Merge tag 'pm-7.2-rc1' of gitolite.kernel.org:pub/scm/linux/kernel/git/rafael/linux-pm
Pull power management updates from Rafael Wysocki:
"Over a half of the changes here are cpufreq updates that include core
modifications, fixes of the old-style governors, new hardware support
in drivers, assorded driver fixes and cleanups, and the removal of one
driver (AMD Elan SC4*).
Apart from that, the intel_idle driver will now be able to avoid
exposing redundant C-states if PC6 is disabled and there are new
sysctl knobs for device suspend/resume watchdog timeouts, hibernation
gets built-in LZ4 support for image compression and there is the usual
collection of assorted fixes and cleanups.
Specifics:
- Fix a race between cpufreq suspend and CPU hotplug during system
shutdown (Tianxiang Chen)
- Avoid redundant target() calls for unchanged limits and fix a typo
in a comment in the cpufreq core (Viresh Kumar)
- Fix concurrency issues related to sysfs attributes access that
affect cpufreq governors using the common governor code (Zhongqiu
Han)
- Simplify frequency limit handling in the conservative cpufreq
governor (Lifeng Zheng)
- Fix descriptions of the conservative governor freq_step tunable and
the ondemand governor sampling_down_factor tunable in the cpufreq
documentation (Pengjie Zhang)
- Fix use-after-free and double free during _OSC evaluation in the
PCC cpufreq driver (Yuho Choi)
- Rework the handling of policy min and max frequency values in the
cpufreq core to allow drivers to specify special initial values for
the scaling_min_freq and scaling_max_freq sysfs attributes (Pierre
Gondois)
- Add cpufreq scaling support for Qualcomm Shikra SoC (Taniya Das,
Imran Shaik).
- Improve the warning message on HWP-disabled hybrid processors
printed by the intel_pstate driver and sync policy->cur during CPU
offline in it (Yohei Kojima, Fushuai Wang)
- Drop cpufreq support for AMD Elan SC4* (Sean Young)
- Minor fixes for cpufreq drivers (Krzysztof Kozlowski, Akashdeep
Kaur, Hans Zhang, Guangshuo Li, Xueqin Luo)
- Clean up dead dependencies on X86 in the cpufreq Kconfig (Julian
Braha)
- Allow the intel_idle driver to avoid exposing C-states that are
redundant when PC6 is disabled (Artem Bityutskiy)
- Fix memory leak and a potential race in the OPP core (Abdun Nihaal,
Di Shen)
- Mark Rust OPP methods as inline (Nicolás Antinori)
- Fix misc device registration failure path in the PM QoS core (Yuho
Choi)
- Add sysctl interface for DPM watchdog timeouts (Tzung-Bi Shih)
- Use complete() instead of complete_all() in device_pm_sleep_init()
to avoid a false-positive warning from lockdep_assert_RT_in_threaded_ctx()
when CONFIG_PROVE_RAW_LOCK_NESTING is enabled (Jiakai Xu)
- Use a flexible array for CRC uncompressed buffers during
hibernation image saving (Rosen Penev)
- Make the LZ4 algorithm available for hibernation compression
(l1rox3)
- Move the preallocate_image() call during hibernation after the
"prepare" phase of the "freeze" transition (Matthew Leach)
- Fix a memory leak in rapl_add_package_cpuslocked() in the
intel_rapl power capping driver and use sysfs_emit() in
cpumask_show() in that driver (Sumeet Pawnikar, Yury Norov)
- Fix ValueError when parsing incomplete device properties in the
pm-graph utility (Gongwei Li)"
* tag 'pm-7.2-rc1' of gitolite.kernel.org:pub/scm/linux/kernel/git/rafael/linux-pm: (40 commits)
PM: dpm_watchdog: Add sysctl interface for DPM watchdog timeouts
PM: QoS: Fix misc device registration unwind
cpufreq: Use policy->min/max init as QoS request
cpufreq: Remove driver default policy->min/max init
cpufreq: Set default policy->min/max values for all drivers
cpufreq: Extract cpufreq_policy_init_qos() function
cpufreq: Documentation: fix conservative governor freq_step description
cpufreq: ti: Add EPROBE_DEFER for K3 SoCs
cpufreq: qcom: Add cpufreq scaling support for Qualcomm Shikra SoC
dt-bindings: cpufreq: Document Qualcomm Shikra SoC EPSS
powercap: intel_rapl: Use sysfs_emit() in cpumask_show()
cpufreq: governor: Fix stale prev_cpu_nice spike when enabling ignore_nice_load
cpufreq: governor: Fix data races on per-CPU idle/nice baselines
PM: hibernate: Use flexible array for CRC uncompressed buffers
powercap: intel_rapl: Fix memory leak in rapl_add_package_cpuslocked()
PM: hibernate: make LZ4 available for hibernation compression
PM: sleep: Use complete() in device_pm_sleep_init()
opp: rust: mark OPP methods as inline
cpufreq: intel_pstate: Improve warning message on HWP-disabled hybrid CPUs
cpufreq: elanfreq: Drop support for AMD Elan SC4*
...
diff --git a/.gitignore b/.gitignore
index 3044b95..38365b8 100644
--- a/.gitignore
+++ b/.gitignore
@@ -57,6 +57,7 @@
*.zst
Module.symvers
dtbs-list
+builtin.order
modules.order
#
@@ -68,6 +69,7 @@
/vmlinux.32
/vmlinux.map
/vmlinux.symvers
+/vmlinux.thinlto-index
/vmlinux.unstripped
/vmlinux-gdb.py
/vmlinuz
@@ -186,5 +188,8 @@
# Rust analyzer configuration
/rust-project.json
+# rustc error message long types
+*.long-type-*.txt
+
# bc language scripts (not LLVM bitcode)
!kernel/time/timeconst.bc
diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt
index 1abdb31..b69482b 100644
--- a/Documentation/admin-guide/kernel-parameters.txt
+++ b/Documentation/admin-guide/kernel-parameters.txt
@@ -5840,13 +5840,13 @@
use a call_rcu[_hurry]() path. Please note, this is for a
normal grace period.
- How to enable it:
+ How to disable it:
- echo 1 > /sys/module/rcutree/parameters/rcu_normal_wake_from_gp
- or pass a boot parameter "rcutree.rcu_normal_wake_from_gp=1"
+ echo 0 > /sys/module/rcutree/parameters/rcu_normal_wake_from_gp
+ or pass a boot parameter "rcutree.rcu_normal_wake_from_gp=0"
- Default is 1 if num_possible_cpus() <= 16 and it is not explicitly
- disabled by the boot parameter passing 0.
+ Default is 1 if it is not explicitly disabled by the boot parameter
+ passing 0.
rcuscale.gp_async= [KNL]
Measure performance of asynchronous
diff --git a/Documentation/dev-tools/autofdo.rst b/Documentation/dev-tools/autofdo.rst
index bcf06e7..ae03c4d 100644
--- a/Documentation/dev-tools/autofdo.rst
+++ b/Documentation/dev-tools/autofdo.rst
@@ -61,6 +61,9 @@
the AutoFDO profile via offline tools.
The support requires a Clang compiler LLVM 17 or later.
+Current supported architectures include x86/x86_64 (via LBR) and
+arm64 (via SPE or ETM).
+
Preparation
===========
@@ -141,6 +144,35 @@
$ perf record --pfm-events RETIRED_TAKEN_BRANCH_INSTRUCTIONS:k -a -N -b -c <count> -o <perf_file> -- <loadtest>
+ - For arm64 with SPE:
+
+ There are a few kernel features that must be enabled to collect SPE profiles on Arm.
+ Below is a list of the required features:
+
+ - CONFIG_ARM_SPE_PMU=y
+ - CONFIG_PID_IN_CONTEXTIDR=y
+ - kpti=off
+
+ Use the following command to generate SPE perf data file::
+
+ $ perf record -e ' arm_spe_0/branch_filter=1,load_filter=0,store_filter=0/' -a -c <count> -N --no-switch-events -o <perf_file> -- <loadtest>
+
+ - For arm64 with ETM trace:
+
+ Follow the instructions in `Linaro OpenCSD document
+ <https://github.com/Linaro/OpenCSD/blob/master/decoder/tests/auto-fdo/autofdo.md>`_
+ to record ETM traces for AutoFDO::
+
+ $ perf record -e cs_etm/@tmc_etr0/k -a -o <etm_perf_file> -- <loadtest>
+ $ perf inject -i <etm_perf_file> -o <perf_file> --itrace=i500009il
+
+ For ARM platforms running Android, follow the instructions in `Android simpleperf
+ document <https://android.googlesource.com/kernel/common/+/refs/heads/android-mainline/gki/aarch64/afdo>`_
+ to record ETM traces for AutoFDO::
+
+ $ simpleperf record -e cs-etm:k -a -o <etm_perf_file> -- <loadtest>
+ $ simpleperf inject -i <etm_perf_file> -o <text_perf_file> --symdir <vmlinux_dir>
+
4) (Optional) Download the raw perf file to the host machine.
5) To generate an AutoFDO profile, two offline tools are available:
@@ -162,6 +194,15 @@
$ llvm-profdata merge -o <profile_file> <profile_1> <profile_2> ... <profile_n>
+ For arm64 SPE, use the following command::
+
+ $ create_llvm_prof --binary=<vmlinux> --profile=<perf_file> --profiler=perf_spe --format=extbinary --out=<profile_file>
+
+ For arm64 ETM, use the following command::
+
+ $ create_llvm_prof --binary=<vmlinux> --profile=<text_perf_file> --profiler=text -format=extbinary -out=<profile_file>
+
+
6) Rebuild the kernel using the AutoFDO profile file with the same config as step 1,
(Note CONFIG_AUTOFDO_CLANG needs to be enabled)::
diff --git a/Documentation/dev-tools/propeller.rst b/Documentation/dev-tools/propeller.rst
index 9219595..e927319 100644
--- a/Documentation/dev-tools/propeller.rst
+++ b/Documentation/dev-tools/propeller.rst
@@ -28,8 +28,10 @@
and the linker(ld.lld).
#. In addition to LLVM toolchain, Propeller requires a profiling
- conversion tool: https://github.com/google/autofdo with a release
- after v0.30.1: https://github.com/google/autofdo/releases/tag/v0.30.1.
+ conversion tool: https://github.com/google/llvm-propeller.
+
+Current supported architectures include x86/X86_64 (via LBR),
+and arm64 (via SPE).
The Propeller optimization process involves the following steps:
@@ -124,17 +126,30 @@
$ perf record --pfm-event RETIRED_TAKEN_BRANCH_INSTRUCTIONS:k -a -N -b -c <count> -o <perf_file> -- <loadtest>
- Note you can repeat the above steps to collect multiple <perf_file>s.
+ - For arm64 with SPE::
+ There are a few kernel features that must be enabled to collect SPE profiles on Arm.
+ Below is a list of the required features:
+
+ - CONFIG_ARM_SPE_PMU=y
+ - CONFIG_PID_IN_CONTEXTIDR=y
+ - kpti=off
+
+ Use the following command to generate SPE perf data file::
+
+ $ perf record -e 'arm_spe_0/branch_filter=1,load_filter=0,store_filter=0/' -a -N -c <count> --no-switch-events -o <perf_file> -- <loadtest>
+
+ Note you can repeat the above steps to collect multiple <perf_file>s.
4) (Optional) Download the raw perf file(s) to the host machine.
-5) Use the create_llvm_prof tool (https://github.com/google/autofdo) to
+5) Use the generate_propeller_profiles tool (https://github.com/google/llvm-propeller) to
generate Propeller profile. ::
- $ create_llvm_prof --binary=<vmlinux> --profile=<perf_file>
- --format=propeller --propeller_output_module_name
- --out=<propeller_profile_prefix>_cc_profile.txt
- --propeller_symorder=<propeller_profile_prefix>_ld_profile.txt
+ $ generate_propeller_profiles \
+ --binary=<vmlinux> --profile=<perf_file> \
+ --format=propeller --propeller_output_module_name \
+ --out=<propeller_profile_prefix>_cc_profile.txt \
+ --propeller_symorder=<propeller_profile_prefix>_ld_profile.txt
"<propeller_profile_prefix>" can be something like "/home/user/dir/any_string".
@@ -146,10 +161,20 @@
you can create a temp list file "<perf_file_list>" with each line
containing one perf file name and run::
- $ create_llvm_prof --binary=<vmlinux> --profile=@<perf_file_list>
- --format=propeller --propeller_output_module_name
- --out=<propeller_profile_prefix>_cc_profile.txt
- --propeller_symorder=<propeller_profile_prefix>_ld_profile.txt
+ $ generate_propeller_profiles \
+ --binary=<vmlinux> --profile=@<perf_file_list> \
+ --format=propeller --propeller_output_module_name \
+ --out=<propeller_profile_prefix>_cc_profile.txt \
+ --propeller_symorder=<propeller_profile_prefix>_ld_profile.txt
+
+ For arm64 SPE, add the option '--profiler=perf_spe', like::
+
+ $ generate_propeller_profiles \
+ --binary=<vmlinux> --profile=<perf_file> \
+ --profiler=perf_spe \
+ --format=propeller --propeller_output_module_name \
+ --out=<propeller_profile_prefix>_cc_profile.txt \
+ --propeller_symorder=<propeller_profile_prefix>_ld_profile.txt
6) Rebuild the kernel using the AutoFDO and Propeller
profiles. ::
diff --git a/Documentation/devicetree/bindings/hwmon/pwm-fan.yaml b/Documentation/devicetree/bindings/hwmon/pwm-fan.yaml
index a84cc3a..6a24851 100644
--- a/Documentation/devicetree/bindings/hwmon/pwm-fan.yaml
+++ b/Documentation/devicetree/bindings/hwmon/pwm-fan.yaml
@@ -63,7 +63,8 @@
description: The PWM that is used to control the fan.
maxItems: 1
- "#cooling-cells": true
+ "#cooling-cells":
+ const: 2
required:
- compatible
diff --git a/Documentation/devicetree/bindings/i2c/i2c-mux-gpio.yaml b/Documentation/devicetree/bindings/i2c/i2c-mux-gpio.yaml
index 4a93d1f..6e44510 100644
--- a/Documentation/devicetree/bindings/i2c/i2c-mux-gpio.yaml
+++ b/Documentation/devicetree/bindings/i2c/i2c-mux-gpio.yaml
@@ -7,7 +7,7 @@
title: GPIO-based I2C Bus Mux
maintainers:
- - Wolfram Sang <wsa@kernel.org>
+ - Peter Korsgaard <peter.korsgaard@barco.com>
description: |
This binding describes an I2C bus multiplexer that uses GPIOs to route the I2C signals.
diff --git a/Documentation/devicetree/bindings/thermal/amlogic,thermal.yaml b/Documentation/devicetree/bindings/thermal/amlogic,thermal.yaml
index 70b2732..e286125 100644
--- a/Documentation/devicetree/bindings/thermal/amlogic,thermal.yaml
+++ b/Documentation/devicetree/bindings/thermal/amlogic,thermal.yaml
@@ -21,7 +21,9 @@
- amlogic,g12a-cpu-thermal
- amlogic,g12a-ddr-thermal
- const: amlogic,g12a-thermal
- - const: amlogic,a1-cpu-thermal
+ - enum:
+ - amlogic,a1-cpu-thermal
+ - amlogic,t7-thermal
reg:
maxItems: 1
@@ -42,12 +44,34 @@
'#thermal-sensor-cells':
const: 0
+ amlogic,secure-monitor:
+ description: phandle to the secure monitor
+ $ref: /schemas/types.yaml#/definitions/phandle-array
+ items:
+ - items:
+ - description: phandle to the secure monitor
+ - description: sensor index to get specific calibration data
+
required:
- compatible
- reg
- interrupts
- clocks
- - amlogic,ao-secure
+
+allOf:
+ - if:
+ properties:
+ compatible:
+ contains:
+ enum:
+ - amlogic,a1-cpu-thermal
+ - amlogic,g12a-thermal
+ then:
+ required:
+ - amlogic,ao-secure
+ else:
+ required:
+ - amlogic,secure-monitor
unevaluatedProperties: false
@@ -62,4 +86,13 @@
#thermal-sensor-cells = <0>;
amlogic,ao-secure = <&sec_AO>;
};
+ - |
+ temperature-sensor@20000 {
+ compatible = "amlogic,t7-thermal";
+ reg = <0x0 0x20000 0x0 0x50>;
+ interrupts = <GIC_SPI 31 IRQ_TYPE_LEVEL_HIGH>;
+ clocks = <&clkc_periphs CLKID_TS>;
+ #thermal-sensor-cells = <0>;
+ amlogic,secure-monitor = <&sm 1>;
+ };
...
diff --git a/Documentation/devicetree/bindings/thermal/qcom-tsens.yaml b/Documentation/devicetree/bindings/thermal/qcom-tsens.yaml
index 7d34ba0..f0efaa8 100644
--- a/Documentation/devicetree/bindings/thermal/qcom-tsens.yaml
+++ b/Documentation/devicetree/bindings/thermal/qcom-tsens.yaml
@@ -56,8 +56,10 @@
- enum:
- qcom,eliza-tsens
- qcom,glymur-tsens
+ - qcom,hawi-tsens
- qcom,kaanapali-tsens
- qcom,milos-tsens
+ - qcom,nord-tsens
- qcom,msm8953-tsens
- qcom,msm8996-tsens
- qcom,msm8998-tsens
@@ -74,6 +76,7 @@
- qcom,sdm630-tsens
- qcom,sdm670-tsens
- qcom,sdm845-tsens
+ - qcom,shikra-tsens
- qcom,sm6115-tsens
- qcom,sm6350-tsens
- qcom,sm6375-tsens
diff --git a/Documentation/devicetree/bindings/thermal/qoriq-thermal.yaml b/Documentation/devicetree/bindings/thermal/qoriq-thermal.yaml
index aa756da..f3b136f 100644
--- a/Documentation/devicetree/bindings/thermal/qoriq-thermal.yaml
+++ b/Documentation/devicetree/bindings/thermal/qoriq-thermal.yaml
@@ -25,6 +25,7 @@
enum:
- fsl,qoriq-tmu
- fsl,imx8mq-tmu
+ - fsl,imx93-tmu
reg:
maxItems: 1
diff --git a/Documentation/devicetree/bindings/thermal/spacemit,k1-tsensor.yaml b/Documentation/devicetree/bindings/thermal/spacemit,k1-tsensor.yaml
new file mode 100644
index 0000000..6dad76a
--- /dev/null
+++ b/Documentation/devicetree/bindings/thermal/spacemit,k1-tsensor.yaml
@@ -0,0 +1,76 @@
+# SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause)
+%YAML 1.2
+---
+$id: http://devicetree.org/schemas/thermal/spacemit,k1-tsensor.yaml#
+$schema: http://devicetree.org/meta-schemas/core.yaml#
+
+title: SpacemiT K1 Thermal Sensor
+
+description:
+ The SpacemiT K1 Thermal Sensor monitors the temperature of the SoC
+ using multiple internal sensors (e.g., soc, package, gpu, clusters).
+
+maintainers:
+ - Shuwei Wu <shuwei.wu@mailbox.org>
+
+$ref: thermal-sensor.yaml#
+
+properties:
+ compatible:
+ const: spacemit,k1-tsensor
+
+ reg:
+ maxItems: 1
+
+ clocks:
+ items:
+ - description: Core clock for thermal sensor
+ - description: Bus clock for thermal sensor
+
+ clock-names:
+ items:
+ - const: core
+ - const: bus
+
+ interrupts:
+ maxItems: 1
+
+ resets:
+ items:
+ - description: Reset for the thermal sensor
+
+ "#thermal-sensor-cells":
+ const: 1
+ description:
+ The first cell indicates the sensor ID.
+ 0 = soc
+ 1 = package
+ 2 = gpu
+ 3 = cluster0
+ 4 = cluster1
+
+required:
+ - compatible
+ - reg
+ - clocks
+ - clock-names
+ - interrupts
+ - resets
+ - "#thermal-sensor-cells"
+
+additionalProperties: false
+
+examples:
+ - |
+ #include <dt-bindings/clock/spacemit,k1-syscon.h>
+
+ thermal@d4018000 {
+ compatible = "spacemit,k1-tsensor";
+ reg = <0xd4018000 0x100>;
+ clocks = <&syscon_apbc CLK_TSEN>,
+ <&syscon_apbc CLK_TSEN_BUS>;
+ clock-names = "core", "bus";
+ interrupts = <61>;
+ resets = <&syscon_apbc RESET_TSEN>;
+ #thermal-sensor-cells = <1>;
+ };
diff --git a/Documentation/devicetree/bindings/thermal/thermal-cooling-devices.yaml b/Documentation/devicetree/bindings/thermal/thermal-cooling-devices.yaml
index b9022f1..28f5818 100644
--- a/Documentation/devicetree/bindings/thermal/thermal-cooling-devices.yaml
+++ b/Documentation/devicetree/bindings/thermal/thermal-cooling-devices.yaml
@@ -44,10 +44,14 @@
properties:
"#cooling-cells":
description:
- Must be 2, in order to specify minimum and maximum cooling state used in
+ Must be 2 or 3. If 2, specifies minimum and maximum cooling state used in
the cooling-maps reference. The first cell is the minimum cooling state
and the second cell is the maximum cooling state requested.
- const: 2
+ If 3, the first cell specifies the thermal mitigation device specifier
+ index for devices that support multiple thermal mitigation mechanisms.
+ The two other cells are respectively the minimum cooling state and the
+ maximum cooling state.
+ enum: [2, 3]
additionalProperties: true
diff --git a/Documentation/devicetree/bindings/thermal/thermal-zones.yaml b/Documentation/devicetree/bindings/thermal/thermal-zones.yaml
index 07d9f57..999ad40 100644
--- a/Documentation/devicetree/bindings/thermal/thermal-zones.yaml
+++ b/Documentation/devicetree/bindings/thermal/thermal-zones.yaml
@@ -211,7 +211,8 @@
device. Using the THERMAL_NO_LIMIT (-1UL) constant in the
cooling-device phandle limit specifier lets the framework
use the minimum and maximum cooling state for that cooling
- device automatically.
+ device automatically. If three arguments are specified,
+ the first argument is the cooling device specifier.
contribution:
$ref: /schemas/types.yaml#/definitions/uint32
diff --git a/Documentation/filesystems/adding-new-filesystems.rst b/Documentation/filesystems/adding-new-filesystems.rst
new file mode 100644
index 0000000..a3d0bf1
--- /dev/null
+++ b/Documentation/filesystems/adding-new-filesystems.rst
@@ -0,0 +1,195 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+.. _adding_new_filesystems:
+
+Adding New Filesystems
+======================
+
+This document describes what is involved in adding a new filesystem to the
+Linux kernel.
+
+Every filesystem merged into the kernel becomes the collective responsibility
+of the VFS maintainers and the wider filesystem development community.
+Experience has shown that filesystems which become unmaintained impose a
+significant and ongoing burden: they are hard or impossible to test, they
+block infrastructure changes because someone must update or preserve old APIs
+for code that nobody is actively looking after, and they accumulate unfixed
+bugs. The requirements and expectations described here are informed by this
+experience and are intended to ensure that new filesystems enter the kernel
+on a sustainable footing.
+
+
+Do You Need a New In-Kernel Filesystem?
+---------------------------------------
+
+Before proposing a new in-kernel filesystem, consider whether one of the
+alternatives might be more appropriate.
+
+ - If an existing in-kernel filesystem covers the same use case, improving it
+ is generally preferred over adding a new implementation. The kernel
+ community favors incremental improvement over parallel implementations.
+
+ - If the filesystem serves a niche audience or has a small user base, a FUSE
+ (Filesystem in Userspace) implementation may be a better fit. FUSE
+ filesystems avoid the long-term kernel maintenance commitment and can be
+ developed and released on their own schedule.
+
+ - If kernel-level performance, reliability, or integration is genuinely
+ required, make the case explicitly. Explain who the users are, what the
+ use case is, and why a FUSE implementation would not be sufficient.
+
+
+Technical Requirements
+----------------------
+
+New filesystems must use current kernel interfaces and practices.
+Submitting a filesystem built on outdated APIs creates an unacceptable
+maintenance debt and is likely to face pushback during review.
+
+Use modern VFS interfaces
+ Do not use interfaces listed in
+ :ref:`Documentation/process/deprecated.rst <deprecated>`.
+
+ Use folios rather than raw page operations for page cache management and
+ iomap rather than buffer heads for block mapping and I/O. See
+ ``Documentation/filesystems/iomap/index.rst`` for iomap documentation.
+
+ Block-based filesystems that need functionality not currently provided by
+ iomap should be prepared to explain why adding that functionality to iomap
+ is infeasible, rather than reimplementing their own block mapping layer.
+
+ Network filesystems should consider using the netfs library
+ (``Documentation/filesystems/netfs_library.rst``), or be prepared to explain
+ why it is not a good fit.
+
+Provide userspace utilities
+ A ``mkfs`` tool is expected so that the filesystem can be created and used
+ by testers and users. A ``fsck`` tool is strongly recommended; while not
+ strictly required for every filesystem type, the ability to verify
+ consistency and repair corruption is an important part of a mature
+ filesystem.
+
+Be testable
+ The filesystem must be testable in a meaningful way. The
+ `fstests <https://git.kernel.org/pub/scm/fs/xfs/xfstests-dev.git>`_
+ framework (also known as xfstests) is the standard testing infrastructure
+ for Linux filesystems and its use is highly recommended. At a minimum,
+ there must be a credible and documented way to test the filesystem and
+ detect regressions. When submitting, include a summary of test results
+ indicating which tests pass, fail, or are not applicable.
+
+Provide documentation
+ A documentation file under ``Documentation/filesystems/`` describing the
+ filesystem, its on-disk format, mount options, and any notable design
+ decisions is recommended.
+
+
+Community and Maintainership Expectations
+-----------------------------------------
+
+Merging a filesystem is a long-term commitment. The kernel community
+needs confidence that the filesystem will be actively maintained after it
+is merged.
+
+Identified maintainers
+ The submission must include a ``MAINTAINERS`` entry with at least one
+ maintainer (``M:``), a mailing list (``L:``), and a git tree (``T:``).
+ Having two or more maintainers is strongly preferred so that coverage
+ does not depend on a single person. The maintainers are expected to be
+ the primary points of contact for the filesystem going forward.
+
+Demonstrated commitment
+ A track record of maintaining kernel code -- for example, in other
+ subsystems -- significantly strengthens the case for a new filesystem.
+ Maintainers who are already known and trusted within the community face
+ less friction during review.
+
+Sustained backing
+ Major filesystems in Linux have organizational or corporate support behind
+ their development. Filesystems that depend entirely on volunteer effort
+ face higher scrutiny about their long-term viability.
+
+Responsiveness
+ The maintainer is expected to respond to bug reports, address review
+ feedback, and adapt the filesystem to VFS infrastructure changes such as
+ folio conversions, iomap migration, and mount API updates. Unresponsive
+ maintainership is one of the primary reasons filesystems end up on the
+ path to deprecation.
+
+User base
+ Clearly describe who the users of this filesystem are and the scale of the
+ user base. Filesystems with a very small or unclear user base face a
+ harder path to acceptance and a higher risk of future deprecation.
+
+Building your track record
+ A practical way to demonstrate many of the qualities above is to maintain
+ the filesystem out-of-tree for a period before requesting a merge. This
+ shows sustained commitment, builds a visible user base, and gives reviewers
+ confidence that the code and its maintainer will persist after merging.
+ That said, it is recognized that for some filesystems the user base grows
+ significantly only after upstreaming, so a compelling case for expected
+ adoption can substitute for a large existing user base.
+
+
+Submission Process
+------------------
+
+This section covers what is specific to filesystem submissions, over and
+above the normal submission advice in
+:ref:`Documentation/process/submitting-patches.rst <submittingpatches>` and
+:ref:`Documentation/process/submit-checklist.rst <submitchecklist>`.
+
+ - Send patches to the linux-fsdevel mailing list
+ (``linux-fsdevel@vger.kernel.org``). CC the relevant VFS maintainers as
+ listed in the ``MAINTAINERS`` file under
+ ``FILESYSTEMS (VFS and infrastructure)``.
+
+ - Structure the submission logically. It is neither acceptable to send one
+ large patch containing the entire filesystem, nor is a replay of the full
+ development history helpful to reviewers. Instead, split the series by
+ topic -- for example: superblock and mount handling, inode operations,
+ directory operations, address space operations, and so on -- so that each
+ patch is reviewable in isolation.
+
+ - Separate any filesystem-specific ioctls into their own patches with
+ dedicated justification. Interfaces beyond those already common across
+ other filesystems will receive additional scrutiny because they are hard
+ to maintain and may conflict with future generic interfaces.
+
+ - Expect thorough review. Filesystem code interacts deeply with the VFS,
+ memory management, and block layers, so reviewers will examine the code
+ carefully. Address all review feedback and be prepared for multiple
+ revision cycles.
+
+ - It may be appropriate to mark the filesystem as experimental in its Kconfig
+ help text for the first few releases to set expectations while the code
+ stabilizes in-tree.
+
+
+Ongoing Obligations
+-------------------
+
+Merging is not the finish line. Maintaining a filesystem in the kernel is an
+ongoing commitment.
+
+ - Adapt to VFS infrastructure changes. The VFS layer evolves continuously;
+ maintainers are expected to keep up with conversions such as folio
+ migration, iomap adoption, and mount API updates.
+
+ - Maintain test coverage. As test suites evolve, the filesystem's test
+ results should be kept current.
+
+ - Handle security issues and regression promptly. Both those reported
+ by ordinary users and those reported by test bots and fuzzing tools.
+ The filesystem must handle corrupted input gracefully without corrupting
+ memory, hanging, or crashing the kernel.
+
+ - Engage with the wider filesystem community. Participate on linux-fsdevel,
+ share approaches to common problems, and look for opportunities to reuse
+ shared infrastructure. It is inappropriate to develop in isolation on a
+ private list and surface patches only at merge time.
+
+ - Filesystems that become unmaintained -- where the maintainer stops
+ responding, infrastructure changes go unadapted, and testing becomes
+ impossible -- are candidates for deprecation and eventual removal from
+ the kernel.
diff --git a/Documentation/filesystems/index.rst b/Documentation/filesystems/index.rst
index fc7254d..1f71cf1 100644
--- a/Documentation/filesystems/index.rst
+++ b/Documentation/filesystems/index.rst
@@ -43,6 +43,7 @@
caching/index
porting
+ adding-new-filesystems
Filesystem support layers
=========================
diff --git a/Documentation/filesystems/locking.rst b/Documentation/filesystems/locking.rst
index 8421ea2..a27aca4 100644
--- a/Documentation/filesystems/locking.rst
+++ b/Documentation/filesystems/locking.rst
@@ -416,20 +416,6 @@
lm_breaker_timedout yes no no
====================== ============= ================= =========
-buffer_head
-===========
-
-prototypes::
-
- void (*b_end_io)(struct buffer_head *bh, int uptodate);
-
-locking rules:
-
-called from interrupts. In other words, extreme care is needed here.
-bh is locked, but that's all warranties we have here. Currently only RAID1,
-highmem, fs/buffer.c, and fs/ntfs/aops.c are providing these. Block devices
-call this method upon the IO completion.
-
block_device_operations
=======================
prototypes::
diff --git a/Documentation/filesystems/porting.rst b/Documentation/filesystems/porting.rst
index fdf0744..d13f0a2 100644
--- a/Documentation/filesystems/porting.rst
+++ b/Documentation/filesystems/porting.rst
@@ -1297,7 +1297,6 @@
- kern_path_locked -> start_removing_path
- kern_path_create -> start_creating_path
- user_path_create -> start_creating_user_path
-- user_path_locked_at -> start_removing_user_path_at
- done_path_create -> end_creating_path
---
@@ -1385,3 +1384,20 @@
yet, see if any of the exported primitives could be used instead of
the entire loop. You still need to hold ->i_lock of the inode over
either form of manual loop.
+
+---
+
+**mandatory**
+
+d_alloc_parallel() no longer requires a waitqueue_head.
+
+---
+
+**mandatory**
+
+d_dispose_if_unused() is gone; use __move_to_shrink_list() if you really
+need that functionality, but watch out for memory safety issues - just
+as with d_dispose_if_unused() these are not trivial; with this variant
+of API it's more explicit, since grabbing ->d_lock is caller-side, but
+d_dispose_if_unused() had all the same issues. It's a low-level primitive;
+use only if you have no alternative.
diff --git a/Documentation/filesystems/proc.rst b/Documentation/filesystems/proc.rst
index db6167b..5006644 100644
--- a/Documentation/filesystems/proc.rst
+++ b/Documentation/filesystems/proc.rst
@@ -52,6 +52,7 @@
4 Configuring procfs
4.1 Mount options
+ 4.2 Mount restrictions
5 Filesystem behavior
@@ -2425,7 +2426,9 @@
information about processes information, just add identd to this group.
subset=pid hides all top level files and directories in the procfs that
-are not related to tasks.
+are not related to tasks. This option cannot be changed on an existing
+procfs instance because overmounts that existed before the change could
+otherwise remain reachable after the top level procfs entries are hidden.
pidns= specifies a pid namespace (either as a string path to something like
`/proc/$pid/ns/pid`, or a file descriptor when using `FSCONFIG_SET_FD`) that
@@ -2434,6 +2437,20 @@
namespace of an existing procfs instance cannot be modified (attempting to do
so will give an `-EBUSY` error).
+4.2 Mount restrictions
+--------------------------
+
+If user namespaces are in use, the kernel additionally checks the instances of
+procfs available to the mounter and will not allow procfs to be mounted if:
+
+ 1. This mount is not fully visible unless the new procfs is going to be
+ mounted with subset=pid option.
+
+ a. Its root directory is not the root directory of the filesystem.
+ b. If any file or non-empty procfs directory is hidden by another mount.
+
+ 2. A new mount overrides the readonly option or any option from atime family.
+
Chapter 5: Filesystem behavior
==============================
diff --git a/Documentation/kbuild/makefiles.rst b/Documentation/kbuild/makefiles.rst
index 24a4708..7521cae 100644
--- a/Documentation/kbuild/makefiles.rst
+++ b/Documentation/kbuild/makefiles.rst
@@ -1285,8 +1285,39 @@
In this example, the file target maketools will be processed
before descending down in the subdirectories.
-See also chapter XXX-TODO that describes how kbuild supports
-generating offset header files.
+Generating offset header files
+------------------------------
+
+The ``include/generated/asm-offsets.h`` header exposes C structure
+member offsets and other compile-time constants to assembly code. It
+is generated from ``arch/$(SRCARCH)/kernel/asm-offsets.c``.
+
+The source file uses ``DEFINE()``, ``OFFSET()``, ``BLANK()`` and
+``COMMENT()`` from ``<linux/kbuild.h>``. These emit marker strings
+through inline asm that Kbuild extracts from the compiled assembly
+output.
+
+Example::
+
+ #include <linux/kbuild.h>
+ #include <linux/sched.h>
+
+ int main(void)
+ {
+ OFFSET(TSK_ACTIVE_MM, task_struct, active_mm);
+ DEFINE(THREAD_SIZE, THREAD_SIZE);
+ BLANK();
+ return 0;
+ }
+
+The rules are defined in the top-level ``Kbuild`` and
+``scripts/Makefile.lib``. The header is built during Kbuild's
+``prepare`` phase, after ``archprepare`` and before descending into
+subdirectories.
+
+The same mechanism generates ``include/generated/bounds.h`` from
+``kernel/bounds.c`` and ``include/generated/rq-offsets.h`` from
+``kernel/sched/rq-offsets.c``.
List directories to visit when descending
-----------------------------------------
@@ -1690,9 +1721,3 @@
- Updates by Kai Germaschewski <kai@tp1.ruhr-uni-bochum.de>
- Updates by Sam Ravnborg <sam@ravnborg.org>
- Language QA by Jan Engelhardt <jengelh@gmx.de>
-
-TODO
-====
-
-- Generating offset header files.
-- Add more variables to chapters 7 or 9?
diff --git a/Documentation/process/changes.rst b/Documentation/process/changes.rst
index 9a99037..b9afce7 100644
--- a/Documentation/process/changes.rst
+++ b/Documentation/process/changes.rst
@@ -36,7 +36,7 @@
binutils 2.30 ld -v
bison 2.0 bison --version
btrfs-progs 0.18 btrfs --version
-Clang/LLVM (optional) 15.0.0 clang --version
+Clang/LLVM (optional) 17.0.1 clang --version
e2fsprogs 1.41.4 e2fsck -V
flex 2.5.35 flex --version
gdb 7.2 gdb --version
diff --git a/Documentation/rust/testing.rst b/Documentation/rust/testing.rst
index f43cb77..e3943ac 100644
--- a/Documentation/rust/testing.rst
+++ b/Documentation/rust/testing.rst
@@ -140,11 +140,15 @@
These tests are introduced by the ``kunit_tests`` procedural macro, which takes
the name of the test suite as an argument.
+Each test suite should be guarded by a Kconfig option in
+``rust/kernel/Kconfig.test``.
+
For instance, assume we want to test the function ``f`` from the documentation
tests section. We could write, in the same file where we have our function:
.. code-block:: rust
+ #[cfg(CONFIG_RUST_MYMOD_KUNIT_TEST)]
#[kunit_tests(rust_kernel_mymod)]
mod tests {
use super::*;
@@ -173,6 +177,7 @@
.. code-block:: rust
+ #[cfg(CONFIG_RUST_MYMOD_KUNIT_TEST)]
#[kunit_tests(rust_kernel_mymod)]
mod tests {
use super::*;
diff --git a/Documentation/trace/ftrace.rst b/Documentation/trace/ftrace.rst
index b9efb14..2ed1b96 100644
--- a/Documentation/trace/ftrace.rst
+++ b/Documentation/trace/ftrace.rst
@@ -1624,7 +1624,7 @@
=> blk_queue_bio
=> submit_bio_noacct
=> submit_bio
- => submit_bh
+ => bh_submit
=> __ext3_get_inode_loc
=> ext3_iget
=> ext3_lookup
@@ -1909,7 +1909,7 @@
=> blk_queue_bio
=> submit_bio_noacct
=> submit_bio
- => submit_bh
+ => bh_submit
=> ext3_bread
=> ext3_dir_bread
=> htree_dirblock_to_tree
diff --git a/Documentation/translations/it_IT/process/changes.rst b/Documentation/translations/it_IT/process/changes.rst
index 7e93833..7ee54c9 100644
--- a/Documentation/translations/it_IT/process/changes.rst
+++ b/Documentation/translations/it_IT/process/changes.rst
@@ -33,7 +33,7 @@
Programma Versione minima Comando per verificare la versione
====================== ================= ========================================
GNU C 8.1 gcc --version
-Clang/LLVM (optional) 13.0.0 clang --version
+Clang/LLVM (optional) 17.0.1 clang --version
Rust (opzionale) 1.78.0 rustc --version
bindgen (opzionale) 0.65.1 bindgen --version
GNU make 4.0 make --version
diff --git a/Documentation/translations/pt_BR/process/changes.rst b/Documentation/translations/pt_BR/process/changes.rst
index 1964c1c..6bbfe60 100644
--- a/Documentation/translations/pt_BR/process/changes.rst
+++ b/Documentation/translations/pt_BR/process/changes.rst
@@ -31,7 +31,7 @@
Programa Versão mínima Comando para verificar a versão
====================== =============== ========================================
GNU C 8.1 gcc --version
-Clang/LLVM (optional) 15.0.0 clang --version
+Clang/LLVM (optional) 17.0.1 clang --version
Rust (optional) 1.78.0 rustc --version
bindgen (optional) 0.65.1 bindgen --version
GNU make 4.0 make --version
diff --git a/MAINTAINERS b/MAINTAINERS
index 355e4ab..8c07f13 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -9916,7 +9916,7 @@
F: Documentation/filesystems/nfs/exporting.rst
F: fs/exportfs/
F: fs/fhandle.c
-F: include/linux/exportfs.h
+F: include/linux/exportfs*.h
FILESYSTEMS [IDMAPPED MOUNTS]
M: Christian Brauner <brauner@kernel.org>
@@ -10772,6 +10772,7 @@
M: Peter Korsgaard <peter.korsgaard@barco.com>
L: linux-i2c@vger.kernel.org
S: Supported
+F: Documentation/devicetree/bindings/i2c/i2c-mux-gpio.yaml
F: Documentation/i2c/muxes/i2c-mux-gpio.rst
F: drivers/i2c/muxes/i2c-mux-gpio.c
F: include/linux/platform_data/i2c-mux-gpio.h
@@ -12094,11 +12095,11 @@
F: drivers/i2c/busses/i2c-parport.c
I2C SUBSYSTEM
-M: Wolfram Sang <wsa+renesas@sang-engineering.com>
+M: Andi Shyti <andi.shyti@kernel.org>
L: linux-i2c@vger.kernel.org
S: Maintained
Q: https://patchwork.ozlabs.org/project/linux-i2c/list/
-T: git git://git.kernel.org/pub/scm/linux/kernel/git/wsa/linux.git
+T: git git://git.kernel.org/pub/scm/linux/kernel/git/andi.shyti/linux.git
F: Documentation/i2c/
F: drivers/i2c/*
F: include/dt-bindings/i2c/i2c.h
@@ -20549,6 +20550,7 @@
M: Manivannan Sadhasivam <mani@kernel.org>
M: Krzysztof Wilczyński <kwilczynski@kernel.org>
R: Kishon Vijay Abraham I <kishon@kernel.org>
+R: Frank Li <Frank.Li@kernel.org>
L: linux-pci@vger.kernel.org
S: Supported
Q: https://patchwork.kernel.org/project/linux-pci/list/
@@ -23398,6 +23400,10 @@
R: Alice Ryhl <aliceryhl@google.com>
R: Trevor Gross <tmgross@umich.edu>
R: Danilo Krummrich <dakr@kernel.org>
+R: Daniel Almeida <daniel.almeida@collabora.com>
+R: Tamir Duberstein <tamird@kernel.org>
+R: Alexandre Courbot <acourbot@nvidia.com>
+R: Onur Özkan <work@onurozkan.dev>
L: rust-for-linux@vger.kernel.org
S: Supported
W: https://rust-for-linux.com
@@ -23427,6 +23433,13 @@
F: rust/kernel/alloc.rs
F: rust/kernel/alloc/
+RUST [BITFIELD]
+M: Alexandre Courbot <acourbot@nvidia.com>
+R: Yury Norov <yury.norov@gmail.com>
+L: rust-for-linux@vger.kernel.org
+S: Maintained
+F: rust/kernel/bitfield.rs
+
RUST [INTEROP]
M: Joel Fernandes <joelagnelf@nvidia.com>
M: Alexandre Courbot <acourbot@nvidia.com>
@@ -25925,7 +25938,6 @@
SYNOPSYS DESIGNWARE I2C DRIVER
M: Mika Westerberg <mika.westerberg@linux.intel.com>
R: Andy Shevchenko <andriy.shevchenko@linux.intel.com>
-R: Jan Dabros <jsd@semihalf.com>
L: linux-i2c@vger.kernel.org
S: Supported
F: drivers/i2c/busses/i2c-designware-*
diff --git a/Makefile b/Makefile
index e156e269..5e73bba 100644
--- a/Makefile
+++ b/Makefile
@@ -2,7 +2,7 @@
VERSION = 7
PATCHLEVEL = 1
SUBLEVEL = 0
-EXTRAVERSION = -rc7
+EXTRAVERSION =
NAME = Baby Opossum Posse
# *DOCUMENTATION*
@@ -293,6 +293,7 @@
clean-targets := %clean mrproper cleandocs
no-dot-config-targets := $(clean-targets) \
cscope gtags TAGS tags help% %docs check% coccicheck \
+ kconfig-sym-check \
$(version_h) headers headers_% archheaders archscripts \
%asm-generic kernelversion %src-pkg dt_binding_check \
outputmakefile rustavailable rustfmt rustfmtcheck \
@@ -1074,11 +1075,16 @@
endif
ifdef CONFIG_LTO_CLANG
-ifdef CONFIG_LTO_CLANG_THIN
-CC_FLAGS_LTO := -flto=thin -fsplit-lto-unit
-KBUILD_LDFLAGS += $(call ld-option,--lto-whole-program-visibility -mllvm -always-rename-promoted-locals=false)
-else
+ifdef CONFIG_LTO_CLANG_FULL
CC_FLAGS_LTO := -flto
+else
+CC_FLAGS_LTO := -flto=thin -fsplit-lto-unit
+
+# These LLVM options were initially added with only in-process ThinLTO
+# support, so avoid distributed ThinLTO support for now.
+ifdef CONFIG_LTO_CLANG_THIN
+KBUILD_LDFLAGS += $(call ld-option,--lto-whole-program-visibility -mllvm -always-rename-promoted-locals=false)
+endif
endif
CC_FLAGS_LTO += -fvisibility=hidden
@@ -1294,7 +1300,7 @@
KBUILD_VMLINUX_OBJS := built-in.a $(patsubst %/, %/lib.a, $(filter %/, $(libs-y)))
KBUILD_VMLINUX_LIBS := $(filter-out %/, $(libs-y))
-export KBUILD_VMLINUX_LIBS
+export KBUILD_VMLINUX_OBJS KBUILD_VMLINUX_LIBS
export KBUILD_LDS := arch/$(SRCARCH)/kernel/vmlinux.lds
ifdef CONFIG_TRIM_UNUSED_KSYMS
@@ -1303,16 +1309,12 @@
KBUILD_MODULES := y
endif
-# '$(AR) mPi' needs 'T' to workaround the bug of llvm-ar <= 14
-quiet_cmd_ar_vmlinux.a = AR $@
- cmd_ar_vmlinux.a = \
- rm -f $@; \
- $(AR) cDPrST $@ $(KBUILD_VMLINUX_OBJS); \
- $(AR) mPiT $$($(AR) t $@ | sed -n 1p) $@ $$($(AR) t $@ | grep -F -f $(srctree)/scripts/head-object-list.txt)
+PHONY += vmlinux_a
+vmlinux_a: $(KBUILD_VMLINUX_OBJS) scripts/head-object-list.txt FORCE
+ $(Q)$(MAKE) -f $(srctree)/scripts/Makefile.vmlinux_a
-targets += vmlinux.a
-vmlinux.a: $(KBUILD_VMLINUX_OBJS) scripts/head-object-list.txt FORCE
- $(call if_changed,ar_vmlinux.a)
+vmlinux.a: vmlinux_a
+ @:
PHONY += vmlinux_o
vmlinux_o: vmlinux.a $(KBUILD_VMLINUX_LIBS)
@@ -1691,6 +1693,7 @@
CLEAN_FILES += vmlinux.symvers modules-only.symvers \
modules.builtin modules.builtin.modinfo modules.nsdeps \
modules.builtin.ranges vmlinux.o.map vmlinux.unstripped \
+ vmlinux.thinlto-index builtin.order \
compile_commands.json rust/test \
rust-project.json .vmlinux.objs .vmlinux.export.c \
.builtin-dtbs-list .builtin-dtbs.S
@@ -1706,7 +1709,8 @@
vmlinux-gdb.py \
rpmbuild \
rust/libmacros.so rust/libmacros.dylib \
- rust/libpin_init_internal.so rust/libpin_init_internal.dylib
+ rust/libpin_init_internal.so rust/libpin_init_internal.dylib \
+ rust/libzerocopy_derive.so rust/libzerocopy_derive.dylib
# clean - Delete most, but leave enough to build external modules
#
@@ -1801,14 +1805,15 @@
echo ' (default: $(INSTALL_HDR_PATH))'; \
echo ''
@echo 'Static analysers:'
- @echo ' checkstack - Generate a list of stack hogs and consider all functions'
- @echo ' with a stack size larger than MINSTACKSIZE (default: 100)'
- @echo ' versioncheck - Sanity check on version.h usage'
- @echo ' includecheck - Check for duplicate included header files'
- @echo ' headerdep - Detect inclusion cycles in headers'
- @echo ' coccicheck - Check with Coccinelle'
- @echo ' clang-analyzer - Check with clang static analyzer'
- @echo ' clang-tidy - Check with clang-tidy'
+ @echo ' checkstack - Generate a list of stack hogs and consider all functions'
+ @echo ' with a stack size larger than MINSTACKSIZE (default: 100)'
+ @echo ' versioncheck - Sanity check on version.h usage'
+ @echo ' includecheck - Check for duplicate included header files'
+ @echo ' headerdep - Detect inclusion cycles in headers'
+ @echo ' coccicheck - Check with Coccinelle'
+ @echo ' kconfig-sym-check - Check for dangling Kconfig symbol references'
+ @echo ' clang-analyzer - Check with clang static analyzer'
+ @echo ' clang-tidy - Check with clang-tidy'
@echo ''
@echo 'Tools:'
@echo ' nsdeps - Generate missing symbol namespace dependencies'
@@ -1957,6 +1962,8 @@
-path $(srctree)/rust/proc-macro2 \
-o -path $(srctree)/rust/quote \
-o -path $(srctree)/rust/syn \
+ -o -path $(srctree)/rust/zerocopy \
+ -o -path $(srctree)/rust/zerocopy-derive \
\) -prune -o \
-type f -a -name '*.rs' -a ! -name '*generated*' -print \
| xargs $(RUSTFMT) $(rustfmt_flags)
@@ -2152,7 +2159,7 @@
$(call cmd,rmfiles)
@find . $(RCS_FIND_IGNORE) \
\( -name '*.[aios]' -o -name '*.rsi' -o -name '*.ko' -o -name '.*.cmd' \
- -o -name '*.ko.*' \
+ -o -name '*.ko.*' -o -name '*.o.thinlto.bc' \
-o -name '*.dtb' -o -name '*.dtbo' \
-o -name '*.dtb.S' -o -name '*.dtbo.S' \
-o -name '*.dt.yaml' -o -name 'dtbs-list' \
@@ -2165,6 +2172,7 @@
-o -name '*.c.[012]*.*' \
-o -name '*.ll' \
-o -name '*.gcno' \
+ -o -name '*.long-type-*.txt' \
\) -type f -print \
-o -name '.tmp_*' -print \
| xargs rm -rf
@@ -2228,7 +2236,7 @@
# Scripts to check various things for consistency
# ---------------------------------------------------------------------------
-PHONY += includecheck versioncheck coccicheck
+PHONY += includecheck versioncheck coccicheck kconfig-sym-check
includecheck:
find $(srctree)/* $(RCS_FIND_IGNORE) \
@@ -2243,6 +2251,9 @@
coccicheck:
$(Q)$(BASH) $(srctree)/scripts/$@
+kconfig-sym-check:
+ $(Q)$(PERL) $(srctree)/scripts/kconfig/kconfig-sym-check.pl $(srctree) $(KCONFIG_SYM_CHECK_EXCLUDES)
+
PHONY += checkstack kernelrelease kernelversion image_name
# UML needs a little special treatment here. It wants to use the host
diff --git a/arch/Kconfig b/arch/Kconfig
index e868800..99c2017 100644
--- a/arch/Kconfig
+++ b/arch/Kconfig
@@ -806,9 +806,6 @@
depends on $(success,$(AR) --help | head -n 1 | grep -qi llvm)
depends on ARCH_SUPPORTS_LTO_CLANG
depends on !FTRACE_MCOUNT_USE_RECORDMCOUNT
- # https://github.com/ClangBuiltLinux/linux/issues/1721
- depends on (!KASAN || KASAN_HW_TAGS || CLANG_VERSION >= 170000) || !DEBUG_INFO
- depends on (!KCOV || CLANG_VERSION >= 170000) || !DEBUG_INFO
depends on !GCOV_KERNEL
help
The compiler and Kconfig options support building with Clang's
@@ -861,15 +858,30 @@
https://clang.llvm.org/docs/ThinLTO.html
If unsure, say Y.
-endchoice
-config ARCH_SUPPORTS_AUTOFDO_CLANG
- bool
+config LTO_CLANG_THIN_DIST
+ bool "Clang ThinLTO in distributed mode (EXPERIMENTAL)"
+ depends on HAS_LTO_CLANG && ARCH_SUPPORTS_LTO_CLANG_THIN
+ select LTO_CLANG
+ help
+ This option enables Clang's ThinLTO in distributed build mode.
+ In this mode, the linker performs the thin-link, generating
+ ThinLTO index files. Subsequently, the build system explicitly
+ invokes ThinLTO backend compilation using these index files
+ and pre-linked IR objects. The resulting native object files
+ are with the .thinlto-native.o suffix.
+
+ This build mode offers improved visibility into the ThinLTO
+ process through explicit subcommand exposure. It also makes
+ final native object files directly available, benefiting
+ tools like objtool and kpatch. Additionally, it provides
+ crucial granular control over back-end options, enabling
+ module-specific compiler options, and simplifies debugging.
+endchoice
config AUTOFDO_CLANG
bool "Enable Clang's AutoFDO build (EXPERIMENTAL)"
- depends on ARCH_SUPPORTS_AUTOFDO_CLANG
- depends on CC_IS_CLANG && CLANG_VERSION >= 170000
+ depends on CC_IS_CLANG
help
This option enables Clang’s AutoFDO build. When
an AutoFDO profile is specified in variable
@@ -883,13 +895,10 @@
If unsure, say N.
-config ARCH_SUPPORTS_PROPELLER_CLANG
- bool
-
config PROPELLER_CLANG
bool "Enable Clang's Propeller build"
- depends on ARCH_SUPPORTS_PROPELLER_CLANG
depends on CC_IS_CLANG && CLANG_VERSION >= 190000
+ depends on $(cc-option,-fbasic-block-sections=list=/dev/null)
help
This option enables Clang’s Propeller build. When the Propeller
profiles is specified in variable CLANG_PROPELLER_PROFILE_PREFIX
diff --git a/arch/alpha/include/uapi/asm/errno.h b/arch/alpha/include/uapi/asm/errno.h
index 6791f65..1a99f38 100644
--- a/arch/alpha/include/uapi/asm/errno.h
+++ b/arch/alpha/include/uapi/asm/errno.h
@@ -127,4 +127,6 @@
#define EHWPOISON 139 /* Memory page has hardware error */
+#define EFTYPE 140 /* Wrong file type for the intended operation */
+
#endif
diff --git a/arch/alpha/include/uapi/asm/fcntl.h b/arch/alpha/include/uapi/asm/fcntl.h
index 50bdc8e..c7e1c5c 100644
--- a/arch/alpha/include/uapi/asm/fcntl.h
+++ b/arch/alpha/include/uapi/asm/fcntl.h
@@ -2,20 +2,20 @@
#ifndef _ALPHA_FCNTL_H
#define _ALPHA_FCNTL_H
-#define O_CREAT 01000 /* not fcntl */
-#define O_TRUNC 02000 /* not fcntl */
-#define O_EXCL 04000 /* not fcntl */
-#define O_NOCTTY 010000 /* not fcntl */
+#define O_CREAT (1 << 9) /* not fcntl */
+#define O_TRUNC (1 << 10) /* not fcntl */
+#define O_EXCL (1 << 11) /* not fcntl */
+#define O_NOCTTY (1 << 12) /* not fcntl */
-#define O_NONBLOCK 00004
-#define O_APPEND 00010
-#define O_DSYNC 040000 /* used to be O_SYNC, see below */
-#define O_DIRECTORY 0100000 /* must be a directory */
-#define O_NOFOLLOW 0200000 /* don't follow links */
-#define O_LARGEFILE 0400000 /* will be set by the kernel on every open */
-#define O_DIRECT 02000000 /* direct disk access - should check with OSF/1 */
-#define O_NOATIME 04000000
-#define O_CLOEXEC 010000000 /* set close_on_exec */
+#define O_NONBLOCK (1 << 2)
+#define O_APPEND (1 << 3)
+#define O_DSYNC (1 << 14) /* used to be O_SYNC, see below */
+#define O_DIRECTORY (1 << 15) /* must be a directory */
+#define O_NOFOLLOW (1 << 16) /* don't follow links */
+#define O_LARGEFILE (1 << 17) /* will be set by the kernel on every open */
+#define O_DIRECT (1 << 19) /* direct disk access - should check with OSF/1 */
+#define O_NOATIME (1 << 20)
+#define O_CLOEXEC (1 << 21) /* set close_on_exec */
/*
* Before Linux 2.6.33 only O_DSYNC semantics were implemented, but using
* the O_SYNC flag. We continue to use the existing numerical value
@@ -29,11 +29,11 @@
*
* Note: __O_SYNC must never be used directly.
*/
-#define __O_SYNC 020000000
+#define __O_SYNC (1 << 22)
#define O_SYNC (__O_SYNC|O_DSYNC)
-#define O_PATH 040000000
-#define __O_TMPFILE 0100000000
+#define O_PATH (1 << 23)
+#define __O_TMPFILE (1 << 24)
#define F_GETLK 7
#define F_SETLK 8
diff --git a/arch/arm/Kconfig.platforms b/arch/arm/Kconfig.platforms
index 5c19c1f..386eccc 100644
--- a/arch/arm/Kconfig.platforms
+++ b/arch/arm/Kconfig.platforms
@@ -8,16 +8,12 @@
config ARCH_MULTI_V4
bool "ARMv4 based platforms (FA526, StrongARM)"
depends on !ARCH_MULTI_V6_V7
- # https://github.com/llvm/llvm-project/issues/50764
- depends on !LD_IS_LLD || LLD_VERSION >= 160000
select ARCH_MULTI_V4_V5
select CPU_FA526 if !(CPU_SA110 || CPU_SA1100)
config ARCH_MULTI_V4T
bool "ARMv4T based platforms (ARM720T, ARM920T, ...)"
depends on !ARCH_MULTI_V6_V7
- # https://github.com/llvm/llvm-project/issues/50764
- depends on !LD_IS_LLD || LLD_VERSION >= 160000
select ARCH_MULTI_V4_V5
select CPU_ARM920T if !(CPU_ARM7TDMI || CPU_ARM720T || \
CPU_ARM740T || CPU_ARM9TDMI || CPU_ARM922T || \
diff --git a/arch/arm/include/asm/io.h b/arch/arm/include/asm/io.h
index bae5edf..e6bd9e7 100644
--- a/arch/arm/include/asm/io.h
+++ b/arch/arm/include/asm/io.h
@@ -56,8 +56,19 @@ void __raw_readsl(const volatile void __iomem *addr, void *data, int longlen);
* the bus. Rather than special-case the machine, just let the compiler
* generate the access for CPUs prior to ARMv6.
*/
-#define __raw_readw(a) (__chk_io_ptr(a), *(volatile unsigned short __force *)(a))
-#define __raw_writew(v,a) ((void)(__chk_io_ptr(a), *(volatile unsigned short __force *)(a) = (v)))
+#define __raw_writew __raw_writew
+static __no_kasan_or_inline void __raw_writew(u16 val, volatile void __iomem *addr)
+{
+ __chk_io_ptr(addr);
+ *(volatile unsigned short __force *)addr = val;
+}
+
+#define __raw_readw __raw_readw
+static __no_kasan_or_inline u16 __raw_readw(const volatile void __iomem *addr)
+{
+ __chk_io_ptr(addr);
+ return *(const volatile unsigned short __force *)addr;
+}
#else
/*
* When running under a hypervisor, we want to avoid I/O accesses with
diff --git a/arch/arm/include/uapi/asm/fcntl.h b/arch/arm/include/uapi/asm/fcntl.h
index e6b5d71..b576ff0 100644
--- a/arch/arm/include/uapi/asm/fcntl.h
+++ b/arch/arm/include/uapi/asm/fcntl.h
@@ -2,10 +2,10 @@
#ifndef _ARM_FCNTL_H
#define _ARM_FCNTL_H
-#define O_DIRECTORY 040000 /* must be a directory */
-#define O_NOFOLLOW 0100000 /* don't follow links */
-#define O_DIRECT 0200000 /* direct disk access hint - currently ignored */
-#define O_LARGEFILE 0400000
+#define O_DIRECTORY (1 << 14) /* must be a directory */
+#define O_NOFOLLOW (1 << 15) /* don't follow links */
+#define O_DIRECT (1 << 16) /* direct disk access hint - currently ignored */
+#define O_LARGEFILE (1 << 17)
#include <asm-generic/fcntl.h>
diff --git a/arch/arm/kernel/entry-armv.S b/arch/arm/kernel/entry-armv.S
index ef6a657..a3d050c 100644
--- a/arch/arm/kernel/entry-armv.S
+++ b/arch/arm/kernel/entry-armv.S
@@ -567,7 +567,7 @@
@ are using KASAN
mov_l r2, KASAN_SHADOW_OFFSET
add r2, r2, ip, lsr #KASAN_SHADOW_SCALE_SHIFT
- ldr r2, [r2]
+ ldrb r2, [r2]
#endif
#endif
diff --git a/arch/arm/kernel/hibernate.c b/arch/arm/kernel/hibernate.c
index 38a90a3..231a76a 100644
--- a/arch/arm/kernel/hibernate.c
+++ b/arch/arm/kernel/hibernate.c
@@ -21,6 +21,7 @@
#include <asm/suspend.h>
#include <asm/page.h>
#include <asm/sections.h>
+#include <asm/uaccess.h>
#include "reboot.h"
int pfn_is_nosave(unsigned long pfn)
@@ -82,6 +83,15 @@ static void notrace arch_restore_image(void *unused)
{
struct pbe *pbe;
+ /*
+ * With CONFIG_CPU_TTBR0_PAN enabled, TTBCR.EPD0 is set to block
+ * TTBR0 page-table walks. The identity mapping used here lives at
+ * low (user-space) virtual addresses and is only reachable via
+ * TTBR0, so re-enable those walks before switching page tables.
+ * On non-PAN kernels this is a no-op.
+ */
+ if (IS_ENABLED(CONFIG_CPU_TTBR0_PAN))
+ uaccess_save_and_enable();
cpu_switch_mm(idmap_pgd, &init_mm);
for (pbe = restore_pblist; pbe; pbe = pbe->next)
copy_page(pbe->orig_address, pbe->address);
diff --git a/arch/arm/mach-rockchip/platsmp.c b/arch/arm/mach-rockchip/platsmp.c
index f432d22..f659d89 100644
--- a/arch/arm/mach-rockchip/platsmp.c
+++ b/arch/arm/mach-rockchip/platsmp.c
@@ -34,6 +34,7 @@ static int ncores;
static struct regmap *pmu;
static int has_pmu = true;
+static struct reset_control *cpu_rstc[4];
static int pmu_power_domain_is_on(int pd)
{
@@ -64,9 +65,11 @@ static struct reset_control *rockchip_get_core_reset(int cpu)
static int pmu_set_power_domain(int pd, bool on)
{
u32 val = (on) ? 0 : BIT(pd);
- struct reset_control *rstc = rockchip_get_core_reset(pd);
+ struct reset_control *rstc;
int ret;
+ rstc = pd < ARRAY_SIZE(cpu_rstc) ? cpu_rstc[pd] : ERR_PTR(-EINVAL);
+
if (IS_ERR(rstc) && read_cpuid_part() != ARM_CPU_PART_CORTEX_A9) {
pr_err("%s: could not get reset control for core %d\n",
__func__, pd);
@@ -100,11 +103,8 @@ static int pmu_set_power_domain(int pd, bool on)
}
}
- if (!IS_ERR(rstc)) {
- if (on)
- reset_control_deassert(rstc);
- reset_control_put(rstc);
- }
+ if (!IS_ERR(rstc) && on)
+ reset_control_deassert(rstc);
return 0;
}
@@ -312,6 +312,10 @@ static void __init rockchip_smp_prepare_cpus(unsigned int max_cpus)
ncores = ((l2ctlr >> 24) & 0x3) + 1;
}
+ /* Collect cpu core reset control for each core */
+ for (i = 0; i < ncores; i++)
+ cpu_rstc[i] = rockchip_get_core_reset(i);
+
/* Make sure that all cores except the first are really off */
for (i = 1; i < ncores; i++)
pmu_set_power_domain(0 + i, false);
diff --git a/arch/arm/mm/idmap.c b/arch/arm/mm/idmap.c
index 4a833e8..70403e9 100644
--- a/arch/arm/mm/idmap.c
+++ b/arch/arm/mm/idmap.c
@@ -11,6 +11,7 @@
#include <asm/pgalloc.h>
#include <asm/sections.h>
#include <asm/system_info.h>
+#include <asm/uaccess.h>
/*
* Note: accesses outside of the kernel image and the identity map area
@@ -133,6 +134,17 @@ early_initcall(init_static_idmap);
*/
void setup_mm_for_reboot(void)
{
+ /*
+ * With CONFIG_CPU_TTBR0_PAN enabled, TTBCR.EPD0 is set whenever
+ * user-space access is disabled in order to block TTBR0 page-table
+ * walks. The identity mapping lives at low (user-space) virtual
+ * addresses and can only be reached via TTBR0, so we must re-enable
+ * those walks before switching page tables. On non-PAN kernels this
+ * is a no-op.
+ */
+ if (IS_ENABLED(CONFIG_CPU_TTBR0_PAN))
+ uaccess_save_and_enable();
+
/* Switch to the identity mapping. */
cpu_switch_mm(idmap_pgd, &init_mm);
local_flush_bp_all();
diff --git a/arch/arm64/include/uapi/asm/fcntl.h b/arch/arm64/include/uapi/asm/fcntl.h
index f8db34f..e503fdb 100644
--- a/arch/arm64/include/uapi/asm/fcntl.h
+++ b/arch/arm64/include/uapi/asm/fcntl.h
@@ -20,10 +20,10 @@
/*
* Using our own definitions for AArch32 (compat) support.
*/
-#define O_DIRECTORY 040000 /* must be a directory */
-#define O_NOFOLLOW 0100000 /* don't follow links */
-#define O_DIRECT 0200000 /* direct disk access hint - currently ignored */
-#define O_LARGEFILE 0400000
+#define O_DIRECTORY (1 << 14) /* must be a directory */
+#define O_NOFOLLOW (1 << 15) /* don't follow links */
+#define O_DIRECT (1 << 16) /* direct disk access hint - currently ignored */
+#define O_LARGEFILE (1 << 17)
#include <asm-generic/fcntl.h>
diff --git a/arch/arm64/kernel/mte.c b/arch/arm64/kernel/mte.c
index 6874b16..1a9aad6 100644
--- a/arch/arm64/kernel/mte.c
+++ b/arch/arm64/kernel/mte.c
@@ -8,6 +8,7 @@
#include <linux/kernel.h>
#include <linux/mm.h>
#include <linux/prctl.h>
+#include <linux/ptrace.h>
#include <linux/sched.h>
#include <linux/sched/mm.h>
#include <linux/string.h>
@@ -537,16 +538,13 @@ static int access_remote_tags(struct task_struct *tsk, unsigned long addr,
if (!mm)
return -EPERM;
- if (!tsk->ptrace || (current != tsk->parent) ||
- ((get_dumpable(mm) != SUID_DUMP_USER) &&
- !ptracer_capable(tsk, mm->user_ns))) {
+ if (!ptracer_access_allowed(tsk)) {
mmput(mm);
return -EPERM;
}
ret = __access_remote_tags(mm, addr, kiov, gup_flags);
mmput(mm);
-
return ret;
}
diff --git a/arch/arm64/kernel/vmlinux.lds.S b/arch/arm64/kernel/vmlinux.lds.S
index e1ac8762..8aaf404 100644
--- a/arch/arm64/kernel/vmlinux.lds.S
+++ b/arch/arm64/kernel/vmlinux.lds.S
@@ -368,6 +368,7 @@
STABS_DEBUG
DWARF_DEBUG
+ PROPELLER_DATA
MODINFO
ELF_DETAILS
diff --git a/arch/m68k/include/uapi/asm/fcntl.h b/arch/m68k/include/uapi/asm/fcntl.h
index c6861e6..66c0e55 100644
--- a/arch/m68k/include/uapi/asm/fcntl.h
+++ b/arch/m68k/include/uapi/asm/fcntl.h
@@ -2,10 +2,10 @@
#ifndef _M68K_FCNTL_H
#define _M68K_FCNTL_H
-#define O_DIRECTORY 040000 /* must be a directory */
-#define O_NOFOLLOW 0100000 /* don't follow links */
-#define O_DIRECT 0200000 /* direct disk access hint - currently ignored */
-#define O_LARGEFILE 0400000
+#define O_DIRECTORY (1 << 14) /* must be a directory */
+#define O_NOFOLLOW (1 << 15) /* don't follow links */
+#define O_DIRECT (1 << 16) /* direct disk access hint - currently ignored */
+#define O_LARGEFILE (1 << 17)
#include <asm-generic/fcntl.h>
diff --git a/arch/mips/include/uapi/asm/errno.h b/arch/mips/include/uapi/asm/errno.h
index c01ed91..1835a50 100644
--- a/arch/mips/include/uapi/asm/errno.h
+++ b/arch/mips/include/uapi/asm/errno.h
@@ -126,6 +126,8 @@
#define EHWPOISON 168 /* Memory page has hardware error */
+#define EFTYPE 169 /* Wrong file type for the intended operation */
+
#define EDQUOT 1133 /* Quota exceeded */
diff --git a/arch/mips/include/uapi/asm/fcntl.h b/arch/mips/include/uapi/asm/fcntl.h
index 0369a38..549fc65 100644
--- a/arch/mips/include/uapi/asm/fcntl.h
+++ b/arch/mips/include/uapi/asm/fcntl.h
@@ -11,15 +11,15 @@
#include <asm/sgidefs.h>
-#define O_APPEND 0x0008
-#define O_DSYNC 0x0010 /* used to be O_SYNC, see below */
-#define O_NONBLOCK 0x0080
-#define O_CREAT 0x0100 /* not fcntl */
-#define O_TRUNC 0x0200 /* not fcntl */
-#define O_EXCL 0x0400 /* not fcntl */
-#define O_NOCTTY 0x0800 /* not fcntl */
-#define FASYNC 0x1000 /* fcntl, for BSD compatibility */
-#define O_LARGEFILE 0x2000 /* allow large file opens */
+#define O_APPEND (1 << 3)
+#define O_DSYNC (1 << 4) /* used to be O_SYNC, see below */
+#define O_NONBLOCK (1 << 7)
+#define O_CREAT (1 << 8) /* not fcntl */
+#define O_TRUNC (1 << 9) /* not fcntl */
+#define O_EXCL (1 << 10) /* not fcntl */
+#define O_NOCTTY (1 << 11) /* not fcntl */
+#define FASYNC (1 << 12) /* fcntl, for BSD compatibility */
+#define O_LARGEFILE (1 << 13) /* allow large file opens */
/*
* Before Linux 2.6.33 only O_DSYNC semantics were implemented, but using
* the O_SYNC flag. We continue to use the existing numerical value
@@ -33,9 +33,9 @@
*
* Note: __O_SYNC must never be used directly.
*/
-#define __O_SYNC 0x4000
+#define __O_SYNC (1 << 14)
#define O_SYNC (__O_SYNC|O_DSYNC)
-#define O_DIRECT 0x8000 /* direct disk access hint */
+#define O_DIRECT (1 << 15) /* direct disk access hint */
#define F_GETLK 14
#define F_SETLK 6
diff --git a/arch/parisc/include/uapi/asm/errno.h b/arch/parisc/include/uapi/asm/errno.h
index 8cbc07c..93194fb 100644
--- a/arch/parisc/include/uapi/asm/errno.h
+++ b/arch/parisc/include/uapi/asm/errno.h
@@ -124,4 +124,6 @@
#define EHWPOISON 257 /* Memory page has hardware error */
+#define EFTYPE 258 /* Wrong file type for the intended operation */
+
#endif
diff --git a/arch/parisc/include/uapi/asm/fcntl.h b/arch/parisc/include/uapi/asm/fcntl.h
index 03dee81..2e1bb18 100644
--- a/arch/parisc/include/uapi/asm/fcntl.h
+++ b/arch/parisc/include/uapi/asm/fcntl.h
@@ -2,23 +2,23 @@
#ifndef _PARISC_FCNTL_H
#define _PARISC_FCNTL_H
-#define O_APPEND 000000010
-#define O_CREAT 000000400 /* not fcntl */
-#define O_EXCL 000002000 /* not fcntl */
-#define O_LARGEFILE 000004000
-#define __O_SYNC 000100000
+#define O_APPEND (1 << 3)
+#define O_CREAT (1 << 8) /* not fcntl */
+#define O_EXCL (1 << 10) /* not fcntl */
+#define O_LARGEFILE (1 << 11)
+#define __O_SYNC (1 << 15)
#define O_SYNC (__O_SYNC|O_DSYNC)
-#define O_NONBLOCK 000200000
-#define O_NOCTTY 000400000 /* not fcntl */
-#define O_DSYNC 001000000
-#define O_NOATIME 004000000
-#define O_CLOEXEC 010000000 /* set close_on_exec */
+#define O_NONBLOCK (1 << 16)
+#define O_NOCTTY (1 << 17) /* not fcntl */
+#define O_DSYNC (1 << 18)
+#define O_NOATIME (1 << 20)
+#define O_CLOEXEC (1 << 21) /* set close_on_exec */
-#define O_DIRECTORY 000010000 /* must be a directory */
-#define O_NOFOLLOW 000000200 /* don't follow links */
+#define O_DIRECTORY (1 << 12) /* must be a directory */
+#define O_NOFOLLOW (1 << 7) /* don't follow links */
-#define O_PATH 020000000
-#define __O_TMPFILE 040000000
+#define O_PATH (1 << 22)
+#define __O_TMPFILE (1 << 23)
#define F_GETLK64 8
#define F_SETLK64 9
diff --git a/arch/powerpc/include/uapi/asm/fcntl.h b/arch/powerpc/include/uapi/asm/fcntl.h
index 65ce083..003bc5e 100644
--- a/arch/powerpc/include/uapi/asm/fcntl.h
+++ b/arch/powerpc/include/uapi/asm/fcntl.h
@@ -2,10 +2,10 @@
#ifndef _ASM_FCNTL_H
#define _ASM_FCNTL_H
-#define O_DIRECTORY 040000 /* must be a directory */
-#define O_NOFOLLOW 0100000 /* don't follow links */
-#define O_LARGEFILE 0200000
-#define O_DIRECT 0400000 /* direct disk access hint */
+#define O_DIRECTORY (1 << 14) /* must be a directory */
+#define O_NOFOLLOW (1 << 15) /* don't follow links */
+#define O_LARGEFILE (1 << 16)
+#define O_DIRECT (1 << 17) /* direct disk access hint */
#include <asm-generic/fcntl.h>
diff --git a/arch/powerpc/platforms/cell/spufs/file.c b/arch/powerpc/platforms/cell/spufs/file.c
index 10fa9b8..f6de8c1 100644
--- a/arch/powerpc/platforms/cell/spufs/file.c
+++ b/arch/powerpc/platforms/cell/spufs/file.c
@@ -1430,7 +1430,7 @@ static int spufs_mfc_open(struct inode *inode, struct file *file)
if (ctx->owner != current->mm)
return -EINVAL;
- if (icount_read(inode) != 1)
+ if (icount_read_once(inode) != 1)
return -EBUSY;
mutex_lock(&ctx->mapping_lock);
diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig
index c575494..1a2fadc 100644
--- a/arch/riscv/Kconfig
+++ b/arch/riscv/Kconfig
@@ -61,8 +61,7 @@
select ARCH_OPTIONAL_KERNEL_RWX_DEFAULT
select ARCH_STACKWALK
select ARCH_SUPPORTS_ATOMIC_RMW
- # clang >= 17: https://github.com/llvm/llvm-project/commit/62fa708ceb027713b386c7e0efda994f8bdc27e2
- select ARCH_SUPPORTS_CFI if (!CC_IS_CLANG || CLANG_VERSION >= 170000)
+ select ARCH_SUPPORTS_CFI
select ARCH_SUPPORTS_DEBUG_PAGEALLOC if MMU
select ARCH_SUPPORTS_HUGE_PFNMAP if TRANSPARENT_HUGEPAGE
select ARCH_SUPPORTS_HUGETLBFS if MMU
@@ -874,19 +873,18 @@
and Zifencei are supported in binutils from version 2.36 onwards.
To make life easier, and avoid forcing toolchains that default to a
newer ISA spec to version 2.2, relax the check to binutils >= 2.36.
- For clang < 17 or GCC < 11.3.0, for which this is not possible or need
- special treatment, this is dealt with in TOOLCHAIN_NEEDS_OLD_ISA_SPEC.
+ For GCC < 11.3.0, for which this is not possible or need special
+ treatment, this is dealt with in TOOLCHAIN_NEEDS_OLD_ISA_SPEC.
config TOOLCHAIN_NEEDS_OLD_ISA_SPEC
def_bool y
depends on TOOLCHAIN_NEEDS_EXPLICIT_ZICSR_ZIFENCEI
- # https://github.com/llvm/llvm-project/commit/22e199e6afb1263c943c0c0d4498694e15bf8a16
# https://gcc.gnu.org/git/?p=gcc.git;a=commit;h=d29f5d6ab513c52fd872f532c492e35ae9fd6671
- depends on (CC_IS_CLANG && CLANG_VERSION < 170000) || (CC_IS_GCC && GCC_VERSION < 110300)
+ depends on CC_IS_GCC && GCC_VERSION < 110300
help
- Certain versions of clang and GCC do not support zicsr and zifencei via
- -march. This option causes an older ISA spec compatible with these older
- versions of clang and GCC to be passed to GAS, which has the same result
+ Certain versions of GCC do not support zicsr and zifencei via -march.
+ This option causes an older ISA spec compatible with these older
+ versions of GCC to be passed to GAS, which has the same result
as passing zicsr and zifencei to -march.
config FPU
diff --git a/arch/s390/Kconfig b/arch/s390/Kconfig
index ecbcbb7..9921a37 100644
--- a/arch/s390/Kconfig
+++ b/arch/s390/Kconfig
@@ -29,9 +29,6 @@
config GENERIC_BUG_RELATIVE_POINTERS
def_bool y
-config GENERIC_LOCKBREAK
- def_bool y if PREEMPTION
-
config AUDIT_ARCH
def_bool y
diff --git a/arch/sparc/include/uapi/asm/errno.h b/arch/sparc/include/uapi/asm/errno.h
index 4a41e78..71940ec 100644
--- a/arch/sparc/include/uapi/asm/errno.h
+++ b/arch/sparc/include/uapi/asm/errno.h
@@ -117,4 +117,6 @@
#define EHWPOISON 135 /* Memory page has hardware error */
+#define EFTYPE 136 /* Wrong file type for the intended operation */
+
#endif
diff --git a/arch/sparc/include/uapi/asm/fcntl.h b/arch/sparc/include/uapi/asm/fcntl.h
index 67dae75..29c5639 100644
--- a/arch/sparc/include/uapi/asm/fcntl.h
+++ b/arch/sparc/include/uapi/asm/fcntl.h
@@ -2,23 +2,23 @@
#ifndef _SPARC_FCNTL_H
#define _SPARC_FCNTL_H
-#define O_APPEND 0x0008
-#define FASYNC 0x0040 /* fcntl, for BSD compatibility */
-#define O_CREAT 0x0200 /* not fcntl */
-#define O_TRUNC 0x0400 /* not fcntl */
-#define O_EXCL 0x0800 /* not fcntl */
-#define O_DSYNC 0x2000 /* used to be O_SYNC, see below */
-#define O_NONBLOCK 0x4000
+#define O_APPEND (1 << 3)
+#define FASYNC (1 << 6) /* fcntl, for BSD compatibility */
+#define O_CREAT (1 << 9) /* not fcntl */
+#define O_TRUNC (1 << 10) /* not fcntl */
+#define O_EXCL (1 << 11) /* not fcntl */
+#define O_DSYNC (1 << 13) /* used to be O_SYNC, see below */
+#define O_NONBLOCK (1 << 14)
#if defined(__sparc__) && defined(__arch64__)
-#define O_NDELAY 0x0004
+#define O_NDELAY (1 << 2)
#else
-#define O_NDELAY (0x0004 | O_NONBLOCK)
+#define O_NDELAY ((1 << 2) | O_NONBLOCK)
#endif
-#define O_NOCTTY 0x8000 /* not fcntl */
-#define O_LARGEFILE 0x40000
-#define O_DIRECT 0x100000 /* direct disk access hint */
-#define O_NOATIME 0x200000
-#define O_CLOEXEC 0x400000
+#define O_NOCTTY (1 << 15) /* not fcntl */
+#define O_LARGEFILE (1 << 18)
+#define O_DIRECT (1 << 20) /* direct disk access hint */
+#define O_NOATIME (1 << 21)
+#define O_CLOEXEC (1 << 22)
/*
* Before Linux 2.6.33 only O_DSYNC semantics were implemented, but using
* the O_SYNC flag. We continue to use the existing numerical value
@@ -32,11 +32,11 @@
*
* Note: __O_SYNC must never be used directly.
*/
-#define __O_SYNC 0x800000
+#define __O_SYNC (1 << 23)
#define O_SYNC (__O_SYNC|O_DSYNC)
-#define O_PATH 0x1000000
-#define __O_TMPFILE 0x2000000
+#define O_PATH (1 << 24)
+#define __O_TMPFILE (1 << 25)
#define F_GETOWN 5 /* for sockets. */
#define F_SETOWN 6 /* for sockets. */
diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig
index f3f7cb0..b875d2f 100644
--- a/arch/x86/Kconfig
+++ b/arch/x86/Kconfig
@@ -130,8 +130,6 @@
select ARCH_SUPPORTS_LTO_CLANG
select ARCH_SUPPORTS_LTO_CLANG_THIN
select ARCH_SUPPORTS_RT
- select ARCH_SUPPORTS_AUTOFDO_CLANG
- select ARCH_SUPPORTS_PROPELLER_CLANG if X86_64
select ARCH_USE_BUILTIN_BSWAP
select ARCH_USE_CMPXCHG_LOCKREF if X86_CX8
select ARCH_USE_MEMTEST
diff --git a/arch/x86/Makefile b/arch/x86/Makefile
index 1d526a5..1e5457a 100644
--- a/arch/x86/Makefile
+++ b/arch/x86/Makefile
@@ -128,11 +128,6 @@
include $(srctree)/arch/x86/Makefile_32.cpu
KBUILD_CFLAGS += $(cflags-y)
- ifneq ($(call clang-min-version, 160000),y)
- # https://github.com/llvm/llvm-project/issues/53645
- KBUILD_CFLAGS += -ffreestanding
- endif
-
percpu_seg := fs
else
BITS := 64
diff --git a/arch/x86/entry/vdso/vdso32/sigreturn.S b/arch/x86/entry/vdso/vdso32/sigreturn.S
index b33fcc5..328bd3a 100644
--- a/arch/x86/entry/vdso/vdso32/sigreturn.S
+++ b/arch/x86/entry/vdso/vdso32/sigreturn.S
@@ -22,17 +22,7 @@
CFI_OFFSET cs, IA32_SIGCONTEXT_cs
CFI_OFFSET ss, IA32_SIGCONTEXT_ss
CFI_OFFSET ds, IA32_SIGCONTEXT_ds
-/*
- * .cfi_offset eflags requires LLVM 16 or newer:
- *
- * https://github.com/llvm/llvm-project/commit/67bd3c58c0c7389e39c5a2f4d3b1a30459ccf5b7
- *
- * Check for 16.0.1 to ensure the support is present, as 16.0.0 may be a
- * prerelease version.
- */
-#if defined(CONFIG_AS_IS_GNU) || (defined(CONFIG_AS_IS_LLVM) && CONFIG_AS_VERSION >= 160001)
CFI_OFFSET eflags, IA32_SIGCONTEXT_flags
-#endif
.endm
/*
diff --git a/arch/x86/include/asm/elf.h b/arch/x86/include/asm/elf.h
index c7f9897..0de9df7 100644
--- a/arch/x86/include/asm/elf.h
+++ b/arch/x86/include/asm/elf.h
@@ -54,9 +54,8 @@ typedef struct user_i387_struct elf_fpregset_t;
#define R_X86_64_GLOB_DAT 6 /* Create GOT entry */
#define R_X86_64_JUMP_SLOT 7 /* Create PLT entry */
#define R_X86_64_RELATIVE 8 /* Adjust by program base */
-#define R_X86_64_GOTPCREL 9 /* 32 bit signed pc relative offset to GOT */
-#define R_X86_64_GOTPCRELX 41
-#define R_X86_64_REX_GOTPCRELX 42
+#define R_X86_64_GOTPCREL 9 /* 32 bit signed pc relative
+ offset to GOT */
#define R_X86_64_32 10 /* Direct 32 bit zero extended */
#define R_X86_64_32S 11 /* Direct 32 bit sign extended */
#define R_X86_64_16 12 /* Direct 16 bit zero extended */
diff --git a/arch/x86/include/asm/paravirt.h b/arch/x86/include/asm/paravirt.h
index cdfe400..0591aa3 100644
--- a/arch/x86/include/asm/paravirt.h
+++ b/arch/x86/include/asm/paravirt.h
@@ -483,17 +483,16 @@ static inline void arch_end_context_switch(struct task_struct *next)
static inline void arch_enter_lazy_mmu_mode(void)
{
- PVOP_VCALL0(pv_ops, mmu.lazy_mode.enter);
-}
-
-static inline void arch_leave_lazy_mmu_mode(void)
-{
- PVOP_VCALL0(pv_ops, mmu.lazy_mode.leave);
}
static inline void arch_flush_lazy_mmu_mode(void)
{
- PVOP_VCALL0(pv_ops, mmu.lazy_mode.flush);
+ PVOP_VCALL0(pv_ops, mmu.lazy_mode_flush);
+}
+
+static inline void arch_leave_lazy_mmu_mode(void)
+{
+ arch_flush_lazy_mmu_mode();
}
static inline void __set_fixmap(unsigned /* enum fixed_addresses */ idx,
diff --git a/arch/x86/include/asm/paravirt_types.h b/arch/x86/include/asm/paravirt_types.h
index 4f5ae00..b4c4a23 100644
--- a/arch/x86/include/asm/paravirt_types.h
+++ b/arch/x86/include/asm/paravirt_types.h
@@ -19,15 +19,6 @@ struct cpumask;
struct flush_tlb_info;
struct vm_area_struct;
-#ifdef CONFIG_PARAVIRT_XXL
-struct pv_lazy_ops {
- /* Set deferred update mode, used for batching operations. */
- void (*enter)(void);
- void (*leave)(void);
- void (*flush)(void);
-} __no_randomize_layout;
-#endif
-
struct pv_cpu_ops {
/* hooks for various privileged instructions */
#ifdef CONFIG_PARAVIRT_XXL
@@ -171,7 +162,7 @@ struct pv_mmu_ops {
void (*set_pgd)(pgd_t *pgdp, pgd_t pgdval);
- struct pv_lazy_ops lazy_mode;
+ void (*lazy_mode_flush)(void);
/* dom0 ops */
diff --git a/arch/x86/include/asm/xen/hypervisor.h b/arch/x86/include/asm/xen/hypervisor.h
index c2fc786..f666637 100644
--- a/arch/x86/include/asm/xen/hypervisor.h
+++ b/arch/x86/include/asm/xen/hypervisor.h
@@ -64,30 +64,7 @@ void __init xen_pvh_init(struct boot_params *boot_params);
void __init mem_map_via_hcall(struct boot_params *boot_params_p);
#endif
-/* Lazy mode for batching updates / context switch */
-enum xen_lazy_mode {
- XEN_LAZY_NONE,
- XEN_LAZY_MMU,
- XEN_LAZY_CPU,
-};
-
-DECLARE_PER_CPU(enum xen_lazy_mode, xen_lazy_mode);
-
-static inline void enter_lazy(enum xen_lazy_mode mode)
-{
- BUG_ON(this_cpu_read(xen_lazy_mode) != XEN_LAZY_NONE);
-
- this_cpu_write(xen_lazy_mode, mode);
-}
-
-static inline void leave_lazy(enum xen_lazy_mode mode)
-{
- BUG_ON(this_cpu_read(xen_lazy_mode) != mode);
-
- this_cpu_write(xen_lazy_mode, XEN_LAZY_NONE);
-}
-
-enum xen_lazy_mode xen_get_lazy_mode(void);
+bool xen_is_cpu_lazy_mode(void);
#if defined(CONFIG_XEN_DOM0) && defined(CONFIG_ACPI)
void xen_sanitize_proc_cap_bits(uint32_t *buf);
diff --git a/arch/x86/kernel/module.c b/arch/x86/kernel/module.c
index 11c45ce..b5b4de4 100644
--- a/arch/x86/kernel/module.c
+++ b/arch/x86/kernel/module.c
@@ -19,7 +19,6 @@
#include <linux/jump_label.h>
#include <linux/random.h>
#include <linux/memory.h>
-#include <linux/stackprotector.h>
#include <asm/text-patching.h>
#include <asm/page.h>
@@ -132,20 +131,6 @@ static int __write_relocate_add(Elf64_Shdr *sechdrs,
goto overflow;
size = 4;
break;
-#if defined(CONFIG_STACKPROTECTOR) && \
- defined(CONFIG_CC_IS_CLANG) && CONFIG_CLANG_VERSION < 170000
- case R_X86_64_REX_GOTPCRELX: {
- static unsigned long __percpu *const addr = &__stack_chk_guard;
-
- if (sym->st_value != (u64)addr) {
- pr_err("%s: Unsupported GOTPCREL relocation\n", me->name);
- return -ENOEXEC;
- }
-
- val = (u64)&addr + rel[i].r_addend;
- fallthrough;
- }
-#endif
case R_X86_64_PC32:
case R_X86_64_PLT32:
val -= (u64)loc;
diff --git a/arch/x86/kernel/paravirt.c b/arch/x86/kernel/paravirt.c
index 792fa96..22f7203 100644
--- a/arch/x86/kernel/paravirt.c
+++ b/arch/x86/kernel/paravirt.c
@@ -204,11 +204,7 @@ struct paravirt_patch_template pv_ops = {
.mmu.enter_mmap = paravirt_nop,
- .mmu.lazy_mode = {
- .enter = paravirt_nop,
- .leave = paravirt_nop,
- .flush = paravirt_nop,
- },
+ .mmu.lazy_mode_flush = paravirt_nop,
.mmu.set_fixmap = native_set_fixmap,
#endif /* CONFIG_PARAVIRT_XXL */
diff --git a/arch/x86/kernel/vmlinux.lds.S b/arch/x86/kernel/vmlinux.lds.S
index 4711a35..74e336d 100644
--- a/arch/x86/kernel/vmlinux.lds.S
+++ b/arch/x86/kernel/vmlinux.lds.S
@@ -423,10 +423,7 @@
STABS_DEBUG
DWARF_DEBUG
-#ifdef CONFIG_PROPELLER_CLANG
- .llvm_bb_addr_map : { *(.llvm_bb_addr_map) }
-#endif
-
+ PROPELLER_DATA
MODINFO
ELF_DETAILS
diff --git a/arch/x86/xen/Kconfig b/arch/x86/xen/Kconfig
index aa4040f..51b53ce 100644
--- a/arch/x86/xen/Kconfig
+++ b/arch/x86/xen/Kconfig
@@ -65,13 +65,6 @@
help
Support running as a Xen PVHVM guest.
-config XEN_DEBUG_FS
- bool "Enable Xen debug and tuning parameters in debugfs"
- depends on XEN && DEBUG_FS
- help
- Enable statistics output and various tuning options in debugfs.
- Enabling this option may incur a significant performance overhead.
-
config XEN_PVH
bool "Xen PVH guest support"
depends on XEN && XEN_PVHVM && ACPI
diff --git a/arch/x86/xen/Makefile b/arch/x86/xen/Makefile
index a9ec8c9..717264a 100644
--- a/arch/x86/xen/Makefile
+++ b/arch/x86/xen/Makefile
@@ -43,8 +43,6 @@
obj-$(CONFIG_PARAVIRT_SPINLOCKS)+= spinlock.o
-obj-$(CONFIG_XEN_DEBUG_FS) += debugfs.o
-
obj-$(CONFIG_XEN_DOM0) += vga.o
obj-$(CONFIG_XEN_EFI) += efi.o
diff --git a/arch/x86/xen/debugfs.c b/arch/x86/xen/debugfs.c
deleted file mode 100644
index b8c9f2a7..0000000
--- a/arch/x86/xen/debugfs.c
+++ /dev/null
@@ -1,16 +0,0 @@
-// SPDX-License-Identifier: GPL-2.0
-#include <linux/init.h>
-#include <linux/debugfs.h>
-#include <linux/slab.h>
-
-#include "xen-ops.h"
-
-static struct dentry *d_xen_debug;
-
-struct dentry * __init xen_init_debugfs(void)
-{
- if (!d_xen_debug)
- d_xen_debug = debugfs_create_dir("xen", NULL);
- return d_xen_debug;
-}
-
diff --git a/arch/x86/xen/enlighten_pv.c b/arch/x86/xen/enlighten_pv.c
index ed2d7a3..b310b91 100644
--- a/arch/x86/xen/enlighten_pv.c
+++ b/arch/x86/xen/enlighten_pv.c
@@ -138,14 +138,11 @@ struct tls_descs {
struct desc_struct desc[3];
};
-DEFINE_PER_CPU(enum xen_lazy_mode, xen_lazy_mode) = XEN_LAZY_NONE;
+static DEFINE_PER_CPU(bool, xen_cpu_lazy_mode);
-enum xen_lazy_mode xen_get_lazy_mode(void)
+bool xen_is_cpu_lazy_mode(void)
{
- if (in_interrupt())
- return XEN_LAZY_NONE;
-
- return this_cpu_read(xen_lazy_mode);
+ return !in_interrupt() && this_cpu_read(xen_cpu_lazy_mode);
}
/*
@@ -424,10 +421,8 @@ static void xen_start_context_switch(struct task_struct *prev)
{
BUG_ON(preemptible());
- if (this_cpu_read(xen_lazy_mode) == XEN_LAZY_MMU) {
- arch_leave_lazy_mmu_mode();
- }
- enter_lazy(XEN_LAZY_CPU);
+ __task_lazy_mmu_mode_pause(prev);
+ this_cpu_write(xen_cpu_lazy_mode, true);
}
static void xen_end_context_switch(struct task_struct *next)
@@ -435,9 +430,8 @@ static void xen_end_context_switch(struct task_struct *next)
BUG_ON(preemptible());
xen_mc_flush();
- leave_lazy(XEN_LAZY_CPU);
- if (__task_lazy_mmu_mode_active(next))
- arch_enter_lazy_mmu_mode();
+ this_cpu_write(xen_cpu_lazy_mode, false);
+ __task_lazy_mmu_mode_resume(next);
}
static unsigned long xen_store_tr(void)
@@ -544,7 +538,7 @@ static void xen_set_ldt(const void *addr, unsigned entries)
MULTI_mmuext_op(mcs.mc, op, 1, NULL, DOMID_SELF);
- xen_mc_issue(XEN_LAZY_CPU);
+ xen_mc_issue(!xen_is_cpu_lazy_mode());
}
static void xen_load_gdt(const struct desc_ptr *dtr)
@@ -640,7 +634,7 @@ static void xen_load_tls(struct thread_struct *t, unsigned int cpu)
* exception between the new %fs descriptor being loaded and
* %fs being effectively cleared at __switch_to().
*/
- if (xen_get_lazy_mode() == XEN_LAZY_CPU)
+ if (xen_is_cpu_lazy_mode())
loadsegment(fs, 0);
xen_mc_batch();
@@ -649,7 +643,7 @@ static void xen_load_tls(struct thread_struct *t, unsigned int cpu)
load_TLS_descriptor(t, cpu, 1);
load_TLS_descriptor(t, cpu, 2);
- xen_mc_issue(XEN_LAZY_CPU);
+ xen_mc_issue(!xen_is_cpu_lazy_mode());
}
static void xen_load_gs_index(unsigned int idx)
@@ -1011,7 +1005,7 @@ static void xen_load_sp0(unsigned long sp0)
mcs = xen_mc_entry(0);
MULTI_stack_switch(mcs.mc, __KERNEL_DS, sp0);
- xen_mc_issue(XEN_LAZY_CPU);
+ xen_mc_issue(!xen_is_cpu_lazy_mode());
this_cpu_write(cpu_tss_rw.x86_tss.sp0, sp0);
}
@@ -1071,7 +1065,7 @@ static void xen_write_cr0(unsigned long cr0)
MULTI_fpu_taskswitch(mcs.mc, (cr0 & X86_CR0_TS) != 0);
- xen_mc_issue(XEN_LAZY_CPU);
+ xen_mc_issue(!xen_is_cpu_lazy_mode());
}
static void xen_write_cr4(unsigned long cr4)
diff --git a/arch/x86/xen/mmu_pv.c b/arch/x86/xen/mmu_pv.c
index 3eee5f8..aab5f70 100644
--- a/arch/x86/xen/mmu_pv.c
+++ b/arch/x86/xen/mmu_pv.c
@@ -84,6 +84,14 @@
#include "xen-ops.h"
+enum pt_level {
+ PT_PGD,
+ PT_P4D,
+ PT_PUD,
+ PT_PMD,
+ PT_PTE
+};
+
/*
* Prototypes for functions called via PV_CALLEE_SAVE_REGS_THUNK() in order
* to avoid warnings with "-Wmissing-prototypes".
@@ -282,7 +290,7 @@ static void xen_set_pmd_hyper(pmd_t *ptr, pmd_t val)
u.val = pmd_val_ma(val);
xen_extend_mmu_update(&u);
- xen_mc_issue(XEN_LAZY_MMU);
+ xen_mc_issue(!is_lazy_mmu_mode_active());
preempt_enable();
}
@@ -316,7 +324,7 @@ static bool xen_batched_set_pte(pte_t *ptep, pte_t pteval)
{
struct mmu_update u;
- if (xen_get_lazy_mode() != XEN_LAZY_MMU)
+ if (!is_lazy_mmu_mode_active())
return false;
xen_mc_batch();
@@ -325,7 +333,7 @@ static bool xen_batched_set_pte(pte_t *ptep, pte_t pteval)
u.val = pte_val_ma(pteval);
xen_extend_mmu_update(&u);
- xen_mc_issue(XEN_LAZY_MMU);
+ xen_mc_issue(!is_lazy_mmu_mode_active());
return true;
}
@@ -372,7 +380,7 @@ static void xen_ptep_modify_prot_commit(struct vm_area_struct *vma,
u.val = pte_val_ma(pte);
xen_extend_mmu_update(&u);
- xen_mc_issue(XEN_LAZY_MMU);
+ xen_mc_issue(!is_lazy_mmu_mode_active());
}
/* Assume pteval_t is equivalent to all the other *val_t types. */
@@ -466,7 +474,7 @@ static void xen_set_pud_hyper(pud_t *ptr, pud_t val)
u.val = pud_val_ma(val);
xen_extend_mmu_update(&u);
- xen_mc_issue(XEN_LAZY_MMU);
+ xen_mc_issue(!is_lazy_mmu_mode_active());
preempt_enable();
}
@@ -549,7 +557,7 @@ static void __init xen_set_p4d_hyper(p4d_t *ptr, p4d_t val)
__xen_set_p4d_hyper(ptr, val);
- xen_mc_issue(XEN_LAZY_MMU);
+ xen_mc_issue(!is_lazy_mmu_mode_active());
preempt_enable();
}
@@ -581,7 +589,7 @@ static void xen_set_p4d(p4d_t *ptr, p4d_t val)
if (user_ptr)
__xen_set_p4d_hyper((p4d_t *)user_ptr, val);
- xen_mc_issue(XEN_LAZY_MMU);
+ xen_mc_issue(!is_lazy_mmu_mode_active());
}
__visible p4dval_t xen_p4d_val(p4d_t p4d)
@@ -808,7 +816,7 @@ static void __xen_pgd_pin(struct mm_struct *mm, pgd_t *pgd)
PFN_DOWN(__pa(user_pgd)));
}
- xen_mc_issue(0);
+ xen_mc_issue(true);
}
static void xen_pgd_pin(struct mm_struct *mm)
@@ -925,7 +933,7 @@ static void __xen_pgd_unpin(struct mm_struct *mm, pgd_t *pgd)
__xen_pgd_walk(mm, pgd, xen_unpin_page, USER_LIMIT);
- xen_mc_issue(0);
+ xen_mc_issue(true);
}
static void xen_pgd_unpin(struct mm_struct *mm)
@@ -1301,7 +1309,7 @@ static noinline void xen_flush_tlb(void)
op->cmd = MMUEXT_TLB_FLUSH_LOCAL;
MULTI_mmuext_op(mcs.mc, op, 1, NULL, DOMID_SELF);
- xen_mc_issue(XEN_LAZY_MMU);
+ xen_mc_issue(!is_lazy_mmu_mode_active());
preempt_enable();
}
@@ -1321,7 +1329,7 @@ static void xen_flush_tlb_one_user(unsigned long addr)
op->arg1.linear_addr = addr & PAGE_MASK;
MULTI_mmuext_op(mcs.mc, op, 1, NULL, DOMID_SELF);
- xen_mc_issue(XEN_LAZY_MMU);
+ xen_mc_issue(!is_lazy_mmu_mode_active());
preempt_enable();
}
@@ -1358,7 +1366,7 @@ static void xen_flush_tlb_multi(const struct cpumask *cpus,
MULTI_mmuext_op(mcs.mc, &args->op, 1, NULL, DOMID_SELF);
- xen_mc_issue(XEN_LAZY_MMU);
+ xen_mc_issue(!is_lazy_mmu_mode_active());
}
static unsigned long xen_read_cr3(void)
@@ -1417,7 +1425,7 @@ static void xen_write_cr3(unsigned long cr3)
else
__xen_write_cr3(false, 0);
- xen_mc_issue(XEN_LAZY_CPU); /* interrupts restored */
+ xen_mc_issue(!xen_is_cpu_lazy_mode()); /* interrupts restored */
}
/*
@@ -1452,7 +1460,7 @@ static void __init xen_write_cr3_init(unsigned long cr3)
__xen_write_cr3(true, cr3);
- xen_mc_issue(XEN_LAZY_CPU); /* interrupts restored */
+ xen_mc_issue(!xen_is_cpu_lazy_mode()); /* interrupts restored */
}
static int xen_pgd_alloc(struct mm_struct *mm)
@@ -1614,7 +1622,7 @@ static inline void xen_alloc_ptpage(struct mm_struct *mm, unsigned long pfn,
!pinned)
__pin_pagetable_pfn(MMUEXT_PIN_L1_TABLE, pfn);
- xen_mc_issue(XEN_LAZY_MMU);
+ xen_mc_issue(!is_lazy_mmu_mode_active());
}
}
@@ -1644,7 +1652,7 @@ static inline void xen_release_ptpage(unsigned long pfn, unsigned level)
__set_pfn_prot(pfn, PAGE_KERNEL);
- xen_mc_issue(XEN_LAZY_MMU);
+ xen_mc_issue(!is_lazy_mmu_mode_active());
ClearPagePinned(page);
}
@@ -1867,7 +1875,7 @@ void __init xen_setup_kernel_pagetable(pgd_t *pgd, unsigned long max_pfn)
*/
xen_mc_batch();
__xen_write_cr3(true, __pa(init_top_pgt));
- xen_mc_issue(XEN_LAZY_CPU);
+ xen_mc_issue(!xen_is_cpu_lazy_mode());
/* We can't that easily rip out L3 and L2, as the Xen pagetables are
* set out this way: [L4], [L1], [L2], [L3], [L1], [L1] ... for
@@ -2143,21 +2151,10 @@ static void xen_set_fixmap(unsigned idx, phys_addr_t phys, pgprot_t prot)
#endif
}
-static void xen_enter_lazy_mmu(void)
-{
- preempt_disable();
- if (xen_get_lazy_mode() != XEN_LAZY_MMU)
- enter_lazy(XEN_LAZY_MMU);
- preempt_enable();
-}
-
static void xen_flush_lazy_mmu(void)
{
preempt_disable();
-
- if (xen_get_lazy_mode() == XEN_LAZY_MMU)
- xen_mc_flush();
-
+ xen_mc_flush();
preempt_enable();
}
@@ -2181,15 +2178,6 @@ static void __init xen_post_allocator_init(void)
pv_ops.mmu.write_cr3 = &xen_write_cr3;
}
-static void xen_leave_lazy_mmu(void)
-{
- preempt_disable();
- xen_mc_flush();
- if (xen_get_lazy_mode() != XEN_LAZY_NONE)
- leave_lazy(XEN_LAZY_MMU);
- preempt_enable();
-}
-
void __init xen_init_mmu_ops(void)
{
x86_init.paging.pagetable_init = xen_pagetable_init;
@@ -2229,9 +2217,7 @@ void __init xen_init_mmu_ops(void)
pv_ops.mmu.make_p4d = PV_CALLEE_SAVE(xen_make_p4d);
pv_ops.mmu.enter_mmap = xen_enter_mmap;
pv_ops.mmu.exit_mmap = xen_exit_mmap;
- pv_ops.mmu.lazy_mode.enter = xen_enter_lazy_mmu;
- pv_ops.mmu.lazy_mode.leave = xen_leave_lazy_mmu;
- pv_ops.mmu.lazy_mode.flush = xen_flush_lazy_mmu;
+ pv_ops.mmu.lazy_mode_flush = xen_flush_lazy_mmu;
pv_ops.mmu.set_fixmap = xen_set_fixmap;
memset(dummy_mapping, 0xff, PAGE_SIZE);
@@ -2258,7 +2244,7 @@ static void xen_zap_pfn_range(unsigned long vaddr, unsigned int order,
if (out_frames)
out_frames[i] = virt_to_pfn((void *)vaddr);
}
- xen_mc_issue(0);
+ xen_mc_issue(true);
}
/*
@@ -2301,7 +2287,7 @@ static void xen_remap_exchanged_ptes(unsigned long vaddr, int order,
set_phys_to_machine(virt_to_pfn((void *)vaddr), mfn);
}
- xen_mc_issue(0);
+ xen_mc_issue(true);
}
/*
@@ -2442,7 +2428,7 @@ static noinline void xen_flush_tlb_all(void)
op->cmd = MMUEXT_TLB_FLUSH_ALL;
MULTI_mmuext_op(mcs.mc, op, 1, NULL, DOMID_SELF);
- xen_mc_issue(XEN_LAZY_MMU);
+ xen_mc_issue(!is_lazy_mmu_mode_active());
preempt_enable();
}
diff --git a/arch/x86/xen/p2m.c b/arch/x86/xen/p2m.c
index 2dd12b6..d007ccf 100644
--- a/arch/x86/xen/p2m.c
+++ b/arch/x86/xen/p2m.c
@@ -883,48 +883,3 @@ void __init xen_add_remap_nonram(phys_addr_t maddr, phys_addr_t paddr,
nr_nonram_remap++;
}
-
-#ifdef CONFIG_XEN_DEBUG_FS
-#include <linux/debugfs.h>
-static int p2m_dump_show(struct seq_file *m, void *v)
-{
- static const char * const type_name[] = {
- [P2M_TYPE_IDENTITY] = "identity",
- [P2M_TYPE_MISSING] = "missing",
- [P2M_TYPE_PFN] = "pfn",
- [P2M_TYPE_UNKNOWN] = "abnormal"};
- unsigned long pfn, first_pfn;
- int type, prev_type;
-
- prev_type = xen_p2m_elem_type(0);
- first_pfn = 0;
-
- for (pfn = 0; pfn < xen_p2m_size; pfn++) {
- type = xen_p2m_elem_type(pfn);
- if (type != prev_type) {
- seq_printf(m, " [0x%lx->0x%lx] %s\n", first_pfn, pfn,
- type_name[prev_type]);
- prev_type = type;
- first_pfn = pfn;
- }
- }
- seq_printf(m, " [0x%lx->0x%lx] %s\n", first_pfn, pfn,
- type_name[prev_type]);
- return 0;
-}
-
-DEFINE_SHOW_ATTRIBUTE(p2m_dump);
-
-static struct dentry *d_mmu_debug;
-
-static int __init xen_p2m_debugfs(void)
-{
- struct dentry *d_xen = xen_init_debugfs();
-
- d_mmu_debug = debugfs_create_dir("mmu", d_xen);
-
- debugfs_create_file("p2m", 0600, d_mmu_debug, NULL, &p2m_dump_fops);
- return 0;
-}
-fs_initcall(xen_p2m_debugfs);
-#endif /* CONFIG_XEN_DEBUG_FS */
diff --git a/arch/x86/xen/xen-ops.h b/arch/x86/xen/xen-ops.h
index f6c331b..dc265bd 100644
--- a/arch/x86/xen/xen-ops.h
+++ b/arch/x86/xen/xen-ops.h
@@ -12,28 +12,24 @@
#include <asm/page.h>
+#ifdef CONFIG_XEN_PV
+
#include <trace/events/xen.h>
/* These are code, but not functions. Defined in entry.S */
extern const char xen_failsafe_callback[];
-void xen_entry_SYSENTER_compat(void);
-#ifdef CONFIG_X86_64
-void xen_entry_SYSCALL_64(void);
-void xen_entry_SYSCALL_compat(void);
-#endif
+DECLARE_PER_CPU(unsigned long, xen_cr3);
extern void *xen_initial_gdt;
+extern cpumask_var_t xen_cpu_initialized_map;
struct trap_info;
void xen_copy_trap_info(struct trap_info *traps);
-DECLARE_PER_CPU_ALIGNED(struct vcpu_info, xen_vcpu_info);
-DECLARE_PER_CPU(unsigned long, xen_cr3);
-
-extern struct start_info *xen_start_info;
-extern struct shared_info xen_dummy_shared_info;
-extern struct shared_info *HYPERVISOR_shared_info;
+void xen_entry_SYSENTER_compat(void);
+void xen_entry_SYSCALL_64(void);
+void xen_entry_SYSCALL_compat(void);
void xen_setup_mfn_list_list(void);
void xen_build_mfn_list_list(void);
@@ -44,13 +40,10 @@ void __init xen_pt_check_e820(void);
void xen_mm_pin_all(void);
void xen_mm_unpin_all(void);
-#ifdef CONFIG_X86_64
void __init xen_relocate_p2m(void);
-#endif
void __init xen_do_remap_nonram(void);
void __init xen_add_remap_nonram(phys_addr_t maddr, phys_addr_t paddr,
unsigned long size);
-
void __init xen_chk_is_e820_usable(phys_addr_t start, phys_addr_t size,
const char *component);
unsigned long __ref xen_chk_extra_mem(unsigned long pfn);
@@ -59,17 +52,121 @@ void __init xen_remap_memory(void);
phys_addr_t __init xen_find_free_area(phys_addr_t size);
char * __init xen_memory_setup(void);
void __init xen_arch_setup(void);
-void xen_banner(void);
void xen_enable_syscall(void);
-void xen_vcpu_restore(void);
+void __init xen_build_dynamic_phys_to_machine(void);
+void __init xen_vmalloc_p2m_tree(void);
+void xen_init_irq_ops(void);
+void xen_setup_vcpu_info_placement(void);
+void __init xen_init_apic(void);
+__visible void xen_irq_enable_direct(void);
+__visible void xen_irq_disable_direct(void);
+__visible unsigned long xen_save_fl_direct(void);
+
+__visible unsigned long xen_read_cr2(void);
+__visible unsigned long xen_read_cr2_direct(void);
+
+/* These are not functions, and cannot be called normally */
+__visible void xen_iret(void);
+
+void xen_force_evtchn_callback(void);
+
+void xen_pv_pre_suspend(void);
+void xen_pv_post_suspend(int suspend_cancelled);
+void xen_start_kernel(struct start_info *si);
+
+void set_pte_mfn(unsigned long vaddr, unsigned long pfn, pgprot_t flags);
+void xen_init_mmu_ops(void);
+
+/* Multicalls */
+struct multicall_space
+{
+ struct multicall_entry *mc;
+ void *args;
+};
+
+/* Allocate room for a multicall and its args */
+struct multicall_space __xen_mc_entry(size_t args);
+
+DECLARE_PER_CPU(unsigned long, xen_mc_irq_flags);
+
+/* Call to start a batch of multiple __xen_mc_entry()s. Must be
+ paired with xen_mc_issue() */
+static inline void xen_mc_batch(void)
+{
+ unsigned long flags;
+
+ /* need to disable interrupts until this entry is complete */
+ local_irq_save(flags);
+ trace_xen_mc_batch(flags);
+ __this_cpu_write(xen_mc_irq_flags, flags);
+}
+
+static inline struct multicall_space xen_mc_entry(size_t args)
+{
+ xen_mc_batch();
+ return __xen_mc_entry(args);
+}
+
+/* Flush all pending multicalls */
+void xen_mc_flush(void);
+
+/* Issue a multicall if we're not in a lazy mode */
+static inline void xen_mc_issue(bool flush)
+{
+ unsigned long flags = this_cpu_read(xen_mc_irq_flags);
+
+ trace_xen_mc_issue(flush, flags);
+
+ if (flush)
+ xen_mc_flush();
+
+ /* restore flags saved in xen_mc_batch */
+ local_irq_restore(flags);
+}
+
+/* Set up a callback to be called when the current batch is flushed */
+void xen_mc_callback(void (*fn)(void *), void *data);
+
+/*
+ * Try to extend the arguments of the previous multicall command. The
+ * previous command's op must match. If it does, then it attempts to
+ * extend the argument space allocated to the multicall entry by
+ * arg_size bytes.
+ *
+ * The returned multicall_space will return with mc pointing to the
+ * command on success, or NULL on failure, and args pointing to the
+ * newly allocated space.
+ */
+struct multicall_space xen_mc_extend_args(unsigned long op, size_t arg_size);
+
+extern bool is_xen_pmu;
+
+irqreturn_t xen_pmu_irq_handler(int irq, void *dev_id);
+bool pmu_msr_chk_emulated(u32 msr, u64 *val, bool is_read);
+int pmu_apic_update(uint32_t reg);
+u64 xen_read_pmc(int counter);
+
+void xen_hypercall_pv(void);
+
+#else
+
+static inline void xen_pv_pre_suspend(void) {}
+static inline void xen_pv_post_suspend(int suspend_cancelled) {}
+
+#endif /* CONFIG_XEN_PV */
+
+DECLARE_PER_CPU_ALIGNED(struct vcpu_info, xen_vcpu_info);
+
+extern struct start_info *xen_start_info;
+extern struct shared_info xen_dummy_shared_info;
+extern struct shared_info *HYPERVISOR_shared_info;
+
+void xen_banner(void);
+void xen_vcpu_restore(void);
void xen_hvm_init_shared_info(void);
void xen_unplug_emulated_devices(void);
-void __init xen_build_dynamic_phys_to_machine(void);
-void __init xen_vmalloc_p2m_tree(void);
-
-void xen_init_irq_ops(void);
void xen_setup_timer(int cpu);
void xen_setup_runstate_info(int cpu);
void xen_teardown_timer(int cpu);
@@ -83,13 +180,10 @@ bool xen_vcpu_stolen(int vcpu);
void xen_vcpu_setup(int cpu);
void xen_vcpu_info_reset(int cpu);
-void xen_setup_vcpu_info_placement(void);
#ifdef CONFIG_SMP
void xen_smp_init(void);
void __init xen_hvm_smp_init(void);
-
-extern cpumask_var_t xen_cpu_initialized_map;
#else
static inline void xen_smp_init(void) {}
static inline void xen_hvm_smp_init(void) {}
@@ -125,8 +219,6 @@ static inline void __init xen_init_vga(const struct dom0_vga_console_info *info,
void xen_add_preferred_consoles(void);
-void __init xen_init_apic(void);
-
#ifdef CONFIG_XEN_EFI
extern void xen_efi_init(struct boot_params *boot_params);
#else
@@ -135,16 +227,6 @@ static inline void __init xen_efi_init(struct boot_params *boot_params)
}
#endif
-__visible void xen_irq_enable_direct(void);
-__visible void xen_irq_disable_direct(void);
-__visible unsigned long xen_save_fl_direct(void);
-
-__visible unsigned long xen_read_cr2(void);
-__visible unsigned long xen_read_cr2_direct(void);
-
-/* These are not functions, and cannot be called normally */
-__visible void xen_iret(void);
-
extern int xen_panic_handler_init(void);
int xen_cpuhp_setup(int (*cpu_up_prepare_cb)(unsigned int),
@@ -153,16 +235,6 @@ int xen_cpuhp_setup(int (*cpu_up_prepare_cb)(unsigned int),
void xen_pin_vcpu(int cpu);
void xen_emergency_restart(void);
-void xen_force_evtchn_callback(void);
-
-#ifdef CONFIG_XEN_PV
-void xen_pv_pre_suspend(void);
-void xen_pv_post_suspend(int suspend_cancelled);
-void xen_start_kernel(struct start_info *si);
-#else
-static inline void xen_pv_pre_suspend(void) {}
-static inline void xen_pv_post_suspend(int suspend_cancelled) {}
-#endif
#ifdef CONFIG_XEN_PVHVM
void xen_hvm_post_suspend(int suspend_cancelled);
@@ -184,85 +256,9 @@ static inline void xen_hvm_post_suspend(int suspend_cancelled) {}
void xen_add_extra_mem(unsigned long start_pfn, unsigned long n_pfns);
-struct dentry * __init xen_init_debugfs(void);
-
-enum pt_level {
- PT_PGD,
- PT_P4D,
- PT_PUD,
- PT_PMD,
- PT_PTE
-};
-
bool __set_phys_to_machine(unsigned long pfn, unsigned long mfn);
-void set_pte_mfn(unsigned long vaddr, unsigned long pfn, pgprot_t flags);
-unsigned long xen_read_cr2_direct(void);
-void xen_init_mmu_ops(void);
void xen_hvm_init_mmu_ops(void);
-/* Multicalls */
-struct multicall_space
-{
- struct multicall_entry *mc;
- void *args;
-};
-
-/* Allocate room for a multicall and its args */
-struct multicall_space __xen_mc_entry(size_t args);
-
-DECLARE_PER_CPU(unsigned long, xen_mc_irq_flags);
-
-/* Call to start a batch of multiple __xen_mc_entry()s. Must be
- paired with xen_mc_issue() */
-static inline void xen_mc_batch(void)
-{
- unsigned long flags;
-
- /* need to disable interrupts until this entry is complete */
- local_irq_save(flags);
- trace_xen_mc_batch(xen_get_lazy_mode());
- __this_cpu_write(xen_mc_irq_flags, flags);
-}
-
-static inline struct multicall_space xen_mc_entry(size_t args)
-{
- xen_mc_batch();
- return __xen_mc_entry(args);
-}
-
-/* Flush all pending multicalls */
-void xen_mc_flush(void);
-
-/* Issue a multicall if we're not in a lazy mode */
-static inline void xen_mc_issue(unsigned mode)
-{
- trace_xen_mc_issue(mode);
-
- if ((xen_get_lazy_mode() & mode) == 0)
- xen_mc_flush();
-
- /* restore flags saved in xen_mc_batch */
- local_irq_restore(this_cpu_read(xen_mc_irq_flags));
-}
-
-/* Set up a callback to be called when the current batch is flushed */
-void xen_mc_callback(void (*fn)(void *), void *data);
-
-/*
- * Try to extend the arguments of the previous multicall command. The
- * previous command's op must match. If it does, then it attempts to
- * extend the argument space allocated to the multicall entry by
- * arg_size bytes.
- *
- * The returned multicall_space will return with mc pointing to the
- * command on success, or NULL on failure, and args pointing to the
- * newly allocated space.
- */
-struct multicall_space xen_mc_extend_args(unsigned long op, size_t arg_size);
-
-extern bool is_xen_pmu;
-
-irqreturn_t xen_pmu_irq_handler(int irq, void *dev_id);
#ifdef CONFIG_XEN_HAVE_VPMU
void xen_pmu_init(int cpu);
void xen_pmu_finish(int cpu);
@@ -270,9 +266,6 @@ void xen_pmu_finish(int cpu);
static inline void xen_pmu_init(int cpu) {}
static inline void xen_pmu_finish(int cpu) {}
#endif
-bool pmu_msr_chk_emulated(u32 msr, u64 *val, bool is_read);
-int pmu_apic_update(uint32_t reg);
-u64 xen_read_pmc(int counter);
#ifdef CONFIG_SMP
@@ -321,9 +314,6 @@ static inline void xen_smp_intr_free_pv(unsigned int cpu) {}
static inline void xen_smp_count_cpus(void) { }
#endif /* CONFIG_SMP */
-#ifdef CONFIG_XEN_PV
-void xen_hypercall_pv(void);
-#endif
void xen_hypercall_hvm(void);
void xen_hypercall_amd(void);
void xen_hypercall_intel(void);
diff --git a/drivers/accel/amdxdna/aie2_ctx.c b/drivers/accel/amdxdna/aie2_ctx.c
index 286379d..eed3d0e 100644
--- a/drivers/accel/amdxdna/aie2_ctx.c
+++ b/drivers/accel/amdxdna/aie2_ctx.c
@@ -999,6 +999,7 @@ static int aie2_populate_range(struct amdxdna_gem_obj *abo)
if (ret == -EBUSY) {
amdxdna_umap_put(mapp);
+ mmput(mm);
goto again;
}
@@ -1009,11 +1010,13 @@ static int aie2_populate_range(struct amdxdna_gem_obj *abo)
if (mmu_interval_read_retry(&mapp->notifier, mapp->range.notifier_seq)) {
up_write(&xdna->notifier_lock);
amdxdna_umap_put(mapp);
+ mmput(mm);
goto again;
}
mapp->invalid = false;
up_write(&xdna->notifier_lock);
amdxdna_umap_put(mapp);
+ mmput(mm);
goto again;
put_mm:
diff --git a/drivers/accel/ivpu/ivpu_ipc.c b/drivers/accel/ivpu/ivpu_ipc.c
index f47df09..9347f05 100644
--- a/drivers/accel/ivpu/ivpu_ipc.c
+++ b/drivers/accel/ivpu/ivpu_ipc.c
@@ -276,7 +276,7 @@ int ivpu_ipc_receive(struct ivpu_device *vdev, struct ivpu_ipc_consumer *cons,
if (ipc_buf)
memcpy(ipc_buf, rx_msg->ipc_hdr, sizeof(*ipc_buf));
if (rx_msg->jsm_msg) {
- u32 size = min_t(int, rx_msg->ipc_hdr->data_size, sizeof(*jsm_msg));
+ u32 size = min(rx_msg->ipc_hdr->data_size, sizeof(*jsm_msg));
if (rx_msg->jsm_msg->result != VPU_JSM_STATUS_SUCCESS) {
ivpu_err(vdev, "IPC resp result error: %d\n", rx_msg->jsm_msg->result);
diff --git a/drivers/acpi/ac.c b/drivers/acpi/ac.c
index 27f3174..f19d6dd 100644
--- a/drivers/acpi/ac.c
+++ b/drivers/acpi/ac.c
@@ -193,6 +193,7 @@ static const struct dmi_system_id ac_dmi_table[] __initconst = {
static int acpi_ac_probe(struct platform_device *pdev)
{
struct power_supply_config psy_cfg = {};
+ struct device *dev = &pdev->dev;
struct acpi_device *adev;
struct acpi_ac *ac;
int result;
@@ -201,7 +202,7 @@ static int acpi_ac_probe(struct platform_device *pdev)
if (!adev)
return -ENODEV;
- ac = kzalloc_obj(struct acpi_ac);
+ ac = devm_kzalloc(dev, sizeof(*ac), GFP_KERNEL);
if (!ac)
return -ENOMEM;
@@ -211,7 +212,7 @@ static int acpi_ac_probe(struct platform_device *pdev)
result = acpi_ac_get_state(ac);
if (result)
- goto err_release_ac;
+ return result;
psy_cfg.drv_data = ac;
@@ -220,33 +221,22 @@ static int acpi_ac_probe(struct platform_device *pdev)
ac->charger_desc.properties = ac_props;
ac->charger_desc.num_properties = ARRAY_SIZE(ac_props);
ac->charger_desc.get_property = get_ac_property;
- ac->charger = power_supply_register(&pdev->dev,
- &ac->charger_desc, &psy_cfg);
- if (IS_ERR(ac->charger)) {
- result = PTR_ERR(ac->charger);
- goto err_release_ac;
- }
+ ac->charger = devm_power_supply_register(dev, &ac->charger_desc, &psy_cfg);
+ if (IS_ERR(ac->charger))
+ return PTR_ERR(ac->charger);
pr_info("AC Adapter [%s] (%s-line)\n", acpi_device_bid(adev),
str_on_off(ac->state));
+ result = devm_acpi_install_notify_handler(dev, ACPI_ALL_NOTIFY,
+ acpi_ac_notify, ac);
+ if (result)
+ return result;
+
ac->battery_nb.notifier_call = acpi_ac_battery_notify;
register_acpi_notifier(&ac->battery_nb);
- result = acpi_dev_install_notify_handler(adev, ACPI_ALL_NOTIFY,
- acpi_ac_notify, ac);
- if (result)
- goto err_unregister;
-
return 0;
-
-err_unregister:
- power_supply_unregister(ac->charger);
- unregister_acpi_notifier(&ac->battery_nb);
-err_release_ac:
- kfree(ac);
-
- return result;
}
#ifdef CONFIG_PM_SLEEP
@@ -271,12 +261,7 @@ static void acpi_ac_remove(struct platform_device *pdev)
{
struct acpi_ac *ac = platform_get_drvdata(pdev);
- acpi_dev_remove_notify_handler(ac->device, ACPI_ALL_NOTIFY,
- acpi_ac_notify);
- power_supply_unregister(ac->charger);
unregister_acpi_notifier(&ac->battery_nb);
-
- kfree(ac);
}
static struct platform_driver acpi_ac_driver = {
diff --git a/drivers/acpi/acpi_ipmi.c b/drivers/acpi/acpi_ipmi.c
index 8f1aeae..79ce6e7 100644
--- a/drivers/acpi/acpi_ipmi.c
+++ b/drivers/acpi/acpi_ipmi.c
@@ -550,7 +550,6 @@ acpi_ipmi_space_handler(u32 function, acpi_physical_address address,
return AE_TYPE;
}
- acpi_ipmi_msg_get(tx_msg);
mutex_lock(&driver_data.ipmi_lock);
/* Do not add a tx_msg that can not be flushed. */
if (ipmi_device->dead) {
@@ -558,6 +557,7 @@ acpi_ipmi_space_handler(u32 function, acpi_physical_address address,
ipmi_msg_release(tx_msg);
return AE_NOT_EXIST;
}
+ acpi_ipmi_msg_get(tx_msg);
spin_lock_irqsave(&ipmi_device->tx_msg_lock, flags);
list_add_tail(&tx_msg->head, &ipmi_device->tx_msg_list);
spin_unlock_irqrestore(&ipmi_device->tx_msg_lock, flags);
diff --git a/drivers/acpi/acpi_pad.c b/drivers/acpi/acpi_pad.c
index ec94b09..dc6d009 100644
--- a/drivers/acpi/acpi_pad.c
+++ b/drivers/acpi/acpi_pad.c
@@ -31,6 +31,8 @@
static DEFINE_MUTEX(isolated_cpus_lock);
static DEFINE_MUTEX(round_robin_lock);
+static bool acpi_pad_teardown;
+
static unsigned int power_saving_mwait_eax;
static unsigned char tsc_detected_unstable;
@@ -334,8 +336,8 @@ static ssize_t idlecpus_store(struct device *dev,
static ssize_t idlecpus_show(struct device *dev,
struct device_attribute *attr, char *buf)
{
- return cpumap_print_to_pagebuf(false, buf,
- to_cpumask(pad_busy_cpus_bits));
+ return sysfs_emit(buf, "%*pb\n",
+ cpumask_pr_args(to_cpumask(pad_busy_cpus_bits)));
}
static DEVICE_ATTR_RW(idlecpus);
@@ -359,6 +361,9 @@ static int acpi_pad_pur(acpi_handle handle)
union acpi_object *package;
int num = -1;
+ if (unlikely(acpi_pad_teardown))
+ return -1;
+
if (ACPI_FAILURE(acpi_evaluate_object(handle, "_PUR", NULL, &buffer)))
return num;
@@ -407,40 +412,29 @@ static void acpi_pad_handle_notify(acpi_handle handle)
static void acpi_pad_notify(acpi_handle handle, u32 event, void *data)
{
- struct acpi_device *adev = data;
-
- switch (event) {
- case ACPI_PROCESSOR_AGGREGATOR_NOTIFY:
- acpi_pad_handle_notify(handle);
- acpi_bus_generate_netlink_event("acpi_pad",
- dev_name(&adev->dev), event, 0);
- break;
- default:
+ if (event != ACPI_PROCESSOR_AGGREGATOR_NOTIFY) {
pr_warn("Unsupported event [0x%x]\n", event);
- break;
+ return;
}
+
+ acpi_pad_handle_notify(handle);
+ acpi_bus_generate_netlink_event("acpi_pad", dev_name(data), event, 0);
}
static int acpi_pad_probe(struct platform_device *pdev)
{
- struct acpi_device *adev;
+ acpi_pad_teardown = false;
- adev = ACPI_COMPANION(&pdev->dev);
- if (!adev)
- return -ENODEV;
-
- return acpi_dev_install_notify_handler(adev, ACPI_DEVICE_NOTIFY,
- acpi_pad_notify, adev);
+ return devm_acpi_install_notify_handler(&pdev->dev, ACPI_DEVICE_NOTIFY,
+ acpi_pad_notify, &pdev->dev);
}
static void acpi_pad_remove(struct platform_device *pdev)
{
mutex_lock(&isolated_cpus_lock);
+ acpi_pad_teardown = true;
acpi_pad_idle_cpus(0);
mutex_unlock(&isolated_cpus_lock);
-
- acpi_dev_remove_notify_handler(ACPI_COMPANION(&pdev->dev),
- ACPI_DEVICE_NOTIFY, acpi_pad_notify);
}
static const struct acpi_device_id pad_device_ids[] = {
diff --git a/drivers/acpi/acpi_video.c b/drivers/acpi/acpi_video.c
index 05793dd..f93e877 100644
--- a/drivers/acpi/acpi_video.c
+++ b/drivers/acpi/acpi_video.c
@@ -63,7 +63,7 @@ MODULE_PARM_DESC(hw_changes_brightness,
* Whether the struct acpi_video_device_attrib::device_id_scheme bit should be
* assumed even if not actually set.
*/
-static bool device_id_scheme = false;
+static bool device_id_scheme;
module_param(device_id_scheme, bool, 0444);
static int only_lcd;
@@ -76,7 +76,6 @@ static DEFINE_MUTEX(video_list_lock);
static LIST_HEAD(video_bus_head);
static int acpi_video_bus_probe(struct auxiliary_device *aux_dev,
const struct auxiliary_device_id *id);
-static void acpi_video_bus_remove(struct auxiliary_device *aux);
static void acpi_video_bus_notify(acpi_handle handle, u32 event, void *data);
/*
@@ -99,7 +98,6 @@ MODULE_DEVICE_TABLE(auxiliary, video_bus_auxiliary_id_table);
static struct auxiliary_driver acpi_video_bus = {
.probe = acpi_video_bus_probe,
- .remove = acpi_video_bus_remove,
.id_table = video_bus_auxiliary_id_table,
};
@@ -1494,10 +1492,31 @@ int acpi_video_get_edid(struct acpi_device *device, int type, int device_id,
}
EXPORT_SYMBOL(acpi_video_get_edid);
-static int
-acpi_video_bus_get_devices(struct acpi_video_bus *video,
- struct acpi_device *device)
+static void acpi_video_bus_put_devices(void *data)
{
+ struct acpi_video_bus *video = data;
+ struct acpi_video_device *dev, *next;
+
+ mutex_lock(&video->device_list_lock);
+ list_for_each_entry_safe(dev, next, &video->video_device_list, entry) {
+ list_del(&dev->entry);
+ kfree(dev);
+ }
+ mutex_unlock(&video->device_list_lock);
+
+ kfree(video->attached_array);
+ video->attached_array = NULL;
+}
+
+static int devm_acpi_video_bus_get_devices(struct device *dev,
+ struct acpi_video_bus *video)
+{
+ int ret;
+
+ ret = devm_add_action(dev, acpi_video_bus_put_devices, video);
+ if (ret)
+ return ret;
+
/*
* There are systems where video module known to work fine regardless
* of broken _DOD and ignoring returned value here doesn't cause
@@ -1505,7 +1524,8 @@ acpi_video_bus_get_devices(struct acpi_video_bus *video,
*/
acpi_video_device_enumerate(video);
- return acpi_dev_for_each_child(device, acpi_video_bus_get_one_device, video);
+ return acpi_dev_for_each_child(video->device,
+ acpi_video_bus_get_one_device, video);
}
/* acpi_video interface */
@@ -1923,8 +1943,9 @@ static void acpi_video_dev_remove_notify_handler(struct acpi_video_device *dev)
}
}
-static void acpi_video_bus_remove_notify_handler(struct acpi_video_bus *video)
+static void acpi_video_bus_remove_notify_handler(void *data)
{
+ struct acpi_video_bus *video = data;
struct acpi_video_device *dev;
mutex_lock(&video->device_list_lock);
@@ -1939,18 +1960,23 @@ static void acpi_video_bus_remove_notify_handler(struct acpi_video_bus *video)
video->input = NULL;
}
-static int acpi_video_bus_put_devices(struct acpi_video_bus *video)
+static void acpi_video_bus_free(void *data)
{
- struct acpi_video_device *dev, *next;
+ struct acpi_video_bus *video = data;
- mutex_lock(&video->device_list_lock);
- list_for_each_entry_safe(dev, next, &video->video_device_list, entry) {
- list_del(&dev->entry);
- kfree(dev);
- }
- mutex_unlock(&video->device_list_lock);
+ video->device->driver_data = NULL;
+ kfree(video);
+}
- return 0;
+static void acpi_video_bus_del(void *data)
+{
+ struct acpi_video_bus *video = data;
+
+ mutex_lock(&video_list_lock);
+ list_del(&video->entry);
+ mutex_unlock(&video_list_lock);
+
+ acpi_video_bus_unregister_backlight(video);
}
static int duplicate_dev_check(struct device *sibling, void *data)
@@ -1978,7 +2004,8 @@ static bool acpi_video_bus_dev_is_duplicate(struct device *dev)
static int acpi_video_bus_probe(struct auxiliary_device *aux_dev,
const struct auxiliary_device_id *id_unused)
{
- struct acpi_device *device = ACPI_COMPANION(&aux_dev->dev);
+ struct device *dev = &aux_dev->dev;
+ struct acpi_device *device = ACPI_COMPANION(dev);
static DEFINE_MUTEX(probe_lock);
struct acpi_video_bus *video;
static int instance;
@@ -1988,7 +2015,7 @@ static int acpi_video_bus_probe(struct auxiliary_device *aux_dev,
/* Probe one video bus device at a time in case there are duplicates. */
guard(mutex)(&probe_lock);
- if (!allow_duplicates && acpi_video_bus_dev_is_duplicate(&aux_dev->dev)) {
+ if (!allow_duplicates && acpi_video_bus_dev_is_duplicate(dev)) {
pr_info(FW_BUG
"Duplicate ACPI video bus devices for the"
" same VGA controller, please try module "
@@ -2001,6 +2028,13 @@ static int acpi_video_bus_probe(struct auxiliary_device *aux_dev,
if (!video)
return -ENOMEM;
+ video->device = device;
+ device->driver_data = video;
+
+ error = devm_add_action_or_reset(dev, acpi_video_bus_free, video);
+ if (error)
+ return error;
+
/*
* A hack to fix the duplicate name "VID" problem on T61 and the
* duplicate name "VGA" problem on Pa 3553.
@@ -2015,20 +2049,17 @@ static int acpi_video_bus_probe(struct auxiliary_device *aux_dev,
auxiliary_set_drvdata(aux_dev, video);
- video->device = device;
- device->driver_data = video;
-
acpi_video_bus_find_cap(video);
error = acpi_video_bus_check(video);
if (error)
- goto err_free_video;
+ return error;
mutex_init(&video->device_list_lock);
INIT_LIST_HEAD(&video->video_device_list);
- error = acpi_video_bus_get_devices(video, device);
+ error = devm_acpi_video_bus_get_devices(dev, video);
if (error)
- goto err_put_video;
+ return error;
/*
* HP ZBook Fury 16 G10 requires ACPI video's child devices have _PS0
@@ -2040,10 +2071,6 @@ static int acpi_video_bus_probe(struct auxiliary_device *aux_dev,
acpi_device_bid(device), str_yes_no(video->flags.multihead),
str_yes_no(video->flags.rom), str_yes_no(video->flags.post));
- mutex_lock(&video_list_lock);
- list_add_tail(&video->entry, &video_bus_head);
- mutex_unlock(&video_list_lock);
-
/*
* If backlight-type auto-detection is used then a native backlight may
* show up later and this may change the result from video to native.
@@ -2059,53 +2086,25 @@ static int acpi_video_bus_probe(struct auxiliary_device *aux_dev,
!auto_detect)
acpi_video_bus_register_backlight(video);
- error = acpi_video_bus_add_notify_handler(video, &aux_dev->dev);
- if (error)
- goto err_del;
+ mutex_lock(&video_list_lock);
+ list_add_tail(&video->entry, &video_bus_head);
+ mutex_unlock(&video_list_lock);
- error = acpi_dev_install_notify_handler(device, ACPI_DEVICE_NOTIFY,
+ error = devm_add_action_or_reset(dev, acpi_video_bus_del, video);
+ if (error)
+ return error;
+
+ error = acpi_video_bus_add_notify_handler(video, dev);
+ if (error)
+ return error;
+
+ error = devm_add_action_or_reset(dev, acpi_video_bus_remove_notify_handler,
+ video);
+ if (error)
+ return error;
+
+ return devm_acpi_install_notify_handler(dev, ACPI_DEVICE_NOTIFY,
acpi_video_bus_notify, video);
- if (error)
- goto err_remove;
-
- return 0;
-
-err_remove:
- acpi_video_bus_remove_notify_handler(video);
-err_del:
- mutex_lock(&video_list_lock);
- list_del(&video->entry);
- mutex_unlock(&video_list_lock);
- acpi_video_bus_unregister_backlight(video);
-err_put_video:
- acpi_video_bus_put_devices(video);
- kfree(video->attached_array);
-err_free_video:
- kfree(video);
- device->driver_data = NULL;
-
- return error;
-}
-
-static void acpi_video_bus_remove(struct auxiliary_device *aux_dev)
-{
- struct acpi_video_bus *video = auxiliary_get_drvdata(aux_dev);
- struct acpi_device *device = ACPI_COMPANION(&aux_dev->dev);
-
- acpi_dev_remove_notify_handler(device, ACPI_DEVICE_NOTIFY,
- acpi_video_bus_notify);
-
- mutex_lock(&video_list_lock);
- list_del(&video->entry);
- mutex_unlock(&video_list_lock);
-
- acpi_video_bus_remove_notify_handler(video);
- acpi_video_bus_unregister_backlight(video);
- acpi_video_bus_put_devices(video);
-
- kfree(video->attached_array);
- kfree(video);
- device->driver_data = NULL;
}
static int __init is_i740(struct pci_dev *dev)
diff --git a/drivers/acpi/acpica/acapps.h b/drivers/acpi/acpica/acapps.h
index d7d4649..16e7bff 100644
--- a/drivers/acpi/acpica/acapps.h
+++ b/drivers/acpi/acpica/acapps.h
@@ -3,7 +3,7 @@
*
* Module Name: acapps - common include for ACPI applications/tools
*
- * Copyright (C) 2000 - 2025, Intel Corp.
+ * Copyright (C) 2000 - 2026, Intel Corp.
*
*****************************************************************************/
@@ -17,7 +17,7 @@
/* Common info for tool signons */
#define ACPICA_NAME "Intel ACPI Component Architecture"
-#define ACPICA_COPYRIGHT "Copyright (c) 2000 - 2025 Intel Corporation"
+#define ACPICA_COPYRIGHT "Copyright (c) 2000 - 2026 Intel Corporation"
#if ACPI_MACHINE_WIDTH == 64
#define ACPI_WIDTH " (64-bit version)"
diff --git a/drivers/acpi/acpica/accommon.h b/drivers/acpi/acpica/accommon.h
index 662231f..b1cf926 100644
--- a/drivers/acpi/acpica/accommon.h
+++ b/drivers/acpi/acpica/accommon.h
@@ -3,7 +3,7 @@
*
* Name: accommon.h - Common include files for generation of ACPICA source
*
- * Copyright (C) 2000 - 2025, Intel Corp.
+ * Copyright (C) 2000 - 2026, Intel Corp.
*
*****************************************************************************/
diff --git a/drivers/acpi/acpica/acconvert.h b/drivers/acpi/acpica/acconvert.h
index 24998f2..22bd671 100644
--- a/drivers/acpi/acpica/acconvert.h
+++ b/drivers/acpi/acpica/acconvert.h
@@ -3,7 +3,7 @@
*
* Module Name: acapps - common include for ACPI applications/tools
*
- * Copyright (C) 2000 - 2025, Intel Corp.
+ * Copyright (C) 2000 - 2026, Intel Corp.
*
*****************************************************************************/
diff --git a/drivers/acpi/acpica/acdebug.h b/drivers/acpi/acpica/acdebug.h
index 91241bd..09ab8b3 100644
--- a/drivers/acpi/acpica/acdebug.h
+++ b/drivers/acpi/acpica/acdebug.h
@@ -3,7 +3,7 @@
*
* Name: acdebug.h - ACPI/AML debugger
*
- * Copyright (C) 2000 - 2025, Intel Corp.
+ * Copyright (C) 2000 - 2026, Intel Corp.
*
*****************************************************************************/
diff --git a/drivers/acpi/acpica/acdispat.h b/drivers/acpi/acpica/acdispat.h
index 5d48a34..f6208722 100644
--- a/drivers/acpi/acpica/acdispat.h
+++ b/drivers/acpi/acpica/acdispat.h
@@ -3,7 +3,7 @@
*
* Name: acdispat.h - dispatcher (parser to interpreter interface)
*
- * Copyright (C) 2000 - 2025, Intel Corp.
+ * Copyright (C) 2000 - 2026, Intel Corp.
*
*****************************************************************************/
diff --git a/drivers/acpi/acpica/acevents.h b/drivers/acpi/acpica/acevents.h
index b40fb3a..ccdc372 100644
--- a/drivers/acpi/acpica/acevents.h
+++ b/drivers/acpi/acpica/acevents.h
@@ -3,7 +3,7 @@
*
* Name: acevents.h - Event subcomponent prototypes and defines
*
- * Copyright (C) 2000 - 2025, Intel Corp.
+ * Copyright (C) 2000 - 2026, Intel Corp.
*
*****************************************************************************/
diff --git a/drivers/acpi/acpica/acglobal.h b/drivers/acpi/acpica/acglobal.h
index c8a750d..b6e3151 100644
--- a/drivers/acpi/acpica/acglobal.h
+++ b/drivers/acpi/acpica/acglobal.h
@@ -3,7 +3,7 @@
*
* Name: acglobal.h - Declarations for global variables
*
- * Copyright (C) 2000 - 2025, Intel Corp.
+ * Copyright (C) 2000 - 2026, Intel Corp.
*
*****************************************************************************/
diff --git a/drivers/acpi/acpica/achware.h b/drivers/acpi/acpica/achware.h
index 6aec56c..78323df 100644
--- a/drivers/acpi/acpica/achware.h
+++ b/drivers/acpi/acpica/achware.h
@@ -3,7 +3,7 @@
*
* Name: achware.h -- hardware specific interfaces
*
- * Copyright (C) 2000 - 2025, Intel Corp.
+ * Copyright (C) 2000 - 2026, Intel Corp.
*
*****************************************************************************/
diff --git a/drivers/acpi/acpica/acinterp.h b/drivers/acpi/acpica/acinterp.h
index 1ee6ac9..a62a4e9e 100644
--- a/drivers/acpi/acpica/acinterp.h
+++ b/drivers/acpi/acpica/acinterp.h
@@ -3,7 +3,7 @@
*
* Name: acinterp.h - Interpreter subcomponent prototypes and defines
*
- * Copyright (C) 2000 - 2025, Intel Corp.
+ * Copyright (C) 2000 - 2026, Intel Corp.
*
*****************************************************************************/
diff --git a/drivers/acpi/acpica/aclocal.h b/drivers/acpi/acpica/aclocal.h
index f986400..1ecc700 100644
--- a/drivers/acpi/acpica/aclocal.h
+++ b/drivers/acpi/acpica/aclocal.h
@@ -3,7 +3,7 @@
*
* Name: aclocal.h - Internal data types used across the ACPI subsystem
*
- * Copyright (C) 2000 - 2025, Intel Corp.
+ * Copyright (C) 2000 - 2026, Intel Corp.
*
*****************************************************************************/
@@ -169,6 +169,7 @@ struct acpi_namespace_node {
#define ANOBJ_IS_EXTERNAL 0x08 /* iASL only: This object created via External() */
#define ANOBJ_METHOD_NO_RETVAL 0x10 /* iASL only: Method has no return value */
#define ANOBJ_METHOD_SOME_NO_RETVAL 0x20 /* iASL only: Method has at least one return value */
+#define ANOBJ_IS_ALIAS 0x40 /* iASL only: Node is an alias to another node */
#define ANOBJ_IS_REFERENCED 0x80 /* iASL only: Object was referenced */
/* Internal ACPI table management - master table list */
diff --git a/drivers/acpi/acpica/acmacros.h b/drivers/acpi/acpica/acmacros.h
index 4e9402c..3a7b6db 100644
--- a/drivers/acpi/acpica/acmacros.h
+++ b/drivers/acpi/acpica/acmacros.h
@@ -3,7 +3,7 @@
*
* Name: acmacros.h - C macros for the entire subsystem.
*
- * Copyright (C) 2000 - 2025, Intel Corp.
+ * Copyright (C) 2000 - 2026, Intel Corp.
*
*****************************************************************************/
diff --git a/drivers/acpi/acpica/acnamesp.h b/drivers/acpi/acpica/acnamesp.h
index 13f050f..b5830e7 100644
--- a/drivers/acpi/acpica/acnamesp.h
+++ b/drivers/acpi/acpica/acnamesp.h
@@ -3,7 +3,7 @@
*
* Name: acnamesp.h - Namespace subcomponent prototypes and defines
*
- * Copyright (C) 2000 - 2025, Intel Corp.
+ * Copyright (C) 2000 - 2026, Intel Corp.
*
*****************************************************************************/
diff --git a/drivers/acpi/acpica/acobject.h b/drivers/acpi/acpica/acobject.h
index 6ffcc7a..9a82e4f 100644
--- a/drivers/acpi/acpica/acobject.h
+++ b/drivers/acpi/acpica/acobject.h
@@ -3,7 +3,7 @@
*
* Name: acobject.h - Definition of union acpi_operand_object (Internal object only)
*
- * Copyright (C) 2000 - 2025, Intel Corp.
+ * Copyright (C) 2000 - 2026, Intel Corp.
*
*****************************************************************************/
diff --git a/drivers/acpi/acpica/acopcode.h b/drivers/acpi/acpica/acopcode.h
index a2a9e51..d8a0beb 100644
--- a/drivers/acpi/acpica/acopcode.h
+++ b/drivers/acpi/acpica/acopcode.h
@@ -3,7 +3,7 @@
*
* Name: acopcode.h - AML opcode information for the AML parser and interpreter
*
- * Copyright (C) 2000 - 2025, Intel Corp.
+ * Copyright (C) 2000 - 2026, Intel Corp.
*
*****************************************************************************/
diff --git a/drivers/acpi/acpica/acparser.h b/drivers/acpi/acpica/acparser.h
index 65a15de..5393a11 100644
--- a/drivers/acpi/acpica/acparser.h
+++ b/drivers/acpi/acpica/acparser.h
@@ -3,7 +3,7 @@
*
* Module Name: acparser.h - AML Parser subcomponent prototypes and defines
*
- * Copyright (C) 2000 - 2025, Intel Corp.
+ * Copyright (C) 2000 - 2026, Intel Corp.
*
*****************************************************************************/
diff --git a/drivers/acpi/acpica/acpredef.h b/drivers/acpi/acpica/acpredef.h
index 07d5790..2208dc7 100644
--- a/drivers/acpi/acpica/acpredef.h
+++ b/drivers/acpi/acpica/acpredef.h
@@ -3,7 +3,7 @@
*
* Name: acpredef - Information table for ACPI predefined methods and objects
*
- * Copyright (C) 2000 - 2025, Intel Corp.
+ * Copyright (C) 2000 - 2026, Intel Corp.
*
*****************************************************************************/
diff --git a/drivers/acpi/acpica/acresrc.h b/drivers/acpi/acpica/acresrc.h
index e8a92be..c3dfa92 100644
--- a/drivers/acpi/acpica/acresrc.h
+++ b/drivers/acpi/acpica/acresrc.h
@@ -3,7 +3,7 @@
*
* Name: acresrc.h - Resource Manager function prototypes
*
- * Copyright (C) 2000 - 2025, Intel Corp.
+ * Copyright (C) 2000 - 2026, Intel Corp.
*
*****************************************************************************/
diff --git a/drivers/acpi/acpica/acstruct.h b/drivers/acpi/acpica/acstruct.h
index e690f60..5c8c14f 100644
--- a/drivers/acpi/acpica/acstruct.h
+++ b/drivers/acpi/acpica/acstruct.h
@@ -3,7 +3,7 @@
*
* Name: acstruct.h - Internal structs
*
- * Copyright (C) 2000 - 2025, Intel Corp.
+ * Copyright (C) 2000 - 2026, Intel Corp.
*
*****************************************************************************/
diff --git a/drivers/acpi/acpica/actables.h b/drivers/acpi/acpica/actables.h
index ebef72b..1334f86 100644
--- a/drivers/acpi/acpica/actables.h
+++ b/drivers/acpi/acpica/actables.h
@@ -3,7 +3,7 @@
*
* Name: actables.h - ACPI table management
*
- * Copyright (C) 2000 - 2025, Intel Corp.
+ * Copyright (C) 2000 - 2026, Intel Corp.
*
*****************************************************************************/
diff --git a/drivers/acpi/acpica/acutils.h b/drivers/acpi/acpica/acutils.h
index 3990d50..9a18cdb 100644
--- a/drivers/acpi/acpica/acutils.h
+++ b/drivers/acpi/acpica/acutils.h
@@ -3,7 +3,7 @@
*
* Name: acutils.h -- prototypes for the common (subsystem-wide) procedures
*
- * Copyright (C) 2000 - 2025, Intel Corp.
+ * Copyright (C) 2000 - 2026, Intel Corp.
*
*****************************************************************************/
diff --git a/drivers/acpi/acpica/amlcode.h b/drivers/acpi/acpica/amlcode.h
index c5b544a0..663c614 100644
--- a/drivers/acpi/acpica/amlcode.h
+++ b/drivers/acpi/acpica/amlcode.h
@@ -5,7 +5,7 @@
* Declarations and definitions contained herein are derived
* directly from the ACPI specification.
*
- * Copyright (C) 2000 - 2025, Intel Corp.
+ * Copyright (C) 2000 - 2026, Intel Corp.
*
*****************************************************************************/
diff --git a/drivers/acpi/acpica/amlresrc.h b/drivers/acpi/acpica/amlresrc.h
index 54d6e51..2c72568 100644
--- a/drivers/acpi/acpica/amlresrc.h
+++ b/drivers/acpi/acpica/amlresrc.h
@@ -3,7 +3,7 @@
*
* Module Name: amlresrc.h - AML resource descriptors
*
- * Copyright (C) 2000 - 2025, Intel Corp.
+ * Copyright (C) 2000 - 2026, Intel Corp.
*
*****************************************************************************/
diff --git a/drivers/acpi/acpica/dbhistry.c b/drivers/acpi/acpica/dbhistry.c
index 554ae35..beec446 100644
--- a/drivers/acpi/acpica/dbhistry.c
+++ b/drivers/acpi/acpica/dbhistry.c
@@ -3,7 +3,7 @@
*
* Module Name: dbhistry - debugger HISTORY command
*
- * Copyright (C) 2000 - 2025, Intel Corp.
+ * Copyright (C) 2000 - 2026, Intel Corp.
*
*****************************************************************************/
diff --git a/drivers/acpi/acpica/dsargs.c b/drivers/acpi/acpica/dsargs.c
index e2f00c5..8a72ff3 100644
--- a/drivers/acpi/acpica/dsargs.c
+++ b/drivers/acpi/acpica/dsargs.c
@@ -4,7 +4,7 @@
* Module Name: dsargs - Support for execution of dynamic arguments for static
* objects (regions, fields, buffer fields, etc.)
*
- * Copyright (C) 2000 - 2025, Intel Corp.
+ * Copyright (C) 2000 - 2026, Intel Corp.
*
*****************************************************************************/
diff --git a/drivers/acpi/acpica/dscontrol.c b/drivers/acpi/acpica/dscontrol.c
index c1f79d7..90f16c0 100644
--- a/drivers/acpi/acpica/dscontrol.c
+++ b/drivers/acpi/acpica/dscontrol.c
@@ -4,7 +4,7 @@
* Module Name: dscontrol - Support for execution control opcodes -
* if/else/while/return
*
- * Copyright (C) 2000 - 2025, Intel Corp.
+ * Copyright (C) 2000 - 2026, Intel Corp.
*
*****************************************************************************/
diff --git a/drivers/acpi/acpica/dsdebug.c b/drivers/acpi/acpica/dsdebug.c
index 274b742..691c001 100644
--- a/drivers/acpi/acpica/dsdebug.c
+++ b/drivers/acpi/acpica/dsdebug.c
@@ -3,7 +3,7 @@
*
* Module Name: dsdebug - Parser/Interpreter interface - debugging
*
- * Copyright (C) 2000 - 2025, Intel Corp.
+ * Copyright (C) 2000 - 2026, Intel Corp.
*
*****************************************************************************/
diff --git a/drivers/acpi/acpica/dsfield.c b/drivers/acpi/acpica/dsfield.c
index df132c9..d7d56cc6 100644
--- a/drivers/acpi/acpica/dsfield.c
+++ b/drivers/acpi/acpica/dsfield.c
@@ -3,7 +3,7 @@
*
* Module Name: dsfield - Dispatcher field routines
*
- * Copyright (C) 2000 - 2025, Intel Corp.
+ * Copyright (C) 2000 - 2026, Intel Corp.
*
*****************************************************************************/
diff --git a/drivers/acpi/acpica/dsinit.c b/drivers/acpi/acpica/dsinit.c
index 57cd9e2..f73e680 100644
--- a/drivers/acpi/acpica/dsinit.c
+++ b/drivers/acpi/acpica/dsinit.c
@@ -3,7 +3,7 @@
*
* Module Name: dsinit - Object initialization namespace walk
*
- * Copyright (C) 2000 - 2025, Intel Corp.
+ * Copyright (C) 2000 - 2026, Intel Corp.
*
*****************************************************************************/
diff --git a/drivers/acpi/acpica/dsmethod.c b/drivers/acpi/acpica/dsmethod.c
index 45ec32e..3b2ced5 100644
--- a/drivers/acpi/acpica/dsmethod.c
+++ b/drivers/acpi/acpica/dsmethod.c
@@ -3,7 +3,7 @@
*
* Module Name: dsmethod - Parser/Interpreter interface - control method parsing
*
- * Copyright (C) 2000 - 2025, Intel Corp.
+ * Copyright (C) 2000 - 2026, Intel Corp.
*
*****************************************************************************/
@@ -705,6 +705,8 @@ void
acpi_ds_terminate_control_method(union acpi_operand_object *method_desc,
struct acpi_walk_state *walk_state)
{
+ u32 i;
+ struct acpi_namespace_node *ref_node;
ACPI_FUNCTION_TRACE_PTR(ds_terminate_control_method, walk_state);
@@ -715,6 +717,47 @@ acpi_ds_terminate_control_method(union acpi_operand_object *method_desc,
}
if (walk_state) {
+ /*
+ * Check if the return value is a ref_of reference to a method local
+ * or argument. If so, clear the reference to avoid use-after-free
+ * when the walk state is deleted.
+ */
+ if (walk_state->return_desc &&
+ (walk_state->return_desc->common.type ==
+ ACPI_TYPE_LOCAL_REFERENCE)
+ && (walk_state->return_desc->reference.class ==
+ ACPI_REFCLASS_REFOF)) {
+ ref_node = walk_state->return_desc->reference.object;
+ if (ref_node) {
+
+ /* Check against method locals */
+ for (i = 0; i < ACPI_METHOD_NUM_LOCALS; i++) {
+ if (ref_node ==
+ &walk_state->local_variables[i]) {
+ acpi_ut_remove_reference
+ (walk_state->return_desc);
+ walk_state->return_desc = NULL;
+ break;
+ }
+ }
+
+ /* Check against method arguments if not already cleared */
+ if (walk_state->return_desc) {
+ for (i = 0; i < ACPI_METHOD_NUM_ARGS;
+ i++) {
+ if (ref_node ==
+ &walk_state->arguments[i]) {
+ acpi_ut_remove_reference
+ (walk_state->
+ return_desc);
+ walk_state->
+ return_desc = NULL;
+ break;
+ }
+ }
+ }
+ }
+ }
/* Delete all arguments and locals */
diff --git a/drivers/acpi/acpica/dsobject.c b/drivers/acpi/acpica/dsobject.c
index 1bf7eec..25dd034 100644
--- a/drivers/acpi/acpica/dsobject.c
+++ b/drivers/acpi/acpica/dsobject.c
@@ -3,7 +3,7 @@
*
* Module Name: dsobject - Dispatcher object management routines
*
- * Copyright (C) 2000 - 2025, Intel Corp.
+ * Copyright (C) 2000 - 2026, Intel Corp.
*
*****************************************************************************/
diff --git a/drivers/acpi/acpica/dsopcode.c b/drivers/acpi/acpica/dsopcode.c
index 5699b08..fa13086 100644
--- a/drivers/acpi/acpica/dsopcode.c
+++ b/drivers/acpi/acpica/dsopcode.c
@@ -3,7 +3,7 @@
*
* Module Name: dsopcode - Dispatcher support for regions and fields
*
- * Copyright (C) 2000 - 2025, Intel Corp.
+ * Copyright (C) 2000 - 2026, Intel Corp.
*
*****************************************************************************/
diff --git a/drivers/acpi/acpica/dspkginit.c b/drivers/acpi/acpica/dspkginit.c
index 1ed2386..1a33c2f 100644
--- a/drivers/acpi/acpica/dspkginit.c
+++ b/drivers/acpi/acpica/dspkginit.c
@@ -3,7 +3,7 @@
*
* Module Name: dspkginit - Completion of deferred package initialization
*
- * Copyright (C) 2000 - 2025, Intel Corp.
+ * Copyright (C) 2000 - 2026, Intel Corp.
*
*****************************************************************************/
diff --git a/drivers/acpi/acpica/dswexec.c b/drivers/acpi/acpica/dswexec.c
index 5c5c6d8..675aaa6 100644
--- a/drivers/acpi/acpica/dswexec.c
+++ b/drivers/acpi/acpica/dswexec.c
@@ -4,7 +4,7 @@
* Module Name: dswexec - Dispatcher method execution callbacks;
* dispatch to interpreter.
*
- * Copyright (C) 2000 - 2025, Intel Corp.
+ * Copyright (C) 2000 - 2026, Intel Corp.
*
*****************************************************************************/
diff --git a/drivers/acpi/acpica/dswload.c b/drivers/acpi/acpica/dswload.c
index 666419b..5a3709a 100644
--- a/drivers/acpi/acpica/dswload.c
+++ b/drivers/acpi/acpica/dswload.c
@@ -3,7 +3,7 @@
*
* Module Name: dswload - Dispatcher first pass namespace load callbacks
*
- * Copyright (C) 2000 - 2025, Intel Corp.
+ * Copyright (C) 2000 - 2026, Intel Corp.
*
*****************************************************************************/
diff --git a/drivers/acpi/acpica/dswload2.c b/drivers/acpi/acpica/dswload2.c
index bfc54c9..277ae08 100644
--- a/drivers/acpi/acpica/dswload2.c
+++ b/drivers/acpi/acpica/dswload2.c
@@ -3,7 +3,7 @@
*
* Module Name: dswload2 - Dispatcher second pass namespace load callbacks
*
- * Copyright (C) 2000 - 2025, Intel Corp.
+ * Copyright (C) 2000 - 2026, Intel Corp.
*
*****************************************************************************/
diff --git a/drivers/acpi/acpica/dswscope.c b/drivers/acpi/acpica/dswscope.c
index 375a8fa..7fef0a0 100644
--- a/drivers/acpi/acpica/dswscope.c
+++ b/drivers/acpi/acpica/dswscope.c
@@ -3,7 +3,7 @@
*
* Module Name: dswscope - Scope stack manipulation
*
- * Copyright (C) 2000 - 2025, Intel Corp.
+ * Copyright (C) 2000 - 2026, Intel Corp.
*
*****************************************************************************/
diff --git a/drivers/acpi/acpica/dswstate.c b/drivers/acpi/acpica/dswstate.c
index 02aaddb..5e94885 100644
--- a/drivers/acpi/acpica/dswstate.c
+++ b/drivers/acpi/acpica/dswstate.c
@@ -3,7 +3,7 @@
*
* Module Name: dswstate - Dispatcher parse tree walk management routines
*
- * Copyright (C) 2000 - 2025, Intel Corp.
+ * Copyright (C) 2000 - 2026, Intel Corp.
*
*****************************************************************************/
diff --git a/drivers/acpi/acpica/evevent.c b/drivers/acpi/acpica/evevent.c
index 6cdd39c..8b12e91 100644
--- a/drivers/acpi/acpica/evevent.c
+++ b/drivers/acpi/acpica/evevent.c
@@ -3,7 +3,7 @@
*
* Module Name: evevent - Fixed Event handling and dispatch
*
- * Copyright (C) 2000 - 2025, Intel Corp.
+ * Copyright (C) 2000 - 2026, Intel Corp.
*
*****************************************************************************/
diff --git a/drivers/acpi/acpica/evglock.c b/drivers/acpi/acpica/evglock.c
index df2a4ab..d1a266e 100644
--- a/drivers/acpi/acpica/evglock.c
+++ b/drivers/acpi/acpica/evglock.c
@@ -3,7 +3,7 @@
*
* Module Name: evglock - Global Lock support
*
- * Copyright (C) 2000 - 2025, Intel Corp.
+ * Copyright (C) 2000 - 2026, Intel Corp.
*
*****************************************************************************/
diff --git a/drivers/acpi/acpica/evgpe.c b/drivers/acpi/acpica/evgpe.c
index ba65b2e..53b52ef 100644
--- a/drivers/acpi/acpica/evgpe.c
+++ b/drivers/acpi/acpica/evgpe.c
@@ -3,7 +3,7 @@
*
* Module Name: evgpe - General Purpose Event handling and dispatch
*
- * Copyright (C) 2000 - 2025, Intel Corp.
+ * Copyright (C) 2000 - 2026, Intel Corp.
*
*****************************************************************************/
diff --git a/drivers/acpi/acpica/evgpeblk.c b/drivers/acpi/acpica/evgpeblk.c
index fadd93c..47d8d50 100644
--- a/drivers/acpi/acpica/evgpeblk.c
+++ b/drivers/acpi/acpica/evgpeblk.c
@@ -3,7 +3,7 @@
*
* Module Name: evgpeblk - GPE block creation and initialization.
*
- * Copyright (C) 2000 - 2025, Intel Corp.
+ * Copyright (C) 2000 - 2026, Intel Corp.
*
*****************************************************************************/
diff --git a/drivers/acpi/acpica/evgpeinit.c b/drivers/acpi/acpica/evgpeinit.c
index eb76973..3317449 100644
--- a/drivers/acpi/acpica/evgpeinit.c
+++ b/drivers/acpi/acpica/evgpeinit.c
@@ -3,7 +3,7 @@
*
* Module Name: evgpeinit - System GPE initialization and update
*
- * Copyright (C) 2000 - 2025, Intel Corp.
+ * Copyright (C) 2000 - 2026, Intel Corp.
*
*****************************************************************************/
diff --git a/drivers/acpi/acpica/evgpeutil.c b/drivers/acpi/acpica/evgpeutil.c
index d15b1d7..4f8af92 100644
--- a/drivers/acpi/acpica/evgpeutil.c
+++ b/drivers/acpi/acpica/evgpeutil.c
@@ -3,7 +3,7 @@
*
* Module Name: evgpeutil - GPE utilities
*
- * Copyright (C) 2000 - 2025, Intel Corp.
+ * Copyright (C) 2000 - 2026, Intel Corp.
*
*****************************************************************************/
diff --git a/drivers/acpi/acpica/evhandler.c b/drivers/acpi/acpica/evhandler.c
index 5a35dae..d9a3b1c 100644
--- a/drivers/acpi/acpica/evhandler.c
+++ b/drivers/acpi/acpica/evhandler.c
@@ -3,7 +3,7 @@
*
* Module Name: evhandler - Support for Address Space handlers
*
- * Copyright (C) 2000 - 2025, Intel Corp.
+ * Copyright (C) 2000 - 2026, Intel Corp.
*
*****************************************************************************/
@@ -130,6 +130,14 @@ acpi_ev_has_default_handler(struct acpi_namespace_node *node,
/* Walk the linked list of handlers for this object */
while (handler_obj) {
+
+ /* Validate handler object type before accessing fields */
+
+ if (handler_obj->common.type !=
+ ACPI_TYPE_LOCAL_ADDRESS_HANDLER) {
+ break;
+ }
+
if (handler_obj->address_space.space_id == space_id) {
if (handler_obj->address_space.handler_flags &
ACPI_ADDR_HANDLER_DEFAULT_INSTALLED) {
@@ -292,6 +300,9 @@ union acpi_operand_object *acpi_ev_find_region_handler(acpi_adr_space_type
/* Walk the handler list for this device */
while (handler_obj) {
+ if (handler_obj->common.type != ACPI_TYPE_LOCAL_ADDRESS_HANDLER) {
+ break;
+ }
/* Same space_id indicates a handler is installed */
diff --git a/drivers/acpi/acpica/evmisc.c b/drivers/acpi/acpica/evmisc.c
index 04a23a6..723db68 100644
--- a/drivers/acpi/acpica/evmisc.c
+++ b/drivers/acpi/acpica/evmisc.c
@@ -3,7 +3,7 @@
*
* Module Name: evmisc - Miscellaneous event manager support functions
*
- * Copyright (C) 2000 - 2025, Intel Corp.
+ * Copyright (C) 2000 - 2026, Intel Corp.
*
*****************************************************************************/
diff --git a/drivers/acpi/acpica/evregion.c b/drivers/acpi/acpica/evregion.c
index b6198f7..96423b3 100644
--- a/drivers/acpi/acpica/evregion.c
+++ b/drivers/acpi/acpica/evregion.c
@@ -3,7 +3,7 @@
*
* Module Name: evregion - Operation Region support
*
- * Copyright (C) 2000 - 2025, Intel Corp.
+ * Copyright (C) 2000 - 2026, Intel Corp.
*
*****************************************************************************/
diff --git a/drivers/acpi/acpica/evrgnini.c b/drivers/acpi/acpica/evrgnini.c
index b039527..4e64e58 100644
--- a/drivers/acpi/acpica/evrgnini.c
+++ b/drivers/acpi/acpica/evrgnini.c
@@ -3,7 +3,7 @@
*
* Module Name: evrgnini- ACPI address_space (op_region) init
*
- * Copyright (C) 2000 - 2025, Intel Corp.
+ * Copyright (C) 2000 - 2026, Intel Corp.
*
*****************************************************************************/
diff --git a/drivers/acpi/acpica/evxface.c b/drivers/acpi/acpica/evxface.c
index 86a8d41..da4e45f 100644
--- a/drivers/acpi/acpica/evxface.c
+++ b/drivers/acpi/acpica/evxface.c
@@ -3,7 +3,7 @@
*
* Module Name: evxface - External interfaces for ACPI events
*
- * Copyright (C) 2000 - 2025, Intel Corp.
+ * Copyright (C) 2000 - 2026, Intel Corp.
*
*****************************************************************************/
diff --git a/drivers/acpi/acpica/evxfevnt.c b/drivers/acpi/acpica/evxfevnt.c
index 4b05290..1819737 100644
--- a/drivers/acpi/acpica/evxfevnt.c
+++ b/drivers/acpi/acpica/evxfevnt.c
@@ -3,7 +3,7 @@
*
* Module Name: evxfevnt - External Interfaces, ACPI event disable/enable
*
- * Copyright (C) 2000 - 2025, Intel Corp.
+ * Copyright (C) 2000 - 2026, Intel Corp.
*
*****************************************************************************/
diff --git a/drivers/acpi/acpica/evxfgpe.c b/drivers/acpi/acpica/evxfgpe.c
index 4074b59..4be8f6e 100644
--- a/drivers/acpi/acpica/evxfgpe.c
+++ b/drivers/acpi/acpica/evxfgpe.c
@@ -3,7 +3,7 @@
*
* Module Name: evxfgpe - External Interfaces for General Purpose Events (GPEs)
*
- * Copyright (C) 2000 - 2025, Intel Corp.
+ * Copyright (C) 2000 - 2026, Intel Corp.
*
*****************************************************************************/
diff --git a/drivers/acpi/acpica/evxfregn.c b/drivers/acpi/acpica/evxfregn.c
index bccc672..177f80d 100644
--- a/drivers/acpi/acpica/evxfregn.c
+++ b/drivers/acpi/acpica/evxfregn.c
@@ -4,7 +4,7 @@
* Module Name: evxfregn - External Interfaces, ACPI Operation Regions and
* Address Spaces.
*
- * Copyright (C) 2000 - 2025, Intel Corp.
+ * Copyright (C) 2000 - 2026, Intel Corp.
*
*****************************************************************************/
diff --git a/drivers/acpi/acpica/exconcat.c b/drivers/acpi/acpica/exconcat.c
index c248c9b..107df9d 100644
--- a/drivers/acpi/acpica/exconcat.c
+++ b/drivers/acpi/acpica/exconcat.c
@@ -3,7 +3,7 @@
*
* Module Name: exconcat - Concatenate-type AML operators
*
- * Copyright (C) 2000 - 2025, Intel Corp.
+ * Copyright (C) 2000 - 2026, Intel Corp.
*
*****************************************************************************/
diff --git a/drivers/acpi/acpica/exconfig.c b/drivers/acpi/acpica/exconfig.c
index 4d7dd0f..da39d57 100644
--- a/drivers/acpi/acpica/exconfig.c
+++ b/drivers/acpi/acpica/exconfig.c
@@ -3,7 +3,7 @@
*
* Module Name: exconfig - Namespace reconfiguration (Load/Unload opcodes)
*
- * Copyright (C) 2000 - 2025, Intel Corp.
+ * Copyright (C) 2000 - 2026, Intel Corp.
*
*****************************************************************************/
@@ -90,6 +90,8 @@ acpi_ex_load_table_op(struct acpi_walk_state *walk_state,
union acpi_operand_object *return_obj;
union acpi_operand_object *ddb_handle;
u32 table_index;
+ char oem_id[ACPI_OEM_ID_SIZE + 1];
+ char oem_table_id[ACPI_OEM_TABLE_ID_SIZE + 1];
ACPI_FUNCTION_TRACE(ex_load_table_op);
@@ -102,12 +104,32 @@ acpi_ex_load_table_op(struct acpi_walk_state *walk_state,
*return_desc = return_obj;
+ /*
+ * Validate OEM ID and OEM Table ID string lengths.
+ * acpi_tb_find_table expects strings that can safely read
+ * ACPI_OEM_ID_SIZE and ACPI_OEM_TABLE_ID_SIZE bytes.
+ */
+ if ((operand[1]->string.length > ACPI_OEM_ID_SIZE) ||
+ (operand[2]->string.length > ACPI_OEM_TABLE_ID_SIZE)) {
+ return_ACPI_STATUS(AE_AML_STRING_LIMIT);
+ }
+
+ /*
+ * Copy OEM strings to local buffers with guaranteed null-termination.
+ * This prevents heap-buffer-overflow when acpi_tb_find_table reads
+ * ACPI_OEM_ID_SIZE/ACPI_OEM_TABLE_ID_SIZE bytes.
+ */
+ memcpy(oem_id, operand[1]->string.pointer, operand[1]->string.length);
+ oem_id[operand[1]->string.length] = 0;
+ memcpy(oem_table_id, operand[2]->string.pointer,
+ operand[2]->string.length);
+ oem_table_id[operand[2]->string.length] = 0;
+
/* Find the ACPI table in the RSDT/XSDT */
acpi_ex_exit_interpreter();
status = acpi_tb_find_table(operand[0]->string.pointer,
- operand[1]->string.pointer,
- operand[2]->string.pointer, &table_index);
+ oem_id, oem_table_id, &table_index);
acpi_ex_enter_interpreter();
if (ACPI_FAILURE(status)) {
if (status != AE_NOT_FOUND) {
diff --git a/drivers/acpi/acpica/exconvrt.c b/drivers/acpi/acpica/exconvrt.c
index fded9bf..b51f20d 100644
--- a/drivers/acpi/acpica/exconvrt.c
+++ b/drivers/acpi/acpica/exconvrt.c
@@ -3,7 +3,7 @@
*
* Module Name: exconvrt - Object conversion routines
*
- * Copyright (C) 2000 - 2025, Intel Corp.
+ * Copyright (C) 2000 - 2026, Intel Corp.
*
*****************************************************************************/
diff --git a/drivers/acpi/acpica/excreate.c b/drivers/acpi/acpica/excreate.c
index 052c695..64f3927 100644
--- a/drivers/acpi/acpica/excreate.c
+++ b/drivers/acpi/acpica/excreate.c
@@ -3,7 +3,7 @@
*
* Module Name: excreate - Named object creation
*
- * Copyright (C) 2000 - 2025, Intel Corp.
+ * Copyright (C) 2000 - 2026, Intel Corp.
*
*****************************************************************************/
diff --git a/drivers/acpi/acpica/exdebug.c b/drivers/acpi/acpica/exdebug.c
index 81a07a5..a592bcc 100644
--- a/drivers/acpi/acpica/exdebug.c
+++ b/drivers/acpi/acpica/exdebug.c
@@ -3,7 +3,7 @@
*
* Module Name: exdebug - Support for stores to the AML Debug Object
*
- * Copyright (C) 2000 - 2025, Intel Corp.
+ * Copyright (C) 2000 - 2026, Intel Corp.
*
*****************************************************************************/
diff --git a/drivers/acpi/acpica/exdump.c b/drivers/acpi/acpica/exdump.c
index d8aeeba..56500b2b 100644
--- a/drivers/acpi/acpica/exdump.c
+++ b/drivers/acpi/acpica/exdump.c
@@ -3,7 +3,7 @@
*
* Module Name: exdump - Interpreter debug output routines
*
- * Copyright (C) 2000 - 2025, Intel Corp.
+ * Copyright (C) 2000 - 2026, Intel Corp.
*
*****************************************************************************/
diff --git a/drivers/acpi/acpica/exfield.c b/drivers/acpi/acpica/exfield.c
index ced3ff9..9a55524 100644
--- a/drivers/acpi/acpica/exfield.c
+++ b/drivers/acpi/acpica/exfield.c
@@ -3,7 +3,7 @@
*
* Module Name: exfield - AML execution - field_unit read/write
*
- * Copyright (C) 2000 - 2025, Intel Corp.
+ * Copyright (C) 2000 - 2026, Intel Corp.
*
*****************************************************************************/
diff --git a/drivers/acpi/acpica/exfldio.c b/drivers/acpi/acpica/exfldio.c
index 0771934..bdd8e6e 100644
--- a/drivers/acpi/acpica/exfldio.c
+++ b/drivers/acpi/acpica/exfldio.c
@@ -3,7 +3,7 @@
*
* Module Name: exfldio - Aml Field I/O
*
- * Copyright (C) 2000 - 2025, Intel Corp.
+ * Copyright (C) 2000 - 2026, Intel Corp.
*
*****************************************************************************/
diff --git a/drivers/acpi/acpica/exmisc.c b/drivers/acpi/acpica/exmisc.c
index 07cbac5..e67d3d5 100644
--- a/drivers/acpi/acpica/exmisc.c
+++ b/drivers/acpi/acpica/exmisc.c
@@ -3,7 +3,7 @@
*
* Module Name: exmisc - ACPI AML (p-code) execution - specific opcodes
*
- * Copyright (C) 2000 - 2025, Intel Corp.
+ * Copyright (C) 2000 - 2026, Intel Corp.
*
*****************************************************************************/
diff --git a/drivers/acpi/acpica/exmutex.c b/drivers/acpi/acpica/exmutex.c
index 1fa01319..cc0f997 100644
--- a/drivers/acpi/acpica/exmutex.c
+++ b/drivers/acpi/acpica/exmutex.c
@@ -3,7 +3,7 @@
*
* Module Name: exmutex - ASL Mutex Acquire/Release functions
*
- * Copyright (C) 2000 - 2025, Intel Corp.
+ * Copyright (C) 2000 - 2026, Intel Corp.
*
*****************************************************************************/
diff --git a/drivers/acpi/acpica/exnames.c b/drivers/acpi/acpica/exnames.c
index 76ab73c..3ae2bd8 100644
--- a/drivers/acpi/acpica/exnames.c
+++ b/drivers/acpi/acpica/exnames.c
@@ -3,7 +3,7 @@
*
* Module Name: exnames - interpreter/scanner name load/execute
*
- * Copyright (C) 2000 - 2025, Intel Corp.
+ * Copyright (C) 2000 - 2026, Intel Corp.
*
*****************************************************************************/
diff --git a/drivers/acpi/acpica/exoparg1.c b/drivers/acpi/acpica/exoparg1.c
index 6ac7e0c..7a6e4c6 100644
--- a/drivers/acpi/acpica/exoparg1.c
+++ b/drivers/acpi/acpica/exoparg1.c
@@ -3,7 +3,7 @@
*
* Module Name: exoparg1 - AML execution - opcodes with 1 argument
*
- * Copyright (C) 2000 - 2025, Intel Corp.
+ * Copyright (C) 2000 - 2026, Intel Corp.
*
*****************************************************************************/
diff --git a/drivers/acpi/acpica/exoparg2.c b/drivers/acpi/acpica/exoparg2.c
index a94fa4d..ca43447 100644
--- a/drivers/acpi/acpica/exoparg2.c
+++ b/drivers/acpi/acpica/exoparg2.c
@@ -3,7 +3,7 @@
*
* Module Name: exoparg2 - AML execution - opcodes with 2 arguments
*
- * Copyright (C) 2000 - 2025, Intel Corp.
+ * Copyright (C) 2000 - 2026, Intel Corp.
*
*****************************************************************************/
diff --git a/drivers/acpi/acpica/exoparg3.c b/drivers/acpi/acpica/exoparg3.c
index 2fc80708..3a0559a 100644
--- a/drivers/acpi/acpica/exoparg3.c
+++ b/drivers/acpi/acpica/exoparg3.c
@@ -3,7 +3,7 @@
*
* Module Name: exoparg3 - AML execution - opcodes with 3 arguments
*
- * Copyright (C) 2000 - 2025, Intel Corp.
+ * Copyright (C) 2000 - 2026, Intel Corp.
*
*****************************************************************************/
@@ -159,7 +159,7 @@ acpi_status acpi_ex_opcode_3A_1T_1R(struct acpi_walk_state *walk_state)
/* Truncate request if larger than the actual String/Buffer */
- else if ((index + length) > operand[0]->string.length) {
+ else if ((index + length) > operand[0]->string.length || (index + length) < index) { /* Check for overflow */
length =
(acpi_size)operand[0]->string.length -
(acpi_size)index;
diff --git a/drivers/acpi/acpica/exoparg6.c b/drivers/acpi/acpica/exoparg6.c
index cb078e3..ab502bf 100644
--- a/drivers/acpi/acpica/exoparg6.c
+++ b/drivers/acpi/acpica/exoparg6.c
@@ -3,7 +3,7 @@
*
* Module Name: exoparg6 - AML execution - opcodes with 6 arguments
*
- * Copyright (C) 2000 - 2025, Intel Corp.
+ * Copyright (C) 2000 - 2026, Intel Corp.
*
*****************************************************************************/
diff --git a/drivers/acpi/acpica/exprep.c b/drivers/acpi/acpica/exprep.c
index 1b1a006..3acfa60 100644
--- a/drivers/acpi/acpica/exprep.c
+++ b/drivers/acpi/acpica/exprep.c
@@ -3,7 +3,7 @@
*
* Module Name: exprep - ACPI AML field prep utilities
*
- * Copyright (C) 2000 - 2025, Intel Corp.
+ * Copyright (C) 2000 - 2026, Intel Corp.
*
*****************************************************************************/
diff --git a/drivers/acpi/acpica/exregion.c b/drivers/acpi/acpica/exregion.c
index a390a1c..fc14491 100644
--- a/drivers/acpi/acpica/exregion.c
+++ b/drivers/acpi/acpica/exregion.c
@@ -3,7 +3,7 @@
*
* Module Name: exregion - ACPI default op_region (address space) handlers
*
- * Copyright (C) 2000 - 2025, Intel Corp.
+ * Copyright (C) 2000 - 2026, Intel Corp.
*
*****************************************************************************/
diff --git a/drivers/acpi/acpica/exresnte.c b/drivers/acpi/acpica/exresnte.c
index dd83631..1c6c3d5 100644
--- a/drivers/acpi/acpica/exresnte.c
+++ b/drivers/acpi/acpica/exresnte.c
@@ -3,7 +3,7 @@
*
* Module Name: exresnte - AML Interpreter object resolution
*
- * Copyright (C) 2000 - 2025, Intel Corp.
+ * Copyright (C) 2000 - 2026, Intel Corp.
*
*****************************************************************************/
diff --git a/drivers/acpi/acpica/exresolv.c b/drivers/acpi/acpica/exresolv.c
index 4589de3..029918333 100644
--- a/drivers/acpi/acpica/exresolv.c
+++ b/drivers/acpi/acpica/exresolv.c
@@ -3,7 +3,7 @@
*
* Module Name: exresolv - AML Interpreter object resolution
*
- * Copyright (C) 2000 - 2025, Intel Corp.
+ * Copyright (C) 2000 - 2026, Intel Corp.
*
*****************************************************************************/
diff --git a/drivers/acpi/acpica/exresop.c b/drivers/acpi/acpica/exresop.c
index 782ee35..8127d40 100644
--- a/drivers/acpi/acpica/exresop.c
+++ b/drivers/acpi/acpica/exresop.c
@@ -3,7 +3,7 @@
*
* Module Name: exresop - AML Interpreter operand/object resolution
*
- * Copyright (C) 2000 - 2025, Intel Corp.
+ * Copyright (C) 2000 - 2026, Intel Corp.
*
*****************************************************************************/
diff --git a/drivers/acpi/acpica/exserial.c b/drivers/acpi/acpica/exserial.c
index 6d2581e..835ea4b 100644
--- a/drivers/acpi/acpica/exserial.c
+++ b/drivers/acpi/acpica/exserial.c
@@ -3,7 +3,7 @@
*
* Module Name: exserial - field_unit support for serial address spaces
*
- * Copyright (C) 2000 - 2025, Intel Corp.
+ * Copyright (C) 2000 - 2026, Intel Corp.
*
*****************************************************************************/
diff --git a/drivers/acpi/acpica/exstore.c b/drivers/acpi/acpica/exstore.c
index cbc42207..86be6bc 100644
--- a/drivers/acpi/acpica/exstore.c
+++ b/drivers/acpi/acpica/exstore.c
@@ -3,7 +3,7 @@
*
* Module Name: exstore - AML Interpreter object store support
*
- * Copyright (C) 2000 - 2025, Intel Corp.
+ * Copyright (C) 2000 - 2026, Intel Corp.
*
*****************************************************************************/
diff --git a/drivers/acpi/acpica/exstoren.c b/drivers/acpi/acpica/exstoren.c
index 0470b26..a047f73 100644
--- a/drivers/acpi/acpica/exstoren.c
+++ b/drivers/acpi/acpica/exstoren.c
@@ -4,7 +4,7 @@
* Module Name: exstoren - AML Interpreter object store support,
* Store to Node (namespace object)
*
- * Copyright (C) 2000 - 2025, Intel Corp.
+ * Copyright (C) 2000 - 2026, Intel Corp.
*
*****************************************************************************/
diff --git a/drivers/acpi/acpica/exstorob.c b/drivers/acpi/acpica/exstorob.c
index 5b168fb..8d17412 100644
--- a/drivers/acpi/acpica/exstorob.c
+++ b/drivers/acpi/acpica/exstorob.c
@@ -3,7 +3,7 @@
*
* Module Name: exstorob - AML object store support, store to object
*
- * Copyright (C) 2000 - 2025, Intel Corp.
+ * Copyright (C) 2000 - 2026, Intel Corp.
*
*****************************************************************************/
diff --git a/drivers/acpi/acpica/exsystem.c b/drivers/acpi/acpica/exsystem.c
index 7f843c9..41d6f8f 100644
--- a/drivers/acpi/acpica/exsystem.c
+++ b/drivers/acpi/acpica/exsystem.c
@@ -3,7 +3,7 @@
*
* Module Name: exsystem - Interface to OS services
*
- * Copyright (C) 2000 - 2025, Intel Corp.
+ * Copyright (C) 2000 - 2026, Intel Corp.
*
*****************************************************************************/
diff --git a/drivers/acpi/acpica/extrace.c b/drivers/acpi/acpica/extrace.c
index 36934d4..4a6c7fb 100644
--- a/drivers/acpi/acpica/extrace.c
+++ b/drivers/acpi/acpica/extrace.c
@@ -3,7 +3,7 @@
*
* Module Name: extrace - Support for interpreter execution tracing
*
- * Copyright (C) 2000 - 2025, Intel Corp.
+ * Copyright (C) 2000 - 2026, Intel Corp.
*
*****************************************************************************/
diff --git a/drivers/acpi/acpica/exutils.c b/drivers/acpi/acpica/exutils.c
index cc10c07..a1aa89a 100644
--- a/drivers/acpi/acpica/exutils.c
+++ b/drivers/acpi/acpica/exutils.c
@@ -3,7 +3,7 @@
*
* Module Name: exutils - interpreter/scanner utilities
*
- * Copyright (C) 2000 - 2025, Intel Corp.
+ * Copyright (C) 2000 - 2026, Intel Corp.
*
*****************************************************************************/
diff --git a/drivers/acpi/acpica/hwacpi.c b/drivers/acpi/acpica/hwacpi.c
index a1e1fa7..e70f603 100644
--- a/drivers/acpi/acpica/hwacpi.c
+++ b/drivers/acpi/acpica/hwacpi.c
@@ -3,7 +3,7 @@
*
* Module Name: hwacpi - ACPI Hardware Initialization/Mode Interface
*
- * Copyright (C) 2000 - 2025, Intel Corp.
+ * Copyright (C) 2000 - 2026, Intel Corp.
*
*****************************************************************************/
diff --git a/drivers/acpi/acpica/hwesleep.c b/drivers/acpi/acpica/hwesleep.c
index 631fd8e..73f60d7 100644
--- a/drivers/acpi/acpica/hwesleep.c
+++ b/drivers/acpi/acpica/hwesleep.c
@@ -4,7 +4,7 @@
* Name: hwesleep.c - ACPI Hardware Sleep/Wake Support functions for the
* extended FADT-V5 sleep registers.
*
- * Copyright (C) 2000 - 2025, Intel Corp.
+ * Copyright (C) 2000 - 2026, Intel Corp.
*
*****************************************************************************/
diff --git a/drivers/acpi/acpica/hwgpe.c b/drivers/acpi/acpica/hwgpe.c
index 386f475..98d3662 100644
--- a/drivers/acpi/acpica/hwgpe.c
+++ b/drivers/acpi/acpica/hwgpe.c
@@ -3,7 +3,7 @@
*
* Module Name: hwgpe - Low level GPE enable/disable/clear functions
*
- * Copyright (C) 2000 - 2025, Intel Corp.
+ * Copyright (C) 2000 - 2026, Intel Corp.
*
*****************************************************************************/
diff --git a/drivers/acpi/acpica/hwsleep.c b/drivers/acpi/acpica/hwsleep.c
index 87d78be..d0d4676 100644
--- a/drivers/acpi/acpica/hwsleep.c
+++ b/drivers/acpi/acpica/hwsleep.c
@@ -4,7 +4,7 @@
* Name: hwsleep.c - ACPI Hardware Sleep/Wake Support functions for the
* original/legacy sleep/PM registers.
*
- * Copyright (C) 2000 - 2025, Intel Corp.
+ * Copyright (C) 2000 - 2026, Intel Corp.
*
*****************************************************************************/
diff --git a/drivers/acpi/acpica/hwtimer.c b/drivers/acpi/acpica/hwtimer.c
index a5e0bcc..c08bbaa 100644
--- a/drivers/acpi/acpica/hwtimer.c
+++ b/drivers/acpi/acpica/hwtimer.c
@@ -3,7 +3,7 @@
*
* Name: hwtimer.c - ACPI Power Management Timer Interface
*
- * Copyright (C) 2000 - 2025, Intel Corp.
+ * Copyright (C) 2000 - 2026, Intel Corp.
*
*****************************************************************************/
diff --git a/drivers/acpi/acpica/hwvalid.c b/drivers/acpi/acpica/hwvalid.c
index 496fd9e..abbe304 100644
--- a/drivers/acpi/acpica/hwvalid.c
+++ b/drivers/acpi/acpica/hwvalid.c
@@ -3,7 +3,7 @@
*
* Module Name: hwvalid - I/O request validation
*
- * Copyright (C) 2000 - 2025, Intel Corp.
+ * Copyright (C) 2000 - 2026, Intel Corp.
*
*****************************************************************************/
diff --git a/drivers/acpi/acpica/hwxface.c b/drivers/acpi/acpica/hwxface.c
index 847cd1b..980ea76 100644
--- a/drivers/acpi/acpica/hwxface.c
+++ b/drivers/acpi/acpica/hwxface.c
@@ -3,7 +3,7 @@
*
* Module Name: hwxface - Public ACPICA hardware interfaces
*
- * Copyright (C) 2000 - 2025, Intel Corp.
+ * Copyright (C) 2000 - 2026, Intel Corp.
*
*****************************************************************************/
diff --git a/drivers/acpi/acpica/hwxfsleep.c b/drivers/acpi/acpica/hwxfsleep.c
index 9aabe30..dd70fcb 100644
--- a/drivers/acpi/acpica/hwxfsleep.c
+++ b/drivers/acpi/acpica/hwxfsleep.c
@@ -3,7 +3,7 @@
*
* Name: hwxfsleep.c - ACPI Hardware Sleep/Wake External Interfaces
*
- * Copyright (C) 2000 - 2025, Intel Corp.
+ * Copyright (C) 2000 - 2026, Intel Corp.
*
*****************************************************************************/
diff --git a/drivers/acpi/acpica/nsarguments.c b/drivers/acpi/acpica/nsarguments.c
index 366d54a..e00cce2 100644
--- a/drivers/acpi/acpica/nsarguments.c
+++ b/drivers/acpi/acpica/nsarguments.c
@@ -3,7 +3,7 @@
*
* Module Name: nsarguments - Validation of args for ACPI predefined methods
*
- * Copyright (C) 2000 - 2025, Intel Corp.
+ * Copyright (C) 2000 - 2026, Intel Corp.
*
*****************************************************************************/
diff --git a/drivers/acpi/acpica/nsconvert.c b/drivers/acpi/acpica/nsconvert.c
index f05a92b..b903f70e 100644
--- a/drivers/acpi/acpica/nsconvert.c
+++ b/drivers/acpi/acpica/nsconvert.c
@@ -4,7 +4,7 @@
* Module Name: nsconvert - Object conversions for objects returned by
* predefined methods
*
- * Copyright (C) 2000 - 2025, Intel Corp.
+ * Copyright (C) 2000 - 2026, Intel Corp.
*
*****************************************************************************/
diff --git a/drivers/acpi/acpica/nsdump.c b/drivers/acpi/acpica/nsdump.c
index 6dc2048..15bb57e 100644
--- a/drivers/acpi/acpica/nsdump.c
+++ b/drivers/acpi/acpica/nsdump.c
@@ -3,7 +3,7 @@
*
* Module Name: nsdump - table dumping routines for debug
*
- * Copyright (C) 2000 - 2025, Intel Corp.
+ * Copyright (C) 2000 - 2026, Intel Corp.
*
*****************************************************************************/
diff --git a/drivers/acpi/acpica/nsdumpdv.c b/drivers/acpi/acpica/nsdumpdv.c
index d5b16aa..7cbb078 100644
--- a/drivers/acpi/acpica/nsdumpdv.c
+++ b/drivers/acpi/acpica/nsdumpdv.c
@@ -3,7 +3,7 @@
*
* Module Name: nsdump - table dumping routines for debug
*
- * Copyright (C) 2000 - 2025, Intel Corp.
+ * Copyright (C) 2000 - 2026, Intel Corp.
*
*****************************************************************************/
diff --git a/drivers/acpi/acpica/nsinit.c b/drivers/acpi/acpica/nsinit.c
index 03373e7..70453eb 100644
--- a/drivers/acpi/acpica/nsinit.c
+++ b/drivers/acpi/acpica/nsinit.c
@@ -3,7 +3,7 @@
*
* Module Name: nsinit - namespace initialization
*
- * Copyright (C) 2000 - 2025, Intel Corp.
+ * Copyright (C) 2000 - 2026, Intel Corp.
*
*****************************************************************************/
diff --git a/drivers/acpi/acpica/nsload.c b/drivers/acpi/acpica/nsload.c
index 6ec4c64..e89998d 100644
--- a/drivers/acpi/acpica/nsload.c
+++ b/drivers/acpi/acpica/nsload.c
@@ -3,7 +3,7 @@
*
* Module Name: nsload - namespace loading/expanding/contracting procedures
*
- * Copyright (C) 2000 - 2025, Intel Corp.
+ * Copyright (C) 2000 - 2026, Intel Corp.
*
*****************************************************************************/
diff --git a/drivers/acpi/acpica/nsnames.c b/drivers/acpi/acpica/nsnames.c
index 22aeeeb..19802da 100644
--- a/drivers/acpi/acpica/nsnames.c
+++ b/drivers/acpi/acpica/nsnames.c
@@ -222,6 +222,12 @@ acpi_ns_build_normalized_path(struct acpi_namespace_node *node,
goto build_trailing_null;
}
+ /* Validate the Node to avoid use-after-free vulnerabilities */
+
+ if (ACPI_GET_DESCRIPTOR_TYPE(node) != ACPI_DESC_TYPE_NAMED) {
+ goto build_trailing_null;
+ }
+
next_node = node;
while (next_node && next_node != acpi_gbl_root_node) {
if (next_node != node) {
diff --git a/drivers/acpi/acpica/nsobject.c b/drivers/acpi/acpica/nsobject.c
index 79d86da..a4ccace 100644
--- a/drivers/acpi/acpica/nsobject.c
+++ b/drivers/acpi/acpica/nsobject.c
@@ -173,6 +173,12 @@ void acpi_ns_detach_object(struct acpi_namespace_node *node)
obj_desc = node->object;
+ /* Alias nodes point directly to other namespace nodes; skip teardown */
+ if (node->flags & ANOBJ_IS_ALIAS) {
+ node->object = NULL;
+ return_VOID;
+ }
+
if (!obj_desc || (obj_desc->common.type == ACPI_TYPE_LOCAL_DATA)) {
return_VOID;
}
diff --git a/drivers/acpi/acpica/nsparse.c b/drivers/acpi/acpica/nsparse.c
index 959e637..1e9c70c 100644
--- a/drivers/acpi/acpica/nsparse.c
+++ b/drivers/acpi/acpica/nsparse.c
@@ -3,7 +3,7 @@
*
* Module Name: nsparse - namespace interface to AML parser
*
- * Copyright (C) 2000 - 2025, Intel Corp.
+ * Copyright (C) 2000 - 2026, Intel Corp.
*
*****************************************************************************/
diff --git a/drivers/acpi/acpica/nspredef.c b/drivers/acpi/acpica/nspredef.c
index 81995ee4..d7b4f8d 100644
--- a/drivers/acpi/acpica/nspredef.c
+++ b/drivers/acpi/acpica/nspredef.c
@@ -3,7 +3,7 @@
*
* Module Name: nspredef - Validation of ACPI predefined methods and objects
*
- * Copyright (C) 2000 - 2025, Intel Corp.
+ * Copyright (C) 2000 - 2026, Intel Corp.
*
*****************************************************************************/
diff --git a/drivers/acpi/acpica/nsprepkg.c b/drivers/acpi/acpica/nsprepkg.c
index ca137ce..f1c510c 100644
--- a/drivers/acpi/acpica/nsprepkg.c
+++ b/drivers/acpi/acpica/nsprepkg.c
@@ -3,7 +3,7 @@
*
* Module Name: nsprepkg - Validation of package objects for predefined names
*
- * Copyright (C) 2000 - 2025, Intel Corp.
+ * Copyright (C) 2000 - 2026, Intel Corp.
*
*****************************************************************************/
@@ -631,6 +631,13 @@ acpi_ns_custom_package(struct acpi_evaluate_info *info,
/* Get version number, must be Integer */
+ if (!(*elements)) {
+ ACPI_WARN_PREDEFINED((AE_INFO, info->full_pathname,
+ info->node_flags,
+ "Return Package has a NULL version element"));
+ return_ACPI_STATUS(AE_AML_OPERAND_TYPE);
+ }
+
if ((*elements)->common.type != ACPI_TYPE_INTEGER) {
ACPI_WARN_PREDEFINED((AE_INFO, info->full_pathname,
info->node_flags,
diff --git a/drivers/acpi/acpica/nsrepair.c b/drivers/acpi/acpica/nsrepair.c
index accfdcf..8e5da0c 100644
--- a/drivers/acpi/acpica/nsrepair.c
+++ b/drivers/acpi/acpica/nsrepair.c
@@ -3,7 +3,7 @@
*
* Module Name: nsrepair - Repair for objects returned by predefined methods
*
- * Copyright (C) 2000 - 2025, Intel Corp.
+ * Copyright (C) 2000 - 2026, Intel Corp.
*
*****************************************************************************/
diff --git a/drivers/acpi/acpica/nsrepair2.c b/drivers/acpi/acpica/nsrepair2.c
index 8dbb870..62734b9 100644
--- a/drivers/acpi/acpica/nsrepair2.c
+++ b/drivers/acpi/acpica/nsrepair2.c
@@ -4,7 +4,7 @@
* Module Name: nsrepair2 - Repair for objects returned by specific
* predefined methods
*
- * Copyright (C) 2000 - 2025, Intel Corp.
+ * Copyright (C) 2000 - 2026, Intel Corp.
*
*****************************************************************************/
diff --git a/drivers/acpi/acpica/nsutils.c b/drivers/acpi/acpica/nsutils.c
index 49cc07e2..65b517f 100644
--- a/drivers/acpi/acpica/nsutils.c
+++ b/drivers/acpi/acpica/nsutils.c
@@ -4,7 +4,7 @@
* Module Name: nsutils - Utilities for accessing ACPI namespace, accessing
* parents and siblings and Scope manipulation
*
- * Copyright (C) 2000 - 2025, Intel Corp.
+ * Copyright (C) 2000 - 2026, Intel Corp.
*
*****************************************************************************/
diff --git a/drivers/acpi/acpica/nswalk.c b/drivers/acpi/acpica/nswalk.c
index 5670ff5..a37d75b 100644
--- a/drivers/acpi/acpica/nswalk.c
+++ b/drivers/acpi/acpica/nswalk.c
@@ -3,7 +3,7 @@
*
* Module Name: nswalk - Functions for walking the ACPI namespace
*
- * Copyright (C) 2000 - 2025, Intel Corp.
+ * Copyright (C) 2000 - 2026, Intel Corp.
*
*****************************************************************************/
diff --git a/drivers/acpi/acpica/nsxfname.c b/drivers/acpi/acpica/nsxfname.c
index b6895a4..b653418 100644
--- a/drivers/acpi/acpica/nsxfname.c
+++ b/drivers/acpi/acpica/nsxfname.c
@@ -4,7 +4,7 @@
* Module Name: nsxfname - Public interfaces to the ACPI subsystem
* ACPI Namespace oriented interfaces
*
- * Copyright (C) 2000 - 2025, Intel Corp.
+ * Copyright (C) 2000 - 2026, Intel Corp.
*
*****************************************************************************/
@@ -512,6 +512,10 @@ acpi_status acpi_install_method(u8 *buffer)
parser_state.aml += acpi_ps_get_opcode_size(opcode);
parser_state.pkg_end = acpi_ps_get_next_package_end(&parser_state);
+ if ((parser_state.pkg_end > parser_state.aml_end) ||
+ (parser_state.pkg_end < parser_state.aml)) {
+ return (AE_AML_PACKAGE_LIMIT);
+ }
path = acpi_ps_get_next_namestring(&parser_state);
method_flags = *parser_state.aml++;
diff --git a/drivers/acpi/acpica/psargs.c b/drivers/acpi/acpica/psargs.c
index 6f6ae38..4643c83 100644
--- a/drivers/acpi/acpica/psargs.c
+++ b/drivers/acpi/acpica/psargs.c
@@ -3,7 +3,7 @@
*
* Module Name: psargs - Parse AML opcode arguments
*
- * Copyright (C) 2000 - 2025, Intel Corp.
+ * Copyright (C) 2000 - 2026, Intel Corp.
*
*****************************************************************************/
@@ -48,6 +48,7 @@ acpi_ps_get_next_package_length(struct acpi_parse_state *parser_state)
u32 package_length = 0;
u32 byte_count;
u8 byte_zero_mask = 0x3F; /* Default [0:5] */
+ u32 remaining;
ACPI_FUNCTION_TRACE(ps_get_next_package_length);
@@ -55,7 +56,23 @@ acpi_ps_get_next_package_length(struct acpi_parse_state *parser_state)
* Byte 0 bits [6:7] contain the number of additional bytes
* used to encode the package length, either 0,1,2, or 3
*/
+
+ /* Check if we have at least one byte to read */
+ remaining = (u32)ACPI_PTR_DIFF(parser_state->aml_end, aml);
+ if (remaining == 0) {
+ return_UINT32(0);
+ }
+
byte_count = (aml[0] >> 6);
+
+ /* Validate byte_count and ensure we have enough bytes to read */
+ if (byte_count >= remaining) {
+
+ /* Clamp to available bytes and advance to end */
+ parser_state->aml = parser_state->aml_end;
+ return_UINT32(0);
+ }
+
parser_state->aml += ((acpi_size)byte_count + 1);
/* Get bytes 3, 2, 1 as needed */
@@ -131,10 +148,16 @@ char *acpi_ps_get_next_namestring(struct acpi_parse_state *parser_state)
/* Point past any namestring prefix characters (backslash or carat) */
- while (ACPI_IS_ROOT_PREFIX(*end) || ACPI_IS_PARENT_PREFIX(*end)) {
+ while (end < parser_state->aml_end &&
+ (ACPI_IS_ROOT_PREFIX(*end) || ACPI_IS_PARENT_PREFIX(*end))) {
end++;
}
+ if (end >= parser_state->aml_end) {
+ parser_state->aml = parser_state->aml_end;
+ return_PTR(NULL);
+ }
+
/* Decode the path prefix character */
switch (*end) {
@@ -159,6 +182,11 @@ char *acpi_ps_get_next_namestring(struct acpi_parse_state *parser_state)
/* Multiple name segments, 4 chars each, count in next byte */
+ if ((end + 1) >= parser_state->aml_end) {
+ parser_state->aml = parser_state->aml_end;
+ return_PTR(NULL);
+ }
+
end += 2 + (*(end + 1) * ACPI_NAMESEG_SIZE);
break;
@@ -170,6 +198,11 @@ char *acpi_ps_get_next_namestring(struct acpi_parse_state *parser_state)
break;
}
+ if (end > parser_state->aml_end) {
+ parser_state->aml = parser_state->aml_end;
+ return_PTR(NULL);
+ }
+
parser_state->aml = end;
return_PTR((char *)start);
}
@@ -367,6 +400,8 @@ acpi_ps_get_next_simple_arg(struct acpi_parse_state *parser_state,
u32 length;
u16 opcode;
u8 *aml = parser_state->aml;
+ u32 remaining = (u32)ACPI_PTR_DIFF(parser_state->aml_end, aml);
+ u64 partial_value;
ACPI_FUNCTION_TRACE_U32(ps_get_next_simple_arg, arg_type);
@@ -376,8 +411,13 @@ acpi_ps_get_next_simple_arg(struct acpi_parse_state *parser_state,
/* Get 1 byte from the AML stream */
opcode = AML_BYTE_OP;
- arg->common.value.integer = (u64) *aml;
- length = 1;
+ if (remaining >= 1) {
+ arg->common.value.integer = (u64)*aml;
+ length = 1;
+ } else {
+ arg->common.value.integer = 0;
+ length = 0;
+ }
break;
case ARGP_WORDDATA:
@@ -385,8 +425,19 @@ acpi_ps_get_next_simple_arg(struct acpi_parse_state *parser_state,
/* Get 2 bytes from the AML stream */
opcode = AML_WORD_OP;
- ACPI_MOVE_16_TO_64(&arg->common.value.integer, aml);
- length = 2;
+ if (remaining >= 2) {
+ ACPI_MOVE_16_TO_64(&arg->common.value.integer, aml);
+ length = 2;
+ } else {
+ arg->common.value.integer = 0;
+ length = 0;
+ if (remaining > 0) {
+ partial_value = 0;
+ memcpy(&partial_value, aml, remaining);
+ arg->common.value.integer = partial_value;
+ length = remaining;
+ }
+ }
break;
case ARGP_DWORDDATA:
@@ -394,8 +445,19 @@ acpi_ps_get_next_simple_arg(struct acpi_parse_state *parser_state,
/* Get 4 bytes from the AML stream */
opcode = AML_DWORD_OP;
- ACPI_MOVE_32_TO_64(&arg->common.value.integer, aml);
- length = 4;
+ if (remaining >= 4) {
+ ACPI_MOVE_32_TO_64(&arg->common.value.integer, aml);
+ length = 4;
+ } else {
+ arg->common.value.integer = 0;
+ length = 0;
+ if (remaining > 0) {
+ partial_value = 0;
+ memcpy(&partial_value, aml, remaining);
+ arg->common.value.integer = partial_value;
+ length = remaining;
+ }
+ }
break;
case ARGP_QWORDDATA:
@@ -403,8 +465,19 @@ acpi_ps_get_next_simple_arg(struct acpi_parse_state *parser_state,
/* Get 8 bytes from the AML stream */
opcode = AML_QWORD_OP;
- ACPI_MOVE_64_TO_64(&arg->common.value.integer, aml);
- length = 8;
+ if (remaining >= 8) {
+ ACPI_MOVE_64_TO_64(&arg->common.value.integer, aml);
+ length = 8;
+ } else {
+ arg->common.value.integer = 0;
+ length = 0;
+ if (remaining > 0) {
+ partial_value = 0;
+ memcpy(&partial_value, aml, remaining);
+ arg->common.value.integer = partial_value;
+ length = remaining;
+ }
+ }
break;
case ARGP_CHARLIST:
@@ -417,10 +490,28 @@ acpi_ps_get_next_simple_arg(struct acpi_parse_state *parser_state,
/* Find the null terminator */
length = 0;
- while (aml[length]) {
+ while ((length < remaining) && aml[length]) {
length++;
}
- length++;
+ if (length < remaining) {
+
+ /* Account for the terminating null */
+ length++;
+ } else {
+ /*
+ * No terminator found - add null at buffer boundary
+ * and report a warning
+ */
+ ACPI_WARNING((AE_INFO,
+ "Invalid AML string: no null terminator, truncating at offset %u",
+ (u32)(aml - parser_state->aml)));
+
+ /* Add null terminator at the boundary */
+ if (remaining > 0) {
+ aml[remaining - 1] = 0;
+ length = remaining;
+ }
+ }
break;
case ARGP_NAME:
@@ -474,6 +565,10 @@ static union acpi_parse_object *acpi_ps_get_next_field(struct acpi_parse_state
ASL_CV_CAPTURE_COMMENTS_ONLY(parser_state);
aml = parser_state->aml;
+ if (aml >= parser_state->aml_end) {
+ return_PTR(NULL);
+ }
+
/* Determine field type */
switch (ACPI_GET8(parser_state->aml)) {
@@ -522,6 +617,11 @@ static union acpi_parse_object *acpi_ps_get_next_field(struct acpi_parse_state
/* Get the 4-character name */
+ if ((parser_state->aml + ACPI_NAMESEG_SIZE) >
+ parser_state->aml_end) {
+ acpi_ps_free_op(field);
+ return_PTR(NULL);
+ }
ACPI_MOVE_32_TO_32(&name, parser_state->aml);
acpi_ps_set_name(field, name);
parser_state->aml += ACPI_NAMESEG_SIZE;
@@ -567,6 +667,10 @@ static union acpi_parse_object *acpi_ps_get_next_field(struct acpi_parse_state
/* Get the two bytes (Type/Attribute) */
+ if ((parser_state->aml + 2) > parser_state->aml_end) {
+ acpi_ps_free_op(field);
+ return_PTR(NULL);
+ }
access_type = ACPI_GET8(parser_state->aml);
parser_state->aml++;
access_attribute = ACPI_GET8(parser_state->aml);
@@ -578,6 +682,10 @@ static union acpi_parse_object *acpi_ps_get_next_field(struct acpi_parse_state
/* This opcode has a third byte, access_length */
if (opcode == AML_INT_EXTACCESSFIELD_OP) {
+ if (parser_state->aml >= parser_state->aml_end) {
+ acpi_ps_free_op(field);
+ return_PTR(NULL);
+ }
access_length = ACPI_GET8(parser_state->aml);
parser_state->aml++;
@@ -775,6 +883,10 @@ acpi_ps_get_next_arg(struct acpi_walk_state *walk_state,
parser_state->pkg_end =
acpi_ps_get_next_package_end(parser_state);
+ if ((parser_state->pkg_end > parser_state->aml_end)
+ || (parser_state->pkg_end < parser_state->aml)) {
+ return_ACPI_STATUS(AE_AML_PACKAGE_LIMIT);
+ }
break;
case ARGP_FIELDLIST:
diff --git a/drivers/acpi/acpica/psloop.c b/drivers/acpi/acpica/psloop.c
index c989cad..24a57f9 100644
--- a/drivers/acpi/acpica/psloop.c
+++ b/drivers/acpi/acpica/psloop.c
@@ -3,7 +3,7 @@
*
* Module Name: psloop - Main AML parse loop
*
- * Copyright (C) 2000 - 2025, Intel Corp.
+ * Copyright (C) 2000 - 2026, Intel Corp.
*
*****************************************************************************/
@@ -361,6 +361,13 @@ acpi_status acpi_ps_parse_loop(struct acpi_walk_state *walk_state)
walk_state->parser_state.aml =
acpi_ps_get_next_package_end
(&walk_state->parser_state);
+ if ((walk_state->parser_state.aml >
+ walk_state->parser_state.aml_end)
+ || (walk_state->parser_state.aml <
+ walk_state->aml)) {
+ return_ACPI_STATUS
+ (AE_AML_PACKAGE_LIMIT);
+ }
walk_state->aml =
walk_state->parser_state.aml;
}
@@ -421,11 +428,22 @@ acpi_status acpi_ps_parse_loop(struct acpi_walk_state *walk_state)
parser_state->aml =
acpi_ps_get_next_package_end
(parser_state);
+ if ((parser_state->aml >
+ parser_state->aml_end)
+ || (parser_state->aml <
+ walk_state->control_state->
+ control.aml_predicate_start)) {
+ return_ACPI_STATUS
+ (AE_AML_PACKAGE_LIMIT);
+ }
walk_state->aml = parser_state->aml;
ACPI_ERROR((AE_INFO,
"Skipping While/If block"));
- if (*walk_state->aml == AML_ELSE_OP) {
+ if ((walk_state->aml <
+ parser_state->aml_end)
+ && (*walk_state->aml ==
+ AML_ELSE_OP)) {
ACPI_ERROR((AE_INFO,
"Skipping Else block"));
walk_state->parser_state.aml =
@@ -433,6 +451,16 @@ acpi_status acpi_ps_parse_loop(struct acpi_walk_state *walk_state)
walk_state->parser_state.aml =
acpi_ps_get_next_package_end
(parser_state);
+ if ((walk_state->parser_state.
+ aml >
+ walk_state->parser_state.
+ aml_end)
+ || (walk_state->
+ parser_state.aml <
+ walk_state->aml)) {
+ return_ACPI_STATUS
+ (AE_AML_PACKAGE_LIMIT);
+ }
walk_state->aml =
parser_state->aml;
}
diff --git a/drivers/acpi/acpica/psobject.c b/drivers/acpi/acpica/psobject.c
index 496a1c1..629cafa 100644
--- a/drivers/acpi/acpica/psobject.c
+++ b/drivers/acpi/acpica/psobject.c
@@ -3,7 +3,7 @@
*
* Module Name: psobject - Support for parse objects
*
- * Copyright (C) 2000 - 2025, Intel Corp.
+ * Copyright (C) 2000 - 2026, Intel Corp.
*
*****************************************************************************/
diff --git a/drivers/acpi/acpica/psopcode.c b/drivers/acpi/acpica/psopcode.c
index bf61039..0abb9b0 100644
--- a/drivers/acpi/acpica/psopcode.c
+++ b/drivers/acpi/acpica/psopcode.c
@@ -3,7 +3,7 @@
*
* Module Name: psopcode - Parser/Interpreter opcode information table
*
- * Copyright (C) 2000 - 2025, Intel Corp.
+ * Copyright (C) 2000 - 2026, Intel Corp.
*
*****************************************************************************/
diff --git a/drivers/acpi/acpica/psopinfo.c b/drivers/acpi/acpica/psopinfo.c
index 532ea30..479ce3d 100644
--- a/drivers/acpi/acpica/psopinfo.c
+++ b/drivers/acpi/acpica/psopinfo.c
@@ -3,7 +3,7 @@
*
* Module Name: psopinfo - AML opcode information functions and dispatch tables
*
- * Copyright (C) 2000 - 2025, Intel Corp.
+ * Copyright (C) 2000 - 2026, Intel Corp.
*
*****************************************************************************/
diff --git a/drivers/acpi/acpica/psparse.c b/drivers/acpi/acpica/psparse.c
index 55a416e..42ec8ab 100644
--- a/drivers/acpi/acpica/psparse.c
+++ b/drivers/acpi/acpica/psparse.c
@@ -3,7 +3,7 @@
*
* Module Name: psparse - Parser top level AML parse routines
*
- * Copyright (C) 2000 - 2025, Intel Corp.
+ * Copyright (C) 2000 - 2026, Intel Corp.
*
*****************************************************************************/
@@ -70,6 +70,9 @@ u16 acpi_ps_peek_opcode(struct acpi_parse_state * parser_state)
u16 opcode;
aml = parser_state->aml;
+ if (aml >= parser_state->aml_end) {
+ return (0xFFFF);
+ }
opcode = (u16) ACPI_GET8(aml);
if (opcode == AML_EXTENDED_PREFIX) {
@@ -77,6 +80,9 @@ u16 acpi_ps_peek_opcode(struct acpi_parse_state * parser_state)
/* Extended opcode, get the second opcode byte */
aml++;
+ if (aml >= parser_state->aml_end) {
+ return (0xFFFF);
+ }
opcode = (u16) ((opcode << 8) | ACPI_GET8(aml));
}
@@ -300,6 +306,7 @@ acpi_ps_next_parse_state(struct acpi_walk_state *walk_state,
{
struct acpi_parse_state *parser_state = &walk_state->parser_state;
acpi_status status = AE_CTRL_PENDING;
+ u8 *aml;
ACPI_FUNCTION_TRACE_PTR(ps_next_parse_state, op);
@@ -344,7 +351,14 @@ acpi_ps_next_parse_state(struct acpi_walk_state *walk_state,
* Predicate of an IF was true, and we are at the matching ELSE.
* Just close out this package
*/
+ aml = parser_state->aml;
+
parser_state->aml = acpi_ps_get_next_package_end(parser_state);
+ if ((parser_state->aml > parser_state->aml_end) ||
+ (parser_state->aml < aml)) {
+ status = AE_AML_PACKAGE_LIMIT;
+ break;
+ }
status = AE_CTRL_PENDING;
break;
diff --git a/drivers/acpi/acpica/psscope.c b/drivers/acpi/acpica/psscope.c
index c4e4483..822ea96 100644
--- a/drivers/acpi/acpica/psscope.c
+++ b/drivers/acpi/acpica/psscope.c
@@ -3,7 +3,7 @@
*
* Module Name: psscope - Parser scope stack management routines
*
- * Copyright (C) 2000 - 2025, Intel Corp.
+ * Copyright (C) 2000 - 2026, Intel Corp.
*
*****************************************************************************/
diff --git a/drivers/acpi/acpica/pstree.c b/drivers/acpi/acpica/pstree.c
index 5a285d3..313cbfb 100644
--- a/drivers/acpi/acpica/pstree.c
+++ b/drivers/acpi/acpica/pstree.c
@@ -3,7 +3,7 @@
*
* Module Name: pstree - Parser op tree manipulation/traversal/search
*
- * Copyright (C) 2000 - 2025, Intel Corp.
+ * Copyright (C) 2000 - 2026, Intel Corp.
*
*****************************************************************************/
diff --git a/drivers/acpi/acpica/psutils.c b/drivers/acpi/acpica/psutils.c
index ada1dc3..789d75d 100644
--- a/drivers/acpi/acpica/psutils.c
+++ b/drivers/acpi/acpica/psutils.c
@@ -3,7 +3,7 @@
*
* Module Name: psutils - Parser miscellaneous utilities (Parser only)
*
- * Copyright (C) 2000 - 2025, Intel Corp.
+ * Copyright (C) 2000 - 2026, Intel Corp.
*
*****************************************************************************/
diff --git a/drivers/acpi/acpica/pswalk.c b/drivers/acpi/acpica/pswalk.c
index 2f3ebcd..ac05210 100644
--- a/drivers/acpi/acpica/pswalk.c
+++ b/drivers/acpi/acpica/pswalk.c
@@ -3,7 +3,7 @@
*
* Module Name: pswalk - Parser routines to walk parsed op tree(s)
*
- * Copyright (C) 2000 - 2025, Intel Corp.
+ * Copyright (C) 2000 - 2026, Intel Corp.
*
*****************************************************************************/
@@ -49,8 +49,8 @@ void acpi_ps_delete_parse_tree(union acpi_parse_object *subtree_root)
/* This debug option will print the entire parse tree */
- acpi_os_printf(" %*.s%s %p", (level * 4),
- " ",
+ acpi_os_printf(" %*s%s %p", (level * 4),
+ "",
acpi_ps_get_opcode_name(op->
common.
aml_opcode),
diff --git a/drivers/acpi/acpica/psxface.c b/drivers/acpi/acpica/psxface.c
index d480de0..f8df939 100644
--- a/drivers/acpi/acpica/psxface.c
+++ b/drivers/acpi/acpica/psxface.c
@@ -3,7 +3,7 @@
*
* Module Name: psxface - Parser external interfaces
*
- * Copyright (C) 2000 - 2025, Intel Corp.
+ * Copyright (C) 2000 - 2026, Intel Corp.
*
*****************************************************************************/
diff --git a/drivers/acpi/acpica/rsserial.c b/drivers/acpi/acpica/rsserial.c
index 279bfa2..5ab41e9 100644
--- a/drivers/acpi/acpica/rsserial.c
+++ b/drivers/acpi/acpica/rsserial.c
@@ -315,7 +315,7 @@ struct acpi_rsconvert_info acpi_rs_convert_csi2_serial_bus[14] = {
*
******************************************************************************/
-struct acpi_rsconvert_info acpi_rs_convert_i2c_serial_bus[17] = {
+struct acpi_rsconvert_info acpi_rs_convert_i2c_serial_bus[18] = {
{ACPI_RSC_INITGET, ACPI_RESOURCE_TYPE_SERIAL_BUS,
ACPI_RS_SIZE(struct acpi_resource_i2c_serialbus),
ACPI_RSC_TABLE_SIZE(acpi_rs_convert_i2c_serial_bus)},
@@ -391,6 +391,11 @@ struct acpi_rsconvert_info acpi_rs_convert_i2c_serial_bus[17] = {
AML_OFFSET(i2c_serial_bus.type_specific_flags),
0},
+ /* Read LVR from Type Specific Flags, bits[15:8] */
+ {ACPI_RSC_MOVE8, ACPI_RS_OFFSET(data.i2c_serial_bus.lvr),
+ AML_OFFSET(i2c_serial_bus.type_specific_flags) + 1,
+ 1},
+
{ACPI_RSC_MOVE32, ACPI_RS_OFFSET(data.i2c_serial_bus.connection_speed),
AML_OFFSET(i2c_serial_bus.connection_speed),
1},
diff --git a/drivers/acpi/acpica/tbdata.c b/drivers/acpi/acpica/tbdata.c
index 5b98e09..07af035 100644
--- a/drivers/acpi/acpica/tbdata.c
+++ b/drivers/acpi/acpica/tbdata.c
@@ -3,7 +3,7 @@
*
* Module Name: tbdata - Table manager data structure functions
*
- * Copyright (C) 2000 - 2025, Intel Corp.
+ * Copyright (C) 2000 - 2026, Intel Corp.
*
*****************************************************************************/
diff --git a/drivers/acpi/acpica/tbfadt.c b/drivers/acpi/acpica/tbfadt.c
index c6658b2..4c3ef7c 100644
--- a/drivers/acpi/acpica/tbfadt.c
+++ b/drivers/acpi/acpica/tbfadt.c
@@ -3,7 +3,7 @@
*
* Module Name: tbfadt - FADT table utilities
*
- * Copyright (C) 2000 - 2025, Intel Corp.
+ * Copyright (C) 2000 - 2026, Intel Corp.
*
*****************************************************************************/
@@ -553,8 +553,11 @@ static void acpi_tb_convert_fadt(void)
* Note: If the legacy length field is > 0xFF bits, ignore
* this check. (GPE registers can be larger than the
* 64-bit GAS structure can accommodate, 0xFF bits).
+ * Also skip if bit_width is 0, indicating the 64-bit field
+ * was not populated - legacy length will be used instead.
*/
if ((ACPI_MUL_8(length) <= ACPI_UINT8_MAX) &&
+ (address64->bit_width != 0) &&
(address64->bit_width !=
ACPI_MUL_8(length))) {
ACPI_BIOS_WARNING((AE_INFO,
diff --git a/drivers/acpi/acpica/tbfind.c b/drivers/acpi/acpica/tbfind.c
index d71a732..60a772c 100644
--- a/drivers/acpi/acpica/tbfind.c
+++ b/drivers/acpi/acpica/tbfind.c
@@ -3,7 +3,7 @@
*
* Module Name: tbfind - find table
*
- * Copyright (C) 2000 - 2025, Intel Corp.
+ * Copyright (C) 2000 - 2026, Intel Corp.
*
*****************************************************************************/
diff --git a/drivers/acpi/acpica/tbinstal.c b/drivers/acpi/acpica/tbinstal.c
index ee9b85b..d36a803 100644
--- a/drivers/acpi/acpica/tbinstal.c
+++ b/drivers/acpi/acpica/tbinstal.c
@@ -3,7 +3,7 @@
*
* Module Name: tbinstal - ACPI table installation and removal
*
- * Copyright (C) 2000 - 2025, Intel Corp.
+ * Copyright (C) 2000 - 2026, Intel Corp.
*
*****************************************************************************/
diff --git a/drivers/acpi/acpica/tbprint.c b/drivers/acpi/acpica/tbprint.c
index e563102..acfff3c 100644
--- a/drivers/acpi/acpica/tbprint.c
+++ b/drivers/acpi/acpica/tbprint.c
@@ -3,7 +3,7 @@
*
* Module Name: tbprint - Table output utilities
*
- * Copyright (C) 2000 - 2025, Intel Corp.
+ * Copyright (C) 2000 - 2026, Intel Corp.
*
*****************************************************************************/
diff --git a/drivers/acpi/acpica/tbutils.c b/drivers/acpi/acpica/tbutils.c
index fa64851..adc6e3b 100644
--- a/drivers/acpi/acpica/tbutils.c
+++ b/drivers/acpi/acpica/tbutils.c
@@ -3,7 +3,7 @@
*
* Module Name: tbutils - ACPI Table utilities
*
- * Copyright (C) 2000 - 2025, Intel Corp.
+ * Copyright (C) 2000 - 2026, Intel Corp.
*
*****************************************************************************/
diff --git a/drivers/acpi/acpica/tbxface.c b/drivers/acpi/acpica/tbxface.c
index a8f07d2..de69ec7 100644
--- a/drivers/acpi/acpica/tbxface.c
+++ b/drivers/acpi/acpica/tbxface.c
@@ -3,7 +3,7 @@
*
* Module Name: tbxface - ACPI table-oriented external interfaces
*
- * Copyright (C) 2000 - 2025, Intel Corp.
+ * Copyright (C) 2000 - 2026, Intel Corp.
*
*****************************************************************************/
diff --git a/drivers/acpi/acpica/tbxfload.c b/drivers/acpi/acpica/tbxfload.c
index 2a17c60..c0e34b0 100644
--- a/drivers/acpi/acpica/tbxfload.c
+++ b/drivers/acpi/acpica/tbxfload.c
@@ -3,7 +3,7 @@
*
* Module Name: tbxfload - Table load/unload external interfaces
*
- * Copyright (C) 2000 - 2025, Intel Corp.
+ * Copyright (C) 2000 - 2026, Intel Corp.
*
*****************************************************************************/
diff --git a/drivers/acpi/acpica/tbxfroot.c b/drivers/acpi/acpica/tbxfroot.c
index 961577b..27e2e16 100644
--- a/drivers/acpi/acpica/tbxfroot.c
+++ b/drivers/acpi/acpica/tbxfroot.c
@@ -3,7 +3,7 @@
*
* Module Name: tbxfroot - Find the root ACPI table (RSDT)
*
- * Copyright (C) 2000 - 2025, Intel Corp.
+ * Copyright (C) 2000 - 2026, Intel Corp.
*
*****************************************************************************/
diff --git a/drivers/acpi/acpica/utaddress.c b/drivers/acpi/acpica/utaddress.c
index c673d6c..48f55a3 100644
--- a/drivers/acpi/acpica/utaddress.c
+++ b/drivers/acpi/acpica/utaddress.c
@@ -3,7 +3,7 @@
*
* Module Name: utaddress - op_region address range check
*
- * Copyright (C) 2000 - 2025, Intel Corp.
+ * Copyright (C) 2000 - 2026, Intel Corp.
*
*****************************************************************************/
diff --git a/drivers/acpi/acpica/utalloc.c b/drivers/acpi/acpica/utalloc.c
index 2418a31..da12b6f 100644
--- a/drivers/acpi/acpica/utalloc.c
+++ b/drivers/acpi/acpica/utalloc.c
@@ -3,7 +3,7 @@
*
* Module Name: utalloc - local memory allocation routines
*
- * Copyright (C) 2000 - 2025, Intel Corp.
+ * Copyright (C) 2000 - 2026, Intel Corp.
*
*****************************************************************************/
diff --git a/drivers/acpi/acpica/utascii.c b/drivers/acpi/acpica/utascii.c
index 259c28d..9ad2bb9 100644
--- a/drivers/acpi/acpica/utascii.c
+++ b/drivers/acpi/acpica/utascii.c
@@ -3,7 +3,7 @@
*
* Module Name: utascii - Utility ascii functions
*
- * Copyright (C) 2000 - 2025, Intel Corp.
+ * Copyright (C) 2000 - 2026, Intel Corp.
*
*****************************************************************************/
diff --git a/drivers/acpi/acpica/utbuffer.c b/drivers/acpi/acpica/utbuffer.c
index f6e6e98..4193ceb 100644
--- a/drivers/acpi/acpica/utbuffer.c
+++ b/drivers/acpi/acpica/utbuffer.c
@@ -3,7 +3,7 @@
*
* Module Name: utbuffer - Buffer dump routines
*
- * Copyright (C) 2000 - 2025, Intel Corp.
+ * Copyright (C) 2000 - 2026, Intel Corp.
*
*****************************************************************************/
diff --git a/drivers/acpi/acpica/utcache.c b/drivers/acpi/acpica/utcache.c
index cabec19..6ec2a6e 100644
--- a/drivers/acpi/acpica/utcache.c
+++ b/drivers/acpi/acpica/utcache.c
@@ -3,7 +3,7 @@
*
* Module Name: utcache - local cache allocation routines
*
- * Copyright (C) 2000 - 2025, Intel Corp.
+ * Copyright (C) 2000 - 2026, Intel Corp.
*
*****************************************************************************/
diff --git a/drivers/acpi/acpica/utcksum.c b/drivers/acpi/acpica/utcksum.c
index e6f6030..040e345 100644
--- a/drivers/acpi/acpica/utcksum.c
+++ b/drivers/acpi/acpica/utcksum.c
@@ -3,7 +3,7 @@
*
* Module Name: utcksum - Support generating table checksums
*
- * Copyright (C) 2000 - 2025, Intel Corp.
+ * Copyright (C) 2000 - 2026, Intel Corp.
*
*****************************************************************************/
diff --git a/drivers/acpi/acpica/utcopy.c b/drivers/acpi/acpica/utcopy.c
index 80458e7..e4d2f51 100644
--- a/drivers/acpi/acpica/utcopy.c
+++ b/drivers/acpi/acpica/utcopy.c
@@ -3,7 +3,7 @@
*
* Module Name: utcopy - Internal to external object translation utilities
*
- * Copyright (C) 2000 - 2025, Intel Corp.
+ * Copyright (C) 2000 - 2026, Intel Corp.
*
*****************************************************************************/
@@ -731,7 +731,15 @@ acpi_ut_copy_simple_object(union acpi_operand_object *source_desc,
break;
}
- acpi_ut_add_reference(source_desc->reference.object);
+ /*
+ * Local/Arg/Debug references do not have a valid Object pointer
+ * that can be referenced
+ */
+ if ((source_desc->reference.class != ACPI_REFCLASS_LOCAL) &&
+ (source_desc->reference.class != ACPI_REFCLASS_ARG) &&
+ (source_desc->reference.class != ACPI_REFCLASS_DEBUG)) {
+ acpi_ut_add_reference(source_desc->reference.object);
+ }
break;
case ACPI_TYPE_REGION:
diff --git a/drivers/acpi/acpica/utdebug.c b/drivers/acpi/acpica/utdebug.c
index 9f197e2..a31e074 100644
--- a/drivers/acpi/acpica/utdebug.c
+++ b/drivers/acpi/acpica/utdebug.c
@@ -3,7 +3,7 @@
*
* Module Name: utdebug - Debug print/trace routines
*
- * Copyright (C) 2000 - 2025, Intel Corp.
+ * Copyright (C) 2000 - 2026, Intel Corp.
*
*****************************************************************************/
diff --git a/drivers/acpi/acpica/utdecode.c b/drivers/acpi/acpica/utdecode.c
index b82130d..10482ca 100644
--- a/drivers/acpi/acpica/utdecode.c
+++ b/drivers/acpi/acpica/utdecode.c
@@ -3,7 +3,7 @@
*
* Module Name: utdecode - Utility decoding routines (value-to-string)
*
- * Copyright (C) 2000 - 2025, Intel Corp.
+ * Copyright (C) 2000 - 2026, Intel Corp.
*
*****************************************************************************/
diff --git a/drivers/acpi/acpica/uteval.c b/drivers/acpi/acpica/uteval.c
index abc6583..c6cc677 100644
--- a/drivers/acpi/acpica/uteval.c
+++ b/drivers/acpi/acpica/uteval.c
@@ -3,7 +3,7 @@
*
* Module Name: uteval - Object evaluation
*
- * Copyright (C) 2000 - 2025, Intel Corp.
+ * Copyright (C) 2000 - 2026, Intel Corp.
*
*****************************************************************************/
diff --git a/drivers/acpi/acpica/utglobal.c b/drivers/acpi/acpica/utglobal.c
index 97c55a1..68b6656 100644
--- a/drivers/acpi/acpica/utglobal.c
+++ b/drivers/acpi/acpica/utglobal.c
@@ -3,7 +3,7 @@
*
* Module Name: utglobal - Global variables for the ACPI subsystem
*
- * Copyright (C) 2000 - 2025, Intel Corp.
+ * Copyright (C) 2000 - 2026, Intel Corp.
*
*****************************************************************************/
diff --git a/drivers/acpi/acpica/uthex.c b/drivers/acpi/acpica/uthex.c
index 8cd050e..df09a1a 100644
--- a/drivers/acpi/acpica/uthex.c
+++ b/drivers/acpi/acpica/uthex.c
@@ -3,7 +3,7 @@
*
* Module Name: uthex -- Hex/ASCII support functions
*
- * Copyright (C) 2000 - 2025, Intel Corp.
+ * Copyright (C) 2000 - 2026, Intel Corp.
*
*****************************************************************************/
diff --git a/drivers/acpi/acpica/utids.c b/drivers/acpi/acpica/utids.c
index eb88335..e177229 100644
--- a/drivers/acpi/acpica/utids.c
+++ b/drivers/acpi/acpica/utids.c
@@ -3,7 +3,7 @@
*
* Module Name: utids - support for device Ids - HID, UID, CID, SUB, CLS
*
- * Copyright (C) 2000 - 2025, Intel Corp.
+ * Copyright (C) 2000 - 2026, Intel Corp.
*
*****************************************************************************/
diff --git a/drivers/acpi/acpica/utinit.c b/drivers/acpi/acpica/utinit.c
index 4bef97e..79d9412 100644
--- a/drivers/acpi/acpica/utinit.c
+++ b/drivers/acpi/acpica/utinit.c
@@ -3,7 +3,7 @@
*
* Module Name: utinit - Common ACPI subsystem initialization
*
- * Copyright (C) 2000 - 2025, Intel Corp.
+ * Copyright (C) 2000 - 2026, Intel Corp.
*
*****************************************************************************/
diff --git a/drivers/acpi/acpica/utlock.c b/drivers/acpi/acpica/utlock.c
index 123dbcb..cf6b931 100644
--- a/drivers/acpi/acpica/utlock.c
+++ b/drivers/acpi/acpica/utlock.c
@@ -3,7 +3,7 @@
*
* Module Name: utlock - Reader/Writer lock interfaces
*
- * Copyright (C) 2000 - 2025, Intel Corp.
+ * Copyright (C) 2000 - 2026, Intel Corp.
*
*****************************************************************************/
diff --git a/drivers/acpi/acpica/utobject.c b/drivers/acpi/acpica/utobject.c
index 8362204..60513df 100644
--- a/drivers/acpi/acpica/utobject.c
+++ b/drivers/acpi/acpica/utobject.c
@@ -3,7 +3,7 @@
*
* Module Name: utobject - ACPI object create/delete/size/cache routines
*
- * Copyright (C) 2000 - 2025, Intel Corp.
+ * Copyright (C) 2000 - 2026, Intel Corp.
*
*****************************************************************************/
diff --git a/drivers/acpi/acpica/utosi.c b/drivers/acpi/acpica/utosi.c
index 88d0418..64e26fb 100644
--- a/drivers/acpi/acpica/utosi.c
+++ b/drivers/acpi/acpica/utosi.c
@@ -3,7 +3,7 @@
*
* Module Name: utosi - Support for the _OSI predefined control method
*
- * Copyright (C) 2000 - 2025, Intel Corp.
+ * Copyright (C) 2000 - 2026, Intel Corp.
*
*****************************************************************************/
diff --git a/drivers/acpi/acpica/utpredef.c b/drivers/acpi/acpica/utpredef.c
index d9bd80e..dd15482 100644
--- a/drivers/acpi/acpica/utpredef.c
+++ b/drivers/acpi/acpica/utpredef.c
@@ -3,7 +3,7 @@
*
* Module Name: utpredef - support functions for predefined names
*
- * Copyright (C) 2000 - 2025, Intel Corp.
+ * Copyright (C) 2000 - 2026, Intel Corp.
*
*****************************************************************************/
diff --git a/drivers/acpi/acpica/utprint.c b/drivers/acpi/acpica/utprint.c
index 423d105..52f41d7 100644
--- a/drivers/acpi/acpica/utprint.c
+++ b/drivers/acpi/acpica/utprint.c
@@ -3,7 +3,7 @@
*
* Module Name: utprint - Formatted printing routines
*
- * Copyright (C) 2000 - 2025, Intel Corp.
+ * Copyright (C) 2000 - 2026, Intel Corp.
*
*****************************************************************************/
diff --git a/drivers/acpi/acpica/utresrc.c b/drivers/acpi/acpica/utresrc.c
index e1cc3d3..86ebd9f 100644
--- a/drivers/acpi/acpica/utresrc.c
+++ b/drivers/acpi/acpica/utresrc.c
@@ -165,6 +165,28 @@ acpi_ut_walk_aml_resources(struct acpi_walk_state *walk_state,
/* Walk the byte list, abort on any invalid descriptor type or length */
while (aml < end_aml) {
+ /*
+ * Validate that the remaining buffer space can hold enough
+ * bytes to safely access fields during validation.
+ * For large resource descriptors (bit 7 set), we need enough
+ * bytes to access the Type field in serial_bus resources.
+ * Small resource descriptors only need sizeof(struct aml_resource_end_tag).
+ */
+ if ((acpi_size)(end_aml - aml) <
+ sizeof(struct aml_resource_end_tag)) {
+ return_ACPI_STATUS(AE_AML_BUFFER_LENGTH);
+ }
+
+ /*
+ * For large resource descriptors, ensure enough space for
+ * the header plus serial_bus Type field access.
+ */
+ if ((ACPI_GET8(aml) & ACPI_RESOURCE_NAME_LARGE) &&
+ ((acpi_size)(end_aml - aml) <
+ ACPI_OFFSET(struct aml_resource_common_serialbus,
+ type) + 1)) {
+ return_ACPI_STATUS(AE_AML_BUFFER_LENGTH);
+ }
/* Validate the Resource Type and Resource Length */
@@ -182,6 +204,14 @@ acpi_ut_walk_aml_resources(struct acpi_walk_state *walk_state,
length = acpi_ut_get_descriptor_length(aml);
+ /*
+ * Validate that the descriptor length doesn't exceed the
+ * remaining buffer size to prevent reading beyond the end.
+ */
+ if (length > (acpi_size)(end_aml - aml)) {
+ return_ACPI_STATUS(AE_AML_BUFFER_LENGTH);
+ }
+
/* Invoke the user function */
if (user_function) {
diff --git a/drivers/acpi/acpica/uttrack.c b/drivers/acpi/acpica/uttrack.c
index a99c4c9..f013fb9c 100644
--- a/drivers/acpi/acpica/uttrack.c
+++ b/drivers/acpi/acpica/uttrack.c
@@ -3,7 +3,7 @@
*
* Module Name: uttrack - Memory allocation tracking routines (debug only)
*
- * Copyright (C) 2000 - 2025, Intel Corp.
+ * Copyright (C) 2000 - 2026, Intel Corp.
*
*****************************************************************************/
diff --git a/drivers/acpi/acpica/utuuid.c b/drivers/acpi/acpica/utuuid.c
index 0682554..aa8985b 100644
--- a/drivers/acpi/acpica/utuuid.c
+++ b/drivers/acpi/acpica/utuuid.c
@@ -3,7 +3,7 @@
*
* Module Name: utuuid -- UUID support functions
*
- * Copyright (C) 2000 - 2025, Intel Corp.
+ * Copyright (C) 2000 - 2026, Intel Corp.
*
*****************************************************************************/
diff --git a/drivers/acpi/acpica/utxface.c b/drivers/acpi/acpica/utxface.c
index 56942b5..e1d08a1 100644
--- a/drivers/acpi/acpica/utxface.c
+++ b/drivers/acpi/acpica/utxface.c
@@ -3,7 +3,7 @@
*
* Module Name: utxface - External interfaces, miscellaneous utility functions
*
- * Copyright (C) 2000 - 2025, Intel Corp.
+ * Copyright (C) 2000 - 2026, Intel Corp.
*
*****************************************************************************/
diff --git a/drivers/acpi/acpica/utxfinit.c b/drivers/acpi/acpica/utxfinit.c
index c1702f8..61f50e0 100644
--- a/drivers/acpi/acpica/utxfinit.c
+++ b/drivers/acpi/acpica/utxfinit.c
@@ -3,7 +3,7 @@
*
* Module Name: utxfinit - External interfaces for ACPICA initialization
*
- * Copyright (C) 2000 - 2025, Intel Corp.
+ * Copyright (C) 2000 - 2026, Intel Corp.
*
*****************************************************************************/
diff --git a/drivers/acpi/battery.c b/drivers/acpi/battery.c
index b82dd67..f5e0eb2 100644
--- a/drivers/acpi/battery.c
+++ b/drivers/acpi/battery.c
@@ -1182,6 +1182,26 @@ static const struct dmi_system_id bat_dmi_table[] __initconst = {
{},
};
+static void acpi_battery_wakeup_cleanup(void *data)
+{
+ device_init_wakeup(data, false);
+}
+
+static int devm_acpi_battery_init_wakeup(struct device *dev)
+{
+ device_init_wakeup(dev, true);
+ return devm_add_action_or_reset(dev, acpi_battery_wakeup_cleanup, dev);
+}
+
+static void sysfs_battery_cleanup(void *data)
+{
+ struct acpi_battery *battery = data;
+
+ guard(mutex)(&battery->update_lock);
+
+ sysfs_remove_battery(battery);
+}
+
/*
* Some machines'(E,G Lenovo Z480) ECs are not stable
* during boot up and this causes battery driver fails to be
@@ -1190,10 +1210,15 @@ static const struct dmi_system_id bat_dmi_table[] __initconst = {
* may work. So add retry code here and 20ms sleep between
* every retries.
*/
-static int acpi_battery_update_retry(struct acpi_battery *battery)
+static int devm_acpi_battery_update_retry(struct device *dev,
+ struct acpi_battery *battery)
{
int retry, ret;
+ ret = devm_add_action(dev, sysfs_battery_cleanup, battery);
+ if (ret)
+ return ret;
+
guard(mutex)(&battery->update_lock);
for (retry = 5; retry; retry--) {
@@ -1206,27 +1231,21 @@ static int acpi_battery_update_retry(struct acpi_battery *battery)
return ret;
}
-static void sysfs_battery_cleanup(struct acpi_battery *battery)
-{
- guard(mutex)(&battery->update_lock);
-
- sysfs_remove_battery(battery);
-}
-
static int acpi_battery_probe(struct platform_device *pdev)
{
+ struct device *dev = &pdev->dev;
struct acpi_battery *battery;
struct acpi_device *device;
int result;
- device = ACPI_COMPANION(&pdev->dev);
+ device = ACPI_COMPANION(dev);
if (!device)
return -ENODEV;
if (device->dep_unmet)
return -EPROBE_DEFER;
- battery = devm_kzalloc(&pdev->dev, sizeof(*battery), GFP_KERNEL);
+ battery = devm_kzalloc(dev, sizeof(*battery), GFP_KERNEL);
if (!battery)
return -ENOMEM;
@@ -1235,54 +1254,38 @@ static int acpi_battery_probe(struct platform_device *pdev)
battery->phys_dev = &pdev->dev;
battery->device = device;
- result = devm_mutex_init(&pdev->dev, &battery->update_lock);
+ result = devm_mutex_init(dev, &battery->update_lock);
if (result)
return result;
if (acpi_has_method(battery->device->handle, "_BIX"))
set_bit(ACPI_BATTERY_XINFO_PRESENT, &battery->flags);
- result = acpi_battery_update_retry(battery);
+ result = devm_acpi_battery_update_retry(dev, battery);
if (result)
- goto fail;
+ return result;
pr_info("Slot [%s] (battery %s)\n", acpi_device_bid(device),
device->status.battery_present ? "present" : "absent");
+ result = devm_acpi_battery_init_wakeup(dev);
+ if (result)
+ return result;
+
+ result = devm_acpi_install_notify_handler(dev, ACPI_ALL_NOTIFY,
+ acpi_battery_notify, battery);
+ if (result)
+ return result;
+
battery->pm_nb.notifier_call = battery_notify;
- result = register_pm_notifier(&battery->pm_nb);
- if (result)
- goto fail;
-
- device_init_wakeup(&pdev->dev, true);
-
- result = acpi_dev_install_notify_handler(device, ACPI_ALL_NOTIFY,
- acpi_battery_notify, battery);
- if (result)
- goto fail_pm;
-
- return 0;
-
-fail_pm:
- device_init_wakeup(&pdev->dev, false);
- unregister_pm_notifier(&battery->pm_nb);
-fail:
- sysfs_battery_cleanup(battery);
-
- return result;
+ return register_pm_notifier(&battery->pm_nb);
}
static void acpi_battery_remove(struct platform_device *pdev)
{
struct acpi_battery *battery = platform_get_drvdata(pdev);
- acpi_dev_remove_notify_handler(battery->device, ACPI_ALL_NOTIFY,
- acpi_battery_notify);
-
- device_init_wakeup(&pdev->dev, false);
unregister_pm_notifier(&battery->pm_nb);
-
- sysfs_battery_cleanup(battery);
}
/* this is needed to learn about changes made in suspended state */
diff --git a/drivers/acpi/bus.c b/drivers/acpi/bus.c
index 2ec095e2..b0e7181a 100644
--- a/drivers/acpi/bus.c
+++ b/drivers/acpi/bus.c
@@ -679,6 +679,73 @@ void acpi_dev_remove_notify_handler(struct acpi_device *adev,
}
EXPORT_SYMBOL_GPL(acpi_dev_remove_notify_handler);
+struct acpi_notify_handler_devres {
+ struct acpi_device *adev;
+ acpi_notify_handler handler;
+ u32 handler_type;
+};
+
+static void devm_acpi_notify_handler_release(struct device *dev, void *res)
+{
+ struct acpi_notify_handler_devres *dr = res;
+
+ acpi_dev_remove_notify_handler(dr->adev, dr->handler_type, dr->handler);
+}
+
+/**
+ * devm_acpi_install_notify_handler - Install an ACPI notify handler for a
+ * managed device
+ * @dev: Device to install a notify handler for
+ * @handler_type: Type of the notify handler
+ * @handler: Handler function to install
+ * @context: Data passed back to the handler function
+ *
+ * This function performs the same function as acpi_dev_install_notify_handler()
+ * called for the ACPI companion of @dev with the same @handler_type, @handler,
+ * and @context arguments, but the ACPI notify handler installed by it will be
+ * automatically removed on driver detach.
+ *
+ * Callers should ensure that all resources used by @handler have been allocated
+ * prior to invoking this function, in which case those resources should be
+ * devres-managed so that they won't be released before the notify handler
+ * removal. Otherwise, special synchronization between @handler and the
+ * management of those resources is required.
+ *
+ * When the request fails, an error message is printed. Don't add extra error
+ * messages at the call sites.
+ *
+ * Return: 0 on success or a negative error number.
+ */
+int devm_acpi_install_notify_handler(struct device *dev, u32 handler_type,
+ acpi_notify_handler handler, void *context)
+{
+ struct acpi_notify_handler_devres *dr;
+ struct acpi_device *adev;
+ int ret;
+
+ adev = ACPI_COMPANION(dev);
+ if (!adev)
+ return dev_err_probe(dev, -ENODEV, "No ACPI companion\n");
+
+ dr = devres_alloc(devm_acpi_notify_handler_release, sizeof(*dr), GFP_KERNEL);
+ if (!dr)
+ return -ENOMEM;
+
+ ret = acpi_dev_install_notify_handler(adev, handler_type, handler, context);
+ if (ret) {
+ devres_free(dr);
+ return dev_err_probe(dev, ret, "Failed to install an ACPI notify handler\n");
+ }
+
+ dr->adev = adev;
+ dr->handler = handler;
+ dr->handler_type = handler_type;
+ devres_add(dev, dr);
+
+ return 0;
+}
+EXPORT_SYMBOL_GPL(devm_acpi_install_notify_handler);
+
/* Handle events targeting \_SB device (at present only graceful shutdown) */
#define ACPI_SB_NOTIFY_SHUTDOWN_REQUEST 0x81
diff --git a/drivers/acpi/button.c b/drivers/acpi/button.c
index d802763..3836ee7 100644
--- a/drivers/acpi/button.c
+++ b/drivers/acpi/button.c
@@ -28,14 +28,15 @@
#define ACPI_BUTTON_NOTIFY_WAKE 0x02
#define ACPI_BUTTON_NOTIFY_STATUS 0x80
-#define ACPI_BUTTON_SUBCLASS_POWER "power"
+#define ACPI_BUTTON_CLASS_POWER "button/power"
#define ACPI_BUTTON_DEVICE_NAME_POWER "Power Button"
#define ACPI_BUTTON_TYPE_POWER 0x01
-#define ACPI_BUTTON_SUBCLASS_SLEEP "sleep"
+#define ACPI_BUTTON_CLASS_SLEEP "button/sleep"
#define ACPI_BUTTON_DEVICE_NAME_SLEEP "Sleep Button"
#define ACPI_BUTTON_TYPE_SLEEP 0x03
+#define ACPI_BUTTON_CLASS_LID "button/lid"
#define ACPI_BUTTON_SUBCLASS_LID "lid"
#define ACPI_BUTTON_DEVICE_NAME_LID "Lid Switch"
#define ACPI_BUTTON_TYPE_LID 0x05
@@ -59,11 +60,11 @@ MODULE_DESCRIPTION("ACPI Button Driver");
MODULE_LICENSE("GPL");
static const struct acpi_device_id button_device_ids[] = {
- {ACPI_BUTTON_HID_LID, 0},
- {ACPI_BUTTON_HID_SLEEP, 0},
- {ACPI_BUTTON_HID_SLEEPF, 0},
- {ACPI_BUTTON_HID_POWER, 0},
- {ACPI_BUTTON_HID_POWERF, 0},
+ {ACPI_BUTTON_HID_LID, ACPI_BUTTON_TYPE_LID},
+ {ACPI_BUTTON_HID_SLEEP, ACPI_BUTTON_TYPE_SLEEP},
+ {ACPI_BUTTON_HID_SLEEPF, ACPI_BUTTON_TYPE_SLEEP},
+ {ACPI_BUTTON_HID_POWER, ACPI_BUTTON_TYPE_POWER},
+ {ACPI_BUTTON_HID_POWERF, ACPI_BUTTON_TYPE_POWER},
{"", 0},
};
MODULE_DEVICE_TABLE(acpi, button_device_ids);
@@ -173,16 +174,16 @@ struct acpi_button {
struct device *dev; /* physical button device */
unsigned int type;
struct input_dev *input;
+ const char *class; /* for netlink messages */
char phys[32]; /* for input device */
unsigned long pushed;
- int last_state;
+ bool last_state;
ktime_t last_time;
bool suspended;
bool lid_state_initialized;
bool gpe_enabled;
};
-static struct acpi_device *lid_device;
static long lid_init_state = -1;
static unsigned long lid_report_interval __read_mostly = 500;
@@ -193,19 +194,19 @@ MODULE_PARM_DESC(lid_report_interval, "Interval (ms) between lid key events");
static struct proc_dir_entry *acpi_button_dir;
static struct proc_dir_entry *acpi_lid_dir;
-static int acpi_lid_evaluate_state(struct acpi_device *device)
+static int acpi_lid_evaluate_state(acpi_handle lid_handle)
{
unsigned long long lid_state;
acpi_status status;
- status = acpi_evaluate_integer(device->handle, "_LID", NULL, &lid_state);
+ status = acpi_evaluate_integer(lid_handle, "_LID", NULL, &lid_state);
if (ACPI_FAILURE(status))
return -ENODEV;
- return lid_state ? 1 : 0;
+ return !!lid_state;
}
-static int acpi_lid_notify_state(struct acpi_button *button, int state)
+static void acpi_lid_notify_state(struct acpi_button *button, bool state)
{
struct acpi_device *device = button->adev;
ktime_t next_report;
@@ -218,18 +219,14 @@ static int acpi_lid_notify_state(struct acpi_button *button, int state)
* So "last_time" is only updated after a timeout or an actual
* switch.
*/
- if (lid_init_state != ACPI_BUTTON_LID_INIT_IGNORE ||
- button->last_state != !!state)
- do_update = true;
- else
- do_update = false;
-
+ do_update = lid_init_state != ACPI_BUTTON_LID_INIT_IGNORE ||
+ button->last_state != state;
next_report = ktime_add(button->last_time,
ms_to_ktime(lid_report_interval));
- if (button->last_state == !!state &&
+ if (button->last_state == state &&
ktime_after(ktime_get(), next_report)) {
- /* Complain the buggy firmware */
- pr_warn_once("The lid device is not compliant to SW_LID.\n");
+ /* Complain about the buggy firmware. */
+ pr_warn_once(FW_BUG "Unexpected lid state reported by firmware\n");
/*
* Send the unreliable complement switch event:
@@ -280,11 +277,9 @@ static int acpi_lid_notify_state(struct acpi_button *button, int state)
state ? "open" : "closed");
input_report_switch(button->input, SW_LID, !state);
input_sync(button->input);
- button->last_state = !!state;
+ button->last_state = state;
button->last_time = ktime_get();
}
-
- return 0;
}
static int __maybe_unused acpi_button_state_seq_show(struct seq_file *seq,
@@ -293,21 +288,16 @@ static int __maybe_unused acpi_button_state_seq_show(struct seq_file *seq,
struct acpi_button *button = seq->private;
int state;
- state = acpi_lid_evaluate_state(button->adev);
+ state = acpi_lid_evaluate_state(button->adev->handle);
seq_printf(seq, "state: %s\n",
state < 0 ? "unsupported" : (state ? "open" : "closed"));
return 0;
}
-static int acpi_button_add_fs(struct acpi_button *button)
+static int acpi_lid_add_fs(struct acpi_button *button)
{
struct acpi_device *device = button->adev;
struct proc_dir_entry *entry = NULL;
- int ret = 0;
-
- /* procfs I/F for ACPI lid device only */
- if (button->type != ACPI_BUTTON_TYPE_LID)
- return 0;
if (acpi_button_dir || acpi_lid_dir) {
pr_info("More than one Lid device found!\n");
@@ -321,33 +311,25 @@ static int acpi_button_add_fs(struct acpi_button *button)
/* create /proc/acpi/button/lid */
acpi_lid_dir = proc_mkdir(ACPI_BUTTON_SUBCLASS_LID, acpi_button_dir);
- if (!acpi_lid_dir) {
- ret = -ENODEV;
+ if (!acpi_lid_dir)
goto remove_button_dir;
- }
/* create /proc/acpi/button/lid/LID/ */
acpi_device_dir(device) = proc_mkdir(acpi_device_bid(device), acpi_lid_dir);
- if (!acpi_device_dir(device)) {
- ret = -ENODEV;
+ if (!acpi_device_dir(device))
goto remove_lid_dir;
- }
/* create /proc/acpi/button/lid/LID/state */
entry = proc_create_single_data(ACPI_BUTTON_FILE_STATE, S_IRUGO,
acpi_device_dir(device), acpi_button_state_seq_show,
button);
- if (!entry) {
- ret = -ENODEV;
+ if (!entry)
goto remove_dev_dir;
- }
-done:
- return ret;
+ return 0;
remove_dev_dir:
- remove_proc_entry(acpi_device_bid(device),
- acpi_lid_dir);
+ remove_proc_entry(acpi_device_bid(device), acpi_lid_dir);
acpi_device_dir(device) = NULL;
remove_lid_dir:
remove_proc_entry(ACPI_BUTTON_SUBCLASS_LID, acpi_button_dir);
@@ -355,16 +337,14 @@ static int acpi_button_add_fs(struct acpi_button *button)
remove_button_dir:
remove_proc_entry(ACPI_BUTTON_CLASS, acpi_root_dir);
acpi_button_dir = NULL;
- goto done;
+ return -ENODEV;
}
-static int acpi_button_remove_fs(struct acpi_button *button)
+static void acpi_lid_remove_fs(void *data)
{
+ struct acpi_button *button = data;
struct acpi_device *device = button->adev;
- if (button->type != ACPI_BUTTON_TYPE_LID)
- return 0;
-
remove_proc_entry(ACPI_BUTTON_FILE_STATE,
acpi_device_dir(device));
remove_proc_entry(acpi_device_bid(device),
@@ -374,44 +354,71 @@ static int acpi_button_remove_fs(struct acpi_button *button)
acpi_lid_dir = NULL;
remove_proc_entry(ACPI_BUTTON_CLASS, acpi_root_dir);
acpi_button_dir = NULL;
+}
- return 0;
+static int devm_acpi_lid_add_fs(struct device *dev, struct acpi_button *button)
+{
+ int ret;
+
+ ret = acpi_lid_add_fs(button);
+ if (ret)
+ return ret;
+
+ return devm_add_action_or_reset(dev, acpi_lid_remove_fs, button);
+}
+
+static acpi_handle saved_lid_handle;
+static DEFINE_MUTEX(acpi_lid_lock);
+
+static void acpi_lid_save(struct acpi_device *adev)
+{
+ guard(mutex)(&acpi_lid_lock);
+
+ saved_lid_handle = adev->handle;
+}
+
+static void acpi_lid_forget(struct acpi_device *adev)
+{
+ guard(mutex)(&acpi_lid_lock);
+
+ if (saved_lid_handle == adev->handle)
+ saved_lid_handle = NULL;
}
/* Driver Interface */
int acpi_lid_open(void)
{
- if (!lid_device)
+ guard(mutex)(&acpi_lid_lock);
+
+ if (!saved_lid_handle)
return -ENODEV;
- return acpi_lid_evaluate_state(lid_device);
+ return acpi_lid_evaluate_state(saved_lid_handle);
}
EXPORT_SYMBOL(acpi_lid_open);
-static int acpi_lid_update_state(struct acpi_button *button,
- bool signal_wakeup)
+static void acpi_lid_update_state(struct acpi_button *button, bool signal_wakeup)
{
- struct acpi_device *device = button->adev;
int state;
- state = acpi_lid_evaluate_state(device);
+ state = acpi_lid_evaluate_state(button->adev->handle);
if (state < 0)
- return state;
+ return;
if (state && signal_wakeup)
acpi_pm_wakeup_event(button->dev);
- return acpi_lid_notify_state(button, state);
+ acpi_lid_notify_state(button, state);
}
static void acpi_lid_initialize_state(struct acpi_button *button)
{
switch (lid_init_state) {
case ACPI_BUTTON_LID_INIT_OPEN:
- (void)acpi_lid_notify_state(button, 1);
+ acpi_lid_notify_state(button, true);
break;
case ACPI_BUTTON_LID_INIT_METHOD:
- (void)acpi_lid_update_state(button, false);
+ acpi_lid_update_state(button, false);
break;
case ACPI_BUTTON_LID_INIT_IGNORE:
default:
@@ -469,8 +476,7 @@ static void acpi_button_notify(acpi_handle handle, u32 event, void *data)
input_report_key(input, keycode, 0);
input_sync(input);
- acpi_bus_generate_netlink_event(acpi_device_class(device),
- dev_name(&device->dev),
+ acpi_bus_generate_netlink_event(button->class, dev_name(&device->dev),
event, ++button->pushed);
}
@@ -497,12 +503,11 @@ static int acpi_button_suspend(struct device *dev)
static int acpi_button_resume(struct device *dev)
{
struct acpi_button *button = dev_get_drvdata(dev);
- struct acpi_device *device = ACPI_COMPANION(dev);
struct input_dev *input;
button->suspended = false;
if (button->type == ACPI_BUTTON_TYPE_LID) {
- button->last_state = !!acpi_lid_evaluate_state(device);
+ button->last_state = !!acpi_lid_evaluate_state(ACPI_HANDLE(dev));
button->last_time = ktime_get();
acpi_lid_initialize_state(button);
}
@@ -521,179 +526,36 @@ static int acpi_button_resume(struct device *dev)
static int acpi_lid_input_open(struct input_dev *input)
{
struct acpi_button *button = input_get_drvdata(input);
- struct acpi_device *device = button->adev;
- button->last_state = !!acpi_lid_evaluate_state(device);
+ button->last_state = !!acpi_lid_evaluate_state(button->adev->handle);
button->last_time = ktime_get();
acpi_lid_initialize_state(button);
return 0;
}
-static int acpi_button_probe(struct platform_device *pdev)
+static acpi_notify_handler acpi_button_notify_handler(struct acpi_button *button)
{
- acpi_notify_handler handler;
- struct acpi_device *device;
- struct acpi_button *button;
- struct input_dev *input;
- acpi_status status;
- char *name, *class;
- const char *hid;
- int error = 0;
+ if (button->type == ACPI_BUTTON_TYPE_LID)
+ return acpi_lid_notify;
- device = ACPI_COMPANION(&pdev->dev);
- if (!device)
- return -ENODEV;
-
- hid = acpi_device_hid(device);
- if (!strcmp(hid, ACPI_BUTTON_HID_LID) &&
- lid_init_state == ACPI_BUTTON_LID_INIT_DISABLED)
- return -ENODEV;
-
- button = kzalloc_obj(struct acpi_button);
- if (!button)
- return -ENOMEM;
-
- platform_set_drvdata(pdev, button);
-
- button->dev = &pdev->dev;
- button->adev = device;
- button->input = input = input_allocate_device();
- if (!input) {
- error = -ENOMEM;
- goto err_free_button;
- }
-
- class = acpi_device_class(device);
-
- if (!strcmp(hid, ACPI_BUTTON_HID_POWER) ||
- !strcmp(hid, ACPI_BUTTON_HID_POWERF)) {
- button->type = ACPI_BUTTON_TYPE_POWER;
- handler = acpi_button_notify;
- name = ACPI_BUTTON_DEVICE_NAME_POWER;
- sprintf(class, "%s/%s",
- ACPI_BUTTON_CLASS, ACPI_BUTTON_SUBCLASS_POWER);
- } else if (!strcmp(hid, ACPI_BUTTON_HID_SLEEP) ||
- !strcmp(hid, ACPI_BUTTON_HID_SLEEPF)) {
- button->type = ACPI_BUTTON_TYPE_SLEEP;
- handler = acpi_button_notify;
- name = ACPI_BUTTON_DEVICE_NAME_SLEEP;
- sprintf(class, "%s/%s",
- ACPI_BUTTON_CLASS, ACPI_BUTTON_SUBCLASS_SLEEP);
- } else if (!strcmp(hid, ACPI_BUTTON_HID_LID)) {
- button->type = ACPI_BUTTON_TYPE_LID;
- handler = acpi_lid_notify;
- name = ACPI_BUTTON_DEVICE_NAME_LID;
- sprintf(class, "%s/%s",
- ACPI_BUTTON_CLASS, ACPI_BUTTON_SUBCLASS_LID);
- input->open = acpi_lid_input_open;
- } else {
- pr_info("Unsupported hid [%s]\n", hid);
- error = -ENODEV;
- }
-
- if (!error)
- error = acpi_button_add_fs(button);
-
- if (error) {
- input_free_device(input);
- goto err_free_button;
- }
-
- snprintf(button->phys, sizeof(button->phys), "%s/button/input0", hid);
-
- input->name = name;
- input->phys = button->phys;
- input->id.bustype = BUS_HOST;
- input->id.product = button->type;
- input->dev.parent = &pdev->dev;
-
- switch (button->type) {
- case ACPI_BUTTON_TYPE_POWER:
- input_set_capability(input, EV_KEY, KEY_POWER);
- input_set_capability(input, EV_KEY, KEY_WAKEUP);
- break;
-
- case ACPI_BUTTON_TYPE_SLEEP:
- input_set_capability(input, EV_KEY, KEY_SLEEP);
- break;
-
- case ACPI_BUTTON_TYPE_LID:
- input_set_capability(input, EV_SW, SW_LID);
- break;
- }
-
- input_set_drvdata(input, button);
- error = input_register_device(input);
- if (error) {
- input_free_device(input);
- goto err_remove_fs;
- }
-
- device_init_wakeup(button->dev, true);
-
- switch (device->device_type) {
- case ACPI_BUS_TYPE_POWER_BUTTON:
- status = acpi_install_fixed_event_handler(ACPI_EVENT_POWER_BUTTON,
- acpi_button_event,
- button);
- break;
- case ACPI_BUS_TYPE_SLEEP_BUTTON:
- status = acpi_install_fixed_event_handler(ACPI_EVENT_SLEEP_BUTTON,
- acpi_button_event,
- button);
- break;
- default:
- status = acpi_install_notify_handler(device->handle,
- ACPI_ALL_NOTIFY, handler,
- button);
- if (ACPI_SUCCESS(status) && device->wakeup.flags.valid) {
- acpi_status st;
-
- /*
- * If the wakeup GPE has a handler method, enable it in
- * case it is also used for signaling runtime events.
- */
- st = acpi_enable_gpe_cond(device->wakeup.gpe_device,
- device->wakeup.gpe_number,
- ACPI_GPE_DISPATCH_METHOD);
- button->gpe_enabled = ACPI_SUCCESS(st);
- if (button->gpe_enabled)
- dev_dbg(button->dev, "Enabled ACPI GPE%02llx\n",
- device->wakeup.gpe_number);
- }
- break;
- }
- if (ACPI_FAILURE(status)) {
- error = -ENODEV;
- goto err_input_unregister;
- }
-
- if (button->type == ACPI_BUTTON_TYPE_LID) {
- /*
- * This assumes there's only one lid device, or if there are
- * more we only care about the last one...
- */
- lid_device = device;
- }
-
- pr_info("%s [%s]\n", name, acpi_device_bid(device));
- return 0;
-
-err_input_unregister:
- device_init_wakeup(button->dev, false);
- input_unregister_device(input);
-err_remove_fs:
- acpi_button_remove_fs(button);
-err_free_button:
- kfree(button);
- memset(acpi_device_class(device), 0, sizeof(acpi_device_class));
- return error;
+ return acpi_button_notify;
}
-static void acpi_button_remove(struct platform_device *pdev)
+static void acpi_button_wakeup_cleanup(void *data)
{
- struct acpi_button *button = platform_get_drvdata(pdev);
+ device_init_wakeup(data, false);
+}
+
+static int devm_acpi_button_init_wakeup(struct device *dev)
+{
+ device_init_wakeup(dev, true);
+ return devm_add_action_or_reset(dev, acpi_button_wakeup_cleanup, dev);
+}
+
+static void acpi_button_remove_event_handler(void *data)
+{
+ struct acpi_button *button = data;
struct acpi_device *adev = button->adev;
switch (adev->device_type) {
@@ -701,10 +563,12 @@ static void acpi_button_remove(struct platform_device *pdev)
acpi_remove_fixed_event_handler(ACPI_EVENT_POWER_BUTTON,
acpi_button_event);
break;
+
case ACPI_BUS_TYPE_SLEEP_BUTTON:
acpi_remove_fixed_event_handler(ACPI_EVENT_SLEEP_BUTTON,
acpi_button_event);
break;
+
default:
if (button->gpe_enabled) {
dev_dbg(button->dev, "Disabling ACPI GPE%02llx\n",
@@ -713,20 +577,180 @@ static void acpi_button_remove(struct platform_device *pdev)
adev->wakeup.gpe_number);
}
acpi_remove_notify_handler(adev->handle, ACPI_ALL_NOTIFY,
- button->type == ACPI_BUTTON_TYPE_LID ?
- acpi_lid_notify :
- acpi_button_notify);
+ acpi_button_notify_handler(button));
break;
}
acpi_os_wait_events_complete();
+}
- device_init_wakeup(button->dev, false);
+static int acpi_button_add_fixed_event_handler(u32 event,
+ struct acpi_button *button)
+{
+ acpi_status status;
- acpi_button_remove_fs(button);
- input_unregister_device(button->input);
- kfree(button);
+ status = acpi_install_fixed_event_handler(event, acpi_button_event, button);
+ if (ACPI_FAILURE(status))
+ return -ENODEV;
- memset(acpi_device_class(adev), 0, sizeof(acpi_device_class));
+ return 0;
+}
+
+static int acpi_button_add_event_handler(struct acpi_button *button)
+{
+ struct acpi_device *adev = button->adev;
+ acpi_status status;
+
+ if (adev->device_type == ACPI_BUS_TYPE_POWER_BUTTON)
+ return acpi_button_add_fixed_event_handler(ACPI_EVENT_POWER_BUTTON,
+ button);
+
+ if (adev->device_type == ACPI_BUS_TYPE_SLEEP_BUTTON)
+ return acpi_button_add_fixed_event_handler(ACPI_EVENT_SLEEP_BUTTON,
+ button);
+
+ status = acpi_install_notify_handler(adev->handle, ACPI_ALL_NOTIFY,
+ acpi_button_notify_handler(button),
+ button);
+ if (ACPI_FAILURE(status))
+ return -ENODEV;
+
+ if (!adev->wakeup.flags.valid)
+ return 0;
+
+ /*
+ * If the wakeup GPE has a handler method, enable it in case it is also
+ * used for signaling runtime events.
+ */
+ status = acpi_enable_gpe_cond(adev->wakeup.gpe_device,
+ adev->wakeup.gpe_number,
+ ACPI_GPE_DISPATCH_METHOD);
+ button->gpe_enabled = ACPI_SUCCESS(status);
+ if (button->gpe_enabled)
+ dev_dbg(button->dev, "Enabled ACPI GPE%02llx\n",
+ adev->wakeup.gpe_number);
+
+ return 0;
+}
+
+static int devm_acpi_button_add_event_handler(struct device *dev,
+ struct acpi_button *button)
+{
+ int ret;
+
+ ret = acpi_button_add_event_handler(button);
+ if (ret)
+ return ret;
+
+ return devm_add_action_or_reset(dev, acpi_button_remove_event_handler,
+ button);
+}
+
+static int acpi_button_probe(struct platform_device *pdev)
+{
+ struct device *dev = &pdev->dev;
+ struct acpi_device *device = ACPI_COMPANION(dev);
+ const struct acpi_device_id *id;
+ struct acpi_button *button;
+ struct input_dev *input;
+ u8 button_type;
+ int error = 0;
+
+ id = acpi_match_acpi_device(button_device_ids, device);
+ if (!id || strcmp(acpi_device_hid(device), id->id))
+ return dev_err_probe(dev, -ENODEV, "Unsupported device\n");
+
+ button_type = id->driver_data;
+ if (button_type == ACPI_BUTTON_TYPE_LID &&
+ lid_init_state == ACPI_BUTTON_LID_INIT_DISABLED)
+ return -ENODEV;
+
+ button = devm_kzalloc(dev, sizeof(*button), GFP_KERNEL);
+ if (!button)
+ return -ENOMEM;
+
+ platform_set_drvdata(pdev, button);
+
+ button->dev = dev;
+ button->adev = device;
+ input = devm_input_allocate_device(dev);
+ if (!input)
+ return -ENOMEM;
+
+ button->input = input;
+ button->type = button_type;
+
+ switch (button_type) {
+ case ACPI_BUTTON_TYPE_LID:
+ button->class = ACPI_BUTTON_CLASS_LID;
+
+ input->name = ACPI_BUTTON_DEVICE_NAME_LID;
+ input_set_capability(input, EV_SW, SW_LID);
+ input->open = acpi_lid_input_open;
+
+ error = devm_acpi_lid_add_fs(dev, button);
+ if (error)
+ return error;
+
+ break;
+
+ case ACPI_BUTTON_TYPE_POWER:
+ button->class = ACPI_BUTTON_CLASS_POWER;
+
+ input->name = ACPI_BUTTON_DEVICE_NAME_POWER;
+ input_set_capability(input, EV_KEY, KEY_POWER);
+ input_set_capability(input, EV_KEY, KEY_WAKEUP);
+ break;
+
+ case ACPI_BUTTON_TYPE_SLEEP:
+ button->class = ACPI_BUTTON_CLASS_SLEEP;
+
+ input->name = ACPI_BUTTON_DEVICE_NAME_SLEEP;
+ input_set_capability(input, EV_KEY, KEY_SLEEP);
+ break;
+
+ default:
+ return dev_err_probe(dev, -ENODEV, "Unrecognized button type\n");
+ }
+
+ snprintf(button->phys, sizeof(button->phys), "%s/button/input0",
+ acpi_device_hid(device));
+
+ input->phys = button->phys;
+ input->id.bustype = BUS_HOST;
+ input->id.product = button_type;
+
+ input_set_drvdata(input, button);
+ error = input_register_device(input);
+ if (error)
+ return error;
+
+ error = devm_acpi_button_init_wakeup(dev);
+ if (error)
+ return error;
+
+ error = devm_acpi_button_add_event_handler(dev, button);
+ if (error)
+ return error;
+
+ if (button_type == ACPI_BUTTON_TYPE_LID) {
+ /*
+ * This assumes there's only one lid device, or if there are
+ * more we only care about the last one...
+ */
+ acpi_lid_save(device);
+ }
+
+ pr_info("%s [%s]\n", input->name, acpi_device_bid(device));
+
+ return 0;
+}
+
+static void acpi_button_remove(struct platform_device *pdev)
+{
+ struct acpi_button *button = platform_get_drvdata(pdev);
+
+ if (button->type == ACPI_BUTTON_TYPE_LID)
+ acpi_lid_forget(button->adev);
}
static int param_set_lid_init_state(const char *val,
diff --git a/drivers/acpi/cppc_acpi.c b/drivers/acpi/cppc_acpi.c
index f370be8..9f572f4 100644
--- a/drivers/acpi/cppc_acpi.c
+++ b/drivers/acpi/cppc_acpi.c
@@ -134,7 +134,7 @@ static DEFINE_PER_CPU(struct cpc_desc *, cpc_desc_ptr);
* cpc_regs[] with the corresponding index. 0 means mandatory and 1
* means optional.
*/
-#define REG_OPTIONAL (0x1FC7D0)
+#define REG_OPTIONAL (0x7FC7D0)
/*
* Use the index of the register in per-cpu cpc_regs[] to check if
@@ -185,8 +185,13 @@ show_cppc_data(cppc_get_perf_caps, cppc_perf_caps, nominal_freq);
show_cppc_data(cppc_get_perf_ctrs, cppc_perf_fb_ctrs, wraparound_time);
-/* Check for valid access_width, otherwise, fallback to using bit_width */
-#define GET_BIT_WIDTH(reg) ((reg)->access_width ? (8 << ((reg)->access_width - 1)) : (reg)->bit_width)
+/*
+ * PCC reuses the access_width field as the subspace id, so only decode access
+ * size for non-PCC registers. Otherwise, use the bit_width.
+ */
+#define GET_BIT_WIDTH(reg) (((reg)->access_width && \
+ (reg)->space_id != ACPI_ADR_SPACE_PLATFORM_COMM) ? \
+ (8 << ((reg)->access_width - 1)) : (reg)->bit_width)
/* Shift and apply the mask for CPC reads/writes */
#define MASK_VAL_READ(reg, val) (((val) >> (reg)->bit_offset) & \
@@ -751,18 +756,19 @@ int acpi_cppc_processor_probe(struct acpi_processor *pr)
/*
* Disregard _CPC if the number of entries in the return package is not
* as expected, but support future revisions being proper supersets of
- * the v3 and only causing more entries to be returned by _CPC.
+ * the v4 and only causing more entries to be returned by _CPC.
*/
if ((cpc_rev == CPPC_V2_REV && num_ent != CPPC_V2_NUM_ENT) ||
(cpc_rev == CPPC_V3_REV && num_ent != CPPC_V3_NUM_ENT) ||
- (cpc_rev > CPPC_V3_REV && num_ent <= CPPC_V3_NUM_ENT)) {
+ (cpc_rev == CPPC_V4_REV && num_ent != CPPC_V4_NUM_ENT) ||
+ (cpc_rev > CPPC_V4_REV && num_ent <= CPPC_V4_NUM_ENT)) {
pr_debug("Unexpected number of _CPC return package entries (%d) for CPU:%d\n",
num_ent, pr->id);
goto out_free;
}
- if (cpc_rev > CPPC_V3_REV) {
- num_ent = CPPC_V3_NUM_ENT;
- cpc_rev = CPPC_V3_REV;
+ if (cpc_rev > CPPC_V4_REV) {
+ num_ent = CPPC_V4_NUM_ENT;
+ cpc_rev = CPPC_V4_REV;
}
cpc_ptr->num_entries = num_ent;
@@ -845,6 +851,16 @@ int acpi_cppc_processor_probe(struct acpi_processor *pr)
cpc_ptr->cpc_regs[i-2].type = ACPI_TYPE_BUFFER;
memcpy(&cpc_ptr->cpc_regs[i-2].cpc_entry.reg, gas_t, sizeof(*gas_t));
+ } else if (cpc_obj->type == ACPI_TYPE_PACKAGE && (i - 2) == RESOURCE_PRIORITY) {
+ /*
+ * ACPI 6.6, s8.4.6.1.2.7 defines Resource Priority as a
+ * Package of Resource Priority Register Descriptor sub-packages.
+ * Parsing the full structure is not yet supported.
+ * Mark the register as unsupported for now.
+ */
+ pr_debug("CPU:%d Resource Priority not supported\n", pr->id);
+ cpc_ptr->cpc_regs[i-2].type = ACPI_TYPE_INTEGER;
+ cpc_ptr->cpc_regs[i-2].cpc_entry.int_value = 0;
} else {
pr_debug("Invalid entry type (%d) in _CPC for CPU:%d\n",
i, pr->id);
@@ -1045,7 +1061,6 @@ static int cpc_read(int cpu, struct cpc_register_resource *reg_res, u64 *val)
* by the bit width field; the access size is used to indicate
* the PCC subspace id.
*/
- size = reg->bit_width;
vaddr = GET_PCC_VADDR(reg->address, pcc_ss_id);
}
else if (reg->space_id == ACPI_ADR_SPACE_SYSTEM_MEMORY)
@@ -1118,7 +1133,6 @@ static int cpc_write(int cpu, struct cpc_register_resource *reg_res, u64 val)
* by the bit width field; the access size is used to indicate
* the PCC subspace id.
*/
- size = reg->bit_width;
vaddr = GET_PCC_VADDR(reg->address, pcc_ss_id);
}
else if (reg->space_id == ACPI_ADR_SPACE_SYSTEM_MEMORY)
diff --git a/drivers/acpi/hed.c b/drivers/acpi/hed.c
index 060e8d6..48562f5 100644
--- a/drivers/acpi/hed.c
+++ b/drivers/acpi/hed.c
@@ -22,7 +22,7 @@ static const struct acpi_device_id acpi_hed_ids[] = {
};
MODULE_DEVICE_TABLE(acpi, acpi_hed_ids);
-static acpi_handle hed_handle;
+static bool hed_present;
static BLOCKING_NOTIFIER_HEAD(acpi_hed_notify_list);
@@ -50,33 +50,24 @@ static void acpi_hed_notify(acpi_handle handle, u32 event, void *data)
static int acpi_hed_probe(struct platform_device *pdev)
{
- struct acpi_device *device;
int err;
- device = ACPI_COMPANION(&pdev->dev);
- if (!device)
- return -ENODEV;
-
/* Only one hardware error device */
- if (hed_handle)
+ if (hed_present)
return -EINVAL;
- hed_handle = device->handle;
- err = acpi_dev_install_notify_handler(device, ACPI_DEVICE_NOTIFY,
- acpi_hed_notify, device);
+ err = devm_acpi_install_notify_handler(&pdev->dev, ACPI_DEVICE_NOTIFY,
+ acpi_hed_notify, NULL);
if (err)
- hed_handle = NULL;
+ return err;
- return err;
+ hed_present = true;
+ return 0;
}
static void acpi_hed_remove(struct platform_device *pdev)
{
- struct acpi_device *device = ACPI_COMPANION(&pdev->dev);
-
- acpi_dev_remove_notify_handler(device, ACPI_DEVICE_NOTIFY,
- acpi_hed_notify);
- hed_handle = NULL;
+ hed_present = false;
}
static struct platform_driver acpi_hed_driver = {
diff --git a/drivers/acpi/nfit/core.c b/drivers/acpi/nfit/core.c
index 9304ac9..cb771d9 100644
--- a/drivers/acpi/nfit/core.c
+++ b/drivers/acpi/nfit/core.c
@@ -56,6 +56,8 @@ MODULE_PARM_DESC(force_labels, "Opt-in to labels despite missing methods");
LIST_HEAD(acpi_descs);
DEFINE_MUTEX(acpi_desc_lock);
+DEFINE_MUTEX(acpi_notify_lock);
+
static struct workqueue_struct *nfit_wq;
struct nfit_table_prev {
@@ -1680,7 +1682,6 @@ static struct nvdimm *acpi_nfit_dimm_by_handle(struct acpi_nfit_desc *acpi_desc,
void __acpi_nvdimm_notify(struct device *dev, u32 event)
{
struct nfit_mem *nfit_mem;
- struct acpi_nfit_desc *acpi_desc;
dev_dbg(dev->parent, "%s: event: %d\n", dev_name(dev),
event);
@@ -1691,12 +1692,11 @@ void __acpi_nvdimm_notify(struct device *dev, u32 event)
return;
}
- acpi_desc = dev_get_drvdata(dev->parent);
- if (!acpi_desc)
+ if (!dev_get_drvdata(dev->parent))
return;
/*
- * If we successfully retrieved acpi_desc, then we know nfit_mem data
+ * If the parent's driver data pointer is not NULL, then nfit_mem data
* is still valid.
*/
nfit_mem = dev_get_drvdata(dev);
@@ -1710,9 +1710,15 @@ static void acpi_nvdimm_notify(acpi_handle handle, u32 event, void *data)
struct acpi_device *adev = data;
struct device *dev = &adev->dev;
- device_lock(dev->parent);
+ /*
+ * Locking is needed here for synchronization with driver probe and
+ * removal and the parent's driver data pointer is NULL when teardown
+ * is in progress (while the parent here is expected to be the ACPI
+ * companion of the platform device used for driver binding).
+ */
+ guard(mutex)(&acpi_notify_lock);
+
__acpi_nvdimm_notify(dev, event);
- device_unlock(dev->parent);
}
static bool acpi_nvdimm_has_method(struct acpi_device *adev, char *method)
@@ -3069,6 +3075,8 @@ static void acpi_nfit_unregister(void *data)
struct acpi_nfit_desc *acpi_desc = data;
nvdimm_bus_unregister(acpi_desc->nvdimm_bus);
+ /* The nvdimm_bus object may have been freed, so clear the pointer. */
+ acpi_desc->nvdimm_bus = NULL;
}
int acpi_nfit_init(struct acpi_nfit_desc *acpi_desc, void *data, acpi_size sz)
@@ -3156,11 +3164,10 @@ EXPORT_SYMBOL_GPL(acpi_nfit_init);
static int acpi_nfit_flush_probe(struct nvdimm_bus_descriptor *nd_desc)
{
struct acpi_nfit_desc *acpi_desc = to_acpi_desc(nd_desc);
- struct device *dev = acpi_desc->dev;
- /* Bounce the device lock to flush acpi_nfit_add / acpi_nfit_notify */
- device_lock(dev);
- device_unlock(dev);
+ /* Bounce the notify lock to flush acpi_nfit_probe / acpi_nfit_notify */
+ mutex_lock(&acpi_notify_lock);
+ mutex_unlock(&acpi_notify_lock);
/* Bounce the init_mutex to complete initial registration */
mutex_lock(&acpi_desc->init_mutex);
@@ -3292,24 +3299,26 @@ static void acpi_nfit_put_table(void *table)
static void acpi_nfit_notify(acpi_handle handle, u32 event, void *data)
{
struct device *dev = data;
+ struct acpi_device *adev = ACPI_COMPANION(dev);
- device_lock(dev);
- __acpi_nfit_notify(dev, handle, event);
- device_unlock(dev);
-}
+ /*
+ * Locking is needed here for synchronization with driver probe and
+ * removal and the ACPI companion's driver data pointer is NULL when
+ * teardown is in progress.
+ */
+ guard(mutex)(&acpi_notify_lock);
-static void acpi_nfit_remove_notify_handler(void *data)
-{
- struct acpi_device *adev = data;
-
- acpi_dev_remove_notify_handler(adev, ACPI_DEVICE_NOTIFY,
- acpi_nfit_notify);
+ if (dev_get_drvdata(&adev->dev))
+ __acpi_nfit_notify(dev, handle, event);
}
void acpi_nfit_shutdown(void *data)
{
struct acpi_nfit_desc *acpi_desc = data;
- struct device *bus_dev = to_nvdimm_bus_dev(acpi_desc->nvdimm_bus);
+ struct device *bus_dev;
+
+ if (!acpi_desc || !acpi_desc->nvdimm_bus)
+ return;
/*
* Destruct under acpi_desc_lock so that nfit_handle_mce does not
@@ -3324,6 +3333,7 @@ void acpi_nfit_shutdown(void *data)
mutex_unlock(&acpi_desc->init_mutex);
cancel_delayed_work_sync(&acpi_desc->dwork);
+ bus_dev = to_nvdimm_bus_dev(acpi_desc->nvdimm_bus);
/*
* Bounce the nvdimm bus lock to make sure any in-flight
* acpi_nfit_ars_rescan() submissions have had a chance to
@@ -3341,23 +3351,20 @@ static int acpi_nfit_probe(struct platform_device *pdev)
struct acpi_buffer buf = { ACPI_ALLOCATE_BUFFER, NULL };
struct acpi_nfit_desc *acpi_desc;
struct device *dev = &pdev->dev;
+ struct acpi_device *adev = ACPI_COMPANION(dev);
struct acpi_table_header *tbl;
- struct acpi_device *adev;
acpi_status status = AE_OK;
acpi_size sz;
int rc = 0;
- adev = ACPI_COMPANION(&pdev->dev);
- if (!adev)
- return -ENODEV;
+ /*
+ * Prevent acpi_nfit_notify() from progressing until the probe is
+ * complete in case there is a concurrent event to process.
+ */
+ guard(mutex)(&acpi_notify_lock);
- rc = acpi_dev_install_notify_handler(adev, ACPI_DEVICE_NOTIFY,
- acpi_nfit_notify, dev);
- if (rc)
- return rc;
-
- rc = devm_add_action_or_reset(dev, acpi_nfit_remove_notify_handler,
- adev);
+ rc = devm_acpi_install_notify_handler(dev, ACPI_DEVICE_NOTIFY,
+ acpi_nfit_notify, dev);
if (rc)
return rc;
@@ -3371,6 +3378,11 @@ static int acpi_nfit_probe(struct platform_device *pdev)
* data in the format of a series of NFIT Structures.
*/
dev_dbg(dev, "failed to find NFIT at startup\n");
+ /*
+ * Let acpi_nfit_update_notify() run in case it will need to
+ * allocate the acpi_desc object.
+ */
+ dev_set_drvdata(&adev->dev, dev);
return 0;
}
@@ -3405,10 +3417,28 @@ static int acpi_nfit_probe(struct platform_device *pdev)
+ sizeof(struct acpi_table_nfit),
sz - sizeof(struct acpi_table_nfit));
- if (rc)
+ if (rc) {
+ acpi_nfit_shutdown(acpi_desc);
return rc;
+ }
- return devm_add_action_or_reset(dev, acpi_nfit_shutdown, acpi_desc);
+ /*
+ * Let notify handlers operate (the actual value of the ACPI companion's
+ * driver data pointer does not matter here so long as it is not NULL).
+ */
+ dev_set_drvdata(&adev->dev, dev);
+ return 0;
+}
+
+static void acpi_nfit_remove(struct platform_device *pdev)
+{
+ struct acpi_device *adev = ACPI_COMPANION(&pdev->dev);
+
+ guard(mutex)(&acpi_notify_lock);
+
+ /* Make notify handlers bail out early going forward. */
+ dev_set_drvdata(&adev->dev, NULL);
+ acpi_nfit_shutdown(platform_get_drvdata(pdev));
}
static void acpi_nfit_update_notify(struct device *dev, acpi_handle handle)
@@ -3460,6 +3490,9 @@ static void acpi_nfit_uc_error_notify(struct device *dev, acpi_handle handle)
{
struct acpi_nfit_desc *acpi_desc = dev_get_drvdata(dev);
+ if (!acpi_desc)
+ return;
+
if (acpi_desc->scrub_mode == HW_ERROR_SCRUB_ON)
acpi_nfit_ars_rescan(acpi_desc, ARS_REQ_LONG);
else
@@ -3489,6 +3522,7 @@ MODULE_DEVICE_TABLE(acpi, acpi_nfit_ids);
static struct platform_driver acpi_nfit_driver = {
.probe = acpi_nfit_probe,
+ .remove = acpi_nfit_remove,
.driver = {
.name = "acpi-nfit",
.acpi_match_table = acpi_nfit_ids,
diff --git a/drivers/acpi/pci_root.c b/drivers/acpi/pci_root.c
index a0ba64e..4c06c3f 100644
--- a/drivers/acpi/pci_root.c
+++ b/drivers/acpi/pci_root.c
@@ -755,6 +755,10 @@ static int acpi_pci_root_add(struct acpi_device *device,
pci_lock_rescan_remove();
pci_bus_add_devices(root->bus);
pci_unlock_rescan_remove();
+
+ /* Clear _DEP dependencies to allow consumers to enumerate */
+ acpi_dev_clear_dependencies(device);
+
return 1;
remove_dmar:
diff --git a/drivers/acpi/pmic/intel_pmic.c b/drivers/acpi/pmic/intel_pmic.c
index 134e9ca8..9bd46de 100644
--- a/drivers/acpi/pmic/intel_pmic.c
+++ b/drivers/acpi/pmic/intel_pmic.c
@@ -67,14 +67,12 @@ static acpi_status intel_pmic_power_handler(u32 function,
if (result == -ENOENT)
return AE_BAD_PARAMETER;
- mutex_lock(&opregion->lock);
+ guard(mutex)(&opregion->lock);
result = function == ACPI_READ ?
d->get_power(regmap, reg, bit, value64) :
d->update_power(regmap, reg, bit, *value64 == 1);
- mutex_unlock(&opregion->lock);
-
return result ? AE_ERROR : AE_OK;
}
@@ -182,19 +180,16 @@ static acpi_status intel_pmic_thermal_handler(u32 function,
if (result == -ENOENT)
return AE_BAD_PARAMETER;
- mutex_lock(&opregion->lock);
-
- if (pmic_thermal_is_temp(address))
- result = pmic_thermal_temp(opregion, reg, function, value64);
- else if (pmic_thermal_is_aux(address))
- result = pmic_thermal_aux(opregion, reg, function, value64);
- else if (pmic_thermal_is_pen(address))
- result = pmic_thermal_pen(opregion, reg, bit,
- function, value64);
- else
- result = -EINVAL;
-
- mutex_unlock(&opregion->lock);
+ scoped_guard(mutex, &opregion->lock) {
+ if (pmic_thermal_is_temp(address))
+ result = pmic_thermal_temp(opregion, reg, function, value64);
+ else if (pmic_thermal_is_aux(address))
+ result = pmic_thermal_aux(opregion, reg, function, value64);
+ else if (pmic_thermal_is_pen(address))
+ result = pmic_thermal_pen(opregion, reg, bit, function, value64);
+ else
+ result = -EINVAL;
+ }
if (result < 0) {
if (result == -EINVAL)
@@ -354,13 +349,15 @@ int intel_soc_pmic_exec_mipi_pmic_seq_element(u16 i2c_address, u32 reg_address,
d = intel_pmic_opregion->data;
- mutex_lock(&intel_pmic_opregion->lock);
+ guard(mutex)(&intel_pmic_opregion->lock);
if (d->exec_mipi_pmic_seq_element) {
- ret = d->exec_mipi_pmic_seq_element(intel_pmic_opregion->regmap,
+ return d->exec_mipi_pmic_seq_element(intel_pmic_opregion->regmap,
i2c_address, reg_address,
value, mask);
- } else if (d->pmic_i2c_address) {
+ }
+
+ if (d->pmic_i2c_address) {
if (i2c_address == d->pmic_i2c_address) {
ret = regmap_update_bits(intel_pmic_opregion->regmap,
reg_address, mask, value);
@@ -376,8 +373,6 @@ int intel_soc_pmic_exec_mipi_pmic_seq_element(u16 i2c_address, u32 reg_address,
ret = -EOPNOTSUPP;
}
- mutex_unlock(&intel_pmic_opregion->lock);
-
return ret;
}
EXPORT_SYMBOL_GPL(intel_soc_pmic_exec_mipi_pmic_seq_element);
diff --git a/drivers/acpi/processor_idle.c b/drivers/acpi/processor_idle.c
index ee5facc..390ab5f 100644
--- a/drivers/acpi/processor_idle.c
+++ b/drivers/acpi/processor_idle.c
@@ -1355,6 +1355,15 @@ void acpi_processor_register_idle_driver(void)
int ret = -ENODEV;
int cpu;
+ /*
+ * If a cpuidle driver is already registered, there is no need to
+ * evaluate _CST or attempt to register the ACPI idle driver.
+ */
+ if (cpuidle_get_driver()) {
+ pr_debug("cpuidle driver %pS already registered.\n", cpuidle_get_driver());
+ return;
+ }
+
acpi_processor_update_max_cstate();
/*
diff --git a/drivers/acpi/scan.c b/drivers/acpi/scan.c
index 530547c..075798d 100644
--- a/drivers/acpi/scan.c
+++ b/drivers/acpi/scan.c
@@ -848,8 +848,6 @@ static bool acpi_info_matches_ids(struct acpi_device_info *info,
static const char * const acpi_ignore_dep_ids[] = {
"PNP0D80", /* Windows-compatible System Power Management Controller */
"INT33BD", /* Intel Baytrail Mailbox Device */
- "INTC10DE", /* Intel CVS LNL */
- "INTC10E0", /* Intel CVS ARL */
"LATT2021", /* Lattice FW Update Client Driver */
NULL
};
@@ -861,11 +859,15 @@ static const char * const acpi_honor_dep_ids[] = {
"INTC1095", /* IVSC (ADL) driver must be loaded to allow i2c access to camera sensors */
"INTC100A", /* IVSC (RPL) driver must be loaded to allow i2c access to camera sensors */
"INTC10CF", /* IVSC (MTL) driver must be loaded to allow i2c access to camera sensors */
+ "INTC10DE", /* CVS (LNL) driver must be loaded to allow camera streaming */
+ "INTC10E0", /* CVS (ARL) driver must be loaded to allow camera streaming */
+ "INTC10E1", /* CVS (PTL) driver must be loaded to allow camera streaming */
"RSCV0001", /* RISC-V PLIC */
"RSCV0002", /* RISC-V APLIC */
"RSCV0005", /* RISC-V SBI MPXY MBOX */
"RSCV0006", /* RISC-V RPMI SYSMSI */
"PNP0C0F", /* PCI Link Device */
+ "ACPI0016", /* CXL/PCIe host bridge: CXL root (ACPI0017) depends on PCI root attach */
NULL
};
diff --git a/drivers/acpi/thermal.c b/drivers/acpi/thermal.c
index dfc7daa..dd7666c 100644
--- a/drivers/acpi/thermal.c
+++ b/drivers/acpi/thermal.c
@@ -655,8 +655,12 @@ static int acpi_thermal_register_thermal_zone(struct acpi_thermal *tz,
return result;
}
-static void acpi_thermal_unregister_thermal_zone(struct acpi_thermal *tz)
+static void acpi_thermal_zone_unregister(void *data)
{
+ struct acpi_thermal *tz = data;
+
+ flush_workqueue(acpi_thermal_pm_queue);
+
thermal_zone_device_disable(tz->thermal_zone);
acpi_thermal_zone_sysfs_remove(tz);
thermal_zone_device_unregister(tz->thermal_zone);
@@ -765,8 +769,9 @@ static void acpi_thermal_check_fn(struct work_struct *work)
mutex_unlock(&tz->thermal_check_lock);
}
-static void acpi_thermal_free_thermal_zone(struct acpi_thermal *tz)
+static void acpi_thermal_zone_free(void *data)
{
+ struct acpi_thermal *tz = data;
int i;
acpi_handle_list_free(&tz->trips.passive.trip.devices);
@@ -779,7 +784,8 @@ static void acpi_thermal_free_thermal_zone(struct acpi_thermal *tz)
static int acpi_thermal_probe(struct platform_device *pdev)
{
struct thermal_trip trip_table[ACPI_THERMAL_MAX_NR_TRIPS] = { 0 };
- struct acpi_device *device = ACPI_COMPANION(&pdev->dev);
+ struct device *dev = &pdev->dev;
+ struct acpi_device *device = ACPI_COMPANION(dev);
struct acpi_thermal_trip *acpi_trip;
struct thermal_trip *trip;
struct acpi_thermal *tz;
@@ -795,6 +801,10 @@ static int acpi_thermal_probe(struct platform_device *pdev)
if (!tz)
return -ENOMEM;
+ result = devm_add_action_or_reset(dev, acpi_thermal_zone_free, tz);
+ if (result)
+ return result;
+
platform_set_drvdata(pdev, tz);
tz->device = device;
@@ -817,7 +827,7 @@ static int acpi_thermal_probe(struct platform_device *pdev)
/* Get temperature [_TMP] (required). */
result = acpi_thermal_get_temperature(tz);
if (result)
- goto free_memory;
+ return result;
/* Determine the default polling frequency [_TZP]. */
if (tzp)
@@ -870,7 +880,11 @@ static int acpi_thermal_probe(struct platform_device *pdev)
trip - trip_table,
passive_delay);
if (result)
- goto free_memory;
+ return result;
+
+ result = devm_add_action_or_reset(dev, acpi_thermal_zone_unregister, tz);
+ if (result)
+ return result;
refcount_set(&tz->thermal_check_count, 3);
mutex_init(&tz->thermal_check_lock);
@@ -879,32 +893,8 @@ static int acpi_thermal_probe(struct platform_device *pdev)
pr_info("Thermal Zone [%s] (%ld C)\n", acpi_device_bid(device),
deci_kelvin_to_celsius(tz->temp_dk));
- result = acpi_dev_install_notify_handler(device, ACPI_DEVICE_NOTIFY,
- acpi_thermal_notify, tz);
- if (result)
- goto flush_wq;
-
- return 0;
-
-flush_wq:
- flush_workqueue(acpi_thermal_pm_queue);
- acpi_thermal_unregister_thermal_zone(tz);
-free_memory:
- acpi_thermal_free_thermal_zone(tz);
-
- return result;
-}
-
-static void acpi_thermal_remove(struct platform_device *pdev)
-{
- struct acpi_thermal *tz = platform_get_drvdata(pdev);
-
- acpi_dev_remove_notify_handler(tz->device, ACPI_DEVICE_NOTIFY,
- acpi_thermal_notify);
-
- flush_workqueue(acpi_thermal_pm_queue);
- acpi_thermal_unregister_thermal_zone(tz);
- acpi_thermal_free_thermal_zone(tz);
+ return devm_acpi_install_notify_handler(dev, ACPI_DEVICE_NOTIFY,
+ acpi_thermal_notify, tz);
}
#ifdef CONFIG_PM_SLEEP
@@ -937,7 +927,6 @@ MODULE_DEVICE_TABLE(acpi, thermal_device_ids);
static struct platform_driver acpi_thermal_driver = {
.probe = acpi_thermal_probe,
- .remove = acpi_thermal_remove,
.driver = {
.name = "acpi-thermal",
.acpi_match_table = thermal_device_ids,
diff --git a/drivers/clk/qcom/dispcc-sc8280xp.c b/drivers/clk/qcom/dispcc-sc8280xp.c
index e91dfed..acc927c 100644
--- a/drivers/clk/qcom/dispcc-sc8280xp.c
+++ b/drivers/clk/qcom/dispcc-sc8280xp.c
@@ -977,7 +977,7 @@ static struct clk_rcg2 disp0_cc_mdss_mdp_clk_src = {
.name = "disp0_cc_mdss_mdp_clk_src",
.parent_data = disp0_cc_parent_data_5,
.num_parents = ARRAY_SIZE(disp0_cc_parent_data_5),
- .ops = &clk_rcg2_shared_ops,
+ .ops = &clk_rcg2_shared_no_init_park_ops,
},
};
@@ -991,7 +991,7 @@ static struct clk_rcg2 disp1_cc_mdss_mdp_clk_src = {
.name = "disp1_cc_mdss_mdp_clk_src",
.parent_data = disp1_cc_parent_data_5,
.num_parents = ARRAY_SIZE(disp1_cc_parent_data_5),
- .ops = &clk_rcg2_shared_ops,
+ .ops = &clk_rcg2_shared_no_init_park_ops,
},
};
diff --git a/drivers/clk/qcom/dispcc-x1e80100.c b/drivers/clk/qcom/dispcc-x1e80100.c
index aa7fd43..cd45bed 100644
--- a/drivers/clk/qcom/dispcc-x1e80100.c
+++ b/drivers/clk/qcom/dispcc-x1e80100.c
@@ -580,7 +580,7 @@ static struct clk_rcg2 disp_cc_mdss_mdp_clk_src = {
.parent_data = disp_cc_parent_data_6,
.num_parents = ARRAY_SIZE(disp_cc_parent_data_6),
.flags = CLK_SET_RATE_PARENT,
- .ops = &clk_rcg2_shared_ops,
+ .ops = &clk_rcg2_shared_no_init_park_ops,
},
};
diff --git a/drivers/clk/samsung/clk-gs101.c b/drivers/clk/samsung/clk-gs101.c
index d2bcd3a9..b44bb31 100644
--- a/drivers/clk/samsung/clk-gs101.c
+++ b/drivers/clk/samsung/clk-gs101.c
@@ -3921,7 +3921,7 @@ static const unsigned long peric0_clk_regs[] __initconst = {
CLK_CON_DIV_DIV_CLK_PERIC0_USI4_USI,
CLK_CON_DIV_DIV_CLK_PERIC0_USI5_USI,
CLK_CON_DIV_DIV_CLK_PERIC0_USI6_USI,
- CLK_CON_DIV_DIV_CLK_PERIC0_USI6_USI,
+ CLK_CON_DIV_DIV_CLK_PERIC0_USI7_USI,
CLK_CON_DIV_DIV_CLK_PERIC0_USI8_USI,
CLK_CON_BUF_CLKBUF_PERIC0_IP,
CLK_CON_GAT_CLK_BLK_PERIC0_UID_PERIC0_CMU_PERIC0_IPCLKPORT_PCLK,
diff --git a/drivers/firmware/efi/efi.c b/drivers/firmware/efi/efi.c
index 318d1cc9..0327a39 100644
--- a/drivers/firmware/efi/efi.c
+++ b/drivers/firmware/efi/efi.c
@@ -73,7 +73,6 @@ struct mm_struct efi_mm = {
MMAP_LOCK_INITIALIZER(efi_mm)
.page_table_lock = __SPIN_LOCK_UNLOCKED(efi_mm.page_table_lock),
.mmlist = LIST_HEAD_INIT(efi_mm.mmlist),
- .user_ns = &init_user_ns,
#ifdef CONFIG_SCHED_MM_CID
.mm_cid.lock = __RAW_SPIN_LOCK_UNLOCKED(efi_mm.mm_cid.lock),
#endif
diff --git a/drivers/firmware/meson/meson_sm.c b/drivers/firmware/meson/meson_sm.c
index 3ab67aaa..ab9751a 100644
--- a/drivers/firmware/meson/meson_sm.c
+++ b/drivers/firmware/meson/meson_sm.c
@@ -41,12 +41,13 @@ static const struct meson_sm_chip gxbb_chip = {
.cmd_shmem_in_base = 0x82000020,
.cmd_shmem_out_base = 0x82000021,
.cmd = {
- CMD(SM_EFUSE_READ, 0x82000030),
- CMD(SM_EFUSE_WRITE, 0x82000031),
+ CMD(SM_EFUSE_READ, 0x82000030),
+ CMD(SM_EFUSE_WRITE, 0x82000031),
CMD(SM_EFUSE_USER_MAX, 0x82000033),
- CMD(SM_GET_CHIP_ID, 0x82000044),
- CMD(SM_A1_PWRC_SET, 0x82000093),
- CMD(SM_A1_PWRC_GET, 0x82000095),
+ CMD(SM_GET_CHIP_ID, 0x82000044),
+ CMD(SM_THERMAL_CALIB_READ, 0x82000047),
+ CMD(SM_A1_PWRC_SET, 0x82000093),
+ CMD(SM_A1_PWRC_GET, 0x82000095),
{ /* sentinel */ },
},
};
@@ -245,6 +246,23 @@ struct meson_sm_firmware *meson_sm_get(struct device_node *sm_node)
}
EXPORT_SYMBOL_GPL(meson_sm_get);
+/**
+ * meson_sm_get_thermal_calib - Read thermal sensor calibration data.
+ * @fw: Pointer to secure-monitor firmware.
+ * @trim_info: Pointer to store the returned calibration data.
+ * @tsensor_id: Sensor index to identify which sensor's calibration data
+ * to retrieve
+ *
+ * Return: 0 on success, negative error code on failure.
+ */
+int meson_sm_get_thermal_calib(struct meson_sm_firmware *fw, u32 *trim_info,
+ u32 tsensor_id)
+{
+ return meson_sm_call(fw, SM_THERMAL_CALIB_READ, trim_info, tsensor_id,
+ 0, 0, 0, 0);
+}
+EXPORT_SYMBOL_GPL(meson_sm_get_thermal_calib);
+
#define SM_CHIP_ID_LENGTH 119
#define SM_CHIP_ID_OFFSET 4
#define SM_CHIP_ID_SIZE 12
diff --git a/drivers/firmware/stratix10-rsu.c b/drivers/firmware/stratix10-rsu.c
index e191210..2a7a0f7 100644
--- a/drivers/firmware/stratix10-rsu.c
+++ b/drivers/firmware/stratix10-rsu.c
@@ -723,15 +723,9 @@ static int stratix10_rsu_probe(struct platform_device *pdev)
return -ENOMEM;
priv->client.dev = dev;
- priv->client.receive_cb = NULL;
priv->client.priv = priv;
- priv->status.current_image = 0;
- priv->status.fail_image = 0;
- priv->status.error_location = 0;
- priv->status.error_details = 0;
- priv->status.version = 0;
- priv->status.state = 0;
priv->retry_counter = INVALID_RETRY_COUNTER;
+ priv->max_retry = INVALID_RETRY_COUNTER;
priv->dcmf_version.dcmf0 = INVALID_DCMF_VERSION;
priv->dcmf_version.dcmf1 = INVALID_DCMF_VERSION;
priv->dcmf_version.dcmf2 = INVALID_DCMF_VERSION;
@@ -740,11 +734,11 @@ static int stratix10_rsu_probe(struct platform_device *pdev)
priv->dcmf_status.dcmf1 = INVALID_DCMF_STATUS;
priv->dcmf_status.dcmf2 = INVALID_DCMF_STATUS;
priv->dcmf_status.dcmf3 = INVALID_DCMF_STATUS;
- priv->max_retry = INVALID_RETRY_COUNTER;
- priv->spt0_address = INVALID_SPT_ADDRESS;
- priv->spt1_address = INVALID_SPT_ADDRESS;
+ /* spt0/1_address and status fields default to 0 from kzalloc */
mutex_init(&priv->lock);
+ init_completion(&priv->completion);
+
priv->chan = stratix10_svc_request_channel_byname(&priv->client,
SVC_CLIENT_RSU);
if (IS_ERR(priv->chan)) {
@@ -756,11 +750,9 @@ static int stratix10_rsu_probe(struct platform_device *pdev)
ret = stratix10_svc_add_async_client(priv->chan, false);
if (ret) {
dev_err(dev, "failed to add async client\n");
- stratix10_svc_free_channel(priv->chan);
- return ret;
+ goto free_channel;
}
- init_completion(&priv->completion);
platform_set_drvdata(pdev, priv);
/* get the initial state from firmware */
@@ -768,41 +760,44 @@ static int stratix10_rsu_probe(struct platform_device *pdev)
rsu_async_status_callback);
if (ret) {
dev_err(dev, "Error, getting RSU status %i\n", ret);
- stratix10_svc_remove_async_client(priv->chan);
- stratix10_svc_free_channel(priv->chan);
- return ret;
+ goto remove_async_client;
}
/* get DCMF version from firmware */
- ret = rsu_send_msg(priv, COMMAND_RSU_DCMF_VERSION,
- 0, rsu_dcmf_version_callback);
+ ret = rsu_send_msg(priv, COMMAND_RSU_DCMF_VERSION, 0,
+ rsu_dcmf_version_callback);
if (ret) {
dev_err(dev, "Error, getting DCMF version %i\n", ret);
- stratix10_svc_free_channel(priv->chan);
+ goto remove_async_client;
}
- ret = rsu_send_msg(priv, COMMAND_RSU_DCMF_STATUS,
- 0, rsu_dcmf_status_callback);
+ ret = rsu_send_msg(priv, COMMAND_RSU_DCMF_STATUS, 0,
+ rsu_dcmf_status_callback);
if (ret) {
dev_err(dev, "Error, getting DCMF status %i\n", ret);
- stratix10_svc_free_channel(priv->chan);
+ goto remove_async_client;
}
ret = rsu_send_msg(priv, COMMAND_RSU_MAX_RETRY, 0,
rsu_max_retry_callback);
if (ret) {
dev_err(dev, "Error, getting RSU max retry %i\n", ret);
- stratix10_svc_free_channel(priv->chan);
+ goto remove_async_client;
}
-
ret = rsu_send_async_msg(dev, priv, COMMAND_RSU_GET_SPT_TABLE, 0,
rsu_async_get_spt_table_callback);
if (ret) {
dev_err(dev, "Error, getting SPT table %i\n", ret);
- stratix10_svc_free_channel(priv->chan);
+ goto remove_async_client;
}
+ return 0;
+
+remove_async_client:
+ stratix10_svc_remove_async_client(priv->chan);
+free_channel:
+ stratix10_svc_free_channel(priv->chan);
return ret;
}
diff --git a/drivers/firmware/stratix10-svc.c b/drivers/firmware/stratix10-svc.c
index e9e35d6..39eb78f 100644
--- a/drivers/firmware/stratix10-svc.c
+++ b/drivers/firmware/stratix10-svc.c
@@ -212,6 +212,7 @@ struct stratix10_async_chan {
/**
* struct stratix10_async_ctrl - Control structure for Stratix10
* asynchronous operations
+ * @supported: Flag indicating whether the system supports async operations
* @initialized: Flag indicating whether the control structure has
* been initialized
* @invoke_fn: Function pointer for invoking Stratix10 service calls
@@ -228,6 +229,7 @@ struct stratix10_async_chan {
*/
struct stratix10_async_ctrl {
+ bool supported;
bool initialized;
void (*invoke_fn)(struct stratix10_async_ctrl *actrl,
const struct arm_smccc_1_2_regs *args,
@@ -1103,6 +1105,7 @@ EXPORT_SYMBOL_GPL(stratix10_svc_request_channel_byname);
* Return: 0 on success, or a negative error code on failure:
* -EINVAL if the channel is NULL or the async controller is
* not initialized.
+ * -EOPNOTSUPP if async operations are not supported.
* -EALREADY if the async channel is already allocated.
* -ENOMEM if memory allocation fails.
* Other negative values if ID allocation fails.
@@ -1121,6 +1124,9 @@ int stratix10_svc_add_async_client(struct stratix10_svc_chan *chan,
ctrl = chan->ctrl;
actrl = &ctrl->actrl;
+ if (!actrl->supported)
+ return -EOPNOTSUPP;
+
if (!actrl->initialized) {
dev_err(ctrl->dev, "Async controller not initialized\n");
return -EINVAL;
@@ -1562,6 +1568,7 @@ static inline void stratix10_smc_1_2(struct stratix10_async_ctrl *actrl,
* initialized, -ENOMEM if memory allocation fails,
* -EADDRINUSE if the client ID is already reserved, or other
* negative error codes on failure.
+ * -EOPNOTSUPP if system doesn't support async operations.
*/
static int stratix10_svc_async_init(struct stratix10_svc_controller *controller)
{
@@ -1585,10 +1592,12 @@ static int stratix10_svc_async_init(struct stratix10_svc_controller *controller)
!(res.a1 > ASYNC_ATF_MINIMUM_MAJOR_VERSION ||
(res.a1 == ASYNC_ATF_MINIMUM_MAJOR_VERSION &&
res.a2 >= ASYNC_ATF_MINIMUM_MINOR_VERSION))) {
- dev_err(dev,
- "Intel Service Layer Driver: ATF version is not compatible for async operation\n");
- return -EINVAL;
+ dev_info(dev,
+ "Intel Service Layer Driver: ATF version is not compatible for async operation\n");
+ actrl->supported = false;
+ return -EOPNOTSUPP;
}
+ actrl->supported = true;
actrl->invoke_fn = stratix10_smc_1_2;
@@ -1952,10 +1961,14 @@ static int stratix10_svc_drv_probe(struct platform_device *pdev)
init_completion(&controller->complete_status);
ret = stratix10_svc_async_init(controller);
- if (ret) {
+ if (ret == -EOPNOTSUPP) {
+ dev_info(dev, "Intel Service Layer Driver Initialized (sync-only mode)\n");
+ } else if (ret) {
dev_dbg(dev, "Intel Service Layer Driver: Error on stratix10_svc_async_init %d\n",
ret);
goto err_destroy_pool;
+ } else {
+ dev_info(dev, "Intel Service Layer Driver Initialized\n");
}
fifo_size = sizeof(struct stratix10_svc_data) * SVC_NUM_DATA_IN_FIFO;
diff --git a/drivers/gpio/gpio-mockup.c b/drivers/gpio/gpio-mockup.c
index a7d69f3..91ff789 100644
--- a/drivers/gpio/gpio-mockup.c
+++ b/drivers/gpio/gpio-mockup.c
@@ -17,6 +17,7 @@
#include <linux/irq.h>
#include <linux/irq_sim.h>
#include <linux/irqdomain.h>
+#include <linux/limits.h>
#include <linux/mod_devicetable.h>
#include <linux/module.h>
#include <linux/platform_device.h>
@@ -578,7 +579,7 @@ static int __init gpio_mockup_register_chip(int idx)
static int __init gpio_mockup_init(void)
{
- int i, num_chips, err;
+ int i, num_chips, err, base, ngpio;
if ((gpio_mockup_num_ranges % 2) ||
(gpio_mockup_num_ranges > GPIO_MOCKUP_MAX_RANGES))
@@ -592,8 +593,19 @@ static int __init gpio_mockup_init(void)
* always be greater than 0.
*/
for (i = 0; i < num_chips; i++) {
- if (gpio_mockup_range_ngpio(i) < 0)
+ base = gpio_mockup_range_base(i);
+ ngpio = gpio_mockup_range_ngpio(i);
+
+ if (ngpio <= 0)
return -EINVAL;
+
+ if (base < 0) {
+ if (ngpio > U16_MAX)
+ return -EINVAL;
+ } else {
+ if (ngpio <= base || ngpio - base > U16_MAX)
+ return -EINVAL;
+ }
}
gpio_mockup_dbg_dir = debugfs_create_dir("gpio-mockup", NULL);
diff --git a/drivers/gpio/gpio-mvebu.c b/drivers/gpio/gpio-mvebu.c
index 22c36b7..c030d1f 100644
--- a/drivers/gpio/gpio-mvebu.c
+++ b/drivers/gpio/gpio-mvebu.c
@@ -996,7 +996,7 @@ static int mvebu_gpio_suspend(struct platform_device *pdev, pm_message_t state)
BUG();
}
- if (IS_REACHABLE(CONFIG_PWM))
+ if (IS_REACHABLE(CONFIG_PWM) && mvchip->mvpwm)
mvebu_pwm_suspend(mvchip);
return 0;
@@ -1048,7 +1048,7 @@ static int mvebu_gpio_resume(struct platform_device *pdev)
BUG();
}
- if (IS_REACHABLE(CONFIG_PWM))
+ if (IS_REACHABLE(CONFIG_PWM) && mvchip->mvpwm)
mvebu_pwm_resume(mvchip);
return 0;
diff --git a/drivers/gpio/gpio-rockchip.c b/drivers/gpio/gpio-rockchip.c
index bc97d5d..9478a58 100644
--- a/drivers/gpio/gpio-rockchip.c
+++ b/drivers/gpio/gpio-rockchip.c
@@ -802,8 +802,10 @@ static void rockchip_gpio_remove(struct platform_device *pdev)
struct rockchip_pin_bank *bank = platform_get_drvdata(pdev);
irq_set_chained_handler_and_data(bank->irq, NULL, NULL);
- if (bank->domain)
+ if (bank->domain) {
+ irq_domain_remove_generic_chips(bank->domain);
irq_domain_remove(bank->domain);
+ }
gpiochip_remove(&bank->gpio_chip);
}
diff --git a/drivers/gpio/gpio-zynq.c b/drivers/gpio/gpio-zynq.c
index 571e366..fafca91 100644
--- a/drivers/gpio/gpio-zynq.c
+++ b/drivers/gpio/gpio-zynq.c
@@ -1014,6 +1014,7 @@ static void zynq_gpio_remove(struct platform_device *pdev)
gpiochip_remove(&gpio->chip);
device_set_wakeup_capable(&pdev->dev, 0);
pm_runtime_disable(&pdev->dev);
+ pm_runtime_put_noidle(&pdev->dev);
}
static struct platform_driver zynq_gpio_driver = {
diff --git a/drivers/gpio/gpiolib-of.c b/drivers/gpio/gpiolib-of.c
index 2c923d1..813dbcb 100644
--- a/drivers/gpio/gpiolib-of.c
+++ b/drivers/gpio/gpiolib-of.c
@@ -1066,11 +1066,6 @@ int of_gpiochip_add(struct gpio_chip *chip)
of_node_get(np);
- for_each_available_child_of_node_scoped(np, child) {
- if (of_property_read_bool(child, "gpio-hog"))
- of_node_set_flag(child, OF_POPULATED);
- }
-
return ret;
}
diff --git a/drivers/gpio/gpiolib.c b/drivers/gpio/gpiolib.c
index 1e6dce4..c1f9c0d 100644
--- a/drivers/gpio/gpiolib.c
+++ b/drivers/gpio/gpiolib.c
@@ -1031,9 +1031,17 @@ static int gpiochip_hog_lines(struct gpio_chip *gc)
if (!fwnode_property_present(fwnode, "gpio-hog"))
continue;
+ /* The hog may have been handled by another gpio_chip on the same fwnode */
+ if (is_of_node(fwnode) &&
+ of_node_check_flag(to_of_node(fwnode), OF_POPULATED))
+ continue;
+
ret = gpiochip_add_hog(gc, fwnode);
if (ret)
return ret;
+
+ if (is_of_node(fwnode))
+ of_node_set_flag(to_of_node(fwnode), OF_POPULATED);
}
return 0;
@@ -1291,7 +1299,7 @@ int gpiochip_add_data_with_key(struct gpio_chip *gc, void *data,
ret = gpiochip_hog_lines(gc);
if (ret)
- goto err_remove_of_chip;
+ goto err_free_hogs;
ret = gpiochip_irqchip_init_valid_mask(gc);
if (ret)
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c
index b24d5d2..583fd67 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c
@@ -1277,6 +1277,7 @@ static int amdgpu_cs_submit(struct amdgpu_cs_parser *p,
{
struct amdgpu_fpriv *fpriv = p->filp->driver_priv;
struct amdgpu_job *leader = p->gang_leader;
+ struct amdgpu_vm *vm = &fpriv->vm;
struct amdgpu_bo_list_entry *e;
struct drm_gem_object *gobj;
unsigned long index;
@@ -1322,7 +1323,8 @@ static int amdgpu_cs_submit(struct amdgpu_cs_parser *p,
amdgpu_hmm_range_free(e->range);
e->range = NULL;
}
- if (r) {
+
+ if (r || !list_empty(&vm->invalidated)) {
r = -EAGAIN;
mutex_unlock(&p->adev->notifier_lock);
return r;
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_userq_fence.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_userq_fence.c
index a41fb72..f74ad37 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_userq_fence.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_userq_fence.c
@@ -593,7 +593,7 @@ int amdgpu_userq_signal_ioctl(struct drm_device *dev, void *data,
static int
amdgpu_userq_wait_count_fences(struct drm_file *filp,
struct drm_amdgpu_userq_wait *wait_info,
- u32 *syncobj_handles, u32 *timeline_points,
+ u32 *syncobj_handles, u64 *timeline_points,
u32 *timeline_handles,
struct drm_gem_object **gobj_write,
struct drm_gem_object **gobj_read)
@@ -703,7 +703,7 @@ amdgpu_userq_wait_add_fence(struct drm_amdgpu_userq_wait *wait_info,
static int
amdgpu_userq_wait_return_fence_info(struct drm_file *filp,
struct drm_amdgpu_userq_wait *wait_info,
- u32 *syncobj_handles, u32 *timeline_points,
+ u32 *syncobj_handles, u64 *timeline_points,
u32 *timeline_handles,
struct drm_gem_object **gobj_write,
struct drm_gem_object **gobj_read)
@@ -906,7 +906,8 @@ int amdgpu_userq_wait_ioctl(struct drm_device *dev, void *data,
struct drm_file *filp)
{
int num_points, num_syncobj, num_read_bo_handles, num_write_bo_handles;
- u32 *syncobj_handles, *timeline_points, *timeline_handles;
+ u32 *syncobj_handles, *timeline_handles;
+ u64 *timeline_points;
struct drm_amdgpu_userq_wait *wait_info = data;
struct drm_gem_object **gobj_write;
struct drm_gem_object **gobj_read;
@@ -935,7 +936,7 @@ int amdgpu_userq_wait_ioctl(struct drm_device *dev, void *data,
}
ptr = u64_to_user_ptr(wait_info->syncobj_timeline_points);
- timeline_points = memdup_array_user(ptr, num_points, sizeof(u32));
+ timeline_points = memdup_array_user(ptr, num_points, sizeof(u64));
if (IS_ERR(timeline_points)) {
r = PTR_ERR(timeline_points);
goto free_timeline_handles;
diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_debug.c b/drivers/gpu/drm/amd/amdkfd/kfd_debug.c
index 0f7aa51..0dd1fd4 100644
--- a/drivers/gpu/drm/amd/amdkfd/kfd_debug.c
+++ b/drivers/gpu/drm/amd/amdkfd/kfd_debug.c
@@ -832,6 +832,12 @@ int kfd_dbg_trap_enable(struct kfd_process *target, uint32_t fd,
if (copy_to_user(runtime_info, (void *)&target->runtime_info, copy_size)) {
kfd_dbg_trap_deactivate(target, false, 0);
+ fput(target->dbg_ev_file);
+ target->dbg_ev_file = NULL;
+ if (target->debugger_process)
+ atomic_dec(&target->debugger_process->debugged_process_count);
+ target->debug_trap_enabled = false;
+ kfd_unref_process(target);
r = -EFAULT;
}
diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_events.c b/drivers/gpu/drm/amd/amdkfd/kfd_events.c
index 44150a7..e65b323 100644
--- a/drivers/gpu/drm/amd/amdkfd/kfd_events.c
+++ b/drivers/gpu/drm/amd/amdkfd/kfd_events.c
@@ -795,6 +795,8 @@ static struct kfd_event_waiter *alloc_event_waiters(uint32_t num_events)
struct kfd_event_waiter *event_waiters;
uint32_t i;
+ if (num_events > KFD_SIGNAL_EVENT_LIMIT)
+ return NULL;
event_waiters = kzalloc_objs(struct kfd_event_waiter, num_events);
if (!event_waiters)
return NULL;
diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_smi_events.c b/drivers/gpu/drm/amd/amdkfd/kfd_smi_events.c
index 15975c2..dfbde5a 100644
--- a/drivers/gpu/drm/amd/amdkfd/kfd_smi_events.c
+++ b/drivers/gpu/drm/amd/amdkfd/kfd_smi_events.c
@@ -254,8 +254,10 @@ void kfd_smi_event_update_vmfault(struct kfd_node *dev, uint16_t pasid)
if (task_info) {
/* Report VM faults from user applications, not retry from kernel */
if (task_info->task.pid)
- kfd_smi_event_add(0, dev, KFD_SMI_EVENT_VMFAULT, KFD_EVENT_FMT_VMFAULT(
- task_info->task.pid, task_info->task.comm));
+ kfd_smi_event_add(task_info->tgid, dev,
+ KFD_SMI_EVENT_VMFAULT,
+ KFD_EVENT_FMT_VMFAULT(task_info->task.pid,
+ task_info->task.comm));
amdgpu_vm_put_task_info(task_info);
}
}
@@ -356,7 +358,7 @@ void kfd_smi_event_process(struct kfd_process_device *pdd, bool start)
task_info = amdgpu_vm_get_task_info_vm(avm);
if (task_info) {
- kfd_smi_event_add(0, pdd->dev,
+ kfd_smi_event_add(task_info->tgid, pdd->dev,
start ? KFD_SMI_EVENT_PROCESS_START :
KFD_SMI_EVENT_PROCESS_END,
KFD_EVENT_FMT_PROCESS(task_info->task.pid,
diff --git a/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm.c b/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm.c
index 5fc5d56..f8c13ba 100644
--- a/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm.c
+++ b/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm.c
@@ -10083,7 +10083,7 @@ static void amdgpu_dm_commit_planes(struct drm_atomic_state *state,
continue;
bundle->surface_updates[planes_count].surface = dc_plane;
- if (new_pcrtc_state->color_mgmt_changed) {
+ if (new_pcrtc_state->color_mgmt_changed || new_plane_state->color_mgmt_changed) {
bundle->surface_updates[planes_count].gamma = &dc_plane->gamma_correction;
bundle->surface_updates[planes_count].in_transfer_func = &dc_plane->in_transfer_func;
bundle->surface_updates[planes_count].gamut_remap_matrix = &dc_plane->gamut_remap_matrix;
@@ -11824,6 +11824,10 @@ static bool should_reset_plane(struct drm_atomic_state *state,
if (new_crtc_state->color_mgmt_changed)
return true;
+ /* Plane color pipeline or its colorop changes. */
+ if (new_plane_state->color_mgmt_changed)
+ return true;
+
/*
* On zpos change, planes need to be reordered by removing and re-adding
* them one by one to the dc state, in order of descending zpos.
@@ -13446,17 +13450,15 @@ void amdgpu_dm_update_freesync_caps(struct drm_connector *connector,
}
/* Handle MCCS */
- if (do_mccs)
+ if (do_mccs) {
dm_helpers_read_mccs_caps(adev->dm.dc->ctx, amdgpu_dm_connector->dc_link, sink);
- if ((sink->sink_signal == SIGNAL_TYPE_HDMI_TYPE_A ||
- as_type == FREESYNC_TYPE_PCON_IN_WHITELIST) &&
- (!sink->edid_caps.freesync_vcp_code ||
- (sink->edid_caps.freesync_vcp_code && !sink->mccs_caps.freesync_supported)))
- freesync_capable = false;
+ if (sink->edid_caps.freesync_vcp_code && !sink->mccs_caps.freesync_supported)
+ freesync_capable = false;
- if (do_mccs && sink->mccs_caps.freesync_supported && freesync_capable)
- dm_helpers_mccs_vcp_set(adev->dm.dc->ctx, amdgpu_dm_connector->dc_link, sink);
+ if (sink->mccs_caps.freesync_supported && freesync_capable)
+ dm_helpers_mccs_vcp_set(adev->dm.dc->ctx, amdgpu_dm_connector->dc_link, sink);
+ }
update:
if (dm_con_state)
diff --git a/drivers/gpu/drm/drm_atomic.c b/drivers/gpu/drm/drm_atomic.c
index 41c5706..0eb52d1 100644
--- a/drivers/gpu/drm/drm_atomic.c
+++ b/drivers/gpu/drm/drm_atomic.c
@@ -830,7 +830,7 @@ static void drm_atomic_colorop_print_state(struct drm_printer *p,
case DRM_COLOROP_1D_LUT:
drm_printf(p, "\tsize=%d\n", colorop->size);
drm_printf(p, "\tinterpolation=%s\n",
- drm_get_colorop_lut1d_interpolation_name(colorop->lut1d_interpolation));
+ drm_get_colorop_lut1d_interpolation_name(state->lut1d_interpolation));
drm_printf(p, "\tdata blob id=%d\n", state->data ? state->data->base.id : 0);
break;
case DRM_COLOROP_CTM_3X4:
@@ -842,7 +842,7 @@ static void drm_atomic_colorop_print_state(struct drm_printer *p,
case DRM_COLOROP_3D_LUT:
drm_printf(p, "\tsize=%d\n", colorop->size);
drm_printf(p, "\tinterpolation=%s\n",
- drm_get_colorop_lut3d_interpolation_name(colorop->lut3d_interpolation));
+ drm_get_colorop_lut3d_interpolation_name(state->lut3d_interpolation));
drm_printf(p, "\tdata blob id=%d\n", state->data ? state->data->base.id : 0);
break;
default:
diff --git a/drivers/gpu/drm/drm_atomic_uapi.c b/drivers/gpu/drm/drm_atomic_uapi.c
index 5bd5bf6..5eaf0e8 100644
--- a/drivers/gpu/drm/drm_atomic_uapi.c
+++ b/drivers/gpu/drm/drm_atomic_uapi.c
@@ -265,13 +265,19 @@ EXPORT_SYMBOL(drm_atomic_set_fb_for_plane);
*
* Helper function to select the color pipeline on a plane by setting
* it to the first drm_colorop element of the pipeline.
+ *
+ * Return: true if plane color pipeline value changed, false otherwise.
*/
-void
+bool
drm_atomic_set_colorop_for_plane(struct drm_plane_state *plane_state,
struct drm_colorop *colorop)
{
struct drm_plane *plane = plane_state->plane;
+ /* Color pipeline didn't change */
+ if (plane_state->color_pipeline == colorop)
+ return false;
+
if (colorop)
drm_dbg_atomic(plane->dev,
"Set [COLOROP:%d] for [PLANE:%d:%s] state %p\n",
@@ -283,6 +289,8 @@ drm_atomic_set_colorop_for_plane(struct drm_plane_state *plane_state,
plane->base.id, plane->name, plane_state);
plane_state->color_pipeline = colorop;
+
+ return true;
}
EXPORT_SYMBOL(drm_atomic_set_colorop_for_plane);
@@ -604,7 +612,7 @@ static int drm_atomic_plane_set_property(struct drm_plane *plane,
if (val && !colorop)
return -EACCES;
- drm_atomic_set_colorop_for_plane(state, colorop);
+ state->color_mgmt_changed |= drm_atomic_set_colorop_for_plane(state, colorop);
} else if (property == config->prop_fb_damage_clips) {
ret = drm_property_replace_blob_from_id(dev,
&state->fb_damage_clips,
@@ -713,11 +721,11 @@ drm_atomic_plane_get_property(struct drm_plane *plane,
static int drm_atomic_color_set_data_property(struct drm_colorop *colorop,
struct drm_colorop_state *state,
struct drm_property *property,
- uint64_t val)
+ uint64_t val,
+ bool *replaced)
{
ssize_t elem_size = -1;
ssize_t size = -1;
- bool replaced = false;
switch (colorop->type) {
case DRM_COLOROP_1D_LUT:
@@ -739,28 +747,45 @@ static int drm_atomic_color_set_data_property(struct drm_colorop *colorop,
&state->data,
val,
-1, size, elem_size,
- &replaced);
+ replaced);
}
static int drm_atomic_colorop_set_property(struct drm_colorop *colorop,
struct drm_colorop_state *state,
struct drm_file *file_priv,
struct drm_property *property,
- uint64_t val)
+ uint64_t val,
+ bool *replaced)
{
if (property == colorop->bypass_property) {
- state->bypass = val;
+ if (state->bypass != val) {
+ state->bypass = val;
+ *replaced = true;
+ }
} else if (property == colorop->lut1d_interpolation_property) {
- colorop->lut1d_interpolation = val;
+ if (state->lut1d_interpolation != val) {
+ state->lut1d_interpolation = val;
+ *replaced = true;
+ }
} else if (property == colorop->curve_1d_type_property) {
- state->curve_1d_type = val;
+ if (state->curve_1d_type != val) {
+ state->curve_1d_type = val;
+ *replaced = true;
+ }
} else if (property == colorop->multiplier_property) {
- state->multiplier = val;
+ if (state->multiplier != val) {
+ state->multiplier = val;
+ *replaced = true;
+ }
} else if (property == colorop->lut3d_interpolation_property) {
- colorop->lut3d_interpolation = val;
+ if (state->lut3d_interpolation != val) {
+ state->lut3d_interpolation = val;
+ *replaced = true;
+ }
} else if (property == colorop->data_property) {
return drm_atomic_color_set_data_property(colorop, state,
- property, val);
+ property, val,
+ replaced);
} else {
drm_dbg_atomic(colorop->dev,
"[COLOROP:%d:%d] unknown property [PROP:%d:%s]\n",
@@ -782,7 +807,7 @@ drm_atomic_colorop_get_property(struct drm_colorop *colorop,
else if (property == colorop->bypass_property)
*val = state->bypass;
else if (property == colorop->lut1d_interpolation_property)
- *val = colorop->lut1d_interpolation;
+ *val = state->lut1d_interpolation;
else if (property == colorop->curve_1d_type_property)
*val = state->curve_1d_type;
else if (property == colorop->multiplier_property)
@@ -790,7 +815,7 @@ drm_atomic_colorop_get_property(struct drm_colorop *colorop,
else if (property == colorop->size_property)
*val = colorop->size;
else if (property == colorop->lut3d_interpolation_property)
- *val = colorop->lut3d_interpolation;
+ *val = state->lut3d_interpolation;
else if (property == colorop->data_property)
*val = (state->data) ? state->data->base.id : 0;
else
@@ -1275,8 +1300,10 @@ int drm_atomic_set_property(struct drm_atomic_state *state,
break;
}
case DRM_MODE_OBJECT_COLOROP: {
+ struct drm_plane_state *plane_state;
struct drm_colorop *colorop = obj_to_colorop(obj);
struct drm_colorop_state *colorop_state;
+ bool replaced = false;
colorop_state = drm_atomic_get_colorop_state(state, colorop);
if (IS_ERR(colorop_state)) {
@@ -1285,7 +1312,18 @@ int drm_atomic_set_property(struct drm_atomic_state *state,
}
ret = drm_atomic_colorop_set_property(colorop, colorop_state,
- file_priv, prop, prop_value);
+ file_priv, prop, prop_value,
+ &replaced);
+ if (ret || !replaced)
+ break;
+
+ plane_state = drm_atomic_get_plane_state(state, colorop->plane);
+ if (IS_ERR(plane_state)) {
+ ret = PTR_ERR(plane_state);
+ break;
+ }
+ plane_state->color_mgmt_changed |= replaced;
+
break;
}
default:
diff --git a/drivers/gpu/drm/drm_colorop.c b/drivers/gpu/drm/drm_colorop.c
index 566816e..509678e 100644
--- a/drivers/gpu/drm/drm_colorop.c
+++ b/drivers/gpu/drm/drm_colorop.c
@@ -342,7 +342,6 @@ int drm_plane_colorop_curve_1d_lut_init(struct drm_device *dev, struct drm_color
colorop->lut1d_interpolation_property = prop;
drm_object_attach_property(&colorop->base, prop, interpolation);
- colorop->lut1d_interpolation = interpolation;
/* data */
ret = drm_colorop_create_data_prop(dev, colorop);
@@ -442,7 +441,6 @@ int drm_plane_colorop_3dlut_init(struct drm_device *dev, struct drm_colorop *col
colorop->lut3d_interpolation_property = prop;
drm_object_attach_property(&colorop->base, prop, interpolation);
- colorop->lut3d_interpolation = interpolation;
/* data */
ret = drm_colorop_create_data_prop(dev, colorop);
@@ -521,6 +519,20 @@ static void __drm_colorop_state_reset(struct drm_colorop_state *colorop_state,
&val);
colorop_state->curve_1d_type = val;
}
+
+ if (colorop->lut1d_interpolation_property) {
+ if (!drm_object_property_get_default_value(&colorop->base,
+ colorop->lut1d_interpolation_property,
+ &val))
+ colorop_state->lut1d_interpolation = val;
+ }
+
+ if (colorop->lut3d_interpolation_property) {
+ if (!drm_object_property_get_default_value(&colorop->base,
+ colorop->lut3d_interpolation_property,
+ &val))
+ colorop_state->lut3d_interpolation = val;
+ }
}
/**
diff --git a/drivers/gpu/drm/etnaviv/etnaviv_gpu.c b/drivers/gpu/drm/etnaviv/etnaviv_gpu.c
index a891d4f..552631c 100644
--- a/drivers/gpu/drm/etnaviv/etnaviv_gpu.c
+++ b/drivers/gpu/drm/etnaviv/etnaviv_gpu.c
@@ -1791,8 +1791,9 @@ static int etnaviv_gpu_bind(struct device *dev, struct device *master,
int ret;
if (IS_ENABLED(CONFIG_DRM_ETNAVIV_THERMAL)) {
- gpu->cooling = thermal_of_cooling_device_register(dev->of_node,
- (char *)dev_name(dev), gpu, &cooling_ops);
+ gpu->cooling = thermal_of_cooling_device_register(dev->of_node, 0,
+ dev_name(dev),
+ gpu, &cooling_ops);
if (IS_ERR(gpu->cooling))
return PTR_ERR(gpu->cooling);
}
diff --git a/drivers/gpu/drm/i915/display/intel_dp.c b/drivers/gpu/drm/i915/display/intel_dp.c
index 6ef2a00..5c3e816 100644
--- a/drivers/gpu/drm/i915/display/intel_dp.c
+++ b/drivers/gpu/drm/i915/display/intel_dp.c
@@ -4678,10 +4678,17 @@ intel_edp_set_sink_rates(struct intel_dp *intel_dp)
if (intel_dp->edp_dpcd[0] >= DP_EDP_14) {
__le16 sink_rates[DP_MAX_SUPPORTED_RATES];
+ int ret;
int i;
- drm_dp_dpcd_read(&intel_dp->aux, DP_SUPPORTED_LINK_RATES,
- sink_rates, sizeof(sink_rates));
+ ret = drm_dp_dpcd_read_data(&intel_dp->aux,
+ DP_SUPPORTED_LINK_RATES,
+ sink_rates, sizeof(sink_rates));
+ if (ret < 0) {
+ drm_dbg_kms(display->drm,
+ "Unable to read eDP supported link rates, using default rates\n");
+ memset(sink_rates, 0, sizeof(sink_rates));
+ }
for (i = 0; i < ARRAY_SIZE(sink_rates); i++) {
int rate;
diff --git a/drivers/gpu/drm/i915/gem/i915_gem_phys.c b/drivers/gpu/drm/i915/gem/i915_gem_phys.c
index e375afb..d53129e 100644
--- a/drivers/gpu/drm/i915/gem/i915_gem_phys.c
+++ b/drivers/gpu/drm/i915/gem/i915_gem_phys.c
@@ -18,6 +18,17 @@
#include "i915_gem_tiling.h"
#include "i915_scatterlist.h"
+/* Abuse scatterlist to store pointer instead of struct page. */
+static inline void __set_phys_vaddr(struct scatterlist *sg, void *vaddr)
+{
+ sg_assign_page(sg, (struct page *)vaddr);
+}
+
+static inline void *__get_phys_vaddr(struct scatterlist *sg)
+{
+ return (void *)sg_page(sg);
+}
+
static int i915_gem_object_get_pages_phys(struct drm_i915_gem_object *obj)
{
struct address_space *mapping = obj->base.filp->f_mapping;
@@ -58,7 +69,7 @@ static int i915_gem_object_get_pages_phys(struct drm_i915_gem_object *obj)
sg->offset = 0;
sg->length = obj->base.size;
- sg_assign_page(sg, (struct page *)vaddr);
+ __set_phys_vaddr(sg, vaddr);
sg_dma_address(sg) = dma;
sg_dma_len(sg) = obj->base.size;
@@ -99,7 +110,7 @@ i915_gem_object_put_pages_phys(struct drm_i915_gem_object *obj,
struct sg_table *pages)
{
dma_addr_t dma = sg_dma_address(pages->sgl);
- void *vaddr = sg_page(pages->sgl);
+ void *vaddr = __get_phys_vaddr(pages->sgl);
__i915_gem_object_release_shmem(obj, pages, false);
@@ -139,7 +150,7 @@ i915_gem_object_put_pages_phys(struct drm_i915_gem_object *obj,
int i915_gem_object_pwrite_phys(struct drm_i915_gem_object *obj,
const struct drm_i915_gem_pwrite *args)
{
- void *vaddr = sg_page(obj->mm.pages->sgl) + args->offset;
+ void *vaddr = __get_phys_vaddr(obj->mm.pages->sgl) + args->offset;
char __user *user_data = u64_to_user_ptr(args->data_ptr);
struct drm_i915_private *i915 = to_i915(obj->base.dev);
int err;
@@ -170,7 +181,7 @@ int i915_gem_object_pwrite_phys(struct drm_i915_gem_object *obj,
int i915_gem_object_pread_phys(struct drm_i915_gem_object *obj,
const struct drm_i915_gem_pread *args)
{
- void *vaddr = sg_page(obj->mm.pages->sgl) + args->offset;
+ void *vaddr = __get_phys_vaddr(obj->mm.pages->sgl) + args->offset;
char __user *user_data = u64_to_user_ptr(args->data_ptr);
int err;
diff --git a/drivers/gpu/drm/vc4/vc4_validate_shaders.c b/drivers/gpu/drm/vc4/vc4_validate_shaders.c
index d48cf76..66502a6 100644
--- a/drivers/gpu/drm/vc4/vc4_validate_shaders.c
+++ b/drivers/gpu/drm/vc4/vc4_validate_shaders.c
@@ -290,15 +290,16 @@ static bool require_uniform_address_uniform(struct vc4_validated_shader_info *va
{
uint32_t o = validated_shader->num_uniform_addr_offsets;
uint32_t num_uniforms = validated_shader->uniforms_size / 4;
+ u32 *offsets;
- validated_shader->uniform_addr_offsets =
- krealloc(validated_shader->uniform_addr_offsets,
- (o + 1) *
- sizeof(*validated_shader->uniform_addr_offsets),
- GFP_KERNEL);
- if (!validated_shader->uniform_addr_offsets)
+ offsets = krealloc_array(validated_shader->uniform_addr_offsets,
+ o + 1,
+ sizeof(*validated_shader->uniform_addr_offsets),
+ GFP_KERNEL);
+ if (!offsets)
return false;
+ validated_shader->uniform_addr_offsets = offsets;
validated_shader->uniform_addr_offsets[o] = num_uniforms;
validated_shader->num_uniform_addr_offsets++;
diff --git a/drivers/gpu/drm/virtio/virtgpu_drv.c b/drivers/gpu/drm/virtio/virtgpu_drv.c
index a5ce96f..9af740b 100644
--- a/drivers/gpu/drm/virtio/virtgpu_drv.c
+++ b/drivers/gpu/drm/virtio/virtgpu_drv.c
@@ -124,7 +124,10 @@ static void virtio_gpu_remove(struct virtio_device *vdev)
struct drm_device *dev = vdev->priv;
drm_dev_unplug(dev);
- drm_atomic_helper_shutdown(dev);
+
+ if (drm_core_check_feature(dev, DRIVER_ATOMIC))
+ drm_atomic_helper_shutdown(dev);
+
virtio_gpu_deinit(dev);
drm_dev_put(dev);
}
diff --git a/drivers/gpu/drm/virtio/virtgpu_submit.c b/drivers/gpu/drm/virtio/virtgpu_submit.c
index dae761f..32cb1e4 100644
--- a/drivers/gpu/drm/virtio/virtgpu_submit.c
+++ b/drivers/gpu/drm/virtio/virtgpu_submit.c
@@ -65,8 +65,10 @@ static int virtio_gpu_dma_fence_wait(struct virtio_gpu_submit *submit,
dma_fence_unwrap_for_each(f, &itr, fence) {
err = virtio_gpu_do_fence_wait(submit, f);
- if (err)
+ if (err) {
+ dma_fence_put(itr.chain);
return err;
+ }
}
return 0;
diff --git a/drivers/gpu/drm/xe/display/xe_display.c b/drivers/gpu/drm/xe/display/xe_display.c
index 00dfa68..b17fb69 100644
--- a/drivers/gpu/drm/xe/display/xe_display.c
+++ b/drivers/gpu/drm/xe/display/xe_display.c
@@ -124,6 +124,15 @@ int xe_display_init_early(struct xe_device *xe)
intel_display_driver_early_probe(display);
+ intel_display_device_info_runtime_init(display);
+
+ /* Display may have been disabled at runtime init */
+ if (!intel_display_device_present(display)) {
+ xe->info.probe_display = false;
+ unset_display_features(xe);
+ return 0;
+ }
+
/* Early display init.. */
intel_opregion_setup(display);
@@ -137,8 +146,6 @@ int xe_display_init_early(struct xe_device *xe)
intel_bw_init_hw(display);
- intel_display_device_info_runtime_init(display);
-
err = intel_display_driver_probe_noirq(display);
if (err)
goto err_opregion;
diff --git a/drivers/gpu/drm/xe/xe_drm_ras.c b/drivers/gpu/drm/xe/xe_drm_ras.c
index e07dc23..c6cd32b 100644
--- a/drivers/gpu/drm/xe/xe_drm_ras.c
+++ b/drivers/gpu/drm/xe/xe_drm_ras.c
@@ -52,7 +52,7 @@ static struct xe_drm_ras_counter *allocate_and_copy_counters(struct xe_device *x
struct xe_drm_ras_counter *counter;
int i;
- counter = kcalloc(DRM_XE_RAS_ERR_COMP_MAX, sizeof(*counter), GFP_KERNEL);
+ counter = drmm_kcalloc(&xe->drm, DRM_XE_RAS_ERR_COMP_MAX, sizeof(*counter), GFP_KERNEL);
if (!counter)
return ERR_PTR(-ENOMEM);
@@ -100,54 +100,47 @@ static int assign_node_params(struct xe_device *xe, struct drm_ras_node *node,
return 0;
}
-static void cleanup_node_param(struct xe_drm_ras *ras, const enum drm_xe_ras_error_severity severity)
+static void cleanup_node_param(struct drm_ras_node *node)
{
- struct drm_ras_node *node = &ras->node[severity];
-
- kfree(ras->info[severity]);
- ras->info[severity] = NULL;
-
kfree(node->device_name);
node->device_name = NULL;
}
+static void cleanup_node(struct drm_device *drm, void *node)
+{
+ drm_ras_node_unregister(node);
+ cleanup_node_param(node);
+}
+
static int register_nodes(struct xe_device *xe)
{
struct xe_drm_ras *ras = &xe->ras;
- int i;
+ struct drm_ras_node *node;
+ int i, ret;
for_each_error_severity(i) {
- struct drm_ras_node *node = &ras->node[i];
- int ret;
+ node = &ras->node[i];
ret = assign_node_params(xe, node, i);
- if (ret) {
- cleanup_node_param(ras, i);
- return ret;
- }
+ if (ret)
+ goto free_param;
ret = drm_ras_node_register(node);
- if (ret) {
- cleanup_node_param(ras, i);
- return ret;
- }
+ if (ret)
+ goto free_param;
+
+ ret = drmm_add_action_or_reset(&xe->drm, cleanup_node, node);
+ if (ret)
+ goto null_info;
}
return 0;
-}
-static void xe_drm_ras_unregister_nodes(struct drm_device *device, void *arg)
-{
- struct xe_device *xe = arg;
- struct xe_drm_ras *ras = &xe->ras;
- int i;
-
- for_each_error_severity(i) {
- struct drm_ras_node *node = &ras->node[i];
-
- drm_ras_node_unregister(node);
- cleanup_node_param(ras, i);
- }
+free_param:
+ cleanup_node_param(node);
+null_info:
+ ras->info[i] = NULL;
+ return ret;
}
/**
@@ -176,11 +169,5 @@ int xe_drm_ras_init(struct xe_device *xe)
return err;
}
- err = drmm_add_action_or_reset(&xe->drm, xe_drm_ras_unregister_nodes, xe);
- if (err) {
- drm_err(&xe->drm, "Failed to add action for Xe DRM RAS (%pe)\n", ERR_PTR(err));
- return err;
- }
-
return 0;
}
diff --git a/drivers/gpu/drm/xe/xe_guc_submit.c b/drivers/gpu/drm/xe/xe_guc_submit.c
index a4a8f0d..42110e0 100644
--- a/drivers/gpu/drm/xe/xe_guc_submit.c
+++ b/drivers/gpu/drm/xe/xe_guc_submit.c
@@ -157,6 +157,11 @@ static void set_exec_queue_banned(struct xe_exec_queue *q)
atomic_or(EXEC_QUEUE_STATE_BANNED, &q->guc->state);
}
+static void clear_exec_queue_banned(struct xe_exec_queue *q)
+{
+ atomic_andnot(EXEC_QUEUE_STATE_BANNED, &q->guc->state);
+}
+
static bool exec_queue_suspended(struct xe_exec_queue *q)
{
return atomic_read(&q->guc->state) & EXEC_QUEUE_STATE_SUSPENDED;
@@ -1361,7 +1366,8 @@ static bool check_timeout(struct xe_exec_queue *q, struct xe_sched_job *job)
xe_sched_job_seqno(job), xe_sched_job_lrc_seqno(job),
q->guc->id);
- return xe_sched_invalidate_job(job, 2);
+ /* GuC never scheduled this job - let the caller trigger a GT reset. */
+ return true;
}
ctx_timestamp = lower_32_bits(xe_lrc_timestamp(q->lrc[0]));
@@ -1458,6 +1464,21 @@ static void disable_scheduling(struct xe_exec_queue *q, bool immediate)
G2H_LEN_DW_SCHED_CONTEXT_MODE_SET, 1);
}
+/*
+ * Recover via GT reset for a kernel queue, or for a GuC scheduling failure (job
+ * never started) on a queue that was not already killed or banned. An already
+ * banned queue must stay banned, so its unstarted jobs do not clear the ban or
+ * trigger a reset.
+ */
+static bool timeout_needs_gt_reset(struct xe_exec_queue *q, struct xe_sched_job *job,
+ bool skip_timeout_check)
+{
+ if (q->flags & EXEC_QUEUE_FLAG_KERNEL)
+ return true;
+
+ return !skip_timeout_check && !xe_sched_job_started(job);
+}
+
static enum drm_gpu_sched_stat
guc_exec_queue_timedout_job(struct drm_sched_job *drm_job)
{
@@ -1606,19 +1627,19 @@ guc_exec_queue_timedout_job(struct drm_sched_job *drm_job)
xe_sched_job_seqno(job), xe_sched_job_lrc_seqno(job),
q->guc->id, q->flags);
- /*
- * Kernel jobs should never fail, nor should VM jobs if they do
- * somethings has gone wrong and the GT needs a reset
- */
- xe_gt_WARN(q->gt, q->flags & EXEC_QUEUE_FLAG_KERNEL,
- "Kernel-submitted job timed out\n");
- xe_gt_WARN(q->gt, q->flags & EXEC_QUEUE_FLAG_VM && !exec_queue_killed(q),
- "VM job timed out on non-killed execqueue\n");
- if (!wedged && (q->flags & EXEC_QUEUE_FLAG_KERNEL ||
- (q->flags & EXEC_QUEUE_FLAG_VM && !exec_queue_killed(q)))) {
- if (!xe_sched_invalidate_job(job, 2)) {
- xe_gt_reset_async(q->gt);
- goto rearm;
+ if (!wedged) {
+ if (timeout_needs_gt_reset(q, job, skip_timeout_check)) {
+ if (!xe_sched_invalidate_job(job, 2)) {
+ clear_exec_queue_banned(q);
+ xe_gt_reset_async(q->gt);
+ goto rearm;
+ }
+ if (q->flags & EXEC_QUEUE_FLAG_KERNEL) {
+ xe_gt_WARN(q->gt, true, "Kernel-submitted job timed out\n");
+ xe_device_declare_wedged(gt_to_xe(q->gt));
+ }
+ } else if (q->flags & EXEC_QUEUE_FLAG_VM && !exec_queue_killed(q)) {
+ xe_gt_WARN(q->gt, true, "VM job timed out on non-killed execqueue\n");
}
}
diff --git a/drivers/gpu/drm/xe/xe_guc_tlb_inval.c b/drivers/gpu/drm/xe/xe_guc_tlb_inval.c
index ced58f4..cf6d106 100644
--- a/drivers/gpu/drm/xe/xe_guc_tlb_inval.c
+++ b/drivers/gpu/drm/xe/xe_guc_tlb_inval.c
@@ -255,9 +255,8 @@ static int send_tlb_inval_ctx_ppgtt(struct xe_tlb_inval *tlb_inval, u32 seqno,
#undef EXEC_QUEUE_COUNT_FULL_THRESHOLD
/*
- * Move exec queues to a temporary list to issue invalidations. The exec
- * queue must active and a reference must be taken to prevent concurrent
- * deregistrations.
+ * Move exec queues to a temporary list to issue invalidations. A
+ * reference must be taken to prevent concurrent deregistrations.
*
* List modification is safe because we hold 'vm->exec_queues.lock' for
* reading, which prevents external modifications. Using a per-GT list
@@ -266,7 +265,7 @@ static int send_tlb_inval_ctx_ppgtt(struct xe_tlb_inval *tlb_inval, u32 seqno,
*/
list_for_each_entry_safe(q, next, &vm->exec_queues.list[id],
vm_exec_queue_link) {
- if (q->ops->active(q) && xe_exec_queue_get_unless_zero(q)) {
+ if (xe_exec_queue_get_unless_zero(q)) {
last_q = q;
list_move_tail(&q->vm_exec_queue_link, &tlb_inval_list);
}
diff --git a/drivers/gpu/drm/xe/xe_hw_error.c b/drivers/gpu/drm/xe/xe_hw_error.c
index 2a31b43..e869bc3 100644
--- a/drivers/gpu/drm/xe/xe_hw_error.c
+++ b/drivers/gpu/drm/xe/xe_hw_error.c
@@ -219,9 +219,9 @@ static void log_hw_error(struct xe_tile *tile, const char *name,
struct xe_device *xe = tile_to_xe(tile);
if (severity == DRM_XE_RAS_ERR_SEV_CORRECTABLE)
- drm_warn(&xe->drm, "%s %s detected\n", name, severity_str);
+ drm_warn(&xe->drm, HW_ERR "%s %s detected\n", name, severity_str);
else
- drm_err_ratelimited(&xe->drm, "%s %s detected\n", name, severity_str);
+ drm_err_ratelimited(&xe->drm, HW_ERR "%s %s detected\n", name, severity_str);
}
static void log_gt_err(struct xe_tile *tile, const char *name, int i, u32 err,
@@ -231,10 +231,10 @@ static void log_gt_err(struct xe_tile *tile, const char *name, int i, u32 err,
struct xe_device *xe = tile_to_xe(tile);
if (severity == DRM_XE_RAS_ERR_SEV_CORRECTABLE)
- drm_warn(&xe->drm, "%s %s detected, ERROR_STAT_GT_VECTOR%d:0x%08x\n",
+ drm_warn(&xe->drm, HW_ERR "%s %s detected, ERROR_STAT_GT_VECTOR%d:0x%08x\n",
name, severity_str, i, err);
else
- drm_err_ratelimited(&xe->drm, "%s %s detected, ERROR_STAT_GT_VECTOR%d:0x%08x\n",
+ drm_err_ratelimited(&xe->drm, HW_ERR "%s %s detected, ERROR_STAT_GT_VECTOR%d:0x%08x\n",
name, severity_str, i, err);
}
@@ -251,9 +251,9 @@ static void log_soc_error(struct xe_tile *tile, const char * const *reg_info,
if (strcmp(name, "Undefined")) {
if (severity == DRM_XE_RAS_ERR_SEV_CORRECTABLE)
- drm_warn(&xe->drm, "%s SOC %s detected", name, severity_str);
+ drm_warn(&xe->drm, HW_ERR "%s SOC %s detected", name, severity_str);
else
- drm_err_ratelimited(&xe->drm, "%s SOC %s detected", name, severity_str);
+ drm_err_ratelimited(&xe->drm, HW_ERR "%s SOC %s detected", name, severity_str);
atomic_inc(&info[index].counter);
}
}
diff --git a/drivers/gpu/drm/xe/xe_range_fence.c b/drivers/gpu/drm/xe/xe_range_fence.c
index 372378e..3d8fa19 100644
--- a/drivers/gpu/drm/xe/xe_range_fence.c
+++ b/drivers/gpu/drm/xe/xe_range_fence.c
@@ -77,6 +77,8 @@ int xe_range_fence_insert(struct xe_range_fence_tree *tree,
} else if (err == 0) {
xe_range_fence_tree_insert(rfence, &tree->root);
return 0;
+ } else {
+ dma_fence_put(fence);
}
free:
diff --git a/drivers/gpu/nova-core/bitfield.rs b/drivers/gpu/nova-core/bitfield.rs
index 02efdcf..660c391 100644
--- a/drivers/gpu/nova-core/bitfield.rs
+++ b/drivers/gpu/nova-core/bitfield.rs
@@ -170,7 +170,7 @@ impl $name {
(@check_field_bounds $hi:tt:$lo:tt $field:ident as bool) => {
#[allow(clippy::eq_op)]
const _: () = {
- ::kernel::build_assert!(
+ ::kernel::build_assert::build_assert!(
$hi == $lo,
concat!("boolean field `", stringify!($field), "` covers more than one bit")
);
@@ -181,7 +181,7 @@ impl $name {
(@check_field_bounds $hi:tt:$lo:tt $field:ident as $type:tt) => {
#[allow(clippy::eq_op)]
const _: () = {
- ::kernel::build_assert!(
+ ::kernel::build_assert::build_assert!(
$hi >= $lo,
concat!("field `", stringify!($field), "`'s MSB is smaller than its LSB")
);
diff --git a/drivers/gpu/nova-core/firmware.rs b/drivers/gpu/nova-core/firmware.rs
index 6c2ab69..ad37994 100644
--- a/drivers/gpu/nova-core/firmware.rs
+++ b/drivers/gpu/nova-core/firmware.rs
@@ -48,7 +48,7 @@ fn request_firmware(
/// Structure used to describe some firmwares, notably FWSEC-FRTS.
#[repr(C)]
-#[derive(Debug, Clone)]
+#[derive(Debug, Clone, FromBytes)]
pub(crate) struct FalconUCodeDescV2 {
/// Header defined by 'NV_BIT_FALCON_UCODE_DESC_HEADER_VDESC*' in OpenRM.
hdr: u32,
@@ -84,9 +84,6 @@ pub(crate) struct FalconUCodeDescV2 {
pub(crate) alt_dmem_load_size: u32,
}
-// SAFETY: all bit patterns are valid for this type, and it doesn't use interior mutability.
-unsafe impl FromBytes for FalconUCodeDescV2 {}
-
/// Structure used to describe some firmwares, notably FWSEC-FRTS.
#[repr(C)]
#[derive(Debug, Clone)]
diff --git a/drivers/gpu/nova-core/gsp/cmdq.rs b/drivers/gpu/nova-core/gsp/cmdq.rs
index 275da9b..1c9b208 100644
--- a/drivers/gpu/nova-core/gsp/cmdq.rs
+++ b/drivers/gpu/nova-core/gsp/cmdq.rs
@@ -237,7 +237,7 @@ fn new(dev: &device::Device<device::Bound>) -> Result<Self> {
let start = gsp_mem.dma_handle();
// Write values one by one to avoid an on-stack instance of `PteArray`.
for i in 0..GspMem::PTE_ARRAY_SIZE {
- dma_write!(gsp_mem, .ptes.0[i], PteArray::<0>::entry(start, i)?);
+ dma_write!(gsp_mem, .ptes.0[build: i], PteArray::<0>::entry(start, i)?);
}
dma_write!(
@@ -260,7 +260,7 @@ fn new(dev: &device::Device<device::Bound>) -> Result<Self> {
let rx = self.gsp_read_ptr();
// Pointer to the first entry of the CPU message queue.
- let data = ptr::project!(mut self.0.as_mut_ptr(), .cpuq.msgq.data[0]);
+ let data = ptr::project!(mut self.0.as_mut_ptr(), .cpuq.msgq.data[build: 0]);
let (tail_end, wrap_end) = if rx == 0 {
// The write area is non-wrapping, and stops at the second-to-last entry of the command
@@ -322,7 +322,7 @@ fn driver_write_area_size(&self) -> usize {
let rx = self.cpu_read_ptr();
// Pointer to the first entry of the GSP message queue.
- let data = ptr::project!(self.0.as_ptr(), .gspq.msgq.data[0]);
+ let data = ptr::project!(self.0.as_ptr(), .gspq.msgq.data[build: 0]);
let (tail_end, wrap_end) = if rx <= tx {
// Read area is non-wrapping and stops right before `tx`.
diff --git a/drivers/gpu/nova-core/num.rs b/drivers/gpu/nova-core/num.rs
index 6c824b8..6eb174d 100644
--- a/drivers/gpu/nova-core/num.rs
+++ b/drivers/gpu/nova-core/num.rs
@@ -49,7 +49,7 @@ macro_rules! impl_safe_as {
#[allow(unused)]
#[inline(always)]
pub(crate) const fn [<$from _as_ $into>](value: $from) -> $into {
- kernel::static_assert!(size_of::<$into>() >= size_of::<$from>());
+ ::kernel::build_assert::static_assert!(size_of::<$into>() >= size_of::<$from>());
value as $into
}
diff --git a/drivers/gpu/nova-core/vbios.rs b/drivers/gpu/nova-core/vbios.rs
index ebda28e..8b7d17a 100644
--- a/drivers/gpu/nova-core/vbios.rs
+++ b/drivers/gpu/nova-core/vbios.rs
@@ -16,6 +16,8 @@
transmute::FromBytes,
};
+use zerocopy::FromBytes as _;
+
use crate::{
driver::Bar0,
firmware::{
@@ -1011,8 +1013,8 @@ pub(crate) fn header(&self) -> Result<FalconUCodeDesc> {
let data = self.base.data.get(falcon_ucode_offset..).ok_or(EINVAL)?;
match ver {
2 => {
- let v2 = FalconUCodeDescV2::from_bytes_copy_prefix(data)
- .ok_or(EINVAL)?
+ let v2 = FalconUCodeDescV2::read_from_prefix(data)
+ .map_err(|_| EINVAL)?
.0;
Ok(FalconUCodeDesc::V2(v2))
}
diff --git a/drivers/hwmon/amc6821.c b/drivers/hwmon/amc6821.c
index d5f864b..8e5926b 100644
--- a/drivers/hwmon/amc6821.c
+++ b/drivers/hwmon/amc6821.c
@@ -1076,7 +1076,7 @@ static int amc6821_probe(struct i2c_client *client)
"Failed to initialize hwmon\n");
if (IS_ENABLED(CONFIG_THERMAL) && fan_np && data->fan_cooling_levels)
- return PTR_ERR_OR_ZERO(devm_thermal_of_cooling_device_register(dev,
+ return PTR_ERR_OR_ZERO(devm_thermal_of_child_cooling_device_register(dev,
fan_np, client->name, data, &amc6821_cooling_ops));
return 0;
diff --git a/drivers/hwmon/aspeed-pwm-tacho.c b/drivers/hwmon/aspeed-pwm-tacho.c
index aa159bf..1c5945d 100644
--- a/drivers/hwmon/aspeed-pwm-tacho.c
+++ b/drivers/hwmon/aspeed-pwm-tacho.c
@@ -841,8 +841,9 @@ static int aspeed_create_pwm_cooling(struct device *dev,
}
snprintf(cdev->name, MAX_CDEV_NAME_LEN, "%pOFn%d", child, pwm_port);
- cdev->tcdev = devm_thermal_of_cooling_device_register(dev, child,
- cdev->name, cdev, &aspeed_pwm_cool_ops);
+ cdev->tcdev = devm_thermal_of_child_cooling_device_register(dev, child,
+ cdev->name, cdev,
+ &aspeed_pwm_cool_ops);
if (IS_ERR(cdev->tcdev))
return PTR_ERR(cdev->tcdev);
diff --git a/drivers/hwmon/cros_ec_hwmon.c b/drivers/hwmon/cros_ec_hwmon.c
index 6cf5ab0..77dd9f2 100644
--- a/drivers/hwmon/cros_ec_hwmon.c
+++ b/drivers/hwmon/cros_ec_hwmon.c
@@ -532,8 +532,8 @@ static void cros_ec_hwmon_register_fan_cooling_devices(struct device *dev,
cpriv->hwmon_priv = priv;
cpriv->index = i;
- cdev = devm_thermal_of_cooling_device_register(dev, NULL, type, cpriv,
- &cros_ec_thermal_cooling_ops);
+ cdev = devm_thermal_cooling_device_register(dev, type, cpriv,
+ &cros_ec_thermal_cooling_ops);
if (IS_ERR(cdev)) {
dev_warn(dev, "failed to register fan %zu as a cooling device: %pe\n", i,
cdev);
diff --git a/drivers/hwmon/dell-smm-hwmon.c b/drivers/hwmon/dell-smm-hwmon.c
index 038edff..47b373e 100644
--- a/drivers/hwmon/dell-smm-hwmon.c
+++ b/drivers/hwmon/dell-smm-hwmon.c
@@ -1161,8 +1161,8 @@ static int dell_smm_init_cdev(struct device *dev, u8 fan_num)
if (cdata) {
cdata->fan_num = fan_num;
cdata->data = data;
- cdev = devm_thermal_of_cooling_device_register(dev, NULL, name, cdata,
- &dell_smm_cooling_ops);
+ cdev = devm_thermal_cooling_device_register(dev, name, cdata,
+ &dell_smm_cooling_ops);
if (IS_ERR(cdev)) {
devm_kfree(dev, cdata);
ret = PTR_ERR(cdev);
diff --git a/drivers/hwmon/emc2305.c b/drivers/hwmon/emc2305.c
index 64b213e..2505e9f 100644
--- a/drivers/hwmon/emc2305.c
+++ b/drivers/hwmon/emc2305.c
@@ -309,9 +309,9 @@ static int emc2305_set_single_tz(struct device *dev, struct device_node *fan_nod
pwm = data->pwm_min[cdev_idx];
data->cdev_data[cdev_idx].cdev =
- devm_thermal_of_cooling_device_register(dev, fan_node,
- emc2305_fan_name[idx], data,
- &emc2305_cooling_ops);
+ devm_thermal_of_child_cooling_device_register(dev, fan_node,
+ emc2305_fan_name[idx], data,
+ &emc2305_cooling_ops);
if (IS_ERR(data->cdev_data[cdev_idx].cdev)) {
dev_err(dev, "Failed to register cooling device %s\n", emc2305_fan_name[idx]);
diff --git a/drivers/hwmon/gpio-fan.c b/drivers/hwmon/gpio-fan.c
index a8892ce..084828e 100644
--- a/drivers/hwmon/gpio-fan.c
+++ b/drivers/hwmon/gpio-fan.c
@@ -592,8 +592,10 @@ static int gpio_fan_probe(struct platform_device *pdev)
}
/* Optional cooling device register for Device tree platforms */
- fan_data->cdev = devm_thermal_of_cooling_device_register(dev, np,
- "gpio-fan", fan_data, &gpio_fan_cool_ops);
+ fan_data->cdev = devm_thermal_of_child_cooling_device_register(dev, np,
+ "gpio-fan",
+ fan_data,
+ &gpio_fan_cool_ops);
dev_info(dev, "GPIO fan initialized\n");
diff --git a/drivers/hwmon/hwmon.c b/drivers/hwmon/hwmon.c
index 6812d1f..c16e6da 100644
--- a/drivers/hwmon/hwmon.c
+++ b/drivers/hwmon/hwmon.c
@@ -1082,6 +1082,7 @@ EXPORT_SYMBOL_GPL(hwmon_device_register_with_info);
* @dev: the parent device
* @name: hwmon name attribute
* @drvdata: driver data to attach to created device
+ * @extra_groups: pointer to list of additional non-standard attribute groups
*
* The use of this function is restricted. It is provided for legacy reasons
* and must only be called from the thermal subsystem.
@@ -1093,12 +1094,13 @@ EXPORT_SYMBOL_GPL(hwmon_device_register_with_info);
*/
struct device *
hwmon_device_register_for_thermal(struct device *dev, const char *name,
- void *drvdata)
+ void *drvdata,
+ const struct attribute_group **extra_groups)
{
if (!name || !dev)
return ERR_PTR(-EINVAL);
- return __hwmon_device_register(dev, name, drvdata, NULL, NULL);
+ return __hwmon_device_register(dev, name, drvdata, NULL, extra_groups);
}
EXPORT_SYMBOL_NS_GPL(hwmon_device_register_for_thermal, "HWMON_THERMAL");
diff --git a/drivers/hwmon/max6650.c b/drivers/hwmon/max6650.c
index 56b8157..3466edd 100644
--- a/drivers/hwmon/max6650.c
+++ b/drivers/hwmon/max6650.c
@@ -794,9 +794,9 @@ static int max6650_probe(struct i2c_client *client)
return err;
if (IS_ENABLED(CONFIG_THERMAL)) {
- cooling_dev = devm_thermal_of_cooling_device_register(dev,
- dev->of_node, client->name,
- data, &max6650_cooling_ops);
+ cooling_dev = devm_thermal_of_child_cooling_device_register(dev, dev->of_node,
+ client->name, data,
+ &max6650_cooling_ops);
if (IS_ERR(cooling_dev)) {
dev_warn(dev, "thermal cooling device register failed: %ld\n",
PTR_ERR(cooling_dev));
diff --git a/drivers/hwmon/mlxreg-fan.c b/drivers/hwmon/mlxreg-fan.c
index 137a90d..860de6c 100644
--- a/drivers/hwmon/mlxreg-fan.c
+++ b/drivers/hwmon/mlxreg-fan.c
@@ -583,8 +583,8 @@ static int mlxreg_fan_cooling_config(struct device *dev, struct mlxreg_fan *fan)
pwm->fan = fan;
/* Set minimal PWM speed. */
pwm->last_hwmon_state = MLXREG_FAN_PWM_DUTY2STATE(MLXREG_FAN_MIN_DUTY);
- pwm->cdev = devm_thermal_of_cooling_device_register(dev, NULL, mlxreg_fan_name[i],
- pwm, &mlxreg_fan_cooling_ops);
+ pwm->cdev = devm_thermal_cooling_device_register(dev, mlxreg_fan_name[i],
+ pwm, &mlxreg_fan_cooling_ops);
if (IS_ERR(pwm->cdev)) {
dev_err(dev, "Failed to register cooling device\n");
return PTR_ERR(pwm->cdev);
diff --git a/drivers/hwmon/npcm750-pwm-fan.c b/drivers/hwmon/npcm750-pwm-fan.c
index c8f5e69..aea0b86 100644
--- a/drivers/hwmon/npcm750-pwm-fan.c
+++ b/drivers/hwmon/npcm750-pwm-fan.c
@@ -857,8 +857,10 @@ static int npcm7xx_create_pwm_cooling(struct device *dev,
snprintf(cdev->name, THERMAL_NAME_LENGTH, "%pOFn%d", child,
pwm_port);
- cdev->tcdev = devm_thermal_of_cooling_device_register(dev, child,
- cdev->name, cdev, &npcm7xx_pwm_cool_ops);
+ cdev->tcdev = devm_thermal_of_child_cooling_device_register(dev, child,
+ cdev->name,
+ cdev,
+ &npcm7xx_pwm_cool_ops);
if (IS_ERR(cdev->tcdev))
return PTR_ERR(cdev->tcdev);
diff --git a/drivers/hwmon/pwm-fan.c b/drivers/hwmon/pwm-fan.c
index 37269db..e6a567d 100644
--- a/drivers/hwmon/pwm-fan.c
+++ b/drivers/hwmon/pwm-fan.c
@@ -685,8 +685,9 @@ static int pwm_fan_probe(struct platform_device *pdev)
ctx->pwm_fan_state = ctx->pwm_fan_max_state;
if (IS_ENABLED(CONFIG_THERMAL)) {
- cdev = devm_thermal_of_cooling_device_register(dev,
- dev->of_node, "pwm-fan", ctx, &pwm_fan_cooling_ops);
+ cdev = devm_thermal_of_child_cooling_device_register(dev, dev->of_node,
+ "pwm-fan", ctx,
+ &pwm_fan_cooling_ops);
if (IS_ERR(cdev)) {
ret = PTR_ERR(cdev);
dev_err(dev,
diff --git a/drivers/hwmon/qnap-mcu-hwmon.c b/drivers/hwmon/qnap-mcu-hwmon.c
index e86e64c..c1c1e9d 100644
--- a/drivers/hwmon/qnap-mcu-hwmon.c
+++ b/drivers/hwmon/qnap-mcu-hwmon.c
@@ -337,9 +337,9 @@ static int qnap_mcu_hwmon_probe(struct platform_device *pdev)
* levels and only succeed with either no or correct cooling levels.
*/
if (IS_ENABLED(CONFIG_THERMAL) && hwm->fan_cooling_levels) {
- cdev = devm_thermal_of_cooling_device_register(dev,
- to_of_node(hwm->fan_node), "qnap-mcu-hwmon",
- hwm, &qnap_mcu_hwmon_cooling_ops);
+ cdev = devm_thermal_of_child_cooling_device_register(dev, to_of_node(hwm->fan_node),
+ "qnap-mcu-hwmon", hwm,
+ &qnap_mcu_hwmon_cooling_ops);
if (IS_ERR(cdev))
return dev_err_probe(dev, PTR_ERR(cdev),
"Failed to register qnap-mcu-hwmon as cooling device\n");
diff --git a/drivers/hwmon/tc654.c b/drivers/hwmon/tc654.c
index 39fe583..ba18b44 100644
--- a/drivers/hwmon/tc654.c
+++ b/drivers/hwmon/tc654.c
@@ -541,8 +541,9 @@ static int tc654_probe(struct i2c_client *client)
if (IS_ENABLED(CONFIG_THERMAL)) {
struct thermal_cooling_device *cdev;
- cdev = devm_thermal_of_cooling_device_register(dev, dev->of_node, client->name,
- hwmon_dev, &tc654_fan_cool_ops);
+ cdev = devm_thermal_of_child_cooling_device_register(dev, dev->of_node,
+ client->name, hwmon_dev,
+ &tc654_fan_cool_ops);
return PTR_ERR_OR_ZERO(cdev);
}
diff --git a/drivers/i2c/busses/i2c-imx-lpi2c.c b/drivers/i2c/busses/i2c-imx-lpi2c.c
index a01c236..cd4da50 100644
--- a/drivers/i2c/busses/i2c-imx-lpi2c.c
+++ b/drivers/i2c/busses/i2c-imx-lpi2c.c
@@ -1383,55 +1383,66 @@ static int lpi2c_imx_init_recovery_info(struct lpi2c_imx_struct *lpi2c_imx,
return 0;
}
-static void dma_exit(struct device *dev, struct lpi2c_imx_dma *dma)
-{
- if (dma->chan_rx)
- dma_release_channel(dma->chan_rx);
-
- if (dma->chan_tx)
- dma_release_channel(dma->chan_tx);
-
- devm_kfree(dev, dma);
-}
-
static int lpi2c_dma_init(struct device *dev, dma_addr_t phy_addr)
{
struct lpi2c_imx_struct *lpi2c_imx = dev_get_drvdata(dev);
struct lpi2c_imx_dma *dma;
+ void *group;
int ret;
- dma = devm_kzalloc(dev, sizeof(*dma), GFP_KERNEL);
- if (!dma)
+ /*
+ * Open a devres group so that all resources allocated within
+ * this function can be released together if DMA init fails but
+ * probe continues in PIO mode.
+ */
+ group = devres_open_group(dev, NULL, GFP_KERNEL);
+ if (!group)
return -ENOMEM;
+ dma = devm_kzalloc(dev, sizeof(*dma), GFP_KERNEL);
+ if (!dma) {
+ ret = -ENOMEM;
+ goto release_group;
+ }
+
dma->phy_addr = phy_addr;
/* Prepare for TX DMA: */
- dma->chan_tx = dma_request_chan(dev, "tx");
+ dma->chan_tx = devm_dma_request_chan(dev, "tx");
if (IS_ERR(dma->chan_tx)) {
ret = PTR_ERR(dma->chan_tx);
if (ret != -ENODEV && ret != -EPROBE_DEFER)
dev_err(dev, "can't request DMA tx channel (%d)\n", ret);
- dma->chan_tx = NULL;
- goto dma_exit;
+ goto release_group;
}
/* Prepare for RX DMA: */
- dma->chan_rx = dma_request_chan(dev, "rx");
+ dma->chan_rx = devm_dma_request_chan(dev, "rx");
if (IS_ERR(dma->chan_rx)) {
ret = PTR_ERR(dma->chan_rx);
if (ret != -ENODEV && ret != -EPROBE_DEFER)
dev_err(dev, "can't request DMA rx channel (%d)\n", ret);
- dma->chan_rx = NULL;
- goto dma_exit;
+ goto release_group;
}
+ /*
+ * DMA init succeeded. Remove the group marker but keep all resources
+ * bound to the device, they will be freed at device removal.
+ */
+ devres_remove_group(dev, group);
+
lpi2c_imx->can_use_dma = true;
lpi2c_imx->dma = dma;
return 0;
-dma_exit:
- dma_exit(dev, dma);
+release_group:
+ /*
+ * DMA init failed. Release ALL resources allocated inside this
+ * group (dma memory, TX channel if already acquired, etc.) so
+ * that a successful PIO-mode probe does not hold unused resources
+ * for the entire device lifetime.
+ */
+ devres_release_group(dev, group);
return ret;
}
diff --git a/drivers/i2c/busses/i2c-imx.c b/drivers/i2c/busses/i2c-imx.c
index a208fef..28313d0 100644
--- a/drivers/i2c/busses/i2c-imx.c
+++ b/drivers/i2c/busses/i2c-imx.c
@@ -1892,9 +1892,15 @@ static void i2c_imx_remove(struct platform_device *pdev)
static int i2c_imx_runtime_suspend(struct device *dev)
{
struct imx_i2c_struct *i2c_imx = dev_get_drvdata(dev);
+ int ret;
+
+ ret = pinctrl_pm_select_sleep_state(dev);
+ if (ret)
+ return ret;
clk_disable(i2c_imx->clk);
- return pinctrl_pm_select_sleep_state(dev);
+
+ return 0;
}
static int i2c_imx_runtime_resume(struct device *dev)
@@ -1907,10 +1913,13 @@ static int i2c_imx_runtime_resume(struct device *dev)
return ret;
ret = clk_enable(i2c_imx->clk);
- if (ret)
+ if (ret) {
dev_err(dev, "can't enable I2C clock, ret=%d\n", ret);
+ pinctrl_pm_select_sleep_state(dev);
+ return ret;
+ }
- return ret;
+ return 0;
}
static int i2c_imx_suspend(struct device *dev)
diff --git a/drivers/i2c/busses/i2c-qcom-cci.c b/drivers/i2c/busses/i2c-qcom-cci.c
index f3ccfbb..01e440b 100644
--- a/drivers/i2c/busses/i2c-qcom-cci.c
+++ b/drivers/i2c/busses/i2c-qcom-cci.c
@@ -660,8 +660,8 @@ static void cci_remove(struct platform_device *pdev)
if (cci->master[i].cci) {
i2c_del_adapter(&cci->master[i].adap);
of_node_put(cci->master[i].adap.dev.of_node);
+ cci_halt(cci, i);
}
- cci_halt(cci, i);
}
disable_irq(cci->irq);
diff --git a/drivers/i2c/busses/i2c-riic.c b/drivers/i2c/busses/i2c-riic.c
index 9e3595b..6d2ebf6 100644
--- a/drivers/i2c/busses/i2c-riic.c
+++ b/drivers/i2c/busses/i2c-riic.c
@@ -725,8 +725,10 @@ static int riic_i2c_resume_noirq(struct device *dev)
return ret;
ret = pm_runtime_force_resume(dev);
- if (ret)
+ if (ret) {
+ reset_control_assert(riic->rstc);
return ret;
+ }
ret = riic_init_hw(riic);
if (ret) {
diff --git a/drivers/i2c/busses/i2c-stm32f7.c b/drivers/i2c/busses/i2c-stm32f7.c
index 53d9df7..067af25 100644
--- a/drivers/i2c/busses/i2c-stm32f7.c
+++ b/drivers/i2c/busses/i2c-stm32f7.c
@@ -694,6 +694,9 @@ static int stm32f7_i2c_setup_timing(struct stm32f7_i2c_dev *i2c_dev,
if (!of_property_read_bool(i2c_dev->dev->of_node, "i2c-digital-filter"))
i2c_dev->dnf_dt = STM32F7_I2C_DNF_DEFAULT;
+ i2c_dev->analog_filter = of_property_read_bool(i2c_dev->dev->of_node,
+ "i2c-analog-filter");
+
do {
ret = stm32f7_i2c_compute_timing(i2c_dev, setup,
&i2c_dev->timing);
@@ -715,9 +718,6 @@ static int stm32f7_i2c_setup_timing(struct stm32f7_i2c_dev *i2c_dev,
return ret;
}
- i2c_dev->analog_filter = of_property_read_bool(i2c_dev->dev->of_node,
- "i2c-analog-filter");
-
dev_dbg(i2c_dev->dev, "I2C Speed(%i), Clk Source(%i)\n",
setup->speed_freq, setup->clock_src);
dev_dbg(i2c_dev->dev, "I2C Rise(%i) and Fall(%i) Time\n",
diff --git a/drivers/i2c/busses/i2c-tegra.c b/drivers/i2c/busses/i2c-tegra.c
index 479a166..400c908 100644
--- a/drivers/i2c/busses/i2c-tegra.c
+++ b/drivers/i2c/busses/i2c-tegra.c
@@ -2115,9 +2115,9 @@ static const struct tegra_i2c_hw_feature tegra264_i2c_hw = {
static const struct tegra_i2c_hw_feature tegra410_i2c_hw = {
.has_continue_xfer_support = true,
.has_per_pkt_xfer_complete_irq = true,
- .clk_divisor_hs_mode = 1,
+ .clk_divisor_hs_mode = 2,
.clk_divisor_std_mode = 0x3f,
- .clk_divisor_fast_mode = 0x2c,
+ .clk_divisor_fast_mode = 0x2f,
.clk_divisor_fast_plus_mode = 0x11,
.has_config_load_reg = true,
.has_multi_master_mode = true,
@@ -2133,8 +2133,8 @@ static const struct tegra_i2c_hw_feature tegra410_i2c_hw = {
.thigh_fast_mode = 0x2,
.tlow_fastplus_mode = 0x2,
.thigh_fastplus_mode = 0x2,
- .tlow_hs_mode = 0x8,
- .thigh_hs_mode = 0x6,
+ .tlow_hs_mode = 0x5,
+ .thigh_hs_mode = 0x2,
.setup_hold_time_std_mode = 0x08080808,
.setup_hold_time_fast_mode = 0x02020202,
.setup_hold_time_fastplus_mode = 0x02020202,
@@ -2402,28 +2402,37 @@ static int __maybe_unused tegra_i2c_runtime_suspend(struct device *dev)
static int __maybe_unused tegra_i2c_suspend(struct device *dev)
{
+ /*
+ * Bring the controller up and hold a usage count so it stays
+ * available until the noirq phase.
+ */
+ return pm_runtime_resume_and_get(dev);
+}
+
+static int __maybe_unused tegra_i2c_suspend_noirq(struct device *dev)
+{
struct tegra_i2c_dev *i2c_dev = dev_get_drvdata(dev);
- int err;
i2c_mark_adapter_suspended(&i2c_dev->adapter);
- if (!pm_runtime_status_suspended(dev)) {
- err = tegra_i2c_runtime_suspend(dev);
- if (err)
- return err;
- }
-
- return 0;
+ /*
+ * Runtime PM is already disabled at this point, so invoke the
+ * runtime_suspend callback directly to put the controller down.
+ */
+ return tegra_i2c_runtime_suspend(dev);
}
-static int __maybe_unused tegra_i2c_resume(struct device *dev)
+static int __maybe_unused tegra_i2c_resume_noirq(struct device *dev)
{
struct tegra_i2c_dev *i2c_dev = dev_get_drvdata(dev);
int err;
/*
- * We need to ensure that clocks are enabled so that registers can be
- * restored in tegra_i2c_init().
+ * Runtime PM is still disabled at this point, so invoke the
+ * runtime_resume callback directly to bring the controller back up
+ * before re-initializing the hardware. The adapter is then marked
+ * resumed so that consumers can issue transfers from their own
+ * resume_noirq() handlers and onwards.
*/
err = tegra_i2c_runtime_resume(dev);
if (err)
@@ -2433,24 +2442,22 @@ static int __maybe_unused tegra_i2c_resume(struct device *dev)
if (err)
return err;
- /*
- * In case we are runtime suspended, disable clocks again so that we
- * don't unbalance the clock reference counts during the next runtime
- * resume transition.
- */
- if (pm_runtime_status_suspended(dev)) {
- err = tegra_i2c_runtime_suspend(dev);
- if (err)
- return err;
- }
-
i2c_mark_adapter_resumed(&i2c_dev->adapter);
return 0;
}
+static int __maybe_unused tegra_i2c_resume(struct device *dev)
+{
+ pm_runtime_put(dev);
+
+ return 0;
+}
+
static const struct dev_pm_ops tegra_i2c_pm = {
- SET_NOIRQ_SYSTEM_SLEEP_PM_OPS(tegra_i2c_suspend, tegra_i2c_resume)
+ SET_SYSTEM_SLEEP_PM_OPS(tegra_i2c_suspend, tegra_i2c_resume)
+ SET_NOIRQ_SYSTEM_SLEEP_PM_OPS(tegra_i2c_suspend_noirq,
+ tegra_i2c_resume_noirq)
SET_RUNTIME_PM_OPS(tegra_i2c_runtime_suspend, tegra_i2c_runtime_resume,
NULL)
};
diff --git a/drivers/iommu/dma-iommu.c b/drivers/iommu/dma-iommu.c
index 54d96e8..381b60d 100644
--- a/drivers/iommu/dma-iommu.c
+++ b/drivers/iommu/dma-iommu.c
@@ -1918,12 +1918,18 @@ static int iommu_dma_iova_link_swiotlb(struct device *dev,
return 0;
}
+ /*
+ * After removing the partial head and tail, there may be no aligned
+ * middle left to map. The tail still gets bounced below.
+ */
size -= iova_end_pad;
- error = __dma_iova_link(dev, addr + mapped, phys + mapped, size, dir,
- attrs);
- if (error)
- goto out_unmap;
- mapped += size;
+ if (size) {
+ error = __dma_iova_link(dev, addr + mapped, phys + mapped,
+ size, dir, attrs);
+ if (error)
+ goto out_unmap;
+ mapped += size;
+ }
if (iova_end_pad) {
error = iommu_dma_iova_bounce_and_link(dev, addr + mapped,
@@ -1936,7 +1942,8 @@ static int iommu_dma_iova_link_swiotlb(struct device *dev,
return 0;
out_unmap:
- dma_iova_unlink(dev, state, 0, mapped, dir, attrs);
+ if (mapped)
+ dma_iova_unlink(dev, state, offset, mapped, dir, attrs);
return error;
}
diff --git a/drivers/md/md-bitmap.c b/drivers/md/md-bitmap.c
index 028b9ca..7d778fe 100644
--- a/drivers/md/md-bitmap.c
+++ b/drivers/md/md-bitmap.c
@@ -502,6 +502,18 @@ static void write_sb_page(struct bitmap *bitmap, unsigned long pg_index,
static void md_bitmap_file_kick(struct bitmap *bitmap);
#ifdef CONFIG_MD_BITMAP_FILE
+static void end_bitmap_write(struct bio *bio)
+{
+ struct buffer_head *bh;
+ bool uptodate = bio_endio_bh(bio, &bh);
+ struct bitmap *bitmap = bh->b_private;
+
+ if (!uptodate)
+ set_bit(BITMAP_WRITE_ERROR, &bitmap->flags);
+ if (atomic_dec_and_test(&bitmap->pending_writes))
+ wake_up(&bitmap->write_wait);
+}
+
static void write_file_page(struct bitmap *bitmap, struct page *page, int wait)
{
struct buffer_head *bh = page_buffers(page);
@@ -510,7 +522,7 @@ static void write_file_page(struct bitmap *bitmap, struct page *page, int wait)
atomic_inc(&bitmap->pending_writes);
set_buffer_locked(bh);
set_buffer_mapped(bh);
- submit_bh(REQ_OP_WRITE | REQ_SYNC, bh);
+ bh_submit(bh, REQ_OP_WRITE | REQ_SYNC, end_bitmap_write);
bh = bh->b_this_page;
}
@@ -519,16 +531,6 @@ static void write_file_page(struct bitmap *bitmap, struct page *page, int wait)
atomic_read(&bitmap->pending_writes) == 0);
}
-static void end_bitmap_write(struct buffer_head *bh, int uptodate)
-{
- struct bitmap *bitmap = bh->b_private;
-
- if (!uptodate)
- set_bit(BITMAP_WRITE_ERROR, &bitmap->flags);
- if (atomic_dec_and_test(&bitmap->pending_writes))
- wake_up(&bitmap->write_wait);
-}
-
static void free_buffers(struct page *page)
{
struct buffer_head *bh;
@@ -592,12 +594,11 @@ static int read_file_page(struct file *file, unsigned long index,
else
count -= blocksize;
- bh->b_end_io = end_bitmap_write;
bh->b_private = bitmap;
atomic_inc(&bitmap->pending_writes);
set_buffer_locked(bh);
set_buffer_mapped(bh);
- submit_bh(REQ_OP_READ, bh);
+ bh_submit(bh, REQ_OP_READ, end_bitmap_write);
}
blk_cur++;
bh = bh->b_this_page;
diff --git a/drivers/md/raid5.h b/drivers/md/raid5.h
index 1c7b710..1dfa60a 100644
--- a/drivers/md/raid5.h
+++ b/drivers/md/raid5.h
@@ -38,7 +38,7 @@
* Clean -> Dirty - on compute_parity to satisfy write/sync (RECONSTRUCT or RMW)
*
* The Want->Empty, Want->Clean, Dirty->Clean, transitions
- * all happen in b_end_io at interrupt time.
+ * all happen in end_io at interrupt time.
* Each sets the Uptodate bit before releasing the Lock bit.
* This leaves one multi-stage transition:
* Want->Dirty->Clean
@@ -64,7 +64,7 @@
* together, but we are not guaranteed of that so we allow for more.
*
* If a buffer is on the read list when the associated cache buffer is
- * Uptodate, the data is copied into the read buffer and it's b_end_io
+ * Uptodate, the data is copied into the read buffer and it's end_io
* routine is called. This may happen in the end_request routine only
* if the buffer has just successfully been read. end_request should
* remove the buffers from the list and then set the Uptodate bit on
@@ -76,7 +76,7 @@
* into the cache buffer, which is then marked dirty, and moved onto a
* third list, the written list (bh_written). Once both the parity
* block and the cached buffer are successfully written, any buffer on
- * a written list can be returned with b_end_io.
+ * a written list can be returned with end_io.
*
* The write list and read list both act as fifos. The read list,
* write list and written list are protected by the device_lock.
diff --git a/drivers/memory/tegra/tegra210-emc-core.c b/drivers/memory/tegra/tegra210-emc-core.c
index e96ca41..065ae8b 100644
--- a/drivers/memory/tegra/tegra210-emc-core.c
+++ b/drivers/memory/tegra/tegra210-emc-core.c
@@ -1966,8 +1966,8 @@ static int tegra210_emc_probe(struct platform_device *pdev)
tegra210_emc_debugfs_init(emc);
- cd = devm_thermal_of_cooling_device_register(emc->dev, np, "emc", emc,
- &tegra210_emc_cd_ops);
+ cd = devm_thermal_of_child_cooling_device_register(emc->dev, np, "emc", emc,
+ &tegra210_emc_cd_ops);
if (IS_ERR(cd)) {
err = PTR_ERR(cd);
dev_err(emc->dev, "failed to register cooling device: %d\n",
diff --git a/drivers/misc/fastrpc.c b/drivers/misc/fastrpc.c
index 1080f9a..f3a4938 100644
--- a/drivers/misc/fastrpc.c
+++ b/drivers/misc/fastrpc.c
@@ -310,6 +310,8 @@ struct fastrpc_user {
spinlock_t lock;
/* lock for allocations */
struct mutex mutex;
+ /* Reference count */
+ struct kref refcount;
};
/* Extract SMMU PA from consolidated IOVA */
@@ -386,7 +388,7 @@ static int fastrpc_map_get(struct fastrpc_map *map)
static int fastrpc_map_lookup(struct fastrpc_user *fl, int fd,
- struct fastrpc_map **ppmap)
+ struct fastrpc_map **ppmap, bool take_ref)
{
struct fastrpc_map *map = NULL;
struct dma_buf *buf;
@@ -401,6 +403,12 @@ static int fastrpc_map_lookup(struct fastrpc_user *fl, int fd,
if (map->fd != fd || map->buf != buf)
continue;
+ if (take_ref) {
+ ret = fastrpc_map_get(map);
+ if (ret)
+ break;
+ }
+
*ppmap = map;
ret = 0;
break;
@@ -497,15 +505,57 @@ static void fastrpc_channel_ctx_put(struct fastrpc_channel_ctx *cctx)
kref_put(&cctx->refcount, fastrpc_channel_ctx_free);
}
+static void fastrpc_context_put(struct fastrpc_invoke_ctx *ctx);
+
+static void fastrpc_user_free(struct kref *ref)
+{
+ struct fastrpc_user *fl = container_of(ref, struct fastrpc_user, refcount);
+ struct fastrpc_invoke_ctx *ctx, *n;
+ struct fastrpc_map *map, *m;
+ struct fastrpc_buf *buf, *b;
+
+ if (fl->init_mem)
+ fastrpc_buf_free(fl->init_mem);
+
+ list_for_each_entry_safe(ctx, n, &fl->pending, node) {
+ list_del(&ctx->node);
+ fastrpc_context_put(ctx);
+ }
+
+ list_for_each_entry_safe(map, m, &fl->maps, node)
+ fastrpc_map_put(map);
+
+ list_for_each_entry_safe(buf, b, &fl->mmaps, node) {
+ list_del(&buf->node);
+ fastrpc_buf_free(buf);
+ }
+
+ fastrpc_channel_ctx_put(fl->cctx);
+ mutex_destroy(&fl->mutex);
+ kfree(fl);
+}
+
+static void fastrpc_user_get(struct fastrpc_user *fl)
+{
+ kref_get(&fl->refcount);
+}
+
+static void fastrpc_user_put(struct fastrpc_user *fl)
+{
+ kref_put(&fl->refcount, fastrpc_user_free);
+}
+
static void fastrpc_context_free(struct kref *ref)
{
struct fastrpc_invoke_ctx *ctx;
struct fastrpc_channel_ctx *cctx;
+ struct fastrpc_user *fl;
unsigned long flags;
int i;
ctx = container_of(ref, struct fastrpc_invoke_ctx, refcount);
cctx = ctx->cctx;
+ fl = ctx->fl;
for (i = 0; i < ctx->nbufs; i++)
fastrpc_map_put(ctx->maps[i]);
@@ -521,6 +571,8 @@ static void fastrpc_context_free(struct kref *ref)
kfree(ctx->olaps);
kfree(ctx);
+ /* Release the reference taken in fastrpc_context_alloc() */
+ fastrpc_user_put(fl);
fastrpc_channel_ctx_put(cctx);
}
@@ -628,6 +680,8 @@ static struct fastrpc_invoke_ctx *fastrpc_context_alloc(
/* Released in fastrpc_context_put() */
fastrpc_channel_ctx_get(cctx);
+ /* Take a reference to user, released in fastrpc_context_free() */
+ fastrpc_user_get(user);
ctx->sc = sc;
ctx->retval = -1;
@@ -658,6 +712,7 @@ static struct fastrpc_invoke_ctx *fastrpc_context_alloc(
spin_lock(&user->lock);
list_del(&ctx->node);
spin_unlock(&user->lock);
+ fastrpc_user_put(user);
fastrpc_channel_ctx_put(cctx);
kfree(ctx->maps);
kfree(ctx->olaps);
@@ -871,19 +926,10 @@ static int fastrpc_map_attach(struct fastrpc_user *fl, int fd,
static int fastrpc_map_create(struct fastrpc_user *fl, int fd,
u64 len, u32 attr, struct fastrpc_map **ppmap)
{
- struct fastrpc_session_ctx *sess = fl->sctx;
- int err = 0;
+ if (!fastrpc_map_lookup(fl, fd, ppmap, true))
+ return 0;
- if (!fastrpc_map_lookup(fl, fd, ppmap)) {
- if (!fastrpc_map_get(*ppmap))
- return 0;
- dev_dbg(sess->dev, "%s: Failed to get map fd=%d\n",
- __func__, fd);
- }
-
- err = fastrpc_map_attach(fl, fd, len, attr, ppmap);
-
- return err;
+ return fastrpc_map_attach(fl, fd, len, attr, ppmap);
}
/*
@@ -1041,7 +1087,7 @@ static int fastrpc_get_args(u32 kernel, struct fastrpc_invoke_ctx *ctx)
pages[i].addr = ctx->maps[i]->dma_addr;
mmap_read_lock(current->mm);
- vma = find_vma(current->mm, ctx->args[i].ptr);
+ vma = vma_lookup(current->mm, ctx->args[i].ptr);
if (vma)
pages[i].addr += (ctx->args[i].ptr & PAGE_MASK) -
vma->vm_start;
@@ -1153,7 +1199,7 @@ static int fastrpc_put_args(struct fastrpc_invoke_ctx *ctx,
for (i = 0; i < FASTRPC_MAX_FDLIST; i++) {
if (!fdlist[i])
break;
- if (!fastrpc_map_lookup(fl, (int)fdlist[i], &mmap))
+ if (!fastrpc_map_lookup(fl, (int)fdlist[i], &mmap, false))
fastrpc_map_put(mmap);
}
@@ -1579,9 +1625,6 @@ static int fastrpc_device_release(struct inode *inode, struct file *file)
{
struct fastrpc_user *fl = (struct fastrpc_user *)file->private_data;
struct fastrpc_channel_ctx *cctx = fl->cctx;
- struct fastrpc_invoke_ctx *ctx, *n;
- struct fastrpc_map *map, *m;
- struct fastrpc_buf *buf, *b;
unsigned long flags;
fastrpc_release_current_dsp_process(fl);
@@ -1590,28 +1633,10 @@ static int fastrpc_device_release(struct inode *inode, struct file *file)
list_del(&fl->user);
spin_unlock_irqrestore(&cctx->lock, flags);
- if (fl->init_mem)
- fastrpc_buf_free(fl->init_mem);
-
- list_for_each_entry_safe(ctx, n, &fl->pending, node) {
- list_del(&ctx->node);
- fastrpc_context_put(ctx);
- }
-
- list_for_each_entry_safe(map, m, &fl->maps, node)
- fastrpc_map_put(map);
-
- list_for_each_entry_safe(buf, b, &fl->mmaps, node) {
- list_del(&buf->node);
- fastrpc_buf_free(buf);
- }
-
fastrpc_session_free(cctx, fl->sctx);
- fastrpc_channel_ctx_put(cctx);
-
- mutex_destroy(&fl->mutex);
- kfree(fl);
file->private_data = NULL;
+ /* Release the reference taken in fastrpc_device_open */
+ fastrpc_user_put(fl);
return 0;
}
@@ -1655,6 +1680,7 @@ static int fastrpc_device_open(struct inode *inode, struct file *filp)
spin_lock_irqsave(&cctx->lock, flags);
list_add_tail(&fl->user, &cctx->users);
spin_unlock_irqrestore(&cctx->lock, flags);
+ kref_init(&fl->refcount);
return 0;
}
@@ -2431,7 +2457,6 @@ static int fastrpc_rpmsg_probe(struct rpmsg_device *rpdev)
kref_init(&data->refcount);
- dev_set_drvdata(&rpdev->dev, data);
rdev->dma_mask = &data->dma_mask;
dma_set_mask_and_coherent(rdev, DMA_BIT_MASK(32));
INIT_LIST_HEAD(&data->users);
@@ -2440,6 +2465,7 @@ static int fastrpc_rpmsg_probe(struct rpmsg_device *rpdev)
idr_init(&data->ctx_idr);
data->domain_id = domain_id;
data->rpdev = rpdev;
+ dev_set_drvdata(&rpdev->dev, data);
err = of_platform_populate(rdev->of_node, NULL, NULL, rdev);
if (err)
@@ -2513,6 +2539,9 @@ static int fastrpc_rpmsg_callback(struct rpmsg_device *rpdev, void *data,
if (len < sizeof(*rsp))
return -EINVAL;
+ if (!cctx)
+ return -ENODEV;
+
ctxid = ((rsp->ctx & FASTRPC_CTXID_MASK) >> 4);
spin_lock_irqsave(&cctx->lock, flags);
diff --git a/drivers/net/ethernet/airoha/airoha_eth.c b/drivers/net/ethernet/airoha/airoha_eth.c
index eab6a98..31cdb11 100644
--- a/drivers/net/ethernet/airoha/airoha_eth.c
+++ b/drivers/net/ethernet/airoha/airoha_eth.c
@@ -1153,6 +1153,9 @@ static int airoha_qdma_init_hfwd_queues(struct airoha_qdma *qdma)
rmem = of_reserved_mem_lookup(np);
of_node_put(np);
+ if (!rmem)
+ return -ENODEV;
+
dma_addr = rmem->base;
/* Compute the number of hw descriptors according to the
* reserved memory size and the payload buffer size
diff --git a/drivers/net/ethernet/amazon/ena/ena_com.c b/drivers/net/ethernet/amazon/ena/ena_com.c
index 8c86789..297fb36 100644
--- a/drivers/net/ethernet/amazon/ena/ena_com.c
+++ b/drivers/net/ethernet/amazon/ena/ena_com.c
@@ -1880,6 +1880,11 @@ int ena_com_phc_get_timestamp(struct ena_com_dev *ena_dev, u64 *timestamp)
continue;
}
+ /* Ensure PHC payload (timestamp, error_flags) is read
+ * after req_id update is observed
+ */
+ dma_rmb();
+
/* req_id was updated by the device which indicates that
* PHC timestamp and error_flags are updated too,
* checking errors before retrieving timestamp
diff --git a/drivers/net/ethernet/broadcom/bnge/bnge_hwrm_lib.c b/drivers/net/ethernet/broadcom/bnge/bnge_hwrm_lib.c
index eb11800..1c9cfec 100644
--- a/drivers/net/ethernet/broadcom/bnge/bnge_hwrm_lib.c
+++ b/drivers/net/ethernet/broadcom/bnge/bnge_hwrm_lib.c
@@ -277,7 +277,7 @@ int bnge_hwrm_func_backing_store_qcaps(struct bnge_dev *bd)
struct hwrm_func_backing_store_qcaps_v2_output *resp;
struct hwrm_func_backing_store_qcaps_v2_input *req;
struct bnge_ctx_mem_info *ctx;
- u16 type;
+ u16 type, next_type;
int rc;
if (bd->ctx)
@@ -294,8 +294,8 @@ int bnge_hwrm_func_backing_store_qcaps(struct bnge_dev *bd)
resp = bnge_hwrm_req_hold(bd, req);
- for (type = 0; type < BNGE_CTX_V2_MAX; ) {
- struct bnge_ctx_mem_type *ctxm = &ctx->ctx_arr[type];
+ for (type = 0; type < BNGE_CTX_INV; type = next_type) {
+ struct bnge_ctx_mem_type *ctxm;
u8 init_val, init_off, i;
__le32 *p;
u32 flags;
@@ -304,8 +304,14 @@ int bnge_hwrm_func_backing_store_qcaps(struct bnge_dev *bd)
rc = bnge_hwrm_req_send(bd, req);
if (rc)
goto ctx_done;
+
+ next_type = le16_to_cpu(resp->next_valid_type);
+ if (type >= BNGE_CTX_V2_MAX)
+ continue;
+
+ ctxm = &ctx->ctx_arr[type];
flags = le32_to_cpu(resp->flags);
- type = le16_to_cpu(resp->next_valid_type);
+
if (!(flags &
FUNC_BACKING_STORE_QCAPS_V2_RESP_FLAGS_TYPE_VALID))
continue;
diff --git a/drivers/net/ethernet/broadcom/bnxt/bnxt.c b/drivers/net/ethernet/broadcom/bnxt/bnxt.c
index 35e1f8f..c999f97 100644
--- a/drivers/net/ethernet/broadcom/bnxt/bnxt.c
+++ b/drivers/net/ethernet/broadcom/bnxt/bnxt.c
@@ -5748,7 +5748,7 @@ static void bnxt_disable_int_sync(struct bnxt *bp)
{
int i;
- if (!bp->irq_tbl)
+ if (!bp->irq_tbl || !bp->bnapi)
return;
atomic_inc(&bp->intr_sem);
diff --git a/drivers/net/ethernet/ibm/emac/core.c b/drivers/net/ethernet/ibm/emac/core.c
index 417dfa1..4e503b3 100644
--- a/drivers/net/ethernet/ibm/emac/core.c
+++ b/drivers/net/ethernet/ibm/emac/core.c
@@ -3144,7 +3144,7 @@ static int emac_probe(struct platform_device *ofdev)
netif_carrier_off(ndev);
- err = devm_register_netdev(&ofdev->dev, ndev);
+ err = register_netdev(ndev);
if (err) {
printk(KERN_ERR "%pOF: failed to register net device (%d)!\n",
np, err);
@@ -3197,6 +3197,13 @@ static void emac_remove(struct platform_device *ofdev)
DBG(dev, "remove" NL);
+ /* Unregister network device before tearing down hardware
+ * to prevent use-after-free during deferred cleanup. This ensures
+ * the network stack stops all operations before hardware resources
+ * are released.
+ */
+ unregister_netdev(dev->ndev);
+
cancel_work_sync(&dev->reset_work);
if (emac_has_feature(dev, EMAC_FTR_HAS_TAH))
diff --git a/drivers/net/ethernet/intel/ice/ice_dpll.c b/drivers/net/ethernet/intel/ice/ice_dpll.c
index 892bc7c..0704e92 100644
--- a/drivers/net/ethernet/intel/ice/ice_dpll.c
+++ b/drivers/net/ethernet/intel/ice/ice_dpll.c
@@ -2633,6 +2633,8 @@ static const struct dpll_pin_ops ice_dpll_pin_ufl_ops = {
.state_on_dpll_set = ice_dpll_ufl_pin_state_set,
.state_on_dpll_get = ice_dpll_sw_pin_state_get,
.direction_get = ice_dpll_pin_sw_direction_get,
+ .prio_get = ice_dpll_sw_input_prio_get,
+ .prio_set = ice_dpll_sw_input_prio_set,
.frequency_get = ice_dpll_sw_pin_frequency_get,
.frequency_set = ice_dpll_sw_pin_frequency_set,
.esync_set = ice_dpll_sw_esync_set,
diff --git a/drivers/net/ethernet/intel/idpf/idpf_ptp.c b/drivers/net/ethernet/intel/idpf/idpf_ptp.c
index 4a51d27..71fe8b2a 100644
--- a/drivers/net/ethernet/intel/idpf/idpf_ptp.c
+++ b/drivers/net/ethernet/intel/idpf/idpf_ptp.c
@@ -51,7 +51,7 @@ void idpf_ptp_get_features_access(const struct idpf_adapter *adapter)
/* Set the device clock time */
direct = VIRTCHNL2_CAP_PTP_SET_DEVICE_CLK_TIME;
- mailbox = VIRTCHNL2_CAP_PTP_SET_DEVICE_CLK_TIME;
+ mailbox = VIRTCHNL2_CAP_PTP_SET_DEVICE_CLK_TIME_MB;
ptp->set_dev_clk_time_access = idpf_ptp_get_access(adapter,
direct,
mailbox);
diff --git a/drivers/net/ethernet/marvell/mv643xx_eth.c b/drivers/net/ethernet/marvell/mv643xx_eth.c
index f9055b3..1881583 100644
--- a/drivers/net/ethernet/marvell/mv643xx_eth.c
+++ b/drivers/net/ethernet/marvell/mv643xx_eth.c
@@ -2780,7 +2780,7 @@ static int mv643xx_eth_shared_of_add_port(struct platform_device *pdev,
goto put_err;
}
ppdev->dev.coherent_dma_mask = DMA_BIT_MASK(32);
- ppdev->dev.of_node = pnp;
+ ppdev->dev.of_node = of_node_get(pnp);
ret = platform_device_add_resources(ppdev, &res, 1);
if (ret)
diff --git a/drivers/net/ethernet/marvell/mvpp2/mvpp2_main.c b/drivers/net/ethernet/marvell/mvpp2/mvpp2_main.c
index f442b87..ccc24a1 100644
--- a/drivers/net/ethernet/marvell/mvpp2/mvpp2_main.c
+++ b/drivers/net/ethernet/marvell/mvpp2/mvpp2_main.c
@@ -3917,10 +3917,10 @@ static int mvpp2_rx(struct mvpp2_port *port, struct napi_struct *napi,
struct mvpp2_bm_pool *bm_pool;
struct page_pool *pp = NULL;
struct sk_buff *skb;
- unsigned int frag_size;
+ unsigned int frag_size, rx_sync_size;
dma_addr_t dma_addr;
phys_addr_t phys_addr;
- int pool, rx_bytes, err, ret;
+ int pool, rx_bytes, rx_offset, err, ret;
struct page *page;
void *data;
@@ -3933,6 +3933,8 @@ static int mvpp2_rx(struct mvpp2_port *port, struct napi_struct *napi,
rx_status = mvpp2_rxdesc_status_get(port, rx_desc);
rx_bytes = mvpp2_rxdesc_size_get(port, rx_desc);
rx_bytes -= MVPP2_MH_SIZE;
+ rx_sync_size = rx_bytes + MVPP2_MH_SIZE;
+ rx_offset = MVPP2_MH_SIZE + MVPP2_SKB_HEADROOM;
dma_addr = mvpp2_rxdesc_dma_addr_get(port, rx_desc);
pool = (rx_status & MVPP2_RXD_BM_POOL_ID_MASK) >>
@@ -3946,9 +3948,10 @@ static int mvpp2_rx(struct mvpp2_port *port, struct napi_struct *napi,
dma_dir = DMA_FROM_DEVICE;
}
- dma_sync_single_for_cpu(dev->dev.parent, dma_addr,
- rx_bytes + MVPP2_MH_SIZE,
- dma_dir);
+ dma_sync_single_range_for_cpu(dev->dev.parent, dma_addr,
+ MVPP2_SKB_HEADROOM,
+ rx_sync_size,
+ dma_dir);
/* Buffer header not supported */
if (rx_status & MVPP2_RXD_BUF_HDR)
@@ -3970,6 +3973,12 @@ static int mvpp2_rx(struct mvpp2_port *port, struct napi_struct *napi,
else
frag_size = bm_pool->frag_size;
+ err = mvpp2_rx_refill(port, bm_pool, pp, pool);
+ if (err) {
+ netdev_err(port->dev, "failed to refill BM pools\n");
+ goto err_drop_frame;
+ }
+
if (xdp_prog) {
struct xdp_rxq_info *xdp_rxq;
@@ -3978,7 +3987,7 @@ static int mvpp2_rx(struct mvpp2_port *port, struct napi_struct *napi,
else
xdp_rxq = &rxq->xdp_rxq_long;
- xdp_init_buff(&xdp, PAGE_SIZE, xdp_rxq);
+ xdp_init_buff(&xdp, bm_pool->frag_size, xdp_rxq);
xdp_prepare_buff(&xdp, data,
MVPP2_MH_SIZE + MVPP2_SKB_HEADROOM,
rx_bytes, true);
@@ -3987,17 +3996,19 @@ static int mvpp2_rx(struct mvpp2_port *port, struct napi_struct *napi,
if (ret) {
xdp_ret |= ret;
- err = mvpp2_rx_refill(port, bm_pool, pp, pool);
- if (err) {
- netdev_err(port->dev, "failed to refill BM pools\n");
- goto err_drop_frame;
- }
-
ps.rx_packets++;
ps.rx_bytes += rx_bytes;
continue;
}
+ rx_sync_size = max_t(unsigned int, rx_sync_size,
+ xdp.data_end - xdp.data_hard_start -
+ MVPP2_SKB_HEADROOM);
+
+ /* Update offset and length to reflect any XDP adjustments. */
+ rx_offset = xdp.data - data;
+ rx_bytes = xdp.data_end - xdp.data;
+
metasize = xdp.data - xdp.data_meta;
}
@@ -4007,8 +4018,20 @@ static int mvpp2_rx(struct mvpp2_port *port, struct napi_struct *napi,
skb = slab_build_skb(data);
if (!skb) {
netdev_warn(port->dev, "skb build failed\n");
- goto err_drop_frame;
+ if (pp) {
+ page_pool_put_page(pp, virt_to_head_page(data),
+ rx_sync_size, true);
+ } else {
+ dma_unmap_single_attrs(dev->dev.parent, dma_addr,
+ bm_pool->buf_size,
+ DMA_FROM_DEVICE,
+ DMA_ATTR_SKIP_CPU_SYNC);
+ mvpp2_frag_free(bm_pool, pp, data);
+ }
+ goto err_drop_frame_retired;
}
+ if (pp)
+ skb_mark_for_recycle(skb);
/* If we have RX hardware timestamping enabled, grab the
* timestamp from the queue and convert.
@@ -4019,16 +4042,7 @@ static int mvpp2_rx(struct mvpp2_port *port, struct napi_struct *napi,
skb_hwtstamps(skb));
}
- err = mvpp2_rx_refill(port, bm_pool, pp, pool);
- if (err) {
- netdev_err(port->dev, "failed to refill BM pools\n");
- dev_kfree_skb_any(skb);
- goto err_drop_frame;
- }
-
- if (pp)
- skb_mark_for_recycle(skb);
- else
+ if (!pp)
dma_unmap_single_attrs(dev->dev.parent, dma_addr,
bm_pool->buf_size, DMA_FROM_DEVICE,
DMA_ATTR_SKIP_CPU_SYNC);
@@ -4036,7 +4050,7 @@ static int mvpp2_rx(struct mvpp2_port *port, struct napi_struct *napi,
ps.rx_packets++;
ps.rx_bytes += rx_bytes;
- skb_reserve(skb, MVPP2_MH_SIZE + MVPP2_SKB_HEADROOM);
+ skb_reserve(skb, rx_offset);
skb_put(skb, rx_bytes);
if (metasize)
skb_metadata_set(skb, metasize);
@@ -4047,13 +4061,14 @@ static int mvpp2_rx(struct mvpp2_port *port, struct napi_struct *napi,
continue;
err_drop_frame:
- dev->stats.rx_errors++;
- mvpp2_rx_error(port, rx_desc);
/* Return the buffer to the pool */
if (rx_status & MVPP2_RXD_BUF_HDR)
mvpp2_buff_hdr_pool_put(port, rx_desc, pool, rx_status);
else
mvpp2_bm_pool_put(port, pool, dma_addr, phys_addr);
+err_drop_frame_retired:
+ dev->stats.rx_errors++;
+ mvpp2_rx_error(port, rx_desc);
}
if (xdp_ret & MVPP2_XDP_REDIR)
diff --git a/drivers/net/ethernet/marvell/octeontx2/af/cn20k/npc.c b/drivers/net/ethernet/marvell/octeontx2/af/cn20k/npc.c
index 6b3f453..fe8c4ff 100644
--- a/drivers/net/ethernet/marvell/octeontx2/af/cn20k/npc.c
+++ b/drivers/net/ethernet/marvell/octeontx2/af/cn20k/npc.c
@@ -1571,53 +1571,49 @@ static u8 npc_map2cn20k_flag(u8 flag)
return 0xff;
}
+static void npc_cn20k_translate_action_flags(struct npc_kpu_profile_action *act)
+{
+ u8 ltype, val;
+
+ if (act->lid != NPC_LID_LC)
+ return;
+
+ ltype = act->ltype;
+ if (ltype != NPC_LT_LC_IP &&
+ ltype != NPC_LT_LC_IP6 &&
+ ltype != NPC_LT_LC_IP_OPT &&
+ ltype != NPC_LT_LC_IP6_EXT)
+ return;
+
+ switch (act->flags) {
+ case NPC_F_LC_U_IP_FRAG:
+ case NPC_F_LC_U_IP6_FRAG:
+ case NPC_F_LC_L_6TO4:
+ case NPC_F_LC_L_MPLS_IN_IP:
+ case NPC_F_LC_L_IP6_TUN_IP6:
+ case NPC_F_LC_L_IP6_MPLS_IN_IP:
+ val = npc_map2cn20k_flag(act->flags);
+ if (val != 0xFF)
+ act->flags = val;
+ break;
+ default:
+ break;
+ }
+}
+
void
npc_cn20k_update_action_entries_n_flags(struct rvu *rvu,
struct npc_kpu_profile_adapter *pfl)
{
struct npc_kpu_profile_action *action;
- int entries, ltype;
- u8 flags, val;
+ int entries;
for (int i = 0; i < pfl->kpus; i++) {
action = pfl->kpu[i].action;
entries = pfl->kpu[i].action_entries;
- for (int j = 0; j < entries; j++) {
- if (action[j].lid != NPC_LID_LC)
- continue;
-
- ltype = action[j].ltype;
-
- if (ltype != NPC_LT_LC_IP &&
- ltype != NPC_LT_LC_IP6 &&
- ltype != NPC_LT_LC_IP_OPT &&
- ltype != NPC_LT_LC_IP6_EXT)
- continue;
-
- flags = action[j].flags;
-
- switch (flags) {
- case NPC_F_LC_U_IP_FRAG:
- case NPC_F_LC_U_IP6_FRAG:
- case NPC_F_LC_L_6TO4:
- case NPC_F_LC_L_MPLS_IN_IP:
- case NPC_F_LC_L_IP6_TUN_IP6:
- case NPC_F_LC_L_IP6_MPLS_IN_IP:
- val = npc_map2cn20k_flag(flags);
- if (val == 0xFF) {
- dev_err(rvu->dev,
- "%s: Error to get flag value\n",
- __func__);
- return;
- }
-
- action[j].flags = val;
- break;
- default:
- break;
- }
- }
+ for (int j = 0; j < entries; j++)
+ npc_cn20k_translate_action_flags(&action[j]);
}
}
@@ -1709,9 +1705,9 @@ int npc_cn20k_apply_custom_kpu(struct rvu *rvu,
for (entry = 0; entry < entries; entry++) {
profile->kpu[kpu].cam[entry] = cam[entry];
profile->kpu[kpu].action[entry] = action[entry];
+ npc_cn20k_translate_action_flags(&profile->kpu[kpu].action[entry]);
}
}
- npc_cn20k_update_action_entries_n_flags(rvu, profile);
return 0;
}
diff --git a/drivers/net/ethernet/marvell/octeontx2/af/rvu.c b/drivers/net/ethernet/marvell/octeontx2/af/rvu.c
index 3cf1315..6e907ee 100644
--- a/drivers/net/ethernet/marvell/octeontx2/af/rvu.c
+++ b/drivers/net/ethernet/marvell/octeontx2/af/rvu.c
@@ -1160,7 +1160,7 @@ static int rvu_setup_hw_resources(struct rvu *rvu)
err = rvu_npc_exact_init(rvu);
if (err) {
dev_err(rvu->dev, "failed to initialize exact match table\n");
- return err;
+ goto cgx_err;
}
/* Assign MACs for CGX mapped functions */
diff --git a/drivers/net/ethernet/mellanox/mlx4/cq.c b/drivers/net/ethernet/mellanox/mlx4/cq.c
index e130e72..5c55971 100644
--- a/drivers/net/ethernet/mellanox/mlx4/cq.c
+++ b/drivers/net/ethernet/mellanox/mlx4/cq.c
@@ -290,6 +290,7 @@ static void mlx4_cq_free_icm(struct mlx4_dev *dev, int cqn)
static int mlx4_init_user_cqes(void *buf, int entries, int cqe_size)
{
int entries_per_copy = PAGE_SIZE / cqe_size;
+ size_t copy_bytes;
void *init_ents;
int err = 0;
int i;
@@ -314,8 +315,14 @@ static int mlx4_init_user_cqes(void *buf, int entries, int cqe_size)
buf += PAGE_SIZE;
}
} else {
+ copy_bytes = array_size(entries, cqe_size);
+ if (WARN_ON_ONCE(copy_bytes > PAGE_SIZE)) {
+ err = -EINVAL;
+ goto out;
+ }
+
err = copy_to_user((void __user *)buf, init_ents,
- array_size(entries, cqe_size)) ?
+ copy_bytes) ?
-EFAULT : 0;
}
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en/xdp.c b/drivers/net/ethernet/mellanox/mlx5/core/en/xdp.c
index d3bab19..d8c7cb8 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en/xdp.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en/xdp.c
@@ -103,9 +103,15 @@ mlx5e_xmit_xdp_buff(struct mlx5e_xdpsq *sq, struct mlx5e_rq *rq,
xdptxd->dma_addr = dma_addr;
- if (unlikely(!INDIRECT_CALL_2(sq->xmit_xdp_frame, mlx5e_xmit_xdp_frame_mpwqe,
- mlx5e_xmit_xdp_frame, sq, xdptxd, 0, NULL)))
+ if (unlikely(!INDIRECT_CALL_2(sq->xmit_xdp_frame,
+ mlx5e_xmit_xdp_frame_mpwqe,
+ mlx5e_xmit_xdp_frame,
+ sq, xdptxd, 0, NULL))) {
+ dma_unmap_single(sq->pdev, dma_addr, xdptxd->len,
+ DMA_TO_DEVICE);
+ xdp_return_frame(xdpf);
return false;
+ }
/* xmit_mode == MLX5E_XDP_XMIT_MODE_FRAME */
mlx5e_xdpi_fifo_push(&sq->db.xdpi_fifo,
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/eswitch.c b/drivers/net/ethernet/mellanox/mlx5/core/eswitch.c
index 7c8311f..236f89a 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/eswitch.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/eswitch.c
@@ -533,23 +533,16 @@ static void esw_update_vport_addr_list(struct mlx5_eswitch *esw,
struct mlx5_vport *vport, int list_type)
{
bool is_uc = list_type == MLX5_NVPRT_LIST_TYPE_UC;
- u8 (*mac_list)[ETH_ALEN];
+ u8 (*mac_list)[ETH_ALEN] = NULL;
struct l2addr_node *node;
struct vport_addr *addr;
struct hlist_head *hash;
struct hlist_node *tmp;
- int size;
+ int size = 0;
int err;
int hi;
int i;
- size = is_uc ? MLX5_MAX_UC_PER_VPORT(esw->dev) :
- MLX5_MAX_MC_PER_VPORT(esw->dev);
-
- mac_list = kcalloc(size, ETH_ALEN, GFP_KERNEL);
- if (!mac_list)
- return;
-
hash = is_uc ? vport->uc_list : vport->mc_list;
for_each_l2hash_node(node, tmp, hash, hi) {
@@ -561,7 +554,7 @@ static void esw_update_vport_addr_list(struct mlx5_eswitch *esw,
goto out;
err = mlx5_query_nic_vport_mac_list(esw->dev, vport->vport, list_type,
- mac_list, &size);
+ &mac_list, &size);
if (err)
goto out;
esw_debug(esw->dev, "vport[%d] context update %s list size (%d)\n",
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/irq_affinity.c b/drivers/net/ethernet/mellanox/mlx5/core/irq_affinity.c
index 994fe83..a0bb8ee 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/irq_affinity.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/irq_affinity.c
@@ -105,9 +105,12 @@ irq_pool_find_least_loaded(struct mlx5_irq_pool *pool, const struct cpumask *req
lockdep_assert_held(&pool->lock);
xa_for_each_range(&pool->irqs, index, iter, start, end) {
- struct cpumask *iter_mask = mlx5_irq_get_affinity_mask(iter);
int iter_refcount = mlx5_irq_read_locked(iter);
+ const struct cpumask *iter_mask;
+ iter_mask = irq_get_effective_affinity_mask(mlx5_irq_get_irq(iter));
+ if (!iter_mask)
+ continue;
if (!cpumask_subset(iter_mask, req_mask))
/* skip IRQs with a mask which is not subset of req_mask */
continue;
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/vport.c b/drivers/net/ethernet/mellanox/mlx5/core/vport.c
index 4effe37..d63b0e88 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/vport.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/vport.c
@@ -324,35 +324,63 @@ int mlx5_modify_nic_vport_mtu(struct mlx5_core_dev *mdev, u16 mtu)
}
EXPORT_SYMBOL_GPL(mlx5_modify_nic_vport_mtu);
+static int mlx5_vport_max_mac_list_size(struct mlx5_core_dev *dev, u16 vport,
+ enum mlx5_list_type list_type)
+{
+ void *query_ctx, *hca_caps;
+ int ret = 0;
+
+ if (!vport && !mlx5_core_is_ecpf(dev))
+ return list_type == MLX5_NVPRT_LIST_TYPE_UC ?
+ 1 << MLX5_CAP_GEN(dev, log_max_current_uc_list) :
+ 1 << MLX5_CAP_GEN(dev, log_max_current_mc_list);
+
+ query_ctx = kzalloc(MLX5_ST_SZ_BYTES(query_hca_cap_out), GFP_KERNEL);
+ if (!query_ctx)
+ return -ENOMEM;
+
+ ret = mlx5_vport_get_other_func_general_cap(dev, vport, query_ctx);
+ if (ret)
+ goto out;
+
+ hca_caps = MLX5_ADDR_OF(query_hca_cap_out, query_ctx, capability);
+ ret = list_type == MLX5_NVPRT_LIST_TYPE_UC ?
+ 1 << MLX5_GET(cmd_hca_cap, hca_caps, log_max_current_uc_list) :
+ 1 << MLX5_GET(cmd_hca_cap, hca_caps, log_max_current_mc_list);
+
+out:
+ kfree(query_ctx);
+
+ return ret;
+}
+
int mlx5_query_nic_vport_mac_list(struct mlx5_core_dev *dev,
u16 vport,
enum mlx5_list_type list_type,
- u8 addr_list[][ETH_ALEN],
- int *list_size)
+ u8 (**addr_list)[ETH_ALEN],
+ int *addr_list_size)
{
u32 in[MLX5_ST_SZ_DW(query_nic_vport_context_in)] = {0};
+ int allowed_list_size;
void *nic_vport_ctx;
int max_list_size;
- int req_list_size;
int out_sz;
void *out;
int err;
int i;
- req_list_size = *list_size;
+ if (!addr_list || !addr_list_size)
+ return -EINVAL;
- max_list_size = list_type == MLX5_NVPRT_LIST_TYPE_UC ?
- 1 << MLX5_CAP_GEN(dev, log_max_current_uc_list) :
- 1 << MLX5_CAP_GEN(dev, log_max_current_mc_list);
+ *addr_list = NULL;
+ *addr_list_size = 0;
- if (req_list_size > max_list_size) {
- mlx5_core_warn(dev, "Requested list size (%d) > (%d) max_list_size\n",
- req_list_size, max_list_size);
- req_list_size = max_list_size;
- }
+ max_list_size = mlx5_vport_max_mac_list_size(dev, vport, list_type);
+ if (max_list_size < 0)
+ return max_list_size;
out_sz = MLX5_ST_SZ_BYTES(query_nic_vport_context_out) +
- req_list_size * MLX5_ST_SZ_BYTES(mac_address_layout);
+ max_list_size * MLX5_ST_SZ_BYTES(mac_address_layout);
out = kvzalloc(out_sz, GFP_KERNEL);
if (!out)
@@ -371,16 +399,24 @@ int mlx5_query_nic_vport_mac_list(struct mlx5_core_dev *dev,
nic_vport_ctx = MLX5_ADDR_OF(query_nic_vport_context_out, out,
nic_vport_context);
- req_list_size = MLX5_GET(nic_vport_context, nic_vport_ctx,
- allowed_list_size);
+ allowed_list_size = MLX5_GET(nic_vport_context, nic_vport_ctx,
+ allowed_list_size);
+ if (!allowed_list_size)
+ goto out;
- *list_size = req_list_size;
- for (i = 0; i < req_list_size; i++) {
+ *addr_list = kcalloc(allowed_list_size, ETH_ALEN, GFP_KERNEL);
+ if (!*addr_list) {
+ err = -ENOMEM;
+ goto out;
+ }
+
+ for (i = 0; i < allowed_list_size; i++) {
u8 *mac_addr = MLX5_ADDR_OF(nic_vport_context,
nic_vport_ctx,
current_uc_mac_address[i]) + 2;
- ether_addr_copy(addr_list[i], mac_addr);
+ ether_addr_copy((*addr_list)[i], mac_addr);
}
+ *addr_list_size = allowed_list_size;
out:
kvfree(out);
return err;
diff --git a/drivers/net/ethernet/wangxun/txgbe/txgbe_aml.c b/drivers/net/ethernet/wangxun/txgbe/txgbe_aml.c
index f051425..8fc32df 100644
--- a/drivers/net/ethernet/wangxun/txgbe/txgbe_aml.c
+++ b/drivers/net/ethernet/wangxun/txgbe/txgbe_aml.c
@@ -204,7 +204,7 @@ int txgbe_set_phy_link(struct wx *wx)
static int txgbe_sfp_to_linkmodes(struct wx *wx, struct txgbe_sff_id *id)
{
__ETHTOOL_DECLARE_LINK_MODE_MASK(modes) = { 0, };
- DECLARE_PHY_INTERFACE_MASK(interfaces);
+ DECLARE_PHY_INTERFACE_MASK_ZERO(interfaces);
struct txgbe *txgbe = wx->priv;
if (id->cable_tech & TXGBE_SFF_DA_PASSIVE_CABLE) {
@@ -271,7 +271,7 @@ static int txgbe_sfp_to_linkmodes(struct wx *wx, struct txgbe_sff_id *id)
static int txgbe_qsfp_to_linkmodes(struct wx *wx, struct txgbe_sff_id *id)
{
__ETHTOOL_DECLARE_LINK_MODE_MASK(modes) = { 0, };
- DECLARE_PHY_INTERFACE_MASK(interfaces);
+ DECLARE_PHY_INTERFACE_MASK_ZERO(interfaces);
struct txgbe *txgbe = wx->priv;
if (id->transceiver_type & TXGBE_SFF_ETHERNET_40G_CR4) {
@@ -335,7 +335,7 @@ static int txgbe_qsfp_to_linkmodes(struct wx *wx, struct txgbe_sff_id *id)
int txgbe_identify_module(struct wx *wx)
{
- struct txgbe_hic_get_module_info buffer;
+ struct txgbe_hic_get_module_info buffer = { 0 };
struct txgbe_sff_id *id;
int err = 0;
u32 mod_abs;
@@ -357,18 +357,16 @@ int txgbe_identify_module(struct wx *wx)
}
id = &buffer.id;
- if (id->identifier != TXGBE_SFF_IDENTIFIER_SFP &&
- id->identifier != TXGBE_SFF_IDENTIFIER_QSFP &&
- id->identifier != TXGBE_SFF_IDENTIFIER_QSFP_PLUS &&
- id->identifier != TXGBE_SFF_IDENTIFIER_QSFP28) {
- wx_err(wx, "Invalid module\n");
- return -ENODEV;
- }
-
- if (id->transceiver_type == 0xFF)
+ if (id->identifier == TXGBE_SFF_IDENTIFIER_SFP)
return txgbe_sfp_to_linkmodes(wx, id);
- return txgbe_qsfp_to_linkmodes(wx, id);
+ if (id->identifier == TXGBE_SFF_IDENTIFIER_QSFP ||
+ id->identifier == TXGBE_SFF_IDENTIFIER_QSFP_PLUS ||
+ id->identifier == TXGBE_SFF_IDENTIFIER_QSFP28)
+ return txgbe_qsfp_to_linkmodes(wx, id);
+
+ wx_err(wx, "Invalid module\n");
+ return -EINVAL;
}
void txgbe_setup_link(struct wx *wx)
diff --git a/drivers/net/ethernet/wangxun/txgbe/txgbe_type.h b/drivers/net/ethernet/wangxun/txgbe/txgbe_type.h
index 6b05f32..877234e 100644
--- a/drivers/net/ethernet/wangxun/txgbe/txgbe_type.h
+++ b/drivers/net/ethernet/wangxun/txgbe/txgbe_type.h
@@ -315,6 +315,9 @@ void txgbe_up(struct wx *wx);
int txgbe_setup_tc(struct net_device *dev, u8 tc);
void txgbe_do_reset(struct net_device *netdev);
+#define DECLARE_PHY_INTERFACE_MASK_ZERO(name) \
+ unsigned long name[PHY_INTERFACE_MODE_MAX] = { 0, }
+
#define TXGBE_LINK_SPEED_UNKNOWN 0
#define TXGBE_LINK_SPEED_10GB_FULL 4
#define TXGBE_LINK_SPEED_25GB_FULL 0x10
diff --git a/drivers/net/hyperv/netvsc.c b/drivers/net/hyperv/netvsc.c
index 59e9534..4d319c5 100644
--- a/drivers/net/hyperv/netvsc.c
+++ b/drivers/net/hyperv/netvsc.c
@@ -12,6 +12,7 @@
#include <linux/sched.h>
#include <linux/wait.h>
#include <linux/mm.h>
+#include <linux/highmem.h>
#include <linux/delay.h>
#include <linux/io.h>
#include <linux/slab.h>
@@ -965,12 +966,22 @@ static void netvsc_copy_to_send_buf(struct netvsc_device *net_device,
}
for (i = 0; i < page_count; i++) {
- char *src = phys_to_virt(pb[i].pfn << HV_HYP_PAGE_SHIFT);
- u32 offset = pb[i].offset;
+ phys_addr_t paddr = (pb[i].pfn << HV_HYP_PAGE_SHIFT) +
+ pb[i].offset;
u32 len = pb[i].len;
- memcpy(dest, (src + offset), len);
- dest += len;
+ while (len) {
+ struct page *page = phys_to_page(paddr);
+ u32 off = offset_in_page(paddr);
+ u32 chunk = min_t(u32, len, PAGE_SIZE - off);
+ char *src = kmap_local_page(page);
+
+ memcpy(dest, src + off, chunk);
+ kunmap_local(src);
+ dest += chunk;
+ paddr += chunk;
+ len -= chunk;
+ }
}
if (padding)
diff --git a/drivers/net/mctp/mctp-usb.c b/drivers/net/mctp/mctp-usb.c
index 3b5dff1..fade65f 100644
--- a/drivers/net/mctp/mctp-usb.c
+++ b/drivers/net/mctp/mctp-usb.c
@@ -22,7 +22,6 @@
struct mctp_usb {
struct usb_device *usbdev;
struct usb_interface *intf;
- bool stopped;
struct net_device *netdev;
@@ -32,6 +31,9 @@ struct mctp_usb {
struct urb *tx_urb;
struct urb *rx_urb;
+ /* enforces atomic access to rx_stopped and requeuing the retry work */
+ spinlock_t rx_lock;
+ bool rx_stopped;
struct delayed_work rx_retry_work;
};
@@ -122,6 +124,7 @@ static const unsigned long RX_RETRY_DELAY = HZ / 4;
static int mctp_usb_rx_queue(struct mctp_usb *mctp_usb, gfp_t gfp)
{
+ unsigned long flags;
struct sk_buff *skb;
int rc;
@@ -147,8 +150,11 @@ static int mctp_usb_rx_queue(struct mctp_usb *mctp_usb, gfp_t gfp)
return rc;
err_retry:
- schedule_delayed_work(&mctp_usb->rx_retry_work, RX_RETRY_DELAY);
- return rc;
+ spin_lock_irqsave(&mctp_usb->rx_lock, flags);
+ if (!mctp_usb->rx_stopped)
+ schedule_delayed_work(&mctp_usb->rx_retry_work, RX_RETRY_DELAY);
+ spin_unlock_irqrestore(&mctp_usb->rx_lock, flags);
+ return 0;
}
static void mctp_usb_in_complete(struct urb *urb)
@@ -248,9 +254,6 @@ static void mctp_usb_rx_retry_work(struct work_struct *work)
struct mctp_usb *mctp_usb = container_of(work, struct mctp_usb,
rx_retry_work.work);
- if (READ_ONCE(mctp_usb->stopped))
- return;
-
mctp_usb_rx_queue(mctp_usb, GFP_KERNEL);
}
@@ -258,7 +261,7 @@ static int mctp_usb_open(struct net_device *dev)
{
struct mctp_usb *mctp_usb = netdev_priv(dev);
- WRITE_ONCE(mctp_usb->stopped, false);
+ WRITE_ONCE(mctp_usb->rx_stopped, false);
netif_start_queue(dev);
@@ -268,17 +271,21 @@ static int mctp_usb_open(struct net_device *dev)
static int mctp_usb_stop(struct net_device *dev)
{
struct mctp_usb *mctp_usb = netdev_priv(dev);
+ unsigned long flags;
netif_stop_queue(dev);
/* prevent RX submission retry */
- WRITE_ONCE(mctp_usb->stopped, true);
+ spin_lock_irqsave(&mctp_usb->rx_lock, flags);
+ mctp_usb->rx_stopped = true;
+ cancel_delayed_work(&mctp_usb->rx_retry_work);
+ spin_unlock_irqrestore(&mctp_usb->rx_lock, flags);
+
+ flush_delayed_work(&mctp_usb->rx_retry_work);
usb_kill_urb(mctp_usb->rx_urb);
usb_kill_urb(mctp_usb->tx_urb);
- cancel_delayed_work_sync(&mctp_usb->rx_retry_work);
-
return 0;
}
@@ -331,6 +338,7 @@ static int mctp_usb_probe(struct usb_interface *intf,
dev->netdev = netdev;
dev->usbdev = interface_to_usbdev(intf);
dev->intf = intf;
+ spin_lock_init(&dev->rx_lock);
usb_set_intfdata(intf, dev);
dev->ep_in = ep_in->bEndpointAddress;
diff --git a/drivers/net/phy/phy_device.c b/drivers/net/phy/phy_device.c
index 3370eb8..1511385 100644
--- a/drivers/net/phy/phy_device.c
+++ b/drivers/net/phy/phy_device.c
@@ -1718,6 +1718,9 @@ static int phy_sfp_probe(struct phy_device *phydev)
ret = sfp_bus_add_upstream(bus, phydev, &sfp_phydev_ops);
sfp_bus_put(bus);
+
+ if (ret)
+ phydev->sfp_bus = NULL;
}
if (!ret && phydev->sfp_bus)
@@ -3509,9 +3512,15 @@ static int phy_setup_ports(struct phy_device *phydev)
if (ret)
return ret;
- ret = phy_sfp_probe(phydev);
- if (ret)
- goto out;
+ /* We don't support SFP with genphy drivers. Also, genphy driver
+ * binding occurs with RTNL help, which will deadlock the call to
+ * sfp_bus_add_upstream().
+ */
+ if (!phydev->is_genphy_driven) {
+ ret = phy_sfp_probe(phydev);
+ if (ret)
+ goto out;
+ }
if (phydev->n_ports < phydev->max_n_ports) {
ret = phy_default_setup_single_port(phydev);
@@ -3775,6 +3784,11 @@ static int phy_probe(struct device *dev)
return 0;
out:
+ sfp_bus_del_upstream(phydev->sfp_bus);
+ phydev->sfp_bus = NULL;
+
+ phy_cleanup_ports(phydev);
+
if (!phydev->is_on_sfp_module)
phy_led_triggers_unregister(phydev);
@@ -3798,11 +3812,11 @@ static int phy_remove(struct device *dev)
phydev->state = PHY_DOWN;
- phy_cleanup_ports(phydev);
-
sfp_bus_del_upstream(phydev->sfp_bus);
phydev->sfp_bus = NULL;
+ phy_cleanup_ports(phydev);
+
if (phydev->drv && phydev->drv->remove)
phydev->drv->remove(phydev);
diff --git a/drivers/net/tun.c b/drivers/net/tun.c
index 9e7744e..fed9dfd 100644
--- a/drivers/net/tun.c
+++ b/drivers/net/tun.c
@@ -2070,6 +2070,7 @@ static ssize_t tun_put_user(struct tun_struct *tun,
struct virtio_net_hdr_v1_hash_tunnel hdr;
struct virtio_net_hdr *gso;
+ memset(&hdr, 0, sizeof(hdr));
ret = tun_vnet_hdr_tnl_from_skb(tun->flags, tun->dev, skb,
&hdr);
if (ret)
diff --git a/drivers/net/usb/r8152.c b/drivers/net/usb/r8152.c
index 1ace1d2..b126855 100644
--- a/drivers/net/usb/r8152.c
+++ b/drivers/net/usb/r8152.c
@@ -9851,7 +9851,12 @@ static int rtl8152_probe_once(struct usb_interface *intf,
struct net_device *netdev;
int ret;
- usb_reset_device(udev);
+ ret = usb_reset_device(udev);
+ if (ret < 0) {
+ dev_err(&intf->dev, "USB reset failed, errno=%d\n", ret);
+ return ret;
+ }
+
netdev = alloc_etherdev(sizeof(struct r8152));
if (!netdev) {
dev_err(&intf->dev, "Out of memory\n");
diff --git a/drivers/nvmem/core.c b/drivers/nvmem/core.c
index 311cb2e..e871181 100644
--- a/drivers/nvmem/core.c
+++ b/drivers/nvmem/core.c
@@ -1468,18 +1468,16 @@ struct nvmem_cell *of_nvmem_cell_get(struct device_node *np, const char *id)
cell_entry = nvmem_find_cell_entry_by_node(nvmem, cell_np);
of_node_put(cell_np);
if (!cell_entry) {
- __nvmem_device_put(nvmem);
nvmem_layout_module_put(nvmem);
- if (nvmem->layout)
- return ERR_PTR(-EPROBE_DEFER);
- else
- return ERR_PTR(-ENOENT);
+ ret = nvmem->layout ? -EPROBE_DEFER : -ENOENT;
+ __nvmem_device_put(nvmem);
+ return ERR_PTR(ret);
}
cell = nvmem_create_cell(cell_entry, id, cell_index);
if (IS_ERR(cell)) {
- __nvmem_device_put(nvmem);
nvmem_layout_module_put(nvmem);
+ __nvmem_device_put(nvmem);
}
return cell;
@@ -1593,8 +1591,8 @@ void nvmem_cell_put(struct nvmem_cell *cell)
kfree_const(cell->id);
kfree(cell);
- __nvmem_device_put(nvmem);
nvmem_layout_module_put(nvmem);
+ __nvmem_device_put(nvmem);
}
EXPORT_SYMBOL_GPL(nvmem_cell_put);
diff --git a/drivers/nvmem/layouts/onie-tlv.c b/drivers/nvmem/layouts/onie-tlv.c
index 0967a32..8b0f3c1 100644
--- a/drivers/nvmem/layouts/onie-tlv.c
+++ b/drivers/nvmem/layouts/onie-tlv.c
@@ -119,7 +119,7 @@ static int onie_tlv_add_cells(struct device *dev, struct nvmem_device *nvmem,
cell.name = onie_tlv_cell_name(tlv.type);
if (!cell.name)
- continue;
+ goto next;
cell.offset = hdr_len + offset + sizeof(tlv.type) + sizeof(tlv.len);
cell.bytes = tlv.len;
@@ -132,6 +132,7 @@ static int onie_tlv_add_cells(struct device *dev, struct nvmem_device *nvmem,
return ret;
}
+next:
offset += sizeof(tlv) + tlv.len;
}
diff --git a/drivers/pinctrl/pinctrl-amd.c b/drivers/pinctrl/pinctrl-amd.c
index 64315b0..e3128b0 100644
--- a/drivers/pinctrl/pinctrl-amd.c
+++ b/drivers/pinctrl/pinctrl-amd.c
@@ -26,7 +26,6 @@
#include <linux/interrupt.h>
#include <linux/bitops.h>
#include <linux/pinctrl/pinconf.h>
-#include <linux/dmi.h>
#include <linux/pinctrl/pinconf-generic.h>
#include <linux/pinctrl/pinmux.h>
#include <linux/string_choices.h>
@@ -40,39 +39,6 @@
static struct amd_gpio *pinctrl_dev;
#endif
-static const struct dmi_system_id amd_gpio_quirk_yoga7_14agp11[] = {
- {
- .matches = {
- DMI_MATCH(DMI_SYS_VENDOR, "LENOVO"),
- DMI_MATCH(DMI_PRODUCT_NAME, "83TD"),
- DMI_MATCH(DMI_BOARD_NAME, "LNVNB161216"),
- },
- },
- { }
-};
-
-static void amd_gpio_apply_quirks(struct amd_gpio *gpio_dev)
-{
- const unsigned int pin = 157; /* WACF2200 GpioInt per ACPI _CRS */
- unsigned long flags;
- u32 reg;
-
- if (!dmi_check_system(amd_gpio_quirk_yoga7_14agp11))
- return;
- if (pin >= gpio_dev->gc.ngpio)
- return;
-
- raw_spin_lock_irqsave(&gpio_dev->lock, flags);
- reg = readl(gpio_dev->base + pin * 4);
- reg |= BIT(INTERRUPT_ENABLE_OFF) | BIT(INTERRUPT_MASK_OFF);
- writel(reg, gpio_dev->base + pin * 4);
- raw_spin_unlock_irqrestore(&gpio_dev->lock, flags);
-
- dev_info(&gpio_dev->pdev->dev,
- "Enabled IRQ for GPIO %u (Yoga 7 14AGP11 touchscreen)\n",
- pin);
-}
-
static int amd_gpio_get_direction(struct gpio_chip *gc, unsigned offset)
{
unsigned long flags;
@@ -1253,7 +1219,6 @@ static int amd_gpio_probe(struct platform_device *pdev)
/* Disable and mask interrupts */
amd_gpio_irq_init(gpio_dev);
- amd_gpio_apply_quirks(gpio_dev);
girq = &gpio_dev->gc.irq;
gpio_irq_chip_set_chip(girq, &amd_gpio_irqchip);
diff --git a/drivers/pinctrl/pinctrl-mcp23s08_spi.c b/drivers/pinctrl/pinctrl-mcp23s08_spi.c
index 54f61c8..30775d3 100644
--- a/drivers/pinctrl/pinctrl-mcp23s08_spi.c
+++ b/drivers/pinctrl/pinctrl-mcp23s08_spi.c
@@ -10,6 +10,7 @@
#include "pinctrl-mcp23s08.h"
#define MCP_MAX_DEV_PER_CS 8
+#define MCP23S08_SPI_BASE 0x40
/*
* A given spi_device can represent up to eight mcp23sxx chips
@@ -143,13 +144,13 @@ static int mcp23s08_probe(struct spi_device *spi)
unsigned int addr;
int chips;
int ret;
- u32 v;
+ u8 v;
info = spi_get_device_match_data(spi);
- ret = device_property_read_u32(dev, "microchip,spi-present-mask", &v);
+ ret = device_property_read_u8(dev, "microchip,spi-present-mask", &v);
if (ret) {
- ret = device_property_read_u32(dev, "mcp,spi-present-mask", &v);
+ ret = device_property_read_u8(dev, "mcp,spi-present-mask", &v);
if (ret) {
dev_err(dev, "missing spi-present-mask");
return ret;
@@ -173,6 +174,8 @@ static int mcp23s08_probe(struct spi_device *spi)
for_each_set_bit(addr, &spi_present_mask, MCP_MAX_DEV_PER_CS) {
data->mcp[addr] = &data->chip[--chips];
data->mcp[addr]->irq = spi->irq;
+ data->mcp[addr]->dev = dev;
+ data->mcp[addr]->addr = MCP23S08_SPI_BASE | (addr << 1);
ret = mcp23s08_spi_regmap_init(data->mcp[addr], dev, addr, info);
if (ret)
@@ -184,7 +187,7 @@ static int mcp23s08_probe(struct spi_device *spi)
if (!data->mcp[addr]->pinctrl_desc.name)
return -ENOMEM;
- ret = mcp23s08_probe_one(data->mcp[addr], dev, 0x40 | (addr << 1),
+ ret = mcp23s08_probe_one(data->mcp[addr], dev, MCP23S08_SPI_BASE | (addr << 1),
info->type, -1);
if (ret < 0)
return ret;
diff --git a/drivers/pmdomain/imx/gpc.c b/drivers/pmdomain/imx/gpc.c
index de695f1..42e50c9 100644
--- a/drivers/pmdomain/imx/gpc.c
+++ b/drivers/pmdomain/imx/gpc.c
@@ -487,7 +487,7 @@ static int imx_gpc_probe(struct platform_device *pdev)
domain->ipg_rate_mhz = ipg_rate_mhz;
pd_pdev->dev.parent = &pdev->dev;
- pd_pdev->dev.of_node = np;
+ pd_pdev->dev.of_node = of_node_get(np);
pd_pdev->dev.fwnode = of_fwnode_handle(np);
ret = platform_device_add(pd_pdev);
diff --git a/drivers/pmdomain/ti/ti_sci_pm_domains.c b/drivers/pmdomain/ti/ti_sci_pm_domains.c
index 18d33bc..949e411 100644
--- a/drivers/pmdomain/ti/ti_sci_pm_domains.c
+++ b/drivers/pmdomain/ti/ti_sci_pm_domains.c
@@ -86,7 +86,7 @@ static inline void ti_sci_pd_set_wkup_constraint(struct device *dev)
const struct ti_sci_handle *ti_sci = pd->parent->ti_sci;
int ret;
- if (device_may_wakeup(dev)) {
+ if (device_may_wakeup(dev) || device_wakeup_path(dev)) {
/*
* If device can wakeup using IO daisy chain wakeups,
* we do not want to set a constraint.
diff --git a/drivers/ptp/ptp_ocp.c b/drivers/ptp/ptp_ocp.c
index beacc2f..7353855 100644
--- a/drivers/ptp/ptp_ocp.c
+++ b/drivers/ptp/ptp_ocp.c
@@ -2479,8 +2479,13 @@ ptp_ocp_ts_enable(void *priv, u32 req, bool enable)
iowrite32(1, ®->intr_mask);
iowrite32(1, ®->intr);
} else {
+ int irq_vec = pci_irq_vector(bp->pdev, ext->irq_vec);
+
iowrite32(0, ®->intr_mask);
iowrite32(0, ®->enable);
+ ioread32(®->intr_mask);
+ if (irq_vec > 0)
+ synchronize_irq(irq_vec);
}
return 0;
@@ -4867,6 +4872,22 @@ ptp_ocp_detach(struct ptp_ocp *bp)
ptp_ocp_detach_sysfs(bp);
ptp_ocp_attr_group_del(bp);
timer_delete_sync(&bp->watchdog);
+ /* Disable interrupts on all timestampers */
+ if (bp->ts0)
+ ptp_ocp_ts_enable(bp->ts0, 0, false);
+ if (bp->ts1)
+ ptp_ocp_ts_enable(bp->ts1, 0, false);
+ if (bp->ts2)
+ ptp_ocp_ts_enable(bp->ts2, 0, false);
+ if (bp->ts3)
+ ptp_ocp_ts_enable(bp->ts3, 0, false);
+ if (bp->ts4)
+ ptp_ocp_ts_enable(bp->ts4, 0, false);
+ if (bp->pps)
+ ptp_ocp_ts_enable(bp->pps, ~0, false);
+ if (bp->ptp)
+ ptp_clock_unregister(bp->ptp);
+ kfree(bp->ptp_info.pin_config);
ptp_ocp_unregister_ext(bp->ts0);
ptp_ocp_unregister_ext(bp->ts1);
ptp_ocp_unregister_ext(bp->ts2);
@@ -4884,9 +4905,6 @@ ptp_ocp_detach(struct ptp_ocp *bp)
clk_hw_unregister_fixed_rate(bp->i2c_clk);
if (bp->n_irqs)
pci_free_irq_vectors(bp->pdev);
- if (bp->ptp)
- ptp_clock_unregister(bp->ptp);
- kfree(bp->ptp_info.pin_config);
device_unregister(&bp->dev);
}
diff --git a/drivers/slimbus/qcom-ngd-ctrl.c b/drivers/slimbus/qcom-ngd-ctrl.c
index 1ed6be6..3071e46 100644
--- a/drivers/slimbus/qcom-ngd-ctrl.c
+++ b/drivers/slimbus/qcom-ngd-ctrl.c
@@ -1471,15 +1471,12 @@ static int qcom_slim_ngd_ssr_pdr_notify(struct qcom_slim_ngd_ctrl *ctrl,
switch (action) {
case QCOM_SSR_BEFORE_SHUTDOWN:
case SERVREG_SERVICE_STATE_DOWN:
- /* Make sure the last dma xfer is finished */
- mutex_lock(&ctrl->tx_lock);
if (ctrl->state != QCOM_SLIM_NGD_CTRL_DOWN) {
pm_runtime_get_noresume(ctrl->ctrl.dev);
ctrl->state = QCOM_SLIM_NGD_CTRL_DOWN;
qcom_slim_ngd_down(ctrl);
qcom_slim_ngd_exit_dma(ctrl);
}
- mutex_unlock(&ctrl->tx_lock);
break;
case QCOM_SSR_AFTER_POWERUP:
case SERVREG_SERVICE_STATE_UP:
@@ -1542,7 +1539,7 @@ static int of_qcom_slim_ngd_register(struct device *parent,
kfree(ngd);
return ret;
}
- ngd->pdev->dev.of_node = node;
+ ngd->pdev->dev.of_node = of_node_get(node);
ctrl->ngd = ngd;
ret = platform_device_add(ngd->pdev);
@@ -1560,6 +1557,13 @@ static int of_qcom_slim_ngd_register(struct device *parent,
return -ENODEV;
}
+static void qcom_slim_ngd_unregister(struct qcom_slim_ngd_ctrl *ctrl)
+{
+ struct qcom_slim_ngd *ngd = ctrl->ngd;
+
+ platform_device_del(ngd->pdev);
+}
+
static int qcom_slim_ngd_probe(struct platform_device *pdev)
{
struct device *dev = &pdev->dev;
@@ -1577,24 +1581,10 @@ static int qcom_slim_ngd_probe(struct platform_device *pdev)
ret = qcom_slim_ngd_qmi_svc_event_init(ctrl);
if (ret) {
dev_err(&pdev->dev, "QMI service registration failed:%d", ret);
- return ret;
+ pm_runtime_dont_use_autosuspend(dev);
+ pm_runtime_disable(dev);
}
- INIT_WORK(&ctrl->m_work, qcom_slim_ngd_master_worker);
- INIT_WORK(&ctrl->ngd_up_work, qcom_slim_ngd_up_worker);
- ctrl->mwq = create_singlethread_workqueue("ngd_master");
- if (!ctrl->mwq) {
- dev_err(&pdev->dev, "Failed to start master worker\n");
- ret = -ENOMEM;
- goto wq_err;
- }
-
- return 0;
-wq_err:
- qcom_slim_ngd_qmi_svc_event_deinit(&ctrl->qmi);
- if (ctrl->mwq)
- destroy_workqueue(ctrl->mwq);
-
return ret;
}
@@ -1602,6 +1592,7 @@ static int qcom_slim_ngd_ctrl_probe(struct platform_device *pdev)
{
struct device *dev = &pdev->dev;
struct qcom_slim_ngd_ctrl *ctrl;
+ int irq;
int ret;
struct pdr_service *pds;
@@ -1615,20 +1606,16 @@ static int qcom_slim_ngd_ctrl_probe(struct platform_device *pdev)
if (IS_ERR(ctrl->base))
return PTR_ERR(ctrl->base);
- ret = platform_get_irq(pdev, 0);
- if (ret < 0)
- return ret;
+ irq = platform_get_irq(pdev, 0);
+ if (irq < 0)
+ return irq;
- ret = devm_request_irq(dev, ret, qcom_slim_ngd_interrupt,
- IRQF_TRIGGER_HIGH, "slim-ngd", ctrl);
+ ret = devm_request_irq(dev, irq, qcom_slim_ngd_interrupt,
+ IRQF_TRIGGER_HIGH | IRQF_NO_AUTOEN,
+ "slim-ngd", ctrl);
if (ret)
return dev_err_probe(&pdev->dev, ret, "request IRQ failed\n");
- ctrl->nb.notifier_call = qcom_slim_ngd_ssr_notify;
- ctrl->notifier = qcom_register_ssr_notifier("lpass", &ctrl->nb);
- if (IS_ERR(ctrl->notifier))
- return PTR_ERR(ctrl->notifier);
-
ctrl->dev = dev;
ctrl->framer.rootfreq = SLIM_ROOT_FREQ >> 3;
ctrl->framer.superfreq =
@@ -1649,48 +1636,71 @@ static int qcom_slim_ngd_ctrl_probe(struct platform_device *pdev)
init_completion(&ctrl->qmi.qmi_comp);
init_completion(&ctrl->qmi_up);
+ INIT_WORK(&ctrl->m_work, qcom_slim_ngd_master_worker);
+ INIT_WORK(&ctrl->ngd_up_work, qcom_slim_ngd_up_worker);
+
+ ctrl->mwq = create_singlethread_workqueue("ngd_master");
+ if (!ctrl->mwq)
+ return dev_err_probe(dev, -ENOMEM, "Failed to start master worker\n");
+
ctrl->pdr = pdr_handle_alloc(slim_pd_status, ctrl);
if (IS_ERR(ctrl->pdr)) {
- ret = dev_err_probe(dev, PTR_ERR(ctrl->pdr),
- "Failed to init PDR handle\n");
- goto err_pdr_alloc;
+ ret = dev_err_probe(dev, PTR_ERR(ctrl->pdr), "Failed to init PDR handle\n");
+ goto err_destroy_mwq;
}
+ ret = of_qcom_slim_ngd_register(dev, ctrl);
+ if (ret)
+ goto err_pdr_release;
+
pds = pdr_add_lookup(ctrl->pdr, "avs/audio", "msm/adsp/audio_pd");
if (IS_ERR(pds) && PTR_ERR(pds) != -EALREADY) {
ret = dev_err_probe(dev, PTR_ERR(pds), "pdr add lookup failed\n");
- goto err_pdr_lookup;
+ goto err_unregister_ngd;
}
- platform_driver_register(&qcom_slim_ngd_driver);
- return of_qcom_slim_ngd_register(dev, ctrl);
+ ctrl->nb.notifier_call = qcom_slim_ngd_ssr_notify;
+ ctrl->notifier = qcom_register_ssr_notifier("lpass", &ctrl->nb);
+ if (IS_ERR(ctrl->notifier)) {
+ ret = PTR_ERR(ctrl->notifier);
+ goto err_unregister_ngd;
+ }
-err_pdr_alloc:
- qcom_unregister_ssr_notifier(ctrl->notifier, &ctrl->nb);
+ enable_irq(irq);
-err_pdr_lookup:
+ return 0;
+
+err_unregister_ngd:
+ qcom_slim_ngd_unregister(ctrl);
+err_pdr_release:
pdr_handle_release(ctrl->pdr);
+err_destroy_mwq:
+ destroy_workqueue(ctrl->mwq);
return ret;
}
static void qcom_slim_ngd_ctrl_remove(struct platform_device *pdev)
{
- platform_driver_unregister(&qcom_slim_ngd_driver);
+ struct qcom_slim_ngd_ctrl *ctrl = platform_get_drvdata(pdev);
+
+ pdr_handle_release(ctrl->pdr);
+ qcom_unregister_ssr_notifier(ctrl->notifier, &ctrl->nb);
+
+ qcom_slim_ngd_unregister(ctrl);
+
+ destroy_workqueue(ctrl->mwq);
}
static void qcom_slim_ngd_remove(struct platform_device *pdev)
{
struct qcom_slim_ngd_ctrl *ctrl = platform_get_drvdata(pdev);
+ pm_runtime_dont_use_autosuspend(&pdev->dev);
pm_runtime_disable(&pdev->dev);
- pdr_handle_release(ctrl->pdr);
- qcom_unregister_ssr_notifier(ctrl->notifier, &ctrl->nb);
qcom_slim_ngd_enable(ctrl, false);
qcom_slim_ngd_exit_dma(ctrl);
qcom_slim_ngd_qmi_svc_event_deinit(&ctrl->qmi);
- if (ctrl->mwq)
- destroy_workqueue(ctrl->mwq);
kfree(ctrl->ngd);
ctrl->ngd = NULL;
@@ -1752,6 +1762,28 @@ static struct platform_driver qcom_slim_ngd_driver = {
},
};
-module_platform_driver(qcom_slim_ngd_ctrl_driver);
+static int qcom_slim_ngd_init(void)
+{
+ int ret;
+
+ ret = platform_driver_register(&qcom_slim_ngd_driver);
+ if (ret)
+ return ret;
+
+ ret = platform_driver_register(&qcom_slim_ngd_ctrl_driver);
+ if (ret)
+ platform_driver_unregister(&qcom_slim_ngd_driver);
+
+ return ret;
+}
+
+static void qcom_slim_ngd_exit(void)
+{
+ platform_driver_unregister(&qcom_slim_ngd_ctrl_driver);
+ platform_driver_unregister(&qcom_slim_ngd_driver);
+}
+
+module_init(qcom_slim_ngd_init);
+module_exit(qcom_slim_ngd_exit);
MODULE_LICENSE("GPL v2");
MODULE_DESCRIPTION("Qualcomm SLIMBus NGD controller");
diff --git a/drivers/soc/microchip/mpfs-sys-controller.c b/drivers/soc/microchip/mpfs-sys-controller.c
index 92d1142..0400a01 100644
--- a/drivers/soc/microchip/mpfs-sys-controller.c
+++ b/drivers/soc/microchip/mpfs-sys-controller.c
@@ -158,8 +158,8 @@ static int mpfs_sys_controller_probe(struct platform_device *pdev)
of_data = (struct mpfs_syscon_config *) device_get_match_data(dev);
if (!of_data) {
- dev_err(dev, "Error getting match data\n");
- return -EINVAL;
+ ret = dev_err_probe(dev, -EINVAL, "Error getting match data\n");
+ goto out_free_channel;
}
for (i = 0; i < of_data->nb_subdevs; i++) {
@@ -173,6 +173,8 @@ static int mpfs_sys_controller_probe(struct platform_device *pdev)
return 0;
+out_free_channel:
+ mbox_free_channel(sys_controller->chan);
out_free:
kfree(sys_controller);
return ret;
diff --git a/drivers/soc/qcom/qcom_aoss.c b/drivers/soc/qcom/qcom_aoss.c
index c255662..259c41f0 100644
--- a/drivers/soc/qcom/qcom_aoss.c
+++ b/drivers/soc/qcom/qcom_aoss.c
@@ -381,7 +381,7 @@ static int qmp_cooling_device_add(struct qmp *qmp,
qmp_cdev->qmp = qmp;
qmp_cdev->state = !qmp_cdev_max_state;
qmp_cdev->name = cdev_name;
- qmp_cdev->cdev = devm_thermal_of_cooling_device_register
+ qmp_cdev->cdev = devm_thermal_of_child_cooling_device_register
(qmp->dev, node,
cdev_name,
qmp_cdev, &qmp_cooling_device_ops);
diff --git a/drivers/spi/spi-dw-core.c b/drivers/spi/spi-dw-core.c
index b476378..4672bc2 100644
--- a/drivers/spi/spi-dw-core.c
+++ b/drivers/spi/spi-dw-core.c
@@ -472,7 +472,9 @@ static inline void dw_spi_abort(struct spi_controller *ctlr)
if (dws->dma_mapped)
dws->dma_ops->dma_stop(dws);
+ disable_irq(dws->irq);
dw_spi_reset_chip(dws);
+ enable_irq(dws->irq);
}
static void dw_spi_handle_err(struct spi_controller *ctlr,
diff --git a/drivers/spi/spi-geni-qcom.c b/drivers/spi/spi-geni-qcom.c
index d5fb0ed..23c6d3a 100644
--- a/drivers/spi/spi-geni-qcom.c
+++ b/drivers/spi/spi-geni-qcom.c
@@ -440,10 +440,15 @@ static int setup_gsi_xfer(struct spi_transfer *xfer, struct spi_geni_master *mas
return ret;
}
- if (!xfer->cs_change) {
- if (!list_is_last(&xfer->transfer_list, &spi->cur_msg->transfers))
- peripheral.fragmentation = FRAGMENTATION;
- }
+ /*
+ * Set fragmentation to keep CS asserted after this transfer when:
+ * - non-last transfer with cs_change=0: keep CS asserted between chained transfers
+ * - last transfer with cs_change=1: keep CS asserted after the message
+ * (e.g. TPM TIS SPI uses cs_change=1 on single-transfer messages to
+ * keep CS asserted across header, wait-state and data phases)
+ */
+ peripheral.fragmentation = list_is_last(&xfer->transfer_list, &spi->cur_msg->transfers) ?
+ xfer->cs_change : !xfer->cs_change;
if (peripheral.cmd & SPI_RX) {
dmaengine_slave_config(mas->rx, &config);
@@ -849,10 +854,16 @@ static int setup_se_xfer(struct spi_transfer *xfer,
mas->cur_xfer_mode = GENI_SE_DMA;
geni_se_select_mode(se, mas->cur_xfer_mode);
- if (!xfer->cs_change) {
- if (!list_is_last(&xfer->transfer_list, &spi->cur_msg->transfers))
- m_params = FRAGMENTATION;
- }
+ /*
+ * Set FRAGMENTATION to keep CS asserted after this transfer when:
+ * - non-last transfer with cs_change=0: keep CS asserted between chained transfers
+ * - last transfer with cs_change=1: keep CS asserted after the message
+ * (e.g. TPM TIS SPI uses cs_change=1 on single-transfer messages to
+ * keep CS asserted across header, wait-state and data phases)
+ */
+ if (list_is_last(&xfer->transfer_list, &spi->cur_msg->transfers) ?
+ xfer->cs_change : !xfer->cs_change)
+ m_params = FRAGMENTATION;
/*
* Lock around right before we start the transfer since our
diff --git a/drivers/spi/spi-rzv2h-rspi.c b/drivers/spi/spi-rzv2h-rspi.c
index 1655efd..6ed3fad 100644
--- a/drivers/spi/spi-rzv2h-rspi.c
+++ b/drivers/spi/spi-rzv2h-rspi.c
@@ -135,8 +135,9 @@ static inline void rzv2h_rspi_rx_##type(struct rzv2h_rspi_priv *rspi, \
RZV2H_RSPI_TX(writel, u32)
RZV2H_RSPI_TX(writew, u16)
RZV2H_RSPI_TX(writeb, u8)
+/* The read access size for RSPI_SPDR is fixed at 32 bits */
RZV2H_RSPI_RX(readl, u32)
-RZV2H_RSPI_RX(readw, u16)
+RZV2H_RSPI_RX(readl, u16)
RZV2H_RSPI_RX(readl, u8)
static void rzv2h_rspi_reg_rmw(const struct rzv2h_rspi_priv *rspi,
diff --git a/drivers/staging/rtl8723bs/core/rtw_mlme.c b/drivers/staging/rtl8723bs/core/rtw_mlme.c
index ddfc56f..9f21a22 100644
--- a/drivers/staging/rtl8723bs/core/rtw_mlme.c
+++ b/drivers/staging/rtl8723bs/core/rtw_mlme.c
@@ -464,8 +464,11 @@ static void update_current_network(struct adapter *adapter, struct wlan_bssid_ex
if (check_fwstate(pmlmepriv, _FW_LINKED) && (is_same_network(&pmlmepriv->cur_network.network, pnetwork, 0))) {
update_network(&pmlmepriv->cur_network.network, pnetwork, adapter, true);
+ if (pmlmepriv->cur_network.network.ie_length < sizeof(struct ndis_802_11_fix_ie))
+ return;
+
rtw_update_protection(adapter, (pmlmepriv->cur_network.network.ies) + sizeof(struct ndis_802_11_fix_ie),
- pmlmepriv->cur_network.network.ie_length);
+ pmlmepriv->cur_network.network.ie_length - sizeof(struct ndis_802_11_fix_ie));
}
}
@@ -601,6 +604,8 @@ static bool rtw_is_desired_network(struct adapter *adapter, struct wlan_network
privacy = pnetwork->network.privacy;
if (check_fwstate(pmlmepriv, WIFI_UNDER_WPS)) {
+ if (pnetwork->network.ie_length < _FIXED_IE_LENGTH_)
+ return false;
if (rtw_get_wps_ie(pnetwork->network.ies + _FIXED_IE_LENGTH_, pnetwork->network.ie_length - _FIXED_IE_LENGTH_, NULL, &wps_ielen))
return true;
else
@@ -614,11 +619,15 @@ static bool rtw_is_desired_network(struct adapter *adapter, struct wlan_network
bselected = false;
if (psecuritypriv->ndisauthtype == Ndis802_11AuthModeWPA2PSK) {
- p = rtw_get_ie(pnetwork->network.ies + _BEACON_IE_OFFSET_, WLAN_EID_RSN, &ie_len, (pnetwork->network.ie_length - _BEACON_IE_OFFSET_));
- if (p && ie_len > 0)
- bselected = true;
- else
+ if (pnetwork->network.ie_length < _BEACON_IE_OFFSET_) {
bselected = false;
+ } else {
+ p = rtw_get_ie(pnetwork->network.ies + _BEACON_IE_OFFSET_, WLAN_EID_RSN, &ie_len, (pnetwork->network.ie_length - _BEACON_IE_OFFSET_));
+ if (p && ie_len > 0)
+ bselected = true;
+ else
+ bselected = false;
+ }
}
}
@@ -1072,8 +1081,11 @@ static void rtw_joinbss_update_network(struct adapter *padapter, struct wlan_net
break;
}
+ if (cur_network->network.ie_length < sizeof(struct ndis_802_11_fix_ie))
+ return;
+
rtw_update_protection(padapter, (cur_network->network.ies) + sizeof(struct ndis_802_11_fix_ie),
- (cur_network->network.ie_length));
+ (cur_network->network.ie_length - sizeof(struct ndis_802_11_fix_ie)));
rtw_update_ht_cap(padapter, cur_network->network.ies, cur_network->network.ie_length, (u8) cur_network->network.configuration.ds_config);
}
diff --git a/drivers/thermal/Kconfig b/drivers/thermal/Kconfig
index b10080d..810eecc 100644
--- a/drivers/thermal/Kconfig
+++ b/drivers/thermal/Kconfig
@@ -436,6 +436,7 @@
tristate "Amlogic Thermal Support"
default ARCH_MESON
depends on OF && ARCH_MESON
+ depends on MESON_SM
help
If you say yes here you get support for Amlogic Thermal
for G12 SoC Family.
@@ -472,6 +473,8 @@
source "drivers/thermal/renesas/Kconfig"
+source "drivers/thermal/spacemit/Kconfig"
+
source "drivers/thermal/tegra/Kconfig"
config GENERIC_ADC_THERMAL
diff --git a/drivers/thermal/Makefile b/drivers/thermal/Makefile
index bb21e7e..3b24919 100644
--- a/drivers/thermal/Makefile
+++ b/drivers/thermal/Makefile
@@ -65,6 +65,7 @@
obj-$(CONFIG_GENERIC_ADC_THERMAL) += thermal-generic-adc.o
obj-$(CONFIG_UNIPHIER_THERMAL) += uniphier_thermal.o
obj-$(CONFIG_AMLOGIC_THERMAL) += amlogic_thermal.o
+obj-y += spacemit/
obj-$(CONFIG_SPRD_THERMAL) += sprd_thermal.o
obj-$(CONFIG_KHADAS_MCU_FAN_THERMAL) += khadas_mcu_fan.o
obj-$(CONFIG_LOONGSON2_THERMAL) += loongson2_thermal.o
diff --git a/drivers/thermal/amlogic_thermal.c b/drivers/thermal/amlogic_thermal.c
index 5448d77..a0b5306 100644
--- a/drivers/thermal/amlogic_thermal.c
+++ b/drivers/thermal/amlogic_thermal.c
@@ -25,6 +25,7 @@
#include <linux/platform_device.h>
#include <linux/regmap.h>
#include <linux/thermal.h>
+#include <linux/firmware/meson/meson_sm.h>
#include "thermal_hwmon.h"
@@ -84,12 +85,14 @@ struct amlogic_thermal_soc_calib_data {
* @u_efuse_off: register offset to read fused calibration value
* @calibration_parameters: calibration parameters structure pointer
* @regmap_config: regmap config for the device
+ * @use_sm: read data from secure monitor instead of efuse
* This structure is required for configuration of amlogic thermal driver.
*/
struct amlogic_thermal_data {
int u_efuse_off;
const struct amlogic_thermal_soc_calib_data *calibration_parameters;
const struct regmap_config *regmap_config;
+ bool use_sm;
};
struct amlogic_thermal {
@@ -100,6 +103,8 @@ struct amlogic_thermal {
struct clk *clk;
struct thermal_zone_device *tzd;
u32 trim_info;
+ struct meson_sm_firmware *sm_fw;
+ u32 tsensor_id;
};
/*
@@ -133,26 +138,6 @@ static int amlogic_thermal_code_to_millicelsius(struct amlogic_thermal *pdata,
return temp;
}
-static int amlogic_thermal_initialize(struct amlogic_thermal *pdata)
-{
- int ret = 0;
- int ver;
-
- regmap_read(pdata->sec_ao_map, pdata->data->u_efuse_off,
- &pdata->trim_info);
-
- ver = TSENSOR_TRIM_VERSION(pdata->trim_info);
-
- if ((ver & TSENSOR_TRIM_CALIB_VALID_MASK) == 0) {
- ret = -EINVAL;
- dev_err(&pdata->pdev->dev,
- "tsensor thermal calibration not supported: 0x%x!\n",
- ver);
- }
-
- return ret;
-}
-
static int amlogic_thermal_enable(struct amlogic_thermal *data)
{
int ret;
@@ -190,6 +175,67 @@ static int amlogic_thermal_get_temp(struct thermal_zone_device *tz, int *temp)
return 0;
}
+static int amlogic_thermal_probe_sm(struct platform_device *pdev,
+ struct amlogic_thermal *pdata)
+{
+ struct device *dev = &pdev->dev;
+ struct of_phandle_args ph_args;
+ int ret;
+
+ ret = of_parse_phandle_with_fixed_args(pdev->dev.of_node,
+ "amlogic,secure-monitor",
+ 1, 0, &ph_args);
+ if (ret)
+ return ret;
+
+ if (!ph_args.np) {
+ dev_err(dev, "Failed to parse secure monitor phandle\n");
+ return -ENODEV;
+ }
+
+ pdata->sm_fw = meson_sm_get(ph_args.np);
+ of_node_put(ph_args.np);
+ if (!pdata->sm_fw) {
+ dev_err(dev, "Failed to get secure monitor firmware\n");
+ return -EPROBE_DEFER;
+ }
+
+ pdata->tsensor_id = ph_args.args[0];
+
+ return meson_sm_get_thermal_calib(pdata->sm_fw,
+ &pdata->trim_info,
+ pdata->tsensor_id);
+}
+
+static int amlogic_thermal_probe_syscon(struct platform_device *pdev,
+ struct amlogic_thermal *pdata)
+{
+ struct device *dev = &pdev->dev;
+ int ver;
+
+ pdata->sec_ao_map =
+ syscon_regmap_lookup_by_phandle(pdev->dev.of_node,
+ "amlogic,ao-secure");
+ if (IS_ERR(pdata->sec_ao_map)) {
+ dev_err(dev, "syscon regmap lookup failed.\n");
+ return PTR_ERR(pdata->sec_ao_map);
+ }
+
+ regmap_read(pdata->sec_ao_map, pdata->data->u_efuse_off,
+ &pdata->trim_info);
+
+ ver = TSENSOR_TRIM_VERSION(pdata->trim_info);
+
+ if ((ver & TSENSOR_TRIM_CALIB_VALID_MASK) == 0) {
+ dev_err(&pdata->pdev->dev,
+ "tsensor thermal calibration not supported: 0x%x!\n",
+ ver);
+ return -EINVAL;
+ }
+
+ return 0;
+}
+
static const struct thermal_zone_device_ops amlogic_thermal_ops = {
.get_temp = amlogic_thermal_get_temp,
};
@@ -226,6 +272,12 @@ static const struct amlogic_thermal_data amlogic_thermal_a1_cpu_param = {
.regmap_config = &amlogic_thermal_regmap_config_g12a,
};
+static const struct amlogic_thermal_data amlogic_thermal_t7_param = {
+ .use_sm = true,
+ .calibration_parameters = &amlogic_thermal_g12a,
+ .regmap_config = &amlogic_thermal_regmap_config_g12a,
+};
+
static const struct of_device_id of_amlogic_thermal_match[] = {
{
.compatible = "amlogic,g12a-ddr-thermal",
@@ -239,6 +291,10 @@ static const struct of_device_id of_amlogic_thermal_match[] = {
.compatible = "amlogic,a1-cpu-thermal",
.data = &amlogic_thermal_a1_cpu_param,
},
+ {
+ .compatible = "amlogic,t7-thermal",
+ .data = &amlogic_thermal_t7_param,
+ },
{ /* sentinel */ }
};
MODULE_DEVICE_TABLE(of, of_amlogic_thermal_match);
@@ -271,12 +327,12 @@ static int amlogic_thermal_probe(struct platform_device *pdev)
if (IS_ERR(pdata->clk))
return dev_err_probe(dev, PTR_ERR(pdata->clk), "failed to get clock\n");
- pdata->sec_ao_map = syscon_regmap_lookup_by_phandle
- (pdev->dev.of_node, "amlogic,ao-secure");
- if (IS_ERR(pdata->sec_ao_map)) {
- dev_err(dev, "syscon regmap lookup failed.\n");
- return PTR_ERR(pdata->sec_ao_map);
- }
+ if (pdata->data->use_sm)
+ ret = amlogic_thermal_probe_sm(pdev, pdata);
+ else
+ ret = amlogic_thermal_probe_syscon(pdev, pdata);
+ if (ret)
+ return ret;
pdata->tzd = devm_thermal_of_zone_register(&pdev->dev,
0,
@@ -290,10 +346,6 @@ static int amlogic_thermal_probe(struct platform_device *pdev)
devm_thermal_add_hwmon_sysfs(&pdev->dev, pdata->tzd);
- ret = amlogic_thermal_initialize(pdata);
- if (ret)
- return ret;
-
ret = amlogic_thermal_enable(pdata);
return ret;
diff --git a/drivers/thermal/cpufreq_cooling.c b/drivers/thermal/cpufreq_cooling.c
index 32bf5ab..768859a 100644
--- a/drivers/thermal/cpufreq_cooling.c
+++ b/drivers/thermal/cpufreq_cooling.c
@@ -592,7 +592,7 @@ __cpufreq_cooling_register(struct device_node *np,
if (!name)
goto remove_qos_req;
- cdev = thermal_of_cooling_device_register(np, name, cpufreq_cdev,
+ cdev = thermal_of_cooling_device_register(np, 0, name, cpufreq_cdev,
cooling_ops);
kfree(name);
diff --git a/drivers/thermal/cpuidle_cooling.c b/drivers/thermal/cpuidle_cooling.c
index 425f596..bbd2e91 100644
--- a/drivers/thermal/cpuidle_cooling.c
+++ b/drivers/thermal/cpuidle_cooling.c
@@ -207,7 +207,7 @@ static int __cpuidle_cooling_register(struct device_node *np,
goto out_unregister;
}
- cdev = thermal_of_cooling_device_register(np, name, idle_cdev,
+ cdev = thermal_of_cooling_device_register(np, 0, name, idle_cdev,
&cpuidle_cooling_ops);
if (IS_ERR(cdev)) {
ret = PTR_ERR(cdev);
diff --git a/drivers/thermal/devfreq_cooling.c b/drivers/thermal/devfreq_cooling.c
index 1c7dffc..0330a81 100644
--- a/drivers/thermal/devfreq_cooling.c
+++ b/drivers/thermal/devfreq_cooling.c
@@ -454,7 +454,7 @@ of_devfreq_cooling_register_power(struct device_node *np, struct devfreq *df,
if (!name)
goto remove_qos_req;
- cdev = thermal_of_cooling_device_register(np, name, dfc, ops);
+ cdev = thermal_of_cooling_device_register(np, 0, name, dfc, ops);
kfree(name);
if (IS_ERR(cdev)) {
diff --git a/drivers/thermal/imx_thermal.c b/drivers/thermal/imx_thermal.c
index 38c993d..5aaacbc 100644
--- a/drivers/thermal/imx_thermal.c
+++ b/drivers/thermal/imx_thermal.c
@@ -693,8 +693,8 @@ static int imx_thermal_probe(struct platform_device *pdev)
goto clk_disable;
}
- dev_info(dev, "%s CPU temperature grade - max:%dC"
- " critical:%dC passive:%dC\n", data->temp_grade,
+ dev_info(dev, "%s CPU temperature grade - max:%dC critical:%dC passive:%dC\n",
+ data->temp_grade,
data->temp_max / 1000, trips[IMX_TRIP_CRITICAL].temperature / 1000,
trips[IMX_TRIP_PASSIVE].temperature / 1000);
diff --git a/drivers/thermal/intel/int340x_thermal/platform_temperature_control.c b/drivers/thermal/intel/int340x_thermal/platform_temperature_control.c
index 0ccc72c..d92a6f8 100644
--- a/drivers/thermal/intel/int340x_thermal/platform_temperature_control.c
+++ b/drivers/thermal/intel/int340x_thermal/platform_temperature_control.c
@@ -138,7 +138,7 @@ static void ptc_mmio_write(struct pci_dev *pdev, u32 offset, int index, u32 valu
reg_val = readq((void __iomem *) (proc_priv->mmio_base + offset));
reg_val &= ~mask;
- reg_val |= (value << ptc_mmio_regs[index].shift);
+ reg_val |= ((u64)value << ptc_mmio_regs[index].shift);
writeq(reg_val, (void __iomem *) (proc_priv->mmio_base + offset));
}
@@ -278,12 +278,18 @@ static void ptc_delete_debugfs(void)
int proc_thermal_ptc_add(struct pci_dev *pdev, struct proc_thermal_device *proc_priv)
{
if (proc_priv->mmio_feature_mask & PROC_THERMAL_FEATURE_PTC) {
- int i;
+ int i, ret;
for (i = 0; i < PTC_MAX_INSTANCES; i++) {
ptc_instance[i].offset = ptc_offsets[i];
ptc_instance[i].pdev = pdev;
- ptc_create_groups(pdev, i, &ptc_instance[i]);
+ ret = ptc_create_groups(pdev, i, &ptc_instance[i]);
+ if (ret) {
+ while (i--)
+ sysfs_remove_group(&pdev->dev.kobj,
+ &ptc_instance[i].ptc_attr_group);
+ return ret;
+ }
}
ptc_create_debugfs();
diff --git a/drivers/thermal/intel/intel_powerclamp.c b/drivers/thermal/intel/intel_powerclamp.c
index ccf380d..bd7fd98 100644
--- a/drivers/thermal/intel/intel_powerclamp.c
+++ b/drivers/thermal/intel/intel_powerclamp.c
@@ -200,7 +200,7 @@ static int cpumask_get(char *buf, const struct kernel_param *kp)
if (!cpumask_available(idle_injection_cpu_mask))
return -ENODEV;
- return cpumap_print_to_pagebuf(false, buf, idle_injection_cpu_mask);
+ return sysfs_emit(buf, "%*pb\n", cpumask_pr_args(idle_injection_cpu_mask));
}
static const struct kernel_param_ops cpumask_ops = {
diff --git a/drivers/thermal/intel/intel_tcc_cooling.c b/drivers/thermal/intel/intel_tcc_cooling.c
index 6c2ba0b..52ea4f7 100644
--- a/drivers/thermal/intel/intel_tcc_cooling.c
+++ b/drivers/thermal/intel/intel_tcc_cooling.c
@@ -65,6 +65,9 @@ static const struct x86_cpu_id tcc_ids[] __initconst = {
X86_MATCH_VFM(INTEL_RAPTORLAKE, NULL),
X86_MATCH_VFM(INTEL_RAPTORLAKE_P, NULL),
X86_MATCH_VFM(INTEL_RAPTORLAKE_S, NULL),
+ X86_MATCH_VFM(INTEL_ARROWLAKE, NULL),
+ X86_MATCH_VFM(INTEL_ARROWLAKE_U, NULL),
+ X86_MATCH_VFM(INTEL_ARROWLAKE_H, NULL),
X86_MATCH_VFM(INTEL_PANTHERLAKE_L, NULL),
X86_MATCH_VFM(INTEL_WILDCATLAKE_L, NULL),
X86_MATCH_VFM(INTEL_NOVALAKE, NULL),
diff --git a/drivers/thermal/khadas_mcu_fan.c b/drivers/thermal/khadas_mcu_fan.c
index d35e531..21b3d0a 100644
--- a/drivers/thermal/khadas_mcu_fan.c
+++ b/drivers/thermal/khadas_mcu_fan.c
@@ -90,9 +90,10 @@ static int khadas_mcu_fan_probe(struct platform_device *pdev)
ctx->mcu = mcu;
platform_set_drvdata(pdev, ctx);
- cdev = devm_thermal_of_cooling_device_register(dev->parent,
- dev->parent->of_node, "khadas-mcu-fan", ctx,
- &khadas_mcu_fan_cooling_ops);
+ cdev = devm_thermal_of_child_cooling_device_register(dev->parent,
+ dev->parent->of_node,
+ "khadas-mcu-fan", ctx,
+ &khadas_mcu_fan_cooling_ops);
if (IS_ERR(cdev)) {
ret = PTR_ERR(cdev);
dev_err(dev, "Failed to register khadas-mcu-fan as cooling device: %d\n",
diff --git a/drivers/thermal/qcom/tsens-v2.c b/drivers/thermal/qcom/tsens-v2.c
index 8d9698e..2ee117a 100644
--- a/drivers/thermal/qcom/tsens-v2.c
+++ b/drivers/thermal/qcom/tsens-v2.c
@@ -263,7 +263,6 @@ static int __init init_tsens_v2_no_rpm(struct tsens_priv *priv)
static const struct tsens_ops ops_generic_v2 = {
.init = init_common,
.get_temp = get_temp_tsens_valid,
- .resume = tsens_resume_common,
};
struct tsens_plat_data data_tsens_v2 = {
@@ -307,3 +306,11 @@ struct tsens_plat_data data_8996 = {
.feat = &tsens_v2_feat,
.fields = tsens_v2_regfields,
};
+
+/* Do not enable wakeup capable interrupts for automotive platforms */
+struct tsens_plat_data data_automotive_v2 = {
+ .ops = &ops_generic_v2,
+ .feat = &tsens_v2_feat,
+ .fields = tsens_v2_regfields,
+ .no_irq_wake = true,
+};
diff --git a/drivers/thermal/qcom/tsens.c b/drivers/thermal/qcom/tsens.c
index a2422eb..6e3714e 100644
--- a/drivers/thermal/qcom/tsens.c
+++ b/drivers/thermal/qcom/tsens.c
@@ -27,7 +27,7 @@
* @up_viol: upper threshold violated
* @up_thresh: upper threshold temperature value
* @up_irq_mask: mask register for upper threshold irqs
- * @up_irq_clear: clear register for uppper threshold irqs
+ * @up_irq_clear: clear register for upper threshold irqs
* @low_viol: lower threshold violated
* @low_thresh: lower threshold temperature value
* @low_irq_mask: mask register for lower threshold irqs
@@ -316,9 +316,66 @@ static inline int code_to_degc(u32 adc_code, const struct tsens_sensor *s)
}
/**
- * tsens_hw_to_mC - Return sign-extended temperature in mCelsius.
+ * tsens_read_temp - Retrieve temperature readings from the hardware.
* @s: Pointer to sensor struct
* @field: Index into regmap_field array pointing to temperature data
+ * @temp: temperature in deciCelsius to be read from hardware
+ *
+ * This function handles temperature returned in ADC code or deciCelsius
+ * depending on IP version.
+ *
+ * Return: 0 on success, a negative errno will be returned in error cases
+ */
+static int tsens_read_temp(const struct tsens_sensor *s, int field, int *temp)
+{
+ struct tsens_priv *priv = s->priv;
+ int temp_val[MAX_READ_RETRY] = {0};
+ u32 status;
+ int ret;
+ u32 last_temp_mask = GENMASK(priv->fields[LAST_TEMP_0].msb,
+ priv->fields[LAST_TEMP_0].lsb);
+ u32 valid_bit = priv->rf[VALID_0] ? BIT(priv->fields[VALID_0].lsb) : 0;
+
+ for (int i = 0; i < MAX_READ_RETRY; i++) {
+ ret = regmap_read(priv->tm_map, priv->fields[field].reg, &status);
+ if (ret)
+ return ret;
+
+ /* VER_0 doesn't have a VALID bit */
+ if (!valid_bit) {
+ *temp = status & last_temp_mask;
+ return 0;
+ }
+
+ temp_val[i] = status & last_temp_mask;
+
+ if (status & valid_bit) {
+ *temp = temp_val[i];
+ return 0;
+ }
+ }
+
+ /*
+ * As per the HW guidelines, if none of the attempts observe a
+ * valid sample, a stable fallback value must be returned. If the
+ * first and second samples match, the second value is returned;
+ * otherwise, if the second and third samples match, the third
+ * value is returned.
+ */
+ if (temp_val[0] == temp_val[1])
+ *temp = temp_val[1];
+ else if (temp_val[1] == temp_val[2])
+ *temp = temp_val[2];
+ else
+ return -EAGAIN;
+
+ return 0;
+}
+
+/**
+ * tsens_hw_to_mC - Return sign-extended temperature in mCelsius.
+ * @s: Pointer to sensor struct
+ * @temp: temperature in milliCelsius to be read from hardware
*
* This function handles temperature returned in ADC code or deciCelsius
* depending on IP version.
@@ -326,20 +383,14 @@ static inline int code_to_degc(u32 adc_code, const struct tsens_sensor *s)
* Return: Temperature in milliCelsius on success, a negative errno will
* be returned in error cases
*/
-static int tsens_hw_to_mC(const struct tsens_sensor *s, int field)
+static int tsens_hw_to_mC(const struct tsens_sensor *s, int temp)
{
struct tsens_priv *priv = s->priv;
u32 resolution;
- u32 temp = 0;
- int ret;
resolution = priv->fields[LAST_TEMP_0].msb -
priv->fields[LAST_TEMP_0].lsb;
- ret = regmap_field_read(priv->rf[field], &temp);
- if (ret)
- return ret;
-
/* Convert temperature from ADC code to milliCelsius */
if (priv->feat->adc)
return code_to_degc(temp, s) * 1000;
@@ -514,8 +565,10 @@ static int tsens_read_irq_state(struct tsens_priv *priv, u32 hw_id,
&d->crit_irq_mask);
if (ret)
return ret;
-
- d->crit_thresh = tsens_hw_to_mC(s, CRIT_THRESH_0 + hw_id);
+ ret = regmap_field_read(priv->rf[CRIT_THRESH_0 + hw_id], &d->crit_thresh);
+ if (ret)
+ return ret;
+ d->crit_thresh = tsens_hw_to_mC(s, d->crit_thresh);
} else {
/* No mask register on older TSENS */
d->up_irq_mask = 0;
@@ -525,8 +578,16 @@ static int tsens_read_irq_state(struct tsens_priv *priv, u32 hw_id,
d->crit_thresh = 0;
}
- d->up_thresh = tsens_hw_to_mC(s, UP_THRESH_0 + hw_id);
- d->low_thresh = tsens_hw_to_mC(s, LOW_THRESH_0 + hw_id);
+ ret = regmap_field_read(priv->rf[UP_THRESH_0 + hw_id], &d->up_thresh);
+ if (ret)
+ return ret;
+
+ d->up_thresh = tsens_hw_to_mC(s, d->up_thresh);
+ ret = regmap_field_read(priv->rf[LOW_THRESH_0 + hw_id], &d->low_thresh);
+ if (ret)
+ return ret;
+
+ d->low_thresh = tsens_hw_to_mC(s, d->low_thresh);
dev_dbg(priv->dev, "[%u] %s%s: status(%u|%u|%u) | clr(%u|%u|%u) | mask(%u|%u|%u)\n",
hw_id, __func__,
@@ -750,33 +811,15 @@ static void tsens_disable_irq(struct tsens_priv *priv)
int get_temp_tsens_valid(const struct tsens_sensor *s, int *temp)
{
- struct tsens_priv *priv = s->priv;
int hw_id = s->hw_id;
u32 temp_idx = LAST_TEMP_0 + hw_id;
- u32 valid_idx = VALID_0 + hw_id;
- u32 valid;
int ret;
- /* VER_0 doesn't have VALID bit */
- if (tsens_version(priv) == VER_0)
- goto get_temp;
+ ret = tsens_read_temp(s, temp_idx, temp);
+ if (!ret)
+ *temp = tsens_hw_to_mC(s, *temp);
- /* Valid bit is 0 for 6 AHB clock cycles.
- * At 19.2MHz, 1 AHB clock is ~60ns.
- * We should enter this loop very, very rarely.
- * Wait 1 us since it's the min of poll_timeout macro.
- * Old value was 400 ns.
- */
- ret = regmap_field_read_poll_timeout(priv->rf[valid_idx], valid,
- valid, 1, 20 * USEC_PER_MSEC);
- if (ret)
- return ret;
-
-get_temp:
- /* Valid bit is set, OK to read the temperature */
- *temp = tsens_hw_to_mC(s, temp_idx);
-
- return 0;
+ return ret;
}
int get_temp_common(const struct tsens_sensor *s, int *temp)
@@ -1086,22 +1129,30 @@ static int tsens_get_temp(struct thermal_zone_device *tz, int *temp)
static int __maybe_unused tsens_suspend(struct device *dev)
{
+ int ret = 0;
struct tsens_priv *priv = dev_get_drvdata(dev);
- if (priv->ops && priv->ops->suspend)
- return priv->ops->suspend(priv);
+ if (priv->ops && priv->ops->suspend) {
+ ret = priv->ops->suspend(priv);
+ if (ret)
+ return ret;
+ }
- return 0;
+ return tsens_suspend_common(priv);
}
static int __maybe_unused tsens_resume(struct device *dev)
{
+ int ret = 0;
struct tsens_priv *priv = dev_get_drvdata(dev);
- if (priv->ops && priv->ops->resume)
- return priv->ops->resume(priv);
+ if (priv->ops && priv->ops->resume) {
+ ret = priv->ops->resume(priv);
+ if (ret)
+ return ret;
+ }
- return 0;
+ return tsens_resume_common(priv);
}
static SIMPLE_DEV_PM_OPS(tsens_pm_ops, tsens_suspend, tsens_resume);
@@ -1161,6 +1212,12 @@ static const struct of_device_id tsens_table[] = {
}, {
.compatible = "qcom,tsens-v2",
.data = &data_tsens_v2,
+ }, {
+ .compatible = "qcom,sa8775p-tsens",
+ .data = &data_automotive_v2,
+ }, {
+ .compatible = "qcom,sa8255p-tsens",
+ .data = &data_automotive_v2,
},
{}
};
@@ -1172,7 +1229,7 @@ static const struct thermal_zone_device_ops tsens_of_ops = {
};
static int tsens_register_irq(struct tsens_priv *priv, char *irqname,
- irq_handler_t thread_fn)
+ irq_handler_t thread_fn, int *irq_num)
{
struct platform_device *pdev;
int ret, irq;
@@ -1205,7 +1262,7 @@ static int tsens_register_irq(struct tsens_priv *priv, char *irqname,
dev_err(&pdev->dev, "%s: failed to get irq\n",
__func__);
else
- enable_irq_wake(irq);
+ *irq_num = irq;
}
put_device(&pdev->dev);
@@ -1232,11 +1289,38 @@ static int tsens_reinit(struct tsens_priv *priv)
return 0;
}
+int tsens_suspend_common(struct tsens_priv *priv)
+{
+ if (!device_may_wakeup(priv->dev))
+ return 0;
+
+ if (priv->feat->combo_int)
+ enable_irq_wake(priv->combined_irq);
+ else {
+ enable_irq_wake(priv->uplow_irq);
+ if (priv->feat->crit_int)
+ enable_irq_wake(priv->crit_irq);
+ }
+
+ return 0;
+}
+
int tsens_resume_common(struct tsens_priv *priv)
{
if (pm_suspend_target_state == PM_SUSPEND_MEM)
tsens_reinit(priv);
+ if (!device_may_wakeup(priv->dev))
+ return 0;
+
+ if (priv->feat->combo_int)
+ disable_irq_wake(priv->combined_irq);
+ else {
+ disable_irq_wake(priv->uplow_irq);
+ if (priv->feat->crit_int)
+ disable_irq_wake(priv->crit_irq);
+ }
+
return 0;
}
@@ -1276,15 +1360,18 @@ static int tsens_register(struct tsens_priv *priv)
if (priv->feat->combo_int) {
ret = tsens_register_irq(priv, "combined",
- tsens_combined_irq_thread);
+ tsens_combined_irq_thread, &priv->combined_irq);
} else {
- ret = tsens_register_irq(priv, "uplow", tsens_irq_thread);
+ ret = tsens_register_irq(priv, "uplow", tsens_irq_thread,
+ &priv->uplow_irq);
if (ret < 0)
return ret;
- if (priv->feat->crit_int)
+ if (priv->feat->crit_int) {
ret = tsens_register_irq(priv, "critical",
- tsens_critical_irq_thread);
+ tsens_critical_irq_thread,
+ &priv->crit_irq);
+ }
}
return ret;
@@ -1343,6 +1430,8 @@ static int tsens_probe(struct platform_device *pdev)
platform_set_drvdata(pdev, priv);
+ device_init_wakeup(dev, !data->no_irq_wake);
+
if (!priv->ops || !priv->ops->init || !priv->ops->get_temp)
return -EINVAL;
diff --git a/drivers/thermal/qcom/tsens.h b/drivers/thermal/qcom/tsens.h
index 2a7afa4..e8376ac 100644
--- a/drivers/thermal/qcom/tsens.h
+++ b/drivers/thermal/qcom/tsens.h
@@ -21,6 +21,7 @@
#define THRESHOLD_MIN_ADC_CODE 0x0
#define MAX_SENSORS 16
+#define MAX_READ_RETRY 3
#include <linux/interrupt.h>
#include <linux/thermal.h>
@@ -531,6 +532,7 @@ struct tsens_features {
* @hw_ids: Subset of sensors ids supported by platform, if not the first n
* @feat: features of the IP
* @fields: bitfield locations
+ * @no_irq_wake: if set, TSENS interrupts will not be configured as wakeup sources
*/
struct tsens_plat_data {
const u32 num_sensors;
@@ -538,6 +540,7 @@ struct tsens_plat_data {
unsigned int *hw_ids;
struct tsens_features *feat;
const struct reg_field *fields;
+ bool no_irq_wake;
};
/**
@@ -567,6 +570,9 @@ struct tsens_context {
* @ops: pointer to list of callbacks supported by this device
* @debug_root: pointer to debugfs dentry for all tsens
* @debug: pointer to debugfs dentry for tsens controller
+ * @uplow_irq: IRQ number for uplow (upper/lower) threshold interrupts
+ * @crit_irq: IRQ number for critical threshold interrupts
+ * @combined_irq: IRQ number for combined threshold interrupts
* @sensor: list of sensors attached to this device
*/
struct tsens_priv {
@@ -588,6 +594,10 @@ struct tsens_priv {
struct dentry *debug_root;
struct dentry *debug;
+ int uplow_irq;
+ int crit_irq;
+ int combined_irq;
+
struct tsens_sensor sensor[] __counted_by(num_sensors);
};
@@ -639,8 +649,17 @@ int get_temp_tsens_valid(const struct tsens_sensor *s, int *temp);
int get_temp_common(const struct tsens_sensor *s, int *temp);
#ifdef CONFIG_SUSPEND
int tsens_resume_common(struct tsens_priv *priv);
+int tsens_suspend_common(struct tsens_priv *priv);
#else
-#define tsens_resume_common NULL
+static inline int tsens_resume_common(struct tsens_priv *priv)
+{
+ return 0;
+}
+
+static inline int tsens_suspend_common(struct tsens_priv *priv)
+{
+ return 0;
+}
#endif
/* TSENS target */
@@ -659,4 +678,7 @@ extern const struct tsens_plat_data data_ipq5018;
extern struct tsens_plat_data data_8996, data_ipq8074, data_tsens_v2;
extern const struct tsens_plat_data data_ipq5332, data_ipq5424;
+/* TSENS automotive targets */
+extern struct tsens_plat_data data_automotive_v2;
+
#endif /* __QCOM_TSENS_H__ */
diff --git a/drivers/thermal/qoriq_thermal.c b/drivers/thermal/qoriq_thermal.c
index 01b58be..35439ec 100644
--- a/drivers/thermal/qoriq_thermal.c
+++ b/drivers/thermal/qoriq_thermal.c
@@ -1,7 +1,9 @@
// SPDX-License-Identifier: GPL-2.0
//
// Copyright 2016 Freescale Semiconductor, Inc.
+// Copyright 2025 NXP
+#include <linux/bitfield.h>
#include <linux/clk.h>
#include <linux/err.h>
#include <linux/io.h>
@@ -24,10 +26,14 @@
#define TMTMIR_DEFAULT 0x0000000f
#define TIER_DISABLE 0x0
#define TEUMR0_V2 0x51009c00
+#define TEUMR0_V21 0x55000c00
#define TMSARA_V2 0xe
#define TMU_VER1 0x1
#define TMU_VER2 0x2
+/* errata ID info define */
+#define TMU_ERR052243 BIT(0)
+
#define REGS_TMR 0x000 /* Mode Register */
#define TMR_DISABLE 0x0
#define TMR_ME 0x80000000
@@ -43,6 +49,15 @@
#define REGS_TIER 0x020 /* Interrupt Enable Register */
#define TIER_DISABLE 0x0
+#define REGS_TIDR 0x24
+#define TEMP_RATE_IRQ_MASK GENMASK(25, 24)
+#define TMRTRCTR 0x70
+#define TMRTRCTR_EN BIT(31)
+#define TMRTRCTR_TEMP_MASK GENMASK(7, 0)
+#define TMFTRCTR 0x74
+#define TMFTRCTR_EN BIT(31)
+#define TMFTRCTR_TEMP_MASK GENMASK(7, 0)
+#define TEMP_RATE_THR_LVL 0x7
#define REGS_TTCFGR 0x080 /* Temperature Configuration Register */
#define REGS_TSCFGR 0x084 /* Sensor Configuration Register */
@@ -73,14 +88,26 @@ struct qoriq_sensor {
int id;
};
+struct tmu_drvdata {
+ u32 teumr0;
+ u32 tmu_errata;
+};
+
struct qoriq_tmu_data {
int ver;
u32 ttrcr[NUM_TTRCR_MAX];
struct regmap *regmap;
struct clk *clk;
struct qoriq_sensor sensor[SITES_MAX];
+ const struct tmu_drvdata *drvdata;
};
+static inline bool qoriq_tmu_has_errata(const struct tmu_drvdata *drvdata,
+ u32 flag)
+{
+ return drvdata->tmu_errata & flag;
+}
+
static struct qoriq_tmu_data *qoriq_sensor_to_data(struct qoriq_sensor *s)
{
return container_of(s, struct qoriq_tmu_data, sensor[s->id]);
@@ -90,7 +117,7 @@ static int tmu_get_temp(struct thermal_zone_device *tz, int *temp)
{
struct qoriq_sensor *qsensor = thermal_zone_device_priv(tz);
struct qoriq_tmu_data *qdata = qoriq_sensor_to_data(qsensor);
- u32 val;
+ u32 val, tidr;
/*
* REGS_TRITSR(id) has the following layout:
*
@@ -123,6 +150,15 @@ static int tmu_get_temp(struct thermal_zone_device *tz, int *temp)
10 * USEC_PER_MSEC))
return -ENODATA;
+ /*ERR052243: If a raising or falling edge happens, try later */
+ if (qoriq_tmu_has_errata(qdata->drvdata, TMU_ERR052243)) {
+ regmap_read(qdata->regmap, REGS_TIDR, &tidr);
+ if (tidr & TEMP_RATE_IRQ_MASK) {
+ regmap_write(qdata->regmap, REGS_TIDR, TEMP_RATE_IRQ_MASK);
+ return -EAGAIN;
+ }
+ }
+
if (qdata->ver == TMU_VER1) {
*temp = (val & GENMASK(7, 0)) * MILLIDEGREE_PER_DEGREE;
} else {
@@ -234,9 +270,18 @@ static void qoriq_tmu_init_device(struct qoriq_tmu_data *data)
regmap_write(data->regmap, REGS_TMTMIR, TMTMIR_DEFAULT);
} else {
regmap_write(data->regmap, REGS_V2_TMTMIR, TMTMIR_DEFAULT);
- regmap_write(data->regmap, REGS_V2_TEUMR(0), TEUMR0_V2);
+ regmap_write(data->regmap, REGS_V2_TEUMR(0),
+ data->drvdata->teumr0);
}
+ /* ERR052243: Set the raising & falling edge monitor */
+ if (qoriq_tmu_has_errata(data->drvdata, TMU_ERR052243)) {
+ regmap_write(data->regmap, TMRTRCTR, TMRTRCTR_EN |
+ FIELD_PREP(TMRTRCTR_TEMP_MASK, TEMP_RATE_THR_LVL));
+ regmap_write(data->regmap, TMFTRCTR, TMFTRCTR_EN |
+ FIELD_PREP(TMFTRCTR_TEMP_MASK, TEMP_RATE_THR_LVL));
+
+ }
/* Disable monitoring */
regmap_write(data->regmap, REGS_TMR, TMR_DISABLE);
}
@@ -319,6 +364,10 @@ static int qoriq_tmu_probe(struct platform_device *pdev)
data->ver = (ver >> 8) & 0xff;
+ data->drvdata = of_device_get_match_data(&pdev->dev);
+ if (!data->drvdata)
+ return dev_err_probe(dev, -EINVAL, "Failed to get match data\n");
+
qoriq_tmu_init_device(data); /* TMU initialization */
ret = qoriq_tmu_calibration(dev, data); /* TMU calibration */
@@ -376,9 +425,23 @@ static int qoriq_tmu_resume(struct device *dev)
static DEFINE_SIMPLE_DEV_PM_OPS(qoriq_tmu_pm_ops,
qoriq_tmu_suspend, qoriq_tmu_resume);
+static const struct tmu_drvdata qoriq_tmu_data = {
+ .teumr0 = TEUMR0_V2,
+};
+
+static const struct tmu_drvdata imx8mq_tmu_data = {
+ .teumr0 = TEUMR0_V2,
+};
+
+static const struct tmu_drvdata imx93_data = {
+ .teumr0 = TEUMR0_V21,
+ .tmu_errata = TMU_ERR052243,
+};
+
static const struct of_device_id qoriq_tmu_match[] = {
- { .compatible = "fsl,qoriq-tmu", },
- { .compatible = "fsl,imx8mq-tmu", },
+ { .compatible = "fsl,qoriq-tmu", .data = &qoriq_tmu_data },
+ { .compatible = "fsl,imx8mq-tmu", .data = &imx8mq_tmu_data },
+ { .compatible = "fsl,imx93-tmu", .data = &imx93_data },
{},
};
MODULE_DEVICE_TABLE(of, qoriq_tmu_match);
diff --git a/drivers/thermal/samsung/Kconfig b/drivers/thermal/samsung/Kconfig
index f4eff5a..e1e8035 100644
--- a/drivers/thermal/samsung/Kconfig
+++ b/drivers/thermal/samsung/Kconfig
@@ -3,6 +3,7 @@
tristate "Exynos thermal management unit driver"
depends on THERMAL_OF
depends on HAS_IOMEM
+ default ARCH_EXYNOS
help
If you say yes here you get support for the TMU (Thermal Management
Unit) driver for Samsung Exynos series of SoCs. This driver initialises
diff --git a/drivers/thermal/spacemit/Kconfig b/drivers/thermal/spacemit/Kconfig
new file mode 100644
index 0000000..de7b5ec
--- /dev/null
+++ b/drivers/thermal/spacemit/Kconfig
@@ -0,0 +1,19 @@
+# SPDX-License-Identifier: GPL-2.0-only
+menu "SpacemiT thermal drivers"
+depends on ARCH_SPACEMIT || COMPILE_TEST
+
+config SPACEMIT_K1_TSENSOR
+ tristate "SpacemiT K1 thermal sensor driver"
+ depends on THERMAL_OF
+ help
+ This driver provides support for the thermal sensor
+ integrated in the SpacemiT K1 SoC.
+
+ The thermal sensor monitors temperatures for five thermal zones:
+ soc, package, gpu, cluster0, and cluster1. It supports reporting
+ temperature values and handling high/low threshold interrupts.
+
+ Say Y here if you want to enable thermal monitoring on SpacemiT K1.
+ If compiled as a module, it will be called k1_tsensor.
+
+endmenu
diff --git a/drivers/thermal/spacemit/Makefile b/drivers/thermal/spacemit/Makefile
new file mode 100644
index 0000000..82b3074
--- /dev/null
+++ b/drivers/thermal/spacemit/Makefile
@@ -0,0 +1,3 @@
+# SPDX-License-Identifier: GPL-2.0-only
+
+obj-$(CONFIG_SPACEMIT_K1_TSENSOR) += k1_tsensor.o
diff --git a/drivers/thermal/spacemit/k1_tsensor.c b/drivers/thermal/spacemit/k1_tsensor.c
new file mode 100644
index 0000000..79222d2
--- /dev/null
+++ b/drivers/thermal/spacemit/k1_tsensor.c
@@ -0,0 +1,280 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Thermal sensor driver for SpacemiT K1 SoC
+ *
+ * Copyright (C) 2026 Shuwei Wu <shuwei.wu@mailbox.org>
+ */
+#include <linux/bitfield.h>
+#include <linux/clk.h>
+#include <linux/err.h>
+#include <linux/interrupt.h>
+#include <linux/io.h>
+#include <linux/kernel.h>
+#include <linux/module.h>
+#include <linux/of.h>
+#include <linux/platform_device.h>
+#include <linux/reset.h>
+#include <linux/slab.h>
+#include <linux/thermal.h>
+
+#include "../thermal_hwmon.h"
+
+#define K1_TSENSOR_PCTRL_REG 0x00
+#define K1_TSENSOR_PCTRL_ENABLE BIT(0)
+#define K1_TSENSOR_PCTRL_TEMP_MODE BIT(3)
+#define K1_TSENSOR_PCTRL_RAW_SEL BIT(7)
+
+#define K1_TSENSOR_PCTRL_CTUNE GENMASK(11, 8)
+#define K1_TSENSOR_PCTRL_SW_CTRL GENMASK(21, 18)
+#define K1_TSENSOR_PCTRL_HW_AUTO_MODE BIT(23)
+
+#define K1_TSENSOR_EN_REG 0x08
+#define K1_TSENSOR_EN_ALL GENMASK(MAX_SENSOR_NUMBER - 1, 0)
+
+#define K1_TSENSOR_TIME_REG 0x0C
+#define K1_TSENSOR_TIME_WAIT_REF_CNT GENMASK(3, 0)
+#define K1_TSENSOR_TIME_ADC_CNT_RST GENMASK(7, 4)
+#define K1_TSENSOR_TIME_FILTER_PERIOD GENMASK(21, 20)
+#define K1_TSENSOR_TIME_MASK GENMASK(23, 0)
+
+#define K1_TSENSOR_INT_CLR_REG 0x10
+#define K1_TSENSOR_INT_EN_REG 0x14
+#define K1_TSENSOR_INT_STA_REG 0x18
+
+#define K1_TSENSOR_INT_EN_MASK BIT(0)
+#define K1_TSENSOR_INT_MASK(x) (GENMASK(2, 1) << ((x) * 2))
+
+#define K1_TSENSOR_DATA_BASE_REG 0x20
+#define K1_TSENSOR_DATA_REG(x) (K1_TSENSOR_DATA_BASE_REG + ((x) / 2) * 4)
+#define K1_TSENSOR_DATA_LOW_MASK GENMASK(15, 0)
+#define K1_TSENSOR_DATA_HIGH_MASK GENMASK(31, 16)
+
+#define K1_TSENSOR_THRSH_BASE_REG 0x40
+#define K1_TSENSOR_THRSH_REG(x) (K1_TSENSOR_THRSH_BASE_REG + ((x) * 4))
+#define K1_TSENSOR_THRSH_LOW_MASK GENMASK(15, 0)
+#define K1_TSENSOR_THRSH_HIGH_MASK GENMASK(31, 16)
+
+#define MAX_SENSOR_NUMBER 5
+
+/* Hardware offset value required for temperature calculation */
+#define TEMPERATURE_OFFSET 278
+
+struct k1_tsensor_channel {
+ struct k1_tsensor *ts;
+ struct thermal_zone_device *tzd;
+ int id;
+};
+
+struct k1_tsensor {
+ void __iomem *base;
+ struct k1_tsensor_channel ch[MAX_SENSOR_NUMBER];
+};
+
+static void k1_tsensor_init(struct k1_tsensor *ts)
+{
+ u32 val;
+
+ /* Disable all the interrupts */
+ writel(0xffffffff, ts->base + K1_TSENSOR_INT_EN_REG);
+
+ /* Configure ADC sampling time and filter period */
+ val = readl(ts->base + K1_TSENSOR_TIME_REG);
+ val &= ~K1_TSENSOR_TIME_MASK;
+ val |= K1_TSENSOR_TIME_FILTER_PERIOD |
+ K1_TSENSOR_TIME_ADC_CNT_RST |
+ K1_TSENSOR_TIME_WAIT_REF_CNT;
+ writel(val, ts->base + K1_TSENSOR_TIME_REG);
+
+ /*
+ * Enable all sensors' auto mode, enable dither control,
+ * consecutive mode, and power up sensor.
+ */
+ val = readl(ts->base + K1_TSENSOR_PCTRL_REG);
+ val &= ~K1_TSENSOR_PCTRL_SW_CTRL;
+ val &= ~K1_TSENSOR_PCTRL_CTUNE;
+ val |= K1_TSENSOR_PCTRL_RAW_SEL |
+ K1_TSENSOR_PCTRL_TEMP_MODE |
+ K1_TSENSOR_PCTRL_HW_AUTO_MODE |
+ K1_TSENSOR_PCTRL_ENABLE;
+ writel(val, ts->base + K1_TSENSOR_PCTRL_REG);
+
+ /* Enable each sensor */
+ val = readl(ts->base + K1_TSENSOR_EN_REG);
+ val |= K1_TSENSOR_EN_ALL;
+ writel(val, ts->base + K1_TSENSOR_EN_REG);
+}
+
+static void k1_tsensor_enable_irq(struct k1_tsensor_channel *ch)
+{
+ struct k1_tsensor *ts = ch->ts;
+ u32 val;
+
+ val = readl(ts->base + K1_TSENSOR_INT_CLR_REG);
+ val |= K1_TSENSOR_INT_MASK(ch->id);
+ writel(val, ts->base + K1_TSENSOR_INT_CLR_REG);
+
+ val = readl(ts->base + K1_TSENSOR_INT_EN_REG);
+ val &= ~K1_TSENSOR_INT_MASK(ch->id);
+ writel(val, ts->base + K1_TSENSOR_INT_EN_REG);
+
+ /* Enable thermal interrupt */
+ val = readl(ts->base + K1_TSENSOR_INT_EN_REG);
+ val |= K1_TSENSOR_INT_EN_MASK;
+ writel(val, ts->base + K1_TSENSOR_INT_EN_REG);
+}
+
+/*
+ * The conversion formula used is:
+ * T(m°C) = (((raw_value & mask) >> shift) - TEMPERATURE_OFFSET) * 1000
+ */
+static int k1_tsensor_get_temp(struct thermal_zone_device *tz, int *temp)
+{
+ struct k1_tsensor_channel *ch = thermal_zone_device_priv(tz);
+ struct k1_tsensor *ts = ch->ts;
+ u32 val;
+
+ val = readl(ts->base + K1_TSENSOR_DATA_REG(ch->id));
+ if (ch->id % 2)
+ *temp = FIELD_GET(K1_TSENSOR_DATA_HIGH_MASK, val);
+ else
+ *temp = FIELD_GET(K1_TSENSOR_DATA_LOW_MASK, val);
+
+ *temp -= TEMPERATURE_OFFSET;
+ *temp *= 1000;
+
+ return 0;
+}
+
+/*
+ * For each sensor, the hardware threshold register is 32 bits:
+ * - Lower 16 bits [15:0] configure the low threshold temperature.
+ * - Upper 16 bits [31:16] configure the high threshold temperature.
+ */
+static int k1_tsensor_set_trips(struct thermal_zone_device *tz, int low, int high)
+{
+ struct k1_tsensor_channel *ch = thermal_zone_device_priv(tz);
+ struct k1_tsensor *ts = ch->ts;
+ u32 val;
+
+ if (low >= high)
+ return -EINVAL;
+
+ low = clamp_val(low / 1000 + TEMPERATURE_OFFSET, TEMPERATURE_OFFSET,
+ FIELD_MAX(K1_TSENSOR_THRSH_LOW_MASK));
+ high = clamp_val(high / 1000 + TEMPERATURE_OFFSET, TEMPERATURE_OFFSET,
+ FIELD_MAX(K1_TSENSOR_THRSH_HIGH_MASK));
+
+ val = readl(ts->base + K1_TSENSOR_THRSH_REG(ch->id));
+
+ val &= ~(K1_TSENSOR_THRSH_LOW_MASK | K1_TSENSOR_THRSH_HIGH_MASK);
+ val |= FIELD_PREP(K1_TSENSOR_THRSH_LOW_MASK, low);
+ val |= FIELD_PREP(K1_TSENSOR_THRSH_HIGH_MASK, high);
+
+ writel(val, ts->base + K1_TSENSOR_THRSH_REG(ch->id));
+
+ return 0;
+}
+
+static const struct thermal_zone_device_ops k1_tsensor_ops = {
+ .get_temp = k1_tsensor_get_temp,
+ .set_trips = k1_tsensor_set_trips,
+};
+
+static irqreturn_t k1_tsensor_irq_thread(int irq, void *data)
+{
+ struct k1_tsensor *ts = (struct k1_tsensor *)data;
+ int mask, status, i;
+
+ status = readl(ts->base + K1_TSENSOR_INT_STA_REG);
+
+ for (i = 0; i < MAX_SENSOR_NUMBER; i++) {
+ if (status & K1_TSENSOR_INT_MASK(i)) {
+ mask = readl(ts->base + K1_TSENSOR_INT_CLR_REG);
+ mask |= K1_TSENSOR_INT_MASK(i);
+ writel(mask, ts->base + K1_TSENSOR_INT_CLR_REG);
+ thermal_zone_device_update(ts->ch[i].tzd, THERMAL_EVENT_UNSPECIFIED);
+ }
+ }
+
+ return IRQ_HANDLED;
+}
+
+static int k1_tsensor_probe(struct platform_device *pdev)
+{
+ struct device *dev = &pdev->dev;
+ struct k1_tsensor *ts;
+ struct reset_control *reset;
+ struct clk *clk;
+ int i, irq, ret;
+
+ ts = devm_kzalloc(dev, sizeof(*ts), GFP_KERNEL);
+ if (!ts)
+ return -ENOMEM;
+
+ ts->base = devm_platform_ioremap_resource(pdev, 0);
+ if (IS_ERR(ts->base))
+ return dev_err_probe(dev, PTR_ERR(ts->base), "Failed to get reg\n");
+
+ reset = devm_reset_control_get_exclusive_deasserted(dev, NULL);
+ if (IS_ERR(reset))
+ return dev_err_probe(dev, PTR_ERR(reset), "Failed to get/deassert reset control\n");
+
+ clk = devm_clk_get_enabled(dev, "core");
+ if (IS_ERR(clk))
+ return dev_err_probe(dev, PTR_ERR(clk), "Failed to get core clock\n");
+
+ clk = devm_clk_get_enabled(dev, "bus");
+ if (IS_ERR(clk))
+ return dev_err_probe(dev, PTR_ERR(clk), "Failed to get bus clock\n");
+
+ k1_tsensor_init(ts);
+
+ irq = platform_get_irq(pdev, 0);
+ if (irq < 0)
+ return irq;
+
+ ret = devm_request_threaded_irq(dev, irq, NULL,
+ k1_tsensor_irq_thread,
+ IRQF_ONESHOT, "k1_tsensor", ts);
+ if (ret < 0)
+ return ret;
+
+ for (i = 0; i < MAX_SENSOR_NUMBER; ++i) {
+ ts->ch[i].id = i;
+ ts->ch[i].ts = ts;
+ ts->ch[i].tzd = devm_thermal_of_zone_register(dev, i, ts->ch + i, &k1_tsensor_ops);
+ if (IS_ERR(ts->ch[i].tzd))
+ return PTR_ERR(ts->ch[i].tzd);
+
+ /* Attach sysfs hwmon attributes for userspace monitoring */
+ ret = devm_thermal_add_hwmon_sysfs(dev, ts->ch[i].tzd);
+ if (ret)
+ dev_warn(dev, "Failed to add hwmon sysfs attributes\n");
+
+ k1_tsensor_enable_irq(ts->ch + i);
+ }
+
+ platform_set_drvdata(pdev, ts);
+
+ return 0;
+}
+
+static const struct of_device_id k1_tsensor_dt_ids[] = {
+ { .compatible = "spacemit,k1-tsensor" },
+ { /* sentinel */ }
+};
+
+MODULE_DEVICE_TABLE(of, k1_tsensor_dt_ids);
+
+static struct platform_driver k1_tsensor_driver = {
+ .driver = {
+ .name = "k1_tsensor",
+ .of_match_table = k1_tsensor_dt_ids,
+ },
+ .probe = k1_tsensor_probe,
+};
+module_platform_driver(k1_tsensor_driver);
+
+MODULE_DESCRIPTION("SpacemiT K1 Thermal Sensor Driver");
+MODULE_AUTHOR("Shuwei Wu <shuwei.wu@mailbox.org>");
+MODULE_LICENSE("GPL");
diff --git a/drivers/thermal/tegra/soctherm.c b/drivers/thermal/tegra/soctherm.c
index 5d26b52..d8e988a 100644
--- a/drivers/thermal/tegra/soctherm.c
+++ b/drivers/thermal/tegra/soctherm.c
@@ -1499,6 +1499,13 @@ static int soctherm_clk_enable(struct platform_device *pdev, bool enable)
return 0;
}
+static void soctherm_clk_disable(void *data)
+{
+ struct platform_device *pdev = data;
+
+ soctherm_clk_enable(pdev, false);
+}
+
static int throt_get_cdev_max_state(struct thermal_cooling_device *cdev,
unsigned long *max_state)
{
@@ -1700,9 +1707,9 @@ static void soctherm_init_hw_throt_cdev(struct platform_device *pdev)
stc->init = true;
} else {
- tcd = thermal_of_cooling_device_register(np_stcc,
- (char *)name, ts,
- &throt_cooling_ops);
+ tcd = devm_thermal_of_child_cooling_device_register(dev, np_stcc,
+ (char *)name, ts,
+ &throt_cooling_ops);
if (IS_ERR_OR_NULL(tcd)) {
dev_err(dev,
"throttle-cfg: %s: failed to register cooling device\n",
@@ -2175,6 +2182,10 @@ static int tegra_soctherm_probe(struct platform_device *pdev)
if (err)
return err;
+ err = devm_add_action_or_reset(&pdev->dev, soctherm_clk_disable, pdev);
+ if (err)
+ return err;
+
soctherm_thermtrips_parse(pdev);
soctherm_init_hw_throt_cdev(pdev);
@@ -2184,10 +2195,8 @@ static int tegra_soctherm_probe(struct platform_device *pdev)
for (i = 0; i < soc->num_ttgs; ++i) {
struct tegra_thermctl_zone *zone =
devm_kzalloc(&pdev->dev, sizeof(*zone), GFP_KERNEL);
- if (!zone) {
- err = -ENOMEM;
- goto disable_clocks;
- }
+ if (!zone)
+ return -ENOMEM;
zone->reg = tegra->regs + soc->ttgs[i]->sensor_temp_offset;
zone->dev = &pdev->dev;
@@ -2201,7 +2210,7 @@ static int tegra_soctherm_probe(struct platform_device *pdev)
err = PTR_ERR(z);
dev_err(&pdev->dev, "failed to register sensor: %d\n",
err);
- goto disable_clocks;
+ return err;
}
zone->tz = z;
@@ -2210,7 +2219,7 @@ static int tegra_soctherm_probe(struct platform_device *pdev)
/* Configure hw trip points */
err = tegra_soctherm_set_hwtrips(&pdev->dev, soc->ttgs[i], z);
if (err)
- goto disable_clocks;
+ return err;
}
err = soctherm_interrupts_init(pdev, tegra);
@@ -2218,11 +2227,6 @@ static int tegra_soctherm_probe(struct platform_device *pdev)
soctherm_debug_init(pdev);
return 0;
-
-disable_clocks:
- soctherm_clk_enable(pdev, false);
-
- return err;
}
static void tegra_soctherm_remove(struct platform_device *pdev)
@@ -2230,8 +2234,6 @@ static void tegra_soctherm_remove(struct platform_device *pdev)
struct tegra_soctherm *tegra = platform_get_drvdata(pdev);
debugfs_remove_recursive(tegra->debugfs_dir);
-
- soctherm_clk_enable(pdev, false);
}
static int __maybe_unused soctherm_suspend(struct device *dev)
diff --git a/drivers/thermal/testing/command.c b/drivers/thermal/testing/command.c
index 1159ece..fbf7ab9 100644
--- a/drivers/thermal/testing/command.c
+++ b/drivers/thermal/testing/command.c
@@ -116,18 +116,30 @@ static int tt_command_exec(int index, const char *arg)
break;
case TT_CMD_DELTZ:
+ if (!arg || !*arg)
+ return -EINVAL;
+
ret = tt_del_tz(arg);
break;
case TT_CMD_TZADDTRIP:
+ if (!arg || !*arg)
+ return -EINVAL;
+
ret = tt_zone_add_trip(arg);
break;
case TT_CMD_TZREG:
+ if (!arg || !*arg)
+ return -EINVAL;
+
ret = tt_zone_reg(arg);
break;
case TT_CMD_TZUNREG:
+ if (!arg || !*arg)
+ return -EINVAL;
+
ret = tt_zone_unreg(arg);
break;
diff --git a/drivers/thermal/testing/zone.c b/drivers/thermal/testing/zone.c
index 3c33924..f7f9ca2 100644
--- a/drivers/thermal/testing/zone.c
+++ b/drivers/thermal/testing/zone.c
@@ -239,9 +239,9 @@ int tt_del_tz(const char *arg)
int ret;
int id;
- ret = sscanf(arg, "%d", &id);
- if (ret != 1)
- return -EINVAL;
+ ret = kstrtoint(arg, 10, &id);
+ if (ret < 0)
+ return ret;
struct tt_work *tt_work __free(kfree) = kzalloc_obj(*tt_work);
if (!tt_work)
@@ -279,9 +279,9 @@ static struct tt_thermal_zone *tt_get_tt_zone(const char *arg)
struct tt_thermal_zone *tt_zone;
int ret, id;
- ret = sscanf(arg, "%d", &id);
- if (ret != 1)
- return ERR_PTR(-EINVAL);
+ ret = kstrtoint(arg, 10, &id);
+ if (ret < 0)
+ return ERR_PTR(ret);
guard(mutex)(&tt_thermal_zones_lock);
diff --git a/drivers/thermal/thermal_core.c b/drivers/thermal/thermal_core.c
index 2f4e2dc..28a20d4b 100644
--- a/drivers/thermal/thermal_core.c
+++ b/drivers/thermal/thermal_core.c
@@ -116,78 +116,24 @@ static int thermal_set_governor(struct thermal_zone_device *tz,
return ret;
}
-int thermal_register_governor(struct thermal_governor *governor)
+static int __init thermal_register_governor(struct thermal_governor *governor)
{
- int err;
- const char *name;
- struct thermal_zone_device *pos;
if (!governor)
return -EINVAL;
- guard(mutex)(&thermal_governor_lock);
+ if (__find_governor(governor->name))
+ return -EBUSY;
- err = -EBUSY;
- if (!__find_governor(governor->name)) {
- bool match_default;
+ list_add(&governor->governor_list, &thermal_governor_list);
- err = 0;
- list_add(&governor->governor_list, &thermal_governor_list);
- match_default = !strncmp(governor->name,
- DEFAULT_THERMAL_GOVERNOR,
- THERMAL_NAME_LENGTH);
+ if (strncmp(governor->name, DEFAULT_THERMAL_GOVERNOR, THERMAL_NAME_LENGTH))
+ return 0;
- if (!def_governor && match_default)
- def_governor = governor;
- }
+ if (!def_governor)
+ def_governor = governor;
- guard(mutex)(&thermal_list_lock);
-
- list_for_each_entry(pos, &thermal_tz_list, node) {
- /*
- * only thermal zones with specified tz->tzp->governor_name
- * may run with tz->govenor unset
- */
- if (pos->governor)
- continue;
-
- name = pos->tzp->governor_name;
-
- if (!strncasecmp(name, governor->name, THERMAL_NAME_LENGTH)) {
- int ret;
-
- ret = thermal_set_governor(pos, governor);
- if (ret)
- dev_err(&pos->device,
- "Failed to set governor %s for thermal zone %s: %d\n",
- governor->name, pos->type, ret);
- }
- }
-
- return err;
-}
-
-void thermal_unregister_governor(struct thermal_governor *governor)
-{
- struct thermal_zone_device *pos;
-
- if (!governor)
- return;
-
- guard(mutex)(&thermal_governor_lock);
-
- if (!__find_governor(governor->name))
- return;
-
- list_del(&governor->governor_list);
-
- guard(mutex)(&thermal_list_lock);
-
- list_for_each_entry(pos, &thermal_tz_list, node) {
- if (!strncasecmp(pos->governor->name, governor->name,
- THERMAL_NAME_LENGTH))
- thermal_set_governor(pos, NULL);
- }
+ return 0;
}
int thermal_zone_device_set_policy(struct thermal_zone_device *tz,
@@ -225,40 +171,34 @@ int thermal_build_list_of_policies(char *buf)
static void __init thermal_unregister_governors(void)
{
- struct thermal_governor **governor;
+ struct thermal_governor *gov, *pos;
- for_each_governor_table(governor)
- thermal_unregister_governor(*governor);
+ guard(mutex)(&thermal_governor_lock);
+
+ list_for_each_entry_safe(gov, pos, &thermal_governor_list, governor_list)
+ list_del(&gov->governor_list);
}
static int __init thermal_register_governors(void)
{
- int ret = 0;
struct thermal_governor **governor;
+ guard(mutex)(&thermal_governor_lock);
+
for_each_governor_table(governor) {
+ int ret;
+
ret = thermal_register_governor(*governor);
if (ret) {
pr_err("Failed to register governor: '%s'",
(*governor)->name);
- break;
+ return ret;
}
- pr_info("Registered thermal governor '%s'",
- (*governor)->name);
+ pr_info("Registered thermal governor '%s'", (*governor)->name);
}
- if (ret) {
- struct thermal_governor **gov;
-
- for_each_governor_table(gov) {
- if (gov == governor)
- break;
- thermal_unregister_governor(*gov);
- }
- }
-
- return ret;
+ return 0;
}
static int __thermal_zone_device_set_mode(struct thermal_zone_device *tz,
@@ -949,34 +889,7 @@ static void thermal_unbind_cdev_from_trip(struct thermal_zone_device *tz,
kfree(pos);
}
-static void thermal_release(struct device *dev)
-{
- struct thermal_zone_device *tz;
- struct thermal_cooling_device *cdev;
-
- if (!strncmp(dev_name(dev), "thermal_zone",
- sizeof("thermal_zone") - 1)) {
- tz = to_thermal_zone(dev);
- thermal_zone_destroy_device_groups(tz);
- thermal_set_governor(tz, NULL);
- ida_destroy(&tz->ida);
- mutex_destroy(&tz->lock);
- complete(&tz->removal);
- } else if (!strncmp(dev_name(dev), "cooling_device",
- sizeof("cooling_device") - 1)) {
- cdev = to_cooling_device(dev);
- thermal_cooling_device_destroy_sysfs(cdev);
- kfree_const(cdev->type);
- ida_free(&thermal_cdev_ida, cdev->id);
- kfree(cdev);
- }
-}
-
-static const struct class thermal_class = {
- .name = "thermal",
- .dev_release = thermal_release,
-};
-static bool thermal_class_unavailable __ro_after_init = true;
+static struct class *thermal_class __ro_after_init;
static inline
void print_bind_err_msg(struct thermal_zone_device *tz,
@@ -1040,42 +953,35 @@ static void thermal_cooling_device_init_complete(struct thermal_cooling_device *
thermal_zone_cdev_bind(tz, cdev);
}
-/**
- * __thermal_cooling_device_register() - register a new thermal cooling device
- * @np: a pointer to a device tree node.
- * @type: the thermal cooling device type.
- * @devdata: device private data.
- * @ops: standard thermal cooling devices callbacks.
- *
- * This interface function adds a new thermal cooling device (fan/processor/...)
- * to /sys/class/thermal/ folder as cooling_device[0-*]. It tries to bind itself
- * to all the thermal zone devices registered at the same time.
- * It also gives the opportunity to link the cooling device to a device tree
- * node, so that it can be bound to a thermal zone created out of device tree.
- *
- * Return: a pointer to the created struct thermal_cooling_device or an
- * ERR_PTR. Caller must check return value with IS_ERR*() helpers.
- */
-static struct thermal_cooling_device *
-__thermal_cooling_device_register(struct device_node *np,
- const char *type, void *devdata,
- const struct thermal_cooling_device_ops *ops)
+static void thermal_cdev_release(struct device *dev)
+{
+ struct thermal_cooling_device *cdev = to_cooling_device(dev);
+
+ thermal_cooling_device_destroy_sysfs(cdev);
+ kfree_const(cdev->type);
+ ida_free(&thermal_cdev_ida, cdev->id);
+ kfree(cdev);
+}
+
+struct thermal_cooling_device *
+thermal_cooling_device_alloc(const char *type, const struct thermal_cooling_device_ops *ops)
{
struct thermal_cooling_device *cdev;
- unsigned long current_state;
int ret;
if (!ops || !ops->get_max_state || !ops->get_cur_state ||
!ops->set_cur_state)
return ERR_PTR(-EINVAL);
- if (thermal_class_unavailable)
+ if (!thermal_class)
return ERR_PTR(-ENODEV);
cdev = kzalloc_obj(*cdev);
if (!cdev)
return ERR_PTR(-ENOMEM);
+ cdev->ops = ops;
+
ret = ida_alloc(&thermal_cdev_ida, GFP_KERNEL);
if (ret < 0)
goto out_kfree_cdev;
@@ -1087,17 +993,35 @@ __thermal_cooling_device_register(struct device_node *np,
goto out_ida_remove;
}
+ return cdev;
+
+out_ida_remove:
+ ida_free(&thermal_cdev_ida, cdev->id);
+out_kfree_cdev:
+ kfree(cdev);
+ return ERR_PTR(ret);
+}
+
+int thermal_cooling_device_add(struct thermal_cooling_device *cdev, void *devdata)
+{
+ unsigned long current_state;
+ int ret;
+
mutex_init(&cdev->lock);
INIT_LIST_HEAD(&cdev->thermal_instances);
- cdev->np = np;
- cdev->ops = ops;
cdev->updated = false;
- cdev->device.class = &thermal_class;
+ cdev->device.class = thermal_class;
+ cdev->device.release = thermal_cdev_release;
+ device_initialize(&cdev->device);
cdev->devdata = devdata;
+ ret = dev_set_name(&cdev->device, "cooling_device%d", cdev->id);
+ if (ret)
+ goto out_put_device;
+
ret = cdev->ops->get_max_state(cdev, &cdev->max_state);
if (ret)
- goto out_cdev_type;
+ goto out_put_device;
/*
* The cooling device's current state is only needed for debug
@@ -1113,40 +1037,32 @@ __thermal_cooling_device_register(struct device_node *np,
thermal_cooling_device_setup_sysfs(cdev);
- ret = dev_set_name(&cdev->device, "cooling_device%d", cdev->id);
+ ret = device_add(&cdev->device);
if (ret)
- goto out_cooling_dev;
-
- ret = device_register(&cdev->device);
- if (ret) {
- /* thermal_release() handles rest of the cleanup */
- put_device(&cdev->device);
- return ERR_PTR(ret);
- }
+ goto out_put_device;
if (current_state <= cdev->max_state)
thermal_debug_cdev_add(cdev, current_state);
thermal_cooling_device_init_complete(cdev);
- return cdev;
+ return 0;
-out_cooling_dev:
- thermal_cooling_device_destroy_sysfs(cdev);
-out_cdev_type:
- kfree_const(cdev->type);
-out_ida_remove:
- ida_free(&thermal_cdev_ida, cdev->id);
-out_kfree_cdev:
- kfree(cdev);
- return ERR_PTR(ret);
+out_put_device:
+ /*
+ * The device core will release the memory via
+ * thermal_release() after put_device() is called in the error
+ * path
+ */
+ put_device(&cdev->device);
+ return ret;
}
/**
* thermal_cooling_device_register() - register a new thermal cooling device
* @type: the thermal cooling device type.
* @devdata: device private data.
- * @ops: standard thermal cooling devices callbacks.
+ * @ops: standard thermal cooling devices callbacks.
*
* This interface function adds a new thermal cooling device (fan/processor/...)
* to /sys/class/thermal/ folder as cooling_device[0-*]. It tries to bind itself
@@ -1159,82 +1075,62 @@ struct thermal_cooling_device *
thermal_cooling_device_register(const char *type, void *devdata,
const struct thermal_cooling_device_ops *ops)
{
- return __thermal_cooling_device_register(NULL, type, devdata, ops);
+ struct thermal_cooling_device *cdev;
+ int ret;
+
+ cdev = thermal_cooling_device_alloc(type, ops);
+ if (IS_ERR(cdev))
+ return cdev;
+
+ ret = thermal_cooling_device_add(cdev, devdata);
+ if (ret)
+ return ERR_PTR(ret);
+
+ return cdev;
}
EXPORT_SYMBOL_GPL(thermal_cooling_device_register);
-/**
- * thermal_of_cooling_device_register() - register an OF thermal cooling device
- * @np: a pointer to a device tree node.
- * @type: the thermal cooling device type.
- * @devdata: device private data.
- * @ops: standard thermal cooling devices callbacks.
- *
- * This function will register a cooling device with device tree node reference.
- * This interface function adds a new thermal cooling device (fan/processor/...)
- * to /sys/class/thermal/ folder as cooling_device[0-*]. It tries to bind itself
- * to all the thermal zone devices registered at the same time.
- *
- * Return: a pointer to the created struct thermal_cooling_device or an
- * ERR_PTR. Caller must check return value with IS_ERR*() helpers.
- */
-struct thermal_cooling_device *
-thermal_of_cooling_device_register(struct device_node *np,
- const char *type, void *devdata,
- const struct thermal_cooling_device_ops *ops)
+static void thermal_cooling_device_release(void *data)
{
- return __thermal_cooling_device_register(np, type, devdata, ops);
-}
-EXPORT_SYMBOL_GPL(thermal_of_cooling_device_register);
+ struct thermal_cooling_device *cdev = data;
-static void thermal_cooling_device_release(struct device *dev, void *res)
-{
- thermal_cooling_device_unregister(
- *(struct thermal_cooling_device **)res);
+ thermal_cooling_device_unregister(cdev);
}
/**
- * devm_thermal_of_cooling_device_register() - register an OF thermal cooling
- * device
+ * devm_thermal_cooling_device_register() - register a thermal cooling device
+ *
* @dev: a valid struct device pointer of a sensor device.
- * @np: a pointer to a device tree node.
* @type: the thermal cooling device type.
* @devdata: device private data.
* @ops: standard thermal cooling devices callbacks.
*
- * This function will register a cooling device with device tree node reference.
- * This interface function adds a new thermal cooling device (fan/processor/...)
- * to /sys/class/thermal/ folder as cooling_device[0-*]. It tries to bind itself
- * to all the thermal zone devices registered at the same time.
+ * This function will register a cooling device. This interface
+ * function adds a new thermal cooling device (fan/processor/...) to
+ * /sys/class/thermal/ folder as cooling_device[0-*]. It tries to bind
+ * itself to all the thermal zone devices registered at the same time.
*
* Return: a pointer to the created struct thermal_cooling_device or an
* ERR_PTR. Caller must check return value with IS_ERR*() helpers.
*/
struct thermal_cooling_device *
-devm_thermal_of_cooling_device_register(struct device *dev,
- struct device_node *np,
- const char *type, void *devdata,
- const struct thermal_cooling_device_ops *ops)
+devm_thermal_cooling_device_register(struct device *dev, const char *type, void *devdata,
+ const struct thermal_cooling_device_ops *ops)
{
- struct thermal_cooling_device **ptr, *tcd;
+ struct thermal_cooling_device *cdev;
+ int ret;
- ptr = devres_alloc(thermal_cooling_device_release, sizeof(*ptr),
- GFP_KERNEL);
- if (!ptr)
- return ERR_PTR(-ENOMEM);
+ cdev = thermal_cooling_device_register(type, devdata, ops);
+ if (IS_ERR(cdev))
+ return cdev;
- tcd = __thermal_cooling_device_register(np, type, devdata, ops);
- if (IS_ERR(tcd)) {
- devres_free(ptr);
- return tcd;
- }
+ ret = devm_add_action_or_reset(dev, thermal_cooling_device_release, cdev);
+ if (ret)
+ return ERR_PTR(ret);
- *ptr = tcd;
- devres_add(dev, ptr);
-
- return tcd;
+ return cdev;
}
-EXPORT_SYMBOL_GPL(devm_thermal_of_cooling_device_register);
+EXPORT_SYMBOL_GPL(devm_thermal_cooling_device_register);
static bool thermal_cooling_device_present(struct thermal_cooling_device *cdev)
{
@@ -1470,6 +1366,17 @@ static void thermal_zone_init_complete(struct thermal_zone_device *tz)
__thermal_zone_device_update(tz, THERMAL_EVENT_UNSPECIFIED);
}
+static void thermal_zone_device_release(struct device *dev)
+{
+ struct thermal_zone_device *tz = to_thermal_zone(dev);
+
+ thermal_zone_destroy_device_groups(tz);
+ thermal_set_governor(tz, NULL);
+ ida_destroy(&tz->ida);
+ mutex_destroy(&tz->lock);
+ complete(&tz->removal);
+}
+
/**
* thermal_zone_device_register_with_trips() - register a new thermal zone device
* @type: the thermal zone device type
@@ -1540,7 +1447,7 @@ thermal_zone_device_register_with_trips(const char *type,
if (polling_delay && passive_delay > polling_delay)
return ERR_PTR(-EINVAL);
- if (thermal_class_unavailable)
+ if (!thermal_class)
return ERR_PTR(-ENODEV);
tz = kzalloc_flex(*tz, trips, num_trips);
@@ -1576,7 +1483,8 @@ thermal_zone_device_register_with_trips(const char *type,
if (!tz->ops.critical)
tz->ops.critical = thermal_zone_device_critical;
- tz->device.class = &thermal_class;
+ tz->device.class = thermal_class;
+ tz->device.release = thermal_zone_device_release;
tz->devdata = devdata;
tz->num_trips = num_trips;
for_each_trip_desc(tz, td) {
@@ -1837,7 +1745,7 @@ static void __thermal_pm_prepare(void)
void thermal_pm_prepare(void)
{
- if (thermal_class_unavailable)
+ if (!thermal_class)
return;
__thermal_pm_prepare();
@@ -1868,7 +1776,7 @@ void thermal_pm_complete(void)
{
struct thermal_zone_device *tz;
- if (thermal_class_unavailable)
+ if (!thermal_class)
return;
guard(mutex)(&thermal_list_lock);
@@ -1881,6 +1789,7 @@ void thermal_pm_complete(void)
static int __init thermal_init(void)
{
+ struct class *tc;
int result;
thermal_debug_init();
@@ -1889,7 +1798,7 @@ static int __init thermal_init(void)
if (result)
goto error;
- thermal_wq = alloc_workqueue("thermal_events", WQ_POWER_EFFICIENT, 0);
+ thermal_wq = alloc_workqueue("thermal_events", WQ_UNBOUND, 0);
if (!thermal_wq) {
result = -ENOMEM;
goto unregister_netlink;
@@ -1897,19 +1806,19 @@ static int __init thermal_init(void)
result = thermal_register_governors();
if (result)
- goto destroy_workqueue;
-
- result = class_register(&thermal_class);
- if (result)
goto unregister_governors;
- thermal_class_unavailable = false;
+ tc = class_create("thermal");
+ if (IS_ERR(tc)) {
+ result = PTR_ERR(tc);
+ goto unregister_governors;
+ }
+ thermal_class = tc;
return 0;
unregister_governors:
thermal_unregister_governors();
-destroy_workqueue:
destroy_workqueue(thermal_wq);
unregister_netlink:
thermal_netlink_exit();
diff --git a/drivers/thermal/thermal_core.h b/drivers/thermal/thermal_core.h
index d3acff6..e98b0aa 100644
--- a/drivers/thermal/thermal_core.h
+++ b/drivers/thermal/thermal_core.h
@@ -258,8 +258,6 @@ struct thermal_instance {
#define to_cooling_device(_dev) \
container_of(_dev, struct thermal_cooling_device, device)
-int thermal_register_governor(struct thermal_governor *);
-void thermal_unregister_governor(struct thermal_governor *);
int thermal_zone_device_set_policy(struct thermal_zone_device *, char *);
int thermal_build_list_of_policies(char *buf);
void __thermal_zone_device_update(struct thermal_zone_device *tz,
@@ -269,6 +267,11 @@ void thermal_zone_device_critical_shutdown(struct thermal_zone_device *tz);
void thermal_governor_update_tz(struct thermal_zone_device *tz,
enum thermal_notify_event reason);
+struct thermal_cooling_device *
+thermal_cooling_device_alloc(const char *type, const struct thermal_cooling_device_ops *ops);
+
+int thermal_cooling_device_add(struct thermal_cooling_device *cdev, void *devdata);
+
/* Helpers */
#define for_each_trip_desc(__tz, __td) \
for (__td = __tz->trips; __td - __tz->trips < __tz->num_trips; __td++)
diff --git a/drivers/thermal/thermal_hwmon.c b/drivers/thermal/thermal_hwmon.c
index b624892..386dfb9 100644
--- a/drivers/thermal/thermal_hwmon.c
+++ b/drivers/thermal/thermal_hwmon.c
@@ -19,27 +19,19 @@
#include "thermal_hwmon.h"
#include "thermal_core.h"
+/*
+ * Needs to be large enough to hold a thermal zone type string followed by an
+ * underline character and a 32-bit integer in decimal representation.
+ */
+#define THERMAL_HWMON_NAME_LENGTH (THERMAL_NAME_LENGTH + 11)
+
/* hwmon sys I/F */
/* thermal zone devices with the same type share one hwmon device */
struct thermal_hwmon_device {
- char type[THERMAL_NAME_LENGTH];
+ char name[THERMAL_HWMON_NAME_LENGTH];
struct device *device;
- int count;
- struct list_head tz_list;
struct list_head node;
-};
-
-struct thermal_hwmon_attr {
- struct device_attribute attr;
- char name[16];
-};
-
-/* one temperature input for each thermal zone */
-struct thermal_hwmon_temp {
- struct list_head hwmon_node;
struct thermal_zone_device *tz;
- struct thermal_hwmon_attr temp_input; /* hwmon sys attr */
- struct thermal_hwmon_attr temp_crit; /* hwmon sys attr */
};
static LIST_HEAD(thermal_hwmon_list);
@@ -47,19 +39,14 @@ static LIST_HEAD(thermal_hwmon_list);
static DEFINE_MUTEX(thermal_hwmon_list_lock);
static ssize_t
-temp_input_show(struct device *dev, struct device_attribute *attr, char *buf)
+temp1_input_show(struct device *dev, struct device_attribute *attr, char *buf)
{
+ struct thermal_hwmon_device *hwmon = dev_get_drvdata(dev);
+ struct thermal_zone_device *tz = hwmon->tz;
int temperature;
int ret;
- struct thermal_hwmon_attr *hwmon_attr
- = container_of(attr, struct thermal_hwmon_attr, attr);
- struct thermal_hwmon_temp *temp
- = container_of(hwmon_attr, struct thermal_hwmon_temp,
- temp_input);
- struct thermal_zone_device *tz = temp->tz;
ret = thermal_zone_get_temp(tz, &temperature);
-
if (ret)
return ret;
@@ -67,14 +54,10 @@ temp_input_show(struct device *dev, struct device_attribute *attr, char *buf)
}
static ssize_t
-temp_crit_show(struct device *dev, struct device_attribute *attr, char *buf)
+temp1_crit_show(struct device *dev, struct device_attribute *attr, char *buf)
{
- struct thermal_hwmon_attr *hwmon_attr
- = container_of(attr, struct thermal_hwmon_attr, attr);
- struct thermal_hwmon_temp *temp
- = container_of(hwmon_attr, struct thermal_hwmon_temp,
- temp_crit);
- struct thermal_zone_device *tz = temp->tz;
+ struct thermal_hwmon_device *hwmon = dev_get_drvdata(dev);
+ struct thermal_zone_device *tz = hwmon->tz;
int temperature;
int ret;
@@ -87,167 +70,98 @@ temp_crit_show(struct device *dev, struct device_attribute *attr, char *buf)
return sysfs_emit(buf, "%d\n", temperature);
}
+static DEVICE_ATTR_RO(temp1_input);
+static DEVICE_ATTR_RO(temp1_crit);
-static struct thermal_hwmon_device *
-thermal_hwmon_lookup_by_type(const struct thermal_zone_device *tz)
+static struct attribute *thermal_hwmon_attrs[] = {
+ &dev_attr_temp1_input.attr,
+ &dev_attr_temp1_crit.attr,
+ NULL,
+};
+
+static umode_t thermal_hwmon_attr_is_visible(struct kobject *kobj,
+ struct attribute *a, int n)
{
- struct thermal_hwmon_device *hwmon;
- char type[THERMAL_NAME_LENGTH];
+ if (a == &dev_attr_temp1_input.attr)
+ return a->mode;
- mutex_lock(&thermal_hwmon_list_lock);
- list_for_each_entry(hwmon, &thermal_hwmon_list, node) {
- strscpy(type, tz->type);
- strreplace(type, '-', '_');
- if (!strcmp(hwmon->type, type)) {
- mutex_unlock(&thermal_hwmon_list_lock);
- return hwmon;
- }
+ if (a == &dev_attr_temp1_crit.attr) {
+ struct thermal_hwmon_device *hwmon = dev_get_drvdata(kobj_to_dev(kobj));
+ struct thermal_zone_device *tz = hwmon->tz;
+ int dummy;
+
+ if (tz->ops.get_crit_temp && !tz->ops.get_crit_temp(tz, &dummy))
+ return a->mode;
}
- mutex_unlock(&thermal_hwmon_list_lock);
- return NULL;
+ return 0;
}
-/* Find the temperature input matching a given thermal zone */
-static struct thermal_hwmon_temp *
-thermal_hwmon_lookup_temp(const struct thermal_hwmon_device *hwmon,
- const struct thermal_zone_device *tz)
-{
- struct thermal_hwmon_temp *temp;
+static const struct attribute_group thermal_hwmon_group = {
+ .attrs = thermal_hwmon_attrs,
+ .is_visible = thermal_hwmon_attr_is_visible,
+};
- mutex_lock(&thermal_hwmon_list_lock);
- list_for_each_entry(temp, &hwmon->tz_list, hwmon_node)
- if (temp->tz == tz) {
- mutex_unlock(&thermal_hwmon_list_lock);
- return temp;
- }
- mutex_unlock(&thermal_hwmon_list_lock);
-
- return NULL;
-}
-
-static bool thermal_zone_crit_temp_valid(struct thermal_zone_device *tz)
-{
- int temp;
- return tz->ops.get_crit_temp && !tz->ops.get_crit_temp(tz, &temp);
-}
+__ATTRIBUTE_GROUPS(thermal_hwmon);
int thermal_add_hwmon_sysfs(struct thermal_zone_device *tz)
{
struct thermal_hwmon_device *hwmon;
- struct thermal_hwmon_temp *temp;
- int new_hwmon_device = 1;
- int result;
-
- hwmon = thermal_hwmon_lookup_by_type(tz);
- if (hwmon) {
- new_hwmon_device = 0;
- goto register_sys_interface;
- }
hwmon = kzalloc_obj(*hwmon);
if (!hwmon)
return -ENOMEM;
- INIT_LIST_HEAD(&hwmon->tz_list);
- strscpy(hwmon->type, tz->type, THERMAL_NAME_LENGTH);
- strreplace(hwmon->type, '-', '_');
+ hwmon->tz = tz;
+ /*
+ * Append the thermal zone ID preceded by an underline character to the
+ * type to disambiguate the sensors command output.
+ */
+ scnprintf(hwmon->name, THERMAL_HWMON_NAME_LENGTH, "%s_%d", tz->type, tz->id);
+ strreplace(hwmon->name, '-', '_');
hwmon->device = hwmon_device_register_for_thermal(&tz->device,
- hwmon->type, hwmon);
+ hwmon->name, hwmon,
+ thermal_hwmon_groups);
if (IS_ERR(hwmon->device)) {
- result = PTR_ERR(hwmon->device);
- goto free_mem;
+ int result = PTR_ERR(hwmon->device);
+
+ kfree(hwmon);
+ return result;
}
- register_sys_interface:
- temp = kzalloc_obj(*temp);
- if (!temp) {
- result = -ENOMEM;
- goto unregister_name;
- }
-
- temp->tz = tz;
- hwmon->count++;
-
- snprintf(temp->temp_input.name, sizeof(temp->temp_input.name),
- "temp%d_input", hwmon->count);
- temp->temp_input.attr.attr.name = temp->temp_input.name;
- temp->temp_input.attr.attr.mode = 0444;
- temp->temp_input.attr.show = temp_input_show;
- sysfs_attr_init(&temp->temp_input.attr.attr);
- result = device_create_file(hwmon->device, &temp->temp_input.attr);
- if (result)
- goto free_temp_mem;
-
- if (thermal_zone_crit_temp_valid(tz)) {
- snprintf(temp->temp_crit.name,
- sizeof(temp->temp_crit.name),
- "temp%d_crit", hwmon->count);
- temp->temp_crit.attr.attr.name = temp->temp_crit.name;
- temp->temp_crit.attr.attr.mode = 0444;
- temp->temp_crit.attr.show = temp_crit_show;
- sysfs_attr_init(&temp->temp_crit.attr.attr);
- result = device_create_file(hwmon->device,
- &temp->temp_crit.attr);
- if (result)
- goto unregister_input;
- }
-
+ /* The list is needed for hwmon lookup during removal. */
mutex_lock(&thermal_hwmon_list_lock);
- if (new_hwmon_device)
- list_add_tail(&hwmon->node, &thermal_hwmon_list);
- list_add_tail(&temp->hwmon_node, &hwmon->tz_list);
+ list_add_tail(&hwmon->node, &thermal_hwmon_list);
mutex_unlock(&thermal_hwmon_list_lock);
return 0;
-
- unregister_input:
- device_remove_file(hwmon->device, &temp->temp_input.attr);
- free_temp_mem:
- kfree(temp);
- unregister_name:
- if (new_hwmon_device)
- hwmon_device_unregister(hwmon->device);
- free_mem:
- kfree(hwmon);
-
- return result;
}
EXPORT_SYMBOL_GPL(thermal_add_hwmon_sysfs);
+static struct thermal_hwmon_device *
+thermal_hwmon_lookup(const struct thermal_zone_device *tz)
+{
+ struct thermal_hwmon_device *hwmon;
+
+ list_for_each_entry(hwmon, &thermal_hwmon_list, node) {
+ if (hwmon->tz == tz)
+ return hwmon;
+ }
+ return NULL;
+}
+
void thermal_remove_hwmon_sysfs(struct thermal_zone_device *tz)
{
struct thermal_hwmon_device *hwmon;
- struct thermal_hwmon_temp *temp;
- hwmon = thermal_hwmon_lookup_by_type(tz);
- if (unlikely(!hwmon)) {
- /* Should never happen... */
- dev_dbg(&tz->device, "hwmon device lookup failed!\n");
- return;
+ scoped_guard(mutex, &thermal_hwmon_list_lock) {
+ hwmon = thermal_hwmon_lookup(tz);
+ if (!hwmon)
+ return;
+
+ list_del(&hwmon->node);
}
- temp = thermal_hwmon_lookup_temp(hwmon, tz);
- if (unlikely(!temp)) {
- /* Should never happen... */
- dev_dbg(&tz->device, "temperature input lookup failed!\n");
- return;
- }
-
- device_remove_file(hwmon->device, &temp->temp_input.attr);
- if (thermal_zone_crit_temp_valid(tz))
- device_remove_file(hwmon->device, &temp->temp_crit.attr);
-
- mutex_lock(&thermal_hwmon_list_lock);
- list_del(&temp->hwmon_node);
- kfree(temp);
- if (!list_empty(&hwmon->tz_list)) {
- mutex_unlock(&thermal_hwmon_list_lock);
- return;
- }
- list_del(&hwmon->node);
- mutex_unlock(&thermal_hwmon_list_lock);
-
hwmon_device_unregister(hwmon->device);
kfree(hwmon);
}
diff --git a/drivers/thermal/thermal_of.c b/drivers/thermal/thermal_of.c
index 99085c8..100fd8a 100644
--- a/drivers/thermal/thermal_of.c
+++ b/drivers/thermal/thermal_of.c
@@ -98,7 +98,7 @@ static struct thermal_trip *thermal_of_trips_init(struct device_node *np, int *n
int ret, count;
*ntrips = 0;
-
+
struct device_node *trips __free(device_node) = of_get_child_by_name(np, "trips");
if (!trips)
return NULL;
@@ -259,16 +259,34 @@ static bool thermal_of_get_cooling_spec(struct device_node *map_np, int index,
of_node_put(cooling_spec.np);
- if (cooling_spec.args_count < 2) {
- pr_err("wrong reference to cooling device, missing limits\n");
+ /*
+ * There are two formats:
+ * - Legacy format : <&cdev lower upper>
+ * - New format : <&cdev cdev_id lower upper>
+ *
+ * With the new format, along with the device node pointer,
+ * the cdev_id must match with the cooling device cdev_id in
+ * order to bind
+ */
+ if (cooling_spec.args_count < 2 || cooling_spec.args_count > 3) {
+ pr_err("Invalid number of cooling device parameters\n");
return false;
}
if (cooling_spec.np != cdev->np)
return false;
- c->lower = cooling_spec.args[0];
- c->upper = cooling_spec.args[1];
+ if (cooling_spec.args_count == 3 &&
+ cooling_spec.args[0] != cdev->cdev_id)
+ return false;
+
+ if (cooling_spec.args_count != 3) {
+ c->lower = cooling_spec.args[0];
+ c->upper = cooling_spec.args[1];
+ } else {
+ c->lower = cooling_spec.args[1];
+ c->upper = cooling_spec.args[2];
+ }
c->weight = weight;
return true;
@@ -494,7 +512,7 @@ EXPORT_SYMBOL_GPL(devm_thermal_of_zone_register);
/**
* devm_thermal_of_zone_unregister - Resource managed version of
* thermal_of_zone_unregister().
- * @dev: Device for which which resource was allocated.
+ * @dev: Device for which resource was allocated.
* @tz: a pointer to struct thermal_zone where the sensor is registered.
*
* This function removes the sensor callbacks and private data from the
@@ -510,3 +528,125 @@ void devm_thermal_of_zone_unregister(struct device *dev, struct thermal_zone_dev
devm_thermal_of_zone_match, tz));
}
EXPORT_SYMBOL_GPL(devm_thermal_of_zone_unregister);
+
+/**
+ * thermal_of_cooling_device_register() - register an OF thermal cooling device
+ * @np: a pointer to a device tree node.
+ * @cdev_id: a cooling device id in the cooling controller
+ * @type: the thermal cooling device type.
+ * @devdata: device private data.
+ * @ops: standard thermal cooling devices callbacks.
+ *
+ * This interface function adds a new thermal cooling device (fan/processor/...)
+ * to /sys/class/thermal/ folder as cooling_device[0-*]. It tries to bind itself
+ * to all the thermal zone devices registered at the same time.
+ * It also gives the opportunity to link the cooling device to a device tree
+ * node, so that it can be bound to a thermal zone created out of device tree.
+ *
+ * Return: a pointer to the created struct thermal_cooling_device or an
+ * ERR_PTR. Caller must check return value with IS_ERR*() helpers.
+ */
+struct thermal_cooling_device *
+thermal_of_cooling_device_register(struct device_node *np, u32 cdev_id,
+ const char *type, void *devdata,
+ const struct thermal_cooling_device_ops *ops)
+{
+ struct thermal_cooling_device *cdev;
+ int ret;
+
+ cdev = thermal_cooling_device_alloc(type, ops);
+ if (IS_ERR(cdev))
+ return cdev;
+
+ cdev->np = np;
+ cdev->cdev_id = cdev_id;
+
+ ret = thermal_cooling_device_add(cdev, devdata);
+ if (ret)
+ return ERR_PTR(ret);
+
+ return cdev;
+}
+EXPORT_SYMBOL_GPL(thermal_of_cooling_device_register);
+
+static void thermal_of_cooling_device_release(void *data)
+{
+ struct thermal_cooling_device *cdev = data;
+
+ thermal_cooling_device_unregister(cdev);
+}
+
+static struct thermal_cooling_device *
+__devm_thermal_of_cooling_device_register(struct device *dev, struct device_node *np,
+ u32 cdev_id, const char *type, void *devdata,
+ const struct thermal_cooling_device_ops *ops)
+{
+ struct thermal_cooling_device *cdev;
+ int ret;
+
+ cdev = thermal_of_cooling_device_register(np, cdev_id, type, devdata, ops);
+ if (IS_ERR(cdev))
+ return cdev;
+
+ ret = devm_add_action_or_reset(dev, thermal_of_cooling_device_release, cdev);
+ if (ret)
+ return ERR_PTR(ret);
+
+ return cdev;
+}
+
+/**
+ * devm_thermal_of_cooling_device_register() - register an OF thermal cooling device
+ * @dev: a valid struct device pointer of a sensor device.
+ * @cdev_id: a cooling device index in the cooling controller
+ * @type: the thermal cooling device type.
+ * @devdata: device private data.
+ * @ops: standard thermal cooling devices callbacks.
+ *
+ * This function will register a cooling device with device tree node reference.
+ * This interface function adds a new thermal cooling device (fan/processor/...)
+ * to /sys/class/thermal/ folder as cooling_device[0-*]. It tries to bind itself
+ * to all the thermal zone devices registered at the same time.
+ *
+ * Return: a pointer to the created struct thermal_cooling_device or an
+ * ERR_PTR. Caller must check return value with IS_ERR*() helpers.
+ */
+struct thermal_cooling_device *
+devm_thermal_of_cooling_device_register(struct device *dev, u32 cdev_id,
+ const char *type, void *devdata,
+ const struct thermal_cooling_device_ops *ops)
+{
+ return __devm_thermal_of_cooling_device_register(dev, dev->of_node, cdev_id,
+ type, devdata, ops);
+}
+EXPORT_SYMBOL_GPL(devm_thermal_of_cooling_device_register);
+
+/**
+ * devm_thermal_of_child_cooling_device_register() - register an OF thermal cooling
+ * device
+ * @dev: a valid struct device pointer of a sensor device.
+ * @np: a pointer to a device tree node.
+ * @type: the thermal cooling device type.
+ * @devdata: device private data.
+ * @ops: standard thermal cooling devices callbacks.
+ *
+ * This function will register a cooling device with device tree node reference.
+ * This interface function adds a new thermal cooling device (fan/processor/...)
+ * to /sys/class/thermal/ folder as cooling_device[0-*]. It tries to bind itself
+ * to all the thermal zone devices registered at the same time.
+ *
+ * This function should be used when a cooling controller has child
+ * nodes which are referenced in the thermal zone cooling map.
+ *
+ * Return: a pointer to the created struct thermal_cooling_device or an
+ * ERR_PTR. Caller must check return value with IS_ERR*() helpers.
+ */
+struct thermal_cooling_device *
+devm_thermal_of_child_cooling_device_register(struct device *dev,
+ struct device_node *np,
+ const char *type, void *devdata,
+ const struct thermal_cooling_device_ops *ops)
+{
+ return __devm_thermal_of_cooling_device_register(dev, np, 0, type, devdata, ops);
+}
+EXPORT_SYMBOL_GPL(devm_thermal_of_child_cooling_device_register);
diff --git a/drivers/thermal/thermal_sysfs.c b/drivers/thermal/thermal_sysfs.c
index 5eecae1..b44abfc9 100644
--- a/drivers/thermal/thermal_sysfs.c
+++ b/drivers/thermal/thermal_sysfs.c
@@ -82,7 +82,7 @@ mode_store(struct device *dev, struct device_attribute *attr,
}
#define thermal_trip_of_attr(_ptr_, _attr_) \
- ({ \
+ ({ \
struct thermal_trip_desc *td; \
\
td = container_of(_ptr_, struct thermal_trip_desc, \
@@ -536,11 +536,9 @@ cur_state_store(struct device *dev, struct device_attribute *attr,
unsigned long state;
int result;
- if (sscanf(buf, "%ld\n", &state) != 1)
- return -EINVAL;
-
- if ((long)state < 0)
- return -EINVAL;
+ result = kstrtoul(buf, 10, &state);
+ if (result < 0)
+ return result;
/* Requested state should be less than max_state + 1 */
if (state > cdev->max_state)
diff --git a/drivers/thunderbolt/property.c b/drivers/thunderbolt/property.c
index da2c59a..59beab4 100644
--- a/drivers/thunderbolt/property.c
+++ b/drivers/thunderbolt/property.c
@@ -60,6 +60,8 @@ static bool tb_property_entry_valid(const struct tb_property_entry *entry,
case TB_PROPERTY_TYPE_DIRECTORY:
case TB_PROPERTY_TYPE_DATA:
case TB_PROPERTY_TYPE_TEXT:
+ if (!entry->length)
+ return false;
if (entry->length > block_len)
return false;
if (check_add_overflow(entry->value, entry->length, &end) ||
@@ -185,6 +187,10 @@ static struct tb_property_dir *__tb_property_parse_dir(const u32 *block,
if (is_root) {
content_offset = dir_offset + 2;
content_len = dir_len;
+ if (content_offset + content_len > block_len) {
+ tb_property_free_dir(dir);
+ return NULL;
+ }
} else {
if (dir_len < 4) {
tb_property_free_dir(dir);
diff --git a/drivers/thunderbolt/xdomain.c b/drivers/thunderbolt/xdomain.c
index 754808c..1fd1cf4 100644
--- a/drivers/thunderbolt/xdomain.c
+++ b/drivers/thunderbolt/xdomain.c
@@ -55,6 +55,7 @@ static const char * const state_names[] = {
struct xdomain_request_work {
struct work_struct work;
struct tb_xdp_header *pkg;
+ size_t pkg_len;
struct tb *tb;
};
@@ -122,7 +123,9 @@ static bool tb_xdomain_match(const struct tb_cfg_request *req,
static bool tb_xdomain_copy(struct tb_cfg_request *req,
const struct ctl_pkg *pkg)
{
- memcpy(req->response, pkg->buffer, req->response_size);
+ size_t len = min_t(size_t, pkg->frame.size, req->response_size);
+
+ memcpy(req->response, pkg->buffer, len);
req->result.err = 0;
return true;
}
@@ -393,6 +396,8 @@ static int tb_xdp_properties_request(struct tb_ctl *ctl, u64 route,
}
}
+ if (req.offset + len > data_len)
+ len = data_len - req.offset;
memcpy(data + req.offset, res->data, len * 4);
req.offset += len;
} while (!data_len || req.offset < data_len);
@@ -731,6 +736,7 @@ static void tb_xdp_handle_request(struct work_struct *work)
struct xdomain_request_work *xw = container_of(work, typeof(*xw), work);
const struct tb_xdp_header *pkg = xw->pkg;
const struct tb_xdomain_header *xhdr = &pkg->xd_hdr;
+ size_t pkg_len = xw->pkg_len;
struct tb *tb = xw->tb;
struct tb_ctl *ctl = tb->ctl;
struct tb_xdomain *xd;
@@ -762,7 +768,7 @@ static void tb_xdp_handle_request(struct work_struct *work)
switch (pkg->type) {
case PROPERTIES_REQUEST:
tb_dbg(tb, "%llx: received XDomain properties request\n", route);
- if (xd) {
+ if (xd && pkg_len >= sizeof(struct tb_xdp_properties)) {
ret = tb_xdp_properties_response(tb, ctl, xd, sequence,
(const struct tb_xdp_properties *)pkg);
}
@@ -816,7 +822,8 @@ static void tb_xdp_handle_request(struct work_struct *work)
tb_dbg(tb, "%llx: received XDomain link state change request\n",
route);
- if (xd && xd->state == XDOMAIN_STATE_BONDING_UUID_HIGH) {
+ if (xd && xd->state == XDOMAIN_STATE_BONDING_UUID_HIGH &&
+ pkg_len >= sizeof(struct tb_xdp_link_state_change)) {
const struct tb_xdp_link_state_change *lsc =
(const struct tb_xdp_link_state_change *)pkg;
@@ -868,6 +875,7 @@ tb_xdp_schedule_request(struct tb *tb, const struct tb_xdp_header *hdr,
kfree(xw);
return false;
}
+ xw->pkg_len = size;
xw->tb = tb_domain_get(tb);
schedule_work(&xw->work);
diff --git a/drivers/usb/serial/io_ti.c b/drivers/usb/serial/io_ti.c
index cb55370..d48819d 100644
--- a/drivers/usb/serial/io_ti.c
+++ b/drivers/usb/serial/io_ti.c
@@ -773,6 +773,12 @@ static int get_manuf_info(struct edgeport_serial *serial, u8 *buffer)
}
/* Read the descriptor data */
+ if (le16_to_cpu(rom_desc->Size) != sizeof(struct edge_ti_manuf_descriptor)) {
+ dev_err(dev, "unexpected Edge descriptor length: %u\n",
+ le16_to_cpu(rom_desc->Size));
+ status = -EINVAL;
+ goto exit;
+ }
status = read_rom(serial, start_address+sizeof(struct ti_i2c_desc),
le16_to_cpu(rom_desc->Size), buffer);
if (status)
@@ -838,6 +844,11 @@ static int build_i2c_fw_hdr(u8 *header, const struct firmware *fw)
/* Pointer to fw_down memory image */
img_header = (struct ti_i2c_image_header *)&fw->data[4];
+ if (le16_to_cpu(img_header->Length) >
+ buffer_size - sizeof(struct ti_i2c_firmware_rec)) {
+ kfree(buffer);
+ return -EINVAL;
+ }
memcpy(buffer + sizeof(struct ti_i2c_firmware_rec),
&fw->data[4 + sizeof(struct ti_i2c_image_header)],
le16_to_cpu(img_header->Length));
diff --git a/drivers/usb/serial/kl5kusb105.c b/drivers/usb/serial/kl5kusb105.c
index ed8531a..e72a0b4 100644
--- a/drivers/usb/serial/kl5kusb105.c
+++ b/drivers/usb/serial/kl5kusb105.c
@@ -330,8 +330,8 @@ static int klsi_105_prepare_write_buffer(struct usb_serial_port *port,
unsigned char *buf = dest;
int count;
- count = kfifo_out_locked(&port->write_fifo, buf + KLSI_HDR_LEN, size,
- &port->lock);
+ count = kfifo_out_locked(&port->write_fifo, buf + KLSI_HDR_LEN,
+ size - KLSI_HDR_LEN, &port->lock);
put_unaligned_le16(count, buf);
return count + KLSI_HDR_LEN;
diff --git a/drivers/usb/serial/option.c b/drivers/usb/serial/option.c
index 48ae018..a34e79c 100644
--- a/drivers/usb/serial/option.c
+++ b/drivers/usb/serial/option.c
@@ -202,6 +202,7 @@ static void option_instat_callback(struct urb *urb);
#define DELL_PRODUCT_5821E_ESIM 0x81e0
#define DELL_PRODUCT_5829E_ESIM 0x81e4
#define DELL_PRODUCT_5829E 0x81e6
+#define DELL_PRODUCT_5826E_ESIM 0x81ea
#define DELL_PRODUCT_FM101R_ESIM 0x8213
#define DELL_PRODUCT_FM101R 0x8215
@@ -1123,6 +1124,8 @@ static const struct usb_device_id option_ids[] = {
.driver_info = RSVD(0) | RSVD(6) },
{ USB_DEVICE(DELL_VENDOR_ID, DELL_PRODUCT_5829E_ESIM),
.driver_info = RSVD(0) | RSVD(6) },
+ { USB_DEVICE_INTERFACE_CLASS(DELL_VENDOR_ID, DELL_PRODUCT_5826E_ESIM, 0xff),
+ .driver_info = RSVD(1) | RSVD(4) },
{ USB_DEVICE_INTERFACE_CLASS(DELL_VENDOR_ID, DELL_PRODUCT_FM101R, 0xff) },
{ USB_DEVICE_INTERFACE_CLASS(DELL_VENDOR_ID, DELL_PRODUCT_FM101R_ESIM, 0xff) },
{ USB_DEVICE(ANYDATA_VENDOR_ID, ANYDATA_PRODUCT_ADU_E100A) }, /* ADU-E100, ADU-310 */
diff --git a/drivers/xen/mcelog.c b/drivers/xen/mcelog.c
index 53a8720..32ab419 100644
--- a/drivers/xen/mcelog.c
+++ b/drivers/xen/mcelog.c
@@ -54,8 +54,8 @@
#include <asm/xen/hypervisor.h>
static struct mc_info g_mi;
-static struct mcinfo_logical_cpu *g_physinfo;
-static uint32_t ncpus;
+static struct mcinfo_logical_cpu *g_physinfo __ro_after_init;
+static uint32_t ncpus __ro_after_init;
static DEFINE_MUTEX(mcelog_lock);
@@ -182,7 +182,7 @@ static const struct file_operations xen_mce_chrdev_ops = {
.unlocked_ioctl = xen_mce_chrdev_ioctl,
};
-static struct miscdevice xen_mce_chrdev_device = {
+static struct miscdevice xen_mce_chrdev_device __ro_after_init = {
MISC_MCELOG_MINOR,
"mcelog",
&xen_mce_chrdev_ops,
diff --git a/drivers/xen/platform-pci.c b/drivers/xen/platform-pci.c
index 1db82da..f243823 100644
--- a/drivers/xen/platform-pci.c
+++ b/drivers/xen/platform-pci.c
@@ -174,11 +174,9 @@ static int platform_pci_probe(struct pci_dev *pdev,
}
static const struct pci_device_id platform_pci_tbl[] = {
- {PCI_VENDOR_ID_XEN, PCI_DEVICE_ID_XEN_PLATFORM,
- PCI_ANY_ID, PCI_ANY_ID, 0, 0, 0},
- {PCI_VENDOR_ID_XEN, PCI_DEVICE_ID_XEN_PLATFORM_XS61,
- PCI_ANY_ID, PCI_ANY_ID, 0, 0, 0},
- {0,}
+ { PCI_VDEVICE(XEN, PCI_DEVICE_ID_XEN_PLATFORM) },
+ { PCI_VDEVICE(XEN, PCI_DEVICE_ID_XEN_PLATFORM_XS61) },
+ { }
};
static const struct dev_pm_ops platform_pm_ops = {
diff --git a/drivers/xen/xen-balloon.c b/drivers/xen/xen-balloon.c
index b293d76..67b0e2d 100644
--- a/drivers/xen/xen-balloon.c
+++ b/drivers/xen/xen-balloon.c
@@ -138,7 +138,7 @@ EXPORT_SYMBOL_GPL(xen_balloon_init);
struct device_attribute *attr, \
char *buf) \
{ \
- return sprintf(buf, format, ##args); \
+ return sysfs_emit(buf, format, ##args); \
} \
static DEVICE_ATTR_RO(name)
@@ -155,7 +155,7 @@ static DEVICE_BOOL_ATTR(scrub_pages, 0644, xen_scrub_pages);
static ssize_t target_kb_show(struct device *dev, struct device_attribute *attr,
char *buf)
{
- return sprintf(buf, "%lu\n", PAGES2KB(balloon_stats.target_pages));
+ return sysfs_emit(buf, "%lu\n", PAGES2KB(balloon_stats.target_pages));
}
static ssize_t target_kb_store(struct device *dev,
@@ -180,7 +180,7 @@ static DEVICE_ATTR_RW(target_kb);
static ssize_t target_show(struct device *dev, struct device_attribute *attr,
char *buf)
{
- return sprintf(buf, "%llu\n",
+ return sysfs_emit(buf, "%llu\n",
(unsigned long long)balloon_stats.target_pages
<< PAGE_SHIFT);
}
diff --git a/drivers/xen/xenbus/xenbus_probe.c b/drivers/xen/xenbus/xenbus_probe.c
index eb260ece..fafb2b8 100644
--- a/drivers/xen/xenbus/xenbus_probe.c
+++ b/drivers/xen/xenbus/xenbus_probe.c
@@ -514,7 +514,7 @@ int xenbus_probe_node(struct xen_bus_type *bus,
char devname[XEN_BUS_ID_SIZE];
int err;
struct xenbus_device *xendev;
- size_t stringlen;
+ size_t name_len, type_len;
char *tmpstring;
enum xenbus_state state = xenbus_read_driver_state(NULL, nodename);
@@ -525,8 +525,9 @@ int xenbus_probe_node(struct xen_bus_type *bus,
return 0;
}
- stringlen = strlen(nodename) + 1 + strlen(type) + 1;
- xendev = kzalloc(sizeof(*xendev) + stringlen, GFP_KERNEL);
+ name_len = strlen(nodename);
+ type_len = strlen(type);
+ xendev = kzalloc(sizeof(*xendev) + name_len + 1 + type_len + 1, GFP_KERNEL);
if (!xendev)
return -ENOMEM;
@@ -535,11 +536,11 @@ int xenbus_probe_node(struct xen_bus_type *bus,
/* Copy the strings into the extra space. */
tmpstring = (char *)(xendev + 1);
- strcpy(tmpstring, nodename);
+ memcpy(tmpstring, nodename, name_len);
xendev->nodename = tmpstring;
- tmpstring += strlen(tmpstring) + 1;
- strcpy(tmpstring, type);
+ tmpstring += name_len + 1;
+ memcpy(tmpstring, type, type_len);
xendev->devicetype = tmpstring;
init_completion(&xendev->down);
diff --git a/fs/affs/affs.h b/fs/affs/affs.h
index a0caf6a..44a3f69 100644
--- a/fs/affs/affs.h
+++ b/fs/affs/affs.h
@@ -227,11 +227,6 @@ static inline bool affs_validblock(struct super_block *sb, int block)
block < AFFS_SB(sb)->s_partition_size);
}
-static inline void
-affs_set_blocksize(struct super_block *sb, int size)
-{
- sb_set_blocksize(sb, size);
-}
static inline struct buffer_head *
affs_bread(struct super_block *sb, int block)
{
diff --git a/fs/affs/super.c b/fs/affs/super.c
index 079f36e..b232251 100644
--- a/fs/affs/super.c
+++ b/fs/affs/super.c
@@ -358,7 +358,8 @@ static int affs_fill_super(struct super_block *sb, struct fs_context *fc)
size = bdev_nr_sectors(sb->s_bdev);
pr_debug("initial blocksize=%d, #blocks=%d\n", 512, size);
- affs_set_blocksize(sb, PAGE_SIZE);
+ if (!sb_set_blocksize(sb, PAGE_SIZE))
+ return -EINVAL;
/* Try to find root block. Its location depends on the block size. */
i = bdev_logical_block_size(sb->s_bdev);
@@ -374,7 +375,8 @@ static int affs_fill_super(struct super_block *sb, struct fs_context *fc)
if (ctx->root_block < 0)
sbi->s_root_block = (ctx->reserved + size - 1) / 2;
pr_debug("setting blocksize to %d\n", blocksize);
- affs_set_blocksize(sb, blocksize);
+ if (!sb_set_blocksize(sb, blocksize))
+ return -EINVAL;
sbi->s_partition_size = size;
/* The root block location that was calculated above is not
diff --git a/fs/afs/dir_silly.c b/fs/afs/dir_silly.c
index a748fd1..982bb6e 100644
--- a/fs/afs/dir_silly.c
+++ b/fs/afs/dir_silly.c
@@ -248,13 +248,11 @@ int afs_silly_iput(struct dentry *dentry, struct inode *inode)
struct dentry *alias;
int ret;
- DECLARE_WAIT_QUEUE_HEAD_ONSTACK(wq);
-
_enter("%p{%pd},%llx", dentry, dentry, vnode->fid.vnode);
down_read(&dvnode->rmdir_lock);
- alias = d_alloc_parallel(dentry->d_parent, &dentry->d_name, &wq);
+ alias = d_alloc_parallel(dentry->d_parent, &dentry->d_name);
if (IS_ERR(alias)) {
up_read(&dvnode->rmdir_lock);
return 0;
diff --git a/fs/aio.c b/fs/aio.c
index 7224765..f57fa21a 100644
--- a/fs/aio.c
+++ b/fs/aio.c
@@ -318,7 +318,6 @@ static int aio_init_fs_context(struct fs_context *fc)
pfc = init_pseudo(fc, AIO_RING_MAGIC);
if (!pfc)
return -ENOMEM;
- fc->s_iflags |= SB_I_NOEXEC;
pfc->ops = &aio_super_operations;
return 0;
}
diff --git a/fs/anon_inodes.c b/fs/anon_inodes.c
index b8381c7..a7b9b94 100644
--- a/fs/anon_inodes.c
+++ b/fs/anon_inodes.c
@@ -86,8 +86,6 @@ static int anon_inodefs_init_fs_context(struct fs_context *fc)
struct pseudo_fs_context *ctx = init_pseudo(fc, ANON_INODE_FS_MAGIC);
if (!ctx)
return -ENOMEM;
- fc->s_iflags |= SB_I_NOEXEC;
- fc->s_iflags |= SB_I_NODEV;
ctx->dops = &anon_inodefs_dentry_operations;
return 0;
}
diff --git a/fs/attr.c b/fs/attr.c
index ded221d..4f437fa 100644
--- a/fs/attr.c
+++ b/fs/attr.c
@@ -547,7 +547,7 @@ int notify_change(struct mnt_idmap *idmap, struct dentry *dentry,
* breaking the delegation in this case.
*/
if (!(ia_valid & ATTR_DELEG)) {
- error = try_break_deleg(inode, delegated_inode);
+ error = try_break_deleg(inode, 0, delegated_inode);
if (error)
return error;
}
diff --git a/fs/backing-file.c b/fs/backing-file.c
index 1f3bbfc..080c996 100644
--- a/fs/backing-file.c
+++ b/fs/backing-file.c
@@ -18,17 +18,18 @@
/**
* backing_file_open - open a backing file for kernel internal use
- * @user_path: path that the user reuqested to open
+ * @user_file: file the user requested to open
* @flags: open flags
* @real_path: path of the backing file
* @cred: credentials for open
*
* Open a backing file for a stackable filesystem (e.g., overlayfs).
- * @user_path may be on the stackable filesystem and @real_path on the
- * underlying filesystem. In this case, we want to be able to return the
- * @user_path of the stackable filesystem. This is done by embedding the
- * returned file into a container structure that also stores the stacked
- * file's path, which can be retrieved using backing_file_user_path().
+ * @user_file->f_path may be on the stackable filesystem and @real_path
+ * on the underlying filesystem. In this case, we want to be able to
+ * return the path of the stackable filesystem. This is done by
+ * embedding the returned file into a container structure that also
+ * stores the stacked file's path, which can be retrieved using
+ * backing_file_user_path().
*/
struct file *backing_file_open(const struct file *user_file, int flags,
const struct path *real_path,
diff --git a/fs/befs/linuxvfs.c b/fs/befs/linuxvfs.c
index c12caae..ee0cbae 100644
--- a/fs/befs/linuxvfs.c
+++ b/fs/befs/linuxvfs.c
@@ -860,7 +860,8 @@ befs_fill_super(struct super_block *sb, struct fs_context *fc)
*/
sb->s_magic = BEFS_SUPER_MAGIC;
/* Set real blocksize of fs */
- sb_set_blocksize(sb, (ulong) befs_sb->block_size);
+ if (!sb_set_blocksize(sb, (ulong) befs_sb->block_size))
+ goto unacquire_priv_sbp;
sb->s_op = &befs_sops;
sb->s_export_op = &befs_export_operations;
sb->s_time_min = 0;
diff --git a/fs/bfs/inode.c b/fs/bfs/inode.c
index 19e49c8..e41efdd 100644
--- a/fs/bfs/inode.c
+++ b/fs/bfs/inode.c
@@ -311,7 +311,7 @@ void bfs_dump_imap(const char *prefix, struct super_block *s)
{
#ifdef DEBUG
int i;
- char *tmpbuf = (char *)get_zeroed_page(GFP_KERNEL);
+ char *tmpbuf = kzalloc(PAGE_SIZE, GFP_KERNEL);
if (!tmpbuf)
return;
@@ -323,7 +323,7 @@ void bfs_dump_imap(const char *prefix, struct super_block *s)
strcat(tmpbuf, "0");
}
printf("%s: lasti=%08lx <%s>\n", prefix, BFS_SB(s)->si_lasti, tmpbuf);
- free_page((unsigned long)tmpbuf);
+ kfree(tmpbuf);
#endif
}
@@ -346,7 +346,8 @@ static int bfs_fill_super(struct super_block *s, struct fs_context *fc)
s->s_time_min = 0;
s->s_time_max = U32_MAX;
- sb_set_blocksize(s, BFS_BSIZE);
+ if (!sb_set_blocksize(s, BFS_BSIZE))
+ goto out;
sbh = sb_bread(s, 0);
if (!sbh)
diff --git a/fs/binfmt_misc.c b/fs/binfmt_misc.c
index b3d8fd7..84349fc 100644
--- a/fs/binfmt_misc.c
+++ b/fs/binfmt_misc.c
@@ -704,7 +704,7 @@ bm_entry_read(struct file *file, char __user *buf, size_t nbytes, loff_t *ppos)
ssize_t res;
char *page;
- page = (char *) __get_free_page(GFP_KERNEL);
+ page = kmalloc(PAGE_SIZE, GFP_KERNEL);
if (!page)
return -ENOMEM;
@@ -712,7 +712,7 @@ bm_entry_read(struct file *file, char __user *buf, size_t nbytes, loff_t *ppos)
res = simple_read_from_buffer(buf, nbytes, ppos, page, strlen(page));
- free_page((unsigned long) page);
+ kfree(page);
return res;
}
diff --git a/fs/bpf_fs_kfuncs.c b/fs/bpf_fs_kfuncs.c
index e4e51a1..11841c3 100644
--- a/fs/bpf_fs_kfuncs.c
+++ b/fs/bpf_fs_kfuncs.c
@@ -100,7 +100,7 @@ static bool match_security_bpf_prefix(const char *name__str)
static int bpf_xattr_read_permission(const char *name, struct inode *inode)
{
- if (WARN_ON(!inode))
+ if (!inode)
return -EINVAL;
/* Allow reading xattr with user. and security.bpf. prefix */
@@ -170,7 +170,7 @@ __bpf_kfunc_end_defs();
static int bpf_xattr_write_permission(const char *name, struct inode *inode)
{
- if (WARN_ON(!inode))
+ if (!inode)
return -EINVAL;
/* Only allow setting and removing security.bpf. xattrs */
@@ -289,6 +289,9 @@ __bpf_kfunc int bpf_set_dentry_xattr(struct dentry *dentry, const char *name__st
struct inode *inode = d_inode(dentry);
int ret;
+ if (!inode)
+ return -EINVAL;
+
inode_lock(inode);
ret = bpf_set_dentry_xattr_locked(dentry, name__str, value_p, flags);
inode_unlock(inode);
@@ -314,6 +317,9 @@ __bpf_kfunc int bpf_remove_dentry_xattr(struct dentry *dentry, const char *name_
struct inode *inode = d_inode(dentry);
int ret;
+ if (!inode)
+ return -EINVAL;
+
inode_lock(inode);
ret = bpf_remove_dentry_xattr_locked(dentry, name__str);
inode_unlock(inode);
@@ -353,6 +359,21 @@ __bpf_kfunc int bpf_cgroup_read_xattr(struct cgroup *cgroup, const char *name__s
}
#endif /* CONFIG_CGROUPS */
+/**
+ * bpf_real_inode - get the real inode backing a dentry
+ * @dentry: dentry to resolve
+ *
+ * If the dentry is on a union/overlay filesystem, return the underlying, real
+ * inode that hosts the data. Otherwise return the inode attached to the
+ * dentry itself.
+ *
+ * Return: The real inode backing the dentry, or NULL for a negative dentry.
+ */
+__bpf_kfunc struct inode *bpf_real_inode(struct dentry *dentry)
+{
+ return d_real_inode(dentry);
+}
+
__bpf_kfunc_end_defs();
BTF_KFUNCS_START(bpf_fs_kfunc_set_ids)
@@ -363,6 +384,7 @@ BTF_ID_FLAGS(func, bpf_get_dentry_xattr, KF_SLEEPABLE)
BTF_ID_FLAGS(func, bpf_get_file_xattr, KF_SLEEPABLE)
BTF_ID_FLAGS(func, bpf_set_dentry_xattr, KF_SLEEPABLE)
BTF_ID_FLAGS(func, bpf_remove_dentry_xattr, KF_SLEEPABLE)
+BTF_ID_FLAGS(func, bpf_real_inode, KF_SLEEPABLE | KF_RET_NULL)
BTF_KFUNCS_END(bpf_fs_kfunc_set_ids)
static int bpf_fs_kfuncs_filter(const struct bpf_prog *prog, u32 kfunc_id)
diff --git a/fs/btrfs/inode.c b/fs/btrfs/inode.c
index 1ca1cbd..509042a 100644
--- a/fs/btrfs/inode.c
+++ b/fs/btrfs/inode.c
@@ -4745,7 +4745,7 @@ static void btrfs_prune_dentries(struct btrfs_root *root)
inode = btrfs_find_first_inode(root, min_ino);
while (inode) {
- if (icount_read(&inode->vfs_inode) > 1)
+ if (icount_read_once(&inode->vfs_inode) > 1)
d_prune_aliases(&inode->vfs_inode);
min_ino = btrfs_ino(inode) + 1;
diff --git a/fs/btrfs/super.c b/fs/btrfs/super.c
index b26aa91..6361548 100644
--- a/fs/btrfs/super.c
+++ b/fs/btrfs/super.c
@@ -2052,7 +2052,7 @@ static int btrfs_get_tree_subvol(struct fs_context *fc)
* then open_ctree will properly initialize the file system specific
* settings later. btrfs_init_fs_info initializes the static elements
* of the fs_info (locks and such) to make cleanup easier if we find a
- * superblock with our given fs_devices later on at sget() time.
+ * superblock with our given fs_devices later on at sget_fc() time.
*/
fs_info = kvzalloc_obj(struct btrfs_fs_info);
if (!fs_info)
diff --git a/fs/buffer.c b/fs/buffer.c
index b0b3792..9af5f06 100644
--- a/fs/buffer.c
+++ b/fs/buffer.c
@@ -54,9 +54,6 @@
#include "internal.h"
-static void submit_bh_wbc(blk_opf_t opf, struct buffer_head *bh,
- enum rw_hint hint, struct writeback_control *wbc);
-
#define BH_ENTRY(list) list_entry((list), struct buffer_head, b_assoc_buffers)
inline void touch_buffer(struct buffer_head *bh)
@@ -74,9 +71,7 @@ EXPORT_SYMBOL(__lock_buffer);
void unlock_buffer(struct buffer_head *bh)
{
- clear_bit_unlock(BH_Lock, &bh->b_state);
- smp_mb__after_atomic();
- wake_up_bit(&bh->b_state, BH_Lock);
+ clear_and_wake_up_bit(BH_Lock, &bh->b_state);
}
EXPORT_SYMBOL(unlock_buffer);
@@ -132,15 +127,43 @@ static void buffer_io_error(struct buffer_head *bh, char *msg)
bh->b_bdev, (unsigned long long)bh->b_blocknr, msg);
}
-/*
- * End-of-IO handler helper function which does not touch the bh after
- * unlocking it.
- * Note: unlock_buffer() sort-of does touch the bh after unlocking it, but
- * a race there is benign: unlock_buffer() only use the bh's address for
- * hashing after unlocking the buffer, so it doesn't actually touch the bh
- * itself.
+/**
+ * bio_endio_bh - Discard the bio used to submit a buffer.
+ * @bio: The bio.
+ * @bhp: Where to return the buffer_head.
+ *
+ * Call this in your bio_end_io handler to retrieve the buffer_head
+ * submitted in bh_submit(). If you did not call bh_submit(), do not
+ * call this function; it will return garbage.
+ *
+ * This function consumes the bio refcount which will probably free the
+ * bio.
+ *
+ * Return: True if the I/O succeeded.
*/
-static void __end_buffer_read_notouch(struct buffer_head *bh, int uptodate)
+bool bio_endio_bh(struct bio *bio, struct buffer_head **bhp)
+{
+ bool success = bio->bi_status == BLK_STS_OK;
+ struct buffer_head *bh = bio->bi_private;
+
+ if (unlikely(bio_flagged(bio, BIO_QUIET)))
+ set_bit(BH_Quiet, &bh->b_state);
+ bio_put(bio);
+
+ *bhp = bh;
+ return success;
+}
+EXPORT_SYMBOL(bio_endio_bh);
+
+/**
+ * end_buffer_read_sync - Handle buffer reads finishing
+ * @bh: The buffer.
+ * @uptodate: True if the read was successful.
+ *
+ * If a buffer is read through a mechanism that isn't bh_submit(), you
+ * can call this function to finish the read.
+ */
+void end_buffer_read_sync(struct buffer_head *bh, int uptodate)
{
if (uptodate) {
set_buffer_uptodate(bh);
@@ -150,21 +173,36 @@ static void __end_buffer_read_notouch(struct buffer_head *bh, int uptodate)
}
unlock_buffer(bh);
}
-
-/*
- * Default synchronous end-of-IO handler.. Just mark it up-to-date and
- * unlock the buffer.
- */
-void end_buffer_read_sync(struct buffer_head *bh, int uptodate)
-{
- put_bh(bh);
- __end_buffer_read_notouch(bh, uptodate);
-}
EXPORT_SYMBOL(end_buffer_read_sync);
-void end_buffer_write_sync(struct buffer_head *bh, int uptodate)
+/**
+ * bh_end_read - I/O end handler for reads
+ * @bio: The bio being completed.
+ *
+ * Pass this function to bh_submit() if you're reading into the buffer,
+ * unless you need your own special I/O end handler.
+ */
+void bh_end_read(struct bio *bio)
{
- if (uptodate) {
+ struct buffer_head *bh;
+ bool uptodate = bio_endio_bh(bio, &bh);
+ end_buffer_read_sync(bh, uptodate);
+}
+EXPORT_SYMBOL(bh_end_read);
+
+/**
+ * bh_end_write - I/O end handler for writes
+ * @bio: The bio being completed.
+ *
+ * Pass this function to bh_submit() if you're writing from the buffer,
+ * unless you need your own special I/O end handler.
+ */
+void bh_end_write(struct bio *bio)
+{
+ struct buffer_head *bh;
+ bool success = bio_endio_bh(bio, &bh);
+
+ if (success) {
set_buffer_uptodate(bh);
} else {
buffer_io_error(bh, ", lost sync page write");
@@ -172,9 +210,8 @@ void end_buffer_write_sync(struct buffer_head *bh, int uptodate)
clear_buffer_uptodate(bh);
}
unlock_buffer(bh);
- put_bh(bh);
}
-EXPORT_SYMBOL(end_buffer_write_sync);
+EXPORT_SYMBOL(bh_end_write);
static struct buffer_head *
__find_get_block_slow(struct block_device *bdev, sector_t block, bool atomic)
@@ -342,11 +379,13 @@ static void decrypt_bh(struct work_struct *work)
}
/*
- * I/O completion handler for block_read_full_folio() - pages
+ * I/O completion handler for block_read_full_folio() - folios
* which come unlocked at the end of I/O.
*/
-static void end_buffer_async_read_io(struct buffer_head *bh, int uptodate)
+static void bh_end_async_read(struct bio *bio)
{
+ struct buffer_head *bh;
+ bool uptodate = bio_endio_bh(bio, &bh);
struct inode *inode = bh->b_folio->mapping->host;
bool decrypt = fscrypt_inode_uses_fs_layer_crypto(inode);
struct fsverity_info *vi = NULL;
@@ -371,17 +410,24 @@ static void end_buffer_async_read_io(struct buffer_head *bh, int uptodate)
}
return;
}
- uptodate = 0;
+ uptodate = false;
}
end_buffer_async_read(bh, uptodate);
}
-/*
- * Completion handler for block_write_full_folio() - folios which are unlocked
- * during I/O, and which have the writeback flag cleared upon I/O completion.
+/**
+ * bh_end_async_write - I/O end handler for async folio writes
+ * @bio: The bio being completed.
+ *
+ * Pass this function to bh_submit() if you're doing the equivalent of
+ * block_write_full_folio(). That is, the folio is unlocked, and will
+ * have its writeback flag cleared once all async write buffers have
+ * completed.
*/
-static void end_buffer_async_write(struct buffer_head *bh, int uptodate)
+void bh_end_async_write(struct bio *bio)
{
+ struct buffer_head *bh;
+ bool success = bio_endio_bh(bio, &bh);
unsigned long flags;
struct buffer_head *first;
struct buffer_head *tmp;
@@ -390,7 +436,7 @@ static void end_buffer_async_write(struct buffer_head *bh, int uptodate)
BUG_ON(!buffer_async_write(bh));
folio = bh->b_folio;
- if (uptodate) {
+ if (success) {
set_buffer_uptodate(bh);
} else {
buffer_io_error(bh, ", lost async page write");
@@ -418,46 +464,7 @@ static void end_buffer_async_write(struct buffer_head *bh, int uptodate)
still_busy:
spin_unlock_irqrestore(&first->b_uptodate_lock, flags);
}
-
-/*
- * If a page's buffers are under async readin (end_buffer_async_read
- * completion) then there is a possibility that another thread of
- * control could lock one of the buffers after it has completed
- * but while some of the other buffers have not completed. This
- * locked buffer would confuse end_buffer_async_read() into not unlocking
- * the page. So the absence of BH_Async_Read tells end_buffer_async_read()
- * that this buffer is not under async I/O.
- *
- * The page comes unlocked when it has no locked buffer_async buffers
- * left.
- *
- * PageLocked prevents anyone starting new async I/O reads any of
- * the buffers.
- *
- * PageWriteback is used to prevent simultaneous writeout of the same
- * page.
- *
- * PageLocked prevents anyone from starting writeback of a page which is
- * under read I/O (PageWriteback is only ever set against a locked page).
- */
-static void mark_buffer_async_read(struct buffer_head *bh)
-{
- bh->b_end_io = end_buffer_async_read_io;
- set_buffer_async_read(bh);
-}
-
-static void mark_buffer_async_write_endio(struct buffer_head *bh,
- bh_end_io_t *handler)
-{
- bh->b_end_io = handler;
- set_buffer_async_write(bh);
-}
-
-void mark_buffer_async_write(struct buffer_head *bh)
-{
- mark_buffer_async_write_endio(bh, end_buffer_async_write);
-}
-EXPORT_SYMBOL(mark_buffer_async_write);
+EXPORT_SYMBOL(bh_end_async_write);
/*
@@ -916,7 +923,6 @@ static sector_t folio_init_buffers(struct folio *folio,
do {
if (!buffer_mapped(bh)) {
- bh->b_end_io = NULL;
bh->b_private = NULL;
bh->b_bdev = bdev;
bh->b_blocknr = block;
@@ -1157,6 +1163,83 @@ void __bforget(struct buffer_head *bh)
}
EXPORT_SYMBOL(__bforget);
+static void buffer_set_crypto_ctx(struct bio *bio, const struct buffer_head *bh,
+ gfp_t gfp_mask)
+{
+ const struct address_space *mapping = folio_mapping(bh->b_folio);
+
+ /*
+ * The ext4 journal (jbd2) can submit a buffer_head it directly created
+ * for a non-pagecache page. fscrypt doesn't care about these.
+ */
+ if (!mapping)
+ return;
+ fscrypt_set_bio_crypt_ctx(bio, mapping->host,
+ folio_pos(bh->b_folio) + bh_offset(bh), gfp_mask);
+}
+
+static void __bh_submit(struct buffer_head *bh, blk_opf_t opf,
+ enum rw_hint write_hint, struct writeback_control *wbc,
+ bio_end_io_t end_bio)
+{
+ const enum req_op op = opf & REQ_OP_MASK;
+ struct bio *bio;
+
+ BUG_ON(!buffer_locked(bh));
+ BUG_ON(!buffer_mapped(bh));
+ BUG_ON(buffer_delay(bh));
+ BUG_ON(buffer_unwritten(bh));
+
+ /*
+ * Only clear out a write error when rewriting
+ */
+ if (test_set_buffer_req(bh) && (op == REQ_OP_WRITE))
+ clear_buffer_write_io_error(bh);
+
+ if (buffer_meta(bh))
+ opf |= REQ_META;
+ if (buffer_prio(bh))
+ opf |= REQ_PRIO;
+
+ bio = bio_alloc(bh->b_bdev, 1, opf, GFP_NOIO);
+
+ if (IS_ENABLED(CONFIG_FS_ENCRYPTION))
+ buffer_set_crypto_ctx(bio, bh, GFP_NOIO);
+
+ bio->bi_iter.bi_sector = bh->b_blocknr * (bh->b_size >> 9);
+ bio->bi_write_hint = write_hint;
+
+ bio_add_folio_nofail(bio, bh->b_folio, bh->b_size, bh_offset(bh));
+
+ bio->bi_end_io = end_bio;
+ bio->bi_private = bh;
+
+ /* Take care of bh's that straddle the end of the device */
+ guard_bio_eod(bio);
+
+ if (wbc) {
+ wbc_init_bio(wbc, bio);
+ wbc_account_cgroup_owner(wbc, bh->b_folio, bh->b_size);
+ }
+
+ blk_crypto_submit_bio(bio);
+}
+
+/**
+ * bh_submit - Start I/O against a buffer head
+ * @bh: The buffer head to perform I/O on.
+ * @opf: Operation and flags for bio.
+ * @end_io: The routine to call when I/O has completed.
+ *
+ * If you need to do I/O on an individual bh (instead of allowing the
+ * page cache to do I/O on the folio that it is in), call this function.
+ */
+void bh_submit(struct buffer_head *bh, blk_opf_t opf, bio_end_io_t end_io)
+{
+ __bh_submit(bh, opf, WRITE_LIFE_NOT_SET, NULL, end_io);
+}
+EXPORT_SYMBOL(bh_submit);
+
static struct buffer_head *__bread_slow(struct buffer_head *bh)
{
lock_buffer(bh);
@@ -1164,9 +1247,7 @@ static struct buffer_head *__bread_slow(struct buffer_head *bh)
unlock_buffer(bh);
return bh;
} else {
- get_bh(bh);
- bh->b_end_io = end_buffer_read_sync;
- submit_bh(REQ_OP_READ, bh);
+ bh_submit(bh, REQ_OP_READ, bh_end_read);
wait_on_buffer(bh);
if (buffer_uptodate(bh))
return bh;
@@ -1716,15 +1797,15 @@ static struct buffer_head *folio_create_buffers(struct folio *folio,
/*
* While block_write_full_folio is writing back the dirty buffers under
- * the page lock, whoever dirtied the buffers may decide to clean them
+ * the folio lock, whoever dirtied the buffers may decide to clean them
* again at any time. We handle that by only looking at the buffer
* state inside lock_buffer().
*
* If block_write_full_folio() is called for regular writeback
- * (wbc->sync_mode == WB_SYNC_NONE) then it will redirty a page which has a
- * locked buffer. This only can happen if someone has written the buffer
- * directly, with submit_bh(). At the address_space level PageWriteback
- * prevents this contention from occurring.
+ * (wbc->sync_mode == WB_SYNC_NONE) then it will redirty a folio which
+ * has a locked buffer. This only can happen if someone has written
+ * the buffer directly, with bh_submit(). At the address_space level
+ * the folio writeback flag prevents this contention from occurring.
*
* If block_write_full_folio() is called with wbc->sync_mode ==
* WB_SYNC_ALL, the writes are posted using REQ_SYNC; this
@@ -1810,8 +1891,7 @@ int __block_write_full_folio(struct inode *inode, struct folio *folio,
continue;
}
if (test_clear_buffer_dirty(bh)) {
- mark_buffer_async_write_endio(bh,
- end_buffer_async_write);
+ set_buffer_async_write(bh);
} else {
unlock_buffer(bh);
}
@@ -1827,8 +1907,9 @@ int __block_write_full_folio(struct inode *inode, struct folio *folio,
do {
struct buffer_head *next = bh->b_this_page;
if (buffer_async_write(bh)) {
- submit_bh_wbc(REQ_OP_WRITE | write_flags, bh,
- inode->i_write_hint, wbc);
+ __bh_submit(bh, REQ_OP_WRITE | write_flags,
+ inode->i_write_hint, wbc,
+ bh_end_async_write);
nr_underway++;
}
bh = next;
@@ -1841,7 +1922,7 @@ int __block_write_full_folio(struct inode *inode, struct folio *folio,
/*
* The folio was marked dirty, but the buffers were
* clean. Someone wrote them back by hand with
- * write_dirty_buffer/submit_bh. A rare case.
+ * write_dirty_buffer/bh_submit. A rare case.
*/
folio_end_writeback(folio);
@@ -1865,8 +1946,7 @@ int __block_write_full_folio(struct inode *inode, struct folio *folio,
if (buffer_mapped(bh) && buffer_dirty(bh) &&
!buffer_delay(bh)) {
lock_buffer(bh);
- mark_buffer_async_write_endio(bh,
- end_buffer_async_write);
+ set_buffer_async_write(bh);
} else {
/*
* The buffer may have been set dirty during
@@ -1882,8 +1962,9 @@ int __block_write_full_folio(struct inode *inode, struct folio *folio,
struct buffer_head *next = bh->b_this_page;
if (buffer_async_write(bh)) {
clear_buffer_dirty(bh);
- submit_bh_wbc(REQ_OP_WRITE | write_flags, bh,
- inode->i_write_hint, wbc);
+ __bh_submit(bh, REQ_OP_WRITE | write_flags,
+ inode->i_write_hint, wbc,
+ bh_end_async_write);
nr_underway++;
}
bh = next;
@@ -2339,9 +2420,33 @@ int block_read_full_folio(struct folio *folio, get_block_t *get_block)
continue;
}
- mark_buffer_async_read(bh);
+ /*
+ * If a folio's buffers are under async readin
+ * (end_buffer_async_read completion) then there is a
+ * possibility that another thread of control could lock
+ * one of the buffers after it has completed but while
+ * some of the other buffers have not completed. This
+ * locked buffer would confuse end_buffer_async_read()
+ * into not unlocking the folio. So the absence of
+ * BH_Async_Read tells end_buffer_async_read() that this
+ * buffer is not under async I/O.
+ *
+ * The folio comes unlocked when it has no locked
+ * buffer_async buffers left.
+ *
+ * The folio lock prevents anyone starting new async
+ * I/O reads into any of the buffers.
+ *
+ * The writeback flag is used to prevent simultaneous
+ * writeout of the same folio.
+ *
+ * The folio lock prevents anyone from starting writeback
+ * of a folio which is under read I/O (the writeback
+ * flag is only ever set on a locked folio).
+ */
+ set_buffer_async_read(bh);
if (prev)
- submit_bh(REQ_OP_READ, prev);
+ bh_submit(prev, REQ_OP_READ, bh_end_async_read);
prev = bh;
} while (iblock++, (bh = bh->b_this_page) != head);
@@ -2355,7 +2460,7 @@ int block_read_full_folio(struct folio *folio, get_block_t *get_block)
* in this folio.
*/
if (prev)
- submit_bh(REQ_OP_READ, prev);
+ bh_submit(prev, REQ_OP_READ, bh_end_async_read);
else
folio_end_read(folio, !page_error);
@@ -2663,86 +2768,6 @@ sector_t generic_block_bmap(struct address_space *mapping, sector_t block,
}
EXPORT_SYMBOL(generic_block_bmap);
-static void end_bio_bh_io_sync(struct bio *bio)
-{
- struct buffer_head *bh = bio->bi_private;
-
- if (unlikely(bio_flagged(bio, BIO_QUIET)))
- set_bit(BH_Quiet, &bh->b_state);
-
- bh->b_end_io(bh, !bio->bi_status);
- bio_put(bio);
-}
-
-static void buffer_set_crypto_ctx(struct bio *bio, const struct buffer_head *bh,
- gfp_t gfp_mask)
-{
- const struct address_space *mapping = folio_mapping(bh->b_folio);
-
- /*
- * The ext4 journal (jbd2) can submit a buffer_head it directly created
- * for a non-pagecache page. fscrypt doesn't care about these.
- */
- if (!mapping)
- return;
- fscrypt_set_bio_crypt_ctx(bio, mapping->host,
- folio_pos(bh->b_folio) + bh_offset(bh), gfp_mask);
-}
-
-static void submit_bh_wbc(blk_opf_t opf, struct buffer_head *bh,
- enum rw_hint write_hint,
- struct writeback_control *wbc)
-{
- const enum req_op op = opf & REQ_OP_MASK;
- struct bio *bio;
-
- BUG_ON(!buffer_locked(bh));
- BUG_ON(!buffer_mapped(bh));
- BUG_ON(!bh->b_end_io);
- BUG_ON(buffer_delay(bh));
- BUG_ON(buffer_unwritten(bh));
-
- /*
- * Only clear out a write error when rewriting
- */
- if (test_set_buffer_req(bh) && (op == REQ_OP_WRITE))
- clear_buffer_write_io_error(bh);
-
- if (buffer_meta(bh))
- opf |= REQ_META;
- if (buffer_prio(bh))
- opf |= REQ_PRIO;
-
- bio = bio_alloc(bh->b_bdev, 1, opf, GFP_NOIO);
-
- if (IS_ENABLED(CONFIG_FS_ENCRYPTION))
- buffer_set_crypto_ctx(bio, bh, GFP_NOIO);
-
- bio->bi_iter.bi_sector = bh->b_blocknr * (bh->b_size >> 9);
- bio->bi_write_hint = write_hint;
-
- bio_add_folio_nofail(bio, bh->b_folio, bh->b_size, bh_offset(bh));
-
- bio->bi_end_io = end_bio_bh_io_sync;
- bio->bi_private = bh;
-
- /* Take care of bh's that straddle the end of the device */
- guard_bio_eod(bio);
-
- if (wbc) {
- wbc_init_bio(wbc, bio);
- wbc_account_cgroup_owner(wbc, bh->b_folio, bh->b_size);
- }
-
- blk_crypto_submit_bio(bio);
-}
-
-void submit_bh(blk_opf_t opf, struct buffer_head *bh)
-{
- submit_bh_wbc(opf, bh, WRITE_LIFE_NOT_SET, NULL);
-}
-EXPORT_SYMBOL(submit_bh);
-
void write_dirty_buffer(struct buffer_head *bh, blk_opf_t op_flags)
{
lock_buffer(bh);
@@ -2750,9 +2775,7 @@ void write_dirty_buffer(struct buffer_head *bh, blk_opf_t op_flags)
unlock_buffer(bh);
return;
}
- bh->b_end_io = end_buffer_write_sync;
- get_bh(bh);
- submit_bh(REQ_OP_WRITE | op_flags, bh);
+ bh_submit(bh, REQ_OP_WRITE | op_flags, bh_end_write);
}
EXPORT_SYMBOL(write_dirty_buffer);
@@ -2775,9 +2798,7 @@ int __sync_dirty_buffer(struct buffer_head *bh, blk_opf_t op_flags)
return -EIO;
}
- get_bh(bh);
- bh->b_end_io = end_buffer_write_sync;
- submit_bh(REQ_OP_WRITE | op_flags, bh);
+ bh_submit(bh, REQ_OP_WRITE | op_flags, bh_end_write);
wait_on_buffer(bh);
if (!buffer_uptodate(bh))
return -EIO;
@@ -3009,9 +3030,7 @@ int __bh_read(struct buffer_head *bh, blk_opf_t op_flags, bool wait)
BUG_ON(!buffer_locked(bh));
- get_bh(bh);
- bh->b_end_io = end_buffer_read_sync;
- submit_bh(REQ_OP_READ | op_flags, bh);
+ bh_submit(bh, REQ_OP_READ | op_flags, bh_end_read);
if (wait) {
wait_on_buffer(bh);
if (!buffer_uptodate(bh))
@@ -3053,9 +3072,7 @@ void __bh_read_batch(int nr, struct buffer_head *bhs[],
continue;
}
- bh->b_end_io = end_buffer_read_sync;
- get_bh(bh);
- submit_bh(REQ_OP_READ | op_flags, bh);
+ bh_submit(bh, REQ_OP_READ | op_flags, bh_end_read);
}
}
EXPORT_SYMBOL(__bh_read_batch);
diff --git a/fs/ceph/file.c b/fs/ceph/file.c
index d54d716..0ad42e1 100644
--- a/fs/ceph/file.c
+++ b/fs/ceph/file.c
@@ -996,6 +996,10 @@ int ceph_atomic_open(struct inode *dir, struct dentry *dentry,
ceph_init_inode_acls(newino, &as_ctx);
file->f_mode |= FMODE_CREATED;
}
+ if ((flags & __O_REGULAR) && !d_is_reg(dentry)) {
+ err = -EFTYPE;
+ goto out_req;
+ }
err = finish_open(file, dentry, ceph_open);
}
out_req:
diff --git a/fs/ceph/mds_client.c b/fs/ceph/mds_client.c
index ed17e00..0edb6a2 100644
--- a/fs/ceph/mds_client.c
+++ b/fs/ceph/mds_client.c
@@ -2267,7 +2267,7 @@ static int trim_caps_cb(struct inode *inode, int mds, void *arg)
int count;
dput(dentry);
d_prune_aliases(inode);
- count = icount_read(inode);
+ count = icount_read_once(inode);
if (count == 1)
(*remaining)--;
doutc(cl, "%p %llx.%llx cap %p pruned, count now %d\n",
diff --git a/fs/configfs/configfs_internal.h b/fs/configfs/configfs_internal.h
index 0b969d0..acdeea8 100644
--- a/fs/configfs/configfs_internal.h
+++ b/fs/configfs/configfs_internal.h
@@ -76,7 +76,6 @@ extern int configfs_make_dirent(struct configfs_dirent *, struct dentry *,
extern int configfs_dirent_is_ready(struct configfs_dirent *);
extern const unsigned char * configfs_get_name(struct configfs_dirent *sd);
-extern void configfs_drop_dentry(struct configfs_dirent *sd, struct dentry *parent);
extern int configfs_setattr(struct mnt_idmap *idmap,
struct dentry *dentry, struct iattr *iattr);
diff --git a/fs/configfs/dir.c b/fs/configfs/dir.c
index 362b6ff..3c88f13 100644
--- a/fs/configfs/dir.c
+++ b/fs/configfs/dir.c
@@ -235,15 +235,16 @@ static int configfs_dirent_exists(struct dentry *dentry)
const unsigned char *new = dentry->d_name.name;
struct configfs_dirent *sd;
+ spin_lock(&configfs_dirent_lock);
list_for_each_entry(sd, &parent_sd->s_children, s_sibling) {
if (sd->s_element) {
- const unsigned char *existing = configfs_get_name(sd);
- if (strcmp(existing, new))
- continue;
- else
+ if (strcmp(configfs_get_name(sd), new) == 0) {
+ spin_unlock(&configfs_dirent_lock);
return -EEXIST;
+ }
}
}
+ spin_unlock(&configfs_dirent_lock);
return 0;
}
@@ -295,6 +296,7 @@ static int configfs_create_dir(struct config_item *item, struct dentry *dentry,
int error;
umode_t mode = S_IFDIR| S_IRWXU | S_IRUGO | S_IXUGO;
struct dentry *p = dentry->d_parent;
+ struct inode *p_inode = d_inode(p);
struct inode *inode;
BUG_ON(!item);
@@ -314,10 +316,9 @@ static int configfs_create_dir(struct config_item *item, struct dentry *dentry,
inode->i_fop = &configfs_dir_operations;
/* directory inodes start off with i_nlink == 2 (for "." entry) */
inc_nlink(inode);
- d_instantiate(dentry, inode);
- /* already hashed */
- dget(dentry); /* pin directory dentries in core */
- inc_nlink(d_inode(p));
+ d_make_persistent(dentry, inode);
+ inc_nlink(p_inode);
+ inode_set_mtime_to_ts(p_inode, inode_set_ctime_current(p_inode));
item->ci_dentry = dentry;
return 0;
@@ -371,6 +372,7 @@ int configfs_create_link(struct configfs_dirent *target, struct dentry *parent,
int err = 0;
umode_t mode = S_IFLNK | S_IRWXUGO;
struct configfs_dirent *p = parent->d_fsdata;
+ struct inode *p_inode = d_inode(parent);
struct inode *inode;
err = configfs_make_dirent(p, dentry, target, mode, CONFIGFS_ITEM_LINK,
@@ -384,8 +386,8 @@ int configfs_create_link(struct configfs_dirent *target, struct dentry *parent,
inode->i_link = body;
inode->i_op = &configfs_symlink_inode_operations;
- d_instantiate(dentry, inode);
- dget(dentry); /* pin link dentries in core */
+ d_make_persistent(dentry, inode);
+ inode_set_mtime_to_ts(p_inode, inode_set_ctime_current(p_inode));
return 0;
out_remove:
@@ -394,29 +396,9 @@ int configfs_create_link(struct configfs_dirent *target, struct dentry *parent,
return PTR_ERR(inode);
}
-static void remove_dir(struct dentry * d)
-{
- struct dentry * parent = dget(d->d_parent);
-
- configfs_remove_dirent(d);
-
- if (d_really_is_positive(d)) {
- if (likely(simple_empty(d))) {
- __simple_rmdir(d_inode(parent),d);
- dput(d);
- } else {
- pr_warn("remove_dir (%pd): attributes remain", d);
- }
- }
-
- pr_debug(" o %pd removing done (%d)\n", d, d_count(d));
-
- dput(parent);
-}
-
/**
* configfs_remove_dir - remove an config_item's directory.
- * @item: config_item we're removing.
+ * @d: dentry we're removing.
*
* The only thing special about this is that we remove any files in
* the directory before we remove the directory, and we've inlined
@@ -425,18 +407,20 @@ static void remove_dir(struct dentry * d)
* Caller holds the mutex of the item's inode
*/
-static void configfs_remove_dir(struct config_item * item)
+static void configfs_remove_dir(struct dentry *d)
{
- struct dentry * dentry = dget(item->ci_dentry);
+ struct dentry * parent = dget(d->d_parent);
- if (!dentry)
- return;
+ configfs_remove_dirent(d);
- remove_dir(dentry);
- /**
- * Drop reference from dget() on entrance.
- */
- dput(dentry);
+ if (d_really_is_positive(d)) {
+ if (unlikely(simple_rmdir(d_inode(parent), d)))
+ pr_warn("remove_dir (%pd): attributes remain", d);
+ }
+
+ pr_debug(" o %pd removing done (%d)\n", d, d_count(d));
+
+ dput(parent);
}
static struct dentry * configfs_lookup(struct inode *dir,
@@ -486,6 +470,9 @@ static struct dentry * configfs_lookup(struct inode *dir,
inode = configfs_create(dentry, mode);
if (IS_ERR(inode)) {
+ spin_lock(&configfs_dirent_lock);
+ sd->s_dentry = NULL;
+ spin_unlock(&configfs_dirent_lock);
configfs_put(sd);
return ERR_CAST(inode);
}
@@ -501,8 +488,7 @@ static struct dentry * configfs_lookup(struct inode *dir,
}
spin_unlock(&configfs_dirent_lock);
done:
- d_add(dentry, inode);
- return NULL;
+ return d_splice_alias(inode, dentry);
}
/*
@@ -513,9 +499,8 @@ static struct dentry * configfs_lookup(struct inode *dir,
* If there is an error, the caller will reset the flags via
* configfs_detach_rollback().
*/
-static int configfs_detach_prep(struct dentry *dentry, struct dentry **wait)
+static int configfs_detach_prep(struct configfs_dirent *parent_sd, struct dentry **wait)
{
- struct configfs_dirent *parent_sd = dentry->d_fsdata;
struct configfs_dirent *sd;
int ret;
@@ -543,7 +528,7 @@ static int configfs_detach_prep(struct dentry *dentry, struct dentry **wait)
* Yup, recursive. If there's a problem, blame
* deep nesting of default_groups
*/
- ret = configfs_detach_prep(sd->s_dentry, wait);
+ ret = configfs_detach_prep(sd, wait);
if (!ret)
continue;
} else
@@ -560,45 +545,77 @@ static int configfs_detach_prep(struct dentry *dentry, struct dentry **wait)
* Walk the tree, resetting CONFIGFS_USET_DROPPING wherever it was
* set.
*/
-static void configfs_detach_rollback(struct dentry *dentry)
+static void configfs_detach_rollback(struct configfs_dirent *parent_sd)
{
- struct configfs_dirent *parent_sd = dentry->d_fsdata;
struct configfs_dirent *sd;
parent_sd->s_type &= ~CONFIGFS_USET_DROPPING;
list_for_each_entry(sd, &parent_sd->s_children, s_sibling)
if (sd->s_type & CONFIGFS_USET_DEFAULT)
- configfs_detach_rollback(sd->s_dentry);
+ configfs_detach_rollback(sd);
}
-static void detach_attrs(struct config_item * item)
+/*
+ * Find the next non-cursor. configfs_dirent_lock held by caller.
+ */
+static struct configfs_dirent *next_dirent(struct configfs_dirent *parent,
+ struct configfs_dirent *last)
{
- struct dentry * dentry = dget(item->ci_dentry);
- struct configfs_dirent * parent_sd;
- struct configfs_dirent * sd, * tmp;
+ struct configfs_dirent *s;
- if (!dentry)
- return;
+ s = list_prepare_entry(last, &parent->s_children, s_sibling);
- pr_debug("configfs %s: dropping attrs for dir\n",
- dentry->d_name.name);
+ list_for_each_entry_continue(s, &parent->s_children, s_sibling) {
+ if (s->s_element)
+ return s;
+ }
+ return NULL;
+}
+
+static void detach_attrs(struct dentry *dentry)
+{
+ struct configfs_dirent *parent_sd;
+ struct configfs_dirent *sd, *next;
+
+ pr_debug("configfs %pd: dropping attrs for dir\n", dentry);
parent_sd = dentry->d_fsdata;
- list_for_each_entry_safe(sd, tmp, &parent_sd->s_children, s_sibling) {
- if (!sd->s_element || !(sd->s_type & CONFIGFS_NOT_PINNED))
- continue;
- spin_lock(&configfs_dirent_lock);
- list_del_init(&sd->s_sibling);
- spin_unlock(&configfs_dirent_lock);
- configfs_drop_dentry(sd, dentry);
- configfs_put(sd);
- }
- /**
- * Drop reference from dget() on entrance.
- */
- dput(dentry);
+ spin_lock(&configfs_dirent_lock);
+ for (sd = next_dirent(parent_sd, NULL); sd; sd = next) {
+ struct dentry *child;
+
+ next = next_dirent(parent_sd, sd);
+ if (!(sd->s_type & CONFIGFS_NOT_PINNED))
+ continue;
+ list_del_init(&sd->s_sibling);
+ child = sd->s_dentry;
+ if (child) {
+ spin_lock(&child->d_lock);
+ if (simple_positive(child)) {
+ dget_dlock(child);
+ __d_drop(child);
+ spin_unlock(&child->d_lock);
+ } else {
+ spin_unlock(&child->d_lock);
+ child = NULL;
+ }
+ }
+ spin_unlock(&configfs_dirent_lock);
+ if (child) {
+ struct inode *inode = child->d_inode;
+
+ inode_lock_nested(inode, I_MUTEX_NONDIR2);
+ inode_set_ctime_current(inode);
+ drop_nlink(inode);
+ inode_unlock(inode);
+ dput(child);
+ }
+ configfs_put(sd);
+ spin_lock(&configfs_dirent_lock);
+ }
+ spin_unlock(&configfs_dirent_lock);
}
static int populate_attrs(struct config_item *item)
@@ -620,54 +637,49 @@ static int populate_attrs(struct config_item *item)
if (ops && ops->is_visible && !ops->is_visible(item, attr, i))
continue;
- if ((error = configfs_create_file(item, attr)))
- break;
+ error = configfs_create_file(item, attr);
+ if (error)
+ return error;
}
}
- if (!error && t->ct_bin_attrs) {
+ if (t->ct_bin_attrs) {
for (i = 0; (bin_attr = t->ct_bin_attrs[i]) != NULL; i++) {
if (ops && ops->is_bin_visible && !ops->is_bin_visible(item, bin_attr, i))
continue;
error = configfs_create_bin_file(item, bin_attr);
if (error)
- break;
+ return error;
}
}
- if (error)
- detach_attrs(item);
-
- return error;
+ return 0;
}
-static int configfs_attach_group(struct config_item *parent_item,
- struct config_item *item,
+static int configfs_attach_group(struct config_item *item,
struct dentry *dentry,
struct configfs_fragment *frag);
-static void configfs_detach_group(struct config_item *item);
+static void configfs_detach_group(struct dentry *dentry);
-static void detach_groups(struct config_group *group)
+static void detach_groups(struct dentry *dentry)
{
- struct dentry * dentry = dget(group->cg_item.ci_dentry);
struct dentry *child;
struct configfs_dirent *parent_sd;
- struct configfs_dirent *sd, *tmp;
-
- if (!dentry)
- return;
+ struct configfs_dirent *sd, *next;
parent_sd = dentry->d_fsdata;
- list_for_each_entry_safe(sd, tmp, &parent_sd->s_children, s_sibling) {
- if (!sd->s_element ||
- !(sd->s_type & CONFIGFS_USET_DEFAULT))
+ spin_lock(&configfs_dirent_lock);
+ for (sd = next_dirent(parent_sd, NULL); sd; sd = next) {
+ next = next_dirent(parent_sd, sd);
+ if (!(sd->s_type & CONFIGFS_USET_DEFAULT))
continue;
- child = sd->s_dentry;
+ child = dget(sd->s_dentry);
+ spin_unlock(&configfs_dirent_lock);
inode_lock(d_inode(child));
- configfs_detach_group(sd->s_element);
+ configfs_detach_group(child);
d_inode(child)->i_flags |= S_DEAD;
dont_mount(child);
@@ -675,12 +687,9 @@ static void detach_groups(struct config_group *group)
d_delete(child);
dput(child);
+ spin_lock(&configfs_dirent_lock);
}
-
- /**
- * Drop reference from dget() on entrance.
- */
- dput(dentry);
+ spin_unlock(&configfs_dirent_lock);
}
/*
@@ -691,14 +700,14 @@ static void detach_groups(struct config_group *group)
* We could, perhaps, tweak our parent's ->mkdir for a minute and
* try using vfs_mkdir. Just a thought.
*/
-static int create_default_group(struct config_group *parent_group,
+static int create_default_group(struct dentry *parent,
struct config_group *group,
struct configfs_fragment *frag)
{
int ret;
struct configfs_dirent *sd;
/* We trust the caller holds a reference to parent */
- struct dentry *child, *parent = parent_group->cg_item.ci_dentry;
+ struct dentry *child;
if (!group->cg_item.ci_name)
group->cg_item.ci_name = group->cg_item.ci_namebuf;
@@ -708,16 +717,15 @@ static int create_default_group(struct config_group *parent_group,
if (child) {
d_add(child, NULL);
- ret = configfs_attach_group(&parent_group->cg_item,
- &group->cg_item, child, frag);
+ ret = configfs_attach_group(&group->cg_item, child, frag);
if (!ret) {
sd = child->d_fsdata;
sd->s_type |= CONFIGFS_USET_DEFAULT;
} else {
BUG_ON(d_inode(child));
d_drop(child);
- dput(child);
}
+ dput(child);
}
return ret;
@@ -726,18 +734,15 @@ static int create_default_group(struct config_group *parent_group,
static int populate_groups(struct config_group *group,
struct configfs_fragment *frag)
{
+ struct dentry *parent = group->cg_item.ci_dentry;
struct config_group *new_group;
- int ret = 0;
list_for_each_entry(new_group, &group->default_groups, group_entry) {
- ret = create_default_group(group, new_group, frag);
- if (ret) {
- detach_groups(group);
- break;
- }
+ int ret = create_default_group(parent, new_group, frag);
+ if (ret)
+ return ret;
}
-
- return ret;
+ return 0;
}
void configfs_remove_default_groups(struct config_group *group)
@@ -827,6 +832,13 @@ static void link_group(struct config_group *parent_group, struct config_group *g
link_group(group, new_group);
}
+/* Caller holds the mutex of the item's inode */
+static void configfs_detach_item(struct dentry *dentry)
+{
+ detach_attrs(dentry);
+ configfs_remove_dir(dentry);
+}
+
/*
* The goal is that configfs_attach_item() (and
* configfs_attach_group()) can be called from either the VFS or this
@@ -842,8 +854,7 @@ static void link_group(struct config_group *parent_group, struct config_group *g
* clean up the configfs items, and they expect their callers will
* handle the dcache bits.
*/
-static int configfs_attach_item(struct config_item *parent_item,
- struct config_item *item,
+static int configfs_attach_item(struct config_item *item,
struct dentry *dentry,
struct configfs_fragment *frag)
{
@@ -859,7 +870,7 @@ static int configfs_attach_item(struct config_item *parent_item,
* we must lock them as rmdir() would.
*/
inode_lock(d_inode(dentry));
- configfs_remove_dir(item);
+ configfs_detach_item(dentry);
d_inode(dentry)->i_flags |= S_DEAD;
dont_mount(dentry);
inode_unlock(d_inode(dentry));
@@ -870,22 +881,21 @@ static int configfs_attach_item(struct config_item *parent_item,
return ret;
}
-/* Caller holds the mutex of the item's inode */
-static void configfs_detach_item(struct config_item *item)
+/* Caller holds the mutex of the group's inode */
+static void configfs_detach_group(struct dentry *dentry)
{
- detach_attrs(item);
- configfs_remove_dir(item);
+ detach_groups(dentry);
+ configfs_detach_item(dentry);
}
-static int configfs_attach_group(struct config_item *parent_item,
- struct config_item *item,
+static int configfs_attach_group(struct config_item *item,
struct dentry *dentry,
struct configfs_fragment *frag)
{
int ret;
struct configfs_dirent *sd;
- ret = configfs_attach_item(parent_item, item, dentry, frag);
+ ret = configfs_attach_item(item, dentry, frag);
if (!ret) {
sd = dentry->d_fsdata;
sd->s_type |= CONFIGFS_USET_DIR;
@@ -903,7 +913,7 @@ static int configfs_attach_group(struct config_item *parent_item,
configfs_adjust_dir_dirent_depth_before_populate(sd);
ret = populate_groups(to_config_group(item), frag);
if (ret) {
- configfs_detach_item(item);
+ configfs_detach_group(dentry);
d_inode(dentry)->i_flags |= S_DEAD;
dont_mount(dentry);
}
@@ -916,13 +926,6 @@ static int configfs_attach_group(struct config_item *parent_item,
return ret;
}
-/* Caller holds the mutex of the group's inode */
-static void configfs_detach_group(struct config_item *item)
-{
- detach_groups(to_config_group(item));
- configfs_detach_item(item);
-}
-
/*
* After the item has been detached from the filesystem view, we are
* ready to tear it out of the hierarchy. Notify the client before
@@ -1065,15 +1068,12 @@ static int configfs_dump(struct configfs_dirent *sd, int level)
* much on the stack, though, so folks that need this function - be careful
* about your stack! Patches will be accepted to make it iterative.
*/
-static int configfs_depend_prep(struct dentry *origin,
+static int configfs_depend_prep(struct configfs_dirent *sd,
struct config_item *target)
{
- struct configfs_dirent *child_sd, *sd;
+ struct configfs_dirent *child_sd;
int ret = 0;
- BUG_ON(!origin || !origin->d_fsdata);
- sd = origin->d_fsdata;
-
if (sd->s_element == target) /* Boo-yah */
goto out;
@@ -1081,8 +1081,7 @@ static int configfs_depend_prep(struct dentry *origin,
if ((child_sd->s_type & CONFIGFS_DIR) &&
!(child_sd->s_type & CONFIGFS_USET_DROPPING) &&
!(child_sd->s_type & CONFIGFS_USET_CREATING)) {
- ret = configfs_depend_prep(child_sd->s_dentry,
- target);
+ ret = configfs_depend_prep(child_sd, target);
if (!ret)
goto out; /* Child path boo-yah */
}
@@ -1095,7 +1094,7 @@ static int configfs_depend_prep(struct dentry *origin,
return ret;
}
-static int configfs_do_depend_item(struct dentry *subsys_dentry,
+static int configfs_do_depend_item(struct configfs_dirent *subsys_sd,
struct config_item *target)
{
struct configfs_dirent *p;
@@ -1103,7 +1102,7 @@ static int configfs_do_depend_item(struct dentry *subsys_dentry,
spin_lock(&configfs_dirent_lock);
/* Scan the tree, return 0 if found */
- ret = configfs_depend_prep(subsys_dentry, target);
+ ret = configfs_depend_prep(subsys_sd, target);
if (ret)
goto out_unlock_dirent_lock;
@@ -1127,6 +1126,7 @@ configfs_find_subsys_dentry(struct configfs_dirent *root_sd,
struct configfs_dirent *p;
struct configfs_dirent *ret = NULL;
+ spin_lock(&configfs_dirent_lock);
list_for_each_entry(p, &root_sd->s_children, s_sibling) {
if (p->s_type & CONFIGFS_DIR &&
p->s_element == subsys_item) {
@@ -1134,6 +1134,7 @@ configfs_find_subsys_dentry(struct configfs_dirent *root_sd,
break;
}
}
+ spin_unlock(&configfs_dirent_lock);
return ret;
}
@@ -1169,7 +1170,7 @@ int configfs_depend_item(struct configfs_subsystem *subsys,
}
/* Ok, now we can trust subsys/s_item */
- ret = configfs_do_depend_item(subsys_sd->s_dentry, target);
+ ret = configfs_do_depend_item(subsys_sd, target);
out_unlock_fs:
inode_unlock(d_inode(root));
@@ -1271,7 +1272,7 @@ int configfs_depend_item_unlocked(struct configfs_subsystem *caller_subsys,
}
/* Now we can execute core of depend item */
- ret = configfs_do_depend_item(subsys_sd->s_dentry, target);
+ ret = configfs_do_depend_item(subsys_sd, target);
if (target_subsys != caller_subsys)
out_root_unlock:
@@ -1298,7 +1299,11 @@ static struct dentry *configfs_mkdir(struct mnt_idmap *idmap, struct inode *dir,
const struct config_item_type *type;
struct module *subsys_owner = NULL, *new_item_owner = NULL;
struct configfs_fragment *frag;
- char *name;
+ struct name_snapshot n;
+ const char *name;
+
+ take_dentry_name_snapshot(&n, dentry);
+ name = n.name.name;
sd = dentry->d_parent->d_fsdata;
@@ -1350,14 +1355,6 @@ static struct dentry *configfs_mkdir(struct mnt_idmap *idmap, struct inode *dir,
goto out_put;
}
- name = kmalloc(dentry->d_name.len + 1, GFP_KERNEL);
- if (!name) {
- ret = -ENOMEM;
- goto out_subsys_put;
- }
-
- snprintf(name, dentry->d_name.len + 1, "%s", dentry->d_name.name);
-
mutex_lock(&subsys->su_mutex);
if (type->ct_group_ops->make_group) {
group = type->ct_group_ops->make_group(to_config_group(parent_item), name);
@@ -1379,7 +1376,6 @@ static struct dentry *configfs_mkdir(struct mnt_idmap *idmap, struct inode *dir,
}
mutex_unlock(&subsys->su_mutex);
- kfree(name);
if (ret) {
/*
* If ret != 0, then link_obj() was never called.
@@ -1424,9 +1420,9 @@ static struct dentry *configfs_mkdir(struct mnt_idmap *idmap, struct inode *dir,
spin_unlock(&configfs_dirent_lock);
if (group)
- ret = configfs_attach_group(parent_item, item, dentry, frag);
+ ret = configfs_attach_group(item, dentry, frag);
else
- ret = configfs_attach_item(parent_item, item, dentry, frag);
+ ret = configfs_attach_item(item, dentry, frag);
spin_lock(&configfs_dirent_lock);
sd->s_type &= ~CONFIGFS_USET_IN_MKDIR;
@@ -1466,6 +1462,7 @@ static struct dentry *configfs_mkdir(struct mnt_idmap *idmap, struct inode *dir,
put_fragment(frag);
out:
+ release_dentry_name_snapshot(&n);
return ERR_PTR(ret);
}
@@ -1513,9 +1510,9 @@ static int configfs_rmdir(struct inode *dir, struct dentry *dentry)
*/
ret = sd->s_dependent_count ? -EBUSY : 0;
if (!ret) {
- ret = configfs_detach_prep(dentry, &wait);
+ ret = configfs_detach_prep(sd, &wait);
if (ret)
- configfs_detach_rollback(dentry);
+ configfs_detach_rollback(sd);
}
spin_unlock(&configfs_dirent_lock);
mutex_unlock(&configfs_symlink_mutex);
@@ -1536,7 +1533,7 @@ static int configfs_rmdir(struct inode *dir, struct dentry *dentry)
frag = sd->s_frag;
if (down_write_killable(&frag->frag_sem)) {
spin_lock(&configfs_dirent_lock);
- configfs_detach_rollback(dentry);
+ configfs_detach_rollback(sd);
spin_unlock(&configfs_dirent_lock);
config_item_put(parent_item);
return -EINTR;
@@ -1554,13 +1551,13 @@ static int configfs_rmdir(struct inode *dir, struct dentry *dentry)
dead_item_owner = item->ci_type->ct_owner;
if (sd->s_type & CONFIGFS_USET_DIR) {
- configfs_detach_group(item);
+ configfs_detach_group(dentry);
mutex_lock(&subsys->su_mutex);
client_disconnect_notify(parent_item, item);
unlink_group(to_config_group(item));
} else {
- configfs_detach_item(item);
+ configfs_detach_item(dentry);
mutex_lock(&subsys->su_mutex);
client_disconnect_notify(parent_item, item);
@@ -1770,7 +1767,7 @@ int configfs_register_group(struct config_group *parent_group,
parent = parent_group->cg_item.ci_dentry;
inode_lock_nested(d_inode(parent), I_MUTEX_PARENT);
- ret = create_default_group(parent_group, group, frag);
+ ret = create_default_group(parent, group, frag);
if (ret)
goto err_out;
@@ -1799,7 +1796,7 @@ EXPORT_SYMBOL(configfs_register_group);
void configfs_unregister_group(struct config_group *group)
{
struct configfs_subsystem *subsys = group->cg_subsys;
- struct dentry *dentry = group->cg_item.ci_dentry;
+ struct dentry *dentry = dget(group->cg_item.ci_dentry);
struct dentry *parent = group->cg_item.ci_parent->ci_dentry;
struct configfs_dirent *sd = dentry->d_fsdata;
struct configfs_fragment *frag = sd->s_frag;
@@ -1810,10 +1807,10 @@ void configfs_unregister_group(struct config_group *group)
inode_lock_nested(d_inode(parent), I_MUTEX_PARENT);
spin_lock(&configfs_dirent_lock);
- configfs_detach_prep(dentry, NULL);
+ configfs_detach_prep(sd, NULL);
spin_unlock(&configfs_dirent_lock);
- configfs_detach_group(&group->cg_item);
+ configfs_detach_group(dentry);
d_inode(dentry)->i_flags |= S_DEAD;
dont_mount(dentry);
d_drop(dentry);
@@ -1908,18 +1905,17 @@ int configfs_register_subsystem(struct configfs_subsystem *subsys)
err = configfs_dirent_exists(dentry);
if (!err)
- err = configfs_attach_group(sd->s_element,
- &group->cg_item,
+ err = configfs_attach_group(&group->cg_item,
dentry, frag);
if (err) {
BUG_ON(d_inode(dentry));
d_drop(dentry);
- dput(dentry);
} else {
spin_lock(&configfs_dirent_lock);
configfs_dir_set_ready(dentry->d_fsdata);
spin_unlock(&configfs_dirent_lock);
}
+ dput(dentry);
}
inode_unlock(d_inode(root));
@@ -1938,7 +1934,7 @@ int configfs_register_subsystem(struct configfs_subsystem *subsys)
void configfs_unregister_subsystem(struct configfs_subsystem *subsys)
{
struct config_group *group = &subsys->su_group;
- struct dentry *dentry = group->cg_item.ci_dentry;
+ struct dentry *dentry = dget(group->cg_item.ci_dentry);
struct dentry *root = dentry->d_sb->s_root;
struct configfs_dirent *sd = dentry->d_fsdata;
struct configfs_fragment *frag = sd->s_frag;
@@ -1957,12 +1953,12 @@ void configfs_unregister_subsystem(struct configfs_subsystem *subsys)
inode_lock_nested(d_inode(dentry), I_MUTEX_CHILD);
mutex_lock(&configfs_symlink_mutex);
spin_lock(&configfs_dirent_lock);
- if (configfs_detach_prep(dentry, NULL)) {
+ if (configfs_detach_prep(sd, NULL)) {
pr_err("Tried to unregister non-empty subsystem!\n");
}
spin_unlock(&configfs_dirent_lock);
mutex_unlock(&configfs_symlink_mutex);
- configfs_detach_group(&group->cg_item);
+ configfs_detach_group(dentry);
d_inode(dentry)->i_flags |= S_DEAD;
dont_mount(dentry);
inode_unlock(d_inode(dentry));
diff --git a/fs/configfs/file.c b/fs/configfs/file.c
index ef8c3cd..a48cece 100644
--- a/fs/configfs/file.c
+++ b/fs/configfs/file.c
@@ -59,7 +59,7 @@ static int fill_read_buffer(struct file *file, struct configfs_buffer *buffer)
ssize_t count = -ENOENT;
if (!buffer->page)
- buffer->page = (char *) get_zeroed_page(GFP_KERNEL);
+ buffer->page = kzalloc(PAGE_SIZE, GFP_KERNEL);
if (!buffer->page)
return -ENOMEM;
@@ -184,7 +184,7 @@ static int fill_write_buffer(struct configfs_buffer *buffer,
int copied;
if (!buffer->page)
- buffer->page = (char *)__get_free_pages(GFP_KERNEL, 0);
+ buffer->page = kmalloc(PAGE_SIZE, GFP_KERNEL);
if (!buffer->page)
return -ENOMEM;
@@ -381,8 +381,7 @@ static int configfs_release(struct inode *inode, struct file *filp)
struct configfs_buffer *buffer = filp->private_data;
module_put(buffer->owner);
- if (buffer->page)
- free_page((unsigned long)buffer->page);
+ kfree(buffer->page);
mutex_destroy(&buffer->mutex);
kfree(buffer);
return 0;
diff --git a/fs/configfs/inode.c b/fs/configfs/inode.c
index bc507b7..68290fe 100644
--- a/fs/configfs/inode.c
+++ b/fs/configfs/inode.c
@@ -157,7 +157,6 @@ struct inode *configfs_create(struct dentry *dentry, umode_t mode)
{
struct inode *inode = NULL;
struct configfs_dirent *sd;
- struct inode *p_inode;
if (!dentry)
return ERR_PTR(-ENOENT);
@@ -170,8 +169,6 @@ struct inode *configfs_create(struct dentry *dentry, umode_t mode)
if (!inode)
return ERR_PTR(-ENOMEM);
- p_inode = d_inode(dentry->d_parent);
- inode_set_mtime_to_ts(p_inode, inode_set_ctime_current(p_inode));
configfs_set_inode_lock_class(sd, inode);
return inode;
}
@@ -195,25 +192,3 @@ const unsigned char * configfs_get_name(struct configfs_dirent *sd)
}
return NULL;
}
-
-
-/*
- * Unhashes the dentry corresponding to given configfs_dirent
- * Called with parent inode's i_mutex held.
- */
-void configfs_drop_dentry(struct configfs_dirent * sd, struct dentry * parent)
-{
- struct dentry * dentry = sd->s_dentry;
-
- if (dentry) {
- spin_lock(&dentry->d_lock);
- if (simple_positive(dentry)) {
- dget_dlock(dentry);
- __d_drop(dentry);
- spin_unlock(&dentry->d_lock);
- __simple_unlink(d_inode(parent), dentry);
- dput(dentry);
- } else
- spin_unlock(&dentry->d_lock);
- }
-}
diff --git a/fs/configfs/symlink.c b/fs/configfs/symlink.c
index f3f79c6..31eb28b 100644
--- a/fs/configfs/symlink.c
+++ b/fs/configfs/symlink.c
@@ -229,9 +229,8 @@ int configfs_unlink(struct inode *dir, struct dentry *dentry)
spin_lock(&configfs_dirent_lock);
list_del_init(&sd->s_sibling);
spin_unlock(&configfs_dirent_lock);
- configfs_drop_dentry(sd, dentry->d_parent);
- dput(dentry);
configfs_put(sd);
+ simple_unlink(dir, dentry);
/*
* drop_link() must be called before
diff --git a/fs/coredump.c b/fs/coredump.c
index bb6fdb1..e68a76f 100644
--- a/fs/coredump.c
+++ b/fs/coredump.c
@@ -395,8 +395,7 @@ static bool coredump_parse(struct core_name *cn, struct coredump_params *cprm,
cred->gid));
break;
case 'd':
- err = cn_printf(cn, "%d",
- __get_dumpable(cprm->mm_flags));
+ err = cn_printf(cn, "%d", cprm->dumpable);
break;
/* signal that caused the coredump */
case 's':
@@ -869,11 +868,11 @@ static inline void coredump_sock_shutdown(struct file *file) { }
static inline bool coredump_socket(struct core_name *cn, struct coredump_params *cprm) { return false; }
#endif
-/* cprm->mm_flags contains a stable snapshot of dumpability flags. */
+/* cprm->dumpable is the snapshot of task dumpability at dump start. */
static inline bool coredump_force_suid_safe(const struct coredump_params *cprm)
{
/* Require nonrelative corefile path and be extra careful. */
- return __get_dumpable(cprm->mm_flags) == SUID_DUMP_ROOT;
+ return cprm->dumpable == TASK_DUMPABLE_ROOT;
}
static bool coredump_file(struct core_name *cn, struct coredump_params *cprm,
@@ -1085,7 +1084,7 @@ static inline bool coredump_skip(const struct coredump_params *cprm,
return true;
if (!binfmt->core_dump)
return true;
- if (!__get_dumpable(cprm->mm_flags))
+ if (cprm->dumpable == TASK_DUMPABLE_OFF)
return true;
return false;
}
@@ -1170,14 +1169,9 @@ void vfs_coredump(const kernel_siginfo_t *siginfo)
struct coredump_params cprm = {
.siginfo = siginfo,
.limit = rlimit(RLIMIT_CORE),
- /*
- * We must use the same mm->flags while dumping core to avoid
- * inconsistency of bit flags, since this flag is not protected
- * by any locks.
- *
- * Note that we only care about MMF_DUMP* flags.
- */
- .mm_flags = __mm_flags_get_dumpable(mm),
+ /* Snapshot MMF_DUMP_FILTER_* (unlocked) and dumpable for the dump. */
+ .mm_flags = __mm_flags_get_word(mm),
+ .dumpable = task_exec_state_get_dumpable(current),
.vma_meta = NULL,
.cpu = raw_smp_processor_id(),
};
@@ -1419,7 +1413,7 @@ EXPORT_SYMBOL(dump_align);
void validate_coredump_safety(void)
{
- if (suid_dumpable == SUID_DUMP_ROOT &&
+ if (suid_dumpable == TASK_DUMPABLE_ROOT &&
core_pattern[0] != '/' && core_pattern[0] != '|' && core_pattern[0] != '@') {
coredump_report_failure("Unsafe core_pattern used with fs.suid_dumpable=2: "
@@ -1488,7 +1482,8 @@ static int proc_dostring_coredump(const struct ctl_table *table, int write,
return -EINVAL;
}
- validate_coredump_safety();
+ if (strncmp(old_core_pattern, core_pattern, CORENAME_MAX_SIZE))
+ validate_coredump_safety();
return error;
}
diff --git a/fs/dcache.c b/fs/dcache.c
index 2c61aee..3e9af9d 100644
--- a/fs/dcache.c
+++ b/fs/dcache.c
@@ -43,8 +43,8 @@
* - i_dentry, d_alias, d_inode of aliases
* dcache_hash_bucket lock protects:
* - the dcache hash table
- * s_roots bl list spinlock protects:
- * - the s_roots list (see __d_drop)
+ * s_roots_lock protects:
+ * - the s_roots list (see __d_move()/dentry_unlist()/d_obtain_root())
* dentry->d_sb->s_dentry_lru_lock protects:
* - the dcache lru lists and counters
* d_lock protects:
@@ -426,9 +426,16 @@ static inline void __d_clear_type_and_inode(struct dentry *dentry)
this_cpu_inc(nr_dentry_negative);
}
+#define DENTRY_WARN_ONCE(condition, dentry) \
+ WARN_ONCE((condition), "dentry=%p d_flags=0x%x\n", (dentry), (dentry)->d_flags)
+#define D_FLAG_VERIFY(dentry, x) \
+ DENTRY_WARN_ONCE(((dentry)->d_flags & (DCACHE_LRU_LIST | DCACHE_SHRINK_LIST)) != (x), (dentry))
+
static void dentry_free(struct dentry *dentry)
{
- WARN_ON(d_really_is_positive(dentry));
+ DENTRY_WARN_ONCE(d_really_is_positive(dentry), dentry);
+ DENTRY_WARN_ONCE(dentry->d_lockref.count >= 0, dentry);
+ D_FLAG_VERIFY(dentry, 0);
if (unlikely(dname_external(dentry))) {
struct external_name *p = external_name(dentry);
if (likely(atomic_dec_and_test(&p->count))) {
@@ -455,14 +462,10 @@ static void dentry_unlink_inode(struct dentry * dentry)
raw_write_seqcount_begin(&dentry->d_seq);
__d_clear_type_and_inode(dentry);
- hlist_del_init(&dentry->d_alias);
+ __hlist_del(&dentry->d_alias);
/*
* dentry becomes negative, so the space occupied by ->d_alias
- * belongs to ->waiters now; we could use __hlist_del() instead
- * of hlist_del_init(), if not for the stunt pulled by nfs
- * dummy root dentries - positive dentry *not* included into
- * the alias list of its inode. Open-coding hlist_del_init()
- * and removing zeroing would be too clumsy...
+ * belongs to ->waiters now.
*/
dentry->waiters = NULL;
raw_write_seqcount_end(&dentry->d_seq);
@@ -495,7 +498,6 @@ static void dentry_unlink_inode(struct dentry * dentry)
* These helper functions make sure we always follow the
* rules. d_lock must be held by the caller.
*/
-#define D_FLAG_VERIFY(dentry,x) WARN_ON_ONCE(((dentry)->d_flags & (DCACHE_LRU_LIST | DCACHE_SHRINK_LIST)) != (x))
static void d_lru_add(struct dentry *dentry)
{
D_FLAG_VERIFY(dentry, 0);
@@ -562,16 +564,7 @@ static void d_lru_shrink_move(struct list_lru_one *lru, struct dentry *dentry,
static void ___d_drop(struct dentry *dentry)
{
- struct hlist_bl_head *b;
- /*
- * Hashed dentries are normally on the dentry hashtable,
- * with the exception of those newly allocated by
- * d_obtain_root, which are always IS_ROOT:
- */
- if (unlikely(IS_ROOT(dentry)))
- b = &dentry->d_sb->s_roots;
- else
- b = d_hash(dentry->d_name.hash);
+ struct hlist_bl_head *b = d_hash(dentry->d_name.hash);
hlist_bl_lock(b);
__hlist_bl_del(&dentry->d_hash);
@@ -630,12 +623,14 @@ struct completion_list {
* Use ->waiters for a single-linked list of struct completion_list of
* waiters.
*/
-static inline void d_add_waiter(struct dentry *dentry, struct completion_list *p)
+static inline bool d_add_waiter(struct dentry *dentry, struct completion_list *p)
{
- struct completion_list *v = dentry->waiters;
+ if (unlikely(dentry->d_flags & DCACHE_DENTRY_KILLED))
+ return false;
init_completion(&p->completion);
- p->next = v;
+ p->next = dentry->waiters;
dentry->waiters = p;
+ return true;
}
static inline void d_complete_waiters(struct dentry *dentry)
@@ -652,6 +647,13 @@ static inline void d_complete_waiters(struct dentry *dentry)
}
}
+static void unlink_secondary_root(struct dentry *dentry)
+{
+ spin_lock(&dentry->d_sb->s_roots_lock);
+ hlist_del_init(&dentry->d_sib);
+ spin_unlock(&dentry->d_sb->s_roots_lock);
+}
+
static inline void dentry_unlist(struct dentry *dentry)
{
struct dentry *next;
@@ -663,6 +665,10 @@ static inline void dentry_unlist(struct dentry *dentry)
d_complete_waiters(dentry);
if (unlikely(hlist_unhashed(&dentry->d_sib)))
return;
+ if (unlikely(IS_ROOT(dentry))) {
+ unlink_secondary_root(dentry); // secondary root goes away
+ return;
+ }
__hlist_del(&dentry->d_sib);
/*
* Cursors can move around the list of children. While we'd been
@@ -691,11 +697,113 @@ static inline void dentry_unlist(struct dentry *dentry)
}
}
-static struct dentry *__dentry_kill(struct dentry *dentry)
+/*
+ * Prepare locking environment for killing a dentry.
+ * Called under dentry->d_lock. To proceed with eviction of a positive dentry
+ * we need to get ->i_lock of the inode of that dentry as well.
+ * However, ->i_lock nests outside of ->d_lock, so if trylock fails we might
+ * have to drop and regain the latter. Dentry state can change while its
+ * ->d_lock is not held - it might end up getting killed, becoming busy,
+ * negative, etc., so we need to be careful.
+ *
+ * For NORCU dentries memory safety relies upon having only one call of
+ * lock_for_kill() in the entire lifetime of dentry and dentry_free() being
+ * called only by the caller of lock_for_kill(). That this is NORCU-specific;
+ * the crucial part is that refcounts of NORCU dentries never grow once having
+ * dropped to zero.
+ *
+ * For normal dentries we can not assume that there won't be concurrent calls
+ * of dentry_free() - dentry might end up being evicted by another thread
+ * while we are dropping/retaking locks on the slow path. Memory safety is
+ * provided by keeping the RCU read-side critical area contiguous with
+ * an explicit rcu_read_lock() scope bridging over the break in spinlock scopes.
+ *
+ * If dentry is busy (or busy dying, or already dead), unlock dentry
+ * and return false. Otherwise, return true and have that dentry's
+ * inode (if any) locked in addition to dentry itself.
+ */
+static bool lock_for_kill(struct dentry *dentry)
+{
+ struct inode *inode = dentry->d_inode;
+
+ if (unlikely(dentry->d_lockref.count)) {
+ spin_unlock(&dentry->d_lock);
+ return false;
+ }
+
+ if (!inode || likely(spin_trylock(&inode->i_lock)))
+ return true;
+
+ // Too bad - we need to drop ->d_lock and take locks in correct order.
+ // To avoid breaking RCU read-side critical area when we drop ->d_lock,
+ // take an explicit rcu_read_lock() while we are switching locks.
+ rcu_read_lock();
+ do {
+ spin_unlock(&dentry->d_lock);
+ spin_lock(&inode->i_lock);
+ spin_lock(&dentry->d_lock);
+ // make sure we'd locked the right inode - ->d_inode might've
+ // changed while we were not holding ->d_lock
+ if (likely(inode == dentry->d_inode))
+ break;
+ spin_unlock(&inode->i_lock);
+ inode = dentry->d_inode;
+ } while (inode);
+ rcu_read_unlock();
+ if (likely(!dentry->d_lockref.count))
+ return true;
+ if (inode)
+ spin_unlock(&inode->i_lock);
+ spin_unlock(&dentry->d_lock);
+ return false;
+}
+
+/**
+ * dentry_kill - evict a dentry
+ * @dentry: dentry to be evicted
+ *
+ * All dentry evictions are done by this function. The reference we are
+ * passed does not contribute to the refcount; the caller had either
+ * already decremented the refcount or it had never held one in the
+ * first place. @dentry->d_lock is held by the caller and dropped
+ * by dentry_kill(@dentry).
+ *
+ * We are guaranteed that nobody had called dentry_free(@dentry)
+ * prior to the beginning of RCU read-side critical area we are in.
+ *
+ * Caller must not access @dentry after the call.
+ *
+ * If eviction of @dentry drops the last reference to its parent,
+ * the reference to parent is returned to caller. In that case
+ * it is guaranteed to satisfy the requirements for dentry_kill()
+ * argument - its ->d_lock is held and we are guaranteed that nobody
+ * had passed it to dentry_free() prior to acquisition of its ->d_lock.
+ * Otherwise %NULL is returned.
+ *
+ * If @dentry is idle and remains such after we assemble the full
+ * locking environment for eviction (see lock_for_kill() for details)
+ * we mark it doomed (->d_lockref.count < 0) and proceed to detaching
+ * it from any filesystem objects. Otherwise we drop ->d_lock and
+ * return %NULL.
+ *
+ * Once @dentry is detached from the filesystem objects, we complete
+ * detaching it from dentry tree. The parent, if any, gets locked
+ * and its refcount is decremented; dentry is carefully removed from
+ * the tree (see dentry_unlist() for details) and marked killed
+ * (%DCACHE_DENTRY_KILLED set in ->d_flags). At that point it's just
+ * an inert chunk of memory, accessible only via RCU references
+ * and possibly via a shrink list. If it is not on any shrink lists,
+ * we call dentry_free(), which schedules actual freeing of memory.
+ * Othewise freeing is left to the owner of the shrink list in question.
+ */
+static struct dentry *dentry_kill(struct dentry *dentry)
{
struct dentry *parent = NULL;
bool can_free = true;
+ if (unlikely(!lock_for_kill(dentry)))
+ return NULL;
+
/*
* The dentry is now unrecoverably dead to the world.
*/
@@ -743,43 +851,6 @@ static struct dentry *__dentry_kill(struct dentry *dentry)
}
/*
- * Lock a dentry for feeding it to __dentry_kill().
- * Called under rcu_read_lock() and dentry->d_lock; the former
- * guarantees that nothing we access will be freed under us.
- * Note that dentry is *not* protected from concurrent dentry_kill(),
- * d_delete(), etc.
- *
- * Return false if dentry is busy. Otherwise, return true and have
- * that dentry's inode locked.
- */
-
-static bool lock_for_kill(struct dentry *dentry)
-{
- struct inode *inode = dentry->d_inode;
-
- if (unlikely(dentry->d_lockref.count))
- return false;
-
- if (!inode || likely(spin_trylock(&inode->i_lock)))
- return true;
-
- do {
- spin_unlock(&dentry->d_lock);
- spin_lock(&inode->i_lock);
- spin_lock(&dentry->d_lock);
- if (likely(inode == dentry->d_inode))
- break;
- spin_unlock(&inode->i_lock);
- inode = dentry->d_inode;
- } while (inode);
- if (likely(!dentry->d_lockref.count))
- return true;
- if (inode)
- spin_unlock(&inode->i_lock);
- return false;
-}
-
-/*
* Decide if dentry is worth retaining. Usually this is called with dentry
* locked; if not locked, we are more limited and might not be able to tell
* without a lock. False in this case means "punt to locked path and recheck".
@@ -854,17 +925,17 @@ EXPORT_SYMBOL(d_mark_dontcache);
* If unsuccessful, we return false, having already taken the dentry lock.
* In that case refcount is guaranteed to be zero and we have already
* decided that it's not worth keeping around.
- *
- * The caller needs to hold the RCU read lock, so that the dentry is
- * guaranteed to stay around even if the refcount goes down to zero!
*/
static inline bool fast_dput(struct dentry *dentry)
{
int ret;
/*
- * try to decrement the lockref optimistically.
+ * Try to decrement the lockref optimistically.
+ * RCU read lock held so that dentry is guaranteed to stay around
+ * even if the refcount goes down to zero.
*/
+ rcu_read_lock();
ret = lockref_put_return(&dentry->d_lockref);
/*
@@ -874,6 +945,7 @@ static inline bool fast_dput(struct dentry *dentry)
*/
if (unlikely(ret < 0)) {
spin_lock(&dentry->d_lock);
+ rcu_read_unlock();
if (WARN_ON_ONCE(dentry->d_lockref.count <= 0)) {
spin_unlock(&dentry->d_lock);
return true;
@@ -885,8 +957,10 @@ static inline bool fast_dput(struct dentry *dentry)
/*
* If we weren't the last ref, we're done.
*/
- if (ret)
+ if (ret) {
+ rcu_read_unlock();
return true;
+ }
/*
* Can we decide that decrement of refcount is all we needed without
@@ -894,8 +968,10 @@ static inline bool fast_dput(struct dentry *dentry)
* dentry looks like it ought to be retained and there's nothing else
* to do.
*/
- if (retain_dentry(dentry, false))
+ if (retain_dentry(dentry, false)) {
+ rcu_read_unlock();
return true;
+ }
/*
* Either not worth retaining or we can't tell without the lock.
@@ -903,6 +979,7 @@ static inline bool fast_dput(struct dentry *dentry)
* but we'll need to re-check the situation after getting the lock.
*/
spin_lock(&dentry->d_lock);
+ rcu_read_unlock();
/*
* Did somebody else grab a reference to it in the meantime, and
@@ -920,21 +997,13 @@ static inline bool fast_dput(struct dentry *dentry)
static void finish_dput(struct dentry *dentry)
__releases(dentry->d_lock)
- __releases(RCU)
{
- while (lock_for_kill(dentry)) {
- rcu_read_unlock();
- dentry = __dentry_kill(dentry);
- if (!dentry)
- return;
+ while ((dentry = dentry_kill(dentry)) != NULL) {
if (retain_dentry(dentry, true)) {
spin_unlock(&dentry->d_lock);
return;
}
- rcu_read_lock();
}
- rcu_read_unlock();
- spin_unlock(&dentry->d_lock);
}
/*
@@ -968,11 +1037,8 @@ void dput(struct dentry *dentry)
if (!dentry)
return;
might_sleep();
- rcu_read_lock();
- if (likely(fast_dput(dentry))) {
- rcu_read_unlock();
+ if (likely(fast_dput(dentry)))
return;
- }
finish_dput(dentry);
}
EXPORT_SYMBOL(dput);
@@ -983,30 +1049,46 @@ void d_make_discardable(struct dentry *dentry)
WARN_ON(!(dentry->d_flags & DCACHE_PERSISTENT));
dentry->d_flags &= ~DCACHE_PERSISTENT;
dentry->d_lockref.count--;
- rcu_read_lock();
finish_dput(dentry);
}
EXPORT_SYMBOL(d_make_discardable);
-static void to_shrink_list(struct dentry *dentry, struct list_head *list)
+/**
+ * __move_to_shrink_list - try to place a dentry into a shrink list
+ * @dentry: dentry to try putting into shrink list
+ * @list: the list to put @dentry into.
+ * Returns: true @dentry had been placed into @list, false otherwise
+ *
+ * If @dentry is idle and not already include into a shrink list, move
+ * it into @list and return %true; otherwise do nothing and return %false.
+ *
+ * Caller must be holding @dentry->d_lock. There must have been no calls of
+ * dentry_free(@dentry) prior to the beginning of the RCU read-side critical
+ * area in which __move_to_shrink_list(@dentry, @list) is called.
+ *
+ * @list should be thread-private and eventually emptied by passing it to
+ * shrink_dentry_list().
+ */
+
+bool __move_to_shrink_list(struct dentry *dentry, struct list_head *list)
__must_hold(&dentry->d_lock)
{
- if (!(dentry->d_flags & DCACHE_SHRINK_LIST)) {
+ if (likely(!dentry->d_lockref.count &&
+ !(dentry->d_flags & DCACHE_SHRINK_LIST))) {
if (dentry->d_flags & DCACHE_LRU_LIST)
d_lru_del(dentry);
d_shrink_add(dentry, list);
+ return true;
}
+ return false;
}
+EXPORT_SYMBOL(__move_to_shrink_list);
void dput_to_list(struct dentry *dentry, struct list_head *list)
{
- rcu_read_lock();
- if (likely(fast_dput(dentry))) {
- rcu_read_unlock();
+ if (likely(fast_dput(dentry)))
return;
- }
- rcu_read_unlock();
- to_shrink_list(dentry, list);
+ __move_to_shrink_list(dentry, list);
spin_unlock(&dentry->d_lock);
}
@@ -1052,7 +1134,10 @@ struct dentry *dget_parent(struct dentry *dentry)
}
EXPORT_SYMBOL(dget_parent);
-static struct dentry * __d_find_any_alias(struct inode *inode)
+/*
+ * inode is a directory, inode->i_lock is held by the caller
+ */
+static struct dentry * __d_find_dir_alias(struct inode *inode)
{
struct dentry *alias;
@@ -1063,6 +1148,18 @@ static struct dentry * __d_find_any_alias(struct inode *inode)
return alias;
}
+static struct dentry * __d_find_any_alias(struct inode *inode)
+{
+ struct dentry *alias;
+
+ if (hlist_empty(&inode->i_dentry))
+ return NULL;
+ for_each_alias(alias, inode)
+ if (dget_alias_ilocked(alias))
+ return alias;
+ return NULL;
+}
+
/**
* d_find_any_alias - find any alias for a given inode
* @inode: inode to find an alias for
@@ -1086,7 +1183,7 @@ static struct dentry *__d_find_alias(struct inode *inode)
struct dentry *alias;
if (S_ISDIR(inode->i_mode))
- return __d_find_any_alias(inode);
+ return __d_find_dir_alias(inode);
for_each_alias(alias, inode) {
spin_lock(&alias->d_lock);
@@ -1152,25 +1249,6 @@ struct dentry *d_find_alias_rcu(struct inode *inode)
return de;
}
-/**
- * d_dispose_if_unused - move unreferenced dentries to shrink list
- * @dentry: dentry in question
- * @dispose: head of shrink list
- *
- * If dentry has no external references, move it to shrink list.
- *
- * NOTE!!! The caller is responsible for preventing eviction of the dentry by
- * holding dentry->d_inode->i_lock or equivalent.
- */
-void d_dispose_if_unused(struct dentry *dentry, struct list_head *dispose)
-{
- spin_lock(&dentry->d_lock);
- if (!dentry->d_lockref.count)
- to_shrink_list(dentry, dispose);
- spin_unlock(&dentry->d_lock);
-}
-EXPORT_SYMBOL(d_dispose_if_unused);
-
/*
* Try to kill dentries associated with this inode.
* WARNING: you must own a reference to inode.
@@ -1181,8 +1259,12 @@ void d_prune_aliases(struct inode *inode)
struct dentry *dentry;
spin_lock(&inode->i_lock);
- for_each_alias(dentry, inode)
- d_dispose_if_unused(dentry, &dispose);
+ for_each_alias(dentry, inode) {
+ spin_lock(&dentry->d_lock);
+ if (likely(!(dentry->d_flags & DCACHE_NORCU)))
+ __move_to_shrink_list(dentry, &dispose);
+ spin_unlock(&dentry->d_lock);
+ }
spin_unlock(&inode->i_lock);
shrink_dentry_list(&dispose);
}
@@ -1190,14 +1272,8 @@ EXPORT_SYMBOL(d_prune_aliases);
static inline void shrink_kill(struct dentry *victim)
{
- do {
- rcu_read_unlock();
- victim = __dentry_kill(victim);
- rcu_read_lock();
- } while (victim && lock_for_kill(victim));
- rcu_read_unlock();
- if (victim)
- spin_unlock(&victim->d_lock);
+ while ((victim = dentry_kill(victim)) != NULL)
+ ;
}
void shrink_dentry_list(struct list_head *list)
@@ -1207,18 +1283,12 @@ void shrink_dentry_list(struct list_head *list)
dentry = list_entry(list->prev, struct dentry, d_lru);
spin_lock(&dentry->d_lock);
- rcu_read_lock();
- if (!lock_for_kill(dentry)) {
- bool can_free;
- rcu_read_unlock();
- d_shrink_del(dentry);
- can_free = dentry->d_flags & DCACHE_DENTRY_KILLED;
+ d_shrink_del(dentry);
+ if (unlikely(dentry->d_flags & DCACHE_DENTRY_KILLED)) {
spin_unlock(&dentry->d_lock);
- if (can_free)
- dentry_free(dentry);
+ dentry_free(dentry);
continue;
}
- d_shrink_del(dentry);
shrink_kill(dentry);
}
}
@@ -1379,6 +1449,8 @@ static void d_walk(struct dentry *parent, void *data,
read_seqbegin_or_lock(&rename_lock, &seq);
this_parent = parent;
spin_lock(&this_parent->d_lock);
+ if (unlikely(this_parent->d_flags & DCACHE_DENTRY_CURSOR))
+ goto out_unlock;
ret = enter(data, this_parent);
switch (ret) {
@@ -1427,14 +1499,15 @@ static void d_walk(struct dentry *parent, void *data,
/*
* All done at this level ... ascend and resume the search.
*/
- rcu_read_lock();
ascend:
if (this_parent != parent) {
dentry = this_parent;
this_parent = dentry->d_parent;
+ rcu_read_lock();
spin_unlock(&dentry->d_lock);
spin_lock(&this_parent->d_lock);
+ rcu_read_unlock();
/* might go back up the wrong parent if we have had a rename. */
if (need_seqretry(&rename_lock, seq))
@@ -1442,7 +1515,6 @@ static void d_walk(struct dentry *parent, void *data,
/* go into the first sibling still alive */
hlist_for_each_entry_continue(dentry, d_sib) {
if (likely(!(dentry->d_flags & DCACHE_DENTRY_KILLED))) {
- rcu_read_unlock();
goto resume;
}
}
@@ -1450,7 +1522,6 @@ static void d_walk(struct dentry *parent, void *data,
}
if (need_seqretry(&rename_lock, seq))
goto rename_retry;
- rcu_read_unlock();
out_unlock:
spin_unlock(&this_parent->d_lock);
@@ -1459,7 +1530,6 @@ static void d_walk(struct dentry *parent, void *data,
rename_retry:
spin_unlock(&this_parent->d_lock);
- rcu_read_unlock();
BUG_ON(seq & 1);
if (!retry)
return;
@@ -1574,12 +1644,8 @@ static enum d_walk_ret select_collect(void *_data, struct dentry *dentry)
if (data->start == dentry)
goto out;
- if (dentry->d_flags & DCACHE_SHRINK_LIST) {
- data->found++;
- } else if (!dentry->d_lockref.count) {
- to_shrink_list(dentry, &data->dispose);
- data->found++;
- } else if (dentry->d_lockref.count < 0) {
+ if (dentry->d_lockref.count <= 0) {
+ __move_to_shrink_list(dentry, &data->dispose);
data->found++;
}
/*
@@ -1610,17 +1676,21 @@ static enum d_walk_ret select_collect2(void *_data, struct dentry *dentry)
if (data->start == dentry)
goto out;
- if (!dentry->d_lockref.count) {
- if (dentry->d_flags & DCACHE_SHRINK_LIST) {
+ if (dentry->d_lockref.count <= 0) {
+ if (!__move_to_shrink_list(dentry, &data->dispose)) {
+ /*
+ * We need an enter RCU read-side critical area that
+ * would extend past the return from d_walk() and
+ * we are in the scope of ->d_lock that will terminate
+ * before that, so we use rcu_read_lock() to bridge
+ * over to the scope of ->d_lock in d_walk() caller.
+ * The scope of rcu_read_lock() spans from here to
+ * paired rcu_read_unlock() in shrink_dcache_tree().
+ */
rcu_read_lock();
data->victim = dentry;
return D_WALK_QUIT;
}
- to_shrink_list(dentry, &data->dispose);
- } else if (dentry->d_lockref.count < 0) {
- rcu_read_lock();
- data->victim = dentry;
- return D_WALK_QUIT;
}
/*
* We can return to the caller if we have found some (this
@@ -1643,7 +1713,9 @@ static enum d_walk_ret select_collect2(void *_data, struct dentry *dentry)
static void shrink_dcache_tree(struct dentry *parent, bool for_umount)
{
for (;;) {
- struct select_data data = {.start = parent};
+ struct completion_list wait;
+ bool need_wait = false;
+ struct select_data data = { .start = parent };
INIT_LIST_HEAD(&data.dispose);
d_walk(parent, &data,
@@ -1661,30 +1733,32 @@ static void shrink_dcache_tree(struct dentry *parent, bool for_umount)
d_walk(parent, &data, select_collect2);
if (data.victim) {
struct dentry *v = data.victim;
-
+ /*
+ * select_collect2() has picked a dentry that was
+ * either dying or on a shrink list and arranged
+ * for it to be returned to us. We are still in
+ * the RCU read-side critical area started there
+ * (rcu_read_lock() scope opened in select_collect2()),
+ * so dentry couldn't have been freed yet, but its
+ * state might've changed since we dropped ->d_lock
+ * on the way out. Switch over to ->d_lock scope
+ * and recheck the dentry state.
+ */
spin_lock(&v->d_lock);
- if (v->d_lockref.count < 0 &&
- !(v->d_flags & DCACHE_DENTRY_KILLED)) {
- struct completion_list wait;
- // It's busy dying; have it notify us once
- // it becomes invisible to d_walk().
- d_add_waiter(v, &wait);
+ rcu_read_unlock();
+
+ if (unlikely(v->d_lockref.count < 0)) {
+ // It's doomed; if it isn't dead yet, notify us
+ // once it becomes invisible to d_walk().
+ need_wait = d_add_waiter(v, &wait);
spin_unlock(&v->d_lock);
- rcu_read_unlock();
- if (!list_empty(&data.dispose))
- shrink_dentry_list(&data.dispose);
- wait_for_completion(&wait.completion);
- continue;
- }
- if (!lock_for_kill(v)) {
- spin_unlock(&v->d_lock);
- rcu_read_unlock();
} else {
shrink_kill(v);
}
}
- if (!list_empty(&data.dispose))
- shrink_dentry_list(&data.dispose);
+ shrink_dentry_list(&data.dispose);
+ if (unlikely(need_wait))
+ wait_for_completion(&wait.completion);
}
}
@@ -1737,9 +1811,30 @@ void shrink_dcache_for_umount(struct super_block *sb)
sb->s_root = NULL;
do_one_tree(dentry);
- while (!hlist_bl_empty(&sb->s_roots)) {
- dentry = dget(hlist_bl_entry(hlist_bl_first(&sb->s_roots), struct dentry, d_hash));
- do_one_tree(dentry);
+ for (;;) {
+ spin_lock(&sb->s_roots_lock);
+ dentry = hlist_entry_safe(sb->s_roots.first,
+ struct dentry, d_sib);
+ if (!dentry) {
+ spin_unlock(&sb->s_roots_lock);
+ break;
+ }
+ rcu_read_lock();
+ spin_unlock(&sb->s_roots_lock);
+ spin_lock(&dentry->d_lock);
+ rcu_read_unlock();
+ if (unlikely(dentry->d_lockref.count < 0)) {
+ struct completion_list wait;
+ bool need_wait = d_add_waiter(dentry, &wait);
+
+ spin_unlock(&dentry->d_lock);
+ if (need_wait)
+ wait_for_completion(&wait.completion);
+ } else {
+ dget_dlock(dentry);
+ spin_unlock(&dentry->d_lock);
+ do_one_tree(dentry);
+ }
}
}
@@ -1820,10 +1915,10 @@ static struct dentry *__d_alloc(struct super_block *sb, const struct qstr *name)
name = &slash_name;
dname = dentry->d_shortname.string;
} else if (name->len > DNAME_INLINE_LEN-1) {
- size_t size = offsetof(struct external_name, name[1]);
- struct external_name *p = kmalloc(size + name->len,
- GFP_KERNEL_ACCOUNT |
- __GFP_RECLAIMABLE);
+ struct external_name *p;
+
+ p = kmalloc_flex(*p, name, name->len + 1,
+ GFP_KERNEL_ACCOUNT | __GFP_RECLAIMABLE);
if (!p) {
kmem_cache_free(dentry_cache, dentry);
return NULL;
@@ -1909,7 +2004,7 @@ struct dentry *d_alloc_cursor(struct dentry * parent)
{
struct dentry *dentry = d_alloc_anon(parent->d_sb);
if (dentry) {
- dentry->d_flags |= DCACHE_DENTRY_CURSOR;
+ dentry->d_flags |= DCACHE_DENTRY_CURSOR | DCACHE_NORCU;
dentry->d_parent = dget(parent);
}
return dentry;
@@ -2100,6 +2195,10 @@ void d_instantiate_new(struct dentry *entry, struct inode *inode)
__d_instantiate(entry, inode);
spin_unlock(&entry->d_lock);
WARN_ON(!(inode_state_read(inode) & I_NEW));
+ /*
+ * Paired with igrab_from_hash()
+ */
+ smp_wmb();
inode_state_clear(inode, I_NEW | I_CREATING);
inode_wake_up_bit(inode, __I_NEW);
spin_unlock(&inode->i_lock);
@@ -2156,9 +2255,9 @@ static struct dentry *__d_obtain_alias(struct inode *inode, bool disconnected)
__d_set_inode_and_type(new, inode, add_flags);
hlist_add_head(&new->d_alias, &inode->i_dentry);
if (!disconnected) {
- hlist_bl_lock(&sb->s_roots);
- hlist_bl_add_head(&new->d_hash, &sb->s_roots);
- hlist_bl_unlock(&sb->s_roots);
+ spin_lock(&sb->s_roots_lock);
+ hlist_add_head(&new->d_sib, &sb->s_roots);
+ spin_unlock(&sb->s_roots_lock);
}
spin_unlock(&new->d_lock);
spin_unlock(&inode->i_lock);
@@ -2250,8 +2349,7 @@ struct dentry *d_add_ci(struct dentry *dentry, struct inode *inode,
return found;
}
if (d_in_lookup(dentry)) {
- found = d_alloc_parallel(dentry->d_parent, name,
- dentry->d_wait);
+ found = d_alloc_parallel(dentry->d_parent, name);
if (IS_ERR(found) || !d_in_lookup(found)) {
iput(inode);
return found;
@@ -2638,32 +2736,24 @@ static inline unsigned start_dir_add(struct inode *dir)
}
}
-static inline void end_dir_add(struct inode *dir, unsigned int n,
- wait_queue_head_t *d_wait)
+static inline void end_dir_add(struct inode *dir, unsigned int n)
{
smp_store_release(&dir->i_dir_seq, n + 2);
preempt_enable_nested();
- if (wq_has_sleeper(d_wait))
- wake_up_all(d_wait);
}
static void d_wait_lookup(struct dentry *dentry)
{
- if (d_in_lookup(dentry)) {
- DECLARE_WAITQUEUE(wait, current);
- add_wait_queue(dentry->d_wait, &wait);
- do {
- set_current_state(TASK_UNINTERRUPTIBLE);
- spin_unlock(&dentry->d_lock);
- schedule();
- spin_lock(&dentry->d_lock);
- } while (d_in_lookup(dentry));
+ if (likely(d_in_lookup(dentry))) {
+ dentry->d_flags |= DCACHE_LOOKUP_WAITERS;
+ wait_var_event_spinlock(&dentry->d_flags,
+ !d_in_lookup(dentry),
+ &dentry->d_lock);
}
}
struct dentry *d_alloc_parallel(struct dentry *parent,
- const struct qstr *name,
- wait_queue_head_t *wq)
+ const struct qstr *name)
{
unsigned int hash = name->hash;
struct hlist_bl_head *b = in_lookup_hash(parent, hash);
@@ -2684,38 +2774,33 @@ struct dentry *d_alloc_parallel(struct dentry *parent,
spin_unlock(&parent->d_lock);
retry:
- rcu_read_lock();
seq = smp_load_acquire(&parent->d_inode->i_dir_seq);
r_seq = read_seqbegin(&rename_lock);
+ rcu_read_lock();
dentry = __d_lookup_rcu(parent, name, &d_seq);
if (unlikely(dentry)) {
if (!lockref_get_not_dead(&dentry->d_lockref)) {
rcu_read_unlock();
goto retry;
}
+ rcu_read_unlock();
if (read_seqcount_retry(&dentry->d_seq, d_seq)) {
- rcu_read_unlock();
dput(dentry);
goto retry;
}
- rcu_read_unlock();
dput(new);
return dentry;
}
- if (unlikely(read_seqretry(&rename_lock, r_seq))) {
- rcu_read_unlock();
+ rcu_read_unlock();
+ if (unlikely(read_seqretry(&rename_lock, r_seq)))
goto retry;
- }
- if (unlikely(seq & 1)) {
- rcu_read_unlock();
+ if (unlikely(seq & 1))
goto retry;
- }
hlist_bl_lock(b);
if (unlikely(READ_ONCE(parent->d_inode->i_dir_seq) != seq)) {
hlist_bl_unlock(b);
- rcu_read_unlock();
goto retry;
}
/*
@@ -2732,19 +2817,20 @@ struct dentry *d_alloc_parallel(struct dentry *parent,
continue;
if (!d_same_name(dentry, parent, name))
continue;
+ rcu_read_lock();
hlist_bl_unlock(b);
+ spin_lock(&dentry->d_lock);
+ rcu_read_unlock();
/* now we can try to grab a reference */
- if (!lockref_get_not_dead(&dentry->d_lockref)) {
- rcu_read_unlock();
+ if (unlikely(dentry->d_lockref.count < 0)) {
+ spin_unlock(&dentry->d_lock);
goto retry;
}
-
- rcu_read_unlock();
/*
* somebody is likely to be still doing lookup for it;
- * wait for them to finish
+ * pin it and wait for them to finish
*/
- spin_lock(&dentry->d_lock);
+ dget_dlock(dentry);
d_wait_lookup(dentry);
/*
* it's not in-lookup anymore; in principle we should repeat
@@ -2765,8 +2851,6 @@ struct dentry *d_alloc_parallel(struct dentry *parent,
dput(new);
return dentry;
}
- rcu_read_unlock();
- new->d_wait = wq;
hlist_bl_add_head(&new->d_in_lookup_hash, b);
hlist_bl_unlock(b);
return new;
@@ -2778,13 +2862,26 @@ struct dentry *d_alloc_parallel(struct dentry *parent,
EXPORT_SYMBOL(d_alloc_parallel);
/*
- * - Unhash the dentry
- * - Retrieve and clear the waitqueue head in dentry
- * - Return the waitqueue head
+ * Move dentry from in-lookup state to busy-negative one.
+ *
+ * From now on d_in_lookup(dentry) will return false and dentry is gone from
+ * in-lookup hash.
+ *
+ * Anyone who had been waiting on it in d_alloc_parallel() is free to
+ * proceed after that. Note that waking such waiters up is left to
+ * the callers; PREEMPT_RT kernels can't have that wakeup done while
+ * in write-side critical area for ->i_dir_seq, so it's done by calling
+ * __d_wake_in_lookup_waiters() once it's safe to do so.
+ *
+ * Both __d_lookup_unhash() and __d_wake_in_lookup_waiters() should
+ * be called within the same ->d_lock scope. PAR_LOOKUP is cleared
+ * here, while LOOKUP_WAITERS (set by somebody finding dentry in
+ * the in-lookup hash and setting down to wait) is checked and cleared
+ * in __d_wake_in_lookup_waiters(). Both are gone by the end of
+ * ->d_lock scope.
*/
-static wait_queue_head_t *__d_lookup_unhash(struct dentry *dentry)
+static void __d_lookup_unhash(struct dentry *dentry)
{
- wait_queue_head_t *d_wait;
struct hlist_bl_head *b;
lockdep_assert_held(&dentry->d_lock);
@@ -2793,18 +2890,23 @@ static wait_queue_head_t *__d_lookup_unhash(struct dentry *dentry)
hlist_bl_lock(b);
dentry->d_flags &= ~DCACHE_PAR_LOOKUP;
__hlist_bl_del(&dentry->d_in_lookup_hash);
- d_wait = dentry->d_wait;
- dentry->d_wait = NULL;
hlist_bl_unlock(b);
dentry->waiters = NULL;
- INIT_LIST_HEAD(&dentry->d_lru);
- return d_wait;
+}
+
+static inline void __d_wake_in_lookup_waiters(struct dentry *dentry)
+{
+ if (dentry->d_flags & DCACHE_LOOKUP_WAITERS) {
+ wake_up_var_locked(&dentry->d_flags, &dentry->d_lock);
+ dentry->d_flags &= ~DCACHE_LOOKUP_WAITERS;
+ }
}
void __d_lookup_unhash_wake(struct dentry *dentry)
{
spin_lock(&dentry->d_lock);
- wake_up_all(__d_lookup_unhash(dentry));
+ __d_lookup_unhash(dentry);
+ __d_wake_in_lookup_waiters(dentry);
spin_unlock(&dentry->d_lock);
}
EXPORT_SYMBOL(__d_lookup_unhash_wake);
@@ -2814,14 +2916,13 @@ EXPORT_SYMBOL(__d_lookup_unhash_wake);
static inline void __d_add(struct dentry *dentry, struct inode *inode,
const struct dentry_operations *ops)
{
- wait_queue_head_t *d_wait;
struct inode *dir = NULL;
unsigned n;
spin_lock(&dentry->d_lock);
if (unlikely(d_in_lookup(dentry))) {
dir = dentry->d_parent->d_inode;
n = start_dir_add(dir);
- d_wait = __d_lookup_unhash(dentry);
+ __d_lookup_unhash(dentry);
}
if (unlikely(ops))
d_set_d_op(dentry, ops);
@@ -2834,8 +2935,10 @@ static inline void __d_add(struct dentry *dentry, struct inode *inode,
fsnotify_update_flags(dentry);
}
__d_rehash(dentry);
- if (dir)
- end_dir_add(dir, n, d_wait);
+ if (dir) {
+ end_dir_add(dir, n);
+ __d_wake_in_lookup_waiters(dentry);
+ }
spin_unlock(&dentry->d_lock);
if (inode)
spin_unlock(&inode->i_lock);
@@ -2948,7 +3051,6 @@ static void __d_move(struct dentry *dentry, struct dentry *target,
bool exchange)
{
struct dentry *old_parent, *p;
- wait_queue_head_t *d_wait;
struct inode *dir = NULL;
unsigned n;
@@ -2979,7 +3081,7 @@ static void __d_move(struct dentry *dentry, struct dentry *target,
if (unlikely(d_in_lookup(target))) {
dir = target->d_parent->d_inode;
n = start_dir_add(dir);
- d_wait = __d_lookup_unhash(target);
+ __d_lookup_unhash(target);
}
write_seqcount_begin(&dentry->d_seq);
@@ -3018,9 +3120,10 @@ static void __d_move(struct dentry *dentry, struct dentry *target,
write_seqcount_end(&target->d_seq);
write_seqcount_end(&dentry->d_seq);
- if (dir)
- end_dir_add(dir, n, d_wait);
-
+ if (dir) {
+ end_dir_add(dir, n);
+ __d_wake_in_lookup_waiters(target);
+ }
if (dentry->d_parent != old_parent)
spin_unlock(&dentry->d_parent->d_lock);
if (dentry != old_parent)
@@ -3141,7 +3244,7 @@ struct dentry *d_splice_alias_ops(struct inode *inode, struct dentry *dentry,
security_d_instantiate(dentry, inode);
spin_lock(&inode->i_lock);
if (S_ISDIR(inode->i_mode)) {
- struct dentry *new = __d_find_any_alias(inode);
+ struct dentry *new = __d_find_dir_alias(inode);
if (unlikely(new)) {
/* The reference to new ensures it remains an alias */
spin_unlock(&inode->i_lock);
@@ -3166,6 +3269,12 @@ struct dentry *d_splice_alias_ops(struct inode *inode, struct dentry *dentry,
}
dput(old_parent);
} else {
+ if (unlikely(!hlist_unhashed(&new->d_sib))) {
+ // secondary root getting spliced
+ spin_lock(&new->d_lock);
+ unlink_secondary_root(new);
+ spin_unlock(&new->d_lock);
+ }
__d_move(new, dentry, false);
write_sequnlock(&rename_lock);
}
diff --git a/fs/ecryptfs/inode.c b/fs/ecryptfs/inode.c
index 546c1fe..7aaf191 100644
--- a/fs/ecryptfs/inode.c
+++ b/fs/ecryptfs/inode.c
@@ -350,11 +350,9 @@ static struct dentry *ecryptfs_lookup_interpose(struct dentry *dentry,
*/
lower_inode = READ_ONCE(lower_dentry->d_inode);
- if (!lower_inode) {
- /* We want to add because we couldn't find in lower */
- d_add(dentry, NULL);
- return NULL;
- }
+ if (!lower_inode) /* We want to add because we couldn't find in lower */
+ return d_splice_alias(NULL, dentry);
+
inode = __ecryptfs_get_inode(lower_inode, dentry->d_sb);
if (IS_ERR(inode)) {
printk(KERN_ERR "%s: Error interposing; rc = [%ld]\n",
diff --git a/fs/eventpoll.c b/fs/eventpoll.c
index a3090b4..0e65c74 100644
--- a/fs/eventpoll.c
+++ b/fs/eventpoll.c
@@ -38,48 +38,174 @@
#include <linux/compat.h>
#include <linux/rculist.h>
#include <linux/capability.h>
+#include <linux/seqlock.h>
#include <net/busy_poll.h>
/*
- * LOCKING:
- * There are three level of locking required by epoll :
+ * fs/eventpoll.c - Efficient event polling ("epoll") kernel implementation.
*
- * 1) epnested_mutex (mutex)
- * 2) ep->mtx (mutex)
- * 3) ep->lock (spinlock)
*
- * The acquire order is the one listed above, from 1 to 3.
- * We need a spinlock (ep->lock) because we manipulate objects
- * from inside the poll callback, that might be triggered from
- * a wake_up() that in turn might be called from IRQ context.
- * So we can't sleep inside the poll callback and hence we need
- * a spinlock. During the event transfer loop (from kernel to
- * user space) we could end up sleeping due a copy_to_user(), so
- * we need a lock that will allow us to sleep. This lock is a
- * mutex (ep->mtx). It is acquired during the event transfer loop,
- * during epoll_ctl(EPOLL_CTL_DEL) and during eventpoll_release_file().
- * The epnested_mutex is acquired when inserting an epoll fd onto another
- * epoll fd. We do this so that we walk the epoll tree and ensure that this
- * insertion does not create a cycle of epoll file descriptors, which
- * could lead to deadlock. We need a global mutex to prevent two
- * simultaneous inserts (A into B and B into A) from racing and
- * constructing a cycle without either insert observing that it is
- * going to.
- * It is necessary to acquire multiple "ep->mtx"es at once in the
- * case when one epoll fd is added to another. In this case, we
- * always acquire the locks in the order of nesting (i.e. after
- * epoll_ctl(e1, EPOLL_CTL_ADD, e2), e1->mtx will always be acquired
- * before e2->mtx). Since we disallow cycles of epoll file
- * descriptors, this ensures that the mutexes are well-ordered. In
- * order to communicate this nesting to lockdep, when walking a tree
- * of epoll file descriptors, we use the current recursion depth as
- * the lockdep subkey.
- * It is possible to drop the "ep->mtx" and to use the global
- * mutex "epnested_mutex" (together with "ep->lock") to have it working,
- * but having "ep->mtx" will make the interface more scalable.
- * Events that require holding "epnested_mutex" are very rare, while for
- * normal operations the epoll private "ep->mtx" will guarantee
- * a better scalability.
+ * Overview
+ * --------
+ *
+ * Each epoll_create(2) returns an anonymous [eventpoll] file whose
+ * ->private_data is a struct eventpoll. Each EPOLL_CTL_ADD installs
+ * a struct epitem linking one (watched file, fd) pair back to that
+ * eventpoll via the watched file's f_op->poll() wait queue(s). When
+ * the watched file signals readiness, ep_poll_callback() fires and
+ * marks the epitem ready. epoll_wait(2) drains the ready list under
+ * ep->mtx, re-queueing items in level-triggered mode.
+ *
+ * epoll instances can watch other epoll instances up to EP_MAX_NESTS
+ * deep; cycles are forbidden and detected at EPOLL_CTL_ADD time.
+ *
+ *
+ * Locking
+ * -------
+ *
+ * Three levels, acquired from outer to inner:
+ *
+ * epnested_mutex (global; rare; taken only for EPOLL_CTL_ADD
+ * loop / path checks)
+ * > ep->mtx (per-eventpoll; sleepable; serializes most ops)
+ * > ep->lock (per-eventpoll; IRQ-safe spinlock)
+ *
+ * file->f_lock (per-file; NOT IRQ-safe; guards f_ep hlist ops;
+ * nested inside ep->mtx, outside ep->lock)
+ *
+ * Rationale:
+ * - ep->lock is a spinlock because ep_poll_callback() is called from
+ * wake_up() which may run in hard-IRQ context. All ep->lock
+ * critical sections use spin_lock_irqsave().
+ * - ep->mtx is a sleepable mutex because the event delivery loop
+ * calls copy_to_user(), and ep_insert() may sleep in
+ * kmem_cache_alloc() and f_op->poll().
+ * - epnested_mutex is global because cycle detection needs a global
+ * view of the epoll topology; a per-object scheme would let two
+ * concurrent inserts (A into B, B into A) construct a cycle
+ * without either observer seeing it.
+ * - Per-ep ep->mtx is preferred for scalability elsewhere. Events
+ * that require epnested_mutex are rare.
+ *
+ * When EPOLL_CTL_ADD nests one eventpoll inside another we acquire
+ * ep->mtx on both: outer first, target second. Since cycles are
+ * forbidden the set of live ep->mtx holds is always a strict chain,
+ * communicated to lockdep via mutex_lock_nested() subclasses derived
+ * from the current recursion depth.
+ *
+ *
+ * Field protection
+ * ----------------
+ *
+ * struct eventpoll:
+ * mtx - self
+ * rbr - ep->mtx
+ * ovflist, rdllist - ep->lock (IRQ-safe)
+ * wq - ep->lock for queue mutation
+ * poll_wait - internal waitqueue spinlock
+ * refs - file->f_lock for adds; ep->mtx for removes;
+ * RCU for readers (hlist_del_rcu + kfree_rcu(ep))
+ * ws - ep->mtx
+ * gen, loop_check_depth - epnested_mutex
+ * file, user - immutable after setup
+ * refcount - atomic (refcount_t)
+ * napi_* - READ_ONCE / WRITE_ONCE
+ *
+ * struct epitem:
+ * rbn / rcu union - rbn: ep->mtx (while epi is linked in ep->rbr).
+ * rcu: written only by kfree_rcu(epi) on the free
+ * path; otherwise untouched by epoll code.
+ * rdllink, next - ep->lock
+ * ffd, ep - immutable after ep_insert()
+ * pwqlist - ep->mtx for writes; POLLFREE clears pwq->whead
+ * via smp_store_release(), see below
+ * fllink - file->f_lock for mutation; hlist_del_rcu +
+ * kfree_rcu(epi) for safe RCU readers
+ * ws - RCU (rcu_assign_pointer /
+ * rcu_dereference_check(mtx))
+ * event - ep->mtx for writes; lockless read in
+ * ep_poll_callback pairs with smp_mb() in
+ * ep_modify()
+ *
+ *
+ * Ready-list state machine
+ * ------------------------
+ *
+ * Readiness is tracked in two lists under ep->lock:
+ *
+ * rdllist - doubly-linked FIFO; the "current" ready list.
+ * ovflist - singly-linked LIFO; used during a scan to catch
+ * events that arrive while rdllist is being iterated
+ * without ep->lock.
+ *
+ * Encoded in ep->ovflist:
+ * EP_UNACTIVE_PTR - no scan active; callback appends to rdllist.
+ * NULL - scan active, no spill yet.
+ * pointer to epi - scan active with spilled items (LIFO).
+ *
+ * Encoded in epi->ovflist_next:
+ * EP_UNACTIVE_PTR - epi is not on ovflist.
+ * otherwise - next epi on ovflist (NULL at tail).
+ *
+ * ep_start_scan() flips "not scanning" to "scanning" and splices
+ * rdllist into a caller-local scan_batch. ep_done_scan() drains ovflist
+ * back to rdllist (list_add head-insert reverses LIFO to FIFO),
+ * flips back to "not scanning", and re-splices any items the caller
+ * left in scan_batch (e.g., level-triggered re-queues).
+ *
+ *
+ * Removal paths
+ * -------------
+ *
+ * Three paths dispose of epitems and/or eventpolls:
+ *
+ * A. ep_remove() - EPOLL_CTL_DEL and ep_insert()
+ * rollback. Caller holds ep->mtx.
+ * B. ep_clear_and_put() - close of the epoll fd itself
+ * (ep_eventpoll_release).
+ * C. eventpoll_release_file() - close of a watched file, invoked
+ * from __fput().
+ *
+ * Coordination:
+ * A and C exclude each other via the watched file's refcount.
+ * A pins the file with epi_fget() before touching file->f_ep or
+ * file->f_lock; if the pin fails, __fput() is in flight and C
+ * will clean this epi up. See the epi_fget() block comment.
+ * A and B both hold ep->mtx serially. B walks the rbtree with
+ * rb_next() captured before ep_remove() erases the current node.
+ * B and C both take ep->mtx; the loser sees fewer entries or an
+ * empty file->f_ep.
+ *
+ * Within every path the internal order is strict:
+ * ep_unregister_pollwait() - drain pwqlist; synchronizes with any
+ * in-flight ep_poll_callback via the
+ * watched wait-queue head's lock.
+ * ep_remove_file() - hlist_del_rcu of epi->fllink and,
+ * if last watcher, clear file->f_ep,
+ * under file->f_lock.
+ * ep_remove_epi() - rb_erase, rdllist unlink (ep->lock),
+ * wakeup_source_unregister,
+ * kfree_rcu(epi).
+ *
+ * kfree_rcu(epi) defers the free past RCU readers in
+ * reverse_path_check_proc(); kfree_rcu(ep) defers past readers in
+ * ep_get_upwards_depth_proc().
+ *
+ *
+ * POLLFREE handshake
+ * ------------------
+ *
+ * When a subsystem tears down a wait-queue head that an epitem is
+ * registered on (binder, signalfd, ...), it wakes the callback with
+ * POLLFREE and must RCU-defer the head's free. The store/load pair:
+ *
+ * ep_poll_callback() POLLFREE branch:
+ * smp_store_release(&pwq->whead, NULL)
+ *
+ * ep_remove_wait_queue():
+ * smp_load_acquire(&pwq->whead)
+ *
+ * See those sites for the full argument.
*/
/* Epoll private bits inside the event mask */
@@ -99,11 +225,6 @@
#define EP_ITEM_COST (sizeof(struct epitem) + sizeof(struct eppoll_entry))
-struct epoll_filefd {
- struct file *file;
- int fd;
-} __packed;
-
/* Wait structure used by the poll hooks */
struct eppoll_entry {
/* List header used to link this structure to the "struct epitem" */
@@ -136,17 +257,19 @@ struct epitem {
struct rcu_head rcu;
};
- /* List header used to link this structure to the eventpoll ready list */
+ /* Link on the owning eventpoll's ready list (ep->rdllist). */
struct list_head rdllink;
/*
- * Works together "struct eventpoll"->ovflist in keeping the
- * single linked chain of items.
+ * Link on the owning eventpoll's scan-overflow list (ep->ovflist),
+ * EP_UNACTIVE_PTR when not linked. See epi_on_ovflist() /
+ * epi_clear_ovflist() and the "Ready-list state machine" section
+ * in the top-of-file banner.
*/
- struct epitem *next;
+ struct epitem *ovflist_next;
/* The file descriptor information this item refers to */
- struct epoll_filefd ffd;
+ struct epoll_key ffd;
/* List containing poll wait queues */
struct eppoll_entry *pwqlist;
@@ -190,6 +313,9 @@ struct eventpoll {
/* Lock which protects rdllist and ovflist */
spinlock_t lock;
+ /* Protect switching between rdllist and ovflist */
+ seqcount_spinlock_t seq;
+
/* RB tree root used to store monitored fd structs */
struct rb_root_cached rbr;
@@ -247,13 +373,81 @@ struct ep_pqueue {
/* Maximum number of epoll watched descriptors, per user */
static long max_user_watches __read_mostly;
-/* Used for cycles detection */
+/*
+ * Cycle and path-length checks at EPOLL_CTL_ADD
+ * ---------------------------------------------
+ *
+ * When EPOLL_CTL_ADD creates a link that either targets an eventpoll
+ * file or extends an existing chain of eventpolls, two checks run:
+ *
+ * 1. no cycle is being formed -- ep_loop_check() walks downward
+ * from the candidate target, and ep_get_upwards_depth_proc()
+ * walks upward from the outer ep, both bounded by EP_MAX_NESTS.
+ * 2. no file accumulates more than path_limits[depth] wakeup paths
+ * of a given length -- reverse_path_check().
+ *
+ * Both need a global view of the epoll topology and must be atomic
+ * with the insertion, so the check is serialized by epnested_mutex
+ * and carries its scratch state on a stack-allocated struct
+ * ep_ctl_ctx scoped to one do_epoll_ctl() call. Non-nested inserts
+ * skip this machinery entirely and take only ep->mtx.
+ *
+ * epnested_mutex Serializes the whole check.
+ * loop_check_gen Global monotonic stamp, bumped at the start of
+ * a check and again at the end. ep->gen caches
+ * the value under which ep was last visited by
+ * ep_loop_check_proc() or
+ * ep_get_upwards_depth_proc(); the post-check
+ * bump ensures those cached stamps can no longer
+ * equal loop_check_gen, so the
+ * "ep->gen == loop_check_gen" trigger in
+ * ep_ctl_lock() only fires while another check
+ * is in flight.
+ *
+ * struct ep_ctl_ctx carries the rest (inserting_into, tfile_check_list,
+ * path_count[]) through the walk; see its declaration below.
+ *
+ * Commits fdcfce93073d ("eventpoll: Fix integer overflow in
+ * ep_loop_check_proc()") and f2e467a48287 ("eventpoll: Fix
+ * semi-unbounded recursion") hardened the walk; any refactor must
+ * preserve both bail-outs.
+ */
static DEFINE_MUTEX(epnested_mutex);
-
static u64 loop_check_gen = 0;
-/* Used to check for epoll file descriptor inclusion loops */
-static struct eventpoll *inserting_into;
+#define PATH_ARR_SIZE 5
+
+/*
+ * Per-do_epoll_ctl() scratch for the loop / path checks. Allocated on
+ * the caller's stack; populated by ep_ctl_lock() and the downward
+ * walk; consumed by reverse_path_check(); released by ep_ctl_unlock().
+ * Only valid while the caller holds epnested_mutex.
+ */
+struct ep_ctl_ctx {
+ /*
+ * Outer eventpoll for one ep_loop_check(); if the downward walk
+ * reaches it the insert would form a cycle.
+ */
+ struct eventpoll *inserting_into;
+
+ /*
+ * Singly-linked list of epitems_head objects collected during
+ * ep_loop_check_proc(), then walked by reverse_path_check().
+ * Terminated by EP_UNACTIVE_PTR, not NULL: epitems_head->next
+ * doubles as a membership flag (a NULL ->next means "not on this
+ * list", see ep_remove_file()), so the list uses a non-NULL
+ * sentinel to keep the tail head distinguishable from an unlisted
+ * one.
+ */
+ struct epitems_head *tfile_check_list;
+
+ /*
+ * Per-depth wakeup-path tally used by reverse_path_check_proc();
+ * reinitialized to zero at the start of each reverse_path_check()
+ * iteration.
+ */
+ int path_count[PATH_ARR_SIZE];
+};
/* Slab cache used to allocate "struct epitem" */
static struct kmem_cache *epi_cache __ro_after_init;
@@ -262,14 +456,15 @@ static struct kmem_cache *epi_cache __ro_after_init;
static struct kmem_cache *pwq_cache __ro_after_init;
/*
- * List of files with newly added links, where we may need to limit the number
- * of emanating paths. Protected by the epnested_mutex.
+ * Wrapper anchor for file->f_ep when the watched file is not itself an
+ * eventpoll; for the epoll-watches-epoll case, file->f_ep points at
+ * &watched_ep->refs directly. The ->next field threads
+ * ctx->tfile_check_list during one EPOLL_CTL_ADD path check.
*/
struct epitems_head {
struct hlist_head epitems;
struct epitems_head *next;
};
-static struct epitems_head *tfile_check_list = EP_UNACTIVE_PTR;
static struct kmem_cache *ephead_cache __ro_after_init;
@@ -279,14 +474,14 @@ static inline void free_ephead(struct epitems_head *head)
kmem_cache_free(ephead_cache, head);
}
-static void list_file(struct file *file)
+static void list_file(struct file *file, struct ep_ctl_ctx *ctx)
{
struct epitems_head *head;
head = container_of(file->f_ep, struct epitems_head, epitems);
if (!head->next) {
- head->next = tfile_check_list;
- tfile_check_list = head;
+ head->next = ctx->tfile_check_list;
+ ctx->tfile_check_list = head;
}
}
@@ -334,29 +529,20 @@ static void __init epoll_sysctls_init(void)
static const struct file_operations eventpoll_fops;
-static inline int is_file_epoll(struct file *f)
+bool is_file_epoll(struct file *f)
{
return f->f_op == &eventpoll_fops;
}
-/* Setup the structure that is used as key for the RB tree */
-static inline void ep_set_ffd(struct epoll_filefd *ffd,
- struct file *file, int fd)
-{
- ffd->file = file;
- ffd->fd = fd;
-}
-
/* Compare RB tree keys */
-static inline int ep_cmp_ffd(struct epoll_filefd *p1,
- struct epoll_filefd *p2)
+static inline int ep_cmp_ffd(struct epoll_key *p1, struct epoll_key *p2)
{
return (p1->file > p2->file ? +1:
(p1->file < p2->file ? -1 : p1->fd - p2->fd));
}
-/* Tells us if the item is currently linked */
-static inline int ep_is_linked(struct epitem *epi)
+/* True iff @epi is on its owning ep's ready list. */
+static inline bool ep_is_linked(struct epitem *epi)
{
return !list_empty(&epi->rdllink);
}
@@ -372,18 +558,50 @@ static inline struct epitem *ep_item_from_wait(wait_queue_entry_t *p)
return container_of(p, struct eppoll_entry, wait)->base;
}
-/**
- * ep_events_available - Checks if ready events might be available.
- *
- * @ep: Pointer to the eventpoll context.
- *
- * Return: a value different than %zero if ready events are available,
- * or %zero otherwise.
+/*
+ * Ready-list / ovflist state (see "Ready-list state machine" in the
+ * top-of-file banner for the full state machine). EP_UNACTIVE_PTR is
+ * the sentinel; these wrappers name each transition and each test so
+ * call sites do not need to know the sentinel's value.
*/
-static inline int ep_events_available(struct eventpoll *ep)
+
+/* True iff @ep is between ep_enter_scan() and ep_exit_scan(). */
+static inline bool ep_is_scanning(struct eventpoll *ep)
{
- return !list_empty_careful(&ep->rdllist) ||
- READ_ONCE(ep->ovflist) != EP_UNACTIVE_PTR;
+ return READ_ONCE(ep->ovflist) != EP_UNACTIVE_PTR;
+}
+
+/* Called by ep_start_scan(): divert ep_poll_callback() to ovflist. */
+static inline void ep_enter_scan(struct eventpoll *ep)
+{
+ WRITE_ONCE(ep->ovflist, NULL);
+}
+
+/* Called by ep_done_scan(): redirect ep_poll_callback() back to rdllist. */
+static inline void ep_exit_scan(struct eventpoll *ep)
+{
+ WRITE_ONCE(ep->ovflist, EP_UNACTIVE_PTR);
+}
+
+/* True iff @epi is currently linked on its ep's ovflist. */
+static inline bool epi_on_ovflist(const struct epitem *epi)
+{
+ return epi->ovflist_next != EP_UNACTIVE_PTR;
+}
+
+/* Mark @epi as not on any ovflist (init and post-drain). */
+static inline void epi_clear_ovflist(struct epitem *epi)
+{
+ epi->ovflist_next = EP_UNACTIVE_PTR;
+}
+
+/* True iff @ep has ready events that epoll_wait() might harvest. */
+static inline bool ep_events_available(struct eventpoll *ep)
+{
+ unsigned int seq = read_seqcount_begin(&ep->seq);
+
+ return !list_empty_careful(&ep->rdllist) || ep_is_scanning(ep) ||
+ read_seqcount_retry(&ep->seq, seq);
}
#ifdef CONFIG_NET_RX_BUSY_POLL
@@ -659,10 +877,15 @@ static void ep_remove_wait_queue(struct eppoll_entry *pwq)
rcu_read_lock();
/*
- * If it is cleared by POLLFREE, it should be rcu-safe.
- * If we read NULL we need a barrier paired with
- * smp_store_release() in ep_poll_callback(), otherwise
- * we rely on whead->lock.
+ * POLLFREE handshake, acquire side; see "POLLFREE handshake"
+ * at the top of this file.
+ *
+ * A NULL load is paired with the smp_store_release(&whead, NULL)
+ * in ep_poll_callback()'s POLLFREE branch: the teardown is
+ * complete and we must not touch whead again. On a non-NULL load
+ * rcu_read_lock() keeps the waitqueue memory alive (POLLFREE
+ * firers RCU-defer the free) and whead->lock inside
+ * remove_wait_queue() serializes us against the store side.
*/
whead = smp_load_acquire(&pwq->whead);
if (whead)
@@ -723,7 +946,7 @@ static inline void ep_pm_stay_awake_rcu(struct epitem *epi)
* ep->mutex needs to be held because we could be hit by
* eventpoll_release_file() and epoll_ctl().
*/
-static void ep_start_scan(struct eventpoll *ep, struct list_head *txlist)
+static void ep_start_scan(struct eventpoll *ep, struct list_head *scan_batch)
{
/*
* Steal the ready list, and re-init the original one to the
@@ -735,13 +958,17 @@ static void ep_start_scan(struct eventpoll *ep, struct list_head *txlist)
*/
lockdep_assert_irqs_enabled();
spin_lock_irq(&ep->lock);
- list_splice_init(&ep->rdllist, txlist);
- WRITE_ONCE(ep->ovflist, NULL);
+ write_seqcount_begin(&ep->seq);
+
+ list_splice_init(&ep->rdllist, scan_batch);
+ ep_enter_scan(ep);
+
+ write_seqcount_end(&ep->seq);
spin_unlock_irq(&ep->lock);
}
static void ep_done_scan(struct eventpoll *ep,
- struct list_head *txlist)
+ struct list_head *scan_batch)
{
struct epitem *epi, *nepi;
@@ -751,34 +978,35 @@ static void ep_done_scan(struct eventpoll *ep,
* other events might have been queued by the poll callback.
* We re-insert them inside the main ready-list here.
*/
- for (nepi = READ_ONCE(ep->ovflist); (epi = nepi) != NULL;
- nepi = epi->next, epi->next = EP_UNACTIVE_PTR) {
+ for (nepi = READ_ONCE(ep->ovflist); (epi = nepi) != NULL; ) {
+ nepi = epi->ovflist_next;
+ epi_clear_ovflist(epi);
/*
- * We need to check if the item is already in the list.
- * During the "sproc" callback execution time, items are
- * queued into ->ovflist but the "txlist" might already
- * contain them, and the list_splice() below takes care of them.
+ * Skip items that the caller already returned via @scan_batch
+ * -- the list_splice() below takes care of those.
*/
if (!ep_is_linked(epi)) {
/*
- * ->ovflist is LIFO, so we have to reverse it in order
- * to keep in FIFO.
+ * ovflist is LIFO; list_add() head-insert here
+ * reverses the iteration order into FIFO.
*/
list_add(&epi->rdllink, &ep->rdllist);
ep_pm_stay_awake(epi);
}
}
- /*
- * We need to set back ep->ovflist to EP_UNACTIVE_PTR, so that after
- * releasing the lock, events will be queued in the normal way inside
- * ep->rdllist.
- */
- WRITE_ONCE(ep->ovflist, EP_UNACTIVE_PTR);
+
+ write_seqcount_begin(&ep->seq);
+
+ /* Back out of scan mode; callbacks target ep->rdllist again. */
+ ep_exit_scan(ep);
/*
- * Quickly re-inject items left on "txlist".
+ * Quickly re-inject items left on "scan_batch".
*/
- list_splice(txlist, &ep->rdllist);
+ list_splice(scan_batch, &ep->rdllist);
+
+ write_seqcount_end(&ep->seq);
+
__pm_relax(ep->ws);
if (!list_empty(&ep->rdllist)) {
@@ -795,9 +1023,10 @@ static void ep_get(struct eventpoll *ep)
}
/*
- * Returns true if the event poll can be disposed
+ * Drop a reference to @ep; returns true iff it was the last, in which
+ * case the caller is responsible for ep_free().
*/
-static bool ep_refcount_dec_and_test(struct eventpoll *ep)
+static bool ep_put(struct eventpoll *ep)
{
if (!refcount_dec_and_test(&ep->refcount))
return false;
@@ -817,22 +1046,23 @@ static void ep_free(struct eventpoll *ep)
}
/*
- * The ffd.file pointer may be in the process of being torn down due to
- * being closed, but we may not have finished eventpoll_release() yet.
+ * Pin @epi->ffd.file for operations that require both safe dereference
+ * and exclusion from __fput().
*
- * Normally, even with the atomic_long_inc_not_zero, the file may have
- * been free'd and then gotten re-allocated to something else (since
- * files are not RCU-delayed, they are SLAB_TYPESAFE_BY_RCU).
+ * struct file uses SLAB_TYPESAFE_BY_RCU, so a freed slot can be
+ * reassigned at any time. The bare load of epi->ffd.file is safe here
+ * because the caller holds ep->mtx and eventpoll_release_file() blocks
+ * on that mutex while tearing down the epi, so the backing file
+ * allocation cannot be freed and reused under us. An rcu_read_lock()
+ * is therefore unnecessary for the load.
*
- * But for epoll, users hold the ep->mtx mutex, and as such any file in
- * the process of being free'd will block in eventpoll_release_file()
- * and thus the underlying file allocation will not be free'd, and the
- * file re-use cannot happen.
- *
- * For the same reason we can avoid a rcu_read_lock() around the
- * operation - 'ffd.file' cannot go away even if the refcount has
- * reached zero (but we must still not call out to ->poll() functions
- * etc).
+ * A successful file_ref_get() additionally blocks __fput() from
+ * starting on this file: once the refcount has reached zero it cannot
+ * come back. ep_remove() relies on that to touch file->f_lock and
+ * file->f_ep without racing eventpoll_release_file() (see commit
+ * a6dc643c6931). A NULL return means __fput() is already in flight;
+ * the caller must bail without touching the file, and
+ * eventpoll_release_file() will clean the epi up from its side.
*/
static struct file *epi_fget(const struct epitem *epi)
{
@@ -858,7 +1088,13 @@ static void ep_remove_file(struct eventpoll *ep, struct epitem *epi,
spin_lock(&file->f_lock);
head = file->f_ep;
if (hlist_is_singular_node(&epi->fllink, head)) {
- /* See eventpoll_release() for details. */
+ /*
+ * Last watcher: publish NULL so the eventpoll_release()
+ * fastpath in include/linux/eventpoll.h can skip the slow
+ * path on a future __fput(). Safe because every f_ep writer
+ * either holds a pin on @file via epi_fget() or is __fput()
+ * itself -- see the comment in eventpoll_release().
+ */
WRITE_ONCE(file->f_ep, NULL);
if (!is_file_epoll(file)) {
struct epitems_head *v;
@@ -919,47 +1155,82 @@ static void ep_remove(struct eventpoll *ep, struct epitem *epi)
ep_remove_file(ep, epi, file);
ep_remove_epi(ep, epi);
- WARN_ON_ONCE(ep_refcount_dec_and_test(ep));
+ WARN_ON_ONCE(ep_put(ep));
}
-static void ep_clear_and_put(struct eventpoll *ep)
+/*
+ * Pass 1 of ep_clear_and_put(): drain every epi's pwqlist.
+ * ep_unregister_pollwait() takes each watched wait-queue head's lock,
+ * which synchronizes with any in-flight ep_poll_callback(); after
+ * this returns no callback can still be about to dereference an epi
+ * on this ep. Must strictly precede ep_drain_tree() -- fusing the
+ * two walks would let a callback queued on epi_i still fire after
+ * epi_{i+k} had already been freed.
+ */
+static void ep_drain_pollwaits(struct eventpoll *ep)
{
- struct rb_node *rbp, *next;
+ struct rb_node *rbp;
struct epitem *epi;
- /* We need to release all tasks waiting for these file */
- if (waitqueue_active(&ep->poll_wait))
- ep_poll_safewake(ep, NULL, 0);
+ lockdep_assert_held(&ep->mtx);
- mutex_lock(&ep->mtx);
-
- /*
- * Walks through the whole tree by unregistering poll callbacks.
- */
for (rbp = rb_first_cached(&ep->rbr); rbp; rbp = rb_next(rbp)) {
epi = rb_entry(rbp, struct epitem, rbn);
ep_unregister_pollwait(ep, epi);
cond_resched();
}
+}
- /*
- * Walks through the whole tree and try to free each "struct epitem".
- * Note that ep_remove() will not remove the epitem in case of a
- * racing eventpoll_release_file(); the latter will do the removal.
- * At this point we are sure no poll callbacks will be lingering around.
- * Since we still own a reference to the eventpoll struct, the loop can't
- * dispose it.
- */
+/*
+ * Pass 2 of ep_clear_and_put(): ep_remove() every epi. The per-epi
+ * pwqlist is already empty (ep_drain_pollwaits ran), but the rest of
+ * ep_remove() still runs: epi_fget() pin, f_ep clear under f_lock,
+ * rbtree erase, rdllist unlink, kfree_rcu(epi). rb_next() is captured
+ * before each erase so the iteration is stable.
+ *
+ * A concurrent eventpoll_release_file() (removal path C) on a watched
+ * file serializes with us via ep->mtx; ep_remove() transparently
+ * hands off any epi whose file is in __fput() by bailing when
+ * epi_fget() returns NULL, and path C will clean that epi up.
+ */
+static void ep_drain_tree(struct eventpoll *ep)
+{
+ struct rb_node *rbp, *next;
+ struct epitem *epi;
+
+ lockdep_assert_held(&ep->mtx);
+
for (rbp = rb_first_cached(&ep->rbr); rbp; rbp = next) {
next = rb_next(rbp);
epi = rb_entry(rbp, struct epitem, rbn);
ep_remove(ep, epi);
cond_resched();
}
+}
+/*
+ * Removal path B (see "Removal paths" in the top-of-file banner):
+ * close of the epoll fd itself, reached via ep_eventpoll_release().
+ *
+ * Two passes under ep->mtx: first ep_drain_pollwaits() quiesces
+ * in-flight callbacks, then ep_drain_tree() frees the epis. The
+ * ep->refcount is kept > 0 across the walk by the ep file's own
+ * share, which we drop below; ep_free() runs iff we were the last
+ * holder after the tree drained.
+ */
+static void ep_clear_and_put(struct eventpoll *ep)
+{
+ /* Release any threads blocked in poll-on-ep. */
+ if (waitqueue_active(&ep->poll_wait))
+ ep_poll_safewake(ep, NULL, 0);
+
+ mutex_lock(&ep->mtx);
+ ep_drain_pollwaits(ep);
+ ep_drain_tree(ep);
mutex_unlock(&ep->mtx);
- if (ep_refcount_dec_and_test(ep))
+
+ if (ep_put(ep))
ep_free(ep);
}
@@ -999,7 +1270,7 @@ static __poll_t ep_item_poll(const struct epitem *epi, poll_table *pt, int depth
static __poll_t __ep_eventpoll_poll(struct file *file, poll_table *wait, int depth)
{
struct eventpoll *ep = file->private_data;
- LIST_HEAD(txlist);
+ LIST_HEAD(scan_batch);
struct epitem *epi, *tmp;
poll_table pt;
__poll_t res = 0;
@@ -1014,8 +1285,8 @@ static __poll_t __ep_eventpoll_poll(struct file *file, poll_table *wait, int dep
* the ready list.
*/
mutex_lock_nested(&ep->mtx, depth);
- ep_start_scan(ep, &txlist);
- list_for_each_entry_safe(epi, tmp, &txlist, rdllink) {
+ ep_start_scan(ep, &scan_batch);
+ list_for_each_entry_safe(epi, tmp, &scan_batch, rdllink) {
if (ep_item_poll(epi, &pt, depth + 1)) {
res = EPOLLIN | EPOLLRDNORM;
break;
@@ -1029,7 +1300,7 @@ static __poll_t __ep_eventpoll_poll(struct file *file, poll_table *wait, int dep
list_del_init(&epi->rdllink);
}
}
- ep_done_scan(ep, &txlist);
+ ep_done_scan(ep, &scan_batch);
mutex_unlock(&ep->mtx);
return res;
}
@@ -1138,7 +1409,7 @@ void eventpoll_release_file(struct file *file)
mutex_unlock(&ep->mtx);
- if (ep_refcount_dec_and_test(ep))
+ if (ep_put(ep))
ep_free(ep);
goto again;
}
@@ -1155,11 +1426,12 @@ static int ep_alloc(struct eventpoll **pep)
mutex_init(&ep->mtx);
spin_lock_init(&ep->lock);
+ seqcount_spinlock_init(&ep->seq, &ep->lock);
init_waitqueue_head(&ep->wq);
init_waitqueue_head(&ep->poll_wait);
INIT_LIST_HEAD(&ep->rdllist);
ep->rbr = RB_ROOT_CACHED;
- ep->ovflist = EP_UNACTIVE_PTR;
+ ep->ovflist = EP_UNACTIVE_PTR; /* not scanning */
ep->user = get_current_user();
refcount_set(&ep->refcount, 1);
@@ -1173,17 +1445,15 @@ static int ep_alloc(struct eventpoll **pep)
* are protected by the "mtx" mutex, and ep_find() must be called with
* "mtx" held.
*/
-static struct epitem *ep_find(struct eventpoll *ep, struct file *file, int fd)
+static struct epitem *ep_find(struct eventpoll *ep, struct epoll_key *tf)
{
int kcmp;
struct rb_node *rbp;
struct epitem *epi, *epir = NULL;
- struct epoll_filefd ffd;
- ep_set_ffd(&ffd, file, fd);
for (rbp = ep->rbr.rb_root.rb_node; rbp; ) {
epi = rb_entry(rbp, struct epitem, rbn);
- kcmp = ep_cmp_ffd(&ffd, &epi->ffd);
+ kcmp = ep_cmp_ffd(tf, &epi->ffd);
if (kcmp > 0)
rbp = rbp->rb_right;
else if (kcmp < 0)
@@ -1197,50 +1467,6 @@ static struct epitem *ep_find(struct eventpoll *ep, struct file *file, int fd)
return epir;
}
-#ifdef CONFIG_KCMP
-static struct epitem *ep_find_tfd(struct eventpoll *ep, int tfd, unsigned long toff)
-{
- struct rb_node *rbp;
- struct epitem *epi;
-
- for (rbp = rb_first_cached(&ep->rbr); rbp; rbp = rb_next(rbp)) {
- epi = rb_entry(rbp, struct epitem, rbn);
- if (epi->ffd.fd == tfd) {
- if (toff == 0)
- return epi;
- else
- toff--;
- }
- cond_resched();
- }
-
- return NULL;
-}
-
-struct file *get_epoll_tfile_raw_ptr(struct file *file, int tfd,
- unsigned long toff)
-{
- struct file *file_raw;
- struct eventpoll *ep;
- struct epitem *epi;
-
- if (!is_file_epoll(file))
- return ERR_PTR(-EINVAL);
-
- ep = file->private_data;
-
- mutex_lock(&ep->mtx);
- epi = ep_find_tfd(ep, tfd, toff);
- if (epi)
- file_raw = epi->ffd.file;
- else
- file_raw = ERR_PTR(-ENOENT);
- mutex_unlock(&ep->mtx);
-
- return file_raw;
-}
-#endif /* CONFIG_KCMP */
-
/*
* This is the callback that is passed to the wait queue wakeup
* mechanism. It is called by the stored file descriptors when they
@@ -1283,9 +1509,9 @@ static int ep_poll_callback(wait_queue_entry_t *wait, unsigned mode, int sync, v
* semantics). All the events that happen during that period of time are
* chained in ep->ovflist and requeued later on.
*/
- if (READ_ONCE(ep->ovflist) != EP_UNACTIVE_PTR) {
- if (epi->next == EP_UNACTIVE_PTR) {
- epi->next = READ_ONCE(ep->ovflist);
+ if (ep_is_scanning(ep)) {
+ if (!epi_on_ovflist(epi)) {
+ epi->ovflist_next = READ_ONCE(ep->ovflist);
WRITE_ONCE(ep->ovflist, epi);
ep_pm_stay_awake_rcu(epi);
}
@@ -1336,17 +1562,24 @@ static int ep_poll_callback(wait_queue_entry_t *wait, unsigned mode, int sync, v
if (pollflags & POLLFREE) {
/*
- * If we race with ep_remove_wait_queue() it can miss
- * ->whead = NULL and do another remove_wait_queue() after
- * us, so we can't use __remove_wait_queue().
+ * POLLFREE handshake, release side; see "POLLFREE handshake"
+ * at the top of this file.
+ *
+ * Unlink our wait entry with list_del_init rather than
+ * __remove_wait_queue: a concurrent ep_remove_wait_queue()
+ * that already loaded a non-NULL whead may still call
+ * remove_wait_queue() after us, and list_del_init() tolerates
+ * the second delete.
+ *
+ * smp_store_release(&whead, NULL) publishes the teardown to
+ * ep_remove_wait_queue()'s smp_load_acquire(). Before this
+ * store, a racing ep_clear_and_put() / ep_remove() reaches
+ * ep_remove_wait_queue() which sees whead != NULL and takes
+ * whead->lock -- the same lock held by our caller, so it
+ * serializes behind us. Once whead is zeroed, nothing else
+ * protects ep / epi / wait.
*/
list_del_init(&wait->entry);
- /*
- * ->whead != NULL protects us from the race with
- * ep_clear_and_put() or ep_remove(), ep_remove_wait_queue()
- * takes whead->lock held by the caller. Once we nullify it,
- * nothing protects ep/epi or even wait.
- */
smp_store_release(&ep_pwq_from_wait(wait)->whead, NULL);
}
@@ -1407,41 +1640,40 @@ static void ep_rbtree_insert(struct eventpoll *ep, struct epitem *epi)
-#define PATH_ARR_SIZE 5
/*
- * These are the number paths of length 1 to 5, that we are allowing to emanate
- * from a single file of interest. For example, we allow 1000 paths of length
- * 1, to emanate from each file of interest. This essentially represents the
- * potential wakeup paths, which need to be limited in order to avoid massive
- * uncontrolled wakeup storms. The common use case should be a single ep which
- * is connected to n file sources. In this case each file source has 1 path
- * of length 1. Thus, the numbers below should be more than sufficient. These
- * path limits are enforced during an EPOLL_CTL_ADD operation, since a modify
- * and delete can't add additional paths. Protected by the epnested_mutex.
+ * Upper bound on wakeup paths emanating from any one watched file,
+ * indexed by path depth (1..PATH_ARR_SIZE). For example, we allow
+ * 1000 paths of length 1 from each watched file. These caps limit
+ * the wakeup amplification that can be built from epoll-watches-
+ * epoll topologies without rejecting reasonable usage.
+ *
+ * Enforced at EPOLL_CTL_ADD; CTL_MOD and CTL_DEL cannot add paths.
+ * The running tallies live in ctx->path_count[] and are protected by
+ * epnested_mutex.
*/
static const int path_limits[PATH_ARR_SIZE] = { 1000, 500, 100, 50, 10 };
-static int path_count[PATH_ARR_SIZE];
-static int path_count_inc(int nests)
+static int path_count_inc(struct ep_ctl_ctx *ctx, int nests)
{
/* Allow an arbitrary number of depth 1 paths */
if (nests == 0)
return 0;
- if (++path_count[nests] > path_limits[nests])
+ if (++ctx->path_count[nests] > path_limits[nests])
return -1;
return 0;
}
-static void path_count_init(void)
+static void path_count_init(struct ep_ctl_ctx *ctx)
{
int i;
for (i = 0; i < PATH_ARR_SIZE; i++)
- path_count[i] = 0;
+ ctx->path_count[i] = 0;
}
-static int reverse_path_check_proc(struct hlist_head *refs, int depth)
+static int reverse_path_check_proc(struct ep_ctl_ctx *ctx,
+ struct hlist_head *refs, int depth)
{
int error = 0;
struct epitem *epi;
@@ -1453,9 +1685,9 @@ static int reverse_path_check_proc(struct hlist_head *refs, int depth)
hlist_for_each_entry_rcu(epi, refs, fllink) {
struct hlist_head *refs = &epi->ep->refs;
if (hlist_empty(refs))
- error = path_count_inc(depth);
+ error = path_count_inc(ctx, depth);
else
- error = reverse_path_check_proc(refs, depth + 1);
+ error = reverse_path_check_proc(ctx, refs, depth + 1);
if (error != 0)
break;
}
@@ -1463,24 +1695,24 @@ static int reverse_path_check_proc(struct hlist_head *refs, int depth)
}
/**
- * reverse_path_check - The tfile_check_list is list of epitem_head, which have
- * links that are proposed to be newly added. We need to
- * make sure that those added links don't add too many
- * paths such that we will spend all our time waking up
- * eventpoll objects.
+ * reverse_path_check - ctx->tfile_check_list is a list of epitems_head
+ * anchoring files with newly proposed links; make
+ * sure those links don't push any path-length bucket
+ * over its limit in path_limits[].
+ * @ctx: Per-do_epoll_ctl() scratch for the loop / path checks.
*
* Return: %zero if the proposed links don't create too many paths,
* %-1 otherwise.
*/
-static int reverse_path_check(void)
+static int reverse_path_check(struct ep_ctl_ctx *ctx)
{
struct epitems_head *p;
- for (p = tfile_check_list; p != EP_UNACTIVE_PTR; p = p->next) {
+ for (p = ctx->tfile_check_list; p != EP_UNACTIVE_PTR; p = p->next) {
int error;
- path_count_init();
+ path_count_init(ctx);
rcu_read_lock();
- error = reverse_path_check_proc(&p->epitems, 0);
+ error = reverse_path_check_proc(ctx, &p->epitems, 0);
rcu_read_unlock();
if (error)
return error;
@@ -1526,7 +1758,7 @@ static noinline void ep_destroy_wakeup_source(struct epitem *epi)
wakeup_source_unregister(ws);
}
-static int attach_epitem(struct file *file, struct epitem *epi)
+static int ep_attach_file(struct file *file, struct epitem *epi)
{
struct epitems_head *to_free = NULL;
struct hlist_head *head = NULL;
@@ -1561,10 +1793,93 @@ static int attach_epitem(struct file *file, struct epitem *epi)
}
/*
+ * Charge the user's epoll_watches quota, allocate a fresh epitem for
+ * @tf, and initialize its fields. The returned item is not yet linked
+ * into any data structure; the caller must install it via
+ * ep_register_epitem() (which takes over on success) or kmem_cache_free()
+ * it and decrement epoll_watches on its own.
+ *
+ * Returns ERR_PTR(-ENOSPC) if the quota is exceeded, ERR_PTR(-ENOMEM)
+ * if the slab allocation fails.
+ */
+static struct epitem *ep_alloc_epitem(struct eventpoll *ep,
+ const struct epoll_event *event,
+ struct epoll_key *tf)
+{
+ struct epitem *epi;
+
+ if (unlikely(percpu_counter_compare(&ep->user->epoll_watches,
+ max_user_watches) >= 0))
+ return ERR_PTR(-ENOSPC);
+ percpu_counter_inc(&ep->user->epoll_watches);
+
+ epi = kmem_cache_zalloc(epi_cache, GFP_KERNEL);
+ if (unlikely(!epi)) {
+ percpu_counter_dec(&ep->user->epoll_watches);
+ return ERR_PTR(-ENOMEM);
+ }
+
+ INIT_LIST_HEAD(&epi->rdllink);
+ epi->ep = ep;
+ epi->ffd = *tf;
+ epi->event = *event;
+ epi_clear_ovflist(epi);
+
+ return epi;
+}
+
+/*
+ * Install @epi into its target file's f_ep hlist and into @ep's rbtree,
+ * taking one additional reference on @ep for the lifetime of the item.
+ *
+ * If @tep is non-NULL, the target file is itself an eventpoll; we hold
+ * tep->mtx at subclass 1 across the attach + rbtree insert to serialize
+ * with the target side. RB tree ops are protected by @ep->mtx, which
+ * the caller already holds.
+ *
+ * On failure the epi is freed and the epoll_watches counter decremented,
+ * matching ep_alloc_epitem()'s allocation. After this returns
+ * successfully, ep_insert()'s later error paths use ep_remove() for
+ * unwind; that cannot drop @ep's refcount to zero because the ep file
+ * itself still holds the original reference.
+ */
+static int ep_register_epitem(struct ep_ctl_ctx *ctx, struct eventpoll *ep,
+ struct epitem *epi, struct eventpoll *tep,
+ int full_check)
+{
+ struct file *tfile = epi->ffd.file;
+ int error;
+
+ if (tep)
+ mutex_lock_nested(&tep->mtx, 1);
+
+ error = ep_attach_file(tfile, epi);
+ if (unlikely(error)) {
+ if (tep)
+ mutex_unlock(&tep->mtx);
+ kmem_cache_free(epi_cache, epi);
+ percpu_counter_dec(&ep->user->epoll_watches);
+ return error;
+ }
+
+ if (full_check && !tep)
+ list_file(tfile, ctx);
+
+ ep_rbtree_insert(ep, epi);
+
+ if (tep)
+ mutex_unlock(&tep->mtx);
+
+ ep_get(ep);
+ return 0;
+}
+
+/*
* Must be called with "mtx" held.
*/
-static int ep_insert(struct eventpoll *ep, const struct epoll_event *event,
- struct file *tfile, int fd, int full_check)
+static int ep_insert(struct ep_ctl_ctx *ctx, struct eventpoll *ep,
+ const struct epoll_event *event, struct epoll_key *tf,
+ int full_check)
{
int error, pwake = 0;
__poll_t revents;
@@ -1572,58 +1887,21 @@ static int ep_insert(struct eventpoll *ep, const struct epoll_event *event,
struct ep_pqueue epq;
struct eventpoll *tep = NULL;
- if (is_file_epoll(tfile))
- tep = tfile->private_data;
+ if (is_file_epoll(tf->file))
+ tep = tf->file->private_data;
lockdep_assert_irqs_enabled();
- if (unlikely(percpu_counter_compare(&ep->user->epoll_watches,
- max_user_watches) >= 0))
- return -ENOSPC;
- percpu_counter_inc(&ep->user->epoll_watches);
+ epi = ep_alloc_epitem(ep, event, tf);
+ if (IS_ERR(epi))
+ return PTR_ERR(epi);
- if (!(epi = kmem_cache_zalloc(epi_cache, GFP_KERNEL))) {
- percpu_counter_dec(&ep->user->epoll_watches);
- return -ENOMEM;
- }
+ error = ep_register_epitem(ctx, ep, epi, tep, full_check);
+ if (error)
+ return error;
- /* Item initialization follow here ... */
- INIT_LIST_HEAD(&epi->rdllink);
- epi->ep = ep;
- ep_set_ffd(&epi->ffd, tfile, fd);
- epi->event = *event;
- epi->next = EP_UNACTIVE_PTR;
-
- if (tep)
- mutex_lock_nested(&tep->mtx, 1);
- /* Add the current item to the list of active epoll hook for this file */
- if (unlikely(attach_epitem(tfile, epi) < 0)) {
- if (tep)
- mutex_unlock(&tep->mtx);
- kmem_cache_free(epi_cache, epi);
- percpu_counter_dec(&ep->user->epoll_watches);
- return -ENOMEM;
- }
-
- if (full_check && !tep)
- list_file(tfile);
-
- /*
- * Add the current item to the RB tree. All RB tree operations are
- * protected by "mtx", and ep_insert() is called with "mtx" held.
- */
- ep_rbtree_insert(ep, epi);
- if (tep)
- mutex_unlock(&tep->mtx);
-
- /*
- * ep_remove() calls in the later error paths can't lead to
- * ep_free() as the ep file itself still holds an ep reference.
- */
- ep_get(ep);
-
- /* now check if we've created too many backpaths */
- if (unlikely(full_check && reverse_path_check())) {
+ /* Reject the insert if the new link would create too many back-paths. */
+ if (unlikely(full_check && reverse_path_check(ctx))) {
ep_remove(ep, epi);
return -EINVAL;
}
@@ -1649,28 +1927,21 @@ static int ep_insert(struct eventpoll *ep, const struct epoll_event *event,
*/
revents = ep_item_poll(epi, &epq.pt, 1);
- /*
- * We have to check if something went wrong during the poll wait queue
- * install process. Namely an allocation for a wait queue failed due
- * high memory pressure.
- */
+ /* ep_ptable_queue_proc() signals allocation failure by clearing epq.epi. */
if (unlikely(!epq.epi)) {
ep_remove(ep, epi);
return -ENOMEM;
}
- /* We have to drop the new item inside our item list to keep track of it */
+ /* Drop the new item onto the ready list if it is already ready. */
spin_lock_irq(&ep->lock);
- /* record NAPI ID of new item if present */
ep_set_busy_poll_napi_id(epi);
- /* If the file is already "ready" we drop it inside the ready list */
if (revents && !ep_is_linked(epi)) {
list_add_tail(&epi->rdllink, &ep->rdllist);
ep_pm_stay_awake(epi);
- /* Notify waiting tasks that events are available */
if (waitqueue_active(&ep->wq))
wake_up(&ep->wq);
if (waitqueue_active(&ep->poll_wait))
@@ -1762,11 +2033,87 @@ static int ep_modify(struct eventpoll *ep, struct epitem *epi,
return 0;
}
+/*
+ * Attempt to deliver one event for @epi into @*uevents.
+ *
+ * Returns 1 if an event was delivered (with *uevents advanced to the
+ * next slot), 0 if the re-poll reported no caller-requested events
+ * (@epi drops out of the ready list; a future callback will re-add
+ * it), or -EFAULT if copy_to_user() faulted (in which case @epi is
+ * re-inserted at the head of @scan_batch so ep_done_scan() merges it
+ * back to rdllist for the next attempt).
+ *
+ * PM bookkeeping and level-triggered re-queue are handled here.
+ * Caller holds ep->mtx and the scan is active.
+ */
+static int ep_deliver_event(struct eventpoll *ep, struct epitem *epi,
+ poll_table *pt,
+ struct epoll_event __user **uevents,
+ struct list_head *scan_batch)
+{
+ struct epoll_event __user *next;
+ struct wakeup_source *ws;
+ __poll_t revents;
+
+ /*
+ * Activate ep->ws before deactivating epi->ws to prevent
+ * triggering auto-suspend here (in case we reactivate epi->ws
+ * below). Rearranging to delay the deactivation would let
+ * epi->ws drift out of sync with ep_is_linked().
+ */
+ ws = ep_wakeup_source(epi);
+ if (ws) {
+ if (ws->active)
+ __pm_stay_awake(ep->ws);
+ __pm_relax(ws);
+ }
+
+ list_del_init(&epi->rdllink);
+
+ /*
+ * Re-poll under ep->mtx so userspace cannot change the item
+ * out from under us. If no caller-requested events remain,
+ * @epi stays off the ready list; the poll callback will
+ * re-queue it when events next appear.
+ */
+ revents = ep_item_poll(epi, pt, 1);
+ if (!revents)
+ return 0;
+
+ next = epoll_put_uevent(revents, epi->event.data, *uevents);
+ if (!next) {
+ /*
+ * copy_to_user() faulted: put the item back so
+ * ep_done_scan() splices it onto rdllist for the next
+ * attempt.
+ */
+ list_add(&epi->rdllink, scan_batch);
+ ep_pm_stay_awake(epi);
+ return -EFAULT;
+ }
+ *uevents = next;
+
+ if (epi->event.events & EPOLLONESHOT) {
+ epi->event.events &= EP_PRIVATE_BITS;
+ } else if (!(epi->event.events & EPOLLET)) {
+ /*
+ * Level-triggered: re-queue so the next epoll_wait()
+ * rechecks availability. We are the sole writer to
+ * rdllist here -- epoll_ctl() callers are locked out
+ * by ep->mtx, and the poll callback queues to ovflist
+ * during scans.
+ */
+ list_add_tail(&epi->rdllink, &ep->rdllist);
+ ep_pm_stay_awake(epi);
+ }
+ return 1;
+}
+
static int ep_send_events(struct eventpoll *ep,
struct epoll_event __user *events, int maxevents)
{
struct epitem *epi, *tmp;
- LIST_HEAD(txlist);
+ LIST_HEAD(scan_batch);
poll_table pt;
int res = 0;
@@ -1781,74 +2128,28 @@ static int ep_send_events(struct eventpoll *ep,
init_poll_funcptr(&pt, NULL);
mutex_lock(&ep->mtx);
- ep_start_scan(ep, &txlist);
+ ep_start_scan(ep, &scan_batch);
/*
- * We can loop without lock because we are passed a task private list.
- * Items cannot vanish during the loop we are holding ep->mtx.
+ * We can loop without lock because we are passed a task-private
+ * scan_batch; items cannot vanish while we hold ep->mtx.
*/
- list_for_each_entry_safe(epi, tmp, &txlist, rdllink) {
- struct wakeup_source *ws;
- __poll_t revents;
+ list_for_each_entry_safe(epi, tmp, &scan_batch, rdllink) {
+ int delivered;
if (res >= maxevents)
break;
- /*
- * Activate ep->ws before deactivating epi->ws to prevent
- * triggering auto-suspend here (in case we reactive epi->ws
- * below).
- *
- * This could be rearranged to delay the deactivation of epi->ws
- * instead, but then epi->ws would temporarily be out of sync
- * with ep_is_linked().
- */
- ws = ep_wakeup_source(epi);
- if (ws) {
- if (ws->active)
- __pm_stay_awake(ep->ws);
- __pm_relax(ws);
- }
-
- list_del_init(&epi->rdllink);
-
- /*
- * If the event mask intersect the caller-requested one,
- * deliver the event to userspace. Again, we are holding ep->mtx,
- * so no operations coming from userspace can change the item.
- */
- revents = ep_item_poll(epi, &pt, 1);
- if (!revents)
- continue;
-
- events = epoll_put_uevent(revents, epi->event.data, events);
- if (!events) {
- list_add(&epi->rdllink, &txlist);
- ep_pm_stay_awake(epi);
+ delivered = ep_deliver_event(ep, epi, &pt, &events, &scan_batch);
+ if (delivered < 0) {
if (!res)
- res = -EFAULT;
+ res = delivered;
break;
}
- res++;
- if (epi->event.events & EPOLLONESHOT)
- epi->event.events &= EP_PRIVATE_BITS;
- else if (!(epi->event.events & EPOLLET)) {
- /*
- * If this file has been added with Level
- * Trigger mode, we need to insert back inside
- * the ready list, so that the next call to
- * epoll_wait() will check again the events
- * availability. At this point, no one can insert
- * into ep->rdllist besides us. The epoll_ctl()
- * callers are locked out by
- * ep_send_events() holding "mtx" and the
- * poll callback will queue them in ep->ovflist.
- */
- list_add_tail(&epi->rdllink, &ep->rdllist);
- ep_pm_stay_awake(epi);
- }
+ res += delivered;
}
- ep_done_scan(ep, &txlist);
+
+ ep_done_scan(ep, &scan_batch);
mutex_unlock(&ep->mtx);
return res;
@@ -1938,7 +2239,8 @@ static int ep_schedule_timeout(ktime_t *to)
static int ep_poll(struct eventpoll *ep, struct epoll_event __user *events,
int maxevents, struct timespec64 *timeout)
{
- int res, eavail, timed_out = 0;
+ int res, timed_out = 0;
+ bool eavail;
u64 slack = 0;
wait_queue_entry_t wait;
ktime_t expires, *to = NULL;
@@ -2036,7 +2338,7 @@ static int ep_poll(struct eventpoll *ep, struct epoll_event __user *events,
* If timed out and still on the wait queue, recheck eavail
* carefully under lock, below.
*/
- eavail = 1;
+ eavail = true;
if (!list_empty_careful(&wait.entry)) {
spin_lock_irq(&ep->lock);
@@ -2060,13 +2362,15 @@ static int ep_poll(struct eventpoll *ep, struct epoll_event __user *events,
* epoll file does not create closed loops, and
* determine the depth of the subtree starting at @ep
*
+ * @ctx: Per-do_epoll_ctl() scratch for the loop / path checks.
* @ep: the &struct eventpoll to be currently checked.
* @depth: Current depth of the path being checked.
*
* Return: depth of the subtree, or a value bigger than EP_MAX_NESTS if we found
* a loop or went too deep.
*/
-static int ep_loop_check_proc(struct eventpoll *ep, int depth)
+static int ep_loop_check_proc(struct ep_ctl_ctx *ctx,
+ struct eventpoll *ep, int depth)
{
int result = 0;
struct rb_node *rbp;
@@ -2082,22 +2386,23 @@ static int ep_loop_check_proc(struct eventpoll *ep, int depth)
if (unlikely(is_file_epoll(epi->ffd.file))) {
struct eventpoll *ep_tovisit;
ep_tovisit = epi->ffd.file->private_data;
- if (ep_tovisit == inserting_into || depth > EP_MAX_NESTS)
+ if (ep_tovisit == ctx->inserting_into ||
+ depth > EP_MAX_NESTS)
result = EP_MAX_NESTS+1;
else
- result = max(result, ep_loop_check_proc(ep_tovisit, depth + 1) + 1);
+ result = max(result,
+ ep_loop_check_proc(ctx, ep_tovisit,
+ depth + 1) + 1);
if (result > EP_MAX_NESTS)
break;
} else {
/*
- * If we've reached a file that is not associated with
- * an ep, then we need to check if the newly added
- * links are going to add too many wakeup paths. We do
- * this by adding it to the tfile_check_list, if it's
- * not already there, and calling reverse_path_check()
- * during ep_insert().
+ * A non-epoll leaf. Queue it for the companion
+ * reverse_path_check() that runs after this walk so
+ * any new links we propose don't add too many wakeup
+ * paths.
*/
- list_file(epi->ffd.file);
+ list_file(epi->ffd.file, ctx);
}
}
ep->loop_check_depth = result;
@@ -2126,22 +2431,24 @@ static int ep_get_upwards_depth_proc(struct eventpoll *ep, int depth)
* into another epoll file (represented by @ep) does not create
* closed loops or too deep chains.
*
- * @ep: Pointer to the epoll we are inserting into.
- * @to: Pointer to the epoll to be inserted.
+ * @ctx: Per-CTL_ADD scratch context.
+ * @ep: Pointer to the epoll we are inserting into.
+ * @to: Pointer to the epoll to be inserted.
*
* Return: %zero if adding the epoll @to inside the epoll @from
* does not violate the constraints, or %-1 otherwise.
*/
-static int ep_loop_check(struct eventpoll *ep, struct eventpoll *to)
+static int ep_loop_check(struct ep_ctl_ctx *ctx, struct eventpoll *ep,
+ struct eventpoll *to)
{
int depth, upwards_depth;
- inserting_into = ep;
+ ctx->inserting_into = ep;
/*
* Check how deep down we can get from @to, and whether it is possible
* to loop up to @ep.
*/
- depth = ep_loop_check_proc(to, 0);
+ depth = ep_loop_check_proc(ctx, to, 0);
if (depth > EP_MAX_NESTS)
return -1;
/* Check how far up we can go from @ep. */
@@ -2152,12 +2459,12 @@ static int ep_loop_check(struct eventpoll *ep, struct eventpoll *to)
return (depth+1+upwards_depth > EP_MAX_NESTS) ? -1 : 0;
}
-static void clear_tfile_check_list(void)
+static void clear_tfile_check_list(struct ep_ctl_ctx *ctx)
{
rcu_read_lock();
- while (tfile_check_list != EP_UNACTIVE_PTR) {
- struct epitems_head *head = tfile_check_list;
- tfile_check_list = head->next;
+ while (ctx->tfile_check_list != EP_UNACTIVE_PTR) {
+ struct epitems_head *head = ctx->tfile_check_list;
+ ctx->tfile_check_list = head->next;
unlist_file(head);
}
rcu_read_unlock();
@@ -2223,38 +2530,107 @@ static inline void ep_take_care_of_epollwakeup(struct epoll_event *epev)
}
#endif
-static inline int epoll_mutex_lock(struct mutex *mutex, int depth,
- bool nonblock)
+static inline int epoll_mutex_lock(struct mutex *mutex, bool nonblock)
{
if (!nonblock) {
- mutex_lock_nested(mutex, depth);
+ mutex_lock(mutex);
return 0;
}
- if (mutex_trylock(mutex))
- return 0;
- return -EAGAIN;
+ return mutex_trylock(mutex) ? 0 : -EAGAIN;
}
-int do_epoll_ctl(int epfd, int op, int fd, struct epoll_event *epds,
- bool nonblock)
+/*
+ * Acquire the locks required for do_epoll_ctl() on @ep for @op.
+ *
+ * Always takes ep->mtx. For EPOLL_CTL_ADD, additionally runs the
+ * loop / path check under epnested_mutex when the topology can
+ * change: @ep is already watched (epfile->f_ep non-NULL), @ep was
+ * recently loop-checked (ep->gen == loop_check_gen), or @tfile is
+ * itself an eventpoll.
+ *
+ * Return value encodes both outcome and lock state:
+ *
+ * 0 success; ep->mtx held.
+ * 1 success; ep->mtx held AND the full check ran under
+ * epnested_mutex (which is also still held). The value
+ * doubles as the @full_check argument to ep_insert().
+ * -errno failure; no locks held.
+ *
+ * The caller releases what was taken with ep_ctl_unlock(ep, ret).
+ *
+ * Holding epnested_mutex on add is what prevents two racing
+ * EPOLL_CTL_ADDs on different eps from building a cycle without
+ * either walker observing it.
+ */
+static int ep_ctl_lock(struct ep_ctl_ctx *ctx, struct eventpoll *ep, int op,
+ struct file *epfile, struct file *tfile, bool nonblock)
+{
+ struct eventpoll *tep;
+ int error;
+
+ error = epoll_mutex_lock(&ep->mtx, nonblock);
+ if (error)
+ return error;
+
+ if (op != EPOLL_CTL_ADD)
+ return 0;
+ if (!READ_ONCE(epfile->f_ep) && ep->gen != loop_check_gen &&
+ !is_file_epoll(tfile))
+ return 0;
+
+ /* Full check needed: drop ep->mtx so we can take epnested_mutex. */
+ mutex_unlock(&ep->mtx);
+ error = epoll_mutex_lock(&epnested_mutex, nonblock);
+ if (error)
+ return error;
+
+ loop_check_gen++;
+
+ if (is_file_epoll(tfile)) {
+ tep = tfile->private_data;
+ if (ep_loop_check(ctx, ep, tep) != 0) {
+ error = -ELOOP;
+ goto err_unlock_nested;
+ }
+ }
+
+ error = epoll_mutex_lock(&ep->mtx, nonblock);
+ if (error)
+ goto err_unlock_nested;
+
+ return 1;
+
+err_unlock_nested:
+ clear_tfile_check_list(ctx);
+ loop_check_gen++;
+ mutex_unlock(&epnested_mutex);
+ return error;
+}
+
+static void ep_ctl_unlock(struct ep_ctl_ctx *ctx, struct eventpoll *ep,
+ int full_check)
+{
+ mutex_unlock(&ep->mtx);
+ if (full_check) {
+ clear_tfile_check_list(ctx);
+ loop_check_gen++;
+ mutex_unlock(&epnested_mutex);
+ }
+}
+
+int do_epoll_ctl_file(struct file *f, int op, struct epoll_key *tf,
+ struct epoll_event *epds, bool nonblock)
{
int error;
- int full_check = 0;
+ int full_check;
struct eventpoll *ep;
struct epitem *epi;
- struct eventpoll *tep = NULL;
-
- CLASS(fd, f)(epfd);
- if (fd_empty(f))
- return -EBADF;
-
- /* Get the "struct file *" for the target file */
- CLASS(fd, tf)(fd);
- if (fd_empty(tf))
- return -EBADF;
+ struct ep_ctl_ctx ctx = {
+ .tfile_check_list = EP_UNACTIVE_PTR,
+ };
/* The target file descriptor must support poll */
- if (!file_can_poll(fd_file(tf)))
+ if (!file_can_poll(tf->file))
return -EPERM;
/* Check if EPOLLWAKEUP is allowed */
@@ -2262,85 +2638,43 @@ int do_epoll_ctl(int epfd, int op, int fd, struct epoll_event *epds,
ep_take_care_of_epollwakeup(epds);
/*
- * We have to check that the file structure underneath the file descriptor
- * the user passed to us _is_ an eventpoll file. And also we do not permit
+ * The @f file must itself be an eventpoll, and we do not permit
* adding an epoll file descriptor inside itself.
*/
- error = -EINVAL;
- if (fd_file(f) == fd_file(tf) || !is_file_epoll(fd_file(f)))
- goto error_tgt_fput;
+ if (f == tf->file || !is_file_epoll(f))
+ return -EINVAL;
/*
* epoll adds to the wakeup queue at EPOLL_CTL_ADD time only,
* so EPOLLEXCLUSIVE is not allowed for a EPOLL_CTL_MOD operation.
- * Also, we do not currently supported nested exclusive wakeups.
+ * Also, nested exclusive wakeups are not supported.
*/
if (ep_op_has_event(op) && (epds->events & EPOLLEXCLUSIVE)) {
if (op == EPOLL_CTL_MOD)
- goto error_tgt_fput;
- if (op == EPOLL_CTL_ADD && (is_file_epoll(fd_file(tf)) ||
+ return -EINVAL;
+ if (op == EPOLL_CTL_ADD && (is_file_epoll(tf->file) ||
(epds->events & ~EPOLLEXCLUSIVE_OK_BITS)))
- goto error_tgt_fput;
+ return -EINVAL;
}
- /*
- * At this point it is safe to assume that the "private_data" contains
- * our own data structure.
- */
- ep = fd_file(f)->private_data;
+ ep = f->private_data;
+
+ full_check = ep_ctl_lock(&ctx, ep, op, f, tf->file, nonblock);
+ if (full_check < 0)
+ return full_check;
/*
- * When we insert an epoll file descriptor inside another epoll file
- * descriptor, there is the chance of creating closed loops, which are
- * better be handled here, than in more critical paths. While we are
- * checking for loops we also determine the list of files reachable
- * and hang them on the tfile_check_list, so we can check that we
- * haven't created too many possible wakeup paths.
- *
- * We do not need to take the global 'epumutex' on EPOLL_CTL_ADD when
- * the epoll file descriptor is attaching directly to a wakeup source,
- * unless the epoll file descriptor is nested. The purpose of taking the
- * 'epnested_mutex' on add is to prevent complex toplogies such as loops and
- * deep wakeup paths from forming in parallel through multiple
- * EPOLL_CTL_ADD operations.
+ * Look the target up in ep's RB tree. We hold ep->mtx, so the
+ * item stays valid until we release.
*/
- error = epoll_mutex_lock(&ep->mtx, 0, nonblock);
- if (error)
- goto error_tgt_fput;
- if (op == EPOLL_CTL_ADD) {
- if (READ_ONCE(fd_file(f)->f_ep) || ep->gen == loop_check_gen ||
- is_file_epoll(fd_file(tf))) {
- mutex_unlock(&ep->mtx);
- error = epoll_mutex_lock(&epnested_mutex, 0, nonblock);
- if (error)
- goto error_tgt_fput;
- loop_check_gen++;
- full_check = 1;
- if (is_file_epoll(fd_file(tf))) {
- tep = fd_file(tf)->private_data;
- error = -ELOOP;
- if (ep_loop_check(ep, tep) != 0)
- goto error_tgt_fput;
- }
- error = epoll_mutex_lock(&ep->mtx, 0, nonblock);
- if (error)
- goto error_tgt_fput;
- }
- }
-
- /*
- * Try to lookup the file inside our RB tree. Since we grabbed "mtx"
- * above, we can be sure to be able to use the item looked up by
- * ep_find() till we release the mutex.
- */
- epi = ep_find(ep, fd_file(tf), fd);
+ epi = ep_find(ep, tf);
error = -EINVAL;
switch (op) {
case EPOLL_CTL_ADD:
if (!epi) {
epds->events |= EPOLLERR | EPOLLHUP;
- error = ep_insert(ep, epds, fd_file(tf), fd, full_check);
+ error = ep_insert(&ctx, ep, epds, tf, full_check);
} else
error = -EEXIST;
break;
@@ -2366,17 +2700,30 @@ int do_epoll_ctl(int epfd, int op, int fd, struct epoll_event *epds,
error = -ENOENT;
break;
}
- mutex_unlock(&ep->mtx);
-error_tgt_fput:
- if (full_check) {
- clear_tfile_check_list();
- loop_check_gen++;
- mutex_unlock(&epnested_mutex);
- }
+ ep_ctl_unlock(&ctx, ep, full_check);
return error;
}
+int do_epoll_ctl(int epfd, int op, int fd, struct epoll_event *epds,
+ bool nonblock)
+{
+ struct epoll_key efd;
+
+ CLASS(fd, f)(epfd);
+ if (fd_empty(f))
+ return -EBADF;
+
+ /* Get the "struct file *" for the target file */
+ CLASS(fd, tf)(fd);
+ if (fd_empty(tf))
+ return -EBADF;
+
+ efd.file = fd_file(tf);
+ efd.fd = fd;
+ return do_epoll_ctl_file(fd_file(f), op, &efd, epds, nonblock);
+}
+
/*
* The following function implements the controller interface for
* the eventpoll file that enables the insertion/removal/change of
@@ -2527,6 +2874,50 @@ SYSCALL_DEFINE6(epoll_pwait2, int, epfd, struct epoll_event __user *, events,
sigmask, sigsetsize);
}
+#ifdef CONFIG_KCMP
+static struct epitem *ep_find_tfd(struct eventpoll *ep, int tfd, unsigned long toff)
+{
+ struct rb_node *rbp;
+ struct epitem *epi;
+
+ for (rbp = rb_first_cached(&ep->rbr); rbp; rbp = rb_next(rbp)) {
+ epi = rb_entry(rbp, struct epitem, rbn);
+ if (epi->ffd.fd == tfd) {
+ if (toff == 0)
+ return epi;
+ else
+ toff--;
+ }
+ cond_resched();
+ }
+
+ return NULL;
+}
+
+struct file *get_epoll_tfile_raw_ptr(struct file *file, int tfd,
+ unsigned long toff)
+{
+ struct file *file_raw;
+ struct eventpoll *ep;
+ struct epitem *epi;
+
+ if (!is_file_epoll(file))
+ return ERR_PTR(-EINVAL);
+
+ ep = file->private_data;
+
+ mutex_lock(&ep->mtx);
+ epi = ep_find_tfd(ep, tfd, toff);
+ if (epi)
+ file_raw = epi->ffd.file;
+ else
+ file_raw = ERR_PTR(-ENOENT);
+ mutex_unlock(&ep->mtx);
+
+ return file_raw;
+}
+#endif /* CONFIG_KCMP */
+
#ifdef CONFIG_COMPAT
static int do_compat_epoll_pwait(int epfd, struct epoll_event __user *events,
int maxevents, struct timespec64 *timeout,
diff --git a/fs/exec.c b/fs/exec.c
index ba12b4c..b92fe7d 100644
--- a/fs/exec.c
+++ b/fs/exec.c
@@ -35,6 +35,7 @@
#include <linux/init.h>
#include <linux/sched/mm.h>
#include <linux/sched/coredump.h>
+#include <linux/sched/exec_state.h>
#include <linux/sched/signal.h>
#include <linux/sched/numa_balancing.h>
#include <linux/sched/task.h>
@@ -263,6 +264,9 @@ static int bprm_mm_init(struct linux_binprm *bprm)
if (!mm)
goto err;
+ /* Staged for would_dump() narrowing; consumed by begin_new_exec(). */
+ bprm->user_ns = get_user_ns(current_user_ns());
+
/* Save current stack limit for all calculations made during exec. */
task_lock(current->group_leader);
bprm->rlim_stack = current->signal->rlim[RLIMIT_STACK];
@@ -832,14 +836,21 @@ EXPORT_SYMBOL(read_code);
/*
* Maps the mm_struct mm into the current task struct.
* On success, this function returns with exec_update_lock
- * held for writing.
+ * held for writing. The replaced address space is stashed in
+ * bprm->old_mm for setup_new_exec() to release outside the lock.
*/
-static int exec_mmap(struct mm_struct *mm)
+static int exec_mmap(struct linux_binprm *bprm)
{
+ struct task_exec_state *exec_state __free(put_task_exec_state) = NULL;
+ struct mm_struct *mm = bprm->mm;
struct task_struct *tsk;
struct mm_struct *old_mm, *active_mm;
int ret;
+ exec_state = alloc_task_exec_state(bprm->user_ns);
+ if (!exec_state)
+ return -ENOMEM;
+
/* Notify parent that we're no longer interested in the old VM */
tsk = current;
old_mm = current->mm;
@@ -870,6 +881,7 @@ static int exec_mmap(struct mm_struct *mm)
tsk->active_mm = mm;
tsk->mm = mm;
mm_init_cid(mm, tsk);
+ exec_state = task_exec_state_replace(tsk, exec_state);
/*
* This prevents preemption while active_mm is being loaded and
* it and mm are being updated, which could cause problems for
@@ -888,15 +900,22 @@ static int exec_mmap(struct mm_struct *mm)
if (old_mm) {
mmap_read_unlock(old_mm);
BUG_ON(active_mm != old_mm);
- setmax_mm_hiwater_rss(&tsk->signal->maxrss, old_mm);
- mm_update_next_owner(old_mm);
- mmput(old_mm);
+ /* Defer teardown to setup_new_exec(), outside the exec locks. */
+ bprm->old_mm = old_mm;
return 0;
}
mmdrop_lazy_tlb(active_mm);
return 0;
}
+/* Release the address space replaced by exec, outside the exec locks. */
+static void exec_mm_put_old(struct mm_struct *old_mm)
+{
+ setmax_mm_hiwater_rss(¤t->signal->maxrss, old_mm);
+ mm_update_next_owner(old_mm);
+ mmput(old_mm);
+}
+
static int de_thread(struct task_struct *tsk)
{
struct signal_struct *sig = tsk->signal;
@@ -1145,7 +1164,7 @@ int begin_new_exec(struct linux_binprm * bprm)
* Release all of the old mmap stuff
*/
acct_arg_size(bprm, 0);
- retval = exec_mmap(bprm->mm);
+ retval = exec_mmap(bprm);
if (retval)
goto out;
@@ -1210,9 +1229,9 @@ int begin_new_exec(struct linux_binprm * bprm)
if (bprm->interp_flags & BINPRM_FLAGS_ENFORCE_NONDUMP ||
!(uid_eq(current_euid(), current_uid()) &&
gid_eq(current_egid(), current_gid())))
- set_dumpable(current->mm, suid_dumpable);
+ task_exec_state_set_dumpable(suid_dumpable);
else
- set_dumpable(current->mm, SUID_DUMP_USER);
+ task_exec_state_set_dumpable(TASK_DUMPABLE_OWNER);
perf_event_exec();
@@ -1261,7 +1280,7 @@ int begin_new_exec(struct linux_binprm * bprm)
* wait until new credentials are committed
* by commit_creds() above
*/
- if (get_dumpable(me->mm) != SUID_DUMP_USER)
+ if (task_exec_state_get_dumpable(me) != TASK_DUMPABLE_OWNER)
perf_event_exit_task(me);
/*
* cred_guard_mutex must be held at least to this point to prevent
@@ -1298,14 +1317,14 @@ void would_dump(struct linux_binprm *bprm, struct file *file)
struct user_namespace *old, *user_ns;
bprm->interp_flags |= BINPRM_FLAGS_ENFORCE_NONDUMP;
- /* Ensure mm->user_ns contains the executable */
- user_ns = old = bprm->mm->user_ns;
+ /* Ensure bprm->user_ns contains the executable. */
+ user_ns = old = bprm->user_ns;
while ((user_ns != &init_user_ns) &&
!privileged_wrt_inode_uidgid(user_ns, idmap, inode))
user_ns = user_ns->parent;
if (old != user_ns) {
- bprm->mm->user_ns = get_user_ns(user_ns);
+ bprm->user_ns = get_user_ns(user_ns);
put_user_ns(old);
}
}
@@ -1328,6 +1347,12 @@ void setup_new_exec(struct linux_binprm * bprm)
me->mm->task_size = TASK_SIZE;
up_write(&me->signal->exec_update_lock);
mutex_unlock(&me->signal->cred_guard_mutex);
+
+ /* The exec locks are dropped: release the old address space now. */
+ if (bprm->old_mm) {
+ exec_mm_put_old(bprm->old_mm);
+ bprm->old_mm = NULL;
+ }
}
EXPORT_SYMBOL(setup_new_exec);
@@ -1375,6 +1400,8 @@ static void free_bprm(struct linux_binprm *bprm)
acct_arg_size(bprm, 0);
mmput(bprm->mm);
}
+ if (bprm->user_ns)
+ put_user_ns(bprm->user_ns);
free_arg_pages(bprm);
if (bprm->cred) {
/* in case exec fails before de_thread() succeeds */
@@ -1382,6 +1409,9 @@ static void free_bprm(struct linux_binprm *bprm)
mutex_unlock(¤t->signal->cred_guard_mutex);
abort_creds(bprm->cred);
}
+ /* exec swapped the mm but failed before setup_new_exec() freed it */
+ if (bprm->old_mm)
+ exec_mm_put_old(bprm->old_mm);
do_close_execat(bprm->file);
if (bprm->executable)
fput(bprm->executable);
@@ -1905,17 +1935,6 @@ void set_binfmt(struct linux_binfmt *new)
}
EXPORT_SYMBOL(set_binfmt);
-/*
- * set_dumpable stores three-value SUID_DUMP_* into mm->flags.
- */
-void set_dumpable(struct mm_struct *mm, int value)
-{
- if (WARN_ON((unsigned)value > SUID_DUMP_ROOT))
- return;
-
- __mm_flags_set_mask_dumpable(mm, value);
-}
-
static inline struct user_arg_ptr native_arg(const char __user *const __user *p)
{
return (struct user_arg_ptr){.ptr.native = p};
@@ -1975,9 +1994,11 @@ COMPAT_SYSCALL_DEFINE5(execveat, int, fd,
static int proc_dointvec_minmax_coredump(const struct ctl_table *table, int write,
void *buffer, size_t *lenp, loff_t *ppos)
{
- int error = proc_dointvec_minmax(table, write, buffer, lenp, ppos);
+ int error, old = READ_ONCE(suid_dumpable);
- if (!error && write)
+ error = proc_dointvec_minmax(table, write, buffer, lenp, ppos);
+
+ if (!error && write && (old != READ_ONCE(suid_dumpable)))
validate_coredump_safety();
return error;
}
diff --git a/fs/exfat/exfat_fs.h b/fs/exfat/exfat_fs.h
index 89ef536..aff4dcd 100644
--- a/fs/exfat/exfat_fs.h
+++ b/fs/exfat/exfat_fs.h
@@ -496,6 +496,8 @@ int exfat_setattr(struct mnt_idmap *idmap, struct dentry *dentry,
int exfat_getattr(struct mnt_idmap *idmap, const struct path *path,
struct kstat *stat, unsigned int request_mask,
unsigned int query_flags);
+struct file_kattr;
+int exfat_fileattr_get(struct dentry *dentry, struct file_kattr *fa);
int exfat_file_fsync(struct file *file, loff_t start, loff_t end, int datasync);
long exfat_ioctl(struct file *filp, unsigned int cmd, unsigned long arg);
long exfat_compat_ioctl(struct file *filp, unsigned int cmd,
diff --git a/fs/exfat/file.c b/fs/exfat/file.c
index 354bdcf..91e5511 100644
--- a/fs/exfat/file.c
+++ b/fs/exfat/file.c
@@ -14,6 +14,7 @@
#include <linux/writeback.h>
#include <linux/filelock.h>
#include <linux/falloc.h>
+#include <linux/fileattr.h>
#include "exfat_raw.h"
#include "exfat_fs.h"
@@ -323,6 +324,18 @@ int exfat_getattr(struct mnt_idmap *idmap, const struct path *path,
return 0;
}
+int exfat_fileattr_get(struct dentry *dentry, struct file_kattr *fa)
+{
+ /*
+ * exFAT compares filenames through an upcase table, so lookup
+ * is always case-insensitive. Long names are stored in UTF-16
+ * with case intact; CASENONPRESERVING stays clear.
+ */
+ fa->fsx_xflags |= FS_XFLAG_CASEFOLD;
+ fa->flags |= FS_CASEFOLD_FL;
+ return 0;
+}
+
int exfat_setattr(struct mnt_idmap *idmap, struct dentry *dentry,
struct iattr *attr)
{
@@ -817,6 +830,7 @@ const struct file_operations exfat_file_operations = {
};
const struct inode_operations exfat_file_inode_operations = {
- .setattr = exfat_setattr,
- .getattr = exfat_getattr,
+ .setattr = exfat_setattr,
+ .getattr = exfat_getattr,
+ .fileattr_get = exfat_fileattr_get,
};
diff --git a/fs/exfat/namei.c b/fs/exfat/namei.c
index 2c56366..94002e4 100644
--- a/fs/exfat/namei.c
+++ b/fs/exfat/namei.c
@@ -1311,4 +1311,5 @@ const struct inode_operations exfat_dir_inode_operations = {
.rename = exfat_rename,
.setattr = exfat_setattr,
.getattr = exfat_getattr,
+ .fileattr_get = exfat_fileattr_get,
};
diff --git a/fs/exportfs/expfs.c b/fs/exportfs/expfs.c
index fbd45e7..eafd995 100644
--- a/fs/exportfs/expfs.c
+++ b/fs/exportfs/expfs.c
@@ -53,10 +53,10 @@ find_acceptable_alias(struct dentry *result,
inode = result->d_inode;
spin_lock(&inode->i_lock);
for_each_alias(dentry, inode) {
- dget(dentry);
+ if (!dget_alias_ilocked(dentry))
+ continue;
spin_unlock(&inode->i_lock);
- if (toput)
- dput(toput);
+ dput(toput);
if (dentry != result && acceptable(context, dentry)) {
dput(result);
return dentry;
@@ -66,8 +66,7 @@ find_acceptable_alias(struct dentry *result,
}
spin_unlock(&inode->i_lock);
- if (toput)
- dput(toput);
+ dput(toput);
return NULL;
}
diff --git a/fs/ext4/ext4.h b/fs/ext4/ext4.h
index 94283a9..6af11f0 100644
--- a/fs/ext4/ext4.h
+++ b/fs/ext4/ext4.h
@@ -2959,7 +2959,7 @@ extern unsigned long ext4_count_dirs(struct super_block *);
extern void ext4_mark_bitmap_end(int start_bit, int end_bit, char *bitmap);
extern int ext4_init_inode_table(struct super_block *sb,
ext4_group_t group, int barrier);
-extern void ext4_end_bitmap_read(struct buffer_head *bh, int uptodate);
+void ext4_end_bitmap_read(struct bio *bio);
/* fast_commit.c */
int ext4_fc_info_show(struct seq_file *seq, void *v);
@@ -3184,10 +3184,10 @@ extern struct buffer_head *ext4_sb_bread_unmovable(struct super_block *sb,
sector_t block);
extern struct buffer_head *ext4_sb_bread_nofail(struct super_block *sb,
sector_t block);
-extern void ext4_read_bh_nowait(struct buffer_head *bh, blk_opf_t op_flags,
- bh_end_io_t *end_io, bool simu_fail);
-extern int ext4_read_bh(struct buffer_head *bh, blk_opf_t op_flags,
- bh_end_io_t *end_io, bool simu_fail);
+void ext4_read_bh_nowait(struct buffer_head *bh, blk_opf_t op_flags,
+ bio_end_io_t end_io, bool simu_fail);
+int ext4_read_bh(struct buffer_head *bh, blk_opf_t op_flags,
+ bio_end_io_t end_io, bool simu_fail);
extern int ext4_read_bh_lock(struct buffer_head *bh, blk_opf_t op_flags, bool wait);
extern void ext4_sb_breadahead_unmovable(struct super_block *sb, sector_t block);
extern int ext4_seq_options_show(struct seq_file *seq, void *offset);
diff --git a/fs/ext4/extents-test.c b/fs/ext4/extents-test.c
index 6b53a3f..bd7795a 100644
--- a/fs/ext4/extents-test.c
+++ b/fs/ext4/extents-test.c
@@ -37,6 +37,7 @@
#include <kunit/test.h>
#include <kunit/static_stub.h>
+#include <linux/fs_context.h>
#include <linux/gfp_types.h>
#include <linux/stddef.h>
@@ -130,14 +131,20 @@ static void ext_kill_sb(struct super_block *sb)
generic_shutdown_super(sb);
}
-static int ext_set(struct super_block *sb, void *data)
+static int ext_init_fs_context(struct fs_context *fc)
+{
+ return 0;
+}
+
+static int ext_set(struct super_block *sb, struct fs_context *fc)
{
return 0;
}
static struct file_system_type ext_fs_type = {
- .name = "extents test",
- .kill_sb = ext_kill_sb,
+ .name = "extents test",
+ .init_fs_context = ext_init_fs_context,
+ .kill_sb = ext_kill_sb,
};
static void extents_kunit_exit(struct kunit *test)
@@ -223,6 +230,7 @@ static int extents_kunit_init(struct kunit *test)
struct ext4_inode_info *ei;
struct inode *inode;
struct super_block *sb;
+ struct fs_context *fc;
struct ext4_sb_info *sbi = NULL;
struct kunit_ext_test_param *param =
(struct kunit_ext_test_param *)(test->param_value);
@@ -232,7 +240,13 @@ static int extents_kunit_init(struct kunit *test)
if (sbi == NULL)
return -ENOMEM;
- sb = sget(&ext_fs_type, NULL, ext_set, 0, NULL);
+ fc = fs_context_for_mount(&ext_fs_type, 0);
+ if (IS_ERR(fc)) {
+ kfree(sbi);
+ return PTR_ERR(fc);
+ }
+ sb = sget_fc(fc, NULL, ext_set);
+ put_fs_context(fc);
if (IS_ERR(sb)) {
kfree(sbi);
return PTR_ERR(sb);
diff --git a/fs/ext4/fast_commit.c b/fs/ext4/fast_commit.c
index b3c2263..5773b85 100644
--- a/fs/ext4/fast_commit.c
+++ b/fs/ext4/fast_commit.c
@@ -184,8 +184,11 @@
#include <trace/events/ext4.h>
static struct kmem_cache *ext4_fc_dentry_cachep;
-static void ext4_end_buffer_io_sync(struct buffer_head *bh, int uptodate)
+static void ext4_end_buffer_io_sync(struct bio *bio)
{
+ struct buffer_head *bh;
+ bool uptodate = bio_endio_bh(bio, &bh);
+
BUFFER_TRACE(bh, "");
if (uptodate) {
ext4_debug("%s: Block %lld up-to-date",
@@ -659,8 +662,7 @@ static void ext4_fc_submit_bh(struct super_block *sb, bool is_tail)
lock_buffer(bh);
set_buffer_dirty(bh);
set_buffer_uptodate(bh);
- bh->b_end_io = ext4_end_buffer_io_sync;
- submit_bh(REQ_OP_WRITE | write_flags, bh);
+ bh_submit(bh, REQ_OP_WRITE | write_flags, ext4_end_buffer_io_sync);
EXT4_SB(sb)->s_fc_bh = NULL;
}
diff --git a/fs/ext4/ialloc.c b/fs/ext4/ialloc.c
index 3fd8f00..a40cb27 100644
--- a/fs/ext4/ialloc.c
+++ b/fs/ext4/ialloc.c
@@ -66,14 +66,16 @@ void ext4_mark_bitmap_end(int start_bit, int end_bit, char *bitmap)
memset(bitmap + (i >> 3), 0xff, (end_bit - i) >> 3);
}
-void ext4_end_bitmap_read(struct buffer_head *bh, int uptodate)
+void ext4_end_bitmap_read(struct bio *bio)
{
+ struct buffer_head *bh;
+ bool uptodate = bio_endio_bh(bio, &bh);
+
if (uptodate) {
set_buffer_uptodate(bh);
set_bitmap_uptodate(bh);
}
unlock_buffer(bh);
- put_bh(bh);
}
static int ext4_validate_inode_bitmap(struct super_block *sb,
@@ -252,10 +254,10 @@ void ext4_free_inode(handle_t *handle, struct inode *inode)
"nonexistent device\n", __func__, __LINE__);
return;
}
- if (icount_read(inode) > 1) {
+ if (icount_read_once(inode) > 1) {
ext4_msg(sb, KERN_ERR, "%s:%d: inode #%llu: count=%d",
__func__, __LINE__, inode->i_ino,
- icount_read(inode));
+ icount_read_once(inode));
return;
}
if (inode->i_nlink) {
diff --git a/fs/ext4/mballoc-test.c b/fs/ext4/mballoc-test.c
index 90ed505..d90da44 100644
--- a/fs/ext4/mballoc-test.c
+++ b/fs/ext4/mballoc-test.c
@@ -5,6 +5,7 @@
#include <kunit/test.h>
#include <kunit/static_stub.h>
+#include <linux/fs_context.h>
#include <linux/random.h>
#include "ext4.h"
@@ -63,8 +64,14 @@ static void mbt_kill_sb(struct super_block *sb)
generic_shutdown_super(sb);
}
+static int mbt_init_fs_context(struct fs_context *fc)
+{
+ return 0;
+}
+
static struct file_system_type mbt_fs_type = {
.name = "mballoc test",
+ .init_fs_context = mbt_init_fs_context,
.kill_sb = mbt_kill_sb,
};
@@ -127,7 +134,7 @@ static void mbt_mb_release(struct super_block *sb)
kfree(sb->s_bdev);
}
-static int mbt_set(struct super_block *sb, void *data)
+static int mbt_set(struct super_block *sb, struct fs_context *fc)
{
return 0;
}
@@ -136,13 +143,19 @@ static struct super_block *mbt_ext4_alloc_super_block(void)
{
struct mbt_ext4_super_block *fsb;
struct super_block *sb;
+ struct fs_context *fc;
struct ext4_sb_info *sbi;
fsb = kzalloc_obj(*fsb);
if (fsb == NULL)
return NULL;
- sb = sget(&mbt_fs_type, NULL, mbt_set, 0, NULL);
+ fc = fs_context_for_mount(&mbt_fs_type, 0);
+ if (IS_ERR(fc))
+ goto out;
+
+ sb = sget_fc(fc, NULL, mbt_set);
+ put_fs_context(fc);
if (IS_ERR(sb))
goto out;
diff --git a/fs/ext4/mmp.c b/fs/ext4/mmp.c
index 6f57c18..7ce3614 100644
--- a/fs/ext4/mmp.c
+++ b/fs/ext4/mmp.c
@@ -46,9 +46,8 @@ static int write_mmp_block_thawed(struct super_block *sb,
ext4_mmp_csum_set(sb, mmp);
lock_buffer(bh);
- bh->b_end_io = end_buffer_write_sync;
- get_bh(bh);
- submit_bh(REQ_OP_WRITE | REQ_SYNC | REQ_META | REQ_PRIO, bh);
+ bh_submit(bh, REQ_OP_WRITE | REQ_SYNC | REQ_META | REQ_PRIO,
+ bh_end_write);
wait_on_buffer(bh);
if (unlikely(!buffer_uptodate(bh)))
return -EIO;
diff --git a/fs/ext4/super.c b/fs/ext4/super.c
index 6a77db4d..7283108 100644
--- a/fs/ext4/super.c
+++ b/fs/ext4/super.c
@@ -161,7 +161,7 @@ MODULE_ALIAS("ext3");
static inline void __ext4_read_bh(struct buffer_head *bh, blk_opf_t op_flags,
- bh_end_io_t *end_io, bool simu_fail)
+ bio_end_io_t end_io, bool simu_fail)
{
if (simu_fail) {
clear_buffer_uptodate(bh);
@@ -176,13 +176,13 @@ static inline void __ext4_read_bh(struct buffer_head *bh, blk_opf_t op_flags,
*/
clear_buffer_verified(bh);
- bh->b_end_io = end_io ? end_io : end_buffer_read_sync;
- get_bh(bh);
- submit_bh(REQ_OP_READ | op_flags, bh);
+ if (!end_io)
+ end_io = bh_end_read;
+ bh_submit(bh, REQ_OP_READ | op_flags, end_io);
}
void ext4_read_bh_nowait(struct buffer_head *bh, blk_opf_t op_flags,
- bh_end_io_t *end_io, bool simu_fail)
+ bio_end_io_t end_io, bool simu_fail)
{
BUG_ON(!buffer_locked(bh));
@@ -194,7 +194,7 @@ void ext4_read_bh_nowait(struct buffer_head *bh, blk_opf_t op_flags,
}
int ext4_read_bh(struct buffer_head *bh, blk_opf_t op_flags,
- bh_end_io_t *end_io, bool simu_fail)
+ bio_end_io_t end_io, bool simu_fail)
{
BUG_ON(!buffer_locked(bh));
@@ -6316,12 +6316,10 @@ static int ext4_commit_super(struct super_block *sb)
clear_buffer_write_io_error(sbh);
set_buffer_uptodate(sbh);
}
- get_bh(sbh);
/* Clear potential dirty bit if it was journalled update */
clear_buffer_dirty(sbh);
- sbh->b_end_io = end_buffer_write_sync;
- submit_bh(REQ_OP_WRITE | REQ_SYNC |
- (test_opt(sb, BARRIER) ? REQ_FUA : 0), sbh);
+ bh_submit(sbh, REQ_OP_WRITE | REQ_SYNC |
+ (test_opt(sb, BARRIER) ? REQ_FUA : 0), bh_end_write);
wait_on_buffer(sbh);
if (buffer_write_io_error(sbh)) {
ext4_msg(sb, KERN_ERR, "I/O error while writing "
diff --git a/fs/fat/fat.h b/fs/fat/fat.h
index 5a58f0b..99ed922 100644
--- a/fs/fat/fat.h
+++ b/fs/fat/fat.h
@@ -10,6 +10,8 @@
#include <linux/fs_context.h>
#include <linux/fs_parser.h>
+struct file_kattr;
+
/*
* vfat shortname flags
*/
@@ -408,6 +410,7 @@ extern void fat_truncate_blocks(struct inode *inode, loff_t offset);
extern int fat_getattr(struct mnt_idmap *idmap,
const struct path *path, struct kstat *stat,
u32 request_mask, unsigned int flags);
+int fat_fileattr_get(struct dentry *dentry, struct file_kattr *fa);
extern int fat_file_fsync(struct file *file, loff_t start, loff_t end,
int datasync);
diff --git a/fs/fat/file.c b/fs/fat/file.c
index becccdd..37e7049 100644
--- a/fs/fat/file.c
+++ b/fs/fat/file.c
@@ -17,6 +17,7 @@
#include <linux/fsnotify.h>
#include <linux/security.h>
#include <linux/falloc.h>
+#include <linux/fileattr.h>
#include "fat.h"
static long fat_fallocate(struct file *file, int mode,
@@ -398,6 +399,40 @@ void fat_truncate_blocks(struct inode *inode, loff_t offset)
fat_flush_inodes(inode->i_sb, inode, NULL);
}
+int fat_fileattr_get(struct dentry *dentry, struct file_kattr *fa)
+{
+ struct msdos_sb_info *sbi = MSDOS_SB(dentry->d_sb);
+ bool case_sensitive;
+
+ /*
+ * FAT filesystems are case-insensitive by default. VFAT
+ * becomes case-sensitive when mounted with 'check=strict',
+ * which installs vfat_dentry_ops. MSDOS has no such option;
+ * its 'nocase' mount option selects case-sensitive matching.
+ *
+ * VFAT long filename entries preserve case. Without VFAT, only
+ * uppercased 8.3 short names are stored. MSDOS with 'nocase'
+ * also preserves case.
+ */
+ if (sbi->options.isvfat)
+ case_sensitive = sbi->options.name_check == 's';
+ else
+ case_sensitive = sbi->options.nocase;
+
+ if (!case_sensitive) {
+ fa->fsx_xflags |= FS_XFLAG_CASEFOLD;
+ fa->flags |= FS_CASEFOLD_FL;
+ if (!sbi->options.isvfat)
+ fa->fsx_xflags |= FS_XFLAG_CASENONPRESERVING;
+ }
+ if (d_inode(dentry)->i_flags & S_IMMUTABLE) {
+ fa->fsx_xflags |= FS_XFLAG_IMMUTABLE;
+ fa->flags |= FS_IMMUTABLE_FL;
+ }
+ return 0;
+}
+EXPORT_SYMBOL_GPL(fat_fileattr_get);
+
int fat_getattr(struct mnt_idmap *idmap, const struct path *path,
struct kstat *stat, u32 request_mask, unsigned int flags)
{
@@ -575,5 +610,6 @@ EXPORT_SYMBOL_GPL(fat_setattr);
const struct inode_operations fat_file_inode_operations = {
.setattr = fat_setattr,
.getattr = fat_getattr,
+ .fileattr_get = fat_fileattr_get,
.update_time = fat_update_time,
};
diff --git a/fs/fat/namei_msdos.c b/fs/fat/namei_msdos.c
index 4cc65f3..0fd2971 100644
--- a/fs/fat/namei_msdos.c
+++ b/fs/fat/namei_msdos.c
@@ -644,6 +644,7 @@ static const struct inode_operations msdos_dir_inode_operations = {
.rename = msdos_rename,
.setattr = fat_setattr,
.getattr = fat_getattr,
+ .fileattr_get = fat_fileattr_get,
.update_time = fat_update_time,
};
diff --git a/fs/fat/namei_vfat.c b/fs/fat/namei_vfat.c
index 918b375..e909447 100644
--- a/fs/fat/namei_vfat.c
+++ b/fs/fat/namei_vfat.c
@@ -1185,6 +1185,7 @@ static const struct inode_operations vfat_dir_inode_operations = {
.rename = vfat_rename2,
.setattr = fat_setattr,
.getattr = fat_getattr,
+ .fileattr_get = fat_fileattr_get,
.update_time = fat_update_time,
};
diff --git a/fs/fcntl.c b/fs/fcntl.c
index beab808..c158f08 100644
--- a/fs/fcntl.c
+++ b/fs/fcntl.c
@@ -929,11 +929,11 @@ void send_sigio(struct fown_struct *fown, int fd, int band)
send_sigio_to_task(p, fown, fd, band, type);
rcu_read_unlock();
} else {
- read_lock(&tasklist_lock);
+ rcu_read_lock();
do_each_pid_task(pid, type, p) {
send_sigio_to_task(p, fown, fd, band, type);
} while_each_pid_task(pid, type, p);
- read_unlock(&tasklist_lock);
+ rcu_read_unlock();
}
out_unlock_fown:
read_unlock_irqrestore(&fown->lock, flags);
@@ -975,11 +975,11 @@ int send_sigurg(struct file *file)
send_sigurg_to_task(p, fown, type);
rcu_read_unlock();
} else {
- read_lock(&tasklist_lock);
+ rcu_read_lock();
do_each_pid_task(pid, type, p) {
send_sigurg_to_task(p, fown, type);
} while_each_pid_task(pid, type, p);
- read_unlock(&tasklist_lock);
+ rcu_read_unlock();
}
out_unlock_fown:
read_unlock_irqrestore(&fown->lock, flags);
@@ -1169,10 +1169,10 @@ static int __init fcntl_init(void)
* Exceptions: O_NONBLOCK is a two bit define on parisc; O_NDELAY
* is defined as O_NONBLOCK on some platforms and not on others.
*/
- BUILD_BUG_ON(20 - 1 /* for O_RDONLY being 0 */ !=
+ BUILD_BUG_ON(22 - 1 /* for O_RDONLY being 0 */ !=
HWEIGHT32(
(VALID_OPEN_FLAGS & ~(O_NONBLOCK | O_NDELAY)) |
- __FMODE_EXEC));
+ __FMODE_EXEC | __O_REGULAR));
fasync_cache = kmem_cache_create("fasync_cache",
sizeof(struct fasync_struct), 0,
diff --git a/fs/file.c b/fs/file.c
index 2c81c0b..628ca07 100644
--- a/fs/file.c
+++ b/fs/file.c
@@ -544,24 +544,23 @@ struct files_struct init_files = {
static unsigned int find_next_fd(struct fdtable *fdt, unsigned int start)
{
unsigned int maxfd = fdt->max_fds; /* always multiple of BITS_PER_LONG */
- unsigned int maxbit = maxfd / BITS_PER_LONG;
- unsigned int bitbit = start / BITS_PER_LONG;
+ unsigned int max_fds_words = maxfd / BITS_PER_LONG;
+ unsigned int fds_word_idx = start / BITS_PER_LONG;
unsigned int bit;
/*
* Try to avoid looking at the second level bitmap
*/
- bit = find_next_zero_bit(&fdt->open_fds[bitbit], BITS_PER_LONG,
+ bit = find_next_zero_bit(&fdt->open_fds[fds_word_idx], BITS_PER_LONG,
start & (BITS_PER_LONG - 1));
if (bit < BITS_PER_LONG)
- return bit + bitbit * BITS_PER_LONG;
+ return bit + (fds_word_idx * BITS_PER_LONG);
- bitbit = find_next_zero_bit(fdt->full_fds_bits, maxbit, bitbit) * BITS_PER_LONG;
- if (bitbit >= maxfd)
+ bit = BITS_PER_LONG *
+ find_next_zero_bit(fdt->full_fds_bits, max_fds_words, fds_word_idx + 1);
+ if (bit >= maxfd)
return maxfd;
- if (bitbit > start)
- start = bitbit;
- return find_next_zero_bit(fdt->open_fds, maxfd, start);
+ return find_next_zero_bit(fdt->open_fds, maxfd, bit);
}
/*
@@ -1134,7 +1133,6 @@ struct file *fget_task(struct task_struct *task, unsigned int fd)
struct file *fget_task_next(struct task_struct *task, unsigned int *ret_fd)
{
- /* Must be called with rcu_read_lock held */
struct files_struct *files;
unsigned int fd = *ret_fd;
struct file *file = NULL;
diff --git a/fs/file_attr.c b/fs/file_attr.c
index da983e1..bfb00d2 100644
--- a/fs/file_attr.c
+++ b/fs/file_attr.c
@@ -15,12 +15,10 @@
* @fa: fileattr pointer
* @xflags: FS_XFLAG_* flags
*
- * Set ->fsx_xflags, ->fsx_valid and ->flags (translated xflags). All
- * other fields are zeroed.
+ * Set ->fsx_xflags, ->fsx_valid and ->flags (translated xflags).
*/
void fileattr_fill_xflags(struct file_kattr *fa, u32 xflags)
{
- memset(fa, 0, sizeof(*fa));
fa->fsx_valid = true;
fa->fsx_xflags = xflags;
if (fa->fsx_xflags & FS_XFLAG_IMMUTABLE)
@@ -39,6 +37,8 @@ void fileattr_fill_xflags(struct file_kattr *fa, u32 xflags)
fa->flags |= FS_PROJINHERIT_FL;
if (fa->fsx_xflags & FS_XFLAG_VERITY)
fa->flags |= FS_VERITY_FL;
+ if (fa->fsx_xflags & FS_XFLAG_CASEFOLD)
+ fa->flags |= FS_CASEFOLD_FL;
}
EXPORT_SYMBOL(fileattr_fill_xflags);
@@ -48,11 +48,9 @@ EXPORT_SYMBOL(fileattr_fill_xflags);
* @flags: FS_*_FL flags
*
* Set ->flags, ->flags_valid and ->fsx_xflags (translated flags).
- * All other fields are zeroed.
*/
void fileattr_fill_flags(struct file_kattr *fa, u32 flags)
{
- memset(fa, 0, sizeof(*fa));
fa->flags_valid = true;
fa->flags = flags;
if (fa->flags & FS_SYNC_FL)
@@ -71,6 +69,8 @@ void fileattr_fill_flags(struct file_kattr *fa, u32 flags)
fa->fsx_xflags |= FS_XFLAG_PROJINHERIT;
if (fa->flags & FS_VERITY_FL)
fa->fsx_xflags |= FS_XFLAG_VERITY;
+ if (fa->flags & FS_CASEFOLD_FL)
+ fa->fsx_xflags |= FS_XFLAG_CASEFOLD;
}
EXPORT_SYMBOL(fileattr_fill_flags);
@@ -325,7 +325,7 @@ int ioctl_setflags(struct file *file, unsigned int __user *argp)
{
struct mnt_idmap *idmap = file_mnt_idmap(file);
struct dentry *dentry = file->f_path.dentry;
- struct file_kattr fa;
+ struct file_kattr fa = {};
unsigned int flags;
int err;
@@ -357,7 +357,7 @@ int ioctl_fssetxattr(struct file *file, void __user *argp)
{
struct mnt_idmap *idmap = file_mnt_idmap(file);
struct dentry *dentry = file->f_path.dentry;
- struct file_kattr fa;
+ struct file_kattr fa = {};
int err;
err = copy_fsxattr_from_user(&fa, argp);
@@ -431,7 +431,7 @@ SYSCALL_DEFINE5(file_setattr, int, dfd, const char __user *, filename,
struct path filepath __free(path_put) = {};
unsigned int lookup_flags = 0;
struct file_attr fattr;
- struct file_kattr fa;
+ struct file_kattr fa = {};
int error;
BUILD_BUG_ON(sizeof(struct file_attr) < FILE_ATTR_SIZE_VER0);
diff --git a/fs/file_table.c b/fs/file_table.c
index 16e52e7..c68b8c0 100644
--- a/fs/file_table.c
+++ b/fs/file_table.c
@@ -231,13 +231,13 @@ static int init_file(struct file *f, int flags, const struct cred *cred)
}
/* Find an unused file structure and return a pointer to it.
- * Returns an error pointer if some error happend e.g. we over file
+ * Returns an error pointer if some error happened, e.g., we exceed the file
* structures limit, run out of memory or operation is not permitted.
*
* Be very careful using this. You are responsible for
* getting write access to any mount that you might assign
* to this filp, if it is opened for write. If this is not
- * done, you will imbalance int the mount's writer count
+ * done, the mount's writer count will be wrong
* and a warning at __fput() time.
*/
struct file *alloc_empty_file(int flags, const struct cred *cred)
@@ -402,6 +402,8 @@ static struct file *alloc_file(const struct path *path, int flags,
static inline int alloc_path_pseudo(const char *name, struct inode *inode,
struct vfsmount *mnt, struct path *path)
{
+ if (WARN_ON_ONCE(S_ISDIR(inode->i_mode)))
+ return -EINVAL;
path->dentry = d_alloc_pseudo(mnt->mnt_sb, &QSTR(name));
if (!path->dentry)
return -ENOMEM;
diff --git a/fs/filesystems.c b/fs/filesystems.c
index 0c7d2b7..673a03b 100644
--- a/fs/filesystems.c
+++ b/fs/filesystems.c
@@ -17,22 +17,49 @@
#include <linux/slab.h>
#include <linux/uaccess.h>
#include <linux/fs_parser.h>
+#include <linux/rculist.h>
/*
- * Handling of filesystem drivers list.
- * Rules:
- * Inclusion to/removals from/scanning of list are protected by spinlock.
- * During the unload module must call unregister_filesystem().
- * We can access the fields of list element if:
- * 1) spinlock is held or
- * 2) we hold the reference to the module.
- * The latter can be guaranteed by call of try_module_get(); if it
- * returned 0 we must skip the element, otherwise we got the reference.
- * Once the reference is obtained we can drop the spinlock.
+ * Read-mostly filesystem drivers list.
+ *
+ * Readers walk under rcu_read_lock(); writers take file_systems_lock
+ * and publish via _rcu hlist primitives. unregister_filesystem()
+ * synchronize_rcu()s after unlock so the embedded file_system_type
+ * can't go away under a reader. To keep using a filesystem after
+ * the RCU section ends, take a module reference via try_module_get().
*/
+static HLIST_HEAD(file_systems);
+static DEFINE_SPINLOCK(file_systems_lock);
-static struct file_system_type *file_systems;
-static DEFINE_RWLOCK(file_systems_lock);
+#ifdef CONFIG_PROC_FS
+/*
+ * Cache a stringified version of the filesystem list.
+ *
+ * The fs list gets queried a lot by userspace because of libselinux, including
+ * rather surprising programs (would you guess *sed* is on the list?). In order
+ * to reduce the overhead we cache the resulting string, which normally hangs
+ * around below 512 bytes in size.
+ *
+ * As the list almost never changes, its creation is not particularly optimized
+ * to keep things simple.
+ *
+ * We sort it out on read in order to not introduce a failure point for fs
+ * registration (in principle we may be unable to alloc memory for the list).
+ */
+struct file_systems_string {
+ struct rcu_head rcu;
+ unsigned long gen;
+ size_t len;
+ char string[];
+};
+
+static unsigned long file_systems_gen;
+static struct file_systems_string __read_mostly __rcu *file_systems_string;
+
+static void invalidate_filesystems_string(void);
+#else
+static inline void invalidate_filesystems_string(void) { }
+#endif
/* WARNING: This can be used only if we _already_ own a reference */
struct file_system_type *get_filesystem(struct file_system_type *fs)
@@ -46,14 +73,15 @@ void put_filesystem(struct file_system_type *fs)
module_put(fs->owner);
}
-static struct file_system_type **find_filesystem(const char *name, unsigned len)
+static struct file_system_type *find_filesystem(const char *name, unsigned len)
{
- struct file_system_type **p;
- for (p = &file_systems; *p; p = &(*p)->next)
- if (strncmp((*p)->name, name, len) == 0 &&
- !(*p)->name[len])
- break;
- return p;
+ struct file_system_type *fs;
+
+ hlist_for_each_entry_rcu(fs, &file_systems, list,
+ lockdep_is_held(&file_systems_lock))
+ if (strncmp(fs->name, name, len) == 0 && !fs->name[len])
+ return fs;
+ return NULL;
}
/**
@@ -64,33 +92,27 @@ static struct file_system_type **find_filesystem(const char *name, unsigned len)
* is aware of for mount and other syscalls. Returns 0 on success,
* or a negative errno code on an error.
*
- * The &struct file_system_type that is passed is linked into the kernel
+ * The &struct file_system_type that is passed is linked into the kernel
* structures and must not be freed until the file system has been
* unregistered.
*/
-
-int register_filesystem(struct file_system_type * fs)
+int register_filesystem(struct file_system_type *fs)
{
- int res = 0;
- struct file_system_type ** p;
-
if (fs->parameters &&
!fs_validate_description(fs->name, fs->parameters))
return -EINVAL;
BUG_ON(strchr(fs->name, '.'));
- if (fs->next)
+ if (!hlist_unhashed_lockless(&fs->list))
return -EBUSY;
- write_lock(&file_systems_lock);
- p = find_filesystem(fs->name, strlen(fs->name));
- if (*p)
- res = -EBUSY;
- else
- *p = fs;
- write_unlock(&file_systems_lock);
- return res;
-}
+ guard(spinlock)(&file_systems_lock);
+ if (find_filesystem(fs->name, strlen(fs->name)))
+ return -EBUSY;
+ hlist_add_tail_rcu(&fs->list, &file_systems);
+ invalidate_filesystems_string();
+ return 0;
+}
EXPORT_SYMBOL(register_filesystem);
/**
@@ -100,94 +122,79 @@ EXPORT_SYMBOL(register_filesystem);
* Remove a file system that was previously successfully registered
* with the kernel. An error is returned if the file system is not found.
* Zero is returned on a success.
- *
+ *
* Once this function has returned the &struct file_system_type structure
* may be freed or reused.
*/
-
-int unregister_filesystem(struct file_system_type * fs)
+int unregister_filesystem(struct file_system_type *fs)
{
- struct file_system_type ** tmp;
-
- write_lock(&file_systems_lock);
- tmp = &file_systems;
- while (*tmp) {
- if (fs == *tmp) {
- *tmp = fs->next;
- fs->next = NULL;
- write_unlock(&file_systems_lock);
- synchronize_rcu();
- return 0;
- }
- tmp = &(*tmp)->next;
+ scoped_guard(spinlock, &file_systems_lock) {
+ if (hlist_unhashed(&fs->list))
+ return -EINVAL;
+ hlist_del_init_rcu(&fs->list);
+ invalidate_filesystems_string();
}
- write_unlock(&file_systems_lock);
-
- return -EINVAL;
+ synchronize_rcu();
+ return 0;
}
-
EXPORT_SYMBOL(unregister_filesystem);
#ifdef CONFIG_SYSFS_SYSCALL
-static int fs_index(const char __user * __name)
+static int fs_index(const char __user *__name)
{
- struct file_system_type * tmp;
+ struct file_system_type *p;
char *name __free(kfree) = strndup_user(__name, PATH_MAX);
- int err, index;
+ int index = 0;
if (IS_ERR(name))
return PTR_ERR(name);
- err = -EINVAL;
- read_lock(&file_systems_lock);
- for (tmp=file_systems, index=0 ; tmp ; tmp=tmp->next, index++) {
- if (strcmp(tmp->name, name) == 0) {
- err = index;
- break;
- }
+ guard(rcu)();
+ hlist_for_each_entry_rcu(p, &file_systems, list) {
+ if (strcmp(p->name, name) == 0)
+ return index;
+ index++;
}
- read_unlock(&file_systems_lock);
- return err;
+ return -EINVAL;
}
-static int fs_name(unsigned int index, char __user * buf)
+static int fs_name(unsigned int index, char __user *buf)
{
- struct file_system_type * tmp;
- int len, res = -EINVAL;
+ struct file_system_type *p, *found = NULL;
+ int len, res;
- read_lock(&file_systems_lock);
- for (tmp = file_systems; tmp; tmp = tmp->next, index--) {
- if (index == 0) {
- if (try_module_get(tmp->owner))
- res = 0;
+ scoped_guard(rcu) {
+ hlist_for_each_entry_rcu(p, &file_systems, list) {
+ if (index--)
+ continue;
+ if (try_module_get(p->owner))
+ found = p;
break;
}
}
- read_unlock(&file_systems_lock);
- if (res)
- return res;
+ if (!found)
+ return -EINVAL;
/* OK, we got the reference, so we can safely block */
- len = strlen(tmp->name) + 1;
- res = copy_to_user(buf, tmp->name, len) ? -EFAULT : 0;
- put_filesystem(tmp);
+ len = strlen(found->name) + 1;
+ res = copy_to_user(buf, found->name, len) ? -EFAULT : 0;
+ put_filesystem(found);
return res;
}
static int fs_maxindex(void)
{
- struct file_system_type * tmp;
- int index;
+ struct file_system_type *p;
+ int index = 0;
- read_lock(&file_systems_lock);
- for (tmp = file_systems, index = 0 ; tmp ; tmp = tmp->next, index++)
- ;
- read_unlock(&file_systems_lock);
+ guard(rcu)();
+ hlist_for_each_entry_rcu(p, &file_systems, list)
+ index++;
return index;
}
/*
- * Whee.. Weird sysv syscall.
+ * Whee.. Weird sysv syscall.
*/
SYSCALL_DEFINE3(sysfs, int, option, unsigned long, arg1, unsigned long, arg2)
{
@@ -216,8 +223,8 @@ int __init list_bdev_fs_names(char *buf, size_t size)
size_t len;
int count = 0;
- read_lock(&file_systems_lock);
- for (p = file_systems; p; p = p->next) {
+ guard(rcu)();
+ hlist_for_each_entry_rcu(p, &file_systems, list) {
if (!(p->fs_flags & FS_REQUIRES_DEV))
continue;
len = strlen(p->name) + 1;
@@ -230,30 +237,143 @@ int __init list_bdev_fs_names(char *buf, size_t size)
size -= len;
count++;
}
- read_unlock(&file_systems_lock);
return count;
}
#ifdef CONFIG_PROC_FS
+static void invalidate_filesystems_string(void)
+{
+ struct file_systems_string *old;
+
+ lockdep_assert_held_write(&file_systems_lock);
+ file_systems_gen++;
+ old = rcu_replace_pointer(file_systems_string, NULL,
+ lockdep_is_held(&file_systems_lock));
+ if (old)
+ kfree_rcu(old, rcu);
+}
+
+static __cold noinline int regen_filesystems_string(void)
+{
+ struct file_system_type *p;
+ struct file_systems_string *old, *new;
+ size_t newlen, usedlen;
+ unsigned long gen;
+
+retry:
+ newlen = 0;
+
+ /* pre-calc space for each fs */
+ spin_lock(&file_systems_lock);
+ gen = file_systems_gen;
+ hlist_for_each_entry_rcu(p, &file_systems, list) {
+ if (!(p->fs_flags & FS_REQUIRES_DEV))
+ newlen += strlen("nodev");
+ newlen += strlen("\t") + strlen(p->name) + strlen("\n");
+ }
+ spin_unlock(&file_systems_lock);
+
+ new = kmalloc(offsetof(struct file_systems_string, string) + newlen + 1,
+ GFP_KERNEL);
+ if (!new)
+ return -ENOMEM;
+
+ new->gen = gen;
+ new->len = newlen;
+ new->string[newlen] = '\0';
+
+ spin_lock(&file_systems_lock);
+ old = file_systems_string;
+
+ /*
+ * Did someone beat us to it?
+ */
+ if (old && old->gen == file_systems_gen) {
+ spin_unlock(&file_systems_lock);
+ kfree(new);
+ return 0;
+ }
+
+ /*
+ * Did the list change in the meantime?
+ */
+ if (gen != file_systems_gen) {
+ spin_unlock(&file_systems_lock);
+ kfree(new);
+ goto retry;
+ }
+
+ /*
+ * Populate the string.
+ *
+ * We know we have just enough space because we calculated the right
+ * size the previous time we had the lock and confirmed the list has
+ * not changed after reacquiring it.
+ */
+ usedlen = 0;
+ hlist_for_each_entry_rcu(p, &file_systems, list) {
+ usedlen += sprintf(&new->string[usedlen], "%s\t%s\n",
+ (p->fs_flags & FS_REQUIRES_DEV) ? "" : "nodev",
+ p->name);
+ }
+
+ if (WARN_ON_ONCE(new->len != strlen(new->string))) {
+ /*
+ * Should never happen of course, keep this in case someone changes string
+ * generation above and messes it up.
+ */
+ spin_unlock(&file_systems_lock);
+ kfree(new);
+ return -EINVAL;
+ }
+
+ rcu_assign_pointer(file_systems_string, new);
+ spin_unlock(&file_systems_lock);
+ if (old)
+ kfree_rcu(old, rcu);
+ return 0;
+}
+
+static __cold noinline int filesystems_proc_show_fallback(struct seq_file *m, void *v)
+{
+ struct file_system_type *p;
+
+ guard(rcu)();
+ hlist_for_each_entry_rcu(p, &file_systems, list) {
+ seq_printf(m, "%s\t%s\n",
+ (p->fs_flags & FS_REQUIRES_DEV) ? "" : "nodev",
+ p->name);
+ }
+ return 0;
+}
+
static int filesystems_proc_show(struct seq_file *m, void *v)
{
- struct file_system_type * tmp;
+ struct file_systems_string *fss;
- read_lock(&file_systems_lock);
- tmp = file_systems;
- while (tmp) {
- seq_printf(m, "%s\t%s\n",
- (tmp->fs_flags & FS_REQUIRES_DEV) ? "" : "nodev",
- tmp->name);
- tmp = tmp->next;
+ for (;;) {
+ scoped_guard(rcu) {
+ fss = rcu_dereference(file_systems_string);
+ if (likely(fss)) {
+ seq_write(m, fss->string, fss->len);
+ return 0;
+ }
+ }
+
+ int err = regen_filesystems_string();
+ if (unlikely(err))
+ return filesystems_proc_show_fallback(m, v);
}
- read_unlock(&file_systems_lock);
- return 0;
}
static int __init proc_filesystems_init(void)
{
- proc_create_single("filesystems", 0, NULL, filesystems_proc_show);
+ struct proc_dir_entry *pde;
+
+ pde = proc_create_single("filesystems", 0, NULL, filesystems_proc_show);
+ if (!pde)
+ return -ENOMEM;
+ proc_make_permanent(pde);
return 0;
}
module_init(proc_filesystems_init);
@@ -263,11 +383,10 @@ static struct file_system_type *__get_fs_type(const char *name, int len)
{
struct file_system_type *fs;
- read_lock(&file_systems_lock);
- fs = *(find_filesystem(name, len));
+ guard(rcu)();
+ fs = find_filesystem(name, len);
if (fs && !try_module_get(fs->owner))
fs = NULL;
- read_unlock(&file_systems_lock);
return fs;
}
@@ -291,5 +410,4 @@ struct file_system_type *get_fs_type(const char *name)
}
return fs;
}
-
EXPORT_SYMBOL(get_fs_type);
diff --git a/fs/fs-writeback.c b/fs/fs-writeback.c
index a65694c..fdb8766 100644
--- a/fs/fs-writeback.c
+++ b/fs/fs-writeback.c
@@ -432,6 +432,10 @@ static bool inode_do_switch_wbs(struct inode *inode,
long nr = folio_nr_pages(folio);
wb_stat_mod(old_wb, WB_RECLAIMABLE, -nr);
wb_stat_mod(new_wb, WB_RECLAIMABLE, nr);
+ if (folio_test_dropbehind(folio)) {
+ wb_stat_mod(old_wb, WB_DONTCACHE_DIRTY, -nr);
+ wb_stat_mod(new_wb, WB_DONTCACHE_DIRTY, nr);
+ }
}
}
@@ -497,6 +501,23 @@ static bool inode_do_switch_wbs(struct inode *inode,
return switched;
}
+static inline void cgroup_writeback_pin(struct super_block *sb)
+{
+ atomic_inc(&sb->s_isw_nr_in_flight);
+}
+
+static inline void cgroup_writeback_unpin(struct super_block *sb)
+{
+ if (atomic_dec_and_test(&sb->s_isw_nr_in_flight))
+ wake_up_var(&sb->s_isw_nr_in_flight);
+}
+
+static inline void cgroup_writeback_drain(struct super_block *sb)
+{
+ wait_var_event(&sb->s_isw_nr_in_flight,
+ !atomic_read(&sb->s_isw_nr_in_flight));
+}
+
static void process_inode_switch_wbs(struct bdi_writeback *new_wb,
struct inode_switch_wbs_context *isw)
{
@@ -554,8 +575,12 @@ static void process_inode_switch_wbs(struct bdi_writeback *new_wb,
wb_put_many(old_wb, nr_switched);
}
- for (inodep = isw->inodes; *inodep; inodep++)
+ for (inodep = isw->inodes; *inodep; inodep++) {
+ struct super_block *sb = (*inodep)->i_sb;
+
iput(*inodep);
+ cgroup_writeback_unpin(sb);
+ }
wb_put(new_wb);
kfree(isw);
atomic_dec(&isw_nr_in_flight);
@@ -598,16 +623,19 @@ void inode_switch_wbs_work_fn(struct work_struct *work)
static bool inode_prepare_wbs_switch(struct inode *inode,
struct bdi_writeback *new_wb)
{
+ /* Avoid the atomic_inc/smp_mb dance once SB_ACTIVE is gone. */
+ if (!(inode->i_sb->s_flags & SB_ACTIVE))
+ return false;
+
/*
- * Paired with smp_mb() in cgroup_writeback_umount().
- * isw_nr_in_flight must be increased before checking SB_ACTIVE and
- * grabbing an inode, otherwise isw_nr_in_flight can be observed as 0
- * in cgroup_writeback_umount() and the isw_wq will be not flushed.
+ * Pairs with smp_mb() in cgroup_writeback_umount(): the umounter either
+ * sees a non-zero counter and waits, or we see SB_ACTIVE clear below.
*/
+ cgroup_writeback_pin(inode->i_sb);
smp_mb();
if (IS_DAX(inode))
- return false;
+ goto out_unpin;
/* while holding I_WB_SWITCH, no one else can update the association */
spin_lock(&inode->i_lock);
@@ -615,13 +643,17 @@ static bool inode_prepare_wbs_switch(struct inode *inode,
inode_state_read(inode) & (I_WB_SWITCH | I_FREEING | I_WILL_FREE) ||
inode_to_wb(inode) == new_wb) {
spin_unlock(&inode->i_lock);
- return false;
+ goto out_unpin;
}
inode_state_set(inode, I_WB_SWITCH);
__iget(inode);
spin_unlock(&inode->i_lock);
return true;
+
+out_unpin:
+ cgroup_writeback_unpin(inode->i_sb);
+ return false;
}
static void wb_queue_isw(struct bdi_writeback *wb,
@@ -1198,36 +1230,27 @@ int cgroup_writeback_by_id(u64 bdi_id, int memcg_id,
}
/**
- * cgroup_writeback_umount - flush inode wb switches for umount
+ * cgroup_writeback_umount - wait for in-flight inode wb switches on @sb
* @sb: target super_block
*
- * This function is called when a super_block is about to be destroyed and
- * flushes in-flight inode wb switches. An inode wb switch goes through
- * RCU and then workqueue, so the two need to be flushed in order to ensure
- * that all previously scheduled switches are finished. As wb switches are
- * rare occurrences and synchronize_rcu() can take a while, perform
- * flushing iff wb switches are in flight.
+ * Wait until every inode wb switch that already passed the SB_ACTIVE
+ * check on this superblock has been completed by the worker. Since
+ * SB_ACTIVE is cleared before this is called, no new switches can start
+ * for @sb, so s_isw_nr_in_flight will monotonically drop to zero.
*/
void cgroup_writeback_umount(struct super_block *sb)
{
-
if (!(sb->s_bdi->capabilities & BDI_CAP_WRITEBACK))
return;
/*
- * SB_ACTIVE should be reliably cleared before checking
- * isw_nr_in_flight, see generic_shutdown_super().
+ * Pairs with smp_mb() in inode_prepare_wbs_switch(): we either observe
+ * a non-zero counter and wait, or the switcher sees SB_ACTIVE clear
+ * (cleared by generic_shutdown_super()) and bails before grabbing the
+ * inode.
*/
smp_mb();
-
- if (atomic_read(&isw_nr_in_flight)) {
- /*
- * Use rcu_barrier() to wait for all pending callbacks to
- * ensure that all in-flight wb switches are in the workqueue.
- */
- rcu_barrier();
- flush_workqueue(isw_wq);
- }
+ cgroup_writeback_drain(sb);
}
static int __init cgroup_writeback_init(void)
@@ -2373,6 +2396,27 @@ static long wb_check_start_all(struct bdi_writeback *wb)
return nr_pages;
}
+static long wb_check_start_dontcache(struct bdi_writeback *wb)
+{
+ long nr_pages;
+
+ if (!test_and_clear_bit(WB_start_dontcache, &wb->state))
+ return 0;
+
+ nr_pages = wb_stat_sum(wb, WB_DONTCACHE_DIRTY);
+ if (nr_pages) {
+ struct wb_writeback_work work = {
+ .nr_pages = nr_pages,
+ .sync_mode = WB_SYNC_NONE,
+ .range_cyclic = 1,
+ .reason = WB_REASON_DONTCACHE,
+ };
+
+ nr_pages = wb_writeback(wb, &work);
+ }
+
+ return nr_pages;
+}
/*
* Retrieve work items and do the writeback they describe
@@ -2395,6 +2439,11 @@ static long wb_do_writeback(struct bdi_writeback *wb)
wrote += wb_check_start_all(wb);
/*
+ * Check for dontcache writeback request
+ */
+ wrote += wb_check_start_dontcache(wb);
+
+ /*
* Check for periodic writeback, kupdated() style
*/
wrote += wb_check_old_data_flush(wb);
@@ -2468,6 +2517,43 @@ void wakeup_flusher_threads_bdi(struct backing_dev_info *bdi,
rcu_read_unlock();
}
+/**
+ * filemap_dontcache_kick_writeback - kick flusher for IOCB_DONTCACHE writes
+ * @mapping: address_space that was just written to
+ *
+ * Kick the writeback flusher thread to expedite writeback of dontcache dirty
+ * pages. Queue writeback for the inode's wb for as many pages as there are
+ * dontcache pages, but don't restrict writeback to dontcache pages only.
+ *
+ * This significantly improves performance over either writing all wb's pages
+ * or writing only dontcache pages. Although it doesn't guarantee quick
+ * writeback and reclaim of dontcache pages, it keeps the amount of dirty pages
+ * in check. Over longer term dontcache pages get written and reclaimed by
+ * background writeback even with this rough heuristic.
+ */
+void filemap_dontcache_kick_writeback(struct address_space *mapping)
+{
+ struct inode *inode = mapping->host;
+ struct bdi_writeback *wb;
+ struct wb_lock_cookie cookie = {};
+ bool need_wakeup = false;
+
+ wb = unlocked_inode_to_wb_begin(inode, &cookie);
+ if (wb_has_dirty_io(wb) &&
+ !test_bit(WB_start_dontcache, &wb->state) &&
+ !test_and_set_bit(WB_start_dontcache, &wb->state)) {
+ wb_get(wb);
+ need_wakeup = true;
+ }
+ unlocked_inode_to_wb_end(inode, &cookie);
+
+ if (need_wakeup) {
+ wb_wakeup(wb);
+ wb_put(wb);
+ }
+}
+EXPORT_SYMBOL_GPL(filemap_dontcache_kick_writeback);
+
/*
* Wakeup the flusher threads to start writeback of all currently dirty pages
*/
diff --git a/fs/fuse/dir.c b/fs/fuse/dir.c
index b658b6b..d8e8ea7 100644
--- a/fs/fuse/dir.c
+++ b/fs/fuse/dir.c
@@ -177,8 +177,8 @@ static void fuse_dentry_tree_work(struct work_struct *work)
spin_lock(&fd->dentry->d_lock);
/* If dentry is still referenced, let next dput release it */
fd->dentry->d_flags |= DCACHE_OP_DELETE;
+ __move_to_shrink_list(fd->dentry, &dispose);
spin_unlock(&fd->dentry->d_lock);
- d_dispose_if_unused(fd->dentry, &dispose);
if (need_resched()) {
spin_unlock(&dentry_hash[i].lock);
cond_resched();
diff --git a/fs/fuse/ioctl.c b/fs/fuse/ioctl.c
index fdc175e..3614ea6 100644
--- a/fs/fuse/ioctl.c
+++ b/fs/fuse/ioctl.c
@@ -10,6 +10,7 @@
#include <linux/fileattr.h>
#include <linux/fsverity.h>
+#include <linux/slab.h>
#define FUSE_VERITY_ENABLE_ARG_MAX_PAGES 256
static ssize_t fuse_send_ioctl(struct fuse_mount *fm, struct fuse_args *args,
@@ -252,7 +253,7 @@ long fuse_do_ioctl(struct file *file, unsigned int cmd, unsigned long arg,
err = -ENOMEM;
ap.folios = fuse_folios_alloc(fm->fc->max_pages, GFP_KERNEL, &ap.descs);
- iov_page = (struct iovec *) __get_free_page(GFP_KERNEL);
+ iov_page = kmalloc(PAGE_SIZE, GFP_KERNEL);
if (!ap.folios || !iov_page)
goto out;
@@ -400,7 +401,7 @@ long fuse_do_ioctl(struct file *file, unsigned int cmd, unsigned long arg,
}
err = 0;
out:
- free_page((unsigned long) iov_page);
+ kfree(iov_page);
while (ap.num_folios)
folio_put(ap.folios[--ap.num_folios]);
kfree(ap.folios);
diff --git a/fs/fuse/readdir.c b/fs/fuse/readdir.c
index db5ae8e..a2361f1 100644
--- a/fs/fuse/readdir.c
+++ b/fs/fuse/readdir.c
@@ -164,7 +164,6 @@ static int fuse_direntplus_link(struct file *file,
struct inode *dir = d_inode(parent);
struct fuse_conn *fc;
struct inode *inode;
- DECLARE_WAIT_QUEUE_HEAD_ONSTACK(wq);
int epoch;
if (!o->nodeid) {
@@ -201,7 +200,7 @@ static int fuse_direntplus_link(struct file *file,
dentry = d_lookup(parent, &name);
if (!dentry) {
retry:
- dentry = d_alloc_parallel(parent, &name, &wq);
+ dentry = d_alloc_parallel(parent, &name);
if (IS_ERR(dentry))
return PTR_ERR(dentry);
}
diff --git a/fs/gfs2/bmap.c b/fs/gfs2/bmap.c
index b3d7fcd..d158c4b 100644
--- a/fs/gfs2/bmap.c
+++ b/fs/gfs2/bmap.c
@@ -304,14 +304,15 @@ static void gfs2_metapath_ra(struct gfs2_glock *gl, __be64 *start, __be64 *end)
rabh = gfs2_getbuf(gl, be64_to_cpu(*t), CREATE);
if (trylock_buffer(rabh)) {
if (!buffer_uptodate(rabh)) {
- rabh->b_end_io = end_buffer_read_sync;
- submit_bh(REQ_OP_READ | REQ_RAHEAD | REQ_META |
- REQ_PRIO, rabh);
- continue;
+ bh_submit(rabh,
+ REQ_OP_READ | REQ_RAHEAD | REQ_META |
+ REQ_PRIO,
+ bh_end_read);
+ } else {
+ unlock_buffer(rabh);
}
- unlock_buffer(rabh);
}
- brelse(rabh);
+ put_bh(rabh);
}
}
diff --git a/fs/gfs2/dir.c b/fs/gfs2/dir.c
index 022dbb3..0237b36 100644
--- a/fs/gfs2/dir.c
+++ b/fs/gfs2/dir.c
@@ -1505,15 +1505,13 @@ static void gfs2_dir_readahead(struct inode *inode, unsigned hsize, u32 index,
if (trylock_buffer(bh)) {
if (buffer_uptodate(bh)) {
unlock_buffer(bh);
- brelse(bh);
- continue;
+ } else {
+ bh_submit(bh, REQ_OP_READ | REQ_RAHEAD |
+ REQ_META | REQ_PRIO,
+ bh_end_read);
}
- bh->b_end_io = end_buffer_read_sync;
- submit_bh(REQ_OP_READ | REQ_RAHEAD | REQ_META |
- REQ_PRIO, bh);
- continue;
}
- brelse(bh);
+ put_bh(bh);
}
}
diff --git a/fs/gfs2/inode.c b/fs/gfs2/inode.c
index e9bf4879..8a77794b 100644
--- a/fs/gfs2/inode.c
+++ b/fs/gfs2/inode.c
@@ -738,6 +738,13 @@ static int gfs2_create_inode(struct inode *dir, struct dentry *dentry,
inode = gfs2_dir_search(dir, &dentry->d_name, !S_ISREG(mode) || excl);
error = PTR_ERR(inode);
if (!IS_ERR(inode)) {
+ if (file && (file->f_flags & __O_REGULAR) &&
+ !S_ISREG(inode->i_mode)) {
+ iput(inode);
+ inode = NULL;
+ error = -EFTYPE;
+ goto fail_gunlock;
+ }
if (S_ISDIR(inode->i_mode)) {
iput(inode);
inode = NULL;
@@ -987,12 +994,8 @@ static struct dentry *__gfs2_lookup(struct inode *dir, struct dentry *dentry,
int error;
inode = gfs2_lookupi(dir, &dentry->d_name, 0);
- if (inode == NULL) {
- d_add(dentry, NULL);
- return NULL;
- }
- if (IS_ERR(inode))
- return ERR_CAST(inode);
+ if (inode == NULL || IS_ERR(inode))
+ return d_splice_alias(inode, dentry);
gl = GFS2_I(inode)->i_gl;
error = gfs2_glock_nq_init(gl, LM_ST_SHARED, LM_FLAG_ANY, &gh);
diff --git a/fs/gfs2/meta_io.c b/fs/gfs2/meta_io.c
index d407dd4..a87cfbf 100644
--- a/fs/gfs2/meta_io.c
+++ b/fs/gfs2/meta_io.c
@@ -59,7 +59,7 @@ static void gfs2_aspace_write_folio(struct folio *folio,
continue;
}
if (test_clear_buffer_dirty(bh)) {
- mark_buffer_async_write(bh);
+ set_buffer_async_write(bh);
} else {
unlock_buffer(bh);
}
@@ -75,7 +75,8 @@ static void gfs2_aspace_write_folio(struct folio *folio,
do {
struct buffer_head *next = bh->b_this_page;
if (buffer_async_write(bh)) {
- submit_bh(REQ_OP_WRITE | write_flags, bh);
+ bh_submit(bh, REQ_OP_WRITE | write_flags,
+ bh_end_async_write);
nr_underway++;
}
bh = next;
@@ -212,7 +213,7 @@ static void gfs2_meta_read_endio(struct bio *bio)
do {
struct buffer_head *next = bh->b_this_page;
len -= bh->b_size;
- bh->b_end_io(bh, !bio->bi_status);
+ end_buffer_read_sync(bh, bio->bi_status == BLK_STS_OK);
bh = next;
} while (bh && len);
}
@@ -221,7 +222,7 @@ static void gfs2_meta_read_endio(struct bio *bio)
/*
* Submit several consecutive buffer head I/O requests as a single bio I/O
- * request. (See submit_bh_wbc.)
+ * request. (See bh_submit.)
*/
static void gfs2_submit_bhs(blk_opf_t opf, struct buffer_head *bhs[], int num)
{
@@ -275,7 +276,6 @@ int gfs2_meta_read(struct gfs2_glock *gl, u64 blkno, int flags,
unlock_buffer(bh);
flags &= ~DIO_WAIT;
} else {
- bh->b_end_io = end_buffer_read_sync;
get_bh(bh);
bhs[num++] = bh;
}
@@ -286,11 +286,10 @@ int gfs2_meta_read(struct gfs2_glock *gl, u64 blkno, int flags,
lock_buffer(bh);
if (buffer_uptodate(bh)) {
unlock_buffer(bh);
- brelse(bh);
} else {
- bh->b_end_io = end_buffer_read_sync;
bhs[num++] = bh;
}
+ brelse(bh);
}
gfs2_submit_bhs(REQ_OP_READ | REQ_META | REQ_PRIO, bhs, num);
diff --git a/fs/hfs/dir.c b/fs/hfs/dir.c
index f5e7efe..c4c6e16 100644
--- a/fs/hfs/dir.c
+++ b/fs/hfs/dir.c
@@ -328,4 +328,5 @@ const struct inode_operations hfs_dir_inode_operations = {
.rmdir = hfs_remove,
.rename = hfs_rename,
.setattr = hfs_inode_setattr,
+ .fileattr_get = hfs_fileattr_get,
};
diff --git a/fs/hfs/hfs_fs.h b/fs/hfs/hfs_fs.h
index ac0e83f7..1b23448 100644
--- a/fs/hfs/hfs_fs.h
+++ b/fs/hfs/hfs_fs.h
@@ -177,6 +177,8 @@ extern int hfs_get_block(struct inode *inode, sector_t block,
extern const struct address_space_operations hfs_aops;
extern const struct address_space_operations hfs_btree_aops;
+struct file_kattr;
+int hfs_fileattr_get(struct dentry *dentry, struct file_kattr *fa);
int hfs_write_begin(const struct kiocb *iocb, struct address_space *mapping,
loff_t pos, unsigned int len, struct folio **foliop,
void **fsdata);
diff --git a/fs/hfs/inode.c b/fs/hfs/inode.c
index 89b33a9..f41cc26 100644
--- a/fs/hfs/inode.c
+++ b/fs/hfs/inode.c
@@ -18,6 +18,7 @@
#include <linux/uio.h>
#include <linux/xattr.h>
#include <linux/blkdev.h>
+#include <linux/fileattr.h>
#include "hfs_fs.h"
#include "btree.h"
@@ -699,6 +700,18 @@ static int hfs_file_fsync(struct file *filp, loff_t start, loff_t end,
return ret;
}
+int hfs_fileattr_get(struct dentry *dentry, struct file_kattr *fa)
+{
+ /*
+ * HFS compares filenames using Mac OS Roman case folding, so
+ * lookup is always case-insensitive. Names are stored on disk
+ * with case intact; CASENONPRESERVING stays clear.
+ */
+ fa->fsx_xflags |= FS_XFLAG_CASEFOLD;
+ fa->flags |= FS_CASEFOLD_FL;
+ return 0;
+}
+
static const struct file_operations hfs_file_operations = {
.llseek = generic_file_llseek,
.read_iter = generic_file_read_iter,
@@ -715,4 +728,5 @@ static const struct inode_operations hfs_file_inode_operations = {
.lookup = hfs_file_lookup,
.setattr = hfs_inode_setattr,
.listxattr = generic_listxattr,
+ .fileattr_get = hfs_fileattr_get,
};
diff --git a/fs/hfsplus/inode.c b/fs/hfsplus/inode.c
index d05891e..5565c14 100644
--- a/fs/hfsplus/inode.c
+++ b/fs/hfsplus/inode.c
@@ -740,6 +740,7 @@ int hfsplus_fileattr_get(struct dentry *dentry, struct file_kattr *fa)
{
struct inode *inode = d_inode(dentry);
struct hfsplus_inode_info *hip = HFSPLUS_I(inode);
+ struct hfsplus_sb_info *sbi = HFSPLUS_SB(inode->i_sb);
unsigned int flags = 0;
if (inode->i_flags & S_IMMUTABLE)
@@ -748,6 +749,8 @@ int hfsplus_fileattr_get(struct dentry *dentry, struct file_kattr *fa)
flags |= FS_APPEND_FL;
if (hip->userflags & HFSPLUS_FLG_NODUMP)
flags |= FS_NODUMP_FL;
+ if (test_bit(HFSPLUS_SB_CASEFOLD, &sbi->flags))
+ flags |= FS_CASEFOLD_FL;
fileattr_fill_flags(fa, flags);
@@ -759,13 +762,24 @@ int hfsplus_fileattr_set(struct mnt_idmap *idmap,
{
struct inode *inode = d_inode(dentry);
struct hfsplus_inode_info *hip = HFSPLUS_I(inode);
+ struct hfsplus_sb_info *sbi = HFSPLUS_SB(inode->i_sb);
+ unsigned int allowed = FS_IMMUTABLE_FL | FS_APPEND_FL | FS_NODUMP_FL;
unsigned int new_fl = 0;
if (fileattr_has_fsx(fa))
return -EOPNOTSUPP;
+ /*
+ * FS_CASEFOLD_FL reflects HFSPLUS_SB_CASEFOLD, a mount-time
+ * property. Accept it as a no-op so chattr's RMW round-trip
+ * succeeds; reject any attempt to enable it on a volume that
+ * was not formatted case-insensitive.
+ */
+ if (test_bit(HFSPLUS_SB_CASEFOLD, &sbi->flags))
+ allowed |= FS_CASEFOLD_FL;
+
/* don't silently ignore unsupported ext2 flags */
- if (fa->flags & ~(FS_IMMUTABLE_FL|FS_APPEND_FL|FS_NODUMP_FL))
+ if (fa->flags & ~allowed)
return -EOPNOTSUPP;
if (fa->flags & FS_IMMUTABLE_FL)
diff --git a/fs/hpfs/inode.c b/fs/hpfs/inode.c
index 0e932cc..1b4fcf7 100644
--- a/fs/hpfs/inode.c
+++ b/fs/hpfs/inode.c
@@ -184,7 +184,7 @@ void hpfs_write_inode(struct inode *i)
struct hpfs_inode_info *hpfs_inode = hpfs_i(i);
struct inode *parent;
if (i->i_ino == hpfs_sb(i->i_sb)->sb_root) return;
- if (hpfs_inode->i_rddir_off && !icount_read(i)) {
+ if (hpfs_inode->i_rddir_off && !icount_read_once(i)) {
if (*hpfs_inode->i_rddir_off)
pr_err("write_inode: some position still there\n");
kfree(hpfs_inode->i_rddir_off);
diff --git a/fs/hpfs/super.c b/fs/hpfs/super.c
index c16d5d4..8fbdbf08 100644
--- a/fs/hpfs/super.c
+++ b/fs/hpfs/super.c
@@ -523,7 +523,8 @@ static int hpfs_fill_super(struct super_block *s, struct fs_context *fc)
hpfs_lock(s);
/*sbi->sb_mounting = 1;*/
- sb_set_blocksize(s, 512);
+ if (!sb_set_blocksize(s, 512))
+ goto bail0;
sbi->sb_fs_size = -1;
if (!(bootblock = hpfs_map_sector(s, 0, &bh0, 0))) goto bail1;
if (!(superblock = hpfs_map_sector(s, 16, &bh1, 1))) goto bail2;
diff --git a/fs/inode.c b/fs/inode.c
index 62c579a..acf206b 100644
--- a/fs/inode.c
+++ b/fs/inode.c
@@ -53,11 +53,7 @@
* inode->i_lock
*
* inode_hash_lock
- * inode->i_sb->s_inode_list_lock
* inode->i_lock
- *
- * iunique_lock
- * inode_hash_lock
*/
static unsigned int i_hash_mask __ro_after_init;
@@ -518,15 +514,6 @@ static void init_once(void *foo)
inode_init_once(inode);
}
-/*
- * get additional reference to inode; caller must already hold one.
- */
-void ihold(struct inode *inode)
-{
- WARN_ON(atomic_inc_return(&inode->i_count) < 2);
-}
-EXPORT_SYMBOL(ihold);
-
struct wait_queue_head *inode_bit_waitqueue(struct wait_bit_queue_entry *wqe,
struct inode *inode, u32 bit)
{
@@ -902,7 +889,7 @@ void evict_inodes(struct super_block *sb)
again:
spin_lock(&sb->s_inode_list_lock);
list_for_each_entry(inode, &sb->s_inodes, i_sb_list) {
- if (icount_read(inode))
+ if (icount_read_once(inode))
continue;
spin_lock(&inode->i_lock);
@@ -1032,6 +1019,7 @@ long prune_icache_sb(struct super_block *sb, struct shrink_control *sc)
}
static void __wait_on_freeing_inode(struct inode *inode, bool hash_locked, bool rcu_locked);
+static bool igrab_from_hash(struct inode *inode);
/*
* Called with the inode lock held.
@@ -1056,6 +1044,11 @@ static struct inode *find_inode(struct super_block *sb,
continue;
if (!test(inode, data))
continue;
+ if (igrab_from_hash(inode)) {
+ rcu_read_unlock();
+ *isnew = false;
+ return inode;
+ }
spin_lock(&inode->i_lock);
if (inode_state_read(inode) & (I_FREEING | I_WILL_FREE)) {
__wait_on_freeing_inode(inode, hash_locked, true);
@@ -1098,6 +1091,11 @@ static struct inode *find_inode_fast(struct super_block *sb,
continue;
if (inode->i_sb != sb)
continue;
+ if (igrab_from_hash(inode)) {
+ rcu_read_unlock();
+ *isnew = false;
+ return inode;
+ }
spin_lock(&inode->i_lock);
if (inode_state_read(inode) & (I_FREEING | I_WILL_FREE)) {
__wait_on_freeing_inode(inode, hash_locked, true);
@@ -1215,6 +1213,10 @@ void unlock_new_inode(struct inode *inode)
lockdep_annotate_inode_mutex_key(inode);
spin_lock(&inode->i_lock);
WARN_ON(!(inode_state_read(inode) & I_NEW));
+ /*
+ * Paired with igrab_from_hash()
+ */
+ smp_wmb();
inode_state_clear(inode, I_NEW | I_CREATING);
inode_wake_up_bit(inode, __I_NEW);
spin_unlock(&inode->i_lock);
@@ -1226,6 +1228,10 @@ void discard_new_inode(struct inode *inode)
lockdep_annotate_inode_mutex_key(inode);
spin_lock(&inode->i_lock);
WARN_ON(!(inode_state_read(inode) & I_NEW));
+ /*
+ * Paired with igrab_from_hash()
+ */
+ smp_wmb();
inode_state_clear(inode, I_NEW);
inode_wake_up_bit(inode, __I_NEW);
spin_unlock(&inode->i_lock);
@@ -1572,8 +1578,27 @@ ino_t iunique(struct super_block *sb, ino_t max_reserved)
}
EXPORT_SYMBOL(iunique);
+/**
+ * ihold - get a reference on the inode, provided you already have one
+ * @inode: inode to operate on
+ */
+void ihold(struct inode *inode)
+{
+ VFS_BUG_ON_INODE(icount_read_once(inode) < 1, inode);
+ WARN_ON(atomic_inc_return(&inode->i_count) < 2);
+}
+EXPORT_SYMBOL(ihold);
+
struct inode *igrab(struct inode *inode)
{
+ /*
+ * Read commentary above igrab_from_hash() for an explanation why this works.
+ */
+ if (atomic_add_unless(&inode->i_count, 1, 0)) {
+ VFS_BUG_ON_INODE(inode_state_read_once(inode) & (I_FREEING | I_WILL_FREE), inode);
+ return inode;
+ }
+
spin_lock(&inode->i_lock);
if (!(inode_state_read(inode) & (I_FREEING | I_WILL_FREE))) {
__iget(inode);
@@ -1591,6 +1616,43 @@ struct inode *igrab(struct inode *inode)
}
EXPORT_SYMBOL(igrab);
+/*
+ * igrab_from_hash - special inode refcount acquire primitive for the inode hash
+ *
+ * It provides lockless refcount acquire in the common case of no problematic
+ * flags being set and the count being > 0.
+ *
+ * There are 4 state flags to worry about and the routine makes sure to not bump the
+ * ref if any of them is present.
+ *
+ * I_NEW and I_CREATING can only legally get set *before* the inode becomes visible
+ * during lookup. Thus if the flags are not spotted, they are guaranteed to not be
+ * a factor. However, we need an acquire fence before returning the inode just
+ * in case we raced against clearing the state to make sure our consumer picks up
+ * any other changes made prior. atomic_add_unless provides a full fence, which
+ * takes care of it.
+ *
+ * I_FREEING and I_WILL_FREE can only legally get set if ->i_count == 0 and it is
+ * illegal to bump the ref if either is present. Consequently if atomic_add_unless
+ * managed to replace a non-0 value with a bigger one, we have a guarantee neither
+ * of these flags is set. Note this means explicitly checking of these flags below
+ * is not necessary, it is only done because it does not cost anything on top of the
+ * load which already needs to be done to handle the other flags.
+ */
+static bool igrab_from_hash(struct inode *inode)
+{
+ if (inode_state_read_once(inode) & (I_NEW | I_CREATING | I_FREEING | I_WILL_FREE))
+ return false;
+ /*
+ * Paired with routines clearing I_NEW
+ */
+ if (atomic_add_unless(&inode->i_count, 1, 0)) {
+ VFS_BUG_ON_INODE(inode_state_read_once(inode) & (I_FREEING | I_WILL_FREE), inode);
+ return true;
+ }
+ return false;
+}
+
/**
* ilookup5_nowait - search for an inode in the inode cache
* @sb: super block of file system to search
@@ -1920,7 +1982,7 @@ static void iput_final(struct inode *inode)
int drop;
WARN_ON(inode_state_read(inode) & I_NEW);
- VFS_BUG_ON_INODE(atomic_read(&inode->i_count) != 0, inode);
+ VFS_BUG_ON_INODE(icount_read(inode) != 0, inode);
if (op->drop_inode)
drop = op->drop_inode(inode);
@@ -1939,7 +2001,7 @@ static void iput_final(struct inode *inode)
* Re-check ->i_count in case the ->drop_inode() hooks played games.
* Note we only execute this if the verdict was to drop the inode.
*/
- VFS_BUG_ON_INODE(atomic_read(&inode->i_count) != 0, inode);
+ VFS_BUG_ON_INODE(icount_read(inode) != 0, inode);
if (drop) {
inode_state_set(inode, I_FREEING);
@@ -1983,7 +2045,7 @@ void iput(struct inode *inode)
* equal to one, then two CPUs racing to further drop it can both
* conclude it's fine.
*/
- VFS_BUG_ON_INODE(atomic_read(&inode->i_count) < 1, inode);
+ VFS_BUG_ON_INODE(icount_read_once(inode) < 1, inode);
if (atomic_add_unless(&inode->i_count, -1, 1))
return;
@@ -2017,7 +2079,7 @@ EXPORT_SYMBOL(iput);
void iput_not_last(struct inode *inode)
{
VFS_BUG_ON_INODE(inode_state_read_once(inode) & (I_FREEING | I_CLEAR), inode);
- VFS_BUG_ON_INODE(atomic_read(&inode->i_count) < 2, inode);
+ VFS_BUG_ON_INODE(icount_read_once(inode) < 2, inode);
WARN_ON(atomic_sub_return(1, &inode->i_count) == 0);
}
@@ -3046,7 +3108,7 @@ void dump_inode(struct inode *inode, const char *reason)
}
state = inode_state_read_once(inode);
- count = atomic_read(&inode->i_count);
+ count = icount_read_once(inode);
if (!sb ||
get_kernel_nofault(s_type, &sb->s_type) || !s_type ||
diff --git a/fs/iomap/buffered-io.c b/fs/iomap/buffered-io.c
index d55b936..8d4806d 100644
--- a/fs/iomap/buffered-io.c
+++ b/fs/iomap/buffered-io.c
@@ -9,6 +9,7 @@
#include <linux/swap.h>
#include <linux/migrate.h>
#include <linux/fserror.h>
+#include <linux/fsverity.h>
#include "internal.h"
#include "trace.h"
@@ -353,9 +354,26 @@ static inline bool iomap_block_needs_zeroing(const struct iomap_iter *iter,
{
const struct iomap *srcmap = iomap_iter_srcmap(iter);
- return srcmap->type != IOMAP_MAPPED ||
- (srcmap->flags & IOMAP_F_NEW) ||
- pos >= i_size_read(iter->inode);
+ /*
+ * If this block has not been written, there's nothing to read
+ */
+ if (srcmap->type != IOMAP_MAPPED)
+ return true;
+
+ /*
+ * Newly allocated blocks have not been written
+ */
+ if (srcmap->flags & IOMAP_F_NEW)
+ return true;
+
+ /*
+ * fsverity metadata is stored past i_size, we need to read it instead
+ * of zeroing
+ */
+ if (srcmap->flags & IOMAP_F_FSVERITY)
+ return false;
+
+ return pos >= i_size_read(iter->inode);
}
/**
@@ -544,9 +562,27 @@ static int iomap_read_folio_iter(struct iomap_iter *iter,
if (plen == 0)
return 0;
- /* zero post-eof blocks as the page may be mapped */
- if (iomap_block_needs_zeroing(iter, pos)) {
+ /*
+ * Handling of fsverity "holes". We hit this for two case:
+ * 1. No need to go further, the hole after fsverity
+ * descriptor is the end of the fsverity metadata.
+ *
+ * 2. This folio contains merkle tree blocks which need to be
+ * synthesized. If we already have fsverity info (ctx->vi)
+ * synthesize these blocks.
+ */
+ if ((iomap->flags & IOMAP_F_FSVERITY) &&
+ iomap->type == IOMAP_HOLE) {
+ if (ctx->vi)
+ fsverity_fill_zerohash(folio, poff, plen,
+ ctx->vi);
+ iomap_set_range_uptodate(folio, poff, plen);
+ } else if (iomap_block_needs_zeroing(iter, pos)) {
+ /* zero post-eof blocks as the page may be mapped */
folio_zero_range(folio, poff, plen);
+ if (ctx->vi &&
+ !fsverity_verify_blocks(ctx->vi, folio, plen, poff))
+ return -EIO;
iomap_set_range_uptodate(folio, poff, plen);
} else {
if (!*bytes_submitted)
@@ -597,6 +633,15 @@ void iomap_read_folio(const struct iomap_ops *ops,
trace_iomap_readpage(iter.inode, 1);
+ /*
+ * Fetch fsverity_info for both data and fsverity metadata, as iomap
+ * needs zeroed hash for merkle tree block synthesis
+ */
+ ctx->vi = fsverity_get_info(iter.inode);
+ if (ctx->vi && iter.pos < i_size_read(iter.inode))
+ fsverity_readahead(ctx->vi, folio->index,
+ folio_nr_pages(folio));
+
while ((ret = iomap_iter(&iter, ops)) > 0)
iter.status = iomap_read_folio_iter(&iter, ctx,
&bytes_submitted);
@@ -664,6 +709,15 @@ void iomap_readahead(const struct iomap_ops *ops,
trace_iomap_readahead(rac->mapping->host, readahead_count(rac));
+ /*
+ * Fetch fsverity_info for both data and fsverity metadata, as iomap
+ * needs zeroed hash for merkle tree block synthesis
+ */
+ ctx->vi = fsverity_get_info(iter.inode);
+ if (ctx->vi && iter.pos < i_size_read(iter.inode))
+ fsverity_readahead(ctx->vi, readahead_index(rac),
+ readahead_count(rac));
+
while (iomap_iter(&iter, ops) > 0)
iter.status = iomap_readahead_iter(&iter, ctx,
&cur_bytes_submitted);
@@ -836,6 +890,7 @@ static int __iomap_write_begin(const struct iomap_iter *iter,
return -EIO;
folio_zero_segments(folio, poff, from, to, poff + plen);
} else {
+ const struct iomap *iomap = iomap_iter_srcmap(iter);
int status;
if (iter->flags & IOMAP_NOWAIT)
@@ -850,9 +905,12 @@ static int __iomap_write_begin(const struct iomap_iter *iter,
if (status < 0)
fserror_report_io(iter->inode,
FSERR_BUFFERED_READ, pos,
- len, status, GFP_NOFS);
+ plen, status, GFP_NOFS);
if (status)
return status;
+
+ if (iomap->flags & IOMAP_F_ZERO_TAIL)
+ folio_zero_segment(folio, to, poff + plen);
}
iomap_set_range_uptodate(folio, poff, plen);
} while ((block_start += plen) < block_end);
@@ -1058,7 +1116,6 @@ static bool iomap_write_end_inline(const struct iomap_iter *iter,
void *addr;
WARN_ON_ONCE(!folio_test_uptodate(folio));
- BUG_ON(!iomap_inline_data_valid(iomap));
if (WARN_ON_ONCE(!iomap->inline_data))
return false;
@@ -1167,13 +1224,14 @@ static int iomap_write_iter(struct iomap_iter *iter, struct iov_iter *i,
* unlock and release the folio.
*/
old_size = iter->inode->i_size;
- if (pos + written > old_size) {
+ if (pos + written > old_size &&
+ !(iter->iomap.flags & IOMAP_F_FSVERITY)) {
i_size_write(iter->inode, pos + written);
iter->iomap.flags |= IOMAP_F_SIZE_CHANGED;
}
__iomap_put_folio(iter, write_ops, written, folio);
- if (old_size < pos)
+ if (old_size < pos && !(iter->iomap.flags & IOMAP_F_FSVERITY))
pagecache_isize_extended(iter->inode, old_size, pos);
cond_resched();
@@ -1232,6 +1290,31 @@ iomap_file_buffered_write(struct kiocb *iocb, struct iov_iter *i,
}
EXPORT_SYMBOL_GPL(iomap_file_buffered_write);
+int iomap_fsverity_write(struct file *file, loff_t pos, size_t length,
+ const void *buf, const struct iomap_ops *ops,
+ const struct iomap_write_ops *write_ops)
+{
+ int ret;
+ struct iov_iter iiter;
+ struct kvec kvec = {
+ .iov_base = (void *)buf,
+ .iov_len = length,
+ };
+ struct kiocb iocb = {
+ .ki_filp = file,
+ .ki_ioprio = get_current_ioprio(),
+ .ki_pos = pos,
+ };
+
+ iov_iter_kvec(&iiter, WRITE, &kvec, 1, length);
+
+ ret = iomap_file_buffered_write(&iocb, &iiter, ops, write_ops, NULL);
+ if (ret < 0)
+ return ret;
+ return ret == length ? 0 : -EIO;
+}
+EXPORT_SYMBOL_GPL(iomap_fsverity_write);
+
static void iomap_write_delalloc_ifs_punch(struct inode *inode,
struct folio *folio, loff_t start_byte, loff_t end_byte,
struct iomap *iomap, iomap_punch_t punch)
@@ -1543,6 +1626,8 @@ static int iomap_zero_iter(struct iomap_iter *iter, bool *did_zero,
size_t offset;
bool ret;
+ balance_dirty_pages_ratelimited(iter->inode->i_mapping);
+
bytes = min_t(u64, SIZE_MAX, bytes);
status = iomap_write_begin(iter, write_ops, &folio, &offset,
&bytes);
@@ -1797,13 +1882,20 @@ static int iomap_writeback_range(struct iomap_writepage_ctx *wpc,
* Check interaction of the folio with the file end.
*
* If the folio is entirely beyond i_size, return false. If it straddles
- * i_size, adjust end_pos and zero all data beyond i_size.
+ * i_size, adjust end_pos and zero all data beyond i_size. Don't skip fsverity
+ * folios as those are beyond i_size.
*/
-static bool iomap_writeback_handle_eof(struct folio *folio, struct inode *inode,
- u64 *end_pos)
+static bool iomap_writeback_handle_eof(struct folio *folio,
+ struct iomap_writepage_ctx *wpc, u64 *end_pos)
{
+ struct inode *inode = wpc->inode;
u64 isize = i_size_read(inode);
+ if (wpc->iomap.flags & IOMAP_F_FSVERITY) {
+ WARN_ON_ONCE(folio_pos(folio) < isize);
+ return true;
+ }
+
if (*end_pos > isize) {
size_t poff = offset_in_folio(folio, isize);
pgoff_t end_index = isize >> PAGE_SHIFT;
@@ -1869,7 +1961,7 @@ int iomap_writeback_folio(struct iomap_writepage_ctx *wpc, struct folio *folio)
trace_iomap_writeback_folio(inode, pos, folio_size(folio));
- if (!iomap_writeback_handle_eof(folio, inode, &end_pos))
+ if (!iomap_writeback_handle_eof(folio, wpc, &end_pos))
return 0;
WARN_ON_ONCE(end_pos <= pos);
diff --git a/fs/iomap/direct-io.c b/fs/iomap/direct-io.c
index b36ee61..b485e3b 100644
--- a/fs/iomap/direct-io.c
+++ b/fs/iomap/direct-io.c
@@ -69,7 +69,7 @@ static void iomap_dio_submit_bio(const struct iomap_iter *iter,
/* Sync dio can't be polled reliably */
if ((iocb->ki_flags & IOCB_HIPRI) && !is_sync_kiocb(iocb)) {
- bio_set_polled(bio, iocb);
+ bio->bi_opf |= REQ_POLLED;
WRITE_ONCE(iocb->private, bio);
}
@@ -603,9 +603,6 @@ static int iomap_dio_inline_iter(struct iomap_iter *iomi, struct iomap_dio *dio)
if (WARN_ON_ONCE(!inline_data))
return -EIO;
- if (WARN_ON_ONCE(!iomap_inline_data_valid(iomap)))
- return -EIO;
-
if (dio->flags & IOMAP_DIO_WRITE) {
loff_t size = iomi->inode->i_size;
diff --git a/fs/iomap/ioend.c b/fs/iomap/ioend.c
index acf3cf9..f7c3e0c 100644
--- a/fs/iomap/ioend.c
+++ b/fs/iomap/ioend.c
@@ -28,6 +28,7 @@ struct iomap_ioend *iomap_init_ioend(struct inode *inode,
ioend->io_offset = file_offset;
ioend->io_size = bio->bi_iter.bi_size;
ioend->io_sector = bio->bi_iter.bi_sector;
+ ioend->io_vi = NULL;
ioend->io_private = NULL;
return ioend;
}
diff --git a/fs/iomap/iter.c b/fs/iomap/iter.c
index c04796f..e4a2982 100644
--- a/fs/iomap/iter.c
+++ b/fs/iomap/iter.c
@@ -6,17 +6,13 @@
#include <linux/iomap.h>
#include "trace.h"
-static inline void iomap_iter_reset_iomap(struct iomap_iter *iter)
+static inline void iomap_iter_clean_fbatch(struct iomap_iter *iter)
{
if (iter->iomap.flags & IOMAP_F_FOLIO_BATCH) {
folio_batch_release(iter->fbatch);
folio_batch_reinit(iter->fbatch);
iter->iomap.flags &= ~IOMAP_F_FOLIO_BATCH;
}
-
- iter->status = 0;
- memset(&iter->iomap, 0, sizeof(iter->iomap));
- memset(&iter->srcmap, 0, sizeof(iter->srcmap));
}
/* Advance the current iterator position and decrement the remaining length */
@@ -102,10 +98,14 @@ int iomap_iter(struct iomap_iter *iter, const struct iomap_ops *ops)
ret = 0;
else
ret = 1;
- iomap_iter_reset_iomap(iter);
+ iomap_iter_clean_fbatch(iter);
+ iter->status = 0;
if (ret <= 0)
return ret;
+ memset(&iter->iomap, 0, sizeof(iter->iomap));
+ memset(&iter->srcmap, 0, sizeof(iter->srcmap));
+
begin:
ret = ops->iomap_begin(iter->inode, iter->pos, iter->len, iter->flags,
&iter->iomap, &iter->srcmap);
diff --git a/fs/iomap/trace.h b/fs/iomap/trace.h
index 097773c..e1ea839 100644
--- a/fs/iomap/trace.h
+++ b/fs/iomap/trace.h
@@ -118,7 +118,9 @@ DEFINE_RANGE_EVENT(iomap_zero_iter);
{ IOMAP_F_ATOMIC_BIO, "ATOMIC_BIO" }, \
{ IOMAP_F_PRIVATE, "PRIVATE" }, \
{ IOMAP_F_SIZE_CHANGED, "SIZE_CHANGED" }, \
- { IOMAP_F_STALE, "STALE" }
+ { IOMAP_F_STALE, "STALE" }, \
+ { IOMAP_F_FSVERITY, "FSVERITY" }, \
+ { IOMAP_F_ZERO_TAIL, "ZERO TAIL" }
#define IOMAP_DIO_STRINGS \
diff --git a/fs/isofs/dir.c b/fs/isofs/dir.c
index 2fd9948..cc587cd 100644
--- a/fs/isofs/dir.c
+++ b/fs/isofs/dir.c
@@ -13,7 +13,9 @@
*/
#include <linux/gfp.h>
#include <linux/filelock.h>
+#include <linux/slab.h>
#include "isofs.h"
+#include <linux/fileattr.h>
int isofs_name_translate(struct iso_directory_record *de, char *new, struct inode *inode)
{
@@ -255,7 +257,7 @@ static int isofs_readdir(struct file *file, struct dir_context *ctx)
struct iso_directory_record *tmpde;
struct inode *inode = file_inode(file);
- tmpname = (char *)__get_free_page(GFP_KERNEL);
+ tmpname = kmalloc(PAGE_SIZE, GFP_KERNEL);
if (tmpname == NULL)
return -ENOMEM;
@@ -263,10 +265,24 @@ static int isofs_readdir(struct file *file, struct dir_context *ctx)
result = do_isofs_readdir(inode, file, ctx, tmpname, tmpde);
- free_page((unsigned long) tmpname);
+ kfree(tmpname);
return result;
}
+int isofs_fileattr_get(struct dentry *dentry, struct file_kattr *fa)
+{
+ struct isofs_sb_info *sbi = ISOFS_SB(dentry->d_sb);
+
+ if (sbi->s_check == 'r') {
+ fa->fsx_xflags |= FS_XFLAG_CASEFOLD;
+ fa->flags |= FS_CASEFOLD_FL;
+ }
+ if (!sbi->s_joliet_level && !sbi->s_rock &&
+ (sbi->s_mapping == 'n' || sbi->s_mapping == 'a'))
+ fa->fsx_xflags |= FS_XFLAG_CASENONPRESERVING;
+ return 0;
+}
+
const struct file_operations isofs_dir_operations =
{
.llseek = generic_file_llseek,
@@ -281,6 +297,7 @@ const struct file_operations isofs_dir_operations =
const struct inode_operations isofs_dir_inode_operations =
{
.lookup = isofs_lookup,
+ .fileattr_get = isofs_fileattr_get,
};
diff --git a/fs/isofs/inode.c b/fs/isofs/inode.c
index efee537..337836a 100644
--- a/fs/isofs/inode.c
+++ b/fs/isofs/inode.c
@@ -818,7 +818,8 @@ static int isofs_fill_super(struct super_block *s, struct fs_context *fc)
* entries. By forcing the blocksize in this way, we ensure
* that we will never be required to do this.
*/
- sb_set_blocksize(s, orig_zonesize);
+ if (!sb_set_blocksize(s, orig_zonesize))
+ goto out_freesbi;
sbi->s_nls_iocharset = NULL;
diff --git a/fs/isofs/isofs.h b/fs/isofs/isofs.h
index 5065558..0ec8b24 100644
--- a/fs/isofs/isofs.h
+++ b/fs/isofs/isofs.h
@@ -197,6 +197,9 @@ isofs_normalize_block_and_offset(struct iso_directory_record* de,
}
}
+struct file_kattr;
+int isofs_fileattr_get(struct dentry *dentry, struct file_kattr *fa);
+
extern const struct inode_operations isofs_dir_inode_operations;
extern const struct file_operations isofs_dir_operations;
extern const struct address_space_operations isofs_symlink_aops;
diff --git a/fs/jbd2/commit.c b/fs/jbd2/commit.c
index 8cf61e7..d857772 100644
--- a/fs/jbd2/commit.c
+++ b/fs/jbd2/commit.c
@@ -29,8 +29,10 @@
/*
* IO end handler for temporary buffer_heads handling writes to the journal.
*/
-static void journal_end_buffer_io_sync(struct buffer_head *bh, int uptodate)
+static void journal_end_buffer_io_sync(struct bio *bio)
{
+ struct buffer_head *bh;
+ bool uptodate = bio_endio_bh(bio, &bh);
struct buffer_head *orig_bh = bh->b_private;
BUFFER_TRACE(bh, "");
@@ -39,9 +41,7 @@ static void journal_end_buffer_io_sync(struct buffer_head *bh, int uptodate)
else
clear_buffer_uptodate(bh);
if (orig_bh) {
- clear_bit_unlock(BH_Shadow, &orig_bh->b_state);
- smp_mb__after_atomic();
- wake_up_bit(&orig_bh->b_state, BH_Shadow);
+ clear_and_wake_up_bit(BH_Shadow, &orig_bh->b_state);
}
unlock_buffer(bh);
}
@@ -147,13 +147,12 @@ static int journal_submit_commit_record(journal_t *journal,
lock_buffer(bh);
clear_buffer_dirty(bh);
set_buffer_uptodate(bh);
- bh->b_end_io = journal_end_buffer_io_sync;
if (journal->j_flags & JBD2_BARRIER &&
!jbd2_has_feature_async_commit(journal))
write_flags |= REQ_PREFLUSH | REQ_FUA;
- submit_bh(write_flags, bh);
+ bh_submit(bh, write_flags, journal_end_buffer_io_sync);
*cbh = bh;
return 0;
}
@@ -751,9 +750,9 @@ void jbd2_journal_commit_transaction(journal_t *journal)
lock_buffer(bh);
clear_buffer_dirty(bh);
set_buffer_uptodate(bh);
- bh->b_end_io = journal_end_buffer_io_sync;
- submit_bh(REQ_OP_WRITE | JBD2_JOURNAL_REQ_FLAGS,
- bh);
+ bh_submit(bh,
+ REQ_OP_WRITE | JBD2_JOURNAL_REQ_FLAGS,
+ journal_end_buffer_io_sync);
}
cond_resched();
diff --git a/fs/jbd2/journal.c b/fs/jbd2/journal.c
index 4f397fc..e827986 100644
--- a/fs/jbd2/journal.c
+++ b/fs/jbd2/journal.c
@@ -1820,9 +1820,7 @@ static int jbd2_write_superblock(journal_t *journal, blk_opf_t write_flags)
}
if (jbd2_journal_has_csum_v2or3(journal))
sb->s_checksum = jbd2_superblock_csum(sb);
- get_bh(bh);
- bh->b_end_io = end_buffer_write_sync;
- submit_bh(REQ_OP_WRITE | write_flags, bh);
+ bh_submit(bh, REQ_OP_WRITE | write_flags, bh_end_write);
wait_on_buffer(bh);
if (buffer_write_io_error(bh)) {
clear_buffer_write_io_error(bh);
@@ -2784,7 +2782,7 @@ void *jbd2_alloc(size_t size, gfp_t flags)
if (size < PAGE_SIZE)
ptr = kmem_cache_alloc(get_slab(size), flags);
else
- ptr = (void *)__get_free_pages(flags, get_order(size));
+ ptr = kmalloc(size, flags);
/* Check alignment; SLUB has gotten this wrong in the past,
* and this can lead to user data corruption! */
@@ -2795,10 +2793,7 @@ void *jbd2_alloc(size_t size, gfp_t flags)
void jbd2_free(void *ptr, size_t size)
{
- if (size < PAGE_SIZE)
- kmem_cache_free(get_slab(size), ptr);
- else
- free_pages((unsigned long)ptr, get_order(size));
+ kfree(ptr);
};
/*
diff --git a/fs/jfs/jfs_dtree.c b/fs/jfs/jfs_dtree.c
index ac0f79f..8ce6e44 100644
--- a/fs/jfs/jfs_dtree.c
+++ b/fs/jfs/jfs_dtree.c
@@ -2729,7 +2729,7 @@ int jfs_readdir(struct file *file, struct dir_context *ctx)
struct ldtentry *d;
struct dtslot *t;
int d_namleft, len, outlen;
- unsigned long dirent_buf;
+ void *dirent_buf;
char *name_ptr;
u32 dir_index;
int do_index = 0;
@@ -2884,7 +2884,7 @@ int jfs_readdir(struct file *file, struct dir_context *ctx)
}
}
- dirent_buf = __get_free_page(GFP_KERNEL);
+ dirent_buf = kmalloc(PAGE_SIZE, GFP_KERNEL);
if (dirent_buf == 0) {
DT_PUTPAGE(mp);
jfs_warn("jfs_readdir: __get_free_page failed!");
@@ -2893,7 +2893,7 @@ int jfs_readdir(struct file *file, struct dir_context *ctx)
}
while (1) {
- jfs_dirent = (struct jfs_dirent *) dirent_buf;
+ jfs_dirent = dirent_buf;
jfs_dirents = 0;
overflow = fix_page = 0;
@@ -2903,7 +2903,7 @@ int jfs_readdir(struct file *file, struct dir_context *ctx)
if (stbl[i] < 0) {
jfs_err("JFS: Invalid stbl[%d] = %d for inode %ld, block = %lld",
i, stbl[i], (long)ip->i_ino, (long long)bn);
- free_page(dirent_buf);
+ kfree(dirent_buf);
DT_PUTPAGE(mp);
return -EIO;
}
@@ -2911,7 +2911,7 @@ int jfs_readdir(struct file *file, struct dir_context *ctx)
d = (struct ldtentry *) & p->slot[stbl[i]];
if (((long) jfs_dirent + d->namlen + 1) >
- (dirent_buf + PAGE_SIZE)) {
+ ((long)dirent_buf + PAGE_SIZE)) {
/* DBCS codepages could overrun dirent_buf */
index = i;
overflow = 1;
@@ -3014,7 +3014,7 @@ int jfs_readdir(struct file *file, struct dir_context *ctx)
/* unpin previous leaf page */
DT_PUTPAGE(mp);
- jfs_dirent = (struct jfs_dirent *) dirent_buf;
+ jfs_dirent = dirent_buf;
while (jfs_dirents--) {
ctx->pos = jfs_dirent->position;
if (!dir_emit(ctx, jfs_dirent->name,
@@ -3037,13 +3037,13 @@ int jfs_readdir(struct file *file, struct dir_context *ctx)
DT_GETPAGE(ip, bn, mp, PSIZE, p, rc);
if (rc) {
- free_page(dirent_buf);
+ kfree(dirent_buf);
return rc;
}
}
out:
- free_page(dirent_buf);
+ kfree(dirent_buf);
return rc;
}
diff --git a/fs/jfs/super.c b/fs/jfs/super.c
index 61575f7..8180d83 100644
--- a/fs/jfs/super.c
+++ b/fs/jfs/super.c
@@ -491,7 +491,8 @@ static int jfs_fill_super(struct super_block *sb, struct fs_context *fc)
/*
* Initialize blocksize to 4K.
*/
- sb_set_blocksize(sb, PSIZE);
+ if (!sb_set_blocksize(sb, PSIZE))
+ goto out_unload;
/*
* Set method vectors.
diff --git a/fs/kernfs/dir.c b/fs/kernfs/dir.c
index 4f9ade8..6d47b84 100644
--- a/fs/kernfs/dir.c
+++ b/fs/kernfs/dir.c
@@ -605,11 +605,8 @@ void kernfs_put(struct kernfs_node *kn)
if (kernfs_type(kn) == KERNFS_LINK)
kernfs_put(kn->symlink.target_kn);
- if (kn->iattr && kn->iattr->xattrs) {
- simple_xattrs_free(kn->iattr->xattrs, NULL);
- kfree(kn->iattr->xattrs);
- kn->iattr->xattrs = NULL;
- }
+ if (kn->iattr)
+ simple_xattrs_free(&root->xa_cache, &kn->iattr->xattrs, NULL);
spin_lock(&root->kernfs_idr_lock);
idr_remove(&root->ino_idr, (u32)kernfs_ino(kn));
@@ -624,6 +621,7 @@ void kernfs_put(struct kernfs_node *kn)
} else {
/* just released the root kn, free @root too */
idr_destroy(&root->ino_idr);
+ simple_xattr_cache_cleanup(&root->xa_cache);
kfree_rcu(root, rcu);
}
}
@@ -700,6 +698,9 @@ static struct kernfs_node *__kernfs_new_node(struct kernfs_root *root,
}
if (parent) {
+ kernfs_get(parent);
+ rcu_assign_pointer(kn->__parent, parent);
+
ret = security_kernfs_init_security(parent, kn);
if (ret)
goto err_out4;
@@ -708,11 +709,10 @@ static struct kernfs_node *__kernfs_new_node(struct kernfs_root *root,
return kn;
err_out4:
+ RCU_INIT_POINTER(kn->__parent, NULL);
+ kernfs_put(parent);
if (kn->iattr) {
- if (kn->iattr->xattrs) {
- simple_xattrs_free(kn->iattr->xattrs, NULL);
- kfree(kn->iattr->xattrs);
- }
+ simple_xattrs_free(&root->xa_cache, &kn->iattr->xattrs, NULL);
kmem_cache_free(kernfs_iattrs_cache, kn->iattr);
}
err_out3:
@@ -747,10 +747,6 @@ struct kernfs_node *kernfs_new_node(struct kernfs_node *parent,
kn = __kernfs_new_node(kernfs_root(parent), parent,
name, mode, uid, gid, flags);
- if (kn) {
- kernfs_get(parent);
- rcu_assign_pointer(kn->__parent, parent);
- }
return kn;
}
diff --git a/fs/kernfs/file.c b/fs/kernfs/file.c
index 1163aa7..8e0e90c 100644
--- a/fs/kernfs/file.c
+++ b/fs/kernfs/file.c
@@ -40,22 +40,15 @@ struct kernfs_open_node {
static DEFINE_SPINLOCK(kernfs_notify_lock);
static struct kernfs_node *kernfs_notify_list = KERNFS_NOTIFY_EOL;
+/* Compatibility wrappers - use the common hashed node lock */
static inline struct mutex *kernfs_open_file_mutex_ptr(struct kernfs_node *kn)
{
- int idx = hash_ptr(kn, NR_KERNFS_LOCK_BITS);
-
- return &kernfs_locks->open_file_mutex[idx];
+ return kernfs_node_lock_ptr(kn);
}
static inline struct mutex *kernfs_open_file_mutex_lock(struct kernfs_node *kn)
{
- struct mutex *lock;
-
- lock = kernfs_open_file_mutex_ptr(kn);
-
- mutex_lock(lock);
-
- return lock;
+ return kernfs_node_lock(kn);
}
/**
diff --git a/fs/kernfs/inode.c b/fs/kernfs/inode.c
index 38b28aa..2cb2029 100644
--- a/fs/kernfs/inode.c
+++ b/fs/kernfs/inode.c
@@ -37,6 +37,7 @@ static struct kernfs_iattrs *__kernfs_iattrs(struct kernfs_node *kn, bool alloc)
if (!ret)
return NULL;
+ INIT_LIST_HEAD_RCU(&ret->xattrs);
/* assign default attributes */
ret->ia_uid = GLOBAL_ROOT_UID;
ret->ia_gid = GLOBAL_ROOT_GID;
@@ -144,8 +145,7 @@ ssize_t kernfs_iop_listxattr(struct dentry *dentry, char *buf, size_t size)
if (!attrs)
return -ENOMEM;
- return simple_xattr_list(d_inode(dentry), READ_ONCE(attrs->xattrs),
- buf, size);
+ return simple_xattr_list(d_inode(dentry), &attrs->xattrs, buf, size);
}
static inline void set_default_inode_attr(struct inode *inode, umode_t mode)
@@ -297,34 +297,35 @@ int kernfs_xattr_get(struct kernfs_node *kn, const char *name,
void *value, size_t size)
{
struct kernfs_iattrs *attrs = kernfs_iattrs_noalloc(kn);
- struct simple_xattrs *xattrs;
+ struct simple_xattr_cache *cache = &kernfs_root(kn)->xa_cache;
if (!attrs)
return -ENODATA;
- xattrs = READ_ONCE(attrs->xattrs);
- if (!xattrs)
- return -ENODATA;
-
- return simple_xattr_get(xattrs, name, value, size);
+ return simple_xattr_get(cache, &attrs->xattrs, name, value, size);
}
int kernfs_xattr_set(struct kernfs_node *kn, const char *name,
const void *value, size_t size, int flags)
{
struct simple_xattr *old_xattr;
- struct simple_xattrs *xattrs;
struct kernfs_iattrs *attrs;
+ struct simple_xattr_cache *cache = &kernfs_root(kn)->xa_cache;
attrs = kernfs_iattrs(kn);
if (!attrs)
return -ENOMEM;
- xattrs = simple_xattrs_lazy_alloc(&attrs->xattrs, value, flags);
- if (IS_ERR_OR_NULL(xattrs))
- return PTR_ERR(xattrs);
+ /*
+ * Protect xattr modifications with the hashed per-node mutex.
+ * Multiple superblocks (with different namespaces) can share the same
+ * kernfs_node, so inode locking alone is insufficient. The hashed mutex
+ * ensures serialization of concurrent xattr operations on the same node,
+ * including the lazy allocation of the xattrs structure itself.
+ */
+ CLASS(kernfs_node_lock, lock)(kn);
- old_xattr = simple_xattr_set(xattrs, name, value, size, flags);
+ old_xattr = simple_xattr_set(cache, &attrs->xattrs, name, value, size, flags);
if (IS_ERR(old_xattr))
return PTR_ERR(old_xattr);
@@ -362,7 +363,6 @@ static int kernfs_vfs_user_xattr_set(const struct xattr_handler *handler,
{
const char *full_name = xattr_full_name(handler, suffix);
struct kernfs_node *kn = inode->i_private;
- struct simple_xattrs *xattrs;
struct kernfs_iattrs *attrs;
if (!(kernfs_root(kn)->flags & KERNFS_ROOT_SUPPORT_USER_XATTR))
@@ -372,11 +372,11 @@ static int kernfs_vfs_user_xattr_set(const struct xattr_handler *handler,
if (!attrs)
return -ENOMEM;
- xattrs = simple_xattrs_lazy_alloc(&attrs->xattrs, value, flags);
- if (IS_ERR_OR_NULL(xattrs))
- return PTR_ERR(xattrs);
+ /* See comment in kernfs_xattr_set() about locking. */
+ CLASS(kernfs_node_lock, lock)(kn);
- return simple_xattr_set_limited(xattrs, &attrs->xattr_limits,
+ return simple_xattr_set_limited(&kernfs_root(kn)->xa_cache,
+ &attrs->xattrs, &attrs->xattr_limits,
full_name, value, size, flags);
}
diff --git a/fs/kernfs/kernfs-internal.h b/fs/kernfs/kernfs-internal.h
index 8d8912f..aa784b5 100644
--- a/fs/kernfs/kernfs-internal.h
+++ b/fs/kernfs/kernfs-internal.h
@@ -26,7 +26,7 @@ struct kernfs_iattrs {
struct timespec64 ia_mtime;
struct timespec64 ia_ctime;
- struct simple_xattrs *xattrs;
+ struct list_head xattrs;
struct simple_xattr_limits xattr_limits;
};
@@ -54,6 +54,8 @@ struct kernfs_root {
rwlock_t kernfs_rename_lock;
struct rcu_head rcu;
+
+ struct simple_xattr_cache xa_cache;
};
/* +1 to avoid triggering overflow warning when negating it */
@@ -211,4 +213,24 @@ extern const struct inode_operations kernfs_symlink_iops;
* kernfs locks
*/
extern struct kernfs_global_locks *kernfs_locks;
+
+/* Hashed mutex helpers - protect per-node data structures */
+static inline struct mutex *kernfs_node_lock_ptr(struct kernfs_node *kn)
+{
+ int idx = hash_ptr(kn, NR_KERNFS_LOCK_BITS);
+
+ return &kernfs_locks->node_mutex[idx];
+}
+
+static inline struct mutex *kernfs_node_lock(struct kernfs_node *kn)
+{
+ struct mutex *lock = kernfs_node_lock_ptr(kn);
+
+ mutex_lock(lock);
+ return lock;
+}
+
+DEFINE_CLASS(kernfs_node_lock, struct mutex *,
+ mutex_unlock(_T), kernfs_node_lock(kn), struct kernfs_node *kn)
+
#endif /* __KERNFS_INTERNAL_H */
diff --git a/fs/kernfs/mount.c b/fs/kernfs/mount.c
index 6e3217b..f183a96 100644
--- a/fs/kernfs/mount.c
+++ b/fs/kernfs/mount.c
@@ -446,7 +446,7 @@ static void __init kernfs_mutex_init(void)
int count;
for (count = 0; count < NR_KERNFS_LOCKS; count++)
- mutex_init(&kernfs_locks->open_file_mutex[count]);
+ mutex_init(&kernfs_locks->node_mutex[count]);
}
static void __init kernfs_lock_init(void)
diff --git a/fs/libfs.c b/fs/libfs.c
index 1bbea5e..5a0d276 100644
--- a/fs/libfs.c
+++ b/fs/libfs.c
@@ -78,8 +78,7 @@ struct dentry *simple_lookup(struct inode *dir, struct dentry *dentry, unsigned
if (IS_ENABLED(CONFIG_UNICODE) && IS_CASEFOLDED(dir))
return NULL;
- d_add(dentry, NULL);
- return NULL;
+ return d_splice_alias(NULL, dentry);
}
EXPORT_SYMBOL(simple_lookup);
@@ -736,6 +735,7 @@ struct pseudo_fs_context *init_pseudo(struct fs_context *fc,
fc->fs_private = ctx;
fc->ops = &pseudo_fs_context_ops;
fc->sb_flags |= SB_NOUSER;
+ fc->s_iflags |= SB_I_NOEXEC | SB_I_NODEV;
fc->global = true;
}
return ctx;
@@ -1258,7 +1258,7 @@ char *simple_transaction_get(struct file *file, const char __user *buf, size_t s
if (size > SIMPLE_TRANSACTION_LIMIT - 1)
return ERR_PTR(-EFBIG);
- ar = (struct simple_transaction_argresp *)get_zeroed_page(GFP_KERNEL);
+ ar = kzalloc(PAGE_SIZE, GFP_KERNEL);
if (!ar)
return ERR_PTR(-ENOMEM);
@@ -1267,7 +1267,7 @@ char *simple_transaction_get(struct file *file, const char __user *buf, size_t s
/* only one write allowed per open */
if (file->private_data) {
spin_unlock(&simple_transaction_lock);
- free_page((unsigned long)ar);
+ kfree(ar);
return ERR_PTR(-EBUSY);
}
@@ -1294,7 +1294,7 @@ EXPORT_SYMBOL(simple_transaction_read);
int simple_transaction_release(struct inode *inode, struct file *file)
{
- free_page((unsigned long)file->private_data);
+ kfree(file->private_data);
return 0;
}
EXPORT_SYMBOL(simple_transaction_release);
diff --git a/fs/locks.c b/fs/locks.c
index fead534..6e4ff7f 100644
--- a/fs/locks.c
+++ b/fs/locks.c
@@ -1582,30 +1582,96 @@ static bool leases_conflict(struct file_lock_core *lc, struct file_lock_core *bc
return rc;
}
-static bool
-any_leases_conflict(struct inode *inode, struct file_lease *breaker)
+#define IGNORE_MASK (FL_IGN_DIR_CREATE | FL_IGN_DIR_DELETE | FL_IGN_DIR_RENAME)
+
+/**
+ * inode_lease_ignore_mask - return union of all ignored inode events for this inode
+ * @inode: inode of which to get ignore mask
+ *
+ * Walk the list of leases, and return the result of all of
+ * their FL_IGN_DIR_* bits or'ed together.
+ */
+u32
+inode_lease_ignore_mask(struct inode *inode)
{
- struct file_lock_context *ctx = inode->i_flctx;
+ struct file_lock_context *ctx;
struct file_lock_core *flc;
+ u32 mask = 0;
- lockdep_assert_held(&ctx->flc_lock);
+ ctx = locks_inode_context(inode);
+ if (!ctx)
+ return 0;
+ spin_lock(&ctx->flc_lock);
list_for_each_entry(flc, &ctx->flc_lease, flc_list) {
- if (leases_conflict(flc, &breaker->c))
- return true;
+ mask |= flc->flc_flags & IGNORE_MASK;
+ /* If we already have everything, we can stop */
+ if (mask == IGNORE_MASK)
+ break;
}
+ spin_unlock(&ctx->flc_lock);
+ return mask;
+}
+EXPORT_SYMBOL_GPL(inode_lease_ignore_mask);
+
+static bool
+ignore_dir_deleg_break(struct file_lease *fl, unsigned int flags)
+{
+ if ((flags & LEASE_BREAK_DIR_CREATE) && (fl->c.flc_flags & FL_IGN_DIR_CREATE))
+ return true;
+ if ((flags & LEASE_BREAK_DIR_DELETE) && (fl->c.flc_flags & FL_IGN_DIR_DELETE))
+ return true;
+ if ((flags & LEASE_BREAK_DIR_RENAME) && (fl->c.flc_flags & FL_IGN_DIR_RENAME))
+ return true;
+
return false;
}
+static unsigned int
+break_lease_flags_to_type(unsigned int flags)
+{
+ if (flags & LEASE_BREAK_LEASE)
+ return FL_LEASE;
+ else if (flags & LEASE_BREAK_DELEG)
+ return FL_DELEG;
+ else if (flags & LEASE_BREAK_LAYOUT)
+ return FL_LAYOUT;
+ else
+ return 0;
+
+}
+
+static struct file_lease *
+first_visible_lease(struct inode *inode, struct file_lease *new_fl, unsigned int flags)
+{
+ struct file_lock_context *ctx = locks_inode_context(inode);
+ struct file_lease *fl;
+
+ lockdep_assert_held(&ctx->flc_lock);
+
+ list_for_each_entry(fl, &ctx->flc_lease, c.flc_list) {
+ if (!leases_conflict(&fl->c, &new_fl->c))
+ continue;
+ if (S_ISDIR(inode->i_mode) && ignore_dir_deleg_break(fl, flags))
+ continue;
+ return fl;
+ }
+ return NULL;
+}
+
+
/**
- * __break_lease - revoke all outstanding leases on file
- * @inode: the inode of the file to return
- * @flags: LEASE_BREAK_* flags
+ * __break_lease - revoke all outstanding leases on file
+ * @inode: the inode of the file to return
+ * @flags: LEASE_BREAK_* flags
*
- * break_lease (inlined for speed) has checked there already is at least
- * some kind of lock (maybe a lease) on this file. Leases are broken on
- * a call to open() or truncate(). This function can block waiting for the
- * lease break unless you specify LEASE_BREAK_NONBLOCK.
+ * break_lease (inlined for speed) has checked there already is at least
+ * some kind of lock (maybe a lease) on this file. Leases and Delegations
+ * are broken on a call to open() or truncate(). Delegations are also
+ * broken on any event that would change the ctime. Directory delegations
+ * are broken whenever the directory changes (unless the delegation is set
+ * up to ignore the event). This function can block waiting for the lease
+ * break unless you specify LEASE_BREAK_NONBLOCK.
*/
int __break_lease(struct inode *inode, unsigned int flags)
{
@@ -1617,13 +1683,10 @@ int __break_lease(struct inode *inode, unsigned int flags)
bool want_write = !(flags & LEASE_BREAK_OPEN_RDONLY);
int error = 0;
- if (flags & LEASE_BREAK_LEASE)
- type = FL_LEASE;
- else if (flags & LEASE_BREAK_DELEG)
- type = FL_DELEG;
- else if (flags & LEASE_BREAK_LAYOUT)
- type = FL_LAYOUT;
- else
+ trace_break_lease(inode, flags);
+
+ type = break_lease_flags_to_type(flags);
+ if (!type)
return -EINVAL;
new_fl = lease_alloc(NULL, type, want_write ? F_WRLCK : F_RDLCK);
@@ -1642,7 +1705,7 @@ int __break_lease(struct inode *inode, unsigned int flags)
time_out_leases(inode, &dispose);
- if (!any_leases_conflict(inode, new_fl))
+ if (!first_visible_lease(inode, new_fl, flags))
goto out;
break_time = 0;
@@ -1655,6 +1718,8 @@ int __break_lease(struct inode *inode, unsigned int flags)
list_for_each_entry_safe(fl, tmp, &ctx->flc_lease, c.flc_list) {
if (!leases_conflict(&fl->c, &new_fl->c))
continue;
+ if (S_ISDIR(inode->i_mode) && ignore_dir_deleg_break(fl, flags))
+ continue;
if (want_write) {
if (fl->c.flc_flags & FL_UNLOCK_PENDING)
continue;
@@ -1670,7 +1735,8 @@ int __break_lease(struct inode *inode, unsigned int flags)
locks_delete_lock_ctx(&fl->c, &dispose);
}
- if (list_empty(&ctx->flc_lease))
+ fl = first_visible_lease(inode, new_fl, flags);
+ if (!fl)
goto out;
if (flags & LEASE_BREAK_NONBLOCK) {
@@ -1680,7 +1746,6 @@ int __break_lease(struct inode *inode, unsigned int flags)
}
restart:
- fl = list_first_entry(&ctx->flc_lease, struct file_lease, c.flc_list);
break_time = fl->fl_break_time;
if (break_time != 0) {
if (time_after(jiffies, break_time)) {
@@ -1691,7 +1756,7 @@ int __break_lease(struct inode *inode, unsigned int flags)
} else
break_time++;
locks_insert_block(&fl->c, &new_fl->c, leases_conflict);
- trace_break_lease_block(inode, new_fl);
+ trace_break_lease_block(inode, fl);
spin_unlock(&ctx->flc_lock);
percpu_up_read(&file_rwsem);
@@ -1702,7 +1767,7 @@ int __break_lease(struct inode *inode, unsigned int flags)
percpu_down_read(&file_rwsem);
spin_lock(&ctx->flc_lock);
- trace_break_lease_unblock(inode, new_fl);
+ trace_break_lease_unblock(inode, NULL);
__locks_delete_block(&new_fl->c);
if (error >= 0) {
/*
@@ -1711,7 +1776,8 @@ int __break_lease(struct inode *inode, unsigned int flags)
*/
if (error == 0)
time_out_leases(inode, &dispose);
- if (any_leases_conflict(inode, new_fl))
+ fl = first_visible_lease(inode, new_fl, flags);
+ if (fl)
goto restart;
error = 0;
}
diff --git a/fs/minix/inode.c b/fs/minix/inode.c
index 9c6bac2..c30cc590 100644
--- a/fs/minix/inode.c
+++ b/fs/minix/inode.c
@@ -292,7 +292,8 @@ static int minix_fill_super(struct super_block *s, struct fs_context *fc)
sbi->s_namelen = 60;
sbi->s_version = MINIX_V3;
sbi->s_mount_state = MINIX_VALID_FS;
- sb_set_blocksize(s, m3s->s_blocksize);
+ if (!sb_set_blocksize(s, m3s->s_blocksize))
+ goto out_release;
s->s_max_links = MINIX2_LINK_MAX;
} else
goto out_no_fs;
diff --git a/fs/mount.h b/fs/mount.h
index 5c120f8..94fcc30 100644
--- a/fs/mount.h
+++ b/fs/mount.h
@@ -25,6 +25,7 @@ struct mnt_namespace {
__u32 n_fsnotify_mask;
struct fsnotify_mark_connector __rcu *n_fsnotify_marks;
#endif
+ struct hlist_head mnt_visible_mounts; /* SB_I_USERNS_VISIBLE mounts */
unsigned int nr_mounts; /* # of mounts in the namespace */
unsigned int pending_mounts;
refcount_t passive; /* number references not pinning @mounts */
@@ -98,6 +99,7 @@ struct mount {
int mnt_expiry_mark; /* true if marked for expiry */
struct hlist_head mnt_pins;
struct hlist_head mnt_stuck_children;
+ struct hlist_node mnt_ns_visible; /* link in ns->mnt_visible_mounts */
struct mount *overmount; /* mounted on ->mnt_root */
} __randomize_layout;
@@ -215,6 +217,8 @@ static inline void move_from_ns(struct mount *mnt)
ns->mnt_first_node = rb_next(&mnt->mnt_node);
rb_erase(&mnt->mnt_node, &ns->mounts);
RB_CLEAR_NODE(&mnt->mnt_node);
+ if (!hlist_unhashed(&mnt->mnt_ns_visible))
+ hlist_del_init(&mnt->mnt_ns_visible);
}
bool has_locked_children(struct mount *mnt, struct dentry *dentry);
diff --git a/fs/namei.c b/fs/namei.c
index 4787244..5cc9f0f 100644
--- a/fs/namei.c
+++ b/fs/namei.c
@@ -129,6 +129,11 @@
static struct kmem_cache *__names_cache __ro_after_init;
#define names_cache runtime_const_ptr(__names_cache)
+/*
+ * Type of the last component on LOOKUP_PARENT
+ */
+enum last_type {LAST_NORM, LAST_ROOT, LAST_DOT, LAST_DOTDOT};
+
void __init filename_init(void)
{
__names_cache = kmem_cache_create_usercopy("names_cache", sizeof(struct filename), 0,
@@ -727,7 +732,7 @@ struct nameidata {
struct inode *inode; /* path.dentry.d_inode */
unsigned int flags, state;
unsigned seq, next_seq, m_seq, r_seq;
- int last_type;
+ enum last_type last_type;
unsigned depth;
int total_link_count;
struct saved {
@@ -1891,13 +1896,12 @@ static struct dentry *__lookup_slow(const struct qstr *name,
{
struct dentry *dentry, *old;
struct inode *inode = dir->d_inode;
- DECLARE_WAIT_QUEUE_HEAD_ONSTACK(wq);
/* Don't go there if it's already dead */
if (unlikely(IS_DEADDIR(inode)))
return ERR_PTR(-ENOENT);
again:
- dentry = d_alloc_parallel(dir, name, &wq);
+ dentry = d_alloc_parallel(dir, name);
if (IS_ERR(dentry))
return dentry;
if (unlikely(!d_in_lookup(dentry))) {
@@ -2220,7 +2224,7 @@ static struct dentry *follow_dotdot(struct nameidata *nd)
return dget(nd->path.dentry);
}
-static const char *handle_dots(struct nameidata *nd, int type)
+static const char *handle_dots(struct nameidata *nd, enum last_type type)
{
if (type == LAST_DOTDOT) {
const char *error = NULL;
@@ -2868,7 +2872,7 @@ static int path_parentat(struct nameidata *nd, unsigned flags,
/* Note: this does not consume "name" */
static int __filename_parentat(int dfd, struct filename *name,
unsigned int flags, struct path *parent,
- struct qstr *last, int *type,
+ struct qstr *last, enum last_type *type,
const struct path *root)
{
int retval;
@@ -2893,7 +2897,7 @@ static int __filename_parentat(int dfd, struct filename *name,
static int filename_parentat(int dfd, struct filename *name,
unsigned int flags, struct path *parent,
- struct qstr *last, int *type)
+ struct qstr *last, enum last_type *type)
{
return __filename_parentat(dfd, name, flags, parent, last, type, NULL);
}
@@ -2955,15 +2959,17 @@ void end_dirop(struct dentry *de)
EXPORT_SYMBOL(end_dirop);
/* does lookup, returns the object with parent locked */
-static struct dentry *__start_removing_path(int dfd, struct filename *name,
- struct path *path)
+struct dentry *start_removing_path(const char *name, struct path *path)
{
+ CLASS(filename_kernel, filename)(name);
struct path parent_path __free(path_put) = {};
struct dentry *d;
struct qstr last;
- int type, error;
+ enum last_type type;
+ int error;
- error = filename_parentat(dfd, name, 0, &parent_path, &last, &type);
+ error = filename_parentat(AT_FDCWD, filename, 0, &parent_path, &last,
+ &type);
if (error)
return ERR_PTR(error);
if (unlikely(type != LAST_NORM))
@@ -3007,7 +3013,8 @@ struct dentry *kern_path_parent(const char *name, struct path *path)
CLASS(filename_kernel, filename)(name);
struct dentry *d;
struct qstr last;
- int type, error;
+ enum last_type type;
+ int error;
error = filename_parentat(AT_FDCWD, filename, 0, &parent_path, &last, &type);
if (error)
@@ -3023,21 +3030,6 @@ struct dentry *kern_path_parent(const char *name, struct path *path)
return d;
}
-struct dentry *start_removing_path(const char *name, struct path *path)
-{
- CLASS(filename_kernel, filename)(name);
- return __start_removing_path(AT_FDCWD, filename, path);
-}
-
-struct dentry *start_removing_user_path_at(int dfd,
- const char __user *name,
- struct path *path)
-{
- CLASS(filename, filename)(name);
- return __start_removing_path(dfd, filename, path);
-}
-EXPORT_SYMBOL(start_removing_user_path_at);
-
int kern_path(const char *name, unsigned int flags, struct path *path)
{
CLASS(filename_kernel, filename)(name);
@@ -3051,15 +3043,22 @@ EXPORT_SYMBOL(kern_path);
* @flags: lookup flags
* @parent: pointer to struct path to fill
* @last: last component
- * @type: type of the last component
* @root: pointer to struct path of the base directory
*/
int vfs_path_parent_lookup(struct filename *filename, unsigned int flags,
- struct path *parent, struct qstr *last, int *type,
+ struct path *parent, struct qstr *last,
const struct path *root)
{
- return __filename_parentat(AT_FDCWD, filename, flags, parent, last,
- type, root);
+ enum last_type type;
+ int err = __filename_parentat(AT_FDCWD, filename, flags, parent, last,
+ &type, root);
+ if (err)
+ return err;
+ if (unlikely(type != LAST_NORM)) {
+ path_put(parent);
+ return -EINVAL;
+ }
+ return 0;
}
EXPORT_SYMBOL(vfs_path_parent_lookup);
@@ -3617,7 +3616,6 @@ int path_pts(struct path *path)
*/
struct dentry *parent = dget_parent(path->dentry);
struct dentry *child;
- struct qstr this = QSTR_INIT("pts", 3);
if (unlikely(!path_connected(path->mnt, parent))) {
dput(parent);
@@ -3625,7 +3623,7 @@ int path_pts(struct path *path)
}
dput(path->dentry);
path->dentry = parent;
- child = d_hash_and_lookup(parent, &this);
+ child = d_hash_and_lookup(parent, &QSTR("pts"));
if (IS_ERR_OR_NULL(child))
return -ENOENT;
@@ -4198,7 +4196,7 @@ int vfs_create(struct mnt_idmap *idmap, struct dentry *dentry, umode_t mode,
error = security_inode_create(dir, dentry, mode);
if (error)
return error;
- error = try_break_deleg(dir, di);
+ error = try_break_deleg(dir, LEASE_BREAK_DIR_CREATE, di);
if (error)
return error;
error = dir->i_op->create(idmap, dir, dentry, mode, true);
@@ -4414,7 +4412,6 @@ static struct dentry *lookup_open(struct nameidata *nd, struct file *file,
struct dentry *dentry;
int error, create_error = 0;
umode_t mode = op->mode;
- DECLARE_WAIT_QUEUE_HEAD_ONSTACK(wq);
if (unlikely(IS_DEADDIR(dir_inode)))
return ERR_PTR(-ENOENT);
@@ -4423,7 +4420,7 @@ static struct dentry *lookup_open(struct nameidata *nd, struct file *file,
dentry = d_lookup(dir, &nd->last);
for (;;) {
if (!dentry) {
- dentry = d_alloc_parallel(dir, &nd->last, &wq);
+ dentry = d_alloc_parallel(dir, &nd->last);
if (IS_ERR(dentry))
return dentry;
}
@@ -4497,7 +4494,7 @@ static struct dentry *lookup_open(struct nameidata *nd, struct file *file,
/* Negative dentry, just create the file */
if (!dentry->d_inode && (open_flag & O_CREAT)) {
/* but break the directory lease first! */
- error = try_break_deleg(dir_inode, delegated_inode);
+ error = try_break_deleg(dir_inode, LEASE_BREAK_DIR_CREATE, delegated_inode);
if (error)
goto out_dput;
@@ -4679,6 +4676,10 @@ static int do_open(struct nameidata *nd,
if (unlikely(error))
return error;
}
+
+ if ((open_flag & __O_REGULAR) && !d_is_reg(nd->path.dentry))
+ return -EFTYPE;
+
if ((nd->flags & LOOKUP_DIRECTORY) && !d_can_lookup(nd->path.dentry))
return -ENOTDIR;
@@ -4925,7 +4926,7 @@ static struct dentry *filename_create(int dfd, struct filename *name,
bool want_dir = lookup_flags & LOOKUP_DIRECTORY;
unsigned int reval_flag = lookup_flags & LOOKUP_REVAL;
unsigned int create_flags = LOOKUP_CREATE | LOOKUP_EXCL;
- int type;
+ enum last_type type;
int error;
error = filename_parentat(dfd, name, reval_flag, path, &last, &type);
@@ -5123,7 +5124,7 @@ int vfs_mknod(struct mnt_idmap *idmap, struct inode *dir,
if (error)
return error;
- error = try_break_deleg(dir, delegated_inode);
+ error = try_break_deleg(dir, LEASE_BREAK_DIR_CREATE, delegated_inode);
if (error)
return error;
@@ -5264,7 +5265,7 @@ struct dentry *vfs_mkdir(struct mnt_idmap *idmap, struct inode *dir,
if (max_links && dir->i_nlink >= max_links)
goto err;
- error = try_break_deleg(dir, delegated_inode);
+ error = try_break_deleg(dir, LEASE_BREAK_DIR_CREATE, delegated_inode);
if (error)
goto err;
@@ -5369,7 +5370,7 @@ int vfs_rmdir(struct mnt_idmap *idmap, struct inode *dir,
if (error)
goto out;
- error = try_break_deleg(dir, delegated_inode);
+ error = try_break_deleg(dir, LEASE_BREAK_DIR_DELETE, delegated_inode);
if (error)
goto out;
@@ -5397,7 +5398,7 @@ int filename_rmdir(int dfd, struct filename *name)
struct dentry *dentry;
struct path path;
struct qstr last;
- int type;
+ enum last_type type;
unsigned int lookup_flags = 0;
struct delegated_inode delegated_inode = { };
retry:
@@ -5406,6 +5407,8 @@ int filename_rmdir(int dfd, struct filename *name)
return error;
switch (type) {
+ case LAST_NORM:
+ break;
case LAST_DOTDOT:
error = -ENOTEMPTY;
goto exit2;
@@ -5499,10 +5502,10 @@ int vfs_unlink(struct mnt_idmap *idmap, struct inode *dir,
else {
error = security_inode_unlink(dir, dentry);
if (!error) {
- error = try_break_deleg(dir, delegated_inode);
+ error = try_break_deleg(dir, LEASE_BREAK_DIR_DELETE, delegated_inode);
if (error)
goto out;
- error = try_break_deleg(target, delegated_inode);
+ error = try_break_deleg(target, 0, delegated_inode);
if (error)
goto out;
error = dir->i_op->unlink(dir, dentry);
@@ -5539,7 +5542,7 @@ int filename_unlinkat(int dfd, struct filename *name)
struct dentry *dentry;
struct path path;
struct qstr last;
- int type;
+ enum last_type type;
struct inode *inode;
struct delegated_inode delegated_inode = { };
unsigned int lookup_flags = 0;
@@ -5646,7 +5649,7 @@ int vfs_symlink(struct mnt_idmap *idmap, struct inode *dir,
if (error)
return error;
- error = try_break_deleg(dir, delegated_inode);
+ error = try_break_deleg(dir, LEASE_BREAK_DIR_CREATE, delegated_inode);
if (error)
return error;
@@ -5777,9 +5780,9 @@ int vfs_link(struct dentry *old_dentry, struct mnt_idmap *idmap,
else if (max_links && inode->i_nlink >= max_links)
error = -EMLINK;
else {
- error = try_break_deleg(dir, delegated_inode);
+ error = try_break_deleg(dir, LEASE_BREAK_DIR_CREATE, delegated_inode);
if (!error)
- error = try_break_deleg(inode, delegated_inode);
+ error = try_break_deleg(inode, 0, delegated_inode);
if (!error)
error = dir->i_op->link(old_dentry, dir, new_dentry);
}
@@ -6043,21 +6046,24 @@ int vfs_rename(struct renamedata *rd)
old_dir->i_nlink >= max_links)
goto out;
}
- error = try_break_deleg(old_dir, delegated_inode);
+ error = try_break_deleg(old_dir,
+ old_dir == new_dir ? LEASE_BREAK_DIR_RENAME :
+ LEASE_BREAK_DIR_DELETE,
+ delegated_inode);
if (error)
goto out;
if (new_dir != old_dir) {
- error = try_break_deleg(new_dir, delegated_inode);
+ error = try_break_deleg(new_dir, LEASE_BREAK_DIR_CREATE, delegated_inode);
if (error)
goto out;
}
if (!is_dir) {
- error = try_break_deleg(source, delegated_inode);
+ error = try_break_deleg(source, 0, delegated_inode);
if (error)
goto out;
}
if (target && !new_is_dir) {
- error = try_break_deleg(target, delegated_inode);
+ error = try_break_deleg(target, 0, delegated_inode);
if (error)
goto out;
}
@@ -6106,7 +6112,7 @@ int filename_renameat2(int olddfd, struct filename *from,
struct renamedata rd;
struct path old_path, new_path;
struct qstr old_last, new_last;
- int old_type, new_type;
+ enum last_type old_type, new_type;
struct delegated_inode delegated_inode = { };
unsigned int lookup_flags = 0;
bool should_retry = false;
diff --git a/fs/namespace.c b/fs/namespace.c
index 341ddd3..3d5cd5b 100644
--- a/fs/namespace.c
+++ b/fs/namespace.c
@@ -321,6 +321,7 @@ static struct mount *alloc_vfsmnt(const char *name)
INIT_HLIST_NODE(&mnt->mnt_slave);
INIT_HLIST_NODE(&mnt->mnt_mp_list);
INIT_HLIST_HEAD(&mnt->mnt_stuck_children);
+ INIT_HLIST_NODE(&mnt->mnt_ns_visible);
RB_CLEAR_NODE(&mnt->mnt_node);
mnt->mnt.mnt_idmap = &nop_mnt_idmap;
}
@@ -1098,6 +1099,10 @@ static void mnt_add_to_ns(struct mnt_namespace *ns, struct mount *mnt)
rb_link_node(&mnt->mnt_node, parent, link);
rb_insert_color(&mnt->mnt_node, &ns->mounts);
+ if ((mnt->mnt.mnt_sb->s_type->fs_flags & FS_USERNS_MOUNT_RESTRICTED) &&
+ mnt->mnt.mnt_root == mnt->mnt.mnt_sb->s_root)
+ hlist_add_head(&mnt->mnt_ns_visible, &ns->mnt_visible_mounts);
+
mnt_notify_add(mnt);
}
@@ -3306,9 +3311,9 @@ static void mnt_warn_timestamp_expiry(const struct path *mountpoint,
(ktime_get_real_seconds() + TIME_UPTIME_SEC_MAX > sb->s_time_max)) {
char *buf, *mntpath;
- buf = (char *)__get_free_page(GFP_KERNEL);
+ buf = __getname();
if (buf)
- mntpath = d_path(mountpoint, buf, PAGE_SIZE);
+ mntpath = d_path(mountpoint, buf, PATH_MAX);
else
mntpath = ERR_PTR(-ENOMEM);
if (IS_ERR(mntpath))
@@ -3321,8 +3326,7 @@ static void mnt_warn_timestamp_expiry(const struct path *mountpoint,
(unsigned long long)sb->s_time_max);
sb->s_iflags |= SB_I_TS_EXPIRY_WARNED;
- if (buf)
- free_page((unsigned long)buf);
+ __putname(buf);
}
}
@@ -4502,6 +4506,10 @@ SYSCALL_DEFINE3(fsmount, int, fs_fd, unsigned int, flags,
new_mnt = vfs_create_mount(fc);
if (IS_ERR(new_mnt))
return PTR_ERR(new_mnt);
+ if (new_mnt->mnt_sb->s_flags & SB_NOUSER) {
+ mntput(new_mnt);
+ return -EINVAL;
+ }
new_mnt->mnt_flags = mnt_flags;
new_path.dentry = dget(fc->root);
@@ -6343,20 +6351,26 @@ static bool mnt_already_visible(struct mnt_namespace *ns,
int *new_mnt_flags)
{
int new_flags = *new_mnt_flags;
- struct mount *mnt, *n;
+ struct mount *mnt;
+
+ /* Don't acquire namespace semaphore without a good reason. */
+ if (hlist_empty(&ns->mnt_visible_mounts))
+ return false;
guard(namespace_shared)();
- rbtree_postorder_for_each_entry_safe(mnt, n, &ns->mounts, mnt_node) {
+ hlist_for_each_entry(mnt, &ns->mnt_visible_mounts, mnt_ns_visible) {
+ const struct super_block *sb_visible = mnt->mnt.mnt_sb;
struct mount *child;
int mnt_flags;
- if (mnt->mnt.mnt_sb->s_type != sb->s_type)
+ if (sb_visible->s_type != sb->s_type)
continue;
- /* This mount is not fully visible if it's root directory
- * is not the root directory of the filesystem.
+ /*
+ * Restricted variants are not compatible with anything, even
+ * other restricted variants.
*/
- if (mnt->mnt.mnt_root != mnt->mnt.mnt_sb->s_root)
+ if (sb_visible->s_iflags & SB_I_RESTRICTED_VARIANT)
continue;
/* A local view of the mount flags */
@@ -6408,16 +6422,23 @@ static bool mount_too_revealing(const struct super_block *sb, int *new_mnt_flags
return false;
/* Can this filesystem be too revealing? */
- s_iflags = sb->s_iflags;
- if (!(s_iflags & SB_I_USERNS_VISIBLE))
+ if (!(sb->s_type->fs_flags & FS_USERNS_MOUNT_RESTRICTED))
return false;
+ s_iflags = sb->s_iflags;
if ((s_iflags & required_iflags) != required_iflags) {
WARN_ONCE(1, "Expected s_iflags to contain 0x%lx\n",
required_iflags);
return true;
}
+ /*
+ * Restricted variants don't need an already visible mount because they
+ * don't expose the full filesystem view.
+ */
+ if (s_iflags & SB_I_RESTRICTED_VARIANT)
+ return false;
+
return !mnt_already_visible(ns, sb, new_mnt_flags);
}
diff --git a/fs/nfs/client.c b/fs/nfs/client.c
index be02bb2..73b9531 100644
--- a/fs/nfs/client.c
+++ b/fs/nfs/client.c
@@ -914,6 +914,7 @@ static void nfs_server_set_fsinfo(struct nfs_server *server,
*/
static int nfs_probe_fsinfo(struct nfs_server *server, struct nfs_fh *mntfh, struct nfs_fattr *fattr)
{
+ struct nfs_pathconf pathinfo = { };
struct nfs_fsinfo fsinfo;
struct nfs_client *clp = server->nfs_client;
int error;
@@ -933,15 +934,28 @@ static int nfs_probe_fsinfo(struct nfs_server *server, struct nfs_fh *mntfh, str
nfs_server_set_fsinfo(server, &fsinfo);
- /* Get some general file system info */
- if (server->namelen == 0) {
- struct nfs_pathconf pathinfo;
+ pathinfo.fattr = fattr;
+ nfs_fattr_init(fattr);
- pathinfo.fattr = fattr;
- nfs_fattr_init(fattr);
+ if (clp->rpc_ops->version < 4 || server->namelen == 0) {
+ if (clp->rpc_ops->pathconf(server, mntfh, &pathinfo) >= 0) {
+ if (server->namelen == 0)
+ server->namelen = pathinfo.max_namelen;
+ if (clp->rpc_ops->version < 4) {
+ unsigned int caps = server->caps;
- if (clp->rpc_ops->pathconf(server, mntfh, &pathinfo) >= 0)
- server->namelen = pathinfo.max_namelen;
+ caps &= ~(NFS_CAP_CASE_INSENSITIVE |
+ NFS_CAP_CASE_NONPRESERVING);
+ if (pathinfo.case_insensitive)
+ caps |= NFS_CAP_CASE_INSENSITIVE;
+ if (!pathinfo.case_preserving)
+ caps |= NFS_CAP_CASE_NONPRESERVING;
+ server->caps = caps;
+ }
+ } else if (clp->rpc_ops->version < 4) {
+ server->caps &= ~(NFS_CAP_CASE_INSENSITIVE |
+ NFS_CAP_CASE_NONPRESERVING);
+ }
}
if (clp->rpc_ops->discover_trunking != NULL &&
diff --git a/fs/nfs/dir.c b/fs/nfs/dir.c
index e9ce188..2f5f26f 100644
--- a/fs/nfs/dir.c
+++ b/fs/nfs/dir.c
@@ -726,7 +726,6 @@ void nfs_prime_dcache(struct dentry *parent, struct nfs_entry *entry,
unsigned long dir_verifier)
{
struct qstr filename = QSTR_INIT(entry->name, entry->len);
- DECLARE_WAIT_QUEUE_HEAD_ONSTACK(wq);
struct dentry *dentry;
struct dentry *alias;
struct inode *inode;
@@ -755,7 +754,7 @@ void nfs_prime_dcache(struct dentry *parent, struct nfs_entry *entry,
dentry = d_lookup(parent, &filename);
again:
if (!dentry) {
- dentry = d_alloc_parallel(parent, &filename, &wq);
+ dentry = d_alloc_parallel(parent, &filename);
if (IS_ERR(dentry))
return;
}
@@ -2106,7 +2105,6 @@ int nfs_atomic_open(struct inode *dir, struct dentry *dentry,
struct file *file, unsigned open_flags,
umode_t mode)
{
- DECLARE_WAIT_QUEUE_HEAD_ONSTACK(wq);
struct nfs_open_context *ctx;
struct dentry *res;
struct iattr attr = { .ia_valid = ATTR_OPEN };
@@ -2162,7 +2160,7 @@ int nfs_atomic_open(struct inode *dir, struct dentry *dentry,
d_drop(dentry);
switched = true;
dentry = d_alloc_parallel(dentry->d_parent,
- &dentry->d_name, &wq);
+ &dentry->d_name);
if (IS_ERR(dentry))
return PTR_ERR(dentry);
if (unlikely(!d_in_lookup(dentry)))
@@ -2194,6 +2192,10 @@ int nfs_atomic_open(struct inode *dir, struct dentry *dentry,
break;
case -EISDIR:
case -ENOTDIR:
+ if (open_flags & __O_REGULAR) {
+ err = -EFTYPE;
+ break;
+ }
goto no_open;
case -ELOOP:
if (!(open_flags & O_NOFOLLOW))
diff --git a/fs/nfs/fs_context.c b/fs/nfs/fs_context.c
index c105882..1967de7 100644
--- a/fs/nfs/fs_context.c
+++ b/fs/nfs/fs_context.c
@@ -1769,7 +1769,9 @@ struct file_system_type nfs_fs_type = {
.init_fs_context = nfs_init_fs_context,
.parameters = nfs_fs_parameters,
.kill_sb = nfs_kill_super,
- .fs_flags = FS_RENAME_DOES_D_MOVE|FS_BINARY_MOUNTDATA,
+ .fs_flags = FS_RENAME_DOES_D_MOVE |
+ FS_BINARY_MOUNTDATA |
+ FS_USERNS_DELEGATABLE,
};
MODULE_ALIAS_FS("nfs");
EXPORT_SYMBOL_GPL(nfs_fs_type);
@@ -1781,7 +1783,9 @@ struct file_system_type nfs4_fs_type = {
.init_fs_context = nfs_init_fs_context,
.parameters = nfs_fs_parameters,
.kill_sb = nfs_kill_super,
- .fs_flags = FS_RENAME_DOES_D_MOVE|FS_BINARY_MOUNTDATA,
+ .fs_flags = FS_RENAME_DOES_D_MOVE |
+ FS_BINARY_MOUNTDATA |
+ FS_USERNS_DELEGATABLE,
};
MODULE_ALIAS_FS("nfs4");
MODULE_ALIAS("nfs4");
diff --git a/fs/nfs/getroot.c b/fs/nfs/getroot.c
index eef0736..ff7424b 100644
--- a/fs/nfs/getroot.c
+++ b/fs/nfs/getroot.c
@@ -33,35 +33,6 @@
#define NFSDBG_FACILITY NFSDBG_CLIENT
/*
- * Set the superblock root dentry.
- * Note that this function frees the inode in case of error.
- */
-static int nfs_superblock_set_dummy_root(struct super_block *sb, struct inode *inode)
-{
- /* The mntroot acts as the dummy root dentry for this superblock */
- if (sb->s_root == NULL) {
- sb->s_root = d_make_root(inode);
- if (sb->s_root == NULL)
- return -ENOMEM;
- ihold(inode);
- /*
- * Ensure that this dentry is invisible to d_find_alias().
- * Otherwise, it may be spliced into the tree by
- * d_splice_alias if a parent directory from the same
- * filesystem gets mounted at a later time.
- * This again causes shrink_dcache_for_umount_subtree() to
- * Oops, since the test for IS_ROOT() will fail.
- */
- spin_lock(&d_inode(sb->s_root)->i_lock);
- spin_lock(&sb->s_root->d_lock);
- hlist_del_init(&sb->s_root->d_alias);
- spin_unlock(&sb->s_root->d_lock);
- spin_unlock(&d_inode(sb->s_root)->i_lock);
- }
- return 0;
-}
-
-/*
* get a root dentry from the root filehandle
*/
int nfs_get_root(struct super_block *s, struct fs_context *fc)
@@ -99,10 +70,6 @@ int nfs_get_root(struct super_block *s, struct fs_context *fc)
goto out_fattr;
}
- error = nfs_superblock_set_dummy_root(s, inode);
- if (error != 0)
- goto out_fattr;
-
/* root dentries normally start off anonymous and get spliced in later
* if the dentry tree reaches them; however if the dentry already
* exists, we'll pick it up at this point and use it as the root
@@ -123,6 +90,8 @@ int nfs_get_root(struct super_block *s, struct fs_context *fc)
name = NULL;
}
spin_unlock(&root->d_lock);
+ if (!s->s_root)
+ s->s_root = dget(root);
fc->root = root;
if (server->caps & NFS_CAP_SECURITY_LABEL)
kflags |= SECURITY_LSM_NATIVE_LABELS;
diff --git a/fs/nfs/inode.c b/fs/nfs/inode.c
index e26030e..6227df9 100644
--- a/fs/nfs/inode.c
+++ b/fs/nfs/inode.c
@@ -41,6 +41,7 @@
#include <linux/freezer.h>
#include <linux/uaccess.h>
#include <linux/iversion.h>
+#include <linux/fileattr.h>
#include "nfs4_fs.h"
#include "callback.h"
@@ -608,7 +609,7 @@ nfs_fhget(struct super_block *sb, struct nfs_fh *fh, struct nfs_fattr *fattr)
inode->i_sb->s_id,
(unsigned long long)NFS_FILEID(inode),
nfs_display_fhandle_hash(fh),
- icount_read(inode));
+ icount_read_once(inode));
out:
return inode;
@@ -1095,6 +1096,20 @@ int nfs_getattr(struct mnt_idmap *idmap, const struct path *path,
}
EXPORT_SYMBOL_GPL(nfs_getattr);
+int nfs_fileattr_get(struct dentry *dentry, struct file_kattr *fa)
+{
+ struct inode *inode = d_inode(dentry);
+
+ if (nfs_server_capable(inode, NFS_CAP_CASE_INSENSITIVE)) {
+ fa->fsx_xflags |= FS_XFLAG_CASEFOLD;
+ fa->flags |= FS_CASEFOLD_FL;
+ }
+ if (nfs_server_capable(inode, NFS_CAP_CASE_NONPRESERVING))
+ fa->fsx_xflags |= FS_XFLAG_CASENONPRESERVING;
+ return 0;
+}
+EXPORT_SYMBOL_GPL(nfs_fileattr_get);
+
static void nfs_init_lock_context(struct nfs_lock_context *l_ctx)
{
refcount_set(&l_ctx->count, 1);
@@ -2255,7 +2270,7 @@ static int nfs_update_inode(struct inode *inode, struct nfs_fattr *fattr)
dfprintk(VFS, "NFS: %s(%s/%llu fh_crc=0x%08x ct=%d info=0x%llx)\n",
__func__, inode->i_sb->s_id, inode->i_ino,
nfs_display_fhandle_hash(NFS_FH(inode)),
- icount_read(inode), fattr->valid);
+ icount_read_once(inode), fattr->valid);
if (!(fattr->valid & NFS_ATTR_FATTR_FILEID)) {
/* Only a mounted-on-fileid? Just exit */
diff --git a/fs/nfs/internal.h b/fs/nfs/internal.h
index 18d46b0..ec2b3d9 100644
--- a/fs/nfs/internal.h
+++ b/fs/nfs/internal.h
@@ -451,6 +451,9 @@ extern void nfs_set_cache_invalid(struct inode *inode, unsigned long flags);
extern bool nfs_check_cache_invalid(struct inode *, unsigned long);
extern int nfs_wait_bit_killable(struct wait_bit_key *key, int mode);
+struct file_kattr;
+int nfs_fileattr_get(struct dentry *dentry, struct file_kattr *fa);
+
#if IS_ENABLED(CONFIG_NFS_LOCALIO)
/* localio.c */
struct nfs_local_dio {
diff --git a/fs/nfs/namespace.c b/fs/nfs/namespace.c
index af9be0c..6d0073c 100644
--- a/fs/nfs/namespace.c
+++ b/fs/nfs/namespace.c
@@ -246,11 +246,13 @@ nfs_namespace_setattr(struct mnt_idmap *idmap, struct dentry *dentry,
const struct inode_operations nfs_mountpoint_inode_operations = {
.getattr = nfs_getattr,
.setattr = nfs_setattr,
+ .fileattr_get = nfs_fileattr_get,
};
const struct inode_operations nfs_referral_inode_operations = {
.getattr = nfs_namespace_getattr,
.setattr = nfs_namespace_setattr,
+ .fileattr_get = nfs_fileattr_get,
};
static void nfs_expire_automounts(struct work_struct *work)
diff --git a/fs/nfs/nfs3proc.c b/fs/nfs/nfs3proc.c
index 95d7cd5..b80d0c5 100644
--- a/fs/nfs/nfs3proc.c
+++ b/fs/nfs/nfs3proc.c
@@ -1053,6 +1053,7 @@ static const struct inode_operations nfs3_dir_inode_operations = {
.permission = nfs_permission,
.getattr = nfs_getattr,
.setattr = nfs_setattr,
+ .fileattr_get = nfs_fileattr_get,
#ifdef CONFIG_NFS_V3_ACL
.listxattr = nfs3_listxattr,
.get_inode_acl = nfs3_get_acl,
@@ -1064,6 +1065,7 @@ static const struct inode_operations nfs3_file_inode_operations = {
.permission = nfs_permission,
.getattr = nfs_getattr,
.setattr = nfs_setattr,
+ .fileattr_get = nfs_fileattr_get,
#ifdef CONFIG_NFS_V3_ACL
.listxattr = nfs3_listxattr,
.get_inode_acl = nfs3_get_acl,
diff --git a/fs/nfs/nfs3xdr.c b/fs/nfs/nfs3xdr.c
index e17d729..e745e78 100644
--- a/fs/nfs/nfs3xdr.c
+++ b/fs/nfs/nfs3xdr.c
@@ -2276,8 +2276,11 @@ static int decode_pathconf3resok(struct xdr_stream *xdr,
if (unlikely(!p))
return -EIO;
result->max_link = be32_to_cpup(p++);
- result->max_namelen = be32_to_cpup(p);
- /* ignore remaining fields */
+ result->max_namelen = be32_to_cpup(p++);
+ p++; /* ignore no_trunc */
+ p++; /* ignore chown_restricted */
+ result->case_insensitive = be32_to_cpup(p++) != 0;
+ result->case_preserving = be32_to_cpup(p) != 0;
return 0;
}
diff --git a/fs/nfs/nfs4namespace.c b/fs/nfs/nfs4namespace.c
index 14f72ba..2a03f02 100644
--- a/fs/nfs/nfs4namespace.c
+++ b/fs/nfs/nfs4namespace.c
@@ -481,7 +481,6 @@ int nfs4_submount(struct fs_context *fc, struct nfs_server *server)
* Returns zero on success, or a negative errno value.
*/
static int nfs4_try_replacing_one_location(struct nfs_server *server,
- char *page, char *page2,
const struct nfs4_fs_location *location)
{
struct net *net = rpc_net_ns(server->client);
@@ -541,21 +540,12 @@ static int nfs4_try_replacing_one_location(struct nfs_server *server,
int nfs4_replace_transport(struct nfs_server *server,
const struct nfs4_fs_locations *locations)
{
- char *page = NULL, *page2 = NULL;
int loc, error;
error = -ENOENT;
if (locations == NULL || locations->nlocations <= 0)
goto out;
- error = -ENOMEM;
- page = (char *) __get_free_page(GFP_USER);
- if (!page)
- goto out;
- page2 = (char *) __get_free_page(GFP_USER);
- if (!page2)
- goto out;
-
for (loc = 0; loc < locations->nlocations; loc++) {
const struct nfs4_fs_location *location =
&locations->locations[loc];
@@ -564,14 +554,11 @@ int nfs4_replace_transport(struct nfs_server *server,
location->rootpath.ncomponents == 0)
continue;
- error = nfs4_try_replacing_one_location(server, page,
- page2, location);
+ error = nfs4_try_replacing_one_location(server, location);
if (error == 0)
break;
}
out:
- free_page((unsigned long)page);
- free_page((unsigned long)page2);
return error;
}
diff --git a/fs/nfs/nfs4proc.c b/fs/nfs/nfs4proc.c
index a9b8d48..0715a67 100644
--- a/fs/nfs/nfs4proc.c
+++ b/fs/nfs/nfs4proc.c
@@ -3933,7 +3933,8 @@ static int _nfs4_server_capabilities(struct nfs_server *server, struct nfs_fh *f
server->caps &=
~(NFS_CAP_ACLS | NFS_CAP_HARDLINKS | NFS_CAP_SYMLINKS |
NFS_CAP_SECURITY_LABEL | NFS_CAP_FS_LOCATIONS |
- NFS_CAP_OPEN_XOR | NFS_CAP_DELEGTIME);
+ NFS_CAP_OPEN_XOR | NFS_CAP_DELEGTIME |
+ NFS_CAP_CASE_INSENSITIVE | NFS_CAP_CASE_NONPRESERVING);
server->fattr_valid = NFS_ATTR_FATTR_V4;
if (res.attr_bitmask[0] & FATTR4_WORD0_ACL &&
res.acl_bitmask & ACL4_SUPPORT_ALLOW_ACL)
@@ -3944,8 +3945,9 @@ static int _nfs4_server_capabilities(struct nfs_server *server, struct nfs_fh *f
server->caps |= NFS_CAP_SYMLINKS;
if (res.case_insensitive)
server->caps |= NFS_CAP_CASE_INSENSITIVE;
- if (res.case_preserving)
- server->caps |= NFS_CAP_CASE_PRESERVING;
+ if ((res.attr_bitmask[0] & FATTR4_WORD0_CASE_PRESERVING) &&
+ !res.case_preserving)
+ server->caps |= NFS_CAP_CASE_NONPRESERVING;
#ifdef CONFIG_NFS_V4_SECURITY_LABEL
if (res.attr_bitmask[2] & FATTR4_WORD2_SECURITY_LABEL)
server->caps |= NFS_CAP_SECURITY_LABEL;
@@ -10617,6 +10619,7 @@ static const struct inode_operations nfs4_dir_inode_operations = {
.getattr = nfs_getattr,
.setattr = nfs_setattr,
.listxattr = nfs4_listxattr,
+ .fileattr_get = nfs_fileattr_get,
};
static const struct inode_operations nfs4_file_inode_operations = {
@@ -10624,6 +10627,7 @@ static const struct inode_operations nfs4_file_inode_operations = {
.getattr = nfs_getattr,
.setattr = nfs_setattr,
.listxattr = nfs4_listxattr,
+ .fileattr_get = nfs_fileattr_get,
};
static struct nfs_server *nfs4_clone_server(struct nfs_server *source,
diff --git a/fs/nfs/proc.c b/fs/nfs/proc.c
index 7079568..03c2c1f 100644
--- a/fs/nfs/proc.c
+++ b/fs/nfs/proc.c
@@ -598,6 +598,7 @@ nfs_proc_pathconf(struct nfs_server *server, struct nfs_fh *fhandle,
{
info->max_link = 0;
info->max_namelen = NFS2_MAXNAMLEN;
+ info->case_preserving = true;
return 0;
}
@@ -718,12 +719,14 @@ static const struct inode_operations nfs_dir_inode_operations = {
.permission = nfs_permission,
.getattr = nfs_getattr,
.setattr = nfs_setattr,
+ .fileattr_get = nfs_fileattr_get,
};
static const struct inode_operations nfs_file_inode_operations = {
.permission = nfs_permission,
.getattr = nfs_getattr,
.setattr = nfs_setattr,
+ .fileattr_get = nfs_fileattr_get,
};
const struct nfs_rpc_ops nfs_v2_clientops = {
diff --git a/fs/nfs/super.c b/fs/nfs/super.c
index 4cd420b..8f8a03a 100644
--- a/fs/nfs/super.c
+++ b/fs/nfs/super.c
@@ -623,7 +623,7 @@ static void show_implementation_id(struct seq_file *m, struct nfs_server *nfss)
int nfs_show_devname(struct seq_file *m, struct dentry *root)
{
- char *page = (char *) __get_free_page(GFP_KERNEL);
+ char *page = kmalloc(PAGE_SIZE, GFP_KERNEL);
char *devname, *dummy;
int err = 0;
if (!page)
@@ -633,7 +633,7 @@ int nfs_show_devname(struct seq_file *m, struct dentry *root)
err = PTR_ERR(devname);
else
seq_escape(m, devname, " \t\n\\");
- free_page((unsigned long)page);
+ kfree(page);
return err;
}
EXPORT_SYMBOL_GPL(nfs_show_devname);
diff --git a/fs/nfs/symlink.c b/fs/nfs/symlink.c
index 58146e9..74a0728 100644
--- a/fs/nfs/symlink.c
+++ b/fs/nfs/symlink.c
@@ -22,6 +22,8 @@
#include <linux/mm.h>
#include <linux/string.h>
+#include "internal.h"
+
/* Symlink caching in the page cache is even more simplistic
* and straight-forward than readdir caching.
*/
@@ -74,4 +76,5 @@ const struct inode_operations nfs_symlink_inode_operations = {
.get_link = nfs_get_link,
.getattr = nfs_getattr,
.setattr = nfs_setattr,
+ .fileattr_get = nfs_fileattr_get,
};
diff --git a/fs/nfs/unlink.c b/fs/nfs/unlink.c
index df3ca46..43ea897 100644
--- a/fs/nfs/unlink.c
+++ b/fs/nfs/unlink.c
@@ -124,7 +124,7 @@ static int nfs_call_unlink(struct dentry *dentry, struct inode *inode, struct nf
struct dentry *alias;
down_read_non_owner(&NFS_I(dir)->rmdir_sem);
- alias = d_alloc_parallel(dentry->d_parent, &data->args.name, &data->wq);
+ alias = d_alloc_parallel(dentry->d_parent, &data->args.name);
if (IS_ERR(alias)) {
up_read_non_owner(&NFS_I(dir)->rmdir_sem);
return 0;
@@ -185,7 +185,6 @@ nfs_async_unlink(struct dentry *dentry, const struct qstr *name)
data->cred = get_current_cred();
data->res.dir_attr = &data->dir_attr;
- init_waitqueue_head(&data->wq);
status = -EBUSY;
spin_lock(&dentry->d_lock);
diff --git a/fs/nfsd/blocklayout.c b/fs/nfsd/blocklayout.c
index 9d829c8..5be7721 100644
--- a/fs/nfsd/blocklayout.c
+++ b/fs/nfsd/blocklayout.c
@@ -2,7 +2,7 @@
/*
* Copyright (c) 2014-2016 Christoph Hellwig.
*/
-#include <linux/exportfs.h>
+#include <linux/exportfs_block.h>
#include <linux/iomap.h>
#include <linux/slab.h>
#include <linux/pr.h>
@@ -32,8 +32,8 @@ nfsd4_block_map_extent(struct inode *inode, const struct svc_fh *fhp,
u32 device_generation = 0;
int error;
- error = sb->s_export_op->map_blocks(inode, offset, length, &iomap,
- iomode != IOMODE_READ, &device_generation);
+ error = sb->s_export_op->block_ops->map_blocks(inode, offset, length,
+ &iomap, iomode != IOMODE_READ, &device_generation);
if (error) {
if (error == -ENXIO)
return nfserr_layoutunavailable;
@@ -179,23 +179,20 @@ static __be32
nfsd4_block_commit_blocks(struct inode *inode, struct nfsd4_layoutcommit *lcp,
struct iomap *iomaps, int nr_iomaps)
{
- struct timespec64 mtime = inode_get_mtime(inode);
- struct iattr iattr = { .ia_valid = 0 };
int error;
- if (lcp->lc_mtime.tv_nsec == UTIME_NOW ||
- timespec64_compare(&lcp->lc_mtime, &mtime) < 0)
- lcp->lc_mtime = current_time(inode);
- iattr.ia_valid |= ATTR_ATIME | ATTR_CTIME | ATTR_MTIME;
- iattr.ia_atime = iattr.ia_ctime = iattr.ia_mtime = lcp->lc_mtime;
-
- if (lcp->lc_size_chg) {
- iattr.ia_valid |= ATTR_SIZE;
- iattr.ia_size = lcp->lc_newsize;
- }
-
- error = inode->i_sb->s_export_op->commit_blocks(inode, iomaps,
- nr_iomaps, &iattr);
+ /*
+ * This ignores the client provided mtime in loca_time_modify, as a
+ * fully client specified mtime doesn't really fit into the Linux
+ * multi-grain timestamp architecture.
+ *
+ * RFC 8881 Section 18.42 makes it clear that the client provided
+ * timestamp is a "may" condition, and clients that want to force a
+ * specific timestamp should send a separate SETATTR in the compound.
+ */
+ error = inode->i_sb->s_export_op->block_ops->commit_blocks(inode,
+ iomaps, nr_iomaps,
+ lcp->lc_size_chg ? lcp->lc_newsize : 0);
kfree(iomaps);
return nfserrno(error);
}
@@ -218,8 +215,8 @@ nfsd4_block_get_device_info_simple(struct super_block *sb,
b->type = PNFS_BLOCK_VOLUME_SIMPLE;
b->simple.sig_len = PNFS_BLOCK_UUID_LEN;
- return sb->s_export_op->get_uuid(sb, b->simple.sig, &b->simple.sig_len,
- &b->simple.offset);
+ return sb->s_export_op->block_ops->get_uuid(sb, b->simple.sig,
+ &b->simple.sig_len, &b->simple.offset);
}
static __be32
diff --git a/fs/nfsd/export.c b/fs/nfsd/export.c
index 665153f1..35fef31 100644
--- a/fs/nfsd/export.c
+++ b/fs/nfsd/export.c
@@ -735,7 +735,8 @@ static int svc_export_parse(struct cache_detail *cd, char *mesg, int mlen)
goto out4;
err = 0;
- nfsd4_setup_layout_type(&exp);
+ if (exp.ex_flags & NFSEXP_PNFS)
+ nfsd4_setup_layout_type(&exp);
}
expp = svc_export_lookup(&exp);
diff --git a/fs/nfsd/nfs3proc.c b/fs/nfsd/nfs3proc.c
index 42adc54..aeda7a8 100644
--- a/fs/nfsd/nfs3proc.c
+++ b/fs/nfsd/nfs3proc.c
@@ -710,23 +710,46 @@ nfsd3_proc_pathconf(struct svc_rqst *rqstp)
resp->p_name_max = 255; /* at least */
resp->p_no_trunc = 0;
resp->p_chown_restricted = 1;
- resp->p_case_insensitive = 0;
- resp->p_case_preserving = 1;
+ resp->p_case_insensitive = false;
+ resp->p_case_preserving = true;
resp->status = fh_verify(rqstp, &argp->fh, 0, NFSD_MAY_NOP);
if (resp->status == nfs_ok) {
struct super_block *sb = argp->fh.fh_dentry->d_sb;
+ int err;
- /* Note that we don't care for remote fs's here */
- switch (sb->s_magic) {
- case EXT2_SUPER_MAGIC:
+ if (sb->s_magic == EXT2_SUPER_MAGIC) {
resp->p_link_max = EXT2_LINK_MAX;
resp->p_name_max = EXT2_NAME_LEN;
+ }
+
+ err = nfsd_get_case_info(argp->fh.fh_dentry,
+ &resp->p_case_insensitive,
+ &resp->p_case_preserving);
+ /*
+ * RFC 1813 lists NFS3ERR_STALE, NFS3ERR_BADHANDLE, and
+ * NFS3ERR_SERVERFAULT as the only PATHCONF errors.
+ */
+ switch (err) {
+ case 0:
+ case -EOPNOTSUPP:
+ /* Both arms leave the output booleans valid. */
break;
- case MSDOS_SUPER_MAGIC:
- resp->p_case_insensitive = 1;
- resp->p_case_preserving = 0;
+ case -EACCES:
+ case -EPERM:
+ /*
+ * Policy denied the query. Report STALE so the
+ * handle is unusable without implying a server
+ * malfunction.
+ */
+ resp->status = nfserr_stale;
+ break;
+ case -ESTALE:
+ resp->status = nfserr_stale;
+ break;
+ default:
+ resp->status = nfserr_serverfault;
break;
}
}
diff --git a/fs/nfsd/nfs4layouts.c b/fs/nfsd/nfs4layouts.c
index 69e4110..c3543d4 100644
--- a/fs/nfsd/nfs4layouts.c
+++ b/fs/nfsd/nfs4layouts.c
@@ -2,7 +2,7 @@
/*
* Copyright (c) 2014 Christoph Hellwig.
*/
-#include <linux/blkdev.h>
+#include <linux/exportfs_block.h>
#include <linux/kmod.h>
#include <linux/file.h>
#include <linux/jhash.h>
@@ -127,30 +127,17 @@ nfsd4_set_deviceid(struct nfsd4_deviceid *id, const struct svc_fh *fhp,
void nfsd4_setup_layout_type(struct svc_export *exp)
{
-#if defined(CONFIG_NFSD_BLOCKLAYOUT) || defined(CONFIG_NFSD_SCSILAYOUT)
struct super_block *sb = exp->ex_path.mnt->mnt_sb;
-#endif
+ expfs_block_layouts_t block_supported = exportfs_layouts_supported(sb);
- if (!(exp->ex_flags & NFSEXP_PNFS))
- return;
-
-#ifdef CONFIG_NFSD_FLEXFILELAYOUT
- exp->ex_layout_types |= 1 << LAYOUT_FLEX_FILES;
-#endif
-#ifdef CONFIG_NFSD_BLOCKLAYOUT
- if (sb->s_export_op->get_uuid &&
- sb->s_export_op->map_blocks &&
- sb->s_export_op->commit_blocks)
+ if (IS_ENABLED(CONFIG_NFSD_FLEXFILELAYOUT))
+ exp->ex_layout_types |= 1 << LAYOUT_FLEX_FILES;
+ if (IS_ENABLED(CONFIG_NFSD_BLOCKLAYOUT) &&
+ (block_supported & EXPFS_BLOCK_IN_BAND_ID))
exp->ex_layout_types |= 1 << LAYOUT_BLOCK_VOLUME;
-#endif
-#ifdef CONFIG_NFSD_SCSILAYOUT
- if (sb->s_export_op->map_blocks &&
- sb->s_export_op->commit_blocks &&
- sb->s_bdev &&
- sb->s_bdev->bd_disk->fops->pr_ops &&
- sb->s_bdev->bd_disk->fops->get_unique_id)
+ if (IS_ENABLED(CONFIG_NFSD_SCSILAYOUT) &&
+ (block_supported & EXPFS_BLOCK_OUT_OF_BAND_ID))
exp->ex_layout_types |= 1 << LAYOUT_SCSI;
-#endif
}
void nfsd4_close_layout(struct nfs4_layout_stateid *ls)
diff --git a/fs/nfsd/nfs4xdr.c b/fs/nfsd/nfs4xdr.c
index 2a0946c..20355dc 100644
--- a/fs/nfsd/nfs4xdr.c
+++ b/fs/nfsd/nfs4xdr.c
@@ -3158,6 +3158,8 @@ struct nfsd4_fattr_args {
u32 rdattr_err;
bool contextsupport;
bool ignore_crossmnt;
+ bool case_insensitive;
+ bool case_preserving;
};
typedef __be32(*nfsd4_enc_attr)(struct xdr_stream *xdr,
@@ -3356,6 +3358,33 @@ static __be32 nfsd4_encode_fattr4_acl(struct xdr_stream *xdr,
return nfs_ok;
}
+static __be32 nfsd4_encode_fattr4_case_insensitive(struct xdr_stream *xdr,
+ const struct nfsd4_fattr_args *args)
+{
+ return nfsd4_encode_bool(xdr, args->case_insensitive);
+}
+
+static __be32 nfsd4_encode_fattr4_case_preserving(struct xdr_stream *xdr,
+ const struct nfsd4_fattr_args *args)
+{
+ return nfsd4_encode_bool(xdr, args->case_preserving);
+}
+
+static __be32 nfsd4_encode_fattr4_homogeneous(struct xdr_stream *xdr,
+ const struct nfsd4_fattr_args *args)
+{
+ /*
+ * Casefold-capable filesystems (e.g. ext4 or f2fs with the
+ * casefold feature) attach a Unicode encoding at mount time
+ * but apply case folding per directory. The per-file-system
+ * case_insensitive and case_preserving values can therefore
+ * legitimately differ across objects that share the same fsid.
+ * Report FATTR4_HOMOGENEOUS = FALSE on such filesystems to
+ * keep that variation consistent with RFC 8881 Section 5.8.2.16.
+ */
+ return nfsd4_encode_bool(xdr, !sb_has_encoding(args->dentry->d_sb));
+}
+
static __be32 nfsd4_encode_fattr4_filehandle(struct xdr_stream *xdr,
const struct nfsd4_fattr_args *args)
{
@@ -3748,8 +3777,8 @@ static const nfsd4_enc_attr nfsd4_enc_fattr4_encode_ops[] = {
[FATTR4_ACLSUPPORT] = nfsd4_encode_fattr4_aclsupport,
[FATTR4_ARCHIVE] = nfsd4_encode_fattr4__noop,
[FATTR4_CANSETTIME] = nfsd4_encode_fattr4__true,
- [FATTR4_CASE_INSENSITIVE] = nfsd4_encode_fattr4__false,
- [FATTR4_CASE_PRESERVING] = nfsd4_encode_fattr4__true,
+ [FATTR4_CASE_INSENSITIVE] = nfsd4_encode_fattr4_case_insensitive,
+ [FATTR4_CASE_PRESERVING] = nfsd4_encode_fattr4_case_preserving,
[FATTR4_CHOWN_RESTRICTED] = nfsd4_encode_fattr4__true,
[FATTR4_FILEHANDLE] = nfsd4_encode_fattr4_filehandle,
[FATTR4_FILEID] = nfsd4_encode_fattr4_fileid,
@@ -3758,7 +3787,7 @@ static const nfsd4_enc_attr nfsd4_enc_fattr4_encode_ops[] = {
[FATTR4_FILES_TOTAL] = nfsd4_encode_fattr4_files_total,
[FATTR4_FS_LOCATIONS] = nfsd4_encode_fattr4_fs_locations,
[FATTR4_HIDDEN] = nfsd4_encode_fattr4__noop,
- [FATTR4_HOMOGENEOUS] = nfsd4_encode_fattr4__true,
+ [FATTR4_HOMOGENEOUS] = nfsd4_encode_fattr4_homogeneous,
[FATTR4_MAXFILESIZE] = nfsd4_encode_fattr4_maxfilesize,
[FATTR4_MAXLINK] = nfsd4_encode_fattr4_maxlink,
[FATTR4_MAXNAME] = nfsd4_encode_fattr4_maxname,
@@ -3854,13 +3883,16 @@ static const nfsd4_enc_attr nfsd4_enc_fattr4_encode_ops[] = {
/*
* Note: @fhp can be NULL; in this case, we might have to compose the filehandle
- * ourselves.
+ * ourselves. @case_cache is NULL for callers that encode a single dentry
+ * (GETATTR, the buffer wrapper); READDIR passes a per-request cache so
+ * non-directory children share the parent's case-folding probe result.
*/
static __be32
nfsd4_encode_fattr4(struct svc_rqst *rqstp, struct xdr_stream *xdr,
struct svc_fh *fhp, struct svc_export *exp,
struct dentry *dentry, const u32 *bmval,
- int ignore_crossmnt)
+ int ignore_crossmnt,
+ struct nfsd_case_attrs_cache *case_cache)
{
DECLARE_BITMAP(attr_bitmap, ARRAY_SIZE(nfsd4_enc_fattr4_encode_ops));
struct nfs4_delegation *dp = NULL;
@@ -3968,6 +4000,47 @@ nfsd4_encode_fattr4(struct svc_rqst *rqstp, struct xdr_stream *xdr,
args.fhp = tempfh;
} else
args.fhp = fhp;
+ if (attrmask[0] & (FATTR4_WORD0_CASE_INSENSITIVE |
+ FATTR4_WORD0_CASE_PRESERVING)) {
+ /*
+ * In a batched encoder (READDIR) every non-directory
+ * child shares the same case-folding answer, so the
+ * directory being read is probed once and the result is
+ * cached. The probe targets case_cache->dir, the held
+ * readdir filehandle's dentry, instead of the child's
+ * locklessly-acquired dentry, which a concurrent rename
+ * could move under an unrelated parent. Directory
+ * entries are queried directly because casefold-capable
+ * filesystems answer per directory.
+ *
+ * Per RFC 8881 Section 18.7.3, an attribute advertised
+ * in SUPPORTED_ATTRS must come back with a value or the
+ * GETATTR must fail. nfsd_get_case_info() fills POSIX
+ * defaults and returns -EOPNOTSUPP when the underlying
+ * filesystem does not expose case state; encode those
+ * defaults so the reply agrees with what SUPPORTED_ATTRS
+ * advertises. Other errors fail the operation as the
+ * spec requires.
+ */
+ if (case_cache && !d_is_dir(dentry)) {
+ if (!case_cache->valid) {
+ err = nfsd_get_case_info(case_cache->dir,
+ &case_cache->insensitive,
+ &case_cache->preserving);
+ if (err && err != -EOPNOTSUPP)
+ goto out_nfserr;
+ case_cache->valid = true;
+ }
+ args.case_insensitive = case_cache->insensitive;
+ args.case_preserving = case_cache->preserving;
+ } else {
+ err = nfsd_get_case_info(dentry,
+ &args.case_insensitive,
+ &args.case_preserving);
+ if (err && err != -EOPNOTSUPP)
+ goto out_nfserr;
+ }
+ }
if (attrmask[0] & FATTR4_WORD0_ACL) {
err = nfsd4_get_nfs4_acl(rqstp, dentry, &args.acl);
@@ -4124,7 +4197,7 @@ __be32 nfsd4_encode_fattr_to_buf(__be32 **p, int words,
svcxdr_init_encode_from_buffer(&xdr, &dummy, *p, words << 2);
ret = nfsd4_encode_fattr4(rqstp, &xdr, fhp, exp, dentry, bmval,
- ignore_crossmnt);
+ ignore_crossmnt, NULL);
*p = xdr.p;
return ret;
}
@@ -4162,6 +4235,7 @@ nfsd4_encode_entry4_fattr(struct nfsd4_readdir *cd, const char *name,
struct dentry *dentry;
__be32 nfserr;
int ignore_crossmnt = 0;
+ bool crossed = false;
dentry = lookup_one_positive_unlocked(&nop_mnt_idmap,
&QSTR_LEN(name, namlen),
@@ -4198,11 +4272,18 @@ nfsd4_encode_entry4_fattr(struct nfsd4_readdir *cd, const char *name,
nfserr = check_nfsd_access(exp, cd->rd_rqstp, false);
if (nfserr)
goto out_put;
+ crossed = true;
}
out_encode:
+ /*
+ * A crossed entry no longer shares a parent with the directory
+ * being read, so it must neither consume nor populate the
+ * per-readdir case-folding cache.
+ */
nfserr = nfsd4_encode_fattr4(cd->rd_rqstp, cd->xdr, NULL, exp, dentry,
- cd->rd_bmval, ignore_crossmnt);
+ cd->rd_bmval, ignore_crossmnt,
+ crossed ? NULL : &cd->rd_case_cache);
out_put:
dput(dentry);
exp_put(exp);
@@ -4449,7 +4530,7 @@ nfsd4_encode_getattr(struct nfsd4_compoundres *resp, __be32 nfserr,
/* obj_attributes */
return nfsd4_encode_fattr4(resp->rqstp, xdr, fhp, fhp->fh_export,
- fhp->fh_dentry, getattr->ga_bmval, 0);
+ fhp->fh_dentry, getattr->ga_bmval, 0, NULL);
}
static __be32
@@ -4976,6 +5057,8 @@ static __be32 nfsd4_encode_dirlist4(struct xdr_stream *xdr,
readdir->rd_maxcount = maxcount;
readdir->common.err = 0;
readdir->cookie_offset = 0;
+ readdir->rd_case_cache.dir = readdir->rd_fhp->fh_dentry;
+ readdir->rd_case_cache.valid = false;
offset = readdir->rd_cookie;
status = nfsd_readdir(readdir->rd_rqstp, readdir->rd_fhp, &offset,
&readdir->common, nfsd4_encode_entry4);
diff --git a/fs/nfsd/vfs.c b/fs/nfsd/vfs.c
index eafdf7b..653cd6f 100644
--- a/fs/nfsd/vfs.c
+++ b/fs/nfsd/vfs.c
@@ -32,6 +32,7 @@
#include <linux/writeback.h>
#include <linux/security.h>
#include <linux/sunrpc/xdr.h>
+#include <linux/fileattr.h>
#include "xdr3.h"
@@ -2407,7 +2408,7 @@ static __be32 nfsd_buffered_readdir(struct file *file, struct svc_fh *fhp,
loff_t offset;
struct readdir_data buf = {
.ctx.actor = nfsd_buffered_filldir,
- .dirent = (void *)__get_free_page(GFP_KERNEL)
+ .dirent = kmalloc(PAGE_SIZE, GFP_KERNEL)
};
if (!buf.dirent)
@@ -2458,7 +2459,7 @@ static __be32 nfsd_buffered_readdir(struct file *file, struct svc_fh *fhp,
offset = vfs_llseek(file, 0, SEEK_CUR);
}
- free_page((unsigned long)(buf.dirent));
+ kfree((buf.dirent));
if (host_err)
return nfserrno(host_err);
@@ -2891,3 +2892,88 @@ nfsd_permission(struct svc_cred *cred, struct svc_export *exp,
return err? nfserrno(err) : 0;
}
+
+/**
+ * nfsd_get_case_info - get case sensitivity info for a dentry
+ * @dentry: dentry to query
+ * @case_insensitive: set to true if name comparison ignores case
+ * @case_preserving: set to true if case is preserved on disk
+ *
+ * On casefold-capable filesystems the flag lives on the directory,
+ * not on its entries, so for a non-directory @dentry the parent is
+ * queried instead. A directory (including an export root, whose
+ * parent lies outside the export) is queried as-is so its own
+ * contents' lookup behavior is reported. NFSD advertises
+ * fattr4_homogeneous as FALSE, so per-directory answers may differ
+ * within an export.
+ *
+ * The probe runs with kernel credentials. case_insensitive and
+ * case_preserving describe the directory's structural lookup
+ * behavior, not the caller's identity; running under the calling
+ * client's mapped credentials would let per-client MAC policy on
+ * the parent directory turn this query into NFS4ERR_ACCESS even
+ * though the underlying property is the same for every client.
+ *
+ * When the filesystem does not expose case-folding state (no
+ * ->fileattr_get, or the callback returns -EOPNOTSUPP /
+ * -ENOIOCTLCMD / -ENOTTY / -EINVAL), the outputs are filled with
+ * POSIX defaults (case-sensitive, case-preserving) on the premise
+ * that a filesystem with case-folding support wires up
+ * fileattr_get.
+ *
+ * Return: 0 with outputs filled, -EOPNOTSUPP with outputs filled
+ * to POSIX defaults, or a negative errno (e.g., -EIO,
+ * -ESTALE, -ENOMEM) with outputs unmodified.
+ */
+int
+nfsd_get_case_info(struct dentry *dentry, bool *case_insensitive,
+ bool *case_preserving)
+{
+ struct file_kattr fa = {};
+ const struct cred *saved;
+ struct cred *probe;
+ struct dentry *cd;
+ bool put = false;
+ int err;
+
+ if (d_is_dir(dentry)) {
+ cd = dentry;
+ } else {
+ cd = dget_parent(dentry);
+ put = true;
+ }
+
+ probe = prepare_kernel_cred(&init_task);
+ if (!probe) {
+ err = -ENOMEM;
+ goto out;
+ }
+ saved = override_creds(probe);
+
+ err = vfs_fileattr_get(cd, &fa);
+
+ put_cred(revert_creds(saved));
+out:
+ if (put)
+ dput(cd);
+ switch (err) {
+ case 0:
+ *case_insensitive = fa.fsx_xflags & FS_XFLAG_CASEFOLD;
+ *case_preserving =
+ !(fa.fsx_xflags & FS_XFLAG_CASENONPRESERVING);
+ return 0;
+ case -EINVAL:
+ case -ENOTTY:
+ case -ENOIOCTLCMD:
+ case -EOPNOTSUPP:
+ /*
+ * Filesystem does not expose case state.
+ * Report POSIX defaults.
+ */
+ *case_insensitive = false;
+ *case_preserving = true;
+ return -EOPNOTSUPP;
+ default:
+ return err;
+ }
+}
diff --git a/fs/nfsd/vfs.h b/fs/nfsd/vfs.h
index 702a844..e09ea04 100644
--- a/fs/nfsd/vfs.h
+++ b/fs/nfsd/vfs.h
@@ -156,6 +156,9 @@ __be32 nfsd_readdir(struct svc_rqst *, struct svc_fh *,
loff_t *, struct readdir_cd *, nfsd_filldir_t);
__be32 nfsd_statfs(struct svc_rqst *, struct svc_fh *,
struct kstatfs *, int access);
+int nfsd_get_case_info(struct dentry *dentry,
+ bool *case_insensitive,
+ bool *case_preserving);
__be32 nfsd_permission(struct svc_cred *cred, struct svc_export *exp,
struct dentry *dentry, int acc);
diff --git a/fs/nfsd/xdr3.h b/fs/nfsd/xdr3.h
index 522067b..a7c9714 100644
--- a/fs/nfsd/xdr3.h
+++ b/fs/nfsd/xdr3.h
@@ -209,8 +209,8 @@ struct nfsd3_pathconfres {
__u32 p_name_max;
__u32 p_no_trunc;
__u32 p_chown_restricted;
- __u32 p_case_insensitive;
- __u32 p_case_preserving;
+ bool p_case_insensitive;
+ bool p_case_preserving;
};
struct nfsd3_commitres {
diff --git a/fs/nfsd/xdr4.h b/fs/nfsd/xdr4.h
index 9a4124c..85574b2 100644
--- a/fs/nfsd/xdr4.h
+++ b/fs/nfsd/xdr4.h
@@ -432,6 +432,19 @@ struct nfsd4_read {
u32 rd_eof; /* response */
};
+/*
+ * Cache the case-folding properties of @dir so a batched encoder
+ * (e.g., READDIR) does not re-probe per child. @dir is the
+ * directory being read, held by the request, so it is stable
+ * against rename for the duration of the cache's lifetime.
+ */
+struct nfsd_case_attrs_cache {
+ struct dentry *dir;
+ bool valid;
+ bool insensitive;
+ bool preserving;
+};
+
struct nfsd4_readdir {
u64 rd_cookie; /* request */
nfs4_verifier rd_verf; /* request */
@@ -444,6 +457,7 @@ struct nfsd4_readdir {
struct readdir_cd common;
struct xdr_stream *xdr;
int cookie_offset;
+ struct nfsd_case_attrs_cache rd_case_cache;
};
struct nfsd4_release_lockowner {
diff --git a/fs/nilfs2/btnode.c b/fs/nilfs2/btnode.c
index 2e553d6..680e400 100644
--- a/fs/nilfs2/btnode.c
+++ b/fs/nilfs2/btnode.c
@@ -134,9 +134,7 @@ int nilfs_btnode_submit_block(struct address_space *btnc, __u64 blocknr,
}
set_buffer_mapped(bh);
bh->b_blocknr = pblocknr; /* set block address for read */
- bh->b_end_io = end_buffer_read_sync;
- get_bh(bh);
- submit_bh(opf, bh);
+ bh_submit(bh, opf, bh_end_read);
bh->b_blocknr = blocknr; /* set back to the given block address */
*submit_ptr = pblocknr;
err = 0;
diff --git a/fs/nilfs2/gcinode.c b/fs/nilfs2/gcinode.c
index 62d4c1b..85379ac 100644
--- a/fs/nilfs2/gcinode.c
+++ b/fs/nilfs2/gcinode.c
@@ -83,9 +83,7 @@ int nilfs_gccache_submit_read_data(struct inode *inode, sector_t blkoff,
if (!buffer_mapped(bh))
set_buffer_mapped(bh);
bh->b_blocknr = pbn;
- bh->b_end_io = end_buffer_read_sync;
- get_bh(bh);
- submit_bh(REQ_OP_READ, bh);
+ bh_submit(bh, REQ_OP_READ, bh_end_read);
if (vbn)
bh->b_blocknr = vbn;
out:
diff --git a/fs/nilfs2/ioctl.c b/fs/nilfs2/ioctl.c
index e0a6066..b73f2c5 100644
--- a/fs/nilfs2/ioctl.c
+++ b/fs/nilfs2/ioctl.c
@@ -69,7 +69,7 @@ static int nilfs_ioctl_wrap_copy(struct the_nilfs *nilfs,
if (argv->v_index > ~(__u64)0 - argv->v_nmembs)
return -EINVAL;
- buf = (void *)get_zeroed_page(GFP_NOFS);
+ buf = kzalloc(PAGE_SIZE, GFP_NOFS);
if (unlikely(!buf))
return -ENOMEM;
maxmembs = PAGE_SIZE / argv->v_size;
@@ -107,7 +107,7 @@ static int nilfs_ioctl_wrap_copy(struct the_nilfs *nilfs,
}
argv->v_nmembs = total;
- free_pages((unsigned long)buf, 0);
+ kfree(buf);
return ret;
}
diff --git a/fs/nilfs2/mdt.c b/fs/nilfs2/mdt.c
index 09adb40..2a43534 100644
--- a/fs/nilfs2/mdt.c
+++ b/fs/nilfs2/mdt.c
@@ -148,9 +148,7 @@ nilfs_mdt_submit_block(struct inode *inode, unsigned long blkoff, blk_opf_t opf,
}
map_bh(bh, inode->i_sb, (sector_t)blknum);
- bh->b_end_io = end_buffer_read_sync;
- get_bh(bh);
- submit_bh(opf, bh);
+ bh_submit(bh, opf, bh_end_read);
ret = 0;
trace_nilfs2_mdt_submit_block(inode, inode->i_ino, blkoff,
diff --git a/fs/notify/fsnotify.c b/fs/notify/fsnotify.c
index 2dac70b..7e2f330 100644
--- a/fs/notify/fsnotify.c
+++ b/fs/notify/fsnotify.c
@@ -14,6 +14,9 @@
#include <linux/fsnotify_backend.h>
#include "fsnotify.h"
+#define CREATE_TRACE_POINTS
+#include <trace/events/fsnotify.h>
+
/*
* Clear all of the marks on an inode when it is being evicted from core
*/
@@ -504,6 +507,8 @@ int fsnotify(__u32 mask, const void *data, int data_type, struct inode *dir,
int ret = 0;
__u32 test_mask, marks_mask = 0;
+ trace_fsnotify(mask, data, data_type, dir, file_name, inode, cookie);
+
if (path)
mnt = real_mount(path->mnt);
diff --git a/fs/notify/mark.c b/fs/notify/mark.c
index e256b42..b2640d8 100644
--- a/fs/notify/mark.c
+++ b/fs/notify/mark.c
@@ -343,6 +343,35 @@ void fsnotify_recalc_mask(struct fsnotify_mark_connector *conn)
fsnotify_conn_set_children_dentry_flags(conn);
}
+/**
+ * fsnotify_modify_mark_mask - set and/or clear flags in a mark's mask
+ * @mark: mark to be modified
+ * @set: bits to be set in mask
+ * @clear: bits to be cleared in mask
+ *
+ * Modify a fsnotify_mark mask as directed, and update its associated conn.
+ * The caller is expected to hold a reference to the mark.
+ */
+void fsnotify_modify_mark_mask(struct fsnotify_mark *mark, u32 set, u32 clear)
+{
+ bool recalc = false;
+ u32 mask;
+
+ WARN_ON_ONCE(clear & set);
+
+ spin_lock(&mark->lock);
+ mask = mark->mask;
+ mark->mask |= set;
+ mark->mask &= ~clear;
+ if (mark->mask != mask)
+ recalc = true;
+ spin_unlock(&mark->lock);
+
+ if (recalc)
+ fsnotify_recalc_mask(mark->connector);
+}
+EXPORT_SYMBOL_GPL(fsnotify_modify_mark_mask);
+
/* Free all connectors queued for freeing once SRCU period ends */
static void fsnotify_connector_destroy_workfn(struct work_struct *work)
{
diff --git a/fs/nsfs.c b/fs/nsfs.c
index 160018c..c3b6ae7 100644
--- a/fs/nsfs.c
+++ b/fs/nsfs.c
@@ -664,7 +664,6 @@ static int nsfs_init_fs_context(struct fs_context *fc)
struct pseudo_fs_context *ctx = init_pseudo(fc, NSFS_MAGIC);
if (!ctx)
return -ENOMEM;
- fc->s_iflags |= SB_I_NOEXEC | SB_I_NODEV;
ctx->s_d_flags |= DCACHE_DONTCACHE;
ctx->ops = &nsfs_ops;
ctx->eops = &nsfs_export_operations;
diff --git a/fs/ntfs/namei.c b/fs/ntfs/namei.c
index c4f8284..3e21753 100644
--- a/fs/ntfs/namei.c
+++ b/fs/ntfs/namei.c
@@ -230,9 +230,8 @@ static struct dentry *ntfs_lookup(struct inode *dir_ino, struct dentry *dent,
if (MREF_ERR(mref) == -ENOENT) {
ntfs_debug("Entry was not found, adding negative dentry.");
/* The dcache will handle negative entries. */
- d_add(dent, NULL);
ntfs_debug("Done.");
- return NULL;
+ return d_splice_alias(NULL, dent);
}
ntfs_error(vol->sb, "ntfs_lookup_ino_by_name() failed with error code %i.",
-MREF_ERR(mref));
diff --git a/fs/ntfs3/file.c b/fs/ntfs3/file.c
index b041639..ad9350d 100644
--- a/fs/ntfs3/file.c
+++ b/fs/ntfs3/file.c
@@ -181,6 +181,34 @@ long ntfs_compat_ioctl(struct file *filp, u32 cmd, unsigned long arg)
#endif
/*
+ * ntfs_fileattr_get - inode_operations::fileattr_get
+ */
+int ntfs_fileattr_get(struct dentry *dentry, struct file_kattr *fa)
+{
+ struct inode *inode = d_inode(dentry);
+ struct ntfs_sb_info *sbi = inode->i_sb->s_fs_info;
+
+ /* Avoid any operation if inode is bad. */
+ if (unlikely(is_bad_ni(ntfs_i(inode))))
+ return -EINVAL;
+
+ /*
+ * NTFS preserves case (the default). Case sensitivity depends on
+ * mount options: with "nocase", NTFS is case-insensitive;
+ * otherwise it is case-sensitive.
+ */
+ if (sbi->options->nocase) {
+ fa->fsx_xflags |= FS_XFLAG_CASEFOLD;
+ fa->flags |= FS_CASEFOLD_FL;
+ }
+ if (inode->i_flags & S_IMMUTABLE) {
+ fa->fsx_xflags |= FS_XFLAG_IMMUTABLE;
+ fa->flags |= FS_IMMUTABLE_FL;
+ }
+ return 0;
+}
+
+/*
* ntfs_getattr - inode_operations::getattr
*/
int ntfs_getattr(struct mnt_idmap *idmap, const struct path *path,
@@ -1547,6 +1575,7 @@ const struct inode_operations ntfs_file_inode_operations = {
.get_acl = ntfs_get_acl,
.set_acl = ntfs_set_acl,
.fiemap = ntfs_fiemap,
+ .fileattr_get = ntfs_fileattr_get,
};
const struct file_operations ntfs_file_operations = {
diff --git a/fs/ntfs3/namei.c b/fs/ntfs3/namei.c
index b2af8f6..e159ba6 100644
--- a/fs/ntfs3/namei.c
+++ b/fs/ntfs3/namei.c
@@ -518,6 +518,7 @@ const struct inode_operations ntfs_dir_inode_operations = {
.getattr = ntfs_getattr,
.listxattr = ntfs_listxattr,
.fiemap = ntfs_fiemap,
+ .fileattr_get = ntfs_fileattr_get,
};
const struct inode_operations ntfs_special_inode_operations = {
diff --git a/fs/ntfs3/ntfs_fs.h b/fs/ntfs3/ntfs_fs.h
index bbf3b6a..41db22d 100644
--- a/fs/ntfs3/ntfs_fs.h
+++ b/fs/ntfs3/ntfs_fs.h
@@ -529,6 +529,7 @@ bool dir_is_empty(struct inode *dir);
extern const struct file_operations ntfs_dir_operations;
/* Globals from file.c */
+int ntfs_fileattr_get(struct dentry *dentry, struct file_kattr *fa);
int ntfs_getattr(struct mnt_idmap *idmap, const struct path *path,
struct kstat *stat, u32 request_mask, u32 flags);
int ntfs_setattr(struct mnt_idmap *idmap, struct dentry *dentry,
diff --git a/fs/ntfs3/super.c b/fs/ntfs3/super.c
index 004f599..3305fe4 100644
--- a/fs/ntfs3/super.c
+++ b/fs/ntfs3/super.c
@@ -1174,7 +1174,10 @@ static int ntfs_init_from_boot(struct super_block *sb, u32 sector_size,
rec->total = cpu_to_le32(sbi->record_size);
((struct ATTRIB *)Add2Ptr(rec, ao))->type = ATTR_END;
- sb_set_blocksize(sb, min_t(u32, sbi->cluster_size, PAGE_SIZE));
+ if (!sb_set_blocksize(sb, min_t(u32, sbi->cluster_size, PAGE_SIZE))) {
+ err = -EINVAL;
+ goto out;
+ }
sbi->block_mask = sb->s_blocksize - 1;
sbi->blocks_per_cluster = sbi->cluster_size >> sb->s_blocksize_bits;
@@ -1225,7 +1228,8 @@ static int ntfs_init_from_boot(struct super_block *sb, u32 sector_size,
/*
* Try alternative boot (last sector)
*/
- sb_set_blocksize(sb, block_size);
+ if (!sb_set_blocksize(sb, block_size))
+ return -EINVAL;
hint = "Alternative boot";
dev_size = dev_size0; /* restore original size. */
goto read_boot;
diff --git a/fs/ocfs2/buffer_head_io.c b/fs/ocfs2/buffer_head_io.c
index 701d27d..8e67eba 100644
--- a/fs/ocfs2/buffer_head_io.c
+++ b/fs/ocfs2/buffer_head_io.c
@@ -62,9 +62,7 @@ int ocfs2_write_block(struct ocfs2_super *osb, struct buffer_head *bh,
/* remove from dirty list before I/O. */
clear_buffer_dirty(bh);
- get_bh(bh); /* for end_buffer_write_sync() */
- bh->b_end_io = end_buffer_write_sync;
- submit_bh(REQ_OP_WRITE, bh);
+ bh_submit(bh, REQ_OP_WRITE, bh_end_write);
wait_on_buffer(bh);
@@ -145,9 +143,7 @@ int ocfs2_read_blocks_sync(struct ocfs2_super *osb, u64 block,
#endif
}
- get_bh(bh); /* for end_buffer_read_sync() */
- bh->b_end_io = end_buffer_read_sync;
- submit_bh(REQ_OP_READ, bh);
+ bh_submit(bh, REQ_OP_READ, bh_end_read);
}
read_failure:
@@ -323,11 +319,9 @@ int ocfs2_read_blocks(struct ocfs2_caching_info *ci, u64 block, int nr,
continue;
}
- get_bh(bh); /* for end_buffer_read_sync() */
if (validate)
set_buffer_needs_validate(bh);
- bh->b_end_io = end_buffer_read_sync;
- submit_bh(REQ_OP_READ, bh);
+ bh_submit(bh, REQ_OP_READ, bh_end_read);
continue;
}
}
@@ -446,10 +440,8 @@ int ocfs2_write_super_or_backup(struct ocfs2_super *osb,
/* remove from dirty list before I/O. */
clear_buffer_dirty(bh);
- get_bh(bh); /* for end_buffer_write_sync() */
- bh->b_end_io = end_buffer_write_sync;
ocfs2_compute_meta_ecc(osb->sb, bh->b_data, &di->i_check);
- submit_bh(REQ_OP_WRITE, bh);
+ bh_submit(bh, REQ_OP_WRITE, bh_end_write);
wait_on_buffer(bh);
diff --git a/fs/ocfs2/dlm/dlmdebug.c b/fs/ocfs2/dlm/dlmdebug.c
index fe4fdd09..6ca8b3b 100644
--- a/fs/ocfs2/dlm/dlmdebug.c
+++ b/fs/ocfs2/dlm/dlmdebug.c
@@ -260,10 +260,10 @@ void dlm_print_one_mle(struct dlm_master_list_entry *mle)
{
char *buf;
- buf = (char *) get_zeroed_page(GFP_ATOMIC);
+ buf = kzalloc(PAGE_SIZE, GFP_ATOMIC);
if (buf) {
dump_mle(mle, buf, PAGE_SIZE - 1);
- free_page((unsigned long)buf);
+ kfree(buf);
}
}
@@ -280,7 +280,7 @@ static struct dentry *dlm_debugfs_root;
/* begin - utils funcs */
static int debug_release(struct inode *inode, struct file *file)
{
- free_page((unsigned long)file->private_data);
+ kfree(file->private_data);
return 0;
}
@@ -327,17 +327,15 @@ static int debug_purgelist_open(struct inode *inode, struct file *file)
struct dlm_ctxt *dlm = inode->i_private;
char *buf = NULL;
- buf = (char *) get_zeroed_page(GFP_NOFS);
+ buf = kzalloc(PAGE_SIZE, GFP_NOFS);
if (!buf)
- goto bail;
+ return -ENOMEM;
i_size_write(inode, debug_purgelist_print(dlm, buf, PAGE_SIZE - 1));
file->private_data = buf;
return 0;
-bail:
- return -ENOMEM;
}
static const struct file_operations debug_purgelist_fops = {
@@ -384,17 +382,15 @@ static int debug_mle_open(struct inode *inode, struct file *file)
struct dlm_ctxt *dlm = inode->i_private;
char *buf = NULL;
- buf = (char *) get_zeroed_page(GFP_NOFS);
+ buf = kzalloc(PAGE_SIZE, GFP_NOFS);
if (!buf)
- goto bail;
+ return -ENOMEM;
i_size_write(inode, debug_mle_print(dlm, buf, PAGE_SIZE - 1));
file->private_data = buf;
return 0;
-bail:
- return -ENOMEM;
}
static const struct file_operations debug_mle_fops = {
@@ -775,17 +771,15 @@ static int debug_state_open(struct inode *inode, struct file *file)
struct dlm_ctxt *dlm = inode->i_private;
char *buf = NULL;
- buf = (char *) get_zeroed_page(GFP_NOFS);
+ buf = kzalloc(PAGE_SIZE, GFP_NOFS);
if (!buf)
- goto bail;
+ return -ENOMEM;
i_size_write(inode, debug_state_print(dlm, buf, PAGE_SIZE - 1));
file->private_data = buf;
return 0;
-bail:
- return -ENOMEM;
}
static const struct file_operations debug_state_fops = {
diff --git a/fs/ocfs2/dlm/dlmdomain.c b/fs/ocfs2/dlm/dlmdomain.c
index dc9da91..97bb940 100644
--- a/fs/ocfs2/dlm/dlmdomain.c
+++ b/fs/ocfs2/dlm/dlmdomain.c
@@ -63,7 +63,7 @@ static inline void byte_copymap(u8 dmap[], unsigned long smap[],
static void dlm_free_pagevec(void **vec, int pages)
{
while (pages--)
- free_page((unsigned long)vec[pages]);
+ kfree(vec[pages]);
kfree(vec);
}
@@ -75,9 +75,11 @@ static void **dlm_alloc_pagevec(int pages)
if (!vec)
return NULL;
- for (i = 0; i < pages; i++)
- if (!(vec[i] = (void *)__get_free_page(GFP_KERNEL)))
+ for (i = 0; i < pages; i++) {
+ vec[i] = kmalloc(PAGE_SIZE, GFP_KERNEL);
+ if (!vec[i])
goto out_free;
+ }
mlog(0, "Allocated DLM hash pagevec; %d pages (%lu expected), %lu buckets per page\n",
pages, (unsigned long)DLM_HASH_PAGES,
diff --git a/fs/ocfs2/dlm/dlmmaster.c b/fs/ocfs2/dlm/dlmmaster.c
index 93eff38..aee3b4c 100644
--- a/fs/ocfs2/dlm/dlmmaster.c
+++ b/fs/ocfs2/dlm/dlmmaster.c
@@ -2548,7 +2548,7 @@ static int dlm_migrate_lockres(struct dlm_ctxt *dlm,
/* preallocate up front. if this fails, abort */
ret = -ENOMEM;
- mres = (struct dlm_migratable_lockres *) __get_free_page(GFP_NOFS);
+ mres = kmalloc(PAGE_SIZE, GFP_NOFS);
if (!mres) {
mlog_errno(ret);
goto leave;
@@ -2725,8 +2725,7 @@ static int dlm_migrate_lockres(struct dlm_ctxt *dlm,
if (wake)
wake_up(&res->wq);
- if (mres)
- free_page((unsigned long)mres);
+ kfree(mres);
dlm_put(dlm);
diff --git a/fs/ocfs2/dlm/dlmrecovery.c b/fs/ocfs2/dlm/dlmrecovery.c
index 128872b..9b97bf7 100644
--- a/fs/ocfs2/dlm/dlmrecovery.c
+++ b/fs/ocfs2/dlm/dlmrecovery.c
@@ -837,7 +837,7 @@ int dlm_request_all_locks_handler(struct o2net_msg *msg, u32 len, void *data,
}
/* this will get freed by dlm_request_all_locks_worker */
- buf = (char *) __get_free_page(GFP_NOFS);
+ buf = kmalloc(PAGE_SIZE, GFP_NOFS);
if (!buf) {
kfree(item);
dlm_put(dlm);
@@ -933,7 +933,7 @@ static void dlm_request_all_locks_worker(struct dlm_work_item *item, void *data)
}
}
leave:
- free_page((unsigned long)data);
+ kfree(data);
}
diff --git a/fs/ocfs2/super.c b/fs/ocfs2/super.c
index b875f01..4870e68 100644
--- a/fs/ocfs2/super.c
+++ b/fs/ocfs2/super.c
@@ -1224,7 +1224,6 @@ static struct file_system_type ocfs2_fs_type = {
.name = "ocfs2",
.kill_sb = kill_block_super,
.fs_flags = FS_REQUIRES_DEV|FS_RENAME_DOES_D_MOVE,
- .next = NULL,
.init_fs_context = ocfs2_init_fs_context,
.parameters = ocfs2_param_spec,
};
diff --git a/fs/omfs/inode.c b/fs/omfs/inode.c
index 834cae1..1d915ef72 100644
--- a/fs/omfs/inode.c
+++ b/fs/omfs/inode.c
@@ -478,7 +478,8 @@ static int omfs_fill_super(struct super_block *sb, struct fs_context *fc)
sb->s_time_min = 0;
sb->s_time_max = U64_MAX / MSEC_PER_SEC;
- sb_set_blocksize(sb, 0x200);
+ if (!sb_set_blocksize(sb, 0x200))
+ goto end;
bh = sb_bread(sb, 0);
if (!bh)
@@ -530,7 +531,8 @@ static int omfs_fill_super(struct super_block *sb, struct fs_context *fc)
* Use sys_blocksize as the fs block since it is smaller than a
* page while the fs blocksize can be larger.
*/
- sb_set_blocksize(sb, sbi->s_sys_blocksize);
+ if (!sb_set_blocksize(sb, sbi->s_sys_blocksize))
+ goto out_brelse_bh;
/*
* ...and the difference goes into a shift. sys_blocksize is always
diff --git a/fs/open.c b/fs/open.c
index 681d405..5458668 100644
--- a/fs/open.c
+++ b/fs/open.c
@@ -960,7 +960,7 @@ static int do_dentry_open(struct file *f,
if (f->f_mapping->a_ops && f->f_mapping->a_ops->direct_IO)
f->f_mode |= FMODE_CAN_ODIRECT;
- f->f_flags &= ~(O_CREAT | O_EXCL | O_NOCTTY | O_TRUNC);
+ f->f_flags &= ~(O_CREAT | O_EXCL | O_NOCTTY | O_TRUNC | __O_REGULAR);
f->f_iocb_flags = iocb_flags(f);
file_ra_state_init(&f->f_ra, f->f_mapping->host->i_mapping);
@@ -1158,7 +1158,7 @@ struct file *kernel_file_open(const struct path *path, int flags,
EXPORT_SYMBOL_GPL(kernel_file_open);
#define WILL_CREATE(flags) (flags & (O_CREAT | __O_TMPFILE))
-#define O_PATH_FLAGS (O_DIRECTORY | O_NOFOLLOW | O_PATH | O_CLOEXEC)
+#define O_PATH_FLAGS (O_DIRECTORY | O_NOFOLLOW | O_PATH | O_CLOEXEC | O_EMPTYPATH)
inline struct open_how build_open_how(int flags, umode_t mode)
{
@@ -1184,7 +1184,15 @@ inline int build_open_flags(const struct open_how *how, struct open_flags *op)
int acc_mode = ACC_MODE(flags);
BUILD_BUG_ON_MSG(upper_32_bits(VALID_OPEN_FLAGS),
- "struct open_flags doesn't yet handle flags > 32 bits");
+ "VALID_OPEN_FLAGS must fit in 32 bits");
+ /* The whole point: OPENAT2_REGULAR must be unrepresentable in int. */
+ BUILD_BUG_ON_MSG(!upper_32_bits(OPENAT2_REGULAR),
+ "OPENAT2_REGULAR must live in the upper 32 bits of open_how::flags");
+ /* Prevent a future bit collision between UAPI and internal carrier. */
+ BUILD_BUG_ON_MSG(OPENAT2_REGULAR & VALID_OPEN_FLAGS,
+ "OPENAT2_REGULAR must not alias any open()/openat() flag");
+ BUILD_BUG_ON_MSG(__O_REGULAR & VALID_OPENAT2_FLAGS,
+ "__O_REGULAR must not alias any user-visible flag");
/*
* Strip flags that aren't relevant in determining struct open_flags.
@@ -1196,7 +1204,7 @@ inline int build_open_flags(const struct open_how *how, struct open_flags *op)
* values before calling build_open_flags(), but openat2(2) checks all
* of its arguments.
*/
- if (flags & ~VALID_OPEN_FLAGS)
+ if (flags & ~VALID_OPENAT2_FLAGS)
return -EINVAL;
if (how->resolve & ~VALID_RESOLVE_FLAGS)
return -EINVAL;
@@ -1236,6 +1244,14 @@ inline int build_open_flags(const struct open_how *how, struct open_flags *op)
if (!(acc_mode & MAY_WRITE))
return -EINVAL;
}
+ /*
+ * Asking to open a directory and a regular file at the same time is
+ * contradictory.
+ */
+ if ((flags & (O_DIRECTORY | OPENAT2_REGULAR)) ==
+ (O_DIRECTORY | OPENAT2_REGULAR))
+ return -EINVAL;
+
if (flags & O_PATH) {
/* O_PATH only permits certain other flags to be set. */
if (flags & ~O_PATH_FLAGS)
@@ -1252,6 +1268,19 @@ inline int build_open_flags(const struct open_how *how, struct open_flags *op)
if (flags & __O_SYNC)
flags |= O_DSYNC;
+ /*
+ * Translate the upper-32-bit UAPI bit OPENAT2_REGULAR into the
+ * kernel-internal lower-32-bit __O_REGULAR carrier so the bit
+ * survives the assignment to op->open_flag (an int) below and the
+ * subsequent flow through f->f_flags (unsigned int) and the
+ * i_op->atomic_open() callback (unsigned). do_dentry_open() strips
+ * __O_REGULAR before the file becomes visible to userspace.
+ */
+ if (flags & OPENAT2_REGULAR) {
+ flags &= ~OPENAT2_REGULAR;
+ flags |= __O_REGULAR;
+ }
+
op->open_flag = flags;
/* O_TRUNC implies we need access checks for write permissions */
@@ -1279,6 +1308,8 @@ inline int build_open_flags(const struct open_how *how, struct open_flags *op)
lookup_flags |= LOOKUP_DIRECTORY;
if (!(flags & O_NOFOLLOW))
lookup_flags |= LOOKUP_FOLLOW;
+ if (flags & O_EMPTYPATH)
+ lookup_flags |= LOOKUP_EMPTY;
if (how->resolve & RESOLVE_NO_XDEV)
lookup_flags |= LOOKUP_NO_XDEV;
@@ -1360,7 +1391,7 @@ static int do_sys_openat2(int dfd, const char __user *filename,
if (unlikely(err))
return err;
- CLASS(filename, name)(filename);
+ CLASS(filename_flags, name)(filename, op.lookup_flags);
return FD_ADD(how->flags, do_file_open(dfd, name, &op));
}
diff --git a/fs/pidfs.c b/fs/pidfs.c
index 1cce4f3..fdd75c3 100644
--- a/fs/pidfs.c
+++ b/fs/pidfs.c
@@ -37,6 +37,8 @@ static struct kmem_cache *pidfs_attr_cachep __ro_after_init;
static struct path pidfs_root_path = {};
+static struct simple_xattr_cache pidfs_xa_cache;
+
void pidfs_get_root(struct path *path)
{
*path = pidfs_root_path;
@@ -96,7 +98,7 @@ static const struct rhashtable_params pidfs_ino_ht_params = {
* use file handles.
*/
struct pidfs_attr {
- struct simple_xattrs *xattrs;
+ struct list_head xattrs;
union {
struct pidfs_anon_attr;
struct llist_node pidfs_llist;
@@ -196,12 +198,7 @@ static void pidfs_free_attr_work(struct work_struct *work)
head = llist_del_all(&pidfs_free_list);
llist_for_each_entry_safe(attr, next, head, pidfs_llist) {
- struct simple_xattrs *xattrs = attr->xattrs;
-
- if (xattrs) {
- simple_xattrs_free(xattrs, NULL);
- kfree(xattrs);
- }
+ simple_xattrs_free(&pidfs_xa_cache, &attr->xattrs, NULL);
kfree(attr);
}
}
@@ -229,7 +226,7 @@ void pidfs_free_pid(struct pid *pid)
if (IS_ERR(attr))
return;
- if (likely(!attr->xattrs))
+ if (likely(list_empty(&attr->xattrs)))
kfree(attr);
else if (llist_add(&attr->pidfs_llist, &pidfs_free_list))
schedule_work(&pidfs_free_work);
@@ -338,14 +335,14 @@ static inline bool pid_in_current_pidns(const struct pid *pid)
return false;
}
-static __u32 pidfs_coredump_mask(unsigned long mm_flags)
+static __u32 pidfs_coredump_mask(enum task_dumpable dumpable)
{
- switch (__get_dumpable(mm_flags)) {
- case SUID_DUMP_USER:
+ switch (dumpable) {
+ case TASK_DUMPABLE_OWNER:
return PIDFD_COREDUMP_USER;
- case SUID_DUMP_ROOT:
+ case TASK_DUMPABLE_ROOT:
return PIDFD_COREDUMP_ROOT;
- case SUID_DUMP_DISABLE:
+ case TASK_DUMPABLE_OFF:
return PIDFD_COREDUMP_SKIP;
default:
WARN_ON_ONCE(true);
@@ -433,14 +430,9 @@ static long pidfd_info(struct file *file, unsigned int cmd, unsigned long arg)
return -ESRCH;
if ((mask & PIDFD_INFO_COREDUMP) && !kinfo.coredump_mask) {
- guard(task_lock)(task);
- if (task->mm) {
- unsigned long flags = __mm_flags_get_dumpable(task->mm);
-
- kinfo.coredump_mask = pidfs_coredump_mask(flags);
- kinfo.mask |= PIDFD_INFO_COREDUMP;
- /* No coredump actually took place, so no coredump signal. */
- }
+ kinfo.coredump_mask = pidfs_coredump_mask(task_exec_state_get_dumpable(task));
+ kinfo.mask |= PIDFD_INFO_COREDUMP;
+ /* No coredump actually took place, so no coredump signal. */
}
/* Unconditionally return identifiers and credentials, the rest only on request */
@@ -779,7 +771,7 @@ void pidfs_coredump(const struct coredump_params *cprm)
VFS_WARN_ON_ONCE(attr == PIDFS_PID_DEAD);
/* Note how we were coredumped and that we coredumped. */
- attr->coredump_mask = pidfs_coredump_mask(cprm->mm_flags) |
+ attr->coredump_mask = pidfs_coredump_mask(cprm->dumpable) |
PIDFD_COREDUMPED;
/* If coredumping is set to skip we should never end up here. */
VFS_WARN_ON_ONCE(attr->coredump_mask & PIDFD_COREDUMP_SKIP);
@@ -815,14 +807,8 @@ static ssize_t pidfs_listxattr(struct dentry *dentry, char *buf, size_t size)
{
struct inode *inode = d_inode(dentry);
struct pid *pid = inode->i_private;
- struct pidfs_attr *attr = pid->attr;
- struct simple_xattrs *xattrs;
- xattrs = READ_ONCE(attr->xattrs);
- if (!xattrs)
- return 0;
-
- return simple_xattr_list(inode, xattrs, buf, size);
+ return simple_xattr_list(inode, &pid->attr->xattrs, buf, size);
}
static const struct inode_operations pidfs_inode_operations = {
@@ -1018,6 +1004,8 @@ int pidfs_register_pid(struct pid *pid)
if (!new_attr)
return -ENOMEM;
+ INIT_LIST_HEAD_RCU(&new_attr->xattrs);
+
/* Synchronize with pidfs_exit(). */
guard(spinlock_irq)(&pid->wait_pidfd.lock);
@@ -1057,16 +1045,9 @@ static int pidfs_xattr_get(const struct xattr_handler *handler,
const char *suffix, void *value, size_t size)
{
struct pid *pid = inode->i_private;
- struct pidfs_attr *attr = pid->attr;
- const char *name;
- struct simple_xattrs *xattrs;
+ const char *name = xattr_full_name(handler, suffix);
- xattrs = READ_ONCE(attr->xattrs);
- if (!xattrs)
- return -ENODATA;
-
- name = xattr_full_name(handler, suffix);
- return simple_xattr_get(xattrs, name, value, size);
+ return simple_xattr_get(&pidfs_xa_cache, &pid->attr->xattrs, name, value, size);
}
static int pidfs_xattr_set(const struct xattr_handler *handler,
@@ -1075,20 +1056,13 @@ static int pidfs_xattr_set(const struct xattr_handler *handler,
const void *value, size_t size, int flags)
{
struct pid *pid = inode->i_private;
- struct pidfs_attr *attr = pid->attr;
- const char *name;
- struct simple_xattrs *xattrs;
+ const char *name = xattr_full_name(handler, suffix);
struct simple_xattr *old_xattr;
/* Ensure we're the only one to set @attr->xattrs. */
WARN_ON_ONCE(!inode_is_locked(inode));
- xattrs = simple_xattrs_lazy_alloc(&attr->xattrs, value, flags);
- if (IS_ERR_OR_NULL(xattrs))
- return PTR_ERR(xattrs);
-
- name = xattr_full_name(handler, suffix);
- old_xattr = simple_xattr_set(xattrs, name, value, size, flags);
+ old_xattr = simple_xattr_set(&pidfs_xa_cache, &pid->attr->xattrs, name, value, size, flags);
if (IS_ERR(old_xattr))
return PTR_ERR(old_xattr);
@@ -1115,8 +1089,6 @@ static int pidfs_init_fs_context(struct fs_context *fc)
if (!ctx)
return -ENOMEM;
- fc->s_iflags |= SB_I_NOEXEC;
- fc->s_iflags |= SB_I_NODEV;
ctx->s_d_flags |= DCACHE_DONTCACHE;
ctx->ops = &pidfs_sops;
ctx->eops = &pidfs_export_operations;
diff --git a/fs/pipe.c b/fs/pipe.c
index 9841648..429b071 100644
--- a/fs/pipe.c
+++ b/fs/pipe.c
@@ -111,16 +111,76 @@ void pipe_double_lock(struct pipe_inode_info *pipe1,
pipe_lock(pipe2);
}
-static struct page *anon_pipe_get_page(struct pipe_inode_info *pipe)
+#define PIPE_PREALLOC_MAX 8
+
+struct anon_pipe_prealloc {
+ struct page *pages[PIPE_PREALLOC_MAX];
+ unsigned int count;
+};
+
+/*
+ * Pre-allocate pages outside pipe->mutex for multi-page writes.
+ * alloc_page() with GFP_HIGHUSER can sleep in reclaim and runs memcg
+ * charging; doing it under the mutex stalls a concurrent reader.
+ *
+ * Loop alloc_page() instead of alloc_pages_bulk_*(): the bulk path refuses
+ * __GFP_ACCOUNT under memcg (see commit 8dcb3060d81d "memcg: page_alloc:
+ * skip bulk allocator for __GFP_ACCOUNT") and silently degrades to a single
+ * page. A per-page loop keeps memcg accounting and the task NUMA mempolicy
+ * honoured for every page; the per-call overhead is small compared to the
+ * pipe->mutex hold-time being shrunk. Any shortfall is covered by the
+ * in-lock alloc_page() fallback in anon_pipe_get_page().
+ */
+static void anon_pipe_get_page_prealloc(struct anon_pipe_prealloc *prealloc,
+ size_t total_len)
{
+ unsigned int want, i;
+ struct page *page;
+
+ prealloc->count = 0;
+ if (total_len <= PAGE_SIZE)
+ return;
+
+ want = min_t(unsigned int, DIV_ROUND_UP(total_len, PAGE_SIZE),
+ PIPE_PREALLOC_MAX);
+
+ for (i = 0; i < want; i++) {
+ page = alloc_page(GFP_HIGHUSER | __GFP_ACCOUNT);
+ if (!page)
+ break;
+ prealloc->pages[prealloc->count++] = page;
+ }
+}
+
+static struct page *anon_pipe_prealloc_pop(struct anon_pipe_prealloc *prealloc)
+{
+ if (!prealloc->count)
+ return NULL;
+
+ prealloc->count--;
+
+ return prealloc->pages[prealloc->count];
+}
+
+static struct page *anon_pipe_get_page(struct pipe_inode_info *pipe,
+ struct anon_pipe_prealloc *prealloc)
+{
+ struct page *page;
+
+ /* Drain prealloc first to keep tmp_page[] hot for later small writes. */
+ page = anon_pipe_prealloc_pop(prealloc);
+ if (page)
+ return page;
+
for (int i = 0; i < ARRAY_SIZE(pipe->tmp_page); i++) {
if (pipe->tmp_page[i]) {
- struct page *page = pipe->tmp_page[i];
+ page = pipe->tmp_page[i];
pipe->tmp_page[i] = NULL;
return page;
}
}
+ /* FWIW: This is called with pipe->mutex held */
return alloc_page(GFP_HIGHUSER | __GFP_ACCOUNT);
}
@@ -139,6 +199,38 @@ static void anon_pipe_put_page(struct pipe_inode_info *pipe,
put_page(page);
}
+/*
+ * Stash leftover prealloc pages in tmp_page[] so the next write to this
+ * pipe gets a hot page without entering the allocator.
+ */
+static void anon_pipe_refill_tmp_pages(struct pipe_inode_info *pipe,
+ struct anon_pipe_prealloc *prealloc)
+{
+ int i, idx;
+
+ if (!prealloc->count)
+ return;
+
+ for (i = 0; i < ARRAY_SIZE(pipe->tmp_page); i++) {
+ if (pipe->tmp_page[i])
+ continue;
+ if (!prealloc->count)
+ return;
+ idx = --prealloc->count;
+ pipe->tmp_page[i] = prealloc->pages[idx];
+ prealloc->pages[idx] = NULL;
+ }
+}
+
+/* Runs after mutex_unlock() to keep put_page() out of the critical section. */
+static void anon_pipe_free_pages(struct anon_pipe_prealloc *prealloc)
+{
+ while (prealloc->count) {
+ prealloc->count--;
+ put_page(prealloc->pages[prealloc->count]);
+ }
+}
+
static void anon_pipe_buf_release(struct pipe_inode_info *pipe,
struct pipe_buffer *buf)
{
@@ -432,6 +524,7 @@ anon_pipe_write(struct kiocb *iocb, struct iov_iter *from)
{
struct file *filp = iocb->ki_filp;
struct pipe_inode_info *pipe = filp->private_data;
+ struct anon_pipe_prealloc prealloc;
unsigned int head;
ssize_t ret = 0;
size_t total_len = iov_iter_count(from);
@@ -455,6 +548,8 @@ anon_pipe_write(struct kiocb *iocb, struct iov_iter *from)
if (unlikely(total_len == 0))
return 0;
+ anon_pipe_get_page_prealloc(&prealloc, total_len);
+
mutex_lock(&pipe->mutex);
if (!pipe->readers) {
@@ -512,7 +607,7 @@ anon_pipe_write(struct kiocb *iocb, struct iov_iter *from)
struct page *page;
int copied;
- page = anon_pipe_get_page(pipe);
+ page = anon_pipe_get_page(pipe, &prealloc);
if (unlikely(!page)) {
if (!ret)
ret = -ENOMEM;
@@ -576,9 +671,11 @@ anon_pipe_write(struct kiocb *iocb, struct iov_iter *from)
wake_next_writer = true;
}
out:
+ anon_pipe_refill_tmp_pages(pipe, &prealloc);
if (pipe_is_full(pipe))
wake_next_writer = false;
mutex_unlock(&pipe->mutex);
+ anon_pipe_free_pages(&prealloc);
/*
* If we do do a wakeup event, we do a 'sync' wakeup, because we
@@ -664,7 +761,8 @@ pipe_poll(struct file *filp, poll_table *wait)
union pipe_index idx;
/* Epoll has some historical nasty semantics, this enables them */
- WRITE_ONCE(pipe->poll_usage, true);
+ if (unlikely(!READ_ONCE(pipe->poll_usage)))
+ WRITE_ONCE(pipe->poll_usage, true);
/*
* Reading pipe state only -- no need for acquiring the semaphore.
diff --git a/fs/posix_acl.c b/fs/posix_acl.c
index 12591c9..b4bfe4d 100644
--- a/fs/posix_acl.c
+++ b/fs/posix_acl.c
@@ -1126,7 +1126,7 @@ int vfs_set_acl(struct mnt_idmap *idmap, struct dentry *dentry,
if (error)
goto out_inode_unlock;
- error = try_break_deleg(inode, &delegated_inode);
+ error = try_break_deleg(inode, 0, &delegated_inode);
if (error)
goto out_inode_unlock;
@@ -1234,7 +1234,7 @@ int vfs_remove_acl(struct mnt_idmap *idmap, struct dentry *dentry,
if (error)
goto out_inode_unlock;
- error = try_break_deleg(inode, &delegated_inode);
+ error = try_break_deleg(inode, 0, &delegated_inode);
if (error)
goto out_inode_unlock;
diff --git a/fs/proc/array.c b/fs/proc/array.c
index 90fb0c6..479ea8c 100644
--- a/fs/proc/array.c
+++ b/fs/proc/array.c
@@ -482,6 +482,11 @@ static int do_task_stat(struct seq_file *m, struct pid_namespace *ns,
unsigned long flags;
int exit_code = task->exit_code;
struct signal_struct *sig = task->signal;
+ int ret;
+
+ ret = down_read_killable(&task->signal->exec_update_lock);
+ if (ret)
+ return ret;
state = *get_task_state(task);
vsize = eip = esp = 0;
@@ -657,6 +662,7 @@ static int do_task_stat(struct seq_file *m, struct pid_namespace *ns,
seq_puts(m, " 0");
seq_putc(m, '\n');
+ up_read(&task->signal->exec_update_lock);
if (mm)
mmput(mm);
return 0;
diff --git a/fs/proc/base.c b/fs/proc/base.c
index d9acfa8..aae47c6 100644
--- a/fs/proc/base.c
+++ b/fs/proc/base.c
@@ -91,6 +91,7 @@
#include <linux/sched/mm.h>
#include <linux/sched/coredump.h>
#include <linux/sched/debug.h>
+#include <linux/sched/exec_state.h>
#include <linux/sched/stat.h>
#include <linux/posix-timers.h>
#include <linux/time_namespace.h>
@@ -218,33 +219,24 @@ static int get_task_root(struct task_struct *task, struct path *root)
return result;
}
-static int proc_cwd_link(struct dentry *dentry, struct path *path)
+static int proc_cwd_link(struct dentry *dentry, struct path *path,
+ struct task_struct *task)
{
- struct task_struct *task = get_proc_task(d_inode(dentry));
int result = -ENOENT;
- if (task) {
- task_lock(task);
- if (task->fs) {
- get_fs_pwd(task->fs, path);
- result = 0;
- }
- task_unlock(task);
- put_task_struct(task);
+ task_lock(task);
+ if (task->fs) {
+ get_fs_pwd(task->fs, path);
+ result = 0;
}
+ task_unlock(task);
return result;
}
-static int proc_root_link(struct dentry *dentry, struct path *path)
+static int proc_root_link(struct dentry *dentry, struct path *path,
+ struct task_struct *task)
{
- struct task_struct *task = get_proc_task(d_inode(dentry));
- int result = -ENOENT;
-
- if (task) {
- result = get_task_root(task, path);
- put_task_struct(task);
- }
- return result;
+ return get_task_root(task, path);
}
/*
@@ -261,7 +253,7 @@ static ssize_t get_mm_proctitle(struct mm_struct *mm, char __user *buf,
if (pos >= PAGE_SIZE)
return 0;
- page = (char *)__get_free_page(GFP_KERNEL);
+ page = kmalloc(PAGE_SIZE, GFP_KERNEL);
if (!page)
return -ENOMEM;
@@ -284,7 +276,7 @@ static ssize_t get_mm_proctitle(struct mm_struct *mm, char __user *buf,
ret = len;
}
}
- free_page((unsigned long)page);
+ kfree(page);
return ret;
}
@@ -347,7 +339,7 @@ static ssize_t get_mm_cmdline(struct mm_struct *mm, char __user *buf,
if (count > arg_end - pos)
count = arg_end - pos;
- page = (char *)__get_free_page(GFP_KERNEL);
+ page = kmalloc(PAGE_SIZE, GFP_KERNEL);
if (!page)
return -ENOMEM;
@@ -371,7 +363,7 @@ static ssize_t get_mm_cmdline(struct mm_struct *mm, char __user *buf,
count -= got;
}
- free_page((unsigned long)page);
+ kfree(page);
return len;
}
@@ -423,18 +415,24 @@ static int proc_pid_wchan(struct seq_file *m, struct pid_namespace *ns,
{
unsigned long wchan;
char symname[KSYM_NAME_LEN];
+ int err;
+ err = down_read_killable(&task->signal->exec_update_lock);
+ if (err)
+ return err;
if (!ptrace_may_access(task, PTRACE_MODE_READ_FSCREDS))
goto print0;
wchan = get_wchan(task);
if (wchan && !lookup_symbol_name(wchan, symname)) {
seq_puts(m, symname);
+ up_read(&task->signal->exec_update_lock);
return 0;
}
print0:
seq_putc(m, '0');
+ up_read(&task->signal->exec_update_lock);
return 0;
}
#endif /* CONFIG_KALLSYMS */
@@ -704,23 +702,6 @@ static int proc_pid_syscall(struct seq_file *m, struct pid_namespace *ns,
/* Here the fs part begins */
/************************************************************************/
-/* permission checks */
-static bool proc_fd_access_allowed(struct inode *inode)
-{
- struct task_struct *task;
- bool allowed = false;
- /* Allow access to a task's file descriptors if it is us or we
- * may use ptrace attach to the process and find out that
- * information.
- */
- task = get_proc_task(inode);
- if (task) {
- allowed = ptrace_may_access(task, PTRACE_MODE_READ_FSCREDS);
- put_task_struct(task);
- }
- return allowed;
-}
-
int proc_nochmod_setattr(struct mnt_idmap *idmap, struct dentry *dentry,
struct iattr *attr)
{
@@ -908,7 +889,7 @@ static ssize_t mem_rw(struct file *file, char __user *buf,
if (!mm)
return 0;
- page = (char *)__get_free_page(GFP_KERNEL);
+ page = kmalloc(PAGE_SIZE, GFP_KERNEL);
if (!page)
return -ENOMEM;
@@ -949,7 +930,7 @@ static ssize_t mem_rw(struct file *file, char __user *buf,
mmput(mm);
free:
- free_page((unsigned long) page);
+ kfree(page);
return copied;
}
@@ -1016,7 +997,7 @@ static ssize_t environ_read(struct file *file, char __user *buf,
if (!mm || !mm->env_end)
return 0;
- page = (char *)__get_free_page(GFP_KERNEL);
+ page = kmalloc(PAGE_SIZE, GFP_KERNEL);
if (!page)
return -ENOMEM;
@@ -1062,7 +1043,7 @@ static ssize_t environ_read(struct file *file, char __user *buf,
mmput(mm);
free:
- free_page((unsigned long) page);
+ kfree(page);
return ret;
}
@@ -1777,16 +1758,12 @@ static const struct file_operations proc_pid_set_comm_operations = {
.release = single_release,
};
-static int proc_exe_link(struct dentry *dentry, struct path *exe_path)
+static int proc_exe_link(struct dentry *dentry, struct path *exe_path,
+ struct task_struct *task)
{
- struct task_struct *task;
struct file *exe_file;
- task = get_proc_task(d_inode(dentry));
- if (!task)
- return -ENOENT;
exe_file = get_task_exe_file(task);
- put_task_struct(task);
if (exe_file) {
*exe_path = exe_file->f_path;
path_get(&exe_file->f_path);
@@ -1796,26 +1773,42 @@ static int proc_exe_link(struct dentry *dentry, struct path *exe_path)
return -ENOENT;
}
+static int call_proc_get_link(struct dentry *dentry, struct inode *inode, struct path *path_out)
+{
+ struct task_struct *task;
+ int ret;
+
+ task = get_proc_task(inode);
+ if (!task)
+ return -ENOENT;
+ ret = down_read_killable(&task->signal->exec_update_lock);
+ if (ret)
+ goto out_put_task;
+ if (!ptrace_may_access(task, PTRACE_MODE_READ_FSCREDS)) {
+ ret = -EACCES;
+ goto out;
+ }
+ ret = PROC_I(inode)->op.proc_get_link(dentry, path_out, task);
+
+out:
+ up_read(&task->signal->exec_update_lock);
+out_put_task:
+ put_task_struct(task);
+ return ret;
+}
+
static const char *proc_pid_get_link(struct dentry *dentry,
struct inode *inode,
struct delayed_call *done)
{
struct path path;
- int error = -EACCES;
+ int error;
if (!dentry)
return ERR_PTR(-ECHILD);
-
- /* Are we allowed to snoop on the tasks file descriptors? */
- if (!proc_fd_access_allowed(inode))
- goto out;
-
- error = PROC_I(inode)->op.proc_get_link(dentry, &path);
- if (error)
- goto out;
-
- error = nd_jump_link(&path);
-out:
+ error = call_proc_get_link(dentry, inode, &path);
+ if (!error)
+ error = nd_jump_link(&path);
return ERR_PTR(error);
}
@@ -1849,17 +1842,11 @@ static int proc_pid_readlink(struct dentry * dentry, char __user * buffer, int b
struct inode *inode = d_inode(dentry);
struct path path;
- /* Are we allowed to snoop on the tasks file descriptors? */
- if (!proc_fd_access_allowed(inode))
- goto out;
-
- error = PROC_I(inode)->op.proc_get_link(dentry, &path);
- if (error)
- goto out;
-
- error = do_proc_readlink(&path, buffer, buflen);
- path_put(&path);
-out:
+ error = call_proc_get_link(dentry, inode, &path);
+ if (!error) {
+ error = do_proc_readlink(&path, buffer, buflen);
+ path_put(&path);
+ }
return error;
}
@@ -1893,7 +1880,6 @@ void task_dump_owner(struct task_struct *task, umode_t mode,
cred = __task_cred(task);
uid = cred->euid;
gid = cred->egid;
- rcu_read_unlock();
/*
* Before the /proc/pid/status file was created the only way to read
@@ -1903,29 +1889,22 @@ void task_dump_owner(struct task_struct *task, umode_t mode,
* made this apply to all per process world readable and executable
* directories.
*/
- if (mode != (S_IFDIR|S_IRUGO|S_IXUGO)) {
- struct mm_struct *mm;
- task_lock(task);
- mm = task->mm;
- /* Make non-dumpable tasks owned by some root */
- if (mm) {
- if (get_dumpable(mm) != SUID_DUMP_USER) {
- struct user_namespace *user_ns = mm->user_ns;
+ if (mode != (S_IFDIR | S_IRUGO | S_IXUGO)) {
+ struct task_exec_state *exec_state;
- uid = make_kuid(user_ns, 0);
- if (!uid_valid(uid))
- uid = GLOBAL_ROOT_UID;
+ exec_state = task_exec_state_rcu(task);
+ if (READ_ONCE(exec_state->dumpable) != TASK_DUMPABLE_OWNER) {
+ uid = make_kuid(exec_state->user_ns, 0);
+ if (!uid_valid(uid))
+ uid = GLOBAL_ROOT_UID;
- gid = make_kgid(user_ns, 0);
- if (!gid_valid(gid))
- gid = GLOBAL_ROOT_GID;
- }
- } else {
- uid = GLOBAL_ROOT_UID;
- gid = GLOBAL_ROOT_GID;
+ gid = make_kgid(exec_state->user_ns, 0);
+ if (!gid_valid(gid))
+ gid = GLOBAL_ROOT_GID;
}
- task_unlock(task);
}
+ rcu_read_unlock();
+
*ruid = uid;
*rgid = gid;
}
@@ -2132,8 +2111,7 @@ bool proc_fill_cache(struct file *file, struct dir_context *ctx,
goto end_instantiate;
if (!child) {
- DECLARE_WAIT_QUEUE_HEAD_ONSTACK(wq);
- child = d_alloc_parallel(dir, &qname, &wq);
+ child = d_alloc_parallel(dir, &qname);
if (IS_ERR(child))
goto end_instantiate;
if (d_in_lookup(child)) {
@@ -2250,21 +2228,16 @@ static const struct dentry_operations tid_map_files_dentry_operations = {
.d_delete = pid_delete_dentry,
};
-static int map_files_get_link(struct dentry *dentry, struct path *path)
+static int map_files_get_link(struct dentry *dentry, struct path *path,
+ struct task_struct *task)
{
unsigned long vm_start, vm_end;
struct vm_area_struct *vma;
- struct task_struct *task;
struct mm_struct *mm;
int rc;
rc = -ENOENT;
- task = get_proc_task(d_inode(dentry));
- if (!task)
- goto out;
-
mm = get_task_mm(task);
- put_task_struct(task);
if (!mm)
goto out;
@@ -2360,17 +2333,15 @@ static struct dentry *proc_map_files_lookup(struct inode *dir,
if (!task)
goto out;
- result = ERR_PTR(-EACCES);
- if (!ptrace_may_access(task, PTRACE_MODE_READ_FSCREDS))
- goto out_put_task;
-
result = ERR_PTR(-ENOENT);
if (dname_to_vma_addr(dentry, &vm_start, &vm_end))
goto out_put_task;
- mm = get_task_mm(task);
- if (!mm)
+ mm = mm_access(task, PTRACE_MODE_READ_FSCREDS);
+ if (IS_ERR(mm)) {
+ result = ERR_CAST(mm);
goto out_put_task;
+ }
result = ERR_PTR(-EINTR);
if (mmap_read_lock_killable(mm))
@@ -2420,24 +2391,23 @@ proc_map_files_readdir(struct file *file, struct dir_context *ctx)
if (!task)
goto out;
- ret = -EACCES;
- if (!ptrace_may_access(task, PTRACE_MODE_READ_FSCREDS))
- goto out_put_task;
-
ret = 0;
if (!dir_emit_dots(file, ctx))
goto out_put_task;
- mm = get_task_mm(task);
- if (!mm)
- goto out_put_task;
-
- ret = mmap_read_lock_killable(mm);
- if (ret) {
- mmput(mm);
+ mm = mm_access(task, PTRACE_MODE_READ_FSCREDS);
+ if (IS_ERR(mm)) {
+ ret = PTR_ERR(mm);
+ /* if the task has no mm, the directory should just be empty */
+ if (ret == -ESRCH)
+ ret = 0;
goto out_put_task;
}
+ ret = mmap_read_lock_killable(mm);
+ if (ret)
+ goto out_put_mm;
+
nr_files = 0;
/*
@@ -2462,8 +2432,7 @@ proc_map_files_readdir(struct file *file, struct dir_context *ctx)
if (!p) {
ret = -ENOMEM;
mmap_read_unlock(mm);
- mmput(mm);
- goto out_put_task;
+ goto out_put_mm;
}
p->start = vma->vm_start;
@@ -2471,7 +2440,6 @@ proc_map_files_readdir(struct file *file, struct dir_context *ctx)
p->mode = vma->vm_file->f_mode;
}
mmap_read_unlock(mm);
- mmput(mm);
for (i = 0; i < nr_files; i++) {
char buf[4 * sizeof(long) + 2]; /* max: %lx-%lx\0 */
@@ -2488,6 +2456,8 @@ proc_map_files_readdir(struct file *file, struct dir_context *ctx)
ctx->pos++;
}
+out_put_mm:
+ mmput(mm);
out_put_task:
put_task_struct(task);
out:
@@ -2965,7 +2935,7 @@ static ssize_t proc_coredump_filter_read(struct file *file, char __user *buf,
ret = 0;
mm = get_task_mm(task);
if (mm) {
- unsigned long flags = __mm_flags_get_dumpable(mm);
+ unsigned long flags = __mm_flags_get_word(mm);
len = snprintf(buffer, sizeof(buffer), "%08lx\n",
((flags & MMF_DUMP_FILTER_MASK) >>
diff --git a/fs/proc/fd.c b/fs/proc/fd.c
index 05c7513..0f9a155 100644
--- a/fs/proc/fd.c
+++ b/fs/proc/fd.c
@@ -171,24 +171,19 @@ static const struct dentry_operations tid_fd_dentry_operations = {
.d_delete = pid_delete_dentry,
};
-static int proc_fd_link(struct dentry *dentry, struct path *path)
+static int proc_fd_link(struct dentry *dentry, struct path *path,
+ struct task_struct *task)
{
- struct task_struct *task;
int ret = -ENOENT;
+ unsigned int fd = proc_fd(d_inode(dentry));
+ struct file *fd_file;
- task = get_proc_task(d_inode(dentry));
- if (task) {
- unsigned int fd = proc_fd(d_inode(dentry));
- struct file *fd_file;
-
- fd_file = fget_task(task, fd);
- if (fd_file) {
- *path = fd_file->f_path;
- path_get(&fd_file->f_path);
- ret = 0;
- fput(fd_file);
- }
- put_task_struct(task);
+ fd_file = fget_task(task, fd);
+ if (fd_file) {
+ *path = fd_file->f_path;
+ path_get(&fd_file->f_path);
+ ret = 0;
+ fput(fd_file);
}
return ret;
diff --git a/fs/proc/generic.c b/fs/proc/generic.c
index 8bb81e5..c6ae076 100644
--- a/fs/proc/generic.c
+++ b/fs/proc/generic.c
@@ -841,3 +841,13 @@ ssize_t proc_simple_write(struct file *f, const char __user *ubuf, size_t size,
kfree(buf);
return ret == 0 ? size : ret;
}
+
+/*
+ * Not exported to modules:
+ * modules' /proc files aren't permanent because modules aren't permanent.
+ */
+void impl_proc_make_permanent(struct proc_dir_entry *pde)
+{
+ if (pde)
+ pde_make_permanent(pde);
+}
diff --git a/fs/proc/internal.h b/fs/proc/internal.h
index 64dc448..b232e10 100644
--- a/fs/proc/internal.h
+++ b/fs/proc/internal.h
@@ -79,8 +79,11 @@ static inline bool pde_is_permanent(const struct proc_dir_entry *pde)
return pde->flags & PROC_ENTRY_PERMANENT;
}
+/* This is for builtin code, not even for modules which are compiled in. */
static inline void pde_make_permanent(struct proc_dir_entry *pde)
{
+ /* Ensure magic flag does something. */
+ static_assert(PROC_ENTRY_PERMANENT != 0);
pde->flags |= PROC_ENTRY_PERMANENT;
}
@@ -107,7 +110,7 @@ extern struct kmem_cache *proc_dir_entry_cache;
void pde_free(struct proc_dir_entry *pde);
union proc_op {
- int (*proc_get_link)(struct dentry *, struct path *);
+ int (*proc_get_link)(struct dentry *, struct path *, struct task_struct *);
int (*proc_show)(struct seq_file *m,
struct pid_namespace *ns, struct pid *pid,
struct task_struct *task);
diff --git a/fs/proc/namespaces.c b/fs/proc/namespaces.c
index 39f4169f..2f46f13 100644
--- a/fs/proc/namespaces.c
+++ b/fs/proc/namespaces.c
@@ -55,6 +55,10 @@ static const char *proc_ns_get_link(struct dentry *dentry,
if (!task)
return ERR_PTR(-EACCES);
+ error = down_read_killable(&task->signal->exec_update_lock);
+ if (error)
+ goto out_put_task;
+
if (!ptrace_may_access(task, PTRACE_MODE_READ_FSCREDS))
goto out;
@@ -64,6 +68,8 @@ static const char *proc_ns_get_link(struct dentry *dentry,
error = nd_jump_link(&ns_path);
out:
+ up_read(&task->signal->exec_update_lock);
+out_put_task:
put_task_struct(task);
return ERR_PTR(error);
}
@@ -80,11 +86,17 @@ static int proc_ns_readlink(struct dentry *dentry, char __user *buffer, int bufl
if (!task)
return res;
+ res = down_read_killable(&task->signal->exec_update_lock);
+ if (res)
+ goto out_put_task;
+
if (ptrace_may_access(task, PTRACE_MODE_READ_FSCREDS)) {
res = ns_get_name(name, sizeof(name), task, ns_ops);
if (res >= 0)
res = readlink_copy(buffer, buflen, name, strlen(name));
}
+ up_read(&task->signal->exec_update_lock);
+out_put_task:
put_task_struct(task);
return res;
}
diff --git a/fs/proc/proc_net.c b/fs/proc/proc_net.c
index 184cdde..00cc385 100644
--- a/fs/proc/proc_net.c
+++ b/fs/proc/proc_net.c
@@ -23,6 +23,7 @@
#include <linux/uidgid.h>
#include <net/net_namespace.h>
#include <linux/seq_file.h>
+#include <linux/security.h>
#include "internal.h"
@@ -270,6 +271,7 @@ static struct net *get_proc_task_net(struct inode *dir)
struct task_struct *task;
struct nsproxy *ns;
struct net *net = NULL;
+ struct proc_fs_info *fs_info = proc_sb_info(dir->i_sb);
rcu_read_lock();
task = pid_task(proc_pid(dir), PIDTYPE_PID);
@@ -282,6 +284,12 @@ static struct net *get_proc_task_net(struct inode *dir)
}
rcu_read_unlock();
+ if (net && (fs_info->pidonly == PROC_PIDONLY_ON) &&
+ security_capable(fs_info->mounter_cred, net->user_ns, CAP_NET_ADMIN, CAP_OPT_NONE) < 0) {
+ put_net(net);
+ net = NULL;
+ }
+
return net;
}
diff --git a/fs/proc/proc_sysctl.c b/fs/proc/proc_sysctl.c
index 49ab74e..04a3821 100644
--- a/fs/proc/proc_sysctl.c
+++ b/fs/proc/proc_sysctl.c
@@ -692,8 +692,7 @@ static bool proc_sys_fill_cache(struct file *file,
child = d_lookup(dir, &qname);
if (!child) {
- DECLARE_WAIT_QUEUE_HEAD_ONSTACK(wq);
- child = d_alloc_parallel(dir, &qname, &wq);
+ child = d_alloc_parallel(dir, &qname);
if (IS_ERR(child))
return false;
if (d_in_lookup(child)) {
diff --git a/fs/proc/root.c b/fs/proc/root.c
index 0f91005..99adddf 100644
--- a/fs/proc/root.c
+++ b/fs/proc/root.c
@@ -223,12 +223,17 @@ static int proc_parse_param(struct fs_context *fc, struct fs_parameter *param)
return 0;
}
-static void proc_apply_options(struct proc_fs_info *fs_info,
+static int proc_apply_options(struct proc_fs_info *fs_info,
struct fs_context *fc,
struct user_namespace *user_ns)
{
struct proc_fs_context *ctx = fc->fs_private;
+ if ((ctx->mask & (1 << Opt_subset)) &&
+ fc->purpose == FS_CONTEXT_FOR_RECONFIGURE &&
+ ctx->pidonly != fs_info->pidonly)
+ return invalf(fc, "proc: subset=pid cannot be changed\n");
+
if (ctx->mask & (1 << Opt_gid))
fs_info->pid_gid = make_kgid(user_ns, ctx->gid);
if (ctx->mask & (1 << Opt_hidepid))
@@ -240,6 +245,7 @@ static void proc_apply_options(struct proc_fs_info *fs_info,
put_pid_ns(fs_info->pid_ns);
fs_info->pid_ns = get_pid_ns(ctx->pid_ns);
}
+ return 0;
}
static int proc_fill_super(struct super_block *s, struct fs_context *fc)
@@ -254,10 +260,13 @@ static int proc_fill_super(struct super_block *s, struct fs_context *fc)
return -ENOMEM;
fs_info->pid_ns = get_pid_ns(ctx->pid_ns);
- proc_apply_options(fs_info, fc, current_user_ns());
+ fs_info->mounter_cred = get_cred(fc->cred);
+ ret = proc_apply_options(fs_info, fc, current_user_ns());
+ if (ret)
+ return ret;
/* User space would break if executables or devices appear on proc */
- s->s_iflags |= SB_I_USERNS_VISIBLE | SB_I_NOEXEC | SB_I_NODEV;
+ s->s_iflags |= SB_I_NOEXEC | SB_I_NODEV;
s->s_flags |= SB_NODIRATIME | SB_NOSUID | SB_NOEXEC;
s->s_blocksize = 1024;
s->s_blocksize_bits = 10;
@@ -266,6 +275,9 @@ static int proc_fill_super(struct super_block *s, struct fs_context *fc)
s->s_time_gran = 1;
s->s_fs_info = fs_info;
+ if (fs_info->pidonly == PROC_PIDONLY_ON)
+ s->s_iflags |= SB_I_RESTRICTED_VARIANT;
+
/*
* procfs isn't actually a stacking filesystem; however, there is
* too much magic going on inside it to permit stacking things on
@@ -303,8 +315,7 @@ static int proc_reconfigure(struct fs_context *fc)
sync_filesystem(sb);
- proc_apply_options(fs_info, fc, current_user_ns());
- return 0;
+ return proc_apply_options(fs_info, fc, current_user_ns());
}
static int proc_get_tree(struct fs_context *fc)
@@ -350,6 +361,7 @@ static void proc_kill_sb(struct super_block *sb)
kill_anon_super(sb);
if (fs_info) {
put_pid_ns(fs_info->pid_ns);
+ put_cred(fs_info->mounter_cred);
kfree_rcu(fs_info, rcu);
}
}
@@ -359,7 +371,7 @@ static struct file_system_type proc_fs_type = {
.init_fs_context = proc_init_fs_context,
.parameters = proc_fs_parameters,
.kill_sb = proc_kill_sb,
- .fs_flags = FS_USERNS_MOUNT | FS_DISALLOW_NOTIFY_PERM,
+ .fs_flags = FS_USERNS_MOUNT | FS_USERNS_MOUNT_RESTRICTED | FS_DISALLOW_NOTIFY_PERM,
};
void __init proc_root_init(void)
diff --git a/fs/qnx4/inode.c b/fs/qnx4/inode.c
index 4deb0ee..42fcd500 100644
--- a/fs/qnx4/inode.c
+++ b/fs/qnx4/inode.c
@@ -202,7 +202,8 @@ static int qnx4_fill_super(struct super_block *s, struct fs_context *fc)
return -ENOMEM;
s->s_fs_info = qs;
- sb_set_blocksize(s, QNX4_BLOCK_SIZE);
+ if (!sb_set_blocksize(s, QNX4_BLOCK_SIZE))
+ return -EINVAL;
s->s_op = &qnx4_sops;
s->s_magic = QNX4_SUPER_MAGIC;
diff --git a/fs/quota/dquot.c b/fs/quota/dquot.c
index 64cf427..9850de3 100644
--- a/fs/quota/dquot.c
+++ b/fs/quota/dquot.c
@@ -3022,7 +3022,7 @@ static const struct ctl_table fs_dqstats_table[] = {
static int __init dquot_init(void)
{
int i, ret;
- unsigned long nr_hash, order;
+ unsigned long nr_hash;
struct shrinker *dqcache_shrinker;
printk(KERN_NOTICE "VFS: Disk quotas %s\n", __DQUOT_VERSION__);
@@ -3035,8 +3035,7 @@ static int __init dquot_init(void)
SLAB_PANIC),
NULL);
- order = 0;
- dquot_hash = (struct hlist_head *)__get_free_pages(GFP_KERNEL, order);
+ dquot_hash = kmalloc(PAGE_SIZE, GFP_KERNEL);
if (!dquot_hash)
panic("Cannot create dquot hash table");
@@ -3046,7 +3045,7 @@ static int __init dquot_init(void)
panic("Cannot create dquot stat counters");
/* Find power-of-two hlist_heads which can fit into allocation */
- nr_hash = (1UL << order) * PAGE_SIZE / sizeof(struct hlist_head);
+ nr_hash = PAGE_SIZE / sizeof(struct hlist_head);
dq_hash_bits = ilog2(nr_hash);
nr_hash = 1UL << dq_hash_bits;
@@ -3054,8 +3053,8 @@ static int __init dquot_init(void)
for (i = 0; i < nr_hash; i++)
INIT_HLIST_HEAD(dquot_hash + i);
- pr_info("VFS: Dquot-cache hash table entries: %ld (order %ld,"
- " %ld bytes)\n", nr_hash, order, (PAGE_SIZE << order));
+ pr_info("VFS: Dquot-cache hash table entries: %ld (%ld bytes)\n",
+ nr_hash, PAGE_SIZE);
dqcache_shrinker = shrinker_alloc(0, "dquota-cache");
if (!dqcache_shrinker)
diff --git a/fs/read_write.c b/fs/read_write.c
index 50bff7e..e8c14e2 100644
--- a/fs/read_write.c
+++ b/fs/read_write.c
@@ -641,13 +641,12 @@ ssize_t __kernel_write(struct file *file, const void *buf, size_t count, loff_t
return __kernel_write_iter(file, &iter, pos);
}
/*
- * This "EXPORT_SYMBOL_GPL()" is more of a "EXPORT_SYMBOL_DONTUSE()",
- * but autofs is one of the few internal kernel users that actually
+ * autofs is one of the few internal kernel users that actually
* wants this _and_ can be built as a module. So we need to export
* this symbol for autofs, even though it really isn't appropriate
* for any other kernel modules.
*/
-EXPORT_SYMBOL_GPL(__kernel_write);
+EXPORT_SYMBOL_FOR_MODULES(__kernel_write, "autofs4");
ssize_t kernel_write(struct file *file, const void *buf, size_t count,
loff_t *pos)
diff --git a/fs/select.c b/fs/select.c
index bf71c98..95d7653 100644
--- a/fs/select.c
+++ b/fs/select.c
@@ -150,7 +150,7 @@ void poll_freewait(struct poll_wqueues *pwq)
} while (entry > p->entries);
old = p;
p = p->next;
- free_page((unsigned long) old);
+ kfree(old);
}
}
EXPORT_SYMBOL(poll_freewait);
@@ -165,7 +165,7 @@ static struct poll_table_entry *poll_get_entry(struct poll_wqueues *p)
if (!table || POLL_TABLE_FULL(table)) {
struct poll_table_page *new_table;
- new_table = (struct poll_table_page *) __get_free_page(GFP_KERNEL);
+ new_table = kmalloc(PAGE_SIZE, GFP_KERNEL);
if (!new_table) {
p->error = -ENOMEM;
return NULL;
diff --git a/fs/smb/client/cifsfs.c b/fs/smb/client/cifsfs.c
index ce23924..834859f 100644
--- a/fs/smb/client/cifsfs.c
+++ b/fs/smb/client/cifsfs.c
@@ -12,6 +12,7 @@
#include <linux/module.h>
#include <linux/fs.h>
+#include <linux/fs_context.h>
#include <linux/filelock.h>
#include <linux/mount.h>
#include <linux/slab.h>
@@ -30,6 +31,7 @@
#include <linux/xattr.h>
#include <linux/mm.h>
#include <linux/key-type.h>
+#include <linux/fileattr.h>
#include <uapi/linux/magic.h>
#include <net/ipv6.h>
#include "cifsfs.h"
@@ -969,26 +971,19 @@ cifs_get_root(struct smb3_fs_context *ctx, struct super_block *sb)
return dentry;
}
-static int cifs_set_super(struct super_block *sb, void *data)
-{
- struct cifs_mnt_data *mnt_data = data;
- sb->s_fs_info = mnt_data->cifs_sb;
- return set_anon_super(sb, NULL);
-}
-
struct dentry *
-cifs_smb3_do_mount(struct file_system_type *fs_type,
- int flags, struct smb3_fs_context *old_ctx)
+cifs_smb3_do_mount(struct fs_context *fc, struct smb3_fs_context *old_ctx)
{
struct cifs_mnt_data mnt_data;
struct cifs_sb_info *cifs_sb;
struct super_block *sb;
struct dentry *root;
+ unsigned int saved_sb_flags;
int rc;
if (cifsFYI) {
- cifs_dbg(FYI, "%s: devname=%s flags=0x%x\n", __func__,
- old_ctx->source, flags);
+ cifs_dbg(FYI, "%s: devname=%s sb_flags=0x%x\n", __func__,
+ old_ctx->source, fc->sb_flags);
} else {
cifs_info("Attempting to mount %s\n", old_ctx->source);
}
@@ -1015,7 +1010,7 @@ cifs_smb3_do_mount(struct file_system_type *fs_type,
rc = cifs_mount(cifs_sb, cifs_sb->ctx);
if (rc) {
- if (!(flags & SB_SILENT))
+ if (!(fc->sb_flags & SB_SILENT))
cifs_dbg(VFS, "cifs_mount failed w/return code = %d\n",
rc);
root = ERR_PTR(rc);
@@ -1024,12 +1019,27 @@ cifs_smb3_do_mount(struct file_system_type *fs_type,
mnt_data.ctx = cifs_sb->ctx;
mnt_data.cifs_sb = cifs_sb;
- mnt_data.flags = flags;
+ mnt_data.flags = 0;
- /* BB should we make this contingent on mount parm? */
- flags |= SB_NODIRATIME | SB_NOATIME;
-
- sb = sget(fs_type, cifs_match_super, cifs_set_super, flags, &mnt_data);
+ /*
+ * sb->s_flags is set from fc->sb_flags by alloc_super(). CIFS has
+ * historically forced SB_NODIRATIME | SB_NOATIME on every mount and
+ * ignored the caller-supplied SB_* flags. Preserve that behaviour by
+ * overriding fc->sb_flags around the sget_fc() call.
+ *
+ * Hand cifs_sb to sget_fc() via fc->s_fs_info; sget_fc() copies it
+ * onto sb->s_fs_info before running set() and clears fc->s_fs_info
+ * on successful publish. Pass the rest of the per-mount context to
+ * cifs_match_super() through fc->sget_key.
+ */
+ saved_sb_flags = fc->sb_flags;
+ fc->sb_flags = SB_NODIRATIME | SB_NOATIME;
+ fc->s_fs_info = cifs_sb;
+ fc->sget_key = &mnt_data;
+ sb = sget_fc(fc, cifs_match_super, set_anon_super_fc);
+ fc->sget_key = NULL;
+ fc->s_fs_info = NULL;
+ fc->sb_flags = saved_sb_flags;
if (IS_ERR(sb)) {
cifs_umount(cifs_sb);
return ERR_CAST(sb);
@@ -1165,6 +1175,56 @@ struct file_system_type smb3_fs_type = {
MODULE_ALIAS_FS("smb3");
MODULE_ALIAS("smb3");
+int cifs_fileattr_get(struct dentry *dentry, struct file_kattr *fa)
+{
+ struct cifs_sb_info *cifs_sb = CIFS_SB(dentry->d_sb);
+ struct cifs_tcon *tcon = cifs_sb_master_tcon(cifs_sb);
+ struct inode *inode = d_inode(dentry);
+ u32 attrs;
+
+ /* Preserve FS_COMPR_FL previously reported by cifs_ioctl(). */
+ if (CIFS_I(inode)->cifsAttrs & ATTR_COMPRESSED)
+ fa->flags |= FS_COMPR_FL;
+
+ /*
+ * FS_CASEFOLD_FL is defined by UAPI as a folder attribute,
+ * and userspace tools (e.g., lsattr) display it only on
+ * directories. Confine the case-handling bits to directories
+ * to match that convention; for non-directories the share's
+ * case semantics are still discoverable through the parent.
+ */
+ if (!S_ISDIR(inode->i_mode))
+ return 0;
+
+ /*
+ * The server's FS_ATTRIBUTE_INFORMATION response, cached on
+ * the tcon at mount, reflects the share's case-handling
+ * semantics after any POSIX extensions negotiation. Prefer
+ * it over the client-local nocase mount option, which only
+ * governs dentry comparison on this superblock.
+ *
+ * QueryFSInfo is best-effort at mount; when it did not
+ * populate fsAttrInfo, MaxPathNameComponentLength remains
+ * zero. In that case fall back to nocase so the reporting
+ * matches the comparison behavior installed on the sb.
+ */
+ if (le32_to_cpu(tcon->fsAttrInfo.MaxPathNameComponentLength) == 0) {
+ if (tcon->nocase) {
+ fa->fsx_xflags |= FS_XFLAG_CASEFOLD;
+ fa->flags |= FS_CASEFOLD_FL;
+ }
+ return 0;
+ }
+ attrs = le32_to_cpu(tcon->fsAttrInfo.Attributes);
+ if (!(attrs & FILE_CASE_SENSITIVE_SEARCH)) {
+ fa->fsx_xflags |= FS_XFLAG_CASEFOLD;
+ fa->flags |= FS_CASEFOLD_FL;
+ }
+ if (!(attrs & FILE_CASE_PRESERVED_NAMES))
+ fa->fsx_xflags |= FS_XFLAG_CASENONPRESERVING;
+ return 0;
+}
+
const struct inode_operations cifs_dir_inode_ops = {
.create = cifs_create,
.atomic_open = cifs_atomic_open,
@@ -1183,6 +1243,7 @@ const struct inode_operations cifs_dir_inode_ops = {
.listxattr = cifs_listxattr,
.get_acl = cifs_get_acl,
.set_acl = cifs_set_acl,
+ .fileattr_get = cifs_fileattr_get,
};
const struct inode_operations cifs_file_inode_ops = {
@@ -1193,6 +1254,7 @@ const struct inode_operations cifs_file_inode_ops = {
.fiemap = cifs_fiemap,
.get_acl = cifs_get_acl,
.set_acl = cifs_set_acl,
+ .fileattr_get = cifs_fileattr_get,
};
const char *cifs_get_link(struct dentry *dentry, struct inode *inode,
diff --git a/fs/smb/client/cifsfs.h b/fs/smb/client/cifsfs.h
index c455b15..008b1d3 100644
--- a/fs/smb/client/cifsfs.h
+++ b/fs/smb/client/cifsfs.h
@@ -89,6 +89,9 @@ extern const struct inode_operations cifs_file_inode_ops;
extern const struct inode_operations cifs_symlink_inode_ops;
extern const struct inode_operations cifs_namespace_inode_operations;
+struct file_kattr;
+int cifs_fileattr_get(struct dentry *dentry, struct file_kattr *fa);
+
/* Functions related to files and directories */
extern const struct netfs_request_ops cifs_req_ops;
@@ -144,8 +147,9 @@ ssize_t cifs_file_copychunk_range(unsigned int xid, struct file *src_file,
long cifs_ioctl(struct file *filep, unsigned int command, unsigned long arg);
void cifs_setsize(struct inode *inode, loff_t offset);
+struct fs_context;
struct smb3_fs_context;
-struct dentry *cifs_smb3_do_mount(struct file_system_type *fs_type, int flags,
+struct dentry *cifs_smb3_do_mount(struct fs_context *fc,
struct smb3_fs_context *old_ctx);
char *cifs_silly_fullpath(struct dentry *dentry);
diff --git a/fs/smb/client/cifsproto.h b/fs/smb/client/cifsproto.h
index 79d891f..c4ababc 100644
--- a/fs/smb/client/cifsproto.h
+++ b/fs/smb/client/cifsproto.h
@@ -19,6 +19,7 @@
struct statfs;
struct smb_rqst;
struct smb3_fs_context;
+struct fs_context;
/*
*****************************************************************
@@ -235,7 +236,7 @@ void cifs_mount_put_conns(struct cifs_mount_ctx *mnt_ctx);
int cifs_mount_get_session(struct cifs_mount_ctx *mnt_ctx);
int cifs_is_path_remote(struct cifs_mount_ctx *mnt_ctx);
int cifs_mount_get_tcon(struct cifs_mount_ctx *mnt_ctx);
-int cifs_match_super(struct super_block *sb, void *data);
+int cifs_match_super(struct super_block *sb, struct fs_context *fc);
int cifs_mount(struct cifs_sb_info *cifs_sb, struct smb3_fs_context *ctx);
void cifs_umount(struct cifs_sb_info *cifs_sb);
void cifs_mark_open_files_invalid(struct cifs_tcon *tcon);
diff --git a/fs/smb/client/connect.c b/fs/smb/client/connect.c
index dcde25d..79762e6 100644
--- a/fs/smb/client/connect.c
+++ b/fs/smb/client/connect.c
@@ -6,6 +6,7 @@
*
*/
#include <linux/fs.h>
+#include <linux/fs_context.h>
#include <linux/net.h>
#include <linux/string.h>
#include <linux/sched/mm.h>
@@ -2991,9 +2992,9 @@ static int match_prepath(struct super_block *sb,
}
int
-cifs_match_super(struct super_block *sb, void *data)
+cifs_match_super(struct super_block *sb, struct fs_context *fc)
{
- struct cifs_mnt_data *mnt_data = data;
+ struct cifs_mnt_data *mnt_data = fc->sget_key;
struct smb3_fs_context *ctx;
struct cifs_sb_info *cifs_sb;
struct TCP_Server_Info *tcp_srv;
diff --git a/fs/smb/client/dir.c b/fs/smb/client/dir.c
index e4295a5..88a4a17 100644
--- a/fs/smb/client/dir.c
+++ b/fs/smb/client/dir.c
@@ -241,6 +241,12 @@ static int __cifs_do_create(struct inode *dir, struct dentry *direntry,
goto cifs_create_get_file_info;
}
+ if ((oflags & __O_REGULAR) && !S_ISREG(newinode->i_mode)) {
+ CIFSSMBClose(xid, tcon, fid->netfid);
+ iput(newinode);
+ return -EFTYPE;
+ }
+
if (S_ISDIR(newinode->i_mode)) {
CIFSSMBClose(xid, tcon, fid->netfid);
iput(newinode);
@@ -458,9 +464,15 @@ static int __cifs_do_create(struct inode *dir, struct dentry *direntry,
goto out_err;
}
- if (newinode && S_ISDIR(newinode->i_mode)) {
- rc = -EISDIR;
- goto out_err;
+ if (newinode) {
+ if ((oflags & __O_REGULAR) && !S_ISREG(newinode->i_mode)) {
+ rc = -EFTYPE;
+ goto out_err;
+ }
+ if (S_ISDIR(newinode->i_mode)) {
+ rc = -EISDIR;
+ goto out_err;
+ }
}
*inode = newinode;
diff --git a/fs/smb/client/fs_context.c b/fs/smb/client/fs_context.c
index 2f86158..9bb4111 100644
--- a/fs/smb/client/fs_context.c
+++ b/fs/smb/client/fs_context.c
@@ -889,7 +889,7 @@ static int smb3_get_tree_common(struct fs_context *fc)
struct dentry *root;
int rc = 0;
- root = cifs_smb3_do_mount(fc->fs_type, 0, ctx);
+ root = cifs_smb3_do_mount(fc, ctx);
if (IS_ERR(root))
return PTR_ERR(root);
diff --git a/fs/smb/client/inode.c b/fs/smb/client/inode.c
index 9472c0a..6c22fc3 100644
--- a/fs/smb/client/inode.c
+++ b/fs/smb/client/inode.c
@@ -2845,7 +2845,7 @@ int cifs_revalidate_dentry_attr(struct dentry *dentry)
}
cifs_dbg(FYI, "Update attributes: %s inode 0x%p count %d dentry: 0x%p d_time %ld jiffies %ld\n",
- full_path, inode, icount_read(inode),
+ full_path, inode, icount_read_once(inode),
dentry, cifs_get_time(dentry), jiffies);
again:
diff --git a/fs/smb/client/namespace.c b/fs/smb/client/namespace.c
index 52a5203..52a51b0 100644
--- a/fs/smb/client/namespace.c
+++ b/fs/smb/client/namespace.c
@@ -294,4 +294,5 @@ struct vfsmount *cifs_d_automount(struct path *path)
}
const struct inode_operations cifs_namespace_inode_operations = {
+ .fileattr_get = cifs_fileattr_get,
};
diff --git a/fs/smb/client/readdir.c b/fs/smb/client/readdir.c
index e860fa0..1ff77f3 100644
--- a/fs/smb/client/readdir.c
+++ b/fs/smb/client/readdir.c
@@ -73,7 +73,6 @@ cifs_prime_dcache(struct dentry *parent, struct qstr *name,
struct cifs_sb_info *cifs_sb = CIFS_SB(sb);
bool posix = cifs_sb_master_tcon(cifs_sb)->posix_extensions;
bool reparse_need_reval = false;
- DECLARE_WAIT_QUEUE_HEAD_ONSTACK(wq);
int rc;
cifs_dbg(FYI, "%s: for %s\n", __func__, name->name);
@@ -105,7 +104,7 @@ cifs_prime_dcache(struct dentry *parent, struct qstr *name,
(fattr->cf_flags & CIFS_FATTR_NEED_REVAL))
return;
- dentry = d_alloc_parallel(parent, name, &wq);
+ dentry = d_alloc_parallel(parent, name);
}
if (IS_ERR(dentry))
return;
diff --git a/fs/smb/server/smb2pdu.c b/fs/smb/server/smb2pdu.c
index 3eb3b17..00e63de 100644
--- a/fs/smb/server/smb2pdu.c
+++ b/fs/smb/server/smb2pdu.c
@@ -14,6 +14,7 @@
#include <linux/falloc.h>
#include <linux/mount.h>
#include <linux/filelock.h>
+#include <linux/fileattr.h>
#include "glob.h"
#include "smbfsctl.h"
@@ -5561,16 +5562,33 @@ static int smb2_get_info_filesystem(struct ksmbd_work *work,
case FS_ATTRIBUTE_INFORMATION:
{
FILE_SYSTEM_ATTRIBUTE_INFO *info;
+ struct file_kattr fa = {};
size_t sz;
+ u32 attrs;
+ int err;
info = (FILE_SYSTEM_ATTRIBUTE_INFO *)rsp->Buffer;
- info->Attributes = cpu_to_le32(FILE_SUPPORTS_OBJECT_IDS |
- FILE_PERSISTENT_ACLS |
- FILE_UNICODE_ON_DISK |
- FILE_CASE_PRESERVED_NAMES |
- FILE_CASE_SENSITIVE_SEARCH |
- FILE_SUPPORTS_BLOCK_REFCOUNTING);
+ attrs = FILE_SUPPORTS_OBJECT_IDS |
+ FILE_PERSISTENT_ACLS |
+ FILE_UNICODE_ON_DISK |
+ FILE_SUPPORTS_BLOCK_REFCOUNTING;
+ err = vfs_fileattr_get(path.dentry, &fa);
+ /*
+ * -EINVAL, -EOPNOTSUPP: ntfs-3g and other FUSE
+ * filesystems that lack FS_IOC_FSGETXATTR support.
+ */
+ if (err && err != -ENOIOCTLCMD && err != -ENOTTY &&
+ err != -EINVAL && err != -EOPNOTSUPP) {
+ path_put(&path);
+ return err;
+ }
+ if (!(fa.fsx_xflags & FS_XFLAG_CASEFOLD))
+ attrs |= FILE_CASE_SENSITIVE_SEARCH;
+ if (!(fa.fsx_xflags & FS_XFLAG_CASENONPRESERVING))
+ attrs |= FILE_CASE_PRESERVED_NAMES;
+
+ info->Attributes = cpu_to_le32(attrs);
info->Attributes |= cpu_to_le32(server_conf.share_fake_fscaps);
if (test_share_config_flag(work->tcon->share_conf,
diff --git a/fs/smb/server/vfs.c b/fs/smb/server/vfs.c
index d08973b..cd1dbca 100644
--- a/fs/smb/server/vfs.c
+++ b/fs/smb/server/vfs.c
@@ -19,7 +19,6 @@
#include <linux/vmalloc.h>
#include <linux/sched/xacct.h>
#include <linux/crc32c.h>
-#include <linux/namei.h>
#include <linux/splice.h>
#include "glob.h"
@@ -56,7 +55,7 @@ static int ksmbd_vfs_path_lookup(struct ksmbd_share_config *share_conf,
{
struct qstr last;
const struct path *root_share_path = &share_conf->vfs_path;
- int err, type;
+ int err;
struct dentry *d;
if (pathname[0] == '\0') {
@@ -67,17 +66,11 @@ static int ksmbd_vfs_path_lookup(struct ksmbd_share_config *share_conf,
}
CLASS(filename_kernel, filename)(pathname);
- err = vfs_path_parent_lookup(filename, flags,
- path, &last, &type,
+ err = vfs_path_parent_lookup(filename, flags, path, &last,
root_share_path);
if (err)
return err;
- if (unlikely(type != LAST_NORM)) {
- path_put(path);
- return -ENOENT;
- }
-
if (for_remove) {
err = mnt_want_write(path->mnt);
if (err) {
@@ -668,7 +661,6 @@ int ksmbd_vfs_rename(struct ksmbd_work *work, const struct path *old_path,
struct renamedata rd;
struct ksmbd_share_config *share_conf = work->tcon->share_conf;
struct ksmbd_file *parent_fp;
- int new_type;
int err, lookup_flags = LOOKUP_NO_SYMLINKS;
if (ksmbd_override_fsids(work))
@@ -678,8 +670,7 @@ int ksmbd_vfs_rename(struct ksmbd_work *work, const struct path *old_path,
retry:
err = vfs_path_parent_lookup(to, lookup_flags | LOOKUP_BENEATH,
- &new_path, &new_last, &new_type,
- &share_conf->vfs_path);
+ &new_path, &new_last, &share_conf->vfs_path);
if (err)
goto out1;
diff --git a/fs/super.c b/fs/super.c
index 378e81e..a8fd611 100644
--- a/fs/super.c
+++ b/fs/super.c
@@ -328,7 +328,7 @@ static struct super_block *alloc_super(struct file_system_type *type, int flags,
init_rwsem(&s->s_umount);
lockdep_set_class(&s->s_umount, &type->s_umount_key);
/*
- * sget() can have s_umount recursion.
+ * sget_fc() can have s_umount recursion.
*
* When it cannot find a suitable sb, it allocates a new
* one (this one), and tries again to find a suitable old
@@ -359,6 +359,7 @@ static struct super_block *alloc_super(struct file_system_type *type, int flags,
s->s_iflags |= SB_I_NODEV;
INIT_HLIST_NODE(&s->s_instances);
INIT_HLIST_BL_HEAD(&s->s_roots);
+ spin_lock_init(&s->s_roots_lock);
mutex_init(&s->s_sync_lock);
INIT_LIST_HEAD(&s->s_inodes);
spin_lock_init(&s->s_inode_list_lock);
@@ -439,7 +440,7 @@ static void kill_super_notify(struct super_block *sb)
/*
* Remove it from @fs_supers so it isn't found by new
- * sget{_fc}() walkers anymore. Any concurrent mounter still
+ * sget_fc() walkers anymore. Any concurrent mounter still
* managing to grab a temporary reference is guaranteed to
* already see SB_DYING and will wait until we notify them about
* SB_DEAD.
@@ -517,7 +518,7 @@ EXPORT_SYMBOL(deactivate_super);
* @sb: superblock to acquire
*
* Acquire a temporary reference on a superblock and try to trade it for
- * an active reference. This is used in sget{_fc}() to wait for a
+ * an active reference. This is used in sget_fc() to wait for a
* superblock to either become SB_BORN or for it to pass through
* sb->kill() and be marked as SB_DEAD.
*
@@ -673,11 +674,11 @@ void generic_shutdown_super(struct super_block *sb)
/*
* Broadcast to everyone that grabbed a temporary reference to this
* superblock before we removed it from @fs_supers that the superblock
- * is dying. Every walker of @fs_supers outside of sget{_fc}() will now
+ * is dying. Every walker of @fs_supers outside of sget_fc() will now
* discard this superblock and treat it as dead.
*
* We leave the superblock on @fs_supers so it can be found by
- * sget{_fc}() until we passed sb->kill_sb().
+ * sget_fc() until we passed sb->kill_sb().
*/
super_wake(sb, SB_DYING);
super_unlock_excl(sb);
@@ -741,12 +742,13 @@ struct super_block *sget_fc(struct fs_context *fc,
int err;
/*
- * Never allow s_user_ns != &init_user_ns when FS_USERNS_MOUNT is
- * not set, as the filesystem is likely unprepared to handle it.
- * This can happen when fsconfig() is called from init_user_ns with
- * an fs_fd opened in another user namespace.
+ * Never allow s_user_ns != &init_user_ns when FS_USERNS_MOUNT or
+ * FS_USERNS_DELEGATABLE is not set, as the filesystem is likely
+ * unprepared to handle it. This can happen when fsconfig() is called
+ * from init_user_ns with an fs_fd opened in another user namespace.
*/
- if (user_ns != &init_user_ns && !(fc->fs_type->fs_flags & FS_USERNS_MOUNT)) {
+ if (user_ns != &init_user_ns &&
+ !(fc->fs_type->fs_flags & (FS_USERNS_MOUNT | FS_USERNS_DELEGATABLE))) {
errorfc(fc, "VFS: Mounting from non-initial user namespace is not allowed");
return ERR_PTR(-EPERM);
}
@@ -808,67 +810,6 @@ struct super_block *sget_fc(struct fs_context *fc,
}
EXPORT_SYMBOL(sget_fc);
-/**
- * sget - find or create a superblock
- * @type: filesystem type superblock should belong to
- * @test: comparison callback
- * @set: setup callback
- * @flags: mount flags
- * @data: argument to each of them
- */
-struct super_block *sget(struct file_system_type *type,
- int (*test)(struct super_block *,void *),
- int (*set)(struct super_block *,void *),
- int flags,
- void *data)
-{
- struct user_namespace *user_ns = current_user_ns();
- struct super_block *s = NULL;
- struct super_block *old;
- int err;
-
-retry:
- spin_lock(&sb_lock);
- if (test) {
- hlist_for_each_entry(old, &type->fs_supers, s_instances) {
- if (!test(old, data))
- continue;
- if (user_ns != old->s_user_ns) {
- spin_unlock(&sb_lock);
- destroy_unused_super(s);
- return ERR_PTR(-EBUSY);
- }
- if (!grab_super(old))
- goto retry;
- destroy_unused_super(s);
- return old;
- }
- }
- if (!s) {
- spin_unlock(&sb_lock);
- s = alloc_super(type, flags, user_ns);
- if (!s)
- return ERR_PTR(-ENOMEM);
- goto retry;
- }
-
- err = set(s, data);
- if (err) {
- spin_unlock(&sb_lock);
- destroy_unused_super(s);
- return ERR_PTR(err);
- }
- s->s_type = type;
- strscpy(s->s_id, type->name, sizeof(s->s_id));
- list_add_tail(&s->s_list, &super_blocks);
- hlist_add_head(&s->s_instances, &type->fs_supers);
- spin_unlock(&sb_lock);
- get_filesystem(type);
- shrinker_register(s->s_shrink);
- return s;
-}
-EXPORT_SYMBOL(sget);
-
void drop_super(struct super_block *sb)
{
super_unlock_shared(sb);
@@ -882,7 +823,6 @@ void drop_super_exclusive(struct super_block *sb)
super_unlock_excl(sb);
put_super(sb);
}
-EXPORT_SYMBOL(drop_super_exclusive);
enum super_iter_flags_t {
SUPER_ITER_EXCL = (1U << 0),
diff --git a/fs/sync.c b/fs/sync.c
index 942a60c..4a84dd8 100644
--- a/fs/sync.c
+++ b/fs/sync.c
@@ -266,8 +266,7 @@ int sync_file_range(struct file *file, loff_t offset, loff_t nbytes,
i_mode = file_inode(file)->i_mode;
ret = -ESPIPE;
- if (!S_ISREG(i_mode) && !S_ISBLK(i_mode) && !S_ISDIR(i_mode) &&
- !S_ISLNK(i_mode))
+ if (!S_ISREG(i_mode) && !S_ISBLK(i_mode) && !S_ISDIR(i_mode))
goto out;
mapping = file->f_mapping;
diff --git a/fs/sysfs/mount.c b/fs/sysfs/mount.c
index b199e8f..88c1082 100644
--- a/fs/sysfs/mount.c
+++ b/fs/sysfs/mount.c
@@ -23,20 +23,6 @@
static struct kernfs_root *sysfs_root;
struct kernfs_node *sysfs_root_kn;
-static int sysfs_get_tree(struct fs_context *fc)
-{
- struct kernfs_fs_context *kfc = fc->fs_private;
- int ret;
-
- ret = kernfs_get_tree(fc);
- if (ret)
- return ret;
-
- if (kfc->new_sb_created)
- fc->root->d_sb->s_iflags |= SB_I_USERNS_VISIBLE;
- return 0;
-}
-
static void sysfs_fs_context_free(struct fs_context *fc)
{
struct kernfs_fs_context *kfc = fc->fs_private;
@@ -49,7 +35,7 @@ static void sysfs_fs_context_free(struct fs_context *fc)
static const struct fs_context_operations sysfs_fs_context_ops = {
.free = sysfs_fs_context_free,
- .get_tree = sysfs_get_tree,
+ .get_tree = kernfs_get_tree,
};
static int sysfs_init_fs_context(struct fs_context *fc)
@@ -93,7 +79,7 @@ static struct file_system_type sysfs_fs_type = {
.name = "sysfs",
.init_fs_context = sysfs_init_fs_context,
.kill_sb = sysfs_kill_sb,
- .fs_flags = FS_USERNS_MOUNT,
+ .fs_flags = FS_USERNS_MOUNT | FS_USERNS_MOUNT_RESTRICTED,
};
int __init sysfs_init(void)
diff --git a/fs/tracefs/event_inode.c b/fs/tracefs/event_inode.c
index 26b6453..39c7a34 100644
--- a/fs/tracefs/event_inode.c
+++ b/fs/tracefs/event_inode.c
@@ -389,8 +389,7 @@ static struct dentry *lookup_file(struct eventfs_inode *parent_ei,
// Files have their parent's ei as their fsdata
dentry->d_fsdata = get_ei(parent_ei);
- d_add(dentry, inode);
- return NULL;
+ return d_splice_alias(inode, dentry);
};
/**
@@ -420,8 +419,7 @@ static struct dentry *lookup_dir_entry(struct dentry *dentry,
dentry->d_fsdata = get_ei(ei);
- d_add(dentry, inode);
- return NULL;
+ return d_splice_alias(inode, dentry);
}
static inline struct eventfs_inode *init_ei(struct eventfs_inode *ei, const char *name)
diff --git a/fs/ubifs/super.c b/fs/ubifs/super.c
index 9a77d8b..3897278 100644
--- a/fs/ubifs/super.c
+++ b/fs/ubifs/super.c
@@ -358,7 +358,7 @@ static void ubifs_evict_inode(struct inode *inode)
goto out;
dbg_gen("inode %llu, mode %#x", inode->i_ino, (int)inode->i_mode);
- ubifs_assert(c, !icount_read(inode));
+ ubifs_assert(c, !icount_read_once(inode));
truncate_inode_pages_final(&inode->i_data);
diff --git a/fs/vboxsf/dir.c b/fs/vboxsf/dir.c
index 42bedc4..c5bd327 100644
--- a/fs/vboxsf/dir.c
+++ b/fs/vboxsf/dir.c
@@ -477,4 +477,5 @@ const struct inode_operations vboxsf_dir_iops = {
.symlink = vboxsf_dir_symlink,
.getattr = vboxsf_getattr,
.setattr = vboxsf_setattr,
+ .fileattr_get = vboxsf_fileattr_get,
};
diff --git a/fs/vboxsf/file.c b/fs/vboxsf/file.c
index 7a7a3fb..9439538 100644
--- a/fs/vboxsf/file.c
+++ b/fs/vboxsf/file.c
@@ -222,7 +222,8 @@ const struct file_operations vboxsf_reg_fops = {
const struct inode_operations vboxsf_reg_iops = {
.getattr = vboxsf_getattr,
- .setattr = vboxsf_setattr
+ .setattr = vboxsf_setattr,
+ .fileattr_get = vboxsf_fileattr_get,
};
static int vboxsf_read_folio(struct file *file, struct folio *folio)
@@ -389,5 +390,6 @@ static const char *vboxsf_get_link(struct dentry *dentry, struct inode *inode,
}
const struct inode_operations vboxsf_lnk_iops = {
- .get_link = vboxsf_get_link
+ .get_link = vboxsf_get_link,
+ .fileattr_get = vboxsf_fileattr_get,
};
diff --git a/fs/vboxsf/super.c b/fs/vboxsf/super.c
index a618cb0..a61fbab 100644
--- a/fs/vboxsf/super.c
+++ b/fs/vboxsf/super.c
@@ -185,6 +185,13 @@ static int vboxsf_fill_super(struct super_block *sb, struct fs_context *fc)
if (err)
goto fail_unmap;
+ /*
+ * A failed query leaves sbi->case_insensitive false, so the
+ * mount defaults to reporting case-sensitive behavior. Do not
+ * fail the mount over an advisory attribute.
+ */
+ vboxsf_query_case_sensitive(sbi);
+
sb->s_magic = VBOXSF_SUPER_MAGIC;
sb->s_blocksize = 1024;
sb->s_maxbytes = MAX_LFS_FILESIZE;
diff --git a/fs/vboxsf/utils.c b/fs/vboxsf/utils.c
index 440e8c5..298bfc9 100644
--- a/fs/vboxsf/utils.c
+++ b/fs/vboxsf/utils.c
@@ -11,6 +11,7 @@
#include <linux/sizes.h>
#include <linux/pagemap.h>
#include <linux/vfs.h>
+#include <linux/fileattr.h>
#include "vfsmod.h"
struct inode *vboxsf_new_inode(struct super_block *sb)
@@ -567,3 +568,32 @@ int vboxsf_dir_read_all(struct vboxsf_sbi *sbi, struct vboxsf_dir_info *sf_d,
return err;
}
+
+int vboxsf_query_case_sensitive(struct vboxsf_sbi *sbi)
+{
+ struct shfl_volinfo volinfo = {};
+ u32 buf_len;
+ int err;
+
+ buf_len = sizeof(volinfo);
+ err = vboxsf_fsinfo(sbi->root, 0, SHFL_INFO_GET | SHFL_INFO_VOLUME,
+ &buf_len, &volinfo);
+ if (err)
+ return err;
+ if (buf_len < sizeof(volinfo))
+ return 0;
+
+ sbi->case_insensitive = !volinfo.properties.case_sensitive;
+ return 0;
+}
+
+int vboxsf_fileattr_get(struct dentry *dentry, struct file_kattr *fa)
+{
+ struct vboxsf_sbi *sbi = VBOXSF_SBI(dentry->d_sb);
+
+ if (sbi->case_insensitive) {
+ fa->fsx_xflags |= FS_XFLAG_CASEFOLD;
+ fa->flags |= FS_CASEFOLD_FL;
+ }
+ return 0;
+}
diff --git a/fs/vboxsf/vfsmod.h b/fs/vboxsf/vfsmod.h
index 05973eb..b61afd0 100644
--- a/fs/vboxsf/vfsmod.h
+++ b/fs/vboxsf/vfsmod.h
@@ -47,6 +47,7 @@ struct vboxsf_sbi {
u32 next_generation;
u32 root;
int bdi_id;
+ bool case_insensitive;
};
/* per-inode information */
@@ -111,6 +112,11 @@ void vboxsf_dir_info_free(struct vboxsf_dir_info *p);
int vboxsf_dir_read_all(struct vboxsf_sbi *sbi, struct vboxsf_dir_info *sf_d,
u64 handle);
+int vboxsf_query_case_sensitive(struct vboxsf_sbi *sbi);
+
+struct file_kattr;
+int vboxsf_fileattr_get(struct dentry *dentry, struct file_kattr *fa);
+
/* from vboxsf_wrappers.c */
int vboxsf_connect(void);
void vboxsf_disconnect(void);
diff --git a/fs/verity/fsverity_private.h b/fs/verity/fsverity_private.h
index 6e6854c..881d46f 100644
--- a/fs/verity/fsverity_private.h
+++ b/fs/verity/fsverity_private.h
@@ -53,6 +53,9 @@ struct merkle_tree_params {
u64 tree_size; /* Merkle tree size in bytes */
unsigned long tree_pages; /* Merkle tree size in pages */
+ /* the hash of an all-zeroes block */
+ u8 zero_digest[FS_VERITY_MAX_DIGEST_SIZE];
+
/*
* Starting block index for each tree level, ordered from leaf level (0)
* to root level ('num_levels - 1')
diff --git a/fs/verity/measure.c b/fs/verity/measure.c
index 6a35623..8180835 100644
--- a/fs/verity/measure.c
+++ b/fs/verity/measure.c
@@ -68,8 +68,8 @@ EXPORT_SYMBOL_GPL(fsverity_ioctl_measure);
* @alg: (out) the digest's algorithm, as a FS_VERITY_HASH_ALG_* value
* @halg: (out) the digest's algorithm, as a HASH_ALGO_* value
*
- * Retrieves the fsverity digest of the given file. The file must have been
- * opened at least once since the inode was last loaded into the inode cache;
+ * Retrieves the fsverity digest of the given file. The
+ * fsverity_ensure_verity_info() must be called on the inode beforehand;
* otherwise this function will not recognize when fsverity is enabled.
*
* The file's fsverity digest consists of @raw_digest in combination with either
diff --git a/fs/verity/open.c b/fs/verity/open.c
index dfa0d1a..d0c56a7 100644
--- a/fs/verity/open.c
+++ b/fs/verity/open.c
@@ -153,6 +153,9 @@ int fsverity_init_merkle_tree_params(struct merkle_tree_params *params,
goto out_err;
}
+ fsverity_hash_block(params, page_address(ZERO_PAGE(0)),
+ params->zero_digest);
+
params->tree_size = offset << log_blocksize;
params->tree_pages = PAGE_ALIGN(params->tree_size) >> PAGE_SHIFT;
return 0;
diff --git a/fs/verity/pagecache.c b/fs/verity/pagecache.c
index 1819314..99f5f53 100644
--- a/fs/verity/pagecache.c
+++ b/fs/verity/pagecache.c
@@ -2,6 +2,7 @@
/*
* Copyright 2019 Google LLC
*/
+#include "fsverity_private.h"
#include <linux/export.h>
#include <linux/fsverity.h>
@@ -56,3 +57,24 @@ void generic_readahead_merkle_tree(struct inode *inode, pgoff_t index,
folio_put(folio);
}
EXPORT_SYMBOL_GPL(generic_readahead_merkle_tree);
+
+/**
+ * fsverity_fill_zerohash() - fill folio with hashes of zero data block
+ * @folio: folio to fill
+ * @offset: offset in the folio to start
+ * @len: length of the range to fill with hashes
+ * @vi: fsverity info
+ */
+void fsverity_fill_zerohash(struct folio *folio, size_t offset, size_t len,
+ struct fsverity_info *vi)
+{
+ size_t off = offset;
+
+ WARN_ON_ONCE(!IS_ALIGNED(offset, vi->tree_params.digest_size));
+ WARN_ON_ONCE(!IS_ALIGNED(len, vi->tree_params.digest_size));
+
+ for (; off < (offset + len); off += vi->tree_params.digest_size)
+ memcpy_to_folio(folio, off, vi->tree_params.zero_digest,
+ vi->tree_params.digest_size);
+}
+EXPORT_SYMBOL_GPL(fsverity_fill_zerohash);
diff --git a/fs/xattr.c b/fs/xattr.c
index 09ecbaa..e1fc688 100644
--- a/fs/xattr.c
+++ b/fs/xattr.c
@@ -28,6 +28,11 @@
#include "internal.h"
+struct sx_key {
+ const struct list_head *parent;
+ const char *name;
+};
+
static const char *
strcmp_prefix(const char *a, const char *a_prefix)
{
@@ -306,7 +311,7 @@ __vfs_setxattr_locked(struct mnt_idmap *idmap, struct dentry *dentry,
if (error)
goto out;
- error = try_break_deleg(inode, delegated_inode);
+ error = try_break_deleg(inode, 0, delegated_inode);
if (error)
goto out;
@@ -564,7 +569,7 @@ __vfs_removexattr_locked(struct mnt_idmap *idmap,
if (error)
goto out;
- error = try_break_deleg(inode, delegated_inode);
+ error = try_break_deleg(inode, 0, delegated_inode);
if (error)
goto out;
@@ -1269,23 +1274,32 @@ struct simple_xattr *simple_xattr_alloc(const void *value, size_t size)
return new_xattr;
}
+static u32 sx_hashfn(const char *name, const struct list_head *parent, u32 seed)
+{
+ return jhash(name, strlen(name), jhash(&parent, sizeof(parent), seed));
+}
+
static u32 simple_xattr_hashfn(const void *data, u32 len, u32 seed)
{
- const char *name = data;
- return jhash(name, strlen(name), seed);
+ const struct sx_key *key = data;
+
+ return sx_hashfn(key->name, key->parent, seed);
}
static u32 simple_xattr_obj_hashfn(const void *obj, u32 len, u32 seed)
{
const struct simple_xattr *xattr = obj;
- return jhash(xattr->name, strlen(xattr->name), seed);
+
+ return sx_hashfn(xattr->name, xattr->parent, seed);
}
static int simple_xattr_obj_cmpfn(struct rhashtable_compare_arg *arg,
const void *obj)
{
const struct simple_xattr *xattr = obj;
- return strcmp(xattr->name, arg->key);
+ const struct sx_key *key = arg->key;
+
+ return xattr->parent != key->parent || strcmp(xattr->name, key->name);
}
static const struct rhashtable_params simple_xattr_params = {
@@ -1298,6 +1312,7 @@ static const struct rhashtable_params simple_xattr_params = {
/**
* simple_xattr_get - get an xattr object
+ * @cache: anchor for the hash table
* @xattrs: the header of the xattr object
* @name: the name of the xattr to retrieve
* @buffer: the buffer to store the value into
@@ -1311,14 +1326,19 @@ static const struct rhashtable_params simple_xattr_params = {
* Return: On success the length of the xattr value is returned. On error a
* negative error code is returned.
*/
-int simple_xattr_get(struct simple_xattrs *xattrs, const char *name,
- void *buffer, size_t size)
+int simple_xattr_get(struct simple_xattr_cache *cache, struct list_head *xattrs,
+ const char *name, void *buffer, size_t size)
{
struct simple_xattr *xattr;
+ struct sx_key key = { .parent = xattrs, .name = name };
+ struct rhashtable *ht = READ_ONCE(cache->ht);
int ret = -ENODATA;
+ if (!ht)
+ return ret;
+
guard(rcu)();
- xattr = rhashtable_lookup(&xattrs->ht, name, simple_xattr_params);
+ xattr = rhashtable_lookup(ht, &key, simple_xattr_params);
if (xattr) {
ret = xattr->size;
if (buffer) {
@@ -1331,8 +1351,45 @@ int simple_xattr_get(struct simple_xattrs *xattrs, const char *name,
return ret;
}
+static struct rhashtable *simple_xattrs_lazy_alloc(struct simple_xattr_cache *cache,
+ const void *value, int flags)
+{
+ struct rhashtable *oldht, *ht = READ_ONCE(cache->ht);
+ int err;
+
+ if (unlikely(!ht)) {
+ if (!value)
+ return (flags & XATTR_REPLACE) ? ERR_PTR(-ENODATA) : NULL;
+
+ ht = kzalloc_obj(*ht);
+ if (!ht)
+ return ERR_PTR(-ENOMEM);
+
+ err = rhashtable_init(ht, &simple_xattr_params);
+ if (err) {
+ kfree(ht);
+ return ERR_PTR(err);
+ }
+
+ /*
+ * Provides release semantics on success, so that use of a
+ * non-NULL READ_ONCE(cache->ht) will be ordered relative to the
+ * above initialization, due to implicit address dependency.
+ */
+ oldht = cmpxchg_release(&cache->ht, NULL, ht);
+ if (oldht) {
+ /* Race lost */
+ rhashtable_destroy(ht);
+ kfree(ht);
+ ht = oldht;
+ }
+ }
+ return ht;
+}
+
/**
* simple_xattr_set - set an xattr object
+ * @cache: anchor for the hash table
* @xattrs: the header of the xattr object
* @name: the name of the xattr to retrieve
* @value: the value to store along the xattr
@@ -1362,45 +1419,58 @@ int simple_xattr_get(struct simple_xattrs *xattrs, const char *name,
* Return: On success, the removed or replaced xattr is returned, to be freed
* by the caller; or NULL if none. On failure a negative error code is returned.
*/
-struct simple_xattr *simple_xattr_set(struct simple_xattrs *xattrs,
+struct simple_xattr *simple_xattr_set(struct simple_xattr_cache *cache, struct list_head *xattrs,
const char *name, const void *value,
size_t size, int flags)
{
+ struct sx_key key = { .parent = xattrs, .name = name };
struct simple_xattr *old_xattr = NULL;
+ struct rhashtable *ht;
int err;
+ ht = simple_xattrs_lazy_alloc(cache, value, flags);
+ if (IS_ERR_OR_NULL(ht))
+ return ERR_CAST(ht);
+
CLASS(simple_xattr, new_xattr)(value, size);
if (IS_ERR(new_xattr))
return new_xattr;
if (new_xattr) {
+ new_xattr->parent = xattrs;
new_xattr->name = kstrdup(name, GFP_KERNEL_ACCOUNT);
if (!new_xattr->name)
return ERR_PTR(-ENOMEM);
}
- /* Lookup is safe without RCU here since writes are serialized. */
- old_xattr = rhashtable_lookup_fast(&xattrs->ht, name,
- simple_xattr_params);
-
+ /*
+ * Hash table lookup/replace/remove will grab RCU read lock themselves.
+ * This makes sure that hash table lookup is safe against concurrent
+ * modification on another inode.
+ */
+ old_xattr = rhashtable_lookup_fast(ht, &key, simple_xattr_params);
if (old_xattr) {
/* Fail if XATTR_CREATE is requested and the xattr exists. */
if (flags & XATTR_CREATE)
return ERR_PTR(-EEXIST);
if (new_xattr) {
- err = rhashtable_replace_fast(&xattrs->ht,
+ err = rhashtable_replace_fast(ht,
&old_xattr->hash_node,
&new_xattr->hash_node,
simple_xattr_params);
if (err)
return ERR_PTR(err);
+
+ list_replace_rcu(&old_xattr->node, &new_xattr->node);
} else {
- err = rhashtable_remove_fast(&xattrs->ht,
+ err = rhashtable_remove_fast(ht,
&old_xattr->hash_node,
simple_xattr_params);
if (err)
return ERR_PTR(err);
+
+ list_del_rcu(&old_xattr->node);
}
} else {
/* Fail if XATTR_REPLACE is requested but no xattr is found. */
@@ -1412,11 +1482,13 @@ struct simple_xattr *simple_xattr_set(struct simple_xattrs *xattrs,
* new value simply insert it.
*/
if (new_xattr) {
- err = rhashtable_insert_fast(&xattrs->ht,
+ err = rhashtable_insert_fast(ht,
&new_xattr->hash_node,
simple_xattr_params);
if (err)
return ERR_PTR(err);
+
+ list_add_tail_rcu(&new_xattr->node, xattrs);
}
/*
@@ -1453,6 +1525,7 @@ static inline int simple_xattr_limits_inc(struct simple_xattr_limits *limits,
/**
* simple_xattr_set_limited - set an xattr with per-inode user.* limits
+ * @cache: anchor for the hash table
* @xattrs: the header of the xattr object
* @limits: per-inode limit counters for user.* xattrs
* @name: the name of the xattr to set or remove
@@ -1467,7 +1540,7 @@ static inline int simple_xattr_limits_inc(struct simple_xattr_limits *limits,
* Return: On success zero is returned. On failure a negative error code is
* returned.
*/
-int simple_xattr_set_limited(struct simple_xattrs *xattrs,
+int simple_xattr_set_limited(struct simple_xattr_cache *cache, struct list_head *xattrs,
struct simple_xattr_limits *limits,
const char *name, const void *value,
size_t size, int flags)
@@ -1481,7 +1554,7 @@ int simple_xattr_set_limited(struct simple_xattrs *xattrs,
return ret;
}
- old_xattr = simple_xattr_set(xattrs, name, value, size, flags);
+ old_xattr = simple_xattr_set(cache, xattrs, name, value, size, flags);
if (IS_ERR(old_xattr)) {
if (value)
simple_xattr_limits_dec(limits, size);
@@ -1527,11 +1600,10 @@ static bool xattr_is_maclabel(const char *name)
* Return: On success the required size or the size of the copied xattrs is
* returned. On error a negative error code is returned.
*/
-ssize_t simple_xattr_list(struct inode *inode, struct simple_xattrs *xattrs,
+ssize_t simple_xattr_list(struct inode *inode, struct list_head *xattrs,
char *buffer, size_t size)
{
bool trusted = ns_capable_noaudit(&init_user_ns, CAP_SYS_ADMIN);
- struct rhashtable_iter iter;
struct simple_xattr *xattr;
ssize_t remaining_size = size;
int err = 0;
@@ -1555,17 +1627,8 @@ ssize_t simple_xattr_list(struct inode *inode, struct simple_xattrs *xattrs,
if (!xattrs)
return size - remaining_size;
- rhashtable_walk_enter(&xattrs->ht, &iter);
- rhashtable_walk_start(&iter);
-
- while ((xattr = rhashtable_walk_next(&iter)) != NULL) {
- if (IS_ERR(xattr)) {
- if (PTR_ERR(xattr) == -EAGAIN)
- continue;
- err = PTR_ERR(xattr);
- break;
- }
-
+ rcu_read_lock();
+ list_for_each_entry_rcu(xattr, xattrs, node) {
/* skip "trusted." attributes for unprivileged callers */
if (!trusted && xattr_is_trusted(xattr->name))
continue;
@@ -1578,15 +1641,14 @@ ssize_t simple_xattr_list(struct inode *inode, struct simple_xattrs *xattrs,
if (err)
break;
}
-
- rhashtable_walk_stop(&iter);
- rhashtable_walk_exit(&iter);
+ rcu_read_unlock();
return err ? err : size - remaining_size;
}
/**
* simple_xattr_add - add xattr objects
+ * @cache: anchor for the hash table
* @xattrs: the header of the xattr object
* @new_xattr: the xattr object to add
*
@@ -1597,112 +1659,100 @@ ssize_t simple_xattr_list(struct inode *inode, struct simple_xattrs *xattrs,
* Return: On success zero is returned. On failure a negative error code is
* returned.
*/
-int simple_xattr_add(struct simple_xattrs *xattrs,
+int simple_xattr_add(struct simple_xattr_cache *cache, struct list_head *xattrs,
struct simple_xattr *new_xattr)
{
- return rhashtable_insert_fast(&xattrs->ht, &new_xattr->hash_node,
- simple_xattr_params);
+ struct rhashtable *ht;
+ int err;
+
+ ht = simple_xattrs_lazy_alloc(cache, new_xattr->value, 0);
+ if (IS_ERR(ht))
+ return PTR_ERR(ht);
+
+ new_xattr->parent = xattrs;
+ err = rhashtable_insert_fast(ht, &new_xattr->hash_node, simple_xattr_params);
+ if (err)
+ return err;
+
+ list_add_tail_rcu(&new_xattr->node, xattrs);
+ return 0;
}
/**
- * simple_xattrs_init - initialize new xattr header
- * @xattrs: header to initialize
+ * simple_xattr_add_limited - add an xattr object, charging per-inode limits
+ * @cache: anchor for the hash table
+ * @xattrs: the header of the xattr object
+ * @limits: per-inode limit counters
+ * @new_xattr: the xattr object to add
*
- * Initialize the rhashtable used to store xattr objects.
+ * Like simple_xattr_add(), but also accounts @new_xattr against @limits so
+ * that a later removal or replacement of it through simple_xattr_set_limited()
+ * decrements counters that were actually incremented, rather than underflowing
+ * them. Use this instead of simple_xattr_add() when seeding initial xattrs
+ * that share a namespace with the limited set/remove path.
*
* Return: On success zero is returned. On failure a negative error code is
* returned.
*/
-int simple_xattrs_init(struct simple_xattrs *xattrs)
+int simple_xattr_add_limited(struct simple_xattr_cache *cache,
+ struct list_head *xattrs,
+ struct simple_xattr_limits *limits,
+ struct simple_xattr *new_xattr)
{
- return rhashtable_init(&xattrs->ht, &simple_xattr_params);
-}
+ int err;
-/**
- * simple_xattrs_alloc - allocate and initialize a new xattr header
- *
- * Dynamically allocate a simple_xattrs header and initialize the
- * underlying rhashtable. This is intended for consumers that want
- * to lazily allocate xattr storage only when the first xattr is set,
- * avoiding the per-inode rhashtable overhead when no xattrs are used.
- *
- * Return: On success a new simple_xattrs is returned. On failure an
- * ERR_PTR is returned.
- */
-struct simple_xattrs *simple_xattrs_alloc(void)
-{
- struct simple_xattrs *xattrs __free(kfree) = NULL;
- int ret;
+ err = simple_xattr_limits_inc(limits, new_xattr->size);
+ if (err)
+ return err;
- xattrs = kzalloc(sizeof(*xattrs), GFP_KERNEL);
- if (!xattrs)
- return ERR_PTR(-ENOMEM);
-
- ret = simple_xattrs_init(xattrs);
- if (ret)
- return ERR_PTR(ret);
-
- return no_free_ptr(xattrs);
-}
-
-/**
- * simple_xattrs_lazy_alloc - get or allocate xattrs for a set operation
- * @xattrsp: pointer to the xattrs pointer (may point to NULL)
- * @value: value being set (NULL means remove)
- * @flags: xattr set flags
- *
- * For lazily-allocated xattrs on the write path. If no xattrs exist yet
- * and this is a remove operation, returns the appropriate result without
- * allocating. Otherwise ensures xattrs is allocated and published with
- * store-release semantics.
- *
- * Return: On success a valid pointer to the xattrs is returned. On
- * failure or early-exit an ERR_PTR or NULL is returned. Callers should
- * check with IS_ERR_OR_NULL() and propagate with PTR_ERR() which
- * correctly returns 0 for the NULL no-op case.
- */
-struct simple_xattrs *simple_xattrs_lazy_alloc(struct simple_xattrs **xattrsp,
- const void *value, int flags)
-{
- struct simple_xattrs *xattrs;
-
- xattrs = READ_ONCE(*xattrsp);
- if (xattrs)
- return xattrs;
-
- if (!value)
- return (flags & XATTR_REPLACE) ? ERR_PTR(-ENODATA) : NULL;
-
- xattrs = simple_xattrs_alloc();
- if (!IS_ERR(xattrs))
- smp_store_release(xattrsp, xattrs);
- return xattrs;
-}
-
-static void simple_xattr_ht_free(void *ptr, void *arg)
-{
- struct simple_xattr *xattr = ptr;
- size_t *freed_space = arg;
-
- if (freed_space)
- *freed_space += simple_xattr_space(xattr->name, xattr->size);
- simple_xattr_free(xattr);
+ err = simple_xattr_add(cache, xattrs, new_xattr);
+ if (err)
+ simple_xattr_limits_dec(limits, new_xattr->size);
+ return err;
}
/**
* simple_xattrs_free - free xattrs
+ * @cache: anchor for the hash table
* @xattrs: xattr header whose xattrs to destroy
* @freed_space: approximate number of bytes of memory freed from @xattrs
*
- * Destroy all xattrs in @xattr. When this is called no one can hold a
+ * Destroy all xattrs in @xattrs. When this is called no one can hold a
* reference to any of the xattrs anymore.
*/
-void simple_xattrs_free(struct simple_xattrs *xattrs, size_t *freed_space)
+void simple_xattrs_free(struct simple_xattr_cache *cache, struct list_head *xattrs,
+ size_t *freed_space)
{
- might_sleep();
-
if (freed_space)
*freed_space = 0;
- rhashtable_free_and_destroy(&xattrs->ht, simple_xattr_ht_free,
- freed_space);
+
+ while (!list_empty(xattrs)) {
+ struct simple_xattr *xattr = list_first_entry(xattrs, typeof(*xattr), node);
+
+ rhashtable_remove_fast(cache->ht, &xattr->hash_node, simple_xattr_params);
+ list_del(&xattr->node);
+ if (freed_space)
+ *freed_space += simple_xattr_space(xattr->name, xattr->size);
+ /*
+ * Free with RCU, since the xattr might still get accessed by
+ * the hash compare function
+ */
+ simple_xattr_free_rcu(xattr);
+ }
+}
+
+/**
+ * simple_xattr_cache_cleanup - free the cache
+ * @cache: anchor for the hash table
+ *
+ * Destroy the cache table, which was lazily allocated on adding the first xattr.
+ */
+void simple_xattr_cache_cleanup(struct simple_xattr_cache *cache)
+{
+ if (cache->ht) {
+ WARN_ON(atomic_read(&cache->ht->nelems));
+ rhashtable_destroy(cache->ht);
+ kfree(cache->ht);
+ cache->ht = NULL;
+ }
}
diff --git a/fs/xfs/libxfs/xfs_inode_util.c b/fs/xfs/libxfs/xfs_inode_util.c
index 551fa51..82be54b 100644
--- a/fs/xfs/libxfs/xfs_inode_util.c
+++ b/fs/xfs/libxfs/xfs_inode_util.c
@@ -130,6 +130,8 @@ xfs_ip2xflags(
if (xfs_inode_has_attr_fork(ip))
flags |= FS_XFLAG_HASATTR;
+ if (xfs_has_asciici(ip->i_mount))
+ flags |= FS_XFLAG_CASEFOLD;
return flags;
}
diff --git a/fs/xfs/xfs_export.c b/fs/xfs/xfs_export.c
index e3e3c3c..9b2ad37 100644
--- a/fs/xfs/xfs_export.c
+++ b/fs/xfs/xfs_export.c
@@ -244,8 +244,6 @@ const struct export_operations xfs_export_operations = {
.get_parent = xfs_fs_get_parent,
.commit_metadata = xfs_fs_nfs_commit_metadata,
#ifdef CONFIG_EXPORTFS_BLOCK_OPS
- .get_uuid = xfs_fs_get_uuid,
- .map_blocks = xfs_fs_map_blocks,
- .commit_blocks = xfs_fs_commit_blocks,
+ .block_ops = &xfs_export_block_ops,
#endif
};
diff --git a/fs/xfs/xfs_inode.c b/fs/xfs/xfs_inode.c
index 9978ac1..ddf2707 100644
--- a/fs/xfs/xfs_inode.c
+++ b/fs/xfs/xfs_inode.c
@@ -1040,7 +1040,7 @@ xfs_itruncate_extents_flags(
int error = 0;
xfs_assert_ilocked(ip, XFS_ILOCK_EXCL);
- if (icount_read(VFS_I(ip)))
+ if (icount_read_once(VFS_I(ip)))
xfs_assert_ilocked(ip, XFS_IOLOCK_EXCL);
if (whichfork == XFS_DATA_FORK)
ASSERT(new_size <= XFS_ISIZE(ip));
diff --git a/fs/xfs/xfs_ioctl.c b/fs/xfs/xfs_ioctl.c
index 96af6b6..904fc3d 100644
--- a/fs/xfs/xfs_ioctl.c
+++ b/fs/xfs/xfs_ioctl.c
@@ -524,7 +524,7 @@ xfs_ioc_fsgetxattra(
xfs_inode_t *ip,
void __user *arg)
{
- struct file_kattr fa;
+ struct file_kattr fa = {};
xfs_ilock(ip, XFS_ILOCK_SHARED);
xfs_fill_fsxattr(ip, XFS_ATTR_FORK, &fa);
@@ -762,9 +762,23 @@ xfs_fileattr_set(
trace_xfs_ioctl_setattr(ip);
if (!fa->fsx_valid) {
- if (fa->flags & ~(FS_IMMUTABLE_FL | FS_APPEND_FL |
- FS_NOATIME_FL | FS_NODUMP_FL |
- FS_SYNC_FL | FS_DAX_FL | FS_PROJINHERIT_FL))
+ unsigned int allowed = FS_IMMUTABLE_FL | FS_APPEND_FL |
+ FS_NOATIME_FL | FS_NODUMP_FL |
+ FS_SYNC_FL | FS_DAX_FL |
+ FS_PROJINHERIT_FL;
+
+ /*
+ * FS_CASEFOLD_FL reflects the ASCIICI superblock feature,
+ * a read-only property. Accept it as a no-op so chattr's
+ * RMW round-trip succeeds; reject any attempt to enable
+ * it on a non-ASCIICI filesystem. xfs_flags2diflags()
+ * has no clause for CASEFOLD, so the bit is dropped from
+ * the on-disk diflags regardless.
+ */
+ if (xfs_has_asciici(mp))
+ allowed |= FS_CASEFOLD_FL;
+
+ if (fa->flags & ~allowed)
return -EOPNOTSUPP;
}
diff --git a/fs/xfs/xfs_pnfs.c b/fs/xfs/xfs_pnfs.c
index d929933..f8535ec 100644
--- a/fs/xfs/xfs_pnfs.c
+++ b/fs/xfs/xfs_pnfs.c
@@ -13,6 +13,7 @@
#include "xfs_bmap.h"
#include "xfs_iomap.h"
#include "xfs_pnfs.h"
+#include <linux/exportfs_block.h>
/*
* Ensure that we do not have any outstanding pNFS layouts that can be used by
@@ -45,11 +46,22 @@ xfs_break_leased_layouts(
return error;
}
+static expfs_block_layouts_t
+xfs_fs_layouts_supported(
+ struct super_block *sb)
+{
+ expfs_block_layouts_t supported = EXPFS_BLOCK_IN_BAND_ID;
+
+ if (exportfs_bdev_supports_out_of_band_id(sb->s_bdev))
+ supported |= EXPFS_BLOCK_OUT_OF_BAND_ID;
+ return supported;
+}
+
/*
* Get a unique ID including its location so that the client can identify
* the exported device.
*/
-int
+static int
xfs_fs_get_uuid(
struct super_block *sb,
u8 *buf,
@@ -104,7 +116,7 @@ xfs_fs_map_update_inode(
/*
* Get a layout for the pNFS client.
*/
-int
+static int
xfs_fs_map_blocks(
struct inode *inode,
loff_t offset,
@@ -255,28 +267,27 @@ xfs_pnfs_validate_isize(
* to manually flush the cache here similar to what the fsync code path does
* for datasyncs on files that have no dirty metadata.
*/
-int
+static int
xfs_fs_commit_blocks(
struct inode *inode,
struct iomap *maps,
int nr_maps,
- struct iattr *iattr)
+ loff_t new_size)
{
struct xfs_inode *ip = XFS_I(inode);
struct xfs_mount *mp = ip->i_mount;
struct xfs_trans *tp;
+ struct timespec64 now;
bool update_isize = false;
int error, i;
loff_t size;
- ASSERT(iattr->ia_valid & (ATTR_ATIME|ATTR_CTIME|ATTR_MTIME));
-
xfs_ilock(ip, XFS_IOLOCK_EXCL);
size = i_size_read(inode);
- if ((iattr->ia_valid & ATTR_SIZE) && iattr->ia_size > size) {
+ if (new_size > size) {
update_isize = true;
- size = iattr->ia_size;
+ size = new_size;
}
for (i = 0; i < nr_maps; i++) {
@@ -321,11 +332,13 @@ xfs_fs_commit_blocks(
xfs_trans_ijoin(tp, ip, XFS_ILOCK_EXCL);
xfs_trans_log_inode(tp, ip, XFS_ILOG_CORE);
- ASSERT(!(iattr->ia_valid & (ATTR_UID | ATTR_GID)));
- setattr_copy(&nop_mnt_idmap, inode, iattr);
+ now = inode_set_ctime_current(inode);
+ inode_set_atime_to_ts(inode, now);
+ inode_set_mtime_to_ts(inode, now);
+
if (update_isize) {
- i_size_write(inode, iattr->ia_size);
- ip->i_disk_size = iattr->ia_size;
+ i_size_write(inode, new_size);
+ ip->i_disk_size = new_size;
}
xfs_trans_set_sync(tp);
@@ -335,3 +348,10 @@ xfs_fs_commit_blocks(
xfs_iunlock(ip, XFS_IOLOCK_EXCL);
return error;
}
+
+const struct exportfs_block_ops xfs_export_block_ops = {
+ .layouts_supported = xfs_fs_layouts_supported,
+ .get_uuid = xfs_fs_get_uuid,
+ .map_blocks = xfs_fs_map_blocks,
+ .commit_blocks = xfs_fs_commit_blocks,
+};
diff --git a/fs/xfs/xfs_pnfs.h b/fs/xfs/xfs_pnfs.h
index 940c6c2..bf43b20 100644
--- a/fs/xfs/xfs_pnfs.h
+++ b/fs/xfs/xfs_pnfs.h
@@ -2,13 +2,9 @@
#ifndef _XFS_PNFS_H
#define _XFS_PNFS_H 1
-#ifdef CONFIG_EXPORTFS_BLOCK_OPS
-int xfs_fs_get_uuid(struct super_block *sb, u8 *buf, u32 *len, u64 *offset);
-int xfs_fs_map_blocks(struct inode *inode, loff_t offset, u64 length,
- struct iomap *iomap, bool write, u32 *device_generation);
-int xfs_fs_commit_blocks(struct inode *inode, struct iomap *maps, int nr_maps,
- struct iattr *iattr);
+#include <linux/exportfs_block.h>
+#ifdef CONFIG_EXPORTFS_BLOCK_OPS
int xfs_break_leased_layouts(struct inode *inode, uint *iolock,
bool *did_unlock);
#else
@@ -18,4 +14,7 @@ xfs_break_leased_layouts(struct inode *inode, uint *iolock, bool *did_unlock)
return 0;
}
#endif /* CONFIG_EXPORTFS_BLOCK_OPS */
+
+extern const struct exportfs_block_ops xfs_export_block_ops;
+
#endif /* _XFS_PNFS_H */
diff --git a/fs/xfs/xfs_trace.h b/fs/xfs/xfs_trace.h
index 1c098cf..f87c738 100644
--- a/fs/xfs/xfs_trace.h
+++ b/fs/xfs/xfs_trace.h
@@ -1158,7 +1158,7 @@ DECLARE_EVENT_CLASS(xfs_iref_class,
TP_fast_assign(
__entry->dev = VFS_I(ip)->i_sb->s_dev;
__entry->ino = ip->i_ino;
- __entry->count = icount_read(VFS_I(ip));
+ __entry->count = icount_read_once(VFS_I(ip));
__entry->pincount = atomic_read(&ip->i_pincount);
__entry->iflags = ip->i_flags;
__entry->caller_ip = caller_ip;
diff --git a/include/acpi/acbuffer.h b/include/acpi/acbuffer.h
index cbc9aea..7823df1 100644
--- a/include/acpi/acbuffer.h
+++ b/include/acpi/acbuffer.h
@@ -3,7 +3,7 @@
*
* Name: acbuffer.h - Support for buffers returned by ACPI predefined names
*
- * Copyright (C) 2000 - 2025, Intel Corp.
+ * Copyright (C) 2000 - 2026, Intel Corp.
*
*****************************************************************************/
diff --git a/include/acpi/acconfig.h b/include/acpi/acconfig.h
index 521d4bf..6bce050 100644
--- a/include/acpi/acconfig.h
+++ b/include/acpi/acconfig.h
@@ -3,7 +3,7 @@
*
* Name: acconfig.h - Global configuration constants
*
- * Copyright (C) 2000 - 2025, Intel Corp.
+ * Copyright (C) 2000 - 2026, Intel Corp.
*
*****************************************************************************/
diff --git a/include/acpi/acexcep.h b/include/acpi/acexcep.h
index a2db36d..d7b22c56 100644
--- a/include/acpi/acexcep.h
+++ b/include/acpi/acexcep.h
@@ -3,7 +3,7 @@
*
* Name: acexcep.h - Exception codes returned by the ACPI subsystem
*
- * Copyright (C) 2000 - 2025, Intel Corp.
+ * Copyright (C) 2000 - 2026, Intel Corp.
*
*****************************************************************************/
diff --git a/include/acpi/acnames.h b/include/acpi/acnames.h
index cb6a4dc..714d8f2 100644
--- a/include/acpi/acnames.h
+++ b/include/acpi/acnames.h
@@ -3,7 +3,7 @@
*
* Name: acnames.h - Global names and strings
*
- * Copyright (C) 2000 - 2025, Intel Corp.
+ * Copyright (C) 2000 - 2026, Intel Corp.
*
*****************************************************************************/
diff --git a/include/acpi/acoutput.h b/include/acpi/acoutput.h
index 3584f33..a8d8460 100644
--- a/include/acpi/acoutput.h
+++ b/include/acpi/acoutput.h
@@ -3,7 +3,7 @@
*
* Name: acoutput.h -- debug output
*
- * Copyright (C) 2000 - 2025, Intel Corp.
+ * Copyright (C) 2000 - 2026, Intel Corp.
*
*****************************************************************************/
diff --git a/include/acpi/acpi.h b/include/acpi/acpi.h
index 92bf809..07e3d6e 100644
--- a/include/acpi/acpi.h
+++ b/include/acpi/acpi.h
@@ -3,7 +3,7 @@
*
* Name: acpi.h - Master public include file used to interface to ACPICA
*
- * Copyright (C) 2000 - 2025, Intel Corp.
+ * Copyright (C) 2000 - 2026, Intel Corp.
*
*****************************************************************************/
diff --git a/include/acpi/acpi_bus.h b/include/acpi/acpi_bus.h
index c41d9a75..7e57f96 100644
--- a/include/acpi/acpi_bus.h
+++ b/include/acpi/acpi_bus.h
@@ -629,6 +629,8 @@ int acpi_dev_install_notify_handler(struct acpi_device *adev,
void acpi_dev_remove_notify_handler(struct acpi_device *adev,
u32 handler_type,
acpi_notify_handler handler);
+int devm_acpi_install_notify_handler(struct device *dev, u32 handler_type,
+ acpi_notify_handler handler, void *context);
extern int acpi_notifier_call_chain(const char *device_class,
const char *bus_id, u32 type, u32 data);
extern int register_acpi_notifier(struct notifier_block *);
diff --git a/include/acpi/acpiosxf.h b/include/acpi/acpiosxf.h
index 65c5737..f5bf17f 100644
--- a/include/acpi/acpiosxf.h
+++ b/include/acpi/acpiosxf.h
@@ -5,7 +5,7 @@
* interfaces must be implemented by OSL to interface the
* ACPI components to the host operating system.
*
- * Copyright (C) 2000 - 2025, Intel Corp.
+ * Copyright (C) 2000 - 2026, Intel Corp.
*
*****************************************************************************/
diff --git a/include/acpi/acpixf.h b/include/acpi/acpixf.h
index a4b5627..9557462 100644
--- a/include/acpi/acpixf.h
+++ b/include/acpi/acpixf.h
@@ -3,7 +3,7 @@
*
* Name: acpixf.h - External interfaces to the ACPI subsystem
*
- * Copyright (C) 2000 - 2025, Intel Corp.
+ * Copyright (C) 2000 - 2026, Intel Corp.
*
*****************************************************************************/
@@ -12,7 +12,7 @@
/* Current ACPICA subsystem version in YYYYMMDD format */
-#define ACPI_CA_VERSION 0x20251212
+#define ACPI_CA_VERSION 0x20260408
#include <acpi/acconfig.h>
#include <acpi/actypes.h>
diff --git a/include/acpi/acrestyp.h b/include/acpi/acrestyp.h
index 842f932..0e25ef6 100644
--- a/include/acpi/acrestyp.h
+++ b/include/acpi/acrestyp.h
@@ -3,7 +3,7 @@
*
* Name: acrestyp.h - Defines, types, and structures for resource descriptors
*
- * Copyright (C) 2000 - 2025, Intel Corp.
+ * Copyright (C) 2000 - 2026, Intel Corp.
*
*****************************************************************************/
@@ -423,6 +423,7 @@ struct acpi_resource_i2c_serialbus {
ACPI_RESOURCE_SERIAL_COMMON u8 access_mode;
u16 slave_address;
u32 connection_speed;
+ u8 lvr;
};
/* Values for access_mode field above */
diff --git a/include/acpi/actbl.h b/include/acpi/actbl.h
index 8a67d4e..606efcb 100644
--- a/include/acpi/actbl.h
+++ b/include/acpi/actbl.h
@@ -3,7 +3,7 @@
*
* Name: actbl.h - Basic ACPI Table Definitions
*
- * Copyright (C) 2000 - 2025, Intel Corp.
+ * Copyright (C) 2000 - 2026, Intel Corp.
*
*****************************************************************************/
diff --git a/include/acpi/actbl1.h b/include/acpi/actbl1.h
index f72e005..d824838 100644
--- a/include/acpi/actbl1.h
+++ b/include/acpi/actbl1.h
@@ -3,7 +3,7 @@
*
* Name: actbl1.h - Additional ACPI table definitions
*
- * Copyright (C) 2000 - 2025, Intel Corp.
+ * Copyright (C) 2000 - 2026, Intel Corp.
*
*****************************************************************************/
diff --git a/include/acpi/actbl2.h b/include/acpi/actbl2.h
index 5c0b55e..baef525 100644
--- a/include/acpi/actbl2.h
+++ b/include/acpi/actbl2.h
@@ -3,7 +3,7 @@
*
* Name: actbl2.h - ACPI Table Definitions
*
- * Copyright (C) 2000 - 2025, Intel Corp.
+ * Copyright (C) 2000 - 2026, Intel Corp.
*
*****************************************************************************/
@@ -1524,7 +1524,7 @@ struct acpi_madt_generic_translator {
#define ACPI_MADT_ITS_NON_COHERENT (1)
-/* 16: Multiprocessor wakeup (ACPI 6.4) */
+/* 16: Multiprocessor wakeup (ACPI 6.6) */
struct acpi_madt_multiproc_wakeup {
struct acpi_subtable_header header;
diff --git a/include/acpi/actbl3.h b/include/acpi/actbl3.h
index 7ca456e..331ecbf 100644
--- a/include/acpi/actbl3.h
+++ b/include/acpi/actbl3.h
@@ -3,7 +3,7 @@
*
* Name: actbl3.h - ACPI Table Definitions
*
- * Copyright (C) 2000 - 2025, Intel Corp.
+ * Copyright (C) 2000 - 2026, Intel Corp.
*
*****************************************************************************/
diff --git a/include/acpi/actypes.h b/include/acpi/actypes.h
index 8fe893d..00c1eb1 100644
--- a/include/acpi/actypes.h
+++ b/include/acpi/actypes.h
@@ -3,7 +3,7 @@
*
* Name: actypes.h - Common data types for the entire ACPI subsystem
*
- * Copyright (C) 2000 - 2025, Intel Corp.
+ * Copyright (C) 2000 - 2026, Intel Corp.
*
*****************************************************************************/
@@ -591,9 +591,9 @@ typedef u64 acpi_integer;
#define ACPI_STATE_D1 (u8) 1
#define ACPI_STATE_D2 (u8) 2
#define ACPI_STATE_D3_HOT (u8) 3
-#define ACPI_STATE_D3 (u8) 4
-#define ACPI_STATE_D3_COLD ACPI_STATE_D3
-#define ACPI_D_STATES_MAX ACPI_STATE_D3
+#define ACPI_STATE_D3_COLD (u8) 4
+#define ACPI_STATE_D3 ACPI_STATE_D3_COLD
+#define ACPI_D_STATES_MAX ACPI_STATE_D3_COLD
#define ACPI_D_STATE_COUNT 5
#define ACPI_STATE_C0 (u8) 0
diff --git a/include/acpi/acuuid.h b/include/acpi/acuuid.h
index b2e29da..9ccf5bb 100644
--- a/include/acpi/acuuid.h
+++ b/include/acpi/acuuid.h
@@ -3,7 +3,7 @@
*
* Name: acuuid.h - ACPI-related UUID/GUID definitions
*
- * Copyright (C) 2000 - 2025, Intel Corp.
+ * Copyright (C) 2000 - 2026, Intel Corp.
*
*****************************************************************************/
@@ -63,6 +63,11 @@
#define UUID_CACHE_PROPERTIES "6DC63E77-257E-4E78-A973-A21F2796898D"
#define UUID_PHYSICAL_PROPERTY "DDE4D59A-AA42-4349-B407-EA40F57D9FB7"
+/* Modern Standby */
+#define UUID_LPS0_MICROSOFT "11E00D56-CE64-47CE-837B-1F898F9AA461"
+#define UUID_LPS0_INTEL "C4EB40A0-6CD2-11E2-BCFD-0800200C9A66"
+#define UUID_LPS0_AMD "E3F32452-FEBC-43CE-9039-932122D37721"
+
/* Miscellaneous */
#define UUID_PLATFORM_CAPABILITIES "0811b06e-4a27-44f9-8d60-3cbbc22e7b48"
diff --git a/include/acpi/cppc_acpi.h b/include/acpi/cppc_acpi.h
index d1f02ce..8693890 100644
--- a/include/acpi/cppc_acpi.h
+++ b/include/acpi/cppc_acpi.h
@@ -17,16 +17,18 @@
#include <acpi/pcc.h>
#include <acpi/processor.h>
-/* CPPCv2 and CPPCv3 support */
+/* CPPCv2, CPPCv3 and CPPCv4 support */
#define CPPC_V2_REV 2
#define CPPC_V3_REV 3
+#define CPPC_V4_REV 4
#define CPPC_V2_NUM_ENT 21
#define CPPC_V3_NUM_ENT 23
+#define CPPC_V4_NUM_ENT 25
#define PCC_CMD_COMPLETE_MASK (1 << 0)
#define PCC_ERROR_MASK (1 << 2)
-#define MAX_CPC_REG_ENT 21
+#define MAX_CPC_REG_ENT 23
/* CPPC specific PCC commands. */
#define CMD_READ 0
@@ -109,6 +111,8 @@ enum cppc_regs {
REFERENCE_PERF,
LOWEST_FREQ,
NOMINAL_FREQ,
+ OSPM_NOMINAL_PERF,
+ RESOURCE_PRIORITY,
};
/*
diff --git a/include/acpi/platform/acenv.h b/include/acpi/platform/acenv.h
index a11fa83..1c66cfe 100644
--- a/include/acpi/platform/acenv.h
+++ b/include/acpi/platform/acenv.h
@@ -3,7 +3,7 @@
*
* Name: acenv.h - Host and compiler configuration
*
- * Copyright (C) 2000 - 2025, Intel Corp.
+ * Copyright (C) 2000 - 2026, Intel Corp.
*
*****************************************************************************/
diff --git a/include/acpi/platform/acenvex.h b/include/acpi/platform/acenvex.h
index 8ffc4e1..f7797ca 100644
--- a/include/acpi/platform/acenvex.h
+++ b/include/acpi/platform/acenvex.h
@@ -3,7 +3,7 @@
*
* Name: acenvex.h - Extra host and compiler configuration
*
- * Copyright (C) 2000 - 2025, Intel Corp.
+ * Copyright (C) 2000 - 2026, Intel Corp.
*
*****************************************************************************/
diff --git a/include/acpi/platform/acgcc.h b/include/acpi/platform/acgcc.h
index 8e4cf2f..31c6245 100644
--- a/include/acpi/platform/acgcc.h
+++ b/include/acpi/platform/acgcc.h
@@ -3,7 +3,7 @@
*
* Name: acgcc.h - GCC specific defines, etc.
*
- * Copyright (C) 2000 - 2025, Intel Corp.
+ * Copyright (C) 2000 - 2026, Intel Corp.
*
*****************************************************************************/
diff --git a/include/acpi/platform/acgccex.h b/include/acpi/platform/acgccex.h
index 4a3c019..89cd8a6 100644
--- a/include/acpi/platform/acgccex.h
+++ b/include/acpi/platform/acgccex.h
@@ -3,7 +3,7 @@
*
* Name: acgccex.h - Extra GCC specific defines, etc.
*
- * Copyright (C) 2000 - 2025, Intel Corp.
+ * Copyright (C) 2000 - 2026, Intel Corp.
*
*****************************************************************************/
diff --git a/include/acpi/platform/aclinux.h b/include/acpi/platform/aclinux.h
index edbbc90..9b30bb9 100644
--- a/include/acpi/platform/aclinux.h
+++ b/include/acpi/platform/aclinux.h
@@ -3,7 +3,7 @@
*
* Name: aclinux.h - OS specific defines, etc. for Linux
*
- * Copyright (C) 2000 - 2025, Intel Corp.
+ * Copyright (C) 2000 - 2026, Intel Corp.
*
*****************************************************************************/
diff --git a/include/acpi/platform/aclinuxex.h b/include/acpi/platform/aclinuxex.h
index 7326565..aeb74e2 100644
--- a/include/acpi/platform/aclinuxex.h
+++ b/include/acpi/platform/aclinuxex.h
@@ -3,7 +3,7 @@
*
* Name: aclinuxex.h - Extra OS specific defines, etc. for Linux
*
- * Copyright (C) 2000 - 2025, Intel Corp.
+ * Copyright (C) 2000 - 2026, Intel Corp.
*
*****************************************************************************/
diff --git a/include/acpi/platform/aczephyr.h b/include/acpi/platform/aczephyr.h
index 03d9a4a..b698ec1 100644
--- a/include/acpi/platform/aczephyr.h
+++ b/include/acpi/platform/aczephyr.h
@@ -3,7 +3,7 @@
*
* Module Name: aczephyr.h - OS specific defines, etc.
*
- * Copyright (C) 2000 - 2025, Intel Corp.
+ * Copyright (C) 2000 - 2026, Intel Corp.
*
*****************************************************************************/
diff --git a/include/asm-generic/vmlinux.lds.h b/include/asm-generic/vmlinux.lds.h
index 60c8c22..5659f4b5 100644
--- a/include/asm-generic/vmlinux.lds.h
+++ b/include/asm-generic/vmlinux.lds.h
@@ -1011,6 +1011,12 @@
#define PERCPU_DECRYPTED_SECTION
#endif
+#ifdef CONFIG_PROPELLER_CLANG
+#define PROPELLER_DATA \
+ .llvm_bb_addr_map : { *(.llvm_bb_addr_map) }
+#else
+#define PROPELLER_DATA
+#endif
/*
* Default discarded sections.
diff --git a/include/drm/drm_atomic_uapi.h b/include/drm/drm_atomic_uapi.h
index 4363155..4e7e78f 100644
--- a/include/drm/drm_atomic_uapi.h
+++ b/include/drm/drm_atomic_uapi.h
@@ -29,6 +29,8 @@
#ifndef DRM_ATOMIC_UAPI_H_
#define DRM_ATOMIC_UAPI_H_
+#include <linux/types.h>
+
struct drm_crtc_state;
struct drm_display_mode;
struct drm_property_blob;
@@ -50,7 +52,7 @@ drm_atomic_set_crtc_for_plane(struct drm_plane_state *plane_state,
struct drm_crtc *crtc);
void drm_atomic_set_fb_for_plane(struct drm_plane_state *plane_state,
struct drm_framebuffer *fb);
-void drm_atomic_set_colorop_for_plane(struct drm_plane_state *plane_state,
+bool drm_atomic_set_colorop_for_plane(struct drm_plane_state *plane_state,
struct drm_colorop *colorop);
int __must_check
drm_atomic_set_crtc_for_connector(struct drm_connector_state *conn_state,
diff --git a/include/drm/drm_colorop.h b/include/drm/drm_colorop.h
index bd08285..d5b45339 100644
--- a/include/drm/drm_colorop.h
+++ b/include/drm/drm_colorop.h
@@ -183,6 +183,20 @@ struct drm_colorop_state {
*/
struct drm_property_blob *data;
+ /**
+ * @lut1d_interpolation:
+ *
+ * Interpolation for DRM_COLOROP_1D_LUT
+ */
+ enum drm_colorop_lut1d_interpolation_type lut1d_interpolation;
+
+ /**
+ * @lut3d_interpolation:
+ *
+ * Interpolation for DRM_COLOROP_3D_LUT
+ */
+ enum drm_colorop_lut3d_interpolation_type lut3d_interpolation;
+
/** @state: backpointer to global drm_atomic_state */
struct drm_atomic_state *state;
};
@@ -307,25 +321,9 @@ struct drm_colorop {
uint32_t size;
/**
- * @lut1d_interpolation:
- *
- * Read-only
- * Interpolation for DRM_COLOROP_1D_LUT
- */
- enum drm_colorop_lut1d_interpolation_type lut1d_interpolation;
-
- /**
- * @lut3d_interpolation:
- *
- * Read-only
- * Interpolation for DRM_COLOROP_3D_LUT
- */
- enum drm_colorop_lut3d_interpolation_type lut3d_interpolation;
-
- /**
* @lut1d_interpolation_property:
*
- * Read-only property for DRM_COLOROP_1D_LUT interpolation
+ * Property for DRM_COLOROP_1D_LUT interpolation
*/
struct drm_property *lut1d_interpolation_property;
@@ -353,7 +351,7 @@ struct drm_colorop {
/**
* @lut3d_interpolation_property:
*
- * Read-only property for DRM_COLOROP_3D_LUT interpolation
+ * Property for DRM_COLOROP_3D_LUT interpolation
*/
struct drm_property *lut3d_interpolation_property;
diff --git a/include/linux/backing-dev-defs.h b/include/linux/backing-dev-defs.h
index a06b934..4f10849 100644
--- a/include/linux/backing-dev-defs.h
+++ b/include/linux/backing-dev-defs.h
@@ -26,6 +26,7 @@ enum wb_state {
WB_writeback_running, /* Writeback is in progress */
WB_has_dirty_io, /* Dirty inodes on ->b_{dirty|io|more_io} */
WB_start_all, /* nr_pages == 0 (all) work pending */
+ WB_start_dontcache, /* dontcache writeback pending */
};
enum wb_stat_item {
@@ -33,6 +34,7 @@ enum wb_stat_item {
WB_WRITEBACK,
WB_DIRTIED,
WB_WRITTEN,
+ WB_DONTCACHE_DIRTY,
NR_WB_STAT_ITEMS
};
@@ -55,6 +57,7 @@ enum wb_reason {
*/
WB_REASON_FORKER_THREAD,
WB_REASON_FOREIGN_FLUSH,
+ WB_REASON_DONTCACHE,
WB_REASON_MAX,
};
diff --git a/include/linux/binfmts.h b/include/linux/binfmts.h
index 65abd5a..2c77e38 100644
--- a/include/linux/binfmts.h
+++ b/include/linux/binfmts.h
@@ -25,6 +25,9 @@ struct linux_binprm {
struct page *page[MAX_ARG_PAGES];
#endif
struct mm_struct *mm;
+ struct mm_struct *old_mm; /* replaced address space, freed by setup_new_exec() */
+ /* user_ns published to task->exec_state at execve, narrowed by would_dump(). */
+ struct user_namespace *user_ns;
unsigned long p; /* current top of mem */
unsigned int
/* Should an execfd be passed to userspace? */
diff --git a/include/linux/bio.h b/include/linux/bio.h
index dc17780..8300d55 100644
--- a/include/linux/bio.h
+++ b/include/linux/bio.h
@@ -703,20 +703,6 @@ static inline bool bioset_initialized(struct bio_set *bs)
return bs->bio_slab != NULL;
}
-/*
- * Mark a bio as polled. Note that for async polled IO, the caller must
- * expect -EWOULDBLOCK if we cannot allocate a request (or other resources).
- * We cannot block waiting for requests on polled IO, as those completions
- * must be found by the caller. This is different than IRQ driven IO, where
- * it's safe to wait for IO to complete.
- */
-static inline void bio_set_polled(struct bio *bio, struct kiocb *kiocb)
-{
- bio->bi_opf |= REQ_POLLED;
- if (kiocb->ki_flags & IOCB_NOWAIT)
- bio->bi_opf |= REQ_NOWAIT;
-}
-
static inline void bio_clear_polled(struct bio *bio)
{
bio->bi_opf &= ~REQ_POLLED;
diff --git a/include/linux/bpf.h b/include/linux/bpf.h
index cd191c5..64efc3f 100644
--- a/include/linux/bpf.h
+++ b/include/linux/bpf.h
@@ -31,6 +31,7 @@
#include <linux/static_call.h>
#include <linux/memcontrol.h>
#include <linux/cfi.h>
+#include <linux/xattr.h>
#include <asm/rqspinlock.h>
struct bpf_verifier_env;
@@ -1918,6 +1919,8 @@ struct bpf_mount_opts {
u64 delegate_maps;
u64 delegate_progs;
u64 delegate_attachs;
+
+ struct simple_xattr_cache xa_cache;
};
struct bpf_token {
diff --git a/include/linux/buffer_head.h b/include/linux/buffer_head.h
index e4939e3..8b23bc9 100644
--- a/include/linux/buffer_head.h
+++ b/include/linux/buffer_head.h
@@ -46,7 +46,6 @@ enum bh_state_bits {
struct page;
struct buffer_head;
struct address_space;
-typedef void (bh_end_io_t)(struct buffer_head *bh, int uptodate);
/*
* Historically, a buffer_head was used to map a single block
@@ -55,7 +54,7 @@ typedef void (bh_end_io_t)(struct buffer_head *bh, int uptodate);
* is the bio, and buffer_heads are used for extracting block
* mappings (via a get_block_t call), for tracking state within
* a folio (via a folio_mapping) and for wrapping bio submission
- * for backward compatibility reasons (e.g. submit_bh).
+ * for backward compatibility reasons (e.g. bh_submit).
*/
struct buffer_head {
unsigned long b_state; /* buffer state bitmap (see above) */
@@ -70,8 +69,7 @@ struct buffer_head {
char *b_data; /* pointer to data within the page */
struct block_device *b_bdev;
- bh_end_io_t *b_end_io; /* I/O completion */
- void *b_private; /* reserved for b_end_io */
+ void *b_private; /* reserved for bio_end_io */
struct list_head b_assoc_buffers; /* associated with another mapping */
struct mapping_metadata_bhs *b_mmb; /* head of the list of metadata bhs
* this buffer is associated with */
@@ -203,7 +201,12 @@ struct buffer_head *alloc_page_buffers(struct page *page, unsigned long size);
struct buffer_head *create_empty_buffers(struct folio *folio,
unsigned long blocksize, unsigned long b_state);
void end_buffer_read_sync(struct buffer_head *bh, int uptodate);
-void end_buffer_write_sync(struct buffer_head *bh, int uptodate);
+bool bio_endio_bh(struct bio *bio, struct buffer_head **bhp);
+
+/* Completion routines suitable for passing to bh_submit() */
+void bh_end_read(struct bio *bio);
+void bh_end_write(struct bio *bio);
+void bh_end_async_write(struct bio *bio);
/* Things to do with metadata buffers list */
void mmb_mark_buffer_dirty(struct buffer_head *bh, struct mapping_metadata_bhs *mmb);
@@ -218,7 +221,6 @@ static inline void clean_bdev_bh_alias(struct buffer_head *bh)
clean_bdev_aliases(bh->b_bdev, bh->b_blocknr, 1);
}
-void mark_buffer_async_write(struct buffer_head *bh);
void __wait_on_buffer(struct buffer_head *);
wait_queue_head_t *bh_waitq_head(struct buffer_head *bh);
struct buffer_head *__find_get_block(struct block_device *bdev, sector_t block,
@@ -239,7 +241,7 @@ void __lock_buffer(struct buffer_head *bh);
int sync_dirty_buffer(struct buffer_head *bh);
int __sync_dirty_buffer(struct buffer_head *bh, blk_opf_t op_flags);
void write_dirty_buffer(struct buffer_head *bh, blk_opf_t op_flags);
-void submit_bh(blk_opf_t, struct buffer_head *);
+void bh_submit(struct buffer_head *, blk_opf_t, bio_end_io_t);
void write_boundary_block(struct block_device *bdev,
sector_t bblock, unsigned blocksize);
int bh_uptodate_or_lock(struct buffer_head *bh);
diff --git a/include/linux/compiler-clang.h b/include/linux/compiler-clang.h
index 527e4e1..e606ed6 100644
--- a/include/linux/compiler-clang.h
+++ b/include/linux/compiler-clang.h
@@ -5,15 +5,6 @@
/* Compiler specific definitions for Clang compiler */
-/*
- * Clang prior to 17 is being silly and considers many __cleanup() variables
- * as unused (because they are, their sole purpose is to go out of scope).
- *
- * https://github.com/llvm/llvm-project/commit/877210faa447f4cc7db87812f8ed80e398fedd61
- */
-#undef __cleanup
-#define __cleanup(func) __maybe_unused __attribute__((__cleanup__(func)))
-
/* all clang versions usable with the kernel support KASAN ABI version 5 */
#define KASAN_ABI_VERSION 5
@@ -137,10 +128,10 @@
#define __diag_clang_23(s)
#endif
-#define __diag_clang_13(s) __diag(s)
+#define __diag_clang_all(s) __diag(s)
#define __diag_ignore_all(option, comment) \
- __diag_clang(13, ignore, option)
+ __diag_clang(all, ignore, option)
/*
* clang has horrible behavior with "g" or "rm" constraints for asm
diff --git a/include/linux/coredump.h b/include/linux/coredump.h
index 68861da..7b38ee2 100644
--- a/include/linux/coredump.h
+++ b/include/linux/coredump.h
@@ -5,6 +5,7 @@
#include <linux/types.h>
#include <linux/mm.h>
#include <linux/fs.h>
+#include <linux/sched/coredump.h>
#include <asm/siginfo.h>
#ifdef CONFIG_COREDUMP
@@ -20,7 +21,10 @@ struct coredump_params {
const kernel_siginfo_t *siginfo;
struct file *file;
unsigned long limit;
+ /* MMF_DUMP_FILTER_* bits, snapshot of mm->flags at dump start. */
unsigned long mm_flags;
+ /* Snapshot of dumpable at dump start. */
+ enum task_dumpable dumpable;
int cpu;
loff_t written;
loff_t pos;
diff --git a/include/linux/dcache.h b/include/linux/dcache.h
index 2577c05..4b1ff996 100644
--- a/include/linux/dcache.h
+++ b/include/linux/dcache.h
@@ -116,10 +116,7 @@ struct dentry {
* possible!
*/
- union {
- struct list_head d_lru; /* LRU list */
- wait_queue_head_t *d_wait; /* in-lookup ones only */
- };
+ struct list_head d_lru; /* LRU list */
struct hlist_node d_sib; /* child of parent list */
struct hlist_head d_children; /* our children */
/*
@@ -210,6 +207,9 @@ enum dentry_flags {
DCACHE_REFERENCED = BIT(6), /* Recently used, don't discard. */
DCACHE_DONTCACHE = BIT(7), /* Purge from memory on final dput() */
DCACHE_CANT_MOUNT = BIT(8),
+ DCACHE_LOOKUP_WAITERS = BIT(9), /* A thread is waiting for
+ * PAR_LOOKUP to clear
+ */
DCACHE_SHRINK_LIST = BIT(10),
DCACHE_OP_WEAK_REVALIDATE = BIT(11),
/*
@@ -256,8 +256,7 @@ extern void d_delete(struct dentry *);
/* allocate/de-allocate */
extern struct dentry * d_alloc(struct dentry *, const struct qstr *);
extern struct dentry * d_alloc_anon(struct super_block *);
-extern struct dentry * d_alloc_parallel(struct dentry *, const struct qstr *,
- wait_queue_head_t *);
+extern struct dentry * d_alloc_parallel(struct dentry *, const struct qstr *);
extern struct dentry * d_splice_alias(struct inode *, struct dentry *);
/* weird procfs mess; *NOT* exported */
extern struct dentry * d_splice_alias_ops(struct inode *, struct dentry *,
@@ -281,7 +280,7 @@ extern void d_tmpfile(struct file *, struct inode *);
extern struct dentry *d_find_alias(struct inode *);
extern void d_prune_aliases(struct inode *);
-extern void d_dispose_if_unused(struct dentry *, struct list_head *);
+extern bool __move_to_shrink_list(struct dentry *, struct list_head *);
extern void shrink_dentry_list(struct list_head *);
extern struct dentry *d_find_alias_rcu(struct inode *);
@@ -366,6 +365,24 @@ static inline struct dentry *dget(struct dentry *dentry)
return dentry;
}
+/* dentry->d_inode->i_lock must be held by caller */
+static inline bool dget_alias_ilocked(struct dentry *dentry)
+{
+ if (likely(!(READ_ONCE(dentry->d_flags) & DCACHE_NORCU))) {
+ lockref_get(&dentry->d_lockref);
+ return true;
+ }
+ // NORCU dentries with zero refcount MUST NOT be grabbed
+ spin_lock(&dentry->d_lock);
+ if (dentry->d_lockref.count > 0) {
+ dget_dlock(dentry);
+ spin_unlock(&dentry->d_lock);
+ return true;
+ }
+ spin_unlock(&dentry->d_lock);
+ return false;
+}
+
extern struct dentry *dget_parent(struct dentry *dentry);
/**
diff --git a/include/linux/eventpoll.h b/include/linux/eventpoll.h
index 728fb5d..de1c738 100644
--- a/include/linux/eventpoll.h
+++ b/include/linux/eventpoll.h
@@ -61,8 +61,16 @@ static inline void eventpoll_release(struct file *file)
eventpoll_release_file(file);
}
+struct epoll_key {
+ struct file *file;
+ int fd;
+} __packed;
+
+int do_epoll_ctl_file(struct file *f, int op, struct epoll_key *tf,
+ struct epoll_event *epds, bool nonblock);
int do_epoll_ctl(int epfd, int op, int fd, struct epoll_event *epds,
bool nonblock);
+bool is_file_epoll(struct file *f);
/* Tells if the epoll_ctl(2) operation needs an event copy from userspace */
static inline int ep_op_has_event(int op)
diff --git a/include/linux/exportfs.h b/include/linux/exportfs.h
index 8bcdba2..c835bc6 100644
--- a/include/linux/exportfs.h
+++ b/include/linux/exportfs.h
@@ -6,9 +6,8 @@
#include <linux/path.h>
struct dentry;
-struct iattr;
+struct exportfs_block_ops;
struct inode;
-struct iomap;
struct super_block;
struct vfsmount;
@@ -260,19 +259,13 @@ struct handle_to_path_ctx {
* @commit_metadata:
* @commit_metadata should commit metadata changes to stable storage.
*
- * @get_uuid:
- * Get a filesystem unique signature exposed to clients.
- *
- * @map_blocks:
- * Map and, if necessary, allocate blocks for a layout.
- *
- * @commit_blocks:
- * Commit blocks in a layout once the client is done with them.
- *
* @flags:
* Allows the filesystem to communicate to nfsd that it may want to do things
* differently when dealing with it.
*
+ * @block_ops:
+ * Operations for layout grants to block on the underlying device.
+ *
* Locking rules:
* get_parent is called with child->d_inode->i_rwsem down
* get_name is not (which is possibly inconsistent)
@@ -290,12 +283,6 @@ struct export_operations {
struct dentry * (*get_parent)(struct dentry *child);
int (*commit_metadata)(struct inode *inode);
- int (*get_uuid)(struct super_block *sb, u8 *buf, u32 *len, u64 *offset);
- int (*map_blocks)(struct inode *inode, loff_t offset,
- u64 len, struct iomap *iomap,
- bool write, u32 *device_generation);
- int (*commit_blocks)(struct inode *inode, struct iomap *iomaps,
- int nr_iomaps, struct iattr *iattr);
int (*permission)(struct handle_to_path_ctx *ctx, unsigned int oflags);
struct file * (*open)(const struct path *path, unsigned int oflags);
#define EXPORT_OP_NOWCC (0x1) /* don't collect v3 wcc data */
@@ -308,6 +295,10 @@ struct export_operations {
#define EXPORT_OP_FLUSH_ON_CLOSE (0x20) /* fs flushes file data on close */
#define EXPORT_OP_NOLOCKS (0x40) /* no file locking support */
unsigned long flags;
+
+#ifdef CONFIG_EXPORTFS_BLOCK_OPS
+ const struct exportfs_block_ops *block_ops;
+#endif
};
/**
diff --git a/include/linux/exportfs_block.h b/include/linux/exportfs_block.h
new file mode 100644
index 0000000..de519b7
--- /dev/null
+++ b/include/linux/exportfs_block.h
@@ -0,0 +1,88 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (c) 2014-2026 Christoph Hellwig.
+ *
+ * Support for exportfs-based layout grants for direct block device access.
+ */
+#ifndef LINUX_EXPORTFS_BLOCK_H
+#define LINUX_EXPORTFS_BLOCK_H 1
+
+#include <linux/blkdev.h>
+#include <linux/exportfs.h>
+#include <linux/fs.h>
+
+struct inode;
+struct iomap;
+struct super_block;
+
+/*
+ * There are the two types of block-style layout support:
+ * - In-band implies a device identified by a unique cookie inside the actual
+ * device address space checked by the ->get_uuid method as used by the pNFS
+ * block layout. This is a bit dangerous and deprecated.
+ * - Out of band implies identification by out of band unique identifiers
+ * specified by the storage protocol, which is much safer and used by the
+ * pNFS SCSI/NVMe layouts.
+ */
+typedef unsigned int __bitwise expfs_block_layouts_t;
+#define EXPFS_BLOCK_FLAG(__bit) \
+ ((__force expfs_block_layouts_t)(1u << __bit))
+#define EXPFS_BLOCK_IN_BAND_ID EXPFS_BLOCK_FLAG(0)
+#define EXPFS_BLOCK_OUT_OF_BAND_ID EXPFS_BLOCK_FLAG(1)
+
+struct exportfs_block_ops {
+ /*
+ * Returns the EXPFS_BLOCK_* bitmap of supported layout types.
+ */
+ expfs_block_layouts_t (*layouts_supported)(struct super_block *sb);
+
+ /*
+ * Get the in-band device unique signature exposed to clients.
+ */
+ int (*get_uuid)(struct super_block *sb, u8 *buf, u32 *len, u64 *offset);
+
+ /*
+ * Map blocks for direct block access.
+ * If @write is %true, also allocate the blocks for the range if needed.
+ */
+ int (*map_blocks)(struct inode *inode, loff_t offset, u64 len,
+ struct iomap *iomap, bool write,
+ u32 *device_generation);
+
+ /*
+ * Commit blocks previously handed out by ->map_blocks and written to by
+ * the client.
+ */
+ int (*commit_blocks)(struct inode *inode, struct iomap *iomaps,
+ int nr_iomaps, loff_t new_size);
+};
+
+static inline bool
+exportfs_bdev_supports_out_of_band_id(struct block_device *bdev)
+{
+ return bdev->bd_disk->fops->pr_ops &&
+ bdev->bd_disk->fops->get_unique_id;
+}
+
+#ifdef CONFIG_EXPORTFS_BLOCK_OPS
+static inline expfs_block_layouts_t
+exportfs_layouts_supported(struct super_block *sb)
+{
+ const struct exportfs_block_ops *bops = sb->s_export_op->block_ops;
+
+ if (!bops ||
+ !bops->layouts_supported ||
+ WARN_ON_ONCE(!bops->map_blocks) ||
+ WARN_ON_ONCE(!bops->commit_blocks))
+ return 0;
+ return bops->layouts_supported(sb);
+}
+#else
+static inline expfs_block_layouts_t
+exportfs_layouts_supported(struct super_block *sb)
+{
+ return 0;
+}
+#endif /* CONFIG_EXPORTFS_BLOCK_OPS */
+
+#endif /* LINUX_EXPORTFS_BLOCK_H */
diff --git a/include/linux/fcntl.h b/include/linux/fcntl.h
index a332e79..6ad6b9e 100644
--- a/include/linux/fcntl.h
+++ b/include/linux/fcntl.h
@@ -4,13 +4,31 @@
#include <linux/stat.h>
#include <uapi/linux/fcntl.h>
+#include <uapi/linux/openat2.h>
/* List of all valid flags for the open/openat flags argument: */
#define VALID_OPEN_FLAGS \
(O_RDONLY | O_WRONLY | O_RDWR | O_CREAT | O_EXCL | O_NOCTTY | O_TRUNC | \
O_APPEND | O_NDELAY | O_NONBLOCK | __O_SYNC | O_DSYNC | \
FASYNC | O_DIRECT | O_LARGEFILE | O_DIRECTORY | O_NOFOLLOW | \
- O_NOATIME | O_CLOEXEC | O_PATH | __O_TMPFILE)
+ O_NOATIME | O_CLOEXEC | O_PATH | __O_TMPFILE | O_EMPTYPATH)
+
+/* List of all valid flags for openat2(2)'s how->flags argument. */
+#define VALID_OPENAT2_FLAGS (VALID_OPEN_FLAGS | OPENAT2_REGULAR)
+
+/*
+ * Kernel-internal carrier for OPENAT2_REGULAR. The UAPI bit lives in the
+ * upper 32 bits of open_how::flags so open()/openat() cannot encode it.
+ * build_open_flags() translates it to this internal flag, which then
+ * propagates through op->open_flag and f->f_flags exactly like __FMODE_EXEC.
+ * do_dentry_open() strips it so userspace cannot observe it via
+ * fcntl(F_GETFL).
+ *
+ * Bit 30 is not claimed by any O_* flag on any architecture and stays clear
+ * of the sign bit of the int op->open_flag. fcntl_init() enforces that it
+ * never aliases an open-flag bit.
+ */
+#define __O_REGULAR (1 << 30)
/* List of all valid flags for the how->resolve argument: */
#define VALID_RESOLVE_FLAGS \
diff --git a/include/linux/fileattr.h b/include/linux/fileattr.h
index 3780904..58044b5 100644
--- a/include/linux/fileattr.h
+++ b/include/linux/fileattr.h
@@ -16,7 +16,8 @@
/* Read-only inode flags */
#define FS_XFLAG_RDONLY_MASK \
- (FS_XFLAG_PREALLOC | FS_XFLAG_HASATTR | FS_XFLAG_VERITY)
+ (FS_XFLAG_PREALLOC | FS_XFLAG_HASATTR | FS_XFLAG_VERITY | \
+ FS_XFLAG_CASEFOLD | FS_XFLAG_CASENONPRESERVING)
/* Flags to indicate valid value of fsx_ fields */
#define FS_XFLAG_VALUES_MASK \
diff --git a/include/linux/filelock.h b/include/linux/filelock.h
index 5f0a2fb..ec11cc6 100644
--- a/include/linux/filelock.h
+++ b/include/linux/filelock.h
@@ -4,19 +4,22 @@
#include <linux/fs.h>
-#define FL_POSIX 1
-#define FL_FLOCK 2
-#define FL_DELEG 4 /* NFSv4 delegation */
-#define FL_ACCESS 8 /* not trying to lock, just looking */
-#define FL_EXISTS 16 /* when unlocking, test for existence */
-#define FL_LEASE 32 /* lease held on this file */
-#define FL_CLOSE 64 /* unlock on close */
-#define FL_SLEEP 128 /* A blocking lock */
-#define FL_DOWNGRADE_PENDING 256 /* Lease is being downgraded */
-#define FL_UNLOCK_PENDING 512 /* Lease is being broken */
-#define FL_OFDLCK 1024 /* lock is "owned" by struct file */
-#define FL_LAYOUT 2048 /* outstanding pNFS layout */
-#define FL_RECLAIM 4096 /* reclaiming from a reboot server */
+#define FL_POSIX BIT(0) /* POSIX lock */
+#define FL_FLOCK BIT(1) /* BSD lock */
+#define FL_DELEG BIT(2) /* NFSv4 delegation */
+#define FL_ACCESS BIT(3) /* not trying to lock, just looking */
+#define FL_EXISTS BIT(4) /* when unlocking, test for existence */
+#define FL_LEASE BIT(5) /* file lease */
+#define FL_CLOSE BIT(6) /* unlock on close */
+#define FL_SLEEP BIT(7) /* A blocking lock */
+#define FL_DOWNGRADE_PENDING BIT(8) /* Lease is being downgraded */
+#define FL_UNLOCK_PENDING BIT(9) /* Lease is being broken */
+#define FL_OFDLCK BIT(10) /* POSIX lock "owned" by struct file */
+#define FL_LAYOUT BIT(11) /* outstanding pNFS layout */
+#define FL_RECLAIM BIT(12) /* reclaiming from a reboot server */
+#define FL_IGN_DIR_CREATE BIT(13) /* ignore DIR_CREATE events */
+#define FL_IGN_DIR_DELETE BIT(14) /* ignore DIR_DELETE events */
+#define FL_IGN_DIR_RENAME BIT(15) /* ignore DIR_RENAME events */
#define FL_CLOSE_POSIX (FL_POSIX | FL_CLOSE)
@@ -26,6 +29,15 @@
*/
#define FILE_LOCK_DEFERRED 1
+#define LEASE_BREAK_LEASE BIT(0) // break leases and delegations
+#define LEASE_BREAK_DELEG BIT(1) // break delegations only
+#define LEASE_BREAK_LAYOUT BIT(2) // break layouts only
+#define LEASE_BREAK_NONBLOCK BIT(3) // non-blocking break
+#define LEASE_BREAK_OPEN_RDONLY BIT(4) // readonly open event
+#define LEASE_BREAK_DIR_CREATE BIT(5) // dir deleg create event
+#define LEASE_BREAK_DIR_DELETE BIT(6) // dir deleg delete event
+#define LEASE_BREAK_DIR_RENAME BIT(7) // dir deleg rename event
+
struct file_lock;
struct file_lease;
@@ -216,19 +228,13 @@ int locks_lock_inode_wait(struct inode *inode, struct file_lock *fl);
void locks_init_lease(struct file_lease *);
void locks_free_lease(struct file_lease *fl);
struct file_lease *locks_alloc_lease(void);
-
-#define LEASE_BREAK_LEASE BIT(0) // break leases and delegations
-#define LEASE_BREAK_DELEG BIT(1) // break delegations only
-#define LEASE_BREAK_LAYOUT BIT(2) // break layouts only
-#define LEASE_BREAK_NONBLOCK BIT(3) // non-blocking break
-#define LEASE_BREAK_OPEN_RDONLY BIT(4) // readonly open event
-
int __break_lease(struct inode *inode, unsigned int flags);
void lease_get_mtime(struct inode *, struct timespec64 *time);
int generic_setlease(struct file *, int, struct file_lease **, void **priv);
int kernel_setlease(struct file *, int, struct file_lease **, void **);
int vfs_setlease(struct file *, int, struct file_lease **, void **);
int lease_modify(struct file_lease *, int, struct list_head *);
+u32 inode_lease_ignore_mask(struct inode *inode);
struct notifier_block;
int lease_register_notifier(struct notifier_block *);
@@ -516,12 +522,26 @@ static inline bool is_delegated(struct delegated_inode *di)
return di->di_inode;
}
-static inline int try_break_deleg(struct inode *inode,
+/**
+ * try_break_deleg - do a non-blocking delegation break
+ * @inode: inode that should have its delegations broken
+ * @flags: extra LEASE_BREAK_* flags to pass to break_deleg()
+ * @di: returns pointer to delegated inode (may be NULL)
+ *
+ * Break delegations in a non-blocking fashion. If there are
+ * outstanding delegations and @di is set, then an extra reference
+ * will be taken on @inode and @di->di_inode will be populated so
+ * that it may be waited upon.
+ *
+ * Returns 0 if there is no need to wait or an error. If -EWOULDBLOCK
+ * is returned, then @di will be populated (if non-NULL).
+ */
+static inline int try_break_deleg(struct inode *inode, unsigned int flags,
struct delegated_inode *di)
{
int ret;
- ret = break_deleg(inode, LEASE_BREAK_NONBLOCK);
+ ret = break_deleg(inode, flags | LEASE_BREAK_NONBLOCK);
if (ret == -EWOULDBLOCK && di) {
di->di_inode = inode;
ihold(inode);
@@ -564,7 +584,7 @@ static inline bool is_delegated(struct delegated_inode *di)
return false;
}
-static inline int break_lease(struct inode *inode, bool wait)
+static inline int break_lease(struct inode *inode, unsigned int mode)
{
return 0;
}
@@ -574,7 +594,7 @@ static inline int break_deleg(struct inode *inode, unsigned int flags)
return 0;
}
-static inline int try_break_deleg(struct inode *inode,
+static inline int try_break_deleg(struct inode *inode, unsigned int flags,
struct delegated_inode *delegated_inode)
{
return 0;
diff --git a/include/linux/firmware/meson/meson_sm.h b/include/linux/firmware/meson/meson_sm.h
index 8eaf892..3ebc2bd 100644
--- a/include/linux/firmware/meson/meson_sm.h
+++ b/include/linux/firmware/meson/meson_sm.h
@@ -12,6 +12,7 @@ enum {
SM_EFUSE_WRITE,
SM_EFUSE_USER_MAX,
SM_GET_CHIP_ID,
+ SM_THERMAL_CALIB_READ,
SM_A1_PWRC_SET,
SM_A1_PWRC_GET,
};
@@ -27,5 +28,7 @@ int meson_sm_call_read(struct meson_sm_firmware *fw, void *buffer,
unsigned int bsize, unsigned int cmd_index, u32 arg0,
u32 arg1, u32 arg2, u32 arg3, u32 arg4);
struct meson_sm_firmware *meson_sm_get(struct device_node *firmware_node);
+int meson_sm_get_thermal_calib(struct meson_sm_firmware *fw, u32 *trim_info,
+ u32 tsensor_id);
#endif /* _MESON_SM_FW_H_ */
diff --git a/include/linux/fs.h b/include/linux/fs.h
index 11559c5..6da4457 100644
--- a/include/linux/fs.h
+++ b/include/linux/fs.h
@@ -2218,8 +2218,21 @@ static inline void mark_inode_dirty_sync(struct inode *inode)
__mark_inode_dirty(inode, I_DIRTY_SYNC);
}
+/*
+ * returns the refcount on the inode. it can change arbitrarily.
+ */
+static inline int icount_read_once(const struct inode *inode)
+{
+ return atomic_read(&inode->i_count);
+}
+
+/*
+ * returns the refcount on the inode. The lock guarantees no 0->1 or 1->0 transitions
+ * of the count are going to take place, otherwise it changes arbitrarily.
+ */
static inline int icount_read(const struct inode *inode)
{
+ lockdep_assert_held(&inode->i_lock);
return atomic_read(&inode->i_count);
}
@@ -2281,12 +2294,14 @@ struct file_system_type {
#define FS_MGTIME 64 /* FS uses multigrain timestamps */
#define FS_LBS 128 /* FS supports LBS */
#define FS_POWER_FREEZE 256 /* Always freeze on suspend/hibernate */
+#define FS_USERNS_MOUNT_RESTRICTED 512 /* Restrict mount in userns if not already visible */
+#define FS_USERNS_DELEGATABLE 1024 /* Can be mounted inside userns from outside */
#define FS_RENAME_DOES_D_MOVE 32768 /* FS will handle d_move() during rename() internally. */
int (*init_fs_context)(struct fs_context *);
const struct fs_parameter_spec *parameters;
void (*kill_sb) (struct super_block *);
struct module *owner;
- struct file_system_type * next;
+ struct hlist_node list;
struct hlist_head fs_supers;
struct lock_class_key s_lock_key;
@@ -2327,10 +2342,6 @@ void free_anon_bdev(dev_t);
struct super_block *sget_fc(struct fs_context *fc,
int (*test)(struct super_block *, struct fs_context *),
int (*set)(struct super_block *, struct fs_context *));
-struct super_block *sget(struct file_system_type *type,
- int (*test)(struct super_block *,void *),
- int (*set)(struct super_block *,void *),
- int flags, void *data);
struct super_block *sget_dev(struct fs_context *fc, dev_t dev);
/* Alas, no aliases. Too much hassle with bringing module.h everywhere */
@@ -2624,6 +2635,7 @@ extern int __must_check file_write_and_wait_range(struct file *file,
loff_t start, loff_t end);
int filemap_flush_range(struct address_space *mapping, loff_t start,
loff_t end);
+void filemap_dontcache_kick_writeback(struct address_space *mapping);
static inline int file_write_and_wait(struct file *file)
{
@@ -2657,10 +2669,7 @@ static inline ssize_t generic_write_sync(struct kiocb *iocb, ssize_t count)
if (ret)
return ret;
} else if (iocb->ki_flags & IOCB_DONTCACHE) {
- struct address_space *mapping = iocb->ki_filp->f_mapping;
-
- filemap_flush_range(mapping, iocb->ki_pos - count,
- iocb->ki_pos - 1);
+ filemap_dontcache_kick_writeback(iocb->ki_filp->f_mapping);
}
return count;
diff --git a/include/linux/fs/super_types.h b/include/linux/fs/super_types.h
index 383050e..ef7941e 100644
--- a/include/linux/fs/super_types.h
+++ b/include/linux/fs/super_types.h
@@ -162,7 +162,8 @@ struct super_block {
struct unicode_map *s_encoding;
__u16 s_encoding_flags;
#endif
- struct hlist_bl_head s_roots; /* alternate root dentries for NFS */
+ struct hlist_head s_roots; /* alternate root dentries for NFS */
+ spinlock_t s_roots_lock;
struct mount *s_mounts; /* list of mounts; _not_ for fs use */
struct block_device *s_bdev; /* can go away once we use an accessor for @s_bdev_file */
struct file *s_bdev_file;
@@ -274,6 +275,14 @@ struct super_block {
/* number of fserrors that are being sent to fsnotify/filesystems */
refcount_t s_pending_errors;
+
+#ifdef CONFIG_CGROUP_WRITEBACK
+ /*
+ * Number of in-flight inode wb switches for this sb. Drained by
+ * cgroup_writeback_umount() before tear-down.
+ */
+ atomic_t s_isw_nr_in_flight;
+#endif
} __randomize_layout;
/*
@@ -326,7 +335,7 @@ struct super_block {
#define SB_I_STABLE_WRITES 0x00000008 /* don't modify blks until WB is done */
/* sb->s_iflags to limit user namespace mounts */
-#define SB_I_USERNS_VISIBLE 0x00000010 /* fstype already mounted */
+#define SB_I_RESTRICTED_VARIANT 0x00000010
#define SB_I_IMA_UNVERIFIABLE_SIGNATURE 0x00000020
#define SB_I_UNTRUSTED_MOUNTER 0x00000040
#define SB_I_EVM_HMAC_UNSUPPORTED 0x00000080
diff --git a/include/linux/fsnotify.h b/include/linux/fsnotify.h
index 079c18b..bda798b 100644
--- a/include/linux/fsnotify.h
+++ b/include/linux/fsnotify.h
@@ -257,6 +257,10 @@ static inline void fsnotify_move(struct inode *old_dir, struct inode *new_dir,
__u32 new_dir_mask = FS_MOVED_TO;
__u32 rename_mask = FS_RENAME;
const struct qstr *new_name = &moved->d_name;
+ struct fsnotify_rename_data rd = {
+ .moved = moved,
+ .target = target,
+ };
if (isdir) {
old_dir_mask |= FS_ISDIR;
@@ -265,12 +269,12 @@ static inline void fsnotify_move(struct inode *old_dir, struct inode *new_dir,
}
/* Event with information about both old and new parent+name */
- fsnotify_name(rename_mask, moved, FSNOTIFY_EVENT_DENTRY,
+ fsnotify_name(rename_mask, &rd, FSNOTIFY_EVENT_RENAME,
old_dir, old_name, 0);
fsnotify_name(old_dir_mask, source, FSNOTIFY_EVENT_INODE,
old_dir, old_name, fs_cookie);
- fsnotify_name(new_dir_mask, source, FSNOTIFY_EVENT_INODE,
+ fsnotify_name(new_dir_mask, &rd, FSNOTIFY_EVENT_RENAME,
new_dir, new_name, fs_cookie);
if (target)
diff --git a/include/linux/fsnotify_backend.h b/include/linux/fsnotify_backend.h
index e5cde39..618eed4 100644
--- a/include/linux/fsnotify_backend.h
+++ b/include/linux/fsnotify_backend.h
@@ -311,6 +311,7 @@ enum fsnotify_data_type {
FSNOTIFY_EVENT_DENTRY,
FSNOTIFY_EVENT_MNT,
FSNOTIFY_EVENT_ERROR,
+ FSNOTIFY_EVENT_RENAME,
};
struct fs_error_report {
@@ -335,6 +336,11 @@ struct fsnotify_mnt {
u64 mnt_id;
};
+struct fsnotify_rename_data {
+ struct dentry *moved; /* the dentry that was renamed */
+ struct inode *target; /* inode overwritten by rename, or NULL */
+};
+
static inline struct inode *fsnotify_data_inode(const void *data, int data_type)
{
switch (data_type) {
@@ -348,6 +354,8 @@ static inline struct inode *fsnotify_data_inode(const void *data, int data_type)
return d_inode(file_range_path(data)->dentry);
case FSNOTIFY_EVENT_ERROR:
return ((struct fs_error_report *)data)->inode;
+ case FSNOTIFY_EVENT_RENAME:
+ return d_inode(((const struct fsnotify_rename_data *)data)->moved);
default:
return NULL;
}
@@ -363,6 +371,8 @@ static inline struct dentry *fsnotify_data_dentry(const void *data, int data_typ
return ((const struct path *)data)->dentry;
case FSNOTIFY_EVENT_FILE_RANGE:
return file_range_path(data)->dentry;
+ case FSNOTIFY_EVENT_RENAME:
+ return ((struct fsnotify_rename_data *)data)->moved;
default:
return NULL;
}
@@ -395,6 +405,8 @@ static inline struct super_block *fsnotify_data_sb(const void *data,
return file_range_path(data)->dentry->d_sb;
case FSNOTIFY_EVENT_ERROR:
return ((struct fs_error_report *) data)->sb;
+ case FSNOTIFY_EVENT_RENAME:
+ return ((const struct fsnotify_rename_data *)data)->moved->d_sb;
default:
return NULL;
}
@@ -430,6 +442,14 @@ static inline struct fs_error_report *fsnotify_data_error_report(
}
}
+static inline struct inode *fsnotify_data_rename_target(const void *data,
+ int data_type)
+{
+ if (data_type == FSNOTIFY_EVENT_RENAME)
+ return ((const struct fsnotify_rename_data *)data)->target;
+ return NULL;
+}
+
static inline const struct file_range *fsnotify_data_file_range(
const void *data,
int data_type)
@@ -918,6 +938,7 @@ extern void fsnotify_put_mark(struct fsnotify_mark *mark);
struct fsnotify_mark *fsnotify_next_mark(struct fsnotify_mark *mark);
extern void fsnotify_finish_user_wait(struct fsnotify_iter_info *iter_info);
extern bool fsnotify_prepare_user_wait(struct fsnotify_iter_info *iter_info);
+extern void fsnotify_modify_mark_mask(struct fsnotify_mark *mark, u32 set, u32 clear);
static inline void fsnotify_init_event(struct fsnotify_event *event)
{
diff --git a/include/linux/fsverity.h b/include/linux/fsverity.h
index a8f9aa7..6c467de 100644
--- a/include/linux/fsverity.h
+++ b/include/linux/fsverity.h
@@ -201,6 +201,8 @@ bool fsverity_verify_blocks(struct fsverity_info *vi, struct folio *folio,
size_t len, size_t offset);
void fsverity_verify_bio(struct fsverity_info *vi, struct bio *bio);
void fsverity_enqueue_verify_work(struct work_struct *work);
+void fsverity_fill_zerohash(struct folio *folio, size_t offset, size_t len,
+ struct fsverity_info *vi);
#else /* !CONFIG_FS_VERITY */
@@ -281,6 +283,12 @@ static inline void fsverity_enqueue_verify_work(struct work_struct *work)
WARN_ON_ONCE(1);
}
+static inline void fsverity_fill_zerohash(struct folio *folio, size_t offset,
+ size_t len, struct fsverity_info *vi)
+{
+ WARN_ON_ONCE(1);
+}
+
#endif /* !CONFIG_FS_VERITY */
static inline bool fsverity_verify_folio(struct fsverity_info *vi,
diff --git a/include/linux/hwmon.h b/include/linux/hwmon.h
index 301a83a..a578a10 100644
--- a/include/linux/hwmon.h
+++ b/include/linux/hwmon.h
@@ -477,7 +477,8 @@ hwmon_device_register_with_info(struct device *dev,
const struct attribute_group **extra_groups);
struct device *
hwmon_device_register_for_thermal(struct device *dev, const char *name,
- void *drvdata);
+ void *drvdata,
+ const struct attribute_group **extra_groups);
struct device *
devm_hwmon_device_register_with_info(struct device *dev,
const char *name, void *drvdata,
diff --git a/include/linux/iomap.h b/include/linux/iomap.h
index 2c5685a..3582ed1 100644
--- a/include/linux/iomap.h
+++ b/include/linux/iomap.h
@@ -67,6 +67,9 @@ struct vm_fault;
* bio, i.e. set REQ_ATOMIC.
*
* IOMAP_F_INTEGRITY indicates that the filesystems handles integrity metadata.
+ *
+ * IOMAP_F_ZERO_TAIL indicates the remainder of the block after the data
+ * written should be zeroed.
*/
#define IOMAP_F_NEW (1U << 0)
#define IOMAP_F_DIRTY (1U << 1)
@@ -86,6 +89,15 @@ struct vm_fault;
#else
#define IOMAP_F_INTEGRITY 0
#endif /* CONFIG_BLK_DEV_INTEGRITY */
+#define IOMAP_F_ZERO_TAIL (1U << 10)
+
+/*
+ * Indicates reads and writes of fsverity metadata.
+ *
+ * Fsverity metadata is stored after the regular file data and thus beyond
+ * i_size.
+ */
+#define IOMAP_F_FSVERITY (1U << 11)
/*
* Flag reserved for file system specific usage
@@ -143,16 +155,6 @@ static inline void *iomap_inline_data(const struct iomap *iomap, loff_t pos)
}
/*
- * Check if the mapping's length is within the valid range for inline data.
- * This is used to guard against accessing data beyond the page inline_data
- * points at.
- */
-static inline bool iomap_inline_data_valid(const struct iomap *iomap)
-{
- return iomap->length <= PAGE_SIZE - offset_in_page(iomap->inline_data);
-}
-
-/*
* When get_folio succeeds, put_folio will always be called to do any
* cleanup work necessary. put_folio is responsible for unlocking and putting
* @folio.
@@ -351,6 +353,9 @@ static inline bool iomap_want_unshare_iter(const struct iomap_iter *iter)
ssize_t iomap_file_buffered_write(struct kiocb *iocb, struct iov_iter *from,
const struct iomap_ops *ops,
const struct iomap_write_ops *write_ops, void *private);
+int iomap_fsverity_write(struct file *file, loff_t pos, size_t length,
+ const void *buf, const struct iomap_ops *ops,
+ const struct iomap_write_ops *write_ops);
void iomap_read_folio(const struct iomap_ops *ops,
struct iomap_read_folio_ctx *ctx, void *private);
void iomap_readahead(const struct iomap_ops *ops,
@@ -427,6 +432,7 @@ struct iomap_ioend {
loff_t io_offset; /* offset in the file */
sector_t io_sector; /* start sector of ioend */
void *io_private; /* file system private data */
+ struct fsverity_info *io_vi; /* fsverity info */
struct bio io_bio; /* MUST BE LAST! */
};
@@ -501,6 +507,7 @@ struct iomap_read_folio_ctx {
struct readahead_control *rac;
void *read_ctx;
loff_t read_ctx_file_offset;
+ struct fsverity_info *vi;
};
struct iomap_read_ops {
diff --git a/include/linux/kcsan-checks.h b/include/linux/kcsan-checks.h
index 92f3843..e135dac 100644
--- a/include/linux/kcsan-checks.h
+++ b/include/linux/kcsan-checks.h
@@ -282,7 +282,7 @@ static inline void __kcsan_disable_current(void) { }
* @size: size of access
*/
#define __kcsan_check_write(ptr, size) \
- __kcsan_check_access(ptr, size, KCSAN_ACCESS_WRITE)
+ __kcsan_check_access(absolute_pointer(ptr), size, KCSAN_ACCESS_WRITE)
/**
* __kcsan_check_read_write - check regular read-write access for races
@@ -308,7 +308,7 @@ static inline void __kcsan_disable_current(void) { }
* @size: size of access
*/
#define kcsan_check_write(ptr, size) \
- kcsan_check_access(ptr, size, KCSAN_ACCESS_WRITE)
+ kcsan_check_access(absolute_pointer(ptr), size, KCSAN_ACCESS_WRITE)
/**
* kcsan_check_read_write - check regular read-write access for races
@@ -331,7 +331,7 @@ static inline void __kcsan_disable_current(void) { }
#define kcsan_check_atomic_read(ptr, size) \
kcsan_check_access(ptr, size, KCSAN_ACCESS_ATOMIC)
#define kcsan_check_atomic_write(ptr, size) \
- kcsan_check_access(ptr, size, KCSAN_ACCESS_ATOMIC | KCSAN_ACCESS_WRITE)
+ kcsan_check_access(absolute_pointer(ptr), size, KCSAN_ACCESS_ATOMIC | KCSAN_ACCESS_WRITE)
#define kcsan_check_atomic_read_write(ptr, size) \
kcsan_check_access(ptr, size, KCSAN_ACCESS_ATOMIC | KCSAN_ACCESS_WRITE | KCSAN_ACCESS_COMPOUND)
#endif
diff --git a/include/linux/kernfs.h b/include/linux/kernfs.h
index e21b2f7..351a510 100644
--- a/include/linux/kernfs.h
+++ b/include/linux/kernfs.h
@@ -76,20 +76,25 @@ struct kernfs_iattrs;
* kernfs_open_file.
*
* kernfs_open_files are chained at kernfs_open_node->files, which is
- * protected by kernfs_global_locks.open_file_mutex[i].
+ * protected by kernfs_global_locks.node_mutex[i].
*
* To reduce possible contention in sysfs access, arising due to single
- * locks, use an array of locks (e.g. open_file_mutex) and use kernfs_node
+ * locks, use an array of locks (e.g. node_mutex) and use kernfs_node
* object address as hash keys to get the index of these locks.
*
* Hashed mutexes are safe to use here because operations using these don't
* rely on global exclusion.
*
+ * The hashed mutex array protects per-node data: the kernfs_open_node for
+ * open file management, and kernfs_node xattr operations (necessary because
+ * multiple superblocks with different namespaces can share the same
+ * kernfs_node, making per-inode locking insufficient).
+ *
* In future we intend to replace other global locks with hashed ones as well.
* kernfs_global_locks acts as a holder for all such hash tables.
*/
struct kernfs_global_locks {
- struct mutex open_file_mutex[NR_KERNFS_LOCKS];
+ struct mutex node_mutex[NR_KERNFS_LOCKS];
};
enum kernfs_node_type {
diff --git a/include/linux/kstrtox.h b/include/linux/kstrtox.h
index 6ea8972..6c92828 100644
--- a/include/linux/kstrtox.h
+++ b/include/linux/kstrtox.h
@@ -142,10 +142,9 @@ static inline int __must_check kstrtos32_from_user(const char __user *s, size_t
* Keep in mind above caveat.
*/
-extern unsigned long simple_strtoul(const char *,char **,unsigned int);
-extern unsigned long simple_strntoul(const char *,char **,unsigned int,size_t);
-extern long simple_strtol(const char *,char **,unsigned int);
-extern unsigned long long simple_strtoull(const char *,char **,unsigned int);
-extern long long simple_strtoll(const char *,char **,unsigned int);
+unsigned long simple_strtoul(const char *cp, char **endp, unsigned int base);
+long simple_strtol(const char *cp, char **endp, unsigned int base);
+unsigned long long simple_strtoull(const char *cp, char **endp, unsigned int base);
+long long simple_strtoll(const char *cp, char **endp, unsigned int base);
#endif /* _LINUX_KSTRTOX_H */
diff --git a/include/linux/mlx5/vport.h b/include/linux/mlx5/vport.h
index dfa2fe3..282ed54 100644
--- a/include/linux/mlx5/vport.h
+++ b/include/linux/mlx5/vport.h
@@ -102,8 +102,8 @@ int mlx5_query_hca_vport_node_guid(struct mlx5_core_dev *dev,
int mlx5_query_nic_vport_mac_list(struct mlx5_core_dev *dev,
u16 vport,
enum mlx5_list_type list_type,
- u8 addr_list[][ETH_ALEN],
- int *list_size);
+ u8 (**mac_list)[ETH_ALEN],
+ int *mac_list_size);
int mlx5_modify_nic_vport_mac_list(struct mlx5_core_dev *dev,
enum mlx5_list_type list_type,
u8 addr_list[][ETH_ALEN],
diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h
index a308e2c..9588ce3 100644
--- a/include/linux/mm_types.h
+++ b/include/linux/mm_types.h
@@ -1342,7 +1342,6 @@ struct mm_struct {
*/
struct task_struct __rcu *owner;
#endif
- struct user_namespace *user_ns;
/* store ref to file /proc/<pid>/exe symlink points to */
struct file __rcu *exe_file;
@@ -1907,11 +1906,11 @@ enum {
/* mm flags */
/*
- * The first two bits represent core dump modes for set-user-ID,
- * the modes are SUID_DUMP_* defined in linux/sched/coredump.h
+ * Bits 0 and 1 were dumpability; that moved to task->exec_state. Reserve
+ * the bits so MMF_DUMP_FILTER_* positions stay stable for the
+ * /proc/<pid>/coredump_filter ABI.
*/
#define MMF_DUMPABLE_BITS 2
-#define MMF_DUMPABLE_MASK (BIT(MMF_DUMPABLE_BITS) - 1)
/* coredump filter bits */
#define MMF_DUMP_ANON_PRIVATE 2
#define MMF_DUMP_ANON_SHARED 3
@@ -1972,7 +1971,7 @@ enum {
#define MMF_TOPDOWN 31 /* mm searches top down by default */
#define MMF_TOPDOWN_MASK BIT(MMF_TOPDOWN)
-#define MMF_INIT_LEGACY_MASK (MMF_DUMPABLE_MASK | MMF_DUMP_FILTER_MASK |\
+#define MMF_INIT_LEGACY_MASK (MMF_DUMP_FILTER_MASK |\
MMF_DISABLE_THP_MASK | MMF_HAS_MDWE_MASK |\
MMF_VM_MERGE_ANY_MASK | MMF_TOPDOWN_MASK)
diff --git a/include/linux/namei.h b/include/linux/namei.h
index 2ad6dd9..ebe6e29 100644
--- a/include/linux/namei.h
+++ b/include/linux/namei.h
@@ -13,11 +13,6 @@ enum { MAX_NESTED_LINKS = 8 };
#define MAXSYMLINKS 40
-/*
- * Type of the last component on LOOKUP_PARENT
- */
-enum {LAST_NORM, LAST_ROOT, LAST_DOT, LAST_DOTDOT};
-
/* pathwalk mode */
#define LOOKUP_FOLLOW BIT(0) /* follow links at the end */
#define LOOKUP_DIRECTORY BIT(1) /* require a directory */
@@ -61,13 +56,12 @@ extern struct dentry *start_creating_path(int, const char *, struct path *, unsi
extern struct dentry *start_creating_user_path(int, const char __user *, struct path *, unsigned int);
extern void end_creating_path(const struct path *, struct dentry *);
extern struct dentry *start_removing_path(const char *, struct path *);
-extern struct dentry *start_removing_user_path_at(int , const char __user *, struct path *);
static inline void end_removing_path(const struct path *path , struct dentry *dentry)
{
end_creating_path(path, dentry);
}
int vfs_path_parent_lookup(struct filename *filename, unsigned int flags,
- struct path *parent, struct qstr *last, int *type,
+ struct path *parent, struct qstr *last,
const struct path *root);
int vfs_path_lookup(struct dentry *, struct vfsmount *, const char *,
unsigned int, struct path *);
diff --git a/include/linux/nfs_fs_sb.h b/include/linux/nfs_fs_sb.h
index 4daee27..34d2947 100644
--- a/include/linux/nfs_fs_sb.h
+++ b/include/linux/nfs_fs_sb.h
@@ -306,7 +306,7 @@ struct nfs_server {
#define NFS_CAP_ATOMIC_OPEN (1U << 4)
#define NFS_CAP_LGOPEN (1U << 5)
#define NFS_CAP_CASE_INSENSITIVE (1U << 6)
-#define NFS_CAP_CASE_PRESERVING (1U << 7)
+#define NFS_CAP_CASE_NONPRESERVING (1U << 7)
#define NFS_CAP_REBOOT_LAYOUTRETURN (1U << 8)
#define NFS_CAP_OFFLOAD_STATUS (1U << 9)
#define NFS_CAP_ZERO_RANGE (1U << 10)
diff --git a/include/linux/nfs_xdr.h b/include/linux/nfs_xdr.h
index fcbd21b..35ea18a 100644
--- a/include/linux/nfs_xdr.h
+++ b/include/linux/nfs_xdr.h
@@ -182,6 +182,8 @@ struct nfs_pathconf {
struct nfs_fattr *fattr; /* Post-op attributes */
__u32 max_link; /* max # of hard links */
__u32 max_namelen; /* max name length */
+ bool case_insensitive;
+ bool case_preserving;
};
struct nfs4_change_info {
@@ -1743,7 +1745,6 @@ struct nfs_unlinkdata {
struct nfs_removeargs args;
struct nfs_removeres res;
struct dentry *dentry;
- wait_queue_head_t wq;
const struct cred *cred;
struct nfs_fattr dir_attr;
long timeout;
diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h
index cdd68ed..07483ed 100644
--- a/include/linux/pgtable.h
+++ b/include/linux/pgtable.h
@@ -301,6 +301,28 @@ static inline void lazy_mmu_mode_disable(void)
}
/**
+ * __task_lazy_mmu_mode_pause() - Pause the lazy MMU mode for a task.
+ * @tsk: The task to check.
+ *
+ * Pauses the lazy MMU mode of @tsk.
+ *
+ * This function only operates on the state saved in task_struct; to pause
+ * current lazy_mmu_mode_pause() should be used instead.
+ *
+ * This function is intended for architectures that implement the lazy MMU
+ * mode; it must not be called from generic code.
+ */
+static inline void __task_lazy_mmu_mode_pause(struct task_struct *tsk)
+{
+ struct lazy_mmu_state *state = &tsk->lazy_mmu_state;
+
+ VM_WARN_ON_ONCE(state->pause_count == U8_MAX);
+
+ if (state->pause_count++ == 0 && state->enable_count > 0)
+ arch_leave_lazy_mmu_mode();
+}
+
+/**
* lazy_mmu_mode_pause() - Pause the lazy MMU mode.
*
* Pauses the lazy MMU mode; if it is currently active, disables it and calls
@@ -315,15 +337,32 @@ static inline void lazy_mmu_mode_disable(void)
*/
static inline void lazy_mmu_mode_pause(void)
{
- struct lazy_mmu_state *state = ¤t->lazy_mmu_state;
-
if (in_interrupt())
return;
- VM_WARN_ON_ONCE(state->pause_count == U8_MAX);
+ __task_lazy_mmu_mode_pause(current);
+}
- if (state->pause_count++ == 0 && state->enable_count > 0)
- arch_leave_lazy_mmu_mode();
+/**
+ * __task_lazy_mmu_mode_resume() - Resume the lazy MMU mode for a task.
+ * @tsk: The task to check.
+ *
+ * Resumes the lazy MMU mode of @tsk.
+ *
+ * This function only operates on the state saved in task_struct; to resume
+ * current lazy_mmu_mode_resume() should be used instead.
+ *
+ * This function is intended for architectures that implement the lazy MMU
+ * mode; it must not be called from generic code.
+ */
+static inline void __task_lazy_mmu_mode_resume(struct task_struct *tsk)
+{
+ struct lazy_mmu_state *state = &tsk->lazy_mmu_state;
+
+ VM_WARN_ON_ONCE(state->pause_count == 0);
+
+ if (--state->pause_count == 0 && state->enable_count > 0)
+ arch_enter_lazy_mmu_mode();
}
/**
@@ -341,15 +380,10 @@ static inline void lazy_mmu_mode_pause(void)
*/
static inline void lazy_mmu_mode_resume(void)
{
- struct lazy_mmu_state *state = ¤t->lazy_mmu_state;
-
if (in_interrupt())
return;
- VM_WARN_ON_ONCE(state->pause_count == 0);
-
- if (--state->pause_count == 0 && state->enable_count > 0)
- arch_enter_lazy_mmu_mode();
+ __task_lazy_mmu_mode_resume(current);
}
#else
static inline void lazy_mmu_mode_enable(void) {}
diff --git a/include/linux/proc_fs.h b/include/linux/proc_fs.h
index 19d1c5e..47d7dea 100644
--- a/include/linux/proc_fs.h
+++ b/include/linux/proc_fs.h
@@ -67,6 +67,7 @@ enum proc_pidonly {
struct proc_fs_info {
struct pid_namespace *pid_ns;
kgid_t pid_gid;
+ const struct cred *mounter_cred;
enum proc_hidepid hide_pid;
enum proc_pidonly pidonly;
struct rcu_head rcu;
@@ -248,4 +249,16 @@ static inline struct pid_namespace *proc_pid_ns(struct super_block *sb)
bool proc_ns_file(const struct file *file);
+#if defined CONFIG_PROC_FS && !defined MODULE
+void impl_proc_make_permanent(struct proc_dir_entry *pde);
+#endif
+
+static inline void proc_make_permanent(struct proc_dir_entry *pde)
+{
+ /* Don't give matches to modules. */
+#if defined CONFIG_PROC_FS && !defined MODULE
+ impl_proc_make_permanent(pde);
+#endif
+}
+
#endif /* _LINUX_PROC_FS_H */
diff --git a/include/linux/ptrace.h b/include/linux/ptrace.h
index 90507d4..ef314f7 100644
--- a/include/linux/ptrace.h
+++ b/include/linux/ptrace.h
@@ -17,6 +17,7 @@ struct syscall_info {
struct seccomp_data data;
};
+bool ptracer_access_allowed(struct task_struct *tsk);
extern int ptrace_access_vm(struct task_struct *tsk, unsigned long addr,
void *buf, int len, unsigned int gup_flags);
diff --git a/include/linux/rcupdate.h b/include/linux/rcupdate.h
index bfa76513..5e95acc 100644
--- a/include/linux/rcupdate.h
+++ b/include/linux/rcupdate.h
@@ -592,11 +592,13 @@ context_unsafe( \
* lockdep checks for being in an RCU read-side critical section. This is
* useful when the value of this pointer is accessed, but the pointer is
* not dereferenced, for example, when testing an RCU-protected pointer
- * against NULL. Although rcu_access_pointer() may also be used in cases
- * where update-side locks prevent the value of the pointer from changing,
- * you should instead use rcu_dereference_protected() for this use case.
- * Within an RCU read-side critical section, there is little reason to
- * use rcu_access_pointer().
+ * against NULL. Within an RCU read-side critical section, there is little
+ * reason to use rcu_access_pointer(). Although rcu_access_pointer() may
+ * also be used in cases where update-side locks prevent the value of the
+ * pointer from changing, you should instead use rcu_dereference_protected()
+ * for this use case. It is also permissible to use rcu_access_pointer()
+ * within lockless updaters to obtain the old value for an atomic operation,
+ * for example, for cmpxchg().
*
* It is usually best to test the rcu_access_pointer() return value
* directly in order to avoid accidental dereferences being introduced
diff --git a/include/linux/rhashtable-types.h b/include/linux/rhashtable-types.h
index fc2f596..57c11ec 100644
--- a/include/linux/rhashtable-types.h
+++ b/include/linux/rhashtable-types.h
@@ -136,12 +136,26 @@ struct rhashtable_iter {
bool end_of_table;
};
-int rhashtable_init_noprof(struct rhashtable *ht,
- const struct rhashtable_params *params);
+int __rhashtable_init_noprof(struct rhashtable *ht,
+ const struct rhashtable_params *params,
+ struct lock_class_key *key);
+#define rhashtable_init_noprof(ht, params) \
+({ \
+ static struct lock_class_key __key; \
+ \
+ __rhashtable_init_noprof(ht, params, &__key); \
+})
#define rhashtable_init(...) alloc_hooks(rhashtable_init_noprof(__VA_ARGS__))
-int rhltable_init_noprof(struct rhltable *hlt,
- const struct rhashtable_params *params);
+int __rhltable_init_noprof(struct rhltable *hlt,
+ const struct rhashtable_params *params,
+ struct lock_class_key *key);
+#define rhltable_init_noprof(hlt, params) \
+({ \
+ static struct lock_class_key __key; \
+ \
+ __rhltable_init_noprof(hlt, params, &__key); \
+})
#define rhltable_init(...) alloc_hooks(rhltable_init_noprof(__VA_ARGS__))
#endif /* _LINUX_RHASHTABLE_TYPES_H */
diff --git a/include/linux/sched.h b/include/linux/sched.h
index ee06cba..258cb07 100644
--- a/include/linux/sched.h
+++ b/include/linux/sched.h
@@ -85,6 +85,7 @@ struct seq_file;
struct sighand_struct;
struct signal_struct;
struct task_delay_info;
+struct task_exec_state;
struct task_group;
struct task_struct;
struct timespec64;
@@ -962,6 +963,8 @@ struct task_struct {
struct mm_struct *mm;
struct mm_struct *active_mm;
+ struct task_exec_state __rcu *exec_state;
+
int exit_state;
int exit_code;
int exit_signal;
@@ -1002,9 +1005,6 @@ struct task_struct {
unsigned sched_rt_mutex:1;
#endif
- /* Save user-dumpable when mm goes away */
- unsigned user_dumpable:1;
-
/* Bit to tell TOMOYO we're in execve(): */
unsigned in_execve:1;
unsigned in_iowait:1;
diff --git a/include/linux/sched/coredump.h b/include/linux/sched/coredump.h
index 624fda1..20957cc 100644
--- a/include/linux/sched/coredump.h
+++ b/include/linux/sched/coredump.h
@@ -2,43 +2,18 @@
#ifndef _LINUX_SCHED_COREDUMP_H
#define _LINUX_SCHED_COREDUMP_H
-#include <linux/mm_types.h>
-
-#define SUID_DUMP_DISABLE 0 /* No setuid dumping */
-#define SUID_DUMP_USER 1 /* Dump as user of process */
-#define SUID_DUMP_ROOT 2 /* Dump as root */
-
-static inline unsigned long __mm_flags_get_dumpable(const struct mm_struct *mm)
-{
- /*
- * By convention, dumpable bits are contained in first 32 bits of the
- * bitmap, so we can simply access this first unsigned long directly.
- */
- return __mm_flags_get_word(mm);
-}
-
-static inline void __mm_flags_set_mask_dumpable(struct mm_struct *mm, int value)
-{
- __mm_flags_set_mask_bits_word(mm, MMF_DUMPABLE_MASK, value);
-}
-
-extern void set_dumpable(struct mm_struct *mm, int value);
/*
- * This returns the actual value of the suid_dumpable flag. For things
- * that are using this for checking for privilege transitions, it must
- * test against SUID_DUMP_USER rather than treating it as a boolean
- * value.
+ * Task dumpability mode. Gates core dump production and ptrace_attach()
+ * authorization. The numeric values are stable ABI (suid_dumpable
+ * sysctl, prctl(PR_SET_DUMPABLE)); do not renumber.
*/
-static inline int __get_dumpable(unsigned long mm_flags)
-{
- return mm_flags & MMF_DUMPABLE_MASK;
-}
+enum task_dumpable {
+ TASK_DUMPABLE_OFF = 0, /* no dump; ptrace needs CAP_SYS_PTRACE */
+ TASK_DUMPABLE_OWNER = 1, /* default; dump and ptrace by uid match */
+ TASK_DUMPABLE_ROOT = 2, /* dump as root; ptrace needs CAP_SYS_PTRACE */
+};
-static inline int get_dumpable(struct mm_struct *mm)
-{
- unsigned long flags = __mm_flags_get_dumpable(mm);
-
- return __get_dumpable(flags);
-}
+void task_exec_state_set_dumpable(enum task_dumpable value);
+enum task_dumpable task_exec_state_get_dumpable(struct task_struct *task);
#endif /* _LINUX_SCHED_COREDUMP_H */
diff --git a/include/linux/sched/exec_state.h b/include/linux/sched/exec_state.h
new file mode 100644
index 0000000..9b61782
--- /dev/null
+++ b/include/linux/sched/exec_state.h
@@ -0,0 +1,31 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2026 Christian Brauner <brauner@kernel.org> */
+#ifndef _LINUX_SCHED_EXEC_STATE_H
+#define _LINUX_SCHED_EXEC_STATE_H
+
+#include <linux/init.h>
+#include <linux/rcupdate.h>
+#include <linux/refcount.h>
+#include <linux/sched/coredump.h>
+#include <linux/user_namespace.h>
+
+struct task_exec_state {
+ refcount_t count;
+ enum task_dumpable dumpable;
+ struct user_namespace *user_ns;
+ struct rcu_head rcu;
+};
+
+extern struct task_exec_state init_task_exec_state;
+
+struct task_exec_state *alloc_task_exec_state(struct user_namespace *user_ns);
+void put_task_exec_state(struct task_exec_state *exec_state);
+struct task_exec_state *task_exec_state_rcu(const struct task_struct *tsk);
+struct task_exec_state *task_exec_state_replace(struct task_struct *tsk,
+ struct task_exec_state *exec_state);
+int task_exec_state_copy(struct task_struct *tsk);
+void __init exec_state_init(void);
+
+DEFINE_FREE(put_task_exec_state, struct task_exec_state *, put_task_exec_state(_T))
+
+#endif /* _LINUX_SCHED_EXEC_STATE_H */
diff --git a/include/linux/shmem_fs.h b/include/linux/shmem_fs.h
index 93a0ba8..69b0177 100644
--- a/include/linux/shmem_fs.h
+++ b/include/linux/shmem_fs.h
@@ -48,7 +48,7 @@ struct shmem_inode_info {
};
struct timespec64 i_crtime; /* file creation time */
struct shared_policy policy; /* NUMA memory alloc policy */
- struct simple_xattrs *xattrs; /* list of xattrs */
+ struct list_head xattrs; /* list of xattrs */
pgoff_t fallocend; /* highest fallocate endindex */
unsigned int fsflags; /* for FS_IOC_[SG]ETFLAGS */
atomic_t stop_eviction; /* hold when working on inode */
@@ -89,6 +89,7 @@ struct shmem_sb_info {
struct list_head shrinklist; /* List of shinkable inodes */
unsigned long shrinklist_len; /* Length of shrinklist */
struct shmem_quota_limits qlimits; /* Default quota limits */
+ struct simple_xattr_cache xa_cache;
};
static inline struct shmem_inode_info *SHMEM_I(struct inode *inode)
diff --git a/include/linux/sockptr.h b/include/linux/sockptr.h
index 3e6c8e9..9c2429c 100644
--- a/include/linux/sockptr.h
+++ b/include/linux/sockptr.h
@@ -87,24 +87,10 @@ static inline int copy_safe_from_sockptr(void *dst, size_t ksize,
static inline int copy_struct_from_sockptr(void *dst, size_t ksize,
sockptr_t src, size_t usize)
{
- size_t size = min(ksize, usize);
- size_t rest = max(ksize, usize) - size;
-
if (!sockptr_is_kernel(src))
- return copy_struct_from_user(dst, ksize, src.user, size);
+ return copy_struct_from_user(dst, ksize, src.user, usize);
- if (usize < ksize) {
- memset(dst + size, 0, rest);
- } else if (usize > ksize) {
- char *p = src.kernel;
-
- while (rest--) {
- if (*p++)
- return -E2BIG;
- }
- }
- memcpy(dst, src.kernel, size);
- return 0;
+ return copy_struct_from_bounce_buffer(dst, ksize, src.kernel, usize);
}
static inline int copy_to_sockptr_offset(sockptr_t dst, size_t offset,
@@ -121,6 +107,16 @@ static inline int copy_to_sockptr(sockptr_t dst, const void *src, size_t size)
return copy_to_sockptr_offset(dst, 0, src, size);
}
+static inline int
+copy_struct_to_sockptr(sockptr_t dst, size_t usize, const void *src,
+ size_t ksize, bool *ignored_trailing)
+{
+ if (!sockptr_is_kernel(dst))
+ return copy_struct_to_user(dst.user, usize, src, ksize, ignored_trailing);
+
+ return copy_struct_to_bounce_buffer(dst.kernel, usize, src, ksize, ignored_trailing);
+}
+
static inline void *memdup_sockptr_noprof(sockptr_t src, size_t len)
{
void *p = kmalloc_track_caller_noprof(len, GFP_USER | __GFP_NOWARN);
diff --git a/include/linux/srcu.h b/include/linux/srcu.h
index 81b1938..a54ce9e 100644
--- a/include/linux/srcu.h
+++ b/include/linux/srcu.h
@@ -397,7 +397,7 @@ static inline struct srcu_ctr __percpu *srcu_read_lock_fast_notrace(struct srcu_
*
* The same srcu_struct may be used concurrently by srcu_down_read_fast()
* and srcu_read_lock_fast(). However, the same definition/initialization
- * requirements called out for srcu_read_lock_safe() apply.
+ * requirements called out for srcu_read_lock_fast_updown() apply.
*/
static inline struct srcu_ctr __percpu *srcu_down_read_fast(struct srcu_struct *ssp) __acquires_shared(ssp)
{
diff --git a/include/linux/thermal.h b/include/linux/thermal.h
index 0ddc77a..083b4f5 100644
--- a/include/linux/thermal.h
+++ b/include/linux/thermal.h
@@ -125,7 +125,6 @@ struct thermal_cooling_device {
const char *type;
unsigned long max_state;
struct device device;
- struct device_node *np;
void *devdata;
void *stats;
const struct thermal_cooling_device_ops *ops;
@@ -133,6 +132,10 @@ struct thermal_cooling_device {
struct mutex lock; /* protect thermal_instances list */
struct list_head thermal_instances;
struct list_head node;
+#ifdef CONFIG_THERMAL_OF
+ struct device_node *np;
+ u32 cdev_id;
+#endif
#ifdef CONFIG_THERMAL_DEBUGFS
struct thermal_debugfs *debugfs;
#endif
@@ -198,6 +201,21 @@ struct thermal_zone_device *devm_thermal_of_zone_register(struct device *dev, in
void devm_thermal_of_zone_unregister(struct device *dev, struct thermal_zone_device *tz);
+struct thermal_cooling_device *
+thermal_of_cooling_device_register(struct device_node *np, u32 cdev_id,
+ const char *type, void *data,
+ const struct thermal_cooling_device_ops *ops);
+
+struct thermal_cooling_device *
+devm_thermal_of_cooling_device_register(struct device *dev, u32 cdev_id,
+ const char *type, void *devdata,
+ const struct thermal_cooling_device_ops *ops);
+
+struct thermal_cooling_device *
+devm_thermal_of_child_cooling_device_register(struct device *dev,
+ struct device_node *np,
+ const char *type, void *devdata,
+ const struct thermal_cooling_device_ops *ops);
#else
static inline
@@ -211,6 +229,31 @@ static inline void devm_thermal_of_zone_unregister(struct device *dev,
struct thermal_zone_device *tz)
{
}
+
+static inline struct thermal_cooling_device *
+thermal_of_cooling_device_register(struct device_node *np, u32 cdev_id,
+ const char *type, void *devdata,
+ const struct thermal_cooling_device_ops *ops)
+{
+ return ERR_PTR(-ENODEV);
+}
+
+static inline struct thermal_cooling_device *
+devm_thermal_of_cooling_device_register(struct device *dev, u32 cdev_id,
+ const char *type, void *devdata,
+ const struct thermal_cooling_device_ops *ops)
+{
+ return ERR_PTR(-ENODEV);
+}
+
+static inline struct thermal_cooling_device *
+devm_thermal_of_child_cooling_device_register(struct device *dev,
+ struct device_node *np,
+ const char *type, void *devdata,
+ const struct thermal_cooling_device_ops *ops)
+{
+ return ERR_PTR(-ENODEV);
+}
#endif
int for_each_thermal_trip(struct thermal_zone_device *tz,
@@ -252,14 +295,11 @@ void thermal_zone_device_update(struct thermal_zone_device *,
struct thermal_cooling_device *thermal_cooling_device_register(const char *,
void *, const struct thermal_cooling_device_ops *);
+
struct thermal_cooling_device *
-thermal_of_cooling_device_register(struct device_node *np, const char *, void *,
- const struct thermal_cooling_device_ops *);
-struct thermal_cooling_device *
-devm_thermal_of_cooling_device_register(struct device *dev,
- struct device_node *np,
- const char *type, void *devdata,
- const struct thermal_cooling_device_ops *ops);
+devm_thermal_cooling_device_register(struct device *dev, const char *type, void *devdata,
+ const struct thermal_cooling_device_ops *ops);
+
void thermal_cooling_device_update(struct thermal_cooling_device *);
void thermal_cooling_device_unregister(struct thermal_cooling_device *);
struct thermal_zone_device *thermal_zone_get_zone_by_name(const char *name);
@@ -304,19 +344,12 @@ static inline struct thermal_cooling_device *
thermal_cooling_device_register(const char *type, void *devdata,
const struct thermal_cooling_device_ops *ops)
{ return ERR_PTR(-ENODEV); }
+
static inline struct thermal_cooling_device *
-thermal_of_cooling_device_register(struct device_node *np,
- const char *type, void *devdata,
- const struct thermal_cooling_device_ops *ops)
+devm_thermal_cooling_device_register(struct device *dev, const char *type, void *devdata,
+ const struct thermal_cooling_device_ops *ops)
{ return ERR_PTR(-ENODEV); }
-static inline struct thermal_cooling_device *
-devm_thermal_of_cooling_device_register(struct device *dev,
- struct device_node *np,
- const char *type, void *devdata,
- const struct thermal_cooling_device_ops *ops)
-{
- return ERR_PTR(-ENODEV);
-}
+
static inline void thermal_cooling_device_unregister(
struct thermal_cooling_device *cdev)
{ }
diff --git a/include/linux/torture.h b/include/linux/torture.h
index 1b59056..c9b47d1 100644
--- a/include/linux/torture.h
+++ b/include/linux/torture.h
@@ -129,6 +129,7 @@ void _torture_stop_kthread(char *m, struct task_struct **tp);
#else
#define torture_preempt_schedule() do { } while (0)
#endif
+void torture_sched_set_normal(struct task_struct *t, int nice);
#if IS_ENABLED(CONFIG_RCU_TORTURE_TEST) || IS_MODULE(CONFIG_RCU_TORTURE_TEST) || IS_ENABLED(CONFIG_LOCK_TORTURE_TEST) || IS_MODULE(CONFIG_LOCK_TORTURE_TEST)
long torture_sched_setaffinity(pid_t pid, const struct cpumask *in_mask, bool dowarn);
diff --git a/include/linux/uaccess.h b/include/linux/uaccess.h
index 5632860..e4a6497 100644
--- a/include/linux/uaccess.h
+++ b/include/linux/uaccess.h
@@ -510,7 +510,7 @@ copy_struct_to_user(void __user *dst, size_t usize, const void *src,
return -EFAULT;
}
if (ignored_trailing)
- *ignored_trailing = ksize < usize &&
+ *ignored_trailing = usize < ksize &&
memchr_inv(src + size, 0, rest) != NULL;
/* Copy the interoperable parts of the struct. */
if (copy_to_user(dst, src, size))
@@ -518,6 +518,69 @@ copy_struct_to_user(void __user *dst, size_t usize, const void *src,
return 0;
}
+static __always_inline void
+__copy_struct_generic_bounce_buffer(void *dst, size_t dstsize,
+ const void *src, size_t srcsize,
+ bool *ignored_trailing)
+{
+ size_t size = min(dstsize, srcsize);
+ size_t rest = max(dstsize, srcsize) - size;
+
+ /* Deal with trailing bytes. */
+ if (dstsize > srcsize)
+ memset(dst + size, 0, rest);
+ if (ignored_trailing)
+ *ignored_trailing = dstsize < srcsize &&
+ memchr_inv(src + size, 0, rest) != NULL;
+ /* Copy the interoperable parts of the struct. */
+ memcpy(dst, src, size);
+}
+
+/**
+ * This is like copy_struct_from_user(), but the
+ * src buffer was already copied into a kernel
+ * bounce buffer, so it will never return -EFAULT.
+ */
+static __always_inline __must_check int
+copy_struct_from_bounce_buffer(void *dst, size_t dstsize,
+ const void *src, size_t srcsize)
+{
+ bool ignored_trailing;
+
+ /* Double check if ksize is larger than a known object size. */
+ if (WARN_ON_ONCE(dstsize > __builtin_object_size(dst, 1)))
+ return -E2BIG;
+
+ __copy_struct_generic_bounce_buffer(dst, dstsize,
+ src, srcsize,
+ &ignored_trailing);
+ if (unlikely(ignored_trailing))
+ return -E2BIG;
+
+ return 0;
+}
+
+/**
+ * This is like copy_struct_to_user(), but the
+ * dst buffer is a kernel bounce buffer instead
+ * of a direct userspace buffer, so it will never return -EFAULT.
+ */
+static __always_inline __must_check int
+copy_struct_to_bounce_buffer(void *dst, size_t dstsize,
+ const void *src,
+ size_t srcsize,
+ bool *ignored_trailing)
+{
+ /* Double check if srcsize is larger than a known object size. */
+ if (WARN_ON_ONCE(srcsize > __builtin_object_size(src, 1)))
+ return -E2BIG;
+
+ __copy_struct_generic_bounce_buffer(dst, dstsize,
+ src, srcsize,
+ ignored_trailing);
+ return 0;
+}
+
bool copy_from_kernel_nofault_allowed(const void *unsafe_src, size_t size);
long copy_from_kernel_nofault(void *dst, const void *src, size_t size);
diff --git a/include/linux/xattr.h b/include/linux/xattr.h
index 8b66013..54ac3cb 100644
--- a/include/linux/xattr.h
+++ b/include/linux/xattr.h
@@ -106,12 +106,14 @@ static inline const char *xattr_prefix(const struct xattr_handler *handler)
return handler->prefix ?: handler->name;
}
-struct simple_xattrs {
- struct rhashtable ht;
+struct simple_xattr_cache {
+ struct rhashtable *ht;
};
struct simple_xattr {
struct rhash_head hash_node;
+ struct list_head *parent;
+ struct list_head node;
struct rcu_head rcu;
char *name;
size_t size;
@@ -132,40 +134,39 @@ static inline void simple_xattr_limits_init(struct simple_xattr_limits *limits)
atomic_set(&limits->xattr_size, 0);
}
-int simple_xattrs_init(struct simple_xattrs *xattrs);
-struct simple_xattrs *simple_xattrs_alloc(void);
-struct simple_xattrs *simple_xattrs_lazy_alloc(struct simple_xattrs **xattrsp,
- const void *value, int flags);
-void simple_xattrs_free(struct simple_xattrs *xattrs, size_t *freed_space);
+void simple_xattrs_free(struct simple_xattr_cache *cache, struct list_head *xattrs,
+ size_t *freed_space);
size_t simple_xattr_space(const char *name, size_t size);
struct simple_xattr *simple_xattr_alloc(const void *value, size_t size);
void simple_xattr_free(struct simple_xattr *xattr);
void simple_xattr_free_rcu(struct simple_xattr *xattr);
-int simple_xattr_get(struct simple_xattrs *xattrs, const char *name,
- void *buffer, size_t size);
-struct simple_xattr *simple_xattr_set(struct simple_xattrs *xattrs,
+int simple_xattr_get(struct simple_xattr_cache *cache, struct list_head *xattrs,
+ const char *name, void *buffer, size_t size);
+struct simple_xattr *simple_xattr_set(struct simple_xattr_cache *cache,
+ struct list_head *xattrs,
const char *name, const void *value,
size_t size, int flags);
-int simple_xattr_set_limited(struct simple_xattrs *xattrs,
+int simple_xattr_set_limited(struct simple_xattr_cache *cache,
+ struct list_head *xattrs,
struct simple_xattr_limits *limits,
const char *name, const void *value,
size_t size, int flags);
-ssize_t simple_xattr_list(struct inode *inode, struct simple_xattrs *xattrs,
+ssize_t simple_xattr_list(struct inode *inode, struct list_head *xattrs,
char *buffer, size_t size);
-int simple_xattr_add(struct simple_xattrs *xattrs,
+int simple_xattr_add(struct simple_xattr_cache *cache, struct list_head *xattrs,
struct simple_xattr *new_xattr);
+int simple_xattr_add_limited(struct simple_xattr_cache *cache,
+ struct list_head *xattrs,
+ struct simple_xattr_limits *limits,
+ struct simple_xattr *new_xattr);
int xattr_list_one(char **buffer, ssize_t *remaining_size, const char *name);
+void simple_xattr_cache_cleanup(struct simple_xattr_cache *cache);
+
DEFINE_CLASS(simple_xattr,
struct simple_xattr *,
if (!IS_ERR_OR_NULL(_T)) simple_xattr_free(_T),
simple_xattr_alloc(value, size),
const void *value, size_t size)
-DEFINE_CLASS(simple_xattrs,
- struct simple_xattrs *,
- if (!IS_ERR_OR_NULL(_T)) { simple_xattrs_free(_T, NULL); kfree(_T); },
- simple_xattrs_alloc(),
- void)
-
#endif /* _LINUX_XATTR_H */
diff --git a/include/net/netfilter/nf_conntrack_helper.h b/include/net/netfilter/nf_conntrack_helper.h
index de2f956..24cf3d2 100644
--- a/include/net/netfilter/nf_conntrack_helper.h
+++ b/include/net/netfilter/nf_conntrack_helper.h
@@ -155,6 +155,7 @@ void nf_ct_helper_log(struct sk_buff *skb, const struct nf_conn *ct,
void nf_ct_helper_expectfn_register(struct nf_ct_helper_expectfn *n);
void nf_ct_helper_expectfn_unregister(struct nf_ct_helper_expectfn *n);
+void nf_ct_helper_expectfn_destroy(const struct nf_ct_helper_expectfn *n);
struct nf_ct_helper_expectfn *
nf_ct_helper_expectfn_find_by_name(const char *name);
struct nf_ct_helper_expectfn *
diff --git a/include/net/sock.h b/include/net/sock.h
index dccd373..95e157e 100644
--- a/include/net/sock.h
+++ b/include/net/sock.h
@@ -1856,6 +1856,7 @@ struct sk_buff *sock_omalloc(struct sock *sk, unsigned long size,
gfp_t priority);
void skb_orphan_partial(struct sk_buff *skb);
void sock_rfree(struct sk_buff *skb);
+void sock_rmem_free(struct sk_buff *skb);
void sock_efree(struct sk_buff *skb);
#ifdef CONFIG_INET
void sock_edemux(struct sk_buff *skb);
diff --git a/include/trace/events/filelock.h b/include/trace/events/filelock.h
index 1167748..f11284be 100644
--- a/include/trace/events/filelock.h
+++ b/include/trace/events/filelock.h
@@ -28,7 +28,10 @@
{ FL_DOWNGRADE_PENDING, "FL_DOWNGRADE_PENDING" }, \
{ FL_UNLOCK_PENDING, "FL_UNLOCK_PENDING" }, \
{ FL_OFDLCK, "FL_OFDLCK" }, \
- { FL_RECLAIM, "FL_RECLAIM"})
+ { FL_RECLAIM, "FL_RECLAIM" }, \
+ { FL_IGN_DIR_CREATE, "FL_IGN_DIR_CREATE" }, \
+ { FL_IGN_DIR_DELETE, "FL_IGN_DIR_DELETE" }, \
+ { FL_IGN_DIR_RENAME, "FL_IGN_DIR_RENAME" })
#define show_fl_type(val) \
__print_symbolic(val, \
@@ -117,6 +120,39 @@ DEFINE_EVENT(filelock_lock, flock_lock_inode,
TP_PROTO(struct inode *inode, struct file_lock *fl, int ret),
TP_ARGS(inode, fl, ret));
+#define show_lease_break_flags(val) \
+ __print_flags(val, "|", \
+ { LEASE_BREAK_LEASE, "LEASE" }, \
+ { LEASE_BREAK_DELEG, "DELEG" }, \
+ { LEASE_BREAK_LAYOUT, "LAYOUT" }, \
+ { LEASE_BREAK_NONBLOCK, "NONBLOCK" }, \
+ { LEASE_BREAK_OPEN_RDONLY, "OPEN_RDONLY" }, \
+ { LEASE_BREAK_DIR_CREATE, "DIR_CREATE" }, \
+ { LEASE_BREAK_DIR_DELETE, "DIR_DELETE" }, \
+ { LEASE_BREAK_DIR_RENAME, "DIR_RENAME" })
+
+TRACE_EVENT(break_lease,
+ TP_PROTO(struct inode *inode, unsigned int flags),
+
+ TP_ARGS(inode, flags),
+
+ TP_STRUCT__entry(
+ __field(unsigned long, i_ino)
+ __field(dev_t, s_dev)
+ __field(unsigned int, flags)
+ ),
+
+ TP_fast_assign(
+ __entry->s_dev = inode->i_sb->s_dev;
+ __entry->i_ino = inode->i_ino;
+ __entry->flags = flags;
+ ),
+
+ TP_printk("dev=0x%x:0x%x ino=0x%lx flags=%s",
+ MAJOR(__entry->s_dev), MINOR(__entry->s_dev),
+ __entry->i_ino, show_lease_break_flags(__entry->flags))
+);
+
DECLARE_EVENT_CLASS(filelock_lease,
TP_PROTO(struct inode *inode, struct file_lease *fl),
@@ -190,7 +226,7 @@ TRACE_EVENT(generic_add_lease,
__entry->i_ino = inode->i_ino;
__entry->wcount = atomic_read(&inode->i_writecount);
__entry->rcount = atomic_read(&inode->i_readcount);
- __entry->icount = icount_read(inode);
+ __entry->icount = icount_read_once(inode);
__entry->owner = fl->c.flc_owner;
__entry->flags = fl->c.flc_flags;
__entry->type = fl->c.flc_type;
diff --git a/include/trace/events/fsnotify.h b/include/trace/events/fsnotify.h
new file mode 100644
index 0000000..341bbd5
--- /dev/null
+++ b/include/trace/events/fsnotify.h
@@ -0,0 +1,51 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#undef TRACE_SYSTEM
+#define TRACE_SYSTEM fsnotify
+
+#if !defined(_TRACE_FSNOTIFY_H) || defined(TRACE_HEADER_MULTI_READ)
+#define _TRACE_FSNOTIFY_H
+
+#include <linux/tracepoint.h>
+
+#include <trace/misc/fsnotify.h>
+
+TRACE_EVENT(fsnotify,
+ TP_PROTO(__u32 mask, const void *data, int data_type,
+ struct inode *dir, const struct qstr *file_name,
+ struct inode *inode, u32 cookie),
+
+ TP_ARGS(mask, data, data_type, dir, file_name, inode, cookie),
+
+ TP_STRUCT__entry(
+ __field(__u32, mask)
+ __field(unsigned long, dir_ino)
+ __field(unsigned long, ino)
+ __field(dev_t, s_dev)
+ __field(int, data_type)
+ __field(u32, cookie)
+ __string(file_name, file_name ? (const char *)file_name->name : "")
+ ),
+
+ TP_fast_assign(
+ __entry->mask = mask;
+ __entry->dir_ino = dir ? dir->i_ino : 0;
+ __entry->ino = inode ? inode->i_ino : 0;
+ __entry->s_dev = dir ? dir->i_sb->s_dev :
+ inode ? inode->i_sb->s_dev : 0;
+ __entry->data_type = data_type;
+ __entry->cookie = cookie;
+ __assign_str(file_name);
+ ),
+
+ TP_printk("dev=%d:%d dir=%lu ino=%lu data_type=%d cookie=0x%x mask=0x%x %s name=%s",
+ MAJOR(__entry->s_dev), MINOR(__entry->s_dev),
+ __entry->dir_ino, __entry->ino,
+ __entry->data_type, __entry->cookie,
+ __entry->mask, show_fsnotify_mask(__entry->mask),
+ __get_str(file_name))
+);
+
+#endif /* _TRACE_FSNOTIFY_H */
+
+/* This part must be outside protection */
+#include <trace/define_trace.h>
diff --git a/include/trace/events/writeback.h b/include/trace/events/writeback.h
index bdac0d6..13ee076 100644
--- a/include/trace/events/writeback.h
+++ b/include/trace/events/writeback.h
@@ -44,7 +44,8 @@
EM( WB_REASON_PERIODIC, "periodic") \
EM( WB_REASON_FS_FREE_SPACE, "fs_free_space") \
EM( WB_REASON_FORKER_THREAD, "forker_thread") \
- EMe(WB_REASON_FOREIGN_FLUSH, "foreign_flush")
+ EM( WB_REASON_FOREIGN_FLUSH, "foreign_flush") \
+ EMe(WB_REASON_DONTCACHE, "dontcache")
WB_WORK_REASON
diff --git a/include/trace/events/xen.h b/include/trace/events/xen.h
index 0577f0c..ad38496 100644
--- a/include/trace/events/xen.h
+++ b/include/trace/events/xen.h
@@ -12,24 +12,29 @@
struct multicall_entry;
/* Multicalls */
-DECLARE_EVENT_CLASS(xen_mc__batch,
- TP_PROTO(enum xen_lazy_mode mode),
- TP_ARGS(mode),
+TRACE_EVENT(xen_mc_batch,
+ TP_PROTO(unsigned long flags),
+ TP_ARGS(flags),
TP_STRUCT__entry(
- __field(enum xen_lazy_mode, mode)
+ __field(unsigned long, flags)
),
- TP_fast_assign(__entry->mode = mode),
- TP_printk("start batch LAZY_%s",
- (__entry->mode == XEN_LAZY_MMU) ? "MMU" :
- (__entry->mode == XEN_LAZY_CPU) ? "CPU" : "NONE")
+ TP_fast_assign(__entry->flags = flags),
+ TP_printk("start batch lazy flags %lx", __entry->flags)
);
-#define DEFINE_XEN_MC_BATCH(name) \
- DEFINE_EVENT(xen_mc__batch, name, \
- TP_PROTO(enum xen_lazy_mode mode), \
- TP_ARGS(mode))
-DEFINE_XEN_MC_BATCH(xen_mc_batch);
-DEFINE_XEN_MC_BATCH(xen_mc_issue);
+TRACE_EVENT(xen_mc_issue,
+ TP_PROTO(bool flush, unsigned long flags),
+ TP_ARGS(flush, flags),
+ TP_STRUCT__entry(
+ __field(unsigned long, flags)
+ __field(bool, flush)
+ ),
+ TP_fast_assign(__entry->flush = flush;
+ __entry->flags = flags;
+ ),
+ TP_printk("flush: %s, flags %lx",
+ __entry->flush ? "yes" : "no", __entry->flags)
+ );
TRACE_DEFINE_SIZEOF(ulong);
@@ -129,9 +134,10 @@ TRACE_EVENT(xen_mc_extend_args,
__entry->res == XEN_MC_XE_NO_SPACE ? "NO_SPACE" : "???")
);
-TRACE_DEFINE_SIZEOF(pteval_t);
/* mmu */
-DECLARE_EVENT_CLASS(xen_mmu__set_pte,
+TRACE_DEFINE_SIZEOF(pteval_t);
+
+TRACE_EVENT(xen_mmu_set_pte,
TP_PROTO(pte_t *ptep, pte_t pteval),
TP_ARGS(ptep, pteval),
TP_STRUCT__entry(
@@ -146,13 +152,6 @@ DECLARE_EVENT_CLASS(xen_mmu__set_pte,
(int)sizeof(pteval_t) * 2, (unsigned long long)__entry->pteval)
);
-#define DEFINE_XEN_MMU_SET_PTE(name) \
- DEFINE_EVENT(xen_mmu__set_pte, name, \
- TP_PROTO(pte_t *ptep, pte_t pteval), \
- TP_ARGS(ptep, pteval))
-
-DEFINE_XEN_MMU_SET_PTE(xen_mmu_set_pte);
-
TRACE_DEFINE_SIZEOF(pmdval_t);
TRACE_EVENT(xen_mmu_set_pmd,
@@ -170,37 +169,6 @@ TRACE_EVENT(xen_mmu_set_pmd,
(int)sizeof(pmdval_t) * 2, (unsigned long long)__entry->pmdval)
);
-#ifdef CONFIG_X86_PAE
-DEFINE_XEN_MMU_SET_PTE(xen_mmu_set_pte_atomic);
-
-TRACE_EVENT(xen_mmu_pte_clear,
- TP_PROTO(struct mm_struct *mm, unsigned long addr, pte_t *ptep),
- TP_ARGS(mm, addr, ptep),
- TP_STRUCT__entry(
- __field(struct mm_struct *, mm)
- __field(unsigned long, addr)
- __field(pte_t *, ptep)
- ),
- TP_fast_assign(__entry->mm = mm;
- __entry->addr = addr;
- __entry->ptep = ptep),
- TP_printk("mm %p addr %lx ptep %p",
- __entry->mm, __entry->addr, __entry->ptep)
- );
-
-TRACE_EVENT(xen_mmu_pmd_clear,
- TP_PROTO(pmd_t *pmdp),
- TP_ARGS(pmdp),
- TP_STRUCT__entry(
- __field(pmd_t *, pmdp)
- ),
- TP_fast_assign(__entry->pmdp = pmdp),
- TP_printk("pmdp %p", __entry->pmdp)
- );
-#endif
-
-#if CONFIG_PGTABLE_LEVELS >= 4
-
TRACE_DEFINE_SIZEOF(pudval_t);
TRACE_EVENT(xen_mmu_set_pud,
@@ -236,24 +204,6 @@ TRACE_EVENT(xen_mmu_set_p4d,
(int)sizeof(p4dval_t) * 2, (unsigned long long)pgd_val(native_make_pgd(__entry->p4dval)),
(int)sizeof(p4dval_t) * 2, (unsigned long long)__entry->p4dval)
);
-#else
-
-TRACE_EVENT(xen_mmu_set_pud,
- TP_PROTO(pud_t *pudp, pud_t pudval),
- TP_ARGS(pudp, pudval),
- TP_STRUCT__entry(
- __field(pud_t *, pudp)
- __field(pudval_t, pudval)
- ),
- TP_fast_assign(__entry->pudp = pudp;
- __entry->pudval = native_pud_val(pudval)),
- TP_printk("pudp %p pudval %0*llx (raw %0*llx)",
- __entry->pudp,
- (int)sizeof(pudval_t) * 2, (unsigned long long)pgd_val(native_make_pgd(__entry->pudval)),
- (int)sizeof(pudval_t) * 2, (unsigned long long)__entry->pudval)
- );
-
-#endif
DECLARE_EVENT_CLASS(xen_mmu_ptep_modify_prot,
TP_PROTO(struct mm_struct *mm, unsigned long addr,
@@ -452,7 +402,6 @@ TRACE_EVENT(xen_cpu_set_ldt,
__entry->addr, __entry->entries)
);
-
#endif /* _TRACE_XEN_H */
/* This part must be outside protection */
diff --git a/include/trace/misc/fsnotify.h b/include/trace/misc/fsnotify.h
new file mode 100644
index 0000000..a201e1b
--- /dev/null
+++ b/include/trace/misc/fsnotify.h
@@ -0,0 +1,35 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Display helpers for fsnotify events
+ */
+
+#include <linux/fsnotify_backend.h>
+
+#define show_fsnotify_mask(mask) \
+ __print_flags(mask, "|", \
+ { FS_ACCESS, "ACCESS" }, \
+ { FS_MODIFY, "MODIFY" }, \
+ { FS_ATTRIB, "ATTRIB" }, \
+ { FS_CLOSE_WRITE, "CLOSE_WRITE" }, \
+ { FS_CLOSE_NOWRITE, "CLOSE_NOWRITE" }, \
+ { FS_OPEN, "OPEN" }, \
+ { FS_MOVED_FROM, "MOVED_FROM" }, \
+ { FS_MOVED_TO, "MOVED_TO" }, \
+ { FS_CREATE, "CREATE" }, \
+ { FS_DELETE, "DELETE" }, \
+ { FS_DELETE_SELF, "DELETE_SELF" }, \
+ { FS_MOVE_SELF, "MOVE_SELF" }, \
+ { FS_OPEN_EXEC, "OPEN_EXEC" }, \
+ { FS_UNMOUNT, "UNMOUNT" }, \
+ { FS_Q_OVERFLOW, "Q_OVERFLOW" }, \
+ { FS_ERROR, "ERROR" }, \
+ { FS_OPEN_PERM, "OPEN_PERM" }, \
+ { FS_ACCESS_PERM, "ACCESS_PERM" }, \
+ { FS_OPEN_EXEC_PERM, "OPEN_EXEC_PERM" }, \
+ { FS_PRE_ACCESS, "PRE_ACCESS" }, \
+ { FS_MNT_ATTACH, "MNT_ATTACH" }, \
+ { FS_MNT_DETACH, "MNT_DETACH" }, \
+ { FS_EVENT_ON_CHILD, "EVENT_ON_CHILD" }, \
+ { FS_RENAME, "RENAME" }, \
+ { FS_DN_MULTISHOT, "DN_MULTISHOT" }, \
+ { FS_ISDIR, "ISDIR" })
diff --git a/include/uapi/asm-generic/errno.h b/include/uapi/asm-generic/errno.h
index 92e7ae4..bd78e69 100644
--- a/include/uapi/asm-generic/errno.h
+++ b/include/uapi/asm-generic/errno.h
@@ -122,4 +122,6 @@
#define EHWPOISON 133 /* Memory page has hardware error */
+#define EFTYPE 134 /* Wrong file type for the intended operation */
+
#endif
diff --git a/include/uapi/asm-generic/fcntl.h b/include/uapi/asm-generic/fcntl.h
index 61347528..883cfd7 100644
--- a/include/uapi/asm-generic/fcntl.h
+++ b/include/uapi/asm-generic/fcntl.h
@@ -15,51 +15,55 @@
* When introducing new O_* bits, please check its uniqueness in fcntl_init().
*/
-#define O_ACCMODE 00000003
-#define O_RDONLY 00000000
-#define O_WRONLY 00000001
-#define O_RDWR 00000002
+#define O_ACCMODE 3
+#define O_RDONLY 0
+#define O_WRONLY (1 << 0)
+#define O_RDWR (1 << 1)
+/* (1 << 2) must not be used -- it collides with flags on alpha, sparc */
+/* (1 << 3) must not be used -- it collides with flags on alpha, mips, parisc, sparc */
+/* (1 << 4) must not be used -- it collides with flags on mips */
+/* (1 << 5) is free */
#ifndef O_CREAT
-#define O_CREAT 00000100 /* not fcntl */
+#define O_CREAT (1 << 6) /* not fcntl */
#endif
#ifndef O_EXCL
-#define O_EXCL 00000200 /* not fcntl */
+#define O_EXCL (1 << 7) /* not fcntl */
#endif
#ifndef O_NOCTTY
-#define O_NOCTTY 00000400 /* not fcntl */
+#define O_NOCTTY (1 << 8) /* not fcntl */
#endif
#ifndef O_TRUNC
-#define O_TRUNC 00001000 /* not fcntl */
+#define O_TRUNC (1 << 9) /* not fcntl */
#endif
#ifndef O_APPEND
-#define O_APPEND 00002000
+#define O_APPEND (1 << 10)
#endif
#ifndef O_NONBLOCK
-#define O_NONBLOCK 00004000
+#define O_NONBLOCK (1 << 11)
#endif
#ifndef O_DSYNC
-#define O_DSYNC 00010000 /* used to be O_SYNC, see below */
+#define O_DSYNC (1 << 12) /* used to be O_SYNC, see below */
#endif
#ifndef FASYNC
-#define FASYNC 00020000 /* fcntl, for BSD compatibility */
+#define FASYNC (1 << 13) /* fcntl, for BSD compatibility */
#endif
#ifndef O_DIRECT
-#define O_DIRECT 00040000 /* direct disk access hint */
+#define O_DIRECT (1 << 14) /* direct disk access hint */
#endif
#ifndef O_LARGEFILE
-#define O_LARGEFILE 00100000
+#define O_LARGEFILE (1 << 15)
#endif
#ifndef O_DIRECTORY
-#define O_DIRECTORY 00200000 /* must be a directory */
+#define O_DIRECTORY (1 << 16) /* must be a directory */
#endif
#ifndef O_NOFOLLOW
-#define O_NOFOLLOW 00400000 /* don't follow links */
+#define O_NOFOLLOW (1 << 17) /* don't follow links */
#endif
#ifndef O_NOATIME
-#define O_NOATIME 01000000
+#define O_NOATIME (1 << 18)
#endif
#ifndef O_CLOEXEC
-#define O_CLOEXEC 02000000 /* set close_on_exec */
+#define O_CLOEXEC (1 << 19) /* set close_on_exec */
#endif
/*
@@ -76,16 +80,20 @@
* Note: __O_SYNC must never be used directly.
*/
#ifndef O_SYNC
-#define __O_SYNC 04000000
+#define __O_SYNC (1 << 20)
#define O_SYNC (__O_SYNC|O_DSYNC)
#endif
#ifndef O_PATH
-#define O_PATH 010000000
+#define O_PATH (1 << 21)
#endif
#ifndef __O_TMPFILE
-#define __O_TMPFILE 020000000
+#define __O_TMPFILE (1 << 22)
+#endif
+
+#ifndef O_EMPTYPATH
+#define O_EMPTYPATH (1 << 26) /* allow empty path */
#endif
/* a horrid kludge trying to make sure that this will fail on old kernels */
@@ -95,6 +103,10 @@
#define O_NDELAY O_NONBLOCK
#endif
+/* (1 << 23) must not be used -- it collides with flags on alpha, parisc, sparc */
+/* (1 << 24) must not be used -- it collides with flags on alpha, sparc */
+/* (1 << 25) must not be used -- it collides with flags on sparc */
+
#define F_DUPFD 0 /* dup */
#define F_GETFD 1 /* get close_on_exec */
#define F_SETFD 2 /* set/clear close_on_exec */
diff --git a/include/uapi/linux/fs.h b/include/uapi/linux/fs.h
index 13f7120..bd87262 100644
--- a/include/uapi/linux/fs.h
+++ b/include/uapi/linux/fs.h
@@ -254,6 +254,13 @@ struct file_attr {
#define FS_XFLAG_DAX 0x00008000 /* use DAX for IO */
#define FS_XFLAG_COWEXTSIZE 0x00010000 /* CoW extent size allocator hint */
#define FS_XFLAG_VERITY 0x00020000 /* fs-verity enabled */
+/*
+ * Case handling flags (read-only, cannot be set via ioctl).
+ * Default (neither set) indicates POSIX semantics: case-sensitive
+ * lookups and case-preserving storage.
+ */
+#define FS_XFLAG_CASEFOLD 0x00040000 /* case-insensitive lookups */
+#define FS_XFLAG_CASENONPRESERVING 0x00080000 /* case not preserved */
#define FS_XFLAG_HASATTR 0x80000000 /* no DIFLAG for this */
/* the read-only stuff doesn't really belong here, but any other place is
@@ -388,7 +395,16 @@ struct file_attr {
#define FS_DAX_FL 0x02000000 /* Inode is DAX */
#define FS_INLINE_DATA_FL 0x10000000 /* Reserved for ext4 */
#define FS_PROJINHERIT_FL 0x20000000 /* Create with parents projid */
-#define FS_CASEFOLD_FL 0x40000000 /* Folder is case insensitive */
+/*
+ * FS_CASEFOLD_FL indicates case-insensitive name lookup. The
+ * bit is most often reported on directories, where it controls
+ * lookups of entries within. Filesystems that derive
+ * case-insensitivity from mount or volume state may also report
+ * it on non-directory inodes; userspace must not assume the bit
+ * is directory-only. FS_XFLAG_CASEFOLD reports the same
+ * information read-only via FS_IOC_FSGETXATTR.
+ */
+#define FS_CASEFOLD_FL 0x40000000
#define FS_RESERVED_FL 0x80000000 /* reserved for ext2 lib */
#define FS_FL_USER_VISIBLE 0x0003DFFF /* User visible flags */
diff --git a/include/uapi/linux/openat2.h b/include/uapi/linux/openat2.h
index a5feb76..575c2c5 100644
--- a/include/uapi/linux/openat2.h
+++ b/include/uapi/linux/openat2.h
@@ -22,6 +22,13 @@ struct open_how {
__u64 resolve;
};
+/*
+ * how->flags bits exclusive to openat2(2). These live in the upper 32 bits
+ * of @flags so that they cannot be expressed by open(2) / openat(2), whose
+ * @flags argument is a C int.
+ */
+#define OPENAT2_REGULAR ((__u64)1 << 32) /* Only open regular files. */
+
/* how->resolve flags for openat2(2). */
#define RESOLVE_NO_XDEV 0x01 /* Block mount-point crossings
(includes bind-mounts). */
diff --git a/include/xen/interface/io/xs_wire.h b/include/xen/interface/io/xs_wire.h
index b623654..29d0394 100644
--- a/include/xen/interface/io/xs_wire.h
+++ b/include/xen/interface/io/xs_wire.h
@@ -51,7 +51,7 @@ struct xsd_errors
const char *errstring;
};
#define XSD_ERROR(x) { x, #x }
-static struct xsd_errors xsd_errors[] __attribute__((unused)) = {
+static const struct xsd_errors xsd_errors[] __maybe_unused = {
XSD_ERROR(EINVAL),
XSD_ERROR(EACCES),
XSD_ERROR(EEXIST),
diff --git a/init/Kconfig b/init/Kconfig
index 2937c4d..147da63 100644
--- a/init/Kconfig
+++ b/init/Kconfig
@@ -118,10 +118,7 @@
config CC_HAS_ASM_GOTO_OUTPUT
def_bool y
depends on !GCC_ASM_GOTO_OUTPUT_BROKEN
- # Detect basic support
depends on $(success,echo 'int foo(int x) { asm goto ("": "=r"(x) ::: bar); return x; bar: return 0; }' | $(CC) -x c - -c -o /dev/null)
- # Detect clang (< v17) scoped label issues
- depends on $(success,echo 'void b(void **);void* c(void);int f(void){{asm goto(""::::l0);return 0;l0:return 1;}void *x __attribute__((cleanup(b)))=c();{asm goto(""::::l1);return 2;l1:return 3;}}' | $(CC) -x c - -c -o /dev/null)
config CC_HAS_ASM_GOTO_TIED_OUTPUT
depends on CC_HAS_ASM_GOTO_OUTPUT
@@ -2198,7 +2195,8 @@
depends on !DEBUG_INFO_BTF || (PAHOLE_HAS_LANG_EXCLUDE && !LTO)
depends on !CFI || HAVE_CFI_ICALL_NORMALIZE_INTEGERS_RUSTC
select CFI_ICALL_NORMALIZE_INTEGERS if CFI
- depends on !KASAN_SW_TAGS
+ depends on !KASAN || CC_IS_CLANG
+ depends on !KASAN_SW_TAGS || RUSTC_VERSION >= 109600
help
Enables Rust support in the kernel.
@@ -2212,6 +2210,8 @@
If unsure, say N.
+source "rust/kernel/Kconfig.test"
+
config RUSTC_VERSION_TEXT
string
depends on RUST
diff --git a/init/do_mounts.c b/init/do_mounts.c
index 55ed3ac..95e0b3a 100644
--- a/init/do_mounts.c
+++ b/init/do_mounts.c
@@ -143,16 +143,14 @@ static int __init do_mount_root(const char *name, const char *fs,
const int flags, const void *data)
{
struct super_block *s;
- struct page *p = NULL;
char *data_page = NULL;
int ret;
if (data) {
/* init_mount() requires a full page as fifth argument */
- p = alloc_page(GFP_KERNEL);
- if (!p)
+ data_page = kmalloc(PAGE_SIZE, GFP_KERNEL);
+ if (!data_page)
return -ENOMEM;
- data_page = page_address(p);
strscpy_pad(data_page, data, PAGE_SIZE);
}
@@ -170,19 +168,20 @@ static int __init do_mount_root(const char *name, const char *fs,
MAJOR(ROOT_DEV), MINOR(ROOT_DEV));
out:
- if (p)
- put_page(p);
+ kfree(data_page);
return ret;
}
void __init mount_root_generic(char *name, char *pretty_name, int flags)
{
- struct page *page = alloc_page(GFP_KERNEL);
- char *fs_names = page_address(page);
+ char *fs_names = kmalloc(PAGE_SIZE, GFP_KERNEL);
char *p;
char b[BDEVNAME_SIZE];
int num_fs, i;
+ if (!fs_names)
+ panic("VFS: Unable to mount root fs: not enough memory");
+
scnprintf(b, BDEVNAME_SIZE, "unknown-block(%u,%u)",
MAJOR(ROOT_DEV), MINOR(ROOT_DEV));
if (root_fs_names)
@@ -242,7 +241,7 @@ void __init mount_root_generic(char *name, char *pretty_name, int flags)
printk("\n");
panic("VFS: Unable to mount root fs on \"%s\" or %s", pretty_name, b);
out:
- put_page(page);
+ kfree(fs_names);
}
#ifdef CONFIG_ROOT_NFS
@@ -343,7 +342,7 @@ static int __init mount_nodev_root(char *root_device_name)
int err = -EINVAL;
int num_fs, i;
- fs_names = (void *)__get_free_page(GFP_KERNEL);
+ fs_names = kmalloc(PAGE_SIZE, GFP_KERNEL);
if (!fs_names)
return -EINVAL;
num_fs = split_fs_names(fs_names, PAGE_SIZE);
@@ -360,7 +359,7 @@ static int __init mount_nodev_root(char *root_device_name)
break;
}
- free_page((unsigned long)fs_names);
+ kfree(fs_names);
return err;
}
diff --git a/init/init_task.c b/init/init_task.c
index b5f48eb..8cad78d 100644
--- a/init/init_task.c
+++ b/init/init_task.c
@@ -7,6 +7,8 @@
#include <linux/sched/rt.h>
#include <linux/sched/task.h>
#include <linux/sched/ext.h>
+#include <linux/sched/exec_state.h>
+#include <linux/user_namespace.h>
#include <linux/init.h>
#include <linux/fs.h>
#include <linux/mm.h>
@@ -56,6 +58,13 @@ static struct sighand_struct init_sighand = {
.signalfd_wqh = __WAIT_QUEUE_HEAD_INITIALIZER(init_sighand.signalfd_wqh),
};
+/* init to 2 - one for init_task, one to ensure it is never freed */
+struct task_exec_state init_task_exec_state = {
+ .count = REFCOUNT_INIT(2),
+ .dumpable = TASK_DUMPABLE_OWNER,
+ .user_ns = &init_user_ns,
+};
+
#ifdef CONFIG_SHADOW_CALL_STACK
unsigned long init_shadow_call_stack[SCS_SIZE / sizeof(long)] = {
[(SCS_SIZE / sizeof(long)) - 1] = SCS_END_MAGIC
@@ -113,6 +122,7 @@ struct task_struct init_task __aligned(L1_CACHE_BYTES) = {
.nr_cpus_allowed= NR_CPUS,
.mm = NULL,
.active_mm = &init_mm,
+ .exec_state = &init_task_exec_state,
.restart_block = {
.fn = do_no_restart_syscall,
},
diff --git a/init/initramfs.c b/init/initramfs.c
index 58db15f..20a18fc 100644
--- a/init/initramfs.c
+++ b/init/initramfs.c
@@ -1,25 +1,28 @@
// SPDX-License-Identifier: GPL-2.0
-#include <linux/init.h>
#include <linux/async.h>
-#include <linux/export.h>
-#include <linux/fs.h>
-#include <linux/slab.h>
-#include <linux/types.h>
-#include <linux/fcntl.h>
#include <linux/delay.h>
-#include <linux/string.h>
#include <linux/dirent.h>
-#include <linux/syscalls.h>
-#include <linux/utime.h>
+#include <linux/export.h>
+#include <linux/fcntl.h>
#include <linux/file.h>
+#include <linux/fs.h>
+#include <linux/hex.h>
+#include <linux/init.h>
+#include <linux/init_syscalls.h>
#include <linux/kstrtox.h>
#include <linux/memblock.h>
#include <linux/mm.h>
#include <linux/namei.h>
-#include <linux/init_syscalls.h>
-#include <linux/umh.h>
-#include <linux/security.h>
#include <linux/overflow.h>
+#include <linux/security.h>
+#include <linux/slab.h>
+#include <linux/string.h>
+#include <linux/syscalls.h>
+#include <linux/types.h>
+#include <linux/umh.h>
+#include <linux/utime.h>
+
+#include <asm/byteorder.h>
#include "do_mounts.h"
#include "initramfs_internal.h"
@@ -190,26 +193,30 @@ static __initdata gid_t gid;
static __initdata unsigned rdev;
static __initdata u32 hdr_csum;
-static void __init parse_header(char *s)
+static int __init parse_header(char *s)
{
- unsigned long parsed[13];
- int i;
+ __be32 header[13];
+ int ret;
- for (i = 0, s += 6; i < 13; i++, s += 8)
- parsed[i] = simple_strntoul(s, NULL, 16, 8);
+ ret = hex2bin((u8 *)header, s + 6, sizeof(header));
+ if (ret) {
+ error("damaged header");
+ return ret;
+ }
- ino = parsed[0];
- mode = parsed[1];
- uid = parsed[2];
- gid = parsed[3];
- nlink = parsed[4];
- mtime = parsed[5]; /* breaks in y2106 */
- body_len = parsed[6];
- major = parsed[7];
- minor = parsed[8];
- rdev = new_encode_dev(MKDEV(parsed[9], parsed[10]));
- name_len = parsed[11];
- hdr_csum = parsed[12];
+ ino = be32_to_cpu(header[0]);
+ mode = be32_to_cpu(header[1]);
+ uid = be32_to_cpu(header[2]);
+ gid = be32_to_cpu(header[3]);
+ nlink = be32_to_cpu(header[4]);
+ mtime = be32_to_cpu(header[5]); /* breaks in y2106 */
+ body_len = be32_to_cpu(header[6]);
+ major = be32_to_cpu(header[7]);
+ minor = be32_to_cpu(header[8]);
+ rdev = new_encode_dev(MKDEV(be32_to_cpu(header[9]), be32_to_cpu(header[10])));
+ name_len = be32_to_cpu(header[11]);
+ hdr_csum = be32_to_cpu(header[12]);
+ return 0;
}
/* Finite-state machine */
@@ -289,7 +296,8 @@ static int __init do_header(void)
error("no cpio magic");
return 1;
}
- parse_header(collected);
+ if (parse_header(collected))
+ return 1;
next_header = this_header + N_ALIGN(name_len) + body_len;
next_header = (next_header + 3) & ~3;
state = SkipIt;
diff --git a/init/initramfs_test.c b/init/initramfs_test.c
index 2ce38d9..bc55306 100644
--- a/init/initramfs_test.c
+++ b/init/initramfs_test.c
@@ -3,7 +3,9 @@
#include <linux/fcntl.h>
#include <linux/file.h>
#include <linux/fs.h>
+#include <linux/init.h>
#include <linux/init_syscalls.h>
+#include <linux/initrd.h>
#include <linux/stringify.h>
#include <linux/timekeeping.h>
#include "initramfs_internal.h"
@@ -27,7 +29,18 @@ struct initramfs_test_cpio {
char *data;
};
-static size_t fill_cpio(struct initramfs_test_cpio *cs, size_t csz, char *out)
+/* regular newc header format */
+#define CPIO_HDR_FMT "%s%08x%08x%08x%08x%08x%08x%08x%08x%08x%08x%08x%08x%08x%s"
+/*
+ * Bogus newc header with "0x" prefixes on the uid, gid, and namesize values.
+ * parse_header()/simple_str[n]toul() accepted this, contrary to the initramfs
+ * specification. hex2bin() now fails.
+ */
+#define CPIO_HDR_OX_INJECT \
+ "%s%08x%08x0x%06x0X%06x%08x%08x%08x%08x%08x%08x%08x0x%06x%08x%s"
+
+static size_t fill_cpio(struct initramfs_test_cpio *cs, size_t csz,
+ bool inject_ox, char *out)
{
int i;
size_t off = 0;
@@ -38,9 +51,8 @@ static size_t fill_cpio(struct initramfs_test_cpio *cs, size_t csz, char *out)
size_t thislen;
/* +1 to account for nulterm */
- thislen = sprintf(pos, "%s"
- "%08x%08x%08x%08x%08x%08x%08x%08x%08x%08x%08x%08x%08x"
- "%s",
+ thislen = sprintf(pos,
+ inject_ox ? CPIO_HDR_OX_INJECT : CPIO_HDR_FMT,
c->magic, c->ino, c->mode, c->uid, c->gid, c->nlink,
c->mtime, c->filesize, c->devmajor, c->devminor,
c->rdevmajor, c->rdevminor, c->namesize, c->csum,
@@ -102,7 +114,7 @@ static void __init initramfs_test_extract(struct kunit *test)
/* +3 to cater for any 4-byte end-alignment */
cpio_srcbuf = kzalloc(ARRAY_SIZE(c) * (CPIO_HDRLEN + PATH_MAX + 3),
GFP_KERNEL);
- len = fill_cpio(c, ARRAY_SIZE(c), cpio_srcbuf);
+ len = fill_cpio(c, ARRAY_SIZE(c), false, cpio_srcbuf);
ktime_get_real_ts64(&ts_before);
err = unpack_to_rootfs(cpio_srcbuf, len);
@@ -177,7 +189,7 @@ static void __init initramfs_test_fname_overrun(struct kunit *test)
/* limit overrun to avoid crashes / filp_open() ENAMETOOLONG */
cpio_srcbuf[CPIO_HDRLEN + strlen(c[0].fname) + 20] = '\0';
- len = fill_cpio(c, ARRAY_SIZE(c), cpio_srcbuf);
+ len = fill_cpio(c, ARRAY_SIZE(c), false, cpio_srcbuf);
/* overwrite trailing fname terminator and padding */
suffix_off = len - 1;
while (cpio_srcbuf[suffix_off] == '\0') {
@@ -219,7 +231,7 @@ static void __init initramfs_test_data(struct kunit *test)
cpio_srcbuf = kmalloc(CPIO_HDRLEN + c[0].namesize + c[0].filesize + 6,
GFP_KERNEL);
- len = fill_cpio(c, ARRAY_SIZE(c), cpio_srcbuf);
+ len = fill_cpio(c, ARRAY_SIZE(c), false, cpio_srcbuf);
err = unpack_to_rootfs(cpio_srcbuf, len);
KUNIT_EXPECT_NULL(test, err);
@@ -274,7 +286,7 @@ static void __init initramfs_test_csum(struct kunit *test)
cpio_srcbuf = kmalloc(8192, GFP_KERNEL);
- len = fill_cpio(c, ARRAY_SIZE(c), cpio_srcbuf);
+ len = fill_cpio(c, ARRAY_SIZE(c), false, cpio_srcbuf);
err = unpack_to_rootfs(cpio_srcbuf, len);
KUNIT_EXPECT_NULL(test, err);
@@ -284,7 +296,7 @@ static void __init initramfs_test_csum(struct kunit *test)
/* mess up the csum and confirm that unpack fails */
c[0].csum--;
- len = fill_cpio(c, ARRAY_SIZE(c), cpio_srcbuf);
+ len = fill_cpio(c, ARRAY_SIZE(c), false, cpio_srcbuf);
err = unpack_to_rootfs(cpio_srcbuf, len);
KUNIT_EXPECT_NOT_NULL(test, err);
@@ -306,7 +318,7 @@ static void __init initramfs_test_hardlink(struct kunit *test)
{
char *err, *cpio_srcbuf;
size_t len;
- struct kstat st0, st1;
+ struct kstat st0 = {}, st1 = {};
struct initramfs_test_cpio c[] = { {
.magic = "070701",
.ino = 1,
@@ -330,7 +342,7 @@ static void __init initramfs_test_hardlink(struct kunit *test)
cpio_srcbuf = kmalloc(8192, GFP_KERNEL);
- len = fill_cpio(c, ARRAY_SIZE(c), cpio_srcbuf);
+ len = fill_cpio(c, ARRAY_SIZE(c), false, cpio_srcbuf);
err = unpack_to_rootfs(cpio_srcbuf, len);
KUNIT_EXPECT_NULL(test, err);
@@ -371,7 +383,7 @@ static void __init initramfs_test_many(struct kunit *test)
};
c.namesize = 1 + sprintf(thispath, "initramfs_test_many-%d", i);
- p += fill_cpio(&c, 1, p);
+ p += fill_cpio(&c, 1, false, p);
}
len = p - cpio_srcbuf;
@@ -425,7 +437,7 @@ static void __init initramfs_test_fname_pad(struct kunit *test)
} };
memcpy(tbufs->padded_fname, "padded_fname", sizeof("padded_fname"));
- len = fill_cpio(c, ARRAY_SIZE(c), tbufs->cpio_srcbuf);
+ len = fill_cpio(c, ARRAY_SIZE(c), false, tbufs->cpio_srcbuf);
err = unpack_to_rootfs(tbufs->cpio_srcbuf, len);
KUNIT_EXPECT_NULL(test, err);
@@ -451,7 +463,7 @@ static void __init initramfs_test_fname_path_max(struct kunit *test)
{
char *err;
size_t len;
- struct kstat st0, st1;
+ struct kstat st0 = {}, st1 = {};
char fdata[] = "this file data will not be unpacked";
struct test_fname_path_max {
char fname_oversize[PATH_MAX + 1];
@@ -481,7 +493,7 @@ static void __init initramfs_test_fname_path_max(struct kunit *test)
memcpy(tbufs->fname_oversize, "fname_oversize",
sizeof("fname_oversize") - 1);
memcpy(tbufs->fname_ok, "fname_ok", sizeof("fname_ok") - 1);
- len = fill_cpio(c, ARRAY_SIZE(c), tbufs->cpio_src);
+ len = fill_cpio(c, ARRAY_SIZE(c), false, tbufs->cpio_src);
/* unpack skips over fname_oversize instead of returning an error */
err = unpack_to_rootfs(tbufs->cpio_src, len);
@@ -494,6 +506,45 @@ static void __init initramfs_test_fname_path_max(struct kunit *test)
kfree(tbufs);
}
+static void __init initramfs_test_hdr_hex(struct kunit *test)
+{
+ char *err;
+ size_t len;
+ char fdata[] = "this file data will not be unpacked";
+ struct initramfs_test_bufs {
+ char cpio_src[(CPIO_HDRLEN + PATH_MAX + 3 + sizeof(fdata)) * 2];
+ } *tbufs = kzalloc(sizeof(struct initramfs_test_bufs), GFP_KERNEL);
+ struct initramfs_test_cpio c[] = { {
+ .magic = "070701",
+ .ino = 1,
+ .mode = S_IFREG | 0777,
+ .uid = 0x123456,
+ .gid = 0x123457,
+ .nlink = 1,
+ .namesize = sizeof("initramfs_test_hdr_hex_0"),
+ .fname = "initramfs_test_hdr_hex_0",
+ .filesize = sizeof(fdata),
+ .data = fdata,
+ }, {
+ .magic = "070701",
+ .ino = 2,
+ .mode = S_IFDIR | 0777,
+ .uid = 0x000056,
+ .gid = 0x000057,
+ .nlink = 1,
+ .namesize = sizeof("initramfs_test_hdr_hex_1"),
+ .fname = "initramfs_test_hdr_hex_1",
+ } };
+
+ /* inject_ox=true to add "0x" cpio field prefixes */
+ len = fill_cpio(c, ARRAY_SIZE(c), true, tbufs->cpio_src);
+
+ err = unpack_to_rootfs(tbufs->cpio_src, len);
+ KUNIT_EXPECT_NOT_NULL(test, err);
+
+ kfree(tbufs);
+}
+
/*
* The kunit_case/_suite struct cannot be marked as __initdata as this will be
* used in debugfs to retrieve results after test has run.
@@ -507,11 +558,25 @@ static struct kunit_case __refdata initramfs_test_cases[] = {
KUNIT_CASE(initramfs_test_many),
KUNIT_CASE(initramfs_test_fname_pad),
KUNIT_CASE(initramfs_test_fname_path_max),
+ KUNIT_CASE(initramfs_test_hdr_hex),
{},
};
-static struct kunit_suite initramfs_test_suite = {
+static int __init initramfs_test_init(struct kunit_suite *suite)
+{
+ /*
+ * unpack_to_rootfs() uses module-static state (victim, byte_count,
+ * state, ...). The boot-time async do_populate_rootfs() may still be
+ * running, so wait for it to finish before we call unpack_to_rootfs()
+ * from the test thread, otherwise the two writers race and crash.
+ */
+ wait_for_initramfs();
+ return 0;
+}
+
+static struct kunit_suite __refdata initramfs_test_suite = {
.name = "initramfs",
+ .suite_init = initramfs_test_init,
.test_cases = initramfs_test_cases,
};
kunit_test_init_section_suites(&initramfs_test_suite);
diff --git a/io_uring/kbuf.c b/io_uring/kbuf.c
index 63061aa..926254b 100644
--- a/io_uring/kbuf.c
+++ b/io_uring/kbuf.c
@@ -305,7 +305,6 @@ static int io_ring_buffers_peek(struct io_kiocb *req, struct buf_sel_arg *arg,
arg->partial_map = 1;
if (iov != arg->iovs)
break;
- WRITE_ONCE(buf->len, len);
}
}
diff --git a/io_uring/wait.c b/io_uring/wait.c
index ec01e78..d005ea1 100644
--- a/io_uring/wait.c
+++ b/io_uring/wait.c
@@ -103,7 +103,7 @@ static enum hrtimer_restart io_cqring_min_timer_wakeup(struct hrtimer *timer)
}
/* any generated CQE posted past this time should wake us up */
- iowq->cq_tail = iowq->cq_min_tail;
+ iowq->cq_tail = iowq->cq_min_tail + 1;
hrtimer_update_function(&iowq->t, io_cqring_timer_wakeup);
hrtimer_set_expires(timer, iowq->timeout);
diff --git a/ipc/sem.c b/ipc/sem.c
index 6cdf862..5ec41de7 100644
--- a/ipc/sem.c
+++ b/ipc/sem.c
@@ -1981,7 +1981,7 @@ static struct sem_undo *find_alloc_undo(struct ipc_namespace *ns, int semid)
}
long __do_semtimedop(int semid, struct sembuf *sops,
- unsigned nsops, const struct timespec64 *timeout,
+ unsigned int nsops, const struct timespec64 *timeout,
struct ipc_namespace *ns)
{
int error = -EINVAL;
@@ -2220,7 +2220,7 @@ long __do_semtimedop(int semid, struct sembuf *sops,
}
static long do_semtimedop(int semid, struct sembuf __user *tsops,
- unsigned nsops, const struct timespec64 *timeout)
+ unsigned int nsops, const struct timespec64 *timeout)
{
struct sembuf fast_sops[SEMOPM_FAST];
struct sembuf *sops = fast_sops;
@@ -2294,7 +2294,7 @@ SYSCALL_DEFINE4(semtimedop_time32, int, semid, struct sembuf __user *, tsems,
#endif
SYSCALL_DEFINE3(semop, int, semid, struct sembuf __user *, tsops,
- unsigned, nsops)
+ unsigned int, nsops)
{
return do_semtimedop(semid, tsops, nsops, NULL);
}
diff --git a/kernel/Makefile b/kernel/Makefile
index 6785982..1e1a316 100644
--- a/kernel/Makefile
+++ b/kernel/Makefile
@@ -3,7 +3,7 @@
# Makefile for the linux kernel.
#
-obj-y = fork.o exec_domain.o panic.o \
+obj-y = fork.o exec_domain.o exec_state.o panic.o \
cpu.o exit.o softirq.o resource.o \
sysctl.o capability.o ptrace.o user.o \
signal.o sys.o umh.o workqueue.o pid.o task_work.o \
diff --git a/kernel/acct.c b/kernel/acct.c
index cbbf79d..c440d43 100644
--- a/kernel/acct.c
+++ b/kernel/acct.c
@@ -249,7 +249,7 @@ static int acct_on(const char __user *name)
return -EINVAL;
/* Exclude procfs and sysfs. */
- if (file_inode(file)->i_sb->s_iflags & SB_I_USERNS_VISIBLE)
+ if (file_inode(file)->i_sb->s_type->fs_flags & FS_USERNS_MOUNT_RESTRICTED)
return -EINVAL;
if (!(file->f_mode & FMODE_CAN_WRITE))
diff --git a/kernel/bpf/inode.c b/kernel/bpf/inode.c
index 25c06a0..c3f79b5 100644
--- a/kernel/bpf/inode.c
+++ b/kernel/bpf/inode.c
@@ -21,6 +21,9 @@
#include <linux/bpf.h>
#include <linux/bpf_trace.h>
#include <linux/kstrtox.h>
+#include <linux/xattr.h>
+#include <linux/security.h>
+
#include "preload/bpf_preload.h"
enum bpf_type {
@@ -30,6 +33,23 @@ enum bpf_type {
BPF_TYPE_LINK,
};
+struct bpf_fs_inode {
+ struct list_head xattrs;
+ struct simple_xattr_limits xlimits;
+ struct inode vfs_inode;
+};
+
+static inline struct bpf_fs_inode *BPF_FS_I(struct inode *inode)
+{
+ return container_of(inode, struct bpf_fs_inode, vfs_inode);
+}
+
+static struct kmem_cache *bpf_fs_inode_cachep __ro_after_init;
+
+static int bpf_fs_initxattrs(struct inode *inode,
+ const struct xattr *xattr_array, void *fs_info);
+static ssize_t bpf_fs_listxattr(struct dentry *dentry, char *buf, size_t size);
+
static void *bpf_any_get(void *raw, enum bpf_type type)
{
switch (type) {
@@ -94,10 +114,17 @@ static void *bpf_fd_probe_obj(u32 ufd, enum bpf_type *type)
}
static const struct inode_operations bpf_dir_iops;
+static const struct inode_operations bpf_symlink_iops;
-static const struct inode_operations bpf_prog_iops = { };
-static const struct inode_operations bpf_map_iops = { };
-static const struct inode_operations bpf_link_iops = { };
+static const struct inode_operations bpf_prog_iops = {
+ .listxattr = bpf_fs_listxattr,
+};
+static const struct inode_operations bpf_map_iops = {
+ .listxattr = bpf_fs_listxattr,
+};
+static const struct inode_operations bpf_link_iops = {
+ .listxattr = bpf_fs_listxattr,
+};
struct inode *bpf_get_inode(struct super_block *sb,
const struct inode *dir,
@@ -153,11 +180,19 @@ static struct dentry *bpf_mkdir(struct mnt_idmap *idmap, struct inode *dir,
struct dentry *dentry, umode_t mode)
{
struct inode *inode;
+ int ret;
inode = bpf_get_inode(dir->i_sb, dir, mode | S_IFDIR);
if (IS_ERR(inode))
return ERR_CAST(inode);
+ ret = security_inode_init_security(inode, dir, &dentry->d_name,
+ bpf_fs_initxattrs, NULL);
+ if (ret && ret != -EOPNOTSUPP) {
+ iput(inode);
+ return ERR_PTR(ret);
+ }
+
inode->i_op = &bpf_dir_iops;
inode->i_fop = &simple_dir_operations;
@@ -330,10 +365,20 @@ static int bpf_mkobj_ops(struct dentry *dentry, umode_t mode, void *raw,
const struct file_operations *fops)
{
struct inode *dir = dentry->d_parent->d_inode;
- struct inode *inode = bpf_get_inode(dir->i_sb, dir, mode);
+ struct inode *inode;
+ int ret;
+
+ inode = bpf_get_inode(dir->i_sb, dir, mode);
if (IS_ERR(inode))
return PTR_ERR(inode);
+ ret = security_inode_init_security(inode, dir, &dentry->d_name,
+ bpf_fs_initxattrs, NULL);
+ if (ret && ret != -EOPNOTSUPP) {
+ iput(inode);
+ return ret;
+ }
+
inode->i_op = iops;
inode->i_fop = fops;
inode->i_private = raw;
@@ -382,9 +427,11 @@ bpf_lookup(struct inode *dir, struct dentry *dentry, unsigned flags)
static int bpf_symlink(struct mnt_idmap *idmap, struct inode *dir,
struct dentry *dentry, const char *target)
{
- char *link = kstrdup(target, GFP_USER | __GFP_NOWARN);
struct inode *inode;
+ char *link;
+ int ret;
+ link = kstrdup(target, GFP_KERNEL_ACCOUNT | __GFP_NOWARN);
if (!link)
return -ENOMEM;
@@ -394,13 +441,25 @@ static int bpf_symlink(struct mnt_idmap *idmap, struct inode *dir,
return PTR_ERR(inode);
}
- inode->i_op = &simple_symlink_inode_operations;
+ inode->i_op = &bpf_symlink_iops;
inode->i_link = link;
+ ret = security_inode_init_security(inode, dir, &dentry->d_name,
+ bpf_fs_initxattrs, NULL);
+ if (ret && ret != -EOPNOTSUPP) {
+ iput(inode);
+ return ret;
+ }
+
bpf_dentry_finalize(dentry, inode, dir);
return 0;
}
+static const struct inode_operations bpf_symlink_iops = {
+ .get_link = simple_get_link,
+ .listxattr = bpf_fs_listxattr,
+};
+
static const struct inode_operations bpf_dir_iops = {
.lookup = bpf_lookup,
.mkdir = bpf_mkdir,
@@ -409,6 +468,7 @@ static const struct inode_operations bpf_dir_iops = {
.rename = simple_rename,
.link = simple_link,
.unlink = simple_unlink,
+ .listxattr = bpf_fs_listxattr,
};
/* pin iterator link into bpffs */
@@ -762,22 +822,147 @@ static int bpf_show_options(struct seq_file *m, struct dentry *root)
return 0;
}
+static struct inode *bpf_fs_alloc_inode(struct super_block *sb)
+{
+ struct bpf_fs_inode *bi;
+
+ bi = alloc_inode_sb(sb, bpf_fs_inode_cachep, GFP_KERNEL);
+ if (!bi)
+ return NULL;
+ INIT_LIST_HEAD_RCU(&bi->xattrs);
+ simple_xattr_limits_init(&bi->xlimits);
+ return &bi->vfs_inode;
+}
+
static void bpf_destroy_inode(struct inode *inode)
{
+ struct bpf_mount_opts *opts = inode->i_sb->s_fs_info;
+ struct bpf_fs_inode *bi = BPF_FS_I(inode);
enum bpf_type type;
- if (S_ISLNK(inode->i_mode))
- kfree(inode->i_link);
if (!bpf_inode_type(inode, &type))
bpf_any_put(inode->i_private, type);
- free_inode_nonrcu(inode);
+ simple_xattrs_free(&opts->xa_cache, &bi->xattrs, NULL);
+}
+
+static void bpf_free_inode(struct inode *inode)
+{
+ if (S_ISLNK(inode->i_mode))
+ kfree(inode->i_link);
+ kmem_cache_free(bpf_fs_inode_cachep, BPF_FS_I(inode));
+}
+
+static int bpf_fs_xattr_get(const struct xattr_handler *handler,
+ struct dentry *unused, struct inode *inode,
+ const char *name, void *value, size_t size)
+{
+ struct bpf_mount_opts *opts = inode->i_sb->s_fs_info;
+ struct bpf_fs_inode *bi = BPF_FS_I(inode);
+
+ name = xattr_full_name(handler, name);
+ return simple_xattr_get(&opts->xa_cache, &bi->xattrs, name, value, size);
+}
+
+enum {
+ BPF_FS_XATTR_UNSPEC,
+ BPF_FS_XATTR_SECURITY,
+ BPF_FS_XATTR_TRUSTED,
+};
+
+static int bpf_fs_xattr_set(const struct xattr_handler *handler,
+ struct mnt_idmap *idmap, struct dentry *unused,
+ struct inode *inode, const char *name,
+ const void *value, size_t size, int flags)
+{
+ struct bpf_mount_opts *opts = inode->i_sb->s_fs_info;
+ struct bpf_fs_inode *bi = BPF_FS_I(inode);
+ struct simple_xattr *old;
+ int err = -EINVAL;
+
+ name = xattr_full_name(handler, name);
+ switch (handler->flags) {
+ case BPF_FS_XATTR_SECURITY:
+ err = simple_xattr_set_limited(&opts->xa_cache, &bi->xattrs,
+ &bi->xlimits, name, value, size,
+ flags);
+ break;
+ case BPF_FS_XATTR_TRUSTED:
+ old = simple_xattr_set(&opts->xa_cache, &bi->xattrs, name,
+ value, size, flags);
+ err = IS_ERR(old) ? PTR_ERR(old) : 0;
+ if (!err)
+ simple_xattr_free_rcu(old);
+ break;
+ }
+ if (err)
+ return err;
+ inode_set_ctime_current(inode);
+ return 0;
+}
+
+static const struct xattr_handler bpf_fs_trusted_xattr_handler = {
+ .prefix = XATTR_TRUSTED_PREFIX,
+ .flags = BPF_FS_XATTR_TRUSTED,
+ .get = bpf_fs_xattr_get,
+ .set = bpf_fs_xattr_set,
+};
+
+static const struct xattr_handler bpf_fs_security_xattr_handler = {
+ .prefix = XATTR_SECURITY_PREFIX,
+ .flags = BPF_FS_XATTR_SECURITY,
+ .get = bpf_fs_xattr_get,
+ .set = bpf_fs_xattr_set,
+};
+
+static const struct xattr_handler * const bpf_fs_xattr_handlers[] = {
+ &bpf_fs_trusted_xattr_handler,
+ &bpf_fs_security_xattr_handler,
+ NULL,
+};
+
+static ssize_t bpf_fs_listxattr(struct dentry *dentry, char *buf, size_t size)
+{
+ struct inode *inode = d_inode(dentry);
+
+ return simple_xattr_list(inode, &BPF_FS_I(inode)->xattrs, buf, size);
+}
+
+static int bpf_fs_initxattrs(struct inode *inode,
+ const struct xattr *xattr_array, void *fs_info)
+{
+ struct bpf_mount_opts *opts = inode->i_sb->s_fs_info;
+ struct bpf_fs_inode *bi = BPF_FS_I(inode);
+ const struct xattr *xattr;
+ int err;
+
+ for (xattr = xattr_array; xattr->name != NULL; xattr++) {
+ CLASS(simple_xattr, new_xattr)(xattr->value, xattr->value_len);
+ if (IS_ERR(new_xattr))
+ return PTR_ERR(new_xattr);
+
+ new_xattr->name = kasprintf(GFP_KERNEL_ACCOUNT,
+ XATTR_SECURITY_PREFIX "%s",
+ xattr->name);
+ if (!new_xattr->name)
+ return -ENOMEM;
+
+ err = simple_xattr_add_limited(&opts->xa_cache, &bi->xattrs,
+ &bi->xlimits, new_xattr);
+ if (err)
+ return err;
+
+ retain_and_null_ptr(new_xattr);
+ }
+ return 0;
}
const struct super_operations bpf_super_ops = {
.statfs = simple_statfs,
.drop_inode = inode_just_drop,
.show_options = bpf_show_options,
+ .alloc_inode = bpf_fs_alloc_inode,
.destroy_inode = bpf_destroy_inode,
+ .free_inode = bpf_free_inode,
};
enum {
@@ -996,25 +1181,38 @@ static int populate_bpffs(struct dentry *parent)
static int bpf_fill_super(struct super_block *sb, struct fs_context *fc)
{
- static const struct tree_descr bpf_rfiles[] = { { "" } };
struct bpf_mount_opts *opts = sb->s_fs_info;
struct inode *inode;
- int ret;
/* Mounting an instance of BPF FS requires privileges */
if (fc->user_ns != &init_user_ns && !capable(CAP_SYS_ADMIN))
return -EPERM;
- ret = simple_fill_super(sb, BPF_FS_MAGIC, bpf_rfiles);
- if (ret)
- return ret;
-
+ sb->s_blocksize = PAGE_SIZE;
+ sb->s_blocksize_bits = PAGE_SHIFT;
+ sb->s_magic = BPF_FS_MAGIC;
sb->s_op = &bpf_super_ops;
+ sb->s_xattr = bpf_fs_xattr_handlers;
+ sb->s_iflags |= SB_I_NOEXEC;
+ sb->s_iflags |= SB_I_NODEV;
+ sb->s_time_gran = 1;
- inode = sb->s_root->d_inode;
+ inode = bpf_get_inode(sb, NULL, S_IFDIR | 0777);
+ if (IS_ERR(inode))
+ return PTR_ERR(inode);
+
+ inode->i_ino = 1;
+ inode->i_op = &bpf_dir_iops;
+ inode->i_fop = &simple_dir_operations;
+ set_nlink(inode, 2);
+
+ sb->s_root = d_make_root(inode);
+ if (!sb->s_root)
+ return -ENOMEM;
+
+ inode = d_inode(sb->s_root);
inode->i_uid = opts->uid;
inode->i_gid = opts->gid;
- inode->i_op = &bpf_dir_iops;
inode->i_mode &= ~S_IALLUGO;
populate_bpffs(sb->s_root);
inode->i_mode |= S_ISVTX | opts->mode;
@@ -1068,6 +1266,7 @@ static void bpf_kill_super(struct super_block *sb)
struct bpf_mount_opts *opts = sb->s_fs_info;
kill_anon_super(sb);
+ simple_xattr_cache_cleanup(&opts->xa_cache);
kfree(opts);
}
@@ -1080,18 +1279,37 @@ static struct file_system_type bpf_fs_type = {
.fs_flags = FS_USERNS_MOUNT,
};
+static void bpf_fs_inode_init_once(void *foo)
+{
+ struct bpf_fs_inode *bi = foo;
+
+ inode_init_once(&bi->vfs_inode);
+}
+
static int __init bpf_init(void)
{
int ret;
+ bpf_fs_inode_cachep = kmem_cache_create("bpf_fs_inode_cache",
+ sizeof(struct bpf_fs_inode),
+ 0, SLAB_ACCOUNT,
+ bpf_fs_inode_init_once);
+ if (!bpf_fs_inode_cachep)
+ return -ENOMEM;
+
ret = sysfs_create_mount_point(fs_kobj, "bpf");
if (ret)
- return ret;
+ goto out_cache;
ret = register_filesystem(&bpf_fs_type);
- if (ret)
+ if (ret) {
sysfs_remove_mount_point(fs_kobj, "bpf");
+ goto out_cache;
+ }
+ return 0;
+out_cache:
+ kmem_cache_destroy(bpf_fs_inode_cachep);
return ret;
}
fs_initcall(bpf_init);
diff --git a/kernel/cred.c b/kernel/cred.c
index 12a7b1c..3df4e15 100644
--- a/kernel/cred.c
+++ b/kernel/cred.c
@@ -384,8 +384,9 @@ int commit_creds(struct cred *new)
!uid_eq(old->fsuid, new->fsuid) ||
!gid_eq(old->fsgid, new->fsgid) ||
!cred_cap_issubset(old, new)) {
+ /* mm-less tasks share init_task's exec_state */
if (task->mm)
- set_dumpable(task->mm, suid_dumpable);
+ task_exec_state_set_dumpable(suid_dumpable);
task->pdeath_signal = 0;
/*
* If a task drops privileges and becomes nondumpable,
diff --git a/kernel/dma/debug.c b/kernel/dma/debug.c
index 3248f8b..2c0e2cd 100644
--- a/kernel/dma/debug.c
+++ b/kernel/dma/debug.c
@@ -1556,7 +1556,7 @@ void debug_dma_sync_sg_for_device(struct device *dev, struct scatterlist *sg,
struct dma_debug_entry ref = {
.type = dma_debug_sg,
.dev = dev,
- .paddr = sg_phys(sg),
+ .paddr = sg_phys(s),
.dev_addr = sg_dma_address(s),
.size = sg_dma_len(s),
.direction = direction,
diff --git a/kernel/dma/direct.c b/kernel/dma/direct.c
index 583c592..4391b79 100644
--- a/kernel/dma/direct.c
+++ b/kernel/dma/direct.c
@@ -476,7 +476,7 @@ int dma_direct_map_sg(struct device *dev, struct scatterlist *sgl, int nents,
* must be mapped with CPU physical address and not PCI
* bus addresses.
*/
- break;
+ fallthrough;
case PCI_P2PDMA_MAP_NONE:
need_sync = true;
sg->dma_address = dma_direct_map_phys(dev, sg_phys(sg),
diff --git a/kernel/exec_state.c b/kernel/exec_state.c
new file mode 100644
index 0000000..6034f4b
--- /dev/null
+++ b/kernel/exec_state.c
@@ -0,0 +1,119 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2026 Christian Brauner <brauner@kernel.org> */
+#include <linux/init.h>
+#include <linux/rcupdate.h>
+#include <linux/refcount.h>
+#include <linux/sched.h>
+#include <linux/sched/coredump.h>
+#include <linux/sched/exec_state.h>
+#include <linux/sched/signal.h>
+#include <linux/slab.h>
+#include <linux/user_namespace.h>
+
+static struct kmem_cache *task_exec_state_cachep;
+
+static void __free_task_exec_state(struct rcu_head *rcu)
+{
+ struct task_exec_state *exec_state = container_of(rcu, struct task_exec_state, rcu);
+
+ put_user_ns(exec_state->user_ns);
+ kmem_cache_free(task_exec_state_cachep, exec_state);
+}
+
+void put_task_exec_state(struct task_exec_state *exec_state)
+{
+ if (exec_state && refcount_dec_and_test(&exec_state->count))
+ call_rcu(&exec_state->rcu, __free_task_exec_state);
+}
+
+struct task_exec_state *alloc_task_exec_state(struct user_namespace *user_ns)
+{
+ struct task_exec_state *exec_state;
+
+ exec_state = kmem_cache_alloc(task_exec_state_cachep, GFP_KERNEL);
+ if (!exec_state)
+ return NULL;
+ refcount_set(&exec_state->count, 1);
+ exec_state->dumpable = TASK_DUMPABLE_OFF;
+ exec_state->user_ns = get_user_ns(user_ns);
+ return exec_state;
+}
+
+struct task_exec_state *task_exec_state_rcu(const struct task_struct *tsk)
+{
+ struct task_exec_state *exec_state;
+
+ exec_state = rcu_dereference_check(tsk->exec_state,
+ lockdep_is_held(&tsk->alloc_lock));
+ WARN_ON_ONCE(!exec_state);
+ return exec_state;
+}
+
+struct task_exec_state *task_exec_state_replace(struct task_struct *tsk,
+ struct task_exec_state *exec_state)
+{
+ /*
+ * Updates must hold both locks so callers needing a consistent
+ * snapshot of mm + dumpability are covered.
+ */
+ lockdep_assert_held(&tsk->alloc_lock);
+ lockdep_assert_held_write(&tsk->signal->exec_update_lock);
+
+ return rcu_replace_pointer(tsk->exec_state, exec_state, true);
+}
+
+/*
+ * The non-CLONE_VM clone path: allocate a fresh exec_state and
+ * inherit the parent's dumpable mode and user_ns reference. CLONE_VM
+ * siblings refcount-share via copy_exec_state() in fork.c; only this
+ * path and execve() ever allocate.
+ */
+int task_exec_state_copy(struct task_struct *tsk)
+{
+ struct task_exec_state *src, *dst;
+
+ src = rcu_dereference_protected(current->exec_state, true);
+ dst = alloc_task_exec_state(src->user_ns);
+ if (!dst)
+ return -ENOMEM;
+ dst->dumpable = READ_ONCE(src->dumpable);
+ rcu_assign_pointer(tsk->exec_state, dst);
+ return 0;
+}
+
+/*
+ * Store TASK_DUMPABLE_* on current->exec_state. All callers
+ * (commit_creds, begin_new_exec, prctl(PR_SET_DUMPABLE)) act on the
+ * running task, which guarantees ->exec_state is allocated and cannot
+ * be replaced under us.
+ */
+void task_exec_state_set_dumpable(enum task_dumpable value)
+{
+ struct task_exec_state *exec_state;
+
+ if (WARN_ON_ONCE(value > TASK_DUMPABLE_ROOT))
+ value = TASK_DUMPABLE_OFF;
+
+ exec_state = rcu_dereference_protected(current->exec_state, true);
+ /* mm-less tasks share init_task's exec_state; never mutate it */
+ if (WARN_ON_ONCE(exec_state == &init_task_exec_state))
+ return;
+ WRITE_ONCE(exec_state->dumpable, value);
+}
+
+enum task_dumpable task_exec_state_get_dumpable(struct task_struct *task)
+{
+ struct task_exec_state *exec_state;
+
+ guard(rcu)();
+ exec_state = rcu_dereference(task->exec_state);
+ return READ_ONCE(exec_state->dumpable);
+}
+
+void __init exec_state_init(void)
+{
+ task_exec_state_cachep = kmem_cache_create("task_exec_state",
+ sizeof(struct task_exec_state), 0,
+ SLAB_HWCACHE_ALIGN | SLAB_PANIC | SLAB_ACCOUNT,
+ NULL);
+}
diff --git a/kernel/exit.c b/kernel/exit.c
index f50d73c..9a90999 100644
--- a/kernel/exit.c
+++ b/kernel/exit.c
@@ -571,7 +571,6 @@ static void exit_mm(void)
*/
smp_mb__after_spinlock();
local_irq_disable();
- current->user_dumpable = (get_dumpable(mm) == SUID_DUMP_USER);
current->mm = NULL;
membarrier_update_current_mm(NULL);
enter_lazy_tlb(mm, current);
diff --git a/kernel/fork.c b/kernel/fork.c
index 8ac38be..ba6b03d 100644
--- a/kernel/fork.c
+++ b/kernel/fork.c
@@ -23,6 +23,7 @@
#include <linux/sched/task_stack.h>
#include <linux/sched/cputime.h>
#include <linux/sched/ext.h>
+#include <linux/sched/exec_state.h>
#include <linux/seq_file.h>
#include <linux/rtmutex.h>
#include <linux/init.h>
@@ -555,6 +556,7 @@ void free_task(struct task_struct *tsk)
if (tsk->flags & PF_KTHREAD)
free_kthread_struct(tsk);
bpf_task_storage_free(tsk);
+ put_task_exec_state(rcu_access_pointer(tsk->exec_state));
free_task_struct(tsk);
}
EXPORT_SYMBOL(free_task);
@@ -731,7 +733,6 @@ void __mmdrop(struct mm_struct *mm)
destroy_context(mm);
mmu_notifier_subscriptions_destroy(mm);
check_mm(mm);
- put_user_ns(mm->user_ns);
mm_pasid_drop(mm);
mm_destroy_cid(mm);
percpu_counter_destroy_many(mm->rss_stat, NR_MM_COUNTERS);
@@ -946,6 +947,8 @@ static struct task_struct *dup_task_struct(struct task_struct *orig, int node)
tsk->seccomp.filter = NULL;
#endif
+ RCU_INIT_POINTER(tsk->exec_state, NULL);
+
setup_thread_stack(tsk, orig);
clear_user_return_notifier(tsk);
clear_tsk_need_resched(tsk);
@@ -1072,8 +1075,7 @@ static void mmap_init_lock(struct mm_struct *mm)
#endif
}
-static struct mm_struct *mm_init(struct mm_struct *mm, struct task_struct *p,
- struct user_namespace *user_ns)
+static struct mm_struct *mm_init(struct mm_struct *mm, struct task_struct *p)
{
mt_init_flags(&mm->mm_mt, MM_MT_FLAGS);
mt_set_external_lock(&mm->mm_mt, &mm->mmap_lock);
@@ -1132,7 +1134,6 @@ static struct mm_struct *mm_init(struct mm_struct *mm, struct task_struct *p,
NR_MM_COUNTERS))
goto fail_pcpu;
- mm->user_ns = get_user_ns(user_ns);
lru_gen_init_mm(mm);
return mm;
@@ -1163,7 +1164,7 @@ struct mm_struct *mm_alloc(void)
return NULL;
memset(mm, 0, sizeof(*mm));
- return mm_init(mm, current, current_user_ns());
+ return mm_init(mm, current);
}
EXPORT_SYMBOL_IF_KUNIT(mm_alloc);
@@ -1527,7 +1528,7 @@ static struct mm_struct *dup_mm(struct task_struct *tsk,
memcpy(mm, oldmm, sizeof(*mm));
- if (!mm_init(mm, tsk, mm->user_ns))
+ if (!mm_init(mm, tsk))
goto fail_nomem;
uprobe_start_dup_mmap();
@@ -1593,6 +1594,22 @@ static int copy_mm(u64 clone_flags, struct task_struct *tsk)
return 0;
}
+static int copy_exec_state(u64 clone_flags, struct task_struct *tsk)
+{
+ struct task_exec_state *exec_state;
+
+ /* CLONE_VM siblings refcount-share the parent's exec_state. */
+ if (clone_flags & CLONE_VM) {
+ exec_state = rcu_dereference_protected(current->exec_state, true);
+ refcount_inc(&exec_state->count);
+ rcu_assign_pointer(tsk->exec_state, exec_state);
+ return 0;
+ }
+
+ /* Everyone else inherits a fresh copy. */
+ return task_exec_state_copy(tsk);
+}
+
static int copy_fs(u64 clone_flags, struct task_struct *tsk)
{
struct fs_struct *fs = current->fs;
@@ -2090,6 +2107,9 @@ __latent_entropy struct task_struct *copy_process(
p = dup_task_struct(current, node);
if (!p)
goto fork_out;
+ retval = copy_exec_state(clone_flags, p);
+ if (retval)
+ goto bad_fork_free;
p->flags &= ~PF_KTHREAD;
if (args->kthread)
p->flags |= PF_KTHREAD;
@@ -3097,6 +3117,7 @@ void __init proc_caches_init(void)
sizeof(struct signal_struct), 0,
SLAB_HWCACHE_ALIGN|SLAB_PANIC|SLAB_ACCOUNT,
NULL);
+ exec_state_init();
files_cachep = kmem_cache_create("files_cache",
sizeof(struct files_struct), 0,
SLAB_HWCACHE_ALIGN|SLAB_PANIC|SLAB_ACCOUNT,
diff --git a/kernel/kthread.c b/kernel/kthread.c
index 791210d..63beb59 100644
--- a/kernel/kthread.c
+++ b/kernel/kthread.c
@@ -1619,7 +1619,6 @@ void kthread_use_mm(struct mm_struct *mm)
WARN_ON_ONCE(!(tsk->flags & PF_KTHREAD));
WARN_ON_ONCE(tsk->mm);
- WARN_ON_ONCE(!mm->user_ns);
/*
* It is possible for mm to be the same as tsk->active_mm, but
diff --git a/kernel/ptrace.c b/kernel/ptrace.c
index 130043b..d041645 100644
--- a/kernel/ptrace.c
+++ b/kernel/ptrace.c
@@ -13,6 +13,7 @@
#include <linux/sched.h>
#include <linux/sched/mm.h>
#include <linux/sched/coredump.h>
+#include <linux/sched/exec_state.h>
#include <linux/sched/task.h>
#include <linux/errno.h>
#include <linux/mm.h>
@@ -36,6 +37,30 @@
#include <asm/syscall.h> /* for syscall_get_* */
+/**
+ * ptracer_access_allowed - may current peek/poke @tsk's address space?
+ * @tsk: tracee
+ *
+ * Per-access check used by ptrace_access_vm() and architecture-specific
+ * tag/register accessors. Returns true iff current is the registered
+ * ptracer of @tsk and either @tsk is owner-dumpable or current holds
+ * CAP_SYS_PTRACE in @tsk's exec namespace. Lighter than
+ * __ptrace_may_access(): it re-validates only dumpability and
+ * capability on every access, without re-running LSM hooks or
+ * cred_cap_issubset() checks performed at attach time.
+ */
+bool ptracer_access_allowed(struct task_struct *tsk)
+{
+ const struct task_exec_state *es;
+
+ guard(rcu)();
+ if (ptrace_parent(tsk) != current)
+ return false;
+ es = task_exec_state_rcu(tsk);
+ return READ_ONCE(es->dumpable) == TASK_DUMPABLE_OWNER ||
+ ptracer_capable(tsk, es->user_ns);
+}
+
/*
* Access another process' address space via ptrace.
* Source/target buffer must be kernel space,
@@ -45,21 +70,14 @@ int ptrace_access_vm(struct task_struct *tsk, unsigned long addr,
void *buf, int len, unsigned int gup_flags)
{
struct mm_struct *mm;
- int ret;
+ int ret = 0;
mm = get_task_mm(tsk);
if (!mm)
return 0;
- if (!tsk->ptrace ||
- (current != tsk->parent) ||
- ((get_dumpable(mm) != SUID_DUMP_USER) &&
- !ptracer_capable(tsk, mm->user_ns))) {
- mmput(mm);
- return 0;
- }
-
- ret = access_remote_vm(mm, addr, buf, len, gup_flags);
+ if (ptracer_access_allowed(tsk))
+ ret = access_remote_vm(mm, addr, buf, len, gup_flags);
mmput(mm);
return ret;
@@ -274,16 +292,13 @@ static bool ptrace_has_cap(struct user_namespace *ns, unsigned int mode)
static bool task_still_dumpable(struct task_struct *task, unsigned int mode)
{
- struct mm_struct *mm = task->mm;
- if (mm) {
- if (get_dumpable(mm) == SUID_DUMP_USER)
- return true;
- return ptrace_has_cap(mm->user_ns, mode);
- }
+ const struct task_exec_state *exec_state;
- if (task->user_dumpable)
+ guard(rcu)();
+ exec_state = task_exec_state_rcu(task);
+ if (READ_ONCE(exec_state->dumpable) == TASK_DUMPABLE_OWNER)
return true;
- return ptrace_has_cap(&init_user_ns, mode);
+ return ptrace_has_cap(exec_state->user_ns, mode);
}
/* Returns 0 on success, -errno on denial. */
diff --git a/kernel/rcu/rcutorture.c b/kernel/rcu/rcutorture.c
index 5f2848b..882a158 100644
--- a/kernel/rcu/rcutorture.c
+++ b/kernel/rcu/rcutorture.c
@@ -572,7 +572,7 @@ static unsigned long rcu_no_completed(void)
static void rcu_torture_deferred_free(struct rcu_torture *p)
{
- call_rcu_hurry(&p->rtort_rcu, rcu_torture_cb);
+ call_rcu(&p->rtort_rcu, rcu_torture_cb);
}
static void rcu_sync_torture_init(void)
@@ -619,7 +619,7 @@ static struct rcu_torture_ops rcu_ops = {
.poll_gp_state_exp = poll_state_synchronize_rcu,
.cond_sync_exp = cond_synchronize_rcu_expedited,
.cond_sync_exp_full = cond_synchronize_rcu_expedited_full,
- .call = call_rcu_hurry,
+ .call = call_rcu,
.cb_barrier = rcu_barrier,
.fqs = rcu_force_quiescent_state,
.gp_kthread_dbg = show_rcu_gp_kthreads,
@@ -1145,7 +1145,7 @@ static void rcu_tasks_torture_deferred_free(struct rcu_torture *p)
static void synchronize_rcu_mult_test(void)
{
- synchronize_rcu_mult(call_rcu_tasks, call_rcu_hurry);
+ synchronize_rcu_mult(call_rcu_tasks, call_rcu);
}
static struct rcu_torture_ops tasks_ops = {
@@ -1632,6 +1632,17 @@ static void do_rtws_sync(struct torture_random_state *trsp, void (*sync)(void))
}
/*
+ * Do an rcu_barrier() to motivate lazy callbacks during a stutter
+ * pause. Without this, we can get false-positives rtort_pipe_count
+ * splats.
+ */
+static void rcu_torture_writer_work(struct work_struct *work)
+{
+ if (cur_ops->cb_barrier)
+ cur_ops->cb_barrier();
+}
+
+/*
* RCU torture writer kthread. Repeatedly substitutes a new structure
* for that pointed to by rcu_torture_current, freeing the old structure
* after a series of grace periods (the "pipeline").
@@ -1651,6 +1662,7 @@ rcu_torture_writer(void *arg)
int i;
int idx;
unsigned long j;
+ struct work_struct lazy_work;
int oldnice = task_nice(current);
struct rcu_gp_oldstate *rgo = NULL;
int rgo_size = 0;
@@ -1703,6 +1715,9 @@ rcu_torture_writer(void *arg)
pr_alert("%s" TORTURE_FLAG " Waited %lu jiffies for boot to complete.\n",
torture_type, jiffies - j);
+ if (IS_ENABLED(CONFIG_RCU_LAZY))
+ INIT_WORK_ONSTACK(&lazy_work, rcu_torture_writer_work);
+
do {
rcu_torture_writer_state = RTWS_FIXED_DELAY;
torture_hrtimeout_us(500, 1000, &rand);
@@ -1895,6 +1910,8 @@ rcu_torture_writer(void *arg)
!rcu_gp_is_normal();
}
rcu_torture_writer_state = RTWS_STUTTER;
+ if (IS_ENABLED(CONFIG_RCU_LAZY))
+ queue_work(system_percpu_wq, &lazy_work);
stutter_waited = stutter_wait("rcu_torture_writer");
if (stutter_waited &&
!atomic_read(&rcu_fwd_cb_nodelay) &&
@@ -1925,6 +1942,12 @@ rcu_torture_writer(void *arg)
pr_alert("%s" TORTURE_FLAG
" Dynamic grace-period expediting was disabled.\n",
torture_type);
+
+ if (IS_ENABLED(CONFIG_RCU_LAZY)) {
+ cancel_work_sync(&lazy_work);
+ destroy_work_on_stack(&lazy_work);
+ }
+
kfree(ulo);
kfree(rgo);
rcu_torture_writer_state = RTWS_STOPPING;
diff --git a/kernel/rcu/tasks.h b/kernel/rcu/tasks.h
index 48f0d80..f4da5fa 100644
--- a/kernel/rcu/tasks.h
+++ b/kernel/rcu/tasks.h
@@ -373,7 +373,8 @@ static void call_rcu_tasks_generic(struct rcu_head *rhp, rcu_callback_t func,
// Queuing callbacks before initialization not yet supported.
if (WARN_ON_ONCE(!rcu_segcblist_is_enabled(&rtpcp->cblist)))
rcu_segcblist_init(&rtpcp->cblist);
- needwake = (func == wakeme_after_rcu) ||
+ needwake = (!havekthread && rcu_segcblist_empty(&rtpcp->cblist)) ||
+ (func == wakeme_after_rcu) ||
(rcu_segcblist_n_cbs(&rtpcp->cblist) == rcu_task_lazy_lim);
if (havekthread && !needwake && !timer_pending(&rtpcp->lazy_timer)) {
if (rtp->lazy_jiffies)
diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c
index 55df6d3..afb9e7d 100644
--- a/kernel/rcu/tree.c
+++ b/kernel/rcu/tree.c
@@ -492,7 +492,7 @@ static int param_set_next_fqs_jiffies(const char *val, const struct kernel_param
int ret = kstrtoul(val, 0, &j);
if (!ret) {
- WRITE_ONCE(*(ulong *)kp->arg, (j > HZ) ? HZ : (j ?: 1));
+ WRITE_ONCE(*(ulong *)kp->arg, clamp_val(j, 1, HZ));
adjust_jiffies_till_sched_qs();
}
return ret;
@@ -1632,17 +1632,21 @@ static void rcu_sr_put_wait_head(struct llist_node *node)
atomic_set_release(&sr_wn->inuse, 0);
}
-/* Enable rcu_normal_wake_from_gp automatically on small systems. */
-#define WAKE_FROM_GP_CPU_THRESHOLD 16
-
-static int rcu_normal_wake_from_gp = -1;
+static int rcu_normal_wake_from_gp = 1;
module_param(rcu_normal_wake_from_gp, int, 0644);
static struct workqueue_struct *sync_wq;
+#define RCU_SR_NORMAL_LATCH_THR 64
+
+/* Number of in-flight synchronize_rcu() calls queued on srs_next. */
+static atomic_long_t rcu_sr_normal_count;
+static int rcu_sr_normal_latched; /* 0/1 */
+
static void rcu_sr_normal_complete(struct llist_node *node)
{
struct rcu_synchronize *rs = container_of(
(struct rcu_head *) node, struct rcu_synchronize, head);
+ long nr;
WARN_ONCE(IS_ENABLED(CONFIG_PROVE_RCU) &&
!poll_state_synchronize_rcu_full(&rs->oldstate),
@@ -1650,6 +1654,15 @@ static void rcu_sr_normal_complete(struct llist_node *node)
/* Finally. */
complete(&rs->completion);
+ nr = atomic_long_dec_return(&rcu_sr_normal_count);
+ WARN_ON_ONCE(nr < 0);
+
+ /*
+ * Unlatch: switch back to normal path when fully
+ * drained and if it has been latched.
+ */
+ if (nr == 0)
+ (void)cmpxchg_relaxed(&rcu_sr_normal_latched, 1, 0);
}
static void rcu_sr_normal_gp_cleanup_work(struct work_struct *work)
@@ -1795,6 +1808,24 @@ static bool rcu_sr_normal_gp_init(void)
static void rcu_sr_normal_add_req(struct rcu_synchronize *rs)
{
+ /*
+ * Increment before publish to avoid a complete
+ * vs enqueue race on latch.
+ */
+ long nr = atomic_long_inc_return(&rcu_sr_normal_count);
+
+ /*
+ * Latch when threshold is reached. Checking for an exact match
+ * restricts cmpxchg() to a single context.
+ *
+ * This latch is intentionally relaxed and best-effort. Concurrent
+ * set/clear can race and temporarily lose the latch, which is OK
+ * because it only selects between the fast and fallback paths.
+ */
+ if (nr == RCU_SR_NORMAL_LATCH_THR)
+ (void)cmpxchg_relaxed(&rcu_sr_normal_latched, 0, 1);
+
+ /* Publish for the GP kthread/worker. */
llist_add((struct llist_node *) &rs->head, &rcu_state.srs_next);
}
@@ -2584,7 +2615,7 @@ static void rcu_do_batch(struct rcu_data *rdp)
const long npj = NSEC_PER_SEC / HZ;
long rrn = READ_ONCE(rcu_resched_ns);
- rrn = rrn < NSEC_PER_MSEC ? NSEC_PER_MSEC : rrn > NSEC_PER_SEC ? NSEC_PER_SEC : rrn;
+ rrn = clamp(rrn, NSEC_PER_MSEC, NSEC_PER_SEC);
tlimit = local_clock() + rrn;
jlimit = jiffies + (rrn + npj + 1) / npj;
jlimit_check = true;
@@ -3278,14 +3309,15 @@ static void synchronize_rcu_normal(void)
{
struct rcu_synchronize rs;
+ init_rcu_head_on_stack(&rs.head);
trace_rcu_sr_normal(rcu_state.name, &rs.head, TPS("request"));
- if (READ_ONCE(rcu_normal_wake_from_gp) < 1) {
+ if (READ_ONCE(rcu_normal_wake_from_gp) < 1 ||
+ READ_ONCE(rcu_sr_normal_latched)) {
wait_rcu_gp(call_rcu_hurry);
goto trace_complete_out;
}
- init_rcu_head_on_stack(&rs.head);
init_completion(&rs.completion);
/*
@@ -3302,10 +3334,10 @@ static void synchronize_rcu_normal(void)
/* Now we can wait. */
wait_for_completion(&rs.completion);
- destroy_rcu_head_on_stack(&rs.head);
trace_complete_out:
trace_rcu_sr_normal(rcu_state.name, &rs.head, TPS("complete"));
+ destroy_rcu_head_on_stack(&rs.head);
}
/**
@@ -4904,12 +4936,6 @@ void __init rcu_init(void)
sync_wq = alloc_workqueue("sync_wq", WQ_MEM_RECLAIM | WQ_UNBOUND, 0);
WARN_ON(!sync_wq);
- /* Respect if explicitly disabled via a boot parameter. */
- if (rcu_normal_wake_from_gp < 0) {
- if (num_possible_cpus() <= WAKE_FROM_GP_CPU_THRESHOLD)
- rcu_normal_wake_from_gp = 1;
- }
-
/* Fill in default value for rcutree.qovld boot parameter. */
/* -After- the rcu_node ->lock fields are initialized! */
if (qovld < 0)
diff --git a/kernel/rcu/tree_nocb.h b/kernel/rcu/tree_nocb.h
index 1047b30..373b877 100644
--- a/kernel/rcu/tree_nocb.h
+++ b/kernel/rcu/tree_nocb.h
@@ -655,7 +655,7 @@ static void nocb_gp_sleep(struct rcu_data *my_rdp, int cpu)
* No-CBs GP kthreads come here to wait for additional callbacks to show up
* or for grace periods to end.
*/
-static void nocb_gp_wait(struct rcu_data *my_rdp)
+static noinline_for_stack void nocb_gp_wait(struct rcu_data *my_rdp)
{
bool bypass = false;
int __maybe_unused cpu = my_rdp->cpu;
diff --git a/kernel/sys.c b/kernel/sys.c
index 62e8420..df69bd7 100644
--- a/kernel/sys.c
+++ b/kernel/sys.c
@@ -2565,14 +2565,14 @@ SYSCALL_DEFINE5(prctl, int, option, unsigned long, arg2, unsigned long, arg3,
error = put_user(me->pdeath_signal, (int __user *)arg2);
break;
case PR_GET_DUMPABLE:
- error = get_dumpable(me->mm);
+ error = task_exec_state_get_dumpable(me);
break;
case PR_SET_DUMPABLE:
- if (arg2 != SUID_DUMP_DISABLE && arg2 != SUID_DUMP_USER) {
+ if (arg2 != TASK_DUMPABLE_OFF && arg2 != TASK_DUMPABLE_OWNER) {
error = -EINVAL;
break;
}
- set_dumpable(me->mm, arg2);
+ task_exec_state_set_dumpable(arg2);
break;
case PR_SET_UNALIGN:
diff --git a/kernel/torture.c b/kernel/torture.c
index 62c1ac7..77cb358 100644
--- a/kernel/torture.c
+++ b/kernel/torture.c
@@ -972,3 +972,19 @@ void _torture_stop_kthread(char *m, struct task_struct **tp)
*tp = NULL;
}
EXPORT_SYMBOL_GPL(_torture_stop_kthread);
+
+/*
+ * Set the specified task's niceness value, saturating at limits.
+ * Saturating noisily, but saturating.
+ */
+void torture_sched_set_normal(struct task_struct *t, int nice)
+{
+ int realnice = nice;
+
+ if (WARN_ON_ONCE(realnice > MAX_NICE))
+ realnice = MAX_NICE;
+ if (WARN_ON_ONCE(realnice < MIN_NICE))
+ realnice = MIN_NICE;
+ sched_set_normal(t, realnice);
+}
+EXPORT_SYMBOL_GPL(torture_sched_set_normal);
diff --git a/lib/debugobjects.c b/lib/debugobjects.c
index b18a682..6fb00e0 100644
--- a/lib/debugobjects.c
+++ b/lib/debugobjects.c
@@ -720,6 +720,41 @@ static inline bool debug_objects_is_pi_blocked_on(void)
#endif
}
+static inline bool can_fill_pool(void)
+{
+ /*
+ * On !RT enabled kernels there are no restrictions and spinlock_t and
+ * raw_spinlock_t are the same types.
+ */
+ if (!IS_ENABLED(CONFIG_PREEMPT_RT))
+ return true;
+
+ /*
+ * On RT enabled kernels, the task must not be blocked on a lock as
+ * that could corrupt the PI state when blocking on a lock in the
+ * allocation path.
+ */
+ if (debug_objects_is_pi_blocked_on())
+ return false;
+
+ /*
+ * On RT enabled kernels the pool refill should happen in preemptible
+ * context.
+ */
+ if (preemptible())
+ return true;
+
+ /*
+ * Though during system boot before scheduling is set up, preemption is
+ * disabled and the pool can get exhausted. Before scheduling is active
+ * a task cannot be blocked on a sleeping lock, but it might hold a lock
+ * and if interrupted then hard interrupt context might run into a lock
+ * inversion. So exclude hard interrupt context from allocations before
+ * scheduling is active.
+ */
+ return system_state < SYSTEM_SCHEDULING && !in_hardirq();
+}
+
static void debug_objects_fill_pool(void)
{
if (!static_branch_likely(&obj_cache_enabled))
@@ -734,18 +769,11 @@ static void debug_objects_fill_pool(void)
if (likely(!pool_should_refill(&pool_global)))
return;
- /*
- * On RT enabled kernels the pool refill must happen in preemptible
- * context and not enqueued on an rt_mutex -- for !RT kernels we rely
- * on the fact that spinlock_t and raw_spinlock_t are basically the
- * same type and this lock-type inversion works just fine.
- */
- if (!IS_ENABLED(CONFIG_PREEMPT_RT) || system_state < SYSTEM_SCHEDULING ||
- (preemptible() && !debug_objects_is_pi_blocked_on())) {
+ if (can_fill_pool()) {
/*
* Annotate away the spinlock_t inside raw_spinlock_t warning
* by temporarily raising the wait-type to LD_WAIT_CONFIG, matching
- * the preemptible() condition above.
+ * the preemptible() condition in can_fill_pool().
*/
static DEFINE_WAIT_OVERRIDE_MAP(fill_pool_map, LD_WAIT_CONFIG);
lock_map_acquire_try(&fill_pool_map);
diff --git a/lib/iov_iter.c b/lib/iov_iter.c
index 243662a..273919b 100644
--- a/lib/iov_iter.c
+++ b/lib/iov_iter.c
@@ -1224,13 +1224,13 @@ const void *dup_iter(struct iov_iter *new, struct iov_iter *old, gfp_t flags)
{
*new = *old;
if (iov_iter_is_bvec(new))
- return new->bvec = kmemdup(new->bvec,
- new->nr_segs * sizeof(struct bio_vec),
+ return new->bvec = kmemdup_array(new->bvec,
+ new->nr_segs, sizeof(struct bio_vec),
flags);
else if (iov_iter_is_kvec(new) || iter_is_iovec(new))
/* iovec and kvec have identical layout */
- return new->__iov = kmemdup(new->__iov,
- new->nr_segs * sizeof(struct iovec),
+ return new->__iov = kmemdup_array(new->__iov,
+ new->nr_segs, sizeof(struct iovec),
flags);
return NULL;
}
diff --git a/lib/rhashtable.c b/lib/rhashtable.c
index 04b3a80..c0ba34ead 100644
--- a/lib/rhashtable.c
+++ b/lib/rhashtable.c
@@ -1057,8 +1057,9 @@ static u32 rhashtable_jhash2(const void *key, u32 length, u32 seed)
* .obj_hashfn = my_hash_fn,
* };
*/
-int rhashtable_init_noprof(struct rhashtable *ht,
- const struct rhashtable_params *params)
+int __rhashtable_init_noprof(struct rhashtable *ht,
+ const struct rhashtable_params *params,
+ struct lock_class_key *key)
{
struct bucket_table *tbl;
size_t size;
@@ -1068,7 +1069,7 @@ int rhashtable_init_noprof(struct rhashtable *ht,
return -EINVAL;
memset(ht, 0, sizeof(*ht));
- mutex_init(&ht->mutex);
+ mutex_init_with_key(&ht->mutex, key);
spin_lock_init(&ht->lock);
memcpy(&ht->p, params, sizeof(*params));
@@ -1120,7 +1121,7 @@ int rhashtable_init_noprof(struct rhashtable *ht,
return 0;
}
-EXPORT_SYMBOL_GPL(rhashtable_init_noprof);
+EXPORT_SYMBOL_GPL(__rhashtable_init_noprof);
/**
* rhltable_init - initialize a new hash list table
@@ -1131,15 +1132,17 @@ EXPORT_SYMBOL_GPL(rhashtable_init_noprof);
*
* See documentation for rhashtable_init.
*/
-int rhltable_init_noprof(struct rhltable *hlt, const struct rhashtable_params *params)
+int __rhltable_init_noprof(struct rhltable *hlt,
+ const struct rhashtable_params *params,
+ struct lock_class_key *key)
{
int err;
- err = rhashtable_init_noprof(&hlt->ht, params);
+ err = __rhashtable_init_noprof(&hlt->ht, params, key);
hlt->ht.rhlist = true;
return err;
}
-EXPORT_SYMBOL_GPL(rhltable_init_noprof);
+EXPORT_SYMBOL_GPL(__rhltable_init_noprof);
static void rhashtable_free_one(struct rhashtable *ht, struct rhash_head *obj,
void (*free_fn)(void *ptr, void *arg),
diff --git a/lib/vsprintf.c b/lib/vsprintf.c
index 9f359b3..a6169e9 100644
--- a/lib/vsprintf.c
+++ b/lib/vsprintf.c
@@ -129,13 +129,6 @@ unsigned long simple_strtoul(const char *cp, char **endp, unsigned int base)
}
EXPORT_SYMBOL(simple_strtoul);
-unsigned long simple_strntoul(const char *cp, char **endp, unsigned int base,
- size_t max_chars)
-{
- return simple_strntoull(cp, endp, base, max_chars);
-}
-EXPORT_SYMBOL(simple_strntoul);
-
/**
* simple_strtol - convert a string to a signed long
* @cp: The start of the string
diff --git a/mm/filemap.c b/mm/filemap.c
index 4e63664..179f288 100644
--- a/mm/filemap.c
+++ b/mm/filemap.c
@@ -2052,8 +2052,19 @@ struct folio *__filemap_get_folio_mpol(struct address_space *mapping,
if (!folio)
return ERR_PTR(-ENOENT);
/* not an uncached lookup, clear uncached if set */
- if (folio_test_dropbehind(folio) && !(fgp_flags & FGP_DONTCACHE))
- folio_clear_dropbehind(folio);
+ if (!(fgp_flags & FGP_DONTCACHE) && folio_test_clear_dropbehind(folio)) {
+ if (folio_test_dirty(folio) &&
+ mapping_can_writeback(mapping)) {
+ struct inode *inode = mapping->host;
+ struct bdi_writeback *wb;
+ struct wb_lock_cookie cookie = {};
+ long nr = folio_nr_pages(folio);
+
+ wb = unlocked_inode_to_wb_begin(inode, &cookie);
+ wb_stat_mod(wb, WB_DONTCACHE_DIRTY, -nr);
+ unlocked_inode_to_wb_end(inode, &cookie);
+ }
+ }
return folio;
}
EXPORT_SYMBOL(__filemap_get_folio_mpol);
diff --git a/mm/huge_memory.c b/mm/huge_memory.c
index b118bcd..d29e854 100644
--- a/mm/huge_memory.c
+++ b/mm/huge_memory.c
@@ -3644,6 +3644,7 @@ static void __split_folio_to_order(struct folio *folio, int old_order,
(1L << PG_arch_3) |
#endif
(1L << PG_dirty) |
+ (1L << PG_dropbehind) |
LRU_GEN_MASK | LRU_REFS_MASK));
if (handle_hwpoison &&
diff --git a/mm/init-mm.c b/mm/init-mm.c
index c5556bb..3e792aad 100644
--- a/mm/init-mm.c
+++ b/mm/init-mm.c
@@ -43,7 +43,6 @@ struct mm_struct init_mm = {
.vma_writer_wait = __RCUWAIT_INITIALIZER(init_mm.vma_writer_wait),
.mm_lock_seq = SEQCNT_ZERO(init_mm.mm_lock_seq),
#endif
- .user_ns = &init_user_ns,
#ifdef CONFIG_SCHED_MM_CID
.mm_cid.lock = __RAW_SPIN_LOCK_UNLOCKED(init_mm.mm_cid.lock),
#endif
diff --git a/mm/page-writeback.c b/mm/page-writeback.c
index 833f743..e987481 100644
--- a/mm/page-writeback.c
+++ b/mm/page-writeback.c
@@ -2626,6 +2626,8 @@ static void folio_account_dirtied(struct folio *folio,
wb = inode_to_wb(inode);
lruvec_stat_mod_folio(folio, NR_FILE_DIRTY, nr);
+ if (folio_test_dropbehind(folio))
+ wb_stat_mod(wb, WB_DONTCACHE_DIRTY, nr);
__zone_stat_mod_folio(folio, NR_ZONE_WRITE_PENDING, nr);
__node_stat_mod_folio(folio, NR_DIRTIED, nr);
wb_stat_mod(wb, WB_RECLAIMABLE, nr);
@@ -2647,6 +2649,8 @@ void folio_account_cleaned(struct folio *folio, struct bdi_writeback *wb)
long nr = folio_nr_pages(folio);
lruvec_stat_mod_folio(folio, NR_FILE_DIRTY, -nr);
+ if (folio_test_dropbehind(folio))
+ wb_stat_mod(wb, WB_DONTCACHE_DIRTY, -nr);
zone_stat_mod_folio(folio, NR_ZONE_WRITE_PENDING, -nr);
wb_stat_mod(wb, WB_RECLAIMABLE, -nr);
task_io_account_cancelled_write(nr * PAGE_SIZE);
@@ -2916,6 +2920,8 @@ bool folio_clear_dirty_for_io(struct folio *folio)
if (folio_test_clear_dirty(folio)) {
long nr = folio_nr_pages(folio);
lruvec_stat_mod_folio(folio, NR_FILE_DIRTY, -nr);
+ if (folio_test_dropbehind(folio))
+ wb_stat_mod(wb, WB_DONTCACHE_DIRTY, -nr);
zone_stat_mod_folio(folio, NR_ZONE_WRITE_PENDING, -nr);
wb_stat_mod(wb, WB_RECLAIMABLE, -nr);
ret = true;
diff --git a/mm/secretmem.c b/mm/secretmem.c
index 5f57ac4..4877c26 100644
--- a/mm/secretmem.c
+++ b/mm/secretmem.c
@@ -245,8 +245,6 @@ static int secretmem_init_fs_context(struct fs_context *fc)
if (!ctx)
return -ENOMEM;
- fc->s_iflags |= SB_I_NOEXEC;
- fc->s_iflags |= SB_I_NODEV;
return 0;
}
diff --git a/mm/shmem.c b/mm/shmem.c
index 3b5dc21..7b1ea9f 100644
--- a/mm/shmem.c
+++ b/mm/shmem.c
@@ -1425,10 +1425,8 @@ static void shmem_evict_inode(struct inode *inode)
}
}
- if (info->xattrs) {
- simple_xattrs_free(info->xattrs, sbinfo->max_inodes ? &freed : NULL);
- kfree(info->xattrs);
- }
+ simple_xattrs_free(&sbinfo->xa_cache, &info->xattrs, sbinfo->max_inodes ? &freed : NULL);
+
shmem_free_inode(inode->i_sb, freed);
WARN_ON(inode->i_blocks);
clear_inode(inode);
@@ -3086,6 +3084,7 @@ static struct inode *__shmem_get_inode(struct mnt_idmap *idmap,
inode->i_generation = get_random_u32();
info = SHMEM_I(inode);
memset(info, 0, (char *)inode - (char *)info);
+ INIT_LIST_HEAD_RCU(&info->xattrs);
spin_lock_init(&info->lock);
atomic_set(&info->stop_eviction, 0);
info->seals = F_SEAL_SEAL;
@@ -4232,11 +4231,6 @@ static int shmem_initxattrs(struct inode *inode,
struct shmem_sb_info *sbinfo = SHMEM_SB(inode->i_sb);
const struct xattr *xattr;
size_t ispace = 0;
- size_t len;
-
- CLASS(simple_xattrs, xattrs)();
- if (IS_ERR(xattrs))
- return PTR_ERR(xattrs);
if (sbinfo->max_inodes) {
for (xattr = xattr_array; xattr->name != NULL; xattr++) {
@@ -4260,19 +4254,16 @@ static int shmem_initxattrs(struct inode *inode,
if (IS_ERR(new_xattr))
break;
- len = strlen(xattr->name) + 1;
- new_xattr->name = kmalloc(XATTR_SECURITY_PREFIX_LEN + len,
- GFP_KERNEL_ACCOUNT);
+ new_xattr->name = kasprintf(GFP_KERNEL_ACCOUNT,
+ XATTR_SECURITY_PREFIX "%s", xattr->name);
if (!new_xattr->name)
break;
- memcpy(new_xattr->name, XATTR_SECURITY_PREFIX,
- XATTR_SECURITY_PREFIX_LEN);
- memcpy(new_xattr->name + XATTR_SECURITY_PREFIX_LEN,
- xattr->name, len);
-
- if (simple_xattr_add(xattrs, new_xattr))
+ if (simple_xattr_add(&sbinfo->xa_cache, &info->xattrs, new_xattr))
break;
+
+ if (sbinfo->max_inodes)
+ ispace -= simple_xattr_space(new_xattr->name, new_xattr->size);
retain_and_null_ptr(new_xattr);
}
@@ -4284,8 +4275,8 @@ static int shmem_initxattrs(struct inode *inode,
}
return -ENOMEM;
}
+ WARN_ON(ispace);
- smp_store_release(&info->xattrs, no_free_ptr(xattrs));
return 0;
}
@@ -4293,15 +4284,11 @@ static int shmem_xattr_handler_get(const struct xattr_handler *handler,
struct dentry *unused, struct inode *inode,
const char *name, void *buffer, size_t size)
{
+ struct shmem_sb_info *sbinfo = SHMEM_SB(inode->i_sb);
struct shmem_inode_info *info = SHMEM_I(inode);
- struct simple_xattrs *xattrs;
-
- xattrs = READ_ONCE(info->xattrs);
- if (!xattrs)
- return -ENODATA;
name = xattr_full_name(handler, name);
- return simple_xattr_get(xattrs, name, buffer, size);
+ return simple_xattr_get(&sbinfo->xa_cache, &info->xattrs, name, buffer, size);
}
static int shmem_xattr_handler_set(const struct xattr_handler *handler,
@@ -4312,16 +4299,11 @@ static int shmem_xattr_handler_set(const struct xattr_handler *handler,
{
struct shmem_inode_info *info = SHMEM_I(inode);
struct shmem_sb_info *sbinfo = SHMEM_SB(inode->i_sb);
- struct simple_xattrs *xattrs;
struct simple_xattr *old_xattr;
size_t ispace = 0;
name = xattr_full_name(handler, name);
- xattrs = simple_xattrs_lazy_alloc(&info->xattrs, value, flags);
- if (IS_ERR_OR_NULL(xattrs))
- return PTR_ERR(xattrs);
-
if (value && sbinfo->max_inodes) {
ispace = simple_xattr_space(name, size);
raw_spin_lock(&sbinfo->stat_lock);
@@ -4334,7 +4316,7 @@ static int shmem_xattr_handler_set(const struct xattr_handler *handler,
return -ENOSPC;
}
- old_xattr = simple_xattr_set(xattrs, name, value, size, flags);
+ old_xattr = simple_xattr_set(&sbinfo->xa_cache, &info->xattrs, name, value, size, flags);
if (!IS_ERR(old_xattr)) {
ispace = 0;
if (old_xattr && sbinfo->max_inodes)
@@ -4382,8 +4364,7 @@ static ssize_t shmem_listxattr(struct dentry *dentry, char *buffer, size_t size)
{
struct shmem_inode_info *info = SHMEM_I(d_inode(dentry));
- return simple_xattr_list(d_inode(dentry), READ_ONCE(info->xattrs),
- buffer, size);
+ return simple_xattr_list(d_inode(dentry), &info->xattrs, buffer, size);
}
#endif /* CONFIG_TMPFS_XATTR */
@@ -4984,6 +4965,9 @@ static void shmem_put_super(struct super_block *sb)
free_percpu(sbinfo->ino_batch);
percpu_counter_destroy(&sbinfo->used_blocks);
mpol_put(sbinfo->mpol);
+#ifdef CONFIG_TMPFS_XATTR
+ simple_xattr_cache_cleanup(&sbinfo->xa_cache);
+#endif
kfree(sbinfo);
sb->s_fs_info = NULL;
}
diff --git a/mm/vmscan.c b/mm/vmscan.c
index bd1b1aa..67231d3 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -1449,7 +1449,7 @@ static unsigned int shrink_folio_list(struct list_head *folio_list,
* is possible for a folio to have the dirty flag set,
* but it is actually clean (all its buffers are clean).
* This happens if the buffers were written out directly,
- * with submit_bh(). ext3 will do this, as well as
+ * with bh_submit(). ext3 will do this, as well as
* the blockdev mapping. filemap_release_folio() will
* discover that cleanness and will drop the buffers
* and mark the folio clean - it can be freed.
diff --git a/net/bridge/netfilter/ebt_dnat.c b/net/bridge/netfilter/ebt_dnat.c
index 3fda71a..73f185c 100644
--- a/net/bridge/netfilter/ebt_dnat.c
+++ b/net/bridge/netfilter/ebt_dnat.c
@@ -39,7 +39,9 @@ ebt_dnat_tg(struct sk_buff *skb, const struct xt_action_param *par)
dev = xt_in(par);
break;
case NF_BR_PRE_ROUTING:
- dev = br_port_get_rcu(xt_in(par))->br->dev;
+ dev = netdev_master_upper_dev_get_rcu(xt_in(par));
+ if (!dev) /* bridge port removed? */
+ return EBT_DROP;
break;
default:
dev = NULL;
diff --git a/net/bridge/netfilter/ebt_redirect.c b/net/bridge/netfilter/ebt_redirect.c
index 3077905..83486cd 100644
--- a/net/bridge/netfilter/ebt_redirect.c
+++ b/net/bridge/netfilter/ebt_redirect.c
@@ -24,12 +24,18 @@ ebt_redirect_tg(struct sk_buff *skb, const struct xt_action_param *par)
if (skb_ensure_writable(skb, 0))
return EBT_DROP;
- if (xt_hooknum(par) != NF_BR_BROUTING)
- /* rcu_read_lock()ed by nf_hook_thresh */
- ether_addr_copy(eth_hdr(skb)->h_dest,
- br_port_get_rcu(xt_in(par))->br->dev->dev_addr);
- else
+ if (xt_hooknum(par) != NF_BR_BROUTING) {
+ const struct net_device *dev;
+
+ dev = netdev_master_upper_dev_get_rcu(xt_in(par));
+ if (!dev)
+ return EBT_DROP;
+
+ ether_addr_copy(eth_hdr(skb)->h_dest, dev->dev_addr);
+ } else {
ether_addr_copy(eth_hdr(skb)->h_dest, xt_in(par)->dev_addr);
+ }
+
skb->pkt_type = PACKET_HOST;
return info->target;
}
diff --git a/net/bridge/netfilter/nft_meta_bridge.c b/net/bridge/netfilter/nft_meta_bridge.c
index 7763e78..219c406 100644
--- a/net/bridge/netfilter/nft_meta_bridge.c
+++ b/net/bridge/netfilter/nft_meta_bridge.c
@@ -64,6 +64,8 @@ static void nft_meta_bridge_get_eval(const struct nft_expr *expr,
if (!br_dev)
goto err;
+ /* ETH_ALEN (6) is shorter than the destination register span (8) */
+ dest[1] = 0;
memcpy(dest, br_dev->dev_addr, ETH_ALEN);
return;
default:
diff --git a/net/core/gro.c b/net/core/gro.c
index a847539..35f2f70 100644
--- a/net/core/gro.c
+++ b/net/core/gro.c
@@ -232,6 +232,11 @@ int skb_gro_receive_list(struct sk_buff *p, struct sk_buff *skb)
if (unlikely(p->len + skb->len >= 65536))
return -E2BIG;
+ if (!pskb_may_pull(skb, skb_gro_offset(skb))) {
+ NAPI_GRO_CB(skb)->flush = 1;
+ return -ENOMEM;
+ }
+
if (NAPI_GRO_CB(p)->last == p)
skb_shinfo(p)->frag_list = skb;
else
diff --git a/net/core/netdev-genl.c b/net/core/netdev-genl.c
index b8f6076d..119eaa6 100644
--- a/net/core/netdev-genl.c
+++ b/net/core/netdev-genl.c
@@ -1095,8 +1095,6 @@ int netdev_nl_bind_rx_doit(struct sk_buff *skb, struct genl_info *info)
genlmsg_end(rsp, hdr);
err = genlmsg_reply(rsp, info);
- if (err)
- goto err_unbind;
bitmap_free(rxq_bitmap);
@@ -1104,7 +1102,7 @@ int netdev_nl_bind_rx_doit(struct sk_buff *skb, struct genl_info *info)
mutex_unlock(&priv->lock);
- return 0;
+ return err < 0 ? err : 0;
err_unbind:
net_devmem_unbind_dmabuf(binding);
diff --git a/net/core/skbuff.c b/net/core/skbuff.c
index c02f0a5..8eab8eb 100644
--- a/net/core/skbuff.c
+++ b/net/core/skbuff.c
@@ -5450,7 +5450,7 @@ int skb_cow_data(struct sk_buff *skb, int tailbits, struct sk_buff **trailer)
}
EXPORT_SYMBOL_GPL(skb_cow_data);
-static void sock_rmem_free(struct sk_buff *skb)
+void sock_rmem_free(struct sk_buff *skb)
{
struct sock *sk = skb->sk;
@@ -5459,8 +5459,8 @@ static void sock_rmem_free(struct sk_buff *skb)
static void skb_set_err_queue(struct sk_buff *skb)
{
- /* pkt_type of skbs received on local sockets is never PACKET_OUTGOING.
- * So, it is safe to (mis)use it to mark skbs on the error queue.
+ /* The error-queue test in skb_is_err_queue() matches this marker
+ * with the sock_rmem_free destructor installed by sock_queue_err_skb().
*/
skb->pkt_type = PACKET_OUTGOING;
BUILD_BUG_ON(PACKET_OUTGOING == 0);
diff --git a/net/core/sock.c b/net/core/sock.c
index d097025..cab041b 100644
--- a/net/core/sock.c
+++ b/net/core/sock.c
@@ -1465,6 +1465,11 @@ int sk_setsockopt(struct sock *sk, int level, int optname,
case SO_ATTACH_FILTER: {
struct sock_fprog fprog;
+ if (sk_is_tcp(sk) &&
+ !sockopt_ns_capable(sock_net(sk)->user_ns, CAP_NET_ADMIN)) {
+ ret = -EPERM;
+ break;
+ }
ret = copy_bpf_fprog_from_user(&fprog, optval, optlen);
if (!ret)
ret = sk_attach_filter(&fprog, sk);
diff --git a/net/ipv4/esp4.c b/net/ipv4/esp4.c
index 513c821..dfc81ee 100644
--- a/net/ipv4/esp4.c
+++ b/net/ipv4/esp4.c
@@ -96,7 +96,7 @@ static inline struct scatterlist *esp_req_sg(struct crypto_aead *aead,
__alignof__(struct scatterlist));
}
-static void esp_ssg_unref(struct xfrm_state *x, void *tmp, struct sk_buff *skb)
+static void esp_ssg_unref(struct xfrm_state *x, void *tmp, struct sk_buff *skb, bool already_unref)
{
struct crypto_aead *aead = x->data;
int extralen = 0;
@@ -113,10 +113,13 @@ static void esp_ssg_unref(struct xfrm_state *x, void *tmp, struct sk_buff *skb)
/* Unref skb_frag_pages in the src scatterlist if necessary.
* Skip the first sg which comes from skb->data.
*/
- if (req->src != req->dst)
- for (sg = sg_next(req->src); sg; sg = sg_next(sg))
+ if (already_unref || req->src != req->dst) {
+ struct scatterlist *src = already_unref ? esp_req_sg(aead, req) : req->src;
+
+ for (sg = sg_next(src); sg; sg = sg_next(sg))
skb_page_unref(page_to_netmem(sg_page(sg)),
skb->pp_recycle);
+ }
}
#ifdef CONFIG_INET_ESPINTCP
@@ -220,7 +223,7 @@ static void esp_output_done(void *data, int err)
}
tmp = ESP_SKB_CB(skb)->tmp;
- esp_ssg_unref(x, tmp, skb);
+ esp_ssg_unref(x, tmp, skb, false);
kfree(tmp);
if (xo && (xo->flags & XFRM_DEV_RESUME)) {
@@ -569,8 +572,10 @@ int esp_output_tail(struct xfrm_state *x, struct sk_buff *skb, struct esp_info *
err = skb_to_sgvec(skb, dsg,
(unsigned char *)esph - skb->data,
assoclen + ivlen + esp->clen + alen);
- if (unlikely(err < 0))
+ if (unlikely(err < 0)) {
+ esp_ssg_unref(x, tmp, skb, true);
goto error_free;
+ }
}
if ((x->props.flags & XFRM_STATE_ESN))
@@ -602,7 +607,7 @@ int esp_output_tail(struct xfrm_state *x, struct sk_buff *skb, struct esp_info *
}
if (sg != dsg)
- esp_ssg_unref(x, tmp, skb);
+ esp_ssg_unref(x, tmp, skb, false);
if (!err && x->encap && x->encap->encap_type == TCP_ENCAP_ESPINTCP)
err = esp_output_tail_tcp(x, skb);
diff --git a/net/ipv4/inet_fragment.c b/net/ipv4/inet_fragment.c
index 3937709..1127519 100644
--- a/net/ipv4/inet_fragment.c
+++ b/net/ipv4/inet_fragment.c
@@ -328,6 +328,9 @@ void inet_frag_queue_flush(struct inet_frag_queue *q,
reason = reason ?: SKB_DROP_REASON_FRAG_REASM_TIMEOUT;
sum = inet_frag_rbtree_purge(&q->rb_fragments, reason);
sub_frag_mem_limit(q->fqdir, sum);
+ q->rb_fragments = RB_ROOT;
+ q->fragments_tail = NULL;
+ q->last_run_head = NULL;
}
EXPORT_SYMBOL(inet_frag_queue_flush);
diff --git a/net/ipv4/ip_fragment.c b/net/ipv4/ip_fragment.c
index 56b0f73..c790d2f 100644
--- a/net/ipv4/ip_fragment.c
+++ b/net/ipv4/ip_fragment.c
@@ -250,9 +250,6 @@ static int ip_frag_reinit(struct ipq *qp)
qp->q.flags = 0;
qp->q.len = 0;
qp->q.meat = 0;
- qp->q.rb_fragments = RB_ROOT;
- qp->q.fragments_tail = NULL;
- qp->q.last_run_head = NULL;
qp->iif = 0;
qp->ecn = 0;
diff --git a/net/ipv4/netfilter/arp_tables.c b/net/ipv4/netfilter/arp_tables.c
index ad22596..0ea513b 100644
--- a/net/ipv4/netfilter/arp_tables.c
+++ b/net/ipv4/netfilter/arp_tables.c
@@ -702,14 +702,12 @@ static int copy_entries_to_user(unsigned int total_size,
const struct xt_entry_target *t;
e = loc_cpu_entry + off;
- if (copy_to_user(userptr + off, e, sizeof(*e))) {
- ret = -EFAULT;
- goto free_counters;
- }
- if (copy_to_user(userptr + off
+ if (copy_to_user(userptr + off, e,
+ offsetof(struct arpt_entry, counters)) ||
+ copy_to_user(userptr + off
+ offsetof(struct arpt_entry, counters),
&counters[num],
- sizeof(counters[num])) != 0) {
+ sizeof(counters[num]))) {
ret = -EFAULT;
goto free_counters;
}
@@ -1327,9 +1325,8 @@ static int compat_copy_entry_to_user(struct arpt_entry *e, void __user **dstptr,
origsize = *size;
ce = *dstptr;
- if (copy_to_user(ce, e, sizeof(struct arpt_entry)) != 0 ||
- copy_to_user(&ce->counters, &counters[i],
- sizeof(counters[i])) != 0)
+ if (copy_to_user(ce, e, offsetof(struct compat_arpt_entry, counters)) ||
+ copy_to_user(&ce->counters, &counters[i], sizeof(counters[i])))
return -EFAULT;
*dstptr += sizeof(struct compat_arpt_entry);
diff --git a/net/ipv4/netfilter/ip_tables.c b/net/ipv4/netfilter/ip_tables.c
index 5cbdb08..ca8ff0a 100644
--- a/net/ipv4/netfilter/ip_tables.c
+++ b/net/ipv4/netfilter/ip_tables.c
@@ -832,14 +832,12 @@ copy_entries_to_user(unsigned int total_size,
const struct xt_entry_target *t;
e = loc_cpu_entry + off;
- if (copy_to_user(userptr + off, e, sizeof(*e))) {
- ret = -EFAULT;
- goto free_counters;
- }
- if (copy_to_user(userptr + off
+ if (copy_to_user(userptr + off, e,
+ offsetof(struct ipt_entry, counters)) ||
+ copy_to_user(userptr + off
+ offsetof(struct ipt_entry, counters),
&counters[num],
- sizeof(counters[num])) != 0) {
+ sizeof(counters[num]))) {
ret = -EFAULT;
goto free_counters;
}
@@ -1228,9 +1226,8 @@ compat_copy_entry_to_user(struct ipt_entry *e, void __user **dstptr,
origsize = *size;
ce = *dstptr;
- if (copy_to_user(ce, e, sizeof(struct ipt_entry)) != 0 ||
- copy_to_user(&ce->counters, &counters[i],
- sizeof(counters[i])) != 0)
+ if (copy_to_user(ce, e, offsetof(struct compat_ipt_entry, counters)) ||
+ copy_to_user(&ce->counters, &counters[i], sizeof(counters[i])))
return -EFAULT;
*dstptr += sizeof(struct compat_ipt_entry);
diff --git a/net/ipv4/netfilter/nf_nat_h323.c b/net/ipv4/netfilter/nf_nat_h323.c
index faee20a..10e1b08 100644
--- a/net/ipv4/netfilter/nf_nat_h323.c
+++ b/net/ipv4/netfilter/nf_nat_h323.c
@@ -555,6 +555,8 @@ static void __exit nf_nat_h323_fini(void)
nf_ct_helper_expectfn_unregister(&q931_nat);
nf_ct_helper_expectfn_unregister(&callforwarding_nat);
synchronize_rcu();
+ nf_ct_helper_expectfn_destroy(&q931_nat);
+ nf_ct_helper_expectfn_destroy(&callforwarding_nat);
}
/****************************************************************************/
diff --git a/net/ipv4/netfilter/nft_fib_ipv4.c b/net/ipv4/netfilter/nft_fib_ipv4.c
index 9d0c6d7..177d738 100644
--- a/net/ipv4/netfilter/nft_fib_ipv4.c
+++ b/net/ipv4/netfilter/nft_fib_ipv4.c
@@ -128,7 +128,7 @@ void nft_fib4_eval(const struct nft_expr *expr, struct nft_regs *regs,
fl4.saddr = get_saddr(iph->daddr);
}
- *dest = 0;
+ nft_fib_store_result(dest, priv, NULL);
if (fib_lookup(nft_net(pkt), &fl4, &res, FIB_LOOKUP_IGNORE_LINKSTATE))
return;
diff --git a/net/ipv6/addrconf.c b/net/ipv6/addrconf.c
index bb84a78..c9e5d3e 100644
--- a/net/ipv6/addrconf.c
+++ b/net/ipv6/addrconf.c
@@ -1265,6 +1265,7 @@ static void
cleanup_prefix_route(struct inet6_ifaddr *ifp, unsigned long expires,
bool del_rt, bool del_peer)
{
+ struct net *net = dev_net(ifp->idev->dev);
struct fib6_table *table;
struct fib6_info *f6i;
@@ -1273,9 +1274,10 @@ cleanup_prefix_route(struct inet6_ifaddr *ifp, unsigned long expires,
ifp->idev->dev, 0, RTF_DEFAULT, true);
if (f6i) {
if (del_rt)
- ip6_del_rt(dev_net(ifp->idev->dev), f6i, false);
+ ip6_del_rt(net, f6i, false);
else {
- if (!(f6i->fib6_flags & RTF_EXPIRES)) {
+ if (f6i != net->ipv6.fib6_null_entry &&
+ !(f6i->fib6_flags & RTF_EXPIRES)) {
table = f6i->fib6_table;
spin_lock_bh(&table->tb6_lock);
diff --git a/net/ipv6/esp6.c b/net/ipv6/esp6.c
index 57481e42..296b579 100644
--- a/net/ipv6/esp6.c
+++ b/net/ipv6/esp6.c
@@ -113,7 +113,7 @@ static inline struct scatterlist *esp_req_sg(struct crypto_aead *aead,
__alignof__(struct scatterlist));
}
-static void esp_ssg_unref(struct xfrm_state *x, void *tmp, struct sk_buff *skb)
+static void esp_ssg_unref(struct xfrm_state *x, void *tmp, struct sk_buff *skb, bool already_unref)
{
struct crypto_aead *aead = x->data;
int extralen = 0;
@@ -130,10 +130,13 @@ static void esp_ssg_unref(struct xfrm_state *x, void *tmp, struct sk_buff *skb)
/* Unref skb_frag_pages in the src scatterlist if necessary.
* Skip the first sg which comes from skb->data.
*/
- if (req->src != req->dst)
- for (sg = sg_next(req->src); sg; sg = sg_next(sg))
+ if (already_unref || req->src != req->dst) {
+ struct scatterlist *src = already_unref ? esp_req_sg(aead, req) : req->src;
+
+ for (sg = sg_next(src); sg; sg = sg_next(sg))
skb_page_unref(page_to_netmem(sg_page(sg)),
skb->pp_recycle);
+ }
}
#ifdef CONFIG_INET6_ESPINTCP
@@ -254,7 +257,7 @@ static void esp_output_done(void *data, int err)
}
tmp = ESP_SKB_CB(skb)->tmp;
- esp_ssg_unref(x, tmp, skb);
+ esp_ssg_unref(x, tmp, skb, false);
kfree(tmp);
esp_output_encap_csum(skb);
@@ -600,8 +603,10 @@ int esp6_output_tail(struct xfrm_state *x, struct sk_buff *skb, struct esp_info
err = skb_to_sgvec(skb, dsg,
(unsigned char *)esph - skb->data,
assoclen + ivlen + esp->clen + alen);
- if (unlikely(err < 0))
+ if (unlikely(err < 0)) {
+ esp_ssg_unref(x, tmp, skb, true);
goto error_free;
+ }
}
if ((x->props.flags & XFRM_STATE_ESN))
@@ -634,7 +639,7 @@ int esp6_output_tail(struct xfrm_state *x, struct sk_buff *skb, struct esp_info
}
if (sg != dsg)
- esp_ssg_unref(x, tmp, skb);
+ esp_ssg_unref(x, tmp, skb, false);
if (!err && x->encap && x->encap->encap_type == TCP_ENCAP_ESPINTCP)
err = esp_output_tail_tcp(x, skb);
diff --git a/net/ipv6/ip6_vti.c b/net/ipv6/ip6_vti.c
index df793c8..d871cab 100644
--- a/net/ipv6/ip6_vti.c
+++ b/net/ipv6/ip6_vti.c
@@ -106,6 +106,7 @@ vti6_tnl_lookup(struct net *net, const struct in6_addr *remote,
hash = HASH(&any, local);
for_each_vti6_tunnel_rcu(ip6n->tnls_r_l[hash]) {
if (ipv6_addr_equal(local, &t->parms.laddr) &&
+ ipv6_addr_any(&t->parms.raddr) &&
(t->dev->flags & IFF_UP))
return t;
}
@@ -113,6 +114,7 @@ vti6_tnl_lookup(struct net *net, const struct in6_addr *remote,
hash = HASH(remote, &any);
for_each_vti6_tunnel_rcu(ip6n->tnls_r_l[hash]) {
if (ipv6_addr_equal(remote, &t->parms.raddr) &&
+ ipv6_addr_any(&t->parms.laddr) &&
(t->dev->flags & IFF_UP))
return t;
}
@@ -1159,6 +1161,7 @@ static int __net_init vti6_init_net(struct net *net)
goto err_alloc_dev;
dev_net_set(ip6n->fb_tnl_dev, net);
ip6n->fb_tnl_dev->rtnl_link_ops = &vti6_link_ops;
+ ip6n->fb_tnl_dev->netns_immutable = true;
err = vti6_fb_tnl_dev_init(ip6n->fb_tnl_dev);
if (err < 0)
diff --git a/net/ipv6/netfilter/ip6_tables.c b/net/ipv6/netfilter/ip6_tables.c
index 9d9c376..e34d5ba 100644
--- a/net/ipv6/netfilter/ip6_tables.c
+++ b/net/ipv6/netfilter/ip6_tables.c
@@ -848,14 +848,12 @@ copy_entries_to_user(unsigned int total_size,
const struct xt_entry_target *t;
e = loc_cpu_entry + off;
- if (copy_to_user(userptr + off, e, sizeof(*e))) {
- ret = -EFAULT;
- goto free_counters;
- }
- if (copy_to_user(userptr + off
+ if (copy_to_user(userptr + off, e,
+ offsetof(struct ip6t_entry, counters)) ||
+ copy_to_user(userptr + off
+ offsetof(struct ip6t_entry, counters),
&counters[num],
- sizeof(counters[num])) != 0) {
+ sizeof(counters[num]))) {
ret = -EFAULT;
goto free_counters;
}
@@ -1244,9 +1242,8 @@ compat_copy_entry_to_user(struct ip6t_entry *e, void __user **dstptr,
origsize = *size;
ce = *dstptr;
- if (copy_to_user(ce, e, sizeof(struct ip6t_entry)) != 0 ||
- copy_to_user(&ce->counters, &counters[i],
- sizeof(counters[i])) != 0)
+ if (copy_to_user(ce, e, offsetof(struct compat_ip6t_entry, counters)) ||
+ copy_to_user(&ce->counters, &counters[i], sizeof(counters[i])))
return -EFAULT;
*dstptr += sizeof(struct compat_ip6t_entry);
diff --git a/net/ipv6/netfilter/nft_fib_ipv6.c b/net/ipv6/netfilter/nft_fib_ipv6.c
index 2dbe447..b9ad7ca 100644
--- a/net/ipv6/netfilter/nft_fib_ipv6.c
+++ b/net/ipv6/netfilter/nft_fib_ipv6.c
@@ -239,7 +239,7 @@ void nft_fib6_eval(const struct nft_expr *expr, struct nft_regs *regs,
lookup_flags = nft_fib6_flowi_init(&fl6, priv, pkt, oif, iph);
- *dest = 0;
+ nft_fib_store_result(dest, priv, NULL);
ret = nft_fib6_lookup(nft_net(pkt), &fl6, &res, lookup_flags);
if (ret || res.fib6_flags & (RTF_REJECT | RTF_ANYCAST | RTF_LOCAL))
return;
diff --git a/net/ipv6/sit.c b/net/ipv6/sit.c
index 201347b..b41e231 100644
--- a/net/ipv6/sit.c
+++ b/net/ipv6/sit.c
@@ -961,6 +961,7 @@ static netdev_tx_t ipip6_tunnel_xmit(struct sk_buff *skb,
ip_rt_put(rt);
goto tx_error;
}
+ iph6 = ipv6_hdr(skb);
if (df) {
mtu = dst4_mtu(&rt->dst) - t_hlen;
diff --git a/net/netfilter/nf_conntrack_helper.c b/net/netfilter/nf_conntrack_helper.c
index 17e971b..2c5a717 100644
--- a/net/netfilter/nf_conntrack_helper.c
+++ b/net/netfilter/nf_conntrack_helper.c
@@ -283,6 +283,25 @@ void nf_ct_helper_expectfn_unregister(struct nf_ct_helper_expectfn *n)
}
EXPORT_SYMBOL_GPL(nf_ct_helper_expectfn_unregister);
+static bool expect_iter_expectfn(struct nf_conntrack_expect *exp, void *data)
+{
+ const struct nf_ct_helper_expectfn *n = data;
+
+ /* Relies on registered expectfn descriptors having unique ->expectfn
+ * pointers, which holds for the in-tree NAT helpers.
+ */
+ return exp->expectfn == n->expectfn;
+}
+
+/* Destroy expectations still pointing at @n->expectfn; call after the
+ * caller's RCU grace period so none outlives the (often modular) callback.
+ */
+void nf_ct_helper_expectfn_destroy(const struct nf_ct_helper_expectfn *n)
+{
+ nf_ct_expect_iterate_destroy(expect_iter_expectfn, (void *)n);
+}
+EXPORT_SYMBOL_GPL(nf_ct_helper_expectfn_destroy);
+
/* Caller should hold the rcu lock */
struct nf_ct_helper_expectfn *
nf_ct_helper_expectfn_find_by_name(const char *name)
diff --git a/net/netfilter/nf_dup_netdev.c b/net/netfilter/nf_dup_netdev.c
index 3b0a70e1..3d88ef9 100644
--- a/net/netfilter/nf_dup_netdev.c
+++ b/net/netfilter/nf_dup_netdev.c
@@ -74,16 +74,18 @@ int nft_fwd_dup_netdev_offload(struct nft_offload_ctx *ctx,
struct flow_action_entry *entry;
struct net_device *dev;
- /* nft_flow_rule_destroy() releases the reference on this device. */
dev = dev_get_by_index(ctx->net, oif);
if (!dev)
return -EOPNOTSUPP;
entry = nft_flow_action_entry_next(ctx, flow);
- if (!entry)
+ if (!entry) {
+ dev_put(dev);
return -E2BIG;
+ }
entry->id = id;
+ /* nft_flow_rule_destroy() releases the reference on this device. */
entry->dev = dev;
return 0;
diff --git a/net/netfilter/nf_log_syslog.c b/net/netfilter/nf_log_syslog.c
index 7a8952b..e37b09b 100644
--- a/net/netfilter/nf_log_syslog.c
+++ b/net/netfilter/nf_log_syslog.c
@@ -815,8 +815,8 @@ static void dump_mac_header(struct nf_log_buf *m,
fallback:
nf_log_buf_add(m, "MAC=");
- if (dev->hard_header_len &&
- skb->mac_header != skb->network_header) {
+ if (dev->hard_header_len && skb_mac_header_was_set(skb) &&
+ skb_mac_header_len(skb) != 0) {
const unsigned char *p = skb_mac_header(skb);
unsigned int i;
diff --git a/net/netfilter/nf_nat_core.c b/net/netfilter/nf_nat_core.c
index 74ec224..2bbf516 100644
--- a/net/netfilter/nf_nat_core.c
+++ b/net/netfilter/nf_nat_core.c
@@ -1341,6 +1341,7 @@ static int __init nf_nat_init(void)
RCU_INIT_POINTER(nf_nat_hook, NULL);
nf_ct_helper_expectfn_unregister(&follow_master_nat);
synchronize_net();
+ nf_ct_helper_expectfn_destroy(&follow_master_nat);
unregister_pernet_subsys(&nat_net_ops);
kvfree(nf_nat_bysource);
}
@@ -1358,6 +1359,7 @@ static void __exit nf_nat_cleanup(void)
RCU_INIT_POINTER(nf_nat_hook, NULL);
synchronize_net();
+ nf_ct_helper_expectfn_destroy(&follow_master_nat);
kvfree(nf_nat_bysource);
unregister_pernet_subsys(&nat_net_ops);
}
diff --git a/net/netfilter/nf_nat_sip.c b/net/netfilter/nf_nat_sip.c
index 9fbfc6b..00838c0 100644
--- a/net/netfilter/nf_nat_sip.c
+++ b/net/netfilter/nf_nat_sip.c
@@ -655,6 +655,7 @@ static void __exit nf_nat_sip_fini(void)
RCU_INIT_POINTER(nf_nat_sip_hooks, NULL);
nf_ct_helper_expectfn_unregister(&sip_nat);
synchronize_rcu();
+ nf_ct_helper_expectfn_destroy(&sip_nat);
}
static const struct nf_nat_sip_hooks sip_hooks = {
diff --git a/net/netfilter/nfnetlink_log.c b/net/netfilter/nfnetlink_log.c
index 2439cbb..fa36575 100644
--- a/net/netfilter/nfnetlink_log.c
+++ b/net/netfilter/nfnetlink_log.c
@@ -451,6 +451,23 @@ static int nfulnl_put_bridge(struct nfulnl_instance *inst, const struct sk_buff
return -1;
}
+#if IS_ENABLED(CONFIG_BRIDGE_NETFILTER)
+static int nflog_put_master_ifindex(struct sk_buff *nlskb, int attr,
+ const struct net_device *dev)
+{
+ const struct net_device *upper;
+
+ if (dev && !netif_is_bridge_port(dev))
+ return 0;
+
+ upper = netdev_master_upper_dev_get_rcu((struct net_device *)dev);
+ if (upper && nla_put_be32(nlskb, attr, htonl(upper->ifindex)))
+ return -EMSGSIZE;
+
+ return 0;
+}
+#endif
+
/* This is an inline function, we don't really care about a long
* list of arguments */
static inline int
@@ -505,8 +522,7 @@ __build_packet_message(struct nfnl_log_net *log,
/* rcu_read_lock()ed by nf_hook_thresh or
* nf_log_packet.
*/
- nla_put_be32(inst->skb, NFULA_IFINDEX_INDEV,
- htonl(br_port_get_rcu(indev)->br->dev->ifindex)))
+ nflog_put_master_ifindex(inst->skb, NFULA_IFINDEX_INDEV, indev))
goto nla_put_failure;
} else {
int physinif;
@@ -542,8 +558,7 @@ __build_packet_message(struct nfnl_log_net *log,
/* rcu_read_lock()ed by nf_hook_thresh or
* nf_log_packet.
*/
- nla_put_be32(inst->skb, NFULA_IFINDEX_OUTDEV,
- htonl(br_port_get_rcu(outdev)->br->dev->ifindex)))
+ nflog_put_master_ifindex(inst->skb, NFULA_IFINDEX_OUTDEV, outdev))
goto nla_put_failure;
} else {
struct net_device *physoutdev;
diff --git a/net/netfilter/nfnetlink_queue.c b/net/netfilter/nfnetlink_queue.c
index 60ab88d..c5e29fe 100644
--- a/net/netfilter/nfnetlink_queue.c
+++ b/net/netfilter/nfnetlink_queue.c
@@ -440,10 +440,47 @@ static bool nf_ct_drop_unconfirmed(const struct nf_queue_entry *entry, bool *is_
return false;
}
+static bool nf_bridge_port_valid(const struct net_device *dev)
+{
+ if (!dev)
+ return true;
+
+ return netif_is_bridge_port(dev);
+}
+
+/* queued skbs leave rcu protection. We bump device refcount so that
+ * the device cannot go away. However, while packet was out the port
+ * could have been removed from the bridge.
+ *
+ * Ensure in+outdev are still part of a bridge at reinject time.
+ *
+ * The device rx_handler_data could even be pointing at data that is
+ * not a net_bridge_port structure.
+ */
+static bool nf_bridge_ports_valid(const struct nf_queue_entry *entry)
+{
+#if IS_ENABLED(CONFIG_BRIDGE_NETFILTER)
+ if (!nf_bridge_port_valid(entry->physin) ||
+ !nf_bridge_port_valid(entry->physout))
+ return false;
+#endif
+ if (entry->state.pf != PF_BRIDGE)
+ return true;
+
+ if (!nf_bridge_port_valid(entry->state.in) ||
+ !nf_bridge_port_valid(entry->state.out))
+ return false;
+
+ return true;
+}
+
static void nfqnl_reinject(struct nf_queue_entry *entry, unsigned int verdict)
{
const struct nf_ct_hook *ct_hook;
+ if (!nf_bridge_ports_valid(entry))
+ verdict = NF_DROP;
+
if (verdict == NF_ACCEPT ||
verdict == NF_REPEAT ||
verdict == NF_STOP) {
@@ -636,6 +673,23 @@ static int nf_queue_checksum_help(struct sk_buff *entskb)
return skb_checksum_help(entskb);
}
+#if IS_ENABLED(CONFIG_BRIDGE_NETFILTER)
+static int nfqnl_put_master_ifindex(struct sk_buff *nlskb, int attr,
+ const struct net_device *dev)
+{
+ const struct net_device *upper;
+
+ if (dev && !netif_is_bridge_port(dev))
+ return 0;
+
+ upper = netdev_master_upper_dev_get_rcu((struct net_device *)dev);
+ if (upper && nla_put_be32(nlskb, attr, htonl(upper->ifindex)))
+ return -EMSGSIZE;
+
+ return 0;
+}
+#endif
+
static struct sk_buff *
nfqnl_build_packet_message(struct net *net, struct nfqnl_instance *queue,
struct nf_queue_entry *entry,
@@ -771,10 +825,7 @@ nfqnl_build_packet_message(struct net *net, struct nfqnl_instance *queue,
* netfilter_bridge) */
if (nla_put_be32(skb, NFQA_IFINDEX_PHYSINDEV,
htonl(indev->ifindex)) ||
- /* this is the bridge group "brX" */
- /* rcu_read_lock()ed by __nf_queue */
- nla_put_be32(skb, NFQA_IFINDEX_INDEV,
- htonl(br_port_get_rcu(indev)->br->dev->ifindex)))
+ nfqnl_put_master_ifindex(skb, NFQA_IFINDEX_INDEV, indev))
goto nla_put_failure;
} else {
int physinif;
@@ -805,10 +856,7 @@ nfqnl_build_packet_message(struct net *net, struct nfqnl_instance *queue,
* netfilter_bridge) */
if (nla_put_be32(skb, NFQA_IFINDEX_PHYSOUTDEV,
htonl(outdev->ifindex)) ||
- /* this is the bridge group "brX" */
- /* rcu_read_lock()ed by __nf_queue */
- nla_put_be32(skb, NFQA_IFINDEX_OUTDEV,
- htonl(br_port_get_rcu(outdev)->br->dev->ifindex)))
+ nfqnl_put_master_ifindex(skb, NFQA_IFINDEX_OUTDEV, outdev))
goto nla_put_failure;
} else {
int physoutif;
diff --git a/net/netfilter/nft_exthdr.c b/net/netfilter/nft_exthdr.c
index e6a07c0..d3fc796 100644
--- a/net/netfilter/nft_exthdr.c
+++ b/net/netfilter/nft_exthdr.c
@@ -532,6 +532,9 @@ static int nft_exthdr_init(const struct nft_ctx *ctx,
return err;
}
+ if ((flags & NFT_EXTHDR_F_PRESENT) && len != 1)
+ return -EINVAL;
+
priv->type = nla_get_u8(tb[NFTA_EXTHDR_TYPE]);
priv->offset = offset;
priv->len = len;
diff --git a/net/netfilter/nft_fib.c b/net/netfilter/nft_fib.c
index 327a5f3..a1632e3 100644
--- a/net/netfilter/nft_fib.c
+++ b/net/netfilter/nft_fib.c
@@ -107,6 +107,12 @@ int nft_fib_init(const struct nft_ctx *ctx, const struct nft_expr *expr,
return -EINVAL;
}
+ if (priv->flags & NFTA_FIB_F_PRESENT) {
+ if (priv->result != NFT_FIB_RESULT_OIF)
+ return -EINVAL;
+ len = sizeof(u8);
+ }
+
err = nft_parse_register_store(ctx, tb[NFTA_FIB_DREG], &priv->dreg,
NULL, NFT_DATA_VALUE, len);
if (err < 0)
diff --git a/net/netlabel/netlabel_unlabeled.c b/net/netlabel/netlabel_unlabeled.c
index ca7a9e2..870e769 100644
--- a/net/netlabel/netlabel_unlabeled.c
+++ b/net/netlabel/netlabel_unlabeled.c
@@ -114,14 +114,14 @@ static struct genl_family netlbl_unlabel_gnl_family;
/* NetLabel Netlink attribute policy */
static const struct nla_policy netlbl_unlabel_genl_policy[NLBL_UNLABEL_A_MAX + 1] = {
[NLBL_UNLABEL_A_ACPTFLG] = { .type = NLA_U8 },
- [NLBL_UNLABEL_A_IPV6ADDR] = { .type = NLA_BINARY,
- .len = sizeof(struct in6_addr) },
- [NLBL_UNLABEL_A_IPV6MASK] = { .type = NLA_BINARY,
- .len = sizeof(struct in6_addr) },
- [NLBL_UNLABEL_A_IPV4ADDR] = { .type = NLA_BINARY,
- .len = sizeof(struct in_addr) },
- [NLBL_UNLABEL_A_IPV4MASK] = { .type = NLA_BINARY,
- .len = sizeof(struct in_addr) },
+ [NLBL_UNLABEL_A_IPV6ADDR] =
+ NLA_POLICY_EXACT_LEN(sizeof(struct in6_addr)),
+ [NLBL_UNLABEL_A_IPV6MASK] =
+ NLA_POLICY_EXACT_LEN(sizeof(struct in6_addr)),
+ [NLBL_UNLABEL_A_IPV4ADDR] =
+ NLA_POLICY_EXACT_LEN(sizeof(struct in_addr)),
+ [NLBL_UNLABEL_A_IPV4MASK] =
+ NLA_POLICY_EXACT_LEN(sizeof(struct in_addr)),
[NLBL_UNLABEL_A_IFACE] = { .type = NLA_NUL_STRING,
.len = IFNAMSIZ - 1 },
[NLBL_UNLABEL_A_SECCTX] = { .type = NLA_BINARY }
@@ -757,24 +757,14 @@ static int netlbl_unlabel_addrinfo_get(struct genl_info *info,
void **mask,
u32 *len)
{
- u32 addr_len;
-
if (info->attrs[NLBL_UNLABEL_A_IPV4ADDR] &&
info->attrs[NLBL_UNLABEL_A_IPV4MASK]) {
- addr_len = nla_len(info->attrs[NLBL_UNLABEL_A_IPV4ADDR]);
- if (addr_len != sizeof(struct in_addr) &&
- addr_len != nla_len(info->attrs[NLBL_UNLABEL_A_IPV4MASK]))
- return -EINVAL;
- *len = addr_len;
+ *len = sizeof(struct in_addr);
*addr = nla_data(info->attrs[NLBL_UNLABEL_A_IPV4ADDR]);
*mask = nla_data(info->attrs[NLBL_UNLABEL_A_IPV4MASK]);
return 0;
} else if (info->attrs[NLBL_UNLABEL_A_IPV6ADDR]) {
- addr_len = nla_len(info->attrs[NLBL_UNLABEL_A_IPV6ADDR]);
- if (addr_len != sizeof(struct in6_addr) &&
- addr_len != nla_len(info->attrs[NLBL_UNLABEL_A_IPV6MASK]))
- return -EINVAL;
- *len = addr_len;
+ *len = sizeof(struct in6_addr);
*addr = nla_data(info->attrs[NLBL_UNLABEL_A_IPV6ADDR]);
*mask = nla_data(info->attrs[NLBL_UNLABEL_A_IPV6MASK]);
return 0;
diff --git a/net/openvswitch/datapath.c b/net/openvswitch/datapath.c
index bbbde50..f016481 100644
--- a/net/openvswitch/datapath.c
+++ b/net/openvswitch/datapath.c
@@ -1316,6 +1316,7 @@ static int ovs_flow_cmd_set(struct sk_buff *skb, struct genl_info *info)
if (IS_ERR(reply)) {
error = PTR_ERR(reply);
+ reply = NULL;
goto err_unlock_ovs;
}
}
diff --git a/net/phonet/pn_dev.c b/net/phonet/pn_dev.c
index 86325b7..ad44831 100644
--- a/net/phonet/pn_dev.c
+++ b/net/phonet/pn_dev.c
@@ -108,7 +108,7 @@ static void phonet_device_destroy(struct net_device *dev)
for_each_set_bit(addr, pnd->addrs, 64)
phonet_address_notify(net, RTM_DELADDR, ifindex, addr);
- kfree(pnd);
+ kfree_rcu(pnd, rcu);
}
}
diff --git a/net/qrtr/af_qrtr.c b/net/qrtr/af_qrtr.c
index 7cec6a7..db82317 100644
--- a/net/qrtr/af_qrtr.c
+++ b/net/qrtr/af_qrtr.c
@@ -707,13 +707,13 @@ static void qrtr_port_remove(struct qrtr_sock *ipc)
if (port == QRTR_PORT_CTRL)
port = 0;
- __sock_put(&ipc->sk);
-
xa_erase(&qrtr_ports, port);
/* Ensure that if qrtr_port_lookup() did enter the RCU read section we
* wait for it to up increment the refcount */
synchronize_rcu();
+
+ __sock_put(&ipc->sk);
}
/* Assign port number to socket.
diff --git a/net/rds/ib_send.c b/net/rds/ib_send.c
index fcd04c2..d6be955 100644
--- a/net/rds/ib_send.c
+++ b/net/rds/ib_send.c
@@ -170,6 +170,8 @@ static struct rds_message *rds_ib_send_unmap_op(struct rds_ib_connection *ic,
break;
case IB_WR_ATOMIC_FETCH_AND_ADD:
case IB_WR_ATOMIC_CMP_AND_SWP:
+ case IB_WR_MASKED_ATOMIC_FETCH_AND_ADD:
+ case IB_WR_MASKED_ATOMIC_CMP_AND_SWP:
if (send->s_op) {
rm = container_of(send->s_op, struct rds_message, atomic);
rds_ib_send_unmap_atomic(ic, send->s_op, wc_status);
diff --git a/net/rds/info.c b/net/rds/info.c
index f1b2999..17061f6 100644
--- a/net/rds/info.c
+++ b/net/rds/info.c
@@ -235,7 +235,7 @@ int rds_info_getsockopt(struct socket *sock, int optname, char __user *optval,
out:
if (pages)
- unpin_user_pages(pages, nr_pages);
+ unpin_user_pages_dirty_lock(pages, nr_pages, true);
kfree(pages);
return ret;
diff --git a/net/rxrpc/input.c b/net/rxrpc/input.c
index 24aceb1..ce76146 100644
--- a/net/rxrpc/input.c
+++ b/net/rxrpc/input.c
@@ -963,23 +963,34 @@ static void rxrpc_input_soft_acks(struct rxrpc_call *call,
struct rxrpc_skb_priv *sp = rxrpc_skb(skb);
struct rxrpc_txqueue *tq = call->tx_queue;
unsigned long extracted = ~0UL;
- unsigned int nr = 0;
+ unsigned int nr = 0, nsack;
rxrpc_seq_t seq = call->acks_hard_ack + 1;
rxrpc_seq_t lowest_nak = seq + sp->ack.nr_acks;
- u8 *acks = skb->data + sizeof(struct rxrpc_wire_header) + sizeof(struct rxrpc_ackpacket);
+ u8 sack[256] __aligned(sizeof(unsigned long));
+ u8 *acks = sack;
_enter("%x,%x,%u", tq->qbase, seq, sp->ack.nr_acks);
while (after(seq, tq->qbase + RXRPC_NR_TXQUEUE - 1))
tq = tq->next;
+ /* Extract an individual SACK table. A normal SACK table is up to 255
+ * bytes with 1 ACK flag per byte, but an extended SACK table can be up
+ * to 256 bytes with up to 8 ACK/NACK flags per byte. The ACK flags go
+ * across all bit 0's then all bit 1's, then all bit 2's, ...
+ */
+ memset(sack, 0, sizeof(sack));
+ nsack = umin(sp->ack.nr_acks, 256);
+ if (skb_copy_bits(skb,
+ sizeof(struct rxrpc_wire_header) + sizeof(struct rxrpc_ackpacket),
+ sack, nsack) < 0)
+ return;
+
for (unsigned int i = 0; i < sp->ack.nr_acks; i++) {
/* Decant ACKs until we hit a txqueue boundary. */
+ if ((i & 255) == 0)
+ acks = sack;
shiftr_adv_rotr(acks, extracted);
- if (i == 256) {
- acks -= i;
- i = 0;
- }
seq++;
nr++;
if ((seq & RXRPC_TXQ_MASK) != 0)
@@ -1117,9 +1128,6 @@ static void rxrpc_input_ack(struct rxrpc_call *call, struct sk_buff *skb)
skb_copy_bits(skb, ioffset, &trailer, sizeof(trailer)) < 0)
return rxrpc_proto_abort(call, 0, rxrpc_badmsg_short_ack_trailer);
- if (nr_acks > 0)
- skb_condense(skb);
-
call->acks_latest_ts = ktime_get_real();
call->acks_hard_ack = hard_ack;
call->acks_prev_seq = prev_pkt;
diff --git a/net/sctp/bind_addr.c b/net/sctp/bind_addr.c
index 75e3e61..31737f1 100644
--- a/net/sctp/bind_addr.c
+++ b/net/sctp/bind_addr.c
@@ -275,6 +275,16 @@ int sctp_raw_to_bind_addrs(struct sctp_bind_addr *bp, __u8 *raw_addr_list,
param = (struct sctp_paramhdr *)raw_addr_list;
rawaddr = (union sctp_addr_param *)raw_addr_list;
+ if (addrs_len < sizeof(*param)) {
+ retval = -EINVAL;
+ goto out_err;
+ }
+ len = ntohs(param->length);
+ if (addrs_len < len) {
+ retval = -EINVAL;
+ goto out_err;
+ }
+
af = sctp_get_af_specific(param_type2af(param->type));
if (unlikely(!af) ||
!af->from_addr_param(&addr, rawaddr, htons(port), 0)) {
@@ -291,7 +301,6 @@ int sctp_raw_to_bind_addrs(struct sctp_bind_addr *bp, __u8 *raw_addr_list,
goto out_err;
next:
- len = ntohs(param->length);
addrs_len -= len;
raw_addr_list += len;
}
diff --git a/net/sctp/input.c b/net/sctp/input.c
index e119e46..864741f 100644
--- a/net/sctp/input.c
+++ b/net/sctp/input.c
@@ -1204,6 +1204,14 @@ static struct sctp_association *__sctp_rcv_asconf_lookup(
/* Skip over the ADDIP header and find the Address parameter */
param = (union sctp_addr_param *)(asconf + 1);
+ /* The whole address parameter must lie within the chunk before
+ * af->from_addr_param() reads the variable-length address; otherwise a
+ * truncated trailing ASCONF chunk lets it read uninitialized bytes past
+ * the parameter.
+ */
+ if (sizeof(*asconf) + ntohs(param->p.length) > ntohs(ch->length))
+ return NULL;
+
af = sctp_get_af_specific(param_type2af(param->p.type));
if (unlikely(!af))
return NULL;
diff --git a/net/sctp/sm_make_chunk.c b/net/sctp/sm_make_chunk.c
index 8526486..1741a9f 100644
--- a/net/sctp/sm_make_chunk.c
+++ b/net/sctp/sm_make_chunk.c
@@ -1731,8 +1731,8 @@ struct sctp_association *sctp_unpack_cookie(
struct sk_buff *skb = chunk->skb;
struct sctp_cookie *bear_cookie;
struct sctp_chunkhdr *ch;
+ unsigned int len, chlen;
enum sctp_scope scope;
- unsigned int len;
ktime_t kt;
/* Header size is static data prior to the actual cookie, including
@@ -1761,7 +1761,12 @@ struct sctp_association *sctp_unpack_cookie(
bear_cookie = &cookie->c;
ch = (struct sctp_chunkhdr *)(bear_cookie + 1);
- if (ntohs(ch->length) > len - fixed_size)
+ chlen = ntohs(ch->length);
+ if (chlen < sizeof(struct sctp_init_chunk))
+ goto malformed;
+ if (chlen > len - fixed_size)
+ goto malformed;
+ if (bear_cookie->raw_addr_list_len > len - fixed_size - chlen)
goto malformed;
/* Verify the cookie's MAC, if cookie authentication is enabled. */
diff --git a/net/sctp/stream.c b/net/sctp/stream.c
index c224779..5c2fded 100644
--- a/net/sctp/stream.c
+++ b/net/sctp/stream.c
@@ -1038,6 +1038,7 @@ struct sctp_chunk *sctp_process_strreset_resp(
stsn, rtsn, GFP_ATOMIC);
} else if (req->type == SCTP_PARAM_RESET_ADD_OUT_STREAMS) {
struct sctp_strreset_addstrm *addstrm;
+ const struct sctp_sched_ops *sched;
__u16 number;
addstrm = (struct sctp_strreset_addstrm *)req;
@@ -1048,7 +1049,10 @@ struct sctp_chunk *sctp_process_strreset_resp(
for (i = number; i < stream->outcnt; i++)
SCTP_SO(stream, i)->state = SCTP_STREAM_OPEN;
} else {
- sctp_stream_shrink_out(stream, number);
+ sched = sctp_sched_ops_from_stream(stream);
+ sched->unsched_all(stream);
+ sctp_stream_outq_migrate(stream, NULL, number);
+ sched->sched_all(stream);
stream->outcnt = number;
}
diff --git a/net/socket.c b/net/socket.c
index 22a412f..f51bdcb 100644
--- a/net/socket.c
+++ b/net/socket.c
@@ -310,8 +310,10 @@ static int move_addr_to_user(struct sockaddr_storage *kaddr, int klen,
static struct kmem_cache *sock_inode_cachep __ro_after_init;
+static struct simple_xattr_cache sockfs_xa_cache;
+
struct sockfs_inode {
- struct simple_xattrs *xattrs;
+ struct list_head xattrs;
struct simple_xattr_limits xattr_limits;
struct socket_alloc;
};
@@ -328,7 +330,7 @@ static struct inode *sock_alloc_inode(struct super_block *sb)
si = alloc_inode_sb(sb, sock_inode_cachep, GFP_KERNEL);
if (!si)
return NULL;
- si->xattrs = NULL;
+ INIT_LIST_HEAD_RCU(&si->xattrs);
simple_xattr_limits_init(&si->xattr_limits);
init_waitqueue_head(&si->socket.wq.wait);
@@ -347,12 +349,8 @@ static struct inode *sock_alloc_inode(struct super_block *sb)
static void sock_evict_inode(struct inode *inode)
{
struct sockfs_inode *si = SOCKFS_I(inode);
- struct simple_xattrs *xattrs = si->xattrs;
- if (xattrs) {
- simple_xattrs_free(xattrs, NULL);
- kfree(xattrs);
- }
+ simple_xattrs_free(&sockfs_xa_cache, &si->xattrs, NULL);
clear_inode(inode);
}
@@ -443,13 +441,9 @@ static int sockfs_user_xattr_get(const struct xattr_handler *handler,
const char *suffix, void *value, size_t size)
{
const char *name = xattr_full_name(handler, suffix);
- struct simple_xattrs *xattrs;
+ struct sockfs_inode *si = SOCKFS_I(inode);
- xattrs = READ_ONCE(SOCKFS_I(inode)->xattrs);
- if (!xattrs)
- return -ENODATA;
-
- return simple_xattr_get(xattrs, name, value, size);
+ return simple_xattr_get(&sockfs_xa_cache, &si->xattrs, name, value, size);
}
static int sockfs_user_xattr_set(const struct xattr_handler *handler,
@@ -460,13 +454,8 @@ static int sockfs_user_xattr_set(const struct xattr_handler *handler,
{
const char *name = xattr_full_name(handler, suffix);
struct sockfs_inode *si = SOCKFS_I(inode);
- struct simple_xattrs *xattrs;
- xattrs = simple_xattrs_lazy_alloc(&si->xattrs, value, flags);
- if (IS_ERR_OR_NULL(xattrs))
- return PTR_ERR(xattrs);
-
- return simple_xattr_set_limited(xattrs, &si->xattr_limits,
+ return simple_xattr_set_limited(&sockfs_xa_cache, &si->xattrs, &si->xattr_limits,
name, value, size, flags);
}
@@ -635,8 +624,7 @@ static ssize_t sockfs_listxattr(struct dentry *dentry, char *buffer,
struct sockfs_inode *si = SOCKFS_I(d_inode(dentry));
ssize_t len, used;
- len = simple_xattr_list(d_inode(dentry), READ_ONCE(si->xattrs),
- buffer, size);
+ len = simple_xattr_list(d_inode(dentry), &si->xattrs, buffer, size);
if (len < 0)
return len;
@@ -852,12 +840,13 @@ EXPORT_SYMBOL(kernel_sendmsg);
static bool skb_is_err_queue(const struct sk_buff *skb)
{
- /* pkt_type of skbs enqueued on the error queue are set to
- * PACKET_OUTGOING in skb_set_err_queue(). This is only safe to do
- * in recvmsg, since skbs received on a local socket will never
- * have a pkt_type of PACKET_OUTGOING.
+ /* Error-queue skbs are marked as PACKET_OUTGOING in
+ * skb_set_err_queue() and use the destructor installed by
+ * sock_queue_err_skb(). PACKET_OUTGOING alone is not unique:
+ * AF_PACKET outgoing taps use the same pkt_type.
*/
- return skb->pkt_type == PACKET_OUTGOING;
+ return skb->pkt_type == PACKET_OUTGOING &&
+ skb->destructor == sock_rmem_free;
}
/* On transmit, software and hardware timestamps are returned independently.
diff --git a/net/xfrm/espintcp.c b/net/xfrm/espintcp.c
index a275618..d903554 100644
--- a/net/xfrm/espintcp.c
+++ b/net/xfrm/espintcp.c
@@ -346,6 +346,10 @@ static int espintcp_sendmsg(struct sock *sk, struct msghdr *msg, size_t size)
err = -ENOBUFS;
goto unlock;
}
+ if (emsg->len) {
+ err = -ENOBUFS;
+ goto unlock;
+ }
sk_msg_init(&emsg->skmsg);
while (1) {
diff --git a/net/xfrm/xfrm_iptfs.c b/net/xfrm/xfrm_iptfs.c
index 6c6bbc0..ad810d1 100644
--- a/net/xfrm/xfrm_iptfs.c
+++ b/net/xfrm/xfrm_iptfs.c
@@ -954,6 +954,7 @@ static bool __input_process_payload(struct xfrm_state *x, u32 data,
u32 first_iplen, iphlen, iplen, remaining, tail;
u32 capturelen;
u64 seq;
+ bool first_skb_partial = false;
xtfs = x->mode_data;
net = xs_net(x);
@@ -1161,6 +1162,7 @@ static bool __input_process_payload(struct xfrm_state *x, u32 data,
spin_unlock(&xtfs->drop_lock);
+ first_skb_partial = (first_skb == skb);
break;
}
@@ -1172,7 +1174,7 @@ static bool __input_process_payload(struct xfrm_state *x, u32 data,
/* this should not happen from the above code */
XFRM_INC_STATS(net, LINUX_MIB_XFRMINIPTFSERROR);
- if (first_skb && first_iplen && !defer && first_skb != xtfs->ra_newskb) {
+ if (first_skb && first_iplen && !defer && !first_skb_partial) {
/* first_skb is queued b/c !defer and not partial */
if (pskb_trim(first_skb, first_iplen)) {
/* error trimming */
@@ -2168,6 +2170,8 @@ static void iptfs_consume_frags(struct sk_buff *to, struct sk_buff *from)
memcpy(&toi->frags[toi->nr_frags], fromi->frags,
sizeof(fromi->frags[0]) * fromi->nr_frags);
toi->nr_frags += fromi->nr_frags;
+ if (fromi->nr_frags)
+ toi->flags |= fromi->flags & SKBFL_SHARED_FRAG;
fromi->nr_frags = 0;
from->data_len = 0;
from->len = 0;
@@ -2726,8 +2730,9 @@ static void iptfs_destroy_state(struct xfrm_state *x)
if (!xtfs)
return;
- spin_lock_bh(&xtfs->x->lock);
hrtimer_cancel(&xtfs->iptfs_timer);
+
+ spin_lock_bh(&xtfs->x->lock);
__skb_queue_head_init(&list);
skb_queue_splice_init(&xtfs->queue, &list);
spin_unlock_bh(&xtfs->x->lock);
@@ -2735,9 +2740,7 @@ static void iptfs_destroy_state(struct xfrm_state *x)
while ((skb = __skb_dequeue(&list)))
kfree_skb(skb);
- spin_lock_bh(&xtfs->drop_lock);
hrtimer_cancel(&xtfs->drop_timer);
- spin_unlock_bh(&xtfs->drop_lock);
if (xtfs->ra_newskb)
kfree_skb(xtfs->ra_newskb);
diff --git a/net/xfrm/xfrm_policy.c b/net/xfrm/xfrm_policy.c
index dd09d20..9595444 100644
--- a/net/xfrm/xfrm_policy.c
+++ b/net/xfrm/xfrm_policy.c
@@ -1156,15 +1156,6 @@ static void __xfrm_policy_inexact_prune_bin(struct xfrm_pol_inexact_bin *b, bool
}
}
-static void xfrm_policy_inexact_prune_bin(struct xfrm_pol_inexact_bin *b)
-{
- struct net *net = read_pnet(&b->k.net);
-
- spin_lock_bh(&net->xfrm.xfrm_policy_lock);
- __xfrm_policy_inexact_prune_bin(b, false);
- spin_unlock_bh(&net->xfrm.xfrm_policy_lock);
-}
-
static void __xfrm_policy_inexact_flush(struct net *net)
{
struct xfrm_pol_inexact_bin *bin, *t;
@@ -1707,12 +1698,12 @@ xfrm_policy_bysel_ctx(struct net *net, const struct xfrm_mark *mark, u32 if_id,
}
ret = pol;
}
+ if (bin && delete)
+ __xfrm_policy_inexact_prune_bin(bin, false);
spin_unlock_bh(&net->xfrm.xfrm_policy_lock);
if (ret && delete)
xfrm_policy_kill(ret);
- if (bin && delete)
- xfrm_policy_inexact_prune_bin(bin);
return ret;
}
EXPORT_SYMBOL(xfrm_policy_bysel_ctx);
diff --git a/rust/Makefile b/rust/Makefile
index b9e9f51..2fbdebb 100644
--- a/rust/Makefile
+++ b/rust/Makefile
@@ -6,6 +6,8 @@
obj-$(CONFIG_RUST) += core.o compiler_builtins.o ffi.o
always-$(CONFIG_RUST) += exports_core_generated.h
+obj-$(CONFIG_RUST) += zerocopy.o
+
ifdef CONFIG_RUST_INLINE_HELPERS
always-$(CONFIG_RUST) += helpers/helpers.bc helpers/helpers_module.bc
else
@@ -47,13 +49,14 @@
# Avoids running `$(RUSTC)` when it may not be available.
ifdef CONFIG_RUST
-libmacros_name := $(shell MAKEFLAGS= $(RUSTC) --print file-names --crate-name macros --crate-type proc-macro - </dev/null)
-libmacros_extension := $(patsubst libmacros.%,%,$(libmacros_name))
+procmacro-name = $(shell MAKEFLAGS= $(RUSTC) --print file-names --crate-name $(1) --crate-type proc-macro - </dev/null)
+procmacro-extension := $(patsubst libname.%,%,$(call procmacro-name,name))
-libpin_init_internal_name := $(shell MAKEFLAGS= $(RUSTC) --print file-names --crate-name pin_init_internal --crate-type proc-macro - </dev/null)
-libpin_init_internal_extension := $(patsubst libpin_init_internal.%,%,$(libpin_init_internal_name))
+libzerocopy_derive_name := $(call procmacro-name,zerocopy_derive)
+libmacros_name := $(call procmacro-name,macros)
+libpin_init_internal_name := $(call procmacro-name,pin_init_internal)
-always-$(CONFIG_RUST) += $(libmacros_name) $(libpin_init_internal_name)
+always-$(CONFIG_RUST) += $(libzerocopy_derive_name) $(libmacros_name) $(libpin_init_internal_name)
# `$(rust_flags)` is passed in case the user added `--sysroot`.
rustc_sysroot := $(shell MAKEFLAGS= $(RUSTC) $(rust_flags) --print sysroot)
@@ -81,6 +84,12 @@
--edition=$(core-edition) \
$(call cfgs-to-flags,$(core-cfgs))
+zerocopy-flags := \
+ --cap-lints=allow
+
+zerocopy-envs := \
+ CARGO_PKG_VERSION=0.8.50
+
proc_macro2-cfgs := \
feature="proc-macro" \
wrap_proc_macro \
@@ -118,6 +127,12 @@
--extern quote \
$(call cfgs-to-flags,$(syn-cfgs))
+zerocopy_derive-flags := \
+ --cap-lints=allow \
+ --extern proc_macro2 \
+ --extern quote \
+ --extern syn
+
pin_init_internal-cfgs := \
kernel USE_RUSTC_FEATURES
@@ -144,6 +159,7 @@
quiet_cmd_rustdoc = RUSTDOC $(if $(rustdoc_host),H, ) $<
cmd_rustdoc = \
+ $(rustc_target_envs) \
OBJTREE=$(abspath $(objtree)) \
$(RUSTDOC) $(filter-out $(skip_flags) --remap-path-scope=%,$(if $(rustdoc_host),$(rust_common_flags),$(rust_flags))) \
$(rustc_target_flags) -L$(objtree)/$(obj) \
@@ -166,7 +182,7 @@
# command-like flags to solve the issue. Meanwhile, we use the non-custom case
# and then retouch the generated files.
rustdoc: rustdoc-core rustdoc-macros rustdoc-compiler_builtins \
- rustdoc-kernel rustdoc-pin_init
+ rustdoc-kernel rustdoc-pin_init rustdoc-zerocopy rustdoc-zerocopy_derive
$(Q)grep -Ehro '<a href="srctree/([^"]+)"' $(rustdoc_output) | \
cut -d'"' -f2 | cut -d/ -f2- | while read f; do \
if [ ! -e "$(srctree)/$$f" ]; then \
@@ -199,6 +215,12 @@
rustdoc-syn: $(src)/syn/lib.rs rustdoc-clean rustdoc-quote FORCE
+$(call if_changed,rustdoc)
+rustdoc-zerocopy_derive: private rustdoc_host = yes
+rustdoc-zerocopy_derive: private rustc_target_flags = $(zerocopy_derive-flags) \
+ --extern proc_macro --crate-type proc-macro
+rustdoc-zerocopy_derive: $(src)/zerocopy-derive/lib.rs rustdoc-clean rustdoc-syn FORCE
+ +$(call if_changed,rustdoc)
+
rustdoc-macros: private rustdoc_host = yes
rustdoc-macros: private rustc_target_flags = --crate-type proc-macro \
--extern proc_macro --extern proc_macro2 --extern quote --extern syn
@@ -218,6 +240,13 @@
rustdoc-compiler_builtins: $(src)/compiler_builtins.rs rustdoc-core FORCE
+$(call if_changed,rustdoc)
+rustdoc-zerocopy: private rustc_target_envs := $(zerocopy-envs)
+rustdoc-zerocopy: private is-kernel-object := y
+rustdoc-zerocopy: private rustc_target_flags = $(zerocopy-flags) \
+ --extend-css $(src)/zerocopy/rustdoc/style.css
+rustdoc-zerocopy: $(src)/zerocopy/src/lib.rs rustdoc-clean rustdoc-core FORCE
+ +$(call if_changed,rustdoc)
+
rustdoc-ffi: private is-kernel-object := y
rustdoc-ffi: $(src)/ffi.rs rustdoc-core FORCE
+$(call if_changed,rustdoc)
@@ -239,7 +268,8 @@
rustdoc-kernel: private is-kernel-object := y
rustdoc-kernel: private rustc_target_flags = --extern ffi --extern pin_init \
--extern build_error --extern macros \
- --extern bindings --extern uapi
+ --extern bindings --extern uapi \
+ --extern zerocopy --extern zerocopy_derive
rustdoc-kernel: $(src)/kernel/lib.rs rustdoc-core rustdoc-ffi rustdoc-macros \
rustdoc-pin_init rustdoc-compiler_builtins $(obj)/$(libmacros_name) \
$(obj)/bindings.o FORCE
@@ -250,6 +280,7 @@
quiet_cmd_rustc_test_library = $(RUSTC_OR_CLIPPY_QUIET) TL $<
cmd_rustc_test_library = \
+ $(rustc_target_envs) \
OBJTREE=$(abspath $(objtree)) \
$(RUSTC_OR_CLIPPY) $(filter-out $(skip_flags),$(rust_common_flags) $(rustc_target_flags)) \
@$(objtree)/include/generated/rustc_cfg \
@@ -258,6 +289,11 @@
-L$(objtree)/$(obj)/test \
--crate-name $(subst rusttest-,,$(subst rusttestlib-,,$@)) $<
+rusttestlib-zerocopy: private rustc_target_envs := $(zerocopy-envs)
+rusttestlib-zerocopy: private rustc_target_flags = $(zerocopy-flags)
+rusttestlib-zerocopy: $(src)/zerocopy/src/lib.rs FORCE
+ +$(call if_changed,rustc_test_library)
+
rusttestlib-build_error: $(src)/build_error.rs FORCE
+$(call if_changed,rustc_test_library)
@@ -277,6 +313,12 @@
rusttestlib-syn: $(src)/syn/lib.rs rusttestlib-quote FORCE
+$(call if_changed,rustc_test_library)
+rusttestlib-zerocopy_derive: private rustc_target_flags = $(zerocopy_derive-flags) \
+ --extern proc_macro
+rusttestlib-zerocopy_derive: private rustc_test_library_proc = yes
+rusttestlib-zerocopy_derive: $(src)/zerocopy-derive/lib.rs rusttestlib-syn FORCE
+ +$(call if_changed,rustc_test_library)
+
rusttestlib-macros: private rustc_target_flags = --extern proc_macro \
--extern proc_macro2 --extern quote --extern syn
rusttestlib-macros: private rustc_test_library_proc = yes
@@ -298,10 +340,11 @@
rusttestlib-kernel: private rustc_target_flags = --extern ffi \
--extern build_error --extern macros --extern pin_init \
- --extern bindings --extern uapi
+ --extern bindings --extern uapi \
+ --extern zerocopy --extern zerocopy_derive
rusttestlib-kernel: $(src)/kernel/lib.rs rusttestlib-bindings rusttestlib-uapi \
rusttestlib-build_error rusttestlib-pin_init $(obj)/$(libmacros_name) \
- $(obj)/bindings.o FORCE
+ $(obj)/bindings.o rusttestlib-zerocopy rusttestlib-zerocopy_derive FORCE
+$(call if_changed,rustc_test_library)
rusttestlib-bindings: private rustc_target_flags = --extern ffi --extern pin_init
@@ -314,6 +357,7 @@
quiet_cmd_rustdoc_test = RUSTDOC T $<
cmd_rustdoc_test = \
+ $(rustc_target_envs) \
RUST_MODFILE=test.rs \
OBJTREE=$(abspath $(objtree)) \
$(RUSTDOC) --test $(rust_common_flags) \
@@ -328,11 +372,13 @@
cmd_rustdoc_test_kernel = \
rm -rf $(objtree)/$(obj)/test/doctests/kernel; \
mkdir -p $(objtree)/$(obj)/test/doctests/kernel; \
+ $(rustc_target_envs) \
OBJTREE=$(abspath $(objtree)) \
$(RUSTDOC) --test $(filter-out --remap-path-scope=%,$(rust_flags)) \
-L$(objtree)/$(obj) --extern ffi --extern pin_init \
--extern kernel --extern build_error --extern macros \
--extern bindings --extern uapi \
+ --extern zerocopy --extern zerocopy_derive \
--no-run --crate-name kernel -Zunstable-options \
--sysroot=/dev/null \
$(doctests_modifiers_workaround) \
@@ -350,6 +396,7 @@
# so for the moment we skip `-Cpanic=abort`.
quiet_cmd_rustc_test = $(RUSTC_OR_CLIPPY_QUIET) T $<
cmd_rustc_test = \
+ $(rustc_target_envs) \
OBJTREE=$(abspath $(objtree)) \
$(RUSTC_OR_CLIPPY) --test $(rust_common_flags) \
@$(objtree)/include/generated/rustc_cfg \
@@ -517,8 +564,9 @@
$(obj)/exports_kernel_generated.h: $(obj)/kernel.o FORCE
$(call if_changed,exports)
-quiet_cmd_rustc_procmacrolibrary = $(RUSTC_OR_CLIPPY_QUIET) PL $@
+quiet_cmd_rustc_procmacrolibrary = $(if $(skip_clippy),RUSTC,$(RUSTC_OR_CLIPPY_QUIET)) PL $@
cmd_rustc_procmacrolibrary = \
+ $(rustc_target_envs) \
$(if $(skip_clippy),$(RUSTC),$(RUSTC_OR_CLIPPY)) \
$(filter-out $(skip_flags),$(rust_common_flags) $(rustc_target_flags)) \
--emit=dep-info=$(depfile) --emit=link=$@ --crate-type rlib -O \
@@ -541,17 +589,24 @@
$(obj)/libsyn.rlib: $(src)/syn/lib.rs $(obj)/libquote.rlib FORCE
+$(call if_changed_dep,rustc_procmacrolibrary)
-quiet_cmd_rustc_procmacro = $(RUSTC_OR_CLIPPY_QUIET) P $@
+quiet_cmd_rustc_procmacro = $(if $(skip_clippy),RUSTC,$(RUSTC_OR_CLIPPY_QUIET)) P $@
cmd_rustc_procmacro = \
- $(RUSTC_OR_CLIPPY) $(rust_common_flags) $(rustc_target_flags) \
+ $(rustc_target_envs) \
+ $(if $(skip_clippy),$(RUSTC),$(RUSTC_OR_CLIPPY)) $(rust_common_flags) $(rustc_target_flags) \
-Clinker-flavor=gcc -Clinker=$(HOSTCC) \
-Clink-args='$(call escsq,$(KBUILD_PROCMACROLDFLAGS))' \
--emit=dep-info=$(depfile) --emit=link=$@ --extern proc_macro \
--crate-type proc-macro -L$(objtree)/$(obj) \
- --crate-name $(patsubst lib%.$(libmacros_extension),%,$(notdir $@)) \
+ --crate-name $(patsubst lib%.$(procmacro-extension),%,$(notdir $@)) \
@$(objtree)/include/generated/rustc_cfg $<
# Procedural macros can only be used with the `rustc` that compiled it.
+$(obj)/$(libzerocopy_derive_name): private skip_clippy = 1
+$(obj)/$(libzerocopy_derive_name): private rustc_target_flags = $(zerocopy_derive-flags)
+$(obj)/$(libzerocopy_derive_name): $(src)/zerocopy-derive/lib.rs $(obj)/libproc_macro2.rlib \
+ $(obj)/libquote.rlib $(obj)/libsyn.rlib FORCE
+ +$(call if_changed_dep,rustc_procmacro)
+
$(obj)/$(libmacros_name): private rustc_target_flags = \
--extern proc_macro2 --extern quote --extern syn
$(obj)/$(libmacros_name): $(src)/macros/lib.rs $(obj)/libproc_macro2.rlib \
@@ -567,6 +622,7 @@
# since Rust 1.95.0 (https://github.com/rust-lang/rust/pull/151534).
quiet_cmd_rustc_library = $(if $(skip_clippy),RUSTC,$(RUSTC_OR_CLIPPY_QUIET)) L $@
cmd_rustc_library = \
+ $(rustc_target_envs) \
OBJTREE=$(abspath $(objtree)) \
$(if $(skip_clippy),$(RUSTC),$(RUSTC_OR_CLIPPY)) \
$(filter-out $(skip_flags),$(rust_flags)) $(rustc_target_flags) \
@@ -591,6 +647,7 @@
--cfgs='syn=$(syn-cfgs)' \
--cfgs='pin_init_internal=$(pin_init_internal-cfgs)' \
--cfgs='pin_init=$(pin_init-cfgs)' \
+ --envs='zerocopy=$(zerocopy-envs)' \
$(realpath $(srctree)) $(realpath $(objtree)) \
$(rustc_sysroot) $(RUST_LIB_SRC) $(if $(KBUILD_EXTMOD),$(srcroot)) \
> rust-project.json
@@ -662,6 +719,13 @@
$(obj)/compiler_builtins.o: $(src)/compiler_builtins.rs $(obj)/core.o FORCE
+$(call if_changed_rule,rustc_library)
+$(obj)/zerocopy.o: private skip_clippy = 1
+$(obj)/zerocopy.o: private skip_gendwarfksyms = 1
+$(obj)/zerocopy.o: private rustc_target_envs := $(zerocopy-envs)
+$(obj)/zerocopy.o: private rustc_target_flags = $(zerocopy-flags)
+$(obj)/zerocopy.o: $(src)/zerocopy/src/lib.rs $(obj)/compiler_builtins.o FORCE
+ +$(call if_changed_rule,rustc_library)
+
$(obj)/pin_init.o: private skip_gendwarfksyms = 1
$(obj)/pin_init.o: private rustc_target_flags = $(pin_init-flags)
$(obj)/pin_init.o: $(src)/pin-init/src/lib.rs $(obj)/compiler_builtins.o \
@@ -697,9 +761,11 @@
+$(call if_changed_rule,rustc_library)
$(obj)/kernel.o: private rustc_target_flags = --extern ffi --extern pin_init \
- --extern build_error --extern macros --extern bindings --extern uapi
+ --extern build_error --extern macros --extern bindings --extern uapi \
+ --extern zerocopy --extern zerocopy_derive
$(obj)/kernel.o: $(src)/kernel/lib.rs $(obj)/build_error.o $(obj)/pin_init.o \
- $(obj)/$(libmacros_name) $(obj)/bindings.o $(obj)/uapi.o FORCE
+ $(obj)/$(libmacros_name) $(obj)/bindings.o $(obj)/uapi.o \
+ $(obj)/zerocopy.o $(obj)/$(libzerocopy_derive_name) FORCE
+$(call if_changed_rule,rustc_library)
ifdef CONFIG_JUMP_LABEL
diff --git a/rust/kernel/Kconfig.test b/rust/kernel/Kconfig.test
new file mode 100644
index 0000000..e6a5c7a
--- /dev/null
+++ b/rust/kernel/Kconfig.test
@@ -0,0 +1,86 @@
+# SPDX-License-Identifier: GPL-2.0-only
+menuconfig RUST_KUNIT_TESTS
+ bool "Rust KUnit tests"
+ depends on KUNIT && RUST
+ default KUNIT_ALL_TESTS
+ help
+ This menu collects all options for Rust KUnit tests.
+ See Documentation/rust/testing.rst for how to protect
+ unit tests with these options.
+
+ Say Y here to enable Rust KUnit tests.
+
+ If unsure, say N.
+
+if RUST_KUNIT_TESTS
+config RUST_ALLOCATOR_KUNIT_TEST
+ bool "KUnit tests for Rust allocator API" if !KUNIT_ALL_TESTS
+ default KUNIT_ALL_TESTS
+ help
+ This option enables KUnit tests for the Rust allocator API.
+ These are only for development and testing, not for regular
+ kernel use cases.
+
+ If unsure, say N.
+
+config RUST_KVEC_KUNIT_TEST
+ bool "KUnit tests for Rust KVec API" if !KUNIT_ALL_TESTS
+ default KUNIT_ALL_TESTS
+ help
+ This option enables KUnit tests for the Rust KVec API.
+ These are only for development and testing, not for
+ regular kernel use cases.
+
+ If unsure, say N.
+
+config RUST_BITMAP_KUNIT_TEST
+ bool "KUnit tests for Rust bitmap API" if !KUNIT_ALL_TESTS
+ default KUNIT_ALL_TESTS
+ help
+ This option enables KUnit tests for the Rust bitmap API.
+ These are only for development and testing, not for regular
+ kernel use cases.
+
+ If unsure, say N.
+
+config RUST_KUNIT_SELFTEST
+ bool "KUnit selftests for Rust" if !KUNIT_ALL_TESTS
+ default KUNIT_ALL_TESTS
+ help
+ This option enables KUnit selftests. These are only
+ for development and testing, not for regular kernel
+ use cases.
+
+ If unsure, say N.
+
+config RUST_STR_KUNIT_TEST
+ bool "KUnit tests for Rust strings API" if !KUNIT_ALL_TESTS
+ default KUNIT_ALL_TESTS
+ help
+ This option enables KUnit tests for the Rust strings API.
+ These are only for development and testing, not for regular
+ kernel use cases.
+
+ If unsure, say N.
+
+config RUST_ATOMICS_KUNIT_TEST
+ bool "KUnit tests for Rust atomics API" if !KUNIT_ALL_TESTS
+ default KUNIT_ALL_TESTS
+ help
+ This option enables KUnit tests for the Rust atomics API.
+ These are only for development and testing, not for regular
+ kernel use cases.
+
+ If unsure, say N.
+
+config RUST_BITFIELD_KUNIT_TEST
+ bool "KUnit tests for the Rust `bitfield!` macro" if !KUNIT_ALL_TESTS
+ default KUNIT_ALL_TESTS
+ help
+ This option enables KUnit tests for the Rust `bitfield!` macro.
+ These are only for development and testing, not for regular
+ kernel use cases.
+
+ If unsure, say N.
+
+endif
diff --git a/rust/kernel/alloc.rs b/rust/kernel/alloc.rs
index e387203..21067bd 100644
--- a/rust/kernel/alloc.rs
+++ b/rust/kernel/alloc.rs
@@ -22,8 +22,12 @@
#[derive(Copy, Clone, PartialEq, Eq, Debug)]
pub struct AllocError;
-use crate::error::{code::EINVAL, Result};
-use core::{alloc::Layout, ptr::NonNull};
+use crate::prelude::*;
+
+use core::{
+ alloc::Layout,
+ ptr::NonNull, //
+};
/// Flags to be used when allocating memory.
///
diff --git a/rust/kernel/alloc/allocator.rs b/rust/kernel/alloc/allocator.rs
index 63bfb91..cd4203f 100644
--- a/rust/kernel/alloc/allocator.rs
+++ b/rust/kernel/alloc/allocator.rs
@@ -8,14 +8,25 @@
//!
//! Reference: <https://docs.kernel.org/core-api/memory-allocation.html>
-use super::Flags;
-use core::alloc::Layout;
-use core::ptr;
-use core::ptr::NonNull;
+use super::{
+ AllocError,
+ Allocator,
+ Flags,
+ NumaNode, //
+};
-use crate::alloc::{AllocError, Allocator, NumaNode};
-use crate::bindings;
-use crate::page;
+use crate::{
+ bindings,
+ page, //
+};
+
+use core::{
+ alloc::Layout,
+ ptr::{
+ self,
+ NonNull, //
+ }, //
+};
const ARCH_KMALLOC_MINALIGN: usize = bindings::ARCH_KMALLOC_MINALIGN;
@@ -163,8 +174,11 @@ impl Vmalloc {
/// # Examples
///
/// ```
- /// # use core::ptr::{NonNull, from_mut};
- /// # use kernel::{page, prelude::*};
+ /// # use core::ptr::{
+ /// # from_mut,
+ /// # NonNull, //
+ /// # };
+ /// # use kernel::page;
/// use kernel::alloc::allocator::Vmalloc;
///
/// let mut vbox = VBox::<[u8; page::PAGE_SIZE]>::new_uninit(GFP_KERNEL)?;
@@ -251,6 +265,7 @@ unsafe fn realloc(
}
}
+#[cfg(CONFIG_RUST_ALLOCATOR_KUNIT_TEST)]
#[macros::kunit_tests(rust_allocator)]
mod tests {
use super::*;
diff --git a/rust/kernel/alloc/allocator/iter.rs b/rust/kernel/alloc/allocator/iter.rs
index e0a70b7..02fda3e 100644
--- a/rust/kernel/alloc/allocator/iter.rs
+++ b/rust/kernel/alloc/allocator/iter.rs
@@ -1,9 +1,13 @@
// SPDX-License-Identifier: GPL-2.0
use super::Vmalloc;
+
use crate::page;
-use core::marker::PhantomData;
-use core::ptr::NonNull;
+
+use core::{
+ marker::PhantomData,
+ ptr::NonNull, //
+};
/// An [`Iterator`] of [`page::BorrowedPage`] items owned by a [`Vmalloc`] allocation.
///
diff --git a/rust/kernel/alloc/kbox.rs b/rust/kernel/alloc/kbox.rs
index bd6da02..80eb393 100644
--- a/rust/kernel/alloc/kbox.rs
+++ b/rust/kernel/alloc/kbox.rs
@@ -3,24 +3,47 @@
//! Implementation of [`Box`].
#[allow(unused_imports)] // Used in doc comments.
-use super::allocator::{KVmalloc, Kmalloc, Vmalloc, VmallocPageIter};
-use super::{AllocError, Allocator, Flags, NumaNode};
-use core::alloc::Layout;
-use core::borrow::{Borrow, BorrowMut};
-use core::marker::PhantomData;
-use core::mem::ManuallyDrop;
-use core::mem::MaybeUninit;
-use core::ops::{Deref, DerefMut};
-use core::pin::Pin;
-use core::ptr::NonNull;
-use core::result::Result;
+use super::allocator::{
+ KVmalloc,
+ Kmalloc,
+ Vmalloc,
+ VmallocPageIter, //
+};
-use crate::ffi::c_void;
-use crate::fmt;
-use crate::init::InPlaceInit;
-use crate::page::AsPageIter;
-use crate::types::ForeignOwnable;
-use pin_init::{InPlaceWrite, Init, PinInit, ZeroableOption};
+use super::{
+ AllocError,
+ Allocator,
+ Flags,
+ NumaNode, //
+};
+
+use crate::{
+ fmt,
+ page::AsPageIter,
+ prelude::*,
+ types::ForeignOwnable, //
+};
+
+use core::{
+ alloc::Layout,
+ borrow::{
+ Borrow,
+ BorrowMut, //
+ },
+ marker::PhantomData,
+ mem::{
+ ManuallyDrop,
+ MaybeUninit, //
+ },
+ ops::{
+ Deref,
+ DerefMut, //
+ },
+ ptr::NonNull,
+ result::Result, //
+};
+
+use pin_init::ZeroableOption;
/// The kernel's [`Box`] type -- a heap allocation for a single value of type `T`.
///
@@ -274,7 +297,10 @@ pub fn pin(x: T, flags: Flags) -> Result<Pin<Box<T, A>>, AllocError>
/// # Examples
///
/// ```
- /// use kernel::sync::{new_spinlock, SpinLock};
+ /// use kernel::sync::{
+ /// new_spinlock,
+ /// SpinLock, //
+ /// };
///
/// struct Inner {
/// a: u32,
@@ -411,6 +437,7 @@ impl<T, A> InPlaceWrite<T> for Box<MaybeUninit<T>, A>
{
type Initialized = Box<T, A>;
+ #[inline]
fn write_init<E>(mut self, init: impl Init<T, E>) -> Result<Self::Initialized, E> {
let slot = self.as_mut_ptr();
// SAFETY: When init errors/panics, slot will get deallocated but not dropped,
@@ -420,6 +447,7 @@ fn write_init<E>(mut self, init: impl Init<T, E>) -> Result<Self::Initialized, E
Ok(unsafe { Box::assume_init(self) })
}
+ #[inline]
fn write_pin_init<E>(mut self, init: impl PinInit<T, E>) -> Result<Pin<Self::Initialized>, E> {
let slot = self.as_mut_ptr();
// SAFETY: When init errors/panics, slot will get deallocated but not dropped,
@@ -567,7 +595,6 @@ fn deref_mut(&mut self) -> &mut T {
///
/// ```
/// # use core::borrow::Borrow;
-/// # use kernel::alloc::KBox;
/// struct Foo<B: Borrow<u32>>(B);
///
/// // Owned instance.
@@ -595,7 +622,6 @@ fn borrow(&self) -> &T {
///
/// ```
/// # use core::borrow::BorrowMut;
-/// # use kernel::alloc::KBox;
/// struct Foo<B: BorrowMut<u32>>(B);
///
/// // Owned instance.
@@ -660,9 +686,13 @@ fn drop(&mut self) {
/// # Examples
///
/// ```
-/// # use kernel::prelude::*;
-/// use kernel::alloc::allocator::VmallocPageIter;
-/// use kernel::page::{AsPageIter, PAGE_SIZE};
+/// use kernel::{
+/// alloc::allocator::VmallocPageIter,
+/// page::{
+/// AsPageIter,
+/// PAGE_SIZE, //
+/// }, //
+/// };
///
/// let mut vbox = VBox::new((), GFP_KERNEL)?;
///
diff --git a/rust/kernel/alloc/kvec.rs b/rust/kernel/alloc/kvec.rs
index 6438385..f7af628 100644
--- a/rust/kernel/alloc/kvec.rs
+++ b/rust/kernel/alloc/kvec.rs
@@ -3,29 +3,52 @@
//! Implementation of [`Vec`].
use super::{
- allocator::{KVmalloc, Kmalloc, Vmalloc, VmallocPageIter},
+ allocator::{
+ KVmalloc,
+ Kmalloc,
+ Vmalloc,
+ VmallocPageIter, //
+ },
layout::ArrayLayout,
- AllocError, Allocator, Box, Flags, NumaNode,
+ AllocError,
+ Allocator,
+ Box,
+ Flags,
+ NumaNode, //
};
+
use crate::{
fmt,
page::{
AsPageIter,
PAGE_SIZE, //
- },
+ }, //
};
+
use core::{
- borrow::{Borrow, BorrowMut},
+ borrow::{
+ Borrow,
+ BorrowMut, //
+ },
marker::PhantomData,
- mem::{ManuallyDrop, MaybeUninit},
- ops::Deref,
- ops::DerefMut,
- ops::Index,
- ops::IndexMut,
- ptr,
- ptr::NonNull,
- slice,
- slice::SliceIndex,
+ mem::{
+ ManuallyDrop,
+ MaybeUninit, //
+ },
+ ops::{
+ Deref,
+ DerefMut,
+ Index,
+ IndexMut, //
+ },
+ ptr::{
+ self,
+ NonNull, //
+ },
+ slice::{
+ self,
+ SliceIndex, //
+ }, //
};
mod errors;
@@ -614,7 +637,7 @@ pub fn clear(&mut self) {
///
/// v.reserve(10, GFP_KERNEL)?;
/// let cap = v.capacity();
- /// assert!(cap >= 10);
+ /// assert!(cap >= v.len() + 10);
///
/// v.reserve(10, GFP_KERNEL)?;
/// let new_cap = v.capacity();
@@ -849,6 +872,24 @@ pub fn shrink_to(&mut self, min_capacity: usize, flags: Flags) -> Result<(), All
impl<T: Clone, A: Allocator> Vec<T, A> {
/// Extend the vector by `n` clones of `value`.
+ ///
+ /// # Examples
+ ///
+ /// ```
+ /// let mut v = KVec::new();
+ /// v.push(1, GFP_KERNEL)?;
+ ///
+ /// v.extend_with(3, 5, GFP_KERNEL)?;
+ /// assert_eq!(&v, &[1, 5, 5, 5]);
+ ///
+ /// v.extend_with(2, 8, GFP_KERNEL)?;
+ /// assert_eq!(&v, &[1, 5, 5, 5, 8, 8]);
+ ///
+ /// v.extend_with(0, 3, GFP_KERNEL)?;
+ /// assert_eq!(&v, &[1, 5, 5, 5, 8, 8]);
+ ///
+ /// # Ok::<(), Error>(())
+ /// ```
pub fn extend_with(&mut self, n: usize, value: T, flags: Flags) -> Result<(), AllocError> {
if n == 0 {
return Ok(());
@@ -866,7 +907,7 @@ pub fn extend_with(&mut self, n: usize, value: T, flags: Flags) -> Result<(), Al
spare[n - 1].write(value);
// SAFETY:
- // - `self.len() + n < self.capacity()` due to the call to reserve above,
+ // - `self.len() + n <= self.capacity()` due to the call to reserve above,
// - the loop and the line above initialized the next `n` elements.
unsafe { self.inc_len(n) };
@@ -1146,9 +1187,13 @@ fn into_iter(self) -> Self::IntoIter {
/// # Examples
///
/// ```
-/// # use kernel::prelude::*;
-/// use kernel::alloc::allocator::VmallocPageIter;
-/// use kernel::page::{AsPageIter, PAGE_SIZE};
+/// use kernel::{
+/// alloc::allocator::VmallocPageIter,
+/// page::{
+/// AsPageIter,
+/// PAGE_SIZE, //
+/// }, //
+/// };
///
/// let mut vec = VVec::<u8>::new();
///
@@ -1463,6 +1508,7 @@ fn drop(&mut self) {
}
}
+#[cfg(CONFIG_RUST_KVEC_KUNIT_TEST)]
#[macros::kunit_tests(rust_kvec)]
mod tests {
use super::*;
diff --git a/rust/kernel/alloc/kvec/errors.rs b/rust/kernel/alloc/kvec/errors.rs
index 985c5f2..aaca644 100644
--- a/rust/kernel/alloc/kvec/errors.rs
+++ b/rust/kernel/alloc/kvec/errors.rs
@@ -2,8 +2,10 @@
//! Errors for the [`Vec`] type.
-use kernel::fmt;
-use kernel::prelude::*;
+use crate::{
+ fmt,
+ prelude::*, //
+};
/// Error type for [`Vec::push_within_capacity`].
pub struct PushError<T>(pub T);
diff --git a/rust/kernel/alloc/layout.rs b/rust/kernel/alloc/layout.rs
index 9f8be72..62a459c 100644
--- a/rust/kernel/alloc/layout.rs
+++ b/rust/kernel/alloc/layout.rs
@@ -4,7 +4,10 @@
//!
//! Custom layout types extending or improving [`Layout`].
-use core::{alloc::Layout, marker::PhantomData};
+use core::{
+ alloc::Layout,
+ marker::PhantomData, //
+};
/// Error when constructing an [`ArrayLayout`].
pub struct LayoutError;
@@ -47,7 +50,10 @@ pub const fn empty() -> Self {
/// # Examples
///
/// ```
- /// # use kernel::alloc::layout::{ArrayLayout, LayoutError};
+ /// # use kernel::alloc::layout::{
+ /// # ArrayLayout,
+ /// # LayoutError, //
+ /// # };
/// let layout = ArrayLayout::<i32>::new(15)?;
/// assert_eq!(layout.len(), 15);
///
diff --git a/rust/kernel/bitfield.rs b/rust/kernel/bitfield.rs
new file mode 100644
index 0000000..554a5a2
--- /dev/null
+++ b/rust/kernel/bitfield.rs
@@ -0,0 +1,862 @@
+// SPDX-License-Identifier: GPL-2.0
+
+//! Support for defining bitfields as Rust structures.
+//!
+//! The [`bitfield!`](kernel::bitfield!) macro declares integer types that are split into distinct
+//! bit fields of arbitrary length. Each field is typed using [`Bounded`](kernel::num::Bounded) to
+//! ensure values are properly validated and to avoid implicit data loss.
+//!
+//! # Example
+//!
+//! ```rust
+//! use kernel::bitfield;
+//! use kernel::num::Bounded;
+//!
+//! bitfield! {
+//! pub struct Rgb(u16) {
+//! 15:11 blue;
+//! 10:5 green;
+//! 4:0 red;
+//! }
+//! }
+//!
+//! // Valid value for the `blue` field.
+//! let blue = Bounded::<u16, 5>::new::<0x18>();
+//!
+//! // Setters can be chained. Values ranges are checked at compile-time.
+//! let color = Rgb::zeroed()
+//! // Compile-time bounds check of constant value.
+//! .with_const_red::<0x10>()
+//! .with_const_green::<0x1f>()
+//! // A `Bounded` can also be passed.
+//! .with_blue(blue);
+//!
+//! assert_eq!(color.red(), 0x10);
+//! assert_eq!(color.green(), 0x1f);
+//! assert_eq!(color.blue(), 0x18);
+//! assert_eq!(
+//! color.into_raw(),
+//! (0x18 << Rgb::BLUE_SHIFT) + (0x1f << Rgb::GREEN_SHIFT) + 0x10,
+//! );
+//!
+//! // Convert to/from the backing storage type.
+//! let raw: u16 = color.into();
+//! assert_eq!(Rgb::from(raw), color);
+//! ```
+//!
+//! # Syntax
+//!
+//! ```text
+//! bitfield! {
+//! #[attributes]
+//! // Documentation for `Name`.
+//! pub struct Name(storage_type) {
+//! // `field_1` documentation.
+//! hi:lo field_1;
+//! // `field_2` documentation.
+//! hi:lo field_2 => ConvertedType;
+//! // `field_3` documentation.
+//! hi:lo field_3 ?=> ConvertedType;
+//! ...
+//! }
+//! }
+//! ```
+//!
+//! - `storage_type`: The underlying unsigned integer type ([`u8`], [`u16`], [`u32`], [`u64`]).
+//! Signed integer storage types are not supported.
+//! - `hi:lo`: Bit range (inclusive), where `hi >= lo`.
+//! - `=> Type`: Optional infallible conversion (see [below](#infallible-conversion-)).
+//! - `?=> Type`: Optional fallible conversion (see [below](#fallible-conversion-)).
+//! - Documentation strings and attributes are optional.
+//!
+//! # Generated code
+//!
+//! Each field is internally represented as a [`Bounded`] parameterized by its bit width. Field
+//! values can either be set/retrieved directly, or converted from/to another type.
+//!
+//! The use of [`Bounded`] for each field enforces bounds-checking (at build time or runtime) of
+//! every value assigned to a field. This ensures that data is never accidentally truncated.
+//!
+//! The macro generates the bitfield type, [`From`] and [`Into`] implementations for its storage
+//! type, as well as [`Debug`] and [`Zeroable`](pin_init::Zeroable) implementations.
+//!
+//! For each field, it also generates:
+//!
+//! - `field()`: Getter method for the field value.
+//! - `with_field(value)`: Infallible setter; the argument type must fit within the field's width.
+//! - `with_const_field::<VALUE>()`: `const` setter; the value is validated at compile time.
+//! Usually shorter to use than `with_field` for constant values as it doesn't require
+//! constructing a [`Bounded`].
+//! - `try_with_field(value)`: Fallible setter. Returns an error if the value is out of range.
+//! - `FIELD_MASK`, `FIELD_SHIFT`, `FIELD_RANGE`: Constants for manual bit manipulation.
+//!
+//! # Reserved names for field identifiers
+//!
+//! Field identifiers are used to generate methods and associated constants on the bitfield type.
+//! For a field named `field`, the macro may generate methods named `field`, `with_field`,
+//! `with_const_field`, `try_with_field`, `__field` and `__with_field`, as well as constants named
+//! `FIELD_MASK`, `FIELD_SHIFT` and `FIELD_RANGE`.
+//!
+//! Therefore, field identifiers must not use names that would collide with generated items for
+//! any field in the same bitfield. The following prefixes are thus reserved for field identifiers:
+//!
+//! - `with_`
+//! - `const_`
+//! - `try_with_`
+//! - `__`
+//!
+//! The field identifiers `from_raw`, `into_raw`, and `into` are also reserved.
+//!
+//! In addition, field identifiers should follow Rust `snake_case` conventions, since the associated
+//! constants are generated by uppercasing the field name.
+//!
+//! # Implicit conversions
+//!
+//! Types that fit entirely within a field's bit width can be used directly with setters. For
+//! example, [`bool`] works with single-bit fields, and [`u8`] works with 8-bit fields:
+//!
+//! ```rust
+//! use kernel::bitfield;
+//!
+//! bitfield! {
+//! pub struct Flags(u32) {
+//! 15:8 byte_field;
+//! 0:0 flag;
+//! }
+//! }
+//!
+//! let flags = Flags::zeroed()
+//! .with_byte_field(0x42_u8)
+//! .with_flag(true);
+//!
+//! assert_eq!(flags.into_raw(), (0x42 << Flags::BYTE_FIELD_SHIFT) | 1);
+//! ```
+//!
+//! # Runtime bounds checking
+//!
+//! When a value is not known at compile time, use `try_with_field()` to check bounds at runtime:
+//!
+//! ```rust
+//! use kernel::bitfield;
+//!
+//! bitfield! {
+//! pub struct Config(u8) {
+//! 3:0 nibble;
+//! }
+//! }
+//!
+//! fn set_nibble(config: Config, value: u8) -> Result<Config, Error> {
+//! // Returns `EOVERFLOW` if `value > 0xf`.
+//! config.try_with_nibble(value)
+//! }
+//! # Ok::<(), Error>(())
+//! ```
+//!
+//! # Type conversion
+//!
+//! Fields can be automatically converted to/from a custom type using `=>` (infallible) or `?=>`
+//! (fallible). The custom type must implement the appropriate [`From`] or [`TryFrom`] traits with
+//! [`Bounded`].
+//!
+//! ## Infallible conversion (`=>`)
+//!
+//! Use this when all possible bit patterns of a field map to valid values:
+//!
+//! ```rust
+//! use kernel::bitfield;
+//! use kernel::num::Bounded;
+//!
+//! #[derive(Debug, Clone, Copy, PartialEq)]
+//! enum Power {
+//! Off,
+//! On,
+//! }
+//!
+//! impl From<Bounded<u32, 1>> for Power {
+//! fn from(v: Bounded<u32, 1>) -> Self {
+//! match *v {
+//! 0 => Power::Off,
+//! _ => Power::On,
+//! }
+//! }
+//! }
+//!
+//! impl From<Power> for Bounded<u32, 1> {
+//! fn from(p: Power) -> Self {
+//! (p as u32 != 0).into()
+//! }
+//! }
+//!
+//! bitfield! {
+//! pub struct Control(u32) {
+//! 0:0 power => Power;
+//! }
+//! }
+//!
+//! let ctrl = Control::zeroed().with_power(Power::On);
+//! assert_eq!(ctrl.power(), Power::On);
+//! ```
+//!
+//! ## Fallible conversion (`?=>`)
+//!
+//! Use this when some bit patterns of a field are invalid. The getter returns a [`Result`]:
+//!
+//! ```rust
+//! use kernel::bitfield;
+//! use kernel::num::Bounded;
+//!
+//! #[derive(Debug, Clone, Copy, PartialEq)]
+//! enum Mode {
+//! Low = 0,
+//! High = 1,
+//! Auto = 2,
+//! // 3 is invalid
+//! }
+//!
+//! impl TryFrom<Bounded<u32, 2>> for Mode {
+//! type Error = u32;
+//!
+//! fn try_from(v: Bounded<u32, 2>) -> Result<Self, u32> {
+//! match *v {
+//! 0 => Ok(Mode::Low),
+//! 1 => Ok(Mode::High),
+//! 2 => Ok(Mode::Auto),
+//! n => Err(n),
+//! }
+//! }
+//! }
+//!
+//! impl From<Mode> for Bounded<u32, 2> {
+//! fn from(m: Mode) -> Self {
+//! match m {
+//! Mode::Low => Bounded::<u32, _>::new::<0>(),
+//! Mode::High => Bounded::<u32, _>::new::<1>(),
+//! Mode::Auto => Bounded::<u32, _>::new::<2>(),
+//! }
+//! }
+//! }
+//!
+//! bitfield! {
+//! pub struct Config(u32) {
+//! 1:0 mode ?=> Mode;
+//! }
+//! }
+//!
+//! let cfg = Config::zeroed().with_mode(Mode::Auto);
+//! assert_eq!(cfg.mode(), Ok(Mode::Auto));
+//!
+//! // Invalid bit pattern returns an error.
+//! assert_eq!(Config::from(0b11).mode(), Err(3));
+//! ```
+//!
+//! # Bits outside of declared fields
+//!
+//! Bits of the storage type that are not part of any declared field are preserved by the setter
+//! methods, and can only be modified through `from_raw` or the [`From`] implementation from the
+//! storage type.
+//!
+//! ```rust
+//! use kernel::bitfield;
+//!
+//! bitfield! {
+//! pub struct Sparse(u8) {
+//! 7:6 high;
+//! // Bits 5:1 are not covered by any field.
+//! 0:0 low;
+//! }
+//! }
+//!
+//! // Set the gap bits via `from_raw`, then mutate the declared fields.
+//! let val = Sparse::from_raw(0b0010_1010)
+//! .with_const_high::<0b11>()
+//! .with_low(true);
+//!
+//! // Bits 5:1 are unchanged.
+//! assert_eq!(val.into_raw(), 0b1110_1011);
+//! ```
+//!
+//! # Signed field values
+//!
+//! Bitfield storage types are unsigned. Since field getter methods return a [`Bounded`] of the
+//! storage type, fields are also unsigned by default.
+//!
+//! If a field needs to encode a signed value, use a custom conversion type with `=>` or `?=>` to
+//! perform the sign interpretation explicitly.
+//!
+//! [`Bounded`]: kernel::num::Bounded
+
+/// Defines a bitfield struct with bounds-checked accessors for individual bit ranges.
+///
+/// See the [`mod@kernel::bitfield`] module for full documentation and examples.
+#[macro_export]
+macro_rules! bitfield {
+ // Entry point defining the bitfield struct, its implementations and its field accessors.
+ (
+ $(#[$attr:meta])* $vis:vis struct $name:ident($storage:ty) { $($fields:tt)* }
+ ) => {
+ $crate::bitfield!(@core
+ #[allow(non_camel_case_types)]
+ $(#[$attr])* $vis $name $storage
+ );
+ $crate::bitfield!(@fields $vis $name $storage { $($fields)* });
+ };
+
+ // All rules below are helpers.
+
+ // Defines the wrapper `$name` type and its conversions from/to the storage type.
+ (@core $(#[$attr:meta])* $vis:vis $name:ident $storage:ty) => {
+ $(#[$attr])*
+ #[repr(transparent)]
+ #[derive(Clone, Copy, PartialEq, Eq)]
+ $vis struct $name {
+ inner: $storage,
+ }
+
+ #[allow(dead_code)]
+ impl $name {
+ /// Creates a bitfield from a raw value.
+ #[inline(always)]
+ $vis const fn from_raw(value: $storage) -> Self {
+ Self{ inner: value }
+ }
+
+ /// Turns this bitfield into its raw value.
+ ///
+ /// This is similar to the [`From`] implementation, but is shorter to invoke in
+ /// most cases.
+ #[inline(always)]
+ $vis const fn into_raw(self) -> $storage {
+ self.inner
+ }
+ }
+
+ // SAFETY: `$storage` is `Zeroable` and `$name` is transparent.
+ unsafe impl ::pin_init::Zeroable for $name {}
+
+ impl ::core::convert::From<$name> for $storage {
+ #[inline(always)]
+ fn from(val: $name) -> $storage {
+ val.into_raw()
+ }
+ }
+
+ impl ::core::convert::From<$storage> for $name {
+ #[inline(always)]
+ fn from(val: $storage) -> $name {
+ Self::from_raw(val)
+ }
+ }
+ };
+
+ // Definitions requiring knowledge of individual fields: private and public field accessors,
+ // and `Debug` implementation.
+ (@fields $vis:vis $name:ident $storage:ty {
+ $($(#[doc = $doc:expr])* $hi:literal:$lo:literal $field:ident
+ $(?=> $try_into_type:ty)?
+ $(=> $into_type:ty)?
+ ;
+ )*
+ }
+ ) => {
+ #[allow(dead_code)]
+ impl $name {
+ $(
+ $crate::bitfield!(@private_field_accessors $vis $name $storage : $hi:$lo $field);
+ $crate::bitfield!(
+ @public_field_accessors $(#[doc = $doc])* $vis $name $storage : $hi:$lo $field
+ $(?=> $try_into_type)?
+ $(=> $into_type)?
+ );
+ )*
+ }
+
+ $crate::bitfield!(@debug $name { $($field;)* });
+ };
+
+ // Private field accessors working with the exact `Bounded` type for the field.
+ (
+ @private_field_accessors $vis:vis $name:ident $storage:ty : $hi:tt:$lo:tt $field:ident
+ ) => {
+ ::kernel::macros::paste!(
+ $vis const [<$field:upper _RANGE>]: ::core::ops::RangeInclusive<u8> = $lo..=$hi;
+ $vis const [<$field:upper _MASK>]: $storage =
+ ((((1 << $hi) - 1) << 1) + 1) - ((1 << $lo) - 1);
+ $vis const [<$field:upper _SHIFT>]: u32 = $lo;
+ );
+
+ ::kernel::macros::paste!(
+ #[inline(always)]
+ fn [<__ $field>](self) ->
+ ::kernel::num::Bounded<$storage, { $hi + 1 - $lo }> {
+ // Left shift to align the field's MSB with the storage MSB.
+ const ALIGN_TOP: u32 = $storage::BITS - ($hi + 1);
+ // Right shift to move the top-aligned field to bit 0 of the storage.
+ const ALIGN_BOTTOM: u32 = ALIGN_TOP + $lo;
+
+ // Extract the field using two shifts. `Bounded::shr` produces the correctly-sized
+ // output type.
+ let val = ::kernel::num::Bounded::<$storage, { $storage::BITS }>::from(
+ self.inner << ALIGN_TOP
+ );
+ val.shr::<ALIGN_BOTTOM, { $hi + 1 - $lo } >()
+ }
+
+ #[inline(always)]
+ const fn [<__with_ $field>](
+ mut self,
+ value: ::kernel::num::Bounded<$storage, { $hi + 1 - $lo }>,
+ ) -> Self
+ {
+ const MASK: $storage = <$name>::[<$field:upper _MASK>];
+ const SHIFT: u32 = <$name>::[<$field:upper _SHIFT>];
+
+ let value = value.get() << SHIFT;
+ self.inner = (self.inner & !MASK) | value;
+
+ self
+ }
+ );
+ };
+
+ // Public accessors for fields infallibly (`=>`) converted to a type.
+ (
+ @public_field_accessors $(#[doc = $doc:expr])* $vis:vis $name:ident $storage:ty :
+ $hi:literal:$lo:literal $field:ident => $into_type:ty
+ ) => {
+ ::kernel::macros::paste!(
+
+ $(#[doc = $doc])*
+ #[doc = "Returns the value of this field."]
+ #[inline(always)]
+ $vis fn $field(self) -> $into_type
+ {
+ self.[<__ $field>]().into()
+ }
+
+ $(#[doc = $doc])*
+ #[doc = "Sets this field to the given `value`."]
+ #[inline(always)]
+ $vis fn [<with_ $field>](self, value: $into_type) -> Self
+ {
+ self.[<__with_ $field>](value.into())
+ }
+
+ );
+ };
+
+ // Public accessors for fields fallibly (`?=>`) converted to a type.
+ (
+ @public_field_accessors $(#[doc = $doc:expr])* $vis:vis $name:ident $storage:ty :
+ $hi:tt:$lo:tt $field:ident ?=> $try_into_type:ty
+ ) => {
+ ::kernel::macros::paste!(
+
+ $(#[doc = $doc])*
+ #[doc = "Returns the value of this field."]
+ #[inline(always)]
+ $vis fn $field(self) ->
+ ::core::result::Result<
+ $try_into_type,
+ <$try_into_type as ::core::convert::TryFrom<
+ ::kernel::num::Bounded<$storage, { $hi + 1 - $lo }>
+ >>::Error
+ >
+ {
+ self.[<__ $field>]().try_into()
+ }
+
+ $(#[doc = $doc])*
+ #[doc = "Sets this field to the given `value`."]
+ #[inline(always)]
+ $vis fn [<with_ $field>](self, value: $try_into_type) -> Self
+ {
+ self.[<__with_ $field>](value.into())
+ }
+
+ );
+ };
+
+ // Public accessors for fields not converted to a type.
+ (
+ @public_field_accessors $(#[doc = $doc:expr])* $vis:vis $name:ident $storage:ty :
+ $hi:tt:$lo:tt $field:ident
+ ) => {
+ ::kernel::macros::paste!(
+
+ $(#[doc = $doc])*
+ #[doc = "Returns the value of this field."]
+ #[inline(always)]
+ $vis fn $field(self) ->
+ ::kernel::num::Bounded<$storage, { $hi + 1 - $lo }>
+ {
+ self.[<__ $field>]()
+ }
+
+ $(#[doc = $doc])*
+ #[doc = "Sets this field to the compile-time constant `VALUE`."]
+ #[inline(always)]
+ $vis const fn [<with_const_ $field>]<const VALUE: $storage>(self) -> Self {
+ self.[<__with_ $field>](
+ ::kernel::num::Bounded::<$storage, { $hi + 1 - $lo }>::new::<VALUE>()
+ )
+ }
+
+ $(#[doc = $doc])*
+ #[doc = "Sets this field to the given `value`."]
+ #[inline(always)]
+ $vis fn [<with_ $field>]<T>(
+ self,
+ value: T,
+ ) -> Self
+ where T: ::core::convert::Into<::kernel::num::Bounded<$storage, { $hi + 1 - $lo }>>,
+ {
+ self.[<__with_ $field>](value.into())
+ }
+
+ $(#[doc = $doc])*
+ #[doc = "Tries to set this field to `value`, returning an error if it is out of range."]
+ #[inline(always)]
+ $vis fn [<try_with_ $field>]<T>(
+ self,
+ value: T,
+ ) -> ::kernel::error::Result<Self>
+ where T: ::kernel::num::TryIntoBounded<$storage, { $hi + 1 - $lo }>,
+ {
+ Ok(
+ self.[<__with_ $field>](
+ value.try_into_bounded().ok_or(::kernel::error::code::EOVERFLOW)?
+ )
+ )
+ }
+
+ );
+ };
+
+ // `Debug` implementation.
+ (@debug $name:ident { $($field:ident;)* }) => {
+ impl ::kernel::fmt::Debug for $name {
+ fn fmt(&self, f: &mut ::kernel::fmt::Formatter<'_>) -> ::kernel::fmt::Result {
+ f.debug_struct(stringify!($name))
+ .field("<raw>", &::kernel::prelude::fmt!("{:#x}", self.inner))
+ $(
+ .field(stringify!($field), &self.$field())
+ )*
+ .finish()
+ }
+ }
+ };
+}
+
+#[cfg(CONFIG_RUST_BITFIELD_KUNIT_TEST)]
+#[::kernel::macros::kunit_tests(rust_kernel_bitfield)]
+mod tests {
+ use core::convert::TryFrom;
+
+ use pin_init::Zeroable;
+
+ use kernel::num::Bounded;
+
+ // Enum types for testing `=>` and `?=>` conversions.
+
+ #[derive(Debug, Clone, Copy, PartialEq)]
+ enum MemoryType {
+ Unmapped = 0,
+ Normal = 1,
+ Device = 2,
+ Reserved = 3,
+ }
+
+ impl TryFrom<Bounded<u64, 4>> for MemoryType {
+ type Error = u64;
+ fn try_from(value: Bounded<u64, 4>) -> Result<Self, Self::Error> {
+ match value.get() {
+ 0 => Ok(MemoryType::Unmapped),
+ 1 => Ok(MemoryType::Normal),
+ 2 => Ok(MemoryType::Device),
+ 3 => Ok(MemoryType::Reserved),
+ _ => Err(value.get()),
+ }
+ }
+ }
+
+ impl From<MemoryType> for Bounded<u64, 4> {
+ fn from(mt: MemoryType) -> Bounded<u64, 4> {
+ Bounded::from_expr(mt as u64)
+ }
+ }
+
+ #[derive(Debug, Clone, Copy, PartialEq)]
+ enum Priority {
+ Low = 0,
+ Medium = 1,
+ High = 2,
+ Critical = 3,
+ }
+
+ impl From<Bounded<u16, 2>> for Priority {
+ fn from(value: Bounded<u16, 2>) -> Self {
+ match value & 0x3 {
+ 0 => Priority::Low,
+ 1 => Priority::Medium,
+ 2 => Priority::High,
+ _ => Priority::Critical,
+ }
+ }
+ }
+
+ impl From<Priority> for Bounded<u16, 2> {
+ fn from(p: Priority) -> Bounded<u16, 2> {
+ Bounded::from_expr(p as u16)
+ }
+ }
+
+ bitfield! {
+ struct TestU64(u64) {
+ 63:63 field_63;
+ 61:52 field_61_52;
+ 51:16 field_51_16;
+ 15:12 field_15_12 ?=> MemoryType;
+ 11:9 field_11_9;
+ 1:1 field_1;
+ 0:0 field_0;
+ }
+ }
+
+ bitfield! {
+ struct TestU16(u16) {
+ 15:8 field_15_8;
+ 7:4 field_7_4; // Partial overlap with `field_5_4`.
+ 5:4 field_5_4 => Priority;
+ 3:1 field_3_1;
+ 0:0 field_0;
+ }
+ }
+
+ bitfield! {
+ struct TestU8(u8) {
+ 7:0 field_7_0; // Full byte overlap.
+ 7:4 field_7_4;
+ 3:2 field_3_2;
+ 1:1 field_1;
+ 0:0 field_0;
+ }
+ }
+
+ // Single and multi-bit fields basic access.
+ #[test]
+ fn test_basic_access() {
+ // `TestU64`.
+ let mut val = TestU64::zeroed();
+ assert_eq!(val.into_raw(), 0x0);
+
+ val = val.with_field_0(true);
+ assert!(val.field_0().into_bool());
+ assert_eq!(val.into_raw(), 0x1);
+
+ val = val.with_field_1(true);
+ assert!(val.field_1().into_bool());
+ val = val.with_field_1(false);
+ assert!(!val.field_1().into_bool());
+ assert_eq!(val.into_raw(), 0x1);
+
+ val = val.with_const_field_11_9::<0x5>();
+ assert_eq!(val.field_11_9(), 0x5);
+ assert_eq!(val.into_raw(), 0xA01);
+
+ val = val.with_const_field_51_16::<0x123456>();
+ assert_eq!(val.field_51_16(), 0x123456);
+ assert_eq!(val.into_raw(), 0x0012_3456_0A01);
+
+ const MAX_FIELD_51_16: u64 = ::kernel::bits::genmask_u64(0..=35);
+ val = val.with_const_field_51_16::<{ MAX_FIELD_51_16 }>();
+ assert_eq!(val.field_51_16(), MAX_FIELD_51_16);
+
+ val = val.with_const_field_61_52::<0x3FF>();
+ assert_eq!(val.field_61_52(), 0x3FF);
+
+ val = val.with_field_63(true);
+ assert!(val.field_63().into_bool());
+
+ // `TestU16`.
+ let mut val = TestU16::zeroed();
+ assert_eq!(val.into_raw(), 0x0);
+
+ val = val.with_field_0(true);
+ assert!(val.field_0().into_bool());
+ assert_eq!(val.into_raw(), 0x1);
+
+ val = val.with_const_field_3_1::<0x5>();
+ assert_eq!(val.field_3_1(), 0x5);
+ assert_eq!(val.into_raw(), 0xB);
+
+ val = val.with_const_field_7_4::<0xA>();
+ assert_eq!(val.field_7_4(), 0xA);
+ assert_eq!(val.into_raw(), 0xAB);
+
+ val = val.with_const_field_15_8::<0x42>();
+ assert_eq!(val.field_15_8(), 0x42);
+ assert_eq!(val.into_raw(), 0x42AB);
+
+ // `TestU8`.
+ let mut val = TestU8::zeroed();
+ assert_eq!(val.into_raw(), 0x0);
+
+ val = val.with_field_0(true);
+ assert!(val.field_0().into_bool());
+ assert_eq!(val.into_raw(), 0x1);
+
+ val = val.with_field_1(true);
+ assert!(val.field_1().into_bool());
+ assert_eq!(val.into_raw(), 0x3);
+
+ val = val.with_const_field_3_2::<0x3>();
+ assert_eq!(val.field_3_2(), 0x3);
+ assert_eq!(val.into_raw(), 0xF);
+
+ val = val.with_const_field_7_4::<0xA>();
+ assert_eq!(val.field_7_4(), 0xA);
+ assert_eq!(val.into_raw(), 0xAF);
+ }
+
+ // `=>` infallible conversion.
+ #[test]
+ fn test_infallible_conversion() {
+ let mut val = TestU16::zeroed();
+
+ val = val.with_field_5_4(Priority::Low);
+ assert_eq!(val.field_5_4(), Priority::Low);
+ assert_eq!(val.into_raw() & 0x30, 0x00);
+
+ val = val.with_field_5_4(Priority::Medium);
+ assert_eq!(val.field_5_4(), Priority::Medium);
+ assert_eq!(val.into_raw() & 0x30, 0x10);
+
+ val = val.with_field_5_4(Priority::High);
+ assert_eq!(val.field_5_4(), Priority::High);
+ assert_eq!(val.into_raw() & 0x30, 0x20);
+
+ val = val.with_field_5_4(Priority::Critical);
+ assert_eq!(val.field_5_4(), Priority::Critical);
+ assert_eq!(val.into_raw() & 0x30, 0x30);
+ }
+
+ // `?=>` fallible conversion.
+ #[test]
+ fn test_fallible_conversion() {
+ let mut val = TestU64::zeroed();
+
+ val = val.with_field_15_12(MemoryType::Unmapped);
+ assert_eq!(val.field_15_12(), Ok(MemoryType::Unmapped));
+ val = val.with_field_15_12(MemoryType::Normal);
+ assert_eq!(val.field_15_12(), Ok(MemoryType::Normal));
+ val = val.with_field_15_12(MemoryType::Device);
+ assert_eq!(val.field_15_12(), Ok(MemoryType::Device));
+ val = val.with_field_15_12(MemoryType::Reserved);
+ assert_eq!(val.field_15_12(), Ok(MemoryType::Reserved));
+
+ // `field_15_12` is 4 bits wide (0-15); `MemoryType` only covers 0-3, so 4-15 return `Err`.
+ let raw = (val.into_raw() & !::kernel::bits::genmask_u64(12..=15)) | (0x7 << 12);
+ assert_eq!(TestU64::from_raw(raw).field_15_12(), Err(0x7));
+ }
+
+ // Test that setting an overlapping field affects the overlapped one as expected.
+ #[test]
+ fn test_overlapping_fields() {
+ let mut val = TestU16::zeroed();
+
+ val = val.with_field_5_4(Priority::High); // High == 2 == 0b10.
+ assert_eq!(val.field_5_4(), Priority::High);
+ assert_eq!(val.field_7_4(), 0x2); // Bits 7:6 == 0, bits 5:4 == 0b10.
+
+ val = val.with_const_field_7_4::<0xF>();
+ assert_eq!(val.field_7_4(), 0xF);
+ assert_eq!(val.field_5_4(), Priority::Critical); // Bits 5:4 == 0b11.
+
+ // `field_7_0` should encompass all other fields.
+ let mut val = TestU8::zeroed()
+ .with_field_0(true)
+ .with_field_1(true)
+ .with_const_field_3_2::<0x3>()
+ .with_const_field_7_4::<0xA>();
+ assert_eq!(val.into_raw(), 0xAF);
+
+ val = val.with_field_7_0(0x55);
+ assert_eq!(val.field_7_0(), 0x55);
+ assert!(val.field_0().into_bool());
+ assert!(!val.field_1().into_bool());
+ assert_eq!(val.field_3_2(), 0x1);
+ assert_eq!(val.field_7_4(), 0x5);
+ }
+
+ // Checks that bits not mapped to any field are left untouched.
+ #[test]
+ fn test_unallocated_bits() {
+ let gap_bits = (1u64 << 62) | 0x1FC;
+
+ let set_all_fields = |val: TestU64| {
+ val.with_field_63(true)
+ .with_const_field_61_52::<0x155>()
+ .with_const_field_51_16::<0x123456>()
+ .with_field_15_12(MemoryType::Device)
+ .with_const_field_11_9::<0x5>()
+ .with_field_1(true)
+ .with_field_0(true)
+ };
+
+ // Gap bits to 0.
+ let val = set_all_fields(TestU64::from_raw(0));
+ assert_eq!(val.into_raw() & gap_bits, 0);
+
+ // Gap bits to 1.
+ let val = set_all_fields(TestU64::from_raw(gap_bits));
+ assert_eq!(val.into_raw() & gap_bits, gap_bits);
+ }
+
+ #[test]
+ fn test_try_with() {
+ let val = TestU64::zeroed().try_with_field_51_16(0x123456).unwrap();
+ assert_eq!(val.field_51_16(), 0x123456);
+
+ let err = TestU64::zeroed().try_with_field_51_16(u64::MAX);
+ assert_eq!(err, Err(::kernel::error::code::EOVERFLOW));
+
+ let val = TestU64::zeroed()
+ .try_with_field_51_16(0xABCDEF)
+ .and_then(|p| p.try_with_field_0(1))
+ .unwrap();
+ assert_eq!(val.field_51_16(), 0xABCDEF);
+ assert!(val.field_0().into_bool());
+ }
+
+ // `from_raw`/`into_raw` and `From`/`Into` round-trips.
+ #[test]
+ fn test_raw() {
+ let raw: u64 = 0xBFF0_0000_3123_3E03;
+ let val = TestU64::from_raw(raw);
+ assert_eq!(u64::from(val), raw);
+ assert!(val.field_0().into_bool());
+ assert!(val.field_1().into_bool());
+ assert_eq!(val.field_11_9(), 0x7);
+ assert_eq!(val.field_51_16(), 0x3123);
+ assert_eq!(val.field_15_12(), Ok(MemoryType::Reserved));
+ assert_eq!(val.field_61_52(), 0x3FF);
+ assert!(val.field_63().into_bool());
+
+ let raw: u16 = 0x42AB;
+ let val = TestU16::from_raw(raw);
+ assert_eq!(u16::from(val), raw);
+ assert!(val.field_0().into_bool());
+ assert_eq!(val.field_3_1(), 0x5);
+ assert_eq!(val.field_7_4(), 0xA);
+ assert_eq!(val.field_15_8(), 0x42);
+
+ let raw: u8 = 0xAF;
+ let val = TestU8::from_raw(raw);
+ assert_eq!(u8::from(val), raw);
+ assert!(val.field_0().into_bool());
+ assert!(val.field_1().into_bool());
+ assert_eq!(val.field_3_2(), 0x3);
+ assert_eq!(val.field_7_4(), 0xA);
+ assert_eq!(val.field_7_0(), 0xAF);
+ }
+}
diff --git a/rust/kernel/bitmap.rs b/rust/kernel/bitmap.rs
index 83d7dea..b27e0ec 100644
--- a/rust/kernel/bitmap.rs
+++ b/rust/kernel/bitmap.rs
@@ -499,9 +499,8 @@ pub fn next_zero_bit(&self, start: usize) -> Option<usize> {
}
}
-use macros::kunit_tests;
-
-#[kunit_tests(rust_kernel_bitmap)]
+#[cfg(CONFIG_RUST_BITMAP_KUNIT_TEST)]
+#[macros::kunit_tests(rust_kernel_bitmap)]
mod tests {
use super::*;
use kernel::alloc::flags::GFP_KERNEL;
diff --git a/rust/kernel/build_assert.rs b/rust/kernel/build_assert.rs
index 2ea2154..c3acb9b 100644
--- a/rust/kernel/build_assert.rs
+++ b/rust/kernel/build_assert.rs
@@ -61,15 +61,16 @@
//! undefined symbols and linker errors, it is not developer friendly to debug, so it is recommended
//! to avoid it and prefer other two assertions where possible.
+#[doc(inline)]
pub use crate::{
- build_assert,
+ build_assert_macro as build_assert,
build_error,
const_assert,
static_assert, //
};
#[doc(hidden)]
-pub use build_error::build_error;
+pub use build_error::build_error as build_error_fn;
/// Static assert (i.e. compile-time assert).
///
@@ -105,6 +106,7 @@
/// static_assert!(f(40) == 42, "f(x) must add 2 to the given input.");
/// ```
#[macro_export]
+#[doc(hidden)]
macro_rules! static_assert {
($condition:expr $(,$arg:literal)?) => {
const _: () = ::core::assert!($condition $(,$arg)?);
@@ -133,6 +135,7 @@ macro_rules! static_assert {
/// }
/// ```
#[macro_export]
+#[doc(hidden)]
macro_rules! const_assert {
($condition:expr $(,$arg:literal)?) => {
const { ::core::assert!($condition $(,$arg)?) };
@@ -157,12 +160,13 @@ macro_rules! const_assert {
/// // foo(usize::MAX); // Fails to compile.
/// ```
#[macro_export]
+#[doc(hidden)]
macro_rules! build_error {
() => {{
- $crate::build_assert::build_error("")
+ $crate::build_assert::build_error_fn("")
}};
($msg:expr) => {{
- $crate::build_assert::build_error($msg)
+ $crate::build_assert::build_error_fn($msg)
}};
}
@@ -200,15 +204,16 @@ macro_rules! build_error {
/// const _: () = const_bar(2);
/// ```
#[macro_export]
-macro_rules! build_assert {
+#[doc(hidden)]
+macro_rules! build_assert_macro {
($cond:expr $(,)?) => {{
if !$cond {
- $crate::build_assert::build_error(concat!("assertion failed: ", stringify!($cond)));
+ $crate::build_assert::build_error_fn(concat!("assertion failed: ", stringify!($cond)));
}
}};
($cond:expr, $msg:expr) => {{
if !$cond {
- $crate::build_assert::build_error($msg);
+ $crate::build_assert::build_error_fn($msg);
}
}};
}
diff --git a/rust/kernel/cpufreq.rs b/rust/kernel/cpufreq.rs
index d8d2687..a20bd50 100644
--- a/rust/kernel/cpufreq.rs
+++ b/rust/kernel/cpufreq.rs
@@ -1323,7 +1323,7 @@ impl<T: Driver> Registration<T> {
// SAFETY: The C API guarantees that `cpu` refers to a valid CPU number.
let cpu_id = unsafe { CpuId::from_u32_unchecked(cpu) };
- PolicyCpu::from_cpu(cpu_id).map_or(0, |mut policy| T::get(&mut policy).map_or(0, |f| f))
+ PolicyCpu::from_cpu(cpu_id).map_or(0, |mut policy| T::get(&mut policy).unwrap_or(0))
}
/// Driver's `update_limit` callback.
diff --git a/rust/kernel/dma.rs b/rust/kernel/dma.rs
index 4995ee5..642ccff 100644
--- a/rust/kernel/dma.rs
+++ b/rust/kernel/dma.rs
@@ -1152,8 +1152,8 @@ unsafe impl Sync for CoherentHandle {}
/// unsafe impl kernel::transmute::AsBytes for MyStruct{};
///
/// # fn test(alloc: &kernel::dma::Coherent<[MyStruct]>) -> Result {
-/// let whole = kernel::dma_read!(alloc, [2]?);
-/// let field = kernel::dma_read!(alloc, [1]?.field);
+/// let whole = kernel::dma_read!(alloc, [try: 2]);
+/// let field = kernel::dma_read!(alloc, [panic: 1].field);
/// # Ok::<(), Error>(()) }
/// ```
#[macro_export]
@@ -1189,8 +1189,8 @@ macro_rules! dma_read {
/// unsafe impl kernel::transmute::AsBytes for MyStruct{};
///
/// # fn test(alloc: &kernel::dma::Coherent<[MyStruct]>) -> Result {
-/// kernel::dma_write!(alloc, [2]?.member, 0xf);
-/// kernel::dma_write!(alloc, [1]?, MyStruct { member: 0xf });
+/// kernel::dma_write!(alloc, [try: 2].member, 0xf);
+/// kernel::dma_write!(alloc, [panic: 1], MyStruct { member: 0xf });
/// # Ok::<(), Error>(()) }
/// ```
#[macro_export]
@@ -1207,11 +1207,8 @@ macro_rules! dma_write {
(@parse [$dma:expr] [$($proj:tt)*] [.$field:tt $($rest:tt)*]) => {
$crate::dma_write!(@parse [$dma] [$($proj)* .$field] [$($rest)*])
};
- (@parse [$dma:expr] [$($proj:tt)*] [[$index:expr]? $($rest:tt)*]) => {
- $crate::dma_write!(@parse [$dma] [$($proj)* [$index]?] [$($rest)*])
- };
- (@parse [$dma:expr] [$($proj:tt)*] [[$index:expr] $($rest:tt)*]) => {
- $crate::dma_write!(@parse [$dma] [$($proj)* [$index]] [$($rest)*])
+ (@parse [$dma:expr] [$($proj:tt)*] [[$flavor:ident: $index:expr] $($rest:tt)*]) => {
+ $crate::dma_write!(@parse [$dma] [$($proj)* [$flavor: $index]] [$($rest)*])
};
($dma:expr, $($rest:tt)*) => {
$crate::dma_write!(@parse [$dma] [] [$($rest)*])
diff --git a/rust/kernel/error.rs b/rust/kernel/error.rs
index 05cf869..a56ba63 100644
--- a/rust/kernel/error.rs
+++ b/rust/kernel/error.rs
@@ -25,10 +25,8 @@ macro_rules! declare_err {
#[doc = $doc]
)*
pub const $err: super::Error =
- match super::Error::try_from_errno(-(crate::bindings::$err as i32)) {
- Some(err) => err,
- None => panic!("Invalid errno in `declare_err!`"),
- };
+ super::Error::try_from_errno(-(crate::bindings::$err as i32))
+ .expect("Invalid errno in `declare_err!`");
};
}
diff --git a/rust/kernel/fmt.rs b/rust/kernel/fmt.rs
index 1e8725e..73afbc5 100644
--- a/rust/kernel/fmt.rs
+++ b/rust/kernel/fmt.rs
@@ -4,7 +4,14 @@
//!
//! This module is intended to be used in place of `core::fmt` in kernel code.
-pub use core::fmt::{Arguments, Debug, Error, Formatter, Result, Write};
+pub use core::fmt::{
+ Arguments,
+ Debug,
+ Error,
+ Formatter,
+ Result,
+ Write, //
+};
/// Internal adapter used to route and allow implementations of formatting traits for foreign types.
///
@@ -27,7 +34,15 @@ fn fmt(&self, f: &mut Formatter<'_>) -> Result {
};
}
-use core::fmt::{Binary, LowerExp, LowerHex, Octal, Pointer, UpperExp, UpperHex};
+use core::fmt::{
+ Binary,
+ LowerExp,
+ LowerHex,
+ Octal,
+ Pointer,
+ UpperExp,
+ UpperHex, //
+};
impl_fmt_adapter_forward!(Debug, LowerHex, UpperHex, Octal, Binary, Pointer, LowerExp, UpperExp);
/// A copy of [`core::fmt::Display`] that allows us to implement it for foreign types.
diff --git a/rust/kernel/i2c.rs b/rust/kernel/i2c.rs
index 7b908f0..c084a45 100644
--- a/rust/kernel/i2c.rs
+++ b/rust/kernel/i2c.rs
@@ -405,7 +405,9 @@ pub fn get(index: i32) -> Result<ARef<Self>> {
// SAFETY: `adapter` is non-null and points to a live `i2c_adapter`.
// `I2cAdapter` is #[repr(transparent)], so this cast is valid.
- Ok(unsafe { (&*adapter.as_ptr().cast::<I2cAdapter<device::Normal>>()).into() })
+ // `i2c_get_adapter` returned the adapter with an incremented refcount, which we pass to
+ // the `ARef`.
+ Ok(unsafe { ARef::from_raw(adapter.cast::<I2cAdapter<device::Normal>>()) })
}
}
diff --git a/rust/kernel/init.rs b/rust/kernel/init.rs
index 7a0d455..05a12e8 100644
--- a/rust/kernel/init.rs
+++ b/rust/kernel/init.rs
@@ -151,6 +151,7 @@ fn try_pin_init<E>(init: impl PinInit<T, E>, flags: Flags) -> Result<Self::Pinne
/// type.
///
/// If `T: !Unpin` it will not be able to move afterwards.
+ #[inline]
fn pin_init<E>(init: impl PinInit<T, E>, flags: Flags) -> error::Result<Self::PinnedSelf>
where
Error: From<E>,
@@ -168,6 +169,7 @@ fn try_init<E>(init: impl Init<T, E>, flags: Flags) -> Result<Self, E>
E: From<AllocError>;
/// Use the given initializer to in-place initialize a `T`.
+ #[inline]
fn init<E>(init: impl Init<T, E>, flags: Flags) -> error::Result<Self>
where
Error: From<E>,
diff --git a/rust/kernel/io/register.rs b/rust/kernel/io/register.rs
index abc4992..f924c7c 100644
--- a/rust/kernel/io/register.rs
+++ b/rust/kernel/io/register.rs
@@ -108,9 +108,10 @@
use core::marker::PhantomData;
-use crate::io::IoLoc;
-
-use kernel::build_assert;
+use crate::{
+ build_assert::build_assert,
+ io::IoLoc, //
+};
/// Trait implemented by all registers.
pub trait Register: Sized {
@@ -872,7 +873,7 @@ macro_rules! register {
@reg $(#[$attr:meta])* $vis:vis $name:ident ($storage:ty)
[ $size:expr, stride = $stride:expr ] @ $offset:literal { $($fields:tt)* }
) => {
- ::kernel::static_assert!(::core::mem::size_of::<$storage>() <= $stride);
+ $crate::build_assert::static_assert!(::core::mem::size_of::<$storage>() <= $stride);
$crate::register!(@bitfield $(#[$attr])* $vis struct $name($storage) { $($fields)* });
$crate::register!(@io_base $name($storage) @ $offset);
@@ -895,7 +896,9 @@ macro_rules! register {
@reg $(#[$attr:meta])* $vis:vis $name:ident ($storage:ty) => $alias:ident [ $idx:expr ]
{ $($fields:tt)* }
) => {
- ::kernel::static_assert!($idx < <$alias as $crate::io::register::RegisterArray>::SIZE);
+ $crate::build_assert::static_assert!(
+ $idx < <$alias as $crate::io::register::RegisterArray>::SIZE
+ );
$crate::register!(@bitfield $(#[$attr])* $vis struct $name($storage) { $($fields)* });
$crate::register!(
@@ -912,7 +915,7 @@ macro_rules! register {
[ $size:expr, stride = $stride:expr ]
@ $base:ident + $offset:literal { $($fields:tt)* }
) => {
- ::kernel::static_assert!(::core::mem::size_of::<$storage>() <= $stride);
+ $crate::build_assert::static_assert!(::core::mem::size_of::<$storage>() <= $stride);
$crate::register!(@bitfield $(#[$attr])* $vis struct $name($storage) { $($fields)* });
$crate::register!(@io_base $name($storage) @ $offset);
@@ -938,7 +941,9 @@ macro_rules! register {
@reg $(#[$attr:meta])* $vis:vis $name:ident ($storage:ty)
=> $base:ident + $alias:ident [ $idx:expr ] { $($fields:tt)* }
) => {
- ::kernel::static_assert!($idx < <$alias as $crate::io::register::RegisterArray>::SIZE);
+ $crate::build_assert::static_assert!(
+ $idx < <$alias as $crate::io::register::RegisterArray>::SIZE
+ );
$crate::register!(@bitfield $(#[$attr])* $vis struct $name($storage) { $($fields)* });
$crate::register!(
@@ -956,11 +961,10 @@ macro_rules! register {
(
@bitfield $(#[$attr:meta])* $vis:vis struct $name:ident($storage:ty) { $($fields:tt)* }
) => {
- $crate::register!(@bitfield_core
+ $crate::bitfield!(
#[allow(non_camel_case_types)]
- $(#[$attr])* $vis $name $storage
+ $(#[$attr])* $vis struct $name($storage) { $($fields)* }
);
- $crate::register!(@bitfield_fields $vis $name $storage { $($fields)* });
};
// Implementations shared by all registers types.
@@ -1016,245 +1020,4 @@ impl $crate::io::register::RegisterArray for $name {
impl $crate::io::register::RelativeRegisterArray for $name {}
};
-
- // Defines the wrapper `$name` type and its conversions from/to the storage type.
- (@bitfield_core $(#[$attr:meta])* $vis:vis $name:ident $storage:ty) => {
- $(#[$attr])*
- #[repr(transparent)]
- #[derive(Clone, Copy, PartialEq, Eq)]
- $vis struct $name {
- inner: $storage,
- }
-
- #[allow(dead_code)]
- impl $name {
- /// Creates a bitfield from a raw value.
- #[inline(always)]
- $vis const fn from_raw(value: $storage) -> Self {
- Self{ inner: value }
- }
-
- /// Turns this bitfield into its raw value.
- ///
- /// This is similar to the [`From`] implementation, but is shorter to invoke in
- /// most cases.
- #[inline(always)]
- $vis const fn into_raw(self) -> $storage {
- self.inner
- }
- }
-
- // SAFETY: `$storage` is `Zeroable` and `$name` is transparent.
- unsafe impl ::pin_init::Zeroable for $name {}
-
- impl ::core::convert::From<$name> for $storage {
- #[inline(always)]
- fn from(val: $name) -> $storage {
- val.into_raw()
- }
- }
-
- impl ::core::convert::From<$storage> for $name {
- #[inline(always)]
- fn from(val: $storage) -> $name {
- Self::from_raw(val)
- }
- }
- };
-
- // Definitions requiring knowledge of individual fields: private and public field accessors,
- // and `Debug` implementation.
- (@bitfield_fields $vis:vis $name:ident $storage:ty {
- $($(#[doc = $doc:expr])* $hi:literal:$lo:literal $field:ident
- $(?=> $try_into_type:ty)?
- $(=> $into_type:ty)?
- ;
- )*
- }
- ) => {
- #[allow(dead_code)]
- impl $name {
- $(
- $crate::register!(@private_field_accessors $vis $name $storage : $hi:$lo $field);
- $crate::register!(
- @public_field_accessors $(#[doc = $doc])* $vis $name $storage : $hi:$lo $field
- $(?=> $try_into_type)?
- $(=> $into_type)?
- );
- )*
- }
-
- $crate::register!(@debug $name { $($field;)* });
- };
-
- // Private field accessors working with the exact `Bounded` type for the field.
- (
- @private_field_accessors $vis:vis $name:ident $storage:ty : $hi:tt:$lo:tt $field:ident
- ) => {
- ::kernel::macros::paste!(
- $vis const [<$field:upper _RANGE>]: ::core::ops::RangeInclusive<u8> = $lo..=$hi;
- $vis const [<$field:upper _MASK>]: $storage =
- ((((1 << $hi) - 1) << 1) + 1) - ((1 << $lo) - 1);
- $vis const [<$field:upper _SHIFT>]: u32 = $lo;
- );
-
- ::kernel::macros::paste!(
- fn [<__ $field>](self) ->
- ::kernel::num::Bounded<$storage, { $hi + 1 - $lo }> {
- // Left shift to align the field's MSB with the storage MSB.
- const ALIGN_TOP: u32 = $storage::BITS - ($hi + 1);
- // Right shift to move the top-aligned field to bit 0 of the storage.
- const ALIGN_BOTTOM: u32 = ALIGN_TOP + $lo;
-
- // Extract the field using two shifts. `Bounded::shr` produces the correctly-sized
- // output type.
- let val = ::kernel::num::Bounded::<$storage, { $storage::BITS }>::from(
- self.inner << ALIGN_TOP
- );
- val.shr::<ALIGN_BOTTOM, { $hi + 1 - $lo } >()
- }
-
- const fn [<__with_ $field>](
- mut self,
- value: ::kernel::num::Bounded<$storage, { $hi + 1 - $lo }>,
- ) -> Self
- {
- const MASK: $storage = <$name>::[<$field:upper _MASK>];
- const SHIFT: u32 = <$name>::[<$field:upper _SHIFT>];
-
- let value = value.get() << SHIFT;
- self.inner = (self.inner & !MASK) | value;
-
- self
- }
- );
- };
-
- // Public accessors for fields infallibly (`=>`) converted to a type.
- (
- @public_field_accessors $(#[doc = $doc:expr])* $vis:vis $name:ident $storage:ty :
- $hi:literal:$lo:literal $field:ident => $into_type:ty
- ) => {
- ::kernel::macros::paste!(
-
- $(#[doc = $doc])*
- #[doc = "Returns the value of this field."]
- #[inline(always)]
- $vis fn $field(self) -> $into_type
- {
- self.[<__ $field>]().into()
- }
-
- $(#[doc = $doc])*
- #[doc = "Sets this field to the given `value`."]
- #[inline(always)]
- $vis fn [<with_ $field>](self, value: $into_type) -> Self
- {
- self.[<__with_ $field>](value.into())
- }
-
- );
- };
-
- // Public accessors for fields fallibly (`?=>`) converted to a type.
- (
- @public_field_accessors $(#[doc = $doc:expr])* $vis:vis $name:ident $storage:ty :
- $hi:tt:$lo:tt $field:ident ?=> $try_into_type:ty
- ) => {
- ::kernel::macros::paste!(
-
- $(#[doc = $doc])*
- #[doc = "Returns the value of this field."]
- #[inline(always)]
- $vis fn $field(self) ->
- Result<
- $try_into_type,
- <$try_into_type as ::core::convert::TryFrom<
- ::kernel::num::Bounded<$storage, { $hi + 1 - $lo }>
- >>::Error
- >
- {
- self.[<__ $field>]().try_into()
- }
-
- $(#[doc = $doc])*
- #[doc = "Sets this field to the given `value`."]
- #[inline(always)]
- $vis fn [<with_ $field>](self, value: $try_into_type) -> Self
- {
- self.[<__with_ $field>](value.into())
- }
-
- );
- };
-
- // Public accessors for fields not converted to a type.
- (
- @public_field_accessors $(#[doc = $doc:expr])* $vis:vis $name:ident $storage:ty :
- $hi:tt:$lo:tt $field:ident
- ) => {
- ::kernel::macros::paste!(
-
- $(#[doc = $doc])*
- #[doc = "Returns the value of this field."]
- #[inline(always)]
- $vis fn $field(self) ->
- ::kernel::num::Bounded<$storage, { $hi + 1 - $lo }>
- {
- self.[<__ $field>]()
- }
-
- $(#[doc = $doc])*
- #[doc = "Sets this field to the compile-time constant `VALUE`."]
- #[inline(always)]
- $vis const fn [<with_const_ $field>]<const VALUE: $storage>(self) -> Self {
- self.[<__with_ $field>](
- ::kernel::num::Bounded::<$storage, { $hi + 1 - $lo }>::new::<VALUE>()
- )
- }
-
- $(#[doc = $doc])*
- #[doc = "Sets this field to the given `value`."]
- #[inline(always)]
- $vis fn [<with_ $field>]<T>(
- self,
- value: T,
- ) -> Self
- where T: Into<::kernel::num::Bounded<$storage, { $hi + 1 - $lo }>>,
- {
- self.[<__with_ $field>](value.into())
- }
-
- $(#[doc = $doc])*
- #[doc = "Tries to set this field to `value`, returning an error if it is out of range."]
- #[inline(always)]
- $vis fn [<try_with_ $field>]<T>(
- self,
- value: T,
- ) -> ::kernel::error::Result<Self>
- where T: ::kernel::num::TryIntoBounded<$storage, { $hi + 1 - $lo }>,
- {
- Ok(
- self.[<__with_ $field>](
- value.try_into_bounded().ok_or(::kernel::error::code::EOVERFLOW)?
- )
- )
- }
-
- );
- };
-
- // `Debug` implementation.
- (@debug $name:ident { $($field:ident;)* }) => {
- impl ::kernel::fmt::Debug for $name {
- fn fmt(&self, f: &mut ::kernel::fmt::Formatter<'_>) -> ::kernel::fmt::Result {
- f.debug_struct(stringify!($name))
- .field("<raw>", &::kernel::prelude::fmt!("{:#x}", self.inner))
- $(
- .field(stringify!($field), &self.$field())
- )*
- .finish()
- }
- }
- };
}
diff --git a/rust/kernel/io/resource.rs b/rust/kernel/io/resource.rs
index b7ac9fa..17b0c17 100644
--- a/rust/kernel/io/resource.rs
+++ b/rust/kernel/io/resource.rs
@@ -229,7 +229,7 @@ impl Flags {
// Always inline to optimize out error path of `build_assert`.
#[inline(always)]
const fn new(value: u32) -> Self {
- crate::build_assert!(value as u64 <= c_ulong::MAX as u64);
+ build_assert!(value as u64 <= c_ulong::MAX as u64);
Flags(value as c_ulong)
}
}
diff --git a/rust/kernel/ioctl.rs b/rust/kernel/ioctl.rs
index 2fc7662..5bb5b48 100644
--- a/rust/kernel/ioctl.rs
+++ b/rust/kernel/ioctl.rs
@@ -6,7 +6,7 @@
#![expect(non_snake_case)]
-use crate::build_assert;
+use crate::build_assert::build_assert;
/// Build an ioctl number, analogous to the C macro of the same name.
#[inline(always)]
diff --git a/rust/kernel/kunit.rs b/rust/kernel/kunit.rs
index a1edf74..cdee5f2 100644
--- a/rust/kernel/kunit.rs
+++ b/rust/kernel/kunit.rs
@@ -329,6 +329,7 @@ pub fn in_kunit_test() -> bool {
!unsafe { bindings::kunit_get_current_test() }.is_null()
}
+#[cfg(CONFIG_RUST_KUNIT_SELFTEST)]
#[kunit_tests(rust_kernel_kunit)]
mod tests {
use super::*;
diff --git a/rust/kernel/lib.rs b/rust/kernel/lib.rs
index b72b2fb..9512af7 100644
--- a/rust/kernel/lib.rs
+++ b/rust/kernel/lib.rs
@@ -44,6 +44,7 @@
pub mod alloc;
#[cfg(CONFIG_AUXILIARY_BUS)]
pub mod auxiliary;
+pub mod bitfield;
pub mod bitmap;
pub mod bits;
#[cfg(CONFIG_BLOCK)]
diff --git a/rust/kernel/net/phy/reg.rs b/rust/kernel/net/phy/reg.rs
index a7db006..80e22c2 100644
--- a/rust/kernel/net/phy/reg.rs
+++ b/rust/kernel/net/phy/reg.rs
@@ -9,9 +9,11 @@
//! defined in IEEE 802.3.
use super::Device;
-use crate::build_assert;
-use crate::error::*;
-use crate::uapi;
+use crate::{
+ build_assert::build_assert,
+ error::*,
+ uapi, //
+};
mod private {
/// Marker that a trait cannot be implemented outside of this crate
diff --git a/rust/kernel/num/bounded.rs b/rust/kernel/num/bounded.rs
index f9f90d6..dafe777 100644
--- a/rust/kernel/num/bounded.rs
+++ b/rust/kernel/num/bounded.rs
@@ -364,7 +364,7 @@ pub fn try_new(value: T) -> Option<Self> {
// Always inline to optimize out error path of `build_assert`.
#[inline(always)]
pub fn from_expr(expr: T) -> Self {
- crate::build_assert!(
+ crate::build_assert::build_assert!(
fits_within(expr, N),
"Requested value larger than maximal representable value."
);
diff --git a/rust/kernel/page.rs b/rust/kernel/page.rs
index adecb20..8affd82 100644
--- a/rust/kernel/page.rs
+++ b/rust/kernel/page.rs
@@ -3,17 +3,25 @@
//! Kernel page allocation and management.
use crate::{
- alloc::{AllocError, Flags},
+ alloc::{
+ AllocError,
+ Flags, //
+ },
bindings,
- error::code::*,
- error::Result,
- uaccess::UserSliceReader,
+ error::{
+ code::*,
+ Result, //
+ },
+ uaccess::UserSliceReader, //
};
use core::{
marker::PhantomData,
mem::ManuallyDrop,
ops::Deref,
- ptr::{self, NonNull},
+ ptr::{
+ self,
+ NonNull, //
+ }, //
};
/// A bitwise shift for the page size.
diff --git a/rust/kernel/prelude.rs b/rust/kernel/prelude.rs
index 44edf72..8a6da92 100644
--- a/rust/kernel/prelude.rs
+++ b/rust/kernel/prelude.rs
@@ -22,6 +22,7 @@
pin::Pin, //
};
+#[doc(no_inline)]
pub use ::ffi::{
c_char,
c_int,
@@ -47,6 +48,7 @@
vtable, //
};
+#[doc(no_inline)]
pub use pin_init::{
init,
pin_data,
@@ -58,6 +60,13 @@
Zeroable, //
};
+#[doc(no_inline)]
+pub use zerocopy::FromBytes;
+
+#[doc(no_inline)]
+pub use zerocopy_derive::FromBytes;
+
+#[doc(no_inline)]
pub use super::{
alloc::{
flags::*,
@@ -70,9 +79,12 @@
VVec,
Vec, //
},
- build_assert,
- build_error,
- const_assert,
+ build_assert::{
+ build_assert,
+ build_error,
+ const_assert,
+ static_assert, //
+ },
current,
dev_alert,
dev_crit,
@@ -96,7 +108,6 @@
pr_info,
pr_notice,
pr_warn,
- static_assert,
str::CStrExt as _,
try_init,
try_pin_init,
diff --git a/rust/kernel/ptr/projection.rs b/rust/kernel/ptr/projection.rs
index 140ea8e..af72d3b 100644
--- a/rust/kernel/ptr/projection.rs
+++ b/rust/kernel/ptr/projection.rs
@@ -26,14 +26,14 @@ fn from(_: OutOfBound) -> Self {
///
/// # Safety
///
-/// The implementation of `index` and `get` (if [`Some`] is returned) must ensure that, if provided
-/// input pointer `slice` and returned pointer `output`, then:
+/// For a given input pointer `slice` and return value `output`, the implementation of `index`,
+/// `build_index` and `get` (if [`Some`] is returned) must ensure that:
/// - `output` has the same provenance as `slice`;
/// - `output.byte_offset_from(slice)` is between 0 to
/// `KnownSize::size(slice) - KnownSize::size(output)`.
///
-/// This means that if the input pointer is valid, then pointer returned by `get` or `index` is
-/// also valid.
+/// This means that if the input pointer is valid, then the pointer returned by `get`, `index`
+/// or `build_index` is also valid.
#[diagnostic::on_unimplemented(message = "`{Self}` cannot be used to index `{T}`")]
#[doc(hidden)]
pub unsafe trait ProjectIndex<T: ?Sized>: Sized {
@@ -42,10 +42,16 @@ pub unsafe trait ProjectIndex<T: ?Sized>: Sized {
/// Returns an index-projected pointer, if in bounds.
fn get(self, slice: *mut T) -> Option<*mut Self::Output>;
+ /// Returns an index-projected pointer; panic if out of bounds.
+ fn index(self, slice: *mut T) -> *mut Self::Output;
+
/// Returns an index-projected pointer; fail the build if it cannot be proved to be in bounds.
#[inline(always)]
- fn index(self, slice: *mut T) -> *mut Self::Output {
- Self::get(self, slice).unwrap_or_else(|| build_error!())
+ fn build_index(self, slice: *mut T) -> *mut Self::Output {
+ match Self::get(self, slice) {
+ Some(v) => v,
+ None => build_error!(),
+ }
}
}
@@ -67,6 +73,11 @@ fn index(self, slice: *mut T) -> *mut Self::Output {
fn index(self, slice: *mut [T; N]) -> *mut Self::Output {
<I as ProjectIndex<[T]>>::index(self, slice)
}
+
+ #[inline(always)]
+ fn build_index(self, slice: *mut [T; N]) -> *mut Self::Output {
+ <I as ProjectIndex<[T]>>::build_index(self, slice)
+ }
}
// SAFETY: `get`-returned pointer has the same provenance as `slice` and the offset is checked to
@@ -82,6 +93,16 @@ fn get(self, slice: *mut [T]) -> Option<*mut T> {
Some(slice.cast::<T>().wrapping_add(self))
}
}
+
+ #[inline(always)]
+ fn index(self, slice: *mut [T]) -> *mut T {
+ // Leverage Rust built-in operators for bounds checking.
+ // SAFETY: All non-null and aligned pointers are valid for ZST read.
+ let zst_slice =
+ unsafe { core::slice::from_raw_parts::<()>(core::ptr::dangling(), slice.len()) };
+ let () = zst_slice[self];
+ slice.cast::<T>().wrapping_add(self)
+ }
}
// SAFETY: `get`-returned pointer has the same provenance as `slice` and the offset is checked to
@@ -100,6 +121,18 @@ fn get(self, slice: *mut [T]) -> Option<*mut [T]> {
new_len,
))
}
+
+ #[inline(always)]
+ fn index(self, slice: *mut [T]) -> *mut [T] {
+ // Leverage Rust built-in operators for bounds checking.
+ // SAFETY: All non-null and aligned pointers are valid for ZST read.
+ let zst_slice =
+ unsafe { core::slice::from_raw_parts::<()>(core::ptr::dangling(), slice.len()) };
+ _ = zst_slice[self.clone()];
+
+ // SAFETY: Bounds checked.
+ unsafe { self.get(slice).unwrap_unchecked() }
+ }
}
// SAFETY: Safety requirement guaranteed by the forwarded impl.
@@ -110,6 +143,11 @@ unsafe impl<T> ProjectIndex<[T]> for core::ops::RangeTo<usize> {
fn get(self, slice: *mut [T]) -> Option<*mut [T]> {
(0..self.end).get(slice)
}
+
+ #[inline(always)]
+ fn index(self, slice: *mut [T]) -> *mut [T] {
+ (0..self.end).index(slice)
+ }
}
// SAFETY: Safety requirement guaranteed by the forwarded impl.
@@ -120,6 +158,11 @@ unsafe impl<T> ProjectIndex<[T]> for core::ops::RangeFrom<usize> {
fn get(self, slice: *mut [T]) -> Option<*mut [T]> {
(self.start..slice.len()).get(slice)
}
+
+ #[inline(always)]
+ fn index(self, slice: *mut [T]) -> *mut [T] {
+ (self.start..slice.len()).index(slice)
+ }
}
// SAFETY: `get` returned the pointer as is, so it always has the same provenance and offset of 0.
@@ -130,6 +173,11 @@ unsafe impl<T> ProjectIndex<[T]> for core::ops::RangeFull {
fn get(self, slice: *mut [T]) -> Option<*mut [T]> {
Some(slice)
}
+
+ #[inline(always)]
+ fn index(self, slice: *mut [T]) -> *mut [T] {
+ slice
+ }
}
/// A helper trait to perform field projection.
@@ -207,10 +255,13 @@ unsafe fn proj<F>(_: *mut Self, _: impl FnOnce(*mut Self) -> *mut F) -> *mut F {
/// If a mutable pointer is needed, the macro input can be prefixed with the `mut` keyword, i.e.
/// `kernel::ptr::project!(mut ptr, projection)`. By default, a const pointer is created.
///
-/// `ptr::project!` macro can perform both fallible indexing and build-time checked indexing.
-/// `[index]` form performs build-time bounds checking; if compiler fails to prove `[index]` is in
-/// bounds, compilation will fail. `[index]?` can be used to perform runtime bounds checking;
-/// `OutOfBound` error is raised via `?` if the index is out of bounds.
+/// The `ptr::project!` macro can perform both fallible indexing and build-time checked indexing.
+/// The syntax is of the form `[<flavor>: index]` where `flavor` indicates the way of handling
+/// index out-of-bounds errors.
+/// - `try` will raise an [`OutOfBound`] error (which is convertible to [`ERANGE`]).
+/// - `build` will use the [`build_assert!`] mechanism to have the compiler validate the index is
+/// in bounds.
+/// - `panic` will cause a Rust [`panic!`] if the index goes out of bounds.
///
/// # Examples
///
@@ -228,17 +279,21 @@ unsafe fn proj<F>(_: *mut Self, _: impl FnOnce(*mut Self) -> *mut F) -> *mut F {
/// }
/// ```
///
-/// Index projections are performed with `[index]`:
+/// Index projections are performed with `[<flavor>: index]`, where `flavor` is `try`, `build` or
+/// `panic`:
///
/// ```
/// fn proj(ptr: *const [u8; 32]) -> Result {
-/// let field_ptr: *const u8 = kernel::ptr::project!(ptr, [1]);
+/// let field_ptr: *const u8 = kernel::ptr::project!(ptr, [build: 1]);
/// // The following invocation, if uncommented, would fail the build.
/// //
-/// // kernel::ptr::project!(ptr, [128]);
+/// // kernel::ptr::project!(ptr, [build: 128]);
///
/// // This will raise an `OutOfBound` error (which is convertible to `ERANGE`).
-/// kernel::ptr::project!(ptr, [128]?);
+/// kernel::ptr::project!(ptr, [try: 128]);
+///
+/// // This will panic at runtime if executed.
+/// kernel::ptr::project!(ptr, [panic: 128]);
/// Ok(())
/// }
/// ```
@@ -248,7 +303,7 @@ unsafe fn proj<F>(_: *mut Self, _: impl FnOnce(*mut Self) -> *mut F) -> *mut F {
/// ```
/// let ptr: *const [u8; 32] = core::ptr::dangling();
/// let field_ptr: Result<*const u8> = (|| -> Result<_> {
-/// Ok(kernel::ptr::project!(ptr, [128]?))
+/// Ok(kernel::ptr::project!(ptr, [try: 128]))
/// })();
/// assert!(field_ptr.is_err());
/// ```
@@ -257,7 +312,7 @@ unsafe fn proj<F>(_: *mut Self, _: impl FnOnce(*mut Self) -> *mut F) -> *mut F {
///
/// ```
/// let ptr: *mut [(u8, u16); 32] = core::ptr::dangling_mut();
-/// let field_ptr: *mut u16 = kernel::ptr::project!(mut ptr, [1].1);
+/// let field_ptr: *mut u16 = kernel::ptr::project!(mut ptr, [build: 1].1);
/// ```
#[macro_export]
macro_rules! project_pointer {
@@ -280,16 +335,22 @@ macro_rules! project_pointer {
$crate::ptr::project!(@gen $ptr, $($rest)*)
};
// Fallible index projection.
- (@gen $ptr:ident, [$index:expr]? $($rest:tt)*) => {
+ (@gen $ptr:ident, [try: $index:expr] $($rest:tt)*) => {
let $ptr = $crate::ptr::projection::ProjectIndex::get($index, $ptr)
.ok_or($crate::ptr::projection::OutOfBound)?;
$crate::ptr::project!(@gen $ptr, $($rest)*)
};
- // Build-time checked index projection.
- (@gen $ptr:ident, [$index:expr] $($rest:tt)*) => {
+ // Panicking index projection.
+ (@gen $ptr:ident, [panic: $index:expr] $($rest:tt)*) => {
let $ptr = $crate::ptr::projection::ProjectIndex::index($index, $ptr);
$crate::ptr::project!(@gen $ptr, $($rest)*)
};
+ // Build-time checked index projection.
+ (@gen $ptr:ident, [build: $index:expr] $($rest:tt)*) => {
+ let $ptr = $crate::ptr::projection::ProjectIndex::build_index($index, $ptr);
+ $crate::ptr::project!(@gen $ptr, $($rest)*)
+ };
+
(mut $ptr:expr, $($proj:tt)*) => {{
let ptr: *mut _ = $ptr;
$crate::ptr::project!(@gen ptr, $($proj)*);
diff --git a/rust/kernel/str.rs b/rust/kernel/str.rs
index 8311d91..b3caa9a 100644
--- a/rust/kernel/str.rs
+++ b/rust/kernel/str.rs
@@ -3,14 +3,27 @@
//! String representations.
use crate::{
- alloc::{flags::*, AllocError, KVec},
- error::{to_result, Result},
- fmt::{self, Write},
- prelude::*,
+ alloc::{
+ AllocError,
+ KVec, //
+ },
+ error::{
+ to_result,
+ Result, //
+ },
+ fmt::{
+ self,
+ Write, //
+ },
+ prelude::*, //
};
use core::{
marker::PhantomData,
- ops::{Deref, DerefMut, Index},
+ ops::{
+ Deref,
+ DerefMut,
+ Index, //
+ }, //
};
pub use crate::prelude::CStr;
@@ -415,6 +428,7 @@ macro_rules! c_str {
}};
}
+#[cfg(CONFIG_RUST_STR_KUNIT_TEST)]
#[kunit_tests(rust_kernel_str)]
mod tests {
use super::*;
diff --git a/rust/kernel/sync/arc.rs b/rust/kernel/sync/arc.rs
index 18d6c0d..5ac4961 100644
--- a/rust/kernel/sync/arc.rs
+++ b/rust/kernel/sync/arc.rs
@@ -712,6 +712,7 @@ fn try_init<E>(init: impl Init<T, E>, flags: Flags) -> Result<Self, E>
impl<T> InPlaceWrite<T> for UniqueArc<MaybeUninit<T>> {
type Initialized = UniqueArc<T>;
+ #[inline]
fn write_init<E>(mut self, init: impl Init<T, E>) -> Result<Self::Initialized, E> {
let slot = self.as_mut_ptr();
// SAFETY: When init errors/panics, slot will get deallocated but not dropped,
@@ -721,6 +722,7 @@ fn write_init<E>(mut self, init: impl Init<T, E>) -> Result<Self::Initialized, E
Ok(unsafe { self.assume_init() })
}
+ #[inline]
fn write_pin_init<E>(mut self, init: impl PinInit<T, E>) -> Result<Pin<Self::Initialized>, E> {
let slot = self.as_mut_ptr();
// SAFETY: When init errors/panics, slot will get deallocated but not dropped,
@@ -758,6 +760,14 @@ pub fn new_uninit(flags: Flags) -> Result<UniqueArc<MaybeUninit<T>>, AllocError>
}
}
+impl<T: ?Sized> UniqueArc<T> {
+ /// Return a raw pointer to the data in this [`UniqueArc`].
+ #[inline]
+ pub fn as_ptr(this: &Self) -> *const T {
+ Arc::as_ptr(&this.inner)
+ }
+}
+
impl<T> UniqueArc<MaybeUninit<T>> {
/// Converts a `UniqueArc<MaybeUninit<T>>` into a `UniqueArc<T>` by writing a value into it.
pub fn write(mut self, value: T) -> UniqueArc<T> {
@@ -782,6 +792,7 @@ pub unsafe fn assume_init(self) -> UniqueArc<T> {
}
/// Initialize `self` using the given initializer.
+ #[inline]
pub fn init_with<E>(mut self, init: impl Init<T, E>) -> core::result::Result<UniqueArc<T>, E> {
// SAFETY: The supplied pointer is valid for initialization.
match unsafe { init.__init(self.as_mut_ptr()) } {
@@ -792,6 +803,7 @@ pub fn init_with<E>(mut self, init: impl Init<T, E>) -> core::result::Result<Uni
}
/// Pin-initialize `self` using the given pin-initializer.
+ #[inline]
pub fn pin_init_with<E>(
mut self,
init: impl PinInit<T, E>,
diff --git a/rust/kernel/sync/aref.rs b/rust/kernel/sync/aref.rs
index 9989f56..b721b2e 100644
--- a/rust/kernel/sync/aref.rs
+++ b/rust/kernel/sync/aref.rs
@@ -17,7 +17,12 @@
//! [`Arc`]: crate::sync::Arc
//! [`Arc<T>`]: crate::sync::Arc
-use core::{marker::PhantomData, mem::ManuallyDrop, ops::Deref, ptr::NonNull};
+use core::{
+ marker::PhantomData,
+ mem::ManuallyDrop,
+ ops::Deref,
+ ptr::NonNull, //
+};
/// Types that are _always_ reference counted.
///
diff --git a/rust/kernel/sync/atomic/internal.rs b/rust/kernel/sync/atomic/internal.rs
index ad810c2..9c8a7a2 100644
--- a/rust/kernel/sync/atomic/internal.rs
+++ b/rust/kernel/sync/atomic/internal.rs
@@ -4,8 +4,11 @@
//!
//! Provides 1:1 mapping to the C atomic operations.
-use crate::bindings;
-use crate::macros::paste;
+use crate::{
+ bindings,
+ build_assert::static_assert,
+ macros::paste, //
+};
use core::cell::UnsafeCell;
use ffi::c_void;
@@ -46,7 +49,7 @@ pub trait AtomicImpl: Sized + Copy + private::Sealed {
// In the future when a CONFIG_ARCH_SUPPORTS_ATOMIC_RMW=n architecture plans to support Rust, the
// load/store helpers that guarantee atomicity against RmW operations (usually via a lock) need to
// be added.
-crate::static_assert!(
+static_assert!(
cfg!(CONFIG_ARCH_SUPPORTS_ATOMIC_RMW),
"The current implementation of atomic i8/i16/ptr relies on the architecure being \
ARCH_SUPPORTS_ATOMIC_RMW"
diff --git a/rust/kernel/sync/atomic/predefine.rs b/rust/kernel/sync/atomic/predefine.rs
index 1d53834..3d63f40 100644
--- a/rust/kernel/sync/atomic/predefine.rs
+++ b/rust/kernel/sync/atomic/predefine.rs
@@ -2,9 +2,7 @@
//! Pre-defined atomic types
-use crate::static_assert;
-use core::mem::{align_of, size_of};
-use ffi::c_void;
+use crate::prelude::*;
// Ensure size and alignment requirements are checked.
static_assert!(size_of::<bool>() == size_of::<i8>());
@@ -154,9 +152,8 @@ fn rhs_into_delta(rhs: usize) -> isize_atomic_repr {
}
}
-use crate::macros::kunit_tests;
-
-#[kunit_tests(rust_atomics)]
+#[cfg(CONFIG_RUST_ATOMICS_KUNIT_TEST)]
+#[macros::kunit_tests(rust_atomics)]
mod tests {
use super::super::*;
diff --git a/rust/kernel/sync/lock/global.rs b/rust/kernel/sync/lock/global.rs
index aecbdc3..ec2dd84 100644
--- a/rust/kernel/sync/lock/global.rs
+++ b/rust/kernel/sync/lock/global.rs
@@ -85,6 +85,7 @@ pub fn lock(&'static self) -> GlobalGuard<B> {
}
/// Try to lock this global lock.
+ #[must_use = "if unused, the lock will be immediately unlocked"]
#[inline]
pub fn try_lock(&'static self) -> Option<GlobalGuard<B>> {
Some(GlobalGuard {
@@ -96,6 +97,7 @@ pub fn try_lock(&'static self) -> Option<GlobalGuard<B>> {
/// A guard for a [`GlobalLock`].
///
/// See [`global_lock!`] for examples.
+#[must_use = "the lock unlocks immediately when the guard is unused"]
pub struct GlobalGuard<B: GlobalLockBackend> {
inner: Guard<'static, B::Item, B::Backend>,
}
diff --git a/rust/kernel/sync/locked_by.rs b/rust/kernel/sync/locked_by.rs
index 61f100a..fb4a143 100644
--- a/rust/kernel/sync/locked_by.rs
+++ b/rust/kernel/sync/locked_by.rs
@@ -3,7 +3,7 @@
//! A wrapper for data protected by a lock that does not wrap it.
use super::{lock::Backend, lock::Lock};
-use crate::build_assert;
+use crate::build_assert::build_assert;
use core::{cell::UnsafeCell, mem::size_of, ptr};
/// Allows access to some data to be serialised by a lock that does not wrap it.
diff --git a/rust/kernel/sync/refcount.rs b/rust/kernel/sync/refcount.rs
index 6c7ae8b..23a5d20 100644
--- a/rust/kernel/sync/refcount.rs
+++ b/rust/kernel/sync/refcount.rs
@@ -4,9 +4,11 @@
//!
//! C header: [`include/linux/refcount.h`](srctree/include/linux/refcount.h)
-use crate::build_assert;
-use crate::sync::atomic::Atomic;
-use crate::types::Opaque;
+use crate::{
+ build_assert::build_assert,
+ sync::atomic::Atomic,
+ types::Opaque, //
+};
/// Atomic reference counter.
///
diff --git a/rust/kernel/xarray.rs b/rust/kernel/xarray.rs
index 46e5f43..987c9c0 100644
--- a/rust/kernel/xarray.rs
+++ b/rust/kernel/xarray.rs
@@ -5,10 +5,16 @@
//! C header: [`include/linux/xarray.h`](srctree/include/linux/xarray.h)
use crate::{
- alloc, bindings, build_assert,
+ alloc,
+ bindings,
+ build_assert::build_assert,
error::{Error, Result},
ffi::c_void,
- types::{ForeignOwnable, NotThreadSafe, Opaque},
+ types::{
+ ForeignOwnable,
+ NotThreadSafe,
+ Opaque, //
+ }, //
};
use core::{iter, marker::PhantomData, pin::Pin, ptr::NonNull};
use pin_init::{pin_data, pin_init, pinned_drop, PinInit};
diff --git a/rust/pin-init/README.md b/rust/pin-init/README.md
index 9095d66..2312c9e 100644
--- a/rust/pin-init/README.md
+++ b/rust/pin-init/README.md
@@ -3,7 +3,7 @@
[](https://deps.rs/repo/github/Rust-for-Linux/pin-init)
[](#nightly-only)
-
+
# `pin-init`
> [!NOTE]
diff --git a/rust/pin-init/examples/big_struct_in_place.rs b/rust/pin-init/examples/big_struct_in_place.rs
index 80f89b5..c051399 100644
--- a/rust/pin-init/examples/big_struct_in_place.rs
+++ b/rust/pin-init/examples/big_struct_in_place.rs
@@ -1,8 +1,5 @@
// SPDX-License-Identifier: Apache-2.0 OR MIT
-#![cfg_attr(USE_RUSTC_FEATURES, feature(lint_reasons))]
-#![cfg_attr(USE_RUSTC_FEATURES, feature(raw_ref_op))]
-
use pin_init::*;
// Struct with size over 1GiB
diff --git a/rust/pin-init/examples/error.rs b/rust/pin-init/examples/error.rs
index 8f4e135..96f0953 100644
--- a/rust/pin-init/examples/error.rs
+++ b/rust/pin-init/examples/error.rs
@@ -11,6 +11,7 @@
pub struct Error;
impl From<Infallible> for Error {
+ #[inline]
fn from(e: Infallible) -> Self {
match e {}
}
@@ -18,6 +19,7 @@ fn from(e: Infallible) -> Self {
#[cfg(feature = "alloc")]
impl From<AllocError> for Error {
+ #[inline]
fn from(_: AllocError) -> Self {
Self
}
diff --git a/rust/pin-init/examples/linked_list.rs b/rust/pin-init/examples/linked_list.rs
index 119169e..424585fe 100644
--- a/rust/pin-init/examples/linked_list.rs
+++ b/rust/pin-init/examples/linked_list.rs
@@ -2,8 +2,6 @@
#![allow(clippy::undocumented_unsafe_blocks)]
#![cfg_attr(feature = "alloc", feature(allocator_api))]
-#![cfg_attr(USE_RUSTC_FEATURES, feature(lint_reasons))]
-#![cfg_attr(USE_RUSTC_FEATURES, feature(raw_ref_op))]
use core::{
cell::Cell,
diff --git a/rust/pin-init/examples/mutex.rs b/rust/pin-init/examples/mutex.rs
index d53671f..35ecb5f 100644
--- a/rust/pin-init/examples/mutex.rs
+++ b/rust/pin-init/examples/mutex.rs
@@ -2,8 +2,6 @@
#![allow(clippy::undocumented_unsafe_blocks)]
#![cfg_attr(feature = "alloc", feature(allocator_api))]
-#![cfg_attr(USE_RUSTC_FEATURES, feature(lint_reasons))]
-#![cfg_attr(USE_RUSTC_FEATURES, feature(raw_ref_op))]
#![allow(clippy::missing_safety_doc)]
use core::{
@@ -220,7 +218,7 @@ fn main() {
for h in handles {
h.join().expect("thread panicked");
}
- println!("{:?}", &*mtx.lock());
+ println!("{:?}", *mtx.lock());
assert_eq!(*mtx.lock(), workload * thread_count * 2);
}
}
diff --git a/rust/pin-init/examples/pthread_mutex.rs b/rust/pin-init/examples/pthread_mutex.rs
index f3b5cc9b..00f457e 100644
--- a/rust/pin-init/examples/pthread_mutex.rs
+++ b/rust/pin-init/examples/pthread_mutex.rs
@@ -3,8 +3,6 @@
// inspired by <https://github.com/nbdd0121/pin-init/blob/trunk/examples/pthread_mutex.rs>
#![allow(clippy::undocumented_unsafe_blocks)]
#![cfg_attr(feature = "alloc", feature(allocator_api))]
-#![cfg_attr(USE_RUSTC_FEATURES, feature(lint_reasons))]
-#![cfg_attr(USE_RUSTC_FEATURES, feature(raw_ref_op))]
#[cfg(not(windows))]
mod pthread_mtx {
@@ -179,7 +177,7 @@ fn main() {
for h in handles {
h.join().expect("thread panicked");
}
- println!("{:?}", &*mtx.lock());
+ println!("{:?}", *mtx.lock());
assert_eq!(*mtx.lock(), workload * thread_count * 2);
}
}
diff --git a/rust/pin-init/examples/static_init.rs b/rust/pin-init/examples/static_init.rs
index f7e53d1..58cd424 100644
--- a/rust/pin-init/examples/static_init.rs
+++ b/rust/pin-init/examples/static_init.rs
@@ -2,8 +2,6 @@
#![allow(clippy::undocumented_unsafe_blocks)]
#![cfg_attr(feature = "alloc", feature(allocator_api))]
-#![cfg_attr(USE_RUSTC_FEATURES, feature(lint_reasons))]
-#![cfg_attr(USE_RUSTC_FEATURES, feature(raw_ref_op))]
#![allow(unused_imports)]
use core::{
@@ -119,7 +117,7 @@ fn main() {
for h in handles {
h.join().expect("thread panicked");
}
- println!("{:?}, {:?}", &*mtx.lock(), &*COUNT.lock());
+ println!("{:?}, {:?}", *mtx.lock(), *COUNT.lock());
assert_eq!(*mtx.lock(), workload * thread_count * 2);
}
}
diff --git a/rust/pin-init/internal/src/diagnostics.rs b/rust/pin-init/internal/src/diagnostics.rs
index 3bdb477..c7d9b3e 100644
--- a/rust/pin-init/internal/src/diagnostics.rs
+++ b/rust/pin-init/internal/src/diagnostics.rs
@@ -3,6 +3,7 @@
use std::fmt::Display;
use proc_macro2::TokenStream;
+use quote::quote_spanned;
use syn::{spanned::Spanned, Error};
pub(crate) struct DiagCtxt(TokenStream);
@@ -15,6 +16,19 @@ pub(crate) fn error(&mut self, span: impl Spanned, msg: impl Display) -> ErrorGu
ErrorGuaranteed(())
}
+ pub(crate) fn warn(&mut self, span: impl Spanned, msg: impl Display) {
+ // Have the message start on a new line for visual clarity.
+ let msg = format!("\n{}", msg);
+ self.0.extend(quote_spanned!(span.span() =>
+ // Approximate using deprecated warning while `proc_macro_diagnostic` is unstable.
+ const _: () = {
+ #[deprecated = #msg]
+ const fn warn() {}
+ warn();
+ };
+ ));
+ }
+
pub(crate) fn with(
fun: impl FnOnce(&mut DiagCtxt) -> Result<TokenStream, ErrorGuaranteed>,
) -> TokenStream {
diff --git a/rust/pin-init/internal/src/init.rs b/rust/pin-init/internal/src/init.rs
index 487ee00..28d3080 100644
--- a/rust/pin-init/internal/src/init.rs
+++ b/rust/pin-init/internal/src/init.rs
@@ -1,7 +1,7 @@
// SPDX-License-Identifier: Apache-2.0 OR MIT
use proc_macro2::{Span, TokenStream};
-use quote::{format_ident, quote, quote_spanned};
+use quote::{format_ident, quote};
use syn::{
braced,
parse::{End, Parse},
@@ -103,17 +103,15 @@ pub(crate) fn expand(
|(_, err)| Box::new(err),
);
let slot = format_ident!("slot");
- let (has_data_trait, data_trait, get_data, init_from_closure) = if pinned {
+ let (has_data_trait, get_data, init_from_closure) = if pinned {
(
format_ident!("HasPinData"),
- format_ident!("PinData"),
format_ident!("__pin_data"),
format_ident!("pin_init_from_closure"),
)
} else {
(
format_ident!("HasInitData"),
- format_ident!("InitData"),
format_ident!("__init_data"),
format_ident!("init_from_closure"),
)
@@ -157,8 +155,7 @@ fn assert_zeroable<T: ?::core::marker::Sized>(_: *mut T)
#path::#get_data()
};
// Ensure that `#data` really is of type `#data` and help with type inference:
- let init = ::pin_init::__internal::#data_trait::make_closure::<_, #error>(
- #data,
+ let init = #data.__make_closure::<_, #error>(
move |slot| {
#zeroable_check
#this
@@ -172,14 +169,7 @@ fn assert_zeroable<T: ?::core::marker::Sized>(_: *mut T)
init(slot).map(|__InitOk| ())
};
// SAFETY: TODO
- let init = unsafe { ::pin_init::#init_from_closure::<_, #error>(init) };
- // FIXME: this let binding is required to avoid a compiler error (cycle when computing the
- // opaque type returned by this function) before Rust 1.81. Remove after MSRV bump.
- #[allow(
- clippy::let_and_return,
- reason = "some clippy versions warn about the let binding"
- )]
- init
+ unsafe { ::pin_init::#init_from_closure::<_, #error>(init) }
}})
}
@@ -238,107 +228,82 @@ fn init_fields(
cfgs.retain(|attr| attr.path().is_ident("cfg"));
cfgs
};
- let init = match kind {
- InitializerKind::Value { ident, value } => {
- let mut value_ident = ident.clone();
- let value_prep = value.as_ref().map(|value| &value.1).map(|value| {
- // Setting the span of `value_ident` to `value`'s span improves error messages
- // when the type of `value` is wrong.
- value_ident.set_span(value.span());
- quote!(let #value_ident = #value;)
+
+ let ident = match kind {
+ InitializerKind::Value { ident, .. } => ident,
+ InitializerKind::Init { ident, .. } => ident,
+ InitializerKind::Code { block, .. } => {
+ res.extend(quote! {
+ #(#attrs)*
+ #[allow(unused_braces)]
+ #block
});
- // Again span for better diagnostics
- let write = quote_spanned!(ident.span()=> ::core::ptr::write);
- quote! {
- #(#attrs)*
- {
- #value_prep
- // SAFETY: TODO
- unsafe { #write(&raw mut (*#slot).#ident, #value_ident) };
- }
- }
+ continue;
}
- InitializerKind::Init { ident, value, .. } => {
- // Again span for better diagnostics
- let init = format_ident!("init", span = value.span());
- let value_init = if pinned {
- quote! {
- // SAFETY:
- // - `slot` is valid, because we are inside of an initializer closure, we
- // return when an error/panic occurs.
- // - We also use `#data` to require the correct trait (`Init` or `PinInit`)
- // for `#ident`.
- unsafe { #data.#ident(&raw mut (*#slot).#ident, #init)? };
- }
- } else {
- quote! {
- // SAFETY: `slot` is valid, because we are inside of an initializer
- // closure, we return when an error/panic occurs.
- unsafe {
- ::pin_init::Init::__init(
- #init,
- &raw mut (*#slot).#ident,
- )?
- };
- }
- };
- quote! {
- #(#attrs)*
- {
- let #init = #value;
- #value_init
- }
- }
- }
- InitializerKind::Code { block: value, .. } => quote! {
- #(#attrs)*
- #[allow(unused_braces)]
- #value
- },
};
- res.extend(init);
- if let Some(ident) = kind.ident() {
- // `mixed_site` ensures that the guard is not accessible to the user-controlled code.
- let guard = format_ident!("__{ident}_guard", span = Span::mixed_site());
- // NOTE: The reference is derived from the guard so that it only lives as long as the
- // guard does and cannot escape the scope. If it's created via `&mut (*#slot).#ident`
- // like the unaligned field guard, it will become effectively `'static`.
- let accessor = if pinned {
- let project_ident = format_ident!("__project_{ident}");
- quote! {
- // SAFETY: the initialization is pinned.
- unsafe { #data.#project_ident(#guard.let_binding()) }
- }
- } else {
- quote! {
- #guard.let_binding()
- }
- };
-
- res.extend(quote! {
- #(#cfgs)*
- // Create the drop guard.
- //
+ let slot = if pinned {
+ quote! {
+ // SAFETY:
+ // - `slot` is valid and properly aligned.
+ // - `make_field_check` checks that `&raw mut (*slot).#ident` is properly aligned.
+ // - `make_field_check` prevents `#ident` from being used twice, therefore
+ // `(*slot).#ident` is exclusively accessed and has not been initialized.
+ (unsafe { #data.#ident(#slot) })
+ }
+ } else {
+ quote! {
+ // For `init!()` macro, everything is unpinned.
// SAFETY:
// - `&raw mut (*slot).#ident` is valid.
// - `make_field_check` checks that `&raw mut (*slot).#ident` is properly aligned.
- // - `(*slot).#ident` has been initialized above.
- // - We only need the ownership to the pointee back when initialization has
- // succeeded, where we `forget` the guard.
- let mut #guard = unsafe {
- ::pin_init::__internal::DropGuard::new(
- &raw mut (*slot).#ident
+ // - `make_field_check` prevents `#ident` from being used twice, therefore
+ // `(*slot).#ident` is exclusively accessed and has not been initialized.
+ (unsafe {
+ ::pin_init::__internal::Slot::<::pin_init::__internal::Unpinned, _>::new(
+ &raw mut (*#slot).#ident
)
- };
+ })
+ }
+ };
- #(#cfgs)*
- #[allow(unused_variables)]
- let #ident = #accessor;
- });
- guards.push(guard);
- guard_attrs.push(cfgs);
- }
+ // `mixed_site` ensures that the guard is not accessible to the user-controlled code.
+ let guard = format_ident!("__{ident}_guard", span = Span::mixed_site());
+
+ let init = match kind {
+ InitializerKind::Value { ident, value } => {
+ let value = value
+ .as_ref()
+ .map(|(_, value)| quote!(#value))
+ .unwrap_or_else(|| quote!(#ident));
+
+ quote! {
+ #(#attrs)*
+ let mut #guard = #slot.write(#value);
+
+ }
+ }
+ InitializerKind::Init { value, .. } => {
+ quote! {
+ #(#attrs)*
+ let mut #guard = #slot.init(#value)?;
+ }
+ }
+ InitializerKind::Code { .. } => unreachable!(),
+ };
+
+ res.extend(quote! {
+ #init
+
+ #(#cfgs)*
+ // Allow `non_snake_case` since the same warning is going to be reported for the struct
+ // field.
+ #[allow(unused_variables, non_snake_case)]
+ let #ident = #guard.let_binding();
+ });
+
+ guards.push(guard);
+ guard_attrs.push(cfgs);
}
quote! {
#res
@@ -389,7 +354,7 @@ fn make_field_check(
::core::ptr::write(slot, #path {
#(
#(#field_attrs)*
- #field_name: ::core::panic!(),
+ #field_name: loop {},
)*
#zeroing_trailer
})
diff --git a/rust/pin-init/internal/src/lib.rs b/rust/pin-init/internal/src/lib.rs
index b08dfe0..60d5093 100644
--- a/rust/pin-init/internal/src/lib.rs
+++ b/rust/pin-init/internal/src/lib.rs
@@ -6,7 +6,6 @@
//! `pin-init` proc macros.
-#![cfg_attr(USE_RUSTC_FEATURES, feature(lint_reasons))]
// Documentation is done in the pin-init crate instead.
#![allow(missing_docs)]
diff --git a/rust/pin-init/internal/src/pin_data.rs b/rust/pin-init/internal/src/pin_data.rs
index 7d87123..9fbbd25 100644
--- a/rust/pin-init/internal/src/pin_data.rs
+++ b/rust/pin-init/internal/src/pin_data.rs
@@ -7,7 +7,7 @@
parse_quote, parse_quote_spanned,
spanned::Spanned,
visit_mut::VisitMut,
- Field, Generics, Ident, Item, PathSegment, Type, TypePath, Visibility, WhereClause,
+ Attribute, Field, Generics, Ident, Item, PathSegment, Type, TypePath, Visibility, WhereClause,
};
use crate::diagnostics::{DiagCtxt, ErrorGuaranteed};
@@ -35,6 +35,12 @@ fn parse(input: syn::parse::ParseStream<'_>) -> syn::Result<Self> {
}
}
+struct FieldInfo<'a> {
+ field: &'a Field,
+ pinned: bool,
+ cfg_attrs: Vec<&'a Attribute>,
+}
+
pub(crate) fn pin_data(
args: Args,
input: Item,
@@ -73,24 +79,37 @@ pub(crate) fn pin_data(
replacer.visit_generics_mut(&mut struct_.generics);
replacer.visit_fields_mut(&mut struct_.fields);
- let fields: Vec<(bool, &Field)> = struct_
+ let fields: Vec<FieldInfo<'_>> = struct_
.fields
.iter_mut()
.map(|field| {
let len = field.attrs.len();
field.attrs.retain(|a| !a.path().is_ident("pin"));
- (len != field.attrs.len(), &*field)
+ let pinned = len != field.attrs.len();
+
+ let cfg_attrs = field
+ .attrs
+ .iter()
+ .filter(|a| a.path().is_ident("cfg"))
+ .collect();
+
+ FieldInfo {
+ field: &*field,
+ pinned,
+ cfg_attrs,
+ }
})
.collect();
- for (pinned, field) in &fields {
- if !pinned && is_phantom_pinned(&field.ty) {
- dcx.error(
- field,
+ for field in &fields {
+ let ident = field.field.ident.as_ref().unwrap();
+
+ if !field.pinned && is_phantom_pinned(&field.field.ty) {
+ dcx.warn(
+ field.field,
format!(
- "The field `{}` of type `PhantomPinned` only has an effect \
+ "The field `{ident}` of type `PhantomPinned` only has an effect \
if it has the `#[pin]` attribute",
- field.ident.as_ref().unwrap(),
),
);
}
@@ -143,7 +162,7 @@ fn is_phantom_pinned(ty: &Type) -> bool {
fn generate_unpin_impl(
ident: &Ident,
generics: &Generics,
- fields: &[(bool, &Field)],
+ fields: &[FieldInfo<'_>],
) -> TokenStream {
let (_, ty_generics, _) = generics.split_for_impl();
let mut generics_with_pin_lt = generics.clone();
@@ -160,19 +179,28 @@ fn generate_unpin_impl(
else {
unreachable!()
};
- let pinned_fields = fields.iter().filter_map(|(b, f)| b.then_some(f));
+ let pinned_fields = fields.iter().filter(|f| f.pinned).map(|f| {
+ let ident = f.field.ident.as_ref().unwrap();
+ let ty = &f.field.ty;
+ let cfg_attrs = &f.cfg_attrs;
+ quote!(
+ #(#cfg_attrs)*
+ #ident: #ty
+ )
+ });
quote! {
// This struct will be used for the unpin analysis. It is needed, because only structurally
// pinned fields are relevant whether the struct should implement `Unpin`.
- #[allow(dead_code)] // The fields below are never used.
+ #[allow(
+ dead_code, // The fields below are never used.
+ non_snake_case // The warning will be emitted on the struct definition.
+ )]
struct __Unpin #generics_with_pin_lt
#where_token
#predicates
{
- __phantom_pin: ::core::marker::PhantomData<fn(&'__pin ()) -> &'__pin ()>,
- __phantom: ::core::marker::PhantomData<
- fn(#ident #ty_generics) -> #ident #ty_generics
- >,
+ __phantom_pin: ::pin_init::__internal::PhantomInvariantLifetime<'__pin>,
+ __phantom: ::pin_init::__internal::PhantomInvariant<#ident #ty_generics>,
#(#pinned_fields),*
}
@@ -238,7 +266,7 @@ fn generate_projections(
vis: &Visibility,
ident: &Ident,
generics: &Generics,
- fields: &[(bool, &Field)],
+ fields: &[FieldInfo<'_>],
) -> TokenStream {
let (impl_generics, ty_generics, _) = generics.split_for_impl();
let mut generics_with_pin_lt = generics.clone();
@@ -247,32 +275,23 @@ fn generate_projections(
let projection = format_ident!("{ident}Projection");
let this = format_ident!("this");
- let (fields_decl, fields_proj) = collect_tuple(fields.iter().map(
- |(
- pinned,
- Field {
- vis,
- ident,
- ty,
- attrs,
- ..
- },
- )| {
- let mut attrs = attrs.clone();
- attrs.retain(|a| !a.path().is_ident("pin"));
- let mut no_doc_attrs = attrs.clone();
- no_doc_attrs.retain(|a| !a.path().is_ident("doc"));
+ let (fields_decl, fields_proj): (Vec<_>, Vec<_>) = fields
+ .iter()
+ .map(|field| {
+ let Field { vis, ident, ty, .. } = &field.field;
+ let cfg_attrs = &field.cfg_attrs;
+
let ident = ident
.as_ref()
.expect("only structs with named fields are supported");
- if *pinned {
+ if field.pinned {
(
quote!(
- #(#attrs)*
+ #(#cfg_attrs)*
#vis #ident: ::core::pin::Pin<&'__pin mut #ty>,
),
quote!(
- #(#no_doc_attrs)*
+ #(#cfg_attrs)*
// SAFETY: this field is structurally pinned.
#ident: unsafe { ::core::pin::Pin::new_unchecked(&mut #this.#ident) },
),
@@ -280,31 +299,35 @@ fn generate_projections(
} else {
(
quote!(
- #(#attrs)*
+ #(#cfg_attrs)*
#vis #ident: &'__pin mut #ty,
),
quote!(
- #(#no_doc_attrs)*
+ #(#cfg_attrs)*
#ident: &mut #this.#ident,
),
)
}
- },
- ));
+ })
+ .collect();
let structurally_pinned_fields_docs = fields
.iter()
- .filter_map(|(pinned, field)| pinned.then_some(field))
- .map(|Field { ident, .. }| format!(" - `{}`", ident.as_ref().unwrap()));
+ .filter(|f| f.pinned)
+ .map(|f| format!(" - `{}`", f.field.ident.as_ref().unwrap()));
let not_structurally_pinned_fields_docs = fields
.iter()
- .filter_map(|(pinned, field)| (!pinned).then_some(field))
- .map(|Field { ident, .. }| format!(" - `{}`", ident.as_ref().unwrap()));
+ .filter(|f| !f.pinned)
+ .map(|f| format!(" - `{}`", f.field.ident.as_ref().unwrap()));
let docs = format!(" Pin-projections of [`{ident}`]");
quote! {
#[doc = #docs]
- #[allow(dead_code)]
+ // Allow `non_snake_case` since the same warning will be emitted on
+ // the struct definition.
+ #[allow(dead_code, non_snake_case)]
#[doc(hidden)]
- #vis struct #projection #generics_with_pin_lt {
+ #vis struct #projection #generics_with_pin_lt
+ #whr
+ {
#(#fields_decl)*
___pin_phantom_data: ::core::marker::PhantomData<&'__pin mut ()>,
}
@@ -336,91 +359,54 @@ impl #impl_generics #ident #ty_generics
fn generate_the_pin_data(
vis: &Visibility,
- ident: &Ident,
+ struct_name: &Ident,
generics: &Generics,
- fields: &[(bool, &Field)],
+ fields: &[FieldInfo<'_>],
) -> TokenStream {
let (impl_generics, ty_generics, whr) = generics.split_for_impl();
// For every field, we create an initializing projection function according to its projection
- // type. If a field is structurally pinned, then it must be initialized via `PinInit`, if it is
- // not structurally pinned, then it can be initialized via `Init`.
- //
- // The functions are `unsafe` to prevent accidentally calling them.
- fn handle_field(
- Field {
- vis,
- ident,
- ty,
- attrs,
- ..
- }: &Field,
- struct_ident: &Ident,
- pinned: bool,
- ) -> TokenStream {
- let mut attrs = attrs.clone();
- attrs.retain(|a| !a.path().is_ident("pin"));
- let ident = ident
- .as_ref()
- .expect("only structs with named fields are supported");
- let project_ident = format_ident!("__project_{ident}");
- let (init_ty, init_fn, project_ty, project_body, pin_safety) = if pinned {
- (
- quote!(PinInit),
- quote!(__pinned_init),
- quote!(::core::pin::Pin<&'__slot mut #ty>),
- // SAFETY: this field is structurally pinned.
- quote!(unsafe { ::core::pin::Pin::new_unchecked(slot) }),
- quote!(
- /// - `slot` will not move until it is dropped, i.e. it will be pinned.
- ),
- )
- } else {
- (
- quote!(Init),
- quote!(__init),
- quote!(&'__slot mut #ty),
- quote!(slot),
- quote!(),
- )
- };
- let slot_safety = format!(
- " `slot` points at the field `{ident}` inside of `{struct_ident}`, which is pinned.",
- );
- quote! {
- /// # Safety
- ///
- /// - `slot` is a valid pointer to uninitialized memory.
- /// - the caller does not touch `slot` when `Err` is returned, they are only permitted
- /// to deallocate.
- #pin_safety
- #(#attrs)*
- #vis unsafe fn #ident<E>(
- self,
- slot: *mut #ty,
- init: impl ::pin_init::#init_ty<#ty, E>,
- ) -> ::core::result::Result<(), E> {
- // SAFETY: this function has the same safety requirements as the __init function
- // called below.
- unsafe { ::pin_init::#init_ty::#init_fn(init, slot) }
- }
-
- /// # Safety
- ///
- #[doc = #slot_safety]
- #(#attrs)*
- #vis unsafe fn #project_ident<'__slot>(
- self,
- slot: &'__slot mut #ty,
- ) -> #project_ty {
- #project_body
- }
- }
- }
-
+ // type. If a field is structurally pinned, we create a `Slot` with `Pinned` which must be
+ // initialized via `PinInit`; if it is not structurally pinned, then we create a `Slot` with
+ // `Unpinned` which allows initialization via `Init`.
let field_accessors = fields
.iter()
- .map(|(pinned, field)| handle_field(field, ident, *pinned))
+ .map(|f| {
+ let Field { vis, ident, ty, .. } = f.field;
+ let cfg_attrs = &f.cfg_attrs;
+
+ let field_name = ident
+ .as_ref()
+ .expect("only structs with named fields are supported");
+ let pin_marker = if f.pinned {
+ quote!(Pinned)
+ } else {
+ quote!(Unpinned)
+ };
+ quote! {
+ /// # Safety
+ ///
+ /// - `slot` is valid and properly aligned.
+ /// - `(*slot).#field_name` is properly aligned.
+ /// - `(*slot).#field_name` points to uninitialized and exclusively accessed
+ /// memory.
+ #(#cfg_attrs)*
+ // Allow `non_snake_case` since the same warning will be emitted on
+ // the struct definition.
+ #[allow(non_snake_case)]
+ #[inline(always)]
+ #vis unsafe fn #field_name(
+ self,
+ slot: *mut #struct_name #ty_generics,
+ ) -> ::pin_init::__internal::Slot<::pin_init::__internal::#pin_marker, #ty> {
+ // SAFETY:
+ // - If `#pin_marker` is `Pinned`, the corresponding field is structurally
+ // pinned.
+ // - Other safety requirements follows the safety requirement.
+ unsafe { ::pin_init::__internal::Slot::new(&raw mut (*slot).#field_name) }
+ }
+ }
+ })
.collect::<TokenStream>();
quote! {
// We declare this struct which will host all of the projection function for our type. It
@@ -429,9 +415,7 @@ fn handle_field(
#vis struct __ThePinData #generics
#whr
{
- __phantom: ::core::marker::PhantomData<
- fn(#ident #ty_generics) -> #ident #ty_generics
- >,
+ __phantom: ::pin_init::__internal::PhantomInvariant<#struct_name #ty_generics>,
}
impl #impl_generics ::core::clone::Clone for __ThePinData #ty_generics
@@ -449,27 +433,30 @@ impl #impl_generics ::core::marker::Copy for __ThePinData #ty_generics
impl #impl_generics __ThePinData #ty_generics
#whr
{
+ /// Type inference helper function.
+ #[inline(always)]
+ #vis fn __make_closure<__F, __E>(self, f: __F) -> __F
+ where
+ __F: FnOnce(*mut #struct_name #ty_generics) ->
+ ::core::result::Result<::pin_init::__internal::InitOk, __E>,
+ {
+ f
+ }
+
#field_accessors
}
// SAFETY: We have added the correct projection functions above to `__ThePinData` and
// we also use the least restrictive generics possible.
- unsafe impl #impl_generics ::pin_init::__internal::HasPinData for #ident #ty_generics
+ unsafe impl #impl_generics ::pin_init::__internal::HasPinData for #struct_name #ty_generics
#whr
{
type PinData = __ThePinData #ty_generics;
unsafe fn __pin_data() -> Self::PinData {
- __ThePinData { __phantom: ::core::marker::PhantomData }
+ __ThePinData { __phantom: ::pin_init::__internal::PhantomInvariant::new() }
}
}
-
- // SAFETY: TODO
- unsafe impl #impl_generics ::pin_init::__internal::PinData for __ThePinData #ty_generics
- #whr
- {
- type Datee = #ident #ty_generics;
- }
}
}
@@ -500,14 +487,3 @@ fn visit_item_mut(&mut self, _: &mut Item) {
// Do not descend into items, since items reset/change what `Self` refers to.
}
}
-
-// replace with `.collect()` once MSRV is above 1.79
-fn collect_tuple<A, B>(iter: impl Iterator<Item = (A, B)>) -> (Vec<A>, Vec<B>) {
- let mut res_a = vec![];
- let mut res_b = vec![];
- for (a, b) in iter {
- res_a.push(a);
- res_b.push(b);
- }
- (res_a, res_b)
-}
diff --git a/rust/pin-init/internal/src/zeroable.rs b/rust/pin-init/internal/src/zeroable.rs
index 0568331..b11feae 100644
--- a/rust/pin-init/internal/src/zeroable.rs
+++ b/rust/pin-init/internal/src/zeroable.rs
@@ -1,4 +1,4 @@
-// SPDX-License-Identifier: GPL-2.0
+// SPDX-License-Identifier: Apache-2.0 OR MIT
use proc_macro2::TokenStream;
use quote::quote;
diff --git a/rust/pin-init/src/__internal.rs b/rust/pin-init/src/__internal.rs
index 5720a62..56dc655 100644
--- a/rust/pin-init/src/__internal.rs
+++ b/rust/pin-init/src/__internal.rs
@@ -7,42 +7,54 @@
use super::*;
+/// Zero-sized type used to mark a type as invariant.
+///
+/// This is a polyfill for the [unstable type] in the standard library of the same name.
+///
/// See the [nomicon] for what subtyping is. See also [this table].
///
-/// The reason for not using `PhantomData<*mut T>` is that that type never implements [`Send`] and
-/// [`Sync`]. Hence `fn(*mut T) -> *mut T` is used, as that type always implements them.
-///
+/// [unstable type]: https://doc.rust-lang.org/nightly/std/marker/struct.PhantomInvariant.html
/// [nomicon]: https://doc.rust-lang.org/nomicon/subtyping.html
/// [this table]: https://doc.rust-lang.org/nomicon/phantom-data.html#table-of-phantomdata-patterns
-pub(crate) type Invariant<T> = PhantomData<fn(*mut T) -> *mut T>;
+#[repr(transparent)]
+pub struct PhantomInvariant<T: ?Sized>(PhantomData<fn(T) -> T>);
-/// Module-internal type implementing `PinInit` and `Init`.
-///
-/// It is unsafe to create this type, since the closure needs to fulfill the same safety
-/// requirement as the `__pinned_init`/`__init` functions.
-pub(crate) struct InitClosure<F, T: ?Sized, E>(pub(crate) F, pub(crate) Invariant<(E, T)>);
-
-// SAFETY: While constructing the `InitClosure`, the user promised that it upholds the
-// `__init` invariants.
-unsafe impl<T: ?Sized, F, E> Init<T, E> for InitClosure<F, T, E>
-where
- F: FnOnce(*mut T) -> Result<(), E>,
-{
- #[inline]
- unsafe fn __init(self, slot: *mut T) -> Result<(), E> {
- (self.0)(slot)
+impl<T: ?Sized> Clone for PhantomInvariant<T> {
+ #[inline(always)]
+ fn clone(&self) -> Self {
+ *self
}
}
-// SAFETY: While constructing the `InitClosure`, the user promised that it upholds the
-// `__pinned_init` invariants.
-unsafe impl<T: ?Sized, F, E> PinInit<T, E> for InitClosure<F, T, E>
-where
- F: FnOnce(*mut T) -> Result<(), E>,
-{
- #[inline]
- unsafe fn __pinned_init(self, slot: *mut T) -> Result<(), E> {
- (self.0)(slot)
+impl<T: ?Sized> Copy for PhantomInvariant<T> {}
+
+impl<T: ?Sized> Default for PhantomInvariant<T> {
+ #[inline(always)]
+ fn default() -> Self {
+ Self::new()
+ }
+}
+
+impl<T: ?Sized> PhantomInvariant<T> {
+ #[inline(always)]
+ pub const fn new() -> Self {
+ Self(PhantomData)
+ }
+}
+
+/// Zero-sized type used to mark a lifetime as invariant.
+///
+/// This is a polyfill for the [unstable type] in the standard library of the same name.
+///
+/// [unstable type]: https://doc.rust-lang.org/nightly/std/marker/struct.PhantomInvariantLifetime.html
+#[repr(transparent)]
+#[derive(Clone, Copy, Default)]
+pub struct PhantomInvariantLifetime<'a>(PhantomInvariant<&'a ()>);
+
+impl PhantomInvariantLifetime<'_> {
+ #[inline(always)]
+ pub const fn new() -> Self {
+ Self(PhantomInvariant::new())
}
}
@@ -71,30 +83,12 @@ pub unsafe fn new() -> Self {
///
/// Only the `init` module is allowed to use this trait.
pub unsafe trait HasPinData {
- type PinData: PinData;
+ type PinData;
#[expect(clippy::missing_safety_doc)]
unsafe fn __pin_data() -> Self::PinData;
}
-/// Marker trait for pinning data of structs.
-///
-/// # Safety
-///
-/// Only the `init` module is allowed to use this trait.
-pub unsafe trait PinData: Copy {
- type Datee: ?Sized + HasPinData;
-
- /// Type inference helper function.
- #[inline(always)]
- fn make_closure<F, E>(self, f: F) -> F
- where
- F: FnOnce(*mut Self::Datee) -> Result<InitOk, E>,
- {
- f
- }
-}
-
/// This trait is automatically implemented for every type. It aims to provide the same type
/// inference help as `HasPinData`.
///
@@ -102,31 +96,13 @@ fn make_closure<F, E>(self, f: F) -> F
///
/// Only the `init` module is allowed to use this trait.
pub unsafe trait HasInitData {
- type InitData: InitData;
+ type InitData;
#[expect(clippy::missing_safety_doc)]
unsafe fn __init_data() -> Self::InitData;
}
-/// Same function as `PinData`, but for arbitrary data.
-///
-/// # Safety
-///
-/// Only the `init` module is allowed to use this trait.
-pub unsafe trait InitData: Copy {
- type Datee: ?Sized + HasInitData;
-
- /// Type inference helper function.
- #[inline(always)]
- fn make_closure<F, E>(self, f: F) -> F
- where
- F: FnOnce(*mut Self::Datee) -> Result<InitOk, E>,
- {
- f
- }
-}
-
-pub struct AllData<T: ?Sized>(Invariant<T>);
+pub struct AllData<T: ?Sized>(PhantomInvariant<T>);
impl<T: ?Sized> Clone for AllData<T> {
fn clone(&self) -> Self {
@@ -136,9 +112,15 @@ fn clone(&self) -> Self {
impl<T: ?Sized> Copy for AllData<T> {}
-// SAFETY: TODO.
-unsafe impl<T: ?Sized> InitData for AllData<T> {
- type Datee = T;
+impl<T: ?Sized> AllData<T> {
+ /// Type inference helper function.
+ #[inline(always)]
+ pub fn __make_closure<F, E>(self, f: F) -> F
+ where
+ F: FnOnce(*mut T) -> Result<InitOk, E>,
+ {
+ f
+ }
}
// SAFETY: TODO.
@@ -146,7 +128,7 @@ unsafe impl<T: ?Sized> HasInitData for T {
type InitData = AllData<T>;
unsafe fn __init_data() -> Self::InitData {
- AllData(PhantomData)
+ AllData(PhantomInvariant::new())
}
}
@@ -235,6 +217,87 @@ struct Foo {
println!("{value:?}");
}
+// Marker types that determines type of `DropGuard`'s let bindings.
+pub struct Pinned;
+pub struct Unpinned;
+
+/// Represent an uninitialized field.
+///
+/// # Invariants
+///
+/// - `ptr` is valid, properly aligned and points to uninitialized and exclusively accessed memory.
+/// - If `P` is `Pinned`, then `ptr` is structurally pinned.
+pub struct Slot<P, T: ?Sized> {
+ ptr: *mut T,
+ _phantom: PhantomData<P>,
+}
+
+impl<P, T: ?Sized> Slot<P, T> {
+ /// # Safety
+ ///
+ /// - `ptr` is valid, properly aligned and points to uninitialized and exclusively accessed
+ /// memory.
+ /// - If `P` is `Pinned`, then `ptr` is structurally pinned.
+ #[inline(always)]
+ pub unsafe fn new(ptr: *mut T) -> Self {
+ // INVARIANT: Per safety requirement.
+ Self {
+ ptr,
+ _phantom: PhantomData,
+ }
+ }
+
+ /// Initialize the field by value.
+ #[inline(always)]
+ pub fn write(self, value: T) -> DropGuard<P, T>
+ where
+ T: Sized,
+ {
+ // SAFETY: `self.ptr` is a valid and aligned pointer for write.
+ unsafe { self.ptr.write(value) }
+ // SAFETY:
+ // - `self.ptr` is valid and properly aligned per type invariant.
+ // - `*self.ptr` is initialized above and the ownership is transferred to the guard.
+ // - If `P` is `Pinned`, `self.ptr` is pinned.
+ unsafe { DropGuard::new(self.ptr) }
+ }
+}
+
+impl<T: ?Sized> Slot<Unpinned, T> {
+ /// Initialize the field.
+ #[inline(always)]
+ pub fn init<E>(self, init: impl Init<T, E>) -> Result<DropGuard<Unpinned, T>, E> {
+ // SAFETY:
+ // - `self.ptr` is valid and properly aligned.
+ // - when `Err` is returned, we also propagate the error without touching `slot`;
+ // also `self` is consumed so it cannot be touched further.
+ unsafe { init.__init(self.ptr)? };
+
+ // SAFETY:
+ // - `self.ptr` is valid and properly aligned per type invariant.
+ // - `*self.ptr` is initialized above and the ownership is transferred to the guard.
+ Ok(unsafe { DropGuard::new(self.ptr) })
+ }
+}
+
+impl<T: ?Sized> Slot<Pinned, T> {
+ /// Initialize the field.
+ #[inline(always)]
+ pub fn init<E>(self, init: impl PinInit<T, E>) -> Result<DropGuard<Pinned, T>, E> {
+ // SAFETY:
+ // - `self.ptr` is valid and properly aligned.
+ // - when `Err` is returned, we also propagate the error without touching `ptr`;
+ // also `self` is consumed so it cannot be touched further.
+ // - the drop guard will not hand out `&mut` (only `Pin<&mut T>`).
+ unsafe { init.__pinned_init(self.ptr)? };
+
+ // SAFETY:
+ // - `self.ptr` is valid, properly aligned and pinned per type invariant.
+ // - `*self.ptr` is initialized above and the ownership is transferred to the guard.
+ Ok(unsafe { DropGuard::new(self.ptr) })
+ }
+}
+
/// When a value of this type is dropped, it drops a `T`.
///
/// Can be forgotten to prevent the drop.
@@ -243,11 +306,13 @@ struct Foo {
///
/// - `ptr` is valid and properly aligned.
/// - `*ptr` is initialized and owned by this guard.
-pub struct DropGuard<T: ?Sized> {
+/// - if `P` is `Pinned`, `ptr` is pinned.
+pub struct DropGuard<P, T: ?Sized> {
ptr: *mut T,
+ phantom: PhantomData<P>,
}
-impl<T: ?Sized> DropGuard<T> {
+impl<P, T: ?Sized> DropGuard<P, T> {
/// Creates a drop guard and transfer the ownership of the pointer content.
///
/// The ownership is only relinguished if the guard is forgotten via [`core::mem::forget`].
@@ -256,12 +321,18 @@ impl<T: ?Sized> DropGuard<T> {
///
/// - `ptr` is valid and properly aligned.
/// - `*ptr` is initialized, and the ownership is transferred to this guard.
+ /// - if `P` is `Pinned`, `ptr` is pinned.
#[inline]
pub unsafe fn new(ptr: *mut T) -> Self {
// INVARIANT: By safety requirement.
- Self { ptr }
+ Self {
+ ptr,
+ phantom: PhantomData,
+ }
}
+}
+impl<T: ?Sized> DropGuard<Unpinned, T> {
/// Create a let binding for accessor use.
#[inline]
pub fn let_binding(&mut self) -> &mut T {
@@ -270,7 +341,17 @@ pub fn let_binding(&mut self) -> &mut T {
}
}
-impl<T: ?Sized> Drop for DropGuard<T> {
+impl<T: ?Sized> DropGuard<Pinned, T> {
+ /// Create a let binding for accessor use.
+ #[inline]
+ pub fn let_binding(&mut self) -> Pin<&mut T> {
+ // SAFETY: `self.ptr` is valid, properly aligned, initialized, exclusively accessible and
+ // pinned per type invariant.
+ unsafe { Pin::new_unchecked(&mut *self.ptr) }
+ }
+}
+
+impl<P, T: ?Sized> Drop for DropGuard<P, T> {
#[inline]
fn drop(&mut self) {
// SAFETY: `self.ptr` is valid, properly aligned and `*self.ptr` is owned by this guard.
diff --git a/rust/pin-init/src/lib.rs b/rust/pin-init/src/lib.rs
index 64eec09..fd40c8f 100644
--- a/rust/pin-init/src/lib.rs
+++ b/rust/pin-init/src/lib.rs
@@ -263,12 +263,6 @@
//! [`impl Init<T, E>`]: crate::Init
//! [Rust-for-Linux]: https://rust-for-linux.com/
-#![cfg_attr(USE_RUSTC_FEATURES, feature(lint_reasons))]
-#![cfg_attr(USE_RUSTC_FEATURES, feature(raw_ref_op))]
-#![cfg_attr(
- all(any(feature = "alloc", feature = "std"), USE_RUSTC_FEATURES),
- feature(new_uninit)
-)]
#![forbid(missing_docs, unsafe_op_in_unsafe_fn)]
#![cfg_attr(not(feature = "std"), no_std)]
#![cfg_attr(feature = "alloc", feature(allocator_api))]
@@ -437,7 +431,7 @@
/// ```
/// use pin_init::MaybeZeroable;
///
-/// // implmements `Zeroable`
+/// // implements `Zeroable`
/// #[derive(MaybeZeroable)]
/// pub struct DriverData {
/// pub(crate) id: i64,
@@ -445,7 +439,7 @@
/// len: usize,
/// }
///
-/// // does not implmement `Zeroable`
+/// // does not implement `Zeroable`
/// #[derive(MaybeZeroable)]
/// pub struct DriverData2 {
/// pub(crate) id: i64,
@@ -873,12 +867,12 @@ macro_rules! stack_try_pin_init {
#[macro_export]
macro_rules! assert_pinned {
($ty:ty, $field:ident, $field_ty:ty, inline) => {
- let _ = move |ptr: *mut $field_ty| {
- // SAFETY: This code is unreachable.
- let data = unsafe { <$ty as $crate::__internal::HasPinData>::__pin_data() };
- let init = $crate::__internal::AlwaysFail::<$field_ty>::new();
- // SAFETY: This code is unreachable.
- unsafe { data.$field(ptr, init) }.ok();
+ // SAFETY: This code is unreachable.
+ let _ = move |ptr: *mut $ty| unsafe {
+ let data = <$ty as $crate::__internal::HasPinData>::__pin_data();
+ _ = data
+ .$field(ptr)
+ .init($crate::__internal::AlwaysFail::<$field_ty>::new());
};
};
@@ -953,12 +947,12 @@ fn pin_chain<F>(self, f: F) -> ChainPinInit<Self, F, T, E>
where
F: FnOnce(Pin<&mut T>) -> Result<(), E>,
{
- ChainPinInit(self, f, PhantomData)
+ ChainPinInit(self, f, __internal::PhantomInvariant::new())
}
}
/// An initializer returned by [`PinInit::pin_chain`].
-pub struct ChainPinInit<I, F, T: ?Sized, E>(I, F, __internal::Invariant<(E, T)>);
+pub struct ChainPinInit<I, F, T: ?Sized, E>(I, F, __internal::PhantomInvariant<(E, T)>);
// SAFETY: The `__pinned_init` function is implemented such that it
// - returns `Ok(())` on successful initialization,
@@ -1061,12 +1055,12 @@ fn chain<F>(self, f: F) -> ChainInit<Self, F, T, E>
where
F: FnOnce(&mut T) -> Result<(), E>,
{
- ChainInit(self, f, PhantomData)
+ ChainInit(self, f, __internal::PhantomInvariant::new())
}
}
/// An initializer returned by [`Init::chain`].
-pub struct ChainInit<I, F, T: ?Sized, E>(I, F, __internal::Invariant<(E, T)>);
+pub struct ChainInit<I, F, T: ?Sized, E>(I, F, __internal::PhantomInvariant<(E, T)>);
// SAFETY: The `__init` function is implemented such that it
// - returns `Ok(())` on successful initialization,
@@ -1098,6 +1092,36 @@ unsafe fn __pinned_init(self, slot: *mut T) -> Result<(), E> {
}
}
+/// Implement `PinInit` and `Init` for closures.
+///
+/// It is unsafe to create this type, since the closure needs to fulfill the same safety
+/// requirement as the `__pinned_init`/`__init` functions.
+struct InitClosure<F, T: ?Sized>(F, __internal::PhantomInvariant<T>);
+
+// SAFETY: While constructing the `InitClosure`, the user promised that it upholds the
+// `__init` invariants.
+unsafe impl<T: ?Sized, F, E> Init<T, E> for InitClosure<F, T>
+where
+ F: FnOnce(*mut T) -> Result<(), E>,
+{
+ #[inline]
+ unsafe fn __init(self, slot: *mut T) -> Result<(), E> {
+ (self.0)(slot)
+ }
+}
+
+// SAFETY: While constructing the `InitClosure`, the user promised that it upholds the
+// `__pinned_init` invariants.
+unsafe impl<T: ?Sized, F, E> PinInit<T, E> for InitClosure<F, T>
+where
+ F: FnOnce(*mut T) -> Result<(), E>,
+{
+ #[inline]
+ unsafe fn __pinned_init(self, slot: *mut T) -> Result<(), E> {
+ (self.0)(slot)
+ }
+}
+
/// Creates a new [`PinInit<T, E>`] from the given closure.
///
/// # Safety
@@ -1114,7 +1138,7 @@ unsafe fn __pinned_init(self, slot: *mut T) -> Result<(), E> {
pub const unsafe fn pin_init_from_closure<T: ?Sized, E>(
f: impl FnOnce(*mut T) -> Result<(), E>,
) -> impl PinInit<T, E> {
- __internal::InitClosure(f, PhantomData)
+ InitClosure(f, __internal::PhantomInvariant::new())
}
/// Creates a new [`Init<T, E>`] from the given closure.
@@ -1133,7 +1157,7 @@ unsafe fn __pinned_init(self, slot: *mut T) -> Result<(), E> {
pub const unsafe fn init_from_closure<T: ?Sized, E>(
f: impl FnOnce(*mut T) -> Result<(), E>,
) -> impl Init<T, E> {
- __internal::InitClosure(f, PhantomData)
+ InitClosure(f, __internal::PhantomInvariant::new())
}
/// Changes the to be initialized type.
@@ -1145,14 +1169,7 @@ unsafe fn __pinned_init(self, slot: *mut T) -> Result<(), E> {
pub const unsafe fn cast_pin_init<T, U, E>(init: impl PinInit<T, E>) -> impl PinInit<U, E> {
// SAFETY: initialization delegated to a valid initializer. Cast is valid by function safety
// requirements.
- let res = unsafe { pin_init_from_closure(|ptr: *mut U| init.__pinned_init(ptr.cast::<T>())) };
- // FIXME: this let binding is required to avoid a compiler error (cycle when computing the opaque
- // type returned by this function) before Rust 1.81. Remove after MSRV bump.
- #[allow(
- clippy::let_and_return,
- reason = "some clippy versions warn about the let binding"
- )]
- res
+ unsafe { pin_init_from_closure(|ptr: *mut U| init.__pinned_init(ptr.cast::<T>())) }
}
/// Changes the to be initialized type.
@@ -1164,14 +1181,7 @@ unsafe fn __pinned_init(self, slot: *mut T) -> Result<(), E> {
pub const unsafe fn cast_init<T, U, E>(init: impl Init<T, E>) -> impl Init<U, E> {
// SAFETY: initialization delegated to a valid initializer. Cast is valid by function safety
// requirements.
- let res = unsafe { init_from_closure(|ptr: *mut U| init.__init(ptr.cast::<T>())) };
- // FIXME: this let binding is required to avoid a compiler error (cycle when computing the opaque
- // type returned by this function) before Rust 1.81. Remove after MSRV bump.
- #[allow(
- clippy::let_and_return,
- reason = "some clippy versions warn about the let binding"
- )]
- res
+ unsafe { init_from_closure(|ptr: *mut U| init.__init(ptr.cast::<T>())) }
}
/// An initializer that leaves the memory uninitialized.
@@ -1523,27 +1533,6 @@ fn zeroed() -> Self
}
}
-/// Marker trait for types that allow `Option<Self>` to be set to all zeroes in order to write
-/// `None` to that location.
-///
-/// # Safety
-///
-/// The implementer needs to ensure that `unsafe impl Zeroable for Option<Self> {}` is sound.
-pub unsafe trait ZeroableOption {}
-
-// SAFETY: by the safety requirement of `ZeroableOption`, this is valid.
-unsafe impl<T: ZeroableOption> Zeroable for Option<T> {}
-
-// SAFETY: `Option<&T>` is part of the option layout optimization guarantee:
-// <https://doc.rust-lang.org/stable/std/option/index.html#representation>.
-unsafe impl<T> ZeroableOption for &T {}
-// SAFETY: `Option<&mut T>` is part of the option layout optimization guarantee:
-// <https://doc.rust-lang.org/stable/std/option/index.html#representation>.
-unsafe impl<T> ZeroableOption for &mut T {}
-// SAFETY: `Option<NonNull<T>>` is part of the option layout optimization guarantee:
-// <https://doc.rust-lang.org/stable/std/option/index.html#representation>.
-unsafe impl<T> ZeroableOption for NonNull<T> {}
-
/// Create an initializer for a zeroed `T`.
///
/// The returned initializer will write `0x00` to every byte of the given `slot`.
@@ -1649,6 +1638,17 @@ unsafe impl<$first: Zeroable, $($t: Zeroable),*> Zeroable for ($first, $($t),*)
impl_tuple_zeroable!(A, B, C, D, E, F, G, H, I, J);
+/// Marker trait for types that allow `Option<Self>` to be set to all zeroes in order to write
+/// `None` to that location.
+///
+/// # Safety
+///
+/// The implementer needs to ensure that `unsafe impl Zeroable for Option<Self> {}` is sound.
+pub unsafe trait ZeroableOption {}
+
+// SAFETY: by the safety requirement of `ZeroableOption`, this is valid.
+unsafe impl<T: ZeroableOption> Zeroable for Option<T> {}
+
macro_rules! impl_fn_zeroable_option {
([$($abi:literal),* $(,)?] $args:tt) => {
$(impl_fn_zeroable_option!({extern $abi} $args);)*
@@ -1674,18 +1674,27 @@ unsafe impl<$ret, $($rest),*> ZeroableOption for $($prefix)* fn($($rest),*) -> $
impl_fn_zeroable_option!(["Rust", "C"] { A, B, C, D, E, F, G, H, I, J, K, L, M, N, O, P, Q, R, S, T, U });
-macro_rules! impl_non_zero_int_zeroable_option {
- ($($int:ty),* $(,)?) => {
- // SAFETY: Safety comment written in the macro invocation.
- $(unsafe impl ZeroableOption for $int {})*
+macro_rules! impl_zeroable_option {
+ ($($({$($generics:tt)*})? $t:ty, )*) => {
+ // SAFETY: Safety comments written in the macro invocation.
+ $(unsafe impl$($($generics)*)? ZeroableOption for $t {})*
};
}
-impl_non_zero_int_zeroable_option! {
+impl_zeroable_option! {
+ // SAFETY: `Option<&T>` is part of the option layout optimization guarantee:
+ // <https://doc.rust-lang.org/stable/std/option/index.html#representation>.
+ {<T: ?Sized>} &T,
+ // SAFETY: `Option<&mut T>` is part of the option layout optimization guarantee:
+ // <https://doc.rust-lang.org/stable/std/option/index.html#representation>.
+ {<T: ?Sized>} &mut T,
+ // SAFETY: `Option<NonNull<T>>` is part of the option layout optimization guarantee:
+ // <https://doc.rust-lang.org/stable/std/option/index.html#representation>.
+ {<T: ?Sized>} NonNull<T>,
// SAFETY: All zeros is equivalent to `None` (option layout optimization guarantee:
// <https://doc.rust-lang.org/stable/std/option/index.html#representation>).
- NonZeroU8, NonZeroU16, NonZeroU32, NonZeroU64, NonZeroU128, NonZeroUsize,
- NonZeroI8, NonZeroI16, NonZeroI32, NonZeroI64, NonZeroI128, NonZeroIsize,
+ NonZero<u8>, NonZero<u16>, NonZero<u32>, NonZero<u64>, NonZero<u128>, NonZero<usize>,
+ NonZero<i8>, NonZero<i16>, NonZero<i32>, NonZero<i64>, NonZero<i128>, NonZero<isize>,
}
/// This trait allows creating an instance of `Self` which contains exactly one
diff --git a/rust/zerocopy-derive/README.md b/rust/zerocopy-derive/README.md
new file mode 100644
index 0000000..110f4a401
--- /dev/null
+++ b/rust/zerocopy-derive/README.md
@@ -0,0 +1,14 @@
+# `zerocopy-derive`
+
+These source files come from the Rust `zerocopy-derive` crate, version v0.8.50
+(released 2026-05-31), hosted in the <https://github.com/google/zerocopy>
+repository, licensed under "BSD-2-Clause OR Apache-2.0 OR MIT" and only
+modified to add the SPDX license identifiers and to remove the generation of
+non-ASCII identifiers.
+
+For copyright details, please see:
+
+ https://github.com/google/zerocopy/blob/v0.8.50/README.md?plain=1
+ https://github.com/google/zerocopy/blob/v0.8.50/LICENSE-BSD
+ https://github.com/google/zerocopy/blob/v0.8.50/LICENSE-APACHE
+ https://github.com/google/zerocopy/blob/v0.8.50/LICENSE-MIT
diff --git a/rust/zerocopy-derive/derive/from_bytes.rs b/rust/zerocopy-derive/derive/from_bytes.rs
new file mode 100644
index 0000000..d693a63
--- /dev/null
+++ b/rust/zerocopy-derive/derive/from_bytes.rs
@@ -0,0 +1,192 @@
+// SPDX-License-Identifier: (BSD-2-Clause OR Apache-2.0) OR MIT
+
+use proc_macro2::{Span, TokenStream};
+use syn::{
+ parse_quote, Data, DataEnum, DataStruct, DataUnion, Error, Expr, ExprLit, ExprUnary, Lit, UnOp,
+ WherePredicate,
+};
+
+use crate::{
+ derive::try_from_bytes::derive_try_from_bytes,
+ repr::{CompoundRepr, EnumRepr, Repr, Spanned},
+ util::{enum_size_from_repr, Ctx, FieldBounds, ImplBlockBuilder, Trait, TraitBound},
+};
+/// Returns `Ok(index)` if variant `index` of the enum has a discriminant of
+/// zero. If `Err(bool)` is returned, the boolean is true if the enum has
+/// unknown discriminants (e.g. discriminants set to const expressions which we
+/// can't evaluate in a proc macro). If the enum has unknown discriminants, then
+/// it might have a zero variant that we just can't detect.
+pub(crate) fn find_zero_variant(enm: &DataEnum) -> Result<usize, bool> {
+ // Discriminants can be anywhere in the range [i128::MIN, u128::MAX] because
+ // the discriminant type may be signed or unsigned. Since we only care about
+ // tracking the discriminant when it's less than or equal to zero, we can
+ // avoid u128 -> i128 conversions and bounds checking by making the "next
+ // discriminant" value implicitly negative.
+ // Technically 64 bits is enough, but 128 is better for future compatibility
+ // with https://github.com/rust-lang/rust/issues/56071
+ let mut next_negative_discriminant = Some(0);
+
+ // Sometimes we encounter explicit discriminants that we can't know the
+ // value of (e.g. a constant expression that requires evaluation). These
+ // could evaluate to zero or a negative number, but we can't assume that
+ // they do (no false positives allowed!). So we treat them like strictly-
+ // positive values that can't result in any zero variants, and track whether
+ // we've encountered any unknown discriminants.
+ let mut has_unknown_discriminants = false;
+
+ for (i, v) in enm.variants.iter().enumerate() {
+ match v.discriminant.as_ref() {
+ // Implicit discriminant
+ None => {
+ match next_negative_discriminant.as_mut() {
+ Some(0) => return Ok(i),
+ // n is nonzero so subtraction is always safe
+ Some(n) => *n -= 1,
+ None => (),
+ }
+ }
+ // Explicit positive discriminant
+ Some((_, Expr::Lit(ExprLit { lit: Lit::Int(int), .. }))) => {
+ match int.base10_parse::<u128>().ok() {
+ Some(0) => return Ok(i),
+ Some(_) => next_negative_discriminant = None,
+ None => {
+ // Numbers should never fail to parse, but just in case:
+ has_unknown_discriminants = true;
+ next_negative_discriminant = None;
+ }
+ }
+ }
+ // Explicit negative discriminant
+ Some((_, Expr::Unary(ExprUnary { op: UnOp::Neg(_), expr, .. }))) => match &**expr {
+ Expr::Lit(ExprLit { lit: Lit::Int(int), .. }) => {
+ match int.base10_parse::<u128>().ok() {
+ Some(0) => return Ok(i),
+ // x is nonzero so subtraction is always safe
+ Some(x) => next_negative_discriminant = Some(x - 1),
+ None => {
+ // Numbers should never fail to parse, but just in
+ // case:
+ has_unknown_discriminants = true;
+ next_negative_discriminant = None;
+ }
+ }
+ }
+ // Unknown negative discriminant (e.g. const repr)
+ _ => {
+ has_unknown_discriminants = true;
+ next_negative_discriminant = None;
+ }
+ },
+ // Unknown discriminant (e.g. const expr)
+ _ => {
+ has_unknown_discriminants = true;
+ next_negative_discriminant = None;
+ }
+ }
+ }
+
+ Err(has_unknown_discriminants)
+}
+pub(crate) fn derive_from_zeros(ctx: &Ctx, top_level: Trait) -> Result<TokenStream, Error> {
+ let try_from_bytes = derive_try_from_bytes(ctx, top_level)?;
+ let from_zeros = match &ctx.ast.data {
+ Data::Struct(strct) => derive_from_zeros_struct(ctx, strct),
+ Data::Enum(enm) => derive_from_zeros_enum(ctx, enm)?,
+ Data::Union(unn) => derive_from_zeros_union(ctx, unn),
+ };
+ Ok(IntoIterator::into_iter([try_from_bytes, from_zeros]).collect())
+}
+pub(crate) fn derive_from_bytes(ctx: &Ctx, top_level: Trait) -> Result<TokenStream, Error> {
+ let from_zeros = derive_from_zeros(ctx, top_level)?;
+ let from_bytes = match &ctx.ast.data {
+ Data::Struct(strct) => derive_from_bytes_struct(ctx, strct),
+ Data::Enum(enm) => derive_from_bytes_enum(ctx, enm)?,
+ Data::Union(unn) => derive_from_bytes_union(ctx, unn),
+ };
+
+ Ok(IntoIterator::into_iter([from_zeros, from_bytes]).collect())
+}
+fn derive_from_zeros_struct(ctx: &Ctx, strct: &DataStruct) -> TokenStream {
+ ImplBlockBuilder::new(ctx, strct, Trait::FromZeros, FieldBounds::ALL_SELF).build()
+}
+fn derive_from_zeros_enum(ctx: &Ctx, enm: &DataEnum) -> Result<TokenStream, Error> {
+ let repr = EnumRepr::from_attrs(&ctx.ast.attrs)?;
+
+ // We don't actually care what the repr is; we just care that it's one of
+ // the allowed ones.
+ match repr {
+ Repr::Compound(Spanned { t: CompoundRepr::C | CompoundRepr::Primitive(_), span: _ }, _) => {
+ }
+ Repr::Transparent(_) | Repr::Compound(Spanned { t: CompoundRepr::Rust, span: _ }, _) => {
+ return ctx.error_or_skip(
+ Error::new(
+ Span::call_site(),
+ "must have #[repr(C)] or #[repr(Int)] attribute in order to guarantee this type's memory layout",
+ ),
+ );
+ }
+ }
+
+ let zero_variant = match find_zero_variant(enm) {
+ Ok(index) => enm.variants.iter().nth(index).unwrap(),
+ // Has unknown variants
+ Err(true) => {
+ return ctx.error_or_skip(Error::new_spanned(
+ &ctx.ast,
+ "FromZeros only supported on enums with a variant that has a discriminant of `0`\n\
+ help: This enum has discriminants which are not literal integers. One of those may \
+ define or imply which variant has a discriminant of zero. Use a literal integer to \
+ define or imply the variant with a discriminant of zero.",
+ ));
+ }
+ // Does not have unknown variants
+ Err(false) => {
+ return ctx.error_or_skip(Error::new_spanned(
+ &ctx.ast,
+ "FromZeros only supported on enums with a variant that has a discriminant of `0`",
+ ));
+ }
+ };
+
+ let zerocopy_crate = &ctx.zerocopy_crate;
+ let explicit_bounds = zero_variant
+ .fields
+ .iter()
+ .map(|field| {
+ let ty = &field.ty;
+ parse_quote! { #ty: #zerocopy_crate::FromZeros }
+ })
+ .collect::<Vec<WherePredicate>>();
+
+ Ok(ImplBlockBuilder::new(ctx, enm, Trait::FromZeros, FieldBounds::Explicit(explicit_bounds))
+ .build())
+}
+fn derive_from_zeros_union(ctx: &Ctx, unn: &DataUnion) -> TokenStream {
+ let field_type_trait_bounds = FieldBounds::All(&[TraitBound::Slf]);
+ ImplBlockBuilder::new(ctx, unn, Trait::FromZeros, field_type_trait_bounds).build()
+}
+fn derive_from_bytes_struct(ctx: &Ctx, strct: &DataStruct) -> TokenStream {
+ ImplBlockBuilder::new(ctx, strct, Trait::FromBytes, FieldBounds::ALL_SELF).build()
+}
+fn derive_from_bytes_enum(ctx: &Ctx, enm: &DataEnum) -> Result<TokenStream, Error> {
+ let repr = EnumRepr::from_attrs(&ctx.ast.attrs)?;
+
+ let variants_required = 1usize << enum_size_from_repr(&repr)?;
+ if enm.variants.len() != variants_required {
+ return ctx.error_or_skip(Error::new_spanned(
+ &ctx.ast,
+ format!(
+ "FromBytes only supported on {} enum with {} variants",
+ repr.repr_type_name(),
+ variants_required
+ ),
+ ));
+ }
+
+ Ok(ImplBlockBuilder::new(ctx, enm, Trait::FromBytes, FieldBounds::ALL_SELF).build())
+}
+fn derive_from_bytes_union(ctx: &Ctx, unn: &DataUnion) -> TokenStream {
+ let field_type_trait_bounds = FieldBounds::All(&[TraitBound::Slf]);
+ ImplBlockBuilder::new(ctx, unn, Trait::FromBytes, field_type_trait_bounds).build()
+}
diff --git a/rust/zerocopy-derive/derive/into_bytes.rs b/rust/zerocopy-derive/derive/into_bytes.rs
new file mode 100644
index 0000000..ad52a6b
--- /dev/null
+++ b/rust/zerocopy-derive/derive/into_bytes.rs
@@ -0,0 +1,164 @@
+// SPDX-License-Identifier: (BSD-2-Clause OR Apache-2.0) OR MIT
+
+use proc_macro2::{Span, TokenStream};
+use quote::quote;
+use syn::{Data, DataEnum, DataStruct, DataUnion, Error, Type};
+
+use crate::{
+ repr::{EnumRepr, StructUnionRepr},
+ util::{
+ generate_tag_enum, Ctx, DataExt, FieldBounds, ImplBlockBuilder, PaddingCheck, Trait,
+ TraitBound,
+ },
+};
+pub(crate) fn derive_into_bytes(ctx: &Ctx, _top_level: Trait) -> Result<TokenStream, Error> {
+ match &ctx.ast.data {
+ Data::Struct(strct) => derive_into_bytes_struct(ctx, strct),
+ Data::Enum(enm) => derive_into_bytes_enum(ctx, enm),
+ Data::Union(unn) => derive_into_bytes_union(ctx, unn),
+ }
+}
+fn derive_into_bytes_struct(ctx: &Ctx, strct: &DataStruct) -> Result<TokenStream, Error> {
+ let repr = StructUnionRepr::from_attrs(&ctx.ast.attrs)?;
+
+ let is_transparent = repr.is_transparent();
+ let is_c = repr.is_c();
+ let is_packed_1 = repr.is_packed_1();
+ let num_fields = strct.fields().len();
+
+ let (padding_check, require_unaligned_fields) = if is_transparent || is_packed_1 {
+ // No padding check needed.
+ // - repr(transparent): The layout and ABI of the whole struct is the
+ // same as its only non-ZST field (meaning there's no padding outside
+ // of that field) and we require that field to be `IntoBytes` (meaning
+ // there's no padding in that field).
+ // - repr(packed): Any inter-field padding bytes are removed, meaning
+ // that any padding bytes would need to come from the fields, all of
+ // which we require to be `IntoBytes` (meaning they don't have any
+ // padding). Note that this holds regardless of other `repr`
+ // attributes, including `repr(Rust)`. [1]
+ //
+ // [1] Per https://doc.rust-lang.org/1.81.0/reference/type-layout.html#the-alignment-modifiers:
+ //
+ // An important consequence of these rules is that a type with
+ // `#[repr(packed(1))]`` (or `#[repr(packed)]``) will have no
+ // inter-field padding.
+ (None, false)
+ } else if is_c && !repr.is_align_gt_1() && num_fields <= 1 {
+ // No padding check needed. A repr(C) struct with zero or one field has
+ // no padding unless #[repr(align)] explicitly adds padding, which we
+ // check for in this branch's condition.
+ (None, false)
+ } else if ctx.ast.generics.params.is_empty() {
+ // Is the last field a syntactic slice, i.e., `[SomeType]`.
+ let is_syntactic_dst =
+ strct.fields().last().map(|(_, _, ty)| matches!(ty, Type::Slice(_))).unwrap_or(false);
+ // Since there are no generics, we can emit a padding check. All reprs
+ // guarantee that fields won't overlap [1], so the padding check is
+ // sound. This is more permissive than the next case, which requires
+ // that all field types implement `Unaligned`.
+ //
+ // [1] Per https://doc.rust-lang.org/1.81.0/reference/type-layout.html#the-rust-representation:
+ //
+ // The only data layout guarantees made by [`repr(Rust)`] are those
+ // required for soundness. They are:
+ // ...
+ // 2. The fields do not overlap.
+ // ...
+ if is_c && is_syntactic_dst {
+ (Some(PaddingCheck::ReprCStruct), false)
+ } else {
+ (Some(PaddingCheck::Struct), false)
+ }
+ } else if is_c && !repr.is_align_gt_1() {
+ // We can't use a padding check since there are generic type arguments.
+ // Instead, we require all field types to implement `Unaligned`. This
+ // ensures that the `repr(C)` layout algorithm will not insert any
+ // padding unless #[repr(align)] explicitly adds padding, which we check
+ // for in this branch's condition.
+ //
+ // FIXME(#10): Support type parameters for non-transparent, non-packed
+ // structs without requiring `Unaligned`.
+ (None, true)
+ } else {
+ return ctx.error_or_skip(Error::new(
+ Span::call_site(),
+ "must have a non-align #[repr(...)] attribute in order to guarantee this type's memory layout",
+ ));
+ };
+
+ let field_bounds = if require_unaligned_fields {
+ FieldBounds::All(&[TraitBound::Slf, TraitBound::Other(Trait::Unaligned)])
+ } else {
+ FieldBounds::ALL_SELF
+ };
+
+ Ok(ImplBlockBuilder::new(ctx, strct, Trait::IntoBytes, field_bounds)
+ .padding_check(padding_check)
+ .build())
+}
+
+fn derive_into_bytes_enum(ctx: &Ctx, enm: &DataEnum) -> Result<TokenStream, Error> {
+ let repr = EnumRepr::from_attrs(&ctx.ast.attrs)?;
+ if !repr.is_c() && !repr.is_primitive() {
+ return ctx.error_or_skip(Error::new(
+ Span::call_site(),
+ "must have #[repr(C)] or #[repr(Int)] attribute in order to guarantee this type's memory layout",
+ ));
+ }
+
+ let tag_type_definition = generate_tag_enum(ctx, &repr, enm);
+ Ok(ImplBlockBuilder::new(ctx, enm, Trait::IntoBytes, FieldBounds::ALL_SELF)
+ .padding_check(PaddingCheck::Enum { tag_type_definition })
+ .build())
+}
+
+fn derive_into_bytes_union(ctx: &Ctx, unn: &DataUnion) -> Result<TokenStream, Error> {
+ // See #1792 for more context.
+ //
+ // By checking for `zerocopy_derive_union_into_bytes` both here and in the
+ // generated code, we ensure that `--cfg zerocopy_derive_union_into_bytes`
+ // need only be passed *either* when compiling this crate *or* when
+ // compiling the user's crate. The former is preferable, but in some
+ // situations (such as when cross-compiling using `cargo build --target`),
+ // it doesn't get propagated to this crate's build by default.
+ let cfg_compile_error = if cfg!(zerocopy_derive_union_into_bytes) {
+ quote!()
+ } else {
+ let core = ctx.core_path();
+ let error_message = "requires --cfg zerocopy_derive_union_into_bytes;
+please let us know you use this feature: https://github.com/google/zerocopy/discussions/1802";
+ quote!(
+ #[allow(unused_attributes, unexpected_cfgs)]
+ const _: () = {
+ #[cfg(not(zerocopy_derive_union_into_bytes))]
+ #core::compile_error!(#error_message);
+ };
+ )
+ };
+
+ // FIXME(#10): Support type parameters.
+ if !ctx.ast.generics.params.is_empty() {
+ return ctx.error_or_skip(Error::new(
+ Span::call_site(),
+ "unsupported on types with type parameters",
+ ));
+ }
+
+ // Because we don't support generics, we don't need to worry about
+ // special-casing different reprs. So long as there is *some* repr which
+ // guarantees the layout, our `PaddingCheck::Union` guarantees that there is
+ // no padding.
+ let repr = StructUnionRepr::from_attrs(&ctx.ast.attrs)?;
+ if !repr.is_c() && !repr.is_transparent() && !repr.is_packed_1() {
+ return ctx.error_or_skip(Error::new(
+ Span::call_site(),
+ "must be #[repr(C)], #[repr(packed)], or #[repr(transparent)]",
+ ));
+ }
+
+ let impl_block = ImplBlockBuilder::new(ctx, unn, Trait::IntoBytes, FieldBounds::ALL_SELF)
+ .padding_check(PaddingCheck::Union)
+ .build();
+ Ok(quote!(#cfg_compile_error #impl_block))
+}
diff --git a/rust/zerocopy-derive/derive/known_layout.rs b/rust/zerocopy-derive/derive/known_layout.rs
new file mode 100644
index 0000000..fddffd1
--- /dev/null
+++ b/rust/zerocopy-derive/derive/known_layout.rs
@@ -0,0 +1,350 @@
+// SPDX-License-Identifier: (BSD-2-Clause OR Apache-2.0) OR MIT
+
+use proc_macro2::TokenStream;
+use quote::quote;
+use syn::{parse_quote, Data, Error, Type};
+
+use crate::{
+ repr::StructUnionRepr,
+ util::{Ctx, DataExt, FieldBounds, ImplBlockBuilder, SelfBounds, Trait},
+};
+
+fn derive_known_layout_for_repr_c_struct<'a>(
+ ctx: &'a Ctx,
+ repr: &StructUnionRepr,
+ fields: &[(&'a syn::Visibility, TokenStream, &'a Type)],
+) -> Option<(SelfBounds<'a>, TokenStream, Option<TokenStream>)> {
+ let (trailing_field, leading_fields) = fields.split_last()?;
+
+ let (_vis, trailing_field_name, trailing_field_ty) = trailing_field;
+ let leading_fields_tys = leading_fields.iter().map(|(_vis, _name, ty)| ty);
+
+ let core = ctx.core_path();
+ let repr_align = repr
+ .get_align()
+ .map(|align| {
+ let align = align.t.get();
+ quote!(#core::num::NonZeroUsize::new(#align as usize))
+ })
+ .unwrap_or_else(|| quote!(#core::option::Option::None));
+ let repr_packed = repr
+ .get_packed()
+ .map(|packed| {
+ let packed = packed.get();
+ quote!(#core::num::NonZeroUsize::new(#packed as usize))
+ })
+ .unwrap_or_else(|| quote!(#core::option::Option::None));
+
+ let zerocopy_crate = &ctx.zerocopy_crate;
+ let make_methods = |trailing_field_ty| {
+ quote! {
+ // SAFETY:
+ // - The returned pointer has the same address and provenance as
+ // `bytes`:
+ // - The recursive call to `raw_from_ptr_len` preserves both
+ // address and provenance.
+ // - The `as` cast preserves both address and provenance.
+ // - `NonNull::new_unchecked` preserves both address and
+ // provenance.
+ // - If `Self` is a slice DST, the returned pointer encodes
+ // `elems` elements in the trailing slice:
+ // - This is true of the recursive call to `raw_from_ptr_len`.
+ // - `trailing.as_ptr() as *mut Self` preserves trailing slice
+ // element count [1].
+ // - `NonNull::new_unchecked` preserves trailing slice element
+ // count.
+ //
+ // [1] Per https://doc.rust-lang.org/reference/expressions/operator-expr.html#pointer-to-pointer-cast:
+ //
+ // `*const T`` / `*mut T` can be cast to `*const U` / `*mut U`
+ // with the following behavior:
+ // ...
+ // - If `T` and `U` are both unsized, the pointer is also
+ // returned unchanged. In particular, the metadata is
+ // preserved exactly.
+ //
+ // For instance, a cast from `*const [T]` to `*const [U]`
+ // preserves the number of elements. ... The same holds
+ // for str and any compound type whose unsized tail is a
+ // slice type, such as struct `Foo(i32, [u8])` or
+ // `(u64, Foo)`.
+ #[inline(always)]
+ fn raw_from_ptr_len(
+ bytes: #core::ptr::NonNull<u8>,
+ meta: <Self as #zerocopy_crate::KnownLayout>::PointerMetadata,
+ ) -> #core::ptr::NonNull<Self> {
+ let trailing = <#trailing_field_ty as #zerocopy_crate::KnownLayout>::raw_from_ptr_len(bytes, meta);
+ let slf = trailing.as_ptr() as *mut Self;
+ // SAFETY: Constructed from `trailing`, which is non-null.
+ unsafe { #core::ptr::NonNull::new_unchecked(slf) }
+ }
+
+ #[inline(always)]
+ fn pointer_to_metadata(ptr: *mut Self) -> <Self as #zerocopy_crate::KnownLayout>::PointerMetadata {
+ <#trailing_field_ty>::pointer_to_metadata(ptr as *mut _)
+ }
+ }
+ };
+
+ let inner_extras = {
+ let leading_fields_tys = leading_fields_tys.clone();
+ let methods = make_methods(*trailing_field_ty);
+ let (_, ty_generics, _) = ctx.ast.generics.split_for_impl();
+
+ quote!(
+ type PointerMetadata = <#trailing_field_ty as #zerocopy_crate::KnownLayout>::PointerMetadata;
+
+ type MaybeUninit = __ZerocopyKnownLayoutMaybeUninit #ty_generics;
+
+ // SAFETY: `LAYOUT` accurately describes the layout of `Self`.
+ // The documentation of `DstLayout::for_repr_c_struct` vows that
+ // invocations in this manner will accurately describe a type,
+ // so long as:
+ //
+ // - that type is `repr(C)`,
+ // - its fields are enumerated in the order they appear,
+ // - the presence of `repr_align` and `repr_packed` are
+ // correctly accounted for.
+ //
+ // We respect all three of these preconditions here. This
+ // expansion is only used if `is_repr_c_struct`, we enumerate
+ // the fields in order, and we extract the values of `align(N)`
+ // and `packed(N)`.
+ const LAYOUT: #zerocopy_crate::DstLayout = #zerocopy_crate::DstLayout::for_repr_c_struct(
+ #repr_align,
+ #repr_packed,
+ &[
+ #(#zerocopy_crate::DstLayout::for_type::<#leading_fields_tys>(),)*
+ <#trailing_field_ty as #zerocopy_crate::KnownLayout>::LAYOUT
+ ],
+ );
+
+ #methods
+ )
+ };
+
+ let outer_extras = {
+ let ident = &ctx.ast.ident;
+ let vis = &ctx.ast.vis;
+ let params = &ctx.ast.generics.params;
+ let (impl_generics, ty_generics, where_clause) = ctx.ast.generics.split_for_impl();
+
+ let predicates = if let Some(where_clause) = where_clause {
+ where_clause.predicates.clone()
+ } else {
+ Default::default()
+ };
+
+ // Generate a valid ident for a type-level handle to a field of a
+ // given `name`.
+ let field_index = |name: &TokenStream| ident!(("__Zerocopy_Field_{}", name), ident.span());
+
+ let field_indices: Vec<_> =
+ fields.iter().map(|(_vis, name, _ty)| field_index(name)).collect();
+
+ // Define the collection of type-level field handles.
+ let field_defs = field_indices.iter().zip(fields).map(|(idx, (vis, _, _))| {
+ quote! {
+ #vis struct #idx;
+ }
+ });
+
+ let field_impls = field_indices.iter().zip(fields).map(|(idx, (_, _, ty))| quote! {
+ // SAFETY: `#ty` is the type of `#ident`'s field at `#idx`.
+ //
+ // We implement `Field` for each field of the struct to create a
+ // projection from the field index to its type. This allows us
+ // to refer to the field's type in a way that respects `Self`
+ // hygiene. If we just copy-pasted the tokens of `#ty`, we
+ // would not respect `Self` hygiene, as `Self` would refer to
+ // the helper struct we are generating, not the derive target
+ // type.
+ unsafe impl #impl_generics #zerocopy_crate::util::macro_util::Field<#idx> for #ident #ty_generics
+ where
+ #predicates
+ {
+ type Type = #ty;
+ }
+ });
+
+ let trailing_field_index = field_index(trailing_field_name);
+ let leading_field_indices =
+ leading_fields.iter().map(|(_vis, name, _ty)| field_index(name));
+
+ // We use `Field` to project the type of the trailing field. This is
+ // required to ensure that if the field type uses `Self`, it
+ // resolves to the derive target type, not the helper struct we are
+ // generating.
+ let trailing_field_ty = quote! {
+ <#ident #ty_generics as
+ #zerocopy_crate::util::macro_util::Field<#trailing_field_index>
+ >::Type
+ };
+
+ let methods = make_methods(&parse_quote! {
+ <#trailing_field_ty as #zerocopy_crate::KnownLayout>::MaybeUninit
+ });
+
+ let core = ctx.core_path();
+
+ quote! {
+ #(#field_defs)*
+
+ #(#field_impls)*
+
+ // SAFETY: This has the same layout as the derive target type,
+ // except that it admits uninit bytes. This is ensured by using
+ // the same repr as the target type, and by using field types
+ // which have the same layout as the target type's fields,
+ // except that they admit uninit bytes. We indirect through
+ // `Field` to ensure that occurrences of `Self` resolve to
+ // `#ty`, not `__ZerocopyKnownLayoutMaybeUninit` (see #2116).
+ #repr
+ #[doc(hidden)]
+ #vis struct __ZerocopyKnownLayoutMaybeUninit<#params> (
+ #(#core::mem::MaybeUninit<
+ <#ident #ty_generics as
+ #zerocopy_crate::util::macro_util::Field<#leading_field_indices>
+ >::Type
+ >,)*
+ // NOTE(#2302): We wrap in `ManuallyDrop` here in case the
+ // type we're operating on is both generic and
+ // `repr(packed)`. In that case, Rust needs to know that the
+ // type is *either* `Sized` or has a trivial `Drop`.
+ // `ManuallyDrop` has a trivial `Drop`, and so satisfies
+ // this requirement.
+ #core::mem::ManuallyDrop<
+ <#trailing_field_ty as #zerocopy_crate::KnownLayout>::MaybeUninit
+ >
+ )
+ where
+ #trailing_field_ty: #zerocopy_crate::KnownLayout,
+ #predicates;
+
+ // SAFETY: We largely defer to the `KnownLayout` implementation
+ // on the derive target type (both by using the same tokens, and
+ // by deferring to impl via type-level indirection). This is
+ // sound, since `__ZerocopyKnownLayoutMaybeUninit` is guaranteed
+ // to have the same layout as the derive target type, except
+ // that `__ZerocopyKnownLayoutMaybeUninit` admits uninit bytes.
+ unsafe impl #impl_generics #zerocopy_crate::KnownLayout for __ZerocopyKnownLayoutMaybeUninit #ty_generics
+ where
+ #trailing_field_ty: #zerocopy_crate::KnownLayout,
+ #predicates
+ {
+ fn only_derive_is_allowed_to_implement_this_trait() {}
+
+ type PointerMetadata = <#ident #ty_generics as #zerocopy_crate::KnownLayout>::PointerMetadata;
+
+ type MaybeUninit = Self;
+
+ const LAYOUT: #zerocopy_crate::DstLayout = <#ident #ty_generics as #zerocopy_crate::KnownLayout>::LAYOUT;
+
+ #methods
+ }
+ }
+ };
+
+ Some((SelfBounds::None, inner_extras, Some(outer_extras)))
+}
+
+pub(crate) fn derive(ctx: &Ctx, _top_level: Trait) -> Result<TokenStream, Error> {
+ // If this is a `repr(C)` struct, then `c_struct_repr` contains the entire
+ // `repr` attribute.
+ let c_struct_repr = match &ctx.ast.data {
+ Data::Struct(..) => {
+ let repr = StructUnionRepr::from_attrs(&ctx.ast.attrs)?;
+ if repr.is_c() {
+ Some(repr)
+ } else {
+ None
+ }
+ }
+ Data::Enum(..) | Data::Union(..) => None,
+ };
+
+ let fields = ctx.ast.data.fields();
+
+ let (self_bounds, inner_extras, outer_extras) = c_struct_repr
+ .as_ref()
+ .and_then(|repr| {
+ derive_known_layout_for_repr_c_struct(ctx, repr, &fields)
+ })
+ .unwrap_or_else(|| {
+ let zerocopy_crate = &ctx.zerocopy_crate;
+ let core = ctx.core_path();
+
+ // For enums, unions, and non-`repr(C)` structs, we require that
+ // `Self` is sized, and as a result don't need to reason about the
+ // internals of the type.
+ (
+ SelfBounds::SIZED,
+ quote!(
+ type PointerMetadata = ();
+ type MaybeUninit =
+ #core::mem::MaybeUninit<Self>;
+
+ // SAFETY: `LAYOUT` is guaranteed to accurately describe the
+ // layout of `Self`, because that is the documented safety
+ // contract of `DstLayout::for_type`.
+ const LAYOUT: #zerocopy_crate::DstLayout = #zerocopy_crate::DstLayout::for_type::<Self>();
+
+ // SAFETY: `.cast` preserves address and provenance.
+ //
+ // FIXME(#429): Add documentation to `.cast` that promises that
+ // it preserves provenance.
+ #[inline(always)]
+ fn raw_from_ptr_len(bytes: #core::ptr::NonNull<u8>, _meta: ()) -> #core::ptr::NonNull<Self> {
+ bytes.cast::<Self>()
+ }
+
+ #[inline(always)]
+ fn pointer_to_metadata(_ptr: *mut Self) -> () {}
+ ),
+ None,
+ )
+ });
+ Ok(match &ctx.ast.data {
+ Data::Struct(strct) => {
+ let require_trait_bound_on_field_types =
+ if matches!(self_bounds, SelfBounds::All(&[Trait::Sized])) {
+ FieldBounds::None
+ } else {
+ FieldBounds::TRAILING_SELF
+ };
+
+ // A bound on the trailing field is required, since structs are
+ // unsized if their trailing field is unsized. Reflecting the layout
+ // of an usized trailing field requires that the field is
+ // `KnownLayout`.
+ ImplBlockBuilder::new(
+ ctx,
+ strct,
+ Trait::KnownLayout,
+ require_trait_bound_on_field_types,
+ )
+ .self_type_trait_bounds(self_bounds)
+ .inner_extras(inner_extras)
+ .outer_extras(outer_extras)
+ .build()
+ }
+ Data::Enum(enm) => {
+ // A bound on the trailing field is not required, since enums cannot
+ // currently be unsized.
+ ImplBlockBuilder::new(ctx, enm, Trait::KnownLayout, FieldBounds::None)
+ .self_type_trait_bounds(SelfBounds::SIZED)
+ .inner_extras(inner_extras)
+ .outer_extras(outer_extras)
+ .build()
+ }
+ Data::Union(unn) => {
+ // A bound on the trailing field is not required, since unions
+ // cannot currently be unsized.
+ ImplBlockBuilder::new(ctx, unn, Trait::KnownLayout, FieldBounds::None)
+ .self_type_trait_bounds(SelfBounds::SIZED)
+ .inner_extras(inner_extras)
+ .outer_extras(outer_extras)
+ .build()
+ }
+ })
+}
diff --git a/rust/zerocopy-derive/derive/mod.rs b/rust/zerocopy-derive/derive/mod.rs
new file mode 100644
index 0000000..665ba7d
--- /dev/null
+++ b/rust/zerocopy-derive/derive/mod.rs
@@ -0,0 +1,132 @@
+// SPDX-License-Identifier: (BSD-2-Clause OR Apache-2.0) OR MIT
+
+pub mod from_bytes;
+pub mod into_bytes;
+pub mod known_layout;
+pub mod try_from_bytes;
+pub mod unaligned;
+
+use proc_macro2::{Span, TokenStream};
+use quote::quote;
+use syn::{Data, Error};
+
+use crate::{
+ repr::StructUnionRepr,
+ util::{Ctx, DataExt, FieldBounds, ImplBlockBuilder, Trait},
+};
+
+pub(crate) fn derive_immutable(ctx: &Ctx, _top_level: Trait) -> TokenStream {
+ match &ctx.ast.data {
+ Data::Struct(strct) => {
+ ImplBlockBuilder::new(ctx, strct, Trait::Immutable, FieldBounds::ALL_SELF).build()
+ }
+ Data::Enum(enm) => {
+ ImplBlockBuilder::new(ctx, enm, Trait::Immutable, FieldBounds::ALL_SELF).build()
+ }
+ Data::Union(unn) => {
+ ImplBlockBuilder::new(ctx, unn, Trait::Immutable, FieldBounds::ALL_SELF).build()
+ }
+ }
+}
+
+pub(crate) fn derive_hash(ctx: &Ctx, _top_level: Trait) -> Result<TokenStream, Error> {
+ // This doesn't delegate to `impl_block` because `impl_block` assumes it is
+ // deriving a `zerocopy`-defined trait, and these trait impls share a common
+ // shape that `Hash` does not. In particular, `zerocopy` traits contain a
+ // method that only `zerocopy_derive` macros are supposed to implement, and
+ // `impl_block` generating this trait method is incompatible with `Hash`.
+ let type_ident = &ctx.ast.ident;
+ let (impl_generics, ty_generics, where_clause) = ctx.ast.generics.split_for_impl();
+ let where_predicates = where_clause.map(|clause| &clause.predicates);
+ let zerocopy_crate = &ctx.zerocopy_crate;
+ let core = ctx.core_path();
+ Ok(quote! {
+ impl #impl_generics #core::hash::Hash for #type_ident #ty_generics
+ where
+ Self: #zerocopy_crate::IntoBytes + #zerocopy_crate::Immutable,
+ #where_predicates
+ {
+ fn hash<H: #core::hash::Hasher>(&self, state: &mut H) {
+ #core::hash::Hasher::write(state, #zerocopy_crate::IntoBytes::as_bytes(self))
+ }
+
+ fn hash_slice<H: #core::hash::Hasher>(data: &[Self], state: &mut H) {
+ #core::hash::Hasher::write(state, #zerocopy_crate::IntoBytes::as_bytes(data))
+ }
+ }
+ })
+}
+
+pub(crate) fn derive_eq(ctx: &Ctx, _top_level: Trait) -> Result<TokenStream, Error> {
+ // This doesn't delegate to `impl_block` because `impl_block` assumes it is
+ // deriving a `zerocopy`-defined trait, and these trait impls share a common
+ // shape that `Eq` does not. In particular, `zerocopy` traits contain a
+ // method that only `zerocopy_derive` macros are supposed to implement, and
+ // `impl_block` generating this trait method is incompatible with `Eq`.
+ let type_ident = &ctx.ast.ident;
+ let (impl_generics, ty_generics, where_clause) = ctx.ast.generics.split_for_impl();
+ let where_predicates = where_clause.map(|clause| &clause.predicates);
+ let zerocopy_crate = &ctx.zerocopy_crate;
+ let core = ctx.core_path();
+ Ok(quote! {
+ impl #impl_generics #core::cmp::PartialEq for #type_ident #ty_generics
+ where
+ Self: #zerocopy_crate::IntoBytes + #zerocopy_crate::Immutable,
+ #where_predicates
+ {
+ fn eq(&self, other: &Self) -> bool {
+ #core::cmp::PartialEq::eq(
+ #zerocopy_crate::IntoBytes::as_bytes(self),
+ #zerocopy_crate::IntoBytes::as_bytes(other),
+ )
+ }
+ }
+
+ impl #impl_generics #core::cmp::Eq for #type_ident #ty_generics
+ where
+ Self: #zerocopy_crate::IntoBytes + #zerocopy_crate::Immutable,
+ #where_predicates
+ {
+ }
+ })
+}
+
+pub(crate) fn derive_split_at(ctx: &Ctx, _top_level: Trait) -> Result<TokenStream, Error> {
+ let repr = StructUnionRepr::from_attrs(&ctx.ast.attrs)?;
+
+ match &ctx.ast.data {
+ Data::Struct(_) => {}
+ Data::Enum(_) | Data::Union(_) => {
+ return Err(Error::new(Span::call_site(), "can only be applied to structs"));
+ }
+ };
+
+ if repr.get_packed().is_some() {
+ return Err(Error::new(Span::call_site(), "must not have #[repr(packed)] attribute"));
+ }
+
+ if !(repr.is_c() || repr.is_transparent()) {
+ return Err(Error::new(
+ Span::call_site(),
+ "must have #[repr(C)] or #[repr(transparent)] in order to guarantee this type's layout is splitable",
+ ));
+ }
+
+ let fields = ctx.ast.data.fields();
+ let trailing_field = if let Some(((_, _, trailing_field), _)) = fields.split_last() {
+ trailing_field
+ } else {
+ return Err(Error::new(Span::call_site(), "must at least one field"));
+ };
+
+ let zerocopy_crate = &ctx.zerocopy_crate;
+ // SAFETY: `#ty`, per the above checks, is `repr(C)` or `repr(transparent)`
+ // and is not packed; its trailing field is guaranteed to be well-aligned
+ // for its type. By invariant on `FieldBounds::TRAILING_SELF`, the trailing
+ // slice of the trailing field is also well-aligned for its type.
+ Ok(ImplBlockBuilder::new(ctx, &ctx.ast.data, Trait::SplitAt, FieldBounds::TRAILING_SELF)
+ .inner_extras(quote! {
+ type Elem = <#trailing_field as #zerocopy_crate::SplitAt>::Elem;
+ })
+ .build())
+}
diff --git a/rust/zerocopy-derive/derive/try_from_bytes.rs b/rust/zerocopy-derive/derive/try_from_bytes.rs
new file mode 100644
index 0000000..a3e4a75
--- /dev/null
+++ b/rust/zerocopy-derive/derive/try_from_bytes.rs
@@ -0,0 +1,765 @@
+// SPDX-License-Identifier: (BSD-2-Clause OR Apache-2.0) OR MIT
+
+use proc_macro2::TokenStream;
+use quote::quote;
+use syn::{
+ parse_quote, spanned::Spanned as _, Data, DataEnum, DataStruct, DataUnion, DeriveInput, Error,
+ Expr, Fields, Ident, Index, Type,
+};
+
+use crate::{
+ repr::{EnumRepr, StructUnionRepr},
+ util::{
+ const_block, enum_size_from_repr, generate_tag_enum, Ctx, DataExt, FieldBounds,
+ ImplBlockBuilder, Trait, TraitBound,
+ },
+};
+fn tag_ident(variant_ident: &Ident) -> Ident {
+ ident!(("___ZEROCOPY_TAG_{}", variant_ident), variant_ident.span())
+}
+
+/// Generates a constant for the tag associated with each variant of the enum.
+/// When we match on the enum's tag, each arm matches one of these constants. We
+/// have to use constants here because:
+///
+/// - The type that we're matching on is not the type of the tag, it's an
+/// integer of the same size as the tag type and with the same bit patterns.
+/// - We can't read the enum tag as an enum because the bytes may not represent
+/// a valid variant.
+/// - Patterns do not currently support const expressions, so we have to assign
+/// these constants to names rather than use them inline in the `match`
+/// statement.
+fn generate_tag_consts(data: &DataEnum) -> TokenStream {
+ let tags = data.variants.iter().map(|v| {
+ let variant_ident = &v.ident;
+ let tag_ident = tag_ident(variant_ident);
+
+ quote! {
+ // This casts the enum variant to its discriminant, and then
+ // converts the discriminant to the target integral type via a
+ // numeric cast [1].
+ //
+ // Because these are the same size, this is defined to be a no-op
+ // and therefore is a lossless conversion [2].
+ //
+ // [1] Per https://doc.rust-lang.org/1.81.0/reference/expressions/operator-expr.html#enum-cast:
+ //
+ // Casts an enum to its discriminant.
+ //
+ // [2] Per https://doc.rust-lang.org/1.81.0/reference/expressions/operator-expr.html#numeric-cast:
+ //
+ // Casting between two integers of the same size (e.g. i32 -> u32)
+ // is a no-op.
+ const #tag_ident: ___ZerocopyTagPrimitive =
+ ___ZerocopyTag::#variant_ident as ___ZerocopyTagPrimitive;
+ }
+ });
+
+ quote! {
+ #(#tags)*
+ }
+}
+
+fn variant_struct_ident(variant_ident: &Ident) -> Ident {
+ ident!(("___ZerocopyVariantStruct_{}", variant_ident), variant_ident.span())
+}
+
+/// Generates variant structs for the given enum variant.
+///
+/// These are structs associated with each variant of an enum. They are
+/// `repr(C)` tuple structs with the same fields as the variant after a
+/// `MaybeUninit<___ZerocopyInnerTag>`.
+///
+/// In order to unify the generated types for `repr(C)` and `repr(int)` enums,
+/// we use a "fused" representation with fields for both an inner tag and an
+/// outer tag. Depending on the repr, we will set one of these tags to the tag
+/// type and the other to `()`. This lets us generate the same code but put the
+/// tags in different locations.
+fn generate_variant_structs(ctx: &Ctx, data: &DataEnum) -> TokenStream {
+ let (impl_generics, ty_generics, where_clause) = ctx.ast.generics.split_for_impl();
+
+ let enum_name = &ctx.ast.ident;
+
+ // All variant structs have a `PhantomData<MyEnum<...>>` field because we
+ // don't know which generic parameters each variant will use, and unused
+ // generic parameters are a compile error.
+ let core = ctx.core_path();
+ let phantom_ty = quote! {
+ #core::marker::PhantomData<#enum_name #ty_generics>
+ };
+
+ let variant_structs = data.variants.iter().filter_map(|variant| {
+ // We don't generate variant structs for unit variants because we only
+ // need to check the tag. This helps cut down our generated code a bit.
+ if matches!(variant.fields, Fields::Unit) {
+ return None;
+ }
+
+ let variant_struct_ident = variant_struct_ident(&variant.ident);
+ let field_types = variant.fields.iter().map(|f| &f.ty);
+
+ let variant_struct = parse_quote! {
+ #[repr(C)]
+ struct #variant_struct_ident #impl_generics (
+ #core::mem::MaybeUninit<___ZerocopyInnerTag>,
+ #(#field_types,)*
+ #phantom_ty,
+ ) #where_clause;
+ };
+
+ // We do this rather than emitting `#[derive(::zerocopy::TryFromBytes)]`
+ // because that is not hygienic, and this is also more performant.
+ let try_from_bytes_impl =
+ derive_try_from_bytes(&ctx.with_input(&variant_struct), Trait::TryFromBytes)
+ .expect("derive_try_from_bytes should not fail on synthesized type");
+
+ Some(quote! {
+ #variant_struct
+ #try_from_bytes_impl
+ })
+ });
+
+ quote! {
+ #(#variant_structs)*
+ }
+}
+
+fn variants_union_field_ident(ident: &Ident) -> Ident {
+ // Field names are prefixed with `__field_` to prevent name collision
+ // with the `__nonempty` field.
+ ident!(("__field_{}", ident), ident.span())
+}
+
+fn generate_variants_union(ctx: &Ctx, data: &DataEnum) -> TokenStream {
+ let generics = &ctx.ast.generics;
+ let (_, ty_generics, _) = generics.split_for_impl();
+
+ let fields = data.variants.iter().filter_map(|variant| {
+ // We don't generate variant structs for unit variants because we only
+ // need to check the tag. This helps cut down our generated code a bit.
+ if matches!(variant.fields, Fields::Unit) {
+ return None;
+ }
+
+ let field_name = variants_union_field_ident(&variant.ident);
+ let variant_struct_ident = variant_struct_ident(&variant.ident);
+
+ let core = ctx.core_path();
+ Some(quote! {
+ #field_name: #core::mem::ManuallyDrop<#variant_struct_ident #ty_generics>,
+ })
+ });
+
+ let variants_union = parse_quote! {
+ #[repr(C)]
+ union ___ZerocopyVariants #generics {
+ #(#fields)*
+ // Enums can have variants with no fields, but unions must
+ // have at least one field. So we just add a trailing unit
+ // to ensure that this union always has at least one field.
+ // Because this union is `repr(C)`, this unit type does not
+ // affect the layout.
+ __nonempty: (),
+ }
+ };
+
+ let has_field =
+ derive_has_field_struct_union(&ctx.with_input(&variants_union), &variants_union.data);
+
+ quote! {
+ #variants_union
+ #has_field
+ }
+}
+
+/// Generates an implementation of `is_bit_valid` for an arbitrary enum.
+///
+/// The general process is:
+///
+/// 1. Generate a tag enum. This is an enum with the same repr, variants, and
+/// corresponding discriminants as the original enum, but without any fields
+/// on the variants. This gives us access to an enum where the variants have
+/// the same discriminants as the one we're writing `is_bit_valid` for.
+/// 2. Make constants from the variants of the tag enum. We need these because
+/// we can't put const exprs in match arms.
+/// 3. Generate variant structs. These are structs which have the same fields as
+/// each variant of the enum, and are `#[repr(C)]` with an optional "inner
+/// tag".
+/// 4. Generate a variants union, with one field for each variant struct type.
+/// 5. And finally, our raw enum is a `#[repr(C)]` struct of an "outer tag" and
+/// the variants union.
+///
+/// See these reference links for fully-worked example decompositions.
+///
+/// - `repr(C)`: <https://doc.rust-lang.org/reference/type-layout.html#reprc-enums-with-fields>
+/// - `repr(int)`: <https://doc.rust-lang.org/reference/type-layout.html#primitive-representation-of-enums-with-fields>
+/// - `repr(C, int)`: <https://doc.rust-lang.org/reference/type-layout.html#combining-primitive-representations-of-enums-with-fields-and-reprc>
+pub(crate) fn derive_is_bit_valid(
+ ctx: &Ctx,
+ data: &DataEnum,
+ repr: &EnumRepr,
+) -> Result<TokenStream, Error> {
+ let trait_path = Trait::TryFromBytes.crate_path(ctx);
+ let tag_enum = generate_tag_enum(ctx, repr, data);
+ let tag_consts = generate_tag_consts(data);
+
+ let (outer_tag_type, inner_tag_type) = if repr.is_c() {
+ (quote! { ___ZerocopyTag }, quote! { () })
+ } else if repr.is_primitive() {
+ (quote! { () }, quote! { ___ZerocopyTag })
+ } else {
+ return Err(Error::new(
+ ctx.ast.span(),
+ "must have #[repr(C)] or #[repr(Int)] attribute in order to guarantee this type's memory layout",
+ ));
+ };
+
+ let variant_structs = generate_variant_structs(ctx, data);
+ let variants_union = generate_variants_union(ctx, data);
+
+ let (impl_generics, ty_generics, where_clause) = ctx.ast.generics.split_for_impl();
+
+ let zerocopy_crate = &ctx.zerocopy_crate;
+ let has_tag = ImplBlockBuilder::new(ctx, data, Trait::HasTag, FieldBounds::None)
+ .inner_extras(quote! {
+ type Tag = ___ZerocopyTag;
+ type ProjectToTag = #zerocopy_crate::pointer::cast::CastSized;
+ })
+ .build();
+ let has_fields = data.variants().into_iter().flat_map(|(variant, fields)| {
+ let variant_ident = &variant.unwrap().ident;
+ let variants_union_field_ident = variants_union_field_ident(variant_ident);
+ let field: Box<syn::Type> = parse_quote!(());
+ fields.into_iter().enumerate().map(move |(idx, (vis, ident, ty))| {
+ // Rust does not presently support explicit visibility modifiers on
+ // enum fields, but we guard against the possibility to ensure this
+ // derive remains sound.
+ assert!(matches!(vis, syn::Visibility::Inherited));
+ let variant_struct_field_index = Index::from(idx + 1);
+ let (_, ty_generics, _) = ctx.ast.generics.split_for_impl();
+ let has_field_trait = Trait::HasField {
+ variant_id: parse_quote!({ #zerocopy_crate::ident_id!(#variant_ident) }),
+ // Since Rust does not presently support explicit visibility
+ // modifiers on enum fields, any public type is suitable here;
+ // we use `()`.
+ field: field.clone(),
+ field_id: parse_quote!({ #zerocopy_crate::ident_id!(#ident) }),
+ };
+ let has_field_path = has_field_trait.crate_path(ctx);
+ let has_field = ImplBlockBuilder::new(
+ ctx,
+ data,
+ has_field_trait,
+ FieldBounds::None,
+ )
+ .inner_extras(quote! {
+ type Type = #ty;
+
+ #[inline(always)]
+ fn project(slf: #zerocopy_crate::pointer::PtrInner<'_, Self>) -> *mut <Self as #has_field_path>::Type {
+ use #zerocopy_crate::pointer::cast::{CastSized, Projection};
+
+ slf.project::<___ZerocopyRawEnum #ty_generics, CastSized>()
+ .project::<_, Projection<_, { #zerocopy_crate::STRUCT_VARIANT_ID }, { #zerocopy_crate::ident_id!(variants) }>>()
+ .project::<_, Projection<_, { #zerocopy_crate::REPR_C_UNION_VARIANT_ID }, { #zerocopy_crate::ident_id!(#variants_union_field_ident) }>>()
+ .project::<_, Projection<_, { #zerocopy_crate::STRUCT_VARIANT_ID }, { #zerocopy_crate::ident_id!(value) }>>()
+ .project::<_, Projection<_, { #zerocopy_crate::STRUCT_VARIANT_ID }, { #zerocopy_crate::ident_id!(#variant_struct_field_index) }>>()
+ .as_ptr()
+ }
+ })
+ .build();
+
+ let project = ImplBlockBuilder::new(
+ ctx,
+ data,
+ Trait::ProjectField {
+ variant_id: parse_quote!({ #zerocopy_crate::ident_id!(#variant_ident) }),
+ // Since Rust does not presently support explicit visibility
+ // modifiers on enum fields, any public type is suitable
+ // here; we use `()`.
+ field: field.clone(),
+ field_id: parse_quote!({ #zerocopy_crate::ident_id!(#ident) }),
+ invariants: parse_quote!((Aliasing, Alignment, #zerocopy_crate::invariant::Initialized)),
+ },
+ FieldBounds::None,
+ )
+ .param_extras(vec![
+ parse_quote!(Aliasing: #zerocopy_crate::invariant::Aliasing),
+ parse_quote!(Alignment: #zerocopy_crate::invariant::Alignment),
+ ])
+ .inner_extras(quote! {
+ type Error = #zerocopy_crate::util::macro_util::core_reexport::convert::Infallible;
+ type Invariants = (Aliasing, Alignment, #zerocopy_crate::invariant::Initialized);
+ })
+ .build();
+
+ quote! {
+ #has_field
+ #project
+ }
+ })
+ });
+
+ let core = ctx.core_path();
+ let match_arms = data.variants.iter().map(|variant| {
+ let tag_ident = tag_ident(&variant.ident);
+ let variant_struct_ident = variant_struct_ident(&variant.ident);
+ let variants_union_field_ident = variants_union_field_ident(&variant.ident);
+
+ if matches!(variant.fields, Fields::Unit) {
+ // Unit variants don't need any further validation beyond checking
+ // the tag.
+ quote! {
+ #tag_ident => true
+ }
+ } else {
+ quote! {
+ #tag_ident => {
+ // SAFETY: Since we know that the tag is `#tag_ident`, we
+ // know that no other `&`s exist which refer to this enum
+ // as any other variant.
+ let variant_md = variants.cast::<
+ _,
+ #zerocopy_crate::pointer::cast::Projection<
+ // #zerocopy_crate::ReadOnly<_>,
+ _,
+ { #zerocopy_crate::REPR_C_UNION_VARIANT_ID },
+ { #zerocopy_crate::ident_id!(#variants_union_field_ident) }
+ >,
+ _
+ >();
+ let variant = variant_md.cast::<
+ #zerocopy_crate::ReadOnly<#variant_struct_ident #ty_generics>,
+ #zerocopy_crate::pointer::cast::CastSized,
+ (#zerocopy_crate::pointer::BecauseRead, _)
+ >();
+ <
+ #variant_struct_ident #ty_generics as #trait_path
+ >::is_bit_valid(variant)
+ }
+ }
+ }
+ });
+
+ let generics = &ctx.ast.generics;
+ let raw_enum: DeriveInput = parse_quote! {
+ #[repr(C)]
+ struct ___ZerocopyRawEnum #generics {
+ tag: ___ZerocopyOuterTag,
+ variants: ___ZerocopyVariants #ty_generics,
+ }
+ };
+
+ let self_ident = &ctx.ast.ident;
+ let invariants_eq_impl = quote! {
+ // SAFETY: `___ZerocopyRawEnum` is designed to have the same layout,
+ // validity, and invariants as `Self`.
+ unsafe impl #impl_generics #zerocopy_crate::pointer::InvariantsEq<___ZerocopyRawEnum #ty_generics> for #self_ident #ty_generics #where_clause {}
+ };
+
+ let raw_enum_projections =
+ derive_has_field_struct_union(&ctx.with_input(&raw_enum), &raw_enum.data);
+
+ let raw_enum = quote! {
+ #raw_enum
+ #invariants_eq_impl
+ #raw_enum_projections
+ };
+
+ Ok(quote! {
+ // SAFETY: We use `is_bit_valid` to validate that the bit pattern of the
+ // enum's tag corresponds to one of the enum's discriminants. Then, we
+ // check the bit validity of each field of the corresponding variant.
+ // Thus, this is a sound implementation of `is_bit_valid`.
+ #[inline]
+ fn is_bit_valid<___ZcAlignment>(
+ mut candidate: #zerocopy_crate::Maybe<'_, Self, ___ZcAlignment>,
+ ) -> #core::primitive::bool
+ where
+ ___ZcAlignment: #zerocopy_crate::invariant::Alignment,
+ {
+ #tag_enum
+
+ type ___ZerocopyTagPrimitive = #zerocopy_crate::util::macro_util::SizeToTag<
+ { #core::mem::size_of::<___ZerocopyTag>() },
+ >;
+
+ #tag_consts
+
+ type ___ZerocopyOuterTag = #outer_tag_type;
+ type ___ZerocopyInnerTag = #inner_tag_type;
+
+ #variant_structs
+
+ #variants_union
+
+ #raw_enum
+
+ #has_tag
+
+ #(#has_fields)*
+
+ let tag = {
+ // SAFETY:
+ // - The provided cast addresses a subset of the bytes addressed
+ // by `candidate` because it addresses the starting tag of the
+ // enum.
+ // - Because the pointer is cast from `candidate`, it has the
+ // same provenance as it.
+ // - There are no `UnsafeCell`s in the tag because it is a
+ // primitive integer.
+ // - `tag_ptr` is casted from `candidate`, whose referent is
+ // `Initialized`. Since we have not written uninitialized
+ // bytes into the referent, `tag_ptr` is also `Initialized`.
+ //
+ // FIXME(#2874): Revise this to a `cast` once `candidate`
+ // references a `ReadOnly<Self>`.
+ let tag_ptr = unsafe {
+ candidate.reborrow().project_transmute_unchecked::<
+ _,
+ #zerocopy_crate::invariant::Initialized,
+ #zerocopy_crate::pointer::cast::CastSized
+ >()
+ };
+ tag_ptr.recall_validity::<_, (_, (_, _))>().read::<#zerocopy_crate::BecauseImmutable>()
+ };
+
+ let mut raw_enum = candidate.cast::<
+ #zerocopy_crate::ReadOnly<___ZerocopyRawEnum #ty_generics>,
+ #zerocopy_crate::pointer::cast::CastSized,
+ (#zerocopy_crate::pointer::BecauseRead, _)
+ >();
+
+ let variants = #zerocopy_crate::into_inner!(raw_enum.project::<
+ _,
+ { #zerocopy_crate::STRUCT_VARIANT_ID },
+ { #zerocopy_crate::ident_id!(variants) }
+ >());
+
+ match tag {
+ #(#match_arms,)*
+ _ => false,
+ }
+ }
+ })
+}
+pub(crate) fn derive_try_from_bytes(ctx: &Ctx, top_level: Trait) -> Result<TokenStream, Error> {
+ match &ctx.ast.data {
+ Data::Struct(strct) => derive_try_from_bytes_struct(ctx, strct, top_level),
+ Data::Enum(enm) => derive_try_from_bytes_enum(ctx, enm, top_level),
+ Data::Union(unn) => Ok(derive_try_from_bytes_union(ctx, unn, top_level)),
+ }
+}
+fn derive_has_field_struct_union(ctx: &Ctx, data: &dyn DataExt) -> TokenStream {
+ let fields = ctx.ast.data.fields();
+ if fields.is_empty() {
+ return quote! {};
+ }
+
+ let field_tokens = fields.iter().map(|(vis, ident, _)| {
+ let ident = ident!(("__z{}", ident), ident.span());
+ quote!(
+ #vis enum #ident {}
+ )
+ });
+
+ let zerocopy_crate = &ctx.zerocopy_crate;
+ let variant_id: Box<Expr> = match &ctx.ast.data {
+ Data::Struct(_) => parse_quote!({ #zerocopy_crate::STRUCT_VARIANT_ID }),
+ Data::Union(_) => {
+ let is_repr_c = StructUnionRepr::from_attrs(&ctx.ast.attrs)
+ .map(|repr| repr.is_c())
+ .unwrap_or(false);
+ if is_repr_c {
+ parse_quote!({ #zerocopy_crate::REPR_C_UNION_VARIANT_ID })
+ } else {
+ parse_quote!({ #zerocopy_crate::UNION_VARIANT_ID })
+ }
+ }
+ _ => unreachable!(),
+ };
+
+ let core = ctx.core_path();
+ let has_tag = ImplBlockBuilder::new(ctx, data, Trait::HasTag, FieldBounds::None)
+ .inner_extras(quote! {
+ type Tag = ();
+ type ProjectToTag = #zerocopy_crate::pointer::cast::CastToUnit;
+ })
+ .build();
+ let has_fields = fields.iter().map(move |(_, ident, ty)| {
+ let field_token = ident!(("__z{}", ident), ident.span());
+ let field: Box<Type> = parse_quote!(#field_token);
+ let field_id: Box<Expr> = parse_quote!({ #zerocopy_crate::ident_id!(#ident) });
+ let has_field_trait = Trait::HasField {
+ variant_id: variant_id.clone(),
+ field: field.clone(),
+ field_id: field_id.clone(),
+ };
+ let has_field_path = has_field_trait.crate_path(ctx);
+ ImplBlockBuilder::new(
+ ctx,
+ data,
+ has_field_trait,
+ FieldBounds::None,
+ )
+ .inner_extras(quote! {
+ type Type = #ty;
+
+ #[inline(always)]
+ fn project(slf: #zerocopy_crate::pointer::PtrInner<'_, Self>) -> *mut <Self as #has_field_path>::Type {
+ let slf = slf.as_ptr();
+ // SAFETY: By invariant on `PtrInner`, `slf` is a non-null
+ // pointer whose referent is zero-sized or lives in a valid
+ // allocation. Since `#ident` is a struct or union field of
+ // `Self`, this projection preserves or shrinks the referent
+ // size, and so the resulting referent also fits in the same
+ // allocation.
+ unsafe { #core::ptr::addr_of_mut!((*slf).#ident) }
+ }
+ }).outer_extras(if matches!(&ctx.ast.data, Data::Struct(..)) {
+ let fields_preserve_alignment = StructUnionRepr::from_attrs(&ctx.ast.attrs)
+ .map(|repr| repr.get_packed().is_none())
+ .unwrap();
+ let alignment = if fields_preserve_alignment {
+ quote! { Alignment }
+ } else {
+ quote! { #zerocopy_crate::invariant::Unaligned }
+ };
+ // SAFETY: See comments on items.
+ ImplBlockBuilder::new(
+ ctx,
+ data,
+ Trait::ProjectField {
+ variant_id: variant_id.clone(),
+ field: field.clone(),
+ field_id: field_id.clone(),
+ invariants: parse_quote!((Aliasing, Alignment, #zerocopy_crate::invariant::Initialized)),
+ },
+ FieldBounds::None,
+ )
+ .param_extras(vec![
+ parse_quote!(Aliasing: #zerocopy_crate::invariant::Aliasing),
+ parse_quote!(Alignment: #zerocopy_crate::invariant::Alignment),
+ ])
+ .inner_extras(quote! {
+ // SAFETY: Projection into structs is always infallible.
+ type Error = #zerocopy_crate::util::macro_util::core_reexport::convert::Infallible;
+ // SAFETY: The alignment of the projected `Ptr` is `Unaligned`
+ // if the structure is packed; otherwise inherited from the
+ // outer `Ptr`. If the validity of the outer pointer is
+ // `Initialized`, so too is the validity of its fields.
+ type Invariants = (Aliasing, #alignment, #zerocopy_crate::invariant::Initialized);
+ })
+ .build()
+ } else {
+ quote! {}
+ })
+ .build()
+ });
+
+ const_block(field_tokens.into_iter().chain(Some(has_tag)).chain(has_fields).map(Some))
+}
+fn derive_try_from_bytes_struct(
+ ctx: &Ctx,
+ strct: &DataStruct,
+ top_level: Trait,
+) -> Result<TokenStream, Error> {
+ let extras = try_gen_trivial_is_bit_valid(ctx, top_level).unwrap_or_else(|| {
+ let zerocopy_crate = &ctx.zerocopy_crate;
+ let fields = strct.fields();
+ let field_names = fields.iter().map(|(_vis, name, _ty)| name);
+ let field_tys = fields.iter().map(|(_vis, _name, ty)| ty);
+ let core = ctx.core_path();
+ quote!(
+ // SAFETY: We use `is_bit_valid` to validate that each field is
+ // bit-valid, and only return `true` if all of them are. The bit
+ // validity of a struct is just the composition of the bit
+ // validities of its fields, so this is a sound implementation
+ // of `is_bit_valid`.
+ #[inline]
+ fn is_bit_valid<___ZcAlignment>(
+ mut candidate: #zerocopy_crate::Maybe<'_, Self, ___ZcAlignment>,
+ ) -> #core::primitive::bool
+ where
+ ___ZcAlignment: #zerocopy_crate::invariant::Alignment,
+ {
+ true #(&& {
+ let field_candidate = #zerocopy_crate::into_inner!(candidate.reborrow().project::<
+ _,
+ { #zerocopy_crate::STRUCT_VARIANT_ID },
+ { #zerocopy_crate::ident_id!(#field_names) }
+ >());
+ <#field_tys as #zerocopy_crate::TryFromBytes>::is_bit_valid(field_candidate)
+ })*
+ }
+ )
+ });
+ Ok(ImplBlockBuilder::new(ctx, strct, Trait::TryFromBytes, FieldBounds::ALL_SELF)
+ .inner_extras(extras)
+ .outer_extras(derive_has_field_struct_union(ctx, strct))
+ .build())
+}
+fn derive_try_from_bytes_union(ctx: &Ctx, unn: &DataUnion, top_level: Trait) -> TokenStream {
+ let field_type_trait_bounds = FieldBounds::All(&[TraitBound::Slf]);
+
+ let zerocopy_crate = &ctx.zerocopy_crate;
+ let variant_id: Box<Expr> = {
+ let is_repr_c =
+ StructUnionRepr::from_attrs(&ctx.ast.attrs).map(|repr| repr.is_c()).unwrap_or(false);
+ if is_repr_c {
+ parse_quote!({ #zerocopy_crate::REPR_C_UNION_VARIANT_ID })
+ } else {
+ parse_quote!({ #zerocopy_crate::UNION_VARIANT_ID })
+ }
+ };
+
+ let extras = try_gen_trivial_is_bit_valid(ctx, top_level).unwrap_or_else(|| {
+ let fields = unn.fields();
+ let field_names = fields.iter().map(|(_vis, name, _ty)| name);
+ let field_tys = fields.iter().map(|(_vis, _name, ty)| ty);
+ let core = ctx.core_path();
+ quote!(
+ // SAFETY: We use `is_bit_valid` to validate that any field is
+ // bit-valid; we only return `true` if at least one of them is.
+ // The bit validity of a union is not yet well defined in Rust,
+ // but it is guaranteed to be no more strict than this
+ // definition. See #696 for a more in-depth discussion.
+ #[inline]
+ fn is_bit_valid<___ZcAlignment>(
+ mut candidate: #zerocopy_crate::Maybe<'_, Self, ___ZcAlignment>,
+ ) -> #core::primitive::bool
+ where
+ ___ZcAlignment: #zerocopy_crate::invariant::Alignment,
+ {
+ false #(|| {
+ // SAFETY:
+ // - Since `ReadOnly<Self>: Immutable` unconditionally,
+ // neither `*slf` nor the returned pointer's referent
+ // permit interior mutation.
+ // - Both source and destination validity are
+ // `Initialized`, which is always a sound
+ // transmutation.
+ let field_candidate = unsafe {
+ candidate.reborrow().project_transmute_unchecked::<
+ _,
+ _,
+ #zerocopy_crate::pointer::cast::Projection<
+ _,
+ #variant_id,
+ { #zerocopy_crate::ident_id!(#field_names) }
+ >
+ >()
+ };
+
+ <#field_tys as #zerocopy_crate::TryFromBytes>::is_bit_valid(field_candidate)
+ })*
+ }
+ )
+ });
+ ImplBlockBuilder::new(ctx, unn, Trait::TryFromBytes, field_type_trait_bounds)
+ .inner_extras(extras)
+ .outer_extras(derive_has_field_struct_union(ctx, unn))
+ .build()
+}
+fn derive_try_from_bytes_enum(
+ ctx: &Ctx,
+ enm: &DataEnum,
+ top_level: Trait,
+) -> Result<TokenStream, Error> {
+ let repr = EnumRepr::from_attrs(&ctx.ast.attrs)?;
+
+ // If an enum has no fields, it has a well-defined integer representation,
+ // and every possible bit pattern corresponds to a valid discriminant tag,
+ // then it *could* be `FromBytes` (even if the user hasn't derived
+ // `FromBytes`). This holds if, for `repr(uN)` or `repr(iN)`, there are 2^N
+ // variants.
+ let could_be_from_bytes = enum_size_from_repr(&repr)
+ .map(|size| enm.fields().is_empty() && enm.variants.len() == 1usize << size)
+ .unwrap_or(false);
+
+ let trivial_is_bit_valid = try_gen_trivial_is_bit_valid(ctx, top_level);
+ let extra = match (trivial_is_bit_valid, could_be_from_bytes) {
+ (Some(is_bit_valid), _) => is_bit_valid,
+ // SAFETY: It would be sound for the enum to implement `FromBytes`, as
+ // required by `gen_trivial_is_bit_valid_unchecked`.
+ (None, true) => unsafe { gen_trivial_is_bit_valid_unchecked(ctx) },
+ (None, false) => match derive_is_bit_valid(ctx, enm, &repr) {
+ Ok(extra) => extra,
+ Err(_) if ctx.skip_on_error => return Ok(TokenStream::new()),
+ Err(e) => return Err(e),
+ },
+ };
+
+ Ok(ImplBlockBuilder::new(ctx, enm, Trait::TryFromBytes, FieldBounds::ALL_SELF)
+ .inner_extras(extra)
+ .build())
+}
+fn try_gen_trivial_is_bit_valid(ctx: &Ctx, top_level: Trait) -> Option<proc_macro2::TokenStream> {
+ // If the top-level trait is `FromBytes` and `Self` has no type parameters,
+ // then the `FromBytes` derive will fail compilation if `Self` is not
+ // actually soundly `FromBytes`, and so we can rely on that for our
+ // `is_bit_valid` impl. It's plausible that we could make changes - or Rust
+ // could make changes (such as the "trivial bounds" language feature) - that
+ // make this no longer true. To hedge against these, we include an explicit
+ // `Self: FromBytes` check in the generated `is_bit_valid`, which is
+ // bulletproof.
+ //
+ // If `ctx.skip_on_error` is true, we can't rely on the `FromBytes` derive
+ // to fail compilation if `Self` is not actually soundly `FromBytes`.
+ if matches!(top_level, Trait::FromBytes)
+ && ctx.ast.generics.params.is_empty()
+ && !ctx.skip_on_error
+ {
+ let zerocopy_crate = &ctx.zerocopy_crate;
+ let core = ctx.core_path();
+ Some(quote!(
+ // SAFETY: See inline.
+ #[inline(always)]
+ fn is_bit_valid<___ZcAlignment>(
+ _candidate: #zerocopy_crate::Maybe<'_, Self, ___ZcAlignment>,
+ ) -> #core::primitive::bool
+ where
+ ___ZcAlignment: #zerocopy_crate::invariant::Alignment,
+ {
+ if false {
+ fn assert_is_from_bytes<T>()
+ where
+ T: #zerocopy_crate::FromBytes,
+ T: ?#core::marker::Sized,
+ {
+ }
+
+ assert_is_from_bytes::<Self>();
+ }
+
+ // SAFETY: The preceding code only compiles if `Self:
+ // FromBytes`. Thus, this code only compiles if all initialized
+ // byte sequences represent valid instances of `Self`.
+ true
+ }
+ ))
+ } else {
+ None
+ }
+}
+
+/// # Safety
+///
+/// All initialized bit patterns must be valid for `Self`.
+unsafe fn gen_trivial_is_bit_valid_unchecked(ctx: &Ctx) -> proc_macro2::TokenStream {
+ let zerocopy_crate = &ctx.zerocopy_crate;
+ let core = ctx.core_path();
+ quote!(
+ // SAFETY: The caller of `gen_trivial_is_bit_valid_unchecked` has
+ // promised that all initialized bit patterns are valid for `Self`.
+ #[inline(always)]
+ fn is_bit_valid<___ZcAlignment>(
+ _candidate: #zerocopy_crate::Maybe<'_, Self, ___ZcAlignment>,
+ ) -> #core::primitive::bool
+ where
+ ___ZcAlignment: #zerocopy_crate::invariant::Alignment,
+ {
+ true
+ }
+ )
+}
diff --git a/rust/zerocopy-derive/derive/unaligned.rs b/rust/zerocopy-derive/derive/unaligned.rs
new file mode 100644
index 0000000..d6dea0a
--- /dev/null
+++ b/rust/zerocopy-derive/derive/unaligned.rs
@@ -0,0 +1,80 @@
+// SPDX-License-Identifier: (BSD-2-Clause OR Apache-2.0) OR MIT
+
+use proc_macro2::{Span, TokenStream};
+use syn::{Data, DataEnum, DataStruct, DataUnion, Error};
+
+use crate::{
+ repr::{EnumRepr, StructUnionRepr},
+ util::{Ctx, FieldBounds, ImplBlockBuilder, Trait},
+};
+
+pub(crate) fn derive_unaligned(ctx: &Ctx, _top_level: Trait) -> Result<TokenStream, Error> {
+ match &ctx.ast.data {
+ Data::Struct(strct) => derive_unaligned_struct(ctx, strct),
+ Data::Enum(enm) => derive_unaligned_enum(ctx, enm),
+ Data::Union(unn) => derive_unaligned_union(ctx, unn),
+ }
+}
+
+/// A struct is `Unaligned` if:
+/// - `repr(align)` is no more than 1 and either
+/// - `repr(C)` or `repr(transparent)` and
+/// - all fields `Unaligned`
+/// - `repr(packed)`
+fn derive_unaligned_struct(ctx: &Ctx, strct: &DataStruct) -> Result<TokenStream, Error> {
+ let repr = StructUnionRepr::from_attrs(&ctx.ast.attrs)?;
+ repr.unaligned_validate_no_align_gt_1()?;
+
+ let field_bounds = if repr.is_packed_1() {
+ FieldBounds::None
+ } else if repr.is_c() || repr.is_transparent() {
+ FieldBounds::ALL_SELF
+ } else {
+ return ctx.error_or_skip(Error::new(
+ Span::call_site(),
+ "must have #[repr(C)], #[repr(transparent)], or #[repr(packed)] attribute in order to guarantee this type's alignment",
+ ));
+ };
+
+ Ok(ImplBlockBuilder::new(ctx, strct, Trait::Unaligned, field_bounds).build())
+}
+
+/// An enum is `Unaligned` if:
+/// - No `repr(align(N > 1))`
+/// - `repr(u8)` or `repr(i8)`
+fn derive_unaligned_enum(ctx: &Ctx, enm: &DataEnum) -> Result<TokenStream, Error> {
+ let repr = EnumRepr::from_attrs(&ctx.ast.attrs)?;
+ repr.unaligned_validate_no_align_gt_1()?;
+
+ if !repr.is_u8() && !repr.is_i8() {
+ return ctx.error_or_skip(Error::new(
+ Span::call_site(),
+ "must have #[repr(u8)] or #[repr(i8)] attribute in order to guarantee this type's alignment",
+ ));
+ }
+
+ Ok(ImplBlockBuilder::new(ctx, enm, Trait::Unaligned, FieldBounds::ALL_SELF).build())
+}
+
+/// Like structs, a union is `Unaligned` if:
+/// - `repr(align)` is no more than 1 and either
+/// - `repr(C)` or `repr(transparent)` and
+/// - all fields `Unaligned`
+/// - `repr(packed)`
+fn derive_unaligned_union(ctx: &Ctx, unn: &DataUnion) -> Result<TokenStream, Error> {
+ let repr = StructUnionRepr::from_attrs(&ctx.ast.attrs)?;
+ repr.unaligned_validate_no_align_gt_1()?;
+
+ let field_type_trait_bounds = if repr.is_packed_1() {
+ FieldBounds::None
+ } else if repr.is_c() || repr.is_transparent() {
+ FieldBounds::ALL_SELF
+ } else {
+ return ctx.error_or_skip(Error::new(
+ Span::call_site(),
+ "must have #[repr(C)], #[repr(transparent)], or #[repr(packed)] attribute in order to guarantee this type's alignment",
+ ));
+ };
+
+ Ok(ImplBlockBuilder::new(ctx, unn, Trait::Unaligned, field_type_trait_bounds).build())
+}
diff --git a/rust/zerocopy-derive/lib.rs b/rust/zerocopy-derive/lib.rs
new file mode 100644
index 0000000..c517ea7
--- /dev/null
+++ b/rust/zerocopy-derive/lib.rs
@@ -0,0 +1,146 @@
+// SPDX-License-Identifier: (BSD-2-Clause OR Apache-2.0) OR MIT
+
+// Copyright 2019 The Fuchsia Authors
+//
+// Licensed under a BSD-style license <LICENSE-BSD>, Apache License, Version 2.0
+// <LICENSE-APACHE or https://www.apache.org/licenses/LICENSE-2.0>, or the MIT
+// license <LICENSE-MIT or https://opensource.org/licenses/MIT>, at your option.
+// This file may not be copied, modified, or distributed except according to
+// those terms.
+
+//! Derive macros for [zerocopy]'s traits.
+//!
+//! [zerocopy]: https://docs.rs/zerocopy
+
+// Sometimes we want to use lints which were added after our MSRV.
+// `unknown_lints` is `warn` by default and we deny warnings in CI, so without
+// this attribute, any unknown lint would cause a CI failure when testing with
+// our MSRV.
+#![allow(unknown_lints)]
+#![deny(renamed_and_removed_lints)]
+#![deny(
+ clippy::all,
+ clippy::missing_safety_doc,
+ clippy::multiple_unsafe_ops_per_block,
+ clippy::undocumented_unsafe_blocks
+)]
+// We defer to own discretion on type complexity.
+#![allow(clippy::type_complexity)]
+// Inlining format args isn't supported on our MSRV.
+#![allow(clippy::uninlined_format_args)]
+#![deny(
+ rustdoc::bare_urls,
+ rustdoc::broken_intra_doc_links,
+ rustdoc::invalid_codeblock_attributes,
+ rustdoc::invalid_html_tags,
+ rustdoc::invalid_rust_codeblocks,
+ rustdoc::missing_crate_level_docs,
+ rustdoc::private_intra_doc_links
+)]
+#![recursion_limit = "128"]
+
+macro_rules! ident {
+ (($fmt:literal $(, $arg:expr)*), $span:expr) => {
+ syn::Ident::new(&format!($fmt $(, crate::util::to_ident_str($arg))*), $span)
+ };
+}
+
+mod derive;
+#[cfg(test)]
+mod output_tests;
+mod repr;
+mod util;
+
+use syn::{DeriveInput, Error};
+
+use crate::util::*;
+
+// FIXME(https://github.com/rust-lang/rust/issues/54140): Some errors could be
+// made better if we could add multiple lines of error output like this:
+//
+// error: unsupported representation
+// --> enum.rs:28:8
+// |
+// 28 | #[repr(transparent)]
+// |
+// help: required by the derive of FromBytes
+//
+// Instead, we have more verbose error messages like "unsupported representation
+// for deriving FromZeros, FromBytes, IntoBytes, or Unaligned on an enum"
+//
+// This will probably require Span::error
+// (https://doc.rust-lang.org/nightly/proc_macro/struct.Span.html#method.error),
+// which is currently unstable. Revisit this once it's stable.
+
+/// Defines a derive function named `$outer` which parses its input
+/// `TokenStream` as a `DeriveInput` and then invokes the `$inner` function.
+///
+/// Note that the separate `$outer` parameter is required - proc macro functions
+/// are currently required to live at the crate root, and so the caller must
+/// specify the name in order to avoid name collisions.
+macro_rules! derive {
+ ($trait:ident => $outer:ident => $inner:path) => {
+ #[proc_macro_derive($trait, attributes(zerocopy))]
+ pub fn $outer(ts: proc_macro::TokenStream) -> proc_macro::TokenStream {
+ let ast = syn::parse_macro_input!(ts as DeriveInput);
+ let ctx = match Ctx::try_from_derive_input(ast) {
+ Ok(ctx) => ctx,
+ Err(e) => return e.into_compile_error().into(),
+ };
+ let ts = $inner(&ctx, Trait::$trait).into_ts();
+ // We wrap in `const_block` as a backstop in case any derive fails
+ // to wrap its output in `const_block` (and thus fails to annotate)
+ // with the full set of `#[allow(...)]` attributes).
+ let ts = const_block([Some(ts)]);
+ #[cfg(test)]
+ crate::util::testutil::check_hygiene(ts.clone());
+ ts.into()
+ }
+ };
+}
+
+trait IntoTokenStream {
+ fn into_ts(self) -> proc_macro2::TokenStream;
+}
+
+impl IntoTokenStream for proc_macro2::TokenStream {
+ fn into_ts(self) -> proc_macro2::TokenStream {
+ self
+ }
+}
+
+impl IntoTokenStream for Result<proc_macro2::TokenStream, Error> {
+ fn into_ts(self) -> proc_macro2::TokenStream {
+ match self {
+ Ok(ts) => ts,
+ Err(err) => err.to_compile_error(),
+ }
+ }
+}
+
+derive!(KnownLayout => derive_known_layout => crate::derive::known_layout::derive);
+derive!(Immutable => derive_immutable => crate::derive::derive_immutable);
+derive!(TryFromBytes => derive_try_from_bytes => crate::derive::try_from_bytes::derive_try_from_bytes);
+derive!(FromZeros => derive_from_zeros => crate::derive::from_bytes::derive_from_zeros);
+derive!(FromBytes => derive_from_bytes => crate::derive::from_bytes::derive_from_bytes);
+derive!(IntoBytes => derive_into_bytes => crate::derive::into_bytes::derive_into_bytes);
+derive!(Unaligned => derive_unaligned => crate::derive::unaligned::derive_unaligned);
+derive!(ByteHash => derive_hash => crate::derive::derive_hash);
+derive!(ByteEq => derive_eq => crate::derive::derive_eq);
+derive!(SplitAt => derive_split_at => crate::derive::derive_split_at);
+
+/// Deprecated: prefer [`FromZeros`] instead.
+#[deprecated(since = "0.8.0", note = "`FromZeroes` was renamed to `FromZeros`")]
+#[doc(hidden)]
+#[proc_macro_derive(FromZeroes)]
+pub fn derive_from_zeroes(ts: proc_macro::TokenStream) -> proc_macro::TokenStream {
+ derive_from_zeros(ts)
+}
+
+/// Deprecated: prefer [`IntoBytes`] instead.
+#[deprecated(since = "0.8.0", note = "`AsBytes` was renamed to `IntoBytes`")]
+#[doc(hidden)]
+#[proc_macro_derive(AsBytes)]
+pub fn derive_as_bytes(ts: proc_macro::TokenStream) -> proc_macro::TokenStream {
+ derive_into_bytes(ts)
+}
diff --git a/rust/zerocopy-derive/repr.rs b/rust/zerocopy-derive/repr.rs
new file mode 100644
index 0000000..74fd376
--- /dev/null
+++ b/rust/zerocopy-derive/repr.rs
@@ -0,0 +1,851 @@
+// SPDX-License-Identifier: (BSD-2-Clause OR Apache-2.0) OR MIT
+
+// Copyright 2019 The Fuchsia Authors
+//
+// Licensed under a BSD-style license <LICENSE-BSD>, Apache License, Version 2.0
+// <LICENSE-APACHE or https://www.apache.org/licenses/LICENSE-2.0>, or the MIT
+// license <LICENSE-MIT or https://opensource.org/licenses/MIT>, at your option.
+// This file may not be copied, modified, or distributed except according to
+// those terms.
+
+use core::{
+ convert::{Infallible, TryFrom},
+ num::NonZeroU32,
+};
+
+use proc_macro2::{Span, TokenStream};
+use quote::{quote_spanned, ToTokens, TokenStreamExt as _};
+use syn::{
+ punctuated::Punctuated, spanned::Spanned as _, token::Comma, Attribute, Error, LitInt, Meta,
+ MetaList,
+};
+
+/// The computed representation of a type.
+///
+/// This is the result of processing all `#[repr(...)]` attributes on a type, if
+/// any. A `Repr` is only capable of representing legal combinations of
+/// `#[repr(...)]` attributes.
+#[cfg_attr(test, derive(Copy, Clone, Debug))]
+pub(crate) enum Repr<Prim, Packed> {
+ /// `#[repr(transparent)]`
+ Transparent(Span),
+ /// A compound representation: `repr(C)`, `repr(Rust)`, or `repr(Int)`
+ /// optionally combined with `repr(packed(...))` or `repr(align(...))`
+ Compound(Spanned<CompoundRepr<Prim>>, Option<Spanned<AlignRepr<Packed>>>),
+}
+
+/// A compound representation: `repr(C)`, `repr(Rust)`, or `repr(Int)`.
+#[cfg_attr(test, derive(Copy, Clone, Debug, Eq, PartialEq))]
+pub(crate) enum CompoundRepr<Prim> {
+ C,
+ Rust,
+ Primitive(Prim),
+}
+
+/// `repr(Int)`
+#[derive(Copy, Clone)]
+#[cfg_attr(test, derive(Debug, Eq, PartialEq))]
+pub(crate) enum PrimitiveRepr {
+ U8,
+ U16,
+ U32,
+ U64,
+ U128,
+ Usize,
+ I8,
+ I16,
+ I32,
+ I64,
+ I128,
+ Isize,
+}
+
+/// `repr(packed(...))` or `repr(align(...))`
+#[cfg_attr(test, derive(Copy, Clone, Debug, Eq, PartialEq))]
+pub(crate) enum AlignRepr<Packed> {
+ Packed(Packed),
+ Align(NonZeroU32),
+}
+
+/// The representations which can legally appear on a struct or union type.
+pub(crate) type StructUnionRepr = Repr<Infallible, NonZeroU32>;
+
+/// The representations which can legally appear on an enum type.
+pub(crate) type EnumRepr = Repr<PrimitiveRepr, Infallible>;
+
+impl<Prim, Packed> Repr<Prim, Packed> {
+ /// Gets the name of this "repr type" - the non-align `repr(X)` that is used
+ /// in prose to refer to this type.
+ ///
+ /// For example, we would refer to `#[repr(C, align(4))] struct Foo { ... }`
+ /// as a "`repr(C)` struct".
+ pub(crate) fn repr_type_name(&self) -> &str
+ where
+ Prim: Copy + With<PrimitiveRepr>,
+ {
+ use CompoundRepr::*;
+ use PrimitiveRepr::*;
+ use Repr::*;
+ match self {
+ Transparent(_span) => "repr(transparent)",
+ Compound(Spanned { t: repr, span: _ }, _align) => match repr {
+ C => "repr(C)",
+ Rust => "repr(Rust)",
+ Primitive(prim) => prim.with(|prim| match prim {
+ U8 => "repr(u8)",
+ U16 => "repr(u16)",
+ U32 => "repr(u32)",
+ U64 => "repr(u64)",
+ U128 => "repr(u128)",
+ Usize => "repr(usize)",
+ I8 => "repr(i8)",
+ I16 => "repr(i16)",
+ I32 => "repr(i32)",
+ I64 => "repr(i64)",
+ I128 => "repr(i128)",
+ Isize => "repr(isize)",
+ }),
+ },
+ }
+ }
+
+ pub(crate) fn is_transparent(&self) -> bool {
+ matches!(self, Repr::Transparent(_))
+ }
+
+ pub(crate) fn is_c(&self) -> bool {
+ use CompoundRepr::*;
+ matches!(self, Repr::Compound(Spanned { t: C, span: _ }, _align))
+ }
+
+ pub(crate) fn is_primitive(&self) -> bool {
+ use CompoundRepr::*;
+ matches!(self, Repr::Compound(Spanned { t: Primitive(_), span: _ }, _align))
+ }
+
+ pub(crate) fn get_packed(&self) -> Option<&Packed> {
+ use AlignRepr::*;
+ use Repr::*;
+ if let Compound(_, Some(Spanned { t: Packed(p), span: _ })) = self {
+ Some(p)
+ } else {
+ None
+ }
+ }
+
+ pub(crate) fn get_align(&self) -> Option<Spanned<NonZeroU32>> {
+ use AlignRepr::*;
+ use Repr::*;
+ if let Compound(_, Some(Spanned { t: Align(n), span })) = self {
+ Some(Spanned::new(*n, *span))
+ } else {
+ None
+ }
+ }
+
+ pub(crate) fn is_align_gt_1(&self) -> bool {
+ self.get_align().map(|n| n.t.get() > 1).unwrap_or(false)
+ }
+
+ /// When deriving `Unaligned`, validate that the decorated type has no
+ /// `#[repr(align(N))]` attribute where `N > 1`. If no such attribute exists
+ /// (including if `N == 1`), this returns `Ok(())`, and otherwise it returns
+ /// a descriptive error.
+ pub(crate) fn unaligned_validate_no_align_gt_1(&self) -> Result<(), Error> {
+ if let Some(n) = self.get_align().filter(|n| n.t.get() > 1) {
+ Err(Error::new(
+ n.span,
+ "cannot derive `Unaligned` on type with alignment greater than 1",
+ ))
+ } else {
+ Ok(())
+ }
+ }
+}
+
+impl<Prim> Repr<Prim, NonZeroU32> {
+ /// Does `self` describe a `#[repr(packed)]` or `#[repr(packed(1))]` type?
+ pub(crate) fn is_packed_1(&self) -> bool {
+ self.get_packed().map(|n| n.get() == 1).unwrap_or(false)
+ }
+}
+
+impl<Packed> Repr<PrimitiveRepr, Packed> {
+ fn get_primitive(&self) -> Option<&PrimitiveRepr> {
+ use CompoundRepr::*;
+ use Repr::*;
+ if let Compound(Spanned { t: Primitive(p), span: _ }, _align) = self {
+ Some(p)
+ } else {
+ None
+ }
+ }
+
+ /// Does `self` describe a `#[repr(u8)]` type?
+ pub(crate) fn is_u8(&self) -> bool {
+ matches!(self.get_primitive(), Some(PrimitiveRepr::U8))
+ }
+
+ /// Does `self` describe a `#[repr(i8)]` type?
+ pub(crate) fn is_i8(&self) -> bool {
+ matches!(self.get_primitive(), Some(PrimitiveRepr::I8))
+ }
+}
+
+impl<Prim, Packed> ToTokens for Repr<Prim, Packed>
+where
+ Prim: With<PrimitiveRepr> + Copy,
+ Packed: With<NonZeroU32> + Copy,
+{
+ fn to_tokens(&self, ts: &mut TokenStream) {
+ use Repr::*;
+ match self {
+ Transparent(span) => ts.append_all(quote_spanned! { *span=> #[repr(transparent)] }),
+ Compound(repr, align) => {
+ repr.to_tokens(ts);
+ if let Some(align) = align {
+ align.to_tokens(ts);
+ }
+ }
+ }
+ }
+}
+
+impl<Prim: With<PrimitiveRepr> + Copy> ToTokens for Spanned<CompoundRepr<Prim>> {
+ fn to_tokens(&self, ts: &mut TokenStream) {
+ use CompoundRepr::*;
+ match &self.t {
+ C => ts.append_all(quote_spanned! { self.span=> #[repr(C)] }),
+ Rust => ts.append_all(quote_spanned! { self.span=> #[repr(Rust)] }),
+ Primitive(prim) => prim.with(|prim| Spanned::new(prim, self.span).to_tokens(ts)),
+ }
+ }
+}
+
+impl ToTokens for Spanned<PrimitiveRepr> {
+ fn to_tokens(&self, ts: &mut TokenStream) {
+ use PrimitiveRepr::*;
+ match self.t {
+ U8 => ts.append_all(quote_spanned! { self.span => #[repr(u8)] }),
+ U16 => ts.append_all(quote_spanned! { self.span => #[repr(u16)] }),
+ U32 => ts.append_all(quote_spanned! { self.span => #[repr(u32)] }),
+ U64 => ts.append_all(quote_spanned! { self.span => #[repr(u64)] }),
+ U128 => ts.append_all(quote_spanned! { self.span => #[repr(u128)] }),
+ Usize => ts.append_all(quote_spanned! { self.span => #[repr(usize)] }),
+ I8 => ts.append_all(quote_spanned! { self.span => #[repr(i8)] }),
+ I16 => ts.append_all(quote_spanned! { self.span => #[repr(i16)] }),
+ I32 => ts.append_all(quote_spanned! { self.span => #[repr(i32)] }),
+ I64 => ts.append_all(quote_spanned! { self.span => #[repr(i64)] }),
+ I128 => ts.append_all(quote_spanned! { self.span => #[repr(i128)] }),
+ Isize => ts.append_all(quote_spanned! { self.span => #[repr(isize)] }),
+ }
+ }
+}
+
+impl<Packed: With<NonZeroU32> + Copy> ToTokens for Spanned<AlignRepr<Packed>> {
+ fn to_tokens(&self, ts: &mut TokenStream) {
+ use AlignRepr::*;
+ // We use `syn::Index` instead of `u32` because `quote_spanned!`
+ // serializes `u32` literals as `123u32`, not just `123`. Rust doesn't
+ // recognize that as a valid argument to `#[repr(align(...))]` or
+ // `#[repr(packed(...))]`.
+ let to_index = |n: NonZeroU32| syn::Index { index: n.get(), span: self.span };
+ match self.t {
+ Packed(n) => n.with(|n| {
+ let n = to_index(n);
+ ts.append_all(quote_spanned! { self.span => #[repr(packed(#n))] })
+ }),
+ Align(n) => {
+ let n = to_index(n);
+ ts.append_all(quote_spanned! { self.span => #[repr(align(#n))] })
+ }
+ }
+ }
+}
+
+/// The result of parsing a single `#[repr(...)]` attribute or a single
+/// directive inside a compound `#[repr(..., ...)]` attribute.
+#[derive(Copy, Clone, PartialEq, Eq)]
+#[cfg_attr(test, derive(Debug))]
+pub(crate) enum RawRepr {
+ Transparent,
+ C,
+ Rust,
+ U8,
+ U16,
+ U32,
+ U64,
+ U128,
+ Usize,
+ I8,
+ I16,
+ I32,
+ I64,
+ I128,
+ Isize,
+ Align(NonZeroU32),
+ PackedN(NonZeroU32),
+ Packed,
+}
+
+/// The error from converting from a `RawRepr`.
+#[cfg_attr(test, derive(Debug, Eq, PartialEq))]
+pub(crate) enum FromRawReprError<E> {
+ /// The `RawRepr` doesn't affect the high-level repr we're parsing (e.g.
+ /// it's `align(...)` and we're parsing a `CompoundRepr`).
+ None,
+ /// The `RawRepr` is invalid for the high-level repr we're parsing (e.g.
+ /// it's `packed` repr and we're parsing an `AlignRepr` for an enum type).
+ Err(E),
+}
+
+/// The representation hint is not supported for the decorated type.
+#[cfg_attr(test, derive(Copy, Clone, Debug, Eq, PartialEq))]
+pub(crate) struct UnsupportedReprError;
+
+impl<Prim: With<PrimitiveRepr>> TryFrom<RawRepr> for CompoundRepr<Prim> {
+ type Error = FromRawReprError<UnsupportedReprError>;
+ fn try_from(
+ raw: RawRepr,
+ ) -> Result<CompoundRepr<Prim>, FromRawReprError<UnsupportedReprError>> {
+ use RawRepr::*;
+ match raw {
+ C => Ok(CompoundRepr::C),
+ Rust => Ok(CompoundRepr::Rust),
+ raw @ (U8 | U16 | U32 | U64 | U128 | Usize | I8 | I16 | I32 | I64 | I128 | Isize) => {
+ Prim::try_with_or(
+ || match raw {
+ U8 => Ok(PrimitiveRepr::U8),
+ U16 => Ok(PrimitiveRepr::U16),
+ U32 => Ok(PrimitiveRepr::U32),
+ U64 => Ok(PrimitiveRepr::U64),
+ U128 => Ok(PrimitiveRepr::U128),
+ Usize => Ok(PrimitiveRepr::Usize),
+ I8 => Ok(PrimitiveRepr::I8),
+ I16 => Ok(PrimitiveRepr::I16),
+ I32 => Ok(PrimitiveRepr::I32),
+ I64 => Ok(PrimitiveRepr::I64),
+ I128 => Ok(PrimitiveRepr::I128),
+ Isize => Ok(PrimitiveRepr::Isize),
+ Transparent | C | Rust | Align(_) | PackedN(_) | Packed => {
+ Err(UnsupportedReprError)
+ }
+ },
+ UnsupportedReprError,
+ )
+ .map(CompoundRepr::Primitive)
+ .map_err(FromRawReprError::Err)
+ }
+ Transparent | Align(_) | PackedN(_) | Packed => Err(FromRawReprError::None),
+ }
+ }
+}
+
+impl<Pcked: With<NonZeroU32>> TryFrom<RawRepr> for AlignRepr<Pcked> {
+ type Error = FromRawReprError<UnsupportedReprError>;
+ fn try_from(raw: RawRepr) -> Result<AlignRepr<Pcked>, FromRawReprError<UnsupportedReprError>> {
+ use RawRepr::*;
+ match raw {
+ Packed | PackedN(_) => Pcked::try_with_or(
+ || match raw {
+ Packed => Ok(NonZeroU32::new(1).unwrap()),
+ PackedN(n) => Ok(n),
+ U8 | U16 | U32 | U64 | U128 | Usize | I8 | I16 | I32 | I64 | I128 | Isize
+ | Transparent | C | Rust | Align(_) => Err(UnsupportedReprError),
+ },
+ UnsupportedReprError,
+ )
+ .map(AlignRepr::Packed)
+ .map_err(FromRawReprError::Err),
+ Align(n) => Ok(AlignRepr::Align(n)),
+ U8 | U16 | U32 | U64 | U128 | Usize | I8 | I16 | I32 | I64 | I128 | Isize
+ | Transparent | C | Rust => Err(FromRawReprError::None),
+ }
+ }
+}
+
+/// The error from extracting a high-level repr type from a list of `RawRepr`s.
+#[cfg_attr(test, derive(Copy, Clone, Debug, Eq, PartialEq))]
+enum FromRawReprsError<E> {
+ /// One of the `RawRepr`s is invalid for the high-level repr we're parsing
+ /// (e.g. there's a `packed` repr and we're parsing an `AlignRepr` for an
+ /// enum type).
+ Single(E),
+ /// Two `RawRepr`s appear which both affect the high-level repr we're
+ /// parsing (e.g., the list is `#[repr(align(2), packed)]`). Note that we
+ /// conservatively treat redundant reprs as conflicting (e.g.
+ /// `#[repr(packed, packed)]`).
+ Conflict,
+}
+
+/// Tries to extract a high-level repr from a list of `RawRepr`s.
+fn try_from_raw_reprs<'a, E, R: TryFrom<RawRepr, Error = FromRawReprError<E>>>(
+ r: impl IntoIterator<Item = &'a Spanned<RawRepr>>,
+) -> Result<Option<Spanned<R>>, Spanned<FromRawReprsError<E>>> {
+ // Walk the list of `RawRepr`s and attempt to convert each to an `R`. Bail
+ // if we find any errors. If we find more than one which converts to an `R`,
+ // bail with a `Conflict` error.
+ r.into_iter().try_fold(None, |found: Option<Spanned<R>>, raw| {
+ let new = match Spanned::<R>::try_from(*raw) {
+ Ok(r) => r,
+ // This `RawRepr` doesn't convert to an `R`, so keep the current
+ // found `R`, if any.
+ Err(FromRawReprError::None) => return Ok(found),
+ // This repr is unsupported for the decorated type (e.g.
+ // `repr(packed)` on an enum).
+ Err(FromRawReprError::Err(Spanned { t: err, span })) => {
+ return Err(Spanned::new(FromRawReprsError::Single(err), span))
+ }
+ };
+
+ if let Some(found) = found {
+ // We already found an `R`, but this `RawRepr` also converts to an
+ // `R`, so that's a conflict.
+ //
+ // `Span::join` returns `None` if the two spans are from different
+ // files or if we're not on the nightly compiler. In that case, just
+ // use `new`'s span.
+ let span = found.span.join(new.span).unwrap_or(new.span);
+ Err(Spanned::new(FromRawReprsError::Conflict, span))
+ } else {
+ Ok(Some(new))
+ }
+ })
+}
+
+/// The error returned from [`Repr::from_attrs`].
+#[cfg_attr(test, derive(Copy, Clone, Debug, Eq, PartialEq))]
+enum FromAttrsError {
+ FromRawReprs(FromRawReprsError<UnsupportedReprError>),
+ Unrecognized,
+}
+
+impl From<FromRawReprsError<UnsupportedReprError>> for FromAttrsError {
+ fn from(err: FromRawReprsError<UnsupportedReprError>) -> FromAttrsError {
+ FromAttrsError::FromRawReprs(err)
+ }
+}
+
+impl From<UnrecognizedReprError> for FromAttrsError {
+ fn from(_err: UnrecognizedReprError) -> FromAttrsError {
+ FromAttrsError::Unrecognized
+ }
+}
+
+impl From<Spanned<FromAttrsError>> for Error {
+ fn from(err: Spanned<FromAttrsError>) -> Error {
+ let Spanned { t: err, span } = err;
+ match err {
+ FromAttrsError::FromRawReprs(FromRawReprsError::Single(
+ _err @ UnsupportedReprError,
+ )) => Error::new(span, "unsupported representation hint for the decorated type"),
+ FromAttrsError::FromRawReprs(FromRawReprsError::Conflict) => {
+ // NOTE: This says "another" rather than "a preceding" because
+ // when one of the reprs involved is `transparent`, we detect
+ // that condition in `Repr::from_attrs`, and at that point we
+ // can't tell which repr came first, so we might report this on
+ // the first involved repr rather than the second, third, etc.
+ Error::new(span, "this conflicts with another representation hint")
+ }
+ FromAttrsError::Unrecognized => Error::new(span, "unrecognized representation hint"),
+ }
+ }
+}
+
+impl<Prim, Packed> Repr<Prim, Packed> {
+ fn from_attrs_inner(attrs: &[Attribute]) -> Result<Repr<Prim, Packed>, Spanned<FromAttrsError>>
+ where
+ Prim: With<PrimitiveRepr>,
+ Packed: With<NonZeroU32>,
+ {
+ let raw_reprs = RawRepr::from_attrs(attrs).map_err(Spanned::from)?;
+
+ let transparent = {
+ let mut transparents = raw_reprs.iter().filter_map(|Spanned { t, span }| match t {
+ RawRepr::Transparent => Some(span),
+ _ => None,
+ });
+ let first = transparents.next();
+ let second = transparents.next();
+ match (first, second) {
+ (None, None) => None,
+ (Some(span), None) => Some(*span),
+ (Some(_), Some(second)) => {
+ return Err(Spanned::new(
+ FromAttrsError::FromRawReprs(FromRawReprsError::Conflict),
+ *second,
+ ))
+ }
+ // An iterator can't produce a value only on the second call to
+ // `.next()`.
+ (None, Some(_)) => unreachable!(),
+ }
+ };
+
+ let compound: Option<Spanned<CompoundRepr<Prim>>> =
+ try_from_raw_reprs(raw_reprs.iter()).map_err(Spanned::from)?;
+ let align: Option<Spanned<AlignRepr<Packed>>> =
+ try_from_raw_reprs(raw_reprs.iter()).map_err(Spanned::from)?;
+
+ if let Some(span) = transparent {
+ if compound.is_some() || align.is_some() {
+ // Arbitrarily report the problem on the `transparent` span. Any
+ // span will do.
+ return Err(Spanned::new(FromRawReprsError::Conflict.into(), span));
+ }
+
+ Ok(Repr::Transparent(span))
+ } else {
+ Ok(Repr::Compound(
+ compound.unwrap_or(Spanned::new(CompoundRepr::Rust, Span::call_site())),
+ align,
+ ))
+ }
+ }
+}
+
+impl<Prim, Packed> Repr<Prim, Packed> {
+ pub(crate) fn from_attrs(attrs: &[Attribute]) -> Result<Repr<Prim, Packed>, Error>
+ where
+ Prim: With<PrimitiveRepr>,
+ Packed: With<NonZeroU32>,
+ {
+ Repr::from_attrs_inner(attrs).map_err(Into::into)
+ }
+}
+
+/// The representation hint could not be parsed or was unrecognized.
+struct UnrecognizedReprError;
+
+impl RawRepr {
+ fn from_attrs(
+ attrs: &[Attribute],
+ ) -> Result<Vec<Spanned<RawRepr>>, Spanned<UnrecognizedReprError>> {
+ let mut reprs = Vec::new();
+ for attr in attrs {
+ // Ignore documentation attributes.
+ if attr.path().is_ident("doc") {
+ continue;
+ }
+ if let Meta::List(ref meta_list) = attr.meta {
+ if meta_list.path.is_ident("repr") {
+ let parsed: Punctuated<Meta, Comma> =
+ match meta_list.parse_args_with(Punctuated::parse_terminated) {
+ Ok(parsed) => parsed,
+ Err(_) => {
+ return Err(Spanned::new(
+ UnrecognizedReprError,
+ meta_list.tokens.span(),
+ ))
+ }
+ };
+ for meta in parsed {
+ let s = meta.span();
+ reprs.push(
+ RawRepr::from_meta(&meta)
+ .map(|r| Spanned::new(r, s))
+ .map_err(|e| Spanned::new(e, s))?,
+ );
+ }
+ }
+ }
+ }
+
+ Ok(reprs)
+ }
+
+ fn from_meta(meta: &Meta) -> Result<RawRepr, UnrecognizedReprError> {
+ let (path, list) = match meta {
+ Meta::Path(path) => (path, None),
+ Meta::List(list) => (&list.path, Some(list)),
+ _ => return Err(UnrecognizedReprError),
+ };
+
+ let ident = path.get_ident().ok_or(UnrecognizedReprError)?;
+
+ // Only returns `Ok` for non-zero power-of-two values.
+ let parse_nzu64 = |list: &MetaList| {
+ list.parse_args::<LitInt>()
+ .and_then(|int| int.base10_parse::<NonZeroU32>())
+ .map_err(|_| UnrecognizedReprError)
+ .and_then(|nz| {
+ if nz.get().is_power_of_two() {
+ Ok(nz)
+ } else {
+ Err(UnrecognizedReprError)
+ }
+ })
+ };
+
+ use RawRepr::*;
+ Ok(match (ident.to_string().as_str(), list) {
+ ("u8", None) => U8,
+ ("u16", None) => U16,
+ ("u32", None) => U32,
+ ("u64", None) => U64,
+ ("u128", None) => U128,
+ ("usize", None) => Usize,
+ ("i8", None) => I8,
+ ("i16", None) => I16,
+ ("i32", None) => I32,
+ ("i64", None) => I64,
+ ("i128", None) => I128,
+ ("isize", None) => Isize,
+ ("C", None) => C,
+ ("transparent", None) => Transparent,
+ ("Rust", None) => Rust,
+ ("packed", None) => Packed,
+ ("packed", Some(list)) => PackedN(parse_nzu64(list)?),
+ ("align", Some(list)) => Align(parse_nzu64(list)?),
+ _ => return Err(UnrecognizedReprError),
+ })
+ }
+}
+
+pub(crate) use util::*;
+mod util {
+ use super::*;
+ /// A value with an associated span.
+ #[derive(Copy, Clone)]
+ #[cfg_attr(test, derive(Debug))]
+ pub(crate) struct Spanned<T> {
+ pub(crate) t: T,
+ pub(crate) span: Span,
+ }
+
+ impl<T> Spanned<T> {
+ pub(super) fn new(t: T, span: Span) -> Spanned<T> {
+ Spanned { t, span }
+ }
+
+ pub(super) fn from<U>(s: Spanned<U>) -> Spanned<T>
+ where
+ T: From<U>,
+ {
+ let Spanned { t: u, span } = s;
+ Spanned::new(u.into(), span)
+ }
+
+ /// Delegates to `T: TryFrom`, preserving span information in both the
+ /// success and error cases.
+ pub(super) fn try_from<E, U>(
+ u: Spanned<U>,
+ ) -> Result<Spanned<T>, FromRawReprError<Spanned<E>>>
+ where
+ T: TryFrom<U, Error = FromRawReprError<E>>,
+ {
+ let Spanned { t: u, span } = u;
+ T::try_from(u).map(|t| Spanned { t, span }).map_err(|err| match err {
+ FromRawReprError::None => FromRawReprError::None,
+ FromRawReprError::Err(e) => FromRawReprError::Err(Spanned::new(e, span)),
+ })
+ }
+ }
+
+ // Used to permit implementing `With<T> for T: Inhabited` and for
+ // `Infallible` without a blanket impl conflict.
+ pub(crate) trait Inhabited {}
+ impl Inhabited for PrimitiveRepr {}
+ impl Inhabited for NonZeroU32 {}
+
+ pub(crate) trait With<T> {
+ fn with<O, F: FnOnce(T) -> O>(self, f: F) -> O;
+ fn try_with_or<E, F: FnOnce() -> Result<T, E>>(f: F, err: E) -> Result<Self, E>
+ where
+ Self: Sized;
+ }
+
+ impl<T: Inhabited> With<T> for T {
+ fn with<O, F: FnOnce(T) -> O>(self, f: F) -> O {
+ f(self)
+ }
+
+ fn try_with_or<E, F: FnOnce() -> Result<T, E>>(f: F, _err: E) -> Result<Self, E> {
+ f()
+ }
+ }
+
+ impl<T> With<T> for Infallible {
+ fn with<O, F: FnOnce(T) -> O>(self, _f: F) -> O {
+ match self {}
+ }
+
+ fn try_with_or<E, F: FnOnce() -> Result<T, E>>(_f: F, err: E) -> Result<Self, E> {
+ Err(err)
+ }
+ }
+}
+
+#[cfg(test)]
+mod tests {
+ use syn::parse_quote;
+
+ use super::*;
+
+ impl<T> From<T> for Spanned<T> {
+ fn from(t: T) -> Spanned<T> {
+ Spanned::new(t, Span::call_site())
+ }
+ }
+
+ // We ignore spans for equality in testing since real spans are hard to
+ // synthesize and don't implement `PartialEq`.
+ impl<T: PartialEq> PartialEq for Spanned<T> {
+ fn eq(&self, other: &Spanned<T>) -> bool {
+ self.t.eq(&other.t)
+ }
+ }
+
+ impl<T: Eq> Eq for Spanned<T> {}
+
+ impl<Prim: PartialEq, Packed: PartialEq> PartialEq for Repr<Prim, Packed> {
+ fn eq(&self, other: &Repr<Prim, Packed>) -> bool {
+ match (self, other) {
+ (Repr::Transparent(_), Repr::Transparent(_)) => true,
+ (Repr::Compound(sc, sa), Repr::Compound(oc, oa)) => (sc, sa) == (oc, oa),
+ _ => false,
+ }
+ }
+ }
+
+ fn s() -> Span {
+ Span::call_site()
+ }
+
+ #[test]
+ fn test() {
+ // Test that a given `#[repr(...)]` attribute parses and returns the
+ // given `Repr` or error.
+ macro_rules! test {
+ ($(#[$attr:meta])* => $repr:expr) => {
+ test!(@inner $(#[$attr])* => Repr => Ok($repr));
+ };
+ // In the error case, the caller must explicitly provide the name of
+ // the `Repr` type to assist in type inference.
+ (@error $(#[$attr:meta])* => $typ:ident => $repr:expr) => {
+ test!(@inner $(#[$attr])* => $typ => Err($repr));
+ };
+ (@inner $(#[$attr:meta])* => $typ:ident => $repr:expr) => {
+ let attr: Attribute = parse_quote!($(#[$attr])*);
+ let mut got = $typ::from_attrs_inner(&[attr]);
+ let expect: Result<Repr<_, _>, _> = $repr;
+ if false {
+ // Force Rust to infer `got` as having the same type as
+ // `expect`.
+ got = expect;
+ }
+ assert_eq!(got, expect, stringify!($(#[$attr])*));
+ };
+ }
+
+ use AlignRepr::*;
+ use CompoundRepr::*;
+ use PrimitiveRepr::*;
+ let nz = |n: u32| NonZeroU32::new(n).unwrap();
+
+ test!(#[repr(transparent)] => StructUnionRepr::Transparent(s()));
+ test!(#[repr()] => StructUnionRepr::Compound(Rust.into(), None));
+ test!(#[repr(packed)] => StructUnionRepr::Compound(Rust.into(), Some(Packed(nz(1)).into())));
+ test!(#[repr(packed(2))] => StructUnionRepr::Compound(Rust.into(), Some(Packed(nz(2)).into())));
+ test!(#[repr(align(1))] => StructUnionRepr::Compound(Rust.into(), Some(Align(nz(1)).into())));
+ test!(#[repr(align(2))] => StructUnionRepr::Compound(Rust.into(), Some(Align(nz(2)).into())));
+ test!(#[repr(C)] => StructUnionRepr::Compound(C.into(), None));
+ test!(#[repr(C, packed)] => StructUnionRepr::Compound(C.into(), Some(Packed(nz(1)).into())));
+ test!(#[repr(C, packed(2))] => StructUnionRepr::Compound(C.into(), Some(Packed(nz(2)).into())));
+ test!(#[repr(C, align(1))] => StructUnionRepr::Compound(C.into(), Some(Align(nz(1)).into())));
+ test!(#[repr(C, align(2))] => StructUnionRepr::Compound(C.into(), Some(Align(nz(2)).into())));
+
+ test!(#[repr(transparent)] => EnumRepr::Transparent(s()));
+ test!(#[repr()] => EnumRepr::Compound(Rust.into(), None));
+ test!(#[repr(align(1))] => EnumRepr::Compound(Rust.into(), Some(Align(nz(1)).into())));
+ test!(#[repr(align(2))] => EnumRepr::Compound(Rust.into(), Some(Align(nz(2)).into())));
+
+ macro_rules! for_each_compound_repr {
+ ($($r:tt => $var:expr),*) => {
+ $(
+ test!(#[repr($r)] => EnumRepr::Compound($var.into(), None));
+ test!(#[repr($r, align(1))] => EnumRepr::Compound($var.into(), Some(Align(nz(1)).into())));
+ test!(#[repr($r, align(2))] => EnumRepr::Compound($var.into(), Some(Align(nz(2)).into())));
+ )*
+ }
+ }
+
+ for_each_compound_repr!(
+ C => C,
+ u8 => Primitive(U8),
+ u16 => Primitive(U16),
+ u32 => Primitive(U32),
+ u64 => Primitive(U64),
+ usize => Primitive(Usize),
+ i8 => Primitive(I8),
+ i16 => Primitive(I16),
+ i32 => Primitive(I32),
+ i64 => Primitive(I64),
+ isize => Primitive(Isize)
+ );
+
+ use FromAttrsError::*;
+ use FromRawReprsError::*;
+
+ // Run failure tests which are valid for both `StructUnionRepr` and
+ // `EnumRepr`.
+ macro_rules! for_each_repr_type {
+ ($($repr:ident),*) => {
+ $(
+ // Invalid packed or align attributes
+ test!(@error #[repr(packed(0))] => $repr => Unrecognized.into());
+ test!(@error #[repr(packed(3))] => $repr => Unrecognized.into());
+ test!(@error #[repr(align(0))] => $repr => Unrecognized.into());
+ test!(@error #[repr(align(3))] => $repr => Unrecognized.into());
+
+ // Conflicts
+ test!(@error #[repr(transparent, transparent)] => $repr => FromRawReprs(Conflict).into());
+ test!(@error #[repr(transparent, C)] => $repr => FromRawReprs(Conflict).into());
+ test!(@error #[repr(transparent, Rust)] => $repr => FromRawReprs(Conflict).into());
+
+ test!(@error #[repr(C, transparent)] => $repr => FromRawReprs(Conflict).into());
+ test!(@error #[repr(C, C)] => $repr => FromRawReprs(Conflict).into());
+ test!(@error #[repr(C, Rust)] => $repr => FromRawReprs(Conflict).into());
+
+ test!(@error #[repr(Rust, transparent)] => $repr => FromRawReprs(Conflict).into());
+ test!(@error #[repr(Rust, C)] => $repr => FromRawReprs(Conflict).into());
+ test!(@error #[repr(Rust, Rust)] => $repr => FromRawReprs(Conflict).into());
+ )*
+ }
+ }
+
+ for_each_repr_type!(StructUnionRepr, EnumRepr);
+
+ // Enum-specific conflicts.
+ //
+ // We don't bother to test every combination since that would be a huge
+ // number (enums can have primitive reprs u8, u16, u32, u64, usize, i8,
+ // i16, i32, i64, and isize). Instead, since the conflict logic doesn't
+ // care what specific value of `PrimitiveRepr` is present, we assume
+ // that testing against u8 alone is fine.
+ test!(@error #[repr(transparent, u8)] => EnumRepr => FromRawReprs(Conflict).into());
+ test!(@error #[repr(u8, transparent)] => EnumRepr => FromRawReprs(Conflict).into());
+ test!(@error #[repr(C, u8)] => EnumRepr => FromRawReprs(Conflict).into());
+ test!(@error #[repr(u8, C)] => EnumRepr => FromRawReprs(Conflict).into());
+ test!(@error #[repr(Rust, u8)] => EnumRepr => FromRawReprs(Conflict).into());
+ test!(@error #[repr(u8, Rust)] => EnumRepr => FromRawReprs(Conflict).into());
+ test!(@error #[repr(u8, u8)] => EnumRepr => FromRawReprs(Conflict).into());
+
+ // Illegal struct/union reprs
+ test!(@error #[repr(u8)] => StructUnionRepr => FromRawReprs(Single(UnsupportedReprError)).into());
+ test!(@error #[repr(u16)] => StructUnionRepr => FromRawReprs(Single(UnsupportedReprError)).into());
+ test!(@error #[repr(u32)] => StructUnionRepr => FromRawReprs(Single(UnsupportedReprError)).into());
+ test!(@error #[repr(u64)] => StructUnionRepr => FromRawReprs(Single(UnsupportedReprError)).into());
+ test!(@error #[repr(usize)] => StructUnionRepr => FromRawReprs(Single(UnsupportedReprError)).into());
+ test!(@error #[repr(i8)] => StructUnionRepr => FromRawReprs(Single(UnsupportedReprError)).into());
+ test!(@error #[repr(i16)] => StructUnionRepr => FromRawReprs(Single(UnsupportedReprError)).into());
+ test!(@error #[repr(i32)] => StructUnionRepr => FromRawReprs(Single(UnsupportedReprError)).into());
+ test!(@error #[repr(i64)] => StructUnionRepr => FromRawReprs(Single(UnsupportedReprError)).into());
+ test!(@error #[repr(isize)] => StructUnionRepr => FromRawReprs(Single(UnsupportedReprError)).into());
+
+ // Illegal enum reprs
+ test!(@error #[repr(packed)] => EnumRepr => FromRawReprs(Single(UnsupportedReprError)).into());
+ test!(@error #[repr(packed(1))] => EnumRepr => FromRawReprs(Single(UnsupportedReprError)).into());
+ test!(@error #[repr(packed(2))] => EnumRepr => FromRawReprs(Single(UnsupportedReprError)).into());
+ }
+}
diff --git a/rust/zerocopy-derive/util.rs b/rust/zerocopy-derive/util.rs
new file mode 100644
index 0000000..5ba5228
--- /dev/null
+++ b/rust/zerocopy-derive/util.rs
@@ -0,0 +1,845 @@
+// SPDX-License-Identifier: (BSD-2-Clause OR Apache-2.0) OR MIT
+
+// Copyright 2019 The Fuchsia Authors
+//
+// Licensed under a BSD-style license <LICENSE-BSD>, Apache License, Version 2.0
+// <LICENSE-APACHE or https://www.apache.org/licenses/LICENSE-2.0>, or the MIT
+// license <LICENSE-MIT or https://opensource.org/licenses/MIT>, at your option.
+// This file may not be copied, modified, or distributed except according to
+// those terms.
+
+use std::num::NonZeroU32;
+
+use proc_macro2::{Span, TokenStream};
+use quote::{quote, quote_spanned, ToTokens};
+use syn::{
+ parse_quote, spanned::Spanned as _, Data, DataEnum, DataStruct, DataUnion, DeriveInput, Error,
+ Expr, ExprLit, Field, GenericParam, Ident, Index, Lit, LitStr, Meta, Path, Type, Variant,
+ Visibility, WherePredicate,
+};
+
+use crate::repr::{CompoundRepr, EnumRepr, PrimitiveRepr, Repr, Spanned};
+
+pub(crate) struct Ctx {
+ pub(crate) ast: DeriveInput,
+ pub(crate) zerocopy_crate: Path,
+
+ // The value of the last `#[zerocopy(on_error = ...)]` attribute, or `false`
+ // if none is provided.
+ pub(crate) skip_on_error: bool,
+
+ // The span of the last `#[zerocopy(on_error = ...)]` attribute, if any.
+ pub(crate) on_error_span: Option<proc_macro2::Span>,
+}
+
+impl Ctx {
+ /// Attempt to extract a crate path from the provided attributes. Defaults to
+ /// `::zerocopy` if not found.
+ pub(crate) fn try_from_derive_input(ast: DeriveInput) -> Result<Self, Error> {
+ let mut path = parse_quote!(::zerocopy);
+ let mut skip_on_error = false;
+ let mut on_error_span = None;
+
+ for attr in &ast.attrs {
+ if let Meta::List(ref meta_list) = attr.meta {
+ if meta_list.path.is_ident("zerocopy") {
+ attr.parse_nested_meta(|meta| {
+ if meta.path.is_ident("crate") {
+ let expr = meta.value().and_then(|value| value.parse());
+ if let Ok(Expr::Lit(ExprLit { lit: Lit::Str(lit), .. })) = expr {
+ if let Ok(path_lit) = lit.parse::<Ident>() {
+ path = parse_quote!(::#path_lit);
+ return Ok(());
+ }
+ }
+
+ return Err(Error::new(
+ Span::call_site(),
+ "`crate` attribute requires a path as the value",
+ ));
+ }
+
+ if meta.path.is_ident("on_error") {
+ on_error_span = Some(meta.path.span());
+ let value = meta.value()?;
+ let s: LitStr = value.parse()?;
+ match s.value().as_str() {
+ "skip" => skip_on_error = true,
+ "fail" => skip_on_error = false,
+ _ => return Err(Error::new(
+ s.span(),
+ "unrecognized value for `on_error` attribute from `zerocopy`; expected `skip` or `fail`",
+ )),
+ }
+ return Ok(());
+ }
+
+ Err(Error::new(
+ Span::call_site(),
+ format!(
+ "unknown attribute encountered: {}",
+ meta.path.into_token_stream()
+ ),
+ ))
+ })?;
+ }
+ }
+ }
+
+ Ok(Self { ast, zerocopy_crate: path, skip_on_error, on_error_span })
+ }
+
+ pub(crate) fn with_input(&self, input: &DeriveInput) -> Self {
+ Self {
+ ast: input.clone(),
+ zerocopy_crate: self.zerocopy_crate.clone(),
+ skip_on_error: self.skip_on_error,
+ on_error_span: self.on_error_span,
+ }
+ }
+
+ pub(crate) fn core_path(&self) -> TokenStream {
+ let zerocopy_crate = &self.zerocopy_crate;
+ quote!(#zerocopy_crate::util::macro_util::core_reexport)
+ }
+
+ pub(crate) fn cfg_compile_error(&self) -> TokenStream {
+ // By checking both during the compilation of the proc macro *and* in
+ // the generated code, we ensure that `--cfg
+ // zerocopy_unstable_derive_on_error` need only be passed *either* when
+ // compiling this crate *or* when compiling the user's crate. The former
+ // is preferable, but in some situations (such as when cross-compiling
+ // using `cargo build --target`), it doesn't get propagated to this
+ // crate's build by default.
+ if cfg!(zerocopy_unstable_derive_on_error) {
+ quote!()
+ } else if let Some(span) = self.on_error_span {
+ let core = self.core_path();
+ let error_message = "`on_error` is experimental; pass '--cfg zerocopy_unstable_derive_on_error' to enable";
+ quote::quote_spanned! {span=>
+ #[allow(unused_attributes, unexpected_cfgs)]
+ const _: () = {
+ #[cfg(not(zerocopy_unstable_derive_on_error))]
+ #core::compile_error!(#error_message);
+ };
+ }
+ } else {
+ quote!()
+ }
+ }
+
+ pub(crate) fn error_or_skip<E>(&self, error: E) -> Result<TokenStream, E> {
+ if self.skip_on_error {
+ Ok(self.cfg_compile_error())
+ } else {
+ Err(error)
+ }
+ }
+}
+
+pub(crate) trait DataExt {
+ /// Extracts the names and types of all fields. For enums, extracts the
+ /// names and types of fields from each variant. For tuple structs, the
+ /// names are the indices used to index into the struct (ie, `0`, `1`, etc).
+ ///
+ /// FIXME: Extracting field names for enums doesn't really make sense. Types
+ /// makes sense because we don't care about where they live - we just care
+ /// about transitive ownership. But for field names, we'd only use them when
+ /// generating is_bit_valid, which cares about where they live.
+ fn fields(&self) -> Vec<(&Visibility, TokenStream, &Type)>;
+
+ fn variants(&self) -> Vec<(Option<&Variant>, Vec<(&Visibility, TokenStream, &Type)>)>;
+
+ fn tag(&self) -> Option<Ident>;
+}
+
+impl DataExt for Data {
+ fn fields(&self) -> Vec<(&Visibility, TokenStream, &Type)> {
+ match self {
+ Data::Struct(strc) => strc.fields(),
+ Data::Enum(enm) => enm.fields(),
+ Data::Union(un) => un.fields(),
+ }
+ }
+
+ fn variants(&self) -> Vec<(Option<&Variant>, Vec<(&Visibility, TokenStream, &Type)>)> {
+ match self {
+ Data::Struct(strc) => strc.variants(),
+ Data::Enum(enm) => enm.variants(),
+ Data::Union(un) => un.variants(),
+ }
+ }
+
+ fn tag(&self) -> Option<Ident> {
+ match self {
+ Data::Struct(strc) => strc.tag(),
+ Data::Enum(enm) => enm.tag(),
+ Data::Union(un) => un.tag(),
+ }
+ }
+}
+
+impl DataExt for DataStruct {
+ fn fields(&self) -> Vec<(&Visibility, TokenStream, &Type)> {
+ map_fields(&self.fields)
+ }
+
+ fn variants(&self) -> Vec<(Option<&Variant>, Vec<(&Visibility, TokenStream, &Type)>)> {
+ vec![(None, self.fields())]
+ }
+
+ fn tag(&self) -> Option<Ident> {
+ None
+ }
+}
+
+impl DataExt for DataEnum {
+ fn fields(&self) -> Vec<(&Visibility, TokenStream, &Type)> {
+ map_fields(self.variants.iter().flat_map(|var| &var.fields))
+ }
+
+ fn variants(&self) -> Vec<(Option<&Variant>, Vec<(&Visibility, TokenStream, &Type)>)> {
+ self.variants.iter().map(|var| (Some(var), map_fields(&var.fields))).collect()
+ }
+
+ fn tag(&self) -> Option<Ident> {
+ Some(Ident::new("___ZerocopyTag", Span::call_site()))
+ }
+}
+
+impl DataExt for DataUnion {
+ fn fields(&self) -> Vec<(&Visibility, TokenStream, &Type)> {
+ map_fields(&self.fields.named)
+ }
+
+ fn variants(&self) -> Vec<(Option<&Variant>, Vec<(&Visibility, TokenStream, &Type)>)> {
+ vec![(None, self.fields())]
+ }
+
+ fn tag(&self) -> Option<Ident> {
+ None
+ }
+}
+
+fn map_fields<'a>(
+ fields: impl 'a + IntoIterator<Item = &'a Field>,
+) -> Vec<(&'a Visibility, TokenStream, &'a Type)> {
+ fields
+ .into_iter()
+ .enumerate()
+ .map(|(idx, f)| {
+ (
+ &f.vis,
+ f.ident
+ .as_ref()
+ .map(ToTokens::to_token_stream)
+ .unwrap_or_else(|| Index::from(idx).to_token_stream()),
+ &f.ty,
+ )
+ })
+ .collect()
+}
+
+pub(crate) fn to_ident_str(t: &impl ToString) -> String {
+ let s = t.to_string();
+ if let Some(stripped) = s.strip_prefix("r#") {
+ stripped.to_string()
+ } else {
+ s
+ }
+}
+
+/// This enum describes what kind of padding check needs to be generated for the
+/// associated impl.
+pub(crate) enum PaddingCheck {
+ /// Check that the sum of the fields' sizes exactly equals the struct's
+ /// size.
+ Struct,
+ /// Check that a `repr(C)` struct has no padding.
+ ReprCStruct,
+ /// Check that the size of each field exactly equals the union's size.
+ Union,
+ /// Check that every variant of the enum contains no padding.
+ ///
+ /// Because doing so requires a tag enum, this padding check requires an
+ /// additional `TokenStream` which defines the tag enum as `___ZerocopyTag`.
+ Enum { tag_type_definition: TokenStream },
+}
+
+impl PaddingCheck {
+ /// Returns the idents of the trait to use and the macro to call in order to
+ /// validate that a type passes the relevant padding check.
+ pub(crate) fn validator_trait_and_macro_idents(&self) -> (Ident, Ident) {
+ let (trt, mcro) = match self {
+ PaddingCheck::Struct => ("PaddingFree", "struct_padding"),
+ PaddingCheck::ReprCStruct => ("DynamicPaddingFree", "repr_c_struct_has_padding"),
+ PaddingCheck::Union => ("PaddingFree", "union_padding"),
+ PaddingCheck::Enum { .. } => ("PaddingFree", "enum_padding"),
+ };
+
+ let trt = Ident::new(trt, Span::call_site());
+ let mcro = Ident::new(mcro, Span::call_site());
+ (trt, mcro)
+ }
+
+ /// Sometimes performing the padding check requires some additional
+ /// "context" code. For enums, this is the definition of the tag enum.
+ pub(crate) fn validator_macro_context(&self) -> Option<&TokenStream> {
+ match self {
+ PaddingCheck::Struct | PaddingCheck::ReprCStruct | PaddingCheck::Union => None,
+ PaddingCheck::Enum { tag_type_definition } => Some(tag_type_definition),
+ }
+ }
+}
+
+#[derive(Clone)]
+pub(crate) enum Trait {
+ KnownLayout,
+ HasTag,
+ HasField {
+ variant_id: Box<Expr>,
+ field: Box<Type>,
+ field_id: Box<Expr>,
+ },
+ ProjectField {
+ variant_id: Box<Expr>,
+ field: Box<Type>,
+ field_id: Box<Expr>,
+ invariants: Box<Type>,
+ },
+ Immutable,
+ TryFromBytes,
+ FromZeros,
+ FromBytes,
+ IntoBytes,
+ Unaligned,
+ Sized,
+ ByteHash,
+ ByteEq,
+ SplitAt,
+}
+
+impl ToTokens for Trait {
+ fn to_tokens(&self, tokens: &mut TokenStream) {
+ // According to [1], the format of the derived `Debug`` output is not
+ // stable and therefore not guaranteed to represent the variant names.
+ // Indeed with the (unstable) `fmt-debug` compiler flag [2], it can
+ // return only a minimalized output or empty string. To make sure this
+ // code will work in the future and independent of the compiler flag, we
+ // translate the variants to their names manually here.
+ //
+ // [1] https://doc.rust-lang.org/1.81.0/std/fmt/trait.Debug.html#stability
+ // [2] https://doc.rust-lang.org/beta/unstable-book/compiler-flags/fmt-debug.html
+ let s = match self {
+ Trait::HasField { .. } => "HasField",
+ Trait::ProjectField { .. } => "ProjectField",
+ Trait::KnownLayout => "KnownLayout",
+ Trait::HasTag => "HasTag",
+ Trait::Immutable => "Immutable",
+ Trait::TryFromBytes => "TryFromBytes",
+ Trait::FromZeros => "FromZeros",
+ Trait::FromBytes => "FromBytes",
+ Trait::IntoBytes => "IntoBytes",
+ Trait::Unaligned => "Unaligned",
+ Trait::Sized => "Sized",
+ Trait::ByteHash => "ByteHash",
+ Trait::ByteEq => "ByteEq",
+ Trait::SplitAt => "SplitAt",
+ };
+ let ident = Ident::new(s, Span::call_site());
+ let arguments: Option<syn::AngleBracketedGenericArguments> = match self {
+ Trait::HasField { variant_id, field, field_id } => {
+ Some(parse_quote!(<#field, #variant_id, #field_id>))
+ }
+ Trait::ProjectField { variant_id, field, field_id, invariants } => {
+ Some(parse_quote!(<#field, #invariants, #variant_id, #field_id>))
+ }
+ Trait::KnownLayout
+ | Trait::HasTag
+ | Trait::Immutable
+ | Trait::TryFromBytes
+ | Trait::FromZeros
+ | Trait::FromBytes
+ | Trait::IntoBytes
+ | Trait::Unaligned
+ | Trait::Sized
+ | Trait::ByteHash
+ | Trait::ByteEq
+ | Trait::SplitAt => None,
+ };
+ tokens.extend(quote!(#ident #arguments));
+ }
+}
+
+impl Trait {
+ pub(crate) fn crate_path(&self, ctx: &Ctx) -> Path {
+ let zerocopy_crate = &ctx.zerocopy_crate;
+ let core = ctx.core_path();
+ match self {
+ Self::Sized => parse_quote!(#core::marker::#self),
+ _ => parse_quote!(#zerocopy_crate::#self),
+ }
+ }
+}
+
+pub(crate) enum TraitBound {
+ Slf,
+ Other(Trait),
+}
+
+pub(crate) enum FieldBounds<'a> {
+ None,
+ All(&'a [TraitBound]),
+ Trailing(&'a [TraitBound]),
+ Explicit(Vec<WherePredicate>),
+}
+
+impl<'a> FieldBounds<'a> {
+ pub(crate) const ALL_SELF: FieldBounds<'a> = FieldBounds::All(&[TraitBound::Slf]);
+ pub(crate) const TRAILING_SELF: FieldBounds<'a> = FieldBounds::Trailing(&[TraitBound::Slf]);
+}
+
+pub(crate) enum SelfBounds<'a> {
+ None,
+ All(&'a [Trait]),
+}
+
+// FIXME(https://github.com/rust-lang/rust-clippy/issues/12908): This is a false
+// positive. Explicit lifetimes are actually necessary here.
+#[allow(clippy::needless_lifetimes)]
+impl<'a> SelfBounds<'a> {
+ pub(crate) const SIZED: Self = Self::All(&[Trait::Sized]);
+}
+
+/// Normalizes a slice of bounds by replacing [`TraitBound::Slf`] with `slf`.
+pub(crate) fn normalize_bounds<'a>(
+ slf: &'a Trait,
+ bounds: &'a [TraitBound],
+) -> impl 'a + Iterator<Item = Trait> {
+ bounds.iter().map(move |bound| match bound {
+ TraitBound::Slf => slf.clone(),
+ TraitBound::Other(trt) => trt.clone(),
+ })
+}
+
+pub(crate) struct ImplBlockBuilder<'a> {
+ ctx: &'a Ctx,
+ data: &'a dyn DataExt,
+ trt: Trait,
+ field_type_trait_bounds: FieldBounds<'a>,
+ self_type_trait_bounds: SelfBounds<'a>,
+ padding_check: Option<PaddingCheck>,
+ param_extras: Vec<GenericParam>,
+ inner_extras: Option<TokenStream>,
+ outer_extras: Option<TokenStream>,
+}
+
+impl<'a> ImplBlockBuilder<'a> {
+ pub(crate) fn new(
+ ctx: &'a Ctx,
+ data: &'a dyn DataExt,
+ trt: Trait,
+ field_type_trait_bounds: FieldBounds<'a>,
+ ) -> Self {
+ Self {
+ ctx,
+ data,
+ trt,
+ field_type_trait_bounds,
+ self_type_trait_bounds: SelfBounds::None,
+ padding_check: None,
+ param_extras: Vec::new(),
+ inner_extras: None,
+ outer_extras: None,
+ }
+ }
+
+ pub(crate) fn self_type_trait_bounds(mut self, self_type_trait_bounds: SelfBounds<'a>) -> Self {
+ self.self_type_trait_bounds = self_type_trait_bounds;
+ self
+ }
+
+ pub(crate) fn padding_check<P: Into<Option<PaddingCheck>>>(mut self, padding_check: P) -> Self {
+ self.padding_check = padding_check.into();
+ self
+ }
+
+ pub(crate) fn param_extras(mut self, param_extras: Vec<GenericParam>) -> Self {
+ self.param_extras.extend(param_extras);
+ self
+ }
+
+ pub(crate) fn inner_extras(mut self, inner_extras: TokenStream) -> Self {
+ self.inner_extras = Some(inner_extras);
+ self
+ }
+
+ pub(crate) fn outer_extras<T: Into<Option<TokenStream>>>(mut self, outer_extras: T) -> Self {
+ self.outer_extras = outer_extras.into();
+ self
+ }
+
+ pub(crate) fn build(self) -> TokenStream {
+ // In this documentation, we will refer to this hypothetical struct:
+ //
+ // #[derive(FromBytes)]
+ // struct Foo<T, I: Iterator>
+ // where
+ // T: Copy,
+ // I: Clone,
+ // I::Item: Clone,
+ // {
+ // a: u8,
+ // b: T,
+ // c: I::Item,
+ // }
+ //
+ // We extract the field types, which in this case are `u8`, `T`, and
+ // `I::Item`. We re-use the existing parameters and where clauses. If
+ // `require_trait_bound == true` (as it is for `FromBytes), we add where
+ // bounds for each field's type:
+ //
+ // impl<T, I: Iterator> FromBytes for Foo<T, I>
+ // where
+ // T: Copy,
+ // I: Clone,
+ // I::Item: Clone,
+ // T: FromBytes,
+ // I::Item: FromBytes,
+ // {
+ // }
+ //
+ // NOTE: It is standard practice to only emit bounds for the type
+ // parameters themselves, not for field types based on those parameters
+ // (e.g., `T` vs `T::Foo`). For a discussion of why this is standard
+ // practice, see https://github.com/rust-lang/rust/issues/26925.
+ //
+ // The reason we diverge from this standard is that doing it that way
+ // for us would be unsound. E.g., consider a type, `T` where `T:
+ // FromBytes` but `T::Foo: !FromBytes`. It would not be sound for us to
+ // accept a type with a `T::Foo` field as `FromBytes` simply because `T:
+ // FromBytes`.
+ //
+ // While there's no getting around this requirement for us, it does have
+ // the pretty serious downside that, when lifetimes are involved, the
+ // trait solver ties itself in knots:
+ //
+ // #[derive(Unaligned)]
+ // #[repr(C)]
+ // struct Dup<'a, 'b> {
+ // a: PhantomData<&'a u8>,
+ // b: PhantomData<&'b u8>,
+ // }
+ //
+ // error[E0283]: type annotations required: cannot resolve `core::marker::PhantomData<&'a u8>: zerocopy::Unaligned`
+ // --> src/main.rs:6:10
+ // |
+ // 6 | #[derive(Unaligned)]
+ // | ^^^^^^^^^
+ // |
+ // = note: required by `zerocopy::Unaligned`
+
+ let type_ident = &self.ctx.ast.ident;
+ let trait_path = self.trt.crate_path(self.ctx);
+ let fields = self.data.fields();
+ let variants = self.data.variants();
+ let tag = self.data.tag();
+ let zerocopy_crate = &self.ctx.zerocopy_crate;
+
+ fn bound_tt(ty: &Type, traits: impl Iterator<Item = Trait>, ctx: &Ctx) -> WherePredicate {
+ let traits = traits.map(|t| t.crate_path(ctx));
+ parse_quote!(#ty: #(#traits)+*)
+ }
+ let field_type_bounds: Vec<_> = match (self.field_type_trait_bounds, &fields[..]) {
+ (FieldBounds::All(traits), _) => fields
+ .iter()
+ .map(|(_vis, _name, ty)| {
+ bound_tt(ty, normalize_bounds(&self.trt, traits), self.ctx)
+ })
+ .collect(),
+ (FieldBounds::None, _) | (FieldBounds::Trailing(..), []) => vec![],
+ (FieldBounds::Trailing(traits), [.., last]) => {
+ vec![bound_tt(last.2, normalize_bounds(&self.trt, traits), self.ctx)]
+ }
+ (FieldBounds::Explicit(bounds), _) => bounds,
+ };
+
+ let padding_check_bound = self
+ .padding_check
+ .map(|check| {
+ // Parse the repr for `align` and `packed` modifiers. Note that
+ // `Repr::<PrimitiveRepr, NonZeroU32>` is more permissive than
+ // what Rust supports for structs, enums, or unions, and thus
+ // reliably extracts these modifiers for any kind of type.
+ let repr =
+ Repr::<PrimitiveRepr, NonZeroU32>::from_attrs(&self.ctx.ast.attrs).unwrap();
+ let core = self.ctx.core_path();
+ let option = quote! { #core::option::Option };
+ let nonzero = quote! { #core::num::NonZeroUsize };
+ let none = quote! { #option::None::<#nonzero> };
+ let repr_align =
+ repr.get_align().map(|spanned| {
+ let n = spanned.t.get();
+ quote_spanned! { spanned.span => (#nonzero::new(#n as usize)) }
+ }).unwrap_or(quote! { (#none) });
+ let repr_packed =
+ repr.get_packed().map(|packed| {
+ let n = packed.get();
+ quote! { (#nonzero::new(#n as usize)) }
+ }).unwrap_or(quote! { (#none) });
+ let variant_types = variants.iter().map(|(_, fields)| {
+ let types = fields.iter().map(|(_vis, _name, ty)| ty);
+ quote!([#((#types)),*])
+ });
+ let validator_context = check.validator_macro_context();
+ let (trt, validator_macro) = check.validator_trait_and_macro_idents();
+ let t = tag.iter();
+ parse_quote! {
+ (): #zerocopy_crate::util::macro_util::#trt<
+ Self,
+ {
+ #validator_context
+ #zerocopy_crate::#validator_macro!(Self, #repr_align, #repr_packed, #(#t,)* #(#variant_types),*)
+ }
+ >
+ }
+ });
+
+ let self_bounds: Option<WherePredicate> = match self.self_type_trait_bounds {
+ SelfBounds::None => None,
+ SelfBounds::All(traits) => {
+ Some(bound_tt(&parse_quote!(Self), traits.iter().cloned(), self.ctx))
+ }
+ };
+
+ let bounds = self
+ .ctx
+ .ast
+ .generics
+ .where_clause
+ .as_ref()
+ .map(|where_clause| where_clause.predicates.iter())
+ .into_iter()
+ .flatten()
+ .chain(field_type_bounds.iter())
+ .chain(padding_check_bound.iter())
+ .chain(self_bounds.iter());
+
+ // The parameters with trait bounds, but without type defaults.
+ let mut params: Vec<_> = self
+ .ctx
+ .ast
+ .generics
+ .params
+ .clone()
+ .into_iter()
+ .map(|mut param| {
+ match &mut param {
+ GenericParam::Type(ty) => ty.default = None,
+ GenericParam::Const(cnst) => cnst.default = None,
+ GenericParam::Lifetime(_) => {}
+ }
+ parse_quote!(#param)
+ })
+ .chain(self.param_extras)
+ .collect();
+
+ // For MSRV purposes, ensure that lifetimes precede types precede const
+ // generics.
+ params.sort_by_cached_key(|param| match param {
+ GenericParam::Lifetime(_) => 0,
+ GenericParam::Type(_) => 1,
+ GenericParam::Const(_) => 2,
+ });
+
+ // The identifiers of the parameters without trait bounds or type
+ // defaults.
+ let param_idents = self.ctx.ast.generics.params.iter().map(|param| match param {
+ GenericParam::Type(ty) => {
+ let ident = &ty.ident;
+ quote!(#ident)
+ }
+ GenericParam::Lifetime(l) => {
+ let ident = &l.lifetime;
+ quote!(#ident)
+ }
+ GenericParam::Const(cnst) => {
+ let ident = &cnst.ident;
+ quote!({#ident})
+ }
+ });
+
+ let inner_extras = self.inner_extras;
+ let allow_trivial_bounds =
+ if self.ctx.skip_on_error { quote!(#[allow(trivial_bounds)]) } else { quote!() };
+ let impl_tokens = quote! {
+ #allow_trivial_bounds
+ unsafe impl < #(#params),* > #trait_path for #type_ident < #(#param_idents),* >
+ where
+ #(#bounds,)*
+ {
+ fn only_derive_is_allowed_to_implement_this_trait() {}
+
+ #inner_extras
+ }
+ };
+
+ let outer_extras = self.outer_extras.filter(|e| !e.is_empty());
+ let cfg_compile_error = self.ctx.cfg_compile_error();
+ const_block([Some(cfg_compile_error), Some(impl_tokens), outer_extras])
+ }
+}
+
+// A polyfill for `Option::then_some`, which was added after our MSRV.
+//
+// The `#[allow(unused)]` is necessary because, on sufficiently recent toolchain
+// versions, `b.then_some(...)` resolves to the inherent method rather than to
+// this trait, and so this trait is considered unused.
+//
+// FIXME(#67): Remove this once our MSRV is >= 1.62.
+#[allow(unused)]
+trait BoolExt {
+ fn then_some<T>(self, t: T) -> Option<T>;
+}
+
+impl BoolExt for bool {
+ fn then_some<T>(self, t: T) -> Option<T> {
+ if self {
+ Some(t)
+ } else {
+ None
+ }
+ }
+}
+
+pub(crate) fn const_block(items: impl IntoIterator<Item = Option<TokenStream>>) -> TokenStream {
+ let items = items.into_iter().flatten();
+ quote! {
+ #[allow(
+ // FIXME(#553): Add a test that generates a warning when
+ // `#[allow(deprecated)]` isn't present.
+ deprecated,
+ // Required on some rustc versions due to a lint that is only
+ // triggered when `derive(KnownLayout)` is applied to `repr(C)`
+ // structs that are generated by macros. See #2177 for details.
+ private_bounds,
+ non_local_definitions,
+ non_camel_case_types,
+ non_upper_case_globals,
+ non_snake_case,
+ non_ascii_idents,
+ clippy::missing_inline_in_public_items,
+ )]
+ #[deny(ambiguous_associated_items)]
+ // While there are not currently any warnings that this suppresses
+ // (that we're aware of), it's good future-proofing hygiene.
+ #[automatically_derived]
+ const _: () = {
+ #(#items)*
+ };
+ }
+}
+pub(crate) fn generate_tag_enum(ctx: &Ctx, repr: &EnumRepr, data: &DataEnum) -> TokenStream {
+ let zerocopy_crate = &ctx.zerocopy_crate;
+ let variants = data.variants.iter().map(|v| {
+ let ident = &v.ident;
+ if let Some((eq, discriminant)) = &v.discriminant {
+ quote! { #ident #eq #discriminant }
+ } else {
+ quote! { #ident }
+ }
+ });
+
+ // Don't include any `repr(align)` when generating the tag enum, as that
+ // could add padding after the tag but before any variants, which is not the
+ // correct behavior.
+ let repr = match repr {
+ EnumRepr::Transparent(span) => quote::quote_spanned! { *span => #[repr(transparent)] },
+ EnumRepr::Compound(c, _) => quote! { #c },
+ };
+
+ quote! {
+ #repr
+ #[allow(dead_code)]
+ pub enum ___ZerocopyTag {
+ #(#variants,)*
+ }
+
+ // SAFETY: `___ZerocopyTag` has no fields, and so it does not permit
+ // interior mutation.
+ unsafe impl #zerocopy_crate::Immutable for ___ZerocopyTag {
+ fn only_derive_is_allowed_to_implement_this_trait() {}
+ }
+ }
+}
+pub(crate) fn enum_size_from_repr(repr: &EnumRepr) -> Result<usize, Error> {
+ use CompoundRepr::*;
+ use PrimitiveRepr::*;
+ use Repr::*;
+ match repr {
+ Transparent(span)
+ | Compound(
+ Spanned {
+ t: C | Rust | Primitive(U32 | I32 | U64 | I64 | U128 | I128 | Usize | Isize),
+ span,
+ },
+ _,
+ ) => Err(Error::new(
+ *span,
+ "`FromBytes` only supported on enums with `#[repr(...)]` attributes `u8`, `i8`, `u16`, or `i16`",
+ )),
+ Compound(Spanned { t: Primitive(U8 | I8), span: _ }, _align) => Ok(8),
+ Compound(Spanned { t: Primitive(U16 | I16), span: _ }, _align) => Ok(16),
+ }
+}
+
+#[cfg(test)]
+pub(crate) mod testutil {
+ use proc_macro2::TokenStream;
+ use syn::visit::{self, Visit};
+
+ /// Checks for hygiene violations in the generated code.
+ ///
+ /// # Panics
+ ///
+ /// Panics if a hygiene violation is found.
+ pub(crate) fn check_hygiene(ts: TokenStream) {
+ struct AmbiguousItemVisitor;
+
+ impl<'ast> Visit<'ast> for AmbiguousItemVisitor {
+ fn visit_path(&mut self, i: &'ast syn::Path) {
+ if i.segments.len() > 1 && i.segments.first().unwrap().ident == "Self" {
+ panic!(
+ "Found ambiguous path `{}` in generated output. \
+ All associated item access must be fully qualified (e.g., `<Self as Trait>::Item`) \
+ to prevent hygiene issues.",
+ quote::quote!(#i)
+ );
+ }
+ visit::visit_path(self, i);
+ }
+ }
+
+ let file = syn::parse2::<syn::File>(ts).expect("failed to parse generated output as File");
+ AmbiguousItemVisitor.visit_file(&file);
+ }
+
+ #[test]
+ fn test_check_hygiene_success() {
+ check_hygiene(quote::quote! {
+ fn foo() {
+ let _ = <Self as Trait>::Item;
+ }
+ });
+ }
+
+ #[test]
+ #[should_panic(expected = "Found ambiguous path `Self :: Ambiguous`")]
+ fn test_check_hygiene_failure() {
+ check_hygiene(quote::quote! {
+ fn foo() {
+ let _ = Self::Ambiguous;
+ }
+ });
+ }
+}
diff --git a/rust/zerocopy/README.md b/rust/zerocopy/README.md
new file mode 100644
index 0000000..99e6cad
--- /dev/null
+++ b/rust/zerocopy/README.md
@@ -0,0 +1,14 @@
+# `zerocopy`
+
+These source files come from the Rust `zerocopy` crate, version v0.8.50
+(released 2026-05-31), hosted in the <https://github.com/google/zerocopy>
+repository, licensed under "BSD-2-Clause OR Apache-2.0 OR MIT" and only
+modified to add the SPDX license identifiers and to remove `Display`
+for `f32` and `f64`.
+
+For copyright details, please see:
+
+ https://github.com/google/zerocopy/blob/v0.8.50/README.md?plain=1
+ https://github.com/google/zerocopy/blob/v0.8.50/LICENSE-BSD
+ https://github.com/google/zerocopy/blob/v0.8.50/LICENSE-APACHE
+ https://github.com/google/zerocopy/blob/v0.8.50/LICENSE-MIT
diff --git a/rust/zerocopy/benches/as_bytes_dynamic_size.rs b/rust/zerocopy/benches/as_bytes_dynamic_size.rs
new file mode 100644
index 0000000..68cd1d6
--- /dev/null
+++ b/rust/zerocopy/benches/as_bytes_dynamic_size.rs
@@ -0,0 +1,9 @@
+use zerocopy::*;
+
+#[path = "formats/coco_dynamic_size.rs"]
+mod format;
+
+#[unsafe(no_mangle)]
+fn bench_as_bytes_dynamic_size(source: &format::CocoPacket) -> &[u8] {
+ source.as_bytes()
+}
diff --git a/rust/zerocopy/benches/as_bytes_dynamic_size.x86-64 b/rust/zerocopy/benches/as_bytes_dynamic_size.x86-64
new file mode 100644
index 0000000..f68bad6
--- /dev/null
+++ b/rust/zerocopy/benches/as_bytes_dynamic_size.x86-64
@@ -0,0 +1,5 @@
+bench_as_bytes_dynamic_size:
+ mov rax, rdi
+ lea rdx, [2*rsi + 5]
+ and rdx, -2
+ ret
diff --git a/rust/zerocopy/benches/as_bytes_dynamic_size.x86-64.mca b/rust/zerocopy/benches/as_bytes_dynamic_size.x86-64.mca
new file mode 100644
index 0000000..c3b92a9
--- /dev/null
+++ b/rust/zerocopy/benches/as_bytes_dynamic_size.x86-64.mca
@@ -0,0 +1,47 @@
+Iterations: 100
+Instructions: 400
+Total Cycles: 137
+Total uOps: 400
+
+Dispatch Width: 4
+uOps Per Cycle: 2.92
+IPC: 2.92
+Block RThroughput: 1.0
+
+
+Instruction Info:
+[1]: #uOps
+[2]: Latency
+[3]: RThroughput
+[4]: MayLoad
+[5]: MayStore
+[6]: HasSideEffects (U)
+
+[1] [2] [3] [4] [5] [6] Instructions:
+ 1 1 0.33 mov rax, rdi
+ 1 1 0.50 lea rdx, [2*rsi + 5]
+ 1 1 0.33 and rdx, -2
+ 1 1 1.00 U ret
+
+
+Resources:
+[0] - SBDivider
+[1] - SBFPDivider
+[2] - SBPort0
+[3] - SBPort1
+[4] - SBPort4
+[5] - SBPort5
+[6.0] - SBPort23
+[6.1] - SBPort23
+
+
+Resource pressure per iteration:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1]
+ - - 1.33 1.33 - 1.34 - -
+
+Resource pressure by instruction:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions:
+ - - - 0.66 - 0.34 - - mov rax, rdi
+ - - 0.33 0.67 - - - - lea rdx, [2*rsi + 5]
+ - - 1.00 - - - - - and rdx, -2
+ - - - - - 1.00 - - ret
diff --git a/rust/zerocopy/benches/as_bytes_static_size.rs b/rust/zerocopy/benches/as_bytes_static_size.rs
new file mode 100644
index 0000000..2ad738e
--- /dev/null
+++ b/rust/zerocopy/benches/as_bytes_static_size.rs
@@ -0,0 +1,9 @@
+use zerocopy::*;
+
+#[path = "formats/coco_static_size.rs"]
+mod format;
+
+#[unsafe(no_mangle)]
+fn bench_as_bytes_static_size(source: &format::CocoPacket) -> &[u8] {
+ source.as_bytes()
+}
diff --git a/rust/zerocopy/benches/as_bytes_static_size.x86-64 b/rust/zerocopy/benches/as_bytes_static_size.x86-64
new file mode 100644
index 0000000..213e74a
--- /dev/null
+++ b/rust/zerocopy/benches/as_bytes_static_size.x86-64
@@ -0,0 +1,4 @@
+bench_as_bytes_static_size:
+ mov rax, rdi
+ mov edx, 6
+ ret
diff --git a/rust/zerocopy/benches/as_bytes_static_size.x86-64.mca b/rust/zerocopy/benches/as_bytes_static_size.x86-64.mca
new file mode 100644
index 0000000..ae04a6b
--- /dev/null
+++ b/rust/zerocopy/benches/as_bytes_static_size.x86-64.mca
@@ -0,0 +1,45 @@
+Iterations: 100
+Instructions: 300
+Total Cycles: 104
+Total uOps: 300
+
+Dispatch Width: 4
+uOps Per Cycle: 2.88
+IPC: 2.88
+Block RThroughput: 1.0
+
+
+Instruction Info:
+[1]: #uOps
+[2]: Latency
+[3]: RThroughput
+[4]: MayLoad
+[5]: MayStore
+[6]: HasSideEffects (U)
+
+[1] [2] [3] [4] [5] [6] Instructions:
+ 1 1 0.33 mov rax, rdi
+ 1 1 0.33 mov edx, 6
+ 1 1 1.00 U ret
+
+
+Resources:
+[0] - SBDivider
+[1] - SBFPDivider
+[2] - SBPort0
+[3] - SBPort1
+[4] - SBPort4
+[5] - SBPort5
+[6.0] - SBPort23
+[6.1] - SBPort23
+
+
+Resource pressure per iteration:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1]
+ - - 0.99 1.00 - 1.01 - -
+
+Resource pressure by instruction:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions:
+ - - 0.99 - - 0.01 - - mov rax, rdi
+ - - - 1.00 - - - - mov edx, 6
+ - - - - - 1.00 - - ret
diff --git a/rust/zerocopy/benches/extend_vec_zeroed.rs b/rust/zerocopy/benches/extend_vec_zeroed.rs
new file mode 100644
index 0000000..1fbf772
--- /dev/null
+++ b/rust/zerocopy/benches/extend_vec_zeroed.rs
@@ -0,0 +1,9 @@
+use zerocopy::*;
+
+#[path = "formats/coco_static_size.rs"]
+mod format;
+
+#[unsafe(no_mangle)]
+fn bench_extend_vec_zeroed(v: &mut Vec<format::LocoPacket>, additional: usize) -> Option<()> {
+ FromZeros::extend_vec_zeroed(v, additional).ok()
+}
diff --git a/rust/zerocopy/benches/extend_vec_zeroed.x86-64 b/rust/zerocopy/benches/extend_vec_zeroed.x86-64
new file mode 100644
index 0000000..831b2a0
--- /dev/null
+++ b/rust/zerocopy/benches/extend_vec_zeroed.x86-64
@@ -0,0 +1,60 @@
+bench_extend_vec_zeroed:
+ push r15
+ push r14
+ push r13
+ push r12
+ push rbx
+ sub rsp, 32
+ mov rbx, rdi
+ mov rax, qword ptr [rdi]
+ mov r12, qword ptr [rdi + 16]
+ mov rcx, rax
+ sub rcx, r12
+ cmp rsi, rcx
+ jbe .LBB6_3
+ mov r15, r12
+ add r15, rsi
+ jae .LBB6_6
+.LBB6_2:
+ xor eax, eax
+ jmp .LBB6_5
+.LBB6_3:
+ mov rax, qword ptr [rbx + 8]
+ lea r15, [r12 + rsi]
+.LBB6_4:
+ lea rcx, [r12 + 2*r12]
+ lea rdi, [rax + 2*rcx]
+ add rsi, rsi
+ lea rdx, [rsi + 2*rsi]
+ xor esi, esi
+ call qword ptr [rip + memset@GOTPCREL]
+ mov qword ptr [rbx + 16], r15
+ mov al, 1
+.LBB6_5:
+ add rsp, 32
+ pop rbx
+ pop r12
+ pop r13
+ pop r14
+ pop r15
+ ret
+.LBB6_6:
+ mov r13, rsi
+ lea rcx, [rax + rax]
+ cmp r15, rcx
+ cmova rcx, r15
+ cmp rcx, 5
+ mov r14d, 4
+ cmovae r14, rcx
+ mov rdx, qword ptr [rbx + 8]
+ lea rdi, [rsp + 8]
+ mov rsi, rax
+ mov rcx, r14
+ call <alloc::raw_vec::RawVecInner>::finish_grow
+ cmp dword ptr [rsp + 8], 1
+ je .LBB6_2
+ mov rax, qword ptr [rsp + 16]
+ mov qword ptr [rbx + 8], rax
+ mov qword ptr [rbx], r14
+ mov rsi, r13
+ jmp .LBB6_4
diff --git a/rust/zerocopy/benches/extend_vec_zeroed.x86-64.mca b/rust/zerocopy/benches/extend_vec_zeroed.x86-64.mca
new file mode 100644
index 0000000..cfab1ee
--- /dev/null
+++ b/rust/zerocopy/benches/extend_vec_zeroed.x86-64.mca
@@ -0,0 +1,147 @@
+Iterations: 100
+Instructions: 5400
+Total Cycles: 6595
+Total uOps: 6800
+
+Dispatch Width: 4
+uOps Per Cycle: 1.03
+IPC: 0.82
+Block RThroughput: 17.0
+
+
+Instruction Info:
+[1]: #uOps
+[2]: Latency
+[3]: RThroughput
+[4]: MayLoad
+[5]: MayStore
+[6]: HasSideEffects (U)
+
+[1] [2] [3] [4] [5] [6] Instructions:
+ 2 5 1.00 * push r15
+ 2 5 1.00 * push r14
+ 2 5 1.00 * push r13
+ 2 5 1.00 * push r12
+ 2 5 1.00 * push rbx
+ 1 1 0.33 sub rsp, 32
+ 1 1 0.33 mov rbx, rdi
+ 1 5 0.50 * mov rax, qword ptr [rdi]
+ 1 5 0.50 * mov r12, qword ptr [rdi + 16]
+ 1 1 0.33 mov rcx, rax
+ 1 1 0.33 sub rcx, r12
+ 1 1 0.33 cmp rsi, rcx
+ 1 1 1.00 jbe .LBB6_3
+ 1 1 0.33 mov r15, r12
+ 1 1 0.33 add r15, rsi
+ 1 1 1.00 jae .LBB6_6
+ 1 0 0.25 xor eax, eax
+ 1 1 1.00 jmp .LBB6_5
+ 1 5 0.50 * mov rax, qword ptr [rbx + 8]
+ 1 1 0.50 lea r15, [r12 + rsi]
+ 1 1 0.50 lea rcx, [r12 + 2*r12]
+ 1 1 0.50 lea rdi, [rax + 2*rcx]
+ 1 1 0.33 add rsi, rsi
+ 1 1 0.50 lea rdx, [rsi + 2*rsi]
+ 1 0 0.25 xor esi, esi
+ 4 7 1.00 * call qword ptr [rip + memset@GOTPCREL]
+ 1 1 1.00 * mov qword ptr [rbx + 16], r15
+ 1 1 0.33 mov al, 1
+ 1 1 0.33 add rsp, 32
+ 1 6 0.50 * pop rbx
+ 1 6 0.50 * pop r12
+ 1 6 0.50 * pop r13
+ 1 6 0.50 * pop r14
+ 1 6 0.50 * pop r15
+ 1 1 1.00 U ret
+ 1 1 0.33 mov r13, rsi
+ 1 1 0.50 lea rcx, [rax + rax]
+ 1 1 0.33 cmp r15, rcx
+ 3 3 1.00 cmova rcx, r15
+ 1 1 0.33 cmp rcx, 5
+ 1 1 0.33 mov r14d, 4
+ 2 2 0.67 cmovae r14, rcx
+ 1 5 0.50 * mov rdx, qword ptr [rbx + 8]
+ 1 1 0.50 lea rdi, [rsp + 8]
+ 1 1 0.33 mov rsi, rax
+ 1 1 0.33 mov rcx, r14
+ 3 5 1.00 call <alloc::raw_vec::RawVecInner>::finish_grow
+ 2 6 0.50 * cmp dword ptr [rsp + 8], 1
+ 1 1 1.00 je .LBB6_2
+ 1 5 0.50 * mov rax, qword ptr [rsp + 16]
+ 1 1 1.00 * mov qword ptr [rbx + 8], rax
+ 1 1 1.00 * mov qword ptr [rbx], r14
+ 1 1 0.33 mov rsi, r13
+ 1 1 1.00 jmp .LBB6_4
+
+
+Resources:
+[0] - SBDivider
+[1] - SBFPDivider
+[2] - SBPort0
+[3] - SBPort1
+[4] - SBPort4
+[5] - SBPort5
+[6.0] - SBPort23
+[6.1] - SBPort23
+
+
+Resource pressure per iteration:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1]
+ - - 12.00 12.00 10.00 13.00 11.00 11.00
+
+Resource pressure by instruction:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions:
+ - - - - 1.00 - 0.49 0.51 push r15
+ - - - - 1.00 - 0.51 0.49 push r14
+ - - - - 1.00 - 0.50 0.50 push r13
+ - - - - 1.00 - 0.50 0.50 push r12
+ - - - - 1.00 - 0.50 0.50 push rbx
+ - - 0.01 0.99 - - - - sub rsp, 32
+ - - - - - 1.00 - - mov rbx, rdi
+ - - - - - - 0.50 0.50 mov rax, qword ptr [rdi]
+ - - - - - - 0.50 0.50 mov r12, qword ptr [rdi + 16]
+ - - - 1.00 - - - - mov rcx, rax
+ - - - 0.99 - 0.01 - - sub rcx, r12
+ - - - - - 1.00 - - cmp rsi, rcx
+ - - - - - 1.00 - - jbe .LBB6_3
+ - - 0.01 0.98 - 0.01 - - mov r15, r12
+ - - 0.99 0.01 - - - - add r15, rsi
+ - - - - - 1.00 - - jae .LBB6_6
+ - - - - - - - - xor eax, eax
+ - - - - - 1.00 - - jmp .LBB6_5
+ - - - - - - 0.50 0.50 mov rax, qword ptr [rbx + 8]
+ - - 1.00 - - - - - lea r15, [r12 + rsi]
+ - - 0.98 0.02 - - - - lea rcx, [r12 + 2*r12]
+ - - 0.99 0.01 - - - - lea rdi, [rax + 2*rcx]
+ - - - 1.00 - - - - add rsi, rsi
+ - - 0.99 0.01 - - - - lea rdx, [rsi + 2*rsi]
+ - - - - - - - - xor esi, esi
+ - - - - 1.00 1.00 1.00 1.00 call qword ptr [rip + memset@GOTPCREL]
+ - - - - 1.00 - 0.50 0.50 mov qword ptr [rbx + 16], r15
+ - - 0.01 0.99 - - - - mov al, 1
+ - - 1.00 - - - - - add rsp, 32
+ - - - - - - 0.50 0.50 pop rbx
+ - - - - - - 0.50 0.50 pop r12
+ - - - - - - 0.50 0.50 pop r13
+ - - - - - - 0.50 0.50 pop r14
+ - - - - - - 0.50 0.50 pop r15
+ - - - - - 1.00 - - ret
+ - - 1.00 - - - - - mov r13, rsi
+ - - 0.01 0.99 - - - - lea rcx, [rax + rax]
+ - - 0.99 0.01 - - - - cmp r15, rcx
+ - - 2.00 0.01 - 0.99 - - cmova rcx, r15
+ - - 0.01 0.99 - - - - cmp rcx, 5
+ - - 0.01 0.99 - - - - mov r14d, 4
+ - - 1.00 0.01 - 0.99 - - cmovae r14, rcx
+ - - - - - - 0.50 0.50 mov rdx, qword ptr [rbx + 8]
+ - - 0.01 0.99 - - - - lea rdi, [rsp + 8]
+ - - - 1.00 - - - - mov rsi, rax
+ - - - 0.01 - 0.99 - - mov rcx, r14
+ - - - - 1.00 1.00 0.50 0.50 call <alloc::raw_vec::RawVecInner>::finish_grow
+ - - - 0.99 - 0.01 0.50 0.50 cmp dword ptr [rsp + 8], 1
+ - - - - - 1.00 - - je .LBB6_2
+ - - - - - - 0.50 0.50 mov rax, qword ptr [rsp + 16]
+ - - - - 1.00 - 0.49 0.51 mov qword ptr [rbx + 8], rax
+ - - - - 1.00 - 0.51 0.49 mov qword ptr [rbx], r14
+ - - 0.99 0.01 - - - - mov rsi, r13
+ - - - - - 1.00 - - jmp .LBB6_4
diff --git a/rust/zerocopy/benches/formats/coco_dynamic_padding.rs b/rust/zerocopy/benches/formats/coco_dynamic_padding.rs
new file mode 100644
index 0000000..e494bce
--- /dev/null
+++ b/rust/zerocopy/benches/formats/coco_dynamic_padding.rs
@@ -0,0 +1,24 @@
+use zerocopy_derive::*;
+
+// The only valid value of this type are the bytes `0xC0C0`.
+#[derive(TryFromBytes, KnownLayout, Immutable)]
+#[repr(u16)]
+pub enum C0C0 {
+ _XC0C0 = 0xC0C0,
+}
+
+#[derive(FromBytes, KnownLayout, Immutable, SplitAt)]
+#[repr(C, align(4))]
+pub struct Packet<Magic> {
+ magic_number: Magic,
+ milk: u8,
+ mug_size: u8,
+ temperature: [u8; 5],
+ marshmallows: [[u8; 3]],
+}
+
+/// A packet begining with the magic number `0xC0C0`.
+pub type CocoPacket = Packet<C0C0>;
+
+/// A packet beginning with any two initialized bytes.
+pub type LocoPacket = Packet<[u8; 2]>;
diff --git a/rust/zerocopy/benches/formats/coco_dynamic_size.rs b/rust/zerocopy/benches/formats/coco_dynamic_size.rs
new file mode 100644
index 0000000..5936463
--- /dev/null
+++ b/rust/zerocopy/benches/formats/coco_dynamic_size.rs
@@ -0,0 +1,27 @@
+use zerocopy_derive::*;
+
+// The only valid value of this type are the bytes `0xC0C0`.
+#[derive(TryFromBytes, KnownLayout, Immutable, IntoBytes)]
+#[repr(u16)]
+pub enum C0C0 {
+ _XC0C0 = 0xC0C0,
+}
+
+macro_rules! define_packet {
+ ($name: ident, $trait: ident, $leading_field: ty) => {
+ #[derive($trait, KnownLayout, Immutable, IntoBytes, SplitAt)]
+ #[repr(C, align(2))]
+ pub struct $name {
+ magic_number: $leading_field,
+ mug_size: u8,
+ temperature: u8,
+ marshmallows: [[u8; 2]],
+ }
+ };
+}
+
+/// Packet begins with bytes 0xC0C0.
+define_packet!(CocoPacket, TryFromBytes, C0C0);
+
+/// Packet begins with any two bytes.
+define_packet!(LocoPacket, FromBytes, [u8; 2]);
diff --git a/rust/zerocopy/benches/formats/coco_static_size.rs b/rust/zerocopy/benches/formats/coco_static_size.rs
new file mode 100644
index 0000000..0839497
--- /dev/null
+++ b/rust/zerocopy/benches/formats/coco_static_size.rs
@@ -0,0 +1,27 @@
+use zerocopy_derive::*;
+
+// The only valid value of this type are the bytes `0xC0C0`.
+#[derive(TryFromBytes, KnownLayout, Immutable, IntoBytes)]
+#[repr(u16)]
+pub enum C0C0 {
+ _XC0C0 = 0xC0C0,
+}
+
+macro_rules! define_packet {
+ ($name: ident, $trait: ident, $leading_field: ty) => {
+ #[derive($trait, KnownLayout, Immutable, IntoBytes)]
+ #[repr(C, align(2))]
+ pub struct $name {
+ magic_number: $leading_field,
+ mug_size: u8,
+ temperature: u8,
+ marshmallows: [u8; 2],
+ }
+ };
+}
+
+/// Packet begins with bytes 0xC0C0.
+define_packet!(CocoPacket, TryFromBytes, C0C0);
+
+/// Packet begins with any two bytes.
+define_packet!(LocoPacket, FromBytes, [u8; 2]);
diff --git a/rust/zerocopy/benches/insert_vec_zeroed.rs b/rust/zerocopy/benches/insert_vec_zeroed.rs
new file mode 100644
index 0000000..a5d685c
--- /dev/null
+++ b/rust/zerocopy/benches/insert_vec_zeroed.rs
@@ -0,0 +1,13 @@
+use zerocopy::*;
+
+#[path = "formats/coco_static_size.rs"]
+mod format;
+
+#[unsafe(no_mangle)]
+fn bench_insert_vec_zeroed(
+ v: &mut Vec<format::LocoPacket>,
+ position: usize,
+ additional: usize,
+) -> Option<()> {
+ FromZeros::insert_vec_zeroed(v, position, additional).ok()
+}
diff --git a/rust/zerocopy/benches/insert_vec_zeroed.x86-64 b/rust/zerocopy/benches/insert_vec_zeroed.x86-64
new file mode 100644
index 0000000..9db8740
--- /dev/null
+++ b/rust/zerocopy/benches/insert_vec_zeroed.x86-64
@@ -0,0 +1,79 @@
+bench_insert_vec_zeroed:
+ push rbp
+ push r15
+ push r14
+ push r13
+ push r12
+ push rbx
+ sub rsp, 24
+ mov r12, qword ptr [rdi + 16]
+ mov r13, r12
+ sub r13, rsi
+ jb .LBB6_10
+ mov rbx, rdi
+ mov rax, qword ptr [rdi]
+ mov rcx, rax
+ sub rcx, r12
+ cmp rdx, rcx
+ jbe .LBB6_4
+ add r12, rdx
+ jae .LBB6_7
+.LBB6_3:
+ xor eax, eax
+ jmp .LBB6_6
+.LBB6_4:
+ mov rax, qword ptr [rbx + 8]
+ add r12, rdx
+.LBB6_5:
+ lea rcx, [rsi + 2*rsi]
+ lea r14, [rax + 2*rcx]
+ add rdx, rdx
+ lea r15, [rdx + 2*rdx]
+ lea rdi, [r14 + r15]
+ add r13, r13
+ lea rdx, [2*r13]
+ add rdx, r13
+ mov rsi, r14
+ call qword ptr [rip + memmove@GOTPCREL]
+ mov rdi, r14
+ xor esi, esi
+ mov rdx, r15
+ call qword ptr [rip + memset@GOTPCREL]
+ mov qword ptr [rbx + 16], r12
+ mov al, 1
+.LBB6_6:
+ add rsp, 24
+ pop rbx
+ pop r12
+ pop r13
+ pop r14
+ pop r15
+ pop rbp
+ ret
+.LBB6_7:
+ mov r15, rsi
+ mov rbp, rdx
+ lea rcx, [rax + rax]
+ cmp r12, rcx
+ cmova rcx, r12
+ cmp rcx, 5
+ mov r14d, 4
+ cmovae r14, rcx
+ mov rdx, qword ptr [rbx + 8]
+ mov rdi, rsp
+ mov rsi, rax
+ mov rcx, r14
+ call <alloc::raw_vec::RawVecInner>::finish_grow
+ cmp dword ptr [rsp], 1
+ je .LBB6_3
+ mov rax, qword ptr [rsp + 8]
+ mov qword ptr [rbx + 8], rax
+ mov qword ptr [rbx], r14
+ mov rdx, rbp
+ mov rsi, r15
+ jmp .LBB6_5
+.LBB6_10:
+ lea rdi, [rip + .Lanon.HASH.1]
+ lea rdx, [rip + .Lanon.HASH.3]
+ mov esi, 37
+ call qword ptr [rip + core::panicking::panic@GOTPCREL]
diff --git a/rust/zerocopy/benches/insert_vec_zeroed.x86-64.mca b/rust/zerocopy/benches/insert_vec_zeroed.x86-64.mca
new file mode 100644
index 0000000..6652406
--- /dev/null
+++ b/rust/zerocopy/benches/insert_vec_zeroed.x86-64.mca
@@ -0,0 +1,183 @@
+Iterations: 100
+Instructions: 7200
+Total Cycles: 7648
+Total uOps: 9300
+
+Dispatch Width: 4
+uOps Per Cycle: 1.22
+IPC: 0.94
+Block RThroughput: 23.3
+
+
+Instruction Info:
+[1]: #uOps
+[2]: Latency
+[3]: RThroughput
+[4]: MayLoad
+[5]: MayStore
+[6]: HasSideEffects (U)
+
+[1] [2] [3] [4] [5] [6] Instructions:
+ 2 5 1.00 * push rbp
+ 2 5 1.00 * push r15
+ 2 5 1.00 * push r14
+ 2 5 1.00 * push r13
+ 2 5 1.00 * push r12
+ 2 5 1.00 * push rbx
+ 1 1 0.33 sub rsp, 24
+ 1 5 0.50 * mov r12, qword ptr [rdi + 16]
+ 1 1 0.33 mov r13, r12
+ 1 1 0.33 sub r13, rsi
+ 1 1 1.00 jb .LBB6_10
+ 1 1 0.33 mov rbx, rdi
+ 1 5 0.50 * mov rax, qword ptr [rdi]
+ 1 1 0.33 mov rcx, rax
+ 1 1 0.33 sub rcx, r12
+ 1 1 0.33 cmp rdx, rcx
+ 1 1 1.00 jbe .LBB6_4
+ 1 1 0.33 add r12, rdx
+ 1 1 1.00 jae .LBB6_7
+ 1 0 0.25 xor eax, eax
+ 1 1 1.00 jmp .LBB6_6
+ 1 5 0.50 * mov rax, qword ptr [rbx + 8]
+ 1 1 0.33 add r12, rdx
+ 1 1 0.50 lea rcx, [rsi + 2*rsi]
+ 1 1 0.50 lea r14, [rax + 2*rcx]
+ 1 1 0.33 add rdx, rdx
+ 1 1 0.50 lea r15, [rdx + 2*rdx]
+ 1 1 0.50 lea rdi, [r14 + r15]
+ 1 1 0.33 add r13, r13
+ 1 1 0.50 lea rdx, [2*r13]
+ 1 1 0.33 add rdx, r13
+ 1 1 0.33 mov rsi, r14
+ 4 7 1.00 * call qword ptr [rip + memmove@GOTPCREL]
+ 1 1 0.33 mov rdi, r14
+ 1 0 0.25 xor esi, esi
+ 1 1 0.33 mov rdx, r15
+ 4 7 1.00 * call qword ptr [rip + memset@GOTPCREL]
+ 1 1 1.00 * mov qword ptr [rbx + 16], r12
+ 1 1 0.33 mov al, 1
+ 1 1 0.33 add rsp, 24
+ 1 6 0.50 * pop rbx
+ 1 6 0.50 * pop r12
+ 1 6 0.50 * pop r13
+ 1 6 0.50 * pop r14
+ 1 6 0.50 * pop r15
+ 1 6 0.50 * pop rbp
+ 1 1 1.00 U ret
+ 1 1 0.33 mov r15, rsi
+ 1 1 0.33 mov rbp, rdx
+ 1 1 0.50 lea rcx, [rax + rax]
+ 1 1 0.33 cmp r12, rcx
+ 3 3 1.00 cmova rcx, r12
+ 1 1 0.33 cmp rcx, 5
+ 1 1 0.33 mov r14d, 4
+ 2 2 0.67 cmovae r14, rcx
+ 1 5 0.50 * mov rdx, qword ptr [rbx + 8]
+ 1 1 0.33 mov rdi, rsp
+ 1 1 0.33 mov rsi, rax
+ 1 1 0.33 mov rcx, r14
+ 3 5 1.00 call <alloc::raw_vec::RawVecInner>::finish_grow
+ 2 6 0.50 * cmp dword ptr [rsp], 1
+ 1 1 1.00 je .LBB6_3
+ 1 5 0.50 * mov rax, qword ptr [rsp + 8]
+ 1 1 1.00 * mov qword ptr [rbx + 8], rax
+ 1 1 1.00 * mov qword ptr [rbx], r14
+ 1 1 0.33 mov rdx, rbp
+ 1 1 0.33 mov rsi, r15
+ 1 1 1.00 jmp .LBB6_5
+ 1 1 0.50 lea rdi, [rip + .Lanon.HASH.1]
+ 1 1 0.50 lea rdx, [rip + .Lanon.HASH.3]
+ 1 1 0.33 mov esi, 37
+ 4 7 1.00 * call qword ptr [rip + core::panicking::panic@GOTPCREL]
+
+
+Resources:
+[0] - SBDivider
+[1] - SBFPDivider
+[2] - SBPort0
+[3] - SBPort1
+[4] - SBPort4
+[5] - SBPort5
+[6.0] - SBPort23
+[6.1] - SBPort23
+
+
+Resource pressure per iteration:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1]
+ - - 17.02 16.50 13.00 19.48 14.00 14.00
+
+Resource pressure by instruction:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions:
+ - - - - 1.00 - 0.98 0.02 push rbp
+ - - - - 1.00 - 0.02 0.98 push r15
+ - - - - 1.00 - 0.99 0.01 push r14
+ - - - - 1.00 - 0.01 0.99 push r13
+ - - - - 1.00 - 0.99 0.01 push r12
+ - - - - 1.00 - 0.01 0.99 push rbx
+ - - 0.49 0.51 - - - - sub rsp, 24
+ - - - - - - 0.04 0.96 mov r12, qword ptr [rdi + 16]
+ - - 0.49 0.50 - 0.01 - - mov r13, r12
+ - - 0.48 0.51 - 0.01 - - sub r13, rsi
+ - - - - - 1.00 - - jb .LBB6_10
+ - - 0.49 0.49 - 0.02 - - mov rbx, rdi
+ - - - - - - 0.97 0.03 mov rax, qword ptr [rdi]
+ - - 0.51 0.49 - - - - mov rcx, rax
+ - - 0.49 0.02 - 0.49 - - sub rcx, r12
+ - - 0.49 0.50 - 0.01 - - cmp rdx, rcx
+ - - - - - 1.00 - - jbe .LBB6_4
+ - - 0.02 0.49 - 0.49 - - add r12, rdx
+ - - - - - 1.00 - - jae .LBB6_7
+ - - - - - - - - xor eax, eax
+ - - - - - 1.00 - - jmp .LBB6_6
+ - - - - - - 0.97 0.03 mov rax, qword ptr [rbx + 8]
+ - - 0.51 0.49 - - - - add r12, rdx
+ - - 0.49 0.51 - - - - lea rcx, [rsi + 2*rsi]
+ - - 0.50 0.50 - - - - lea r14, [rax + 2*rcx]
+ - - 0.51 0.49 - - - - add rdx, rdx
+ - - 0.50 0.50 - - - - lea r15, [rdx + 2*rdx]
+ - - 0.49 0.51 - - - - lea rdi, [r14 + r15]
+ - - 0.50 0.49 - 0.01 - - add r13, r13
+ - - 0.51 0.49 - - - - lea rdx, [2*r13]
+ - - 0.01 0.01 - 0.98 - - add rdx, r13
+ - - 0.01 - - 0.99 - - mov rsi, r14
+ - - - - 1.00 1.00 1.98 0.02 call qword ptr [rip + memmove@GOTPCREL]
+ - - 0.49 0.50 - 0.01 - - mov rdi, r14
+ - - - - - - - - xor esi, esi
+ - - 0.50 0.50 - - - - mov rdx, r15
+ - - - - 1.00 1.00 1.96 0.04 call qword ptr [rip + memset@GOTPCREL]
+ - - - - 1.00 - 0.01 0.99 mov qword ptr [rbx + 16], r12
+ - - 0.50 - - 0.50 - - mov al, 1
+ - - 0.51 0.49 - - - - add rsp, 24
+ - - - - - - 0.02 0.98 pop rbx
+ - - - - - - 0.03 0.97 pop r12
+ - - - - - - 0.03 0.97 pop r13
+ - - - - - - 0.97 0.03 pop r14
+ - - - - - - 0.03 0.97 pop r15
+ - - - - - - 0.01 0.99 pop rbp
+ - - - - - 1.00 - - ret
+ - - 0.49 0.51 - - - - mov r15, rsi
+ - - 0.51 0.49 - - - - mov rbp, rdx
+ - - 0.49 0.51 - - - - lea rcx, [rax + rax]
+ - - 0.49 0.50 - 0.01 - - cmp r12, rcx
+ - - 1.04 0.50 - 1.46 - - cmova rcx, r12
+ - - 0.49 0.49 - 0.02 - - cmp rcx, 5
+ - - 0.50 - - 0.50 - - mov r14d, 4
+ - - 0.50 0.51 - 0.99 - - cmovae r14, rcx
+ - - - - - - 0.97 0.03 mov rdx, qword ptr [rbx + 8]
+ - - - 0.51 - 0.49 - - mov rdi, rsp
+ - - 0.01 0.50 - 0.49 - - mov rsi, rax
+ - - 0.49 0.50 - 0.01 - - mov rcx, r14
+ - - - - 1.00 1.00 0.99 0.01 call <alloc::raw_vec::RawVecInner>::finish_grow
+ - - 0.51 0.49 - - 0.50 0.50 cmp dword ptr [rsp], 1
+ - - - - - 1.00 - - je .LBB6_3
+ - - - - - - 0.50 0.50 mov rax, qword ptr [rsp + 8]
+ - - - - 1.00 - 0.99 0.01 mov qword ptr [rbx + 8], rax
+ - - - - 1.00 - 0.01 0.99 mov qword ptr [rbx], r14
+ - - 0.49 0.50 - 0.01 - - mov rdx, rbp
+ - - 0.50 0.01 - 0.49 - - mov rsi, r15
+ - - - - - 1.00 - - jmp .LBB6_5
+ - - 0.01 0.99 - - - - lea rdi, [rip + .Lanon.HASH.1]
+ - - 0.99 0.01 - - - - lea rdx, [rip + .Lanon.HASH.3]
+ - - 0.02 0.49 - 0.49 - - mov esi, 37
+ - - - - 1.00 1.00 0.02 1.98 call qword ptr [rip + core::panicking::panic@GOTPCREL]
diff --git a/rust/zerocopy/benches/new_box_zeroed.rs b/rust/zerocopy/benches/new_box_zeroed.rs
new file mode 100644
index 0000000..aa9a66c
--- /dev/null
+++ b/rust/zerocopy/benches/new_box_zeroed.rs
@@ -0,0 +1,9 @@
+use zerocopy::*;
+
+#[path = "formats/coco_static_size.rs"]
+mod format;
+
+#[unsafe(no_mangle)]
+fn bench_new_box_zeroed() -> Option<Box<format::LocoPacket>> {
+ FromZeros::new_box_zeroed().ok()
+}
diff --git a/rust/zerocopy/benches/new_box_zeroed.x86-64 b/rust/zerocopy/benches/new_box_zeroed.x86-64
new file mode 100644
index 0000000..ef74ea53
--- /dev/null
+++ b/rust/zerocopy/benches/new_box_zeroed.x86-64
@@ -0,0 +1,7 @@
+bench_new_box_zeroed:
+ push rax
+ call qword ptr [rip + __rustc::__rust_no_alloc_shim_is_unstable_v2@GOTPCREL]
+ mov edi, 6
+ mov esi, 2
+ pop rax
+ jmp qword ptr [rip + __rustc::__rust_alloc_zeroed@GOTPCREL]
diff --git a/rust/zerocopy/benches/new_box_zeroed.x86-64.mca b/rust/zerocopy/benches/new_box_zeroed.x86-64.mca
new file mode 100644
index 0000000..05afa7f
--- /dev/null
+++ b/rust/zerocopy/benches/new_box_zeroed.x86-64.mca
@@ -0,0 +1,51 @@
+Iterations: 100
+Instructions: 600
+Total Cycles: 1197
+Total uOps: 1100
+
+Dispatch Width: 4
+uOps Per Cycle: 0.92
+IPC: 0.50
+Block RThroughput: 2.8
+
+
+Instruction Info:
+[1]: #uOps
+[2]: Latency
+[3]: RThroughput
+[4]: MayLoad
+[5]: MayStore
+[6]: HasSideEffects (U)
+
+[1] [2] [3] [4] [5] [6] Instructions:
+ 2 5 1.00 * push rax
+ 4 7 1.00 * call qword ptr [rip + __rustc::__rust_no_alloc_shim_is_unstable_v2@GOTPCREL]
+ 1 1 0.33 mov edi, 6
+ 1 1 0.33 mov esi, 2
+ 1 6 0.50 * pop rax
+ 2 6 1.00 * jmp qword ptr [rip + __rustc::__rust_alloc_zeroed@GOTPCREL]
+
+
+Resources:
+[0] - SBDivider
+[1] - SBFPDivider
+[2] - SBPort0
+[3] - SBPort1
+[4] - SBPort4
+[5] - SBPort5
+[6.0] - SBPort23
+[6.1] - SBPort23
+
+
+Resource pressure per iteration:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1]
+ - - 0.99 1.00 2.00 2.01 2.07 2.93
+
+Resource pressure by instruction:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions:
+ - - - - 1.00 - 0.93 0.07 push rax
+ - - - - 1.00 1.00 0.12 1.88 call qword ptr [rip + __rustc::__rust_no_alloc_shim_is_unstable_v2@GOTPCREL]
+ - - 0.99 - - 0.01 - - mov edi, 6
+ - - - 1.00 - - - - mov esi, 2
+ - - - - - - 0.94 0.06 pop rax
+ - - - - - 1.00 0.08 0.92 jmp qword ptr [rip + __rustc::__rust_alloc_zeroed@GOTPCREL]
diff --git a/rust/zerocopy/benches/new_box_zeroed_with_elems_dynamic_padding.rs b/rust/zerocopy/benches/new_box_zeroed_with_elems_dynamic_padding.rs
new file mode 100644
index 0000000..0afde99
--- /dev/null
+++ b/rust/zerocopy/benches/new_box_zeroed_with_elems_dynamic_padding.rs
@@ -0,0 +1,11 @@
+use zerocopy::*;
+
+#[path = "formats/coco_dynamic_padding.rs"]
+mod format;
+
+#[unsafe(no_mangle)]
+fn bench_new_box_zeroed_with_elems_dynamic_padding(
+ count: usize,
+) -> Option<Box<format::LocoPacket>> {
+ FromZeros::new_box_zeroed_with_elems(count).ok()
+}
diff --git a/rust/zerocopy/benches/new_box_zeroed_with_elems_dynamic_padding.x86-64 b/rust/zerocopy/benches/new_box_zeroed_with_elems_dynamic_padding.x86-64
new file mode 100644
index 0000000..22a8d04
--- /dev/null
+++ b/rust/zerocopy/benches/new_box_zeroed_with_elems_dynamic_padding.x86-64
@@ -0,0 +1,24 @@
+bench_new_box_zeroed_with_elems_dynamic_padding:
+ push r14
+ push rbx
+ push rax
+ mov rbx, rdi
+ movabs rax, 3074457345618258598
+ cmp rdi, rax
+ ja .LBB5_1
+ lea r14, [rbx + 2*rbx]
+ or r14, 3
+ add r14, 9
+ call qword ptr [rip + __rustc::__rust_no_alloc_shim_is_unstable_v2@GOTPCREL]
+ mov esi, 4
+ mov rdi, r14
+ call qword ptr [rip + __rustc::__rust_alloc_zeroed@GOTPCREL]
+ jmp .LBB5_3
+.LBB5_1:
+ xor eax, eax
+.LBB5_3:
+ mov rdx, rbx
+ add rsp, 8
+ pop rbx
+ pop r14
+ ret
diff --git a/rust/zerocopy/benches/new_box_zeroed_with_elems_dynamic_padding.x86-64.mca b/rust/zerocopy/benches/new_box_zeroed_with_elems_dynamic_padding.x86-64.mca
new file mode 100644
index 0000000..e6efaede
--- /dev/null
+++ b/rust/zerocopy/benches/new_box_zeroed_with_elems_dynamic_padding.x86-64.mca
@@ -0,0 +1,81 @@
+Iterations: 100
+Instructions: 2100
+Total Cycles: 2990
+Total uOps: 3000
+
+Dispatch Width: 4
+uOps Per Cycle: 1.00
+IPC: 0.70
+Block RThroughput: 7.5
+
+
+Instruction Info:
+[1]: #uOps
+[2]: Latency
+[3]: RThroughput
+[4]: MayLoad
+[5]: MayStore
+[6]: HasSideEffects (U)
+
+[1] [2] [3] [4] [5] [6] Instructions:
+ 2 5 1.00 * push r14
+ 2 5 1.00 * push rbx
+ 2 5 1.00 * push rax
+ 1 1 0.33 mov rbx, rdi
+ 1 1 0.33 movabs rax, 3074457345618258598
+ 1 1 0.33 cmp rdi, rax
+ 1 1 1.00 ja .LBB5_1
+ 1 1 0.50 lea r14, [rbx + 2*rbx]
+ 1 1 0.33 or r14, 3
+ 1 1 0.33 add r14, 9
+ 4 7 1.00 * call qword ptr [rip + __rustc::__rust_no_alloc_shim_is_unstable_v2@GOTPCREL]
+ 1 1 0.33 mov esi, 4
+ 1 1 0.33 mov rdi, r14
+ 4 7 1.00 * call qword ptr [rip + __rustc::__rust_alloc_zeroed@GOTPCREL]
+ 1 1 1.00 jmp .LBB5_3
+ 1 0 0.25 xor eax, eax
+ 1 1 0.33 mov rdx, rbx
+ 1 1 0.33 add rsp, 8
+ 1 6 0.50 * pop rbx
+ 1 6 0.50 * pop r14
+ 1 1 1.00 U ret
+
+
+Resources:
+[0] - SBDivider
+[1] - SBFPDivider
+[2] - SBPort0
+[3] - SBPort1
+[4] - SBPort4
+[5] - SBPort5
+[6.0] - SBPort23
+[6.1] - SBPort23
+
+
+Resource pressure per iteration:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1]
+ - - 4.49 4.50 5.00 6.01 4.50 4.50
+
+Resource pressure by instruction:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions:
+ - - - - 1.00 - 0.50 0.50 push r14
+ - - - - 1.00 - 0.50 0.50 push rbx
+ - - - - 1.00 - 0.50 0.50 push rax
+ - - 0.49 0.50 - 0.01 - - mov rbx, rdi
+ - - 0.50 0.50 - - - - movabs rax, 3074457345618258598
+ - - 0.50 0.50 - - - - cmp rdi, rax
+ - - - - - 1.00 - - ja .LBB5_1
+ - - 0.50 0.50 - - - - lea r14, [rbx + 2*rbx]
+ - - 0.50 0.50 - - - - or r14, 3
+ - - 0.50 - - 0.50 - - add r14, 9
+ - - - - 1.00 1.00 1.00 1.00 call qword ptr [rip + __rustc::__rust_no_alloc_shim_is_unstable_v2@GOTPCREL]
+ - - - 0.50 - 0.50 - - mov esi, 4
+ - - 0.50 0.50 - - - - mov rdi, r14
+ - - - - 1.00 1.00 1.00 1.00 call qword ptr [rip + __rustc::__rust_alloc_zeroed@GOTPCREL]
+ - - - - - 1.00 - - jmp .LBB5_3
+ - - - - - - - - xor eax, eax
+ - - 0.51 0.49 - - - - mov rdx, rbx
+ - - 0.49 0.51 - - - - add rsp, 8
+ - - - - - - 0.50 0.50 pop rbx
+ - - - - - - 0.50 0.50 pop r14
+ - - - - - 1.00 - - ret
diff --git a/rust/zerocopy/benches/new_box_zeroed_with_elems_dynamic_size.rs b/rust/zerocopy/benches/new_box_zeroed_with_elems_dynamic_size.rs
new file mode 100644
index 0000000..1b12ca2
--- /dev/null
+++ b/rust/zerocopy/benches/new_box_zeroed_with_elems_dynamic_size.rs
@@ -0,0 +1,9 @@
+use zerocopy::*;
+
+#[path = "formats/coco_dynamic_size.rs"]
+mod format;
+
+#[unsafe(no_mangle)]
+fn bench_new_box_zeroed_with_elems_dynamic_size(count: usize) -> Option<Box<format::LocoPacket>> {
+ FromZeros::new_box_zeroed_with_elems(count).ok()
+}
diff --git a/rust/zerocopy/benches/new_box_zeroed_with_elems_dynamic_size.x86-64 b/rust/zerocopy/benches/new_box_zeroed_with_elems_dynamic_size.x86-64
new file mode 100644
index 0000000..bff15e5
--- /dev/null
+++ b/rust/zerocopy/benches/new_box_zeroed_with_elems_dynamic_size.x86-64
@@ -0,0 +1,22 @@
+bench_new_box_zeroed_with_elems_dynamic_size:
+ push r14
+ push rbx
+ push rax
+ mov rbx, rdi
+ movabs rax, 4611686018427387901
+ cmp rdi, rax
+ ja .LBB5_1
+ lea r14, [2*rbx + 4]
+ call qword ptr [rip + __rustc::__rust_no_alloc_shim_is_unstable_v2@GOTPCREL]
+ mov esi, 2
+ mov rdi, r14
+ call qword ptr [rip + __rustc::__rust_alloc_zeroed@GOTPCREL]
+ jmp .LBB5_3
+.LBB5_1:
+ xor eax, eax
+.LBB5_3:
+ mov rdx, rbx
+ add rsp, 8
+ pop rbx
+ pop r14
+ ret
diff --git a/rust/zerocopy/benches/new_box_zeroed_with_elems_dynamic_size.x86-64.mca b/rust/zerocopy/benches/new_box_zeroed_with_elems_dynamic_size.x86-64.mca
new file mode 100644
index 0000000..153d36c
--- /dev/null
+++ b/rust/zerocopy/benches/new_box_zeroed_with_elems_dynamic_size.x86-64.mca
@@ -0,0 +1,77 @@
+Iterations: 100
+Instructions: 1900
+Total Cycles: 2990
+Total uOps: 2800
+
+Dispatch Width: 4
+uOps Per Cycle: 0.94
+IPC: 0.64
+Block RThroughput: 7.0
+
+
+Instruction Info:
+[1]: #uOps
+[2]: Latency
+[3]: RThroughput
+[4]: MayLoad
+[5]: MayStore
+[6]: HasSideEffects (U)
+
+[1] [2] [3] [4] [5] [6] Instructions:
+ 2 5 1.00 * push r14
+ 2 5 1.00 * push rbx
+ 2 5 1.00 * push rax
+ 1 1 0.33 mov rbx, rdi
+ 1 1 0.33 movabs rax, 4611686018427387901
+ 1 1 0.33 cmp rdi, rax
+ 1 1 1.00 ja .LBB5_1
+ 1 1 0.50 lea r14, [2*rbx + 4]
+ 4 7 1.00 * call qword ptr [rip + __rustc::__rust_no_alloc_shim_is_unstable_v2@GOTPCREL]
+ 1 1 0.33 mov esi, 2
+ 1 1 0.33 mov rdi, r14
+ 4 7 1.00 * call qword ptr [rip + __rustc::__rust_alloc_zeroed@GOTPCREL]
+ 1 1 1.00 jmp .LBB5_3
+ 1 0 0.25 xor eax, eax
+ 1 1 0.33 mov rdx, rbx
+ 1 1 0.33 add rsp, 8
+ 1 6 0.50 * pop rbx
+ 1 6 0.50 * pop r14
+ 1 1 1.00 U ret
+
+
+Resources:
+[0] - SBDivider
+[1] - SBFPDivider
+[2] - SBPort0
+[3] - SBPort1
+[4] - SBPort4
+[5] - SBPort5
+[6.0] - SBPort23
+[6.1] - SBPort23
+
+
+Resource pressure per iteration:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1]
+ - - 3.97 3.98 5.00 5.05 4.50 4.50
+
+Resource pressure by instruction:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions:
+ - - - - 1.00 - 0.50 0.50 push r14
+ - - - - 1.00 - 0.50 0.50 push rbx
+ - - - - 1.00 - 0.50 0.50 push rax
+ - - 0.05 0.94 - 0.01 - - mov rbx, rdi
+ - - 0.94 0.06 - - - - movabs rax, 4611686018427387901
+ - - 0.06 0.94 - - - - cmp rdi, rax
+ - - - - - 1.00 - - ja .LBB5_1
+ - - 0.94 0.06 - - - - lea r14, [2*rbx + 4]
+ - - - - 1.00 1.00 1.00 1.00 call qword ptr [rip + __rustc::__rust_no_alloc_shim_is_unstable_v2@GOTPCREL]
+ - - 0.98 0.02 - - - - mov esi, 2
+ - - 0.02 0.94 - 0.04 - - mov rdi, r14
+ - - - - 1.00 1.00 1.00 1.00 call qword ptr [rip + __rustc::__rust_alloc_zeroed@GOTPCREL]
+ - - - - - 1.00 - - jmp .LBB5_3
+ - - - - - - - - xor eax, eax
+ - - 0.94 0.06 - - - - mov rdx, rbx
+ - - 0.04 0.96 - - - - add rsp, 8
+ - - - - - - 0.50 0.50 pop rbx
+ - - - - - - 0.50 0.50 pop r14
+ - - - - - 1.00 - - ret
diff --git a/rust/zerocopy/benches/new_vec_zeroed.rs b/rust/zerocopy/benches/new_vec_zeroed.rs
new file mode 100644
index 0000000..3d95b2b
--- /dev/null
+++ b/rust/zerocopy/benches/new_vec_zeroed.rs
@@ -0,0 +1,9 @@
+use zerocopy::*;
+
+#[path = "formats/coco_static_size.rs"]
+mod format;
+
+#[unsafe(no_mangle)]
+fn bench_new_vec_zeroed(len: usize) -> Option<Vec<format::LocoPacket>> {
+ FromZeros::new_vec_zeroed(len).ok()
+}
diff --git a/rust/zerocopy/benches/new_vec_zeroed.x86-64 b/rust/zerocopy/benches/new_vec_zeroed.x86-64
new file mode 100644
index 0000000..b5c083a
--- /dev/null
+++ b/rust/zerocopy/benches/new_vec_zeroed.x86-64
@@ -0,0 +1,40 @@
+bench_new_vec_zeroed:
+ mov rax, rdi
+ movabs rcx, 1537228672809129301
+ cmp rsi, rcx
+ ja .LBB5_5
+ test rsi, rsi
+ je .LBB5_2
+ push r15
+ push r14
+ push rbx
+ lea rcx, [rsi + rsi]
+ lea rbx, [rcx + 2*rcx]
+ mov r14, rax
+ mov r15, rsi
+ call qword ptr [rip + __rustc::__rust_no_alloc_shim_is_unstable_v2@GOTPCREL]
+ mov esi, 2
+ mov rdi, rbx
+ call qword ptr [rip + __rustc::__rust_alloc_zeroed@GOTPCREL]
+ mov rsi, r15
+ mov rcx, rax
+ mov rax, r14
+ test rcx, rcx
+ pop rbx
+ pop r14
+ pop r15
+ je .LBB5_5
+ mov qword ptr [rax], rsi
+ mov qword ptr [rax + 8], rcx
+ mov qword ptr [rax + 16], rsi
+ ret
+.LBB5_5:
+ movabs rcx, -9223372036854775808
+ mov qword ptr [rax], rcx
+ ret
+.LBB5_2:
+ mov ecx, 2
+ mov qword ptr [rax], rsi
+ mov qword ptr [rax + 8], rcx
+ mov qword ptr [rax + 16], rsi
+ ret
diff --git a/rust/zerocopy/benches/new_vec_zeroed.x86-64.mca b/rust/zerocopy/benches/new_vec_zeroed.x86-64.mca
new file mode 100644
index 0000000..b4fb454
--- /dev/null
+++ b/rust/zerocopy/benches/new_vec_zeroed.x86-64.mca
@@ -0,0 +1,113 @@
+Iterations: 100
+Instructions: 3700
+Total Cycles: 3486
+Total uOps: 4600
+
+Dispatch Width: 4
+uOps Per Cycle: 1.32
+IPC: 1.06
+Block RThroughput: 12.0
+
+
+Instruction Info:
+[1]: #uOps
+[2]: Latency
+[3]: RThroughput
+[4]: MayLoad
+[5]: MayStore
+[6]: HasSideEffects (U)
+
+[1] [2] [3] [4] [5] [6] Instructions:
+ 1 1 0.33 mov rax, rdi
+ 1 1 0.33 movabs rcx, 1537228672809129301
+ 1 1 0.33 cmp rsi, rcx
+ 1 1 1.00 ja .LBB5_5
+ 1 1 0.33 test rsi, rsi
+ 1 1 1.00 je .LBB5_2
+ 2 5 1.00 * push r15
+ 2 5 1.00 * push r14
+ 2 5 1.00 * push rbx
+ 1 1 0.50 lea rcx, [rsi + rsi]
+ 1 1 0.50 lea rbx, [rcx + 2*rcx]
+ 1 1 0.33 mov r14, rax
+ 1 1 0.33 mov r15, rsi
+ 4 7 1.00 * call qword ptr [rip + __rustc::__rust_no_alloc_shim_is_unstable_v2@GOTPCREL]
+ 1 1 0.33 mov esi, 2
+ 1 1 0.33 mov rdi, rbx
+ 4 7 1.00 * call qword ptr [rip + __rustc::__rust_alloc_zeroed@GOTPCREL]
+ 1 1 0.33 mov rsi, r15
+ 1 1 0.33 mov rcx, rax
+ 1 1 0.33 mov rax, r14
+ 1 1 0.33 test rcx, rcx
+ 1 6 0.50 * pop rbx
+ 1 6 0.50 * pop r14
+ 1 6 0.50 * pop r15
+ 1 1 1.00 je .LBB5_5
+ 1 1 1.00 * mov qword ptr [rax], rsi
+ 1 1 1.00 * mov qword ptr [rax + 8], rcx
+ 1 1 1.00 * mov qword ptr [rax + 16], rsi
+ 1 1 1.00 U ret
+ 1 1 0.33 movabs rcx, -9223372036854775808
+ 1 1 1.00 * mov qword ptr [rax], rcx
+ 1 1 1.00 U ret
+ 1 1 0.33 mov ecx, 2
+ 1 1 1.00 * mov qword ptr [rax], rsi
+ 1 1 1.00 * mov qword ptr [rax + 8], rcx
+ 1 1 1.00 * mov qword ptr [rax + 16], rsi
+ 1 1 1.00 U ret
+
+
+Resources:
+[0] - SBDivider
+[1] - SBFPDivider
+[2] - SBPort0
+[3] - SBPort1
+[4] - SBPort4
+[5] - SBPort5
+[6.0] - SBPort23
+[6.1] - SBPort23
+
+
+Resource pressure per iteration:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1]
+ - - 6.99 6.99 12.00 10.02 8.00 9.00
+
+Resource pressure by instruction:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions:
+ - - 0.01 0.98 - 0.01 - - mov rax, rdi
+ - - 0.98 0.02 - - - - movabs rcx, 1537228672809129301
+ - - 0.02 0.98 - - - - cmp rsi, rcx
+ - - - - - 1.00 - - ja .LBB5_5
+ - - 0.98 - - 0.02 - - test rsi, rsi
+ - - - - - 1.00 - - je .LBB5_2
+ - - - - 1.00 - - 1.00 push r15
+ - - - - 1.00 - 1.00 - push r14
+ - - - - 1.00 - - 1.00 push rbx
+ - - - 1.00 - - - - lea rcx, [rsi + rsi]
+ - - - 1.00 - - - - lea rbx, [rcx + 2*rcx]
+ - - 1.00 - - - - - mov r14, rax
+ - - 1.00 - - - - - mov r15, rsi
+ - - - - 1.00 1.00 2.00 - call qword ptr [rip + __rustc::__rust_no_alloc_shim_is_unstable_v2@GOTPCREL]
+ - - - 0.01 - 0.99 - - mov esi, 2
+ - - 0.01 0.99 - - - - mov rdi, rbx
+ - - - - 1.00 1.00 - 2.00 call qword ptr [rip + __rustc::__rust_alloc_zeroed@GOTPCREL]
+ - - 0.01 - - 0.99 - - mov rsi, r15
+ - - 0.99 0.01 - - - - mov rcx, rax
+ - - - 0.99 - 0.01 - - mov rax, r14
+ - - 0.99 0.01 - - - - test rcx, rcx
+ - - - - - - - 1.00 pop rbx
+ - - - - - - 1.00 - pop r14
+ - - - - - - - 1.00 pop r15
+ - - - - - 1.00 - - je .LBB5_5
+ - - - - 1.00 - 1.00 - mov qword ptr [rax], rsi
+ - - - - 1.00 - - 1.00 mov qword ptr [rax + 8], rcx
+ - - - - 1.00 - 1.00 - mov qword ptr [rax + 16], rsi
+ - - - - - 1.00 - - ret
+ - - 0.01 0.99 - - - - movabs rcx, -9223372036854775808
+ - - - - 1.00 - - 1.00 mov qword ptr [rax], rcx
+ - - - - - 1.00 - - ret
+ - - 0.99 0.01 - - - - mov ecx, 2
+ - - - - 1.00 - 1.00 - mov qword ptr [rax], rsi
+ - - - - 1.00 - - 1.00 mov qword ptr [rax + 8], rcx
+ - - - - 1.00 - 1.00 - mov qword ptr [rax + 16], rsi
+ - - - - - 1.00 - - ret
diff --git a/rust/zerocopy/benches/new_zeroed.rs b/rust/zerocopy/benches/new_zeroed.rs
new file mode 100644
index 0000000..b49f62e
--- /dev/null
+++ b/rust/zerocopy/benches/new_zeroed.rs
@@ -0,0 +1,9 @@
+use zerocopy::*;
+
+#[path = "formats/coco_static_size.rs"]
+mod format;
+
+#[unsafe(no_mangle)]
+fn bench_new_zeroed() -> format::LocoPacket {
+ FromZeros::new_zeroed()
+}
diff --git a/rust/zerocopy/benches/new_zeroed.x86-64 b/rust/zerocopy/benches/new_zeroed.x86-64
new file mode 100644
index 0000000..b4d305e
--- /dev/null
+++ b/rust/zerocopy/benches/new_zeroed.x86-64
@@ -0,0 +1,3 @@
+bench_new_zeroed:
+ xor eax, eax
+ ret
diff --git a/rust/zerocopy/benches/new_zeroed.x86-64.mca b/rust/zerocopy/benches/new_zeroed.x86-64.mca
new file mode 100644
index 0000000..44583ca
--- /dev/null
+++ b/rust/zerocopy/benches/new_zeroed.x86-64.mca
@@ -0,0 +1,43 @@
+Iterations: 100
+Instructions: 200
+Total Cycles: 103
+Total uOps: 200
+
+Dispatch Width: 4
+uOps Per Cycle: 1.94
+IPC: 1.94
+Block RThroughput: 1.0
+
+
+Instruction Info:
+[1]: #uOps
+[2]: Latency
+[3]: RThroughput
+[4]: MayLoad
+[5]: MayStore
+[6]: HasSideEffects (U)
+
+[1] [2] [3] [4] [5] [6] Instructions:
+ 1 0 0.25 xor eax, eax
+ 1 1 1.00 U ret
+
+
+Resources:
+[0] - SBDivider
+[1] - SBFPDivider
+[2] - SBPort0
+[3] - SBPort1
+[4] - SBPort4
+[5] - SBPort5
+[6.0] - SBPort23
+[6.1] - SBPort23
+
+
+Resource pressure per iteration:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1]
+ - - - - - 1.00 - -
+
+Resource pressure by instruction:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions:
+ - - - - - - - - xor eax, eax
+ - - - - - 1.00 - - ret
diff --git a/rust/zerocopy/benches/read_from_bytes.rs b/rust/zerocopy/benches/read_from_bytes.rs
new file mode 100644
index 0000000..8a3badd
--- /dev/null
+++ b/rust/zerocopy/benches/read_from_bytes.rs
@@ -0,0 +1,7 @@
+#[path = "formats/coco_static_size.rs"]
+mod format;
+
+#[unsafe(no_mangle)]
+fn bench_read_from_bytes_static_size(source: &[u8]) -> Option<format::LocoPacket> {
+ zerocopy::FromBytes::read_from_bytes(source).ok()
+}
diff --git a/rust/zerocopy/benches/read_from_bytes.x86-64 b/rust/zerocopy/benches/read_from_bytes.x86-64
new file mode 100644
index 0000000..9082d79
--- /dev/null
+++ b/rust/zerocopy/benches/read_from_bytes.x86-64
@@ -0,0 +1,15 @@
+bench_read_from_bytes_static_size:
+ mov rcx, rsi
+ cmp rsi, 6
+ jne .LBB5_2
+ mov eax, dword ptr [rdi]
+ movzx ecx, word ptr [rdi + 4]
+ shl rcx, 32
+ or rcx, rax
+.LBB5_2:
+ shl rcx, 16
+ inc rcx
+ xor eax, eax
+ cmp rsi, 6
+ cmove rax, rcx
+ ret
diff --git a/rust/zerocopy/benches/read_from_bytes.x86-64.mca b/rust/zerocopy/benches/read_from_bytes.x86-64.mca
new file mode 100644
index 0000000..77e787c
--- /dev/null
+++ b/rust/zerocopy/benches/read_from_bytes.x86-64.mca
@@ -0,0 +1,65 @@
+Iterations: 100
+Instructions: 1300
+Total Cycles: 377
+Total uOps: 1400
+
+Dispatch Width: 4
+uOps Per Cycle: 3.71
+IPC: 3.45
+Block RThroughput: 3.5
+
+
+Instruction Info:
+[1]: #uOps
+[2]: Latency
+[3]: RThroughput
+[4]: MayLoad
+[5]: MayStore
+[6]: HasSideEffects (U)
+
+[1] [2] [3] [4] [5] [6] Instructions:
+ 1 1 0.33 mov rcx, rsi
+ 1 1 0.33 cmp rsi, 6
+ 1 1 1.00 jne .LBB5_2
+ 1 5 0.50 * mov eax, dword ptr [rdi]
+ 1 5 0.50 * movzx ecx, word ptr [rdi + 4]
+ 1 1 0.50 shl rcx, 32
+ 1 1 0.33 or rcx, rax
+ 1 1 0.50 shl rcx, 16
+ 1 1 0.33 inc rcx
+ 1 0 0.25 xor eax, eax
+ 1 1 0.33 cmp rsi, 6
+ 2 2 0.67 cmove rax, rcx
+ 1 1 1.00 U ret
+
+
+Resources:
+[0] - SBDivider
+[1] - SBFPDivider
+[2] - SBPort0
+[3] - SBPort1
+[4] - SBPort4
+[5] - SBPort5
+[6.0] - SBPort23
+[6.1] - SBPort23
+
+
+Resource pressure per iteration:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1]
+ - - 3.66 3.67 - 3.67 1.00 1.00
+
+Resource pressure by instruction:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions:
+ - - 0.63 0.36 - 0.01 - - mov rcx, rsi
+ - - 0.05 0.05 - 0.90 - - cmp rsi, 6
+ - - - - - 1.00 - - jne .LBB5_2
+ - - - - - - - 1.00 mov eax, dword ptr [rdi]
+ - - - - - - 1.00 - movzx ecx, word ptr [rdi + 4]
+ - - 0.97 - - 0.03 - - shl rcx, 32
+ - - 0.02 0.35 - 0.63 - - or rcx, rax
+ - - 0.98 - - 0.02 - - shl rcx, 16
+ - - - 0.98 - 0.02 - - inc rcx
+ - - - - - - - - xor eax, eax
+ - - 0.03 0.93 - 0.04 - - cmp rsi, 6
+ - - 0.98 1.00 - 0.02 - - cmove rax, rcx
+ - - - - - 1.00 - - ret
diff --git a/rust/zerocopy/benches/read_from_prefix.rs b/rust/zerocopy/benches/read_from_prefix.rs
new file mode 100644
index 0000000..d49bf80
--- /dev/null
+++ b/rust/zerocopy/benches/read_from_prefix.rs
@@ -0,0 +1,10 @@
+#[path = "formats/coco_static_size.rs"]
+mod format;
+
+#[unsafe(no_mangle)]
+fn bench_read_from_prefix_static_size(source: &[u8]) -> Option<format::LocoPacket> {
+ match zerocopy::FromBytes::read_from_prefix(source) {
+ Ok((packet, _rest)) => Some(packet),
+ _ => None,
+ }
+}
diff --git a/rust/zerocopy/benches/read_from_prefix.x86-64 b/rust/zerocopy/benches/read_from_prefix.x86-64
new file mode 100644
index 0000000..c75b06c
--- /dev/null
+++ b/rust/zerocopy/benches/read_from_prefix.x86-64
@@ -0,0 +1,14 @@
+bench_read_from_prefix_static_size:
+ cmp rsi, 5
+ jbe .LBB5_2
+ mov eax, dword ptr [rdi]
+ movzx edi, word ptr [rdi + 4]
+ shl rdi, 32
+ or rdi, rax
+.LBB5_2:
+ shl rdi, 16
+ inc rdi
+ xor eax, eax
+ cmp rsi, 6
+ cmovae rax, rdi
+ ret
diff --git a/rust/zerocopy/benches/read_from_prefix.x86-64.mca b/rust/zerocopy/benches/read_from_prefix.x86-64.mca
new file mode 100644
index 0000000..04e76cd
--- /dev/null
+++ b/rust/zerocopy/benches/read_from_prefix.x86-64.mca
@@ -0,0 +1,63 @@
+Iterations: 100
+Instructions: 1200
+Total Cycles: 905
+Total uOps: 1300
+
+Dispatch Width: 4
+uOps Per Cycle: 1.44
+IPC: 1.33
+Block RThroughput: 3.3
+
+
+Instruction Info:
+[1]: #uOps
+[2]: Latency
+[3]: RThroughput
+[4]: MayLoad
+[5]: MayStore
+[6]: HasSideEffects (U)
+
+[1] [2] [3] [4] [5] [6] Instructions:
+ 1 1 0.33 cmp rsi, 5
+ 1 1 1.00 jbe .LBB5_2
+ 1 5 0.50 * mov eax, dword ptr [rdi]
+ 1 5 0.50 * movzx edi, word ptr [rdi + 4]
+ 1 1 0.50 shl rdi, 32
+ 1 1 0.33 or rdi, rax
+ 1 1 0.50 shl rdi, 16
+ 1 1 0.33 inc rdi
+ 1 0 0.25 xor eax, eax
+ 1 1 0.33 cmp rsi, 6
+ 2 2 0.67 cmovae rax, rdi
+ 1 1 1.00 U ret
+
+
+Resources:
+[0] - SBDivider
+[1] - SBFPDivider
+[2] - SBPort0
+[3] - SBPort1
+[4] - SBPort4
+[5] - SBPort5
+[6.0] - SBPort23
+[6.1] - SBPort23
+
+
+Resource pressure per iteration:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1]
+ - - 3.32 3.32 - 3.36 1.00 1.00
+
+Resource pressure by instruction:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions:
+ - - 0.05 0.94 - 0.01 - - cmp rsi, 5
+ - - - - - 1.00 - - jbe .LBB5_2
+ - - - - - - - 1.00 mov eax, dword ptr [rdi]
+ - - - - - - 1.00 - movzx edi, word ptr [rdi + 4]
+ - - 0.71 - - 0.29 - - shl rdi, 32
+ - - - 0.64 - 0.36 - - or rdi, rax
+ - - 1.00 - - - - - shl rdi, 16
+ - - 0.31 0.40 - 0.29 - - inc rdi
+ - - - - - - - - xor eax, eax
+ - - 0.34 0.35 - 0.31 - - cmp rsi, 6
+ - - 0.91 0.99 - 0.10 - - cmovae rax, rdi
+ - - - - - 1.00 - - ret
diff --git a/rust/zerocopy/benches/read_from_suffix.rs b/rust/zerocopy/benches/read_from_suffix.rs
new file mode 100644
index 0000000..4eaadb0
--- /dev/null
+++ b/rust/zerocopy/benches/read_from_suffix.rs
@@ -0,0 +1,10 @@
+#[path = "formats/coco_static_size.rs"]
+mod format;
+
+#[unsafe(no_mangle)]
+fn bench_read_from_suffix_static_size(source: &[u8]) -> Option<format::LocoPacket> {
+ match zerocopy::FromBytes::read_from_suffix(source) {
+ Ok((_rest, packet)) => Some(packet),
+ _ => None,
+ }
+}
diff --git a/rust/zerocopy/benches/read_from_suffix.x86-64 b/rust/zerocopy/benches/read_from_suffix.x86-64
new file mode 100644
index 0000000..5cff2a0
--- /dev/null
+++ b/rust/zerocopy/benches/read_from_suffix.x86-64
@@ -0,0 +1,15 @@
+bench_read_from_suffix_static_size:
+ mov rcx, rsi
+ cmp rsi, 6
+ jb .LBB5_2
+ mov eax, dword ptr [rdi + rsi - 6]
+ movzx ecx, word ptr [rdi + rsi - 2]
+ shl rcx, 32
+ or rcx, rax
+.LBB5_2:
+ shl rcx, 16
+ inc rcx
+ xor eax, eax
+ cmp rsi, 6
+ cmovae rax, rcx
+ ret
diff --git a/rust/zerocopy/benches/read_from_suffix.x86-64.mca b/rust/zerocopy/benches/read_from_suffix.x86-64.mca
new file mode 100644
index 0000000..0107de8
--- /dev/null
+++ b/rust/zerocopy/benches/read_from_suffix.x86-64.mca
@@ -0,0 +1,65 @@
+Iterations: 100
+Instructions: 1300
+Total Cycles: 377
+Total uOps: 1400
+
+Dispatch Width: 4
+uOps Per Cycle: 3.71
+IPC: 3.45
+Block RThroughput: 3.5
+
+
+Instruction Info:
+[1]: #uOps
+[2]: Latency
+[3]: RThroughput
+[4]: MayLoad
+[5]: MayStore
+[6]: HasSideEffects (U)
+
+[1] [2] [3] [4] [5] [6] Instructions:
+ 1 1 0.33 mov rcx, rsi
+ 1 1 0.33 cmp rsi, 6
+ 1 1 1.00 jb .LBB5_2
+ 1 5 0.50 * mov eax, dword ptr [rdi + rsi - 6]
+ 1 5 0.50 * movzx ecx, word ptr [rdi + rsi - 2]
+ 1 1 0.50 shl rcx, 32
+ 1 1 0.33 or rcx, rax
+ 1 1 0.50 shl rcx, 16
+ 1 1 0.33 inc rcx
+ 1 0 0.25 xor eax, eax
+ 1 1 0.33 cmp rsi, 6
+ 2 2 0.67 cmovae rax, rcx
+ 1 1 1.00 U ret
+
+
+Resources:
+[0] - SBDivider
+[1] - SBFPDivider
+[2] - SBPort0
+[3] - SBPort1
+[4] - SBPort4
+[5] - SBPort5
+[6.0] - SBPort23
+[6.1] - SBPort23
+
+
+Resource pressure per iteration:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1]
+ - - 3.66 3.67 - 3.67 1.00 1.00
+
+Resource pressure by instruction:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions:
+ - - 0.63 0.36 - 0.01 - - mov rcx, rsi
+ - - 0.05 0.05 - 0.90 - - cmp rsi, 6
+ - - - - - 1.00 - - jb .LBB5_2
+ - - - - - - - 1.00 mov eax, dword ptr [rdi + rsi - 6]
+ - - - - - - 1.00 - movzx ecx, word ptr [rdi + rsi - 2]
+ - - 0.97 - - 0.03 - - shl rcx, 32
+ - - 0.02 0.35 - 0.63 - - or rcx, rax
+ - - 0.98 - - 0.02 - - shl rcx, 16
+ - - - 0.98 - 0.02 - - inc rcx
+ - - - - - - - - xor eax, eax
+ - - 0.03 0.93 - 0.04 - - cmp rsi, 6
+ - - 0.98 1.00 - 0.02 - - cmovae rax, rcx
+ - - - - - 1.00 - - ret
diff --git a/rust/zerocopy/benches/ref_from_bytes_dynamic_padding.rs b/rust/zerocopy/benches/ref_from_bytes_dynamic_padding.rs
new file mode 100644
index 0000000..29708df
--- /dev/null
+++ b/rust/zerocopy/benches/ref_from_bytes_dynamic_padding.rs
@@ -0,0 +1,7 @@
+#[path = "formats/coco_dynamic_padding.rs"]
+mod format;
+
+#[unsafe(no_mangle)]
+fn bench_ref_from_bytes_dynamic_padding(source: &[u8]) -> Option<&format::LocoPacket> {
+ zerocopy::FromBytes::ref_from_bytes(source).ok()
+}
diff --git a/rust/zerocopy/benches/ref_from_bytes_dynamic_padding.x86-64 b/rust/zerocopy/benches/ref_from_bytes_dynamic_padding.x86-64
new file mode 100644
index 0000000..e844a46
--- /dev/null
+++ b/rust/zerocopy/benches/ref_from_bytes_dynamic_padding.x86-64
@@ -0,0 +1,22 @@
+bench_ref_from_bytes_dynamic_padding:
+ test dil, 3
+ jne .LBB5_3
+ movabs rax, 9223372036854775804
+ and rax, rsi
+ cmp rax, 9
+ jb .LBB5_3
+ add rax, -9
+ movabs rcx, -6148914691236517205
+ mul rcx
+ shr rdx
+ lea rax, [rdx + 2*rdx]
+ or rax, 3
+ add rax, 9
+ cmp rsi, rax
+ je .LBB5_4
+.LBB5_3:
+ xor edi, edi
+ mov rdx, rsi
+.LBB5_4:
+ mov rax, rdi
+ ret
diff --git a/rust/zerocopy/benches/ref_from_bytes_dynamic_padding.x86-64.mca b/rust/zerocopy/benches/ref_from_bytes_dynamic_padding.x86-64.mca
new file mode 100644
index 0000000..423ed38
--- /dev/null
+++ b/rust/zerocopy/benches/ref_from_bytes_dynamic_padding.x86-64.mca
@@ -0,0 +1,77 @@
+Iterations: 100
+Instructions: 1900
+Total Cycles: 645
+Total uOps: 2000
+
+Dispatch Width: 4
+uOps Per Cycle: 3.10
+IPC: 2.95
+Block RThroughput: 5.0
+
+
+Instruction Info:
+[1]: #uOps
+[2]: Latency
+[3]: RThroughput
+[4]: MayLoad
+[5]: MayStore
+[6]: HasSideEffects (U)
+
+[1] [2] [3] [4] [5] [6] Instructions:
+ 1 1 0.33 test dil, 3
+ 1 1 1.00 jne .LBB5_3
+ 1 1 0.33 movabs rax, 9223372036854775804
+ 1 1 0.33 and rax, rsi
+ 1 1 0.33 cmp rax, 9
+ 1 1 1.00 jb .LBB5_3
+ 1 1 0.33 add rax, -9
+ 1 1 0.33 movabs rcx, -6148914691236517205
+ 2 4 1.00 mul rcx
+ 1 1 0.50 shr rdx
+ 1 1 0.50 lea rax, [rdx + 2*rdx]
+ 1 1 0.33 or rax, 3
+ 1 1 0.33 add rax, 9
+ 1 1 0.33 cmp rsi, rax
+ 1 1 1.00 je .LBB5_4
+ 1 0 0.25 xor edi, edi
+ 1 1 0.33 mov rdx, rsi
+ 1 1 0.33 mov rax, rdi
+ 1 1 1.00 U ret
+
+
+Resources:
+[0] - SBDivider
+[1] - SBFPDivider
+[2] - SBPort0
+[3] - SBPort1
+[4] - SBPort4
+[5] - SBPort5
+[6.0] - SBPort23
+[6.1] - SBPort23
+
+
+Resource pressure per iteration:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1]
+ - - 6.32 6.33 - 6.35 - -
+
+Resource pressure by instruction:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions:
+ - - 0.64 0.35 - 0.01 - - test dil, 3
+ - - - - - 1.00 - - jne .LBB5_3
+ - - 0.34 0.65 - 0.01 - - movabs rax, 9223372036854775804
+ - - 0.35 0.65 - - - - and rax, rsi
+ - - 0.33 0.34 - 0.33 - - cmp rax, 9
+ - - - - - 1.00 - - jb .LBB5_3
+ - - 0.35 - - 0.65 - - add rax, -9
+ - - 0.97 0.01 - 0.02 - - movabs rcx, -6148914691236517205
+ - - 1.00 1.00 - - - - mul rcx
+ - - 0.99 - - 0.01 - - shr rdx
+ - - 0.33 0.67 - - - - lea rax, [rdx + 2*rdx]
+ - - 0.34 0.66 - - - - or rax, 3
+ - - 0.33 0.66 - 0.01 - - add rax, 9
+ - - 0.01 0.99 - - - - cmp rsi, rax
+ - - - - - 1.00 - - je .LBB5_4
+ - - - - - - - - xor edi, edi
+ - - 0.32 0.01 - 0.67 - - mov rdx, rsi
+ - - 0.02 0.34 - 0.64 - - mov rax, rdi
+ - - - - - 1.00 - - ret
diff --git a/rust/zerocopy/benches/ref_from_bytes_dynamic_size.rs b/rust/zerocopy/benches/ref_from_bytes_dynamic_size.rs
new file mode 100644
index 0000000..4eb4f97
--- /dev/null
+++ b/rust/zerocopy/benches/ref_from_bytes_dynamic_size.rs
@@ -0,0 +1,7 @@
+#[path = "formats/coco_dynamic_size.rs"]
+mod format;
+
+#[unsafe(no_mangle)]
+fn bench_ref_from_bytes_dynamic_size(source: &[u8]) -> Option<&format::LocoPacket> {
+ zerocopy::FromBytes::ref_from_bytes(source).ok()
+}
diff --git a/rust/zerocopy/benches/ref_from_bytes_dynamic_size.x86-64 b/rust/zerocopy/benches/ref_from_bytes_dynamic_size.x86-64
new file mode 100644
index 0000000..cc905b7
--- /dev/null
+++ b/rust/zerocopy/benches/ref_from_bytes_dynamic_size.x86-64
@@ -0,0 +1,20 @@
+bench_ref_from_bytes_dynamic_size:
+ mov rdx, rsi
+ cmp rsi, 4
+ setb al
+ or al, dil
+ test al, 1
+ je .LBB5_2
+ xor eax, eax
+ ret
+.LBB5_2:
+ lea rcx, [rdx - 4]
+ mov rsi, rcx
+ and rsi, -2
+ add rsi, 4
+ shr rcx
+ xor eax, eax
+ cmp rdx, rsi
+ cmove rdx, rcx
+ cmove rax, rdi
+ ret
diff --git a/rust/zerocopy/benches/ref_from_bytes_dynamic_size.x86-64.mca b/rust/zerocopy/benches/ref_from_bytes_dynamic_size.x86-64.mca
new file mode 100644
index 0000000..68aea58
--- /dev/null
+++ b/rust/zerocopy/benches/ref_from_bytes_dynamic_size.x86-64.mca
@@ -0,0 +1,75 @@
+Iterations: 100
+Instructions: 1800
+Total Cycles: 704
+Total uOps: 2000
+
+Dispatch Width: 4
+uOps Per Cycle: 2.84
+IPC: 2.56
+Block RThroughput: 5.0
+
+
+Instruction Info:
+[1]: #uOps
+[2]: Latency
+[3]: RThroughput
+[4]: MayLoad
+[5]: MayStore
+[6]: HasSideEffects (U)
+
+[1] [2] [3] [4] [5] [6] Instructions:
+ 1 1 0.33 mov rdx, rsi
+ 1 1 0.33 cmp rsi, 4
+ 1 1 0.50 setb al
+ 1 1 0.33 or al, dil
+ 1 1 0.33 test al, 1
+ 1 1 1.00 je .LBB5_2
+ 1 0 0.25 xor eax, eax
+ 1 1 1.00 U ret
+ 1 1 0.50 lea rcx, [rdx - 4]
+ 1 1 0.33 mov rsi, rcx
+ 1 1 0.33 and rsi, -2
+ 1 1 0.33 add rsi, 4
+ 1 1 0.50 shr rcx
+ 1 0 0.25 xor eax, eax
+ 1 1 0.33 cmp rdx, rsi
+ 2 2 0.67 cmove rdx, rcx
+ 2 2 0.67 cmove rax, rdi
+ 1 1 1.00 U ret
+
+
+Resources:
+[0] - SBDivider
+[1] - SBFPDivider
+[2] - SBPort0
+[3] - SBPort1
+[4] - SBPort4
+[5] - SBPort5
+[6.0] - SBPort23
+[6.1] - SBPort23
+
+
+Resource pressure per iteration:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1]
+ - - 5.97 5.98 - 6.05 - -
+
+Resource pressure by instruction:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions:
+ - - 0.97 0.01 - 0.02 - - mov rdx, rsi
+ - - 0.01 0.02 - 0.97 - - cmp rsi, 4
+ - - 0.03 - - 0.97 - - setb al
+ - - 0.01 0.02 - 0.97 - - or al, dil
+ - - - 0.98 - 0.02 - - test al, 1
+ - - - - - 1.00 - - je .LBB5_2
+ - - - - - - - - xor eax, eax
+ - - - - - 1.00 - - ret
+ - - 0.98 0.02 - - - - lea rcx, [rdx - 4]
+ - - 0.01 0.99 - - - - mov rsi, rcx
+ - - - 0.98 - 0.02 - - and rsi, -2
+ - - 0.98 0.01 - 0.01 - - add rsi, 4
+ - - 0.99 - - 0.01 - - shr rcx
+ - - - - - - - - xor eax, eax
+ - - 0.02 0.97 - 0.01 - - cmp rdx, rsi
+ - - 0.99 0.99 - 0.02 - - cmove rdx, rcx
+ - - 0.98 0.99 - 0.03 - - cmove rax, rdi
+ - - - - - 1.00 - - ret
diff --git a/rust/zerocopy/benches/ref_from_bytes_static_size.rs b/rust/zerocopy/benches/ref_from_bytes_static_size.rs
new file mode 100644
index 0000000..3742bba
--- /dev/null
+++ b/rust/zerocopy/benches/ref_from_bytes_static_size.rs
@@ -0,0 +1,7 @@
+#[path = "formats/coco_static_size.rs"]
+mod format;
+
+#[unsafe(no_mangle)]
+fn bench_ref_from_bytes_static_size(source: &[u8]) -> Option<&format::LocoPacket> {
+ zerocopy::FromBytes::ref_from_bytes(source).ok()
+}
diff --git a/rust/zerocopy/benches/ref_from_bytes_static_size.x86-64 b/rust/zerocopy/benches/ref_from_bytes_static_size.x86-64
new file mode 100644
index 0000000..2c8da68
--- /dev/null
+++ b/rust/zerocopy/benches/ref_from_bytes_static_size.x86-64
@@ -0,0 +1,8 @@
+bench_ref_from_bytes_static_size:
+ mov ecx, edi
+ and ecx, 1
+ xor rsi, 6
+ xor eax, eax
+ or rsi, rcx
+ cmove rax, rdi
+ ret
diff --git a/rust/zerocopy/benches/ref_from_bytes_static_size.x86-64.mca b/rust/zerocopy/benches/ref_from_bytes_static_size.x86-64.mca
new file mode 100644
index 0000000..8326978
--- /dev/null
+++ b/rust/zerocopy/benches/ref_from_bytes_static_size.x86-64.mca
@@ -0,0 +1,53 @@
+Iterations: 100
+Instructions: 700
+Total Cycles: 240
+Total uOps: 800
+
+Dispatch Width: 4
+uOps Per Cycle: 3.33
+IPC: 2.92
+Block RThroughput: 2.0
+
+
+Instruction Info:
+[1]: #uOps
+[2]: Latency
+[3]: RThroughput
+[4]: MayLoad
+[5]: MayStore
+[6]: HasSideEffects (U)
+
+[1] [2] [3] [4] [5] [6] Instructions:
+ 1 1 0.33 mov ecx, edi
+ 1 1 0.33 and ecx, 1
+ 1 1 0.33 xor rsi, 6
+ 1 0 0.25 xor eax, eax
+ 1 1 0.33 or rsi, rcx
+ 2 2 0.67 cmove rax, rdi
+ 1 1 1.00 U ret
+
+
+Resources:
+[0] - SBDivider
+[1] - SBFPDivider
+[2] - SBPort0
+[3] - SBPort1
+[4] - SBPort4
+[5] - SBPort5
+[6.0] - SBPort23
+[6.1] - SBPort23
+
+
+Resource pressure per iteration:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1]
+ - - 2.33 2.33 - 2.34 - -
+
+Resource pressure by instruction:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions:
+ - - 0.01 0.98 - 0.01 - - mov ecx, edi
+ - - 0.02 0.66 - 0.32 - - and ecx, 1
+ - - 0.33 0.66 - 0.01 - - xor rsi, 6
+ - - - - - - - - xor eax, eax
+ - - 0.98 0.02 - - - - or rsi, rcx
+ - - 0.99 0.01 - 1.00 - - cmove rax, rdi
+ - - - - - 1.00 - - ret
diff --git a/rust/zerocopy/benches/ref_from_bytes_with_elems_dynamic_padding.rs b/rust/zerocopy/benches/ref_from_bytes_with_elems_dynamic_padding.rs
new file mode 100644
index 0000000..b4fea4a
--- /dev/null
+++ b/rust/zerocopy/benches/ref_from_bytes_with_elems_dynamic_padding.rs
@@ -0,0 +1,10 @@
+#[path = "formats/coco_dynamic_padding.rs"]
+mod format;
+
+#[unsafe(no_mangle)]
+fn bench_ref_from_bytes_with_elems_dynamic_padding(
+ source: &[u8],
+ count: usize,
+) -> Option<&format::LocoPacket> {
+ zerocopy::FromBytes::ref_from_bytes_with_elems(source, count).ok()
+}
diff --git a/rust/zerocopy/benches/ref_from_bytes_with_elems_dynamic_padding.x86-64 b/rust/zerocopy/benches/ref_from_bytes_with_elems_dynamic_padding.x86-64
new file mode 100644
index 0000000..d579b3f
--- /dev/null
+++ b/rust/zerocopy/benches/ref_from_bytes_with_elems_dynamic_padding.x86-64
@@ -0,0 +1,19 @@
+bench_ref_from_bytes_with_elems_dynamic_padding:
+ movabs rax, 3074457345618258598
+ cmp rdx, rax
+ seta cl
+ mov rax, rdi
+ test al, 3
+ setne dil
+ or dil, cl
+ jne .LBB5_2
+ lea rcx, [rdx + 2*rdx]
+ or rcx, 3
+ add rcx, 9
+ cmp rsi, rcx
+ je .LBB5_3
+.LBB5_2:
+ xor eax, eax
+ mov rdx, rsi
+.LBB5_3:
+ ret
diff --git a/rust/zerocopy/benches/ref_from_bytes_with_elems_dynamic_padding.x86-64.mca b/rust/zerocopy/benches/ref_from_bytes_with_elems_dynamic_padding.x86-64.mca
new file mode 100644
index 0000000..ea2d83db
--- /dev/null
+++ b/rust/zerocopy/benches/ref_from_bytes_with_elems_dynamic_padding.x86-64.mca
@@ -0,0 +1,71 @@
+Iterations: 100
+Instructions: 1600
+Total Cycles: 539
+Total uOps: 1700
+
+Dispatch Width: 4
+uOps Per Cycle: 3.15
+IPC: 2.97
+Block RThroughput: 4.3
+
+
+Instruction Info:
+[1]: #uOps
+[2]: Latency
+[3]: RThroughput
+[4]: MayLoad
+[5]: MayStore
+[6]: HasSideEffects (U)
+
+[1] [2] [3] [4] [5] [6] Instructions:
+ 1 1 0.33 movabs rax, 3074457345618258598
+ 1 1 0.33 cmp rdx, rax
+ 2 2 1.00 seta cl
+ 1 1 0.33 mov rax, rdi
+ 1 1 0.33 test al, 3
+ 1 1 0.50 setne dil
+ 1 1 0.33 or dil, cl
+ 1 1 1.00 jne .LBB5_2
+ 1 1 0.50 lea rcx, [rdx + 2*rdx]
+ 1 1 0.33 or rcx, 3
+ 1 1 0.33 add rcx, 9
+ 1 1 0.33 cmp rsi, rcx
+ 1 1 1.00 je .LBB5_3
+ 1 0 0.25 xor eax, eax
+ 1 1 0.33 mov rdx, rsi
+ 1 1 1.00 U ret
+
+
+Resources:
+[0] - SBDivider
+[1] - SBFPDivider
+[2] - SBPort0
+[3] - SBPort1
+[4] - SBPort4
+[5] - SBPort5
+[6.0] - SBPort23
+[6.1] - SBPort23
+
+
+Resource pressure per iteration:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1]
+ - - 5.33 5.32 - 5.35 - -
+
+Resource pressure by instruction:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions:
+ - - 0.01 0.98 - 0.01 - - movabs rax, 3074457345618258598
+ - - - 1.00 - - - - cmp rdx, rax
+ - - 1.98 - - 0.02 - - seta cl
+ - - 0.02 0.98 - - - - mov rax, rdi
+ - - - 0.67 - 0.33 - - test al, 3
+ - - 0.67 - - 0.33 - - setne dil
+ - - 0.99 - - 0.01 - - or dil, cl
+ - - - - - 1.00 - - jne .LBB5_2
+ - - 0.01 0.99 - - - - lea rcx, [rdx + 2*rdx]
+ - - - 0.01 - 0.99 - - or rcx, 3
+ - - 0.65 0.02 - 0.33 - - add rcx, 9
+ - - 0.99 0.01 - - - - cmp rsi, rcx
+ - - - - - 1.00 - - je .LBB5_3
+ - - - - - - - - xor eax, eax
+ - - 0.01 0.66 - 0.33 - - mov rdx, rsi
+ - - - - - 1.00 - - ret
diff --git a/rust/zerocopy/benches/ref_from_bytes_with_elems_dynamic_size.rs b/rust/zerocopy/benches/ref_from_bytes_with_elems_dynamic_size.rs
new file mode 100644
index 0000000..9d33a7c
--- /dev/null
+++ b/rust/zerocopy/benches/ref_from_bytes_with_elems_dynamic_size.rs
@@ -0,0 +1,10 @@
+#[path = "formats/coco_dynamic_size.rs"]
+mod format;
+
+#[unsafe(no_mangle)]
+fn bench_ref_from_bytes_with_elems_dynamic_size(
+ source: &[u8],
+ count: usize,
+) -> Option<&format::LocoPacket> {
+ zerocopy::FromBytes::ref_from_bytes_with_elems(source, count).ok()
+}
diff --git a/rust/zerocopy/benches/ref_from_bytes_with_elems_dynamic_size.x86-64 b/rust/zerocopy/benches/ref_from_bytes_with_elems_dynamic_size.x86-64
new file mode 100644
index 0000000..3d8d15b
--- /dev/null
+++ b/rust/zerocopy/benches/ref_from_bytes_with_elems_dynamic_size.x86-64
@@ -0,0 +1,16 @@
+bench_ref_from_bytes_with_elems_dynamic_size:
+ movabs rax, 4611686018427387901
+ cmp rdx, rax
+ seta cl
+ mov rax, rdi
+ or dil, cl
+ test dil, 1
+ jne .LBB5_2
+ lea rcx, [2*rdx + 4]
+ cmp rsi, rcx
+ je .LBB5_3
+.LBB5_2:
+ xor eax, eax
+ mov rdx, rsi
+.LBB5_3:
+ ret
diff --git a/rust/zerocopy/benches/ref_from_bytes_with_elems_dynamic_size.x86-64.mca b/rust/zerocopy/benches/ref_from_bytes_with_elems_dynamic_size.x86-64.mca
new file mode 100644
index 0000000..602179f
--- /dev/null
+++ b/rust/zerocopy/benches/ref_from_bytes_with_elems_dynamic_size.x86-64.mca
@@ -0,0 +1,65 @@
+Iterations: 100
+Instructions: 1300
+Total Cycles: 439
+Total uOps: 1400
+
+Dispatch Width: 4
+uOps Per Cycle: 3.19
+IPC: 2.96
+Block RThroughput: 3.5
+
+
+Instruction Info:
+[1]: #uOps
+[2]: Latency
+[3]: RThroughput
+[4]: MayLoad
+[5]: MayStore
+[6]: HasSideEffects (U)
+
+[1] [2] [3] [4] [5] [6] Instructions:
+ 1 1 0.33 movabs rax, 4611686018427387901
+ 1 1 0.33 cmp rdx, rax
+ 2 2 1.00 seta cl
+ 1 1 0.33 mov rax, rdi
+ 1 1 0.33 or dil, cl
+ 1 1 0.33 test dil, 1
+ 1 1 1.00 jne .LBB5_2
+ 1 1 0.50 lea rcx, [2*rdx + 4]
+ 1 1 0.33 cmp rsi, rcx
+ 1 1 1.00 je .LBB5_3
+ 1 0 0.25 xor eax, eax
+ 1 1 0.33 mov rdx, rsi
+ 1 1 1.00 U ret
+
+
+Resources:
+[0] - SBDivider
+[1] - SBFPDivider
+[2] - SBPort0
+[3] - SBPort1
+[4] - SBPort4
+[5] - SBPort5
+[6.0] - SBPort23
+[6.1] - SBPort23
+
+
+Resource pressure per iteration:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1]
+ - - 4.32 4.33 - 4.35 - -
+
+Resource pressure by instruction:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions:
+ - - - 0.99 - 0.01 - - movabs rax, 4611686018427387901
+ - - 0.33 0.67 - - - - cmp rdx, rax
+ - - 1.98 - - 0.02 - - seta cl
+ - - 0.01 0.99 - - - - mov rax, rdi
+ - - 1.00 - - - - - or dil, cl
+ - - 0.99 0.01 - - - - test dil, 1
+ - - - - - 1.00 - - jne .LBB5_2
+ - - - 1.00 - - - - lea rcx, [2*rdx + 4]
+ - - 0.01 - - 0.99 - - cmp rsi, rcx
+ - - - - - 1.00 - - je .LBB5_3
+ - - - - - - - - xor eax, eax
+ - - - 0.67 - 0.33 - - mov rdx, rsi
+ - - - - - 1.00 - - ret
diff --git a/rust/zerocopy/benches/ref_from_prefix_dynamic_padding.rs b/rust/zerocopy/benches/ref_from_prefix_dynamic_padding.rs
new file mode 100644
index 0000000..53c707b
--- /dev/null
+++ b/rust/zerocopy/benches/ref_from_prefix_dynamic_padding.rs
@@ -0,0 +1,10 @@
+#[path = "formats/coco_dynamic_padding.rs"]
+mod format;
+
+#[unsafe(no_mangle)]
+fn bench_ref_from_prefix_dynamic_padding(source: &[u8]) -> Option<&format::LocoPacket> {
+ match zerocopy::FromBytes::ref_from_prefix(source) {
+ Ok((packet, _rest)) => Some(packet),
+ _ => None,
+ }
+}
diff --git a/rust/zerocopy/benches/ref_from_prefix_dynamic_padding.x86-64 b/rust/zerocopy/benches/ref_from_prefix_dynamic_padding.x86-64
new file mode 100644
index 0000000..a58592a
--- /dev/null
+++ b/rust/zerocopy/benches/ref_from_prefix_dynamic_padding.x86-64
@@ -0,0 +1,22 @@
+bench_ref_from_prefix_dynamic_padding:
+ xor edx, edx
+ mov eax, 0
+ test dil, 3
+ je .LBB5_1
+ ret
+.LBB5_1:
+ movabs rax, 9223372036854775804
+ and rsi, rax
+ cmp rsi, 9
+ jae .LBB5_3
+ mov edx, 1
+ xor eax, eax
+ ret
+.LBB5_3:
+ add rsi, -9
+ movabs rcx, -6148914691236517205
+ mov rax, rsi
+ mul rcx
+ shr rdx
+ mov rax, rdi
+ ret
diff --git a/rust/zerocopy/benches/ref_from_prefix_dynamic_padding.x86-64.mca b/rust/zerocopy/benches/ref_from_prefix_dynamic_padding.x86-64.mca
new file mode 100644
index 0000000..62ea4ba
--- /dev/null
+++ b/rust/zerocopy/benches/ref_from_prefix_dynamic_padding.x86-64.mca
@@ -0,0 +1,77 @@
+Iterations: 100
+Instructions: 1900
+Total Cycles: 608
+Total uOps: 2000
+
+Dispatch Width: 4
+uOps Per Cycle: 3.29
+IPC: 3.13
+Block RThroughput: 5.0
+
+
+Instruction Info:
+[1]: #uOps
+[2]: Latency
+[3]: RThroughput
+[4]: MayLoad
+[5]: MayStore
+[6]: HasSideEffects (U)
+
+[1] [2] [3] [4] [5] [6] Instructions:
+ 1 0 0.25 xor edx, edx
+ 1 1 0.33 mov eax, 0
+ 1 1 0.33 test dil, 3
+ 1 1 1.00 je .LBB5_1
+ 1 1 1.00 U ret
+ 1 1 0.33 movabs rax, 9223372036854775804
+ 1 1 0.33 and rsi, rax
+ 1 1 0.33 cmp rsi, 9
+ 1 1 1.00 jae .LBB5_3
+ 1 1 0.33 mov edx, 1
+ 1 0 0.25 xor eax, eax
+ 1 1 1.00 U ret
+ 1 1 0.33 add rsi, -9
+ 1 1 0.33 movabs rcx, -6148914691236517205
+ 1 1 0.33 mov rax, rsi
+ 2 4 1.00 mul rcx
+ 1 1 0.50 shr rdx
+ 1 1 0.33 mov rax, rdi
+ 1 1 1.00 U ret
+
+
+Resources:
+[0] - SBDivider
+[1] - SBFPDivider
+[2] - SBPort0
+[3] - SBPort1
+[4] - SBPort4
+[5] - SBPort5
+[6.0] - SBPort23
+[6.1] - SBPort23
+
+
+Resource pressure per iteration:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1]
+ - - 6.00 6.00 - 6.00 - -
+
+Resource pressure by instruction:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions:
+ - - - - - - - - xor edx, edx
+ - - 0.01 0.98 - 0.01 - - mov eax, 0
+ - - 0.98 0.01 - 0.01 - - test dil, 3
+ - - - - - 1.00 - - je .LBB5_1
+ - - - - - 1.00 - - ret
+ - - 0.01 0.99 - - - - movabs rax, 9223372036854775804
+ - - - 1.00 - - - - and rsi, rax
+ - - - 1.00 - - - - cmp rsi, 9
+ - - - - - 1.00 - - jae .LBB5_3
+ - - 1.00 - - - - - mov edx, 1
+ - - - - - - - - xor eax, eax
+ - - - - - 1.00 - - ret
+ - - 0.02 0.02 - 0.96 - - add rsi, -9
+ - - 0.99 0.01 - - - - movabs rcx, -6148914691236517205
+ - - 0.01 0.99 - - - - mov rax, rsi
+ - - 1.00 1.00 - - - - mul rcx
+ - - 1.00 - - - - - shr rdx
+ - - 0.98 - - 0.02 - - mov rax, rdi
+ - - - - - 1.00 - - ret
diff --git a/rust/zerocopy/benches/ref_from_prefix_dynamic_size.rs b/rust/zerocopy/benches/ref_from_prefix_dynamic_size.rs
new file mode 100644
index 0000000..a3f26f6
--- /dev/null
+++ b/rust/zerocopy/benches/ref_from_prefix_dynamic_size.rs
@@ -0,0 +1,10 @@
+#[path = "formats/coco_dynamic_size.rs"]
+mod format;
+
+#[unsafe(no_mangle)]
+fn bench_ref_from_prefix_dynamic_size(source: &[u8]) -> Option<&format::LocoPacket> {
+ match zerocopy::FromBytes::ref_from_prefix(source) {
+ Ok((packet, _rest)) => Some(packet),
+ _ => None,
+ }
+}
diff --git a/rust/zerocopy/benches/ref_from_prefix_dynamic_size.x86-64 b/rust/zerocopy/benches/ref_from_prefix_dynamic_size.x86-64
new file mode 100644
index 0000000..fe6332c
--- /dev/null
+++ b/rust/zerocopy/benches/ref_from_prefix_dynamic_size.x86-64
@@ -0,0 +1,17 @@
+bench_ref_from_prefix_dynamic_size:
+ xor edx, edx
+ mov eax, 0
+ test dil, 1
+ jne .LBB5_4
+ cmp rsi, 4
+ jae .LBB5_3
+ mov edx, 1
+ xor eax, eax
+ ret
+.LBB5_3:
+ add rsi, -4
+ shr rsi
+ mov rdx, rsi
+ mov rax, rdi
+.LBB5_4:
+ ret
diff --git a/rust/zerocopy/benches/ref_from_prefix_dynamic_size.x86-64.mca b/rust/zerocopy/benches/ref_from_prefix_dynamic_size.x86-64.mca
new file mode 100644
index 0000000..3900a59
--- /dev/null
+++ b/rust/zerocopy/benches/ref_from_prefix_dynamic_size.x86-64.mca
@@ -0,0 +1,67 @@
+Iterations: 100
+Instructions: 1400
+Total Cycles: 405
+Total uOps: 1400
+
+Dispatch Width: 4
+uOps Per Cycle: 3.46
+IPC: 3.46
+Block RThroughput: 4.0
+
+
+Instruction Info:
+[1]: #uOps
+[2]: Latency
+[3]: RThroughput
+[4]: MayLoad
+[5]: MayStore
+[6]: HasSideEffects (U)
+
+[1] [2] [3] [4] [5] [6] Instructions:
+ 1 0 0.25 xor edx, edx
+ 1 1 0.33 mov eax, 0
+ 1 1 0.33 test dil, 1
+ 1 1 1.00 jne .LBB5_4
+ 1 1 0.33 cmp rsi, 4
+ 1 1 1.00 jae .LBB5_3
+ 1 1 0.33 mov edx, 1
+ 1 0 0.25 xor eax, eax
+ 1 1 1.00 U ret
+ 1 1 0.33 add rsi, -4
+ 1 1 0.50 shr rsi
+ 1 1 0.33 mov rdx, rsi
+ 1 1 0.33 mov rax, rdi
+ 1 1 1.00 U ret
+
+
+Resources:
+[0] - SBDivider
+[1] - SBFPDivider
+[2] - SBPort0
+[3] - SBPort1
+[4] - SBPort4
+[5] - SBPort5
+[6.0] - SBPort23
+[6.1] - SBPort23
+
+
+Resource pressure per iteration:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1]
+ - - 3.99 3.99 - 4.02 - -
+
+Resource pressure by instruction:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions:
+ - - - - - - - - xor edx, edx
+ - - 0.01 0.98 - 0.01 - - mov eax, 0
+ - - 0.98 0.02 - - - - test dil, 1
+ - - - - - 1.00 - - jne .LBB5_4
+ - - 0.02 0.98 - - - - cmp rsi, 4
+ - - - - - 1.00 - - jae .LBB5_3
+ - - 0.98 0.01 - 0.01 - - mov edx, 1
+ - - - - - - - - xor eax, eax
+ - - - - - 1.00 - - ret
+ - - 0.01 0.99 - - - - add rsi, -4
+ - - 1.00 - - - - - shr rsi
+ - - - 1.00 - - - - mov rdx, rsi
+ - - 0.99 0.01 - - - - mov rax, rdi
+ - - - - - 1.00 - - ret
diff --git a/rust/zerocopy/benches/ref_from_prefix_static_size.rs b/rust/zerocopy/benches/ref_from_prefix_static_size.rs
new file mode 100644
index 0000000..834fa39
--- /dev/null
+++ b/rust/zerocopy/benches/ref_from_prefix_static_size.rs
@@ -0,0 +1,10 @@
+#[path = "formats/coco_static_size.rs"]
+mod format;
+
+#[unsafe(no_mangle)]
+fn bench_ref_from_prefix_static_size(source: &[u8]) -> Option<&format::LocoPacket> {
+ match zerocopy::FromBytes::ref_from_prefix(source) {
+ Ok((packet, _rest)) => Some(packet),
+ _ => None,
+ }
+}
diff --git a/rust/zerocopy/benches/ref_from_prefix_static_size.x86-64 b/rust/zerocopy/benches/ref_from_prefix_static_size.x86-64
new file mode 100644
index 0000000..7c1bf45
--- /dev/null
+++ b/rust/zerocopy/benches/ref_from_prefix_static_size.x86-64
@@ -0,0 +1,8 @@
+bench_ref_from_prefix_static_size:
+ xor eax, eax
+ cmp rsi, 6
+ mov rcx, rdi
+ cmovb rcx, rax
+ test dil, 1
+ cmove rax, rcx
+ ret
diff --git a/rust/zerocopy/benches/ref_from_prefix_static_size.x86-64.mca b/rust/zerocopy/benches/ref_from_prefix_static_size.x86-64.mca
new file mode 100644
index 0000000..9691b88
--- /dev/null
+++ b/rust/zerocopy/benches/ref_from_prefix_static_size.x86-64.mca
@@ -0,0 +1,53 @@
+Iterations: 100
+Instructions: 700
+Total Cycles: 274
+Total uOps: 900
+
+Dispatch Width: 4
+uOps Per Cycle: 3.28
+IPC: 2.55
+Block RThroughput: 2.3
+
+
+Instruction Info:
+[1]: #uOps
+[2]: Latency
+[3]: RThroughput
+[4]: MayLoad
+[5]: MayStore
+[6]: HasSideEffects (U)
+
+[1] [2] [3] [4] [5] [6] Instructions:
+ 1 0 0.25 xor eax, eax
+ 1 1 0.33 cmp rsi, 6
+ 1 1 0.33 mov rcx, rdi
+ 2 2 0.67 cmovb rcx, rax
+ 1 1 0.33 test dil, 1
+ 2 2 0.67 cmove rax, rcx
+ 1 1 1.00 U ret
+
+
+Resources:
+[0] - SBDivider
+[1] - SBFPDivider
+[2] - SBPort0
+[3] - SBPort1
+[4] - SBPort4
+[5] - SBPort5
+[6.0] - SBPort23
+[6.1] - SBPort23
+
+
+Resource pressure per iteration:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1]
+ - - 2.66 2.67 - 2.67 - -
+
+Resource pressure by instruction:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions:
+ - - - - - - - - xor eax, eax
+ - - - 0.01 - 0.99 - - cmp rsi, 6
+ - - 0.01 0.67 - 0.32 - - mov rcx, rdi
+ - - 1.00 0.99 - 0.01 - - cmovb rcx, rax
+ - - 0.66 0.01 - 0.33 - - test dil, 1
+ - - 0.99 0.99 - 0.02 - - cmove rax, rcx
+ - - - - - 1.00 - - ret
diff --git a/rust/zerocopy/benches/ref_from_prefix_with_elems_dynamic_padding.rs b/rust/zerocopy/benches/ref_from_prefix_with_elems_dynamic_padding.rs
new file mode 100644
index 0000000..55d495e
--- /dev/null
+++ b/rust/zerocopy/benches/ref_from_prefix_with_elems_dynamic_padding.rs
@@ -0,0 +1,13 @@
+#[path = "formats/coco_dynamic_padding.rs"]
+mod format;
+
+#[unsafe(no_mangle)]
+fn bench_ref_from_prefix_with_elems_dynamic_padding(
+ source: &[u8],
+ count: usize,
+) -> Option<&format::LocoPacket> {
+ match zerocopy::FromBytes::ref_from_prefix_with_elems(source, count) {
+ Ok((packet, _rest)) => Some(packet),
+ _ => None,
+ }
+}
diff --git a/rust/zerocopy/benches/ref_from_prefix_with_elems_dynamic_padding.x86-64 b/rust/zerocopy/benches/ref_from_prefix_with_elems_dynamic_padding.x86-64
new file mode 100644
index 0000000..5b31277b
--- /dev/null
+++ b/rust/zerocopy/benches/ref_from_prefix_with_elems_dynamic_padding.x86-64
@@ -0,0 +1,26 @@
+bench_ref_from_prefix_with_elems_dynamic_padding:
+ movabs rax, 3074457345618258598
+ cmp rdx, rax
+ ja .LBB5_1
+ xor ecx, ecx
+ mov eax, 0
+ test dil, 3
+ je .LBB5_3
+ mov rdx, rcx
+ ret
+.LBB5_1:
+ mov edx, 1
+ xor eax, eax
+ ret
+.LBB5_3:
+ lea rax, [rdx + 2*rdx]
+ or rax, 3
+ add rax, 9
+ xor r8d, r8d
+ cmp rax, rsi
+ mov ecx, 1
+ cmovbe rcx, rdx
+ cmova rdi, r8
+ mov rax, rdi
+ mov rdx, rcx
+ ret
diff --git a/rust/zerocopy/benches/ref_from_prefix_with_elems_dynamic_padding.x86-64.mca b/rust/zerocopy/benches/ref_from_prefix_with_elems_dynamic_padding.x86-64.mca
new file mode 100644
index 0000000..2f212ec
--- /dev/null
+++ b/rust/zerocopy/benches/ref_from_prefix_with_elems_dynamic_padding.x86-64.mca
@@ -0,0 +1,85 @@
+Iterations: 100
+Instructions: 2300
+Total Cycles: 807
+Total uOps: 2700
+
+Dispatch Width: 4
+uOps Per Cycle: 3.35
+IPC: 2.85
+Block RThroughput: 6.8
+
+
+Instruction Info:
+[1]: #uOps
+[2]: Latency
+[3]: RThroughput
+[4]: MayLoad
+[5]: MayStore
+[6]: HasSideEffects (U)
+
+[1] [2] [3] [4] [5] [6] Instructions:
+ 1 1 0.33 movabs rax, 3074457345618258598
+ 1 1 0.33 cmp rdx, rax
+ 1 1 1.00 ja .LBB5_1
+ 1 0 0.25 xor ecx, ecx
+ 1 1 0.33 mov eax, 0
+ 1 1 0.33 test dil, 3
+ 1 1 1.00 je .LBB5_3
+ 1 1 0.33 mov rdx, rcx
+ 1 1 1.00 U ret
+ 1 1 0.33 mov edx, 1
+ 1 0 0.25 xor eax, eax
+ 1 1 1.00 U ret
+ 1 1 0.50 lea rax, [rdx + 2*rdx]
+ 1 1 0.33 or rax, 3
+ 1 1 0.33 add rax, 9
+ 1 0 0.25 xor r8d, r8d
+ 1 1 0.33 cmp rax, rsi
+ 1 1 0.33 mov ecx, 1
+ 3 3 1.00 cmovbe rcx, rdx
+ 3 3 1.00 cmova rdi, r8
+ 1 1 0.33 mov rax, rdi
+ 1 1 0.33 mov rdx, rcx
+ 1 1 1.00 U ret
+
+
+Resources:
+[0] - SBDivider
+[1] - SBFPDivider
+[2] - SBPort0
+[3] - SBPort1
+[4] - SBPort4
+[5] - SBPort5
+[6.0] - SBPort23
+[6.1] - SBPort23
+
+
+Resource pressure per iteration:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1]
+ - - 7.99 7.99 - 8.02 - -
+
+Resource pressure by instruction:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions:
+ - - 0.47 0.52 - 0.01 - - movabs rax, 3074457345618258598
+ - - 0.94 0.01 - 0.05 - - cmp rdx, rax
+ - - - - - 1.00 - - ja .LBB5_1
+ - - - - - - - - xor ecx, ecx
+ - - 0.03 0.97 - - - - mov eax, 0
+ - - 0.01 0.52 - 0.47 - - test dil, 3
+ - - - - - 1.00 - - je .LBB5_3
+ - - 0.03 0.51 - 0.46 - - mov rdx, rcx
+ - - - - - 1.00 - - ret
+ - - 0.04 0.96 - - - - mov edx, 1
+ - - - - - - - - xor eax, eax
+ - - - - - 1.00 - - ret
+ - - 0.01 0.99 - - - - lea rax, [rdx + 2*rdx]
+ - - 0.52 0.48 - - - - or rax, 3
+ - - 0.51 0.49 - - - - add rax, 9
+ - - - - - - - - xor r8d, r8d
+ - - 0.97 0.03 - - - - cmp rax, rsi
+ - - 0.01 0.99 - - - - mov ecx, 1
+ - - 1.04 0.97 - 0.99 - - cmovbe rcx, rdx
+ - - 1.44 0.54 - 1.02 - - cmova rdi, r8
+ - - 0.97 0.01 - 0.02 - - mov rax, rdi
+ - - 1.00 - - - - - mov rdx, rcx
+ - - - - - 1.00 - - ret
diff --git a/rust/zerocopy/benches/ref_from_prefix_with_elems_dynamic_size.rs b/rust/zerocopy/benches/ref_from_prefix_with_elems_dynamic_size.rs
new file mode 100644
index 0000000..e9663c7
--- /dev/null
+++ b/rust/zerocopy/benches/ref_from_prefix_with_elems_dynamic_size.rs
@@ -0,0 +1,13 @@
+#[path = "formats/coco_dynamic_size.rs"]
+mod format;
+
+#[unsafe(no_mangle)]
+fn bench_ref_from_prefix_with_elems_dynamic_size(
+ source: &[u8],
+ count: usize,
+) -> Option<&format::LocoPacket> {
+ match zerocopy::FromBytes::ref_from_prefix_with_elems(source, count) {
+ Ok((packet, _rest)) => Some(packet),
+ _ => None,
+ }
+}
diff --git a/rust/zerocopy/benches/ref_from_prefix_with_elems_dynamic_size.x86-64 b/rust/zerocopy/benches/ref_from_prefix_with_elems_dynamic_size.x86-64
new file mode 100644
index 0000000..069fd48
--- /dev/null
+++ b/rust/zerocopy/benches/ref_from_prefix_with_elems_dynamic_size.x86-64
@@ -0,0 +1,22 @@
+bench_ref_from_prefix_with_elems_dynamic_size:
+ movabs rax, 4611686018427387901
+ cmp rdx, rax
+ ja .LBB5_1
+ mov rcx, rdx
+ xor edx, edx
+ mov eax, 0
+ test dil, 1
+ jne .LBB5_4
+ lea rax, [2*rcx + 4]
+ xor r8d, r8d
+ cmp rax, rsi
+ mov edx, 1
+ cmovbe rdx, rcx
+ cmova rdi, r8
+ mov rax, rdi
+.LBB5_4:
+ ret
+.LBB5_1:
+ mov edx, 1
+ xor eax, eax
+ ret
diff --git a/rust/zerocopy/benches/ref_from_prefix_with_elems_dynamic_size.x86-64.mca b/rust/zerocopy/benches/ref_from_prefix_with_elems_dynamic_size.x86-64.mca
new file mode 100644
index 0000000..6f22726
--- /dev/null
+++ b/rust/zerocopy/benches/ref_from_prefix_with_elems_dynamic_size.x86-64.mca
@@ -0,0 +1,77 @@
+Iterations: 100
+Instructions: 1900
+Total Cycles: 672
+Total uOps: 2300
+
+Dispatch Width: 4
+uOps Per Cycle: 3.42
+IPC: 2.83
+Block RThroughput: 5.8
+
+
+Instruction Info:
+[1]: #uOps
+[2]: Latency
+[3]: RThroughput
+[4]: MayLoad
+[5]: MayStore
+[6]: HasSideEffects (U)
+
+[1] [2] [3] [4] [5] [6] Instructions:
+ 1 1 0.33 movabs rax, 4611686018427387901
+ 1 1 0.33 cmp rdx, rax
+ 1 1 1.00 ja .LBB5_1
+ 1 1 0.33 mov rcx, rdx
+ 1 0 0.25 xor edx, edx
+ 1 1 0.33 mov eax, 0
+ 1 1 0.33 test dil, 1
+ 1 1 1.00 jne .LBB5_4
+ 1 1 0.50 lea rax, [2*rcx + 4]
+ 1 0 0.25 xor r8d, r8d
+ 1 1 0.33 cmp rax, rsi
+ 1 1 0.33 mov edx, 1
+ 3 3 1.00 cmovbe rdx, rcx
+ 3 3 1.00 cmova rdi, r8
+ 1 1 0.33 mov rax, rdi
+ 1 1 1.00 U ret
+ 1 1 0.33 mov edx, 1
+ 1 0 0.25 xor eax, eax
+ 1 1 1.00 U ret
+
+
+Resources:
+[0] - SBDivider
+[1] - SBFPDivider
+[2] - SBPort0
+[3] - SBPort1
+[4] - SBPort4
+[5] - SBPort5
+[6.0] - SBPort23
+[6.1] - SBPort23
+
+
+Resource pressure per iteration:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1]
+ - - 6.66 6.66 - 6.68 - -
+
+Resource pressure by instruction:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions:
+ - - - 0.99 - 0.01 - - movabs rax, 4611686018427387901
+ - - 0.37 0.63 - - - - cmp rdx, rax
+ - - - - - 1.00 - - ja .LBB5_1
+ - - 0.63 0.37 - - - - mov rcx, rdx
+ - - - - - - - - xor edx, edx
+ - - 0.01 0.98 - 0.01 - - mov eax, 0
+ - - 0.98 0.02 - - - - test dil, 1
+ - - - - - 1.00 - - jne .LBB5_4
+ - - 0.01 0.99 - - - - lea rax, [2*rcx + 4]
+ - - - - - - - - xor r8d, r8d
+ - - 1.00 - - - - - cmp rax, rsi
+ - - - 0.67 - 0.33 - - mov edx, 1
+ - - 0.73 0.98 - 1.29 - - cmovbe rdx, rcx
+ - - 1.60 0.36 - 1.04 - - cmova rdi, r8
+ - - 0.99 0.01 - - - - mov rax, rdi
+ - - - - - 1.00 - - ret
+ - - 0.34 0.66 - - - - mov edx, 1
+ - - - - - - - - xor eax, eax
+ - - - - - 1.00 - - ret
diff --git a/rust/zerocopy/benches/ref_from_suffix_dynamic_padding.rs b/rust/zerocopy/benches/ref_from_suffix_dynamic_padding.rs
new file mode 100644
index 0000000..5a6ea3a
--- /dev/null
+++ b/rust/zerocopy/benches/ref_from_suffix_dynamic_padding.rs
@@ -0,0 +1,10 @@
+#[path = "formats/coco_dynamic_padding.rs"]
+mod format;
+
+#[unsafe(no_mangle)]
+fn bench_ref_from_suffix_dynamic_padding(source: &[u8]) -> Option<&format::LocoPacket> {
+ match zerocopy::FromBytes::ref_from_suffix(source) {
+ Ok((_rest, packet)) => Some(packet),
+ _ => None,
+ }
+}
diff --git a/rust/zerocopy/benches/ref_from_suffix_dynamic_padding.x86-64 b/rust/zerocopy/benches/ref_from_suffix_dynamic_padding.x86-64
new file mode 100644
index 0000000..3e05f60
--- /dev/null
+++ b/rust/zerocopy/benches/ref_from_suffix_dynamic_padding.x86-64
@@ -0,0 +1,23 @@
+bench_ref_from_suffix_dynamic_padding:
+ lea eax, [rsi + rdi]
+ test al, 3
+ jne .LBB5_1
+ movabs rax, 9223372036854775804
+ and rax, rsi
+ cmp rax, 9
+ jae .LBB5_3
+.LBB5_1:
+ xor eax, eax
+ ret
+.LBB5_3:
+ add rax, -9
+ movabs rcx, -6148914691236517205
+ mul rcx
+ shr rdx
+ lea rax, [rdx + 2*rdx]
+ sub rsi, rax
+ or rax, -4
+ add rsi, rdi
+ add rax, rsi
+ add rax, -8
+ ret
diff --git a/rust/zerocopy/benches/ref_from_suffix_dynamic_padding.x86-64.mca b/rust/zerocopy/benches/ref_from_suffix_dynamic_padding.x86-64.mca
new file mode 100644
index 0000000..73599d5
--- /dev/null
+++ b/rust/zerocopy/benches/ref_from_suffix_dynamic_padding.x86-64.mca
@@ -0,0 +1,79 @@
+Iterations: 100
+Instructions: 2000
+Total Cycles: 682
+Total uOps: 2100
+
+Dispatch Width: 4
+uOps Per Cycle: 3.08
+IPC: 2.93
+Block RThroughput: 5.3
+
+
+Instruction Info:
+[1]: #uOps
+[2]: Latency
+[3]: RThroughput
+[4]: MayLoad
+[5]: MayStore
+[6]: HasSideEffects (U)
+
+[1] [2] [3] [4] [5] [6] Instructions:
+ 1 1 0.50 lea eax, [rsi + rdi]
+ 1 1 0.33 test al, 3
+ 1 1 1.00 jne .LBB5_1
+ 1 1 0.33 movabs rax, 9223372036854775804
+ 1 1 0.33 and rax, rsi
+ 1 1 0.33 cmp rax, 9
+ 1 1 1.00 jae .LBB5_3
+ 1 0 0.25 xor eax, eax
+ 1 1 1.00 U ret
+ 1 1 0.33 add rax, -9
+ 1 1 0.33 movabs rcx, -6148914691236517205
+ 2 4 1.00 mul rcx
+ 1 1 0.50 shr rdx
+ 1 1 0.50 lea rax, [rdx + 2*rdx]
+ 1 1 0.33 sub rsi, rax
+ 1 1 0.33 or rax, -4
+ 1 1 0.33 add rsi, rdi
+ 1 1 0.33 add rax, rsi
+ 1 1 0.33 add rax, -8
+ 1 1 1.00 U ret
+
+
+Resources:
+[0] - SBDivider
+[1] - SBFPDivider
+[2] - SBPort0
+[3] - SBPort1
+[4] - SBPort4
+[5] - SBPort5
+[6.0] - SBPort23
+[6.1] - SBPort23
+
+
+Resource pressure per iteration:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1]
+ - - 6.65 6.67 - 6.68 - -
+
+Resource pressure by instruction:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions:
+ - - 0.90 0.10 - - - - lea eax, [rsi + rdi]
+ - - 0.93 - - 0.07 - - test al, 3
+ - - - - - 1.00 - - jne .LBB5_1
+ - - 0.51 0.47 - 0.02 - - movabs rax, 9223372036854775804
+ - - - - - 1.00 - - and rax, rsi
+ - - - 0.09 - 0.91 - - cmp rax, 9
+ - - - - - 1.00 - - jae .LBB5_3
+ - - - - - - - - xor eax, eax
+ - - - - - 1.00 - - ret
+ - - 0.43 0.47 - 0.10 - - add rax, -9
+ - - 0.42 0.39 - 0.19 - - movabs rcx, -6148914691236517205
+ - - 1.00 1.00 - - - - mul rcx
+ - - 0.69 - - 0.31 - - shr rdx
+ - - 0.54 0.46 - - - - lea rax, [rdx + 2*rdx]
+ - - 0.07 0.91 - 0.02 - - sub rsi, rax
+ - - 0.91 0.05 - 0.04 - - or rax, -4
+ - - 0.08 0.90 - 0.02 - - add rsi, rdi
+ - - 0.09 0.91 - - - - add rax, rsi
+ - - 0.08 0.92 - - - - add rax, -8
+ - - - - - 1.00 - - ret
diff --git a/rust/zerocopy/benches/ref_from_suffix_dynamic_size.rs b/rust/zerocopy/benches/ref_from_suffix_dynamic_size.rs
new file mode 100644
index 0000000..3437b14
--- /dev/null
+++ b/rust/zerocopy/benches/ref_from_suffix_dynamic_size.rs
@@ -0,0 +1,10 @@
+#[path = "formats/coco_dynamic_size.rs"]
+mod format;
+
+#[unsafe(no_mangle)]
+fn bench_ref_from_suffix_dynamic_size(source: &[u8]) -> Option<&format::LocoPacket> {
+ match zerocopy::FromBytes::ref_from_suffix(source) {
+ Ok((_rest, packet)) => Some(packet),
+ _ => None,
+ }
+}
diff --git a/rust/zerocopy/benches/ref_from_suffix_dynamic_size.x86-64 b/rust/zerocopy/benches/ref_from_suffix_dynamic_size.x86-64
new file mode 100644
index 0000000..bd4ace8
--- /dev/null
+++ b/rust/zerocopy/benches/ref_from_suffix_dynamic_size.x86-64
@@ -0,0 +1,13 @@
+bench_ref_from_suffix_dynamic_size:
+ mov rdx, rsi
+ lea ecx, [rsi + rdi]
+ mov eax, edx
+ and eax, 1
+ add rax, rdi
+ xor esi, esi
+ sub rdx, 4
+ cmovb rax, rsi
+ shr rdx
+ test cl, 1
+ cmovne rax, rsi
+ ret
diff --git a/rust/zerocopy/benches/ref_from_suffix_dynamic_size.x86-64.mca b/rust/zerocopy/benches/ref_from_suffix_dynamic_size.x86-64.mca
new file mode 100644
index 0000000..1398bcf
--- /dev/null
+++ b/rust/zerocopy/benches/ref_from_suffix_dynamic_size.x86-64.mca
@@ -0,0 +1,63 @@
+Iterations: 100
+Instructions: 1200
+Total Cycles: 439
+Total uOps: 1400
+
+Dispatch Width: 4
+uOps Per Cycle: 3.19
+IPC: 2.73
+Block RThroughput: 3.5
+
+
+Instruction Info:
+[1]: #uOps
+[2]: Latency
+[3]: RThroughput
+[4]: MayLoad
+[5]: MayStore
+[6]: HasSideEffects (U)
+
+[1] [2] [3] [4] [5] [6] Instructions:
+ 1 1 0.33 mov rdx, rsi
+ 1 1 0.50 lea ecx, [rsi + rdi]
+ 1 1 0.33 mov eax, edx
+ 1 1 0.33 and eax, 1
+ 1 1 0.33 add rax, rdi
+ 1 0 0.25 xor esi, esi
+ 1 1 0.33 sub rdx, 4
+ 2 2 0.67 cmovb rax, rsi
+ 1 1 0.50 shr rdx
+ 1 1 0.33 test cl, 1
+ 2 2 0.67 cmovne rax, rsi
+ 1 1 1.00 U ret
+
+
+Resources:
+[0] - SBDivider
+[1] - SBFPDivider
+[2] - SBPort0
+[3] - SBPort1
+[4] - SBPort4
+[5] - SBPort5
+[6.0] - SBPort23
+[6.1] - SBPort23
+
+
+Resource pressure per iteration:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1]
+ - - 4.33 4.33 - 4.34 - -
+
+Resource pressure by instruction:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions:
+ - - 0.02 0.32 - 0.66 - - mov rdx, rsi
+ - - 0.32 0.68 - - - - lea ecx, [rsi + rdi]
+ - - 0.66 - - 0.34 - - mov eax, edx
+ - - 0.02 0.33 - 0.65 - - and eax, 1
+ - - - 0.99 - 0.01 - - add rax, rdi
+ - - - - - - - - xor esi, esi
+ - - 0.65 - - 0.35 - - sub rdx, 4
+ - - 1.00 1.00 - - - - cmovb rax, rsi
+ - - 0.66 - - 0.34 - - shr rdx
+ - - - 0.01 - 0.99 - - test cl, 1
+ - - 1.00 1.00 - - - - cmovne rax, rsi
+ - - - - - 1.00 - - ret
diff --git a/rust/zerocopy/benches/ref_from_suffix_static_size.rs b/rust/zerocopy/benches/ref_from_suffix_static_size.rs
new file mode 100644
index 0000000..c8435d1
--- /dev/null
+++ b/rust/zerocopy/benches/ref_from_suffix_static_size.rs
@@ -0,0 +1,10 @@
+#[path = "formats/coco_static_size.rs"]
+mod format;
+
+#[unsafe(no_mangle)]
+fn bench_ref_from_suffix_static_size(source: &[u8]) -> Option<&format::LocoPacket> {
+ match zerocopy::FromBytes::ref_from_suffix(source) {
+ Ok((_rest, packet)) => Some(packet),
+ _ => None,
+ }
+}
diff --git a/rust/zerocopy/benches/ref_from_suffix_static_size.x86-64 b/rust/zerocopy/benches/ref_from_suffix_static_size.x86-64
new file mode 100644
index 0000000..9e90b9e
--- /dev/null
+++ b/rust/zerocopy/benches/ref_from_suffix_static_size.x86-64
@@ -0,0 +1,13 @@
+bench_ref_from_suffix_static_size:
+ lea eax, [rsi + rdi]
+ cmp rsi, 6
+ setb cl
+ or cl, al
+ test cl, 1
+ je .LBB5_2
+ xor eax, eax
+ ret
+.LBB5_2:
+ lea rax, [rdi + rsi]
+ add rax, -6
+ ret
diff --git a/rust/zerocopy/benches/ref_from_suffix_static_size.x86-64.mca b/rust/zerocopy/benches/ref_from_suffix_static_size.x86-64.mca
new file mode 100644
index 0000000..ef58926
--- /dev/null
+++ b/rust/zerocopy/benches/ref_from_suffix_static_size.x86-64.mca
@@ -0,0 +1,61 @@
+Iterations: 100
+Instructions: 1100
+Total Cycles: 338
+Total uOps: 1100
+
+Dispatch Width: 4
+uOps Per Cycle: 3.25
+IPC: 3.25
+Block RThroughput: 3.0
+
+
+Instruction Info:
+[1]: #uOps
+[2]: Latency
+[3]: RThroughput
+[4]: MayLoad
+[5]: MayStore
+[6]: HasSideEffects (U)
+
+[1] [2] [3] [4] [5] [6] Instructions:
+ 1 1 0.50 lea eax, [rsi + rdi]
+ 1 1 0.33 cmp rsi, 6
+ 1 1 0.50 setb cl
+ 1 1 0.33 or cl, al
+ 1 1 0.33 test cl, 1
+ 1 1 1.00 je .LBB5_2
+ 1 0 0.25 xor eax, eax
+ 1 1 1.00 U ret
+ 1 1 0.50 lea rax, [rdi + rsi]
+ 1 1 0.33 add rax, -6
+ 1 1 1.00 U ret
+
+
+Resources:
+[0] - SBDivider
+[1] - SBFPDivider
+[2] - SBPort0
+[3] - SBPort1
+[4] - SBPort4
+[5] - SBPort5
+[6.0] - SBPort23
+[6.1] - SBPort23
+
+
+Resource pressure per iteration:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1]
+ - - 3.32 3.33 - 3.35 - -
+
+Resource pressure by instruction:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions:
+ - - 0.97 0.03 - - - - lea eax, [rsi + rdi]
+ - - 0.33 0.32 - 0.35 - - cmp rsi, 6
+ - - 1.00 - - - - - setb cl
+ - - - 1.00 - - - - or cl, al
+ - - - 1.00 - - - - test cl, 1
+ - - - - - 1.00 - - je .LBB5_2
+ - - - - - - - - xor eax, eax
+ - - - - - 1.00 - - ret
+ - - 0.34 0.66 - - - - lea rax, [rdi + rsi]
+ - - 0.68 0.32 - - - - add rax, -6
+ - - - - - 1.00 - - ret
diff --git a/rust/zerocopy/benches/ref_from_suffix_with_elems_dynamic_padding.rs b/rust/zerocopy/benches/ref_from_suffix_with_elems_dynamic_padding.rs
new file mode 100644
index 0000000..73d91ce
--- /dev/null
+++ b/rust/zerocopy/benches/ref_from_suffix_with_elems_dynamic_padding.rs
@@ -0,0 +1,13 @@
+#[path = "formats/coco_dynamic_padding.rs"]
+mod format;
+
+#[unsafe(no_mangle)]
+fn bench_ref_from_suffix_with_elems_dynamic_padding(
+ source: &[u8],
+ count: usize,
+) -> Option<&format::LocoPacket> {
+ match zerocopy::FromBytes::ref_from_suffix_with_elems(source, count) {
+ Ok((_rest, packet)) => Some(packet),
+ _ => None,
+ }
+}
diff --git a/rust/zerocopy/benches/ref_from_suffix_with_elems_dynamic_padding.x86-64 b/rust/zerocopy/benches/ref_from_suffix_with_elems_dynamic_padding.x86-64
new file mode 100644
index 0000000..c3d10b5
--- /dev/null
+++ b/rust/zerocopy/benches/ref_from_suffix_with_elems_dynamic_padding.x86-64
@@ -0,0 +1,27 @@
+bench_ref_from_suffix_with_elems_dynamic_padding:
+ movabs rax, 3074457345618258598
+ cmp rdx, rax
+ ja .LBB5_1
+ lea r8d, [rsi + rdi]
+ xor ecx, ecx
+ mov eax, 0
+ test r8b, 3
+ je .LBB5_3
+ mov rdx, rcx
+ ret
+.LBB5_3:
+ lea rax, [rdx + 2*rdx]
+ or rax, 3
+ add rax, 9
+ sub rsi, rax
+ jae .LBB5_4
+.LBB5_1:
+ xor eax, eax
+ mov edx, 1
+ ret
+.LBB5_4:
+ add rdi, rsi
+ mov rcx, rdx
+ mov rax, rdi
+ mov rdx, rcx
+ ret
diff --git a/rust/zerocopy/benches/ref_from_suffix_with_elems_dynamic_padding.x86-64.mca b/rust/zerocopy/benches/ref_from_suffix_with_elems_dynamic_padding.x86-64.mca
new file mode 100644
index 0000000..92e6280
--- /dev/null
+++ b/rust/zerocopy/benches/ref_from_suffix_with_elems_dynamic_padding.x86-64.mca
@@ -0,0 +1,85 @@
+Iterations: 100
+Instructions: 2300
+Total Cycles: 706
+Total uOps: 2300
+
+Dispatch Width: 4
+uOps Per Cycle: 3.26
+IPC: 3.26
+Block RThroughput: 6.0
+
+
+Instruction Info:
+[1]: #uOps
+[2]: Latency
+[3]: RThroughput
+[4]: MayLoad
+[5]: MayStore
+[6]: HasSideEffects (U)
+
+[1] [2] [3] [4] [5] [6] Instructions:
+ 1 1 0.33 movabs rax, 3074457345618258598
+ 1 1 0.33 cmp rdx, rax
+ 1 1 1.00 ja .LBB5_1
+ 1 1 0.50 lea r8d, [rsi + rdi]
+ 1 0 0.25 xor ecx, ecx
+ 1 1 0.33 mov eax, 0
+ 1 1 0.33 test r8b, 3
+ 1 1 1.00 je .LBB5_3
+ 1 1 0.33 mov rdx, rcx
+ 1 1 1.00 U ret
+ 1 1 0.50 lea rax, [rdx + 2*rdx]
+ 1 1 0.33 or rax, 3
+ 1 1 0.33 add rax, 9
+ 1 1 0.33 sub rsi, rax
+ 1 1 1.00 jae .LBB5_4
+ 1 0 0.25 xor eax, eax
+ 1 1 0.33 mov edx, 1
+ 1 1 1.00 U ret
+ 1 1 0.33 add rdi, rsi
+ 1 1 0.33 mov rcx, rdx
+ 1 1 0.33 mov rax, rdi
+ 1 1 0.33 mov rdx, rcx
+ 1 1 1.00 U ret
+
+
+Resources:
+[0] - SBDivider
+[1] - SBFPDivider
+[2] - SBPort0
+[3] - SBPort1
+[4] - SBPort4
+[5] - SBPort5
+[6.0] - SBPort23
+[6.1] - SBPort23
+
+
+Resource pressure per iteration:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1]
+ - - 6.99 7.00 - 7.01 - -
+
+Resource pressure by instruction:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions:
+ - - - 0.99 - 0.01 - - movabs rax, 3074457345618258598
+ - - 0.01 0.50 - 0.49 - - cmp rdx, rax
+ - - - - - 1.00 - - ja .LBB5_1
+ - - - 1.00 - - - - lea r8d, [rsi + rdi]
+ - - - - - - - - xor ecx, ecx
+ - - 0.50 0.49 - 0.01 - - mov eax, 0
+ - - 0.49 0.51 - - - - test r8b, 3
+ - - - - - 1.00 - - je .LBB5_3
+ - - 0.51 0.49 - - - - mov rdx, rcx
+ - - - - - 1.00 - - ret
+ - - 0.50 0.50 - - - - lea rax, [rdx + 2*rdx]
+ - - 1.00 - - - - - or rax, 3
+ - - 1.00 - - - - - add rax, 9
+ - - 0.99 0.01 - - - - sub rsi, rax
+ - - - - - 1.00 - - jae .LBB5_4
+ - - - - - - - - xor eax, eax
+ - - - 1.00 - - - - mov edx, 1
+ - - - - - 1.00 - - ret
+ - - 1.00 - - - - - add rdi, rsi
+ - - - 1.00 - - - - mov rcx, rdx
+ - - 0.99 0.01 - - - - mov rax, rdi
+ - - - 0.50 - 0.50 - - mov rdx, rcx
+ - - - - - 1.00 - - ret
diff --git a/rust/zerocopy/benches/ref_from_suffix_with_elems_dynamic_size.rs b/rust/zerocopy/benches/ref_from_suffix_with_elems_dynamic_size.rs
new file mode 100644
index 0000000..68a28ba
--- /dev/null
+++ b/rust/zerocopy/benches/ref_from_suffix_with_elems_dynamic_size.rs
@@ -0,0 +1,13 @@
+#[path = "formats/coco_dynamic_size.rs"]
+mod format;
+
+#[unsafe(no_mangle)]
+fn bench_ref_from_suffix_with_elems_dynamic_size(
+ source: &[u8],
+ count: usize,
+) -> Option<&format::LocoPacket> {
+ match zerocopy::FromBytes::ref_from_suffix_with_elems(source, count) {
+ Ok((_rest, packet)) => Some(packet),
+ _ => None,
+ }
+}
diff --git a/rust/zerocopy/benches/ref_from_suffix_with_elems_dynamic_size.x86-64 b/rust/zerocopy/benches/ref_from_suffix_with_elems_dynamic_size.x86-64
new file mode 100644
index 0000000..bdca571
--- /dev/null
+++ b/rust/zerocopy/benches/ref_from_suffix_with_elems_dynamic_size.x86-64
@@ -0,0 +1,23 @@
+bench_ref_from_suffix_with_elems_dynamic_size:
+ movabs rax, 4611686018427387901
+ cmp rdx, rax
+ ja .LBB5_1
+ lea r8d, [rsi + rdi]
+ xor ecx, ecx
+ mov eax, 0
+ test r8b, 1
+ jne .LBB5_5
+ lea rax, [2*rdx + 4]
+ sub rsi, rax
+ jae .LBB5_4
+.LBB5_1:
+ xor eax, eax
+ mov edx, 1
+ ret
+.LBB5_4:
+ add rdi, rsi
+ mov rcx, rdx
+ mov rax, rdi
+.LBB5_5:
+ mov rdx, rcx
+ ret
diff --git a/rust/zerocopy/benches/ref_from_suffix_with_elems_dynamic_size.x86-64.mca b/rust/zerocopy/benches/ref_from_suffix_with_elems_dynamic_size.x86-64.mca
new file mode 100644
index 0000000..6d9de0b
--- /dev/null
+++ b/rust/zerocopy/benches/ref_from_suffix_with_elems_dynamic_size.x86-64.mca
@@ -0,0 +1,77 @@
+Iterations: 100
+Instructions: 1900
+Total Cycles: 571
+Total uOps: 1900
+
+Dispatch Width: 4
+uOps Per Cycle: 3.33
+IPC: 3.33
+Block RThroughput: 5.0
+
+
+Instruction Info:
+[1]: #uOps
+[2]: Latency
+[3]: RThroughput
+[4]: MayLoad
+[5]: MayStore
+[6]: HasSideEffects (U)
+
+[1] [2] [3] [4] [5] [6] Instructions:
+ 1 1 0.33 movabs rax, 4611686018427387901
+ 1 1 0.33 cmp rdx, rax
+ 1 1 1.00 ja .LBB5_1
+ 1 1 0.50 lea r8d, [rsi + rdi]
+ 1 0 0.25 xor ecx, ecx
+ 1 1 0.33 mov eax, 0
+ 1 1 0.33 test r8b, 1
+ 1 1 1.00 jne .LBB5_5
+ 1 1 0.50 lea rax, [2*rdx + 4]
+ 1 1 0.33 sub rsi, rax
+ 1 1 1.00 jae .LBB5_4
+ 1 0 0.25 xor eax, eax
+ 1 1 0.33 mov edx, 1
+ 1 1 1.00 U ret
+ 1 1 0.33 add rdi, rsi
+ 1 1 0.33 mov rcx, rdx
+ 1 1 0.33 mov rax, rdi
+ 1 1 0.33 mov rdx, rcx
+ 1 1 1.00 U ret
+
+
+Resources:
+[0] - SBDivider
+[1] - SBFPDivider
+[2] - SBPort0
+[3] - SBPort1
+[4] - SBPort4
+[5] - SBPort5
+[6.0] - SBPort23
+[6.1] - SBPort23
+
+
+Resource pressure per iteration:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1]
+ - - 5.66 5.66 - 5.68 - -
+
+Resource pressure by instruction:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions:
+ - - 0.66 0.33 - 0.01 - - movabs rax, 4611686018427387901
+ - - 0.01 0.99 - - - - cmp rdx, rax
+ - - - - - 1.00 - - ja .LBB5_1
+ - - 0.99 0.01 - - - - lea r8d, [rsi + rdi]
+ - - - - - - - - xor ecx, ecx
+ - - 0.33 0.33 - 0.34 - - mov eax, 0
+ - - 0.33 0.34 - 0.33 - - test r8b, 1
+ - - - - - 1.00 - - jne .LBB5_5
+ - - 0.34 0.66 - - - - lea rax, [2*rdx + 4]
+ - - - 1.00 - - - - sub rsi, rax
+ - - - - - 1.00 - - jae .LBB5_4
+ - - - - - - - - xor eax, eax
+ - - 1.00 - - - - - mov edx, 1
+ - - - - - 1.00 - - ret
+ - - - 1.00 - - - - add rdi, rsi
+ - - 1.00 - - - - - mov rcx, rdx
+ - - 0.32 0.68 - - - - mov rax, rdi
+ - - 0.68 0.32 - - - - mov rdx, rcx
+ - - - - - 1.00 - - ret
diff --git a/rust/zerocopy/benches/split_at_dynamic_padding.rs b/rust/zerocopy/benches/split_at_dynamic_padding.rs
new file mode 100644
index 0000000..bed90f6
--- /dev/null
+++ b/rust/zerocopy/benches/split_at_dynamic_padding.rs
@@ -0,0 +1,12 @@
+use zerocopy::*;
+
+#[path = "formats/coco_dynamic_padding.rs"]
+mod format;
+
+#[unsafe(no_mangle)]
+fn bench_split_at_dynamic_padding(
+ source: &format::CocoPacket,
+ len: usize,
+) -> Option<Split<&format::CocoPacket>> {
+ source.split_at(len)
+}
diff --git a/rust/zerocopy/benches/split_at_dynamic_padding.x86-64 b/rust/zerocopy/benches/split_at_dynamic_padding.x86-64
new file mode 100644
index 0000000..6eaf5a0
--- /dev/null
+++ b/rust/zerocopy/benches/split_at_dynamic_padding.x86-64
@@ -0,0 +1,12 @@
+bench_split_at_dynamic_padding:
+ mov rax, rdi
+ cmp rcx, rdx
+ jbe .LBB5_2
+ xor esi, esi
+ mov qword ptr [rax], rsi
+ ret
+.LBB5_2:
+ mov qword ptr [rax + 8], rdx
+ mov qword ptr [rax + 16], rcx
+ mov qword ptr [rax], rsi
+ ret
diff --git a/rust/zerocopy/benches/split_at_dynamic_padding.x86-64.mca b/rust/zerocopy/benches/split_at_dynamic_padding.x86-64.mca
new file mode 100644
index 0000000..19ab341
--- /dev/null
+++ b/rust/zerocopy/benches/split_at_dynamic_padding.x86-64.mca
@@ -0,0 +1,59 @@
+Iterations: 100
+Instructions: 1000
+Total Cycles: 404
+Total uOps: 1000
+
+Dispatch Width: 4
+uOps Per Cycle: 2.48
+IPC: 2.48
+Block RThroughput: 4.0
+
+
+Instruction Info:
+[1]: #uOps
+[2]: Latency
+[3]: RThroughput
+[4]: MayLoad
+[5]: MayStore
+[6]: HasSideEffects (U)
+
+[1] [2] [3] [4] [5] [6] Instructions:
+ 1 1 0.33 mov rax, rdi
+ 1 1 0.33 cmp rcx, rdx
+ 1 1 1.00 jbe .LBB5_2
+ 1 0 0.25 xor esi, esi
+ 1 1 1.00 * mov qword ptr [rax], rsi
+ 1 1 1.00 U ret
+ 1 1 1.00 * mov qword ptr [rax + 8], rdx
+ 1 1 1.00 * mov qword ptr [rax + 16], rcx
+ 1 1 1.00 * mov qword ptr [rax], rsi
+ 1 1 1.00 U ret
+
+
+Resources:
+[0] - SBDivider
+[1] - SBFPDivider
+[2] - SBPort0
+[3] - SBPort1
+[4] - SBPort4
+[5] - SBPort5
+[6.0] - SBPort23
+[6.1] - SBPort23
+
+
+Resource pressure per iteration:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1]
+ - - 0.99 1.00 4.00 3.01 2.00 2.00
+
+Resource pressure by instruction:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions:
+ - - 0.99 - - 0.01 - - mov rax, rdi
+ - - - 1.00 - - - - cmp rcx, rdx
+ - - - - - 1.00 - - jbe .LBB5_2
+ - - - - - - - - xor esi, esi
+ - - - - 1.00 - - 1.00 mov qword ptr [rax], rsi
+ - - - - - 1.00 - - ret
+ - - - - 1.00 - 1.00 - mov qword ptr [rax + 8], rdx
+ - - - - 1.00 - - 1.00 mov qword ptr [rax + 16], rcx
+ - - - - 1.00 - 1.00 - mov qword ptr [rax], rsi
+ - - - - - 1.00 - - ret
diff --git a/rust/zerocopy/benches/split_at_dynamic_size.rs b/rust/zerocopy/benches/split_at_dynamic_size.rs
new file mode 100644
index 0000000..07a22ba
--- /dev/null
+++ b/rust/zerocopy/benches/split_at_dynamic_size.rs
@@ -0,0 +1,12 @@
+use zerocopy::*;
+
+#[path = "formats/coco_dynamic_size.rs"]
+mod format;
+
+#[unsafe(no_mangle)]
+fn bench_split_at_dynamic_size(
+ source: &format::CocoPacket,
+ len: usize,
+) -> Option<Split<&format::CocoPacket>> {
+ source.split_at(len)
+}
diff --git a/rust/zerocopy/benches/split_at_dynamic_size.x86-64 b/rust/zerocopy/benches/split_at_dynamic_size.x86-64
new file mode 100644
index 0000000..8d81b98
--- /dev/null
+++ b/rust/zerocopy/benches/split_at_dynamic_size.x86-64
@@ -0,0 +1,12 @@
+bench_split_at_dynamic_size:
+ mov rax, rdi
+ cmp rcx, rdx
+ jbe .LBB5_2
+ xor esi, esi
+ mov qword ptr [rax], rsi
+ ret
+.LBB5_2:
+ mov qword ptr [rax + 8], rdx
+ mov qword ptr [rax + 16], rcx
+ mov qword ptr [rax], rsi
+ ret
diff --git a/rust/zerocopy/benches/split_at_dynamic_size.x86-64.mca b/rust/zerocopy/benches/split_at_dynamic_size.x86-64.mca
new file mode 100644
index 0000000..19ab341
--- /dev/null
+++ b/rust/zerocopy/benches/split_at_dynamic_size.x86-64.mca
@@ -0,0 +1,59 @@
+Iterations: 100
+Instructions: 1000
+Total Cycles: 404
+Total uOps: 1000
+
+Dispatch Width: 4
+uOps Per Cycle: 2.48
+IPC: 2.48
+Block RThroughput: 4.0
+
+
+Instruction Info:
+[1]: #uOps
+[2]: Latency
+[3]: RThroughput
+[4]: MayLoad
+[5]: MayStore
+[6]: HasSideEffects (U)
+
+[1] [2] [3] [4] [5] [6] Instructions:
+ 1 1 0.33 mov rax, rdi
+ 1 1 0.33 cmp rcx, rdx
+ 1 1 1.00 jbe .LBB5_2
+ 1 0 0.25 xor esi, esi
+ 1 1 1.00 * mov qword ptr [rax], rsi
+ 1 1 1.00 U ret
+ 1 1 1.00 * mov qword ptr [rax + 8], rdx
+ 1 1 1.00 * mov qword ptr [rax + 16], rcx
+ 1 1 1.00 * mov qword ptr [rax], rsi
+ 1 1 1.00 U ret
+
+
+Resources:
+[0] - SBDivider
+[1] - SBFPDivider
+[2] - SBPort0
+[3] - SBPort1
+[4] - SBPort4
+[5] - SBPort5
+[6.0] - SBPort23
+[6.1] - SBPort23
+
+
+Resource pressure per iteration:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1]
+ - - 0.99 1.00 4.00 3.01 2.00 2.00
+
+Resource pressure by instruction:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions:
+ - - 0.99 - - 0.01 - - mov rax, rdi
+ - - - 1.00 - - - - cmp rcx, rdx
+ - - - - - 1.00 - - jbe .LBB5_2
+ - - - - - - - - xor esi, esi
+ - - - - 1.00 - - 1.00 mov qword ptr [rax], rsi
+ - - - - - 1.00 - - ret
+ - - - - 1.00 - 1.00 - mov qword ptr [rax + 8], rdx
+ - - - - 1.00 - - 1.00 mov qword ptr [rax + 16], rcx
+ - - - - 1.00 - 1.00 - mov qword ptr [rax], rsi
+ - - - - - 1.00 - - ret
diff --git a/rust/zerocopy/benches/split_at_unchecked_dynamic_padding.rs b/rust/zerocopy/benches/split_at_unchecked_dynamic_padding.rs
new file mode 100644
index 0000000..3c147d3
--- /dev/null
+++ b/rust/zerocopy/benches/split_at_unchecked_dynamic_padding.rs
@@ -0,0 +1,12 @@
+use zerocopy::*;
+
+#[path = "formats/coco_dynamic_padding.rs"]
+mod format;
+
+#[unsafe(no_mangle)]
+unsafe fn bench_split_at_unchecked_dynamic_padding(
+ source: &format::CocoPacket,
+ len: usize,
+) -> Split<&format::CocoPacket> {
+ unsafe { source.split_at_unchecked(len) }
+}
diff --git a/rust/zerocopy/benches/split_at_unchecked_dynamic_padding.x86-64 b/rust/zerocopy/benches/split_at_unchecked_dynamic_padding.x86-64
new file mode 100644
index 0000000..74c3b52f
--- /dev/null
+++ b/rust/zerocopy/benches/split_at_unchecked_dynamic_padding.x86-64
@@ -0,0 +1,6 @@
+bench_split_at_unchecked_dynamic_padding:
+ mov rax, rdi
+ mov qword ptr [rdi], rsi
+ mov qword ptr [rdi + 8], rdx
+ mov qword ptr [rdi + 16], rcx
+ ret
diff --git a/rust/zerocopy/benches/split_at_unchecked_dynamic_padding.x86-64.mca b/rust/zerocopy/benches/split_at_unchecked_dynamic_padding.x86-64.mca
new file mode 100644
index 0000000..e8c6159
--- /dev/null
+++ b/rust/zerocopy/benches/split_at_unchecked_dynamic_padding.x86-64.mca
@@ -0,0 +1,49 @@
+Iterations: 100
+Instructions: 500
+Total Cycles: 303
+Total uOps: 500
+
+Dispatch Width: 4
+uOps Per Cycle: 1.65
+IPC: 1.65
+Block RThroughput: 3.0
+
+
+Instruction Info:
+[1]: #uOps
+[2]: Latency
+[3]: RThroughput
+[4]: MayLoad
+[5]: MayStore
+[6]: HasSideEffects (U)
+
+[1] [2] [3] [4] [5] [6] Instructions:
+ 1 1 0.33 mov rax, rdi
+ 1 1 1.00 * mov qword ptr [rdi], rsi
+ 1 1 1.00 * mov qword ptr [rdi + 8], rdx
+ 1 1 1.00 * mov qword ptr [rdi + 16], rcx
+ 1 1 1.00 U ret
+
+
+Resources:
+[0] - SBDivider
+[1] - SBFPDivider
+[2] - SBPort0
+[3] - SBPort1
+[4] - SBPort4
+[5] - SBPort5
+[6.0] - SBPort23
+[6.1] - SBPort23
+
+
+Resource pressure per iteration:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1]
+ - - 0.49 0.50 3.00 1.01 1.50 1.50
+
+Resource pressure by instruction:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions:
+ - - 0.49 0.50 - 0.01 - - mov rax, rdi
+ - - - - 1.00 - 0.50 0.50 mov qword ptr [rdi], rsi
+ - - - - 1.00 - 0.50 0.50 mov qword ptr [rdi + 8], rdx
+ - - - - 1.00 - 0.50 0.50 mov qword ptr [rdi + 16], rcx
+ - - - - - 1.00 - - ret
diff --git a/rust/zerocopy/benches/split_at_unchecked_dynamic_size.rs b/rust/zerocopy/benches/split_at_unchecked_dynamic_size.rs
new file mode 100644
index 0000000..b1aa1df
--- /dev/null
+++ b/rust/zerocopy/benches/split_at_unchecked_dynamic_size.rs
@@ -0,0 +1,12 @@
+use zerocopy::*;
+
+#[path = "formats/coco_dynamic_size.rs"]
+mod format;
+
+#[unsafe(no_mangle)]
+unsafe fn bench_split_at_unchecked_dynamic_size(
+ source: &format::CocoPacket,
+ len: usize,
+) -> Split<&format::CocoPacket> {
+ unsafe { source.split_at_unchecked(len) }
+}
diff --git a/rust/zerocopy/benches/split_at_unchecked_dynamic_size.x86-64 b/rust/zerocopy/benches/split_at_unchecked_dynamic_size.x86-64
new file mode 100644
index 0000000..56671d1
--- /dev/null
+++ b/rust/zerocopy/benches/split_at_unchecked_dynamic_size.x86-64
@@ -0,0 +1,6 @@
+bench_split_at_unchecked_dynamic_size:
+ mov rax, rdi
+ mov qword ptr [rdi], rsi
+ mov qword ptr [rdi + 8], rdx
+ mov qword ptr [rdi + 16], rcx
+ ret
diff --git a/rust/zerocopy/benches/split_at_unchecked_dynamic_size.x86-64.mca b/rust/zerocopy/benches/split_at_unchecked_dynamic_size.x86-64.mca
new file mode 100644
index 0000000..e8c6159
--- /dev/null
+++ b/rust/zerocopy/benches/split_at_unchecked_dynamic_size.x86-64.mca
@@ -0,0 +1,49 @@
+Iterations: 100
+Instructions: 500
+Total Cycles: 303
+Total uOps: 500
+
+Dispatch Width: 4
+uOps Per Cycle: 1.65
+IPC: 1.65
+Block RThroughput: 3.0
+
+
+Instruction Info:
+[1]: #uOps
+[2]: Latency
+[3]: RThroughput
+[4]: MayLoad
+[5]: MayStore
+[6]: HasSideEffects (U)
+
+[1] [2] [3] [4] [5] [6] Instructions:
+ 1 1 0.33 mov rax, rdi
+ 1 1 1.00 * mov qword ptr [rdi], rsi
+ 1 1 1.00 * mov qword ptr [rdi + 8], rdx
+ 1 1 1.00 * mov qword ptr [rdi + 16], rcx
+ 1 1 1.00 U ret
+
+
+Resources:
+[0] - SBDivider
+[1] - SBFPDivider
+[2] - SBPort0
+[3] - SBPort1
+[4] - SBPort4
+[5] - SBPort5
+[6.0] - SBPort23
+[6.1] - SBPort23
+
+
+Resource pressure per iteration:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1]
+ - - 0.49 0.50 3.00 1.01 1.50 1.50
+
+Resource pressure by instruction:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions:
+ - - 0.49 0.50 - 0.01 - - mov rax, rdi
+ - - - - 1.00 - 0.50 0.50 mov qword ptr [rdi], rsi
+ - - - - 1.00 - 0.50 0.50 mov qword ptr [rdi + 8], rdx
+ - - - - 1.00 - 0.50 0.50 mov qword ptr [rdi + 16], rcx
+ - - - - - 1.00 - - ret
diff --git a/rust/zerocopy/benches/split_via_immutable_dynamic_padding.rs b/rust/zerocopy/benches/split_via_immutable_dynamic_padding.rs
new file mode 100644
index 0000000..b86ad36
--- /dev/null
+++ b/rust/zerocopy/benches/split_via_immutable_dynamic_padding.rs
@@ -0,0 +1,11 @@
+use zerocopy::*;
+
+#[path = "formats/coco_dynamic_padding.rs"]
+mod format;
+
+#[unsafe(no_mangle)]
+fn bench_split_via_immutable_dynamic_padding(
+ split: Split<&format::CocoPacket>,
+) -> (&format::CocoPacket, &[[u8; 3]]) {
+ split.via_immutable()
+}
diff --git a/rust/zerocopy/benches/split_via_immutable_dynamic_padding.x86-64 b/rust/zerocopy/benches/split_via_immutable_dynamic_padding.x86-64
new file mode 100644
index 0000000..dac1834
--- /dev/null
+++ b/rust/zerocopy/benches/split_via_immutable_dynamic_padding.x86-64
@@ -0,0 +1,14 @@
+bench_split_via_immutable_dynamic_padding:
+ mov rax, rdi
+ mov rcx, qword ptr [rsi]
+ mov rdx, qword ptr [rsi + 8]
+ mov rsi, qword ptr [rsi + 16]
+ lea rdi, [rsi + 2*rsi]
+ add rdi, rcx
+ add rdi, 9
+ sub rdx, rsi
+ mov qword ptr [rax], rcx
+ mov qword ptr [rax + 8], rsi
+ mov qword ptr [rax + 16], rdi
+ mov qword ptr [rax + 24], rdx
+ ret
diff --git a/rust/zerocopy/benches/split_via_immutable_dynamic_padding.x86-64.mca b/rust/zerocopy/benches/split_via_immutable_dynamic_padding.x86-64.mca
new file mode 100644
index 0000000..6ab4e83
--- /dev/null
+++ b/rust/zerocopy/benches/split_via_immutable_dynamic_padding.x86-64.mca
@@ -0,0 +1,65 @@
+Iterations: 100
+Instructions: 1300
+Total Cycles: 510
+Total uOps: 1300
+
+Dispatch Width: 4
+uOps Per Cycle: 2.55
+IPC: 2.55
+Block RThroughput: 4.0
+
+
+Instruction Info:
+[1]: #uOps
+[2]: Latency
+[3]: RThroughput
+[4]: MayLoad
+[5]: MayStore
+[6]: HasSideEffects (U)
+
+[1] [2] [3] [4] [5] [6] Instructions:
+ 1 1 0.33 mov rax, rdi
+ 1 5 0.50 * mov rcx, qword ptr [rsi]
+ 1 5 0.50 * mov rdx, qword ptr [rsi + 8]
+ 1 5 0.50 * mov rsi, qword ptr [rsi + 16]
+ 1 1 0.50 lea rdi, [rsi + 2*rsi]
+ 1 1 0.33 add rdi, rcx
+ 1 1 0.33 add rdi, 9
+ 1 1 0.33 sub rdx, rsi
+ 1 1 1.00 * mov qword ptr [rax], rcx
+ 1 1 1.00 * mov qword ptr [rax + 8], rsi
+ 1 1 1.00 * mov qword ptr [rax + 16], rdi
+ 1 1 1.00 * mov qword ptr [rax + 24], rdx
+ 1 1 1.00 U ret
+
+
+Resources:
+[0] - SBDivider
+[1] - SBFPDivider
+[2] - SBPort0
+[3] - SBPort1
+[4] - SBPort4
+[5] - SBPort5
+[6.0] - SBPort23
+[6.1] - SBPort23
+
+
+Resource pressure per iteration:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1]
+ - - 2.00 2.00 4.00 2.00 3.50 3.50
+
+Resource pressure by instruction:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions:
+ - - 0.03 0.93 - 0.04 - - mov rax, rdi
+ - - - - - - 0.49 0.51 mov rcx, qword ptr [rsi]
+ - - - - - - 1.00 - mov rdx, qword ptr [rsi + 8]
+ - - - - - - 0.01 0.99 mov rsi, qword ptr [rsi + 16]
+ - - 0.93 0.07 - - - - lea rdi, [rsi + 2*rsi]
+ - - 0.05 0.02 - 0.93 - - add rdi, rcx
+ - - 0.49 0.49 - 0.02 - - add rdi, 9
+ - - 0.50 0.49 - 0.01 - - sub rdx, rsi
+ - - - - 1.00 - 0.50 0.50 mov qword ptr [rax], rcx
+ - - - - 1.00 - 0.50 0.50 mov qword ptr [rax + 8], rsi
+ - - - - 1.00 - 0.49 0.51 mov qword ptr [rax + 16], rdi
+ - - - - 1.00 - 0.51 0.49 mov qword ptr [rax + 24], rdx
+ - - - - - 1.00 - - ret
diff --git a/rust/zerocopy/benches/split_via_immutable_dynamic_size.rs b/rust/zerocopy/benches/split_via_immutable_dynamic_size.rs
new file mode 100644
index 0000000..7d115ca
--- /dev/null
+++ b/rust/zerocopy/benches/split_via_immutable_dynamic_size.rs
@@ -0,0 +1,11 @@
+use zerocopy::*;
+
+#[path = "formats/coco_dynamic_size.rs"]
+mod format;
+
+#[unsafe(no_mangle)]
+fn bench_split_via_immutable_dynamic_size(
+ split: Split<&format::CocoPacket>,
+) -> (&format::CocoPacket, &[[u8; 2]]) {
+ split.via_immutable()
+}
diff --git a/rust/zerocopy/benches/split_via_immutable_dynamic_size.x86-64 b/rust/zerocopy/benches/split_via_immutable_dynamic_size.x86-64
new file mode 100644
index 0000000..58f6b09
--- /dev/null
+++ b/rust/zerocopy/benches/split_via_immutable_dynamic_size.x86-64
@@ -0,0 +1,13 @@
+bench_split_via_immutable_dynamic_size:
+ mov rax, rdi
+ mov rcx, qword ptr [rsi]
+ mov rdx, qword ptr [rsi + 8]
+ mov rsi, qword ptr [rsi + 16]
+ lea rdi, [rcx + 2*rsi]
+ add rdi, 4
+ sub rdx, rsi
+ mov qword ptr [rax], rcx
+ mov qword ptr [rax + 8], rsi
+ mov qword ptr [rax + 16], rdi
+ mov qword ptr [rax + 24], rdx
+ ret
diff --git a/rust/zerocopy/benches/split_via_immutable_dynamic_size.x86-64.mca b/rust/zerocopy/benches/split_via_immutable_dynamic_size.x86-64.mca
new file mode 100644
index 0000000..4549f20
--- /dev/null
+++ b/rust/zerocopy/benches/split_via_immutable_dynamic_size.x86-64.mca
@@ -0,0 +1,63 @@
+Iterations: 100
+Instructions: 1200
+Total Cycles: 509
+Total uOps: 1200
+
+Dispatch Width: 4
+uOps Per Cycle: 2.36
+IPC: 2.36
+Block RThroughput: 4.0
+
+
+Instruction Info:
+[1]: #uOps
+[2]: Latency
+[3]: RThroughput
+[4]: MayLoad
+[5]: MayStore
+[6]: HasSideEffects (U)
+
+[1] [2] [3] [4] [5] [6] Instructions:
+ 1 1 0.33 mov rax, rdi
+ 1 5 0.50 * mov rcx, qword ptr [rsi]
+ 1 5 0.50 * mov rdx, qword ptr [rsi + 8]
+ 1 5 0.50 * mov rsi, qword ptr [rsi + 16]
+ 1 1 0.50 lea rdi, [rcx + 2*rsi]
+ 1 1 0.33 add rdi, 4
+ 1 1 0.33 sub rdx, rsi
+ 1 1 1.00 * mov qword ptr [rax], rcx
+ 1 1 1.00 * mov qword ptr [rax + 8], rsi
+ 1 1 1.00 * mov qword ptr [rax + 16], rdi
+ 1 1 1.00 * mov qword ptr [rax + 24], rdx
+ 1 1 1.00 U ret
+
+
+Resources:
+[0] - SBDivider
+[1] - SBFPDivider
+[2] - SBPort0
+[3] - SBPort1
+[4] - SBPort4
+[5] - SBPort5
+[6.0] - SBPort23
+[6.1] - SBPort23
+
+
+Resource pressure per iteration:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1]
+ - - 1.66 1.66 4.00 1.68 3.50 3.50
+
+Resource pressure by instruction:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions:
+ - - 0.34 0.33 - 0.33 - - mov rax, rdi
+ - - - - - - 0.49 0.51 mov rcx, qword ptr [rsi]
+ - - - - - - 0.51 0.49 mov rdx, qword ptr [rsi + 8]
+ - - - - - - 0.01 0.99 mov rsi, qword ptr [rsi + 16]
+ - - 0.33 0.67 - - - - lea rdi, [rcx + 2*rsi]
+ - - 0.63 0.34 - 0.03 - - add rdi, 4
+ - - 0.36 0.32 - 0.32 - - sub rdx, rsi
+ - - - - 1.00 - 0.50 0.50 mov qword ptr [rax], rcx
+ - - - - 1.00 - 0.50 0.50 mov qword ptr [rax + 8], rsi
+ - - - - 1.00 - 0.98 0.02 mov qword ptr [rax + 16], rdi
+ - - - - 1.00 - 0.51 0.49 mov qword ptr [rax + 24], rdx
+ - - - - - 1.00 - - ret
diff --git a/rust/zerocopy/benches/split_via_runtime_check_dynamic_padding.rs b/rust/zerocopy/benches/split_via_runtime_check_dynamic_padding.rs
new file mode 100644
index 0000000..edba7bf
--- /dev/null
+++ b/rust/zerocopy/benches/split_via_runtime_check_dynamic_padding.rs
@@ -0,0 +1,11 @@
+use zerocopy::*;
+
+#[path = "formats/coco_dynamic_padding.rs"]
+mod format;
+
+#[unsafe(no_mangle)]
+fn bench_split_via_runtime_check_dynamic_padding(
+ split: Split<&format::CocoPacket>,
+) -> Option<(&format::CocoPacket, &[[u8; 3]])> {
+ split.via_runtime_check().ok()
+}
diff --git a/rust/zerocopy/benches/split_via_runtime_check_dynamic_padding.x86-64 b/rust/zerocopy/benches/split_via_runtime_check_dynamic_padding.x86-64
new file mode 100644
index 0000000..03684cb
--- /dev/null
+++ b/rust/zerocopy/benches/split_via_runtime_check_dynamic_padding.x86-64
@@ -0,0 +1,22 @@
+bench_split_via_runtime_check_dynamic_padding:
+ mov rax, rdi
+ mov rdx, qword ptr [rsi + 16]
+ mov ecx, edx
+ and ecx, 3
+ cmp ecx, 1
+ jne .LBB5_1
+ mov rcx, qword ptr [rsi]
+ mov rsi, qword ptr [rsi + 8]
+ lea rdi, [rdx + 2*rdx]
+ add rdi, rcx
+ add rdi, 9
+ sub rsi, rdx
+ mov qword ptr [rax + 8], rdx
+ mov qword ptr [rax + 16], rdi
+ mov qword ptr [rax + 24], rsi
+ mov qword ptr [rax], rcx
+ ret
+.LBB5_1:
+ xor ecx, ecx
+ mov qword ptr [rax], rcx
+ ret
diff --git a/rust/zerocopy/benches/split_via_runtime_check_dynamic_padding.x86-64.mca b/rust/zerocopy/benches/split_via_runtime_check_dynamic_padding.x86-64.mca
new file mode 100644
index 0000000..5034ab0
--- /dev/null
+++ b/rust/zerocopy/benches/split_via_runtime_check_dynamic_padding.x86-64.mca
@@ -0,0 +1,79 @@
+Iterations: 100
+Instructions: 2000
+Total Cycles: 708
+Total uOps: 2000
+
+Dispatch Width: 4
+uOps Per Cycle: 2.82
+IPC: 2.82
+Block RThroughput: 5.0
+
+
+Instruction Info:
+[1]: #uOps
+[2]: Latency
+[3]: RThroughput
+[4]: MayLoad
+[5]: MayStore
+[6]: HasSideEffects (U)
+
+[1] [2] [3] [4] [5] [6] Instructions:
+ 1 1 0.33 mov rax, rdi
+ 1 5 0.50 * mov rdx, qword ptr [rsi + 16]
+ 1 1 0.33 mov ecx, edx
+ 1 1 0.33 and ecx, 3
+ 1 1 0.33 cmp ecx, 1
+ 1 1 1.00 jne .LBB5_1
+ 1 5 0.50 * mov rcx, qword ptr [rsi]
+ 1 5 0.50 * mov rsi, qword ptr [rsi + 8]
+ 1 1 0.50 lea rdi, [rdx + 2*rdx]
+ 1 1 0.33 add rdi, rcx
+ 1 1 0.33 add rdi, 9
+ 1 1 0.33 sub rsi, rdx
+ 1 1 1.00 * mov qword ptr [rax + 8], rdx
+ 1 1 1.00 * mov qword ptr [rax + 16], rdi
+ 1 1 1.00 * mov qword ptr [rax + 24], rsi
+ 1 1 1.00 * mov qword ptr [rax], rcx
+ 1 1 1.00 U ret
+ 1 0 0.25 xor ecx, ecx
+ 1 1 1.00 * mov qword ptr [rax], rcx
+ 1 1 1.00 U ret
+
+
+Resources:
+[0] - SBDivider
+[1] - SBFPDivider
+[2] - SBPort0
+[3] - SBPort1
+[4] - SBPort4
+[5] - SBPort5
+[6.0] - SBPort23
+[6.1] - SBPort23
+
+
+Resource pressure per iteration:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1]
+ - - 3.00 3.02 5.00 4.98 4.00 4.00
+
+Resource pressure by instruction:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions:
+ - - - 0.99 - 0.01 - - mov rax, rdi
+ - - - - - - - 1.00 mov rdx, qword ptr [rsi + 16]
+ - - 0.99 0.01 - - - - mov ecx, edx
+ - - 0.99 0.01 - - - - and ecx, 3
+ - - 0.97 0.03 - - - - cmp ecx, 1
+ - - - - - 1.00 - - jne .LBB5_1
+ - - - - - - 1.00 - mov rcx, qword ptr [rsi]
+ - - - - - - 0.99 0.01 mov rsi, qword ptr [rsi + 8]
+ - - 0.01 0.99 - - - - lea rdi, [rdx + 2*rdx]
+ - - - 0.96 - 0.04 - - add rdi, rcx
+ - - 0.03 - - 0.97 - - add rdi, 9
+ - - 0.01 0.03 - 0.96 - - sub rsi, rdx
+ - - - - 1.00 - 1.00 - mov qword ptr [rax + 8], rdx
+ - - - - 1.00 - - 1.00 mov qword ptr [rax + 16], rdi
+ - - - - 1.00 - 0.01 0.99 mov qword ptr [rax + 24], rsi
+ - - - - 1.00 - 0.99 0.01 mov qword ptr [rax], rcx
+ - - - - - 1.00 - - ret
+ - - - - - - - - xor ecx, ecx
+ - - - - 1.00 - 0.01 0.99 mov qword ptr [rax], rcx
+ - - - - - 1.00 - - ret
diff --git a/rust/zerocopy/benches/split_via_runtime_check_dynamic_size.rs b/rust/zerocopy/benches/split_via_runtime_check_dynamic_size.rs
new file mode 100644
index 0000000..ce22de9
--- /dev/null
+++ b/rust/zerocopy/benches/split_via_runtime_check_dynamic_size.rs
@@ -0,0 +1,11 @@
+use zerocopy::*;
+
+#[path = "formats/coco_dynamic_size.rs"]
+mod format;
+
+#[unsafe(no_mangle)]
+fn bench_split_via_runtime_check_dynamic_size(
+ split: Split<&format::CocoPacket>,
+) -> Option<(&format::CocoPacket, &[[u8; 2]])> {
+ split.via_runtime_check().ok()
+}
diff --git a/rust/zerocopy/benches/split_via_runtime_check_dynamic_size.x86-64 b/rust/zerocopy/benches/split_via_runtime_check_dynamic_size.x86-64
new file mode 100644
index 0000000..e54276b
--- /dev/null
+++ b/rust/zerocopy/benches/split_via_runtime_check_dynamic_size.x86-64
@@ -0,0 +1,13 @@
+bench_split_via_runtime_check_dynamic_size:
+ mov rax, rdi
+ mov rcx, qword ptr [rsi]
+ mov rdx, qword ptr [rsi + 8]
+ mov rsi, qword ptr [rsi + 16]
+ lea rdi, [rcx + 2*rsi]
+ add rdi, 4
+ sub rdx, rsi
+ mov qword ptr [rax], rcx
+ mov qword ptr [rax + 8], rsi
+ mov qword ptr [rax + 16], rdi
+ mov qword ptr [rax + 24], rdx
+ ret
diff --git a/rust/zerocopy/benches/split_via_runtime_check_dynamic_size.x86-64.mca b/rust/zerocopy/benches/split_via_runtime_check_dynamic_size.x86-64.mca
new file mode 100644
index 0000000..4549f20
--- /dev/null
+++ b/rust/zerocopy/benches/split_via_runtime_check_dynamic_size.x86-64.mca
@@ -0,0 +1,63 @@
+Iterations: 100
+Instructions: 1200
+Total Cycles: 509
+Total uOps: 1200
+
+Dispatch Width: 4
+uOps Per Cycle: 2.36
+IPC: 2.36
+Block RThroughput: 4.0
+
+
+Instruction Info:
+[1]: #uOps
+[2]: Latency
+[3]: RThroughput
+[4]: MayLoad
+[5]: MayStore
+[6]: HasSideEffects (U)
+
+[1] [2] [3] [4] [5] [6] Instructions:
+ 1 1 0.33 mov rax, rdi
+ 1 5 0.50 * mov rcx, qword ptr [rsi]
+ 1 5 0.50 * mov rdx, qword ptr [rsi + 8]
+ 1 5 0.50 * mov rsi, qword ptr [rsi + 16]
+ 1 1 0.50 lea rdi, [rcx + 2*rsi]
+ 1 1 0.33 add rdi, 4
+ 1 1 0.33 sub rdx, rsi
+ 1 1 1.00 * mov qword ptr [rax], rcx
+ 1 1 1.00 * mov qword ptr [rax + 8], rsi
+ 1 1 1.00 * mov qword ptr [rax + 16], rdi
+ 1 1 1.00 * mov qword ptr [rax + 24], rdx
+ 1 1 1.00 U ret
+
+
+Resources:
+[0] - SBDivider
+[1] - SBFPDivider
+[2] - SBPort0
+[3] - SBPort1
+[4] - SBPort4
+[5] - SBPort5
+[6.0] - SBPort23
+[6.1] - SBPort23
+
+
+Resource pressure per iteration:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1]
+ - - 1.66 1.66 4.00 1.68 3.50 3.50
+
+Resource pressure by instruction:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions:
+ - - 0.34 0.33 - 0.33 - - mov rax, rdi
+ - - - - - - 0.49 0.51 mov rcx, qword ptr [rsi]
+ - - - - - - 0.51 0.49 mov rdx, qword ptr [rsi + 8]
+ - - - - - - 0.01 0.99 mov rsi, qword ptr [rsi + 16]
+ - - 0.33 0.67 - - - - lea rdi, [rcx + 2*rsi]
+ - - 0.63 0.34 - 0.03 - - add rdi, 4
+ - - 0.36 0.32 - 0.32 - - sub rdx, rsi
+ - - - - 1.00 - 0.50 0.50 mov qword ptr [rax], rcx
+ - - - - 1.00 - 0.50 0.50 mov qword ptr [rax + 8], rsi
+ - - - - 1.00 - 0.98 0.02 mov qword ptr [rax + 16], rdi
+ - - - - 1.00 - 0.51 0.49 mov qword ptr [rax + 24], rdx
+ - - - - - 1.00 - - ret
diff --git a/rust/zerocopy/benches/split_via_unchecked_dynamic_padding.rs b/rust/zerocopy/benches/split_via_unchecked_dynamic_padding.rs
new file mode 100644
index 0000000..21d74db
--- /dev/null
+++ b/rust/zerocopy/benches/split_via_unchecked_dynamic_padding.rs
@@ -0,0 +1,11 @@
+use zerocopy::*;
+
+#[path = "formats/coco_dynamic_padding.rs"]
+mod format;
+
+#[unsafe(no_mangle)]
+unsafe fn bench_split_via_unchecked_dynamic_padding(
+ split: Split<&format::CocoPacket>,
+) -> (&format::CocoPacket, &[[u8; 3]]) {
+ unsafe { split.via_unchecked() }
+}
diff --git a/rust/zerocopy/benches/split_via_unchecked_dynamic_padding.x86-64 b/rust/zerocopy/benches/split_via_unchecked_dynamic_padding.x86-64
new file mode 100644
index 0000000..3c2c4ec
--- /dev/null
+++ b/rust/zerocopy/benches/split_via_unchecked_dynamic_padding.x86-64
@@ -0,0 +1,14 @@
+bench_split_via_unchecked_dynamic_padding:
+ mov rax, rdi
+ mov rcx, qword ptr [rsi]
+ mov rdx, qword ptr [rsi + 8]
+ mov rsi, qword ptr [rsi + 16]
+ lea rdi, [rsi + 2*rsi]
+ add rdi, rcx
+ add rdi, 9
+ sub rdx, rsi
+ mov qword ptr [rax], rcx
+ mov qword ptr [rax + 8], rsi
+ mov qword ptr [rax + 16], rdi
+ mov qword ptr [rax + 24], rdx
+ ret
diff --git a/rust/zerocopy/benches/split_via_unchecked_dynamic_padding.x86-64.mca b/rust/zerocopy/benches/split_via_unchecked_dynamic_padding.x86-64.mca
new file mode 100644
index 0000000..6ab4e83
--- /dev/null
+++ b/rust/zerocopy/benches/split_via_unchecked_dynamic_padding.x86-64.mca
@@ -0,0 +1,65 @@
+Iterations: 100
+Instructions: 1300
+Total Cycles: 510
+Total uOps: 1300
+
+Dispatch Width: 4
+uOps Per Cycle: 2.55
+IPC: 2.55
+Block RThroughput: 4.0
+
+
+Instruction Info:
+[1]: #uOps
+[2]: Latency
+[3]: RThroughput
+[4]: MayLoad
+[5]: MayStore
+[6]: HasSideEffects (U)
+
+[1] [2] [3] [4] [5] [6] Instructions:
+ 1 1 0.33 mov rax, rdi
+ 1 5 0.50 * mov rcx, qword ptr [rsi]
+ 1 5 0.50 * mov rdx, qword ptr [rsi + 8]
+ 1 5 0.50 * mov rsi, qword ptr [rsi + 16]
+ 1 1 0.50 lea rdi, [rsi + 2*rsi]
+ 1 1 0.33 add rdi, rcx
+ 1 1 0.33 add rdi, 9
+ 1 1 0.33 sub rdx, rsi
+ 1 1 1.00 * mov qword ptr [rax], rcx
+ 1 1 1.00 * mov qword ptr [rax + 8], rsi
+ 1 1 1.00 * mov qword ptr [rax + 16], rdi
+ 1 1 1.00 * mov qword ptr [rax + 24], rdx
+ 1 1 1.00 U ret
+
+
+Resources:
+[0] - SBDivider
+[1] - SBFPDivider
+[2] - SBPort0
+[3] - SBPort1
+[4] - SBPort4
+[5] - SBPort5
+[6.0] - SBPort23
+[6.1] - SBPort23
+
+
+Resource pressure per iteration:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1]
+ - - 2.00 2.00 4.00 2.00 3.50 3.50
+
+Resource pressure by instruction:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions:
+ - - 0.03 0.93 - 0.04 - - mov rax, rdi
+ - - - - - - 0.49 0.51 mov rcx, qword ptr [rsi]
+ - - - - - - 1.00 - mov rdx, qword ptr [rsi + 8]
+ - - - - - - 0.01 0.99 mov rsi, qword ptr [rsi + 16]
+ - - 0.93 0.07 - - - - lea rdi, [rsi + 2*rsi]
+ - - 0.05 0.02 - 0.93 - - add rdi, rcx
+ - - 0.49 0.49 - 0.02 - - add rdi, 9
+ - - 0.50 0.49 - 0.01 - - sub rdx, rsi
+ - - - - 1.00 - 0.50 0.50 mov qword ptr [rax], rcx
+ - - - - 1.00 - 0.50 0.50 mov qword ptr [rax + 8], rsi
+ - - - - 1.00 - 0.49 0.51 mov qword ptr [rax + 16], rdi
+ - - - - 1.00 - 0.51 0.49 mov qword ptr [rax + 24], rdx
+ - - - - - 1.00 - - ret
diff --git a/rust/zerocopy/benches/split_via_unchecked_dynamic_size.rs b/rust/zerocopy/benches/split_via_unchecked_dynamic_size.rs
new file mode 100644
index 0000000..824e22d
--- /dev/null
+++ b/rust/zerocopy/benches/split_via_unchecked_dynamic_size.rs
@@ -0,0 +1,11 @@
+use zerocopy::*;
+
+#[path = "formats/coco_dynamic_size.rs"]
+mod format;
+
+#[unsafe(no_mangle)]
+unsafe fn bench_split_via_unchecked_dynamic_size(
+ split: Split<&format::CocoPacket>,
+) -> (&format::CocoPacket, &[[u8; 2]]) {
+ unsafe { split.via_unchecked() }
+}
diff --git a/rust/zerocopy/benches/split_via_unchecked_dynamic_size.x86-64 b/rust/zerocopy/benches/split_via_unchecked_dynamic_size.x86-64
new file mode 100644
index 0000000..1e31268
--- /dev/null
+++ b/rust/zerocopy/benches/split_via_unchecked_dynamic_size.x86-64
@@ -0,0 +1,13 @@
+bench_split_via_unchecked_dynamic_size:
+ mov rax, rdi
+ mov rcx, qword ptr [rsi]
+ mov rdx, qword ptr [rsi + 8]
+ mov rsi, qword ptr [rsi + 16]
+ lea rdi, [rcx + 2*rsi]
+ add rdi, 4
+ sub rdx, rsi
+ mov qword ptr [rax], rcx
+ mov qword ptr [rax + 8], rsi
+ mov qword ptr [rax + 16], rdi
+ mov qword ptr [rax + 24], rdx
+ ret
diff --git a/rust/zerocopy/benches/split_via_unchecked_dynamic_size.x86-64.mca b/rust/zerocopy/benches/split_via_unchecked_dynamic_size.x86-64.mca
new file mode 100644
index 0000000..4549f20
--- /dev/null
+++ b/rust/zerocopy/benches/split_via_unchecked_dynamic_size.x86-64.mca
@@ -0,0 +1,63 @@
+Iterations: 100
+Instructions: 1200
+Total Cycles: 509
+Total uOps: 1200
+
+Dispatch Width: 4
+uOps Per Cycle: 2.36
+IPC: 2.36
+Block RThroughput: 4.0
+
+
+Instruction Info:
+[1]: #uOps
+[2]: Latency
+[3]: RThroughput
+[4]: MayLoad
+[5]: MayStore
+[6]: HasSideEffects (U)
+
+[1] [2] [3] [4] [5] [6] Instructions:
+ 1 1 0.33 mov rax, rdi
+ 1 5 0.50 * mov rcx, qword ptr [rsi]
+ 1 5 0.50 * mov rdx, qword ptr [rsi + 8]
+ 1 5 0.50 * mov rsi, qword ptr [rsi + 16]
+ 1 1 0.50 lea rdi, [rcx + 2*rsi]
+ 1 1 0.33 add rdi, 4
+ 1 1 0.33 sub rdx, rsi
+ 1 1 1.00 * mov qword ptr [rax], rcx
+ 1 1 1.00 * mov qword ptr [rax + 8], rsi
+ 1 1 1.00 * mov qword ptr [rax + 16], rdi
+ 1 1 1.00 * mov qword ptr [rax + 24], rdx
+ 1 1 1.00 U ret
+
+
+Resources:
+[0] - SBDivider
+[1] - SBFPDivider
+[2] - SBPort0
+[3] - SBPort1
+[4] - SBPort4
+[5] - SBPort5
+[6.0] - SBPort23
+[6.1] - SBPort23
+
+
+Resource pressure per iteration:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1]
+ - - 1.66 1.66 4.00 1.68 3.50 3.50
+
+Resource pressure by instruction:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions:
+ - - 0.34 0.33 - 0.33 - - mov rax, rdi
+ - - - - - - 0.49 0.51 mov rcx, qword ptr [rsi]
+ - - - - - - 0.51 0.49 mov rdx, qword ptr [rsi + 8]
+ - - - - - - 0.01 0.99 mov rsi, qword ptr [rsi + 16]
+ - - 0.33 0.67 - - - - lea rdi, [rcx + 2*rsi]
+ - - 0.63 0.34 - 0.03 - - add rdi, 4
+ - - 0.36 0.32 - 0.32 - - sub rdx, rsi
+ - - - - 1.00 - 0.50 0.50 mov qword ptr [rax], rcx
+ - - - - 1.00 - 0.50 0.50 mov qword ptr [rax + 8], rsi
+ - - - - 1.00 - 0.98 0.02 mov qword ptr [rax + 16], rdi
+ - - - - 1.00 - 0.51 0.49 mov qword ptr [rax + 24], rdx
+ - - - - - 1.00 - - ret
diff --git a/rust/zerocopy/benches/transmute.rs b/rust/zerocopy/benches/transmute.rs
new file mode 100644
index 0000000..e60bfb25
--- /dev/null
+++ b/rust/zerocopy/benches/transmute.rs
@@ -0,0 +1,16 @@
+use zerocopy::Unalign;
+use zerocopy_derive::*;
+
+#[path = "formats/coco_static_size.rs"]
+mod format;
+
+#[derive(IntoBytes, KnownLayout, Immutable)]
+#[repr(C)]
+struct MinimalViableSource {
+ bytes: [u8; 6],
+}
+
+#[unsafe(no_mangle)]
+fn bench_transmute(source: MinimalViableSource) -> Unalign<format::LocoPacket> {
+ zerocopy::transmute!(source)
+}
diff --git a/rust/zerocopy/benches/transmute.x86-64 b/rust/zerocopy/benches/transmute.x86-64
new file mode 100644
index 0000000..a4c6299
--- /dev/null
+++ b/rust/zerocopy/benches/transmute.x86-64
@@ -0,0 +1,3 @@
+bench_transmute:
+ mov rax, rdi
+ ret
diff --git a/rust/zerocopy/benches/transmute.x86-64.mca b/rust/zerocopy/benches/transmute.x86-64.mca
new file mode 100644
index 0000000..f297729
--- /dev/null
+++ b/rust/zerocopy/benches/transmute.x86-64.mca
@@ -0,0 +1,43 @@
+Iterations: 100
+Instructions: 200
+Total Cycles: 104
+Total uOps: 200
+
+Dispatch Width: 4
+uOps Per Cycle: 1.92
+IPC: 1.92
+Block RThroughput: 1.0
+
+
+Instruction Info:
+[1]: #uOps
+[2]: Latency
+[3]: RThroughput
+[4]: MayLoad
+[5]: MayStore
+[6]: HasSideEffects (U)
+
+[1] [2] [3] [4] [5] [6] Instructions:
+ 1 1 0.33 mov rax, rdi
+ 1 1 1.00 U ret
+
+
+Resources:
+[0] - SBDivider
+[1] - SBFPDivider
+[2] - SBPort0
+[3] - SBPort1
+[4] - SBPort4
+[5] - SBPort5
+[6.0] - SBPort23
+[6.1] - SBPort23
+
+
+Resource pressure per iteration:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1]
+ - - 0.49 0.50 - 1.01 - -
+
+Resource pressure by instruction:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions:
+ - - 0.49 0.50 - 0.01 - - mov rax, rdi
+ - - - - - 1.00 - - ret
diff --git a/rust/zerocopy/benches/transmute_ref_dynamic_size.rs b/rust/zerocopy/benches/transmute_ref_dynamic_size.rs
new file mode 100644
index 0000000..825f0f2
--- /dev/null
+++ b/rust/zerocopy/benches/transmute_ref_dynamic_size.rs
@@ -0,0 +1,16 @@
+use zerocopy_derive::*;
+
+#[path = "formats/coco_dynamic_size.rs"]
+mod format;
+
+#[derive(IntoBytes, KnownLayout, Immutable)]
+#[repr(C, align(2))]
+struct MinimalViableSource {
+ header: [u8; 6],
+ trailer: [[u8; 2]],
+}
+
+#[unsafe(no_mangle)]
+fn bench_transmute_ref_dynamic_size(source: &MinimalViableSource) -> &format::LocoPacket {
+ zerocopy::transmute_ref!(source)
+}
diff --git a/rust/zerocopy/benches/transmute_ref_dynamic_size.x86-64 b/rust/zerocopy/benches/transmute_ref_dynamic_size.x86-64
new file mode 100644
index 0000000..80a0f59
--- /dev/null
+++ b/rust/zerocopy/benches/transmute_ref_dynamic_size.x86-64
@@ -0,0 +1,4 @@
+bench_transmute_ref_dynamic_size:
+ mov rax, rdi
+ lea rdx, [rsi + 1]
+ ret
diff --git a/rust/zerocopy/benches/transmute_ref_dynamic_size.x86-64.mca b/rust/zerocopy/benches/transmute_ref_dynamic_size.x86-64.mca
new file mode 100644
index 0000000..ef1bcdc
--- /dev/null
+++ b/rust/zerocopy/benches/transmute_ref_dynamic_size.x86-64.mca
@@ -0,0 +1,45 @@
+Iterations: 100
+Instructions: 300
+Total Cycles: 104
+Total uOps: 300
+
+Dispatch Width: 4
+uOps Per Cycle: 2.88
+IPC: 2.88
+Block RThroughput: 1.0
+
+
+Instruction Info:
+[1]: #uOps
+[2]: Latency
+[3]: RThroughput
+[4]: MayLoad
+[5]: MayStore
+[6]: HasSideEffects (U)
+
+[1] [2] [3] [4] [5] [6] Instructions:
+ 1 1 0.33 mov rax, rdi
+ 1 1 0.50 lea rdx, [rsi + 1]
+ 1 1 1.00 U ret
+
+
+Resources:
+[0] - SBDivider
+[1] - SBFPDivider
+[2] - SBPort0
+[3] - SBPort1
+[4] - SBPort4
+[5] - SBPort5
+[6.0] - SBPort23
+[6.1] - SBPort23
+
+
+Resource pressure per iteration:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1]
+ - - 0.99 1.00 - 1.01 - -
+
+Resource pressure by instruction:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions:
+ - - 0.99 - - 0.01 - - mov rax, rdi
+ - - - 1.00 - - - - lea rdx, [rsi + 1]
+ - - - - - 1.00 - - ret
diff --git a/rust/zerocopy/benches/transmute_ref_static_size.rs b/rust/zerocopy/benches/transmute_ref_static_size.rs
new file mode 100644
index 0000000..a6db611
--- /dev/null
+++ b/rust/zerocopy/benches/transmute_ref_static_size.rs
@@ -0,0 +1,15 @@
+use zerocopy_derive::*;
+
+#[path = "formats/coco_static_size.rs"]
+mod format;
+
+#[derive(IntoBytes, KnownLayout, Immutable)]
+#[repr(C, align(2))]
+struct MinimalViableSource {
+ bytes: [u8; 6],
+}
+
+#[unsafe(no_mangle)]
+fn bench_transmute_ref_static_size(source: &MinimalViableSource) -> &format::LocoPacket {
+ zerocopy::transmute_ref!(source)
+}
diff --git a/rust/zerocopy/benches/transmute_ref_static_size.x86-64 b/rust/zerocopy/benches/transmute_ref_static_size.x86-64
new file mode 100644
index 0000000..7a9229c
--- /dev/null
+++ b/rust/zerocopy/benches/transmute_ref_static_size.x86-64
@@ -0,0 +1,3 @@
+bench_transmute_ref_static_size:
+ mov rax, rdi
+ ret
diff --git a/rust/zerocopy/benches/transmute_ref_static_size.x86-64.mca b/rust/zerocopy/benches/transmute_ref_static_size.x86-64.mca
new file mode 100644
index 0000000..f297729
--- /dev/null
+++ b/rust/zerocopy/benches/transmute_ref_static_size.x86-64.mca
@@ -0,0 +1,43 @@
+Iterations: 100
+Instructions: 200
+Total Cycles: 104
+Total uOps: 200
+
+Dispatch Width: 4
+uOps Per Cycle: 1.92
+IPC: 1.92
+Block RThroughput: 1.0
+
+
+Instruction Info:
+[1]: #uOps
+[2]: Latency
+[3]: RThroughput
+[4]: MayLoad
+[5]: MayStore
+[6]: HasSideEffects (U)
+
+[1] [2] [3] [4] [5] [6] Instructions:
+ 1 1 0.33 mov rax, rdi
+ 1 1 1.00 U ret
+
+
+Resources:
+[0] - SBDivider
+[1] - SBFPDivider
+[2] - SBPort0
+[3] - SBPort1
+[4] - SBPort4
+[5] - SBPort5
+[6.0] - SBPort23
+[6.1] - SBPort23
+
+
+Resource pressure per iteration:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1]
+ - - 0.49 0.50 - 1.01 - -
+
+Resource pressure by instruction:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions:
+ - - 0.49 0.50 - 0.01 - - mov rax, rdi
+ - - - - - 1.00 - - ret
diff --git a/rust/zerocopy/benches/try_read_from_bytes.rs b/rust/zerocopy/benches/try_read_from_bytes.rs
new file mode 100644
index 0000000..f8384b3
--- /dev/null
+++ b/rust/zerocopy/benches/try_read_from_bytes.rs
@@ -0,0 +1,7 @@
+#[path = "formats/coco_static_size.rs"]
+mod format;
+
+#[unsafe(no_mangle)]
+fn bench_try_read_from_bytes_static_size(source: &[u8]) -> Option<format::CocoPacket> {
+ zerocopy::TryFromBytes::try_read_from_bytes(source).ok()
+}
diff --git a/rust/zerocopy/benches/try_read_from_bytes.x86-64 b/rust/zerocopy/benches/try_read_from_bytes.x86-64
new file mode 100644
index 0000000..08088a0
--- /dev/null
+++ b/rust/zerocopy/benches/try_read_from_bytes.x86-64
@@ -0,0 +1,23 @@
+bench_try_read_from_bytes_static_size:
+ mov ax, -16191
+ cmp rsi, 6
+ jne .LBB5_1
+ mov ecx, dword ptr [rdi]
+ movzx edx, cx
+ cmp edx, 49344
+ jne .LBB5_4
+ movzx eax, word ptr [rdi + 4]
+ shl rax, 32
+ or rcx, rax
+ shr rcx, 16
+ mov ax, -16192
+.LBB5_4:
+ shl rcx, 16
+ movzx eax, ax
+ or rax, rcx
+ ret
+.LBB5_1:
+ shl rcx, 16
+ movzx eax, ax
+ or rax, rcx
+ ret
diff --git a/rust/zerocopy/benches/try_read_from_bytes.x86-64.mca b/rust/zerocopy/benches/try_read_from_bytes.x86-64.mca
new file mode 100644
index 0000000..385e6a4
--- /dev/null
+++ b/rust/zerocopy/benches/try_read_from_bytes.x86-64.mca
@@ -0,0 +1,79 @@
+Iterations: 100
+Instructions: 2000
+Total Cycles: 608
+Total uOps: 2000
+
+Dispatch Width: 4
+uOps Per Cycle: 3.29
+IPC: 3.29
+Block RThroughput: 5.0
+
+
+Instruction Info:
+[1]: #uOps
+[2]: Latency
+[3]: RThroughput
+[4]: MayLoad
+[5]: MayStore
+[6]: HasSideEffects (U)
+
+[1] [2] [3] [4] [5] [6] Instructions:
+ 1 1 0.33 mov ax, -16191
+ 1 1 0.33 cmp rsi, 6
+ 1 1 1.00 jne .LBB5_1
+ 1 5 0.50 * mov ecx, dword ptr [rdi]
+ 1 1 0.33 movzx edx, cx
+ 1 1 0.33 cmp edx, 49344
+ 1 1 1.00 jne .LBB5_4
+ 1 5 0.50 * movzx eax, word ptr [rdi + 4]
+ 1 1 0.50 shl rax, 32
+ 1 1 0.33 or rcx, rax
+ 1 1 0.50 shr rcx, 16
+ 1 1 0.33 mov ax, -16192
+ 1 1 0.50 shl rcx, 16
+ 1 1 0.33 movzx eax, ax
+ 1 1 0.33 or rax, rcx
+ 1 1 1.00 U ret
+ 1 1 0.50 shl rcx, 16
+ 1 1 0.33 movzx eax, ax
+ 1 1 0.33 or rax, rcx
+ 1 1 1.00 U ret
+
+
+Resources:
+[0] - SBDivider
+[1] - SBFPDivider
+[2] - SBPort0
+[3] - SBPort1
+[4] - SBPort4
+[5] - SBPort5
+[6.0] - SBPort23
+[6.1] - SBPort23
+
+
+Resource pressure per iteration:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1]
+ - - 5.99 5.99 - 6.02 1.00 1.00
+
+Resource pressure by instruction:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions:
+ - - - 0.99 - 0.01 - - mov ax, -16191
+ - - - 0.01 - 0.99 - - cmp rsi, 6
+ - - - - - 1.00 - - jne .LBB5_1
+ - - - - - - - 1.00 mov ecx, dword ptr [rdi]
+ - - 0.98 - - 0.02 - - movzx edx, cx
+ - - 0.99 0.01 - - - - cmp edx, 49344
+ - - - - - 1.00 - - jne .LBB5_4
+ - - - - - - 1.00 - movzx eax, word ptr [rdi + 4]
+ - - 0.01 - - 0.99 - - shl rax, 32
+ - - 0.02 0.98 - - - - or rcx, rax
+ - - 1.00 - - - - - shr rcx, 16
+ - - 0.99 0.01 - - - - mov ax, -16192
+ - - 1.00 - - - - - shl rcx, 16
+ - - - 1.00 - - - - movzx eax, ax
+ - - - 1.00 - - - - or rax, rcx
+ - - - - - 1.00 - - ret
+ - - 1.00 - - - - - shl rcx, 16
+ - - - 1.00 - - - - movzx eax, ax
+ - - - 0.99 - 0.01 - - or rax, rcx
+ - - - - - 1.00 - - ret
diff --git a/rust/zerocopy/benches/try_read_from_prefix.rs b/rust/zerocopy/benches/try_read_from_prefix.rs
new file mode 100644
index 0000000..fabbac3
--- /dev/null
+++ b/rust/zerocopy/benches/try_read_from_prefix.rs
@@ -0,0 +1,10 @@
+#[path = "formats/coco_static_size.rs"]
+mod format;
+
+#[unsafe(no_mangle)]
+fn bench_try_read_from_prefix_static_size(source: &[u8]) -> Option<format::CocoPacket> {
+ match zerocopy::TryFromBytes::try_read_from_prefix(source) {
+ Ok((packet, _rest)) => Some(packet),
+ _ => None,
+ }
+}
diff --git a/rust/zerocopy/benches/try_read_from_prefix.x86-64 b/rust/zerocopy/benches/try_read_from_prefix.x86-64
new file mode 100644
index 0000000..d3e1edc
--- /dev/null
+++ b/rust/zerocopy/benches/try_read_from_prefix.x86-64
@@ -0,0 +1,16 @@
+bench_try_read_from_prefix_static_size:
+ mov eax, 49345
+ cmp rsi, 6
+ jb .LBB5_2
+ mov eax, dword ptr [rdi]
+ movzx ecx, word ptr [rdi + 4]
+ shl rcx, 32
+ or rcx, rax
+ movzx eax, cx
+ and rcx, -65536
+ or rcx, 49344
+ cmp eax, 49344
+ mov eax, 49345
+ cmove rax, rcx
+.LBB5_2:
+ ret
diff --git a/rust/zerocopy/benches/try_read_from_prefix.x86-64.mca b/rust/zerocopy/benches/try_read_from_prefix.x86-64.mca
new file mode 100644
index 0000000..40401d8
--- /dev/null
+++ b/rust/zerocopy/benches/try_read_from_prefix.x86-64.mca
@@ -0,0 +1,67 @@
+Iterations: 100
+Instructions: 1400
+Total Cycles: 442
+Total uOps: 1500
+
+Dispatch Width: 4
+uOps Per Cycle: 3.39
+IPC: 3.17
+Block RThroughput: 3.8
+
+
+Instruction Info:
+[1]: #uOps
+[2]: Latency
+[3]: RThroughput
+[4]: MayLoad
+[5]: MayStore
+[6]: HasSideEffects (U)
+
+[1] [2] [3] [4] [5] [6] Instructions:
+ 1 1 0.33 mov eax, 49345
+ 1 1 0.33 cmp rsi, 6
+ 1 1 1.00 jb .LBB5_2
+ 1 5 0.50 * mov eax, dword ptr [rdi]
+ 1 5 0.50 * movzx ecx, word ptr [rdi + 4]
+ 1 1 0.50 shl rcx, 32
+ 1 1 0.33 or rcx, rax
+ 1 1 0.33 movzx eax, cx
+ 1 1 0.33 and rcx, -65536
+ 1 1 0.33 or rcx, 49344
+ 1 1 0.33 cmp eax, 49344
+ 1 1 0.33 mov eax, 49345
+ 2 2 0.67 cmove rax, rcx
+ 1 1 1.00 U ret
+
+
+Resources:
+[0] - SBDivider
+[1] - SBFPDivider
+[2] - SBPort0
+[3] - SBPort1
+[4] - SBPort4
+[5] - SBPort5
+[6.0] - SBPort23
+[6.1] - SBPort23
+
+
+Resource pressure per iteration:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1]
+ - - 4.33 4.33 - 4.34 1.00 1.00
+
+Resource pressure by instruction:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions:
+ - - 0.65 0.01 - 0.34 - - mov eax, 49345
+ - - 0.01 0.33 - 0.66 - - cmp rsi, 6
+ - - - - - 1.00 - - jb .LBB5_2
+ - - - - - - - 1.00 mov eax, dword ptr [rdi]
+ - - - - - - 1.00 - movzx ecx, word ptr [rdi + 4]
+ - - 0.65 - - 0.35 - - shl rcx, 32
+ - - - 0.67 - 0.33 - - or rcx, rax
+ - - 0.01 0.99 - - - - movzx eax, cx
+ - - 0.99 0.01 - - - - and rcx, -65536
+ - - 0.01 0.99 - - - - or rcx, 49344
+ - - 0.99 0.01 - - - - cmp eax, 49344
+ - - 0.02 0.33 - 0.65 - - mov eax, 49345
+ - - 1.00 0.99 - 0.01 - - cmove rax, rcx
+ - - - - - 1.00 - - ret
diff --git a/rust/zerocopy/benches/try_read_from_suffix.rs b/rust/zerocopy/benches/try_read_from_suffix.rs
new file mode 100644
index 0000000..1e96164
--- /dev/null
+++ b/rust/zerocopy/benches/try_read_from_suffix.rs
@@ -0,0 +1,10 @@
+#[path = "formats/coco_static_size.rs"]
+mod format;
+
+#[unsafe(no_mangle)]
+fn bench_try_read_from_suffix_static_size(source: &[u8]) -> Option<format::CocoPacket> {
+ match zerocopy::TryFromBytes::try_read_from_suffix(source) {
+ Ok((_rest, packet)) => Some(packet),
+ _ => None,
+ }
+}
diff --git a/rust/zerocopy/benches/try_read_from_suffix.x86-64 b/rust/zerocopy/benches/try_read_from_suffix.x86-64
new file mode 100644
index 0000000..095e326
--- /dev/null
+++ b/rust/zerocopy/benches/try_read_from_suffix.x86-64
@@ -0,0 +1,18 @@
+bench_try_read_from_suffix_static_size:
+ mov eax, 49345
+ cmp rsi, 6
+ jb .LBB5_2
+ mov eax, dword ptr [rdi + rsi - 6]
+ movzx ecx, word ptr [rdi + rsi - 2]
+ shl rcx, 32
+ or rcx, rax
+ movzx edx, cx
+ xor eax, eax
+ cmp edx, 49344
+ cmovne rcx, rsi
+ sete al
+ and rcx, -65536
+ xor rax, 49345
+ or rax, rcx
+.LBB5_2:
+ ret
diff --git a/rust/zerocopy/benches/try_read_from_suffix.x86-64.mca b/rust/zerocopy/benches/try_read_from_suffix.x86-64.mca
new file mode 100644
index 0000000..d3eaadb
--- /dev/null
+++ b/rust/zerocopy/benches/try_read_from_suffix.x86-64.mca
@@ -0,0 +1,71 @@
+Iterations: 100
+Instructions: 1600
+Total Cycles: 478
+Total uOps: 1700
+
+Dispatch Width: 4
+uOps Per Cycle: 3.56
+IPC: 3.35
+Block RThroughput: 4.3
+
+
+Instruction Info:
+[1]: #uOps
+[2]: Latency
+[3]: RThroughput
+[4]: MayLoad
+[5]: MayStore
+[6]: HasSideEffects (U)
+
+[1] [2] [3] [4] [5] [6] Instructions:
+ 1 1 0.33 mov eax, 49345
+ 1 1 0.33 cmp rsi, 6
+ 1 1 1.00 jb .LBB5_2
+ 1 5 0.50 * mov eax, dword ptr [rdi + rsi - 6]
+ 1 5 0.50 * movzx ecx, word ptr [rdi + rsi - 2]
+ 1 1 0.50 shl rcx, 32
+ 1 1 0.33 or rcx, rax
+ 1 1 0.33 movzx edx, cx
+ 1 0 0.25 xor eax, eax
+ 1 1 0.33 cmp edx, 49344
+ 2 2 0.67 cmovne rcx, rsi
+ 1 1 0.50 sete al
+ 1 1 0.33 and rcx, -65536
+ 1 1 0.33 xor rax, 49345
+ 1 1 0.33 or rax, rcx
+ 1 1 1.00 U ret
+
+
+Resources:
+[0] - SBDivider
+[1] - SBFPDivider
+[2] - SBPort0
+[3] - SBPort1
+[4] - SBPort4
+[5] - SBPort5
+[6.0] - SBPort23
+[6.1] - SBPort23
+
+
+Resource pressure per iteration:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1]
+ - - 4.66 4.66 - 4.68 1.00 1.00
+
+Resource pressure by instruction:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions:
+ - - 0.32 0.01 - 0.67 - - mov eax, 49345
+ - - 0.62 0.02 - 0.36 - - cmp rsi, 6
+ - - - - - 1.00 - - jb .LBB5_2
+ - - - - - - - 1.00 mov eax, dword ptr [rdi + rsi - 6]
+ - - - - - - 1.00 - movzx ecx, word ptr [rdi + rsi - 2]
+ - - 0.37 - - 0.63 - - shl rcx, 32
+ - - 0.99 0.01 - - - - or rcx, rax
+ - - 1.00 - - - - - movzx edx, cx
+ - - - - - - - - xor eax, eax
+ - - 0.35 0.64 - 0.01 - - cmp edx, 49344
+ - - 1.00 1.00 - - - - cmovne rcx, rsi
+ - - - - - 1.00 - - sete al
+ - - 0.01 0.99 - - - - and rcx, -65536
+ - - - 1.00 - - - - xor rax, 49345
+ - - - 0.99 - 0.01 - - or rax, rcx
+ - - - - - 1.00 - - ret
diff --git a/rust/zerocopy/benches/try_ref_from_bytes_dynamic_padding.rs b/rust/zerocopy/benches/try_ref_from_bytes_dynamic_padding.rs
new file mode 100644
index 0000000..126009c
--- /dev/null
+++ b/rust/zerocopy/benches/try_ref_from_bytes_dynamic_padding.rs
@@ -0,0 +1,7 @@
+#[path = "formats/coco_dynamic_padding.rs"]
+mod format;
+
+#[unsafe(no_mangle)]
+fn bench_try_ref_from_bytes_dynamic_padding(source: &[u8]) -> Option<&format::CocoPacket> {
+ zerocopy::TryFromBytes::try_ref_from_bytes(source).ok()
+}
diff --git a/rust/zerocopy/benches/try_ref_from_bytes_dynamic_padding.x86-64 b/rust/zerocopy/benches/try_ref_from_bytes_dynamic_padding.x86-64
new file mode 100644
index 0000000..217c5fc
--- /dev/null
+++ b/rust/zerocopy/benches/try_ref_from_bytes_dynamic_padding.x86-64
@@ -0,0 +1,24 @@
+bench_try_ref_from_bytes_dynamic_padding:
+ test dil, 3
+ jne .LBB5_4
+ movabs rax, 9223372036854775804
+ and rax, rsi
+ cmp rax, 9
+ jb .LBB5_4
+ add rax, -9
+ movabs rcx, -6148914691236517205
+ mul rcx
+ shr rdx
+ lea rax, [rdx + 2*rdx]
+ or rax, 3
+ add rax, 9
+ cmp rsi, rax
+ jne .LBB5_4
+ cmp word ptr [rdi], -16192
+ je .LBB5_5
+.LBB5_4:
+ xor edi, edi
+ mov rdx, rsi
+.LBB5_5:
+ mov rax, rdi
+ ret
diff --git a/rust/zerocopy/benches/try_ref_from_bytes_dynamic_padding.x86-64.mca b/rust/zerocopy/benches/try_ref_from_bytes_dynamic_padding.x86-64.mca
new file mode 100644
index 0000000..95b993c
--- /dev/null
+++ b/rust/zerocopy/benches/try_ref_from_bytes_dynamic_padding.x86-64.mca
@@ -0,0 +1,81 @@
+Iterations: 100
+Instructions: 2100
+Total Cycles: 709
+Total uOps: 2300
+
+Dispatch Width: 4
+uOps Per Cycle: 3.24
+IPC: 2.96
+Block RThroughput: 5.8
+
+
+Instruction Info:
+[1]: #uOps
+[2]: Latency
+[3]: RThroughput
+[4]: MayLoad
+[5]: MayStore
+[6]: HasSideEffects (U)
+
+[1] [2] [3] [4] [5] [6] Instructions:
+ 1 1 0.33 test dil, 3
+ 1 1 1.00 jne .LBB5_4
+ 1 1 0.33 movabs rax, 9223372036854775804
+ 1 1 0.33 and rax, rsi
+ 1 1 0.33 cmp rax, 9
+ 1 1 1.00 jb .LBB5_4
+ 1 1 0.33 add rax, -9
+ 1 1 0.33 movabs rcx, -6148914691236517205
+ 2 4 1.00 mul rcx
+ 1 1 0.50 shr rdx
+ 1 1 0.50 lea rax, [rdx + 2*rdx]
+ 1 1 0.33 or rax, 3
+ 1 1 0.33 add rax, 9
+ 1 1 0.33 cmp rsi, rax
+ 1 1 1.00 jne .LBB5_4
+ 2 6 0.50 * cmp word ptr [rdi], -16192
+ 1 1 1.00 je .LBB5_5
+ 1 0 0.25 xor edi, edi
+ 1 1 0.33 mov rdx, rsi
+ 1 1 0.33 mov rax, rdi
+ 1 1 1.00 U ret
+
+
+Resources:
+[0] - SBDivider
+[1] - SBFPDivider
+[2] - SBPort0
+[3] - SBPort1
+[4] - SBPort4
+[5] - SBPort5
+[6.0] - SBPort23
+[6.1] - SBPort23
+
+
+Resource pressure per iteration:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1]
+ - - 6.98 6.99 - 7.03 0.50 0.50
+
+Resource pressure by instruction:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions:
+ - - 0.48 0.51 - 0.01 - - test dil, 3
+ - - - - - 1.00 - - jne .LBB5_4
+ - - 0.51 0.49 - - - - movabs rax, 9223372036854775804
+ - - 0.01 0.99 - - - - and rax, rsi
+ - - 0.51 0.49 - - - - cmp rax, 9
+ - - - - - 1.00 - - jb .LBB5_4
+ - - 0.98 - - 0.02 - - add rax, -9
+ - - 0.98 0.02 - - - - movabs rcx, -6148914691236517205
+ - - 1.00 1.00 - - - - mul rcx
+ - - 0.99 - - 0.01 - - shr rdx
+ - - - 1.00 - - - - lea rax, [rdx + 2*rdx]
+ - - - 0.51 - 0.49 - - or rax, 3
+ - - 0.01 0.49 - 0.50 - - add rax, 9
+ - - - 0.02 - 0.98 - - cmp rsi, rax
+ - - - - - 1.00 - - jne .LBB5_4
+ - - 0.51 0.49 - - 0.50 0.50 cmp word ptr [rdi], -16192
+ - - - - - 1.00 - - je .LBB5_5
+ - - - - - - - - xor edi, edi
+ - - 0.50 0.50 - - - - mov rdx, rsi
+ - - 0.50 0.48 - 0.02 - - mov rax, rdi
+ - - - - - 1.00 - - ret
diff --git a/rust/zerocopy/benches/try_ref_from_bytes_dynamic_size.rs b/rust/zerocopy/benches/try_ref_from_bytes_dynamic_size.rs
new file mode 100644
index 0000000..fc3cfba
--- /dev/null
+++ b/rust/zerocopy/benches/try_ref_from_bytes_dynamic_size.rs
@@ -0,0 +1,7 @@
+#[path = "formats/coco_dynamic_size.rs"]
+mod format;
+
+#[unsafe(no_mangle)]
+fn bench_try_ref_from_bytes_dynamic_size(source: &[u8]) -> Option<&format::CocoPacket> {
+ zerocopy::TryFromBytes::try_ref_from_bytes(source).ok()
+}
diff --git a/rust/zerocopy/benches/try_ref_from_bytes_dynamic_size.x86-64 b/rust/zerocopy/benches/try_ref_from_bytes_dynamic_size.x86-64
new file mode 100644
index 0000000..cf67afd
--- /dev/null
+++ b/rust/zerocopy/benches/try_ref_from_bytes_dynamic_size.x86-64
@@ -0,0 +1,22 @@
+bench_try_ref_from_bytes_dynamic_size:
+ mov rdx, rsi
+ mov rax, rdi
+ cmp rsi, 4
+ setb cl
+ or cl, al
+ test cl, 1
+ jne .LBB5_4
+ lea rcx, [rdx - 4]
+ mov rsi, rcx
+ and rsi, -2
+ add rsi, 4
+ cmp rdx, rsi
+ jne .LBB5_4
+ cmp word ptr [rax], -16192
+ jne .LBB5_4
+ shr rcx
+ mov rdx, rcx
+ ret
+.LBB5_4:
+ xor eax, eax
+ ret
diff --git a/rust/zerocopy/benches/try_ref_from_bytes_dynamic_size.x86-64.mca b/rust/zerocopy/benches/try_ref_from_bytes_dynamic_size.x86-64.mca
new file mode 100644
index 0000000..ecd7a18
--- /dev/null
+++ b/rust/zerocopy/benches/try_ref_from_bytes_dynamic_size.x86-64.mca
@@ -0,0 +1,79 @@
+Iterations: 100
+Instructions: 2000
+Total Cycles: 639
+Total uOps: 2100
+
+Dispatch Width: 4
+uOps Per Cycle: 3.29
+IPC: 3.13
+Block RThroughput: 5.3
+
+
+Instruction Info:
+[1]: #uOps
+[2]: Latency
+[3]: RThroughput
+[4]: MayLoad
+[5]: MayStore
+[6]: HasSideEffects (U)
+
+[1] [2] [3] [4] [5] [6] Instructions:
+ 1 1 0.33 mov rdx, rsi
+ 1 1 0.33 mov rax, rdi
+ 1 1 0.33 cmp rsi, 4
+ 1 1 0.50 setb cl
+ 1 1 0.33 or cl, al
+ 1 1 0.33 test cl, 1
+ 1 1 1.00 jne .LBB5_4
+ 1 1 0.50 lea rcx, [rdx - 4]
+ 1 1 0.33 mov rsi, rcx
+ 1 1 0.33 and rsi, -2
+ 1 1 0.33 add rsi, 4
+ 1 1 0.33 cmp rdx, rsi
+ 1 1 1.00 jne .LBB5_4
+ 2 6 0.50 * cmp word ptr [rax], -16192
+ 1 1 1.00 jne .LBB5_4
+ 1 1 0.50 shr rcx
+ 1 1 0.33 mov rdx, rcx
+ 1 1 1.00 U ret
+ 1 0 0.25 xor eax, eax
+ 1 1 1.00 U ret
+
+
+Resources:
+[0] - SBDivider
+[1] - SBFPDivider
+[2] - SBPort0
+[3] - SBPort1
+[4] - SBPort4
+[5] - SBPort5
+[6.0] - SBPort23
+[6.1] - SBPort23
+
+
+Resource pressure per iteration:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1]
+ - - 6.32 6.32 - 6.36 0.50 0.50
+
+Resource pressure by instruction:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions:
+ - - 0.33 0.66 - 0.01 - - mov rdx, rsi
+ - - 0.66 0.34 - - - - mov rax, rdi
+ - - 0.34 0.66 - - - - cmp rsi, 4
+ - - 0.99 - - 0.01 - - setb cl
+ - - 0.01 0.99 - - - - or cl, al
+ - - - 1.00 - - - - test cl, 1
+ - - - - - 1.00 - - jne .LBB5_4
+ - - 0.66 0.34 - - - - lea rcx, [rdx - 4]
+ - - 0.33 0.66 - 0.01 - - mov rsi, rcx
+ - - 1.00 - - - - - and rsi, -2
+ - - 0.66 0.34 - - - - add rsi, 4
+ - - - 1.00 - - - - cmp rdx, rsi
+ - - - - - 1.00 - - jne .LBB5_4
+ - - - - - 1.00 0.50 0.50 cmp word ptr [rax], -16192
+ - - - - - 1.00 - - jne .LBB5_4
+ - - 0.67 - - 0.33 - - shr rcx
+ - - 0.67 0.33 - - - - mov rdx, rcx
+ - - - - - 1.00 - - ret
+ - - - - - - - - xor eax, eax
+ - - - - - 1.00 - - ret
diff --git a/rust/zerocopy/benches/try_ref_from_bytes_static_size.rs b/rust/zerocopy/benches/try_ref_from_bytes_static_size.rs
new file mode 100644
index 0000000..5215571
--- /dev/null
+++ b/rust/zerocopy/benches/try_ref_from_bytes_static_size.rs
@@ -0,0 +1,7 @@
+#[path = "formats/coco_static_size.rs"]
+mod format;
+
+#[unsafe(no_mangle)]
+fn bench_try_ref_from_bytes_static_size(source: &[u8]) -> Option<&format::CocoPacket> {
+ zerocopy::TryFromBytes::try_ref_from_bytes(source).ok()
+}
diff --git a/rust/zerocopy/benches/try_ref_from_bytes_static_size.x86-64 b/rust/zerocopy/benches/try_ref_from_bytes_static_size.x86-64
new file mode 100644
index 0000000..a11f27189
--- /dev/null
+++ b/rust/zerocopy/benches/try_ref_from_bytes_static_size.x86-64
@@ -0,0 +1,13 @@
+bench_try_ref_from_bytes_static_size:
+ mov rax, rdi
+ cmp rsi, 6
+ setne cl
+ or cl, al
+ test cl, 1
+ jne .LBB5_2
+ cmp word ptr [rax], -16192
+ je .LBB5_3
+.LBB5_2:
+ xor eax, eax
+.LBB5_3:
+ ret
diff --git a/rust/zerocopy/benches/try_ref_from_bytes_static_size.x86-64.mca b/rust/zerocopy/benches/try_ref_from_bytes_static_size.x86-64.mca
new file mode 100644
index 0000000..e6bd205
--- /dev/null
+++ b/rust/zerocopy/benches/try_ref_from_bytes_static_size.x86-64.mca
@@ -0,0 +1,59 @@
+Iterations: 100
+Instructions: 1000
+Total Cycles: 308
+Total uOps: 1100
+
+Dispatch Width: 4
+uOps Per Cycle: 3.57
+IPC: 3.25
+Block RThroughput: 3.0
+
+
+Instruction Info:
+[1]: #uOps
+[2]: Latency
+[3]: RThroughput
+[4]: MayLoad
+[5]: MayStore
+[6]: HasSideEffects (U)
+
+[1] [2] [3] [4] [5] [6] Instructions:
+ 1 1 0.33 mov rax, rdi
+ 1 1 0.33 cmp rsi, 6
+ 1 1 0.50 setne cl
+ 1 1 0.33 or cl, al
+ 1 1 0.33 test cl, 1
+ 1 1 1.00 jne .LBB5_2
+ 2 6 0.50 * cmp word ptr [rax], -16192
+ 1 1 1.00 je .LBB5_3
+ 1 0 0.25 xor eax, eax
+ 1 1 1.00 U ret
+
+
+Resources:
+[0] - SBDivider
+[1] - SBFPDivider
+[2] - SBPort0
+[3] - SBPort1
+[4] - SBPort4
+[5] - SBPort5
+[6.0] - SBPort23
+[6.1] - SBPort23
+
+
+Resource pressure per iteration:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1]
+ - - 2.98 2.98 - 3.04 0.50 0.50
+
+Resource pressure by instruction:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions:
+ - - 0.02 0.97 - 0.01 - - mov rax, rdi
+ - - 0.02 0.98 - - - - cmp rsi, 6
+ - - 1.00 - - - - - setne cl
+ - - 0.97 0.02 - 0.01 - - or cl, al
+ - - 0.96 0.03 - 0.01 - - test cl, 1
+ - - - - - 1.00 - - jne .LBB5_2
+ - - 0.01 0.98 - 0.01 0.50 0.50 cmp word ptr [rax], -16192
+ - - - - - 1.00 - - je .LBB5_3
+ - - - - - - - - xor eax, eax
+ - - - - - 1.00 - - ret
diff --git a/rust/zerocopy/benches/try_ref_from_bytes_with_elems_dynamic_padding.rs b/rust/zerocopy/benches/try_ref_from_bytes_with_elems_dynamic_padding.rs
new file mode 100644
index 0000000..8b9e735
--- /dev/null
+++ b/rust/zerocopy/benches/try_ref_from_bytes_with_elems_dynamic_padding.rs
@@ -0,0 +1,10 @@
+#[path = "formats/coco_dynamic_padding.rs"]
+mod format;
+
+#[unsafe(no_mangle)]
+fn bench_try_ref_from_bytes_with_elems_dynamic_padding(
+ source: &[u8],
+ count: usize,
+) -> Option<&format::CocoPacket> {
+ zerocopy::TryFromBytes::try_ref_from_bytes_with_elems(source, count).ok()
+}
diff --git a/rust/zerocopy/benches/try_ref_from_bytes_with_elems_dynamic_padding.x86-64 b/rust/zerocopy/benches/try_ref_from_bytes_with_elems_dynamic_padding.x86-64
new file mode 100644
index 0000000..3ef8d14
--- /dev/null
+++ b/rust/zerocopy/benches/try_ref_from_bytes_with_elems_dynamic_padding.x86-64
@@ -0,0 +1,21 @@
+bench_try_ref_from_bytes_with_elems_dynamic_padding:
+ movabs rax, 3074457345618258598
+ cmp rdx, rax
+ seta cl
+ mov rax, rdi
+ test al, 3
+ setne dil
+ or dil, cl
+ jne .LBB5_3
+ lea rcx, [rdx + 2*rdx]
+ or rcx, 3
+ add rcx, 9
+ cmp rsi, rcx
+ jne .LBB5_3
+ cmp word ptr [rax], -16192
+ je .LBB5_4
+.LBB5_3:
+ xor eax, eax
+ mov rdx, rsi
+.LBB5_4:
+ ret
diff --git a/rust/zerocopy/benches/try_ref_from_bytes_with_elems_dynamic_padding.x86-64.mca b/rust/zerocopy/benches/try_ref_from_bytes_with_elems_dynamic_padding.x86-64.mca
new file mode 100644
index 0000000..8131f3b
--- /dev/null
+++ b/rust/zerocopy/benches/try_ref_from_bytes_with_elems_dynamic_padding.x86-64.mca
@@ -0,0 +1,75 @@
+Iterations: 100
+Instructions: 1800
+Total Cycles: 607
+Total uOps: 2000
+
+Dispatch Width: 4
+uOps Per Cycle: 3.29
+IPC: 2.97
+Block RThroughput: 5.0
+
+
+Instruction Info:
+[1]: #uOps
+[2]: Latency
+[3]: RThroughput
+[4]: MayLoad
+[5]: MayStore
+[6]: HasSideEffects (U)
+
+[1] [2] [3] [4] [5] [6] Instructions:
+ 1 1 0.33 movabs rax, 3074457345618258598
+ 1 1 0.33 cmp rdx, rax
+ 2 2 1.00 seta cl
+ 1 1 0.33 mov rax, rdi
+ 1 1 0.33 test al, 3
+ 1 1 0.50 setne dil
+ 1 1 0.33 or dil, cl
+ 1 1 1.00 jne .LBB5_3
+ 1 1 0.50 lea rcx, [rdx + 2*rdx]
+ 1 1 0.33 or rcx, 3
+ 1 1 0.33 add rcx, 9
+ 1 1 0.33 cmp rsi, rcx
+ 1 1 1.00 jne .LBB5_3
+ 2 6 0.50 * cmp word ptr [rax], -16192
+ 1 1 1.00 je .LBB5_4
+ 1 0 0.25 xor eax, eax
+ 1 1 0.33 mov rdx, rsi
+ 1 1 1.00 U ret
+
+
+Resources:
+[0] - SBDivider
+[1] - SBFPDivider
+[2] - SBPort0
+[3] - SBPort1
+[4] - SBPort4
+[5] - SBPort5
+[6.0] - SBPort23
+[6.1] - SBPort23
+
+
+Resource pressure per iteration:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1]
+ - - 5.99 5.99 - 6.02 0.50 0.50
+
+Resource pressure by instruction:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions:
+ - - - 0.99 - 0.01 - - movabs rax, 3074457345618258598
+ - - - 1.00 - - - - cmp rdx, rax
+ - - - - - 2.00 - - seta cl
+ - - 1.00 - - - - - mov rax, rdi
+ - - 0.99 0.01 - - - - test al, 3
+ - - 1.00 - - - - - setne dil
+ - - - 0.99 - 0.01 - - or dil, cl
+ - - - - - 1.00 - - jne .LBB5_3
+ - - 0.01 0.99 - - - - lea rcx, [rdx + 2*rdx]
+ - - - 1.00 - - - - or rcx, 3
+ - - 0.99 0.01 - - - - add rcx, 9
+ - - - 1.00 - - - - cmp rsi, rcx
+ - - - - - 1.00 - - jne .LBB5_3
+ - - 1.00 - - - 0.50 0.50 cmp word ptr [rax], -16192
+ - - - - - 1.00 - - je .LBB5_4
+ - - - - - - - - xor eax, eax
+ - - 1.00 - - - - - mov rdx, rsi
+ - - - - - 1.00 - - ret
diff --git a/rust/zerocopy/benches/try_ref_from_bytes_with_elems_dynamic_size.rs b/rust/zerocopy/benches/try_ref_from_bytes_with_elems_dynamic_size.rs
new file mode 100644
index 0000000..9ccd6fe
--- /dev/null
+++ b/rust/zerocopy/benches/try_ref_from_bytes_with_elems_dynamic_size.rs
@@ -0,0 +1,10 @@
+#[path = "formats/coco_dynamic_size.rs"]
+mod format;
+
+#[unsafe(no_mangle)]
+fn bench_try_ref_from_bytes_with_elems_dynamic_size(
+ source: &[u8],
+ count: usize,
+) -> Option<&format::CocoPacket> {
+ zerocopy::TryFromBytes::try_ref_from_bytes_with_elems(source, count).ok()
+}
diff --git a/rust/zerocopy/benches/try_ref_from_bytes_with_elems_dynamic_size.x86-64 b/rust/zerocopy/benches/try_ref_from_bytes_with_elems_dynamic_size.x86-64
new file mode 100644
index 0000000..ba34b18
--- /dev/null
+++ b/rust/zerocopy/benches/try_ref_from_bytes_with_elems_dynamic_size.x86-64
@@ -0,0 +1,18 @@
+bench_try_ref_from_bytes_with_elems_dynamic_size:
+ movabs rax, 4611686018427387901
+ cmp rdx, rax
+ seta cl
+ mov rax, rdi
+ or dil, cl
+ test dil, 1
+ jne .LBB5_3
+ lea rcx, [2*rdx + 4]
+ cmp rsi, rcx
+ jne .LBB5_3
+ cmp word ptr [rax], -16192
+ je .LBB5_4
+.LBB5_3:
+ xor eax, eax
+ mov rdx, rsi
+.LBB5_4:
+ ret
diff --git a/rust/zerocopy/benches/try_ref_from_bytes_with_elems_dynamic_size.x86-64.mca b/rust/zerocopy/benches/try_ref_from_bytes_with_elems_dynamic_size.x86-64.mca
new file mode 100644
index 0000000..ae049c0
--- /dev/null
+++ b/rust/zerocopy/benches/try_ref_from_bytes_with_elems_dynamic_size.x86-64.mca
@@ -0,0 +1,69 @@
+Iterations: 100
+Instructions: 1500
+Total Cycles: 507
+Total uOps: 1700
+
+Dispatch Width: 4
+uOps Per Cycle: 3.35
+IPC: 2.96
+Block RThroughput: 4.3
+
+
+Instruction Info:
+[1]: #uOps
+[2]: Latency
+[3]: RThroughput
+[4]: MayLoad
+[5]: MayStore
+[6]: HasSideEffects (U)
+
+[1] [2] [3] [4] [5] [6] Instructions:
+ 1 1 0.33 movabs rax, 4611686018427387901
+ 1 1 0.33 cmp rdx, rax
+ 2 2 1.00 seta cl
+ 1 1 0.33 mov rax, rdi
+ 1 1 0.33 or dil, cl
+ 1 1 0.33 test dil, 1
+ 1 1 1.00 jne .LBB5_3
+ 1 1 0.50 lea rcx, [2*rdx + 4]
+ 1 1 0.33 cmp rsi, rcx
+ 1 1 1.00 jne .LBB5_3
+ 2 6 0.50 * cmp word ptr [rax], -16192
+ 1 1 1.00 je .LBB5_4
+ 1 0 0.25 xor eax, eax
+ 1 1 0.33 mov rdx, rsi
+ 1 1 1.00 U ret
+
+
+Resources:
+[0] - SBDivider
+[1] - SBFPDivider
+[2] - SBPort0
+[3] - SBPort1
+[4] - SBPort4
+[5] - SBPort5
+[6.0] - SBPort23
+[6.1] - SBPort23
+
+
+Resource pressure per iteration:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1]
+ - - 4.98 4.99 - 5.03 0.50 0.50
+
+Resource pressure by instruction:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions:
+ - - - 0.99 - 0.01 - - movabs rax, 4611686018427387901
+ - - 0.50 0.50 - - - - cmp rdx, rax
+ - - 1.96 - - 0.04 - - seta cl
+ - - 0.01 0.99 - - - - mov rax, rdi
+ - - 1.00 - - - - - or dil, cl
+ - - 0.99 0.01 - - - - test dil, 1
+ - - - - - 1.00 - - jne .LBB5_3
+ - - 0.01 0.99 - - - - lea rcx, [2*rdx + 4]
+ - - 0.02 0.49 - 0.49 - - cmp rsi, rcx
+ - - - - - 1.00 - - jne .LBB5_3
+ - - - 0.51 - 0.49 0.50 0.50 cmp word ptr [rax], -16192
+ - - - - - 1.00 - - je .LBB5_4
+ - - - - - - - - xor eax, eax
+ - - 0.49 0.51 - - - - mov rdx, rsi
+ - - - - - 1.00 - - ret
diff --git a/rust/zerocopy/benches/try_ref_from_prefix_dynamic_padding.rs b/rust/zerocopy/benches/try_ref_from_prefix_dynamic_padding.rs
new file mode 100644
index 0000000..23b346f
--- /dev/null
+++ b/rust/zerocopy/benches/try_ref_from_prefix_dynamic_padding.rs
@@ -0,0 +1,10 @@
+#[path = "formats/coco_dynamic_padding.rs"]
+mod format;
+
+#[unsafe(no_mangle)]
+fn bench_try_ref_from_prefix_dynamic_padding(source: &[u8]) -> Option<&format::CocoPacket> {
+ match zerocopy::TryFromBytes::try_ref_from_prefix(source) {
+ Ok((packet, _rest)) => Some(packet),
+ _ => None,
+ }
+}
diff --git a/rust/zerocopy/benches/try_ref_from_prefix_dynamic_padding.x86-64 b/rust/zerocopy/benches/try_ref_from_prefix_dynamic_padding.x86-64
new file mode 100644
index 0000000..d832cb7
--- /dev/null
+++ b/rust/zerocopy/benches/try_ref_from_prefix_dynamic_padding.x86-64
@@ -0,0 +1,29 @@
+bench_try_ref_from_prefix_dynamic_padding:
+ xor edx, edx
+ mov eax, 0
+ test dil, 3
+ je .LBB5_1
+ ret
+.LBB5_1:
+ movabs rax, 9223372036854775804
+ and rsi, rax
+ cmp rsi, 9
+ jae .LBB5_3
+ mov edx, 1
+ xor eax, eax
+ ret
+.LBB5_3:
+ add rsi, -9
+ movabs rcx, -6148914691236517205
+ mov rax, rsi
+ mul rcx
+ mov rax, rdx
+ shr rax
+ movzx ecx, word ptr [rdi]
+ cmp cx, -16192
+ mov edx, 2
+ cmove rdx, rax
+ xor eax, eax
+ cmp ecx, 49344
+ cmove rax, rdi
+ ret
diff --git a/rust/zerocopy/benches/try_ref_from_prefix_dynamic_padding.x86-64.mca b/rust/zerocopy/benches/try_ref_from_prefix_dynamic_padding.x86-64.mca
new file mode 100644
index 0000000..482112a
--- /dev/null
+++ b/rust/zerocopy/benches/try_ref_from_prefix_dynamic_padding.x86-64.mca
@@ -0,0 +1,91 @@
+Iterations: 100
+Instructions: 2600
+Total Cycles: 843
+Total uOps: 2900
+
+Dispatch Width: 4
+uOps Per Cycle: 3.44
+IPC: 3.08
+Block RThroughput: 7.3
+
+
+Instruction Info:
+[1]: #uOps
+[2]: Latency
+[3]: RThroughput
+[4]: MayLoad
+[5]: MayStore
+[6]: HasSideEffects (U)
+
+[1] [2] [3] [4] [5] [6] Instructions:
+ 1 0 0.25 xor edx, edx
+ 1 1 0.33 mov eax, 0
+ 1 1 0.33 test dil, 3
+ 1 1 1.00 je .LBB5_1
+ 1 1 1.00 U ret
+ 1 1 0.33 movabs rax, 9223372036854775804
+ 1 1 0.33 and rsi, rax
+ 1 1 0.33 cmp rsi, 9
+ 1 1 1.00 jae .LBB5_3
+ 1 1 0.33 mov edx, 1
+ 1 0 0.25 xor eax, eax
+ 1 1 1.00 U ret
+ 1 1 0.33 add rsi, -9
+ 1 1 0.33 movabs rcx, -6148914691236517205
+ 1 1 0.33 mov rax, rsi
+ 2 4 1.00 mul rcx
+ 1 1 0.33 mov rax, rdx
+ 1 1 0.50 shr rax
+ 1 5 0.50 * movzx ecx, word ptr [rdi]
+ 1 1 0.33 cmp cx, -16192
+ 1 1 0.33 mov edx, 2
+ 2 2 0.67 cmove rdx, rax
+ 1 0 0.25 xor eax, eax
+ 1 1 0.33 cmp ecx, 49344
+ 2 2 0.67 cmove rax, rdi
+ 1 1 1.00 U ret
+
+
+Resources:
+[0] - SBDivider
+[1] - SBFPDivider
+[2] - SBPort0
+[3] - SBPort1
+[4] - SBPort4
+[5] - SBPort5
+[6.0] - SBPort23
+[6.1] - SBPort23
+
+
+Resource pressure per iteration:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1]
+ - - 8.33 8.33 - 8.34 0.50 0.50
+
+Resource pressure by instruction:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions:
+ - - - - - - - - xor edx, edx
+ - - 0.32 0.34 - 0.34 - - mov eax, 0
+ - - 0.34 0.33 - 0.33 - - test dil, 3
+ - - - - - 1.00 - - je .LBB5_1
+ - - - - - 1.00 - - ret
+ - - 0.35 0.65 - - - - movabs rax, 9223372036854775804
+ - - 0.96 0.03 - 0.01 - - and rsi, rax
+ - - 0.01 0.97 - 0.02 - - cmp rsi, 9
+ - - - - - 1.00 - - jae .LBB5_3
+ - - 0.67 0.01 - 0.32 - - mov edx, 1
+ - - - - - - - - xor eax, eax
+ - - - - - 1.00 - - ret
+ - - 0.02 0.34 - 0.64 - - add rsi, -9
+ - - 0.33 0.66 - 0.01 - - movabs rcx, -6148914691236517205
+ - - 0.66 0.34 - - - - mov rax, rsi
+ - - 1.00 1.00 - - - - mul rcx
+ - - 0.01 0.99 - - - - mov rax, rdx
+ - - 0.99 - - 0.01 - - shr rax
+ - - - - - - 0.50 0.50 movzx ecx, word ptr [rdi]
+ - - 0.33 0.03 - 0.64 - - cmp cx, -16192
+ - - 0.01 0.31 - 0.68 - - mov edx, 2
+ - - 1.00 1.00 - - - - cmove rdx, rax
+ - - - - - - - - xor eax, eax
+ - - 0.33 0.33 - 0.34 - - cmp ecx, 49344
+ - - 1.00 1.00 - - - - cmove rax, rdi
+ - - - - - 1.00 - - ret
diff --git a/rust/zerocopy/benches/try_ref_from_prefix_dynamic_size.rs b/rust/zerocopy/benches/try_ref_from_prefix_dynamic_size.rs
new file mode 100644
index 0000000..41a466e
--- /dev/null
+++ b/rust/zerocopy/benches/try_ref_from_prefix_dynamic_size.rs
@@ -0,0 +1,10 @@
+#[path = "formats/coco_dynamic_size.rs"]
+mod format;
+
+#[unsafe(no_mangle)]
+fn bench_try_ref_from_prefix_dynamic_size(source: &[u8]) -> Option<&format::CocoPacket> {
+ match zerocopy::TryFromBytes::try_ref_from_prefix(source) {
+ Ok((packet, _rest)) => Some(packet),
+ _ => None,
+ }
+}
diff --git a/rust/zerocopy/benches/try_ref_from_prefix_dynamic_size.x86-64 b/rust/zerocopy/benches/try_ref_from_prefix_dynamic_size.x86-64
new file mode 100644
index 0000000..be7f34b
--- /dev/null
+++ b/rust/zerocopy/benches/try_ref_from_prefix_dynamic_size.x86-64
@@ -0,0 +1,22 @@
+bench_try_ref_from_prefix_dynamic_size:
+ xor edx, edx
+ mov eax, 0
+ test dil, 1
+ jne .LBB5_4
+ cmp rsi, 4
+ jae .LBB5_3
+ mov edx, 1
+ xor eax, eax
+ ret
+.LBB5_3:
+ add rsi, -4
+ shr rsi
+ movzx ecx, word ptr [rdi]
+ cmp ecx, 49344
+ mov edx, 2
+ cmove rdx, rsi
+ xor eax, eax
+ cmp cx, -16192
+ cmove rax, rdi
+.LBB5_4:
+ ret
diff --git a/rust/zerocopy/benches/try_ref_from_prefix_dynamic_size.x86-64.mca b/rust/zerocopy/benches/try_ref_from_prefix_dynamic_size.x86-64.mca
new file mode 100644
index 0000000..11706de
--- /dev/null
+++ b/rust/zerocopy/benches/try_ref_from_prefix_dynamic_size.x86-64.mca
@@ -0,0 +1,77 @@
+Iterations: 100
+Instructions: 1900
+Total Cycles: 573
+Total uOps: 2100
+
+Dispatch Width: 4
+uOps Per Cycle: 3.66
+IPC: 3.32
+Block RThroughput: 5.3
+
+
+Instruction Info:
+[1]: #uOps
+[2]: Latency
+[3]: RThroughput
+[4]: MayLoad
+[5]: MayStore
+[6]: HasSideEffects (U)
+
+[1] [2] [3] [4] [5] [6] Instructions:
+ 1 0 0.25 xor edx, edx
+ 1 1 0.33 mov eax, 0
+ 1 1 0.33 test dil, 1
+ 1 1 1.00 jne .LBB5_4
+ 1 1 0.33 cmp rsi, 4
+ 1 1 1.00 jae .LBB5_3
+ 1 1 0.33 mov edx, 1
+ 1 0 0.25 xor eax, eax
+ 1 1 1.00 U ret
+ 1 1 0.33 add rsi, -4
+ 1 1 0.50 shr rsi
+ 1 5 0.50 * movzx ecx, word ptr [rdi]
+ 1 1 0.33 cmp ecx, 49344
+ 1 1 0.33 mov edx, 2
+ 2 2 0.67 cmove rdx, rsi
+ 1 0 0.25 xor eax, eax
+ 1 1 0.33 cmp cx, -16192
+ 2 2 0.67 cmove rax, rdi
+ 1 1 1.00 U ret
+
+
+Resources:
+[0] - SBDivider
+[1] - SBFPDivider
+[2] - SBPort0
+[3] - SBPort1
+[4] - SBPort4
+[5] - SBPort5
+[6.0] - SBPort23
+[6.1] - SBPort23
+
+
+Resource pressure per iteration:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1]
+ - - 5.66 5.67 - 5.67 0.50 0.50
+
+Resource pressure by instruction:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions:
+ - - - - - - - - xor edx, edx
+ - - 0.30 0.37 - 0.33 - - mov eax, 0
+ - - 0.35 0.32 - 0.33 - - test dil, 1
+ - - - - - 1.00 - - jne .LBB5_4
+ - - 0.32 0.33 - 0.35 - - cmp rsi, 4
+ - - - - - 1.00 - - jae .LBB5_3
+ - - 0.33 0.35 - 0.32 - - mov edx, 1
+ - - - - - - - - xor eax, eax
+ - - - - - 1.00 - - ret
+ - - 0.34 0.64 - 0.02 - - add rsi, -4
+ - - 1.00 - - - - - shr rsi
+ - - - - - - 0.50 0.50 movzx ecx, word ptr [rdi]
+ - - 0.60 0.40 - - - - cmp ecx, 49344
+ - - 0.05 0.95 - - - - mov edx, 2
+ - - 1.00 1.00 - - - - cmove rdx, rsi
+ - - - - - - - - xor eax, eax
+ - - 0.37 0.31 - 0.32 - - cmp cx, -16192
+ - - 1.00 1.00 - - - - cmove rax, rdi
+ - - - - - 1.00 - - ret
diff --git a/rust/zerocopy/benches/try_ref_from_prefix_static_size.rs b/rust/zerocopy/benches/try_ref_from_prefix_static_size.rs
new file mode 100644
index 0000000..5f13d48
--- /dev/null
+++ b/rust/zerocopy/benches/try_ref_from_prefix_static_size.rs
@@ -0,0 +1,10 @@
+#[path = "formats/coco_static_size.rs"]
+mod format;
+
+#[unsafe(no_mangle)]
+fn bench_try_ref_from_prefix_static_size(source: &[u8]) -> Option<&format::CocoPacket> {
+ match zerocopy::TryFromBytes::try_ref_from_prefix(source) {
+ Ok((packet, _rest)) => Some(packet),
+ _ => None,
+ }
+}
diff --git a/rust/zerocopy/benches/try_ref_from_prefix_static_size.x86-64 b/rust/zerocopy/benches/try_ref_from_prefix_static_size.x86-64
new file mode 100644
index 0000000..83212f7
--- /dev/null
+++ b/rust/zerocopy/benches/try_ref_from_prefix_static_size.x86-64
@@ -0,0 +1,15 @@
+bench_try_ref_from_prefix_static_size:
+ cmp rsi, 6
+ setb al
+ or al, dil
+ test al, 1
+ jne .LBB5_2
+ movzx eax, word ptr [rdi]
+ cmp eax, 49344
+ mov eax, 2
+ cmove rax, rdi
+ je .LBB5_3
+.LBB5_2:
+ xor eax, eax
+.LBB5_3:
+ ret
diff --git a/rust/zerocopy/benches/try_ref_from_prefix_static_size.x86-64.mca b/rust/zerocopy/benches/try_ref_from_prefix_static_size.x86-64.mca
new file mode 100644
index 0000000..5d02b86
--- /dev/null
+++ b/rust/zerocopy/benches/try_ref_from_prefix_static_size.x86-64.mca
@@ -0,0 +1,63 @@
+Iterations: 100
+Instructions: 1200
+Total Cycles: 374
+Total uOps: 1300
+
+Dispatch Width: 4
+uOps Per Cycle: 3.48
+IPC: 3.21
+Block RThroughput: 3.3
+
+
+Instruction Info:
+[1]: #uOps
+[2]: Latency
+[3]: RThroughput
+[4]: MayLoad
+[5]: MayStore
+[6]: HasSideEffects (U)
+
+[1] [2] [3] [4] [5] [6] Instructions:
+ 1 1 0.33 cmp rsi, 6
+ 1 1 0.50 setb al
+ 1 1 0.33 or al, dil
+ 1 1 0.33 test al, 1
+ 1 1 1.00 jne .LBB5_2
+ 1 5 0.50 * movzx eax, word ptr [rdi]
+ 1 1 0.33 cmp eax, 49344
+ 1 1 0.33 mov eax, 2
+ 2 2 0.67 cmove rax, rdi
+ 1 1 1.00 je .LBB5_3
+ 1 0 0.25 xor eax, eax
+ 1 1 1.00 U ret
+
+
+Resources:
+[0] - SBDivider
+[1] - SBFPDivider
+[2] - SBPort0
+[3] - SBPort1
+[4] - SBPort4
+[5] - SBPort5
+[6.0] - SBPort23
+[6.1] - SBPort23
+
+
+Resource pressure per iteration:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1]
+ - - 3.66 3.65 - 3.69 0.50 0.50
+
+Resource pressure by instruction:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions:
+ - - 0.35 0.64 - 0.01 - - cmp rsi, 6
+ - - 1.00 - - - - - setb al
+ - - 0.02 0.66 - 0.32 - - or al, dil
+ - - 0.03 0.65 - 0.32 - - test al, 1
+ - - - - - 1.00 - - jne .LBB5_2
+ - - - - - - 0.50 0.50 movzx eax, word ptr [rdi]
+ - - 0.92 0.07 - 0.01 - - cmp eax, 49344
+ - - 0.37 0.63 - - - - mov eax, 2
+ - - 0.97 1.00 - 0.03 - - cmove rax, rdi
+ - - - - - 1.00 - - je .LBB5_3
+ - - - - - - - - xor eax, eax
+ - - - - - 1.00 - - ret
diff --git a/rust/zerocopy/benches/try_ref_from_prefix_with_elems_dynamic_padding.rs b/rust/zerocopy/benches/try_ref_from_prefix_with_elems_dynamic_padding.rs
new file mode 100644
index 0000000..1744a40
--- /dev/null
+++ b/rust/zerocopy/benches/try_ref_from_prefix_with_elems_dynamic_padding.rs
@@ -0,0 +1,13 @@
+#[path = "formats/coco_dynamic_padding.rs"]
+mod format;
+
+#[unsafe(no_mangle)]
+fn bench_try_ref_from_prefix_with_elems_dynamic_padding(
+ source: &[u8],
+ count: usize,
+) -> Option<&format::CocoPacket> {
+ match zerocopy::TryFromBytes::try_ref_from_prefix_with_elems(source, count) {
+ Ok((packet, _rest)) => Some(packet),
+ _ => None,
+ }
+}
diff --git a/rust/zerocopy/benches/try_ref_from_prefix_with_elems_dynamic_padding.x86-64 b/rust/zerocopy/benches/try_ref_from_prefix_with_elems_dynamic_padding.x86-64
new file mode 100644
index 0000000..80e66ba
--- /dev/null
+++ b/rust/zerocopy/benches/try_ref_from_prefix_with_elems_dynamic_padding.x86-64
@@ -0,0 +1,30 @@
+bench_try_ref_from_prefix_with_elems_dynamic_padding:
+ movabs rax, 3074457345618258598
+ cmp rdx, rax
+ ja .LBB5_1
+ xor ecx, ecx
+ mov eax, 0
+ test dil, 3
+ je .LBB5_3
+ mov rdx, rcx
+ ret
+.LBB5_3:
+ lea rax, [rdx + 2*rdx]
+ or rax, 3
+ add rax, 9
+ cmp rax, rsi
+ jbe .LBB5_4
+.LBB5_1:
+ xor eax, eax
+ mov edx, 1
+ ret
+.LBB5_4:
+ movzx esi, word ptr [rdi]
+ cmp si, -16192
+ mov ecx, 2
+ cmove rcx, rdx
+ xor eax, eax
+ cmp esi, 49344
+ cmove rax, rdi
+ mov rdx, rcx
+ ret
diff --git a/rust/zerocopy/benches/try_ref_from_prefix_with_elems_dynamic_padding.x86-64.mca b/rust/zerocopy/benches/try_ref_from_prefix_with_elems_dynamic_padding.x86-64.mca
new file mode 100644
index 0000000..512e8ce
--- /dev/null
+++ b/rust/zerocopy/benches/try_ref_from_prefix_with_elems_dynamic_padding.x86-64.mca
@@ -0,0 +1,91 @@
+Iterations: 100
+Instructions: 2600
+Total Cycles: 806
+Total uOps: 2800
+
+Dispatch Width: 4
+uOps Per Cycle: 3.47
+IPC: 3.23
+Block RThroughput: 7.0
+
+
+Instruction Info:
+[1]: #uOps
+[2]: Latency
+[3]: RThroughput
+[4]: MayLoad
+[5]: MayStore
+[6]: HasSideEffects (U)
+
+[1] [2] [3] [4] [5] [6] Instructions:
+ 1 1 0.33 movabs rax, 3074457345618258598
+ 1 1 0.33 cmp rdx, rax
+ 1 1 1.00 ja .LBB5_1
+ 1 0 0.25 xor ecx, ecx
+ 1 1 0.33 mov eax, 0
+ 1 1 0.33 test dil, 3
+ 1 1 1.00 je .LBB5_3
+ 1 1 0.33 mov rdx, rcx
+ 1 1 1.00 U ret
+ 1 1 0.50 lea rax, [rdx + 2*rdx]
+ 1 1 0.33 or rax, 3
+ 1 1 0.33 add rax, 9
+ 1 1 0.33 cmp rax, rsi
+ 1 1 1.00 jbe .LBB5_4
+ 1 0 0.25 xor eax, eax
+ 1 1 0.33 mov edx, 1
+ 1 1 1.00 U ret
+ 1 5 0.50 * movzx esi, word ptr [rdi]
+ 1 1 0.33 cmp si, -16192
+ 1 1 0.33 mov ecx, 2
+ 2 2 0.67 cmove rcx, rdx
+ 1 0 0.25 xor eax, eax
+ 1 1 0.33 cmp esi, 49344
+ 2 2 0.67 cmove rax, rdi
+ 1 1 0.33 mov rdx, rcx
+ 1 1 1.00 U ret
+
+
+Resources:
+[0] - SBDivider
+[1] - SBFPDivider
+[2] - SBPort0
+[3] - SBPort1
+[4] - SBPort4
+[5] - SBPort5
+[6.0] - SBPort23
+[6.1] - SBPort23
+
+
+Resource pressure per iteration:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1]
+ - - 7.98 7.99 - 8.03 0.50 0.50
+
+Resource pressure by instruction:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions:
+ - - 0.98 - - 0.02 - - movabs rax, 3074457345618258598
+ - - - 1.00 - - - - cmp rdx, rax
+ - - - - - 1.00 - - ja .LBB5_1
+ - - - - - - - - xor ecx, ecx
+ - - 0.99 0.01 - - - - mov eax, 0
+ - - 0.01 0.96 - 0.03 - - test dil, 3
+ - - - - - 1.00 - - je .LBB5_3
+ - - 0.97 0.01 - 0.02 - - mov rdx, rcx
+ - - - - - 1.00 - - ret
+ - - 0.03 0.97 - - - - lea rax, [rdx + 2*rdx]
+ - - 0.03 0.97 - - - - or rax, 3
+ - - 0.01 0.99 - - - - add rax, 9
+ - - - 1.00 - - - - cmp rax, rsi
+ - - - - - 1.00 - - jbe .LBB5_4
+ - - - - - - - - xor eax, eax
+ - - 0.98 0.01 - 0.01 - - mov edx, 1
+ - - - - - 1.00 - - ret
+ - - - - - - 0.50 0.50 movzx esi, word ptr [rdi]
+ - - 0.97 0.03 - - - - cmp si, -16192
+ - - 0.98 0.01 - 0.01 - - mov ecx, 2
+ - - 1.00 0.03 - 0.97 - - cmove rcx, rdx
+ - - - - - - - - xor eax, eax
+ - - 0.03 0.97 - - - - cmp esi, 49344
+ - - 1.00 1.00 - - - - cmove rax, rdi
+ - - - 0.03 - 0.97 - - mov rdx, rcx
+ - - - - - 1.00 - - ret
diff --git a/rust/zerocopy/benches/try_ref_from_prefix_with_elems_dynamic_size.rs b/rust/zerocopy/benches/try_ref_from_prefix_with_elems_dynamic_size.rs
new file mode 100644
index 0000000..ed0f509
--- /dev/null
+++ b/rust/zerocopy/benches/try_ref_from_prefix_with_elems_dynamic_size.rs
@@ -0,0 +1,13 @@
+#[path = "formats/coco_dynamic_size.rs"]
+mod format;
+
+#[unsafe(no_mangle)]
+fn bench_try_ref_from_prefix_with_elems_dynamic_size(
+ source: &[u8],
+ count: usize,
+) -> Option<&format::CocoPacket> {
+ match zerocopy::TryFromBytes::try_ref_from_prefix_with_elems(source, count) {
+ Ok((packet, _rest)) => Some(packet),
+ _ => None,
+ }
+}
diff --git a/rust/zerocopy/benches/try_ref_from_prefix_with_elems_dynamic_size.x86-64 b/rust/zerocopy/benches/try_ref_from_prefix_with_elems_dynamic_size.x86-64
new file mode 100644
index 0000000..c12e87c
--- /dev/null
+++ b/rust/zerocopy/benches/try_ref_from_prefix_with_elems_dynamic_size.x86-64
@@ -0,0 +1,26 @@
+bench_try_ref_from_prefix_with_elems_dynamic_size:
+ movabs rax, 4611686018427387901
+ cmp rdx, rax
+ ja .LBB5_1
+ mov rcx, rdx
+ xor edx, edx
+ mov eax, 0
+ test dil, 1
+ jne .LBB5_5
+ lea rax, [2*rcx + 4]
+ cmp rax, rsi
+ jbe .LBB5_4
+.LBB5_1:
+ xor eax, eax
+ mov edx, 1
+ ret
+.LBB5_4:
+ movzx esi, word ptr [rdi]
+ cmp si, -16192
+ mov edx, 2
+ cmove rdx, rcx
+ xor eax, eax
+ cmp esi, 49344
+ cmove rax, rdi
+.LBB5_5:
+ ret
diff --git a/rust/zerocopy/benches/try_ref_from_prefix_with_elems_dynamic_size.x86-64.mca b/rust/zerocopy/benches/try_ref_from_prefix_with_elems_dynamic_size.x86-64.mca
new file mode 100644
index 0000000..6c3f1a1
--- /dev/null
+++ b/rust/zerocopy/benches/try_ref_from_prefix_with_elems_dynamic_size.x86-64.mca
@@ -0,0 +1,83 @@
+Iterations: 100
+Instructions: 2200
+Total Cycles: 674
+Total uOps: 2400
+
+Dispatch Width: 4
+uOps Per Cycle: 3.56
+IPC: 3.26
+Block RThroughput: 6.0
+
+
+Instruction Info:
+[1]: #uOps
+[2]: Latency
+[3]: RThroughput
+[4]: MayLoad
+[5]: MayStore
+[6]: HasSideEffects (U)
+
+[1] [2] [3] [4] [5] [6] Instructions:
+ 1 1 0.33 movabs rax, 4611686018427387901
+ 1 1 0.33 cmp rdx, rax
+ 1 1 1.00 ja .LBB5_1
+ 1 1 0.33 mov rcx, rdx
+ 1 0 0.25 xor edx, edx
+ 1 1 0.33 mov eax, 0
+ 1 1 0.33 test dil, 1
+ 1 1 1.00 jne .LBB5_5
+ 1 1 0.50 lea rax, [2*rcx + 4]
+ 1 1 0.33 cmp rax, rsi
+ 1 1 1.00 jbe .LBB5_4
+ 1 0 0.25 xor eax, eax
+ 1 1 0.33 mov edx, 1
+ 1 1 1.00 U ret
+ 1 5 0.50 * movzx esi, word ptr [rdi]
+ 1 1 0.33 cmp si, -16192
+ 1 1 0.33 mov edx, 2
+ 2 2 0.67 cmove rdx, rcx
+ 1 0 0.25 xor eax, eax
+ 1 1 0.33 cmp esi, 49344
+ 2 2 0.67 cmove rax, rdi
+ 1 1 1.00 U ret
+
+
+Resources:
+[0] - SBDivider
+[1] - SBFPDivider
+[2] - SBPort0
+[3] - SBPort1
+[4] - SBPort4
+[5] - SBPort5
+[6.0] - SBPort23
+[6.1] - SBPort23
+
+
+Resource pressure per iteration:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1]
+ - - 6.65 6.66 - 6.69 0.50 0.50
+
+Resource pressure by instruction:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions:
+ - - 0.66 0.33 - 0.01 - - movabs rax, 4611686018427387901
+ - - 0.02 0.66 - 0.32 - - cmp rdx, rax
+ - - - - - 1.00 - - ja .LBB5_1
+ - - 0.66 0.33 - 0.01 - - mov rcx, rdx
+ - - - - - - - - xor edx, edx
+ - - 0.33 0.01 - 0.66 - - mov eax, 0
+ - - 0.34 0.65 - 0.01 - - test dil, 1
+ - - - - - 1.00 - - jne .LBB5_5
+ - - 0.65 0.35 - - - - lea rax, [2*rcx + 4]
+ - - - 1.00 - - - - cmp rax, rsi
+ - - - - - 1.00 - - jbe .LBB5_4
+ - - - - - - - - xor eax, eax
+ - - 0.34 0.01 - 0.65 - - mov edx, 1
+ - - - - - 1.00 - - ret
+ - - - - - - 0.50 0.50 movzx esi, word ptr [rdi]
+ - - 0.65 0.34 - 0.01 - - cmp si, -16192
+ - - 0.66 0.34 - - - - mov edx, 2
+ - - 1.00 0.99 - 0.01 - - cmove rdx, rcx
+ - - - - - - - - xor eax, eax
+ - - 0.34 0.66 - - - - cmp esi, 49344
+ - - 1.00 0.99 - 0.01 - - cmove rax, rdi
+ - - - - - 1.00 - - ret
diff --git a/rust/zerocopy/benches/try_ref_from_suffix_dynamic_padding.rs b/rust/zerocopy/benches/try_ref_from_suffix_dynamic_padding.rs
new file mode 100644
index 0000000..981feca
--- /dev/null
+++ b/rust/zerocopy/benches/try_ref_from_suffix_dynamic_padding.rs
@@ -0,0 +1,10 @@
+#[path = "formats/coco_dynamic_padding.rs"]
+mod format;
+
+#[unsafe(no_mangle)]
+fn bench_try_ref_from_suffix_dynamic_padding(source: &[u8]) -> Option<&format::CocoPacket> {
+ match zerocopy::TryFromBytes::try_ref_from_suffix(source) {
+ Ok((_rest, packet)) => Some(packet),
+ _ => None,
+ }
+}
diff --git a/rust/zerocopy/benches/try_ref_from_suffix_dynamic_padding.x86-64 b/rust/zerocopy/benches/try_ref_from_suffix_dynamic_padding.x86-64
new file mode 100644
index 0000000..b3e9244
--- /dev/null
+++ b/rust/zerocopy/benches/try_ref_from_suffix_dynamic_padding.x86-64
@@ -0,0 +1,26 @@
+bench_try_ref_from_suffix_dynamic_padding:
+ lea eax, [rsi + rdi]
+ test al, 3
+ jne .LBB5_1
+ movabs rax, 9223372036854775804
+ and rax, rsi
+ cmp rax, 9
+ jae .LBB5_3
+.LBB5_1:
+ xor eax, eax
+ ret
+.LBB5_3:
+ add rax, -9
+ movabs rcx, -6148914691236517205
+ mul rcx
+ shr rdx
+ lea rcx, [rdx + 2*rdx]
+ sub rsi, rcx
+ or rcx, -4
+ add rsi, rdi
+ lea rdi, [rcx + rsi]
+ add rdi, -8
+ xor eax, eax
+ cmp word ptr [rcx + rsi - 8], -16192
+ cmove rax, rdi
+ ret
diff --git a/rust/zerocopy/benches/try_ref_from_suffix_dynamic_padding.x86-64.mca b/rust/zerocopy/benches/try_ref_from_suffix_dynamic_padding.x86-64.mca
new file mode 100644
index 0000000..d56ae56
--- /dev/null
+++ b/rust/zerocopy/benches/try_ref_from_suffix_dynamic_padding.x86-64.mca
@@ -0,0 +1,85 @@
+Iterations: 100
+Instructions: 2300
+Total Cycles: 791
+Total uOps: 2600
+
+Dispatch Width: 4
+uOps Per Cycle: 3.29
+IPC: 2.91
+Block RThroughput: 6.5
+
+
+Instruction Info:
+[1]: #uOps
+[2]: Latency
+[3]: RThroughput
+[4]: MayLoad
+[5]: MayStore
+[6]: HasSideEffects (U)
+
+[1] [2] [3] [4] [5] [6] Instructions:
+ 1 1 0.50 lea eax, [rsi + rdi]
+ 1 1 0.33 test al, 3
+ 1 1 1.00 jne .LBB5_1
+ 1 1 0.33 movabs rax, 9223372036854775804
+ 1 1 0.33 and rax, rsi
+ 1 1 0.33 cmp rax, 9
+ 1 1 1.00 jae .LBB5_3
+ 1 0 0.25 xor eax, eax
+ 1 1 1.00 U ret
+ 1 1 0.33 add rax, -9
+ 1 1 0.33 movabs rcx, -6148914691236517205
+ 2 4 1.00 mul rcx
+ 1 1 0.50 shr rdx
+ 1 1 0.50 lea rcx, [rdx + 2*rdx]
+ 1 1 0.33 sub rsi, rcx
+ 1 1 0.33 or rcx, -4
+ 1 1 0.33 add rsi, rdi
+ 1 1 0.50 lea rdi, [rcx + rsi]
+ 1 1 0.33 add rdi, -8
+ 1 0 0.25 xor eax, eax
+ 2 6 0.50 * cmp word ptr [rcx + rsi - 8], -16192
+ 2 2 0.67 cmove rax, rdi
+ 1 1 1.00 U ret
+
+
+Resources:
+[0] - SBDivider
+[1] - SBFPDivider
+[2] - SBPort0
+[3] - SBPort1
+[4] - SBPort4
+[5] - SBPort5
+[6.0] - SBPort23
+[6.1] - SBPort23
+
+
+Resource pressure per iteration:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1]
+ - - 7.70 7.58 - 7.72 0.50 0.50
+
+Resource pressure by instruction:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions:
+ - - 0.26 0.74 - - - - lea eax, [rsi + rdi]
+ - - 0.19 0.28 - 0.53 - - test al, 3
+ - - - - - 1.00 - - jne .LBB5_1
+ - - 0.93 0.06 - 0.01 - - movabs rax, 9223372036854775804
+ - - 0.81 0.14 - 0.05 - - and rax, rsi
+ - - 0.55 0.43 - 0.02 - - cmp rax, 9
+ - - - - - 1.00 - - jae .LBB5_3
+ - - - - - - - - xor eax, eax
+ - - - - - 1.00 - - ret
+ - - 0.42 0.56 - 0.02 - - add rax, -9
+ - - 0.67 0.30 - 0.03 - - movabs rcx, -6148914691236517205
+ - - 1.00 1.00 - - - - mul rcx
+ - - 0.71 - - 0.29 - - shr rdx
+ - - 0.32 0.68 - - - - lea rcx, [rdx + 2*rdx]
+ - - 0.57 0.04 - 0.39 - - sub rsi, rcx
+ - - 0.28 0.67 - 0.05 - - or rcx, -4
+ - - 0.29 0.29 - 0.42 - - add rsi, rdi
+ - - 0.02 0.98 - - - - lea rdi, [rcx + rsi]
+ - - 0.02 0.41 - 0.57 - - add rdi, -8
+ - - - - - - - - xor eax, eax
+ - - 0.57 0.01 - 0.42 0.50 0.50 cmp word ptr [rcx + rsi - 8], -16192
+ - - 0.09 0.99 - 0.92 - - cmove rax, rdi
+ - - - - - 1.00 - - ret
diff --git a/rust/zerocopy/benches/try_ref_from_suffix_dynamic_size.rs b/rust/zerocopy/benches/try_ref_from_suffix_dynamic_size.rs
new file mode 100644
index 0000000..c3d75ac
--- /dev/null
+++ b/rust/zerocopy/benches/try_ref_from_suffix_dynamic_size.rs
@@ -0,0 +1,10 @@
+#[path = "formats/coco_dynamic_size.rs"]
+mod format;
+
+#[unsafe(no_mangle)]
+fn bench_try_ref_from_suffix_dynamic_size(source: &[u8]) -> Option<&format::CocoPacket> {
+ match zerocopy::TryFromBytes::try_ref_from_suffix(source) {
+ Ok((_rest, packet)) => Some(packet),
+ _ => None,
+ }
+}
diff --git a/rust/zerocopy/benches/try_ref_from_suffix_dynamic_size.x86-64 b/rust/zerocopy/benches/try_ref_from_suffix_dynamic_size.x86-64
new file mode 100644
index 0000000..d51f781
--- /dev/null
+++ b/rust/zerocopy/benches/try_ref_from_suffix_dynamic_size.x86-64
@@ -0,0 +1,18 @@
+bench_try_ref_from_suffix_dynamic_size:
+ lea eax, [rsi + rdi]
+ cmp rsi, 4
+ setb cl
+ or cl, al
+ test cl, 1
+ je .LBB5_2
+ xor eax, eax
+ ret
+.LBB5_2:
+ lea rdx, [rsi - 4]
+ shr rdx
+ and esi, 1
+ lea rcx, [rdi + rsi]
+ xor eax, eax
+ cmp word ptr [rdi + rsi], -16192
+ cmove rax, rcx
+ ret
diff --git a/rust/zerocopy/benches/try_ref_from_suffix_dynamic_size.x86-64.mca b/rust/zerocopy/benches/try_ref_from_suffix_dynamic_size.x86-64.mca
new file mode 100644
index 0000000..6cf7f8e
--- /dev/null
+++ b/rust/zerocopy/benches/try_ref_from_suffix_dynamic_size.x86-64.mca
@@ -0,0 +1,71 @@
+Iterations: 100
+Instructions: 1600
+Total Cycles: 510
+Total uOps: 1800
+
+Dispatch Width: 4
+uOps Per Cycle: 3.53
+IPC: 3.14
+Block RThroughput: 4.5
+
+
+Instruction Info:
+[1]: #uOps
+[2]: Latency
+[3]: RThroughput
+[4]: MayLoad
+[5]: MayStore
+[6]: HasSideEffects (U)
+
+[1] [2] [3] [4] [5] [6] Instructions:
+ 1 1 0.50 lea eax, [rsi + rdi]
+ 1 1 0.33 cmp rsi, 4
+ 1 1 0.50 setb cl
+ 1 1 0.33 or cl, al
+ 1 1 0.33 test cl, 1
+ 1 1 1.00 je .LBB5_2
+ 1 0 0.25 xor eax, eax
+ 1 1 1.00 U ret
+ 1 1 0.50 lea rdx, [rsi - 4]
+ 1 1 0.50 shr rdx
+ 1 1 0.33 and esi, 1
+ 1 1 0.50 lea rcx, [rdi + rsi]
+ 1 0 0.25 xor eax, eax
+ 2 6 0.50 * cmp word ptr [rdi + rsi], -16192
+ 2 2 0.67 cmove rax, rcx
+ 1 1 1.00 U ret
+
+
+Resources:
+[0] - SBDivider
+[1] - SBFPDivider
+[2] - SBPort0
+[3] - SBPort1
+[4] - SBPort4
+[5] - SBPort5
+[6.0] - SBPort23
+[6.1] - SBPort23
+
+
+Resource pressure per iteration:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1]
+ - - 4.99 5.00 - 5.01 0.50 0.50
+
+Resource pressure by instruction:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions:
+ - - 0.98 0.02 - - - - lea eax, [rsi + rdi]
+ - - - 0.98 - 0.02 - - cmp rsi, 4
+ - - 1.00 - - - - - setb cl
+ - - 0.01 0.99 - - - - or cl, al
+ - - 0.01 0.07 - 0.92 - - test cl, 1
+ - - - - - 1.00 - - je .LBB5_2
+ - - - - - - - - xor eax, eax
+ - - - - - 1.00 - - ret
+ - - 0.93 0.07 - - - - lea rdx, [rsi - 4]
+ - - 0.93 - - 0.07 - - shr rdx
+ - - 0.06 0.93 - 0.01 - - and esi, 1
+ - - 0.07 0.93 - - - - lea rcx, [rdi + rsi]
+ - - - - - - - - xor eax, eax
+ - - - 0.01 - 0.99 0.50 0.50 cmp word ptr [rdi + rsi], -16192
+ - - 1.00 1.00 - - - - cmove rax, rcx
+ - - - - - 1.00 - - ret
diff --git a/rust/zerocopy/benches/try_ref_from_suffix_static_size.rs b/rust/zerocopy/benches/try_ref_from_suffix_static_size.rs
new file mode 100644
index 0000000..d4b92f6
--- /dev/null
+++ b/rust/zerocopy/benches/try_ref_from_suffix_static_size.rs
@@ -0,0 +1,10 @@
+#[path = "formats/coco_static_size.rs"]
+mod format;
+
+#[unsafe(no_mangle)]
+fn bench_try_ref_from_suffix_static_size(source: &[u8]) -> Option<&format::CocoPacket> {
+ match zerocopy::TryFromBytes::try_ref_from_suffix(source) {
+ Ok((_rest, packet)) => Some(packet),
+ _ => None,
+ }
+}
diff --git a/rust/zerocopy/benches/try_ref_from_suffix_static_size.x86-64 b/rust/zerocopy/benches/try_ref_from_suffix_static_size.x86-64
new file mode 100644
index 0000000..cd39f70
--- /dev/null
+++ b/rust/zerocopy/benches/try_ref_from_suffix_static_size.x86-64
@@ -0,0 +1,16 @@
+bench_try_ref_from_suffix_static_size:
+ lea eax, [rsi + rdi]
+ cmp rsi, 6
+ setb cl
+ or cl, al
+ test cl, 1
+ je .LBB5_2
+ xor eax, eax
+ ret
+.LBB5_2:
+ lea rcx, [rdi + rsi]
+ add rcx, -6
+ xor eax, eax
+ cmp word ptr [rdi + rsi - 6], -16192
+ cmove rax, rcx
+ ret
diff --git a/rust/zerocopy/benches/try_ref_from_suffix_static_size.x86-64.mca b/rust/zerocopy/benches/try_ref_from_suffix_static_size.x86-64.mca
new file mode 100644
index 0000000..087d1e7
--- /dev/null
+++ b/rust/zerocopy/benches/try_ref_from_suffix_static_size.x86-64.mca
@@ -0,0 +1,67 @@
+Iterations: 100
+Instructions: 1400
+Total Cycles: 443
+Total uOps: 1600
+
+Dispatch Width: 4
+uOps Per Cycle: 3.61
+IPC: 3.16
+Block RThroughput: 4.0
+
+
+Instruction Info:
+[1]: #uOps
+[2]: Latency
+[3]: RThroughput
+[4]: MayLoad
+[5]: MayStore
+[6]: HasSideEffects (U)
+
+[1] [2] [3] [4] [5] [6] Instructions:
+ 1 1 0.50 lea eax, [rsi + rdi]
+ 1 1 0.33 cmp rsi, 6
+ 1 1 0.50 setb cl
+ 1 1 0.33 or cl, al
+ 1 1 0.33 test cl, 1
+ 1 1 1.00 je .LBB5_2
+ 1 0 0.25 xor eax, eax
+ 1 1 1.00 U ret
+ 1 1 0.50 lea rcx, [rdi + rsi]
+ 1 1 0.33 add rcx, -6
+ 1 0 0.25 xor eax, eax
+ 2 6 0.50 * cmp word ptr [rdi + rsi - 6], -16192
+ 2 2 0.67 cmove rax, rcx
+ 1 1 1.00 U ret
+
+
+Resources:
+[0] - SBDivider
+[1] - SBFPDivider
+[2] - SBPort0
+[3] - SBPort1
+[4] - SBPort4
+[5] - SBPort5
+[6.0] - SBPort23
+[6.1] - SBPort23
+
+
+Resource pressure per iteration:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1]
+ - - 4.33 4.33 - 4.34 0.50 0.50
+
+Resource pressure by instruction:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions:
+ - - 0.32 0.68 - - - - lea eax, [rsi + rdi]
+ - - 0.05 0.94 - 0.01 - - cmp rsi, 6
+ - - 1.00 - - - - - setb cl
+ - - 0.95 0.05 - - - - or cl, al
+ - - 0.95 0.02 - 0.03 - - test cl, 1
+ - - - - - 1.00 - - je .LBB5_2
+ - - - - - - - - xor eax, eax
+ - - - - - 1.00 - - ret
+ - - 0.04 0.96 - - - - lea rcx, [rdi + rsi]
+ - - 0.02 0.97 - 0.01 - - add rcx, -6
+ - - - - - - - - xor eax, eax
+ - - 0.03 0.66 - 0.31 0.50 0.50 cmp word ptr [rdi + rsi - 6], -16192
+ - - 0.97 0.05 - 0.98 - - cmove rax, rcx
+ - - - - - 1.00 - - ret
diff --git a/rust/zerocopy/benches/try_ref_from_suffix_with_elems_dynamic_padding.rs b/rust/zerocopy/benches/try_ref_from_suffix_with_elems_dynamic_padding.rs
new file mode 100644
index 0000000..1da455c
--- /dev/null
+++ b/rust/zerocopy/benches/try_ref_from_suffix_with_elems_dynamic_padding.rs
@@ -0,0 +1,13 @@
+#[path = "formats/coco_dynamic_padding.rs"]
+mod format;
+
+#[unsafe(no_mangle)]
+fn bench_try_ref_from_suffix_with_elems_dynamic_padding(
+ source: &[u8],
+ count: usize,
+) -> Option<&format::CocoPacket> {
+ match zerocopy::TryFromBytes::try_ref_from_suffix_with_elems(source, count) {
+ Ok((_rest, packet)) => Some(packet),
+ _ => None,
+ }
+}
diff --git a/rust/zerocopy/benches/try_ref_from_suffix_with_elems_dynamic_padding.x86-64 b/rust/zerocopy/benches/try_ref_from_suffix_with_elems_dynamic_padding.x86-64
new file mode 100644
index 0000000..c7530d8
--- /dev/null
+++ b/rust/zerocopy/benches/try_ref_from_suffix_with_elems_dynamic_padding.x86-64
@@ -0,0 +1,32 @@
+bench_try_ref_from_suffix_with_elems_dynamic_padding:
+ movabs rax, 3074457345618258598
+ cmp rdx, rax
+ ja .LBB5_1
+ lea r8d, [rsi + rdi]
+ xor ecx, ecx
+ mov eax, 0
+ test r8b, 3
+ je .LBB5_3
+ mov rdx, rcx
+ ret
+.LBB5_3:
+ lea rax, [rdx + 2*rdx]
+ or rax, 3
+ add rax, 9
+ sub rsi, rax
+ jae .LBB5_4
+.LBB5_1:
+ xor eax, eax
+ mov edx, 1
+ ret
+.LBB5_4:
+ lea r8, [rdi + rsi]
+ movzx esi, word ptr [rdi + rsi]
+ cmp si, -16192
+ mov ecx, 2
+ cmove rcx, rdx
+ xor eax, eax
+ cmp esi, 49344
+ cmove rax, r8
+ mov rdx, rcx
+ ret
diff --git a/rust/zerocopy/benches/try_ref_from_suffix_with_elems_dynamic_padding.x86-64.mca b/rust/zerocopy/benches/try_ref_from_suffix_with_elems_dynamic_padding.x86-64.mca
new file mode 100644
index 0000000..be736c0
--- /dev/null
+++ b/rust/zerocopy/benches/try_ref_from_suffix_with_elems_dynamic_padding.x86-64.mca
@@ -0,0 +1,95 @@
+Iterations: 100
+Instructions: 2800
+Total Cycles: 878
+Total uOps: 3000
+
+Dispatch Width: 4
+uOps Per Cycle: 3.42
+IPC: 3.19
+Block RThroughput: 7.5
+
+
+Instruction Info:
+[1]: #uOps
+[2]: Latency
+[3]: RThroughput
+[4]: MayLoad
+[5]: MayStore
+[6]: HasSideEffects (U)
+
+[1] [2] [3] [4] [5] [6] Instructions:
+ 1 1 0.33 movabs rax, 3074457345618258598
+ 1 1 0.33 cmp rdx, rax
+ 1 1 1.00 ja .LBB5_1
+ 1 1 0.50 lea r8d, [rsi + rdi]
+ 1 0 0.25 xor ecx, ecx
+ 1 1 0.33 mov eax, 0
+ 1 1 0.33 test r8b, 3
+ 1 1 1.00 je .LBB5_3
+ 1 1 0.33 mov rdx, rcx
+ 1 1 1.00 U ret
+ 1 1 0.50 lea rax, [rdx + 2*rdx]
+ 1 1 0.33 or rax, 3
+ 1 1 0.33 add rax, 9
+ 1 1 0.33 sub rsi, rax
+ 1 1 1.00 jae .LBB5_4
+ 1 0 0.25 xor eax, eax
+ 1 1 0.33 mov edx, 1
+ 1 1 1.00 U ret
+ 1 1 0.50 lea r8, [rdi + rsi]
+ 1 5 0.50 * movzx esi, word ptr [rdi + rsi]
+ 1 1 0.33 cmp si, -16192
+ 1 1 0.33 mov ecx, 2
+ 2 2 0.67 cmove rcx, rdx
+ 1 0 0.25 xor eax, eax
+ 1 1 0.33 cmp esi, 49344
+ 2 2 0.67 cmove rax, r8
+ 1 1 0.33 mov rdx, rcx
+ 1 1 1.00 U ret
+
+
+Resources:
+[0] - SBDivider
+[1] - SBFPDivider
+[2] - SBPort0
+[3] - SBPort1
+[4] - SBPort4
+[5] - SBPort5
+[6.0] - SBPort23
+[6.1] - SBPort23
+
+
+Resource pressure per iteration:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1]
+ - - 8.65 8.65 - 8.70 0.50 0.50
+
+Resource pressure by instruction:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions:
+ - - 0.67 0.30 - 0.03 - - movabs rax, 3074457345618258598
+ - - 0.01 0.99 - - - - cmp rdx, rax
+ - - - - - 1.00 - - ja .LBB5_1
+ - - 0.99 0.01 - - - - lea r8d, [rsi + rdi]
+ - - - - - - - - xor ecx, ecx
+ - - 0.35 0.62 - 0.03 - - mov eax, 0
+ - - 0.99 0.01 - - - - test r8b, 3
+ - - - - - 1.00 - - je .LBB5_3
+ - - 0.68 0.30 - 0.02 - - mov rdx, rcx
+ - - - - - 1.00 - - ret
+ - - 0.07 0.93 - - - - lea rax, [rdx + 2*rdx]
+ - - 0.06 0.35 - 0.59 - - or rax, 3
+ - - 0.02 0.07 - 0.91 - - add rax, 9
+ - - 0.01 0.04 - 0.95 - - sub rsi, rax
+ - - - - - 1.00 - - jae .LBB5_4
+ - - - - - - - - xor eax, eax
+ - - 0.92 0.01 - 0.07 - - mov edx, 1
+ - - - - - 1.00 - - ret
+ - - - 1.00 - - - - lea r8, [rdi + rsi]
+ - - - - - - 0.50 0.50 movzx esi, word ptr [rdi + rsi]
+ - - 0.01 0.99 - - - - cmp si, -16192
+ - - 0.88 0.04 - 0.08 - - mov ecx, 2
+ - - 1.00 0.99 - 0.01 - - cmove rcx, rdx
+ - - - - - - - - xor eax, eax
+ - - 0.99 0.01 - - - - cmp esi, 49344
+ - - 1.00 1.00 - - - - cmove rax, r8
+ - - - 0.99 - 0.01 - - mov rdx, rcx
+ - - - - - 1.00 - - ret
diff --git a/rust/zerocopy/benches/try_ref_from_suffix_with_elems_dynamic_size.rs b/rust/zerocopy/benches/try_ref_from_suffix_with_elems_dynamic_size.rs
new file mode 100644
index 0000000..8c2b80f
--- /dev/null
+++ b/rust/zerocopy/benches/try_ref_from_suffix_with_elems_dynamic_size.rs
@@ -0,0 +1,13 @@
+#[path = "formats/coco_dynamic_size.rs"]
+mod format;
+
+#[unsafe(no_mangle)]
+fn bench_try_ref_from_suffix_with_elems_dynamic_size(
+ source: &[u8],
+ count: usize,
+) -> Option<&format::CocoPacket> {
+ match zerocopy::TryFromBytes::try_ref_from_suffix_with_elems(source, count) {
+ Ok((_rest, packet)) => Some(packet),
+ _ => None,
+ }
+}
diff --git a/rust/zerocopy/benches/try_ref_from_suffix_with_elems_dynamic_size.x86-64 b/rust/zerocopy/benches/try_ref_from_suffix_with_elems_dynamic_size.x86-64
new file mode 100644
index 0000000..952eb12
--- /dev/null
+++ b/rust/zerocopy/benches/try_ref_from_suffix_with_elems_dynamic_size.x86-64
@@ -0,0 +1,28 @@
+bench_try_ref_from_suffix_with_elems_dynamic_size:
+ movabs rax, 4611686018427387901
+ cmp rdx, rax
+ ja .LBB5_1
+ lea r8d, [rsi + rdi]
+ xor ecx, ecx
+ mov eax, 0
+ test r8b, 1
+ jne .LBB5_5
+ lea rax, [2*rdx + 4]
+ sub rsi, rax
+ jae .LBB5_4
+.LBB5_1:
+ xor eax, eax
+ mov edx, 1
+ ret
+.LBB5_4:
+ lea r8, [rdi + rsi]
+ movzx esi, word ptr [rdi + rsi]
+ cmp si, -16192
+ mov ecx, 2
+ cmove rcx, rdx
+ xor eax, eax
+ cmp esi, 49344
+ cmove rax, r8
+.LBB5_5:
+ mov rdx, rcx
+ ret
diff --git a/rust/zerocopy/benches/try_ref_from_suffix_with_elems_dynamic_size.x86-64.mca b/rust/zerocopy/benches/try_ref_from_suffix_with_elems_dynamic_size.x86-64.mca
new file mode 100644
index 0000000..d4f78f6
--- /dev/null
+++ b/rust/zerocopy/benches/try_ref_from_suffix_with_elems_dynamic_size.x86-64.mca
@@ -0,0 +1,87 @@
+Iterations: 100
+Instructions: 2400
+Total Cycles: 1107
+Total uOps: 2600
+
+Dispatch Width: 4
+uOps Per Cycle: 2.35
+IPC: 2.17
+Block RThroughput: 6.5
+
+
+Instruction Info:
+[1]: #uOps
+[2]: Latency
+[3]: RThroughput
+[4]: MayLoad
+[5]: MayStore
+[6]: HasSideEffects (U)
+
+[1] [2] [3] [4] [5] [6] Instructions:
+ 1 1 0.33 movabs rax, 4611686018427387901
+ 1 1 0.33 cmp rdx, rax
+ 1 1 1.00 ja .LBB5_1
+ 1 1 0.50 lea r8d, [rsi + rdi]
+ 1 0 0.25 xor ecx, ecx
+ 1 1 0.33 mov eax, 0
+ 1 1 0.33 test r8b, 1
+ 1 1 1.00 jne .LBB5_5
+ 1 1 0.50 lea rax, [2*rdx + 4]
+ 1 1 0.33 sub rsi, rax
+ 1 1 1.00 jae .LBB5_4
+ 1 0 0.25 xor eax, eax
+ 1 1 0.33 mov edx, 1
+ 1 1 1.00 U ret
+ 1 1 0.50 lea r8, [rdi + rsi]
+ 1 5 0.50 * movzx esi, word ptr [rdi + rsi]
+ 1 1 0.33 cmp si, -16192
+ 1 1 0.33 mov ecx, 2
+ 2 2 0.67 cmove rcx, rdx
+ 1 0 0.25 xor eax, eax
+ 1 1 0.33 cmp esi, 49344
+ 2 2 0.67 cmove rax, r8
+ 1 1 0.33 mov rdx, rcx
+ 1 1 1.00 U ret
+
+
+Resources:
+[0] - SBDivider
+[1] - SBFPDivider
+[2] - SBPort0
+[3] - SBPort1
+[4] - SBPort4
+[5] - SBPort5
+[6.0] - SBPort23
+[6.1] - SBPort23
+
+
+Resource pressure per iteration:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1]
+ - - 6.99 7.00 - 8.01 0.50 0.50
+
+Resource pressure by instruction:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions:
+ - - 0.02 0.95 - 0.03 - - movabs rax, 4611686018427387901
+ - - 0.93 0.04 - 0.03 - - cmp rdx, rax
+ - - - - - 1.00 - - ja .LBB5_1
+ - - 0.96 0.04 - - - - lea r8d, [rsi + rdi]
+ - - - - - - - - xor ecx, ecx
+ - - 0.95 0.02 - 0.03 - - mov eax, 0
+ - - 0.95 0.05 - - - - test r8b, 1
+ - - - - - 1.00 - - jne .LBB5_5
+ - - 0.06 0.94 - - - - lea rax, [2*rdx + 4]
+ - - 0.93 0.07 - - - - sub rsi, rax
+ - - - - - 1.00 - - jae .LBB5_4
+ - - - - - - - - xor eax, eax
+ - - 0.03 0.95 - 0.02 - - mov edx, 1
+ - - - - - 1.00 - - ret
+ - - 0.97 0.03 - - - - lea r8, [rdi + rsi]
+ - - - - - - 0.50 0.50 movzx esi, word ptr [rdi + rsi]
+ - - 0.03 0.97 - - - - cmp si, -16192
+ - - 0.05 0.94 - 0.01 - - mov ecx, 2
+ - - 0.06 0.98 - 0.96 - - cmove rcx, rdx
+ - - - - - - - - xor eax, eax
+ - - 0.97 0.03 - - - - cmp esi, 49344
+ - - 0.06 0.96 - 0.98 - - cmove rax, r8
+ - - 0.02 0.03 - 0.95 - - mov rdx, rcx
+ - - - - - 1.00 - - ret
diff --git a/rust/zerocopy/benches/try_transmute.rs b/rust/zerocopy/benches/try_transmute.rs
new file mode 100644
index 0000000..c0de07a
--- /dev/null
+++ b/rust/zerocopy/benches/try_transmute.rs
@@ -0,0 +1,16 @@
+use zerocopy::Unalign;
+use zerocopy_derive::*;
+
+#[path = "formats/coco_static_size.rs"]
+mod format;
+
+#[derive(IntoBytes, KnownLayout, Immutable)]
+#[repr(C)]
+struct MinimalViableSource {
+ bytes: [u8; 6],
+}
+
+#[unsafe(no_mangle)]
+fn bench_try_transmute(source: MinimalViableSource) -> Option<Unalign<format::CocoPacket>> {
+ zerocopy::try_transmute!(source).ok()
+}
diff --git a/rust/zerocopy/benches/try_transmute.x86-64 b/rust/zerocopy/benches/try_transmute.x86-64
new file mode 100644
index 0000000..9e16a66
--- /dev/null
+++ b/rust/zerocopy/benches/try_transmute.x86-64
@@ -0,0 +1,9 @@
+bench_try_transmute:
+ movzx ecx, di
+ xor eax, eax
+ cmp ecx, 49344
+ sete al
+ and rdi, -65536
+ xor rax, 49345
+ or rax, rdi
+ ret
diff --git a/rust/zerocopy/benches/try_transmute.x86-64.mca b/rust/zerocopy/benches/try_transmute.x86-64.mca
new file mode 100644
index 0000000..33abc3b
--- /dev/null
+++ b/rust/zerocopy/benches/try_transmute.x86-64.mca
@@ -0,0 +1,55 @@
+Iterations: 100
+Instructions: 800
+Total Cycles: 238
+Total uOps: 800
+
+Dispatch Width: 4
+uOps Per Cycle: 3.36
+IPC: 3.36
+Block RThroughput: 2.0
+
+
+Instruction Info:
+[1]: #uOps
+[2]: Latency
+[3]: RThroughput
+[4]: MayLoad
+[5]: MayStore
+[6]: HasSideEffects (U)
+
+[1] [2] [3] [4] [5] [6] Instructions:
+ 1 1 0.33 movzx ecx, di
+ 1 0 0.25 xor eax, eax
+ 1 1 0.33 cmp ecx, 49344
+ 1 1 0.50 sete al
+ 1 1 0.33 and rdi, -65536
+ 1 1 0.33 xor rax, 49345
+ 1 1 0.33 or rax, rdi
+ 1 1 1.00 U ret
+
+
+Resources:
+[0] - SBDivider
+[1] - SBFPDivider
+[2] - SBPort0
+[3] - SBPort1
+[4] - SBPort4
+[5] - SBPort5
+[6.0] - SBPort23
+[6.1] - SBPort23
+
+
+Resource pressure per iteration:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1]
+ - - 2.33 2.33 - 2.34 - -
+
+Resource pressure by instruction:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions:
+ - - 0.32 0.67 - 0.01 - - movzx ecx, di
+ - - - - - - - - xor eax, eax
+ - - 0.33 0.67 - - - - cmp ecx, 49344
+ - - 1.00 - - - - - sete al
+ - - 0.67 0.33 - - - - and rdi, -65536
+ - - - 0.66 - 0.34 - - xor rax, 49345
+ - - 0.01 - - 0.99 - - or rax, rdi
+ - - - - - 1.00 - - ret
diff --git a/rust/zerocopy/benches/try_transmute_ref_dynamic_size.rs b/rust/zerocopy/benches/try_transmute_ref_dynamic_size.rs
new file mode 100644
index 0000000..c9236e1
--- /dev/null
+++ b/rust/zerocopy/benches/try_transmute_ref_dynamic_size.rs
@@ -0,0 +1,18 @@
+use zerocopy_derive::*;
+
+#[path = "formats/coco_dynamic_size.rs"]
+mod format;
+
+#[derive(IntoBytes, KnownLayout, Immutable)]
+#[repr(C, align(2))]
+struct MinimalViableSource {
+ header: [u8; 6],
+ trailer: [[u8; 2]],
+}
+
+#[unsafe(no_mangle)]
+fn bench_try_transmute_ref_dynamic_size(
+ source: &MinimalViableSource,
+) -> Option<&format::CocoPacket> {
+ zerocopy::try_transmute_ref!(source).ok()
+}
diff --git a/rust/zerocopy/benches/try_transmute_ref_dynamic_size.x86-64 b/rust/zerocopy/benches/try_transmute_ref_dynamic_size.x86-64
new file mode 100644
index 0000000..d34d2b3
--- /dev/null
+++ b/rust/zerocopy/benches/try_transmute_ref_dynamic_size.x86-64
@@ -0,0 +1,6 @@
+bench_try_transmute_ref_dynamic_size:
+ lea rdx, [rsi + 1]
+ xor eax, eax
+ cmp word ptr [rdi], -16192
+ cmove rax, rdi
+ ret
diff --git a/rust/zerocopy/benches/try_transmute_ref_dynamic_size.x86-64.mca b/rust/zerocopy/benches/try_transmute_ref_dynamic_size.x86-64.mca
new file mode 100644
index 0000000..bc771b5
--- /dev/null
+++ b/rust/zerocopy/benches/try_transmute_ref_dynamic_size.x86-64.mca
@@ -0,0 +1,49 @@
+Iterations: 100
+Instructions: 500
+Total Cycles: 209
+Total uOps: 700
+
+Dispatch Width: 4
+uOps Per Cycle: 3.35
+IPC: 2.39
+Block RThroughput: 1.8
+
+
+Instruction Info:
+[1]: #uOps
+[2]: Latency
+[3]: RThroughput
+[4]: MayLoad
+[5]: MayStore
+[6]: HasSideEffects (U)
+
+[1] [2] [3] [4] [5] [6] Instructions:
+ 1 1 0.50 lea rdx, [rsi + 1]
+ 1 0 0.25 xor eax, eax
+ 2 6 0.50 * cmp word ptr [rdi], -16192
+ 2 2 0.67 cmove rax, rdi
+ 1 1 1.00 U ret
+
+
+Resources:
+[0] - SBDivider
+[1] - SBFPDivider
+[2] - SBPort0
+[3] - SBPort1
+[4] - SBPort4
+[5] - SBPort5
+[6.0] - SBPort23
+[6.1] - SBPort23
+
+
+Resource pressure per iteration:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1]
+ - - 1.50 1.51 - 1.99 0.50 0.50
+
+Resource pressure by instruction:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions:
+ - - 0.51 0.49 - - - - lea rdx, [rsi + 1]
+ - - - - - - - - xor eax, eax
+ - - - 0.02 - 0.98 0.50 0.50 cmp word ptr [rdi], -16192
+ - - 0.99 1.00 - 0.01 - - cmove rax, rdi
+ - - - - - 1.00 - - ret
diff --git a/rust/zerocopy/benches/try_transmute_ref_static_size.rs b/rust/zerocopy/benches/try_transmute_ref_static_size.rs
new file mode 100644
index 0000000..631cce2
--- /dev/null
+++ b/rust/zerocopy/benches/try_transmute_ref_static_size.rs
@@ -0,0 +1,17 @@
+use zerocopy_derive::*;
+
+#[path = "formats/coco_static_size.rs"]
+mod format;
+
+#[derive(IntoBytes, KnownLayout, Immutable)]
+#[repr(C, align(2))]
+struct MinimalViableSource {
+ bytes: [u8; 6],
+}
+
+#[unsafe(no_mangle)]
+fn bench_try_transmute_ref_static_size(
+ source: &MinimalViableSource,
+) -> Option<&format::CocoPacket> {
+ zerocopy::try_transmute_ref!(source).ok()
+}
diff --git a/rust/zerocopy/benches/try_transmute_ref_static_size.x86-64 b/rust/zerocopy/benches/try_transmute_ref_static_size.x86-64
new file mode 100644
index 0000000..8c16fac
--- /dev/null
+++ b/rust/zerocopy/benches/try_transmute_ref_static_size.x86-64
@@ -0,0 +1,5 @@
+bench_try_transmute_ref_static_size:
+ xor eax, eax
+ cmp word ptr [rdi], -16192
+ cmove rax, rdi
+ ret
diff --git a/rust/zerocopy/benches/try_transmute_ref_static_size.x86-64.mca b/rust/zerocopy/benches/try_transmute_ref_static_size.x86-64.mca
new file mode 100644
index 0000000..cf73849
--- /dev/null
+++ b/rust/zerocopy/benches/try_transmute_ref_static_size.x86-64.mca
@@ -0,0 +1,47 @@
+Iterations: 100
+Instructions: 400
+Total Cycles: 160
+Total uOps: 600
+
+Dispatch Width: 4
+uOps Per Cycle: 3.75
+IPC: 2.50
+Block RThroughput: 1.5
+
+
+Instruction Info:
+[1]: #uOps
+[2]: Latency
+[3]: RThroughput
+[4]: MayLoad
+[5]: MayStore
+[6]: HasSideEffects (U)
+
+[1] [2] [3] [4] [5] [6] Instructions:
+ 1 0 0.25 xor eax, eax
+ 2 6 0.50 * cmp word ptr [rdi], -16192
+ 2 2 0.67 cmove rax, rdi
+ 1 1 1.00 U ret
+
+
+Resources:
+[0] - SBDivider
+[1] - SBFPDivider
+[2] - SBPort0
+[3] - SBPort1
+[4] - SBPort4
+[5] - SBPort5
+[6.0] - SBPort23
+[6.1] - SBPort23
+
+
+Resource pressure per iteration:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1]
+ - - 1.02 1.48 - 1.50 0.50 0.50
+
+Resource pressure by instruction:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions:
+ - - - - - - - - xor eax, eax
+ - - 0.02 0.49 - 0.49 0.50 0.50 cmp word ptr [rdi], -16192
+ - - 1.00 0.99 - 0.01 - - cmove rax, rdi
+ - - - - - 1.00 - - ret
diff --git a/rust/zerocopy/benches/write_to_dynamic_size.rs b/rust/zerocopy/benches/write_to_dynamic_size.rs
new file mode 100644
index 0000000..c126a14
--- /dev/null
+++ b/rust/zerocopy/benches/write_to_dynamic_size.rs
@@ -0,0 +1,9 @@
+use zerocopy::*;
+
+#[path = "formats/coco_dynamic_size.rs"]
+mod format;
+
+#[unsafe(no_mangle)]
+fn bench_write_to_dynamic_size(source: &format::CocoPacket, destination: &mut [u8]) -> Option<()> {
+ source.write_to(destination).ok()
+}
diff --git a/rust/zerocopy/benches/write_to_dynamic_size.x86-64 b/rust/zerocopy/benches/write_to_dynamic_size.x86-64
new file mode 100644
index 0000000..c5abb17
--- /dev/null
+++ b/rust/zerocopy/benches/write_to_dynamic_size.x86-64
@@ -0,0 +1,21 @@
+bench_write_to_dynamic_size:
+ push r14
+ push rbx
+ push rax
+ mov rbx, rcx
+ lea r14, [2*rsi + 5]
+ and r14, -2
+ cmp rcx, r14
+ jne .LBB5_2
+ mov rax, rdi
+ mov rdi, rdx
+ mov rsi, rax
+ mov rdx, rbx
+ call qword ptr [rip + memcpy@GOTPCREL]
+.LBB5_2:
+ cmp rbx, r14
+ sete al
+ add rsp, 8
+ pop rbx
+ pop r14
+ ret
diff --git a/rust/zerocopy/benches/write_to_dynamic_size.x86-64.mca b/rust/zerocopy/benches/write_to_dynamic_size.x86-64.mca
new file mode 100644
index 0000000..5b2c08a
--- /dev/null
+++ b/rust/zerocopy/benches/write_to_dynamic_size.x86-64.mca
@@ -0,0 +1,77 @@
+Iterations: 100
+Instructions: 1900
+Total Cycles: 2890
+Total uOps: 2500
+
+Dispatch Width: 4
+uOps Per Cycle: 0.87
+IPC: 0.66
+Block RThroughput: 6.3
+
+
+Instruction Info:
+[1]: #uOps
+[2]: Latency
+[3]: RThroughput
+[4]: MayLoad
+[5]: MayStore
+[6]: HasSideEffects (U)
+
+[1] [2] [3] [4] [5] [6] Instructions:
+ 2 5 1.00 * push r14
+ 2 5 1.00 * push rbx
+ 2 5 1.00 * push rax
+ 1 1 0.33 mov rbx, rcx
+ 1 1 0.50 lea r14, [2*rsi + 5]
+ 1 1 0.33 and r14, -2
+ 1 1 0.33 cmp rcx, r14
+ 1 1 1.00 jne .LBB5_2
+ 1 1 0.33 mov rax, rdi
+ 1 1 0.33 mov rdi, rdx
+ 1 1 0.33 mov rsi, rax
+ 1 1 0.33 mov rdx, rbx
+ 4 7 1.00 * call qword ptr [rip + memcpy@GOTPCREL]
+ 1 1 0.33 cmp rbx, r14
+ 1 1 0.50 sete al
+ 1 1 0.33 add rsp, 8
+ 1 6 0.50 * pop rbx
+ 1 6 0.50 * pop r14
+ 1 1 1.00 U ret
+
+
+Resources:
+[0] - SBDivider
+[1] - SBFPDivider
+[2] - SBPort0
+[3] - SBPort1
+[4] - SBPort4
+[5] - SBPort5
+[6.0] - SBPort23
+[6.1] - SBPort23
+
+
+Resource pressure per iteration:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1]
+ - - 4.66 4.64 4.00 4.70 4.00 3.00
+
+Resource pressure by instruction:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions:
+ - - - - 1.00 - - 1.00 push r14
+ - - - - 1.00 - 1.00 - push rbx
+ - - - - 1.00 - - 1.00 push rax
+ - - 0.02 0.97 - 0.01 - - mov rbx, rcx
+ - - 0.97 0.03 - - - - lea r14, [2*rsi + 5]
+ - - 0.63 0.35 - 0.02 - - and r14, -2
+ - - 0.31 0.34 - 0.35 - - cmp rcx, r14
+ - - - - - 1.00 - - jne .LBB5_2
+ - - 0.33 0.33 - 0.34 - - mov rax, rdi
+ - - 0.36 0.31 - 0.33 - - mov rdi, rdx
+ - - 0.33 0.35 - 0.32 - - mov rsi, rax
+ - - 0.35 0.63 - 0.02 - - mov rdx, rbx
+ - - - - 1.00 1.00 2.00 - call qword ptr [rip + memcpy@GOTPCREL]
+ - - 0.65 0.35 - - - - cmp rbx, r14
+ - - 0.69 - - 0.31 - - sete al
+ - - 0.02 0.98 - - - - add rsp, 8
+ - - - - - - - 1.00 pop rbx
+ - - - - - - 1.00 - pop r14
+ - - - - - 1.00 - - ret
diff --git a/rust/zerocopy/benches/write_to_prefix_dynamic_size.rs b/rust/zerocopy/benches/write_to_prefix_dynamic_size.rs
new file mode 100644
index 0000000..a54d327
--- /dev/null
+++ b/rust/zerocopy/benches/write_to_prefix_dynamic_size.rs
@@ -0,0 +1,12 @@
+use zerocopy::*;
+
+#[path = "formats/coco_dynamic_size.rs"]
+mod format;
+
+#[unsafe(no_mangle)]
+fn bench_write_to_prefix_dynamic_size(
+ source: &format::CocoPacket,
+ destination: &mut [u8],
+) -> Option<()> {
+ source.write_to_prefix(destination).ok()
+}
diff --git a/rust/zerocopy/benches/write_to_prefix_dynamic_size.x86-64 b/rust/zerocopy/benches/write_to_prefix_dynamic_size.x86-64
new file mode 100644
index 0000000..d7779c6
--- /dev/null
+++ b/rust/zerocopy/benches/write_to_prefix_dynamic_size.x86-64
@@ -0,0 +1,21 @@
+bench_write_to_prefix_dynamic_size:
+ push r14
+ push rbx
+ push rax
+ mov rbx, rcx
+ lea r14, [2*rsi + 5]
+ and r14, -2
+ cmp r14, rcx
+ ja .LBB5_2
+ mov rax, rdi
+ mov rdi, rdx
+ mov rsi, rax
+ mov rdx, r14
+ call qword ptr [rip + memcpy@GOTPCREL]
+.LBB5_2:
+ cmp r14, rbx
+ setbe al
+ add rsp, 8
+ pop rbx
+ pop r14
+ ret
diff --git a/rust/zerocopy/benches/write_to_prefix_dynamic_size.x86-64.mca b/rust/zerocopy/benches/write_to_prefix_dynamic_size.x86-64.mca
new file mode 100644
index 0000000..4cebe24
--- /dev/null
+++ b/rust/zerocopy/benches/write_to_prefix_dynamic_size.x86-64.mca
@@ -0,0 +1,77 @@
+Iterations: 100
+Instructions: 1900
+Total Cycles: 2890
+Total uOps: 2600
+
+Dispatch Width: 4
+uOps Per Cycle: 0.90
+IPC: 0.66
+Block RThroughput: 6.5
+
+
+Instruction Info:
+[1]: #uOps
+[2]: Latency
+[3]: RThroughput
+[4]: MayLoad
+[5]: MayStore
+[6]: HasSideEffects (U)
+
+[1] [2] [3] [4] [5] [6] Instructions:
+ 2 5 1.00 * push r14
+ 2 5 1.00 * push rbx
+ 2 5 1.00 * push rax
+ 1 1 0.33 mov rbx, rcx
+ 1 1 0.50 lea r14, [2*rsi + 5]
+ 1 1 0.33 and r14, -2
+ 1 1 0.33 cmp r14, rcx
+ 1 1 1.00 ja .LBB5_2
+ 1 1 0.33 mov rax, rdi
+ 1 1 0.33 mov rdi, rdx
+ 1 1 0.33 mov rsi, rax
+ 1 1 0.33 mov rdx, r14
+ 4 7 1.00 * call qword ptr [rip + memcpy@GOTPCREL]
+ 1 1 0.33 cmp r14, rbx
+ 2 2 1.00 setbe al
+ 1 1 0.33 add rsp, 8
+ 1 6 0.50 * pop rbx
+ 1 6 0.50 * pop r14
+ 1 1 1.00 U ret
+
+
+Resources:
+[0] - SBDivider
+[1] - SBFPDivider
+[2] - SBPort0
+[3] - SBPort1
+[4] - SBPort4
+[5] - SBPort5
+[6.0] - SBPort23
+[6.1] - SBPort23
+
+
+Resource pressure per iteration:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1]
+ - - 5.47 4.49 4.00 5.04 4.00 3.00
+
+Resource pressure by instruction:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions:
+ - - - - 1.00 - - 1.00 push r14
+ - - - - 1.00 - 1.00 - push rbx
+ - - - - 1.00 - - 1.00 push rax
+ - - 0.48 0.51 - 0.01 - - mov rbx, rcx
+ - - 0.51 0.49 - - - - lea r14, [2*rsi + 5]
+ - - 0.48 0.05 - 0.47 - - and r14, -2
+ - - 0.48 0.49 - 0.03 - - cmp r14, rcx
+ - - - - - 1.00 - - ja .LBB5_2
+ - - 0.04 0.47 - 0.49 - - mov rax, rdi
+ - - 0.49 0.03 - 0.48 - - mov rdi, rdx
+ - - 0.03 0.48 - 0.49 - - mov rsi, rax
+ - - 0.48 0.51 - 0.01 - - mov rdx, r14
+ - - - - 1.00 1.00 2.00 - call qword ptr [rip + memcpy@GOTPCREL]
+ - - 0.51 0.49 - - - - cmp r14, rbx
+ - - 1.94 - - 0.06 - - setbe al
+ - - 0.03 0.97 - - - - add rsp, 8
+ - - - - - - - 1.00 pop rbx
+ - - - - - - 1.00 - pop r14
+ - - - - - 1.00 - - ret
diff --git a/rust/zerocopy/benches/write_to_prefix_static_size.rs b/rust/zerocopy/benches/write_to_prefix_static_size.rs
new file mode 100644
index 0000000..826222c
--- /dev/null
+++ b/rust/zerocopy/benches/write_to_prefix_static_size.rs
@@ -0,0 +1,12 @@
+use zerocopy::*;
+
+#[path = "formats/coco_static_size.rs"]
+mod format;
+
+#[unsafe(no_mangle)]
+fn bench_write_to_prefix_static_size(
+ source: &format::CocoPacket,
+ destination: &mut [u8],
+) -> Option<()> {
+ source.write_to_prefix(destination).ok()
+}
diff --git a/rust/zerocopy/benches/write_to_prefix_static_size.x86-64 b/rust/zerocopy/benches/write_to_prefix_static_size.x86-64
new file mode 100644
index 0000000..9cf0662
--- /dev/null
+++ b/rust/zerocopy/benches/write_to_prefix_static_size.x86-64
@@ -0,0 +1,11 @@
+bench_write_to_prefix_static_size:
+ cmp rdx, 6
+ jb .LBB5_2
+ movzx eax, word ptr [rdi + 4]
+ mov word ptr [rsi + 4], ax
+ mov eax, dword ptr [rdi]
+ mov dword ptr [rsi], eax
+.LBB5_2:
+ cmp rdx, 6
+ setae al
+ ret
diff --git a/rust/zerocopy/benches/write_to_prefix_static_size.x86-64.mca b/rust/zerocopy/benches/write_to_prefix_static_size.x86-64.mca
new file mode 100644
index 0000000..5d17200
--- /dev/null
+++ b/rust/zerocopy/benches/write_to_prefix_static_size.x86-64.mca
@@ -0,0 +1,57 @@
+Iterations: 100
+Instructions: 900
+Total Cycles: 233
+Total uOps: 900
+
+Dispatch Width: 4
+uOps Per Cycle: 3.86
+IPC: 3.86
+Block RThroughput: 2.3
+
+
+Instruction Info:
+[1]: #uOps
+[2]: Latency
+[3]: RThroughput
+[4]: MayLoad
+[5]: MayStore
+[6]: HasSideEffects (U)
+
+[1] [2] [3] [4] [5] [6] Instructions:
+ 1 1 0.33 cmp rdx, 6
+ 1 1 1.00 jb .LBB5_2
+ 1 5 0.50 * movzx eax, word ptr [rdi + 4]
+ 1 1 1.00 * mov word ptr [rsi + 4], ax
+ 1 5 0.50 * mov eax, dword ptr [rdi]
+ 1 1 1.00 * mov dword ptr [rsi], eax
+ 1 1 0.33 cmp rdx, 6
+ 1 1 0.50 setae al
+ 1 1 1.00 U ret
+
+
+Resources:
+[0] - SBDivider
+[1] - SBFPDivider
+[2] - SBPort0
+[3] - SBPort1
+[4] - SBPort4
+[5] - SBPort5
+[6.0] - SBPort23
+[6.1] - SBPort23
+
+
+Resource pressure per iteration:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1]
+ - - 1.50 1.49 2.00 2.01 2.00 2.00
+
+Resource pressure by instruction:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions:
+ - - 0.25 0.74 - 0.01 - - cmp rdx, 6
+ - - - - - 1.00 - - jb .LBB5_2
+ - - - - - - 0.50 0.50 movzx eax, word ptr [rdi + 4]
+ - - - - 1.00 - 0.48 0.52 mov word ptr [rsi + 4], ax
+ - - - - - - 0.52 0.48 mov eax, dword ptr [rdi]
+ - - - - 1.00 - 0.50 0.50 mov dword ptr [rsi], eax
+ - - 0.25 0.75 - - - - cmp rdx, 6
+ - - 1.00 - - - - - setae al
+ - - - - - 1.00 - - ret
diff --git a/rust/zerocopy/benches/write_to_static_size.rs b/rust/zerocopy/benches/write_to_static_size.rs
new file mode 100644
index 0000000..3bb9435
--- /dev/null
+++ b/rust/zerocopy/benches/write_to_static_size.rs
@@ -0,0 +1,9 @@
+use zerocopy::*;
+
+#[path = "formats/coco_static_size.rs"]
+mod format;
+
+#[unsafe(no_mangle)]
+fn bench_write_to_static_size(source: &format::CocoPacket, destination: &mut [u8]) -> Option<()> {
+ source.write_to(destination).ok()
+}
diff --git a/rust/zerocopy/benches/write_to_static_size.x86-64 b/rust/zerocopy/benches/write_to_static_size.x86-64
new file mode 100644
index 0000000..d6413e0
--- /dev/null
+++ b/rust/zerocopy/benches/write_to_static_size.x86-64
@@ -0,0 +1,11 @@
+bench_write_to_static_size:
+ cmp rdx, 6
+ jne .LBB5_2
+ movzx eax, word ptr [rdi + 4]
+ mov word ptr [rsi + 4], ax
+ mov eax, dword ptr [rdi]
+ mov dword ptr [rsi], eax
+.LBB5_2:
+ cmp rdx, 6
+ sete al
+ ret
diff --git a/rust/zerocopy/benches/write_to_static_size.x86-64.mca b/rust/zerocopy/benches/write_to_static_size.x86-64.mca
new file mode 100644
index 0000000..cc5bb1d
--- /dev/null
+++ b/rust/zerocopy/benches/write_to_static_size.x86-64.mca
@@ -0,0 +1,57 @@
+Iterations: 100
+Instructions: 900
+Total Cycles: 233
+Total uOps: 900
+
+Dispatch Width: 4
+uOps Per Cycle: 3.86
+IPC: 3.86
+Block RThroughput: 2.3
+
+
+Instruction Info:
+[1]: #uOps
+[2]: Latency
+[3]: RThroughput
+[4]: MayLoad
+[5]: MayStore
+[6]: HasSideEffects (U)
+
+[1] [2] [3] [4] [5] [6] Instructions:
+ 1 1 0.33 cmp rdx, 6
+ 1 1 1.00 jne .LBB5_2
+ 1 5 0.50 * movzx eax, word ptr [rdi + 4]
+ 1 1 1.00 * mov word ptr [rsi + 4], ax
+ 1 5 0.50 * mov eax, dword ptr [rdi]
+ 1 1 1.00 * mov dword ptr [rsi], eax
+ 1 1 0.33 cmp rdx, 6
+ 1 1 0.50 sete al
+ 1 1 1.00 U ret
+
+
+Resources:
+[0] - SBDivider
+[1] - SBFPDivider
+[2] - SBPort0
+[3] - SBPort1
+[4] - SBPort4
+[5] - SBPort5
+[6.0] - SBPort23
+[6.1] - SBPort23
+
+
+Resource pressure per iteration:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1]
+ - - 1.50 1.49 2.00 2.01 2.00 2.00
+
+Resource pressure by instruction:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions:
+ - - 0.25 0.74 - 0.01 - - cmp rdx, 6
+ - - - - - 1.00 - - jne .LBB5_2
+ - - - - - - 0.50 0.50 movzx eax, word ptr [rdi + 4]
+ - - - - 1.00 - 0.48 0.52 mov word ptr [rsi + 4], ax
+ - - - - - - 0.52 0.48 mov eax, dword ptr [rdi]
+ - - - - 1.00 - 0.50 0.50 mov dword ptr [rsi], eax
+ - - 0.25 0.75 - - - - cmp rdx, 6
+ - - 1.00 - - - - - sete al
+ - - - - - 1.00 - - ret
diff --git a/rust/zerocopy/benches/write_to_suffix_dynamic_size.rs b/rust/zerocopy/benches/write_to_suffix_dynamic_size.rs
new file mode 100644
index 0000000..9fa6b91c
--- /dev/null
+++ b/rust/zerocopy/benches/write_to_suffix_dynamic_size.rs
@@ -0,0 +1,12 @@
+use zerocopy::*;
+
+#[path = "formats/coco_dynamic_size.rs"]
+mod format;
+
+#[unsafe(no_mangle)]
+fn bench_write_to_suffix_dynamic_size(
+ source: &format::CocoPacket,
+ destination: &mut [u8],
+) -> Option<()> {
+ source.write_to_suffix(destination).ok()
+}
diff --git a/rust/zerocopy/benches/write_to_suffix_dynamic_size.x86-64 b/rust/zerocopy/benches/write_to_suffix_dynamic_size.x86-64
new file mode 100644
index 0000000..75f3495
--- /dev/null
+++ b/rust/zerocopy/benches/write_to_suffix_dynamic_size.x86-64
@@ -0,0 +1,22 @@
+bench_write_to_suffix_dynamic_size:
+ push r14
+ push rbx
+ push rax
+ mov rbx, rcx
+ lea r14, [2*rsi + 5]
+ and r14, -2
+ sub rcx, r14
+ jb .LBB5_2
+ mov rax, rdi
+ add rdx, rcx
+ mov rdi, rdx
+ mov rsi, rax
+ mov rdx, r14
+ call qword ptr [rip + memcpy@GOTPCREL]
+.LBB5_2:
+ cmp rbx, r14
+ setae al
+ add rsp, 8
+ pop rbx
+ pop r14
+ ret
diff --git a/rust/zerocopy/benches/write_to_suffix_dynamic_size.x86-64.mca b/rust/zerocopy/benches/write_to_suffix_dynamic_size.x86-64.mca
new file mode 100644
index 0000000..95cb9df
--- /dev/null
+++ b/rust/zerocopy/benches/write_to_suffix_dynamic_size.x86-64.mca
@@ -0,0 +1,79 @@
+Iterations: 100
+Instructions: 2000
+Total Cycles: 2890
+Total uOps: 2600
+
+Dispatch Width: 4
+uOps Per Cycle: 0.90
+IPC: 0.69
+Block RThroughput: 6.5
+
+
+Instruction Info:
+[1]: #uOps
+[2]: Latency
+[3]: RThroughput
+[4]: MayLoad
+[5]: MayStore
+[6]: HasSideEffects (U)
+
+[1] [2] [3] [4] [5] [6] Instructions:
+ 2 5 1.00 * push r14
+ 2 5 1.00 * push rbx
+ 2 5 1.00 * push rax
+ 1 1 0.33 mov rbx, rcx
+ 1 1 0.50 lea r14, [2*rsi + 5]
+ 1 1 0.33 and r14, -2
+ 1 1 0.33 sub rcx, r14
+ 1 1 1.00 jb .LBB5_2
+ 1 1 0.33 mov rax, rdi
+ 1 1 0.33 add rdx, rcx
+ 1 1 0.33 mov rdi, rdx
+ 1 1 0.33 mov rsi, rax
+ 1 1 0.33 mov rdx, r14
+ 4 7 1.00 * call qword ptr [rip + memcpy@GOTPCREL]
+ 1 1 0.33 cmp rbx, r14
+ 1 1 0.50 setae al
+ 1 1 0.33 add rsp, 8
+ 1 6 0.50 * pop rbx
+ 1 6 0.50 * pop r14
+ 1 1 1.00 U ret
+
+
+Resources:
+[0] - SBDivider
+[1] - SBFPDivider
+[2] - SBPort0
+[3] - SBPort1
+[4] - SBPort4
+[5] - SBPort5
+[6.0] - SBPort23
+[6.1] - SBPort23
+
+
+Resource pressure per iteration:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1]
+ - - 4.98 4.98 4.00 5.04 4.00 3.00
+
+Resource pressure by instruction:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions:
+ - - - - 1.00 - - 1.00 push r14
+ - - - - 1.00 - 1.00 - push rbx
+ - - - - 1.00 - - 1.00 push rax
+ - - 0.94 0.05 - 0.01 - - mov rbx, rcx
+ - - 0.06 0.94 - - - - lea r14, [2*rsi + 5]
+ - - 0.93 0.02 - 0.05 - - and r14, -2
+ - - 0.05 0.94 - 0.01 - - sub rcx, r14
+ - - - - - 1.00 - - jb .LBB5_2
+ - - 0.02 0.04 - 0.94 - - mov rax, rdi
+ - - 0.03 0.97 - - - - add rdx, rcx
+ - - 0.95 0.05 - - - - mov rdi, rdx
+ - - 0.94 0.03 - 0.03 - - mov rsi, rax
+ - - 0.01 0.03 - 0.96 - - mov rdx, r14
+ - - - - 1.00 1.00 2.00 - call qword ptr [rip + memcpy@GOTPCREL]
+ - - 0.05 0.94 - 0.01 - - cmp rbx, r14
+ - - 0.97 - - 0.03 - - setae al
+ - - 0.03 0.97 - - - - add rsp, 8
+ - - - - - - - 1.00 pop rbx
+ - - - - - - 1.00 - pop r14
+ - - - - - 1.00 - - ret
diff --git a/rust/zerocopy/benches/write_to_suffix_static_size.rs b/rust/zerocopy/benches/write_to_suffix_static_size.rs
new file mode 100644
index 0000000..1c95aba
--- /dev/null
+++ b/rust/zerocopy/benches/write_to_suffix_static_size.rs
@@ -0,0 +1,12 @@
+use zerocopy::*;
+
+#[path = "formats/coco_static_size.rs"]
+mod format;
+
+#[unsafe(no_mangle)]
+fn bench_write_to_suffix_static_size(
+ source: &format::CocoPacket,
+ destination: &mut [u8],
+) -> Option<()> {
+ source.write_to_suffix(destination).ok()
+}
diff --git a/rust/zerocopy/benches/write_to_suffix_static_size.x86-64 b/rust/zerocopy/benches/write_to_suffix_static_size.x86-64
new file mode 100644
index 0000000..934aa37
--- /dev/null
+++ b/rust/zerocopy/benches/write_to_suffix_static_size.x86-64
@@ -0,0 +1,11 @@
+bench_write_to_suffix_static_size:
+ cmp rdx, 6
+ jb .LBB5_2
+ movzx eax, word ptr [rdi + 4]
+ mov word ptr [rsi + rdx - 2], ax
+ mov eax, dword ptr [rdi]
+ mov dword ptr [rsi + rdx - 6], eax
+.LBB5_2:
+ cmp rdx, 6
+ setae al
+ ret
diff --git a/rust/zerocopy/benches/write_to_suffix_static_size.x86-64.mca b/rust/zerocopy/benches/write_to_suffix_static_size.x86-64.mca
new file mode 100644
index 0000000..6b18e4a
--- /dev/null
+++ b/rust/zerocopy/benches/write_to_suffix_static_size.x86-64.mca
@@ -0,0 +1,57 @@
+Iterations: 100
+Instructions: 900
+Total Cycles: 233
+Total uOps: 900
+
+Dispatch Width: 4
+uOps Per Cycle: 3.86
+IPC: 3.86
+Block RThroughput: 2.3
+
+
+Instruction Info:
+[1]: #uOps
+[2]: Latency
+[3]: RThroughput
+[4]: MayLoad
+[5]: MayStore
+[6]: HasSideEffects (U)
+
+[1] [2] [3] [4] [5] [6] Instructions:
+ 1 1 0.33 cmp rdx, 6
+ 1 1 1.00 jb .LBB5_2
+ 1 5 0.50 * movzx eax, word ptr [rdi + 4]
+ 1 1 1.00 * mov word ptr [rsi + rdx - 2], ax
+ 1 5 0.50 * mov eax, dword ptr [rdi]
+ 1 1 1.00 * mov dword ptr [rsi + rdx - 6], eax
+ 1 1 0.33 cmp rdx, 6
+ 1 1 0.50 setae al
+ 1 1 1.00 U ret
+
+
+Resources:
+[0] - SBDivider
+[1] - SBFPDivider
+[2] - SBPort0
+[3] - SBPort1
+[4] - SBPort4
+[5] - SBPort5
+[6.0] - SBPort23
+[6.1] - SBPort23
+
+
+Resource pressure per iteration:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1]
+ - - 1.50 1.49 2.00 2.01 2.00 2.00
+
+Resource pressure by instruction:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions:
+ - - 0.25 0.74 - 0.01 - - cmp rdx, 6
+ - - - - - 1.00 - - jb .LBB5_2
+ - - - - - - 0.50 0.50 movzx eax, word ptr [rdi + 4]
+ - - - - 1.00 - 0.48 0.52 mov word ptr [rsi + rdx - 2], ax
+ - - - - - - 0.52 0.48 mov eax, dword ptr [rdi]
+ - - - - 1.00 - 0.50 0.50 mov dword ptr [rsi + rdx - 6], eax
+ - - 0.25 0.75 - - - - cmp rdx, 6
+ - - 1.00 - - - - - setae al
+ - - - - - 1.00 - - ret
diff --git a/rust/zerocopy/benches/zero_dynamic_padding.rs b/rust/zerocopy/benches/zero_dynamic_padding.rs
new file mode 100644
index 0000000..8eda095
--- /dev/null
+++ b/rust/zerocopy/benches/zero_dynamic_padding.rs
@@ -0,0 +1,9 @@
+use zerocopy::*;
+
+#[path = "formats/coco_dynamic_padding.rs"]
+mod format;
+
+#[unsafe(no_mangle)]
+fn bench_zero_dynamic_padding(source: &mut format::LocoPacket) {
+ source.zero()
+}
diff --git a/rust/zerocopy/benches/zero_dynamic_padding.x86-64 b/rust/zerocopy/benches/zero_dynamic_padding.x86-64
new file mode 100644
index 0000000..7dccf17
--- /dev/null
+++ b/rust/zerocopy/benches/zero_dynamic_padding.x86-64
@@ -0,0 +1,7 @@
+bench_zero_dynamic_padding:
+ lea rax, [rsi + 2*rsi]
+ movabs rdx, 9223372036854775804
+ and rdx, rax
+ add rdx, 12
+ xor esi, esi
+ jmp qword ptr [rip + memset@GOTPCREL]
diff --git a/rust/zerocopy/benches/zero_dynamic_padding.x86-64.mca b/rust/zerocopy/benches/zero_dynamic_padding.x86-64.mca
new file mode 100644
index 0000000..098fc10
--- /dev/null
+++ b/rust/zerocopy/benches/zero_dynamic_padding.x86-64.mca
@@ -0,0 +1,51 @@
+Iterations: 100
+Instructions: 600
+Total Cycles: 209
+Total uOps: 700
+
+Dispatch Width: 4
+uOps Per Cycle: 3.35
+IPC: 2.87
+Block RThroughput: 1.8
+
+
+Instruction Info:
+[1]: #uOps
+[2]: Latency
+[3]: RThroughput
+[4]: MayLoad
+[5]: MayStore
+[6]: HasSideEffects (U)
+
+[1] [2] [3] [4] [5] [6] Instructions:
+ 1 1 0.50 lea rax, [rsi + 2*rsi]
+ 1 1 0.33 movabs rdx, 9223372036854775804
+ 1 1 0.33 and rdx, rax
+ 1 1 0.33 add rdx, 12
+ 1 0 0.25 xor esi, esi
+ 2 6 1.00 * jmp qword ptr [rip + memset@GOTPCREL]
+
+
+Resources:
+[0] - SBDivider
+[1] - SBFPDivider
+[2] - SBPort0
+[3] - SBPort1
+[4] - SBPort4
+[5] - SBPort5
+[6.0] - SBPort23
+[6.1] - SBPort23
+
+
+Resource pressure per iteration:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1]
+ - - 1.66 1.66 - 1.68 0.50 0.50
+
+Resource pressure by instruction:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions:
+ - - 0.33 0.67 - - - - lea rax, [rsi + 2*rsi]
+ - - 0.98 - - 0.02 - - movabs rdx, 9223372036854775804
+ - - 0.01 0.66 - 0.33 - - and rdx, rax
+ - - 0.34 0.33 - 0.33 - - add rdx, 12
+ - - - - - - - - xor esi, esi
+ - - - - - 1.00 0.50 0.50 jmp qword ptr [rip + memset@GOTPCREL]
diff --git a/rust/zerocopy/benches/zero_dynamic_size.rs b/rust/zerocopy/benches/zero_dynamic_size.rs
new file mode 100644
index 0000000..536d800
--- /dev/null
+++ b/rust/zerocopy/benches/zero_dynamic_size.rs
@@ -0,0 +1,9 @@
+use zerocopy::*;
+
+#[path = "formats/coco_dynamic_size.rs"]
+mod format;
+
+#[unsafe(no_mangle)]
+fn bench_zero_dynamic_size(source: &mut format::LocoPacket) {
+ source.zero()
+}
diff --git a/rust/zerocopy/benches/zero_dynamic_size.x86-64 b/rust/zerocopy/benches/zero_dynamic_size.x86-64
new file mode 100644
index 0000000..2b31ed6
--- /dev/null
+++ b/rust/zerocopy/benches/zero_dynamic_size.x86-64
@@ -0,0 +1,5 @@
+bench_zero_dynamic_size:
+ lea rdx, [2*rsi + 5]
+ and rdx, -2
+ xor esi, esi
+ jmp qword ptr [rip + memset@GOTPCREL]
diff --git a/rust/zerocopy/benches/zero_dynamic_size.x86-64.mca b/rust/zerocopy/benches/zero_dynamic_size.x86-64.mca
new file mode 100644
index 0000000..0b086a2
--- /dev/null
+++ b/rust/zerocopy/benches/zero_dynamic_size.x86-64.mca
@@ -0,0 +1,47 @@
+Iterations: 100
+Instructions: 400
+Total Cycles: 142
+Total uOps: 500
+
+Dispatch Width: 4
+uOps Per Cycle: 3.52
+IPC: 2.82
+Block RThroughput: 1.3
+
+
+Instruction Info:
+[1]: #uOps
+[2]: Latency
+[3]: RThroughput
+[4]: MayLoad
+[5]: MayStore
+[6]: HasSideEffects (U)
+
+[1] [2] [3] [4] [5] [6] Instructions:
+ 1 1 0.50 lea rdx, [2*rsi + 5]
+ 1 1 0.33 and rdx, -2
+ 1 0 0.25 xor esi, esi
+ 2 6 1.00 * jmp qword ptr [rip + memset@GOTPCREL]
+
+
+Resources:
+[0] - SBDivider
+[1] - SBFPDivider
+[2] - SBPort0
+[3] - SBPort1
+[4] - SBPort4
+[5] - SBPort5
+[6.0] - SBPort23
+[6.1] - SBPort23
+
+
+Resource pressure per iteration:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1]
+ - - 0.99 1.00 - 1.01 0.50 0.50
+
+Resource pressure by instruction:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions:
+ - - 0.99 0.01 - - - - lea rdx, [2*rsi + 5]
+ - - - 0.99 - 0.01 - - and rdx, -2
+ - - - - - - - - xor esi, esi
+ - - - - - 1.00 0.50 0.50 jmp qword ptr [rip + memset@GOTPCREL]
diff --git a/rust/zerocopy/benches/zero_static_size.rs b/rust/zerocopy/benches/zero_static_size.rs
new file mode 100644
index 0000000..fa7fa08
--- /dev/null
+++ b/rust/zerocopy/benches/zero_static_size.rs
@@ -0,0 +1,9 @@
+use zerocopy::*;
+
+#[path = "formats/coco_static_size.rs"]
+mod format;
+
+#[unsafe(no_mangle)]
+fn bench_zero_static_size(source: &mut format::LocoPacket) {
+ source.zero()
+}
diff --git a/rust/zerocopy/benches/zero_static_size.x86-64 b/rust/zerocopy/benches/zero_static_size.x86-64
new file mode 100644
index 0000000..ced8e18
--- /dev/null
+++ b/rust/zerocopy/benches/zero_static_size.x86-64
@@ -0,0 +1,4 @@
+bench_zero_static_size:
+ mov word ptr [rdi + 4], 0
+ mov dword ptr [rdi], 0
+ ret
diff --git a/rust/zerocopy/benches/zero_static_size.x86-64.mca b/rust/zerocopy/benches/zero_static_size.x86-64.mca
new file mode 100644
index 0000000..042897e
--- /dev/null
+++ b/rust/zerocopy/benches/zero_static_size.x86-64.mca
@@ -0,0 +1,45 @@
+Iterations: 100
+Instructions: 300
+Total Cycles: 203
+Total uOps: 300
+
+Dispatch Width: 4
+uOps Per Cycle: 1.48
+IPC: 1.48
+Block RThroughput: 2.0
+
+
+Instruction Info:
+[1]: #uOps
+[2]: Latency
+[3]: RThroughput
+[4]: MayLoad
+[5]: MayStore
+[6]: HasSideEffects (U)
+
+[1] [2] [3] [4] [5] [6] Instructions:
+ 1 1 1.00 * mov word ptr [rdi + 4], 0
+ 1 1 1.00 * mov dword ptr [rdi], 0
+ 1 1 1.00 U ret
+
+
+Resources:
+[0] - SBDivider
+[1] - SBFPDivider
+[2] - SBPort0
+[3] - SBPort1
+[4] - SBPort4
+[5] - SBPort5
+[6.0] - SBPort23
+[6.1] - SBPort23
+
+
+Resource pressure per iteration:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1]
+ - - - - 2.00 1.00 1.00 1.00
+
+Resource pressure by instruction:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions:
+ - - - - 1.00 - - 1.00 mov word ptr [rdi + 4], 0
+ - - - - 1.00 - 1.00 - mov dword ptr [rdi], 0
+ - - - - - 1.00 - - ret
diff --git a/rust/zerocopy/rustdoc/style.css b/rust/zerocopy/rustdoc/style.css
new file mode 100644
index 0000000..4143489
--- /dev/null
+++ b/rust/zerocopy/rustdoc/style.css
@@ -0,0 +1,57 @@
+/* SPDX-License-Identifier: (BSD-2-Clause OR Apache-2.0) OR MIT */
+
+/*
+Copyright 2026 The Fuchsia Authors
+
+Licensed under a BSD-style license <LICENSE-BSD>, Apache License, Version 2.0
+<LICENSE-APACHE or https://www.apache.org/licenses/LICENSE-2.0>, or the MIT
+license <LICENSE-MIT or https://opensource.org/licenses/MIT>, at your option.
+This file may not be copied, modified, or distributed except according to
+those terms.
+*/
+
+.codegen-tabs {
+ display: grid;
+ grid-template-columns: repeat(var(--arity), minmax(200px, 1fr));
+ grid-template-rows: auto 1fr;
+ column-gap: 1rem;
+}
+
+.codegen-tabs:not(:has(> details[open]))::after {
+ grid-column: 1/-1;
+ content: 'Click one of the above headers to expand its contents.';
+ font-style: italic;
+ font-size: small;
+ text-align: center;
+}
+
+.codegen-tabs details {
+ display: grid;
+ grid-column: 1 / -1;
+ grid-row: 1 / span 2;
+ grid-template-columns: subgrid;
+ grid-template-rows: subgrid;
+}
+
+.codegen-tabs summary {
+ display: grid;
+ grid-column: var(--n) / span 1;
+ grid-row: 1;
+ z-index: 1;
+ border-bottom: 2px solid var(--headings-border-bottom-color);
+ cursor: pointer;
+}
+
+.codegen-tabs details[open] > summary {
+ background-color: var(--code-block-background-color);
+ border-bottom-color: var(--target-border-color);
+}
+
+.codegen-tabs details::details-content {
+ grid-column: 1 / -1;
+ grid-row: 2;
+}
+
+.codegen-tabs details:not([open])::details-content {
+ display: none;
+}
diff --git a/rust/zerocopy/src/byte_slice.rs b/rust/zerocopy/src/byte_slice.rs
new file mode 100644
index 0000000..a5ded4a
--- /dev/null
+++ b/rust/zerocopy/src/byte_slice.rs
@@ -0,0 +1,434 @@
+// SPDX-License-Identifier: (BSD-2-Clause OR Apache-2.0) OR MIT
+
+// Copyright 2024 The Fuchsia Authors
+//
+// Licensed under a BSD-style license <LICENSE-BSD>, Apache License, Version 2.0
+// <LICENSE-APACHE or https://www.apache.org/licenses/LICENSE-2.0>, or the MIT
+// license <LICENSE-MIT or https://opensource.org/licenses/MIT>, at your option.
+// This file may not be copied, modified, or distributed except according to
+// those terms.
+
+//! Traits for types that encapsulate a `[u8]`.
+//!
+//! These traits are used to bound the `B` parameter of [`Ref`].
+
+use core::{
+ cell,
+ ops::{Deref, DerefMut},
+};
+
+// For each trait polyfill, as soon as the corresponding feature is stable, the
+// polyfill import will be unused because method/function resolution will prefer
+// the inherent method/function over a trait method/function. Thus, we suppress
+// the `unused_imports` warning.
+//
+// See the documentation on `util::polyfills` for more information.
+#[allow(unused_imports)]
+use crate::util::polyfills::{self, NonNullExt as _, NumExt as _};
+#[cfg(doc)]
+use crate::Ref;
+
+/// A mutable or immutable reference to a byte slice.
+///
+/// `ByteSlice` abstracts over the mutability of a byte slice reference, and is
+/// implemented for various special reference types such as
+/// [`Ref<[u8]>`](core::cell::Ref) and [`RefMut<[u8]>`](core::cell::RefMut).
+///
+/// # Safety
+///
+/// Implementations of `ByteSlice` must promise that their implementations of
+/// [`Deref`] and [`DerefMut`] are "stable". In particular, given `B: ByteSlice`
+/// and `b: B`, two calls, each to either `b.deref()` or `b.deref_mut()`, must
+/// return a byte slice with the same address and length. This must hold even if
+/// the two calls are separated by an arbitrary sequence of calls to methods on
+/// `ByteSlice`, [`ByteSliceMut`], [`IntoByteSlice`], or [`IntoByteSliceMut`],
+/// or on their super-traits. This does *not* need to hold if the two calls are
+/// separated by any method calls, field accesses, or field modifications *other
+/// than* those from these traits.
+///
+/// Note that this also implies that, given `b: B`, the address and length
+/// cannot be modified via objects other than `b`, either on the same thread or
+/// on another thread.
+pub unsafe trait ByteSlice: Deref<Target = [u8]> + Sized {}
+
+/// A mutable reference to a byte slice.
+///
+/// `ByteSliceMut` abstracts over various ways of storing a mutable reference to
+/// a byte slice, and is implemented for various special reference types such as
+/// `RefMut<[u8]>`.
+///
+/// `ByteSliceMut` is a shorthand for [`ByteSlice`] and [`DerefMut`].
+pub trait ByteSliceMut: ByteSlice + DerefMut {}
+impl<B: ByteSlice + DerefMut> ByteSliceMut for B {}
+
+/// A [`ByteSlice`] which can be copied without violating dereference stability.
+///
+/// # Safety
+///
+/// If `B: CopyableByteSlice`, then the dereference stability properties
+/// required by [`ByteSlice`] (see that trait's safety documentation) do not
+/// only hold regarding two calls to `b.deref()` or `b.deref_mut()`, but also
+/// hold regarding `c.deref()` or `c.deref_mut()`, where `c` is produced by
+/// copying `b`.
+pub unsafe trait CopyableByteSlice: ByteSlice + Copy + CloneableByteSlice {}
+
+/// A [`ByteSlice`] which can be cloned without violating dereference stability.
+///
+/// # Safety
+///
+/// If `B: CloneableByteSlice`, then the dereference stability properties
+/// required by [`ByteSlice`] (see that trait's safety documentation) do not
+/// only hold regarding two calls to `b.deref()` or `b.deref_mut()`, but also
+/// hold regarding `c.deref()` or `c.deref_mut()`, where `c` is produced by
+/// `b.clone()`, `b.clone().clone()`, etc.
+pub unsafe trait CloneableByteSlice: ByteSlice + Clone {}
+
+/// A [`ByteSlice`] that can be split in two.
+///
+/// # Safety
+///
+/// Unsafe code may depend for its soundness on the assumption that `split_at`
+/// and `split_at_unchecked` are implemented correctly. In particular, given `B:
+/// SplitByteSlice` and `b: B`, if `b.deref()` returns a byte slice with address
+/// `addr` and length `len`, then if `split <= len`, both of these
+/// invocations:
+/// - `b.split_at(split)`
+/// - `b.split_at_unchecked(split)`
+///
+/// ...will return `(first, second)` such that:
+/// - `first`'s address is `addr` and its length is `split`
+/// - `second`'s address is `addr + split` and its length is `len - split`
+pub unsafe trait SplitByteSlice: ByteSlice {
+ /// Attempts to split `self` at the midpoint.
+ ///
+ /// `s.split_at(mid)` returns `Ok((s[..mid], s[mid..]))` if `mid <=
+ /// s.deref().len()` and otherwise returns `Err(s)`.
+ ///
+ /// # Safety
+ ///
+ /// Unsafe code may rely on this function correctly implementing the above
+ /// functionality.
+ #[inline]
+ fn split_at(self, mid: usize) -> Result<(Self, Self), Self> {
+ if mid <= self.deref().len() {
+ // SAFETY: Above, we ensure that `mid <= self.deref().len()`. By
+ // invariant on `ByteSlice`, a supertrait of `SplitByteSlice`,
+ // `.deref()` is guaranteed to be "stable"; i.e., it will always
+ // dereference to a byte slice of the same address and length. Thus,
+ // we can be sure that the above precondition remains satisfied
+ // through the call to `split_at_unchecked`.
+ unsafe { Ok(self.split_at_unchecked(mid)) }
+ } else {
+ Err(self)
+ }
+ }
+
+ /// Splits the slice at the midpoint, possibly omitting bounds checks.
+ ///
+ /// `s.split_at_unchecked(mid)` returns `s[..mid]` and `s[mid..]`.
+ ///
+ /// # Safety
+ ///
+ /// `mid` must not be greater than `self.deref().len()`.
+ ///
+ /// # Panics
+ ///
+ /// Implementations of this method may choose to perform a bounds check and
+ /// panic if `mid > self.deref().len()`. They may also panic for any other
+ /// reason. Since it is optional, callers must not rely on this behavior for
+ /// soundness.
+ #[must_use]
+ unsafe fn split_at_unchecked(self, mid: usize) -> (Self, Self);
+}
+
+/// A shorthand for [`SplitByteSlice`] and [`ByteSliceMut`].
+pub trait SplitByteSliceMut: SplitByteSlice + ByteSliceMut {}
+impl<B: SplitByteSlice + ByteSliceMut> SplitByteSliceMut for B {}
+
+#[allow(clippy::missing_safety_doc)] // There's a `Safety` section on `into_byte_slice`.
+/// A [`ByteSlice`] that conveys no ownership, and so can be converted into a
+/// byte slice.
+///
+/// Some `ByteSlice` types (notably, the standard library's [`Ref`] type) convey
+/// ownership, and so they cannot soundly be moved by-value into a byte slice
+/// type (`&[u8]`). Some methods in this crate's API (such as [`Ref::into_ref`])
+/// are only compatible with `ByteSlice` types without these ownership
+/// semantics.
+///
+/// [`Ref`]: core::cell::Ref
+pub unsafe trait IntoByteSlice<'a>: ByteSlice {
+ /// Coverts `self` into a `&[u8]`.
+ ///
+ /// # Safety
+ ///
+ /// The returned reference has the same address and length as `self.deref()`
+ /// and `self.deref_mut()`.
+ ///
+ /// Note that, combined with the safety invariant on [`ByteSlice`], this
+ /// safety invariant implies that the returned reference is "stable" in the
+ /// sense described in the `ByteSlice` docs.
+ fn into_byte_slice(self) -> &'a [u8];
+}
+
+#[allow(clippy::missing_safety_doc)] // There's a `Safety` section on `into_byte_slice_mut`.
+/// A [`ByteSliceMut`] that conveys no ownership, and so can be converted into a
+/// mutable byte slice.
+///
+/// Some `ByteSliceMut` types (notably, the standard library's [`RefMut`] type)
+/// convey ownership, and so they cannot soundly be moved by-value into a byte
+/// slice type (`&mut [u8]`). Some methods in this crate's API (such as
+/// [`Ref::into_mut`]) are only compatible with `ByteSliceMut` types without
+/// these ownership semantics.
+///
+/// [`RefMut`]: core::cell::RefMut
+pub unsafe trait IntoByteSliceMut<'a>: IntoByteSlice<'a> + ByteSliceMut {
+ /// Coverts `self` into a `&mut [u8]`.
+ ///
+ /// # Safety
+ ///
+ /// The returned reference has the same address and length as `self.deref()`
+ /// and `self.deref_mut()`.
+ ///
+ /// Note that, combined with the safety invariant on [`ByteSlice`], this
+ /// safety invariant implies that the returned reference is "stable" in the
+ /// sense described in the `ByteSlice` docs.
+ fn into_byte_slice_mut(self) -> &'a mut [u8];
+}
+
+// FIXME(#429): Add a "SAFETY" comment and remove this `allow`.
+#[allow(clippy::undocumented_unsafe_blocks)]
+unsafe impl ByteSlice for &[u8] {}
+
+// FIXME(#429): Add a "SAFETY" comment and remove this `allow`.
+#[allow(clippy::undocumented_unsafe_blocks)]
+unsafe impl CopyableByteSlice for &[u8] {}
+
+// FIXME(#429): Add a "SAFETY" comment and remove this `allow`.
+#[allow(clippy::undocumented_unsafe_blocks)]
+unsafe impl CloneableByteSlice for &[u8] {}
+
+// SAFETY: This delegates to `polyfills:split_at_unchecked`, which is documented
+// to correctly split `self` into two slices at the given `mid` point.
+unsafe impl SplitByteSlice for &[u8] {
+ #[inline]
+ unsafe fn split_at_unchecked(self, mid: usize) -> (Self, Self) {
+ // SAFETY: By contract on caller, `mid` is not greater than
+ // `self.len()`.
+ #[allow(clippy::multiple_unsafe_ops_per_block)]
+ unsafe {
+ (<[u8]>::get_unchecked(self, ..mid), <[u8]>::get_unchecked(self, mid..))
+ }
+ }
+}
+
+// SAFETY: See inline.
+unsafe impl<'a> IntoByteSlice<'a> for &'a [u8] {
+ #[inline(always)]
+ fn into_byte_slice(self) -> &'a [u8] {
+ // SAFETY: It would be patently insane to implement `<Deref for
+ // &[u8]>::deref` as anything other than `fn deref(&self) -> &[u8] {
+ // *self }`. Assuming this holds, then `self` is stable as required by
+ // `into_byte_slice`.
+ self
+ }
+}
+
+// FIXME(#429): Add a "SAFETY" comment and remove this `allow`.
+#[allow(clippy::undocumented_unsafe_blocks)]
+unsafe impl ByteSlice for &mut [u8] {}
+
+// SAFETY: This delegates to `polyfills:split_at_mut_unchecked`, which is
+// documented to correctly split `self` into two slices at the given `mid`
+// point.
+unsafe impl SplitByteSlice for &mut [u8] {
+ #[inline]
+ unsafe fn split_at_unchecked(self, mid: usize) -> (Self, Self) {
+ use core::slice::from_raw_parts_mut;
+
+ // `l_ptr` is non-null, because `self` is non-null, by invariant on
+ // `&mut [u8]`.
+ let l_ptr = self.as_mut_ptr();
+
+ // SAFETY: By contract on caller, `mid` is not greater than
+ // `self.len()`.
+ let r_ptr = unsafe { l_ptr.add(mid) };
+
+ let l_len = mid;
+
+ // SAFETY: By contract on caller, `mid` is not greater than
+ // `self.len()`.
+ //
+ // FIXME(#67): Remove this allow. See NumExt for more details.
+ #[allow(unstable_name_collisions)]
+ let r_len = unsafe { self.len().unchecked_sub(mid) };
+
+ // SAFETY: These invocations of `from_raw_parts_mut` satisfy its
+ // documented safety preconditions [1]:
+ // - The data `l_ptr` and `r_ptr` are valid for both reads and writes of
+ // `l_len` and `r_len` bytes, respectively, and they are trivially
+ // aligned. In particular:
+ // - The entire memory range of each slice is contained within a
+ // single allocated object, since `l_ptr` and `r_ptr` are both
+ // derived from within the address range of `self`.
+ // - Both `l_ptr` and `r_ptr` are non-null and trivially aligned.
+ // `self` is non-null by invariant on `&mut [u8]`, and the
+ // operations that derive `l_ptr` and `r_ptr` from `self` do not
+ // nullify either pointer.
+ // - The data `l_ptr` and `r_ptr` point to `l_len` and `r_len`,
+ // respectively, consecutive properly initialized values of type `u8`.
+ // This is true for `self` by invariant on `&mut [u8]`, and remains
+ // true for these two sub-slices of `self`.
+ // - The memory referenced by the returned slice cannot be accessed
+ // through any other pointer (not derived from the return value) for
+ // the duration of lifetime `'a``, because:
+ // - `split_at_unchecked` consumes `self` (which is not `Copy`),
+ // - `split_at_unchecked` does not exfiltrate any references to this
+ // memory, besides those references returned below,
+ // - the returned slices are non-overlapping.
+ // - The individual sizes of the sub-slices of `self` are no larger than
+ // `isize::MAX`, because their combined sizes are no larger than
+ // `isize::MAX`, by invariant on `self`.
+ //
+ // [1] https://doc.rust-lang.org/std/slice/fn.from_raw_parts_mut.html#safety
+ #[allow(clippy::multiple_unsafe_ops_per_block)]
+ unsafe {
+ (from_raw_parts_mut(l_ptr, l_len), from_raw_parts_mut(r_ptr, r_len))
+ }
+ }
+}
+
+// SAFETY: See inline.
+unsafe impl<'a> IntoByteSlice<'a> for &'a mut [u8] {
+ #[inline(always)]
+ fn into_byte_slice(self) -> &'a [u8] {
+ // SAFETY: It would be patently insane to implement `<Deref for &mut
+ // [u8]>::deref` as anything other than `fn deref(&self) -> &[u8] {
+ // *self }`. Assuming this holds, then `self` is stable as required by
+ // `into_byte_slice`.
+ self
+ }
+}
+
+// SAFETY: See inline.
+unsafe impl<'a> IntoByteSliceMut<'a> for &'a mut [u8] {
+ #[inline(always)]
+ fn into_byte_slice_mut(self) -> &'a mut [u8] {
+ // SAFETY: It would be patently insane to implement `<DerefMut for &mut
+ // [u8]>::deref` as anything other than `fn deref_mut(&mut self) -> &mut
+ // [u8] { *self }`. Assuming this holds, then `self` is stable as
+ // required by `into_byte_slice_mut`.
+ self
+ }
+}
+
+// FIXME(#429): Add a "SAFETY" comment and remove this `allow`.
+#[allow(clippy::undocumented_unsafe_blocks)]
+unsafe impl ByteSlice for cell::Ref<'_, [u8]> {}
+
+// SAFETY: This delegates to stdlib implementation of `Ref::map_split`, which is
+// assumed to be correct, and `SplitByteSlice::split_at_unchecked`, which is
+// documented to correctly split `self` into two slices at the given `mid`
+// point.
+unsafe impl SplitByteSlice for cell::Ref<'_, [u8]> {
+ #[inline]
+ unsafe fn split_at_unchecked(self, mid: usize) -> (Self, Self) {
+ cell::Ref::map_split(self, |slice|
+ // SAFETY: By precondition on caller, `mid` is not greater than
+ // `slice.len()`.
+ unsafe {
+ SplitByteSlice::split_at_unchecked(slice, mid)
+ })
+ }
+}
+
+// FIXME(#429): Add a "SAFETY" comment and remove this `allow`.
+#[allow(clippy::undocumented_unsafe_blocks)]
+unsafe impl ByteSlice for cell::RefMut<'_, [u8]> {}
+
+// SAFETY: This delegates to stdlib implementation of `RefMut::map_split`, which
+// is assumed to be correct, and `SplitByteSlice::split_at_unchecked`, which is
+// documented to correctly split `self` into two slices at the given `mid`
+// point.
+unsafe impl SplitByteSlice for cell::RefMut<'_, [u8]> {
+ #[inline]
+ unsafe fn split_at_unchecked(self, mid: usize) -> (Self, Self) {
+ cell::RefMut::map_split(self, |slice|
+ // SAFETY: By precondition on caller, `mid` is not greater than
+ // `slice.len()`
+ unsafe {
+ SplitByteSlice::split_at_unchecked(slice, mid)
+ })
+ }
+}
+
+#[cfg(kani)]
+mod proofs {
+ use super::*;
+
+ fn any_vec() -> Vec<u8> {
+ let len = kani::any();
+ kani::assume(len <= crate::DstLayout::MAX_SIZE);
+ vec![0u8; len]
+ }
+
+ #[kani::proof]
+ fn prove_split_at_unchecked() {
+ let v = any_vec();
+ let slc = v.as_slice();
+ let mid = kani::any();
+ kani::assume(mid <= slc.len());
+ let (l, r) = unsafe { slc.split_at_unchecked(mid) };
+ assert_eq!(l.len() + r.len(), slc.len());
+
+ let slc: *const _ = slc;
+ let l: *const _ = l;
+ let r: *const _ = r;
+
+ assert_eq!(slc.cast::<u8>(), l.cast::<u8>());
+ assert_eq!(unsafe { slc.cast::<u8>().add(mid) }, r.cast::<u8>());
+
+ let mut v = any_vec();
+ let slc = v.as_mut_slice();
+ let len = slc.len();
+ let mid = kani::any();
+ kani::assume(mid <= slc.len());
+ let (l, r) = unsafe { slc.split_at_unchecked(mid) };
+ assert_eq!(l.len() + r.len(), len);
+
+ let l: *mut _ = l;
+ let r: *mut _ = r;
+ let slc: *mut _ = slc;
+
+ assert_eq!(slc.cast::<u8>(), l.cast::<u8>());
+ assert_eq!(unsafe { slc.cast::<u8>().add(mid) }, r.cast::<u8>());
+ }
+}
+
+#[cfg(test)]
+mod tests {
+ use core::cell::RefCell;
+
+ use super::*;
+
+ #[test]
+ fn test_ref_split_at_unchecked() {
+ let cell = RefCell::new([1, 2, 3, 4]);
+ let borrow = cell.borrow();
+ let slice_ref: cell::Ref<'_, [u8]> = cell::Ref::map(borrow, |a| &a[..]);
+ // SAFETY: 2 is within bounds of [1, 2, 3, 4]
+ let (l, r) = unsafe { slice_ref.split_at_unchecked(2) };
+ assert_eq!(*l, [1, 2]);
+ assert_eq!(*r, [3, 4]);
+ }
+
+ #[test]
+ fn test_ref_mut_split_at_unchecked() {
+ let cell = RefCell::new([1, 2, 3, 4]);
+ let borrow_mut = cell.borrow_mut();
+ let slice_ref_mut: cell::RefMut<'_, [u8]> = cell::RefMut::map(borrow_mut, |a| &mut a[..]);
+ // SAFETY: 2 is within bounds of [1, 2, 3, 4]
+ let (l, r) = unsafe { slice_ref_mut.split_at_unchecked(2) };
+ assert_eq!(*l, [1, 2]);
+ assert_eq!(*r, [3, 4]);
+ }
+}
diff --git a/rust/zerocopy/src/byteorder.rs b/rust/zerocopy/src/byteorder.rs
new file mode 100644
index 0000000..8f70048
--- /dev/null
+++ b/rust/zerocopy/src/byteorder.rs
@@ -0,0 +1,1564 @@
+// SPDX-License-Identifier: (BSD-2-Clause OR Apache-2.0) OR MIT
+
+// Copyright 2019 The Fuchsia Authors
+//
+// Licensed under a BSD-style license <LICENSE-BSD>, Apache License, Version 2.0
+// <LICENSE-APACHE or https://www.apache.org/licenses/LICENSE-2.0>, or the MIT
+// license <LICENSE-MIT or https://opensource.org/licenses/MIT>, at your option.
+// This file may not be copied, modified, or distributed except according to
+// those terms.
+
+//! Byte order-aware numeric primitives.
+//!
+//! This module contains equivalents of the native multi-byte integer types with
+//! no alignment requirement and supporting byte order conversions.
+//!
+//! For each native multi-byte integer type - `u16`, `i16`, `u32`, etc - and
+//! floating point type - `f32` and `f64` - an equivalent type is defined by
+//! this module - [`U16`], [`I16`], [`U32`], [`F32`], [`F64`], etc. Unlike their
+//! native counterparts, these types have alignment 1, and take a type parameter
+//! specifying the byte order in which the bytes are stored in memory. Each type
+//! implements this crate's relevant conversion and marker traits.
+//!
+//! These two properties, taken together, make these types useful for defining
+//! data structures whose memory layout matches a wire format such as that of a
+//! network protocol or a file format. Such formats often have multi-byte values
+//! at offsets that do not respect the alignment requirements of the equivalent
+//! native types, and stored in a byte order not necessarily the same as that of
+//! the target platform.
+//!
+//! Type aliases are provided for common byte orders in the [`big_endian`],
+//! [`little_endian`], [`network_endian`], and [`native_endian`] submodules.
+//! Note that network-endian is a synonym for big-endian.
+//!
+//! # Example
+//!
+//! One use of these types is for representing network packet formats, such as
+//! UDP:
+//!
+//! ```rust
+//! use zerocopy::{*, byteorder::network_endian::U16};
+//! # use zerocopy_derive::*;
+//!
+//! #[derive(FromBytes, IntoBytes, KnownLayout, Immutable, Unaligned)]
+//! #[repr(C)]
+//! struct UdpHeader {
+//! src_port: U16,
+//! dst_port: U16,
+//! length: U16,
+//! checksum: U16,
+//! }
+//!
+//! #[derive(FromBytes, IntoBytes, KnownLayout, Immutable, Unaligned)]
+//! #[repr(C, packed)]
+//! struct UdpPacket {
+//! header: UdpHeader,
+//! body: [u8],
+//! }
+//!
+//! impl UdpPacket {
+//! fn parse(bytes: &[u8]) -> Option<&UdpPacket> {
+//! UdpPacket::ref_from_bytes(bytes).ok()
+//! }
+//! }
+//! ```
+
+use core::{
+ convert::{TryFrom, TryInto},
+ fmt::{Binary, Debug, LowerHex, Octal, UpperHex},
+ hash::Hash,
+ num::TryFromIntError,
+};
+
+use super::*;
+
+/// A type-level representation of byte order.
+///
+/// This type is implemented by [`BigEndian`] and [`LittleEndian`], which
+/// represent big-endian and little-endian byte order respectively. This module
+/// also provides a number of useful aliases for those types: [`NativeEndian`],
+/// [`NetworkEndian`], [`BE`], and [`LE`].
+///
+/// `ByteOrder` types can be used to specify the byte order of the types in this
+/// module - for example, [`U32<BigEndian>`] is a 32-bit integer stored in
+/// big-endian byte order.
+///
+/// [`U32<BigEndian>`]: U32
+pub trait ByteOrder:
+ Copy + Clone + Debug + Display + Eq + PartialEq + Ord + PartialOrd + Hash + private::Sealed
+{
+ #[doc(hidden)]
+ const ORDER: Order;
+}
+
+mod private {
+ pub trait Sealed {}
+
+ impl Sealed for super::BigEndian {}
+ impl Sealed for super::LittleEndian {}
+}
+
+#[allow(missing_copy_implementations, missing_debug_implementations)]
+#[doc(hidden)]
+pub enum Order {
+ BigEndian,
+ LittleEndian,
+}
+
+/// Big-endian byte order.
+///
+/// See [`ByteOrder`] for more details.
+#[derive(Copy, Clone, Debug, Eq, PartialEq, Ord, PartialOrd, Hash)]
+pub enum BigEndian {}
+
+impl ByteOrder for BigEndian {
+ const ORDER: Order = Order::BigEndian;
+}
+
+impl Display for BigEndian {
+ #[inline]
+ fn fmt(&self, _: &mut Formatter<'_>) -> fmt::Result {
+ match *self {}
+ }
+}
+
+/// Little-endian byte order.
+///
+/// See [`ByteOrder`] for more details.
+#[derive(Copy, Clone, Debug, Eq, PartialEq, Ord, PartialOrd, Hash)]
+pub enum LittleEndian {}
+
+impl ByteOrder for LittleEndian {
+ const ORDER: Order = Order::LittleEndian;
+}
+
+impl Display for LittleEndian {
+ #[inline]
+ fn fmt(&self, _: &mut Formatter<'_>) -> fmt::Result {
+ match *self {}
+ }
+}
+
+/// The endianness used by this platform.
+///
+/// This is a type alias for [`BigEndian`] or [`LittleEndian`] depending on the
+/// endianness of the target platform.
+#[cfg(target_endian = "big")]
+pub type NativeEndian = BigEndian;
+
+/// The endianness used by this platform.
+///
+/// This is a type alias for [`BigEndian`] or [`LittleEndian`] depending on the
+/// endianness of the target platform.
+#[cfg(target_endian = "little")]
+pub type NativeEndian = LittleEndian;
+
+/// The endianness used in many network protocols.
+///
+/// This is a type alias for [`BigEndian`].
+pub type NetworkEndian = BigEndian;
+
+/// A type alias for [`BigEndian`].
+pub type BE = BigEndian;
+
+/// A type alias for [`LittleEndian`].
+pub type LE = LittleEndian;
+
+macro_rules! impl_fmt_trait {
+ ($name:ident, $native:ident, $trait:ident) => {
+ impl<O: ByteOrder> $trait for $name<O> {
+ #[inline(always)]
+ fn fmt(&self, f: &mut Formatter<'_>) -> fmt::Result {
+ $trait::fmt(&self.get(), f)
+ }
+ }
+ };
+}
+
+macro_rules! impl_fmt_traits {
+ ($name:ident, $native:ident, "floating point number") => {
+ };
+ ($name:ident, $native:ident, "unsigned integer") => {
+ impl_fmt_traits!($name, $native, @all_types);
+ };
+ ($name:ident, $native:ident, "signed integer") => {
+ impl_fmt_traits!($name, $native, @all_types);
+ };
+ ($name:ident, $native:ident, @all_types) => {
+ impl_fmt_trait!($name, $native, Display);
+ impl_fmt_trait!($name, $native, Octal);
+ impl_fmt_trait!($name, $native, LowerHex);
+ impl_fmt_trait!($name, $native, UpperHex);
+ impl_fmt_trait!($name, $native, Binary);
+ };
+}
+
+macro_rules! impl_ops_traits {
+ ($name:ident, $native:ident, "floating point number") => {
+ impl_ops_traits!($name, $native, @all_types);
+ impl_ops_traits!($name, $native, @signed_integer_floating_point);
+
+ impl<O: ByteOrder> PartialOrd for $name<O> {
+ #[inline(always)]
+ fn partial_cmp(&self, other: &Self) -> Option<Ordering> {
+ self.get().partial_cmp(&other.get())
+ }
+ }
+ };
+ ($name:ident, $native:ident, "unsigned integer") => {
+ impl_ops_traits!($name, $native, @signed_unsigned_integer);
+ impl_ops_traits!($name, $native, @all_types);
+ };
+ ($name:ident, $native:ident, "signed integer") => {
+ impl_ops_traits!($name, $native, @signed_unsigned_integer);
+ impl_ops_traits!($name, $native, @signed_integer_floating_point);
+ impl_ops_traits!($name, $native, @all_types);
+ };
+ ($name:ident, $native:ident, @signed_unsigned_integer) => {
+ impl_ops_traits!(@without_byteorder_swap $name, $native, BitAnd, bitand, BitAndAssign, bitand_assign);
+ impl_ops_traits!(@without_byteorder_swap $name, $native, BitOr, bitor, BitOrAssign, bitor_assign);
+ impl_ops_traits!(@without_byteorder_swap $name, $native, BitXor, bitxor, BitXorAssign, bitxor_assign);
+ impl_ops_traits!(@with_byteorder_swap $name, $native, Shl, shl, ShlAssign, shl_assign);
+ impl_ops_traits!(@with_byteorder_swap $name, $native, Shr, shr, ShrAssign, shr_assign);
+
+ impl<O> core::ops::Not for $name<O> {
+ type Output = $name<O>;
+
+ #[inline(always)]
+ fn not(self) -> $name<O> {
+ let self_native = $native::from_ne_bytes(self.0);
+ $name((!self_native).to_ne_bytes(), PhantomData)
+ }
+ }
+
+ impl<O: ByteOrder> PartialOrd for $name<O> {
+ #[inline(always)]
+ fn partial_cmp(&self, other: &Self) -> Option<Ordering> {
+ Some(self.cmp(other))
+ }
+ }
+
+ impl<O: ByteOrder> Ord for $name<O> {
+ #[inline(always)]
+ fn cmp(&self, other: &Self) -> Ordering {
+ self.get().cmp(&other.get())
+ }
+ }
+
+ impl<O: ByteOrder> PartialOrd<$native> for $name<O> {
+ #[inline(always)]
+ fn partial_cmp(&self, other: &$native) -> Option<Ordering> {
+ self.get().partial_cmp(other)
+ }
+ }
+ };
+ ($name:ident, $native:ident, @signed_integer_floating_point) => {
+ impl<O: ByteOrder> core::ops::Neg for $name<O> {
+ type Output = $name<O>;
+
+ #[inline(always)]
+ fn neg(self) -> $name<O> {
+ let self_native: $native = self.get();
+ #[allow(clippy::arithmetic_side_effects)]
+ $name::<O>::new(-self_native)
+ }
+ }
+ };
+ ($name:ident, $native:ident, @all_types) => {
+ impl_ops_traits!(@with_byteorder_swap $name, $native, Add, add, AddAssign, add_assign);
+ impl_ops_traits!(@with_byteorder_swap $name, $native, Div, div, DivAssign, div_assign);
+ impl_ops_traits!(@with_byteorder_swap $name, $native, Mul, mul, MulAssign, mul_assign);
+ impl_ops_traits!(@with_byteorder_swap $name, $native, Rem, rem, RemAssign, rem_assign);
+ impl_ops_traits!(@with_byteorder_swap $name, $native, Sub, sub, SubAssign, sub_assign);
+ };
+ (@with_byteorder_swap $name:ident, $native:ident, $trait:ident, $method:ident, $trait_assign:ident, $method_assign:ident) => {
+ impl<O: ByteOrder> core::ops::$trait<$name<O>> for $name<O> {
+ type Output = $name<O>;
+
+ #[inline(always)]
+ fn $method(self, rhs: $name<O>) -> $name<O> {
+ let self_native: $native = self.get();
+ let rhs_native: $native = rhs.get();
+ let result_native = core::ops::$trait::$method(self_native, rhs_native);
+ $name::<O>::new(result_native)
+ }
+ }
+
+ impl<O: ByteOrder> core::ops::$trait<$name<O>> for $native {
+ type Output = $name<O>;
+
+ #[inline(always)]
+ fn $method(self, rhs: $name<O>) -> $name<O> {
+ let rhs_native: $native = rhs.get();
+ let result_native = core::ops::$trait::$method(self, rhs_native);
+ $name::<O>::new(result_native)
+ }
+ }
+
+ impl<O: ByteOrder> core::ops::$trait<$native> for $name<O> {
+ type Output = $name<O>;
+
+ #[inline(always)]
+ fn $method(self, rhs: $native) -> $name<O> {
+ let self_native: $native = self.get();
+ let result_native = core::ops::$trait::$method(self_native, rhs);
+ $name::<O>::new(result_native)
+ }
+ }
+
+ impl<O: ByteOrder> core::ops::$trait_assign<$name<O>> for $name<O> {
+ #[inline(always)]
+ fn $method_assign(&mut self, rhs: $name<O>) {
+ *self = core::ops::$trait::$method(*self, rhs);
+ }
+ }
+
+ impl<O: ByteOrder> core::ops::$trait_assign<$name<O>> for $native {
+ #[inline(always)]
+ fn $method_assign(&mut self, rhs: $name<O>) {
+ let rhs_native: $native = rhs.get();
+ *self = core::ops::$trait::$method(*self, rhs_native);
+ }
+ }
+
+ impl<O: ByteOrder> core::ops::$trait_assign<$native> for $name<O> {
+ #[inline(always)]
+ fn $method_assign(&mut self, rhs: $native) {
+ *self = core::ops::$trait::$method(*self, rhs);
+ }
+ }
+ };
+ // Implement traits in terms of the same trait on the native type, but
+ // without performing a byte order swap when both operands are byteorder
+ // types. This only works for bitwise operations like `&`, `|`, etc.
+ //
+ // When only one operand is a byteorder type, we still need to perform a
+ // byteorder swap.
+ (@without_byteorder_swap $name:ident, $native:ident, $trait:ident, $method:ident, $trait_assign:ident, $method_assign:ident) => {
+ impl<O: ByteOrder> core::ops::$trait<$name<O>> for $name<O> {
+ type Output = $name<O>;
+
+ #[inline(always)]
+ fn $method(self, rhs: $name<O>) -> $name<O> {
+ let self_native = $native::from_ne_bytes(self.0);
+ let rhs_native = $native::from_ne_bytes(rhs.0);
+ let result_native = core::ops::$trait::$method(self_native, rhs_native);
+ $name(result_native.to_ne_bytes(), PhantomData)
+ }
+ }
+
+ impl<O: ByteOrder> core::ops::$trait<$name<O>> for $native {
+ type Output = $name<O>;
+
+ #[inline(always)]
+ fn $method(self, rhs: $name<O>) -> $name<O> {
+ // No runtime cost - just byte packing
+ let rhs_native = $native::from_ne_bytes(rhs.0);
+ // (Maybe) runtime cost - byte order swap
+ let slf_byteorder = $name::<O>::new(self);
+ // No runtime cost - just byte packing
+ let slf_native = $native::from_ne_bytes(slf_byteorder.0);
+ // Runtime cost - perform the operation
+ let result_native = core::ops::$trait::$method(slf_native, rhs_native);
+ // No runtime cost - just byte unpacking
+ $name(result_native.to_ne_bytes(), PhantomData)
+ }
+ }
+
+ impl<O: ByteOrder> core::ops::$trait<$native> for $name<O> {
+ type Output = $name<O>;
+
+ #[inline(always)]
+ fn $method(self, rhs: $native) -> $name<O> {
+ // (Maybe) runtime cost - byte order swap
+ let rhs_byteorder = $name::<O>::new(rhs);
+ // No runtime cost - just byte packing
+ let rhs_native = $native::from_ne_bytes(rhs_byteorder.0);
+ // No runtime cost - just byte packing
+ let slf_native = $native::from_ne_bytes(self.0);
+ // Runtime cost - perform the operation
+ let result_native = core::ops::$trait::$method(slf_native, rhs_native);
+ // No runtime cost - just byte unpacking
+ $name(result_native.to_ne_bytes(), PhantomData)
+ }
+ }
+
+ impl<O: ByteOrder> core::ops::$trait_assign<$name<O>> for $name<O> {
+ #[inline(always)]
+ fn $method_assign(&mut self, rhs: $name<O>) {
+ *self = core::ops::$trait::$method(*self, rhs);
+ }
+ }
+
+ impl<O: ByteOrder> core::ops::$trait_assign<$name<O>> for $native {
+ #[inline(always)]
+ fn $method_assign(&mut self, rhs: $name<O>) {
+ // (Maybe) runtime cost - byte order swap
+ let rhs_native = rhs.get();
+ // Runtime cost - perform the operation
+ *self = core::ops::$trait::$method(*self, rhs_native);
+ }
+ }
+
+ impl<O: ByteOrder> core::ops::$trait_assign<$native> for $name<O> {
+ #[inline(always)]
+ fn $method_assign(&mut self, rhs: $native) {
+ *self = core::ops::$trait::$method(*self, rhs);
+ }
+ }
+ };
+}
+
+macro_rules! doc_comment {
+ ($x:expr, $($tt:tt)*) => {
+ #[doc = $x]
+ $($tt)*
+ };
+}
+
+macro_rules! define_max_value_constant {
+ ($name:ident, $bytes:expr, "unsigned integer") => {
+ /// The maximum value.
+ ///
+ /// This constant should be preferred to constructing a new value using
+ /// `new`, as `new` may perform an endianness swap depending on the
+ /// endianness `O` and the endianness of the platform.
+ pub const MAX_VALUE: $name<O> = $name([0xFFu8; $bytes], PhantomData);
+ };
+ // We don't provide maximum and minimum value constants for signed values
+ // and floats because there's no way to do it generically - it would require
+ // a different value depending on the value of the `ByteOrder` type
+ // parameter. Currently, one workaround would be to provide implementations
+ // for concrete implementations of that trait. In the long term, if we are
+ // ever able to make the `new` constructor a const fn, we could use that
+ // instead.
+ ($name:ident, $bytes:expr, "signed integer") => {};
+ ($name:ident, $bytes:expr, "floating point number") => {};
+}
+
+macro_rules! define_type {
+ (
+ $article:ident,
+ $description:expr,
+ $name:ident,
+ $native:ident,
+ $bits:expr,
+ $bytes:expr,
+ $from_be_fn:path,
+ $to_be_fn:path,
+ $from_le_fn:path,
+ $to_le_fn:path,
+ $number_kind:tt,
+ [$($larger_native:ty),*],
+ [$($larger_native_try:ty),*],
+ [$($larger_byteorder:ident),*],
+ [$($larger_byteorder_try:ident),*]
+ ) => {
+ doc_comment! {
+ concat!($description, " stored in a given byte order.
+
+`", stringify!($name), "` is like the native `", stringify!($native), "` type with
+two major differences: First, it has no alignment requirement (its alignment is 1).
+Second, the endianness of its memory layout is given by the type parameter `O`,
+which can be any type which implements [`ByteOrder`]. In particular, this refers
+to [`BigEndian`], [`LittleEndian`], [`NativeEndian`], and [`NetworkEndian`].
+
+", stringify!($article), " `", stringify!($name), "` can be constructed using
+the [`new`] method, and its contained value can be obtained as a native
+`",stringify!($native), "` using the [`get`] method, or updated in place with
+the [`set`] method. In all cases, if the endianness `O` is not the same as the
+endianness of the current platform, an endianness swap will be performed in
+order to uphold the invariants that a) the layout of `", stringify!($name), "`
+has endianness `O` and that, b) the layout of `", stringify!($native), "` has
+the platform's native endianness.
+
+`", stringify!($name), "` implements [`FromBytes`], [`IntoBytes`], and [`Unaligned`],
+making it useful for parsing and serialization. See the module documentation for an
+example of how it can be used for parsing UDP packets.
+
+[`new`]: crate::byteorder::", stringify!($name), "::new
+[`get`]: crate::byteorder::", stringify!($name), "::get
+[`set`]: crate::byteorder::", stringify!($name), "::set
+[`FromBytes`]: crate::FromBytes
+[`IntoBytes`]: crate::IntoBytes
+[`Unaligned`]: crate::Unaligned"),
+ #[derive(Copy, Clone, Eq, PartialEq, Hash)]
+ #[cfg_attr(any(feature = "derive", test), derive(KnownLayout, Immutable, FromBytes, IntoBytes, Unaligned))]
+ #[repr(transparent)]
+ pub struct $name<O>([u8; $bytes], PhantomData<O>);
+ }
+
+ #[cfg(not(any(feature = "derive", test)))]
+ impl_known_layout!(O => $name<O>);
+
+ #[allow(unused_unsafe)] // Unused when `feature = "derive"`.
+ // SAFETY: `$name<O>` is `repr(transparent)`, and so it has the same
+ // layout as its only non-zero field, which is a `u8` array. `u8` arrays
+ // are `Immutable`, `TryFromBytes`, `FromZeros`, `FromBytes`,
+ // `IntoBytes`, and `Unaligned`.
+ #[allow(clippy::multiple_unsafe_ops_per_block)]
+ const _: () = unsafe {
+ impl_or_verify!(O => Immutable for $name<O>);
+ impl_or_verify!(O => TryFromBytes for $name<O>);
+ impl_or_verify!(O => FromZeros for $name<O>);
+ impl_or_verify!(O => FromBytes for $name<O>);
+ impl_or_verify!(O => IntoBytes for $name<O>);
+ impl_or_verify!(O => Unaligned for $name<O>);
+ };
+
+ impl<O> Default for $name<O> {
+ #[inline(always)]
+ fn default() -> $name<O> {
+ $name::ZERO
+ }
+ }
+
+ impl<O> $name<O> {
+ /// The value zero.
+ ///
+ /// This constant should be preferred to constructing a new value
+ /// using `new`, as `new` may perform an endianness swap depending
+ /// on the endianness and platform.
+ pub const ZERO: $name<O> = $name([0u8; $bytes], PhantomData);
+
+ define_max_value_constant!($name, $bytes, $number_kind);
+
+ /// Constructs a new value from bytes which are already in `O` byte
+ /// order.
+ #[must_use = "has no side effects"]
+ #[inline(always)]
+ pub const fn from_bytes(bytes: [u8; $bytes]) -> $name<O> {
+ $name(bytes, PhantomData)
+ }
+
+ /// Extracts the bytes of `self` without swapping the byte order.
+ ///
+ /// The returned bytes will be in `O` byte order.
+ #[must_use = "has no side effects"]
+ #[inline(always)]
+ pub const fn to_bytes(self) -> [u8; $bytes] {
+ self.0
+ }
+ }
+
+ impl<O: ByteOrder> $name<O> {
+ maybe_const_trait_bounded_fn! {
+ /// Constructs a new value, possibly performing an endianness
+ /// swap to guarantee that the returned value has endianness
+ /// `O`.
+ #[must_use = "has no side effects"]
+ #[inline(always)]
+ pub const fn new(n: $native) -> $name<O> {
+ let bytes = match O::ORDER {
+ Order::BigEndian => $to_be_fn(n),
+ Order::LittleEndian => $to_le_fn(n),
+ };
+
+ $name(bytes, PhantomData)
+ }
+ }
+
+ maybe_const_trait_bounded_fn! {
+ /// Returns the value as a primitive type, possibly performing
+ /// an endianness swap to guarantee that the return value has
+ /// the endianness of the native platform.
+ #[must_use = "has no side effects"]
+ #[inline(always)]
+ pub const fn get(self) -> $native {
+ match O::ORDER {
+ Order::BigEndian => $from_be_fn(self.0),
+ Order::LittleEndian => $from_le_fn(self.0),
+ }
+ }
+ }
+
+ /// Updates the value in place as a primitive type, possibly
+ /// performing an endianness swap to guarantee that the stored value
+ /// has the endianness `O`.
+ #[inline(always)]
+ pub fn set(&mut self, n: $native) {
+ *self = Self::new(n);
+ }
+ }
+
+ // The reasoning behind which traits to implement here is to only
+ // implement traits which won't cause inference issues. Notably,
+ // comparison traits like PartialEq and PartialOrd tend to cause
+ // inference issues.
+
+ impl<O: ByteOrder> From<$name<O>> for [u8; $bytes] {
+ #[inline(always)]
+ fn from(x: $name<O>) -> [u8; $bytes] {
+ x.0
+ }
+ }
+
+ impl<O: ByteOrder> From<[u8; $bytes]> for $name<O> {
+ #[inline(always)]
+ fn from(bytes: [u8; $bytes]) -> $name<O> {
+ $name(bytes, PhantomData)
+ }
+ }
+
+ impl<O: ByteOrder> From<$name<O>> for $native {
+ #[inline(always)]
+ fn from(x: $name<O>) -> $native {
+ x.get()
+ }
+ }
+
+ impl<O: ByteOrder> From<$native> for $name<O> {
+ #[inline(always)]
+ fn from(x: $native) -> $name<O> {
+ $name::new(x)
+ }
+ }
+
+ $(
+ impl<O: ByteOrder> From<$name<O>> for $larger_native {
+ #[inline(always)]
+ fn from(x: $name<O>) -> $larger_native {
+ x.get().into()
+ }
+ }
+ )*
+
+ $(
+ impl<O: ByteOrder> TryFrom<$larger_native_try> for $name<O> {
+ type Error = TryFromIntError;
+ #[inline(always)]
+ fn try_from(x: $larger_native_try) -> Result<$name<O>, TryFromIntError> {
+ $native::try_from(x).map($name::new)
+ }
+ }
+ )*
+
+ $(
+ impl<O: ByteOrder, P: ByteOrder> From<$name<O>> for $larger_byteorder<P> {
+ #[inline(always)]
+ fn from(x: $name<O>) -> $larger_byteorder<P> {
+ $larger_byteorder::new(x.get().into())
+ }
+ }
+ )*
+
+ $(
+ impl<O: ByteOrder, P: ByteOrder> TryFrom<$larger_byteorder_try<P>> for $name<O> {
+ type Error = TryFromIntError;
+ #[inline(always)]
+ fn try_from(x: $larger_byteorder_try<P>) -> Result<$name<O>, TryFromIntError> {
+ x.get().try_into().map($name::new)
+ }
+ }
+ )*
+
+ impl<O> AsRef<[u8; $bytes]> for $name<O> {
+ #[inline(always)]
+ fn as_ref(&self) -> &[u8; $bytes] {
+ &self.0
+ }
+ }
+
+ impl<O> AsMut<[u8; $bytes]> for $name<O> {
+ #[inline(always)]
+ fn as_mut(&mut self) -> &mut [u8; $bytes] {
+ &mut self.0
+ }
+ }
+
+ impl<O> PartialEq<$name<O>> for [u8; $bytes] {
+ #[inline(always)]
+ fn eq(&self, other: &$name<O>) -> bool {
+ self.eq(&other.0)
+ }
+ }
+
+ impl<O> PartialEq<[u8; $bytes]> for $name<O> {
+ #[inline(always)]
+ fn eq(&self, other: &[u8; $bytes]) -> bool {
+ self.0.eq(other)
+ }
+ }
+
+ impl<O: ByteOrder> PartialEq<$native> for $name<O> {
+ #[inline(always)]
+ fn eq(&self, other: &$native) -> bool {
+ self.get().eq(other)
+ }
+ }
+
+ impl_fmt_traits!($name, $native, $number_kind);
+ impl_ops_traits!($name, $native, $number_kind);
+
+ impl<O: ByteOrder> Debug for $name<O> {
+ #[inline]
+ fn fmt(&self, f: &mut Formatter<'_>) -> fmt::Result {
+ // This results in a format like "U16(42)".
+ f.debug_tuple(stringify!($name)).field(&self.get()).finish()
+ }
+ }
+ };
+}
+
+define_type!(
+ A,
+ "A 16-bit unsigned integer",
+ U16,
+ u16,
+ 16,
+ 2,
+ u16::from_be_bytes,
+ u16::to_be_bytes,
+ u16::from_le_bytes,
+ u16::to_le_bytes,
+ "unsigned integer",
+ [u32, u64, u128, usize],
+ [u32, u64, u128, usize],
+ [U32, U64, U128, Usize],
+ [U32, U64, U128, Usize]
+);
+define_type!(
+ A,
+ "A 32-bit unsigned integer",
+ U32,
+ u32,
+ 32,
+ 4,
+ u32::from_be_bytes,
+ u32::to_be_bytes,
+ u32::from_le_bytes,
+ u32::to_le_bytes,
+ "unsigned integer",
+ [u64, u128],
+ [u64, u128],
+ [U64, U128],
+ [U64, U128]
+);
+define_type!(
+ A,
+ "A 64-bit unsigned integer",
+ U64,
+ u64,
+ 64,
+ 8,
+ u64::from_be_bytes,
+ u64::to_be_bytes,
+ u64::from_le_bytes,
+ u64::to_le_bytes,
+ "unsigned integer",
+ [u128],
+ [u128],
+ [U128],
+ [U128]
+);
+define_type!(
+ A,
+ "A 128-bit unsigned integer",
+ U128,
+ u128,
+ 128,
+ 16,
+ u128::from_be_bytes,
+ u128::to_be_bytes,
+ u128::from_le_bytes,
+ u128::to_le_bytes,
+ "unsigned integer",
+ [],
+ [],
+ [],
+ []
+);
+define_type!(
+ A,
+ "A word-sized unsigned integer",
+ Usize,
+ usize,
+ mem::size_of::<usize>() * 8,
+ mem::size_of::<usize>(),
+ usize::from_be_bytes,
+ usize::to_be_bytes,
+ usize::from_le_bytes,
+ usize::to_le_bytes,
+ "unsigned integer",
+ [],
+ [],
+ [],
+ []
+);
+define_type!(
+ An,
+ "A 16-bit signed integer",
+ I16,
+ i16,
+ 16,
+ 2,
+ i16::from_be_bytes,
+ i16::to_be_bytes,
+ i16::from_le_bytes,
+ i16::to_le_bytes,
+ "signed integer",
+ [i32, i64, i128, isize],
+ [i32, i64, i128, isize],
+ [I32, I64, I128, Isize],
+ [I32, I64, I128, Isize]
+);
+define_type!(
+ An,
+ "A 32-bit signed integer",
+ I32,
+ i32,
+ 32,
+ 4,
+ i32::from_be_bytes,
+ i32::to_be_bytes,
+ i32::from_le_bytes,
+ i32::to_le_bytes,
+ "signed integer",
+ [i64, i128],
+ [i64, i128],
+ [I64, I128],
+ [I64, I128]
+);
+define_type!(
+ An,
+ "A 64-bit signed integer",
+ I64,
+ i64,
+ 64,
+ 8,
+ i64::from_be_bytes,
+ i64::to_be_bytes,
+ i64::from_le_bytes,
+ i64::to_le_bytes,
+ "signed integer",
+ [i128],
+ [i128],
+ [I128],
+ [I128]
+);
+define_type!(
+ An,
+ "A 128-bit signed integer",
+ I128,
+ i128,
+ 128,
+ 16,
+ i128::from_be_bytes,
+ i128::to_be_bytes,
+ i128::from_le_bytes,
+ i128::to_le_bytes,
+ "signed integer",
+ [],
+ [],
+ [],
+ []
+);
+define_type!(
+ An,
+ "A word-sized signed integer",
+ Isize,
+ isize,
+ mem::size_of::<isize>() * 8,
+ mem::size_of::<isize>(),
+ isize::from_be_bytes,
+ isize::to_be_bytes,
+ isize::from_le_bytes,
+ isize::to_le_bytes,
+ "signed integer",
+ [],
+ [],
+ [],
+ []
+);
+
+// FIXME(https://github.com/rust-lang/rust/issues/72447): Use the endianness
+// conversion methods directly once those are const-stable.
+macro_rules! define_float_conversion {
+ ($ty:ty, $bits:ident, $bytes:expr, $mod:ident) => {
+ mod $mod {
+ use super::*;
+
+ define_float_conversion!($ty, $bits, $bytes, from_be_bytes, to_be_bytes);
+ define_float_conversion!($ty, $bits, $bytes, from_le_bytes, to_le_bytes);
+ }
+ };
+ ($ty:ty, $bits:ident, $bytes:expr, $from:ident, $to:ident) => {
+ // Clippy: The suggestion of using `from_bits()` instead doesn't work
+ // because `from_bits` is not const-stable on our MSRV.
+ #[allow(clippy::unnecessary_transmutes)]
+ pub(crate) const fn $from(bytes: [u8; $bytes]) -> $ty {
+ transmute!($bits::$from(bytes))
+ }
+
+ pub(crate) const fn $to(f: $ty) -> [u8; $bytes] {
+ // Clippy: The suggestion of using `f.to_bits()` instead doesn't
+ // work because `to_bits` is not const-stable on our MSRV.
+ #[allow(clippy::unnecessary_transmutes)]
+ let bits: $bits = transmute!(f);
+ bits.$to()
+ }
+ };
+}
+
+define_float_conversion!(f32, u32, 4, f32_ext);
+define_float_conversion!(f64, u64, 8, f64_ext);
+
+define_type!(
+ An,
+ "A 32-bit floating point number",
+ F32,
+ f32,
+ 32,
+ 4,
+ f32_ext::from_be_bytes,
+ f32_ext::to_be_bytes,
+ f32_ext::from_le_bytes,
+ f32_ext::to_le_bytes,
+ "floating point number",
+ [f64],
+ [],
+ [F64],
+ []
+);
+define_type!(
+ An,
+ "A 64-bit floating point number",
+ F64,
+ f64,
+ 64,
+ 8,
+ f64_ext::from_be_bytes,
+ f64_ext::to_be_bytes,
+ f64_ext::from_le_bytes,
+ f64_ext::to_le_bytes,
+ "floating point number",
+ [],
+ [],
+ [],
+ []
+);
+
+macro_rules! module {
+ ($name:ident, $trait:ident, $endianness_str:expr) => {
+ /// Numeric primitives stored in
+ #[doc = $endianness_str]
+ /// byte order.
+ pub mod $name {
+ use super::$trait;
+
+ module!(@ty U16, $trait, "16-bit unsigned integer", $endianness_str);
+ module!(@ty U32, $trait, "32-bit unsigned integer", $endianness_str);
+ module!(@ty U64, $trait, "64-bit unsigned integer", $endianness_str);
+ module!(@ty U128, $trait, "128-bit unsigned integer", $endianness_str);
+ module!(@ty I16, $trait, "16-bit signed integer", $endianness_str);
+ module!(@ty I32, $trait, "32-bit signed integer", $endianness_str);
+ module!(@ty I64, $trait, "64-bit signed integer", $endianness_str);
+ module!(@ty I128, $trait, "128-bit signed integer", $endianness_str);
+ module!(@ty F32, $trait, "32-bit floating point number", $endianness_str);
+ module!(@ty F64, $trait, "64-bit floating point number", $endianness_str);
+ }
+ };
+ (@ty $ty:ident, $trait:ident, $desc_str:expr, $endianness_str:expr) => {
+ /// A
+ #[doc = $desc_str]
+ /// stored in
+ #[doc = $endianness_str]
+ /// byte order.
+ pub type $ty = crate::byteorder::$ty<$trait>;
+ };
+}
+
+module!(big_endian, BigEndian, "big-endian");
+module!(little_endian, LittleEndian, "little-endian");
+module!(network_endian, NetworkEndian, "network-endian");
+module!(native_endian, NativeEndian, "native-endian");
+
+#[cfg(any(test, kani))]
+mod tests {
+ use super::*;
+
+ #[cfg(not(kani))]
+ mod compatibility {
+ pub(super) use rand::{
+ distributions::{Distribution, Standard},
+ rngs::SmallRng,
+ Rng, SeedableRng,
+ };
+
+ pub(crate) trait Arbitrary {}
+
+ impl<T> Arbitrary for T {}
+ }
+
+ #[cfg(kani)]
+ mod compatibility {
+ pub(crate) use kani::Arbitrary;
+
+ pub(crate) struct SmallRng;
+
+ impl SmallRng {
+ pub(crate) fn seed_from_u64(_state: u64) -> Self {
+ Self
+ }
+ }
+
+ pub(crate) trait Rng {
+ fn sample<T, D: Distribution<T>>(&mut self, _distr: D) -> T
+ where
+ T: Arbitrary,
+ {
+ kani::any()
+ }
+ }
+
+ impl Rng for SmallRng {}
+
+ pub(crate) trait Distribution<T> {}
+ impl<T, U> Distribution<T> for U {}
+
+ pub(crate) struct Standard;
+ }
+
+ use compatibility::*;
+
+ // A native integer type (u16, i32, etc).
+ trait Native: Arbitrary + FromBytes + IntoBytes + Immutable + Copy + PartialEq + Debug {
+ const ZERO: Self;
+ const MAX_VALUE: Self;
+
+ type Distribution: Distribution<Self>;
+ const DIST: Self::Distribution;
+
+ fn rand<R: Rng>(rng: &mut R) -> Self {
+ rng.sample(Self::DIST)
+ }
+
+ #[cfg_attr(kani, allow(unused))]
+ fn checked_add(self, rhs: Self) -> Option<Self>;
+
+ #[cfg_attr(kani, allow(unused))]
+ fn checked_div(self, rhs: Self) -> Option<Self>;
+
+ #[cfg_attr(kani, allow(unused))]
+ fn checked_mul(self, rhs: Self) -> Option<Self>;
+
+ #[cfg_attr(kani, allow(unused))]
+ fn checked_rem(self, rhs: Self) -> Option<Self>;
+
+ #[cfg_attr(kani, allow(unused))]
+ fn checked_sub(self, rhs: Self) -> Option<Self>;
+
+ #[cfg_attr(kani, allow(unused))]
+ fn checked_shl(self, rhs: Self) -> Option<Self>;
+
+ #[cfg_attr(kani, allow(unused))]
+ fn checked_shr(self, rhs: Self) -> Option<Self>;
+
+ fn is_nan(self) -> bool;
+
+ /// For `f32` and `f64`, NaN values are not considered equal to
+ /// themselves. This method is like `assert_eq!`, but it treats NaN
+ /// values as equal.
+ fn assert_eq_or_nan(self, other: Self) {
+ let slf = (!self.is_nan()).then(|| self);
+ let other = (!other.is_nan()).then(|| other);
+ assert_eq!(slf, other);
+ }
+ }
+
+ trait ByteArray:
+ FromBytes + IntoBytes + Immutable + Copy + AsRef<[u8]> + AsMut<[u8]> + Debug + Default + Eq
+ {
+ /// Invert the order of the bytes in the array.
+ fn invert(self) -> Self;
+ }
+
+ trait ByteOrderType:
+ FromBytes + IntoBytes + Unaligned + Copy + Eq + Debug + Hash + From<Self::Native>
+ {
+ type Native: Native;
+ type ByteArray: ByteArray;
+
+ const ZERO: Self;
+
+ fn new(native: Self::Native) -> Self;
+ fn get(self) -> Self::Native;
+ fn set(&mut self, native: Self::Native);
+ fn from_bytes(bytes: Self::ByteArray) -> Self;
+ fn into_bytes(self) -> Self::ByteArray;
+
+ /// For `f32` and `f64`, NaN values are not considered equal to
+ /// themselves. This method is like `assert_eq!`, but it treats NaN
+ /// values as equal.
+ fn assert_eq_or_nan(self, other: Self) {
+ let slf = (!self.get().is_nan()).then(|| self);
+ let other = (!other.get().is_nan()).then(|| other);
+ assert_eq!(slf, other);
+ }
+ }
+
+ trait ByteOrderTypeUnsigned: ByteOrderType {
+ const MAX_VALUE: Self;
+ }
+
+ macro_rules! impl_byte_array {
+ ($bytes:expr) => {
+ impl ByteArray for [u8; $bytes] {
+ fn invert(mut self) -> [u8; $bytes] {
+ self.reverse();
+ self
+ }
+ }
+ };
+ }
+
+ impl_byte_array!(2);
+ impl_byte_array!(4);
+ impl_byte_array!(8);
+ impl_byte_array!(16);
+
+ macro_rules! impl_byte_order_type_unsigned {
+ ($name:ident, unsigned) => {
+ impl<O: ByteOrder> ByteOrderTypeUnsigned for $name<O> {
+ const MAX_VALUE: $name<O> = $name::MAX_VALUE;
+ }
+ };
+ ($name:ident, signed) => {};
+ }
+
+ macro_rules! impl_traits {
+ ($name:ident, $native:ident, $sign:ident $(, @$float:ident)?) => {
+ impl Native for $native {
+ // For some types, `0 as $native` is required (for example, when
+ // `$native` is a floating-point type; `0` is an integer), but
+ // for other types, it's a trivial cast. In all cases, Clippy
+ // thinks it's dangerous.
+ #[allow(trivial_numeric_casts, clippy::as_conversions)]
+ const ZERO: $native = 0 as $native;
+ const MAX_VALUE: $native = $native::MAX;
+
+ type Distribution = Standard;
+ const DIST: Standard = Standard;
+
+ impl_traits!(@float_dependent_methods $(@$float)?);
+ }
+
+ impl<O: ByteOrder> ByteOrderType for $name<O> {
+ type Native = $native;
+ type ByteArray = [u8; mem::size_of::<$native>()];
+
+ const ZERO: $name<O> = $name::ZERO;
+
+ fn new(native: $native) -> $name<O> {
+ $name::new(native)
+ }
+
+ fn get(self) -> $native {
+ $name::get(self)
+ }
+
+ fn set(&mut self, native: $native) {
+ $name::set(self, native)
+ }
+
+ fn from_bytes(bytes: [u8; mem::size_of::<$native>()]) -> $name<O> {
+ $name::from(bytes)
+ }
+
+ fn into_bytes(self) -> [u8; mem::size_of::<$native>()] {
+ <[u8; mem::size_of::<$native>()]>::from(self)
+ }
+ }
+
+ impl_byte_order_type_unsigned!($name, $sign);
+ };
+ (@float_dependent_methods) => {
+ fn checked_add(self, rhs: Self) -> Option<Self> { self.checked_add(rhs) }
+ fn checked_div(self, rhs: Self) -> Option<Self> { self.checked_div(rhs) }
+ fn checked_mul(self, rhs: Self) -> Option<Self> { self.checked_mul(rhs) }
+ fn checked_rem(self, rhs: Self) -> Option<Self> { self.checked_rem(rhs) }
+ fn checked_sub(self, rhs: Self) -> Option<Self> { self.checked_sub(rhs) }
+ fn checked_shl(self, rhs: Self) -> Option<Self> { self.checked_shl(rhs.try_into().unwrap_or(u32::MAX)) }
+ fn checked_shr(self, rhs: Self) -> Option<Self> { self.checked_shr(rhs.try_into().unwrap_or(u32::MAX)) }
+ fn is_nan(self) -> bool { false }
+ };
+ (@float_dependent_methods @float) => {
+ fn checked_add(self, rhs: Self) -> Option<Self> { Some(self + rhs) }
+ fn checked_div(self, rhs: Self) -> Option<Self> { Some(self / rhs) }
+ fn checked_mul(self, rhs: Self) -> Option<Self> { Some(self * rhs) }
+ fn checked_rem(self, rhs: Self) -> Option<Self> { Some(self % rhs) }
+ fn checked_sub(self, rhs: Self) -> Option<Self> { Some(self - rhs) }
+ fn checked_shl(self, _rhs: Self) -> Option<Self> { unimplemented!() }
+ fn checked_shr(self, _rhs: Self) -> Option<Self> { unimplemented!() }
+ fn is_nan(self) -> bool { self.is_nan() }
+ };
+ }
+
+ impl_traits!(U16, u16, unsigned);
+ impl_traits!(U32, u32, unsigned);
+ impl_traits!(U64, u64, unsigned);
+ impl_traits!(U128, u128, unsigned);
+ impl_traits!(Usize, usize, unsigned);
+ impl_traits!(I16, i16, signed);
+ impl_traits!(I32, i32, signed);
+ impl_traits!(I64, i64, signed);
+ impl_traits!(I128, i128, signed);
+ impl_traits!(Isize, isize, unsigned);
+ impl_traits!(F32, f32, signed, @float);
+ impl_traits!(F64, f64, signed, @float);
+
+ macro_rules! call_for_unsigned_types {
+ ($fn:ident, $byteorder:ident) => {
+ $fn::<U16<$byteorder>>();
+ $fn::<U32<$byteorder>>();
+ $fn::<U64<$byteorder>>();
+ $fn::<U128<$byteorder>>();
+ $fn::<Usize<$byteorder>>();
+ };
+ }
+
+ macro_rules! call_for_signed_types {
+ ($fn:ident, $byteorder:ident) => {
+ $fn::<I16<$byteorder>>();
+ $fn::<I32<$byteorder>>();
+ $fn::<I64<$byteorder>>();
+ $fn::<I128<$byteorder>>();
+ $fn::<Isize<$byteorder>>();
+ };
+ }
+
+ macro_rules! call_for_float_types {
+ ($fn:ident, $byteorder:ident) => {
+ $fn::<F32<$byteorder>>();
+ $fn::<F64<$byteorder>>();
+ };
+ }
+
+ macro_rules! call_for_all_types {
+ ($fn:ident, $byteorder:ident) => {
+ call_for_unsigned_types!($fn, $byteorder);
+ call_for_signed_types!($fn, $byteorder);
+ call_for_float_types!($fn, $byteorder);
+ };
+ }
+
+ #[cfg(target_endian = "big")]
+ type NonNativeEndian = LittleEndian;
+ #[cfg(target_endian = "little")]
+ type NonNativeEndian = BigEndian;
+
+ // We use a `u64` seed so that we can use `SeedableRng::seed_from_u64`.
+ // `SmallRng`'s `SeedableRng::Seed` differs by platform, so if we wanted to
+ // call `SeedableRng::from_seed`, which takes a `Seed`, we would need
+ // conditional compilation by `target_pointer_width`.
+ const RNG_SEED: u64 = 0x7A03CAE2F32B5B8F;
+
+ const RAND_ITERS: usize = if cfg!(any(miri, kani)) {
+ // The tests below which use this constant used to take a very long time
+ // on Miri, which slows down local development and CI jobs. We're not
+ // using Miri to check for the correctness of our code, but rather its
+ // soundness, and at least in the context of these particular tests, a
+ // single loop iteration is just as good for surfacing UB as multiple
+ // iterations are.
+ //
+ // As of the writing of this comment, here's one set of measurements:
+ //
+ // $ # RAND_ITERS == 1
+ // $ cargo miri test -- -Z unstable-options --report-time endian
+ // test byteorder::tests::test_native_endian ... ok <0.049s>
+ // test byteorder::tests::test_non_native_endian ... ok <0.061s>
+ //
+ // $ # RAND_ITERS == 1024
+ // $ cargo miri test -- -Z unstable-options --report-time endian
+ // test byteorder::tests::test_native_endian ... ok <25.716s>
+ // test byteorder::tests::test_non_native_endian ... ok <38.127s>
+ 1
+ } else {
+ 1024
+ };
+
+ #[test]
+ fn test_const_methods() {
+ use big_endian::*;
+
+ #[rustversion::since(1.61.0)]
+ const _U: U16 = U16::new(0);
+ #[rustversion::since(1.61.0)]
+ const _NATIVE: u16 = _U.get();
+ const _FROM_BYTES: U16 = U16::from_bytes([0, 1]);
+ const _BYTES: [u8; 2] = _FROM_BYTES.to_bytes();
+ }
+
+ #[cfg_attr(test, test)]
+ #[cfg_attr(kani, kani::proof)]
+ fn test_zero() {
+ fn test_zero<T: ByteOrderType>() {
+ assert_eq!(T::ZERO.get(), T::Native::ZERO);
+ }
+
+ call_for_all_types!(test_zero, NativeEndian);
+ call_for_all_types!(test_zero, NonNativeEndian);
+ }
+
+ #[cfg_attr(test, test)]
+ #[cfg_attr(kani, kani::proof)]
+ fn test_max_value() {
+ fn test_max_value<T: ByteOrderTypeUnsigned>() {
+ assert_eq!(T::MAX_VALUE.get(), T::Native::MAX_VALUE);
+ }
+
+ call_for_unsigned_types!(test_max_value, NativeEndian);
+ call_for_unsigned_types!(test_max_value, NonNativeEndian);
+ }
+
+ #[cfg_attr(test, test)]
+ #[cfg_attr(kani, kani::proof)]
+ fn test_endian() {
+ fn test<T: ByteOrderType>(invert: bool) {
+ let mut r = SmallRng::seed_from_u64(RNG_SEED);
+ for _ in 0..RAND_ITERS {
+ let native = T::Native::rand(&mut r);
+ let mut bytes = T::ByteArray::default();
+ bytes.as_mut_bytes().copy_from_slice(native.as_bytes());
+ if invert {
+ bytes = bytes.invert();
+ }
+ let mut from_native = T::new(native);
+ let from_bytes = T::from_bytes(bytes);
+
+ from_native.assert_eq_or_nan(from_bytes);
+ from_native.get().assert_eq_or_nan(native);
+ from_bytes.get().assert_eq_or_nan(native);
+
+ assert_eq!(from_native.into_bytes(), bytes);
+ assert_eq!(from_bytes.into_bytes(), bytes);
+
+ let updated = T::Native::rand(&mut r);
+ from_native.set(updated);
+ from_native.get().assert_eq_or_nan(updated);
+ }
+ }
+
+ fn test_native<T: ByteOrderType>() {
+ test::<T>(false);
+ }
+
+ fn test_non_native<T: ByteOrderType>() {
+ test::<T>(true);
+ }
+
+ call_for_all_types!(test_native, NativeEndian);
+ call_for_all_types!(test_non_native, NonNativeEndian);
+ }
+
+ #[test]
+ fn test_ops_impls() {
+ // Test implementations of traits in `core::ops`. Some of these are
+ // fairly banal, but some are optimized to perform the operation without
+ // swapping byte order (namely, bit-wise operations which are identical
+ // regardless of byte order). These are important to test, and while
+ // we're testing those anyway, it's trivial to test all of the impls.
+
+ fn test<T, FTT, FTN, FNT, FNN, FNNChecked, FATT, FATN, FANT>(
+ op_t_t: FTT,
+ op_t_n: FTN,
+ op_n_t: FNT,
+ op_n_n: FNN,
+ op_n_n_checked: Option<FNNChecked>,
+ op_assign: Option<(FATT, FATN, FANT)>,
+ ) where
+ T: ByteOrderType,
+ FTT: Fn(T, T) -> T,
+ FTN: Fn(T, T::Native) -> T,
+ FNT: Fn(T::Native, T) -> T,
+ FNN: Fn(T::Native, T::Native) -> T::Native,
+ FNNChecked: Fn(T::Native, T::Native) -> Option<T::Native>,
+ FATT: Fn(&mut T, T),
+ FATN: Fn(&mut T, T::Native),
+ FANT: Fn(&mut T::Native, T),
+ {
+ let mut r = SmallRng::seed_from_u64(RNG_SEED);
+ for _ in 0..RAND_ITERS {
+ let n0 = T::Native::rand(&mut r);
+ let n1 = T::Native::rand(&mut r);
+ let t0 = T::new(n0);
+ let t1 = T::new(n1);
+
+ // If this operation would overflow/underflow, skip it rather
+ // than attempt to catch and recover from panics.
+ if matches!(&op_n_n_checked, Some(checked) if checked(n0, n1).is_none()) {
+ continue;
+ }
+
+ let t_t_res = op_t_t(t0, t1);
+ let t_n_res = op_t_n(t0, n1);
+ let n_t_res = op_n_t(n0, t1);
+ let n_n_res = op_n_n(n0, n1);
+
+ // For `f32` and `f64`, NaN values are not considered equal to
+ // themselves. We store `Option<f32>`/`Option<f64>` and store
+ // NaN as `None` so they can still be compared.
+ let val_or_none = |t: T| (!T::Native::is_nan(t.get())).then(|| t.get());
+ let t_t_res = val_or_none(t_t_res);
+ let t_n_res = val_or_none(t_n_res);
+ let n_t_res = val_or_none(n_t_res);
+ let n_n_res = (!T::Native::is_nan(n_n_res)).then(|| n_n_res);
+ assert_eq!(t_t_res, n_n_res);
+ assert_eq!(t_n_res, n_n_res);
+ assert_eq!(n_t_res, n_n_res);
+
+ if let Some((op_assign_t_t, op_assign_t_n, op_assign_n_t)) = &op_assign {
+ let mut t_t_res = t0;
+ op_assign_t_t(&mut t_t_res, t1);
+ let mut t_n_res = t0;
+ op_assign_t_n(&mut t_n_res, n1);
+ let mut n_t_res = n0;
+ op_assign_n_t(&mut n_t_res, t1);
+
+ // For `f32` and `f64`, NaN values are not considered equal to
+ // themselves. We store `Option<f32>`/`Option<f64>` and store
+ // NaN as `None` so they can still be compared.
+ let t_t_res = val_or_none(t_t_res);
+ let t_n_res = val_or_none(t_n_res);
+ let n_t_res = (!T::Native::is_nan(n_t_res)).then(|| n_t_res);
+ assert_eq!(t_t_res, n_n_res);
+ assert_eq!(t_n_res, n_n_res);
+ assert_eq!(n_t_res, n_n_res);
+ }
+ }
+ }
+
+ macro_rules! test {
+ (
+ @binary
+ $trait:ident,
+ $method:ident $([$checked_method:ident])?,
+ $trait_assign:ident,
+ $method_assign:ident,
+ $($call_for_macros:ident),*
+ ) => {{
+ fn t<T>()
+ where
+ T: ByteOrderType,
+ T: core::ops::$trait<T, Output = T>,
+ T: core::ops::$trait<T::Native, Output = T>,
+ T::Native: core::ops::$trait<T, Output = T>,
+ T::Native: core::ops::$trait<T::Native, Output = T::Native>,
+
+ T: core::ops::$trait_assign<T>,
+ T: core::ops::$trait_assign<T::Native>,
+ T::Native: core::ops::$trait_assign<T>,
+ T::Native: core::ops::$trait_assign<T::Native>,
+ {
+ test::<T, _, _, _, _, _, _, _, _>(
+ core::ops::$trait::$method,
+ core::ops::$trait::$method,
+ core::ops::$trait::$method,
+ core::ops::$trait::$method,
+ {
+ #[allow(unused_mut, unused_assignments)]
+ let mut op_native_checked = None::<fn(T::Native, T::Native) -> Option<T::Native>>;
+ $(
+ op_native_checked = Some(T::Native::$checked_method);
+ )?
+ op_native_checked
+ },
+ Some((
+ <T as core::ops::$trait_assign<T>>::$method_assign,
+ <T as core::ops::$trait_assign::<T::Native>>::$method_assign,
+ <T::Native as core::ops::$trait_assign::<T>>::$method_assign
+ )),
+ );
+ }
+
+ $(
+ $call_for_macros!(t, NativeEndian);
+ $call_for_macros!(t, NonNativeEndian);
+ )*
+ }};
+ (
+ @unary
+ $trait:ident,
+ $method:ident,
+ $($call_for_macros:ident),*
+ ) => {{
+ fn t<T>()
+ where
+ T: ByteOrderType,
+ T: core::ops::$trait<Output = T>,
+ T::Native: core::ops::$trait<Output = T::Native>,
+ {
+ test::<T, _, _, _, _, _, _, _, _>(
+ |slf, _rhs| core::ops::$trait::$method(slf),
+ |slf, _rhs| core::ops::$trait::$method(slf),
+ |slf, _rhs| core::ops::$trait::$method(slf).into(),
+ |slf, _rhs| core::ops::$trait::$method(slf),
+ None::<fn(T::Native, T::Native) -> Option<T::Native>>,
+ None::<(fn(&mut T, T), fn(&mut T, T::Native), fn(&mut T::Native, T))>,
+ );
+ }
+
+ $(
+ $call_for_macros!(t, NativeEndian);
+ $call_for_macros!(t, NonNativeEndian);
+ )*
+ }};
+ }
+
+ test!(@binary Add, add[checked_add], AddAssign, add_assign, call_for_all_types);
+ test!(@binary Div, div[checked_div], DivAssign, div_assign, call_for_all_types);
+ test!(@binary Mul, mul[checked_mul], MulAssign, mul_assign, call_for_all_types);
+ test!(@binary Rem, rem[checked_rem], RemAssign, rem_assign, call_for_all_types);
+ test!(@binary Sub, sub[checked_sub], SubAssign, sub_assign, call_for_all_types);
+
+ test!(@binary BitAnd, bitand, BitAndAssign, bitand_assign, call_for_unsigned_types, call_for_signed_types);
+ test!(@binary BitOr, bitor, BitOrAssign, bitor_assign, call_for_unsigned_types, call_for_signed_types);
+ test!(@binary BitXor, bitxor, BitXorAssign, bitxor_assign, call_for_unsigned_types, call_for_signed_types);
+ test!(@binary Shl, shl[checked_shl], ShlAssign, shl_assign, call_for_unsigned_types, call_for_signed_types);
+ test!(@binary Shr, shr[checked_shr], ShrAssign, shr_assign, call_for_unsigned_types, call_for_signed_types);
+
+ test!(@unary Not, not, call_for_signed_types, call_for_unsigned_types);
+ test!(@unary Neg, neg, call_for_signed_types, call_for_float_types);
+ }
+
+ #[test]
+ fn test_debug_impl() {
+ // Ensure that Debug applies format options to the inner value.
+ let val = U16::<LE>::new(10);
+ assert_eq!(format!("{:?}", val), "U16(10)");
+ assert_eq!(format!("{:03?}", val), "U16(010)");
+ assert_eq!(format!("{:x?}", val), "U16(a)");
+ }
+
+ #[test]
+ fn test_byteorder_traits_coverage() {
+ let val_be = U16::<BigEndian>::from_bytes([0, 1]);
+ let val_le = U16::<LittleEndian>::from_bytes([1, 0]);
+
+ assert_eq!(val_be.get(), 1);
+ assert_eq!(val_le.get(), 1);
+
+ // Debug
+ assert_eq!(format!("{:?}", val_be), "U16(1)");
+ assert_eq!(format!("{:?}", val_le), "U16(1)");
+
+ // PartialOrd, Ord with same type
+ assert!(val_be >= val_be);
+ assert!(val_be <= val_be);
+ assert_eq!(val_be.cmp(&val_be), core::cmp::Ordering::Equal);
+
+ // PartialOrd with native
+ assert!(val_be == 1u16);
+ assert!(val_be >= 1u16);
+
+ // Default
+ let default_be: U16<BigEndian> = Default::default();
+ assert_eq!(default_be.get(), 0);
+
+ // I16
+ let val_be_i16 = I16::<BigEndian>::from_bytes([0, 1]);
+ assert_eq!(val_be_i16.get(), 1);
+ assert_eq!(format!("{:?}", val_be_i16), "I16(1)");
+ assert_eq!(val_be_i16.cmp(&val_be_i16), core::cmp::Ordering::Equal);
+ }
+}
diff --git a/rust/zerocopy/src/deprecated.rs b/rust/zerocopy/src/deprecated.rs
new file mode 100644
index 0000000..24bafbf
--- /dev/null
+++ b/rust/zerocopy/src/deprecated.rs
@@ -0,0 +1,281 @@
+// SPDX-License-Identifier: (BSD-2-Clause OR Apache-2.0) OR MIT
+
+// Copyright 2024 The Fuchsia Authors
+//
+// Licensed under the 2-Clause BSD License <LICENSE-BSD or
+// https://opensource.org/license/bsd-2-clause>, Apache License, Version 2.0
+// <LICENSE-APACHE or https://www.apache.org/licenses/LICENSE-2.0>, or the MIT
+// license <LICENSE-MIT or https://opensource.org/licenses/MIT>, at your option.
+// This file may not be copied, modified, or distributed except according to
+// those terms.
+
+//! Deprecated items. These are kept separate so that they don't clutter up
+//! other modules.
+
+use super::*;
+
+impl<B, T> Ref<B, T>
+where
+ B: ByteSlice,
+ T: KnownLayout + Immutable + ?Sized,
+{
+ #[deprecated(since = "0.8.0", note = "renamed to `Ref::from_bytes`")]
+ #[doc(hidden)]
+ #[must_use = "has no side effects"]
+ #[inline(always)]
+ pub fn new(bytes: B) -> Option<Ref<B, T>> {
+ Self::from_bytes(bytes).ok()
+ }
+}
+
+impl<B, T> Ref<B, T>
+where
+ B: SplitByteSlice,
+ T: KnownLayout + Immutable + ?Sized,
+{
+ #[deprecated(since = "0.8.0", note = "renamed to `Ref::from_prefix`")]
+ #[doc(hidden)]
+ #[must_use = "has no side effects"]
+ #[inline(always)]
+ pub fn new_from_prefix(bytes: B) -> Option<(Ref<B, T>, B)> {
+ Self::from_prefix(bytes).ok()
+ }
+}
+
+impl<B, T> Ref<B, T>
+where
+ B: SplitByteSlice,
+ T: KnownLayout + Immutable + ?Sized,
+{
+ #[deprecated(since = "0.8.0", note = "renamed to `Ref::from_suffix`")]
+ #[doc(hidden)]
+ #[must_use = "has no side effects"]
+ #[inline(always)]
+ pub fn new_from_suffix(bytes: B) -> Option<(B, Ref<B, T>)> {
+ Self::from_suffix(bytes).ok()
+ }
+}
+
+impl<B, T> Ref<B, T>
+where
+ B: ByteSlice,
+ T: Unaligned + KnownLayout + Immutable + ?Sized,
+{
+ #[deprecated(
+ since = "0.8.0",
+ note = "use `Ref::from_bytes`; for `T: Unaligned`, the returned `CastError` implements `Into<SizeError>`"
+ )]
+ #[doc(hidden)]
+ #[must_use = "has no side effects"]
+ #[inline(always)]
+ pub fn new_unaligned(bytes: B) -> Option<Ref<B, T>> {
+ Self::from_bytes(bytes).ok()
+ }
+}
+
+impl<B, T> Ref<B, T>
+where
+ B: SplitByteSlice,
+ T: Unaligned + KnownLayout + Immutable + ?Sized,
+{
+ #[deprecated(
+ since = "0.8.0",
+ note = "use `Ref::from_prefix`; for `T: Unaligned`, the returned `CastError` implements `Into<SizeError>`"
+ )]
+ #[doc(hidden)]
+ #[must_use = "has no side effects"]
+ #[inline(always)]
+ pub fn new_unaligned_from_prefix(bytes: B) -> Option<(Ref<B, T>, B)> {
+ Self::from_prefix(bytes).ok()
+ }
+}
+
+impl<B, T> Ref<B, T>
+where
+ B: SplitByteSlice,
+ T: Unaligned + KnownLayout + Immutable + ?Sized,
+{
+ #[deprecated(
+ since = "0.8.0",
+ note = "use `Ref::from_suffix`; for `T: Unaligned`, the returned `CastError` implements `Into<SizeError>`"
+ )]
+ #[doc(hidden)]
+ #[must_use = "has no side effects"]
+ #[inline(always)]
+ pub fn new_unaligned_from_suffix(bytes: B) -> Option<(B, Ref<B, T>)> {
+ Self::from_suffix(bytes).ok()
+ }
+}
+
+impl<B, T> Ref<B, [T]>
+where
+ B: ByteSlice,
+ T: Immutable,
+{
+ #[deprecated(since = "0.8.0", note = "`Ref::from_bytes` now supports slices")]
+ #[doc(hidden)]
+ #[inline(always)]
+ pub fn new_slice(bytes: B) -> Option<Ref<B, [T]>> {
+ Self::from_bytes(bytes).ok()
+ }
+}
+
+impl<B, T> Ref<B, [T]>
+where
+ B: ByteSlice,
+ T: Unaligned + Immutable,
+{
+ #[deprecated(
+ since = "0.8.0",
+ note = "`Ref::from_bytes` now supports slices; for `T: Unaligned`, the returned `CastError` implements `Into<SizeError>`"
+ )]
+ #[doc(hidden)]
+ #[inline(always)]
+ pub fn new_slice_unaligned(bytes: B) -> Option<Ref<B, [T]>> {
+ Ref::from_bytes(bytes).ok()
+ }
+}
+
+impl<'a, B, T> Ref<B, [T]>
+where
+ B: 'a + IntoByteSlice<'a>,
+ T: FromBytes + Immutable,
+{
+ #[deprecated(since = "0.8.0", note = "`Ref::into_ref` now supports slices")]
+ #[doc(hidden)]
+ #[inline(always)]
+ pub fn into_slice(self) -> &'a [T] {
+ Ref::into_ref(self)
+ }
+}
+
+impl<'a, B, T> Ref<B, [T]>
+where
+ B: 'a + IntoByteSliceMut<'a>,
+ T: FromBytes + IntoBytes + Immutable,
+{
+ #[deprecated(since = "0.8.0", note = "`Ref::into_mut` now supports slices")]
+ #[doc(hidden)]
+ #[inline(always)]
+ pub fn into_mut_slice(self) -> &'a mut [T] {
+ Ref::into_mut(self)
+ }
+}
+
+impl<B, T> Ref<B, [T]>
+where
+ B: SplitByteSlice,
+ T: Immutable,
+{
+ #[deprecated(since = "0.8.0", note = "replaced by `Ref::from_prefix_with_elems`")]
+ #[must_use = "has no side effects"]
+ #[doc(hidden)]
+ #[inline(always)]
+ pub fn new_slice_from_prefix(bytes: B, count: usize) -> Option<(Ref<B, [T]>, B)> {
+ Ref::from_prefix_with_elems(bytes, count).ok()
+ }
+
+ #[deprecated(since = "0.8.0", note = "replaced by `Ref::from_suffix_with_elems`")]
+ #[must_use = "has no side effects"]
+ #[doc(hidden)]
+ #[inline(always)]
+ pub fn new_slice_from_suffix(bytes: B, count: usize) -> Option<(B, Ref<B, [T]>)> {
+ Ref::from_suffix_with_elems(bytes, count).ok()
+ }
+}
+
+impl<B, T> Ref<B, [T]>
+where
+ B: SplitByteSlice,
+ T: Unaligned + Immutable,
+{
+ #[deprecated(
+ since = "0.8.0",
+ note = "use `Ref::from_prefix_with_elems`; for `T: Unaligned`, the returned `CastError` implements `Into<SizeError>`"
+ )]
+ #[doc(hidden)]
+ #[must_use = "has no side effects"]
+ #[inline(always)]
+ pub fn new_slice_unaligned_from_prefix(bytes: B, count: usize) -> Option<(Ref<B, [T]>, B)> {
+ Ref::from_prefix_with_elems(bytes, count).ok()
+ }
+
+ #[deprecated(
+ since = "0.8.0",
+ note = "use `Ref::from_suffix_with_elems`; for `T: Unaligned`, the returned `CastError` implements `Into<SizeError>`"
+ )]
+ #[doc(hidden)]
+ #[must_use = "has no side effects"]
+ #[inline(always)]
+ pub fn new_slice_unaligned_from_suffix(bytes: B, count: usize) -> Option<(B, Ref<B, [T]>)> {
+ Ref::from_suffix_with_elems(bytes, count).ok()
+ }
+}
+
+#[cfg(test)]
+mod tests {
+ use super::*;
+
+ #[test]
+ #[allow(deprecated)]
+ fn test_deprecated_ref_methods() {
+ let bytes = &[0u8; 1][..];
+ let bytes_slice = &[0u8; 4][..];
+
+ let r: Option<Ref<&[u8], u8>> = Ref::new(bytes);
+ assert!(r.is_some());
+
+ let r: Option<(Ref<&[u8], u8>, &[u8])> = Ref::new_from_prefix(bytes);
+ assert!(r.is_some());
+
+ let r: Option<(&[u8], Ref<&[u8], u8>)> = Ref::new_from_suffix(bytes);
+ assert!(r.is_some());
+
+ let r: Option<Ref<&[u8], u8>> = Ref::new_unaligned(bytes);
+ assert!(r.is_some());
+
+ let r: Option<(Ref<&[u8], u8>, &[u8])> = Ref::new_unaligned_from_prefix(bytes);
+ assert!(r.is_some());
+
+ let r: Option<(&[u8], Ref<&[u8], u8>)> = Ref::new_unaligned_from_suffix(bytes);
+ assert!(r.is_some());
+
+ let r: Option<Ref<&[u8], [u8]>> = Ref::new_slice(bytes_slice);
+ assert!(r.is_some());
+
+ let r: Option<Ref<&[u8], [u8]>> = Ref::new_slice_unaligned(bytes_slice);
+ assert!(r.is_some());
+
+ let r: Option<(Ref<&[u8], [u8]>, &[u8])> = Ref::new_slice_from_prefix(bytes_slice, 1);
+ assert!(r.is_some());
+
+ let r: Option<(&[u8], Ref<&[u8], [u8]>)> = Ref::new_slice_from_suffix(bytes_slice, 1);
+ assert!(r.is_some());
+
+ let r: Option<(Ref<&[u8], [u8]>, &[u8])> =
+ Ref::new_slice_unaligned_from_prefix(bytes_slice, 1);
+ assert!(r.is_some());
+
+ let r: Option<(&[u8], Ref<&[u8], [u8]>)> =
+ Ref::new_slice_unaligned_from_suffix(bytes_slice, 1);
+ assert!(r.is_some());
+ }
+
+ #[test]
+ #[allow(deprecated)]
+ fn test_deprecated_into_slice() {
+ let bytes = &[0u8; 4][..];
+ let r: Ref<&[u8], [u8]> = Ref::from_bytes(bytes).unwrap();
+ let slice: &[u8] = r.into_slice();
+ assert_eq!(slice.len(), 4);
+ }
+
+ #[test]
+ #[allow(deprecated)]
+ fn test_deprecated_into_mut_slice() {
+ let mut bytes = [0u8; 4];
+ let r: Ref<&mut [u8], [u8]> = Ref::from_bytes(&mut bytes[..]).unwrap();
+ let slice: &mut [u8] = r.into_mut_slice();
+ assert_eq!(slice.len(), 4);
+ }
+}
diff --git a/rust/zerocopy/src/error.rs b/rust/zerocopy/src/error.rs
new file mode 100644
index 0000000..7cb08c3
--- /dev/null
+++ b/rust/zerocopy/src/error.rs
@@ -0,0 +1,1350 @@
+// SPDX-License-Identifier: (BSD-2-Clause OR Apache-2.0) OR MIT
+
+// Copyright 2024 The Fuchsia Authors
+//
+// Licensed under the 2-Clause BSD License <LICENSE-BSD or
+// https://opensource.org/license/bsd-2-clause>, Apache License, Version 2.0
+// <LICENSE-APACHE or https://www.apache.org/licenses/LICENSE-2.0>, or the MIT
+// license <LICENSE-MIT or https://opensource.org/licenses/MIT>, at your option.
+// This file may not be copied, modified, or distributed except according to
+// those terms.
+
+//! Types related to error reporting.
+//!
+//! ## Single failure mode errors
+//!
+//! Generally speaking, zerocopy's conversions may fail for one of up to three
+//! reasons:
+//! - [`AlignmentError`]: the conversion source was improperly aligned
+//! - [`SizeError`]: the conversion source was of incorrect size
+//! - [`ValidityError`]: the conversion source contained invalid data
+//!
+//! Methods that only have one failure mode, like
+//! [`FromBytes::read_from_bytes`], return that mode's corresponding error type
+//! directly.
+//!
+//! ## Compound errors
+//!
+//! Conversion methods that have either two or three possible failure modes
+//! return one of these error types:
+//! - [`CastError`]: the error type of reference conversions
+//! - [`TryCastError`]: the error type of fallible reference conversions
+//! - [`TryReadError`]: the error type of fallible read conversions
+//!
+//! ## [`Unaligned`] destination types
+//!
+//! For [`Unaligned`] destination types, alignment errors are impossible. All
+//! compound error types support infallibly discarding the alignment error via
+//! [`From`] so long as `Dst: Unaligned`. For example, see [`<SizeError as
+//! From<ConvertError>>::from`][size-error-from].
+//!
+//! [size-error-from]: struct.SizeError.html#method.from-1
+//!
+//! ## Accessing the conversion source
+//!
+//! All error types provide an `into_src` method that converts the error into
+//! the source value underlying the failed conversion.
+//!
+//! ## Display formatting
+//!
+//! All error types provide a `Display` implementation that produces a
+//! human-readable error message. When `debug_assertions` are enabled, these
+//! error messages are verbose and may include potentially sensitive
+//! information, including:
+//!
+//! - the names of the involved types
+//! - the sizes of the involved types
+//! - the addresses of the involved types
+//! - the contents of the involved types
+//!
+//! When `debug_assertions` are disabled (as is default for `release` builds),
+//! such potentially sensitive information is excluded.
+//!
+//! In the future, we may support manually configuring this behavior. If you are
+//! interested in this feature, [let us know on GitHub][issue-1457] so we know
+//! to prioritize it.
+//!
+//! [issue-1457]: https://github.com/google/zerocopy/issues/1457
+//!
+//! ## Validation order
+//!
+//! Our conversion methods typically check alignment, then size, then bit
+//! validity. However, we do not guarantee that this is always the case, and
+//! this behavior may change between releases.
+//!
+//! ## `Send`, `Sync`, and `'static`
+//!
+//! Our error types are `Send`, `Sync`, and `'static` when their `Src` parameter
+//! is `Send`, `Sync`, or `'static`, respectively. This can cause issues when an
+//! error is sent or synchronized across threads; e.g.:
+//!
+//! ```compile_fail,E0515
+//! use zerocopy::*;
+//!
+//! let result: SizeError<&[u8], u32> = std::thread::spawn(|| {
+//! let source = &mut [0u8, 1, 2][..];
+//! // Try (and fail) to read a `u32` from `source`.
+//! u32::read_from_bytes(source).unwrap_err()
+//! }).join().unwrap();
+//! ```
+//!
+//! To work around this, use [`map_src`][CastError::map_src] to convert the
+//! source parameter to an unproblematic type; e.g.:
+//!
+//! ```
+//! use zerocopy::*;
+//!
+//! let result: SizeError<(), u32> = std::thread::spawn(|| {
+//! let source = &mut [0u8, 1, 2][..];
+//! // Try (and fail) to read a `u32` from `source`.
+//! u32::read_from_bytes(source).unwrap_err()
+//! // Erase the error source.
+//! .map_src(drop)
+//! }).join().unwrap();
+//! ```
+//!
+//! Alternatively, use `.to_string()` to eagerly convert the error into a
+//! human-readable message; e.g.:
+//!
+//! ```
+//! use zerocopy::*;
+//!
+//! let result: Result<u32, String> = std::thread::spawn(|| {
+//! let source = &mut [0u8, 1, 2][..];
+//! // Try (and fail) to read a `u32` from `source`.
+//! u32::read_from_bytes(source)
+//! // Eagerly render the error message.
+//! .map_err(|err| err.to_string())
+//! }).join().unwrap();
+//! ```
+#[cfg(not(no_zerocopy_core_error_1_81_0))]
+use core::error::Error;
+use core::{
+ convert::Infallible,
+ fmt::{self, Debug, Write},
+ ops::Deref,
+};
+#[cfg(all(no_zerocopy_core_error_1_81_0, any(feature = "std", test)))]
+use std::error::Error;
+
+use crate::{util::SendSyncPhantomData, KnownLayout, TryFromBytes, Unaligned};
+#[cfg(doc)]
+use crate::{FromBytes, Ref};
+
+/// Zerocopy's generic error type.
+///
+/// Generally speaking, zerocopy's conversions may fail for one of up to three
+/// reasons:
+/// - [`AlignmentError`]: the conversion source was improperly aligned
+/// - [`SizeError`]: the conversion source was of incorrect size
+/// - [`ValidityError`]: the conversion source contained invalid data
+///
+/// However, not all conversions produce all errors. For instance,
+/// [`FromBytes::ref_from_bytes`] may fail due to alignment or size issues, but
+/// not validity issues. This generic error type captures these
+/// (im)possibilities via parameterization: `A` is parameterized with
+/// [`AlignmentError`], `S` is parameterized with [`SizeError`], and `V` is
+/// parameterized with [`Infallible`].
+///
+/// Zerocopy never uses this type directly in its API. Rather, we provide three
+/// pre-parameterized aliases:
+/// - [`CastError`]: the error type of reference conversions
+/// - [`TryCastError`]: the error type of fallible reference conversions
+/// - [`TryReadError`]: the error type of fallible read conversions
+#[derive(PartialEq, Eq, Clone)]
+pub enum ConvertError<A, S, V> {
+ /// The conversion source was improperly aligned.
+ Alignment(A),
+ /// The conversion source was of incorrect size.
+ Size(S),
+ /// The conversion source contained invalid data.
+ Validity(V),
+}
+
+impl<Src, Dst: ?Sized + Unaligned, S, V> From<ConvertError<AlignmentError<Src, Dst>, S, V>>
+ for ConvertError<Infallible, S, V>
+{
+ /// Infallibly discards the alignment error from this `ConvertError` since
+ /// `Dst` is unaligned.
+ ///
+ /// Since [`Dst: Unaligned`], it is impossible to encounter an alignment
+ /// error. This method permits discarding that alignment error infallibly
+ /// and replacing it with [`Infallible`].
+ ///
+ /// [`Dst: Unaligned`]: crate::Unaligned
+ ///
+ /// # Examples
+ ///
+ /// ```
+ /// use core::convert::Infallible;
+ /// use zerocopy::*;
+ /// # use zerocopy_derive::*;
+ ///
+ /// #[derive(TryFromBytes, KnownLayout, Unaligned, Immutable)]
+ /// #[repr(C, packed)]
+ /// struct Bools {
+ /// one: bool,
+ /// two: bool,
+ /// many: [bool],
+ /// }
+ ///
+ /// impl Bools {
+ /// fn parse(bytes: &[u8]) -> Result<&Bools, AlignedTryCastError<&[u8], Bools>> {
+ /// // Since `Bools: Unaligned`, we can infallibly discard
+ /// // the alignment error.
+ /// Bools::try_ref_from_bytes(bytes).map_err(Into::into)
+ /// }
+ /// }
+ /// ```
+ #[inline]
+ fn from(err: ConvertError<AlignmentError<Src, Dst>, S, V>) -> ConvertError<Infallible, S, V> {
+ match err {
+ ConvertError::Alignment(e) => {
+ #[allow(unreachable_code)]
+ return ConvertError::Alignment(Infallible::from(e));
+ }
+ ConvertError::Size(e) => ConvertError::Size(e),
+ ConvertError::Validity(e) => ConvertError::Validity(e),
+ }
+ }
+}
+
+impl<A: fmt::Debug, S: fmt::Debug, V: fmt::Debug> fmt::Debug for ConvertError<A, S, V> {
+ #[inline]
+ fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
+ match self {
+ Self::Alignment(e) => f.debug_tuple("Alignment").field(e).finish(),
+ Self::Size(e) => f.debug_tuple("Size").field(e).finish(),
+ Self::Validity(e) => f.debug_tuple("Validity").field(e).finish(),
+ }
+ }
+}
+
+/// Produces a human-readable error message.
+///
+/// The message differs between debug and release builds. When
+/// `debug_assertions` are enabled, this message is verbose and includes
+/// potentially sensitive information.
+impl<A: fmt::Display, S: fmt::Display, V: fmt::Display> fmt::Display for ConvertError<A, S, V> {
+ #[inline]
+ fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
+ match self {
+ Self::Alignment(e) => e.fmt(f),
+ Self::Size(e) => e.fmt(f),
+ Self::Validity(e) => e.fmt(f),
+ }
+ }
+}
+
+#[cfg(any(not(no_zerocopy_core_error_1_81_0), feature = "std", test))]
+#[cfg_attr(doc_cfg, doc(cfg(all(rust = "1.81.0", feature = "std"))))]
+impl<A, S, V> Error for ConvertError<A, S, V>
+where
+ A: fmt::Display + fmt::Debug,
+ S: fmt::Display + fmt::Debug,
+ V: fmt::Display + fmt::Debug,
+{
+}
+
+/// The error emitted if the conversion source is improperly aligned.
+pub struct AlignmentError<Src, Dst: ?Sized> {
+ /// The source value involved in the conversion.
+ src: Src,
+ /// The inner destination type involved in the conversion.
+ ///
+ /// INVARIANT: An `AlignmentError` may only be constructed if `Dst`'s
+ /// alignment requirement is greater than one.
+ _dst: SendSyncPhantomData<Dst>,
+}
+
+impl<Src, Dst: ?Sized> AlignmentError<Src, Dst> {
+ /// # Safety
+ ///
+ /// The caller must ensure that `Dst`'s alignment requirement is greater
+ /// than one.
+ pub(crate) unsafe fn new_unchecked(src: Src) -> Self {
+ // INVARIANT: The caller guarantees that `Dst`'s alignment requirement
+ // is greater than one.
+ Self { src, _dst: SendSyncPhantomData::default() }
+ }
+
+ /// Produces the source underlying the failed conversion.
+ #[inline]
+ pub fn into_src(self) -> Src {
+ self.src
+ }
+
+ pub(crate) fn with_src<NewSrc>(self, new_src: NewSrc) -> AlignmentError<NewSrc, Dst> {
+ // INVARIANT: `with_src` doesn't change the type of `Dst`, so the
+ // invariant that `Dst`'s alignment requirement is greater than one is
+ // preserved.
+ AlignmentError { src: new_src, _dst: SendSyncPhantomData::default() }
+ }
+
+ /// Maps the source value associated with the conversion error.
+ ///
+ /// This can help mitigate [issues with `Send`, `Sync` and `'static`
+ /// bounds][self#send-sync-and-static].
+ ///
+ /// # Examples
+ ///
+ /// ```
+ /// use zerocopy::*;
+ ///
+ /// let unaligned = Unalign::new(0u16);
+ ///
+ /// // Attempt to deref `unaligned`. This might fail with an alignment error.
+ /// let maybe_n: Result<&u16, AlignmentError<&Unalign<u16>, u16>> = unaligned.try_deref();
+ ///
+ /// // Map the error's source to its address as a usize.
+ /// let maybe_n: Result<&u16, AlignmentError<usize, u16>> = maybe_n.map_err(|err| {
+ /// err.map_src(|src| src as *const _ as usize)
+ /// });
+ /// ```
+ #[inline]
+ pub fn map_src<NewSrc>(self, f: impl FnOnce(Src) -> NewSrc) -> AlignmentError<NewSrc, Dst> {
+ AlignmentError { src: f(self.src), _dst: SendSyncPhantomData::default() }
+ }
+
+ pub(crate) fn into<S, V>(self) -> ConvertError<Self, S, V> {
+ ConvertError::Alignment(self)
+ }
+
+ /// Format extra details for a verbose, human-readable error message.
+ ///
+ /// This formatting may include potentially sensitive information.
+ fn display_verbose_extras(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result
+ where
+ Src: Deref,
+ Dst: KnownLayout,
+ {
+ #[allow(clippy::as_conversions)]
+ let addr = self.src.deref() as *const _ as *const ();
+ let addr_align = 2usize.pow((crate::util::AsAddress::addr(addr)).trailing_zeros());
+
+ f.write_str("\n\nSource type: ")?;
+ f.write_str(core::any::type_name::<Src>())?;
+
+ f.write_str("\nSource address: ")?;
+ addr.fmt(f)?;
+ f.write_str(" (a multiple of ")?;
+ addr_align.fmt(f)?;
+ f.write_str(")")?;
+
+ f.write_str("\nDestination type: ")?;
+ f.write_str(core::any::type_name::<Dst>())?;
+
+ f.write_str("\nDestination alignment: ")?;
+ <Dst as KnownLayout>::LAYOUT.align.get().fmt(f)?;
+
+ Ok(())
+ }
+}
+
+impl<Src: Clone, Dst: ?Sized> Clone for AlignmentError<Src, Dst> {
+ #[inline]
+ fn clone(&self) -> Self {
+ Self { src: self.src.clone(), _dst: SendSyncPhantomData::default() }
+ }
+}
+
+impl<Src: PartialEq, Dst: ?Sized> PartialEq for AlignmentError<Src, Dst> {
+ #[inline]
+ fn eq(&self, other: &Self) -> bool {
+ self.src == other.src
+ }
+}
+
+impl<Src: Eq, Dst: ?Sized> Eq for AlignmentError<Src, Dst> {}
+
+impl<Src, Dst: ?Sized + Unaligned> From<AlignmentError<Src, Dst>> for Infallible {
+ #[inline(always)]
+ fn from(_: AlignmentError<Src, Dst>) -> Infallible {
+ // SAFETY: `AlignmentError`s can only be constructed when `Dst`'s
+ // alignment requirement is greater than one. In this block, `Dst:
+ // Unaligned`, which means that its alignment requirement is equal to
+ // one. Thus, it's not possible to reach here at runtime.
+ unsafe { core::hint::unreachable_unchecked() }
+ }
+}
+
+#[cfg(test)]
+impl<Src, Dst> AlignmentError<Src, Dst> {
+ // A convenience constructor so that test code doesn't need to write
+ // `unsafe`.
+ fn new_checked(src: Src) -> AlignmentError<Src, Dst> {
+ assert_ne!(core::mem::align_of::<Dst>(), 1);
+ // SAFETY: The preceding assertion guarantees that `Dst`'s alignment
+ // requirement is greater than one.
+ unsafe { AlignmentError::new_unchecked(src) }
+ }
+}
+
+impl<Src, Dst: ?Sized> fmt::Debug for AlignmentError<Src, Dst> {
+ #[inline]
+ fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
+ f.debug_struct("AlignmentError").finish()
+ }
+}
+
+/// Produces a human-readable error message.
+///
+/// The message differs between debug and release builds. When
+/// `debug_assertions` are enabled, this message is verbose and includes
+/// potentially sensitive information.
+impl<Src, Dst: ?Sized> fmt::Display for AlignmentError<Src, Dst>
+where
+ Src: Deref,
+ Dst: KnownLayout,
+{
+ #[inline]
+ fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
+ f.write_str("The conversion failed because the address of the source is not a multiple of the alignment of the destination type.")?;
+
+ if cfg!(debug_assertions) {
+ self.display_verbose_extras(f)
+ } else {
+ Ok(())
+ }
+ }
+}
+
+#[cfg(any(not(no_zerocopy_core_error_1_81_0), feature = "std", test))]
+#[cfg_attr(doc_cfg, doc(cfg(all(rust = "1.81.0", feature = "std"))))]
+impl<Src, Dst: ?Sized> Error for AlignmentError<Src, Dst>
+where
+ Src: Deref,
+ Dst: KnownLayout,
+{
+}
+
+impl<Src, Dst: ?Sized, S, V> From<AlignmentError<Src, Dst>>
+ for ConvertError<AlignmentError<Src, Dst>, S, V>
+{
+ #[inline(always)]
+ fn from(err: AlignmentError<Src, Dst>) -> Self {
+ Self::Alignment(err)
+ }
+}
+
+/// The error emitted if the conversion source is of incorrect size.
+pub struct SizeError<Src, Dst: ?Sized> {
+ /// The source value involved in the conversion.
+ src: Src,
+ /// The inner destination type involved in the conversion.
+ _dst: SendSyncPhantomData<Dst>,
+}
+
+impl<Src, Dst: ?Sized> SizeError<Src, Dst> {
+ pub(crate) fn new(src: Src) -> Self {
+ Self { src, _dst: SendSyncPhantomData::default() }
+ }
+
+ /// Produces the source underlying the failed conversion.
+ #[inline]
+ pub fn into_src(self) -> Src {
+ self.src
+ }
+
+ /// Sets the source value associated with the conversion error.
+ pub(crate) fn with_src<NewSrc>(self, new_src: NewSrc) -> SizeError<NewSrc, Dst> {
+ SizeError { src: new_src, _dst: SendSyncPhantomData::default() }
+ }
+
+ /// Maps the source value associated with the conversion error.
+ ///
+ /// This can help mitigate [issues with `Send`, `Sync` and `'static`
+ /// bounds][self#send-sync-and-static].
+ ///
+ /// # Examples
+ ///
+ /// ```
+ /// use zerocopy::*;
+ ///
+ /// let source: [u8; 3] = [0, 1, 2];
+ ///
+ /// // Try to read a `u32` from `source`. This will fail because there are insufficient
+ /// // bytes in `source`.
+ /// let maybe_u32: Result<u32, SizeError<&[u8], u32>> = u32::read_from_bytes(&source[..]);
+ ///
+ /// // Map the error's source to its size.
+ /// let maybe_u32: Result<u32, SizeError<usize, u32>> = maybe_u32.map_err(|err| {
+ /// err.map_src(|src| src.len())
+ /// });
+ /// ```
+ #[inline]
+ pub fn map_src<NewSrc>(self, f: impl FnOnce(Src) -> NewSrc) -> SizeError<NewSrc, Dst> {
+ SizeError { src: f(self.src), _dst: SendSyncPhantomData::default() }
+ }
+
+ /// Sets the destination type associated with the conversion error.
+ pub(crate) fn with_dst<NewDst: ?Sized>(self) -> SizeError<Src, NewDst> {
+ SizeError { src: self.src, _dst: SendSyncPhantomData::default() }
+ }
+
+ /// Converts the error into a general [`ConvertError`].
+ pub(crate) fn into<A, V>(self) -> ConvertError<A, Self, V> {
+ ConvertError::Size(self)
+ }
+
+ /// Format extra details for a verbose, human-readable error message.
+ ///
+ /// This formatting may include potentially sensitive information.
+ fn display_verbose_extras(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result
+ where
+ Src: Deref,
+ Dst: KnownLayout,
+ {
+ // include the source type
+ f.write_str("\nSource type: ")?;
+ f.write_str(core::any::type_name::<Src>())?;
+
+ // include the source.deref() size
+ let src_size = core::mem::size_of_val(&*self.src);
+ f.write_str("\nSource size: ")?;
+ src_size.fmt(f)?;
+ f.write_str(" byte")?;
+ if src_size != 1 {
+ f.write_char('s')?;
+ }
+
+ // if `Dst` is `Sized`, include the `Dst` size
+ if let crate::SizeInfo::Sized { size } = Dst::LAYOUT.size_info {
+ f.write_str("\nDestination size: ")?;
+ size.fmt(f)?;
+ f.write_str(" byte")?;
+ if size != 1 {
+ f.write_char('s')?;
+ }
+ }
+
+ // include the destination type
+ f.write_str("\nDestination type: ")?;
+ f.write_str(core::any::type_name::<Dst>())?;
+
+ Ok(())
+ }
+}
+
+impl<Src: Clone, Dst: ?Sized> Clone for SizeError<Src, Dst> {
+ #[inline]
+ fn clone(&self) -> Self {
+ Self { src: self.src.clone(), _dst: SendSyncPhantomData::default() }
+ }
+}
+
+impl<Src: PartialEq, Dst: ?Sized> PartialEq for SizeError<Src, Dst> {
+ #[inline]
+ fn eq(&self, other: &Self) -> bool {
+ self.src == other.src
+ }
+}
+
+impl<Src: Eq, Dst: ?Sized> Eq for SizeError<Src, Dst> {}
+
+impl<Src, Dst: ?Sized> fmt::Debug for SizeError<Src, Dst> {
+ #[inline]
+ fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
+ f.debug_struct("SizeError").finish()
+ }
+}
+
+/// Produces a human-readable error message.
+///
+/// The message differs between debug and release builds. When
+/// `debug_assertions` are enabled, this message is verbose and includes
+/// potentially sensitive information.
+impl<Src, Dst: ?Sized> fmt::Display for SizeError<Src, Dst>
+where
+ Src: Deref,
+ Dst: KnownLayout,
+{
+ #[inline]
+ fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
+ f.write_str("The conversion failed because the source was incorrectly sized to complete the conversion into the destination type.")?;
+ if cfg!(debug_assertions) {
+ f.write_str("\n")?;
+ self.display_verbose_extras(f)?;
+ }
+ Ok(())
+ }
+}
+
+#[cfg(any(not(no_zerocopy_core_error_1_81_0), feature = "std", test))]
+#[cfg_attr(doc_cfg, doc(cfg(all(rust = "1.81.0", feature = "std"))))]
+impl<Src, Dst: ?Sized> Error for SizeError<Src, Dst>
+where
+ Src: Deref,
+ Dst: KnownLayout,
+{
+}
+
+impl<Src, Dst: ?Sized, A, V> From<SizeError<Src, Dst>> for ConvertError<A, SizeError<Src, Dst>, V> {
+ #[inline(always)]
+ fn from(err: SizeError<Src, Dst>) -> Self {
+ Self::Size(err)
+ }
+}
+
+/// The error emitted if the conversion source contains invalid data.
+pub struct ValidityError<Src, Dst: ?Sized + TryFromBytes> {
+ /// The source value involved in the conversion.
+ pub(crate) src: Src,
+ /// The inner destination type involved in the conversion.
+ _dst: SendSyncPhantomData<Dst>,
+}
+
+impl<Src, Dst: ?Sized + TryFromBytes> ValidityError<Src, Dst> {
+ pub(crate) fn new(src: Src) -> Self {
+ Self { src, _dst: SendSyncPhantomData::default() }
+ }
+
+ /// Produces the source underlying the failed conversion.
+ #[inline]
+ pub fn into_src(self) -> Src {
+ self.src
+ }
+
+ /// Maps the source value associated with the conversion error.
+ ///
+ /// This can help mitigate [issues with `Send`, `Sync` and `'static`
+ /// bounds][self#send-sync-and-static].
+ ///
+ /// # Examples
+ ///
+ /// ```
+ /// use zerocopy::*;
+ ///
+ /// let source: u8 = 42;
+ ///
+ /// // Try to transmute the `source` to a `bool`. This will fail.
+ /// let maybe_bool: Result<bool, ValidityError<u8, bool>> = try_transmute!(source);
+ ///
+ /// // Drop the error's source.
+ /// let maybe_bool: Result<bool, ValidityError<(), bool>> = maybe_bool.map_err(|err| {
+ /// err.map_src(drop)
+ /// });
+ /// ```
+ #[inline]
+ pub fn map_src<NewSrc>(self, f: impl FnOnce(Src) -> NewSrc) -> ValidityError<NewSrc, Dst> {
+ ValidityError { src: f(self.src), _dst: SendSyncPhantomData::default() }
+ }
+
+ /// Converts the error into a general [`ConvertError`].
+ pub(crate) fn into<A, S>(self) -> ConvertError<A, S, Self> {
+ ConvertError::Validity(self)
+ }
+
+ /// Format extra details for a verbose, human-readable error message.
+ ///
+ /// This formatting may include potentially sensitive information.
+ fn display_verbose_extras(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result
+ where
+ Dst: KnownLayout,
+ {
+ f.write_str("Destination type: ")?;
+ f.write_str(core::any::type_name::<Dst>())?;
+ Ok(())
+ }
+}
+
+impl<Src: Clone, Dst: ?Sized + TryFromBytes> Clone for ValidityError<Src, Dst> {
+ #[inline]
+ fn clone(&self) -> Self {
+ Self { src: self.src.clone(), _dst: SendSyncPhantomData::default() }
+ }
+}
+
+// SAFETY: `ValidityError` contains a single `Self::Inner = Src`, and no other
+// non-ZST fields. `map` passes ownership of `self`'s sole `Self::Inner` to `f`.
+unsafe impl<Src, NewSrc, Dst> crate::pointer::TryWithError<NewSrc>
+ for crate::ValidityError<Src, Dst>
+where
+ Dst: TryFromBytes + ?Sized,
+{
+ type Inner = Src;
+ type Mapped = crate::ValidityError<NewSrc, Dst>;
+ #[inline]
+ fn map<F: FnOnce(Src) -> NewSrc>(self, f: F) -> Self::Mapped {
+ self.map_src(f)
+ }
+}
+
+impl<Src: PartialEq, Dst: ?Sized + TryFromBytes> PartialEq for ValidityError<Src, Dst> {
+ #[inline]
+ fn eq(&self, other: &Self) -> bool {
+ self.src == other.src
+ }
+}
+
+impl<Src: Eq, Dst: ?Sized + TryFromBytes> Eq for ValidityError<Src, Dst> {}
+
+impl<Src, Dst: ?Sized + TryFromBytes> fmt::Debug for ValidityError<Src, Dst> {
+ #[inline]
+ fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
+ f.debug_struct("ValidityError").finish()
+ }
+}
+
+/// Produces a human-readable error message.
+///
+/// The message differs between debug and release builds. When
+/// `debug_assertions` are enabled, this message is verbose and includes
+/// potentially sensitive information.
+impl<Src, Dst: ?Sized> fmt::Display for ValidityError<Src, Dst>
+where
+ Dst: KnownLayout + TryFromBytes,
+{
+ #[inline]
+ fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
+ f.write_str("The conversion failed because the source bytes are not a valid value of the destination type.")?;
+ if cfg!(debug_assertions) {
+ f.write_str("\n\n")?;
+ self.display_verbose_extras(f)?;
+ }
+ Ok(())
+ }
+}
+
+#[cfg(any(not(no_zerocopy_core_error_1_81_0), feature = "std", test))]
+#[cfg_attr(doc_cfg, doc(cfg(all(rust = "1.81.0", feature = "std"))))]
+impl<Src, Dst: ?Sized> Error for ValidityError<Src, Dst> where Dst: KnownLayout + TryFromBytes {}
+
+impl<Src, Dst: ?Sized + TryFromBytes, A, S> From<ValidityError<Src, Dst>>
+ for ConvertError<A, S, ValidityError<Src, Dst>>
+{
+ #[inline(always)]
+ fn from(err: ValidityError<Src, Dst>) -> Self {
+ Self::Validity(err)
+ }
+}
+
+/// The error type of reference conversions.
+///
+/// Reference conversions, like [`FromBytes::ref_from_bytes`] may emit
+/// [alignment](AlignmentError) and [size](SizeError) errors.
+// Bounds on generic parameters are not enforced in type aliases, but they do
+// appear in rustdoc.
+#[allow(type_alias_bounds)]
+pub type CastError<Src, Dst: ?Sized> =
+ ConvertError<AlignmentError<Src, Dst>, SizeError<Src, Dst>, Infallible>;
+
+impl<Src, Dst: ?Sized> CastError<Src, Dst> {
+ /// Produces the source underlying the failed conversion.
+ #[inline]
+ pub fn into_src(self) -> Src {
+ match self {
+ Self::Alignment(e) => e.src,
+ Self::Size(e) => e.src,
+ Self::Validity(i) => match i {},
+ }
+ }
+
+ /// Sets the source value associated with the conversion error.
+ pub(crate) fn with_src<NewSrc>(self, new_src: NewSrc) -> CastError<NewSrc, Dst> {
+ match self {
+ Self::Alignment(e) => CastError::Alignment(e.with_src(new_src)),
+ Self::Size(e) => CastError::Size(e.with_src(new_src)),
+ Self::Validity(i) => match i {},
+ }
+ }
+
+ /// Maps the source value associated with the conversion error.
+ ///
+ /// This can help mitigate [issues with `Send`, `Sync` and `'static`
+ /// bounds][self#send-sync-and-static].
+ ///
+ /// # Examples
+ ///
+ /// ```
+ /// use zerocopy::*;
+ ///
+ /// let source: [u8; 3] = [0, 1, 2];
+ ///
+ /// // Try to read a `u32` from `source`. This will fail because there are insufficient
+ /// // bytes in `source`.
+ /// let maybe_u32: Result<&u32, CastError<&[u8], u32>> = u32::ref_from_bytes(&source[..]);
+ ///
+ /// // Map the error's source to its size and address.
+ /// let maybe_u32: Result<&u32, CastError<(usize, usize), u32>> = maybe_u32.map_err(|err| {
+ /// err.map_src(|src| (src.len(), src.as_ptr() as usize))
+ /// });
+ /// ```
+ #[inline]
+ pub fn map_src<NewSrc>(self, f: impl FnOnce(Src) -> NewSrc) -> CastError<NewSrc, Dst> {
+ match self {
+ Self::Alignment(e) => CastError::Alignment(e.map_src(f)),
+ Self::Size(e) => CastError::Size(e.map_src(f)),
+ Self::Validity(i) => match i {},
+ }
+ }
+
+ /// Converts the error into a general [`ConvertError`].
+ pub(crate) fn into(self) -> TryCastError<Src, Dst>
+ where
+ Dst: TryFromBytes,
+ {
+ match self {
+ Self::Alignment(e) => TryCastError::Alignment(e),
+ Self::Size(e) => TryCastError::Size(e),
+ Self::Validity(i) => match i {},
+ }
+ }
+}
+
+// SAFETY: `CastError` is either a single `AlignmentError` or a single
+// `SizeError`. In either case, it contains a single `Self::Inner = Src`, and no
+// other non-ZST fields. `map` passes ownership of `self`'s sole `Self::Inner`
+// to `f`.
+unsafe impl<Src, NewSrc, Dst> crate::pointer::TryWithError<NewSrc> for crate::CastError<Src, Dst>
+where
+ Dst: ?Sized,
+{
+ type Inner = Src;
+ type Mapped = crate::CastError<NewSrc, Dst>;
+
+ #[inline]
+ fn map<F: FnOnce(Src) -> NewSrc>(self, f: F) -> Self::Mapped {
+ self.map_src(f)
+ }
+}
+
+impl<Src, Dst: ?Sized + Unaligned> From<CastError<Src, Dst>> for SizeError<Src, Dst> {
+ /// Infallibly extracts the [`SizeError`] from this `CastError` since `Dst`
+ /// is unaligned.
+ ///
+ /// Since [`Dst: Unaligned`], it is impossible to encounter an alignment
+ /// error, and so the only error that can be encountered at runtime is a
+ /// [`SizeError`]. This method permits extracting that `SizeError`
+ /// infallibly.
+ ///
+ /// [`Dst: Unaligned`]: crate::Unaligned
+ ///
+ /// # Examples
+ ///
+ /// ```rust
+ /// use zerocopy::*;
+ /// # use zerocopy_derive::*;
+ ///
+ /// #[derive(FromBytes, IntoBytes, KnownLayout, Immutable, Unaligned)]
+ /// #[repr(C)]
+ /// struct UdpHeader {
+ /// src_port: [u8; 2],
+ /// dst_port: [u8; 2],
+ /// length: [u8; 2],
+ /// checksum: [u8; 2],
+ /// }
+ ///
+ /// #[derive(FromBytes, IntoBytes, KnownLayout, Immutable, Unaligned)]
+ /// #[repr(C, packed)]
+ /// struct UdpPacket {
+ /// header: UdpHeader,
+ /// body: [u8],
+ /// }
+ ///
+ /// impl UdpPacket {
+ /// pub fn parse(bytes: &[u8]) -> Result<&UdpPacket, SizeError<&[u8], UdpPacket>> {
+ /// // Since `UdpPacket: Unaligned`, we can map the `CastError` to a `SizeError`.
+ /// UdpPacket::ref_from_bytes(bytes).map_err(Into::into)
+ /// }
+ /// }
+ /// ```
+ #[inline(always)]
+ fn from(err: CastError<Src, Dst>) -> SizeError<Src, Dst> {
+ match err {
+ #[allow(unreachable_code)]
+ CastError::Alignment(e) => match Infallible::from(e) {},
+ CastError::Size(e) => e,
+ CastError::Validity(i) => match i {},
+ }
+ }
+}
+
+/// The error type of fallible reference conversions.
+///
+/// Fallible reference conversions, like [`TryFromBytes::try_ref_from_bytes`]
+/// may emit [alignment](AlignmentError), [size](SizeError), and
+/// [validity](ValidityError) errors.
+// Bounds on generic parameters are not enforced in type aliases, but they do
+// appear in rustdoc.
+#[allow(type_alias_bounds)]
+pub type TryCastError<Src, Dst: ?Sized + TryFromBytes> =
+ ConvertError<AlignmentError<Src, Dst>, SizeError<Src, Dst>, ValidityError<Src, Dst>>;
+
+// FIXME(#1139): Remove the `TryFromBytes` here and in other downstream
+// locations (all the way to `ValidityError`) if we determine it's not necessary
+// for rich validity errors.
+impl<Src, Dst: ?Sized + TryFromBytes> TryCastError<Src, Dst> {
+ /// Produces the source underlying the failed conversion.
+ #[inline]
+ pub fn into_src(self) -> Src {
+ match self {
+ Self::Alignment(e) => e.src,
+ Self::Size(e) => e.src,
+ Self::Validity(e) => e.src,
+ }
+ }
+
+ /// Maps the source value associated with the conversion error.
+ ///
+ /// This can help mitigate [issues with `Send`, `Sync` and `'static`
+ /// bounds][self#send-sync-and-static].
+ ///
+ /// # Examples
+ ///
+ /// ```
+ /// use core::num::NonZeroU32;
+ /// use zerocopy::*;
+ ///
+ /// let source: [u8; 3] = [0, 0, 0];
+ ///
+ /// // Try to read a `NonZeroU32` from `source`.
+ /// let maybe_u32: Result<&NonZeroU32, TryCastError<&[u8], NonZeroU32>>
+ /// = NonZeroU32::try_ref_from_bytes(&source[..]);
+ ///
+ /// // Map the error's source to its size and address.
+ /// let maybe_u32: Result<&NonZeroU32, TryCastError<(usize, usize), NonZeroU32>> =
+ /// maybe_u32.map_err(|err| {
+ /// err.map_src(|src| (src.len(), src.as_ptr() as usize))
+ /// });
+ /// ```
+ #[inline]
+ pub fn map_src<NewSrc>(self, f: impl FnOnce(Src) -> NewSrc) -> TryCastError<NewSrc, Dst> {
+ match self {
+ Self::Alignment(e) => TryCastError::Alignment(e.map_src(f)),
+ Self::Size(e) => TryCastError::Size(e.map_src(f)),
+ Self::Validity(e) => TryCastError::Validity(e.map_src(f)),
+ }
+ }
+}
+
+impl<Src, Dst: ?Sized + TryFromBytes> From<CastError<Src, Dst>> for TryCastError<Src, Dst> {
+ #[inline]
+ fn from(value: CastError<Src, Dst>) -> Self {
+ match value {
+ CastError::Alignment(e) => Self::Alignment(e),
+ CastError::Size(e) => Self::Size(e),
+ CastError::Validity(i) => match i {},
+ }
+ }
+}
+
+/// The error type of fallible read-conversions.
+///
+/// Fallible read-conversions, like [`TryFromBytes::try_read_from_bytes`] may
+/// emit [size](SizeError) and [validity](ValidityError) errors, but not
+/// alignment errors.
+// Bounds on generic parameters are not enforced in type aliases, but they do
+// appear in rustdoc.
+#[allow(type_alias_bounds)]
+pub type TryReadError<Src, Dst: ?Sized + TryFromBytes> =
+ ConvertError<Infallible, SizeError<Src, Dst>, ValidityError<Src, Dst>>;
+
+impl<Src, Dst: ?Sized + TryFromBytes> TryReadError<Src, Dst> {
+ /// Produces the source underlying the failed conversion.
+ #[inline]
+ pub fn into_src(self) -> Src {
+ match self {
+ Self::Alignment(i) => match i {},
+ Self::Size(e) => e.src,
+ Self::Validity(e) => e.src,
+ }
+ }
+
+ /// Maps the source value associated with the conversion error.
+ ///
+ /// This can help mitigate [issues with `Send`, `Sync` and `'static`
+ /// bounds][self#send-sync-and-static].
+ ///
+ /// # Examples
+ ///
+ /// ```
+ /// use core::num::NonZeroU32;
+ /// use zerocopy::*;
+ ///
+ /// let source: [u8; 3] = [0, 0, 0];
+ ///
+ /// // Try to read a `NonZeroU32` from `source`.
+ /// let maybe_u32: Result<NonZeroU32, TryReadError<&[u8], NonZeroU32>>
+ /// = NonZeroU32::try_read_from_bytes(&source[..]);
+ ///
+ /// // Map the error's source to its size.
+ /// let maybe_u32: Result<NonZeroU32, TryReadError<usize, NonZeroU32>> =
+ /// maybe_u32.map_err(|err| {
+ /// err.map_src(|src| src.len())
+ /// });
+ /// ```
+ #[inline]
+ pub fn map_src<NewSrc>(self, f: impl FnOnce(Src) -> NewSrc) -> TryReadError<NewSrc, Dst> {
+ match self {
+ Self::Alignment(i) => match i {},
+ Self::Size(e) => TryReadError::Size(e.map_src(f)),
+ Self::Validity(e) => TryReadError::Validity(e.map_src(f)),
+ }
+ }
+}
+
+/// The error type of well-aligned, fallible casts.
+///
+/// This is like [`TryCastError`], but for casts that are always well-aligned.
+/// It is identical to `TryCastError`, except that its alignment error is
+/// [`Infallible`].
+///
+/// As of this writing, none of zerocopy's API produces this error directly.
+/// However, it is useful since it permits users to infallibly discard alignment
+/// errors when they can prove statically that alignment errors are impossible.
+///
+/// # Examples
+///
+/// ```
+/// use core::convert::Infallible;
+/// use zerocopy::*;
+/// # use zerocopy_derive::*;
+///
+/// #[derive(TryFromBytes, KnownLayout, Unaligned, Immutable)]
+/// #[repr(C, packed)]
+/// struct Bools {
+/// one: bool,
+/// two: bool,
+/// many: [bool],
+/// }
+///
+/// impl Bools {
+/// fn parse(bytes: &[u8]) -> Result<&Bools, AlignedTryCastError<&[u8], Bools>> {
+/// // Since `Bools: Unaligned`, we can infallibly discard
+/// // the alignment error.
+/// Bools::try_ref_from_bytes(bytes).map_err(Into::into)
+/// }
+/// }
+/// ```
+#[allow(type_alias_bounds)]
+pub type AlignedTryCastError<Src, Dst: ?Sized + TryFromBytes> =
+ ConvertError<Infallible, SizeError<Src, Dst>, ValidityError<Src, Dst>>;
+
+/// The error type of a failed allocation.
+///
+/// This type is intended to be deprecated in favor of the standard library's
+/// [`AllocError`] type once it is stabilized. When that happens, this type will
+/// be replaced by a type alias to the standard library type. We do not intend
+/// to treat this as a breaking change; users who wish to avoid breakage should
+/// avoid writing code which assumes that this is *not* such an alias. For
+/// example, implementing the same trait for both types will result in an impl
+/// conflict once this type is an alias.
+///
+/// [`AllocError`]: https://doc.rust-lang.org/alloc/alloc/struct.AllocError.html
+#[derive(Copy, Clone, PartialEq, Eq, Debug)]
+pub struct AllocError;
+
+#[cfg(test)]
+mod tests {
+ use core::convert::Infallible;
+
+ use super::*;
+
+ #[test]
+ fn test_send_sync() {
+ // Test that all error types are `Send + Sync` even if `Dst: !Send +
+ // !Sync`.
+
+ #[allow(dead_code)]
+ fn is_send_sync<T: Send + Sync>(_t: T) {}
+
+ #[allow(dead_code)]
+ fn alignment_err_is_send_sync<Src: Send + Sync, Dst>(err: AlignmentError<Src, Dst>) {
+ is_send_sync(err)
+ }
+
+ #[allow(dead_code)]
+ fn size_err_is_send_sync<Src: Send + Sync, Dst>(err: SizeError<Src, Dst>) {
+ is_send_sync(err)
+ }
+
+ #[allow(dead_code)]
+ fn validity_err_is_send_sync<Src: Send + Sync, Dst: TryFromBytes>(
+ err: ValidityError<Src, Dst>,
+ ) {
+ is_send_sync(err)
+ }
+
+ #[allow(dead_code)]
+ fn convert_error_is_send_sync<Src: Send + Sync, Dst: TryFromBytes>(
+ err: ConvertError<
+ AlignmentError<Src, Dst>,
+ SizeError<Src, Dst>,
+ ValidityError<Src, Dst>,
+ >,
+ ) {
+ is_send_sync(err)
+ }
+ }
+
+ #[test]
+ fn test_eq_partial_eq_clone() {
+ // Test that all error types implement `Eq`, `PartialEq`
+ // and `Clone` if src does
+ // even if `Dst: !Eq`, `!PartialEq`, `!Clone`.
+
+ #[allow(dead_code)]
+ fn is_eq_partial_eq_clone<T: Eq + PartialEq + Clone>(_t: T) {}
+
+ #[allow(dead_code)]
+ fn alignment_err_is_eq_partial_eq_clone<Src: Eq + PartialEq + Clone, Dst>(
+ err: AlignmentError<Src, Dst>,
+ ) {
+ is_eq_partial_eq_clone(err)
+ }
+
+ #[allow(dead_code)]
+ fn size_err_is_eq_partial_eq_clone<Src: Eq + PartialEq + Clone, Dst>(
+ err: SizeError<Src, Dst>,
+ ) {
+ is_eq_partial_eq_clone(err)
+ }
+
+ #[allow(dead_code)]
+ fn validity_err_is_eq_partial_eq_clone<Src: Eq + PartialEq + Clone, Dst: TryFromBytes>(
+ err: ValidityError<Src, Dst>,
+ ) {
+ is_eq_partial_eq_clone(err)
+ }
+
+ #[allow(dead_code)]
+ fn convert_error_is_eq_partial_eq_clone<Src: Eq + PartialEq + Clone, Dst: TryFromBytes>(
+ err: ConvertError<
+ AlignmentError<Src, Dst>,
+ SizeError<Src, Dst>,
+ ValidityError<Src, Dst>,
+ >,
+ ) {
+ is_eq_partial_eq_clone(err)
+ }
+ }
+
+ #[test]
+ fn alignment_display() {
+ #[repr(C, align(128))]
+ struct Aligned {
+ bytes: [u8; 128],
+ }
+
+ impl_known_layout!(elain::Align::<8>);
+
+ let aligned = Aligned { bytes: [0; 128] };
+
+ let bytes = &aligned.bytes[1..];
+ let addr = crate::util::AsAddress::addr(bytes);
+ assert_eq!(
+ AlignmentError::<_, elain::Align::<8>>::new_checked(bytes).to_string(),
+ format!("The conversion failed because the address of the source is not a multiple of the alignment of the destination type.\n\
+ \nSource type: &[u8]\
+ \nSource address: 0x{:x} (a multiple of 1)\
+ \nDestination type: elain::Align<8>\
+ \nDestination alignment: 8", addr)
+ );
+
+ let bytes = &aligned.bytes[2..];
+ let addr = crate::util::AsAddress::addr(bytes);
+ assert_eq!(
+ AlignmentError::<_, elain::Align::<8>>::new_checked(bytes).to_string(),
+ format!("The conversion failed because the address of the source is not a multiple of the alignment of the destination type.\n\
+ \nSource type: &[u8]\
+ \nSource address: 0x{:x} (a multiple of 2)\
+ \nDestination type: elain::Align<8>\
+ \nDestination alignment: 8", addr)
+ );
+
+ let bytes = &aligned.bytes[3..];
+ let addr = crate::util::AsAddress::addr(bytes);
+ assert_eq!(
+ AlignmentError::<_, elain::Align::<8>>::new_checked(bytes).to_string(),
+ format!("The conversion failed because the address of the source is not a multiple of the alignment of the destination type.\n\
+ \nSource type: &[u8]\
+ \nSource address: 0x{:x} (a multiple of 1)\
+ \nDestination type: elain::Align<8>\
+ \nDestination alignment: 8", addr)
+ );
+
+ let bytes = &aligned.bytes[4..];
+ let addr = crate::util::AsAddress::addr(bytes);
+ assert_eq!(
+ AlignmentError::<_, elain::Align::<8>>::new_checked(bytes).to_string(),
+ format!("The conversion failed because the address of the source is not a multiple of the alignment of the destination type.\n\
+ \nSource type: &[u8]\
+ \nSource address: 0x{:x} (a multiple of 4)\
+ \nDestination type: elain::Align<8>\
+ \nDestination alignment: 8", addr)
+ );
+ }
+
+ #[test]
+ fn size_display() {
+ assert_eq!(
+ SizeError::<_, [u8]>::new(&[0u8; 2][..]).to_string(),
+ "The conversion failed because the source was incorrectly sized to complete the conversion into the destination type.\n\
+ \nSource type: &[u8]\
+ \nSource size: 2 bytes\
+ \nDestination type: [u8]"
+ );
+
+ assert_eq!(
+ SizeError::<_, [u8; 2]>::new(&[0u8; 1][..]).to_string(),
+ "The conversion failed because the source was incorrectly sized to complete the conversion into the destination type.\n\
+ \nSource type: &[u8]\
+ \nSource size: 1 byte\
+ \nDestination size: 2 bytes\
+ \nDestination type: [u8; 2]"
+ );
+ }
+
+ #[test]
+ fn validity_display() {
+ assert_eq!(
+ ValidityError::<_, bool>::new(&[2u8; 1][..]).to_string(),
+ "The conversion failed because the source bytes are not a valid value of the destination type.\n\
+ \n\
+ Destination type: bool"
+ );
+ }
+
+ #[test]
+ fn test_convert_error_debug() {
+ let err: ConvertError<
+ AlignmentError<&[u8], u16>,
+ SizeError<&[u8], u16>,
+ ValidityError<&[u8], bool>,
+ > = ConvertError::Alignment(AlignmentError::new_checked(&[0u8]));
+ assert_eq!(format!("{:?}", err), "Alignment(AlignmentError)");
+
+ let err: ConvertError<
+ AlignmentError<&[u8], u16>,
+ SizeError<&[u8], u16>,
+ ValidityError<&[u8], bool>,
+ > = ConvertError::Size(SizeError::new(&[0u8]));
+ assert_eq!(format!("{:?}", err), "Size(SizeError)");
+
+ let err: ConvertError<
+ AlignmentError<&[u8], u16>,
+ SizeError<&[u8], u16>,
+ ValidityError<&[u8], bool>,
+ > = ConvertError::Validity(ValidityError::new(&[0u8]));
+ assert_eq!(format!("{:?}", err), "Validity(ValidityError)");
+ }
+
+ #[test]
+ fn test_convert_error_from_unaligned() {
+ // u8 is Unaligned
+ let err: ConvertError<
+ AlignmentError<&[u8], u8>,
+ SizeError<&[u8], u8>,
+ ValidityError<&[u8], bool>,
+ > = ConvertError::Size(SizeError::new(&[0u8]));
+ let converted: ConvertError<Infallible, SizeError<&[u8], u8>, ValidityError<&[u8], bool>> =
+ ConvertError::from(err);
+ match converted {
+ ConvertError::Size(_) => {}
+ _ => panic!("Expected Size error"),
+ }
+ }
+
+ #[test]
+ fn test_alignment_error_display_debug() {
+ let err: AlignmentError<&[u8], u16> = AlignmentError::new_checked(&[0u8]);
+ assert!(format!("{:?}", err).contains("AlignmentError"));
+ assert!(format!("{}", err).contains("address of the source is not a multiple"));
+ }
+
+ #[test]
+ fn test_size_error_display_debug() {
+ let err: SizeError<&[u8], u16> = SizeError::new(&[0u8]);
+ assert!(format!("{:?}", err).contains("SizeError"));
+ assert!(format!("{}", err).contains("source was incorrectly sized"));
+ }
+
+ #[test]
+ fn test_validity_error_display_debug() {
+ let err: ValidityError<&[u8], bool> = ValidityError::new(&[0u8]);
+ assert!(format!("{:?}", err).contains("ValidityError"));
+ assert!(format!("{}", err).contains("source bytes are not a valid value"));
+ }
+
+ #[test]
+ fn test_convert_error_display_debug_more() {
+ let err: ConvertError<
+ AlignmentError<&[u8], u16>,
+ SizeError<&[u8], u16>,
+ ValidityError<&[u8], bool>,
+ > = ConvertError::Alignment(AlignmentError::new_checked(&[0u8]));
+ assert!(format!("{}", err).contains("address of the source is not a multiple"));
+
+ let err: ConvertError<
+ AlignmentError<&[u8], u16>,
+ SizeError<&[u8], u16>,
+ ValidityError<&[u8], bool>,
+ > = ConvertError::Size(SizeError::new(&[0u8]));
+ assert!(format!("{}", err).contains("source was incorrectly sized"));
+
+ let err: ConvertError<
+ AlignmentError<&[u8], u16>,
+ SizeError<&[u8], u16>,
+ ValidityError<&[u8], bool>,
+ > = ConvertError::Validity(ValidityError::new(&[0u8]));
+ assert!(format!("{}", err).contains("source bytes are not a valid value"));
+ }
+
+ #[test]
+ fn test_alignment_error_methods() {
+ let err: AlignmentError<&[u8], u16> = AlignmentError::new_checked(&[0u8]);
+
+ // into_src
+ let src = err.clone().into_src();
+ assert_eq!(src, &[0u8]);
+
+ // into
+ let converted: ConvertError<
+ AlignmentError<&[u8], u16>,
+ SizeError<&[u8], u16>,
+ ValidityError<&[u8], bool>,
+ > = err.clone().into();
+ match converted {
+ ConvertError::Alignment(_) => {}
+ _ => panic!("Expected Alignment error"),
+ }
+
+ // clone
+ let cloned = err.clone();
+ assert_eq!(err, cloned);
+
+ // eq
+ assert_eq!(err, cloned);
+ let err2: AlignmentError<&[u8], u16> = AlignmentError::new_checked(&[1u8]);
+ assert_ne!(err, err2);
+ }
+
+ #[test]
+ fn test_convert_error_from_unaligned_variants() {
+ // u8 is Unaligned
+ let err: ConvertError<
+ AlignmentError<&[u8], u8>,
+ SizeError<&[u8], u8>,
+ ValidityError<&[u8], bool>,
+ > = ConvertError::Validity(ValidityError::new(&[0u8]));
+ let converted: ConvertError<Infallible, SizeError<&[u8], u8>, ValidityError<&[u8], bool>> =
+ ConvertError::from(err);
+ match converted {
+ ConvertError::Validity(_) => {}
+ _ => panic!("Expected Validity error"),
+ }
+
+ let err: ConvertError<
+ AlignmentError<&[u8], u8>,
+ SizeError<&[u8], u8>,
+ ValidityError<&[u8], bool>,
+ > = ConvertError::Size(SizeError::new(&[0u8]));
+ let converted: ConvertError<Infallible, SizeError<&[u8], u8>, ValidityError<&[u8], bool>> =
+ ConvertError::from(err);
+ match converted {
+ ConvertError::Size(_) => {}
+ _ => panic!("Expected Size error"),
+ }
+ }
+}
diff --git a/rust/zerocopy/src/impls.rs b/rust/zerocopy/src/impls.rs
new file mode 100644
index 0000000..22fd6c3
--- /dev/null
+++ b/rust/zerocopy/src/impls.rs
@@ -0,0 +1,2389 @@
+// SPDX-License-Identifier: (BSD-2-Clause OR Apache-2.0) OR MIT
+
+// Copyright 2024 The Fuchsia Authors
+//
+// Licensed under the 2-Clause BSD License <LICENSE-BSD or
+// https://opensource.org/license/bsd-2-clause>, Apache License, Version 2.0
+// <LICENSE-APACHE or https://www.apache.org/licenses/LICENSE-2.0>, or the MIT
+// license <LICENSE-MIT or https://opensource.org/licenses/MIT>, at your option.
+// This file may not be copied, modified, or distributed except according to
+// those terms.
+
+use core::{
+ cell::{Cell, UnsafeCell},
+ mem::MaybeUninit as CoreMaybeUninit,
+ ptr::NonNull,
+};
+
+use super::*;
+use crate::pointer::cast::{CastSizedExact, CastUnsized};
+
+// SAFETY: Per the reference [1], "the unit tuple (`()`) ... is guaranteed as a
+// zero-sized type to have a size of 0 and an alignment of 1."
+// - `Immutable`: `()` self-evidently does not contain any `UnsafeCell`s.
+// - `TryFromBytes` (with no validator), `FromZeros`, `FromBytes`: There is only
+// one possible sequence of 0 bytes, and `()` is inhabited.
+// - `IntoBytes`: Since `()` has size 0, it contains no padding bytes.
+// - `Unaligned`: `()` has alignment 1.
+//
+// [1] https://doc.rust-lang.org/1.81.0/reference/type-layout.html#tuple-layout
+#[allow(clippy::multiple_unsafe_ops_per_block)]
+const _: () = unsafe {
+ unsafe_impl!((): Immutable, TryFromBytes, FromZeros, FromBytes, IntoBytes, Unaligned);
+ assert_unaligned!(());
+};
+
+// SAFETY:
+// - `Immutable`: These types self-evidently do not contain any `UnsafeCell`s.
+// - `TryFromBytes` (with no validator), `FromZeros`, `FromBytes`: all bit
+// patterns are valid for numeric types [1]
+// - `IntoBytes`: numeric types have no padding bytes [1]
+// - `Unaligned` (`u8` and `i8` only): The reference [2] specifies the size of
+// `u8` and `i8` as 1 byte. We also know that:
+// - Alignment is >= 1 [3]
+// - Size is an integer multiple of alignment [4]
+// - The only value >= 1 for which 1 is an integer multiple is 1 Therefore,
+// the only possible alignment for `u8` and `i8` is 1.
+//
+// [1] Per https://doc.rust-lang.org/1.81.0/reference/types/numeric.html#bit-validity:
+//
+// For every numeric type, `T`, the bit validity of `T` is equivalent to
+// the bit validity of `[u8; size_of::<T>()]`. An uninitialized byte is
+// not a valid `u8`.
+//
+// [2] https://doc.rust-lang.org/1.81.0/reference/type-layout.html#primitive-data-layout
+//
+// [3] Per https://doc.rust-lang.org/1.81.0/reference/type-layout.html#size-and-alignment:
+//
+// Alignment is measured in bytes, and must be at least 1.
+//
+// [4] Per https://doc.rust-lang.org/1.81.0/reference/type-layout.html#size-and-alignment:
+//
+// The size of a value is always a multiple of its alignment.
+//
+// FIXME(#278): Once we've updated the trait docs to refer to `u8`s rather than
+// bits or bytes, update this comment, especially the reference to [1].
+#[allow(clippy::multiple_unsafe_ops_per_block)]
+const _: () = unsafe {
+ unsafe_impl!(u8: Immutable, TryFromBytes, FromZeros, FromBytes, IntoBytes, Unaligned);
+ unsafe_impl!(i8: Immutable, TryFromBytes, FromZeros, FromBytes, IntoBytes, Unaligned);
+ assert_unaligned!(u8, i8);
+ unsafe_impl!(u16: Immutable, TryFromBytes, FromZeros, FromBytes, IntoBytes);
+ unsafe_impl!(i16: Immutable, TryFromBytes, FromZeros, FromBytes, IntoBytes);
+ unsafe_impl!(u32: Immutable, TryFromBytes, FromZeros, FromBytes, IntoBytes);
+ unsafe_impl!(i32: Immutable, TryFromBytes, FromZeros, FromBytes, IntoBytes);
+ unsafe_impl!(u64: Immutable, TryFromBytes, FromZeros, FromBytes, IntoBytes);
+ unsafe_impl!(i64: Immutable, TryFromBytes, FromZeros, FromBytes, IntoBytes);
+ unsafe_impl!(u128: Immutable, TryFromBytes, FromZeros, FromBytes, IntoBytes);
+ unsafe_impl!(i128: Immutable, TryFromBytes, FromZeros, FromBytes, IntoBytes);
+ unsafe_impl!(usize: Immutable, TryFromBytes, FromZeros, FromBytes, IntoBytes);
+ unsafe_impl!(isize: Immutable, TryFromBytes, FromZeros, FromBytes, IntoBytes);
+ unsafe_impl!(f32: Immutable, TryFromBytes, FromZeros, FromBytes, IntoBytes);
+ unsafe_impl!(f64: Immutable, TryFromBytes, FromZeros, FromBytes, IntoBytes);
+ #[cfg(feature = "float-nightly")]
+ unsafe_impl!(#[cfg_attr(doc_cfg, doc(cfg(feature = "float-nightly")))] f16: Immutable, TryFromBytes, FromZeros, FromBytes, IntoBytes);
+ #[cfg(feature = "float-nightly")]
+ unsafe_impl!(#[cfg_attr(doc_cfg, doc(cfg(feature = "float-nightly")))] f128: Immutable, TryFromBytes, FromZeros, FromBytes, IntoBytes);
+};
+
+// SAFETY:
+// - `Immutable`: `bool` self-evidently does not contain any `UnsafeCell`s.
+// - `FromZeros`: Valid since "[t]he value false has the bit pattern 0x00" [1].
+// - `IntoBytes`: Since "the boolean type has a size and alignment of 1 each"
+// and "The value false has the bit pattern 0x00 and the value true has the
+// bit pattern 0x01" [1]. Thus, the only byte of the bool is always
+// initialized.
+// - `Unaligned`: Per the reference [1], "[a]n object with the boolean type has
+// a size and alignment of 1 each."
+//
+// [1] https://doc.rust-lang.org/1.81.0/reference/types/boolean.html
+#[allow(clippy::multiple_unsafe_ops_per_block)]
+const _: () = unsafe { unsafe_impl!(bool: Immutable, FromZeros, IntoBytes, Unaligned) };
+assert_unaligned!(bool);
+
+// SAFETY: The impl must only return `true` for its argument if the original
+// `Maybe<bool>` refers to a valid `bool`. We only return true if the `u8` value
+// is 0 or 1, and both of these are valid values for `bool` [1].
+//
+// [1] Per https://doc.rust-lang.org/1.81.0/reference/types/boolean.html:
+//
+// The value false has the bit pattern 0x00 and the value true has the bit
+// pattern 0x01.
+const _: () = unsafe {
+ unsafe_impl!(=> TryFromBytes for bool; |byte| {
+ let byte = byte.transmute_with::<u8, invariant::Valid, CastSizedExact, BecauseImmutable>();
+ *byte.unaligned_as_ref() < 2
+ })
+};
+
+// SAFETY:
+// - `Immutable`: `char` self-evidently does not contain any `UnsafeCell`s.
+// - `FromZeros`: Per reference [1], "[a] value of type char is a Unicode scalar
+// value (i.e. a code point that is not a surrogate), represented as a 32-bit
+// unsigned word in the 0x0000 to 0xD7FF or 0xE000 to 0x10FFFF range" which
+// contains 0x0000.
+// - `IntoBytes`: `char` is per reference [1] "represented as a 32-bit unsigned
+// word" (`u32`) which is `IntoBytes`. Note that unlike `u32`, not all bit
+// patterns are valid for `char`.
+//
+// [1] https://doc.rust-lang.org/1.81.0/reference/types/textual.html
+#[allow(clippy::multiple_unsafe_ops_per_block)]
+const _: () = unsafe { unsafe_impl!(char: Immutable, FromZeros, IntoBytes) };
+
+// SAFETY: The impl must only return `true` for its argument if the original
+// `Maybe<char>` refers to a valid `char`. `char::from_u32` guarantees that it
+// returns `None` if its input is not a valid `char` [1].
+//
+// [1] Per https://doc.rust-lang.org/core/primitive.char.html#method.from_u32:
+//
+// `from_u32()` will return `None` if the input is not a valid value for a
+// `char`.
+const _: () = unsafe {
+ unsafe_impl!(=> TryFromBytes for char; |c| {
+ let c = c.transmute_with::<Unalign<u32>, invariant::Valid, CastSizedExact, BecauseImmutable>();
+ let c = c.read().into_inner();
+ char::from_u32(c).is_some()
+ });
+};
+
+// SAFETY: Per the Reference [1], `str` has the same layout as `[u8]`.
+// - `Immutable`: `[u8]` does not contain any `UnsafeCell`s.
+// - `FromZeros`, `IntoBytes`, `Unaligned`: `[u8]` is `FromZeros`, `IntoBytes`,
+// and `Unaligned`.
+//
+// Note that we don't `assert_unaligned!(str)` because `assert_unaligned!` uses
+// `align_of`, which only works for `Sized` types.
+//
+// FIXME(#429): Improve safety proof for `FromZeros` and `IntoBytes`; having the same
+// layout as `[u8]` isn't sufficient.
+//
+// [1] Per https://doc.rust-lang.org/1.81.0/reference/type-layout.html#str-layout:
+//
+// String slices are a UTF-8 representation of characters that have the same
+// layout as slices of type `[u8]`.
+#[allow(clippy::multiple_unsafe_ops_per_block)]
+const _: () = unsafe { unsafe_impl!(str: Immutable, FromZeros, IntoBytes, Unaligned) };
+
+// SAFETY: The impl must only return `true` for its argument if the original
+// `Maybe<str>` refers to a valid `str`. `str::from_utf8` guarantees that it
+// returns `Err` if its input is not a valid `str` [1].
+//
+// [1] Per https://doc.rust-lang.org/core/str/fn.from_utf8.html#errors:
+//
+// Returns `Err` if the slice is not UTF-8.
+const _: () = unsafe {
+ unsafe_impl!(=> TryFromBytes for str; |c| {
+ let c = c.transmute_with::<[u8], invariant::Valid, CastUnsized, BecauseImmutable>();
+ let c = c.unaligned_as_ref();
+ core::str::from_utf8(c).is_ok()
+ })
+};
+
+macro_rules! unsafe_impl_try_from_bytes_for_nonzero {
+ ($($nonzero:ident[$prim:ty]),*) => {
+ $(
+ unsafe_impl!(=> TryFromBytes for $nonzero; |n| {
+ let n = n.transmute_with::<Unalign<$prim>, invariant::Valid, CastSizedExact, BecauseImmutable>();
+ $nonzero::new(n.read().into_inner()).is_some()
+ });
+ )*
+ }
+}
+
+// `NonZeroXxx` is `IntoBytes`, but not `FromZeros` or `FromBytes`.
+//
+// SAFETY:
+// - `IntoBytes`: `NonZeroXxx` has the same layout as its associated primitive.
+// Since it is the same size, this guarantees it has no padding - integers
+// have no padding, and there's no room for padding if it can represent all
+// of the same values except 0.
+// - `Unaligned`: `NonZeroU8` and `NonZeroI8` document that `Option<NonZeroU8>`
+// and `Option<NonZeroI8>` both have size 1. [1] [2] This is worded in a way
+// that makes it unclear whether it's meant as a guarantee, but given the
+// purpose of those types, it's virtually unthinkable that that would ever
+// change. `Option` cannot be smaller than its contained type, which implies
+// that, and `NonZeroX8` are of size 1 or 0. `NonZeroX8` can represent
+// multiple states, so they cannot be 0 bytes, which means that they must be 1
+// byte. The only valid alignment for a 1-byte type is 1.
+//
+// FIXME(#429):
+// - Add quotes from documentation.
+// - Add safety comment for `Immutable`. How can we prove that `NonZeroXxx`
+// doesn't contain any `UnsafeCell`s? It's obviously true, but it's not clear
+// how we'd prove it short of adding text to the stdlib docs that says so
+// explicitly, which likely wouldn't be accepted.
+//
+// [1] Per https://doc.rust-lang.org/1.81.0/std/num/type.NonZeroU8.html:
+//
+// `NonZeroU8` is guaranteed to have the same layout and bit validity as `u8` with
+// the exception that 0 is not a valid instance.
+//
+// [2] Per https://doc.rust-lang.org/1.81.0/std/num/type.NonZeroI8.html:
+//
+// `NonZeroI8` is guaranteed to have the same layout and bit validity as `i8` with
+// the exception that 0 is not a valid instance.
+#[allow(clippy::multiple_unsafe_ops_per_block)]
+const _: () = unsafe {
+ unsafe_impl!(NonZeroU8: Immutable, IntoBytes, Unaligned);
+ unsafe_impl!(NonZeroI8: Immutable, IntoBytes, Unaligned);
+ assert_unaligned!(NonZeroU8, NonZeroI8);
+ unsafe_impl!(NonZeroU16: Immutable, IntoBytes);
+ unsafe_impl!(NonZeroI16: Immutable, IntoBytes);
+ unsafe_impl!(NonZeroU32: Immutable, IntoBytes);
+ unsafe_impl!(NonZeroI32: Immutable, IntoBytes);
+ unsafe_impl!(NonZeroU64: Immutable, IntoBytes);
+ unsafe_impl!(NonZeroI64: Immutable, IntoBytes);
+ unsafe_impl!(NonZeroU128: Immutable, IntoBytes);
+ unsafe_impl!(NonZeroI128: Immutable, IntoBytes);
+ unsafe_impl!(NonZeroUsize: Immutable, IntoBytes);
+ unsafe_impl!(NonZeroIsize: Immutable, IntoBytes);
+ unsafe_impl_try_from_bytes_for_nonzero!(
+ NonZeroU8[u8],
+ NonZeroI8[i8],
+ NonZeroU16[u16],
+ NonZeroI16[i16],
+ NonZeroU32[u32],
+ NonZeroI32[i32],
+ NonZeroU64[u64],
+ NonZeroI64[i64],
+ NonZeroU128[u128],
+ NonZeroI128[i128],
+ NonZeroUsize[usize],
+ NonZeroIsize[isize]
+ );
+};
+
+// SAFETY:
+// - `TryFromBytes` (with no validator), `FromZeros`, `FromBytes`, `IntoBytes`:
+// The Rust compiler reuses `0` value to represent `None`, so
+// `size_of::<Option<NonZeroXxx>>() == size_of::<xxx>()`; see `NonZeroXxx`
+// documentation.
+// - `Unaligned`: `NonZeroU8` and `NonZeroI8` document that `Option<NonZeroU8>`
+// and `Option<NonZeroI8>` both have size 1. [1] [2] This is worded in a way
+// that makes it unclear whether it's meant as a guarantee, but given the
+// purpose of those types, it's virtually unthinkable that that would ever
+// change. The only valid alignment for a 1-byte type is 1.
+//
+// [1] Per https://doc.rust-lang.org/1.81.0/std/num/type.NonZeroU8.html:
+//
+// `Option<NonZeroU8>` is guaranteed to be compatible with `u8`, including in FFI.
+//
+// Thanks to the null pointer optimization, `NonZeroU8` and `Option<NonZeroU8>`
+// are guaranteed to have the same size and alignment:
+//
+// [2] Per https://doc.rust-lang.org/1.81.0/std/num/type.NonZeroI8.html:
+//
+// `Option<NonZeroI8>` is guaranteed to be compatible with `i8`, including in FFI.
+//
+// Thanks to the null pointer optimization, `NonZeroI8` and `Option<NonZeroI8>`
+// are guaranteed to have the same size and alignment:
+#[allow(clippy::multiple_unsafe_ops_per_block)]
+const _: () = unsafe {
+ unsafe_impl!(Option<NonZeroU8>: TryFromBytes, FromZeros, FromBytes, IntoBytes, Unaligned);
+ unsafe_impl!(Option<NonZeroI8>: TryFromBytes, FromZeros, FromBytes, IntoBytes, Unaligned);
+ assert_unaligned!(Option<NonZeroU8>, Option<NonZeroI8>);
+ unsafe_impl!(Option<NonZeroU16>: TryFromBytes, FromZeros, FromBytes, IntoBytes);
+ unsafe_impl!(Option<NonZeroI16>: TryFromBytes, FromZeros, FromBytes, IntoBytes);
+ unsafe_impl!(Option<NonZeroU32>: TryFromBytes, FromZeros, FromBytes, IntoBytes);
+ unsafe_impl!(Option<NonZeroI32>: TryFromBytes, FromZeros, FromBytes, IntoBytes);
+ unsafe_impl!(Option<NonZeroU64>: TryFromBytes, FromZeros, FromBytes, IntoBytes);
+ unsafe_impl!(Option<NonZeroI64>: TryFromBytes, FromZeros, FromBytes, IntoBytes);
+ unsafe_impl!(Option<NonZeroU128>: TryFromBytes, FromZeros, FromBytes, IntoBytes);
+ unsafe_impl!(Option<NonZeroI128>: TryFromBytes, FromZeros, FromBytes, IntoBytes);
+ unsafe_impl!(Option<NonZeroUsize>: TryFromBytes, FromZeros, FromBytes, IntoBytes);
+ unsafe_impl!(Option<NonZeroIsize>: TryFromBytes, FromZeros, FromBytes, IntoBytes);
+};
+
+// SAFETY: While it's not fully documented, the consensus is that `Box<T>` does
+// not contain any `UnsafeCell`s for `T: Sized` [1]. This is not a complete
+// proof, but we are accepting this as a known risk per #1358.
+//
+// [1] https://github.com/rust-lang/unsafe-code-guidelines/issues/492
+#[cfg(feature = "alloc")]
+const _: () = unsafe {
+ unsafe_impl!(
+ #[cfg_attr(doc_cfg, doc(cfg(feature = "alloc")))]
+ T: Sized => Immutable for Box<T>
+ )
+};
+
+// SAFETY: The following types can be transmuted from `[0u8; size_of::<T>()]`. [1]
+//
+// [1] Per https://doc.rust-lang.org/1.89.0/core/option/index.html#representation:
+//
+// Rust guarantees to optimize the following types `T` such that [`Option<T>`]
+// has the same size and alignment as `T`. In some of these cases, Rust
+// further guarantees that `transmute::<_, Option<T>>([0u8; size_of::<T>()])`
+// is sound and produces `Option::<T>::None`. These cases are identified by
+// the second column:
+//
+// | `T` | `transmute::<_, Option<T>>([0u8; size_of::<T>()])` sound? |
+// |-----------------------------------|-----------------------------------------------------------|
+// | [`Box<U>`] | when `U: Sized` |
+// | `&U` | when `U: Sized` |
+// | `&mut U` | when `U: Sized` |
+// | [`ptr::NonNull<U>`] | when `U: Sized` |
+// | `fn`, `extern "C" fn`[^extern_fn] | always |
+//
+// [^extern_fn]: this remains true for `unsafe` variants, any argument/return
+// types, and any other ABI: `[unsafe] extern "abi" fn` (_e.g._, `extern
+// "system" fn`)
+#[allow(clippy::multiple_unsafe_ops_per_block)]
+const _: () = unsafe {
+ #[cfg(feature = "alloc")]
+ unsafe_impl!(
+ #[cfg_attr(doc_cfg, doc(cfg(feature = "alloc")))]
+ T => TryFromBytes for Option<Box<T>>; |c| pointer::is_zeroed(c)
+ );
+ #[cfg(feature = "alloc")]
+ unsafe_impl!(
+ #[cfg_attr(doc_cfg, doc(cfg(feature = "alloc")))]
+ T => FromZeros for Option<Box<T>>
+ );
+ unsafe_impl!(
+ T => TryFromBytes for Option<&'_ T>; |c| pointer::is_zeroed(c)
+ );
+ unsafe_impl!(T => FromZeros for Option<&'_ T>);
+ unsafe_impl!(
+ T => TryFromBytes for Option<&'_ mut T>; |c| pointer::is_zeroed(c)
+ );
+ unsafe_impl!(T => FromZeros for Option<&'_ mut T>);
+ unsafe_impl!(
+ T => TryFromBytes for Option<NonNull<T>>; |c| pointer::is_zeroed(c)
+ );
+ unsafe_impl!(T => FromZeros for Option<NonNull<T>>);
+ unsafe_impl_for_power_set!(A, B, C, D, E, F, G, H, I, J, K, L -> M => FromZeros for opt_fn!(...));
+ unsafe_impl_for_power_set!(
+ A, B, C, D, E, F, G, H, I, J, K, L -> M => TryFromBytes for opt_fn!(...);
+ |c| pointer::is_zeroed(c)
+ );
+ unsafe_impl_for_power_set!(A, B, C, D, E, F, G, H, I, J, K, L -> M => FromZeros for opt_unsafe_fn!(...));
+ unsafe_impl_for_power_set!(
+ A, B, C, D, E, F, G, H, I, J, K, L -> M => TryFromBytes for opt_unsafe_fn!(...);
+ |c| pointer::is_zeroed(c)
+ );
+ unsafe_impl_for_power_set!(A, B, C, D, E, F, G, H, I, J, K, L -> M => FromZeros for opt_extern_c_fn!(...));
+ unsafe_impl_for_power_set!(
+ A, B, C, D, E, F, G, H, I, J, K, L -> M => TryFromBytes for opt_extern_c_fn!(...);
+ |c| pointer::is_zeroed(c)
+ );
+ unsafe_impl_for_power_set!(A, B, C, D, E, F, G, H, I, J, K, L -> M => FromZeros for opt_unsafe_extern_c_fn!(...));
+ unsafe_impl_for_power_set!(
+ A, B, C, D, E, F, G, H, I, J, K, L -> M => TryFromBytes for opt_unsafe_extern_c_fn!(...);
+ |c| pointer::is_zeroed(c)
+ );
+};
+
+// SAFETY: `[unsafe] [extern "C"] fn()` self-evidently do not contain
+// `UnsafeCell`s. This is not a proof, but we are accepting this as a known risk
+// per #1358.
+#[allow(clippy::multiple_unsafe_ops_per_block)]
+const _: () = unsafe {
+ unsafe_impl_for_power_set!(A, B, C, D, E, F, G, H, I, J, K, L -> M => Immutable for opt_fn!(...));
+ unsafe_impl_for_power_set!(A, B, C, D, E, F, G, H, I, J, K, L -> M => Immutable for opt_unsafe_fn!(...));
+ unsafe_impl_for_power_set!(A, B, C, D, E, F, G, H, I, J, K, L -> M => Immutable for opt_extern_c_fn!(...));
+ unsafe_impl_for_power_set!(A, B, C, D, E, F, G, H, I, J, K, L -> M => Immutable for opt_unsafe_extern_c_fn!(...));
+};
+
+#[cfg(all(
+ not(no_zerocopy_target_has_atomics_1_60_0),
+ any(
+ target_has_atomic = "8",
+ target_has_atomic = "16",
+ target_has_atomic = "32",
+ target_has_atomic = "64",
+ target_has_atomic = "ptr"
+ )
+))]
+#[cfg_attr(doc_cfg, doc(cfg(rust = "1.60.0")))]
+mod atomics {
+ use super::*;
+
+ macro_rules! impl_traits_for_atomics {
+ ($($atomics:tt [$primitives:ty]),* $(,)?) => {
+ $(
+ impl_known_layout!($atomics);
+ impl_for_transmute_from!(=> FromZeros for $atomics [$primitives]);
+ impl_for_transmute_from!(=> FromBytes for $atomics [$primitives]);
+ impl_for_transmute_from!(=> TryFromBytes for $atomics [$primitives]);
+ impl_for_transmute_from!(=> IntoBytes for $atomics [$primitives]);
+ )*
+ };
+ }
+
+ /// Implements `TransmuteFrom` for `$atomic`, `$prim`, and
+ /// `UnsafeCell<$prim>`.
+ ///
+ /// # Safety
+ ///
+ /// `$atomic` must have the same size and bit validity as `$prim`.
+ macro_rules! unsafe_impl_transmute_from_for_atomic {
+ ($($($tyvar:ident)? => $atomic:ty [$prim:ty]),*) => {{
+ crate::util::macros::__unsafe();
+
+ use crate::pointer::{SizeEq, TransmuteFrom, invariant::Valid};
+
+ $(
+ // SAFETY: The caller promised that `$atomic` and `$prim` have
+ // the same size and bit validity.
+ unsafe impl<$($tyvar)?> TransmuteFrom<$atomic, Valid, Valid> for $prim {}
+ // SAFETY: The caller promised that `$atomic` and `$prim` have
+ // the same size and bit validity.
+ unsafe impl<$($tyvar)?> TransmuteFrom<$prim, Valid, Valid> for $atomic {}
+
+ impl<$($tyvar)?> SizeEq<ReadOnly<$atomic>> for ReadOnly<$prim> {
+ type CastFrom = $crate::pointer::cast::CastSizedExact;
+ }
+
+ // SAFETY: The caller promised that `$atomic` and `$prim` have
+ // the same bit validity. `UnsafeCell<T>` has the same bit
+ // validity as `T` [1].
+ //
+ // [1] Per https://doc.rust-lang.org/1.85.0/std/cell/struct.UnsafeCell.html#memory-layout:
+ //
+ // `UnsafeCell<T>` has the same in-memory representation as
+ // its inner type `T`. A consequence of this guarantee is that
+ // it is possible to convert between `T` and `UnsafeCell<T>`.
+ unsafe impl<$($tyvar)?> TransmuteFrom<$atomic, Valid, Valid> for core::cell::UnsafeCell<$prim> {}
+ // SAFETY: See previous safety comment.
+ unsafe impl<$($tyvar)?> TransmuteFrom<core::cell::UnsafeCell<$prim>, Valid, Valid> for $atomic {}
+ )*
+ }};
+ }
+
+ #[cfg(target_has_atomic = "8")]
+ #[cfg_attr(doc_cfg, doc(cfg(target_has_atomic = "8")))]
+ mod atomic_8 {
+ use core::sync::atomic::{AtomicBool, AtomicI8, AtomicU8};
+
+ use super::*;
+
+ impl_traits_for_atomics!(AtomicU8[u8], AtomicI8[i8]);
+
+ impl_known_layout!(AtomicBool);
+ impl_for_transmute_from!(=> FromZeros for AtomicBool [bool]);
+ impl_for_transmute_from!(=> TryFromBytes for AtomicBool [bool]);
+ impl_for_transmute_from!(=> IntoBytes for AtomicBool [bool]);
+
+ // SAFETY: Per [1], `AtomicBool`, `AtomicU8`, and `AtomicI8` have the
+ // same size as `bool`, `u8`, and `i8` respectively. Since a type's
+ // alignment cannot be smaller than 1 [2], and since its alignment
+ // cannot be greater than its size [3], the only possible value for the
+ // alignment is 1. Thus, it is sound to implement `Unaligned`.
+ //
+ // [1] Per (for example) https://doc.rust-lang.org/1.81.0/std/sync/atomic/struct.AtomicU8.html:
+ //
+ // This type has the same size, alignment, and bit validity as the
+ // underlying integer type
+ //
+ // [2] Per https://doc.rust-lang.org/1.81.0/reference/type-layout.html#size-and-alignment:
+ //
+ // Alignment is measured in bytes, and must be at least 1.
+ //
+ // [3] Per https://doc.rust-lang.org/1.81.0/reference/type-layout.html#size-and-alignment:
+ //
+ // The size of a value is always a multiple of its alignment.
+ #[allow(clippy::multiple_unsafe_ops_per_block)]
+ const _: () = unsafe {
+ unsafe_impl!(AtomicBool: Unaligned);
+ unsafe_impl!(AtomicU8: Unaligned);
+ unsafe_impl!(AtomicI8: Unaligned);
+ assert_unaligned!(AtomicBool, AtomicU8, AtomicI8);
+ };
+
+ // SAFETY: `AtomicU8`, `AtomicI8`, and `AtomicBool` have the same size
+ // and bit validity as `u8`, `i8`, and `bool` respectively [1][2][3].
+ //
+ // [1] Per https://doc.rust-lang.org/1.85.0/std/sync/atomic/struct.AtomicU8.html:
+ //
+ // This type has the same size, alignment, and bit validity as the
+ // underlying integer type, `u8`.
+ //
+ // [2] Per https://doc.rust-lang.org/1.85.0/std/sync/atomic/struct.AtomicI8.html:
+ //
+ // This type has the same size, alignment, and bit validity as the
+ // underlying integer type, `i8`.
+ //
+ // [3] Per https://doc.rust-lang.org/1.85.0/std/sync/atomic/struct.AtomicBool.html:
+ //
+ // This type has the same size, alignment, and bit validity a `bool`.
+ #[allow(clippy::multiple_unsafe_ops_per_block)]
+ const _: () = unsafe {
+ unsafe_impl_transmute_from_for_atomic!(
+ => AtomicU8 [u8],
+ => AtomicI8 [i8],
+ => AtomicBool [bool]
+ )
+ };
+ }
+
+ #[cfg(target_has_atomic = "16")]
+ #[cfg_attr(doc_cfg, doc(cfg(target_has_atomic = "16")))]
+ mod atomic_16 {
+ use core::sync::atomic::{AtomicI16, AtomicU16};
+
+ use super::*;
+
+ impl_traits_for_atomics!(AtomicU16[u16], AtomicI16[i16]);
+
+ // SAFETY: `AtomicU16` and `AtomicI16` have the same size and bit
+ // validity as `u16` and `i16` respectively [1][2].
+ //
+ // [1] Per https://doc.rust-lang.org/1.85.0/std/sync/atomic/struct.AtomicU16.html:
+ //
+ // This type has the same size and bit validity as the underlying
+ // integer type, `u16`.
+ //
+ // [2] Per https://doc.rust-lang.org/1.85.0/std/sync/atomic/struct.AtomicI16.html:
+ //
+ // This type has the same size and bit validity as the underlying
+ // integer type, `i16`.
+ #[allow(clippy::multiple_unsafe_ops_per_block)]
+ const _: () = unsafe {
+ unsafe_impl_transmute_from_for_atomic!(=> AtomicU16 [u16], => AtomicI16 [i16])
+ };
+ }
+
+ #[cfg(target_has_atomic = "32")]
+ #[cfg_attr(doc_cfg, doc(cfg(target_has_atomic = "32")))]
+ mod atomic_32 {
+ use core::sync::atomic::{AtomicI32, AtomicU32};
+
+ use super::*;
+
+ impl_traits_for_atomics!(AtomicU32[u32], AtomicI32[i32]);
+
+ // SAFETY: `AtomicU32` and `AtomicI32` have the same size and bit
+ // validity as `u32` and `i32` respectively [1][2].
+ //
+ // [1] Per https://doc.rust-lang.org/1.85.0/std/sync/atomic/struct.AtomicU32.html:
+ //
+ // This type has the same size and bit validity as the underlying
+ // integer type, `u32`.
+ //
+ // [2] Per https://doc.rust-lang.org/1.85.0/std/sync/atomic/struct.AtomicI32.html:
+ //
+ // This type has the same size and bit validity as the underlying
+ // integer type, `i32`.
+ #[allow(clippy::multiple_unsafe_ops_per_block)]
+ const _: () = unsafe {
+ unsafe_impl_transmute_from_for_atomic!(=> AtomicU32 [u32], => AtomicI32 [i32])
+ };
+ }
+
+ #[cfg(target_has_atomic = "64")]
+ #[cfg_attr(doc_cfg, doc(cfg(target_has_atomic = "64")))]
+ mod atomic_64 {
+ use core::sync::atomic::{AtomicI64, AtomicU64};
+
+ use super::*;
+
+ impl_traits_for_atomics!(AtomicU64[u64], AtomicI64[i64]);
+
+ // SAFETY: `AtomicU64` and `AtomicI64` have the same size and bit
+ // validity as `u64` and `i64` respectively [1][2].
+ //
+ // [1] Per https://doc.rust-lang.org/1.85.0/std/sync/atomic/struct.AtomicU64.html:
+ //
+ // This type has the same size and bit validity as the underlying
+ // integer type, `u64`.
+ //
+ // [2] Per https://doc.rust-lang.org/1.85.0/std/sync/atomic/struct.AtomicI64.html:
+ //
+ // This type has the same size and bit validity as the underlying
+ // integer type, `i64`.
+ #[allow(clippy::multiple_unsafe_ops_per_block)]
+ const _: () = unsafe {
+ unsafe_impl_transmute_from_for_atomic!(=> AtomicU64 [u64], => AtomicI64 [i64])
+ };
+ }
+
+ #[cfg(target_has_atomic = "ptr")]
+ #[cfg_attr(doc_cfg, doc(cfg(target_has_atomic = "ptr")))]
+ mod atomic_ptr {
+ use core::sync::atomic::{AtomicIsize, AtomicPtr, AtomicUsize};
+
+ use super::*;
+
+ impl_traits_for_atomics!(AtomicUsize[usize], AtomicIsize[isize]);
+
+ // FIXME(#170): Implement `FromBytes` and `IntoBytes` once we implement
+ // those traits for `*mut T`.
+ impl_known_layout!(T => AtomicPtr<T>);
+ impl_for_transmute_from!(T => TryFromBytes for AtomicPtr<T> [*mut T]);
+ impl_for_transmute_from!(T => FromZeros for AtomicPtr<T> [*mut T]);
+
+ // SAFETY: `AtomicUsize` and `AtomicIsize` have the same size and bit
+ // validity as `usize` and `isize` respectively [1][2].
+ //
+ // [1] Per https://doc.rust-lang.org/1.85.0/std/sync/atomic/struct.AtomicUsize.html:
+ //
+ // This type has the same size and bit validity as the underlying
+ // integer type, `usize`.
+ //
+ // [2] Per https://doc.rust-lang.org/1.85.0/std/sync/atomic/struct.AtomicIsize.html:
+ //
+ // This type has the same size and bit validity as the underlying
+ // integer type, `isize`.
+ #[allow(clippy::multiple_unsafe_ops_per_block)]
+ const _: () = unsafe {
+ unsafe_impl_transmute_from_for_atomic!(=> AtomicUsize [usize], => AtomicIsize [isize])
+ };
+
+ // SAFETY: Per
+ // https://doc.rust-lang.org/1.85.0/std/sync/atomic/struct.AtomicPtr.html:
+ //
+ // This type has the same size and bit validity as a `*mut T`.
+ #[allow(clippy::multiple_unsafe_ops_per_block)]
+ const _: () = unsafe { unsafe_impl_transmute_from_for_atomic!(T => AtomicPtr<T> [*mut T]) };
+ }
+}
+
+// SAFETY: Per reference [1]: "For all T, the following are guaranteed:
+// size_of::<PhantomData<T>>() == 0 align_of::<PhantomData<T>>() == 1". This
+// gives:
+// - `Immutable`: `PhantomData` has no fields.
+// - `TryFromBytes` (with no validator), `FromZeros`, `FromBytes`: There is only
+// one possible sequence of 0 bytes, and `PhantomData` is inhabited.
+// - `IntoBytes`: Since `PhantomData` has size 0, it contains no padding bytes.
+// - `Unaligned`: Per the preceding reference, `PhantomData` has alignment 1.
+//
+// [1] https://doc.rust-lang.org/1.81.0/std/marker/struct.PhantomData.html#layout-1
+#[allow(clippy::multiple_unsafe_ops_per_block)]
+const _: () = unsafe {
+ unsafe_impl!(T: ?Sized => Immutable for PhantomData<T>);
+ unsafe_impl!(T: ?Sized => TryFromBytes for PhantomData<T>);
+ unsafe_impl!(T: ?Sized => FromZeros for PhantomData<T>);
+ unsafe_impl!(T: ?Sized => FromBytes for PhantomData<T>);
+ unsafe_impl!(T: ?Sized => IntoBytes for PhantomData<T>);
+ unsafe_impl!(T: ?Sized => Unaligned for PhantomData<T>);
+ assert_unaligned!(PhantomData<()>, PhantomData<u8>, PhantomData<u64>);
+};
+
+impl_for_transmute_from!(T: TryFromBytes => TryFromBytes for Wrapping<T>[T]);
+impl_for_transmute_from!(T: FromZeros => FromZeros for Wrapping<T>[T]);
+impl_for_transmute_from!(T: FromBytes => FromBytes for Wrapping<T>[T]);
+impl_for_transmute_from!(T: IntoBytes => IntoBytes for Wrapping<T>[T]);
+assert_unaligned!(Wrapping<()>, Wrapping<u8>);
+
+// SAFETY: Per [1], `Wrapping<T>` has the same layout as `T`. Since its single
+// field (of type `T`) is public, it would be a breaking change to add or remove
+// fields. Thus, we know that `Wrapping<T>` contains a `T` (as opposed to just
+// having the same size and alignment as `T`) with no pre- or post-padding.
+// Thus, `Wrapping<T>` must have `UnsafeCell`s covering the same byte ranges as
+// `Inner = T`.
+//
+// [1] Per https://doc.rust-lang.org/1.81.0/std/num/struct.Wrapping.html#layout-1:
+//
+// `Wrapping<T>` is guaranteed to have the same layout and ABI as `T`
+const _: () = unsafe { unsafe_impl!(T: Immutable => Immutable for Wrapping<T>) };
+
+// SAFETY: Per [1] in the preceding safety comment, `Wrapping<T>` has the same
+// alignment as `T`.
+const _: () = unsafe { unsafe_impl!(T: Unaligned => Unaligned for Wrapping<T>) };
+
+// SAFETY: `TryFromBytes` (with no validator), `FromZeros`, `FromBytes`:
+// `MaybeUninit<T>` has no restrictions on its contents.
+#[allow(clippy::multiple_unsafe_ops_per_block)]
+const _: () = unsafe {
+ unsafe_impl!(T => TryFromBytes for CoreMaybeUninit<T>);
+ unsafe_impl!(T => FromZeros for CoreMaybeUninit<T>);
+ unsafe_impl!(T => FromBytes for CoreMaybeUninit<T>);
+};
+
+// SAFETY: `MaybeUninit<T>` has `UnsafeCell`s covering the same byte ranges as
+// `Inner = T`. This is not explicitly documented, but it can be inferred. Per
+// [1], `MaybeUninit<T>` has the same size as `T`. Further, note the signature
+// of `MaybeUninit::assume_init_ref` [2]:
+//
+// pub unsafe fn assume_init_ref(&self) -> &T
+//
+// If the argument `&MaybeUninit<T>` and the returned `&T` had `UnsafeCell`s at
+// different offsets, this would be unsound. Its existence is proof that this is
+// not the case.
+//
+// [1] Per https://doc.rust-lang.org/1.81.0/std/mem/union.MaybeUninit.html#layout-1:
+//
+// `MaybeUninit<T>` is guaranteed to have the same size, alignment, and ABI as
+// `T`.
+//
+// [2] https://doc.rust-lang.org/1.81.0/std/mem/union.MaybeUninit.html#method.assume_init_ref
+const _: () = unsafe { unsafe_impl!(T: Immutable => Immutable for CoreMaybeUninit<T>) };
+
+// SAFETY: Per [1] in the preceding safety comment, `MaybeUninit<T>` has the
+// same alignment as `T`.
+const _: () = unsafe { unsafe_impl!(T: Unaligned => Unaligned for CoreMaybeUninit<T>) };
+assert_unaligned!(CoreMaybeUninit<()>, CoreMaybeUninit<u8>);
+
+// SAFETY: `ManuallyDrop<T>` has the same layout as `T` [1]. This strongly
+// implies, but does not guarantee, that it contains `UnsafeCell`s covering the
+// same byte ranges as in `T`. However, it also implements `Defer<Target = T>`
+// [2], which provides the ability to convert `&ManuallyDrop<T> -> &T`. This,
+// combined with having the same size as `T`, implies that `ManuallyDrop<T>`
+// exactly contains a `T` with the same fields and `UnsafeCell`s covering the
+// same byte ranges, or else the `Deref` impl would permit safe code to obtain
+// different shared references to the same region of memory with different
+// `UnsafeCell` coverage, which would in turn permit interior mutation that
+// would violate the invariants of a shared reference.
+//
+// [1] Per https://doc.rust-lang.org/1.85.0/std/mem/struct.ManuallyDrop.html:
+//
+// `ManuallyDrop<T>` is guaranteed to have the same layout and bit validity as
+// `T`
+//
+// [2] https://doc.rust-lang.org/1.85.0/std/mem/struct.ManuallyDrop.html#impl-Deref-for-ManuallyDrop%3CT%3E
+const _: () = unsafe { unsafe_impl!(T: ?Sized + Immutable => Immutable for ManuallyDrop<T>) };
+
+impl_for_transmute_from!(T: ?Sized + TryFromBytes => TryFromBytes for ManuallyDrop<T>[T]);
+impl_for_transmute_from!(T: ?Sized + FromZeros => FromZeros for ManuallyDrop<T>[T]);
+impl_for_transmute_from!(T: ?Sized + FromBytes => FromBytes for ManuallyDrop<T>[T]);
+impl_for_transmute_from!(T: ?Sized + IntoBytes => IntoBytes for ManuallyDrop<T>[T]);
+// SAFETY: `ManuallyDrop<T>` has the same layout as `T` [1], and thus has the
+// same alignment as `T`.
+//
+// [1] Per https://doc.rust-lang.org/1.81.0/std/mem/struct.ManuallyDrop.html:
+//
+// `ManuallyDrop<T>` is guaranteed to have the same layout and bit validity as
+// `T`
+const _: () = unsafe { unsafe_impl!(T: ?Sized + Unaligned => Unaligned for ManuallyDrop<T>) };
+assert_unaligned!(ManuallyDrop<()>, ManuallyDrop<u8>);
+
+const _: () = {
+ #[allow(
+ non_camel_case_types,
+ missing_copy_implementations,
+ missing_debug_implementations,
+ missing_docs
+ )]
+ pub enum value {}
+
+ // SAFETY: See safety comment on `ProjectToTag`.
+ unsafe impl<T: ?Sized> HasTag for ManuallyDrop<T> {
+ #[inline]
+ fn only_derive_is_allowed_to_implement_this_trait()
+ where
+ Self: Sized,
+ {
+ }
+
+ type Tag = ();
+
+ // SAFETY: It is trivially sound to project any pointer to a pointer to
+ // a type of size zero and alignment 1 (which `()` is [1]). Such a
+ // pointer will trivially satisfy its aliasing and validity requirements
+ // (since it has a zero-sized referent), and its alignment requirement
+ // (since it is aligned to 1).
+ //
+ // [1] Per https://doc.rust-lang.org/1.92.0/reference/type-layout.html#r-layout.tuple.unit:
+ //
+ // [T]he unit tuple (`()`)... is guaranteed as a zero-sized type to
+ // have a size of 0 and an alignment of 1.
+ type ProjectToTag = crate::pointer::cast::CastToUnit;
+ }
+
+ // SAFETY: `ManuallyDrop<T>` has a field of type `T` at offset `0` without
+ // any safety invariants beyond those of `T`. Its existence is not
+ // explicitly documented, but it can be inferred; per [1] `ManuallyDrop<T>`
+ // has the same size and bit validity as `T`. This field is not literally
+ // public, but is effectively so; the field can be transparently:
+ //
+ // - initialized via `ManuallyDrop::new`
+ // - moved via `ManuallyDrop::into_inner`
+ // - referenced via `ManuallyDrop::deref`
+ // - exclusively referenced via `ManuallyDrop::deref_mut`
+ //
+ // We call this field `value`, both because that is both the name of this
+ // private field, and because it is the name it is referred to in the public
+ // documentation of `ManuallyDrop::new`, `ManuallyDrop::into_inner`,
+ // `ManuallyDrop::take` and `ManuallyDrop::drop`.
+ unsafe impl<T: ?Sized>
+ HasField<value, { crate::STRUCT_VARIANT_ID }, { crate::ident_id!(value) }>
+ for ManuallyDrop<T>
+ {
+ #[inline]
+ fn only_derive_is_allowed_to_implement_this_trait()
+ where
+ Self: Sized,
+ {
+ }
+
+ type Type = T;
+
+ #[inline(always)]
+ fn project(slf: PtrInner<'_, Self>) -> *mut T {
+ // SAFETY: `ManuallyDrop<T>` has the same layout and bit validity as
+ // `T` [1].
+ //
+ // [1] Per https://doc.rust-lang.org/1.85.0/std/mem/struct.ManuallyDrop.html:
+ //
+ // `ManuallyDrop<T>` is guaranteed to have the same layout and bit
+ // validity as `T`
+ #[allow(clippy::as_conversions)]
+ return slf.as_ptr() as *mut T;
+ }
+ }
+};
+
+impl_for_transmute_from!(T: ?Sized + TryFromBytes => TryFromBytes for Cell<T>[T]);
+impl_for_transmute_from!(T: ?Sized + FromZeros => FromZeros for Cell<T>[T]);
+impl_for_transmute_from!(T: ?Sized + FromBytes => FromBytes for Cell<T>[T]);
+impl_for_transmute_from!(T: ?Sized + IntoBytes => IntoBytes for Cell<T>[T]);
+// SAFETY: `Cell<T>` has the same in-memory representation as `T` [1], and thus
+// has the same alignment as `T`.
+//
+// [1] Per https://doc.rust-lang.org/1.81.0/core/cell/struct.Cell.html#memory-layout:
+//
+// `Cell<T>` has the same in-memory representation as its inner type `T`.
+const _: () = unsafe { unsafe_impl!(T: ?Sized + Unaligned => Unaligned for Cell<T>) };
+
+impl_for_transmute_from!(T: ?Sized + FromZeros => FromZeros for UnsafeCell<T>[T]);
+impl_for_transmute_from!(T: ?Sized + FromBytes => FromBytes for UnsafeCell<T>[T]);
+impl_for_transmute_from!(T: ?Sized + IntoBytes => IntoBytes for UnsafeCell<T>[T]);
+// SAFETY: `UnsafeCell<T>` has the same in-memory representation as `T` [1], and
+// thus has the same alignment as `T`.
+//
+// [1] Per https://doc.rust-lang.org/1.81.0/core/cell/struct.UnsafeCell.html#memory-layout:
+//
+// `UnsafeCell<T>` has the same in-memory representation as its inner type
+// `T`.
+const _: () = unsafe { unsafe_impl!(T: ?Sized + Unaligned => Unaligned for UnsafeCell<T>) };
+assert_unaligned!(UnsafeCell<()>, UnsafeCell<u8>);
+
+// SAFETY: See safety comment in `is_bit_valid` impl.
+unsafe impl<T: TryFromBytes + ?Sized> TryFromBytes for UnsafeCell<T> {
+ #[allow(clippy::missing_inline_in_public_items)]
+ fn only_derive_is_allowed_to_implement_this_trait()
+ where
+ Self: Sized,
+ {
+ }
+
+ #[inline(always)]
+ fn is_bit_valid<A>(candidate: Maybe<'_, Self, A>) -> bool
+ where
+ A: invariant::Alignment,
+ {
+ T::is_bit_valid(candidate.transmute::<_, _, BecauseImmutable>())
+ }
+}
+
+// SAFETY: Per the reference [1]:
+//
+// An array of `[T; N]` has a size of `size_of::<T>() * N` and the same
+// alignment of `T`. Arrays are laid out so that the zero-based `nth` element
+// of the array is offset from the start of the array by `n * size_of::<T>()`
+// bytes.
+//
+// ...
+//
+// Slices have the same layout as the section of the array they slice.
+//
+// In other words, the layout of a `[T]` or `[T; N]` is a sequence of `T`s laid
+// out back-to-back with no bytes in between. Therefore, `[T]` or `[T; N]` are
+// `Immutable`, `TryFromBytes`, `FromZeros`, `FromBytes`, and `IntoBytes` if `T`
+// is (respectively). Furthermore, since an array/slice has "the same alignment
+// of `T`", `[T]` and `[T; N]` are `Unaligned` if `T` is.
+//
+// Note that we don't `assert_unaligned!` for slice types because
+// `assert_unaligned!` uses `align_of`, which only works for `Sized` types.
+//
+// [1] https://doc.rust-lang.org/1.81.0/reference/type-layout.html#array-layout
+#[allow(clippy::multiple_unsafe_ops_per_block)]
+const _: () = unsafe {
+ unsafe_impl!(const N: usize, T: Immutable => Immutable for [T; N]);
+ unsafe_impl!(const N: usize, T: TryFromBytes => TryFromBytes for [T; N]; |c| {
+ let c: Ptr<'_, [ReadOnly<T>; N], _> = c.cast::<_, crate::pointer::cast::CastSized, _>();
+ let c: Ptr<'_, [ReadOnly<T>], _> = c.as_slice();
+ let c: Ptr<'_, ReadOnly<[T]>, _> = c.cast::<_, crate::pointer::cast::CastUnsized, _>();
+
+ // Note that this call may panic, but it would still be sound even if it
+ // did. `is_bit_valid` does not promise that it will not panic (in fact,
+ // it explicitly warns that it's a possibility), and we have not
+ // violated any safety invariants that we must fix before returning.
+ <[T] as TryFromBytes>::is_bit_valid(c)
+ });
+ unsafe_impl!(const N: usize, T: FromZeros => FromZeros for [T; N]);
+ unsafe_impl!(const N: usize, T: FromBytes => FromBytes for [T; N]);
+ unsafe_impl!(const N: usize, T: IntoBytes => IntoBytes for [T; N]);
+ unsafe_impl!(const N: usize, T: Unaligned => Unaligned for [T; N]);
+ assert_unaligned!([(); 0], [(); 1], [u8; 0], [u8; 1]);
+ unsafe_impl!(T: Immutable => Immutable for [T]);
+ unsafe_impl!(T: TryFromBytes => TryFromBytes for [T]; |c| {
+ let c: Ptr<'_, [ReadOnly<T>], _> = c.cast::<_, crate::pointer::cast::CastUnsized, _>();
+
+ // SAFETY: Per the reference [1]:
+ //
+ // An array of `[T; N]` has a size of `size_of::<T>() * N` and the
+ // same alignment of `T`. Arrays are laid out so that the zero-based
+ // `nth` element of the array is offset from the start of the array by
+ // `n * size_of::<T>()` bytes.
+ //
+ // ...
+ //
+ // Slices have the same layout as the section of the array they slice.
+ //
+ // In other words, the layout of a `[T] is a sequence of `T`s laid out
+ // back-to-back with no bytes in between. If all elements in `candidate`
+ // are `is_bit_valid`, so too is `candidate`.
+ //
+ // Note that any of the below calls may panic, but it would still be
+ // sound even if it did. `is_bit_valid` does not promise that it will
+ // not panic (in fact, it explicitly warns that it's a possibility), and
+ // we have not violated any safety invariants that we must fix before
+ // returning.
+ c.iter().all(<T as TryFromBytes>::is_bit_valid)
+ });
+ unsafe_impl!(T: FromZeros => FromZeros for [T]);
+ unsafe_impl!(T: FromBytes => FromBytes for [T]);
+ unsafe_impl!(T: IntoBytes => IntoBytes for [T]);
+ unsafe_impl!(T: Unaligned => Unaligned for [T]);
+};
+
+// SAFETY:
+// - `Immutable`: Raw pointers do not contain any `UnsafeCell`s.
+// - `FromZeros`: For thin pointers (note that `T: Sized`), the zero pointer is
+// considered "null". [1] No operations which require provenance are legal on
+// null pointers, so this is not a footgun.
+// - `TryFromBytes`: By the same reasoning as for `FromZeroes`, we can implement
+// `TryFromBytes` for thin pointers provided that
+// [`TryFromByte::is_bit_valid`] only produces `true` for zeroed bytes.
+//
+// NOTE(#170): Implementing `FromBytes` and `IntoBytes` for raw pointers would
+// be sound, but carries provenance footguns. We want to support `FromBytes` and
+// `IntoBytes` for raw pointers eventually, but we are holding off until we can
+// figure out how to address those footguns.
+//
+// [1] Per https://doc.rust-lang.org/1.81.0/std/ptr/fn.null.html:
+//
+// Creates a null raw pointer.
+//
+// This function is equivalent to zero-initializing the pointer:
+// `MaybeUninit::<*const T>::zeroed().assume_init()`.
+//
+// The resulting pointer has the address 0.
+#[allow(clippy::multiple_unsafe_ops_per_block)]
+const _: () = unsafe {
+ unsafe_impl!(T: ?Sized => Immutable for *const T);
+ unsafe_impl!(T: ?Sized => Immutable for *mut T);
+ unsafe_impl!(T => TryFromBytes for *const T; |c| pointer::is_zeroed(c));
+ unsafe_impl!(T => FromZeros for *const T);
+ unsafe_impl!(T => TryFromBytes for *mut T; |c| pointer::is_zeroed(c));
+ unsafe_impl!(T => FromZeros for *mut T);
+};
+
+// SAFETY: `NonNull<T>` self-evidently does not contain `UnsafeCell`s. This is
+// not a proof, but we are accepting this as a known risk per #1358.
+const _: () = unsafe { unsafe_impl!(T: ?Sized => Immutable for NonNull<T>) };
+
+// SAFETY: Reference types do not contain any `UnsafeCell`s.
+#[allow(clippy::multiple_unsafe_ops_per_block)]
+const _: () = unsafe {
+ unsafe_impl!(T: ?Sized => Immutable for &'_ T);
+ unsafe_impl!(T: ?Sized => Immutable for &'_ mut T);
+};
+
+// SAFETY: `Option` is not `#[non_exhaustive]` [1], which means that the types
+// in its variants cannot change, and no new variants can be added. `Option<T>`
+// does not contain any `UnsafeCell`s outside of `T`. [1]
+//
+// [1] https://doc.rust-lang.org/core/option/enum.Option.html
+const _: () = unsafe { unsafe_impl!(T: Immutable => Immutable for Option<T>) };
+
+mod tuples {
+ use super::*;
+
+ /// Generates various trait implementations for tuples.
+ ///
+ /// # Safety
+ ///
+ /// `impl_tuple!` should be provided name-number pairs, where each number is
+ /// the ordinal of the preceding type name.
+ macro_rules! impl_tuple {
+ // Entry point.
+ ($($T:ident $I:tt),+ $(,)?) => {
+ crate::util::macros::__unsafe();
+ impl_tuple!(@all [] [$($T $I)+]);
+ };
+
+ // Build up the set of tuple types (i.e., `(A,)`, `(A, B)`, `(A, B, C)`,
+ // etc.) Trait implementations that do not depend on field index may be
+ // added to this branch.
+ (@all [$($head_T:ident $head_I:tt)*] [$next_T:ident $next_I:tt $($tail:tt)*]) => {
+ // SAFETY: If all fields of the tuple `Self` are `Immutable`, so too is `Self`.
+ unsafe_impl!($($head_T: Immutable,)* $next_T: Immutable => Immutable for ($($head_T,)* $next_T,));
+
+ // SAFETY: If all fields in `c` are `is_bit_valid`, so too is `c`.
+ unsafe_impl!($($head_T: TryFromBytes,)* $next_T: TryFromBytes => TryFromBytes for ($($head_T,)* $next_T,); |c| {
+ let mut c = c;
+ $(TryFromBytes::is_bit_valid(into_inner!(c.reborrow().project::<_, { crate::STRUCT_VARIANT_ID }, { crate::ident_id!($head_I) }>())) &&)*
+ TryFromBytes::is_bit_valid(into_inner!(c.reborrow().project::<_, { crate::STRUCT_VARIANT_ID }, { crate::ident_id!($next_I) }>()))
+ });
+
+ // SAFETY: If all fields in `Self` are `FromZeros`, so too is `Self`.
+ unsafe_impl!($($head_T: FromZeros,)* $next_T: FromZeros => FromZeros for ($($head_T,)* $next_T,));
+
+ // SAFETY: If all fields in `Self` are `FromBytes`, so too is `Self`.
+ unsafe_impl!($($head_T: FromBytes,)* $next_T: FromBytes => FromBytes for ($($head_T,)* $next_T,));
+
+ // SAFETY: See safety comment on `ProjectToTag`.
+ unsafe impl<$($head_T,)* $next_T> crate::HasTag for ($($head_T,)* $next_T,) {
+ #[inline]
+ fn only_derive_is_allowed_to_implement_this_trait()
+ where
+ Self: Sized
+ {}
+
+ type Tag = ();
+
+ // SAFETY: It is trivially sound to project any pointer to a
+ // pointer to a type of size zero and alignment 1 (which `()` is
+ // [1]). Such a pointer will trivially satisfy its aliasing and
+ // validity requirements (since it has a zero-sized referent),
+ // and its alignment requirement (since it is aligned to 1).
+ //
+ // [1] Per https://doc.rust-lang.org/1.92.0/reference/type-layout.html#r-layout.tuple.unit:
+ //
+ // [T]he unit tuple (`()`)... is guaranteed as a zero-sized
+ // type to have a size of 0 and an alignment of 1.
+ type ProjectToTag = crate::pointer::cast::CastToUnit;
+ }
+
+ // Generate impls that depend on tuple index.
+ impl_tuple!(@variants
+ [$($head_T $head_I)* $next_T $next_I]
+ []
+ [$($head_T $head_I)* $next_T $next_I]
+ );
+
+ // Recurse to next tuple size
+ impl_tuple!(@all [$($head_T $head_I)* $next_T $next_I] [$($tail)*]);
+ };
+ (@all [$($head_T:ident $head_I:tt)*] []) => {};
+
+ // Emit trait implementations that depend on field index.
+ (@variants
+ // The full tuple definition in type–index pairs.
+ [$($AllT:ident $AllI:tt)+]
+ // Types before the current index.
+ [$($BeforeT:ident)*]
+ // The types and indices at and after the current index.
+ [$CurrT:ident $CurrI:tt $($AfterT:ident $AfterI:tt)*]
+ ) => {
+ // SAFETY:
+ // - `Self` is a struct (albeit anonymous), so `VARIANT_ID` is
+ // `STRUCT_VARIANT_ID`.
+ // - `$CurrI` is the field at index `$CurrI`, so `FIELD_ID` is
+ // `zerocopy::ident_id!($CurrI)`
+ // - `()` has the same visibility as the `.$CurrI` field (ie, `.0`,
+ // `.1`, etc)
+ // - `Type` has the same type as `$CurrI`; i.e., `$CurrT`.
+ unsafe impl<$($AllT),+> crate::HasField<
+ (),
+ { crate::STRUCT_VARIANT_ID },
+ { crate::ident_id!($CurrI)}
+ > for ($($AllT,)+) {
+ #[inline]
+ fn only_derive_is_allowed_to_implement_this_trait()
+ where
+ Self: Sized
+ {}
+
+ type Type = $CurrT;
+
+ #[inline(always)]
+ fn project(slf: crate::PtrInner<'_, Self>) -> *mut Self::Type {
+ let slf = slf.as_non_null().as_ptr();
+ // SAFETY: `PtrInner` promises it references either a zero-sized
+ // byte range, or else will reference a byte range that is
+ // entirely contained within an allocated object. In either
+ // case, this guarantees that `(*slf).$CurrI` is in-bounds of
+ // `slf`.
+ unsafe { core::ptr::addr_of_mut!((*slf).$CurrI) }
+ }
+ }
+
+ // SAFETY: See comments on items.
+ unsafe impl<Aliasing, Alignment, $($AllT),+> crate::ProjectField<
+ (),
+ (Aliasing, Alignment, crate::invariant::Uninit),
+ { crate::STRUCT_VARIANT_ID },
+ { crate::ident_id!($CurrI)}
+ > for ($($AllT,)+)
+ where
+ Aliasing: crate::invariant::Aliasing,
+ Alignment: crate::invariant::Alignment,
+ {
+ #[inline]
+ fn only_derive_is_allowed_to_implement_this_trait()
+ where
+ Self: Sized
+ {}
+
+ // SAFETY: Tuples are product types whose fields are
+ // well-aligned, so projection preserves both the alignment and
+ // validity invariants of the outer pointer.
+ type Invariants = (Aliasing, Alignment, crate::invariant::Uninit);
+
+ // SAFETY: Tuples are product types and so projection is infallible;
+ type Error = core::convert::Infallible;
+ }
+
+ // SAFETY: See comments on items.
+ unsafe impl<Aliasing, Alignment, $($AllT),+> crate::ProjectField<
+ (),
+ (Aliasing, Alignment, crate::invariant::Initialized),
+ { crate::STRUCT_VARIANT_ID },
+ { crate::ident_id!($CurrI)}
+ > for ($($AllT,)+)
+ where
+ Aliasing: crate::invariant::Aliasing,
+ Alignment: crate::invariant::Alignment,
+ {
+ #[inline]
+ fn only_derive_is_allowed_to_implement_this_trait()
+ where
+ Self: Sized
+ {}
+
+ // SAFETY: Tuples are product types whose fields are
+ // well-aligned, so projection preserves both the alignment and
+ // validity invariants of the outer pointer.
+ type Invariants = (Aliasing, Alignment, crate::invariant::Initialized);
+
+ // SAFETY: Tuples are product types and so projection is infallible;
+ type Error = core::convert::Infallible;
+ }
+
+ // SAFETY: See comments on items.
+ unsafe impl<Aliasing, Alignment, $($AllT),+> crate::ProjectField<
+ (),
+ (Aliasing, Alignment, crate::invariant::Valid),
+ { crate::STRUCT_VARIANT_ID },
+ { crate::ident_id!($CurrI)}
+ > for ($($AllT,)+)
+ where
+ Aliasing: crate::invariant::Aliasing,
+ Alignment: crate::invariant::Alignment,
+ {
+ #[inline]
+ fn only_derive_is_allowed_to_implement_this_trait()
+ where
+ Self: Sized
+ {}
+
+ // SAFETY: Tuples are product types whose fields are
+ // well-aligned, so projection preserves both the alignment and
+ // validity invariants of the outer pointer.
+ type Invariants = (Aliasing, Alignment, crate::invariant::Valid);
+
+ // SAFETY: Tuples are product types and so projection is infallible;
+ type Error = core::convert::Infallible;
+ }
+
+ // Recurse to the next index.
+ impl_tuple!(@variants [$($AllT $AllI)+] [$($BeforeT)* $CurrT] [$($AfterT $AfterI)*]);
+ };
+ (@variants [$($AllT:ident $AllI:tt)+] [$($BeforeT:ident)*] []) => {};
+ }
+
+ // SAFETY: `impl_tuple` is provided name-number pairs, where number is the
+ // ordinal of the name.
+ #[allow(clippy::multiple_unsafe_ops_per_block)]
+ const _: () = unsafe {
+ impl_tuple! {
+ A 0,
+ B 1,
+ C 2,
+ D 3,
+ E 4,
+ F 5,
+ G 6,
+ H 7,
+ I 8,
+ J 9,
+ K 10,
+ L 11,
+ M 12,
+ N 13,
+ O 14,
+ P 15,
+ Q 16,
+ R 17,
+ S 18,
+ T 19,
+ U 20,
+ V 21,
+ W 22,
+ X 23,
+ Y 24,
+ Z 25,
+ };
+ };
+}
+
+// SIMD support
+//
+// Per the Unsafe Code Guidelines Reference [1]:
+//
+// Packed SIMD vector types are `repr(simd)` homogeneous tuple-structs
+// containing `N` elements of type `T` where `N` is a power-of-two and the
+// size and alignment requirements of `T` are equal:
+//
+// ```rust
+// #[repr(simd)]
+// struct Vector<T, N>(T_0, ..., T_(N - 1));
+// ```
+//
+// ...
+//
+// The size of `Vector` is `N * size_of::<T>()` and its alignment is an
+// implementation-defined function of `T` and `N` greater than or equal to
+// `align_of::<T>()`.
+//
+// ...
+//
+// Vector elements are laid out in source field order, enabling random access
+// to vector elements by reinterpreting the vector as an array:
+//
+// ```rust
+// union U {
+// vec: Vector<T, N>,
+// arr: [T; N]
+// }
+//
+// assert_eq!(size_of::<Vector<T, N>>(), size_of::<[T; N]>());
+// assert!(align_of::<Vector<T, N>>() >= align_of::<[T; N]>());
+//
+// unsafe {
+// let u = U { vec: Vector<T, N>(t_0, ..., t_(N - 1)) };
+//
+// assert_eq!(u.vec.0, u.arr[0]);
+// // ...
+// assert_eq!(u.vec.(N - 1), u.arr[N - 1]);
+// }
+// ```
+//
+// Given this background, we can observe that:
+// - The size and bit pattern requirements of a SIMD type are equivalent to the
+// equivalent array type. Thus, for any SIMD type whose primitive `T` is
+// `Immutable`, `TryFromBytes`, `FromZeros`, `FromBytes`, or `IntoBytes`, that
+// SIMD type is also `Immutable`, `TryFromBytes`, `FromZeros`, `FromBytes`, or
+// `IntoBytes` respectively.
+// - Since no upper bound is placed on the alignment, no SIMD type can be
+// guaranteed to be `Unaligned`.
+//
+// Also per [1]:
+//
+// This chapter represents the consensus from issue #38. The statements in
+// here are not (yet) "guaranteed" not to change until an RFC ratifies them.
+//
+// See issue #38 [2]. While this behavior is not technically guaranteed, the
+// likelihood that the behavior will change such that SIMD types are no longer
+// `TryFromBytes`, `FromZeros`, `FromBytes`, or `IntoBytes` is next to zero, as
+// that would defeat the entire purpose of SIMD types. Nonetheless, we put this
+// behavior behind the `simd` Cargo feature, which requires consumers to opt
+// into this stability hazard.
+//
+// [1] https://rust-lang.github.io/unsafe-code-guidelines/layout/packed-simd-vectors.html
+// [2] https://github.com/rust-lang/unsafe-code-guidelines/issues/38
+#[cfg(feature = "simd")]
+#[cfg_attr(doc_cfg, doc(cfg(feature = "simd")))]
+mod simd {
+ /// Defines a module which implements `TryFromBytes`, `FromZeros`,
+ /// `FromBytes`, and `IntoBytes` for a set of types from a module in
+ /// `core::arch`.
+ ///
+ /// `$arch` is both the name of the defined module and the name of the
+ /// module in `core::arch`, and `$typ` is the list of items from that module
+ /// to implement `FromZeros`, `FromBytes`, and `IntoBytes` for.
+ #[allow(unused_macros)] // `allow(unused_macros)` is needed because some
+ // target/feature combinations don't emit any impls
+ // and thus don't use this macro.
+ macro_rules! simd_arch_mod {
+ ($(#[cfg $cfg:tt])* $(#[cfg_attr $cfg_attr:tt])? $arch:ident, $mod:ident, $($typ:ident),*) => {
+ $(#[cfg $cfg])*
+ #[cfg_attr(doc_cfg, doc(cfg $($cfg)*))]
+ $(#[cfg_attr $cfg_attr])?
+ mod $mod {
+ use core::arch::$arch::{$($typ),*};
+
+ use crate::*;
+ impl_known_layout!($($typ),*);
+ // SAFETY: See comment on module definition for justification.
+ #[allow(clippy::multiple_unsafe_ops_per_block)]
+ const _: () = unsafe {
+ $( unsafe_impl!($typ: Immutable, TryFromBytes, FromZeros, FromBytes, IntoBytes); )*
+ };
+ }
+ };
+ }
+
+ #[rustfmt::skip]
+ const _: () = {
+ simd_arch_mod!(
+ #[cfg(target_arch = "x86")]
+ x86, x86, __m128, __m128d, __m128i, __m256, __m256d, __m256i
+ );
+ #[cfg(not(no_zerocopy_simd_x86_avx12_1_89_0))]
+ simd_arch_mod!(
+ #[cfg(target_arch = "x86")]
+ #[cfg_attr(doc_cfg, doc(cfg(rust = "1.89.0")))]
+ x86, x86_nightly, __m512bh, __m512, __m512d, __m512i
+ );
+ simd_arch_mod!(
+ #[cfg(target_arch = "x86_64")]
+ x86_64, x86_64, __m128, __m128d, __m128i, __m256, __m256d, __m256i
+ );
+ #[cfg(not(no_zerocopy_simd_x86_avx12_1_89_0))]
+ simd_arch_mod!(
+ #[cfg(target_arch = "x86_64")]
+ #[cfg_attr(doc_cfg, doc(cfg(rust = "1.89.0")))]
+ x86_64, x86_64_nightly, __m512bh, __m512, __m512d, __m512i
+ );
+ simd_arch_mod!(
+ #[cfg(target_arch = "wasm32")]
+ wasm32, wasm32, v128
+ );
+ simd_arch_mod!(
+ #[cfg(all(feature = "simd-nightly", target_arch = "powerpc"))]
+ powerpc, powerpc, vector_bool_long, vector_double, vector_signed_long, vector_unsigned_long
+ );
+ simd_arch_mod!(
+ #[cfg(all(feature = "simd-nightly", target_arch = "powerpc64"))]
+ powerpc64, powerpc64, vector_bool_long, vector_double, vector_signed_long, vector_unsigned_long
+ );
+ // NOTE: NEON intrinsics were broken on big-endian platforms from their stabilization up to
+ // Rust 1.87. (Context in https://github.com/rust-lang/stdarch/issues/1484). Support is
+ // split in two different version ranges on top of the base configuration, requiring either
+ // little endian or the more recent version to be detected as well.
+ #[cfg(not(no_zerocopy_aarch64_simd_1_59_0))]
+ simd_arch_mod!(
+ #[cfg(all(
+ target_arch = "aarch64",
+ any(
+ target_endian = "little",
+ not(no_zerocopy_aarch64_simd_be_1_87_0)
+ )
+ ))]
+ #[cfg_attr(
+ doc_cfg,
+ doc(cfg(all(target_arch = "aarch64", any(
+ all(rust = "1.59.0", target_endian = "little"),
+ rust = "1.87.0",
+ ))))
+ )]
+ aarch64, aarch64, float32x2_t, float32x4_t, float64x1_t, float64x2_t, int8x8_t, int8x8x2_t,
+ int8x8x3_t, int8x8x4_t, int8x16_t, int8x16x2_t, int8x16x3_t, int8x16x4_t, int16x4_t,
+ int16x8_t, int32x2_t, int32x4_t, int64x1_t, int64x2_t, poly8x8_t, poly8x8x2_t, poly8x8x3_t,
+ poly8x8x4_t, poly8x16_t, poly8x16x2_t, poly8x16x3_t, poly8x16x4_t, poly16x4_t, poly16x8_t,
+ poly64x1_t, poly64x2_t, uint8x8_t, uint8x8x2_t, uint8x8x3_t, uint8x8x4_t, uint8x16_t,
+ uint8x16x2_t, uint8x16x3_t, uint8x16x4_t, uint16x4_t, uint16x4x2_t, uint16x4x3_t,
+ uint16x4x4_t, uint16x8_t, uint32x2_t, uint32x4_t, uint64x1_t, uint64x2_t
+ );
+ };
+}
+
+#[cfg(test)]
+mod tests {
+ use super::*;
+
+ #[test]
+ fn test_impls() {
+ // A type that can supply test cases for testing
+ // `TryFromBytes::is_bit_valid`. All types passed to `assert_impls!`
+ // must implement this trait; that macro uses it to generate runtime
+ // tests for `TryFromBytes` impls.
+ //
+ // All `T: FromBytes` types are provided with a blanket impl. Other
+ // types must implement `TryFromBytesTestable` directly (ie using
+ // `impl_try_from_bytes_testable!`).
+ trait TryFromBytesTestable {
+ fn with_passing_test_cases<F: Fn(Box<ReadOnly<Self>>)>(f: F);
+ fn with_failing_test_cases<F: Fn(&mut [u8])>(f: F);
+ }
+
+ impl<T: FromBytes> TryFromBytesTestable for T {
+ fn with_passing_test_cases<F: Fn(Box<ReadOnly<Self>>)>(f: F) {
+ // Test with a zeroed value.
+ f(ReadOnly::<Self>::new_box_zeroed().unwrap());
+
+ let ffs = {
+ let mut t = ReadOnly::new(Self::new_zeroed());
+ let ptr: *mut T = ReadOnly::as_mut(&mut t);
+ // SAFETY: `T: FromBytes`
+ unsafe { ptr::write_bytes(ptr.cast::<u8>(), 0xFF, mem::size_of::<T>()) };
+ t
+ };
+
+ // Test with a value initialized with 0xFF.
+ f(Box::new(ffs));
+ }
+
+ fn with_failing_test_cases<F: Fn(&mut [u8])>(_f: F) {}
+ }
+
+ macro_rules! impl_try_from_bytes_testable_for_null_pointer_optimization {
+ ($($tys:ty),*) => {
+ $(
+ impl TryFromBytesTestable for Option<$tys> {
+ fn with_passing_test_cases<F: Fn(Box<ReadOnly<Self>>)>(f: F) {
+ // Test with a zeroed value.
+ f(Box::new(ReadOnly::new(None)));
+ }
+
+ fn with_failing_test_cases<F: Fn(&mut [u8])>(f: F) {
+ for pos in 0..mem::size_of::<Self>() {
+ let mut bytes = [0u8; mem::size_of::<Self>()];
+ bytes[pos] = 0x01;
+ f(&mut bytes[..]);
+ }
+ }
+ }
+ )*
+ };
+ }
+
+ // Implements `TryFromBytesTestable`.
+ macro_rules! impl_try_from_bytes_testable {
+ // Base case for recursion (when the list of types has run out).
+ (=> @success $($success_case:expr),* $(, @failure $($failure_case:expr),*)?) => {};
+ // Implements for type(s) with no type parameters.
+ ($ty:ty $(,$tys:ty)* => @success $($success_case:expr),* $(, @failure $($failure_case:expr),*)?) => {
+ impl TryFromBytesTestable for $ty {
+ impl_try_from_bytes_testable!(
+ @methods @success $($success_case),*
+ $(, @failure $($failure_case),*)?
+ );
+ }
+ impl_try_from_bytes_testable!($($tys),* => @success $($success_case),* $(, @failure $($failure_case),*)?);
+ };
+ // Implements for multiple types with no type parameters.
+ ($($($ty:ty),* => @success $($success_case:expr), * $(, @failure $($failure_case:expr),*)?;)*) => {
+ $(
+ impl_try_from_bytes_testable!($($ty),* => @success $($success_case),* $(, @failure $($failure_case),*)*);
+ )*
+ };
+ // Implements only the methods; caller must invoke this from inside
+ // an impl block.
+ (@methods @success $($success_case:expr),* $(, @failure $($failure_case:expr),*)?) => {
+ fn with_passing_test_cases<F: Fn(Box<ReadOnly<Self>>)>(_f: F) {
+ $(
+ let bx = Box::<Self>::from($success_case);
+ let ro: Box<ReadOnly<_>> = {
+ let raw = Box::into_raw(bx);
+ // SAFETY: `ReadOnly<T>` has the same layout and bit
+ // validity as `T`.
+ #[allow(clippy::as_conversions)]
+ unsafe { Box::from_raw(raw as *mut _) }
+ };
+ _f(ro);
+ )*
+ }
+
+ fn with_failing_test_cases<F: Fn(&mut [u8])>(_f: F) {
+ $($(
+ let mut case = $failure_case;
+ _f(case.as_mut_bytes());
+ )*)?
+ }
+ };
+ }
+
+ impl_try_from_bytes_testable_for_null_pointer_optimization!(
+ Box<UnsafeCell<NotZerocopy>>,
+ &'static UnsafeCell<NotZerocopy>,
+ &'static mut UnsafeCell<NotZerocopy>,
+ NonNull<UnsafeCell<NotZerocopy>>,
+ fn(),
+ FnManyArgs,
+ extern "C" fn(),
+ ECFnManyArgs
+ );
+
+ macro_rules! bx {
+ ($e:expr) => {
+ Box::new($e)
+ };
+ }
+
+ // Note that these impls are only for types which are not `FromBytes`.
+ // `FromBytes` types are covered by a preceding blanket impl.
+ impl_try_from_bytes_testable!(
+ bool => @success true, false,
+ @failure 2u8, 3u8, 0xFFu8;
+ char => @success '\u{0}', '\u{D7FF}', '\u{E000}', '\u{10FFFF}',
+ @failure 0xD800u32, 0xDFFFu32, 0x110000u32;
+ str => @success "", "hello", "❤️🧡💛💚💙💜",
+ @failure [0, 159, 146, 150];
+ [u8] => @success vec![].into_boxed_slice(), vec![0, 1, 2].into_boxed_slice();
+ NonZeroU8, NonZeroI8, NonZeroU16, NonZeroI16, NonZeroU32,
+ NonZeroI32, NonZeroU64, NonZeroI64, NonZeroU128, NonZeroI128,
+ NonZeroUsize, NonZeroIsize
+ => @success Self::new(1).unwrap(),
+ // Doing this instead of `0` ensures that we always satisfy
+ // the size and alignment requirements of `Self` (whereas `0`
+ // may be any integer type with a different size or alignment
+ // than some `NonZeroXxx` types).
+ @failure Option::<Self>::None;
+ [bool; 0] => @success [];
+ [bool; 1]
+ => @success [true], [false],
+ @failure [2u8], [3u8], [0xFFu8];
+ [bool]
+ => @success vec![true, false].into_boxed_slice(), vec![false, true].into_boxed_slice(),
+ @failure [2u8], [3u8], [0xFFu8], [0u8, 1u8, 2u8];
+ Unalign<bool>
+ => @success Unalign::new(false), Unalign::new(true),
+ @failure 2u8, 0xFFu8;
+ ManuallyDrop<bool>
+ => @success ManuallyDrop::new(false), ManuallyDrop::new(true),
+ @failure 2u8, 0xFFu8;
+ ManuallyDrop<[u8]>
+ => @success bx!(ManuallyDrop::new([])), bx!(ManuallyDrop::new([0u8])), bx!(ManuallyDrop::new([0u8, 1u8]));
+ ManuallyDrop<[bool]>
+ => @success bx!(ManuallyDrop::new([])), bx!(ManuallyDrop::new([false])), bx!(ManuallyDrop::new([false, true])),
+ @failure [2u8], [3u8], [0xFFu8], [0u8, 1u8, 2u8];
+ ManuallyDrop<[UnsafeCell<u8>]>
+ => @success bx!(ManuallyDrop::new([UnsafeCell::new(0)])), bx!(ManuallyDrop::new([UnsafeCell::new(0), UnsafeCell::new(1)]));
+ ManuallyDrop<[UnsafeCell<bool>]>
+ => @success bx!(ManuallyDrop::new([UnsafeCell::new(false)])), bx!(ManuallyDrop::new([UnsafeCell::new(false), UnsafeCell::new(true)])),
+ @failure [2u8], [3u8], [0xFFu8], [0u8, 1u8, 2u8];
+ Wrapping<bool>
+ => @success Wrapping(false), Wrapping(true),
+ @failure 2u8, 0xFFu8;
+ *const NotZerocopy
+ => @success ptr::null::<NotZerocopy>(),
+ @failure [0x01; mem::size_of::<*const NotZerocopy>()];
+ *mut NotZerocopy
+ => @success ptr::null_mut::<NotZerocopy>(),
+ @failure [0x01; mem::size_of::<*mut NotZerocopy>()];
+ );
+
+ // Use the trick described in [1] to allow us to call methods
+ // conditional on certain trait bounds.
+ //
+ // In all of these cases, methods return `Option<R>`, where `R` is the
+ // return type of the method we're conditionally calling. The "real"
+ // implementations (the ones defined in traits using `&self`) return
+ // `Some`, and the default implementations (the ones defined as inherent
+ // methods using `&mut self`) return `None`.
+ //
+ // [1] https://github.com/dtolnay/case-studies/blob/master/autoref-specialization/README.md
+ mod autoref_trick {
+ use super::*;
+
+ pub(super) struct AutorefWrapper<T: ?Sized>(pub(super) PhantomData<T>);
+
+ pub(super) trait TestIsBitValidShared<T: ?Sized> {
+ #[allow(clippy::needless_lifetimes)]
+ fn test_is_bit_valid_shared<'ptr>(&self, candidate: Maybe<'ptr, T>)
+ -> Option<bool>;
+ }
+
+ impl<T: TryFromBytes + Immutable + ?Sized> TestIsBitValidShared<T> for AutorefWrapper<T> {
+ #[allow(clippy::needless_lifetimes)]
+ fn test_is_bit_valid_shared<'ptr>(
+ &self,
+ candidate: Maybe<'ptr, T>,
+ ) -> Option<bool> {
+ Some(T::is_bit_valid(candidate))
+ }
+ }
+
+ pub(super) trait TestTryFromRef<T: ?Sized> {
+ #[allow(clippy::needless_lifetimes)]
+ fn test_try_from_ref<'bytes>(
+ &self,
+ bytes: &'bytes [u8],
+ ) -> Option<Option<&'bytes T>>;
+ }
+
+ impl<T: TryFromBytes + Immutable + KnownLayout + ?Sized> TestTryFromRef<T> for AutorefWrapper<T> {
+ #[allow(clippy::needless_lifetimes)]
+ fn test_try_from_ref<'bytes>(
+ &self,
+ bytes: &'bytes [u8],
+ ) -> Option<Option<&'bytes T>> {
+ Some(T::try_ref_from_bytes(bytes).ok())
+ }
+ }
+
+ pub(super) trait TestTryFromMut<T: ?Sized> {
+ #[allow(clippy::needless_lifetimes)]
+ fn test_try_from_mut<'bytes>(
+ &self,
+ bytes: &'bytes mut [u8],
+ ) -> Option<Option<&'bytes mut T>>;
+ }
+
+ impl<T: TryFromBytes + IntoBytes + KnownLayout + ?Sized> TestTryFromMut<T> for AutorefWrapper<T> {
+ #[allow(clippy::needless_lifetimes)]
+ fn test_try_from_mut<'bytes>(
+ &self,
+ bytes: &'bytes mut [u8],
+ ) -> Option<Option<&'bytes mut T>> {
+ Some(T::try_mut_from_bytes(bytes).ok())
+ }
+ }
+
+ pub(super) trait TestTryReadFrom<T> {
+ fn test_try_read_from(&self, bytes: &[u8]) -> Option<Option<T>>;
+ }
+
+ impl<T: TryFromBytes> TestTryReadFrom<T> for AutorefWrapper<T> {
+ fn test_try_read_from(&self, bytes: &[u8]) -> Option<Option<T>> {
+ Some(T::try_read_from_bytes(bytes).ok())
+ }
+ }
+
+ pub(super) trait TestAsBytes<T: ?Sized> {
+ #[allow(clippy::needless_lifetimes)]
+ fn test_as_bytes<'slf, 't>(&'slf self, t: &'t ReadOnly<T>) -> Option<&'t [u8]>;
+ }
+
+ impl<T: IntoBytes + Immutable + ?Sized> TestAsBytes<T> for AutorefWrapper<T> {
+ #[allow(clippy::needless_lifetimes)]
+ fn test_as_bytes<'slf, 't>(&'slf self, t: &'t ReadOnly<T>) -> Option<&'t [u8]> {
+ Some(t.as_bytes())
+ }
+ }
+ }
+
+ use autoref_trick::*;
+
+ // Asserts that `$ty` is one of a list of types which are allowed to not
+ // provide a "real" implementation for `$fn_name`. Since the
+ // `autoref_trick` machinery fails silently, this allows us to ensure
+ // that the "default" impls are only being used for types which we
+ // expect.
+ //
+ // Note that, since this is a runtime test, it is possible to have an
+ // allowlist which is too restrictive if the function in question is
+ // never called for a particular type. For example, if `as_bytes` is not
+ // supported for a particular type, and so `test_as_bytes` returns
+ // `None`, methods such as `test_try_from_ref` may never be called for
+ // that type. As a result, it's possible that, for example, adding
+ // `as_bytes` support for a type would cause other allowlist assertions
+ // to fail. This means that allowlist assertion failures should not
+ // automatically be taken as a sign of a bug.
+ macro_rules! assert_on_allowlist {
+ ($fn_name:ident($ty:ty) $(: $($tys:ty),*)?) => {{
+ use core::any::TypeId;
+
+ let allowlist: &[TypeId] = &[ $($(TypeId::of::<$tys>()),*)? ];
+ let allowlist_names: &[&str] = &[ $($(stringify!($tys)),*)? ];
+
+ let id = TypeId::of::<$ty>();
+ assert!(allowlist.contains(&id), "{} is not on allowlist for {}: {:?}", stringify!($ty), stringify!($fn_name), allowlist_names);
+ }};
+ }
+
+ // Asserts that `$ty` implements any `$trait` and doesn't implement any
+ // `!$trait`. Note that all `$trait`s must come before any `!$trait`s.
+ //
+ // For `T: TryFromBytes`, uses `TryFromBytesTestable` to test success
+ // and failure cases.
+ macro_rules! assert_impls {
+ ($ty:ty: TryFromBytes) => {
+ // "Default" implementations that match the "real"
+ // implementations defined in the `autoref_trick` module above.
+ #[allow(unused, non_local_definitions)]
+ impl AutorefWrapper<$ty> {
+ #[allow(clippy::needless_lifetimes)]
+ fn test_is_bit_valid_shared<'ptr>(
+ &mut self,
+ candidate: Maybe<'ptr, $ty>,
+ ) -> Option<bool> {
+ assert_on_allowlist!(
+ test_is_bit_valid_shared($ty):
+ ManuallyDrop<UnsafeCell<()>>,
+ ManuallyDrop<[UnsafeCell<u8>]>,
+ ManuallyDrop<[UnsafeCell<bool>]>,
+ CoreMaybeUninit<NotZerocopy>,
+ CoreMaybeUninit<UnsafeCell<()>>,
+ Wrapping<UnsafeCell<()>>
+ );
+
+ None
+ }
+
+ #[allow(clippy::needless_lifetimes)]
+ fn test_try_from_ref<'bytes>(&mut self, _bytes: &'bytes [u8]) -> Option<Option<&'bytes $ty>> {
+ assert_on_allowlist!(
+ test_try_from_ref($ty):
+ ManuallyDrop<[UnsafeCell<bool>]>
+ );
+
+ None
+ }
+
+ #[allow(clippy::needless_lifetimes)]
+ fn test_try_from_mut<'bytes>(&mut self, _bytes: &'bytes mut [u8]) -> Option<Option<&'bytes mut $ty>> {
+ assert_on_allowlist!(
+ test_try_from_mut($ty):
+ Option<Box<UnsafeCell<NotZerocopy>>>,
+ Option<&'static UnsafeCell<NotZerocopy>>,
+ Option<&'static mut UnsafeCell<NotZerocopy>>,
+ Option<NonNull<UnsafeCell<NotZerocopy>>>,
+ Option<fn()>,
+ Option<FnManyArgs>,
+ Option<extern "C" fn()>,
+ Option<ECFnManyArgs>,
+ *const NotZerocopy,
+ *mut NotZerocopy
+ );
+
+ None
+ }
+
+ fn test_try_read_from(&mut self, _bytes: &[u8]) -> Option<Option<&$ty>> {
+ assert_on_allowlist!(
+ test_try_read_from($ty):
+ str,
+ ManuallyDrop<[u8]>,
+ ManuallyDrop<[bool]>,
+ ManuallyDrop<[UnsafeCell<bool>]>,
+ [u8],
+ [bool]
+ );
+
+ None
+ }
+
+ fn test_as_bytes(&mut self, _t: &ReadOnly<$ty>) -> Option<&[u8]> {
+ assert_on_allowlist!(
+ test_as_bytes($ty):
+ Option<&'static UnsafeCell<NotZerocopy>>,
+ Option<&'static mut UnsafeCell<NotZerocopy>>,
+ Option<NonNull<UnsafeCell<NotZerocopy>>>,
+ Option<Box<UnsafeCell<NotZerocopy>>>,
+ Option<fn()>,
+ Option<FnManyArgs>,
+ Option<extern "C" fn()>,
+ Option<ECFnManyArgs>,
+ CoreMaybeUninit<u8>,
+ CoreMaybeUninit<NotZerocopy>,
+ CoreMaybeUninit<UnsafeCell<()>>,
+ ManuallyDrop<UnsafeCell<()>>,
+ ManuallyDrop<[UnsafeCell<u8>]>,
+ ManuallyDrop<[UnsafeCell<bool>]>,
+ Wrapping<UnsafeCell<()>>,
+ *const NotZerocopy,
+ *mut NotZerocopy
+ );
+
+ None
+ }
+ }
+
+ <$ty as TryFromBytesTestable>::with_passing_test_cases(|mut val| {
+ // FIXME(#494): These tests only get exercised for types
+ // which are `IntoBytes`. Once we implement #494, we should
+ // be able to support non-`IntoBytes` types by zeroing
+ // padding.
+
+ // We define `w` and `ww` since, in the case of the inherent
+ // methods, Rust thinks they're both borrowed mutably at the
+ // same time (given how we use them below). If we just
+ // defined a single `w` and used it for multiple operations,
+ // this would conflict.
+ //
+ // We `#[allow(unused_mut]` for the cases where the "real"
+ // impls are used, which take `&self`.
+ #[allow(unused_mut)]
+ let (mut w, mut ww) = (AutorefWrapper::<$ty>(PhantomData), AutorefWrapper::<$ty>(PhantomData));
+
+ let c = Ptr::from_ref(&*val);
+ let c = c.forget_aligned();
+ // SAFETY: FIXME(#899): This is unsound. `$ty` is not
+ // necessarily `IntoBytes`, but that's the corner we've
+ // backed ourselves into by using `Ptr::from_ref`.
+ let c = unsafe { c.assume_initialized() };
+ let res = w.test_is_bit_valid_shared(c);
+ if let Some(res) = res {
+ assert!(res, "{}::is_bit_valid (shared `Ptr`): got false, expected true", stringify!($ty));
+ }
+
+ let c = Ptr::from_mut(&mut *val);
+ let c = c.forget_aligned();
+ // SAFETY: FIXME(#899): This is unsound. `$ty` is not
+ // necessarily `IntoBytes`, but that's the corner we've
+ // backed ourselves into by using `Ptr::from_ref`.
+ let mut c = unsafe { c.assume_initialized() };
+ let res = <$ty as TryFromBytes>::is_bit_valid(c.reborrow_shared());
+ assert!(res, "{}::is_bit_valid (exclusive `Ptr`): got false, expected true", stringify!($ty));
+
+ // `bytes` is `Some(val.as_bytes())` if `$ty: IntoBytes +
+ // Immutable` and `None` otherwise.
+ let bytes = w.test_as_bytes(&*val);
+
+ // The inner closure returns
+ // `Some($ty::try_ref_from_bytes(bytes))` if `$ty:
+ // Immutable` and `None` otherwise.
+ let res = bytes.and_then(|bytes| ww.test_try_from_ref(bytes));
+ if let Some(res) = res {
+ assert!(res.is_some(), "{}::try_ref_from_bytes: got `None`, expected `Some`", stringify!($ty));
+ }
+
+ if let Some(bytes) = bytes {
+ // We need to get a mutable byte slice, and so we clone
+ // into a `Vec`. However, we also need these bytes to
+ // satisfy `$ty`'s alignment requirement, which isn't
+ // guaranteed for `Vec<u8>`. In order to get around
+ // this, we create a `Vec` which is twice as long as we
+ // need. There is guaranteed to be an aligned byte range
+ // of size `size_of_val(val)` within that range.
+ let val = &*val;
+ let size = mem::size_of_val(val);
+ let align = mem::align_of_val(val);
+
+ let mut vec = bytes.to_vec();
+ vec.extend(bytes);
+ let slc = vec.as_slice();
+ let offset = slc.as_ptr().align_offset(align);
+ let bytes_mut = &mut vec.as_mut_slice()[offset..offset+size];
+ bytes_mut.copy_from_slice(bytes);
+
+ let res = ww.test_try_from_mut(bytes_mut);
+ if let Some(res) = res {
+ assert!(res.is_some(), "{}::try_mut_from_bytes: got `None`, expected `Some`", stringify!($ty));
+ }
+ }
+
+ let res = bytes.and_then(|bytes| ww.test_try_read_from(bytes));
+ if let Some(res) = res {
+ assert!(res.is_some(), "{}::try_read_from_bytes: got `None`, expected `Some`", stringify!($ty));
+ }
+ });
+ #[allow(clippy::as_conversions)]
+ <$ty as TryFromBytesTestable>::with_failing_test_cases(|c| {
+ #[allow(unused_mut)] // For cases where the "real" impls are used, which take `&self`.
+ let mut w = AutorefWrapper::<$ty>(PhantomData);
+
+ // This is `Some($ty::try_ref_from_bytes(c))` if `$ty:
+ // Immutable` and `None` otherwise.
+ let res = w.test_try_from_ref(c);
+ if let Some(res) = res {
+ assert!(res.is_none(), "{}::try_ref_from_bytes({:?}): got Some, expected None", stringify!($ty), c);
+ }
+
+ let res = w.test_try_from_mut(c);
+ if let Some(res) = res {
+ assert!(res.is_none(), "{}::try_mut_from_bytes({:?}): got Some, expected None", stringify!($ty), c);
+ }
+
+
+ let res = w.test_try_read_from(c);
+ if let Some(res) = res {
+ assert!(res.is_none(), "{}::try_read_from_bytes({:?}): got Some, expected None", stringify!($ty), c);
+ }
+ });
+
+ #[allow(dead_code)]
+ const _: () = { static_assertions::assert_impl_all!($ty: TryFromBytes); };
+ };
+ ($ty:ty: $trait:ident) => {
+ #[allow(dead_code)]
+ const _: () = { static_assertions::assert_impl_all!($ty: $trait); };
+ };
+ ($ty:ty: !$trait:ident) => {
+ #[allow(dead_code)]
+ const _: () = { static_assertions::assert_not_impl_any!($ty: $trait); };
+ };
+ ($ty:ty: $($trait:ident),* $(,)? $(!$negative_trait:ident),*) => {
+ $(
+ assert_impls!($ty: $trait);
+ )*
+
+ $(
+ assert_impls!($ty: !$negative_trait);
+ )*
+ };
+ }
+
+ // NOTE: The negative impl assertions here are not necessarily
+ // prescriptive. They merely serve as change detectors to make sure
+ // we're aware of what trait impls are getting added with a given
+ // change. Of course, some impls would be invalid (e.g., `bool:
+ // FromBytes`), and so this change detection is very important.
+
+ assert_impls!(
+ (): KnownLayout,
+ Immutable,
+ TryFromBytes,
+ FromZeros,
+ FromBytes,
+ IntoBytes,
+ Unaligned
+ );
+ assert_impls!(
+ u8: KnownLayout,
+ Immutable,
+ TryFromBytes,
+ FromZeros,
+ FromBytes,
+ IntoBytes,
+ Unaligned
+ );
+ assert_impls!(
+ i8: KnownLayout,
+ Immutable,
+ TryFromBytes,
+ FromZeros,
+ FromBytes,
+ IntoBytes,
+ Unaligned
+ );
+ assert_impls!(
+ u16: KnownLayout,
+ Immutable,
+ TryFromBytes,
+ FromZeros,
+ FromBytes,
+ IntoBytes,
+ !Unaligned
+ );
+ assert_impls!(
+ i16: KnownLayout,
+ Immutable,
+ TryFromBytes,
+ FromZeros,
+ FromBytes,
+ IntoBytes,
+ !Unaligned
+ );
+ assert_impls!(
+ u32: KnownLayout,
+ Immutable,
+ TryFromBytes,
+ FromZeros,
+ FromBytes,
+ IntoBytes,
+ !Unaligned
+ );
+ assert_impls!(
+ i32: KnownLayout,
+ Immutable,
+ TryFromBytes,
+ FromZeros,
+ FromBytes,
+ IntoBytes,
+ !Unaligned
+ );
+ assert_impls!(
+ u64: KnownLayout,
+ Immutable,
+ TryFromBytes,
+ FromZeros,
+ FromBytes,
+ IntoBytes,
+ !Unaligned
+ );
+ assert_impls!(
+ i64: KnownLayout,
+ Immutable,
+ TryFromBytes,
+ FromZeros,
+ FromBytes,
+ IntoBytes,
+ !Unaligned
+ );
+ assert_impls!(
+ u128: KnownLayout,
+ Immutable,
+ TryFromBytes,
+ FromZeros,
+ FromBytes,
+ IntoBytes,
+ !Unaligned
+ );
+ assert_impls!(
+ i128: KnownLayout,
+ Immutable,
+ TryFromBytes,
+ FromZeros,
+ FromBytes,
+ IntoBytes,
+ !Unaligned
+ );
+ assert_impls!(
+ usize: KnownLayout,
+ Immutable,
+ TryFromBytes,
+ FromZeros,
+ FromBytes,
+ IntoBytes,
+ !Unaligned
+ );
+ assert_impls!(
+ isize: KnownLayout,
+ Immutable,
+ TryFromBytes,
+ FromZeros,
+ FromBytes,
+ IntoBytes,
+ !Unaligned
+ );
+ #[cfg(feature = "float-nightly")]
+ assert_impls!(
+ f16: KnownLayout,
+ Immutable,
+ TryFromBytes,
+ FromZeros,
+ FromBytes,
+ IntoBytes,
+ !Unaligned
+ );
+ assert_impls!(
+ f32: KnownLayout,
+ Immutable,
+ TryFromBytes,
+ FromZeros,
+ FromBytes,
+ IntoBytes,
+ !Unaligned
+ );
+ assert_impls!(
+ f64: KnownLayout,
+ Immutable,
+ TryFromBytes,
+ FromZeros,
+ FromBytes,
+ IntoBytes,
+ !Unaligned
+ );
+ #[cfg(feature = "float-nightly")]
+ assert_impls!(
+ f128: KnownLayout,
+ Immutable,
+ TryFromBytes,
+ FromZeros,
+ FromBytes,
+ IntoBytes,
+ !Unaligned
+ );
+ assert_impls!(
+ bool: KnownLayout,
+ Immutable,
+ TryFromBytes,
+ FromZeros,
+ IntoBytes,
+ Unaligned,
+ !FromBytes
+ );
+ assert_impls!(
+ char: KnownLayout,
+ Immutable,
+ TryFromBytes,
+ FromZeros,
+ IntoBytes,
+ !FromBytes,
+ !Unaligned
+ );
+ assert_impls!(
+ str: KnownLayout,
+ Immutable,
+ TryFromBytes,
+ FromZeros,
+ IntoBytes,
+ Unaligned,
+ !FromBytes
+ );
+
+ assert_impls!(
+ NonZeroU8: KnownLayout,
+ Immutable,
+ TryFromBytes,
+ IntoBytes,
+ Unaligned,
+ !FromZeros,
+ !FromBytes
+ );
+ assert_impls!(
+ NonZeroI8: KnownLayout,
+ Immutable,
+ TryFromBytes,
+ IntoBytes,
+ Unaligned,
+ !FromZeros,
+ !FromBytes
+ );
+ assert_impls!(
+ NonZeroU16: KnownLayout,
+ Immutable,
+ TryFromBytes,
+ IntoBytes,
+ !FromBytes,
+ !Unaligned
+ );
+ assert_impls!(
+ NonZeroI16: KnownLayout,
+ Immutable,
+ TryFromBytes,
+ IntoBytes,
+ !FromBytes,
+ !Unaligned
+ );
+ assert_impls!(
+ NonZeroU32: KnownLayout,
+ Immutable,
+ TryFromBytes,
+ IntoBytes,
+ !FromBytes,
+ !Unaligned
+ );
+ assert_impls!(
+ NonZeroI32: KnownLayout,
+ Immutable,
+ TryFromBytes,
+ IntoBytes,
+ !FromBytes,
+ !Unaligned
+ );
+ assert_impls!(
+ NonZeroU64: KnownLayout,
+ Immutable,
+ TryFromBytes,
+ IntoBytes,
+ !FromBytes,
+ !Unaligned
+ );
+ assert_impls!(
+ NonZeroI64: KnownLayout,
+ Immutable,
+ TryFromBytes,
+ IntoBytes,
+ !FromBytes,
+ !Unaligned
+ );
+ assert_impls!(
+ NonZeroU128: KnownLayout,
+ Immutable,
+ TryFromBytes,
+ IntoBytes,
+ !FromBytes,
+ !Unaligned
+ );
+ assert_impls!(
+ NonZeroI128: KnownLayout,
+ Immutable,
+ TryFromBytes,
+ IntoBytes,
+ !FromBytes,
+ !Unaligned
+ );
+ assert_impls!(
+ NonZeroUsize: KnownLayout,
+ Immutable,
+ TryFromBytes,
+ IntoBytes,
+ !FromBytes,
+ !Unaligned
+ );
+ assert_impls!(
+ NonZeroIsize: KnownLayout,
+ Immutable,
+ TryFromBytes,
+ IntoBytes,
+ !FromBytes,
+ !Unaligned
+ );
+
+ assert_impls!(Option<NonZeroU8>: KnownLayout, Immutable, TryFromBytes, FromZeros, FromBytes, IntoBytes, Unaligned);
+ assert_impls!(Option<NonZeroI8>: KnownLayout, Immutable, TryFromBytes, FromZeros, FromBytes, IntoBytes, Unaligned);
+ assert_impls!(Option<NonZeroU16>: KnownLayout, Immutable, TryFromBytes, FromZeros, FromBytes, IntoBytes, !Unaligned);
+ assert_impls!(Option<NonZeroI16>: KnownLayout, Immutable, TryFromBytes, FromZeros, FromBytes, IntoBytes, !Unaligned);
+ assert_impls!(Option<NonZeroU32>: KnownLayout, Immutable, TryFromBytes, FromZeros, FromBytes, IntoBytes, !Unaligned);
+ assert_impls!(Option<NonZeroI32>: KnownLayout, Immutable, TryFromBytes, FromZeros, FromBytes, IntoBytes, !Unaligned);
+ assert_impls!(Option<NonZeroU64>: KnownLayout, Immutable, TryFromBytes, FromZeros, FromBytes, IntoBytes, !Unaligned);
+ assert_impls!(Option<NonZeroI64>: KnownLayout, Immutable, TryFromBytes, FromZeros, FromBytes, IntoBytes, !Unaligned);
+ assert_impls!(Option<NonZeroU128>: KnownLayout, Immutable, TryFromBytes, FromZeros, FromBytes, IntoBytes, !Unaligned);
+ assert_impls!(Option<NonZeroI128>: KnownLayout, Immutable, TryFromBytes, FromZeros, FromBytes, IntoBytes, !Unaligned);
+ assert_impls!(Option<NonZeroUsize>: KnownLayout, Immutable, TryFromBytes, FromZeros, FromBytes, IntoBytes, !Unaligned);
+ assert_impls!(Option<NonZeroIsize>: KnownLayout, Immutable, TryFromBytes, FromZeros, FromBytes, IntoBytes, !Unaligned);
+
+ // Implements none of the ZC traits.
+ struct NotZerocopy;
+
+ #[rustfmt::skip]
+ type FnManyArgs = fn(
+ NotZerocopy, u8, u8, u8, u8, u8, u8, u8, u8, u8, u8, u8,
+ ) -> (NotZerocopy, NotZerocopy);
+
+ // Allowed, because we're not actually using this type for FFI.
+ #[allow(improper_ctypes_definitions)]
+ #[rustfmt::skip]
+ type ECFnManyArgs = extern "C" fn(
+ NotZerocopy, u8, u8, u8, u8, u8, u8, u8, u8, u8, u8, u8,
+ ) -> (NotZerocopy, NotZerocopy);
+
+ #[cfg(feature = "alloc")]
+ assert_impls!(Option<Box<UnsafeCell<NotZerocopy>>>: KnownLayout, Immutable, TryFromBytes, FromZeros, !FromBytes, !IntoBytes, !Unaligned);
+ assert_impls!(Option<Box<[UnsafeCell<NotZerocopy>]>>: KnownLayout, !Immutable, !TryFromBytes, !FromZeros, !FromBytes, !IntoBytes, !Unaligned);
+ assert_impls!(Option<&'static UnsafeCell<NotZerocopy>>: KnownLayout, Immutable, TryFromBytes, FromZeros, !FromBytes, !IntoBytes, !Unaligned);
+ assert_impls!(Option<&'static [UnsafeCell<NotZerocopy>]>: KnownLayout, Immutable, !TryFromBytes, !FromZeros, !FromBytes, !IntoBytes, !Unaligned);
+ assert_impls!(Option<&'static mut UnsafeCell<NotZerocopy>>: KnownLayout, Immutable, TryFromBytes, FromZeros, !FromBytes, !IntoBytes, !Unaligned);
+ assert_impls!(Option<&'static mut [UnsafeCell<NotZerocopy>]>: KnownLayout, Immutable, !TryFromBytes, !FromZeros, !FromBytes, !IntoBytes, !Unaligned);
+ assert_impls!(Option<NonNull<UnsafeCell<NotZerocopy>>>: KnownLayout, TryFromBytes, FromZeros, Immutable, !FromBytes, !IntoBytes, !Unaligned);
+ assert_impls!(Option<NonNull<[UnsafeCell<NotZerocopy>]>>: KnownLayout, Immutable, !TryFromBytes, !FromZeros, !FromBytes, !IntoBytes, !Unaligned);
+ assert_impls!(Option<fn()>: KnownLayout, Immutable, TryFromBytes, FromZeros, !FromBytes, !IntoBytes, !Unaligned);
+ assert_impls!(Option<FnManyArgs>: KnownLayout, Immutable, TryFromBytes, FromZeros, !FromBytes, !IntoBytes, !Unaligned);
+ assert_impls!(Option<extern "C" fn()>: KnownLayout, Immutable, TryFromBytes, FromZeros, !FromBytes, !IntoBytes, !Unaligned);
+ assert_impls!(Option<ECFnManyArgs>: KnownLayout, Immutable, TryFromBytes, FromZeros, !FromBytes, !IntoBytes, !Unaligned);
+
+ assert_impls!(PhantomData<NotZerocopy>: KnownLayout, Immutable, TryFromBytes, FromZeros, FromBytes, IntoBytes, Unaligned);
+ assert_impls!(PhantomData<UnsafeCell<()>>: KnownLayout, Immutable, TryFromBytes, FromZeros, FromBytes, IntoBytes, Unaligned);
+ assert_impls!(PhantomData<[u8]>: KnownLayout, Immutable, TryFromBytes, FromZeros, FromBytes, IntoBytes, Unaligned);
+
+ assert_impls!(ManuallyDrop<u8>: KnownLayout, Immutable, TryFromBytes, FromZeros, FromBytes, IntoBytes, Unaligned);
+ // This test is important because it allows us to test our hand-rolled
+ // implementation of `<ManuallyDrop<T> as TryFromBytes>::is_bit_valid`.
+ assert_impls!(ManuallyDrop<bool>: KnownLayout, Immutable, TryFromBytes, FromZeros, IntoBytes, Unaligned, !FromBytes);
+ assert_impls!(ManuallyDrop<[u8]>: KnownLayout, Immutable, TryFromBytes, FromZeros, FromBytes, IntoBytes, Unaligned);
+ // This test is important because it allows us to test our hand-rolled
+ // implementation of `<ManuallyDrop<T> as TryFromBytes>::is_bit_valid`.
+ assert_impls!(ManuallyDrop<[bool]>: KnownLayout, Immutable, TryFromBytes, FromZeros, IntoBytes, Unaligned, !FromBytes);
+ assert_impls!(ManuallyDrop<NotZerocopy>: !Immutable, !TryFromBytes, !KnownLayout, !FromZeros, !FromBytes, !IntoBytes, !Unaligned);
+ assert_impls!(ManuallyDrop<[NotZerocopy]>: KnownLayout, !Immutable, !TryFromBytes, !FromZeros, !FromBytes, !IntoBytes, !Unaligned);
+ assert_impls!(ManuallyDrop<UnsafeCell<()>>: KnownLayout, TryFromBytes, FromZeros, FromBytes, IntoBytes, Unaligned, !Immutable);
+ assert_impls!(ManuallyDrop<[UnsafeCell<u8>]>: KnownLayout, TryFromBytes, FromZeros, FromBytes, IntoBytes, Unaligned, !Immutable);
+ assert_impls!(ManuallyDrop<[UnsafeCell<bool>]>: KnownLayout, TryFromBytes, FromZeros, IntoBytes, Unaligned, !Immutable, !FromBytes);
+
+ assert_impls!(CoreMaybeUninit<u8>: KnownLayout, Immutable, TryFromBytes, FromZeros, FromBytes, Unaligned, !IntoBytes);
+ assert_impls!(CoreMaybeUninit<NotZerocopy>: KnownLayout, TryFromBytes, FromZeros, FromBytes, !Immutable, !IntoBytes, !Unaligned);
+ assert_impls!(CoreMaybeUninit<UnsafeCell<()>>: KnownLayout, TryFromBytes, FromZeros, FromBytes, Unaligned, !Immutable, !IntoBytes);
+
+ assert_impls!(Wrapping<u8>: KnownLayout, Immutable, TryFromBytes, FromZeros, FromBytes, IntoBytes, Unaligned);
+ // This test is important because it allows us to test our hand-rolled
+ // implementation of `<Wrapping<T> as TryFromBytes>::is_bit_valid`.
+ assert_impls!(Wrapping<bool>: KnownLayout, Immutable, TryFromBytes, FromZeros, IntoBytes, Unaligned, !FromBytes);
+ assert_impls!(Wrapping<NotZerocopy>: KnownLayout, !Immutable, !TryFromBytes, !FromZeros, !FromBytes, !IntoBytes, !Unaligned);
+ assert_impls!(Wrapping<UnsafeCell<()>>: KnownLayout, TryFromBytes, FromZeros, FromBytes, IntoBytes, Unaligned, !Immutable);
+
+ assert_impls!(Unalign<u8>: KnownLayout, Immutable, TryFromBytes, FromZeros, FromBytes, IntoBytes, Unaligned);
+ // This test is important because it allows us to test our hand-rolled
+ // implementation of `<Unalign<T> as TryFromBytes>::is_bit_valid`.
+ assert_impls!(Unalign<bool>: KnownLayout, Immutable, TryFromBytes, FromZeros, IntoBytes, Unaligned, !FromBytes);
+ assert_impls!(Unalign<NotZerocopy>: KnownLayout, Unaligned, !Immutable, !TryFromBytes, !FromZeros, !FromBytes, !IntoBytes);
+
+ assert_impls!(
+ [u8]: KnownLayout,
+ Immutable,
+ TryFromBytes,
+ FromZeros,
+ FromBytes,
+ IntoBytes,
+ Unaligned
+ );
+ assert_impls!(
+ [bool]: KnownLayout,
+ Immutable,
+ TryFromBytes,
+ FromZeros,
+ IntoBytes,
+ Unaligned,
+ !FromBytes
+ );
+ assert_impls!([NotZerocopy]: KnownLayout, !Immutable, !TryFromBytes, !FromZeros, !FromBytes, !IntoBytes, !Unaligned);
+ assert_impls!(
+ [u8; 0]: KnownLayout,
+ Immutable,
+ TryFromBytes,
+ FromZeros,
+ FromBytes,
+ IntoBytes,
+ Unaligned,
+ );
+ assert_impls!(
+ [NotZerocopy; 0]: KnownLayout,
+ !Immutable,
+ !TryFromBytes,
+ !FromZeros,
+ !FromBytes,
+ !IntoBytes,
+ !Unaligned
+ );
+ assert_impls!(
+ [u8; 1]: KnownLayout,
+ Immutable,
+ TryFromBytes,
+ FromZeros,
+ FromBytes,
+ IntoBytes,
+ Unaligned,
+ );
+ assert_impls!(
+ [NotZerocopy; 1]: KnownLayout,
+ !Immutable,
+ !TryFromBytes,
+ !FromZeros,
+ !FromBytes,
+ !IntoBytes,
+ !Unaligned
+ );
+
+ assert_impls!(*const NotZerocopy: KnownLayout, Immutable, TryFromBytes, FromZeros, !FromBytes, !IntoBytes, !Unaligned);
+ assert_impls!(*mut NotZerocopy: KnownLayout, Immutable, TryFromBytes, FromZeros, !FromBytes, !IntoBytes, !Unaligned);
+ assert_impls!(*const [NotZerocopy]: KnownLayout, Immutable, !TryFromBytes, !FromZeros, !FromBytes, !IntoBytes, !Unaligned);
+ assert_impls!(*mut [NotZerocopy]: KnownLayout, Immutable, !TryFromBytes, !FromZeros, !FromBytes, !IntoBytes, !Unaligned);
+ assert_impls!(*const dyn Debug: KnownLayout, Immutable, !TryFromBytes, !FromZeros, !FromBytes, !IntoBytes, !Unaligned);
+ assert_impls!(*mut dyn Debug: KnownLayout, Immutable, !TryFromBytes, !FromZeros, !FromBytes, !IntoBytes, !Unaligned);
+
+ #[cfg(feature = "simd")]
+ {
+ #[allow(unused_macros)]
+ macro_rules! test_simd_arch_mod {
+ ($arch:ident, $($typ:ident),*) => {
+ {
+ use core::arch::$arch::{$($typ),*};
+ use crate::*;
+ $( assert_impls!($typ: KnownLayout, Immutable, TryFromBytes, FromZeros, FromBytes, IntoBytes, !Unaligned); )*
+ }
+ };
+ }
+ #[cfg(target_arch = "x86")]
+ test_simd_arch_mod!(x86, __m128, __m128d, __m128i, __m256, __m256d, __m256i);
+
+ #[cfg(all(not(no_zerocopy_simd_x86_avx12_1_89_0), target_arch = "x86"))]
+ test_simd_arch_mod!(x86, __m512bh, __m512, __m512d, __m512i);
+
+ #[cfg(target_arch = "x86_64")]
+ test_simd_arch_mod!(x86_64, __m128, __m128d, __m128i, __m256, __m256d, __m256i);
+
+ #[cfg(all(not(no_zerocopy_simd_x86_avx12_1_89_0), target_arch = "x86_64"))]
+ test_simd_arch_mod!(x86_64, __m512bh, __m512, __m512d, __m512i);
+
+ #[cfg(target_arch = "wasm32")]
+ test_simd_arch_mod!(wasm32, v128);
+
+ #[cfg(all(feature = "simd-nightly", target_arch = "powerpc"))]
+ test_simd_arch_mod!(
+ powerpc,
+ vector_bool_long,
+ vector_double,
+ vector_signed_long,
+ vector_unsigned_long
+ );
+
+ #[cfg(all(feature = "simd-nightly", target_arch = "powerpc64"))]
+ test_simd_arch_mod!(
+ powerpc64,
+ vector_bool_long,
+ vector_double,
+ vector_signed_long,
+ vector_unsigned_long
+ );
+ #[cfg(all(target_arch = "aarch64", not(no_zerocopy_aarch64_simd_1_59_0)))]
+ #[rustfmt::skip]
+ test_simd_arch_mod!(
+ aarch64, float32x2_t, float32x4_t, float64x1_t, float64x2_t, int8x8_t, int8x8x2_t,
+ int8x8x3_t, int8x8x4_t, int8x16_t, int8x16x2_t, int8x16x3_t, int8x16x4_t, int16x4_t,
+ int16x8_t, int32x2_t, int32x4_t, int64x1_t, int64x2_t, poly8x8_t, poly8x8x2_t, poly8x8x3_t,
+ poly8x8x4_t, poly8x16_t, poly8x16x2_t, poly8x16x3_t, poly8x16x4_t, poly16x4_t, poly16x8_t,
+ poly64x1_t, poly64x2_t, uint8x8_t, uint8x8x2_t, uint8x8x3_t, uint8x8x4_t, uint8x16_t,
+ uint8x16x2_t, uint8x16x3_t, uint8x16x4_t, uint16x4_t, uint16x4x2_t, uint16x4x3_t,
+ uint16x4x4_t, uint16x8_t, uint32x2_t, uint32x4_t, uint64x1_t, uint64x2_t
+ );
+ }
+ }
+}
diff --git a/rust/zerocopy/src/layout.rs b/rust/zerocopy/src/layout.rs
new file mode 100644
index 0000000..6015d0f
--- /dev/null
+++ b/rust/zerocopy/src/layout.rs
@@ -0,0 +1,2225 @@
+// SPDX-License-Identifier: (BSD-2-Clause OR Apache-2.0) OR MIT
+
+// Copyright 2024 The Fuchsia Authors
+//
+// Licensed under the 2-Clause BSD License <LICENSE-BSD or
+// https://opensource.org/license/bsd-2-clause>, Apache License, Version 2.0
+// <LICENSE-APACHE or https://www.apache.org/licenses/LICENSE-2.0>, or the MIT
+// license <LICENSE-MIT or https://opensource.org/licenses/MIT>, at your option.
+// This file may not be copied, modified, or distributed except according to
+// those terms.
+
+use core::{mem, num::NonZeroUsize};
+
+use crate::util;
+
+/// The target pointer width, counted in bits.
+const POINTER_WIDTH_BITS: usize = mem::size_of::<usize>() * 8;
+
+/// The layout of a type which might be dynamically-sized.
+///
+/// `DstLayout` describes the layout of sized types, slice types, and "slice
+/// DSTs" - ie, those that are known by the type system to have a trailing slice
+/// (as distinguished from `dyn Trait` types - such types *might* have a
+/// trailing slice type, but the type system isn't aware of it).
+///
+/// Note that `DstLayout` does not have any internal invariants, so no guarantee
+/// is made that a `DstLayout` conforms to any of Rust's requirements regarding
+/// the layout of real Rust types or instances of types.
+#[doc(hidden)]
+#[allow(missing_debug_implementations, missing_copy_implementations)]
+#[cfg_attr(any(kani, test), derive(Debug, PartialEq, Eq))]
+#[derive(Copy, Clone)]
+pub struct DstLayout {
+ pub(crate) align: NonZeroUsize,
+ pub(crate) size_info: SizeInfo,
+ // Is it guaranteed statically (without knowing a value's runtime metadata)
+ // that the top-level type contains no padding? This does *not* apply
+ // recursively - for example, `[(u8, u16)]` has `statically_shallow_unpadded
+ // = true` even though this type likely has padding inside each `(u8, u16)`.
+ pub(crate) statically_shallow_unpadded: bool,
+}
+
+#[cfg_attr(any(kani, test), derive(Debug, PartialEq, Eq))]
+#[derive(Copy, Clone)]
+pub(crate) enum SizeInfo<E = usize> {
+ Sized { size: usize },
+ SliceDst(TrailingSliceLayout<E>),
+}
+
+#[cfg_attr(any(kani, test), derive(Debug, PartialEq, Eq))]
+#[derive(Copy, Clone)]
+pub(crate) struct TrailingSliceLayout<E = usize> {
+ // The offset of the first byte of the trailing slice field. Note that this
+ // is NOT the same as the minimum size of the type. For example, consider
+ // the following type:
+ //
+ // struct Foo {
+ // a: u16,
+ // b: u8,
+ // c: [u8],
+ // }
+ //
+ // In `Foo`, `c` is at byte offset 3. When `c.len() == 0`, `c` is followed
+ // by a padding byte.
+ pub(crate) offset: usize,
+ // The size of the element type of the trailing slice field.
+ pub(crate) elem_size: E,
+}
+
+impl SizeInfo {
+ /// Attempts to create a `SizeInfo` from `Self` in which `elem_size` is a
+ /// `NonZeroUsize`. If `elem_size` is 0, returns `None`.
+ #[allow(unused)]
+ const fn try_to_nonzero_elem_size(&self) -> Option<SizeInfo<NonZeroUsize>> {
+ Some(match *self {
+ SizeInfo::Sized { size } => SizeInfo::Sized { size },
+ SizeInfo::SliceDst(TrailingSliceLayout { offset, elem_size }) => {
+ if let Some(elem_size) = NonZeroUsize::new(elem_size) {
+ SizeInfo::SliceDst(TrailingSliceLayout { offset, elem_size })
+ } else {
+ return None;
+ }
+ }
+ })
+ }
+}
+
+#[doc(hidden)]
+#[derive(Copy, Clone)]
+#[cfg_attr(test, derive(Debug))]
+#[allow(missing_debug_implementations)]
+pub enum CastType {
+ Prefix,
+ Suffix,
+}
+
+#[cfg_attr(test, derive(Debug))]
+pub(crate) enum MetadataCastError {
+ Alignment,
+ Size,
+}
+
+impl DstLayout {
+ /// The minimum possible alignment of a type.
+ const MIN_ALIGN: NonZeroUsize = match NonZeroUsize::new(1) {
+ Some(min_align) => min_align,
+ None => const_unreachable!(),
+ };
+
+ /// The maximum theoretic possible alignment of a type.
+ ///
+ /// For compatibility with future Rust versions, this is defined as the
+ /// maximum power-of-two that fits into a `usize`. See also
+ /// [`DstLayout::CURRENT_MAX_ALIGN`].
+ pub(crate) const THEORETICAL_MAX_ALIGN: NonZeroUsize =
+ match NonZeroUsize::new(1 << (POINTER_WIDTH_BITS - 1)) {
+ Some(max_align) => max_align,
+ None => const_unreachable!(),
+ };
+
+ /// The current, documented max alignment of a type \[1\].
+ ///
+ /// \[1\] Per <https://doc.rust-lang.org/reference/type-layout.html#the-alignment-modifiers>:
+ ///
+ /// The alignment value must be a power of two from 1 up to
+ /// 2<sup>29</sup>.
+ #[cfg(not(kani))]
+ #[cfg(not(target_pointer_width = "16"))]
+ pub(crate) const CURRENT_MAX_ALIGN: NonZeroUsize = match NonZeroUsize::new(1 << 28) {
+ Some(max_align) => max_align,
+ None => const_unreachable!(),
+ };
+
+ #[cfg(not(kani))]
+ #[cfg(target_pointer_width = "16")]
+ pub(crate) const CURRENT_MAX_ALIGN: NonZeroUsize = match NonZeroUsize::new(1 << 15) {
+ Some(max_align) => max_align,
+ None => const_unreachable!(),
+ };
+
+ /// The maximum size of an allocation \[1\].
+ ///
+ /// \[1\] Per <https://doc.rust-lang.org/1.91.1/std/ptr/index.html#allocation>:
+ ///
+ /// For any allocation with base `address`, `size`, and a set of `addresses`,
+ /// the following are guaranteed: [..]
+ ///
+ /// - `size <= isize::MAX`
+ ///
+ #[allow(clippy::as_conversions)]
+ pub(crate) const MAX_SIZE: usize = isize::MAX as usize;
+
+ /// Assumes that this layout lacks static shallow padding.
+ ///
+ /// # Panics
+ ///
+ /// This method does not panic.
+ ///
+ /// # Safety
+ ///
+ /// If `self` describes the size and alignment of type that lacks static
+ /// shallow padding, unsafe code may assume that the result of this method
+ /// accurately reflects the size, alignment, and lack of static shallow
+ /// padding of that type.
+ const fn assume_shallow_unpadded(self) -> Self {
+ Self { statically_shallow_unpadded: true, ..self }
+ }
+
+ /// Constructs a `DstLayout` for a zero-sized type with `repr_align`
+ /// alignment (or 1). If `repr_align` is provided, then it must be a power
+ /// of two.
+ ///
+ /// # Panics
+ ///
+ /// This function panics if the supplied `repr_align` is not a power of two.
+ ///
+ /// # Safety
+ ///
+ /// Unsafe code may assume that the contract of this function is satisfied.
+ #[doc(hidden)]
+ #[must_use]
+ #[inline]
+ pub const fn new_zst(repr_align: Option<NonZeroUsize>) -> DstLayout {
+ let align = match repr_align {
+ Some(align) => align,
+ None => Self::MIN_ALIGN,
+ };
+
+ const_assert!(align.get().is_power_of_two());
+
+ DstLayout {
+ align,
+ size_info: SizeInfo::Sized { size: 0 },
+ statically_shallow_unpadded: true,
+ }
+ }
+
+ /// Constructs a `DstLayout` which describes `T` and assumes `T` may contain
+ /// padding.
+ ///
+ /// # Safety
+ ///
+ /// Unsafe code may assume that `DstLayout` is the correct layout for `T`.
+ #[doc(hidden)]
+ #[must_use]
+ #[inline]
+ pub const fn for_type<T>() -> DstLayout {
+ // SAFETY: `align` is correct by construction. `T: Sized`, and so it is
+ // sound to initialize `size_info` to `SizeInfo::Sized { size }`; the
+ // `size` field is also correct by construction. `unpadded` can safely
+ // default to `false`.
+ DstLayout {
+ align: match NonZeroUsize::new(mem::align_of::<T>()) {
+ Some(align) => align,
+ None => const_unreachable!(),
+ },
+ size_info: SizeInfo::Sized { size: mem::size_of::<T>() },
+ statically_shallow_unpadded: false,
+ }
+ }
+
+ /// Constructs a `DstLayout` which describes a `T` that does not contain
+ /// padding.
+ ///
+ /// # Safety
+ ///
+ /// Unsafe code may assume that `DstLayout` is the correct layout for `T`.
+ #[doc(hidden)]
+ #[must_use]
+ #[inline]
+ pub const fn for_unpadded_type<T>() -> DstLayout {
+ Self::for_type::<T>().assume_shallow_unpadded()
+ }
+
+ /// Constructs a `DstLayout` which describes `[T]`.
+ ///
+ /// # Safety
+ ///
+ /// Unsafe code may assume that `DstLayout` is the correct layout for `[T]`.
+ pub(crate) const fn for_slice<T>() -> DstLayout {
+ // SAFETY: The alignment of a slice is equal to the alignment of its
+ // element type, and so `align` is initialized correctly.
+ //
+ // Since this is just a slice type, there is no offset between the
+ // beginning of the type and the beginning of the slice, so it is
+ // correct to set `offset: 0`. The `elem_size` is correct by
+ // construction. Since `[T]` is a (degenerate case of a) slice DST, it
+ // is correct to initialize `size_info` to `SizeInfo::SliceDst`.
+ DstLayout {
+ align: match NonZeroUsize::new(mem::align_of::<T>()) {
+ Some(align) => align,
+ None => const_unreachable!(),
+ },
+ size_info: SizeInfo::SliceDst(TrailingSliceLayout {
+ offset: 0,
+ elem_size: mem::size_of::<T>(),
+ }),
+ statically_shallow_unpadded: true,
+ }
+ }
+
+ /// Constructs a complete `DstLayout` reflecting a `repr(C)` struct with the
+ /// given alignment modifiers and fields.
+ ///
+ /// This method cannot be used to match the layout of a record with the
+ /// default representation, as that representation is mostly unspecified.
+ ///
+ /// # Safety
+ ///
+ /// For any definition of a `repr(C)` struct, if this method is invoked with
+ /// alignment modifiers and fields corresponding to that definition, the
+ /// resulting `DstLayout` will correctly encode the layout of that struct.
+ ///
+ /// We make no guarantees to the behavior of this method when it is invoked
+ /// with arguments that cannot correspond to a valid `repr(C)` struct.
+ #[must_use]
+ #[inline]
+ pub const fn for_repr_c_struct(
+ repr_align: Option<NonZeroUsize>,
+ repr_packed: Option<NonZeroUsize>,
+ fields: &[DstLayout],
+ ) -> DstLayout {
+ let mut layout = DstLayout::new_zst(repr_align);
+
+ let mut i = 0;
+ #[allow(clippy::arithmetic_side_effects)]
+ while i < fields.len() {
+ #[allow(clippy::indexing_slicing)]
+ let field = fields[i];
+ layout = layout.extend(field, repr_packed);
+ i += 1;
+ }
+
+ layout = layout.pad_to_align();
+
+ // SAFETY: `layout` accurately describes the layout of a `repr(C)`
+ // struct with `repr_align` or `repr_packed` alignment modifications and
+ // the given `fields`. The `layout` is constructed using a sequence of
+ // invocations of `DstLayout::{new_zst,extend,pad_to_align}`. The
+ // documentation of these items vows that invocations in this manner
+ // will accurately describe a type, so long as:
+ //
+ // - that type is `repr(C)`,
+ // - its fields are enumerated in the order they appear,
+ // - the presence of `repr_align` and `repr_packed` are correctly accounted for.
+ //
+ // We respect all three of these preconditions above.
+ layout
+ }
+
+ /// Like `Layout::extend`, this creates a layout that describes a record
+ /// whose layout consists of `self` followed by `next` that includes the
+ /// necessary inter-field padding, but not any trailing padding.
+ ///
+ /// In order to match the layout of a `#[repr(C)]` struct, this method
+ /// should be invoked for each field in declaration order. To add trailing
+ /// padding, call `DstLayout::pad_to_align` after extending the layout for
+ /// all fields. If `self` corresponds to a type marked with
+ /// `repr(packed(N))`, then `repr_packed` should be set to `Some(N)`,
+ /// otherwise `None`.
+ ///
+ /// This method cannot be used to match the layout of a record with the
+ /// default representation, as that representation is mostly unspecified.
+ ///
+ /// # Safety
+ ///
+ /// If a (potentially hypothetical) valid `repr(C)` Rust type begins with
+ /// fields whose layout are `self`, and those fields are immediately
+ /// followed by a field whose layout is `field`, then unsafe code may rely
+ /// on `self.extend(field, repr_packed)` producing a layout that correctly
+ /// encompasses those two components.
+ ///
+ /// We make no guarantees to the behavior of this method if these fragments
+ /// cannot appear in a valid Rust type (e.g., the concatenation of the
+ /// layouts would lead to a size larger than `isize::MAX`).
+ #[doc(hidden)]
+ #[must_use]
+ #[inline]
+ pub const fn extend(self, field: DstLayout, repr_packed: Option<NonZeroUsize>) -> Self {
+ use util::{max, min, padding_needed_for};
+
+ // If `repr_packed` is `None`, there are no alignment constraints, and
+ // the value can be defaulted to `THEORETICAL_MAX_ALIGN`.
+ let max_align = match repr_packed {
+ Some(max_align) => max_align,
+ None => Self::THEORETICAL_MAX_ALIGN,
+ };
+
+ const_assert!(max_align.get().is_power_of_two());
+
+ // We use Kani to prove that this method is robust to future increases
+ // in Rust's maximum allowed alignment. However, if such a change ever
+ // actually occurs, we'd like to be notified via assertion failures.
+ #[cfg(not(kani))]
+ {
+ const_debug_assert!(self.align.get() <= DstLayout::CURRENT_MAX_ALIGN.get());
+ const_debug_assert!(field.align.get() <= DstLayout::CURRENT_MAX_ALIGN.get());
+ if let Some(repr_packed) = repr_packed {
+ const_debug_assert!(repr_packed.get() <= DstLayout::CURRENT_MAX_ALIGN.get());
+ }
+ }
+
+ // The field's alignment is clamped by `repr_packed` (i.e., the
+ // `repr(packed(N))` attribute, if any) [1].
+ //
+ // [1] Per https://doc.rust-lang.org/reference/type-layout.html#the-alignment-modifiers:
+ //
+ // The alignments of each field, for the purpose of positioning
+ // fields, is the smaller of the specified alignment and the alignment
+ // of the field's type.
+ let field_align = min(field.align, max_align);
+
+ // The struct's alignment is the maximum of its previous alignment and
+ // `field_align`.
+ let align = max(self.align, field_align);
+
+ let (interfield_padding, size_info) = match self.size_info {
+ // If the layout is already a DST, we panic; DSTs cannot be extended
+ // with additional fields.
+ SizeInfo::SliceDst(..) => const_panic!("Cannot extend a DST with additional fields."),
+
+ SizeInfo::Sized { size: preceding_size } => {
+ // Compute the minimum amount of inter-field padding needed to
+ // satisfy the field's alignment, and offset of the trailing
+ // field. [1]
+ //
+ // [1] Per https://doc.rust-lang.org/reference/type-layout.html#the-alignment-modifiers:
+ //
+ // Inter-field padding is guaranteed to be the minimum
+ // required in order to satisfy each field's (possibly
+ // altered) alignment.
+ let padding = padding_needed_for(preceding_size, field_align);
+
+ // This will not panic (and is proven to not panic, with Kani)
+ // if the layout components can correspond to a leading layout
+ // fragment of a valid Rust type, but may panic otherwise (e.g.,
+ // combining or aligning the components would create a size
+ // exceeding `isize::MAX`).
+ let offset = match preceding_size.checked_add(padding) {
+ Some(offset) => offset,
+ None => const_panic!("Adding padding to `self`'s size overflows `usize`."),
+ };
+
+ (
+ padding,
+ match field.size_info {
+ SizeInfo::Sized { size: field_size } => {
+ // If the trailing field is sized, the resulting layout
+ // will be sized. Its size will be the sum of the
+ // preceding layout, the size of the new field, and the
+ // size of inter-field padding between the two.
+ //
+ // This will not panic (and is proven with Kani to not
+ // panic) if the layout components can correspond to a
+ // leading layout fragment of a valid Rust type, but may
+ // panic otherwise (e.g., combining or aligning the
+ // components would create a size exceeding
+ // `usize::MAX`).
+ let size = match offset.checked_add(field_size) {
+ Some(size) => size,
+ None => const_panic!("`field` cannot be appended without the total size overflowing `usize`"),
+ };
+ SizeInfo::Sized { size }
+ }
+ SizeInfo::SliceDst(TrailingSliceLayout {
+ offset: trailing_offset,
+ elem_size,
+ }) => {
+ // If the trailing field is dynamically sized, so too
+ // will the resulting layout. The offset of the trailing
+ // slice component is the sum of the offset of the
+ // trailing field and the trailing slice offset within
+ // that field.
+ //
+ // This will not panic (and is proven with Kani to not
+ // panic) if the layout components can correspond to a
+ // leading layout fragment of a valid Rust type, but may
+ // panic otherwise (e.g., combining or aligning the
+ // components would create a size exceeding
+ // `usize::MAX`).
+ let offset = match offset.checked_add(trailing_offset) {
+ Some(offset) => offset,
+ None => const_panic!("`field` cannot be appended without the total size overflowing `usize`"),
+ };
+ SizeInfo::SliceDst(TrailingSliceLayout { offset, elem_size })
+ }
+ },
+ )
+ }
+ };
+
+ let statically_shallow_unpadded = self.statically_shallow_unpadded
+ && field.statically_shallow_unpadded
+ && interfield_padding == 0;
+
+ DstLayout { align, size_info, statically_shallow_unpadded }
+ }
+
+ /// Like `Layout::pad_to_align`, this routine rounds the size of this layout
+ /// up to the nearest multiple of this type's alignment or `repr_packed`
+ /// (whichever is less). This method leaves DST layouts unchanged, since the
+ /// trailing padding of DSTs is computed at runtime.
+ ///
+ /// The accompanying boolean is `true` if the resulting composition of
+ /// fields necessitated static (as opposed to dynamic) padding; otherwise
+ /// `false`.
+ ///
+ /// In order to match the layout of a `#[repr(C)]` struct, this method
+ /// should be invoked after the invocations of [`DstLayout::extend`]. If
+ /// `self` corresponds to a type marked with `repr(packed(N))`, then
+ /// `repr_packed` should be set to `Some(N)`, otherwise `None`.
+ ///
+ /// This method cannot be used to match the layout of a record with the
+ /// default representation, as that representation is mostly unspecified.
+ ///
+ /// # Safety
+ ///
+ /// If a (potentially hypothetical) valid `repr(C)` type begins with fields
+ /// whose layout are `self` followed only by zero or more bytes of trailing
+ /// padding (not included in `self`), then unsafe code may rely on
+ /// `self.pad_to_align(repr_packed)` producing a layout that correctly
+ /// encapsulates the layout of that type.
+ ///
+ /// We make no guarantees to the behavior of this method if `self` cannot
+ /// appear in a valid Rust type (e.g., because the addition of trailing
+ /// padding would lead to a size larger than `isize::MAX`).
+ #[doc(hidden)]
+ #[must_use]
+ #[inline]
+ pub const fn pad_to_align(self) -> Self {
+ use util::padding_needed_for;
+
+ let (static_padding, size_info) = match self.size_info {
+ // For sized layouts, we add the minimum amount of trailing padding
+ // needed to satisfy alignment.
+ SizeInfo::Sized { size: unpadded_size } => {
+ let padding = padding_needed_for(unpadded_size, self.align);
+ let size = match unpadded_size.checked_add(padding) {
+ Some(size) => size,
+ None => const_panic!("Adding padding caused size to overflow `usize`."),
+ };
+ (padding, SizeInfo::Sized { size })
+ }
+ // For DST layouts, trailing padding depends on the length of the
+ // trailing DST and is computed at runtime. This does not alter the
+ // offset or element size of the layout, so we leave `size_info`
+ // unchanged.
+ size_info @ SizeInfo::SliceDst(_) => (0, size_info),
+ };
+
+ let statically_shallow_unpadded = self.statically_shallow_unpadded && static_padding == 0;
+
+ DstLayout { align: self.align, size_info, statically_shallow_unpadded }
+ }
+
+ /// Produces `true` if `self` requires static padding; otherwise `false`.
+ #[must_use]
+ #[inline(always)]
+ pub const fn requires_static_padding(self) -> bool {
+ !self.statically_shallow_unpadded
+ }
+
+ /// Produces `true` if there exists any metadata for which a type of layout
+ /// `self` would require dynamic trailing padding; otherwise `false`.
+ #[must_use]
+ #[inline(always)]
+ pub const fn requires_dynamic_padding(self) -> bool {
+ // A `% self.align.get()` cannot panic, since `align` is non-zero.
+ #[allow(clippy::arithmetic_side_effects)]
+ match self.size_info {
+ SizeInfo::Sized { .. } => false,
+ SizeInfo::SliceDst(trailing_slice_layout) => {
+ // SAFETY: This predicate is formally proved sound by
+ // `proofs::prove_requires_dynamic_padding`.
+ trailing_slice_layout.offset % self.align.get() != 0
+ || trailing_slice_layout.elem_size % self.align.get() != 0
+ }
+ }
+ }
+
+ /// Validates that a cast is sound from a layout perspective.
+ ///
+ /// Validates that the size and alignment requirements of a type with the
+ /// layout described in `self` would not be violated by performing a
+ /// `cast_type` cast from a pointer with address `addr` which refers to a
+ /// memory region of size `bytes_len`.
+ ///
+ /// If the cast is valid, `validate_cast_and_convert_metadata` returns
+ /// `(elems, split_at)`. If `self` describes a dynamically-sized type, then
+ /// `elems` is the maximum number of trailing slice elements for which a
+ /// cast would be valid (for sized types, `elem` is meaningless and should
+ /// be ignored). `split_at` is the index at which to split the memory region
+ /// in order for the prefix (suffix) to contain the result of the cast, and
+ /// in order for the remaining suffix (prefix) to contain the leftover
+ /// bytes.
+ ///
+ /// There are three conditions under which a cast can fail:
+ /// - The smallest possible value for the type is larger than the provided
+ /// memory region
+ /// - A prefix cast is requested, and `addr` does not satisfy `self`'s
+ /// alignment requirement
+ /// - A suffix cast is requested, and `addr + bytes_len` does not satisfy
+ /// `self`'s alignment requirement (as a consequence, since all instances
+ /// of the type are a multiple of its alignment, no size for the type will
+ /// result in a starting address which is properly aligned)
+ ///
+ /// # Safety
+ ///
+ /// The caller may assume that this implementation is correct, and may rely
+ /// on that assumption for the soundness of their code. In particular, the
+ /// caller may assume that, if `validate_cast_and_convert_metadata` returns
+ /// `Some((elems, split_at))`, then:
+ /// - A pointer to the type (for dynamically sized types, this includes
+ /// `elems` as its pointer metadata) describes an object of size `size <=
+ /// bytes_len`
+ /// - If this is a prefix cast:
+ /// - `addr` satisfies `self`'s alignment
+ /// - `size == split_at`
+ /// - If this is a suffix cast:
+ /// - `split_at == bytes_len - size`
+ /// - `addr + split_at` satisfies `self`'s alignment
+ ///
+ /// Note that this method does *not* ensure that a pointer constructed from
+ /// its return values will be a valid pointer. In particular, this method
+ /// does not reason about `isize` overflow, which is a requirement of many
+ /// Rust pointer APIs, and may at some point be determined to be a validity
+ /// invariant of pointer types themselves. This should never be a problem so
+ /// long as the arguments to this method are derived from a known-valid
+ /// pointer (e.g., one derived from a safe Rust reference), but it is
+ /// nonetheless the caller's responsibility to justify that pointer
+ /// arithmetic will not overflow based on a safety argument *other than* the
+ /// mere fact that this method returned successfully.
+ ///
+ /// # Panics
+ ///
+ /// `validate_cast_and_convert_metadata` will panic if `self` describes a
+ /// DST whose trailing slice element is zero-sized.
+ ///
+ /// If `addr + bytes_len` overflows `usize`,
+ /// `validate_cast_and_convert_metadata` may panic, or it may return
+ /// incorrect results. No guarantees are made about when
+ /// `validate_cast_and_convert_metadata` will panic. The caller should not
+ /// rely on `validate_cast_and_convert_metadata` panicking in any particular
+ /// condition, even if `debug_assertions` are enabled.
+ #[allow(unused)]
+ #[inline(always)]
+ pub(crate) const fn validate_cast_and_convert_metadata(
+ &self,
+ addr: usize,
+ bytes_len: usize,
+ cast_type: CastType,
+ ) -> Result<(usize, usize), MetadataCastError> {
+ // `debug_assert!`, but with `#[allow(clippy::arithmetic_side_effects)]`.
+ macro_rules! __const_debug_assert {
+ ($e:expr $(, $msg:expr)?) => {
+ const_debug_assert!({
+ #[allow(clippy::arithmetic_side_effects)]
+ let e = $e;
+ e
+ } $(, $msg)?);
+ };
+ }
+
+ // Note that, in practice, `self` is always a compile-time constant. We
+ // do this check earlier than needed to ensure that we always panic as a
+ // result of bugs in the program (such as calling this function on an
+ // invalid type) instead of allowing this panic to be hidden if the cast
+ // would have failed anyway for runtime reasons (such as a too-small
+ // memory region).
+ //
+ // FIXME(#67): Once our MSRV is 1.65, use let-else:
+ // https://blog.rust-lang.org/2022/11/03/Rust-1.65.0.html#let-else-statements
+ let size_info = match self.size_info.try_to_nonzero_elem_size() {
+ Some(size_info) => size_info,
+ None => const_panic!("attempted to cast to slice type with zero-sized element"),
+ };
+
+ // Precondition
+ __const_debug_assert!(
+ addr.checked_add(bytes_len).is_some(),
+ "`addr` + `bytes_len` > usize::MAX"
+ );
+
+ // Alignment checks go in their own block to avoid introducing variables
+ // into the top-level scope.
+ {
+ // We check alignment for `addr` (for prefix casts) or `addr +
+ // bytes_len` (for suffix casts). For a prefix cast, the correctness
+ // of this check is trivial - `addr` is the address the object will
+ // live at.
+ //
+ // For a suffix cast, we know that all valid sizes for the type are
+ // a multiple of the alignment (and by safety precondition, we know
+ // `DstLayout` may only describe valid Rust types). Thus, a
+ // validly-sized instance which lives at a validly-aligned address
+ // must also end at a validly-aligned address. Thus, if the end
+ // address for a suffix cast (`addr + bytes_len`) is not aligned,
+ // then no valid start address will be aligned either.
+ let offset = match cast_type {
+ CastType::Prefix => 0,
+ CastType::Suffix => bytes_len,
+ };
+
+ // Addition is guaranteed not to overflow because `offset <=
+ // bytes_len`, and `addr + bytes_len <= usize::MAX` is a
+ // precondition of this method. Modulus is guaranteed not to divide
+ // by 0 because `align` is non-zero.
+ #[allow(clippy::arithmetic_side_effects)]
+ if (addr + offset) % self.align.get() != 0 {
+ return Err(MetadataCastError::Alignment);
+ }
+ }
+
+ let (elems, self_bytes) = match size_info {
+ SizeInfo::Sized { size } => {
+ if size > bytes_len {
+ return Err(MetadataCastError::Size);
+ }
+ (0, size)
+ }
+ SizeInfo::SliceDst(TrailingSliceLayout { offset, elem_size }) => {
+ // Calculate the maximum number of bytes that could be consumed
+ // - any number of bytes larger than this will either not be a
+ // multiple of the alignment, or will be larger than
+ // `bytes_len`.
+ let max_total_bytes =
+ util::round_down_to_next_multiple_of_alignment(bytes_len, self.align);
+ // Calculate the maximum number of bytes that could be consumed
+ // by the trailing slice.
+ //
+ // FIXME(#67): Once our MSRV is 1.65, use let-else:
+ // https://blog.rust-lang.org/2022/11/03/Rust-1.65.0.html#let-else-statements
+ let max_slice_and_padding_bytes = match max_total_bytes.checked_sub(offset) {
+ Some(max) => max,
+ // `bytes_len` too small even for 0 trailing slice elements.
+ None => return Err(MetadataCastError::Size),
+ };
+
+ // Calculate the number of elements that fit in
+ // `max_slice_and_padding_bytes`; any remaining bytes will be
+ // considered padding.
+ //
+ // Guaranteed not to divide by zero: `elem_size` is non-zero.
+ #[allow(clippy::arithmetic_side_effects)]
+ let elems = max_slice_and_padding_bytes / elem_size.get();
+ // Guaranteed not to overflow on multiplication: `usize::MAX >=
+ // max_slice_and_padding_bytes >= (max_slice_and_padding_bytes /
+ // elem_size) * elem_size`.
+ //
+ // Guaranteed not to overflow on addition:
+ // - max_slice_and_padding_bytes == max_total_bytes - offset
+ // - elems * elem_size <= max_slice_and_padding_bytes == max_total_bytes - offset
+ // - elems * elem_size + offset <= max_total_bytes <= usize::MAX
+ #[allow(clippy::arithmetic_side_effects)]
+ let without_padding = offset + elems * elem_size.get();
+ // `self_bytes` is equal to the offset bytes plus the bytes
+ // consumed by the trailing slice plus any padding bytes
+ // required to satisfy the alignment. Note that we have computed
+ // the maximum number of trailing slice elements that could fit
+ // in `self_bytes`, so any padding is guaranteed to be less than
+ // the size of an extra element.
+ //
+ // Guaranteed not to overflow:
+ // - By previous comment: without_padding == elems * elem_size +
+ // offset <= max_total_bytes
+ // - By construction, `max_total_bytes` is a multiple of
+ // `self.align`.
+ // - At most, adding padding needed to round `without_padding`
+ // up to the next multiple of the alignment will bring
+ // `self_bytes` up to `max_total_bytes`.
+ #[allow(clippy::arithmetic_side_effects)]
+ let self_bytes =
+ without_padding + util::padding_needed_for(without_padding, self.align);
+ (elems, self_bytes)
+ }
+ };
+
+ __const_debug_assert!(self_bytes <= bytes_len);
+
+ let split_at = match cast_type {
+ CastType::Prefix => self_bytes,
+ // Guaranteed not to underflow:
+ // - In the `Sized` branch, only returns `size` if `size <=
+ // bytes_len`.
+ // - In the `SliceDst` branch, calculates `self_bytes <=
+ // max_toatl_bytes`, which is upper-bounded by `bytes_len`.
+ #[allow(clippy::arithmetic_side_effects)]
+ CastType::Suffix => bytes_len - self_bytes,
+ };
+
+ Ok((elems, split_at))
+ }
+}
+
+pub(crate) use cast_from::CastFrom;
+mod cast_from {
+ use crate::*;
+
+ pub(crate) struct CastFrom<Dst: ?Sized> {
+ _never: core::convert::Infallible,
+ _marker: PhantomData<Dst>,
+ }
+
+ // SAFETY: The implementation of `Project::project` preserves the address
+ // of the referent – it only modifies pointer metadata.
+ unsafe impl<Src, Dst> crate::pointer::cast::Cast<Src, Dst> for CastFrom<Dst>
+ where
+ Src: KnownLayout + ?Sized,
+ Dst: KnownLayout + ?Sized,
+ {
+ }
+
+ // SAFETY: The implementation of `Project::project` preserves the size of
+ // the referent (see inline comments for a more detailed proof of this).
+ unsafe impl<Src, Dst> crate::pointer::cast::CastExact<Src, Dst> for CastFrom<Dst>
+ where
+ Src: KnownLayout + ?Sized,
+ Dst: KnownLayout + ?Sized,
+ {
+ }
+
+ // SAFETY: `project` produces a pointer which refers to the same referent
+ // bytes as its input, or to a subset of them (see inline comments for a
+ // more detailed proof of this). It does this using provenance-preserving
+ // operations.
+ unsafe impl<Src, Dst> crate::pointer::cast::Project<Src, Dst> for CastFrom<Dst>
+ where
+ Src: KnownLayout + ?Sized,
+ Dst: KnownLayout + ?Sized,
+ {
+ /// # PME
+ ///
+ /// Generates a post-monomorphization error if it is not possible to
+ /// implement soundly.
+ //
+ // FIXME(#1817): Support Sized->Unsized and Unsized->Sized casts
+ fn project(src: PtrInner<'_, Src>) -> *mut Dst {
+ /// The parameters required in order to perform a pointer cast from
+ /// `Src` to `Dst`.
+ ///
+ /// These are a compile-time function of the layouts of `Src`
+ /// and `Dst`.
+ ///
+ /// # Safety
+ ///
+ /// `Src`'s alignment must not be smaller than `Dst`'s alignment.
+ struct CastParams<Src: ?Sized, Dst: ?Sized> {
+ inner: CastParamsInner,
+ _src: PhantomData<Src>,
+ _dst: PhantomData<Dst>,
+ }
+
+ #[derive(Copy, Clone)]
+ enum CastParamsInner {
+ // At compile time (specifically, post-monomorphization time),
+ // we need to compute two things:
+ // - Whether, given *any* `*Src`, it is possible to construct a
+ // `*Dst` which addresses the same number of bytes (ie,
+ // whether, for any `Src` pointer metadata, there exists `Dst`
+ // pointer metadata that addresses the same number of bytes)
+ // - If this is possible, any information necessary to perform
+ // the `Src`->`Dst` metadata conversion at runtime.
+ //
+ // Assume that `Src` and `Dst` are slice DSTs, and define:
+ // - `S_OFF = Src::LAYOUT.size_info.offset`
+ // - `S_ELEM = Src::LAYOUT.size_info.elem_size`
+ // - `D_OFF = Dst::LAYOUT.size_info.offset`
+ // - `D_ELEM = Dst::LAYOUT.size_info.elem_size`
+ //
+ // We are trying to solve the following equation:
+ //
+ // D_OFF + d_meta * D_ELEM = S_OFF + s_meta * S_ELEM
+ //
+ // At runtime, we will be attempting to compute `d_meta`, given
+ // `s_meta` (a runtime value) and all other parameters (which
+ // are compile-time values). We can solve like so:
+ //
+ // D_OFF + d_meta * D_ELEM = S_OFF + s_meta * S_ELEM
+ //
+ // d_meta * D_ELEM = S_OFF - D_OFF + s_meta * S_ELEM
+ //
+ // d_meta = (S_OFF - D_OFF + s_meta * S_ELEM)/D_ELEM
+ //
+ // Since `d_meta` will be a `usize`, we need the right-hand side
+ // to be an integer, and this needs to hold for *any* value of
+ // `s_meta` (in order for our conversion to be infallible - ie,
+ // to not have to reject certain values of `s_meta` at runtime).
+ // This means that:
+ //
+ // - `s_meta * S_ELEM` must be a multiple of `D_ELEM`
+ // - Since this must hold for any value of `s_meta`, `S_ELEM`
+ // must be a multiple of `D_ELEM`
+ // - `S_OFF - D_OFF` must be a multiple of `D_ELEM`
+ //
+ // Thus, let `OFFSET_DELTA_ELEMS = (S_OFF - D_OFF)/D_ELEM` and
+ // `ELEM_MULTIPLE = S_ELEM/D_ELEM`. We can rewrite the above
+ // expression as:
+ //
+ // d_meta = (S_OFF - D_OFF + s_meta * S_ELEM)/D_ELEM
+ //
+ // d_meta = OFFSET_DELTA_ELEMS + s_meta * ELEM_MULTIPLE
+ //
+ // Thus, we just need to compute the following and confirm that
+ // they have integer solutions in order to both a) determine
+ // whether infallible `Src` -> `Dst` casts are possible and, b)
+ // pre-compute the parameters necessary to perform those casts
+ // at runtime. These parameters are encapsulated in
+ // `CastParams`, which acts as a witness that such infallible
+ // casts are possible.
+ /// The parameters required in order to perform an
+ /// unsized-to-unsized pointer cast from `Src` to `Dst` as
+ /// described above.
+ ///
+ /// # Safety
+ ///
+ /// `Src` and `Dst` must both be slice DSTs.
+ ///
+ /// `offset_delta_elems` and `elem_multiple` must be valid as
+ /// described above.
+ UnsizedToUnsized { offset_delta_elems: usize, elem_multiple: usize },
+
+ /// The metadata of a `Dst` which has the same size as `Src:
+ /// Sized`.
+ ///
+ /// # Safety
+ ///
+ /// `Src: Sized` and `Dst` must be a slice DST.
+ ///
+ /// A raw `Dst` pointer with metadata `dst_meta` must address
+ /// `size_of::<Src>()` bytes.
+ SizedToUnsized { dst_meta: usize },
+
+ /// The metadata of a `Dst` which has the same size as `Src:
+ /// Sized`.
+ ///
+ /// # Safety
+ ///
+ /// `Src` and `Dst` must both be `Sized` and `size_of::<Src>()
+ /// == size_of::<Dst>()`.
+ SizedToSized,
+ }
+
+ impl<Src: ?Sized, Dst: ?Sized> Copy for CastParams<Src, Dst> {}
+ impl<Src: ?Sized, Dst: ?Sized> Clone for CastParams<Src, Dst> {
+ fn clone(&self) -> Self {
+ *self
+ }
+ }
+
+ impl<Src: ?Sized, Dst: ?Sized> CastParams<Src, Dst> {
+ const fn try_compute(
+ src: &DstLayout,
+ dst: &DstLayout,
+ ) -> Option<CastParams<Src, Dst>> {
+ if src.align.get() < dst.align.get() {
+ return None;
+ }
+
+ let inner = match (src.size_info, dst.size_info) {
+ (
+ SizeInfo::Sized { size: src_size },
+ SizeInfo::Sized { size: dst_size },
+ ) => {
+ if src_size != dst_size {
+ return None;
+ }
+
+ // SAFETY: We checked above that `src_size ==
+ // dst_size`.
+ CastParamsInner::SizedToSized
+ }
+ (SizeInfo::Sized { size: src_size }, SizeInfo::SliceDst(dst)) => {
+ let offset_delta = if let Some(od) = src_size.checked_sub(dst.offset) {
+ od
+ } else {
+ return None;
+ };
+
+ let dst_elem_size = if let Some(e) = NonZeroUsize::new(dst.elem_size) {
+ e
+ } else {
+ return None;
+ };
+
+ // PANICS: `dst_elem_size: NonZeroUsize`, so this won't
+ // divide by zero.
+ #[allow(clippy::arithmetic_side_effects)]
+ let delta_mod_other_elem = offset_delta % dst_elem_size.get();
+
+ if delta_mod_other_elem != 0 {
+ return None;
+ }
+
+ // PANICS: `dst_elem_size: NonZeroUsize`, so this won't
+ // divide by zero.
+ #[allow(clippy::arithmetic_side_effects)]
+ let dst_meta = offset_delta / dst_elem_size.get();
+
+ // SAFETY: The preceding math ensures that a `Dst`
+ // with `dst_meta` addresses `src_size` bytes.
+ CastParamsInner::SizedToUnsized { dst_meta }
+ }
+ (SizeInfo::SliceDst(src), SizeInfo::SliceDst(dst)) => {
+ let offset_delta = if let Some(od) = src.offset.checked_sub(dst.offset)
+ {
+ od
+ } else {
+ return None;
+ };
+
+ let dst_elem_size = if let Some(e) = NonZeroUsize::new(dst.elem_size) {
+ e
+ } else {
+ return None;
+ };
+
+ // PANICS: `dst_elem_size: NonZeroUsize`, so this won't
+ // divide by zero.
+ #[allow(clippy::arithmetic_side_effects)]
+ let delta_mod_other_elem = offset_delta % dst_elem_size.get();
+
+ // PANICS: `dst_elem_size: NonZeroUsize`, so this won't
+ // divide by zero.
+ #[allow(clippy::arithmetic_side_effects)]
+ let elem_remainder = src.elem_size % dst_elem_size.get();
+
+ if delta_mod_other_elem != 0
+ || src.elem_size < dst.elem_size
+ || elem_remainder != 0
+ {
+ return None;
+ }
+
+ // PANICS: `dst_elem_size: NonZeroUsize`, so this won't
+ // divide by zero.
+ #[allow(clippy::arithmetic_side_effects)]
+ let offset_delta_elems = offset_delta / dst_elem_size.get();
+
+ // PANICS: `dst_elem_size: NonZeroUsize`, so this won't
+ // divide by zero.
+ #[allow(clippy::arithmetic_side_effects)]
+ let elem_multiple = src.elem_size / dst_elem_size.get();
+
+ CastParamsInner::UnsizedToUnsized {
+ // SAFETY: We checked above that this is an exact ratio.
+ offset_delta_elems,
+ // SAFETY: We checked above that this is an exact ratio.
+ elem_multiple,
+ }
+ }
+ _ => return None,
+ };
+
+ // SAFETY: We checked above that `src.align >= dst.align`.
+ Some(CastParams { inner, _src: PhantomData, _dst: PhantomData })
+ }
+ }
+
+ impl<Src: KnownLayout + ?Sized, Dst: KnownLayout + ?Sized> CastParams<Src, Dst> {
+ /// # Safety
+ ///
+ /// `src_meta` describes a `Src` whose size is no larger than
+ /// `isize::MAX`.
+ ///
+ /// The returned metadata describes a `Dst` of the same size as
+ /// the original `Src`.
+ #[inline(always)]
+ unsafe fn cast_metadata(
+ self,
+ src_meta: Src::PointerMetadata,
+ ) -> Dst::PointerMetadata {
+ #[allow(unused)]
+ use crate::util::polyfills::*;
+
+ let dst_meta = match self.inner {
+ CastParamsInner::UnsizedToUnsized { offset_delta_elems, elem_multiple } => {
+ let src_meta = src_meta.to_elem_count();
+ #[allow(
+ unstable_name_collisions,
+ clippy::multiple_unsafe_ops_per_block
+ )]
+ // SAFETY: `self` is a witness that the following
+ // equation holds:
+ //
+ // D_OFF + d_meta * D_ELEM = S_OFF + s_meta * S_ELEM
+ //
+ // Since the caller promises that `src_meta` is
+ // valid `Src` metadata, this math will not
+ // overflow, and the returned value will describe a
+ // `Dst` of the same size.
+ unsafe {
+ offset_delta_elems
+ .unchecked_add(src_meta.unchecked_mul(elem_multiple))
+ }
+ }
+ CastParamsInner::SizedToUnsized { dst_meta } => dst_meta,
+ CastParamsInner::SizedToSized => 0,
+ };
+ Dst::PointerMetadata::from_elem_count(dst_meta)
+ }
+ }
+
+ trait Params<Src: ?Sized> {
+ const CAST_PARAMS: CastParams<Src, Self>;
+ }
+
+ impl<Src, Dst> Params<Src> for Dst
+ where
+ Src: KnownLayout + ?Sized,
+ Dst: KnownLayout + ?Sized,
+ {
+ const CAST_PARAMS: CastParams<Src, Dst> =
+ match CastParams::try_compute(&Src::LAYOUT, &Dst::LAYOUT) {
+ Some(params) => params,
+ None => const_panic!(
+ "cannot `transmute_ref!` or `transmute_mut!` between incompatible types"
+ ),
+ };
+ }
+
+ let src_meta = <Src as KnownLayout>::pointer_to_metadata(src.as_ptr());
+ let params = <Dst as Params<Src>>::CAST_PARAMS;
+
+ // SAFETY: `src: PtrInner` guarantees that `src`'s referent is zero
+ // bytes or lives in a single allocation, which means that it is no
+ // larger than `isize::MAX` bytes [1].
+ //
+ // [1] https://doc.rust-lang.org/1.92.0/std/ptr/index.html#allocation
+ let dst_meta = unsafe { params.cast_metadata(src_meta) };
+
+ <Dst as KnownLayout>::raw_from_ptr_len(src.as_non_null().cast(), dst_meta).as_ptr()
+ }
+ }
+}
+
+// FIXME(#67): For some reason, on our MSRV toolchain, this `allow` isn't
+// enforced despite having `#![allow(unknown_lints)]` at the crate root, but
+// putting it here works. Once our MSRV is high enough that this bug has been
+// fixed, remove this `allow`.
+#[allow(unknown_lints)]
+#[cfg(test)]
+mod tests {
+ use super::*;
+
+ #[test]
+ fn test_dst_layout_for_slice() {
+ let layout = DstLayout::for_slice::<u32>();
+ match layout.size_info {
+ SizeInfo::SliceDst(TrailingSliceLayout { offset, elem_size }) => {
+ assert_eq!(offset, 0);
+ assert_eq!(elem_size, 4);
+ }
+ _ => panic!("Expected SliceDst"),
+ }
+ assert_eq!(layout.align.get(), 4);
+ }
+
+ /// Tests of when a sized `DstLayout` is extended with a sized field.
+ #[allow(clippy::decimal_literal_representation)]
+ #[test]
+ fn test_dst_layout_extend_sized_with_sized() {
+ // This macro constructs a layout corresponding to a `u8` and extends it
+ // with a zero-sized trailing field of given alignment `n`. The macro
+ // tests that the resulting layout has both size and alignment `min(n,
+ // P)` for all valid values of `repr(packed(P))`.
+ macro_rules! test_align_is_size {
+ ($n:expr) => {
+ let base = DstLayout::for_type::<u8>();
+ let trailing_field = DstLayout::for_type::<elain::Align<$n>>();
+
+ let packs =
+ core::iter::once(None).chain((0..29).map(|p| NonZeroUsize::new(2usize.pow(p))));
+
+ for pack in packs {
+ let composite = base.extend(trailing_field, pack);
+ let max_align = pack.unwrap_or(DstLayout::CURRENT_MAX_ALIGN);
+ let align = $n.min(max_align.get());
+ assert_eq!(
+ composite,
+ DstLayout {
+ align: NonZeroUsize::new(align).unwrap(),
+ size_info: SizeInfo::Sized { size: align },
+ statically_shallow_unpadded: false,
+ }
+ )
+ }
+ };
+ }
+
+ test_align_is_size!(1);
+ test_align_is_size!(2);
+ test_align_is_size!(4);
+ test_align_is_size!(8);
+ test_align_is_size!(16);
+ test_align_is_size!(32);
+ test_align_is_size!(64);
+ test_align_is_size!(128);
+ test_align_is_size!(256);
+ test_align_is_size!(512);
+ test_align_is_size!(1024);
+ test_align_is_size!(2048);
+ test_align_is_size!(4096);
+ test_align_is_size!(8192);
+ test_align_is_size!(16384);
+ test_align_is_size!(32768);
+ test_align_is_size!(65536);
+ test_align_is_size!(131072);
+ test_align_is_size!(262144);
+ test_align_is_size!(524288);
+ test_align_is_size!(1048576);
+ test_align_is_size!(2097152);
+ test_align_is_size!(4194304);
+ test_align_is_size!(8388608);
+ test_align_is_size!(16777216);
+ test_align_is_size!(33554432);
+ test_align_is_size!(67108864);
+ test_align_is_size!(33554432);
+ test_align_is_size!(134217728);
+ test_align_is_size!(268435456);
+ }
+
+ /// Tests of when a sized `DstLayout` is extended with a DST field.
+ #[test]
+ fn test_dst_layout_extend_sized_with_dst() {
+ // Test that for all combinations of real-world alignments and
+ // `repr_packed` values, that the extension of a sized `DstLayout`` with
+ // a DST field correctly computes the trailing offset in the composite
+ // layout.
+
+ let aligns = (0..29).map(|p| NonZeroUsize::new(2usize.pow(p)).unwrap());
+ let packs = core::iter::once(None).chain(aligns.clone().map(Some));
+
+ for align in aligns {
+ for pack in packs.clone() {
+ let base = DstLayout::for_type::<u8>();
+ let elem_size = 42;
+ let trailing_field_offset = 11;
+
+ let trailing_field = DstLayout {
+ align,
+ size_info: SizeInfo::SliceDst(TrailingSliceLayout { elem_size, offset: 11 }),
+ statically_shallow_unpadded: false,
+ };
+
+ let composite = base.extend(trailing_field, pack);
+
+ let max_align = pack.unwrap_or(DstLayout::CURRENT_MAX_ALIGN).get();
+
+ let align = align.get().min(max_align);
+
+ assert_eq!(
+ composite,
+ DstLayout {
+ align: NonZeroUsize::new(align).unwrap(),
+ size_info: SizeInfo::SliceDst(TrailingSliceLayout {
+ elem_size,
+ offset: align + trailing_field_offset,
+ }),
+ statically_shallow_unpadded: false,
+ }
+ )
+ }
+ }
+ }
+
+ /// Tests that calling `pad_to_align` on a sized `DstLayout` adds the
+ /// expected amount of trailing padding.
+ #[test]
+ fn test_dst_layout_pad_to_align_with_sized() {
+ // For all valid alignments `align`, construct a one-byte layout aligned
+ // to `align`, call `pad_to_align`, and assert that the size of the
+ // resulting layout is equal to `align`.
+ for align in (0..29).map(|p| NonZeroUsize::new(2usize.pow(p)).unwrap()) {
+ let layout = DstLayout {
+ align,
+ size_info: SizeInfo::Sized { size: 1 },
+ statically_shallow_unpadded: true,
+ };
+
+ assert_eq!(
+ layout.pad_to_align(),
+ DstLayout {
+ align,
+ size_info: SizeInfo::Sized { size: align.get() },
+ statically_shallow_unpadded: align.get() == 1
+ }
+ );
+ }
+
+ // Test explicitly-provided combinations of unpadded and padded
+ // counterparts.
+
+ macro_rules! test {
+ (unpadded { size: $unpadded_size:expr, align: $unpadded_align:expr }
+ => padded { size: $padded_size:expr, align: $padded_align:expr }) => {
+ let unpadded = DstLayout {
+ align: NonZeroUsize::new($unpadded_align).unwrap(),
+ size_info: SizeInfo::Sized { size: $unpadded_size },
+ statically_shallow_unpadded: false,
+ };
+ let padded = unpadded.pad_to_align();
+
+ assert_eq!(
+ padded,
+ DstLayout {
+ align: NonZeroUsize::new($padded_align).unwrap(),
+ size_info: SizeInfo::Sized { size: $padded_size },
+ statically_shallow_unpadded: false,
+ }
+ );
+ };
+ }
+
+ test!(unpadded { size: 0, align: 4 } => padded { size: 0, align: 4 });
+ test!(unpadded { size: 1, align: 4 } => padded { size: 4, align: 4 });
+ test!(unpadded { size: 2, align: 4 } => padded { size: 4, align: 4 });
+ test!(unpadded { size: 3, align: 4 } => padded { size: 4, align: 4 });
+ test!(unpadded { size: 4, align: 4 } => padded { size: 4, align: 4 });
+ test!(unpadded { size: 5, align: 4 } => padded { size: 8, align: 4 });
+ test!(unpadded { size: 6, align: 4 } => padded { size: 8, align: 4 });
+ test!(unpadded { size: 7, align: 4 } => padded { size: 8, align: 4 });
+ test!(unpadded { size: 8, align: 4 } => padded { size: 8, align: 4 });
+
+ let current_max_align = DstLayout::CURRENT_MAX_ALIGN.get();
+
+ test!(unpadded { size: 1, align: current_max_align }
+ => padded { size: current_max_align, align: current_max_align });
+
+ test!(unpadded { size: current_max_align + 1, align: current_max_align }
+ => padded { size: current_max_align * 2, align: current_max_align });
+ }
+
+ /// Tests that calling `pad_to_align` on a DST `DstLayout` is a no-op.
+ #[test]
+ fn test_dst_layout_pad_to_align_with_dst() {
+ for align in (0..29).map(|p| NonZeroUsize::new(2usize.pow(p)).unwrap()) {
+ for offset in 0..10 {
+ for elem_size in 0..10 {
+ let layout = DstLayout {
+ align,
+ size_info: SizeInfo::SliceDst(TrailingSliceLayout { offset, elem_size }),
+ statically_shallow_unpadded: false,
+ };
+ assert_eq!(layout.pad_to_align(), layout);
+ }
+ }
+ }
+ }
+
+ // This test takes a long time when running under Miri, so we skip it in
+ // that case. This is acceptable because this is a logic test that doesn't
+ // attempt to expose UB.
+ #[test]
+ #[cfg_attr(miri, ignore)]
+ fn test_validate_cast_and_convert_metadata() {
+ #[allow(non_local_definitions)]
+ impl From<usize> for SizeInfo {
+ fn from(size: usize) -> SizeInfo {
+ SizeInfo::Sized { size }
+ }
+ }
+
+ #[allow(non_local_definitions)]
+ impl From<(usize, usize)> for SizeInfo {
+ fn from((offset, elem_size): (usize, usize)) -> SizeInfo {
+ SizeInfo::SliceDst(TrailingSliceLayout { offset, elem_size })
+ }
+ }
+
+ fn layout<S: Into<SizeInfo>>(s: S, align: usize) -> DstLayout {
+ DstLayout {
+ size_info: s.into(),
+ align: NonZeroUsize::new(align).unwrap(),
+ statically_shallow_unpadded: false,
+ }
+ }
+
+ /// This macro accepts arguments in the form of:
+ ///
+ /// layout(_, _).validate(_, _, _), Ok(Some((_, _)))
+ /// | | | | | | |
+ /// size ---------+ | | | | | |
+ /// align -----------+ | | | | |
+ /// addr ------------------------+ | | | |
+ /// bytes_len ----------------------+ | | |
+ /// cast_type -------------------------+ | |
+ /// elems ------------------------------------------+ |
+ /// split_at ------------------------------------------+
+ ///
+ /// `.validate` is shorthand for `.validate_cast_and_convert_metadata`
+ /// for brevity.
+ ///
+ /// Each argument can either be an iterator or a wildcard. Each
+ /// wildcarded variable is implicitly replaced by an iterator over a
+ /// representative sample of values for that variable. Each `test!`
+ /// invocation iterates over every combination of values provided by
+ /// each variable's iterator (ie, the cartesian product) and validates
+ /// that the results are expected.
+ ///
+ /// The final argument uses the same syntax, but it has a different
+ /// meaning:
+ /// - If it is `Ok(pat)`, then the pattern `pat` is supplied to
+ /// a matching assert to validate the computed result for each
+ /// combination of input values.
+ /// - If it is `Err(Some(msg) | None)`, then `test!` validates that the
+ /// call to `validate_cast_and_convert_metadata` panics with the given
+ /// panic message or, if the current Rust toolchain version is too
+ /// early to support panicking in `const fn`s, panics with *some*
+ /// message. In the latter case, the `const_panic!` macro is used,
+ /// which emits code which causes a non-panicking error at const eval
+ /// time, but which does panic when invoked at runtime. Thus, it is
+ /// merely difficult to predict the *value* of this panic. We deem
+ /// that testing against the real panic strings on stable and nightly
+ /// toolchains is enough to ensure correctness.
+ ///
+ /// Note that the meta-variables that match these variables have the
+ /// `tt` type, and some valid expressions are not valid `tt`s (such as
+ /// `a..b`). In this case, wrap the expression in parentheses, and it
+ /// will become valid `tt`.
+ macro_rules! test {
+ (
+ layout($size:tt, $align:tt)
+ .validate($addr:tt, $bytes_len:tt, $cast_type:tt), $expect:pat $(,)?
+ ) => {
+ itertools::iproduct!(
+ test!(@generate_size $size),
+ test!(@generate_align $align),
+ test!(@generate_usize $addr),
+ test!(@generate_usize $bytes_len),
+ test!(@generate_cast_type $cast_type)
+ ).for_each(|(size_info, align, addr, bytes_len, cast_type)| {
+ // Temporarily disable the panic hook installed by the test
+ // harness. If we don't do this, all panic messages will be
+ // kept in an internal log. On its own, this isn't a
+ // problem, but if a non-caught panic ever happens (ie, in
+ // code later in this test not in this macro), all of the
+ // previously-buffered messages will be dumped, hiding the
+ // real culprit.
+ let previous_hook = std::panic::take_hook();
+ // I don't understand why, but this seems to be required in
+ // addition to the previous line.
+ std::panic::set_hook(Box::new(|_| {}));
+ let actual = std::panic::catch_unwind(|| {
+ layout(size_info, align).validate_cast_and_convert_metadata(addr, bytes_len, cast_type)
+ }).map_err(|d| {
+ let msg = d.downcast::<&'static str>().ok().map(|s| *s.as_ref());
+ assert!(msg.is_some() || cfg!(no_zerocopy_panic_in_const_and_vec_try_reserve_1_57_0), "non-string panic messages are not permitted when usage of panic in const fn is enabled");
+ msg
+ });
+ std::panic::set_hook(previous_hook);
+
+ assert!(
+ matches!(actual, $expect),
+ "layout({:?}, {}).validate_cast_and_convert_metadata({}, {}, {:?})" ,size_info, align, addr, bytes_len, cast_type
+ );
+ });
+ };
+ (@generate_usize _) => { 0..8 };
+ // Generate sizes for both Sized and !Sized types.
+ (@generate_size _) => {
+ test!(@generate_size (_)).chain(test!(@generate_size (_, _)))
+ };
+ // Generate sizes for both Sized and !Sized types by chaining
+ // specified iterators for each.
+ (@generate_size ($sized_sizes:tt | $unsized_sizes:tt)) => {
+ test!(@generate_size ($sized_sizes)).chain(test!(@generate_size $unsized_sizes))
+ };
+ // Generate sizes for Sized types.
+ (@generate_size (_)) => { test!(@generate_size (0..8)) };
+ (@generate_size ($sizes:expr)) => { $sizes.into_iter().map(Into::<SizeInfo>::into) };
+ // Generate sizes for !Sized types.
+ (@generate_size ($min_sizes:tt, $elem_sizes:tt)) => {
+ itertools::iproduct!(
+ test!(@generate_min_size $min_sizes),
+ test!(@generate_elem_size $elem_sizes)
+ ).map(Into::<SizeInfo>::into)
+ };
+ (@generate_fixed_size _) => { (0..8).into_iter().map(Into::<SizeInfo>::into) };
+ (@generate_min_size _) => { 0..8 };
+ (@generate_elem_size _) => { 1..8 };
+ (@generate_align _) => { [1, 2, 4, 8, 16] };
+ (@generate_opt_usize _) => { [None].into_iter().chain((0..8).map(Some).into_iter()) };
+ (@generate_cast_type _) => { [CastType::Prefix, CastType::Suffix] };
+ (@generate_cast_type $variant:ident) => { [CastType::$variant] };
+ // Some expressions need to be wrapped in parentheses in order to be
+ // valid `tt`s (required by the top match pattern). See the comment
+ // below for more details. This arm removes these parentheses to
+ // avoid generating an `unused_parens` warning.
+ (@$_:ident ($vals:expr)) => { $vals };
+ (@$_:ident $vals:expr) => { $vals };
+ }
+
+ const EVENS: [usize; 8] = [0, 2, 4, 6, 8, 10, 12, 14];
+ const ODDS: [usize; 8] = [1, 3, 5, 7, 9, 11, 13, 15];
+
+ // base_size is too big for the memory region.
+ test!(
+ layout(((1..8) | ((1..8), (1..8))), _).validate([0], [0], _),
+ Ok(Err(MetadataCastError::Size))
+ );
+ test!(
+ layout(((2..8) | ((2..8), (2..8))), _).validate([0], [1], Prefix),
+ Ok(Err(MetadataCastError::Size))
+ );
+ test!(
+ layout(((2..8) | ((2..8), (2..8))), _).validate([0x1000_0000 - 1], [1], Suffix),
+ Ok(Err(MetadataCastError::Size))
+ );
+
+ // addr is unaligned for prefix cast
+ test!(layout(_, [2]).validate(ODDS, _, Prefix), Ok(Err(MetadataCastError::Alignment)));
+ test!(layout(_, [2]).validate(ODDS, _, Prefix), Ok(Err(MetadataCastError::Alignment)));
+
+ // addr is aligned, but end of buffer is unaligned for suffix cast
+ test!(layout(_, [2]).validate(EVENS, ODDS, Suffix), Ok(Err(MetadataCastError::Alignment)));
+ test!(layout(_, [2]).validate(EVENS, ODDS, Suffix), Ok(Err(MetadataCastError::Alignment)));
+
+ // Unfortunately, these constants cannot easily be used in the
+ // implementation of `validate_cast_and_convert_metadata`, since
+ // `panic!` consumes a string literal, not an expression.
+ //
+ // It's important that these messages be in a separate module. If they
+ // were at the function's top level, we'd pass them to `test!` as, e.g.,
+ // `Err(TRAILING)`, which would run into a subtle Rust footgun - the
+ // `TRAILING` identifier would be treated as a pattern to match rather
+ // than a value to check for equality.
+ mod msgs {
+ pub(super) const TRAILING: &str =
+ "attempted to cast to slice type with zero-sized element";
+ pub(super) const OVERFLOW: &str = "`addr` + `bytes_len` > usize::MAX";
+ }
+
+ // casts with ZST trailing element types are unsupported
+ test!(layout((_, [0]), _).validate(_, _, _), Err(Some(msgs::TRAILING) | None),);
+
+ // addr + bytes_len must not overflow usize
+ test!(layout(_, _).validate([usize::MAX], (1..100), _), Err(Some(msgs::OVERFLOW) | None));
+ test!(layout(_, _).validate((1..100), [usize::MAX], _), Err(Some(msgs::OVERFLOW) | None));
+ test!(
+ layout(_, _).validate(
+ [usize::MAX / 2 + 1, usize::MAX],
+ [usize::MAX / 2 + 1, usize::MAX],
+ _
+ ),
+ Err(Some(msgs::OVERFLOW) | None)
+ );
+
+ // Validates that `validate_cast_and_convert_metadata` satisfies its own
+ // documented safety postconditions, and also a few other properties
+ // that aren't documented but we want to guarantee anyway.
+ fn validate_behavior(
+ (layout, addr, bytes_len, cast_type): (DstLayout, usize, usize, CastType),
+ ) {
+ if let Ok((elems, split_at)) =
+ layout.validate_cast_and_convert_metadata(addr, bytes_len, cast_type)
+ {
+ let (size_info, align) = (layout.size_info, layout.align);
+ let debug_str = format!(
+ "layout({:?}, {}).validate_cast_and_convert_metadata({}, {}, {:?}) => ({}, {})",
+ size_info, align, addr, bytes_len, cast_type, elems, split_at
+ );
+
+ // If this is a sized type (no trailing slice), then `elems` is
+ // meaningless, but in practice we set it to 0. Callers are not
+ // allowed to rely on this, but a lot of math is nicer if
+ // they're able to, and some callers might accidentally do that.
+ let sized = matches!(layout.size_info, SizeInfo::Sized { .. });
+ assert!(!(sized && elems != 0), "{}", debug_str);
+
+ let resulting_size = match layout.size_info {
+ SizeInfo::Sized { size } => size,
+ SizeInfo::SliceDst(TrailingSliceLayout { offset, elem_size }) => {
+ let padded_size = |elems| {
+ let without_padding = offset + elems * elem_size;
+ without_padding + util::padding_needed_for(without_padding, align)
+ };
+
+ let resulting_size = padded_size(elems);
+ // Test that `validate_cast_and_convert_metadata`
+ // computed the largest possible value that fits in the
+ // given range.
+ assert!(padded_size(elems + 1) > bytes_len, "{}", debug_str);
+ resulting_size
+ }
+ };
+
+ // Test safety postconditions guaranteed by
+ // `validate_cast_and_convert_metadata`.
+ assert!(resulting_size <= bytes_len, "{}", debug_str);
+ match cast_type {
+ CastType::Prefix => {
+ assert_eq!(addr % align, 0, "{}", debug_str);
+ assert_eq!(resulting_size, split_at, "{}", debug_str);
+ }
+ CastType::Suffix => {
+ assert_eq!(split_at, bytes_len - resulting_size, "{}", debug_str);
+ assert_eq!((addr + split_at) % align, 0, "{}", debug_str);
+ }
+ }
+ } else {
+ let min_size = match layout.size_info {
+ SizeInfo::Sized { size } => size,
+ SizeInfo::SliceDst(TrailingSliceLayout { offset, .. }) => {
+ offset + util::padding_needed_for(offset, layout.align)
+ }
+ };
+
+ // If a cast is invalid, it is either because...
+ // 1. there are insufficient bytes at the given region for type:
+ let insufficient_bytes = bytes_len < min_size;
+ // 2. performing the cast would misalign type:
+ let base = match cast_type {
+ CastType::Prefix => 0,
+ CastType::Suffix => bytes_len,
+ };
+ let misaligned = (base + addr) % layout.align != 0;
+
+ assert!(insufficient_bytes || misaligned);
+ }
+ }
+
+ let sizes = 0..8;
+ let elem_sizes = 1..8;
+ let size_infos = sizes
+ .clone()
+ .map(Into::<SizeInfo>::into)
+ .chain(itertools::iproduct!(sizes, elem_sizes).map(Into::<SizeInfo>::into));
+ let layouts = itertools::iproduct!(size_infos, [1, 2, 4, 8, 16, 32])
+ .filter(|(size_info, align)| !matches!(size_info, SizeInfo::Sized { size } if size % align != 0))
+ .map(|(size_info, align)| layout(size_info, align));
+ itertools::iproduct!(layouts, 0..8, 0..8, [CastType::Prefix, CastType::Suffix])
+ .for_each(validate_behavior);
+ }
+
+ #[test]
+ #[cfg(__ZEROCOPY_INTERNAL_USE_ONLY_NIGHTLY_FEATURES_IN_TESTS)]
+ fn test_validate_rust_layout() {
+ use core::{
+ convert::TryInto as _,
+ ptr::{self, NonNull},
+ };
+
+ use crate::util::testutil::*;
+
+ // This test synthesizes pointers with various metadata and uses Rust's
+ // built-in APIs to confirm that Rust makes decisions about type layout
+ // which are consistent with what we believe is guaranteed by the
+ // language. If this test fails, it doesn't just mean our code is wrong
+ // - it means we're misunderstanding the language's guarantees.
+
+ #[derive(Debug)]
+ struct MacroArgs {
+ offset: usize,
+ align: NonZeroUsize,
+ elem_size: Option<usize>,
+ }
+
+ /// # Safety
+ ///
+ /// `test` promises to only call `addr_of_slice_field` on a `NonNull<T>`
+ /// which points to a valid `T`.
+ ///
+ /// `with_elems` must produce a pointer which points to a valid `T`.
+ fn test<T: ?Sized, W: Fn(usize) -> NonNull<T>>(
+ args: MacroArgs,
+ with_elems: W,
+ addr_of_slice_field: Option<fn(NonNull<T>) -> NonNull<u8>>,
+ ) {
+ let dst = args.elem_size.is_some();
+ let layout = {
+ let size_info = match args.elem_size {
+ Some(elem_size) => {
+ SizeInfo::SliceDst(TrailingSliceLayout { offset: args.offset, elem_size })
+ }
+ None => SizeInfo::Sized {
+ // Rust only supports types whose sizes are a multiple
+ // of their alignment. If the macro created a type like
+ // this:
+ //
+ // #[repr(C, align(2))]
+ // struct Foo([u8; 1]);
+ //
+ // ...then Rust will automatically round the type's size
+ // up to 2.
+ size: args.offset + util::padding_needed_for(args.offset, args.align),
+ },
+ };
+ DstLayout { size_info, align: args.align, statically_shallow_unpadded: false }
+ };
+
+ for elems in 0..128 {
+ let ptr = with_elems(elems);
+
+ if let Some(addr_of_slice_field) = addr_of_slice_field {
+ let slc_field_ptr = addr_of_slice_field(ptr).as_ptr();
+ // SAFETY: Both `slc_field_ptr` and `ptr` are pointers to
+ // the same valid Rust object.
+ // Work around https://github.com/rust-lang/rust-clippy/issues/12280
+ let offset: usize =
+ unsafe { slc_field_ptr.byte_offset_from(ptr.as_ptr()).try_into().unwrap() };
+ assert_eq!(offset, args.offset);
+ }
+
+ // SAFETY: `ptr` points to a valid `T`.
+ #[allow(clippy::multiple_unsafe_ops_per_block)]
+ let (size, align) = unsafe {
+ (mem::size_of_val_raw(ptr.as_ptr()), mem::align_of_val_raw(ptr.as_ptr()))
+ };
+
+ // Avoid expensive allocation when running under Miri.
+ let assert_msg = if !cfg!(miri) {
+ format!("\n{:?}\nsize:{}, align:{}", args, size, align)
+ } else {
+ String::new()
+ };
+
+ let without_padding =
+ args.offset + args.elem_size.map(|elem_size| elems * elem_size).unwrap_or(0);
+ assert!(size >= without_padding, "{}", assert_msg);
+ assert_eq!(align, args.align.get(), "{}", assert_msg);
+
+ // This encodes the most important part of the test: our
+ // understanding of how Rust determines the layout of repr(C)
+ // types. Sized repr(C) types are trivial, but DST types have
+ // some subtlety. Note that:
+ // - For sized types, `without_padding` is just the size of the
+ // type that we constructed for `Foo`. Since we may have
+ // requested a larger alignment, `Foo` may actually be larger
+ // than this, hence `padding_needed_for`.
+ // - For unsized types, `without_padding` is dynamically
+ // computed from the offset, the element size, and element
+ // count. We expect that the size of the object should be
+ // `offset + elem_size * elems` rounded up to the next
+ // alignment.
+ let expected_size =
+ without_padding + util::padding_needed_for(without_padding, args.align);
+ assert_eq!(expected_size, size, "{}", assert_msg);
+
+ // For zero-sized element types,
+ // `validate_cast_and_convert_metadata` just panics, so we skip
+ // testing those types.
+ if args.elem_size.map(|elem_size| elem_size > 0).unwrap_or(true) {
+ let addr = ptr.addr().get();
+ let (got_elems, got_split_at) = layout
+ .validate_cast_and_convert_metadata(addr, size, CastType::Prefix)
+ .unwrap();
+ // Avoid expensive allocation when running under Miri.
+ let assert_msg = if !cfg!(miri) {
+ format!(
+ "{}\nvalidate_cast_and_convert_metadata({}, {})",
+ assert_msg, addr, size,
+ )
+ } else {
+ String::new()
+ };
+ assert_eq!(got_split_at, size, "{}", assert_msg);
+ if dst {
+ assert!(got_elems >= elems, "{}", assert_msg);
+ if got_elems != elems {
+ // If `validate_cast_and_convert_metadata`
+ // returned more elements than `elems`, that
+ // means that `elems` is not the maximum number
+ // of elements that can fit in `size` - in other
+ // words, there is enough padding at the end of
+ // the value to fit at least one more element.
+ // If we use this metadata to synthesize a
+ // pointer, despite having a different element
+ // count, we still expect it to have the same
+ // size.
+ let got_ptr = with_elems(got_elems);
+ // SAFETY: `got_ptr` is a pointer to a valid `T`.
+ let size_of_got_ptr = unsafe { mem::size_of_val_raw(got_ptr.as_ptr()) };
+ assert_eq!(size_of_got_ptr, size, "{}", assert_msg);
+ }
+ } else {
+ // For sized casts, the returned element value is
+ // technically meaningless, and we don't guarantee any
+ // particular value. In practice, it's always zero.
+ assert_eq!(got_elems, 0, "{}", assert_msg)
+ }
+ }
+ }
+ }
+
+ macro_rules! validate_against_rust {
+ ($offset:literal, $align:literal $(, $elem_size:literal)?) => {{
+ #[repr(C, align($align))]
+ struct Foo([u8; $offset]$(, [[u8; $elem_size]])?);
+
+ let args = MacroArgs {
+ offset: $offset,
+ align: $align.try_into().unwrap(),
+ elem_size: {
+ #[allow(unused)]
+ let ret = None::<usize>;
+ $(let ret = Some($elem_size);)?
+ ret
+ }
+ };
+
+ #[repr(C, align($align))]
+ struct FooAlign;
+ // Create an aligned buffer to use in order to synthesize
+ // pointers to `Foo`. We don't ever load values from these
+ // pointers - we just do arithmetic on them - so having a "real"
+ // block of memory as opposed to a validly-aligned-but-dangling
+ // pointer is only necessary to make Miri happy since we run it
+ // with "strict provenance" checking enabled.
+ let aligned_buf = Align::<_, FooAlign>::new([0u8; 1024]);
+ let with_elems = |elems| {
+ let slc = NonNull::slice_from_raw_parts(NonNull::from(&aligned_buf.t), elems);
+ #[allow(clippy::as_conversions)]
+ NonNull::new(slc.as_ptr() as *mut Foo).unwrap()
+ };
+ let addr_of_slice_field = {
+ #[allow(unused)]
+ let f = None::<fn(NonNull<Foo>) -> NonNull<u8>>;
+ $(
+ // SAFETY: `test` promises to only call `f` with a `ptr`
+ // to a valid `Foo`.
+ let f: Option<fn(NonNull<Foo>) -> NonNull<u8>> = Some(|ptr: NonNull<Foo>| unsafe {
+ NonNull::new(ptr::addr_of_mut!((*ptr.as_ptr()).1)).unwrap().cast::<u8>()
+ });
+ let _ = $elem_size;
+ )?
+ f
+ };
+
+ test::<Foo, _>(args, with_elems, addr_of_slice_field);
+ }};
+ }
+
+ // Every permutation of:
+ // - offset in [0, 4]
+ // - align in [1, 16]
+ // - elem_size in [0, 4] (plus no elem_size)
+ validate_against_rust!(0, 1);
+ validate_against_rust!(0, 1, 0);
+ validate_against_rust!(0, 1, 1);
+ validate_against_rust!(0, 1, 2);
+ validate_against_rust!(0, 1, 3);
+ validate_against_rust!(0, 1, 4);
+ validate_against_rust!(0, 2);
+ validate_against_rust!(0, 2, 0);
+ validate_against_rust!(0, 2, 1);
+ validate_against_rust!(0, 2, 2);
+ validate_against_rust!(0, 2, 3);
+ validate_against_rust!(0, 2, 4);
+ validate_against_rust!(0, 4);
+ validate_against_rust!(0, 4, 0);
+ validate_against_rust!(0, 4, 1);
+ validate_against_rust!(0, 4, 2);
+ validate_against_rust!(0, 4, 3);
+ validate_against_rust!(0, 4, 4);
+ validate_against_rust!(0, 8);
+ validate_against_rust!(0, 8, 0);
+ validate_against_rust!(0, 8, 1);
+ validate_against_rust!(0, 8, 2);
+ validate_against_rust!(0, 8, 3);
+ validate_against_rust!(0, 8, 4);
+ validate_against_rust!(0, 16);
+ validate_against_rust!(0, 16, 0);
+ validate_against_rust!(0, 16, 1);
+ validate_against_rust!(0, 16, 2);
+ validate_against_rust!(0, 16, 3);
+ validate_against_rust!(0, 16, 4);
+ validate_against_rust!(1, 1);
+ validate_against_rust!(1, 1, 0);
+ validate_against_rust!(1, 1, 1);
+ validate_against_rust!(1, 1, 2);
+ validate_against_rust!(1, 1, 3);
+ validate_against_rust!(1, 1, 4);
+ validate_against_rust!(1, 2);
+ validate_against_rust!(1, 2, 0);
+ validate_against_rust!(1, 2, 1);
+ validate_against_rust!(1, 2, 2);
+ validate_against_rust!(1, 2, 3);
+ validate_against_rust!(1, 2, 4);
+ validate_against_rust!(1, 4);
+ validate_against_rust!(1, 4, 0);
+ validate_against_rust!(1, 4, 1);
+ validate_against_rust!(1, 4, 2);
+ validate_against_rust!(1, 4, 3);
+ validate_against_rust!(1, 4, 4);
+ validate_against_rust!(1, 8);
+ validate_against_rust!(1, 8, 0);
+ validate_against_rust!(1, 8, 1);
+ validate_against_rust!(1, 8, 2);
+ validate_against_rust!(1, 8, 3);
+ validate_against_rust!(1, 8, 4);
+ validate_against_rust!(1, 16);
+ validate_against_rust!(1, 16, 0);
+ validate_against_rust!(1, 16, 1);
+ validate_against_rust!(1, 16, 2);
+ validate_against_rust!(1, 16, 3);
+ validate_against_rust!(1, 16, 4);
+ validate_against_rust!(2, 1);
+ validate_against_rust!(2, 1, 0);
+ validate_against_rust!(2, 1, 1);
+ validate_against_rust!(2, 1, 2);
+ validate_against_rust!(2, 1, 3);
+ validate_against_rust!(2, 1, 4);
+ validate_against_rust!(2, 2);
+ validate_against_rust!(2, 2, 0);
+ validate_against_rust!(2, 2, 1);
+ validate_against_rust!(2, 2, 2);
+ validate_against_rust!(2, 2, 3);
+ validate_against_rust!(2, 2, 4);
+ validate_against_rust!(2, 4);
+ validate_against_rust!(2, 4, 0);
+ validate_against_rust!(2, 4, 1);
+ validate_against_rust!(2, 4, 2);
+ validate_against_rust!(2, 4, 3);
+ validate_against_rust!(2, 4, 4);
+ validate_against_rust!(2, 8);
+ validate_against_rust!(2, 8, 0);
+ validate_against_rust!(2, 8, 1);
+ validate_against_rust!(2, 8, 2);
+ validate_against_rust!(2, 8, 3);
+ validate_against_rust!(2, 8, 4);
+ validate_against_rust!(2, 16);
+ validate_against_rust!(2, 16, 0);
+ validate_against_rust!(2, 16, 1);
+ validate_against_rust!(2, 16, 2);
+ validate_against_rust!(2, 16, 3);
+ validate_against_rust!(2, 16, 4);
+ validate_against_rust!(3, 1);
+ validate_against_rust!(3, 1, 0);
+ validate_against_rust!(3, 1, 1);
+ validate_against_rust!(3, 1, 2);
+ validate_against_rust!(3, 1, 3);
+ validate_against_rust!(3, 1, 4);
+ validate_against_rust!(3, 2);
+ validate_against_rust!(3, 2, 0);
+ validate_against_rust!(3, 2, 1);
+ validate_against_rust!(3, 2, 2);
+ validate_against_rust!(3, 2, 3);
+ validate_against_rust!(3, 2, 4);
+ validate_against_rust!(3, 4);
+ validate_against_rust!(3, 4, 0);
+ validate_against_rust!(3, 4, 1);
+ validate_against_rust!(3, 4, 2);
+ validate_against_rust!(3, 4, 3);
+ validate_against_rust!(3, 4, 4);
+ validate_against_rust!(3, 8);
+ validate_against_rust!(3, 8, 0);
+ validate_against_rust!(3, 8, 1);
+ validate_against_rust!(3, 8, 2);
+ validate_against_rust!(3, 8, 3);
+ validate_against_rust!(3, 8, 4);
+ validate_against_rust!(3, 16);
+ validate_against_rust!(3, 16, 0);
+ validate_against_rust!(3, 16, 1);
+ validate_against_rust!(3, 16, 2);
+ validate_against_rust!(3, 16, 3);
+ validate_against_rust!(3, 16, 4);
+ validate_against_rust!(4, 1);
+ validate_against_rust!(4, 1, 0);
+ validate_against_rust!(4, 1, 1);
+ validate_against_rust!(4, 1, 2);
+ validate_against_rust!(4, 1, 3);
+ validate_against_rust!(4, 1, 4);
+ validate_against_rust!(4, 2);
+ validate_against_rust!(4, 2, 0);
+ validate_against_rust!(4, 2, 1);
+ validate_against_rust!(4, 2, 2);
+ validate_against_rust!(4, 2, 3);
+ validate_against_rust!(4, 2, 4);
+ validate_against_rust!(4, 4);
+ validate_against_rust!(4, 4, 0);
+ validate_against_rust!(4, 4, 1);
+ validate_against_rust!(4, 4, 2);
+ validate_against_rust!(4, 4, 3);
+ validate_against_rust!(4, 4, 4);
+ validate_against_rust!(4, 8);
+ validate_against_rust!(4, 8, 0);
+ validate_against_rust!(4, 8, 1);
+ validate_against_rust!(4, 8, 2);
+ validate_against_rust!(4, 8, 3);
+ validate_against_rust!(4, 8, 4);
+ validate_against_rust!(4, 16);
+ validate_against_rust!(4, 16, 0);
+ validate_against_rust!(4, 16, 1);
+ validate_against_rust!(4, 16, 2);
+ validate_against_rust!(4, 16, 3);
+ validate_against_rust!(4, 16, 4);
+ }
+}
+
+#[cfg(kani)]
+mod proofs {
+ use core::alloc::Layout;
+
+ use super::*;
+
+ impl kani::Arbitrary for DstLayout {
+ fn any() -> Self {
+ let align: NonZeroUsize = kani::any();
+ let size_info: SizeInfo = kani::any();
+
+ kani::assume(align.is_power_of_two());
+ kani::assume(align < DstLayout::THEORETICAL_MAX_ALIGN);
+
+ // For testing purposes, we most care about instantiations of
+ // `DstLayout` that can correspond to actual Rust types. We use
+ // `Layout` to verify that our `DstLayout` satisfies the validity
+ // conditions of Rust layouts.
+ kani::assume(
+ match size_info {
+ SizeInfo::Sized { size } => Layout::from_size_align(size, align.get()),
+ SizeInfo::SliceDst(TrailingSliceLayout { offset, elem_size: _ }) => {
+ // `SliceDst` cannot encode an exact size, but we know
+ // it is at least `offset` bytes.
+ Layout::from_size_align(offset, align.get())
+ }
+ }
+ .is_ok(),
+ );
+
+ Self { align: align, size_info: size_info, statically_shallow_unpadded: kani::any() }
+ }
+ }
+
+ impl kani::Arbitrary for SizeInfo {
+ fn any() -> Self {
+ let is_sized: bool = kani::any();
+
+ match is_sized {
+ true => {
+ let size: usize = kani::any();
+
+ kani::assume(size <= DstLayout::MAX_SIZE);
+
+ SizeInfo::Sized { size }
+ }
+ false => SizeInfo::SliceDst(kani::any()),
+ }
+ }
+ }
+
+ impl kani::Arbitrary for TrailingSliceLayout {
+ fn any() -> Self {
+ let elem_size: usize = kani::any();
+ let offset: usize = kani::any();
+
+ kani::assume(elem_size < DstLayout::MAX_SIZE);
+ kani::assume(offset < DstLayout::MAX_SIZE);
+
+ TrailingSliceLayout { elem_size, offset }
+ }
+ }
+
+ #[kani::proof]
+ fn prove_requires_dynamic_padding() {
+ let layout: DstLayout = kani::any();
+
+ let SizeInfo::SliceDst(size_info) = layout.size_info else {
+ kani::assume(false);
+ loop {}
+ };
+
+ let meta: usize = kani::any();
+
+ let Some(trailing_slice_size) = size_info.elem_size.checked_mul(meta) else {
+ // The `trailing_slice_size` exceeds `usize::MAX`; `meta` is invalid.
+ kani::assume(false);
+ loop {}
+ };
+
+ let Some(unpadded_size) = size_info.offset.checked_add(trailing_slice_size) else {
+ // The `unpadded_size` exceeds `usize::MAX`; `meta`` is invalid.
+ kani::assume(false);
+ loop {}
+ };
+
+ if unpadded_size >= DstLayout::MAX_SIZE {
+ // The `unpadded_size` exceeds `isize::MAX`; `meta` is invalid.
+ kani::assume(false);
+ loop {}
+ }
+
+ let trailing_padding = util::padding_needed_for(unpadded_size, layout.align);
+
+ if !layout.requires_dynamic_padding() {
+ assert!(trailing_padding == 0);
+ }
+ }
+
+ #[kani::proof]
+ fn prove_dst_layout_extend() {
+ use crate::util::{max, min, padding_needed_for};
+
+ let base: DstLayout = kani::any();
+ let field: DstLayout = kani::any();
+ let packed: Option<NonZeroUsize> = kani::any();
+
+ if let Some(max_align) = packed {
+ kani::assume(max_align.is_power_of_two());
+ kani::assume(base.align <= max_align);
+ }
+
+ // The base can only be extended if it's sized.
+ kani::assume(matches!(base.size_info, SizeInfo::Sized { .. }));
+ let base_size = if let SizeInfo::Sized { size } = base.size_info {
+ size
+ } else {
+ unreachable!();
+ };
+
+ // Under the above conditions, `DstLayout::extend` will not panic.
+ let composite = base.extend(field, packed);
+
+ // The field's alignment is clamped by `max_align` (i.e., the
+ // `packed` attribute, if any) [1].
+ //
+ // [1] Per https://doc.rust-lang.org/reference/type-layout.html#the-alignment-modifiers:
+ //
+ // The alignments of each field, for the purpose of positioning
+ // fields, is the smaller of the specified alignment and the
+ // alignment of the field's type.
+ let field_align = min(field.align, packed.unwrap_or(DstLayout::THEORETICAL_MAX_ALIGN));
+
+ // The struct's alignment is the maximum of its previous alignment and
+ // `field_align`.
+ assert_eq!(composite.align, max(base.align, field_align));
+
+ // Compute the minimum amount of inter-field padding needed to
+ // satisfy the field's alignment, and offset of the trailing field.
+ // [1]
+ //
+ // [1] Per https://doc.rust-lang.org/reference/type-layout.html#the-alignment-modifiers:
+ //
+ // Inter-field padding is guaranteed to be the minimum required in
+ // order to satisfy each field's (possibly altered) alignment.
+ let padding = padding_needed_for(base_size, field_align);
+ let offset = base_size + padding;
+
+ // For testing purposes, we'll also construct `alloc::Layout`
+ // stand-ins for `DstLayout`, and show that `extend` behaves
+ // comparably on both types.
+ let base_analog = Layout::from_size_align(base_size, base.align.get()).unwrap();
+
+ match field.size_info {
+ SizeInfo::Sized { size: field_size } => {
+ if let SizeInfo::Sized { size: composite_size } = composite.size_info {
+ // If the trailing field is sized, the resulting layout will
+ // be sized. Its size will be the sum of the preceding
+ // layout, the size of the new field, and the size of
+ // inter-field padding between the two.
+ assert_eq!(composite_size, offset + field_size);
+
+ let field_analog =
+ Layout::from_size_align(field_size, field_align.get()).unwrap();
+
+ if let Ok((actual_composite, actual_offset)) = base_analog.extend(field_analog)
+ {
+ assert_eq!(actual_offset, offset);
+ assert_eq!(actual_composite.size(), composite_size);
+ assert_eq!(actual_composite.align(), composite.align.get());
+ } else {
+ // An error here reflects that composite of `base`
+ // and `field` cannot correspond to a real Rust type
+ // fragment, because such a fragment would violate
+ // the basic invariants of a valid Rust layout. At
+ // the time of writing, `DstLayout` is a little more
+ // permissive than `Layout`, so we don't assert
+ // anything in this branch (e.g., unreachability).
+ }
+ } else {
+ panic!("The composite of two sized layouts must be sized.")
+ }
+ }
+ SizeInfo::SliceDst(TrailingSliceLayout {
+ offset: field_offset,
+ elem_size: field_elem_size,
+ }) => {
+ if let SizeInfo::SliceDst(TrailingSliceLayout {
+ offset: composite_offset,
+ elem_size: composite_elem_size,
+ }) = composite.size_info
+ {
+ // The offset of the trailing slice component is the sum
+ // of the offset of the trailing field and the trailing
+ // slice offset within that field.
+ assert_eq!(composite_offset, offset + field_offset);
+ // The elem size is unchanged.
+ assert_eq!(composite_elem_size, field_elem_size);
+
+ let field_analog =
+ Layout::from_size_align(field_offset, field_align.get()).unwrap();
+
+ if let Ok((actual_composite, actual_offset)) = base_analog.extend(field_analog)
+ {
+ assert_eq!(actual_offset, offset);
+ assert_eq!(actual_composite.size(), composite_offset);
+ assert_eq!(actual_composite.align(), composite.align.get());
+ } else {
+ // An error here reflects that composite of `base`
+ // and `field` cannot correspond to a real Rust type
+ // fragment, because such a fragment would violate
+ // the basic invariants of a valid Rust layout. At
+ // the time of writing, `DstLayout` is a little more
+ // permissive than `Layout`, so we don't assert
+ // anything in this branch (e.g., unreachability).
+ }
+ } else {
+ panic!("The extension of a layout with a DST must result in a DST.")
+ }
+ }
+ }
+ }
+
+ #[kani::proof]
+ #[kani::should_panic]
+ fn prove_dst_layout_extend_dst_panics() {
+ let base: DstLayout = kani::any();
+ let field: DstLayout = kani::any();
+ let packed: Option<NonZeroUsize> = kani::any();
+
+ if let Some(max_align) = packed {
+ kani::assume(max_align.is_power_of_two());
+ kani::assume(base.align <= max_align);
+ }
+
+ kani::assume(matches!(base.size_info, SizeInfo::SliceDst(..)));
+
+ let _ = base.extend(field, packed);
+ }
+
+ #[kani::proof]
+ fn prove_dst_layout_pad_to_align() {
+ use crate::util::padding_needed_for;
+
+ let layout: DstLayout = kani::any();
+
+ let padded = layout.pad_to_align();
+
+ // Calling `pad_to_align` does not alter the `DstLayout`'s alignment.
+ assert_eq!(padded.align, layout.align);
+
+ if let SizeInfo::Sized { size: unpadded_size } = layout.size_info {
+ if let SizeInfo::Sized { size: padded_size } = padded.size_info {
+ // If the layout is sized, it will remain sized after padding is
+ // added. Its sum will be its unpadded size and the size of the
+ // trailing padding needed to satisfy its alignment
+ // requirements.
+ let padding = padding_needed_for(unpadded_size, layout.align);
+ assert_eq!(padded_size, unpadded_size + padding);
+
+ // Prove that calling `DstLayout::pad_to_align` behaves
+ // identically to `Layout::pad_to_align`.
+ let layout_analog =
+ Layout::from_size_align(unpadded_size, layout.align.get()).unwrap();
+ let padded_analog = layout_analog.pad_to_align();
+ assert_eq!(padded_analog.align(), layout.align.get());
+ assert_eq!(padded_analog.size(), padded_size);
+ } else {
+ panic!("The padding of a sized layout must result in a sized layout.")
+ }
+ } else {
+ // If the layout is a DST, padding cannot be statically added.
+ assert_eq!(padded.size_info, layout.size_info);
+ }
+ }
+}
diff --git a/rust/zerocopy/src/lib.rs b/rust/zerocopy/src/lib.rs
new file mode 100644
index 0000000..3302d67
--- /dev/null
+++ b/rust/zerocopy/src/lib.rs
@@ -0,0 +1,7612 @@
+// SPDX-License-Identifier: (BSD-2-Clause OR Apache-2.0) OR MIT
+
+// Copyright 2018 The Fuchsia Authors
+//
+// Licensed under the 2-Clause BSD License <LICENSE-BSD or
+// https://opensource.org/license/bsd-2-clause>, Apache License, Version 2.0
+// <LICENSE-APACHE or https://www.apache.org/licenses/LICENSE-2.0>, or the MIT
+// license <LICENSE-MIT or https://opensource.org/licenses/MIT>, at your option.
+// This file may not be copied, modified, or distributed except according to
+// those terms.
+
+// After updating the following doc comment, make sure to run the following
+// command to update `README.md` based on its contents:
+//
+// cargo -q run --manifest-path tools/Cargo.toml -p generate-readme > README.md
+
+//! ***<span style="font-size: 140%">Fast, safe, <span
+//! style="color:red;">compile error</span>. Pick two.</span>***
+//!
+//! Zerocopy makes zero-cost memory manipulation effortless. We write `unsafe`
+//! so you don't have to.
+//!
+//! *For an overview of what's changed from zerocopy 0.7, check out our [release
+//! notes][release-notes], which include a step-by-step upgrading guide.*
+//!
+//! *Have questions? Need more out of zerocopy? Submit a [customer request
+//! issue][customer-request-issue] or ask the maintainers on
+//! [GitHub][github-q-a] or [Discord][discord]!*
+//!
+//! [customer-request-issue]: https://github.com/google/zerocopy/issues/new/choose
+//! [release-notes]: https://github.com/google/zerocopy/discussions/1680
+//! [github-q-a]: https://github.com/google/zerocopy/discussions/categories/q-a
+//! [discord]: https://discord.gg/MAvWH2R6zk
+//!
+//! # Overview
+//!
+//! ##### Conversion Traits
+//!
+//! Zerocopy provides four derivable traits for zero-cost conversions:
+//! - [`TryFromBytes`] indicates that a type may safely be converted from
+//! certain byte sequences (conditional on runtime checks)
+//! - [`FromZeros`] indicates that a sequence of zero bytes represents a valid
+//! instance of a type
+//! - [`FromBytes`] indicates that a type may safely be converted from an
+//! arbitrary byte sequence
+//! - [`IntoBytes`] indicates that a type may safely be converted *to* a byte
+//! sequence
+//!
+//! These traits support sized types, slices, and [slice DSTs][slice-dsts].
+//!
+//! [slice-dsts]: KnownLayout#dynamically-sized-types
+//!
+//! ##### Marker Traits
+//!
+//! Zerocopy provides three derivable marker traits that do not provide any
+//! functionality themselves, but are required to call certain methods provided
+//! by the conversion traits:
+//! - [`KnownLayout`] indicates that zerocopy can reason about certain layout
+//! qualities of a type
+//! - [`Immutable`] indicates that a type is free from interior mutability,
+//! except by ownership or an exclusive (`&mut`) borrow
+//! - [`Unaligned`] indicates that a type's alignment requirement is 1
+//!
+//! You should generally derive these marker traits whenever possible.
+//!
+//! ##### Conversion Macros
+//!
+//! Zerocopy provides six macros for safe casting between types:
+//!
+//! - ([`try_`][try_transmute])[`transmute`] (conditionally) converts a value of
+//! one type to a value of another type of the same size
+//! - ([`try_`][try_transmute_mut])[`transmute_mut`] (conditionally) converts a
+//! mutable reference of one type to a mutable reference of another type of
+//! the same size
+//! - ([`try_`][try_transmute_ref])[`transmute_ref`] (conditionally) converts a
+//! mutable or immutable reference of one type to an immutable reference of
+//! another type of the same size
+//!
+//! These macros perform *compile-time* size and alignment checks, meaning that
+//! unconditional casts have zero cost at runtime. Conditional casts do not need
+//! to validate size or alignment runtime, but do need to validate contents.
+//!
+//! These macros cannot be used in generic contexts. For generic conversions,
+//! use the methods defined by the [conversion traits](#conversion-traits).
+//!
+//! ##### Byteorder-Aware Numerics
+//!
+//! Zerocopy provides byte-order aware integer types that support these
+//! conversions; see the [`byteorder`] module. These types are especially useful
+//! for network parsing.
+//!
+//! # Cargo Features
+//!
+//! - **`alloc`**
+//! By default, `zerocopy` is `no_std`. When the `alloc` feature is enabled,
+//! the `alloc` crate is added as a dependency, and some allocation-related
+//! functionality is added.
+//!
+//! - **`std`**
+//! By default, `zerocopy` is `no_std`. When the `std` feature is enabled, the
+//! `std` crate is added as a dependency (ie, `no_std` is disabled), and
+//! support for some `std` types is added. `std` implies `alloc`.
+//!
+//! - **`derive`**
+//! Provides derives for the core marker traits via the `zerocopy-derive`
+//! crate. These derives are re-exported from `zerocopy`, so it is not
+//! necessary to depend on `zerocopy-derive` directly.
+//!
+//! However, you may experience better compile times if you instead directly
+//! depend on both `zerocopy` and `zerocopy-derive` in your `Cargo.toml`,
+//! since doing so will allow Rust to compile these crates in parallel. To do
+//! so, do *not* enable the `derive` feature, and list both dependencies in
+//! your `Cargo.toml` with the same leading non-zero version number; e.g:
+//!
+//! ```toml
+//! [dependencies]
+//! zerocopy = "0.X"
+//! zerocopy-derive = "0.X"
+//! ```
+//!
+//! To avoid the risk of [duplicate import errors][duplicate-import-errors] if
+//! one of your dependencies enables zerocopy's `derive` feature, import
+//! derives as `use zerocopy_derive::*` rather than by name (e.g., `use
+//! zerocopy_derive::FromBytes`).
+//!
+//! - **`simd`**
+//! When the `simd` feature is enabled, `FromZeros`, `FromBytes`, and
+//! `IntoBytes` impls are emitted for all stable SIMD types which exist on the
+//! target platform. Note that the layout of SIMD types is not yet stabilized,
+//! so these impls may be removed in the future if layout changes make them
+//! invalid. For more information, see the Unsafe Code Guidelines Reference
+//! page on the [layout of packed SIMD vectors][simd-layout].
+//!
+//! - **`simd-nightly`**
+//! Enables the `simd` feature and adds support for SIMD types which are only
+//! available on nightly. Since these types are unstable, support for any type
+//! may be removed at any point in the future.
+//!
+//! - **`float-nightly`**
+//! Adds support for the unstable `f16` and `f128` types. These types are
+//! not yet fully implemented and may not be supported on all platforms.
+//!
+//! [duplicate-import-errors]: https://github.com/google/zerocopy/issues/1587
+//! [simd-layout]: https://rust-lang.github.io/unsafe-code-guidelines/layout/packed-simd-vectors.html
+//!
+//! # Build Tuning
+//!
+//! ## `--cfg zerocopy_inline_always`
+//!
+//! Upgrades `#[inline]` to `#[inline(always)]` on many of zerocopy's public
+//! functions and methods. This provides a narrowly-scoped alternative that
+//! *may* improve the optimization of hot paths using zerocopy without the broad
+//! compile-time penalties of configuring `codegen-units=1`.
+//!
+//! # Security Ethos
+//!
+//! Zerocopy is expressly designed for use in security-critical contexts. We
+//! strive to ensure that that zerocopy code is sound under Rust's current
+//! memory model, and *any future memory model*. We ensure this by:
+//! - **...not 'guessing' about Rust's semantics.**
+//! We annotate `unsafe` code with a precise rationale for its soundness that
+//! cites a relevant section of Rust's official documentation. When Rust's
+//! documented semantics are unclear, we work with the Rust Operational
+//! Semantics Team to clarify Rust's documentation.
+//! - **...rigorously testing our implementation.**
+//! We run tests using [Miri], ensuring that zerocopy is sound across a wide
+//! array of supported target platforms of varying endianness and pointer
+//! width, and across both current and experimental memory models of Rust.
+//! - **...formally proving the correctness of our implementation.**
+//! We apply formal verification tools like [Kani][kani] to prove zerocopy's
+//! correctness.
+//!
+//! For more information, see our full [soundness policy].
+//!
+//! [Miri]: https://github.com/rust-lang/miri
+//! [Kani]: https://github.com/model-checking/kani
+//! [soundness policy]: https://github.com/google/zerocopy/blob/main/POLICIES.md#soundness
+//!
+//! # Relationship to Project Safe Transmute
+//!
+//! [Project Safe Transmute] is an official initiative of the Rust Project to
+//! develop language-level support for safer transmutation. The Project consults
+//! with crates like zerocopy to identify aspects of safer transmutation that
+//! would benefit from compiler support, and has developed an [experimental,
+//! compiler-supported analysis][mcp-transmutability] which determines whether,
+//! for a given type, any value of that type may be soundly transmuted into
+//! another type. Once this functionality is sufficiently mature, zerocopy
+//! intends to replace its internal transmutability analysis (implemented by our
+//! custom derives) with the compiler-supported one. This change will likely be
+//! an implementation detail that is invisible to zerocopy's users.
+//!
+//! Project Safe Transmute will not replace the need for most of zerocopy's
+//! higher-level abstractions. The experimental compiler analysis is a tool for
+//! checking the soundness of `unsafe` code, not a tool to avoid writing
+//! `unsafe` code altogether. For the foreseeable future, crates like zerocopy
+//! will still be required in order to provide higher-level abstractions on top
+//! of the building block provided by Project Safe Transmute.
+//!
+//! [Project Safe Transmute]: https://rust-lang.github.io/rfcs/2835-project-safe-transmute.html
+//! [mcp-transmutability]: https://github.com/rust-lang/compiler-team/issues/411
+//!
+//! # MSRV
+//!
+//! See our [MSRV policy].
+//!
+//! [MSRV policy]: https://github.com/google/zerocopy/blob/main/POLICIES.md#msrv
+//!
+//! # Changelog
+//!
+//! Zerocopy uses [GitHub Releases].
+//!
+//! [GitHub Releases]: https://github.com/google/zerocopy/releases
+//!
+//! # Thanks
+//!
+//! Zerocopy is maintained by engineers at Google with help from [many wonderful
+//! contributors][contributors]. Thank you to everyone who has lent a hand in
+//! making Rust a little more secure!
+//!
+//! [contributors]: https://github.com/google/zerocopy/graphs/contributors
+
+// Sometimes we want to use lints which were added after our MSRV.
+// `unknown_lints` is `warn` by default and we deny warnings in CI, so without
+// this attribute, any unknown lint would cause a CI failure when testing with
+// our MSRV.
+#![allow(unknown_lints, non_local_definitions, unreachable_patterns)]
+#![deny(renamed_and_removed_lints)]
+#![deny(
+ anonymous_parameters,
+ deprecated_in_future,
+ late_bound_lifetime_arguments,
+ missing_copy_implementations,
+ missing_debug_implementations,
+ missing_docs,
+ path_statements,
+ patterns_in_fns_without_body,
+ rust_2018_idioms,
+ trivial_numeric_casts,
+ unreachable_pub,
+ unsafe_op_in_unsafe_fn,
+ unused_extern_crates,
+ // We intentionally choose not to deny `unused_qualifications`. When items
+ // are added to the prelude (e.g., `core::mem::size_of`), this has the
+ // consequence of making some uses trigger this lint on the latest toolchain
+ // (e.g., `mem::size_of`), but fixing it (e.g. by replacing with `size_of`)
+ // does not work on older toolchains.
+ //
+ // We tested a more complicated fix in #1413, but ultimately decided that,
+ // since this lint is just a minor style lint, the complexity isn't worth it
+ // - it's fine to occasionally have unused qualifications slip through,
+ // especially since these do not affect our user-facing API in any way.
+ variant_size_differences
+)]
+#![cfg_attr(
+ __ZEROCOPY_INTERNAL_USE_ONLY_NIGHTLY_FEATURES_IN_TESTS,
+ deny(fuzzy_provenance_casts, lossy_provenance_casts)
+)]
+#![deny(
+ clippy::all,
+ clippy::alloc_instead_of_core,
+ clippy::arithmetic_side_effects,
+ clippy::as_underscore,
+ clippy::assertions_on_result_states,
+ clippy::as_conversions,
+ clippy::correctness,
+ clippy::dbg_macro,
+ clippy::decimal_literal_representation,
+ clippy::double_must_use,
+ clippy::get_unwrap,
+ clippy::indexing_slicing,
+ clippy::missing_inline_in_public_items,
+ clippy::missing_safety_doc,
+ clippy::multiple_unsafe_ops_per_block,
+ clippy::must_use_candidate,
+ clippy::must_use_unit,
+ clippy::obfuscated_if_else,
+ clippy::perf,
+ clippy::print_stdout,
+ clippy::return_self_not_must_use,
+ clippy::std_instead_of_core,
+ clippy::style,
+ clippy::suspicious,
+ clippy::todo,
+ clippy::undocumented_unsafe_blocks,
+ clippy::unimplemented,
+ clippy::unnested_or_patterns,
+ clippy::unwrap_used,
+ clippy::use_debug
+)]
+// `clippy::incompatible_msrv` (implied by `clippy::suspicious`): This sometimes
+// has false positives, and we test on our MSRV in CI, so it doesn't help us
+// anyway.
+#![allow(clippy::needless_lifetimes, clippy::type_complexity, clippy::incompatible_msrv)]
+#![deny(
+ rustdoc::bare_urls,
+ rustdoc::broken_intra_doc_links,
+ rustdoc::invalid_codeblock_attributes,
+ rustdoc::invalid_html_tags,
+ rustdoc::invalid_rust_codeblocks,
+ rustdoc::missing_crate_level_docs,
+ rustdoc::private_intra_doc_links
+)]
+// In test code, it makes sense to weight more heavily towards concise, readable
+// code over correct or debuggable code.
+#![cfg_attr(any(test, kani), allow(
+ // In tests, you get line numbers and have access to source code, so panic
+ // messages are less important. You also often unwrap a lot, which would
+ // make expect'ing instead very verbose.
+ clippy::unwrap_used,
+ // In tests, there's no harm to "panic risks" - the worst that can happen is
+ // that your test will fail, and you'll fix it. By contrast, panic risks in
+ // production code introduce the possibly of code panicking unexpectedly "in
+ // the field".
+ clippy::arithmetic_side_effects,
+ clippy::indexing_slicing,
+))]
+#![cfg_attr(not(any(test, kani, feature = "std")), no_std)]
+#![cfg_attr(
+ all(feature = "simd-nightly", target_arch = "arm"),
+ feature(stdarch_arm_neon_intrinsics)
+)]
+#![cfg_attr(
+ all(feature = "simd-nightly", any(target_arch = "powerpc", target_arch = "powerpc64")),
+ feature(stdarch_powerpc)
+)]
+#![cfg_attr(feature = "float-nightly", feature(f16, f128))]
+#![cfg_attr(doc_cfg, feature(doc_cfg))]
+#![cfg_attr(__ZEROCOPY_INTERNAL_USE_ONLY_NIGHTLY_FEATURES_IN_TESTS, feature(coverage_attribute))]
+#![cfg_attr(
+ any(__ZEROCOPY_INTERNAL_USE_ONLY_NIGHTLY_FEATURES_IN_TESTS, miri),
+ feature(layout_for_ptr)
+)]
+#![cfg_attr(all(test, __ZEROCOPY_INTERNAL_USE_ONLY_NIGHTLY_FEATURES_IN_TESTS), feature(test))]
+
+// This is a hack to allow zerocopy-derive derives to work in this crate. They
+// assume that zerocopy is linked as an extern crate, so they access items from
+// it as `zerocopy::Xxx`. This makes that still work.
+#[cfg(any(feature = "derive", test))]
+extern crate self as zerocopy;
+
+#[cfg(all(test, __ZEROCOPY_INTERNAL_USE_ONLY_NIGHTLY_FEATURES_IN_TESTS))]
+extern crate test;
+
+#[doc(hidden)]
+#[macro_use]
+pub mod util;
+
+pub mod byte_slice;
+pub mod byteorder;
+mod deprecated;
+
+#[cfg(__ZEROCOPY_INTERNAL_USE_ONLY_DEV_MODE)]
+pub mod doctests;
+
+// This module is `pub` so that zerocopy's error types and error handling
+// documentation is grouped together in a cohesive module. In practice, we
+// expect most users to use the re-export of `error`'s items to avoid identifier
+// stuttering.
+pub mod error;
+mod impls;
+#[doc(hidden)]
+pub mod layout;
+mod macros;
+#[cfg_attr(not(zerocopy_unstable_ptr), doc(hidden))]
+#[cfg_attr(doc_cfg, doc(cfg(zerocopy_unstable_ptr)))]
+pub mod pointer;
+mod r#ref;
+mod split_at;
+// FIXME(#252): If we make this pub, come up with a better name.
+mod wrappers;
+
+use core::{
+ cell::{Cell, UnsafeCell},
+ cmp::Ordering,
+ fmt::{self, Debug, Display, Formatter},
+ hash::Hasher,
+ marker::PhantomData,
+ mem::{self, ManuallyDrop, MaybeUninit as CoreMaybeUninit},
+ num::{
+ NonZeroI128, NonZeroI16, NonZeroI32, NonZeroI64, NonZeroI8, NonZeroIsize, NonZeroU128,
+ NonZeroU16, NonZeroU32, NonZeroU64, NonZeroU8, NonZeroUsize, Wrapping,
+ },
+ ops::{Deref, DerefMut},
+ ptr::{self, NonNull},
+ slice,
+};
+#[cfg(feature = "std")]
+use std::io;
+
+#[doc(hidden)]
+pub use crate::pointer::{
+ invariant::{self, BecauseExclusive},
+ PtrInner,
+};
+pub use crate::{
+ byte_slice::*,
+ byteorder::*,
+ error::*,
+ r#ref::*,
+ split_at::{Split, SplitAt},
+ wrappers::*,
+};
+
+#[cfg(any(feature = "alloc", test, kani))]
+extern crate alloc;
+#[cfg(any(feature = "alloc", test))]
+use alloc::{boxed::Box, vec::Vec};
+#[cfg(any(feature = "alloc", test))]
+use core::alloc::Layout;
+
+// Used by `KnownLayout`.
+#[doc(hidden)]
+pub use crate::layout::*;
+// Used by `TryFromBytes::is_bit_valid`.
+#[doc(hidden)]
+pub use crate::pointer::{invariant::BecauseImmutable, Maybe, Ptr};
+// For each trait polyfill, as soon as the corresponding feature is stable, the
+// polyfill import will be unused because method/function resolution will prefer
+// the inherent method/function over a trait method/function. Thus, we suppress
+// the `unused_imports` warning.
+//
+// See the documentation on `util::polyfills` for more information.
+#[allow(unused_imports)]
+use crate::util::polyfills::{self, NonNullExt as _, NumExt as _};
+#[cfg_attr(not(zerocopy_unstable_ptr), doc(hidden))]
+#[cfg_attr(doc_cfg, doc(cfg(zerocopy_unstable_ptr)))]
+pub use crate::util::MetadataOf;
+
+#[cfg(all(test, not(__ZEROCOPY_INTERNAL_USE_ONLY_DEV_MODE)))]
+const _: () = {
+ #[deprecated = "Development of zerocopy using cargo is not supported. Please use `cargo.sh` or `win-cargo.bat` instead."]
+ #[allow(unused)]
+ const WARNING: () = ();
+ #[warn(deprecated)]
+ WARNING
+};
+
+/// Implements [`KnownLayout`].
+///
+/// This derive analyzes various aspects of a type's layout that are needed for
+/// some of zerocopy's APIs. It can be applied to structs, enums, and unions;
+/// e.g.:
+///
+/// ```
+/// # use zerocopy_derive::KnownLayout;
+/// #[derive(KnownLayout)]
+/// struct MyStruct {
+/// # /*
+/// ...
+/// # */
+/// }
+///
+/// #[derive(KnownLayout)]
+/// enum MyEnum {
+/// # V00,
+/// # /*
+/// ...
+/// # */
+/// }
+///
+/// #[derive(KnownLayout)]
+/// union MyUnion {
+/// # variant: u8,
+/// # /*
+/// ...
+/// # */
+/// }
+/// ```
+///
+/// # Limitations
+///
+/// This derive cannot currently be applied to unsized structs without an
+/// explicit `repr` attribute.
+///
+/// Some invocations of this derive run afoul of a [known bug] in Rust's type
+/// privacy checker. For example, this code:
+///
+/// ```compile_fail,E0446
+/// use zerocopy::*;
+/// # use zerocopy_derive::*;
+///
+/// #[derive(KnownLayout)]
+/// #[repr(C)]
+/// pub struct PublicType {
+/// leading: Foo,
+/// trailing: Bar,
+/// }
+///
+/// #[derive(KnownLayout)]
+/// struct Foo;
+///
+/// #[derive(KnownLayout)]
+/// struct Bar;
+/// ```
+///
+/// ...results in a compilation error:
+///
+/// ```text
+/// error[E0446]: private type `Bar` in public interface
+/// --> examples/bug.rs:3:10
+/// |
+/// 3 | #[derive(KnownLayout)]
+/// | ^^^^^^^^^^^ can't leak private type
+/// ...
+/// 14 | struct Bar;
+/// | ---------- `Bar` declared as private
+/// |
+/// = note: this error originates in the derive macro `KnownLayout` (in Nightly builds, run with -Z macro-backtrace for more info)
+/// ```
+///
+/// This issue arises when `#[derive(KnownLayout)]` is applied to `repr(C)`
+/// structs whose trailing field type is less public than the enclosing struct.
+///
+/// To work around this, mark the trailing field type `pub` and annotate it with
+/// `#[doc(hidden)]`; e.g.:
+///
+/// ```no_run
+/// use zerocopy::*;
+/// # use zerocopy_derive::*;
+///
+/// #[derive(KnownLayout)]
+/// #[repr(C)]
+/// pub struct PublicType {
+/// leading: Foo,
+/// trailing: Bar,
+/// }
+///
+/// #[derive(KnownLayout)]
+/// struct Foo;
+///
+/// #[doc(hidden)]
+/// #[derive(KnownLayout)]
+/// pub struct Bar; // <- `Bar` is now also `pub`
+/// ```
+///
+/// [known bug]: https://github.com/rust-lang/rust/issues/45713
+#[cfg(any(feature = "derive", test))]
+#[cfg_attr(doc_cfg, doc(cfg(feature = "derive")))]
+pub use zerocopy_derive::KnownLayout;
+// These exist so that code which was written against the old names will get
+// less confusing error messages when they upgrade to a more recent version of
+// zerocopy. On our MSRV toolchain, the error messages read, for example:
+//
+// error[E0603]: trait `FromZeroes` is private
+// --> examples/deprecated.rs:1:15
+// |
+// 1 | use zerocopy::FromZeroes;
+// | ^^^^^^^^^^ private trait
+// |
+// note: the trait `FromZeroes` is defined here
+// --> /Users/josh/workspace/zerocopy/src/lib.rs:1845:5
+// |
+// 1845 | use FromZeros as FromZeroes;
+// | ^^^^^^^^^^^^^^^^^^^^^^^
+//
+// The "note" provides enough context to make it easy to figure out how to fix
+// the error.
+#[allow(unused)]
+use {FromZeros as FromZeroes, IntoBytes as AsBytes, Ref as LayoutVerified};
+
+/// Indicates that zerocopy can reason about certain aspects of a type's layout.
+///
+/// This trait is required by many of zerocopy's APIs. It supports sized types,
+/// slices, and [slice DSTs](#dynamically-sized-types).
+///
+/// # Implementation
+///
+/// **Do not implement this trait yourself!** Instead, use
+/// [`#[derive(KnownLayout)]`][derive]; e.g.:
+///
+/// ```
+/// # use zerocopy_derive::KnownLayout;
+/// #[derive(KnownLayout)]
+/// struct MyStruct {
+/// # /*
+/// ...
+/// # */
+/// }
+///
+/// #[derive(KnownLayout)]
+/// enum MyEnum {
+/// # /*
+/// ...
+/// # */
+/// }
+///
+/// #[derive(KnownLayout)]
+/// union MyUnion {
+/// # variant: u8,
+/// # /*
+/// ...
+/// # */
+/// }
+/// ```
+///
+/// This derive performs a sophisticated analysis to deduce the layout
+/// characteristics of types. You **must** implement this trait via the derive.
+///
+/// # Dynamically-sized types
+///
+/// `KnownLayout` supports slice-based dynamically sized types ("slice DSTs").
+///
+/// A slice DST is a type whose trailing field is either a slice or another
+/// slice DST, rather than a type with fixed size. For example:
+///
+/// ```
+/// #[repr(C)]
+/// struct PacketHeader {
+/// # /*
+/// ...
+/// # */
+/// }
+///
+/// #[repr(C)]
+/// struct Packet {
+/// header: PacketHeader,
+/// body: [u8],
+/// }
+/// ```
+///
+/// It can be useful to think of slice DSTs as a generalization of slices - in
+/// other words, a normal slice is just the special case of a slice DST with
+/// zero leading fields. In particular:
+/// - Like slices, slice DSTs can have different lengths at runtime
+/// - Like slices, slice DSTs cannot be passed by-value, but only by reference
+/// or via other indirection such as `Box`
+/// - Like slices, a reference (or `Box`, or other pointer type) to a slice DST
+/// encodes the number of elements in the trailing slice field
+///
+/// ## Slice DST layout
+///
+/// Just like other composite Rust types, the layout of a slice DST is not
+/// well-defined unless it is specified using an explicit `#[repr(...)]`
+/// attribute such as `#[repr(C)]`. [Other representations are
+/// supported][reprs], but in this section, we'll use `#[repr(C)]` as our
+/// example.
+///
+/// A `#[repr(C)]` slice DST is laid out [just like sized `#[repr(C)]`
+/// types][repr-c-structs], but the presence of a variable-length field
+/// introduces the possibility of *dynamic padding*. In particular, it may be
+/// necessary to add trailing padding *after* the trailing slice field in order
+/// to satisfy the outer type's alignment, and the amount of padding required
+/// may be a function of the length of the trailing slice field. This is just a
+/// natural consequence of the normal `#[repr(C)]` rules applied to slice DSTs,
+/// but it can result in surprising behavior. For example, consider the
+/// following type:
+///
+/// ```
+/// #[repr(C)]
+/// struct Foo {
+/// a: u32,
+/// b: u8,
+/// z: [u16],
+/// }
+/// ```
+///
+/// Assuming that `u32` has alignment 4 (this is not true on all platforms),
+/// then `Foo` has alignment 4 as well. Here is the smallest possible value for
+/// `Foo`:
+///
+/// ```text
+/// byte offset | 01234567
+/// field | aaaab---
+/// ><
+/// ```
+///
+/// In this value, `z` has length 0. Abiding by `#[repr(C)]`, the lowest offset
+/// that we can place `z` at is 5, but since `z` has alignment 2, we need to
+/// round up to offset 6. This means that there is one byte of padding between
+/// `b` and `z`, then 0 bytes of `z` itself (denoted `><` in this diagram), and
+/// then two bytes of padding after `z` in order to satisfy the overall
+/// alignment of `Foo`. The size of this instance is 8 bytes.
+///
+/// What about if `z` has length 1?
+///
+/// ```text
+/// byte offset | 01234567
+/// field | aaaab-zz
+/// ```
+///
+/// In this instance, `z` has length 1, and thus takes up 2 bytes. That means
+/// that we no longer need padding after `z` in order to satisfy `Foo`'s
+/// alignment. We've now seen two different values of `Foo` with two different
+/// lengths of `z`, but they both have the same size - 8 bytes.
+///
+/// What about if `z` has length 2?
+///
+/// ```text
+/// byte offset | 012345678901
+/// field | aaaab-zzzz--
+/// ```
+///
+/// Now `z` has length 2, and thus takes up 4 bytes. This brings our un-padded
+/// size to 10, and so we now need another 2 bytes of padding after `z` to
+/// satisfy `Foo`'s alignment.
+///
+/// Again, all of this is just a logical consequence of the `#[repr(C)]` rules
+/// applied to slice DSTs, but it can be surprising that the amount of trailing
+/// padding becomes a function of the trailing slice field's length, and thus
+/// can only be computed at runtime.
+///
+/// [reprs]: https://doc.rust-lang.org/reference/type-layout.html#representations
+/// [repr-c-structs]: https://doc.rust-lang.org/reference/type-layout.html#reprc-structs
+///
+/// ## What is a valid size?
+///
+/// There are two places in zerocopy's API that we refer to "a valid size" of a
+/// type. In normal casts or conversions, where the source is a byte slice, we
+/// need to know whether the source byte slice is a valid size of the
+/// destination type. In prefix or suffix casts, we need to know whether *there
+/// exists* a valid size of the destination type which fits in the source byte
+/// slice and, if so, what the largest such size is.
+///
+/// As outlined above, a slice DST's size is defined by the number of elements
+/// in its trailing slice field. However, there is not necessarily a 1-to-1
+/// mapping between trailing slice field length and overall size. As we saw in
+/// the previous section with the type `Foo`, instances with both 0 and 1
+/// elements in the trailing `z` field result in a `Foo` whose size is 8 bytes.
+///
+/// When we say "x is a valid size of `T`", we mean one of two things:
+/// - If `T: Sized`, then we mean that `x == size_of::<T>()`
+/// - If `T` is a slice DST, then we mean that there exists a `len` such that the instance of
+/// `T` with `len` trailing slice elements has size `x`
+///
+/// When we say "largest possible size of `T` that fits in a byte slice", we
+/// mean one of two things:
+/// - If `T: Sized`, then we mean `size_of::<T>()` if the byte slice is at least
+/// `size_of::<T>()` bytes long
+/// - If `T` is a slice DST, then we mean to consider all values, `len`, such
+/// that the instance of `T` with `len` trailing slice elements fits in the
+/// byte slice, and to choose the largest such `len`, if any
+///
+///
+/// # Safety
+///
+/// This trait does not convey any safety guarantees to code outside this crate.
+///
+/// You must not rely on the `#[doc(hidden)]` internals of `KnownLayout`. Future
+/// releases of zerocopy may make backwards-breaking changes to these items,
+/// including changes that only affect soundness, which may cause code which
+/// uses those items to silently become unsound.
+///
+#[cfg_attr(feature = "derive", doc = "[derive]: zerocopy_derive::KnownLayout")]
+#[cfg_attr(
+ not(feature = "derive"),
+ doc = concat!("[derive]: https://docs.rs/zerocopy/", env!("CARGO_PKG_VERSION"), "/zerocopy/derive.KnownLayout.html"),
+)]
+#[cfg_attr(
+ not(no_zerocopy_diagnostic_on_unimplemented_1_78_0),
+ diagnostic::on_unimplemented(note = "Consider adding `#[derive(KnownLayout)]` to `{Self}`")
+)]
+pub unsafe trait KnownLayout {
+ // The `Self: Sized` bound makes it so that `KnownLayout` can still be
+ // object safe. It's not currently object safe thanks to `const LAYOUT`, and
+ // it likely won't be in the future, but there's no reason not to be
+ // forwards-compatible with object safety.
+ #[doc(hidden)]
+ fn only_derive_is_allowed_to_implement_this_trait()
+ where
+ Self: Sized;
+
+ /// The type of metadata stored in a pointer to `Self`.
+ ///
+ /// This is `()` for sized types and [`usize`] for slice DSTs.
+ type PointerMetadata: PointerMetadata;
+
+ /// A maybe-uninitialized analog of `Self`
+ ///
+ /// # Safety
+ ///
+ /// `Self::LAYOUT` and `Self::MaybeUninit::LAYOUT` are identical.
+ /// `Self::MaybeUninit` admits uninitialized bytes in all positions.
+ #[doc(hidden)]
+ type MaybeUninit: ?Sized + KnownLayout<PointerMetadata = Self::PointerMetadata>;
+
+ /// The layout of `Self`.
+ ///
+ /// # Safety
+ ///
+ /// Callers may assume that `LAYOUT` accurately reflects the layout of
+ /// `Self`. In particular:
+ /// - `LAYOUT.align` is equal to `Self`'s alignment
+ /// - If `Self: Sized`, then `LAYOUT.size_info == SizeInfo::Sized { size }`
+ /// where `size == size_of::<Self>()`
+ /// - If `Self` is a slice DST, then `LAYOUT.size_info ==
+ /// SizeInfo::SliceDst(slice_layout)` where:
+ /// - The size, `size`, of an instance of `Self` with `elems` trailing
+ /// slice elements is equal to `slice_layout.offset +
+ /// slice_layout.elem_size * elems` rounded up to the nearest multiple
+ /// of `LAYOUT.align`
+ /// - For such an instance, any bytes in the range `[slice_layout.offset +
+ /// slice_layout.elem_size * elems, size)` are padding and must not be
+ /// assumed to be initialized
+ #[doc(hidden)]
+ const LAYOUT: DstLayout;
+
+ /// SAFETY: The returned pointer has the same address and provenance as
+ /// `bytes`. If `Self` is a DST, the returned pointer's referent has `elems`
+ /// elements in its trailing slice.
+ #[doc(hidden)]
+ fn raw_from_ptr_len(bytes: NonNull<u8>, meta: Self::PointerMetadata) -> NonNull<Self>;
+
+ /// Extracts the metadata from a pointer to `Self`.
+ ///
+ /// # Safety
+ ///
+ /// `pointer_to_metadata` always returns the correct metadata stored in
+ /// `ptr`.
+ #[doc(hidden)]
+ fn pointer_to_metadata(ptr: *mut Self) -> Self::PointerMetadata;
+
+ /// Computes the length of the byte range addressed by `ptr`.
+ ///
+ /// Returns `None` if the resulting length would not fit in an `usize`.
+ ///
+ /// # Safety
+ ///
+ /// Callers may assume that `size_of_val_raw` always returns the correct
+ /// size.
+ ///
+ /// Callers may assume that, if `ptr` addresses a byte range whose length
+ /// fits in an `usize`, this will return `Some`.
+ #[doc(hidden)]
+ #[must_use]
+ #[inline(always)]
+ fn size_of_val_raw(ptr: NonNull<Self>) -> Option<usize> {
+ let meta = Self::pointer_to_metadata(ptr.as_ptr());
+ // SAFETY: `size_for_metadata` promises to only return `None` if the
+ // resulting size would not fit in a `usize`.
+ Self::size_for_metadata(meta)
+ }
+
+ #[doc(hidden)]
+ #[must_use]
+ #[inline(always)]
+ fn raw_dangling() -> NonNull<Self> {
+ let meta = Self::PointerMetadata::from_elem_count(0);
+ Self::raw_from_ptr_len(NonNull::dangling(), meta)
+ }
+
+ /// Computes the size of an object of type `Self` with the given pointer
+ /// metadata.
+ ///
+ /// # Safety
+ ///
+ /// `size_for_metadata` promises to return `None` if and only if the
+ /// resulting size would not fit in a [`usize`]. Note that the returned size
+ /// could exceed the actual maximum valid size of an allocated object,
+ /// [`isize::MAX`].
+ ///
+ /// # Examples
+ ///
+ /// ```
+ /// use zerocopy::KnownLayout;
+ ///
+ /// assert_eq!(u8::size_for_metadata(()), Some(1));
+ /// assert_eq!(u16::size_for_metadata(()), Some(2));
+ /// assert_eq!(<[u8]>::size_for_metadata(42), Some(42));
+ /// assert_eq!(<[u16]>::size_for_metadata(42), Some(84));
+ ///
+ /// // This size exceeds the maximum valid object size (`isize::MAX`):
+ /// assert_eq!(<[u8]>::size_for_metadata(usize::MAX), Some(usize::MAX));
+ ///
+ /// // This size, if computed, would exceed `usize::MAX`:
+ /// assert_eq!(<[u16]>::size_for_metadata(usize::MAX), None);
+ /// ```
+ #[inline(always)]
+ fn size_for_metadata(meta: Self::PointerMetadata) -> Option<usize> {
+ meta.size_for_metadata(Self::LAYOUT)
+ }
+
+ /// Computes whether `meta` can describe a valid allocation of `Self`.
+ ///
+ /// # Safety
+ ///
+ /// `is_valid_metadata` promises to return `true` if and only if the size of
+ /// an allocation of `Self` with `meta` would not overflow an
+ /// [`isize::MAX`].
+ #[doc(hidden)]
+ #[inline(always)]
+ fn is_valid_metadata(meta: Self::PointerMetadata) -> bool {
+ meta.to_elem_count() <= maximum_trailing_slice_len::<Self>().to_elem_count()
+ }
+}
+
+/// Efficiently produces the [`TrailingSliceLayout`] of `T`.
+#[inline(always)]
+pub(crate) fn trailing_slice_layout<T>() -> TrailingSliceLayout
+where
+ T: ?Sized + KnownLayout<PointerMetadata = usize>,
+{
+ trait LayoutFacts {
+ const SIZE_INFO: TrailingSliceLayout;
+ }
+
+ impl<T: ?Sized> LayoutFacts for T
+ where
+ T: KnownLayout<PointerMetadata = usize>,
+ {
+ const SIZE_INFO: TrailingSliceLayout = match T::LAYOUT.size_info {
+ crate::SizeInfo::Sized { .. } => const_panic!("unreachable"),
+ crate::SizeInfo::SliceDst(info) => info,
+ };
+ }
+
+ T::SIZE_INFO
+}
+
+/// Efficiently produces the maximum trailing slice length `T`.
+#[inline(always)]
+pub(crate) fn maximum_trailing_slice_len<T>() -> usize
+where
+ T: ?Sized + KnownLayout,
+{
+ trait LayoutFacts {
+ const MAX_LEN: usize;
+ }
+
+ impl<T: ?Sized> LayoutFacts for T
+ where
+ T: KnownLayout,
+ {
+ const MAX_LEN: usize = match T::LAYOUT.size_info {
+ SizeInfo::SliceDst(TrailingSliceLayout { elem_size: 0, .. }) => usize::MAX,
+ _ => match T::LAYOUT.validate_cast_and_convert_metadata(
+ T::LAYOUT.align.get(),
+ DstLayout::MAX_SIZE,
+ CastType::Prefix,
+ ) {
+ Ok((elems, _)) => elems,
+ Err(_) => const_panic!("unreachable"),
+ },
+ };
+ }
+
+ T::MAX_LEN
+}
+
+/// The metadata associated with a [`KnownLayout`] type.
+#[doc(hidden)]
+pub trait PointerMetadata: Copy + Eq + Debug + Ord {
+ /// Constructs a `Self` from an element count.
+ ///
+ /// If `Self = ()`, this returns `()`. If `Self = usize`, this returns
+ /// `elems`. No other types are currently supported.
+ fn from_elem_count(elems: usize) -> Self;
+
+ /// Converts `self` to an element count.
+ ///
+ /// If `Self = ()`, this returns `0`. If `Self = usize`, this returns
+ /// `self`. No other types are currently supported.
+ fn to_elem_count(self) -> usize;
+
+ /// Computes the size of the object with the given layout and pointer
+ /// metadata.
+ ///
+ /// # Panics
+ ///
+ /// If `Self = ()`, `layout` must describe a sized type. If `Self = usize`,
+ /// `layout` must describe a slice DST. Otherwise, `size_for_metadata` may
+ /// panic.
+ ///
+ /// # Safety
+ ///
+ /// `size_for_metadata` promises to only return `None` if the resulting size
+ /// would not fit in a `usize`.
+ fn size_for_metadata(self, layout: DstLayout) -> Option<usize>;
+}
+
+impl PointerMetadata for () {
+ #[inline]
+ #[allow(clippy::unused_unit)]
+ fn from_elem_count(_elems: usize) -> () {}
+
+ #[inline]
+ fn to_elem_count(self) -> usize {
+ 0
+ }
+
+ #[inline]
+ fn size_for_metadata(self, layout: DstLayout) -> Option<usize> {
+ match layout.size_info {
+ SizeInfo::Sized { size } => Some(size),
+ // NOTE: This branch is unreachable, but we return `None` rather
+ // than `unreachable!()` to avoid generating panic paths.
+ SizeInfo::SliceDst(_) => None,
+ }
+ }
+}
+
+impl PointerMetadata for usize {
+ #[inline]
+ fn from_elem_count(elems: usize) -> usize {
+ elems
+ }
+
+ #[inline]
+ fn to_elem_count(self) -> usize {
+ self
+ }
+
+ #[inline]
+ fn size_for_metadata(self, layout: DstLayout) -> Option<usize> {
+ match layout.size_info {
+ SizeInfo::SliceDst(TrailingSliceLayout { offset, elem_size }) => {
+ let slice_len = elem_size.checked_mul(self)?;
+ let without_padding = offset.checked_add(slice_len)?;
+ without_padding.checked_add(util::padding_needed_for(without_padding, layout.align))
+ }
+ // NOTE: This branch is unreachable, but we return `None` rather
+ // than `unreachable!()` to avoid generating panic paths.
+ SizeInfo::Sized { .. } => None,
+ }
+ }
+}
+
+// SAFETY: Delegates safety to `DstLayout::for_slice`.
+unsafe impl<T> KnownLayout for [T] {
+ #[allow(clippy::missing_inline_in_public_items, dead_code)]
+ #[cfg_attr(
+ all(coverage_nightly, __ZEROCOPY_INTERNAL_USE_ONLY_NIGHTLY_FEATURES_IN_TESTS),
+ coverage(off)
+ )]
+ fn only_derive_is_allowed_to_implement_this_trait()
+ where
+ Self: Sized,
+ {
+ }
+
+ type PointerMetadata = usize;
+
+ // SAFETY: `CoreMaybeUninit<T>::LAYOUT` and `T::LAYOUT` are identical
+ // because `CoreMaybeUninit<T>` has the same size and alignment as `T` [1].
+ // Consequently, `[CoreMaybeUninit<T>]::LAYOUT` and `[T]::LAYOUT` are
+ // identical, because they both lack a fixed-sized prefix and because they
+ // inherit the alignments of their inner element type (which are identical)
+ // [2][3].
+ //
+ // `[CoreMaybeUninit<T>]` admits uninitialized bytes at all positions
+ // because `CoreMaybeUninit<T>` admits uninitialized bytes at all positions
+ // and because the inner elements of `[CoreMaybeUninit<T>]` are laid out
+ // back-to-back [2][3].
+ //
+ // [1] Per https://doc.rust-lang.org/1.81.0/std/mem/union.MaybeUninit.html#layout-1:
+ //
+ // `MaybeUninit<T>` is guaranteed to have the same size, alignment, and ABI as
+ // `T`
+ //
+ // [2] Per https://doc.rust-lang.org/1.82.0/reference/type-layout.html#slice-layout:
+ //
+ // Slices have the same layout as the section of the array they slice.
+ //
+ // [3] Per https://doc.rust-lang.org/1.82.0/reference/type-layout.html#array-layout:
+ //
+ // An array of `[T; N]` has a size of `size_of::<T>() * N` and the same
+ // alignment of `T`. Arrays are laid out so that the zero-based `nth`
+ // element of the array is offset from the start of the array by `n *
+ // size_of::<T>()` bytes.
+ type MaybeUninit = [CoreMaybeUninit<T>];
+
+ const LAYOUT: DstLayout = DstLayout::for_slice::<T>();
+
+ // SAFETY: `.cast` preserves address and provenance. The returned pointer
+ // refers to an object with `elems` elements by construction.
+ #[inline(always)]
+ fn raw_from_ptr_len(data: NonNull<u8>, elems: usize) -> NonNull<Self> {
+ // FIXME(#67): Remove this allow. See NonNullExt for more details.
+ #[allow(unstable_name_collisions)]
+ NonNull::slice_from_raw_parts(data.cast::<T>(), elems)
+ }
+
+ #[inline(always)]
+ fn pointer_to_metadata(ptr: *mut [T]) -> usize {
+ #[allow(clippy::as_conversions)]
+ let slc = ptr as *const [()];
+
+ // SAFETY:
+ // - `()` has alignment 1, so `slc` is trivially aligned.
+ // - `slc` was derived from a non-null pointer.
+ // - The size is 0 regardless of the length, so it is sound to
+ // materialize a reference regardless of location.
+ // - By invariant, `self.ptr` has valid provenance.
+ let slc = unsafe { &*slc };
+
+ // This is correct because the preceding `as` cast preserves the number
+ // of slice elements. [1]
+ //
+ // [1] Per https://doc.rust-lang.org/reference/expressions/operator-expr.html#pointer-to-pointer-cast:
+ //
+ // For slice types like `[T]` and `[U]`, the raw pointer types `*const
+ // [T]`, `*mut [T]`, `*const [U]`, and `*mut [U]` encode the number of
+ // elements in this slice. Casts between these raw pointer types
+ // preserve the number of elements. ... The same holds for `str` and
+ // any compound type whose unsized tail is a slice type, such as
+ // struct `Foo(i32, [u8])` or `(u64, Foo)`.
+ slc.len()
+ }
+}
+
+#[rustfmt::skip]
+impl_known_layout!(
+ (),
+ u8, i8, u16, i16, u32, i32, u64, i64, u128, i128, usize, isize, f32, f64,
+ bool, char,
+ NonZeroU8, NonZeroI8, NonZeroU16, NonZeroI16, NonZeroU32, NonZeroI32,
+ NonZeroU64, NonZeroI64, NonZeroU128, NonZeroI128, NonZeroUsize, NonZeroIsize
+);
+#[rustfmt::skip]
+#[cfg(feature = "float-nightly")]
+impl_known_layout!(
+ #[cfg_attr(doc_cfg, doc(cfg(feature = "float-nightly")))]
+ f16,
+ #[cfg_attr(doc_cfg, doc(cfg(feature = "float-nightly")))]
+ f128
+);
+#[rustfmt::skip]
+impl_known_layout!(
+ T => Option<T>,
+ T: ?Sized => PhantomData<T>,
+ T => Wrapping<T>,
+ T => CoreMaybeUninit<T>,
+ T: ?Sized => *const T,
+ T: ?Sized => *mut T,
+ T: ?Sized => &'_ T,
+ T: ?Sized => &'_ mut T,
+);
+impl_known_layout!(const N: usize, T => [T; N]);
+
+// SAFETY: `str` has the same representation as `[u8]`. `ManuallyDrop<T>` [1],
+// `UnsafeCell<T>` [2], and `Cell<T>` [3] have the same representation as `T`.
+//
+// [1] Per https://doc.rust-lang.org/1.85.0/std/mem/struct.ManuallyDrop.html:
+//
+// `ManuallyDrop<T>` is guaranteed to have the same layout and bit validity as
+// `T`
+//
+// [2] Per https://doc.rust-lang.org/1.85.0/core/cell/struct.UnsafeCell.html#memory-layout:
+//
+// `UnsafeCell<T>` has the same in-memory representation as its inner type
+// `T`.
+//
+// [3] Per https://doc.rust-lang.org/1.85.0/core/cell/struct.Cell.html#memory-layout:
+//
+// `Cell<T>` has the same in-memory representation as `T`.
+#[allow(clippy::multiple_unsafe_ops_per_block)]
+const _: () = unsafe {
+ unsafe_impl_known_layout!(
+ #[repr([u8])]
+ str
+ );
+ unsafe_impl_known_layout!(T: ?Sized + KnownLayout => #[repr(T)] ManuallyDrop<T>);
+ unsafe_impl_known_layout!(T: ?Sized + KnownLayout => #[repr(T)] UnsafeCell<T>);
+ unsafe_impl_known_layout!(T: ?Sized + KnownLayout => #[repr(T)] Cell<T>);
+};
+
+// SAFETY:
+// - By consequence of the invariant on `T::MaybeUninit` that `T::LAYOUT` and
+// `T::MaybeUninit::LAYOUT` are equal, `T` and `T::MaybeUninit` have the same:
+// - Fixed prefix size
+// - Alignment
+// - (For DSTs) trailing slice element size
+// - By consequence of the above, referents `T::MaybeUninit` and `T` have the
+// require the same kind of pointer metadata, and thus it is valid to perform
+// an `as` cast from `*mut T` and `*mut T::MaybeUninit`, and this operation
+// preserves referent size (ie, `size_of_val_raw`).
+const _: () = unsafe {
+ unsafe_impl_known_layout!(T: ?Sized + KnownLayout => #[repr(T::MaybeUninit)] MaybeUninit<T>)
+};
+
+// FIXME(#196, #2856): Eventually, we'll want to support enums variants and
+// union fields being treated uniformly since they behave similarly to each
+// other in terms of projecting validity – specifically, for a type `T` with
+// validity `V`, if `T` is a struct type, then its fields straightforwardly also
+// have validity `V`. By contrast, if `T` is an enum or union type, then
+// validity is not straightforwardly recursive in this way.
+#[doc(hidden)]
+pub const STRUCT_VARIANT_ID: i128 = -1;
+#[doc(hidden)]
+pub const UNION_VARIANT_ID: i128 = -2;
+#[doc(hidden)]
+pub const REPR_C_UNION_VARIANT_ID: i128 = -3;
+
+/// # Safety
+///
+/// `Self::ProjectToTag` must satisfy its safety invariant.
+#[doc(hidden)]
+pub unsafe trait HasTag {
+ fn only_derive_is_allowed_to_implement_this_trait()
+ where
+ Self: Sized;
+
+ /// The type's enum tag, or `()` for non-enum types.
+ type Tag: Immutable;
+
+ /// A pointer projection from `Self` to its tag.
+ ///
+ /// # Safety
+ ///
+ /// It must be the case that, for all `slf: Ptr<'_, Self, I>`, it is sound
+ /// to project from `slf` to `Ptr<'_, Self::Tag, I>` using this projection.
+ type ProjectToTag: pointer::cast::Project<Self, Self::Tag>;
+}
+
+/// Projects a given field from `Self`.
+///
+/// All implementations of `HasField` for a particular field `f` in `Self`
+/// should use the same `Field` type; this ensures that `Field` is inferable
+/// given an explicit `VARIANT_ID` and `FIELD_ID`.
+///
+/// # Safety
+///
+/// A field `f` is `HasField` for `Self` if and only if:
+///
+/// - If `Self` has the layout of a struct or union type, then `VARIANT_ID` is
+/// `STRUCT_VARIANT_ID` or `UNION_VARIANT_ID` respectively; otherwise, if
+/// `Self` has the layout of an enum type, `VARIANT_ID` is the numerical index
+/// of the enum variant in which `f` appears. Note that `Self` does not need
+/// to actually *be* such a type – it just needs to have the same layout as
+/// such a type. For example, a `#[repr(transparent)]` wrapper around an enum
+/// has the same layout as that enum.
+/// - If `f` has name `n`, `FIELD_ID` is `zerocopy::ident_id!(n)`; otherwise,
+/// if `f` is at index `i`, `FIELD_ID` is `zerocopy::ident_id!(i)`.
+/// - `Field` is a type with the same visibility as `f`.
+/// - `Type` has the same type as `f`.
+///
+/// The caller must **not** assume that a pointer's referent being aligned
+/// implies that calling `project` on that pointer will result in a pointer to
+/// an aligned referent. For example, `HasField` may be implemented for
+/// `#[repr(packed)]` structs.
+///
+/// The implementation of `project` must satisfy its safety post-condition.
+#[doc(hidden)]
+pub unsafe trait HasField<Field, const VARIANT_ID: i128, const FIELD_ID: i128>:
+ HasTag
+{
+ fn only_derive_is_allowed_to_implement_this_trait()
+ where
+ Self: Sized;
+
+ /// The type of the field.
+ type Type: ?Sized;
+
+ /// Projects from `slf` to the field.
+ ///
+ /// Users should generally not call `project` directly, and instead should
+ /// use high-level APIs like [`PtrInner::project`] or [`Ptr::project`].
+ ///
+ /// # Safety
+ ///
+ /// The returned pointer refers to a non-strict subset of the bytes of
+ /// `slf`'s referent, and has the same provenance as `slf`.
+ #[must_use]
+ fn project(slf: PtrInner<'_, Self>) -> *mut Self::Type;
+}
+
+/// Projects a given field from `Self`.
+///
+/// Implementations of this trait encode the conditions under which a field can
+/// be projected from a `Ptr<'_, Self, I>`, and how the invariants of that
+/// [`Ptr`] (`I`) determine the invariants of pointers projected from it. In
+/// other words, it is a type-level function over invariants; `I` goes in,
+/// `Self::Invariants` comes out.
+///
+/// # Safety
+///
+/// `T: ProjectField<Field, I, VARIANT_ID, FIELD_ID>` if, for a
+/// `ptr: Ptr<'_, T, I>` such that `T::is_projectable(ptr).is_ok()`,
+/// `<T as HasField<Field, VARIANT_ID, FIELD_ID>>::project(ptr.as_inner())`
+/// conforms to `T::Invariants`.
+#[doc(hidden)]
+pub unsafe trait ProjectField<Field, I, const VARIANT_ID: i128, const FIELD_ID: i128>:
+ HasField<Field, VARIANT_ID, FIELD_ID>
+where
+ I: invariant::Invariants,
+{
+ fn only_derive_is_allowed_to_implement_this_trait()
+ where
+ Self: Sized;
+
+ /// The invariants of the projected field pointer, with respect to the
+ /// invariants, `I`, of the containing pointer. The aliasing dimension of
+ /// the invariants is guaranteed to remain unchanged.
+ type Invariants: invariant::Invariants<Aliasing = I::Aliasing>;
+
+ /// The failure mode of projection. `()` if the projection is fallible,
+ /// otherwise [`core::convert::Infallible`].
+ type Error;
+
+ /// Is the given field projectable from `ptr`?
+ ///
+ /// If a field with [`Self::Invariants`] is projectable from the referent,
+ /// this function produces an `Ok(ptr)` from which the projection can be
+ /// made; otherwise `Err`.
+ ///
+ /// This method must be overriden if the field's projectability depends on
+ /// the value of the bytes in `ptr`.
+ #[inline(always)]
+ fn is_projectable<'a>(_ptr: Ptr<'a, Self::Tag, I>) -> Result<(), Self::Error> {
+ trait IsInfallible {
+ const IS_INFALLIBLE: bool;
+ }
+
+ struct Projection<T, Field, I, const VARIANT_ID: i128, const FIELD_ID: i128>(
+ PhantomData<(Field, I, T)>,
+ )
+ where
+ T: ?Sized + HasField<Field, VARIANT_ID, FIELD_ID>,
+ I: invariant::Invariants;
+
+ impl<T, Field, I, const VARIANT_ID: i128, const FIELD_ID: i128> IsInfallible
+ for Projection<T, Field, I, VARIANT_ID, FIELD_ID>
+ where
+ T: ?Sized + HasField<Field, VARIANT_ID, FIELD_ID>,
+ I: invariant::Invariants,
+ {
+ const IS_INFALLIBLE: bool = {
+ let is_infallible = match VARIANT_ID {
+ // For nondestructive projections of struct and union
+ // fields, the projected field's satisfaction of
+ // `Invariants` does not depend on the value of the
+ // referent. This default implementation of `is_projectable`
+ // is non-destructive, as it does not overwrite any part of
+ // the referent.
+ crate::STRUCT_VARIANT_ID | crate::UNION_VARIANT_ID => true,
+ _enum_variant => {
+ use crate::invariant::{Validity, ValidityKind};
+ match I::Validity::KIND {
+ // The `Uninit` and `Initialized` validity
+ // invariants do not depend on the enum's tag. In
+ // particular, we don't actually care about what
+ // variant is present – we can treat *any* range of
+ // uninitialized or initialized memory as containing
+ // an uninitialized or initialized instance of *any*
+ // type – the type itself is irrelevant.
+ ValidityKind::Uninit | ValidityKind::Initialized => true,
+ // The projectability of an enum field from an
+ // `AsInitialized` or `Valid` state is a dynamic
+ // property of its tag.
+ ValidityKind::AsInitialized | ValidityKind::Valid => false,
+ }
+ }
+ };
+ const_assert!(is_infallible);
+ is_infallible
+ };
+ }
+
+ const_assert!(
+ <Projection<Self, Field, I, VARIANT_ID, FIELD_ID> as IsInfallible>::IS_INFALLIBLE
+ );
+
+ Ok(())
+ }
+}
+
+/// Analyzes whether a type is [`FromZeros`].
+///
+/// This derive analyzes, at compile time, whether the annotated type satisfies
+/// the [safety conditions] of `FromZeros` and implements `FromZeros` and its
+/// supertraits if it is sound to do so. This derive can be applied to structs,
+/// enums, and unions; e.g.:
+///
+/// ```
+/// # use zerocopy_derive::{FromZeros, Immutable};
+/// #[derive(FromZeros)]
+/// struct MyStruct {
+/// # /*
+/// ...
+/// # */
+/// }
+///
+/// #[derive(FromZeros)]
+/// #[repr(u8)]
+/// enum MyEnum {
+/// # Variant0,
+/// # /*
+/// ...
+/// # */
+/// }
+///
+/// #[derive(FromZeros, Immutable)]
+/// union MyUnion {
+/// # variant: u8,
+/// # /*
+/// ...
+/// # */
+/// }
+/// ```
+///
+/// [safety conditions]: trait@FromZeros#safety
+///
+/// # Analysis
+///
+/// *This section describes, roughly, the analysis performed by this derive to
+/// determine whether it is sound to implement `FromZeros` for a given type.
+/// Unless you are modifying the implementation of this derive, or attempting to
+/// manually implement `FromZeros` for a type yourself, you don't need to read
+/// this section.*
+///
+/// If a type has the following properties, then this derive can implement
+/// `FromZeros` for that type:
+///
+/// - If the type is a struct, all of its fields must be `FromZeros`.
+/// - If the type is an enum:
+/// - It must have a defined representation (`repr`s `C`, `u8`, `u16`, `u32`,
+/// `u64`, `usize`, `i8`, `i16`, `i32`, `i64`, or `isize`).
+/// - It must have a variant with a discriminant/tag of `0`, and its fields
+/// must be `FromZeros`. See [the reference] for a description of
+/// discriminant values are specified.
+/// - The fields of that variant must be `FromZeros`.
+///
+/// This analysis is subject to change. Unsafe code may *only* rely on the
+/// documented [safety conditions] of `FromZeros`, and must *not* rely on the
+/// implementation details of this derive.
+///
+/// [the reference]: https://doc.rust-lang.org/reference/items/enumerations.html#custom-discriminant-values-for-fieldless-enumerations
+///
+/// ## Why isn't an explicit representation required for structs?
+///
+/// Neither this derive, nor the [safety conditions] of `FromZeros`, requires
+/// that structs are marked with `#[repr(C)]`.
+///
+/// Per the [Rust reference](reference),
+///
+/// > The representation of a type can change the padding between fields, but
+/// > does not change the layout of the fields themselves.
+///
+/// [reference]: https://doc.rust-lang.org/reference/type-layout.html#representations
+///
+/// Since the layout of structs only consists of padding bytes and field bytes,
+/// a struct is soundly `FromZeros` if:
+/// 1. its padding is soundly `FromZeros`, and
+/// 2. its fields are soundly `FromZeros`.
+///
+/// The answer to the first question is always yes: padding bytes do not have
+/// any validity constraints. A [discussion] of this question in the Unsafe Code
+/// Guidelines Working Group concluded that it would be virtually unimaginable
+/// for future versions of rustc to add validity constraints to padding bytes.
+///
+/// [discussion]: https://github.com/rust-lang/unsafe-code-guidelines/issues/174
+///
+/// Whether a struct is soundly `FromZeros` therefore solely depends on whether
+/// its fields are `FromZeros`.
+// FIXME(#146): Document why we don't require an enum to have an explicit `repr`
+// attribute.
+#[cfg(any(feature = "derive", test))]
+#[cfg_attr(doc_cfg, doc(cfg(feature = "derive")))]
+pub use zerocopy_derive::FromZeros;
+/// Analyzes whether a type is [`Immutable`].
+///
+/// This derive analyzes, at compile time, whether the annotated type satisfies
+/// the [safety conditions] of `Immutable` and implements `Immutable` if it is
+/// sound to do so. This derive can be applied to structs, enums, and unions;
+/// e.g.:
+///
+/// ```
+/// # use zerocopy_derive::Immutable;
+/// #[derive(Immutable)]
+/// struct MyStruct {
+/// # /*
+/// ...
+/// # */
+/// }
+///
+/// #[derive(Immutable)]
+/// enum MyEnum {
+/// # Variant0,
+/// # /*
+/// ...
+/// # */
+/// }
+///
+/// #[derive(Immutable)]
+/// union MyUnion {
+/// # variant: u8,
+/// # /*
+/// ...
+/// # */
+/// }
+/// ```
+///
+/// # Analysis
+///
+/// *This section describes, roughly, the analysis performed by this derive to
+/// determine whether it is sound to implement `Immutable` for a given type.
+/// Unless you are modifying the implementation of this derive, you don't need
+/// to read this section.*
+///
+/// If a type has the following properties, then this derive can implement
+/// `Immutable` for that type:
+///
+/// - All fields must be `Immutable`.
+///
+/// This analysis is subject to change. Unsafe code may *only* rely on the
+/// documented [safety conditions] of `Immutable`, and must *not* rely on the
+/// implementation details of this derive.
+///
+/// [safety conditions]: trait@Immutable#safety
+#[cfg(any(feature = "derive", test))]
+#[cfg_attr(doc_cfg, doc(cfg(feature = "derive")))]
+pub use zerocopy_derive::Immutable;
+
+/// Types which are free from interior mutability.
+///
+/// `T: Immutable` indicates that `T` does not permit interior mutation, except
+/// by ownership or an exclusive (`&mut`) borrow.
+///
+/// # Implementation
+///
+/// **Do not implement this trait yourself!** Instead, use
+/// [`#[derive(Immutable)]`][derive] (requires the `derive` Cargo feature);
+/// e.g.:
+///
+/// ```
+/// # use zerocopy_derive::Immutable;
+/// #[derive(Immutable)]
+/// struct MyStruct {
+/// # /*
+/// ...
+/// # */
+/// }
+///
+/// #[derive(Immutable)]
+/// enum MyEnum {
+/// # /*
+/// ...
+/// # */
+/// }
+///
+/// #[derive(Immutable)]
+/// union MyUnion {
+/// # variant: u8,
+/// # /*
+/// ...
+/// # */
+/// }
+/// ```
+///
+/// This derive performs a sophisticated, compile-time safety analysis to
+/// determine whether a type is `Immutable`.
+///
+/// # Safety
+///
+/// Unsafe code outside of this crate must not make any assumptions about `T`
+/// based on `T: Immutable`. We reserve the right to relax the requirements for
+/// `Immutable` in the future, and if unsafe code outside of this crate makes
+/// assumptions based on `T: Immutable`, future relaxations may cause that code
+/// to become unsound.
+///
+// # Safety (Internal)
+//
+// If `T: Immutable`, unsafe code *inside of this crate* may assume that, given
+// `t: &T`, `t` does not permit interior mutation of its referent. Because
+// [`UnsafeCell`] is the only type which permits interior mutation, it is
+// sufficient (though not necessary) to guarantee that `T` contains no
+// `UnsafeCell`s.
+//
+// [`UnsafeCell`]: core::cell::UnsafeCell
+#[cfg_attr(
+ feature = "derive",
+ doc = "[derive]: zerocopy_derive::Immutable",
+ doc = "[derive-analysis]: zerocopy_derive::Immutable#analysis"
+)]
+#[cfg_attr(
+ not(feature = "derive"),
+ doc = concat!("[derive]: https://docs.rs/zerocopy/", env!("CARGO_PKG_VERSION"), "/zerocopy/derive.Immutable.html"),
+ doc = concat!("[derive-analysis]: https://docs.rs/zerocopy/", env!("CARGO_PKG_VERSION"), "/zerocopy/derive.Immutable.html#analysis"),
+)]
+#[cfg_attr(
+ not(no_zerocopy_diagnostic_on_unimplemented_1_78_0),
+ diagnostic::on_unimplemented(note = "Consider adding `#[derive(Immutable)]` to `{Self}`")
+)]
+pub unsafe trait Immutable {
+ // The `Self: Sized` bound makes it so that `Immutable` is still object
+ // safe.
+ #[doc(hidden)]
+ fn only_derive_is_allowed_to_implement_this_trait()
+ where
+ Self: Sized;
+}
+
+/// Implements [`TryFromBytes`].
+///
+/// This derive synthesizes the runtime checks required to check whether a
+/// sequence of initialized bytes corresponds to a valid instance of a type.
+/// This derive can be applied to structs, enums, and unions; e.g.:
+///
+/// ```
+/// # use zerocopy_derive::{TryFromBytes, Immutable};
+/// #[derive(TryFromBytes)]
+/// struct MyStruct {
+/// # /*
+/// ...
+/// # */
+/// }
+///
+/// #[derive(TryFromBytes)]
+/// #[repr(u8)]
+/// enum MyEnum {
+/// # V00,
+/// # /*
+/// ...
+/// # */
+/// }
+///
+/// #[derive(TryFromBytes, Immutable)]
+/// union MyUnion {
+/// # variant: u8,
+/// # /*
+/// ...
+/// # */
+/// }
+/// ```
+///
+/// # Portability
+///
+/// To ensure consistent endianness for enums with multi-byte representations,
+/// explicitly specify and convert each discriminant using `.to_le()` or
+/// `.to_be()`; e.g.:
+///
+/// ```
+/// # use zerocopy_derive::TryFromBytes;
+/// // `DataStoreVersion` is encoded in little-endian.
+/// #[derive(TryFromBytes)]
+/// #[repr(u32)]
+/// pub enum DataStoreVersion {
+/// /// Version 1 of the data store.
+/// V1 = 9u32.to_le(),
+///
+/// /// Version 2 of the data store.
+/// V2 = 10u32.to_le(),
+/// }
+/// ```
+///
+/// [safety conditions]: trait@TryFromBytes#safety
+#[cfg(any(feature = "derive", test))]
+#[cfg_attr(doc_cfg, doc(cfg(feature = "derive")))]
+pub use zerocopy_derive::TryFromBytes;
+
+/// Types for which some bit patterns are valid.
+///
+/// A memory region of the appropriate length which contains initialized bytes
+/// can be viewed as a `TryFromBytes` type so long as the runtime value of those
+/// bytes corresponds to a [*valid instance*] of that type. For example,
+/// [`bool`] is `TryFromBytes`, so zerocopy can transmute a [`u8`] into a
+/// [`bool`] so long as it first checks that the value of the [`u8`] is `0` or
+/// `1`.
+///
+/// # Implementation
+///
+/// **Do not implement this trait yourself!** Instead, use
+/// [`#[derive(TryFromBytes)]`][derive]; e.g.:
+///
+/// ```
+/// # use zerocopy_derive::{TryFromBytes, Immutable};
+/// #[derive(TryFromBytes)]
+/// struct MyStruct {
+/// # /*
+/// ...
+/// # */
+/// }
+///
+/// #[derive(TryFromBytes)]
+/// #[repr(u8)]
+/// enum MyEnum {
+/// # V00,
+/// # /*
+/// ...
+/// # */
+/// }
+///
+/// #[derive(TryFromBytes, Immutable)]
+/// union MyUnion {
+/// # variant: u8,
+/// # /*
+/// ...
+/// # */
+/// }
+/// ```
+///
+/// This derive ensures that the runtime check of whether bytes correspond to a
+/// valid instance is sound. You **must** implement this trait via the derive.
+///
+/// # What is a "valid instance"?
+///
+/// In Rust, each type has *bit validity*, which refers to the set of bit
+/// patterns which may appear in an instance of that type. It is impossible for
+/// safe Rust code to produce values which violate bit validity (ie, values
+/// outside of the "valid" set of bit patterns). If `unsafe` code produces an
+/// invalid value, this is considered [undefined behavior].
+///
+/// Rust's bit validity rules are currently being decided, which means that some
+/// types have three classes of bit patterns: those which are definitely valid,
+/// and whose validity is documented in the language; those which may or may not
+/// be considered valid at some point in the future; and those which are
+/// definitely invalid.
+///
+/// Zerocopy takes a conservative approach, and only considers a bit pattern to
+/// be valid if its validity is a documented guarantee provided by the
+/// language.
+///
+/// For most use cases, Rust's current guarantees align with programmers'
+/// intuitions about what ought to be valid. As a result, zerocopy's
+/// conservatism should not affect most users.
+///
+/// If you are negatively affected by lack of support for a particular type,
+/// we encourage you to let us know by [filing an issue][github-repo].
+///
+/// # `TryFromBytes` is not symmetrical with [`IntoBytes`]
+///
+/// There are some types which implement both `TryFromBytes` and [`IntoBytes`],
+/// but for which `TryFromBytes` is not guaranteed to accept all byte sequences
+/// produced by `IntoBytes`. In other words, for some `T: TryFromBytes +
+/// IntoBytes`, there exist values of `t: T` such that
+/// `TryFromBytes::try_ref_from_bytes(t.as_bytes()) == None`. Code should not
+/// generally assume that values produced by `IntoBytes` will necessarily be
+/// accepted as valid by `TryFromBytes`.
+///
+/// # Safety
+///
+/// On its own, `T: TryFromBytes` does not make any guarantees about the layout
+/// or representation of `T`. It merely provides the ability to perform a
+/// validity check at runtime via methods like [`try_ref_from_bytes`].
+///
+/// You must not rely on the `#[doc(hidden)]` internals of `TryFromBytes`.
+/// Future releases of zerocopy may make backwards-breaking changes to these
+/// items, including changes that only affect soundness, which may cause code
+/// which uses those items to silently become unsound.
+///
+/// [undefined behavior]: https://raphlinus.github.io/programming/rust/2018/08/17/undefined-behavior.html
+/// [github-repo]: https://github.com/google/zerocopy
+/// [`try_ref_from_bytes`]: TryFromBytes::try_ref_from_bytes
+/// [*valid instance*]: #what-is-a-valid-instance
+#[cfg_attr(feature = "derive", doc = "[derive]: zerocopy_derive::TryFromBytes")]
+#[cfg_attr(
+ not(feature = "derive"),
+ doc = concat!("[derive]: https://docs.rs/zerocopy/", env!("CARGO_PKG_VERSION"), "/zerocopy/derive.TryFromBytes.html"),
+)]
+#[cfg_attr(
+ not(no_zerocopy_diagnostic_on_unimplemented_1_78_0),
+ diagnostic::on_unimplemented(note = "Consider adding `#[derive(TryFromBytes)]` to `{Self}`")
+)]
+pub unsafe trait TryFromBytes {
+ // The `Self: Sized` bound makes it so that `TryFromBytes` is still object
+ // safe.
+ #[doc(hidden)]
+ fn only_derive_is_allowed_to_implement_this_trait()
+ where
+ Self: Sized;
+
+ /// Does a given memory range contain a valid instance of `Self`?
+ ///
+ /// # Safety
+ ///
+ /// Unsafe code may assume that, if `is_bit_valid(candidate)` returns true,
+ /// `*candidate` contains a valid `Self`.
+ ///
+ /// # Panics
+ ///
+ /// `is_bit_valid` may panic. Callers are responsible for ensuring that any
+ /// `unsafe` code remains sound even in the face of `is_bit_valid`
+ /// panicking. (We support user-defined validation routines; so long as
+ /// these routines are not required to be `unsafe`, there is no way to
+ /// ensure that these do not generate panics.)
+ ///
+ /// Besides user-defined validation routines panicking, `is_bit_valid` will
+ /// either panic or fail to compile if called on a pointer with [`Shared`]
+ /// aliasing when `Self: !Immutable`.
+ ///
+ /// [`UnsafeCell`]: core::cell::UnsafeCell
+ /// [`Shared`]: invariant::Shared
+ #[doc(hidden)]
+ fn is_bit_valid<A>(candidate: Maybe<'_, Self, A>) -> bool
+ where
+ A: invariant::Alignment;
+
+ /// Attempts to interpret the given `source` as a `&Self`.
+ ///
+ /// If the bytes of `source` are a valid instance of `Self`, this method
+ /// returns a reference to those bytes interpreted as a `Self`. If the
+ /// length of `source` is not a [valid size of `Self`][valid-size], or if
+ /// `source` is not appropriately aligned, or if `source` is not a valid
+ /// instance of `Self`, this returns `Err`. If [`Self:
+ /// Unaligned`][self-unaligned], you can [infallibly discard the alignment
+ /// error][ConvertError::from].
+ ///
+ /// `Self` may be a sized type, a slice, or a [slice DST][slice-dst].
+ ///
+ /// [valid-size]: crate::KnownLayout#what-is-a-valid-size
+ /// [self-unaligned]: Unaligned
+ /// [slice-dst]: KnownLayout#dynamically-sized-types
+ ///
+ /// # Compile-Time Assertions
+ ///
+ /// This method cannot yet be used on unsized types whose dynamically-sized
+ /// component is zero-sized. Attempting to use this method on such types
+ /// results in a compile-time assertion error; e.g.:
+ ///
+ /// ```compile_fail,E0080
+ /// use zerocopy::*;
+ /// # use zerocopy_derive::*;
+ ///
+ /// #[derive(TryFromBytes, Immutable, KnownLayout)]
+ /// #[repr(C)]
+ /// struct ZSTy {
+ /// leading_sized: u16,
+ /// trailing_dst: [()],
+ /// }
+ ///
+ /// let _ = ZSTy::try_ref_from_bytes(0u16.as_bytes()); // ⚠ Compile Error!
+ /// ```
+ ///
+ /// # Examples
+ ///
+ /// ```
+ /// use zerocopy::TryFromBytes;
+ /// # use zerocopy_derive::*;
+ ///
+ /// // The only valid value of this type is the byte `0xC0`
+ /// #[derive(TryFromBytes, KnownLayout, Immutable)]
+ /// #[repr(u8)]
+ /// enum C0 { xC0 = 0xC0 }
+ ///
+ /// // The only valid value of this type is the byte sequence `0xC0C0`.
+ /// #[derive(TryFromBytes, KnownLayout, Immutable)]
+ /// #[repr(C)]
+ /// struct C0C0(C0, C0);
+ ///
+ /// #[derive(TryFromBytes, KnownLayout, Immutable)]
+ /// #[repr(C)]
+ /// struct Packet {
+ /// magic_number: C0C0,
+ /// mug_size: u8,
+ /// temperature: u8,
+ /// marshmallows: [[u8; 2]],
+ /// }
+ ///
+ /// let bytes = &[0xC0, 0xC0, 240, 77, 0, 1, 2, 3, 4, 5][..];
+ ///
+ /// let packet = Packet::try_ref_from_bytes(bytes).unwrap();
+ ///
+ /// assert_eq!(packet.mug_size, 240);
+ /// assert_eq!(packet.temperature, 77);
+ /// assert_eq!(packet.marshmallows, [[0, 1], [2, 3], [4, 5]]);
+ ///
+ /// // These bytes are not valid instance of `Packet`.
+ /// let bytes = &[0x10, 0xC0, 240, 77, 0, 1, 2, 3, 4, 5][..];
+ /// assert!(Packet::try_ref_from_bytes(bytes).is_err());
+ /// ```
+ ///
+ #[doc = codegen_section!(
+ header = "h5",
+ bench = "try_ref_from_bytes",
+ format = "coco",
+ arity = 3,
+ [
+ open
+ @index 1
+ @title "Sized"
+ @variant "static_size"
+ ],
+ [
+ @index 2
+ @title "Unsized"
+ @variant "dynamic_size"
+ ],
+ [
+ @index 3
+ @title "Dynamically Padded"
+ @variant "dynamic_padding"
+ ]
+ )]
+ #[must_use = "has no side effects"]
+ #[cfg_attr(zerocopy_inline_always, inline(always))]
+ #[cfg_attr(not(zerocopy_inline_always), inline)]
+ fn try_ref_from_bytes(source: &[u8]) -> Result<&Self, TryCastError<&[u8], Self>>
+ where
+ Self: KnownLayout + Immutable,
+ {
+ static_assert_dst_is_not_zst!(Self);
+ match Ptr::from_ref(source).try_cast_into_no_leftover::<Self, BecauseImmutable>(None) {
+ Ok(source) => {
+ // This call may panic. If that happens, it doesn't cause any soundness
+ // issues, as we have not generated any invalid state which we need to
+ // fix before returning.
+ match source.try_into_valid() {
+ Ok(valid) => Ok(valid.as_ref()),
+ Err(e) => {
+ Err(e.map_src(|src| src.as_bytes::<BecauseImmutable>().as_ref()).into())
+ }
+ }
+ }
+ Err(e) => Err(e.map_src(Ptr::as_ref).into()),
+ }
+ }
+
+ /// Attempts to interpret the prefix of the given `source` as a `&Self`.
+ ///
+ /// This method computes the [largest possible size of `Self`][valid-size]
+ /// that can fit in the leading bytes of `source`. If that prefix is a valid
+ /// instance of `Self`, this method returns a reference to those bytes
+ /// interpreted as `Self`, and a reference to the remaining bytes. If there
+ /// are insufficient bytes, or if `source` is not appropriately aligned, or
+ /// if those bytes are not a valid instance of `Self`, this returns `Err`.
+ /// If [`Self: Unaligned`][self-unaligned], you can [infallibly discard the
+ /// alignment error][ConvertError::from].
+ ///
+ /// `Self` may be a sized type, a slice, or a [slice DST][slice-dst].
+ ///
+ /// [valid-size]: crate::KnownLayout#what-is-a-valid-size
+ /// [self-unaligned]: Unaligned
+ /// [slice-dst]: KnownLayout#dynamically-sized-types
+ ///
+ /// # Compile-Time Assertions
+ ///
+ /// This method cannot yet be used on unsized types whose dynamically-sized
+ /// component is zero-sized. Attempting to use this method on such types
+ /// results in a compile-time assertion error; e.g.:
+ ///
+ /// ```compile_fail,E0080
+ /// use zerocopy::*;
+ /// # use zerocopy_derive::*;
+ ///
+ /// #[derive(TryFromBytes, Immutable, KnownLayout)]
+ /// #[repr(C)]
+ /// struct ZSTy {
+ /// leading_sized: u16,
+ /// trailing_dst: [()],
+ /// }
+ ///
+ /// let _ = ZSTy::try_ref_from_prefix(0u16.as_bytes()); // ⚠ Compile Error!
+ /// ```
+ ///
+ /// # Examples
+ ///
+ /// ```
+ /// use zerocopy::TryFromBytes;
+ /// # use zerocopy_derive::*;
+ ///
+ /// // The only valid value of this type is the byte `0xC0`
+ /// #[derive(TryFromBytes, KnownLayout, Immutable)]
+ /// #[repr(u8)]
+ /// enum C0 { xC0 = 0xC0 }
+ ///
+ /// // The only valid value of this type is the bytes `0xC0C0`.
+ /// #[derive(TryFromBytes, KnownLayout, Immutable)]
+ /// #[repr(C)]
+ /// struct C0C0(C0, C0);
+ ///
+ /// #[derive(TryFromBytes, KnownLayout, Immutable)]
+ /// #[repr(C)]
+ /// struct Packet {
+ /// magic_number: C0C0,
+ /// mug_size: u8,
+ /// temperature: u8,
+ /// marshmallows: [[u8; 2]],
+ /// }
+ ///
+ /// // These are more bytes than are needed to encode a `Packet`.
+ /// let bytes = &[0xC0, 0xC0, 240, 77, 0, 1, 2, 3, 4, 5, 6][..];
+ ///
+ /// let (packet, suffix) = Packet::try_ref_from_prefix(bytes).unwrap();
+ ///
+ /// assert_eq!(packet.mug_size, 240);
+ /// assert_eq!(packet.temperature, 77);
+ /// assert_eq!(packet.marshmallows, [[0, 1], [2, 3], [4, 5]]);
+ /// assert_eq!(suffix, &[6u8][..]);
+ ///
+ /// // These bytes are not valid instance of `Packet`.
+ /// let bytes = &[0x10, 0xC0, 240, 77, 0, 1, 2, 3, 4, 5, 6][..];
+ /// assert!(Packet::try_ref_from_prefix(bytes).is_err());
+ /// ```
+ ///
+ #[doc = codegen_section!(
+ header = "h5",
+ bench = "try_ref_from_prefix",
+ format = "coco",
+ arity = 3,
+ [
+ open
+ @index 1
+ @title "Sized"
+ @variant "static_size"
+ ],
+ [
+ @index 2
+ @title "Unsized"
+ @variant "dynamic_size"
+ ],
+ [
+ @index 3
+ @title "Dynamically Padded"
+ @variant "dynamic_padding"
+ ]
+ )]
+ #[must_use = "has no side effects"]
+ #[cfg_attr(zerocopy_inline_always, inline(always))]
+ #[cfg_attr(not(zerocopy_inline_always), inline)]
+ fn try_ref_from_prefix(source: &[u8]) -> Result<(&Self, &[u8]), TryCastError<&[u8], Self>>
+ where
+ Self: KnownLayout + Immutable,
+ {
+ static_assert_dst_is_not_zst!(Self);
+ try_ref_from_prefix_suffix(source, CastType::Prefix, None)
+ }
+
+ /// Attempts to interpret the suffix of the given `source` as a `&Self`.
+ ///
+ /// This method computes the [largest possible size of `Self`][valid-size]
+ /// that can fit in the trailing bytes of `source`. If that suffix is a
+ /// valid instance of `Self`, this method returns a reference to those bytes
+ /// interpreted as `Self`, and a reference to the preceding bytes. If there
+ /// are insufficient bytes, or if the suffix of `source` would not be
+ /// appropriately aligned, or if the suffix is not a valid instance of
+ /// `Self`, this returns `Err`. If [`Self: Unaligned`][self-unaligned], you
+ /// can [infallibly discard the alignment error][ConvertError::from].
+ ///
+ /// `Self` may be a sized type, a slice, or a [slice DST][slice-dst].
+ ///
+ /// [valid-size]: crate::KnownLayout#what-is-a-valid-size
+ /// [self-unaligned]: Unaligned
+ /// [slice-dst]: KnownLayout#dynamically-sized-types
+ ///
+ /// # Compile-Time Assertions
+ ///
+ /// This method cannot yet be used on unsized types whose dynamically-sized
+ /// component is zero-sized. Attempting to use this method on such types
+ /// results in a compile-time assertion error; e.g.:
+ ///
+ /// ```compile_fail,E0080
+ /// use zerocopy::*;
+ /// # use zerocopy_derive::*;
+ ///
+ /// #[derive(TryFromBytes, Immutable, KnownLayout)]
+ /// #[repr(C)]
+ /// struct ZSTy {
+ /// leading_sized: u16,
+ /// trailing_dst: [()],
+ /// }
+ ///
+ /// let _ = ZSTy::try_ref_from_suffix(0u16.as_bytes()); // ⚠ Compile Error!
+ /// ```
+ ///
+ /// # Examples
+ ///
+ /// ```
+ /// use zerocopy::TryFromBytes;
+ /// # use zerocopy_derive::*;
+ ///
+ /// // The only valid value of this type is the byte `0xC0`
+ /// #[derive(TryFromBytes, KnownLayout, Immutable)]
+ /// #[repr(u8)]
+ /// enum C0 { xC0 = 0xC0 }
+ ///
+ /// // The only valid value of this type is the bytes `0xC0C0`.
+ /// #[derive(TryFromBytes, KnownLayout, Immutable)]
+ /// #[repr(C)]
+ /// struct C0C0(C0, C0);
+ ///
+ /// #[derive(TryFromBytes, KnownLayout, Immutable)]
+ /// #[repr(C)]
+ /// struct Packet {
+ /// magic_number: C0C0,
+ /// mug_size: u8,
+ /// temperature: u8,
+ /// marshmallows: [[u8; 2]],
+ /// }
+ ///
+ /// // These are more bytes than are needed to encode a `Packet`.
+ /// let bytes = &[0, 0xC0, 0xC0, 240, 77, 2, 3, 4, 5, 6, 7][..];
+ ///
+ /// let (prefix, packet) = Packet::try_ref_from_suffix(bytes).unwrap();
+ ///
+ /// assert_eq!(packet.mug_size, 240);
+ /// assert_eq!(packet.temperature, 77);
+ /// assert_eq!(packet.marshmallows, [[2, 3], [4, 5], [6, 7]]);
+ /// assert_eq!(prefix, &[0u8][..]);
+ ///
+ /// // These bytes are not valid instance of `Packet`.
+ /// let bytes = &[0, 1, 2, 3, 4, 5, 6, 77, 240, 0xC0, 0x10][..];
+ /// assert!(Packet::try_ref_from_suffix(bytes).is_err());
+ /// ```
+ ///
+ #[doc = codegen_section!(
+ header = "h5",
+ bench = "try_ref_from_suffix",
+ format = "coco",
+ arity = 3,
+ [
+ open
+ @index 1
+ @title "Sized"
+ @variant "static_size"
+ ],
+ [
+ @index 2
+ @title "Unsized"
+ @variant "dynamic_size"
+ ],
+ [
+ @index 3
+ @title "Dynamically Padded"
+ @variant "dynamic_padding"
+ ]
+ )]
+ #[must_use = "has no side effects"]
+ #[cfg_attr(zerocopy_inline_always, inline(always))]
+ #[cfg_attr(not(zerocopy_inline_always), inline)]
+ fn try_ref_from_suffix(source: &[u8]) -> Result<(&[u8], &Self), TryCastError<&[u8], Self>>
+ where
+ Self: KnownLayout + Immutable,
+ {
+ static_assert_dst_is_not_zst!(Self);
+ try_ref_from_prefix_suffix(source, CastType::Suffix, None).map(swap)
+ }
+
+ /// Attempts to interpret the given `source` as a `&mut Self` without
+ /// copying.
+ ///
+ /// If the bytes of `source` are a valid instance of `Self`, this method
+ /// returns a reference to those bytes interpreted as a `Self`. If the
+ /// length of `source` is not a [valid size of `Self`][valid-size], or if
+ /// `source` is not appropriately aligned, or if `source` is not a valid
+ /// instance of `Self`, this returns `Err`. If [`Self:
+ /// Unaligned`][self-unaligned], you can [infallibly discard the alignment
+ /// error][ConvertError::from].
+ ///
+ /// `Self` may be a sized type, a slice, or a [slice DST][slice-dst].
+ ///
+ /// [valid-size]: crate::KnownLayout#what-is-a-valid-size
+ /// [self-unaligned]: Unaligned
+ /// [slice-dst]: KnownLayout#dynamically-sized-types
+ ///
+ /// # Compile-Time Assertions
+ ///
+ /// This method cannot yet be used on unsized types whose dynamically-sized
+ /// component is zero-sized. Attempting to use this method on such types
+ /// results in a compile-time assertion error; e.g.:
+ ///
+ /// ```compile_fail,E0080
+ /// use zerocopy::*;
+ /// # use zerocopy_derive::*;
+ ///
+ /// #[derive(TryFromBytes, IntoBytes, KnownLayout)]
+ /// #[repr(C, packed)]
+ /// struct ZSTy {
+ /// leading_sized: [u8; 2],
+ /// trailing_dst: [()],
+ /// }
+ ///
+ /// let mut source = [85, 85];
+ /// let _ = ZSTy::try_mut_from_bytes(&mut source[..]); // ⚠ Compile Error!
+ /// ```
+ ///
+ /// # Examples
+ ///
+ /// ```
+ /// use zerocopy::TryFromBytes;
+ /// # use zerocopy_derive::*;
+ ///
+ /// // The only valid value of this type is the byte `0xC0`
+ /// #[derive(TryFromBytes, IntoBytes, KnownLayout)]
+ /// #[repr(u8)]
+ /// enum C0 { xC0 = 0xC0 }
+ ///
+ /// // The only valid value of this type is the bytes `0xC0C0`.
+ /// #[derive(TryFromBytes, IntoBytes, KnownLayout)]
+ /// #[repr(C)]
+ /// struct C0C0(C0, C0);
+ ///
+ /// #[derive(TryFromBytes, IntoBytes, KnownLayout)]
+ /// #[repr(C, packed)]
+ /// struct Packet {
+ /// magic_number: C0C0,
+ /// mug_size: u8,
+ /// temperature: u8,
+ /// marshmallows: [[u8; 2]],
+ /// }
+ ///
+ /// let bytes = &mut [0xC0, 0xC0, 240, 77, 0, 1, 2, 3, 4, 5][..];
+ ///
+ /// let packet = Packet::try_mut_from_bytes(bytes).unwrap();
+ ///
+ /// assert_eq!(packet.mug_size, 240);
+ /// assert_eq!(packet.temperature, 77);
+ /// assert_eq!(packet.marshmallows, [[0, 1], [2, 3], [4, 5]]);
+ ///
+ /// packet.temperature = 111;
+ ///
+ /// assert_eq!(bytes, [0xC0, 0xC0, 240, 111, 0, 1, 2, 3, 4, 5]);
+ ///
+ /// // These bytes are not valid instance of `Packet`.
+ /// let bytes = &mut [0x10, 0xC0, 240, 77, 0, 1, 2, 3, 4, 5, 6][..];
+ /// assert!(Packet::try_mut_from_bytes(bytes).is_err());
+ /// ```
+ ///
+ #[doc = codegen_header!("h5", "try_mut_from_bytes")]
+ ///
+ /// See [`TryFromBytes::try_ref_from_bytes`](#method.try_ref_from_bytes.codegen).
+ #[must_use = "has no side effects"]
+ #[cfg_attr(zerocopy_inline_always, inline(always))]
+ #[cfg_attr(not(zerocopy_inline_always), inline)]
+ fn try_mut_from_bytes(bytes: &mut [u8]) -> Result<&mut Self, TryCastError<&mut [u8], Self>>
+ where
+ Self: KnownLayout + IntoBytes,
+ {
+ static_assert_dst_is_not_zst!(Self);
+ match Ptr::from_mut(bytes).try_cast_into_no_leftover::<Self, BecauseExclusive>(None) {
+ Ok(source) => {
+ // This call may panic. If that happens, it doesn't cause any soundness
+ // issues, as we have not generated any invalid state which we need to
+ // fix before returning.
+ match source.try_into_valid() {
+ Ok(source) => Ok(source.as_mut()),
+ Err(e) => Err(e.map_src(|src| src.as_bytes().as_mut()).into()),
+ }
+ }
+ Err(e) => Err(e.map_src(Ptr::as_mut).into()),
+ }
+ }
+
+ /// Attempts to interpret the prefix of the given `source` as a `&mut
+ /// Self`.
+ ///
+ /// This method computes the [largest possible size of `Self`][valid-size]
+ /// that can fit in the leading bytes of `source`. If that prefix is a valid
+ /// instance of `Self`, this method returns a reference to those bytes
+ /// interpreted as `Self`, and a reference to the remaining bytes. If there
+ /// are insufficient bytes, or if `source` is not appropriately aligned, or
+ /// if the bytes are not a valid instance of `Self`, this returns `Err`. If
+ /// [`Self: Unaligned`][self-unaligned], you can [infallibly discard the
+ /// alignment error][ConvertError::from].
+ ///
+ /// `Self` may be a sized type, a slice, or a [slice DST][slice-dst].
+ ///
+ /// [valid-size]: crate::KnownLayout#what-is-a-valid-size
+ /// [self-unaligned]: Unaligned
+ /// [slice-dst]: KnownLayout#dynamically-sized-types
+ ///
+ /// # Compile-Time Assertions
+ ///
+ /// This method cannot yet be used on unsized types whose dynamically-sized
+ /// component is zero-sized. Attempting to use this method on such types
+ /// results in a compile-time assertion error; e.g.:
+ ///
+ /// ```compile_fail,E0080
+ /// use zerocopy::*;
+ /// # use zerocopy_derive::*;
+ ///
+ /// #[derive(TryFromBytes, IntoBytes, KnownLayout)]
+ /// #[repr(C, packed)]
+ /// struct ZSTy {
+ /// leading_sized: [u8; 2],
+ /// trailing_dst: [()],
+ /// }
+ ///
+ /// let mut source = [85, 85];
+ /// let _ = ZSTy::try_mut_from_prefix(&mut source[..]); // ⚠ Compile Error!
+ /// ```
+ ///
+ /// # Examples
+ ///
+ /// ```
+ /// use zerocopy::TryFromBytes;
+ /// # use zerocopy_derive::*;
+ ///
+ /// // The only valid value of this type is the byte `0xC0`
+ /// #[derive(TryFromBytes, IntoBytes, KnownLayout)]
+ /// #[repr(u8)]
+ /// enum C0 { xC0 = 0xC0 }
+ ///
+ /// // The only valid value of this type is the bytes `0xC0C0`.
+ /// #[derive(TryFromBytes, IntoBytes, KnownLayout)]
+ /// #[repr(C)]
+ /// struct C0C0(C0, C0);
+ ///
+ /// #[derive(TryFromBytes, IntoBytes, KnownLayout)]
+ /// #[repr(C, packed)]
+ /// struct Packet {
+ /// magic_number: C0C0,
+ /// mug_size: u8,
+ /// temperature: u8,
+ /// marshmallows: [[u8; 2]],
+ /// }
+ ///
+ /// // These are more bytes than are needed to encode a `Packet`.
+ /// let bytes = &mut [0xC0, 0xC0, 240, 77, 0, 1, 2, 3, 4, 5, 6][..];
+ ///
+ /// let (packet, suffix) = Packet::try_mut_from_prefix(bytes).unwrap();
+ ///
+ /// assert_eq!(packet.mug_size, 240);
+ /// assert_eq!(packet.temperature, 77);
+ /// assert_eq!(packet.marshmallows, [[0, 1], [2, 3], [4, 5]]);
+ /// assert_eq!(suffix, &[6u8][..]);
+ ///
+ /// packet.temperature = 111;
+ /// suffix[0] = 222;
+ ///
+ /// assert_eq!(bytes, [0xC0, 0xC0, 240, 111, 0, 1, 2, 3, 4, 5, 222]);
+ ///
+ /// // These bytes are not valid instance of `Packet`.
+ /// let bytes = &mut [0x10, 0xC0, 240, 77, 0, 1, 2, 3, 4, 5, 6][..];
+ /// assert!(Packet::try_mut_from_prefix(bytes).is_err());
+ /// ```
+ ///
+ #[doc = codegen_header!("h5", "try_mut_from_prefix")]
+ ///
+ /// See [`TryFromBytes::try_ref_from_prefix`](#method.try_ref_from_prefix.codegen).
+ #[must_use = "has no side effects"]
+ #[cfg_attr(zerocopy_inline_always, inline(always))]
+ #[cfg_attr(not(zerocopy_inline_always), inline)]
+ fn try_mut_from_prefix(
+ source: &mut [u8],
+ ) -> Result<(&mut Self, &mut [u8]), TryCastError<&mut [u8], Self>>
+ where
+ Self: KnownLayout + IntoBytes,
+ {
+ static_assert_dst_is_not_zst!(Self);
+ try_mut_from_prefix_suffix(source, CastType::Prefix, None)
+ }
+
+ /// Attempts to interpret the suffix of the given `source` as a `&mut
+ /// Self`.
+ ///
+ /// This method computes the [largest possible size of `Self`][valid-size]
+ /// that can fit in the trailing bytes of `source`. If that suffix is a
+ /// valid instance of `Self`, this method returns a reference to those bytes
+ /// interpreted as `Self`, and a reference to the preceding bytes. If there
+ /// are insufficient bytes, or if the suffix of `source` would not be
+ /// appropriately aligned, or if the suffix is not a valid instance of
+ /// `Self`, this returns `Err`. If [`Self: Unaligned`][self-unaligned], you
+ /// can [infallibly discard the alignment error][ConvertError::from].
+ ///
+ /// `Self` may be a sized type, a slice, or a [slice DST][slice-dst].
+ ///
+ /// [valid-size]: crate::KnownLayout#what-is-a-valid-size
+ /// [self-unaligned]: Unaligned
+ /// [slice-dst]: KnownLayout#dynamically-sized-types
+ ///
+ /// # Compile-Time Assertions
+ ///
+ /// This method cannot yet be used on unsized types whose dynamically-sized
+ /// component is zero-sized. Attempting to use this method on such types
+ /// results in a compile-time assertion error; e.g.:
+ ///
+ /// ```compile_fail,E0080
+ /// use zerocopy::*;
+ /// # use zerocopy_derive::*;
+ ///
+ /// #[derive(TryFromBytes, IntoBytes, KnownLayout)]
+ /// #[repr(C, packed)]
+ /// struct ZSTy {
+ /// leading_sized: u16,
+ /// trailing_dst: [()],
+ /// }
+ ///
+ /// let mut source = [85, 85];
+ /// let _ = ZSTy::try_mut_from_suffix(&mut source[..]); // ⚠ Compile Error!
+ /// ```
+ ///
+ /// # Examples
+ ///
+ /// ```
+ /// use zerocopy::TryFromBytes;
+ /// # use zerocopy_derive::*;
+ ///
+ /// // The only valid value of this type is the byte `0xC0`
+ /// #[derive(TryFromBytes, IntoBytes, KnownLayout)]
+ /// #[repr(u8)]
+ /// enum C0 { xC0 = 0xC0 }
+ ///
+ /// // The only valid value of this type is the bytes `0xC0C0`.
+ /// #[derive(TryFromBytes, IntoBytes, KnownLayout)]
+ /// #[repr(C)]
+ /// struct C0C0(C0, C0);
+ ///
+ /// #[derive(TryFromBytes, IntoBytes, KnownLayout)]
+ /// #[repr(C, packed)]
+ /// struct Packet {
+ /// magic_number: C0C0,
+ /// mug_size: u8,
+ /// temperature: u8,
+ /// marshmallows: [[u8; 2]],
+ /// }
+ ///
+ /// // These are more bytes than are needed to encode a `Packet`.
+ /// let bytes = &mut [0, 0xC0, 0xC0, 240, 77, 2, 3, 4, 5, 6, 7][..];
+ ///
+ /// let (prefix, packet) = Packet::try_mut_from_suffix(bytes).unwrap();
+ ///
+ /// assert_eq!(packet.mug_size, 240);
+ /// assert_eq!(packet.temperature, 77);
+ /// assert_eq!(packet.marshmallows, [[2, 3], [4, 5], [6, 7]]);
+ /// assert_eq!(prefix, &[0u8][..]);
+ ///
+ /// prefix[0] = 111;
+ /// packet.temperature = 222;
+ ///
+ /// assert_eq!(bytes, [111, 0xC0, 0xC0, 240, 222, 2, 3, 4, 5, 6, 7]);
+ ///
+ /// // These bytes are not valid instance of `Packet`.
+ /// let bytes = &mut [0, 1, 2, 3, 4, 5, 6, 77, 240, 0xC0, 0x10][..];
+ /// assert!(Packet::try_mut_from_suffix(bytes).is_err());
+ /// ```
+ ///
+ #[doc = codegen_header!("h5", "try_mut_from_suffix")]
+ ///
+ /// See [`TryFromBytes::try_ref_from_suffix`](#method.try_ref_from_suffix.codegen).
+ #[must_use = "has no side effects"]
+ #[cfg_attr(zerocopy_inline_always, inline(always))]
+ #[cfg_attr(not(zerocopy_inline_always), inline)]
+ fn try_mut_from_suffix(
+ source: &mut [u8],
+ ) -> Result<(&mut [u8], &mut Self), TryCastError<&mut [u8], Self>>
+ where
+ Self: KnownLayout + IntoBytes,
+ {
+ static_assert_dst_is_not_zst!(Self);
+ try_mut_from_prefix_suffix(source, CastType::Suffix, None).map(swap)
+ }
+
+ /// Attempts to interpret the given `source` as a `&Self` with a DST length
+ /// equal to `count`.
+ ///
+ /// This method attempts to return a reference to `source` interpreted as a
+ /// `Self` with `count` trailing elements. If the length of `source` is not
+ /// equal to the size of `Self` with `count` elements, if `source` is not
+ /// appropriately aligned, or if `source` does not contain a valid instance
+ /// of `Self`, this returns `Err`. If [`Self: Unaligned`][self-unaligned],
+ /// you can [infallibly discard the alignment error][ConvertError::from].
+ ///
+ /// [self-unaligned]: Unaligned
+ /// [slice-dst]: KnownLayout#dynamically-sized-types
+ ///
+ /// # Examples
+ ///
+ /// ```
+ /// # #![allow(non_camel_case_types)] // For C0::xC0
+ /// use zerocopy::TryFromBytes;
+ /// # use zerocopy_derive::*;
+ ///
+ /// // The only valid value of this type is the byte `0xC0`
+ /// #[derive(TryFromBytes, KnownLayout, Immutable)]
+ /// #[repr(u8)]
+ /// enum C0 { xC0 = 0xC0 }
+ ///
+ /// // The only valid value of this type is the bytes `0xC0C0`.
+ /// #[derive(TryFromBytes, KnownLayout, Immutable)]
+ /// #[repr(C)]
+ /// struct C0C0(C0, C0);
+ ///
+ /// #[derive(TryFromBytes, KnownLayout, Immutable)]
+ /// #[repr(C)]
+ /// struct Packet {
+ /// magic_number: C0C0,
+ /// mug_size: u8,
+ /// temperature: u8,
+ /// marshmallows: [[u8; 2]],
+ /// }
+ ///
+ /// let bytes = &[0xC0, 0xC0, 240, 77, 2, 3, 4, 5, 6, 7][..];
+ ///
+ /// let packet = Packet::try_ref_from_bytes_with_elems(bytes, 3).unwrap();
+ ///
+ /// assert_eq!(packet.mug_size, 240);
+ /// assert_eq!(packet.temperature, 77);
+ /// assert_eq!(packet.marshmallows, [[2, 3], [4, 5], [6, 7]]);
+ ///
+ /// // These bytes are not valid instance of `Packet`.
+ /// let bytes = &[0, 1, 2, 3, 4, 5, 6, 77, 240, 0xC0, 0xC0][..];
+ /// assert!(Packet::try_ref_from_bytes_with_elems(bytes, 3).is_err());
+ /// ```
+ ///
+ /// Since an explicit `count` is provided, this method supports types with
+ /// zero-sized trailing slice elements. Methods such as [`try_ref_from_bytes`]
+ /// which do not take an explicit count do not support such types.
+ ///
+ /// ```
+ /// use core::num::NonZeroU16;
+ /// use zerocopy::*;
+ /// # use zerocopy_derive::*;
+ ///
+ /// #[derive(TryFromBytes, Immutable, KnownLayout)]
+ /// #[repr(C)]
+ /// struct ZSTy {
+ /// leading_sized: NonZeroU16,
+ /// trailing_dst: [()],
+ /// }
+ ///
+ /// let src = 0xCAFEu16.as_bytes();
+ /// let zsty = ZSTy::try_ref_from_bytes_with_elems(src, 42).unwrap();
+ /// assert_eq!(zsty.trailing_dst.len(), 42);
+ /// ```
+ ///
+ /// [`try_ref_from_bytes`]: TryFromBytes::try_ref_from_bytes
+ ///
+ #[doc = codegen_section!(
+ header = "h5",
+ bench = "try_ref_from_bytes_with_elems",
+ format = "coco",
+ arity = 2,
+ [
+ open
+ @index 1
+ @title "Unsized"
+ @variant "dynamic_size"
+ ],
+ [
+ @index 2
+ @title "Dynamically Padded"
+ @variant "dynamic_padding"
+ ]
+ )]
+ #[must_use = "has no side effects"]
+ #[cfg_attr(zerocopy_inline_always, inline(always))]
+ #[cfg_attr(not(zerocopy_inline_always), inline)]
+ fn try_ref_from_bytes_with_elems(
+ source: &[u8],
+ count: usize,
+ ) -> Result<&Self, TryCastError<&[u8], Self>>
+ where
+ Self: KnownLayout<PointerMetadata = usize> + Immutable,
+ {
+ match Ptr::from_ref(source).try_cast_into_no_leftover::<Self, BecauseImmutable>(Some(count))
+ {
+ Ok(source) => {
+ // This call may panic. If that happens, it doesn't cause any soundness
+ // issues, as we have not generated any invalid state which we need to
+ // fix before returning.
+ match source.try_into_valid() {
+ Ok(source) => Ok(source.as_ref()),
+ Err(e) => {
+ Err(e.map_src(|src| src.as_bytes::<BecauseImmutable>().as_ref()).into())
+ }
+ }
+ }
+ Err(e) => Err(e.map_src(Ptr::as_ref).into()),
+ }
+ }
+
+ /// Attempts to interpret the prefix of the given `source` as a `&Self` with
+ /// a DST length equal to `count`.
+ ///
+ /// This method attempts to return a reference to the prefix of `source`
+ /// interpreted as a `Self` with `count` trailing elements, and a reference
+ /// to the remaining bytes. If the length of `source` is less than the size
+ /// of `Self` with `count` elements, if `source` is not appropriately
+ /// aligned, or if the prefix of `source` does not contain a valid instance
+ /// of `Self`, this returns `Err`. If [`Self: Unaligned`][self-unaligned],
+ /// you can [infallibly discard the alignment error][ConvertError::from].
+ ///
+ /// [self-unaligned]: Unaligned
+ /// [slice-dst]: KnownLayout#dynamically-sized-types
+ ///
+ /// # Examples
+ ///
+ /// ```
+ /// # #![allow(non_camel_case_types)] // For C0::xC0
+ /// use zerocopy::TryFromBytes;
+ /// # use zerocopy_derive::*;
+ ///
+ /// // The only valid value of this type is the byte `0xC0`
+ /// #[derive(TryFromBytes, KnownLayout, Immutable)]
+ /// #[repr(u8)]
+ /// enum C0 { xC0 = 0xC0 }
+ ///
+ /// // The only valid value of this type is the bytes `0xC0C0`.
+ /// #[derive(TryFromBytes, KnownLayout, Immutable)]
+ /// #[repr(C)]
+ /// struct C0C0(C0, C0);
+ ///
+ /// #[derive(TryFromBytes, KnownLayout, Immutable)]
+ /// #[repr(C)]
+ /// struct Packet {
+ /// magic_number: C0C0,
+ /// mug_size: u8,
+ /// temperature: u8,
+ /// marshmallows: [[u8; 2]],
+ /// }
+ ///
+ /// let bytes = &[0xC0, 0xC0, 240, 77, 2, 3, 4, 5, 6, 7, 8][..];
+ ///
+ /// let (packet, suffix) = Packet::try_ref_from_prefix_with_elems(bytes, 3).unwrap();
+ ///
+ /// assert_eq!(packet.mug_size, 240);
+ /// assert_eq!(packet.temperature, 77);
+ /// assert_eq!(packet.marshmallows, [[2, 3], [4, 5], [6, 7]]);
+ /// assert_eq!(suffix, &[8u8][..]);
+ ///
+ /// // These bytes are not valid instance of `Packet`.
+ /// let bytes = &mut [0, 1, 2, 3, 4, 5, 6, 7, 8, 77, 240, 0xC0, 0xC0][..];
+ /// assert!(Packet::try_ref_from_prefix_with_elems(bytes, 3).is_err());
+ /// ```
+ ///
+ /// Since an explicit `count` is provided, this method supports types with
+ /// zero-sized trailing slice elements. Methods such as [`try_ref_from_prefix`]
+ /// which do not take an explicit count do not support such types.
+ ///
+ /// ```
+ /// use core::num::NonZeroU16;
+ /// use zerocopy::*;
+ /// # use zerocopy_derive::*;
+ ///
+ /// #[derive(TryFromBytes, Immutable, KnownLayout)]
+ /// #[repr(C)]
+ /// struct ZSTy {
+ /// leading_sized: NonZeroU16,
+ /// trailing_dst: [()],
+ /// }
+ ///
+ /// let src = 0xCAFEu16.as_bytes();
+ /// let (zsty, _) = ZSTy::try_ref_from_prefix_with_elems(src, 42).unwrap();
+ /// assert_eq!(zsty.trailing_dst.len(), 42);
+ /// ```
+ ///
+ /// [`try_ref_from_prefix`]: TryFromBytes::try_ref_from_prefix
+ ///
+ #[doc = codegen_section!(
+ header = "h5",
+ bench = "try_ref_from_prefix_with_elems",
+ format = "coco",
+ arity = 2,
+ [
+ open
+ @index 1
+ @title "Unsized"
+ @variant "dynamic_size"
+ ],
+ [
+ @index 2
+ @title "Dynamically Padded"
+ @variant "dynamic_padding"
+ ]
+ )]
+ #[must_use = "has no side effects"]
+ #[cfg_attr(zerocopy_inline_always, inline(always))]
+ #[cfg_attr(not(zerocopy_inline_always), inline)]
+ fn try_ref_from_prefix_with_elems(
+ source: &[u8],
+ count: usize,
+ ) -> Result<(&Self, &[u8]), TryCastError<&[u8], Self>>
+ where
+ Self: KnownLayout<PointerMetadata = usize> + Immutable,
+ {
+ try_ref_from_prefix_suffix(source, CastType::Prefix, Some(count))
+ }
+
+ /// Attempts to interpret the suffix of the given `source` as a `&Self` with
+ /// a DST length equal to `count`.
+ ///
+ /// This method attempts to return a reference to the suffix of `source`
+ /// interpreted as a `Self` with `count` trailing elements, and a reference
+ /// to the preceding bytes. If the length of `source` is less than the size
+ /// of `Self` with `count` elements, if the suffix of `source` is not
+ /// appropriately aligned, or if the suffix of `source` does not contain a
+ /// valid instance of `Self`, this returns `Err`. If [`Self:
+ /// Unaligned`][self-unaligned], you can [infallibly discard the alignment
+ /// error][ConvertError::from].
+ ///
+ /// [self-unaligned]: Unaligned
+ /// [slice-dst]: KnownLayout#dynamically-sized-types
+ ///
+ /// # Examples
+ ///
+ /// ```
+ /// # #![allow(non_camel_case_types)] // For C0::xC0
+ /// use zerocopy::TryFromBytes;
+ /// # use zerocopy_derive::*;
+ ///
+ /// // The only valid value of this type is the byte `0xC0`
+ /// #[derive(TryFromBytes, KnownLayout, Immutable)]
+ /// #[repr(u8)]
+ /// enum C0 { xC0 = 0xC0 }
+ ///
+ /// // The only valid value of this type is the bytes `0xC0C0`.
+ /// #[derive(TryFromBytes, KnownLayout, Immutable)]
+ /// #[repr(C)]
+ /// struct C0C0(C0, C0);
+ ///
+ /// #[derive(TryFromBytes, KnownLayout, Immutable)]
+ /// #[repr(C)]
+ /// struct Packet {
+ /// magic_number: C0C0,
+ /// mug_size: u8,
+ /// temperature: u8,
+ /// marshmallows: [[u8; 2]],
+ /// }
+ ///
+ /// let bytes = &[123, 0xC0, 0xC0, 240, 77, 2, 3, 4, 5, 6, 7][..];
+ ///
+ /// let (prefix, packet) = Packet::try_ref_from_suffix_with_elems(bytes, 3).unwrap();
+ ///
+ /// assert_eq!(packet.mug_size, 240);
+ /// assert_eq!(packet.temperature, 77);
+ /// assert_eq!(packet.marshmallows, [[2, 3], [4, 5], [6, 7]]);
+ /// assert_eq!(prefix, &[123u8][..]);
+ ///
+ /// // These bytes are not valid instance of `Packet`.
+ /// let bytes = &[0, 1, 2, 3, 4, 5, 6, 7, 8, 77, 240, 0xC0, 0xC0][..];
+ /// assert!(Packet::try_ref_from_suffix_with_elems(bytes, 3).is_err());
+ /// ```
+ ///
+ /// Since an explicit `count` is provided, this method supports types with
+ /// zero-sized trailing slice elements. Methods such as [`try_ref_from_prefix`]
+ /// which do not take an explicit count do not support such types.
+ ///
+ /// ```
+ /// use core::num::NonZeroU16;
+ /// use zerocopy::*;
+ /// # use zerocopy_derive::*;
+ ///
+ /// #[derive(TryFromBytes, Immutable, KnownLayout)]
+ /// #[repr(C)]
+ /// struct ZSTy {
+ /// leading_sized: NonZeroU16,
+ /// trailing_dst: [()],
+ /// }
+ ///
+ /// let src = 0xCAFEu16.as_bytes();
+ /// let (_, zsty) = ZSTy::try_ref_from_suffix_with_elems(src, 42).unwrap();
+ /// assert_eq!(zsty.trailing_dst.len(), 42);
+ /// ```
+ ///
+ /// [`try_ref_from_prefix`]: TryFromBytes::try_ref_from_prefix
+ ///
+ #[doc = codegen_section!(
+ header = "h5",
+ bench = "try_ref_from_suffix_with_elems",
+ format = "coco",
+ arity = 2,
+ [
+ open
+ @index 1
+ @title "Unsized"
+ @variant "dynamic_size"
+ ],
+ [
+ @index 2
+ @title "Dynamically Padded"
+ @variant "dynamic_padding"
+ ]
+ )]
+ #[must_use = "has no side effects"]
+ #[cfg_attr(zerocopy_inline_always, inline(always))]
+ #[cfg_attr(not(zerocopy_inline_always), inline)]
+ fn try_ref_from_suffix_with_elems(
+ source: &[u8],
+ count: usize,
+ ) -> Result<(&[u8], &Self), TryCastError<&[u8], Self>>
+ where
+ Self: KnownLayout<PointerMetadata = usize> + Immutable,
+ {
+ try_ref_from_prefix_suffix(source, CastType::Suffix, Some(count)).map(swap)
+ }
+
+ /// Attempts to interpret the given `source` as a `&mut Self` with a DST
+ /// length equal to `count`.
+ ///
+ /// This method attempts to return a reference to `source` interpreted as a
+ /// `Self` with `count` trailing elements. If the length of `source` is not
+ /// equal to the size of `Self` with `count` elements, if `source` is not
+ /// appropriately aligned, or if `source` does not contain a valid instance
+ /// of `Self`, this returns `Err`. If [`Self: Unaligned`][self-unaligned],
+ /// you can [infallibly discard the alignment error][ConvertError::from].
+ ///
+ /// [self-unaligned]: Unaligned
+ /// [slice-dst]: KnownLayout#dynamically-sized-types
+ ///
+ /// # Examples
+ ///
+ /// ```
+ /// # #![allow(non_camel_case_types)] // For C0::xC0
+ /// use zerocopy::TryFromBytes;
+ /// # use zerocopy_derive::*;
+ ///
+ /// // The only valid value of this type is the byte `0xC0`
+ /// #[derive(TryFromBytes, IntoBytes, KnownLayout)]
+ /// #[repr(u8)]
+ /// enum C0 { xC0 = 0xC0 }
+ ///
+ /// // The only valid value of this type is the bytes `0xC0C0`.
+ /// #[derive(TryFromBytes, IntoBytes, KnownLayout)]
+ /// #[repr(C)]
+ /// struct C0C0(C0, C0);
+ ///
+ /// #[derive(TryFromBytes, IntoBytes, KnownLayout)]
+ /// #[repr(C, packed)]
+ /// struct Packet {
+ /// magic_number: C0C0,
+ /// mug_size: u8,
+ /// temperature: u8,
+ /// marshmallows: [[u8; 2]],
+ /// }
+ ///
+ /// let bytes = &mut [0xC0, 0xC0, 240, 77, 2, 3, 4, 5, 6, 7][..];
+ ///
+ /// let packet = Packet::try_mut_from_bytes_with_elems(bytes, 3).unwrap();
+ ///
+ /// assert_eq!(packet.mug_size, 240);
+ /// assert_eq!(packet.temperature, 77);
+ /// assert_eq!(packet.marshmallows, [[2, 3], [4, 5], [6, 7]]);
+ ///
+ /// packet.temperature = 111;
+ ///
+ /// assert_eq!(bytes, [0xC0, 0xC0, 240, 111, 2, 3, 4, 5, 6, 7]);
+ ///
+ /// // These bytes are not valid instance of `Packet`.
+ /// let bytes = &mut [0, 1, 2, 3, 4, 5, 6, 77, 240, 0xC0, 0xC0][..];
+ /// assert!(Packet::try_mut_from_bytes_with_elems(bytes, 3).is_err());
+ /// ```
+ ///
+ /// Since an explicit `count` is provided, this method supports types with
+ /// zero-sized trailing slice elements. Methods such as [`try_mut_from_bytes`]
+ /// which do not take an explicit count do not support such types.
+ ///
+ /// ```
+ /// use core::num::NonZeroU16;
+ /// use zerocopy::*;
+ /// # use zerocopy_derive::*;
+ ///
+ /// #[derive(TryFromBytes, IntoBytes, KnownLayout)]
+ /// #[repr(C, packed)]
+ /// struct ZSTy {
+ /// leading_sized: NonZeroU16,
+ /// trailing_dst: [()],
+ /// }
+ ///
+ /// let mut src = 0xCAFEu16;
+ /// let src = src.as_mut_bytes();
+ /// let zsty = ZSTy::try_mut_from_bytes_with_elems(src, 42).unwrap();
+ /// assert_eq!(zsty.trailing_dst.len(), 42);
+ /// ```
+ ///
+ /// [`try_mut_from_bytes`]: TryFromBytes::try_mut_from_bytes
+ ///
+ #[doc = codegen_header!("h5", "try_mut_from_bytes_with_elems")]
+ ///
+ /// See [`TryFromBytes::try_ref_from_bytes_with_elems`](#method.try_ref_from_bytes_with_elems.codegen).
+ #[must_use = "has no side effects"]
+ #[cfg_attr(zerocopy_inline_always, inline(always))]
+ #[cfg_attr(not(zerocopy_inline_always), inline)]
+ fn try_mut_from_bytes_with_elems(
+ source: &mut [u8],
+ count: usize,
+ ) -> Result<&mut Self, TryCastError<&mut [u8], Self>>
+ where
+ Self: KnownLayout<PointerMetadata = usize> + IntoBytes,
+ {
+ match Ptr::from_mut(source).try_cast_into_no_leftover::<Self, BecauseExclusive>(Some(count))
+ {
+ Ok(source) => {
+ // This call may panic. If that happens, it doesn't cause any soundness
+ // issues, as we have not generated any invalid state which we need to
+ // fix before returning.
+ match source.try_into_valid() {
+ Ok(source) => Ok(source.as_mut()),
+ Err(e) => Err(e.map_src(|src| src.as_bytes().as_mut()).into()),
+ }
+ }
+ Err(e) => Err(e.map_src(Ptr::as_mut).into()),
+ }
+ }
+
+ /// Attempts to interpret the prefix of the given `source` as a `&mut Self`
+ /// with a DST length equal to `count`.
+ ///
+ /// This method attempts to return a reference to the prefix of `source`
+ /// interpreted as a `Self` with `count` trailing elements, and a reference
+ /// to the remaining bytes. If the length of `source` is less than the size
+ /// of `Self` with `count` elements, if `source` is not appropriately
+ /// aligned, or if the prefix of `source` does not contain a valid instance
+ /// of `Self`, this returns `Err`. If [`Self: Unaligned`][self-unaligned],
+ /// you can [infallibly discard the alignment error][ConvertError::from].
+ ///
+ /// [self-unaligned]: Unaligned
+ /// [slice-dst]: KnownLayout#dynamically-sized-types
+ ///
+ /// # Examples
+ ///
+ /// ```
+ /// # #![allow(non_camel_case_types)] // For C0::xC0
+ /// use zerocopy::TryFromBytes;
+ /// # use zerocopy_derive::*;
+ ///
+ /// // The only valid value of this type is the byte `0xC0`
+ /// #[derive(TryFromBytes, IntoBytes, KnownLayout)]
+ /// #[repr(u8)]
+ /// enum C0 { xC0 = 0xC0 }
+ ///
+ /// // The only valid value of this type is the bytes `0xC0C0`.
+ /// #[derive(TryFromBytes, IntoBytes, KnownLayout)]
+ /// #[repr(C)]
+ /// struct C0C0(C0, C0);
+ ///
+ /// #[derive(TryFromBytes, IntoBytes, KnownLayout)]
+ /// #[repr(C, packed)]
+ /// struct Packet {
+ /// magic_number: C0C0,
+ /// mug_size: u8,
+ /// temperature: u8,
+ /// marshmallows: [[u8; 2]],
+ /// }
+ ///
+ /// let bytes = &mut [0xC0, 0xC0, 240, 77, 2, 3, 4, 5, 6, 7, 8][..];
+ ///
+ /// let (packet, suffix) = Packet::try_mut_from_prefix_with_elems(bytes, 3).unwrap();
+ ///
+ /// assert_eq!(packet.mug_size, 240);
+ /// assert_eq!(packet.temperature, 77);
+ /// assert_eq!(packet.marshmallows, [[2, 3], [4, 5], [6, 7]]);
+ /// assert_eq!(suffix, &[8u8][..]);
+ ///
+ /// packet.temperature = 111;
+ /// suffix[0] = 222;
+ ///
+ /// assert_eq!(bytes, [0xC0, 0xC0, 240, 111, 2, 3, 4, 5, 6, 7, 222]);
+ ///
+ /// // These bytes are not valid instance of `Packet`.
+ /// let bytes = &mut [0, 1, 2, 3, 4, 5, 6, 7, 8, 77, 240, 0xC0, 0xC0][..];
+ /// assert!(Packet::try_mut_from_prefix_with_elems(bytes, 3).is_err());
+ /// ```
+ ///
+ /// Since an explicit `count` is provided, this method supports types with
+ /// zero-sized trailing slice elements. Methods such as [`try_mut_from_prefix`]
+ /// which do not take an explicit count do not support such types.
+ ///
+ /// ```
+ /// use core::num::NonZeroU16;
+ /// use zerocopy::*;
+ /// # use zerocopy_derive::*;
+ ///
+ /// #[derive(TryFromBytes, IntoBytes, KnownLayout)]
+ /// #[repr(C, packed)]
+ /// struct ZSTy {
+ /// leading_sized: NonZeroU16,
+ /// trailing_dst: [()],
+ /// }
+ ///
+ /// let mut src = 0xCAFEu16;
+ /// let src = src.as_mut_bytes();
+ /// let (zsty, _) = ZSTy::try_mut_from_prefix_with_elems(src, 42).unwrap();
+ /// assert_eq!(zsty.trailing_dst.len(), 42);
+ /// ```
+ ///
+ /// [`try_mut_from_prefix`]: TryFromBytes::try_mut_from_prefix
+ ///
+ #[doc = codegen_header!("h5", "try_mut_from_prefix_with_elems")]
+ ///
+ /// See [`TryFromBytes::try_ref_from_prefix_with_elems`](#method.try_ref_from_prefix_with_elems.codegen).
+ #[must_use = "has no side effects"]
+ #[cfg_attr(zerocopy_inline_always, inline(always))]
+ #[cfg_attr(not(zerocopy_inline_always), inline)]
+ fn try_mut_from_prefix_with_elems(
+ source: &mut [u8],
+ count: usize,
+ ) -> Result<(&mut Self, &mut [u8]), TryCastError<&mut [u8], Self>>
+ where
+ Self: KnownLayout<PointerMetadata = usize> + IntoBytes,
+ {
+ try_mut_from_prefix_suffix(source, CastType::Prefix, Some(count))
+ }
+
+ /// Attempts to interpret the suffix of the given `source` as a `&mut Self`
+ /// with a DST length equal to `count`.
+ ///
+ /// This method attempts to return a reference to the suffix of `source`
+ /// interpreted as a `Self` with `count` trailing elements, and a reference
+ /// to the preceding bytes. If the length of `source` is less than the size
+ /// of `Self` with `count` elements, if the suffix of `source` is not
+ /// appropriately aligned, or if the suffix of `source` does not contain a
+ /// valid instance of `Self`, this returns `Err`. If [`Self:
+ /// Unaligned`][self-unaligned], you can [infallibly discard the alignment
+ /// error][ConvertError::from].
+ ///
+ /// [self-unaligned]: Unaligned
+ /// [slice-dst]: KnownLayout#dynamically-sized-types
+ ///
+ /// # Examples
+ ///
+ /// ```
+ /// # #![allow(non_camel_case_types)] // For C0::xC0
+ /// use zerocopy::TryFromBytes;
+ /// # use zerocopy_derive::*;
+ ///
+ /// // The only valid value of this type is the byte `0xC0`
+ /// #[derive(TryFromBytes, IntoBytes, KnownLayout)]
+ /// #[repr(u8)]
+ /// enum C0 { xC0 = 0xC0 }
+ ///
+ /// // The only valid value of this type is the bytes `0xC0C0`.
+ /// #[derive(TryFromBytes, IntoBytes, KnownLayout)]
+ /// #[repr(C)]
+ /// struct C0C0(C0, C0);
+ ///
+ /// #[derive(TryFromBytes, IntoBytes, KnownLayout)]
+ /// #[repr(C, packed)]
+ /// struct Packet {
+ /// magic_number: C0C0,
+ /// mug_size: u8,
+ /// temperature: u8,
+ /// marshmallows: [[u8; 2]],
+ /// }
+ ///
+ /// let bytes = &mut [123, 0xC0, 0xC0, 240, 77, 2, 3, 4, 5, 6, 7][..];
+ ///
+ /// let (prefix, packet) = Packet::try_mut_from_suffix_with_elems(bytes, 3).unwrap();
+ ///
+ /// assert_eq!(packet.mug_size, 240);
+ /// assert_eq!(packet.temperature, 77);
+ /// assert_eq!(packet.marshmallows, [[2, 3], [4, 5], [6, 7]]);
+ /// assert_eq!(prefix, &[123u8][..]);
+ ///
+ /// prefix[0] = 111;
+ /// packet.temperature = 222;
+ ///
+ /// assert_eq!(bytes, [111, 0xC0, 0xC0, 240, 222, 2, 3, 4, 5, 6, 7]);
+ ///
+ /// // These bytes are not valid instance of `Packet`.
+ /// let bytes = &mut [0, 1, 2, 3, 4, 5, 6, 7, 8, 77, 240, 0xC0, 0xC0][..];
+ /// assert!(Packet::try_mut_from_suffix_with_elems(bytes, 3).is_err());
+ /// ```
+ ///
+ /// Since an explicit `count` is provided, this method supports types with
+ /// zero-sized trailing slice elements. Methods such as [`try_mut_from_prefix`]
+ /// which do not take an explicit count do not support such types.
+ ///
+ /// ```
+ /// use core::num::NonZeroU16;
+ /// use zerocopy::*;
+ /// # use zerocopy_derive::*;
+ ///
+ /// #[derive(TryFromBytes, IntoBytes, KnownLayout)]
+ /// #[repr(C, packed)]
+ /// struct ZSTy {
+ /// leading_sized: NonZeroU16,
+ /// trailing_dst: [()],
+ /// }
+ ///
+ /// let mut src = 0xCAFEu16;
+ /// let src = src.as_mut_bytes();
+ /// let (_, zsty) = ZSTy::try_mut_from_suffix_with_elems(src, 42).unwrap();
+ /// assert_eq!(zsty.trailing_dst.len(), 42);
+ /// ```
+ ///
+ /// [`try_mut_from_prefix`]: TryFromBytes::try_mut_from_prefix
+ ///
+ #[doc = codegen_header!("h5", "try_mut_from_suffix_with_elems")]
+ ///
+ /// See [`TryFromBytes::try_ref_from_suffix_with_elems`](#method.try_ref_from_suffix_with_elems.codegen).
+ #[must_use = "has no side effects"]
+ #[cfg_attr(zerocopy_inline_always, inline(always))]
+ #[cfg_attr(not(zerocopy_inline_always), inline)]
+ fn try_mut_from_suffix_with_elems(
+ source: &mut [u8],
+ count: usize,
+ ) -> Result<(&mut [u8], &mut Self), TryCastError<&mut [u8], Self>>
+ where
+ Self: KnownLayout<PointerMetadata = usize> + IntoBytes,
+ {
+ try_mut_from_prefix_suffix(source, CastType::Suffix, Some(count)).map(swap)
+ }
+
+ /// Attempts to read the given `source` as a `Self`.
+ ///
+ /// If `source.len() != size_of::<Self>()` or the bytes are not a valid
+ /// instance of `Self`, this returns `Err`.
+ ///
+ /// # Examples
+ ///
+ /// ```
+ /// use zerocopy::TryFromBytes;
+ /// # use zerocopy_derive::*;
+ ///
+ /// // The only valid value of this type is the byte `0xC0`
+ /// #[derive(TryFromBytes)]
+ /// #[repr(u8)]
+ /// enum C0 { xC0 = 0xC0 }
+ ///
+ /// // The only valid value of this type is the bytes `0xC0C0`.
+ /// #[derive(TryFromBytes)]
+ /// #[repr(C)]
+ /// struct C0C0(C0, C0);
+ ///
+ /// #[derive(TryFromBytes)]
+ /// #[repr(C)]
+ /// struct Packet {
+ /// magic_number: C0C0,
+ /// mug_size: u8,
+ /// temperature: u8,
+ /// }
+ ///
+ /// let bytes = &[0xC0, 0xC0, 240, 77][..];
+ ///
+ /// let packet = Packet::try_read_from_bytes(bytes).unwrap();
+ ///
+ /// assert_eq!(packet.mug_size, 240);
+ /// assert_eq!(packet.temperature, 77);
+ ///
+ /// // These bytes are not valid instance of `Packet`.
+ /// let bytes = &mut [0x10, 0xC0, 240, 77][..];
+ /// assert!(Packet::try_read_from_bytes(bytes).is_err());
+ /// ```
+ ///
+ /// # Performance Considerations
+ ///
+ /// In this version of zerocopy, this method reads the `source` into a
+ /// well-aligned stack allocation and *then* validates that the allocation
+ /// is a valid `Self`. This ensures that validation can be performed using
+ /// aligned reads (which carry a performance advantage over unaligned reads
+ /// on many platforms) at the cost of an unconditional copy.
+ ///
+ #[doc = codegen_section!(
+ header = "h5",
+ bench = "try_read_from_bytes",
+ format = "coco_static_size",
+ )]
+ #[must_use = "has no side effects"]
+ #[cfg_attr(zerocopy_inline_always, inline(always))]
+ #[cfg_attr(not(zerocopy_inline_always), inline)]
+ fn try_read_from_bytes(source: &[u8]) -> Result<Self, TryReadError<&[u8], Self>>
+ where
+ Self: Sized,
+ {
+ // FIXME(#2981): If `align_of::<Self>() == 1`, validate `source` in-place.
+
+ let candidate = match CoreMaybeUninit::<Self>::read_from_bytes(source) {
+ Ok(candidate) => candidate,
+ Err(e) => {
+ return Err(TryReadError::Size(e.with_dst()));
+ }
+ };
+ // SAFETY: `candidate` was copied from from `source: &[u8]`, so all of
+ // its bytes are initialized.
+ unsafe { try_read_from(source, candidate) }
+ }
+
+ /// Attempts to read a `Self` from the prefix of the given `source`.
+ ///
+ /// This attempts to read a `Self` from the first `size_of::<Self>()` bytes
+ /// of `source`, returning that `Self` and any remaining bytes. If
+ /// `source.len() < size_of::<Self>()` or the bytes are not a valid instance
+ /// of `Self`, it returns `Err`.
+ ///
+ /// # Examples
+ ///
+ /// ```
+ /// use zerocopy::TryFromBytes;
+ /// # use zerocopy_derive::*;
+ ///
+ /// // The only valid value of this type is the byte `0xC0`
+ /// #[derive(TryFromBytes)]
+ /// #[repr(u8)]
+ /// enum C0 { xC0 = 0xC0 }
+ ///
+ /// // The only valid value of this type is the bytes `0xC0C0`.
+ /// #[derive(TryFromBytes)]
+ /// #[repr(C)]
+ /// struct C0C0(C0, C0);
+ ///
+ /// #[derive(TryFromBytes)]
+ /// #[repr(C)]
+ /// struct Packet {
+ /// magic_number: C0C0,
+ /// mug_size: u8,
+ /// temperature: u8,
+ /// }
+ ///
+ /// // These are more bytes than are needed to encode a `Packet`.
+ /// let bytes = &[0xC0, 0xC0, 240, 77, 0, 1, 2, 3, 4, 5, 6][..];
+ ///
+ /// let (packet, suffix) = Packet::try_read_from_prefix(bytes).unwrap();
+ ///
+ /// assert_eq!(packet.mug_size, 240);
+ /// assert_eq!(packet.temperature, 77);
+ /// assert_eq!(suffix, &[0u8, 1, 2, 3, 4, 5, 6][..]);
+ ///
+ /// // These bytes are not valid instance of `Packet`.
+ /// let bytes = &[0x10, 0xC0, 240, 77, 0, 1, 2, 3, 4, 5, 6][..];
+ /// assert!(Packet::try_read_from_prefix(bytes).is_err());
+ /// ```
+ ///
+ /// # Performance Considerations
+ ///
+ /// In this version of zerocopy, this method reads the `source` into a
+ /// well-aligned stack allocation and *then* validates that the allocation
+ /// is a valid `Self`. This ensures that validation can be performed using
+ /// aligned reads (which carry a performance advantage over unaligned reads
+ /// on many platforms) at the cost of an unconditional copy.
+ ///
+ #[doc = codegen_section!(
+ header = "h5",
+ bench = "try_read_from_prefix",
+ format = "coco_static_size",
+ )]
+ #[must_use = "has no side effects"]
+ #[cfg_attr(zerocopy_inline_always, inline(always))]
+ #[cfg_attr(not(zerocopy_inline_always), inline)]
+ fn try_read_from_prefix(source: &[u8]) -> Result<(Self, &[u8]), TryReadError<&[u8], Self>>
+ where
+ Self: Sized,
+ {
+ // FIXME(#2981): If `align_of::<Self>() == 1`, validate `source` in-place.
+
+ let (candidate, suffix) = match CoreMaybeUninit::<Self>::read_from_prefix(source) {
+ Ok(candidate) => candidate,
+ Err(e) => {
+ return Err(TryReadError::Size(e.with_dst()));
+ }
+ };
+ // SAFETY: `candidate` was copied from from `source: &[u8]`, so all of
+ // its bytes are initialized.
+ unsafe { try_read_from(source, candidate).map(|slf| (slf, suffix)) }
+ }
+
+ /// Attempts to read a `Self` from the suffix of the given `source`.
+ ///
+ /// This attempts to read a `Self` from the last `size_of::<Self>()` bytes
+ /// of `source`, returning that `Self` and any preceding bytes. If
+ /// `source.len() < size_of::<Self>()` or the bytes are not a valid instance
+ /// of `Self`, it returns `Err`.
+ ///
+ /// # Examples
+ ///
+ /// ```
+ /// # #![allow(non_camel_case_types)] // For C0::xC0
+ /// use zerocopy::TryFromBytes;
+ /// # use zerocopy_derive::*;
+ ///
+ /// // The only valid value of this type is the byte `0xC0`
+ /// #[derive(TryFromBytes)]
+ /// #[repr(u8)]
+ /// enum C0 { xC0 = 0xC0 }
+ ///
+ /// // The only valid value of this type is the bytes `0xC0C0`.
+ /// #[derive(TryFromBytes)]
+ /// #[repr(C)]
+ /// struct C0C0(C0, C0);
+ ///
+ /// #[derive(TryFromBytes)]
+ /// #[repr(C)]
+ /// struct Packet {
+ /// magic_number: C0C0,
+ /// mug_size: u8,
+ /// temperature: u8,
+ /// }
+ ///
+ /// // These are more bytes than are needed to encode a `Packet`.
+ /// let bytes = &[0, 1, 2, 3, 4, 5, 0xC0, 0xC0, 240, 77][..];
+ ///
+ /// let (prefix, packet) = Packet::try_read_from_suffix(bytes).unwrap();
+ ///
+ /// assert_eq!(packet.mug_size, 240);
+ /// assert_eq!(packet.temperature, 77);
+ /// assert_eq!(prefix, &[0u8, 1, 2, 3, 4, 5][..]);
+ ///
+ /// // These bytes are not valid instance of `Packet`.
+ /// let bytes = &[0, 1, 2, 3, 4, 5, 0x10, 0xC0, 240, 77][..];
+ /// assert!(Packet::try_read_from_suffix(bytes).is_err());
+ /// ```
+ ///
+ /// # Performance Considerations
+ ///
+ /// In this version of zerocopy, this method reads the `source` into a
+ /// well-aligned stack allocation and *then* validates that the allocation
+ /// is a valid `Self`. This ensures that validation can be performed using
+ /// aligned reads (which carry a performance advantage over unaligned reads
+ /// on many platforms) at the cost of an unconditional copy.
+ ///
+ #[doc = codegen_section!(
+ header = "h5",
+ bench = "try_read_from_suffix",
+ format = "coco_static_size",
+ )]
+ #[must_use = "has no side effects"]
+ #[cfg_attr(zerocopy_inline_always, inline(always))]
+ #[cfg_attr(not(zerocopy_inline_always), inline)]
+ fn try_read_from_suffix(source: &[u8]) -> Result<(&[u8], Self), TryReadError<&[u8], Self>>
+ where
+ Self: Sized,
+ {
+ // FIXME(#2981): If `align_of::<Self>() == 1`, validate `source` in-place.
+
+ let (prefix, candidate) = match CoreMaybeUninit::<Self>::read_from_suffix(source) {
+ Ok(candidate) => candidate,
+ Err(e) => {
+ return Err(TryReadError::Size(e.with_dst()));
+ }
+ };
+ // SAFETY: `candidate` was copied from from `source: &[u8]`, so all of
+ // its bytes are initialized.
+ unsafe { try_read_from(source, candidate).map(|slf| (prefix, slf)) }
+ }
+}
+
+#[inline(always)]
+fn try_ref_from_prefix_suffix<T: TryFromBytes + KnownLayout + Immutable + ?Sized>(
+ source: &[u8],
+ cast_type: CastType,
+ meta: Option<T::PointerMetadata>,
+) -> Result<(&T, &[u8]), TryCastError<&[u8], T>> {
+ match Ptr::from_ref(source).try_cast_into::<T, BecauseImmutable>(cast_type, meta) {
+ Ok((source, prefix_suffix)) => {
+ // This call may panic. If that happens, it doesn't cause any soundness
+ // issues, as we have not generated any invalid state which we need to
+ // fix before returning.
+ match source.try_into_valid() {
+ Ok(valid) => Ok((valid.as_ref(), prefix_suffix.as_ref())),
+ Err(e) => Err(e.map_src(|src| src.as_bytes::<BecauseImmutable>().as_ref()).into()),
+ }
+ }
+ Err(e) => Err(e.map_src(Ptr::as_ref).into()),
+ }
+}
+
+#[inline(always)]
+fn try_mut_from_prefix_suffix<T: IntoBytes + TryFromBytes + KnownLayout + ?Sized>(
+ candidate: &mut [u8],
+ cast_type: CastType,
+ meta: Option<T::PointerMetadata>,
+) -> Result<(&mut T, &mut [u8]), TryCastError<&mut [u8], T>> {
+ match Ptr::from_mut(candidate).try_cast_into::<T, BecauseExclusive>(cast_type, meta) {
+ Ok((candidate, prefix_suffix)) => {
+ // This call may panic. If that happens, it doesn't cause any soundness
+ // issues, as we have not generated any invalid state which we need to
+ // fix before returning.
+ match candidate.try_into_valid() {
+ Ok(valid) => Ok((valid.as_mut(), prefix_suffix.as_mut())),
+ Err(e) => Err(e.map_src(|src| src.as_bytes().as_mut()).into()),
+ }
+ }
+ Err(e) => Err(e.map_src(Ptr::as_mut).into()),
+ }
+}
+
+#[inline(always)]
+fn swap<T, U>((t, u): (T, U)) -> (U, T) {
+ (u, t)
+}
+
+/// # Safety
+///
+/// All bytes of `candidate` must be initialized.
+#[inline(always)]
+unsafe fn try_read_from<S, T: TryFromBytes>(
+ source: S,
+ mut candidate: CoreMaybeUninit<T>,
+) -> Result<T, TryReadError<S, T>> {
+ // We use `from_mut` despite not mutating via `c_ptr` so that we don't need
+ // to add a `T: Immutable` bound.
+ let c_ptr = Ptr::from_mut(&mut candidate);
+ // SAFETY: `c_ptr` has no uninitialized sub-ranges because it derived from
+ // `candidate`, which the caller promises is entirely initialized. Since
+ // `candidate` is a `MaybeUninit`, it has no validity requirements, and so
+ // no values written to an `Initialized` `c_ptr` can violate its validity.
+ // Since `c_ptr` has `Exclusive` aliasing, no mutations may happen except
+ // via `c_ptr` so long as it is live, so we don't need to worry about the
+ // fact that `c_ptr` may have more restricted validity than `candidate`.
+ let c_ptr = unsafe { c_ptr.assume_validity::<invariant::Initialized>() };
+ let mut c_ptr = c_ptr.cast::<_, crate::pointer::cast::CastSized, _>();
+
+ // Since we don't have `T: KnownLayout`, we hack around that by using
+ // `Wrapping<T>`, which implements `KnownLayout` even if `T` doesn't.
+ //
+ // This call may panic. If that happens, it doesn't cause any soundness
+ // issues, as we have not generated any invalid state which we need to fix
+ // before returning.
+ if !Wrapping::<T>::is_bit_valid(c_ptr.reborrow_shared().forget_aligned()) {
+ return Err(ValidityError::new(source).into());
+ }
+
+ fn _assert_same_size_and_validity<T>()
+ where
+ Wrapping<T>: pointer::TransmuteFrom<T, invariant::Valid, invariant::Valid>,
+ T: pointer::TransmuteFrom<Wrapping<T>, invariant::Valid, invariant::Valid>,
+ {
+ }
+
+ _assert_same_size_and_validity::<T>();
+
+ // SAFETY: We just validated that `candidate` contains a valid
+ // `Wrapping<T>`, which has the same size and bit validity as `T`, as
+ // guaranteed by the preceding type assertion.
+ Ok(unsafe { candidate.assume_init() })
+}
+
+/// Types for which a sequence of `0` bytes is a valid instance.
+///
+/// Any memory region of the appropriate length which is guaranteed to contain
+/// only zero bytes can be viewed as any `FromZeros` type with no runtime
+/// overhead. This is useful whenever memory is known to be in a zeroed state,
+/// such memory returned from some allocation routines.
+///
+/// # Warning: Padding bytes
+///
+/// Note that, when a value is moved or copied, only the non-padding bytes of
+/// that value are guaranteed to be preserved. It is unsound to assume that
+/// values written to padding bytes are preserved after a move or copy. For more
+/// details, see the [`FromBytes` docs][frombytes-warning-padding-bytes].
+///
+/// [frombytes-warning-padding-bytes]: FromBytes#warning-padding-bytes
+///
+/// # Implementation
+///
+/// **Do not implement this trait yourself!** Instead, use
+/// [`#[derive(FromZeros)]`][derive]; e.g.:
+///
+/// ```
+/// # use zerocopy_derive::{FromZeros, Immutable};
+/// #[derive(FromZeros)]
+/// struct MyStruct {
+/// # /*
+/// ...
+/// # */
+/// }
+///
+/// #[derive(FromZeros)]
+/// #[repr(u8)]
+/// enum MyEnum {
+/// # Variant0,
+/// # /*
+/// ...
+/// # */
+/// }
+///
+/// #[derive(FromZeros, Immutable)]
+/// union MyUnion {
+/// # variant: u8,
+/// # /*
+/// ...
+/// # */
+/// }
+/// ```
+///
+/// This derive performs a sophisticated, compile-time safety analysis to
+/// determine whether a type is `FromZeros`.
+///
+/// # Safety
+///
+/// *This section describes what is required in order for `T: FromZeros`, and
+/// what unsafe code may assume of such types. If you don't plan on implementing
+/// `FromZeros` manually, and you don't plan on writing unsafe code that
+/// operates on `FromZeros` types, then you don't need to read this section.*
+///
+/// If `T: FromZeros`, then unsafe code may assume that it is sound to produce a
+/// `T` whose bytes are all initialized to zero. If a type is marked as
+/// `FromZeros` which violates this contract, it may cause undefined behavior.
+///
+/// `#[derive(FromZeros)]` only permits [types which satisfy these
+/// requirements][derive-analysis].
+///
+#[cfg_attr(
+ feature = "derive",
+ doc = "[derive]: zerocopy_derive::FromZeros",
+ doc = "[derive-analysis]: zerocopy_derive::FromZeros#analysis"
+)]
+#[cfg_attr(
+ not(feature = "derive"),
+ doc = concat!("[derive]: https://docs.rs/zerocopy/", env!("CARGO_PKG_VERSION"), "/zerocopy/derive.FromZeros.html"),
+ doc = concat!("[derive-analysis]: https://docs.rs/zerocopy/", env!("CARGO_PKG_VERSION"), "/zerocopy/derive.FromZeros.html#analysis"),
+)]
+#[cfg_attr(
+ not(no_zerocopy_diagnostic_on_unimplemented_1_78_0),
+ diagnostic::on_unimplemented(note = "Consider adding `#[derive(FromZeros)]` to `{Self}`")
+)]
+pub unsafe trait FromZeros: TryFromBytes {
+ // The `Self: Sized` bound makes it so that `FromZeros` is still object
+ // safe.
+ #[doc(hidden)]
+ fn only_derive_is_allowed_to_implement_this_trait()
+ where
+ Self: Sized;
+
+ /// Overwrites `self` with zeros.
+ ///
+ /// Sets every byte in `self` to 0. While this is similar to doing `*self =
+ /// Self::new_zeroed()`, it differs in that `zero` does not semantically
+ /// drop the current value and replace it with a new one — it simply
+ /// modifies the bytes of the existing value.
+ ///
+ /// # Examples
+ ///
+ /// ```
+ /// # use zerocopy::FromZeros;
+ /// # use zerocopy_derive::*;
+ /// #
+ /// #[derive(FromZeros)]
+ /// #[repr(C)]
+ /// struct PacketHeader {
+ /// src_port: [u8; 2],
+ /// dst_port: [u8; 2],
+ /// length: [u8; 2],
+ /// checksum: [u8; 2],
+ /// }
+ ///
+ /// let mut header = PacketHeader {
+ /// src_port: 100u16.to_be_bytes(),
+ /// dst_port: 200u16.to_be_bytes(),
+ /// length: 300u16.to_be_bytes(),
+ /// checksum: 400u16.to_be_bytes(),
+ /// };
+ ///
+ /// header.zero();
+ ///
+ /// assert_eq!(header.src_port, [0, 0]);
+ /// assert_eq!(header.dst_port, [0, 0]);
+ /// assert_eq!(header.length, [0, 0]);
+ /// assert_eq!(header.checksum, [0, 0]);
+ /// ```
+ ///
+ #[doc = codegen_section!(
+ header = "h5",
+ bench = "zero",
+ format = "coco",
+ arity = 3,
+ [
+ open
+ @index 1
+ @title "Sized"
+ @variant "static_size"
+ ],
+ [
+ @index 2
+ @title "Unsized"
+ @variant "dynamic_size"
+ ],
+ [
+ @index 3
+ @title "Dynamically Padded"
+ @variant "dynamic_padding"
+ ]
+ )]
+ #[inline(always)]
+ fn zero(&mut self) {
+ let slf: *mut Self = self;
+ let len = mem::size_of_val(self);
+ // SAFETY:
+ // - `self` is guaranteed by the type system to be valid for writes of
+ // size `size_of_val(self)`.
+ // - `u8`'s alignment is 1, and thus `self` is guaranteed to be aligned
+ // as required by `u8`.
+ // - Since `Self: FromZeros`, the all-zeros instance is a valid instance
+ // of `Self.`
+ //
+ // FIXME(#429): Add references to docs and quotes.
+ unsafe { ptr::write_bytes(slf.cast::<u8>(), 0, len) };
+ }
+
+ /// Creates an instance of `Self` from zeroed bytes.
+ ///
+ /// # Examples
+ ///
+ /// ```
+ /// # use zerocopy::FromZeros;
+ /// # use zerocopy_derive::*;
+ /// #
+ /// #[derive(FromZeros)]
+ /// #[repr(C)]
+ /// struct PacketHeader {
+ /// src_port: [u8; 2],
+ /// dst_port: [u8; 2],
+ /// length: [u8; 2],
+ /// checksum: [u8; 2],
+ /// }
+ ///
+ /// let header: PacketHeader = FromZeros::new_zeroed();
+ ///
+ /// assert_eq!(header.src_port, [0, 0]);
+ /// assert_eq!(header.dst_port, [0, 0]);
+ /// assert_eq!(header.length, [0, 0]);
+ /// assert_eq!(header.checksum, [0, 0]);
+ /// ```
+ ///
+ #[doc = codegen_section!(
+ header = "h5",
+ bench = "new_zeroed",
+ format = "coco_static_size",
+ )]
+ #[must_use = "has no side effects"]
+ #[inline(always)]
+ fn new_zeroed() -> Self
+ where
+ Self: Sized,
+ {
+ // SAFETY: `FromZeros` says that the all-zeros bit pattern is legal.
+ unsafe { mem::zeroed() }
+ }
+
+ /// Creates a `Box<Self>` from zeroed bytes.
+ ///
+ /// This function is useful for allocating large values on the heap and
+ /// zero-initializing them, without ever creating a temporary instance of
+ /// `Self` on the stack. For example, `<[u8; 1048576]>::new_box_zeroed()`
+ /// will allocate `[u8; 1048576]` directly on the heap; it does not require
+ /// storing `[u8; 1048576]` in a temporary variable on the stack.
+ ///
+ /// On systems that use a heap implementation that supports allocating from
+ /// pre-zeroed memory, using `new_box_zeroed` (or related functions) may
+ /// have performance benefits.
+ ///
+ /// # Errors
+ ///
+ /// Returns an error on allocation failure. Allocation failure is guaranteed
+ /// never to cause a panic or an abort.
+ ///
+ #[doc = codegen_section!(
+ header = "h5",
+ bench = "new_box_zeroed",
+ format = "coco_static_size",
+ )]
+ #[must_use = "has no side effects (other than allocation)"]
+ #[cfg(any(feature = "alloc", test))]
+ #[cfg_attr(doc_cfg, doc(cfg(feature = "alloc")))]
+ #[inline]
+ fn new_box_zeroed() -> Result<Box<Self>, AllocError>
+ where
+ Self: Sized,
+ {
+ // If `T` is a ZST, then return a proper boxed instance of it. There is
+ // no allocation, but `Box` does require a correct dangling pointer.
+ let layout = Layout::new::<Self>();
+ if layout.size() == 0 {
+ // Construct the `Box` from a dangling pointer to avoid calling
+ // `Self::new_zeroed`. This ensures that stack space is never
+ // allocated for `Self` even on lower opt-levels where this branch
+ // might not get optimized out.
+
+ // SAFETY: Per [1], when `T` is a ZST, `Box<T>`'s only validity
+ // requirements are that the pointer is non-null and sufficiently
+ // aligned. Per [2], `NonNull::dangling` produces a pointer which
+ // is sufficiently aligned. Since the produced pointer is a
+ // `NonNull`, it is non-null.
+ //
+ // [1] Per https://doc.rust-lang.org/1.81.0/std/boxed/index.html#memory-layout:
+ //
+ // For zero-sized values, the `Box` pointer has to be non-null and sufficiently aligned.
+ //
+ // [2] Per https://doc.rust-lang.org/std/ptr/struct.NonNull.html#method.dangling:
+ //
+ // Creates a new `NonNull` that is dangling, but well-aligned.
+ return Ok(unsafe { Box::from_raw(NonNull::dangling().as_ptr()) });
+ }
+
+ // FIXME(#429): Add a "SAFETY" comment and remove this `allow`.
+ #[allow(clippy::undocumented_unsafe_blocks)]
+ let ptr = unsafe { alloc::alloc::alloc_zeroed(layout).cast::<Self>() };
+ if ptr.is_null() {
+ return Err(AllocError);
+ }
+ // FIXME(#429): Add a "SAFETY" comment and remove this `allow`.
+ #[allow(clippy::undocumented_unsafe_blocks)]
+ Ok(unsafe { Box::from_raw(ptr) })
+ }
+
+ /// Creates a `Box<[Self]>` (a boxed slice) from zeroed bytes.
+ ///
+ /// This function is useful for allocating large values of `[Self]` on the
+ /// heap and zero-initializing them, without ever creating a temporary
+ /// instance of `[Self; _]` on the stack. For example,
+ /// `u8::new_box_slice_zeroed(1048576)` will allocate the slice directly on
+ /// the heap; it does not require storing the slice on the stack.
+ ///
+ /// On systems that use a heap implementation that supports allocating from
+ /// pre-zeroed memory, using `new_box_slice_zeroed` may have performance
+ /// benefits.
+ ///
+ /// If `Self` is a zero-sized type, then this function will return a
+ /// `Box<[Self]>` that has the correct `len`. Such a box cannot contain any
+ /// actual information, but its `len()` property will report the correct
+ /// value.
+ ///
+ /// # Errors
+ ///
+ /// Returns an error on allocation failure. Allocation failure is
+ /// guaranteed never to cause a panic or an abort.
+ ///
+ #[doc = codegen_section!(
+ header = "h5",
+ bench = "new_box_zeroed_with_elems",
+ format = "coco",
+ arity = 2,
+ [
+ open
+ @index 1
+ @title "Unsized"
+ @variant "dynamic_size"
+ ],
+ [
+ @index 2
+ @title "Dynamically Padded"
+ @variant "dynamic_padding"
+ ]
+ )]
+ #[must_use = "has no side effects (other than allocation)"]
+ #[cfg(feature = "alloc")]
+ #[cfg_attr(doc_cfg, doc(cfg(feature = "alloc")))]
+ #[inline]
+ fn new_box_zeroed_with_elems(count: usize) -> Result<Box<Self>, AllocError>
+ where
+ Self: KnownLayout<PointerMetadata = usize>,
+ {
+ // SAFETY: `alloc::alloc::alloc_zeroed` is a valid argument of
+ // `new_box`. The referent of the pointer returned by `alloc_zeroed`
+ // (and, consequently, the `Box` derived from it) is a valid instance of
+ // `Self`, because `Self` is `FromZeros`.
+ unsafe { crate::util::new_box(count, alloc::alloc::alloc_zeroed) }
+ }
+
+ #[deprecated(since = "0.8.0", note = "renamed to `FromZeros::new_box_zeroed_with_elems`")]
+ #[doc(hidden)]
+ #[cfg(feature = "alloc")]
+ #[cfg_attr(doc_cfg, doc(cfg(feature = "alloc")))]
+ #[must_use = "has no side effects (other than allocation)"]
+ #[inline(always)]
+ fn new_box_slice_zeroed(len: usize) -> Result<Box<[Self]>, AllocError>
+ where
+ Self: Sized,
+ {
+ <[Self]>::new_box_zeroed_with_elems(len)
+ }
+
+ /// Creates a `Vec<Self>` from zeroed bytes.
+ ///
+ /// This function is useful for allocating large values of `Vec`s and
+ /// zero-initializing them, without ever creating a temporary instance of
+ /// `[Self; _]` (or many temporary instances of `Self`) on the stack. For
+ /// example, `u8::new_vec_zeroed(1048576)` will allocate directly on the
+ /// heap; it does not require storing intermediate values on the stack.
+ ///
+ /// On systems that use a heap implementation that supports allocating from
+ /// pre-zeroed memory, using `new_vec_zeroed` may have performance benefits.
+ ///
+ /// If `Self` is a zero-sized type, then this function will return a
+ /// `Vec<Self>` that has the correct `len`. Such a `Vec` cannot contain any
+ /// actual information, but its `len()` property will report the correct
+ /// value.
+ ///
+ /// # Errors
+ ///
+ /// Returns an error on allocation failure. Allocation failure is
+ /// guaranteed never to cause a panic or an abort.
+ ///
+ #[doc = codegen_section!(
+ header = "h5",
+ bench = "new_vec_zeroed",
+ format = "coco_static_size",
+ )]
+ #[must_use = "has no side effects (other than allocation)"]
+ #[cfg(feature = "alloc")]
+ #[cfg_attr(doc_cfg, doc(cfg(feature = "alloc")))]
+ #[inline(always)]
+ fn new_vec_zeroed(len: usize) -> Result<Vec<Self>, AllocError>
+ where
+ Self: Sized,
+ {
+ <[Self]>::new_box_zeroed_with_elems(len).map(Into::into)
+ }
+
+ /// Extends a `Vec<Self>` by pushing `additional` new items onto the end of
+ /// the vector. The new items are initialized with zeros.
+ ///
+ #[doc = codegen_section!(
+ header = "h5",
+ bench = "extend_vec_zeroed",
+ format = "coco_static_size",
+ )]
+ #[cfg(not(no_zerocopy_panic_in_const_and_vec_try_reserve_1_57_0))]
+ #[cfg(feature = "alloc")]
+ #[cfg_attr(doc_cfg, doc(cfg(all(rust = "1.57.0", feature = "alloc"))))]
+ #[inline(always)]
+ fn extend_vec_zeroed(v: &mut Vec<Self>, additional: usize) -> Result<(), AllocError>
+ where
+ Self: Sized,
+ {
+ // PANICS: We pass `v.len()` for `position`, so the `position > v.len()`
+ // panic condition is not satisfied.
+ <Self as FromZeros>::insert_vec_zeroed(v, v.len(), additional)
+ }
+
+ /// Inserts `additional` new items into `Vec<Self>` at `position`. The new
+ /// items are initialized with zeros.
+ ///
+ /// # Panics
+ ///
+ /// Panics if `position > v.len()`.
+ ///
+ #[doc = codegen_section!(
+ header = "h5",
+ bench = "insert_vec_zeroed",
+ format = "coco_static_size",
+ )]
+ #[cfg(not(no_zerocopy_panic_in_const_and_vec_try_reserve_1_57_0))]
+ #[cfg(feature = "alloc")]
+ #[cfg_attr(doc_cfg, doc(cfg(all(rust = "1.57.0", feature = "alloc"))))]
+ #[inline]
+ fn insert_vec_zeroed(
+ v: &mut Vec<Self>,
+ position: usize,
+ additional: usize,
+ ) -> Result<(), AllocError>
+ where
+ Self: Sized,
+ {
+ assert!(position <= v.len());
+ // We only conditionally compile on versions on which `try_reserve` is
+ // stable; the Clippy lint is a false positive.
+ v.try_reserve(additional).map_err(|_| AllocError)?;
+ // SAFETY: The `try_reserve` call guarantees that these cannot overflow:
+ // * `ptr.add(position)`
+ // * `position + additional`
+ // * `v.len() + additional`
+ //
+ // `v.len() - position` cannot overflow because we asserted that
+ // `position <= v.len()`.
+ #[allow(clippy::multiple_unsafe_ops_per_block)]
+ unsafe {
+ // This is a potentially overlapping copy.
+ let ptr = v.as_mut_ptr();
+ #[allow(clippy::arithmetic_side_effects)]
+ ptr.add(position).copy_to(ptr.add(position + additional), v.len() - position);
+ ptr.add(position).write_bytes(0, additional);
+ #[allow(clippy::arithmetic_side_effects)]
+ v.set_len(v.len() + additional);
+ }
+
+ Ok(())
+ }
+}
+
+/// Analyzes whether a type is [`FromBytes`].
+///
+/// This derive analyzes, at compile time, whether the annotated type satisfies
+/// the [safety conditions] of `FromBytes` and implements `FromBytes` and its
+/// supertraits if it is sound to do so. This derive can be applied to structs,
+/// enums, and unions;
+/// e.g.:
+///
+/// ```
+/// # use zerocopy_derive::{FromBytes, FromZeros, Immutable};
+/// #[derive(FromBytes)]
+/// struct MyStruct {
+/// # /*
+/// ...
+/// # */
+/// }
+///
+/// #[derive(FromBytes)]
+/// #[repr(u8)]
+/// enum MyEnum {
+/// # V00, V01, V02, V03, V04, V05, V06, V07, V08, V09, V0A, V0B, V0C, V0D, V0E,
+/// # V0F, V10, V11, V12, V13, V14, V15, V16, V17, V18, V19, V1A, V1B, V1C, V1D,
+/// # V1E, V1F, V20, V21, V22, V23, V24, V25, V26, V27, V28, V29, V2A, V2B, V2C,
+/// # V2D, V2E, V2F, V30, V31, V32, V33, V34, V35, V36, V37, V38, V39, V3A, V3B,
+/// # V3C, V3D, V3E, V3F, V40, V41, V42, V43, V44, V45, V46, V47, V48, V49, V4A,
+/// # V4B, V4C, V4D, V4E, V4F, V50, V51, V52, V53, V54, V55, V56, V57, V58, V59,
+/// # V5A, V5B, V5C, V5D, V5E, V5F, V60, V61, V62, V63, V64, V65, V66, V67, V68,
+/// # V69, V6A, V6B, V6C, V6D, V6E, V6F, V70, V71, V72, V73, V74, V75, V76, V77,
+/// # V78, V79, V7A, V7B, V7C, V7D, V7E, V7F, V80, V81, V82, V83, V84, V85, V86,
+/// # V87, V88, V89, V8A, V8B, V8C, V8D, V8E, V8F, V90, V91, V92, V93, V94, V95,
+/// # V96, V97, V98, V99, V9A, V9B, V9C, V9D, V9E, V9F, VA0, VA1, VA2, VA3, VA4,
+/// # VA5, VA6, VA7, VA8, VA9, VAA, VAB, VAC, VAD, VAE, VAF, VB0, VB1, VB2, VB3,
+/// # VB4, VB5, VB6, VB7, VB8, VB9, VBA, VBB, VBC, VBD, VBE, VBF, VC0, VC1, VC2,
+/// # VC3, VC4, VC5, VC6, VC7, VC8, VC9, VCA, VCB, VCC, VCD, VCE, VCF, VD0, VD1,
+/// # VD2, VD3, VD4, VD5, VD6, VD7, VD8, VD9, VDA, VDB, VDC, VDD, VDE, VDF, VE0,
+/// # VE1, VE2, VE3, VE4, VE5, VE6, VE7, VE8, VE9, VEA, VEB, VEC, VED, VEE, VEF,
+/// # VF0, VF1, VF2, VF3, VF4, VF5, VF6, VF7, VF8, VF9, VFA, VFB, VFC, VFD, VFE,
+/// # VFF,
+/// # /*
+/// ...
+/// # */
+/// }
+///
+/// #[derive(FromBytes, Immutable)]
+/// union MyUnion {
+/// # variant: u8,
+/// # /*
+/// ...
+/// # */
+/// }
+/// ```
+///
+/// [safety conditions]: trait@FromBytes#safety
+///
+/// # Analysis
+///
+/// *This section describes, roughly, the analysis performed by this derive to
+/// determine whether it is sound to implement `FromBytes` for a given type.
+/// Unless you are modifying the implementation of this derive, or attempting to
+/// manually implement `FromBytes` for a type yourself, you don't need to read
+/// this section.*
+///
+/// If a type has the following properties, then this derive can implement
+/// `FromBytes` for that type:
+///
+/// - If the type is a struct, all of its fields must be `FromBytes`.
+/// - If the type is an enum:
+/// - It must have a defined representation which is one of `u8`, `u16`, `i8`,
+/// or `i16`.
+/// - The maximum number of discriminants must be used (so that every possible
+/// bit pattern is a valid one).
+/// - Its fields must be `FromBytes`.
+///
+/// This analysis is subject to change. Unsafe code may *only* rely on the
+/// documented [safety conditions] of `FromBytes`, and must *not* rely on the
+/// implementation details of this derive.
+///
+/// ## Why isn't an explicit representation required for structs?
+///
+/// Neither this derive, nor the [safety conditions] of `FromBytes`, requires
+/// that structs are marked with `#[repr(C)]`.
+///
+/// Per the [Rust reference](reference),
+///
+/// > The representation of a type can change the padding between fields, but
+/// > does not change the layout of the fields themselves.
+///
+/// [reference]: https://doc.rust-lang.org/reference/type-layout.html#representations
+///
+/// Since the layout of structs only consists of padding bytes and field bytes,
+/// a struct is soundly `FromBytes` if:
+/// 1. its padding is soundly `FromBytes`, and
+/// 2. its fields are soundly `FromBytes`.
+///
+/// The answer to the first question is always yes: padding bytes do not have
+/// any validity constraints. A [discussion] of this question in the Unsafe Code
+/// Guidelines Working Group concluded that it would be virtually unimaginable
+/// for future versions of rustc to add validity constraints to padding bytes.
+///
+/// [discussion]: https://github.com/rust-lang/unsafe-code-guidelines/issues/174
+///
+/// Whether a struct is soundly `FromBytes` therefore solely depends on whether
+/// its fields are `FromBytes`.
+#[cfg(any(feature = "derive", test))]
+#[cfg_attr(doc_cfg, doc(cfg(feature = "derive")))]
+pub use zerocopy_derive::FromBytes;
+
+/// Types for which any bit pattern is valid.
+///
+/// Any memory region of the appropriate length which contains initialized bytes
+/// can be viewed as any `FromBytes` type with no runtime overhead. This is
+/// useful for efficiently parsing bytes as structured data.
+///
+/// # Warning: Padding bytes
+///
+/// Note that, when a value is moved or copied, only the non-padding bytes of
+/// that value are guaranteed to be preserved. It is unsound to assume that
+/// values written to padding bytes are preserved after a move or copy. For
+/// example, the following is unsound:
+///
+/// ```rust,no_run
+/// use core::mem::{size_of, transmute};
+/// use zerocopy::FromZeros;
+/// # use zerocopy_derive::*;
+///
+/// // Assume `Foo` is a type with padding bytes.
+/// #[derive(FromZeros, Default)]
+/// struct Foo {
+/// # /*
+/// ...
+/// # */
+/// }
+///
+/// let mut foo: Foo = Foo::default();
+/// FromZeros::zero(&mut foo);
+/// // UNSOUND: Although `FromZeros::zero` writes zeros to all bytes of `foo`,
+/// // those writes are not guaranteed to be preserved in padding bytes when
+/// // `foo` is moved, so this may expose padding bytes as `u8`s.
+/// let foo_bytes: [u8; size_of::<Foo>()] = unsafe { transmute(foo) };
+/// ```
+///
+/// # Implementation
+///
+/// **Do not implement this trait yourself!** Instead, use
+/// [`#[derive(FromBytes)]`][derive]; e.g.:
+///
+/// ```
+/// # use zerocopy_derive::{FromBytes, Immutable};
+/// #[derive(FromBytes)]
+/// struct MyStruct {
+/// # /*
+/// ...
+/// # */
+/// }
+///
+/// #[derive(FromBytes)]
+/// #[repr(u8)]
+/// enum MyEnum {
+/// # V00, V01, V02, V03, V04, V05, V06, V07, V08, V09, V0A, V0B, V0C, V0D, V0E,
+/// # V0F, V10, V11, V12, V13, V14, V15, V16, V17, V18, V19, V1A, V1B, V1C, V1D,
+/// # V1E, V1F, V20, V21, V22, V23, V24, V25, V26, V27, V28, V29, V2A, V2B, V2C,
+/// # V2D, V2E, V2F, V30, V31, V32, V33, V34, V35, V36, V37, V38, V39, V3A, V3B,
+/// # V3C, V3D, V3E, V3F, V40, V41, V42, V43, V44, V45, V46, V47, V48, V49, V4A,
+/// # V4B, V4C, V4D, V4E, V4F, V50, V51, V52, V53, V54, V55, V56, V57, V58, V59,
+/// # V5A, V5B, V5C, V5D, V5E, V5F, V60, V61, V62, V63, V64, V65, V66, V67, V68,
+/// # V69, V6A, V6B, V6C, V6D, V6E, V6F, V70, V71, V72, V73, V74, V75, V76, V77,
+/// # V78, V79, V7A, V7B, V7C, V7D, V7E, V7F, V80, V81, V82, V83, V84, V85, V86,
+/// # V87, V88, V89, V8A, V8B, V8C, V8D, V8E, V8F, V90, V91, V92, V93, V94, V95,
+/// # V96, V97, V98, V99, V9A, V9B, V9C, V9D, V9E, V9F, VA0, VA1, VA2, VA3, VA4,
+/// # VA5, VA6, VA7, VA8, VA9, VAA, VAB, VAC, VAD, VAE, VAF, VB0, VB1, VB2, VB3,
+/// # VB4, VB5, VB6, VB7, VB8, VB9, VBA, VBB, VBC, VBD, VBE, VBF, VC0, VC1, VC2,
+/// # VC3, VC4, VC5, VC6, VC7, VC8, VC9, VCA, VCB, VCC, VCD, VCE, VCF, VD0, VD1,
+/// # VD2, VD3, VD4, VD5, VD6, VD7, VD8, VD9, VDA, VDB, VDC, VDD, VDE, VDF, VE0,
+/// # VE1, VE2, VE3, VE4, VE5, VE6, VE7, VE8, VE9, VEA, VEB, VEC, VED, VEE, VEF,
+/// # VF0, VF1, VF2, VF3, VF4, VF5, VF6, VF7, VF8, VF9, VFA, VFB, VFC, VFD, VFE,
+/// # VFF,
+/// # /*
+/// ...
+/// # */
+/// }
+///
+/// #[derive(FromBytes, Immutable)]
+/// union MyUnion {
+/// # variant: u8,
+/// # /*
+/// ...
+/// # */
+/// }
+/// ```
+///
+/// This derive performs a sophisticated, compile-time safety analysis to
+/// determine whether a type is `FromBytes`.
+///
+/// # Safety
+///
+/// *This section describes what is required in order for `T: FromBytes`, and
+/// what unsafe code may assume of such types. If you don't plan on implementing
+/// `FromBytes` manually, and you don't plan on writing unsafe code that
+/// operates on `FromBytes` types, then you don't need to read this section.*
+///
+/// If `T: FromBytes`, then unsafe code may assume that it is sound to produce a
+/// `T` whose bytes are initialized to any sequence of valid `u8`s (in other
+/// words, any byte value which is not uninitialized). If a type is marked as
+/// `FromBytes` which violates this contract, it may cause undefined behavior.
+///
+/// `#[derive(FromBytes)]` only permits [types which satisfy these
+/// requirements][derive-analysis].
+///
+#[cfg_attr(
+ feature = "derive",
+ doc = "[derive]: zerocopy_derive::FromBytes",
+ doc = "[derive-analysis]: zerocopy_derive::FromBytes#analysis"
+)]
+#[cfg_attr(
+ not(feature = "derive"),
+ doc = concat!("[derive]: https://docs.rs/zerocopy/", env!("CARGO_PKG_VERSION"), "/zerocopy/derive.FromBytes.html"),
+ doc = concat!("[derive-analysis]: https://docs.rs/zerocopy/", env!("CARGO_PKG_VERSION"), "/zerocopy/derive.FromBytes.html#analysis"),
+)]
+#[cfg_attr(
+ not(no_zerocopy_diagnostic_on_unimplemented_1_78_0),
+ diagnostic::on_unimplemented(note = "Consider adding `#[derive(FromBytes)]` to `{Self}`")
+)]
+pub unsafe trait FromBytes: FromZeros {
+ // The `Self: Sized` bound makes it so that `FromBytes` is still object
+ // safe.
+ #[doc(hidden)]
+ fn only_derive_is_allowed_to_implement_this_trait()
+ where
+ Self: Sized;
+
+ /// Interprets the given `source` as a `&Self`.
+ ///
+ /// This method attempts to return a reference to `source` interpreted as a
+ /// `Self`. If the length of `source` is not a [valid size of
+ /// `Self`][valid-size], or if `source` is not appropriately aligned, this
+ /// returns `Err`. If [`Self: Unaligned`][self-unaligned], you can
+ /// [infallibly discard the alignment error][size-error-from].
+ ///
+ /// `Self` may be a sized type, a slice, or a [slice DST][slice-dst].
+ ///
+ /// [valid-size]: crate::KnownLayout#what-is-a-valid-size
+ /// [self-unaligned]: Unaligned
+ /// [size-error-from]: error/struct.SizeError.html#method.from-1
+ /// [slice-dst]: KnownLayout#dynamically-sized-types
+ ///
+ /// # Compile-Time Assertions
+ ///
+ /// This method cannot yet be used on unsized types whose dynamically-sized
+ /// component is zero-sized. Attempting to use this method on such types
+ /// results in a compile-time assertion error; e.g.:
+ ///
+ /// ```compile_fail,E0080
+ /// use zerocopy::*;
+ /// # use zerocopy_derive::*;
+ ///
+ /// #[derive(FromBytes, Immutable, KnownLayout)]
+ /// #[repr(C)]
+ /// struct ZSTy {
+ /// leading_sized: u16,
+ /// trailing_dst: [()],
+ /// }
+ ///
+ /// let _ = ZSTy::ref_from_bytes(0u16.as_bytes()); // ⚠ Compile Error!
+ /// ```
+ ///
+ /// # Examples
+ ///
+ /// ```
+ /// use zerocopy::FromBytes;
+ /// # use zerocopy_derive::*;
+ ///
+ /// #[derive(FromBytes, KnownLayout, Immutable)]
+ /// #[repr(C)]
+ /// struct PacketHeader {
+ /// src_port: [u8; 2],
+ /// dst_port: [u8; 2],
+ /// length: [u8; 2],
+ /// checksum: [u8; 2],
+ /// }
+ ///
+ /// #[derive(FromBytes, KnownLayout, Immutable)]
+ /// #[repr(C)]
+ /// struct Packet {
+ /// header: PacketHeader,
+ /// body: [u8],
+ /// }
+ ///
+ /// // These bytes encode a `Packet`.
+ /// let bytes = &[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11][..];
+ ///
+ /// let packet = Packet::ref_from_bytes(bytes).unwrap();
+ ///
+ /// assert_eq!(packet.header.src_port, [0, 1]);
+ /// assert_eq!(packet.header.dst_port, [2, 3]);
+ /// assert_eq!(packet.header.length, [4, 5]);
+ /// assert_eq!(packet.header.checksum, [6, 7]);
+ /// assert_eq!(packet.body, [8, 9, 10, 11]);
+ /// ```
+ ///
+ #[doc = codegen_section!(
+ header = "h5",
+ bench = "ref_from_bytes",
+ format = "coco",
+ arity = 3,
+ [
+ open
+ @index 1
+ @title "Sized"
+ @variant "static_size"
+ ],
+ [
+ @index 2
+ @title "Unsized"
+ @variant "dynamic_size"
+ ],
+ [
+ @index 3
+ @title "Dynamically Padded"
+ @variant "dynamic_padding"
+ ]
+ )]
+ #[must_use = "has no side effects"]
+ #[cfg_attr(zerocopy_inline_always, inline(always))]
+ #[cfg_attr(not(zerocopy_inline_always), inline)]
+ fn ref_from_bytes(source: &[u8]) -> Result<&Self, CastError<&[u8], Self>>
+ where
+ Self: KnownLayout + Immutable,
+ {
+ static_assert_dst_is_not_zst!(Self);
+ match Ptr::from_ref(source).try_cast_into_no_leftover::<_, BecauseImmutable>(None) {
+ Ok(ptr) => Ok(ptr.recall_validity().as_ref()),
+ Err(err) => Err(err.map_src(|src| src.as_ref())),
+ }
+ }
+
+ /// Interprets the prefix of the given `source` as a `&Self` without
+ /// copying.
+ ///
+ /// This method computes the [largest possible size of `Self`][valid-size]
+ /// that can fit in the leading bytes of `source`, then attempts to return
+ /// both a reference to those bytes interpreted as a `Self`, and a reference
+ /// to the remaining bytes. If there are insufficient bytes, or if `source`
+ /// is not appropriately aligned, this returns `Err`. If [`Self:
+ /// Unaligned`][self-unaligned], you can [infallibly discard the alignment
+ /// error][size-error-from].
+ ///
+ /// `Self` may be a sized type, a slice, or a [slice DST][slice-dst].
+ ///
+ /// [valid-size]: crate::KnownLayout#what-is-a-valid-size
+ /// [self-unaligned]: Unaligned
+ /// [size-error-from]: error/struct.SizeError.html#method.from-1
+ /// [slice-dst]: KnownLayout#dynamically-sized-types
+ ///
+ /// # Compile-Time Assertions
+ ///
+ /// This method cannot yet be used on unsized types whose dynamically-sized
+ /// component is zero-sized. See [`ref_from_prefix_with_elems`], which does
+ /// support such types. Attempting to use this method on such types results
+ /// in a compile-time assertion error; e.g.:
+ ///
+ /// ```compile_fail,E0080
+ /// use zerocopy::*;
+ /// # use zerocopy_derive::*;
+ ///
+ /// #[derive(FromBytes, Immutable, KnownLayout)]
+ /// #[repr(C)]
+ /// struct ZSTy {
+ /// leading_sized: u16,
+ /// trailing_dst: [()],
+ /// }
+ ///
+ /// let _ = ZSTy::ref_from_prefix(0u16.as_bytes()); // ⚠ Compile Error!
+ /// ```
+ ///
+ /// [`ref_from_prefix_with_elems`]: FromBytes::ref_from_prefix_with_elems
+ ///
+ /// # Examples
+ ///
+ /// ```
+ /// use zerocopy::FromBytes;
+ /// # use zerocopy_derive::*;
+ ///
+ /// #[derive(FromBytes, KnownLayout, Immutable)]
+ /// #[repr(C)]
+ /// struct PacketHeader {
+ /// src_port: [u8; 2],
+ /// dst_port: [u8; 2],
+ /// length: [u8; 2],
+ /// checksum: [u8; 2],
+ /// }
+ ///
+ /// #[derive(FromBytes, KnownLayout, Immutable)]
+ /// #[repr(C)]
+ /// struct Packet {
+ /// header: PacketHeader,
+ /// body: [[u8; 2]],
+ /// }
+ ///
+ /// // These are more bytes than are needed to encode a `Packet`.
+ /// let bytes = &[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14][..];
+ ///
+ /// let (packet, suffix) = Packet::ref_from_prefix(bytes).unwrap();
+ ///
+ /// assert_eq!(packet.header.src_port, [0, 1]);
+ /// assert_eq!(packet.header.dst_port, [2, 3]);
+ /// assert_eq!(packet.header.length, [4, 5]);
+ /// assert_eq!(packet.header.checksum, [6, 7]);
+ /// assert_eq!(packet.body, [[8, 9], [10, 11], [12, 13]]);
+ /// assert_eq!(suffix, &[14u8][..]);
+ /// ```
+ ///
+ #[doc = codegen_section!(
+ header = "h5",
+ bench = "ref_from_prefix",
+ format = "coco",
+ arity = 3,
+ [
+ open
+ @index 1
+ @title "Sized"
+ @variant "static_size"
+ ],
+ [
+ @index 2
+ @title "Unsized"
+ @variant "dynamic_size"
+ ],
+ [
+ @index 3
+ @title "Dynamically Padded"
+ @variant "dynamic_padding"
+ ]
+ )]
+ #[must_use = "has no side effects"]
+ #[cfg_attr(zerocopy_inline_always, inline(always))]
+ #[cfg_attr(not(zerocopy_inline_always), inline)]
+ fn ref_from_prefix(source: &[u8]) -> Result<(&Self, &[u8]), CastError<&[u8], Self>>
+ where
+ Self: KnownLayout + Immutable,
+ {
+ static_assert_dst_is_not_zst!(Self);
+ ref_from_prefix_suffix(source, None, CastType::Prefix)
+ }
+
+ /// Interprets the suffix of the given bytes as a `&Self`.
+ ///
+ /// This method computes the [largest possible size of `Self`][valid-size]
+ /// that can fit in the trailing bytes of `source`, then attempts to return
+ /// both a reference to those bytes interpreted as a `Self`, and a reference
+ /// to the preceding bytes. If there are insufficient bytes, or if that
+ /// suffix of `source` is not appropriately aligned, this returns `Err`. If
+ /// [`Self: Unaligned`][self-unaligned], you can [infallibly discard the
+ /// alignment error][size-error-from].
+ ///
+ /// `Self` may be a sized type, a slice, or a [slice DST][slice-dst].
+ ///
+ /// [valid-size]: crate::KnownLayout#what-is-a-valid-size
+ /// [self-unaligned]: Unaligned
+ /// [size-error-from]: error/struct.SizeError.html#method.from-1
+ /// [slice-dst]: KnownLayout#dynamically-sized-types
+ ///
+ /// # Compile-Time Assertions
+ ///
+ /// This method cannot yet be used on unsized types whose dynamically-sized
+ /// component is zero-sized. See [`ref_from_suffix_with_elems`], which does
+ /// support such types. Attempting to use this method on such types results
+ /// in a compile-time assertion error; e.g.:
+ ///
+ /// ```compile_fail,E0080
+ /// use zerocopy::*;
+ /// # use zerocopy_derive::*;
+ ///
+ /// #[derive(FromBytes, Immutable, KnownLayout)]
+ /// #[repr(C)]
+ /// struct ZSTy {
+ /// leading_sized: u16,
+ /// trailing_dst: [()],
+ /// }
+ ///
+ /// let _ = ZSTy::ref_from_suffix(0u16.as_bytes()); // ⚠ Compile Error!
+ /// ```
+ ///
+ /// [`ref_from_suffix_with_elems`]: FromBytes::ref_from_suffix_with_elems
+ ///
+ /// # Examples
+ ///
+ /// ```
+ /// use zerocopy::FromBytes;
+ /// # use zerocopy_derive::*;
+ ///
+ /// #[derive(FromBytes, Immutable, KnownLayout)]
+ /// #[repr(C)]
+ /// struct PacketTrailer {
+ /// frame_check_sequence: [u8; 4],
+ /// }
+ ///
+ /// // These are more bytes than are needed to encode a `PacketTrailer`.
+ /// let bytes = &[0, 1, 2, 3, 4, 5, 6, 7, 8, 9][..];
+ ///
+ /// let (prefix, trailer) = PacketTrailer::ref_from_suffix(bytes).unwrap();
+ ///
+ /// assert_eq!(prefix, &[0, 1, 2, 3, 4, 5][..]);
+ /// assert_eq!(trailer.frame_check_sequence, [6, 7, 8, 9]);
+ /// ```
+ ///
+ #[doc = codegen_section!(
+ header = "h5",
+ bench = "ref_from_suffix",
+ format = "coco",
+ arity = 3,
+ [
+ open
+ @index 1
+ @title "Sized"
+ @variant "static_size"
+ ],
+ [
+ @index 2
+ @title "Unsized"
+ @variant "dynamic_size"
+ ],
+ [
+ @index 3
+ @title "Dynamically Padded"
+ @variant "dynamic_padding"
+ ]
+ )]
+ #[must_use = "has no side effects"]
+ #[cfg_attr(zerocopy_inline_always, inline(always))]
+ #[cfg_attr(not(zerocopy_inline_always), inline)]
+ fn ref_from_suffix(source: &[u8]) -> Result<(&[u8], &Self), CastError<&[u8], Self>>
+ where
+ Self: Immutable + KnownLayout,
+ {
+ static_assert_dst_is_not_zst!(Self);
+ ref_from_prefix_suffix(source, None, CastType::Suffix).map(swap)
+ }
+
+ /// Interprets the given `source` as a `&mut Self`.
+ ///
+ /// This method attempts to return a reference to `source` interpreted as a
+ /// `Self`. If the length of `source` is not a [valid size of
+ /// `Self`][valid-size], or if `source` is not appropriately aligned, this
+ /// returns `Err`. If [`Self: Unaligned`][self-unaligned], you can
+ /// [infallibly discard the alignment error][size-error-from].
+ ///
+ /// `Self` may be a sized type, a slice, or a [slice DST][slice-dst].
+ ///
+ /// [valid-size]: crate::KnownLayout#what-is-a-valid-size
+ /// [self-unaligned]: Unaligned
+ /// [size-error-from]: error/struct.SizeError.html#method.from-1
+ /// [slice-dst]: KnownLayout#dynamically-sized-types
+ ///
+ /// # Compile-Time Assertions
+ ///
+ /// This method cannot yet be used on unsized types whose dynamically-sized
+ /// component is zero-sized. See [`mut_from_prefix_with_elems`], which does
+ /// support such types. Attempting to use this method on such types results
+ /// in a compile-time assertion error; e.g.:
+ ///
+ /// ```compile_fail,E0080
+ /// use zerocopy::*;
+ /// # use zerocopy_derive::*;
+ ///
+ /// #[derive(FromBytes, Immutable, IntoBytes, KnownLayout)]
+ /// #[repr(C, packed)]
+ /// struct ZSTy {
+ /// leading_sized: [u8; 2],
+ /// trailing_dst: [()],
+ /// }
+ ///
+ /// let mut source = [85, 85];
+ /// let _ = ZSTy::mut_from_bytes(&mut source[..]); // ⚠ Compile Error!
+ /// ```
+ ///
+ /// [`mut_from_prefix_with_elems`]: FromBytes::mut_from_prefix_with_elems
+ ///
+ /// # Examples
+ ///
+ /// ```
+ /// use zerocopy::FromBytes;
+ /// # use zerocopy_derive::*;
+ ///
+ /// #[derive(FromBytes, IntoBytes, KnownLayout, Immutable)]
+ /// #[repr(C)]
+ /// struct PacketHeader {
+ /// src_port: [u8; 2],
+ /// dst_port: [u8; 2],
+ /// length: [u8; 2],
+ /// checksum: [u8; 2],
+ /// }
+ ///
+ /// // These bytes encode a `PacketHeader`.
+ /// let bytes = &mut [0, 1, 2, 3, 4, 5, 6, 7][..];
+ ///
+ /// let header = PacketHeader::mut_from_bytes(bytes).unwrap();
+ ///
+ /// assert_eq!(header.src_port, [0, 1]);
+ /// assert_eq!(header.dst_port, [2, 3]);
+ /// assert_eq!(header.length, [4, 5]);
+ /// assert_eq!(header.checksum, [6, 7]);
+ ///
+ /// header.checksum = [0, 0];
+ ///
+ /// assert_eq!(bytes, [0, 1, 2, 3, 4, 5, 0, 0]);
+ ///
+ /// ```
+ ///
+ #[doc = codegen_header!("h5", "mut_from_bytes")]
+ ///
+ /// See [`FromBytes::ref_from_bytes`](#method.ref_from_bytes.codegen).
+ #[must_use = "has no side effects"]
+ #[cfg_attr(zerocopy_inline_always, inline(always))]
+ #[cfg_attr(not(zerocopy_inline_always), inline)]
+ fn mut_from_bytes(source: &mut [u8]) -> Result<&mut Self, CastError<&mut [u8], Self>>
+ where
+ Self: IntoBytes + KnownLayout,
+ {
+ static_assert_dst_is_not_zst!(Self);
+ match Ptr::from_mut(source).try_cast_into_no_leftover::<_, BecauseExclusive>(None) {
+ Ok(ptr) => Ok(ptr.recall_validity::<_, (_, (_, _))>().as_mut()),
+ Err(err) => Err(err.map_src(|src| src.as_mut())),
+ }
+ }
+
+ /// Interprets the prefix of the given `source` as a `&mut Self` without
+ /// copying.
+ ///
+ /// This method computes the [largest possible size of `Self`][valid-size]
+ /// that can fit in the leading bytes of `source`, then attempts to return
+ /// both a reference to those bytes interpreted as a `Self`, and a reference
+ /// to the remaining bytes. If there are insufficient bytes, or if `source`
+ /// is not appropriately aligned, this returns `Err`. If [`Self:
+ /// Unaligned`][self-unaligned], you can [infallibly discard the alignment
+ /// error][size-error-from].
+ ///
+ /// `Self` may be a sized type, a slice, or a [slice DST][slice-dst].
+ ///
+ /// [valid-size]: crate::KnownLayout#what-is-a-valid-size
+ /// [self-unaligned]: Unaligned
+ /// [size-error-from]: error/struct.SizeError.html#method.from-1
+ /// [slice-dst]: KnownLayout#dynamically-sized-types
+ ///
+ /// # Compile-Time Assertions
+ ///
+ /// This method cannot yet be used on unsized types whose dynamically-sized
+ /// component is zero-sized. See [`mut_from_suffix_with_elems`], which does
+ /// support such types. Attempting to use this method on such types results
+ /// in a compile-time assertion error; e.g.:
+ ///
+ /// ```compile_fail,E0080
+ /// use zerocopy::*;
+ /// # use zerocopy_derive::*;
+ ///
+ /// #[derive(FromBytes, Immutable, IntoBytes, KnownLayout)]
+ /// #[repr(C, packed)]
+ /// struct ZSTy {
+ /// leading_sized: [u8; 2],
+ /// trailing_dst: [()],
+ /// }
+ ///
+ /// let mut source = [85, 85];
+ /// let _ = ZSTy::mut_from_prefix(&mut source[..]); // ⚠ Compile Error!
+ /// ```
+ ///
+ /// [`mut_from_suffix_with_elems`]: FromBytes::mut_from_suffix_with_elems
+ ///
+ /// # Examples
+ ///
+ /// ```
+ /// use zerocopy::FromBytes;
+ /// # use zerocopy_derive::*;
+ ///
+ /// #[derive(FromBytes, IntoBytes, KnownLayout, Immutable)]
+ /// #[repr(C)]
+ /// struct PacketHeader {
+ /// src_port: [u8; 2],
+ /// dst_port: [u8; 2],
+ /// length: [u8; 2],
+ /// checksum: [u8; 2],
+ /// }
+ ///
+ /// // These are more bytes than are needed to encode a `PacketHeader`.
+ /// let bytes = &mut [0, 1, 2, 3, 4, 5, 6, 7, 8, 9][..];
+ ///
+ /// let (header, body) = PacketHeader::mut_from_prefix(bytes).unwrap();
+ ///
+ /// assert_eq!(header.src_port, [0, 1]);
+ /// assert_eq!(header.dst_port, [2, 3]);
+ /// assert_eq!(header.length, [4, 5]);
+ /// assert_eq!(header.checksum, [6, 7]);
+ /// assert_eq!(body, &[8, 9][..]);
+ ///
+ /// header.checksum = [0, 0];
+ /// body.fill(1);
+ ///
+ /// assert_eq!(bytes, [0, 1, 2, 3, 4, 5, 0, 0, 1, 1]);
+ /// ```
+ ///
+ #[doc = codegen_header!("h5", "mut_from_prefix")]
+ ///
+ /// See [`FromBytes::ref_from_prefix`](#method.ref_from_prefix.codegen).
+ #[must_use = "has no side effects"]
+ #[cfg_attr(zerocopy_inline_always, inline(always))]
+ #[cfg_attr(not(zerocopy_inline_always), inline)]
+ fn mut_from_prefix(
+ source: &mut [u8],
+ ) -> Result<(&mut Self, &mut [u8]), CastError<&mut [u8], Self>>
+ where
+ Self: IntoBytes + KnownLayout,
+ {
+ static_assert_dst_is_not_zst!(Self);
+ mut_from_prefix_suffix(source, None, CastType::Prefix)
+ }
+
+ /// Interprets the suffix of the given `source` as a `&mut Self` without
+ /// copying.
+ ///
+ /// This method computes the [largest possible size of `Self`][valid-size]
+ /// that can fit in the trailing bytes of `source`, then attempts to return
+ /// both a reference to those bytes interpreted as a `Self`, and a reference
+ /// to the preceding bytes. If there are insufficient bytes, or if that
+ /// suffix of `source` is not appropriately aligned, this returns `Err`. If
+ /// [`Self: Unaligned`][self-unaligned], you can [infallibly discard the
+ /// alignment error][size-error-from].
+ ///
+ /// `Self` may be a sized type, a slice, or a [slice DST][slice-dst].
+ ///
+ /// [valid-size]: crate::KnownLayout#what-is-a-valid-size
+ /// [self-unaligned]: Unaligned
+ /// [size-error-from]: error/struct.SizeError.html#method.from-1
+ /// [slice-dst]: KnownLayout#dynamically-sized-types
+ ///
+ /// # Compile-Time Assertions
+ ///
+ /// This method cannot yet be used on unsized types whose dynamically-sized
+ /// component is zero-sized. Attempting to use this method on such types
+ /// results in a compile-time assertion error; e.g.:
+ ///
+ /// ```compile_fail,E0080
+ /// use zerocopy::*;
+ /// # use zerocopy_derive::*;
+ ///
+ /// #[derive(FromBytes, Immutable, IntoBytes, KnownLayout)]
+ /// #[repr(C, packed)]
+ /// struct ZSTy {
+ /// leading_sized: [u8; 2],
+ /// trailing_dst: [()],
+ /// }
+ ///
+ /// let mut source = [85, 85];
+ /// let _ = ZSTy::mut_from_suffix(&mut source[..]); // ⚠ Compile Error!
+ /// ```
+ ///
+ /// # Examples
+ ///
+ /// ```
+ /// use zerocopy::FromBytes;
+ /// # use zerocopy_derive::*;
+ ///
+ /// #[derive(FromBytes, IntoBytes, KnownLayout, Immutable)]
+ /// #[repr(C)]
+ /// struct PacketTrailer {
+ /// frame_check_sequence: [u8; 4],
+ /// }
+ ///
+ /// // These are more bytes than are needed to encode a `PacketTrailer`.
+ /// let bytes = &mut [0, 1, 2, 3, 4, 5, 6, 7, 8, 9][..];
+ ///
+ /// let (prefix, trailer) = PacketTrailer::mut_from_suffix(bytes).unwrap();
+ ///
+ /// assert_eq!(prefix, &[0u8, 1, 2, 3, 4, 5][..]);
+ /// assert_eq!(trailer.frame_check_sequence, [6, 7, 8, 9]);
+ ///
+ /// prefix.fill(0);
+ /// trailer.frame_check_sequence.fill(1);
+ ///
+ /// assert_eq!(bytes, [0, 0, 0, 0, 0, 0, 1, 1, 1, 1]);
+ /// ```
+ ///
+ #[doc = codegen_header!("h5", "mut_from_suffix")]
+ ///
+ /// See [`FromBytes::ref_from_suffix`](#method.ref_from_suffix.codegen).
+ #[must_use = "has no side effects"]
+ #[cfg_attr(zerocopy_inline_always, inline(always))]
+ #[cfg_attr(not(zerocopy_inline_always), inline)]
+ fn mut_from_suffix(
+ source: &mut [u8],
+ ) -> Result<(&mut [u8], &mut Self), CastError<&mut [u8], Self>>
+ where
+ Self: IntoBytes + KnownLayout,
+ {
+ static_assert_dst_is_not_zst!(Self);
+ mut_from_prefix_suffix(source, None, CastType::Suffix).map(swap)
+ }
+
+ /// Interprets the given `source` as a `&Self` with a DST length equal to
+ /// `count`.
+ ///
+ /// This method attempts to return a reference to `source` interpreted as a
+ /// `Self` with `count` trailing elements. If the length of `source` is not
+ /// equal to the size of `Self` with `count` elements, or if `source` is not
+ /// appropriately aligned, this returns `Err`. If [`Self:
+ /// Unaligned`][self-unaligned], you can [infallibly discard the alignment
+ /// error][size-error-from].
+ ///
+ /// [self-unaligned]: Unaligned
+ /// [size-error-from]: error/struct.SizeError.html#method.from-1
+ ///
+ /// # Examples
+ ///
+ /// ```
+ /// use zerocopy::FromBytes;
+ /// # use zerocopy_derive::*;
+ ///
+ /// # #[derive(Debug, PartialEq, Eq)]
+ /// #[derive(FromBytes, Immutable)]
+ /// #[repr(C)]
+ /// struct Pixel {
+ /// r: u8,
+ /// g: u8,
+ /// b: u8,
+ /// a: u8,
+ /// }
+ ///
+ /// let bytes = &[0, 1, 2, 3, 4, 5, 6, 7][..];
+ ///
+ /// let pixels = <[Pixel]>::ref_from_bytes_with_elems(bytes, 2).unwrap();
+ ///
+ /// assert_eq!(pixels, &[
+ /// Pixel { r: 0, g: 1, b: 2, a: 3 },
+ /// Pixel { r: 4, g: 5, b: 6, a: 7 },
+ /// ]);
+ ///
+ /// ```
+ ///
+ /// Since an explicit `count` is provided, this method supports types with
+ /// zero-sized trailing slice elements. Methods such as [`ref_from_bytes`]
+ /// which do not take an explicit count do not support such types.
+ ///
+ /// ```
+ /// use zerocopy::*;
+ /// # use zerocopy_derive::*;
+ ///
+ /// #[derive(FromBytes, Immutable, KnownLayout)]
+ /// #[repr(C)]
+ /// struct ZSTy {
+ /// leading_sized: [u8; 2],
+ /// trailing_dst: [()],
+ /// }
+ ///
+ /// let src = &[85, 85][..];
+ /// let zsty = ZSTy::ref_from_bytes_with_elems(src, 42).unwrap();
+ /// assert_eq!(zsty.trailing_dst.len(), 42);
+ /// ```
+ ///
+ /// [`ref_from_bytes`]: FromBytes::ref_from_bytes
+ ///
+ #[doc = codegen_section!(
+ header = "h5",
+ bench = "ref_from_bytes_with_elems",
+ format = "coco",
+ arity = 2,
+ [
+ open
+ @index 1
+ @title "Unsized"
+ @variant "dynamic_size"
+ ],
+ [
+ @index 2
+ @title "Dynamically Padded"
+ @variant "dynamic_padding"
+ ]
+ )]
+ #[must_use = "has no side effects"]
+ #[cfg_attr(zerocopy_inline_always, inline(always))]
+ #[cfg_attr(not(zerocopy_inline_always), inline)]
+ fn ref_from_bytes_with_elems(
+ source: &[u8],
+ count: usize,
+ ) -> Result<&Self, CastError<&[u8], Self>>
+ where
+ Self: KnownLayout<PointerMetadata = usize> + Immutable,
+ {
+ let source = Ptr::from_ref(source);
+ let maybe_slf = source.try_cast_into_no_leftover::<_, BecauseImmutable>(Some(count));
+ match maybe_slf {
+ Ok(slf) => Ok(slf.recall_validity().as_ref()),
+ Err(err) => Err(err.map_src(|s| s.as_ref())),
+ }
+ }
+
+ /// Interprets the prefix of the given `source` as a DST `&Self` with length
+ /// equal to `count`.
+ ///
+ /// This method attempts to return a reference to the prefix of `source`
+ /// interpreted as a `Self` with `count` trailing elements, and a reference
+ /// to the remaining bytes. If there are insufficient bytes, or if `source`
+ /// is not appropriately aligned, this returns `Err`. If [`Self:
+ /// Unaligned`][self-unaligned], you can [infallibly discard the alignment
+ /// error][size-error-from].
+ ///
+ /// [self-unaligned]: Unaligned
+ /// [size-error-from]: error/struct.SizeError.html#method.from-1
+ ///
+ /// # Examples
+ ///
+ /// ```
+ /// use zerocopy::FromBytes;
+ /// # use zerocopy_derive::*;
+ ///
+ /// # #[derive(Debug, PartialEq, Eq)]
+ /// #[derive(FromBytes, Immutable)]
+ /// #[repr(C)]
+ /// struct Pixel {
+ /// r: u8,
+ /// g: u8,
+ /// b: u8,
+ /// a: u8,
+ /// }
+ ///
+ /// // These are more bytes than are needed to encode two `Pixel`s.
+ /// let bytes = &[0, 1, 2, 3, 4, 5, 6, 7, 8, 9][..];
+ ///
+ /// let (pixels, suffix) = <[Pixel]>::ref_from_prefix_with_elems(bytes, 2).unwrap();
+ ///
+ /// assert_eq!(pixels, &[
+ /// Pixel { r: 0, g: 1, b: 2, a: 3 },
+ /// Pixel { r: 4, g: 5, b: 6, a: 7 },
+ /// ]);
+ ///
+ /// assert_eq!(suffix, &[8, 9]);
+ /// ```
+ ///
+ /// Since an explicit `count` is provided, this method supports types with
+ /// zero-sized trailing slice elements. Methods such as [`ref_from_prefix`]
+ /// which do not take an explicit count do not support such types.
+ ///
+ /// ```
+ /// use zerocopy::*;
+ /// # use zerocopy_derive::*;
+ ///
+ /// #[derive(FromBytes, Immutable, KnownLayout)]
+ /// #[repr(C)]
+ /// struct ZSTy {
+ /// leading_sized: [u8; 2],
+ /// trailing_dst: [()],
+ /// }
+ ///
+ /// let src = &[85, 85][..];
+ /// let (zsty, _) = ZSTy::ref_from_prefix_with_elems(src, 42).unwrap();
+ /// assert_eq!(zsty.trailing_dst.len(), 42);
+ /// ```
+ ///
+ /// [`ref_from_prefix`]: FromBytes::ref_from_prefix
+ ///
+ #[doc = codegen_section!(
+ header = "h5",
+ bench = "ref_from_prefix_with_elems",
+ format = "coco",
+ arity = 2,
+ [
+ open
+ @index 1
+ @title "Unsized"
+ @variant "dynamic_size"
+ ],
+ [
+ @index 2
+ @title "Dynamically Padded"
+ @variant "dynamic_padding"
+ ]
+ )]
+ #[must_use = "has no side effects"]
+ #[cfg_attr(zerocopy_inline_always, inline(always))]
+ #[cfg_attr(not(zerocopy_inline_always), inline)]
+ fn ref_from_prefix_with_elems(
+ source: &[u8],
+ count: usize,
+ ) -> Result<(&Self, &[u8]), CastError<&[u8], Self>>
+ where
+ Self: KnownLayout<PointerMetadata = usize> + Immutable,
+ {
+ ref_from_prefix_suffix(source, Some(count), CastType::Prefix)
+ }
+
+ /// Interprets the suffix of the given `source` as a DST `&Self` with length
+ /// equal to `count`.
+ ///
+ /// This method attempts to return a reference to the suffix of `source`
+ /// interpreted as a `Self` with `count` trailing elements, and a reference
+ /// to the preceding bytes. If there are insufficient bytes, or if that
+ /// suffix of `source` is not appropriately aligned, this returns `Err`. If
+ /// [`Self: Unaligned`][self-unaligned], you can [infallibly discard the
+ /// alignment error][size-error-from].
+ ///
+ /// [self-unaligned]: Unaligned
+ /// [size-error-from]: error/struct.SizeError.html#method.from-1
+ ///
+ /// # Examples
+ ///
+ /// ```
+ /// use zerocopy::FromBytes;
+ /// # use zerocopy_derive::*;
+ ///
+ /// # #[derive(Debug, PartialEq, Eq)]
+ /// #[derive(FromBytes, Immutable)]
+ /// #[repr(C)]
+ /// struct Pixel {
+ /// r: u8,
+ /// g: u8,
+ /// b: u8,
+ /// a: u8,
+ /// }
+ ///
+ /// // These are more bytes than are needed to encode two `Pixel`s.
+ /// let bytes = &[0, 1, 2, 3, 4, 5, 6, 7, 8, 9][..];
+ ///
+ /// let (prefix, pixels) = <[Pixel]>::ref_from_suffix_with_elems(bytes, 2).unwrap();
+ ///
+ /// assert_eq!(prefix, &[0, 1]);
+ ///
+ /// assert_eq!(pixels, &[
+ /// Pixel { r: 2, g: 3, b: 4, a: 5 },
+ /// Pixel { r: 6, g: 7, b: 8, a: 9 },
+ /// ]);
+ /// ```
+ ///
+ /// Since an explicit `count` is provided, this method supports types with
+ /// zero-sized trailing slice elements. Methods such as [`ref_from_suffix`]
+ /// which do not take an explicit count do not support such types.
+ ///
+ /// ```
+ /// use zerocopy::*;
+ /// # use zerocopy_derive::*;
+ ///
+ /// #[derive(FromBytes, Immutable, KnownLayout)]
+ /// #[repr(C)]
+ /// struct ZSTy {
+ /// leading_sized: [u8; 2],
+ /// trailing_dst: [()],
+ /// }
+ ///
+ /// let src = &[85, 85][..];
+ /// let (_, zsty) = ZSTy::ref_from_suffix_with_elems(src, 42).unwrap();
+ /// assert_eq!(zsty.trailing_dst.len(), 42);
+ /// ```
+ ///
+ /// [`ref_from_suffix`]: FromBytes::ref_from_suffix
+ ///
+ #[doc = codegen_section!(
+ header = "h5",
+ bench = "ref_from_suffix_with_elems",
+ format = "coco",
+ arity = 2,
+ [
+ open
+ @index 1
+ @title "Unsized"
+ @variant "dynamic_size"
+ ],
+ [
+ @index 2
+ @title "Dynamically Padded"
+ @variant "dynamic_padding"
+ ]
+ )]
+ #[must_use = "has no side effects"]
+ #[cfg_attr(zerocopy_inline_always, inline(always))]
+ #[cfg_attr(not(zerocopy_inline_always), inline)]
+ fn ref_from_suffix_with_elems(
+ source: &[u8],
+ count: usize,
+ ) -> Result<(&[u8], &Self), CastError<&[u8], Self>>
+ where
+ Self: KnownLayout<PointerMetadata = usize> + Immutable,
+ {
+ ref_from_prefix_suffix(source, Some(count), CastType::Suffix).map(swap)
+ }
+
+ /// Interprets the given `source` as a `&mut Self` with a DST length equal
+ /// to `count`.
+ ///
+ /// This method attempts to return a reference to `source` interpreted as a
+ /// `Self` with `count` trailing elements. If the length of `source` is not
+ /// equal to the size of `Self` with `count` elements, or if `source` is not
+ /// appropriately aligned, this returns `Err`. If [`Self:
+ /// Unaligned`][self-unaligned], you can [infallibly discard the alignment
+ /// error][size-error-from].
+ ///
+ /// [self-unaligned]: Unaligned
+ /// [size-error-from]: error/struct.SizeError.html#method.from-1
+ ///
+ /// # Examples
+ ///
+ /// ```
+ /// use zerocopy::FromBytes;
+ /// # use zerocopy_derive::*;
+ ///
+ /// # #[derive(Debug, PartialEq, Eq)]
+ /// #[derive(KnownLayout, FromBytes, IntoBytes, Immutable)]
+ /// #[repr(C)]
+ /// struct Pixel {
+ /// r: u8,
+ /// g: u8,
+ /// b: u8,
+ /// a: u8,
+ /// }
+ ///
+ /// let bytes = &mut [0, 1, 2, 3, 4, 5, 6, 7][..];
+ ///
+ /// let pixels = <[Pixel]>::mut_from_bytes_with_elems(bytes, 2).unwrap();
+ ///
+ /// assert_eq!(pixels, &[
+ /// Pixel { r: 0, g: 1, b: 2, a: 3 },
+ /// Pixel { r: 4, g: 5, b: 6, a: 7 },
+ /// ]);
+ ///
+ /// pixels[1] = Pixel { r: 0, g: 0, b: 0, a: 0 };
+ ///
+ /// assert_eq!(bytes, [0, 1, 2, 3, 0, 0, 0, 0]);
+ /// ```
+ ///
+ /// Since an explicit `count` is provided, this method supports types with
+ /// zero-sized trailing slice elements. Methods such as [`mut_from_bytes`]
+ /// which do not take an explicit count do not support such types.
+ ///
+ /// ```
+ /// use zerocopy::*;
+ /// # use zerocopy_derive::*;
+ ///
+ /// #[derive(FromBytes, IntoBytes, Immutable, KnownLayout)]
+ /// #[repr(C, packed)]
+ /// struct ZSTy {
+ /// leading_sized: [u8; 2],
+ /// trailing_dst: [()],
+ /// }
+ ///
+ /// let src = &mut [85, 85][..];
+ /// let zsty = ZSTy::mut_from_bytes_with_elems(src, 42).unwrap();
+ /// assert_eq!(zsty.trailing_dst.len(), 42);
+ /// ```
+ ///
+ /// [`mut_from_bytes`]: FromBytes::mut_from_bytes
+ ///
+ #[doc = codegen_header!("h5", "mut_from_bytes_with_elems")]
+ ///
+ /// See [`TryFromBytes::ref_from_bytes_with_elems`](#method.ref_from_bytes_with_elems.codegen).
+ #[must_use = "has no side effects"]
+ #[cfg_attr(zerocopy_inline_always, inline(always))]
+ #[cfg_attr(not(zerocopy_inline_always), inline)]
+ fn mut_from_bytes_with_elems(
+ source: &mut [u8],
+ count: usize,
+ ) -> Result<&mut Self, CastError<&mut [u8], Self>>
+ where
+ Self: IntoBytes + KnownLayout<PointerMetadata = usize> + Immutable,
+ {
+ let source = Ptr::from_mut(source);
+ let maybe_slf = source.try_cast_into_no_leftover::<_, BecauseImmutable>(Some(count));
+ match maybe_slf {
+ Ok(slf) => Ok(slf.recall_validity::<_, (_, (_, BecauseExclusive))>().as_mut()),
+ Err(err) => Err(err.map_src(|s| s.as_mut())),
+ }
+ }
+
+ /// Interprets the prefix of the given `source` as a `&mut Self` with DST
+ /// length equal to `count`.
+ ///
+ /// This method attempts to return a reference to the prefix of `source`
+ /// interpreted as a `Self` with `count` trailing elements, and a reference
+ /// to the preceding bytes. If there are insufficient bytes, or if `source`
+ /// is not appropriately aligned, this returns `Err`. If [`Self:
+ /// Unaligned`][self-unaligned], you can [infallibly discard the alignment
+ /// error][size-error-from].
+ ///
+ /// [self-unaligned]: Unaligned
+ /// [size-error-from]: error/struct.SizeError.html#method.from-1
+ ///
+ /// # Examples
+ ///
+ /// ```
+ /// use zerocopy::FromBytes;
+ /// # use zerocopy_derive::*;
+ ///
+ /// # #[derive(Debug, PartialEq, Eq)]
+ /// #[derive(KnownLayout, FromBytes, IntoBytes, Immutable)]
+ /// #[repr(C)]
+ /// struct Pixel {
+ /// r: u8,
+ /// g: u8,
+ /// b: u8,
+ /// a: u8,
+ /// }
+ ///
+ /// // These are more bytes than are needed to encode two `Pixel`s.
+ /// let bytes = &mut [0, 1, 2, 3, 4, 5, 6, 7, 8, 9][..];
+ ///
+ /// let (pixels, suffix) = <[Pixel]>::mut_from_prefix_with_elems(bytes, 2).unwrap();
+ ///
+ /// assert_eq!(pixels, &[
+ /// Pixel { r: 0, g: 1, b: 2, a: 3 },
+ /// Pixel { r: 4, g: 5, b: 6, a: 7 },
+ /// ]);
+ ///
+ /// assert_eq!(suffix, &[8, 9]);
+ ///
+ /// pixels[1] = Pixel { r: 0, g: 0, b: 0, a: 0 };
+ /// suffix.fill(1);
+ ///
+ /// assert_eq!(bytes, [0, 1, 2, 3, 0, 0, 0, 0, 1, 1]);
+ /// ```
+ ///
+ /// Since an explicit `count` is provided, this method supports types with
+ /// zero-sized trailing slice elements. Methods such as [`mut_from_prefix`]
+ /// which do not take an explicit count do not support such types.
+ ///
+ /// ```
+ /// use zerocopy::*;
+ /// # use zerocopy_derive::*;
+ ///
+ /// #[derive(FromBytes, IntoBytes, Immutable, KnownLayout)]
+ /// #[repr(C, packed)]
+ /// struct ZSTy {
+ /// leading_sized: [u8; 2],
+ /// trailing_dst: [()],
+ /// }
+ ///
+ /// let src = &mut [85, 85][..];
+ /// let (zsty, _) = ZSTy::mut_from_prefix_with_elems(src, 42).unwrap();
+ /// assert_eq!(zsty.trailing_dst.len(), 42);
+ /// ```
+ ///
+ /// [`mut_from_prefix`]: FromBytes::mut_from_prefix
+ ///
+ #[doc = codegen_header!("h5", "mut_from_prefix_with_elems")]
+ ///
+ /// See [`TryFromBytes::ref_from_prefix_with_elems`](#method.ref_from_prefix_with_elems.codegen).
+ #[must_use = "has no side effects"]
+ #[cfg_attr(zerocopy_inline_always, inline(always))]
+ #[cfg_attr(not(zerocopy_inline_always), inline)]
+ fn mut_from_prefix_with_elems(
+ source: &mut [u8],
+ count: usize,
+ ) -> Result<(&mut Self, &mut [u8]), CastError<&mut [u8], Self>>
+ where
+ Self: IntoBytes + KnownLayout<PointerMetadata = usize>,
+ {
+ mut_from_prefix_suffix(source, Some(count), CastType::Prefix)
+ }
+
+ /// Interprets the suffix of the given `source` as a `&mut Self` with DST
+ /// length equal to `count`.
+ ///
+ /// This method attempts to return a reference to the suffix of `source`
+ /// interpreted as a `Self` with `count` trailing elements, and a reference
+ /// to the remaining bytes. If there are insufficient bytes, or if that
+ /// suffix of `source` is not appropriately aligned, this returns `Err`. If
+ /// [`Self: Unaligned`][self-unaligned], you can [infallibly discard the
+ /// alignment error][size-error-from].
+ ///
+ /// [self-unaligned]: Unaligned
+ /// [size-error-from]: error/struct.SizeError.html#method.from-1
+ ///
+ /// # Examples
+ ///
+ /// ```
+ /// use zerocopy::FromBytes;
+ /// # use zerocopy_derive::*;
+ ///
+ /// # #[derive(Debug, PartialEq, Eq)]
+ /// #[derive(FromBytes, IntoBytes, Immutable)]
+ /// #[repr(C)]
+ /// struct Pixel {
+ /// r: u8,
+ /// g: u8,
+ /// b: u8,
+ /// a: u8,
+ /// }
+ ///
+ /// // These are more bytes than are needed to encode two `Pixel`s.
+ /// let bytes = &mut [0, 1, 2, 3, 4, 5, 6, 7, 8, 9][..];
+ ///
+ /// let (prefix, pixels) = <[Pixel]>::mut_from_suffix_with_elems(bytes, 2).unwrap();
+ ///
+ /// assert_eq!(prefix, &[0, 1]);
+ ///
+ /// assert_eq!(pixels, &[
+ /// Pixel { r: 2, g: 3, b: 4, a: 5 },
+ /// Pixel { r: 6, g: 7, b: 8, a: 9 },
+ /// ]);
+ ///
+ /// prefix.fill(9);
+ /// pixels[1] = Pixel { r: 0, g: 0, b: 0, a: 0 };
+ ///
+ /// assert_eq!(bytes, [9, 9, 2, 3, 4, 5, 0, 0, 0, 0]);
+ /// ```
+ ///
+ /// Since an explicit `count` is provided, this method supports types with
+ /// zero-sized trailing slice elements. Methods such as [`mut_from_suffix`]
+ /// which do not take an explicit count do not support such types.
+ ///
+ /// ```
+ /// use zerocopy::*;
+ /// # use zerocopy_derive::*;
+ ///
+ /// #[derive(FromBytes, IntoBytes, Immutable, KnownLayout)]
+ /// #[repr(C, packed)]
+ /// struct ZSTy {
+ /// leading_sized: [u8; 2],
+ /// trailing_dst: [()],
+ /// }
+ ///
+ /// let src = &mut [85, 85][..];
+ /// let (_, zsty) = ZSTy::mut_from_suffix_with_elems(src, 42).unwrap();
+ /// assert_eq!(zsty.trailing_dst.len(), 42);
+ /// ```
+ ///
+ /// [`mut_from_suffix`]: FromBytes::mut_from_suffix
+ ///
+ #[doc = codegen_header!("h5", "mut_from_suffix_with_elems")]
+ ///
+ /// See [`TryFromBytes::ref_from_suffix_with_elems`](#method.ref_from_suffix_with_elems.codegen).
+ #[must_use = "has no side effects"]
+ #[cfg_attr(zerocopy_inline_always, inline(always))]
+ #[cfg_attr(not(zerocopy_inline_always), inline)]
+ fn mut_from_suffix_with_elems(
+ source: &mut [u8],
+ count: usize,
+ ) -> Result<(&mut [u8], &mut Self), CastError<&mut [u8], Self>>
+ where
+ Self: IntoBytes + KnownLayout<PointerMetadata = usize>,
+ {
+ mut_from_prefix_suffix(source, Some(count), CastType::Suffix).map(swap)
+ }
+
+ /// Reads a copy of `Self` from the given `source`.
+ ///
+ /// If `source.len() != size_of::<Self>()`, `read_from_bytes` returns `Err`.
+ ///
+ /// # Examples
+ ///
+ /// ```
+ /// use zerocopy::FromBytes;
+ /// # use zerocopy_derive::*;
+ ///
+ /// #[derive(FromBytes)]
+ /// #[repr(C)]
+ /// struct PacketHeader {
+ /// src_port: [u8; 2],
+ /// dst_port: [u8; 2],
+ /// length: [u8; 2],
+ /// checksum: [u8; 2],
+ /// }
+ ///
+ /// // These bytes encode a `PacketHeader`.
+ /// let bytes = &[0, 1, 2, 3, 4, 5, 6, 7][..];
+ ///
+ /// let header = PacketHeader::read_from_bytes(bytes).unwrap();
+ ///
+ /// assert_eq!(header.src_port, [0, 1]);
+ /// assert_eq!(header.dst_port, [2, 3]);
+ /// assert_eq!(header.length, [4, 5]);
+ /// assert_eq!(header.checksum, [6, 7]);
+ /// ```
+ ///
+ #[doc = codegen_section!(
+ header = "h5",
+ bench = "read_from_bytes",
+ format = "coco_static_size",
+ )]
+ #[must_use = "has no side effects"]
+ #[cfg_attr(zerocopy_inline_always, inline(always))]
+ #[cfg_attr(not(zerocopy_inline_always), inline)]
+ fn read_from_bytes(source: &[u8]) -> Result<Self, SizeError<&[u8], Self>>
+ where
+ Self: Sized,
+ {
+ match Ref::<_, Unalign<Self>>::sized_from(source) {
+ Ok(r) => Ok(Ref::read(&r).into_inner()),
+ Err(CastError::Size(e)) => Err(e.with_dst()),
+ Err(CastError::Alignment(_)) => {
+ // SAFETY: `Unalign<Self>` is trivially aligned, so
+ // `Ref::sized_from` cannot fail due to unmet alignment
+ // requirements.
+ unsafe { core::hint::unreachable_unchecked() }
+ }
+ Err(CastError::Validity(i)) => match i {},
+ }
+ }
+
+ /// Reads a copy of `Self` from the prefix of the given `source`.
+ ///
+ /// This attempts to read a `Self` from the first `size_of::<Self>()` bytes
+ /// of `source`, returning that `Self` and any remaining bytes. If
+ /// `source.len() < size_of::<Self>()`, it returns `Err`.
+ ///
+ /// # Examples
+ ///
+ /// ```
+ /// use zerocopy::FromBytes;
+ /// # use zerocopy_derive::*;
+ ///
+ /// #[derive(FromBytes)]
+ /// #[repr(C)]
+ /// struct PacketHeader {
+ /// src_port: [u8; 2],
+ /// dst_port: [u8; 2],
+ /// length: [u8; 2],
+ /// checksum: [u8; 2],
+ /// }
+ ///
+ /// // These are more bytes than are needed to encode a `PacketHeader`.
+ /// let bytes = &[0, 1, 2, 3, 4, 5, 6, 7, 8, 9][..];
+ ///
+ /// let (header, body) = PacketHeader::read_from_prefix(bytes).unwrap();
+ ///
+ /// assert_eq!(header.src_port, [0, 1]);
+ /// assert_eq!(header.dst_port, [2, 3]);
+ /// assert_eq!(header.length, [4, 5]);
+ /// assert_eq!(header.checksum, [6, 7]);
+ /// assert_eq!(body, [8, 9]);
+ /// ```
+ ///
+ #[doc = codegen_section!(
+ header = "h5",
+ bench = "read_from_prefix",
+ format = "coco_static_size",
+ )]
+ #[must_use = "has no side effects"]
+ #[cfg_attr(zerocopy_inline_always, inline(always))]
+ #[cfg_attr(not(zerocopy_inline_always), inline)]
+ fn read_from_prefix(source: &[u8]) -> Result<(Self, &[u8]), SizeError<&[u8], Self>>
+ where
+ Self: Sized,
+ {
+ match Ref::<_, Unalign<Self>>::sized_from_prefix(source) {
+ Ok((r, suffix)) => Ok((Ref::read(&r).into_inner(), suffix)),
+ Err(CastError::Size(e)) => Err(e.with_dst()),
+ Err(CastError::Alignment(_)) => {
+ // SAFETY: `Unalign<Self>` is trivially aligned, so
+ // `Ref::sized_from_prefix` cannot fail due to unmet alignment
+ // requirements.
+ unsafe { core::hint::unreachable_unchecked() }
+ }
+ Err(CastError::Validity(i)) => match i {},
+ }
+ }
+
+ /// Reads a copy of `Self` from the suffix of the given `source`.
+ ///
+ /// This attempts to read a `Self` from the last `size_of::<Self>()` bytes
+ /// of `source`, returning that `Self` and any preceding bytes. If
+ /// `source.len() < size_of::<Self>()`, it returns `Err`.
+ ///
+ /// # Examples
+ ///
+ /// ```
+ /// use zerocopy::FromBytes;
+ /// # use zerocopy_derive::*;
+ ///
+ /// #[derive(FromBytes)]
+ /// #[repr(C)]
+ /// struct PacketTrailer {
+ /// frame_check_sequence: [u8; 4],
+ /// }
+ ///
+ /// // These are more bytes than are needed to encode a `PacketTrailer`.
+ /// let bytes = &[0, 1, 2, 3, 4, 5, 6, 7, 8, 9][..];
+ ///
+ /// let (prefix, trailer) = PacketTrailer::read_from_suffix(bytes).unwrap();
+ ///
+ /// assert_eq!(prefix, [0, 1, 2, 3, 4, 5]);
+ /// assert_eq!(trailer.frame_check_sequence, [6, 7, 8, 9]);
+ /// ```
+ ///
+ #[doc = codegen_section!(
+ header = "h5",
+ bench = "read_from_suffix",
+ format = "coco_static_size",
+ )]
+ #[must_use = "has no side effects"]
+ #[cfg_attr(zerocopy_inline_always, inline(always))]
+ #[cfg_attr(not(zerocopy_inline_always), inline)]
+ fn read_from_suffix(source: &[u8]) -> Result<(&[u8], Self), SizeError<&[u8], Self>>
+ where
+ Self: Sized,
+ {
+ match Ref::<_, Unalign<Self>>::sized_from_suffix(source) {
+ Ok((prefix, r)) => Ok((prefix, Ref::read(&r).into_inner())),
+ Err(CastError::Size(e)) => Err(e.with_dst()),
+ Err(CastError::Alignment(_)) => {
+ // SAFETY: `Unalign<Self>` is trivially aligned, so
+ // `Ref::sized_from_suffix` cannot fail due to unmet alignment
+ // requirements.
+ unsafe { core::hint::unreachable_unchecked() }
+ }
+ Err(CastError::Validity(i)) => match i {},
+ }
+ }
+
+ /// Reads a copy of `self` from an `io::Read`.
+ ///
+ /// This is useful for interfacing with operating system byte sinks (files,
+ /// sockets, etc.).
+ ///
+ /// # Examples
+ ///
+ /// ```no_run
+ /// use zerocopy::{byteorder::big_endian::*, FromBytes};
+ /// use std::fs::File;
+ /// # use zerocopy_derive::*;
+ ///
+ /// #[derive(FromBytes)]
+ /// #[repr(C)]
+ /// struct BitmapFileHeader {
+ /// signature: [u8; 2],
+ /// size: U32,
+ /// reserved: U64,
+ /// offset: U64,
+ /// }
+ ///
+ /// let mut file = File::open("image.bin").unwrap();
+ /// let header = BitmapFileHeader::read_from_io(&mut file).unwrap();
+ /// ```
+ #[cfg(feature = "std")]
+ #[cfg_attr(doc_cfg, doc(cfg(feature = "std")))]
+ #[inline(always)]
+ fn read_from_io<R>(mut src: R) -> io::Result<Self>
+ where
+ Self: Sized,
+ R: io::Read,
+ {
+ // NOTE(#2319, #2320): We do `buf.zero()` separately rather than
+ // constructing `let buf = CoreMaybeUninit::zeroed()` because, if `Self`
+ // contains padding bytes, then a typed copy of `CoreMaybeUninit<Self>`
+ // will not necessarily preserve zeros written to those padding byte
+ // locations, and so `buf` could contain uninitialized bytes.
+ let mut buf = CoreMaybeUninit::<Self>::uninit();
+ buf.zero();
+
+ let ptr = Ptr::from_mut(&mut buf);
+ // SAFETY: After `buf.zero()`, `buf` consists entirely of initialized,
+ // zeroed bytes. Since `MaybeUninit` has no validity requirements, `ptr`
+ // cannot be used to write values which will violate `buf`'s bit
+ // validity. Since `ptr` has `Exclusive` aliasing, nothing other than
+ // `ptr` may be used to mutate `ptr`'s referent, and so its bit validity
+ // cannot be violated even though `buf` may have more permissive bit
+ // validity than `ptr`.
+ let ptr = unsafe { ptr.assume_validity::<invariant::Initialized>() };
+ let ptr = ptr.as_bytes();
+ src.read_exact(ptr.as_mut())?;
+ // SAFETY: `buf` entirely consists of initialized bytes, and `Self` is
+ // `FromBytes`.
+ Ok(unsafe { buf.assume_init() })
+ }
+
+ #[deprecated(since = "0.8.0", note = "renamed to `FromBytes::ref_from_bytes`")]
+ #[doc(hidden)]
+ #[must_use = "has no side effects"]
+ #[inline(always)]
+ fn ref_from(source: &[u8]) -> Option<&Self>
+ where
+ Self: KnownLayout + Immutable,
+ {
+ Self::ref_from_bytes(source).ok()
+ }
+
+ #[deprecated(since = "0.8.0", note = "renamed to `FromBytes::mut_from_bytes`")]
+ #[doc(hidden)]
+ #[must_use = "has no side effects"]
+ #[inline(always)]
+ fn mut_from(source: &mut [u8]) -> Option<&mut Self>
+ where
+ Self: KnownLayout + IntoBytes,
+ {
+ Self::mut_from_bytes(source).ok()
+ }
+
+ #[deprecated(since = "0.8.0", note = "renamed to `FromBytes::ref_from_prefix_with_elems`")]
+ #[doc(hidden)]
+ #[must_use = "has no side effects"]
+ #[inline(always)]
+ fn slice_from_prefix(source: &[u8], count: usize) -> Option<(&[Self], &[u8])>
+ where
+ Self: Sized + Immutable,
+ {
+ <[Self]>::ref_from_prefix_with_elems(source, count).ok()
+ }
+
+ #[deprecated(since = "0.8.0", note = "renamed to `FromBytes::ref_from_suffix_with_elems`")]
+ #[doc(hidden)]
+ #[must_use = "has no side effects"]
+ #[inline(always)]
+ fn slice_from_suffix(source: &[u8], count: usize) -> Option<(&[u8], &[Self])>
+ where
+ Self: Sized + Immutable,
+ {
+ <[Self]>::ref_from_suffix_with_elems(source, count).ok()
+ }
+
+ #[deprecated(since = "0.8.0", note = "renamed to `FromBytes::mut_from_prefix_with_elems`")]
+ #[doc(hidden)]
+ #[must_use = "has no side effects"]
+ #[inline(always)]
+ fn mut_slice_from_prefix(source: &mut [u8], count: usize) -> Option<(&mut [Self], &mut [u8])>
+ where
+ Self: Sized + IntoBytes,
+ {
+ <[Self]>::mut_from_prefix_with_elems(source, count).ok()
+ }
+
+ #[deprecated(since = "0.8.0", note = "renamed to `FromBytes::mut_from_suffix_with_elems`")]
+ #[doc(hidden)]
+ #[must_use = "has no side effects"]
+ #[inline(always)]
+ fn mut_slice_from_suffix(source: &mut [u8], count: usize) -> Option<(&mut [u8], &mut [Self])>
+ where
+ Self: Sized + IntoBytes,
+ {
+ <[Self]>::mut_from_suffix_with_elems(source, count).ok()
+ }
+
+ #[deprecated(since = "0.8.0", note = "renamed to `FromBytes::read_from_bytes`")]
+ #[doc(hidden)]
+ #[must_use = "has no side effects"]
+ #[inline(always)]
+ fn read_from(source: &[u8]) -> Option<Self>
+ where
+ Self: Sized,
+ {
+ Self::read_from_bytes(source).ok()
+ }
+}
+
+/// Interprets the given affix of the given bytes as a `&Self`.
+///
+/// This method computes the largest possible size of `Self` that can fit in the
+/// prefix or suffix bytes of `source`, then attempts to return both a reference
+/// to those bytes interpreted as a `Self`, and a reference to the excess bytes.
+/// If there are insufficient bytes, or if that affix of `source` is not
+/// appropriately aligned, this returns `Err`.
+#[inline(always)]
+fn ref_from_prefix_suffix<T: FromBytes + KnownLayout + Immutable + ?Sized>(
+ source: &[u8],
+ meta: Option<T::PointerMetadata>,
+ cast_type: CastType,
+) -> Result<(&T, &[u8]), CastError<&[u8], T>> {
+ let (slf, prefix_suffix) = Ptr::from_ref(source)
+ .try_cast_into::<_, BecauseImmutable>(cast_type, meta)
+ .map_err(|err| err.map_src(|s| s.as_ref()))?;
+ Ok((slf.recall_validity().as_ref(), prefix_suffix.as_ref()))
+}
+
+/// Interprets the given affix of the given bytes as a `&mut Self` without
+/// copying.
+///
+/// This method computes the largest possible size of `Self` that can fit in the
+/// prefix or suffix bytes of `source`, then attempts to return both a reference
+/// to those bytes interpreted as a `Self`, and a reference to the excess bytes.
+/// If there are insufficient bytes, or if that affix of `source` is not
+/// appropriately aligned, this returns `Err`.
+#[inline(always)]
+fn mut_from_prefix_suffix<T: FromBytes + IntoBytes + KnownLayout + ?Sized>(
+ source: &mut [u8],
+ meta: Option<T::PointerMetadata>,
+ cast_type: CastType,
+) -> Result<(&mut T, &mut [u8]), CastError<&mut [u8], T>> {
+ let (slf, prefix_suffix) = Ptr::from_mut(source)
+ .try_cast_into::<_, BecauseExclusive>(cast_type, meta)
+ .map_err(|err| err.map_src(|s| s.as_mut()))?;
+ Ok((slf.recall_validity::<_, (_, (_, _))>().as_mut(), prefix_suffix.as_mut()))
+}
+
+/// Analyzes whether a type is [`IntoBytes`].
+///
+/// This derive analyzes, at compile time, whether the annotated type satisfies
+/// the [safety conditions] of `IntoBytes` and implements `IntoBytes` if it is
+/// sound to do so. This derive can be applied to structs and enums (see below
+/// for union support); e.g.:
+///
+/// ```
+/// # use zerocopy_derive::{IntoBytes};
+/// #[derive(IntoBytes)]
+/// #[repr(C)]
+/// struct MyStruct {
+/// # /*
+/// ...
+/// # */
+/// }
+///
+/// #[derive(IntoBytes)]
+/// #[repr(u8)]
+/// enum MyEnum {
+/// # Variant,
+/// # /*
+/// ...
+/// # */
+/// }
+/// ```
+///
+/// [safety conditions]: trait@IntoBytes#safety
+///
+/// # Error Messages
+///
+/// On Rust toolchains prior to 1.78.0, due to the way that the custom derive
+/// for `IntoBytes` is implemented, you may get an error like this:
+///
+/// ```text
+/// error[E0277]: the trait bound `(): PaddingFree<Foo, true>` is not satisfied
+/// --> lib.rs:23:10
+/// |
+/// 1 | #[derive(IntoBytes)]
+/// | ^^^^^^^^^ the trait `PaddingFree<Foo, true>` is not implemented for `()`
+/// |
+/// = help: the following implementations were found:
+/// <() as PaddingFree<T, false>>
+/// ```
+///
+/// This error indicates that the type being annotated has padding bytes, which
+/// is illegal for `IntoBytes` types. Consider reducing the alignment of some
+/// fields by using types in the [`byteorder`] module, wrapping field types in
+/// [`Unalign`], adding explicit struct fields where those padding bytes would
+/// be, or using `#[repr(packed)]`. See the Rust Reference's page on [type
+/// layout] for more information about type layout and padding.
+///
+/// [type layout]: https://doc.rust-lang.org/reference/type-layout.html
+///
+/// # Unions
+///
+/// Currently, union bit validity is [up in the air][union-validity], and so
+/// zerocopy does not support `#[derive(IntoBytes)]` on unions by default.
+/// However, implementing `IntoBytes` on a union type is likely sound on all
+/// existing Rust toolchains - it's just that it may become unsound in the
+/// future. You can opt-in to `#[derive(IntoBytes)]` support on unions by
+/// passing the unstable `zerocopy_derive_union_into_bytes` cfg:
+///
+/// ```shell
+/// $ RUSTFLAGS='--cfg zerocopy_derive_union_into_bytes' cargo build
+/// ```
+///
+/// However, it is your responsibility to ensure that this derive is sound on
+/// the specific versions of the Rust toolchain you are using! We make no
+/// stability or soundness guarantees regarding this cfg, and may remove it at
+/// any point.
+///
+/// We are actively working with Rust to stabilize the necessary language
+/// guarantees to support this in a forwards-compatible way, which will enable
+/// us to remove the cfg gate. As part of this effort, we need to know how much
+/// demand there is for this feature. If you would like to use `IntoBytes` on
+/// unions, [please let us know][discussion].
+///
+/// [union-validity]: https://github.com/rust-lang/unsafe-code-guidelines/issues/438
+/// [discussion]: https://github.com/google/zerocopy/discussions/1802
+///
+/// # Analysis
+///
+/// *This section describes, roughly, the analysis performed by this derive to
+/// determine whether it is sound to implement `IntoBytes` for a given type.
+/// Unless you are modifying the implementation of this derive, or attempting to
+/// manually implement `IntoBytes` for a type yourself, you don't need to read
+/// this section.*
+///
+/// If a type has the following properties, then this derive can implement
+/// `IntoBytes` for that type:
+///
+/// - If the type is a struct, its fields must be [`IntoBytes`]. Additionally:
+/// - if the type is `repr(transparent)` or `repr(packed)`, it is
+/// [`IntoBytes`] if its fields are [`IntoBytes`]; else,
+/// - if the type is `repr(C)` with at most one field, it is [`IntoBytes`]
+/// if its field is [`IntoBytes`]; else,
+/// - if the type has no generic parameters, it is [`IntoBytes`] if the type
+/// is sized and has no padding bytes; else,
+/// - if the type is `repr(C)`, its fields must be [`Unaligned`].
+/// - If the type is an enum:
+/// - It must have a defined representation (`repr`s `C`, `u8`, `u16`, `u32`,
+/// `u64`, `usize`, `i8`, `i16`, `i32`, `i64`, or `isize`).
+/// - It must have no padding bytes.
+/// - Its fields must be [`IntoBytes`].
+///
+/// This analysis is subject to change. Unsafe code may *only* rely on the
+/// documented [safety conditions] of `FromBytes`, and must *not* rely on the
+/// implementation details of this derive.
+///
+/// [Rust Reference]: https://doc.rust-lang.org/reference/type-layout.html
+#[cfg(any(feature = "derive", test))]
+#[cfg_attr(doc_cfg, doc(cfg(feature = "derive")))]
+pub use zerocopy_derive::IntoBytes;
+
+/// Types that can be converted to an immutable slice of initialized bytes.
+///
+/// Any `IntoBytes` type can be converted to a slice of initialized bytes of the
+/// same size. This is useful for efficiently serializing structured data as raw
+/// bytes.
+///
+/// # Implementation
+///
+/// **Do not implement this trait yourself!** Instead, use
+/// [`#[derive(IntoBytes)]`][derive]; e.g.:
+///
+/// ```
+/// # use zerocopy_derive::IntoBytes;
+/// #[derive(IntoBytes)]
+/// #[repr(C)]
+/// struct MyStruct {
+/// # /*
+/// ...
+/// # */
+/// }
+///
+/// #[derive(IntoBytes)]
+/// #[repr(u8)]
+/// enum MyEnum {
+/// # Variant0,
+/// # /*
+/// ...
+/// # */
+/// }
+/// ```
+///
+/// This derive performs a sophisticated, compile-time safety analysis to
+/// determine whether a type is `IntoBytes`. See the [derive
+/// documentation][derive] for guidance on how to interpret error messages
+/// produced by the derive's analysis.
+///
+/// # Safety
+///
+/// *This section describes what is required in order for `T: IntoBytes`, and
+/// what unsafe code may assume of such types. If you don't plan on implementing
+/// `IntoBytes` manually, and you don't plan on writing unsafe code that
+/// operates on `IntoBytes` types, then you don't need to read this section.*
+///
+/// If `T: IntoBytes`, then unsafe code may assume that it is sound to treat any
+/// `t: T` as an immutable `[u8]` of length `size_of_val(t)`. If a type is
+/// marked as `IntoBytes` which violates this contract, it may cause undefined
+/// behavior.
+///
+/// `#[derive(IntoBytes)]` only permits [types which satisfy these
+/// requirements][derive-analysis].
+///
+#[cfg_attr(
+ feature = "derive",
+ doc = "[derive]: zerocopy_derive::IntoBytes",
+ doc = "[derive-analysis]: zerocopy_derive::IntoBytes#analysis"
+)]
+#[cfg_attr(
+ not(feature = "derive"),
+ doc = concat!("[derive]: https://docs.rs/zerocopy/", env!("CARGO_PKG_VERSION"), "/zerocopy/derive.IntoBytes.html"),
+ doc = concat!("[derive-analysis]: https://docs.rs/zerocopy/", env!("CARGO_PKG_VERSION"), "/zerocopy/derive.IntoBytes.html#analysis"),
+)]
+#[cfg_attr(
+ not(no_zerocopy_diagnostic_on_unimplemented_1_78_0),
+ diagnostic::on_unimplemented(note = "Consider adding `#[derive(IntoBytes)]` to `{Self}`")
+)]
+pub unsafe trait IntoBytes {
+ // The `Self: Sized` bound makes it so that this function doesn't prevent
+ // `IntoBytes` from being object safe. Note that other `IntoBytes` methods
+ // prevent object safety, but those provide a benefit in exchange for object
+ // safety. If at some point we remove those methods, change their type
+ // signatures, or move them out of this trait so that `IntoBytes` is object
+ // safe again, it's important that this function not prevent object safety.
+ #[doc(hidden)]
+ fn only_derive_is_allowed_to_implement_this_trait()
+ where
+ Self: Sized;
+
+ /// Gets the bytes of this value.
+ ///
+ /// # Examples
+ ///
+ /// ```
+ /// use zerocopy::IntoBytes;
+ /// # use zerocopy_derive::*;
+ ///
+ /// #[derive(IntoBytes, Immutable)]
+ /// #[repr(C)]
+ /// struct PacketHeader {
+ /// src_port: [u8; 2],
+ /// dst_port: [u8; 2],
+ /// length: [u8; 2],
+ /// checksum: [u8; 2],
+ /// }
+ ///
+ /// let header = PacketHeader {
+ /// src_port: [0, 1],
+ /// dst_port: [2, 3],
+ /// length: [4, 5],
+ /// checksum: [6, 7],
+ /// };
+ ///
+ /// let bytes = header.as_bytes();
+ ///
+ /// assert_eq!(bytes, [0, 1, 2, 3, 4, 5, 6, 7]);
+ /// ```
+ ///
+ #[doc = codegen_section!(
+ header = "h5",
+ bench = "as_bytes",
+ format = "coco",
+ arity = 2,
+ [
+ open
+ @index 1
+ @title "Sized"
+ @variant "static_size"
+ ],
+ [
+ @index 2
+ @title "Unsized"
+ @variant "dynamic_size"
+ ]
+ )]
+ #[must_use = "has no side effects"]
+ #[inline(always)]
+ fn as_bytes(&self) -> &[u8]
+ where
+ Self: Immutable,
+ {
+ // Note that this method does not have a `Self: Sized` bound;
+ // `size_of_val` works for unsized values too.
+ let len = mem::size_of_val(self);
+ let slf: *const Self = self;
+
+ // SAFETY:
+ // - `slf.cast::<u8>()` is valid for reads for `len * size_of::<u8>()`
+ // many bytes because...
+ // - `slf` is the same pointer as `self`, and `self` is a reference
+ // which points to an object whose size is `len`. Thus...
+ // - The entire region of `len` bytes starting at `slf` is contained
+ // within a single allocation.
+ // - `slf` is non-null.
+ // - `slf` is trivially aligned to `align_of::<u8>() == 1`.
+ // - `Self: IntoBytes` ensures that all of the bytes of `slf` are
+ // initialized.
+ // - Since `slf` is derived from `self`, and `self` is an immutable
+ // reference, the only other references to this memory region that
+ // could exist are other immutable references, which by `Self:
+ // Immutable` don't permit mutation.
+ // - The total size of the resulting slice is no larger than
+ // `isize::MAX` because no allocation produced by safe code can be
+ // larger than `isize::MAX`.
+ //
+ // FIXME(#429): Add references to docs and quotes.
+ unsafe { slice::from_raw_parts(slf.cast::<u8>(), len) }
+ }
+
+ /// Gets the bytes of this value mutably.
+ ///
+ /// # Examples
+ ///
+ /// ```
+ /// use zerocopy::IntoBytes;
+ /// # use zerocopy_derive::*;
+ ///
+ /// # #[derive(Eq, PartialEq, Debug)]
+ /// #[derive(FromBytes, IntoBytes, Immutable)]
+ /// #[repr(C)]
+ /// struct PacketHeader {
+ /// src_port: [u8; 2],
+ /// dst_port: [u8; 2],
+ /// length: [u8; 2],
+ /// checksum: [u8; 2],
+ /// }
+ ///
+ /// let mut header = PacketHeader {
+ /// src_port: [0, 1],
+ /// dst_port: [2, 3],
+ /// length: [4, 5],
+ /// checksum: [6, 7],
+ /// };
+ ///
+ /// let bytes = header.as_mut_bytes();
+ ///
+ /// assert_eq!(bytes, [0, 1, 2, 3, 4, 5, 6, 7]);
+ ///
+ /// bytes.reverse();
+ ///
+ /// assert_eq!(header, PacketHeader {
+ /// src_port: [7, 6],
+ /// dst_port: [5, 4],
+ /// length: [3, 2],
+ /// checksum: [1, 0],
+ /// });
+ /// ```
+ ///
+ #[doc = codegen_header!("h5", "as_mut_bytes")]
+ ///
+ /// See [`IntoBytes::as_bytes`](#method.as_bytes.codegen).
+ #[must_use = "has no side effects"]
+ #[inline(always)]
+ fn as_mut_bytes(&mut self) -> &mut [u8]
+ where
+ Self: FromBytes,
+ {
+ // Note that this method does not have a `Self: Sized` bound;
+ // `size_of_val` works for unsized values too.
+ let len = mem::size_of_val(self);
+ let slf: *mut Self = self;
+
+ // SAFETY:
+ // - `slf.cast::<u8>()` is valid for reads and writes for `len *
+ // size_of::<u8>()` many bytes because...
+ // - `slf` is the same pointer as `self`, and `self` is a reference
+ // which points to an object whose size is `len`. Thus...
+ // - The entire region of `len` bytes starting at `slf` is contained
+ // within a single allocation.
+ // - `slf` is non-null.
+ // - `slf` is trivially aligned to `align_of::<u8>() == 1`.
+ // - `Self: IntoBytes` ensures that all of the bytes of `slf` are
+ // initialized.
+ // - `Self: FromBytes` ensures that no write to this memory region
+ // could result in it containing an invalid `Self`.
+ // - Since `slf` is derived from `self`, and `self` is a mutable
+ // reference, no other references to this memory region can exist.
+ // - The total size of the resulting slice is no larger than
+ // `isize::MAX` because no allocation produced by safe code can be
+ // larger than `isize::MAX`.
+ //
+ // FIXME(#429): Add references to docs and quotes.
+ unsafe { slice::from_raw_parts_mut(slf.cast::<u8>(), len) }
+ }
+
+ /// Writes a copy of `self` to `dst`.
+ ///
+ /// If `dst.len() != size_of_val(self)`, `write_to` returns `Err`.
+ ///
+ /// # Examples
+ ///
+ /// ```
+ /// use zerocopy::IntoBytes;
+ /// # use zerocopy_derive::*;
+ ///
+ /// #[derive(IntoBytes, Immutable)]
+ /// #[repr(C)]
+ /// struct PacketHeader {
+ /// src_port: [u8; 2],
+ /// dst_port: [u8; 2],
+ /// length: [u8; 2],
+ /// checksum: [u8; 2],
+ /// }
+ ///
+ /// let header = PacketHeader {
+ /// src_port: [0, 1],
+ /// dst_port: [2, 3],
+ /// length: [4, 5],
+ /// checksum: [6, 7],
+ /// };
+ ///
+ /// let mut bytes = [0, 0, 0, 0, 0, 0, 0, 0];
+ ///
+ /// header.write_to(&mut bytes[..]);
+ ///
+ /// assert_eq!(bytes, [0, 1, 2, 3, 4, 5, 6, 7]);
+ /// ```
+ ///
+ /// If too many or too few target bytes are provided, `write_to` returns
+ /// `Err` and leaves the target bytes unmodified:
+ ///
+ /// ```
+ /// # use zerocopy::IntoBytes;
+ /// # let header = u128::MAX;
+ /// let mut excessive_bytes = &mut [0u8; 128][..];
+ ///
+ /// let write_result = header.write_to(excessive_bytes);
+ ///
+ /// assert!(write_result.is_err());
+ /// assert_eq!(excessive_bytes, [0u8; 128]);
+ /// ```
+ ///
+ #[doc = codegen_section!(
+ header = "h5",
+ bench = "write_to",
+ format = "coco",
+ arity = 2,
+ [
+ open
+ @index 1
+ @title "Sized"
+ @variant "static_size"
+ ],
+ [
+ @index 2
+ @title "Unsized"
+ @variant "dynamic_size"
+ ]
+ )]
+ #[must_use = "callers should check the return value to see if the operation succeeded"]
+ #[cfg_attr(zerocopy_inline_always, inline(always))]
+ #[cfg_attr(not(zerocopy_inline_always), inline)]
+ #[allow(clippy::mut_from_ref)] // False positive: `&self -> &mut [u8]`
+ fn write_to(&self, dst: &mut [u8]) -> Result<(), SizeError<&Self, &mut [u8]>>
+ where
+ Self: Immutable,
+ {
+ let src = self.as_bytes();
+ if dst.len() == src.len() {
+ // SAFETY: Within this branch of the conditional, we have ensured
+ // that `dst.len()` is equal to `src.len()`. Neither the size of the
+ // source nor the size of the destination change between the above
+ // size check and the invocation of `copy_unchecked`.
+ unsafe { util::copy_unchecked(src, dst) }
+ Ok(())
+ } else {
+ Err(SizeError::new(self))
+ }
+ }
+
+ /// Writes a copy of `self` to the prefix of `dst`.
+ ///
+ /// `write_to_prefix` writes `self` to the first `size_of_val(self)` bytes
+ /// of `dst`. If `dst.len() < size_of_val(self)`, it returns `Err`.
+ ///
+ /// # Examples
+ ///
+ /// ```
+ /// use zerocopy::IntoBytes;
+ /// # use zerocopy_derive::*;
+ ///
+ /// #[derive(IntoBytes, Immutable)]
+ /// #[repr(C)]
+ /// struct PacketHeader {
+ /// src_port: [u8; 2],
+ /// dst_port: [u8; 2],
+ /// length: [u8; 2],
+ /// checksum: [u8; 2],
+ /// }
+ ///
+ /// let header = PacketHeader {
+ /// src_port: [0, 1],
+ /// dst_port: [2, 3],
+ /// length: [4, 5],
+ /// checksum: [6, 7],
+ /// };
+ ///
+ /// let mut bytes = [0, 0, 0, 0, 0, 0, 0, 0, 0, 0];
+ ///
+ /// header.write_to_prefix(&mut bytes[..]);
+ ///
+ /// assert_eq!(bytes, [0, 1, 2, 3, 4, 5, 6, 7, 0, 0]);
+ /// ```
+ ///
+ /// If insufficient target bytes are provided, `write_to_prefix` returns
+ /// `Err` and leaves the target bytes unmodified:
+ ///
+ /// ```
+ /// # use zerocopy::IntoBytes;
+ /// # let header = u128::MAX;
+ /// let mut insufficient_bytes = &mut [0, 0][..];
+ ///
+ /// let write_result = header.write_to_suffix(insufficient_bytes);
+ ///
+ /// assert!(write_result.is_err());
+ /// assert_eq!(insufficient_bytes, [0, 0]);
+ /// ```
+ ///
+ #[doc = codegen_section!(
+ header = "h5",
+ bench = "write_to_prefix",
+ format = "coco",
+ arity = 2,
+ [
+ open
+ @index 1
+ @title "Sized"
+ @variant "static_size"
+ ],
+ [
+ @index 2
+ @title "Unsized"
+ @variant "dynamic_size"
+ ]
+ )]
+ #[must_use = "callers should check the return value to see if the operation succeeded"]
+ #[cfg_attr(zerocopy_inline_always, inline(always))]
+ #[cfg_attr(not(zerocopy_inline_always), inline)]
+ #[allow(clippy::mut_from_ref)] // False positive: `&self -> &mut [u8]`
+ fn write_to_prefix(&self, dst: &mut [u8]) -> Result<(), SizeError<&Self, &mut [u8]>>
+ where
+ Self: Immutable,
+ {
+ let src = self.as_bytes();
+ match dst.get_mut(..src.len()) {
+ Some(dst) => {
+ // SAFETY: Within this branch of the `match`, we have ensured
+ // through fallible subslicing that `dst.len()` is equal to
+ // `src.len()`. Neither the size of the source nor the size of
+ // the destination change between the above subslicing operation
+ // and the invocation of `copy_unchecked`.
+ unsafe { util::copy_unchecked(src, dst) }
+ Ok(())
+ }
+ None => Err(SizeError::new(self)),
+ }
+ }
+
+ /// Writes a copy of `self` to the suffix of `dst`.
+ ///
+ /// `write_to_suffix` writes `self` to the last `size_of_val(self)` bytes of
+ /// `dst`. If `dst.len() < size_of_val(self)`, it returns `Err`.
+ ///
+ /// # Examples
+ ///
+ /// ```
+ /// use zerocopy::IntoBytes;
+ /// # use zerocopy_derive::*;
+ ///
+ /// #[derive(IntoBytes, Immutable)]
+ /// #[repr(C)]
+ /// struct PacketHeader {
+ /// src_port: [u8; 2],
+ /// dst_port: [u8; 2],
+ /// length: [u8; 2],
+ /// checksum: [u8; 2],
+ /// }
+ ///
+ /// let header = PacketHeader {
+ /// src_port: [0, 1],
+ /// dst_port: [2, 3],
+ /// length: [4, 5],
+ /// checksum: [6, 7],
+ /// };
+ ///
+ /// let mut bytes = [0, 0, 0, 0, 0, 0, 0, 0, 0, 0];
+ ///
+ /// header.write_to_suffix(&mut bytes[..]);
+ ///
+ /// assert_eq!(bytes, [0, 0, 0, 1, 2, 3, 4, 5, 6, 7]);
+ ///
+ /// let mut insufficient_bytes = &mut [0, 0][..];
+ ///
+ /// let write_result = header.write_to_suffix(insufficient_bytes);
+ ///
+ /// assert!(write_result.is_err());
+ /// assert_eq!(insufficient_bytes, [0, 0]);
+ /// ```
+ ///
+ /// If insufficient target bytes are provided, `write_to_suffix` returns
+ /// `Err` and leaves the target bytes unmodified:
+ ///
+ /// ```
+ /// # use zerocopy::IntoBytes;
+ /// # let header = u128::MAX;
+ /// let mut insufficient_bytes = &mut [0, 0][..];
+ ///
+ /// let write_result = header.write_to_suffix(insufficient_bytes);
+ ///
+ /// assert!(write_result.is_err());
+ /// assert_eq!(insufficient_bytes, [0, 0]);
+ /// ```
+ ///
+ #[doc = codegen_section!(
+ header = "h5",
+ bench = "write_to_suffix",
+ format = "coco",
+ arity = 2,
+ [
+ open
+ @index 1
+ @title "Sized"
+ @variant "static_size"
+ ],
+ [
+ @index 2
+ @title "Unsized"
+ @variant "dynamic_size"
+ ]
+ )]
+ #[must_use = "callers should check the return value to see if the operation succeeded"]
+ #[cfg_attr(zerocopy_inline_always, inline(always))]
+ #[cfg_attr(not(zerocopy_inline_always), inline)]
+ #[allow(clippy::mut_from_ref)] // False positive: `&self -> &mut [u8]`
+ fn write_to_suffix(&self, dst: &mut [u8]) -> Result<(), SizeError<&Self, &mut [u8]>>
+ where
+ Self: Immutable,
+ {
+ let src = self.as_bytes();
+ let start = if let Some(start) = dst.len().checked_sub(src.len()) {
+ start
+ } else {
+ return Err(SizeError::new(self));
+ };
+ let dst = if let Some(dst) = dst.get_mut(start..) {
+ dst
+ } else {
+ // get_mut() should never return None here. We return a `SizeError`
+ // rather than .unwrap() because in the event the branch is not
+ // optimized away, returning a value is generally lighter-weight
+ // than panicking.
+ return Err(SizeError::new(self));
+ };
+ // SAFETY: Through fallible subslicing of `dst`, we have ensured that
+ // `dst.len()` is equal to `src.len()`. Neither the size of the source
+ // nor the size of the destination change between the above subslicing
+ // operation and the invocation of `copy_unchecked`.
+ unsafe {
+ util::copy_unchecked(src, dst);
+ }
+ Ok(())
+ }
+
+ /// Writes a copy of `self` to an `io::Write`.
+ ///
+ /// This is a shorthand for `dst.write_all(self.as_bytes())`, and is useful
+ /// for interfacing with operating system byte sinks (files, sockets, etc.).
+ ///
+ /// # Examples
+ ///
+ /// ```no_run
+ /// use zerocopy::{byteorder::big_endian::U16, FromBytes, IntoBytes};
+ /// use std::fs::File;
+ /// # use zerocopy_derive::*;
+ ///
+ /// #[derive(FromBytes, IntoBytes, Immutable, KnownLayout)]
+ /// #[repr(C, packed)]
+ /// struct GrayscaleImage {
+ /// height: U16,
+ /// width: U16,
+ /// pixels: [U16],
+ /// }
+ ///
+ /// let image = GrayscaleImage::ref_from_bytes(&[0, 0, 0, 0][..]).unwrap();
+ /// let mut file = File::create("image.bin").unwrap();
+ /// image.write_to_io(&mut file).unwrap();
+ /// ```
+ ///
+ /// If the write fails, `write_to_io` returns `Err` and a partial write may
+ /// have occurred; e.g.:
+ ///
+ /// ```
+ /// # use zerocopy::IntoBytes;
+ ///
+ /// let src = u128::MAX;
+ /// let mut dst = [0u8; 2];
+ ///
+ /// let write_result = src.write_to_io(&mut dst[..]);
+ ///
+ /// assert!(write_result.is_err());
+ /// assert_eq!(dst, [255, 255]);
+ /// ```
+ #[cfg(feature = "std")]
+ #[cfg_attr(doc_cfg, doc(cfg(feature = "std")))]
+ #[inline(always)]
+ fn write_to_io<W>(&self, mut dst: W) -> io::Result<()>
+ where
+ Self: Immutable,
+ W: io::Write,
+ {
+ dst.write_all(self.as_bytes())
+ }
+
+ #[deprecated(since = "0.8.0", note = "`IntoBytes::as_bytes_mut` was renamed to `as_mut_bytes`")]
+ #[doc(hidden)]
+ #[inline]
+ fn as_bytes_mut(&mut self) -> &mut [u8]
+ where
+ Self: FromBytes,
+ {
+ self.as_mut_bytes()
+ }
+}
+
+/// Analyzes whether a type is [`Unaligned`].
+///
+/// This derive analyzes, at compile time, whether the annotated type satisfies
+/// the [safety conditions] of `Unaligned` and implements `Unaligned` if it is
+/// sound to do so. This derive can be applied to structs, enums, and unions;
+/// e.g.:
+///
+/// ```
+/// # use zerocopy_derive::Unaligned;
+/// #[derive(Unaligned)]
+/// #[repr(C)]
+/// struct MyStruct {
+/// # /*
+/// ...
+/// # */
+/// }
+///
+/// #[derive(Unaligned)]
+/// #[repr(u8)]
+/// enum MyEnum {
+/// # Variant0,
+/// # /*
+/// ...
+/// # */
+/// }
+///
+/// #[derive(Unaligned)]
+/// #[repr(packed)]
+/// union MyUnion {
+/// # variant: u8,
+/// # /*
+/// ...
+/// # */
+/// }
+/// ```
+///
+/// # Analysis
+///
+/// *This section describes, roughly, the analysis performed by this derive to
+/// determine whether it is sound to implement `Unaligned` for a given type.
+/// Unless you are modifying the implementation of this derive, or attempting to
+/// manually implement `Unaligned` for a type yourself, you don't need to read
+/// this section.*
+///
+/// If a type has the following properties, then this derive can implement
+/// `Unaligned` for that type:
+///
+/// - If the type is a struct or union:
+/// - If `repr(align(N))` is provided, `N` must equal 1.
+/// - If the type is `repr(C)` or `repr(transparent)`, all fields must be
+/// [`Unaligned`].
+/// - If the type is not `repr(C)` or `repr(transparent)`, it must be
+/// `repr(packed)` or `repr(packed(1))`.
+/// - If the type is an enum:
+/// - If `repr(align(N))` is provided, `N` must equal 1.
+/// - It must be a field-less enum (meaning that all variants have no fields).
+/// - It must be `repr(i8)` or `repr(u8)`.
+///
+/// [safety conditions]: trait@Unaligned#safety
+#[cfg(any(feature = "derive", test))]
+#[cfg_attr(doc_cfg, doc(cfg(feature = "derive")))]
+pub use zerocopy_derive::Unaligned;
+
+/// Types with no alignment requirement.
+///
+/// If `T: Unaligned`, then `align_of::<T>() == 1`.
+///
+/// # Implementation
+///
+/// **Do not implement this trait yourself!** Instead, use
+/// [`#[derive(Unaligned)]`][derive]; e.g.:
+///
+/// ```
+/// # use zerocopy_derive::Unaligned;
+/// #[derive(Unaligned)]
+/// #[repr(C)]
+/// struct MyStruct {
+/// # /*
+/// ...
+/// # */
+/// }
+///
+/// #[derive(Unaligned)]
+/// #[repr(u8)]
+/// enum MyEnum {
+/// # Variant0,
+/// # /*
+/// ...
+/// # */
+/// }
+///
+/// #[derive(Unaligned)]
+/// #[repr(packed)]
+/// union MyUnion {
+/// # variant: u8,
+/// # /*
+/// ...
+/// # */
+/// }
+/// ```
+///
+/// This derive performs a sophisticated, compile-time safety analysis to
+/// determine whether a type is `Unaligned`.
+///
+/// # Safety
+///
+/// *This section describes what is required in order for `T: Unaligned`, and
+/// what unsafe code may assume of such types. If you don't plan on implementing
+/// `Unaligned` manually, and you don't plan on writing unsafe code that
+/// operates on `Unaligned` types, then you don't need to read this section.*
+///
+/// If `T: Unaligned`, then unsafe code may assume that it is sound to produce a
+/// reference to `T` at any memory location regardless of alignment. If a type
+/// is marked as `Unaligned` which violates this contract, it may cause
+/// undefined behavior.
+///
+/// `#[derive(Unaligned)]` only permits [types which satisfy these
+/// requirements][derive-analysis].
+///
+#[cfg_attr(
+ feature = "derive",
+ doc = "[derive]: zerocopy_derive::Unaligned",
+ doc = "[derive-analysis]: zerocopy_derive::Unaligned#analysis"
+)]
+#[cfg_attr(
+ not(feature = "derive"),
+ doc = concat!("[derive]: https://docs.rs/zerocopy/", env!("CARGO_PKG_VERSION"), "/zerocopy/derive.Unaligned.html"),
+ doc = concat!("[derive-analysis]: https://docs.rs/zerocopy/", env!("CARGO_PKG_VERSION"), "/zerocopy/derive.Unaligned.html#analysis"),
+)]
+#[cfg_attr(
+ not(no_zerocopy_diagnostic_on_unimplemented_1_78_0),
+ diagnostic::on_unimplemented(note = "Consider adding `#[derive(Unaligned)]` to `{Self}`")
+)]
+pub unsafe trait Unaligned {
+ // The `Self: Sized` bound makes it so that `Unaligned` is still object
+ // safe.
+ #[doc(hidden)]
+ fn only_derive_is_allowed_to_implement_this_trait()
+ where
+ Self: Sized;
+}
+
+/// Derives optimized [`PartialEq`] and [`Eq`] implementations.
+///
+/// This derive can be applied to structs and enums implementing both
+/// [`Immutable`] and [`IntoBytes`]; e.g.:
+///
+/// ```
+/// # use zerocopy_derive::{ByteEq, Immutable, IntoBytes};
+/// #[derive(ByteEq, Immutable, IntoBytes)]
+/// #[repr(C)]
+/// struct MyStruct {
+/// # /*
+/// ...
+/// # */
+/// }
+///
+/// #[derive(ByteEq, Immutable, IntoBytes)]
+/// #[repr(u8)]
+/// enum MyEnum {
+/// # Variant,
+/// # /*
+/// ...
+/// # */
+/// }
+/// ```
+///
+/// The standard library's [`derive(Eq, PartialEq)`][derive@PartialEq] computes
+/// equality by individually comparing each field. Instead, the implementation
+/// of [`PartialEq::eq`] emitted by `derive(ByteHash)` converts the entirety of
+/// `self` and `other` to byte slices and compares those slices for equality.
+/// This may have performance advantages.
+#[cfg(any(feature = "derive", test))]
+#[cfg_attr(doc_cfg, doc(cfg(feature = "derive")))]
+pub use zerocopy_derive::ByteEq;
+/// Derives an optimized [`Hash`] implementation.
+///
+/// This derive can be applied to structs and enums implementing both
+/// [`Immutable`] and [`IntoBytes`]; e.g.:
+///
+/// ```
+/// # use zerocopy_derive::{ByteHash, Immutable, IntoBytes};
+/// #[derive(ByteHash, Immutable, IntoBytes)]
+/// #[repr(C)]
+/// struct MyStruct {
+/// # /*
+/// ...
+/// # */
+/// }
+///
+/// #[derive(ByteHash, Immutable, IntoBytes)]
+/// #[repr(u8)]
+/// enum MyEnum {
+/// # Variant,
+/// # /*
+/// ...
+/// # */
+/// }
+/// ```
+///
+/// The standard library's [`derive(Hash)`][derive@Hash] produces hashes by
+/// individually hashing each field and combining the results. Instead, the
+/// implementations of [`Hash::hash()`] and [`Hash::hash_slice()`] generated by
+/// `derive(ByteHash)` convert the entirety of `self` to a byte slice and hashes
+/// it in a single call to [`Hasher::write()`]. This may have performance
+/// advantages.
+///
+/// [`Hash`]: core::hash::Hash
+/// [`Hash::hash()`]: core::hash::Hash::hash()
+/// [`Hash::hash_slice()`]: core::hash::Hash::hash_slice()
+#[cfg(any(feature = "derive", test))]
+#[cfg_attr(doc_cfg, doc(cfg(feature = "derive")))]
+pub use zerocopy_derive::ByteHash;
+/// Implements [`SplitAt`].
+///
+/// This derive can be applied to structs; e.g.:
+///
+/// ```
+/// # use zerocopy_derive::{ByteEq, Immutable, IntoBytes};
+/// #[derive(ByteEq, Immutable, IntoBytes)]
+/// #[repr(C)]
+/// struct MyStruct {
+/// # /*
+/// ...
+/// # */
+/// }
+/// ```
+#[cfg(any(feature = "derive", test))]
+#[cfg_attr(doc_cfg, doc(cfg(feature = "derive")))]
+pub use zerocopy_derive::SplitAt;
+
+#[cfg(feature = "alloc")]
+#[cfg_attr(doc_cfg, doc(cfg(feature = "alloc")))]
+#[cfg(not(no_zerocopy_panic_in_const_and_vec_try_reserve_1_57_0))]
+mod alloc_support {
+ use super::*;
+
+ /// Extends a `Vec<T>` by pushing `additional` new items onto the end of the
+ /// vector. The new items are initialized with zeros.
+ #[cfg(not(no_zerocopy_panic_in_const_and_vec_try_reserve_1_57_0))]
+ #[doc(hidden)]
+ #[deprecated(since = "0.8.0", note = "moved to `FromZeros`")]
+ #[inline(always)]
+ pub fn extend_vec_zeroed<T: FromZeros>(
+ v: &mut Vec<T>,
+ additional: usize,
+ ) -> Result<(), AllocError> {
+ <T as FromZeros>::extend_vec_zeroed(v, additional)
+ }
+
+ /// Inserts `additional` new items into `Vec<T>` at `position`. The new
+ /// items are initialized with zeros.
+ ///
+ /// # Panics
+ ///
+ /// Panics if `position > v.len()`.
+ #[cfg(not(no_zerocopy_panic_in_const_and_vec_try_reserve_1_57_0))]
+ #[doc(hidden)]
+ #[deprecated(since = "0.8.0", note = "moved to `FromZeros`")]
+ #[inline(always)]
+ pub fn insert_vec_zeroed<T: FromZeros>(
+ v: &mut Vec<T>,
+ position: usize,
+ additional: usize,
+ ) -> Result<(), AllocError> {
+ <T as FromZeros>::insert_vec_zeroed(v, position, additional)
+ }
+}
+
+#[cfg(feature = "alloc")]
+#[cfg(not(no_zerocopy_panic_in_const_and_vec_try_reserve_1_57_0))]
+#[doc(hidden)]
+pub use alloc_support::*;
+
+#[cfg(test)]
+#[allow(clippy::assertions_on_result_states, clippy::unreadable_literal)]
+mod tests {
+ use static_assertions::assert_impl_all;
+
+ use super::*;
+ use crate::util::testutil::*;
+
+ // An unsized type.
+ //
+ // This is used to test the custom derives of our traits. The `[u8]` type
+ // gets a hand-rolled impl, so it doesn't exercise our custom derives.
+ #[derive(Debug, Eq, PartialEq, FromBytes, IntoBytes, Unaligned, Immutable)]
+ #[repr(transparent)]
+ struct Unsized([u8]);
+
+ impl Unsized {
+ fn from_mut_slice(slc: &mut [u8]) -> &mut Unsized {
+ // SAFETY: This *probably* sound - since the layouts of `[u8]` and
+ // `Unsized` are the same, so are the layouts of `&mut [u8]` and
+ // `&mut Unsized`. [1] Even if it turns out that this isn't actually
+ // guaranteed by the language spec, we can just change this since
+ // it's in test code.
+ //
+ // [1] https://github.com/rust-lang/unsafe-code-guidelines/issues/375
+ unsafe { mem::transmute(slc) }
+ }
+ }
+
+ #[test]
+ fn test_known_layout() {
+ // Test that `$ty` and `ManuallyDrop<$ty>` have the expected layout.
+ // Test that `PhantomData<$ty>` has the same layout as `()` regardless
+ // of `$ty`.
+ macro_rules! test {
+ ($ty:ty, $expect:expr) => {
+ let expect = $expect;
+ assert_eq!(<$ty as KnownLayout>::LAYOUT, expect);
+ assert_eq!(<ManuallyDrop<$ty> as KnownLayout>::LAYOUT, expect);
+ assert_eq!(<PhantomData<$ty> as KnownLayout>::LAYOUT, <() as KnownLayout>::LAYOUT);
+ };
+ }
+
+ let layout =
+ |offset, align, trailing_slice_elem_size, statically_shallow_unpadded| DstLayout {
+ align: NonZeroUsize::new(align).unwrap(),
+ size_info: match trailing_slice_elem_size {
+ None => SizeInfo::Sized { size: offset },
+ Some(elem_size) => {
+ SizeInfo::SliceDst(TrailingSliceLayout { offset, elem_size })
+ }
+ },
+ statically_shallow_unpadded,
+ };
+
+ test!((), layout(0, 1, None, false));
+ test!(u8, layout(1, 1, None, false));
+ // Use `align_of` because `u64` alignment may be smaller than 8 on some
+ // platforms.
+ test!(u64, layout(8, mem::align_of::<u64>(), None, false));
+ test!(AU64, layout(8, 8, None, false));
+
+ test!(Option<&'static ()>, usize::LAYOUT);
+
+ test!([()], layout(0, 1, Some(0), true));
+ test!([u8], layout(0, 1, Some(1), true));
+ test!(str, layout(0, 1, Some(1), true));
+ }
+
+ #[cfg(feature = "derive")]
+ #[test]
+ fn test_known_layout_derive() {
+ // In this and other files (`late_compile_pass.rs`,
+ // `mid_compile_pass.rs`, and `struct.rs`), we test success and failure
+ // modes of `derive(KnownLayout)` for the following combination of
+ // properties:
+ //
+ // +------------+--------------------------------------+-----------+
+ // | | trailing field properties | |
+ // | `repr(C)`? | generic? | `KnownLayout`? | `Sized`? | Type Name |
+ // |------------+----------+----------------+----------+-----------|
+ // | N | N | N | N | KL00 |
+ // | N | N | N | Y | KL01 |
+ // | N | N | Y | N | KL02 |
+ // | N | N | Y | Y | KL03 |
+ // | N | Y | N | N | KL04 |
+ // | N | Y | N | Y | KL05 |
+ // | N | Y | Y | N | KL06 |
+ // | N | Y | Y | Y | KL07 |
+ // | Y | N | N | N | KL08 |
+ // | Y | N | N | Y | KL09 |
+ // | Y | N | Y | N | KL10 |
+ // | Y | N | Y | Y | KL11 |
+ // | Y | Y | N | N | KL12 |
+ // | Y | Y | N | Y | KL13 |
+ // | Y | Y | Y | N | KL14 |
+ // | Y | Y | Y | Y | KL15 |
+ // +------------+----------+----------------+----------+-----------+
+
+ struct NotKnownLayout<T = ()> {
+ _t: T,
+ }
+
+ #[derive(KnownLayout)]
+ #[repr(C)]
+ struct AlignSize<const ALIGN: usize, const SIZE: usize>
+ where
+ elain::Align<ALIGN>: elain::Alignment,
+ {
+ _align: elain::Align<ALIGN>,
+ size: [u8; SIZE],
+ }
+
+ type AU16 = AlignSize<2, 2>;
+ type AU32 = AlignSize<4, 4>;
+
+ fn _assert_kl<T: ?Sized + KnownLayout>(_: &T) {}
+
+ let sized_layout = |align, size| DstLayout {
+ align: NonZeroUsize::new(align).unwrap(),
+ size_info: SizeInfo::Sized { size },
+ statically_shallow_unpadded: false,
+ };
+
+ let unsized_layout = |align, elem_size, offset, statically_shallow_unpadded| DstLayout {
+ align: NonZeroUsize::new(align).unwrap(),
+ size_info: SizeInfo::SliceDst(TrailingSliceLayout { offset, elem_size }),
+ statically_shallow_unpadded,
+ };
+
+ // | `repr(C)`? | generic? | `KnownLayout`? | `Sized`? | Type Name |
+ // | N | N | N | Y | KL01 |
+ #[allow(dead_code)]
+ #[derive(KnownLayout)]
+ struct KL01(NotKnownLayout<AU32>, NotKnownLayout<AU16>);
+
+ let expected = DstLayout::for_type::<KL01>();
+
+ assert_eq!(<KL01 as KnownLayout>::LAYOUT, expected);
+ assert_eq!(<KL01 as KnownLayout>::LAYOUT, sized_layout(4, 8));
+
+ // ...with `align(N)`:
+ #[allow(dead_code)]
+ #[derive(KnownLayout)]
+ #[repr(align(64))]
+ struct KL01Align(NotKnownLayout<AU32>, NotKnownLayout<AU16>);
+
+ let expected = DstLayout::for_type::<KL01Align>();
+
+ assert_eq!(<KL01Align as KnownLayout>::LAYOUT, expected);
+ assert_eq!(<KL01Align as KnownLayout>::LAYOUT, sized_layout(64, 64));
+
+ // ...with `packed`:
+ #[allow(dead_code)]
+ #[derive(KnownLayout)]
+ #[repr(packed)]
+ struct KL01Packed(NotKnownLayout<AU32>, NotKnownLayout<AU16>);
+
+ let expected = DstLayout::for_type::<KL01Packed>();
+
+ assert_eq!(<KL01Packed as KnownLayout>::LAYOUT, expected);
+ assert_eq!(<KL01Packed as KnownLayout>::LAYOUT, sized_layout(1, 6));
+
+ // ...with `packed(N)`:
+ #[allow(dead_code)]
+ #[derive(KnownLayout)]
+ #[repr(packed(2))]
+ struct KL01PackedN(NotKnownLayout<AU32>, NotKnownLayout<AU16>);
+
+ assert_impl_all!(KL01PackedN: KnownLayout);
+
+ let expected = DstLayout::for_type::<KL01PackedN>();
+
+ assert_eq!(<KL01PackedN as KnownLayout>::LAYOUT, expected);
+ assert_eq!(<KL01PackedN as KnownLayout>::LAYOUT, sized_layout(2, 6));
+
+ // | `repr(C)`? | generic? | `KnownLayout`? | `Sized`? | Type Name |
+ // | N | N | Y | Y | KL03 |
+ #[allow(dead_code)]
+ #[derive(KnownLayout)]
+ struct KL03(NotKnownLayout, u8);
+
+ let expected = DstLayout::for_type::<KL03>();
+
+ assert_eq!(<KL03 as KnownLayout>::LAYOUT, expected);
+ assert_eq!(<KL03 as KnownLayout>::LAYOUT, sized_layout(1, 1));
+
+ // ... with `align(N)`
+ #[allow(dead_code)]
+ #[derive(KnownLayout)]
+ #[repr(align(64))]
+ struct KL03Align(NotKnownLayout<AU32>, u8);
+
+ let expected = DstLayout::for_type::<KL03Align>();
+
+ assert_eq!(<KL03Align as KnownLayout>::LAYOUT, expected);
+ assert_eq!(<KL03Align as KnownLayout>::LAYOUT, sized_layout(64, 64));
+
+ // ... with `packed`:
+ #[allow(dead_code)]
+ #[derive(KnownLayout)]
+ #[repr(packed)]
+ struct KL03Packed(NotKnownLayout<AU32>, u8);
+
+ let expected = DstLayout::for_type::<KL03Packed>();
+
+ assert_eq!(<KL03Packed as KnownLayout>::LAYOUT, expected);
+ assert_eq!(<KL03Packed as KnownLayout>::LAYOUT, sized_layout(1, 5));
+
+ // ... with `packed(N)`
+ #[allow(dead_code)]
+ #[derive(KnownLayout)]
+ #[repr(packed(2))]
+ struct KL03PackedN(NotKnownLayout<AU32>, u8);
+
+ assert_impl_all!(KL03PackedN: KnownLayout);
+
+ let expected = DstLayout::for_type::<KL03PackedN>();
+
+ assert_eq!(<KL03PackedN as KnownLayout>::LAYOUT, expected);
+ assert_eq!(<KL03PackedN as KnownLayout>::LAYOUT, sized_layout(2, 6));
+
+ // | `repr(C)`? | generic? | `KnownLayout`? | `Sized`? | Type Name |
+ // | N | Y | N | Y | KL05 |
+ #[allow(dead_code)]
+ #[derive(KnownLayout)]
+ struct KL05<T>(u8, T);
+
+ fn _test_kl05<T>(t: T) -> impl KnownLayout {
+ KL05(0u8, t)
+ }
+
+ // | `repr(C)`? | generic? | `KnownLayout`? | `Sized`? | Type Name |
+ // | N | Y | Y | Y | KL07 |
+ #[allow(dead_code)]
+ #[derive(KnownLayout)]
+ struct KL07<T: KnownLayout>(u8, T);
+
+ fn _test_kl07<T: KnownLayout>(t: T) -> impl KnownLayout {
+ let _ = KL07(0u8, t);
+ }
+
+ // | `repr(C)`? | generic? | `KnownLayout`? | `Sized`? | Type Name |
+ // | Y | N | Y | N | KL10 |
+ #[allow(dead_code)]
+ #[derive(KnownLayout)]
+ #[repr(C)]
+ struct KL10(NotKnownLayout<AU32>, [u8]);
+
+ let expected = DstLayout::new_zst(None)
+ .extend(DstLayout::for_type::<NotKnownLayout<AU32>>(), None)
+ .extend(<[u8] as KnownLayout>::LAYOUT, None)
+ .pad_to_align();
+
+ assert_eq!(<KL10 as KnownLayout>::LAYOUT, expected);
+ assert_eq!(<KL10 as KnownLayout>::LAYOUT, unsized_layout(4, 1, 4, false));
+
+ // ...with `align(N)`:
+ #[allow(dead_code)]
+ #[derive(KnownLayout)]
+ #[repr(C, align(64))]
+ struct KL10Align(NotKnownLayout<AU32>, [u8]);
+
+ let repr_align = NonZeroUsize::new(64);
+
+ let expected = DstLayout::new_zst(repr_align)
+ .extend(DstLayout::for_type::<NotKnownLayout<AU32>>(), None)
+ .extend(<[u8] as KnownLayout>::LAYOUT, None)
+ .pad_to_align();
+
+ assert_eq!(<KL10Align as KnownLayout>::LAYOUT, expected);
+ assert_eq!(<KL10Align as KnownLayout>::LAYOUT, unsized_layout(64, 1, 4, false));
+
+ // ...with `packed`:
+ #[allow(dead_code)]
+ #[derive(KnownLayout)]
+ #[repr(C, packed)]
+ struct KL10Packed(NotKnownLayout<AU32>, [u8]);
+
+ let repr_packed = NonZeroUsize::new(1);
+
+ let expected = DstLayout::new_zst(None)
+ .extend(DstLayout::for_type::<NotKnownLayout<AU32>>(), repr_packed)
+ .extend(<[u8] as KnownLayout>::LAYOUT, repr_packed)
+ .pad_to_align();
+
+ assert_eq!(<KL10Packed as KnownLayout>::LAYOUT, expected);
+ assert_eq!(<KL10Packed as KnownLayout>::LAYOUT, unsized_layout(1, 1, 4, false));
+
+ // ...with `packed(N)`:
+ #[allow(dead_code)]
+ #[derive(KnownLayout)]
+ #[repr(C, packed(2))]
+ struct KL10PackedN(NotKnownLayout<AU32>, [u8]);
+
+ let repr_packed = NonZeroUsize::new(2);
+
+ let expected = DstLayout::new_zst(None)
+ .extend(DstLayout::for_type::<NotKnownLayout<AU32>>(), repr_packed)
+ .extend(<[u8] as KnownLayout>::LAYOUT, repr_packed)
+ .pad_to_align();
+
+ assert_eq!(<KL10PackedN as KnownLayout>::LAYOUT, expected);
+ assert_eq!(<KL10PackedN as KnownLayout>::LAYOUT, unsized_layout(2, 1, 4, false));
+
+ // | `repr(C)`? | generic? | `KnownLayout`? | `Sized`? | Type Name |
+ // | Y | N | Y | Y | KL11 |
+ #[allow(dead_code)]
+ #[derive(KnownLayout)]
+ #[repr(C)]
+ struct KL11(NotKnownLayout<AU64>, u8);
+
+ let expected = DstLayout::new_zst(None)
+ .extend(DstLayout::for_type::<NotKnownLayout<AU64>>(), None)
+ .extend(<u8 as KnownLayout>::LAYOUT, None)
+ .pad_to_align();
+
+ assert_eq!(<KL11 as KnownLayout>::LAYOUT, expected);
+ assert_eq!(<KL11 as KnownLayout>::LAYOUT, sized_layout(8, 16));
+
+ // ...with `align(N)`:
+ #[allow(dead_code)]
+ #[derive(KnownLayout)]
+ #[repr(C, align(64))]
+ struct KL11Align(NotKnownLayout<AU64>, u8);
+
+ let repr_align = NonZeroUsize::new(64);
+
+ let expected = DstLayout::new_zst(repr_align)
+ .extend(DstLayout::for_type::<NotKnownLayout<AU64>>(), None)
+ .extend(<u8 as KnownLayout>::LAYOUT, None)
+ .pad_to_align();
+
+ assert_eq!(<KL11Align as KnownLayout>::LAYOUT, expected);
+ assert_eq!(<KL11Align as KnownLayout>::LAYOUT, sized_layout(64, 64));
+
+ // ...with `packed`:
+ #[allow(dead_code)]
+ #[derive(KnownLayout)]
+ #[repr(C, packed)]
+ struct KL11Packed(NotKnownLayout<AU64>, u8);
+
+ let repr_packed = NonZeroUsize::new(1);
+
+ let expected = DstLayout::new_zst(None)
+ .extend(DstLayout::for_type::<NotKnownLayout<AU64>>(), repr_packed)
+ .extend(<u8 as KnownLayout>::LAYOUT, repr_packed)
+ .pad_to_align();
+
+ assert_eq!(<KL11Packed as KnownLayout>::LAYOUT, expected);
+ assert_eq!(<KL11Packed as KnownLayout>::LAYOUT, sized_layout(1, 9));
+
+ // ...with `packed(N)`:
+ #[allow(dead_code)]
+ #[derive(KnownLayout)]
+ #[repr(C, packed(2))]
+ struct KL11PackedN(NotKnownLayout<AU64>, u8);
+
+ let repr_packed = NonZeroUsize::new(2);
+
+ let expected = DstLayout::new_zst(None)
+ .extend(DstLayout::for_type::<NotKnownLayout<AU64>>(), repr_packed)
+ .extend(<u8 as KnownLayout>::LAYOUT, repr_packed)
+ .pad_to_align();
+
+ assert_eq!(<KL11PackedN as KnownLayout>::LAYOUT, expected);
+ assert_eq!(<KL11PackedN as KnownLayout>::LAYOUT, sized_layout(2, 10));
+
+ // | `repr(C)`? | generic? | `KnownLayout`? | `Sized`? | Type Name |
+ // | Y | Y | Y | N | KL14 |
+ #[allow(dead_code)]
+ #[derive(KnownLayout)]
+ #[repr(C)]
+ struct KL14<T: ?Sized + KnownLayout>(u8, T);
+
+ fn _test_kl14<T: ?Sized + KnownLayout>(kl: &KL14<T>) {
+ _assert_kl(kl)
+ }
+
+ // | `repr(C)`? | generic? | `KnownLayout`? | `Sized`? | Type Name |
+ // | Y | Y | Y | Y | KL15 |
+ #[allow(dead_code)]
+ #[derive(KnownLayout)]
+ #[repr(C)]
+ struct KL15<T: KnownLayout>(u8, T);
+
+ fn _test_kl15<T: KnownLayout>(t: T) -> impl KnownLayout {
+ let _ = KL15(0u8, t);
+ }
+
+ // Test a variety of combinations of field types:
+ // - ()
+ // - u8
+ // - AU16
+ // - [()]
+ // - [u8]
+ // - [AU16]
+
+ #[allow(clippy::upper_case_acronyms, dead_code)]
+ #[derive(KnownLayout)]
+ #[repr(C)]
+ struct KLTU<T, U: ?Sized>(T, U);
+
+ assert_eq!(<KLTU<(), ()> as KnownLayout>::LAYOUT, sized_layout(1, 0));
+
+ assert_eq!(<KLTU<(), u8> as KnownLayout>::LAYOUT, sized_layout(1, 1));
+
+ assert_eq!(<KLTU<(), AU16> as KnownLayout>::LAYOUT, sized_layout(2, 2));
+
+ assert_eq!(<KLTU<(), [()]> as KnownLayout>::LAYOUT, unsized_layout(1, 0, 0, false));
+
+ assert_eq!(<KLTU<(), [u8]> as KnownLayout>::LAYOUT, unsized_layout(1, 1, 0, false));
+
+ assert_eq!(<KLTU<(), [AU16]> as KnownLayout>::LAYOUT, unsized_layout(2, 2, 0, false));
+
+ assert_eq!(<KLTU<u8, ()> as KnownLayout>::LAYOUT, sized_layout(1, 1));
+
+ assert_eq!(<KLTU<u8, u8> as KnownLayout>::LAYOUT, sized_layout(1, 2));
+
+ assert_eq!(<KLTU<u8, AU16> as KnownLayout>::LAYOUT, sized_layout(2, 4));
+
+ assert_eq!(<KLTU<u8, [()]> as KnownLayout>::LAYOUT, unsized_layout(1, 0, 1, false));
+
+ assert_eq!(<KLTU<u8, [u8]> as KnownLayout>::LAYOUT, unsized_layout(1, 1, 1, false));
+
+ assert_eq!(<KLTU<u8, [AU16]> as KnownLayout>::LAYOUT, unsized_layout(2, 2, 2, false));
+
+ assert_eq!(<KLTU<AU16, ()> as KnownLayout>::LAYOUT, sized_layout(2, 2));
+
+ assert_eq!(<KLTU<AU16, u8> as KnownLayout>::LAYOUT, sized_layout(2, 4));
+
+ assert_eq!(<KLTU<AU16, AU16> as KnownLayout>::LAYOUT, sized_layout(2, 4));
+
+ assert_eq!(<KLTU<AU16, [()]> as KnownLayout>::LAYOUT, unsized_layout(2, 0, 2, false));
+
+ assert_eq!(<KLTU<AU16, [u8]> as KnownLayout>::LAYOUT, unsized_layout(2, 1, 2, false));
+
+ assert_eq!(<KLTU<AU16, [AU16]> as KnownLayout>::LAYOUT, unsized_layout(2, 2, 2, false));
+
+ // Test a variety of field counts.
+
+ #[derive(KnownLayout)]
+ #[repr(C)]
+ struct KLF0;
+
+ assert_eq!(<KLF0 as KnownLayout>::LAYOUT, sized_layout(1, 0));
+
+ #[derive(KnownLayout)]
+ #[repr(C)]
+ struct KLF1([u8]);
+
+ assert_eq!(<KLF1 as KnownLayout>::LAYOUT, unsized_layout(1, 1, 0, true));
+
+ #[derive(KnownLayout)]
+ #[repr(C)]
+ struct KLF2(NotKnownLayout<u8>, [u8]);
+
+ assert_eq!(<KLF2 as KnownLayout>::LAYOUT, unsized_layout(1, 1, 1, false));
+
+ #[derive(KnownLayout)]
+ #[repr(C)]
+ struct KLF3(NotKnownLayout<u8>, NotKnownLayout<AU16>, [u8]);
+
+ assert_eq!(<KLF3 as KnownLayout>::LAYOUT, unsized_layout(2, 1, 4, false));
+
+ #[derive(KnownLayout)]
+ #[repr(C)]
+ struct KLF4(NotKnownLayout<u8>, NotKnownLayout<AU16>, NotKnownLayout<AU32>, [u8]);
+
+ assert_eq!(<KLF4 as KnownLayout>::LAYOUT, unsized_layout(4, 1, 8, false));
+ }
+
+ #[test]
+ fn test_object_safety() {
+ fn _takes_immutable(_: &dyn Immutable) {}
+ fn _takes_unaligned(_: &dyn Unaligned) {}
+ }
+
+ #[test]
+ fn test_from_zeros_only() {
+ // Test types that implement `FromZeros` but not `FromBytes`.
+
+ assert!(!bool::new_zeroed());
+ assert_eq!(char::new_zeroed(), '\0');
+
+ #[cfg(feature = "alloc")]
+ {
+ assert_eq!(bool::new_box_zeroed(), Ok(Box::new(false)));
+ assert_eq!(char::new_box_zeroed(), Ok(Box::new('\0')));
+
+ assert_eq!(
+ <[bool]>::new_box_zeroed_with_elems(3).unwrap().as_ref(),
+ [false, false, false]
+ );
+ assert_eq!(
+ <[char]>::new_box_zeroed_with_elems(3).unwrap().as_ref(),
+ ['\0', '\0', '\0']
+ );
+
+ assert_eq!(bool::new_vec_zeroed(3).unwrap().as_ref(), [false, false, false]);
+ assert_eq!(char::new_vec_zeroed(3).unwrap().as_ref(), ['\0', '\0', '\0']);
+ }
+
+ let mut string = "hello".to_string();
+ let s: &mut str = string.as_mut();
+ assert_eq!(s, "hello");
+ s.zero();
+ assert_eq!(s, "\0\0\0\0\0");
+ }
+
+ #[test]
+ fn test_zst_count_preserved() {
+ // Test that, when an explicit count is provided to for a type with a
+ // ZST trailing slice element, that count is preserved. This is
+ // important since, for such types, all element counts result in objects
+ // of the same size, and so the correct behavior is ambiguous. However,
+ // preserving the count as requested by the user is the behavior that we
+ // document publicly.
+
+ // FromZeros methods
+ #[cfg(feature = "alloc")]
+ assert_eq!(<[()]>::new_box_zeroed_with_elems(3).unwrap().len(), 3);
+ #[cfg(feature = "alloc")]
+ assert_eq!(<()>::new_vec_zeroed(3).unwrap().len(), 3);
+
+ // FromBytes methods
+ assert_eq!(<[()]>::ref_from_bytes_with_elems(&[][..], 3).unwrap().len(), 3);
+ assert_eq!(<[()]>::ref_from_prefix_with_elems(&[][..], 3).unwrap().0.len(), 3);
+ assert_eq!(<[()]>::ref_from_suffix_with_elems(&[][..], 3).unwrap().1.len(), 3);
+ assert_eq!(<[()]>::mut_from_bytes_with_elems(&mut [][..], 3).unwrap().len(), 3);
+ assert_eq!(<[()]>::mut_from_prefix_with_elems(&mut [][..], 3).unwrap().0.len(), 3);
+ assert_eq!(<[()]>::mut_from_suffix_with_elems(&mut [][..], 3).unwrap().1.len(), 3);
+ }
+
+ #[test]
+ fn test_read_write() {
+ const VAL: u64 = 0x12345678;
+ #[cfg(target_endian = "big")]
+ const VAL_BYTES: [u8; 8] = VAL.to_be_bytes();
+ #[cfg(target_endian = "little")]
+ const VAL_BYTES: [u8; 8] = VAL.to_le_bytes();
+ const ZEROS: [u8; 8] = [0u8; 8];
+
+ // Test `FromBytes::{read_from, read_from_prefix, read_from_suffix}`.
+
+ assert_eq!(u64::read_from_bytes(&VAL_BYTES[..]), Ok(VAL));
+ // The first 8 bytes are from `VAL_BYTES` and the second 8 bytes are all
+ // zeros.
+ let bytes_with_prefix: [u8; 16] = transmute!([VAL_BYTES, [0; 8]]);
+ assert_eq!(u64::read_from_prefix(&bytes_with_prefix[..]), Ok((VAL, &ZEROS[..])));
+ assert_eq!(u64::read_from_suffix(&bytes_with_prefix[..]), Ok((&VAL_BYTES[..], 0)));
+ // The first 8 bytes are all zeros and the second 8 bytes are from
+ // `VAL_BYTES`
+ let bytes_with_suffix: [u8; 16] = transmute!([[0; 8], VAL_BYTES]);
+ assert_eq!(u64::read_from_prefix(&bytes_with_suffix[..]), Ok((0, &VAL_BYTES[..])));
+ assert_eq!(u64::read_from_suffix(&bytes_with_suffix[..]), Ok((&ZEROS[..], VAL)));
+
+ // Test `IntoBytes::{write_to, write_to_prefix, write_to_suffix}`.
+
+ let mut bytes = [0u8; 8];
+ assert_eq!(VAL.write_to(&mut bytes[..]), Ok(()));
+ assert_eq!(bytes, VAL_BYTES);
+ let mut bytes = [0u8; 16];
+ assert_eq!(VAL.write_to_prefix(&mut bytes[..]), Ok(()));
+ let want: [u8; 16] = transmute!([VAL_BYTES, [0; 8]]);
+ assert_eq!(bytes, want);
+ let mut bytes = [0u8; 16];
+ assert_eq!(VAL.write_to_suffix(&mut bytes[..]), Ok(()));
+ let want: [u8; 16] = transmute!([[0; 8], VAL_BYTES]);
+ assert_eq!(bytes, want);
+ }
+
+ #[test]
+ #[cfg(feature = "std")]
+ fn test_read_io_with_padding_soundness() {
+ // This test is designed to exhibit potential UB in
+ // `FromBytes::read_from_io`. (see #2319, #2320).
+
+ // On most platforms (where `align_of::<u16>() == 2`), `WithPadding`
+ // will have inter-field padding between `x` and `y`.
+ #[derive(FromBytes)]
+ #[repr(C)]
+ struct WithPadding {
+ x: u8,
+ y: u16,
+ }
+ struct ReadsInRead;
+ impl std::io::Read for ReadsInRead {
+ fn read(&mut self, buf: &mut [u8]) -> std::io::Result<usize> {
+ // This body branches on every byte of `buf`, ensuring that it
+ // exhibits UB if any byte of `buf` is uninitialized.
+ if buf.iter().all(|&x| x == 0) {
+ Ok(buf.len())
+ } else {
+ buf.iter_mut().for_each(|x| *x = 0);
+ Ok(buf.len())
+ }
+ }
+ }
+ assert!(matches!(WithPadding::read_from_io(ReadsInRead), Ok(WithPadding { x: 0, y: 0 })));
+ }
+
+ #[test]
+ #[cfg(feature = "std")]
+ fn test_read_write_io() {
+ let mut long_buffer = [0, 0, 0, 0];
+ assert!(matches!(u16::MAX.write_to_io(&mut long_buffer[..]), Ok(())));
+ assert_eq!(long_buffer, [255, 255, 0, 0]);
+ assert!(matches!(u16::read_from_io(&long_buffer[..]), Ok(u16::MAX)));
+
+ let mut short_buffer = [0, 0];
+ assert!(u32::MAX.write_to_io(&mut short_buffer[..]).is_err());
+ assert_eq!(short_buffer, [255, 255]);
+ assert!(u32::read_from_io(&short_buffer[..]).is_err());
+ }
+
+ #[test]
+ fn test_try_from_bytes_try_read_from() {
+ assert_eq!(<bool as TryFromBytes>::try_read_from_bytes(&[0]), Ok(false));
+ assert_eq!(<bool as TryFromBytes>::try_read_from_bytes(&[1]), Ok(true));
+
+ assert_eq!(<bool as TryFromBytes>::try_read_from_prefix(&[0, 2]), Ok((false, &[2][..])));
+ assert_eq!(<bool as TryFromBytes>::try_read_from_prefix(&[1, 2]), Ok((true, &[2][..])));
+
+ assert_eq!(<bool as TryFromBytes>::try_read_from_suffix(&[2, 0]), Ok((&[2][..], false)));
+ assert_eq!(<bool as TryFromBytes>::try_read_from_suffix(&[2, 1]), Ok((&[2][..], true)));
+
+ // If we don't pass enough bytes, it fails.
+ assert!(matches!(
+ <u8 as TryFromBytes>::try_read_from_bytes(&[]),
+ Err(TryReadError::Size(_))
+ ));
+ assert!(matches!(
+ <u8 as TryFromBytes>::try_read_from_prefix(&[]),
+ Err(TryReadError::Size(_))
+ ));
+ assert!(matches!(
+ <u8 as TryFromBytes>::try_read_from_suffix(&[]),
+ Err(TryReadError::Size(_))
+ ));
+
+ // If we pass too many bytes, it fails.
+ assert!(matches!(
+ <u8 as TryFromBytes>::try_read_from_bytes(&[0, 0]),
+ Err(TryReadError::Size(_))
+ ));
+
+ // If we pass an invalid value, it fails.
+ assert!(matches!(
+ <bool as TryFromBytes>::try_read_from_bytes(&[2]),
+ Err(TryReadError::Validity(_))
+ ));
+ assert!(matches!(
+ <bool as TryFromBytes>::try_read_from_prefix(&[2, 0]),
+ Err(TryReadError::Validity(_))
+ ));
+ assert!(matches!(
+ <bool as TryFromBytes>::try_read_from_suffix(&[0, 2]),
+ Err(TryReadError::Validity(_))
+ ));
+
+ // Reading from a misaligned buffer should still succeed. Since `AU64`'s
+ // alignment is 8, and since we read from two adjacent addresses one
+ // byte apart, it is guaranteed that at least one of them (though
+ // possibly both) will be misaligned.
+ let bytes: [u8; 9] = [0, 0, 0, 0, 0, 0, 0, 0, 0];
+ assert_eq!(<AU64 as TryFromBytes>::try_read_from_bytes(&bytes[..8]), Ok(AU64(0)));
+ assert_eq!(<AU64 as TryFromBytes>::try_read_from_bytes(&bytes[1..9]), Ok(AU64(0)));
+
+ assert_eq!(
+ <AU64 as TryFromBytes>::try_read_from_prefix(&bytes[..8]),
+ Ok((AU64(0), &[][..]))
+ );
+ assert_eq!(
+ <AU64 as TryFromBytes>::try_read_from_prefix(&bytes[1..9]),
+ Ok((AU64(0), &[][..]))
+ );
+
+ assert_eq!(
+ <AU64 as TryFromBytes>::try_read_from_suffix(&bytes[..8]),
+ Ok((&[][..], AU64(0)))
+ );
+ assert_eq!(
+ <AU64 as TryFromBytes>::try_read_from_suffix(&bytes[1..9]),
+ Ok((&[][..], AU64(0)))
+ );
+ }
+
+ #[test]
+ fn test_ref_from_mut_from_bytes() {
+ // Test `FromBytes::{ref_from_bytes, mut_from_bytes}{,_prefix,Suffix}`
+ // success cases. Exhaustive coverage for these methods is covered by
+ // the `Ref` tests above, which these helper methods defer to.
+
+ let mut buf =
+ Align::<[u8; 16], AU64>::new([0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15]);
+
+ assert_eq!(
+ AU64::ref_from_bytes(&buf.t[8..]).unwrap().0.to_ne_bytes(),
+ [8, 9, 10, 11, 12, 13, 14, 15]
+ );
+ let suffix = AU64::mut_from_bytes(&mut buf.t[8..]).unwrap();
+ suffix.0 = 0x0101010101010101;
+ // The `[u8:9]` is a non-half size of the full buffer, which would catch
+ // `from_prefix` having the same implementation as `from_suffix` (issues #506, #511).
+ assert_eq!(
+ <[u8; 9]>::ref_from_suffix(&buf.t[..]).unwrap(),
+ (&[0, 1, 2, 3, 4, 5, 6][..], &[7u8, 1, 1, 1, 1, 1, 1, 1, 1])
+ );
+ let (prefix, suffix) = AU64::mut_from_suffix(&mut buf.t[1..]).unwrap();
+ assert_eq!(prefix, &mut [1u8, 2, 3, 4, 5, 6, 7][..]);
+ suffix.0 = 0x0202020202020202;
+ let (prefix, suffix) = <[u8; 10]>::mut_from_suffix(&mut buf.t[..]).unwrap();
+ assert_eq!(prefix, &mut [0u8, 1, 2, 3, 4, 5][..]);
+ suffix[0] = 42;
+ assert_eq!(
+ <[u8; 9]>::ref_from_prefix(&buf.t[..]).unwrap(),
+ (&[0u8, 1, 2, 3, 4, 5, 42, 7, 2], &[2u8, 2, 2, 2, 2, 2, 2][..])
+ );
+ <[u8; 2]>::mut_from_prefix(&mut buf.t[..]).unwrap().0[1] = 30;
+ assert_eq!(buf.t, [0, 30, 2, 3, 4, 5, 42, 7, 2, 2, 2, 2, 2, 2, 2, 2]);
+ }
+
+ #[test]
+ fn test_ref_from_mut_from_bytes_error() {
+ // Test `FromBytes::{ref_from_bytes, mut_from_bytes}{,_prefix,Suffix}`
+ // error cases.
+
+ // Fail because the buffer is too large.
+ let mut buf = Align::<[u8; 16], AU64>::default();
+ // `buf.t` should be aligned to 8, so only the length check should fail.
+ assert!(AU64::ref_from_bytes(&buf.t[..]).is_err());
+ assert!(AU64::mut_from_bytes(&mut buf.t[..]).is_err());
+ assert!(<[u8; 8]>::ref_from_bytes(&buf.t[..]).is_err());
+ assert!(<[u8; 8]>::mut_from_bytes(&mut buf.t[..]).is_err());
+
+ // Fail because the buffer is too small.
+ let mut buf = Align::<[u8; 4], AU64>::default();
+ assert!(AU64::ref_from_bytes(&buf.t[..]).is_err());
+ assert!(AU64::mut_from_bytes(&mut buf.t[..]).is_err());
+ assert!(<[u8; 8]>::ref_from_bytes(&buf.t[..]).is_err());
+ assert!(<[u8; 8]>::mut_from_bytes(&mut buf.t[..]).is_err());
+ assert!(AU64::ref_from_prefix(&buf.t[..]).is_err());
+ assert!(AU64::mut_from_prefix(&mut buf.t[..]).is_err());
+ assert!(AU64::ref_from_suffix(&buf.t[..]).is_err());
+ assert!(AU64::mut_from_suffix(&mut buf.t[..]).is_err());
+ assert!(<[u8; 8]>::ref_from_prefix(&buf.t[..]).is_err());
+ assert!(<[u8; 8]>::mut_from_prefix(&mut buf.t[..]).is_err());
+ assert!(<[u8; 8]>::ref_from_suffix(&buf.t[..]).is_err());
+ assert!(<[u8; 8]>::mut_from_suffix(&mut buf.t[..]).is_err());
+
+ // Fail because the alignment is insufficient.
+ let mut buf = Align::<[u8; 13], AU64>::default();
+ assert!(AU64::ref_from_bytes(&buf.t[1..]).is_err());
+ assert!(AU64::mut_from_bytes(&mut buf.t[1..]).is_err());
+ assert!(AU64::ref_from_bytes(&buf.t[1..]).is_err());
+ assert!(AU64::mut_from_bytes(&mut buf.t[1..]).is_err());
+ assert!(AU64::ref_from_prefix(&buf.t[1..]).is_err());
+ assert!(AU64::mut_from_prefix(&mut buf.t[1..]).is_err());
+ assert!(AU64::ref_from_suffix(&buf.t[..]).is_err());
+ assert!(AU64::mut_from_suffix(&mut buf.t[..]).is_err());
+ }
+
+ #[test]
+ fn test_to_methods() {
+ /// Run a series of tests by calling `IntoBytes` methods on `t`.
+ ///
+ /// `bytes` is the expected byte sequence returned from `t.as_bytes()`
+ /// before `t` has been modified. `post_mutation` is the expected
+ /// sequence returned from `t.as_bytes()` after `t.as_mut_bytes()[0]`
+ /// has had its bits flipped (by applying `^= 0xFF`).
+ ///
+ /// `N` is the size of `t` in bytes.
+ fn test<T: FromBytes + IntoBytes + Immutable + Debug + Eq + ?Sized, const N: usize>(
+ t: &mut T,
+ bytes: &[u8],
+ post_mutation: &T,
+ ) {
+ // Test that we can access the underlying bytes, and that we get the
+ // right bytes and the right number of bytes.
+ assert_eq!(t.as_bytes(), bytes);
+
+ // Test that changes to the underlying byte slices are reflected in
+ // the original object.
+ t.as_mut_bytes()[0] ^= 0xFF;
+ assert_eq!(t, post_mutation);
+ t.as_mut_bytes()[0] ^= 0xFF;
+
+ // `write_to` rejects slices that are too small or too large.
+ assert!(t.write_to(&mut vec![0; N - 1][..]).is_err());
+ assert!(t.write_to(&mut vec![0; N + 1][..]).is_err());
+
+ // `write_to` works as expected.
+ let mut bytes = [0; N];
+ assert_eq!(t.write_to(&mut bytes[..]), Ok(()));
+ assert_eq!(bytes, t.as_bytes());
+
+ // `write_to_prefix` rejects slices that are too small.
+ assert!(t.write_to_prefix(&mut vec![0; N - 1][..]).is_err());
+
+ // `write_to_prefix` works with exact-sized slices.
+ let mut bytes = [0; N];
+ assert_eq!(t.write_to_prefix(&mut bytes[..]), Ok(()));
+ assert_eq!(bytes, t.as_bytes());
+
+ // `write_to_prefix` works with too-large slices, and any bytes past
+ // the prefix aren't modified.
+ let mut too_many_bytes = vec![0; N + 1];
+ too_many_bytes[N] = 123;
+ assert_eq!(t.write_to_prefix(&mut too_many_bytes[..]), Ok(()));
+ assert_eq!(&too_many_bytes[..N], t.as_bytes());
+ assert_eq!(too_many_bytes[N], 123);
+
+ // `write_to_suffix` rejects slices that are too small.
+ assert!(t.write_to_suffix(&mut vec![0; N - 1][..]).is_err());
+
+ // `write_to_suffix` works with exact-sized slices.
+ let mut bytes = [0; N];
+ assert_eq!(t.write_to_suffix(&mut bytes[..]), Ok(()));
+ assert_eq!(bytes, t.as_bytes());
+
+ // `write_to_suffix` works with too-large slices, and any bytes
+ // before the suffix aren't modified.
+ let mut too_many_bytes = vec![0; N + 1];
+ too_many_bytes[0] = 123;
+ assert_eq!(t.write_to_suffix(&mut too_many_bytes[..]), Ok(()));
+ assert_eq!(&too_many_bytes[1..], t.as_bytes());
+ assert_eq!(too_many_bytes[0], 123);
+ }
+
+ #[derive(Debug, Eq, PartialEq, FromBytes, IntoBytes, Immutable)]
+ #[repr(C)]
+ struct Foo {
+ a: u32,
+ b: Wrapping<u32>,
+ c: Option<NonZeroU32>,
+ }
+
+ let expected_bytes: Vec<u8> = if cfg!(target_endian = "little") {
+ vec![1, 0, 0, 0, 2, 0, 0, 0, 0, 0, 0, 0]
+ } else {
+ vec![0, 0, 0, 1, 0, 0, 0, 2, 0, 0, 0, 0]
+ };
+ let post_mutation_expected_a =
+ if cfg!(target_endian = "little") { 0x00_00_00_FE } else { 0xFF_00_00_01 };
+ test::<_, 12>(
+ &mut Foo { a: 1, b: Wrapping(2), c: None },
+ expected_bytes.as_bytes(),
+ &Foo { a: post_mutation_expected_a, b: Wrapping(2), c: None },
+ );
+ test::<_, 3>(
+ Unsized::from_mut_slice(&mut [1, 2, 3]),
+ &[1, 2, 3],
+ Unsized::from_mut_slice(&mut [0xFE, 2, 3]),
+ );
+ }
+
+ #[test]
+ fn test_array() {
+ #[derive(FromBytes, IntoBytes, Immutable)]
+ #[repr(C)]
+ struct Foo {
+ a: [u16; 33],
+ }
+
+ let foo = Foo { a: [0xFFFF; 33] };
+ let expected = [0xFFu8; 66];
+ assert_eq!(foo.as_bytes(), &expected[..]);
+ }
+
+ #[test]
+ fn test_new_zeroed() {
+ assert!(!bool::new_zeroed());
+ assert_eq!(u64::new_zeroed(), 0);
+ // This test exists in order to exercise unsafe code, especially when
+ // running under Miri.
+ #[allow(clippy::unit_cmp)]
+ {
+ assert_eq!(<()>::new_zeroed(), ());
+ }
+ }
+
+ #[test]
+ fn test_transparent_packed_generic_struct() {
+ #[derive(IntoBytes, FromBytes, Unaligned)]
+ #[repr(transparent)]
+ #[allow(dead_code)] // We never construct this type
+ struct Foo<T> {
+ _t: T,
+ _phantom: PhantomData<()>,
+ }
+
+ assert_impl_all!(Foo<u32>: FromZeros, FromBytes, IntoBytes);
+ assert_impl_all!(Foo<u8>: Unaligned);
+
+ #[derive(IntoBytes, FromBytes, Unaligned)]
+ #[repr(C, packed)]
+ #[allow(dead_code)] // We never construct this type
+ struct Bar<T, U> {
+ _t: T,
+ _u: U,
+ }
+
+ assert_impl_all!(Bar<u8, AU64>: FromZeros, FromBytes, IntoBytes, Unaligned);
+ }
+
+ #[cfg(feature = "alloc")]
+ mod alloc {
+ use super::*;
+
+ #[cfg(not(no_zerocopy_panic_in_const_and_vec_try_reserve_1_57_0))]
+ #[test]
+ fn test_extend_vec_zeroed() {
+ // Test extending when there is an existing allocation.
+ let mut v = vec![100u16, 200, 300];
+ FromZeros::extend_vec_zeroed(&mut v, 3).unwrap();
+ assert_eq!(v.len(), 6);
+ assert_eq!(&*v, &[100, 200, 300, 0, 0, 0]);
+ drop(v);
+
+ // Test extending when there is no existing allocation.
+ let mut v: Vec<u64> = Vec::new();
+ FromZeros::extend_vec_zeroed(&mut v, 3).unwrap();
+ assert_eq!(v.len(), 3);
+ assert_eq!(&*v, &[0, 0, 0]);
+ drop(v);
+ }
+
+ #[cfg(not(no_zerocopy_panic_in_const_and_vec_try_reserve_1_57_0))]
+ #[test]
+ fn test_extend_vec_zeroed_zst() {
+ // Test extending when there is an existing (fake) allocation.
+ let mut v = vec![(), (), ()];
+ <()>::extend_vec_zeroed(&mut v, 3).unwrap();
+ assert_eq!(v.len(), 6);
+ assert_eq!(&*v, &[(), (), (), (), (), ()]);
+ drop(v);
+
+ // Test extending when there is no existing (fake) allocation.
+ let mut v: Vec<()> = Vec::new();
+ <()>::extend_vec_zeroed(&mut v, 3).unwrap();
+ assert_eq!(&*v, &[(), (), ()]);
+ drop(v);
+ }
+
+ #[cfg(not(no_zerocopy_panic_in_const_and_vec_try_reserve_1_57_0))]
+ #[test]
+ fn test_insert_vec_zeroed() {
+ // Insert at start (no existing allocation).
+ let mut v: Vec<u64> = Vec::new();
+ u64::insert_vec_zeroed(&mut v, 0, 2).unwrap();
+ assert_eq!(v.len(), 2);
+ assert_eq!(&*v, &[0, 0]);
+ drop(v);
+
+ // Insert at start.
+ let mut v = vec![100u64, 200, 300];
+ u64::insert_vec_zeroed(&mut v, 0, 2).unwrap();
+ assert_eq!(v.len(), 5);
+ assert_eq!(&*v, &[0, 0, 100, 200, 300]);
+ drop(v);
+
+ // Insert at middle.
+ let mut v = vec![100u64, 200, 300];
+ u64::insert_vec_zeroed(&mut v, 1, 1).unwrap();
+ assert_eq!(v.len(), 4);
+ assert_eq!(&*v, &[100, 0, 200, 300]);
+ drop(v);
+
+ // Insert at end.
+ let mut v = vec![100u64, 200, 300];
+ u64::insert_vec_zeroed(&mut v, 3, 1).unwrap();
+ assert_eq!(v.len(), 4);
+ assert_eq!(&*v, &[100, 200, 300, 0]);
+ drop(v);
+ }
+
+ #[cfg(not(no_zerocopy_panic_in_const_and_vec_try_reserve_1_57_0))]
+ #[test]
+ fn test_insert_vec_zeroed_zst() {
+ // Insert at start (no existing fake allocation).
+ let mut v: Vec<()> = Vec::new();
+ <()>::insert_vec_zeroed(&mut v, 0, 2).unwrap();
+ assert_eq!(v.len(), 2);
+ assert_eq!(&*v, &[(), ()]);
+ drop(v);
+
+ // Insert at start.
+ let mut v = vec![(), (), ()];
+ <()>::insert_vec_zeroed(&mut v, 0, 2).unwrap();
+ assert_eq!(v.len(), 5);
+ assert_eq!(&*v, &[(), (), (), (), ()]);
+ drop(v);
+
+ // Insert at middle.
+ let mut v = vec![(), (), ()];
+ <()>::insert_vec_zeroed(&mut v, 1, 1).unwrap();
+ assert_eq!(v.len(), 4);
+ assert_eq!(&*v, &[(), (), (), ()]);
+ drop(v);
+
+ // Insert at end.
+ let mut v = vec![(), (), ()];
+ <()>::insert_vec_zeroed(&mut v, 3, 1).unwrap();
+ assert_eq!(v.len(), 4);
+ assert_eq!(&*v, &[(), (), (), ()]);
+ drop(v);
+ }
+
+ #[test]
+ fn test_new_box_zeroed() {
+ assert_eq!(u64::new_box_zeroed(), Ok(Box::new(0)));
+ }
+
+ #[test]
+ fn test_new_box_zeroed_array() {
+ drop(<[u32; 0x1000]>::new_box_zeroed());
+ }
+
+ #[test]
+ fn test_new_box_zeroed_zst() {
+ // This test exists in order to exercise unsafe code, especially
+ // when running under Miri.
+ #[allow(clippy::unit_cmp)]
+ {
+ assert_eq!(<()>::new_box_zeroed(), Ok(Box::new(())));
+ }
+ }
+
+ #[test]
+ fn test_new_box_zeroed_with_elems() {
+ let mut s: Box<[u64]> = <[u64]>::new_box_zeroed_with_elems(3).unwrap();
+ assert_eq!(s.len(), 3);
+ assert_eq!(&*s, &[0, 0, 0]);
+ s[1] = 3;
+ assert_eq!(&*s, &[0, 3, 0]);
+ }
+
+ #[test]
+ fn test_new_box_zeroed_with_elems_empty() {
+ let s: Box<[u64]> = <[u64]>::new_box_zeroed_with_elems(0).unwrap();
+ assert_eq!(s.len(), 0);
+ }
+
+ #[test]
+ fn test_new_box_zeroed_with_elems_zst() {
+ let mut s: Box<[()]> = <[()]>::new_box_zeroed_with_elems(3).unwrap();
+ assert_eq!(s.len(), 3);
+ assert!(s.get(10).is_none());
+ // This test exists in order to exercise unsafe code, especially
+ // when running under Miri.
+ #[allow(clippy::unit_cmp)]
+ {
+ assert_eq!(s[1], ());
+ }
+ s[2] = ();
+ }
+
+ #[test]
+ fn test_new_box_zeroed_with_elems_zst_empty() {
+ let s: Box<[()]> = <[()]>::new_box_zeroed_with_elems(0).unwrap();
+ assert_eq!(s.len(), 0);
+ }
+
+ #[test]
+ fn new_box_zeroed_with_elems_errors() {
+ assert_eq!(<[u16]>::new_box_zeroed_with_elems(usize::MAX), Err(AllocError));
+
+ let max = <usize as core::convert::TryFrom<_>>::try_from(isize::MAX).unwrap();
+ assert_eq!(
+ <[u16]>::new_box_zeroed_with_elems((max / mem::size_of::<u16>()) + 1),
+ Err(AllocError)
+ );
+ }
+ }
+
+ #[test]
+ #[allow(deprecated)]
+ fn test_deprecated_from_bytes() {
+ let val = 0u32;
+ let bytes = val.as_bytes();
+
+ assert!(u32::ref_from(bytes).is_some());
+ // mut_from needs mut bytes
+ let mut val = 0u32;
+ let mut_bytes = val.as_mut_bytes();
+ assert!(u32::mut_from(mut_bytes).is_some());
+
+ assert!(u32::read_from(bytes).is_some());
+
+ let (slc, rest) = <u32>::slice_from_prefix(bytes, 0).unwrap();
+ assert!(slc.is_empty());
+ assert_eq!(rest.len(), 4);
+
+ let (rest, slc) = <u32>::slice_from_suffix(bytes, 0).unwrap();
+ assert!(slc.is_empty());
+ assert_eq!(rest.len(), 4);
+
+ let (slc, rest) = <u32>::mut_slice_from_prefix(mut_bytes, 0).unwrap();
+ assert!(slc.is_empty());
+ assert_eq!(rest.len(), 4);
+
+ let (rest, slc) = <u32>::mut_slice_from_suffix(mut_bytes, 0).unwrap();
+ assert!(slc.is_empty());
+ assert_eq!(rest.len(), 4);
+ }
+
+ #[test]
+ fn test_try_ref_from_prefix_suffix() {
+ use crate::util::testutil::Align;
+ let bytes = &Align::<[u8; 4], u32>::new([0u8; 4]).t[..];
+ let (r, rest): (&u32, &[u8]) = u32::try_ref_from_prefix(bytes).unwrap();
+ assert_eq!(*r, 0);
+ assert_eq!(rest.len(), 0);
+
+ let (rest, r): (&[u8], &u32) = u32::try_ref_from_suffix(bytes).unwrap();
+ assert_eq!(*r, 0);
+ assert_eq!(rest.len(), 0);
+ }
+
+ #[test]
+ fn test_raw_dangling() {
+ use crate::util::AsAddress;
+ let ptr: NonNull<u32> = u32::raw_dangling();
+ assert_eq!(AsAddress::addr(ptr), 1);
+
+ let ptr: NonNull<[u32]> = <[u32]>::raw_dangling();
+ assert_eq!(AsAddress::addr(ptr), 1);
+ }
+
+ #[test]
+ fn test_try_ref_from_prefix_with_elems() {
+ use crate::util::testutil::Align;
+ let bytes = &Align::<[u8; 8], u32>::new([0u8; 8]).t[..];
+ let (r, rest): (&[u32], &[u8]) = <[u32]>::try_ref_from_prefix_with_elems(bytes, 2).unwrap();
+ assert_eq!(r.len(), 2);
+ assert_eq!(rest.len(), 0);
+ }
+
+ #[test]
+ fn test_try_ref_from_suffix_with_elems() {
+ use crate::util::testutil::Align;
+ let bytes = &Align::<[u8; 8], u32>::new([0u8; 8]).t[..];
+ let (rest, r): (&[u8], &[u32]) = <[u32]>::try_ref_from_suffix_with_elems(bytes, 2).unwrap();
+ assert_eq!(r.len(), 2);
+ assert_eq!(rest.len(), 0);
+ }
+}
diff --git a/rust/zerocopy/src/macros.rs b/rust/zerocopy/src/macros.rs
new file mode 100644
index 0000000..b801d86a
--- /dev/null
+++ b/rust/zerocopy/src/macros.rs
@@ -0,0 +1,1825 @@
+// SPDX-License-Identifier: (BSD-2-Clause OR Apache-2.0) OR MIT
+
+// Copyright 2024 The Fuchsia Authors
+//
+// Licensed under the 2-Clause BSD License <LICENSE-BSD or
+// https://opensource.org/license/bsd-2-clause>, Apache License, Version 2.0
+// <LICENSE-APACHE or https://www.apache.org/licenses/LICENSE-2.0>, or the MIT
+// license <LICENSE-MIT or https://opensource.org/licenses/MIT>, at your option.
+// This file may not be copied, modified, or distributed except according to
+// those terms.
+
+/// Safely transmutes a value of one type to a value of another type of the same
+/// size.
+///
+/// This macro behaves like an invocation of this function:
+///
+/// ```ignore
+/// const fn transmute<Src, Dst>(src: Src) -> Dst
+/// where
+/// Src: IntoBytes,
+/// Dst: FromBytes,
+/// size_of::<Src>() == size_of::<Dst>(),
+/// {
+/// # /*
+/// ...
+/// # */
+/// }
+/// ```
+///
+/// However, unlike a function, this macro can only be invoked when the types of
+/// `Src` and `Dst` are completely concrete. The types `Src` and `Dst` are
+/// inferred from the calling context; they cannot be explicitly specified in
+/// the macro invocation.
+///
+/// Note that the `Src` produced by the expression `$e` will *not* be dropped.
+/// Semantically, its bits will be copied into a new value of type `Dst`, the
+/// original `Src` will be forgotten, and the value of type `Dst` will be
+/// returned.
+///
+/// # `#![allow(shrink)]`
+///
+/// If `#![allow(shrink)]` is provided, `transmute!` additionally supports
+/// transmutations that shrink the size of the value; e.g.:
+///
+/// ```
+/// # use zerocopy::transmute;
+/// let u: u32 = transmute!(#![allow(shrink)] 0u64);
+/// assert_eq!(u, 0u32);
+/// ```
+///
+/// # Examples
+///
+/// ```
+/// # use zerocopy::transmute;
+/// let one_dimensional: [u8; 8] = [0, 1, 2, 3, 4, 5, 6, 7];
+///
+/// let two_dimensional: [[u8; 4]; 2] = transmute!(one_dimensional);
+///
+/// assert_eq!(two_dimensional, [[0, 1, 2, 3], [4, 5, 6, 7]]);
+/// ```
+///
+/// # Use in `const` contexts
+///
+/// This macro can be invoked in `const` contexts.
+///
+#[doc = codegen_section!(
+ header = "h2",
+ bench = "transmute",
+ format = "coco_static_size",
+)]
+#[macro_export]
+macro_rules! transmute {
+ // NOTE: This must be a macro (rather than a function with trait bounds)
+ // because there's no way, in a generic context, to enforce that two types
+ // have the same size. `core::mem::transmute` uses compiler magic to enforce
+ // this so long as the types are concrete.
+ (#![allow(shrink)] $e:expr) => {{
+ let mut e = $e;
+ if false {
+ // This branch, though never taken, ensures that the type of `e` is
+ // `IntoBytes` and that the type of the outer macro invocation
+ // expression is `FromBytes`.
+
+ fn transmute<Src, Dst>(src: Src) -> Dst
+ where
+ Src: $crate::IntoBytes,
+ Dst: $crate::FromBytes,
+ {
+ let _ = src;
+ loop {}
+ }
+ loop {}
+ #[allow(unreachable_code)]
+ transmute(e)
+ } else {
+ use $crate::util::macro_util::core_reexport::mem::ManuallyDrop;
+
+ // NOTE: `repr(packed)` is important! It ensures that the size of
+ // `Transmute` won't be rounded up to accommodate `Src`'s or `Dst`'s
+ // alignment, which would break the size comparison logic below.
+ //
+ // As an example of why this is problematic, consider `Src = [u8;
+ // 5]`, `Dst = u32`. The total size of `Transmute<Src, Dst>` would
+ // be 8, and so we would reject a `[u8; 5]` to `u32` transmute as
+ // being size-increasing, which it isn't.
+ #[repr(C, packed)]
+ union Transmute<Src, Dst> {
+ src: ManuallyDrop<Src>,
+ dst: ManuallyDrop<Dst>,
+ }
+
+ // SAFETY: `Transmute` is a `repr(C)` union whose `src` field has
+ // type `ManuallyDrop<Src>`. Thus, the `src` field starts at byte
+ // offset 0 within `Transmute` [1]. `ManuallyDrop<T>` has the same
+ // layout and bit validity as `T`, so it is sound to transmute `Src`
+ // to `Transmute`.
+ //
+ // [1] https://doc.rust-lang.org/1.85.0/reference/type-layout.html#reprc-unions
+ //
+ // [2] Per https://doc.rust-lang.org/1.85.0/std/mem/struct.ManuallyDrop.html:
+ //
+ // `ManuallyDrop<T>` is guaranteed to have the same layout and bit
+ // validity as `T`
+ let u: Transmute<_, _> = unsafe {
+ // Clippy: We can't annotate the types; this macro is designed
+ // to infer the types from the calling context.
+ #[allow(clippy::missing_transmute_annotations)]
+ $crate::util::macro_util::core_reexport::mem::transmute(e)
+ };
+
+ if false {
+ // SAFETY: This code is never executed.
+ e = ManuallyDrop::into_inner(unsafe { u.src });
+ // Suppress the `unused_assignments` lint on the previous line.
+ let _ = e;
+ loop {}
+ } else {
+ // SAFETY: Per the safety comment on `let u` above, the `dst`
+ // field in `Transmute` starts at byte offset 0, and has the
+ // same layout and bit validity as `Dst`.
+ //
+ // Transmuting `Src` to `Transmute<Src, Dst>` above using
+ // `core::mem::transmute` ensures that `size_of::<Src>() ==
+ // size_of::<Transmute<Src, Dst>>()`. A `#[repr(C, packed)]`
+ // union has the maximum size of all of its fields [1], so this
+ // is equivalent to `size_of::<Src>() >= size_of::<Dst>()`.
+ //
+ // The outer `if`'s `false` branch ensures that `Src: IntoBytes`
+ // and `Dst: FromBytes`. This, combined with the size bound,
+ // ensures that this transmute is sound.
+ //
+ // [1] Per https://doc.rust-lang.org/1.85.0/reference/type-layout.html#reprc-unions:
+ //
+ // The union will have a size of the maximum size of all of
+ // its fields rounded to its alignment
+ let dst = unsafe { u.dst };
+ $crate::util::macro_util::must_use(ManuallyDrop::into_inner(dst))
+ }
+ }
+ }};
+ ($e:expr) => {{
+ let e = $e;
+ if false {
+ // This branch, though never taken, ensures that the type of `e` is
+ // `IntoBytes` and that the type of the outer macro invocation
+ // expression is `FromBytes`.
+
+ fn transmute<Src, Dst>(src: Src) -> Dst
+ where
+ Src: $crate::IntoBytes,
+ Dst: $crate::FromBytes,
+ {
+ let _ = src;
+ loop {}
+ }
+ loop {}
+ #[allow(unreachable_code)]
+ transmute(e)
+ } else {
+ // SAFETY: `core::mem::transmute` ensures that the type of `e` and
+ // the type of this macro invocation expression have the same size.
+ // We know this transmute is safe thanks to the `IntoBytes` and
+ // `FromBytes` bounds enforced by the `false` branch.
+ let u = unsafe {
+ // Clippy: We can't annotate the types; this macro is designed
+ // to infer the types from the calling context.
+ #[allow(clippy::missing_transmute_annotations, unnecessary_transmutes)]
+ $crate::util::macro_util::core_reexport::mem::transmute(e)
+ };
+ $crate::util::macro_util::must_use(u)
+ }
+ }};
+}
+
+/// Safely transmutes a mutable or immutable reference of one type to an
+/// immutable reference of another type of the same size and compatible
+/// alignment.
+///
+/// This macro behaves like an invocation of this function:
+///
+/// ```ignore
+/// fn transmute_ref<'src, 'dst, Src, Dst>(src: &'src Src) -> &'dst Dst
+/// where
+/// 'src: 'dst,
+/// Src: IntoBytes + Immutable + ?Sized,
+/// Dst: FromBytes + Immutable + ?Sized,
+/// align_of::<Src>() >= align_of::<Dst>(),
+/// size_compatible::<Src, Dst>(),
+/// {
+/// # /*
+/// ...
+/// # */
+/// }
+/// ```
+///
+/// The types `Src` and `Dst` are inferred from the calling context; they cannot
+/// be explicitly specified in the macro invocation.
+///
+/// # Size compatibility
+///
+/// `transmute_ref!` supports transmuting between `Sized` types, between unsized
+/// (i.e., `?Sized`) types, and from a `Sized` type to an unsized type. It
+/// supports any transmutation that preserves the number of bytes of the
+/// referent, even if doing so requires updating the metadata stored in an
+/// unsized "fat" reference:
+///
+/// ```
+/// # use zerocopy::transmute_ref;
+/// # use core::mem::size_of_val; // Not in the prelude on our MSRV
+/// let src: &[[u8; 2]] = &[[0, 1], [2, 3]][..];
+/// let dst: &[u8] = transmute_ref!(src);
+///
+/// assert_eq!(src.len(), 2);
+/// assert_eq!(dst.len(), 4);
+/// assert_eq!(dst, [0, 1, 2, 3]);
+/// assert_eq!(size_of_val(src), size_of_val(dst));
+/// ```
+///
+/// # Errors
+///
+/// Violations of the alignment and size compatibility checks are detected
+/// *after* the compiler performs monomorphization. This has two important
+/// consequences.
+///
+/// First, it means that generic code will *never* fail these conditions:
+///
+/// ```
+/// # use zerocopy::{transmute_ref, FromBytes, IntoBytes, Immutable};
+/// fn transmute_ref<Src, Dst>(src: &Src) -> &Dst
+/// where
+/// Src: IntoBytes + Immutable,
+/// Dst: FromBytes + Immutable,
+/// {
+/// transmute_ref!(src)
+/// }
+/// ```
+///
+/// Instead, failures will only be detected once generic code is instantiated
+/// with concrete types:
+///
+/// ```compile_fail,E0080
+/// # use zerocopy::{transmute_ref, FromBytes, IntoBytes, Immutable};
+/// #
+/// # fn transmute_ref<Src, Dst>(src: &Src) -> &Dst
+/// # where
+/// # Src: IntoBytes + Immutable,
+/// # Dst: FromBytes + Immutable,
+/// # {
+/// # transmute_ref!(src)
+/// # }
+/// let src: &u16 = &0;
+/// let dst: &u8 = transmute_ref(src);
+/// ```
+///
+/// Second, the fact that violations are detected after monomorphization means
+/// that `cargo check` will usually not detect errors, even when types are
+/// concrete. Instead, `cargo build` must be used to detect such errors.
+///
+/// # Examples
+///
+/// Transmuting between `Sized` types:
+///
+/// ```
+/// # use zerocopy::transmute_ref;
+/// let one_dimensional: [u8; 8] = [0, 1, 2, 3, 4, 5, 6, 7];
+///
+/// let two_dimensional: &[[u8; 4]; 2] = transmute_ref!(&one_dimensional);
+///
+/// assert_eq!(two_dimensional, &[[0, 1, 2, 3], [4, 5, 6, 7]]);
+/// ```
+///
+/// Transmuting between unsized types:
+///
+/// ```
+/// # use {zerocopy::*, zerocopy_derive::*};
+/// # type u16 = zerocopy::byteorder::native_endian::U16;
+/// # type u32 = zerocopy::byteorder::native_endian::U32;
+/// #[derive(KnownLayout, FromBytes, IntoBytes, Immutable)]
+/// #[repr(C)]
+/// struct SliceDst<T, U> {
+/// t: T,
+/// u: [U],
+/// }
+///
+/// type Src = SliceDst<u32, u16>;
+/// type Dst = SliceDst<u16, u8>;
+///
+/// let src = Src::ref_from_bytes(&[0, 1, 2, 3, 4, 5, 6, 7]).unwrap();
+/// let dst: &Dst = transmute_ref!(src);
+///
+/// assert_eq!(src.t.as_bytes(), [0, 1, 2, 3]);
+/// assert_eq!(src.u.len(), 2);
+/// assert_eq!(src.u.as_bytes(), [4, 5, 6, 7]);
+///
+/// assert_eq!(dst.t.as_bytes(), [0, 1]);
+/// assert_eq!(dst.u, [2, 3, 4, 5, 6, 7]);
+/// ```
+///
+/// # Use in `const` contexts
+///
+/// This macro can be invoked in `const` contexts only when `Src: Sized` and
+/// `Dst: Sized`.
+///
+#[doc = codegen_section!(
+ header = "h2",
+ bench = "transmute_ref",
+ format = "coco",
+ arity = 2,
+ [
+ open
+ @index 1
+ @title "Sized"
+ @variant "static_size"
+ ],
+ [
+ @index 2
+ @title "Unsized"
+ @variant "dynamic_size"
+ ]
+)]
+#[macro_export]
+macro_rules! transmute_ref {
+ ($e:expr) => {{
+ // NOTE: This must be a macro (rather than a function with trait bounds)
+ // because there's no way, in a generic context, to enforce that two
+ // types have the same size or alignment.
+
+ // Ensure that the source type is a reference or a mutable reference
+ // (note that mutable references are implicitly reborrowed here).
+ let e: &_ = $e;
+
+ #[allow(unused, clippy::diverging_sub_expression)]
+ if false {
+ // This branch, though never taken, ensures that the type of `e` is
+ // `&T` where `T: IntoBytes + Immutable`, and that the type of this
+ // macro expression is `&U` where `U: FromBytes + Immutable`.
+
+ struct AssertSrcIsIntoBytes<'a, T: ?::core::marker::Sized + $crate::IntoBytes>(&'a T);
+ struct AssertSrcIsImmutable<'a, T: ?::core::marker::Sized + $crate::Immutable>(&'a T);
+ struct AssertDstIsFromBytes<'a, U: ?::core::marker::Sized + $crate::FromBytes>(&'a U);
+ struct AssertDstIsImmutable<'a, T: ?::core::marker::Sized + $crate::Immutable>(&'a T);
+
+ let _ = AssertSrcIsIntoBytes(e);
+ let _ = AssertSrcIsImmutable(e);
+
+ if true {
+ #[allow(unused, unreachable_code)]
+ let u = AssertDstIsFromBytes(loop {});
+ u.0
+ } else {
+ #[allow(unused, unreachable_code)]
+ let u = AssertDstIsImmutable(loop {});
+ u.0
+ }
+ } else {
+ use $crate::util::macro_util::TransmuteRefDst;
+ let t = $crate::util::macro_util::Wrap::new(e);
+
+ if false {
+ // This branch exists solely to force the compiler to infer the
+ // type of `Dst` *before* it attempts to resolve the method call
+ // to `transmute_ref` in the `else` branch.
+ //
+ // Without this, if `Src` is `Sized` but `Dst` is `!Sized`, the
+ // compiler will eagerly select the inherent impl of
+ // `transmute_ref` (which requires `Dst: Sized`) because inherent
+ // methods take priority over trait methods. It does this before
+ // it realizes `Dst` is `!Sized`, leading to a compile error when
+ // it checks the bounds later.
+ //
+ // By calling this helper (which returns `&Dst`), we force `Dst`
+ // to be fully resolved. By the time it gets to the `else`
+ // branch, the compiler knows `Dst` is `!Sized`, properly
+ // disqualifies the inherent method, and falls back to the trait
+ // implementation.
+ t.transmute_ref_inference_helper()
+ } else {
+ // SAFETY: The outer `if false` branch ensures that:
+ // - `Src: IntoBytes + Immutable`
+ // - `Dst: FromBytes + Immutable`
+ unsafe {
+ t.transmute_ref()
+ }
+ }
+ }
+ }}
+}
+
+/// Safely transmutes a mutable reference of one type to a mutable reference of
+/// another type of the same size and compatible alignment.
+///
+/// This macro behaves like an invocation of this function:
+///
+/// ```ignore
+/// const fn transmute_mut<'src, 'dst, Src, Dst>(src: &'src mut Src) -> &'dst mut Dst
+/// where
+/// 'src: 'dst,
+/// Src: FromBytes + IntoBytes + ?Sized,
+/// Dst: FromBytes + IntoBytes + ?Sized,
+/// align_of::<Src>() >= align_of::<Dst>(),
+/// size_compatible::<Src, Dst>(),
+/// {
+/// # /*
+/// ...
+/// # */
+/// }
+/// ```
+///
+/// The types `Src` and `Dst` are inferred from the calling context; they cannot
+/// be explicitly specified in the macro invocation.
+///
+/// # Size compatibility
+///
+/// `transmute_mut!` supports transmuting between `Sized` types, between unsized
+/// (i.e., `?Sized`) types, and from a `Sized` type to an unsized type. It
+/// supports any transmutation that preserves the number of bytes of the
+/// referent, even if doing so requires updating the metadata stored in an
+/// unsized "fat" reference:
+///
+/// ```
+/// # use zerocopy::transmute_mut;
+/// # use core::mem::size_of_val; // Not in the prelude on our MSRV
+/// let src: &mut [[u8; 2]] = &mut [[0, 1], [2, 3]][..];
+/// let dst: &mut [u8] = transmute_mut!(src);
+///
+/// assert_eq!(dst.len(), 4);
+/// assert_eq!(dst, [0, 1, 2, 3]);
+/// let dst_size = size_of_val(dst);
+/// assert_eq!(src.len(), 2);
+/// assert_eq!(size_of_val(src), dst_size);
+/// ```
+///
+/// # Errors
+///
+/// Violations of the alignment and size compatibility checks are detected
+/// *after* the compiler performs monomorphization. This has two important
+/// consequences.
+///
+/// First, it means that generic code will *never* fail these conditions:
+///
+/// ```
+/// # use zerocopy::{transmute_mut, FromBytes, IntoBytes, Immutable};
+/// fn transmute_mut<Src, Dst>(src: &mut Src) -> &mut Dst
+/// where
+/// Src: FromBytes + IntoBytes,
+/// Dst: FromBytes + IntoBytes,
+/// {
+/// transmute_mut!(src)
+/// }
+/// ```
+///
+/// Instead, failures will only be detected once generic code is instantiated
+/// with concrete types:
+///
+/// ```compile_fail,E0080
+/// # use zerocopy::{transmute_mut, FromBytes, IntoBytes, Immutable};
+/// #
+/// # fn transmute_mut<Src, Dst>(src: &mut Src) -> &mut Dst
+/// # where
+/// # Src: FromBytes + IntoBytes,
+/// # Dst: FromBytes + IntoBytes,
+/// # {
+/// # transmute_mut!(src)
+/// # }
+/// let src: &mut u16 = &mut 0;
+/// let dst: &mut u8 = transmute_mut(src);
+/// ```
+///
+/// Second, the fact that violations are detected after monomorphization means
+/// that `cargo check` will usually not detect errors, even when types are
+/// concrete. Instead, `cargo build` must be used to detect such errors.
+///
+///
+/// # Examples
+///
+/// Transmuting between `Sized` types:
+///
+/// ```
+/// # use zerocopy::transmute_mut;
+/// let mut one_dimensional: [u8; 8] = [0, 1, 2, 3, 4, 5, 6, 7];
+///
+/// let two_dimensional: &mut [[u8; 4]; 2] = transmute_mut!(&mut one_dimensional);
+///
+/// assert_eq!(two_dimensional, &[[0, 1, 2, 3], [4, 5, 6, 7]]);
+///
+/// two_dimensional.reverse();
+///
+/// assert_eq!(one_dimensional, [4, 5, 6, 7, 0, 1, 2, 3]);
+/// ```
+///
+/// Transmuting between unsized types:
+///
+/// ```
+/// # use {zerocopy::*, zerocopy_derive::*};
+/// # type u16 = zerocopy::byteorder::native_endian::U16;
+/// # type u32 = zerocopy::byteorder::native_endian::U32;
+/// #[derive(KnownLayout, FromBytes, IntoBytes, Immutable)]
+/// #[repr(C)]
+/// struct SliceDst<T, U> {
+/// t: T,
+/// u: [U],
+/// }
+///
+/// type Src = SliceDst<u32, u16>;
+/// type Dst = SliceDst<u16, u8>;
+///
+/// let mut bytes = [0, 1, 2, 3, 4, 5, 6, 7];
+/// let src = Src::mut_from_bytes(&mut bytes[..]).unwrap();
+/// let dst: &mut Dst = transmute_mut!(src);
+///
+/// assert_eq!(dst.t.as_bytes(), [0, 1]);
+/// assert_eq!(dst.u, [2, 3, 4, 5, 6, 7]);
+///
+/// assert_eq!(src.t.as_bytes(), [0, 1, 2, 3]);
+/// assert_eq!(src.u.len(), 2);
+/// assert_eq!(src.u.as_bytes(), [4, 5, 6, 7]);
+/// ```
+#[macro_export]
+macro_rules! transmute_mut {
+ ($e:expr) => {{
+ // NOTE: This must be a macro (rather than a function with trait bounds)
+ // because, for backwards-compatibility on v0.8.x, we use the autoref
+ // specialization trick to dispatch to different `transmute_mut`
+ // implementations: one which doesn't require `Src: KnownLayout + Dst:
+ // KnownLayout` when `Src: Sized + Dst: Sized`, and one which requires
+ // `KnownLayout` bounds otherwise.
+
+ // Ensure that the source type is a mutable reference.
+ let e: &mut _ = $e;
+
+ #[allow(unused)]
+ use $crate::util::macro_util::TransmuteMutDst as _;
+ let t = $crate::util::macro_util::Wrap::new(e);
+ if false {
+ // This branch exists solely to force the compiler to infer the type
+ // of `Dst` *before* it attempts to resolve the method call to
+ // `transmute_mut` in the `else` branch.
+ //
+ // Without this, if `Src` is `Sized` but `Dst` is `!Sized`, the
+ // compiler will eagerly select the inherent impl of `transmute_mut`
+ // (which requires `Dst: Sized`) because inherent methods take
+ // priority over trait methods. It does this before it realizes
+ // `Dst` is `!Sized`, leading to a compile error when it checks the
+ // bounds later.
+ //
+ // By calling this helper (which returns `&mut Dst`), we force `Dst`
+ // to be fully resolved. By the time it gets to the `else` branch,
+ // the compiler knows `Dst` is `!Sized`, properly disqualifies the
+ // inherent method, and falls back to the trait implementation.
+ t.transmute_mut_inference_helper()
+ } else {
+ t.transmute_mut()
+ }
+ }}
+}
+
+/// Conditionally transmutes a value of one type to a value of another type of
+/// the same size.
+///
+/// This macro behaves like an invocation of this function:
+///
+/// ```ignore
+/// fn try_transmute<Src, Dst>(src: Src) -> Result<Dst, ValidityError<Src, Dst>>
+/// where
+/// Src: IntoBytes,
+/// Dst: TryFromBytes,
+/// size_of::<Src>() == size_of::<Dst>(),
+/// {
+/// # /*
+/// ...
+/// # */
+/// }
+/// ```
+///
+/// However, unlike a function, this macro can only be invoked when the types of
+/// `Src` and `Dst` are completely concrete. The types `Src` and `Dst` are
+/// inferred from the calling context; they cannot be explicitly specified in
+/// the macro invocation.
+///
+/// Note that the `Src` produced by the expression `$e` will *not* be dropped.
+/// Semantically, its bits will be copied into a new value of type `Dst`, the
+/// original `Src` will be forgotten, and the value of type `Dst` will be
+/// returned.
+///
+/// # Examples
+///
+/// ```
+/// # use zerocopy::*;
+/// // 0u8 → bool = false
+/// assert_eq!(try_transmute!(0u8), Ok(false));
+///
+/// // 1u8 → bool = true
+/// assert_eq!(try_transmute!(1u8), Ok(true));
+///
+/// // 2u8 → bool = error
+/// assert!(matches!(
+/// try_transmute!(2u8),
+/// Result::<bool, _>::Err(ValidityError { .. })
+/// ));
+/// ```
+///
+#[doc = codegen_section!(
+ header = "h2",
+ bench = "try_transmute",
+ format = "coco_static_size",
+)]
+#[macro_export]
+macro_rules! try_transmute {
+ ($e:expr) => {{
+ // NOTE: This must be a macro (rather than a function with trait bounds)
+ // because there's no way, in a generic context, to enforce that two
+ // types have the same size. `core::mem::transmute` uses compiler magic
+ // to enforce this so long as the types are concrete.
+
+ let e = $e;
+ if false {
+ // Check that the sizes of the source and destination types are
+ // equal.
+
+ // SAFETY: This code is never executed.
+ Ok(unsafe {
+ // Clippy: We can't annotate the types; this macro is designed
+ // to infer the types from the calling context.
+ #[allow(clippy::missing_transmute_annotations)]
+ $crate::util::macro_util::core_reexport::mem::transmute(e)
+ })
+ } else {
+ $crate::util::macro_util::try_transmute::<_, _>(e)
+ }
+ }}
+}
+
+/// Conditionally transmutes a mutable or immutable reference of one type to an
+/// immutable reference of another type of the same size and compatible
+/// alignment.
+///
+/// *Note that while the **value** of the referent is checked for validity at
+/// runtime, the **size** and **alignment** are checked at compile time. For
+/// conversions which are fallible with respect to size and alignment, see the
+/// methods on [`TryFromBytes`].*
+///
+/// This macro behaves like an invocation of this function:
+///
+/// ```ignore
+/// fn try_transmute_ref<Src, Dst>(src: &Src) -> Result<&Dst, ValidityError<&Src, Dst>>
+/// where
+/// Src: IntoBytes + Immutable + ?Sized,
+/// Dst: TryFromBytes + Immutable + ?Sized,
+/// align_of::<Src>() >= align_of::<Dst>(),
+/// size_compatible::<Src, Dst>(),
+/// {
+/// # /*
+/// ...
+/// # */
+/// }
+/// ```
+///
+/// The types `Src` and `Dst` are inferred from the calling context; they cannot
+/// be explicitly specified in the macro invocation.
+///
+/// [`TryFromBytes`]: crate::TryFromBytes
+///
+/// # Size compatibility
+///
+/// `try_transmute_ref!` supports transmuting between `Sized` types, between
+/// unsized (i.e., `?Sized`) types, and from a `Sized` type to an unsized type.
+/// It supports any transmutation that preserves the number of bytes of the
+/// referent, even if doing so requires updating the metadata stored in an
+/// unsized "fat" reference:
+///
+/// ```
+/// # use zerocopy::try_transmute_ref;
+/// # use core::mem::size_of_val; // Not in the prelude on our MSRV
+/// let src: &[[u8; 2]] = &[[0, 1], [2, 3]][..];
+/// let dst: &[u8] = try_transmute_ref!(src).unwrap();
+///
+/// assert_eq!(src.len(), 2);
+/// assert_eq!(dst.len(), 4);
+/// assert_eq!(dst, [0, 1, 2, 3]);
+/// assert_eq!(size_of_val(src), size_of_val(dst));
+/// ```
+///
+/// # Examples
+///
+/// Transmuting between `Sized` types:
+///
+/// ```
+/// # use zerocopy::*;
+/// // 0u8 → bool = false
+/// assert_eq!(try_transmute_ref!(&0u8), Ok(&false));
+///
+/// // 1u8 → bool = true
+/// assert_eq!(try_transmute_ref!(&1u8), Ok(&true));
+///
+/// // 2u8 → bool = error
+/// assert!(matches!(
+/// try_transmute_ref!(&2u8),
+/// Result::<&bool, _>::Err(ValidityError { .. })
+/// ));
+/// ```
+///
+/// Transmuting between unsized types:
+///
+/// ```
+/// # use {zerocopy::*, zerocopy_derive::*};
+/// # type u16 = zerocopy::byteorder::native_endian::U16;
+/// # type u32 = zerocopy::byteorder::native_endian::U32;
+/// #[derive(KnownLayout, FromBytes, IntoBytes, Immutable)]
+/// #[repr(C)]
+/// struct SliceDst<T, U> {
+/// t: T,
+/// u: [U],
+/// }
+///
+/// type Src = SliceDst<u32, u16>;
+/// type Dst = SliceDst<u16, bool>;
+///
+/// let src = Src::ref_from_bytes(&[0, 1, 0, 1, 0, 1, 0, 1]).unwrap();
+/// let dst: &Dst = try_transmute_ref!(src).unwrap();
+///
+/// assert_eq!(src.t.as_bytes(), [0, 1, 0, 1]);
+/// assert_eq!(src.u.len(), 2);
+/// assert_eq!(src.u.as_bytes(), [0, 1, 0, 1]);
+///
+/// assert_eq!(dst.t.as_bytes(), [0, 1]);
+/// assert_eq!(dst.u, [false, true, false, true, false, true]);
+/// ```
+///
+#[doc = codegen_section!(
+ header = "h2",
+ bench = "try_transmute_ref",
+ format = "coco",
+ arity = 2,
+ [
+ open
+ @index 1
+ @title "Sized"
+ @variant "static_size"
+ ],
+ [
+ @index 2
+ @title "Unsized"
+ @variant "dynamic_size"
+ ]
+)]
+#[macro_export]
+macro_rules! try_transmute_ref {
+ ($e:expr) => {{
+ // Ensure that the source type is a reference or a mutable reference
+ // (note that mutable references are implicitly reborrowed here).
+ let e: &_ = $e;
+
+ #[allow(unused_imports)]
+ use $crate::util::macro_util::TryTransmuteRefDst as _;
+ let t = $crate::util::macro_util::Wrap::new(e);
+ if false {
+ // This branch exists solely to force the compiler to infer the type
+ // of `Dst` *before* it attempts to resolve the method call to
+ // `try_transmute_ref` in the `else` branch.
+ //
+ // Without this, if `Src` is `Sized` but `Dst` is `!Sized`, the
+ // compiler will eagerly select the inherent impl of
+ // `try_transmute_ref` (which requires `Dst: Sized`) because
+ // inherent methods take priority over trait methods. It does this
+ // before it realizes `Dst` is `!Sized`, leading to a compile error
+ // when it checks the bounds later.
+ //
+ // By calling this helper (which returns `&Dst`), we force `Dst`
+ // to be fully resolved. By the time it gets to the `else`
+ // branch, the compiler knows `Dst` is `!Sized`, properly
+ // disqualifies the inherent method, and falls back to the trait
+ // implementation.
+ Ok(t.transmute_ref_inference_helper())
+ } else {
+ t.try_transmute_ref()
+ }
+ }}
+}
+
+/// Conditionally transmutes a mutable reference of one type to a mutable
+/// reference of another type of the same size and compatible alignment.
+///
+/// *Note that while the **value** of the referent is checked for validity at
+/// runtime, the **size** and **alignment** are checked at compile time. For
+/// conversions which are fallible with respect to size and alignment, see the
+/// methods on [`TryFromBytes`].*
+///
+/// This macro behaves like an invocation of this function:
+///
+/// ```ignore
+/// fn try_transmute_mut<Src, Dst>(src: &mut Src) -> Result<&mut Dst, ValidityError<&mut Src, Dst>>
+/// where
+/// Src: FromBytes + IntoBytes + ?Sized,
+/// Dst: TryFromBytes + IntoBytes + ?Sized,
+/// align_of::<Src>() >= align_of::<Dst>(),
+/// size_compatible::<Src, Dst>(),
+/// {
+/// # /*
+/// ...
+/// # */
+/// }
+/// ```
+///
+/// The types `Src` and `Dst` are inferred from the calling context; they cannot
+/// be explicitly specified in the macro invocation.
+///
+/// [`TryFromBytes`]: crate::TryFromBytes
+///
+/// # Size compatibility
+///
+/// `try_transmute_mut!` supports transmuting between `Sized` types, between
+/// unsized (i.e., `?Sized`) types, and from a `Sized` type to an unsized type.
+/// It supports any transmutation that preserves the number of bytes of the
+/// referent, even if doing so requires updating the metadata stored in an
+/// unsized "fat" reference:
+///
+/// ```
+/// # use zerocopy::try_transmute_mut;
+/// # use core::mem::size_of_val; // Not in the prelude on our MSRV
+/// let src: &mut [[u8; 2]] = &mut [[0, 1], [2, 3]][..];
+/// let dst: &mut [u8] = try_transmute_mut!(src).unwrap();
+///
+/// assert_eq!(dst.len(), 4);
+/// assert_eq!(dst, [0, 1, 2, 3]);
+/// let dst_size = size_of_val(dst);
+/// assert_eq!(src.len(), 2);
+/// assert_eq!(size_of_val(src), dst_size);
+/// ```
+///
+/// # Examples
+///
+/// Transmuting between `Sized` types:
+///
+/// ```
+/// # use zerocopy::*;
+/// // 0u8 → bool = false
+/// let src = &mut 0u8;
+/// assert_eq!(try_transmute_mut!(src), Ok(&mut false));
+///
+/// // 1u8 → bool = true
+/// let src = &mut 1u8;
+/// assert_eq!(try_transmute_mut!(src), Ok(&mut true));
+///
+/// // 2u8 → bool = error
+/// let src = &mut 2u8;
+/// assert!(matches!(
+/// try_transmute_mut!(src),
+/// Result::<&mut bool, _>::Err(ValidityError { .. })
+/// ));
+/// ```
+///
+/// Transmuting between unsized types:
+///
+/// ```
+/// # use {zerocopy::*, zerocopy_derive::*};
+/// # type u16 = zerocopy::byteorder::native_endian::U16;
+/// # type u32 = zerocopy::byteorder::native_endian::U32;
+/// #[derive(KnownLayout, FromBytes, IntoBytes, Immutable)]
+/// #[repr(C)]
+/// struct SliceDst<T, U> {
+/// t: T,
+/// u: [U],
+/// }
+///
+/// type Src = SliceDst<u32, u16>;
+/// type Dst = SliceDst<u16, bool>;
+///
+/// let mut bytes = [0, 1, 0, 1, 0, 1, 0, 1];
+/// let src = Src::mut_from_bytes(&mut bytes).unwrap();
+///
+/// assert_eq!(src.t.as_bytes(), [0, 1, 0, 1]);
+/// assert_eq!(src.u.len(), 2);
+/// assert_eq!(src.u.as_bytes(), [0, 1, 0, 1]);
+///
+/// let dst: &Dst = try_transmute_mut!(src).unwrap();
+///
+/// assert_eq!(dst.t.as_bytes(), [0, 1]);
+/// assert_eq!(dst.u, [false, true, false, true, false, true]);
+/// ```
+#[macro_export]
+macro_rules! try_transmute_mut {
+ ($e:expr) => {{
+ // Ensure that the source type is a mutable reference.
+ let e: &mut _ = $e;
+
+ #[allow(unused_imports)]
+ use $crate::util::macro_util::TryTransmuteMutDst as _;
+ let t = $crate::util::macro_util::Wrap::new(e);
+ if false {
+ // This branch exists solely to force the compiler to infer the type
+ // of `Dst` *before* it attempts to resolve the method call to
+ // `try_transmute_mut` in the `else` branch.
+ //
+ // Without this, if `Src` is `Sized` but `Dst` is `!Sized`, the
+ // compiler will eagerly select the inherent impl of
+ // `try_transmute_mut` (which requires `Dst: Sized`) because
+ // inherent methods take priority over trait methods. It does this
+ // before it realizes `Dst` is `!Sized`, leading to a compile error
+ // when it checks the bounds later.
+ //
+ // By calling this helper (which returns `&Dst`), we force `Dst`
+ // to be fully resolved. By the time it gets to the `else`
+ // branch, the compiler knows `Dst` is `!Sized`, properly
+ // disqualifies the inherent method, and falls back to the trait
+ // implementation.
+ Ok(t.transmute_mut_inference_helper())
+ } else {
+ t.try_transmute_mut()
+ }
+ }}
+}
+
+/// Includes a file and safely transmutes it to a value of an arbitrary type.
+///
+/// The file will be included as a byte array, `[u8; N]`, which will be
+/// transmuted to another type, `T`. `T` is inferred from the calling context,
+/// and must implement [`FromBytes`].
+///
+/// The file is located relative to the current file (similarly to how modules
+/// are found). The provided path is interpreted in a platform-specific way at
+/// compile time. So, for instance, an invocation with a Windows path containing
+/// backslashes `\` would not compile correctly on Unix.
+///
+/// `include_value!` is ignorant of byte order. For byte order-aware types, see
+/// the [`byteorder`] module.
+///
+/// [`FromBytes`]: crate::FromBytes
+/// [`byteorder`]: crate::byteorder
+///
+/// # Examples
+///
+/// Assume there are two files in the same directory with the following
+/// contents:
+///
+/// File `data` (no trailing newline):
+///
+/// ```text
+/// abcd
+/// ```
+///
+/// File `main.rs`:
+///
+/// ```rust
+/// use zerocopy::include_value;
+/// # macro_rules! include_value {
+/// # ($file:expr) => { zerocopy::include_value!(concat!("../testdata/include_value/", $file)) };
+/// # }
+///
+/// fn main() {
+/// let as_u32: u32 = include_value!("data");
+/// assert_eq!(as_u32, u32::from_ne_bytes([b'a', b'b', b'c', b'd']));
+/// let as_i32: i32 = include_value!("data");
+/// assert_eq!(as_i32, i32::from_ne_bytes([b'a', b'b', b'c', b'd']));
+/// }
+/// ```
+///
+/// # Use in `const` contexts
+///
+/// This macro can be invoked in `const` contexts.
+#[doc(alias("include_bytes", "include_data", "include_type"))]
+#[macro_export]
+macro_rules! include_value {
+ ($file:expr $(,)?) => {
+ $crate::transmute!(*::core::include_bytes!($file))
+ };
+}
+
+#[doc(hidden)]
+#[macro_export]
+macro_rules! cryptocorrosion_derive_traits {
+ (
+ #[repr($repr:ident)]
+ $(#[$attr:meta])*
+ $vis:vis struct $name:ident $(<$($tyvar:ident),*>)?
+ $(
+ (
+ $($tuple_field_vis:vis $tuple_field_ty:ty),*
+ );
+ )?
+
+ $(
+ {
+ $($field_vis:vis $field_name:ident: $field_ty:ty,)*
+ }
+ )?
+ ) => {
+ $crate::cryptocorrosion_derive_traits!(@assert_allowed_struct_repr #[repr($repr)]);
+
+ $(#[$attr])*
+ #[repr($repr)]
+ $vis struct $name $(<$($tyvar),*>)?
+ $(
+ (
+ $($tuple_field_vis $tuple_field_ty),*
+ );
+ )?
+
+ $(
+ {
+ $($field_vis $field_name: $field_ty,)*
+ }
+ )?
+
+ // SAFETY: See inline.
+ unsafe impl $(<$($tyvar),*>)? $crate::TryFromBytes for $name$(<$($tyvar),*>)?
+ where
+ $(
+ $($tuple_field_ty: $crate::FromBytes,)*
+ )?
+
+ $(
+ $($field_ty: $crate::FromBytes,)*
+ )?
+ {
+ #[inline(always)]
+ fn is_bit_valid<A>(_: $crate::Maybe<'_, Self, A>) -> bool
+ where
+ A: $crate::invariant::Alignment,
+ {
+ // SAFETY: This macro only accepts `#[repr(C)]` and
+ // `#[repr(transparent)]` structs, and this `impl` block
+ // requires all field types to be `FromBytes`. Thus, all
+ // initialized byte sequences constitutes valid instances of
+ // `Self`.
+ true
+ }
+
+ fn only_derive_is_allowed_to_implement_this_trait() {}
+ }
+
+ // SAFETY: This macro only accepts `#[repr(C)]` and
+ // `#[repr(transparent)]` structs, and this `impl` block requires all
+ // field types to be `FromBytes`, which is a sub-trait of `FromZeros`.
+ unsafe impl $(<$($tyvar),*>)? $crate::FromZeros for $name$(<$($tyvar),*>)?
+ where
+ $(
+ $($tuple_field_ty: $crate::FromBytes,)*
+ )?
+
+ $(
+ $($field_ty: $crate::FromBytes,)*
+ )?
+ {
+ fn only_derive_is_allowed_to_implement_this_trait() {}
+ }
+
+ // SAFETY: This macro only accepts `#[repr(C)]` and
+ // `#[repr(transparent)]` structs, and this `impl` block requires all
+ // field types to be `FromBytes`.
+ unsafe impl $(<$($tyvar),*>)? $crate::FromBytes for $name$(<$($tyvar),*>)?
+ where
+ $(
+ $($tuple_field_ty: $crate::FromBytes,)*
+ )?
+
+ $(
+ $($field_ty: $crate::FromBytes,)*
+ )?
+ {
+ fn only_derive_is_allowed_to_implement_this_trait() {}
+ }
+
+ // SAFETY: This macro only accepts `#[repr(C)]` and
+ // `#[repr(transparent)]` structs, this `impl` block requires all field
+ // types to be `IntoBytes`, and a padding check is used to ensures that
+ // there are no padding bytes.
+ unsafe impl $(<$($tyvar),*>)? $crate::IntoBytes for $name$(<$($tyvar),*>)?
+ where
+ $(
+ $($tuple_field_ty: $crate::IntoBytes,)*
+ )?
+
+ $(
+ $($field_ty: $crate::IntoBytes,)*
+ )?
+
+ (): $crate::util::macro_util::PaddingFree<
+ Self,
+ {
+ $crate::cryptocorrosion_derive_traits!(
+ @struct_padding_check #[repr($repr)]
+ $(($($tuple_field_ty),*))?
+ $({$($field_ty),*})?
+ )
+ },
+ >,
+ {
+ fn only_derive_is_allowed_to_implement_this_trait() {}
+ }
+
+ // SAFETY: This macro only accepts `#[repr(C)]` and
+ // `#[repr(transparent)]` structs, and this `impl` block requires all
+ // field types to be `Immutable`.
+ unsafe impl $(<$($tyvar),*>)? $crate::Immutable for $name$(<$($tyvar),*>)?
+ where
+ $(
+ $($tuple_field_ty: $crate::Immutable,)*
+ )?
+
+ $(
+ $($field_ty: $crate::Immutable,)*
+ )?
+ {
+ fn only_derive_is_allowed_to_implement_this_trait() {}
+ }
+ };
+ (@assert_allowed_struct_repr #[repr(transparent)]) => {};
+ (@assert_allowed_struct_repr #[repr(C)]) => {};
+ (@assert_allowed_struct_repr #[$_attr:meta]) => {
+ compile_error!("repr must be `#[repr(transparent)]` or `#[repr(C)]`");
+ };
+ (
+ @struct_padding_check #[repr(transparent)]
+ $(($($tuple_field_ty:ty),*))?
+ $({$($field_ty:ty),*})?
+ ) => {
+ // SAFETY: `#[repr(transparent)]` structs cannot have the same layout as
+ // their single non-zero-sized field, and so cannot have any padding
+ // outside of that field.
+ 0
+ };
+ (
+ @struct_padding_check #[repr(C)]
+ $(($($tuple_field_ty:ty),*))?
+ $({$($field_ty:ty),*})?
+ ) => {
+ $crate::struct_padding!(
+ Self,
+ None,
+ None,
+ [
+ $($($tuple_field_ty),*)?
+ $($($field_ty),*)?
+ ]
+ )
+ };
+ (
+ #[repr(C)]
+ $(#[$attr:meta])*
+ $vis:vis union $name:ident {
+ $(
+ $field_name:ident: $field_ty:ty,
+ )*
+ }
+ ) => {
+ $(#[$attr])*
+ #[repr(C)]
+ $vis union $name {
+ $(
+ $field_name: $field_ty,
+ )*
+ }
+
+ // SAFETY: See inline.
+ unsafe impl $crate::TryFromBytes for $name
+ where
+ $(
+ $field_ty: $crate::FromBytes,
+ )*
+ {
+ #[inline(always)]
+ fn is_bit_valid<A>(_: $crate::Maybe<'_, Self, A>) -> bool
+ where
+ A: $crate::invariant::Alignment,
+ {
+ // SAFETY: This macro only accepts `#[repr(C)]` unions, and this
+ // `impl` block requires all field types to be `FromBytes`.
+ // Thus, all initialized byte sequences constitutes valid
+ // instances of `Self`.
+ true
+ }
+
+ fn only_derive_is_allowed_to_implement_this_trait() {}
+ }
+
+ // SAFETY: This macro only accepts `#[repr(C)]` unions, and this `impl`
+ // block requires all field types to be `FromBytes`, which is a
+ // sub-trait of `FromZeros`.
+ unsafe impl $crate::FromZeros for $name
+ where
+ $(
+ $field_ty: $crate::FromBytes,
+ )*
+ {
+ fn only_derive_is_allowed_to_implement_this_trait() {}
+ }
+
+ // SAFETY: This macro only accepts `#[repr(C)]` unions, and this `impl`
+ // block requires all field types to be `FromBytes`.
+ unsafe impl $crate::FromBytes for $name
+ where
+ $(
+ $field_ty: $crate::FromBytes,
+ )*
+ {
+ fn only_derive_is_allowed_to_implement_this_trait() {}
+ }
+
+ // SAFETY: This macro only accepts `#[repr(C)]` unions, this `impl`
+ // block requires all field types to be `IntoBytes`, and a padding check
+ // is used to ensures that there are no padding bytes before or after
+ // any field.
+ unsafe impl $crate::IntoBytes for $name
+ where
+ $(
+ $field_ty: $crate::IntoBytes,
+ )*
+ (): $crate::util::macro_util::PaddingFree<
+ Self,
+ {
+ $crate::union_padding!(
+ Self,
+ None::<usize>,
+ None::<usize>,
+ [$($field_ty),*]
+ )
+ },
+ >,
+ {
+ fn only_derive_is_allowed_to_implement_this_trait() {}
+ }
+
+ // SAFETY: This macro only accepts `#[repr(C)]` unions, and this `impl`
+ // block requires all field types to be `Immutable`.
+ unsafe impl $crate::Immutable for $name
+ where
+ $(
+ $field_ty: $crate::Immutable,
+ )*
+ {
+ fn only_derive_is_allowed_to_implement_this_trait() {}
+ }
+ };
+}
+
+#[cfg(test)]
+mod tests {
+ use crate::{
+ byteorder::native_endian::{U16, U32},
+ util::testutil::*,
+ *,
+ };
+
+ #[derive(KnownLayout, Immutable, FromBytes, IntoBytes, PartialEq, Debug)]
+ #[repr(C)]
+ struct SliceDst<T, U> {
+ a: T,
+ b: [U],
+ }
+
+ #[test]
+ fn test_transmute() {
+ // Test that memory is transmuted as expected.
+ let array_of_u8s = [0u8, 1, 2, 3, 4, 5, 6, 7];
+ let array_of_arrays = [[0, 1], [2, 3], [4, 5], [6, 7]];
+ let x: [[u8; 2]; 4] = transmute!(array_of_u8s);
+ assert_eq!(x, array_of_arrays);
+ let x: [u8; 8] = transmute!(array_of_arrays);
+ assert_eq!(x, array_of_u8s);
+
+ // Test that memory is transmuted as expected when shrinking.
+ let x: [[u8; 2]; 3] = transmute!(#![allow(shrink)] array_of_u8s);
+ assert_eq!(x, [[0u8, 1], [2, 3], [4, 5]]);
+
+ // Test that the source expression's value is forgotten rather than
+ // dropped.
+ #[derive(IntoBytes)]
+ #[repr(transparent)]
+ struct PanicOnDrop(());
+ impl Drop for PanicOnDrop {
+ fn drop(&mut self) {
+ panic!("PanicOnDrop::drop");
+ }
+ }
+ #[allow(clippy::let_unit_value)]
+ let _: () = transmute!(PanicOnDrop(()));
+ #[allow(clippy::let_unit_value)]
+ let _: () = transmute!(#![allow(shrink)] PanicOnDrop(()));
+
+ // Test that `transmute!` is legal in a const context.
+ const ARRAY_OF_U8S: [u8; 8] = [0u8, 1, 2, 3, 4, 5, 6, 7];
+ const ARRAY_OF_ARRAYS: [[u8; 2]; 4] = [[0, 1], [2, 3], [4, 5], [6, 7]];
+ const X: [[u8; 2]; 4] = transmute!(ARRAY_OF_U8S);
+ assert_eq!(X, ARRAY_OF_ARRAYS);
+ const X_SHRINK: [[u8; 2]; 3] = transmute!(#![allow(shrink)] ARRAY_OF_U8S);
+ assert_eq!(X_SHRINK, [[0u8, 1], [2, 3], [4, 5]]);
+
+ // Test that `transmute!` works with `!Immutable` types.
+ let x: usize = transmute!(UnsafeCell::new(1usize));
+ assert_eq!(x, 1);
+ let x: UnsafeCell<usize> = transmute!(1usize);
+ assert_eq!(x.into_inner(), 1);
+ let x: UnsafeCell<isize> = transmute!(UnsafeCell::new(1usize));
+ assert_eq!(x.into_inner(), 1);
+ }
+
+ // A `Sized` type which doesn't implement `KnownLayout` (it is "not
+ // `KnownLayout`", or `Nkl`).
+ //
+ // This permits us to test that `transmute_ref!` and `transmute_mut!` work
+ // for types which are `Sized + !KnownLayout`. When we added support for
+ // slice DSTs in #1924, this new support relied on `KnownLayout`, but we
+ // need to make sure to remain backwards-compatible with code which uses
+ // these macros with types which are `!KnownLayout`.
+ #[derive(FromBytes, IntoBytes, Immutable, PartialEq, Eq, Debug)]
+ #[repr(transparent)]
+ struct Nkl<T>(T);
+
+ #[test]
+ fn test_transmute_ref() {
+ // Test that memory is transmuted as expected.
+ let array_of_u8s = [0u8, 1, 2, 3, 4, 5, 6, 7];
+ let array_of_arrays = [[0, 1], [2, 3], [4, 5], [6, 7]];
+ let x: &[[u8; 2]; 4] = transmute_ref!(&array_of_u8s);
+ assert_eq!(*x, array_of_arrays);
+ let x: &[u8; 8] = transmute_ref!(&array_of_arrays);
+ assert_eq!(*x, array_of_u8s);
+
+ // Test that `transmute_ref!` is legal in a const context.
+ const ARRAY_OF_U8S: [u8; 8] = [0u8, 1, 2, 3, 4, 5, 6, 7];
+ const ARRAY_OF_ARRAYS: [[u8; 2]; 4] = [[0, 1], [2, 3], [4, 5], [6, 7]];
+ #[allow(clippy::redundant_static_lifetimes)]
+ const X: &'static [[u8; 2]; 4] = transmute_ref!(&ARRAY_OF_U8S);
+ assert_eq!(*X, ARRAY_OF_ARRAYS);
+
+ // Test sized -> unsized transmutation.
+ let array_of_u8s = [0u8, 1, 2, 3, 4, 5, 6, 7];
+ let array_of_arrays = [[0, 1], [2, 3], [4, 5], [6, 7]];
+ let slice_of_arrays = &array_of_arrays[..];
+ let x: &[[u8; 2]] = transmute_ref!(&array_of_u8s);
+ assert_eq!(x, slice_of_arrays);
+
+ // Before 1.61.0, we can't define the `const fn transmute_ref` function
+ // that we do on and after 1.61.0.
+ #[cfg(no_zerocopy_generic_bounds_in_const_fn_1_61_0)]
+ {
+ // Test that `transmute_ref!` supports non-`KnownLayout` `Sized`
+ // types.
+ const ARRAY_OF_NKL_U8S: Nkl<[u8; 8]> = Nkl([0u8, 1, 2, 3, 4, 5, 6, 7]);
+ const ARRAY_OF_NKL_ARRAYS: Nkl<[[u8; 2]; 4]> = Nkl([[0, 1], [2, 3], [4, 5], [6, 7]]);
+ const X_NKL: &Nkl<[[u8; 2]; 4]> = transmute_ref!(&ARRAY_OF_NKL_U8S);
+ assert_eq!(*X_NKL, ARRAY_OF_NKL_ARRAYS);
+ }
+
+ #[cfg(not(no_zerocopy_generic_bounds_in_const_fn_1_61_0))]
+ {
+ // Call through a generic function to make sure our autoref
+ // specialization trick works even when types are generic.
+ const fn transmute_ref<T, U>(t: &T) -> &U
+ where
+ T: IntoBytes + Immutable,
+ U: FromBytes + Immutable,
+ {
+ transmute_ref!(t)
+ }
+
+ // Test that `transmute_ref!` supports non-`KnownLayout` `Sized`
+ // types.
+ const ARRAY_OF_NKL_U8S: Nkl<[u8; 8]> = Nkl([0u8, 1, 2, 3, 4, 5, 6, 7]);
+ const ARRAY_OF_NKL_ARRAYS: Nkl<[[u8; 2]; 4]> = Nkl([[0, 1], [2, 3], [4, 5], [6, 7]]);
+ const X_NKL: &Nkl<[[u8; 2]; 4]> = transmute_ref(&ARRAY_OF_NKL_U8S);
+ assert_eq!(*X_NKL, ARRAY_OF_NKL_ARRAYS);
+ }
+
+ // Test that `transmute_ref!` works on slice DSTs in and that memory is
+ // transmuted as expected.
+ let slice_dst_of_u8s =
+ SliceDst::<U16, [u8; 2]>::ref_from_bytes(&[0, 1, 2, 3, 4, 5][..]).unwrap();
+ let slice_dst_of_u16s =
+ SliceDst::<U16, U16>::ref_from_bytes(&[0, 1, 2, 3, 4, 5][..]).unwrap();
+ let x: &SliceDst<U16, U16> = transmute_ref!(slice_dst_of_u8s);
+ assert_eq!(x, slice_dst_of_u16s);
+
+ let slice_dst_of_u8s =
+ SliceDst::<U16, u8>::ref_from_bytes(&[0, 1, 2, 3, 4, 5][..]).unwrap();
+ let x: &[u8] = transmute_ref!(slice_dst_of_u8s);
+ assert_eq!(x, [0, 1, 2, 3, 4, 5]);
+
+ let x: &[u8] = transmute_ref!(slice_dst_of_u16s);
+ assert_eq!(x, [0, 1, 2, 3, 4, 5]);
+
+ let x: &[U16] = transmute_ref!(slice_dst_of_u16s);
+ let slice_of_u16s: &[U16] = <[U16]>::ref_from_bytes(&[0, 1, 2, 3, 4, 5][..]).unwrap();
+ assert_eq!(x, slice_of_u16s);
+
+ // Test that transmuting from a type with larger trailing slice offset
+ // and larger trailing slice element works.
+ let bytes = &[0, 1, 2, 3, 4, 5, 6, 7][..];
+ let slice_dst_big = SliceDst::<U32, U16>::ref_from_bytes(bytes).unwrap();
+ let slice_dst_small = SliceDst::<U16, u8>::ref_from_bytes(bytes).unwrap();
+ let x: &SliceDst<U16, u8> = transmute_ref!(slice_dst_big);
+ assert_eq!(x, slice_dst_small);
+
+ // Test that it's legal to transmute a reference while shrinking the
+ // lifetime (note that `X` has the lifetime `'static`).
+ let x: &[u8; 8] = transmute_ref!(X);
+ assert_eq!(*x, ARRAY_OF_U8S);
+
+ // Test that `transmute_ref!` supports decreasing alignment.
+ let u = AU64(0);
+ let array = [0, 0, 0, 0, 0, 0, 0, 0];
+ let x: &[u8; 8] = transmute_ref!(&u);
+ assert_eq!(*x, array);
+
+ // Test that a mutable reference can be turned into an immutable one.
+ let mut x = 0u8;
+ #[allow(clippy::useless_transmute)]
+ let y: &u8 = transmute_ref!(&mut x);
+ assert_eq!(*y, 0);
+ }
+
+ #[test]
+ fn test_try_transmute() {
+ // Test that memory is transmuted with `try_transmute` as expected.
+ let array_of_bools = [false, true, false, true, false, true, false, true];
+ let array_of_arrays = [[0, 1], [0, 1], [0, 1], [0, 1]];
+ let x: Result<[[u8; 2]; 4], _> = try_transmute!(array_of_bools);
+ assert_eq!(x, Ok(array_of_arrays));
+ let x: Result<[bool; 8], _> = try_transmute!(array_of_arrays);
+ assert_eq!(x, Ok(array_of_bools));
+
+ // Test that `try_transmute!` works with `!Immutable` types.
+ let x: Result<usize, _> = try_transmute!(UnsafeCell::new(1usize));
+ assert_eq!(x.unwrap(), 1);
+ let x: Result<UnsafeCell<usize>, _> = try_transmute!(1usize);
+ assert_eq!(x.unwrap().into_inner(), 1);
+ let x: Result<UnsafeCell<isize>, _> = try_transmute!(UnsafeCell::new(1usize));
+ assert_eq!(x.unwrap().into_inner(), 1);
+
+ #[derive(FromBytes, IntoBytes, Debug, PartialEq)]
+ #[repr(transparent)]
+ struct PanicOnDrop<T>(T);
+
+ impl<T> Drop for PanicOnDrop<T> {
+ fn drop(&mut self) {
+ panic!("PanicOnDrop dropped");
+ }
+ }
+
+ // Since `try_transmute!` semantically moves its argument on failure,
+ // the `PanicOnDrop` is not dropped, and thus this shouldn't panic.
+ let x: Result<usize, _> = try_transmute!(PanicOnDrop(1usize));
+ assert_eq!(x, Ok(1));
+
+ // Since `try_transmute!` semantically returns ownership of its argument
+ // on failure, the `PanicOnDrop` is returned rather than dropped, and
+ // thus this shouldn't panic.
+ let y: Result<bool, _> = try_transmute!(PanicOnDrop(2u8));
+ // We have to use `map_err` instead of comparing against
+ // `Err(PanicOnDrop(2u8))` because the latter would create and then drop
+ // its `PanicOnDrop` temporary, which would cause a panic.
+ assert_eq!(y.as_ref().map_err(|p| &p.src.0), Err::<&bool, _>(&2u8));
+ mem::forget(y);
+ }
+
+ #[test]
+ fn test_try_transmute_ref() {
+ // Test that memory is transmuted with `try_transmute_ref` as expected.
+ let array_of_bools = &[false, true, false, true, false, true, false, true];
+ let array_of_arrays = &[[0, 1], [0, 1], [0, 1], [0, 1]];
+ let x: Result<&[[u8; 2]; 4], _> = try_transmute_ref!(array_of_bools);
+ assert_eq!(x, Ok(array_of_arrays));
+ let x: Result<&[bool; 8], _> = try_transmute_ref!(array_of_arrays);
+ assert_eq!(x, Ok(array_of_bools));
+
+ // Test that it's legal to transmute a reference while shrinking the
+ // lifetime.
+ {
+ let x: Result<&[[u8; 2]; 4], _> = try_transmute_ref!(array_of_bools);
+ assert_eq!(x, Ok(array_of_arrays));
+ }
+
+ // Test that `try_transmute_ref!` supports decreasing alignment.
+ let u = AU64(0);
+ let array = [0u8, 0, 0, 0, 0, 0, 0, 0];
+ let x: Result<&[u8; 8], _> = try_transmute_ref!(&u);
+ assert_eq!(x, Ok(&array));
+
+ // Test that a mutable reference can be turned into an immutable one.
+ let mut x = 0u8;
+ #[allow(clippy::useless_transmute)]
+ let y: Result<&u8, _> = try_transmute_ref!(&mut x);
+ assert_eq!(y, Ok(&0));
+
+ // Test that sized types work which don't implement `KnownLayout`.
+ let array_of_nkl_u8s = Nkl([0u8, 1, 2, 3, 4, 5, 6, 7]);
+ let array_of_nkl_arrays = Nkl([[0, 1], [2, 3], [4, 5], [6, 7]]);
+ let x: Result<&Nkl<[[u8; 2]; 4]>, _> = try_transmute_ref!(&array_of_nkl_u8s);
+ assert_eq!(x, Ok(&array_of_nkl_arrays));
+
+ // Test sized -> unsized transmutation.
+ let array_of_u8s = [0u8, 1, 2, 3, 4, 5, 6, 7];
+ let array_of_arrays = [[0, 1], [2, 3], [4, 5], [6, 7]];
+ let slice_of_arrays = &array_of_arrays[..];
+ let x: Result<&[[u8; 2]], _> = try_transmute_ref!(&array_of_u8s);
+ assert_eq!(x, Ok(slice_of_arrays));
+
+ // Test unsized -> unsized transmutation.
+ let slice_dst_of_u8s =
+ SliceDst::<U16, [u8; 2]>::ref_from_bytes(&[0, 1, 2, 3, 4, 5][..]).unwrap();
+ let slice_dst_of_u16s =
+ SliceDst::<U16, U16>::ref_from_bytes(&[0, 1, 2, 3, 4, 5][..]).unwrap();
+ let x: Result<&SliceDst<U16, U16>, _> = try_transmute_ref!(slice_dst_of_u8s);
+ assert_eq!(x, Ok(slice_dst_of_u16s));
+ }
+
+ #[test]
+ fn test_try_transmute_mut() {
+ // Test that memory is transmuted with `try_transmute_mut` as expected.
+ let array_of_u8s = &mut [0u8, 1, 0, 1, 0, 1, 0, 1];
+ let array_of_arrays = &mut [[0u8, 1], [0, 1], [0, 1], [0, 1]];
+ let x: Result<&mut [[u8; 2]; 4], _> = try_transmute_mut!(array_of_u8s);
+ assert_eq!(x, Ok(array_of_arrays));
+
+ let array_of_bools = &mut [false, true, false, true, false, true, false, true];
+ let array_of_arrays = &mut [[0u8, 1], [0, 1], [0, 1], [0, 1]];
+ let x: Result<&mut [bool; 8], _> = try_transmute_mut!(array_of_arrays);
+ assert_eq!(x, Ok(array_of_bools));
+
+ // Test that it's legal to transmute a reference while shrinking the
+ // lifetime.
+ let array_of_bools = &mut [false, true, false, true, false, true, false, true];
+ let array_of_arrays = &mut [[0u8, 1], [0, 1], [0, 1], [0, 1]];
+ {
+ let x: Result<&mut [bool; 8], _> = try_transmute_mut!(array_of_arrays);
+ assert_eq!(x, Ok(array_of_bools));
+ }
+
+ // Test that `try_transmute_mut!` supports decreasing alignment.
+ let u = &mut AU64(0);
+ let array = &mut [0u8, 0, 0, 0, 0, 0, 0, 0];
+ let x: Result<&mut [u8; 8], _> = try_transmute_mut!(u);
+ assert_eq!(x, Ok(array));
+
+ // Test that a mutable reference can be turned into an immutable one.
+ let mut x = 0u8;
+ #[allow(clippy::useless_transmute)]
+ let y: Result<&mut u8, _> = try_transmute_mut!(&mut x);
+ assert_eq!(y, Ok(&mut 0));
+
+ // Test that sized types work which don't implement `KnownLayout`.
+ let mut array_of_nkl_u8s = Nkl([0u8, 1, 2, 3, 4, 5, 6, 7]);
+ let mut array_of_nkl_arrays = Nkl([[0, 1], [2, 3], [4, 5], [6, 7]]);
+ let x: Result<&mut Nkl<[[u8; 2]; 4]>, _> = try_transmute_mut!(&mut array_of_nkl_u8s);
+ assert_eq!(x, Ok(&mut array_of_nkl_arrays));
+
+ // Test sized -> unsized transmutation.
+ let mut array_of_u8s = [0u8, 1, 2, 3, 4, 5, 6, 7];
+ let mut array_of_arrays = [[0, 1], [2, 3], [4, 5], [6, 7]];
+ let slice_of_arrays = &mut array_of_arrays[..];
+ let x: Result<&mut [[u8; 2]], _> = try_transmute_mut!(&mut array_of_u8s);
+ assert_eq!(x, Ok(slice_of_arrays));
+
+ // Test unsized -> unsized transmutation.
+ let mut bytes = [0, 1, 2, 3, 4, 5, 6];
+ let slice_dst_of_u8s = SliceDst::<u8, [u8; 2]>::mut_from_bytes(&mut bytes[..]).unwrap();
+ let mut bytes = [0, 1, 2, 3, 4, 5, 6];
+ let slice_dst_of_u16s = SliceDst::<u8, U16>::mut_from_bytes(&mut bytes[..]).unwrap();
+ let x: Result<&mut SliceDst<u8, U16>, _> = try_transmute_mut!(slice_dst_of_u8s);
+ assert_eq!(x, Ok(slice_dst_of_u16s));
+ }
+
+ #[test]
+ fn test_transmute_mut() {
+ // Test that memory is transmuted as expected.
+ let mut array_of_u8s = [0u8, 1, 2, 3, 4, 5, 6, 7];
+ let mut array_of_arrays = [[0, 1], [2, 3], [4, 5], [6, 7]];
+ let x: &mut [[u8; 2]; 4] = transmute_mut!(&mut array_of_u8s);
+ assert_eq!(*x, array_of_arrays);
+ let x: &mut [u8; 8] = transmute_mut!(&mut array_of_arrays);
+ assert_eq!(*x, array_of_u8s);
+
+ {
+ // Test that it's legal to transmute a reference while shrinking the
+ // lifetime.
+ let x: &mut [u8; 8] = transmute_mut!(&mut array_of_arrays);
+ assert_eq!(*x, array_of_u8s);
+ }
+
+ // Test that `transmute_mut!` supports non-`KnownLayout` types.
+ let mut array_of_u8s = Nkl([0u8, 1, 2, 3, 4, 5, 6, 7]);
+ let mut array_of_arrays = Nkl([[0, 1], [2, 3], [4, 5], [6, 7]]);
+ let x: &mut Nkl<[[u8; 2]; 4]> = transmute_mut!(&mut array_of_u8s);
+ assert_eq!(*x, array_of_arrays);
+ let x: &mut Nkl<[u8; 8]> = transmute_mut!(&mut array_of_arrays);
+ assert_eq!(*x, array_of_u8s);
+
+ // Test that `transmute_mut!` supports decreasing alignment.
+ let mut u = AU64(0);
+ let array = [0, 0, 0, 0, 0, 0, 0, 0];
+ let x: &[u8; 8] = transmute_mut!(&mut u);
+ assert_eq!(*x, array);
+
+ // Test that a mutable reference can be turned into an immutable one.
+ let mut x = 0u8;
+ #[allow(clippy::useless_transmute)]
+ let y: &u8 = transmute_mut!(&mut x);
+ assert_eq!(*y, 0);
+
+ // Test that `transmute_mut!` works on slice DSTs in and that memory is
+ // transmuted as expected.
+ let mut bytes = [0, 1, 2, 3, 4, 5, 6];
+ let slice_dst_of_u8s = SliceDst::<u8, [u8; 2]>::mut_from_bytes(&mut bytes[..]).unwrap();
+ let mut bytes = [0, 1, 2, 3, 4, 5, 6];
+ let slice_dst_of_u16s = SliceDst::<u8, U16>::mut_from_bytes(&mut bytes[..]).unwrap();
+ let x: &mut SliceDst<u8, U16> = transmute_mut!(slice_dst_of_u8s);
+ assert_eq!(x, slice_dst_of_u16s);
+
+ // Test that `transmute_mut!` works on slices that memory is transmuted
+ // as expected.
+ let array_of_u16s: &mut [u16] = &mut [0u16, 1, 2];
+ let array_of_i16s: &mut [i16] = &mut [0i16, 1, 2];
+ let x: &mut [i16] = transmute_mut!(array_of_u16s);
+ assert_eq!(x, array_of_i16s);
+
+ // Test that transmuting from a type with larger trailing slice offset
+ // and larger trailing slice element works.
+ let mut bytes = [0, 1, 2, 3, 4, 5, 6, 7];
+ let slice_dst_big = SliceDst::<U32, U16>::mut_from_bytes(&mut bytes[..]).unwrap();
+ let mut bytes = [0, 1, 2, 3, 4, 5, 6, 7];
+ let slice_dst_small = SliceDst::<U16, u8>::mut_from_bytes(&mut bytes[..]).unwrap();
+ let x: &mut SliceDst<U16, u8> = transmute_mut!(slice_dst_big);
+ assert_eq!(x, slice_dst_small);
+
+ // Test sized -> unsized transmutation.
+ let mut array_of_u8s = [0u8, 1, 2, 3, 4, 5, 6, 7];
+ let mut array_of_arrays = [[0, 1], [2, 3], [4, 5], [6, 7]];
+ let slice_of_arrays = &mut array_of_arrays[..];
+ let x: &mut [[u8; 2]] = transmute_mut!(&mut array_of_u8s);
+ assert_eq!(x, slice_of_arrays);
+ }
+
+ #[test]
+ fn test_macros_evaluate_args_once() {
+ let mut ctr = 0;
+ #[allow(clippy::useless_transmute)]
+ let _: usize = transmute!({
+ ctr += 1;
+ 0usize
+ });
+ assert_eq!(ctr, 1);
+
+ let mut ctr = 0;
+ let _: &usize = transmute_ref!({
+ ctr += 1;
+ &0usize
+ });
+ assert_eq!(ctr, 1);
+
+ let mut ctr: usize = 0;
+ let _: &mut usize = transmute_mut!({
+ ctr += 1;
+ &mut ctr
+ });
+ assert_eq!(ctr, 1);
+
+ let mut ctr = 0;
+ #[allow(clippy::useless_transmute)]
+ let _: usize = try_transmute!({
+ ctr += 1;
+ 0usize
+ })
+ .unwrap();
+ assert_eq!(ctr, 1);
+ }
+
+ #[test]
+ fn test_include_value() {
+ const AS_U32: u32 = include_value!("../testdata/include_value/data");
+ assert_eq!(AS_U32, u32::from_ne_bytes([b'a', b'b', b'c', b'd']));
+ const AS_I32: i32 = include_value!("../testdata/include_value/data");
+ assert_eq!(AS_I32, i32::from_ne_bytes([b'a', b'b', b'c', b'd']));
+ }
+
+ #[test]
+ #[allow(non_camel_case_types, unreachable_pub, dead_code)]
+ fn test_cryptocorrosion_derive_traits() {
+ // Test the set of invocations added in
+ // https://github.com/cryptocorrosion/cryptocorrosion/pull/85
+
+ fn assert_impls<T: FromBytes + IntoBytes + Immutable>() {}
+
+ cryptocorrosion_derive_traits! {
+ #[repr(C)]
+ #[derive(Clone, Copy)]
+ pub union vec128_storage {
+ d: [u32; 4],
+ q: [u64; 2],
+ }
+ }
+
+ assert_impls::<vec128_storage>();
+
+ cryptocorrosion_derive_traits! {
+ #[repr(transparent)]
+ #[derive(Copy, Clone, Debug, PartialEq)]
+ pub struct u32x4_generic([u32; 4]);
+ }
+
+ assert_impls::<u32x4_generic>();
+
+ cryptocorrosion_derive_traits! {
+ #[repr(transparent)]
+ #[derive(Copy, Clone, Debug, PartialEq)]
+ pub struct u64x2_generic([u64; 2]);
+ }
+
+ assert_impls::<u64x2_generic>();
+
+ cryptocorrosion_derive_traits! {
+ #[repr(transparent)]
+ #[derive(Copy, Clone, Debug, PartialEq)]
+ pub struct u128x1_generic([u128; 1]);
+ }
+
+ assert_impls::<u128x1_generic>();
+
+ cryptocorrosion_derive_traits! {
+ #[repr(transparent)]
+ #[derive(Copy, Clone, Default)]
+ #[allow(non_camel_case_types)]
+ pub struct x2<W, G>(pub [W; 2], PhantomData<G>);
+ }
+
+ enum NotZerocopy {}
+ assert_impls::<x2<(), NotZerocopy>>();
+
+ cryptocorrosion_derive_traits! {
+ #[repr(transparent)]
+ #[derive(Copy, Clone, Default)]
+ #[allow(non_camel_case_types)]
+ pub struct x4<W>(pub [W; 4]);
+ }
+
+ assert_impls::<x4<()>>();
+
+ #[cfg(feature = "simd")]
+ #[cfg(any(target_arch = "x86", target_arch = "x86_64"))]
+ {
+ #[cfg(target_arch = "x86")]
+ use core::arch::x86::{__m128i, __m256i};
+ #[cfg(target_arch = "x86_64")]
+ use core::arch::x86_64::{__m128i, __m256i};
+
+ cryptocorrosion_derive_traits! {
+ #[repr(C)]
+ #[derive(Copy, Clone)]
+ pub struct X4(__m128i, __m128i, __m128i, __m128i);
+ }
+
+ assert_impls::<X4>();
+
+ cryptocorrosion_derive_traits! {
+ #[repr(C)]
+ /// Generic wrapper for unparameterized storage of any of the
+ /// possible impls. Converting into and out of this type should
+ /// be essentially free, although it may be more aligned than a
+ /// particular impl requires.
+ #[allow(non_camel_case_types)]
+ #[derive(Copy, Clone)]
+ pub union vec128_storage {
+ u32x4: [u32; 4],
+ u64x2: [u64; 2],
+ u128x1: [u128; 1],
+ sse2: __m128i,
+ }
+ }
+
+ assert_impls::<vec128_storage>();
+
+ cryptocorrosion_derive_traits! {
+ #[repr(transparent)]
+ #[allow(non_camel_case_types)]
+ #[derive(Copy, Clone)]
+ pub struct vec<S3, S4, NI> {
+ x: __m128i,
+ s3: PhantomData<S3>,
+ s4: PhantomData<S4>,
+ ni: PhantomData<NI>,
+ }
+ }
+
+ assert_impls::<vec<NotZerocopy, NotZerocopy, NotZerocopy>>();
+
+ cryptocorrosion_derive_traits! {
+ #[repr(transparent)]
+ #[derive(Copy, Clone)]
+ pub struct u32x4x2_avx2<NI> {
+ x: __m256i,
+ ni: PhantomData<NI>,
+ }
+ }
+
+ assert_impls::<u32x4x2_avx2<NotZerocopy>>();
+ }
+
+ // Make sure that our derive works for `#[repr(C)]` structs even though
+ // cryptocorrosion doesn't currently have any.
+ cryptocorrosion_derive_traits! {
+ #[repr(C)]
+ #[derive(Copy, Clone, Debug, PartialEq)]
+ pub struct ReprC(u8, u8, u16);
+ }
+ }
+}
diff --git a/rust/zerocopy/src/pointer/inner.rs b/rust/zerocopy/src/pointer/inner.rs
new file mode 100644
index 0000000..5db0808
--- /dev/null
+++ b/rust/zerocopy/src/pointer/inner.rs
@@ -0,0 +1,754 @@
+// SPDX-License-Identifier: (BSD-2-Clause OR Apache-2.0) OR MIT
+
+// Copyright 2024 The Fuchsia Authors
+//
+// Licensed under a BSD-style license <LICENSE-BSD>, Apache License, Version 2.0
+// <LICENSE-APACHE or https://www.apache.org/licenses/LICENSE-2.0>, or the MIT
+// license <LICENSE-MIT or https://opensource.org/licenses/MIT>, at your option.
+// This file may not be copied, modified, or distributed except according to
+// those terms.
+
+use core::{marker::PhantomData, ops::Range, ptr::NonNull};
+
+pub use _def::PtrInner;
+
+#[allow(unused_imports)]
+use crate::util::polyfills::NumExt as _;
+use crate::{
+ layout::{CastType, MetadataCastError},
+ pointer::cast,
+ util::AsAddress,
+ AlignmentError, CastError, KnownLayout, MetadataOf, SizeError, SplitAt,
+};
+
+mod _def {
+ use super::*;
+ /// The inner pointer stored inside a [`Ptr`][crate::Ptr].
+ ///
+ /// `PtrInner<'a, T>` is [covariant] in `'a` and invariant in `T`.
+ ///
+ /// [covariant]: https://doc.rust-lang.org/reference/subtyping.html
+ #[allow(missing_debug_implementations)]
+ pub struct PtrInner<'a, T>
+ where
+ T: ?Sized,
+ {
+ /// # Invariants
+ ///
+ /// 0. If `ptr`'s referent is not zero sized, then `ptr` has valid
+ /// provenance for its referent, which is entirely contained in some
+ /// Rust allocation, `A`.
+ /// 1. If `ptr`'s referent is not zero sized, `A` is guaranteed to live
+ /// for at least `'a`.
+ ///
+ /// # Postconditions
+ ///
+ /// By virtue of these invariants, code may assume the following, which
+ /// are logical implications of the invariants:
+ /// - `ptr`'s referent is not larger than `isize::MAX` bytes \[1\]
+ /// - `ptr`'s referent does not wrap around the address space \[1\]
+ ///
+ /// \[1\] Per <https://doc.rust-lang.org/1.85.0/std/ptr/index.html#allocated-object>:
+ ///
+ /// For any allocated object with `base` address, `size`, and a set of
+ /// `addresses`, the following are guaranteed:
+ /// ...
+ /// - `size <= isize::MAX`
+ ///
+ /// As a consequence of these guarantees, given any address `a` within
+ /// the set of addresses of an allocated object:
+ /// ...
+ /// - It is guaranteed that, given `o = a - base` (i.e., the offset of
+ /// `a` within the allocated object), `base + o` will not wrap
+ /// around the address space (in other words, will not overflow
+ /// `usize`)
+ ptr: NonNull<T>,
+ // SAFETY: `&'a UnsafeCell<T>` is covariant in `'a` and invariant in `T`
+ // [1]. We use this construction rather than the equivalent `&mut T`,
+ // because our MSRV of 1.65 prohibits `&mut` types in const contexts.
+ //
+ // [1] https://doc.rust-lang.org/1.81.0/reference/subtyping.html#variance
+ _marker: PhantomData<&'a core::cell::UnsafeCell<T>>,
+ }
+
+ impl<'a, T: 'a + ?Sized> Copy for PtrInner<'a, T> {}
+ impl<'a, T: 'a + ?Sized> Clone for PtrInner<'a, T> {
+ #[inline(always)]
+ fn clone(&self) -> PtrInner<'a, T> {
+ // SAFETY: None of the invariants on `ptr` are affected by having
+ // multiple copies of a `PtrInner`.
+ *self
+ }
+ }
+
+ impl<'a, T: 'a + ?Sized> PtrInner<'a, T> {
+ /// Constructs a `Ptr` from a [`NonNull`].
+ ///
+ /// # Safety
+ ///
+ /// The caller promises that:
+ ///
+ /// 0. If `ptr`'s referent is not zero sized, then `ptr` has valid
+ /// provenance for its referent, which is entirely contained in some
+ /// Rust allocation, `A`.
+ /// 1. If `ptr`'s referent is not zero sized, `A` is guaranteed to live
+ /// for at least `'a`.
+ #[inline(always)]
+ #[must_use]
+ pub const unsafe fn new(ptr: NonNull<T>) -> PtrInner<'a, T> {
+ // SAFETY: The caller has promised to satisfy all safety invariants
+ // of `PtrInner`.
+ Self { ptr, _marker: PhantomData }
+ }
+
+ /// Converts this `PtrInner<T>` to a [`NonNull<T>`].
+ ///
+ /// Note that this method does not consume `self`. The caller should
+ /// watch out for `unsafe` code which uses the returned `NonNull` in a
+ /// way that violates the safety invariants of `self`.
+ #[inline(always)]
+ #[must_use]
+ pub const fn as_non_null(&self) -> NonNull<T> {
+ self.ptr
+ }
+
+ /// Converts this `PtrInner<T>` to a [`*mut T`].
+ ///
+ /// Note that this method does not consume `self`. The caller should
+ /// watch out for `unsafe` code which uses the returned `*mut T` in a
+ /// way that violates the safety invariants of `self`.
+ #[inline(always)]
+ #[must_use]
+ pub const fn as_ptr(&self) -> *mut T {
+ self.ptr.as_ptr()
+ }
+ }
+}
+
+impl<'a, T: ?Sized> PtrInner<'a, T> {
+ /// Constructs a `PtrInner` from a reference.
+ #[inline]
+ pub fn from_ref(ptr: &'a T) -> Self {
+ let ptr = NonNull::from(ptr);
+ // SAFETY:
+ // 0. If `ptr`'s referent is not zero sized, then `ptr`, by invariant on
+ // `&'a T` [1], has valid provenance for its referent, which is
+ // entirely contained in some Rust allocation, `A`.
+ // 1. If `ptr`'s referent is not zero sized, then `A`, by invariant on
+ // `&'a T`, is guaranteed to live for at least `'a`.
+ //
+ // [1] Per https://doc.rust-lang.org/1.85.0/std/primitive.reference.html#safety:
+ //
+ // For all types, `T: ?Sized`, and for all `t: &T` or `t: &mut T`,
+ // when such values cross an API boundary, the following invariants
+ // must generally be upheld:
+ // ...
+ // - if `size_of_val(t) > 0`, then `t` is dereferenceable for
+ // `size_of_val(t)` many bytes
+ //
+ // If `t` points at address `a`, being “dereferenceable” for N bytes
+ // means that the memory range `[a, a + N)` is all contained within a
+ // single allocated object.
+ unsafe { Self::new(ptr) }
+ }
+
+ /// Constructs a `PtrInner` from a mutable reference.
+ #[inline]
+ pub fn from_mut(ptr: &'a mut T) -> Self {
+ let ptr = NonNull::from(ptr);
+ // SAFETY:
+ // 0. If `ptr`'s referent is not zero sized, then `ptr`, by invariant on
+ // `&'a mut T` [1], has valid provenance for its referent, which is
+ // entirely contained in some Rust allocation, `A`.
+ // 1. If `ptr`'s referent is not zero sized, then `A`, by invariant on
+ // `&'a mut T`, is guaranteed to live for at least `'a`.
+ //
+ // [1] Per https://doc.rust-lang.org/1.85.0/std/primitive.reference.html#safety:
+ //
+ // For all types, `T: ?Sized`, and for all `t: &T` or `t: &mut T`,
+ // when such values cross an API boundary, the following invariants
+ // must generally be upheld:
+ // ...
+ // - if `size_of_val(t) > 0`, then `t` is dereferenceable for
+ // `size_of_val(t)` many bytes
+ //
+ // If `t` points at address `a`, being “dereferenceable” for N bytes
+ // means that the memory range `[a, a + N)` is all contained within a
+ // single allocated object.
+ unsafe { Self::new(ptr) }
+ }
+
+ /// # Safety
+ ///
+ /// The caller may assume that the resulting `PtrInner` addresses the subset
+ /// of the bytes of `self`'s referent addressed by `C::project(self)`.
+ #[must_use]
+ #[inline(always)]
+ pub fn project<U: ?Sized, C: cast::Project<T, U>>(self) -> PtrInner<'a, U> {
+ let projected_raw = C::project(self);
+
+ // SAFETY: `self`'s referent lives at a `NonNull` address, and is either
+ // zero-sized or lives in an allocation. In either case, it does not
+ // wrap around the address space [1], and so none of the addresses
+ // contained in it or one-past-the-end of it are null.
+ //
+ // By invariant on `C: Project`, `C::project` is a provenance-preserving
+ // projection which preserves or shrinks the set of referent bytes, so
+ // `projected_raw` references a subset of `self`'s referent, and so it
+ // cannot be null.
+ //
+ // [1] https://doc.rust-lang.org/1.92.0/std/ptr/index.html#allocation
+ let projected_non_null = unsafe { NonNull::new_unchecked(projected_raw) };
+
+ // SAFETY: As described in the preceding safety comment, `projected_raw`,
+ // and thus `projected_non_null`, addresses a subset of `self`'s
+ // referent. Thus, `projected_non_null` either:
+ // - Addresses zero bytes or,
+ // - Addresses a subset of the referent of `self`. In this case, `self`
+ // has provenance for its referent, which lives in an allocation.
+ // Since `projected_non_null` was constructed using a sequence of
+ // provenance-preserving operations, it also has provenance for its
+ // referent and that referent lives in an allocation. By invariant on
+ // `self`, that allocation lives for `'a`.
+ unsafe { PtrInner::new(projected_non_null) }
+ }
+}
+
+#[allow(clippy::needless_lifetimes)]
+impl<'a, T> PtrInner<'a, T>
+where
+ T: ?Sized + KnownLayout,
+{
+ /// Extracts the metadata of this `ptr`.
+ #[inline]
+ #[must_use]
+ pub fn meta(self) -> MetadataOf<T> {
+ let meta = T::pointer_to_metadata(self.as_ptr());
+ // SAFETY: By invariant on `PtrInner`, `self.as_non_null()` addresses no
+ // more than `isize::MAX` bytes.
+ unsafe { MetadataOf::new_unchecked(meta) }
+ }
+
+ /// Produces a `PtrInner` with the same address and provenance as `self` but
+ /// the given `meta`.
+ ///
+ /// # Safety
+ ///
+ /// The caller promises that if `self`'s referent is not zero sized, then
+ /// a pointer constructed from its address with the given `meta` metadata
+ /// will address a subset of the allocation pointed to by `self`.
+ #[inline]
+ #[must_use]
+ pub unsafe fn with_meta(self, meta: T::PointerMetadata) -> Self
+ where
+ T: KnownLayout,
+ {
+ let raw = T::raw_from_ptr_len(self.as_non_null().cast(), meta);
+
+ // SAFETY:
+ //
+ // Lemma 0: `raw` either addresses zero bytes, or addresses a subset of
+ // the allocation pointed to by `self` and has the same
+ // provenance as `self`. Proof: `raw` is constructed using
+ // provenance-preserving operations, and the caller has
+ // promised that, if `self`'s referent is not zero-sized, the
+ // resulting pointer addresses a subset of the allocation
+ // pointed to by `self`.
+ //
+ // 0. Per Lemma 0 and by invariant on `self`, if `ptr`'s referent is not
+ // zero sized, then `ptr` is derived from some valid Rust allocation,
+ // `A`.
+ // 1. Per Lemma 0 and by invariant on `self`, if `ptr`'s referent is not
+ // zero sized, then `ptr` has valid provenance for `A`.
+ // 2. Per Lemma 0 and by invariant on `self`, if `ptr`'s referent is not
+ // zero sized, then `ptr` addresses a byte range which is entirely
+ // contained in `A`.
+ // 3. Per Lemma 0 and by invariant on `self`, `ptr` addresses a byte
+ // range whose length fits in an `isize`.
+ // 4. Per Lemma 0 and by invariant on `self`, `ptr` addresses a byte
+ // range which does not wrap around the address space.
+ // 5. Per Lemma 0 and by invariant on `self`, if `ptr`'s referent is not
+ // zero sized, then `A` is guaranteed to live for at least `'a`.
+ unsafe { PtrInner::new(raw) }
+ }
+}
+
+#[allow(clippy::needless_lifetimes)]
+impl<'a, T> PtrInner<'a, T>
+where
+ T: ?Sized + KnownLayout<PointerMetadata = usize>,
+{
+ /// Splits `T` in two.
+ ///
+ /// # Safety
+ ///
+ /// The caller promises that:
+ /// - `l_len.get() <= self.meta()`.
+ ///
+ /// ## (Non-)Overlap
+ ///
+ /// Given `let (left, right) = ptr.split_at(l_len)`, it is guaranteed that
+ /// `left` and `right` are contiguous and non-overlapping if
+ /// `l_len.padding_needed_for() == 0`. This is true for all `[T]`.
+ ///
+ /// If `l_len.padding_needed_for() != 0`, then the left pointer will overlap
+ /// the right pointer to satisfy `T`'s padding requirements.
+ #[inline]
+ #[must_use]
+ pub unsafe fn split_at_unchecked(
+ self,
+ l_len: crate::util::MetadataOf<T>,
+ ) -> (Self, PtrInner<'a, [T::Elem]>)
+ where
+ T: SplitAt,
+ {
+ let l_len = l_len.get();
+
+ // SAFETY: The caller promises that `l_len.get() <= self.meta()`.
+ // Trivially, `0 <= l_len`.
+ let left = unsafe { self.with_meta(l_len) };
+
+ let right = self.trailing_slice();
+ // SAFETY: The caller promises that `l_len <= self.meta() = slf.meta()`.
+ // Trivially, `slf.meta() <= slf.meta()`.
+ let right = unsafe { right.slice_unchecked(l_len..self.meta().get()) };
+
+ // SAFETY: If `l_len.padding_needed_for() == 0`, then `left` and `right`
+ // are non-overlapping. Proof: `left` is constructed `slf` with `l_len`
+ // as its (exclusive) upper bound. If `l_len.padding_needed_for() == 0`,
+ // then `left` requires no trailing padding following its final element.
+ // Since `right` is constructed from `slf`'s trailing slice with `l_len`
+ // as its (inclusive) lower bound, no byte is referred to by both
+ // pointers.
+ //
+ // Conversely, `l_len.padding_needed_for() == N`, where `N
+ // > 0`, `left` requires `N` bytes of trailing padding following its
+ // final element. Since `right` is constructed from the trailing slice
+ // of `slf` with `l_len` as its (inclusive) lower bound, the first `N`
+ // bytes of `right` are aliased by `left`.
+ (left, right)
+ }
+
+ /// Produces the trailing slice of `self`.
+ #[inline]
+ #[must_use]
+ pub fn trailing_slice(self) -> PtrInner<'a, [T::Elem]>
+ where
+ T: SplitAt,
+ {
+ let offset = crate::trailing_slice_layout::<T>().offset;
+
+ let bytes = self.as_non_null().cast::<u8>().as_ptr();
+
+ // SAFETY:
+ // - By invariant on `T: KnownLayout`, `T::LAYOUT` describes `T`'s
+ // layout. `offset` is the offset of the trailing slice within `T`,
+ // which is by definition in-bounds or one byte past the end of any
+ // `T`, regardless of metadata. By invariant on `PtrInner`, `self`
+ // (and thus `bytes`) points to a byte range of size `<= isize::MAX`,
+ // and so `offset <= isize::MAX`. Since `size_of::<u8>() == 1`,
+ // `offset * size_of::<u8>() <= isize::MAX`.
+ // - If `offset > 0`, then by invariant on `PtrInner`, `self` (and thus
+ // `bytes`) points to a byte range entirely contained within the same
+ // allocated object as `self`. As explained above, this offset results
+ // in a pointer to or one byte past the end of this allocated object.
+ let bytes = unsafe { bytes.add(offset) };
+
+ // SAFETY: By the preceding safety argument, `bytes` is within or one
+ // byte past the end of the same allocated object as `self`, which
+ // ensures that it is non-null.
+ let bytes = unsafe { NonNull::new_unchecked(bytes) };
+
+ let ptr = KnownLayout::raw_from_ptr_len(bytes, self.meta().get());
+
+ // SAFETY:
+ // 0. If `ptr`'s referent is not zero sized, then `ptr` is derived from
+ // some valid Rust allocation, `A`, because `ptr` is derived from
+ // the same allocated object as `self`.
+ // 1. If `ptr`'s referent is not zero sized, then `ptr` has valid
+ // provenance for `A` because `raw` is derived from the same
+ // allocated object as `self` via provenance-preserving operations.
+ // 2. If `ptr`'s referent is not zero sized, then `ptr` addresses a byte
+ // range which is entirely contained in `A`, by previous safety proof
+ // on `bytes`.
+ // 3. `ptr` addresses a byte range whose length fits in an `isize`, by
+ // consequence of #2.
+ // 4. `ptr` addresses a byte range which does not wrap around the
+ // address space, by consequence of #2.
+ // 5. If `ptr`'s referent is not zero sized, then `A` is guaranteed to
+ // live for at least `'a`, because `ptr` is derived from `self`.
+ unsafe { PtrInner::new(ptr) }
+ }
+}
+
+#[allow(clippy::needless_lifetimes)]
+impl<'a, T> PtrInner<'a, [T]> {
+ /// Creates a pointer which addresses the given `range` of self.
+ ///
+ /// # Safety
+ ///
+ /// `range` is a valid range (`start <= end`) and `end <= self.meta()`.
+ #[inline]
+ #[must_use]
+ pub unsafe fn slice_unchecked(self, range: Range<usize>) -> Self {
+ let base = self.as_non_null().cast::<T>().as_ptr();
+
+ // SAFETY: The caller promises that `start <= end <= self.meta()`. By
+ // invariant, if `self`'s referent is not zero-sized, then `self` refers
+ // to a byte range which is contained within a single allocation, which
+ // is no more than `isize::MAX` bytes long, and which does not wrap
+ // around the address space. Thus, this pointer arithmetic remains
+ // in-bounds of the same allocation, and does not wrap around the
+ // address space. The offset (in bytes) does not overflow `isize`.
+ //
+ // If `self`'s referent is zero-sized, then these conditions are
+ // trivially satisfied.
+ let base = unsafe { base.add(range.start) };
+
+ // SAFETY: The caller promises that `start <= end`, and so this will not
+ // underflow.
+ #[allow(unstable_name_collisions)]
+ let len = unsafe { range.end.unchecked_sub(range.start) };
+
+ let ptr = core::ptr::slice_from_raw_parts_mut(base, len);
+
+ // SAFETY: By invariant, `self`'s referent is either a ZST or lives
+ // entirely in an allocation. `ptr` points inside of or one byte past
+ // the end of that referent. Thus, in either case, `ptr` is non-null.
+ let ptr = unsafe { NonNull::new_unchecked(ptr) };
+
+ // SAFETY:
+ //
+ // Lemma 0: `ptr` addresses a subset of the bytes addressed by `self`,
+ // and has the same provenance. Proof: The caller guarantees
+ // that `start <= end <= self.meta()`. Thus, `base` is
+ // in-bounds of `self`, and `base + (end - start)` is also
+ // in-bounds of self. Finally, `ptr` is constructed using
+ // provenance-preserving operations.
+ //
+ // 0. Per Lemma 0 and by invariant on `self`, if `ptr`'s referent is not
+ // zero sized, then `ptr` has valid provenance for its referent,
+ // which is entirely contained in some Rust allocation, `A`.
+ // 1. Per Lemma 0 and by invariant on `self`, if `ptr`'s referent is not
+ // zero sized, then `A` is guaranteed to live for at least `'a`.
+ unsafe { PtrInner::new(ptr) }
+ }
+
+ /// Iteratively projects the elements `PtrInner<T>` from `PtrInner<[T]>`.
+ #[inline]
+ pub fn iter(&self) -> impl Iterator<Item = PtrInner<'a, T>> {
+ // FIXME(#429): Once `NonNull::cast` documents that it preserves
+ // provenance, cite those docs.
+ let base = self.as_non_null().cast::<T>().as_ptr();
+ (0..self.meta().get()).map(move |i| {
+ // FIXME(https://github.com/rust-lang/rust/issues/74265): Use
+ // `NonNull::get_unchecked_mut`.
+
+ // SAFETY: If the following conditions are not satisfied
+ // `pointer::cast` may induce Undefined Behavior [1]:
+ //
+ // > - The computed offset, `count * size_of::<T>()` bytes, must not
+ // > overflow `isize``.
+ // > - If the computed offset is non-zero, then `self` must be
+ // > derived from a pointer to some allocated object, and the
+ // > entire memory range between `self` and the result must be in
+ // > bounds of that allocated object. In particular, this range
+ // > must not “wrap around” the edge of the address space.
+ //
+ // [1] https://doc.rust-lang.org/std/primitive.pointer.html#method.add
+ //
+ // We satisfy both of these conditions here:
+ // - By invariant on `Ptr`, `self` addresses a byte range whose
+ // length fits in an `isize`. Since `elem` is contained in `self`,
+ // the computed offset of `elem` must fit within `isize.`
+ // - If the computed offset is non-zero, then this means that the
+ // referent is not zero-sized. In this case, `base` points to an
+ // allocated object (by invariant on `self`). Thus:
+ // - By contract, `self.meta()` accurately reflects the number of
+ // elements in the slice. `i` is in bounds of `c.meta()` by
+ // construction, and so the result of this addition cannot
+ // overflow past the end of the allocation referred to by `c`.
+ // - By invariant on `Ptr`, `self` addresses a byte range which
+ // does not wrap around the address space. Since `elem` is
+ // contained in `self`, the computed offset of `elem` must wrap
+ // around the address space.
+ //
+ // FIXME(#429): Once `pointer::add` documents that it preserves
+ // provenance, cite those docs.
+ let elem = unsafe { base.add(i) };
+
+ // SAFETY: `elem` must not be null. `base` is constructed from a
+ // `NonNull` pointer, and the addition that produces `elem` must not
+ // overflow or wrap around, so `elem >= base > 0`.
+ //
+ // FIXME(#429): Once `NonNull::new_unchecked` documents that it
+ // preserves provenance, cite those docs.
+ let elem = unsafe { NonNull::new_unchecked(elem) };
+
+ // SAFETY: The safety invariants of `Ptr::new` (see definition) are
+ // satisfied:
+ // 0. If `elem`'s referent is not zero sized, then `elem` has valid
+ // provenance for its referent, because it derived from `self`
+ // using a series of provenance-preserving operations, and
+ // because `self` has valid provenance for its referent. By the
+ // same argument, `elem`'s referent is entirely contained within
+ // the same allocated object as `self`'s referent.
+ // 1. If `elem`'s referent is not zero sized, then the allocation of
+ // `elem` is guaranteed to live for at least `'a`, because `elem`
+ // is entirely contained in `self`, which lives for at least `'a`
+ // by invariant on `Ptr`.
+ unsafe { PtrInner::new(elem) }
+ })
+ }
+}
+
+impl<'a, T, const N: usize> PtrInner<'a, [T; N]> {
+ /// Casts this pointer-to-array into a slice.
+ ///
+ /// # Safety
+ ///
+ /// Callers may assume that the returned `PtrInner` references the same
+ /// address and length as `self`.
+ #[allow(clippy::wrong_self_convention)]
+ #[inline]
+ #[must_use]
+ pub fn as_slice(self) -> PtrInner<'a, [T]> {
+ let start = self.as_non_null().cast::<T>().as_ptr();
+ let slice = core::ptr::slice_from_raw_parts_mut(start, N);
+ // SAFETY: `slice` is not null, because it is derived from `start`
+ // which is non-null.
+ let slice = unsafe { NonNull::new_unchecked(slice) };
+ // SAFETY: Lemma: In the following safety arguments, note that `slice`
+ // is derived from `self` in two steps: first, by casting `self: [T; N]`
+ // to `start: T`, then by constructing a pointer to a slice starting at
+ // `start` of length `N`. As a result, `slice` references exactly the
+ // same allocation as `self`, if any.
+ //
+ // 0. By the above lemma, if `slice`'s referent is not zero sized, then
+ // `slice` has the same referent as `self`. By invariant on `self`,
+ // this referent is entirely contained within some allocation, `A`.
+ // Because `slice` was constructed using provenance-preserving
+ // operations, it has provenance for its entire referent.
+ // 1. By the above lemma, if `slice`'s referent is not zero sized, then
+ // `A` is guaranteed to live for at least `'a`, because it is derived
+ // from the same allocation as `self`, which, by invariant on
+ // `PtrInner`, lives for at least `'a`.
+ unsafe { PtrInner::new(slice) }
+ }
+}
+
+impl<'a> PtrInner<'a, [u8]> {
+ /// Attempts to cast `self` to a `U` using the given cast type.
+ ///
+ /// If `U` is a slice DST and pointer metadata (`meta`) is provided, then
+ /// the cast will only succeed if it would produce an object with the given
+ /// metadata.
+ ///
+ /// Returns `None` if the resulting `U` would be invalidly-aligned, if no
+ /// `U` can fit in `self`, or if the provided pointer metadata describes an
+ /// invalid instance of `U`. On success, returns a pointer to the
+ /// largest-possible `U` which fits in `self`.
+ ///
+ /// # Safety
+ ///
+ /// The caller may assume that this implementation is correct, and may rely
+ /// on that assumption for the soundness of their code. In particular, the
+ /// caller may assume that, if `try_cast_into` returns `Some((ptr,
+ /// remainder))`, then `ptr` and `remainder` refer to non-overlapping byte
+ /// ranges within `self`, and that `ptr` and `remainder` entirely cover
+ /// `self`. Finally:
+ /// - If this is a prefix cast, `ptr` has the same address as `self`.
+ /// - If this is a suffix cast, `remainder` has the same address as `self`.
+ #[inline]
+ pub fn try_cast_into<U>(
+ self,
+ cast_type: CastType,
+ meta: Option<U::PointerMetadata>,
+ ) -> Result<(PtrInner<'a, U>, PtrInner<'a, [u8]>), CastError<Self, U>>
+ where
+ U: 'a + ?Sized + KnownLayout,
+ {
+ // PANICS: By invariant, the byte range addressed by
+ // `self.as_non_null()` does not wrap around the address space. This
+ // implies that the sum of the address (represented as a `usize`) and
+ // length do not overflow `usize`, as required by
+ // `validate_cast_and_convert_metadata`. Thus, this call to
+ // `validate_cast_and_convert_metadata` will only panic if `U` is a DST
+ // whose trailing slice element is zero-sized.
+ let maybe_metadata = MetadataOf::<U>::validate_cast_and_convert_metadata(
+ AsAddress::addr(self.as_ptr()),
+ self.meta(),
+ cast_type,
+ meta,
+ );
+
+ let (elems, split_at) = match maybe_metadata {
+ Ok((elems, split_at)) => (elems, split_at),
+ Err(MetadataCastError::Alignment) => {
+ // SAFETY: Since `validate_cast_and_convert_metadata` returned
+ // an alignment error, `U` must have an alignment requirement
+ // greater than one.
+ let err = unsafe { AlignmentError::<_, U>::new_unchecked(self) };
+ return Err(CastError::Alignment(err));
+ }
+ Err(MetadataCastError::Size) => return Err(CastError::Size(SizeError::new(self))),
+ };
+
+ // SAFETY: `validate_cast_and_convert_metadata` promises to return
+ // `split_at <= self.meta()`.
+ //
+ // Lemma 0: `l_slice` and `r_slice` are non-overlapping. Proof: By
+ // contract on `PtrInner::split_at_unchecked`, the produced `PtrInner`s
+ // are always non-overlapping if `self` is a `[T]`; here it is a `[u8]`.
+ let (l_slice, r_slice) = unsafe { self.split_at_unchecked(split_at) };
+
+ let (target, remainder) = match cast_type {
+ CastType::Prefix => (l_slice, r_slice),
+ CastType::Suffix => (r_slice, l_slice),
+ };
+
+ let base = target.as_non_null().cast::<u8>();
+
+ let ptr = U::raw_from_ptr_len(base, elems.get());
+
+ // SAFETY:
+ // 0. By invariant, if `target`'s referent is not zero sized, then
+ // `target` has provenance valid for some Rust allocation, `A`.
+ // Because `ptr` is derived from `target` via provenance-preserving
+ // operations, `ptr` will also have provenance valid for its entire
+ // referent.
+ // 1. `validate_cast_and_convert_metadata` promises that the object
+ // described by `elems` and `split_at` lives at a byte range which is
+ // a subset of the input byte range. Thus, by invariant, if
+ // `target`'s referent is not zero sized, then `target` refers to an
+ // allocation which is guaranteed to live for at least `'a`, and thus
+ // so does `ptr`.
+ Ok((unsafe { PtrInner::new(ptr) }, remainder))
+ }
+}
+
+#[cfg(test)]
+mod tests {
+ use super::*;
+ use crate::*;
+
+ #[test]
+ fn test_meta() {
+ let arr = [1; 16];
+ let dst = <[u8]>::ref_from_bytes(&arr[..]).unwrap();
+ let ptr = PtrInner::from_ref(dst);
+ assert_eq!(ptr.meta().get(), 16);
+
+ // SAFETY: 8 is less than 16
+ let ptr = unsafe { ptr.with_meta(8) };
+
+ assert_eq!(ptr.meta().get(), 8);
+ }
+
+ #[test]
+ fn test_split_at() {
+ fn test_split_at<const OFFSET: usize, const BUFFER_SIZE: usize>() {
+ #[derive(FromBytes, KnownLayout, SplitAt, Immutable)]
+ #[repr(C)]
+ struct SliceDst<const OFFSET: usize> {
+ prefix: [u8; OFFSET],
+ trailing: [u8],
+ }
+
+ let n: usize = BUFFER_SIZE - OFFSET;
+ let arr = [1; BUFFER_SIZE];
+ let dst = SliceDst::<OFFSET>::ref_from_bytes(&arr[..]).unwrap();
+ let ptr = PtrInner::from_ref(dst);
+ for i in 0..=n {
+ assert_eq!(ptr.meta().get(), n);
+ // SAFETY: `i` is in bounds by construction.
+ let i = unsafe { MetadataOf::new_unchecked(i) };
+ // SAFETY: `i` is in bounds by construction.
+ let (l, r) = unsafe { ptr.split_at_unchecked(i) };
+ // SAFETY: Points to a valid value by construction.
+ #[allow(clippy::undocumented_unsafe_blocks, clippy::as_conversions)]
+ // Clippy false positive
+ let l_sum: usize = l
+ .trailing_slice()
+ .iter()
+ .map(
+ #[inline(always)]
+ |ptr| unsafe { core::ptr::read_unaligned(ptr.as_ptr()) } as usize,
+ )
+ .sum();
+ // SAFETY: Points to a valid value by construction.
+ #[allow(clippy::undocumented_unsafe_blocks, clippy::as_conversions)]
+ // Clippy false positive
+ let r_sum: usize = r
+ .iter()
+ .map(
+ #[inline(always)]
+ |ptr| unsafe { core::ptr::read_unaligned(ptr.as_ptr()) } as usize,
+ )
+ .sum();
+ assert_eq!(l_sum, i.get());
+ assert_eq!(r_sum, n - i.get());
+ assert_eq!(l_sum + r_sum, n);
+ }
+ }
+
+ test_split_at::<0, 16>();
+ test_split_at::<1, 17>();
+ test_split_at::<2, 18>();
+ }
+
+ #[test]
+ fn test_trailing_slice() {
+ fn test_trailing_slice<const OFFSET: usize, const BUFFER_SIZE: usize>() {
+ #[derive(FromBytes, KnownLayout, SplitAt, Immutable)]
+ #[repr(C)]
+ struct SliceDst<const OFFSET: usize> {
+ prefix: [u8; OFFSET],
+ trailing: [u8],
+ }
+
+ let n: usize = BUFFER_SIZE - OFFSET;
+ let arr = [1; BUFFER_SIZE];
+ let dst = SliceDst::<OFFSET>::ref_from_bytes(&arr[..]).unwrap();
+ let ptr = PtrInner::from_ref(dst);
+
+ assert_eq!(ptr.meta().get(), n);
+ let trailing = ptr.trailing_slice();
+ assert_eq!(trailing.meta().get(), n);
+
+ assert_eq!(
+ // SAFETY: We assume this to be sound for the sake of this test,
+ // which will fail, here, in miri, if the safety precondition of
+ // `offset_of` is not satisfied.
+ unsafe {
+ #[allow(clippy::as_conversions)]
+ let offset = (trailing.as_ptr() as *mut u8).offset_from(ptr.as_ptr() as *mut _);
+ offset
+ },
+ isize::try_from(OFFSET).unwrap(),
+ );
+
+ // SAFETY: Points to a valid value by construction.
+ #[allow(clippy::undocumented_unsafe_blocks, clippy::as_conversions)]
+ // Clippy false positive
+ let trailing: usize = trailing
+ .iter()
+ .map(|ptr| unsafe { core::ptr::read_unaligned(ptr.as_ptr()) } as usize)
+ .sum();
+
+ assert_eq!(trailing, n);
+ }
+
+ test_trailing_slice::<0, 16>();
+ test_trailing_slice::<1, 17>();
+ test_trailing_slice::<2, 18>();
+ }
+ #[test]
+ fn test_ptr_inner_clone() {
+ let mut x = 0u8;
+ let p = PtrInner::from_mut(&mut x);
+ #[allow(clippy::clone_on_copy)]
+ let p2 = p.clone();
+ assert_eq!(p.as_non_null(), p2.as_non_null());
+ }
+}
diff --git a/rust/zerocopy/src/pointer/invariant.rs b/rust/zerocopy/src/pointer/invariant.rs
new file mode 100644
index 0000000..1802d23
--- /dev/null
+++ b/rust/zerocopy/src/pointer/invariant.rs
@@ -0,0 +1,298 @@
+// SPDX-License-Identifier: (BSD-2-Clause OR Apache-2.0) OR MIT
+
+// Copyright 2024 The Fuchsia Authors
+//
+// Licensed under a BSD-style license <LICENSE-BSD>, Apache License, Version 2.0
+// <LICENSE-APACHE or https://www.apache.org/licenses/LICENSE-2.0>, or the MIT
+// license <LICENSE-MIT or https://opensource.org/licenses/MIT>, at your option.
+// This file may not be copied, modified, or distributed except according to
+// those terms.
+
+#![allow(missing_copy_implementations, missing_debug_implementations, missing_docs)]
+
+//! The parameterized invariants of a [`Ptr`][super::Ptr].
+//!
+//! Invariants are encoded as ([`Aliasing`], [`Alignment`], [`Validity`])
+//! triples implementing the [`Invariants`] trait.
+
+/// The invariants of a [`Ptr`][super::Ptr].
+pub trait Invariants: Sealed {
+ type Aliasing: Aliasing;
+ type Alignment: Alignment;
+ type Validity: Validity;
+}
+
+impl<A: Aliasing, AA: Alignment, V: Validity> Invariants for (A, AA, V) {
+ type Aliasing = A;
+ type Alignment = AA;
+ type Validity = V;
+}
+
+/// The aliasing invariant of a [`Ptr`][super::Ptr].
+///
+/// All aliasing invariants must permit reading from the bytes of a pointer's
+/// referent which are not covered by [`UnsafeCell`]s.
+///
+/// [`UnsafeCell`]: core::cell::UnsafeCell
+pub trait Aliasing: Sealed {
+ /// Is `Self` [`Exclusive`]?
+ #[doc(hidden)]
+ const IS_EXCLUSIVE: bool;
+}
+
+/// The alignment invariant of a [`Ptr`][super::Ptr].
+pub trait Alignment: Sealed {
+ #[doc(hidden)]
+ #[must_use]
+ fn read<T, I, R>(ptr: crate::Ptr<'_, T, I>) -> T
+ where
+ T: Copy + Read<I::Aliasing, R>,
+ I: Invariants<Alignment = Self, Validity = Valid>,
+ I::Aliasing: Reference;
+}
+
+/// The validity invariant of a [`Ptr`][super::Ptr].
+///
+/// # Safety
+///
+/// In this section, we will use `Ptr<T, V>` as a shorthand for `Ptr<T, I:
+/// Invariants<Validity = V>>` for brevity.
+///
+/// Each `V: Validity` defines a set of bit values which may appear in the
+/// referent of a `Ptr<T, V>`, denoted `S(T, V)`. Each `V: Validity`, in its
+/// documentation, provides a definition of `S(T, V)` which must be valid for
+/// all `T: ?Sized`. Any `V: Validity` must guarantee that this set is only a
+/// function of the *bit validity* of the referent type, `T`, and not of any
+/// other property of `T`. As a consequence, given `V: Validity`, `T`, and `U`
+/// where `T` and `U` have the same bit validity, `S(V, T) = S(V, U)`.
+///
+/// It is guaranteed that the referent of any `ptr: Ptr<T, V>` is a member of
+/// `S(T, V)`. Unsafe code must ensure that this guarantee will be upheld for
+/// any existing `Ptr`s or any `Ptr`s that that code creates.
+///
+/// An important implication of this guarantee is that it restricts what
+/// transmutes are sound, where "transmute" is used in this context to refer to
+/// changing the referent type or validity invariant of a `Ptr`, as either
+/// change may change the set of bit values permitted to appear in the referent.
+/// In particular, the following are necessary (but not sufficient) conditions
+/// in order for a transmute from `src: Ptr<T, V>` to `dst: Ptr<U, W>` to be
+/// sound:
+/// - If `S(T, V) = S(U, W)`, then no restrictions apply; otherwise,
+/// - If `dst` permits mutation of its referent (e.g. via `Exclusive` aliasing
+/// or interior mutation under `Shared` aliasing), then it must hold that
+/// `S(T, V) ⊇ S(U, W)` - in other words, the transmute must not expand the
+/// set of allowed referent bit patterns. A violation of this requirement
+/// would permit using `dst` to write `x` where `x ∈ S(U, W)` but `x ∉ S(T,
+/// V)`, which would violate the guarantee that `src`'s referent may only
+/// contain values in `S(T, V)`.
+/// - If the referent may be mutated without going through `dst` while `dst` is
+/// live (e.g. via interior mutation on a `Shared`-aliased `Ptr` or `&`
+/// reference), then it must hold that `S(T, V) ⊆ S(U, W)` - in other words,
+/// the transmute must not shrink the set of allowed referent bit patterns. A
+/// violation of this requirement would permit using `src` or another
+/// mechanism (e.g. a `&` reference used to derive `src`) to write `x` where
+/// `x ∈ S(T, V)` but `x ∉ S(U, W)`, which would violate the guarantee that
+/// `dst`'s referent may only contain values in `S(U, W)`.
+pub unsafe trait Validity: Sealed {
+ const KIND: ValidityKind;
+}
+
+pub enum ValidityKind {
+ Uninit,
+ AsInitialized,
+ Initialized,
+ Valid,
+}
+
+/// An [`Aliasing`] invariant which is either [`Shared`] or [`Exclusive`].
+///
+/// # Safety
+///
+/// Given `A: Reference`, callers may assume that either `A = Shared` or `A =
+/// Exclusive`.
+pub trait Reference: Aliasing + Sealed {}
+
+/// The `Ptr<'a, T>` adheres to the aliasing rules of a `&'a T`.
+///
+/// The referent of a shared-aliased `Ptr` may be concurrently referenced by any
+/// number of shared-aliased `Ptr` or `&T` references, or by any number of
+/// `Ptr<U>` or `&U` references as permitted by `T`'s library safety invariants,
+/// and may not be concurrently referenced by any exclusively-aliased `Ptr`s or
+/// `&mut` references. The referent must not be mutated, except via
+/// [`UnsafeCell`]s, and only when permitted by `T`'s library safety invariants.
+///
+/// [`UnsafeCell`]: core::cell::UnsafeCell
+pub enum Shared {}
+impl Aliasing for Shared {
+ const IS_EXCLUSIVE: bool = false;
+}
+impl Reference for Shared {}
+
+/// The `Ptr<'a, T>` adheres to the aliasing rules of a `&'a mut T`.
+///
+/// The referent of an exclusively-aliased `Ptr` may not be concurrently
+/// referenced by any other `Ptr`s or references, and may not be accessed (read
+/// or written) other than via this `Ptr`.
+pub enum Exclusive {}
+impl Aliasing for Exclusive {
+ const IS_EXCLUSIVE: bool = true;
+}
+impl Reference for Exclusive {}
+
+/// It is unknown whether the pointer is aligned.
+pub enum Unaligned {}
+
+impl Alignment for Unaligned {
+ #[inline(always)]
+ fn read<T, I, R>(ptr: crate::Ptr<'_, T, I>) -> T
+ where
+ T: Copy + Read<I::Aliasing, R>,
+ I: Invariants<Alignment = Self, Validity = Valid>,
+ I::Aliasing: Reference,
+ {
+ (*ptr.into_unalign().as_ref()).into_inner()
+ }
+}
+
+/// The referent is aligned: for `Ptr<T>`, the referent's address is a multiple
+/// of the `T`'s alignment.
+pub enum Aligned {}
+impl Alignment for Aligned {
+ #[inline(always)]
+ fn read<T, I, R>(ptr: crate::Ptr<'_, T, I>) -> T
+ where
+ T: Copy + Read<I::Aliasing, R>,
+ I: Invariants<Alignment = Self, Validity = Valid>,
+ I::Aliasing: Reference,
+ {
+ *ptr.as_ref()
+ }
+}
+
+/// Any bit pattern is allowed in the `Ptr`'s referent, including uninitialized
+/// bytes.
+pub enum Uninit {}
+// SAFETY: `Uninit`'s validity is well-defined for all `T: ?Sized`, and is not a
+// function of any property of `T` other than its bit validity (in fact, it's
+// not even a property of `T`'s bit validity, but this is more than we are
+// required to uphold).
+unsafe impl Validity for Uninit {
+ const KIND: ValidityKind = ValidityKind::Uninit;
+}
+
+/// The byte ranges initialized in `T` are also initialized in the referent of a
+/// `Ptr<T>`.
+///
+/// Formally: uninitialized bytes may only be present in `Ptr<T>`'s referent
+/// where they are guaranteed to be present in `T`. This is a dynamic property:
+/// if, at a particular byte offset, a valid enum discriminant is set, the
+/// subsequent bytes may only have uninitialized bytes as specified by the
+/// corresponding enum.
+///
+/// Formally, given `len = size_of_val_raw(ptr)`, at every byte offset, `b`, in
+/// the range `[0, len)`:
+/// - If, in any instance `t: T` of length `len`, the byte at offset `b` in `t`
+/// is initialized, then the byte at offset `b` within `*ptr` must be
+/// initialized.
+/// - Let `c` be the contents of the byte range `[0, b)` in `*ptr`. Let `S` be
+/// the subset of valid instances of `T` of length `len` which contain `c` in
+/// the offset range `[0, b)`. If, in any instance of `t: T` in `S`, the byte
+/// at offset `b` in `t` is initialized, then the byte at offset `b` in `*ptr`
+/// must be initialized.
+///
+/// Pragmatically, this means that if `*ptr` is guaranteed to contain an enum
+/// type at a particular offset, and the enum discriminant stored in `*ptr`
+/// corresponds to a valid variant of that enum type, then it is guaranteed
+/// that the appropriate bytes of `*ptr` are initialized as defined by that
+/// variant's bit validity (although note that the variant may contain another
+/// enum type, in which case the same rules apply depending on the state of
+/// its discriminant, and so on recursively).
+pub enum AsInitialized {}
+// SAFETY: `AsInitialized`'s validity is well-defined for all `T: ?Sized`, and
+// is not a function of any property of `T` other than its bit validity.
+unsafe impl Validity for AsInitialized {
+ const KIND: ValidityKind = ValidityKind::AsInitialized;
+}
+
+/// The byte ranges in the referent are fully initialized. In other words, if
+/// the referent is `N` bytes long, then it contains a bit-valid `[u8; N]`.
+pub enum Initialized {}
+// SAFETY: `Initialized`'s validity is well-defined for all `T: ?Sized`, and is
+// not a function of any property of `T` other than its bit validity (in fact,
+// it's not even a property of `T`'s bit validity, but this is more than we are
+// required to uphold).
+unsafe impl Validity for Initialized {
+ const KIND: ValidityKind = ValidityKind::Initialized;
+}
+
+/// The referent of a `Ptr<T>` is valid for `T`, upholding bit validity and any
+/// library safety invariants.
+pub enum Valid {}
+// SAFETY: `Valid`'s validity is well-defined for all `T: ?Sized`, and is not a
+// function of any property of `T` other than its bit validity.
+unsafe impl Validity for Valid {
+ const KIND: ValidityKind = ValidityKind::Valid;
+}
+
+/// # Safety
+///
+/// `DT: CastableFrom<ST, SV, DV>` is sound if `SV = DV = Uninit` or `SV = DV =
+/// Initialized`.
+pub unsafe trait CastableFrom<ST: ?Sized, SV, DV> {}
+
+// SAFETY: `SV = DV = Uninit`.
+unsafe impl<ST: ?Sized, DT: ?Sized> CastableFrom<ST, Uninit, Uninit> for DT {}
+// SAFETY: `SV = DV = Initialized`.
+unsafe impl<ST: ?Sized, DT: ?Sized> CastableFrom<ST, Initialized, Initialized> for DT {}
+
+/// [`Ptr`](crate::Ptr) referents that permit unsynchronized read operations.
+///
+/// `T: Read<A, R>` implies that a pointer to `T` with aliasing `A` permits
+/// unsynchronized read operations. This can be because `A` is [`Exclusive`] or
+/// because `T` does not permit interior mutation.
+///
+/// # Safety
+///
+/// `T: Read<A, R>` if either of the following conditions holds:
+/// - `A` is [`Exclusive`]
+/// - `T` implements [`Immutable`](crate::Immutable)
+///
+/// As a consequence, if `T: Read<A, R>`, then any `Ptr<T, (A, ...)>` is
+/// permitted to perform unsynchronized reads from its referent.
+pub trait Read<A: Aliasing, R> {}
+
+impl<A: Aliasing, T: ?Sized + crate::Immutable> Read<A, BecauseImmutable> for T {}
+impl<T: ?Sized> Read<Exclusive, BecauseExclusive> for T {}
+
+/// Unsynchronized reads are permitted because only one live [`Ptr`](crate::Ptr)
+/// or reference may exist to the referent bytes at a time.
+#[derive(Copy, Clone, Debug)]
+pub enum BecauseExclusive {}
+
+/// Unsynchronized reads are permitted because no live [`Ptr`](crate::Ptr)s or
+/// references permit interior mutation.
+#[derive(Copy, Clone, Debug)]
+pub enum BecauseImmutable {}
+
+use sealed::Sealed;
+mod sealed {
+ use super::*;
+
+ pub trait Sealed {}
+
+ impl Sealed for Shared {}
+ impl Sealed for Exclusive {}
+
+ impl Sealed for Unaligned {}
+ impl Sealed for Aligned {}
+
+ impl Sealed for Uninit {}
+ impl Sealed for AsInitialized {}
+ impl Sealed for Initialized {}
+ impl Sealed for Valid {}
+
+ impl<A: Sealed, AA: Sealed, V: Sealed> Sealed for (A, AA, V) {}
+
+ impl Sealed for BecauseImmutable {}
+ impl Sealed for BecauseExclusive {}
+}
diff --git a/rust/zerocopy/src/pointer/mod.rs b/rust/zerocopy/src/pointer/mod.rs
new file mode 100644
index 0000000..3461f7f
--- /dev/null
+++ b/rust/zerocopy/src/pointer/mod.rs
@@ -0,0 +1,410 @@
+// SPDX-License-Identifier: (BSD-2-Clause OR Apache-2.0) OR MIT
+
+// Copyright 2023 The Fuchsia Authors
+//
+// Licensed under a BSD-style license <LICENSE-BSD>, Apache License, Version 2.0
+// <LICENSE-APACHE or https://www.apache.org/licenses/LICENSE-2.0>, or the MIT
+// license <LICENSE-MIT or https://opensource.org/licenses/MIT>, at your option.
+// This file may not be copied, modified, or distributed except according to
+// those terms.
+
+//! Abstractions over raw pointers.
+
+#![allow(missing_docs)]
+
+mod inner;
+pub mod invariant;
+mod ptr;
+pub mod transmute;
+
+pub use inner::PtrInner;
+pub use invariant::{BecauseExclusive, BecauseImmutable, Read};
+pub use ptr::{Ptr, TryWithError};
+pub use transmute::*;
+
+use crate::wrappers::ReadOnly;
+
+/// A shorthand for a maybe-valid, maybe-aligned reference. Used as the argument
+/// to [`TryFromBytes::is_bit_valid`].
+///
+/// [`TryFromBytes::is_bit_valid`]: crate::TryFromBytes::is_bit_valid
+pub type Maybe<'a, T, Alignment = invariant::Unaligned> =
+ Ptr<'a, ReadOnly<T>, (invariant::Shared, Alignment, invariant::Initialized)>;
+
+/// Checks if the referent is zeroed.
+pub(crate) fn is_zeroed<T, I>(ptr: Ptr<'_, T, I>) -> bool
+where
+ T: crate::Immutable + crate::KnownLayout,
+ I: invariant::Invariants<Validity = invariant::Initialized>,
+ I::Aliasing: invariant::Reference,
+{
+ ptr.as_bytes().as_ref().iter().all(
+ #[inline(always)]
+ |&byte| byte == 0,
+ )
+}
+
+pub mod cast {
+ use core::{marker::PhantomData, mem};
+
+ use crate::{
+ layout::{SizeInfo, TrailingSliceLayout},
+ HasField, KnownLayout, PtrInner,
+ };
+
+ /// A pointer cast or projection.
+ ///
+ /// # Safety
+ ///
+ /// The implementation of `project` must satisfy its safety post-condition.
+ pub unsafe trait Project<Src: ?Sized, Dst: ?Sized> {
+ /// Projects a pointer from `Src` to `Dst`.
+ ///
+ /// Users should generally not call `project` directly, and instead
+ /// should use high-level APIs like [`PtrInner::project`] or
+ /// [`Ptr::project`].
+ ///
+ /// [`Ptr::project`]: crate::pointer::Ptr::project
+ ///
+ /// # Safety
+ ///
+ /// The returned pointer refers to a non-strict subset of the bytes of
+ /// `src`'s referent, and has the same provenance as `src`.
+ fn project(src: PtrInner<'_, Src>) -> *mut Dst;
+ }
+
+ /// A [`Project`] which preserves the address of the referent – a pointer
+ /// cast.
+ ///
+ /// # Safety
+ ///
+ /// A `Cast` projection must preserve the address of the referent. It may
+ /// shrink the set of referent bytes, and it may change the referent's type.
+ pub unsafe trait Cast<Src: ?Sized, Dst: ?Sized>: Project<Src, Dst> {}
+
+ /// A [`Cast`] which does not shrink the set of referent bytes.
+ ///
+ /// # Safety
+ ///
+ /// A `CastExact` projection must preserve the set of referent bytes.
+ pub unsafe trait CastExact<Src: ?Sized, Dst: ?Sized>: Cast<Src, Dst> {}
+
+ /// A no-op pointer cast.
+ #[derive(Default, Copy, Clone)]
+ #[allow(missing_debug_implementations)]
+ pub struct IdCast;
+
+ // SAFETY: `project` returns its argument unchanged, and so it is a
+ // provenance-preserving projection which preserves the set of referent
+ // bytes.
+ unsafe impl<T: ?Sized> Project<T, T> for IdCast {
+ #[inline(always)]
+ fn project(src: PtrInner<'_, T>) -> *mut T {
+ src.as_ptr()
+ }
+ }
+
+ // SAFETY: The `Project::project` impl preserves referent address.
+ unsafe impl<T: ?Sized> Cast<T, T> for IdCast {}
+
+ // SAFETY: The `Project::project` impl preserves referent size.
+ unsafe impl<T: ?Sized> CastExact<T, T> for IdCast {}
+
+ /// A pointer cast which preserves or shrinks the set of referent bytes of
+ /// a statically-sized referent.
+ ///
+ /// # Safety
+ ///
+ /// The implementation of [`Project`] uses a compile-time assertion to
+ /// guarantee that `Dst` is no larger than `Src`. Thus, `CastSized` has a
+ /// sound implementation of [`Project`] for all `Src` and `Dst` – the caller
+ /// may pass any `Src` and `Dst` without being responsible for soundness.
+ #[allow(missing_debug_implementations, missing_copy_implementations)]
+ pub enum CastSized {}
+
+ // SAFETY: By the `static_assert!`, `Dst` is no larger than `Src`,
+ // and so all casts preserve or shrink the set of referent bytes. All
+ // operations preserve provenance.
+ unsafe impl<Src, Dst> Project<Src, Dst> for CastSized {
+ #[inline(always)]
+ fn project(src: PtrInner<'_, Src>) -> *mut Dst {
+ static_assert!(Src, Dst => mem::size_of::<Src>() >= mem::size_of::<Dst>());
+ src.as_ptr().cast::<Dst>()
+ }
+ }
+
+ // SAFETY: The `Project::project` impl preserves referent address.
+ unsafe impl<Src, Dst> Cast<Src, Dst> for CastSized {}
+
+ /// A pointer cast which preserves the set of referent bytes of a
+ /// statically-sized referent.
+ ///
+ /// # Safety
+ ///
+ /// The implementation of [`Project`] uses a compile-time assertion to
+ /// guarantee that `Dst` has the same size as `Src`. Thus, `CastSizedExact`
+ /// has a sound implementation of [`Project`] for all `Src` and `Dst` – the
+ /// caller may pass any `Src` and `Dst` without being responsible for
+ /// soundness.
+ #[allow(missing_debug_implementations, missing_copy_implementations)]
+ pub enum CastSizedExact {}
+
+ // SAFETY: By the `static_assert!`, `Dst` has the same size as `Src`,
+ // and so all casts preserve the set of referent bytes. All operations
+ // preserve provenance.
+ unsafe impl<Src, Dst> Project<Src, Dst> for CastSizedExact {
+ #[inline(always)]
+ fn project(src: PtrInner<'_, Src>) -> *mut Dst {
+ static_assert!(Src, Dst => mem::size_of::<Src>() == mem::size_of::<Dst>());
+ src.as_ptr().cast::<Dst>()
+ }
+ }
+
+ // SAFETY: The `Project::project_raw` impl preserves referent address.
+ unsafe impl<Src, Dst> Cast<Src, Dst> for CastSizedExact {}
+
+ // SAFETY: By the `static_assert!`, `Project::project_raw` impl preserves
+ // referent size.
+ unsafe impl<Src, Dst> CastExact<Src, Dst> for CastSizedExact {}
+
+ /// A pointer cast which preserves or shrinks the set of referent bytes of
+ /// a dynamically-sized referent.
+ ///
+ /// # Safety
+ ///
+ /// The implementation of [`Project`] uses a compile-time assertion to
+ /// guarantee that the cast preserves the set of referent bytes. Thus,
+ /// `CastUnsized` has a sound implementation of [`Project`] for all `Src`
+ /// and `Dst` – the caller may pass any `Src` and `Dst` without being
+ /// responsible for soundness.
+ #[allow(missing_debug_implementations, missing_copy_implementations)]
+ pub enum CastUnsized {}
+
+ // SAFETY: By the `static_assert!`, `Src` and `Dst` are either:
+ // - Both sized and equal in size
+ // - Both slice DSTs with the same trailing slice offset and element size
+ // and with align_of::<Src>() == align_of::<Dst>(). These ensure that any
+ // given pointer metadata encodes the same size for both `Src` and `Dst`
+ // (note that the alignment is required as it affects the amount of
+ // trailing padding). Thus, `project` preserves the set of referent bytes.
+ unsafe impl<Src, Dst> Project<Src, Dst> for CastUnsized
+ where
+ Src: ?Sized + KnownLayout,
+ Dst: ?Sized + KnownLayout<PointerMetadata = Src::PointerMetadata>,
+ {
+ #[inline(always)]
+ fn project(src: PtrInner<'_, Src>) -> *mut Dst {
+ // FIXME: Do we want this to support shrinking casts as well? If so,
+ // we'll need to remove the `CastExact` impl.
+ static_assert!(Src: ?Sized + KnownLayout, Dst: ?Sized + KnownLayout => {
+ let src = <Src as KnownLayout>::LAYOUT;
+ let dst = <Dst as KnownLayout>::LAYOUT;
+ match (src.size_info, dst.size_info) {
+ (SizeInfo::Sized { size: src_size }, SizeInfo::Sized { size: dst_size }) => src_size == dst_size,
+ (
+ SizeInfo::SliceDst(TrailingSliceLayout { offset: src_offset, elem_size: src_elem_size }),
+ SizeInfo::SliceDst(TrailingSliceLayout { offset: dst_offset, elem_size: dst_elem_size })
+ ) => src.align.get() == dst.align.get() && src_offset == dst_offset && src_elem_size == dst_elem_size,
+ _ => false,
+ }
+ });
+
+ let metadata = Src::pointer_to_metadata(src.as_ptr());
+ Dst::raw_from_ptr_len(src.as_non_null().cast::<u8>(), metadata).as_ptr()
+ }
+ }
+
+ // SAFETY: The `Project::project` impl preserves referent address.
+ unsafe impl<Src, Dst> Cast<Src, Dst> for CastUnsized
+ where
+ Src: ?Sized + KnownLayout,
+ Dst: ?Sized + KnownLayout<PointerMetadata = Src::PointerMetadata>,
+ {
+ }
+
+ // SAFETY: By the `static_assert!` in `Project::project`, `Src` and `Dst`
+ // are either:
+ // - Both sized and equal in size
+ // - Both slice DSTs with the same alignment, trailing slice offset, and
+ // element size. These ensure that any given pointer metadata encodes the
+ // same size for both `Src` and `Dst` (note that the alignment is required
+ // as it affects the amount of trailing padding).
+ unsafe impl<Src, Dst> CastExact<Src, Dst> for CastUnsized
+ where
+ Src: ?Sized + KnownLayout,
+ Dst: ?Sized + KnownLayout<PointerMetadata = Src::PointerMetadata>,
+ {
+ }
+
+ /// A field projection
+ ///
+ /// A `Projection` is a [`Project`] which implements projection by
+ /// delegating to an implementation of [`HasField::project`].
+ #[allow(missing_debug_implementations, missing_copy_implementations)]
+ pub struct Projection<F: ?Sized, const VARIANT_ID: i128, const FIELD_ID: i128> {
+ _never: core::convert::Infallible,
+ _phantom: PhantomData<F>,
+ }
+
+ // SAFETY: `HasField::project` has the same safety post-conditions as
+ // `Project::project`.
+ unsafe impl<T: ?Sized, F, const VARIANT_ID: i128, const FIELD_ID: i128> Project<T, T::Type>
+ for Projection<F, VARIANT_ID, FIELD_ID>
+ where
+ T: HasField<F, VARIANT_ID, FIELD_ID>,
+ {
+ #[inline(always)]
+ fn project(src: PtrInner<'_, T>) -> *mut T::Type {
+ T::project(src)
+ }
+ }
+
+ // SAFETY: All `repr(C)` union fields exist at offset 0 within the union [1],
+ // and so any union projection is actually a cast (ie, preserves address).
+ //
+ // [1] Per
+ // https://doc.rust-lang.org/1.92.0/reference/type-layout.html#reprc-unions,
+ // it's not *technically* guaranteed that non-maximally-sized fields
+ // are at offset 0, but it's clear that this is the intention of `repr(C)`
+ // unions. It says:
+ //
+ // > A union declared with `#[repr(C)]` will have the same size and
+ // > alignment as an equivalent C union declaration in the C language for
+ // > the target platform.
+ //
+ // Note that this only mentions size and alignment, not layout. However,
+ // C unions *do* guarantee that all fields start at offset 0. [2]
+ //
+ // This is also reinforced by
+ // https://doc.rust-lang.org/1.92.0/reference/items/unions.html#r-items.union.fields.offset:
+ //
+ // > Fields might have a non-zero offset (except when the C
+ // > representation is used); in that case the bits starting at the
+ // > offset of the fields are read
+ //
+ // [2] Per https://port70.net/~nsz/c/c11/n1570.html#6.7.2.1p16:
+ //
+ // > The size of a union is sufficient to contain the largest of its
+ // > members. The value of at most one of the members can be stored in a
+ // > union object at any time. A pointer to a union object, suitably
+ // > converted, points to each of its members (or if a member is a
+ // > bit-field, then to the unit in which it resides), and vice versa.
+ //
+ // FIXME(https://github.com/rust-lang/unsafe-code-guidelines/issues/595):
+ // Cite the documentation once it's updated.
+ unsafe impl<T: ?Sized, F, const FIELD_ID: i128> Cast<T, T::Type>
+ for Projection<F, { crate::REPR_C_UNION_VARIANT_ID }, FIELD_ID>
+ where
+ T: HasField<F, { crate::REPR_C_UNION_VARIANT_ID }, FIELD_ID>,
+ {
+ }
+
+ /// A transitive sequence of projections.
+ ///
+ /// Given `TU: Project` and `UV: Project`, `TransitiveProject<_, TU, UV>` is
+ /// a [`Project`] which projects by applying `TU` followed by `UV`.
+ ///
+ /// If `TU: Cast` and `UV: Cast`, then `TransitiveProject<_, TU, UV>: Cast`.
+ #[allow(missing_debug_implementations)]
+ pub struct TransitiveProject<U: ?Sized, TU, UV> {
+ _never: core::convert::Infallible,
+ _projections: PhantomData<(TU, UV)>,
+ // On our MSRV (1.56), the debuginfo for a tuple containing both an
+ // uninhabited type and a DST causes an ICE. We split `U` from `TU` and
+ // `UV` to avoid this situation.
+ _u: PhantomData<U>,
+ }
+
+ // SAFETY: Since `TU::project` and `UV::project` are each
+ // provenance-preserving operations which preserve or shrink the set of
+ // referent bytes, so is their composition.
+ unsafe impl<T, U, V, TU, UV> Project<T, V> for TransitiveProject<U, TU, UV>
+ where
+ T: ?Sized,
+ U: ?Sized,
+ V: ?Sized,
+ TU: Project<T, U>,
+ UV: Project<U, V>,
+ {
+ #[inline(always)]
+ fn project(t: PtrInner<'_, T>) -> *mut V {
+ t.project::<_, TU>().project::<_, UV>().as_ptr()
+ }
+ }
+
+ // SAFETY: Since the `Project::project` impl delegates to `TU::project` and
+ // `UV::project`, and since `TU` and `UV` are `Cast`, the `Project::project`
+ // impl preserves the address of the referent.
+ unsafe impl<T, U, V, TU, UV> Cast<T, V> for TransitiveProject<U, TU, UV>
+ where
+ T: ?Sized,
+ U: ?Sized,
+ V: ?Sized,
+ TU: Cast<T, U>,
+ UV: Cast<U, V>,
+ {
+ }
+
+ // SAFETY: Since the `Project::project` impl delegates to `TU::project` and
+ // `UV::project`, and since `TU` and `UV` are `CastExact`, the `Project::project`
+ // impl preserves the set of referent bytes.
+ unsafe impl<T, U, V, TU, UV> CastExact<T, V> for TransitiveProject<U, TU, UV>
+ where
+ T: ?Sized,
+ U: ?Sized,
+ V: ?Sized,
+ TU: CastExact<T, U>,
+ UV: CastExact<U, V>,
+ {
+ }
+
+ /// A cast from `T` to `[u8]`.
+ #[allow(missing_copy_implementations, missing_debug_implementations)]
+ pub struct AsBytesCast;
+
+ // SAFETY: `project` constructs a pointer with the same address as `src`
+ // and with a referent of the same size as `*src`. It does this using
+ // provenance-preserving operations.
+ //
+ // FIXME(https://github.com/rust-lang/unsafe-code-guidelines/issues/594):
+ // Technically, this proof assumes that `*src` is contiguous (the same is
+ // true of other proofs in this codebase). Is this guaranteed anywhere?
+ unsafe impl<T: ?Sized + KnownLayout> Project<T, [u8]> for AsBytesCast {
+ #[inline(always)]
+ fn project(src: PtrInner<'_, T>) -> *mut [u8] {
+ let bytes = match T::size_of_val_raw(src.as_non_null()) {
+ Some(bytes) => bytes,
+ // SAFETY: `KnownLayout::size_of_val_raw` promises to always
+ // return `Some` so long as the resulting size fits in a
+ // `usize`. By invariant on `PtrInner`, `src` refers to a range
+ // of bytes whose size fits in an `isize`, which implies that it
+ // also fits in a `usize`.
+ None => unsafe { core::hint::unreachable_unchecked() },
+ };
+
+ core::ptr::slice_from_raw_parts_mut(src.as_ptr().cast::<u8>(), bytes)
+ }
+ }
+
+ // SAFETY: The `Project::project` impl preserves referent address.
+ unsafe impl<T: ?Sized + KnownLayout> Cast<T, [u8]> for AsBytesCast {}
+
+ // SAFETY: The `Project::project` impl preserves the set of referent bytes.
+ unsafe impl<T: ?Sized + KnownLayout> CastExact<T, [u8]> for AsBytesCast {}
+
+ /// A cast from any type to `()`.
+ #[allow(missing_copy_implementations, missing_debug_implementations)]
+ pub struct CastToUnit;
+
+ // SAFETY: The `project` implementation projects to a subset of its
+ // argument's referent using provenance-preserving operations.
+ unsafe impl<T: ?Sized> Project<T, ()> for CastToUnit {
+ #[inline(always)]
+ fn project(src: PtrInner<'_, T>) -> *mut () {
+ src.as_ptr().cast::<()>()
+ }
+ }
+
+ // SAFETY: The `project` implementation preserves referent address.
+ unsafe impl<T: ?Sized> Cast<T, ()> for CastToUnit {}
+}
diff --git a/rust/zerocopy/src/pointer/ptr.rs b/rust/zerocopy/src/pointer/ptr.rs
new file mode 100644
index 0000000..b7c4ea5
--- /dev/null
+++ b/rust/zerocopy/src/pointer/ptr.rs
@@ -0,0 +1,1586 @@
+// SPDX-License-Identifier: (BSD-2-Clause OR Apache-2.0) OR MIT
+
+// Copyright 2023 The Fuchsia Authors
+//
+// Licensed under a BSD-style license <LICENSE-BSD>, Apache License, Version 2.0
+// <LICENSE-APACHE or https://www.apache.org/licenses/LICENSE-2.0>, or the MIT
+// license <LICENSE-MIT or https://opensource.org/licenses/MIT>, at your option.
+// This file may not be copied, modified, or distributed except according to
+// those terms.
+
+#![allow(missing_docs)]
+
+use core::{
+ fmt::{Debug, Formatter},
+ marker::PhantomData,
+};
+
+use crate::{
+ pointer::{
+ inner::PtrInner,
+ invariant::*,
+ transmute::{MutationCompatible, SizeEq, TransmuteFromPtr},
+ },
+ AlignmentError, CastError, CastType, KnownLayout, SizeError, TryFromBytes, ValidityError,
+};
+
+/// Module used to gate access to [`Ptr`]'s fields.
+mod def {
+ #[cfg(doc)]
+ use super::super::invariant;
+ use super::*;
+
+ /// A raw pointer with more restrictions.
+ ///
+ /// `Ptr<T>` is similar to [`NonNull<T>`], but it is more restrictive in the
+ /// following ways (note that these requirements only hold of non-zero-sized
+ /// referents):
+ /// - It must derive from a valid allocation.
+ /// - It must reference a byte range which is contained inside the
+ /// allocation from which it derives.
+ /// - As a consequence, the byte range it references must have a size
+ /// which does not overflow `isize`.
+ ///
+ /// Depending on how `Ptr` is parameterized, it may have additional
+ /// invariants:
+ /// - `ptr` conforms to the aliasing invariant of
+ /// [`I::Aliasing`](invariant::Aliasing).
+ /// - `ptr` conforms to the alignment invariant of
+ /// [`I::Alignment`](invariant::Alignment).
+ /// - `ptr` conforms to the validity invariant of
+ /// [`I::Validity`](invariant::Validity).
+ ///
+ /// `Ptr<'a, T>` is [covariant] in `'a` and invariant in `T`.
+ ///
+ /// [`NonNull<T>`]: core::ptr::NonNull
+ /// [covariant]: https://doc.rust-lang.org/reference/subtyping.html
+ pub struct Ptr<'a, T, I>
+ where
+ T: ?Sized,
+ I: Invariants,
+ {
+ /// # Invariants
+ ///
+ /// 0. `ptr` conforms to the aliasing invariant of
+ /// [`I::Aliasing`](invariant::Aliasing).
+ /// 1. `ptr` conforms to the alignment invariant of
+ /// [`I::Alignment`](invariant::Alignment).
+ /// 2. `ptr` conforms to the validity invariant of
+ /// [`I::Validity`](invariant::Validity).
+ // SAFETY: `PtrInner<'a, T>` is covariant in `'a` and invariant in `T`.
+ ptr: PtrInner<'a, T>,
+ _invariants: PhantomData<I>,
+ }
+
+ impl<'a, T, I> Ptr<'a, T, I>
+ where
+ T: 'a + ?Sized,
+ I: Invariants,
+ {
+ /// Constructs a new `Ptr` from a [`PtrInner`].
+ ///
+ /// # Safety
+ ///
+ /// The caller promises that:
+ ///
+ /// 0. `ptr` conforms to the aliasing invariant of
+ /// [`I::Aliasing`](invariant::Aliasing).
+ /// 1. `ptr` conforms to the alignment invariant of
+ /// [`I::Alignment`](invariant::Alignment).
+ /// 2. `ptr` conforms to the validity invariant of
+ /// [`I::Validity`](invariant::Validity).
+ pub(crate) unsafe fn from_inner(ptr: PtrInner<'a, T>) -> Ptr<'a, T, I> {
+ // SAFETY: The caller has promised to satisfy all safety invariants
+ // of `Ptr`.
+ Self { ptr, _invariants: PhantomData }
+ }
+
+ /// Converts this `Ptr<T>` to a [`PtrInner<T>`].
+ ///
+ /// Note that this method does not consume `self`. The caller should
+ /// watch out for `unsafe` code which uses the returned value in a way
+ /// that violates the safety invariants of `self`.
+ #[inline]
+ #[must_use]
+ pub fn as_inner(&self) -> PtrInner<'a, T> {
+ self.ptr
+ }
+ }
+}
+
+#[allow(unreachable_pub)] // This is a false positive on our MSRV toolchain.
+pub use def::Ptr;
+
+/// External trait implementations on [`Ptr`].
+mod _external {
+ use super::*;
+
+ /// SAFETY: Shared pointers are safely `Copy`. `Ptr`'s other invariants
+ /// (besides aliasing) are unaffected by the number of references that exist
+ /// to `Ptr`'s referent. The notable cases are:
+ /// - Alignment is a property of the referent type (`T`) and the address,
+ /// both of which are unchanged
+ /// - Let `S(T, V)` be the set of bit values permitted to appear in the
+ /// referent of a `Ptr<T, I: Invariants<Validity = V>>`. Since this copy
+ /// does not change `I::Validity` or `T`, `S(T, I::Validity)` is also
+ /// unchanged.
+ ///
+ /// We are required to guarantee that the referents of the original `Ptr`
+ /// and of the copy (which, of course, are actually the same since they
+ /// live in the same byte address range) both remain in the set `S(T,
+ /// I::Validity)`. Since this invariant holds on the original `Ptr`, it
+ /// cannot be violated by the original `Ptr`, and thus the original `Ptr`
+ /// cannot be used to violate this invariant on the copy. The inverse
+ /// holds as well.
+ impl<'a, T, I> Copy for Ptr<'a, T, I>
+ where
+ T: 'a + ?Sized,
+ I: Invariants<Aliasing = Shared>,
+ {
+ }
+
+ /// SAFETY: See the safety comment on `Copy`.
+ impl<'a, T, I> Clone for Ptr<'a, T, I>
+ where
+ T: 'a + ?Sized,
+ I: Invariants<Aliasing = Shared>,
+ {
+ #[inline]
+ fn clone(&self) -> Self {
+ *self
+ }
+ }
+
+ impl<'a, T, I> Debug for Ptr<'a, T, I>
+ where
+ T: 'a + ?Sized,
+ I: Invariants,
+ {
+ #[inline]
+ fn fmt(&self, f: &mut Formatter<'_>) -> core::fmt::Result {
+ self.as_inner().as_non_null().fmt(f)
+ }
+ }
+}
+
+/// Methods for converting to and from `Ptr` and Rust's safe reference types.
+mod _conversions {
+ use super::*;
+ use crate::pointer::cast::{CastExact, CastSized, IdCast};
+
+ /// `&'a T` → `Ptr<'a, T>`
+ impl<'a, T> Ptr<'a, T, (Shared, Aligned, Valid)>
+ where
+ T: 'a + ?Sized,
+ {
+ /// Constructs a `Ptr` from a shared reference.
+ #[inline(always)]
+ pub fn from_ref(ptr: &'a T) -> Self {
+ let inner = PtrInner::from_ref(ptr);
+ // SAFETY:
+ // 0. `ptr`, by invariant on `&'a T`, conforms to the aliasing
+ // invariant of `Shared`.
+ // 1. `ptr`, by invariant on `&'a T`, conforms to the alignment
+ // invariant of `Aligned`.
+ // 2. `ptr`'s referent, by invariant on `&'a T`, is a bit-valid `T`.
+ // This satisfies the requirement that a `Ptr<T, (_, _, Valid)>`
+ // point to a bit-valid `T`. Even if `T` permits interior
+ // mutation, this invariant guarantees that the returned `Ptr`
+ // can only ever be used to modify the referent to store
+ // bit-valid `T`s, which ensures that the returned `Ptr` cannot
+ // be used to violate the soundness of the original `ptr: &'a T`
+ // or of any other references that may exist to the same
+ // referent.
+ unsafe { Self::from_inner(inner) }
+ }
+ }
+
+ /// `&'a mut T` → `Ptr<'a, T>`
+ impl<'a, T> Ptr<'a, T, (Exclusive, Aligned, Valid)>
+ where
+ T: 'a + ?Sized,
+ {
+ /// Constructs a `Ptr` from an exclusive reference.
+ #[inline(always)]
+ pub fn from_mut(ptr: &'a mut T) -> Self {
+ let inner = PtrInner::from_mut(ptr);
+ // SAFETY:
+ // 0. `ptr`, by invariant on `&'a mut T`, conforms to the aliasing
+ // invariant of `Exclusive`.
+ // 1. `ptr`, by invariant on `&'a mut T`, conforms to the alignment
+ // invariant of `Aligned`.
+ // 2. `ptr`'s referent, by invariant on `&'a mut T`, is a bit-valid
+ // `T`. This satisfies the requirement that a `Ptr<T, (_, _,
+ // Valid)>` point to a bit-valid `T`. This invariant guarantees
+ // that the returned `Ptr` can only ever be used to modify the
+ // referent to store bit-valid `T`s, which ensures that the
+ // returned `Ptr` cannot be used to violate the soundness of the
+ // original `ptr: &'a mut T`.
+ unsafe { Self::from_inner(inner) }
+ }
+ }
+
+ /// `Ptr<'a, T>` → `&'a T`
+ impl<'a, T, I> Ptr<'a, T, I>
+ where
+ T: 'a + ?Sized,
+ I: Invariants<Alignment = Aligned, Validity = Valid>,
+ I::Aliasing: Reference,
+ {
+ /// Converts `self` to a shared reference.
+ // This consumes `self`, not `&self`, because `self` is, logically, a
+ // pointer. For `I::Aliasing = invariant::Shared`, `Self: Copy`, and so
+ // this doesn't prevent the caller from still using the pointer after
+ // calling `as_ref`.
+ #[allow(clippy::wrong_self_convention)]
+ #[inline]
+ #[must_use]
+ pub fn as_ref(self) -> &'a T {
+ let raw = self.as_inner().as_non_null();
+ // SAFETY: `self` satisfies the `Aligned` invariant, so we know that
+ // `raw` is validly-aligned for `T`.
+ #[cfg(miri)]
+ unsafe {
+ crate::util::miri_promise_symbolic_alignment(
+ raw.as_ptr().cast(),
+ core::mem::align_of_val_raw(raw.as_ptr()),
+ );
+ }
+ // SAFETY: This invocation of `NonNull::as_ref` satisfies its
+ // documented safety preconditions:
+ //
+ // 1. The pointer is properly aligned. This is ensured by-contract
+ // on `Ptr`, because the `I::Alignment` is `Aligned`.
+ //
+ // 2. If the pointer's referent is not zero-sized, then the pointer
+ // must be “dereferenceable” in the sense defined in the module
+ // documentation; i.e.:
+ //
+ // > The memory range of the given size starting at the pointer
+ // > must all be within the bounds of a single allocated object.
+ // > [2]
+ //
+ // This is ensured by contract on all `PtrInner`s.
+ //
+ // 3. The pointer must point to a validly-initialized instance of
+ // `T`. This is ensured by-contract on `Ptr`, because the
+ // `I::Validity` is `Valid`.
+ //
+ // 4. You must enforce Rust’s aliasing rules. This is ensured by
+ // contract on `Ptr`, because `I::Aliasing: Reference`. Either it
+ // is `Shared` or `Exclusive`. If it is `Shared`, other
+ // references may not mutate the referent outside of
+ // `UnsafeCell`s.
+ //
+ // [1]: https://doc.rust-lang.org/std/ptr/struct.NonNull.html#method.as_ref
+ // [2]: https://doc.rust-lang.org/std/ptr/index.html#safety
+ unsafe { raw.as_ref() }
+ }
+ }
+
+ impl<'a, T, I> Ptr<'a, T, I>
+ where
+ T: 'a + ?Sized,
+ I: Invariants,
+ I::Aliasing: Reference,
+ {
+ /// Reborrows `self`, producing another `Ptr`.
+ ///
+ /// Since `self` is borrowed mutably, this prevents any methods from
+ /// being called on `self` as long as the returned `Ptr` exists.
+ #[inline]
+ #[must_use]
+ #[allow(clippy::needless_lifetimes)] // Allows us to name the lifetime in the safety comment below.
+ pub fn reborrow<'b>(&'b mut self) -> Ptr<'b, T, I>
+ where
+ 'a: 'b,
+ {
+ // SAFETY: The following all hold by invariant on `self`, and thus
+ // hold of `ptr = self.as_inner()`:
+ // 0. SEE BELOW.
+ // 1. `ptr` conforms to the alignment invariant of
+ // [`I::Alignment`](invariant::Alignment).
+ // 2. `ptr` conforms to the validity invariant of
+ // [`I::Validity`](invariant::Validity). `self` and the returned
+ // `Ptr` permit the same bit values in their referents since they
+ // have the same referent type (`T`) and the same validity
+ // (`I::Validity`). Thus, regardless of what mutation is
+ // permitted (`Exclusive` aliasing or `Shared`-aliased interior
+ // mutation), neither can be used to write a value to the
+ // referent which violates the other's validity invariant.
+ //
+ // For aliasing (0 above), since `I::Aliasing: Reference`,
+ // there are two cases for `I::Aliasing`:
+ // - For `invariant::Shared`: `'a` outlives `'b`, and so the
+ // returned `Ptr` does not permit accessing the referent any
+ // longer than is possible via `self`. For shared aliasing, it is
+ // sound for multiple `Ptr`s to exist simultaneously which
+ // reference the same memory, so creating a new one is not
+ // problematic.
+ // - For `invariant::Exclusive`: Since `self` is `&'b mut` and we
+ // return a `Ptr` with lifetime `'b`, `self` is inaccessible to
+ // the caller for the lifetime `'b` - in other words, `self` is
+ // inaccessible to the caller as long as the returned `Ptr`
+ // exists. Since `self` is an exclusive `Ptr`, no other live
+ // references or `Ptr`s may exist which refer to the same memory
+ // while `self` is live. Thus, as long as the returned `Ptr`
+ // exists, no other references or `Ptr`s which refer to the same
+ // memory may be live.
+ unsafe { Ptr::from_inner(self.as_inner()) }
+ }
+
+ /// Reborrows `self` as shared, producing another `Ptr` with `Shared`
+ /// aliasing.
+ ///
+ /// Since `self` is borrowed mutably, this prevents any methods from
+ /// being called on `self` as long as the returned `Ptr` exists.
+ #[inline]
+ #[must_use]
+ #[allow(clippy::needless_lifetimes)] // Allows us to name the lifetime in the safety comment below.
+ pub fn reborrow_shared<'b>(&'b mut self) -> Ptr<'b, T, (Shared, I::Alignment, I::Validity)>
+ where
+ 'a: 'b,
+ {
+ // SAFETY: The following all hold by invariant on `self`, and thus
+ // hold of `ptr = self.as_inner()`:
+ // 0. SEE BELOW.
+ // 1. `ptr` conforms to the alignment invariant of
+ // [`I::Alignment`](invariant::Alignment).
+ // 2. `ptr` conforms to the validity invariant of
+ // [`I::Validity`](invariant::Validity). `self` and the returned
+ // `Ptr` permit the same bit values in their referents since they
+ // have the same referent type (`T`) and the same validity
+ // (`I::Validity`). Thus, regardless of what mutation is
+ // permitted (`Exclusive` aliasing or `Shared`-aliased interior
+ // mutation), neither can be used to write a value to the
+ // referent which violates the other's validity invariant.
+ //
+ // For aliasing (0 above), since `I::Aliasing: Reference`,
+ // there are two cases for `I::Aliasing`:
+ // - For `invariant::Shared`: `'a` outlives `'b`, and so the
+ // returned `Ptr` does not permit accessing the referent any
+ // longer than is possible via `self`. For shared aliasing, it is
+ // sound for multiple `Ptr`s to exist simultaneously which
+ // reference the same memory, so creating a new one is not
+ // problematic.
+ // - For `invariant::Exclusive`: Since `self` is `&'b mut` and we
+ // return a `Ptr` with lifetime `'b`, `self` is inaccessible to
+ // the caller for the lifetime `'b` - in other words, `self` is
+ // inaccessible to the caller as long as the returned `Ptr`
+ // exists. Since `self` is an exclusive `Ptr`, no other live
+ // references or `Ptr`s may exist which refer to the same memory
+ // while `self` is live. Thus, as long as the returned `Ptr`
+ // exists, no other references or `Ptr`s which refer to the same
+ // memory may be live.
+ unsafe { Ptr::from_inner(self.as_inner()) }
+ }
+ }
+
+ /// `Ptr<'a, T>` → `&'a mut T`
+ impl<'a, T> Ptr<'a, T, (Exclusive, Aligned, Valid)>
+ where
+ T: 'a + ?Sized,
+ {
+ /// Converts `self` to a mutable reference.
+ #[allow(clippy::wrong_self_convention)]
+ #[inline]
+ #[must_use]
+ pub fn as_mut(self) -> &'a mut T {
+ let mut raw = self.as_inner().as_non_null();
+ // SAFETY: `self` satisfies the `Aligned` invariant, so we know that
+ // `raw` is validly-aligned for `T`.
+ #[cfg(miri)]
+ unsafe {
+ crate::util::miri_promise_symbolic_alignment(
+ raw.as_ptr().cast(),
+ core::mem::align_of_val_raw(raw.as_ptr()),
+ );
+ }
+ // SAFETY: This invocation of `NonNull::as_mut` satisfies its
+ // documented safety preconditions:
+ //
+ // 1. The pointer is properly aligned. This is ensured by-contract
+ // on `Ptr`, because the `ALIGNMENT_INVARIANT` is `Aligned`.
+ //
+ // 2. If the pointer's referent is not zero-sized, then the pointer
+ // must be “dereferenceable” in the sense defined in the module
+ // documentation; i.e.:
+ //
+ // > The memory range of the given size starting at the pointer
+ // > must all be within the bounds of a single allocated object.
+ // > [2]
+ //
+ // This is ensured by contract on all `PtrInner`s.
+ //
+ // 3. The pointer must point to a validly-initialized instance of
+ // `T`. This is ensured by-contract on `Ptr`, because the
+ // validity invariant is `Valid`.
+ //
+ // 4. You must enforce Rust’s aliasing rules. This is ensured by
+ // contract on `Ptr`, because the `ALIASING_INVARIANT` is
+ // `Exclusive`.
+ //
+ // [1]: https://doc.rust-lang.org/std/ptr/struct.NonNull.html#method.as_mut
+ // [2]: https://doc.rust-lang.org/std/ptr/index.html#safety
+ unsafe { raw.as_mut() }
+ }
+ }
+
+ /// `Ptr<'a, T>` → `Ptr<'a, U>`
+ impl<'a, T: ?Sized, I> Ptr<'a, T, I>
+ where
+ I: Invariants,
+ {
+ #[must_use]
+ #[inline(always)]
+ pub fn transmute<U, V, R>(self) -> Ptr<'a, U, (I::Aliasing, Unaligned, V)>
+ where
+ V: Validity,
+ U: TransmuteFromPtr<T, I::Aliasing, I::Validity, V, <U as SizeEq<T>>::CastFrom, R>
+ + SizeEq<T>
+ + ?Sized,
+ {
+ self.transmute_with::<U, V, <U as SizeEq<T>>::CastFrom, R>()
+ }
+
+ #[inline]
+ #[must_use]
+ pub fn transmute_with<U, V, C, R>(self) -> Ptr<'a, U, (I::Aliasing, Unaligned, V)>
+ where
+ V: Validity,
+ U: TransmuteFromPtr<T, I::Aliasing, I::Validity, V, C, R> + ?Sized,
+ C: CastExact<T, U>,
+ {
+ // SAFETY:
+ // - By `C: CastExact`, `C` preserves referent address, and so we
+ // don't need to consider projections in the following safety
+ // arguments.
+ // - If aliasing is `Shared`, then by `U: TransmuteFromPtr<T>`, at
+ // least one of the following holds:
+ // - `T: Immutable` and `U: Immutable`, in which case it is
+ // trivially sound for shared code to operate on a `&T` and `&U`
+ // at the same time, as neither can perform interior mutation
+ // - It is directly guaranteed that it is sound for shared code to
+ // operate on these references simultaneously
+ // - By `U: TransmuteFromPtr<T, I::Aliasing, I::Validity, C, V>`, it
+ // is sound to perform this transmute using `C`.
+ unsafe { self.project_transmute_unchecked::<_, _, C>() }
+ }
+
+ #[inline]
+ #[must_use]
+ pub fn recall_validity<V, R>(self) -> Ptr<'a, T, (I::Aliasing, I::Alignment, V)>
+ where
+ V: Validity,
+ T: TransmuteFromPtr<T, I::Aliasing, I::Validity, V, IdCast, R>,
+ {
+ let ptr = self.transmute_with::<T, V, IdCast, R>();
+ // SAFETY: `self` and `ptr` have the same address and referent type.
+ // Therefore, if `self` satisfies `I::Alignment`, then so does
+ // `ptr`.
+ unsafe { ptr.assume_alignment::<I::Alignment>() }
+ }
+
+ /// Projects and/or transmutes to a different (unsized) referent type
+ /// without checking interior mutability.
+ ///
+ /// Callers should prefer [`cast`] or [`project`] where possible.
+ ///
+ /// [`cast`]: Ptr::cast
+ /// [`project`]: Ptr::project
+ ///
+ /// # Safety
+ ///
+ /// The caller promises that:
+ /// - If `I::Aliasing` is [`Shared`], it must not be possible for safe
+ /// code, operating on a `&T` and `&U`, with the referents of `self`
+ /// and `self.project_transmute_unchecked()`, respectively, to cause
+ /// undefined behavior.
+ /// - It is sound to project and/or transmute a pointer of type `T` with
+ /// aliasing `I::Aliasing` and validity `I::Validity` to a pointer of
+ /// type `U` with aliasing `I::Aliasing` and validity `V`. This is a
+ /// subtle soundness requirement that is a function of `T`, `U`,
+ /// `I::Aliasing`, `I::Validity`, and `V`, and may depend upon the
+ /// presence, absence, or specific location of `UnsafeCell`s in `T`
+ /// and/or `U`, and on whether interior mutation is ever permitted via
+ /// those `UnsafeCell`s. See [`Validity`] for more details.
+ #[inline]
+ #[must_use]
+ pub unsafe fn project_transmute_unchecked<U: ?Sized, V, P>(
+ self,
+ ) -> Ptr<'a, U, (I::Aliasing, Unaligned, V)>
+ where
+ V: Validity,
+ P: crate::pointer::cast::Project<T, U>,
+ {
+ let ptr = self.as_inner().project::<_, P>();
+
+ // SAFETY:
+ //
+ // The following safety arguments rely on the fact that `P: Project`
+ // guarantees that `P` is a referent-preserving or -shrinking
+ // projection. Thus, `ptr` addresses a subset of the bytes of
+ // `*self`, and so certain properties that hold of `*self` also hold
+ // of `*ptr`.
+ //
+ // 0. `ptr` conforms to the aliasing invariant of `I::Aliasing`:
+ // - `Exclusive`: `self` is the only `Ptr` or reference which is
+ // permitted to read or modify the referent for the lifetime
+ // `'a`. Since we consume `self` by value, the returned pointer
+ // remains the only `Ptr` or reference which is permitted to
+ // read or modify the referent for the lifetime `'a`.
+ // - `Shared`: Since `self` has aliasing `Shared`, we know that
+ // no other code may mutate the referent during the lifetime
+ // `'a`, except via `UnsafeCell`s, and except as permitted by
+ // `T`'s library safety invariants. The caller promises that
+ // any safe operations which can be permitted on a `&T` and a
+ // `&U` simultaneously must be sound. Thus, no operations on a
+ // `&U` could violate `&T`'s library safety invariants, and
+ // vice-versa. Since any mutation via shared references outside
+ // of `UnsafeCell`s is unsound, this must be impossible using
+ // `&T` and `&U`.
+ // - `Inaccessible`: There are no restrictions we need to uphold.
+ // 1. `ptr` trivially satisfies the alignment invariant `Unaligned`.
+ // 2. The caller promises that the returned pointer satisfies the
+ // validity invariant `V` with respect to its referent type, `U`.
+ unsafe { Ptr::from_inner(ptr) }
+ }
+ }
+
+ /// `Ptr<'a, T, (_, _, _)>` → `Ptr<'a, Unalign<T>, (_, Aligned, _)>`
+ impl<'a, T, I> Ptr<'a, T, I>
+ where
+ I: Invariants,
+ {
+ /// Converts a `Ptr` an unaligned `T` into a `Ptr` to an aligned
+ /// `Unalign<T>`.
+ #[inline]
+ #[must_use]
+ pub fn into_unalign(
+ self,
+ ) -> Ptr<'a, crate::Unalign<T>, (I::Aliasing, Aligned, I::Validity)> {
+ // FIXME(#1359): This should be a `transmute_with` call.
+ // Unfortunately, to avoid blanket impl conflicts, we only implement
+ // `TransmuteFrom<T>` for `Unalign<T>` (and vice versa) specifically
+ // for `Valid` validity, not for all validity types.
+
+ // SAFETY:
+ // - By `CastSized: Cast`, `CastSized` preserves referent address,
+ // and so we don't need to consider projections in the following
+ // safety arguments.
+ // - Since `Unalign<T>` has the same layout as `T`, the returned
+ // pointer refers to `UnsafeCell`s at the same locations as
+ // `self`.
+ // - `Unalign<T>` promises to have the same bit validity as `T`. By
+ // invariant on `Validity`, the set of bit patterns allowed in the
+ // referent of a `Ptr<X, (_, _, V)>` is only a function of the
+ // validity of `X` and of `V`. Thus, the set of bit patterns
+ // allowed in the referent of a `Ptr<T, (_, _, I::Validity)>` is
+ // the same as the set of bit patterns allowed in the referent of
+ // a `Ptr<Unalign<T>, (_, _, I::Validity)>`. As a result, `self`
+ // and the returned `Ptr` permit the same set of bit patterns in
+ // their referents, and so neither can be used to violate the
+ // validity of the other.
+ let ptr = unsafe { self.project_transmute_unchecked::<_, _, CastSized>() };
+ ptr.bikeshed_recall_aligned()
+ }
+ }
+
+ impl<'a, T, I> Ptr<'a, T, I>
+ where
+ T: ?Sized,
+ I: Invariants<Validity = Valid>,
+ I::Aliasing: Reference,
+ {
+ /// Reads the referent.
+ #[must_use]
+ #[inline(always)]
+ pub fn read<R>(self) -> T
+ where
+ T: Copy,
+ T: Read<I::Aliasing, R>,
+ {
+ <I::Alignment as Alignment>::read(self)
+ }
+
+ /// Views the value as an aligned reference.
+ ///
+ /// This is only available if `T` is [`Unaligned`].
+ #[must_use]
+ #[inline]
+ pub fn unaligned_as_ref(self) -> &'a T
+ where
+ T: crate::Unaligned,
+ {
+ self.bikeshed_recall_aligned().as_ref()
+ }
+ }
+}
+
+/// State transitions between invariants.
+mod _transitions {
+ use super::*;
+ use crate::{
+ pointer::{cast::IdCast, transmute::TryTransmuteFromPtr},
+ ReadOnly,
+ };
+
+ impl<'a, T, I> Ptr<'a, T, I>
+ where
+ T: 'a + ?Sized,
+ I: Invariants,
+ {
+ /// Assumes that `self` satisfies the invariants `H`.
+ ///
+ /// # Safety
+ ///
+ /// The caller promises that `self` satisfies the invariants `H`.
+ unsafe fn assume_invariants<H: Invariants>(self) -> Ptr<'a, T, H> {
+ // SAFETY: The caller has promised to satisfy all parameterized
+ // invariants of `Ptr`. `Ptr`'s other invariants are satisfied
+ // by-contract by the source `Ptr`.
+ unsafe { Ptr::from_inner(self.as_inner()) }
+ }
+
+ /// Helps the type system unify two distinct invariant types which are
+ /// actually the same.
+ #[inline]
+ #[must_use]
+ pub fn unify_invariants<
+ H: Invariants<Aliasing = I::Aliasing, Alignment = I::Alignment, Validity = I::Validity>,
+ >(
+ self,
+ ) -> Ptr<'a, T, H> {
+ // SAFETY: The associated type bounds on `H` ensure that the
+ // invariants are unchanged.
+ unsafe { self.assume_invariants::<H>() }
+ }
+
+ /// Assumes that `self`'s referent is validly-aligned for `T` if
+ /// required by `A`.
+ ///
+ /// # Safety
+ ///
+ /// The caller promises that `self`'s referent conforms to the alignment
+ /// invariant of `T` if required by `A`.
+ #[inline]
+ pub(crate) unsafe fn assume_alignment<A: Alignment>(
+ self,
+ ) -> Ptr<'a, T, (I::Aliasing, A, I::Validity)> {
+ // SAFETY: The caller promises that `self`'s referent is
+ // well-aligned for `T` if required by `A` .
+ unsafe { self.assume_invariants() }
+ }
+
+ /// Checks the `self`'s alignment at runtime, returning an aligned `Ptr`
+ /// on success.
+ #[inline]
+ pub fn try_into_aligned(
+ self,
+ ) -> Result<Ptr<'a, T, (I::Aliasing, Aligned, I::Validity)>, AlignmentError<Self, T>>
+ where
+ T: Sized,
+ {
+ if let Err(err) =
+ crate::util::validate_aligned_to::<_, T>(self.as_inner().as_non_null())
+ {
+ return Err(err.with_src(self));
+ }
+
+ // SAFETY: We just checked the alignment.
+ Ok(unsafe { self.assume_alignment::<Aligned>() })
+ }
+
+ /// Recalls that `self`'s referent is validly-aligned for `T`.
+ #[inline]
+ // FIXME(#859): Reconsider the name of this method before making it
+ // public.
+ #[must_use]
+ pub fn bikeshed_recall_aligned(self) -> Ptr<'a, T, (I::Aliasing, Aligned, I::Validity)>
+ where
+ T: crate::Unaligned,
+ {
+ // SAFETY: The bound `T: Unaligned` ensures that `T` has no
+ // non-trivial alignment requirement.
+ unsafe { self.assume_alignment::<Aligned>() }
+ }
+
+ /// Assumes that `self`'s referent conforms to the validity requirement
+ /// of `V`.
+ ///
+ /// # Safety
+ ///
+ /// The caller promises that `self`'s referent conforms to the validity
+ /// requirement of `V`.
+ #[must_use]
+ #[inline]
+ pub unsafe fn assume_validity<V: Validity>(
+ self,
+ ) -> Ptr<'a, T, (I::Aliasing, I::Alignment, V)> {
+ // SAFETY: The caller promises that `self`'s referent conforms to
+ // the validity requirement of `V`.
+ unsafe { self.assume_invariants() }
+ }
+
+ /// A shorthand for `self.assume_validity<invariant::Initialized>()`.
+ ///
+ /// # Safety
+ ///
+ /// The caller promises to uphold the safety preconditions of
+ /// `self.assume_validity<invariant::Initialized>()`.
+ #[must_use]
+ #[inline]
+ pub unsafe fn assume_initialized(
+ self,
+ ) -> Ptr<'a, T, (I::Aliasing, I::Alignment, Initialized)> {
+ // SAFETY: The caller has promised to uphold the safety
+ // preconditions.
+ unsafe { self.assume_validity::<Initialized>() }
+ }
+
+ /// A shorthand for `self.assume_validity<Valid>()`.
+ ///
+ /// # Safety
+ ///
+ /// The caller promises to uphold the safety preconditions of
+ /// `self.assume_validity<Valid>()`.
+ #[must_use]
+ #[inline]
+ pub unsafe fn assume_valid(self) -> Ptr<'a, T, (I::Aliasing, I::Alignment, Valid)> {
+ // SAFETY: The caller has promised to uphold the safety
+ // preconditions.
+ unsafe { self.assume_validity::<Valid>() }
+ }
+
+ /// Checks that `self`'s referent is validly initialized for `T`,
+ /// returning a `Ptr` with `Valid` on success.
+ ///
+ /// # Panics
+ ///
+ /// This method will panic if
+ /// [`T::is_bit_valid`][TryFromBytes::is_bit_valid] panics.
+ ///
+ /// # Safety
+ ///
+ /// On error, unsafe code may rely on this method's returned
+ /// `ValidityError` containing `self`.
+ #[inline]
+ pub fn try_into_valid<R, S>(
+ mut self,
+ ) -> Result<Ptr<'a, T, (I::Aliasing, I::Alignment, Valid)>, ValidityError<Self, T>>
+ where
+ T: TryFromBytes
+ + Read<I::Aliasing, R>
+ + TryTransmuteFromPtr<T, I::Aliasing, I::Validity, Valid, IdCast, S>,
+ ReadOnly<T>: Read<I::Aliasing, R>,
+ I::Aliasing: Reference,
+ I: Invariants<Validity = Initialized>,
+ {
+ // This call may panic. If that happens, it doesn't cause any
+ // soundness issues, as we have not generated any invalid state
+ // which we need to fix before returning.
+ if T::is_bit_valid(self.reborrow().transmute::<_, _, _>().reborrow_shared()) {
+ // SAFETY: If `T::is_bit_valid`, code may assume that `self`
+ // contains a bit-valid instance of `T`. By `T:
+ // TryTransmuteFromPtr<T, I::Aliasing, I::Validity, Valid>`, so
+ // long as `self`'s referent conforms to the `Valid` validity
+ // for `T` (which we just confirmed), then this transmute is
+ // sound.
+ Ok(unsafe { self.assume_valid() })
+ } else {
+ Err(ValidityError::new(self))
+ }
+ }
+
+ /// Forgets that `self`'s referent is validly-aligned for `T`.
+ #[inline]
+ #[must_use]
+ pub fn forget_aligned(self) -> Ptr<'a, T, (I::Aliasing, Unaligned, I::Validity)> {
+ // SAFETY: `Unaligned` is less restrictive than `Aligned`.
+ unsafe { self.assume_invariants() }
+ }
+ }
+}
+
+/// Casts of the referent type.
+#[cfg_attr(not(zerocopy_unstable_ptr), allow(unreachable_pub))]
+pub use _casts::TryWithError;
+mod _casts {
+ use core::cell::UnsafeCell;
+
+ use super::*;
+ use crate::{
+ pointer::cast::{AsBytesCast, Cast},
+ HasTag, ProjectField,
+ };
+
+ impl<'a, T, I> Ptr<'a, T, I>
+ where
+ T: 'a + ?Sized,
+ I: Invariants,
+ {
+ /// Casts to a different referent type without checking interior
+ /// mutability.
+ ///
+ /// Callers should prefer [`cast`][Ptr::cast] where possible.
+ ///
+ /// # Safety
+ ///
+ /// If `I::Aliasing` is [`Shared`], it must not be possible for safe
+ /// code, operating on a `&T` and `&U` with the same referent
+ /// simultaneously, to cause undefined behavior.
+ #[inline]
+ #[must_use]
+ pub unsafe fn cast_unchecked<U, C: Cast<T, U>>(
+ self,
+ ) -> Ptr<'a, U, (I::Aliasing, Unaligned, I::Validity)>
+ where
+ U: 'a + CastableFrom<T, I::Validity, I::Validity> + ?Sized,
+ {
+ // SAFETY:
+ // - By `C: Cast`, `C` preserves the address of the referent.
+ // - If `I::Aliasing` is [`Shared`], the caller promises that it
+ // is not possible for safe code, operating on a `&T` and `&U`
+ // with the same referent simultaneously, to cause undefined
+ // behavior.
+ // - By `U: CastableFrom<T, I::Validity, I::Validity>`,
+ // `I::Validity` is either `Uninit` or `Initialized`. In both
+ // cases, the bit validity `I::Validity` has the same semantics
+ // regardless of referent type. In other words, the set of allowed
+ // referent values for `Ptr<T, (_, _, I::Validity)>` and `Ptr<U,
+ // (_, _, I::Validity)>` are identical. As a consequence, neither
+ // `self` nor the returned `Ptr` can be used to write values which
+ // are invalid for the other.
+ unsafe { self.project_transmute_unchecked::<_, _, C>() }
+ }
+
+ /// Casts to a different referent type.
+ #[inline]
+ #[must_use]
+ pub fn cast<U, C, R>(self) -> Ptr<'a, U, (I::Aliasing, Unaligned, I::Validity)>
+ where
+ T: MutationCompatible<U, I::Aliasing, I::Validity, I::Validity, R>,
+ U: 'a + ?Sized + CastableFrom<T, I::Validity, I::Validity>,
+ C: Cast<T, U>,
+ {
+ // SAFETY: Because `T: MutationCompatible<U, I::Aliasing, R>`, one
+ // of the following holds:
+ // - `T: Read<I::Aliasing>` and `U: Read<I::Aliasing>`, in which
+ // case one of the following holds:
+ // - `I::Aliasing` is `Exclusive`
+ // - `T` and `U` are both `Immutable`
+ // - It is sound for safe code to operate on `&T` and `&U` with the
+ // same referent simultaneously.
+ unsafe { self.cast_unchecked::<_, C>() }
+ }
+
+ #[inline(always)]
+ pub fn project<F, const VARIANT_ID: i128, const FIELD_ID: i128>(
+ mut self,
+ ) -> Result<Ptr<'a, T::Type, T::Invariants>, T::Error>
+ where
+ T: ProjectField<F, I, VARIANT_ID, FIELD_ID>,
+ I::Aliasing: Reference,
+ {
+ use crate::pointer::cast::Projection;
+ match T::is_projectable(self.reborrow().project_tag()) {
+ Ok(()) => {
+ let inner = self.as_inner();
+ let projected = inner.project::<_, Projection<F, VARIANT_ID, FIELD_ID>>();
+ // SAFETY: By `T: ProjectField<F, I, VARIANT_ID, FIELD_ID>`,
+ // for `self: Ptr<'_, T, I>` such that `T::is_projectable`
+ // (which we've verified in this match arm),
+ // `T::project(self.as_inner())` conforms to
+ // `T::Invariants`. The `projected` pointer satisfies these
+ // invariants because it is produced by way of an
+ // abstraction that is equivalent to
+ // `T::project(ptr.as_inner())`: by invariant on
+ // `PtrInner::project`, `projected` is guaranteed to address
+ // the subset of the bytes of `inner`'s referent addressed
+ // by `Projection::project(inner)`, and by invariant on
+ // `Projection`, `Projection::project` is implemented by
+ // delegating to an implementation of `HasField::project`.
+ Ok(unsafe { Ptr::from_inner(projected) })
+ }
+ Err(err) => Err(err),
+ }
+ }
+
+ #[must_use]
+ #[inline(always)]
+ pub(crate) fn project_tag(self) -> Ptr<'a, T::Tag, I>
+ where
+ T: HasTag,
+ {
+ // SAFETY: By invariant on `Self::ProjectToTag`, this is a sound
+ // projection.
+ let tag = unsafe { self.project_transmute_unchecked::<_, _, T::ProjectToTag>() };
+ // SAFETY: By invariant on `Self::ProjectToTag`, the projected
+ // pointer has the same alignment as `ptr`.
+ let tag = unsafe { tag.assume_alignment() };
+ tag.unify_invariants()
+ }
+
+ /// Attempts to transform the pointer, restoring the original on
+ /// failure.
+ ///
+ /// # Safety
+ ///
+ /// If `I::Aliasing != Shared`, then if `f` returns `Err(err)`, no copy
+ /// of `f`'s argument must exist outside of `err`.
+ #[inline(always)]
+ pub(crate) unsafe fn try_with_unchecked<U, J, E, F>(
+ self,
+ f: F,
+ ) -> Result<Ptr<'a, U, J>, E::Mapped>
+ where
+ U: 'a + ?Sized,
+ J: Invariants<Aliasing = I::Aliasing>,
+ E: TryWithError<Self>,
+ F: FnOnce(Ptr<'a, T, I>) -> Result<Ptr<'a, U, J>, E>,
+ {
+ let old_inner = self.as_inner();
+ #[rustfmt::skip]
+ let res = f(self).map_err(#[inline(always)] move |err: E| {
+ err.map(#[inline(always)] |src| {
+ drop(src);
+
+ // SAFETY:
+ // 0. Aliasing is either `Shared` or `Exclusive`:
+ // - If aliasing is `Shared`, then it cannot violate
+ // aliasing make another copy of this pointer (in fact,
+ // using `I::Aliasing = Shared`, we could have just
+ // cloned `self`).
+ // - If aliasing is `Exclusive`, then `f` is not allowed
+ // to make another copy of `self`. In `map_err`, we are
+ // consuming the only value in the returned `Result`.
+ // By invariant on `E: TryWithError<Self>`, that `err:
+ // E` only contains a single `Self` and no other
+ // non-ZST fields which could be `Ptr`s or references
+ // to `self`'s referent. By the same invariant, `map`
+ // consumes this single `Self` and passes it to this
+ // closure. Since `self` was, by invariant on
+ // `Exclusive`, the only `Ptr` or reference live for
+ // `'a` with this referent, and since we `drop(src)`
+ // above, there are no copies left, and so we are
+ // creating the only copy.
+ // 1. `self` conforms to `I::Aliasing` by invariant on
+ // `Ptr`, and `old_inner` has the same address, so it
+ // does too.
+ // 2. `f` could not have violated `self`'s validity without
+ // itself being unsound. Assuming that `f` is sound, the
+ // referent of `self` is still valid for `T`.
+ unsafe { Ptr::from_inner(old_inner) }
+ })
+ });
+ res
+ }
+
+ /// Attempts to transform the pointer, restoring the original on
+ /// failure.
+ #[inline(always)]
+ pub fn try_with<U, J, E, F>(self, f: F) -> Result<Ptr<'a, U, J>, E::Mapped>
+ where
+ U: 'a + ?Sized,
+ J: Invariants<Aliasing = I::Aliasing>,
+ E: TryWithError<Self>,
+ F: FnOnce(Ptr<'a, T, I>) -> Result<Ptr<'a, U, J>, E>,
+ I: Invariants<Aliasing = Shared>,
+ {
+ // SAFETY: `I::Aliasing = Shared`, so the safety condition does not
+ // apply.
+ unsafe { self.try_with_unchecked(f) }
+ }
+ }
+
+ /// # Safety
+ ///
+ /// `Self` only contains a single `Self::Inner`, and `Self::Mapped` only
+ /// contains a single `MappedInner`. Other than that, `Self` and
+ /// `Self::Mapped` contain no non-ZST fields.
+ ///
+ /// `map` must pass ownership of `self`'s sole `Self::Inner` to `f`.
+ pub unsafe trait TryWithError<MappedInner> {
+ type Inner;
+ type Mapped;
+ fn map<F: FnOnce(Self::Inner) -> MappedInner>(self, f: F) -> Self::Mapped;
+ }
+
+ impl<'a, T, I> Ptr<'a, T, I>
+ where
+ T: 'a + KnownLayout + ?Sized,
+ I: Invariants,
+ {
+ /// Casts this pointer-to-initialized into a pointer-to-bytes.
+ #[allow(clippy::wrong_self_convention)]
+ #[must_use]
+ #[inline]
+ pub fn as_bytes<R>(self) -> Ptr<'a, [u8], (I::Aliasing, Aligned, Valid)>
+ where
+ [u8]: TransmuteFromPtr<T, I::Aliasing, I::Validity, Valid, AsBytesCast, R>,
+ {
+ self.transmute_with::<[u8], Valid, AsBytesCast, _>().bikeshed_recall_aligned()
+ }
+ }
+
+ impl<'a, T, I, const N: usize> Ptr<'a, [T; N], I>
+ where
+ T: 'a,
+ I: Invariants,
+ {
+ /// Casts this pointer-to-array into a slice.
+ #[allow(clippy::wrong_self_convention)]
+ #[inline]
+ #[must_use]
+ pub fn as_slice(self) -> Ptr<'a, [T], I> {
+ let slice = self.as_inner().as_slice();
+ // SAFETY: Note that, by post-condition on `PtrInner::as_slice`,
+ // `slice` refers to the same byte range as `self.as_inner()`.
+ //
+ // 0. Thus, `slice` conforms to the aliasing invariant of
+ // `I::Aliasing` because `self` does.
+ // 1. By the above lemma, `slice` conforms to the alignment
+ // invariant of `I::Alignment` because `self` does.
+ // 2. Since `[T; N]` and `[T]` have the same bit validity [1][2],
+ // and since `self` and the returned `Ptr` have the same validity
+ // invariant, neither `self` nor the returned `Ptr` can be used
+ // to write a value to the referent which violates the other's
+ // validity invariant.
+ //
+ // [1] Per https://doc.rust-lang.org/1.81.0/reference/type-layout.html#array-layout:
+ //
+ // An array of `[T; N]` has a size of `size_of::<T>() * N` and the
+ // same alignment of `T`. Arrays are laid out so that the
+ // zero-based `nth` element of the array is offset from the start
+ // of the array by `n * size_of::<T>()` bytes.
+ //
+ // ...
+ //
+ // Slices have the same layout as the section of the array they
+ // slice.
+ //
+ // [2] Per https://doc.rust-lang.org/1.81.0/reference/types/array.html#array-types:
+ //
+ // All elements of arrays are always initialized
+ unsafe { Ptr::from_inner(slice) }
+ }
+ }
+
+ /// For caller convenience, these methods are generic over alignment
+ /// invariant. In practice, the referent is always well-aligned, because the
+ /// alignment of `[u8]` is 1.
+ impl<'a, I> Ptr<'a, [u8], I>
+ where
+ I: Invariants<Validity = Valid>,
+ {
+ /// Attempts to cast `self` to a `U` using the given cast type.
+ ///
+ /// If `U` is a slice DST and pointer metadata (`meta`) is provided,
+ /// then the cast will only succeed if it would produce an object with
+ /// the given metadata.
+ ///
+ /// Returns `None` if the resulting `U` would be invalidly-aligned, if
+ /// no `U` can fit in `self`, or if the provided pointer metadata
+ /// describes an invalid instance of `U`. On success, returns a pointer
+ /// to the largest-possible `U` which fits in `self`.
+ ///
+ /// # Safety
+ ///
+ /// The caller may assume that this implementation is correct, and may
+ /// rely on that assumption for the soundness of their code. In
+ /// particular, the caller may assume that, if `try_cast_into` returns
+ /// `Some((ptr, remainder))`, then `ptr` and `remainder` refer to
+ /// non-overlapping byte ranges within `self`, and that `ptr` and
+ /// `remainder` entirely cover `self`. Finally:
+ /// - If this is a prefix cast, `ptr` has the same address as `self`.
+ /// - If this is a suffix cast, `remainder` has the same address as
+ /// `self`.
+ #[inline(always)]
+ pub fn try_cast_into<U, R>(
+ self,
+ cast_type: CastType,
+ meta: Option<U::PointerMetadata>,
+ ) -> Result<
+ (Ptr<'a, U, (I::Aliasing, Aligned, Initialized)>, Ptr<'a, [u8], I>),
+ CastError<Self, U>,
+ >
+ where
+ I::Aliasing: Reference,
+ U: 'a + ?Sized + KnownLayout + Read<I::Aliasing, R>,
+ {
+ let (inner, remainder) = self.as_inner().try_cast_into(cast_type, meta).map_err(
+ #[inline(always)]
+ |err| {
+ err.map_src(
+ #[inline(always)]
+ |inner|
+ // SAFETY: `PtrInner::try_cast_into` promises to return its
+ // original argument on error, which was originally produced
+ // by `self.as_inner()`, which is guaranteed to satisfy
+ // `Ptr`'s invariants.
+ unsafe { Ptr::from_inner(inner) },
+ )
+ },
+ )?;
+
+ // SAFETY:
+ // 0. Since `U: Read<I::Aliasing, _>`, either:
+ // - `I::Aliasing` is `Exclusive`, in which case both `src` and
+ // `ptr` conform to `Exclusive`
+ // - `I::Aliasing` is `Shared` and `U` is `Immutable` (we already
+ // know that `[u8]: Immutable`). In this case, neither `U` nor
+ // `[u8]` permit mutation, and so `Shared` aliasing is
+ // satisfied.
+ // 1. `ptr` conforms to the alignment invariant of `Aligned` because
+ // it is derived from `try_cast_into`, which promises that the
+ // object described by `target` is validly aligned for `U`.
+ // 2. By trait bound, `self` - and thus `target` - is a bit-valid
+ // `[u8]`. `Ptr<[u8], (_, _, Valid)>` and `Ptr<_, (_, _,
+ // Initialized)>` have the same bit validity, and so neither
+ // `self` nor `res` can be used to write a value to the referent
+ // which violates the other's validity invariant.
+ let res = unsafe { Ptr::from_inner(inner) };
+
+ // SAFETY:
+ // 0. `self` and `remainder` both have the type `[u8]`. Thus, they
+ // have `UnsafeCell`s at the same locations. Type casting does
+ // not affect aliasing.
+ // 1. `[u8]` has no alignment requirement.
+ // 2. `self` has validity `Valid` and has type `[u8]`. Since
+ // `remainder` references a subset of `self`'s referent, it is
+ // also a bit-valid `[u8]`. Thus, neither `self` nor `remainder`
+ // can be used to write a value to the referent which violates
+ // the other's validity invariant.
+ let remainder = unsafe { Ptr::from_inner(remainder) };
+
+ Ok((res, remainder))
+ }
+
+ /// Attempts to cast `self` into a `U`, failing if all of the bytes of
+ /// `self` cannot be treated as a `U`.
+ ///
+ /// In particular, this method fails if `self` is not validly-aligned
+ /// for `U` or if `self`'s size is not a valid size for `U`.
+ ///
+ /// # Safety
+ ///
+ /// On success, the caller may assume that the returned pointer
+ /// references the same byte range as `self`.
+ #[allow(unused)]
+ #[inline(always)]
+ pub fn try_cast_into_no_leftover<U, R>(
+ self,
+ meta: Option<U::PointerMetadata>,
+ ) -> Result<Ptr<'a, U, (I::Aliasing, Aligned, Initialized)>, CastError<Self, U>>
+ where
+ I::Aliasing: Reference,
+ U: 'a + ?Sized + KnownLayout + Read<I::Aliasing, R>,
+ [u8]: Read<I::Aliasing, R>,
+ {
+ // SAFETY: The provided closure returns the only copy of `slf`.
+ unsafe {
+ self.try_with_unchecked(
+ #[inline(always)]
+ |slf| match slf.try_cast_into(CastType::Prefix, meta) {
+ Ok((slf, remainder)) => {
+ if remainder.is_empty() {
+ Ok(slf)
+ } else {
+ Err(CastError::Size(SizeError::<_, U>::new(())))
+ }
+ }
+ Err(err) => Err(err.map_src(
+ #[inline(always)]
+ |_slf| (),
+ )),
+ },
+ )
+ }
+ }
+ }
+
+ impl<'a, T, I> Ptr<'a, UnsafeCell<T>, I>
+ where
+ T: 'a + ?Sized,
+ I: Invariants<Aliasing = Exclusive>,
+ {
+ /// Converts this `Ptr` into a pointer to the underlying data.
+ ///
+ /// This call borrows the `UnsafeCell` mutably (at compile-time) which
+ /// guarantees that we possess the only reference.
+ ///
+ /// This is like [`UnsafeCell::get_mut`], but for `Ptr`.
+ ///
+ /// [`UnsafeCell::get_mut`]: core::cell::UnsafeCell::get_mut
+ #[must_use]
+ #[inline(always)]
+ pub fn get_mut(self) -> Ptr<'a, T, I> {
+ // SAFETY: As described below, `UnsafeCell<T>` has the same size
+ // as `T: ?Sized` (same static size or same DST layout). Thus,
+ // `*const UnsafeCell<T> as *const T` is a size-preserving cast.
+ define_cast!(unsafe { Cast<T: ?Sized> = UnsafeCell<T> => T });
+
+ // SAFETY:
+ // - Aliasing is `Exclusive`, and so we are not required to promise
+ // anything about the locations of `UnsafeCell`s.
+ // - `UnsafeCell<T>` has the same bit validity as `T` [1].
+ // Technically the term "representation" doesn't guarantee this,
+ // but the subsequent sentence in the documentation makes it clear
+ // that this is the intention.
+ //
+ // By invariant on `Validity`, since `T` and `UnsafeCell<T>` have
+ // the same bit validity, then the set of values which may appear
+ // in the referent of a `Ptr<T, (_, _, V)>` is the same as the set
+ // which may appear in the referent of a `Ptr<UnsafeCell<T>, (_,
+ // _, V)>`. Thus, neither `self` nor `ptr` may be used to write a
+ // value to the referent which would violate the other's validity
+ // invariant.
+ //
+ // [1] Per https://doc.rust-lang.org/1.81.0/core/cell/struct.UnsafeCell.html#memory-layout:
+ //
+ // `UnsafeCell<T>` has the same in-memory representation as its
+ // inner type `T`. A consequence of this guarantee is that it is
+ // possible to convert between `T` and `UnsafeCell<T>`.
+ let ptr = unsafe { self.project_transmute_unchecked::<_, _, Cast>() };
+
+ // SAFETY: `UnsafeCell<T>` has the same alignment as `T` [1],
+ // and so if `self` is guaranteed to be aligned, then so is the
+ // returned `Ptr`.
+ //
+ // [1] Per https://doc.rust-lang.org/1.81.0/core/cell/struct.UnsafeCell.html#memory-layout:
+ //
+ // `UnsafeCell<T>` has the same in-memory representation as
+ // its inner type `T`. A consequence of this guarantee is that
+ // it is possible to convert between `T` and `UnsafeCell<T>`.
+ let ptr = unsafe { ptr.assume_alignment::<I::Alignment>() };
+ ptr.unify_invariants()
+ }
+ }
+}
+
+/// Projections through the referent.
+mod _project {
+ use super::*;
+
+ impl<'a, T, I> Ptr<'a, [T], I>
+ where
+ T: 'a,
+ I: Invariants,
+ I::Aliasing: Reference,
+ {
+ /// Iteratively projects the elements `Ptr<T>` from `Ptr<[T]>`.
+ #[inline]
+ pub fn iter(self) -> impl Iterator<Item = Ptr<'a, T, I>> {
+ // SAFETY:
+ // 0. `elem` conforms to the aliasing invariant of `I::Aliasing`:
+ // - `Exclusive`: `self` is consumed by value, and therefore
+ // cannot be used to access the slice while any yielded
+ // element `Ptr` is live. Each non-zero-sized element is a
+ // disjoint byte range within the slice, and zero-sized
+ // elements address no bytes, so distinct yielded element
+ // `Ptr`s do not alias each other.
+ // - `Shared`: It is sound for multiple shared `Ptr`s to exist
+ // simultaneously which reference the same memory.
+ // 1. `elem`, conditionally, conforms to the validity invariant of
+ // `I::Alignment`. If `elem` is projected from data well-aligned
+ // for `[T]`, `elem` will be valid for `T`.
+ // 2. `elem` conforms to the validity invariant of `I::Validity`.
+ // Per https://doc.rust-lang.org/1.81.0/reference/type-layout.html#array-layout:
+ //
+ // Slices have the same layout as the section of the array they
+ // slice.
+ //
+ // Arrays are laid out so that the zero-based `nth` element of
+ // the array is offset from the start of the array by `n *
+ // size_of::<T>()` bytes. Thus, `elem` addresses a valid `T`
+ // within the slice. Since `self` satisfies `I::Validity`, `elem`
+ // also satisfies `I::Validity`.
+ self.as_inner().iter().map(
+ #[inline(always)]
+ |elem| unsafe { Ptr::from_inner(elem) },
+ )
+ }
+ }
+
+ #[allow(clippy::needless_lifetimes)]
+ impl<'a, T, I> Ptr<'a, T, I>
+ where
+ T: 'a + ?Sized + KnownLayout<PointerMetadata = usize>,
+ I: Invariants,
+ {
+ /// The number of slice elements in the object referenced by `self`.
+ #[inline]
+ #[must_use]
+ pub fn len(&self) -> usize {
+ self.as_inner().meta().get()
+ }
+
+ /// Returns `true` if the slice pointer has a length of 0.
+ #[inline]
+ #[must_use]
+ pub fn is_empty(&self) -> bool {
+ self.len() == 0
+ }
+ }
+}
+
+#[cfg(test)]
+mod tests {
+ use core::mem::{self, MaybeUninit};
+
+ use super::*;
+ #[allow(unused)] // Needed on our MSRV, but considered unused on later toolchains.
+ use crate::util::AsAddress;
+ use crate::{pointer::BecauseImmutable, util::testutil::AU64, FromBytes, Immutable};
+
+ mod test_ptr_try_cast_into_soundness {
+ use super::*;
+
+ // This test is designed so that if `Ptr::try_cast_into_xxx` are
+ // buggy, it will manifest as unsoundness that Miri can detect.
+
+ // - If `size_of::<T>() == 0`, `N == 4`
+ // - Else, `N == 4 * size_of::<T>()`
+ //
+ // Each test will be run for each metadata in `metas`.
+ fn test<T, I, const N: usize>(metas: I)
+ where
+ T: ?Sized + KnownLayout + Immutable + FromBytes,
+ I: IntoIterator<Item = Option<T::PointerMetadata>> + Clone,
+ {
+ let mut bytes = [MaybeUninit::<u8>::uninit(); N];
+ let initialized = [MaybeUninit::new(0u8); N];
+ for start in 0..=bytes.len() {
+ for end in start..=bytes.len() {
+ // Set all bytes to uninitialized other than those in
+ // the range we're going to pass to `try_cast_from`.
+ // This allows Miri to detect out-of-bounds reads
+ // because they read uninitialized memory. Without this,
+ // some out-of-bounds reads would still be in-bounds of
+ // `bytes`, and so might spuriously be accepted.
+ bytes = [MaybeUninit::<u8>::uninit(); N];
+ let bytes = &mut bytes[start..end];
+ // Initialize only the byte range we're going to pass to
+ // `try_cast_from`.
+ bytes.copy_from_slice(&initialized[start..end]);
+
+ let bytes = {
+ let bytes: *const [MaybeUninit<u8>] = bytes;
+ #[allow(clippy::as_conversions)]
+ let bytes = bytes as *const [u8];
+ // SAFETY: We just initialized these bytes to valid
+ // `u8`s.
+ unsafe { &*bytes }
+ };
+
+ // SAFETY: The bytes in `slf` must be initialized.
+ unsafe fn validate_and_get_len<
+ T: ?Sized + KnownLayout + FromBytes + Immutable,
+ >(
+ slf: Ptr<'_, T, (Shared, Aligned, Initialized)>,
+ ) -> usize {
+ let t = slf.recall_validity().as_ref();
+
+ let bytes = {
+ let len = mem::size_of_val(t);
+ let t: *const T = t;
+ // SAFETY:
+ // - We know `t`'s bytes are all initialized
+ // because we just read it from `slf`, which
+ // points to an initialized range of bytes. If
+ // there's a bug and this doesn't hold, then
+ // that's exactly what we're hoping Miri will
+ // catch!
+ // - Since `T: FromBytes`, `T` doesn't contain
+ // any `UnsafeCell`s, so it's okay for `t: T`
+ // and a `&[u8]` to the same memory to be
+ // alive concurrently.
+ unsafe { core::slice::from_raw_parts(t.cast::<u8>(), len) }
+ };
+
+ // This assertion ensures that `t`'s bytes are read
+ // and compared to another value, which in turn
+ // ensures that Miri gets a chance to notice if any
+ // of `t`'s bytes are uninitialized, which they
+ // shouldn't be (see the comment above).
+ assert_eq!(bytes, vec![0u8; bytes.len()]);
+
+ mem::size_of_val(t)
+ }
+
+ for meta in metas.clone().into_iter() {
+ for cast_type in [CastType::Prefix, CastType::Suffix] {
+ if let Ok((slf, remaining)) = Ptr::from_ref(bytes)
+ .try_cast_into::<T, BecauseImmutable>(cast_type, meta)
+ {
+ // SAFETY: All bytes in `bytes` have been
+ // initialized.
+ let len = unsafe { validate_and_get_len(slf) };
+ assert_eq!(remaining.len(), bytes.len() - len);
+ #[allow(unstable_name_collisions)]
+ let bytes_addr = bytes.as_ptr().addr();
+ #[allow(unstable_name_collisions)]
+ let remaining_addr = remaining.as_inner().as_ptr().addr();
+ match cast_type {
+ CastType::Prefix => {
+ assert_eq!(remaining_addr, bytes_addr + len)
+ }
+ CastType::Suffix => assert_eq!(remaining_addr, bytes_addr),
+ }
+
+ if let Some(want) = meta {
+ let got =
+ KnownLayout::pointer_to_metadata(slf.as_inner().as_ptr());
+ assert_eq!(got, want);
+ }
+ }
+ }
+
+ if let Ok(slf) = Ptr::from_ref(bytes)
+ .try_cast_into_no_leftover::<T, BecauseImmutable>(meta)
+ {
+ // SAFETY: All bytes in `bytes` have been
+ // initialized.
+ let len = unsafe { validate_and_get_len(slf) };
+ assert_eq!(len, bytes.len());
+
+ if let Some(want) = meta {
+ let got = KnownLayout::pointer_to_metadata(slf.as_inner().as_ptr());
+ assert_eq!(got, want);
+ }
+ }
+ }
+ }
+ }
+ }
+
+ #[derive(FromBytes, KnownLayout, Immutable)]
+ #[repr(C)]
+ struct SliceDst<T> {
+ a: u8,
+ trailing: [T],
+ }
+
+ // Each test case becomes its own `#[test]` function. We do this because
+ // this test in particular takes far, far longer to execute under Miri
+ // than all of our other tests combined. Previously, we had these
+ // execute sequentially in a single test function. We run Miri tests in
+ // parallel in CI, but this test being sequential meant that most of
+ // that parallelism was wasted, as all other tests would finish in a
+ // fraction of the total execution time, leaving this test to execute on
+ // a single thread for the remainder of the test. By putting each test
+ // case in its own function, we permit better use of available
+ // parallelism.
+ macro_rules! test {
+ ($test_name:ident: $ty:ty) => {
+ #[test]
+ #[allow(non_snake_case)]
+ fn $test_name() {
+ const S: usize = core::mem::size_of::<$ty>();
+ const N: usize = if S == 0 { 4 } else { S * 4 };
+ test::<$ty, _, N>([None]);
+
+ // If `$ty` is a ZST, then we can't pass `None` as the
+ // pointer metadata, or else computing the correct trailing
+ // slice length will panic.
+ if S == 0 {
+ test::<[$ty], _, N>([Some(0), Some(1), Some(2), Some(3)]);
+ test::<SliceDst<$ty>, _, N>([Some(0), Some(1), Some(2), Some(3)]);
+ } else {
+ test::<[$ty], _, N>([None, Some(0), Some(1), Some(2), Some(3)]);
+ test::<SliceDst<$ty>, _, N>([None, Some(0), Some(1), Some(2), Some(3)]);
+ }
+ }
+ };
+ ($ty:ident) => {
+ test!($ty: $ty);
+ };
+ ($($ty:ident),*) => { $(test!($ty);)* }
+ }
+
+ test!(empty_tuple: ());
+ test!(u8, u16, u32, u64, usize, AU64);
+ test!(i8, i16, i32, i64, isize);
+ test!(f32, f64);
+ }
+
+ #[test]
+ fn test_try_cast_into_explicit_count() {
+ macro_rules! test {
+ ($ty:ty, $bytes:expr, $elems:expr, $expect:expr) => {{
+ let bytes = [0u8; $bytes];
+ let ptr = Ptr::from_ref(&bytes[..]);
+ let res =
+ ptr.try_cast_into::<$ty, BecauseImmutable>(CastType::Prefix, Some($elems));
+ if let Some(expect) = $expect {
+ let (ptr, _) = res.unwrap();
+ assert_eq!(KnownLayout::pointer_to_metadata(ptr.as_inner().as_ptr()), expect);
+ } else {
+ let _ = res.unwrap_err();
+ }
+ }};
+ }
+
+ #[derive(KnownLayout, Immutable)]
+ #[repr(C)]
+ struct ZstDst {
+ u: [u8; 8],
+ slc: [()],
+ }
+
+ test!(ZstDst, 8, 0, Some(0));
+ test!(ZstDst, 7, 0, None);
+
+ test!(ZstDst, 8, usize::MAX, Some(usize::MAX));
+ test!(ZstDst, 7, usize::MAX, None);
+
+ #[derive(KnownLayout, Immutable)]
+ #[repr(C)]
+ struct Dst {
+ u: [u8; 8],
+ slc: [u8],
+ }
+
+ test!(Dst, 8, 0, Some(0));
+ test!(Dst, 7, 0, None);
+
+ test!(Dst, 9, 1, Some(1));
+ test!(Dst, 8, 1, None);
+
+ // If we didn't properly check for overflow, this would cause the
+ // metadata to overflow to 0, and thus the cast would spuriously
+ // succeed.
+ test!(Dst, 8, usize::MAX - 8 + 1, None);
+ }
+
+ #[test]
+ fn test_try_cast_into_no_leftover_restores_original_slice() {
+ let bytes = [0u8; 4];
+ let ptr = Ptr::from_ref(&bytes[..]);
+ let res = ptr.try_cast_into_no_leftover::<[u8; 2], BecauseImmutable>(None);
+ match res {
+ Ok(_) => panic!("should have failed due to leftover bytes"),
+ Err(CastError::Size(e)) => {
+ assert_eq!(e.into_src().len(), 4, "Should return original slice length");
+ }
+ Err(e) => panic!("wrong error type: {:?}", e),
+ }
+ }
+
+ #[test]
+ fn test_iter_exclusive_yields_disjoint_ptrs() {
+ let mut arr = [0u8, 1, 2, 3];
+
+ {
+ let mut iter = Ptr::from_mut(&mut arr[..]).iter();
+ let first = iter.next().unwrap().as_mut();
+ let second = iter.next().unwrap().as_mut();
+
+ *first = 10;
+ *second = 20;
+ *first = 30;
+ }
+
+ assert_eq!(arr, [30, 20, 2, 3]);
+ }
+}
diff --git a/rust/zerocopy/src/pointer/transmute.rs b/rust/zerocopy/src/pointer/transmute.rs
new file mode 100644
index 0000000..a534984
--- /dev/null
+++ b/rust/zerocopy/src/pointer/transmute.rs
@@ -0,0 +1,522 @@
+// SPDX-License-Identifier: (BSD-2-Clause OR Apache-2.0) OR MIT
+
+// Copyright 2025 The Fuchsia Authors
+//
+// Licensed under a BSD-style license <LICENSE-BSD>, Apache License, Version 2.0
+// <LICENSE-APACHE or https://www.apache.org/licenses/LICENSE-2.0>, or the MIT
+// license <LICENSE-MIT or https://opensource.org/licenses/MIT>, at your option.
+// This file may not be copied, modified, or distributed except according to
+// those terms.
+
+#![allow(missing_docs)]
+
+use core::{
+ cell::{Cell, UnsafeCell},
+ mem::{ManuallyDrop, MaybeUninit},
+ num::Wrapping,
+};
+
+use crate::{
+ pointer::{
+ cast::{self, CastExact, CastSizedExact},
+ invariant::*,
+ },
+ FromBytes, Immutable, IntoBytes, Unalign,
+};
+
+/// Transmutations which are sound to attempt, conditional on validating the bit
+/// validity of the destination type.
+///
+/// If a `Ptr` transmutation is `TryTransmuteFromPtr`, then it is sound to
+/// perform that transmutation so long as some additional mechanism is used to
+/// validate that the referent is bit-valid for the destination type. That
+/// validation mechanism could be a type bound (such as `TransmuteFrom`) or a
+/// runtime validity check.
+///
+/// # Safety
+///
+/// ## Post-conditions
+///
+/// Given `Dst: TryTransmuteFromPtr<Src, A, SV, DV, C, _>`, callers may assume
+/// the following:
+///
+/// Given `src: Ptr<'a, Src, (A, _, SV)>`, if the referent of `src` is
+/// `DV`-valid for `Dst`, then it is sound to transmute `src` into `dst: Ptr<'a,
+/// Dst, (A, Unaligned, DV)>` using `C`.
+///
+/// ## Pre-conditions
+///
+/// Given `src: Ptr<Src, (A, _, SV)>` and `dst: Ptr<Dst, (A, Unaligned, DV)>`,
+/// `Dst: TryTransmuteFromPtr<Src, A, SV, DV, C, _>` is sound if all of the
+/// following hold:
+/// - Forwards transmutation: Either of the following hold:
+/// - So long as `dst` is active, no mutation of `dst`'s referent is allowed
+/// except via `dst` itself
+/// - The set of `DV`-valid referents of `dst` is a superset of the set of
+/// `SV`-valid referents of `src` (NOTE: this condition effectively bans
+/// shrinking or overwriting transmutes, which cannot satisfy this
+/// condition)
+/// - Reverse transmutation: Either of the following hold:
+/// - `dst` does not permit mutation of its referent
+/// - The set of `DV`-valid referents of `dst` is a subset of the set of
+/// `SV`-valid referents of `src` (NOTE: this condition effectively bans
+/// shrinking or overwriting transmutes, which cannot satisfy this
+/// condition)
+/// - No safe code, given access to `src` and `dst`, can cause undefined
+/// behavior: Any of the following hold:
+/// - `A` is `Exclusive`
+/// - `Src: Immutable` and `Dst: Immutable`
+/// - It is sound for shared code to operate on a `&Src` and `&Dst` which
+/// reference the same byte range at the same time
+///
+/// ## Proof
+///
+/// Given:
+/// - `src: Ptr<'a, Src, (A, _, SV)>`
+/// - `src`'s referent is `DV`-valid for `Dst`
+///
+/// We are trying to prove that it is sound to perform a cast from `src` to a
+/// `dst: Ptr<'a, Dst, (A, Unaligned, DV)>` using `C`. We need to prove that
+/// such a cast does not violate any of `src`'s invariants, and that it
+/// satisfies all invariants of the destination `Ptr` type.
+///
+/// First, by `C: CastExact`, `src`'s address is unchanged, so it still satisfies
+/// its alignment. Since `dst`'s alignment is `Unaligned`, it trivially satisfies
+/// its alignment.
+///
+/// Second, aliasing is either `Exclusive` or `Shared`:
+/// - If it is `Exclusive`, then both `src` and `dst` satisfy `Exclusive`
+/// aliasing trivially: since `src` and `dst` have the same lifetime, `src` is
+/// inaccessible so long as `dst` is alive, and no other live `Ptr`s or
+/// references may reference the same referent.
+/// - If it is `Shared`, then either:
+/// - `Src: Immutable` and `Dst: Immutable`, and so neither `src` nor `dst`
+/// permit interior mutation.
+/// - It is explicitly sound for safe code to operate on a `&Src` and a `&Dst`
+/// pointing to the same byte range at the same time.
+///
+/// Third, `src`'s validity is satisfied. By invariant, `src`'s referent began
+/// as an `SV`-valid `Src`. It is guaranteed to remain so, as either of the
+/// following hold:
+/// - `dst` does not permit mutation of its referent.
+/// - The set of `DV`-valid referents of `dst` is a subset of the set of
+/// `SV`-valid referents of `src`. Thus, any value written via `dst` is
+/// guaranteed to be an `SV`-valid referent of `src`.
+///
+/// Fourth, `dst`'s validity is satisfied. It is a given of this proof that the
+/// referent is `DV`-valid for `Dst`. It is guaranteed to remain so, as either
+/// of the following hold:
+/// - So long as `dst` is active, no mutation of the referent is allowed except
+/// via `dst` itself.
+/// - The set of `DV`-valid referents of `dst` is a superset of the set of
+/// `SV`-valid referents of `src`. Thus, any value written via `src` is
+/// guaranteed to be a `DV`-valid referent of `dst`.
+pub unsafe trait TryTransmuteFromPtr<
+ Src: ?Sized,
+ A: Aliasing,
+ SV: Validity,
+ DV: Validity,
+ C: CastExact<Src, Self>,
+ R,
+>
+{
+}
+
+#[allow(missing_copy_implementations, missing_debug_implementations)]
+pub enum BecauseMutationCompatible {}
+
+// SAFETY:
+// - Forwards transmutation: By `Dst: MutationCompatible<Src, A, SV, DV, _>`, we
+// know that at least one of the following holds:
+// - So long as `dst: Ptr<Dst>` is active, no mutation of its referent is
+// allowed except via `dst` itself if either of the following hold:
+// - Aliasing is `Exclusive`, in which case, so long as the `Dst` `Ptr`
+// exists, no mutation is permitted except via that `Ptr`
+// - Aliasing is `Shared`, `Src: Immutable`, and `Dst: Immutable`, in which
+// case no mutation is possible via either `Ptr`
+// - Since the underlying cast is size-preserving, `dst` addresses the same
+// referent as `src`. By `Dst: TransmuteFrom<Src, SV, DV>`, the set of
+// `DV`-valid referents of `dst` is a superset of the set of `SV`-valid
+// referents of `src`.
+// - Reverse transmutation: Since the underlying cast is size-preserving, `dst`
+// addresses the same referent as `src`. By `Src: TransmuteFrom<Dst, DV, SV>`,
+// the set of `DV`-valid referents of `src` is a subset of the set of
+// `SV`-valid referents of `dst`.
+// - No safe code, given access to `src` and `dst`, can cause undefined
+// behavior: By `Dst: MutationCompatible<Src, A, SV, DV, _>`, at least one of
+// the following holds:
+// - `A` is `Exclusive`
+// - `Src: Immutable` and `Dst: Immutable`
+// - `Dst: InvariantsEq<Src>`, which guarantees that `Src` and `Dst` have the
+// same invariants, and permit interior mutation on the same byte ranges
+unsafe impl<Src, Dst, SV, DV, A, C, R>
+ TryTransmuteFromPtr<Src, A, SV, DV, C, (BecauseMutationCompatible, R)> for Dst
+where
+ A: Aliasing,
+ SV: Validity,
+ DV: Validity,
+ Src: TransmuteFrom<Dst, DV, SV> + ?Sized,
+ Dst: MutationCompatible<Src, A, SV, DV, R> + ?Sized,
+ C: CastExact<Src, Dst>,
+{
+}
+
+// SAFETY:
+// - Forwards transmutation: Since aliasing is `Shared` and `Src: Immutable`,
+// `src` does not permit mutation of its referent.
+// - Reverse transmutation: Since aliasing is `Shared` and `Dst: Immutable`,
+// `dst` does not permit mutation of its referent.
+// - No safe code, given access to `src` and `dst`, can cause undefined
+// behavior: `Src: Immutable` and `Dst: Immutable`
+unsafe impl<Src, Dst, SV, DV, C> TryTransmuteFromPtr<Src, Shared, SV, DV, C, BecauseImmutable>
+ for Dst
+where
+ SV: Validity,
+ DV: Validity,
+ Src: Immutable + ?Sized,
+ Dst: Immutable + ?Sized,
+ C: CastExact<Src, Dst>,
+{
+}
+
+/// Denotes that `src: Ptr<Src, (A, _, SV)>` and `dst: Ptr<Self, (A, _, DV)>`,
+/// referencing the same referent at the same time, cannot be used by safe code
+/// to break library safety invariants of `Src` or `Self`.
+///
+/// # Safety
+///
+/// At least one of the following must hold:
+/// - `Src: Read<A, _>` and `Self: Read<A, _>`
+/// - `Self: InvariantsEq<Src>`, and, for some `V`:
+/// - `Dst: TransmuteFrom<Src, V, V>`
+/// - `Src: TransmuteFrom<Dst, V, V>`
+pub unsafe trait MutationCompatible<Src: ?Sized, A: Aliasing, SV, DV, R> {}
+
+#[allow(missing_copy_implementations, missing_debug_implementations)]
+pub enum BecauseRead {}
+
+// SAFETY: `Src: Read<A, _>` and `Dst: Read<A, _>`.
+unsafe impl<Src: ?Sized, Dst: ?Sized, A: Aliasing, SV: Validity, DV: Validity, R>
+ MutationCompatible<Src, A, SV, DV, (BecauseRead, R)> for Dst
+where
+ Src: Read<A, R>,
+ Dst: Read<A, R>,
+{
+}
+
+/// Denotes that two types have the same invariants.
+///
+/// # Safety
+///
+/// It is sound for safe code to operate on a `&T` and a `&Self` pointing to the
+/// same referent at the same time - no such safe code can cause undefined
+/// behavior.
+pub unsafe trait InvariantsEq<T: ?Sized> {}
+
+// SAFETY: Trivially sound to have multiple `&T` pointing to the same referent.
+unsafe impl<T: ?Sized> InvariantsEq<T> for T {}
+
+// SAFETY: `Dst: InvariantsEq<Src> + TransmuteFrom<Src, SV, DV>`, and `Src:
+// TransmuteFrom<Dst, DV, SV>`.
+unsafe impl<Src: ?Sized, Dst: ?Sized, A: Aliasing, SV: Validity, DV: Validity>
+ MutationCompatible<Src, A, SV, DV, BecauseInvariantsEq> for Dst
+where
+ Src: TransmuteFrom<Dst, DV, SV>,
+ Dst: TransmuteFrom<Src, SV, DV> + InvariantsEq<Src>,
+{
+}
+
+#[allow(missing_debug_implementations, missing_copy_implementations)]
+pub enum BecauseInvariantsEq {}
+
+macro_rules! unsafe_impl_invariants_eq {
+ ($tyvar:ident => $t:ty, $u:ty) => {{
+ crate::util::macros::__unsafe();
+ // SAFETY: The caller promises that this is sound.
+ unsafe impl<$tyvar> InvariantsEq<$t> for $u {}
+ // SAFETY: The caller promises that this is sound.
+ unsafe impl<$tyvar> InvariantsEq<$u> for $t {}
+ }};
+}
+
+impl_transitive_transmute_from!(T => MaybeUninit<T> => T => Wrapping<T>);
+impl_transitive_transmute_from!(T => Wrapping<T> => T => MaybeUninit<T>);
+
+// SAFETY: `ManuallyDrop<T>` has the same size and bit validity as `T` [1], and
+// implements `Deref<Target = T>` [2]. Thus, it is already possible for safe
+// code to obtain a `&T` and a `&ManuallyDrop<T>` to the same referent at the
+// same time.
+//
+// [1] Per https://doc.rust-lang.org/1.81.0/std/mem/struct.ManuallyDrop.html:
+//
+// `ManuallyDrop<T>` is guaranteed to have the same layout and bit
+// validity as `T`
+//
+// [2] https://doc.rust-lang.org/1.81.0/std/mem/struct.ManuallyDrop.html#impl-Deref-for-ManuallyDrop%3CT%3E
+unsafe impl<T: ?Sized> InvariantsEq<T> for ManuallyDrop<T> {}
+// SAFETY: See previous safety comment.
+unsafe impl<T: ?Sized> InvariantsEq<ManuallyDrop<T>> for T {}
+
+/// Transmutations which are always sound.
+///
+/// `TransmuteFromPtr` is a shorthand for [`TryTransmuteFromPtr`] and
+/// [`TransmuteFrom`].
+///
+/// # Safety
+///
+/// `Dst: TransmuteFromPtr<Src, A, SV, DV, _>` is equivalent to `Dst:
+/// TryTransmuteFromPtr<Src, A, SV, DV, _> + TransmuteFrom<Src, SV, DV>`.
+pub unsafe trait TransmuteFromPtr<
+ Src: ?Sized,
+ A: Aliasing,
+ SV: Validity,
+ DV: Validity,
+ C: CastExact<Src, Self>,
+ R,
+>: TryTransmuteFromPtr<Src, A, SV, DV, C, R> + TransmuteFrom<Src, SV, DV>
+{
+}
+
+// SAFETY: The `where` bounds are equivalent to the safety invariant on
+// `TransmuteFromPtr`.
+unsafe impl<
+ Src: ?Sized,
+ Dst: ?Sized,
+ A: Aliasing,
+ SV: Validity,
+ DV: Validity,
+ C: CastExact<Src, Dst>,
+ R,
+ > TransmuteFromPtr<Src, A, SV, DV, C, R> for Dst
+where
+ Dst: TransmuteFrom<Src, SV, DV> + TryTransmuteFromPtr<Src, A, SV, DV, C, R>,
+{
+}
+
+/// Denotes that any `SV`-valid `Src` may soundly be transmuted into a
+/// `DV`-valid `Self`.
+///
+/// # Safety
+///
+/// Given `src: Ptr<Src, (_, _, SV)>` and `dst: Ptr<Dst, (_, _, DV)>`, if the
+/// referents of `src` and `dst` are the same size, then the set of bit patterns
+/// allowed to appear in `src`'s referent must be a subset of the set allowed to
+/// appear in `dst`'s referent.
+///
+/// If the referents are not the same size, then `Dst: TransmuteFrom<Src, SV,
+/// DV>` conveys no safety guarantee.
+pub unsafe trait TransmuteFrom<Src: ?Sized, SV, DV> {}
+
+/// Carries the ability to perform a size-preserving cast or conversion from a
+/// raw pointer to `Src` to a raw pointer to `Self`.
+///
+/// The cast/conversion is carried by the associated [`CastFrom`] type, and
+/// may be a no-op cast (without updating pointer metadata) or a conversion
+/// which updates pointer metadata.
+///
+/// # Safety
+///
+/// `SizeEq` on its own conveys no safety guarantee. Any safety guarantees come
+/// from the safety invariants on the associated [`CastFrom`] type, specifically
+/// the [`CastExact`] bound.
+///
+/// [`CastFrom`]: SizeEq::CastFrom
+/// [`CastExact`]: CastExact
+pub trait SizeEq<Src: ?Sized> {
+ type CastFrom: CastExact<Src, Self>;
+}
+
+impl<T: ?Sized> SizeEq<T> for T {
+ type CastFrom = cast::IdCast;
+}
+
+// SAFETY: Since `Src: IntoBytes`, the set of valid `Src`'s is the set of
+// initialized bit patterns, which is exactly the set allowed in the referent of
+// any `Initialized` `Ptr`.
+unsafe impl<Src, Dst> TransmuteFrom<Src, Valid, Initialized> for Dst
+where
+ Src: IntoBytes + ?Sized,
+ Dst: ?Sized,
+{
+}
+
+// SAFETY: Since `Dst: FromBytes`, any initialized bit pattern may appear in the
+// referent of a `Ptr<Dst, (_, _, Valid)>`. This is exactly equal to the set of
+// bit patterns which may appear in the referent of any `Initialized` `Ptr`.
+unsafe impl<Src, Dst> TransmuteFrom<Src, Initialized, Valid> for Dst
+where
+ Src: ?Sized,
+ Dst: FromBytes + ?Sized,
+{
+}
+
+// FIXME(#2354): This seems like a smell - the soundness of this bound has
+// nothing to do with `Src` or `Dst` - we're basically just saying `[u8; N]` is
+// transmutable into `[u8; N]`.
+
+// SAFETY: The set of allowed bit patterns in the referent of any `Initialized`
+// `Ptr` is the same regardless of referent type.
+unsafe impl<Src, Dst> TransmuteFrom<Src, Initialized, Initialized> for Dst
+where
+ Src: ?Sized,
+ Dst: ?Sized,
+{
+}
+
+// FIXME(#2354): This seems like a smell - the soundness of this bound has
+// nothing to do with `Dst` - we're basically just saying that any type is
+// transmutable into `MaybeUninit<[u8; N]>`.
+
+// SAFETY: A `Dst` with validity `Uninit` permits any byte sequence, and
+// therefore can be transmuted from any value.
+unsafe impl<Src, Dst, V> TransmuteFrom<Src, V, Uninit> for Dst
+where
+ Src: ?Sized,
+ Dst: ?Sized,
+ V: Validity,
+{
+}
+
+// SAFETY:
+// - `ManuallyDrop<T>` has the same size as `T` [1]
+// - `ManuallyDrop<T>` has the same validity as `T` [1]
+//
+// [1] Per https://doc.rust-lang.org/1.81.0/std/mem/struct.ManuallyDrop.html:
+//
+// `ManuallyDrop<T>` is guaranteed to have the same layout and bit validity as
+// `T`
+#[allow(clippy::multiple_unsafe_ops_per_block)]
+const _: () = unsafe { unsafe_impl_for_transparent_wrapper!(pub T: ?Sized => ManuallyDrop<T>) };
+
+// SAFETY:
+// - `Unalign<T>` promises to have the same size as `T`.
+// - `Unalign<T>` promises to have the same validity as `T`.
+#[allow(clippy::multiple_unsafe_ops_per_block)]
+const _: () = unsafe { unsafe_impl_for_transparent_wrapper!(pub T => Unalign<T>) };
+// SAFETY: `Unalign<T>` promises to have the same size and validity as `T`.
+// Given `u: &Unalign<T>`, it is already possible to obtain `let t =
+// u.try_deref().unwrap()`. Because `Unalign<T>` has the same size as `T`, the
+// returned `&T` must point to the same referent as `u`, and thus it must be
+// sound for these two references to exist at the same time since it's already
+// possible for safe code to get into this state.
+#[allow(clippy::multiple_unsafe_ops_per_block)]
+const _: () = unsafe { unsafe_impl_invariants_eq!(T => T, Unalign<T>) };
+
+// SAFETY:
+// - `Wrapping<T>` has the same size as `T` [1].
+// - `Wrapping<T>` has only one field, which is `pub` [2]. We are also
+// guaranteed per that `Wrapping<T>` has the same layout as `T` [1]. The only
+// way for both of these to be true simultaneously is for `Wrapping<T>` to
+// have the same bit validity as `T`. In particular, in order to change the
+// bit validity, one of the following would need to happen:
+// - `Wrapping` could change its `repr`, but this would violate the layout
+// guarantee.
+// - `Wrapping` could add or change its fields, but this would be a
+// stability-breaking change.
+//
+// [1] Per https://doc.rust-lang.org/1.85.0/core/num/struct.Wrapping.html#layout-1:
+//
+// `Wrapping<T>` is guaranteed to have the same layout and ABI as `T`.
+//
+// [2] Definition from https://doc.rust-lang.org/1.85.0/core/num/struct.Wrapping.html:
+//
+// ```
+// #[repr(transparent)]
+// pub struct Wrapping<T>(pub T);
+// ```
+#[allow(clippy::multiple_unsafe_ops_per_block)]
+const _: () = unsafe { unsafe_impl_for_transparent_wrapper!(pub T => Wrapping<T>) };
+
+// SAFETY: By the preceding safety proof, `Wrapping<T>` and `T` have the same
+// layout and bit validity. Since a `Wrapping<T>`'s `T` field is `pub`, given
+// `w: &Wrapping<T>`, it's possible to do `let t = &w.t`, which means that it's
+// already possible for safe code to obtain a `&Wrapping<T>` and a `&T` pointing
+// to the same referent at the same time. Thus, this must be sound.
+#[allow(clippy::multiple_unsafe_ops_per_block)]
+const _: () = unsafe { unsafe_impl_invariants_eq!(T => T, Wrapping<T>) };
+
+// SAFETY:
+// - `UnsafeCell<T>` has the same size as `T` [1].
+// - Per [1], `UnsafeCell<T>` has the same bit validity as `T`. Technically the
+// term "representation" doesn't guarantee this, but the subsequent sentence
+// in the documentation makes it clear that this is the intention.
+//
+// [1] Per https://doc.rust-lang.org/1.81.0/core/cell/struct.UnsafeCell.html#memory-layout:
+//
+// `UnsafeCell<T>` has the same in-memory representation as its inner type
+// `T`. A consequence of this guarantee is that it is possible to convert
+// between `T` and `UnsafeCell<T>`.
+#[allow(clippy::multiple_unsafe_ops_per_block)]
+const _: () = unsafe { unsafe_impl_for_transparent_wrapper!(pub T: ?Sized => UnsafeCell<T>) };
+
+// SAFETY:
+// - `Cell<T>` has the same size as `T` [1].
+// - Per [1], `Cell<T>` has the same bit validity as `T`. Technically the term
+// "representation" doesn't guarantee this, but it does promise to have the
+// "same memory layout and caveats as `UnsafeCell<T>`." The `UnsafeCell` docs
+// [2] make it clear that bit validity is the intention even if that phrase
+// isn't used.
+//
+// [1] Per https://doc.rust-lang.org/1.85.0/std/cell/struct.Cell.html#memory-layout:
+//
+// `Cell<T>` has the same memory layout and caveats as `UnsafeCell<T>`. In
+// particular, this means that `Cell<T>` has the same in-memory representation
+// as its inner type `T`.
+//
+// [2] Per https://doc.rust-lang.org/1.81.0/core/cell/struct.UnsafeCell.html#memory-layout:
+//
+// `UnsafeCell<T>` has the same in-memory representation as its inner type
+// `T`. A consequence of this guarantee is that it is possible to convert
+// between `T` and `UnsafeCell<T>`.
+#[allow(clippy::multiple_unsafe_ops_per_block)]
+const _: () = unsafe { unsafe_impl_for_transparent_wrapper!(pub T: ?Sized => Cell<T>) };
+
+impl_transitive_transmute_from!(T: ?Sized => Cell<T> => T => UnsafeCell<T>);
+impl_transitive_transmute_from!(T: ?Sized => UnsafeCell<T> => T => Cell<T>);
+
+// SAFETY: `MaybeUninit<T>` has no validity requirements. Currently this is not
+// explicitly guaranteed, but it's obvious from `MaybeUninit`'s documentation
+// that this is the intention:
+// https://doc.rust-lang.org/1.85.0/core/mem/union.MaybeUninit.html
+unsafe impl<T> TransmuteFrom<T, Uninit, Valid> for MaybeUninit<T> {}
+
+impl<T> SizeEq<T> for MaybeUninit<T> {
+ type CastFrom = CastSizedExact;
+}
+
+impl<T> SizeEq<MaybeUninit<T>> for T {
+ type CastFrom = CastSizedExact;
+}
+
+#[cfg(test)]
+mod tests {
+ use super::*;
+ use crate::pointer::cast::Project as _;
+
+ fn test_size_eq<Src, Dst: SizeEq<Src>>(mut src: Src) {
+ let _: *mut Dst =
+ <Dst as SizeEq<Src>>::CastFrom::project(crate::pointer::PtrInner::from_mut(&mut src));
+ }
+
+ #[test]
+ fn test_transmute_coverage() {
+ // SizeEq<T> for MaybeUninit<T>
+ test_size_eq::<u8, MaybeUninit<u8>>(0u8);
+
+ // SizeEq<MaybeUninit<T>> for T
+ test_size_eq::<MaybeUninit<u8>, u8>(MaybeUninit::<u8>::new(0));
+
+ // Transitive: MaybeUninit<T> -> Wrapping<T>
+ // T => MaybeUninit<T> => T => Wrapping<T>
+ test_size_eq::<u8, Wrapping<u8>>(0u8);
+
+ // T => Wrapping<T> => T => MaybeUninit<T>
+ test_size_eq::<Wrapping<u8>, MaybeUninit<u8>>(Wrapping(0u8));
+
+ // T: ?Sized => Cell<T> => T => UnsafeCell<T>
+ test_size_eq::<Cell<u8>, UnsafeCell<u8>>(Cell::new(0u8));
+
+ // T: ?Sized => UnsafeCell<T> => T => Cell<T>
+ test_size_eq::<UnsafeCell<u8>, Cell<u8>>(UnsafeCell::new(0u8));
+ }
+}
diff --git a/rust/zerocopy/src/ref.rs b/rust/zerocopy/src/ref.rs
new file mode 100644
index 0000000..860066d7
--- /dev/null
+++ b/rust/zerocopy/src/ref.rs
@@ -0,0 +1,1358 @@
+// SPDX-License-Identifier: (BSD-2-Clause OR Apache-2.0) OR MIT
+
+// Copyright 2024 The Fuchsia Authors
+//
+// Licensed under the 2-Clause BSD License <LICENSE-BSD or
+// https://opensource.org/license/bsd-2-clause>, Apache License, Version 2.0
+// <LICENSE-APACHE or https://www.apache.org/licenses/LICENSE-2.0>, or the MIT
+// license <LICENSE-MIT or https://opensource.org/licenses/MIT>, at your option.
+// This file may not be copied, modified, or distributed except according to
+// those terms.
+use super::*;
+use crate::pointer::{
+ BecauseInvariantsEq, BecauseMutationCompatible, MutationCompatible, TransmuteFromPtr,
+};
+
+mod def {
+ use core::marker::PhantomData;
+
+ use crate::{
+ ByteSlice, ByteSliceMut, CloneableByteSlice, CopyableByteSlice, IntoByteSlice,
+ IntoByteSliceMut,
+ };
+
+ /// A typed reference derived from a byte slice.
+ ///
+ /// A `Ref<B, T>` is a reference to a `T` which is stored in a byte slice, `B`.
+ /// Unlike a native reference (`&T` or `&mut T`), `Ref<B, T>` has the same
+ /// mutability as the byte slice it was constructed from (`B`).
+ ///
+ /// # Examples
+ ///
+ /// `Ref` can be used to treat a sequence of bytes as a structured type, and
+ /// to read and write the fields of that type as if the byte slice reference
+ /// were simply a reference to that type.
+ ///
+ /// ```rust
+ /// use zerocopy::*;
+ /// # use zerocopy_derive::*;
+ ///
+ /// #[derive(FromBytes, IntoBytes, KnownLayout, Immutable, Unaligned)]
+ /// #[repr(C)]
+ /// struct UdpHeader {
+ /// src_port: [u8; 2],
+ /// dst_port: [u8; 2],
+ /// length: [u8; 2],
+ /// checksum: [u8; 2],
+ /// }
+ ///
+ /// #[derive(FromBytes, IntoBytes, KnownLayout, Immutable, Unaligned)]
+ /// #[repr(C, packed)]
+ /// struct UdpPacket {
+ /// header: UdpHeader,
+ /// body: [u8],
+ /// }
+ ///
+ /// impl UdpPacket {
+ /// pub fn parse<B: ByteSlice>(bytes: B) -> Option<Ref<B, UdpPacket>> {
+ /// Ref::from_bytes(bytes).ok()
+ /// }
+ /// }
+ /// ```
+ pub struct Ref<B, T: ?Sized>(
+ // INVARIANTS: The referent (via `.deref`, `.deref_mut`, `.into`) byte
+ // slice is aligned to `T`'s alignment and its size corresponds to a
+ // valid size for `T`.
+ B,
+ PhantomData<T>,
+ );
+
+ impl<B, T: ?Sized> Ref<B, T> {
+ /// Constructs a new `Ref`.
+ ///
+ /// # Safety
+ ///
+ /// `bytes` dereferences (via [`deref`], [`deref_mut`], and [`into`]) to
+ /// a byte slice which is aligned to `T`'s alignment and whose size is a
+ /// valid size for `T`.
+ ///
+ /// [`deref`]: core::ops::Deref::deref
+ /// [`deref_mut`]: core::ops::DerefMut::deref_mut
+ /// [`into`]: core::convert::Into::into
+ pub(crate) unsafe fn new_unchecked(bytes: B) -> Ref<B, T> {
+ // INVARIANTS: The caller has promised that `bytes`'s referent is
+ // validly-aligned and has a valid size.
+ Ref(bytes, PhantomData)
+ }
+ }
+
+ impl<B: ByteSlice, T: ?Sized> Ref<B, T> {
+ /// Access the byte slice as a [`ByteSlice`].
+ ///
+ /// # Safety
+ ///
+ /// The caller promises not to call methods on the returned
+ /// [`ByteSlice`] other than `ByteSlice` methods (for example, via
+ /// `Any::downcast_ref`).
+ ///
+ /// `as_byte_slice` promises to return a `ByteSlice` whose referent is
+ /// validly-aligned for `T` and has a valid size for `T`.
+ pub(crate) unsafe fn as_byte_slice(&self) -> &impl ByteSlice {
+ // INVARIANTS: The caller promises not to call methods other than
+ // those on `ByteSlice`. Since `B: ByteSlice`, dereference stability
+ // guarantees that calling `ByteSlice` methods will not change the
+ // address or length of `self.0`'s referent.
+ //
+ // SAFETY: By invariant on `self.0`, the alignment and size
+ // post-conditions are upheld.
+ &self.0
+ }
+ }
+
+ impl<B: ByteSliceMut, T: ?Sized> Ref<B, T> {
+ /// Access the byte slice as a [`ByteSliceMut`].
+ ///
+ /// # Safety
+ ///
+ /// The caller promises not to call methods on the returned
+ /// [`ByteSliceMut`] other than `ByteSliceMut` methods (for example, via
+ /// `Any::downcast_mut`).
+ ///
+ /// `as_byte_slice` promises to return a `ByteSlice` whose referent is
+ /// validly-aligned for `T` and has a valid size for `T`.
+ pub(crate) unsafe fn as_byte_slice_mut(&mut self) -> &mut impl ByteSliceMut {
+ // INVARIANTS: The caller promises not to call methods other than
+ // those on `ByteSliceMut`. Since `B: ByteSlice`, dereference
+ // stability guarantees that calling `ByteSlice` methods will not
+ // change the address or length of `self.0`'s referent.
+ //
+ // SAFETY: By invariant on `self.0`, the alignment and size
+ // post-conditions are upheld.
+ &mut self.0
+ }
+ }
+
+ impl<'a, B: IntoByteSlice<'a>, T: ?Sized> Ref<B, T> {
+ /// Access the byte slice as an [`IntoByteSlice`].
+ ///
+ /// # Safety
+ ///
+ /// The caller promises not to call methods on the returned
+ /// [`IntoByteSlice`] other than `IntoByteSlice` methods (for example,
+ /// via `Any::downcast_ref`).
+ ///
+ /// `as_byte_slice` promises to return a `ByteSlice` whose referent is
+ /// validly-aligned for `T` and has a valid size for `T`.
+ pub(crate) unsafe fn into_byte_slice(self) -> impl IntoByteSlice<'a> {
+ // INVARIANTS: The caller promises not to call methods other than
+ // those on `IntoByteSlice`. Since `B: ByteSlice`, dereference
+ // stability guarantees that calling `ByteSlice` methods will not
+ // change the address or length of `self.0`'s referent.
+ //
+ // SAFETY: By invariant on `self.0`, the alignment and size
+ // post-conditions are upheld.
+ self.0
+ }
+ }
+
+ impl<'a, B: IntoByteSliceMut<'a>, T: ?Sized> Ref<B, T> {
+ /// Access the byte slice as an [`IntoByteSliceMut`].
+ ///
+ /// # Safety
+ ///
+ /// The caller promises not to call methods on the returned
+ /// [`IntoByteSliceMut`] other than `IntoByteSliceMut` methods (for
+ /// example, via `Any::downcast_mut`).
+ ///
+ /// `as_byte_slice` promises to return a `ByteSlice` whose referent is
+ /// validly-aligned for `T` and has a valid size for `T`.
+ pub(crate) unsafe fn into_byte_slice_mut(self) -> impl IntoByteSliceMut<'a> {
+ // INVARIANTS: The caller promises not to call methods other than
+ // those on `IntoByteSliceMut`. Since `B: ByteSlice`, dereference
+ // stability guarantees that calling `ByteSlice` methods will not
+ // change the address or length of `self.0`'s referent.
+ //
+ // SAFETY: By invariant on `self.0`, the alignment and size
+ // post-conditions are upheld.
+ self.0
+ }
+ }
+
+ impl<B: CloneableByteSlice + Clone, T: ?Sized> Clone for Ref<B, T> {
+ #[inline]
+ fn clone(&self) -> Ref<B, T> {
+ // INVARIANTS: Since `B: CloneableByteSlice`, `self.0.clone()` has
+ // the same address and length as `self.0`. Since `self.0` upholds
+ // the field invariants, so does `self.0.clone()`.
+ Ref(self.0.clone(), PhantomData)
+ }
+ }
+
+ // INVARIANTS: Since `B: CopyableByteSlice`, the copied `Ref`'s `.0` has the
+ // same address and length as the original `Ref`'s `.0`. Since the original
+ // upholds the field invariants, so does the copy.
+ impl<B: CopyableByteSlice + Copy, T: ?Sized> Copy for Ref<B, T> {}
+}
+
+#[allow(unreachable_pub)] // This is a false positive on our MSRV toolchain.
+pub use def::Ref;
+
+use crate::pointer::{
+ invariant::{Aligned, BecauseExclusive, Initialized, Unaligned, Valid},
+ BecauseRead, PtrInner,
+};
+
+impl<B, T> Ref<B, T>
+where
+ B: ByteSlice,
+{
+ #[must_use = "has no side effects"]
+ pub(crate) fn sized_from(bytes: B) -> Result<Ref<B, T>, CastError<B, T>> {
+ if bytes.len() != mem::size_of::<T>() {
+ return Err(SizeError::new(bytes).into());
+ }
+ if let Err(err) = util::validate_aligned_to::<_, T>(bytes.deref()) {
+ return Err(err.with_src(bytes).into());
+ }
+
+ // SAFETY: We just validated size and alignment.
+ Ok(unsafe { Ref::new_unchecked(bytes) })
+ }
+}
+
+impl<B, T> Ref<B, T>
+where
+ B: SplitByteSlice,
+{
+ #[must_use = "has no side effects"]
+ pub(crate) fn sized_from_prefix(bytes: B) -> Result<(Ref<B, T>, B), CastError<B, T>> {
+ if bytes.len() < mem::size_of::<T>() {
+ return Err(SizeError::new(bytes).into());
+ }
+ if let Err(err) = util::validate_aligned_to::<_, T>(bytes.deref()) {
+ return Err(err.with_src(bytes).into());
+ }
+ let (bytes, suffix) = bytes.split_at(mem::size_of::<T>()).map_err(
+ #[inline(always)]
+ |b| SizeError::new(b).into(),
+ )?;
+ // SAFETY: We just validated alignment and that `bytes` is at least as
+ // large as `T`. `bytes.split_at(mem::size_of::<T>())?` ensures that the
+ // new `bytes` is exactly the size of `T`. By safety postcondition on
+ // `SplitByteSlice::split_at` we can rely on `split_at` to produce the
+ // correct `bytes` and `suffix`.
+ let r = unsafe { Ref::new_unchecked(bytes) };
+ Ok((r, suffix))
+ }
+
+ #[must_use = "has no side effects"]
+ pub(crate) fn sized_from_suffix(bytes: B) -> Result<(B, Ref<B, T>), CastError<B, T>> {
+ let bytes_len = bytes.len();
+ let split_at = if let Some(split_at) = bytes_len.checked_sub(mem::size_of::<T>()) {
+ split_at
+ } else {
+ return Err(SizeError::new(bytes).into());
+ };
+ let (prefix, bytes) = bytes.split_at(split_at).map_err(|b| SizeError::new(b).into())?;
+ if let Err(err) = util::validate_aligned_to::<_, T>(bytes.deref()) {
+ return Err(err.with_src(bytes).into());
+ }
+ // SAFETY: Since `split_at` is defined as `bytes_len - size_of::<T>()`,
+ // the `bytes` which results from `let (prefix, bytes) =
+ // bytes.split_at(split_at)?` has length `size_of::<T>()`. After
+ // constructing `bytes`, we validate that it has the proper alignment.
+ // By safety postcondition on `SplitByteSlice::split_at` we can rely on
+ // `split_at` to produce the correct `prefix` and `bytes`.
+ let r = unsafe { Ref::new_unchecked(bytes) };
+ Ok((prefix, r))
+ }
+}
+
+impl<B, T> Ref<B, T>
+where
+ B: ByteSlice,
+ T: KnownLayout + Immutable + ?Sized,
+{
+ /// Constructs a `Ref` from a byte slice.
+ ///
+ /// If the length of `source` is not a [valid size of `T`][valid-size], or
+ /// if `source` is not appropriately aligned for `T`, this returns `Err`. If
+ /// [`T: Unaligned`][t-unaligned], you can [infallibly discard the alignment
+ /// error][size-error-from].
+ ///
+ /// `T` may be a sized type, a slice, or a [slice DST][slice-dst].
+ ///
+ /// [valid-size]: crate::KnownLayout#what-is-a-valid-size
+ /// [t-unaligned]: crate::Unaligned
+ /// [size-error-from]: error/struct.SizeError.html#method.from-1
+ /// [slice-dst]: KnownLayout#dynamically-sized-types
+ ///
+ /// # Compile-Time Assertions
+ ///
+ /// This method cannot yet be used on unsized types whose dynamically-sized
+ /// component is zero-sized. Attempting to use this method on such types
+ /// results in a compile-time assertion error; e.g.:
+ ///
+ /// ```compile_fail,E0080
+ /// use zerocopy::*;
+ /// # use zerocopy_derive::*;
+ ///
+ /// #[derive(Immutable, KnownLayout)]
+ /// #[repr(C)]
+ /// struct ZSTy {
+ /// leading_sized: u16,
+ /// trailing_dst: [()],
+ /// }
+ ///
+ /// let _ = Ref::<_, ZSTy>::from_bytes(&b"UU"[..]); // ⚠ Compile Error!
+ /// ```
+ #[must_use = "has no side effects"]
+ #[inline]
+ pub fn from_bytes(source: B) -> Result<Ref<B, T>, CastError<B, T>> {
+ static_assert_dst_is_not_zst!(T);
+ if let Err(e) =
+ Ptr::from_ref(source.deref()).try_cast_into_no_leftover::<T, BecauseImmutable>(None)
+ {
+ return Err(e.with_src(()).with_src(source));
+ }
+ // SAFETY: `try_cast_into_no_leftover` validates size and alignment.
+ Ok(unsafe { Ref::new_unchecked(source) })
+ }
+}
+
+impl<B, T> Ref<B, T>
+where
+ B: SplitByteSlice,
+ T: KnownLayout + Immutable + ?Sized,
+{
+ /// Constructs a `Ref` from the prefix of a byte slice.
+ ///
+ /// This method computes the [largest possible size of `T`][valid-size] that
+ /// can fit in the leading bytes of `source`, then attempts to return both a
+ /// `Ref` to those bytes, and a reference to the remaining bytes. If there
+ /// are insufficient bytes, or if `source` is not appropriately aligned,
+ /// this returns `Err`. If [`T: Unaligned`][t-unaligned], you can
+ /// [infallibly discard the alignment error][size-error-from].
+ ///
+ /// `T` may be a sized type, a slice, or a [slice DST][slice-dst].
+ ///
+ /// [valid-size]: crate::KnownLayout#what-is-a-valid-size
+ /// [t-unaligned]: crate::Unaligned
+ /// [size-error-from]: error/struct.SizeError.html#method.from-1
+ /// [slice-dst]: KnownLayout#dynamically-sized-types
+ ///
+ /// # Compile-Time Assertions
+ ///
+ /// This method cannot yet be used on unsized types whose dynamically-sized
+ /// component is zero-sized. Attempting to use this method on such types
+ /// results in a compile-time assertion error; e.g.:
+ ///
+ /// ```compile_fail,E0080
+ /// use zerocopy::*;
+ /// # use zerocopy_derive::*;
+ ///
+ /// #[derive(Immutable, KnownLayout)]
+ /// #[repr(C)]
+ /// struct ZSTy {
+ /// leading_sized: u16,
+ /// trailing_dst: [()],
+ /// }
+ ///
+ /// let _ = Ref::<_, ZSTy>::from_prefix(&b"UU"[..]); // ⚠ Compile Error!
+ /// ```
+ #[must_use = "has no side effects"]
+ #[inline]
+ pub fn from_prefix(source: B) -> Result<(Ref<B, T>, B), CastError<B, T>> {
+ static_assert_dst_is_not_zst!(T);
+ let remainder = match Ptr::from_ref(source.deref())
+ .try_cast_into::<T, BecauseImmutable>(CastType::Prefix, None)
+ {
+ Ok((_, remainder)) => remainder,
+ Err(e) => {
+ return Err(e.with_src(()).with_src(source));
+ }
+ };
+
+ // SAFETY: `remainder` is constructed as a subset of `source`, and so it
+ // cannot have a larger size than `source`. Both of their `len` methods
+ // measure bytes (`source` deref's to `[u8]`, and `remainder` is a
+ // `Ptr<[u8]>`), so `source.len() >= remainder.len()`. Thus, this cannot
+ // underflow.
+ #[allow(unstable_name_collisions)]
+ let split_at = unsafe { source.len().unchecked_sub(remainder.len()) };
+ let (bytes, suffix) = source.split_at(split_at).map_err(|b| SizeError::new(b).into())?;
+ // SAFETY: `try_cast_into` validates size and alignment, and returns a
+ // `split_at` that indicates how many bytes of `source` correspond to a
+ // valid `T`. By safety postcondition on `SplitByteSlice::split_at` we
+ // can rely on `split_at` to produce the correct `source` and `suffix`.
+ let r = unsafe { Ref::new_unchecked(bytes) };
+ Ok((r, suffix))
+ }
+
+ /// Constructs a `Ref` from the suffix of a byte slice.
+ ///
+ /// This method computes the [largest possible size of `T`][valid-size] that
+ /// can fit in the trailing bytes of `source`, then attempts to return both
+ /// a `Ref` to those bytes, and a reference to the preceding bytes. If there
+ /// are insufficient bytes, or if that suffix of `source` is not
+ /// appropriately aligned, this returns `Err`. If [`T:
+ /// Unaligned`][t-unaligned], you can [infallibly discard the alignment
+ /// error][size-error-from].
+ ///
+ /// `T` may be a sized type, a slice, or a [slice DST][slice-dst].
+ ///
+ /// [valid-size]: crate::KnownLayout#what-is-a-valid-size
+ /// [t-unaligned]: crate::Unaligned
+ /// [size-error-from]: error/struct.SizeError.html#method.from-1
+ /// [slice-dst]: KnownLayout#dynamically-sized-types
+ ///
+ /// # Compile-Time Assertions
+ ///
+ /// This method cannot yet be used on unsized types whose dynamically-sized
+ /// component is zero-sized. Attempting to use this method on such types
+ /// results in a compile-time assertion error; e.g.:
+ ///
+ /// ```compile_fail,E0080
+ /// use zerocopy::*;
+ /// # use zerocopy_derive::*;
+ ///
+ /// #[derive(Immutable, KnownLayout)]
+ /// #[repr(C)]
+ /// struct ZSTy {
+ /// leading_sized: u16,
+ /// trailing_dst: [()],
+ /// }
+ ///
+ /// let _ = Ref::<_, ZSTy>::from_suffix(&b"UU"[..]); // ⚠ Compile Error!
+ /// ```
+ #[must_use = "has no side effects"]
+ #[inline]
+ pub fn from_suffix(source: B) -> Result<(B, Ref<B, T>), CastError<B, T>> {
+ static_assert_dst_is_not_zst!(T);
+ let remainder = match Ptr::from_ref(source.deref())
+ .try_cast_into::<T, BecauseImmutable>(CastType::Suffix, None)
+ {
+ Ok((_, remainder)) => remainder,
+ Err(e) => {
+ let e = e.with_src(());
+ return Err(e.with_src(source));
+ }
+ };
+
+ let split_at = remainder.len();
+ let (prefix, bytes) = source.split_at(split_at).map_err(|b| SizeError::new(b).into())?;
+ // SAFETY: `try_cast_into` validates size and alignment, and returns a
+ // `split_at` that indicates how many bytes of `source` correspond to a
+ // valid `T`. By safety postcondition on `SplitByteSlice::split_at` we
+ // can rely on `split_at` to produce the correct `prefix` and `bytes`.
+ let r = unsafe { Ref::new_unchecked(bytes) };
+ Ok((prefix, r))
+ }
+}
+
+impl<B, T> Ref<B, T>
+where
+ B: ByteSlice,
+ T: KnownLayout<PointerMetadata = usize> + Immutable + ?Sized,
+{
+ /// Constructs a `Ref` from the given bytes with DST length equal to `count`
+ /// without copying.
+ ///
+ /// This method attempts to return a `Ref` to the prefix of `source`
+ /// interpreted as a `T` with `count` trailing elements, and a reference to
+ /// the remaining bytes. If the length of `source` is not equal to the size
+ /// of `Self` with `count` elements, or if `source` is not appropriately
+ /// aligned, this returns `Err`. If [`T: Unaligned`][t-unaligned], you can
+ /// [infallibly discard the alignment error][size-error-from].
+ ///
+ /// [t-unaligned]: crate::Unaligned
+ /// [size-error-from]: error/struct.SizeError.html#method.from-1
+ ///
+ /// # Compile-Time Assertions
+ ///
+ /// This method cannot yet be used on unsized types whose dynamically-sized
+ /// component is zero-sized. Attempting to use this method on such types
+ /// results in a compile-time assertion error; e.g.:
+ ///
+ /// ```compile_fail,E0080
+ /// use zerocopy::*;
+ /// # use zerocopy_derive::*;
+ ///
+ /// #[derive(Immutable, KnownLayout)]
+ /// #[repr(C)]
+ /// struct ZSTy {
+ /// leading_sized: u16,
+ /// trailing_dst: [()],
+ /// }
+ ///
+ /// let _ = Ref::<_, ZSTy>::from_bytes_with_elems(&b"UU"[..], 42); // ⚠ Compile Error!
+ /// ```
+ #[inline]
+ pub fn from_bytes_with_elems(source: B, count: usize) -> Result<Ref<B, T>, CastError<B, T>> {
+ static_assert_dst_is_not_zst!(T);
+ let expected_len = match T::size_for_metadata(count) {
+ Some(len) => len,
+ None => return Err(SizeError::new(source).into()),
+ };
+ if source.len() != expected_len {
+ return Err(SizeError::new(source).into());
+ }
+ Self::from_bytes(source)
+ }
+}
+
+impl<B, T> Ref<B, T>
+where
+ B: SplitByteSlice,
+ T: KnownLayout<PointerMetadata = usize> + Immutable + ?Sized,
+{
+ /// Constructs a `Ref` from the prefix of the given bytes with DST
+ /// length equal to `count` without copying.
+ ///
+ /// This method attempts to return a `Ref` to the prefix of `source`
+ /// interpreted as a `T` with `count` trailing elements, and a reference to
+ /// the remaining bytes. If there are insufficient bytes, or if `source` is
+ /// not appropriately aligned, this returns `Err`. If [`T:
+ /// Unaligned`][t-unaligned], you can [infallibly discard the alignment
+ /// error][size-error-from].
+ ///
+ /// [t-unaligned]: crate::Unaligned
+ /// [size-error-from]: error/struct.SizeError.html#method.from-1
+ ///
+ /// # Compile-Time Assertions
+ ///
+ /// This method cannot yet be used on unsized types whose dynamically-sized
+ /// component is zero-sized. Attempting to use this method on such types
+ /// results in a compile-time assertion error; e.g.:
+ ///
+ /// ```compile_fail,E0080
+ /// use zerocopy::*;
+ /// # use zerocopy_derive::*;
+ ///
+ /// #[derive(Immutable, KnownLayout)]
+ /// #[repr(C)]
+ /// struct ZSTy {
+ /// leading_sized: u16,
+ /// trailing_dst: [()],
+ /// }
+ ///
+ /// let _ = Ref::<_, ZSTy>::from_prefix_with_elems(&b"UU"[..], 42); // ⚠ Compile Error!
+ /// ```
+ #[inline]
+ pub fn from_prefix_with_elems(
+ source: B,
+ count: usize,
+ ) -> Result<(Ref<B, T>, B), CastError<B, T>> {
+ static_assert_dst_is_not_zst!(T);
+ let expected_len = match T::size_for_metadata(count) {
+ Some(len) => len,
+ None => return Err(SizeError::new(source).into()),
+ };
+ let (prefix, bytes) = source.split_at(expected_len).map_err(SizeError::new)?;
+ Self::from_bytes(prefix).map(move |l| (l, bytes))
+ }
+
+ /// Constructs a `Ref` from the suffix of the given bytes with DST length
+ /// equal to `count` without copying.
+ ///
+ /// This method attempts to return a `Ref` to the suffix of `source`
+ /// interpreted as a `T` with `count` trailing elements, and a reference to
+ /// the preceding bytes. If there are insufficient bytes, or if that suffix
+ /// of `source` is not appropriately aligned, this returns `Err`. If [`T:
+ /// Unaligned`][t-unaligned], you can [infallibly discard the alignment
+ /// error][size-error-from].
+ ///
+ /// [t-unaligned]: crate::Unaligned
+ /// [size-error-from]: error/struct.SizeError.html#method.from-1
+ ///
+ /// # Compile-Time Assertions
+ ///
+ /// This method cannot yet be used on unsized types whose dynamically-sized
+ /// component is zero-sized. Attempting to use this method on such types
+ /// results in a compile-time assertion error; e.g.:
+ ///
+ /// ```compile_fail,E0080
+ /// use zerocopy::*;
+ /// # use zerocopy_derive::*;
+ ///
+ /// #[derive(Immutable, KnownLayout)]
+ /// #[repr(C)]
+ /// struct ZSTy {
+ /// leading_sized: u16,
+ /// trailing_dst: [()],
+ /// }
+ ///
+ /// let _ = Ref::<_, ZSTy>::from_suffix_with_elems(&b"UU"[..], 42); // ⚠ Compile Error!
+ /// ```
+ #[inline]
+ pub fn from_suffix_with_elems(
+ source: B,
+ count: usize,
+ ) -> Result<(B, Ref<B, T>), CastError<B, T>> {
+ static_assert_dst_is_not_zst!(T);
+ let expected_len = match T::size_for_metadata(count) {
+ Some(len) => len,
+ None => return Err(SizeError::new(source).into()),
+ };
+ let split_at = if let Some(split_at) = source.len().checked_sub(expected_len) {
+ split_at
+ } else {
+ return Err(SizeError::new(source).into());
+ };
+ // SAFETY: The preceding `source.len().checked_sub(expected_len)`
+ // guarantees that `split_at` is in-bounds.
+ let (bytes, suffix) = unsafe { source.split_at_unchecked(split_at) };
+ Self::from_bytes(suffix).map(move |l| (bytes, l))
+ }
+}
+
+impl<'a, B, T> Ref<B, T>
+where
+ B: 'a + IntoByteSlice<'a>,
+ T: FromBytes + KnownLayout + Immutable + ?Sized,
+{
+ /// Converts this `Ref` into a reference.
+ ///
+ /// `into_ref` consumes the `Ref`, and returns a reference to `T`.
+ ///
+ /// Note: this is an associated function, which means that you have to call
+ /// it as `Ref::into_ref(r)` instead of `r.into_ref()`. This is so that
+ /// there is no conflict with a method on the inner type.
+ #[must_use = "has no side effects"]
+ #[inline(always)]
+ pub fn into_ref(r: Self) -> &'a T {
+ // Presumably unreachable, since we've guarded each constructor of `Ref`.
+ static_assert_dst_is_not_zst!(T);
+
+ // SAFETY: We don't call any methods on `b` other than those provided by
+ // `IntoByteSlice`.
+ let b = unsafe { r.into_byte_slice() };
+ let b = b.into_byte_slice();
+
+ if let crate::layout::SizeInfo::Sized { .. } = T::LAYOUT.size_info {
+ let ptr = Ptr::from_ref(b);
+ // SAFETY: We just checked that `T: Sized`. By invariant on `r`,
+ // `b`'s size is equal to `size_of::<T>()`.
+ let ptr = unsafe { cast_for_sized::<T, _, _, _>(ptr) };
+
+ // SAFETY: None of the preceding transformations modifies the
+ // address of the pointer, and by invariant on `r`, we know that it
+ // is validly-aligned.
+ let ptr = unsafe { ptr.assume_alignment::<Aligned>() };
+ return ptr.as_ref();
+ }
+
+ // PANICS: By post-condition on `into_byte_slice`, `b`'s size and
+ // alignment are valid for `T`. By post-condition, `b.into_byte_slice()`
+ // produces a byte slice with identical address and length to that
+ // produced by `b.deref()`.
+ let ptr = Ptr::from_ref(b.into_byte_slice())
+ .try_cast_into_no_leftover::<T, BecauseImmutable>(None)
+ .expect("zerocopy internal error: into_ref should be infallible");
+ let ptr = ptr.recall_validity();
+ ptr.as_ref()
+ }
+}
+
+impl<'a, B, T> Ref<B, T>
+where
+ B: 'a + IntoByteSliceMut<'a>,
+ T: FromBytes + IntoBytes + KnownLayout + ?Sized,
+{
+ /// Converts this `Ref` into a mutable reference.
+ ///
+ /// `into_mut` consumes the `Ref`, and returns a mutable reference to `T`.
+ ///
+ /// Note: this is an associated function, which means that you have to call
+ /// it as `Ref::into_mut(r)` instead of `r.into_mut()`. This is so that
+ /// there is no conflict with a method on the inner type.
+ #[must_use = "has no side effects"]
+ #[inline(always)]
+ pub fn into_mut(r: Self) -> &'a mut T {
+ // Presumably unreachable, since we've guarded each constructor of `Ref`.
+ static_assert_dst_is_not_zst!(T);
+
+ // SAFETY: We don't call any methods on `b` other than those provided by
+ // `IntoByteSliceMut`.
+ let b = unsafe { r.into_byte_slice_mut() };
+ let b = b.into_byte_slice_mut();
+
+ if let crate::layout::SizeInfo::Sized { .. } = T::LAYOUT.size_info {
+ let ptr = Ptr::from_mut(b);
+ // SAFETY: We just checked that `T: Sized`. By invariant on `r`,
+ // `b`'s size is equal to `size_of::<T>()`.
+ let ptr = unsafe {
+ cast_for_sized::<
+ T,
+ _,
+ (BecauseRead, BecauseExclusive),
+ (BecauseMutationCompatible, BecauseInvariantsEq),
+ >(ptr)
+ };
+
+ // SAFETY: None of the preceding transformations modifies the
+ // address of the pointer, and by invariant on `r`, we know that it
+ // is validly-aligned.
+ let ptr = unsafe { ptr.assume_alignment::<Aligned>() };
+ return ptr.as_mut();
+ }
+
+ // PANICS: By post-condition on `into_byte_slice_mut`, `b`'s size and
+ // alignment are valid for `T`. By post-condition,
+ // `b.into_byte_slice_mut()` produces a byte slice with identical
+ // address and length to that produced by `b.deref_mut()`.
+ let ptr = Ptr::from_mut(b.into_byte_slice_mut())
+ .try_cast_into_no_leftover::<T, BecauseExclusive>(None)
+ .expect("zerocopy internal error: into_ref should be infallible");
+ let ptr = ptr.recall_validity::<_, (_, (_, _))>();
+ ptr.as_mut()
+ }
+}
+
+impl<B, T> Ref<B, T>
+where
+ B: ByteSlice,
+ T: ?Sized,
+{
+ /// Gets the underlying bytes.
+ ///
+ /// Note: this is an associated function, which means that you have to call
+ /// it as `Ref::bytes(r)` instead of `r.bytes()`. This is so that there is
+ /// no conflict with a method on the inner type.
+ #[inline]
+ pub fn bytes(r: &Self) -> &[u8] {
+ // SAFETY: We don't call any methods on `b` other than those provided by
+ // `ByteSlice`.
+ unsafe { r.as_byte_slice().deref() }
+ }
+}
+
+impl<B, T> Ref<B, T>
+where
+ B: ByteSliceMut,
+ T: ?Sized,
+{
+ /// Gets the underlying bytes mutably.
+ ///
+ /// Note: this is an associated function, which means that you have to call
+ /// it as `Ref::bytes_mut(r)` instead of `r.bytes_mut()`. This is so that
+ /// there is no conflict with a method on the inner type.
+ #[inline]
+ pub fn bytes_mut(r: &mut Self) -> &mut [u8] {
+ // SAFETY: We don't call any methods on `b` other than those provided by
+ // `ByteSliceMut`.
+ unsafe { r.as_byte_slice_mut().deref_mut() }
+ }
+}
+
+impl<B, T> Ref<B, T>
+where
+ B: ByteSlice,
+ T: FromBytes,
+{
+ /// Reads a copy of `T`.
+ ///
+ /// Note: this is an associated function, which means that you have to call
+ /// it as `Ref::read(r)` instead of `r.read()`. This is so that there is no
+ /// conflict with a method on the inner type.
+ #[must_use = "has no side effects"]
+ #[inline]
+ pub fn read(r: &Self) -> T {
+ // SAFETY: We don't call any methods on `b` other than those provided by
+ // `ByteSlice`.
+ let b = unsafe { r.as_byte_slice() };
+
+ // SAFETY: By postcondition on `as_byte_slice`, we know that `b` is a
+ // valid size and alignment for `T`. By safety invariant on `ByteSlice`,
+ // we know that this is preserved via `.deref()`. Because `T:
+ // FromBytes`, it is sound to interpret these bytes as a `T`.
+ unsafe { ptr::read(b.deref().as_ptr().cast::<T>()) }
+ }
+}
+
+impl<B, T> Ref<B, T>
+where
+ B: ByteSliceMut,
+ T: IntoBytes,
+{
+ /// Writes the bytes of `t` and then forgets `t`.
+ ///
+ /// Note: this is an associated function, which means that you have to call
+ /// it as `Ref::write(r, t)` instead of `r.write(t)`. This is so that there
+ /// is no conflict with a method on the inner type.
+ #[inline]
+ pub fn write(r: &mut Self, t: T) {
+ // SAFETY: We don't call any methods on `b` other than those provided by
+ // `ByteSliceMut`.
+ let b = unsafe { r.as_byte_slice_mut() };
+
+ // SAFETY: By postcondition on `as_byte_slice_mut`, we know that `b` is
+ // a valid size and alignment for `T`. By safety invariant on
+ // `ByteSlice`, we know that this is preserved via `.deref()`. Writing
+ // `t` to the buffer will allow all of the bytes of `t` to be accessed
+ // as a `[u8]`, but because `T: IntoBytes`, we know that this is sound.
+ unsafe { ptr::write(b.deref_mut().as_mut_ptr().cast::<T>(), t) }
+ }
+}
+
+impl<B, T> Deref for Ref<B, T>
+where
+ B: ByteSlice,
+ T: FromBytes + KnownLayout + Immutable + ?Sized,
+{
+ type Target = T;
+ #[inline]
+ fn deref(&self) -> &T {
+ // Presumably unreachable, since we've guarded each constructor of `Ref`.
+ static_assert_dst_is_not_zst!(T);
+
+ // SAFETY: We don't call any methods on `b` other than those provided by
+ // `ByteSlice`.
+ let b = unsafe { self.as_byte_slice() };
+ let b = b.deref();
+
+ if let crate::layout::SizeInfo::Sized { .. } = T::LAYOUT.size_info {
+ let ptr = Ptr::from_ref(b);
+ // SAFETY: We just checked that `T: Sized`. By invariant on `r`,
+ // `b`'s size is equal to `size_of::<T>()`.
+ let ptr = unsafe { cast_for_sized::<T, _, _, _>(ptr) };
+
+ // SAFETY: None of the preceding transformations modifies the
+ // address of the pointer, and by invariant on `r`, we know that it
+ // is validly-aligned.
+ let ptr = unsafe { ptr.assume_alignment::<Aligned>() };
+ return ptr.as_ref();
+ }
+
+ // PANICS: By postcondition on `as_byte_slice`, `b`'s size and alignment
+ // are valid for `T`, and by invariant on `ByteSlice`, these are
+ // preserved through `.deref()`, so this `unwrap` will not panic.
+ let ptr = Ptr::from_ref(b)
+ .try_cast_into_no_leftover::<T, BecauseImmutable>(None)
+ .expect("zerocopy internal error: Deref::deref should be infallible");
+ let ptr = ptr.recall_validity();
+ ptr.as_ref()
+ }
+}
+
+impl<B, T> DerefMut for Ref<B, T>
+where
+ B: ByteSliceMut,
+ // FIXME(#251): We can't remove `Immutable` here because it's required by
+ // the impl of `Deref`, which is a super-trait of `DerefMut`. Maybe we can
+ // add a separate inherent method for this?
+ T: FromBytes + IntoBytes + KnownLayout + Immutable + ?Sized,
+{
+ #[inline]
+ fn deref_mut(&mut self) -> &mut T {
+ // Presumably unreachable, since we've guarded each constructor of `Ref`.
+ static_assert_dst_is_not_zst!(T);
+
+ // SAFETY: We don't call any methods on `b` other than those provided by
+ // `ByteSliceMut`.
+ let b = unsafe { self.as_byte_slice_mut() };
+ let b = b.deref_mut();
+
+ if let crate::layout::SizeInfo::Sized { .. } = T::LAYOUT.size_info {
+ let ptr = Ptr::from_mut(b);
+ // SAFETY: We just checked that `T: Sized`. By invariant on `r`,
+ // `b`'s size is equal to `size_of::<T>()`.
+ let ptr = unsafe {
+ cast_for_sized::<
+ T,
+ _,
+ (BecauseRead, BecauseExclusive),
+ (BecauseMutationCompatible, BecauseInvariantsEq),
+ >(ptr)
+ };
+
+ // SAFETY: None of the preceding transformations modifies the
+ // address of the pointer, and by invariant on `r`, we know that it
+ // is validly-aligned.
+ let ptr = unsafe { ptr.assume_alignment::<Aligned>() };
+ return ptr.as_mut();
+ }
+
+ // PANICS: By postcondition on `as_byte_slice_mut`, `b`'s size and
+ // alignment are valid for `T`, and by invariant on `ByteSlice`, these
+ // are preserved through `.deref_mut()`, so this `unwrap` will not
+ // panic.
+ let ptr = Ptr::from_mut(b)
+ .try_cast_into_no_leftover::<T, BecauseExclusive>(None)
+ .expect("zerocopy internal error: DerefMut::deref_mut should be infallible");
+ let ptr = ptr.recall_validity::<_, (_, (_, BecauseExclusive))>();
+ ptr.as_mut()
+ }
+}
+
+impl<T, B> Display for Ref<B, T>
+where
+ B: ByteSlice,
+ T: FromBytes + Display + KnownLayout + Immutable + ?Sized,
+{
+ #[inline]
+ fn fmt(&self, fmt: &mut Formatter<'_>) -> fmt::Result {
+ let inner: &T = self;
+ inner.fmt(fmt)
+ }
+}
+
+impl<T, B> Debug for Ref<B, T>
+where
+ B: ByteSlice,
+ T: FromBytes + Debug + KnownLayout + Immutable + ?Sized,
+{
+ #[inline]
+ fn fmt(&self, fmt: &mut Formatter<'_>) -> fmt::Result {
+ let inner: &T = self;
+ fmt.debug_tuple("Ref").field(&inner).finish()
+ }
+}
+
+impl<T, B> Eq for Ref<B, T>
+where
+ B: ByteSlice,
+ T: FromBytes + Eq + KnownLayout + Immutable + ?Sized,
+{
+}
+
+impl<T, B> PartialEq for Ref<B, T>
+where
+ B: ByteSlice,
+ T: FromBytes + PartialEq + KnownLayout + Immutable + ?Sized,
+{
+ #[inline]
+ fn eq(&self, other: &Self) -> bool {
+ self.deref().eq(other.deref())
+ }
+}
+
+impl<T, B> Ord for Ref<B, T>
+where
+ B: ByteSlice,
+ T: FromBytes + Ord + KnownLayout + Immutable + ?Sized,
+{
+ #[inline]
+ fn cmp(&self, other: &Self) -> Ordering {
+ let inner: &T = self;
+ let other_inner: &T = other;
+ inner.cmp(other_inner)
+ }
+}
+
+impl<T, B> PartialOrd for Ref<B, T>
+where
+ B: ByteSlice,
+ T: FromBytes + PartialOrd + KnownLayout + Immutable + ?Sized,
+{
+ #[inline]
+ fn partial_cmp(&self, other: &Self) -> Option<Ordering> {
+ let inner: &T = self;
+ let other_inner: &T = other;
+ inner.partial_cmp(other_inner)
+ }
+}
+
+/// # Safety
+///
+/// `T: Sized` and `ptr`'s referent must have size `size_of::<T>()`.
+#[inline(always)]
+unsafe fn cast_for_sized<'a, T, A, R, S>(
+ ptr: Ptr<'a, [u8], (A, Aligned, Valid)>,
+) -> Ptr<'a, T, (A, Unaligned, Valid)>
+where
+ T: FromBytes + KnownLayout + ?Sized,
+ A: crate::invariant::Aliasing,
+ [u8]: MutationCompatible<T, A, Initialized, Initialized, R>,
+ T: TransmuteFromPtr<T, A, Initialized, Valid, crate::pointer::cast::IdCast, S>,
+{
+ use crate::pointer::cast::{Cast, Project};
+
+ enum CastForSized {}
+
+ // SAFETY: `CastForSized` is only used below with the input `ptr`, which the
+ // caller promises has size `size_of::<T>()`. Thus, the referent produced in
+ // this cast has the same size as `ptr`'s referent. All operations preserve
+ // provenance.
+ unsafe impl<T: ?Sized + KnownLayout> Project<[u8], T> for CastForSized {
+ #[inline(always)]
+ fn project(src: PtrInner<'_, [u8]>) -> *mut T {
+ T::raw_from_ptr_len(
+ src.as_non_null().cast(),
+ <T::PointerMetadata as crate::PointerMetadata>::from_elem_count(0),
+ )
+ .as_ptr()
+ }
+ }
+
+ // SAFETY: The `Project::project` impl preserves referent address.
+ unsafe impl<T: ?Sized + KnownLayout> Cast<[u8], T> for CastForSized {}
+
+ ptr.recall_validity::<Initialized, (_, (_, _))>()
+ .cast::<_, CastForSized, _>()
+ .recall_validity::<Valid, _>()
+}
+
+#[cfg(test)]
+#[allow(clippy::assertions_on_result_states)]
+mod tests {
+ use core::convert::TryInto as _;
+
+ use super::*;
+ use crate::util::testutil::*;
+
+ #[test]
+ fn test_mut_slice_into_ref() {
+ // Prior to #1260/#1299, calling `into_ref` on a `&mut [u8]`-backed
+ // `Ref` was not supported.
+ let mut buf = [0u8];
+ let r = Ref::<&mut [u8], u8>::from_bytes(&mut buf).unwrap();
+ assert_eq!(Ref::into_ref(r), &0);
+ }
+
+ #[test]
+ fn test_address() {
+ // Test that the `Deref` and `DerefMut` implementations return a
+ // reference which points to the right region of memory.
+
+ let buf = [0];
+ let r = Ref::<_, u8>::from_bytes(&buf[..]).unwrap();
+ let buf_ptr = buf.as_ptr();
+ let deref_ptr: *const u8 = r.deref();
+ assert_eq!(buf_ptr, deref_ptr);
+
+ let buf = [0];
+ let r = Ref::<_, [u8]>::from_bytes(&buf[..]).unwrap();
+ let buf_ptr = buf.as_ptr();
+ let deref_ptr = r.deref().as_ptr();
+ assert_eq!(buf_ptr, deref_ptr);
+ }
+
+ // Verify that values written to a `Ref` are properly shared between the
+ // typed and untyped representations, that reads via `deref` and `read`
+ // behave the same, and that writes via `deref_mut` and `write` behave the
+ // same.
+ fn test_new_helper(mut r: Ref<&mut [u8], AU64>) {
+ // assert that the value starts at 0
+ assert_eq!(*r, AU64(0));
+ assert_eq!(Ref::read(&r), AU64(0));
+
+ // Assert that values written to the typed value are reflected in the
+ // byte slice.
+ const VAL1: AU64 = AU64(0xFF00FF00FF00FF00);
+ *r = VAL1;
+ assert_eq!(Ref::bytes(&r), &VAL1.to_bytes());
+ *r = AU64(0);
+ Ref::write(&mut r, VAL1);
+ assert_eq!(Ref::bytes(&r), &VAL1.to_bytes());
+
+ // Assert that values written to the byte slice are reflected in the
+ // typed value.
+ const VAL2: AU64 = AU64(!VAL1.0); // different from `VAL1`
+ Ref::bytes_mut(&mut r).copy_from_slice(&VAL2.to_bytes()[..]);
+ assert_eq!(*r, VAL2);
+ assert_eq!(Ref::read(&r), VAL2);
+ }
+
+ // Verify that values written to a `Ref` are properly shared between the
+ // typed and untyped representations; pass a value with `typed_len` `AU64`s
+ // backed by an array of `typed_len * 8` bytes.
+ fn test_new_helper_slice(mut r: Ref<&mut [u8], [AU64]>, typed_len: usize) {
+ // Assert that the value starts out zeroed.
+ assert_eq!(&*r, vec![AU64(0); typed_len].as_slice());
+
+ // Check the backing storage is the exact same slice.
+ let untyped_len = typed_len * 8;
+ assert_eq!(Ref::bytes(&r).len(), untyped_len);
+ assert_eq!(Ref::bytes(&r).as_ptr(), r.as_ptr().cast::<u8>());
+
+ // Assert that values written to the typed value are reflected in the
+ // byte slice.
+ const VAL1: AU64 = AU64(0xFF00FF00FF00FF00);
+ for typed in &mut *r {
+ *typed = VAL1;
+ }
+ assert_eq!(Ref::bytes(&r), VAL1.0.to_ne_bytes().repeat(typed_len).as_slice());
+
+ // Assert that values written to the byte slice are reflected in the
+ // typed value.
+ const VAL2: AU64 = AU64(!VAL1.0); // different from VAL1
+ Ref::bytes_mut(&mut r).copy_from_slice(&VAL2.0.to_ne_bytes().repeat(typed_len));
+ assert!(r.iter().copied().all(|x| x == VAL2));
+ }
+
+ #[test]
+ fn test_new_aligned_sized() {
+ // Test that a properly-aligned, properly-sized buffer works for new,
+ // new_from_prefix, and new_from_suffix, and that new_from_prefix and
+ // new_from_suffix return empty slices. Test that a properly-aligned
+ // buffer whose length is a multiple of the element size works for
+ // new_slice.
+
+ // A buffer with an alignment of 8.
+ let mut buf = Align::<[u8; 8], AU64>::default();
+ // `buf.t` should be aligned to 8, so this should always succeed.
+ test_new_helper(Ref::<_, AU64>::from_bytes(&mut buf.t[..]).unwrap());
+ {
+ // In a block so that `r` and `suffix` don't live too long.
+ buf.set_default();
+ let (r, suffix) = Ref::<_, AU64>::from_prefix(&mut buf.t[..]).unwrap();
+ assert!(suffix.is_empty());
+ test_new_helper(r);
+ }
+ {
+ buf.set_default();
+ let (prefix, r) = Ref::<_, AU64>::from_suffix(&mut buf.t[..]).unwrap();
+ assert!(prefix.is_empty());
+ test_new_helper(r);
+ }
+
+ // A buffer with alignment 8 and length 24. We choose this length very
+ // intentionally: if we instead used length 16, then the prefix and
+ // suffix lengths would be identical. In the past, we used length 16,
+ // which resulted in this test failing to discover the bug uncovered in
+ // #506.
+ let mut buf = Align::<[u8; 24], AU64>::default();
+ // `buf.t` should be aligned to 8 and have a length which is a multiple
+ // of `size_of::<AU64>()`, so this should always succeed.
+ test_new_helper_slice(Ref::<_, [AU64]>::from_bytes(&mut buf.t[..]).unwrap(), 3);
+ buf.set_default();
+ let r = Ref::<_, [AU64]>::from_bytes_with_elems(&mut buf.t[..], 3).unwrap();
+ test_new_helper_slice(r, 3);
+
+ let ascending: [u8; 24] = (0..24).collect::<Vec<_>>().try_into().unwrap();
+ // 16 ascending bytes followed by 8 zeros.
+ let mut ascending_prefix = ascending;
+ ascending_prefix[16..].copy_from_slice(&[0, 0, 0, 0, 0, 0, 0, 0]);
+ // 8 zeros followed by 16 ascending bytes.
+ let mut ascending_suffix = ascending;
+ ascending_suffix[..8].copy_from_slice(&[0, 0, 0, 0, 0, 0, 0, 0]);
+ {
+ buf.t = ascending_suffix;
+ let (r, suffix) = Ref::<_, [AU64]>::from_prefix_with_elems(&mut buf.t[..], 1).unwrap();
+ assert_eq!(suffix, &ascending[8..]);
+ test_new_helper_slice(r, 1);
+ }
+ {
+ buf.t = ascending_prefix;
+ let (prefix, r) = Ref::<_, [AU64]>::from_suffix_with_elems(&mut buf.t[..], 1).unwrap();
+ assert_eq!(prefix, &ascending[..16]);
+ test_new_helper_slice(r, 1);
+ }
+ }
+
+ #[test]
+ fn test_new_oversized() {
+ // Test that a properly-aligned, overly-sized buffer works for
+ // `new_from_prefix` and `new_from_suffix`, and that they return the
+ // remainder and prefix of the slice respectively.
+
+ let mut buf = Align::<[u8; 16], AU64>::default();
+ {
+ // In a block so that `r` and `suffix` don't live too long. `buf.t`
+ // should be aligned to 8, so this should always succeed.
+ let (r, suffix) = Ref::<_, AU64>::from_prefix(&mut buf.t[..]).unwrap();
+ assert_eq!(suffix.len(), 8);
+ test_new_helper(r);
+ }
+ {
+ buf.set_default();
+ // `buf.t` should be aligned to 8, so this should always succeed.
+ let (prefix, r) = Ref::<_, AU64>::from_suffix(&mut buf.t[..]).unwrap();
+ assert_eq!(prefix.len(), 8);
+ test_new_helper(r);
+ }
+ }
+
+ #[test]
+ #[allow(clippy::cognitive_complexity)]
+ fn test_new_error() {
+ // Fail because the buffer is too large.
+
+ // A buffer with an alignment of 8.
+ let buf = Align::<[u8; 16], AU64>::default();
+ // `buf.t` should be aligned to 8, so only the length check should fail.
+ assert!(Ref::<_, AU64>::from_bytes(&buf.t[..]).is_err());
+
+ // Fail because the buffer is too small.
+
+ // A buffer with an alignment of 8.
+ let buf = Align::<[u8; 4], AU64>::default();
+ // `buf.t` should be aligned to 8, so only the length check should fail.
+ assert!(Ref::<_, AU64>::from_bytes(&buf.t[..]).is_err());
+ assert!(Ref::<_, AU64>::from_prefix(&buf.t[..]).is_err());
+ assert!(Ref::<_, AU64>::from_suffix(&buf.t[..]).is_err());
+
+ // Fail because the length is not a multiple of the element size.
+
+ let buf = Align::<[u8; 12], AU64>::default();
+ // `buf.t` has length 12, but element size is 8.
+ assert!(Ref::<_, [AU64]>::from_bytes(&buf.t[..]).is_err());
+
+ // Fail because the buffer is too short.
+ let buf = Align::<[u8; 12], AU64>::default();
+ // `buf.t` has length 12, but the element size is 8 (and we're expecting
+ // two of them). For each function, we test with a length that would
+ // cause the size to overflow `usize`, and with a normal length that
+ // will fail thanks to the buffer being too short; these are different
+ // error paths, and while the error types are the same, the distinction
+ // shows up in code coverage metrics.
+ let n = (usize::MAX / mem::size_of::<AU64>()) + 1;
+ assert!(Ref::<_, [AU64]>::from_bytes_with_elems(&buf.t[..], n).is_err());
+ assert!(Ref::<_, [AU64]>::from_bytes_with_elems(&buf.t[..], 2).is_err());
+ assert!(Ref::<_, [AU64]>::from_prefix_with_elems(&buf.t[..], n).is_err());
+ assert!(Ref::<_, [AU64]>::from_prefix_with_elems(&buf.t[..], 2).is_err());
+ assert!(Ref::<_, [AU64]>::from_suffix_with_elems(&buf.t[..], n).is_err());
+ assert!(Ref::<_, [AU64]>::from_suffix_with_elems(&buf.t[..], 2).is_err());
+
+ // Fail because the alignment is insufficient.
+
+ // A buffer with an alignment of 8. An odd buffer size is chosen so that
+ // the last byte of the buffer has odd alignment.
+ let buf = Align::<[u8; 13], AU64>::default();
+ // Slicing from 1, we get a buffer with size 12 (so the length check
+ // should succeed) but an alignment of only 1, which is insufficient.
+ assert!(Ref::<_, AU64>::from_bytes(&buf.t[1..]).is_err());
+ assert!(Ref::<_, AU64>::from_prefix(&buf.t[1..]).is_err());
+ assert!(Ref::<_, [AU64]>::from_bytes(&buf.t[1..]).is_err());
+ assert!(Ref::<_, [AU64]>::from_bytes_with_elems(&buf.t[1..], 1).is_err());
+ assert!(Ref::<_, [AU64]>::from_prefix_with_elems(&buf.t[1..], 1).is_err());
+ assert!(Ref::<_, [AU64]>::from_suffix_with_elems(&buf.t[1..], 1).is_err());
+ // Slicing is unnecessary here because `new_from_suffix` uses the suffix
+ // of the slice, which has odd alignment.
+ assert!(Ref::<_, AU64>::from_suffix(&buf.t[..]).is_err());
+
+ // Fail due to arithmetic overflow.
+
+ let buf = Align::<[u8; 16], AU64>::default();
+ let unreasonable_len = usize::MAX / mem::size_of::<AU64>() + 1;
+ assert!(Ref::<_, [AU64]>::from_prefix_with_elems(&buf.t[..], unreasonable_len).is_err());
+ assert!(Ref::<_, [AU64]>::from_suffix_with_elems(&buf.t[..], unreasonable_len).is_err());
+ }
+
+ #[test]
+ #[allow(unstable_name_collisions)]
+ #[allow(clippy::as_conversions)]
+ fn test_into_ref_mut() {
+ #[allow(unused)]
+ use crate::util::AsAddress as _;
+
+ let mut buf = Align::<[u8; 8], u64>::default();
+ let r = Ref::<_, u64>::from_bytes(&buf.t[..]).unwrap();
+ let rf = Ref::into_ref(r);
+ assert_eq!(rf, &0u64);
+ let buf_addr = (&buf.t as *const [u8; 8]).addr();
+ assert_eq!((rf as *const u64).addr(), buf_addr);
+
+ let r = Ref::<_, u64>::from_bytes(&mut buf.t[..]).unwrap();
+ let rf = Ref::into_mut(r);
+ assert_eq!(rf, &mut 0u64);
+ assert_eq!((rf as *mut u64).addr(), buf_addr);
+
+ *rf = u64::MAX;
+ assert_eq!(buf.t, [0xFF; 8]);
+ }
+
+ #[test]
+ fn test_display_debug() {
+ let buf = Align::<[u8; 8], u64>::default();
+ let r = Ref::<_, u64>::from_bytes(&buf.t[..]).unwrap();
+ assert_eq!(format!("{}", r), "0");
+ assert_eq!(format!("{:?}", r), "Ref(0)");
+
+ let buf = Align::<[u8; 8], u64>::default();
+ let r = Ref::<_, [u64]>::from_bytes(&buf.t[..]).unwrap();
+ assert_eq!(format!("{:?}", r), "Ref([0])");
+ }
+
+ #[test]
+ fn test_eq() {
+ let buf1 = 0_u64;
+ let r1 = Ref::<_, u64>::from_bytes(buf1.as_bytes()).unwrap();
+ let buf2 = 0_u64;
+ let r2 = Ref::<_, u64>::from_bytes(buf2.as_bytes()).unwrap();
+ assert_eq!(r1, r2);
+ }
+
+ #[test]
+ fn test_ne() {
+ let buf1 = 0_u64;
+ let r1 = Ref::<_, u64>::from_bytes(buf1.as_bytes()).unwrap();
+ let buf2 = 1_u64;
+ let r2 = Ref::<_, u64>::from_bytes(buf2.as_bytes()).unwrap();
+ assert_ne!(r1, r2);
+ }
+
+ #[test]
+ fn test_ord() {
+ let buf1 = 0_u64;
+ let r1 = Ref::<_, u64>::from_bytes(buf1.as_bytes()).unwrap();
+ let buf2 = 1_u64;
+ let r2 = Ref::<_, u64>::from_bytes(buf2.as_bytes()).unwrap();
+ assert!(r1 < r2);
+ assert_eq!(PartialOrd::partial_cmp(&r1, &r2), Some(Ordering::Less));
+ assert_eq!(Ord::cmp(&r1, &r2), Ordering::Less);
+ }
+}
+
+#[cfg(all(test, __ZEROCOPY_INTERNAL_USE_ONLY_NIGHTLY_FEATURES_IN_TESTS))]
+mod benches {
+ use test::{self, Bencher};
+
+ use super::*;
+ use crate::util::testutil::*;
+
+ #[bench]
+ fn bench_from_bytes_sized(b: &mut Bencher) {
+ let buf = Align::<[u8; 8], AU64>::default();
+ // `buf.t` should be aligned to 8, so this should always succeed.
+ let bytes = &buf.t[..];
+ b.iter(|| test::black_box(Ref::<_, AU64>::from_bytes(test::black_box(bytes)).unwrap()));
+ }
+
+ #[bench]
+ fn bench_into_ref_sized(b: &mut Bencher) {
+ let buf = Align::<[u8; 8], AU64>::default();
+ let bytes = &buf.t[..];
+ let r = Ref::<_, AU64>::from_bytes(bytes).unwrap();
+ b.iter(|| test::black_box(Ref::into_ref(test::black_box(r))));
+ }
+
+ #[bench]
+ fn bench_into_mut_sized(b: &mut Bencher) {
+ let mut buf = Align::<[u8; 8], AU64>::default();
+ let buf = &mut buf.t[..];
+ let _ = Ref::<_, AU64>::from_bytes(&mut *buf).unwrap();
+ b.iter(move || {
+ // SAFETY: The preceding `from_bytes` succeeded, and so we know that
+ // `buf` is validly-aligned and has the correct length.
+ let r = unsafe { Ref::<&mut [u8], AU64>::new_unchecked(&mut *buf) };
+ test::black_box(Ref::into_mut(test::black_box(r)));
+ });
+ }
+
+ #[bench]
+ fn bench_deref_sized(b: &mut Bencher) {
+ let buf = Align::<[u8; 8], AU64>::default();
+ let bytes = &buf.t[..];
+ let r = Ref::<_, AU64>::from_bytes(bytes).unwrap();
+ b.iter(|| {
+ let temp = test::black_box(r);
+ test::black_box(temp.deref());
+ });
+ }
+
+ #[bench]
+ fn bench_deref_mut_sized(b: &mut Bencher) {
+ let mut buf = Align::<[u8; 8], AU64>::default();
+ let buf = &mut buf.t[..];
+ let _ = Ref::<_, AU64>::from_bytes(&mut *buf).unwrap();
+ b.iter(|| {
+ // SAFETY: The preceding `from_bytes` succeeded, and so we know that
+ // `buf` is validly-aligned and has the correct length.
+ let r = unsafe { Ref::<&mut [u8], AU64>::new_unchecked(&mut *buf) };
+ let mut temp = test::black_box(r);
+ test::black_box(temp.deref_mut());
+ });
+ }
+}
diff --git a/rust/zerocopy/src/split_at.rs b/rust/zerocopy/src/split_at.rs
new file mode 100644
index 0000000..9a67d5a
--- /dev/null
+++ b/rust/zerocopy/src/split_at.rs
@@ -0,0 +1,1090 @@
+// SPDX-License-Identifier: (BSD-2-Clause OR Apache-2.0) OR MIT
+
+// Copyright 2025 The Fuchsia Authors
+//
+// Licensed under the 2-Clause BSD License <LICENSE-BSD or
+// https://opensource.org/license/bsd-2-clause>, Apache License, Version 2.0
+// <LICENSE-APACHE or https://www.apache.org/licenses/LICENSE-2.0>, or the MIT
+// license <LICENSE-MIT or https://opensource.org/licenses/MIT>, at your option.
+// This file may not be copied, modified, or distributed except according to
+// those terms.
+
+use super::*;
+use crate::pointer::invariant::{Aligned, Exclusive, Invariants, Shared, Valid};
+
+/// Types that can be split in two.
+///
+/// This trait generalizes Rust's existing support for splitting slices to
+/// support slices and slice-based dynamically-sized types ("slice DSTs").
+///
+/// # Implementation
+///
+/// **Do not implement this trait yourself!** Instead, use
+/// [`#[derive(SplitAt)]`][derive]; e.g.:
+///
+/// ```
+/// # use zerocopy_derive::{SplitAt, KnownLayout};
+/// #[derive(SplitAt, KnownLayout)]
+/// #[repr(C)]
+/// struct MyStruct<T: ?Sized> {
+/// # /*
+/// ...,
+/// # */
+/// // `SplitAt` types must have at least one field.
+/// field: T,
+/// }
+/// ```
+///
+/// This derive performs a sophisticated, compile-time safety analysis to
+/// determine whether a type is `SplitAt`.
+///
+/// # Safety
+///
+/// This trait does not convey any safety guarantees to code outside this crate.
+///
+/// You must not rely on the `#[doc(hidden)]` internals of `SplitAt`. Future
+/// releases of zerocopy may make backwards-breaking changes to these items,
+/// including changes that only affect soundness, which may cause code which
+/// uses those items to silently become unsound.
+///
+#[cfg_attr(feature = "derive", doc = "[derive]: zerocopy_derive::SplitAt")]
+#[cfg_attr(
+ not(feature = "derive"),
+ doc = concat!("[derive]: https://docs.rs/zerocopy/", env!("CARGO_PKG_VERSION"), "/zerocopy/derive.SplitAt.html"),
+)]
+#[cfg_attr(
+ not(no_zerocopy_diagnostic_on_unimplemented_1_78_0),
+ diagnostic::on_unimplemented(note = "Consider adding `#[derive(SplitAt)]` to `{Self}`")
+)]
+// # Safety
+//
+// The trailing slice is well-aligned for its element type. `Self` is `[T]`, or
+// a `repr(C)` or `repr(transparent)` slice DST.
+pub unsafe trait SplitAt: KnownLayout<PointerMetadata = usize> {
+ /// The element type of the trailing slice.
+ type Elem;
+
+ #[doc(hidden)]
+ fn only_derive_is_allowed_to_implement_this_trait()
+ where
+ Self: Sized;
+
+ /// Unsafely splits `self` in two.
+ ///
+ /// # Safety
+ ///
+ /// The caller promises that `l_len` is not greater than the length of
+ /// `self`'s trailing slice.
+ ///
+ #[doc = codegen_section!(
+ header = "h5",
+ bench = "split_at_unchecked",
+ format = "coco",
+ arity = 2,
+ [
+ open
+ @index 1
+ @title "Unsized"
+ @variant "dynamic_size"
+ ],
+ [
+ @index 2
+ @title "Dynamically Padded"
+ @variant "dynamic_padding"
+ ]
+ )]
+ #[inline]
+ #[must_use]
+ unsafe fn split_at_unchecked(&self, l_len: usize) -> Split<&Self> {
+ // SAFETY: By precondition on the caller, `l_len <= self.len()`.
+ unsafe { Split::<&Self>::new(self, l_len) }
+ }
+
+ /// Attempts to split `self` in two.
+ ///
+ /// Returns `None` if `l_len` is greater than the length of `self`'s
+ /// trailing slice.
+ ///
+ /// # Examples
+ ///
+ /// ```
+ /// use zerocopy::{SplitAt, FromBytes};
+ /// # use zerocopy_derive::*;
+ ///
+ /// #[derive(SplitAt, FromBytes, KnownLayout, Immutable)]
+ /// #[repr(C)]
+ /// struct Packet {
+ /// length: u8,
+ /// body: [u8],
+ /// }
+ ///
+ /// // These bytes encode a `Packet`.
+ /// let bytes = &[4, 1, 2, 3, 4, 5, 6, 7, 8, 9][..];
+ ///
+ /// let packet = Packet::ref_from_bytes(bytes).unwrap();
+ ///
+ /// assert_eq!(packet.length, 4);
+ /// assert_eq!(packet.body, [1, 2, 3, 4, 5, 6, 7, 8, 9]);
+ ///
+ /// // Attempt to split `packet` at `length`.
+ /// let split = packet.split_at(packet.length as usize).unwrap();
+ ///
+ /// // Use the `Immutable` bound on `Packet` to prove that it's okay to
+ /// // return concurrent references to `packet` and `rest`.
+ /// let (packet, rest) = split.via_immutable();
+ ///
+ /// assert_eq!(packet.length, 4);
+ /// assert_eq!(packet.body, [1, 2, 3, 4]);
+ /// assert_eq!(rest, [5, 6, 7, 8, 9]);
+ /// ```
+ ///
+ #[doc = codegen_section!(
+ header = "h5",
+ bench = "split_at",
+ format = "coco",
+ arity = 2,
+ [
+ open
+ @index 1
+ @title "Unsized"
+ @variant "dynamic_size"
+ ],
+ [
+ @index 2
+ @title "Dynamically Padded"
+ @variant "dynamic_padding"
+ ]
+ )]
+ #[inline]
+ #[must_use = "has no side effects"]
+ fn split_at(&self, l_len: usize) -> Option<Split<&Self>> {
+ MetadataOf::new_in_bounds(self, l_len).map(
+ #[inline(always)]
+ |l_len| {
+ // SAFETY: We have ensured that `l_len <= self.len()` (by
+ // post-condition on `MetadataOf::new_in_bounds`)
+ unsafe { Split::new(self, l_len.get()) }
+ },
+ )
+ }
+
+ /// Unsafely splits `self` in two.
+ ///
+ /// # Safety
+ ///
+ /// The caller promises that `l_len` is not greater than the length of
+ /// `self`'s trailing slice.
+ ///
+ #[doc = codegen_header!("h5", "split_at_mut_unchecked")]
+ ///
+ /// See [`SplitAt::split_at_unchecked`](#method.split_at_unchecked.codegen).
+ #[inline]
+ #[must_use]
+ unsafe fn split_at_mut_unchecked(&mut self, l_len: usize) -> Split<&mut Self> {
+ // SAFETY: By precondition on the caller, `l_len <= self.len()`.
+ unsafe { Split::<&mut Self>::new(self, l_len) }
+ }
+
+ /// Attempts to split `self` in two.
+ ///
+ /// Returns `None` if `l_len` is greater than the length of `self`'s
+ /// trailing slice, or if the given `l_len` would result in [the trailing
+ /// padding](KnownLayout#slice-dst-layout) of the left portion overlapping
+ /// the right portion.
+ ///
+ ///
+ /// # Examples
+ ///
+ /// ```
+ /// use zerocopy::{SplitAt, FromBytes};
+ /// # use zerocopy_derive::*;
+ ///
+ /// #[derive(SplitAt, FromBytes, KnownLayout, IntoBytes)]
+ /// #[repr(C)]
+ /// struct Packet<B: ?Sized> {
+ /// length: u8,
+ /// body: B,
+ /// }
+ ///
+ /// // These bytes encode a `Packet`.
+ /// let mut bytes = &mut [4, 1, 2, 3, 4, 5, 6, 7, 8, 9][..];
+ ///
+ /// let packet = Packet::<[u8]>::mut_from_bytes(bytes).unwrap();
+ ///
+ /// assert_eq!(packet.length, 4);
+ /// assert_eq!(packet.body, [1, 2, 3, 4, 5, 6, 7, 8, 9]);
+ ///
+ /// {
+ /// // Attempt to split `packet` at `length`.
+ /// let split = packet.split_at_mut(packet.length as usize).unwrap();
+ ///
+ /// // Use the `IntoBytes` bound on `Packet` to prove that it's okay to
+ /// // return concurrent references to `packet` and `rest`.
+ /// let (packet, rest) = split.via_into_bytes();
+ ///
+ /// assert_eq!(packet.length, 4);
+ /// assert_eq!(packet.body, [1, 2, 3, 4]);
+ /// assert_eq!(rest, [5, 6, 7, 8, 9]);
+ ///
+ /// rest.fill(0);
+ /// }
+ ///
+ /// assert_eq!(packet.length, 4);
+ /// assert_eq!(packet.body, [1, 2, 3, 4, 0, 0, 0, 0, 0]);
+ /// ```
+ ///
+ #[doc = codegen_header!("h5", "split_at_mut")]
+ ///
+ /// See [`SplitAt::split_at`](#method.split_at.codegen).
+ #[inline]
+ fn split_at_mut(&mut self, l_len: usize) -> Option<Split<&mut Self>> {
+ MetadataOf::new_in_bounds(self, l_len).map(
+ #[inline(always)]
+ |l_len| {
+ // SAFETY: We have ensured that `l_len <= self.len()` (by
+ // post-condition on `MetadataOf::new_in_bounds`)
+ unsafe { Split::new(self, l_len.get()) }
+ },
+ )
+ }
+}
+
+// SAFETY: `[T]`'s trailing slice is `[T]`, which is trivially aligned.
+unsafe impl<T> SplitAt for [T] {
+ type Elem = T;
+
+ #[inline]
+ #[allow(dead_code)]
+ fn only_derive_is_allowed_to_implement_this_trait()
+ where
+ Self: Sized,
+ {
+ }
+}
+
+/// A `T` that has been split into two possibly-overlapping parts.
+///
+/// For some dynamically sized types, the padding that appears after the
+/// trailing slice field [is a dynamic function of the trailing slice
+/// length](KnownLayout#slice-dst-layout). If `T` is split at a length that
+/// requires trailing padding, the trailing padding of the left part of the
+/// split `T` will overlap the right part. If `T` is a mutable reference or
+/// permits interior mutation, you must ensure that the left and right parts do
+/// not overlap. You can do this at zero-cost using using
+/// [`Self::via_immutable`], [`Self::via_into_bytes`], or
+/// [`Self::via_unaligned`], or with a dynamic check by using
+/// [`Self::via_runtime_check`].
+#[derive(Debug)]
+pub struct Split<T> {
+ /// A pointer to the source slice DST.
+ source: T,
+ /// The length of the future left half of `source`.
+ ///
+ /// # Safety
+ ///
+ /// If `source` is a pointer to a slice DST, `l_len` is no greater than
+ /// `source`'s length.
+ l_len: usize,
+}
+
+impl<T> Split<T> {
+ /// Produces a `Split` of `source` with `l_len`.
+ ///
+ /// # Safety
+ ///
+ /// `l_len` is no greater than `source`'s length.
+ #[inline(always)]
+ unsafe fn new(source: T, l_len: usize) -> Self {
+ Self { source, l_len }
+ }
+}
+
+impl<'a, T> Split<&'a T>
+where
+ T: ?Sized + SplitAt,
+{
+ #[inline(always)]
+ fn into_ptr(self) -> Split<Ptr<'a, T, (Shared, Aligned, Valid)>> {
+ let source = Ptr::from_ref(self.source);
+ // SAFETY: `Ptr::from_ref(self.source)` points to exactly `self.source`
+ // and thus maintains the invariants of `self` with respect to `l_len`.
+ unsafe { Split::new(source, self.l_len) }
+ }
+
+ /// Produces the split parts of `self`, using [`Immutable`] to ensure that
+ /// it is sound to have concurrent references to both parts.
+ ///
+ /// # Examples
+ ///
+ /// ```
+ /// use zerocopy::{SplitAt, FromBytes};
+ /// # use zerocopy_derive::*;
+ ///
+ /// #[derive(SplitAt, FromBytes, KnownLayout, Immutable)]
+ /// #[repr(C)]
+ /// struct Packet {
+ /// length: u8,
+ /// body: [u8],
+ /// }
+ ///
+ /// // These bytes encode a `Packet`.
+ /// let bytes = &[4, 1, 2, 3, 4, 5, 6, 7, 8, 9][..];
+ ///
+ /// let packet = Packet::ref_from_bytes(bytes).unwrap();
+ ///
+ /// assert_eq!(packet.length, 4);
+ /// assert_eq!(packet.body, [1, 2, 3, 4, 5, 6, 7, 8, 9]);
+ ///
+ /// // Attempt to split `packet` at `length`.
+ /// let split = packet.split_at(packet.length as usize).unwrap();
+ ///
+ /// // Use the `Immutable` bound on `Packet` to prove that it's okay to
+ /// // return concurrent references to `packet` and `rest`.
+ /// let (packet, rest) = split.via_immutable();
+ ///
+ /// assert_eq!(packet.length, 4);
+ /// assert_eq!(packet.body, [1, 2, 3, 4]);
+ /// assert_eq!(rest, [5, 6, 7, 8, 9]);
+ /// ```
+ ///
+ #[doc = codegen_section!(
+ header = "h5",
+ bench = "split_via_immutable",
+ format = "coco",
+ arity = 2,
+ [
+ open
+ @index 1
+ @title "Unsized"
+ @variant "dynamic_size"
+ ],
+ [
+ @index 2
+ @title "Dynamically Padded"
+ @variant "dynamic_padding"
+ ]
+ )]
+ #[must_use = "has no side effects"]
+ #[inline(always)]
+ pub fn via_immutable(self) -> (&'a T, &'a [T::Elem])
+ where
+ T: Immutable,
+ {
+ let (l, r) = self.into_ptr().via_immutable();
+ (l.as_ref(), r.as_ref())
+ }
+
+ /// Produces the split parts of `self`, using [`IntoBytes`] to ensure that
+ /// it is sound to have concurrent references to both parts.
+ ///
+ /// # Examples
+ ///
+ /// ```
+ /// use zerocopy::{SplitAt, FromBytes};
+ /// # use zerocopy_derive::*;
+ ///
+ /// #[derive(SplitAt, FromBytes, KnownLayout, Immutable, IntoBytes)]
+ /// #[repr(C)]
+ /// struct Packet<B: ?Sized> {
+ /// length: u8,
+ /// body: B,
+ /// }
+ ///
+ /// // These bytes encode a `Packet`.
+ /// let bytes = &[4, 1, 2, 3, 4, 5, 6, 7, 8, 9][..];
+ ///
+ /// let packet = Packet::<[u8]>::ref_from_bytes(bytes).unwrap();
+ ///
+ /// assert_eq!(packet.length, 4);
+ /// assert_eq!(packet.body, [1, 2, 3, 4, 5, 6, 7, 8, 9]);
+ ///
+ /// // Attempt to split `packet` at `length`.
+ /// let split = packet.split_at(packet.length as usize).unwrap();
+ ///
+ /// // Use the `IntoBytes` bound on `Packet` to prove that it's okay to
+ /// // return concurrent references to `packet` and `rest`.
+ /// let (packet, rest) = split.via_into_bytes();
+ ///
+ /// assert_eq!(packet.length, 4);
+ /// assert_eq!(packet.body, [1, 2, 3, 4]);
+ /// assert_eq!(rest, [5, 6, 7, 8, 9]);
+ /// ```
+ ///
+ #[doc = codegen_header!("h5", "split_via_into_bytes")]
+ ///
+ /// See [`Split::via_immutable`](#method.split_via_immutable.codegen).
+ #[must_use = "has no side effects"]
+ #[inline(always)]
+ pub fn via_into_bytes(self) -> (&'a T, &'a [T::Elem])
+ where
+ T: IntoBytes,
+ {
+ let (l, r) = self.into_ptr().via_into_bytes();
+ (l.as_ref(), r.as_ref())
+ }
+
+ /// Produces the split parts of `self`, using [`Unaligned`] to ensure that
+ /// it is sound to have concurrent references to both parts.
+ ///
+ /// # Examples
+ ///
+ /// ```
+ /// use zerocopy::{SplitAt, FromBytes};
+ /// # use zerocopy_derive::*;
+ ///
+ /// #[derive(SplitAt, FromBytes, KnownLayout, Immutable, Unaligned)]
+ /// #[repr(C)]
+ /// struct Packet {
+ /// length: u8,
+ /// body: [u8],
+ /// }
+ ///
+ /// // These bytes encode a `Packet`.
+ /// let bytes = &[4, 1, 2, 3, 4, 5, 6, 7, 8, 9][..];
+ ///
+ /// let packet = Packet::ref_from_bytes(bytes).unwrap();
+ ///
+ /// assert_eq!(packet.length, 4);
+ /// assert_eq!(packet.body, [1, 2, 3, 4, 5, 6, 7, 8, 9]);
+ ///
+ /// // Attempt to split `packet` at `length`.
+ /// let split = packet.split_at(packet.length as usize).unwrap();
+ ///
+ /// // Use the `Unaligned` bound on `Packet` to prove that it's okay to
+ /// // return concurrent references to `packet` and `rest`.
+ /// let (packet, rest) = split.via_unaligned();
+ ///
+ /// assert_eq!(packet.length, 4);
+ /// assert_eq!(packet.body, [1, 2, 3, 4]);
+ /// assert_eq!(rest, [5, 6, 7, 8, 9]);
+ /// ```
+ ///
+ #[doc = codegen_header!("h5", "split_via_unaligned")]
+ ///
+ /// See [`Split::via_immutable`](#method.split_via_immutable.codegen).
+ #[must_use = "has no side effects"]
+ #[inline(always)]
+ pub fn via_unaligned(self) -> (&'a T, &'a [T::Elem])
+ where
+ T: Unaligned,
+ {
+ let (l, r) = self.into_ptr().via_unaligned();
+ (l.as_ref(), r.as_ref())
+ }
+
+ /// Produces the split parts of `self`, using a dynamic check to ensure that
+ /// it is sound to have concurrent references to both parts. You should
+ /// prefer using [`Self::via_immutable`], [`Self::via_into_bytes`], or
+ /// [`Self::via_unaligned`], which have no runtime cost.
+ ///
+ /// Note that this check is overly conservative if `T` is [`Immutable`]; for
+ /// some types, this check will reject some splits which
+ /// [`Self::via_immutable`] will accept.
+ ///
+ /// # Examples
+ ///
+ /// ```
+ /// use zerocopy::{SplitAt, FromBytes, IntoBytes, network_endian::U16};
+ /// # use zerocopy_derive::*;
+ ///
+ /// #[derive(SplitAt, FromBytes, KnownLayout, Immutable, Debug)]
+ /// #[repr(C, align(2))]
+ /// struct Packet {
+ /// length: U16,
+ /// body: [u8],
+ /// }
+ ///
+ /// // These bytes encode a `Packet`.
+ /// let bytes = [
+ /// 4u16.to_be(),
+ /// 1u16.to_be(),
+ /// 2u16.to_be(),
+ /// 3u16.to_be(),
+ /// 4u16.to_be()
+ /// ];
+ ///
+ /// let packet = Packet::ref_from_bytes(bytes.as_bytes()).unwrap();
+ ///
+ /// assert_eq!(packet.length, 4);
+ /// assert_eq!(packet.body, [0, 1, 0, 2, 0, 3, 0, 4]);
+ ///
+ /// // Attempt to split `packet` at `length`.
+ /// let split = packet.split_at(packet.length.into()).unwrap();
+ ///
+ /// // Use a dynamic check to prove that it's okay to return concurrent
+ /// // references to `packet` and `rest`.
+ /// let (packet, rest) = split.via_runtime_check().unwrap();
+ ///
+ /// assert_eq!(packet.length, 4);
+ /// assert_eq!(packet.body, [0, 1, 0, 2]);
+ /// assert_eq!(rest, [0, 3, 0, 4]);
+ ///
+ /// // Attempt to split `packet` at `length - 1`.
+ /// let idx = packet.length.get() - 1;
+ /// let split = packet.split_at(idx as usize).unwrap();
+ ///
+ /// // Attempt (and fail) to use a dynamic check to prove that it's okay
+ /// // to return concurrent references to `packet` and `rest`. Note that
+ /// // this is a case of `via_runtime_check` being overly conservative.
+ /// // Although the left and right parts indeed overlap, the `Immutable`
+ /// // bound ensures that concurrently referencing these overlapping
+ /// // parts is sound.
+ /// assert!(split.via_runtime_check().is_err());
+ /// ```
+ ///
+ #[doc = codegen_section!(
+ header = "h5",
+ bench = "split_via_runtime_check",
+ format = "coco",
+ arity = 2,
+ [
+ open
+ @index 1
+ @title "Unsized"
+ @variant "dynamic_size"
+ ],
+ [
+ @index 2
+ @title "Dynamically Padded"
+ @variant "dynamic_padding"
+ ]
+ )]
+ #[must_use = "has no side effects"]
+ #[inline(always)]
+ pub fn via_runtime_check(self) -> Result<(&'a T, &'a [T::Elem]), Self> {
+ match self.into_ptr().via_runtime_check() {
+ Ok((l, r)) => Ok((l.as_ref(), r.as_ref())),
+ Err(s) => Err(s.into_ref()),
+ }
+ }
+
+ /// Unsafely produces the split parts of `self`.
+ ///
+ /// # Safety
+ ///
+ /// If `T` permits interior mutation, the trailing padding bytes of the left
+ /// portion must not overlap the right portion. For some dynamically sized
+ /// types, the padding that appears after the trailing slice field [is a
+ /// dynamic function of the trailing slice
+ /// length](KnownLayout#slice-dst-layout). Thus, for some types, this
+ /// condition is dependent on the length of the left portion.
+ ///
+ #[doc = codegen_section!(
+ header = "h5",
+ bench = "split_via_unchecked",
+ format = "coco",
+ arity = 2,
+ [
+ open
+ @index 1
+ @title "Unsized"
+ @variant "dynamic_size"
+ ],
+ [
+ @index 2
+ @title "Dynamically Padded"
+ @variant "dynamic_padding"
+ ]
+ )]
+ #[must_use = "has no side effects"]
+ #[inline(always)]
+ pub unsafe fn via_unchecked(self) -> (&'a T, &'a [T::Elem]) {
+ // SAFETY: The aliasing of `self.into_ptr()` is not `Exclusive`, but the
+ // caller has promised that if `T` permits interior mutation then the
+ // left and right portions of `self` split at `l_len` do not overlap.
+ let (l, r) = unsafe { self.into_ptr().via_unchecked() };
+ (l.as_ref(), r.as_ref())
+ }
+}
+
+impl<'a, T> Split<&'a mut T>
+where
+ T: ?Sized + SplitAt,
+{
+ #[inline(always)]
+ fn into_ptr(self) -> Split<Ptr<'a, T, (Exclusive, Aligned, Valid)>> {
+ let source = Ptr::from_mut(self.source);
+ // SAFETY: `Ptr::from_mut(self.source)` points to exactly `self.source`,
+ // and thus maintains the invariants of `self` with respect to `l_len`.
+ unsafe { Split::new(source, self.l_len) }
+ }
+
+ /// Produces the split parts of `self`, using [`IntoBytes`] to ensure that
+ /// it is sound to have concurrent references to both parts.
+ ///
+ /// # Examples
+ ///
+ /// ```
+ /// use zerocopy::{SplitAt, FromBytes};
+ /// # use zerocopy_derive::*;
+ ///
+ /// #[derive(SplitAt, FromBytes, KnownLayout, IntoBytes)]
+ /// #[repr(C)]
+ /// struct Packet<B: ?Sized> {
+ /// length: u8,
+ /// body: B,
+ /// }
+ ///
+ /// // These bytes encode a `Packet`.
+ /// let mut bytes = &mut [4, 1, 2, 3, 4, 5, 6, 7, 8, 9][..];
+ ///
+ /// let packet = Packet::<[u8]>::mut_from_bytes(bytes).unwrap();
+ ///
+ /// assert_eq!(packet.length, 4);
+ /// assert_eq!(packet.body, [1, 2, 3, 4, 5, 6, 7, 8, 9]);
+ ///
+ /// {
+ /// // Attempt to split `packet` at `length`.
+ /// let split = packet.split_at_mut(packet.length as usize).unwrap();
+ ///
+ /// // Use the `IntoBytes` bound on `Packet` to prove that it's okay to
+ /// // return concurrent references to `packet` and `rest`.
+ /// let (packet, rest) = split.via_into_bytes();
+ ///
+ /// assert_eq!(packet.length, 4);
+ /// assert_eq!(packet.body, [1, 2, 3, 4]);
+ /// assert_eq!(rest, [5, 6, 7, 8, 9]);
+ ///
+ /// rest.fill(0);
+ /// }
+ ///
+ /// assert_eq!(packet.length, 4);
+ /// assert_eq!(packet.body, [1, 2, 3, 4, 0, 0, 0, 0, 0]);
+ /// ```
+ ///
+ /// # Code Generation
+ ///
+ /// See [`Split::via_immutable`](#method.split_via_immutable.codegen).
+ #[must_use = "has no side effects"]
+ #[inline(always)]
+ pub fn via_into_bytes(self) -> (&'a mut T, &'a mut [T::Elem])
+ where
+ T: IntoBytes,
+ {
+ let (l, r) = self.into_ptr().via_into_bytes();
+ (l.as_mut(), r.as_mut())
+ }
+
+ /// Produces the split parts of `self`, using [`Unaligned`] to ensure that
+ /// it is sound to have concurrent references to both parts.
+ ///
+ /// # Examples
+ ///
+ /// ```
+ /// use zerocopy::{SplitAt, FromBytes};
+ /// # use zerocopy_derive::*;
+ ///
+ /// #[derive(SplitAt, FromBytes, KnownLayout, IntoBytes, Unaligned)]
+ /// #[repr(C)]
+ /// struct Packet<B: ?Sized> {
+ /// length: u8,
+ /// body: B,
+ /// }
+ ///
+ /// // These bytes encode a `Packet`.
+ /// let mut bytes = &mut [4, 1, 2, 3, 4, 5, 6, 7, 8, 9][..];
+ ///
+ /// let packet = Packet::<[u8]>::mut_from_bytes(bytes).unwrap();
+ ///
+ /// assert_eq!(packet.length, 4);
+ /// assert_eq!(packet.body, [1, 2, 3, 4, 5, 6, 7, 8, 9]);
+ ///
+ /// {
+ /// // Attempt to split `packet` at `length`.
+ /// let split = packet.split_at_mut(packet.length as usize).unwrap();
+ ///
+ /// // Use the `Unaligned` bound on `Packet` to prove that it's okay to
+ /// // return concurrent references to `packet` and `rest`.
+ /// let (packet, rest) = split.via_unaligned();
+ ///
+ /// assert_eq!(packet.length, 4);
+ /// assert_eq!(packet.body, [1, 2, 3, 4]);
+ /// assert_eq!(rest, [5, 6, 7, 8, 9]);
+ ///
+ /// rest.fill(0);
+ /// }
+ ///
+ /// assert_eq!(packet.length, 4);
+ /// assert_eq!(packet.body, [1, 2, 3, 4, 0, 0, 0, 0, 0]);
+ /// ```
+ ///
+ /// # Code Generation
+ ///
+ /// See [`Split::via_immutable`](#method.split_via_immutable.codegen).
+ #[must_use = "has no side effects"]
+ #[inline(always)]
+ pub fn via_unaligned(self) -> (&'a mut T, &'a mut [T::Elem])
+ where
+ T: Unaligned,
+ {
+ let (l, r) = self.into_ptr().via_unaligned();
+ (l.as_mut(), r.as_mut())
+ }
+
+ /// Produces the split parts of `self`, using a dynamic check to ensure that
+ /// it is sound to have concurrent references to both parts. You should
+ /// prefer using [`Self::via_into_bytes`] or [`Self::via_unaligned`], which
+ /// have no runtime cost.
+ ///
+ /// # Examples
+ ///
+ /// ```
+ /// use zerocopy::{SplitAt, FromBytes};
+ /// # use zerocopy_derive::*;
+ ///
+ /// #[derive(SplitAt, FromBytes, KnownLayout, IntoBytes, Debug)]
+ /// #[repr(C)]
+ /// struct Packet<B: ?Sized> {
+ /// length: u8,
+ /// body: B,
+ /// }
+ ///
+ /// // These bytes encode a `Packet`.
+ /// let mut bytes = &mut [4, 1, 2, 3, 4, 5, 6, 7, 8, 9][..];
+ ///
+ /// let packet = Packet::<[u8]>::mut_from_bytes(bytes).unwrap();
+ ///
+ /// assert_eq!(packet.length, 4);
+ /// assert_eq!(packet.body, [1, 2, 3, 4, 5, 6, 7, 8, 9]);
+ ///
+ /// {
+ /// // Attempt to split `packet` at `length`.
+ /// let split = packet.split_at_mut(packet.length as usize).unwrap();
+ ///
+ /// // Use a dynamic check to prove that it's okay to return concurrent
+ /// // references to `packet` and `rest`.
+ /// let (packet, rest) = split.via_runtime_check().unwrap();
+ ///
+ /// assert_eq!(packet.length, 4);
+ /// assert_eq!(packet.body, [1, 2, 3, 4]);
+ /// assert_eq!(rest, [5, 6, 7, 8, 9]);
+ ///
+ /// rest.fill(0);
+ /// }
+ ///
+ /// assert_eq!(packet.length, 4);
+ /// assert_eq!(packet.body, [1, 2, 3, 4, 0, 0, 0, 0, 0]);
+ /// ```
+ ///
+ /// # Code Generation
+ ///
+ /// See [`Split::via_runtime_check`](#method.split_via_runtime_check.codegen).
+ #[must_use = "has no side effects"]
+ #[inline(always)]
+ pub fn via_runtime_check(self) -> Result<(&'a mut T, &'a mut [T::Elem]), Self> {
+ match self.into_ptr().via_runtime_check() {
+ Ok((l, r)) => Ok((l.as_mut(), r.as_mut())),
+ Err(s) => Err(s.into_mut()),
+ }
+ }
+
+ /// Unsafely produces the split parts of `self`.
+ ///
+ /// # Safety
+ ///
+ /// The trailing padding bytes of the left portion must not overlap the
+ /// right portion. For some dynamically sized types, the padding that
+ /// appears after the trailing slice field [is a dynamic function of the
+ /// trailing slice length](KnownLayout#slice-dst-layout). Thus, for some
+ /// types, this condition is dependent on the length of the left portion.
+ ///
+ /// # Code Generation
+ ///
+ /// See [`Split::via_unchecked`](#method.split_via_unchecked.codegen).
+ #[must_use = "has no side effects"]
+ #[inline(always)]
+ pub unsafe fn via_unchecked(self) -> (&'a mut T, &'a mut [T::Elem]) {
+ // SAFETY: The aliasing of `self.into_ptr()` is `Exclusive`, and the
+ // caller has promised that the left and right portions of `self` split
+ // at `l_len` do not overlap.
+ let (l, r) = unsafe { self.into_ptr().via_unchecked() };
+ (l.as_mut(), r.as_mut())
+ }
+}
+
+impl<'a, T, I> Split<Ptr<'a, T, I>>
+where
+ T: ?Sized + SplitAt,
+ I: Invariants<Alignment = Aligned, Validity = Valid>,
+{
+ fn into_ref(self) -> Split<&'a T>
+ where
+ I: Invariants<Aliasing = Shared>,
+ {
+ // SAFETY: `self.source.as_ref()` points to exactly the same referent as
+ // `self.source` and thus maintains the invariants of `self` with
+ // respect to `l_len`.
+ unsafe { Split::new(self.source.as_ref(), self.l_len) }
+ }
+
+ fn into_mut(self) -> Split<&'a mut T>
+ where
+ I: Invariants<Aliasing = Exclusive>,
+ {
+ // SAFETY: `self.source.as_mut()` points to exactly the same referent as
+ // `self.source` and thus maintains the invariants of `self` with
+ // respect to `l_len`.
+ unsafe { Split::new(self.source.unify_invariants().as_mut(), self.l_len) }
+ }
+
+ /// Produces the length of `self`'s left part.
+ #[inline(always)]
+ fn l_len(&self) -> MetadataOf<T> {
+ // SAFETY: By invariant on `Split`, `self.l_len` is not greater than the
+ // length of `self.source`.
+ unsafe { MetadataOf::<T>::new_unchecked(self.l_len) }
+ }
+
+ /// Produces the split parts of `self`, using [`Immutable`] to ensure that
+ /// it is sound to have concurrent references to both parts.
+ #[inline(always)]
+ fn via_immutable(self) -> (Ptr<'a, T, I>, Ptr<'a, [T::Elem], I>)
+ where
+ T: Immutable,
+ I: Invariants<Aliasing = Shared>,
+ {
+ // SAFETY: `Aliasing = Shared` and `T: Immutable`.
+ unsafe { self.via_unchecked() }
+ }
+
+ /// Produces the split parts of `self`, using [`IntoBytes`] to ensure that
+ /// it is sound to have concurrent references to both parts.
+ #[inline(always)]
+ fn via_into_bytes(self) -> (Ptr<'a, T, I>, Ptr<'a, [T::Elem], I>)
+ where
+ T: IntoBytes,
+ {
+ // SAFETY: By `T: IntoBytes`, `T` has no padding for any length.
+ // Consequently, `T` can be split into non-overlapping parts at any
+ // index.
+ unsafe { self.via_unchecked() }
+ }
+
+ /// Produces the split parts of `self`, using [`Unaligned`] to ensure that
+ /// it is sound to have concurrent references to both parts.
+ #[inline(always)]
+ fn via_unaligned(self) -> (Ptr<'a, T, I>, Ptr<'a, [T::Elem], I>)
+ where
+ T: Unaligned,
+ {
+ // SAFETY: By `T: SplitAt + Unaligned`, `T` is either a slice or a
+ // `repr(C)` or `repr(transparent)` slice DST that is well-aligned at
+ // any address and length. If `T` is a slice DST with alignment 1,
+ // `repr(C)` or `repr(transparent)` ensures that no padding is placed
+ // after the final element of the trailing slice. Consequently, `T` can
+ // be split into strictly non-overlapping parts any any index.
+ unsafe { self.via_unchecked() }
+ }
+
+ /// Produces the split parts of `self`, using a dynamic check to ensure that
+ /// it is sound to have concurrent references to both parts. You should
+ /// prefer using [`Self::via_immutable`], [`Self::via_into_bytes`], or
+ /// [`Self::via_unaligned`], which have no runtime cost.
+ #[inline(always)]
+ fn via_runtime_check(self) -> Result<(Ptr<'a, T, I>, Ptr<'a, [T::Elem], I>), Self> {
+ let l_len = self.l_len();
+ // FIXME(#1290): Once we require `KnownLayout` on all fields, add an
+ // `IS_IMMUTABLE` associated const, and add `T::IS_IMMUTABLE ||` to the
+ // below check.
+ if l_len.padding_needed_for() == 0 {
+ // SAFETY: By `T: SplitAt`, `T` is either `[T]`, or a `repr(C)` or
+ // `repr(transparent)` slice DST, for which the trailing padding
+ // needed to accommodate `l_len` trailing elements is
+ // `l_len.padding_needed_for()`. If no trailing padding is required,
+ // the left and right parts are strictly non-overlapping.
+ Ok(unsafe { self.via_unchecked() })
+ } else {
+ Err(self)
+ }
+ }
+
+ /// Unsafely produces the split parts of `self`.
+ ///
+ /// # Safety
+ ///
+ /// The caller promises that if `I::Aliasing` is [`Exclusive`] or `T`
+ /// permits interior mutation, then `l_len.padding_needed_for() == 0`.
+ #[inline(always)]
+ unsafe fn via_unchecked(self) -> (Ptr<'a, T, I>, Ptr<'a, [T::Elem], I>) {
+ let l_len = self.l_len();
+ let inner = self.source.as_inner();
+
+ // SAFETY: By invariant on `Self::l_len`, `l_len` is not greater than
+ // the length of `inner`'s trailing slice.
+ let (left, right) = unsafe { inner.split_at_unchecked(l_len) };
+
+ // Lemma 0: `left` and `right` conform to the aliasing invariant
+ // `I::Aliasing`. Proof: If `I::Aliasing` is `Exclusive` or `T` permits
+ // interior mutation, the caller promises that `l_len.padding_needed_for()
+ // == 0`. Consequently, by post-condition on `PtrInner::split_at_unchecked`,
+ // there is no trailing padding after `left`'s final element that would
+ // overlap into `right`. If `I::Aliasing` is shared and `T` forbids interior
+ // mutation, then overlap between their referents is permissible.
+
+ // SAFETY:
+ // 0. `left` conforms to the aliasing invariant of `I::Aliasing`, by Lemma 0.
+ // 1. `left` conforms to the alignment invariant of `I::Alignment, because
+ // the referents of `left` and `Self` have the same address and type
+ // (and, thus, alignment requirement).
+ // 2. `left` conforms to the validity invariant of `I::Validity`, neither
+ // the type nor bytes of `left`'s referent have been changed.
+ let left = unsafe { Ptr::from_inner(left) };
+
+ // SAFETY:
+ // 0. `right` conforms to the aliasing invariant of `I::Aliasing`, by Lemma
+ // 0.
+ // 1. `right` conforms to the alignment invariant of `I::Alignment, because
+ // if `ptr` with `I::Alignment = Aligned`, then by invariant on `T:
+ // SplitAt`, the trailing slice of `ptr` (from which `right` is derived)
+ // will also be well-aligned.
+ // 2. `right` conforms to the validity invariant of `I::Validity`,
+ // because `right: [T::Elem]` is derived from the trailing slice of
+ // `ptr`, which, by contract on `T: SplitAt::Elem`, has type
+ // `[T::Elem]`. The `left` part cannot be used to invalidate `right`,
+ // because the caller promises that if `I::Aliasing` is `Exclusive`
+ // or `T` permits interior mutation, then `l_len.padding_needed_for()
+ // == 0` and thus the parts will be non-overlapping.
+ let right = unsafe { Ptr::from_inner(right) };
+
+ (left, right)
+ }
+}
+
+#[cfg(test)]
+mod tests {
+ #[cfg(feature = "derive")]
+ #[test]
+ fn test_split_at() {
+ use crate::{FromBytes, Immutable, IntoBytes, KnownLayout, SplitAt};
+
+ #[derive(FromBytes, KnownLayout, SplitAt, IntoBytes, Immutable, Debug)]
+ #[repr(C)]
+ struct SliceDst<const OFFSET: usize> {
+ prefix: [u8; OFFSET],
+ trailing: [u8],
+ }
+
+ #[allow(clippy::as_conversions)]
+ fn test_split_at<const OFFSET: usize, const BUFFER_SIZE: usize>() {
+ // Test `split_at`
+ let n: usize = BUFFER_SIZE - OFFSET;
+ let arr = [1; BUFFER_SIZE];
+ let dst = SliceDst::<OFFSET>::ref_from_bytes(&arr[..]).unwrap();
+ for i in 0..=n {
+ let (l, r) = dst.split_at(i).unwrap().via_runtime_check().unwrap();
+ let l_sum: u8 = l.trailing.iter().sum();
+ let r_sum: u8 = r.iter().sum();
+ assert_eq!(l_sum, i as u8);
+ assert_eq!(r_sum, (n - i) as u8);
+ assert_eq!(l_sum + r_sum, n as u8);
+ }
+
+ // Test `split_at_mut`
+ let n: usize = BUFFER_SIZE - OFFSET;
+ let mut arr = [1; BUFFER_SIZE];
+ let dst = SliceDst::<OFFSET>::mut_from_bytes(&mut arr[..]).unwrap();
+ for i in 0..=n {
+ let (l, r) = dst.split_at_mut(i).unwrap().via_runtime_check().unwrap();
+ let l_sum: u8 = l.trailing.iter().sum();
+ let r_sum: u8 = r.iter().sum();
+ assert_eq!(l_sum, i as u8);
+ assert_eq!(r_sum, (n - i) as u8);
+ assert_eq!(l_sum + r_sum, n as u8);
+ }
+ }
+
+ test_split_at::<0, 16>();
+ test_split_at::<1, 17>();
+ test_split_at::<2, 18>();
+ }
+
+ #[cfg(feature = "derive")]
+ #[test]
+ #[allow(clippy::as_conversions)]
+ fn test_split_at_overlapping() {
+ use crate::{FromBytes, Immutable, IntoBytes, KnownLayout, SplitAt};
+
+ #[derive(FromBytes, KnownLayout, SplitAt, Immutable)]
+ #[repr(C, align(2))]
+ struct SliceDst {
+ prefix: u8,
+ trailing: [u8],
+ }
+
+ const N: usize = 16;
+
+ let arr = [1u16; N];
+ let dst = SliceDst::ref_from_bytes(arr.as_bytes()).unwrap();
+
+ for i in 0..N {
+ let split = dst.split_at(i).unwrap().via_runtime_check();
+ if i % 2 == 1 {
+ assert!(split.is_ok());
+ } else {
+ assert!(split.is_err());
+ }
+ }
+ }
+ #[test]
+ fn test_split_at_unchecked() {
+ use crate::SplitAt;
+ let mut arr = [1, 2, 3, 4];
+ let slice = &arr[..];
+ // SAFETY: 2 <= arr.len() (4)
+ let split = unsafe { SplitAt::split_at_unchecked(slice, 2) };
+ // SAFETY: SplitAt::split_at_unchecked guarantees that the split is valid.
+ let (l, r) = unsafe { split.via_unchecked() };
+ assert_eq!(l, &[1, 2]);
+ assert_eq!(r, &[3, 4]);
+
+ let slice_mut = &mut arr[..];
+ // SAFETY: 2 <= arr.len() (4)
+ let split = unsafe { SplitAt::split_at_mut_unchecked(slice_mut, 2) };
+ // SAFETY: SplitAt::split_at_mut_unchecked guarantees that the split is valid.
+ let (l, r) = unsafe { split.via_unchecked() };
+ assert_eq!(l, &mut [1, 2]);
+ assert_eq!(r, &mut [3, 4]);
+ }
+
+ #[test]
+ fn test_split_at_via_methods() {
+ use crate::{FromBytes, Immutable, IntoBytes, KnownLayout, SplitAt};
+ #[derive(FromBytes, KnownLayout, SplitAt, IntoBytes, Immutable, Debug)]
+ #[repr(C)]
+ struct Packet {
+ length: u8,
+ body: [u8],
+ }
+
+ let arr = [1, 2, 3, 4];
+ let packet = Packet::ref_from_bytes(&arr[..]).unwrap();
+
+ let split1 = packet.split_at(2).unwrap();
+ let (l, r) = split1.via_immutable();
+ assert_eq!(l.length, 1);
+ assert_eq!(r, &[4]);
+
+ let split2 = packet.split_at(2).unwrap();
+ let (l, r) = split2.via_into_bytes();
+ assert_eq!(l.length, 1);
+ assert_eq!(r, &[4]);
+ }
+ #[test]
+ fn test_split_at_via_unaligned() {
+ use crate::{FromBytes, Immutable, IntoBytes, KnownLayout, SplitAt, Unaligned};
+ #[derive(FromBytes, KnownLayout, SplitAt, IntoBytes, Immutable, Unaligned)]
+ #[repr(C)]
+ struct Packet {
+ length: u8,
+ body: [u8],
+ }
+
+ let arr = [1, 2, 3, 4];
+ let packet = Packet::ref_from_bytes(&arr[..]).unwrap();
+
+ let split = packet.split_at(2).unwrap();
+ let (l, r) = split.via_unaligned();
+ assert_eq!(l.length, 1);
+ assert_eq!(r, &[4]);
+ }
+}
diff --git a/rust/zerocopy/src/util/macro_util.rs b/rust/zerocopy/src/util/macro_util.rs
new file mode 100644
index 0000000..1abb0fb
--- /dev/null
+++ b/rust/zerocopy/src/util/macro_util.rs
@@ -0,0 +1,1310 @@
+// SPDX-License-Identifier: (BSD-2-Clause OR Apache-2.0) OR MIT
+
+// Copyright 2022 The Fuchsia Authors
+//
+// Licensed under a BSD-style license <LICENSE-BSD>, Apache License, Version 2.0
+// <LICENSE-APACHE or https://www.apache.org/licenses/LICENSE-2.0>, or the MIT
+// license <LICENSE-MIT or https://opensource.org/licenses/MIT>, at your option.
+// This file may not be copied, modified, or distributed except according to
+// those terms.
+
+//! Utilities used by macros and by `zerocopy-derive`.
+//!
+//! These are defined here `zerocopy` rather than in code generated by macros or
+//! by `zerocopy-derive` so that they can be compiled once rather than
+//! recompiled for every invocation (e.g., if they were defined in generated
+//! code, then deriving `IntoBytes` and `FromBytes` on three different types
+//! would result in the code in question being emitted and compiled six
+//! different times).
+
+#![allow(missing_debug_implementations)]
+
+// FIXME(#29), FIXME(https://github.com/rust-lang/rust/issues/69835): Remove
+// this `cfg` when `size_of_val_raw` is stabilized.
+#[cfg(__ZEROCOPY_INTERNAL_USE_ONLY_NIGHTLY_FEATURES_IN_TESTS)]
+#[cfg(not(target_pointer_width = "16"))]
+use core::ptr::{self, NonNull};
+use core::{marker::PhantomData, mem, num::Wrapping};
+
+use crate::{
+ pointer::{
+ cast::CastSized,
+ invariant::{Aligned, Initialized, Valid},
+ BecauseImmutable,
+ },
+ FromBytes, Immutable, IntoBytes, KnownLayout, Ptr, ReadOnly, TryFromBytes, ValidityError,
+};
+
+/// Projects the type of the field at `Index` in `Self` without regard for field
+/// privacy.
+///
+/// The `Index` parameter is any sort of handle that identifies the field; its
+/// definition is the obligation of the implementer.
+///
+/// # Safety
+///
+/// Unsafe code may assume that this accurately reflects the definition of
+/// `Self`.
+pub unsafe trait Field<Index> {
+ /// The type of the field at `Index`.
+ type Type: ?Sized;
+}
+
+#[cfg_attr(
+ not(no_zerocopy_diagnostic_on_unimplemented_1_78_0),
+ diagnostic::on_unimplemented(
+ message = "`{T}` has {PADDING_BYTES} total byte(s) of padding",
+ label = "types with padding cannot implement `IntoBytes`",
+ note = "consider using `zerocopy::Unalign` to lower the alignment of individual fields",
+ note = "consider adding explicit fields where padding would be",
+ note = "consider using `#[repr(packed)]` to remove padding"
+ )
+)]
+pub trait PaddingFree<T: ?Sized, const PADDING_BYTES: usize> {}
+impl<T: ?Sized> PaddingFree<T, 0> for () {}
+
+// FIXME(#1112): In the slice DST case, we should delegate to *both*
+// `PaddingFree` *and* `DynamicPaddingFree` (and probably rename `PaddingFree`
+// to `StaticPaddingFree` or something - or introduce a third trait with that
+// name) so that we can have more clear error messages.
+
+#[cfg_attr(
+ not(no_zerocopy_diagnostic_on_unimplemented_1_78_0),
+ diagnostic::on_unimplemented(
+ message = "`{T}` has one or more padding bytes",
+ label = "types with padding cannot implement `IntoBytes`",
+ note = "consider using `zerocopy::Unalign` to lower the alignment of individual fields",
+ note = "consider adding explicit fields where padding would be",
+ note = "consider using `#[repr(packed)]` to remove padding"
+ )
+)]
+pub trait DynamicPaddingFree<T: ?Sized, const HAS_PADDING: bool> {}
+impl<T: ?Sized> DynamicPaddingFree<T, false> for () {}
+
+#[cfg(__ZEROCOPY_INTERNAL_USE_ONLY_NIGHTLY_FEATURES_IN_TESTS)]
+#[cfg(not(target_pointer_width = "16"))]
+const _64K: usize = 1 << 16;
+
+// FIXME(#29), FIXME(https://github.com/rust-lang/rust/issues/69835): Remove
+// this `cfg` when `size_of_val_raw` is stabilized.
+#[cfg(__ZEROCOPY_INTERNAL_USE_ONLY_NIGHTLY_FEATURES_IN_TESTS)]
+#[cfg(not(target_pointer_width = "16"))]
+#[repr(C, align(65536))]
+struct Aligned64kAllocation([u8; _64K]);
+
+/// A pointer to an aligned allocation of size 2^16.
+///
+/// # Safety
+///
+/// `ALIGNED_64K_ALLOCATION` is guaranteed to point to the entirety of an
+/// allocation with size and alignment 2^16, and to have valid provenance.
+// FIXME(#29), FIXME(https://github.com/rust-lang/rust/issues/69835): Remove
+// this `cfg` when `size_of_val_raw` is stabilized.
+#[cfg(__ZEROCOPY_INTERNAL_USE_ONLY_NIGHTLY_FEATURES_IN_TESTS)]
+#[cfg(not(target_pointer_width = "16"))]
+pub const ALIGNED_64K_ALLOCATION: NonNull<[u8]> = {
+ const REF: &Aligned64kAllocation = &Aligned64kAllocation([0; _64K]);
+ let ptr: *const Aligned64kAllocation = REF;
+ let ptr: *const [u8] = ptr::slice_from_raw_parts(ptr.cast(), _64K);
+ // SAFETY:
+ // - `ptr` is derived from a Rust reference, which is guaranteed to be
+ // non-null.
+ // - `ptr` is derived from an `&Aligned64kAllocation`, which has size and
+ // alignment `_64K` as promised. Its length is initialized to `_64K`,
+ // which means that it refers to the entire allocation.
+ // - `ptr` is derived from a Rust reference, which is guaranteed to have
+ // valid provenance.
+ //
+ // FIXME(#429): Once `NonNull::new_unchecked` docs document that it
+ // preserves provenance, cite those docs.
+ // FIXME: Replace this `as` with `ptr.cast_mut()` once our MSRV >= 1.65
+ #[allow(clippy::as_conversions)]
+ unsafe {
+ NonNull::new_unchecked(ptr as *mut _)
+ }
+};
+
+/// Computes the offset of the base of the field `$trailing_field_name` within
+/// the type `$ty`.
+///
+/// `trailing_field_offset!` produces code which is valid in a `const` context.
+// FIXME(#29), FIXME(https://github.com/rust-lang/rust/issues/69835): Remove
+// this `cfg` when `size_of_val_raw` is stabilized.
+#[cfg(__ZEROCOPY_INTERNAL_USE_ONLY_NIGHTLY_FEATURES_IN_TESTS)]
+#[doc(hidden)] // `#[macro_export]` bypasses this module's `#[doc(hidden)]`.
+#[macro_export]
+macro_rules! trailing_field_offset {
+ ($ty:ty, $trailing_field_name:tt) => {{
+ let min_size = {
+ let zero_elems: *const [()] =
+ $crate::util::macro_util::core_reexport::ptr::slice_from_raw_parts(
+ $crate::util::macro_util::core_reexport::ptr::NonNull::<()>::dangling()
+ .as_ptr()
+ .cast_const(),
+ 0,
+ );
+ // SAFETY:
+ // - If `$ty` is `Sized`, `size_of_val_raw` is always safe to call.
+ // - Otherwise:
+ // - If `$ty` is not a slice DST, this pointer conversion will
+ // fail due to "mismatched vtable kinds", and compilation will
+ // fail.
+ // - If `$ty` is a slice DST, we have constructed `zero_elems` to
+ // have zero trailing slice elements. Per the `size_of_val_raw`
+ // docs, "For the special case where the dynamic tail length is
+ // 0, this function is safe to call." [1]
+ //
+ // [1] https://doc.rust-lang.org/nightly/std/mem/fn.size_of_val_raw.html
+ unsafe {
+ #[allow(clippy::as_conversions)]
+ $crate::util::macro_util::core_reexport::mem::size_of_val_raw(
+ zero_elems as *const $ty,
+ )
+ }
+ };
+
+ assert!(min_size <= _64K);
+
+ #[allow(clippy::as_conversions)]
+ let ptr = ALIGNED_64K_ALLOCATION.as_ptr() as *const $ty;
+
+ // SAFETY:
+ // - Thanks to the preceding `assert!`, we know that the value with zero
+ // elements fits in `_64K` bytes, and thus in the allocation addressed
+ // by `ALIGNED_64K_ALLOCATION`. The offset of the trailing field is
+ // guaranteed to be no larger than this size, so this field projection
+ // is guaranteed to remain in-bounds of its allocation.
+ // - Because the minimum size is no larger than `_64K` bytes, and
+ // because an object's size must always be a multiple of its alignment
+ // [1], we know that `$ty`'s alignment is no larger than `_64K`. The
+ // allocation addressed by `ALIGNED_64K_ALLOCATION` is guaranteed to
+ // be aligned to `_64K`, so `ptr` is guaranteed to satisfy `$ty`'s
+ // alignment.
+ // - As required by `addr_of!`, we do not write through `field`.
+ //
+ // Note that, as of [2], this requirement is technically unnecessary
+ // for Rust versions >= 1.75.0, but no harm in guaranteeing it anyway
+ // until we bump our MSRV.
+ //
+ // [1] Per https://doc.rust-lang.org/reference/type-layout.html:
+ //
+ // The size of a value is always a multiple of its alignment.
+ //
+ // [2] https://github.com/rust-lang/reference/pull/1387
+ let field = unsafe {
+ $crate::util::macro_util::core_reexport::ptr::addr_of!((*ptr).$trailing_field_name)
+ };
+ // SAFETY:
+ // - Both `ptr` and `field` are derived from the same allocated object.
+ // - By the preceding safety comment, `field` is in bounds of that
+ // allocated object.
+ // - The distance, in bytes, between `ptr` and `field` is required to be
+ // a multiple of the size of `u8`, which is trivially true because
+ // `u8`'s size is 1.
+ // - The distance, in bytes, cannot overflow `isize`. This is guaranteed
+ // because no allocated object can have a size larger than can fit in
+ // `isize`. [1]
+ // - The distance being in-bounds cannot rely on wrapping around the
+ // address space. This is guaranteed because the same is guaranteed of
+ // allocated objects. [1]
+ //
+ // [1] FIXME(#429), FIXME(https://github.com/rust-lang/rust/pull/116675):
+ // Once these are guaranteed in the Reference, cite it.
+ let offset = unsafe { field.cast::<u8>().offset_from(ptr.cast::<u8>()) };
+ // Guaranteed not to be lossy: `field` comes after `ptr`, so the offset
+ // from `ptr` to `field` is guaranteed to be positive.
+ assert!(offset >= 0);
+ Some(
+ #[allow(clippy::as_conversions)]
+ {
+ offset as usize
+ },
+ )
+ }};
+}
+
+/// Computes alignment of `$ty: ?Sized`.
+///
+/// `align_of!` produces code which is valid in a `const` context.
+// FIXME(#29), FIXME(https://github.com/rust-lang/rust/issues/69835): Remove
+// this `cfg` when `size_of_val_raw` is stabilized.
+#[cfg(__ZEROCOPY_INTERNAL_USE_ONLY_NIGHTLY_FEATURES_IN_TESTS)]
+#[doc(hidden)] // `#[macro_export]` bypasses this module's `#[doc(hidden)]`.
+#[macro_export]
+macro_rules! align_of {
+ ($ty:ty) => {{
+ // SAFETY: `OffsetOfTrailingIsAlignment` is `repr(C)`, and its layout is
+ // guaranteed [1] to begin with the single-byte layout for `_byte`,
+ // followed by the padding needed to align `_trailing`, then the layout
+ // for `_trailing`, and finally any trailing padding bytes needed to
+ // correctly-align the entire struct.
+ //
+ // This macro computes the alignment of `$ty` by counting the number of
+ // bytes preceding `_trailing`. For instance, if the alignment of `$ty`
+ // is `1`, then no padding is required align `_trailing` and it will be
+ // located immediately after `_byte` at offset 1. If the alignment of
+ // `$ty` is 2, then a single padding byte is required before
+ // `_trailing`, and `_trailing` will be located at offset 2.
+
+ // This correspondence between offset and alignment holds for all valid
+ // Rust alignments, and we confirm this exhaustively (or, at least up to
+ // the maximum alignment supported by `trailing_field_offset!`) in
+ // `test_align_of_dst`.
+ //
+ // [1]: https://doc.rust-lang.org/nomicon/other-reprs.html#reprc
+
+ #[repr(C)]
+ struct OffsetOfTrailingIsAlignment {
+ _byte: u8,
+ _trailing: $ty,
+ }
+
+ trailing_field_offset!(OffsetOfTrailingIsAlignment, _trailing)
+ }};
+}
+
+mod size_to_tag {
+ pub trait SizeToTag<const SIZE: usize> {
+ type Tag;
+ }
+
+ impl SizeToTag<1> for () {
+ type Tag = u8;
+ }
+ impl SizeToTag<2> for () {
+ type Tag = u16;
+ }
+ impl SizeToTag<4> for () {
+ type Tag = u32;
+ }
+ impl SizeToTag<8> for () {
+ type Tag = u64;
+ }
+ impl SizeToTag<16> for () {
+ type Tag = u128;
+ }
+}
+
+/// An alias for the unsigned integer of the given size in bytes.
+#[doc(hidden)]
+pub type SizeToTag<const SIZE: usize> = <() as size_to_tag::SizeToTag<SIZE>>::Tag;
+
+// We put `Sized` in its own module so it can have the same name as the standard
+// library `Sized` without shadowing it in the parent module.
+#[cfg(not(no_zerocopy_diagnostic_on_unimplemented_1_78_0))]
+mod __size_of {
+ #[diagnostic::on_unimplemented(
+ message = "`{Self}` is unsized",
+ label = "`IntoBytes` needs all field types to be `Sized` in order to determine whether there is padding",
+ note = "consider using `#[repr(packed)]` to remove padding",
+ note = "`IntoBytes` does not require the fields of `#[repr(packed)]` types to be `Sized`"
+ )]
+ pub trait Sized: core::marker::Sized {}
+ impl<T: core::marker::Sized> Sized for T {}
+
+ #[inline(always)]
+ #[must_use]
+ #[allow(clippy::needless_maybe_sized)]
+ pub const fn size_of<T: Sized + ?core::marker::Sized>() -> usize {
+ core::mem::size_of::<T>()
+ }
+}
+
+#[cfg(no_zerocopy_diagnostic_on_unimplemented_1_78_0)]
+pub use core::mem::size_of;
+
+#[cfg(not(no_zerocopy_diagnostic_on_unimplemented_1_78_0))]
+pub use __size_of::size_of;
+
+/// How many padding bytes does the struct type `$t` have?
+///
+/// `$ts` is the list of the type of every field in `$t`. `$t` must be a struct
+/// type, or else `struct_padding!`'s result may be meaningless.
+///
+/// Note that `struct_padding!`'s results are independent of `repcr` since they
+/// only consider the size of the type and the sizes of the fields. Whatever the
+/// repr, the size of the type already takes into account any padding that the
+/// compiler has decided to add. Structs with well-defined representations (such
+/// as `repr(C)`) can use this macro to check for padding. Note that while this
+/// may yield some consistent value for some `repr(Rust)` structs, it is not
+/// guaranteed across platforms or compilations.
+#[doc(hidden)] // `#[macro_export]` bypasses this module's `#[doc(hidden)]`.
+#[macro_export]
+macro_rules! struct_padding {
+ ($t:ty, $_align:expr, $_packed:expr, [$($ts:ty),*]) => {{
+ // The `align` and `packed` directives can be ignored here. Regardless
+ // of if and how they are set, comparing the size of `$t` to the sum of
+ // its field sizes is a reliable indicator of the presence of padding.
+ $crate::util::macro_util::size_of::<$t>() - (0 $(+ $crate::util::macro_util::size_of::<$ts>())*)
+ }};
+}
+
+/// Does the `repr(C)` struct type `$t` have padding?
+///
+/// `$ts` is the list of the type of every field in `$t`. `$t` must be a
+/// `repr(C)` struct type, or else `struct_has_padding!`'s result may be
+/// meaningless.
+#[doc(hidden)] // `#[macro_export]` bypasses this module's `#[doc(hidden)]`.
+#[macro_export]
+macro_rules! repr_c_struct_has_padding {
+ ($t:ty, $align:expr, $packed:expr, [$($ts:tt),*]) => {{
+ let layout = $crate::DstLayout::for_repr_c_struct(
+ $align,
+ $packed,
+ &[$($crate::repr_c_struct_has_padding!(@field $ts),)*]
+ );
+ layout.requires_static_padding() || layout.requires_dynamic_padding()
+ }};
+ (@field ([$t:ty])) => {
+ <[$t] as $crate::KnownLayout>::LAYOUT
+ };
+ (@field ($t:ty)) => {
+ $crate::DstLayout::for_unpadded_type::<$t>()
+ };
+ (@field [$t:ty]) => {
+ <[$t] as $crate::KnownLayout>::LAYOUT
+ };
+ (@field $t:ty) => {
+ $crate::DstLayout::for_unpadded_type::<$t>()
+ };
+}
+
+/// Does the union type `$t` have padding?
+///
+/// `$ts` is the list of the type of every field in `$t`. `$t` must be a union
+/// type, or else `union_padding!`'s result may be meaningless.
+///
+/// Note that `union_padding!`'s results are independent of `repr` since they
+/// only consider the size of the type and the sizes of the fields. Whatever the
+/// repr, the size of the type already takes into account any padding that the
+/// compiler has decided to add. Unions with well-defined representations (such
+/// as `repr(C)`) can use this macro to check for padding. Note that while this
+/// may yield some consistent value for some `repr(Rust)` unions, it is not
+/// guaranteed across platforms or compilations.
+#[doc(hidden)] // `#[macro_export]` bypasses this module's `#[doc(hidden)]`.
+#[macro_export]
+macro_rules! union_padding {
+ ($t:ty, $_align:expr, $_packed:expr, [$($ts:ty),*]) => {{
+ // The `align` and `packed` directives can be ignored here. Regardless
+ // of if and how they are set, comparing the size of `$t` to each of its
+ // field sizes is a reliable indicator of the presence of padding.
+ let mut max = 0;
+ $({
+ let padding = $crate::util::macro_util::size_of::<$t>() - $crate::util::macro_util::size_of::<$ts>();
+ if padding > max {
+ max = padding;
+ }
+ })*
+ max
+ }};
+}
+
+/// How many padding bytes does the enum type `$t` have?
+///
+/// `$disc` is the type of the enum tag, and `$ts` is a list of fields in each
+/// square-bracket-delimited variant. `$t` must be an enum, or else
+/// `enum_padding!`'s result may be meaningless. An enum has padding if any of
+/// its variant structs [1][2] contain padding, and so all of the variants of an
+/// enum must be "full" in order for the enum to not have padding.
+///
+/// The results of `enum_padding!` require that the enum is not `repr(Rust)`, as
+/// `repr(Rust)` enums may niche the enum's tag and reduce the total number of
+/// bytes required to represent the enum as a result. As long as the enum is
+/// `repr(C)`, `repr(int)`, or `repr(C, int)`, this will consistently return
+/// whether the enum contains any padding bytes.
+///
+/// [1]: https://doc.rust-lang.org/1.81.0/reference/type-layout.html#reprc-enums-with-fields
+/// [2]: https://doc.rust-lang.org/1.81.0/reference/type-layout.html#primitive-representation-of-enums-with-fields
+#[doc(hidden)] // `#[macro_export]` bypasses this module's `#[doc(hidden)]`.
+#[macro_export]
+macro_rules! enum_padding {
+ ($t:ty, $_align:expr, $packed:expr, $disc:ty, $([$($ts:ty),*]),*) => {{
+ // The `align` and `packed` directives are irrelevant. `$align` can be
+ // ignored because regardless of if and how it is set, comparing the
+ // size of `$t` to each of its field sizes is a reliable indicator of
+ // the presence of padding. `$packed` is irrelevant because it is
+ // forbidden on enums.
+ #[allow(clippy::as_conversions)]
+ const _: [(); 1] = [(); $packed.is_none() as usize];
+ let mut max = 0;
+ $({
+ let padding = $crate::util::macro_util::size_of::<$t>()
+ - (
+ $crate::util::macro_util::size_of::<$disc>()
+ $(+ $crate::util::macro_util::size_of::<$ts>())*
+ );
+ if padding > max {
+ max = padding;
+ }
+ })*
+ max
+ }};
+}
+
+/// Unwraps an infallible `Result`.
+#[doc(hidden)]
+#[macro_export]
+macro_rules! into_inner {
+ ($e:expr) => {
+ match $e {
+ $crate::util::macro_util::core_reexport::result::Result::Ok(e) => e,
+ $crate::util::macro_util::core_reexport::result::Result::Err(i) => match i {},
+ }
+ };
+}
+
+/// Translates an identifier or tuple index into a numeric identifier.
+#[doc(hidden)] // `#[macro_export]` bypasses this module's `#[doc(hidden)]`.
+#[macro_export]
+macro_rules! ident_id {
+ ($field:ident) => {
+ $crate::util::macro_util::hash_name(stringify!($field))
+ };
+ ($field:literal) => {
+ $field
+ };
+}
+
+/// Computes the hash of a string.
+///
+/// NOTE(#2749) on hash collisions: This function's output only needs to be
+/// deterministic within a particular compilation. Thus, if a user ever reports
+/// a hash collision (very unlikely given the <= 16-byte special case), we can
+/// strengthen the hash function at that point and publish a new version. Since
+/// this is computed at compile time on small strings, we can easily use more
+/// expensive and higher-quality hash functions if need be.
+#[inline(always)]
+#[must_use]
+#[allow(clippy::as_conversions, clippy::indexing_slicing, clippy::arithmetic_side_effects)]
+pub const fn hash_name(name: &str) -> i128 {
+ let name = name.as_bytes();
+
+ // We guarantee freedom from hash collisions between any two strings of
+ // length 16 or less by having the hashes of such strings be equal to
+ // their value. There is still a possibility that such strings will have
+ // the same value as the hash of a string of length > 16.
+ if name.len() <= size_of::<u128>() {
+ let mut bytes = [0u8; 16];
+
+ let mut i = 0;
+ while i < name.len() {
+ bytes[i] = name[i];
+ i += 1;
+ }
+
+ return i128::from_ne_bytes(bytes);
+ };
+
+ // An implementation of FxHasher, although returning a u128. Probably
+ // not as strong as it could be, but probably more collision resistant
+ // than normal 64-bit FxHasher.
+ let mut hash = 0u128;
+ let mut i = 0;
+ while i < name.len() {
+ // This is just FxHasher's `0x517cc1b727220a95` constant
+ // concatenated back-to-back.
+ const K: u128 = 0x517cc1b727220a95517cc1b727220a95;
+ hash = (hash.rotate_left(5) ^ (name[i] as u128)).wrapping_mul(K);
+ i += 1;
+ }
+ i128::from_ne_bytes(hash.to_ne_bytes())
+}
+
+/// Attempts to transmute `Src` into `Dst`.
+///
+/// A helper for `try_transmute!`.
+///
+/// # Panics
+///
+/// `try_transmute` may either produce a post-monomorphization error or a panic
+/// if `Dst` is bigger than `Src`. Otherwise, `try_transmute` panics under the
+/// same circumstances as [`is_bit_valid`].
+///
+/// [`is_bit_valid`]: TryFromBytes::is_bit_valid
+#[inline(always)]
+pub fn try_transmute<Src, Dst>(src: Src) -> Result<Dst, ValidityError<Src, Dst>>
+where
+ Src: IntoBytes,
+ Dst: TryFromBytes,
+{
+ static_assert!(Src, Dst => mem::size_of::<Dst>() == mem::size_of::<Src>());
+
+ let mu_src = mem::MaybeUninit::new(src);
+ // SAFETY: `MaybeUninit` has no validity requirements.
+ let mu_dst: mem::MaybeUninit<ReadOnly<Dst>> =
+ unsafe { crate::util::transmute_unchecked(mu_src) };
+
+ let ptr = Ptr::from_ref(&mu_dst);
+
+ // SAFETY: Since `Src: IntoBytes`, and since `size_of::<Src>() ==
+ // size_of::<Dst>()` by the preceding assertion, all of `mu_dst`'s bytes are
+ // initialized. `MaybeUninit` has no validity requirements, so even if
+ // `ptr` is used to mutate its referent (which it actually can't be - it's
+ // a shared `ReadOnly` pointer), that won't violate its referent's validity.
+ let ptr = unsafe { ptr.assume_validity::<Initialized>() };
+ if Dst::is_bit_valid(ptr.cast::<_, CastSized, _>()) {
+ // SAFETY: Since `Dst::is_bit_valid`, we know that `ptr`'s referent is
+ // bit-valid for `Dst`. `ptr` points to `mu_dst`, and no intervening
+ // operations have mutated it, so it is a bit-valid `Dst`.
+ Ok(ReadOnly::into_inner(unsafe { mu_dst.assume_init() }))
+ } else {
+ // SAFETY: `MaybeUninit` has no validity requirements.
+ let mu_src: mem::MaybeUninit<Src> = unsafe { crate::util::transmute_unchecked(mu_dst) };
+ // SAFETY: `mu_dst`/`mu_src` was constructed from `src` and never
+ // modified, so it is still bit-valid.
+ Err(ValidityError::new(unsafe { mu_src.assume_init() }))
+ }
+}
+
+/// See `try_transmute_ref!` documentation.
+pub trait TryTransmuteRefDst<'a> {
+ type Dst: ?Sized;
+
+ /// See `try_transmute_ref!` documentation.
+ fn try_transmute_ref(self) -> Result<&'a Self::Dst, ValidityError<&'a Self::Src, Self::Dst>>
+ where
+ Self: TryTransmuteRefSrc<'a>,
+ Self::Src: IntoBytes + Immutable + KnownLayout,
+ Self::Dst: TryFromBytes + Immutable + KnownLayout;
+}
+
+pub trait TryTransmuteRefSrc<'a> {
+ type Src: ?Sized;
+}
+
+impl<'a, Src, Dst> TryTransmuteRefSrc<'a> for Wrap<&'a Src, &'a Dst>
+where
+ Src: ?Sized,
+ Dst: ?Sized,
+{
+ type Src = Src;
+}
+
+impl<'a, Src, Dst> TryTransmuteRefDst<'a> for Wrap<&'a Src, &'a Dst>
+where
+ Src: IntoBytes + Immutable + KnownLayout + ?Sized,
+ Dst: TryFromBytes + Immutable + KnownLayout + ?Sized,
+{
+ type Dst = Dst;
+
+ #[inline(always)]
+ fn try_transmute_ref(
+ self,
+ ) -> Result<
+ &'a Dst,
+ ValidityError<&'a <Wrap<&'a Src, &'a Dst> as TryTransmuteRefSrc<'a>>::Src, Dst>,
+ > {
+ let ptr = Ptr::from_ref(self.0);
+ #[rustfmt::skip]
+ let res = ptr.try_with(#[inline(always)] |ptr| {
+ let ptr = ptr.recall_validity::<Initialized, _>();
+ let ptr = ptr.cast::<_, crate::layout::CastFrom<Dst>, _>();
+ ptr.try_into_valid()
+ });
+ match res {
+ Ok(ptr) => {
+ static_assert!(Src: ?Sized + KnownLayout, Dst: ?Sized + KnownLayout => {
+ Src::LAYOUT.align.get() >= Dst::LAYOUT.align.get()
+ }, "cannot transmute reference when destination type has higher alignment than source type");
+ // SAFETY: We have checked that `Dst` does not have a stricter
+ // alignment requirement than `Src`.
+ let ptr = unsafe { ptr.assume_alignment::<Aligned>() };
+ Ok(ptr.as_ref())
+ }
+ Err(err) => Err(err.map_src(Ptr::as_ref)),
+ }
+ }
+}
+
+pub trait TryTransmuteMutDst<'a> {
+ type Dst: ?Sized;
+
+ /// See `try_transmute_mut!` documentation.
+ fn try_transmute_mut(
+ self,
+ ) -> Result<&'a mut Self::Dst, ValidityError<&'a mut Self::Src, Self::Dst>>
+ where
+ Self: TryTransmuteMutSrc<'a>,
+ Self::Src: IntoBytes,
+ Self::Dst: TryFromBytes;
+}
+
+pub trait TryTransmuteMutSrc<'a> {
+ type Src: ?Sized;
+}
+
+impl<'a, Src, Dst> TryTransmuteMutSrc<'a> for Wrap<&'a mut Src, &'a mut Dst>
+where
+ Src: ?Sized,
+ Dst: ?Sized,
+{
+ type Src = Src;
+}
+
+impl<'a, Src, Dst> TryTransmuteMutDst<'a> for Wrap<&'a mut Src, &'a mut Dst>
+where
+ Src: FromBytes + IntoBytes + KnownLayout + ?Sized,
+ Dst: TryFromBytes + IntoBytes + KnownLayout + ?Sized,
+{
+ type Dst = Dst;
+
+ #[inline(always)]
+ fn try_transmute_mut(
+ self,
+ ) -> Result<
+ &'a mut Dst,
+ ValidityError<&'a mut <Wrap<&'a mut Src, &'a mut Dst> as TryTransmuteMutSrc<'a>>::Src, Dst>,
+ > {
+ let ptr = Ptr::from_mut(self.0);
+ // SAFETY: The provided closure returns the only copy of `ptr`.
+ #[rustfmt::skip]
+ let res = unsafe {
+ ptr.try_with_unchecked(#[inline(always)] |ptr| {
+ let ptr = ptr.recall_validity::<Initialized, (_, (_, _))>();
+ let ptr = ptr.cast::<_, crate::layout::CastFrom<Dst>, _>();
+ ptr.try_into_valid()
+ })
+ };
+ match res {
+ Ok(ptr) => {
+ static_assert!(Src: ?Sized + KnownLayout, Dst: ?Sized + KnownLayout => {
+ Src::LAYOUT.align.get() >= Dst::LAYOUT.align.get()
+ }, "cannot transmute reference when destination type has higher alignment than source type");
+ // SAFETY: We have checked that `Dst` does not have a stricter
+ // alignment requirement than `Src`.
+ let ptr = unsafe { ptr.assume_alignment::<Aligned>() };
+ Ok(ptr.as_mut())
+ }
+ Err(err) => Err(err.map_src(Ptr::as_mut)),
+ }
+ }
+}
+
+// Used in `transmute_ref!` and friends.
+//
+// This permits us to use the autoref specialization trick to dispatch to
+// associated functions for `transmute_ref` and `transmute_mut` when both `Src`
+// and `Dst` are `Sized`, and to trait methods otherwise. The associated
+// functions, unlike the trait methods, do not require a `KnownLayout` bound.
+// This permits us to add support for transmuting references to unsized types
+// without breaking backwards-compatibility (on v0.8.x) with the old
+// implementation, which did not require a `KnownLayout` bound to transmute
+// sized types.
+#[derive(Copy, Clone)]
+pub struct Wrap<Src, Dst>(pub Src, pub PhantomData<Dst>);
+
+impl<Src, Dst> Wrap<Src, Dst> {
+ #[inline(always)]
+ pub const fn new(src: Src) -> Self {
+ Wrap(src, PhantomData)
+ }
+}
+
+impl<'a, Src, Dst> Wrap<&'a Src, &'a Dst>
+where
+ Src: ?Sized,
+ Dst: ?Sized,
+{
+ #[allow(clippy::must_use_candidate, clippy::missing_inline_in_public_items, clippy::empty_loop)]
+ pub const fn transmute_ref_inference_helper(self) -> &'a Dst {
+ loop {}
+ }
+}
+
+impl<'a, Src, Dst> Wrap<&'a Src, &'a Dst> {
+ /// # Safety
+ /// The caller must guarantee that:
+ /// - `Src: IntoBytes + Immutable`
+ /// - `Dst: FromBytes + Immutable`
+ ///
+ /// # PME
+ ///
+ /// Instantiating this method PMEs unless both:
+ /// - `mem::size_of::<Dst>() == mem::size_of::<Src>()`
+ /// - `mem::align_of::<Dst>() <= mem::align_of::<Src>()`
+ #[inline(always)]
+ #[must_use]
+ pub const unsafe fn transmute_ref(self) -> &'a Dst {
+ static_assert!(Src, Dst => mem::size_of::<Dst>() == mem::size_of::<Src>());
+ static_assert!(Src, Dst => mem::align_of::<Dst>() <= mem::align_of::<Src>());
+
+ let src: *const Src = self.0;
+ let dst = src.cast::<Dst>();
+ // SAFETY:
+ // - We know that it is sound to view the target type of the input
+ // reference (`Src`) as the target type of the output reference
+ // (`Dst`) because the caller has guaranteed that `Src: IntoBytes`,
+ // `Dst: FromBytes`, and `size_of::<Src>() == size_of::<Dst>()`.
+ // - We know that there are no `UnsafeCell`s, and thus we don't have to
+ // worry about `UnsafeCell` overlap, because `Src: Immutable` and
+ // `Dst: Immutable`.
+ // - The caller has guaranteed that alignment is not increased.
+ // - We know that the returned lifetime will not outlive the input
+ // lifetime thanks to the lifetime bounds on this function.
+ //
+ // FIXME(#67): Once our MSRV is 1.58, replace this `transmute` with
+ // `&*dst`.
+ #[allow(clippy::transmute_ptr_to_ref)]
+ unsafe {
+ mem::transmute(dst)
+ }
+ }
+
+ #[inline(always)]
+ pub fn try_transmute_ref(self) -> Result<&'a Dst, ValidityError<&'a Src, Dst>>
+ where
+ Src: IntoBytes + Immutable,
+ Dst: TryFromBytes + Immutable,
+ {
+ static_assert!(Src => mem::align_of::<Src>() == mem::align_of::<Wrapping<Src>>());
+ static_assert!(Dst => mem::align_of::<Dst>() == mem::align_of::<Wrapping<Dst>>());
+
+ // SAFETY: By the preceding assert, `Src` and `Wrapping<Src>` have the
+ // same alignment.
+ let src: &Wrapping<Src> =
+ unsafe { crate::util::transmute_ref::<_, _, BecauseImmutable>(self.0) };
+ let src = Wrap::new(src);
+ <Wrap<&'a Wrapping<Src>, &'a Wrapping<Dst>> as TryTransmuteRefDst<'a>>::try_transmute_ref(
+ src,
+ )
+ .map(
+ // SAFETY: By the preceding assert, `Dst` and `Wrapping<Dst>` have
+ // the same alignment.
+ #[inline(always)]
+ |dst| unsafe { crate::util::transmute_ref::<_, _, BecauseImmutable>(dst) },
+ )
+ .map_err(
+ #[inline(always)]
+ |err| {
+ // SAFETY: By the preceding assert, `Src` and `Wrapping<Src>` have the
+ // same alignment.
+ ValidityError::new(unsafe {
+ crate::util::transmute_ref::<_, _, BecauseImmutable>(err.into_src())
+ })
+ },
+ )
+ }
+}
+
+impl<'a, Src, Dst> Wrap<&'a mut Src, &'a mut Dst>
+where
+ Src: ?Sized,
+ Dst: ?Sized,
+{
+ #[allow(clippy::must_use_candidate, clippy::missing_inline_in_public_items, clippy::empty_loop)]
+ pub fn transmute_mut_inference_helper(self) -> &'a mut Dst {
+ loop {}
+ }
+}
+
+impl<'a, Src, Dst> Wrap<&'a mut Src, &'a mut Dst> {
+ /// Transmutes a mutable reference of one type to a mutable reference of
+ /// another type.
+ ///
+ /// # PME
+ ///
+ /// Instantiating this method PMEs unless both:
+ /// - `mem::size_of::<Dst>() == mem::size_of::<Src>()`
+ /// - `mem::align_of::<Dst>() <= mem::align_of::<Src>()`
+ #[inline(always)]
+ #[must_use]
+ pub fn transmute_mut(self) -> &'a mut Dst
+ where
+ Src: FromBytes + IntoBytes,
+ Dst: FromBytes + IntoBytes,
+ {
+ static_assert!(Src, Dst => mem::size_of::<Dst>() == mem::size_of::<Src>());
+ static_assert!(Src, Dst => mem::align_of::<Dst>() <= mem::align_of::<Src>());
+
+ let src: *mut Src = self.0;
+ let dst = src.cast::<Dst>();
+ // SAFETY:
+ // - We know that it is sound to view the target type of the input
+ // reference (`Src`) as the target type of the output reference
+ // (`Dst`) and vice-versa because `Src: FromBytes + IntoBytes`, `Dst:
+ // FromBytes + IntoBytes`, and (as asserted above) `size_of::<Src>()
+ // == size_of::<Dst>()`.
+ // - We asserted above that alignment will not increase.
+ // - We know that the returned lifetime will not outlive the input
+ // lifetime thanks to the lifetime bounds on this function.
+ unsafe { &mut *dst }
+ }
+
+ #[inline(always)]
+ pub fn try_transmute_mut(self) -> Result<&'a mut Dst, ValidityError<&'a mut Src, Dst>>
+ where
+ Src: FromBytes + IntoBytes,
+ Dst: TryFromBytes + IntoBytes,
+ {
+ static_assert!(Src => mem::align_of::<Src>() == mem::align_of::<Wrapping<Src>>());
+ static_assert!(Dst => mem::align_of::<Dst>() == mem::align_of::<Wrapping<Dst>>());
+
+ // SAFETY: By the preceding assert, `Src` and `Wrapping<Src>` have the
+ // same alignment.
+ let src: &mut Wrapping<Src> =
+ unsafe { crate::util::transmute_mut::<_, _, (_, (_, _))>(self.0) };
+ let src = Wrap::new(src);
+ <Wrap<&'a mut Wrapping<Src>, &'a mut Wrapping<Dst>> as TryTransmuteMutDst<'a>>
+ ::try_transmute_mut(src)
+ // SAFETY: By the preceding assert, `Dst` and `Wrapping<Dst>` have the
+ // same alignment.
+ .map(|dst| unsafe { crate::util::transmute_mut::<_, _, (_, (_, _))>(dst) })
+ .map_err(|err| {
+ // SAFETY: By the preceding assert, `Src` and `Wrapping<Src>` have the
+ // same alignment.
+ ValidityError::new(unsafe {
+ crate::util::transmute_mut::<_, _, (_, (_, _))>(err.into_src())
+ })
+ })
+ }
+}
+
+pub trait TransmuteRefDst<'a> {
+ type Dst: ?Sized;
+
+ #[must_use]
+ fn transmute_ref(self) -> &'a Self::Dst;
+}
+
+impl<'a, Src: ?Sized, Dst: ?Sized> TransmuteRefDst<'a> for Wrap<&'a Src, &'a Dst>
+where
+ Src: KnownLayout + IntoBytes + Immutable,
+ Dst: KnownLayout<PointerMetadata = usize> + FromBytes + Immutable,
+{
+ type Dst = Dst;
+
+ #[inline(always)]
+ fn transmute_ref(self) -> &'a Dst {
+ let ptr = Ptr::from_ref(self.0)
+ .recall_validity::<Initialized, _>()
+ .transmute_with::<Dst, Initialized, crate::layout::CastFrom<Dst>, (crate::pointer::BecauseMutationCompatible, _)>()
+ .recall_validity::<Valid, _>();
+
+ static_assert!(Src: ?Sized + KnownLayout, Dst: ?Sized + KnownLayout => {
+ Src::LAYOUT.align.get() >= Dst::LAYOUT.align.get()
+ }, "cannot transmute reference when destination type has higher alignment than source type");
+
+ // SAFETY: The preceding `static_assert!` ensures that
+ // `Src::LAYOUT.align >= Dst::LAYOUT.align`. Since `self` is
+ // validly-aligned for `Src`, it is also validly-aligned for `Dst`.
+ let ptr = unsafe { ptr.assume_alignment() };
+
+ ptr.as_ref()
+ }
+}
+
+pub trait TransmuteMutDst<'a> {
+ type Dst: ?Sized;
+ #[must_use]
+ fn transmute_mut(self) -> &'a mut Self::Dst;
+}
+
+impl<'a, Src: ?Sized, Dst: ?Sized> TransmuteMutDst<'a> for Wrap<&'a mut Src, &'a mut Dst>
+where
+ Src: KnownLayout + FromBytes + IntoBytes,
+ Dst: KnownLayout<PointerMetadata = usize> + FromBytes + IntoBytes,
+{
+ type Dst = Dst;
+
+ #[inline(always)]
+ fn transmute_mut(self) -> &'a mut Dst {
+ let ptr = Ptr::from_mut(self.0)
+ .recall_validity::<Initialized, (_, (_, _))>()
+ .transmute_with::<Dst, Initialized, crate::layout::CastFrom<Dst>, _>()
+ .recall_validity::<Valid, (_, (_, _))>();
+
+ static_assert!(Src: ?Sized + KnownLayout, Dst: ?Sized + KnownLayout => {
+ Src::LAYOUT.align.get() >= Dst::LAYOUT.align.get()
+ }, "cannot transmute reference when destination type has higher alignment than source type");
+
+ // SAFETY: The preceding `static_assert!` ensures that
+ // `Src::LAYOUT.align >= Dst::LAYOUT.align`. Since `self` is
+ // validly-aligned for `Src`, it is also validly-aligned for `Dst`.
+ let ptr = unsafe { ptr.assume_alignment() };
+
+ ptr.as_mut()
+ }
+}
+
+/// A function which emits a warning if its return value is not used.
+#[must_use]
+#[inline(always)]
+pub const fn must_use<T>(t: T) -> T {
+ t
+}
+
+// NOTE: We can't change this to a `pub use core as core_reexport` until [1] is
+// fixed or we update to a semver-breaking version (as of this writing, 0.8.0)
+// on the `main` branch.
+//
+// [1] https://github.com/obi1kenobi/cargo-semver-checks/issues/573
+pub mod core_reexport {
+ pub use core::*;
+
+ pub mod mem {
+ pub use core::mem::*;
+ }
+}
+
+#[cfg(test)]
+mod tests {
+ use core::num::NonZeroUsize;
+
+ use crate::util::testutil::*;
+
+ #[cfg(__ZEROCOPY_INTERNAL_USE_ONLY_NIGHTLY_FEATURES_IN_TESTS)]
+ mod nightly {
+ use super::super::*;
+ use crate::util::testutil::*;
+
+ // FIXME(#29), FIXME(https://github.com/rust-lang/rust/issues/69835):
+ // Remove this `cfg` when `size_of_val_raw` is stabilized.
+ #[allow(clippy::decimal_literal_representation)]
+ #[test]
+ fn test_trailing_field_offset() {
+ assert_eq!(mem::align_of::<Aligned64kAllocation>(), _64K);
+
+ macro_rules! test {
+ (#[$cfg:meta] ($($ts:ty),* ; $trailing_field_ty:ty) => $expect:expr) => {{
+ #[$cfg]
+ struct Test($(#[allow(dead_code)] $ts,)* #[allow(dead_code)] $trailing_field_ty);
+ assert_eq!(test!(@offset $($ts),* ; $trailing_field_ty), $expect);
+ }};
+ (#[$cfg:meta] $(#[$cfgs:meta])* ($($ts:ty),* ; $trailing_field_ty:ty) => $expect:expr) => {
+ test!(#[$cfg] ($($ts),* ; $trailing_field_ty) => $expect);
+ test!($(#[$cfgs])* ($($ts),* ; $trailing_field_ty) => $expect);
+ };
+ (@offset ; $_trailing:ty) => { trailing_field_offset!(Test, 0) };
+ (@offset $_t:ty ; $_trailing:ty) => { trailing_field_offset!(Test, 1) };
+ }
+
+ test!(#[repr(C)] #[repr(transparent)] #[repr(packed)](; u8) => Some(0));
+ test!(#[repr(C)] #[repr(transparent)] #[repr(packed)](; [u8]) => Some(0));
+ test!(#[repr(C)] #[repr(C, packed)] (u8; u8) => Some(1));
+ test!(#[repr(C)] (; AU64) => Some(0));
+ test!(#[repr(C)] (; [AU64]) => Some(0));
+ test!(#[repr(C)] (u8; AU64) => Some(8));
+ test!(#[repr(C)] (u8; [AU64]) => Some(8));
+
+ #[derive(
+ Immutable, FromBytes, Eq, PartialEq, Ord, PartialOrd, Default, Debug, Copy, Clone,
+ )]
+ #[repr(C)]
+ pub(crate) struct Nested<T, U: ?Sized> {
+ _t: T,
+ _u: U,
+ }
+
+ test!(#[repr(C)] (; Nested<u8, AU64>) => Some(0));
+ test!(#[repr(C)] (; Nested<u8, [AU64]>) => Some(0));
+ test!(#[repr(C)] (u8; Nested<u8, AU64>) => Some(8));
+ test!(#[repr(C)] (u8; Nested<u8, [AU64]>) => Some(8));
+
+ // Test that `packed(N)` limits the offset of the trailing field.
+ test!(#[repr(C, packed( 1))] (u8; elain::Align< 2>) => Some( 1));
+ test!(#[repr(C, packed( 2))] (u8; elain::Align< 4>) => Some( 2));
+ test!(#[repr(C, packed( 4))] (u8; elain::Align< 8>) => Some( 4));
+ test!(#[repr(C, packed( 8))] (u8; elain::Align< 16>) => Some( 8));
+ test!(#[repr(C, packed( 16))] (u8; elain::Align< 32>) => Some( 16));
+ test!(#[repr(C, packed( 32))] (u8; elain::Align< 64>) => Some( 32));
+ test!(#[repr(C, packed( 64))] (u8; elain::Align< 128>) => Some( 64));
+ test!(#[repr(C, packed( 128))] (u8; elain::Align< 256>) => Some( 128));
+ test!(#[repr(C, packed( 256))] (u8; elain::Align< 512>) => Some( 256));
+ test!(#[repr(C, packed( 512))] (u8; elain::Align< 1024>) => Some( 512));
+ test!(#[repr(C, packed( 1024))] (u8; elain::Align< 2048>) => Some( 1024));
+ test!(#[repr(C, packed( 2048))] (u8; elain::Align< 4096>) => Some( 2048));
+ test!(#[repr(C, packed( 4096))] (u8; elain::Align< 8192>) => Some( 4096));
+ test!(#[repr(C, packed( 8192))] (u8; elain::Align< 16384>) => Some( 8192));
+ test!(#[repr(C, packed( 16384))] (u8; elain::Align< 32768>) => Some( 16384));
+ test!(#[repr(C, packed( 32768))] (u8; elain::Align< 65536>) => Some( 32768));
+ test!(#[repr(C, packed( 65536))] (u8; elain::Align< 131072>) => Some( 65536));
+ /* Alignments above 65536 are not yet supported.
+ test!(#[repr(C, packed( 131072))] (u8; elain::Align< 262144>) => Some( 131072));
+ test!(#[repr(C, packed( 262144))] (u8; elain::Align< 524288>) => Some( 262144));
+ test!(#[repr(C, packed( 524288))] (u8; elain::Align< 1048576>) => Some( 524288));
+ test!(#[repr(C, packed( 1048576))] (u8; elain::Align< 2097152>) => Some( 1048576));
+ test!(#[repr(C, packed( 2097152))] (u8; elain::Align< 4194304>) => Some( 2097152));
+ test!(#[repr(C, packed( 4194304))] (u8; elain::Align< 8388608>) => Some( 4194304));
+ test!(#[repr(C, packed( 8388608))] (u8; elain::Align< 16777216>) => Some( 8388608));
+ test!(#[repr(C, packed( 16777216))] (u8; elain::Align< 33554432>) => Some( 16777216));
+ test!(#[repr(C, packed( 33554432))] (u8; elain::Align< 67108864>) => Some( 33554432));
+ test!(#[repr(C, packed( 67108864))] (u8; elain::Align< 33554432>) => Some( 67108864));
+ test!(#[repr(C, packed( 33554432))] (u8; elain::Align<134217728>) => Some( 33554432));
+ test!(#[repr(C, packed(134217728))] (u8; elain::Align<268435456>) => Some(134217728));
+ test!(#[repr(C, packed(268435456))] (u8; elain::Align<268435456>) => Some(268435456));
+ */
+
+ // Test that `align(N)` does not limit the offset of the trailing field.
+ test!(#[repr(C, align( 1))] (u8; elain::Align< 2>) => Some( 2));
+ test!(#[repr(C, align( 2))] (u8; elain::Align< 4>) => Some( 4));
+ test!(#[repr(C, align( 4))] (u8; elain::Align< 8>) => Some( 8));
+ test!(#[repr(C, align( 8))] (u8; elain::Align< 16>) => Some( 16));
+ test!(#[repr(C, align( 16))] (u8; elain::Align< 32>) => Some( 32));
+ test!(#[repr(C, align( 32))] (u8; elain::Align< 64>) => Some( 64));
+ test!(#[repr(C, align( 64))] (u8; elain::Align< 128>) => Some( 128));
+ test!(#[repr(C, align( 128))] (u8; elain::Align< 256>) => Some( 256));
+ test!(#[repr(C, align( 256))] (u8; elain::Align< 512>) => Some( 512));
+ test!(#[repr(C, align( 512))] (u8; elain::Align< 1024>) => Some( 1024));
+ test!(#[repr(C, align( 1024))] (u8; elain::Align< 2048>) => Some( 2048));
+ test!(#[repr(C, align( 2048))] (u8; elain::Align< 4096>) => Some( 4096));
+ test!(#[repr(C, align( 4096))] (u8; elain::Align< 8192>) => Some( 8192));
+ test!(#[repr(C, align( 8192))] (u8; elain::Align< 16384>) => Some( 16384));
+ test!(#[repr(C, align( 16384))] (u8; elain::Align< 32768>) => Some( 32768));
+ test!(#[repr(C, align( 32768))] (u8; elain::Align< 65536>) => Some( 65536));
+ /* Alignments above 65536 are not yet supported.
+ test!(#[repr(C, align( 65536))] (u8; elain::Align< 131072>) => Some( 131072));
+ test!(#[repr(C, align( 131072))] (u8; elain::Align< 262144>) => Some( 262144));
+ test!(#[repr(C, align( 262144))] (u8; elain::Align< 524288>) => Some( 524288));
+ test!(#[repr(C, align( 524288))] (u8; elain::Align< 1048576>) => Some( 1048576));
+ test!(#[repr(C, align( 1048576))] (u8; elain::Align< 2097152>) => Some( 2097152));
+ test!(#[repr(C, align( 2097152))] (u8; elain::Align< 4194304>) => Some( 4194304));
+ test!(#[repr(C, align( 4194304))] (u8; elain::Align< 8388608>) => Some( 8388608));
+ test!(#[repr(C, align( 8388608))] (u8; elain::Align< 16777216>) => Some( 16777216));
+ test!(#[repr(C, align( 16777216))] (u8; elain::Align< 33554432>) => Some( 33554432));
+ test!(#[repr(C, align( 33554432))] (u8; elain::Align< 67108864>) => Some( 67108864));
+ test!(#[repr(C, align( 67108864))] (u8; elain::Align< 33554432>) => Some( 33554432));
+ test!(#[repr(C, align( 33554432))] (u8; elain::Align<134217728>) => Some(134217728));
+ test!(#[repr(C, align(134217728))] (u8; elain::Align<268435456>) => Some(268435456));
+ */
+ }
+
+ // FIXME(#29), FIXME(https://github.com/rust-lang/rust/issues/69835):
+ // Remove this `cfg` when `size_of_val_raw` is stabilized.
+ #[allow(clippy::decimal_literal_representation)]
+ #[test]
+ fn test_align_of_dst() {
+ // Test that `align_of!` correctly computes the alignment of DSTs.
+ assert_eq!(align_of!([elain::Align<1>]), Some(1));
+ assert_eq!(align_of!([elain::Align<2>]), Some(2));
+ assert_eq!(align_of!([elain::Align<4>]), Some(4));
+ assert_eq!(align_of!([elain::Align<8>]), Some(8));
+ assert_eq!(align_of!([elain::Align<16>]), Some(16));
+ assert_eq!(align_of!([elain::Align<32>]), Some(32));
+ assert_eq!(align_of!([elain::Align<64>]), Some(64));
+ assert_eq!(align_of!([elain::Align<128>]), Some(128));
+ assert_eq!(align_of!([elain::Align<256>]), Some(256));
+ assert_eq!(align_of!([elain::Align<512>]), Some(512));
+ assert_eq!(align_of!([elain::Align<1024>]), Some(1024));
+ assert_eq!(align_of!([elain::Align<2048>]), Some(2048));
+ assert_eq!(align_of!([elain::Align<4096>]), Some(4096));
+ assert_eq!(align_of!([elain::Align<8192>]), Some(8192));
+ assert_eq!(align_of!([elain::Align<16384>]), Some(16384));
+ assert_eq!(align_of!([elain::Align<32768>]), Some(32768));
+ assert_eq!(align_of!([elain::Align<65536>]), Some(65536));
+ /* Alignments above 65536 are not yet supported.
+ assert_eq!(align_of!([elain::Align<131072>]), Some(131072));
+ assert_eq!(align_of!([elain::Align<262144>]), Some(262144));
+ assert_eq!(align_of!([elain::Align<524288>]), Some(524288));
+ assert_eq!(align_of!([elain::Align<1048576>]), Some(1048576));
+ assert_eq!(align_of!([elain::Align<2097152>]), Some(2097152));
+ assert_eq!(align_of!([elain::Align<4194304>]), Some(4194304));
+ assert_eq!(align_of!([elain::Align<8388608>]), Some(8388608));
+ assert_eq!(align_of!([elain::Align<16777216>]), Some(16777216));
+ assert_eq!(align_of!([elain::Align<33554432>]), Some(33554432));
+ assert_eq!(align_of!([elain::Align<67108864>]), Some(67108864));
+ assert_eq!(align_of!([elain::Align<33554432>]), Some(33554432));
+ assert_eq!(align_of!([elain::Align<134217728>]), Some(134217728));
+ assert_eq!(align_of!([elain::Align<268435456>]), Some(268435456));
+ */
+ }
+ }
+
+ #[test]
+ fn test_enum_casts() {
+ // Test that casting the variants of enums with signed integer reprs to
+ // unsigned integers obeys expected signed -> unsigned casting rules.
+
+ #[repr(i8)]
+ enum ReprI8 {
+ MinusOne = -1,
+ Zero = 0,
+ Min = i8::MIN,
+ Max = i8::MAX,
+ }
+
+ #[allow(clippy::as_conversions)]
+ let x = ReprI8::MinusOne as u8;
+ assert_eq!(x, u8::MAX);
+
+ #[allow(clippy::as_conversions)]
+ let x = ReprI8::Zero as u8;
+ assert_eq!(x, 0);
+
+ #[allow(clippy::as_conversions)]
+ let x = ReprI8::Min as u8;
+ assert_eq!(x, 128);
+
+ #[allow(clippy::as_conversions)]
+ let x = ReprI8::Max as u8;
+ assert_eq!(x, 127);
+ }
+
+ #[test]
+ fn test_struct_padding() {
+ // Test that, for each provided repr, `struct_padding!` reports the
+ // expected value.
+ macro_rules! test {
+ (#[$cfg:meta] ($($ts:ty),*) => $expect:expr) => {{
+ #[$cfg]
+ #[allow(dead_code)]
+ struct Test($($ts),*);
+ assert_eq!(struct_padding!(Test, None::<NonZeroUsize>, None::<NonZeroUsize>, [$($ts),*]), $expect);
+ }};
+ (#[$cfg:meta] $(#[$cfgs:meta])* ($($ts:ty),*) => $expect:expr) => {
+ test!(#[$cfg] ($($ts),*) => $expect);
+ test!($(#[$cfgs])* ($($ts),*) => $expect);
+ };
+ }
+
+ test!(#[repr(C)] #[repr(transparent)] #[repr(packed)] () => 0);
+ test!(#[repr(C)] #[repr(transparent)] #[repr(packed)] (u8) => 0);
+ test!(#[repr(C)] #[repr(transparent)] #[repr(packed)] (u8, ()) => 0);
+ test!(#[repr(C)] #[repr(packed)] (u8, u8) => 0);
+
+ test!(#[repr(C)] (u8, AU64) => 7);
+ // Rust won't let you put `#[repr(packed)]` on a type which contains a
+ // `#[repr(align(n > 1))]` type (`AU64`), so we have to use `u64` here.
+ // It's not ideal, but it definitely has align > 1 on /some/ of our CI
+ // targets, and this isn't a particularly complex macro we're testing
+ // anyway.
+ test!(#[repr(packed)] (u8, u64) => 0);
+ }
+
+ #[test]
+ fn test_repr_c_struct_padding() {
+ // Test that, for each provided repr, `repr_c_struct_padding!` reports
+ // the expected value.
+ macro_rules! test {
+ (($($ts:tt),*) => $expect:expr) => {{
+ #[repr(C)]
+ #[allow(dead_code)]
+ struct Test($($ts),*);
+ assert_eq!(repr_c_struct_has_padding!(Test, None::<NonZeroUsize>, None::<NonZeroUsize>, [$($ts),*]), $expect);
+ }};
+ }
+
+ // Test static padding
+ test!(() => false);
+ test!(([u8]) => false);
+ test!((u8) => false);
+ test!((u8, [u8]) => false);
+ test!((u8, ()) => false);
+ test!((u8, (), [u8]) => false);
+ test!((u8, u8) => false);
+ test!((u8, u8, [u8]) => false);
+
+ test!((u8, AU64) => true);
+ test!((u8, AU64, [u8]) => true);
+
+ // Test dynamic padding
+ test!((AU64, [AU64]) => false);
+ test!((u8, [AU64]) => true);
+
+ #[repr(align(4))]
+ struct AU32(#[allow(unused)] u32);
+ test!((AU64, [AU64]) => false);
+ test!((AU64, [AU32]) => true);
+ }
+
+ #[test]
+ fn test_union_padding() {
+ // Test that, for each provided repr, `union_padding!` reports the
+ // expected value.
+ macro_rules! test {
+ (#[$cfg:meta] {$($fs:ident: $ts:ty),*} => $expect:expr) => {{
+ #[$cfg]
+ #[allow(unused)] // fields are never read
+ union Test{ $($fs: $ts),* }
+ assert_eq!(union_padding!(Test, None::<NonZeroUsize>, None::<usize>, [$($ts),*]), $expect);
+ }};
+ (#[$cfg:meta] $(#[$cfgs:meta])* {$($fs:ident: $ts:ty),*} => $expect:expr) => {
+ test!(#[$cfg] {$($fs: $ts),*} => $expect);
+ test!($(#[$cfgs])* {$($fs: $ts),*} => $expect);
+ };
+ }
+
+ test!(#[repr(C)] #[repr(packed)] {a: u8} => 0);
+ test!(#[repr(C)] #[repr(packed)] {a: u8, b: u8} => 0);
+
+ // Rust won't let you put `#[repr(packed)]` on a type which contains a
+ // `#[repr(align(n > 1))]` type (`AU64`), so we have to use `u64` here.
+ // It's not ideal, but it definitely has align > 1 on /some/ of our CI
+ // targets, and this isn't a particularly complex macro we're testing
+ // anyway.
+ test!(#[repr(C)] #[repr(packed)] {a: u8, b: u64} => 7);
+ }
+
+ #[test]
+ fn test_enum_padding() {
+ // Test that, for each provided repr, `enum_has_padding!` reports the
+ // expected value.
+ macro_rules! test {
+ (#[repr($disc:ident $(, $c:ident)?)] { $($vs:ident ($($ts:ty),*),)* } => $expect:expr) => {
+ test!(@case #[repr($disc $(, $c)?)] { $($vs ($($ts),*),)* } => $expect);
+ };
+ (#[repr($disc:ident $(, $c:ident)?)] #[$cfg:meta] $(#[$cfgs:meta])* { $($vs:ident ($($ts:ty),*),)* } => $expect:expr) => {
+ test!(@case #[repr($disc $(, $c)?)] #[$cfg] { $($vs ($($ts),*),)* } => $expect);
+ test!(#[repr($disc $(, $c)?)] $(#[$cfgs])* { $($vs ($($ts),*),)* } => $expect);
+ };
+ (@case #[repr($disc:ident $(, $c:ident)?)] $(#[$cfg:meta])? { $($vs:ident ($($ts:ty),*),)* } => $expect:expr) => {{
+ #[repr($disc $(, $c)?)]
+ $(#[$cfg])?
+ #[allow(unused)] // variants and fields are never used
+ enum Test {
+ $($vs ($($ts),*),)*
+ }
+ assert_eq!(
+ enum_padding!(Test, None::<NonZeroUsize>, None::<NonZeroUsize>, $disc, $([$($ts),*]),*),
+ $expect
+ );
+ }};
+ }
+
+ #[allow(unused)]
+ #[repr(align(2))]
+ struct U16(u16);
+
+ #[allow(unused)]
+ #[repr(align(4))]
+ struct U32(u32);
+
+ test!(#[repr(u8)] #[repr(C)] {
+ A(u8),
+ } => 0);
+ test!(#[repr(u16)] #[repr(C)] {
+ A(u8, u8),
+ B(U16),
+ } => 0);
+ test!(#[repr(u32)] #[repr(C)] {
+ A(u8, u8, u8, u8),
+ B(U16, u8, u8),
+ C(u8, u8, U16),
+ D(U16, U16),
+ E(U32),
+ } => 0);
+
+ // `repr(int)` can pack the discriminant more efficiently
+ test!(#[repr(u8)] {
+ A(u8, U16),
+ } => 0);
+ test!(#[repr(u8)] {
+ A(u8, U16, U32),
+ } => 0);
+
+ // `repr(C)` cannot
+ test!(#[repr(u8, C)] {
+ A(u8, U16),
+ } => 2);
+ test!(#[repr(u8, C)] {
+ A(u8, u8, u8, U32),
+ } => 4);
+
+ // And field ordering can always cause problems
+ test!(#[repr(u8)] #[repr(C)] {
+ A(U16, u8),
+ } => 2);
+ test!(#[repr(u8)] #[repr(C)] {
+ A(U32, u8, u8, u8),
+ } => 4);
+ }
+}
diff --git a/rust/zerocopy/src/util/macros.rs b/rust/zerocopy/src/util/macros.rs
new file mode 100644
index 0000000..43e4fd6
--- /dev/null
+++ b/rust/zerocopy/src/util/macros.rs
@@ -0,0 +1,1067 @@
+// SPDX-License-Identifier: (BSD-2-Clause OR Apache-2.0) OR MIT
+
+// Copyright 2023 The Fuchsia Authors
+//
+// Licensed under a BSD-style license <LICENSE-BSD>, Apache License, Version 2.0
+// <LICENSE-APACHE or https://www.apache.org/licenses/LICENSE-2.0>, or the MIT
+// license <LICENSE-MIT or https://opensource.org/licenses/MIT>, at your option.
+// This file may not be copied, modified, or distributed except according to
+// those terms.
+
+/// Unsafely implements trait(s) for a type.
+///
+/// # Safety
+///
+/// The trait impl must be sound.
+///
+/// When implementing `TryFromBytes`:
+/// - If no `is_bit_valid` impl is provided, then it must be valid for
+/// `is_bit_valid` to unconditionally return `true`. In other words, it must
+/// be the case that any initialized sequence of bytes constitutes a valid
+/// instance of `$ty`.
+/// - If an `is_bit_valid` impl is provided, then the impl of `is_bit_valid`
+/// must only return `true` if its argument refers to a valid `$ty`.
+macro_rules! unsafe_impl {
+ // Implement `$trait` for `$ty` with no bounds.
+ ($(#[$attr:meta])* $ty:ty: $trait:ident $(; |$candidate:ident| $is_bit_valid:expr)?) => {{
+ crate::util::macros::__unsafe();
+
+ $(#[$attr])*
+ // SAFETY: The caller promises that this is sound.
+ unsafe impl $trait for $ty {
+ unsafe_impl!(@method $trait $(; |$candidate| $is_bit_valid)?);
+ }
+ }};
+
+ // Implement all `$traits` for `$ty` with no bounds.
+ //
+ // The 2 arms under this one are there so we can apply
+ // N attributes for each one of M trait implementations.
+ // The simple solution of:
+ //
+ // ($(#[$attrs:meta])* $ty:ty: $($traits:ident),*) => {
+ // $( unsafe_impl!( $(#[$attrs])* $ty: $traits ) );*
+ // }
+ //
+ // Won't work. The macro processor sees that the outer repetition
+ // contains both $attrs and $traits and expects them to match the same
+ // amount of fragments.
+ //
+ // To solve this we must:
+ // 1. Pack the attributes into a single token tree fragment we can match over.
+ // 2. Expand the traits.
+ // 3. Unpack and expand the attributes.
+ ($(#[$attrs:meta])* $ty:ty: $($traits:ident),*) => {
+ unsafe_impl!(@impl_traits_with_packed_attrs { $(#[$attrs])* } $ty: $($traits),*)
+ };
+
+ (@impl_traits_with_packed_attrs $attrs:tt $ty:ty: $($traits:ident),*) => {{
+ $( unsafe_impl!(@unpack_attrs $attrs $ty: $traits); )*
+ }};
+
+ (@unpack_attrs { $(#[$attrs:meta])* } $ty:ty: $traits:ident) => {
+ unsafe_impl!($(#[$attrs])* $ty: $traits);
+ };
+
+ // This arm is identical to the following one, except it contains a
+ // preceding `const`. If we attempt to handle these with a single arm, there
+ // is an inherent ambiguity between `const` (the keyword) and `const` (the
+ // ident match for `$tyvar:ident`).
+ //
+ // To explain how this works, consider the following invocation:
+ //
+ // unsafe_impl!(const N: usize, T: ?Sized + Copy => Clone for Foo<T>);
+ //
+ // In this invocation, here are the assignments to meta-variables:
+ //
+ // |---------------|------------|
+ // | Meta-variable | Assignment |
+ // |---------------|------------|
+ // | $constname | N |
+ // | $constty | usize |
+ // | $tyvar | T |
+ // | $optbound | Sized |
+ // | $bound | Copy |
+ // | $trait | Clone |
+ // | $ty | Foo<T> |
+ // |---------------|------------|
+ //
+ // The following arm has the same behavior with the exception of the lack of
+ // support for a leading `const` parameter.
+ (
+ $(#[$attr:meta])*
+ const $constname:ident : $constty:ident $(,)?
+ $($tyvar:ident $(: $(? $optbound:ident $(+)?)* $($bound:ident $(+)?)* )?),*
+ => $trait:ident for $ty:ty $(; |$candidate:ident| $is_bit_valid:expr)?
+ ) => {
+ unsafe_impl!(
+ @inner
+ $(#[$attr])*
+ @const $constname: $constty,
+ $($tyvar $(: $(? $optbound +)* + $($bound +)*)?,)*
+ => $trait for $ty $(; |$candidate| $is_bit_valid)?
+ );
+ };
+ (
+ $(#[$attr:meta])*
+ $($tyvar:ident $(: $(? $optbound:ident $(+)?)* $($bound:ident $(+)?)* )?),*
+ => $trait:ident for $ty:ty $(; |$candidate:ident| $is_bit_valid:expr)?
+ ) => {{
+ unsafe_impl!(
+ @inner
+ $(#[$attr])*
+ $($tyvar $(: $(? $optbound +)* + $($bound +)*)?,)*
+ => $trait for $ty $(; |$candidate| $is_bit_valid)?
+ );
+ }};
+ (
+ @inner
+ $(#[$attr:meta])*
+ $(@const $constname:ident : $constty:ident,)*
+ $($tyvar:ident $(: $(? $optbound:ident +)* + $($bound:ident +)* )?,)*
+ => $trait:ident for $ty:ty $(; |$candidate:ident| $is_bit_valid:expr)?
+ ) => {{
+ crate::util::macros::__unsafe();
+
+ $(#[$attr])*
+ #[allow(non_local_definitions)]
+ // SAFETY: The caller promises that this is sound.
+ unsafe impl<$($tyvar $(: $(? $optbound +)* $($bound +)*)?),* $(, const $constname: $constty,)*> $trait for $ty {
+ unsafe_impl!(@method $trait $(; |$candidate| $is_bit_valid)?);
+ }
+ }};
+
+ (@method TryFromBytes ; |$candidate:ident| $is_bit_valid:expr) => {
+ #[allow(clippy::missing_inline_in_public_items, dead_code)]
+ #[cfg_attr(all(coverage_nightly, __ZEROCOPY_INTERNAL_USE_ONLY_NIGHTLY_FEATURES_IN_TESTS), coverage(off))]
+ fn only_derive_is_allowed_to_implement_this_trait() {}
+
+ #[inline]
+ fn is_bit_valid<Alignment>($candidate: Maybe<'_, Self, Alignment>) -> bool
+ where
+ Alignment: crate::invariant::Alignment,
+ {
+ $is_bit_valid
+ }
+ };
+ (@method TryFromBytes) => {
+ #[allow(clippy::missing_inline_in_public_items)]
+ #[cfg_attr(all(coverage_nightly, __ZEROCOPY_INTERNAL_USE_ONLY_NIGHTLY_FEATURES_IN_TESTS), coverage(off))]
+ fn only_derive_is_allowed_to_implement_this_trait() {}
+ #[inline(always)]
+ fn is_bit_valid<Alignment>(_candidate: Maybe<'_, Self, Alignment>) -> bool
+ where
+ Alignment: crate::invariant::Alignment,
+ {
+ true
+ }
+ };
+ (@method $trait:ident) => {
+ #[allow(clippy::missing_inline_in_public_items, dead_code)]
+ #[cfg_attr(all(coverage_nightly, __ZEROCOPY_INTERNAL_USE_ONLY_NIGHTLY_FEATURES_IN_TESTS), coverage(off))]
+ fn only_derive_is_allowed_to_implement_this_trait() {}
+ };
+ (@method $trait:ident; |$_candidate:ident| $_is_bit_valid:expr) => {
+ compile_error!("Can't provide `is_bit_valid` impl for trait other than `TryFromBytes`");
+ };
+}
+
+/// Implements `$trait` for `$ty` where `$ty: TransmuteFrom<$repr>` (and
+/// vice-versa).
+///
+/// Calling this macro is safe; the internals of the macro emit appropriate
+/// trait bounds which ensure that the given impl is sound.
+macro_rules! impl_for_transmute_from {
+ (
+ $(#[$attr:meta])*
+ $($tyvar:ident $(: $(? $optbound:ident $(+)?)* $($bound:ident $(+)?)* )?)?
+ => $trait:ident for $ty:ty [$repr:ty]
+ ) => {
+ const _: () = {
+ $(#[$attr])*
+ #[allow(non_local_definitions)]
+
+ // SAFETY: `is_trait<T, R>` (defined and used below) requires `T:
+ // TransmuteFrom<R>`, `R: TransmuteFrom<T>`, and `R: $trait`. It is
+ // called using `$ty` and `$repr`, ensuring that `$ty` and `$repr`
+ // have equivalent bit validity, and ensuring that `$repr: $trait`.
+ // The supported traits - `TryFromBytes`, `FromZeros`, `FromBytes`,
+ // and `IntoBytes` - are defined only in terms of the bit validity
+ // of a type. Therefore, `$repr: $trait` ensures that `$ty: $trait`
+ // is sound.
+ unsafe impl<$($tyvar $(: $(? $optbound +)* $($bound +)*)?)?> $trait for $ty {
+ #[allow(dead_code, clippy::missing_inline_in_public_items)]
+ #[cfg_attr(all(coverage_nightly, __ZEROCOPY_INTERNAL_USE_ONLY_NIGHTLY_FEATURES_IN_TESTS), coverage(off))]
+ fn only_derive_is_allowed_to_implement_this_trait() {
+ use crate::pointer::{*, invariant::Valid};
+
+ impl_for_transmute_from!(@assert_is_supported_trait $trait);
+
+ fn is_trait<T, R>()
+ where
+ T: TransmuteFrom<R, Valid, Valid> + ?Sized,
+ R: TransmuteFrom<T, Valid, Valid> + ?Sized,
+ R: $trait,
+ {
+ }
+
+ #[cfg_attr(all(coverage_nightly, __ZEROCOPY_INTERNAL_USE_ONLY_NIGHTLY_FEATURES_IN_TESTS), coverage(off))]
+ fn f<$($tyvar $(: $(? $optbound +)* $($bound +)*)?)?>() {
+ is_trait::<$ty, $repr>();
+ }
+ }
+
+ impl_for_transmute_from!(
+ @is_bit_valid
+ $(<$tyvar $(: $(? $optbound +)* $($bound +)*)?>)?
+ $trait for $ty [$repr]
+ );
+ }
+ };
+ };
+ (@assert_is_supported_trait TryFromBytes) => {};
+ (@assert_is_supported_trait FromZeros) => {};
+ (@assert_is_supported_trait FromBytes) => {};
+ (@assert_is_supported_trait IntoBytes) => {};
+ (
+ @is_bit_valid
+ $(<$tyvar:ident $(: $(? $optbound:ident $(+)?)* $($bound:ident $(+)?)* )?>)?
+ TryFromBytes for $ty:ty [$repr:ty]
+ ) => {
+ #[inline(always)]
+ fn is_bit_valid<Alignment>(candidate: $crate::Maybe<'_, Self, Alignment>) -> bool
+ where
+ Alignment: $crate::invariant::Alignment,
+ {
+ // SAFETY: This macro ensures that `$repr` and `Self` have the same
+ // size and bit validity. Thus, a bit-valid instance of `$repr` is
+ // also a bit-valid instance of `Self`.
+ <$repr as TryFromBytes>::is_bit_valid(candidate.transmute::<_, _, BecauseImmutable>())
+ }
+ };
+ (
+ @is_bit_valid
+ $(<$tyvar:ident $(: $(? $optbound:ident $(+)?)* $($bound:ident $(+)?)* )?>)?
+ $trait:ident for $ty:ty [$repr:ty]
+ ) => {
+ // Trait other than `TryFromBytes`; no `is_bit_valid` impl.
+ };
+}
+
+/// Implements a trait for a type, bounding on each member of the power set of
+/// a set of type variables. This is useful for implementing traits for tuples
+/// or `fn` types.
+///
+/// The last argument is the name of a macro which will be called in every
+/// `impl` block, and is expected to expand to the name of the type for which to
+/// implement the trait.
+///
+/// For example, the invocation:
+/// ```ignore
+/// unsafe_impl_for_power_set!(A, B => Foo for type!(...))
+/// ```
+/// ...expands to:
+/// ```ignore
+/// unsafe impl Foo for type!() { ... }
+/// unsafe impl<B> Foo for type!(B) { ... }
+/// unsafe impl<A, B> Foo for type!(A, B) { ... }
+/// ```
+macro_rules! unsafe_impl_for_power_set {
+ (
+ $first:ident $(, $rest:ident)* $(-> $ret:ident)? => $trait:ident for $macro:ident!(...)
+ $(; |$candidate:ident| $is_bit_valid:expr)?
+ ) => {
+ unsafe_impl_for_power_set!(
+ $($rest),* $(-> $ret)? => $trait for $macro!(...)
+ $(; |$candidate| $is_bit_valid)?
+ );
+ unsafe_impl_for_power_set!(
+ @impl $first $(, $rest)* $(-> $ret)? => $trait for $macro!(...)
+ $(; |$candidate| $is_bit_valid)?
+ );
+ };
+ (
+ $(-> $ret:ident)? => $trait:ident for $macro:ident!(...)
+ $(; |$candidate:ident| $is_bit_valid:expr)?
+ ) => {
+ unsafe_impl_for_power_set!(
+ @impl $(-> $ret)? => $trait for $macro!(...)
+ $(; |$candidate| $is_bit_valid)?
+ );
+ };
+ (
+ @impl $($vars:ident),* $(-> $ret:ident)? => $trait:ident for $macro:ident!(...)
+ $(; |$candidate:ident| $is_bit_valid:expr)?
+ ) => {
+ unsafe_impl!(
+ $($vars,)* $($ret)? => $trait for $macro!($($vars),* $(-> $ret)?)
+ $(; |$candidate| $is_bit_valid)?
+ );
+ };
+}
+
+/// Expands to an `Option<extern "C" fn>` type with the given argument types and
+/// return type. Designed for use with `unsafe_impl_for_power_set`.
+macro_rules! opt_extern_c_fn {
+ ($($args:ident),* -> $ret:ident) => { Option<extern "C" fn($($args),*) -> $ret> };
+}
+
+/// Expands to an `Option<unsafe extern "C" fn>` type with the given argument
+/// types and return type. Designed for use with `unsafe_impl_for_power_set`.
+macro_rules! opt_unsafe_extern_c_fn {
+ ($($args:ident),* -> $ret:ident) => { Option<unsafe extern "C" fn($($args),*) -> $ret> };
+}
+
+/// Expands to an `Option<fn>` type with the given argument types and return
+/// type. Designed for use with `unsafe_impl_for_power_set`.
+macro_rules! opt_fn {
+ ($($args:ident),* -> $ret:ident) => { Option<fn($($args),*) -> $ret> };
+}
+
+/// Expands to an `Option<unsafe fn>` type with the given argument types and
+/// return type. Designed for use with `unsafe_impl_for_power_set`.
+macro_rules! opt_unsafe_fn {
+ ($($args:ident),* -> $ret:ident) => { Option<unsafe fn($($args),*) -> $ret> };
+}
+
+// This `allow` is needed because, when testing, we export this macro so it can
+// be used in `doctests`.
+#[allow(rustdoc::private_intra_doc_links)]
+/// Implements trait(s) for a type or verifies the given implementation by
+/// referencing an existing (derived) implementation.
+///
+/// This macro exists so that we can provide zerocopy-derive as an optional
+/// dependency and still get the benefit of using its derives to validate that
+/// our trait impls are sound.
+///
+/// When compiling without `--cfg 'feature = "derive"` and without `--cfg test`,
+/// `impl_or_verify!` emits the provided trait impl. When compiling with either
+/// of those cfgs, it is expected that the type in question is deriving the
+/// traits instead. In this case, `impl_or_verify!` emits code which validates
+/// that the given trait impl is at least as restrictive as the the impl emitted
+/// by the custom derive. This has the effect of confirming that the impl which
+/// is emitted when the `derive` feature is disabled is actually sound (on the
+/// assumption that the impl emitted by the custom derive is sound).
+///
+/// The caller is still required to provide a safety comment (e.g. using the
+/// `const _: () = unsafe` macro). The reason for this restriction is that,
+/// while `impl_or_verify!` can guarantee that the provided impl is sound when
+/// it is compiled with the appropriate cfgs, there is no way to guarantee that
+/// it is ever compiled with those cfgs. In particular, it would be possible to
+/// accidentally place an `impl_or_verify!` call in a context that is only ever
+/// compiled when the `derive` feature is disabled. If that were to happen,
+/// there would be nothing to prevent an unsound trait impl from being emitted.
+/// Requiring a safety comment reduces the likelihood of emitting an unsound
+/// impl in this case, and also provides useful documentation for readers of the
+/// code.
+///
+/// Finally, if a `TryFromBytes::is_bit_valid` impl is provided, it must adhere
+/// to the safety preconditions of [`unsafe_impl!`].
+///
+/// ## Example
+///
+/// ```rust,ignore
+/// // Note that these derives are gated by `feature = "derive"`
+/// #[cfg_attr(any(feature = "derive", test), derive(FromZeros, FromBytes, IntoBytes, Unaligned))]
+/// #[repr(transparent)]
+/// struct Wrapper<T>(T);
+///
+/// const _: () = unsafe {
+/// /// SAFETY:
+/// /// `Wrapper<T>` is `repr(transparent)`, so it is sound to implement any
+/// /// zerocopy trait if `T` implements that trait.
+/// impl_or_verify!(T: FromZeros => FromZeros for Wrapper<T>);
+/// impl_or_verify!(T: FromBytes => FromBytes for Wrapper<T>);
+/// impl_or_verify!(T: IntoBytes => IntoBytes for Wrapper<T>);
+/// impl_or_verify!(T: Unaligned => Unaligned for Wrapper<T>);
+/// }
+/// ```
+#[cfg_attr(__ZEROCOPY_INTERNAL_USE_ONLY_DEV_MODE, macro_export)] // Used in `doctests.rs`
+#[doc(hidden)]
+macro_rules! impl_or_verify {
+ // The following two match arms follow the same pattern as their
+ // counterparts in `unsafe_impl!`; see the documentation on those arms for
+ // more details.
+ (
+ const $constname:ident : $constty:ident $(,)?
+ $($tyvar:ident $(: $(? $optbound:ident $(+)?)* $($bound:ident $(+)?)* )?),*
+ => $trait:ident for $ty:ty
+ ) => {
+ impl_or_verify!(@impl { unsafe_impl!(
+ const $constname: $constty, $($tyvar $(: $(? $optbound +)* $($bound +)*)?),* => $trait for $ty
+ ); });
+ impl_or_verify!(@verify $trait, {
+ impl<const $constname: $constty, $($tyvar $(: $(? $optbound +)* $($bound +)*)?),*> Subtrait for $ty {}
+ });
+ };
+ (
+ $($tyvar:ident $(: $(? $optbound:ident $(+)?)* $($bound:ident $(+)?)* )?),*
+ => $trait:ident for $ty:ty $(; |$candidate:ident| $is_bit_valid:expr)?
+ ) => {
+ impl_or_verify!(@impl { unsafe_impl!(
+ $($tyvar $(: $(? $optbound +)* $($bound +)*)?),* => $trait for $ty
+ $(; |$candidate| $is_bit_valid)?
+ ); });
+ impl_or_verify!(@verify $trait, {
+ impl<$($tyvar $(: $(? $optbound +)* $($bound +)*)?),*> Subtrait for $ty {}
+ });
+ };
+ (@impl $impl_block:tt) => {
+ #[cfg(not(any(feature = "derive", test)))]
+ { $impl_block };
+ };
+ (@verify $trait:ident, $impl_block:tt) => {
+ #[cfg(any(feature = "derive", test))]
+ {
+ // On some toolchains, `Subtrait` triggers the `dead_code` lint
+ // because it is implemented but never used.
+ #[allow(dead_code)]
+ trait Subtrait: $trait {}
+ $impl_block
+ };
+ };
+}
+
+/// Implements `KnownLayout` for a sized type.
+macro_rules! impl_known_layout {
+ ($(const $constvar:ident : $constty:ty, $tyvar:ident $(: ?$optbound:ident)? => $ty:ty),* $(,)?) => {
+ $(impl_known_layout!(@inner const $constvar: $constty, $tyvar $(: ?$optbound)? => $ty);)*
+ };
+ ($($tyvar:ident $(: ?$optbound:ident)? => $ty:ty),* $(,)?) => {
+ $(impl_known_layout!(@inner , $tyvar $(: ?$optbound)? => $ty);)*
+ };
+ ($($(#[$attrs:meta])* $ty:ty),*) => { $(impl_known_layout!(@inner , => $(#[$attrs])* $ty);)* };
+ (@inner $(const $constvar:ident : $constty:ty)? , $($tyvar:ident $(: ?$optbound:ident)?)? => $(#[$attrs:meta])* $ty:ty) => {
+ const _: () = {
+ use core::ptr::NonNull;
+
+ #[allow(non_local_definitions)]
+ $(#[$attrs])*
+ // SAFETY: Delegates safety to `DstLayout::for_type`.
+ unsafe impl<$($tyvar $(: ?$optbound)?)? $(, const $constvar : $constty)?> KnownLayout for $ty {
+ #[allow(clippy::missing_inline_in_public_items)]
+ #[cfg_attr(all(coverage_nightly, __ZEROCOPY_INTERNAL_USE_ONLY_NIGHTLY_FEATURES_IN_TESTS), coverage(off))]
+ fn only_derive_is_allowed_to_implement_this_trait() where Self: Sized {}
+
+ type PointerMetadata = ();
+
+ // SAFETY: `CoreMaybeUninit<T>::LAYOUT` and `T::LAYOUT` are
+ // identical because `CoreMaybeUninit<T>` has the same size and
+ // alignment as `T` [1], and `CoreMaybeUninit` admits
+ // uninitialized bytes in all positions.
+ //
+ // [1] Per https://doc.rust-lang.org/1.81.0/std/mem/union.MaybeUninit.html#layout-1:
+ //
+ // `MaybeUninit<T>` is guaranteed to have the same size,
+ // alignment, and ABI as `T`
+ type MaybeUninit = core::mem::MaybeUninit<Self>;
+
+ const LAYOUT: crate::DstLayout = crate::DstLayout::for_type::<$ty>();
+
+ // SAFETY: `.cast` preserves address and provenance.
+ //
+ // FIXME(#429): Add documentation to `.cast` that promises that
+ // it preserves provenance.
+ #[inline(always)]
+ fn raw_from_ptr_len(bytes: NonNull<u8>, _meta: ()) -> NonNull<Self> {
+ bytes.cast::<Self>()
+ }
+
+ #[inline(always)]
+ fn pointer_to_metadata(_ptr: *mut Self) -> () {
+ }
+ }
+ };
+ };
+}
+
+/// Implements `KnownLayout` for a type in terms of the implementation of
+/// another type with the same representation.
+///
+/// # Safety
+///
+/// - `$ty` and `$repr` must have the same:
+/// - Fixed prefix size
+/// - Alignment
+/// - (For DSTs) trailing slice element size
+/// - It must be valid to perform an `as` cast from `*mut $repr` to `*mut $ty`,
+/// and this operation must preserve referent size (ie, `size_of_val_raw`).
+macro_rules! unsafe_impl_known_layout {
+ ($($tyvar:ident: ?Sized + KnownLayout =>)? #[repr($repr:ty)] $ty:ty) => {{
+ use core::ptr::NonNull;
+
+ crate::util::macros::__unsafe();
+
+ #[allow(non_local_definitions)]
+ // SAFETY: The caller promises that this is sound.
+ unsafe impl<$($tyvar: ?Sized + KnownLayout)?> KnownLayout for $ty {
+ #[allow(clippy::missing_inline_in_public_items, dead_code)]
+ #[cfg_attr(all(coverage_nightly, __ZEROCOPY_INTERNAL_USE_ONLY_NIGHTLY_FEATURES_IN_TESTS), coverage(off))]
+ fn only_derive_is_allowed_to_implement_this_trait() {}
+
+ type PointerMetadata = <$repr as KnownLayout>::PointerMetadata;
+ type MaybeUninit = <$repr as KnownLayout>::MaybeUninit;
+
+ const LAYOUT: DstLayout = <$repr as KnownLayout>::LAYOUT;
+
+ // SAFETY: All operations preserve address and provenance. Caller
+ // has promised that the `as` cast preserves size.
+ //
+ // FIXME(#429): Add documentation to `NonNull::new_unchecked` that
+ // it preserves provenance.
+ #[inline(always)]
+ fn raw_from_ptr_len(bytes: NonNull<u8>, meta: <$repr as KnownLayout>::PointerMetadata) -> NonNull<Self> {
+ #[allow(clippy::as_conversions)]
+ let ptr = <$repr>::raw_from_ptr_len(bytes, meta).as_ptr() as *mut Self;
+ // SAFETY: `ptr` was converted from `bytes`, which is non-null.
+ unsafe { NonNull::new_unchecked(ptr) }
+ }
+
+ #[inline(always)]
+ fn pointer_to_metadata(ptr: *mut Self) -> Self::PointerMetadata {
+ #[allow(clippy::as_conversions)]
+ let ptr = ptr as *mut $repr;
+ <$repr>::pointer_to_metadata(ptr)
+ }
+ }
+ }};
+}
+
+/// Uses `align_of` to confirm that a type or set of types have alignment 1.
+///
+/// Note that `align_of<T>` requires `T: Sized`, so this macro doesn't work for
+/// unsized types.
+macro_rules! assert_unaligned {
+ ($($tys:ty),*) => {
+ $(
+ // We only compile this assertion under `cfg(test)` to avoid taking
+ // an extra non-dev dependency (and making this crate more expensive
+ // to compile for our dependents).
+ #[cfg(test)]
+ static_assertions::const_assert_eq!(core::mem::align_of::<$tys>(), 1);
+ )*
+ };
+}
+
+/// Emits a function definition as either `const fn` or `fn` depending on
+/// whether the current toolchain version supports `const fn` with generic trait
+/// bounds.
+macro_rules! maybe_const_trait_bounded_fn {
+ // This case handles both `self` methods (where `self` is by value) and
+ // non-method functions. Each `$args` may optionally be followed by `:
+ // $arg_tys:ty`, which can be omitted for `self`.
+ ($(#[$attr:meta])* $vis:vis const fn $name:ident($($args:ident $(: $arg_tys:ty)?),* $(,)?) $(-> $ret_ty:ty)? $body:block) => {
+ #[cfg(not(no_zerocopy_generic_bounds_in_const_fn_1_61_0))]
+ $(#[$attr])* $vis const fn $name($($args $(: $arg_tys)?),*) $(-> $ret_ty)? $body
+
+ #[cfg(no_zerocopy_generic_bounds_in_const_fn_1_61_0)]
+ $(#[$attr])* $vis fn $name($($args $(: $arg_tys)?),*) $(-> $ret_ty)? $body
+ };
+}
+
+/// Either panic (if the current Rust toolchain supports panicking in `const
+/// fn`) or evaluate a constant that will cause an array indexing error whose
+/// error message will include the format string.
+///
+/// The type that this expression evaluates to must be `Copy`, or else the
+/// non-panicking desugaring will fail to compile.
+macro_rules! const_panic {
+ (@non_panic $($_arg:tt)+) => {{
+ // This will type check to whatever type is expected based on the call
+ // site.
+ let panic: [_; 0] = [];
+ // This will always fail (since we're indexing into an array of size 0.
+ #[allow(unconditional_panic)]
+ panic[0]
+ }};
+ ($($arg:tt)+) => {{
+ #[cfg(not(no_zerocopy_panic_in_const_and_vec_try_reserve_1_57_0))]
+ panic!($($arg)+);
+ #[cfg(no_zerocopy_panic_in_const_and_vec_try_reserve_1_57_0)]
+ const_panic!(@non_panic $($arg)+)
+ }};
+}
+
+/// Either assert (if the current Rust toolchain supports panicking in `const
+/// fn`) or evaluate the expression and, if it evaluates to `false`, call
+/// `const_panic!`. This is used in place of `assert!` in const contexts to
+/// accommodate old toolchains.
+macro_rules! const_assert {
+ ($e:expr) => {{
+ #[cfg(not(no_zerocopy_panic_in_const_and_vec_try_reserve_1_57_0))]
+ assert!($e);
+ #[cfg(no_zerocopy_panic_in_const_and_vec_try_reserve_1_57_0)]
+ {
+ let e = $e;
+ if !e {
+ let _: () = const_panic!(@non_panic concat!("assertion failed: ", stringify!($e)));
+ }
+ }
+ }};
+ ($e:expr, $($args:tt)+) => {{
+ #[cfg(not(no_zerocopy_panic_in_const_and_vec_try_reserve_1_57_0))]
+ assert!($e, $($args)+);
+ #[cfg(no_zerocopy_panic_in_const_and_vec_try_reserve_1_57_0)]
+ {
+ let e = $e;
+ if !e {
+ let _: () = const_panic!(@non_panic concat!("assertion failed: ", stringify!($e), ": ", stringify!($arg)), $($args)*);
+ }
+ }
+ }};
+}
+
+/// Like `const_assert!`, but relative to `debug_assert!`.
+macro_rules! const_debug_assert {
+ ($e:expr $(, $msg:expr)?) => {{
+ #[cfg(not(no_zerocopy_panic_in_const_and_vec_try_reserve_1_57_0))]
+ debug_assert!($e $(, $msg)?);
+ #[cfg(no_zerocopy_panic_in_const_and_vec_try_reserve_1_57_0)]
+ {
+ // Use this (rather than `#[cfg(debug_assertions)]`) to ensure that
+ // `$e` is always compiled even if it will never be evaluated at
+ // runtime.
+ if cfg!(debug_assertions) {
+ let e = $e;
+ if !e {
+ let _: () = const_panic!(@non_panic concat!("assertion failed: ", stringify!($e) $(, ": ", $msg)?));
+ }
+ }
+ }
+ }}
+}
+
+/// Either invoke `unreachable!()` or `loop {}` depending on whether the Rust
+/// toolchain supports panicking in `const fn`.
+macro_rules! const_unreachable {
+ () => {{
+ #[cfg(not(no_zerocopy_panic_in_const_and_vec_try_reserve_1_57_0))]
+ unreachable!();
+
+ #[cfg(no_zerocopy_panic_in_const_and_vec_try_reserve_1_57_0)]
+ loop {}
+ }};
+}
+
+/// Asserts at compile time that `$condition` is true for `Self` or the given
+/// `$tyvar`s. Unlike `const_assert`, this is *strictly* a compile-time check;
+/// it cannot be evaluated in a runtime context. The condition is checked after
+/// monomorphization and, upon failure, emits a compile error.
+macro_rules! static_assert {
+ (Self $(: $(? $optbound:ident $(+)?)* $($bound:ident $(+)?)* )? => $condition:expr $(, $args:tt)*) => {{
+ trait StaticAssert {
+ const ASSERT: bool;
+ }
+
+ impl<T $(: $(? $optbound +)* $($bound +)*)?> StaticAssert for T {
+ const ASSERT: bool = {
+ const_assert!($condition $(, $args)*);
+ $condition
+ };
+ }
+
+ const_assert!(<Self as StaticAssert>::ASSERT);
+ }};
+ ($($tyvar:ident $(: $(? $optbound:ident $(+)?)* $($bound:ident $(+)?)* )?),* => $condition:expr $(, $args:tt)*) => {{
+ trait StaticAssert {
+ const ASSERT: bool;
+ }
+
+ // NOTE: We use `PhantomData` so we can support unsized types.
+ impl<$($tyvar $(: $(? $optbound +)* $($bound +)*)?,)*> StaticAssert for ($(core::marker::PhantomData<$tyvar>,)*) {
+ const ASSERT: bool = {
+ const_assert!($condition $(, $args)*);
+ $condition
+ };
+ }
+
+ const_assert!(<($(core::marker::PhantomData<$tyvar>,)*) as StaticAssert>::ASSERT);
+ }};
+}
+
+/// Assert at compile time that `tyvar` does not have a zero-sized DST
+/// component.
+macro_rules! static_assert_dst_is_not_zst {
+ ($tyvar:ident) => {{
+ use crate::KnownLayout;
+ static_assert!($tyvar: ?Sized + KnownLayout => {
+ let dst_is_zst = match $tyvar::LAYOUT.size_info {
+ crate::SizeInfo::Sized { .. } => false,
+ crate::SizeInfo::SliceDst(TrailingSliceLayout { elem_size, .. }) => {
+ elem_size == 0
+ }
+ };
+ !dst_is_zst
+ }, "cannot call this method on a dynamically-sized type whose trailing slice element is zero-sized");
+ }}
+}
+
+/// Defines a named [`Cast`] implementation.
+///
+/// # Safety
+///
+/// The caller must ensure that, given `src: *mut $src`, `src as *mut $dst` is a
+/// size-preserving or size-shrinking cast.
+///
+/// [`Cast`]: crate::pointer::cast::Cast
+#[macro_export]
+#[doc(hidden)]
+macro_rules! define_cast {
+ // We require the caller to provide an `unsafe` block as part of the input
+ // syntax since a call to `define_cast!` is useless inside of an `unsafe`
+ // block (since it would introduce a type which can't be named outside of
+ // the context of that block).
+ (unsafe { $vis:vis $name:ident $(<$tyvar:ident $(: ?$optbound:ident)?>)? = $src:ty => $dst:ty }) => {
+ #[allow(missing_debug_implementations, missing_copy_implementations, unreachable_pub)]
+ $vis enum $name {}
+
+ // SAFETY: The caller promises that `src as *mut $src` is a size-
+ // preserving or size-shrinking cast. All operations preserve
+ // provenance.
+ unsafe impl $(<$tyvar $(: ?$optbound)?>)? $crate::pointer::cast::Project<$src, $dst> for $name {
+ fn project(src: $crate::pointer::PtrInner<'_, $src>) -> *mut $dst {
+ #[allow(clippy::as_conversions)]
+ return src.as_ptr() as *mut $dst;
+ }
+ }
+
+ // SAFETY: The impl of `Project::project` preserves referent address.
+ unsafe impl $(<$tyvar $(: ?$optbound)?>)? $crate::pointer::cast::Cast<$src, $dst> for $name {}
+ };
+}
+
+/// Implements `TransmuteFrom` and `SizeEq` for `T` and `$wrapper<T>`.
+///
+/// # Safety
+///
+/// `T` and `$wrapper<T>` must have the same bit validity, and must have the
+/// same size in the sense of `CastExact` (specifically, both a
+/// `T`-to-`$wrapper<T>` cast and a `$wrapper<T>`-to-`T` cast must be
+/// size-preserving).
+macro_rules! unsafe_impl_for_transparent_wrapper {
+ ($vis:vis T $(: ?$optbound:ident)? => $wrapper:ident<T>) => {{
+ crate::util::macros::__unsafe();
+
+ use crate::pointer::{TransmuteFrom, cast::{CastExact, TransitiveProject}, SizeEq, invariant::Valid};
+ use crate::wrappers::ReadOnly;
+
+ // SAFETY: The caller promises that `T` and `$wrapper<T>` have the same
+ // bit validity.
+ unsafe impl<T $(: ?$optbound)?> TransmuteFrom<T, Valid, Valid> for $wrapper<T> {}
+ // SAFETY: See previous safety comment.
+ unsafe impl<T $(: ?$optbound)?> TransmuteFrom<$wrapper<T>, Valid, Valid> for T {}
+ // SAFETY: The caller promises that a `T` to `$wrapper<T>` cast is
+ // size-preserving.
+ define_cast!(unsafe { $vis CastToWrapper<T $(: ?$optbound)? > = T => $wrapper<T> });
+ // SAFETY: The caller promises that a `T` to `$wrapper<T>` cast is
+ // size-preserving.
+ unsafe impl<T $(: ?$optbound)?> CastExact<T, $wrapper<T>> for CastToWrapper {}
+ // SAFETY: The caller promises that a `$wrapper<T>` to `T` cast is
+ // size-preserving.
+ define_cast!(unsafe { $vis CastFromWrapper<T $(: ?$optbound)? > = $wrapper<T> => T });
+ // SAFETY: The caller promises that a `$wrapper<T>` to `T` cast is
+ // size-preserving.
+ unsafe impl<T $(: ?$optbound)?> CastExact<$wrapper<T>, T> for CastFromWrapper {}
+
+ impl<T $(: ?$optbound)?> SizeEq<T> for $wrapper<T> {
+ type CastFrom = CastToWrapper;
+ }
+ impl<T $(: ?$optbound)?> SizeEq<$wrapper<T>> for T {
+ type CastFrom = CastFromWrapper;
+ }
+
+ impl<T $(: ?$optbound)?> SizeEq<ReadOnly<T>> for $wrapper<T> {
+ type CastFrom = TransitiveProject<
+ T,
+ <T as SizeEq<ReadOnly<T>>>::CastFrom,
+ CastToWrapper,
+ >;
+ }
+ impl<T $(: ?$optbound)?> SizeEq<$wrapper<T>> for ReadOnly<T> {
+ type CastFrom = TransitiveProject<
+ T,
+ CastFromWrapper,
+ <ReadOnly<T> as SizeEq<T>>::CastFrom,
+ >;
+ }
+
+ impl<T $(: ?$optbound)?> SizeEq<ReadOnly<T>> for ReadOnly<$wrapper<T>> {
+ type CastFrom = TransitiveProject<
+ $wrapper<T>,
+ <$wrapper<T> as SizeEq<ReadOnly<T>>>::CastFrom,
+ <ReadOnly<$wrapper<T>> as SizeEq<$wrapper<T>>>::CastFrom,
+ >;
+ }
+ impl<T $(: ?$optbound)?> SizeEq<ReadOnly<$wrapper<T>>> for ReadOnly<T> {
+ type CastFrom = TransitiveProject<
+ $wrapper<T>,
+ <$wrapper<T> as SizeEq<ReadOnly<$wrapper<T>>>>::CastFrom,
+ <ReadOnly<T> as SizeEq<$wrapper<T>>>::CastFrom,
+ >;
+ }
+ }};
+}
+
+macro_rules! impl_transitive_transmute_from {
+ ($($tyvar:ident $(: ?$optbound:ident)?)? => $t:ty => $u:ty => $v:ty) => {
+ const _: () = {
+ use crate::pointer::{TransmuteFrom, SizeEq, invariant::Valid};
+
+ impl<$($tyvar $(: ?$optbound)?)?> SizeEq<$t> for $v
+ where
+ $u: SizeEq<$t>,
+ $v: SizeEq<$u>,
+ {
+ type CastFrom = cast::TransitiveProject<
+ $u,
+ <$u as SizeEq<$t>>::CastFrom,
+ <$v as SizeEq<$u>>::CastFrom
+ >;
+ }
+
+ // SAFETY: Since `$u: TransmuteFrom<$t, Valid, Valid>`, it is sound
+ // to transmute a bit-valid `$t` to a bit-valid `$u`. Since `$v:
+ // TransmuteFrom<$u, Valid, Valid>`, it is sound to transmute that
+ // bit-valid `$u` to a bit-valid `$v`.
+ unsafe impl<$($tyvar $(: ?$optbound)?)?> TransmuteFrom<$t, Valid, Valid> for $v
+ where
+ $u: TransmuteFrom<$t, Valid, Valid>,
+ $v: TransmuteFrom<$u, Valid, Valid>,
+ {}
+ };
+ };
+}
+
+/// A no-op `unsafe fn` for use in macro expansions.
+///
+/// Calling this function in a macro expansion ensures that the macro's caller
+/// must wrap the call in `unsafe { ... }`.
+#[inline(always)]
+pub(crate) const unsafe fn __unsafe() {}
+
+/// Extracts the contents of doc comments.
+#[allow(unused)]
+macro_rules! docstring {
+ ($(#[doc = $content:expr])*) => {
+ concat!($($content, "\n",)*)
+ }
+}
+
+/// Generate a rustdoc-style header with `$name` as the HTML ID for the 'Code
+/// Generation' section of documentation.
+#[allow(unused)]
+macro_rules! codegen_header {
+ ($level:expr, $name:expr) => {
+ concat!(
+ "
+<",
+ $level,
+ " id='method.",
+ $name,
+ ".codegen'>
+ <a class='doc-anchor' href='#method.",
+ $name,
+ ".codegen'>§</a>
+ Code Generation
+</",
+ $level,
+ ">
+"
+ )
+ };
+}
+
+/// Generates HTML tabs.
+#[rustfmt::skip]
+#[allow(unused)]
+macro_rules! tabs {
+ (
+ name = $name:expr,
+ arity = $arity:literal,
+ $([
+ $($open:ident)?
+ @index $n:literal
+ @title $title:literal
+ $(#[doc = $content:expr])*
+ ]),*
+ ) => {
+ concat!("
+<div class='codegen-tabs' style='--arity: ", $arity ,"'>", $(concat!("
+ <details name='tab-", $name,"' style='--n: ", $n ,"'", $(stringify!($open),)*">
+ <summary><h6>", $title, "</h6></summary>
+ <div>
+
+", $($content, "\n",)* "
+\
+ </div>
+ </details>"),)*
+"</div>")
+ }
+}
+
+/// Generates the HTML for a single benchmark example.
+#[allow(unused)]
+macro_rules! codegen_example {
+ (format = $format:expr, bench = $bench:expr) => {
+ tabs!(
+ name = $bench,
+ arity = 4,
+ [
+ @index 1
+ @title "Format"
+ /// ```ignore
+ #[doc = include_str!(concat!("../benches/formats/", $format, ".rs"))]
+ /// ```
+ ],
+ [
+ @index 2
+ @title "Benchmark"
+ /// ```ignore
+ #[doc = include_str!(concat!("../benches/", $bench, ".rs"))]
+ /// ```
+ ],
+ [
+ open
+ @index 3
+ @title "Assembly"
+ /// ```plain
+ #[doc = include_str!(concat!("../benches/", $bench, ".x86-64"))]
+ /// ```
+ ],
+ [
+ @index 4
+ @title "Machine Code Analysis"
+ /// ```plain
+ #[doc = include_str!(concat!("../benches/", $bench, ".x86-64.mca"))]
+ /// ```
+ ]
+ )
+ }
+}
+
+/// Generate the HTML for a suite of benchmark examples.
+#[allow(unused)]
+macro_rules! codegen_example_suite {
+ (
+ bench = $bench:expr,
+ format = $format:expr,
+ arity = $arity:literal,
+ $([
+ $($open:ident)?
+ @index $index:literal
+ @title $title:literal
+ @variant $variant:literal
+ ]),*
+ ) => {
+ tabs!(
+ name = $bench,
+ arity = $arity,
+ $([
+ $($open)*
+ @index $index
+ @title $title
+ #[doc = codegen_example!(
+ format = concat!($format, "_", $variant),
+ bench = concat!($bench, "_", $variant)
+ )]
+ ]),*
+ )
+ }
+}
+
+/// Generates the string for code generation preamble.
+#[allow(unused)]
+macro_rules! codegen_preamble {
+ () => {
+ docstring!(
+ ///
+ /// This abstraction is safe and cheap, but does not necessarily
+ /// have zero runtime cost. The codegen you experience in practice
+ /// will depend on optimization level, the layout of the destination
+ /// type, and what the compiler can prove about the source.
+ ///
+ )
+ }
+}
+
+/// Stub for rendering codegen documentation; used to break build dependency
+/// between benches and zerocopy when re-blessing codegen tests.
+#[allow(unused)]
+#[cfg(not(doc))]
+macro_rules! codegen_section {
+ (
+ header = $level:expr,
+ bench = $bench:expr,
+ format = $format:expr,
+ arity = $arity:literal,
+ $([
+ $($open:ident)?
+ @index $index:literal
+ @title $title:literal
+ @variant $variant:literal
+ ]),*
+ ) => {
+ ""
+ };
+ (
+ header = $level:expr,
+ bench = $bench:expr,
+ format = $format:expr,
+ ) => {
+ ""
+ };
+}
+
+/// Generates the HTML for code generation documentation.
+#[allow(unused)]
+#[cfg(doc)]
+macro_rules! codegen_section {
+ (
+ header = $level:expr,
+ bench = $bench:expr,
+ format = $format:expr,
+ arity = $arity:literal,
+ $([
+ $($open:ident)?
+ @index $index:literal
+ @title $title:literal
+ @variant $variant:literal
+ ]),*
+ ) => {
+ concat!(
+ codegen_header!($level, $bench),
+ codegen_preamble!(),
+ docstring!(
+ ///
+ /// The below examples illustrate typical codegen for
+ /// increasingly complex types:
+ ///
+ ),
+ codegen_example_suite!(
+ bench = $bench,
+ format = $format,
+ arity = $arity,
+ $([
+ $($open)*
+ @index $index
+ @title $title
+ @variant $variant
+ ]),*
+ )
+ )
+ };
+ (
+ header = $level:expr,
+ bench = $bench:expr,
+ format = $format:expr,
+ ) => {
+ concat!(
+ codegen_header!($level, $bench),
+ codegen_preamble!(),
+ codegen_example!(
+ format = $format,
+ bench = $bench
+ )
+ )
+ }
+}
diff --git a/rust/zerocopy/src/util/mod.rs b/rust/zerocopy/src/util/mod.rs
new file mode 100644
index 0000000..d6d4c6c
--- /dev/null
+++ b/rust/zerocopy/src/util/mod.rs
@@ -0,0 +1,938 @@
+// SPDX-License-Identifier: (BSD-2-Clause OR Apache-2.0) OR MIT
+
+// Copyright 2023 The Fuchsia Authors
+//
+// Licensed under a BSD-style license <LICENSE-BSD>, Apache License, Version 2.0
+// <LICENSE-APACHE or https://www.apache.org/licenses/LICENSE-2.0>, or the MIT
+// license <LICENSE-MIT or https://opensource.org/licenses/MIT>, at your option.
+// This file may not be copied, modified, or distributed except according to
+// those terms.
+
+#[macro_use]
+pub(crate) mod macros;
+
+#[doc(hidden)]
+pub mod macro_util;
+
+use core::{
+ marker::PhantomData,
+ mem::{self, ManuallyDrop},
+ num::NonZeroUsize,
+ ptr::NonNull,
+};
+
+use super::*;
+use crate::pointer::{
+ invariant::{Exclusive, Shared, Valid},
+ SizeEq, TransmuteFromPtr,
+};
+
+/// Like [`PhantomData`], but [`Send`] and [`Sync`] regardless of whether the
+/// wrapped `T` is.
+pub(crate) struct SendSyncPhantomData<T: ?Sized>(PhantomData<T>);
+
+// SAFETY: `SendSyncPhantomData` does not enable any behavior which isn't sound
+// to be called from multiple threads.
+unsafe impl<T: ?Sized> Send for SendSyncPhantomData<T> {}
+// SAFETY: `SendSyncPhantomData` does not enable any behavior which isn't sound
+// to be called from multiple threads.
+unsafe impl<T: ?Sized> Sync for SendSyncPhantomData<T> {}
+
+impl<T: ?Sized> Default for SendSyncPhantomData<T> {
+ fn default() -> SendSyncPhantomData<T> {
+ SendSyncPhantomData(PhantomData)
+ }
+}
+
+impl<T: ?Sized> PartialEq for SendSyncPhantomData<T> {
+ fn eq(&self, _other: &Self) -> bool {
+ true
+ }
+}
+
+impl<T: ?Sized> Eq for SendSyncPhantomData<T> {}
+
+impl<T: ?Sized> Clone for SendSyncPhantomData<T> {
+ fn clone(&self) -> Self {
+ SendSyncPhantomData(PhantomData)
+ }
+}
+
+#[cfg(miri)]
+extern "Rust" {
+ /// Miri-provided intrinsic that marks the pointer `ptr` as aligned to
+ /// `align`.
+ ///
+ /// This intrinsic is used to inform Miri's symbolic alignment checker that
+ /// a pointer is aligned, even if Miri cannot statically deduce that fact.
+ /// This is often required when performing raw pointer arithmetic or casts
+ /// where the alignment is guaranteed by runtime checks or invariants that
+ /// Miri is not aware of.
+ pub(crate) fn miri_promise_symbolic_alignment(ptr: *const (), align: usize);
+}
+
+pub(crate) trait AsAddress {
+ fn addr(self) -> usize;
+}
+
+impl<T: ?Sized> AsAddress for &T {
+ #[inline(always)]
+ fn addr(self) -> usize {
+ let ptr: *const T = self;
+ AsAddress::addr(ptr)
+ }
+}
+
+impl<T: ?Sized> AsAddress for &mut T {
+ #[inline(always)]
+ fn addr(self) -> usize {
+ let ptr: *const T = self;
+ AsAddress::addr(ptr)
+ }
+}
+
+impl<T: ?Sized> AsAddress for NonNull<T> {
+ #[inline(always)]
+ fn addr(self) -> usize {
+ AsAddress::addr(self.as_ptr())
+ }
+}
+
+impl<T: ?Sized> AsAddress for *const T {
+ #[inline(always)]
+ fn addr(self) -> usize {
+ // FIXME(#181), FIXME(https://github.com/rust-lang/rust/issues/95228):
+ // Use `.addr()` instead of `as usize` once it's stable, and get rid of
+ // this `allow`. Currently, `as usize` is the only way to accomplish
+ // this.
+ #[allow(clippy::as_conversions)]
+ #[cfg_attr(
+ __ZEROCOPY_INTERNAL_USE_ONLY_NIGHTLY_FEATURES_IN_TESTS,
+ allow(lossy_provenance_casts)
+ )]
+ return self.cast::<()>() as usize;
+ }
+}
+
+impl<T: ?Sized> AsAddress for *mut T {
+ #[inline(always)]
+ fn addr(self) -> usize {
+ let ptr: *const T = self;
+ AsAddress::addr(ptr)
+ }
+}
+
+/// Validates that `t` is aligned to `align_of::<U>()`.
+#[inline(always)]
+pub(crate) fn validate_aligned_to<T: AsAddress, U>(t: T) -> Result<(), AlignmentError<(), U>> {
+ // `mem::align_of::<U>()` is guaranteed to return a non-zero value, which in
+ // turn guarantees that this mod operation will not panic.
+ #[allow(clippy::arithmetic_side_effects)]
+ let remainder = t.addr() % mem::align_of::<U>();
+ if remainder == 0 {
+ Ok(())
+ } else {
+ // SAFETY: We just confirmed that `t.addr() % align_of::<U>() != 0`.
+ // That's only possible if `align_of::<U>() > 1`.
+ Err(unsafe { AlignmentError::new_unchecked(()) })
+ }
+}
+
+/// Returns the bytes needed to pad `len` to the next multiple of `align`.
+///
+/// This function assumes that align is a power of two; there are no guarantees
+/// on the answer it gives if this is not the case.
+#[cfg_attr(
+ kani,
+ kani::requires(len <= DstLayout::MAX_SIZE),
+ kani::requires(align.is_power_of_two()),
+ kani::ensures(|&p| (len + p) % align.get() == 0),
+ // Ensures that we add the minimum required padding.
+ kani::ensures(|&p| p < align.get()),
+)]
+pub(crate) const fn padding_needed_for(len: usize, align: NonZeroUsize) -> usize {
+ #[cfg(kani)]
+ #[kani::proof_for_contract(padding_needed_for)]
+ fn proof() {
+ padding_needed_for(kani::any(), kani::any());
+ }
+
+ // Abstractly, we want to compute:
+ // align - (len % align).
+ // Handling the case where len%align is 0.
+ // Because align is a power of two, len % align = len & (align-1).
+ // Guaranteed not to underflow as align is nonzero.
+ #[allow(clippy::arithmetic_side_effects)]
+ let mask = align.get() - 1;
+
+ // To efficiently subtract this value from align, we can use the bitwise
+ // complement.
+ // Note that ((!len) & (align-1)) gives us a number that with (len &
+ // (align-1)) sums to align-1. So subtracting 1 from x before taking the
+ // complement subtracts `len` from `align`. Some quick inspection of
+ // cases shows that this also handles the case where `len % align = 0`
+ // correctly too: len-1 % align then equals align-1, so the complement mod
+ // align will be 0, as desired.
+ //
+ // The following reasoning can be verified quickly by an SMT solver
+ // supporting the theory of bitvectors:
+ // ```smtlib
+ // ; Naive implementation of padding
+ // (define-fun padding1 (
+ // (len (_ BitVec 32))
+ // (align (_ BitVec 32))) (_ BitVec 32)
+ // (ite
+ // (= (_ bv0 32) (bvand len (bvsub align (_ bv1 32))))
+ // (_ bv0 32)
+ // (bvsub align (bvand len (bvsub align (_ bv1 32))))))
+ //
+ // ; The implementation below
+ // (define-fun padding2 (
+ // (len (_ BitVec 32))
+ // (align (_ BitVec 32))) (_ BitVec 32)
+ // (bvand (bvnot (bvsub len (_ bv1 32))) (bvsub align (_ bv1 32))))
+ //
+ // (define-fun is-power-of-two ((x (_ BitVec 32))) Bool
+ // (= (_ bv0 32) (bvand x (bvsub x (_ bv1 32)))))
+ //
+ // (declare-const len (_ BitVec 32))
+ // (declare-const align (_ BitVec 32))
+ // ; Search for a case where align is a power of two and padding2 disagrees
+ // ; with padding1
+ // (assert (and (is-power-of-two align)
+ // (not (= (padding1 len align) (padding2 len align)))))
+ // (simplify (padding1 (_ bv300 32) (_ bv32 32))) ; 20
+ // (simplify (padding2 (_ bv300 32) (_ bv32 32))) ; 20
+ // (simplify (padding1 (_ bv322 32) (_ bv32 32))) ; 30
+ // (simplify (padding2 (_ bv322 32) (_ bv32 32))) ; 30
+ // (simplify (padding1 (_ bv8 32) (_ bv8 32))) ; 0
+ // (simplify (padding2 (_ bv8 32) (_ bv8 32))) ; 0
+ // (check-sat) ; unsat, also works for 64-bit bitvectors
+ // ```
+ !(len.wrapping_sub(1)) & mask
+}
+
+/// Rounds `n` down to the largest value `m` such that `m <= n` and `m % align
+/// == 0`.
+///
+/// # Panics
+///
+/// May panic if `align` is not a power of two. Even if it doesn't panic in this
+/// case, it will produce nonsense results.
+#[inline(always)]
+#[cfg_attr(
+ kani,
+ kani::requires(align.is_power_of_two()),
+ kani::ensures(|&m| m <= n && m % align.get() == 0),
+ // Guarantees that `m` is the *largest* value such that `m % align == 0`.
+ kani::ensures(|&m| {
+ // If this `checked_add` fails, then the next multiple would wrap
+ // around, which trivially satisfies the "largest value" requirement.
+ m.checked_add(align.get()).map(|next_mul| next_mul > n).unwrap_or(true)
+ })
+)]
+pub(crate) const fn round_down_to_next_multiple_of_alignment(
+ n: usize,
+ align: NonZeroUsize,
+) -> usize {
+ #[cfg(kani)]
+ #[kani::proof_for_contract(round_down_to_next_multiple_of_alignment)]
+ fn proof() {
+ round_down_to_next_multiple_of_alignment(kani::any(), kani::any());
+ }
+
+ let align = align.get();
+ #[cfg(not(no_zerocopy_panic_in_const_and_vec_try_reserve_1_57_0))]
+ debug_assert!(align.is_power_of_two());
+
+ // Subtraction can't underflow because `align.get() >= 1`.
+ #[allow(clippy::arithmetic_side_effects)]
+ let mask = !(align - 1);
+ n & mask
+}
+
+pub(crate) const fn max(a: NonZeroUsize, b: NonZeroUsize) -> NonZeroUsize {
+ if a.get() < b.get() {
+ b
+ } else {
+ a
+ }
+}
+
+pub(crate) const fn min(a: NonZeroUsize, b: NonZeroUsize) -> NonZeroUsize {
+ if a.get() > b.get() {
+ b
+ } else {
+ a
+ }
+}
+
+/// Copies `src` into the prefix of `dst`.
+///
+/// # Safety
+///
+/// The caller guarantees that `src.len() <= dst.len()`.
+#[inline(always)]
+pub(crate) unsafe fn copy_unchecked(src: &[u8], dst: &mut [u8]) {
+ debug_assert!(src.len() <= dst.len());
+ // SAFETY: This invocation satisfies the safety contract of
+ // copy_nonoverlapping [1]:
+ // - `src.as_ptr()` is trivially valid for reads of `src.len()` bytes
+ // - `dst.as_ptr()` is valid for writes of `src.len()` bytes, because the
+ // caller has promised that `src.len() <= dst.len()`
+ // - `src` and `dst` are, trivially, properly aligned
+ // - the region of memory beginning at `src` with a size of `src.len()`
+ // bytes does not overlap with the region of memory beginning at `dst`
+ // with the same size, because `dst` is derived from an exclusive
+ // reference.
+ unsafe {
+ core::ptr::copy_nonoverlapping(src.as_ptr(), dst.as_mut_ptr(), src.len());
+ };
+}
+
+/// Unsafely transmutes the given `src` into a type `Dst`.
+///
+/// # Safety
+///
+/// The value `src` must be a valid instance of `Dst`.
+#[inline(always)]
+pub(crate) const unsafe fn transmute_unchecked<Src, Dst>(src: Src) -> Dst {
+ static_assert!(Src, Dst => core::mem::size_of::<Src>() == core::mem::size_of::<Dst>());
+
+ #[repr(C)]
+ union Transmute<Src, Dst> {
+ src: ManuallyDrop<Src>,
+ dst: ManuallyDrop<Dst>,
+ }
+
+ // SAFETY: Since `Transmute<Src, Dst>` is `#[repr(C)]`, its `src` and `dst`
+ // fields both start at the same offset and the types of those fields are
+ // transparent wrappers around `Src` and `Dst` [1]. Consequently,
+ // initializing `Transmute` with with `src` and then reading out `dst` is
+ // equivalent to transmuting from `Src` to `Dst` [2]. Transmuting from `src`
+ // to `Dst` is valid because — by contract on the caller — `src` is a valid
+ // instance of `Dst`.
+ //
+ // [1] Per https://doc.rust-lang.org/1.82.0/std/mem/struct.ManuallyDrop.html:
+ //
+ // `ManuallyDrop<T>` is guaranteed to have the same layout and bit
+ // validity as `T`, and is subject to the same layout optimizations as
+ // `T`.
+ //
+ // [2] Per https://doc.rust-lang.org/1.82.0/reference/items/unions.html#reading-and-writing-union-fields:
+ //
+ // Effectively, writing to and then reading from a union with the C
+ // representation is analogous to a transmute from the type used for
+ // writing to the type used for reading.
+ unsafe { ManuallyDrop::into_inner(Transmute { src: ManuallyDrop::new(src) }.dst) }
+}
+
+/// # Safety
+///
+/// `Src` must have a greater or equal alignment to `Dst`.
+pub(crate) unsafe fn transmute_ref<Src, Dst, R>(src: &Src) -> &Dst
+where
+ Src: ?Sized,
+ Dst: SizeEq<Src>
+ + TransmuteFromPtr<Src, Shared, Valid, Valid, <Dst as SizeEq<Src>>::CastFrom, R>
+ + ?Sized,
+{
+ let dst = Ptr::from_ref(src).transmute();
+ // SAFETY: The caller promises that `Src`'s alignment is at least as large
+ // as `Dst`'s alignment.
+ let dst = unsafe { dst.assume_alignment() };
+ dst.as_ref()
+}
+
+/// # Safety
+///
+/// `Src` must have a greater or equal alignment to `Dst`.
+pub(crate) unsafe fn transmute_mut<Src, Dst, R>(src: &mut Src) -> &mut Dst
+where
+ Src: ?Sized,
+ Dst: SizeEq<Src>
+ + TransmuteFromPtr<Src, Exclusive, Valid, Valid, <Dst as SizeEq<Src>>::CastFrom, R>
+ + ?Sized,
+{
+ let dst = Ptr::from_mut(src).transmute();
+ // SAFETY: The caller promises that `Src`'s alignment is at least as large
+ // as `Dst`'s alignment.
+ let dst = unsafe { dst.assume_alignment() };
+ dst.as_mut()
+}
+
+/// Uses `allocate` to create a `Box<T>`.
+///
+/// # Errors
+///
+/// Returns an error on allocation failure. Allocation failure is guaranteed
+/// never to cause a panic or an abort.
+///
+/// # Safety
+///
+/// `allocate` must be either `alloc::alloc::alloc` or
+/// `alloc::alloc::alloc_zeroed`. The referent of the box returned by `new_box`
+/// has the same bit-validity as the referent of the pointer returned by the
+/// given `allocate` and sufficient size to store `T` with `meta`.
+#[must_use = "has no side effects (other than allocation)"]
+#[cfg(feature = "alloc")]
+#[inline]
+pub(crate) unsafe fn new_box<T>(
+ meta: T::PointerMetadata,
+ allocate: unsafe fn(core::alloc::Layout) -> *mut u8,
+) -> Result<alloc::boxed::Box<T>, AllocError>
+where
+ T: ?Sized + crate::KnownLayout,
+{
+ let align = T::LAYOUT.align.get();
+ if !T::is_valid_metadata(meta) {
+ return Err(AllocError);
+ }
+ let size = match T::size_for_metadata(meta) {
+ Some(size) => size,
+ // Thanks to the `!T::is_valid_metadata(meta)` check
+ // above, this branch is unreachable. Fortunately, the
+ // optimizer recognizes this, so replacing this branch
+ // with `unreachable_unchecked` produces no codegen
+ // improvements.
+ None => return Err(AllocError),
+ };
+ let ptr = if size != 0 {
+ // SAFETY:
+ // - `align` is derived from a `NonZeroUsize` and is thus non-zero.
+ // - `align` is a power of two because, by invariant on
+ // `KnownLayout::LAYOUT` `<T as KnownLayout>::LAYOUT` accurately
+ // reflects the layout of `T`.
+ // - `size`, by invariant on `size_for_metadata` is well-aligned for
+ // `align` and, by the check on `T::is_valid_metadata(meta)`, is less
+ // than `isize::MAX`.
+ let layout: Layout = unsafe { Layout::from_size_align_unchecked(size, align) };
+ // SAFETY: By contract on the caller, `allocate` is either
+ // `alloc::alloc::alloc` or `alloc::alloc::alloc_zeroed`. The above
+ // check ensures their shared safety precondition: that the supplied
+ // layout is not zero-sized type [1].
+ //
+ // [1] Per https://doc.rust-lang.org/1.81.0/std/alloc/trait.GlobalAlloc.html#tymethod.alloc:
+ //
+ // This function is unsafe because undefined behavior can result if
+ // the caller does not ensure that layout has non-zero size.
+ let ptr = unsafe { allocate(layout) };
+ match NonNull::new(ptr) {
+ Some(ptr) => ptr,
+ None => return Err(AllocError),
+ }
+ } else {
+ // We use `transmute` instead of an `as` cast since Miri (with strict
+ // provenance enabled) notices and complains that an `as` cast creates a
+ // pointer with no provenance. Miri isn't smart enough to realize that
+ // we're only executing this branch when we're constructing a zero-sized
+ // `Box`, which doesn't require provenance.
+ //
+ // SAFETY: any initialized bit sequence is a bit-valid `*mut u8`. All
+ // bits of a `usize` are initialized.
+ //
+ // `#[allow(unknown_lints)]` is for `integer_to_ptr_transmutes`
+ #[allow(unknown_lints)]
+ #[allow(clippy::useless_transmute, integer_to_ptr_transmutes)]
+ let dangling = unsafe { mem::transmute::<usize, *mut u8>(align) };
+ // SAFETY: `dangling` is constructed from `align`, which is derived from
+ // a `NonZeroUsize`, which is guaranteed to be non-zero.
+ //
+ // `Box<[T]>` does not allocate when `T` is zero-sized or when `len` is
+ // zero, but it does require a non-null dangling pointer for its
+ // allocation.
+ //
+ // FIXME(https://github.com/rust-lang/rust/issues/95228): Use
+ // `std::ptr::without_provenance` once it's stable. That may optimize
+ // better. As written, Rust may assume that this consumes "exposed"
+ // provenance, and thus Rust may have to assume that this may consume
+ // provenance from any pointer whose provenance has been exposed.
+ unsafe { NonNull::new_unchecked(dangling) }
+ };
+
+ let ptr = T::raw_from_ptr_len(ptr, meta);
+
+ // FIXME(#429): Add a "SAFETY" comment and remove this `allow`. Make sure to
+ // include a justification that `ptr.as_ptr()` is validly-aligned in the ZST
+ // case (in which we manually construct a dangling pointer) and to justify
+ // why `Box` is safe to drop (it's because `allocate` uses the system
+ // allocator).
+ #[allow(clippy::undocumented_unsafe_blocks)]
+ Ok(unsafe { alloc::boxed::Box::from_raw(ptr.as_ptr()) })
+}
+
+mod len_of {
+ use super::*;
+
+ /// A witness type for metadata of a valid instance of `&T`.
+ pub struct MetadataOf<T: ?Sized + KnownLayout> {
+ /// # Safety
+ ///
+ /// The size of an instance of `&T` with the given metadata is not
+ /// larger than `isize::MAX`.
+ meta: T::PointerMetadata,
+ _p: PhantomData<T>,
+ }
+
+ impl<T: ?Sized + KnownLayout> Copy for MetadataOf<T> {}
+ impl<T: ?Sized + KnownLayout> Clone for MetadataOf<T> {
+ #[inline]
+ fn clone(&self) -> Self {
+ *self
+ }
+ }
+
+ impl<T: ?Sized + KnownLayout> core::fmt::Debug for MetadataOf<T>
+ where
+ T::PointerMetadata: core::fmt::Debug,
+ {
+ #[inline]
+ fn fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result {
+ f.debug_struct("MetadataOf").field("meta", &self.meta).finish()
+ }
+ }
+
+ impl<T: ?Sized> MetadataOf<T>
+ where
+ T: KnownLayout,
+ {
+ /// Returns `None` if `meta` is greater than `t`'s metadata.
+ #[inline(always)]
+ pub(crate) fn new_in_bounds(t: &T, meta: usize) -> Option<Self>
+ where
+ T: KnownLayout<PointerMetadata = usize>,
+ {
+ if meta <= Ptr::from_ref(t).len() {
+ // SAFETY: We have checked that `meta` is not greater than `t`'s
+ // metadata, which, by invariant on `&T`, addresses no more than
+ // `isize::MAX` bytes [1][2].
+ //
+ // [1] Per https://doc.rust-lang.org/1.85.0/std/primitive.reference.html#safety:
+ //
+ // For all types, `T: ?Sized`, and for all `t: &T` or `t:
+ // &mut T`, when such values cross an API boundary, the
+ // following invariants must generally be upheld:
+ //
+ // * `t` is non-null
+ // * `t` is aligned to `align_of_val(t)`
+ // * if `size_of_val(t) > 0`, then `t` is dereferenceable for
+ // `size_of_val(t)` many bytes
+ //
+ // If `t` points at address `a`, being "dereferenceable" for
+ // N bytes means that the memory range `[a, a + N)` is all
+ // contained within a single allocated object.
+ //
+ // [2] Per https://doc.rust-lang.org/1.85.0/std/ptr/index.html#allocated-object:
+ //
+ // For any allocated object with `base` address, `size`, and
+ // a set of `addresses`, the following are guaranteed:
+ // - For all addresses `a` in `addresses`, `a` is in the
+ // range `base .. (base + size)` (note that this requires
+ // `a < base + size`, not `a <= base + size`)
+ // - `base` is not equal to [`null()`] (i.e., the address
+ // with the numerical value 0)
+ // - `base + size <= usize::MAX`
+ // - `size <= isize::MAX`
+ Some(unsafe { Self::new_unchecked(meta) })
+ } else {
+ None
+ }
+ }
+
+ /// # Safety
+ ///
+ /// The size of an instance of `&T` with the given metadata is not
+ /// larger than `isize::MAX`.
+ pub(crate) unsafe fn new_unchecked(meta: T::PointerMetadata) -> Self {
+ // SAFETY: The caller has promised that the size of an instance of
+ // `&T` with the given metadata is not larger than `isize::MAX`.
+ Self { meta, _p: PhantomData }
+ }
+
+ pub(crate) fn get(&self) -> T::PointerMetadata
+ where
+ T::PointerMetadata: Copy,
+ {
+ self.meta
+ }
+
+ #[inline]
+ pub(crate) fn padding_needed_for(&self) -> usize
+ where
+ T: KnownLayout<PointerMetadata = usize>,
+ {
+ let trailing_slice_layout = crate::trailing_slice_layout::<T>();
+
+ // FIXME(#67): Remove this allow. See NumExt for more details.
+ #[allow(
+ unstable_name_collisions,
+ clippy::incompatible_msrv,
+ clippy::multiple_unsafe_ops_per_block
+ )]
+ // SAFETY: By invariant on `self`, a `&T` with metadata `self.meta`
+ // describes an object of size `<= isize::MAX`. This computes the
+ // size of such a `&T` without any trailing padding, and so neither
+ // the multiplication nor the addition will overflow.
+ let unpadded_size = unsafe {
+ let trailing_size = self.meta.unchecked_mul(trailing_slice_layout.elem_size);
+ trailing_size.unchecked_add(trailing_slice_layout.offset)
+ };
+
+ util::padding_needed_for(unpadded_size, T::LAYOUT.align)
+ }
+
+ #[inline(always)]
+ pub(crate) fn validate_cast_and_convert_metadata(
+ addr: usize,
+ bytes_len: MetadataOf<[u8]>,
+ cast_type: CastType,
+ meta: Option<T::PointerMetadata>,
+ ) -> Result<(MetadataOf<T>, MetadataOf<[u8]>), MetadataCastError> {
+ let layout = match meta {
+ None => T::LAYOUT,
+ // This can return `Err(MetadataCastError::Size)` if the
+ // metadata describes an object which can't fit in an `isize`.
+ Some(meta) => {
+ if !T::is_valid_metadata(meta) {
+ return Err(MetadataCastError::Size);
+ }
+ let size = match T::size_for_metadata(meta) {
+ Some(size) => size,
+ // Thanks to the `!T::is_valid_metadata(meta)` check
+ // above, this branch is unreachable. Fortunately, the
+ // optimizer recognizes this, so replacing this branch
+ // with `unreachable_unchecked` produces no codegen
+ // improvements.
+ None => return Err(MetadataCastError::Size),
+ };
+ DstLayout {
+ align: T::LAYOUT.align,
+ size_info: crate::SizeInfo::Sized { size },
+ statically_shallow_unpadded: false,
+ }
+ }
+ };
+ // Lemma 0: By contract on `validate_cast_and_convert_metadata`, if
+ // the result is `Ok(..)`, then a `&T` with `elems` trailing slice
+ // elements is no larger in size than `bytes_len.get()`.
+ let (elems, split_at) =
+ layout.validate_cast_and_convert_metadata(addr, bytes_len.get(), cast_type)?;
+ let elems = T::PointerMetadata::from_elem_count(elems);
+
+ // For a slice DST type, if `meta` is `Some(elems)`, then we
+ // synthesize `layout` to describe a sized type whose size is equal
+ // to the size of the instance that we are asked to cast. For sized
+ // types, `validate_cast_and_convert_metadata` returns `elems == 0`.
+ // Thus, in this case, we need to use the `elems` passed by the
+ // caller, not the one returned by
+ // `validate_cast_and_convert_metadata`.
+ //
+ // Lemma 1: A `&T` with `elems` trailing slice elements is no larger
+ // in size than `bytes_len.get()`. Proof:
+ // - If `meta` is `None`, then `elems` satisfies this condition by
+ // Lemma 0.
+ // - If `meta` is `Some(meta)`, then `layout` describes an object
+ // whose size is equal to the size of an `&T` with `meta`
+ // metadata. By Lemma 0, that size is not larger than
+ // `bytes_len.get()`.
+ //
+ // Lemma 2: A `&T` with `elems` trailing slice elements is no larger
+ // than `isize::MAX` bytes. Proof: By Lemma 1, a `&T` with metadata
+ // `elems` is not larger in size than `bytes_len.get()`. By
+ // invariant on `MetadataOf<[u8]>`, a `&[u8]` with metadata
+ // `bytes_len` is not larger than `isize::MAX`. Because
+ // `size_of::<u8>()` is `1`, a `&[u8]` with metadata `bytes_len` has
+ // size `bytes_len.get()` bytes. Therefore, a `&T` with metadata
+ // `elems` has size not larger than `isize::MAX`.
+ let elems = meta.unwrap_or(elems);
+
+ // SAFETY: See Lemma 2.
+ let elems = unsafe { MetadataOf::new_unchecked(elems) };
+
+ // SAFETY: Let `size` be the size of a `&T` with metadata `elems`.
+ // By post-condition on `validate_cast_and_convert_metadata`, one of
+ // the following conditions holds:
+ // - `split_at == size`, in which case, by Lemma 2, `split_at <=
+ // isize::MAX`. Since `size_of::<u8>() == 1`, a `[u8]` with
+ // `split_at` elems has size not larger than `isize::MAX`.
+ // - `split_at == bytes_len - size`. Since `bytes_len:
+ // MetadataOf<u8>`, and since `size` is non-negative, `split_at`
+ // addresses no more bytes than `bytes_len` does. Since
+ // `bytes_len: MetadataOf<u8>`, `bytes_len` describes a `[u8]`
+ // which has no more than `isize::MAX` bytes, and thus so does
+ // `split_at`.
+ let split_at = unsafe { MetadataOf::<[u8]>::new_unchecked(split_at) };
+ Ok((elems, split_at))
+ }
+ }
+}
+
+pub use len_of::MetadataOf;
+
+/// Since we support multiple versions of Rust, there are often features which
+/// have been stabilized in the most recent stable release which do not yet
+/// exist (stably) on our MSRV. This module provides polyfills for those
+/// features so that we can write more "modern" code, and just remove the
+/// polyfill once our MSRV supports the corresponding feature. Without this,
+/// we'd have to write worse/more verbose code and leave FIXME comments
+/// sprinkled throughout the codebase to update to the new pattern once it's
+/// stabilized.
+///
+/// Each trait is imported as `_` at the crate root; each polyfill should "just
+/// work" at usage sites.
+pub(crate) mod polyfills {
+ use core::ptr::{self, NonNull};
+
+ // A polyfill for `NonNull::slice_from_raw_parts` that we can use before our
+ // MSRV is 1.70, when that function was stabilized.
+ //
+ // The `#[allow(unused)]` is necessary because, on sufficiently recent
+ // toolchain versions, `ptr.slice_from_raw_parts()` resolves to the inherent
+ // method rather than to this trait, and so this trait is considered unused.
+ //
+ // FIXME(#67): Once our MSRV is 1.70, remove this.
+ #[allow(unused)]
+ pub(crate) trait NonNullExt<T> {
+ fn slice_from_raw_parts(data: Self, len: usize) -> NonNull<[T]>;
+ }
+
+ impl<T> NonNullExt<T> for NonNull<T> {
+ // NOTE on coverage: this will never be tested in nightly since it's a
+ // polyfill for a feature which has been stabilized on our nightly
+ // toolchain.
+ #[cfg_attr(
+ all(coverage_nightly, __ZEROCOPY_INTERNAL_USE_ONLY_NIGHTLY_FEATURES_IN_TESTS),
+ coverage(off)
+ )]
+ #[inline(always)]
+ fn slice_from_raw_parts(data: Self, len: usize) -> NonNull<[T]> {
+ let ptr = ptr::slice_from_raw_parts_mut(data.as_ptr(), len);
+ // SAFETY: `ptr` is converted from `data`, which is non-null.
+ unsafe { NonNull::new_unchecked(ptr) }
+ }
+ }
+
+ // A polyfill for `Self::unchecked_sub` that we can use until methods like
+ // `usize::unchecked_sub` is stabilized.
+ //
+ // The `#[allow(unused)]` is necessary because, on sufficiently recent
+ // toolchain versions, `ptr.slice_from_raw_parts()` resolves to the inherent
+ // method rather than to this trait, and so this trait is considered unused.
+ //
+ // FIXME(#67): Once our MSRV is high enough, remove this.
+ #[allow(unused)]
+ pub(crate) trait NumExt {
+ /// Add without checking for overflow.
+ ///
+ /// # Safety
+ ///
+ /// The caller promises that the addition will not overflow.
+ unsafe fn unchecked_add(self, rhs: Self) -> Self;
+
+ /// Subtract without checking for underflow.
+ ///
+ /// # Safety
+ ///
+ /// The caller promises that the subtraction will not underflow.
+ unsafe fn unchecked_sub(self, rhs: Self) -> Self;
+
+ /// Multiply without checking for overflow.
+ ///
+ /// # Safety
+ ///
+ /// The caller promises that the multiplication will not overflow.
+ unsafe fn unchecked_mul(self, rhs: Self) -> Self;
+ }
+
+ // NOTE on coverage: these will never be tested in nightly since they're
+ // polyfills for a feature which has been stabilized on our nightly
+ // toolchain.
+ impl NumExt for usize {
+ #[cfg_attr(
+ all(coverage_nightly, __ZEROCOPY_INTERNAL_USE_ONLY_NIGHTLY_FEATURES_IN_TESTS),
+ coverage(off)
+ )]
+ #[inline(always)]
+ unsafe fn unchecked_add(self, rhs: usize) -> usize {
+ match self.checked_add(rhs) {
+ Some(x) => x,
+ None => {
+ // SAFETY: The caller promises that the addition will not
+ // underflow.
+ unsafe { core::hint::unreachable_unchecked() }
+ }
+ }
+ }
+
+ #[cfg_attr(
+ all(coverage_nightly, __ZEROCOPY_INTERNAL_USE_ONLY_NIGHTLY_FEATURES_IN_TESTS),
+ coverage(off)
+ )]
+ #[inline(always)]
+ unsafe fn unchecked_sub(self, rhs: usize) -> usize {
+ match self.checked_sub(rhs) {
+ Some(x) => x,
+ None => {
+ // SAFETY: The caller promises that the subtraction will not
+ // underflow.
+ unsafe { core::hint::unreachable_unchecked() }
+ }
+ }
+ }
+
+ #[cfg_attr(
+ all(coverage_nightly, __ZEROCOPY_INTERNAL_USE_ONLY_NIGHTLY_FEATURES_IN_TESTS),
+ coverage(off)
+ )]
+ #[inline(always)]
+ unsafe fn unchecked_mul(self, rhs: usize) -> usize {
+ match self.checked_mul(rhs) {
+ Some(x) => x,
+ None => {
+ // SAFETY: The caller promises that the multiplication will
+ // not overflow.
+ unsafe { core::hint::unreachable_unchecked() }
+ }
+ }
+ }
+ }
+}
+
+#[cfg(test)]
+pub(crate) mod testutil {
+ use crate::*;
+
+ /// A `T` which is aligned to at least `align_of::<A>()`.
+ #[derive(Default)]
+ pub(crate) struct Align<T, A> {
+ pub(crate) t: T,
+ _a: [A; 0],
+ }
+
+ impl<T: Default, A> Align<T, A> {
+ pub(crate) fn set_default(&mut self) {
+ self.t = T::default();
+ }
+ }
+
+ impl<T, A> Align<T, A> {
+ pub(crate) const fn new(t: T) -> Align<T, A> {
+ Align { t, _a: [] }
+ }
+ }
+
+ /// A `T` which is guaranteed not to satisfy `align_of::<A>()`.
+ ///
+ /// It must be the case that `align_of::<T>() < align_of::<A>()` in order
+ /// for this type to work properly.
+ #[repr(C)]
+ pub(crate) struct ForceUnalign<T: Unaligned, A> {
+ // The outer struct is aligned to `A`, and, thanks to `repr(C)`, `t` is
+ // placed at the minimum offset that guarantees its alignment. If
+ // `align_of::<T>() < align_of::<A>()`, then that offset will be
+ // guaranteed *not* to satisfy `align_of::<A>()`.
+ //
+ // Note that we need `T: Unaligned` in order to guarantee that there is
+ // no padding between `_u` and `t`.
+ _u: u8,
+ pub(crate) t: T,
+ _a: [A; 0],
+ }
+
+ impl<T: Unaligned, A> ForceUnalign<T, A> {
+ pub(crate) fn new(t: T) -> ForceUnalign<T, A> {
+ ForceUnalign { _u: 0, t, _a: [] }
+ }
+ }
+ // A `u64` with alignment 8.
+ //
+ // Though `u64` has alignment 8 on some platforms, it's not guaranteed. By
+ // contrast, `AU64` is guaranteed to have alignment 8 on all platforms.
+ #[derive(
+ KnownLayout,
+ Immutable,
+ FromBytes,
+ IntoBytes,
+ Eq,
+ PartialEq,
+ Ord,
+ PartialOrd,
+ Default,
+ Debug,
+ Copy,
+ Clone,
+ )]
+ #[repr(C, align(8))]
+ pub(crate) struct AU64(pub(crate) u64);
+
+ impl AU64 {
+ // Converts this `AU64` to bytes using this platform's endianness.
+ pub(crate) fn to_bytes(self) -> [u8; 8] {
+ crate::transmute!(self)
+ }
+ }
+
+ impl Display for AU64 {
+ #[cfg_attr(
+ all(coverage_nightly, __ZEROCOPY_INTERNAL_USE_ONLY_NIGHTLY_FEATURES_IN_TESTS),
+ coverage(off)
+ )]
+ fn fmt(&self, f: &mut Formatter<'_>) -> fmt::Result {
+ Display::fmt(&self.0, f)
+ }
+ }
+}
+
+#[cfg(test)]
+mod tests {
+ use super::*;
+
+ #[test]
+ fn test_round_down_to_next_multiple_of_alignment() {
+ fn alt_impl(n: usize, align: NonZeroUsize) -> usize {
+ let mul = n / align.get();
+ mul * align.get()
+ }
+
+ for align in [1, 2, 4, 8, 16] {
+ for n in 0..256 {
+ let align = NonZeroUsize::new(align).unwrap();
+ let want = alt_impl(n, align);
+ let got = round_down_to_next_multiple_of_alignment(n, align);
+ assert_eq!(got, want, "round_down_to_next_multiple_of_alignment({}, {})", n, align);
+ }
+ }
+ }
+
+ #[rustversion::since(1.57.0)]
+ #[test]
+ #[should_panic]
+ fn test_round_down_to_next_multiple_of_alignment_zerocopy_panic_in_const_and_vec_try_reserve() {
+ round_down_to_next_multiple_of_alignment(0, NonZeroUsize::new(3).unwrap());
+ }
+ #[test]
+ fn test_send_sync_phantom_data() {
+ let x = SendSyncPhantomData::<u8>::default();
+ let y = x.clone();
+ assert!(x == y);
+ assert!(x == SendSyncPhantomData::<u8>::default());
+ }
+
+ #[test]
+ #[allow(clippy::as_conversions)]
+ fn test_as_address() {
+ let x = 0u8;
+ let r = &x;
+ let mut x_mut = 0u8;
+ let rm = &mut x_mut;
+ let p = r as *const u8;
+ let pm = rm as *mut u8;
+ let nn = NonNull::new(p as *mut u8).unwrap();
+
+ assert_eq!(AsAddress::addr(r), p as usize);
+ assert_eq!(AsAddress::addr(rm), pm as usize);
+ assert_eq!(AsAddress::addr(p), p as usize);
+ assert_eq!(AsAddress::addr(pm), pm as usize);
+ assert_eq!(AsAddress::addr(nn), p as usize);
+ }
+}
diff --git a/rust/zerocopy/src/wrappers.rs b/rust/zerocopy/src/wrappers.rs
new file mode 100644
index 0000000..266aec2
--- /dev/null
+++ b/rust/zerocopy/src/wrappers.rs
@@ -0,0 +1,1034 @@
+// SPDX-License-Identifier: (BSD-2-Clause OR Apache-2.0) OR MIT
+
+// Copyright 2023 The Fuchsia Authors
+//
+// Licensed under a BSD-style license <LICENSE-BSD>, Apache License, Version 2.0
+// <LICENSE-APACHE or https://www.apache.org/licenses/LICENSE-2.0>, or the MIT
+// license <LICENSE-MIT or https://opensource.org/licenses/MIT>, at your option.
+// This file may not be copied, modified, or distributed except according to
+// those terms.
+
+use core::{fmt, hash::Hash};
+
+use super::*;
+use crate::pointer::{invariant::Valid, SizeEq, TransmuteFrom};
+
+/// A type with no alignment requirement.
+///
+/// An `Unalign` wraps a `T`, removing any alignment requirement. `Unalign<T>`
+/// has the same size and bit validity as `T`, but not necessarily the same
+/// alignment [or ABI]. This is useful if a type with an alignment requirement
+/// needs to be read from a chunk of memory which provides no alignment
+/// guarantees.
+///
+/// Since `Unalign` has no alignment requirement, the inner `T` may not be
+/// properly aligned in memory. There are five ways to access the inner `T`:
+/// - by value, using [`get`] or [`into_inner`]
+/// - by reference inside of a callback, using [`update`]
+/// - fallibly by reference, using [`try_deref`] or [`try_deref_mut`]; these can
+/// fail if the `Unalign` does not satisfy `T`'s alignment requirement at
+/// runtime
+/// - unsafely by reference, using [`deref_unchecked`] or
+/// [`deref_mut_unchecked`]; it is the caller's responsibility to ensure that
+/// the `Unalign` satisfies `T`'s alignment requirement
+/// - (where `T: Unaligned`) infallibly by reference, using [`Deref::deref`] or
+/// [`DerefMut::deref_mut`]
+///
+/// [or ABI]: https://github.com/google/zerocopy/issues/164
+/// [`get`]: Unalign::get
+/// [`into_inner`]: Unalign::into_inner
+/// [`update`]: Unalign::update
+/// [`try_deref`]: Unalign::try_deref
+/// [`try_deref_mut`]: Unalign::try_deref_mut
+/// [`deref_unchecked`]: Unalign::deref_unchecked
+/// [`deref_mut_unchecked`]: Unalign::deref_mut_unchecked
+///
+/// # Example
+///
+/// In this example, we need `EthernetFrame` to have no alignment requirement -
+/// and thus implement [`Unaligned`]. `EtherType` is `#[repr(u16)]` and so
+/// cannot implement `Unaligned`. We use `Unalign` to relax `EtherType`'s
+/// alignment requirement so that `EthernetFrame` has no alignment requirement
+/// and can implement `Unaligned`.
+///
+/// ```rust
+/// use zerocopy::*;
+/// # use zerocopy_derive::*;
+/// # #[derive(FromBytes, KnownLayout, Immutable, Unaligned)] #[repr(C)] struct Mac([u8; 6]);
+///
+/// # #[derive(PartialEq, Copy, Clone, Debug)]
+/// #[derive(TryFromBytes, KnownLayout, Immutable)]
+/// #[repr(u16)]
+/// enum EtherType {
+/// Ipv4 = 0x0800u16.to_be(),
+/// Arp = 0x0806u16.to_be(),
+/// Ipv6 = 0x86DDu16.to_be(),
+/// # /*
+/// ...
+/// # */
+/// }
+///
+/// #[derive(TryFromBytes, KnownLayout, Immutable, Unaligned)]
+/// #[repr(C)]
+/// struct EthernetFrame {
+/// src: Mac,
+/// dst: Mac,
+/// ethertype: Unalign<EtherType>,
+/// payload: [u8],
+/// }
+///
+/// let bytes = &[
+/// # 0, 1, 2, 3, 4, 5,
+/// # 6, 7, 8, 9, 10, 11,
+/// # /*
+/// ...
+/// # */
+/// 0x86, 0xDD, // EtherType
+/// 0xDE, 0xAD, 0xBE, 0xEF // Payload
+/// ][..];
+///
+/// // PANICS: Guaranteed not to panic because `bytes` is of the right
+/// // length, has the right contents, and `EthernetFrame` has no
+/// // alignment requirement.
+/// let packet = EthernetFrame::try_ref_from_bytes(&bytes).unwrap();
+///
+/// assert_eq!(packet.ethertype.get(), EtherType::Ipv6);
+/// assert_eq!(packet.payload, [0xDE, 0xAD, 0xBE, 0xEF]);
+/// ```
+///
+/// # Safety
+///
+/// `Unalign<T>` is guaranteed to have the same size and bit validity as `T`,
+/// and to have [`UnsafeCell`]s covering the same byte ranges as `T`.
+/// `Unalign<T>` is guaranteed to have alignment 1.
+// NOTE: This type is sound to use with types that need to be dropped. The
+// reason is that the compiler-generated drop code automatically moves all
+// values to aligned memory slots before dropping them in-place. This is not
+// well-documented, but it's hinted at in places like [1] and [2]. However, this
+// also means that `T` must be `Sized`; unless something changes, we can never
+// support unsized `T`. [3]
+//
+// [1] https://github.com/rust-lang/rust/issues/54148#issuecomment-420529646
+// [2] https://github.com/google/zerocopy/pull/126#discussion_r1018512323
+// [3] https://github.com/google/zerocopy/issues/209
+#[allow(missing_debug_implementations)]
+#[derive(Default, Copy)]
+#[cfg_attr(any(feature = "derive", test), derive(Immutable, FromBytes, IntoBytes, Unaligned))]
+#[repr(C, packed)]
+pub struct Unalign<T>(T);
+
+// We do not use `derive(KnownLayout)` on `Unalign`, because the derive is not
+// smart enough to realize that `Unalign<T>` is always sized and thus emits a
+// `KnownLayout` impl bounded on `T: KnownLayout.` This is overly restrictive.
+impl_known_layout!(T => Unalign<T>);
+
+// FIXME(https://github.com/rust-lang/rust-clippy/issues/16087): Move these
+// attributes below the comment once this Clippy bug is fixed.
+#[cfg_attr(
+ all(__ZEROCOPY_INTERNAL_USE_ONLY_NIGHTLY_FEATURES_IN_TESTS, any(feature = "derive", test)),
+ expect(unused_unsafe)
+)]
+#[cfg_attr(
+ all(
+ not(__ZEROCOPY_INTERNAL_USE_ONLY_NIGHTLY_FEATURES_IN_TESTS),
+ any(feature = "derive", test)
+ ),
+ allow(unused_unsafe)
+)]
+// SAFETY:
+// - `Unalign<T>` promises to have alignment 1, and so we don't require that `T:
+// Unaligned`.
+// - `Unalign<T>` has the same bit validity as `T`, and so it is `FromZeros`,
+// `FromBytes`, or `IntoBytes` exactly when `T` is as well.
+// - `Immutable`: `Unalign<T>` has the same fields as `T`, so it permits
+// interior mutation exactly when `T` does.
+// - `TryFromBytes`: `Unalign<T>` has the same the same bit validity as `T`, so
+// `T::is_bit_valid` is a sound implementation of `is_bit_valid`.
+//
+#[allow(clippy::multiple_unsafe_ops_per_block)]
+const _: () = unsafe {
+ impl_or_verify!(T => Unaligned for Unalign<T>);
+ impl_or_verify!(T: Immutable => Immutable for Unalign<T>);
+ impl_or_verify!(
+ T: TryFromBytes => TryFromBytes for Unalign<T>;
+ |c| T::is_bit_valid(c.transmute::<_, _, BecauseImmutable>())
+ );
+ impl_or_verify!(T: FromZeros => FromZeros for Unalign<T>);
+ impl_or_verify!(T: FromBytes => FromBytes for Unalign<T>);
+ impl_or_verify!(T: IntoBytes => IntoBytes for Unalign<T>);
+};
+
+// Note that `Unalign: Clone` only if `T: Copy`. Since the inner `T` may not be
+// aligned, there's no way to safely call `T::clone`, and so a `T: Clone` bound
+// is not sufficient to implement `Clone` for `Unalign`.
+impl<T: Copy> Clone for Unalign<T> {
+ #[inline(always)]
+ fn clone(&self) -> Unalign<T> {
+ *self
+ }
+}
+
+impl<T> Unalign<T> {
+ /// Constructs a new `Unalign`.
+ #[inline(always)]
+ pub const fn new(val: T) -> Unalign<T> {
+ Unalign(val)
+ }
+
+ /// Consumes `self`, returning the inner `T`.
+ #[inline(always)]
+ pub const fn into_inner(self) -> T {
+ // SAFETY: Since `Unalign` is `#[repr(C, packed)]`, it has the same size
+ // and bit validity as `T`.
+ //
+ // We do this instead of just destructuring in order to prevent
+ // `Unalign`'s `Drop::drop` from being run, since dropping is not
+ // supported in `const fn`s.
+ //
+ // FIXME(https://github.com/rust-lang/rust/issues/73255): Destructure
+ // instead of using unsafe.
+ unsafe { crate::util::transmute_unchecked(self) }
+ }
+
+ /// Attempts to return a reference to the wrapped `T`, failing if `self` is
+ /// not properly aligned.
+ ///
+ /// If `self` does not satisfy `align_of::<T>()`, then `try_deref` returns
+ /// `Err`.
+ ///
+ /// If `T: Unaligned`, then `Unalign<T>` implements [`Deref`], and callers
+ /// may prefer [`Deref::deref`], which is infallible.
+ #[inline(always)]
+ pub fn try_deref(&self) -> Result<&T, AlignmentError<&Self, T>> {
+ let inner = Ptr::from_ref(self).transmute();
+ match inner.try_into_aligned() {
+ Ok(aligned) => Ok(aligned.as_ref()),
+ Err(err) => Err(err.map_src(
+ #[inline(always)]
+ |src| src.into_unalign().as_ref(),
+ )),
+ }
+ }
+
+ /// Attempts to return a mutable reference to the wrapped `T`, failing if
+ /// `self` is not properly aligned.
+ ///
+ /// If `self` does not satisfy `align_of::<T>()`, then `try_deref` returns
+ /// `Err`.
+ ///
+ /// If `T: Unaligned`, then `Unalign<T>` implements [`DerefMut`], and
+ /// callers may prefer [`DerefMut::deref_mut`], which is infallible.
+ #[inline(always)]
+ pub fn try_deref_mut(&mut self) -> Result<&mut T, AlignmentError<&mut Self, T>> {
+ let inner = Ptr::from_mut(self).transmute::<_, _, (_, (_, _))>();
+ match inner.try_into_aligned() {
+ Ok(aligned) => Ok(aligned.as_mut()),
+ Err(err) => Err(err.map_src(|src| src.into_unalign().as_mut())),
+ }
+ }
+
+ /// Returns a reference to the wrapped `T` without checking alignment.
+ ///
+ /// If `T: Unaligned`, then `Unalign<T>` implements[ `Deref`], and callers
+ /// may prefer [`Deref::deref`], which is safe.
+ ///
+ /// # Safety
+ ///
+ /// The caller must guarantee that `self` satisfies `align_of::<T>()`.
+ #[inline(always)]
+ pub const unsafe fn deref_unchecked(&self) -> &T {
+ // SAFETY: `Unalign<T>` is `repr(transparent)`, so there is a valid `T`
+ // at the same memory location as `self`. It has no alignment guarantee,
+ // but the caller has promised that `self` is properly aligned, so we
+ // know that it is sound to create a reference to `T` at this memory
+ // location.
+ //
+ // We use `mem::transmute` instead of `&*self.get_ptr()` because
+ // dereferencing pointers is not stable in `const` on our current MSRV
+ // (1.56 as of this writing).
+ unsafe { mem::transmute(self) }
+ }
+
+ /// Returns a mutable reference to the wrapped `T` without checking
+ /// alignment.
+ ///
+ /// If `T: Unaligned`, then `Unalign<T>` implements[ `DerefMut`], and
+ /// callers may prefer [`DerefMut::deref_mut`], which is safe.
+ ///
+ /// # Safety
+ ///
+ /// The caller must guarantee that `self` satisfies `align_of::<T>()`.
+ #[inline(always)]
+ pub unsafe fn deref_mut_unchecked(&mut self) -> &mut T {
+ // SAFETY: `self.get_mut_ptr()` returns a raw pointer to a valid `T` at
+ // the same memory location as `self`. It has no alignment guarantee,
+ // but the caller has promised that `self` is properly aligned, so we
+ // know that the pointer itself is aligned, and thus that it is sound to
+ // create a reference to a `T` at this memory location.
+ unsafe { &mut *self.get_mut_ptr() }
+ }
+
+ /// Gets an unaligned raw pointer to the inner `T`.
+ ///
+ /// # Safety
+ ///
+ /// The returned raw pointer is not necessarily aligned to
+ /// `align_of::<T>()`. Most functions which operate on raw pointers require
+ /// those pointers to be aligned, so calling those functions with the result
+ /// of `get_ptr` will result in undefined behavior if alignment is not
+ /// guaranteed using some out-of-band mechanism. In general, the only
+ /// functions which are safe to call with this pointer are those which are
+ /// explicitly documented as being sound to use with an unaligned pointer,
+ /// such as [`read_unaligned`].
+ ///
+ /// Even if the caller is permitted to mutate `self` (e.g. they have
+ /// ownership or a mutable borrow), it is not guaranteed to be sound to
+ /// write through the returned pointer. If writing is required, prefer
+ /// [`get_mut_ptr`] instead.
+ ///
+ /// [`read_unaligned`]: core::ptr::read_unaligned
+ /// [`get_mut_ptr`]: Unalign::get_mut_ptr
+ #[inline(always)]
+ pub const fn get_ptr(&self) -> *const T {
+ ptr::addr_of!(self.0)
+ }
+
+ /// Gets an unaligned mutable raw pointer to the inner `T`.
+ ///
+ /// # Safety
+ ///
+ /// The returned raw pointer is not necessarily aligned to
+ /// `align_of::<T>()`. Most functions which operate on raw pointers require
+ /// those pointers to be aligned, so calling those functions with the result
+ /// of `get_ptr` will result in undefined behavior if alignment is not
+ /// guaranteed using some out-of-band mechanism. In general, the only
+ /// functions which are safe to call with this pointer are those which are
+ /// explicitly documented as being sound to use with an unaligned pointer,
+ /// such as [`read_unaligned`].
+ ///
+ /// [`read_unaligned`]: core::ptr::read_unaligned
+ // FIXME(https://github.com/rust-lang/rust/issues/57349): Make this `const`.
+ #[inline(always)]
+ pub fn get_mut_ptr(&mut self) -> *mut T {
+ ptr::addr_of_mut!(self.0)
+ }
+
+ /// Sets the inner `T`, dropping the previous value.
+ // FIXME(https://github.com/rust-lang/rust/issues/57349): Make this `const`.
+ #[inline(always)]
+ pub fn set(&mut self, t: T) {
+ *self = Unalign::new(t);
+ }
+
+ /// Updates the inner `T` by calling a function on it.
+ ///
+ /// If [`T: Unaligned`], then `Unalign<T>` implements [`DerefMut`], and that
+ /// impl should be preferred over this method when performing updates, as it
+ /// will usually be faster and more ergonomic.
+ ///
+ /// For large types, this method may be expensive, as it requires copying
+ /// `2 * size_of::<T>()` bytes. \[1\]
+ ///
+ /// \[1\] Since the inner `T` may not be aligned, it would not be sound to
+ /// invoke `f` on it directly. Instead, `update` moves it into a
+ /// properly-aligned location in the local stack frame, calls `f` on it, and
+ /// then moves it back to its original location in `self`.
+ ///
+ /// [`T: Unaligned`]: Unaligned
+ #[inline]
+ pub fn update<O, F: FnOnce(&mut T) -> O>(&mut self, f: F) -> O {
+ if mem::align_of::<T>() == 1 {
+ // While we advise callers to use `DerefMut` when `T: Unaligned`,
+ // not all callers will be able to guarantee `T: Unaligned` in all
+ // cases. In particular, callers who are themselves providing an API
+ // which is generic over `T` may sometimes be called by *their*
+ // callers with `T` such that `align_of::<T>() == 1`, but cannot
+ // guarantee this in the general case. Thus, this optimization may
+ // sometimes be helpful.
+
+ // SAFETY: Since `T`'s alignment is 1, `self` satisfies its
+ // alignment by definition.
+ let t = unsafe { self.deref_mut_unchecked() };
+ return f(t);
+ }
+
+ // On drop, this moves `copy` out of itself and uses `ptr::write` to
+ // overwrite `slf`.
+ struct WriteBackOnDrop<T> {
+ copy: ManuallyDrop<T>,
+ slf: *mut Unalign<T>,
+ }
+
+ impl<T> Drop for WriteBackOnDrop<T> {
+ fn drop(&mut self) {
+ // SAFETY: We never use `copy` again as required by
+ // `ManuallyDrop::take`.
+ let copy = unsafe { ManuallyDrop::take(&mut self.copy) };
+ // SAFETY: `slf` is the raw pointer value of `self`. We know it
+ // is valid for writes and properly aligned because `self` is a
+ // mutable reference, which guarantees both of these properties.
+ unsafe { ptr::write(self.slf, Unalign::new(copy)) };
+ }
+ }
+
+ // SAFETY: We know that `self` is valid for reads, properly aligned, and
+ // points to an initialized `Unalign<T>` because it is a mutable
+ // reference, which guarantees all of these properties.
+ //
+ // Since `T: !Copy`, it would be unsound in the general case to allow
+ // both the original `Unalign<T>` and the copy to be used by safe code.
+ // We guarantee that the copy is used to overwrite the original in the
+ // `Drop::drop` impl of `WriteBackOnDrop`. So long as this `drop` is
+ // called before any other safe code executes, soundness is upheld.
+ // While this method can terminate in two ways (by returning normally or
+ // by unwinding due to a panic in `f`), in both cases, `write_back` is
+ // dropped - and its `drop` called - before any other safe code can
+ // execute.
+ let copy = unsafe { ptr::read(self) }.into_inner();
+ let mut write_back = WriteBackOnDrop { copy: ManuallyDrop::new(copy), slf: self };
+
+ let ret = f(&mut write_back.copy);
+
+ drop(write_back);
+ ret
+ }
+}
+
+impl<T: Copy> Unalign<T> {
+ /// Gets a copy of the inner `T`.
+ // FIXME(https://github.com/rust-lang/rust/issues/57349): Make this `const`.
+ #[inline(always)]
+ pub fn get(&self) -> T {
+ let Unalign(val) = *self;
+ val
+ }
+}
+
+impl<T: Unaligned> Deref for Unalign<T> {
+ type Target = T;
+
+ #[inline(always)]
+ fn deref(&self) -> &T {
+ Ptr::from_ref(self).transmute().bikeshed_recall_aligned().as_ref()
+ }
+}
+
+impl<T: Unaligned> DerefMut for Unalign<T> {
+ #[inline(always)]
+ fn deref_mut(&mut self) -> &mut T {
+ Ptr::from_mut(self).transmute::<_, _, (_, (_, _))>().bikeshed_recall_aligned().as_mut()
+ }
+}
+
+impl<T: Unaligned + PartialOrd> PartialOrd<Unalign<T>> for Unalign<T> {
+ #[inline(always)]
+ fn partial_cmp(&self, other: &Unalign<T>) -> Option<Ordering> {
+ PartialOrd::partial_cmp(self.deref(), other.deref())
+ }
+}
+
+impl<T: Unaligned + Ord> Ord for Unalign<T> {
+ #[inline(always)]
+ fn cmp(&self, other: &Unalign<T>) -> Ordering {
+ Ord::cmp(self.deref(), other.deref())
+ }
+}
+
+impl<T: Unaligned + PartialEq> PartialEq<Unalign<T>> for Unalign<T> {
+ #[inline(always)]
+ fn eq(&self, other: &Unalign<T>) -> bool {
+ PartialEq::eq(self.deref(), other.deref())
+ }
+}
+
+impl<T: Unaligned + Eq> Eq for Unalign<T> {}
+
+impl<T: Unaligned + Hash> Hash for Unalign<T> {
+ #[inline(always)]
+ fn hash<H>(&self, state: &mut H)
+ where
+ H: Hasher,
+ {
+ self.deref().hash(state);
+ }
+}
+
+impl<T: Unaligned + Debug> Debug for Unalign<T> {
+ #[inline(always)]
+ fn fmt(&self, f: &mut Formatter<'_>) -> fmt::Result {
+ Debug::fmt(self.deref(), f)
+ }
+}
+
+impl<T: Unaligned + Display> Display for Unalign<T> {
+ #[inline(always)]
+ fn fmt(&self, f: &mut Formatter<'_>) -> fmt::Result {
+ Display::fmt(self.deref(), f)
+ }
+}
+
+/// A wrapper type to construct uninitialized instances of `T`.
+///
+/// `MaybeUninit` is identical to the [standard library
+/// `MaybeUninit`][core-maybe-uninit] type except that it supports unsized
+/// types.
+///
+/// # Layout
+///
+/// The same layout guarantees and caveats apply to `MaybeUninit<T>` as apply to
+/// the [standard library `MaybeUninit`][core-maybe-uninit] with one exception:
+/// for `T: !Sized`, there is no single value for `T`'s size. Instead, for such
+/// types, the following are guaranteed:
+/// - Every [valid size][valid-size] for `T` is a valid size for
+/// `MaybeUninit<T>` and vice versa
+/// - Given `t: *const T` and `m: *const MaybeUninit<T>` with identical fat
+/// pointer metadata, `t` and `m` address the same number of bytes (and
+/// likewise for `*mut`)
+///
+/// [core-maybe-uninit]: core::mem::MaybeUninit
+/// [valid-size]: crate::KnownLayout#what-is-a-valid-size
+#[repr(transparent)]
+#[doc(hidden)]
+pub struct MaybeUninit<T: ?Sized + KnownLayout>(
+ // SAFETY: `MaybeUninit<T>` has the same size as `T`, because (by invariant
+ // on `T::MaybeUninit`) `T::MaybeUninit` has `T::LAYOUT` identical to `T`,
+ // and because (invariant on `T::LAYOUT`) we can trust that `LAYOUT`
+ // accurately reflects the layout of `T`. By invariant on `T::MaybeUninit`,
+ // it admits uninitialized bytes in all positions. Because `MaybeUninit` is
+ // marked `repr(transparent)`, these properties additionally hold true for
+ // `Self`.
+ T::MaybeUninit,
+);
+
+#[doc(hidden)]
+impl<T: ?Sized + KnownLayout> MaybeUninit<T> {
+ /// Constructs a `MaybeUninit<T>` initialized with the given value.
+ #[inline(always)]
+ pub fn new(val: T) -> Self
+ where
+ T: Sized,
+ Self: Sized,
+ {
+ // SAFETY: It is valid to transmute `val` to `MaybeUninit<T>` because it
+ // is both valid to transmute `val` to `T::MaybeUninit`, and it is valid
+ // to transmute from `T::MaybeUninit` to `MaybeUninit<T>`.
+ //
+ // First, it is valid to transmute `val` to `T::MaybeUninit` because, by
+ // invariant on `T::MaybeUninit`:
+ // - For `T: Sized`, `T` and `T::MaybeUninit` have the same size.
+ // - All byte sequences of the correct size are valid values of
+ // `T::MaybeUninit`.
+ //
+ // Second, it is additionally valid to transmute from `T::MaybeUninit`
+ // to `MaybeUninit<T>`, because `MaybeUninit<T>` is a
+ // `repr(transparent)` wrapper around `T::MaybeUninit`.
+ //
+ // These two transmutes are collapsed into one so we don't need to add a
+ // `T::MaybeUninit: Sized` bound to this function's `where` clause.
+ unsafe { crate::util::transmute_unchecked(val) }
+ }
+
+ /// Constructs an uninitialized `MaybeUninit<T>`.
+ #[must_use]
+ #[inline(always)]
+ pub fn uninit() -> Self
+ where
+ T: Sized,
+ Self: Sized,
+ {
+ let uninit = CoreMaybeUninit::<T>::uninit();
+ // SAFETY: It is valid to transmute from `CoreMaybeUninit<T>` to
+ // `MaybeUninit<T>` since they both admit uninitialized bytes in all
+ // positions, and they have the same size (i.e., that of `T`).
+ //
+ // `MaybeUninit<T>` has the same size as `T`, because (by invariant on
+ // `T::MaybeUninit`) `T::MaybeUninit` has `T::LAYOUT` identical to `T`,
+ // and because (invariant on `T::LAYOUT`) we can trust that `LAYOUT`
+ // accurately reflects the layout of `T`.
+ //
+ // `CoreMaybeUninit<T>` has the same size as `T` [1] and admits
+ // uninitialized bytes in all positions.
+ //
+ // [1] Per https://doc.rust-lang.org/1.81.0/std/mem/union.MaybeUninit.html#layout-1:
+ //
+ // `MaybeUninit<T>` is guaranteed to have the same size, alignment,
+ // and ABI as `T`
+ unsafe { crate::util::transmute_unchecked(uninit) }
+ }
+
+ /// Creates a `Box<MaybeUninit<T>>`.
+ ///
+ /// This function is useful for allocating large, uninit values on the heap
+ /// without ever creating a temporary instance of `Self` on the stack.
+ ///
+ /// # Errors
+ ///
+ /// Returns an error on allocation failure. Allocation failure is guaranteed
+ /// never to cause a panic or an abort.
+ #[cfg(feature = "alloc")]
+ #[inline]
+ pub fn new_boxed_uninit(meta: T::PointerMetadata) -> Result<Box<Self>, AllocError> {
+ // SAFETY: `alloc::alloc::alloc_zeroed` is a valid argument of
+ // `new_box`. The referent of the pointer returned by `alloc` (and,
+ // consequently, the `Box` derived from it) is a valid instance of
+ // `Self`, because `Self` is `MaybeUninit` and thus admits arbitrary
+ // (un)initialized bytes.
+ unsafe { crate::util::new_box(meta, alloc::alloc::alloc) }
+ }
+
+ /// Extracts the value from the `MaybeUninit<T>` container.
+ ///
+ /// # Safety
+ ///
+ /// The caller must ensure that `self` is in an bit-valid state. Depending
+ /// on subsequent use, it may also need to be in a library-valid state.
+ #[inline(always)]
+ pub unsafe fn assume_init(self) -> T
+ where
+ T: Sized,
+ Self: Sized,
+ {
+ // SAFETY: The caller guarantees that `self` is in an bit-valid state.
+ unsafe { crate::util::transmute_unchecked(self) }
+ }
+}
+
+impl<T: ?Sized + KnownLayout> fmt::Debug for MaybeUninit<T> {
+ #[inline]
+ fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
+ f.pad(core::any::type_name::<Self>())
+ }
+}
+
+#[allow(unreachable_pub)] // False positive on MSRV
+#[doc(hidden)]
+pub use read_only_def::*;
+mod read_only_def {
+ /// A read-only wrapper.
+ ///
+ /// A `ReadOnly<T>` disables any interior mutability in `T`, ensuring that
+ /// a `&ReadOnly<T>` is genuinely read-only. Thus, `ReadOnly<T>` is
+ /// [`Immutable`] regardless of whether `T` is.
+ ///
+ /// Note that `&mut ReadOnly<T>` still permits mutation – the read-only
+ /// property only applies to shared references.
+ ///
+ /// [`Immutable`]: crate::Immutable
+ #[repr(transparent)]
+ pub struct ReadOnly<T: ?Sized> {
+ // INVARIANT: `inner` is never mutated through a `&ReadOnly<T>`
+ // reference.
+ inner: T,
+ }
+
+ impl<T> ReadOnly<T> {
+ /// Creates a new `ReadOnly`.
+ #[must_use]
+ #[inline(always)]
+ pub const fn new(t: T) -> ReadOnly<T> {
+ ReadOnly { inner: t }
+ }
+
+ /// Returns the inner value.
+ #[must_use]
+ #[inline(always)]
+ pub fn into_inner(r: ReadOnly<T>) -> T {
+ r.inner
+ }
+ }
+
+ impl<T: ?Sized> ReadOnly<T> {
+ #[inline(always)]
+ pub(crate) fn as_mut(r: &mut ReadOnly<T>) -> &mut T {
+ // SAFETY: `r: &mut ReadOnly`, so this doesn't violate the invariant
+ // that `inner` is never mutated through a `&ReadOnly<T>` reference.
+ &mut r.inner
+ }
+
+ /// # Safety
+ ///
+ /// The caller promises not to mutate the referent (i.e., via interior
+ /// mutation).
+ pub(crate) const unsafe fn as_ref_unchecked(r: &ReadOnly<T>) -> &T {
+ // SAFETY: The caller promises not to mutate the referent.
+ &r.inner
+ }
+ }
+}
+
+// SAFETY: `ReadOnly<T>` is a `#[repr(transparent)` wrapper around `T`.
+const _: () = unsafe {
+ unsafe_impl_known_layout!(T: ?Sized + KnownLayout => #[repr(T)] ReadOnly<T>);
+};
+
+#[allow(clippy::multiple_unsafe_ops_per_block)]
+// SAFETY:
+// - `ReadOnly<T>` has the same alignment as `T`, and so it is `Unaligned`
+// exactly when `T` is as well.
+// - `ReadOnly<T>` has the same bit validity as `T`, and so this `is_bit_valid`
+// implementation is correct, and thus the `TryFromBytes` impl is sound.
+// - `ReadOnly<T>` has the same bit validity as `T`, and so it is `FromZeros`,
+// `FromBytes`, and `IntoBytes` exactly when `T` is as well.
+const _: () = unsafe {
+ unsafe_impl!(T: ?Sized + Unaligned => Unaligned for ReadOnly<T>);
+ unsafe_impl!(
+ T: ?Sized + TryFromBytes => TryFromBytes for ReadOnly<T>;
+ |c| T::is_bit_valid(c.cast::<_, <ReadOnly<T> as SizeEq<ReadOnly<ReadOnly<T>>>>::CastFrom, _>())
+ );
+ unsafe_impl!(T: ?Sized + FromZeros => FromZeros for ReadOnly<T>);
+ unsafe_impl!(T: ?Sized + FromBytes => FromBytes for ReadOnly<T>);
+ unsafe_impl!(T: ?Sized + IntoBytes => IntoBytes for ReadOnly<T>);
+};
+
+// SAFETY: By invariant, `inner` is never mutated through a `&ReadOnly<T>`
+// reference.
+const _: () = unsafe {
+ unsafe_impl!(T: ?Sized => Immutable for ReadOnly<T>);
+};
+
+const _: () = {
+ use crate::pointer::cast::CastExact;
+
+ // SAFETY: `ReadOnly<T>` has the same layout as `T`.
+ define_cast!(unsafe { pub CastFromReadOnly<T: ?Sized> = ReadOnly<T> => T});
+ // SAFETY: `ReadOnly<T>` has the same layout as `T`.
+ unsafe impl<T: ?Sized> CastExact<ReadOnly<T>, T> for CastFromReadOnly {}
+ // SAFETY: `ReadOnly<T>` has the same layout as `T`.
+ define_cast!(unsafe { pub CastToReadOnly<T: ?Sized> = T => ReadOnly<T>});
+ // SAFETY: `ReadOnly<T>` has the same layout as `T`.
+ unsafe impl<T: ?Sized> CastExact<T, ReadOnly<T>> for CastToReadOnly {}
+
+ impl<T: ?Sized> SizeEq<ReadOnly<T>> for T {
+ type CastFrom = CastFromReadOnly;
+ }
+
+ impl<T: ?Sized> SizeEq<T> for ReadOnly<T> {
+ type CastFrom = CastToReadOnly;
+ }
+};
+
+// SAFETY: `ReadOnly<T>` is a `#[repr(transparent)]` wrapper around `T`, and so
+// it has the same bit validity as `T`.
+unsafe impl<T: ?Sized> TransmuteFrom<T, Valid, Valid> for ReadOnly<T> {}
+
+// SAFETY: `ReadOnly<T>` is a `#[repr(transparent)]` wrapper around `T`, and so
+// it has the same bit validity as `T`.
+unsafe impl<T: ?Sized> TransmuteFrom<ReadOnly<T>, Valid, Valid> for T {}
+
+impl<'a, T: ?Sized + Immutable> From<&'a T> for &'a ReadOnly<T> {
+ #[inline(always)]
+ fn from(t: &'a T) -> &'a ReadOnly<T> {
+ let ro = Ptr::from_ref(t).transmute::<_, _, (_, _)>();
+ // SAFETY: `ReadOnly<T>` has the same alignment as `T`, and
+ // `Ptr::from_ref` produces an aligned `Ptr`.
+ let ro = unsafe { ro.assume_alignment() };
+ ro.as_ref()
+ }
+}
+
+impl<T: ?Sized + Immutable> Deref for ReadOnly<T> {
+ type Target = T;
+
+ #[inline(always)]
+ fn deref(&self) -> &Self::Target {
+ // SAFETY: By `T: Immutable`, `&T` doesn't permit interior mutation.
+ unsafe { ReadOnly::as_ref_unchecked(self) }
+ }
+}
+
+impl<T: ?Sized + Immutable> DerefMut for ReadOnly<T> {
+ #[inline(always)]
+ fn deref_mut(&mut self) -> &mut Self::Target {
+ ReadOnly::as_mut(self)
+ }
+}
+
+impl<T: ?Sized + Immutable + Debug> Debug for ReadOnly<T> {
+ #[inline(always)]
+ fn fmt(&self, f: &mut Formatter<'_>) -> fmt::Result {
+ self.deref().fmt(f)
+ }
+}
+
+// SAFETY: See safety comment on `ProjectToTag`.
+unsafe impl<T: HasTag + ?Sized> HasTag for ReadOnly<T> {
+ #[allow(clippy::missing_inline_in_public_items)]
+ fn only_derive_is_allowed_to_implement_this_trait()
+ where
+ Self: Sized,
+ {
+ }
+
+ type Tag = T::Tag;
+
+ // SAFETY: `<T as SizeEq<ReadOnly<T>>>::CastFrom` is a no-op projection that
+ // produces a pointer with the same referent. By invariant, for any `Ptr<'_,
+ // T, I>` it is sound to use `T::ProjectToTag` to project to a `Ptr<'_,
+ // T::Tag, I>`. Since `ReadOnly<T>` has the same layout and validity as `T`,
+ // the same is true of projecting from a `Ptr<'_, ReadOnly<T>, I>`.
+ type ProjectToTag = crate::pointer::cast::TransitiveProject<
+ T,
+ <T as SizeEq<ReadOnly<T>>>::CastFrom,
+ T::ProjectToTag,
+ >;
+}
+
+// SAFETY: `ReadOnly<T>` is a `#[repr(transparent)]` wrapper around `T`, and so
+// has the same fields at the same offsets. Thus, it satisfies the safety
+// invariants of `HasField<Field, VARIANT_ID, FIELD_ID>` for field `f` exactly
+// when `T` does, as guaranteed by the `T: HasField` bound:
+// - If `VARIANT_ID` is `STRUCT_VARIANT_ID` or `UNION_VARIANT_ID`, then `T` has
+// the layout of a struct or union type. Since `ReadOnly<T>` is a transparent
+// wrapper around `T`, it does too. Otherwise, if `VARIANT_ID` is an enum
+// variant index, then `T` has the layout of an enum type, and `ReadOnly<T>`
+// does too.
+// - By `T: HasField<_, _, FIELD_ID>`:
+// - `T` has a field `f` with name `n` such that
+// `FIELD_ID = zerocopy::ident_id!(n)` or at index `i` such that
+// `FIELD_ID = zerocopy::ident_id!(i)`.
+// - `Field` has the same visibility as `f`.
+// - `T::Type` has the same type as `f`. Thus, `ReadOnly<T::Type>` has the
+// same type as `f`, wrapped in `ReadOnly`.
+//
+// `project` satisfies its post-condition – namely, that the returned pointer
+// refers to a non-strict subset of the bytes of `slf`'s referent, and has the
+// same provenance as `slf` – because all intermediate operations satisfy those
+// same conditions.
+unsafe impl<T, Field, const VARIANT_ID: i128, const FIELD_ID: i128>
+ HasField<Field, VARIANT_ID, FIELD_ID> for ReadOnly<T>
+where
+ T: HasField<Field, VARIANT_ID, FIELD_ID> + ?Sized,
+{
+ #[allow(clippy::missing_inline_in_public_items)]
+ fn only_derive_is_allowed_to_implement_this_trait()
+ where
+ Self: Sized,
+ {
+ }
+
+ type Type = ReadOnly<T::Type>;
+
+ #[inline(always)]
+ fn project(slf: PtrInner<'_, Self>) -> *mut ReadOnly<T::Type> {
+ slf.project::<_, <T as SizeEq<ReadOnly<T>>>::CastFrom>()
+ .project::<_, crate::pointer::cast::Projection<Field, VARIANT_ID, FIELD_ID>>()
+ .project::<_, <ReadOnly<T::Type> as SizeEq<T::Type>>::CastFrom>()
+ .as_non_null()
+ .as_ptr()
+ }
+}
+
+// SAFETY: `ReadOnly<T>` is a `#[repr(transparent)]` wrapper around `T`, and so
+// has the same fields at the same offsets. `is_projectable` simply delegates to
+// `T::is_projectable`, which is sound because a `Ptr<'_, ReadOnly<T>, I>` will
+// be projectable exactly when a `Ptr<'_, T, I>` referent is.
+unsafe impl<T, Field, I, const VARIANT_ID: i128, const FIELD_ID: i128>
+ ProjectField<Field, I, VARIANT_ID, FIELD_ID> for ReadOnly<T>
+where
+ T: ProjectField<Field, I, VARIANT_ID, FIELD_ID> + ?Sized,
+ I: invariant::Invariants,
+{
+ #[allow(clippy::missing_inline_in_public_items)]
+ fn only_derive_is_allowed_to_implement_this_trait()
+ where
+ Self: Sized,
+ {
+ }
+
+ type Invariants = T::Invariants;
+
+ type Error = T::Error;
+
+ #[inline(always)]
+ fn is_projectable<'a>(ptr: Ptr<'a, Self::Tag, I>) -> Result<(), Self::Error> {
+ T::is_projectable(ptr)
+ }
+}
+
+#[cfg(test)]
+mod tests {
+ use core::panic::AssertUnwindSafe;
+
+ use super::*;
+ use crate::util::testutil::*;
+
+ #[test]
+ fn test_unalign() {
+ // Test methods that don't depend on alignment.
+ let mut u = Unalign::new(AU64(123));
+ assert_eq!(u.get(), AU64(123));
+ assert_eq!(u.into_inner(), AU64(123));
+ assert_eq!(u.get_ptr(), <*const _>::cast::<AU64>(&u));
+ assert_eq!(u.get_mut_ptr(), <*mut _>::cast::<AU64>(&mut u));
+ u.set(AU64(321));
+ assert_eq!(u.get(), AU64(321));
+
+ // Test methods that depend on alignment (when alignment is satisfied).
+ let mut u: Align<_, AU64> = Align::new(Unalign::new(AU64(123)));
+ assert_eq!(u.t.try_deref().unwrap(), &AU64(123));
+ assert_eq!(u.t.try_deref_mut().unwrap(), &mut AU64(123));
+ // SAFETY: The `Align<_, AU64>` guarantees proper alignment.
+ assert_eq!(unsafe { u.t.deref_unchecked() }, &AU64(123));
+ // SAFETY: The `Align<_, AU64>` guarantees proper alignment.
+ assert_eq!(unsafe { u.t.deref_mut_unchecked() }, &mut AU64(123));
+ *u.t.try_deref_mut().unwrap() = AU64(321);
+ assert_eq!(u.t.get(), AU64(321));
+
+ // Test methods that depend on alignment (when alignment is not
+ // satisfied).
+ let mut u: ForceUnalign<_, AU64> = ForceUnalign::new(Unalign::new(AU64(123)));
+ assert!(matches!(u.t.try_deref(), Err(AlignmentError { .. })));
+ assert!(matches!(u.t.try_deref_mut(), Err(AlignmentError { .. })));
+
+ // Test methods that depend on `T: Unaligned`.
+ let mut u = Unalign::new(123u8);
+ assert_eq!(u.try_deref(), Ok(&123));
+ assert_eq!(u.try_deref_mut(), Ok(&mut 123));
+ assert_eq!(u.deref(), &123);
+ assert_eq!(u.deref_mut(), &mut 123);
+ *u = 21;
+ assert_eq!(u.get(), 21);
+
+ // Test that some `Unalign` functions and methods are `const`.
+ const _UNALIGN: Unalign<u64> = Unalign::new(0);
+ const _UNALIGN_PTR: *const u64 = _UNALIGN.get_ptr();
+ const _U64: u64 = _UNALIGN.into_inner();
+ // Make sure all code is considered "used".
+ //
+ // FIXME(https://github.com/rust-lang/rust/issues/104084): Remove this
+ // attribute.
+ #[allow(dead_code)]
+ const _: () = {
+ let x: Align<_, AU64> = Align::new(Unalign::new(AU64(123)));
+ // Make sure that `deref_unchecked` is `const`.
+ //
+ // SAFETY: The `Align<_, AU64>` guarantees proper alignment.
+ let au64 = unsafe { x.t.deref_unchecked() };
+ match au64 {
+ AU64(123) => {}
+ _ => const_unreachable!(),
+ }
+ };
+ }
+
+ #[test]
+ fn test_unalign_update() {
+ let mut u = Unalign::new(AU64(123));
+ u.update(|a| a.0 += 1);
+ assert_eq!(u.get(), AU64(124));
+
+ // Test that, even if the callback panics, the original is still
+ // correctly overwritten. Use a `Box` so that Miri is more likely to
+ // catch any unsoundness (which would likely result in two `Box`es for
+ // the same heap object, which is the sort of thing that Miri would
+ // probably catch).
+ let mut u = Unalign::new(Box::new(AU64(123)));
+ let res = std::panic::catch_unwind(AssertUnwindSafe(|| {
+ u.update(|a| {
+ a.0 += 1;
+ panic!();
+ })
+ }));
+ assert!(res.is_err());
+ assert_eq!(u.into_inner(), Box::new(AU64(124)));
+
+ // Test the align_of::<T>() == 1 optimization.
+ let mut u = Unalign::new([0u8, 1]);
+ u.update(|a| a[0] += 1);
+ assert_eq!(u.get(), [1u8, 1]);
+ }
+
+ #[test]
+ fn test_unalign_copy_clone() {
+ // Test that `Copy` and `Clone` do not cause soundness issues. This test
+ // is mainly meant to exercise UB that would be caught by Miri.
+
+ // `u.t` is definitely not validly-aligned for `AU64`'s alignment of 8.
+ let u = ForceUnalign::<_, AU64>::new(Unalign::new(AU64(123)));
+ #[allow(clippy::clone_on_copy)]
+ let v = u.t.clone();
+ let w = u.t;
+ assert_eq!(u.t.get(), v.get());
+ assert_eq!(u.t.get(), w.get());
+ assert_eq!(v.get(), w.get());
+ }
+
+ #[test]
+ fn test_unalign_trait_impls() {
+ let zero = Unalign::new(0u8);
+ let one = Unalign::new(1u8);
+
+ assert!(zero < one);
+ assert_eq!(PartialOrd::partial_cmp(&zero, &one), Some(Ordering::Less));
+ assert_eq!(Ord::cmp(&zero, &one), Ordering::Less);
+
+ assert_ne!(zero, one);
+ assert_eq!(zero, zero);
+ assert!(!PartialEq::eq(&zero, &one));
+ assert!(PartialEq::eq(&zero, &zero));
+
+ fn hash<T: Hash>(t: &T) -> u64 {
+ let mut h = std::collections::hash_map::DefaultHasher::new();
+ t.hash(&mut h);
+ h.finish()
+ }
+
+ assert_eq!(hash(&zero), hash(&0u8));
+ assert_eq!(hash(&one), hash(&1u8));
+
+ assert_eq!(format!("{:?}", zero), format!("{:?}", 0u8));
+ assert_eq!(format!("{:?}", one), format!("{:?}", 1u8));
+ assert_eq!(format!("{}", zero), format!("{}", 0u8));
+ assert_eq!(format!("{}", one), format!("{}", 1u8));
+ }
+
+ #[test]
+ #[allow(clippy::as_conversions)]
+ fn test_maybe_uninit() {
+ // int
+ {
+ let input = 42;
+ let uninit = MaybeUninit::new(input);
+ // SAFETY: `uninit` is in an initialized state
+ let output = unsafe { uninit.assume_init() };
+ assert_eq!(input, output);
+ }
+
+ // thin ref
+ {
+ let input = 42;
+ let uninit = MaybeUninit::new(&input);
+ // SAFETY: `uninit` is in an initialized state
+ let output = unsafe { uninit.assume_init() };
+ assert_eq!(&input as *const _, output as *const _);
+ assert_eq!(input, *output);
+ }
+
+ // wide ref
+ {
+ let input = [1, 2, 3, 4];
+ let uninit = MaybeUninit::new(&input[..]);
+ // SAFETY: `uninit` is in an initialized state
+ let output = unsafe { uninit.assume_init() };
+ assert_eq!(&input[..] as *const _, output as *const _);
+ assert_eq!(input, *output);
+ }
+ }
+ #[test]
+ fn test_maybe_uninit_uninit() {
+ let _uninit = MaybeUninit::<u8>::uninit();
+ // Cannot check value, but can check it compiles and runs
+ }
+
+ #[test]
+ #[cfg(feature = "alloc")]
+ fn test_maybe_uninit_new_boxed_uninit() {
+ let _boxed = MaybeUninit::<u8>::new_boxed_uninit(()).unwrap();
+ }
+
+ #[test]
+ fn test_maybe_uninit_debug() {
+ let uninit = MaybeUninit::<u8>::uninit();
+ assert!(format!("{:?}", uninit).contains("MaybeUninit"));
+ }
+}
diff --git a/samples/rust/rust_dma.rs b/samples/rust/rust_dma.rs
index 129bb4b..a2c34bb 100644
--- a/samples/rust/rust_dma.rs
+++ b/samples/rust/rust_dma.rs
@@ -73,7 +73,7 @@ fn probe(pdev: &pci::Device<Core>, _info: &Self::IdInfo) -> impl PinInit<Self, E
Coherent::zeroed_slice(pdev.as_ref(), TEST_VALUES.len(), GFP_KERNEL)?;
for (i, value) in TEST_VALUES.into_iter().enumerate() {
- kernel::dma_write!(ca, [i]?, MyStruct::new(value.0, value.1));
+ kernel::dma_write!(ca, [try: i], MyStruct::new(value.0, value.1));
}
let size = 4 * page::PAGE_SIZE;
@@ -91,16 +91,14 @@ fn probe(pdev: &pci::Device<Core>, _info: &Self::IdInfo) -> impl PinInit<Self, E
}
impl DmaSampleDriver {
- fn check_dma(&self) -> Result {
+ fn check_dma(&self) {
for (i, value) in TEST_VALUES.into_iter().enumerate() {
- let val0 = kernel::dma_read!(self.ca, [i]?.h);
- let val1 = kernel::dma_read!(self.ca, [i]?.b);
+ let val0 = kernel::dma_read!(self.ca, [panic: i].h);
+ let val1 = kernel::dma_read!(self.ca, [panic: i].b);
assert_eq!(val0, value.0);
assert_eq!(val1, value.1);
}
-
- Ok(())
}
}
@@ -109,7 +107,7 @@ impl PinnedDrop for DmaSampleDriver {
fn drop(self: Pin<&mut Self>) {
dev_info!(self.pdev, "Unload DMA test driver.\n");
- assert!(self.check_dma().is_ok());
+ self.check_dma();
for (i, entry) in self.sgt.iter().enumerate() {
dev_info!(
diff --git a/scripts/Makefile.autofdo b/scripts/Makefile.autofdo
index 1caf245..1442043 100644
--- a/scripts/Makefile.autofdo
+++ b/scripts/Makefile.autofdo
@@ -3,14 +3,18 @@
# Enable available and selected Clang AutoFDO features.
CFLAGS_AUTOFDO_CLANG := -fdebug-info-for-profiling -mllvm -enable-fs-discriminator=true -mllvm -improved-fs-discriminator=true
+RUSTFLAGS_AUTOFDO_CLANG := $(if $(call rustc-min-version,109800),-Zdebuginfo-for-profiling,-Zdebug-info-for-profiling) -Cllvm-args=-enable-fs-discriminator=true -Cllvm-args=-improved-fs-discriminator=true
ifndef CONFIG_DEBUG_INFO
CFLAGS_AUTOFDO_CLANG += -gmlt
+ RUSTFLAGS_AUTOFDO_CLANG += -Cdebuginfo=line-tables-only
endif
ifdef CLANG_AUTOFDO_PROFILE
CFLAGS_AUTOFDO_CLANG += -fprofile-sample-use=$(CLANG_AUTOFDO_PROFILE) -ffunction-sections
CFLAGS_AUTOFDO_CLANG += -fsplit-machine-functions
+ RUSTFLAGS_AUTOFDO_CLANG += -Zprofile-sample-use=$(CLANG_AUTOFDO_PROFILE) -Zfunction-sections=y
+ RUSTFLAGS_AUTOFDO_CLANG += -Cllvm-args=-split-machine-functions
endif
ifdef CONFIG_LTO_CLANG_THIN
@@ -21,4 +25,4 @@
KBUILD_LDFLAGS += -plugin-opt=-split-machine-functions
endif
-export CFLAGS_AUTOFDO_CLANG
+export CFLAGS_AUTOFDO_CLANG RUSTFLAGS_AUTOFDO_CLANG
diff --git a/scripts/Makefile.build b/scripts/Makefile.build
index 3498d25..9117457 100644
--- a/scripts/Makefile.build
+++ b/scripts/Makefile.build
@@ -329,6 +329,7 @@
-Zcrate-attr=no_std \
-Zcrate-attr='feature($(rust_allowed_features))' \
-Zunstable-options --extern pin_init --extern kernel \
+ --extern zerocopy --extern zerocopy_derive \
--crate-type rlib -L $(objtree)/rust/ \
--sysroot=/dev/null \
--out-dir $(dir $@) --emit=dep-info=$(depfile)
diff --git a/scripts/Makefile.kasan b/scripts/Makefile.kasan
index 0ba2aac..91504e8 100644
--- a/scripts/Makefile.kasan
+++ b/scripts/Makefile.kasan
@@ -71,8 +71,6 @@
CFLAGS_KASAN := -fsanitize=kernel-hwaddress
-# This sets flags that will enable SW_TAGS KASAN once enabled in Rust. These
-# will not work today, and is guarded against in dependencies for CONFIG_RUST.
RUSTFLAGS_KASAN := -Zsanitizer=kernel-hwaddress \
-Zsanitizer-recover=kernel-hwaddress
diff --git a/scripts/Makefile.lib b/scripts/Makefile.lib
index 0718e39c..154a646 100644
--- a/scripts/Makefile.lib
+++ b/scripts/Makefile.lib
@@ -123,6 +123,9 @@
_c_flags += $(if $(patsubst n%,, \
$(AUTOFDO_PROFILE_$(target-stem).o)$(AUTOFDO_PROFILE)$(is-kernel-object)), \
$(CFLAGS_AUTOFDO_CLANG))
+_rust_flags += $(if $(patsubst n%,, \
+ $(AUTOFDO_PROFILE_$(target-stem).o)$(AUTOFDO_PROFILE)$(is-kernel-object)), \
+ $(RUSTFLAGS_AUTOFDO_CLANG))
endif
#
@@ -249,6 +252,13 @@
cmd_ld_single = $(if $(objtool-enabled)$(is-single-obj-m), ; $(LD) $(ld_flags) -r -o $(tmp-target) $@; mv $(tmp-target) $@)
endif
+ifdef CONFIG_LTO_CLANG_THIN_DIST
+# Save the _c_flags, sliently.
+quiet_cmd_save_c_flags =
+ saved_c_flags = $(_c_flags) $(modkern_cflags)
+ cmd_save_c_flags = printf '\n%s\n' 'saved_c_flags_$@ := $(call escsq,$(saved_c_flags))' >> $(dot-target).cmd
+endif
+
quiet_cmd_cc_o_c = CC $(quiet_modtag) $@
cmd_cc_o_c = $(CC) $(c_flags) -c -o $@ $< \
$(cmd_ld_single) \
@@ -256,6 +266,7 @@
define rule_cc_o_c
$(call cmd_and_fixdep,cc_o_c)
+ $(call cmd,save_c_flags)
$(call cmd,checksrc)
$(call cmd,checkdoc)
$(call cmd,gen_objtooldep)
diff --git a/scripts/Makefile.modfinal b/scripts/Makefile.modfinal
index adcbcde..01a37ec 100644
--- a/scripts/Makefile.modfinal
+++ b/scripts/Makefile.modfinal
@@ -46,17 +46,9 @@
$(CONFIG_SHELL) $(srctree)/scripts/gen-btf.sh --btf_base $(objtree)/vmlinux $@; \
fi;
-# Same as newer-prereqs, but allows to exclude specified extra dependencies
-newer_prereqs_except = $(filter-out $(PHONY) $(1),$?)
-
-# Same as if_changed, but allows to exclude specified extra dependencies
-if_changed_except = $(if $(call newer_prereqs_except,$(2))$(cmd-check), \
- $(cmd); \
- printf '%s\n' 'savedcmd_$@ := $(make-cmd)' > $(dot-target).cmd, @:)
-
# Re-generate module BTFs if either module's .ko or vmlinux changed
%.ko: %.o %.mod.o .module-common.o $(objtree)/scripts/module.lds $(and $(CONFIG_DEBUG_INFO_BTF_MODULES),$(KBUILD_BUILTIN),$(objtree)/vmlinux) FORCE
- +$(call if_changed_except,ld_ko_o,$(objtree)/vmlinux)
+ +$(call if_changed,ld_ko_o)
ifdef CONFIG_DEBUG_INFO_BTF_MODULES
+$(if $(newer-prereqs),$(call cmd,btf_ko))
endif
diff --git a/scripts/Makefile.thinlto b/scripts/Makefile.thinlto
new file mode 100644
index 0000000..bb83f13
--- /dev/null
+++ b/scripts/Makefile.thinlto
@@ -0,0 +1,40 @@
+PHONY := __default
+__default:
+
+include include/config/auto.conf
+include $(srctree)/scripts/Kbuild.include
+include $(srctree)/scripts/Makefile.lib
+
+native-objs := $(patsubst %.o,%.thinlto-native.o,$(call read-file, vmlinux.thinlto-index))
+
+__default: $(native-objs)
+
+# Generate .thinlto-native.o (obj) from .o (bitcode) and .thinlto.bc (summary) files
+# ---------------------------------------------------------------------------
+quiet_cmd_cc_o_bc = CC $(quiet_modtag) $@
+ be_flags = $(shell sed -n '/saved_c_flags_/s/.*:= //p' \
+ $(dir $(<)).$(notdir $(<)).cmd)
+ cmd_cc_o_bc = \
+ $(CC) $(be_flags) -x ir -fno-lto -Wno-unused-command-line-argument \
+ -fthinlto-index=$(word 2, $^) -c -o $@ $<
+
+targets += $(native-objs)
+$(native-objs): %.thinlto-native.o: %.o %.o.thinlto.bc FORCE
+ $(call if_changed,cc_o_bc)
+
+# Add FORCE to the prerequisites of a target to force it to be always rebuilt.
+# ---------------------------------------------------------------------------
+
+PHONY += FORCE
+FORCE:
+
+# Read all saved command lines and dependencies for the $(targets) we
+# may be building above, using $(if_changed{,_dep}). As an
+# optimization, we don't need to read them if the target does not
+# exist, we will rebuild anyway in that case.
+
+existing-targets := $(wildcard $(sort $(targets)))
+
+-include $(foreach f, $(existing-targets),$(dir $(f)).$(notdir $(f)).cmd)
+
+.PHONY: $(PHONY)
diff --git a/scripts/Makefile.vmlinux_a b/scripts/Makefile.vmlinux_a
new file mode 100644
index 0000000..395e299
--- /dev/null
+++ b/scripts/Makefile.vmlinux_a
@@ -0,0 +1,82 @@
+# SPDX-License-Identifier: GPL-2.0-only
+
+PHONY := __default
+__default: vmlinux.a
+
+include include/config/auto.conf
+include $(srctree)/scripts/Kbuild.include
+include $(srctree)/scripts/Makefile.lib
+
+# Link of built-in-fixup.a
+# ---------------------------------------------------------------------------
+
+quiet_cmd_ar_builtin_fixup = AR $@
+ cmd_ar_builtin_fixup = \
+ rm -f $@; \
+ $(AR) cDPrST $@ $(KBUILD_VMLINUX_OBJS); \
+ $(AR) mPi $$($(AR) t $@ | sed -n 1p) $@ $$($(AR) t $@ | grep -F -f $(srctree)/scripts/head-object-list.txt)
+
+targets += built-in-fixup.a
+built-in-fixup.a: $(KBUILD_VMLINUX_OBJS) scripts/head-object-list.txt FORCE
+ $(call if_changed,ar_builtin_fixup)
+
+ifdef CONFIG_LTO_CLANG_THIN_DIST
+
+quiet_cmd_builtin.order = GEN $@
+ cmd_builtin.order = $(AR) t $< > $@
+
+targets += builtin.order
+builtin.order: built-in-fixup.a FORCE
+ $(call if_changed,builtin.order)
+
+quiet_cmd_ld_thinlto_index = LD $@
+ cmd_ld_thinlto_index = \
+ $(LD) $(KBUILD_LDFLAGS) -r --thinlto-index-only=$@ @$<
+
+targets += vmlinux.thinlto-index
+vmlinux.thinlto-index: builtin.order FORCE
+ $(call if_changed,ld_thinlto_index)
+
+quiet_cmd_ar_vmlinux.a = GEN $@
+ cmd_ar_vmlinux.a = \
+ rm -f $@; \
+ while read -r obj; do \
+ if grep -Fqx $${obj} $(word 2, $^); then \
+ echo $${obj%.o}.thinlto-native.o; \
+ else \
+ echo $${obj}; \
+ fi; \
+ done < $< | xargs $(AR) cDPrS --thin $@
+
+targets += vmlinux.a
+vmlinux.a: builtin.order vmlinux.thinlto-index FORCE
+ $(Q)$(MAKE) -f $(srctree)/scripts/Makefile.thinlto
+ $(call if_changed,ar_vmlinux.a)
+
+else
+
+# vmlinux.a
+# ---------------------------------------------------------------------------
+
+targets += vmlinux.a
+vmlinux.a: built-in-fixup.a FORCE
+ $(call if_changed,copy)
+
+endif
+
+# Add FORCE to the prerequisites of a target to force it to be always rebuilt.
+# ---------------------------------------------------------------------------
+
+PHONY += FORCE
+FORCE:
+
+# Read all saved command lines and dependencies for the $(targets) we
+# may be building above, using $(if_changed{,_dep}). As an
+# optimization, we don't need to read them if the target does not
+# exist, we will rebuild anyway in that case.
+
+existing-targets := $(wildcard $(sort $(targets)))
+
+-include $(foreach f,$(existing-targets),$(dir $(f)).$(notdir $(f)).cmd)
+
+.PHONY: $(PHONY)
diff --git a/scripts/Makefile.warn b/scripts/Makefile.warn
index e77ca87..35af7d6 100644
--- a/scripts/Makefile.warn
+++ b/scripts/Makefile.warn
@@ -135,16 +135,6 @@
KBUILD_CFLAGS += -Wno-override-init # alias for -Wno-initializer-overrides in clang
ifdef CONFIG_CC_IS_CLANG
-# Clang before clang-16 would warn on default argument promotions.
-ifneq ($(call clang-min-version, 160000),y)
-# Disable -Wformat
-KBUILD_CFLAGS += -Wno-format
-# Then re-enable flags that were part of the -Wformat group that aren't
-# problematic.
-KBUILD_CFLAGS += -Wformat-extra-args -Wformat-invalid-specifier
-KBUILD_CFLAGS += -Wformat-zero-length -Wnonnull
-KBUILD_CFLAGS += -Wformat-insufficient-args
-endif
KBUILD_CFLAGS += -Wno-pointer-to-enum-cast
KBUILD_CFLAGS += -Wno-tautological-constant-out-of-range-compare
KBUILD_CFLAGS += -Wno-unaligned-access
diff --git a/scripts/checkpatch.pl b/scripts/checkpatch.pl
index 0492d6a..cc5bbd7 100755
--- a/scripts/checkpatch.pl
+++ b/scripts/checkpatch.pl
@@ -865,8 +865,6 @@
"DEFINE_IDR" => "DEFINE_XARRAY",
"idr_init" => "xa_init",
"idr_init_base" => "xa_init_flags",
- "rcu_read_lock_trace" => "rcu_read_lock_tasks_trace",
- "rcu_read_unlock_trace" => "rcu_read_unlock_tasks_trace",
);
#Create a search pattern for all these strings to speed up a loop below
@@ -7596,12 +7594,15 @@
# Complain about RCU Tasks Trace used outside of BPF (and of course, RCU).
our $rcu_trace_funcs = qr{(?x:
+ rcu_read_lock_tasks_trace |
rcu_read_lock_trace |
rcu_read_lock_trace_held |
rcu_read_unlock_trace |
+ rcu_read_unlock_tasks_trace |
call_rcu_tasks_trace |
synchronize_rcu_tasks_trace |
rcu_barrier_tasks_trace |
+ rcu_tasks_trace_expedite_current |
rcu_request_urgent_qs_task
)};
our $rcu_trace_paths = qr{(?x:
diff --git a/scripts/clang-tools/run-clang-tools.py b/scripts/clang-tools/run-clang-tools.py
index f31ffd0..e78be82 100755
--- a/scripts/clang-tools/run-clang-tools.py
+++ b/scripts/clang-tools/run-clang-tools.py
@@ -79,14 +79,15 @@
def main():
- try:
- args = parse_arguments()
+ args = parse_arguments()
- lock = multiprocessing.Lock()
- pool = multiprocessing.Pool(initializer=init, initargs=(lock, args))
- # Read JSON data into the datastore variable
- with open(args.path, "r") as f:
- datastore = json.load(f)
+ # Read JSON data into the datastore variable
+ with open(args.path) as f:
+ datastore = json.load(f)
+
+ lock = multiprocessing.Lock()
+ try:
+ with multiprocessing.Pool(initializer=init, initargs=(lock, args)) as pool:
pool.map(run_analysis, datastore)
except BrokenPipeError:
# Python flushes standard streams on exit; redirect remaining output
diff --git a/scripts/generate_rust_analyzer.py b/scripts/generate_rust_analyzer.py
index d5f9a0c..dc121973 100755
--- a/scripts/generate_rust_analyzer.py
+++ b/scripts/generate_rust_analyzer.py
@@ -26,6 +26,14 @@
return crates_cfgs
+def args_crates_envs(envs: List[str]) -> Dict[str, Dict[str, str]]:
+ crates_envs = {}
+ for env in envs:
+ crate, vals = env.split("=", 1)
+ crates_envs[crate] = dict(v.split("=", 1) for v in vals.split())
+
+ return crates_envs
+
class Dependency(TypedDict):
crate: int
name: str
@@ -61,6 +69,7 @@
sysroot_src: pathlib.Path,
external_src: Optional[pathlib.Path],
cfgs: List[str],
+ envs: List[str],
core_edition: str,
) -> List[Crate]:
# Generate the configuration list.
@@ -74,6 +83,7 @@
# Now fill the crates list.
crates: List[Crate] = []
crates_cfgs = args_crates_cfgs(cfgs)
+ crates_envs = args_crates_envs(envs)
def get_crate_name(path: pathlib.Path) -> str:
return invoke_rustc(["--print", "crate-name", str(path)])
@@ -92,6 +102,10 @@
is_workspace_member if is_workspace_member is not None else True
)
edition = edition if edition is not None else "2021"
+ crate_env = {
+ "RUST_MODFILE": "This is only for rust-analyzer",
+ **crates_envs.get(display_name, {}),
+ }
return {
"display_name": display_name,
"root_module": str(root_module),
@@ -99,9 +113,7 @@
"deps": deps,
"cfg": cfg,
"edition": edition,
- "env": {
- "RUST_MODFILE": "This is only for rust-analyzer"
- }
+ "env": crate_env,
}
def append_proc_macro_crate(
@@ -240,6 +252,12 @@
[std, proc_macro, proc_macro2, quote, syn],
)
+ zerocopy_derive = append_proc_macro_crate(
+ "zerocopy_derive",
+ srctree / "rust" / "zerocopy-derive" / "lib.rs",
+ [std, proc_macro, proc_macro2, quote, syn],
+ )
+
build_error = append_crate(
"build_error",
srctree / "rust" / "build_error.rs",
@@ -264,6 +282,12 @@
[core, compiler_builtins],
)
+ zerocopy = append_crate(
+ "zerocopy",
+ srctree / "rust" / "zerocopy" / "src" / "lib.rs",
+ [core, compiler_builtins],
+ )
+
def append_crate_with_generated(
display_name: str,
deps: List[Dependency],
@@ -292,7 +316,7 @@
bindings = append_crate_with_generated("bindings", [core, ffi, pin_init])
uapi = append_crate_with_generated("uapi", [core, ffi, pin_init])
kernel = append_crate_with_generated(
- "kernel", [core, macros, build_error, pin_init, ffi, bindings, uapi]
+ "kernel", [core, macros, build_error, pin_init, ffi, bindings, uapi, zerocopy, zerocopy_derive]
)
scripts = srctree / "scripts"
@@ -337,7 +361,7 @@
append_crate(
crate_name,
path,
- [core, kernel, pin_init],
+ [core, kernel, pin_init, zerocopy, zerocopy_derive],
cfg=generated_cfg,
)
@@ -347,6 +371,7 @@
parser = argparse.ArgumentParser()
parser.add_argument('--verbose', '-v', action='store_true')
parser.add_argument('--cfgs', action='append', default=[])
+ parser.add_argument('--envs', action='append', default=[])
parser.add_argument("core_edition")
parser.add_argument("srctree", type=pathlib.Path)
parser.add_argument("objtree", type=pathlib.Path)
@@ -357,6 +382,7 @@
class Args(argparse.Namespace):
verbose: bool
cfgs: List[str]
+ envs: List[str]
srctree: pathlib.Path
objtree: pathlib.Path
sysroot: pathlib.Path
@@ -372,7 +398,7 @@
)
rust_project = {
- "crates": generate_crates(args.srctree, args.objtree, args.sysroot_src, args.exttree, args.cfgs, args.core_edition),
+ "crates": generate_crates(args.srctree, args.objtree, args.sysroot_src, args.exttree, args.cfgs, args.envs, args.core_edition),
"sysroot": str(args.sysroot),
}
diff --git a/scripts/kconfig/conf.c b/scripts/kconfig/conf.c
index a7b44cd..c368bec 100644
--- a/scripts/kconfig/conf.c
+++ b/scripts/kconfig/conf.c
@@ -297,9 +297,7 @@ static int conf_askvalue(struct symbol *sym, const char *def)
line[1] = 0;
if (!sym_is_changeable(sym)) {
- printf("%s\n", def);
- line[0] = '\n';
- line[1] = 0;
+ printf("%s\n", def ?: "");
return 0;
}
@@ -307,7 +305,7 @@ static int conf_askvalue(struct symbol *sym, const char *def)
case oldconfig:
case syncconfig:
if (sym_has_value(sym)) {
- printf("%s\n", def);
+ printf("%s\n", def ?: "");
return 0;
}
/* fall through */
diff --git a/scripts/kconfig/kconfig-sym-check.pl b/scripts/kconfig/kconfig-sym-check.pl
new file mode 100755
index 0000000..daa5285
--- /dev/null
+++ b/scripts/kconfig/kconfig-sym-check.pl
@@ -0,0 +1,132 @@
+#!/usr/bin/env perl
+# SPDX-License-Identifier: GPL-2.0
+
+use warnings;
+use strict;
+
+my $srctree = shift @ARGV;
+unless (defined $srctree) {
+ $srctree = `git rev-parse --show-toplevel 2>/dev/null`;
+ chomp $srctree;
+ my $msg = "Usage: $0 <srctree> [excludes file]\n";
+ $msg .= "Please provide <srctree>.";
+ $msg .= " Is it '$srctree'?" if $srctree;
+ $msg .= "\n";
+ die $msg;
+}
+my $kconfig_sym_check_excludes = defined $ARGV[0] ? $ARGV[0] : undef;
+
+sub indent_depth {
+ my ($ws) = @_;
+ my $col = 0;
+ for my $c (split //, $ws) {
+ $col = $c eq "\t" ? int($col / 8) * 8 + 8 : $col + 1;
+ }
+ return $col;
+}
+
+my @files = `git -C \Q$srctree\E ls-files '*Kconfig*' 2>/dev/null`;
+if (@files) {
+ chomp @files;
+ @files = map { "$srctree/$_" } @files;
+} else {
+ @files = `find \Q$srctree\E -name '*Kconfig*'`;
+ chomp @files;
+}
+
+@files = grep { !m{/scripts/kconfig/tests/} } @files;
+
+my %configs = ();
+my %refs = ();
+
+foreach my $file (@files) {
+ open F, $file or die "Cannot open $file: $!";
+
+ my $help = 0;
+ my $help_level;
+ my $level;
+
+ while (<F>) {
+ chomp;
+
+ while (/\\\s*$/) {
+ s/\\\s*$/ /;
+ my $cont = <F> // last;
+ chomp $cont;
+ $_ .= $cont;
+ }
+
+ next if /^\s*$/;
+ next if /^\s*#/;
+
+ /^(\s*)/;
+ $level = indent_depth($1);
+
+ if ($help && $level < $help_level) {
+ $help = 0;
+ }
+
+ next if ($help);
+
+ if (/^\s*(help|\-\-\-help\-\-\-)$/) {
+ $help = 1;
+ my $next;
+ while (defined($next = <F>)) {
+ last unless $next =~ /^\s*(?:#.*)?$/;
+ }
+ last unless defined $next;
+ $next =~ /^(\s*)/;
+ if (indent_depth($1) >= $level) {
+ $help_level = indent_depth($1);
+ } else {
+ $help = 0;
+ }
+ $_ = $next;
+ redo;
+ }
+
+ if (/^\s*(config|menuconfig)\s+([a-zA-Z0-9_]+)\s*(#.*)?$/) {
+ $configs{$2}++;
+ next;
+ }
+
+ if (/^\s*(default|def_bool|def_tristate|select|depends\s+on|imply|visible\s+if|range|if|bool|tristate|int|hex|string|prompt)\s+(.+)\s*$/) {
+ my $s = $2;
+ $s =~ s/"(?:[^"\\]|\\.)*"|'(?:[^'\\]|\\.)*'//g;
+ $s =~ s/#.*//;
+ $s =~ s/\$\((?:[^()]*|\((?:[^()]*|\([^()]*\))*\))*\)//g;
+ $s =~ s/%%[^%]*%%//g;
+ my @syms = split /[^a-zA-Z0-9_]+/, $s;
+ map {
+ $refs{$_}++ if (/[a-zA-Z]/ && $_ ne "if" && $_ ne "y" && $_ ne "n" && $_ ne "m" && !/^0[xX][0-9a-fA-F]+$/);
+ } @syms
+ }
+ }
+
+ close F;
+}
+
+my %known_syms = ();
+if (defined $kconfig_sym_check_excludes) {
+ my $file = $kconfig_sym_check_excludes;
+ open(F, "<", $file) or die "Cannot open $file: $!";
+ while (<F>) {
+ chomp;
+ next if /^\s*$/;
+ next if /^\s*#/;
+ $known_syms{$1}++ if (/^\s*([a-zA-Z0-9_]+)\s*(#.*)?$/);
+ }
+}
+
+my $ret = 0;
+foreach my $k (sort keys %refs) {
+ next if (exists $configs{$k} || exists $known_syms{$k});
+
+ print "$k";
+ print " - warning: '$k' is probably not what you want; Kconfig tristate literals are always lowercase ('n', 'y', 'm')" if ($k eq "N" || $k eq "Y" || $k eq "M");
+ print "\n";
+
+ $ret = 1;
+}
+
+exit $ret;
diff --git a/scripts/kconfig/tests/no_write_if_dep_unmet/__init__.py b/scripts/kconfig/tests/no_write_if_dep_unmet/__init__.py
index ffd469d..791ed65 100644
--- a/scripts/kconfig/tests/no_write_if_dep_unmet/__init__.py
+++ b/scripts/kconfig/tests/no_write_if_dep_unmet/__init__.py
@@ -8,7 +8,7 @@
This was not working correctly for choice values because choice needs
a bit different symbol computation.
-This checks that no unneeded "# COFIG_... is not set" is contained in
+This checks that no unneeded "# CONFIG_... is not set" is contained in
the .config file.
Related Linux commit: cb67ab2cd2b8abd9650292c986c79901e3073a59
diff --git a/scripts/min-tool-version.sh b/scripts/min-tool-version.sh
index b96ec2d..ea2689b 100755
--- a/scripts/min-tool-version.sh
+++ b/scripts/min-tool-version.sh
@@ -27,7 +27,7 @@
if [ "$SRCARCH" = loongarch ]; then
echo 18.0.0
else
- echo 15.0.0
+ echo 17.0.1
fi
;;
rustc)
diff --git a/scripts/mod/modpost.c b/scripts/mod/modpost.c
index abbcd3f..d592548 100644
--- a/scripts/mod/modpost.c
+++ b/scripts/mod/modpost.c
@@ -765,6 +765,8 @@ static const char *const section_white_list[] =
".gnu.lto*",
".discard.*",
".llvm.call-graph-profile", /* call graph */
+ "__llvm_covfun",
+ "__llvm_covmap",
NULL
};
@@ -1487,13 +1489,22 @@ static void extract_crcs_for_object(const char *object, struct module *mod)
char cmd_file[PATH_MAX];
char *buf, *p;
const char *base;
- int dirlen, ret;
+ int dirlen, baselen_without_suffix, ret;
base = get_basename(object);
dirlen = base - object;
- ret = snprintf(cmd_file, sizeof(cmd_file), "%.*s.%s.cmd",
- dirlen, object, base);
+ baselen_without_suffix = strlen(object) - dirlen - strlen(".o");
+
+ /*
+ * When CONFIG_LTO_CLANG_THIN_DIST=y, the ELF is *.thinlto-native.o
+ * but the symbol CRCs are recorded in *.o.cmd file.
+ */
+ if (strends(object, ".thinlto-native.o"))
+ baselen_without_suffix -= strlen(".thinlto-native");
+
+ ret = snprintf(cmd_file, sizeof(cmd_file), "%.*s.%.*s.o.cmd",
+ dirlen, object, baselen_without_suffix, base);
if (ret >= sizeof(cmd_file)) {
error("%s: too long path was truncated\n", cmd_file);
return;
@@ -1689,8 +1700,17 @@ void __attribute__((format(printf, 2, 3))) buf_printf(struct buffer *buf,
va_start(ap, fmt);
len = vsnprintf(tmp, SZ, fmt, ap);
- buf_write(buf, tmp, len);
va_end(ap);
+
+ if (len < 0) {
+ perror("vsnprintf failed");
+ exit(1);
+ }
+ if (len >= SZ)
+ fatal("buf_printf output truncated for string %s: %d bytes needed, %d available\n",
+ tmp, len + 1, SZ);
+
+ buf_write(buf, tmp, len);
}
void buf_write(struct buffer *buf, const char *s, int len)
diff --git a/scripts/package/PKGBUILD b/scripts/package/PKGBUILD
index 1213c8e0..66e4b6a 100644
--- a/scripts/package/PKGBUILD
+++ b/scripts/package/PKGBUILD
@@ -121,6 +121,9 @@
install -Dt "${debugdir}" -m644 vmlinux
mkdir -p "${builddir}"
ln -sr "${debugdir}/vmlinux" "${builddir}/vmlinux"
+
+ echo "Installing unstripped vDSO(s)..."
+ ${MAKE} INSTALL_MOD_PATH="${pkgdir}/usr" vdso_install
}
for _p in "${pkgname[@]}"; do
diff --git a/scripts/package/kernel.spec b/scripts/package/kernel.spec
index b3c9562..c732415 100644
--- a/scripts/package/kernel.spec
+++ b/scripts/package/kernel.spec
@@ -6,7 +6,7 @@
Name: kernel
Summary: The Linux Kernel
Version: %(echo %{KERNELRELEASE} | sed -e 's/-/_/g')
-Release: %{pkg_release}
+Release: %{pkg_release}%{?dist}
License: GPL
Group: System Environment/Kernel
Vendor: The Linux Community
diff --git a/security/Kconfig.hardening b/security/Kconfig.hardening
index 86f8768..6923036 100644
--- a/security/Kconfig.hardening
+++ b/security/Kconfig.hardening
@@ -188,10 +188,8 @@
synthetic workloads have measured as high as 8%.
config CC_HAS_ZERO_CALL_USED_REGS
+ # supported by gcc-11 or newer and all supported versions of clang
def_bool $(cc-option,-fzero-call-used-regs=used-gpr)
- # https://github.com/ClangBuiltLinux/linux/issues/1766
- # https://github.com/llvm/llvm-project/issues/59242
- depends on !CC_IS_CLANG || CLANG_VERSION > 150006
config ZERO_CALL_USED_REGS
bool "Enable register zeroing on function exit"
@@ -216,8 +214,6 @@
config FORTIFY_SOURCE
bool "Harden common str/mem functions against buffer overflows"
depends on ARCH_HAS_FORTIFY_SOURCE
- # https://github.com/llvm/llvm-project/issues/53645
- depends on !X86_32 || !CC_IS_CLANG || CLANG_VERSION >= 160000
help
Detect overflows of buffers in common string and memory functions
where the compiler can determine and validate the buffer sizes.
@@ -279,9 +275,6 @@
config CC_HAS_RANDSTRUCT
def_bool $(cc-option,-frandomize-layout-seed-file=/dev/null)
- # Randstruct was first added in Clang 15, but it isn't safe to use until
- # Clang 16 due to https://github.com/llvm/llvm-project/issues/60349
- depends on !CC_IS_CLANG || CLANG_VERSION >= 160000
choice
prompt "Randomize layout of sensitive kernel structures"
diff --git a/security/landlock/fs.c b/security/landlock/fs.c
index c1ecfe2..32d560f 100644
--- a/security/landlock/fs.c
+++ b/security/landlock/fs.c
@@ -1278,7 +1278,7 @@ static void hook_sb_delete(struct super_block *const sb)
struct landlock_object *object;
/* Only handles referenced inodes. */
- if (!icount_read(inode))
+ if (!icount_read_once(inode))
continue;
/*
diff --git a/sound/core/timer.c b/sound/core/timer.c
index 57583de..3d72379 100644
--- a/sound/core/timer.c
+++ b/sound/core/timer.c
@@ -430,6 +430,8 @@ static void snd_timer_close_locked(struct snd_timer_instance *timeri,
if (timer) {
guard(spinlock_irq)(&timer->lock);
+ if (timeri->flags & SNDRV_TIMER_IFLG_DEAD)
+ return; /* already closed */
timeri->flags |= SNDRV_TIMER_IFLG_DEAD;
}
@@ -975,18 +977,18 @@ EXPORT_SYMBOL(snd_timer_new);
static int snd_timer_free(struct snd_timer *timer)
{
+ struct snd_timer_instance *ti, *n;
+
if (!timer)
return 0;
guard(mutex)(®ister_mutex);
if (! list_empty(&timer->open_list_head)) {
- struct list_head *p, *n;
- struct snd_timer_instance *ti;
- pr_warn("ALSA: timer %p is busy?\n", timer);
- list_for_each_safe(p, n, &timer->open_list_head) {
- list_del_init(p);
- ti = list_entry(p, struct snd_timer_instance, open_list);
- ti->timer = NULL;
+ list_for_each_entry_safe(ti, n, &timer->open_list_head, open_list) {
+ struct device *card_dev_to_put = NULL;
+
+ snd_timer_close_locked(ti, &card_dev_to_put);
+ put_device(card_dev_to_put);
}
}
list_del(&timer->device_list);
@@ -1809,6 +1811,7 @@ static int snd_timer_user_params(struct file *file,
struct snd_timer *t;
int err;
+ guard(mutex)(®ister_mutex);
tu = file->private_data;
if (!tu->timeri)
return -EBADFD;
diff --git a/sound/soc/amd/yc/acp6x-mach.c b/sound/soc/amd/yc/acp6x-mach.c
index b426cda..ce229f5 100644
--- a/sound/soc/amd/yc/acp6x-mach.c
+++ b/sound/soc/amd/yc/acp6x-mach.c
@@ -808,6 +808,13 @@ static const struct dmi_system_id yc_acp_quirk_table[] = {
DMI_MATCH(DMI_BOARD_NAME, "MS-17LN"),
}
},
+ {
+ .driver_data = &acp6x_card,
+ .matches = {
+ DMI_MATCH(DMI_BOARD_VENDOR, "ASUSTeK COMPUTER INC."),
+ DMI_MATCH(DMI_BOARD_NAME, "PM1403CDA"),
+ }
+ },
{}
};
diff --git a/sound/soc/codecs/wm_adsp.c b/sound/soc/codecs/wm_adsp.c
index a637e22..ca630c9 100644
--- a/sound/soc/codecs/wm_adsp.c
+++ b/sound/soc/codecs/wm_adsp.c
@@ -679,6 +679,9 @@ static void wm_adsp_control_remove(struct cs_dsp_coeff_ctl *cs_ctl)
{
struct wm_coeff_ctl *ctl = cs_ctl->priv;
+ if (!ctl)
+ return;
+
cancel_work_sync(&ctl->work);
kfree(ctl->name);
diff --git a/sound/soc/loongson/loongson_dma.c b/sound/soc/loongson/loongson_dma.c
index a149b64..f3ed14a 100644
--- a/sound/soc/loongson/loongson_dma.c
+++ b/sound/soc/loongson/loongson_dma.c
@@ -199,6 +199,7 @@ loongson_pcm_pointer(struct snd_soc_component *component,
struct snd_pcm_substream *substream)
{
struct snd_pcm_runtime *runtime = substream->runtime;
+ struct device *dev = substream->pcm->card->dev;
struct loongson_runtime_data *prtd = runtime->private_data;
struct loongson_dma_desc *desc;
snd_pcm_uframes_t x;
@@ -207,9 +208,16 @@ loongson_pcm_pointer(struct snd_soc_component *component,
desc = dma_desc_save(prtd);
addr = ((u64)desc->saddr_hi << 32) | desc->saddr;
- x = bytes_to_frames(runtime, addr - runtime->dma_addr);
- if (x == runtime->buffer_size)
+ if (addr < runtime->dma_addr ||
+ addr > runtime->dma_addr + runtime->dma_bytes) {
+ dev_warn(dev, "WARNING! dma_addr:0x%llx\n", addr);
x = 0;
+ } else {
+ x = bytes_to_frames(runtime, addr - runtime->dma_addr);
+ if (x == runtime->buffer_size)
+ x = 0;
+ }
+
return x;
}
diff --git a/sound/soc/sdca/sdca_function_device.c b/sound/soc/sdca/sdca_function_device.c
index feacfbc..b5ca982 100644
--- a/sound/soc/sdca/sdca_function_device.c
+++ b/sound/soc/sdca/sdca_function_device.c
@@ -82,6 +82,9 @@ static struct sdca_dev *sdca_dev_register(struct device *parent,
static void sdca_dev_unregister(struct sdca_dev *sdev)
{
+ if (!sdev)
+ return;
+
auxiliary_device_delete(&sdev->auxdev);
auxiliary_device_uninit(&sdev->auxdev);
}
@@ -90,14 +93,24 @@ int sdca_dev_register_functions(struct sdw_slave *slave)
{
struct sdca_device_data *sdca_data = &slave->sdca_data;
int i;
+ int ret;
for (i = 0; i < sdca_data->num_functions; i++) {
struct sdca_dev *func_dev;
func_dev = sdca_dev_register(&slave->dev,
&sdca_data->function[i]);
- if (IS_ERR(func_dev))
- return PTR_ERR(func_dev);
+ if (IS_ERR(func_dev)) {
+ ret = PTR_ERR(func_dev);
+ /*
+ * Unregister functions that were successfully
+ * registered before this failure. This also
+ * sets func_dev to NULL so the caller will not
+ * try to unregister them again.
+ */
+ sdca_dev_unregister_functions(slave);
+ return ret;
+ }
sdca_data->function[i].func_dev = func_dev;
}
@@ -111,7 +124,12 @@ void sdca_dev_unregister_functions(struct sdw_slave *slave)
struct sdca_device_data *sdca_data = &slave->sdca_data;
int i;
- for (i = 0; i < sdca_data->num_functions; i++)
+ for (i = 0; i < sdca_data->num_functions; i++) {
+ if (!sdca_data->function[i].func_dev)
+ continue;
+
sdca_dev_unregister(sdca_data->function[i].func_dev);
+ sdca_data->function[i].func_dev = NULL;
+ }
}
EXPORT_SYMBOL_NS(sdca_dev_unregister_functions, "SND_SOC_SDCA");
diff --git a/sound/soc/sof/amd/acp-ipc.c b/sound/soc/sof/amd/acp-ipc.c
index 3cd4674..94025bc 100644
--- a/sound/soc/sof/amd/acp-ipc.c
+++ b/sound/soc/sof/amd/acp-ipc.c
@@ -181,14 +181,14 @@ irqreturn_t acp_sof_ipc_irq_thread(int irq, void *context)
}
dsp_msg = snd_sof_dsp_read(sdev, ACP_DSP_BAR, ACP_SCRATCH_REG_0 + dsp_msg_write);
- if (dsp_msg) {
+ if (dsp_msg == ACP_DSP_MSG_SET) {
snd_sof_ipc_msgs_rx(sdev);
acp_dsp_ipc_host_done(sdev);
ipc_irq = true;
}
dsp_ack = snd_sof_dsp_read(sdev, ACP_DSP_BAR, ACP_SCRATCH_REG_0 + dsp_ack_write);
- if (dsp_ack) {
+ if (dsp_ack == ACP_DSP_ACK_SET) {
if (likely(sdev->fw_state == SOF_FW_BOOT_COMPLETE)) {
guard(spinlock_irq)(&sdev->ipc_lock);
diff --git a/sound/soc/sof/amd/acp.c b/sound/soc/sof/amd/acp.c
index f615b8d..e6af892 100644
--- a/sound/soc/sof/amd/acp.c
+++ b/sound/soc/sof/amd/acp.c
@@ -377,6 +377,33 @@ void memcpy_to_scratch(struct snd_sof_dev *sdev, u32 offset, unsigned int *src,
snd_sof_dsp_write(sdev, ACP_DSP_BAR, reg_offset + i, src[j]);
}
+static int acp_init_scratch_mem_ipc_flags(struct snd_sof_dev *sdev)
+{
+ u32 dsp_msg_write, dsp_ack_write, host_msg_write, host_ack_write;
+
+ dsp_msg_write = sdev->debug_box.offset +
+ offsetof(struct scratch_ipc_conf, sof_dsp_msg_write);
+ dsp_ack_write = sdev->debug_box.offset +
+ offsetof(struct scratch_ipc_conf, sof_dsp_ack_write);
+ host_msg_write = sdev->debug_box.offset +
+ offsetof(struct scratch_ipc_conf, sof_host_msg_write);
+ host_ack_write = sdev->debug_box.offset +
+ offsetof(struct scratch_ipc_conf, sof_host_ack_write);
+ /* Initialize host message write flag */
+ snd_sof_dsp_write(sdev, ACP_DSP_BAR, ACP_SCRATCH_REG_0 + host_msg_write, 0);
+
+ /* Initialize host ack write flag */
+ snd_sof_dsp_write(sdev, ACP_DSP_BAR, ACP_SCRATCH_REG_0 + host_ack_write, 0);
+
+ /* Initialize DSP message write flag */
+ snd_sof_dsp_write(sdev, ACP_DSP_BAR, ACP_SCRATCH_REG_0 + dsp_msg_write, 0);
+
+ /* Initialize DSP ack write flag */
+ snd_sof_dsp_write(sdev, ACP_DSP_BAR, ACP_SCRATCH_REG_0 + dsp_ack_write, 0);
+
+ return 0;
+}
+
static int acp_memory_init(struct snd_sof_dev *sdev)
{
struct acp_dev_data *adata = sdev->pdata->hw_pdata;
@@ -384,6 +411,7 @@ static int acp_memory_init(struct snd_sof_dev *sdev)
snd_sof_dsp_update_bits(sdev, ACP_DSP_BAR, desc->dsp_intr_base + DSP_SW_INTR_CNTL_OFFSET,
ACP_DSP_INTR_EN_MASK, ACP_DSP_INTR_EN_MASK);
+ acp_init_scratch_mem_ipc_flags(sdev);
init_dma_descriptor(adata);
return 0;
diff --git a/sound/soc/sof/amd/acp.h b/sound/soc/sof/amd/acp.h
index 2b7ea8c..7bcb766 100644
--- a/sound/soc/sof/amd/acp.h
+++ b/sound/soc/sof/amd/acp.h
@@ -116,6 +116,8 @@
#define ACP_SRAM_PAGE_COUNT 128
#define ACP6X_SDW_MAX_MANAGER_COUNT 2
#define ACP70_SDW_MAX_MANAGER_COUNT ACP6X_SDW_MAX_MANAGER_COUNT
+#define ACP_DSP_MSG_SET 1
+#define ACP_DSP_ACK_SET 1
enum clock_source {
ACP_CLOCK_96M = 0,
diff --git a/tools/arch/alpha/include/uapi/asm/errno.h b/tools/arch/alpha/include/uapi/asm/errno.h
index 6791f65..1a99f38 100644
--- a/tools/arch/alpha/include/uapi/asm/errno.h
+++ b/tools/arch/alpha/include/uapi/asm/errno.h
@@ -127,4 +127,6 @@
#define EHWPOISON 139 /* Memory page has hardware error */
+#define EFTYPE 140 /* Wrong file type for the intended operation */
+
#endif
diff --git a/tools/arch/mips/include/uapi/asm/errno.h b/tools/arch/mips/include/uapi/asm/errno.h
index c01ed91..1835a50 100644
--- a/tools/arch/mips/include/uapi/asm/errno.h
+++ b/tools/arch/mips/include/uapi/asm/errno.h
@@ -126,6 +126,8 @@
#define EHWPOISON 168 /* Memory page has hardware error */
+#define EFTYPE 169 /* Wrong file type for the intended operation */
+
#define EDQUOT 1133 /* Quota exceeded */
diff --git a/tools/arch/parisc/include/uapi/asm/errno.h b/tools/arch/parisc/include/uapi/asm/errno.h
index 8cbc07c..93194fb 100644
--- a/tools/arch/parisc/include/uapi/asm/errno.h
+++ b/tools/arch/parisc/include/uapi/asm/errno.h
@@ -124,4 +124,6 @@
#define EHWPOISON 257 /* Memory page has hardware error */
+#define EFTYPE 258 /* Wrong file type for the intended operation */
+
#endif
diff --git a/tools/arch/sparc/include/uapi/asm/errno.h b/tools/arch/sparc/include/uapi/asm/errno.h
index 4a41e78..71940ec 100644
--- a/tools/arch/sparc/include/uapi/asm/errno.h
+++ b/tools/arch/sparc/include/uapi/asm/errno.h
@@ -117,4 +117,6 @@
#define EHWPOISON 135 /* Memory page has hardware error */
+#define EFTYPE 136 /* Wrong file type for the intended operation */
+
#endif
diff --git a/tools/include/nolibc/Makefile b/tools/include/nolibc/Makefile
index 7455097..00fd2e5 100644
--- a/tools/include/nolibc/Makefile
+++ b/tools/include/nolibc/Makefile
@@ -17,9 +17,11 @@
# it defaults to this nolibc directory.
OUTPUT ?= $(CURDIR)/
-architectures := arm arm64 loongarch m68k mips powerpc riscv s390 sh sparc x86
+architectures := arm arm64 loongarch m68k mips openrisc parisc powerpc riscv s390 sh sparc x86
arch_files := arch.h $(addsuffix .h, $(addprefix arch-, $(architectures)))
all_files := \
+ alloca.h \
+ assert.h \
byteswap.h \
compiler.h \
crt.h \
diff --git a/tools/include/nolibc/alloca.h b/tools/include/nolibc/alloca.h
new file mode 100644
index 0000000..448233a
--- /dev/null
+++ b/tools/include/nolibc/alloca.h
@@ -0,0 +1,15 @@
+/* SPDX-License-Identifier: LGPL-2.1 OR MIT */
+/*
+ * alloca() for NOLIBC
+ * Copyright (C) 2026 Thomas Weißschuh <linux@weissschuh.net>
+ */
+
+/* make sure to include all global symbols */
+#include "nolibc.h"
+
+#ifndef _NOLIBC_ALLOCA_H
+#define _NOLIBC_ALLOCA_H
+
+#define alloca(size) __builtin_alloca(size)
+
+#endif /* _NOLIBC_ALLOCA_H */
diff --git a/tools/include/nolibc/arch-arm.h b/tools/include/nolibc/arch-arm.h
index a4d3a77..8681922 100644
--- a/tools/include/nolibc/arch-arm.h
+++ b/tools/include/nolibc/arch-arm.h
@@ -7,8 +7,11 @@
#ifndef _NOLIBC_ARCH_ARM_H
#define _NOLIBC_ARCH_ARM_H
+#include <linux/unistd.h>
+
#include "compiler.h"
#include "crt.h"
+#include "std.h"
/* Syscalls for ARM in ARM or Thumb modes :
* - registers are 32-bit
@@ -186,7 +189,7 @@
#ifndef NOLIBC_NO_RUNTIME
/* startup code */
-void __attribute__((weak, noreturn)) __nolibc_entrypoint __no_stack_protector _start(void)
+void __attribute__((weak, noreturn)) __nolibc_entrypoint __nolibc_no_stack_protector _start(void)
{
__asm__ volatile (
"mov r0, sp\n" /* save stack pointer to %r0, as arg1 of _start_c */
@@ -196,4 +199,11 @@ void __attribute__((weak, noreturn)) __nolibc_entrypoint __no_stack_protector _s
}
#endif /* NOLIBC_NO_RUNTIME */
+static __attribute__((unused))
+int _sys_ftruncate64(int fd, uint32_t length0, uint32_t length1)
+{
+ return __nolibc_syscall4(__NR_ftruncate64, fd, 0, length0, length1);
+}
+#define _sys_ftruncate64 _sys_ftruncate64
+
#endif /* _NOLIBC_ARCH_ARM_H */
diff --git a/tools/include/nolibc/arch-arm64.h b/tools/include/nolibc/arch-arm64.h
index 28b3c75..814bcc1 100644
--- a/tools/include/nolibc/arch-arm64.h
+++ b/tools/include/nolibc/arch-arm64.h
@@ -143,7 +143,7 @@
#ifndef NOLIBC_NO_RUNTIME
/* startup code */
-void __attribute__((weak, noreturn)) __nolibc_entrypoint __no_stack_protector _start(void)
+void __attribute__((weak, noreturn)) __nolibc_entrypoint __nolibc_no_stack_protector _start(void)
{
__asm__ volatile (
"mov x0, sp\n" /* save stack pointer to x0, as arg1 of _start_c */
diff --git a/tools/include/nolibc/arch-loongarch.h b/tools/include/nolibc/arch-loongarch.h
index 86fb34b..3abed96 100644
--- a/tools/include/nolibc/arch-loongarch.h
+++ b/tools/include/nolibc/arch-loongarch.h
@@ -144,7 +144,7 @@
#ifndef NOLIBC_NO_RUNTIME
/* startup code */
-void __attribute__((weak, noreturn)) __nolibc_entrypoint __no_stack_protector _start(void)
+void __attribute__((weak, noreturn)) __nolibc_entrypoint __nolibc_no_stack_protector _start(void)
{
__asm__ volatile (
"move $a0, $sp\n" /* save stack pointer to $a0, as arg1 of _start_c */
diff --git a/tools/include/nolibc/arch-m68k.h b/tools/include/nolibc/arch-m68k.h
index 81d34c2..341f434 100644
--- a/tools/include/nolibc/arch-m68k.h
+++ b/tools/include/nolibc/arch-m68k.h
@@ -130,7 +130,7 @@
#ifndef NOLIBC_NO_RUNTIME
void _start(void);
-void __attribute__((weak, noreturn)) __nolibc_entrypoint __no_stack_protector _start(void)
+void __attribute__((weak, noreturn)) __nolibc_entrypoint __nolibc_no_stack_protector _start(void)
{
__asm__ volatile (
"movel %sp, %sp@-\n"
diff --git a/tools/include/nolibc/arch-mips.h b/tools/include/nolibc/arch-mips.h
index bb9d580..26ad413 100644
--- a/tools/include/nolibc/arch-mips.h
+++ b/tools/include/nolibc/arch-mips.h
@@ -7,8 +7,11 @@
#ifndef _NOLIBC_ARCH_MIPS_H
#define _NOLIBC_ARCH_MIPS_H
+#include <linux/unistd.h>
+
#include "compiler.h"
#include "crt.h"
+#include "std.h"
#if !defined(_ABIO32) && !defined(_ABIN32) && !defined(_ABI64)
#error Unsupported MIPS ABI
@@ -55,6 +58,8 @@
#define _NOLIBC_SYSCALL_STACK_RESERVE "addiu $sp, $sp, -32\n"
#define _NOLIBC_SYSCALL_STACK_UNRESERVE "addiu $sp, $sp, 32\n"
+#define _NOLIBC_SYSCALL_REG register long
+
#else /* _ABIN32 || _ABI64 */
/* binutils, GCC and clang disagree about register aliases, use numbers instead. */
@@ -66,12 +71,14 @@
#define _NOLIBC_SYSCALL_STACK_RESERVE
#define _NOLIBC_SYSCALL_STACK_UNRESERVE
+#define _NOLIBC_SYSCALL_REG register long long
+
#endif /* _ABIO32 */
#define __nolibc_syscall0(num) \
({ \
- register long _num __asm__ ("v0") = (num); \
- register long _arg4 __asm__ ("a3"); \
+ _NOLIBC_SYSCALL_REG _num __asm__ ("v0") = (num); \
+ _NOLIBC_SYSCALL_REG _arg4 __asm__ ("a3"); \
\
__asm__ volatile ( \
_NOLIBC_SYSCALL_STACK_RESERVE \
@@ -86,9 +93,9 @@
#define __nolibc_syscall1(num, arg1) \
({ \
- register long _num __asm__ ("v0") = (num); \
- register long _arg1 __asm__ ("a0") = (long)(arg1); \
- register long _arg4 __asm__ ("a3"); \
+ _NOLIBC_SYSCALL_REG _num __asm__ ("v0") = (num); \
+ _NOLIBC_SYSCALL_REG _arg1 __asm__ ("a0") = __nolibc_arg_to_reg(arg1); \
+ _NOLIBC_SYSCALL_REG _arg4 __asm__ ("a3"); \
\
__asm__ volatile ( \
_NOLIBC_SYSCALL_STACK_RESERVE \
@@ -104,10 +111,10 @@
#define __nolibc_syscall2(num, arg1, arg2) \
({ \
- register long _num __asm__ ("v0") = (num); \
- register long _arg1 __asm__ ("a0") = (long)(arg1); \
- register long _arg2 __asm__ ("a1") = (long)(arg2); \
- register long _arg4 __asm__ ("a3"); \
+ _NOLIBC_SYSCALL_REG _num __asm__ ("v0") = (num); \
+ _NOLIBC_SYSCALL_REG _arg1 __asm__ ("a0") = __nolibc_arg_to_reg(arg1); \
+ _NOLIBC_SYSCALL_REG _arg2 __asm__ ("a1") = __nolibc_arg_to_reg(arg2); \
+ _NOLIBC_SYSCALL_REG _arg4 __asm__ ("a3"); \
\
__asm__ volatile ( \
_NOLIBC_SYSCALL_STACK_RESERVE \
@@ -123,11 +130,11 @@
#define __nolibc_syscall3(num, arg1, arg2, arg3) \
({ \
- register long _num __asm__ ("v0") = (num); \
- register long _arg1 __asm__ ("a0") = (long)(arg1); \
- register long _arg2 __asm__ ("a1") = (long)(arg2); \
- register long _arg3 __asm__ ("a2") = (long)(arg3); \
- register long _arg4 __asm__ ("a3"); \
+ _NOLIBC_SYSCALL_REG _num __asm__ ("v0") = (num); \
+ _NOLIBC_SYSCALL_REG _arg1 __asm__ ("a0") = __nolibc_arg_to_reg(arg1); \
+ _NOLIBC_SYSCALL_REG _arg2 __asm__ ("a1") = __nolibc_arg_to_reg(arg2); \
+ _NOLIBC_SYSCALL_REG _arg3 __asm__ ("a2") = __nolibc_arg_to_reg(arg3); \
+ _NOLIBC_SYSCALL_REG _arg4 __asm__ ("a3"); \
\
__asm__ volatile ( \
_NOLIBC_SYSCALL_STACK_RESERVE \
@@ -143,11 +150,11 @@
#define __nolibc_syscall4(num, arg1, arg2, arg3, arg4) \
({ \
- register long _num __asm__ ("v0") = (num); \
- register long _arg1 __asm__ ("a0") = (long)(arg1); \
- register long _arg2 __asm__ ("a1") = (long)(arg2); \
- register long _arg3 __asm__ ("a2") = (long)(arg3); \
- register long _arg4 __asm__ ("a3") = (long)(arg4); \
+ _NOLIBC_SYSCALL_REG _num __asm__ ("v0") = (num); \
+ _NOLIBC_SYSCALL_REG _arg1 __asm__ ("a0") = __nolibc_arg_to_reg(arg1); \
+ _NOLIBC_SYSCALL_REG _arg2 __asm__ ("a1") = __nolibc_arg_to_reg(arg2); \
+ _NOLIBC_SYSCALL_REG _arg3 __asm__ ("a2") = __nolibc_arg_to_reg(arg3); \
+ _NOLIBC_SYSCALL_REG _arg4 __asm__ ("a3") = __nolibc_arg_to_reg(arg4); \
\
__asm__ volatile ( \
_NOLIBC_SYSCALL_STACK_RESERVE \
@@ -165,12 +172,12 @@
#define __nolibc_syscall5(num, arg1, arg2, arg3, arg4, arg5) \
({ \
- register long _num __asm__ ("v0") = (num); \
- register long _arg1 __asm__ ("a0") = (long)(arg1); \
- register long _arg2 __asm__ ("a1") = (long)(arg2); \
- register long _arg3 __asm__ ("a2") = (long)(arg3); \
- register long _arg4 __asm__ ("a3") = (long)(arg4); \
- register long _arg5 = (long)(arg5); \
+ _NOLIBC_SYSCALL_REG _num __asm__ ("v0") = (num); \
+ _NOLIBC_SYSCALL_REG _arg1 __asm__ ("a0") = __nolibc_arg_to_reg(arg1); \
+ _NOLIBC_SYSCALL_REG _arg2 __asm__ ("a1") = __nolibc_arg_to_reg(arg2); \
+ _NOLIBC_SYSCALL_REG _arg3 __asm__ ("a2") = __nolibc_arg_to_reg(arg3); \
+ _NOLIBC_SYSCALL_REG _arg4 __asm__ ("a3") = __nolibc_arg_to_reg(arg4); \
+ _NOLIBC_SYSCALL_REG _arg5 = __nolibc_arg_to_reg(arg5); \
\
__asm__ volatile ( \
_NOLIBC_SYSCALL_STACK_RESERVE \
@@ -187,13 +194,13 @@
#define __nolibc_syscall6(num, arg1, arg2, arg3, arg4, arg5, arg6) \
({ \
- register long _num __asm__ ("v0") = (num); \
- register long _arg1 __asm__ ("a0") = (long)(arg1); \
- register long _arg2 __asm__ ("a1") = (long)(arg2); \
- register long _arg3 __asm__ ("a2") = (long)(arg3); \
- register long _arg4 __asm__ ("a3") = (long)(arg4); \
- register long _arg5 = (long)(arg5); \
- register long _arg6 = (long)(arg6); \
+ _NOLIBC_SYSCALL_REG _num __asm__ ("v0") = (num); \
+ _NOLIBC_SYSCALL_REG _arg1 __asm__ ("a0") = __nolibc_arg_to_reg(arg1); \
+ _NOLIBC_SYSCALL_REG _arg2 __asm__ ("a1") = __nolibc_arg_to_reg(arg2); \
+ _NOLIBC_SYSCALL_REG _arg3 __asm__ ("a2") = __nolibc_arg_to_reg(arg3); \
+ _NOLIBC_SYSCALL_REG _arg4 __asm__ ("a3") = __nolibc_arg_to_reg(arg4); \
+ _NOLIBC_SYSCALL_REG _arg5 = __nolibc_arg_to_reg(arg5); \
+ _NOLIBC_SYSCALL_REG _arg6 = __nolibc_arg_to_reg(arg6); \
\
__asm__ volatile ( \
_NOLIBC_SYSCALL_STACK_RESERVE \
@@ -214,12 +221,12 @@
#define __nolibc_syscall5(num, arg1, arg2, arg3, arg4, arg5) \
({ \
- register long _num __asm__ ("v0") = (num); \
- register long _arg1 __asm__ ("$4") = (long)(arg1); \
- register long _arg2 __asm__ ("$5") = (long)(arg2); \
- register long _arg3 __asm__ ("$6") = (long)(arg3); \
- register long _arg4 __asm__ ("$7") = (long)(arg4); \
- register long _arg5 __asm__ ("$8") = (long)(arg5); \
+ _NOLIBC_SYSCALL_REG _num __asm__ ("v0") = (num); \
+ _NOLIBC_SYSCALL_REG _arg1 __asm__ ("$4") = __nolibc_arg_to_reg(arg1); \
+ _NOLIBC_SYSCALL_REG _arg2 __asm__ ("$5") = __nolibc_arg_to_reg(arg2); \
+ _NOLIBC_SYSCALL_REG _arg3 __asm__ ("$6") = __nolibc_arg_to_reg(arg3); \
+ _NOLIBC_SYSCALL_REG _arg4 __asm__ ("$7") = __nolibc_arg_to_reg(arg4); \
+ _NOLIBC_SYSCALL_REG _arg5 __asm__ ("$8") = __nolibc_arg_to_reg(arg5); \
\
__asm__ volatile ( \
"syscall\n" \
@@ -233,13 +240,13 @@
#define __nolibc_syscall6(num, arg1, arg2, arg3, arg4, arg5, arg6) \
({ \
- register long _num __asm__ ("v0") = (num); \
- register long _arg1 __asm__ ("$4") = (long)(arg1); \
- register long _arg2 __asm__ ("$5") = (long)(arg2); \
- register long _arg3 __asm__ ("$6") = (long)(arg3); \
- register long _arg4 __asm__ ("$7") = (long)(arg4); \
- register long _arg5 __asm__ ("$8") = (long)(arg5); \
- register long _arg6 __asm__ ("$9") = (long)(arg6); \
+ _NOLIBC_SYSCALL_REG _num __asm__ ("v0") = (num); \
+ _NOLIBC_SYSCALL_REG _arg1 __asm__ ("$4") = __nolibc_arg_to_reg(arg1); \
+ _NOLIBC_SYSCALL_REG _arg2 __asm__ ("$5") = __nolibc_arg_to_reg(arg2); \
+ _NOLIBC_SYSCALL_REG _arg3 __asm__ ("$6") = __nolibc_arg_to_reg(arg3); \
+ _NOLIBC_SYSCALL_REG _arg4 __asm__ ("$7") = __nolibc_arg_to_reg(arg4); \
+ _NOLIBC_SYSCALL_REG _arg5 __asm__ ("$8") = __nolibc_arg_to_reg(arg5); \
+ _NOLIBC_SYSCALL_REG _arg6 __asm__ ("$9") = __nolibc_arg_to_reg(arg6); \
\
__asm__ volatile ( \
"syscall\n" \
@@ -257,7 +264,7 @@
#ifndef NOLIBC_NO_RUNTIME
/* startup code, note that it's called __start on MIPS */
void __start(void);
-void __attribute__((weak, noreturn)) __nolibc_entrypoint __no_stack_protector __start(void)
+void __attribute__((weak, noreturn)) __nolibc_entrypoint __nolibc_no_stack_protector __start(void)
{
__asm__ volatile (
"move $a0, $sp\n" /* save stack pointer to $a0, as arg1 of _start_c */
@@ -278,4 +285,13 @@ void __attribute__((weak, noreturn)) __nolibc_entrypoint __no_stack_protector __
}
#endif /* NOLIBC_NO_RUNTIME */
+#if defined(_ABIO32)
+static __attribute__((unused))
+int _sys_ftruncate64(int fd, uint32_t length0, uint32_t length1)
+{
+ return __nolibc_syscall4(__NR_ftruncate64, fd, 0, length0, length1);
+}
+#define _sys_ftruncate64 _sys_ftruncate64
+#endif
+
#endif /* _NOLIBC_ARCH_MIPS_H */
diff --git a/tools/include/nolibc/arch-openrisc.h b/tools/include/nolibc/arch-openrisc.h
new file mode 100644
index 0000000..5ef82fd
--- /dev/null
+++ b/tools/include/nolibc/arch-openrisc.h
@@ -0,0 +1,160 @@
+/* SPDX-License-Identifier: LGPL-2.1 OR MIT */
+/*
+ * OpenRISC specific definitions for NOLIBC
+ * Copyright (C) 2026 Thomas Weißschuh <linux@weissschuh.net>
+ */
+
+#ifndef _NOLIBC_ARCH_OPENRISC_H
+#define _NOLIBC_ARCH_OPENRISC_H
+
+#include "compiler.h"
+#include "crt.h"
+
+/*
+ * Syscalls for OpenRISC:
+ * - syscall number is passed in r11
+ * - arguments are in r3, r4, r5, r6, r7, r8
+ * - the system call is performed by calling l.sys 1
+ * - syscall return value is in r11
+ */
+
+#define _NOLIBC_SYSCALL_CLOBBERLIST \
+ "r12", "r13", "r15", "r17", "r19", "r21", "r23", "r25", "r27", "r29", "r31", "memory"
+
+#define __nolibc_syscall0(num) \
+({ \
+ register long _num __asm__ ("r11") = (num); \
+ \
+ __asm__ volatile ( \
+ "l.sys 1\n" \
+ : "+r"(_num) \
+ : \
+ : "r3", "r4", "r5", "r6", "r7", "r8", \
+ _NOLIBC_SYSCALL_CLOBBERLIST \
+ ); \
+ _num; \
+})
+
+#define __nolibc_syscall1(num, arg1) \
+({ \
+ register long _num __asm__ ("r11") = (num); \
+ register long _arg1 __asm__ ("r3") = (long)(arg1); \
+ \
+ __asm__ volatile ( \
+ "l.sys 1\n" \
+ : "+r"(_num) \
+ : "r"(_arg1) \
+ : "r4", "r5", "r6", "r7", "r8", _NOLIBC_SYSCALL_CLOBBERLIST \
+ ); \
+ _num; \
+})
+
+#define __nolibc_syscall2(num, arg1, arg2) \
+({ \
+ register long _num __asm__ ("r11") = (num); \
+ register long _arg1 __asm__ ("r3") = (long)(arg1); \
+ register long _arg2 __asm__ ("r4") = (long)(arg2); \
+ \
+ __asm__ volatile ( \
+ "l.sys 1\n" \
+ : "+r"(_num) \
+ : "r"(_arg1), "r"(_arg2) \
+ : "r5", "r6", "r7", "r8", _NOLIBC_SYSCALL_CLOBBERLIST \
+ ); \
+ _num; \
+})
+
+#define __nolibc_syscall3(num, arg1, arg2, arg3) \
+({ \
+ register long _num __asm__ ("r11") = (num); \
+ register long _arg1 __asm__ ("r3") = (long)(arg1); \
+ register long _arg2 __asm__ ("r4") = (long)(arg2); \
+ register long _arg3 __asm__ ("r5") = (long)(arg3); \
+ \
+ __asm__ volatile ( \
+ "l.sys 1\n" \
+ : "+r"(_num) \
+ : "r"(_arg1), "r"(_arg2), "r"(_arg3) \
+ : "r6", "r7", "r8", _NOLIBC_SYSCALL_CLOBBERLIST \
+ ); \
+ _num; \
+})
+
+#define __nolibc_syscall4(num, arg1, arg2, arg3, arg4) \
+({ \
+ register long _num __asm__ ("r11") = (num); \
+ register long _arg1 __asm__ ("r3") = (long)(arg1); \
+ register long _arg2 __asm__ ("r4") = (long)(arg2); \
+ register long _arg3 __asm__ ("r5") = (long)(arg3); \
+ register long _arg4 __asm__ ("r6") = (long)(arg4); \
+ \
+ __asm__ volatile ( \
+ "l.sys 1\n" \
+ : "+r"(_num) \
+ : "r"(_arg1), "r"(_arg2), "r"(_arg3), "r"(_arg4) \
+ : "r7", "r8", _NOLIBC_SYSCALL_CLOBBERLIST \
+ ); \
+ _num; \
+})
+
+#define __nolibc_syscall5(num, arg1, arg2, arg3, arg4, arg5) \
+({ \
+ register long _num __asm__ ("r11") = (num); \
+ register long _arg1 __asm__ ("r3") = (long)(arg1); \
+ register long _arg2 __asm__ ("r4") = (long)(arg2); \
+ register long _arg3 __asm__ ("r5") = (long)(arg3); \
+ register long _arg4 __asm__ ("r6") = (long)(arg4); \
+ register long _arg5 __asm__ ("r7") = (long)(arg5); \
+ \
+ __asm__ volatile ( \
+ "l.sys 1\n" \
+ : "+r"(_num) \
+ : "r"(_arg1), "r"(_arg2), "r"(_arg3), "r"(_arg4), "r"(_arg5) \
+ : "r8", _NOLIBC_SYSCALL_CLOBBERLIST \
+ ); \
+ _num; \
+})
+
+#define __nolibc_syscall6(num, arg1, arg2, arg3, arg4, arg5, arg6) \
+({ \
+ register long _num __asm__ ("r11") = (num); \
+ register long _arg1 __asm__ ("r3") = (long)(arg1); \
+ register long _arg2 __asm__ ("r4") = (long)(arg2); \
+ register long _arg3 __asm__ ("r5") = (long)(arg3); \
+ register long _arg4 __asm__ ("r6") = (long)(arg4); \
+ register long _arg5 __asm__ ("r7") = (long)(arg5); \
+ register long _arg6 __asm__ ("r8") = (long)(arg6); \
+ \
+ __asm__ volatile ( \
+ "l.sys 1\n" \
+ : "+r"(_num) \
+ : "r"(_arg1), "r"(_arg2), "r"(_arg3), "r"(_arg4), "r"(_arg5), \
+ "r"(_arg6) \
+ : _NOLIBC_SYSCALL_CLOBBERLIST \
+ ); \
+ _num; \
+})
+
+#ifndef NOLIBC_NO_RUNTIME
+/* startup code */
+void _start_wrapper(void);
+void __attribute__((weak,noreturn))
+__nolibc_entrypoint __nolibc_no_stack_protector
+_start_wrapper(void)
+{
+ __asm__ volatile (
+ ".global _start\n" /* The C function will have a prologue, */
+ ".type _start, @function\n" /* corrupting "sp/r1" */
+ ".weak _start\n"
+ "_start:\n"
+
+ "l.jal _start_c\n" /* transfer to c runtime */
+ "l.or r3,r1,r1\n" /* save stack pointer to r3, as arg1 of _start_c */
+
+ ".size _start, .-_start\n"
+ );
+ __nolibc_entrypoint_epilogue();
+}
+#endif /* NOLIBC_NO_RUNTIME */
+
+#endif /* _NOLIBC_ARCH_OPENRISC_H */
diff --git a/tools/include/nolibc/arch-parisc.h b/tools/include/nolibc/arch-parisc.h
new file mode 100644
index 0000000..417043e
--- /dev/null
+++ b/tools/include/nolibc/arch-parisc.h
@@ -0,0 +1,185 @@
+/* SPDX-License-Identifier: LGPL-2.1 OR MIT */
+/*
+ * parisc/hppa (32-bit) specific definitions for NOLIBC
+ * Copyright (C) 2026 Thomas Weißschuh <linux@weissschuh.net>
+ */
+
+#ifndef _NOLIBC_ARCH_PARISC_H
+#define _NOLIBC_ARCH_PARISC_H
+
+#if defined(__LP64__)
+#error 64-bit not supported
+#endif
+
+#include "compiler.h"
+#include "crt.h"
+
+/* Syscalls for parisc :
+ * - syscall number is passed in r20
+ * - arguments are in r26 to r21
+ * - the system call is performed by calling "ble 0x100(%sr2, %r0)",
+ * the instruction after that is in the delay slot and executed before
+ * the jump to 0x100 actually happens, use it to load the syscall number
+ * - syscall return comes in r28
+ * - the arguments are cast to long and assigned into the target
+ * registers which are then simply passed as registers to the asm code,
+ * so that we don't have to experience issues with register constraints.
+ */
+
+#define _NOLIBC_SYSCALL_CLOBBERLIST \
+ "memory", "%r1", "%r2", "%r4", "%r20", "%r29", "%r31"
+
+#define __nolibc_syscall0(num) \
+({ \
+ register long _ret __asm__ ("r28"); \
+ \
+ __asm__ volatile ( \
+ "ble 0x100(%%sr2, %%r0)\n\t" \
+ "copy %1, %%r20\n\t" \
+ : "=r"(_ret) \
+ : "r"(num) \
+ : _NOLIBC_SYSCALL_CLOBBERLIST, \
+ "%r21", "%r22", "%r23", "%r24", "%r25", "%r26" \
+ ); \
+ _ret; \
+})
+
+#define __nolibc_syscall1(num, arg1) \
+({ \
+ register long _ret __asm__ ("r28"); \
+ register long _arg1 __asm__ ("r26") = (long)(arg1); \
+ \
+ __asm__ volatile ( \
+ "ble 0x100(%%sr2, %%r0)\n\t" \
+ "copy %2, %%r20\n\t" \
+ : "=r"(_ret), \
+ "+r"(_arg1) \
+ : "r"(num) \
+ : _NOLIBC_SYSCALL_CLOBBERLIST, \
+ "%r21", "%r22", "%r23", "%r24", "%r25" \
+ ); \
+ _ret; \
+})
+
+#define __nolibc_syscall2(num, arg1, arg2) \
+({ \
+ register long _ret __asm__ ("r28"); \
+ register long _arg1 __asm__ ("r26") = (long)(arg1); \
+ register long _arg2 __asm__ ("r25") = (long)(arg2); \
+ \
+ __asm__ volatile ( \
+ "ble 0x100(%%sr2, %%r0)\n\t" \
+ "copy %3, %%r20\n\t" \
+ : "=r"(_ret), \
+ "+r"(_arg1), "+r"(_arg2) \
+ : "r"(num) \
+ : _NOLIBC_SYSCALL_CLOBBERLIST, \
+ "%r21", "%r22", "%r23", "%r24" \
+ ); \
+ _ret; \
+})
+
+#define __nolibc_syscall3(num, arg1, arg2, arg3) \
+({ \
+ register long _ret __asm__ ("r28"); \
+ register long _arg1 __asm__ ("r26") = (long)(arg1); \
+ register long _arg2 __asm__ ("r25") = (long)(arg2); \
+ register long _arg3 __asm__ ("r24") = (long)(arg3); \
+ \
+ __asm__ volatile ( \
+ "ble 0x100(%%sr2, %%r0)\n\t" \
+ "copy %4, %%r20\n\t" \
+ : "=r"(_ret), \
+ "+r"(_arg1), "+r"(_arg2), "+r"(_arg3) \
+ : "r"(num) \
+ : _NOLIBC_SYSCALL_CLOBBERLIST, \
+ "%r21", "%r22", "%r23" \
+ ); \
+ _ret; \
+})
+
+#define __nolibc_syscall4(num, arg1, arg2, arg3, arg4) \
+({ \
+ register long _ret __asm__ ("r28"); \
+ register long _arg1 __asm__ ("r26") = (long)(arg1); \
+ register long _arg2 __asm__ ("r25") = (long)(arg2); \
+ register long _arg3 __asm__ ("r24") = (long)(arg3); \
+ register long _arg4 __asm__ ("r23") = (long)(arg4); \
+ \
+ __asm__ volatile ( \
+ "ble 0x100(%%sr2, %%r0)\n\t" \
+ "copy %5, %%r20\n\t" \
+ : "=r"(_ret), \
+ "+r"(_arg1), "+r"(_arg2), "+r"(_arg3), "+r"(_arg4) \
+ : "r"(num) \
+ : _NOLIBC_SYSCALL_CLOBBERLIST, \
+ "%r21", "%r22" \
+ ); \
+ _ret; \
+})
+
+#define __nolibc_syscall5(num, arg1, arg2, arg3, arg4, arg5) \
+({ \
+ register long _ret __asm__ ("r28"); \
+ register long _arg1 __asm__ ("r26") = (long)(arg1); \
+ register long _arg2 __asm__ ("r25") = (long)(arg2); \
+ register long _arg3 __asm__ ("r24") = (long)(arg3); \
+ register long _arg4 __asm__ ("r23") = (long)(arg4); \
+ register long _arg5 __asm__ ("r22") = (long)(arg5); \
+ \
+ __asm__ volatile ( \
+ "ble 0x100(%%sr2, %%r0)\n\t" \
+ "copy %6, %%r20\n\t" \
+ : "=r"(_ret), \
+ "+r"(_arg1), "+r"(_arg2), "+r"(_arg3), "+r"(_arg4), \
+ "+r"(_arg5) \
+ : "r"(num) \
+ : _NOLIBC_SYSCALL_CLOBBERLIST, \
+ "%r21" \
+ ); \
+ _ret; \
+})
+
+#define __nolibc_syscall6(num, arg1, arg2, arg3, arg4, arg5, arg6) \
+({ \
+ register long _ret __asm__ ("r28"); \
+ register long _arg1 __asm__ ("r26") = (long)(arg1); \
+ register long _arg2 __asm__ ("r25") = (long)(arg2); \
+ register long _arg3 __asm__ ("r24") = (long)(arg3); \
+ register long _arg4 __asm__ ("r23") = (long)(arg4); \
+ register long _arg5 __asm__ ("r22") = (long)(arg5); \
+ register long _arg6 __asm__ ("r21") = (long)(arg6); \
+ \
+ __asm__ volatile ( \
+ "ble 0x100(%%sr2, %%r0)\n\t" \
+ "copy %7, %%r20\n\t" \
+ : "=r"(_ret), \
+ "+r"(_arg1), "+r"(_arg2), "+r"(_arg3), "+r"(_arg4), \
+ "+r"(_arg5), "+r"(_arg6) \
+ : "r"(num) \
+ : _NOLIBC_SYSCALL_CLOBBERLIST \
+ ); \
+ _ret; \
+})
+
+#ifndef NOLIBC_NO_RUNTIME
+/* startup code */
+void __attribute__((weak, noreturn)) __nolibc_entrypoint __nolibc_no_stack_protector _start(void)
+{
+ __asm__ volatile (
+ ".import $global$\n" /* Set up the dp register */
+ "ldil L%$global$, %dp\n"
+ "ldo R%$global$(%r27), %dp\n"
+
+ "b _start_c\n" /* Call _start_c, the load below is executed first */
+
+ "ldo -4(%r24), %r26\n" /* The sp register is special on parisc.
+ * r24 points to argv. Subtract 4 to get &argc.
+ * Pass that as first argument to _start_c.
+ */
+ );
+ __nolibc_entrypoint_epilogue();
+}
+#endif /* NOLIBC_NO_RUNTIME */
+
+#endif /* _NOLIBC_ARCH_PARISC_H */
diff --git a/tools/include/nolibc/arch-powerpc.h b/tools/include/nolibc/arch-powerpc.h
index ef87886..a1ab91d 100644
--- a/tools/include/nolibc/arch-powerpc.h
+++ b/tools/include/nolibc/arch-powerpc.h
@@ -7,8 +7,11 @@
#ifndef _NOLIBC_ARCH_POWERPC_H
#define _NOLIBC_ARCH_POWERPC_H
+#include <linux/unistd.h>
+
#include "compiler.h"
#include "crt.h"
+#include "std.h"
/* Syscalls for PowerPC :
* - stack is 16-byte aligned
@@ -177,15 +180,15 @@
* "omit-frame-pointer" fails with __attribute__((no_stack_protector)) but
* works with __attribute__((__optimize__("-fno-stack-protector")))
*/
-#ifdef __no_stack_protector
-#undef __no_stack_protector
-#define __no_stack_protector __attribute__((__optimize__("-fno-stack-protector")))
+#ifdef __nolibc_no_stack_protector
+#undef __nolibc_no_stack_protector
+#define __nolibc_no_stack_protector __attribute__((__optimize__("-fno-stack-protector")))
#endif
#endif /* !__powerpc64__ */
#ifndef NOLIBC_NO_RUNTIME
/* startup code */
-void __attribute__((weak, noreturn)) __nolibc_entrypoint __no_stack_protector _start(void)
+void __attribute__((weak, noreturn)) __nolibc_entrypoint __nolibc_no_stack_protector _start(void)
{
#ifdef __powerpc64__
#if _CALL_ELF == 2
@@ -218,4 +221,13 @@ void __attribute__((weak, noreturn)) __nolibc_entrypoint __no_stack_protector _s
}
#endif /* NOLIBC_NO_RUNTIME */
+#if !defined(__powerpc64__)
+static __attribute__((unused))
+int _sys_ftruncate64(int fd, uint32_t length0, uint32_t length1)
+{
+ return __nolibc_syscall4(__NR_ftruncate64, fd, 0, length0, length1);
+}
+#define _sys_ftruncate64 _sys_ftruncate64
+#endif
+
#endif /* _NOLIBC_ARCH_POWERPC_H */
diff --git a/tools/include/nolibc/arch-riscv.h b/tools/include/nolibc/arch-riscv.h
index 386ebb9..1e84ed2 100644
--- a/tools/include/nolibc/arch-riscv.h
+++ b/tools/include/nolibc/arch-riscv.h
@@ -141,7 +141,7 @@
#ifndef NOLIBC_NO_RUNTIME
/* startup code */
-void __attribute__((weak, noreturn)) __nolibc_entrypoint __no_stack_protector _start(void)
+void __attribute__((weak, noreturn)) __nolibc_entrypoint __nolibc_no_stack_protector _start(void)
{
__asm__ volatile (
".option push\n"
diff --git a/tools/include/nolibc/arch-s390.h b/tools/include/nolibc/arch-s390.h
index 4e69123..3f05c01 100644
--- a/tools/include/nolibc/arch-s390.h
+++ b/tools/include/nolibc/arch-s390.h
@@ -145,7 +145,7 @@
#ifndef NOLIBC_NO_RUNTIME
/* startup code */
-void __attribute__((weak, noreturn)) __nolibc_entrypoint __no_stack_protector _start(void)
+void __attribute__((weak, noreturn)) __nolibc_entrypoint __nolibc_no_stack_protector _start(void)
{
__asm__ volatile (
"lgr %r2, %r15\n" /* save stack pointer to %r2, as arg1 of _start_c */
diff --git a/tools/include/nolibc/arch-sh.h b/tools/include/nolibc/arch-sh.h
index b5a64ce..a32378f 100644
--- a/tools/include/nolibc/arch-sh.h
+++ b/tools/include/nolibc/arch-sh.h
@@ -143,7 +143,7 @@
#ifndef NOLIBC_NO_RUNTIME
/* startup code */
void _start_wrapper(void);
-void __attribute__((weak,noreturn)) __nolibc_entrypoint __no_stack_protector _start_wrapper(void)
+void __attribute__((weak, noreturn)) __nolibc_entrypoint __nolibc_no_stack_protector _start_wrapper(void)
{
__asm__ volatile (
".global _start\n" /* The C function will have a prologue, */
diff --git a/tools/include/nolibc/arch-sparc.h b/tools/include/nolibc/arch-sparc.h
index 240539d..ddae9bc1 100644
--- a/tools/include/nolibc/arch-sparc.h
+++ b/tools/include/nolibc/arch-sparc.h
@@ -154,7 +154,7 @@
#ifndef NOLIBC_NO_RUNTIME
/* startup code */
-void __attribute__((weak, noreturn)) __nolibc_entrypoint __no_stack_protector _start(void)
+void __attribute__((weak, noreturn)) __nolibc_entrypoint __nolibc_no_stack_protector _start(void)
{
__asm__ volatile (
/*
diff --git a/tools/include/nolibc/arch-x86.h b/tools/include/nolibc/arch-x86.h
index 769ba01..fe152ac 100644
--- a/tools/include/nolibc/arch-x86.h
+++ b/tools/include/nolibc/arch-x86.h
@@ -165,7 +165,7 @@
* 2) The deepest stack frame should be set to zero
*
*/
-void __attribute__((weak, noreturn)) __nolibc_entrypoint __no_stack_protector _start(void)
+void __attribute__((weak, noreturn)) __nolibc_entrypoint __nolibc_no_stack_protector _start(void)
{
__asm__ volatile (
"xor %ebp, %ebp\n" /* zero the stack frame */
@@ -202,8 +202,8 @@ void __attribute__((weak, noreturn)) __nolibc_entrypoint __no_stack_protector _s
#define __nolibc_syscall0(num) \
({ \
- long _ret; \
- register long _num __asm__ ("rax") = (num); \
+ long long _ret; \
+ register long long _num __asm__ ("rax") = (num); \
\
__asm__ volatile ( \
"syscall\n" \
@@ -216,9 +216,9 @@ void __attribute__((weak, noreturn)) __nolibc_entrypoint __no_stack_protector _s
#define __nolibc_syscall1(num, arg1) \
({ \
- long _ret; \
- register long _num __asm__ ("rax") = (num); \
- register long _arg1 __asm__ ("rdi") = (long)(arg1); \
+ long long _ret; \
+ register long long _num __asm__ ("rax") = (num); \
+ register long long _arg1 __asm__ ("rdi") = __nolibc_arg_to_reg(arg1); \
\
__asm__ volatile ( \
"syscall\n" \
@@ -232,10 +232,10 @@ void __attribute__((weak, noreturn)) __nolibc_entrypoint __no_stack_protector _s
#define __nolibc_syscall2(num, arg1, arg2) \
({ \
- long _ret; \
- register long _num __asm__ ("rax") = (num); \
- register long _arg1 __asm__ ("rdi") = (long)(arg1); \
- register long _arg2 __asm__ ("rsi") = (long)(arg2); \
+ long long _ret; \
+ register long long _num __asm__ ("rax") = (num); \
+ register long long _arg1 __asm__ ("rdi") = __nolibc_arg_to_reg(arg1); \
+ register long long _arg2 __asm__ ("rsi") = __nolibc_arg_to_reg(arg2); \
\
__asm__ volatile ( \
"syscall\n" \
@@ -249,11 +249,11 @@ void __attribute__((weak, noreturn)) __nolibc_entrypoint __no_stack_protector _s
#define __nolibc_syscall3(num, arg1, arg2, arg3) \
({ \
- long _ret; \
- register long _num __asm__ ("rax") = (num); \
- register long _arg1 __asm__ ("rdi") = (long)(arg1); \
- register long _arg2 __asm__ ("rsi") = (long)(arg2); \
- register long _arg3 __asm__ ("rdx") = (long)(arg3); \
+ long long _ret; \
+ register long long _num __asm__ ("rax") = (num); \
+ register long long _arg1 __asm__ ("rdi") = __nolibc_arg_to_reg(arg1); \
+ register long long _arg2 __asm__ ("rsi") = __nolibc_arg_to_reg(arg2); \
+ register long long _arg3 __asm__ ("rdx") = __nolibc_arg_to_reg(arg3); \
\
__asm__ volatile ( \
"syscall\n" \
@@ -267,12 +267,12 @@ void __attribute__((weak, noreturn)) __nolibc_entrypoint __no_stack_protector _s
#define __nolibc_syscall4(num, arg1, arg2, arg3, arg4) \
({ \
- long _ret; \
- register long _num __asm__ ("rax") = (num); \
- register long _arg1 __asm__ ("rdi") = (long)(arg1); \
- register long _arg2 __asm__ ("rsi") = (long)(arg2); \
- register long _arg3 __asm__ ("rdx") = (long)(arg3); \
- register long _arg4 __asm__ ("r10") = (long)(arg4); \
+ long long _ret; \
+ register long long _num __asm__ ("rax") = (num); \
+ register long long _arg1 __asm__ ("rdi") = __nolibc_arg_to_reg(arg1); \
+ register long long _arg2 __asm__ ("rsi") = __nolibc_arg_to_reg(arg2); \
+ register long long _arg3 __asm__ ("rdx") = __nolibc_arg_to_reg(arg3); \
+ register long long _arg4 __asm__ ("r10") = __nolibc_arg_to_reg(arg4); \
\
__asm__ volatile ( \
"syscall\n" \
@@ -286,13 +286,13 @@ void __attribute__((weak, noreturn)) __nolibc_entrypoint __no_stack_protector _s
#define __nolibc_syscall5(num, arg1, arg2, arg3, arg4, arg5) \
({ \
- long _ret; \
- register long _num __asm__ ("rax") = (num); \
- register long _arg1 __asm__ ("rdi") = (long)(arg1); \
- register long _arg2 __asm__ ("rsi") = (long)(arg2); \
- register long _arg3 __asm__ ("rdx") = (long)(arg3); \
- register long _arg4 __asm__ ("r10") = (long)(arg4); \
- register long _arg5 __asm__ ("r8") = (long)(arg5); \
+ long long _ret; \
+ register long long _num __asm__ ("rax") = (num); \
+ register long long _arg1 __asm__ ("rdi") = __nolibc_arg_to_reg(arg1); \
+ register long long _arg2 __asm__ ("rsi") = __nolibc_arg_to_reg(arg2); \
+ register long long _arg3 __asm__ ("rdx") = __nolibc_arg_to_reg(arg3); \
+ register long long _arg4 __asm__ ("r10") = __nolibc_arg_to_reg(arg4); \
+ register long long _arg5 __asm__ ("r8") = __nolibc_arg_to_reg(arg5); \
\
__asm__ volatile ( \
"syscall\n" \
@@ -306,14 +306,14 @@ void __attribute__((weak, noreturn)) __nolibc_entrypoint __no_stack_protector _s
#define __nolibc_syscall6(num, arg1, arg2, arg3, arg4, arg5, arg6) \
({ \
- long _ret; \
- register long _num __asm__ ("rax") = (num); \
- register long _arg1 __asm__ ("rdi") = (long)(arg1); \
- register long _arg2 __asm__ ("rsi") = (long)(arg2); \
- register long _arg3 __asm__ ("rdx") = (long)(arg3); \
- register long _arg4 __asm__ ("r10") = (long)(arg4); \
- register long _arg5 __asm__ ("r8") = (long)(arg5); \
- register long _arg6 __asm__ ("r9") = (long)(arg6); \
+ long long _ret; \
+ register long long _num __asm__ ("rax") = (num); \
+ register long long _arg1 __asm__ ("rdi") = __nolibc_arg_to_reg(arg1); \
+ register long long _arg2 __asm__ ("rsi") = __nolibc_arg_to_reg(arg2); \
+ register long long _arg3 __asm__ ("rdx") = __nolibc_arg_to_reg(arg3); \
+ register long long _arg4 __asm__ ("r10") = __nolibc_arg_to_reg(arg4); \
+ register long long _arg5 __asm__ ("r8") = __nolibc_arg_to_reg(arg5); \
+ register long long _arg6 __asm__ ("r9") = __nolibc_arg_to_reg(arg6); \
\
__asm__ volatile ( \
"syscall\n" \
@@ -333,7 +333,7 @@ void __attribute__((weak, noreturn)) __nolibc_entrypoint __no_stack_protector _s
* 2) The deepest stack frame should be zero (the %rbp).
*
*/
-void __attribute__((weak, noreturn)) __nolibc_entrypoint __no_stack_protector _start(void)
+void __attribute__((weak, noreturn)) __nolibc_entrypoint __nolibc_no_stack_protector _start(void)
{
__asm__ volatile (
"xor %ebp, %ebp\n" /* zero the stack frame */
diff --git a/tools/include/nolibc/arch.h b/tools/include/nolibc/arch.h
index a3adaf4..b69d9c5 100644
--- a/tools/include/nolibc/arch.h
+++ b/tools/include/nolibc/arch.h
@@ -28,6 +28,10 @@
#include "arch-m68k.h"
#elif defined(__sh__)
#include "arch-sh.h"
+#elif defined(__or1k__)
+#include "arch-openrisc.h"
+#elif defined(__hppa__)
+#include "arch-parisc.h"
#else
#error Unsupported Architecture
#endif
diff --git a/tools/include/nolibc/assert.h b/tools/include/nolibc/assert.h
new file mode 100644
index 0000000..84ff8ad
--- /dev/null
+++ b/tools/include/nolibc/assert.h
@@ -0,0 +1,36 @@
+/* SPDX-License-Identifier: LGPL-2.1 OR MIT */
+/*
+ * Assert for NOLIBC
+ * Copyright (C) 2026 Thomas Weißschuh <linux@weissschuh.net>
+ */
+
+/* make sure to include all global symbols */
+#include "nolibc.h"
+
+#ifndef _NOLIBC_ASSERT_H
+#define _NOLIBC_ASSERT_H
+
+#include "errno.h"
+#include "stdio.h"
+#include "stdlib.h"
+
+#endif /* _NOLIBC_ASSERT_H */
+
+/* NDEBUG needs to be evaluated on *each* inclusion */
+#ifdef assert
+#undef assert
+#endif
+
+#ifndef NDEBUG
+#define assert(expr) \
+({ \
+ if (!(expr)) { \
+ fprintf(stderr, "%s: %s:%d: %s: Assertion `%s' failed.\n", \
+ program_invocation_short_name, __FILE__, __LINE__, __func__, \
+ #expr); \
+ abort(); \
+ } \
+})
+#else
+#define assert(expr) ((void)0)
+#endif
diff --git a/tools/include/nolibc/compiler.h b/tools/include/nolibc/compiler.h
index b56570b..f2d7a81 100644
--- a/tools/include/nolibc/compiler.h
+++ b/tools/include/nolibc/compiler.h
@@ -36,9 +36,9 @@
#endif /* defined(__SSP__) ... */
#if __nolibc_has_attribute(no_stack_protector)
-# define __no_stack_protector __attribute__((no_stack_protector))
+# define __nolibc_no_stack_protector __attribute__((no_stack_protector))
#else
-# define __no_stack_protector __attribute__((__optimize__("-fno-stack-protector")))
+# define __nolibc_no_stack_protector __attribute__((__optimize__("-fno-stack-protector")))
#endif /* __nolibc_has_attribute(no_stack_protector) */
#if __nolibc_has_attribute(__fallthrough__)
diff --git a/tools/include/nolibc/crt.h b/tools/include/nolibc/crt.h
index d8ce91f..7142562 100644
--- a/tools/include/nolibc/crt.h
+++ b/tools/include/nolibc/crt.h
@@ -7,6 +7,10 @@
#ifndef _NOLIBC_CRT_H
#define _NOLIBC_CRT_H
+#define __nolibc_arg_to_reg(_a) \
+ __builtin_choose_expr(__builtin_classify_type(_a) == __builtin_classify_type(NULL), \
+ (unsigned long)(_a), (_a))
+
#ifndef NOLIBC_NO_RUNTIME
#include "compiler.h"
@@ -47,7 +51,7 @@ char *__nolibc_program_invocation_short_name(char *long_name)
#endif /* NOLIBC_IGNORE_ERRNO */
void _start_c(long *sp);
-__attribute__((weak,used)) __nolibc_no_sanitize_undefined
+__attribute__((weak,used)) __nolibc_no_sanitize_undefined __nolibc_no_stack_protector
void _start_c(long *sp)
{
long argc;
@@ -89,7 +93,7 @@ void _start_c(long *sp)
/* find _auxv */
for (auxv = (void *)envp; *auxv++;)
- ;
+ __asm__("");
_auxv = auxv;
#ifndef NOLIBC_IGNORE_ERRNO
diff --git a/tools/include/nolibc/errno.h b/tools/include/nolibc/errno.h
index bab8369..a2325596 100644
--- a/tools/include/nolibc/errno.h
+++ b/tools/include/nolibc/errno.h
@@ -15,8 +15,8 @@
#ifndef NOLIBC_IGNORE_ERRNO
#define SET_ERRNO(v) do { errno = (v); } while (0)
int errno __attribute__((weak));
-char *program_invocation_name __attribute__((weak)) = "";
-char *program_invocation_short_name __attribute__((weak)) = "";
+char *program_invocation_name __attribute__((weak)) = (char *)"";
+char *program_invocation_short_name __attribute__((weak)) = (char *)"";
#else
#define SET_ERRNO(v) do { } while (0)
#define program_invocation_name ""
diff --git a/tools/include/nolibc/fcntl.h b/tools/include/nolibc/fcntl.h
index ed2f555..d4b6af6 100644
--- a/tools/include/nolibc/fcntl.h
+++ b/tools/include/nolibc/fcntl.h
@@ -14,6 +14,20 @@
#include "types.h"
#include "sys.h"
+#define __nolibc_open_flags(_flags) ((_flags) | O_LARGEFILE)
+
+#define __nolibc_open_mode(_flags) \
+({ \
+ mode_t _mode; \
+ va_list args; \
+ \
+ va_start(args, (_flags)); \
+ _mode = va_arg(args, mode_t); \
+ va_end(args); \
+ \
+ _mode; \
+})
+
/*
* int openat(int dirfd, const char *path, int flags[, mode_t mode]);
*/
@@ -27,17 +41,8 @@ int _sys_openat(int dirfd, const char *path, int flags, mode_t mode)
static __attribute__((unused))
int openat(int dirfd, const char *path, int flags, ...)
{
- mode_t mode = 0;
-
- if (flags & O_CREAT) {
- va_list args;
-
- va_start(args, flags);
- mode = va_arg(args, mode_t);
- va_end(args);
- }
-
- return __sysret(_sys_openat(dirfd, path, flags, mode));
+ return __sysret(_sys_openat(dirfd, path, __nolibc_open_flags(flags),
+ __nolibc_open_mode(flags)));
}
/*
@@ -53,17 +58,17 @@ int _sys_open(const char *path, int flags, mode_t mode)
static __attribute__((unused))
int open(const char *path, int flags, ...)
{
- mode_t mode = 0;
+ return __sysret(_sys_open(path, __nolibc_open_flags(flags), __nolibc_open_mode(flags)));
+}
- if (flags & O_CREAT) {
- va_list args;
+/*
+ * int creat(const char *path, mode_t mode);
+ */
- va_start(args, flags);
- mode = va_arg(args, mode_t);
- va_end(args);
- }
-
- return __sysret(_sys_open(path, flags, mode));
+static __attribute__((unused))
+int creat(const char *path, mode_t mode)
+{
+ return open(path, O_CREAT | O_WRONLY | O_TRUNC, mode);
}
#endif /* _NOLIBC_FCNTL_H */
diff --git a/tools/include/nolibc/getopt.h b/tools/include/nolibc/getopt.h
index 87565e3..3ad140f 100644
--- a/tools/include/nolibc/getopt.h
+++ b/tools/include/nolibc/getopt.h
@@ -71,7 +71,7 @@ int getopt(int argc, char * const argv[], const char *optstring)
d = optstring[i++];
} while (d && d != c);
- if (d != c || c == ':') {
+ if (!d || d != c || c == ':') {
optopt = c;
if (optstring[0] != ':' && opterr)
fprintf(stderr, "%s: unrecognized option: %c\n", argv[0], *optchar);
diff --git a/tools/include/nolibc/nolibc.h b/tools/include/nolibc/nolibc.h
index f4120f6..faa94f2 100644
--- a/tools/include/nolibc/nolibc.h
+++ b/tools/include/nolibc/nolibc.h
@@ -133,6 +133,8 @@
#include "err.h"
#include "byteswap.h"
#include "endian.h"
+#include "assert.h"
+#include "alloca.h"
/* Used by programs to avoid std includes */
#define NOLIBC
diff --git a/tools/include/nolibc/stackprotector.h b/tools/include/nolibc/stackprotector.h
index ae8b1d3..916a9206 100644
--- a/tools/include/nolibc/stackprotector.h
+++ b/tools/include/nolibc/stackprotector.h
@@ -40,9 +40,10 @@ void __stack_chk_fail_local(void)
__attribute__((weak,used,section(".data.nolibc_stack_chk")))
uintptr_t __stack_chk_guard;
-static __no_stack_protector void __stack_chk_init(void)
+static __nolibc_no_stack_protector void __stack_chk_init(void)
{
- __nolibc_syscall3(__NR_getrandom, &__stack_chk_guard, sizeof(__stack_chk_guard), 0);
+ __nolibc_syscall3(__NR_getrandom, &__stack_chk_guard, sizeof(__stack_chk_guard),
+ GRND_INSECURE | GRND_NONBLOCK);
/* a bit more randomness in case getrandom() fails, ensure the guard is never 0 */
if (__stack_chk_guard != (uintptr_t) &__stack_chk_guard)
__stack_chk_guard ^= (uintptr_t) &__stack_chk_guard;
diff --git a/tools/include/nolibc/sys.h b/tools/include/nolibc/sys.h
index 6335fd5..548f94d 100644
--- a/tools/include/nolibc/sys.h
+++ b/tools/include/nolibc/sys.h
@@ -45,16 +45,41 @@
: __sysret_arg; /* return original value */ \
})
-/* Syscall ENOSYS helper: Avoids unused-parameter warnings and provides a
- * debugging hook.
+/* Syscall ENOSYS helper: Avoids unused-parameter warnings, provides compile
+ * time validation and a debugging hook.
*/
+#if defined(NOLIBC_COMPILE_TIME_ENOSYS)
static __inline__ int __nolibc_enosys(const char *syscall, ...)
{
(void)syscall;
return -ENOSYS;
}
+#elif __nolibc_has_attribute(error)
+__attribute__((error("system call not implemented")))
+extern int __nolibc_enosys(const char *syscall, ...);
+
+#else
+static __inline__ int __nolibc_enosys(const char *syscall, ...)
+{
+ extern int __nolibc_enosys_error;
+ (void)syscall;
+
+ return __nolibc_enosys_error;
+}
+#endif
+
+
+/*
+ * Helper for 32-bit machines where a 64-bit syscall arg needs to be split into
+ * two 32-bit parts while making sure the order of the low/high parts are correct
+ * for the endianness:
+ * __NOLIBC_LLARGPART(x, 0), __NOLIBC_LLARGPART(x, 1)
+ */
+#define __NOLIBC_LLARGPART(_arg, _part) \
+ (((union { long long ll; long l[2]; }) { .ll = _arg }).l[_part])
+
/* Functions in this file only describe syscalls. They're declared static so
* that the compiler usually decides to inline them while still being allowed
@@ -87,7 +112,7 @@ static __inline__ int __nolibc_enosys(const char *syscall, ...)
static __attribute__((unused))
void *_sys_brk(void *addr)
{
- return (void *)__nolibc_syscall1(__NR_brk, addr);
+ return (void *)(unsigned long)__nolibc_syscall1(__NR_brk, addr);
}
static __attribute__((unused))
@@ -597,12 +622,18 @@ int link(const char *old, const char *new)
static __attribute__((unused))
off_t _sys_lseek(int fd, off_t offset, int whence)
{
-#if defined(__NR_llseek)
+#if defined(__NR_llseek) || defined(__NR__llseek)
__kernel_loff_t loff = 0;
+ int ret, nr_llseek;
off_t result;
- int ret;
- ret = __nolibc_syscall5(__NR_llseek, fd, offset >> 32, (uint32_t)offset, &loff, whence);
+#if defined(__NR_llseek)
+ nr_llseek = __NR_llseek;
+#else
+ nr_llseek = __NR__llseek;
+#endif
+
+ ret = __nolibc_syscall5(nr_llseek, fd, offset >> 32, (uint32_t)offset, &loff, whence);
if (ret < 0)
result = ret;
else
diff --git a/tools/include/nolibc/sys/mman.h b/tools/include/nolibc/sys/mman.h
index 91d77a5..72bc1d4 100644
--- a/tools/include/nolibc/sys/mman.h
+++ b/tools/include/nolibc/sys/mman.h
@@ -27,7 +27,7 @@ void *_sys_mmap(void *addr, size_t length, int prot, int flags, int fd,
n = __NR_mmap;
#endif
- return (void *)__nolibc_syscall6(n, addr, length, prot, flags, fd, offset);
+ return (void *)(unsigned long)__nolibc_syscall6(n, addr, length, prot, flags, fd, offset);
}
#endif
@@ -46,8 +46,8 @@ void *mmap(void *addr, size_t length, int prot, int flags, int fd, off_t offset)
static __attribute__((unused))
void *_sys_mremap(void *old_address, size_t old_size, size_t new_size, int flags, void *new_address)
{
- return (void *)__nolibc_syscall5(__NR_mremap, old_address, old_size,
- new_size, flags, new_address);
+ return (void *)(unsigned long)__nolibc_syscall5(__NR_mremap, old_address, old_size,
+ new_size, flags, new_address);
}
static __attribute__((unused))
diff --git a/tools/include/nolibc/unistd.h b/tools/include/nolibc/unistd.h
index 5882a68..79599ce 100644
--- a/tools/include/nolibc/unistd.h
+++ b/tools/include/nolibc/unistd.h
@@ -48,6 +48,30 @@ int access(const char *path, int amode)
return faccessat(AT_FDCWD, path, amode, 0);
}
+#if !defined(_sys_ftruncate64) && defined(__NR_ftruncate64)
+static __attribute__((unused))
+int _sys_ftruncate64(int fd, uint32_t length0, uint32_t length1)
+{
+ return __nolibc_syscall3(__NR_ftruncate64, fd, length0, length1);
+}
+#define _sys_ftruncate64 _sys_ftruncate64
+#endif
+
+static __attribute__((unused))
+int _sys_ftruncate(int fd, off_t length)
+{
+#if defined(_sys_ftruncate64)
+ return _sys_ftruncate64(fd, __NOLIBC_LLARGPART(length, 0), __NOLIBC_LLARGPART(length, 1));
+#else
+ return __nolibc_syscall2(__NR_ftruncate, fd, length);
+#endif
+}
+
+static __attribute__((unused))
+int ftruncate(int fd, off_t length)
+{
+ return __sysret(_sys_ftruncate(fd, length));
+}
static __attribute__((unused))
int msleep(unsigned int msecs)
diff --git a/tools/include/uapi/asm-generic/errno.h b/tools/include/uapi/asm-generic/errno.h
index 92e7ae4..bd78e69 100644
--- a/tools/include/uapi/asm-generic/errno.h
+++ b/tools/include/uapi/asm-generic/errno.h
@@ -122,4 +122,6 @@
#define EHWPOISON 133 /* Memory page has hardware error */
+#define EFTYPE 134 /* Wrong file type for the intended operation */
+
#endif
diff --git a/tools/include/uapi/linux/openat2.h b/tools/include/uapi/linux/openat2.h
new file mode 100644
index 0000000..4759c47
--- /dev/null
+++ b/tools/include/uapi/linux/openat2.h
@@ -0,0 +1,43 @@
+/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */
+#ifndef _LINUX_OPENAT2_H
+#define _LINUX_OPENAT2_H
+
+#include <linux/types.h>
+
+/*
+ * Arguments for how openat2(2) should open the target path. If only @flags and
+ * @mode are non-zero, then openat2(2) operates very similarly to openat(2).
+ *
+ * However, unlike openat(2), unknown or invalid bits in @flags result in
+ * -EINVAL rather than being silently ignored. @mode must be zero unless one of
+ * {O_CREAT, O_TMPFILE} are set.
+ *
+ * @flags: O_* flags.
+ * @mode: O_CREAT/O_TMPFILE file mode.
+ * @resolve: RESOLVE_* flags.
+ */
+struct open_how {
+ __u64 flags;
+ __u64 mode;
+ __u64 resolve;
+};
+
+/* how->resolve flags for openat2(2). */
+#define RESOLVE_NO_XDEV 0x01 /* Block mount-point crossings
+ (includes bind-mounts). */
+#define RESOLVE_NO_MAGICLINKS 0x02 /* Block traversal through procfs-style
+ "magic-links". */
+#define RESOLVE_NO_SYMLINKS 0x04 /* Block traversal through all symlinks
+ (implies OEXT_NO_MAGICLINKS) */
+#define RESOLVE_BENEATH 0x08 /* Block "lexical" trickery like
+ "..", symlinks, and absolute
+ paths which escape the dirfd. */
+#define RESOLVE_IN_ROOT 0x10 /* Make all jumps to "/" and ".."
+ be scoped inside the dirfd
+ (similar to chroot(2)). */
+#define RESOLVE_CACHED 0x20 /* Only complete if resolution can be
+ completed through cached lookup. May
+ return -EAGAIN if that's not
+ possible. */
+
+#endif /* _LINUX_OPENAT2_H */
diff --git a/tools/perf/trace/beauty/include/uapi/linux/fs.h b/tools/perf/trace/beauty/include/uapi/linux/fs.h
index 13f7120..2ea4c81 100644
--- a/tools/perf/trace/beauty/include/uapi/linux/fs.h
+++ b/tools/perf/trace/beauty/include/uapi/linux/fs.h
@@ -254,6 +254,13 @@ struct file_attr {
#define FS_XFLAG_DAX 0x00008000 /* use DAX for IO */
#define FS_XFLAG_COWEXTSIZE 0x00010000 /* CoW extent size allocator hint */
#define FS_XFLAG_VERITY 0x00020000 /* fs-verity enabled */
+/*
+ * Case handling flags (read-only, cannot be set via ioctl).
+ * Default (neither set) indicates POSIX semantics: case-sensitive
+ * lookups and case-preserving storage.
+ */
+#define FS_XFLAG_CASEFOLD 0x00040000 /* case-insensitive lookups */
+#define FS_XFLAG_CASENONPRESERVING 0x00080000 /* case not preserved */
#define FS_XFLAG_HASATTR 0x80000000 /* no DIFLAG for this */
/* the read-only stuff doesn't really belong here, but any other place is
diff --git a/tools/power/acpi/common/cmfsize.c b/tools/power/acpi/common/cmfsize.c
index af0e558..6e578b0 100644
--- a/tools/power/acpi/common/cmfsize.c
+++ b/tools/power/acpi/common/cmfsize.c
@@ -3,7 +3,7 @@
*
* Module Name: cmfsize - Common get file size function
*
- * Copyright (C) 2000 - 2025, Intel Corp.
+ * Copyright (C) 2000 - 2026, Intel Corp.
*
*****************************************************************************/
diff --git a/tools/power/acpi/common/getopt.c b/tools/power/acpi/common/getopt.c
index 3d63626..42735c0 100644
--- a/tools/power/acpi/common/getopt.c
+++ b/tools/power/acpi/common/getopt.c
@@ -3,7 +3,7 @@
*
* Module Name: getopt
*
- * Copyright (C) 2000 - 2025, Intel Corp.
+ * Copyright (C) 2000 - 2026, Intel Corp.
*
*****************************************************************************/
diff --git a/tools/power/acpi/os_specific/service_layers/oslinuxtbl.c b/tools/power/acpi/os_specific/service_layers/oslinuxtbl.c
index de93067..4f9c033 100644
--- a/tools/power/acpi/os_specific/service_layers/oslinuxtbl.c
+++ b/tools/power/acpi/os_specific/service_layers/oslinuxtbl.c
@@ -3,7 +3,7 @@
*
* Module Name: oslinuxtbl - Linux OSL for obtaining ACPI tables
*
- * Copyright (C) 2000 - 2025, Intel Corp.
+ * Copyright (C) 2000 - 2026, Intel Corp.
*
*****************************************************************************/
diff --git a/tools/power/acpi/os_specific/service_layers/osunixdir.c b/tools/power/acpi/os_specific/service_layers/osunixdir.c
index b9bb831..96d0d52 100644
--- a/tools/power/acpi/os_specific/service_layers/osunixdir.c
+++ b/tools/power/acpi/os_specific/service_layers/osunixdir.c
@@ -3,7 +3,7 @@
*
* Module Name: osunixdir - Unix directory access interfaces
*
- * Copyright (C) 2000 - 2025, Intel Corp.
+ * Copyright (C) 2000 - 2026, Intel Corp.
*
*****************************************************************************/
diff --git a/tools/power/acpi/os_specific/service_layers/osunixmap.c b/tools/power/acpi/os_specific/service_layers/osunixmap.c
index b93ebc9..5f6126b 100644
--- a/tools/power/acpi/os_specific/service_layers/osunixmap.c
+++ b/tools/power/acpi/os_specific/service_layers/osunixmap.c
@@ -3,7 +3,7 @@
*
* Module Name: osunixmap - Unix OSL for file mappings
*
- * Copyright (C) 2000 - 2025, Intel Corp.
+ * Copyright (C) 2000 - 2026, Intel Corp.
*
*****************************************************************************/
diff --git a/tools/power/acpi/os_specific/service_layers/osunixxf.c b/tools/power/acpi/os_specific/service_layers/osunixxf.c
index 36f2749..49e2884 100644
--- a/tools/power/acpi/os_specific/service_layers/osunixxf.c
+++ b/tools/power/acpi/os_specific/service_layers/osunixxf.c
@@ -3,7 +3,7 @@
*
* Module Name: osunixxf - UNIX OSL interfaces
*
- * Copyright (C) 2000 - 2025, Intel Corp.
+ * Copyright (C) 2000 - 2026, Intel Corp.
*
*****************************************************************************/
diff --git a/tools/power/acpi/tools/acpidump/acpidump.h b/tools/power/acpi/tools/acpidump/acpidump.h
index fe0d23c..eac946b 100644
--- a/tools/power/acpi/tools/acpidump/acpidump.h
+++ b/tools/power/acpi/tools/acpidump/acpidump.h
@@ -3,7 +3,7 @@
*
* Module Name: acpidump.h - Include file for acpi_dump utility
*
- * Copyright (C) 2000 - 2025, Intel Corp.
+ * Copyright (C) 2000 - 2026, Intel Corp.
*
*****************************************************************************/
diff --git a/tools/power/acpi/tools/acpidump/apdump.c b/tools/power/acpi/tools/acpidump/apdump.c
index 7a6223a..72ad791 100644
--- a/tools/power/acpi/tools/acpidump/apdump.c
+++ b/tools/power/acpi/tools/acpidump/apdump.c
@@ -3,7 +3,7 @@
*
* Module Name: apdump - Dump routines for ACPI tables (acpidump)
*
- * Copyright (C) 2000 - 2025, Intel Corp.
+ * Copyright (C) 2000 - 2026, Intel Corp.
*
*****************************************************************************/
diff --git a/tools/power/acpi/tools/acpidump/apfiles.c b/tools/power/acpi/tools/acpidump/apfiles.c
index d6b8a201..0b64324 100644
--- a/tools/power/acpi/tools/acpidump/apfiles.c
+++ b/tools/power/acpi/tools/acpidump/apfiles.c
@@ -3,7 +3,7 @@
*
* Module Name: apfiles - File-related functions for acpidump utility
*
- * Copyright (C) 2000 - 2025, Intel Corp.
+ * Copyright (C) 2000 - 2026, Intel Corp.
*
*****************************************************************************/
diff --git a/tools/power/acpi/tools/acpidump/apmain.c b/tools/power/acpi/tools/acpidump/apmain.c
index 9f3850e..8403660 100644
--- a/tools/power/acpi/tools/acpidump/apmain.c
+++ b/tools/power/acpi/tools/acpidump/apmain.c
@@ -3,7 +3,7 @@
*
* Module Name: apmain - Main module for the acpidump utility
*
- * Copyright (C) 2000 - 2025, Intel Corp.
+ * Copyright (C) 2000 - 2026, Intel Corp.
*
*****************************************************************************/
diff --git a/tools/testing/selftests/Makefile b/tools/testing/selftests/Makefile
index 6e59b8f..8d4db22 100644
--- a/tools/testing/selftests/Makefile
+++ b/tools/testing/selftests/Makefile
@@ -37,6 +37,7 @@
TARGETS += filesystems/overlayfs
TARGETS += filesystems/statmount
TARGETS += filesystems/mount-notify
+TARGETS += filesystems/nsfs
TARGETS += filesystems/fuse
TARGETS += filesystems/move_mount
TARGETS += filesystems/empty_mntns
@@ -85,12 +86,12 @@
TARGETS += net/rds
TARGETS += net/tcp_ao
TARGETS += nolibc
-TARGETS += nsfs
TARGETS += pci_endpoint
TARGETS += pcie_bwctrl
TARGETS += perf_events
TARGETS += pidfd
TARGETS += pid_namespace
+TARGETS += pipe
TARGETS += power_supply
TARGETS += powerpc
TARGETS += prctl
diff --git a/tools/testing/selftests/clone3/clone3_cap_checkpoint_restore.c b/tools/testing/selftests/clone3/clone3_cap_checkpoint_restore.c
index e82281e..ab62bcf 100644
--- a/tools/testing/selftests/clone3/clone3_cap_checkpoint_restore.c
+++ b/tools/testing/selftests/clone3/clone3_cap_checkpoint_restore.c
@@ -53,9 +53,6 @@ static int call_clone3_set_tid(struct __test_metadata *_metadata,
}
if (pid == 0) {
- int ret;
- char tmp = 0;
-
TH_LOG("I am the child, my PID is %d (expected %d)", getpid(), set_tid[0]);
if (set_tid[0] != getpid())
@@ -87,15 +84,11 @@ static int test_clone3_set_tid(struct __test_metadata *_metadata,
return ret;
}
-struct libcap {
- struct __user_cap_header_struct hdr;
- struct __user_cap_data_struct data[2];
-};
-
static int set_capability(void)
{
- cap_value_t cap_values[] = { CAP_SETUID, CAP_SETGID };
- struct libcap *cap;
+ cap_value_t cap_values[] = {
+ CAP_SETUID, CAP_SETGID, CAP_CHECKPOINT_RESTORE
+ };
int ret = -1;
cap_t caps;
@@ -111,14 +104,8 @@ static int set_capability(void)
goto out;
}
- cap_set_flag(caps, CAP_EFFECTIVE, 2, cap_values, CAP_SET);
- cap_set_flag(caps, CAP_PERMITTED, 2, cap_values, CAP_SET);
-
- cap = (struct libcap *) caps;
-
- /* 40 -> CAP_CHECKPOINT_RESTORE */
- cap->data[1].effective |= 1 << (40 - 32);
- cap->data[1].permitted |= 1 << (40 - 32);
+ cap_set_flag(caps, CAP_EFFECTIVE, 3, cap_values, CAP_SET);
+ cap_set_flag(caps, CAP_PERMITTED, 3, cap_values, CAP_SET);
if (cap_set_proc(caps)) {
perror("cap_set_proc");
@@ -135,7 +122,6 @@ TEST(clone3_cap_checkpoint_restore)
{
pid_t pid;
int status;
- int ret = 0;
pid_t set_tid[1];
test_clone3_supported();
diff --git a/tools/testing/selftests/filesystems/epoll/epoll_wakeup_test.c b/tools/testing/selftests/filesystems/epoll/epoll_wakeup_test.c
index 8bc57a2..f6f1a7f 100644
--- a/tools/testing/selftests/filesystems/epoll/epoll_wakeup_test.c
+++ b/tools/testing/selftests/filesystems/epoll/epoll_wakeup_test.c
@@ -3493,4 +3493,49 @@ TEST(epoll64)
close(ctx.sfd[1]);
}
+static void *epoll65_wait(void *ctx_)
+{
+ struct epoll_mtcontext *ctx = ctx_;
+ struct epoll_event event;
+
+ for (int i = 0; i < 100000; ++i) {
+ if (!epoll_wait(ctx->efd[0], &event, 1, 0))
+ return (void *)ENODATA;
+ }
+
+ return (void *)0;
+}
+
+TEST(epoll65)
+{
+ struct epoll_mtcontext ctx;
+ struct epoll_event event;
+ int64_t dummy_data = 99;
+ pthread_t threads[64];
+ uintptr_t ret;
+ int i, err;
+
+ ctx.efd[0] = epoll_create(1);
+ ASSERT_GE(ctx.efd[0], 0);
+ ctx.efd[1] = eventfd(0, 0);
+ ASSERT_GE(ctx.efd[1], 0);
+
+ event.events = EPOLLIN;
+ err = epoll_ctl(ctx.efd[0], EPOLL_CTL_ADD, ctx.efd[1], &event);
+ ASSERT_EQ(err, 0);
+
+ write(ctx.efd[1], &dummy_data, sizeof(dummy_data));
+
+ for (i = 0; i < ARRAY_SIZE(threads); ++i)
+ ASSERT_EQ(pthread_create(&threads[i], NULL, epoll65_wait, &ctx), 0);
+
+ for (i = 0; i < ARRAY_SIZE(threads); ++i) {
+ ASSERT_EQ(pthread_join(threads[i], (void **)&ret), 0);
+ ASSERT_EQ(ret, 0);
+ }
+
+ close(ctx.efd[0]);
+ close(ctx.efd[1]);
+}
+
TEST_HARNESS_MAIN
diff --git a/tools/testing/selftests/openat2/.gitignore b/tools/testing/selftests/filesystems/openat2/.gitignore
similarity index 100%
rename from tools/testing/selftests/openat2/.gitignore
rename to tools/testing/selftests/filesystems/openat2/.gitignore
diff --git a/tools/testing/selftests/openat2/Makefile b/tools/testing/selftests/filesystems/openat2/Makefile
similarity index 65%
rename from tools/testing/selftests/openat2/Makefile
rename to tools/testing/selftests/filesystems/openat2/Makefile
index 185dc76..d848aac 100644
--- a/tools/testing/selftests/openat2/Makefile
+++ b/tools/testing/selftests/filesystems/openat2/Makefile
@@ -1,7 +1,8 @@
# SPDX-License-Identifier: GPL-2.0-or-later
-CFLAGS += -Wall -O2 -g -fsanitize=address -fsanitize=undefined
-TEST_GEN_PROGS := openat2_test resolve_test rename_attack_test
+CFLAGS += $(KHDR_INCLUDES)
+CFLAGS += -Wall -O2 -g -fsanitize=address -fsanitize=undefined $(TOOLS_INCLUDES)
+TEST_GEN_PROGS := openat2_test resolve_test rename_attack_test emptypath_test
# gcc requires -static-libasan in order to ensure that Address Sanitizer's
# library is the first one loaded. However, clang already statically links the
@@ -13,6 +14,4 @@
LOCAL_HDRS += helpers.h
-include ../lib.mk
-
-$(TEST_GEN_PROGS): helpers.c
+include ../../lib.mk
diff --git a/tools/testing/selftests/filesystems/openat2/emptypath_test.c b/tools/testing/selftests/filesystems/openat2/emptypath_test.c
new file mode 100644
index 0000000..be37ccb
--- /dev/null
+++ b/tools/testing/selftests/filesystems/openat2/emptypath_test.c
@@ -0,0 +1,77 @@
+// SPDX-License-Identifier: GPL-2.0-or-later
+
+#define _GNU_SOURCE
+#define __SANE_USERSPACE_TYPES__
+#include <fcntl.h>
+#include <unistd.h>
+#include <errno.h>
+#include <string.h>
+#include <sys/stat.h>
+
+#include "kselftest_harness.h"
+
+#ifndef O_EMPTYPATH
+#define O_EMPTYPATH (1 << 26)
+#endif
+
+#define EMPTYPATH_TEST_FILE "/tmp/emptypath_test"
+
+FIXTURE(emptypath) {
+ int opath_fd;
+};
+
+FIXTURE_SETUP(emptypath)
+{
+ int fd;
+
+ self->opath_fd = -1;
+
+ fd = open(EMPTYPATH_TEST_FILE, O_CREAT | O_WRONLY, S_IRWXU);
+ ASSERT_GE(fd, 0) {
+ TH_LOG("create %s: %s", EMPTYPATH_TEST_FILE, strerror(errno));
+ }
+ close(fd);
+
+ self->opath_fd = open(EMPTYPATH_TEST_FILE, O_PATH);
+ ASSERT_GE(self->opath_fd, 0) {
+ TH_LOG("open %s O_PATH: %s", EMPTYPATH_TEST_FILE, strerror(errno));
+ }
+}
+
+FIXTURE_TEARDOWN(emptypath)
+{
+ if (self->opath_fd >= 0)
+ close(self->opath_fd);
+ unlink(EMPTYPATH_TEST_FILE);
+}
+
+/* An empty path is rejected with ENOENT unless O_EMPTYPATH is set. */
+TEST_F(emptypath, without_flag_returns_enoent)
+{
+ int fd = openat(self->opath_fd, "", O_RDONLY);
+
+ if (fd >= 0)
+ close(fd);
+ ASSERT_LT(fd, 0) {
+ TH_LOG("empty path without O_EMPTYPATH unexpectedly succeeded");
+ }
+ EXPECT_EQ(errno, ENOENT) {
+ TH_LOG("expected ENOENT, got %s", strerror(errno));
+ }
+}
+
+/* O_EMPTYPATH reopens the O_PATH fd through an empty path. */
+TEST_F(emptypath, reopens_opath_fd)
+{
+ int fd = openat(self->opath_fd, "", O_RDONLY | O_EMPTYPATH);
+
+ if (fd < 0 && errno == EINVAL)
+ SKIP(return, "O_EMPTYPATH not supported");
+
+ ASSERT_GE(fd, 0) {
+ TH_LOG("O_EMPTYPATH failed: %s", strerror(errno));
+ }
+ close(fd);
+}
+
+TEST_HARNESS_MAIN
diff --git a/tools/testing/selftests/filesystems/openat2/helpers.h b/tools/testing/selftests/filesystems/openat2/helpers.h
new file mode 100644
index 0000000..3f01fb6
--- /dev/null
+++ b/tools/testing/selftests/filesystems/openat2/helpers.h
@@ -0,0 +1,135 @@
+// SPDX-License-Identifier: GPL-2.0-or-later
+/*
+ * Author: Aleksa Sarai <cyphar@cyphar.com>
+ * Copyright (C) 2018-2019 SUSE LLC.
+ * Copyright (C) 2026 Amutable GmbH
+ */
+
+#ifndef __RESOLVEAT_H__
+#define __RESOLVEAT_H__
+
+#define _GNU_SOURCE
+#include <stdint.h>
+#include <stdbool.h>
+#include <errno.h>
+#include <limits.h>
+#include <linux/types.h>
+#include <linux/unistd.h>
+#include <linux/openat2.h>
+#include "kselftest_harness.h"
+
+#define BUILD_BUG_ON(e) ((void)(sizeof(struct { int:(-!!(e)); })))
+
+#define OPEN_HOW_SIZE_VER0 24 /* sizeof first published struct */
+#define OPEN_HOW_SIZE_LATEST OPEN_HOW_SIZE_VER0
+
+__maybe_unused
+static bool needs_openat2(const struct open_how *how)
+{
+ return how->resolve != 0;
+}
+
+__maybe_unused
+static int raw_openat2(int dfd, const char *path, void *how, size_t size)
+{
+ int ret = syscall(__NR_openat2, dfd, path, how, size);
+
+ return ret >= 0 ? ret : -errno;
+}
+
+__maybe_unused
+static int sys_openat2(int dfd, const char *path, struct open_how *how)
+{
+ return raw_openat2(dfd, path, how, sizeof(*how));
+}
+
+__maybe_unused
+static int sys_openat(int dfd, const char *path, struct open_how *how)
+{
+ int ret = openat(dfd, path, how->flags, how->mode);
+
+ return ret >= 0 ? ret : -errno;
+}
+
+__maybe_unused
+static int sys_renameat2(int olddirfd, const char *oldpath,
+ int newdirfd, const char *newpath, unsigned int flags)
+{
+ int ret = syscall(__NR_renameat2, olddirfd, oldpath,
+ newdirfd, newpath, flags);
+
+ return ret >= 0 ? ret : -errno;
+}
+
+__maybe_unused
+static int touchat(int dfd, const char *path)
+{
+ int fd = openat(dfd, path, O_CREAT, 0700);
+
+ if (fd >= 0)
+ close(fd);
+ return fd;
+}
+
+__maybe_unused
+static char *fdreadlink(struct __test_metadata *_metadata, int fd)
+{
+ char *target, *tmp;
+
+ ASSERT_GT(asprintf(&tmp, "/proc/self/fd/%d", fd), 0);
+
+ target = malloc(PATH_MAX);
+ ASSERT_NE(target, NULL);
+ memset(target, 0, PATH_MAX);
+
+ ASSERT_GT(readlink(tmp, target, PATH_MAX), 0);
+
+ free(tmp);
+ return target;
+}
+
+__maybe_unused
+static bool fdequal(struct __test_metadata *_metadata, int fd,
+ int dfd, const char *path)
+{
+ char *fdpath, *dfdpath, *other;
+ bool cmp;
+
+ fdpath = fdreadlink(_metadata, fd);
+ dfdpath = fdreadlink(_metadata, dfd);
+
+ if (!path) {
+ ASSERT_GT(asprintf(&other, "%s", dfdpath), 0);
+ } else if (*path == '/') {
+ ASSERT_GT(asprintf(&other, "%s", path), 0);
+ } else {
+ ASSERT_GT(asprintf(&other, "%s/%s", dfdpath, path), 0);
+ }
+
+ cmp = !strcmp(fdpath, other);
+
+ free(fdpath);
+ free(dfdpath);
+ free(other);
+ return cmp;
+}
+
+static bool openat2_supported = false;
+
+__attribute__((constructor))
+static void __detect_openat2_supported(void)
+{
+ struct open_how how = {};
+ int fd;
+
+ BUILD_BUG_ON(sizeof(struct open_how) != OPEN_HOW_SIZE_VER0);
+
+ /* Check openat2(2) support. */
+ fd = sys_openat2(AT_FDCWD, ".", &how);
+ openat2_supported = (fd >= 0);
+
+ if (fd >= 0)
+ close(fd);
+}
+
+#endif /* __RESOLVEAT_H__ */
diff --git a/tools/testing/selftests/openat2/openat2_test.c b/tools/testing/selftests/filesystems/openat2/openat2_test.c
similarity index 63%
rename from tools/testing/selftests/openat2/openat2_test.c
rename to tools/testing/selftests/filesystems/openat2/openat2_test.c
index 0e161ef..6f5afbe 100644
--- a/tools/testing/selftests/openat2/openat2_test.c
+++ b/tools/testing/selftests/filesystems/openat2/openat2_test.c
@@ -15,8 +15,8 @@
#include <stdbool.h>
#include <string.h>
-#include "kselftest.h"
#include "helpers.h"
+#include "kselftest_harness.h"
/*
* O_LARGEFILE is set to 0 by glibc.
@@ -45,13 +45,29 @@ struct struct_test {
int err;
};
-#define NUM_OPENAT2_STRUCT_TESTS 7
-#define NUM_OPENAT2_STRUCT_VARIATIONS 13
+struct flag_test {
+ const char *name;
+ struct open_how how;
+ int err;
+};
-void test_openat2_struct(void)
+FIXTURE(openat2) {};
+
+FIXTURE_SETUP(openat2)
+{
+ if (!openat2_supported)
+ SKIP(return, "openat2(2) not supported");
+}
+
+FIXTURE_TEARDOWN(openat2) {}
+
+/*
+ * Verify that the struct size and misalignment handling for openat2(2) is
+ * correct, including that is_zeroed_user() works.
+ */
+TEST_F(openat2, struct_argument_sizes)
{
int misalignments[] = { 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11, 17, 87 };
-
struct struct_test tests[] = {
/* Normal struct. */
{ .name = "normal struct",
@@ -83,26 +99,14 @@ void test_openat2_struct(void)
.size = sizeof(struct open_how_ext), .err = -E2BIG },
};
- BUILD_BUG_ON(ARRAY_LEN(misalignments) != NUM_OPENAT2_STRUCT_VARIATIONS);
- BUILD_BUG_ON(ARRAY_LEN(tests) != NUM_OPENAT2_STRUCT_TESTS);
-
- for (int i = 0; i < ARRAY_LEN(tests); i++) {
+ for (int i = 0; i < ARRAY_SIZE(tests); i++) {
struct struct_test *test = &tests[i];
struct open_how_ext how_ext = test->arg;
- for (int j = 0; j < ARRAY_LEN(misalignments); j++) {
+ for (int j = 0; j < ARRAY_SIZE(misalignments); j++) {
int fd, misalign = misalignments[j];
- char *fdpath = NULL;
- bool failed;
- void (*resultfn)(const char *msg, ...) = ksft_test_result_pass;
-
void *copy = NULL, *how_copy = &how_ext;
-
- if (!openat2_supported) {
- ksft_print_msg("openat2(2) unsupported\n");
- resultfn = ksft_test_result_skip;
- goto skip;
- }
+ char *fdpath = NULL;
if (misalign) {
/*
@@ -119,50 +123,42 @@ void test_openat2_struct(void)
}
fd = raw_openat2(AT_FDCWD, ".", how_copy, test->size);
- if (test->err >= 0)
- failed = (fd < 0);
- else
- failed = (fd != test->err);
if (fd >= 0) {
- fdpath = fdreadlink(fd);
+ fdpath = fdreadlink(_metadata, fd);
close(fd);
}
- if (failed) {
- resultfn = ksft_test_result_fail;
-
- ksft_print_msg("openat2 unexpectedly returned ");
- if (fdpath)
- ksft_print_msg("%d['%s']\n", fd, fdpath);
- else
- ksft_print_msg("%d (%s)\n", fd, strerror(-fd));
+ if (test->err >= 0) {
+ EXPECT_GE(fd, 0) {
+ TH_LOG("openat2 with %s [misalign=%d] should succeed, got %d (%s)",
+ test->name, misalign,
+ fd, strerror(-fd));
+ }
+ } else {
+ EXPECT_EQ(test->err, fd) {
+ if (fdpath)
+ TH_LOG("openat2 with %s [misalign=%d] should fail with %d (%s), got %d['%s']",
+ test->name, misalign,
+ test->err,
+ strerror(-test->err),
+ fd, fdpath);
+ else
+ TH_LOG("openat2 with %s [misalign=%d] should fail with %d (%s), got %d (%s)",
+ test->name, misalign,
+ test->err,
+ strerror(-test->err),
+ fd, strerror(-fd));
+ }
}
-skip:
- if (test->err >= 0)
- resultfn("openat2 with %s argument [misalign=%d] succeeds\n",
- test->name, misalign);
- else
- resultfn("openat2 with %s argument [misalign=%d] fails with %d (%s)\n",
- test->name, misalign, test->err,
- strerror(-test->err));
-
free(copy);
free(fdpath);
- fflush(stdout);
}
}
}
-struct flag_test {
- const char *name;
- struct open_how how;
- int err;
-};
-
-#define NUM_OPENAT2_FLAG_TESTS 25
-
-void test_openat2_flags(void)
+/* Verify openat2(2) flag and mode validation. */
+TEST_F(openat2, flag_validation)
{
struct flag_test tests[] = {
/* O_TMPFILE is incompatible with O_PATH and O_CREAT. */
@@ -241,20 +237,10 @@ void test_openat2_flags(void)
.how.resolve = 0, .err = -EINVAL },
};
- BUILD_BUG_ON(ARRAY_LEN(tests) != NUM_OPENAT2_FLAG_TESTS);
-
- for (int i = 0; i < ARRAY_LEN(tests); i++) {
+ for (int i = 0; i < ARRAY_SIZE(tests); i++) {
int fd, fdflags = -1;
char *path, *fdpath = NULL;
- bool failed = false;
struct flag_test *test = &tests[i];
- void (*resultfn)(const char *msg, ...) = ksft_test_result_pass;
-
- if (!openat2_supported) {
- ksft_print_msg("openat2(2) unsupported\n");
- resultfn = ksft_test_result_skip;
- goto skip;
- }
path = (test->how.flags & O_CREAT) ? "/tmp/ksft.openat2_tmpfile" : ".";
unlink(path);
@@ -265,74 +251,112 @@ void test_openat2_flags(void)
* Skip the testcase if it failed because not supported
* by FS. (e.g. a valid O_TMPFILE combination on NFS)
*/
- ksft_test_result_skip("openat2 with %s fails with %d (%s)\n",
- test->name, fd, strerror(-fd));
- goto next;
+ TH_LOG("openat2 with %s not supported by FS -- skipping",
+ test->name);
+ continue;
}
- if (test->err >= 0)
- failed = (fd < 0);
- else
- failed = (fd != test->err);
- if (fd >= 0) {
- int otherflags;
+ if (test->err >= 0) {
+ EXPECT_GE(fd, 0) {
+ TH_LOG("openat2 with %s should succeed, got %d (%s)",
+ test->name, fd, strerror(-fd));
+ }
+ if (fd >= 0) {
+ int otherflags;
- fdpath = fdreadlink(fd);
- fdflags = fcntl(fd, F_GETFL);
- otherflags = fcntl(fd, F_GETFD);
- close(fd);
+ fdpath = fdreadlink(_metadata, fd);
+ fdflags = fcntl(fd, F_GETFL);
+ otherflags = fcntl(fd, F_GETFD);
+ close(fd);
- E_assert(fdflags >= 0, "fcntl F_GETFL of new fd");
- E_assert(otherflags >= 0, "fcntl F_GETFD of new fd");
+ ASSERT_GE(fdflags, 0);
+ ASSERT_GE(otherflags, 0);
- /* O_CLOEXEC isn't shown in F_GETFL. */
- if (otherflags & FD_CLOEXEC)
- fdflags |= O_CLOEXEC;
- /* O_CREAT is hidden from F_GETFL. */
- if (test->how.flags & O_CREAT)
- fdflags |= O_CREAT;
- if (!(test->how.flags & O_LARGEFILE))
- fdflags &= ~O_LARGEFILE;
- failed |= (fdflags != test->how.flags);
+ /* O_CLOEXEC isn't shown in F_GETFL. */
+ if (otherflags & FD_CLOEXEC)
+ fdflags |= O_CLOEXEC;
+ /* O_CREAT is hidden from F_GETFL. */
+ if (test->how.flags & O_CREAT)
+ fdflags |= O_CREAT;
+ if (!(test->how.flags & O_LARGEFILE))
+ fdflags &= ~O_LARGEFILE;
+
+ EXPECT_EQ(fdflags, (int)test->how.flags) {
+ TH_LOG("openat2 with %s: flags mismatch %X != %llX",
+ test->name, fdflags,
+ (unsigned long long)test->how.flags);
+ }
+ }
+ } else {
+ EXPECT_EQ(test->err, fd) {
+ if (fd >= 0) {
+ fdpath = fdreadlink(_metadata, fd);
+ TH_LOG("openat2 with %s should fail with %d (%s), got %d['%s']",
+ test->name, test->err,
+ strerror(-test->err),
+ fd, fdpath);
+ } else {
+ TH_LOG("openat2 with %s should fail with %d (%s), got %d (%s)",
+ test->name, test->err,
+ strerror(-test->err),
+ fd, strerror(-fd));
+ }
+ }
+ if (fd >= 0)
+ close(fd);
}
- if (failed) {
- resultfn = ksft_test_result_fail;
-
- ksft_print_msg("openat2 unexpectedly returned ");
- if (fdpath)
- ksft_print_msg("%d['%s'] with %X (!= %llX)\n",
- fd, fdpath, fdflags,
- test->how.flags);
- else
- ksft_print_msg("%d (%s)\n", fd, strerror(-fd));
- }
-
-skip:
- if (test->err >= 0)
- resultfn("openat2 with %s succeeds\n", test->name);
- else
- resultfn("openat2 with %s fails with %d (%s)\n",
- test->name, test->err, strerror(-test->err));
-next:
free(fdpath);
- fflush(stdout);
}
}
-#define NUM_TESTS (NUM_OPENAT2_STRUCT_VARIATIONS * NUM_OPENAT2_STRUCT_TESTS + \
- NUM_OPENAT2_FLAG_TESTS)
+#ifndef OPENAT2_REGULAR
+#define OPENAT2_REGULAR ((__u64)1 << 32)
+#endif
-int main(int argc, char **argv)
+#ifndef EFTYPE
+#define EFTYPE 134
+#endif
+
+/* Kernel-internal carrier for OPENAT2_REGULAR (see __O_REGULAR in fcntl.h). */
+#ifndef __O_REGULAR
+#define __O_REGULAR (1 << 30)
+#endif
+
+/* Verify that OPENAT2_REGULAR rejects non-regular files with EFTYPE. */
+TEST_F(openat2, regular_flag)
{
- ksft_print_header();
- ksft_set_plan(NUM_TESTS);
+ struct open_how how = {
+ .flags = OPENAT2_REGULAR | O_RDONLY,
+ };
+ int fd;
- test_openat2_struct();
- test_openat2_flags();
+ fd = sys_openat2(AT_FDCWD, "/dev/null", &how);
+ if (fd == -ENOENT)
+ SKIP(return, "/dev/null does not exist");
- if (ksft_get_fail_cnt() + ksft_get_error_cnt() > 0)
- ksft_exit_fail();
- else
- ksft_exit_pass();
+ EXPECT_EQ(-EFTYPE, fd) {
+ TH_LOG("openat2 with OPENAT2_REGULAR should fail with %d (%s), got %d (%s)",
+ -EFTYPE, strerror(EFTYPE), fd, strerror(-fd));
+ }
+ if (fd >= 0)
+ close(fd);
}
+
+/* open()/openat() must keep ignoring the internal __O_REGULAR bit. */
+TEST(legacy_openat_ignores_o_regular)
+{
+ int fd;
+
+ fd = openat(AT_FDCWD, "/dev/null", O_RDONLY | __O_REGULAR);
+ if (fd < 0 && errno == ENOENT)
+ SKIP(return, "/dev/null does not exist");
+
+ ASSERT_GE(fd, 0) {
+ TH_LOG("legacy openat() must ignore the __O_REGULAR carrier bit, got errno %d (%s)",
+ errno, strerror(errno));
+ }
+ close(fd);
+}
+
+TEST_HARNESS_MAIN
diff --git a/tools/testing/selftests/filesystems/openat2/rename_attack_test.c b/tools/testing/selftests/filesystems/openat2/rename_attack_test.c
new file mode 100644
index 0000000..1f33c34
--- /dev/null
+++ b/tools/testing/selftests/filesystems/openat2/rename_attack_test.c
@@ -0,0 +1,159 @@
+// SPDX-License-Identifier: GPL-2.0-or-later
+/*
+ * Author: Aleksa Sarai <cyphar@cyphar.com>
+ * Copyright (C) 2018-2019 SUSE LLC.
+ */
+
+#define _GNU_SOURCE
+#include <errno.h>
+#include <fcntl.h>
+#include <sched.h>
+#include <sys/stat.h>
+#include <sys/types.h>
+#include <sys/mount.h>
+#include <sys/mman.h>
+#include <sys/prctl.h>
+#include <signal.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <stdbool.h>
+#include <string.h>
+#include <syscall.h>
+#include <limits.h>
+#include <unistd.h>
+
+#include "helpers.h"
+#include "kselftest_harness.h"
+
+#define ROUNDS 400000
+
+/* Swap @dirfd/@a and @dirfd/@b constantly. Parent must kill this process. */
+pid_t spawn_attack(struct __test_metadata *_metadata,
+ int dirfd, char *a, char *b)
+{
+ pid_t child = fork();
+ if (child != 0)
+ return child;
+
+ /* If the parent (the test process) dies, kill ourselves too. */
+ ASSERT_EQ(prctl(PR_SET_PDEATHSIG, SIGKILL), 0);
+
+ /* Swap @a and @b. */
+ for (;;)
+ renameat2(dirfd, a, dirfd, b, RENAME_EXCHANGE);
+ exit(1);
+}
+
+/*
+ * Construct a test directory with the following structure:
+ *
+ * root/
+ * |-- a/
+ * | `-- c/
+ * `-- b/
+ */
+FIXTURE(rename_attack) {
+ int dfd;
+ int afd;
+ pid_t child;
+};
+
+FIXTURE_SETUP(rename_attack)
+{
+ char dirname[] = "/tmp/ksft-openat2-rename-attack.XXXXXX";
+
+ self->dfd = -1;
+ self->afd = -1;
+ self->child = 0;
+
+ /* Make the top-level directory. */
+ ASSERT_NE(mkdtemp(dirname), NULL);
+ self->dfd = open(dirname, O_PATH | O_DIRECTORY);
+ ASSERT_GE(self->dfd, 0);
+
+ ASSERT_EQ(mkdirat(self->dfd, "a", 0755), 0);
+ ASSERT_EQ(mkdirat(self->dfd, "b", 0755), 0);
+ ASSERT_EQ(mkdirat(self->dfd, "a/c", 0755), 0);
+
+ self->afd = openat(self->dfd, "a", O_PATH);
+ ASSERT_GE(self->afd, 0);
+
+ self->child = spawn_attack(_metadata, self->dfd, "a/c", "b");
+ ASSERT_GT(self->child, 0);
+}
+
+FIXTURE_TEARDOWN(rename_attack)
+{
+ if (self->child > 0)
+ kill(self->child, SIGKILL);
+ if (self->afd >= 0)
+ close(self->afd);
+ if (self->dfd >= 0)
+ close(self->dfd);
+}
+
+FIXTURE_VARIANT(rename_attack) {
+ int resolve;
+ const char *name;
+};
+
+FIXTURE_VARIANT_ADD(rename_attack, resolve_beneath) {
+ .resolve = RESOLVE_BENEATH,
+ .name = "RESOLVE_BENEATH",
+};
+
+FIXTURE_VARIANT_ADD(rename_attack, resolve_in_root) {
+ .resolve = RESOLVE_IN_ROOT,
+ .name = "RESOLVE_IN_ROOT",
+};
+
+TEST_F_TIMEOUT(rename_attack, test, 120)
+{
+ int escapes = 0, successes = 0, other_errs = 0, exdevs = 0, eagains = 0;
+ char *victim_path = "c/../../c/../../c/../../c/../../c/../../c/../../c/../../c/../../c/../../c/../../c/../../c/../../c/../../c/../../c/../../c/../../c/../../c/../../c/../..";
+ struct open_how how = {
+ .flags = O_PATH,
+ .resolve = variant->resolve,
+ };
+
+ if (!openat2_supported) {
+ how.resolve = 0;
+ TH_LOG("openat2(2) unsupported -- using openat(2) instead");
+ }
+
+ for (int i = 0; i < ROUNDS; i++) {
+ int fd;
+
+ if (openat2_supported)
+ fd = sys_openat2(self->afd, victim_path, &how);
+ else
+ fd = sys_openat(self->afd, victim_path, &how);
+
+ if (fd < 0) {
+ if (fd == -EAGAIN)
+ eagains++;
+ else if (fd == -EXDEV)
+ exdevs++;
+ else if (fd == -ENOENT)
+ escapes++; /* escaped outside and got ENOENT... */
+ else
+ other_errs++; /* unexpected error */
+ } else {
+ if (fdequal(_metadata, fd, self->afd, NULL))
+ successes++;
+ else
+ escapes++; /* we got an unexpected fd */
+ }
+ if (fd >= 0)
+ close(fd);
+ }
+
+ TH_LOG("non-escapes: EAGAIN=%d EXDEV=%d E<other>=%d success=%d",
+ eagains, exdevs, other_errs, successes);
+ ASSERT_EQ(escapes, 0) {
+ TH_LOG("rename attack with %s (%d runs, got %d escapes)",
+ variant->name, ROUNDS, escapes);
+ }
+}
+
+TEST_HARNESS_MAIN
diff --git a/tools/testing/selftests/openat2/resolve_test.c b/tools/testing/selftests/filesystems/openat2/resolve_test.c
similarity index 74%
rename from tools/testing/selftests/openat2/resolve_test.c
rename to tools/testing/selftests/filesystems/openat2/resolve_test.c
index a76ef15c..eacde59 100644
--- a/tools/testing/selftests/openat2/resolve_test.c
+++ b/tools/testing/selftests/filesystems/openat2/resolve_test.c
@@ -14,8 +14,81 @@
#include <stdbool.h>
#include <string.h>
-#include "kselftest.h"
#include "helpers.h"
+#include "kselftest_harness.h"
+
+struct resolve_test {
+ const char *name;
+ const char *dir;
+ const char *path;
+ struct open_how how;
+ bool pass;
+ union {
+ int err;
+ const char *path;
+ } out;
+};
+
+/*
+ * Verify a single resolve test case. This must be called from within a TEST_F
+ * function with _metadata in scope.
+ */
+static void verify_resolve_test(struct __test_metadata *_metadata,
+ int rootfd, int hardcoded_fd,
+ const struct resolve_test *test)
+{
+ struct open_how how = test->how;
+ int dfd, fd;
+ char *fdpath = NULL;
+
+ /* Auto-set O_PATH. */
+ if (!(how.flags & O_CREAT))
+ how.flags |= O_PATH;
+
+ if (test->dir)
+ dfd = openat(rootfd, test->dir, O_PATH | O_DIRECTORY);
+ else
+ dfd = dup(rootfd);
+ ASSERT_GE(dfd, 0) TH_LOG("failed to open dir '%s': %m", test->dir ?: ".");
+ ASSERT_EQ(dup2(dfd, hardcoded_fd), hardcoded_fd);
+
+ fd = sys_openat2(dfd, test->path, &how);
+
+ if (test->pass) {
+ EXPECT_GE(fd, 0) {
+ TH_LOG("%s: expected success, got %d (%s)",
+ test->name, fd, strerror(-fd));
+ }
+ if (fd >= 0) {
+ EXPECT_TRUE(fdequal(_metadata, fd, rootfd, test->out.path)) {
+ fdpath = fdreadlink(_metadata, fd);
+ TH_LOG("%s: wrong path '%s', expected '%s'",
+ test->name, fdpath,
+ test->out.path ?: ".");
+ free(fdpath);
+ }
+ }
+ } else {
+ EXPECT_EQ(test->out.err, fd) {
+ if (fd >= 0) {
+ fdpath = fdreadlink(_metadata, fd);
+ TH_LOG("%s: expected %d (%s), got %d['%s']",
+ test->name, test->out.err,
+ strerror(-test->out.err), fd, fdpath);
+ free(fdpath);
+ } else {
+ TH_LOG("%s: expected %d (%s), got %d (%s)",
+ test->name, test->out.err,
+ strerror(-test->out.err),
+ fd, strerror(-fd));
+ }
+ }
+ }
+
+ if (fd >= 0)
+ close(fd);
+ close(dfd);
+}
/*
* Construct a test directory with the following structure:
@@ -39,101 +112,110 @@
* |-- absself -> /
* |-- self -> ../../root/
* |-- garbageself -> /../../root/
- * |-- passwd -> ../cheeky/../cheeky/../etc/../etc/passwd
- * |-- abspasswd -> /../cheeky/../cheeky/../etc/../etc/passwd
+ * |-- passwd -> ../cheeky/../etc/../etc/passwd
+ * |-- abspasswd -> /../cheeky/../etc/../etc/passwd
* |-- dotdotlink -> ../../../../../../../../../../../../../../etc/passwd
* `-- garbagelink -> /../../../../../../../../../../../../../../etc/passwd
*/
-int setup_testdir(void)
+FIXTURE(openat2_resolve) {
+ int rootfd;
+ int hardcoded_fd;
+ char *hardcoded_fdpath;
+ char *procselfexe;
+};
+
+FIXTURE_SETUP(openat2_resolve)
{
- int dfd, tmpfd;
char dirname[] = "/tmp/ksft-openat2-testdir.XXXXXX";
+ int dfd, tmpfd;
+
+ self->rootfd = -1;
+ self->hardcoded_fd = -1;
+ self->hardcoded_fdpath = NULL;
+ self->procselfexe = NULL;
+
+ /* NOTE: We should be checking for CAP_SYS_ADMIN here... */
+ if (geteuid() != 0)
+ SKIP(return, "all tests require euid == 0");
+ if (!openat2_supported)
+ SKIP(return, "openat2(2) not supported");
/* Unshare and make /tmp a new directory. */
- E_unshare(CLONE_NEWNS);
- E_mount("", "/tmp", "", MS_PRIVATE, "");
+ ASSERT_EQ(unshare(CLONE_NEWNS), 0);
+ ASSERT_EQ(mount("", "/tmp", "", MS_PRIVATE, ""), 0);
/* Make the top-level directory. */
- if (!mkdtemp(dirname))
- ksft_exit_fail_msg("setup_testdir: failed to create tmpdir\n");
+ ASSERT_NE(mkdtemp(dirname), NULL);
dfd = open(dirname, O_PATH | O_DIRECTORY);
- if (dfd < 0)
- ksft_exit_fail_msg("setup_testdir: failed to open tmpdir\n");
+ ASSERT_GE(dfd, 0);
/* A sub-directory which is actually used for tests. */
- E_mkdirat(dfd, "root", 0755);
+ ASSERT_EQ(mkdirat(dfd, "root", 0755), 0);
tmpfd = openat(dfd, "root", O_PATH | O_DIRECTORY);
- if (tmpfd < 0)
- ksft_exit_fail_msg("setup_testdir: failed to open tmpdir\n");
+ ASSERT_GE(tmpfd, 0);
close(dfd);
dfd = tmpfd;
- E_symlinkat("/proc/self/exe", dfd, "procexe");
- E_symlinkat("/proc/self/root", dfd, "procroot");
- E_mkdirat(dfd, "root", 0755);
+ ASSERT_EQ(symlinkat("/proc/self/exe", dfd, "procexe"), 0);
+ ASSERT_EQ(symlinkat("/proc/self/root", dfd, "procroot"), 0);
+ ASSERT_EQ(mkdirat(dfd, "root", 0755), 0);
/* There is no mountat(2), so use chdir. */
- E_mkdirat(dfd, "mnt", 0755);
- E_fchdir(dfd);
- E_mount("tmpfs", "./mnt", "tmpfs", MS_NOSUID | MS_NODEV, "");
- E_symlinkat("../mnt/", dfd, "mnt/self");
- E_symlinkat("/mnt/", dfd, "mnt/absself");
+ ASSERT_EQ(mkdirat(dfd, "mnt", 0755), 0);
+ ASSERT_EQ(fchdir(dfd), 0);
+ ASSERT_EQ(mount("tmpfs", "./mnt", "tmpfs", MS_NOSUID | MS_NODEV, ""), 0);
+ ASSERT_EQ(symlinkat("../mnt/", dfd, "mnt/self"), 0);
+ ASSERT_EQ(symlinkat("/mnt/", dfd, "mnt/absself"), 0);
- E_mkdirat(dfd, "etc", 0755);
- E_touchat(dfd, "etc/passwd");
+ ASSERT_EQ(mkdirat(dfd, "etc", 0755), 0);
+ ASSERT_GE(touchat(dfd, "etc/passwd"), 0);
- E_symlinkat("/newfile3", dfd, "creatlink");
- E_symlinkat("etc/", dfd, "reletc");
- E_symlinkat("etc/passwd", dfd, "relsym");
- E_symlinkat("/etc/", dfd, "absetc");
- E_symlinkat("/etc/passwd", dfd, "abssym");
- E_symlinkat("/cheeky", dfd, "abscheeky");
+ ASSERT_EQ(symlinkat("/newfile3", dfd, "creatlink"), 0);
+ ASSERT_EQ(symlinkat("etc/", dfd, "reletc"), 0);
+ ASSERT_EQ(symlinkat("etc/passwd", dfd, "relsym"), 0);
+ ASSERT_EQ(symlinkat("/etc/", dfd, "absetc"), 0);
+ ASSERT_EQ(symlinkat("/etc/passwd", dfd, "abssym"), 0);
+ ASSERT_EQ(symlinkat("/cheeky", dfd, "abscheeky"), 0);
- E_mkdirat(dfd, "cheeky", 0755);
+ ASSERT_EQ(mkdirat(dfd, "cheeky", 0755), 0);
- E_symlinkat("/", dfd, "cheeky/absself");
- E_symlinkat("../../root/", dfd, "cheeky/self");
- E_symlinkat("/../../root/", dfd, "cheeky/garbageself");
+ ASSERT_EQ(symlinkat("/", dfd, "cheeky/absself"), 0);
+ ASSERT_EQ(symlinkat("../../root/", dfd, "cheeky/self"), 0);
+ ASSERT_EQ(symlinkat("/../../root/", dfd, "cheeky/garbageself"), 0);
- E_symlinkat("../cheeky/../etc/../etc/passwd", dfd, "cheeky/passwd");
- E_symlinkat("/../cheeky/../etc/../etc/passwd", dfd, "cheeky/abspasswd");
+ ASSERT_EQ(symlinkat("../cheeky/../etc/../etc/passwd",
+ dfd, "cheeky/passwd"), 0);
+ ASSERT_EQ(symlinkat("/../cheeky/../etc/../etc/passwd",
+ dfd, "cheeky/abspasswd"), 0);
- E_symlinkat("../../../../../../../../../../../../../../etc/passwd",
- dfd, "cheeky/dotdotlink");
- E_symlinkat("/../../../../../../../../../../../../../../etc/passwd",
- dfd, "cheeky/garbagelink");
+ ASSERT_EQ(symlinkat("../../../../../../../../../../../../../../etc/passwd",
+ dfd, "cheeky/dotdotlink"), 0);
+ ASSERT_EQ(symlinkat("/../../../../../../../../../../../../../../etc/passwd",
+ dfd, "cheeky/garbagelink"), 0);
- return dfd;
+ self->rootfd = dfd;
+
+ self->hardcoded_fd = open("/dev/null", O_RDONLY);
+ ASSERT_GE(self->hardcoded_fd, 0);
+ ASSERT_GE(asprintf(&self->hardcoded_fdpath, "self/fd/%d",
+ self->hardcoded_fd), 0);
+ ASSERT_GE(asprintf(&self->procselfexe, "/proc/%d/exe", getpid()), 0);
}
-struct basic_test {
- const char *name;
- const char *dir;
- const char *path;
- struct open_how how;
- bool pass;
- union {
- int err;
- const char *path;
- } out;
-};
-
-#define NUM_OPENAT2_OPATH_TESTS 88
-
-void test_openat2_opath_tests(void)
+FIXTURE_TEARDOWN(openat2_resolve)
{
- int rootfd, hardcoded_fd;
- char *procselfexe, *hardcoded_fdpath;
+ free(self->procselfexe);
+ free(self->hardcoded_fdpath);
+ if (self->hardcoded_fd >= 0)
+ close(self->hardcoded_fd);
+ if (self->rootfd >= 0)
+ close(self->rootfd);
+}
- E_asprintf(&procselfexe, "/proc/%d/exe", getpid());
- rootfd = setup_testdir();
-
- hardcoded_fd = open("/dev/null", O_RDONLY);
- E_assert(hardcoded_fd >= 0, "open fd to hardcode");
- E_asprintf(&hardcoded_fdpath, "self/fd/%d", hardcoded_fd);
-
- struct basic_test tests[] = {
- /** RESOLVE_BENEATH **/
+/* Attempts to cross the dirfd should be blocked with -EXDEV. */
+TEST_F(openat2_resolve, resolve_beneath)
+{
+ struct resolve_test tests[] = {
/* Attempts to cross dirfd should be blocked. */
{ .name = "[beneath] jump to /",
.path = "/", .how.resolve = RESOLVE_BENEATH,
@@ -206,9 +288,17 @@ void test_openat2_opath_tests(void)
{ .name = "[beneath] tricky absolute + garbage link outside $root",
.path = "abscheeky/garbagelink", .how.resolve = RESOLVE_BENEATH,
.out.err = -EXDEV, .pass = false },
+ };
- /** RESOLVE_IN_ROOT **/
- /* All attempts to cross the dirfd will be scoped-to-root. */
+ for (int i = 0; i < ARRAY_SIZE(tests); i++)
+ verify_resolve_test(_metadata, self->rootfd,
+ self->hardcoded_fd, &tests[i]);
+}
+
+/* All attempts to cross the dirfd will be scoped-to-root. */
+TEST_F(openat2_resolve, resolve_in_root)
+{
+ struct resolve_test tests[] = {
{ .name = "[in_root] jump to /",
.path = "/", .how.resolve = RESOLVE_IN_ROOT,
.out.path = NULL, .pass = true },
@@ -297,8 +387,17 @@ void test_openat2_opath_tests(void)
.how.mode = 0700,
.how.resolve = RESOLVE_IN_ROOT,
.out.path = "newfile3", .pass = true },
+ };
- /** RESOLVE_NO_XDEV **/
+ for (int i = 0; i < ARRAY_SIZE(tests); i++)
+ verify_resolve_test(_metadata, self->rootfd,
+ self->hardcoded_fd, &tests[i]);
+}
+
+/* Crossing mount boundaries should be blocked. */
+TEST_F(openat2_resolve, resolve_no_xdev)
+{
+ struct resolve_test tests[] = {
/* Crossing *down* into a mountpoint is disallowed. */
{ .name = "[no_xdev] cross into $mnt",
.path = "mnt", .how.resolve = RESOLVE_NO_XDEV,
@@ -347,10 +446,19 @@ void test_openat2_opath_tests(void)
.out.err = -EXDEV, .pass = false },
/* Except magic-link jumps inside the same vfsmount. */
{ .name = "[no_xdev] jump through magic-link to same procfs",
- .dir = "/proc", .path = hardcoded_fdpath, .how.resolve = RESOLVE_NO_XDEV,
- .out.path = "/proc", .pass = true, },
+ .dir = "/proc", .path = self->hardcoded_fdpath, .how.resolve = RESOLVE_NO_XDEV,
+ .out.path = "/proc", .pass = true, },
+ };
- /** RESOLVE_NO_MAGICLINKS **/
+ for (int i = 0; i < ARRAY_SIZE(tests); i++)
+ verify_resolve_test(_metadata, self->rootfd,
+ self->hardcoded_fd, &tests[i]);
+}
+
+/* Procfs-style magic-link resolution should be blocked. */
+TEST_F(openat2_resolve, resolve_no_magiclinks)
+{
+ struct resolve_test tests[] = {
/* Regular symlinks should work. */
{ .name = "[no_magiclinks] ordinary relative symlink",
.path = "relsym", .how.resolve = RESOLVE_NO_MAGICLINKS,
@@ -365,7 +473,7 @@ void test_openat2_opath_tests(void)
{ .name = "[no_magiclinks] normal path to magic-link with O_NOFOLLOW",
.path = "/proc/self/exe", .how.flags = O_NOFOLLOW,
.how.resolve = RESOLVE_NO_MAGICLINKS,
- .out.path = procselfexe, .pass = true },
+ .out.path = self->procselfexe, .pass = true },
{ .name = "[no_magiclinks] symlink to magic-link path component",
.path = "procroot/etc", .how.resolve = RESOLVE_NO_MAGICLINKS,
.out.err = -ELOOP, .pass = false },
@@ -376,8 +484,17 @@ void test_openat2_opath_tests(void)
.path = "/proc/self/root/etc", .how.flags = O_NOFOLLOW,
.how.resolve = RESOLVE_NO_MAGICLINKS,
.out.err = -ELOOP, .pass = false },
+ };
- /** RESOLVE_NO_SYMLINKS **/
+ for (int i = 0; i < ARRAY_SIZE(tests); i++)
+ verify_resolve_test(_metadata, self->rootfd,
+ self->hardcoded_fd, &tests[i]);
+}
+
+/* All symlink resolution should be blocked. */
+TEST_F(openat2_resolve, resolve_no_symlinks)
+{
+ struct resolve_test tests[] = {
/* Normal paths should work. */
{ .name = "[no_symlinks] ordinary path to '.'",
.path = ".", .how.resolve = RESOLVE_NO_SYMLINKS,
@@ -436,88 +553,9 @@ void test_openat2_opath_tests(void)
.out.err = -ELOOP, .pass = false },
};
- BUILD_BUG_ON(ARRAY_LEN(tests) != NUM_OPENAT2_OPATH_TESTS);
-
- for (int i = 0; i < ARRAY_LEN(tests); i++) {
- int dfd, fd;
- char *fdpath = NULL;
- bool failed;
- void (*resultfn)(const char *msg, ...) = ksft_test_result_pass;
- struct basic_test *test = &tests[i];
-
- if (!openat2_supported) {
- ksft_print_msg("openat2(2) unsupported\n");
- resultfn = ksft_test_result_skip;
- goto skip;
- }
-
- /* Auto-set O_PATH. */
- if (!(test->how.flags & O_CREAT))
- test->how.flags |= O_PATH;
-
- if (test->dir)
- dfd = openat(rootfd, test->dir, O_PATH | O_DIRECTORY);
- else
- dfd = dup(rootfd);
- E_assert(dfd, "failed to openat root '%s': %m", test->dir);
-
- E_dup2(dfd, hardcoded_fd);
-
- fd = sys_openat2(dfd, test->path, &test->how);
- if (test->pass)
- failed = (fd < 0 || !fdequal(fd, rootfd, test->out.path));
- else
- failed = (fd != test->out.err);
- if (fd >= 0) {
- fdpath = fdreadlink(fd);
- close(fd);
- }
- close(dfd);
-
- if (failed) {
- resultfn = ksft_test_result_fail;
-
- ksft_print_msg("openat2 unexpectedly returned ");
- if (fdpath)
- ksft_print_msg("%d['%s']\n", fd, fdpath);
- else
- ksft_print_msg("%d (%s)\n", fd, strerror(-fd));
- }
-
-skip:
- if (test->pass)
- resultfn("%s gives path '%s'\n", test->name,
- test->out.path ?: ".");
- else
- resultfn("%s fails with %d (%s)\n", test->name,
- test->out.err, strerror(-test->out.err));
-
- fflush(stdout);
- free(fdpath);
- }
-
- free(procselfexe);
- close(rootfd);
-
- free(hardcoded_fdpath);
- close(hardcoded_fd);
+ for (int i = 0; i < ARRAY_SIZE(tests); i++)
+ verify_resolve_test(_metadata, self->rootfd,
+ self->hardcoded_fd, &tests[i]);
}
-#define NUM_TESTS NUM_OPENAT2_OPATH_TESTS
-
-int main(int argc, char **argv)
-{
- ksft_print_header();
- ksft_set_plan(NUM_TESTS);
-
- /* NOTE: We should be checking for CAP_SYS_ADMIN here... */
- if (geteuid() != 0)
- ksft_exit_skip("all tests require euid == 0\n");
-
- test_openat2_opath_tests();
-
- if (ksft_get_fail_cnt() + ksft_get_error_cnt() > 0)
- ksft_exit_fail();
- else
- ksft_exit_pass();
-}
+TEST_HARNESS_MAIN
diff --git a/tools/testing/selftests/namespaces/listns_efault_test.c b/tools/testing/selftests/namespaces/listns_efault_test.c
index b570746..26b452c 100644
--- a/tools/testing/selftests/namespaces/listns_efault_test.c
+++ b/tools/testing/selftests/namespaces/listns_efault_test.c
@@ -38,7 +38,7 @@ TEST(listns_partial_fault_with_ns_cleanup)
__u64 *ns_ids;
ssize_t ret;
long page_size;
- pid_t pid, iter_pid;
+ pid_t pid, iter_pid, ns_pids[5];
int pidfds[5];
int sv[5][2];
int iter_pidfd;
@@ -114,6 +114,7 @@ TEST(listns_partial_fault_with_ns_cleanup)
pid = create_child(&pidfds[i], CLONE_NEWNS);
ASSERT_NE(pid, -1);
+ ns_pids[i] = pid;
if (pid == 0) {
close(sv[i][0]); /* Close parent end */
@@ -164,7 +165,7 @@ TEST(listns_partial_fault_with_ns_cleanup)
/* Wait for all mount namespace children to exit and cleanup */
for (i = 0; i < 5; i++) {
- waitpid(-1, NULL, 0);
+ waitpid(ns_pids[i], NULL, 0);
close(sv[i][0]);
close(pidfds[i]);
}
@@ -175,6 +176,12 @@ TEST(listns_partial_fault_with_ns_cleanup)
ASSERT_EQ(ret, iter_pid);
close(iter_pidfd);
+ /* If listns() is not supported the iterator exits cleanly via ENOSYS */
+ if (WIFEXITED(status) && WEXITSTATUS(status) == PIDFD_SKIP) {
+ munmap(map, page_size);
+ SKIP(return, "listns() not supported");
+ }
+
/* Should have been killed */
ASSERT_TRUE(WIFSIGNALED(status));
ASSERT_EQ(WTERMSIG(status), SIGKILL);
@@ -250,7 +257,7 @@ TEST(listns_late_fault_with_ns_cleanup)
__u64 *ns_ids;
ssize_t ret;
long page_size;
- pid_t pid, iter_pid;
+ pid_t pid, iter_pid, ns_pids[10];
int pidfds[10];
int sv[10][2];
int iter_pidfd;
@@ -320,6 +327,7 @@ TEST(listns_late_fault_with_ns_cleanup)
pid = create_child(&pidfds[i], CLONE_NEWNS);
ASSERT_NE(pid, -1);
+ ns_pids[i] = pid;
if (pid == 0) {
close(sv[i][0]); /* Close parent end */
@@ -373,7 +381,7 @@ TEST(listns_late_fault_with_ns_cleanup)
/* Wait for all children and cleanup */
for (i = 0; i < 10; i++) {
- waitpid(-1, NULL, 0);
+ waitpid(ns_pids[i], NULL, 0);
close(sv[i][0]);
close(pidfds[i]);
}
@@ -384,6 +392,12 @@ TEST(listns_late_fault_with_ns_cleanup)
ASSERT_EQ(ret, iter_pid);
close(iter_pidfd);
+ /* If listns() is not supported the iterator exits cleanly via ENOSYS */
+ if (WIFEXITED(status) && WEXITSTATUS(status) == PIDFD_SKIP) {
+ munmap(map, page_size);
+ SKIP(return, "listns() not supported");
+ }
+
/* Should have been killed */
ASSERT_TRUE(WIFSIGNALED(status));
ASSERT_EQ(WTERMSIG(status), SIGKILL);
@@ -402,7 +416,7 @@ TEST(listns_mnt_ns_cleanup_on_fault)
__u64 *ns_ids;
ssize_t ret;
long page_size;
- pid_t pid, iter_pid;
+ pid_t pid, iter_pid, ns_pids[8];
int pidfds[8];
int sv[8][2];
int iter_pidfd;
@@ -462,6 +476,7 @@ TEST(listns_mnt_ns_cleanup_on_fault)
pid = create_child(&pidfds[i], CLONE_NEWNS);
ASSERT_NE(pid, -1);
+ ns_pids[i] = pid;
if (pid == 0) {
close(sv[i][0]); /* Close parent end */
@@ -508,7 +523,7 @@ TEST(listns_mnt_ns_cleanup_on_fault)
/* Wait for children and cleanup */
for (i = 0; i < 8; i++) {
- waitpid(-1, NULL, 0);
+ waitpid(ns_pids[i], NULL, 0);
close(sv[i][0]);
close(pidfds[i]);
}
@@ -519,6 +534,12 @@ TEST(listns_mnt_ns_cleanup_on_fault)
ASSERT_EQ(ret, iter_pid);
close(iter_pidfd);
+ /* If listns() is not supported the iterator exits cleanly via ENOSYS */
+ if (WIFEXITED(status) && WEXITSTATUS(status) == PIDFD_SKIP) {
+ munmap(map, page_size);
+ SKIP(return, "listns() not supported");
+ }
+
/* Should have been killed */
ASSERT_TRUE(WIFSIGNALED(status));
ASSERT_EQ(WTERMSIG(status), SIGKILL);
diff --git a/tools/testing/selftests/namespaces/nsid_test.c b/tools/testing/selftests/namespaces/nsid_test.c
index b4a14c6..46dc838 100644
--- a/tools/testing/selftests/namespaces/nsid_test.c
+++ b/tools/testing/selftests/namespaces/nsid_test.c
@@ -25,14 +25,24 @@
/* Fixture for tests that create child processes */
FIXTURE(nsid) {
pid_t child_pid;
+ pid_t grandchild_pid;
};
FIXTURE_SETUP(nsid) {
self->child_pid = 0;
+ self->grandchild_pid = 0;
}
FIXTURE_TEARDOWN(nsid) {
- /* Clean up any child process that may still be running */
+ /*
+ * Kill grandchild first: timens_separate and pidns_separate fork a
+ * grandchild that calls pause(). It is reparented to init on child
+ * exit and keeps the test runner's tap pipe open, hanging the runner.
+ */
+ if (self->grandchild_pid > 0) {
+ kill(self->grandchild_pid, SIGKILL);
+ waitpid(self->grandchild_pid, NULL, 0);
+ }
if (self->child_pid > 0) {
kill(self->child_pid, SIGKILL);
waitpid(self->child_pid, NULL, 0);
@@ -676,6 +686,7 @@ TEST_F(nsid, timens_separate)
pid_t grandchild_pid;
ASSERT_EQ(read(pipefd[0], &grandchild_pid, sizeof(grandchild_pid)), sizeof(grandchild_pid));
+ self->grandchild_pid = grandchild_pid;
close(pipefd[0]);
/* Open grandchild's time namespace */
@@ -797,6 +808,7 @@ TEST_F(nsid, pidns_separate)
pid_t grandchild_pid;
ASSERT_EQ(read(pipefd[0], &grandchild_pid, sizeof(grandchild_pid)), sizeof(grandchild_pid));
+ self->grandchild_pid = grandchild_pid;
close(pipefd[0]);
/* Open grandchild's PID namespace */
diff --git a/tools/testing/selftests/nolibc/Makefile.include b/tools/testing/selftests/nolibc/Makefile.include
index 96fe2bc..c30ca3a 100644
--- a/tools/testing/selftests/nolibc/Makefile.include
+++ b/tools/testing/selftests/nolibc/Makefile.include
@@ -6,7 +6,7 @@
$(__CFLAGS_STACKPROTECTOR))
_CFLAGS_SANITIZER ?= $(call cc-option,-fsanitize=undefined -fsanitize-trap=all)
CFLAGS_NOLIBC_TEST ?= -Os -fno-ident -fno-asynchronous-unwind-tables -std=c89 \
- -W -Wall -Wextra -Wundef \
+ -W -Wall -Wextra -Wundef -Wwrite-strings \
$(call cc-option,-fno-stack-protector) $(call cc-option,-Wmissing-prototypes) \
$(_CFLAGS_STACKPROTECTOR) $(_CFLAGS_SANITIZER)
diff --git a/tools/testing/selftests/nolibc/Makefile.nolibc b/tools/testing/selftests/nolibc/Makefile.nolibc
index f30bc68..06f881e 100644
--- a/tools/testing/selftests/nolibc/Makefile.nolibc
+++ b/tools/testing/selftests/nolibc/Makefile.nolibc
@@ -64,6 +64,7 @@
ARCH_sparc32 = sparc
ARCH_sparc64 = sparc
ARCH_sh4 = sh
+ARCH_parisc32 = parisc
ARCH := $(or $(ARCH_$(XARCH)),$(XARCH))
# kernel image names by architecture
@@ -74,33 +75,18 @@
IMAGE_arm64 = arch/arm64/boot/Image
IMAGE_arm = arch/arm/boot/zImage
IMAGE_armthumb = arch/arm/boot/zImage
-IMAGE_mips32le = vmlinuz
-IMAGE_mips32be = vmlinuz
-IMAGE_mipsn32le = vmlinuz
-IMAGE_mipsn32be = vmlinuz
-IMAGE_mips64le = vmlinuz
-IMAGE_mips64be = vmlinuz
-IMAGE_ppc = vmlinux
-IMAGE_ppc64 = vmlinux
IMAGE_ppc64le = arch/powerpc/boot/zImage
-IMAGE_riscv = arch/riscv/boot/Image
IMAGE_riscv32 = arch/riscv/boot/Image
IMAGE_riscv64 = arch/riscv/boot/Image
IMAGE_s390x = arch/s390/boot/bzImage
IMAGE_loongarch = arch/loongarch/boot/vmlinuz.efi
IMAGE_sparc32 = arch/sparc/boot/image
IMAGE_sparc64 = arch/sparc/boot/image
-IMAGE_m68k = vmlinux
IMAGE_sh4 = arch/sh/boot/zImage
-IMAGE = $(objtree)/$(IMAGE_$(XARCH))
+IMAGE = $(objtree)/$(or $(IMAGE_$(XARCH)),vmlinux)
IMAGE_NAME = $(notdir $(IMAGE))
# default kernel configurations that appear to be usable
-DEFCONFIG_i386 = defconfig
-DEFCONFIG_x86_64 = defconfig
-DEFCONFIG_x32 = defconfig
-DEFCONFIG_x86 = defconfig
-DEFCONFIG_arm64 = defconfig
DEFCONFIG_arm = multi_v7_defconfig
DEFCONFIG_armthumb = multi_v7_defconfig
DEFCONFIG_mips32le = malta_defconfig
@@ -112,20 +98,18 @@
DEFCONFIG_ppc = pmac32_defconfig
DEFCONFIG_ppc64 = powernv_be_defconfig
DEFCONFIG_ppc64le = powernv_defconfig
-DEFCONFIG_riscv = defconfig
DEFCONFIG_riscv32 = rv32_defconfig
-DEFCONFIG_riscv64 = defconfig
-DEFCONFIG_s390x = defconfig
-DEFCONFIG_loongarch = defconfig
DEFCONFIG_sparc32 = sparc32_defconfig
DEFCONFIG_sparc64 = sparc64_defconfig
DEFCONFIG_m68k = virt_defconfig
DEFCONFIG_sh4 = rts7751r2dplus_defconfig
-DEFCONFIG = $(DEFCONFIG_$(XARCH))
+DEFCONFIG_openrisc = virt_defconfig
+DEFCONFIG = $(or $(DEFCONFIG_$(XARCH)),defconfig)
EXTRACONFIG_x32 = -e CONFIG_X86_X32_ABI
EXTRACONFIG_arm = -e CONFIG_NAMESPACES
EXTRACONFIG_armthumb = -e CONFIG_NAMESPACES
+EXTRACONFIG_sparc32 = -e CONFIG_TMPFS
EXTRACONFIG_m68k = -e CONFIG_BLK_DEV_INITRD
EXTRACONFIG_sh4 = -e CONFIG_BLK_DEV_INITRD -e CONFIG_CMDLINE_FROM_BOOTLOADER
EXTRACONFIG = $(EXTRACONFIG_$(XARCH))
@@ -134,37 +118,27 @@
TEST =
# QEMU_ARCH: arch names used by qemu
-QEMU_ARCH_i386 = i386
-QEMU_ARCH_x86_64 = x86_64
QEMU_ARCH_x32 = x86_64
QEMU_ARCH_x86 = x86_64
QEMU_ARCH_arm64 = aarch64
-QEMU_ARCH_arm = arm
QEMU_ARCH_armthumb = arm
QEMU_ARCH_mips32le = mipsel # works with malta_defconfig
-QEMU_ARCH_mips32be = mips
+QEMU_ARCH_mips32be = mips
QEMU_ARCH_mipsn32le = mips64el
QEMU_ARCH_mipsn32be = mips64
QEMU_ARCH_mips64le = mips64el
QEMU_ARCH_mips64be = mips64
-QEMU_ARCH_ppc = ppc
-QEMU_ARCH_ppc64 = ppc64
QEMU_ARCH_ppc64le = ppc64
-QEMU_ARCH_riscv = riscv64
-QEMU_ARCH_riscv32 = riscv32
-QEMU_ARCH_riscv64 = riscv64
-QEMU_ARCH_s390x = s390x
QEMU_ARCH_loongarch = loongarch64
QEMU_ARCH_sparc32 = sparc
-QEMU_ARCH_sparc64 = sparc64
-QEMU_ARCH_m68k = m68k
-QEMU_ARCH_sh4 = sh4
-QEMU_ARCH = $(QEMU_ARCH_$(XARCH))
+QEMU_ARCH_openrisc = or1k
+QEMU_ARCH_parisc32 = hppa
+QEMU_ARCH = $(or $(QEMU_ARCH_$(XARCH)),$(XARCH))
QEMU_ARCH_USER_ppc64le = ppc64le
QEMU_ARCH_USER_mipsn32le = mipsn32el
QEMU_ARCH_USER_mipsn32be = mipsn32
-QEMU_ARCH_USER = $(or $(QEMU_ARCH_USER_$(XARCH)),$(QEMU_ARCH_$(XARCH)))
+QEMU_ARCH_USER = $(or $(QEMU_ARCH_USER_$(XARCH)),$(QEMU_ARCH))
QEMU_BIOS_DIR = /usr/share/edk2/
QEMU_BIOS_loongarch = $(QEMU_BIOS_DIR)/loongarch64/OVMF_CODE.fd
@@ -190,7 +164,6 @@
QEMU_ARGS_ppc = -M g3beige -append "console=ttyS0 panic=-1 $(TEST:%=NOLIBC_TEST=%)"
QEMU_ARGS_ppc64 = -M powernv -append "console=hvc0 panic=-1 $(TEST:%=NOLIBC_TEST=%)"
QEMU_ARGS_ppc64le = -M powernv -append "console=hvc0 panic=-1 $(TEST:%=NOLIBC_TEST=%)"
-QEMU_ARGS_riscv = -M virt -append "console=ttyS0 panic=-1 $(TEST:%=NOLIBC_TEST=%)"
QEMU_ARGS_riscv32 = -M virt -append "console=ttyS0 panic=-1 $(TEST:%=NOLIBC_TEST=%)"
QEMU_ARGS_riscv64 = -M virt -append "console=ttyS0 panic=-1 $(TEST:%=NOLIBC_TEST=%)"
QEMU_ARGS_s390x = -M s390-ccw-virtio -append "console=ttyS0 panic=-1 $(TEST:%=NOLIBC_TEST=%)"
@@ -199,6 +172,8 @@
QEMU_ARGS_sparc64 = -M sun4u -append "console=ttyS0,115200 panic=-1 $(TEST:%=NOLIBC_TEST=%)"
QEMU_ARGS_m68k = -M virt -append "console=ttyGF0,115200 panic=-1 $(TEST:%=NOLIBC_TEST=%)"
QEMU_ARGS_sh4 = -M r2d -serial file:/dev/stdout -append "console=ttySC1,115200 panic=-1 $(TEST:%=NOLIBC_TEST=%)"
+QEMU_ARGS_openrisc = -M virt -m 512M -append "console=ttyS0 panic=-1 $(TEST:%=NOLIBC_TEST=%)"
+QEMU_ARGS_parisc32 = -M B160L -append "console=ttyS0 panic=-1 $(TEST:%=NOLIBC_TEST=%)"
QEMU_ARGS = -m 1G $(QEMU_ARGS_$(XARCH)) $(QEMU_ARGS_BIOS) $(QEMU_ARGS_EXTRA)
# OUTPUT is only set when run from the main makefile, otherwise
@@ -215,6 +190,7 @@
CFLAGS_x32 = -mx32
CFLAGS_arm = -marm
CFLAGS_armthumb = -mthumb -march=armv6t2
+CFLAGS_parisc32 = -mfast-indirect-calls
CFLAGS_ppc = -m32 -mbig-endian -mno-vsx $(call cc-option,-mmultiple)
CFLAGS_ppc64 = -m64 -mbig-endian -mno-vsx $(call cc-option,-mmultiple)
CFLAGS_ppc64le = -m64 -mlittle-endian -mno-vsx $(call cc-option,-mabi=elfv2)
@@ -233,6 +209,7 @@
endif
LDLIBS_ppc = $(if $(LLVM),,-lgcc)
+LDLIBS_openrisc = $(if $(LLVM),,-lgcc)
LDLIBS = $(LDLIBS_$(XARCH))
include Makefile.include
diff --git a/tools/testing/selftests/nolibc/nolibc-test.c b/tools/testing/selftests/nolibc/nolibc-test.c
index d3c4fac..c1c1ce4 100644
--- a/tools/testing/selftests/nolibc/nolibc-test.c
+++ b/tools/testing/selftests/nolibc/nolibc-test.c
@@ -2,6 +2,7 @@
#define _GNU_SOURCE
#define _LARGEFILE64_SOURCE
+#define _FILE_OFFSET_BITS 64
/* libc-specific include files
* The program may be built in 3 ways:
@@ -45,6 +46,7 @@
#include <stdbool.h>
#include <byteswap.h>
#include <endian.h>
+#include <alloca.h>
#pragma GCC diagnostic ignored "-Wmissing-prototypes"
@@ -647,20 +649,25 @@ int expect_str_buf_eq(size_t expr, const char *buf, size_t val, int llen, const
return 0;
}
+enum strtox_func {
+ strtox_func_strtol,
+ strtox_func_strtoul,
+};
+
#define EXPECT_STRTOX(cond, func, input, base, expected, chars, expected_errno) \
- do { if (!(cond)) result(llen, SKIPPED); else ret += expect_strtox(llen, func, input, base, expected, chars, expected_errno); } while (0)
+ do { if (!(cond)) result(llen, SKIPPED); else ret += expect_strtox(llen, strtox_func_ ## func, input, base, expected, chars, expected_errno); } while (0)
static __attribute__((unused))
-int expect_strtox(int llen, void *func, const char *input, int base, intmax_t expected, int expected_chars, int expected_errno)
+int expect_strtox(int llen, enum strtox_func func, const char *input, int base, intmax_t expected, int expected_chars, int expected_errno)
{
char *endptr;
int actual_errno, actual_chars;
intmax_t r;
errno = 0;
- if (func == strtol) {
+ if (func == strtox_func_strtol) {
r = strtol(input, &endptr, base);
- } else if (func == strtoul) {
+ } else if (func == strtox_func_strtoul) {
r = strtoul(input, &endptr, base);
} else {
result(llen, FAIL);
@@ -797,7 +804,7 @@ int test_getdents64(const char *dir)
int fd, ret;
int err;
- ret = fd = open(dir, O_RDONLY | O_DIRECTORY, 0);
+ ret = fd = open(dir, O_RDONLY | O_DIRECTORY);
if (ret < 0)
return ret;
@@ -1010,6 +1017,57 @@ int test_fork(enum fork_type type)
}
}
+int test_ftruncate(void)
+{
+ struct stat stat_buf;
+ int ret, fd;
+
+ ret = ftruncate(-1, 0);
+ if (ret != -1 || errno != EBADF) {
+ errno = EINVAL;
+ return __LINE__;
+ }
+
+ fd = memfd_create(__func__, 0);
+ if (fd == -1)
+ return __LINE__;
+
+ /*
+ * This also tests that the high 32-bit half is passed through correctly.
+ * If it gets lost, the kernel will see a positive number and not fail.
+ */
+ ret = ftruncate(fd, -1);
+ if (!(ret == -1 && errno == EINVAL)) {
+ if (ret == 0)
+ errno = EINVAL;
+ ret = __LINE__;
+ goto end;
+ }
+
+ ret = ftruncate(fd, 42);
+ if (ret != 0) {
+ ret = __LINE__;
+ goto end;
+ }
+
+ ret = fstat(fd, &stat_buf);
+ if (ret != 0) {
+ ret = __LINE__;
+ goto end;
+ }
+
+ if (stat_buf.st_size != 42) {
+ errno = EINVAL;
+ ret = __LINE__;
+ goto end;
+ }
+
+end:
+ close(fd);
+
+ return ret;
+}
+
int test_stat_timestamps(void)
{
struct stat st;
@@ -1298,6 +1356,45 @@ int test_openat(void)
return 0;
}
+int test_open_mode(void)
+{
+ const mode_t mode = 0444;
+ struct stat stat_buf;
+ int fd, ret;
+
+ fd = open("/tmp", O_TMPFILE | O_RDWR, mode);
+ if (fd == -1)
+ return -1;
+
+ ret = fstat(fd, &stat_buf);
+ close(fd);
+
+ if (ret == -1)
+ return -1;
+
+ if ((stat_buf.st_mode & 0777) != mode)
+ return -1;
+
+ return 0;
+}
+
+int test_nolibc_enosys(void)
+{
+ if (true)
+ return 0;
+
+#if defined(NOLIBC)
+ /*
+ * __nolibc_enosys() will fail the compilation.
+ * Make sure it can be optimized away if not actually called.
+ */
+ if (__nolibc_enosys("something") != -ENOSYS)
+ return 1;
+#endif
+
+ return 0;
+}
+
int test_namespace(void)
{
int original_ns, new_ns, ret;
@@ -1364,6 +1461,52 @@ int test_namespace(void)
return ret;
}
+int test_large_file(void)
+{
+ off_t large_seek = ((off_t)UINT32_MAX) + 100;
+ int fd, ret, saved_errno;
+ ssize_t written;
+ off_t off;
+
+#if defined(__mips__) && defined(_ABIN32)
+ /* https://lore.kernel.org/qemu-devel/fed03914-a95a-4522-a432-f129264cb2ac@t-8ch.de/ */
+ if (getpid() != 1)
+ return 0;
+#endif
+
+ if (large_seek < UINT32_MAX) {
+ errno = EOVERFLOW;
+ return -1;
+ }
+
+ fd = open("/tmp", O_TMPFILE | O_RDWR, 0644);
+ if (fd == -1)
+ return -1;
+
+ off = lseek(fd, large_seek, SEEK_CUR);
+ if (off == -1) {
+ ret = off;
+ goto out;
+ } else if (off != large_seek) {
+ errno = ERANGE;
+ ret = -1;
+ goto out;
+ }
+
+ written = write(fd, "1", 1);
+ if (written == -1) {
+ ret = written;
+ goto out;
+ }
+
+ ret = 0;
+out:
+ saved_errno = errno;
+ close(fd);
+ errno = saved_errno;
+ return ret;
+}
+
/* Run syscall tests between IDs <min> and <max>.
* Return 0 on success, non-zero on failure.
*/
@@ -1441,12 +1584,13 @@ int run_syscall(int min, int max)
CASE_TEST(dup2_m1); tmp = dup2(-1, 100); EXPECT_SYSER(1, tmp, -1, EBADF); if (tmp != -1) close(tmp); break;
CASE_TEST(dup3_0); tmp = dup3(0, 100, 0); EXPECT_SYSNE(1, tmp, -1); close(tmp); break;
CASE_TEST(dup3_m1); tmp = dup3(-1, 100, 0); EXPECT_SYSER(1, tmp, -1, EBADF); if (tmp != -1) close(tmp); break;
- CASE_TEST(execve_root); EXPECT_SYSER(1, execve("/", (char*[]){ [0] = "/", [1] = NULL }, NULL), -1, EACCES); break;
+ CASE_TEST(execve_root); EXPECT_SYSER(1, execve("/", (char*[]){ [0] = (char []){"/"}, [1] = NULL }, NULL), -1, EACCES); break;
CASE_TEST(fchdir_stdin); EXPECT_SYSER(1, fchdir(STDIN_FILENO), -1, ENOTDIR); break;
CASE_TEST(fchdir_badfd); EXPECT_SYSER(1, fchdir(-1), -1, EBADF); break;
CASE_TEST(file_stream); EXPECT_SYSZR(1, test_file_stream()); break;
CASE_TEST(file_stream_wsr); EXPECT_SYSZR(1, test_file_stream_wsr()); break;
CASE_TEST(fork); EXPECT_SYSZR(1, test_fork(FORK_STANDARD)); break;
+ CASE_TEST(ftruncate); EXPECT_SYSZR(1, test_ftruncate()); break;
CASE_TEST(getdents64_root); EXPECT_SYSNE(1, test_getdents64("/"), -1); break;
CASE_TEST(getdents64_null); EXPECT_SYSER(1, test_getdents64("/dev/null"), -1, ENOTDIR); break;
CASE_TEST(directories); EXPECT_SYSZR(is_nolibc && proc, test_dirent()); break;
@@ -1466,9 +1610,11 @@ int run_syscall(int min, int max)
CASE_TEST(munmap_bad); EXPECT_SYSER(1, munmap(NULL, 0), -1, EINVAL); break;
CASE_TEST(mmap_munmap_good); EXPECT_SYSZR(1, test_mmap_munmap()); break;
CASE_TEST(nanosleep); ts.tv_nsec = -1; EXPECT_SYSER(1, nanosleep(&ts, NULL), -1, EINVAL); break;
+ CASE_TEST(nolibc_enosys); EXPECT_ZR(is_nolibc, test_nolibc_enosys()); break;
CASE_TEST(open_tty); EXPECT_SYSNE(1, tmp = open("/dev/null", O_RDONLY), -1); if (tmp != -1) close(tmp); break;
CASE_TEST(open_blah); EXPECT_SYSER(1, tmp = open("/proc/self/blah", O_RDONLY), -1, ENOENT); if (tmp != -1) close(tmp); break;
CASE_TEST(openat_dir); EXPECT_SYSZR(1, test_openat()); break;
+ CASE_TEST(open_mode); EXPECT_SYSZR(1, test_open_mode()); break;
CASE_TEST(pipe); EXPECT_SYSZR(1, test_pipe()); break;
CASE_TEST(poll_null); EXPECT_SYSZR(1, poll(NULL, 0, 0)); break;
CASE_TEST(poll_stdout); EXPECT_SYSNE(1, ({ struct pollfd fds = { 1, POLLOUT, 0}; poll(&fds, 1, 0); }), -1); break;
@@ -1508,6 +1654,7 @@ int run_syscall(int min, int max)
CASE_TEST(_syscall_noargs); EXPECT_SYSEQ(is_nolibc, _syscall(__NR_getpid), getpid()); break;
CASE_TEST(_syscall_args); EXPECT_SYSEQ(is_nolibc, _syscall(__NR_statx, 0, NULL, 0, 0, NULL), -EFAULT); break;
CASE_TEST(namespace); EXPECT_SYSZR(euid0 && proc, test_namespace()); break;
+ CASE_TEST(largefile); EXPECT_SYSZR(1, test_large_file()); break;
case __LINE__:
return ret; /* must be last */
/* note: do not set any defaults so as to permit holes above */
@@ -1516,6 +1663,18 @@ int run_syscall(int min, int max)
return ret;
}
+int test_alloca(void)
+{
+ uint64_t *x;
+
+ x = alloca(sizeof(*x));
+
+ *x = 0x1234;
+ __asm__ ("" : "+r" (x));
+
+ return *x - 0x1234;
+}
+
int test_difftime(void)
{
if (difftime(200., 100.) != 100.)
@@ -1731,6 +1890,7 @@ int run_stdlib(int min, int max)
CASE_TEST(toupper_noop); EXPECT_EQ(1, toupper('A'), 'A'); break;
CASE_TEST(abs); EXPECT_EQ(1, abs(-10), 10); break;
CASE_TEST(abs_noop); EXPECT_EQ(1, abs(10), 10); break;
+ CASE_TEST(alloca); EXPECT_ZR(1, test_alloca()); break;
CASE_TEST(difftime); EXPECT_ZR(1, test_difftime()); break;
CASE_TEST(memchr_foobar6_o); EXPECT_STREQ(1, memchr("foobar", 'o', 6), "oobar"); break;
CASE_TEST(memchr_foobar3_b); EXPECT_STRZR(1, memchr("foobar", 'b', 3)); break;
diff --git a/tools/testing/selftests/nolibc/run-tests.sh b/tools/testing/selftests/nolibc/run-tests.sh
index cd43909..6460e25 100755
--- a/tools/testing/selftests/nolibc/run-tests.sh
+++ b/tools/testing/selftests/nolibc/run-tests.sh
@@ -21,6 +21,7 @@
i386 x86_64 x32
arm64 arm armthumb
mips32le mips32be mipsn32le mipsn32be mips64le mips64be
+ openrisc
ppc ppc64 ppc64le
riscv32 riscv64
s390x
@@ -28,6 +29,7 @@
sparc32 sparc64
m68k
sh4
+ parisc32
)
archs="${all_archs[@]}"
@@ -107,6 +109,7 @@
case "$1" in
arm64) echo aarch64;;
armthumb) echo arm;;
+ openrisc) echo or1k;;
ppc) echo powerpc;;
ppc64) echo powerpc64;;
ppc64le) echo powerpc64;;
@@ -116,6 +119,7 @@
s390*) echo s390;;
sparc*) echo sparc64;;
x32*) echo x86_64;;
+ parisc32) echo hppa;;
*) echo "$1";;
esac
}
@@ -173,6 +177,10 @@
fi
MAKE=(make -f Makefile.nolibc -j"${nproc}" XARCH="${arch}" CROSS_COMPILE="${cross_compile}" LLVM="${llvm}" O="${build_dir}")
+ if [ "$arch" = "parisc32" ]; then
+ MAKE+=("CROSS32CC=${cross_compile}gcc")
+ fi
+
case "$test_mode" in
'system')
test_target=run
@@ -185,7 +193,7 @@
exit 1
esac
printf '%-15s' "$arch:"
- if [ "$arch" = "m68k" -o "$arch" = "sh4" ] && [ "$llvm" = "1" ]; then
+ if [ "$arch" = "m68k" -o "$arch" = "sh4" -o "$arch" = "openrisc" -o "$arch" = "parisc32" ] && [ "$llvm" = "1" ]; then
echo "Unsupported configuration"
return
fi
diff --git a/tools/testing/selftests/openat2/helpers.c b/tools/testing/selftests/openat2/helpers.c
deleted file mode 100644
index 5074681..0000000
--- a/tools/testing/selftests/openat2/helpers.c
+++ /dev/null
@@ -1,109 +0,0 @@
-// SPDX-License-Identifier: GPL-2.0-or-later
-/*
- * Author: Aleksa Sarai <cyphar@cyphar.com>
- * Copyright (C) 2018-2019 SUSE LLC.
- */
-
-#define _GNU_SOURCE
-#include <errno.h>
-#include <fcntl.h>
-#include <stdbool.h>
-#include <string.h>
-#include <syscall.h>
-#include <limits.h>
-
-#include "helpers.h"
-
-bool needs_openat2(const struct open_how *how)
-{
- return how->resolve != 0;
-}
-
-int raw_openat2(int dfd, const char *path, void *how, size_t size)
-{
- int ret = syscall(__NR_openat2, dfd, path, how, size);
- return ret >= 0 ? ret : -errno;
-}
-
-int sys_openat2(int dfd, const char *path, struct open_how *how)
-{
- return raw_openat2(dfd, path, how, sizeof(*how));
-}
-
-int sys_openat(int dfd, const char *path, struct open_how *how)
-{
- int ret = openat(dfd, path, how->flags, how->mode);
- return ret >= 0 ? ret : -errno;
-}
-
-int sys_renameat2(int olddirfd, const char *oldpath,
- int newdirfd, const char *newpath, unsigned int flags)
-{
- int ret = syscall(__NR_renameat2, olddirfd, oldpath,
- newdirfd, newpath, flags);
- return ret >= 0 ? ret : -errno;
-}
-
-int touchat(int dfd, const char *path)
-{
- int fd = openat(dfd, path, O_CREAT, 0700);
- if (fd >= 0)
- close(fd);
- return fd;
-}
-
-char *fdreadlink(int fd)
-{
- char *target, *tmp;
-
- E_asprintf(&tmp, "/proc/self/fd/%d", fd);
-
- target = malloc(PATH_MAX);
- if (!target)
- ksft_exit_fail_msg("fdreadlink: malloc failed\n");
- memset(target, 0, PATH_MAX);
-
- E_readlink(tmp, target, PATH_MAX);
- free(tmp);
- return target;
-}
-
-bool fdequal(int fd, int dfd, const char *path)
-{
- char *fdpath, *dfdpath, *other;
- bool cmp;
-
- fdpath = fdreadlink(fd);
- dfdpath = fdreadlink(dfd);
-
- if (!path)
- E_asprintf(&other, "%s", dfdpath);
- else if (*path == '/')
- E_asprintf(&other, "%s", path);
- else
- E_asprintf(&other, "%s/%s", dfdpath, path);
-
- cmp = !strcmp(fdpath, other);
-
- free(fdpath);
- free(dfdpath);
- free(other);
- return cmp;
-}
-
-bool openat2_supported = false;
-
-void __attribute__((constructor)) init(void)
-{
- struct open_how how = {};
- int fd;
-
- BUILD_BUG_ON(sizeof(struct open_how) != OPEN_HOW_SIZE_VER0);
-
- /* Check openat2(2) support. */
- fd = sys_openat2(AT_FDCWD, ".", &how);
- openat2_supported = (fd >= 0);
-
- if (fd >= 0)
- close(fd);
-}
diff --git a/tools/testing/selftests/openat2/helpers.h b/tools/testing/selftests/openat2/helpers.h
deleted file mode 100644
index 510e606..0000000
--- a/tools/testing/selftests/openat2/helpers.h
+++ /dev/null
@@ -1,108 +0,0 @@
-// SPDX-License-Identifier: GPL-2.0-or-later
-/*
- * Author: Aleksa Sarai <cyphar@cyphar.com>
- * Copyright (C) 2018-2019 SUSE LLC.
- */
-
-#ifndef __RESOLVEAT_H__
-#define __RESOLVEAT_H__
-
-#define _GNU_SOURCE
-#include <stdint.h>
-#include <stdbool.h>
-#include <errno.h>
-#include <linux/types.h>
-#include "kselftest.h"
-
-#define ARRAY_LEN(X) (sizeof (X) / sizeof (*(X)))
-#define BUILD_BUG_ON(e) ((void)(sizeof(struct { int:(-!!(e)); })))
-
-#ifndef SYS_openat2
-#ifndef __NR_openat2
-#define __NR_openat2 437
-#endif /* __NR_openat2 */
-#define SYS_openat2 __NR_openat2
-#endif /* SYS_openat2 */
-
-/*
- * Arguments for how openat2(2) should open the target path. If @resolve is
- * zero, then openat2(2) operates very similarly to openat(2).
- *
- * However, unlike openat(2), unknown bits in @flags result in -EINVAL rather
- * than being silently ignored. @mode must be zero unless one of {O_CREAT,
- * O_TMPFILE} are set.
- *
- * @flags: O_* flags.
- * @mode: O_CREAT/O_TMPFILE file mode.
- * @resolve: RESOLVE_* flags.
- */
-struct open_how {
- __u64 flags;
- __u64 mode;
- __u64 resolve;
-};
-
-#define OPEN_HOW_SIZE_VER0 24 /* sizeof first published struct */
-#define OPEN_HOW_SIZE_LATEST OPEN_HOW_SIZE_VER0
-
-bool needs_openat2(const struct open_how *how);
-
-#ifndef RESOLVE_IN_ROOT
-/* how->resolve flags for openat2(2). */
-#define RESOLVE_NO_XDEV 0x01 /* Block mount-point crossings
- (includes bind-mounts). */
-#define RESOLVE_NO_MAGICLINKS 0x02 /* Block traversal through procfs-style
- "magic-links". */
-#define RESOLVE_NO_SYMLINKS 0x04 /* Block traversal through all symlinks
- (implies OEXT_NO_MAGICLINKS) */
-#define RESOLVE_BENEATH 0x08 /* Block "lexical" trickery like
- "..", symlinks, and absolute
- paths which escape the dirfd. */
-#define RESOLVE_IN_ROOT 0x10 /* Make all jumps to "/" and ".."
- be scoped inside the dirfd
- (similar to chroot(2)). */
-#endif /* RESOLVE_IN_ROOT */
-
-#define E_func(func, ...) \
- do { \
- errno = 0; \
- if (func(__VA_ARGS__) < 0) \
- ksft_exit_fail_msg("%s:%d %s failed - errno:%d\n", \
- __FILE__, __LINE__, #func, errno); \
- } while (0)
-
-#define E_asprintf(...) E_func(asprintf, __VA_ARGS__)
-#define E_chmod(...) E_func(chmod, __VA_ARGS__)
-#define E_dup2(...) E_func(dup2, __VA_ARGS__)
-#define E_fchdir(...) E_func(fchdir, __VA_ARGS__)
-#define E_fstatat(...) E_func(fstatat, __VA_ARGS__)
-#define E_kill(...) E_func(kill, __VA_ARGS__)
-#define E_mkdirat(...) E_func(mkdirat, __VA_ARGS__)
-#define E_mount(...) E_func(mount, __VA_ARGS__)
-#define E_prctl(...) E_func(prctl, __VA_ARGS__)
-#define E_readlink(...) E_func(readlink, __VA_ARGS__)
-#define E_setresuid(...) E_func(setresuid, __VA_ARGS__)
-#define E_symlinkat(...) E_func(symlinkat, __VA_ARGS__)
-#define E_touchat(...) E_func(touchat, __VA_ARGS__)
-#define E_unshare(...) E_func(unshare, __VA_ARGS__)
-
-#define E_assert(expr, msg, ...) \
- do { \
- if (!(expr)) \
- ksft_exit_fail_msg("ASSERT(%s:%d) failed (%s): " msg "\n", \
- __FILE__, __LINE__, #expr, ##__VA_ARGS__); \
- } while (0)
-
-int raw_openat2(int dfd, const char *path, void *how, size_t size);
-int sys_openat2(int dfd, const char *path, struct open_how *how);
-int sys_openat(int dfd, const char *path, struct open_how *how);
-int sys_renameat2(int olddirfd, const char *oldpath,
- int newdirfd, const char *newpath, unsigned int flags);
-
-int touchat(int dfd, const char *path);
-char *fdreadlink(int fd);
-bool fdequal(int fd, int dfd, const char *path);
-
-extern bool openat2_supported;
-
-#endif /* __RESOLVEAT_H__ */
diff --git a/tools/testing/selftests/openat2/rename_attack_test.c b/tools/testing/selftests/openat2/rename_attack_test.c
deleted file mode 100644
index aa5699e..0000000
--- a/tools/testing/selftests/openat2/rename_attack_test.c
+++ /dev/null
@@ -1,160 +0,0 @@
-// SPDX-License-Identifier: GPL-2.0-or-later
-/*
- * Author: Aleksa Sarai <cyphar@cyphar.com>
- * Copyright (C) 2018-2019 SUSE LLC.
- */
-
-#define _GNU_SOURCE
-#include <errno.h>
-#include <fcntl.h>
-#include <sched.h>
-#include <sys/stat.h>
-#include <sys/types.h>
-#include <sys/mount.h>
-#include <sys/mman.h>
-#include <sys/prctl.h>
-#include <signal.h>
-#include <stdio.h>
-#include <stdlib.h>
-#include <stdbool.h>
-#include <string.h>
-#include <syscall.h>
-#include <limits.h>
-#include <unistd.h>
-
-#include "kselftest.h"
-#include "helpers.h"
-
-/* Construct a test directory with the following structure:
- *
- * root/
- * |-- a/
- * | `-- c/
- * `-- b/
- */
-int setup_testdir(void)
-{
- int dfd;
- char dirname[] = "/tmp/ksft-openat2-rename-attack.XXXXXX";
-
- /* Make the top-level directory. */
- if (!mkdtemp(dirname))
- ksft_exit_fail_msg("setup_testdir: failed to create tmpdir\n");
- dfd = open(dirname, O_PATH | O_DIRECTORY);
- if (dfd < 0)
- ksft_exit_fail_msg("setup_testdir: failed to open tmpdir\n");
-
- E_mkdirat(dfd, "a", 0755);
- E_mkdirat(dfd, "b", 0755);
- E_mkdirat(dfd, "a/c", 0755);
-
- return dfd;
-}
-
-/* Swap @dirfd/@a and @dirfd/@b constantly. Parent must kill this process. */
-pid_t spawn_attack(int dirfd, char *a, char *b)
-{
- pid_t child = fork();
- if (child != 0)
- return child;
-
- /* If the parent (the test process) dies, kill ourselves too. */
- E_prctl(PR_SET_PDEATHSIG, SIGKILL);
-
- /* Swap @a and @b. */
- for (;;)
- renameat2(dirfd, a, dirfd, b, RENAME_EXCHANGE);
- exit(1);
-}
-
-#define NUM_RENAME_TESTS 2
-#define ROUNDS 400000
-
-const char *flagname(int resolve)
-{
- switch (resolve) {
- case RESOLVE_IN_ROOT:
- return "RESOLVE_IN_ROOT";
- case RESOLVE_BENEATH:
- return "RESOLVE_BENEATH";
- }
- return "(unknown)";
-}
-
-void test_rename_attack(int resolve)
-{
- int dfd, afd;
- pid_t child;
- void (*resultfn)(const char *msg, ...) = ksft_test_result_pass;
- int escapes = 0, other_errs = 0, exdevs = 0, eagains = 0, successes = 0;
-
- struct open_how how = {
- .flags = O_PATH,
- .resolve = resolve,
- };
-
- if (!openat2_supported) {
- how.resolve = 0;
- ksft_print_msg("openat2(2) unsupported -- using openat(2) instead\n");
- }
-
- dfd = setup_testdir();
- afd = openat(dfd, "a", O_PATH);
- if (afd < 0)
- ksft_exit_fail_msg("test_rename_attack: failed to open 'a'\n");
-
- child = spawn_attack(dfd, "a/c", "b");
-
- for (int i = 0; i < ROUNDS; i++) {
- int fd;
- char *victim_path = "c/../../c/../../c/../../c/../../c/../../c/../../c/../../c/../../c/../../c/../../c/../../c/../../c/../../c/../../c/../../c/../../c/../../c/../../c/../..";
-
- if (openat2_supported)
- fd = sys_openat2(afd, victim_path, &how);
- else
- fd = sys_openat(afd, victim_path, &how);
-
- if (fd < 0) {
- if (fd == -EAGAIN)
- eagains++;
- else if (fd == -EXDEV)
- exdevs++;
- else if (fd == -ENOENT)
- escapes++; /* escaped outside and got ENOENT... */
- else
- other_errs++; /* unexpected error */
- } else {
- if (fdequal(fd, afd, NULL))
- successes++;
- else
- escapes++; /* we got an unexpected fd */
- }
- close(fd);
- }
-
- if (escapes > 0)
- resultfn = ksft_test_result_fail;
- ksft_print_msg("non-escapes: EAGAIN=%d EXDEV=%d E<other>=%d success=%d\n",
- eagains, exdevs, other_errs, successes);
- resultfn("rename attack with %s (%d runs, got %d escapes)\n",
- flagname(resolve), ROUNDS, escapes);
-
- /* Should be killed anyway, but might as well make sure. */
- E_kill(child, SIGKILL);
-}
-
-#define NUM_TESTS NUM_RENAME_TESTS
-
-int main(int argc, char **argv)
-{
- ksft_print_header();
- ksft_set_plan(NUM_TESTS);
-
- test_rename_attack(RESOLVE_BENEATH);
- test_rename_attack(RESOLVE_IN_ROOT);
-
- if (ksft_get_fail_cnt() + ksft_get_error_cnt() > 0)
- ksft_exit_fail();
- else
- ksft_exit_pass();
-}
diff --git a/tools/testing/selftests/pid_namespace/pid_max.c b/tools/testing/selftests/pid_namespace/pid_max.c
index c9519e7..5d686a0 100644
--- a/tools/testing/selftests/pid_namespace/pid_max.c
+++ b/tools/testing/selftests/pid_namespace/pid_max.c
@@ -12,10 +12,74 @@
#include <syscall.h>
#include <sys/mount.h>
#include <sys/wait.h>
+#include <unistd.h>
#include "kselftest_harness.h"
#include "../pidfd/pidfd.h"
+/*
+ * The kernel computes the minimum allowed pid_max as:
+ * max(RESERVED_PIDS + 1, PIDS_PER_CPU_MIN * num_possible_cpus())
+ * Mirror that here so the test values are always valid.
+ *
+ * Note: glibc's get_nprocs_conf() returns the number of *configured*
+ * (present) CPUs, not *possible* CPUs. The kernel uses
+ * num_possible_cpus() which corresponds to /sys/devices/system/cpu/possible.
+ * These can differ significantly (e.g. 16 configured vs 128 possible).
+ */
+#define RESERVED_PIDS 300
+#define PIDS_PER_CPU_MIN 8
+
+/* Count CPUs from a range list like "0-31" or "0-15,32-47". */
+static int num_possible_cpus(void)
+{
+ FILE *f;
+ int count = 0;
+ int lo, hi;
+
+ f = fopen("/sys/devices/system/cpu/possible", "r");
+ if (!f)
+ return 0;
+
+ while (fscanf(f, "%d", &lo) == 1) {
+ if (fscanf(f, "-%d", &hi) == 1)
+ count += hi - lo + 1;
+ else
+ count++;
+ /* skip comma separator */
+ fscanf(f, ",");
+ }
+
+ fclose(f);
+ return count;
+}
+
+static int pid_min(void)
+{
+ int cpu_min = PIDS_PER_CPU_MIN * num_possible_cpus();
+
+ return cpu_min > (RESERVED_PIDS + 1) ? cpu_min : (RESERVED_PIDS + 1);
+}
+
+/*
+ * Outer and inner pid_max limits used by the tests. The outer limit is
+ * the more restrictive ancestor; the inner limit is set higher in a
+ * nested namespace but must still be capped by the outer limit.
+ * Both are derived from the kernel's minimum so they are always writable.
+ *
+ * Global so that clone callbacks can access them without parameter plumbing.
+ */
+static int outer_limit;
+static int inner_limit;
+
+static int write_int_to_fd(int fd, int val)
+{
+ char buf[12];
+ int len = snprintf(buf, sizeof(buf), "%d", val);
+
+ return write(fd, buf, len);
+}
+
#define __STACK_SIZE (8 * 1024 * 1024)
static pid_t do_clone(int (*fn)(void *), void *arg, int flags)
{
@@ -60,18 +124,18 @@ static int pid_max_cb(void *data)
return -1;
}
- ret = write(fd, "500", sizeof("500") - 1);
+ ret = write_int_to_fd(fd, inner_limit);
if (ret < 0) {
fprintf(stderr, "%m - Failed to write pid_max\n");
return -1;
}
- for (int i = 0; i < 501; i++) {
+ for (int i = 0; i < inner_limit + 1; i++) {
pid = fork();
if (pid == 0)
exit(EXIT_SUCCESS);
wait_for_pid(pid);
- if (pid > 500) {
+ if (pid > inner_limit) {
fprintf(stderr, "Managed to create pid number beyond limit\n");
return -1;
}
@@ -106,7 +170,7 @@ static int pid_max_nested_inner(void *data)
return fret;
}
- ret = write(fd, "500", sizeof("500") - 1);
+ ret = write_int_to_fd(fd, inner_limit);
close(fd);
if (ret < 0) {
fprintf(stderr, "%m - Failed to write pid_max\n");
@@ -133,8 +197,8 @@ static int pid_max_nested_inner(void *data)
return fret;
}
- /* Now make sure that we wrap pids at 400. */
- for (i = 0; i < 510; i++) {
+ /* Now make sure that we wrap pids at outer_limit. */
+ for (i = 0; i < inner_limit + 10; i++) {
pid_t pid;
pid = fork();
@@ -145,7 +209,7 @@ static int pid_max_nested_inner(void *data)
exit(EXIT_SUCCESS);
wait_for_pid(pid);
- if (pid >= 500) {
+ if (pid >= inner_limit) {
fprintf(stderr, "Managed to create process with pid %d beyond configured limit\n", pid);
return fret;
}
@@ -156,15 +220,19 @@ static int pid_max_nested_inner(void *data)
static int pid_max_nested_outer(void *data)
{
- int fret = -1, nr_procs = 400;
- pid_t pids[1000];
- int fd, i, ret;
+ int fret = -1, nr_procs = 0;
+ pid_t *pids;
+ int fd, ret;
pid_t pid;
+ pids = malloc(outer_limit * sizeof(pid_t));
+ if (!pids)
+ return -1;
+
ret = mount("", "/", NULL, MS_PRIVATE | MS_REC, 0);
if (ret) {
fprintf(stderr, "%m - Failed to make rootfs private mount\n");
- return fret;
+ goto out;
}
umount2("/proc", MNT_DETACH);
@@ -172,27 +240,28 @@ static int pid_max_nested_outer(void *data)
ret = mount("proc", "/proc", "proc", 0, NULL);
if (ret) {
fprintf(stderr, "%m - Failed to mount proc\n");
- return fret;
+ goto out;
}
fd = open("/proc/sys/kernel/pid_max", O_RDWR | O_CLOEXEC | O_NOCTTY);
if (fd < 0) {
fprintf(stderr, "%m - Failed to open pid_max\n");
- return fret;
+ goto out;
}
- ret = write(fd, "400", sizeof("400") - 1);
+ ret = write_int_to_fd(fd, outer_limit);
close(fd);
if (ret < 0) {
fprintf(stderr, "%m - Failed to write pid_max\n");
- return fret;
+ goto out;
}
/*
- * Create 397 processes. This leaves room for do_clone() (398) and
- * one more 399. So creating another process needs to fail.
+ * Create (outer_limit - 4) processes. This leaves room for
+ * do_clone() and one more. So creating another process needs
+ * to fail.
*/
- for (nr_procs = 0; nr_procs < 396; nr_procs++) {
+ for (nr_procs = 0; nr_procs < outer_limit - 4; nr_procs++) {
pid = fork();
if (pid < 0)
goto reap;
@@ -220,20 +289,26 @@ static int pid_max_nested_outer(void *data)
for (int i = 0; i < nr_procs; i++)
wait_for_pid(pids[i]);
+out:
+ free(pids);
return fret;
}
static int pid_max_nested_limit_inner(void *data)
{
- int fret = -1, nr_procs = 400;
+ int fret = -1, nr_procs = 0;
int fd, ret;
pid_t pid;
- pid_t pids[1000];
+ pid_t *pids;
+
+ pids = malloc(inner_limit * sizeof(pid_t));
+ if (!pids)
+ return -1;
ret = mount("", "/", NULL, MS_PRIVATE | MS_REC, 0);
if (ret) {
fprintf(stderr, "%m - Failed to make rootfs private mount\n");
- return fret;
+ goto out;
}
umount2("/proc", MNT_DETACH);
@@ -241,23 +316,23 @@ static int pid_max_nested_limit_inner(void *data)
ret = mount("proc", "/proc", "proc", 0, NULL);
if (ret) {
fprintf(stderr, "%m - Failed to mount proc\n");
- return fret;
+ goto out;
}
fd = open("/proc/sys/kernel/pid_max", O_RDWR | O_CLOEXEC | O_NOCTTY);
if (fd < 0) {
fprintf(stderr, "%m - Failed to open pid_max\n");
- return fret;
+ goto out;
}
- ret = write(fd, "500", sizeof("500") - 1);
+ ret = write_int_to_fd(fd, inner_limit);
close(fd);
if (ret < 0) {
fprintf(stderr, "%m - Failed to write pid_max\n");
- return fret;
+ goto out;
}
- for (nr_procs = 0; nr_procs < 500; nr_procs++) {
+ for (nr_procs = 0; nr_procs < inner_limit; nr_procs++) {
pid = fork();
if (pid < 0)
break;
@@ -268,7 +343,7 @@ static int pid_max_nested_limit_inner(void *data)
pids[nr_procs] = pid;
}
- if (nr_procs >= 400) {
+ if (nr_procs >= outer_limit) {
fprintf(stderr, "Managed to create processes beyond the configured outer limit\n");
goto reap;
}
@@ -279,6 +354,8 @@ static int pid_max_nested_limit_inner(void *data)
for (int i = 0; i < nr_procs; i++)
wait_for_pid(pids[i]);
+out:
+ free(pids);
return fret;
}
@@ -307,7 +384,7 @@ static int pid_max_nested_limit_outer(void *data)
return -1;
}
- ret = write(fd, "400", sizeof("400") - 1);
+ ret = write_int_to_fd(fd, outer_limit);
close(fd);
if (ret < 0) {
fprintf(stderr, "%m - Failed to write pid_max\n");
@@ -328,17 +405,32 @@ static int pid_max_nested_limit_outer(void *data)
return 0;
}
-TEST(pid_max_simple)
+FIXTURE(pid_max) {
+ int dummy;
+};
+
+FIXTURE_SETUP(pid_max)
+{
+ int min = pid_min();
+
+ outer_limit = min + 100;
+ inner_limit = min + 200;
+}
+
+FIXTURE_TEARDOWN(pid_max)
+{
+}
+
+TEST_F(pid_max, simple)
{
pid_t pid;
-
pid = do_clone(pid_max_cb, NULL, CLONE_NEWPID | CLONE_NEWNS);
ASSERT_GT(pid, 0);
ASSERT_EQ(0, wait_for_pid(pid));
}
-TEST(pid_max_nested_limit)
+TEST_F(pid_max, nested_limit)
{
pid_t pid;
@@ -347,7 +439,7 @@ TEST(pid_max_nested_limit)
ASSERT_EQ(0, wait_for_pid(pid));
}
-TEST(pid_max_nested)
+TEST_F(pid_max, nested)
{
pid_t pid;
diff --git a/tools/testing/selftests/pipe/.gitignore b/tools/testing/selftests/pipe/.gitignore
new file mode 100644
index 0000000..20b5493
--- /dev/null
+++ b/tools/testing/selftests/pipe/.gitignore
@@ -0,0 +1 @@
+pipe_bench
diff --git a/tools/testing/selftests/pipe/Makefile b/tools/testing/selftests/pipe/Makefile
new file mode 100644
index 0000000..1810c68
--- /dev/null
+++ b/tools/testing/selftests/pipe/Makefile
@@ -0,0 +1,9 @@
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (c) 2026 Meta Platforms, Inc. and affiliates
+# Copyright (c) 2026 Breno Leitao <leitao@debian.org>
+
+CFLAGS += -O2 -Wall -Wextra -pthread
+
+TEST_GEN_PROGS := pipe_bench
+
+include ../lib.mk
diff --git a/tools/testing/selftests/pipe/pipe_bench.c b/tools/testing/selftests/pipe/pipe_bench.c
new file mode 100644
index 0000000..7e96429
--- /dev/null
+++ b/tools/testing/selftests/pipe/pipe_bench.c
@@ -0,0 +1,616 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * pipe_bench - exercise concurrent pipe operation
+ *
+ * N writer threads hammer a single pipe with multi-page writes; M reader
+ * threads drain it. Each writer records its own write() latency histogram.
+ * Multi-page writes (msgsize >= PAGE_SIZE) force the loop in
+ * anon_pipe_write() to call alloc_page(GFP_HIGHUSER | __GFP_ACCOUNT) under
+ * pipe->mutex, which is the critical section the patch shrinks.
+ *
+ * By default the benchmark sweeps writers in {1, 2, 5} x readers in
+ * {1, 5, 10} and prints one block per configuration so two runs (e.g.
+ * baseline vs patched) can be diffed directly. Pass -w and -r to run a
+ * single configuration instead. Pass --memory-pressure to spawn stress-ng
+ * alongside the sweep so the per-page alloc_page() path under pipe->mutex
+ * has to dip into reclaim.
+ *
+ * Copyright (c) 2026 Meta Platforms, Inc. and affiliates
+ * Copyright (c) 2026 Breno Leitao <leitao@debian.org>
+ */
+
+#define _GNU_SOURCE
+#include <errno.h>
+#include <fcntl.h>
+#include <getopt.h>
+#include <poll.h>
+#include <pthread.h>
+#include <signal.h>
+#include <stdatomic.h>
+#include <stdint.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <sys/wait.h>
+#include <time.h>
+#include <unistd.h>
+
+#define ARRAY_SIZE(a) (sizeof(a) / sizeof((a)[0]))
+#define HIST_BUCKETS 32
+
+static size_t g_msgsize = 16 * 4096;
+static int g_duration = 3;
+static int g_pipe_size = 1024 * 1024;
+static int g_memory_pressure;
+
+static atomic_int g_stop;
+static int g_pipe[2];
+
+struct wstats {
+ uint64_t writes;
+ uint64_t bytes;
+ uint64_t lat_sum_ns;
+ uint64_t lat_max_ns;
+ uint64_t lat_hist[HIST_BUCKETS];
+ char *buf;
+};
+
+struct rstats {
+ char *buf;
+};
+
+struct hist_totals {
+ uint64_t writes;
+ uint64_t bytes;
+ uint64_t lat_sum;
+ uint64_t lat_max;
+};
+
+static inline uint64_t now_ns(void)
+{
+ struct timespec ts;
+
+ clock_gettime(CLOCK_MONOTONIC, &ts);
+ return (uint64_t)ts.tv_sec * 1000000000ull + (uint64_t)ts.tv_nsec;
+}
+
+static inline int log2_bucket(uint64_t v)
+{
+ int b = 0;
+
+ if (!v)
+ return 0;
+ while (v >>= 1)
+ b++;
+ return b < HIST_BUCKETS ? b : HIST_BUCKETS - 1;
+}
+
+static void *writer(void *arg)
+{
+ struct wstats *s = arg;
+
+ while (!atomic_load_explicit(&g_stop, memory_order_relaxed)) {
+ uint64_t t0 = now_ns();
+ ssize_t n = write(g_pipe[1], s->buf, g_msgsize);
+ uint64_t dt = now_ns() - t0;
+
+ if (n > 0) {
+ s->writes++;
+ s->bytes += (uint64_t)n;
+ s->lat_sum_ns += dt;
+ if (dt > s->lat_max_ns)
+ s->lat_max_ns = dt;
+ s->lat_hist[log2_bucket(dt)]++;
+ } else if (n < 0 && (errno == EPIPE || errno == EBADF)) {
+ break;
+ }
+ }
+ return NULL;
+}
+
+static void *reader(void *arg)
+{
+ struct rstats *s = arg;
+
+ /*
+ * Drain until EOF (write end closed by main). g_stop is not checked
+ * here on purpose: writers may be blocked in write() with the pipe
+ * full when g_stop is set, so the reader must keep draining until
+ * main closes the write end.
+ */
+ for (;;) {
+ ssize_t n = read(g_pipe[0], s->buf, g_msgsize);
+
+ if (n <= 0)
+ break;
+ }
+ return NULL;
+}
+
+/* Sum per-writer stats and per-bucket counts into the caller's aggregates. */
+static void aggregate_wstats(struct wstats *all, int nw,
+ uint64_t agg[HIST_BUCKETS],
+ struct hist_totals *t)
+{
+ memset(t, 0, sizeof(*t));
+ for (int i = 0; i < nw; i++) {
+ t->writes += all[i].writes;
+ t->bytes += all[i].bytes;
+ t->lat_sum += all[i].lat_sum_ns;
+ if (all[i].lat_max_ns > t->lat_max)
+ t->lat_max = all[i].lat_max_ns;
+ for (int b = 0; b < HIST_BUCKETS; b++)
+ agg[b] += all[i].lat_hist[b];
+ }
+}
+
+/*
+ * Walk @agg in order, returning the inclusive upper bound (in ns) of the
+ * log2 bucket where the running sum first reaches @target.
+ *
+ * A percentile is undefined with zero samples, and with very low sample
+ * counts integer truncation could make @target zero -- then "cum >= 0"
+ * would latch on the first (possibly empty) bucket. Callers must pass
+ * @target >= 1.
+ */
+static uint64_t bucket_at(const uint64_t agg[HIST_BUCKETS], uint64_t target)
+{
+ uint64_t cum = 0;
+
+ for (int b = 0; b < HIST_BUCKETS; b++) {
+ /* HIST_BUCKETS <= 63, so (b + 1) is always a safe shift. */
+ uint64_t upper = (1ULL << (b + 1)) - 1;
+
+ cum += agg[b];
+ if (cum >= target)
+ return upper;
+ }
+ return 0;
+}
+
+static void compute_p50_p99(const uint64_t agg[HIST_BUCKETS], uint64_t writes,
+ uint64_t *p50, uint64_t *p99)
+{
+ uint64_t p50_target, p99_target;
+
+ *p50 = *p99 = 0;
+ if (!writes)
+ return;
+
+ p50_target = writes * 50 / 100;
+ p99_target = writes * 99 / 100;
+ if (!p50_target)
+ p50_target = 1;
+ if (!p99_target)
+ p99_target = 1;
+
+ *p50 = bucket_at(agg, p50_target);
+ *p99 = bucket_at(agg, p99_target);
+}
+
+static void print_summary(int nw, int nr, const struct hist_totals *t,
+ uint64_t p50, uint64_t p99)
+{
+ double sec = g_duration;
+ uint64_t avg_ns = t->writes ? t->lat_sum / t->writes : 0;
+
+ printf("config: writers=%d readers=%d msgsize=%zu duration=%d pipe_size=%d memory_pressure=%s\n",
+ nw, nr, g_msgsize, g_duration, g_pipe_size,
+ g_memory_pressure ? "yes" : "no");
+ printf("writes: total=%llu rate=%.0f/s\n",
+ (unsigned long long)t->writes, (double)t->writes / sec);
+ printf("throughput_MBps: %.2f\n",
+ ((double)t->bytes / sec) / (1024.0 * 1024.0));
+ printf("lat_avg_ns: %llu\n", (unsigned long long)avg_ns);
+ printf("lat_p50_ns_upper: %llu\n", (unsigned long long)p50);
+ printf("lat_p99_ns_upper: %llu\n", (unsigned long long)p99);
+ printf("lat_max_ns: %llu\n", (unsigned long long)t->lat_max);
+}
+
+static void summarize(struct wstats *all, int nw, int nr)
+{
+ uint64_t agg[HIST_BUCKETS] = {0};
+ struct hist_totals t;
+ uint64_t p50, p99;
+
+ aggregate_wstats(all, nw, agg, &t);
+ compute_p50_p99(agg, t.writes, &p50, &p99);
+ print_summary(nw, nr, &t, p50, p99);
+}
+
+/*
+ * Child branch of fork(): restore SIGPIPE to default (parent ignores it),
+ * exec stress-ng, and on failure write the reason into @hs_wr before
+ * exiting. The parent observes EOF on hs_wr (closed via O_CLOEXEC) when
+ * exec succeeds.
+ */
+static void stress_ng_child(int hs_wr) __attribute__((noreturn));
+static void stress_ng_child(int hs_wr)
+{
+ char errbuf[256];
+
+ signal(SIGPIPE, SIG_DFL);
+ execlp("stress-ng", "stress-ng",
+ "--vm", "4", "--vm-bytes", "80%",
+ "--vm-method", "all",
+ (char *)NULL);
+ snprintf(errbuf, sizeof(errbuf),
+ "exec stress-ng failed: %s\n", strerror(errno));
+ (void)!write(hs_wr, errbuf, strlen(errbuf));
+ _exit(127);
+}
+
+/*
+ * Read from the O_CLOEXEC handshake pipe. Anything readable means the
+ * child wrote an error before exec; EOF (n == 0) means the write-end
+ * closed because exec succeeded. Returns 0 on exec success, -1 if the
+ * child failed and was reaped.
+ */
+static int stress_ng_wait_handshake(int hs_rd, pid_t pid)
+{
+ struct pollfd pfd = { .fd = hs_rd, .events = POLLIN };
+ char errbuf[256];
+ int status;
+ int ret;
+
+ ret = poll(&pfd, 1, 500);
+ if (ret <= 0)
+ return 0;
+
+ ssize_t n = read(hs_rd, errbuf, sizeof(errbuf) - 1);
+
+ if (n > 0) {
+ errbuf[n] = '\0';
+ fputs(errbuf, stderr);
+ waitpid(pid, &status, 0);
+ return -1;
+ }
+ return 0;
+}
+
+static pid_t spawn_stress_ng(void)
+{
+ int hs[2];
+ pid_t pid;
+
+ /*
+ * Handshake pipe: child writes one byte and _exit()s on exec
+ * failure. On exec success the O_CLOEXEC flag closes the write
+ * end, which the parent observes as EOF. This makes the "is
+ * stress-ng on $PATH?" check fail fast rather than silently.
+ */
+ if (pipe2(hs, O_CLOEXEC) < 0) {
+ perror("pipe2");
+ return -1;
+ }
+
+ pid = fork();
+ if (pid < 0) {
+ perror("fork");
+ close(hs[0]);
+ close(hs[1]);
+ return -1;
+ }
+ if (pid == 0) {
+ close(hs[0]);
+ stress_ng_child(hs[1]);
+ }
+
+ close(hs[1]);
+ if (stress_ng_wait_handshake(hs[0], pid) < 0) {
+ close(hs[0]);
+ return -1;
+ }
+ close(hs[0]);
+
+ /* Give stress-ng a moment to map its VM regions before measuring. */
+ sleep(1);
+ return pid;
+}
+
+static void kill_stress_ng(pid_t pid)
+{
+ int status;
+
+ if (pid <= 0)
+ return;
+ kill(pid, SIGTERM);
+ for (int i = 0; i < 20; i++) {
+ if (waitpid(pid, &status, WNOHANG) > 0)
+ return;
+ usleep(100 * 1000);
+ }
+ kill(pid, SIGKILL);
+ waitpid(pid, &status, 0);
+}
+
+/*
+ * Allocate per-thread page-aligned buffers in main so a failed
+ * aligned_alloc() aborts the run before any thread starts. Workers used
+ * to allocate their own buffer and return NULL on failure, which left
+ * peers blocked in write()/read() with nobody to unblock them.
+ */
+static int alloc_thread_bufs(struct wstats *ws, int nw,
+ struct rstats *rs, int nr)
+{
+ for (int i = 0; i < nw; i++) {
+ ws[i].buf = aligned_alloc(4096, g_msgsize);
+ if (!ws[i].buf) {
+ fprintf(stderr, "writer %d: aligned_alloc(%zu) failed\n",
+ i, g_msgsize);
+ return -1;
+ }
+ memset(ws[i].buf, 0xAA, g_msgsize);
+ }
+ for (int i = 0; i < nr; i++) {
+ rs[i].buf = aligned_alloc(4096, g_msgsize);
+ if (!rs[i].buf) {
+ fprintf(stderr, "reader %d: aligned_alloc(%zu) failed\n",
+ i, g_msgsize);
+ return -1;
+ }
+ }
+ return 0;
+}
+
+static void free_thread_bufs(struct wstats *ws, int nw,
+ struct rstats *rs, int nr)
+{
+ if (ws)
+ for (int i = 0; i < nw; i++)
+ free(ws[i].buf);
+ if (rs)
+ for (int i = 0; i < nr; i++)
+ free(rs[i].buf);
+}
+
+static int start_readers(pthread_t *rt, struct rstats *rs, int nr,
+ int *created)
+{
+ for (int i = 0; i < nr; i++) {
+ int err = pthread_create(&rt[i], NULL, reader, &rs[i]);
+
+ if (err) {
+ fprintf(stderr, "pthread_create reader %d: %s\n",
+ i, strerror(err));
+ return -1;
+ }
+ (*created)++;
+ }
+ return 0;
+}
+
+static int start_writers(pthread_t *wt, struct wstats *ws, int nw,
+ int *created)
+{
+ for (int i = 0; i < nw; i++) {
+ int err = pthread_create(&wt[i], NULL, writer, &ws[i]);
+
+ if (err) {
+ fprintf(stderr, "pthread_create writer %d: %s\n",
+ i, strerror(err));
+ return -1;
+ }
+ (*created)++;
+ }
+ return 0;
+}
+
+static int open_bench_pipe(void)
+{
+ if (pipe(g_pipe) < 0) {
+ perror("pipe");
+ return -1;
+ }
+ if (fcntl(g_pipe[1], F_SETPIPE_SZ, g_pipe_size) < 0)
+ perror("F_SETPIPE_SZ (continuing)");
+ return 0;
+}
+
+/*
+ * Normal termination: g_stop tells writers to leave the loop after the
+ * current write() returns. Closing the shared write-end fd means once
+ * the in-flight writes drain, readers see EOF and exit. Writers are not
+ * unblocked by EPIPE here -- g_pipe[0] stays open so readers can keep
+ * draining.
+ *
+ * Error path: some threads may have been created and others skipped, so
+ * writers could be blocked in write() with no reader making progress.
+ * Close both ends -- closing the read end is what delivers EPIPE to a
+ * blocked writer.
+ */
+static void stop_and_join(pthread_t *wt, int nw_created,
+ pthread_t *rt, int nr_created, int rc)
+{
+ atomic_store(&g_stop, 1);
+ close(g_pipe[1]);
+ if (rc < 0)
+ close(g_pipe[0]);
+ for (int i = 0; i < nw_created; i++)
+ pthread_join(wt[i], NULL);
+ for (int i = 0; i < nr_created; i++)
+ pthread_join(rt[i], NULL);
+ if (rc == 0)
+ close(g_pipe[0]);
+}
+
+static int run_one(int nw, int nr)
+{
+ pthread_t *wt = NULL, *rt = NULL;
+ struct wstats *ws = NULL;
+ struct rstats *rs = NULL;
+ int nw_created = 0, nr_created = 0;
+ int rc = 0;
+
+ atomic_store(&g_stop, 0);
+
+ if (open_bench_pipe() < 0)
+ return -1;
+
+ wt = calloc((size_t)nw, sizeof(*wt));
+ rt = calloc((size_t)nr, sizeof(*rt));
+ ws = calloc((size_t)nw, sizeof(*ws));
+ rs = calloc((size_t)nr, sizeof(*rs));
+ if (!wt || !rt || !ws || !rs) {
+ fprintf(stderr, "alloc failed\n");
+ rc = -1;
+ goto teardown;
+ }
+
+ if (alloc_thread_bufs(ws, nw, rs, nr) < 0) {
+ rc = -1;
+ goto teardown;
+ }
+
+ if (start_readers(rt, rs, nr, &nr_created) < 0 ||
+ start_writers(wt, ws, nw, &nw_created) < 0) {
+ rc = -1;
+ goto teardown;
+ }
+
+ sleep((unsigned int)g_duration);
+
+teardown:
+ stop_and_join(wt, nw_created, rt, nr_created, rc);
+
+ if (rc == 0) {
+ summarize(ws, nw, nr);
+ fflush(stdout);
+ }
+
+ free_thread_bufs(ws, nw, rs, nr);
+ free(wt);
+ free(rt);
+ free(ws);
+ free(rs);
+ return rc;
+}
+
+static void usage(const char *prog)
+{
+ fprintf(stderr,
+ "usage: %s [-w writers] [-r readers] [-s msgsize] [-d secs] [-p pipe_size] [--memory-pressure]\n"
+ " default: sweep writers={1,2,5} x readers={1,5,10}\n"
+ " --memory-pressure: spawn stress-ng (--vm 4 --vm-bytes 80%% --vm-method all) for the run\n",
+ prog);
+}
+
+static int parse_args(int argc, char **argv,
+ int *writers_override, int *readers_override)
+{
+ static const struct option long_opts[] = {
+ {"memory-pressure", no_argument, NULL, 'M'},
+ {0, 0, 0, 0},
+ };
+ int opt;
+
+ while ((opt = getopt_long(argc, argv, "w:r:s:d:p:",
+ long_opts, NULL)) != -1) {
+ switch (opt) {
+ case 'w':
+ *writers_override = atoi(optarg);
+ break;
+ case 'r':
+ *readers_override = atoi(optarg);
+ break;
+ case 's':
+ g_msgsize = (size_t)atol(optarg);
+ break;
+ case 'd':
+ g_duration = atoi(optarg);
+ break;
+ case 'p':
+ g_pipe_size = atoi(optarg);
+ break;
+ case 'M':
+ g_memory_pressure = 1;
+ break;
+ default:
+ usage(argv[0]);
+ return -1;
+ }
+ }
+ return 0;
+}
+
+/*
+ * aligned_alloc(4096, size) requires size to be a multiple of the
+ * alignment (C11); glibc returns NULL otherwise, which would make
+ * writer/reader threads silently exit and the run report zero writes.
+ * Validate up front instead.
+ */
+static int validate_args(void)
+{
+ if (g_msgsize == 0 || g_msgsize % 4096 != 0) {
+ fprintf(stderr,
+ "msgsize must be a positive multiple of 4096 (got %zu)\n",
+ g_msgsize);
+ return -1;
+ }
+ if (g_duration <= 0) {
+ fprintf(stderr, "duration must be > 0 seconds (got %d)\n",
+ g_duration);
+ return -1;
+ }
+ if (g_pipe_size <= 0) {
+ fprintf(stderr, "pipe_size must be > 0 bytes (got %d)\n",
+ g_pipe_size);
+ return -1;
+ }
+ return 0;
+}
+
+static int run_sweep(void)
+{
+ static const int writers_sweep[] = {1, 2, 5};
+ static const int readers_sweep[] = {1, 5, 10};
+
+ for (size_t i = 0; i < ARRAY_SIZE(writers_sweep); i++) {
+ for (size_t j = 0; j < ARRAY_SIZE(readers_sweep); j++) {
+ printf("---\n");
+ if (run_one(writers_sweep[i], readers_sweep[j]) < 0)
+ return -1;
+ }
+ }
+ return 0;
+}
+
+int main(int argc, char **argv)
+{
+ int writers_override = 0, readers_override = 0;
+ pid_t stress_pid = -1;
+ int rc = 0;
+
+ if (parse_args(argc, argv, &writers_override, &readers_override) < 0)
+ return 1;
+ if (validate_args() < 0)
+ return 1;
+
+ signal(SIGPIPE, SIG_IGN);
+ setvbuf(stdout, NULL, _IOLBF, 0);
+ setvbuf(stderr, NULL, _IOLBF, 0);
+
+ fprintf(stderr, "pid=%d\n", getpid());
+ fflush(stderr);
+
+ if (g_memory_pressure) {
+ stress_pid = spawn_stress_ng();
+ if (stress_pid < 0) {
+ fprintf(stderr,
+ "memory_pressure requested but stress-ng could not be spawned\n");
+ return 1;
+ }
+ }
+
+ if (writers_override > 0 || readers_override > 0) {
+ int nw = writers_override > 0 ? writers_override : 1;
+ int nr = readers_override > 0 ? readers_override : 1;
+
+ rc = run_one(nw, nr) < 0 ? 1 : 0;
+ } else {
+ rc = run_sweep() < 0 ? 1 : 0;
+ }
+
+ kill_stress_ng(stress_pid);
+ return rc;
+}
diff --git a/tools/testing/selftests/rcutorture/bin/kvm-series.sh b/tools/testing/selftests/rcutorture/bin/kvm-series.sh
index c4ee5f9..be94125 100755
--- a/tools/testing/selftests/rcutorture/bin/kvm-series.sh
+++ b/tools/testing/selftests/rcutorture/bin/kvm-series.sh
@@ -1,12 +1,13 @@
#!/bin/bash
# SPDX-License-Identifier: GPL-2.0+
#
-# Usage: kvm-series.sh config-list commit-id-list [ kvm.sh parameters ]
+# Usage: kvm-series.sh config-list commit-id-range [ kvm.sh parameters ]
#
-# Tests the specified list of unadorned configs ("TREE01 SRCU-P" but not
-# "CFLIST" or "3*TRACE01") and an indication of a set of commits to test,
-# then runs each commit through the specified list of commits using kvm.sh.
-# The runs are grouped into a -series/config/commit directory tree.
+# Tests the specified list of unadorned configs ("TREE01 SRCU-P" but
+# not "CFLIST" or "3*TRACE01") and an indication of a range of commits
+# ("v7.0-rc1..rcu/dev", but not "cd0ce7bab0408 ff74db28df623 17c52d7b31a1f")
+# to test, then runs each commit through the specified list of commits using
+# kvm.sh. The runs are grouped into a -series/config/commit directory tree.
# Each run defaults to a duration of one minute.
#
# Run in top-level Linux source directory. Please note that this is in
diff --git a/tools/testing/selftests/rcutorture/bin/torture.sh b/tools/testing/selftests/rcutorture/bin/torture.sh
index a33ba10..f008389 100755
--- a/tools/testing/selftests/rcutorture/bin/torture.sh
+++ b/tools/testing/selftests/rcutorture/bin/torture.sh
@@ -184,7 +184,7 @@
do_clocksourcewd=no
do_srcu_lockdep=no
;;
- --do-normal|--do-no-normal|--no-normal)
+ --do-normal|--do-norm|--do-no-normal|--do-no-norm|--no-normal|--no-norm)
do_normal=`doyesno "$1" --do-normal`
explicit_normal=yes
;;
diff --git a/virt/kvm/guest_memfd.c b/virt/kvm/guest_memfd.c
index 69c9d6d..80f20103 100644
--- a/virt/kvm/guest_memfd.c
+++ b/virt/kvm/guest_memfd.c
@@ -973,8 +973,6 @@ static int kvm_gmem_init_fs_context(struct fs_context *fc)
if (!init_pseudo(fc, GUEST_MEMFD_MAGIC))
return -ENOMEM;
- fc->s_iflags |= SB_I_NOEXEC;
- fc->s_iflags |= SB_I_NODEV;
ctx = fc->fs_private;
ctx->ops = &kvm_gmem_super_operations;
