Merge tag 'devicetree-for-6.2-2' of git://git.kernel.org/pub/scm/linux/kernel/git/robh/linux
Pull more devicetree updates from Rob Herring:
"This is mostly a treewide clean-up from Krzysztof. There's also a
couple of fixes and things that fell thru the cracks.
I must say this has been a nice merge window without bindings dumped
in at the last minute introducing warnings.
Summary:
- Treewide dropping of redundant 'binding' or 'schema' from schema
titles. This will be followed up with a automated check to catch
these.
- Re-sort vendor-prefies
- Convert GPIO based watchdog to schema
- Handle all the variations for clocks, resets, power domains in i.MX
PCIe binding
- Document missing 'power-domains' property in mxsfb
- Fix error with path references in Tegra XUSB example
- Honor CONFIG_CMDLINE* even without /chosen node"
* tag 'devicetree-for-6.2-2' of git://git.kernel.org/pub/scm/linux/kernel/git/robh/linux:
dt-bindings: drop redundant part of title (manual)
dt-bindings: clock: drop redundant part of title
dt-bindings: drop redundant part of title (beginning)
dt-bindings: drop redundant part of title (end, part three)
dt-bindings: drop redundant part of title (end, part two)
dt-bindings: drop redundant part of title (end)
dt-bindings: clock: st,stm32mp1-rcc: add proper title
dt-bindings: memory-controllers: ti,gpmc-child: drop redundant part of title
dt-bindings: drop redundant part of title of shared bindings
dt-bindings: watchdog: gpio: Convert bindings to YAML
dt-bindings: imx6q-pcie: Handle more resets on legacy platforms
dt-bindings: imx6q-pcie: Handle various PD configurations
dt-bindings: imx6q-pcie: Handle various clock configurations
dt-bindings: hwmon: ntc-thermistor: drop Naveen Krishna Chatradhi from maintainers
dt-bindings: mxsfb: Document i.MX8M/i.MX6SX/i.MX6SL power-domains property
dt-bindings: vendor-prefixes: sort entries alphabetically
dt-bindings: usb: tegra-xusb: Remove path references
of: fdt: Honor CONFIG_CMDLINE* even without /chosen node
diff --git a/CREDITS b/CREDITS
index 198f675..4e302a4 100644
--- a/CREDITS
+++ b/CREDITS
@@ -1439,6 +1439,10 @@
E: jguyett@andrew.cmu.edu
D: via-rhine net driver hacking
+N: Nitin Gupta
+E: ngupta@vflare.org
+D: zsmalloc memory allocator and zram block device driver
+
N: Danny ter Haar
E: dth@cistron.nl
D: /proc/cpuinfo, reboot on panic , kernel pre-patch tester ;)
diff --git a/Documentation/ABI/stable/sysfs-driver-dma-idxd b/Documentation/ABI/stable/sysfs-driver-dma-idxd
index 8e2c2c4..3becc9a 100644
--- a/Documentation/ABI/stable/sysfs-driver-dma-idxd
+++ b/Documentation/ABI/stable/sysfs-driver-dma-idxd
@@ -22,6 +22,7 @@
KernelVersion: 5.6.0
Contact: dmaengine@vger.kernel.org
Description: The largest number of work descriptors in a batch.
+ It's not visible when the device does not support batch.
What: /sys/bus/dsa/devices/dsa<m>/max_work_queues_size
Date: Oct 25, 2019
@@ -49,6 +50,8 @@
The read buffers represent resources within the DSA
implementation, and these resources are allocated by engines to
support operations. See DSA spec v1.2 9.2.4 Total Read Buffers.
+ It's not visible when the device does not support Read Buffer
+ allocation control.
What: /sys/bus/dsa/devices/dsa<m>/max_transfer_size
Date: Oct 25, 2019
@@ -122,6 +125,8 @@
Description: The maximum number of read buffers that may be in use at
one time by operations that access low bandwidth memory in the
device. See DSA spec v1.2 9.2.8 GENCFG on Global Read Buffer Limit.
+ It's not visible when the device does not support Read Buffer
+ allocation control.
What: /sys/bus/dsa/devices/dsa<m>/cmd_status
Date: Aug 28, 2020
@@ -205,6 +210,7 @@
Contact: dmaengine@vger.kernel.org
Description: The max batch size for this workqueue. Cannot exceed device
max batch size. Configurable parameter.
+ It's not visible when the device does not support batch.
What: /sys/bus/dsa/devices/wq<m>.<n>/ats_disable
Date: Nov 13, 2020
@@ -250,6 +256,8 @@
Contact: dmaengine@vger.kernel.org
Description: Enable the use of global read buffer limit for the group. See DSA
spec v1.2 9.2.18 GRPCFG Use Global Read Buffer Limit.
+ It's not visible when the device does not support Read Buffer
+ allocation control.
What: /sys/bus/dsa/devices/group<m>.<n>/read_buffers_allowed
Date: Dec 10, 2021
@@ -258,6 +266,8 @@
Description: Indicates max number of read buffers that may be in use at one time
by all engines in the group. See DSA spec v1.2 9.2.18 GRPCFG Read
Buffers Allowed.
+ It's not visible when the device does not support Read Buffer
+ allocation control.
What: /sys/bus/dsa/devices/group<m>.<n>/read_buffers_reserved
Date: Dec 10, 2021
@@ -266,6 +276,8 @@
Description: Indicates the number of Read Buffers reserved for the use of
engines in the group. See DSA spec v1.2 9.2.18 GRPCFG Read Buffers
Reserved.
+ It's not visible when the device does not support Read Buffer
+ allocation control.
What: /sys/bus/dsa/devices/group<m>.<n>/desc_progress_limit
Date: Sept 14, 2022
diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt
index 5a0eff3..6cfa6e3 100644
--- a/Documentation/admin-guide/kernel-parameters.txt
+++ b/Documentation/admin-guide/kernel-parameters.txt
@@ -1050,6 +1050,11 @@
them frequently to increase the rate of SLB faults
on kernel addresses.
+ stress_hpt [PPC]
+ Limits the number of kernel HPT entries in the hash
+ page table to increase the rate of hash page table
+ faults on kernel addresses.
+
disable= [IPV6]
See Documentation/networking/ipv6.rst.
@@ -2308,7 +2313,13 @@
Provide an override to the IOAPIC-ID<->DEVICE-ID
mapping provided in the IVRS ACPI table.
By default, PCI segment is 0, and can be omitted.
- For example:
+
+ For example, to map IOAPIC-ID decimal 10 to
+ PCI segment 0x1 and PCI device 00:14.0,
+ write the parameter as:
+ ivrs_ioapic=10@0001:00:14.0
+
+ Deprecated formats:
* To map IOAPIC-ID decimal 10 to PCI device 00:14.0
write the parameter as:
ivrs_ioapic[10]=00:14.0
@@ -2320,7 +2331,13 @@
Provide an override to the HPET-ID<->DEVICE-ID
mapping provided in the IVRS ACPI table.
By default, PCI segment is 0, and can be omitted.
- For example:
+
+ For example, to map HPET-ID decimal 10 to
+ PCI segment 0x1 and PCI device 00:14.0,
+ write the parameter as:
+ ivrs_hpet=10@0001:00:14.0
+
+ Deprecated formats:
* To map HPET-ID decimal 0 to PCI device 00:14.0
write the parameter as:
ivrs_hpet[0]=00:14.0
@@ -2331,15 +2348,20 @@
ivrs_acpihid [HW,X86-64]
Provide an override to the ACPI-HID:UID<->DEVICE-ID
mapping provided in the IVRS ACPI table.
+ By default, PCI segment is 0, and can be omitted.
For example, to map UART-HID:UID AMD0020:0 to
PCI segment 0x1 and PCI device ID 00:14.5,
write the parameter as:
- ivrs_acpihid[0001:00:14.5]=AMD0020:0
+ ivrs_acpihid=AMD0020:0@0001:00:14.5
- By default, PCI segment is 0, and can be omitted.
- For example, PCI device 00:14.5 write the parameter as:
+ Deprecated formats:
+ * To map UART-HID:UID AMD0020:0 to PCI segment is 0,
+ PCI device ID 00:14.5, write the parameter as:
ivrs_acpihid[00:14.5]=AMD0020:0
+ * To map UART-HID:UID AMD0020:0 to PCI segment 0x1 and
+ PCI device ID 00:14.5, write the parameter as:
+ ivrs_acpihid[0001:00:14.5]=AMD0020:0
js= [HW,JOY] Analog joystick
See Documentation/input/joydev/joystick.rst.
diff --git a/Documentation/admin-guide/sysctl/kernel.rst b/Documentation/admin-guide/sysctl/kernel.rst
index 659d7bc..46e3d62 100644
--- a/Documentation/admin-guide/sysctl/kernel.rst
+++ b/Documentation/admin-guide/sysctl/kernel.rst
@@ -436,8 +436,8 @@
On architectures where unaligned accesses cause traps, and where this
feature is supported (``CONFIG_SYSCTL_ARCH_UNALIGN_NO_WARN``;
-currently, ``arc`` and ``ia64``), controls whether all unaligned traps
-are logged.
+currently, ``arc``, ``ia64`` and ``loongarch``), controls whether all
+unaligned traps are logged.
= =============================================================
0 Log all unaligned accesses.
@@ -1492,8 +1492,8 @@
On architectures where unaligned accesses cause traps, and where this
feature is supported (``CONFIG_SYSCTL_ARCH_UNALIGN_ALLOW``; currently,
-``arc`` and ``parisc``), controls whether unaligned traps are caught
-and emulated (instead of failing).
+``arc``, ``parisc`` and ``loongarch``), controls whether unaligned traps
+are caught and emulated (instead of failing).
= ========================================================
0 Do not emulate unaligned accesses.
diff --git a/Documentation/arm64/silicon-errata.rst b/Documentation/arm64/silicon-errata.rst
index ec5f889..808ade4 100644
--- a/Documentation/arm64/silicon-errata.rst
+++ b/Documentation/arm64/silicon-errata.rst
@@ -120,8 +120,6 @@
+----------------+-----------------+-----------------+-----------------------------+
| ARM | Cortex-A710 | #2224489 | ARM64_ERRATUM_2224489 |
+----------------+-----------------+-----------------+-----------------------------+
-| ARM | Cortex-A715 | #2645198 | ARM64_ERRATUM_2645198 |
-+----------------+-----------------+-----------------+-----------------------------+
| ARM | Cortex-X2 | #2119858 | ARM64_ERRATUM_2119858 |
+----------------+-----------------+-----------------+-----------------------------+
| ARM | Cortex-X2 | #2224489 | ARM64_ERRATUM_2224489 |
diff --git a/Documentation/devicetree/bindings/dma/ingenic,dma.yaml b/Documentation/devicetree/bindings/dma/ingenic,dma.yaml
index 2be6ed1..fd5b0a8 100644
--- a/Documentation/devicetree/bindings/dma/ingenic,dma.yaml
+++ b/Documentation/devicetree/bindings/dma/ingenic,dma.yaml
@@ -18,6 +18,7 @@
- enum:
- ingenic,jz4740-dma
- ingenic,jz4725b-dma
+ - ingenic,jz4755-dma
- ingenic,jz4760-dma
- ingenic,jz4760-bdma
- ingenic,jz4760-mdma
diff --git a/Documentation/devicetree/bindings/dma/nvidia,tegra186-gpc-dma.yaml b/Documentation/devicetree/bindings/dma/nvidia,tegra186-gpc-dma.yaml
index c889447..851bd50 100644
--- a/Documentation/devicetree/bindings/dma/nvidia,tegra186-gpc-dma.yaml
+++ b/Documentation/devicetree/bindings/dma/nvidia,tegra186-gpc-dma.yaml
@@ -39,7 +39,7 @@
Should contain all of the per-channel DMA interrupts in
ascending order with respect to the DMA channel index.
minItems: 1
- maxItems: 31
+ maxItems: 32
resets:
maxItems: 1
@@ -52,6 +52,9 @@
dma-coherent: true
+ dma-channel-mask:
+ maxItems: 1
+
required:
- compatible
- reg
@@ -60,6 +63,7 @@
- reset-names
- "#dma-cells"
- iommus
+ - dma-channel-mask
additionalProperties: false
@@ -108,5 +112,6 @@
#dma-cells = <1>;
iommus = <&smmu TEGRA186_SID_GPCDMA_0>;
dma-coherent;
+ dma-channel-mask = <0xfffffffe>;
};
...
diff --git a/Documentation/devicetree/bindings/dma/qcom,gpi.yaml b/Documentation/devicetree/bindings/dma/qcom,gpi.yaml
index eabf8a7..e7ba1c47 100644
--- a/Documentation/devicetree/bindings/dma/qcom,gpi.yaml
+++ b/Documentation/devicetree/bindings/dma/qcom,gpi.yaml
@@ -18,14 +18,24 @@
properties:
compatible:
- enum:
- - qcom,sc7280-gpi-dma
- - qcom,sdm845-gpi-dma
- - qcom,sm6350-gpi-dma
- - qcom,sm8150-gpi-dma
- - qcom,sm8250-gpi-dma
- - qcom,sm8350-gpi-dma
- - qcom,sm8450-gpi-dma
+ oneOf:
+ - enum:
+ - qcom,sdm845-gpi-dma
+ - qcom,sm6350-gpi-dma
+ - items:
+ - enum:
+ - qcom,sc7280-gpi-dma
+ - qcom,sm6115-gpi-dma
+ - qcom,sm6375-gpi-dma
+ - qcom,sm8350-gpi-dma
+ - qcom,sm8450-gpi-dma
+ - const: qcom,sm6350-gpi-dma
+ - items:
+ - enum:
+ - qcom,sdm670-gpi-dma
+ - qcom,sm8150-gpi-dma
+ - qcom,sm8250-gpi-dma
+ - const: qcom,sdm845-gpi-dma
reg:
maxItems: 1
diff --git a/Documentation/devicetree/bindings/iommu/arm,smmu.yaml b/Documentation/devicetree/bindings/iommu/arm,smmu.yaml
index 9066e6d..b28c5c2 100644
--- a/Documentation/devicetree/bindings/iommu/arm,smmu.yaml
+++ b/Documentation/devicetree/bindings/iommu/arm,smmu.yaml
@@ -28,19 +28,50 @@
- enum:
- qcom,msm8996-smmu-v2
- qcom,msm8998-smmu-v2
+ - qcom,sdm630-smmu-v2
- const: qcom,smmu-v2
- - description: Qcom SoCs implementing "arm,mmu-500"
+ - description: Qcom SoCs implementing "qcom,smmu-500" and "arm,mmu-500"
items:
- enum:
- qcom,qcm2290-smmu-500
+ - qcom,qdu1000-smmu-500
+ - qcom,sc7180-smmu-500
+ - qcom,sc7280-smmu-500
+ - qcom,sc8180x-smmu-500
+ - qcom,sc8280xp-smmu-500
+ - qcom,sdm670-smmu-500
+ - qcom,sdm845-smmu-500
+ - qcom,sm6115-smmu-500
+ - qcom,sm6350-smmu-500
+ - qcom,sm6375-smmu-500
+ - qcom,sm8150-smmu-500
+ - qcom,sm8250-smmu-500
+ - qcom,sm8350-smmu-500
+ - qcom,sm8450-smmu-500
+ - const: qcom,smmu-500
+ - const: arm,mmu-500
+
+ - description: Qcom SoCs implementing "arm,mmu-500" (non-qcom implementation)
+ deprecated: true
+ items:
+ - enum:
+ - qcom,sdx55-smmu-500
+ - qcom,sdx65-smmu-500
+ - const: arm,mmu-500
+
+ - description: Qcom SoCs implementing "arm,mmu-500" (legacy binding)
+ deprecated: true
+ items:
+ # Do not add additional SoC to this list. Instead use two previous lists.
+ - enum:
+ - qcom,qcm2290-smmu-500
- qcom,sc7180-smmu-500
- qcom,sc7280-smmu-500
- qcom,sc8180x-smmu-500
- qcom,sc8280xp-smmu-500
- qcom,sdm845-smmu-500
- - qcom,sdx55-smmu-500
- - qcom,sdx65-smmu-500
+ - qcom,sm6115-smmu-500
- qcom,sm6350-smmu-500
- qcom,sm6375-smmu-500
- qcom,sm8150-smmu-500
@@ -48,13 +79,28 @@
- qcom,sm8350-smmu-500
- qcom,sm8450-smmu-500
- const: arm,mmu-500
+
+ - description: Qcom Adreno GPUs implementing "arm,smmu-500"
+ items:
+ - enum:
+ - qcom,sc7280-smmu-500
+ - qcom,sm8250-smmu-500
+ - const: qcom,adreno-smmu
+ - const: arm,mmu-500
- description: Qcom Adreno GPUs implementing "arm,smmu-v2"
items:
- enum:
+ - qcom,msm8996-smmu-v2
- qcom,sc7180-smmu-v2
+ - qcom,sdm630-smmu-v2
- qcom,sdm845-smmu-v2
+ - qcom,sm6350-smmu-v2
- const: qcom,adreno-smmu
- const: qcom,smmu-v2
+ - description: Qcom Adreno GPUs on Google Cheza platform
+ items:
+ - const: qcom,sdm845-smmu-v2
+ - const: qcom,smmu-v2
- description: Marvell SoCs implementing "arm,mmu-500"
items:
- const: marvell,ap806-smmu-500
@@ -147,16 +193,12 @@
present in such cases.
clock-names:
- items:
- - const: bus
- - const: iface
+ minItems: 1
+ maxItems: 7
clocks:
- items:
- - description: bus clock required for downstream bus access and for the
- smmu ptw
- - description: interface clock required to access smmu's registers
- through the TCU's programming interface.
+ minItems: 1
+ maxItems: 7
power-domains:
maxItems: 1
@@ -206,6 +248,124 @@
reg:
maxItems: 1
+ - if:
+ properties:
+ compatible:
+ contains:
+ enum:
+ - qcom,msm8998-smmu-v2
+ - qcom,sdm630-smmu-v2
+ then:
+ anyOf:
+ - properties:
+ clock-names:
+ items:
+ - const: bus
+ clocks:
+ items:
+ - description: bus clock required for downstream bus access and for
+ the smmu ptw
+ - properties:
+ clock-names:
+ items:
+ - const: iface
+ - const: mem
+ - const: mem_iface
+ clocks:
+ items:
+ - description: interface clock required to access smmu's registers
+ through the TCU's programming interface.
+ - description: bus clock required for memory access
+ - description: bus clock required for GPU memory access
+ - properties:
+ clock-names:
+ items:
+ - const: iface-mm
+ - const: iface-smmu
+ - const: bus-mm
+ - const: bus-smmu
+ clocks:
+ items:
+ - description: interface clock required to access mnoc's registers
+ through the TCU's programming interface.
+ - description: interface clock required to access smmu's registers
+ through the TCU's programming interface.
+ - description: bus clock required for downstream bus access
+ - description: bus clock required for the smmu ptw
+
+ - if:
+ properties:
+ compatible:
+ contains:
+ enum:
+ - qcom,msm8996-smmu-v2
+ - qcom,sc7180-smmu-v2
+ - qcom,sdm845-smmu-v2
+ then:
+ properties:
+ clock-names:
+ items:
+ - const: bus
+ - const: iface
+
+ clocks:
+ items:
+ - description: bus clock required for downstream bus access and for
+ the smmu ptw
+ - description: interface clock required to access smmu's registers
+ through the TCU's programming interface.
+
+ - if:
+ properties:
+ compatible:
+ contains:
+ const: qcom,sc7280-smmu-500
+ then:
+ properties:
+ clock-names:
+ items:
+ - const: gcc_gpu_memnoc_gfx_clk
+ - const: gcc_gpu_snoc_dvm_gfx_clk
+ - const: gpu_cc_ahb_clk
+ - const: gpu_cc_hlos1_vote_gpu_smmu_clk
+ - const: gpu_cc_cx_gmu_clk
+ - const: gpu_cc_hub_cx_int_clk
+ - const: gpu_cc_hub_aon_clk
+
+ clocks:
+ items:
+ - description: GPU memnoc_gfx clock
+ - description: GPU snoc_dvm_gfx clock
+ - description: GPU ahb clock
+ - description: GPU hlos1_vote_GPU smmu clock
+ - description: GPU cx_gmu clock
+ - description: GPU hub_cx_int clock
+ - description: GPU hub_aon clock
+
+ - if:
+ properties:
+ compatible:
+ contains:
+ enum:
+ - qcom,sm6350-smmu-v2
+ - qcom,sm8150-smmu-500
+ - qcom,sm8250-smmu-500
+ then:
+ properties:
+ clock-names:
+ items:
+ - const: ahb
+ - const: bus
+ - const: iface
+
+ clocks:
+ items:
+ - description: bus clock required for AHB bus access
+ - description: bus clock required for downstream bus access and for
+ the smmu ptw
+ - description: interface clock required to access smmu's registers
+ through the TCU's programming interface.
+
examples:
- |+
/* SMMU with stream matching or stream indexing */
diff --git a/Documentation/devicetree/bindings/iommu/mediatek,iommu.yaml b/Documentation/devicetree/bindings/iommu/mediatek,iommu.yaml
index 839e3be0..5b6395b 100644
--- a/Documentation/devicetree/bindings/iommu/mediatek,iommu.yaml
+++ b/Documentation/devicetree/bindings/iommu/mediatek,iommu.yaml
@@ -82,6 +82,7 @@
- mediatek,mt8195-iommu-vdo # generation two
- mediatek,mt8195-iommu-vpp # generation two
- mediatek,mt8195-iommu-infra # generation two
+ - mediatek,mt8365-m4u # generation two
- description: mt7623 generation one
items:
@@ -132,6 +133,7 @@
dt-binding/memory/mt8186-memory-port.h for mt8186,
dt-binding/memory/mt8192-larb-port.h for mt8192.
dt-binding/memory/mt8195-memory-port.h for mt8195.
+ dt-binding/memory/mediatek,mt8365-larb-port.h for mt8365.
power-domains:
maxItems: 1
diff --git a/Documentation/devicetree/bindings/leds/common.yaml b/Documentation/devicetree/bindings/leds/common.yaml
index 3c14a98..f5c57a5 100644
--- a/Documentation/devicetree/bindings/leds/common.yaml
+++ b/Documentation/devicetree/bindings/leds/common.yaml
@@ -100,6 +100,7 @@
- pattern
# LED is triggered by SD/MMC activity
- pattern: "^mmc[0-9]+$"
+ - pattern: "^cpu[0-9]*$"
led-pattern:
description: |
diff --git a/Documentation/devicetree/bindings/leds/leds-pm8058.txt b/Documentation/devicetree/bindings/leds/leds-pm8058.txt
deleted file mode 100644
index 89584c4..0000000
--- a/Documentation/devicetree/bindings/leds/leds-pm8058.txt
+++ /dev/null
@@ -1,67 +0,0 @@
-Qualcomm PM8058 LED driver
-
-The Qualcomm PM8058 is a multi-functional device which contains
-an LED driver block for up to six LEDs: three normal LEDs, two
-"flash" LEDs and one "keypad backlight" LED. The names are
-quoted because sometimes these LED drivers are used for wildly
-different things than flash or keypad backlight: their names
-are more of a suggestion than a hard-wired usecase.
-
-Hardware-wise the different LEDs support slightly different
-output currents. The "flash" LEDs do not need to charge nor
-do they support external triggers. They are just powerful LED
-drivers.
-
-The LEDs appear as children to the PM8058 device, with the
-proper compatible string. For the PM8058 bindings see:
-mfd/qcom-pm8xxx.txt.
-
-Each LED is represented as a sub-node of the syscon device. Each
-node's name represents the name of the corresponding LED.
-
-LED sub-node properties:
-
-Required properties:
-- compatible: one of
- "qcom,pm8058-led" (for the normal LEDs at 0x131, 0x132 and 0x133)
- "qcom,pm8058-keypad-led" (for the "keypad" LED at 0x48)
- "qcom,pm8058-flash-led" (for the "flash" LEDs at 0x49 and 0xFB)
-
-Optional properties:
-- label: see Documentation/devicetree/bindings/leds/common.txt
-- default-state: see Documentation/devicetree/bindings/leds/common.txt
-- linux,default-trigger: see Documentation/devicetree/bindings/leds/common.txt
-
-Example:
-
-qcom,ssbi@500000 {
- pmicintc: pmic@0 {
- compatible = "qcom,pm8058";
- led@48 {
- compatible = "qcom,pm8058-keypad-led";
- reg = <0x48>;
- label = "pm8050:white:keypad";
- default-state = "off";
- };
- led@131 {
- compatible = "qcom,pm8058-led";
- reg = <0x131>;
- label = "pm8058:red";
- default-state = "off";
- };
- led@132 {
- compatible = "qcom,pm8058-led";
- reg = <0x132>;
- label = "pm8058:yellow";
- default-state = "off";
- linux,default-trigger = "mmc0";
- };
- led@133 {
- compatible = "qcom,pm8058-led";
- reg = <0x133>;
- label = "pm8058:green";
- default-state = "on";
- linux,default-trigger = "heartbeat";
- };
- };
-};
diff --git a/Documentation/devicetree/bindings/leds/qcom,pm8058-led.yaml b/Documentation/devicetree/bindings/leds/qcom,pm8058-led.yaml
new file mode 100644
index 0000000..fa03e73
--- /dev/null
+++ b/Documentation/devicetree/bindings/leds/qcom,pm8058-led.yaml
@@ -0,0 +1,57 @@
+# SPDX-License-Identifier: GPL-2.0-only OR BSD-2-Clause
+%YAML 1.2
+---
+$id: http://devicetree.org/schemas/leds/qcom,pm8058-led.yaml#
+$schema: http://devicetree.org/meta-schemas/core.yaml#
+
+title: Qualcomm PM8058 PMIC LED
+
+maintainers:
+ - Krzysztof Kozlowski <krzysztof.kozlowski@linaro.org>
+
+description: |
+ The Qualcomm PM8058 contains an LED block for up to six LEDs:: three normal
+ LEDs, two "flash" LEDs and one "keypad backlight" LED. The names are quoted
+ because sometimes these LED drivers are used for wildly different things than
+ flash or keypad backlight:: their names are more of a suggestion than a
+ hard-wired usecase.
+
+ Hardware-wise the different LEDs support slightly different output currents.
+ The "flash" LEDs do not need to charge nor do they support external triggers.
+ They are just powerful LED drivers.
+
+allOf:
+ - $ref: common.yaml#
+
+properties:
+ compatible:
+ enum:
+ - qcom,pm8058-led
+ - qcom,pm8058-keypad-led
+ - qcom,pm8058-flash-led
+
+ reg:
+ maxItems: 1
+
+required:
+ - compatible
+ - reg
+
+unevaluatedProperties: false
+
+examples:
+ - |
+ #include <dt-bindings/leds/common.h>
+
+ pmic {
+ #address-cells = <1>;
+ #size-cells = <0>;
+
+ led@131 {
+ compatible = "qcom,pm8058-led";
+ reg = <0x131>;
+ label = "pm8058:red";
+ color = <LED_COLOR_ID_RED>;
+ default-state = "off";
+ };
+ };
diff --git a/Documentation/devicetree/bindings/mfd/ene-kb930.yaml b/Documentation/devicetree/bindings/mfd/ene-kb930.yaml
index aa29e3f..02c111d 100644
--- a/Documentation/devicetree/bindings/mfd/ene-kb930.yaml
+++ b/Documentation/devicetree/bindings/mfd/ene-kb930.yaml
@@ -13,6 +13,8 @@
maintainers:
- Dmitry Osipenko <digetx@gmail.com>
+$ref: /schemas/power/supply/power-supply.yaml
+
properties:
compatible:
items:
@@ -22,15 +24,13 @@
reg:
maxItems: 1
- monitored-battery: true
- power-supplies: true
system-power-controller: true
required:
- compatible
- reg
-additionalProperties: false
+unevaluatedProperties: false
examples:
- |
diff --git a/Documentation/devicetree/bindings/mfd/qcom-pm8xxx.yaml b/Documentation/devicetree/bindings/mfd/qcom-pm8xxx.yaml
index 61bd0b3..bd6e4ae 100644
--- a/Documentation/devicetree/bindings/mfd/qcom-pm8xxx.yaml
+++ b/Documentation/devicetree/bindings/mfd/qcom-pm8xxx.yaml
@@ -39,6 +39,10 @@
interrupt-controller: true
patternProperties:
+ "led@[0-9a-f]+$":
+ type: object
+ $ref: /schemas/leds/qcom,pm8058-led.yaml#
+
"rtc@[0-9a-f]+$":
type: object
$ref: "../rtc/qcom-pm8xxx-rtc.yaml"
diff --git a/Documentation/devicetree/bindings/mfd/rockchip,rk817.yaml b/Documentation/devicetree/bindings/mfd/rockchip,rk817.yaml
index 935e170..269fb85 100644
--- a/Documentation/devicetree/bindings/mfd/rockchip,rk817.yaml
+++ b/Documentation/devicetree/bindings/mfd/rockchip,rk817.yaml
@@ -124,6 +124,8 @@
The child node for the charger to hold additional properties. If a
battery is not in use, this node can be omitted.
type: object
+ $ref: /schemas/power/supply/power-supply.yaml
+
properties:
monitored-battery:
description: |
diff --git a/Documentation/devicetree/bindings/phy/allwinner,sun6i-a31-mipi-dphy.yaml b/Documentation/devicetree/bindings/phy/allwinner,sun6i-a31-mipi-dphy.yaml
index dfb6a89..fe9702e 100644
--- a/Documentation/devicetree/bindings/phy/allwinner,sun6i-a31-mipi-dphy.yaml
+++ b/Documentation/devicetree/bindings/phy/allwinner,sun6i-a31-mipi-dphy.yaml
@@ -17,13 +17,20 @@
compatible:
oneOf:
- const: allwinner,sun6i-a31-mipi-dphy
+ - const: allwinner,sun50i-a100-mipi-dphy
- items:
- const: allwinner,sun50i-a64-mipi-dphy
- const: allwinner,sun6i-a31-mipi-dphy
+ - items:
+ - const: allwinner,sun20i-d1-mipi-dphy
+ - const: allwinner,sun50i-a100-mipi-dphy
reg:
maxItems: 1
+ interrupts:
+ maxItems: 1
+
clocks:
items:
- description: Bus Clock
@@ -53,6 +60,7 @@
- "#phy-cells"
- compatible
- reg
+ - interrupts
- clocks
- clock-names
- resets
@@ -61,9 +69,12 @@
examples:
- |
+ #include <dt-bindings/interrupt-controller/arm-gic.h>
+
dphy0: d-phy@1ca1000 {
compatible = "allwinner,sun6i-a31-mipi-dphy";
reg = <0x01ca1000 0x1000>;
+ interrupts = <GIC_SPI 89 IRQ_TYPE_LEVEL_HIGH>;
clocks = <&ccu 23>, <&ccu 97>;
clock-names = "bus", "mod";
resets = <&ccu 4>;
diff --git a/Documentation/devicetree/bindings/phy/allwinner,sun8i-h3-usb-phy.yaml b/Documentation/devicetree/bindings/phy/allwinner,sun8i-h3-usb-phy.yaml
index 77539b4..2df012d 100644
--- a/Documentation/devicetree/bindings/phy/allwinner,sun8i-h3-usb-phy.yaml
+++ b/Documentation/devicetree/bindings/phy/allwinner,sun8i-h3-usb-phy.yaml
@@ -36,18 +36,22 @@
- const: pmu3
clocks:
+ minItems: 4
items:
- description: USB OTG PHY bus clock
- description: USB Host 0 PHY bus clock
- description: USB Host 1 PHY bus clock
- description: USB Host 2 PHY bus clock
+ - description: PMU clock for host port 2
clock-names:
+ minItems: 4
items:
- const: usb0_phy
- const: usb1_phy
- const: usb2_phy
- const: usb3_phy
+ - const: pmu2_clk
resets:
items:
@@ -96,6 +100,28 @@
- resets
- reset-names
+allOf:
+ - if:
+ properties:
+ compatible:
+ contains:
+ enum:
+ - allwinner,sun50i-h616-usb-phy
+ then:
+ properties:
+ clocks:
+ minItems: 5
+
+ clock-names:
+ minItems: 5
+ else:
+ properties:
+ clocks:
+ maxItems: 4
+
+ clock-names:
+ maxItems: 4
+
additionalProperties: false
examples:
diff --git a/Documentation/devicetree/bindings/phy/fsl,imx8-pcie-phy.yaml b/Documentation/devicetree/bindings/phy/fsl,imx8-pcie-phy.yaml
index 0af765b..182a219 100644
--- a/Documentation/devicetree/bindings/phy/fsl,imx8-pcie-phy.yaml
+++ b/Documentation/devicetree/bindings/phy/fsl,imx8-pcie-phy.yaml
@@ -16,6 +16,7 @@
compatible:
enum:
- fsl,imx8mm-pcie-phy
+ - fsl,imx8mp-pcie-phy
reg:
maxItems: 1
@@ -28,11 +29,16 @@
- const: ref
resets:
- maxItems: 1
+ minItems: 1
+ maxItems: 2
reset-names:
- items:
- - const: pciephy
+ oneOf:
+ - items: # for iMX8MM
+ - const: pciephy
+ - items: # for IMX8MP
+ - const: pciephy
+ - const: perst
fsl,refclk-pad-mode:
description: |
@@ -60,6 +66,10 @@
description: A boolean property indicating the CLKREQ# signal is
not supported in the board design (optional)
+ power-domains:
+ description: PCIe PHY power domain (optional).
+ maxItems: 1
+
required:
- "#phy-cells"
- compatible
diff --git a/Documentation/devicetree/bindings/phy/qcom,qmp-pcie-phy.yaml b/Documentation/devicetree/bindings/phy/qcom,ipq8074-qmp-pcie-phy.yaml
similarity index 96%
rename from Documentation/devicetree/bindings/phy/qcom,qmp-pcie-phy.yaml
rename to Documentation/devicetree/bindings/phy/qcom,ipq8074-qmp-pcie-phy.yaml
index 324ad7d..62045dc 100644
--- a/Documentation/devicetree/bindings/phy/qcom,qmp-pcie-phy.yaml
+++ b/Documentation/devicetree/bindings/phy/qcom,ipq8074-qmp-pcie-phy.yaml
@@ -1,10 +1,10 @@
# SPDX-License-Identifier: (GPL-2.0 OR BSD-2-Clause)
%YAML 1.2
---
-$id: http://devicetree.org/schemas/phy/qcom,qmp-pcie-phy.yaml#
+$id: http://devicetree.org/schemas/phy/qcom,ipq8074-qmp-pcie-phy.yaml#
$schema: http://devicetree.org/meta-schemas/core.yaml#
-title: Qualcomm QMP PHY controller (PCIe)
+title: Qualcomm QMP PHY controller (PCIe, IPQ8074)
maintainers:
- Vinod Koul <vkoul@kernel.org>
@@ -13,6 +13,9 @@
QMP PHY controller supports physical layer functionality for a number of
controllers on Qualcomm chipsets, such as, PCIe, UFS, and USB.
+ Note that these bindings are for SoCs up to SC8180X. For newer SoCs, see
+ qcom,sc8280xp-qmp-pcie-phy.yaml.
+
properties:
compatible:
enum:
diff --git a/Documentation/devicetree/bindings/phy/qcom,qmp-ufs-phy.yaml b/Documentation/devicetree/bindings/phy/qcom,msm8996-qmp-ufs-phy.yaml
similarity index 92%
rename from Documentation/devicetree/bindings/phy/qcom,qmp-ufs-phy.yaml
rename to Documentation/devicetree/bindings/phy/qcom,msm8996-qmp-ufs-phy.yaml
index 815c375..be41acb 100644
--- a/Documentation/devicetree/bindings/phy/qcom,qmp-ufs-phy.yaml
+++ b/Documentation/devicetree/bindings/phy/qcom,msm8996-qmp-ufs-phy.yaml
@@ -1,10 +1,10 @@
# SPDX-License-Identifier: (GPL-2.0 OR BSD-2-Clause)
%YAML 1.2
---
-$id: http://devicetree.org/schemas/phy/qcom,qmp-ufs-phy.yaml#
+$id: http://devicetree.org/schemas/phy/qcom,msm8996-qmp-ufs-phy.yaml#
$schema: http://devicetree.org/meta-schemas/core.yaml#
-title: Qualcomm QMP PHY controller (UFS)
+title: Qualcomm QMP PHY controller (UFS, MSM8996)
maintainers:
- Vinod Koul <vkoul@kernel.org>
@@ -13,13 +13,15 @@
QMP PHY controller supports physical layer functionality for a number of
controllers on Qualcomm chipsets, such as, PCIe, UFS, and USB.
+ Note that these bindings are for SoCs up to SC8180X. For newer SoCs, see
+ qcom,sc8280xp-qmp-ufs-phy.yaml.
+
properties:
compatible:
enum:
- qcom,msm8996-qmp-ufs-phy
- qcom,msm8998-qmp-ufs-phy
- qcom,sc8180x-qmp-ufs-phy
- - qcom,sc8280xp-qmp-ufs-phy
- qcom,sdm845-qmp-ufs-phy
- qcom,sm6115-qmp-ufs-phy
- qcom,sm6350-qmp-ufs-phy
@@ -119,7 +121,6 @@
enum:
- qcom,msm8998-qmp-ufs-phy
- qcom,sc8180x-qmp-ufs-phy
- - qcom,sc8280xp-qmp-ufs-phy
- qcom,sdm845-qmp-ufs-phy
- qcom,sm6115-qmp-ufs-phy
- qcom,sm6350-qmp-ufs-phy
@@ -156,7 +157,6 @@
contains:
enum:
- qcom,msm8998-qmp-ufs-phy
- - qcom,sc8280xp-qmp-ufs-phy
- qcom,sdm845-qmp-ufs-phy
- qcom,sm6350-qmp-ufs-phy
- qcom,sm8150-qmp-ufs-phy
@@ -211,11 +211,12 @@
examples:
- |
- #include <dt-bindings/clock/qcom,gcc-sc8280xp.h>
+ #include <dt-bindings/clock/qcom,gcc-sm8250.h>
#include <dt-bindings/clock/qcom,rpmh.h>
+
phy-wrapper@1d87000 {
- compatible = "qcom,sc8280xp-qmp-ufs-phy";
- reg = <0x01d87000 0xe10>;
+ compatible = "qcom,sm8250-qmp-ufs-phy";
+ reg = <0x01d87000 0x1c0>;
#address-cells = <1>;
#size-cells = <1>;
ranges = <0x0 0x01d87000 0x1000>;
diff --git a/Documentation/devicetree/bindings/phy/qcom,qmp-usb-phy.yaml b/Documentation/devicetree/bindings/phy/qcom,msm8996-qmp-usb3-phy.yaml
similarity index 95%
rename from Documentation/devicetree/bindings/phy/qcom,qmp-usb-phy.yaml
rename to Documentation/devicetree/bindings/phy/qcom,msm8996-qmp-usb3-phy.yaml
index 7acb4b7..0c6b3ba 100644
--- a/Documentation/devicetree/bindings/phy/qcom,qmp-usb-phy.yaml
+++ b/Documentation/devicetree/bindings/phy/qcom,msm8996-qmp-usb3-phy.yaml
@@ -1,10 +1,10 @@
# SPDX-License-Identifier: (GPL-2.0 OR BSD-2-Clause)
%YAML 1.2
---
-$id: http://devicetree.org/schemas/phy/qcom,qmp-usb-phy.yaml#
+$id: http://devicetree.org/schemas/phy/qcom,msm8996-qmp-usb3-phy.yaml#
$schema: http://devicetree.org/meta-schemas/core.yaml#
-title: Qualcomm QMP PHY controller (USB)
+title: Qualcomm QMP PHY controller (USB, MSM8996)
maintainers:
- Vinod Koul <vkoul@kernel.org>
@@ -13,6 +13,9 @@
QMP PHY controller supports physical layer functionality for a number of
controllers on Qualcomm chipsets, such as, PCIe, UFS, and USB.
+ Note that these bindings are for SoCs up to SC8180X. For newer SoCs, see
+ qcom,sc8280xp-qmp-usb3-uni-phy.yaml.
+
properties:
compatible:
enum:
@@ -23,7 +26,6 @@
- qcom,qcm2290-qmp-usb3-phy
- qcom,sc7180-qmp-usb3-phy
- qcom,sc8180x-qmp-usb3-phy
- - qcom,sc8280xp-qmp-usb3-uni-phy
- qcom,sdm845-qmp-usb3-phy
- qcom,sdm845-qmp-usb3-uni-phy
- qcom,sdx55-qmp-usb3-uni-phy
@@ -201,7 +203,6 @@
compatible:
contains:
enum:
- - qcom,sc8280xp-qmp-usb3-uni-phy
- qcom,sm8150-qmp-usb3-phy
- qcom,sm8150-qmp-usb3-uni-phy
- qcom,sm8250-qmp-usb3-uni-phy
@@ -273,16 +274,6 @@
compatible:
contains:
enum:
- - qcom,sc8280xp-qmp-usb3-uni-phy
- then:
- required:
- - power-domains
-
- - if:
- properties:
- compatible:
- contains:
- enum:
- qcom,sdm845-qmp-usb3-phy
- qcom,sm8150-qmp-usb3-phy
- qcom,sm8350-qmp-usb3-phy
@@ -349,7 +340,6 @@
contains:
enum:
- qcom,msm8996-qmp-usb3-phy
- - qcom,sc8280xp-qmp-usb3-uni-phy
- qcom,sm8250-qmp-usb3-uni-phy
- qcom,sm8350-qmp-usb3-uni-phy
then:
diff --git a/Documentation/devicetree/bindings/phy/qcom,qmp-usb3-dp-phy.yaml b/Documentation/devicetree/bindings/phy/qcom,sc7180-qmp-usb3-dp-phy.yaml
similarity index 91%
rename from Documentation/devicetree/bindings/phy/qcom,qmp-usb3-dp-phy.yaml
rename to Documentation/devicetree/bindings/phy/qcom,sc7180-qmp-usb3-dp-phy.yaml
index 97a7eca..d9d0ab9 100644
--- a/Documentation/devicetree/bindings/phy/qcom,qmp-usb3-dp-phy.yaml
+++ b/Documentation/devicetree/bindings/phy/qcom,sc7180-qmp-usb3-dp-phy.yaml
@@ -2,10 +2,17 @@
%YAML 1.2
---
-$id: "http://devicetree.org/schemas/phy/qcom,qmp-usb3-dp-phy.yaml#"
-$schema: "http://devicetree.org/meta-schemas/core.yaml#"
+$id: http://devicetree.org/schemas/phy/qcom,sc7180-qmp-usb3-dp-phy.yaml#
+$schema: http://devicetree.org/meta-schemas/core.yaml#
-title: Qualcomm QMP USB3 DP PHY controller
+title: Qualcomm QMP USB3 DP PHY controller (SC7180)
+
+description:
+ The QMP PHY controller supports physical layer functionality for a number of
+ controllers on Qualcomm chipsets, such as, PCIe, UFS and USB.
+
+ Note that these bindings are for SoCs up to SC8180X. For newer SoCs, see
+ qcom,sc8280xp-qmp-usb43dp-phy.yaml.
maintainers:
- Wesley Cheng <quic_wcheng@quicinc.com>
@@ -16,7 +23,6 @@
- qcom,sc7180-qmp-usb3-dp-phy
- qcom,sc7280-qmp-usb3-dp-phy
- qcom,sc8180x-qmp-usb3-dp-phy
- - qcom,sc8280xp-qmp-usb43dp-phy
- qcom,sdm845-qmp-usb3-dp-phy
- qcom,sm8250-qmp-usb3-dp-phy
reg:
@@ -162,17 +168,6 @@
additionalProperties: false
-allOf:
- - if:
- properties:
- compatible:
- contains:
- enum:
- - qcom,sc8280xp-qmp-usb43dp-phy
- then:
- required:
- - power-domains
-
examples:
- |
#include <dt-bindings/clock/qcom,gcc-sdm845.h>
diff --git a/Documentation/devicetree/bindings/phy/qcom,sc8280xp-qmp-pcie-phy.yaml b/Documentation/devicetree/bindings/phy/qcom,sc8280xp-qmp-pcie-phy.yaml
new file mode 100644
index 0000000..80aa8d2
--- /dev/null
+++ b/Documentation/devicetree/bindings/phy/qcom,sc8280xp-qmp-pcie-phy.yaml
@@ -0,0 +1,165 @@
+# SPDX-License-Identifier: (GPL-2.0 OR BSD-2-Clause)
+%YAML 1.2
+---
+$id: http://devicetree.org/schemas/phy/qcom,sc8280xp-qmp-pcie-phy.yaml#
+$schema: http://devicetree.org/meta-schemas/core.yaml#
+
+title: Qualcomm QMP PHY controller (PCIe, SC8280XP)
+
+maintainers:
+ - Vinod Koul <vkoul@kernel.org>
+
+description:
+ The QMP PHY controller supports physical layer functionality for a number of
+ controllers on Qualcomm chipsets, such as, PCIe, UFS, and USB.
+
+properties:
+ compatible:
+ enum:
+ - qcom,sc8280xp-qmp-gen3x1-pcie-phy
+ - qcom,sc8280xp-qmp-gen3x2-pcie-phy
+ - qcom,sc8280xp-qmp-gen3x4-pcie-phy
+
+ reg:
+ minItems: 1
+ maxItems: 2
+
+ clocks:
+ maxItems: 6
+
+ clock-names:
+ items:
+ - const: aux
+ - const: cfg_ahb
+ - const: ref
+ - const: rchng
+ - const: pipe
+ - const: pipediv2
+
+ power-domains:
+ maxItems: 1
+
+ resets:
+ maxItems: 1
+
+ reset-names:
+ items:
+ - const: phy
+
+ vdda-phy-supply: true
+
+ vdda-pll-supply: true
+
+ qcom,4ln-config-sel:
+ description: PCIe 4-lane configuration
+ $ref: /schemas/types.yaml#/definitions/phandle-array
+ items:
+ - items:
+ - description: phandle of TCSR syscon
+ - description: offset of PCIe 4-lane configuration register
+ - description: offset of configuration bit for this PHY
+
+ "#clock-cells":
+ const: 0
+
+ clock-output-names:
+ maxItems: 1
+
+ "#phy-cells":
+ const: 0
+
+required:
+ - compatible
+ - reg
+ - clocks
+ - clock-names
+ - power-domains
+ - resets
+ - reset-names
+ - vdda-phy-supply
+ - vdda-pll-supply
+ - "#clock-cells"
+ - clock-output-names
+ - "#phy-cells"
+
+additionalProperties: false
+
+allOf:
+ - if:
+ properties:
+ compatible:
+ contains:
+ enum:
+ - qcom,sc8280xp-qmp-gen3x4-pcie-phy
+ then:
+ properties:
+ reg:
+ items:
+ - description: port a
+ - description: port b
+ required:
+ - qcom,4ln-config-sel
+ else:
+ properties:
+ reg:
+ maxItems: 1
+
+examples:
+ - |
+ #include <dt-bindings/clock/qcom,gcc-sc8280xp.h>
+
+ pcie2b_phy: phy@1c18000 {
+ compatible = "qcom,sc8280xp-qmp-gen3x2-pcie-phy";
+ reg = <0x01c18000 0x2000>;
+
+ clocks = <&gcc GCC_PCIE_2B_AUX_CLK>,
+ <&gcc GCC_PCIE_2B_CFG_AHB_CLK>,
+ <&gcc GCC_PCIE_2A2B_CLKREF_CLK>,
+ <&gcc GCC_PCIE2B_PHY_RCHNG_CLK>,
+ <&gcc GCC_PCIE_2B_PIPE_CLK>,
+ <&gcc GCC_PCIE_2B_PIPEDIV2_CLK>;
+ clock-names = "aux", "cfg_ahb", "ref", "rchng",
+ "pipe", "pipediv2";
+
+ power-domains = <&gcc PCIE_2B_GDSC>;
+
+ resets = <&gcc GCC_PCIE_2B_PHY_BCR>;
+ reset-names = "phy";
+
+ vdda-phy-supply = <&vreg_l6d>;
+ vdda-pll-supply = <&vreg_l4d>;
+
+ #clock-cells = <0>;
+ clock-output-names = "pcie_2b_pipe_clk";
+
+ #phy-cells = <0>;
+ };
+
+ pcie2a_phy: phy@1c24000 {
+ compatible = "qcom,sc8280xp-qmp-gen3x4-pcie-phy";
+ reg = <0x01c24000 0x2000>, <0x01c26000 0x2000>;
+
+ clocks = <&gcc GCC_PCIE_2A_AUX_CLK>,
+ <&gcc GCC_PCIE_2A_CFG_AHB_CLK>,
+ <&gcc GCC_PCIE_2A2B_CLKREF_CLK>,
+ <&gcc GCC_PCIE2A_PHY_RCHNG_CLK>,
+ <&gcc GCC_PCIE_2A_PIPE_CLK>,
+ <&gcc GCC_PCIE_2A_PIPEDIV2_CLK>;
+ clock-names = "aux", "cfg_ahb", "ref", "rchng",
+ "pipe", "pipediv2";
+
+ power-domains = <&gcc PCIE_2A_GDSC>;
+
+ resets = <&gcc GCC_PCIE_2A_PHY_BCR>;
+ reset-names = "phy";
+
+ vdda-phy-supply = <&vreg_l6d>;
+ vdda-pll-supply = <&vreg_l4d>;
+
+ qcom,4ln-config-sel = <&tcsr 0xa044 0>;
+
+ #clock-cells = <0>;
+ clock-output-names = "pcie_2a_pipe_clk";
+
+ #phy-cells = <0>;
+ };
diff --git a/Documentation/devicetree/bindings/phy/qcom,sc8280xp-qmp-ufs-phy.yaml b/Documentation/devicetree/bindings/phy/qcom,sc8280xp-qmp-ufs-phy.yaml
new file mode 100644
index 0000000..dde86a1
--- /dev/null
+++ b/Documentation/devicetree/bindings/phy/qcom,sc8280xp-qmp-ufs-phy.yaml
@@ -0,0 +1,83 @@
+# SPDX-License-Identifier: (GPL-2.0 OR BSD-2-Clause)
+%YAML 1.2
+---
+$id: http://devicetree.org/schemas/phy/qcom,sc8280xp-qmp-ufs-phy.yaml#
+$schema: http://devicetree.org/meta-schemas/core.yaml#
+
+title: Qualcomm QMP PHY controller (UFS, SC8280XP)
+
+maintainers:
+ - Vinod Koul <vkoul@kernel.org>
+
+description:
+ The QMP PHY controller supports physical layer functionality for a number of
+ controllers on Qualcomm chipsets, such as, PCIe, UFS, and USB.
+
+properties:
+ compatible:
+ enum:
+ - qcom,sc8280xp-qmp-ufs-phy
+
+ reg:
+ maxItems: 1
+
+ clocks:
+ maxItems: 2
+
+ clock-names:
+ items:
+ - const: ref
+ - const: ref_aux
+
+ power-domains:
+ maxItems: 1
+
+ resets:
+ maxItems: 1
+
+ reset-names:
+ items:
+ - const: ufsphy
+
+ vdda-phy-supply: true
+
+ vdda-pll-supply: true
+
+ "#phy-cells":
+ const: 0
+
+required:
+ - compatible
+ - reg
+ - clocks
+ - clock-names
+ - power-domains
+ - resets
+ - reset-names
+ - vdda-phy-supply
+ - vdda-pll-supply
+ - "#phy-cells"
+
+additionalProperties: false
+
+examples:
+ - |
+ #include <dt-bindings/clock/qcom,gcc-sc8280xp.h>
+
+ ufs_mem_phy: phy@1d87000 {
+ compatible = "qcom,sc8280xp-qmp-ufs-phy";
+ reg = <0x01d87000 0x1000>;
+
+ clocks = <&gcc GCC_UFS_REF_CLKREF_CLK>, <&gcc GCC_UFS_PHY_PHY_AUX_CLK>;
+ clock-names = "ref", "ref_aux";
+
+ power-domains = <&gcc UFS_PHY_GDSC>;
+
+ resets = <&ufs_mem_hc 0>;
+ reset-names = "ufsphy";
+
+ vdda-phy-supply = <&vreg_l6b>;
+ vdda-pll-supply = <&vreg_l3b>;
+
+ #phy-cells = <0>;
+ };
diff --git a/Documentation/devicetree/bindings/phy/qcom,sc8280xp-qmp-usb3-uni-phy.yaml b/Documentation/devicetree/bindings/phy/qcom,sc8280xp-qmp-usb3-uni-phy.yaml
new file mode 100644
index 0000000..16fce10
--- /dev/null
+++ b/Documentation/devicetree/bindings/phy/qcom,sc8280xp-qmp-usb3-uni-phy.yaml
@@ -0,0 +1,102 @@
+# SPDX-License-Identifier: (GPL-2.0 OR BSD-2-Clause)
+%YAML 1.2
+---
+$id: http://devicetree.org/schemas/phy/qcom,sc8280xp-qmp-usb3-uni-phy.yaml#
+$schema: http://devicetree.org/meta-schemas/core.yaml#
+
+title: Qualcomm QMP PHY controller (USB, SC8280XP)
+
+maintainers:
+ - Vinod Koul <vkoul@kernel.org>
+
+description:
+ The QMP PHY controller supports physical layer functionality for a number of
+ controllers on Qualcomm chipsets, such as, PCIe, UFS, and USB.
+
+properties:
+ compatible:
+ enum:
+ - qcom,sc8280xp-qmp-usb3-uni-phy
+
+ reg:
+ maxItems: 1
+
+ clocks:
+ maxItems: 4
+
+ clock-names:
+ items:
+ - const: aux
+ - const: ref
+ - const: com_aux
+ - const: pipe
+
+ power-domains:
+ maxItems: 1
+
+ resets:
+ maxItems: 2
+
+ reset-names:
+ items:
+ - const: phy
+ - const: phy_phy
+
+ vdda-phy-supply: true
+
+ vdda-pll-supply: true
+
+ "#clock-cells":
+ const: 0
+
+ clock-output-names:
+ maxItems: 1
+
+ "#phy-cells":
+ const: 0
+
+required:
+ - compatible
+ - reg
+ - clocks
+ - clock-names
+ - power-domains
+ - resets
+ - reset-names
+ - vdda-phy-supply
+ - vdda-pll-supply
+ - "#clock-cells"
+ - clock-output-names
+ - "#phy-cells"
+
+additionalProperties: false
+
+examples:
+ - |
+ #include <dt-bindings/clock/qcom,gcc-sc8280xp.h>
+ #include <dt-bindings/clock/qcom,rpmh.h>
+
+ phy@88ef000 {
+ compatible = "qcom,sc8280xp-qmp-usb3-uni-phy";
+ reg = <0x088ef000 0x2000>;
+
+ clocks = <&gcc GCC_USB3_MP_PHY_AUX_CLK>,
+ <&gcc GCC_USB3_MP0_CLKREF_CLK>,
+ <&gcc GCC_USB3_MP_PHY_COM_AUX_CLK>,
+ <&gcc GCC_USB3_MP_PHY_PIPE_0_CLK>;
+ clock-names = "aux", "ref", "com_aux", "pipe";
+
+ power-domains = <&gcc USB30_MP_GDSC>;
+
+ resets = <&gcc GCC_USB3_UNIPHY_MP0_BCR>,
+ <&gcc GCC_USB3UNIPHY_PHY_MP0_BCR>;
+ reset-names = "phy", "phy_phy";
+
+ vdda-phy-supply = <&vreg_l3a>;
+ vdda-pll-supply = <&vreg_l5a>;
+
+ #clock-cells = <0>;
+ clock-output-names = "usb2_phy0_pipe_clk";
+
+ #phy-cells = <0>;
+ };
diff --git a/Documentation/devicetree/bindings/phy/qcom,sc8280xp-qmp-usb43dp-phy.yaml b/Documentation/devicetree/bindings/phy/qcom,sc8280xp-qmp-usb43dp-phy.yaml
new file mode 100644
index 0000000..6f31693
--- /dev/null
+++ b/Documentation/devicetree/bindings/phy/qcom,sc8280xp-qmp-usb43dp-phy.yaml
@@ -0,0 +1,99 @@
+# SPDX-License-Identifier: (GPL-2.0 OR BSD-2-Clause)
+%YAML 1.2
+---
+$id: http://devicetree.org/schemas/phy/qcom,sc8280xp-qmp-usb43dp-phy.yaml#
+$schema: http://devicetree.org/meta-schemas/core.yaml#
+
+title: Qualcomm QMP USB4-USB3-DP PHY controller (SC8280XP)
+
+maintainers:
+ - Vinod Koul <vkoul@kernel.org>
+
+description:
+ The QMP PHY controller supports physical layer functionality for a number of
+ controllers on Qualcomm chipsets, such as, PCIe, UFS and USB.
+
+properties:
+ compatible:
+ enum:
+ - qcom,sc8280xp-qmp-usb43dp-phy
+
+ reg:
+ maxItems: 1
+
+ clocks:
+ maxItems: 4
+
+ clock-names:
+ items:
+ - const: aux
+ - const: ref
+ - const: com_aux
+ - const: usb3_pipe
+
+ power-domains:
+ maxItems: 1
+
+ resets:
+ maxItems: 2
+
+ reset-names:
+ items:
+ - const: phy
+ - const: common
+
+ vdda-phy-supply: true
+
+ vdda-pll-supply: true
+
+ "#clock-cells":
+ const: 1
+ description:
+ See include/dt-bindings/dt-bindings/phy/phy-qcom-qmp.h
+
+ "#phy-cells":
+ const: 1
+ description:
+ See include/dt-bindings/dt-bindings/phy/phy-qcom-qmp.h
+
+required:
+ - compatible
+ - reg
+ - clocks
+ - clock-names
+ - power-domains
+ - resets
+ - reset-names
+ - vdda-phy-supply
+ - vdda-pll-supply
+ - "#clock-cells"
+ - "#phy-cells"
+
+additionalProperties: false
+
+examples:
+ - |
+ #include <dt-bindings/clock/qcom,gcc-sc8280xp.h>
+
+ phy@88eb000 {
+ compatible = "qcom,sc8280xp-qmp-usb43dp-phy";
+ reg = <0x088eb000 0x4000>;
+
+ clocks = <&gcc GCC_USB3_PRIM_PHY_AUX_CLK>,
+ <&gcc GCC_USB4_EUD_CLKREF_CLK>,
+ <&gcc GCC_USB3_PRIM_PHY_COM_AUX_CLK>,
+ <&gcc GCC_USB3_PRIM_PHY_PIPE_CLK>;
+ clock-names = "aux", "ref", "com_aux", "usb3_pipe";
+
+ power-domains = <&gcc USB30_PRIM_GDSC>;
+
+ resets = <&gcc GCC_USB3_PHY_PRIM_BCR>,
+ <&gcc GCC_USB4_DP_PHY_PRIM_BCR>;
+ reset-names = "phy", "common";
+
+ vdda-phy-supply = <&vreg_l9d>;
+ vdda-pll-supply = <&vreg_l4d>;
+
+ #clock-cells = <1>;
+ #phy-cells = <1>;
+ };
diff --git a/Documentation/devicetree/bindings/phy/renesas,r8a779f0-ether-serdes.yaml b/Documentation/devicetree/bindings/phy/renesas,r8a779f0-ether-serdes.yaml
new file mode 100644
index 0000000..93ab728
--- /dev/null
+++ b/Documentation/devicetree/bindings/phy/renesas,r8a779f0-ether-serdes.yaml
@@ -0,0 +1,54 @@
+# SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause)
+%YAML 1.2
+---
+$id: http://devicetree.org/schemas/phy/renesas,r8a779f0-ether-serdes.yaml#
+$schema: http://devicetree.org/meta-schemas/core.yaml#
+
+title: Renesas Ethernet SERDES
+
+maintainers:
+ - Yoshihiro Shimoda <yoshihiro.shimoda.uh@renesas.com>
+
+properties:
+ compatible:
+ const: renesas,r8a779f0-ether-serdes
+
+ reg:
+ maxItems: 1
+
+ clocks:
+ maxItems: 1
+
+ resets:
+ maxItems: 1
+
+ power-domains:
+ maxItems: 1
+
+ '#phy-cells':
+ description: Port number of SERDES.
+ const: 1
+
+required:
+ - compatible
+ - reg
+ - clocks
+ - resets
+ - power-domains
+ - '#phy-cells'
+
+additionalProperties: false
+
+examples:
+ - |
+ #include <dt-bindings/clock/r8a779f0-cpg-mssr.h>
+ #include <dt-bindings/power/r8a779f0-sysc.h>
+
+ phy@e6444000 {
+ compatible = "renesas,r8a779f0-ether-serdes";
+ reg = <0xe6444000 0xc00>;
+ clocks = <&cpg CPG_MOD 1506>;
+ power-domains = <&sysc R8A779F0_PD_ALWAYS_ON>;
+ resets = <&cpg 1506>;
+ #phy-cells = <1>;
+ };
diff --git a/Documentation/devicetree/bindings/phy/ti,phy-gmii-sel.yaml b/Documentation/devicetree/bindings/phy/ti,phy-gmii-sel.yaml
index 7bf5c42..6d46f57 100644
--- a/Documentation/devicetree/bindings/phy/ti,phy-gmii-sel.yaml
+++ b/Documentation/devicetree/bindings/phy/ti,phy-gmii-sel.yaml
@@ -54,6 +54,7 @@
- ti,dm814-phy-gmii-sel
- ti,am654-phy-gmii-sel
- ti,j7200-cpsw5g-phy-gmii-sel
+ - ti,j721e-cpsw9g-phy-gmii-sel
reg:
maxItems: 1
@@ -63,14 +64,17 @@
ti,qsgmii-main-ports:
$ref: /schemas/types.yaml#/definitions/uint32-array
description: |
- Required only for QSGMII mode. Array to select the port for
- QSGMII main mode. Rest of the ports are selected as QSGMII_SUB
- ports automatically. Any one of the 4 CPSW5G ports can act as the
- main port with the rest of them being the QSGMII_SUB ports.
- maxItems: 1
+ Required only for QSGMII mode. Array to select the port/s for QSGMII
+ main mode. The size of the array corresponds to the number of QSGMII
+ interfaces and thus, the number of distinct QSGMII main ports,
+ supported by the device. If the device supports two QSGMII interfaces
+ but only one QSGMII interface is desired, repeat the QSGMII main port
+ value corresponding to the QSGMII interface in the array.
+ minItems: 1
+ maxItems: 2
items:
minimum: 1
- maximum: 4
+ maximum: 8
allOf:
- if:
@@ -81,6 +85,8 @@
- ti,dra7xx-phy-gmii-sel
- ti,dm814-phy-gmii-sel
- ti,am654-phy-gmii-sel
+ - ti,j7200-cpsw5g-phy-gmii-sel
+ - ti,j721e-cpsw9g-phy-gmii-sel
then:
properties:
'#phy-cells':
@@ -88,12 +94,42 @@
description: CPSW port number (starting from 1)
- if:
+ properties:
+ compatible:
+ contains:
+ enum:
+ - ti,j7200-cpsw5g-phy-gmii-sel
+ then:
+ properties:
+ ti,qsgmii-main-ports:
+ maxItems: 1
+ items:
+ minimum: 1
+ maximum: 4
+
+ - if:
+ properties:
+ compatible:
+ contains:
+ enum:
+ - ti,j721e-cpsw9g-phy-gmii-sel
+ then:
+ properties:
+ ti,qsgmii-main-ports:
+ minItems: 2
+ maxItems: 2
+ items:
+ minimum: 1
+ maximum: 8
+
+ - if:
not:
properties:
compatible:
contains:
enum:
- ti,j7200-cpsw5g-phy-gmii-sel
+ - ti,j721e-cpsw9g-phy-gmii-sel
then:
properties:
ti,qsgmii-main-ports: false
diff --git a/Documentation/devicetree/bindings/phy/ti,phy-j721e-wiz.yaml b/Documentation/devicetree/bindings/phy/ti,phy-j721e-wiz.yaml
index 2225925..c54b36c 100644
--- a/Documentation/devicetree/bindings/phy/ti,phy-j721e-wiz.yaml
+++ b/Documentation/devicetree/bindings/phy/ti,phy-j721e-wiz.yaml
@@ -15,8 +15,10 @@
enum:
- ti,j721e-wiz-16g
- ti,j721e-wiz-10g
+ - ti,j721s2-wiz-10g
- ti,am64-wiz-10g
- ti,j7200-wiz-10g
+ - ti,j784s4-wiz-10g
power-domains:
maxItems: 1
diff --git a/Documentation/devicetree/bindings/power/reset/gpio-restart.yaml b/Documentation/devicetree/bindings/power/reset/gpio-restart.yaml
index a72d5c7..d3d18e0 100644
--- a/Documentation/devicetree/bindings/power/reset/gpio-restart.yaml
+++ b/Documentation/devicetree/bindings/power/reset/gpio-restart.yaml
@@ -25,6 +25,9 @@
inactive-delay, the GPIO is driven active again. After a delay specified by wait-delay, the
restart handler completes allowing other restart handlers to be attempted.
+allOf:
+ - $ref: restart-handler.yaml#
+
properties:
compatible:
const: gpio-restart
@@ -41,16 +44,6 @@
in its inactive state.
priority:
- $ref: /schemas/types.yaml#/definitions/uint32
- description: |
- A priority ranging from 0 to 255 (default 129) according to the following guidelines:
-
- 0: Restart handler of last resort, with limited restart capabilities.
- 128: Default restart handler; use if no other restart handler is expected to be available,
- and/or if restart functionality is sufficient to restart the entire system.
- 255: Highest priority restart handler, will preempt all other restart handlers.
- minimum: 0
- maximum: 255
default: 129
active-delay:
diff --git a/Documentation/devicetree/bindings/power/reset/restart-handler.yaml b/Documentation/devicetree/bindings/power/reset/restart-handler.yaml
new file mode 100644
index 0000000..1f9a2aa
--- /dev/null
+++ b/Documentation/devicetree/bindings/power/reset/restart-handler.yaml
@@ -0,0 +1,30 @@
+# SPDX-License-Identifier: GPL-2.0-only or BSD-2-Clause
+%YAML 1.2
+---
+$id: http://devicetree.org/schemas/power/reset/restart-handler.yaml#
+$schema: http://devicetree.org/meta-schemas/core.yaml#
+
+title: Restart and shutdown handler generic binding
+
+maintainers:
+ - Sebastian Reichel <sre@kernel.org>
+
+description:
+ Restart and shutdown handler device is responsible for powering off the
+ system, e.g. my cutting off the power. System might have several restart
+ handlers, which usually are tried from most precise to last resort.
+
+properties:
+ priority:
+ $ref: /schemas/types.yaml#/definitions/uint32
+ description: |
+ A priority ranging from 0 to 255 according to the following guidelines::
+ 0:: Restart handler of last resort, with limited restart capabilities.
+ 128:: Typical, default restart handler; use if no other restart handler
+ is expected to be available, and/or if restart functionality is
+ sufficient to restart the entire system.
+ 255:: Highest priority restart handler, will preempt all other restart handlers.
+ minimum: 0
+ maximum: 255
+
+additionalProperties: true
diff --git a/Documentation/devicetree/bindings/power/supply/bq25890.yaml b/Documentation/devicetree/bindings/power/supply/bq25890.yaml
index 6273b1c..dae27e9 100644
--- a/Documentation/devicetree/bindings/power/supply/bq25890.yaml
+++ b/Documentation/devicetree/bindings/power/supply/bq25890.yaml
@@ -15,11 +15,15 @@
properties:
compatible:
- enum:
- - ti,bq25890
- - ti,bq25892
- - ti,bq25895
- - ti,bq25896
+ oneOf:
+ - enum:
+ - ti,bq25890
+ - items:
+ - enum:
+ - ti,bq25892
+ - ti,bq25895
+ - ti,bq25896
+ - const: ti,bq25890
reg:
maxItems: 1
@@ -93,7 +97,7 @@
- ti,boost-voltage
- ti,boost-max-current
-additionalProperties: false
+unevaluatedProperties: false
examples:
- |
diff --git a/Documentation/devicetree/bindings/power/supply/bq27xxx.yaml b/Documentation/devicetree/bindings/power/supply/bq27xxx.yaml
index 65fc604..347d443 100644
--- a/Documentation/devicetree/bindings/power/supply/bq27xxx.yaml
+++ b/Documentation/devicetree/bindings/power/supply/bq27xxx.yaml
@@ -60,13 +60,11 @@
monitored-battery:
description: |
- phandle of battery characteristics node.
The fuel gauge uses the following battery properties:
- energy-full-design-microwatt-hours
- charge-full-design-microamp-hours
- voltage-min-design-microvolt
Both or neither of the *-full-design-*-hours properties must be set.
- See Documentation/devicetree/bindings/power/supply/battery.yaml
power-supplies: true
diff --git a/Documentation/devicetree/bindings/power/supply/ingenic,battery.yaml b/Documentation/devicetree/bindings/power/supply/ingenic,battery.yaml
index 534388b..741022b 100644
--- a/Documentation/devicetree/bindings/power/supply/ingenic,battery.yaml
+++ b/Documentation/devicetree/bindings/power/supply/ingenic,battery.yaml
@@ -10,6 +10,8 @@
maintainers:
- Artur Rojek <contact@artur-rojek.eu>
+$ref: power-supply.yaml#
+
properties:
compatible:
oneOf:
@@ -28,8 +30,6 @@
monitored-battery:
description: >
- phandle to a "simple-battery" compatible node.
-
This property must be a phandle to a node using the format described
in battery.yaml, with the following properties being required:
- voltage-min-design-microvolt: drained battery voltage,
diff --git a/Documentation/devicetree/bindings/power/supply/maxim,max17042.yaml b/Documentation/devicetree/bindings/power/supply/maxim,max17042.yaml
index aff5d07..64a0edb 100644
--- a/Documentation/devicetree/bindings/power/supply/maxim,max17042.yaml
+++ b/Documentation/devicetree/bindings/power/supply/maxim,max17042.yaml
@@ -59,6 +59,8 @@
Voltage threshold to report battery as over voltage (in mV).
Default is not to report over-voltage events.
+ power-supplies: true
+
required:
- compatible
- reg
diff --git a/Documentation/devicetree/bindings/power/supply/power-supply.yaml b/Documentation/devicetree/bindings/power/supply/power-supply.yaml
index 2f672e6..4e54c93 100644
--- a/Documentation/devicetree/bindings/power/supply/power-supply.yaml
+++ b/Documentation/devicetree/bindings/power/supply/power-supply.yaml
@@ -18,4 +18,10 @@
This property is added to a supply in order to list the devices which
supply it power, referenced by their phandles.
+ monitored-battery:
+ $ref: /schemas/types.yaml#/definitions/phandle
+ description:
+ The battery (with "simple-battery" compatible) being monitored by this
+ power supply.
+
additionalProperties: true
diff --git a/Documentation/devicetree/bindings/power/supply/rohm,bd99954.yaml b/Documentation/devicetree/bindings/power/supply/rohm,bd99954.yaml
index 24b0695..14d9b42 100644
--- a/Documentation/devicetree/bindings/power/supply/rohm,bd99954.yaml
+++ b/Documentation/devicetree/bindings/power/supply/rohm,bd99954.yaml
@@ -18,6 +18,7 @@
provides a Dual-source Battery Charger, two port BC1.2 detection and a
Battery Monitor.
+$ref: power-supply.yaml#
properties:
compatible:
diff --git a/Documentation/devicetree/bindings/power/supply/sc2731-charger.yaml b/Documentation/devicetree/bindings/power/supply/sc2731-charger.yaml
index a734481..a846a4d14 100644
--- a/Documentation/devicetree/bindings/power/supply/sc2731-charger.yaml
+++ b/Documentation/devicetree/bindings/power/supply/sc2731-charger.yaml
@@ -28,7 +28,6 @@
The charger uses the following battery properties
- charge-term-current-microamp: current for charge termination phase.
- constant-charge-voltage-max-microvolt: maximum constant input voltage.
- See Documentation/devicetree/bindings/power/supply/battery.yaml
additionalProperties: false
diff --git a/Documentation/devicetree/bindings/rtc/amlogic,meson6-rtc.yaml b/Documentation/devicetree/bindings/rtc/amlogic,meson6-rtc.yaml
new file mode 100644
index 0000000..8bf7d3a
--- /dev/null
+++ b/Documentation/devicetree/bindings/rtc/amlogic,meson6-rtc.yaml
@@ -0,0 +1,62 @@
+# SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause)
+%YAML 1.2
+---
+$id: http://devicetree.org/schemas/rtc/amlogic,meson6-rtc.yaml#
+$schema: http://devicetree.org/meta-schemas/core.yaml#
+
+title: Amlogic Meson6, Meson8, Meson8b and Meson8m2 RTC
+
+maintainers:
+ - Neil Armstrong <neil.armstrong@linaro.org>
+ - Martin Blumenstingl <martin.blumenstingl@googlemail.com>
+
+allOf:
+ - $ref: rtc.yaml#
+ - $ref: /schemas/nvmem/nvmem.yaml#
+
+properties:
+ compatible:
+ enum:
+ - amlogic,meson6-rtc
+ - amlogic,meson8-rtc
+ - amlogic,meson8b-rtc
+ - amlogic,meson8m2-rtc
+
+ reg:
+ maxItems: 1
+
+ clocks:
+ maxItems: 1
+
+ interrupts:
+ maxItems: 1
+
+ resets:
+ maxItems: 1
+
+ vdd-supply: true
+
+required:
+ - compatible
+ - reg
+
+unevaluatedProperties: false
+
+examples:
+ - |
+ #include <dt-bindings/interrupt-controller/irq.h>
+ #include <dt-bindings/interrupt-controller/arm-gic.h>
+ rtc: rtc@740 {
+ compatible = "amlogic,meson6-rtc";
+ reg = <0x740 0x14>;
+ interrupts = <GIC_SPI 72 IRQ_TYPE_EDGE_RISING>;
+ clocks = <&rtc32k_xtal>;
+ vdd-supply = <&rtc_vdd>;
+ resets = <&reset_rtc>;
+ #address-cells = <1>;
+ #size-cells = <1>;
+
+ mac@0 {
+ reg = <0 6>;
+ };
+ };
diff --git a/Documentation/devicetree/bindings/rtc/haoyu,hym8563.txt b/Documentation/devicetree/bindings/rtc/haoyu,hym8563.txt
deleted file mode 100644
index a8934fe..0000000
--- a/Documentation/devicetree/bindings/rtc/haoyu,hym8563.txt
+++ /dev/null
@@ -1,30 +0,0 @@
-Haoyu Microelectronics HYM8563 Real Time Clock
-
-The HYM8563 provides basic rtc and alarm functionality
-as well as a clock output of up to 32kHz.
-
-Required properties:
-- compatible: should be: "haoyu,hym8563"
-- reg: i2c address
-- #clock-cells: the value should be 0
-
-Optional properties:
-- clock-output-names: From common clock binding
-- interrupts: rtc alarm/event interrupt
-
-Example:
-
-hym8563: hym8563@51 {
- compatible = "haoyu,hym8563";
- reg = <0x51>;
-
- interrupts = <13 IRQ_TYPE_EDGE_FALLING>;
-
- #clock-cells = <0>;
-};
-
-device {
-...
- clocks = <&hym8563>;
-...
-};
diff --git a/Documentation/devicetree/bindings/rtc/haoyu,hym8563.yaml b/Documentation/devicetree/bindings/rtc/haoyu,hym8563.yaml
new file mode 100644
index 0000000..0b9f39e
--- /dev/null
+++ b/Documentation/devicetree/bindings/rtc/haoyu,hym8563.yaml
@@ -0,0 +1,56 @@
+# SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause)
+%YAML 1.2
+---
+$id: http://devicetree.org/schemas/rtc/haoyu,hym8563.yaml#
+$schema: http://devicetree.org/meta-schemas/core.yaml#
+
+title: Haoyu Microelectronics HYM8563 RTC
+
+maintainers:
+ - Alexandre Belloni <alexandre.belloni@bootlin.com>
+
+properties:
+ compatible:
+ const: haoyu,hym8563
+
+ reg:
+ maxItems: 1
+
+ interrupts:
+ maxItems: 1
+
+ "#clock-cells":
+ const: 0
+
+ clock-output-names:
+ description: From common clock binding to override the default output clock name.
+ maxItems: 1
+
+ wakeup-source:
+ description: Enables wake up of host system on alarm.
+
+allOf:
+ - $ref: rtc.yaml
+
+unevaluatedProperties: false
+
+required:
+ - compatible
+ - reg
+ - "#clock-cells"
+
+examples:
+ - |
+ #include <dt-bindings/interrupt-controller/irq.h>
+
+ i2c {
+ #address-cells = <1>;
+ #size-cells = <0>;
+
+ rtc@51 {
+ compatible = "haoyu,hym8563";
+ reg = <0x51>;
+ interrupts = <13 IRQ_TYPE_EDGE_FALLING>;
+ #clock-cells = <0>;
+ };
+ };
diff --git a/Documentation/devicetree/bindings/rtc/qcom-pm8xxx-rtc.yaml b/Documentation/devicetree/bindings/rtc/qcom-pm8xxx-rtc.yaml
index 23ab5bb..0a7aa29 100644
--- a/Documentation/devicetree/bindings/rtc/qcom-pm8xxx-rtc.yaml
+++ b/Documentation/devicetree/bindings/rtc/qcom-pm8xxx-rtc.yaml
@@ -11,12 +11,16 @@
properties:
compatible:
- enum:
- - qcom,pm8058-rtc
- - qcom,pm8921-rtc
- - qcom,pm8941-rtc
- - qcom,pm8018-rtc
- - qcom,pmk8350-rtc
+ oneOf:
+ - enum:
+ - qcom,pm8058-rtc
+ - qcom,pm8921-rtc
+ - qcom,pm8941-rtc
+ - qcom,pmk8350-rtc
+ - items:
+ - enum:
+ - qcom,pm8018-rtc
+ - const: qcom,pm8921-rtc
reg:
minItems: 1
diff --git a/Documentation/devicetree/bindings/rtc/rtc-m41t80.txt b/Documentation/devicetree/bindings/rtc/rtc-m41t80.txt
deleted file mode 100644
index cdd196b..0000000
--- a/Documentation/devicetree/bindings/rtc/rtc-m41t80.txt
+++ /dev/null
@@ -1,39 +0,0 @@
-ST M41T80 family of RTC and compatible
-
-Required properties:
-- compatible: should be one of:
- "st,m41t62",
- "st,m41t65",
- "st,m41t80",
- "st,m41t81",
- "st,m41t81s",
- "st,m41t82",
- "st,m41t83",
- "st,m41t84",
- "st,m41t85",
- "st,m41t87",
- "microcrystal,rv4162",
-- reg: I2C bus address of the device
-
-Optional properties:
-- interrupts: rtc alarm interrupt.
-- clock-output-names: From common clock binding to override the default output
- clock name
-- wakeup-source: Enables wake up of host system on alarm
-
-Optional child node:
-- clock: Provide this if the square wave pin is used as boot-enabled fixed clock.
-
-Example:
- rtc@68 {
- compatible = "st,m41t80";
- reg = <0x68>;
- interrupt-parent = <&UIC0>;
- interrupts = <0x9 0x8>;
-
- clock {
- compatible = "fixed-clock";
- #clock-cells = <0>;
- clock-frequency = <32768>;
- };
- };
diff --git a/Documentation/devicetree/bindings/rtc/rtc-meson.txt b/Documentation/devicetree/bindings/rtc/rtc-meson.txt
deleted file mode 100644
index e921fe6..0000000
--- a/Documentation/devicetree/bindings/rtc/rtc-meson.txt
+++ /dev/null
@@ -1,35 +0,0 @@
-* Amlogic Meson6, Meson8, Meson8b and Meson8m2 RTC
-
-Required properties:
-- compatible: should be one of the following describing the hardware:
- * "amlogic,meson6-rtc"
- * "amlogic,meson8-rtc"
- * "amlogic,meson8b-rtc"
- * "amlogic,meson8m2-rtc"
-
-- reg: physical register space for the controller's memory mapped registers.
-- interrupts: the interrupt line of the RTC block.
-- clocks: reference to the external 32.768kHz crystal oscillator.
-- vdd-supply: reference to the power supply of the RTC block.
-- resets: reset controller reference to allow reset of the controller
-
-Optional properties for the battery-backed non-volatile memory:
-- #address-cells: should be 1 to address the battery-backed non-volatile memory
-- #size-cells: should be 1 to reference the battery-backed non-volatile memory
-
-Optional child nodes:
-- see ../nvmem/nvmem.txt
-
-Example:
-
- rtc: rtc@740 {
- compatible = "amlogic,meson6-rtc";
- reg = <0x740 0x14>;
- interrupts = <GIC_SPI 72 IRQ_TYPE_EDGE_RISING>;
- clocks = <&rtc32k_xtal>;
- vdd-supply = <&rtc_vdd>;
- resets = <&reset RESET_RTC>;
-
- #address-cells = <1>;
- #size-cells = <1>;
- };
diff --git a/Documentation/devicetree/bindings/rtc/st,m41t80.yaml b/Documentation/devicetree/bindings/rtc/st,m41t80.yaml
new file mode 100644
index 0000000..fc9c6da
--- /dev/null
+++ b/Documentation/devicetree/bindings/rtc/st,m41t80.yaml
@@ -0,0 +1,73 @@
+# SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause)
+%YAML 1.2
+---
+$id: http://devicetree.org/schemas/rtc/st,m41t80.yaml#
+$schema: http://devicetree.org/meta-schemas/core.yaml#
+
+title: ST M41T80 family of RTC and compatible
+
+maintainers:
+ - Alexandre Belloni <alexandre.belloni@bootlin.com>
+
+properties:
+ compatible:
+ enum:
+ - st,m41t62
+ - st,m41t65
+ - st,m41t80
+ - st,m41t81
+ - st,m41t81s
+ - st,m41t82
+ - st,m41t83
+ - st,m41t84
+ - st,m41t85
+ - st,m41t87
+ - microcrystal,rv4162
+
+ reg:
+ maxItems: 1
+
+ interrupts:
+ maxItems: 1
+
+ "#clock-cells":
+ const: 1
+
+ clock-output-names:
+ maxItems: 1
+ description: From common clock binding to override the default output clock name.
+
+ clock:
+ type: object
+ $ref: /schemas/clock/fixed-clock.yaml#
+ properties:
+ clock-frequency:
+ const: 32768
+
+allOf:
+ - $ref: rtc.yaml
+
+unevaluatedProperties: false
+
+required:
+ - compatible
+ - reg
+
+examples:
+ - |
+ i2c {
+ #address-cells = <1>;
+ #size-cells = <0>;
+ rtc@68 {
+ compatible = "st,m41t80";
+ reg = <0x68>;
+ interrupt-parent = <&UIC0>;
+ interrupts = <0x9 0x8>;
+
+ clock {
+ compatible = "fixed-clock";
+ #clock-cells = <0>;
+ clock-frequency = <32768>;
+ };
+ };
+ };
diff --git a/Documentation/devicetree/bindings/soundwire/qcom,sdw.txt b/Documentation/devicetree/bindings/soundwire/qcom,sdw.txt
deleted file mode 100644
index c85c257..0000000
--- a/Documentation/devicetree/bindings/soundwire/qcom,sdw.txt
+++ /dev/null
@@ -1,214 +0,0 @@
-Qualcomm SoundWire Controller Bindings
-
-
-This binding describes the Qualcomm SoundWire Controller along with its
-board specific bus parameters.
-
-- compatible:
- Usage: required
- Value type: <stringlist>
- Definition: must be "qcom,soundwire-v<MAJOR>.<MINOR>.<STEP>",
- Example:
- "qcom,soundwire-v1.3.0"
- "qcom,soundwire-v1.5.0"
- "qcom,soundwire-v1.5.1"
- "qcom,soundwire-v1.6.0"
-- reg:
- Usage: required
- Value type: <prop-encoded-array>
- Definition: the base address and size of SoundWire controller
- address space.
-
-- interrupts:
- Usage: required
- Value type: <prop-encoded-array>
- Definition: should specify the SoundWire Controller core and optional
- wake IRQ
-
-- interrupt-names:
- Usage: Optional
- Value type: boolean
- Value type: <stringlist>
- Definition: should be "core" for core and "wakeup" for wake interrupt.
-
-- wakeup-source:
- Usage: Optional
- Value type: boolean
- Definition: should specify if SoundWire Controller is wake up capable.
-
-- clock-names:
- Usage: required
- Value type: <stringlist>
- Definition: should be "iface" for SoundWire Controller interface clock
-
-- clocks:
- Usage: required
- Value type: <prop-encoded-array>
- Definition: should specify the SoundWire Controller interface clock
-
-- #sound-dai-cells:
- Usage: required
- Value type: <u32>
- Definition: must be 1 for digital audio interfaces on the controller.
-
-- qcom,dout-ports:
- Usage: required
- Value type: <u32>
- Definition: must be count of data out ports
-
-- qcom,din-ports:
- Usage: required
- Value type: <u32>
- Definition: must be count of data in ports
-
-- qcom,ports-offset1:
- Usage: required
- Value type: <prop-encoded-array>
- Definition: should specify payload transport window offset1 of each
- data port. Out ports followed by In ports.
- Value of 0xFF indicates that this option is not implemented
- or applicable for the respective data port.
- More info in MIPI Alliance SoundWire 1.0 Specifications.
-
-- qcom,ports-offset2:
- Usage: required
- Value type: <prop-encoded-array>
- Definition: should specify payload transport window offset2 of each
- data port. Out ports followed by In ports.
- Value of 0xFF indicates that this option is not implemented
- or applicable for the respective data port.
- More info in MIPI Alliance SoundWire 1.0 Specifications.
-
-- qcom,ports-sinterval-low:
- Usage: required
- Value type: <prop-encoded-array>
- Definition: should be sample interval low of each data port.
- Out ports followed by In ports. Used for Sample Interval
- calculation.
- Value of 0xFF indicates that this option is not implemented
- or applicable for the respective data port.
- More info in MIPI Alliance SoundWire 1.0 Specifications.
-
-- qcom,ports-word-length:
- Usage: optional
- Value type: <prop-encoded-array>
- Definition: should be size of payload channel sample.
- Value of 0xFF indicates that this option is not implemented
- or applicable for the respective data port.
- More info in MIPI Alliance SoundWire 1.0 Specifications.
-
-- qcom,ports-block-pack-mode:
- Usage: optional
- Value type: <prop-encoded-array>
- Definition: should be 0 or 1 to indicate the block packing mode.
- 0 to indicate Blocks are per Channel
- 1 to indicate Blocks are per Port.
- Out ports followed by In ports.
- Value of 0xFF indicates that this option is not implemented
- or applicable for the respective data port.
- More info in MIPI Alliance SoundWire 1.0 Specifications.
-
-- qcom,ports-block-group-count:
- Usage: optional
- Value type: <prop-encoded-array>
- Definition: should be in range 1 to 4 to indicate how many sample
- intervals are combined into a payload.
- Out ports followed by In ports.
- Value of 0xFF indicates that this option is not implemented
- or applicable for the respective data port.
- More info in MIPI Alliance SoundWire 1.0 Specifications.
-
-- qcom,ports-lane-control:
- Usage: optional
- Value type: <prop-encoded-array>
- Definition: should be in range 0 to 7 to identify which data lane
- the data port uses.
- Out ports followed by In ports.
- Value of 0xFF indicates that this option is not implemented
- or applicable for the respective data port.
- More info in MIPI Alliance SoundWire 1.0 Specifications.
-
-- qcom,ports-hstart:
- Usage: optional
- Value type: <prop-encoded-array>
- Definition: should be number identifying lowerst numbered coloum in
- SoundWire Frame, i.e. left edge of the Transport sub-frame
- for each port. Values between 0 and 15 are valid.
- Out ports followed by In ports.
- Value of 0xFF indicates that this option is not implemented
- or applicable for the respective data port.
- More info in MIPI Alliance SoundWire 1.0 Specifications.
-
-- qcom,ports-hstop:
- Usage: optional
- Value type: <prop-encoded-array>
- Definition: should be number identifying highest numbered coloum in
- SoundWire Frame, i.e. the right edge of the Transport
- sub-frame for each port. Values between 0 and 15 are valid.
- Out ports followed by In ports.
- Value of 0xFF indicates that this option is not implemented
- or applicable for the respective data port.
- More info in MIPI Alliance SoundWire 1.0 Specifications.
-
-- qcom,dports-type:
- Usage: optional
- Value type: <prop-encoded-array>
- Definition: should be one of the following types
- 0 for reduced port
- 1 for simple ports
- 2 for full port
- Out ports followed by In ports.
- Value of 0xFF indicates that this option is not implemented
- or applicable for the respective data port.
- More info in MIPI Alliance SoundWire 1.0 Specifications.
-
-- reset:
- Usage: optional
- Value type: <prop-encoded-array>
- Definition: Should specify the SoundWire audio CSR reset controller interface,
- which is required for SoundWire version 1.6.0 and above.
-
-- reset-names:
- Usage: optional
- Value type: <stringlist>
- Definition: should be "swr_audio_cgcr" for SoundWire audio CSR reset
- controller interface.
-
-Note:
- More Information on detail of encoding of these fields can be
-found in MIPI Alliance SoundWire 1.0 Specifications.
-
-= SoundWire devices
-Each subnode of the bus represents SoundWire device attached to it.
-The properties of these nodes are defined by the individual bindings.
-
-= EXAMPLE
-The following example represents a SoundWire controller on DB845c board
-which has controller integrated inside WCD934x codec on SDM845 SoC.
-
-soundwire: soundwire@c85 {
- compatible = "qcom,soundwire-v1.3.0";
- reg = <0xc85 0x20>;
- interrupts = <20 IRQ_TYPE_EDGE_RISING>;
- clocks = <&wcc>;
- clock-names = "iface";
- resets = <&lpass_audiocc LPASS_AUDIO_SWR_TX_CGCR>;
- reset-names = "swr_audio_cgcr";
- #sound-dai-cells = <1>;
- qcom,dports-type = <0>;
- qcom,dout-ports = <6>;
- qcom,din-ports = <2>;
- qcom,ports-sinterval-low = /bits/ 8 <0x07 0x1F 0x3F 0x7 0x1F 0x3F 0x0F 0x0F>;
- qcom,ports-offset1 = /bits/ 8 <0x01 0x02 0x0C 0x6 0x12 0x0D 0x07 0x0A >;
- qcom,ports-offset2 = /bits/ 8 <0x00 0x00 0x1F 0x00 0x00 0x1F 0x00 0x00>;
-
- /* Left Speaker */
- left{
- ....
- };
-
- /* Right Speaker */
- right{
- ....
- };
-};
diff --git a/Documentation/devicetree/bindings/soundwire/qcom,soundwire.yaml b/Documentation/devicetree/bindings/soundwire/qcom,soundwire.yaml
new file mode 100644
index 0000000..bcbfa71
--- /dev/null
+++ b/Documentation/devicetree/bindings/soundwire/qcom,soundwire.yaml
@@ -0,0 +1,270 @@
+# SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause)
+%YAML 1.2
+---
+$id: http://devicetree.org/schemas/soundwire/qcom,soundwire.yaml#
+$schema: http://devicetree.org/meta-schemas/core.yaml#
+
+title: Qualcomm SoundWire Controller
+
+maintainers:
+ - Srinivas Kandagatla <srinivas.kandagatla@linaro.org>
+ - Srinivasa Rao Mandadapu <quic_srivasam@quicinc.com>
+
+description:
+ The Qualcomm SoundWire controller along with its board specific bus parameters.
+
+properties:
+ compatible:
+ enum:
+ - qcom,soundwire-v1.3.0
+ - qcom,soundwire-v1.5.0
+ - qcom,soundwire-v1.5.1
+ - qcom,soundwire-v1.6.0
+ - qcom,soundwire-v1.7.0
+
+ reg:
+ maxItems: 1
+
+ interrupts:
+ minItems: 1
+ items:
+ - description: specify the SoundWire controller core.
+ - description: specify the Soundwire controller wake IRQ.
+
+ interrupt-names:
+ minItems: 1
+ items:
+ - const: core
+ - const: wakeup
+
+ clocks:
+ items:
+ - description: iface clock
+
+ clock-names:
+ items:
+ - const: iface
+
+ resets:
+ items:
+ - description: SWR_AUDIO_CGCR RESET
+
+ reset-names:
+ items:
+ - const: swr_audio_cgcr
+
+ '#sound-dai-cells':
+ const: 1
+
+ '#address-cells':
+ const: 2
+
+ '#size-cells':
+ const: 0
+
+ wakeup-source: true
+
+ qcom,din-ports:
+ $ref: /schemas/types.yaml#/definitions/uint32
+ description: count of data in ports
+
+ qcom,dout-ports:
+ $ref: /schemas/types.yaml#/definitions/uint32
+ description: count of data out ports
+
+ qcom,ports-word-length:
+ $ref: /schemas/types.yaml#/definitions/uint8-array
+ description:
+ Size of payload channel sample.
+ Value of 0xff indicates that this option is not implemented
+ or applicable for the respective data port.
+ More info in MIPI Alliance SoundWire 1.0 Specifications.
+ minItems: 3
+ maxItems: 5
+
+ qcom,ports-sinterval-low:
+ $ref: /schemas/types.yaml#/definitions/uint8-array
+ description:
+ Sample interval low of each data port.
+ Out ports followed by In ports. Used for Sample Interval calculation.
+ Value of 0xff indicates that this option is not implemented
+ or applicable for the respective data port.
+ More info in MIPI Alliance SoundWire 1.0 Specifications.
+ minItems: 3
+ maxItems: 8
+
+ qcom,ports-offset1:
+ $ref: /schemas/types.yaml#/definitions/uint8-array
+ description:
+ Payload transport window offset1 of each data port.
+ Out ports followed by In ports.
+ Value of 0xff indicates that this option is not implemented
+ or applicable for the respective data port.
+ More info in MIPI Alliance SoundWire 1.0 Specifications.
+ minItems: 3
+ maxItems: 8
+
+ qcom,ports-offset2:
+ $ref: /schemas/types.yaml#/definitions/uint8-array
+ description:
+ Payload transport window offset2 of each data port.
+ Out ports followed by In ports.
+ Value of 0xff indicates that this option is not implemented
+ or applicable for the respective data port.
+ More info in MIPI Alliance SoundWire 1.0 Specifications.
+ minItems: 3
+ maxItems: 8
+
+ qcom,ports-lane-control:
+ $ref: /schemas/types.yaml#/definitions/uint8-array
+ description:
+ Identify which data lane the data port uses.
+ Out ports followed by In ports.
+ Value of 0xff indicates that this option is not implemented
+ or applicable for the respective data port.
+ More info in MIPI Alliance SoundWire 1.0 Specifications.
+ minItems: 3
+ maxItems: 5
+
+ qcom,ports-block-pack-mode:
+ $ref: /schemas/types.yaml#/definitions/uint8-array
+ description:
+ Indicate the block packing mode.
+ 0 to indicate Blocks are per Channel
+ 1 to indicate Blocks are per Port.
+ Out ports followed by In ports.
+ Value of 0xff indicates that this option is not implemented
+ or applicable for the respective data port.
+ More info in MIPI Alliance SoundWire 1.0 Specifications.
+ minItems: 3
+ maxItems: 8
+ items:
+ oneOf:
+ - minimum: 0
+ maximum: 1
+ - const: 0xff
+
+ qcom,ports-hstart:
+ $ref: /schemas/types.yaml#/definitions/uint8-array
+ description:
+ Identifying lowerst numbered coloum in SoundWire Frame,
+ i.e. left edge of the Transport sub-frame for each port.
+ Out ports followed by In ports.
+ Value of 0xff indicates that this option is not implemented
+ or applicable for the respective data port.
+ More info in MIPI Alliance SoundWire 1.0 Specifications.
+ minItems: 3
+ maxItems: 5
+ items:
+ oneOf:
+ - minimum: 0
+ maximum: 15
+ - const: 0xff
+
+ qcom,ports-hstop:
+ $ref: /schemas/types.yaml#/definitions/uint8-array
+ description:
+ Identifying highest numbered coloum in SoundWire Frame,
+ i.e. the right edge of the Transport
+ sub-frame for each port. Out ports followed by In ports.
+ Value of 0xff indicates that this option is not implemented
+ or applicable for the respective data port.
+ More info in MIPI Alliance SoundWire 1.0 Specifications.
+ minItems: 3
+ maxItems: 5
+ items:
+ oneOf:
+ - minimum: 0
+ maximum: 15
+ - const: 0xff
+
+ qcom,ports-block-group-count:
+ $ref: /schemas/types.yaml#/definitions/uint8-array
+ description:
+ In range 1 to 4 to indicate how many sample intervals are combined
+ into a payload. Out ports followed by In ports.
+ Value of 0xff indicates that this option is not implemented
+ or applicable for the respective data port.
+ More info in MIPI Alliance SoundWire 1.0 Specifications.
+ minItems: 3
+ maxItems: 5
+ items:
+ oneOf:
+ - minimum: 0
+ maximum: 4
+ - const: 0xff
+
+ label:
+ maxItems: 1
+
+patternProperties:
+ "^.*@[0-9a-f],[0-9a-f]$":
+ type: object
+ description:
+ Child nodes for a standalone audio codec or speaker amplifier IC.
+ It has RX and TX Soundwire secondary devices.
+ properties:
+ compatible:
+ pattern: "^sdw[0-9a-f]{1}[0-9a-f]{4}[0-9a-f]{4}[0-9a-f]{2}$"
+
+required:
+ - compatible
+ - reg
+ - interrupts
+ - clocks
+ - clock-names
+ - '#sound-dai-cells'
+ - '#address-cells'
+ - '#size-cells'
+ - qcom,dout-ports
+ - qcom,din-ports
+ - qcom,ports-sinterval-low
+ - qcom,ports-offset1
+ - qcom,ports-offset2
+
+additionalProperties: false
+
+examples:
+ - |
+ #include <dt-bindings/interrupt-controller/arm-gic.h>
+ #include <dt-bindings/interrupt-controller/irq.h>
+ #include <dt-bindings/clock/qcom,lpassaudiocc-sc7280.h>
+
+ soundwire@3210000 {
+ compatible = "qcom,soundwire-v1.6.0";
+ reg = <0x03210000 0x2000>;
+
+ interrupts = <GIC_SPI 155 IRQ_TYPE_LEVEL_HIGH>,
+ <&pdc 130 IRQ_TYPE_LEVEL_HIGH>;
+
+ interrupt-names = "core", "wakeup";
+
+ clocks = <&lpass_rx_macro>;
+ clock-names = "iface";
+
+ qcom,din-ports = <0>;
+ qcom,dout-ports = <5>;
+
+ resets = <&lpass_audiocc LPASS_AUDIO_SWR_RX_CGCR>;
+ reset-names = "swr_audio_cgcr";
+
+ qcom,ports-word-length = /bits/ 8 <0x01 0x07 0x04 0xff 0xff>;
+ qcom,ports-sinterval-low = /bits/ 8 <0x03 0x3f 0x1f 0x03 0x03>;
+ qcom,ports-offset1 = /bits/ 8 <0x00 0x00 0x0b 0x01 0x01>;
+ qcom,ports-offset2 = /bits/ 8 <0x00 0x00 0x0b 0x00 0x00>;
+ qcom,ports-lane-control = /bits/ 8 <0x01 0x00 0x00 0x00 0x00>;
+ qcom,ports-block-pack-mode = /bits/ 8 <0xff 0x00 0x01 0xff 0xff>;
+ qcom,ports-hstart = /bits/ 8 <0xff 0x03 0xff 0xff 0xff>;
+ qcom,ports-hstop = /bits/ 8 <0xff 0x06 0xff 0xff 0xff>;
+ qcom,ports-block-group-count = /bits/ 8 <0xff 0xff 0xff 0xff 0x00>;
+
+ #sound-dai-cells = <1>;
+ #address-cells = <2>;
+ #size-cells = <0>;
+
+ codec@0,4 {
+ compatible = "sdw20217010d00";
+ reg = <0 4>;
+ qcom,rx-port-mapping = <1 2 3 4 5>;
+ };
+ };
diff --git a/Documentation/devicetree/bindings/watchdog/mediatek,mtk-wdt.yaml b/Documentation/devicetree/bindings/watchdog/mediatek,mtk-wdt.yaml
new file mode 100644
index 0000000..b360560
--- /dev/null
+++ b/Documentation/devicetree/bindings/watchdog/mediatek,mtk-wdt.yaml
@@ -0,0 +1,80 @@
+# SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause)
+%YAML 1.2
+---
+$id: http://devicetree.org/schemas/watchdog/mediatek,mtk-wdt.yaml#
+$schema: http://devicetree.org/meta-schemas/core.yaml#
+
+title: MediaTek SoCs Watchdog timer
+
+maintainers:
+ - Matthias Brugger <matthias.bgg@gmail.com>
+
+description:
+ The watchdog supports a pre-timeout interrupt that fires
+ timeout-sec/2 before the expiry.
+
+allOf:
+ - $ref: watchdog.yaml#
+
+properties:
+ compatible:
+ oneOf:
+ - enum:
+ - mediatek,mt2712-wdt
+ - mediatek,mt6589-wdt
+ - mediatek,mt6795-wdt
+ - mediatek,mt7986-wdt
+ - mediatek,mt8183-wdt
+ - mediatek,mt8186-wdt
+ - mediatek,mt8188-wdt
+ - mediatek,mt8192-wdt
+ - mediatek,mt8195-wdt
+ - items:
+ - enum:
+ - mediatek,mt2701-wdt
+ - mediatek,mt6582-wdt
+ - mediatek,mt6797-wdt
+ - mediatek,mt7622-wdt
+ - mediatek,mt7623-wdt
+ - mediatek,mt7629-wdt
+ - mediatek,mt8173-wdt
+ - mediatek,mt8516-wdt
+ - const: mediatek,mt6589-wdt
+
+ reg:
+ maxItems: 1
+
+ interrupts:
+ items:
+ - description: Watchdog pre-timeout (bark) interrupt
+
+ mediatek,disable-extrst:
+ description: Disable sending output reset signal
+ type: boolean
+
+ '#reset-cells':
+ const: 1
+
+required:
+ - compatible
+ - reg
+
+unevaluatedProperties: false
+
+examples:
+ - |
+ #include <dt-bindings/interrupt-controller/arm-gic.h>
+
+ soc {
+ #address-cells = <2>;
+ #size-cells = <2>;
+
+ watchdog: watchdog@10007000 {
+ compatible = "mediatek,mt8183-wdt";
+ reg = <0 0x10007000 0 0x100>;
+ interrupts = <GIC_SPI 139 IRQ_TYPE_LEVEL_LOW>;
+ mediatek,disable-extrst;
+ timeout-sec = <10>;
+ #reset-cells = <1>;
+ };
+ };
diff --git a/Documentation/devicetree/bindings/watchdog/mtk-wdt.txt b/Documentation/devicetree/bindings/watchdog/mtk-wdt.txt
deleted file mode 100644
index 762c62e..0000000
--- a/Documentation/devicetree/bindings/watchdog/mtk-wdt.txt
+++ /dev/null
@@ -1,42 +0,0 @@
-Mediatek SoCs Watchdog timer
-
-The watchdog supports a pre-timeout interrupt that fires timeout-sec/2
-before the expiry.
-
-Required properties:
-
-- compatible should contain:
- "mediatek,mt2701-wdt", "mediatek,mt6589-wdt": for MT2701
- "mediatek,mt2712-wdt": for MT2712
- "mediatek,mt6582-wdt", "mediatek,mt6589-wdt": for MT6582
- "mediatek,mt6589-wdt": for MT6589
- "mediatek,mt6797-wdt", "mediatek,mt6589-wdt": for MT6797
- "mediatek,mt7622-wdt", "mediatek,mt6589-wdt": for MT7622
- "mediatek,mt7623-wdt", "mediatek,mt6589-wdt": for MT7623
- "mediatek,mt7629-wdt", "mediatek,mt6589-wdt": for MT7629
- "mediatek,mt7986-wdt", "mediatek,mt6589-wdt": for MT7986
- "mediatek,mt8183-wdt": for MT8183
- "mediatek,mt8186-wdt", "mediatek,mt6589-wdt": for MT8186
- "mediatek,mt8516-wdt", "mediatek,mt6589-wdt": for MT8516
- "mediatek,mt8192-wdt": for MT8192
- "mediatek,mt8195-wdt", "mediatek,mt6589-wdt": for MT8195
-
-- reg : Specifies base physical address and size of the registers.
-
-Optional properties:
-- mediatek,disable-extrst: disable send output reset signal
-- interrupts: Watchdog pre-timeout (bark) interrupt.
-- timeout-sec: contains the watchdog timeout in seconds.
-- #reset-cells: Should be 1.
-
-Example:
-
-watchdog: watchdog@10007000 {
- compatible = "mediatek,mt8183-wdt",
- "mediatek,mt6589-wdt";
- mediatek,disable-extrst;
- reg = <0 0x10007000 0 0x100>;
- interrupts = <GIC_SPI 139 IRQ_TYPE_NONE>;
- timeout-sec = <10>;
- #reset-cells = <1>;
-};
diff --git a/Documentation/driver-api/driver-model/devres.rst b/Documentation/driver-api/driver-model/devres.rst
index d2ef9fb..4249eb4 100644
--- a/Documentation/driver-api/driver-model/devres.rst
+++ b/Documentation/driver-api/driver-model/devres.rst
@@ -450,6 +450,7 @@
SLAVE DMA ENGINE
devm_acpi_dma_controller_register()
+ devm_acpi_dma_controller_free()
SPI
devm_spi_alloc_master()
diff --git a/Documentation/driver-api/phy/phy.rst b/Documentation/driver-api/phy/phy.rst
index 8fc1ce0..8e8b3e8 100644
--- a/Documentation/driver-api/phy/phy.rst
+++ b/Documentation/driver-api/phy/phy.rst
@@ -94,7 +94,8 @@
can use phy_set_drvdata() after creating the PHY and use phy_get_drvdata() in
phy_ops to get back the private data.
-4. Getting a reference to the PHY
+Getting a reference to the PHY
+==============================
Before the controller can make use of the PHY, it has to get a reference to
it. This framework provides the following APIs to get a reference to the PHY.
@@ -130,6 +131,28 @@
phy_power_off() calls are all NOP when applied to a NULL phy. The NULL
phy is useful in devices for handling optional phy devices.
+Order of API calls
+==================
+
+The general order of calls should be::
+
+ [devm_][of_]phy_get()
+ phy_init()
+ phy_power_on()
+ [phy_set_mode[_ext]()]
+ ...
+ phy_power_off()
+ phy_exit()
+ [[of_]phy_put()]
+
+Some PHY drivers may not implement :c:func:`phy_init` or :c:func:`phy_power_on`,
+but controllers should always call these functions to be compatible with other
+PHYs. Some PHYs may require :c:func:`phy_set_mode <phy_set_mode_ext>`, while
+others may use a default mode (typically configured via devicetree or other
+firmware). For compatibility, you should always call this function if you know
+what mode you will be using. Generally, this function should be called after
+:c:func:`phy_power_on`, although some PHY drivers may allow it at any time.
+
Releasing a reference to the PHY
================================
diff --git a/Documentation/powerpc/cpu_families.rst b/Documentation/powerpc/cpu_families.rst
index 9b84e04..eb7e606 100644
--- a/Documentation/powerpc/cpu_families.rst
+++ b/Documentation/powerpc/cpu_families.rst
@@ -10,6 +10,7 @@
------------------
- Hash MMU (except 603 and e300)
+- Radix MMU (POWER9 and later)
- Software loaded TLB (603 and e300)
- Selectable Software loaded TLB in addition to hash MMU (755, 7450, e600)
- Mix of 32 & 64 bit::
@@ -101,6 +102,18 @@
+--------------+
| POWER8 |
+--------------+
+ |
+ |
+ v
+ +--------------+
+ | POWER9 |
+ +--------------+
+ |
+ |
+ v
+ +--------------+
+ | POWER10 |
+ +--------------+
+---------------+
diff --git a/Documentation/process/changes.rst b/Documentation/process/changes.rst
index 9844ca3..ef54086 100644
--- a/Documentation/process/changes.rst
+++ b/Documentation/process/changes.rst
@@ -35,7 +35,7 @@
bindgen (optional) 0.56.0 bindgen --version
GNU make 3.82 make --version
bash 4.2 bash --version
-binutils 2.23 ld -v
+binutils 2.25 ld -v
flex 2.5.35 flex --version
bison 2.0 bison --version
pahole 1.16 pahole --version
@@ -119,7 +119,7 @@
Binutils
--------
-Binutils 2.23 or newer is needed to build the kernel.
+Binutils 2.25 or newer is needed to build the kernel.
pkg-config
----------
diff --git a/MAINTAINERS b/MAINTAINERS
index be45419..7f0b718 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -2330,6 +2330,7 @@
F: Documentation/devicetree/bindings/gpio/intel,ixp4xx-gpio.txt
F: Documentation/devicetree/bindings/interrupt-controller/intel,ixp4xx-interrupt.yaml
F: Documentation/devicetree/bindings/timer/intel,ixp4xx-timer.yaml
+F: arch/arm/boot/dts/intel-ixp*
F: arch/arm/mach-ixp4xx/
F: drivers/bus/intel-ixp4xx-eb.c
F: drivers/clocksource/timer-ixp4xx.c
@@ -10460,11 +10461,6 @@
F: drivers/iommu/intel/
F: include/linux/intel-svm.h
-INTEL IOP-ADMA DMA DRIVER
-R: Dan Williams <dan.j.williams@intel.com>
-S: Odd fixes
-F: drivers/dma/iop-adma.c
-
INTEL IPU3 CSI-2 CIO2 DRIVER
M: Yong Zhi <yong.zhi@intel.com>
M: Sakari Ailus <sakari.ailus@linux.intel.com>
@@ -11732,11 +11728,13 @@
LED SUBSYSTEM
M: Pavel Machek <pavel@ucw.cz>
+M: Lee Jones <lee@kernel.org>
L: linux-leds@vger.kernel.org
S: Maintained
T: git git://git.kernel.org/pub/scm/linux/kernel/git/pavel/linux-leds.git
F: Documentation/devicetree/bindings/leds/
F: drivers/leds/
+F: include/dt-bindings/leds/
F: include/linux/leds.h
LEGACY EEPROM DRIVER
@@ -13627,7 +13625,6 @@
S: Supported
F: Documentation/devicetree/bindings/dma/atmel-dma.txt
F: drivers/dma/at_hdmac.c
-F: drivers/dma/at_hdmac_regs.h
F: drivers/dma/at_xdmac.c
F: include/dt-bindings/dma/at91.h
@@ -16431,7 +16428,7 @@
F: tools/perf/
PERFORMANCE EVENTS TOOLING ARM64
-R: John Garry <john.garry@huawei.com>
+R: John Garry <john.g.garry@oracle.com>
R: Will Deacon <will@kernel.org>
R: James Clark <james.clark@arm.com>
R: Mike Leach <mike.leach@linaro.org>
@@ -23053,8 +23050,7 @@
ZRAM COMPRESSED RAM BLOCK DEVICE DRVIER
M: Minchan Kim <minchan@kernel.org>
-M: Nitin Gupta <ngupta@vflare.org>
-R: Sergey Senozhatsky <senozhatsky@chromium.org>
+M: Sergey Senozhatsky <senozhatsky@chromium.org>
L: linux-kernel@vger.kernel.org
S: Maintained
F: Documentation/admin-guide/blockdev/zram.rst
@@ -23067,8 +23063,7 @@
ZSMALLOC COMPRESSED SLAB MEMORY ALLOCATOR
M: Minchan Kim <minchan@kernel.org>
-M: Nitin Gupta <ngupta@vflare.org>
-R: Sergey Senozhatsky <senozhatsky@chromium.org>
+M: Sergey Senozhatsky <senozhatsky@chromium.org>
L: linux-mm@kvack.org
S: Maintained
F: Documentation/mm/zsmalloc.rst
@@ -23079,7 +23074,7 @@
M: Nick Terrell <terrelln@fb.com>
S: Maintained
B: https://github.com/facebook/zstd/issues
-T: git git://github.com/terrelln/linux.git
+T: git https://github.com/terrelln/linux.git
F: include/linux/zstd*
F: lib/zstd/
F: lib/decompress_unzstd.c
diff --git a/Makefile b/Makefile
index 6aa709d..25247f9 100644
--- a/Makefile
+++ b/Makefile
@@ -11,6 +11,10 @@
# Comments in this file are targeted only to the developer, do not
# expect to learn how to build the kernel reading this file.
+ifeq ($(filter undefine,$(.FEATURES)),)
+$(error GNU Make >= 3.82 is required. Your Make version is $(MAKE_VERSION))
+endif
+
$(if $(filter __%, $(MAKECMDGOALS)), \
$(error targets prefixed with '__' are only for internal use))
@@ -93,10 +97,17 @@
# If the user is running make -s (silent mode), suppress echoing of
# commands
+# make-4.0 (and later) keep single letter options in the 1st word of MAKEFLAGS.
-ifneq ($(findstring s,$(filter-out --%,$(MAKEFLAGS))),)
- quiet=silent_
- KBUILD_VERBOSE = 0
+ifeq ($(filter 3.%,$(MAKE_VERSION)),)
+silence:=$(findstring s,$(firstword -$(MAKEFLAGS)))
+else
+silence:=$(findstring s,$(filter-out --%,$(MAKEFLAGS)))
+endif
+
+ifeq ($(silence),s)
+quiet=silent_
+KBUILD_VERBOSE = 0
endif
export quiet Q KBUILD_VERBOSE
@@ -369,7 +380,7 @@
include $(srctree)/scripts/Kbuild.include
# Read KERNELRELEASE from include/config/kernel.release (if it exists)
-KERNELRELEASE = $(shell cat include/config/kernel.release 2> /dev/null)
+KERNELRELEASE = $(call read-file, include/config/kernel.release)
KERNELVERSION = $(VERSION)$(if $(PATCHLEVEL),.$(PATCHLEVEL)$(if $(SUBLEVEL),.$(SUBLEVEL)))$(EXTRAVERSION)
export VERSION PATCHLEVEL SUBLEVEL KERNELRELEASE KERNELVERSION
@@ -859,7 +870,8 @@
KBUILD_CFLAGS += $(stackp-flags-y)
-KBUILD_CFLAGS-$(CONFIG_WERROR) += -Werror
+KBUILD_CPPFLAGS-$(CONFIG_WERROR) += -Werror
+KBUILD_CPPFLAGS += $(KBUILD_CPPFLAGS-y)
KBUILD_CFLAGS-$(CONFIG_CC_NO_ARRAY_BOUNDS) += -Wno-array-bounds
KBUILD_RUSTFLAGS-$(CONFIG_WERROR) += -Dwarnings
@@ -933,7 +945,9 @@
endif
endif
ifdef CONFIG_FTRACE_MCOUNT_USE_OBJTOOL
- CC_FLAGS_USING += -DCC_USING_NOP_MCOUNT
+ ifdef CONFIG_HAVE_OBJTOOL_NOP_MCOUNT
+ CC_FLAGS_USING += -DCC_USING_NOP_MCOUNT
+ endif
endif
ifdef CONFIG_FTRACE_MCOUNT_USE_RECORDMCOUNT
ifdef CONFIG_HAVE_C_RECORDMCOUNT
@@ -988,7 +1002,7 @@
# Check for frame size exceeding threshold during prolog/epilog insertion
# when using lld < 13.0.0.
ifneq ($(CONFIG_FRAME_WARN),0)
-ifeq ($(shell test $(CONFIG_LLD_VERSION) -lt 130000; echo $$?),0)
+ifeq ($(call test-lt, $(CONFIG_LLD_VERSION), 130000),y)
KBUILD_LDFLAGS += -plugin-opt=-warn-stack-size=$(CONFIG_FRAME_WARN)
endif
endif
@@ -1558,7 +1572,7 @@
rm -f $(MODLIB)/build ; \
ln -s $(CURDIR) $(MODLIB)/build ; \
fi
- @sed 's:^:kernel/:' modules.order > $(MODLIB)/modules.order
+ @sed 's:^\(.*\)\.o$$:kernel/\1.ko:' modules.order > $(MODLIB)/modules.order
@cp -f modules.builtin $(MODLIB)/
@cp -f $(objtree)/modules.builtin.modinfo $(MODLIB)/
diff --git a/arch/Kconfig b/arch/Kconfig
index a3c47c2..12e3dda 100644
--- a/arch/Kconfig
+++ b/arch/Kconfig
@@ -638,7 +638,7 @@
config SHADOW_CALL_STACK
bool "Shadow Call Stack"
depends on ARCH_SUPPORTS_SHADOW_CALL_STACK
- depends on DYNAMIC_FTRACE_WITH_REGS || !FUNCTION_GRAPH_TRACER
+ depends on DYNAMIC_FTRACE_WITH_ARGS || DYNAMIC_FTRACE_WITH_REGS || !FUNCTION_GRAPH_TRACER
help
This option enables the compiler's Shadow Call Stack, which
uses a shadow stack to protect function return addresses from
diff --git a/arch/arm/Kconfig b/arch/arm/Kconfig
index a0da424..43c7773 100644
--- a/arch/arm/Kconfig
+++ b/arch/arm/Kconfig
@@ -345,12 +345,14 @@
config ARCH_MULTI_V4
bool "ARMv4 based platforms (FA526, StrongARM)"
depends on !ARCH_MULTI_V6_V7
+ depends on !LD_IS_LLD
select ARCH_MULTI_V4_V5
select CPU_FA526 if !(CPU_SA110 || CPU_SA1100)
config ARCH_MULTI_V4T
bool "ARMv4T based platforms (ARM720T, ARM920T, ...)"
depends on !ARCH_MULTI_V6_V7
+ depends on !LD_IS_LLD
select ARCH_MULTI_V4_V5
select CPU_ARM920T if !(CPU_ARM7TDMI || CPU_ARM720T || \
CPU_ARM740T || CPU_ARM9TDMI || CPU_ARM922T || \
diff --git a/arch/arm/boot/dts/spear300.dtsi b/arch/arm/boot/dts/spear300.dtsi
index b39bd5a..f1135e8 100644
--- a/arch/arm/boot/dts/spear300.dtsi
+++ b/arch/arm/boot/dts/spear300.dtsi
@@ -46,7 +46,7 @@ sdhci@70000000 {
status = "disabled";
};
- shirq: interrupt-controller@0x50000000 {
+ shirq: interrupt-controller@50000000 {
compatible = "st,spear300-shirq";
reg = <0x50000000 0x1000>;
interrupts = <28>;
diff --git a/arch/arm/boot/dts/spear310.dtsi b/arch/arm/boot/dts/spear310.dtsi
index 7757083..ce08d88 100644
--- a/arch/arm/boot/dts/spear310.dtsi
+++ b/arch/arm/boot/dts/spear310.dtsi
@@ -34,7 +34,7 @@ fsmc: flash@44000000 {
status = "disabled";
};
- shirq: interrupt-controller@0xb4000000 {
+ shirq: interrupt-controller@b4000000 {
compatible = "st,spear310-shirq";
reg = <0xb4000000 0x1000>;
interrupts = <28 29 30 1>;
diff --git a/arch/arm/boot/dts/spear320.dtsi b/arch/arm/boot/dts/spear320.dtsi
index b124744..56f1412 100644
--- a/arch/arm/boot/dts/spear320.dtsi
+++ b/arch/arm/boot/dts/spear320.dtsi
@@ -49,7 +49,7 @@ sdhci@70000000 {
status = "disabled";
};
- shirq: interrupt-controller@0xb3000000 {
+ shirq: interrupt-controller@b3000000 {
compatible = "st,spear320-shirq";
reg = <0xb3000000 0x1000>;
interrupts = <30 28 29 1>;
diff --git a/arch/arm/mach-omap1/sram-init.c b/arch/arm/mach-omap1/sram-init.c
index 27c42e2..dabf0c4 100644
--- a/arch/arm/mach-omap1/sram-init.c
+++ b/arch/arm/mach-omap1/sram-init.c
@@ -10,11 +10,11 @@
#include <linux/kernel.h>
#include <linux/init.h>
#include <linux/io.h>
+#include <linux/set_memory.h>
#include <asm/fncpy.h>
#include <asm/tlb.h>
#include <asm/cacheflush.h>
-#include <asm/set_memory.h>
#include <asm/mach/map.h>
@@ -74,8 +74,7 @@ void *omap_sram_push(void *funcp, unsigned long size)
dst = fncpy(sram, funcp, size);
- set_memory_ro(base, pages);
- set_memory_x(base, pages);
+ set_memory_rox(base, pages);
return dst;
}
@@ -126,8 +125,7 @@ static void __init omap_detect_and_map_sram(void)
base = (unsigned long)omap_sram_base;
pages = PAGE_ALIGN(omap_sram_size) / PAGE_SIZE;
- set_memory_ro(base, pages);
- set_memory_x(base, pages);
+ set_memory_rox(base, pages);
}
static void (*_omap_sram_reprogram_clock)(u32 dpllctl, u32 ckctl);
diff --git a/arch/arm/mach-omap2/sram.c b/arch/arm/mach-omap2/sram.c
index 39cf270..815d390 100644
--- a/arch/arm/mach-omap2/sram.c
+++ b/arch/arm/mach-omap2/sram.c
@@ -14,11 +14,11 @@
#include <linux/kernel.h>
#include <linux/init.h>
#include <linux/io.h>
+#include <linux/set_memory.h>
#include <asm/fncpy.h>
#include <asm/tlb.h>
#include <asm/cacheflush.h>
-#include <asm/set_memory.h>
#include <asm/mach/map.h>
@@ -96,8 +96,7 @@ void *omap_sram_push(void *funcp, unsigned long size)
dst = fncpy(sram, funcp, size);
- set_memory_ro(base, pages);
- set_memory_x(base, pages);
+ set_memory_rox(base, pages);
return dst;
}
@@ -217,8 +216,7 @@ static void __init omap2_map_sram(void)
base = (unsigned long)omap_sram_base;
pages = PAGE_ALIGN(omap_sram_size) / PAGE_SIZE;
- set_memory_ro(base, pages);
- set_memory_x(base, pages);
+ set_memory_rox(base, pages);
}
static void (*_omap2_sram_ddr_init)(u32 *slow_dll_ctrl, u32 fast_dll_ctrl,
diff --git a/arch/arm/mach-pxa/pxa27x.c b/arch/arm/mach-pxa/pxa27x.c
index afbf6ac..eea507f 100644
--- a/arch/arm/mach-pxa/pxa27x.c
+++ b/arch/arm/mach-pxa/pxa27x.c
@@ -133,8 +133,12 @@ void pxa27x_cpu_pm_enter(suspend_state_t state)
#ifndef CONFIG_IWMMXT
u64 acc0;
+#ifndef CONFIG_AS_IS_LLVM
asm volatile(".arch_extension xscale\n\t"
"mra %Q0, %R0, acc0" : "=r" (acc0));
+#else
+ asm volatile("mrrc p0, 0, %Q0, %R0, c0" : "=r" (acc0));
+#endif
#endif
/* ensure voltage-change sequencer not initiated, which hangs */
@@ -153,8 +157,12 @@ void pxa27x_cpu_pm_enter(suspend_state_t state)
case PM_SUSPEND_MEM:
cpu_suspend(pwrmode, pxa27x_finish_suspend);
#ifndef CONFIG_IWMMXT
+#ifndef CONFIG_AS_IS_LLVM
asm volatile(".arch_extension xscale\n\t"
"mar acc0, %Q0, %R0" : "=r" (acc0));
+#else
+ asm volatile("mcrr p0, 0, %Q0, %R0, c0" :: "r" (acc0));
+#endif
#endif
break;
}
diff --git a/arch/arm/mach-pxa/pxa3xx.c b/arch/arm/mach-pxa/pxa3xx.c
index 979642a..b26f00f 100644
--- a/arch/arm/mach-pxa/pxa3xx.c
+++ b/arch/arm/mach-pxa/pxa3xx.c
@@ -108,8 +108,12 @@ static void pxa3xx_cpu_pm_suspend(void)
#ifndef CONFIG_IWMMXT
u64 acc0;
+#ifdef CONFIG_CC_IS_GCC
asm volatile(".arch_extension xscale\n\t"
"mra %Q0, %R0, acc0" : "=r" (acc0));
+#else
+ asm volatile("mrrc p0, 0, %Q0, %R0, c0" : "=r" (acc0));
+#endif
#endif
/* resuming from D2 requires the HSIO2/BOOT/TPM clocks enabled */
@@ -137,8 +141,12 @@ static void pxa3xx_cpu_pm_suspend(void)
AD3ER = 0;
#ifndef CONFIG_IWMMXT
+#ifndef CONFIG_AS_IS_LLVM
asm volatile(".arch_extension xscale\n\t"
"mar acc0, %Q0, %R0" : "=r" (acc0));
+#else
+ asm volatile("mcrr p0, 0, %Q0, %R0, c0" :: "r" (acc0));
+#endif
#endif
}
diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
index cf6d1cd..0393480 100644
--- a/arch/arm64/Kconfig
+++ b/arch/arm64/Kconfig
@@ -972,22 +972,6 @@
If unsure, say Y.
-config ARM64_ERRATUM_2645198
- bool "Cortex-A715: 2645198: Workaround possible [ESR|FAR]_ELx corruption"
- default y
- help
- This option adds the workaround for ARM Cortex-A715 erratum 2645198.
-
- If a Cortex-A715 cpu sees a page mapping permissions change from executable
- to non-executable, it may corrupt the ESR_ELx and FAR_ELx registers on the
- next instruction abort caused by permission fault.
-
- Only user-space does executable to non-executable permission transition via
- mprotect() system call. Workaround the problem by doing a break-before-make
- TLB invalidation, for all changes to executable user space mappings.
-
- If unsure, say Y.
-
config CAVIUM_ERRATUM_22375
bool "Cavium erratum 22375, 24313"
default y
diff --git a/arch/arm64/boot/dts/mediatek/mt8183.dtsi b/arch/arm64/boot/dts/mediatek/mt8183.dtsi
index a70b669..402136b 100644
--- a/arch/arm64/boot/dts/mediatek/mt8183.dtsi
+++ b/arch/arm64/boot/dts/mediatek/mt8183.dtsi
@@ -1678,7 +1678,7 @@ gpu: gpu@13040000 {
<GIC_SPI 278 IRQ_TYPE_LEVEL_LOW>;
interrupt-names = "job", "mmu", "gpu";
- clocks = <&topckgen CLK_TOP_MFGPLL_CK>;
+ clocks = <&mfgcfg CLK_MFG_BG3D>;
power-domains =
<&spm MT8183_POWER_DOMAIN_MFG_CORE0>,
diff --git a/arch/arm64/boot/dts/mediatek/mt8195-demo.dts b/arch/arm64/boot/dts/mediatek/mt8195-demo.dts
index 4fbd99e..dec85d2 100644
--- a/arch/arm64/boot/dts/mediatek/mt8195-demo.dts
+++ b/arch/arm64/boot/dts/mediatek/mt8195-demo.dts
@@ -56,10 +56,10 @@ reserved-memory {
#size-cells = <2>;
ranges;
- /* 192 KiB reserved for ARM Trusted Firmware (BL31) */
+ /* 2 MiB reserved for ARM Trusted Firmware (BL31) */
bl31_secmon_reserved: secmon@54600000 {
no-map;
- reg = <0 0x54600000 0x0 0x30000>;
+ reg = <0 0x54600000 0x0 0x200000>;
};
/* 12 MiB reserved for OP-TEE (BL32)
diff --git a/arch/arm64/include/asm/hugetlb.h b/arch/arm64/include/asm/hugetlb.h
index 6a4a1ab..d20f5da 100644
--- a/arch/arm64/include/asm/hugetlb.h
+++ b/arch/arm64/include/asm/hugetlb.h
@@ -49,15 +49,6 @@ extern pte_t huge_ptep_get(pte_t *ptep);
void __init arm64_hugetlb_cma_reserve(void);
-#define huge_ptep_modify_prot_start huge_ptep_modify_prot_start
-extern pte_t huge_ptep_modify_prot_start(struct vm_area_struct *vma,
- unsigned long addr, pte_t *ptep);
-
-#define huge_ptep_modify_prot_commit huge_ptep_modify_prot_commit
-extern void huge_ptep_modify_prot_commit(struct vm_area_struct *vma,
- unsigned long addr, pte_t *ptep,
- pte_t old_pte, pte_t new_pte);
-
#include <asm-generic/hugetlb.h>
#endif /* __ASM_HUGETLB_H */
diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h
index 6914add..b4bbeed 100644
--- a/arch/arm64/include/asm/pgtable.h
+++ b/arch/arm64/include/asm/pgtable.h
@@ -1093,15 +1093,6 @@ static inline bool pud_sect_supported(void)
}
-#define __HAVE_ARCH_PTEP_MODIFY_PROT_TRANSACTION
-#define ptep_modify_prot_start ptep_modify_prot_start
-extern pte_t ptep_modify_prot_start(struct vm_area_struct *vma,
- unsigned long addr, pte_t *ptep);
-
-#define ptep_modify_prot_commit ptep_modify_prot_commit
-extern void ptep_modify_prot_commit(struct vm_area_struct *vma,
- unsigned long addr, pte_t *ptep,
- pte_t old_pte, pte_t new_pte);
#endif /* !__ASSEMBLY__ */
#endif /* __ASM_PGTABLE_H */
diff --git a/arch/arm64/kernel/cpu_errata.c b/arch/arm64/kernel/cpu_errata.c
index 307faa2..89ac000 100644
--- a/arch/arm64/kernel/cpu_errata.c
+++ b/arch/arm64/kernel/cpu_errata.c
@@ -661,13 +661,6 @@ const struct arm64_cpu_capabilities arm64_errata[] = {
CAP_MIDR_RANGE_LIST(trbe_write_out_of_range_cpus),
},
#endif
-#ifdef CONFIG_ARM64_ERRATUM_2645198
- {
- .desc = "ARM erratum 2645198",
- .capability = ARM64_WORKAROUND_2645198,
- ERRATA_MIDR_ALL_VERSIONS(MIDR_CORTEX_A715)
- },
-#endif
#ifdef CONFIG_ARM64_ERRATUM_2077057
{
.desc = "ARM erratum 2077057",
diff --git a/arch/arm64/mm/hugetlbpage.c b/arch/arm64/mm/hugetlbpage.c
index cd8d96e1..35e9a46 100644
--- a/arch/arm64/mm/hugetlbpage.c
+++ b/arch/arm64/mm/hugetlbpage.c
@@ -559,24 +559,3 @@ bool __init arch_hugetlb_valid_size(unsigned long size)
{
return __hugetlb_valid_size(size);
}
-
-pte_t huge_ptep_modify_prot_start(struct vm_area_struct *vma, unsigned long addr, pte_t *ptep)
-{
- if (IS_ENABLED(CONFIG_ARM64_WORKAROUND_2645198) &&
- cpus_have_const_cap(ARM64_WORKAROUND_2645198)) {
- /*
- * Break-before-make (BBM) is required for all user space mappings
- * when the permission changes from executable to non-executable
- * in cases where cpu is affected with errata #2645198.
- */
- if (pte_user_exec(READ_ONCE(*ptep)))
- return huge_ptep_clear_flush(vma, addr, ptep);
- }
- return huge_ptep_get_and_clear(vma->vm_mm, addr, ptep);
-}
-
-void huge_ptep_modify_prot_commit(struct vm_area_struct *vma, unsigned long addr, pte_t *ptep,
- pte_t old_pte, pte_t pte)
-{
- set_huge_pte_at(vma->vm_mm, addr, ptep, pte);
-}
diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c
index 12915f3..14c87e8 100644
--- a/arch/arm64/mm/mmu.c
+++ b/arch/arm64/mm/mmu.c
@@ -1630,24 +1630,3 @@ static int __init prevent_bootmem_remove_init(void)
}
early_initcall(prevent_bootmem_remove_init);
#endif
-
-pte_t ptep_modify_prot_start(struct vm_area_struct *vma, unsigned long addr, pte_t *ptep)
-{
- if (IS_ENABLED(CONFIG_ARM64_WORKAROUND_2645198) &&
- cpus_have_const_cap(ARM64_WORKAROUND_2645198)) {
- /*
- * Break-before-make (BBM) is required for all user space mappings
- * when the permission changes from executable to non-executable
- * in cases where cpu is affected with errata #2645198.
- */
- if (pte_user_exec(READ_ONCE(*ptep)))
- return ptep_clear_flush(vma, addr, ptep);
- }
- return ptep_get_and_clear(vma->vm_mm, addr, ptep);
-}
-
-void ptep_modify_prot_commit(struct vm_area_struct *vma, unsigned long addr, pte_t *ptep,
- pte_t old_pte, pte_t pte)
-{
- set_pte_at(vma->vm_mm, addr, ptep, pte);
-}
diff --git a/arch/arm64/tools/cpucaps b/arch/arm64/tools/cpucaps
index dfeb2c5..a86ee37 100644
--- a/arch/arm64/tools/cpucaps
+++ b/arch/arm64/tools/cpucaps
@@ -71,7 +71,6 @@
WORKAROUND_2064142
WORKAROUND_2077057
WORKAROUND_2457168
-WORKAROUND_2645198
WORKAROUND_2658417
WORKAROUND_TRBE_OVERWRITE_FILL_MODE
WORKAROUND_TSB_FLUSH_FAILURE
diff --git a/arch/csky/Kconfig b/arch/csky/Kconfig
index adee6ab..dba02da 100644
--- a/arch/csky/Kconfig
+++ b/arch/csky/Kconfig
@@ -9,6 +9,7 @@
select ARCH_USE_BUILTIN_BSWAP
select ARCH_USE_QUEUED_RWLOCKS
select ARCH_USE_QUEUED_SPINLOCKS
+ select ARCH_HAS_CURRENT_STACK_POINTER
select ARCH_INLINE_READ_LOCK if !PREEMPTION
select ARCH_INLINE_READ_LOCK_BH if !PREEMPTION
select ARCH_INLINE_READ_LOCK_IRQ if !PREEMPTION
@@ -93,7 +94,6 @@
select HAVE_PERF_USER_STACK_DUMP
select HAVE_DMA_CONTIGUOUS
select HAVE_REGS_AND_STACK_ACCESS_API
- select HAVE_RSEQ
select HAVE_STACKPROTECTOR
select HAVE_SYSCALL_TRACEPOINTS
select MAY_HAVE_SPARSE_IRQ
@@ -269,7 +269,7 @@
bool "Tightly-Coupled/Sram Memory"
depends on !COMPILE_TEST
help
- The implementation are not only used by TCM (Tightly-Coupled Meory)
+ The implementation are not only used by TCM (Tightly-Coupled Memory)
but also used by sram on SOC bus. It follow existed linux tcm
software interface, so that old tcm application codes could be
re-used directly.
diff --git a/arch/csky/include/asm/processor.h b/arch/csky/include/asm/processor.h
index 63ad71f..ea75d72 100644
--- a/arch/csky/include/asm/processor.h
+++ b/arch/csky/include/asm/processor.h
@@ -84,4 +84,6 @@ unsigned long __get_wchan(struct task_struct *p);
#define cpu_relax() barrier()
+register unsigned long current_stack_pointer __asm__("sp");
+
#endif /* __ASM_CSKY_PROCESSOR_H */
diff --git a/arch/csky/kernel/entry.S b/arch/csky/kernel/entry.S
index 547b4cd..c68cdcc 100644
--- a/arch/csky/kernel/entry.S
+++ b/arch/csky/kernel/entry.S
@@ -54,7 +54,7 @@
lrw r9, __NR_syscalls
cmphs syscallid, r9 /* Check nr of syscall */
- bt 1f
+ bt ret_from_exception
lrw r9, sys_call_table
ixw r9, syscallid
@@ -80,11 +80,6 @@
jsr syscallid
#endif
stw a0, (sp, LSAVE_A0) /* Save return value */
-1:
-#ifdef CONFIG_DEBUG_RSEQ
- mov a0, sp
- jbsr rseq_syscall
-#endif
jmpi ret_from_exception
csky_syscall_trace:
@@ -113,10 +108,6 @@
stw a0, (sp, LSAVE_A0) /* Save return value */
1:
-#ifdef CONFIG_DEBUG_RSEQ
- mov a0, sp
- jbsr rseq_syscall
-#endif
mov a0, sp /* right now, sp --> pt_regs */
jbsr syscall_trace_exit
br ret_from_exception
diff --git a/arch/csky/kernel/signal.c b/arch/csky/kernel/signal.c
index b7b3685..10da0fe 100644
--- a/arch/csky/kernel/signal.c
+++ b/arch/csky/kernel/signal.c
@@ -179,8 +179,6 @@ static void handle_signal(struct ksignal *ksig, struct pt_regs *regs)
sigset_t *oldset = sigmask_to_save();
int ret;
- rseq_signal_deliver(ksig, regs);
-
/* Are we from a system call? */
if (in_syscall(regs)) {
/* Avoid additional syscall restarting via ret_from_exception */
diff --git a/arch/csky/kernel/stacktrace.c b/arch/csky/kernel/stacktrace.c
index 9f78f5d..27ecd63 100644
--- a/arch/csky/kernel/stacktrace.c
+++ b/arch/csky/kernel/stacktrace.c
@@ -23,10 +23,9 @@ void notrace walk_stackframe(struct task_struct *task, struct pt_regs *regs,
sp = user_stack_pointer(regs);
pc = instruction_pointer(regs);
} else if (task == NULL || task == current) {
- const register unsigned long current_sp __asm__ ("sp");
const register unsigned long current_fp __asm__ ("r8");
fp = current_fp;
- sp = current_sp;
+ sp = current_stack_pointer;
pc = (unsigned long)walk_stackframe;
} else {
/* task blocked in __switch_to */
@@ -68,8 +67,7 @@ static void notrace walk_stackframe(struct task_struct *task,
sp = user_stack_pointer(regs);
pc = instruction_pointer(regs);
} else if (task == NULL || task == current) {
- const register unsigned long current_sp __asm__ ("sp");
- sp = current_sp;
+ sp = current_stack_pointer;
pc = (unsigned long)walk_stackframe;
} else {
/* task blocked in __switch_to */
diff --git a/arch/loongarch/Kconfig b/arch/loongarch/Kconfig
index a508813..9cc8b84 100644
--- a/arch/loongarch/Kconfig
+++ b/arch/loongarch/Kconfig
@@ -58,6 +58,7 @@
select ARCH_WANTS_NO_INSTR
select BUILDTIME_TABLE_SORT
select COMMON_CLK
+ select CPU_PM
select EFI
select GENERIC_CLOCKEVENTS
select GENERIC_CMOS_UPDATE
@@ -86,11 +87,18 @@
select HAVE_ARCH_TRANSPARENT_HUGEPAGE
select HAVE_ASM_MODVERSIONS
select HAVE_CONTEXT_TRACKING_USER
+ select HAVE_C_RECORDMCOUNT
select HAVE_DEBUG_STACKOVERFLOW
select HAVE_DMA_CONTIGUOUS
+ select HAVE_DYNAMIC_FTRACE
+ select HAVE_DYNAMIC_FTRACE_WITH_ARGS
+ select HAVE_DYNAMIC_FTRACE_WITH_REGS
select HAVE_EBPF_JIT
select HAVE_EXIT_THREAD
select HAVE_FAST_GUP
+ select HAVE_FTRACE_MCOUNT_RECORD
+ select HAVE_FUNCTION_GRAPH_TRACER
+ select HAVE_FUNCTION_TRACER
select HAVE_GENERIC_VDSO
select HAVE_IOREMAP_PROT
select HAVE_IRQ_EXIT_ON_IRQ_STACK
@@ -104,6 +112,7 @@
select HAVE_REGS_AND_STACK_ACCESS_API
select HAVE_RSEQ
select HAVE_SETUP_PER_CPU_AREA if NUMA
+ select HAVE_STACKPROTECTOR
select HAVE_SYSCALL_TRACEPOINTS
select HAVE_TIF_NOHZ
select HAVE_VIRT_CPU_ACCOUNTING_GEN if !SMP
@@ -113,6 +122,8 @@
select MODULES_USE_ELF_RELA if MODULES
select NEED_PER_CPU_EMBED_FIRST_CHUNK
select NEED_PER_CPU_PAGE_FIRST_CHUNK
+ select OF
+ select OF_EARLY_FLATTREE
select PCI
select PCI_DOMAINS_GENERIC
select PCI_ECAM if ACPI
@@ -123,6 +134,8 @@
select RTC_LIB
select SMP
select SPARSE_IRQ
+ select SYSCTL_ARCH_UNALIGN_ALLOW
+ select SYSCTL_ARCH_UNALIGN_NO_WARN
select SYSCTL_EXCEPTION_TRACE
select SWIOTLB
select TRACE_IRQFLAGS_SUPPORT
@@ -516,6 +529,13 @@
menu "Power management options"
+config ARCH_SUSPEND_POSSIBLE
+ def_bool y
+
+config ARCH_HIBERNATION_POSSIBLE
+ def_bool y
+
+source "kernel/power/Kconfig"
source "drivers/acpi/Kconfig"
endmenu
diff --git a/arch/loongarch/Makefile b/arch/loongarch/Makefile
index 01b57b7..4402387 100644
--- a/arch/loongarch/Makefile
+++ b/arch/loongarch/Makefile
@@ -25,6 +25,11 @@
32bit-emul = elf32loongarch
64bit-emul = elf64loongarch
+ifdef CONFIG_DYNAMIC_FTRACE
+KBUILD_CPPFLAGS += -DCC_USING_PATCHABLE_FUNCTION_ENTRY
+CC_FLAGS_FTRACE := -fpatchable-function-entry=2
+endif
+
ifdef CONFIG_64BIT
tool-archpref = $(64bit-tool-archpref)
UTS_MACHINE := loongarch64
@@ -104,6 +109,9 @@
libs-y += arch/loongarch/lib/
libs-$(CONFIG_EFI_STUB) += $(objtree)/drivers/firmware/efi/libstub/lib.a
+# suspend and hibernation support
+drivers-$(CONFIG_PM) += arch/loongarch/power/
+
ifeq ($(KBUILD_EXTMOD),)
prepare: vdso_prepare
vdso_prepare: prepare0
diff --git a/arch/loongarch/configs/loongson3_defconfig b/arch/loongarch/configs/loongson3_defconfig
index 3540e9c..eb84cae 100644
--- a/arch/loongarch/configs/loongson3_defconfig
+++ b/arch/loongarch/configs/loongson3_defconfig
@@ -34,12 +34,13 @@
CONFIG_RELAY=y
CONFIG_BLK_DEV_INITRD=y
CONFIG_EXPERT=y
-CONFIG_USERFAULTFD=y
+CONFIG_KALLSYMS_ALL=y
CONFIG_PERF_EVENTS=y
-# CONFIG_COMPAT_BRK is not set
CONFIG_LOONGARCH=y
CONFIG_64BIT=y
CONFIG_MACH_LOONGSON64=y
+CONFIG_PAGE_SIZE_16KB=y
+CONFIG_HZ_250=y
CONFIG_DMI=y
CONFIG_EFI=y
CONFIG_SMP=y
@@ -47,14 +48,14 @@
CONFIG_NR_CPUS=64
CONFIG_NUMA=y
CONFIG_KEXEC=y
-CONFIG_PAGE_SIZE_16KB=y
-CONFIG_HZ_250=y
+CONFIG_SUSPEND=y
+CONFIG_HIBERNATION=y
CONFIG_ACPI=y
CONFIG_ACPI_SPCR_TABLE=y
-CONFIG_ACPI_HOTPLUG_CPU=y
CONFIG_ACPI_TAD=y
CONFIG_ACPI_DOCK=y
CONFIG_ACPI_IPMI=m
+CONFIG_ACPI_HOTPLUG_CPU=y
CONFIG_ACPI_PCI_SLOT=y
CONFIG_ACPI_HOTPLUG_MEMORY=y
CONFIG_EFI_ZBOOT=y
@@ -73,17 +74,19 @@
CONFIG_IOSCHED_BFQ=y
CONFIG_BFQ_GROUP_IOSCHED=y
CONFIG_BINFMT_MISC=m
+CONFIG_ZPOOL=y
+CONFIG_ZSWAP=y
+CONFIG_ZSWAP_COMPRESSOR_DEFAULT_ZSTD=y
+CONFIG_ZBUD=y
+CONFIG_Z3FOLD=y
+CONFIG_ZSMALLOC=m
+# CONFIG_COMPAT_BRK is not set
CONFIG_MEMORY_HOTPLUG=y
CONFIG_MEMORY_HOTPLUG_DEFAULT_ONLINE=y
CONFIG_MEMORY_HOTREMOVE=y
CONFIG_KSM=y
CONFIG_TRANSPARENT_HUGEPAGE=y
-CONFIG_ZSWAP=y
-CONFIG_ZSWAP_COMPRESSOR_DEFAULT_ZSTD=y
-CONFIG_ZPOOL=y
-CONFIG_ZBUD=y
-CONFIG_Z3FOLD=y
-CONFIG_ZSMALLOC=m
+CONFIG_USERFAULTFD=y
CONFIG_NET=y
CONFIG_PACKET=y
CONFIG_UNIX=y
@@ -118,7 +121,6 @@
CONFIG_BRIDGE_NETFILTER=m
CONFIG_NETFILTER_NETLINK_LOG=m
CONFIG_NF_CONNTRACK=m
-CONFIG_NF_LOG_NETDEV=m
CONFIG_NF_CONNTRACK_AMANDA=m
CONFIG_NF_CONNTRACK_FTP=m
CONFIG_NF_CONNTRACK_NETBIOS_NS=m
@@ -416,6 +418,7 @@
CONFIG_ATA=y
CONFIG_SATA_AHCI=y
CONFIG_SATA_AHCI_PLATFORM=y
+CONFIG_AHCI_DWC=y
CONFIG_PATA_ATIIXP=y
CONFIG_PATA_PCMCIA=m
CONFIG_MD=y
@@ -469,13 +472,11 @@
# CONFIG_NET_VENDOR_ARC is not set
# CONFIG_NET_VENDOR_ATHEROS is not set
CONFIG_BNX2=y
-# CONFIG_NET_VENDOR_BROCADE is not set
# CONFIG_NET_VENDOR_CAVIUM is not set
CONFIG_CHELSIO_T1=m
CONFIG_CHELSIO_T1_1G=y
CONFIG_CHELSIO_T3=m
CONFIG_CHELSIO_T4=m
-# CONFIG_NET_VENDOR_CIRRUS is not set
# CONFIG_NET_VENDOR_CISCO is not set
# CONFIG_NET_VENDOR_DEC is not set
# CONFIG_NET_VENDOR_DLINK is not set
@@ -496,6 +497,7 @@
# CONFIG_NET_VENDOR_NVIDIA is not set
# CONFIG_NET_VENDOR_OKI is not set
# CONFIG_NET_VENDOR_QLOGIC is not set
+# CONFIG_NET_VENDOR_BROCADE is not set
# CONFIG_NET_VENDOR_QUALCOMM is not set
# CONFIG_NET_VENDOR_RDC is not set
CONFIG_8139CP=m
@@ -505,9 +507,9 @@
# CONFIG_NET_VENDOR_ROCKER is not set
# CONFIG_NET_VENDOR_SAMSUNG is not set
# CONFIG_NET_VENDOR_SEEQ is not set
-# CONFIG_NET_VENDOR_SOLARFLARE is not set
# CONFIG_NET_VENDOR_SILAN is not set
# CONFIG_NET_VENDOR_SIS is not set
+# CONFIG_NET_VENDOR_SOLARFLARE is not set
# CONFIG_NET_VENDOR_SMSC is not set
CONFIG_STMMAC_ETH=y
# CONFIG_NET_VENDOR_SUN is not set
@@ -588,6 +590,7 @@
CONFIG_SERIAL_8250_MANY_PORTS=y
CONFIG_SERIAL_8250_SHARE_IRQ=y
CONFIG_SERIAL_8250_RSA=y
+CONFIG_SERIAL_OF_PLATFORM=y
CONFIG_SERIAL_NONSTANDARD=y
CONFIG_PRINTER=m
CONFIG_VIRTIO_CONSOLE=y
@@ -602,6 +605,11 @@
CONFIG_SPI=y
CONFIG_GPIO_SYSFS=y
CONFIG_GPIO_LOONGSON=y
+CONFIG_POWER_RESET=y
+CONFIG_POWER_RESET_RESTART=y
+CONFIG_POWER_RESET_SYSCON=y
+CONFIG_POWER_RESET_SYSCON_POWEROFF=y
+CONFIG_SYSCON_REBOOT_MODE=y
CONFIG_SENSORS_LM75=m
CONFIG_SENSORS_LM93=m
CONFIG_SENSORS_W83795=m
@@ -609,16 +617,16 @@
CONFIG_RC_CORE=m
CONFIG_LIRC=y
CONFIG_RC_DECODERS=y
+CONFIG_IR_IMON_DECODER=m
+CONFIG_IR_JVC_DECODER=m
+CONFIG_IR_MCE_KBD_DECODER=m
CONFIG_IR_NEC_DECODER=m
CONFIG_IR_RC5_DECODER=m
CONFIG_IR_RC6_DECODER=m
-CONFIG_IR_JVC_DECODER=m
-CONFIG_IR_SONY_DECODER=m
CONFIG_IR_SANYO_DECODER=m
CONFIG_IR_SHARP_DECODER=m
-CONFIG_IR_MCE_KBD_DECODER=m
+CONFIG_IR_SONY_DECODER=m
CONFIG_IR_XMP_DECODER=m
-CONFIG_IR_IMON_DECODER=m
CONFIG_MEDIA_SUPPORT=m
CONFIG_MEDIA_USB_SUPPORT=y
CONFIG_USB_VIDEO_CLASS=m
@@ -638,6 +646,7 @@
CONFIG_FB=y
CONFIG_FB_EFI=y
CONFIG_FB_RADEON=y
+CONFIG_LCD_CLASS_DEVICE=y
CONFIG_LCD_PLATFORM=m
# CONFIG_VGA_CONSOLE is not set
CONFIG_FRAMEBUFFER_CONSOLE=y
@@ -647,7 +656,6 @@
CONFIG_SND=y
CONFIG_SND_SEQUENCER=m
CONFIG_SND_SEQ_DUMMY=m
-# CONFIG_SND_ISA is not set
CONFIG_SND_BT87X=m
CONFIG_SND_BT87X_OVERCLOCK=y
CONFIG_SND_HDA_INTEL=y
@@ -818,10 +826,6 @@
# CONFIG_CRYPTO_MANAGER_DISABLE_TESTS is not set
CONFIG_CRYPTO_PCRYPT=m
CONFIG_CRYPTO_CRYPTD=m
-CONFIG_CRYPTO_CHACHA20POLY1305=m
-CONFIG_CRYPTO_HMAC=y
-CONFIG_CRYPTO_VMAC=m
-CONFIG_CRYPTO_WP512=m
CONFIG_CRYPTO_ANUBIS=m
CONFIG_CRYPTO_BLOWFISH=m
CONFIG_CRYPTO_CAST5=m
@@ -831,6 +835,9 @@
CONFIG_CRYPTO_SERPENT=m
CONFIG_CRYPTO_TEA=m
CONFIG_CRYPTO_TWOFISH=m
+CONFIG_CRYPTO_CHACHA20POLY1305=m
+CONFIG_CRYPTO_VMAC=m
+CONFIG_CRYPTO_WP512=m
CONFIG_CRYPTO_DEFLATE=m
CONFIG_CRYPTO_LZO=m
CONFIG_CRYPTO_842=m
@@ -844,6 +851,7 @@
CONFIG_PRINTK_TIME=y
CONFIG_STRIP_ASM_SYMS=y
CONFIG_MAGIC_SYSRQ=y
+CONFIG_DEBUG_FS=y
# CONFIG_SCHED_DEBUG is not set
CONFIG_SCHEDSTATS=y
# CONFIG_DEBUG_PREEMPT is not set
diff --git a/arch/loongarch/include/asm/acpi.h b/arch/loongarch/include/asm/acpi.h
index 825c251..4198753 100644
--- a/arch/loongarch/include/asm/acpi.h
+++ b/arch/loongarch/include/asm/acpi.h
@@ -35,4 +35,14 @@ extern struct list_head acpi_wakeup_device_list;
#define ACPI_TABLE_UPGRADE_MAX_PHYS ARCH_LOW_ADDRESS_LIMIT
+extern int loongarch_acpi_suspend(void);
+extern int (*acpi_suspend_lowlevel)(void);
+extern void loongarch_suspend_enter(void);
+
+static inline unsigned long acpi_get_wakeup_address(void)
+{
+ extern void loongarch_wakeup_start(void);
+ return (unsigned long)loongarch_wakeup_start;
+}
+
#endif /* _ASM_LOONGARCH_ACPI_H */
diff --git a/arch/loongarch/include/asm/alternative-asm.h b/arch/loongarch/include/asm/alternative-asm.h
new file mode 100644
index 0000000..ff3d10a
--- /dev/null
+++ b/arch/loongarch/include/asm/alternative-asm.h
@@ -0,0 +1,82 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _ASM_ALTERNATIVE_ASM_H
+#define _ASM_ALTERNATIVE_ASM_H
+
+#ifdef __ASSEMBLY__
+
+#include <asm/asm.h>
+
+/*
+ * Issue one struct alt_instr descriptor entry (need to put it into
+ * the section .altinstructions, see below). This entry contains
+ * enough information for the alternatives patching code to patch an
+ * instruction. See apply_alternatives().
+ */
+.macro altinstruction_entry orig alt feature orig_len alt_len
+ .long \orig - .
+ .long \alt - .
+ .short \feature
+ .byte \orig_len
+ .byte \alt_len
+.endm
+
+/*
+ * Define an alternative between two instructions. If @feature is
+ * present, early code in apply_alternatives() replaces @oldinstr with
+ * @newinstr. ".fill" directive takes care of proper instruction padding
+ * in case @newinstr is longer than @oldinstr.
+ */
+.macro ALTERNATIVE oldinstr, newinstr, feature
+140 :
+ \oldinstr
+141 :
+ .fill - (((144f-143f)-(141b-140b)) > 0) * ((144f-143f)-(141b-140b)) / 4, 4, 0x03400000
+142 :
+
+ .pushsection .altinstructions, "a"
+ altinstruction_entry 140b, 143f, \feature, 142b-140b, 144f-143f
+ .popsection
+
+ .subsection 1
+143 :
+ \newinstr
+144 :
+ .previous
+.endm
+
+#define old_len (141b-140b)
+#define new_len1 (144f-143f)
+#define new_len2 (145f-144f)
+
+#define alt_max_short(a, b) ((a) ^ (((a) ^ (b)) & -(-((a) < (b)))))
+
+/*
+ * Same as ALTERNATIVE macro above but for two alternatives. If CPU
+ * has @feature1, it replaces @oldinstr with @newinstr1. If CPU has
+ * @feature2, it replaces @oldinstr with @feature2.
+ */
+.macro ALTERNATIVE_2 oldinstr, newinstr1, feature1, newinstr2, feature2
+140 :
+ \oldinstr
+141 :
+ .fill - ((alt_max_short(new_len1, new_len2) - (old_len)) > 0) * \
+ (alt_max_short(new_len1, new_len2) - (old_len)) / 4, 4, 0x03400000
+142 :
+
+ .pushsection .altinstructions, "a"
+ altinstruction_entry 140b, 143f, \feature1, 142b-140b, 144f-143f, 142b-141b
+ altinstruction_entry 140b, 144f, \feature2, 142b-140b, 145f-144f, 142b-141b
+ .popsection
+
+ .subsection 1
+143 :
+ \newinstr1
+144 :
+ \newinstr2
+145 :
+ .previous
+.endm
+
+#endif /* __ASSEMBLY__ */
+
+#endif /* _ASM_ALTERNATIVE_ASM_H */
diff --git a/arch/loongarch/include/asm/alternative.h b/arch/loongarch/include/asm/alternative.h
new file mode 100644
index 0000000..cee7b29
--- /dev/null
+++ b/arch/loongarch/include/asm/alternative.h
@@ -0,0 +1,111 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _ASM_ALTERNATIVE_H
+#define _ASM_ALTERNATIVE_H
+
+#ifndef __ASSEMBLY__
+
+#include <linux/types.h>
+#include <linux/stddef.h>
+#include <linux/stringify.h>
+#include <asm/asm.h>
+
+struct alt_instr {
+ s32 instr_offset; /* offset to original instruction */
+ s32 replace_offset; /* offset to replacement instruction */
+ u16 feature; /* feature bit set for replacement */
+ u8 instrlen; /* length of original instruction */
+ u8 replacementlen; /* length of new instruction */
+} __packed;
+
+/*
+ * Debug flag that can be tested to see whether alternative
+ * instructions were patched in already:
+ */
+extern int alternatives_patched;
+extern struct alt_instr __alt_instructions[], __alt_instructions_end[];
+
+extern void alternative_instructions(void);
+extern void apply_alternatives(struct alt_instr *start, struct alt_instr *end);
+
+#define b_replacement(num) "664"#num
+#define e_replacement(num) "665"#num
+
+#define alt_end_marker "663"
+#define alt_slen "662b-661b"
+#define alt_total_slen alt_end_marker"b-661b"
+#define alt_rlen(num) e_replacement(num)"f-"b_replacement(num)"f"
+
+#define __OLDINSTR(oldinstr, num) \
+ "661:\n\t" oldinstr "\n662:\n" \
+ ".fill -(((" alt_rlen(num) ")-(" alt_slen ")) > 0) * " \
+ "((" alt_rlen(num) ")-(" alt_slen ")) / 4, 4, 0x03400000\n"
+
+#define OLDINSTR(oldinstr, num) \
+ __OLDINSTR(oldinstr, num) \
+ alt_end_marker ":\n"
+
+#define alt_max_short(a, b) "((" a ") ^ (((" a ") ^ (" b ")) & -(-((" a ") < (" b ")))))"
+
+/*
+ * Pad the second replacement alternative with additional NOPs if it is
+ * additionally longer than the first replacement alternative.
+ */
+#define OLDINSTR_2(oldinstr, num1, num2) \
+ "661:\n\t" oldinstr "\n662:\n" \
+ ".fill -((" alt_max_short(alt_rlen(num1), alt_rlen(num2)) " - (" alt_slen ")) > 0) * " \
+ "(" alt_max_short(alt_rlen(num1), alt_rlen(num2)) " - (" alt_slen ")) / 4, " \
+ "4, 0x03400000\n" \
+ alt_end_marker ":\n"
+
+#define ALTINSTR_ENTRY(feature, num) \
+ " .long 661b - .\n" /* label */ \
+ " .long " b_replacement(num)"f - .\n" /* new instruction */ \
+ " .short " __stringify(feature) "\n" /* feature bit */ \
+ " .byte " alt_total_slen "\n" /* source len */ \
+ " .byte " alt_rlen(num) "\n" /* replacement len */
+
+#define ALTINSTR_REPLACEMENT(newinstr, feature, num) /* replacement */ \
+ b_replacement(num)":\n\t" newinstr "\n" e_replacement(num) ":\n\t"
+
+/* alternative assembly primitive: */
+#define ALTERNATIVE(oldinstr, newinstr, feature) \
+ OLDINSTR(oldinstr, 1) \
+ ".pushsection .altinstructions,\"a\"\n" \
+ ALTINSTR_ENTRY(feature, 1) \
+ ".popsection\n" \
+ ".subsection 1\n" \
+ ALTINSTR_REPLACEMENT(newinstr, feature, 1) \
+ ".previous\n"
+
+#define ALTERNATIVE_2(oldinstr, newinstr1, feature1, newinstr2, feature2)\
+ OLDINSTR_2(oldinstr, 1, 2) \
+ ".pushsection .altinstructions,\"a\"\n" \
+ ALTINSTR_ENTRY(feature1, 1) \
+ ALTINSTR_ENTRY(feature2, 2) \
+ ".popsection\n" \
+ ".subsection 1\n" \
+ ALTINSTR_REPLACEMENT(newinstr1, feature1, 1) \
+ ALTINSTR_REPLACEMENT(newinstr2, feature2, 2) \
+ ".previous\n"
+
+/*
+ * Alternative instructions for different CPU types or capabilities.
+ *
+ * This allows to use optimized instructions even on generic binary
+ * kernels.
+ *
+ * length of oldinstr must be longer or equal the length of newinstr
+ * It can be padded with nops as needed.
+ *
+ * For non barrier like inlines please define new variants
+ * without volatile and memory clobber.
+ */
+#define alternative(oldinstr, newinstr, feature) \
+ (asm volatile (ALTERNATIVE(oldinstr, newinstr, feature) : : : "memory"))
+
+#define alternative_2(oldinstr, newinstr1, feature1, newinstr2, feature2) \
+ (asm volatile(ALTERNATIVE_2(oldinstr, newinstr1, feature1, newinstr2, feature2) ::: "memory"))
+
+#endif /* __ASSEMBLY__ */
+
+#endif /* _ASM_ALTERNATIVE_H */
diff --git a/arch/loongarch/include/asm/asm-extable.h b/arch/loongarch/include/asm/asm-extable.h
new file mode 100644
index 0000000..df05005
--- /dev/null
+++ b/arch/loongarch/include/asm/asm-extable.h
@@ -0,0 +1,65 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+#ifndef __ASM_ASM_EXTABLE_H
+#define __ASM_ASM_EXTABLE_H
+
+#define EX_TYPE_NONE 0
+#define EX_TYPE_FIXUP 1
+#define EX_TYPE_UACCESS_ERR_ZERO 2
+#define EX_TYPE_BPF 3
+
+#ifdef __ASSEMBLY__
+
+#define __ASM_EXTABLE_RAW(insn, fixup, type, data) \
+ .pushsection __ex_table, "a"; \
+ .balign 4; \
+ .long ((insn) - .); \
+ .long ((fixup) - .); \
+ .short (type); \
+ .short (data); \
+ .popsection;
+
+ .macro _asm_extable, insn, fixup
+ __ASM_EXTABLE_RAW(\insn, \fixup, EX_TYPE_FIXUP, 0)
+ .endm
+
+#else /* __ASSEMBLY__ */
+
+#include <linux/bits.h>
+#include <linux/stringify.h>
+#include <asm/gpr-num.h>
+
+#define __ASM_EXTABLE_RAW(insn, fixup, type, data) \
+ ".pushsection __ex_table, \"a\"\n" \
+ ".balign 4\n" \
+ ".long ((" insn ") - .)\n" \
+ ".long ((" fixup ") - .)\n" \
+ ".short (" type ")\n" \
+ ".short (" data ")\n" \
+ ".popsection\n"
+
+#define _ASM_EXTABLE(insn, fixup) \
+ __ASM_EXTABLE_RAW(#insn, #fixup, __stringify(EX_TYPE_FIXUP), "0")
+
+#define EX_DATA_REG_ERR_SHIFT 0
+#define EX_DATA_REG_ERR GENMASK(4, 0)
+#define EX_DATA_REG_ZERO_SHIFT 5
+#define EX_DATA_REG_ZERO GENMASK(9, 5)
+
+#define EX_DATA_REG(reg, gpr) \
+ "((.L__gpr_num_" #gpr ") << " __stringify(EX_DATA_REG_##reg##_SHIFT) ")"
+
+#define _ASM_EXTABLE_UACCESS_ERR_ZERO(insn, fixup, err, zero) \
+ __DEFINE_ASM_GPR_NUMS \
+ __ASM_EXTABLE_RAW(#insn, #fixup, \
+ __stringify(EX_TYPE_UACCESS_ERR_ZERO), \
+ "(" \
+ EX_DATA_REG(ERR, err) " | " \
+ EX_DATA_REG(ZERO, zero) \
+ ")")
+
+#define _ASM_EXTABLE_UACCESS_ERR(insn, fixup, err) \
+ _ASM_EXTABLE_UACCESS_ERR_ZERO(insn, fixup, err, zero)
+
+#endif /* __ASSEMBLY__ */
+
+#endif /* __ASM_ASM_EXTABLE_H */
diff --git a/arch/loongarch/include/asm/bootinfo.h b/arch/loongarch/include/asm/bootinfo.h
index ed0910e..0051b52 100644
--- a/arch/loongarch/include/asm/bootinfo.h
+++ b/arch/loongarch/include/asm/bootinfo.h
@@ -32,6 +32,7 @@ struct loongson_system_configuration {
int cores_per_node;
int cores_per_package;
unsigned long cores_io_master;
+ unsigned long suspend_addr;
const char *cpuname;
};
diff --git a/arch/loongarch/include/asm/bugs.h b/arch/loongarch/include/asm/bugs.h
new file mode 100644
index 0000000..9839653
--- /dev/null
+++ b/arch/loongarch/include/asm/bugs.h
@@ -0,0 +1,15 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * This is included by init/main.c to check for architecture-dependent bugs.
+ *
+ * Copyright (C) 2020-2022 Loongson Technology Corporation Limited
+ */
+#ifndef _ASM_BUGS_H
+#define _ASM_BUGS_H
+
+#include <asm/cpu.h>
+#include <asm/cpu-info.h>
+
+extern void check_bugs(void);
+
+#endif /* _ASM_BUGS_H */
diff --git a/arch/loongarch/include/asm/efi.h b/arch/loongarch/include/asm/efi.h
index 97f16e60..091897d 100644
--- a/arch/loongarch/include/asm/efi.h
+++ b/arch/loongarch/include/asm/efi.h
@@ -9,6 +9,7 @@
void __init efi_init(void);
void __init efi_runtime_init(void);
+void __init *efi_fdt_pointer(void);
void efifb_setup_from_dmi(struct screen_info *si, const char *opt);
#define ARCH_EFI_IRQ_FLAGS_MASK 0x00000004 /* Bit 2: CSR.CRMD.IE */
diff --git a/arch/loongarch/include/asm/extable.h b/arch/loongarch/include/asm/extable.h
new file mode 100644
index 0000000..5abf29f
--- /dev/null
+++ b/arch/loongarch/include/asm/extable.h
@@ -0,0 +1,47 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _ASM_LOONGARCH_EXTABLE_H
+#define _ASM_LOONGARCH_EXTABLE_H
+
+/*
+ * The exception table consists of pairs of relative offsets: the first
+ * is the relative offset to an instruction that is allowed to fault,
+ * and the second is the relative offset at which the program should
+ * continue. No registers are modified, so it is entirely up to the
+ * continuation code to figure out what to do.
+ *
+ * All the routines below use bits of fixup code that are out of line
+ * with the main instruction path. This means when everything is well,
+ * we don't even have to jump over them. Further, they do not intrude
+ * on our cache or tlb entries.
+ */
+
+struct exception_table_entry {
+ int insn, fixup;
+ short type, data;
+};
+
+#define ARCH_HAS_RELATIVE_EXTABLE
+
+#define swap_ex_entry_fixup(a, b, tmp, delta) \
+do { \
+ (a)->fixup = (b)->fixup + (delta); \
+ (b)->fixup = (tmp).fixup - (delta); \
+ (a)->type = (b)->type; \
+ (b)->type = (tmp).type; \
+ (a)->data = (b)->data; \
+ (b)->data = (tmp).data; \
+} while (0)
+
+#ifdef CONFIG_BPF_JIT
+bool ex_handler_bpf(const struct exception_table_entry *ex, struct pt_regs *regs);
+#else
+static inline
+bool ex_handler_bpf(const struct exception_table_entry *ex, struct pt_regs *regs)
+{
+ return false;
+}
+#endif /* !CONFIG_BPF_JIT */
+
+bool fixup_exception(struct pt_regs *regs);
+
+#endif
diff --git a/arch/loongarch/include/asm/ftrace.h b/arch/loongarch/include/asm/ftrace.h
new file mode 100644
index 0000000..90f9d33
--- /dev/null
+++ b/arch/loongarch/include/asm/ftrace.h
@@ -0,0 +1,68 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (C) 2022 Loongson Technology Corporation Limited
+ */
+
+#ifndef _ASM_LOONGARCH_FTRACE_H
+#define _ASM_LOONGARCH_FTRACE_H
+
+#define FTRACE_PLT_IDX 0
+#define FTRACE_REGS_PLT_IDX 1
+#define NR_FTRACE_PLTS 2
+
+#define GRAPH_FAKE_OFFSET (sizeof(struct pt_regs) - offsetof(struct pt_regs, regs[1]))
+
+#ifdef CONFIG_FUNCTION_TRACER
+
+#define MCOUNT_INSN_SIZE 4 /* sizeof mcount call */
+
+#ifndef __ASSEMBLY__
+
+#ifndef CONFIG_DYNAMIC_FTRACE
+
+#define mcount _mcount
+extern void _mcount(void);
+extern void prepare_ftrace_return(unsigned long self_addr, unsigned long callsite_sp, unsigned long old);
+
+#else
+
+struct dyn_ftrace;
+struct dyn_arch_ftrace { };
+
+#define ARCH_SUPPORTS_FTRACE_OPS 1
+#define HAVE_FUNCTION_GRAPH_RET_ADDR_PTR
+
+#define ftrace_init_nop ftrace_init_nop
+int ftrace_init_nop(struct module *mod, struct dyn_ftrace *rec);
+
+static inline unsigned long ftrace_call_adjust(unsigned long addr)
+{
+ return addr;
+}
+
+void prepare_ftrace_return(unsigned long self_addr, unsigned long *parent);
+
+#endif /* CONFIG_DYNAMIC_FTRACE */
+
+#ifdef CONFIG_HAVE_DYNAMIC_FTRACE_WITH_ARGS
+struct ftrace_ops;
+
+struct ftrace_regs {
+ struct pt_regs regs;
+};
+
+static __always_inline struct pt_regs *arch_ftrace_get_regs(struct ftrace_regs *fregs)
+{
+ return &fregs->regs;
+}
+
+#define ftrace_graph_func ftrace_graph_func
+void ftrace_graph_func(unsigned long ip, unsigned long parent_ip,
+ struct ftrace_ops *op, struct ftrace_regs *fregs);
+#endif
+
+#endif /* __ASSEMBLY__ */
+
+#endif /* CONFIG_FUNCTION_TRACER */
+
+#endif /* _ASM_LOONGARCH_FTRACE_H */
diff --git a/arch/loongarch/include/asm/futex.h b/arch/loongarch/include/asm/futex.h
index feb6658..042ca44 100644
--- a/arch/loongarch/include/asm/futex.h
+++ b/arch/loongarch/include/asm/futex.h
@@ -7,6 +7,7 @@
#include <linux/futex.h>
#include <linux/uaccess.h>
+#include <asm/asm-extable.h>
#include <asm/barrier.h>
#include <asm/errno.h>
@@ -18,18 +19,11 @@
"2: sc.w $t0, %2 \n" \
" beqz $t0, 1b \n" \
"3: \n" \
- " .section .fixup,\"ax\" \n" \
- "4: li.w %0, %6 \n" \
- " b 3b \n" \
- " .previous \n" \
- " .section __ex_table,\"a\" \n" \
- " "__UA_ADDR "\t1b, 4b \n" \
- " "__UA_ADDR "\t2b, 4b \n" \
- " .previous \n" \
+ _ASM_EXTABLE_UACCESS_ERR(1b, 3b, %0) \
+ _ASM_EXTABLE_UACCESS_ERR(2b, 3b, %0) \
: "=r" (ret), "=&r" (oldval), \
"=ZC" (*uaddr) \
- : "0" (0), "ZC" (*uaddr), "Jr" (oparg), \
- "i" (-EFAULT) \
+ : "0" (0), "ZC" (*uaddr), "Jr" (oparg) \
: "memory", "t0"); \
}
@@ -86,17 +80,10 @@ futex_atomic_cmpxchg_inatomic(u32 *uval, u32 __user *uaddr, u32 oldval, u32 newv
" beqz $t0, 1b \n"
"3: \n"
__WEAK_LLSC_MB
- " .section .fixup,\"ax\" \n"
- "4: li.d %0, %6 \n"
- " b 3b \n"
- " .previous \n"
- " .section __ex_table,\"a\" \n"
- " "__UA_ADDR "\t1b, 4b \n"
- " "__UA_ADDR "\t2b, 4b \n"
- " .previous \n"
+ _ASM_EXTABLE_UACCESS_ERR(1b, 3b, %0)
+ _ASM_EXTABLE_UACCESS_ERR(2b, 3b, %0)
: "+r" (ret), "=&r" (val), "=ZC" (*uaddr)
- : "ZC" (*uaddr), "Jr" (oldval), "Jr" (newval),
- "i" (-EFAULT)
+ : "ZC" (*uaddr), "Jr" (oldval), "Jr" (newval)
: "memory", "t0");
*uval = val;
diff --git a/arch/loongarch/include/asm/gpr-num.h b/arch/loongarch/include/asm/gpr-num.h
new file mode 100644
index 0000000..e0941af
--- /dev/null
+++ b/arch/loongarch/include/asm/gpr-num.h
@@ -0,0 +1,22 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+#ifndef __ASM_GPR_NUM_H
+#define __ASM_GPR_NUM_H
+
+#ifdef __ASSEMBLY__
+
+ .equ .L__gpr_num_zero, 0
+ .irp num,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31
+ .equ .L__gpr_num_$r\num, \num
+ .endr
+
+#else /* __ASSEMBLY__ */
+
+#define __DEFINE_ASM_GPR_NUMS \
+" .equ .L__gpr_num_zero, 0\n" \
+" .irp num,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31\n" \
+" .equ .L__gpr_num_$r\\num, \\num\n" \
+" .endr\n" \
+
+#endif /* __ASSEMBLY__ */
+
+#endif /* __ASM_GPR_NUM_H */
diff --git a/arch/loongarch/include/asm/inst.h b/arch/loongarch/include/asm/inst.h
index fce1843..c00e151 100644
--- a/arch/loongarch/include/asm/inst.h
+++ b/arch/loongarch/include/asm/inst.h
@@ -8,14 +8,17 @@
#include <linux/types.h>
#include <asm/asm.h>
+#define INSN_NOP 0x03400000
#define INSN_BREAK 0x002a0000
#define ADDR_IMMMASK_LU52ID 0xFFF0000000000000
#define ADDR_IMMMASK_LU32ID 0x000FFFFF00000000
+#define ADDR_IMMMASK_LU12IW 0x00000000FFFFF000
#define ADDR_IMMMASK_ADDU16ID 0x00000000FFFF0000
#define ADDR_IMMSHIFT_LU52ID 52
#define ADDR_IMMSHIFT_LU32ID 32
+#define ADDR_IMMSHIFT_LU12IW 12
#define ADDR_IMMSHIFT_ADDU16ID 16
#define ADDR_IMM(addr, INSN) ((addr & ADDR_IMMMASK_##INSN) >> ADDR_IMMSHIFT_##INSN)
@@ -28,6 +31,7 @@ enum reg0i26_op {
enum reg1i20_op {
lu12iw_op = 0x0a,
lu32id_op = 0x0b,
+ pcaddi_op = 0x0c,
pcaddu12i_op = 0x0e,
pcaddu18i_op = 0x0f,
};
@@ -35,6 +39,8 @@ enum reg1i20_op {
enum reg1i21_op {
beqz_op = 0x10,
bnez_op = 0x11,
+ bceqz_op = 0x12, /* bits[9:8] = 0x00 */
+ bcnez_op = 0x12, /* bits[9:8] = 0x01 */
};
enum reg2_op {
@@ -76,6 +82,10 @@ enum reg2i12_op {
ldbu_op = 0xa8,
ldhu_op = 0xa9,
ldwu_op = 0xaa,
+ flds_op = 0xac,
+ fsts_op = 0xad,
+ fldd_op = 0xae,
+ fstd_op = 0xaf,
};
enum reg2i14_op {
@@ -146,6 +156,10 @@ enum reg3_op {
ldxbu_op = 0x7040,
ldxhu_op = 0x7048,
ldxwu_op = 0x7050,
+ fldxs_op = 0x7060,
+ fldxd_op = 0x7068,
+ fstxs_op = 0x7070,
+ fstxd_op = 0x7078,
amswapw_op = 0x70c0,
amswapd_op = 0x70c1,
amaddw_op = 0x70c2,
@@ -307,6 +321,12 @@ static inline bool is_imm_negative(unsigned long val, unsigned int bit)
return val & (1UL << (bit - 1));
}
+static inline bool is_pc_ins(union loongarch_instruction *ip)
+{
+ return ip->reg1i20_format.opcode >= pcaddi_op &&
+ ip->reg1i20_format.opcode <= pcaddu18i_op;
+}
+
static inline bool is_branch_ins(union loongarch_instruction *ip)
{
return ip->reg1i21_format.opcode >= beqz_op &&
@@ -331,6 +351,18 @@ static inline bool is_stack_alloc_ins(union loongarch_instruction *ip)
is_imm12_negative(ip->reg2i12_format.immediate);
}
+int larch_insn_read(void *addr, u32 *insnp);
+int larch_insn_write(void *addr, u32 insn);
+int larch_insn_patch_text(void *addr, u32 insn);
+
+u32 larch_insn_gen_nop(void);
+u32 larch_insn_gen_b(unsigned long pc, unsigned long dest);
+u32 larch_insn_gen_bl(unsigned long pc, unsigned long dest);
+
+u32 larch_insn_gen_or(enum loongarch_gpr rd, enum loongarch_gpr rj, enum loongarch_gpr rk);
+u32 larch_insn_gen_move(enum loongarch_gpr rd, enum loongarch_gpr rj);
+
+u32 larch_insn_gen_lu12iw(enum loongarch_gpr rd, int imm);
u32 larch_insn_gen_lu32id(enum loongarch_gpr rd, int imm);
u32 larch_insn_gen_lu52id(enum loongarch_gpr rd, enum loongarch_gpr rj, int imm);
u32 larch_insn_gen_jirl(enum loongarch_gpr rd, enum loongarch_gpr rj, unsigned long pc, unsigned long dest);
@@ -345,6 +377,14 @@ static inline bool unsigned_imm_check(unsigned long val, unsigned int bit)
return val < (1UL << bit);
}
+static inline unsigned long sign_extend(unsigned long val, unsigned int idx)
+{
+ if (!is_imm_negative(val, idx + 1))
+ return ((1UL << idx) - 1) & val;
+ else
+ return ~((1UL << idx) - 1) | val;
+}
+
#define DEF_EMIT_REG0I26_FORMAT(NAME, OP) \
static inline void emit_##NAME(union loongarch_instruction *insn, \
int offset) \
@@ -566,4 +606,10 @@ static inline void emit_##NAME(union loongarch_instruction *insn, \
DEF_EMIT_REG3SA2_FORMAT(alsld, alsld_op)
+struct pt_regs;
+
+void emulate_load_store_insn(struct pt_regs *regs, void __user *addr, unsigned int *pc);
+unsigned long unaligned_read(void __user *addr, void *value, unsigned long n, bool sign);
+unsigned long unaligned_write(void __user *addr, unsigned long value, unsigned long n);
+
#endif /* _ASM_INST_H */
diff --git a/arch/loongarch/include/asm/loongson.h b/arch/loongarch/include/asm/loongson.h
index 00db93e..12494cf 100644
--- a/arch/loongarch/include/asm/loongson.h
+++ b/arch/loongarch/include/asm/loongson.h
@@ -136,4 +136,7 @@ typedef enum {
#define ls7a_writel(val, addr) *(volatile unsigned int *)TO_UNCACHE(addr) = (val)
#define ls7a_writeq(val, addr) *(volatile unsigned long *)TO_UNCACHE(addr) = (val)
+void enable_gpe_wakeup(void);
+void enable_pci_wakeup(void);
+
#endif /* __ASM_LOONGSON_H */
diff --git a/arch/loongarch/include/asm/module.h b/arch/loongarch/include/asm/module.h
index b29b19a..12a0f1e 100644
--- a/arch/loongarch/include/asm/module.h
+++ b/arch/loongarch/include/asm/module.h
@@ -11,7 +11,7 @@
#define RELA_STACK_DEPTH 16
struct mod_section {
- Elf_Shdr *shdr;
+ int shndx;
int num_entries;
int max_entries;
};
@@ -20,6 +20,9 @@ struct mod_arch_specific {
struct mod_section got;
struct mod_section plt;
struct mod_section plt_idx;
+
+ /* For CONFIG_DYNAMIC_FTRACE */
+ struct plt_entry *ftrace_trampolines;
};
struct got_entry {
@@ -37,8 +40,8 @@ struct plt_idx_entry {
Elf_Addr symbol_addr;
};
-Elf_Addr module_emit_got_entry(struct module *mod, Elf_Addr val);
-Elf_Addr module_emit_plt_entry(struct module *mod, Elf_Addr val);
+Elf_Addr module_emit_got_entry(struct module *mod, Elf_Shdr *sechdrs, Elf_Addr val);
+Elf_Addr module_emit_plt_entry(struct module *mod, Elf_Shdr *sechdrs, Elf_Addr val);
static inline struct got_entry emit_got_entry(Elf_Addr val)
{
@@ -49,7 +52,7 @@ static inline struct plt_entry emit_plt_entry(unsigned long val)
{
u32 lu12iw, lu32id, lu52id, jirl;
- lu12iw = (lu12iw_op << 25 | (((val >> 12) & 0xfffff) << 5) | LOONGARCH_GPR_T1);
+ lu12iw = larch_insn_gen_lu12iw(LOONGARCH_GPR_T1, ADDR_IMM(val, LU12IW));
lu32id = larch_insn_gen_lu32id(LOONGARCH_GPR_T1, ADDR_IMM(val, LU32ID));
lu52id = larch_insn_gen_lu52id(LOONGARCH_GPR_T1, LOONGARCH_GPR_T1, ADDR_IMM(val, LU52ID));
jirl = larch_insn_gen_jirl(0, LOONGARCH_GPR_T1, 0, (val & 0xfff));
@@ -62,10 +65,10 @@ static inline struct plt_idx_entry emit_plt_idx_entry(unsigned long val)
return (struct plt_idx_entry) { val };
}
-static inline int get_plt_idx(unsigned long val, const struct mod_section *sec)
+static inline int get_plt_idx(unsigned long val, Elf_Shdr *sechdrs, const struct mod_section *sec)
{
int i;
- struct plt_idx_entry *plt_idx = (struct plt_idx_entry *)sec->shdr->sh_addr;
+ struct plt_idx_entry *plt_idx = (struct plt_idx_entry *)sechdrs[sec->shndx].sh_addr;
for (i = 0; i < sec->num_entries; i++) {
if (plt_idx[i].symbol_addr == val)
@@ -76,11 +79,12 @@ static inline int get_plt_idx(unsigned long val, const struct mod_section *sec)
}
static inline struct plt_entry *get_plt_entry(unsigned long val,
- const struct mod_section *sec_plt,
- const struct mod_section *sec_plt_idx)
+ Elf_Shdr *sechdrs,
+ const struct mod_section *sec_plt,
+ const struct mod_section *sec_plt_idx)
{
- int plt_idx = get_plt_idx(val, sec_plt_idx);
- struct plt_entry *plt = (struct plt_entry *)sec_plt->shdr->sh_addr;
+ int plt_idx = get_plt_idx(val, sechdrs, sec_plt_idx);
+ struct plt_entry *plt = (struct plt_entry *)sechdrs[sec_plt->shndx].sh_addr;
if (plt_idx < 0)
return NULL;
@@ -89,10 +93,11 @@ static inline struct plt_entry *get_plt_entry(unsigned long val,
}
static inline struct got_entry *get_got_entry(Elf_Addr val,
+ Elf_Shdr *sechdrs,
const struct mod_section *sec)
{
- struct got_entry *got = (struct got_entry *)sec->shdr->sh_addr;
int i;
+ struct got_entry *got = (struct got_entry *)sechdrs[sec->shndx].sh_addr;
for (i = 0; i < sec->num_entries; i++)
if (got[i].symbol_addr == val)
diff --git a/arch/loongarch/include/asm/module.lds.h b/arch/loongarch/include/asm/module.lds.h
index a3d1bc0..438f09d 100644
--- a/arch/loongarch/include/asm/module.lds.h
+++ b/arch/loongarch/include/asm/module.lds.h
@@ -5,4 +5,5 @@ SECTIONS {
.got : { BYTE(0) }
.plt : { BYTE(0) }
.plt.idx : { BYTE(0) }
+ .ftrace_trampoline : { BYTE(0) }
}
diff --git a/arch/loongarch/include/asm/setup.h b/arch/loongarch/include/asm/setup.h
index ca373f8..72ead58 100644
--- a/arch/loongarch/include/asm/setup.h
+++ b/arch/loongarch/include/asm/setup.h
@@ -13,6 +13,7 @@
extern unsigned long eentry;
extern unsigned long tlbrentry;
+extern char init_command_line[COMMAND_LINE_SIZE];
extern void tlb_init(int cpu);
extern void cpu_cache_init(void);
extern void cache_error_setup(void);
diff --git a/arch/loongarch/include/asm/stackprotector.h b/arch/loongarch/include/asm/stackprotector.h
new file mode 100644
index 0000000..a1a9657
--- /dev/null
+++ b/arch/loongarch/include/asm/stackprotector.h
@@ -0,0 +1,38 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * GCC stack protector support.
+ *
+ * Stack protector works by putting predefined pattern at the start of
+ * the stack frame and verifying that it hasn't been overwritten when
+ * returning from the function. The pattern is called stack canary and
+ * on LoongArch gcc expects it to be defined by a global variable called
+ * "__stack_chk_guard".
+ */
+
+#ifndef _ASM_STACKPROTECTOR_H
+#define _ASM_STACKPROTECTOR_H
+
+#include <linux/random.h>
+#include <linux/version.h>
+
+extern unsigned long __stack_chk_guard;
+
+/*
+ * Initialize the stackprotector canary value.
+ *
+ * NOTE: this must only be called from functions that never return,
+ * and it must always be inlined.
+ */
+static __always_inline void boot_init_stack_canary(void)
+{
+ unsigned long canary;
+
+ /* Try to get a semi random initial value. */
+ get_random_bytes(&canary, sizeof(canary));
+ canary ^= LINUX_VERSION_CODE;
+
+ current->stack_canary = canary;
+ __stack_chk_guard = current->stack_canary;
+}
+
+#endif /* _ASM_STACKPROTECTOR_H */
diff --git a/arch/loongarch/include/asm/string.h b/arch/loongarch/include/asm/string.h
index b07e60d..7b29cc9 100644
--- a/arch/loongarch/include/asm/string.h
+++ b/arch/loongarch/include/asm/string.h
@@ -5,8 +5,13 @@
#ifndef _ASM_STRING_H
#define _ASM_STRING_H
+#define __HAVE_ARCH_MEMSET
extern void *memset(void *__s, int __c, size_t __count);
+
+#define __HAVE_ARCH_MEMCPY
extern void *memcpy(void *__to, __const__ void *__from, size_t __n);
+
+#define __HAVE_ARCH_MEMMOVE
extern void *memmove(void *__dest, __const__ void *__src, size_t __n);
#endif /* _ASM_STRING_H */
diff --git a/arch/loongarch/include/asm/thread_info.h b/arch/loongarch/include/asm/thread_info.h
index b7dd9f1..1a3354c 100644
--- a/arch/loongarch/include/asm/thread_info.h
+++ b/arch/loongarch/include/asm/thread_info.h
@@ -38,7 +38,7 @@ struct thread_info {
#define INIT_THREAD_INFO(tsk) \
{ \
.task = &tsk, \
- .flags = 0, \
+ .flags = _TIF_FIXADE, \
.cpu = 0, \
.preempt_count = INIT_PREEMPT_COUNT, \
}
diff --git a/arch/loongarch/include/asm/time.h b/arch/loongarch/include/asm/time.h
index 2eae219..037a2d1 100644
--- a/arch/loongarch/include/asm/time.h
+++ b/arch/loongarch/include/asm/time.h
@@ -12,6 +12,7 @@
extern u64 cpu_clock_freq;
extern u64 const_clock_freq;
+extern void save_counter(void);
extern void sync_counter(void);
static inline unsigned int calc_const_freq(void)
diff --git a/arch/loongarch/include/asm/uaccess.h b/arch/loongarch/include/asm/uaccess.h
index a8ae2af..255899d 100644
--- a/arch/loongarch/include/asm/uaccess.h
+++ b/arch/loongarch/include/asm/uaccess.h
@@ -15,7 +15,8 @@
#include <linux/string.h>
#include <linux/extable.h>
#include <asm/pgtable.h>
-#include <asm-generic/extable.h>
+#include <asm/extable.h>
+#include <asm/asm-extable.h>
#include <asm-generic/access_ok.h>
extern u64 __ua_limit;
@@ -160,16 +161,9 @@ do { \
__asm__ __volatile__( \
"1: " insn " %1, %2 \n" \
"2: \n" \
- " .section .fixup,\"ax\" \n" \
- "3: li.w %0, %3 \n" \
- " move %1, $zero \n" \
- " b 2b \n" \
- " .previous \n" \
- " .section __ex_table,\"a\" \n" \
- " "__UA_ADDR "\t1b, 3b \n" \
- " .previous \n" \
+ _ASM_EXTABLE_UACCESS_ERR_ZERO(1b, 2b, %0, %1) \
: "+r" (__gu_err), "=r" (__gu_tmp) \
- : "m" (__m(ptr)), "i" (-EFAULT)); \
+ : "m" (__m(ptr))); \
\
(val) = (__typeof__(*(ptr))) __gu_tmp; \
}
@@ -192,15 +186,9 @@ do { \
__asm__ __volatile__( \
"1: " insn " %z2, %1 # __put_user_asm\n" \
"2: \n" \
- " .section .fixup,\"ax\" \n" \
- "3: li.w %0, %3 \n" \
- " b 2b \n" \
- " .previous \n" \
- " .section __ex_table,\"a\" \n" \
- " " __UA_ADDR " 1b, 3b \n" \
- " .previous \n" \
+ _ASM_EXTABLE_UACCESS_ERR(1b, 2b, %0) \
: "+r" (__pu_err), "=m" (__m(ptr)) \
- : "Jr" (__pu_val), "i" (-EFAULT)); \
+ : "Jr" (__pu_val)); \
}
#define __get_kernel_nofault(dst, src, type, err_label) \
diff --git a/arch/loongarch/include/asm/unwind.h b/arch/loongarch/include/asm/unwind.h
index 6af4718..f2b52b9 100644
--- a/arch/loongarch/include/asm/unwind.h
+++ b/arch/loongarch/include/asm/unwind.h
@@ -20,7 +20,8 @@ struct unwind_state {
char type; /* UNWINDER_XXX */
struct stack_info stack_info;
struct task_struct *task;
- bool first, error;
+ bool first, error, is_ftrace;
+ int graph_idx;
unsigned long sp, pc, ra;
};
diff --git a/arch/loongarch/kernel/Makefile b/arch/loongarch/kernel/Makefile
index 42be564..fcaa024 100644
--- a/arch/loongarch/kernel/Makefile
+++ b/arch/loongarch/kernel/Makefile
@@ -7,13 +7,27 @@
obj-y += head.o cpu-probe.o cacheinfo.o env.o setup.o entry.o genex.o \
traps.o irq.o idle.o process.o dma.o mem.o io.o reset.o switch.o \
- elf.o syscall.o signal.o time.o topology.o inst.o ptrace.o vdso.o
+ elf.o syscall.o signal.o time.o topology.o inst.o ptrace.o vdso.o \
+ alternative.o unaligned.o
obj-$(CONFIG_ACPI) += acpi.o
obj-$(CONFIG_EFI) += efi.o
obj-$(CONFIG_CPU_HAS_FPU) += fpu.o
+ifdef CONFIG_FUNCTION_TRACER
+ ifndef CONFIG_DYNAMIC_FTRACE
+ obj-y += mcount.o ftrace.o
+ CFLAGS_REMOVE_ftrace.o = $(CC_FLAGS_FTRACE)
+ else
+ obj-y += mcount_dyn.o ftrace_dyn.o
+ CFLAGS_REMOVE_ftrace_dyn.o = $(CC_FLAGS_FTRACE)
+ endif
+ CFLAGS_REMOVE_inst.o = $(CC_FLAGS_FTRACE)
+ CFLAGS_REMOVE_time.o = $(CC_FLAGS_FTRACE)
+ CFLAGS_REMOVE_perf_event.o = $(CC_FLAGS_FTRACE)
+endif
+
obj-$(CONFIG_MODULES) += module.o module-sections.o
obj-$(CONFIG_STACKTRACE) += stacktrace.o
diff --git a/arch/loongarch/kernel/acpi.c b/arch/loongarch/kernel/acpi.c
index 8319cc4..98f4311 100644
--- a/arch/loongarch/kernel/acpi.c
+++ b/arch/loongarch/kernel/acpi.c
@@ -12,6 +12,7 @@
#include <linux/irq.h>
#include <linux/irqdomain.h>
#include <linux/memblock.h>
+#include <linux/of_fdt.h>
#include <linux/serial_core.h>
#include <asm/io.h>
#include <asm/numa.h>
@@ -139,20 +140,26 @@ static void __init acpi_process_madt(void)
loongson_sysconf.nr_cpus = num_processors;
}
+#ifndef CONFIG_SUSPEND
+int (*acpi_suspend_lowlevel)(void);
+#else
+int (*acpi_suspend_lowlevel)(void) = loongarch_acpi_suspend;
+#endif
+
void __init acpi_boot_table_init(void)
{
/*
* If acpi_disabled, bail out
*/
if (acpi_disabled)
- return;
+ goto fdt_earlycon;
/*
* Initialize the ACPI boot-time table parser.
*/
if (acpi_table_init()) {
disable_acpi();
- return;
+ goto fdt_earlycon;
}
loongson_sysconf.boot_cpu_id = read_csr_cpuid();
@@ -164,6 +171,12 @@ void __init acpi_boot_table_init(void)
/* Do not enable ACPI SPCR console by default */
acpi_parse_spcr(earlycon_acpi_spcr_enable, false);
+
+ return;
+
+fdt_earlycon:
+ if (earlycon_acpi_spcr_enable)
+ early_init_dt_scan_chosen_stdout();
}
#ifdef CONFIG_ACPI_NUMA
diff --git a/arch/loongarch/kernel/alternative.c b/arch/loongarch/kernel/alternative.c
new file mode 100644
index 0000000..c5aebea
--- /dev/null
+++ b/arch/loongarch/kernel/alternative.c
@@ -0,0 +1,246 @@
+// SPDX-License-Identifier: GPL-2.0-only
+#include <linux/mm.h>
+#include <linux/module.h>
+#include <asm/alternative.h>
+#include <asm/cacheflush.h>
+#include <asm/inst.h>
+#include <asm/sections.h>
+
+int __read_mostly alternatives_patched;
+
+EXPORT_SYMBOL_GPL(alternatives_patched);
+
+#define MAX_PATCH_SIZE (((u8)(-1)) / LOONGARCH_INSN_SIZE)
+
+static int __initdata_or_module debug_alternative;
+
+static int __init debug_alt(char *str)
+{
+ debug_alternative = 1;
+ return 1;
+}
+__setup("debug-alternative", debug_alt);
+
+#define DPRINTK(fmt, args...) \
+do { \
+ if (debug_alternative) \
+ printk(KERN_DEBUG "%s: " fmt "\n", __func__, ##args); \
+} while (0)
+
+#define DUMP_WORDS(buf, count, fmt, args...) \
+do { \
+ if (unlikely(debug_alternative)) { \
+ int _j; \
+ union loongarch_instruction *_buf = buf; \
+ \
+ if (!(count)) \
+ break; \
+ \
+ printk(KERN_DEBUG fmt, ##args); \
+ for (_j = 0; _j < count - 1; _j++) \
+ printk(KERN_CONT "<%08x> ", _buf[_j].word); \
+ printk(KERN_CONT "<%08x>\n", _buf[_j].word); \
+ } \
+} while (0)
+
+/* Use this to add nops to a buffer, then text_poke the whole buffer. */
+static void __init_or_module add_nops(union loongarch_instruction *insn, int count)
+{
+ while (count--) {
+ insn->word = INSN_NOP;
+ insn++;
+ }
+}
+
+/* Is the jump addr in local .altinstructions */
+static inline bool in_alt_jump(unsigned long jump, void *start, void *end)
+{
+ return jump >= (unsigned long)start && jump < (unsigned long)end;
+}
+
+static void __init_or_module recompute_jump(union loongarch_instruction *buf,
+ union loongarch_instruction *dest, union loongarch_instruction *src,
+ void *start, void *end)
+{
+ unsigned int si, si_l, si_h;
+ unsigned long cur_pc, jump_addr, pc;
+ long offset;
+
+ cur_pc = (unsigned long)src;
+ pc = (unsigned long)dest;
+
+ si_l = src->reg0i26_format.immediate_l;
+ si_h = src->reg0i26_format.immediate_h;
+ switch (src->reg0i26_format.opcode) {
+ case b_op:
+ case bl_op:
+ jump_addr = cur_pc + sign_extend((si_h << 16 | si_l) << 2, 27);
+ if (in_alt_jump(jump_addr, start, end))
+ return;
+ offset = jump_addr - pc;
+ BUG_ON(offset < -SZ_128M || offset >= SZ_128M);
+ offset >>= 2;
+ buf->reg0i26_format.immediate_h = offset >> 16;
+ buf->reg0i26_format.immediate_l = offset;
+ return;
+ }
+
+ si_l = src->reg1i21_format.immediate_l;
+ si_h = src->reg1i21_format.immediate_h;
+ switch (src->reg1i21_format.opcode) {
+ case bceqz_op: /* bceqz_op = bcnez_op */
+ BUG_ON(buf->reg1i21_format.rj & BIT(4));
+ fallthrough;
+ case beqz_op:
+ case bnez_op:
+ jump_addr = cur_pc + sign_extend((si_h << 16 | si_l) << 2, 22);
+ if (in_alt_jump(jump_addr, start, end))
+ return;
+ offset = jump_addr - pc;
+ BUG_ON(offset < -SZ_4M || offset >= SZ_4M);
+ offset >>= 2;
+ buf->reg1i21_format.immediate_h = offset >> 16;
+ buf->reg1i21_format.immediate_l = offset;
+ return;
+ }
+
+ si = src->reg2i16_format.immediate;
+ switch (src->reg2i16_format.opcode) {
+ case beq_op:
+ case bne_op:
+ case blt_op:
+ case bge_op:
+ case bltu_op:
+ case bgeu_op:
+ jump_addr = cur_pc + sign_extend(si << 2, 17);
+ if (in_alt_jump(jump_addr, start, end))
+ return;
+ offset = jump_addr - pc;
+ BUG_ON(offset < -SZ_128K || offset >= SZ_128K);
+ offset >>= 2;
+ buf->reg2i16_format.immediate = offset;
+ return;
+ }
+}
+
+static int __init_or_module copy_alt_insns(union loongarch_instruction *buf,
+ union loongarch_instruction *dest, union loongarch_instruction *src, int nr)
+{
+ int i;
+
+ for (i = 0; i < nr; i++) {
+ buf[i].word = src[i].word;
+
+ if (is_pc_ins(&src[i])) {
+ pr_err("Not support pcrel instruction at present!");
+ return -EINVAL;
+ }
+
+ if (is_branch_ins(&src[i]) &&
+ src[i].reg2i16_format.opcode != jirl_op) {
+ recompute_jump(&buf[i], &dest[i], &src[i], src, src + nr);
+ }
+ }
+
+ return 0;
+}
+
+/*
+ * text_poke_early - Update instructions on a live kernel at boot time
+ *
+ * When you use this code to patch more than one byte of an instruction
+ * you need to make sure that other CPUs cannot execute this code in parallel.
+ * Also no thread must be currently preempted in the middle of these
+ * instructions. And on the local CPU you need to be protected again NMI or MCE
+ * handlers seeing an inconsistent instruction while you patch.
+ */
+static void *__init_or_module text_poke_early(union loongarch_instruction *insn,
+ union loongarch_instruction *buf, unsigned int nr)
+{
+ int i;
+ unsigned long flags;
+
+ local_irq_save(flags);
+
+ for (i = 0; i < nr; i++)
+ insn[i].word = buf[i].word;
+
+ local_irq_restore(flags);
+
+ wbflush();
+ flush_icache_range((unsigned long)insn, (unsigned long)(insn + nr));
+
+ return insn;
+}
+
+/*
+ * Replace instructions with better alternatives for this CPU type. This runs
+ * before SMP is initialized to avoid SMP problems with self modifying code.
+ * This implies that asymmetric systems where APs have less capabilities than
+ * the boot processor are not handled. Tough. Make sure you disable such
+ * features by hand.
+ */
+void __init_or_module apply_alternatives(struct alt_instr *start, struct alt_instr *end)
+{
+ struct alt_instr *a;
+ unsigned int nr_instr, nr_repl, nr_insnbuf;
+ union loongarch_instruction *instr, *replacement;
+ union loongarch_instruction insnbuf[MAX_PATCH_SIZE];
+
+ DPRINTK("alt table %px, -> %px", start, end);
+ /*
+ * The scan order should be from start to end. A later scanned
+ * alternative code can overwrite previously scanned alternative code.
+ * Some kernel functions (e.g. memcpy, memset, etc) use this order to
+ * patch code.
+ *
+ * So be careful if you want to change the scan order to any other
+ * order.
+ */
+ for (a = start; a < end; a++) {
+ nr_insnbuf = 0;
+
+ instr = (void *)&a->instr_offset + a->instr_offset;
+ replacement = (void *)&a->replace_offset + a->replace_offset;
+
+ BUG_ON(a->instrlen > sizeof(insnbuf));
+ BUG_ON(a->instrlen & 0x3);
+ BUG_ON(a->replacementlen & 0x3);
+
+ nr_instr = a->instrlen / LOONGARCH_INSN_SIZE;
+ nr_repl = a->replacementlen / LOONGARCH_INSN_SIZE;
+
+ if (!cpu_has(a->feature)) {
+ DPRINTK("feat not exist: %d, old: (%px len: %d), repl: (%px, len: %d)",
+ a->feature, instr, a->instrlen,
+ replacement, a->replacementlen);
+
+ continue;
+ }
+
+ DPRINTK("feat: %d, old: (%px len: %d), repl: (%px, len: %d)",
+ a->feature, instr, a->instrlen,
+ replacement, a->replacementlen);
+
+ DUMP_WORDS(instr, nr_instr, "%px: old_insn: ", instr);
+ DUMP_WORDS(replacement, nr_repl, "%px: rpl_insn: ", replacement);
+
+ copy_alt_insns(insnbuf, instr, replacement, nr_repl);
+ nr_insnbuf = nr_repl;
+
+ if (nr_instr > nr_repl) {
+ add_nops(insnbuf + nr_repl, nr_instr - nr_repl);
+ nr_insnbuf += nr_instr - nr_repl;
+ }
+ DUMP_WORDS(insnbuf, nr_insnbuf, "%px: final_insn: ", instr);
+
+ text_poke_early(instr, insnbuf, nr_insnbuf);
+ }
+}
+
+void __init alternative_instructions(void)
+{
+ apply_alternatives(__alt_instructions, __alt_instructions_end);
+
+ alternatives_patched = 1;
+}
diff --git a/arch/loongarch/kernel/asm-offsets.c b/arch/loongarch/kernel/asm-offsets.c
index bdd88ed..4bdb203 100644
--- a/arch/loongarch/kernel/asm-offsets.c
+++ b/arch/loongarch/kernel/asm-offsets.c
@@ -68,6 +68,9 @@ void output_task_defines(void)
OFFSET(TASK_FLAGS, task_struct, flags);
OFFSET(TASK_MM, task_struct, mm);
OFFSET(TASK_PID, task_struct, pid);
+#if defined(CONFIG_STACKPROTECTOR)
+ OFFSET(TASK_STACK_CANARY, task_struct, stack_canary);
+#endif
DEFINE(TASK_STRUCT_SIZE, sizeof(struct task_struct));
BLANK();
}
@@ -257,3 +260,15 @@ void output_smpboot_defines(void)
BLANK();
}
#endif
+
+#ifdef CONFIG_HIBERNATION
+void output_pbe_defines(void)
+{
+ COMMENT(" Linux struct pbe offsets. ");
+ OFFSET(PBE_ADDRESS, pbe, address);
+ OFFSET(PBE_ORIG_ADDRESS, pbe, orig_address);
+ OFFSET(PBE_NEXT, pbe, next);
+ DEFINE(PBE_SIZE, sizeof(struct pbe));
+ BLANK();
+}
+#endif
diff --git a/arch/loongarch/kernel/efi.c b/arch/loongarch/kernel/efi.c
index d75ce73..3d448fe 100644
--- a/arch/loongarch/kernel/efi.c
+++ b/arch/loongarch/kernel/efi.c
@@ -28,16 +28,29 @@ static unsigned long efi_nr_tables;
static unsigned long efi_config_table;
static unsigned long __initdata boot_memmap = EFI_INVALID_TABLE_ADDR;
+static unsigned long __initdata fdt_pointer = EFI_INVALID_TABLE_ADDR;
static efi_system_table_t *efi_systab;
static efi_config_table_type_t arch_tables[] __initdata = {
{LINUX_EFI_BOOT_MEMMAP_GUID, &boot_memmap, "MEMMAP" },
+ {DEVICE_TREE_GUID, &fdt_pointer, "FDTPTR" },
{},
};
+void __init *efi_fdt_pointer(void)
+{
+ if (!efi_systab)
+ return NULL;
+
+ if (fdt_pointer == EFI_INVALID_TABLE_ADDR)
+ return NULL;
+
+ return early_memremap_ro(fdt_pointer, SZ_64K);
+}
+
void __init efi_runtime_init(void)
{
- if (!efi_enabled(EFI_BOOT))
+ if (!efi_enabled(EFI_BOOT) || !efi_systab->runtime)
return;
if (efi_runtime_disabled()) {
diff --git a/arch/loongarch/kernel/env.c b/arch/loongarch/kernel/env.c
index 6d56a463..6b3bfb0 100644
--- a/arch/loongarch/kernel/env.c
+++ b/arch/loongarch/kernel/env.c
@@ -11,6 +11,7 @@
#include <asm/early_ioremap.h>
#include <asm/bootinfo.h>
#include <asm/loongson.h>
+#include <asm/setup.h>
u64 efi_system_table;
struct loongson_system_configuration loongson_sysconf;
@@ -27,6 +28,7 @@ void __init init_environ(void)
clear_bit(EFI_BOOT, &efi.flags);
strscpy(boot_command_line, cmdline, COMMAND_LINE_SIZE);
+ strscpy(init_command_line, cmdline, COMMAND_LINE_SIZE);
early_memunmap(cmdline, COMMAND_LINE_SIZE);
efi_system_table = fw_arg2;
diff --git a/arch/loongarch/kernel/fpu.S b/arch/loongarch/kernel/fpu.S
index 576b337..ccde941 100644
--- a/arch/loongarch/kernel/fpu.S
+++ b/arch/loongarch/kernel/fpu.S
@@ -8,6 +8,7 @@
*/
#include <asm/asm.h>
#include <asm/asmmacro.h>
+#include <asm/asm-extable.h>
#include <asm/asm-offsets.h>
#include <asm/errno.h>
#include <asm/export.h>
@@ -21,9 +22,7 @@
.macro EX insn, reg, src, offs
.ex\@: \insn \reg, \src, \offs
- .section __ex_table,"a"
- PTR .ex\@, fault
- .previous
+ _asm_extable .ex\@, fault
.endm
.macro sc_save_fp base
diff --git a/arch/loongarch/kernel/ftrace.c b/arch/loongarch/kernel/ftrace.c
new file mode 100644
index 0000000..8c3ec1b
--- /dev/null
+++ b/arch/loongarch/kernel/ftrace.c
@@ -0,0 +1,73 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Copyright (C) 2022 Loongson Technology Corporation Limited
+ */
+
+#include <linux/init.h>
+#include <linux/ftrace.h>
+#include <linux/syscalls.h>
+#include <linux/uaccess.h>
+
+#include <asm/asm.h>
+#include <asm/asm-offsets.h>
+#include <asm/cacheflush.h>
+#include <asm/inst.h>
+#include <asm/loongarch.h>
+#include <asm/syscall.h>
+
+#include <asm-generic/sections.h>
+
+#ifdef CONFIG_FUNCTION_GRAPH_TRACER
+
+/*
+ * As `call _mcount` follows LoongArch psABI, ra-saved operation and
+ * stack operation can be found before this insn.
+ */
+
+static int ftrace_get_parent_ra_addr(unsigned long insn_addr, int *ra_off)
+{
+ int limit = 32;
+ union loongarch_instruction *insn;
+
+ insn = (union loongarch_instruction *)insn_addr;
+
+ do {
+ insn--;
+ limit--;
+
+ if (is_ra_save_ins(insn))
+ *ra_off = -((1 << 12) - insn->reg2i12_format.immediate);
+
+ } while (!is_stack_alloc_ins(insn) && limit);
+
+ if (!limit)
+ return -EINVAL;
+
+ return 0;
+}
+
+void prepare_ftrace_return(unsigned long self_addr,
+ unsigned long callsite_sp, unsigned long old)
+{
+ int ra_off;
+ unsigned long return_hooker = (unsigned long)&return_to_handler;
+
+ if (unlikely(ftrace_graph_is_dead()))
+ return;
+
+ if (unlikely(atomic_read(¤t->tracing_graph_pause)))
+ return;
+
+ if (ftrace_get_parent_ra_addr(self_addr, &ra_off))
+ goto out;
+
+ if (!function_graph_enter(old, self_addr, 0, NULL))
+ *(unsigned long *)(callsite_sp + ra_off) = return_hooker;
+
+ return;
+
+out:
+ ftrace_graph_stop();
+ WARN_ON(1);
+}
+#endif /* CONFIG_FUNCTION_GRAPH_TRACER */
diff --git a/arch/loongarch/kernel/ftrace_dyn.c b/arch/loongarch/kernel/ftrace_dyn.c
new file mode 100644
index 0000000..0f07591
--- /dev/null
+++ b/arch/loongarch/kernel/ftrace_dyn.c
@@ -0,0 +1,273 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Based on arch/arm64/kernel/ftrace.c
+ *
+ * Copyright (C) 2022 Loongson Technology Corporation Limited
+ */
+
+#include <linux/ftrace.h>
+#include <linux/uaccess.h>
+
+#include <asm/inst.h>
+#include <asm/module.h>
+
+static int ftrace_modify_code(unsigned long pc, u32 old, u32 new, bool validate)
+{
+ u32 replaced;
+
+ if (validate) {
+ if (larch_insn_read((void *)pc, &replaced))
+ return -EFAULT;
+
+ if (replaced != old)
+ return -EINVAL;
+ }
+
+ if (larch_insn_patch_text((void *)pc, new))
+ return -EPERM;
+
+ return 0;
+}
+
+#ifdef CONFIG_DYNAMIC_FTRACE_WITH_REGS
+
+#ifdef CONFIG_MODULES
+static inline int __get_mod(struct module **mod, unsigned long addr)
+{
+ preempt_disable();
+ *mod = __module_text_address(addr);
+ preempt_enable();
+
+ if (WARN_ON(!(*mod)))
+ return -EINVAL;
+
+ return 0;
+}
+
+static struct plt_entry *get_ftrace_plt(struct module *mod, unsigned long addr)
+{
+ struct plt_entry *plt = mod->arch.ftrace_trampolines;
+
+ if (addr == FTRACE_ADDR)
+ return &plt[FTRACE_PLT_IDX];
+ if (addr == FTRACE_REGS_ADDR &&
+ IS_ENABLED(CONFIG_DYNAMIC_FTRACE_WITH_REGS))
+ return &plt[FTRACE_REGS_PLT_IDX];
+
+ return NULL;
+}
+
+static unsigned long get_plt_addr(struct module *mod, unsigned long addr)
+{
+ struct plt_entry *plt;
+
+ plt = get_ftrace_plt(mod, addr);
+ if (!plt) {
+ pr_err("ftrace: no module PLT for %ps\n", (void *)addr);
+ return -EINVAL;
+ }
+
+ return (unsigned long)plt;
+}
+#endif
+
+int ftrace_modify_call(struct dyn_ftrace *rec, unsigned long old_addr, unsigned long addr)
+{
+ u32 old, new;
+ unsigned long pc;
+ long offset __maybe_unused;
+
+ pc = rec->ip + LOONGARCH_INSN_SIZE;
+
+#ifdef CONFIG_MODULES
+ offset = (long)pc - (long)addr;
+
+ if (offset < -SZ_128M || offset >= SZ_128M) {
+ int ret;
+ struct module *mod;
+
+ ret = __get_mod(&mod, pc);
+ if (ret)
+ return ret;
+
+ addr = get_plt_addr(mod, addr);
+
+ old_addr = get_plt_addr(mod, old_addr);
+ }
+#endif
+
+ new = larch_insn_gen_bl(pc, addr);
+ old = larch_insn_gen_bl(pc, old_addr);
+
+ return ftrace_modify_code(pc, old, new, true);
+}
+
+#endif /* CONFIG_DYNAMIC_FTRACE_WITH_REGS */
+
+int ftrace_update_ftrace_func(ftrace_func_t func)
+{
+ u32 new;
+ unsigned long pc;
+
+ pc = (unsigned long)&ftrace_call;
+ new = larch_insn_gen_bl(pc, (unsigned long)func);
+
+ return ftrace_modify_code(pc, 0, new, false);
+}
+
+/*
+ * The compiler has inserted 2 NOPs before the regular function prologue.
+ * T series registers are available and safe because of LoongArch's psABI.
+ *
+ * At runtime, we can replace nop with bl to enable ftrace call and replace bl
+ * with nop to disable ftrace call. The bl requires us to save the original RA
+ * value, so it saves RA at t0 here.
+ *
+ * Details are:
+ *
+ * | Compiled | Disabled | Enabled |
+ * +------------+------------------------+------------------------+
+ * | nop | move t0, ra | move t0, ra |
+ * | nop | nop | bl ftrace_caller |
+ * | func_body | func_body | func_body |
+ *
+ * The RA value will be recovered by ftrace_regs_entry, and restored into RA
+ * before returning to the regular function prologue. When a function is not
+ * being traced, the "move t0, ra" is not harmful.
+ */
+
+int ftrace_init_nop(struct module *mod, struct dyn_ftrace *rec)
+{
+ u32 old, new;
+ unsigned long pc;
+
+ pc = rec->ip;
+ old = larch_insn_gen_nop();
+ new = larch_insn_gen_move(LOONGARCH_GPR_T0, LOONGARCH_GPR_RA);
+
+ return ftrace_modify_code(pc, old, new, true);
+}
+
+int ftrace_make_call(struct dyn_ftrace *rec, unsigned long addr)
+{
+ u32 old, new;
+ unsigned long pc;
+ long offset __maybe_unused;
+
+ pc = rec->ip + LOONGARCH_INSN_SIZE;
+
+#ifdef CONFIG_MODULES
+ offset = (long)pc - (long)addr;
+
+ if (offset < -SZ_128M || offset >= SZ_128M) {
+ int ret;
+ struct module *mod;
+
+ ret = __get_mod(&mod, pc);
+ if (ret)
+ return ret;
+
+ addr = get_plt_addr(mod, addr);
+ }
+#endif
+
+ old = larch_insn_gen_nop();
+ new = larch_insn_gen_bl(pc, addr);
+
+ return ftrace_modify_code(pc, old, new, true);
+}
+
+int ftrace_make_nop(struct module *mod, struct dyn_ftrace *rec, unsigned long addr)
+{
+ u32 old, new;
+ unsigned long pc;
+ long offset __maybe_unused;
+
+ pc = rec->ip + LOONGARCH_INSN_SIZE;
+
+#ifdef CONFIG_MODULES
+ offset = (long)pc - (long)addr;
+
+ if (offset < -SZ_128M || offset >= SZ_128M) {
+ int ret;
+ struct module *mod;
+
+ ret = __get_mod(&mod, pc);
+ if (ret)
+ return ret;
+
+ addr = get_plt_addr(mod, addr);
+ }
+#endif
+
+ new = larch_insn_gen_nop();
+ old = larch_insn_gen_bl(pc, addr);
+
+ return ftrace_modify_code(pc, old, new, true);
+}
+
+void arch_ftrace_update_code(int command)
+{
+ command |= FTRACE_MAY_SLEEP;
+ ftrace_modify_all_code(command);
+}
+
+int __init ftrace_dyn_arch_init(void)
+{
+ return 0;
+}
+
+#ifdef CONFIG_FUNCTION_GRAPH_TRACER
+void prepare_ftrace_return(unsigned long self_addr, unsigned long *parent)
+{
+ unsigned long old;
+ unsigned long return_hooker = (unsigned long)&return_to_handler;
+
+ if (unlikely(atomic_read(¤t->tracing_graph_pause)))
+ return;
+
+ old = *parent;
+
+ if (!function_graph_enter(old, self_addr, 0, parent))
+ *parent = return_hooker;
+}
+
+#ifdef CONFIG_HAVE_DYNAMIC_FTRACE_WITH_ARGS
+void ftrace_graph_func(unsigned long ip, unsigned long parent_ip,
+ struct ftrace_ops *op, struct ftrace_regs *fregs)
+{
+ struct pt_regs *regs = &fregs->regs;
+ unsigned long *parent = (unsigned long *)®s->regs[1];
+
+ prepare_ftrace_return(ip, (unsigned long *)parent);
+}
+#else
+static int ftrace_modify_graph_caller(bool enable)
+{
+ u32 branch, nop;
+ unsigned long pc, func;
+ extern void ftrace_graph_call(void);
+
+ pc = (unsigned long)&ftrace_graph_call;
+ func = (unsigned long)&ftrace_graph_caller;
+
+ nop = larch_insn_gen_nop();
+ branch = larch_insn_gen_b(pc, func);
+
+ if (enable)
+ return ftrace_modify_code(pc, nop, branch, true);
+ else
+ return ftrace_modify_code(pc, branch, nop, true);
+}
+
+int ftrace_enable_ftrace_graph_caller(void)
+{
+ return ftrace_modify_graph_caller(true);
+}
+
+int ftrace_disable_ftrace_graph_caller(void)
+{
+ return ftrace_modify_graph_caller(false);
+}
+#endif /* CONFIG_HAVE_DYNAMIC_FTRACE_WITH_ARGS */
+#endif /* CONFIG_FUNCTION_GRAPH_TRACER */
diff --git a/arch/loongarch/kernel/inst.c b/arch/loongarch/kernel/inst.c
index b1df0ec..512579d 100644
--- a/arch/loongarch/kernel/inst.c
+++ b/arch/loongarch/kernel/inst.c
@@ -2,8 +2,135 @@
/*
* Copyright (C) 2020-2022 Loongson Technology Corporation Limited
*/
+#include <linux/sizes.h>
+#include <linux/uaccess.h>
+
+#include <asm/cacheflush.h>
#include <asm/inst.h>
+static DEFINE_RAW_SPINLOCK(patch_lock);
+
+int larch_insn_read(void *addr, u32 *insnp)
+{
+ int ret;
+ u32 val;
+
+ ret = copy_from_kernel_nofault(&val, addr, LOONGARCH_INSN_SIZE);
+ if (!ret)
+ *insnp = val;
+
+ return ret;
+}
+
+int larch_insn_write(void *addr, u32 insn)
+{
+ int ret;
+ unsigned long flags = 0;
+
+ raw_spin_lock_irqsave(&patch_lock, flags);
+ ret = copy_to_kernel_nofault(addr, &insn, LOONGARCH_INSN_SIZE);
+ raw_spin_unlock_irqrestore(&patch_lock, flags);
+
+ return ret;
+}
+
+int larch_insn_patch_text(void *addr, u32 insn)
+{
+ int ret;
+ u32 *tp = addr;
+
+ if ((unsigned long)tp & 3)
+ return -EINVAL;
+
+ ret = larch_insn_write(tp, insn);
+ if (!ret)
+ flush_icache_range((unsigned long)tp,
+ (unsigned long)tp + LOONGARCH_INSN_SIZE);
+
+ return ret;
+}
+
+u32 larch_insn_gen_nop(void)
+{
+ return INSN_NOP;
+}
+
+u32 larch_insn_gen_b(unsigned long pc, unsigned long dest)
+{
+ long offset = dest - pc;
+ unsigned int immediate_l, immediate_h;
+ union loongarch_instruction insn;
+
+ if ((offset & 3) || offset < -SZ_128M || offset >= SZ_128M) {
+ pr_warn("The generated b instruction is out of range.\n");
+ return INSN_BREAK;
+ }
+
+ offset >>= 2;
+
+ immediate_l = offset & 0xffff;
+ offset >>= 16;
+ immediate_h = offset & 0x3ff;
+
+ insn.reg0i26_format.opcode = b_op;
+ insn.reg0i26_format.immediate_l = immediate_l;
+ insn.reg0i26_format.immediate_h = immediate_h;
+
+ return insn.word;
+}
+
+u32 larch_insn_gen_bl(unsigned long pc, unsigned long dest)
+{
+ long offset = dest - pc;
+ unsigned int immediate_l, immediate_h;
+ union loongarch_instruction insn;
+
+ if ((offset & 3) || offset < -SZ_128M || offset >= SZ_128M) {
+ pr_warn("The generated bl instruction is out of range.\n");
+ return INSN_BREAK;
+ }
+
+ offset >>= 2;
+
+ immediate_l = offset & 0xffff;
+ offset >>= 16;
+ immediate_h = offset & 0x3ff;
+
+ insn.reg0i26_format.opcode = bl_op;
+ insn.reg0i26_format.immediate_l = immediate_l;
+ insn.reg0i26_format.immediate_h = immediate_h;
+
+ return insn.word;
+}
+
+u32 larch_insn_gen_or(enum loongarch_gpr rd, enum loongarch_gpr rj, enum loongarch_gpr rk)
+{
+ union loongarch_instruction insn;
+
+ insn.reg3_format.opcode = or_op;
+ insn.reg3_format.rd = rd;
+ insn.reg3_format.rj = rj;
+ insn.reg3_format.rk = rk;
+
+ return insn.word;
+}
+
+u32 larch_insn_gen_move(enum loongarch_gpr rd, enum loongarch_gpr rj)
+{
+ return larch_insn_gen_or(rd, rj, 0);
+}
+
+u32 larch_insn_gen_lu12iw(enum loongarch_gpr rd, int imm)
+{
+ union loongarch_instruction insn;
+
+ insn.reg1i20_format.opcode = lu12iw_op;
+ insn.reg1i20_format.rd = rd;
+ insn.reg1i20_format.immediate = imm;
+
+ return insn.word;
+}
+
u32 larch_insn_gen_lu32id(enum loongarch_gpr rd, int imm)
{
union loongarch_instruction insn;
diff --git a/arch/loongarch/kernel/mcount.S b/arch/loongarch/kernel/mcount.S
new file mode 100644
index 0000000..8cdc156
--- /dev/null
+++ b/arch/loongarch/kernel/mcount.S
@@ -0,0 +1,96 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * LoongArch specific _mcount support
+ *
+ * Copyright (C) 2022 Loongson Technology Corporation Limited
+ */
+
+#include <asm/export.h>
+#include <asm/ftrace.h>
+#include <asm/regdef.h>
+#include <asm/stackframe.h>
+
+ .text
+
+#define MCOUNT_S0_OFFSET (0)
+#define MCOUNT_RA_OFFSET (SZREG)
+#define MCOUNT_STACK_SIZE (2 * SZREG)
+
+ .macro MCOUNT_SAVE_REGS
+ PTR_ADDI sp, sp, -MCOUNT_STACK_SIZE
+ PTR_S s0, sp, MCOUNT_S0_OFFSET
+ PTR_S ra, sp, MCOUNT_RA_OFFSET
+ move s0, a0
+ .endm
+
+ .macro MCOUNT_RESTORE_REGS
+ move a0, s0
+ PTR_L ra, sp, MCOUNT_RA_OFFSET
+ PTR_L s0, sp, MCOUNT_S0_OFFSET
+ PTR_ADDI sp, sp, MCOUNT_STACK_SIZE
+ .endm
+
+SYM_FUNC_START(_mcount)
+ la.pcrel t1, ftrace_stub
+ la.pcrel t2, ftrace_trace_function /* Prepare t2 for (1) */
+ PTR_L t2, t2, 0
+ beq t1, t2, fgraph_trace
+
+ MCOUNT_SAVE_REGS
+
+ move a0, ra /* arg0: self return address */
+ move a1, s0 /* arg1: parent's return address */
+ jirl ra, t2, 0 /* (1) call *ftrace_trace_function */
+
+ MCOUNT_RESTORE_REGS
+
+fgraph_trace:
+#ifdef CONFIG_FUNCTION_GRAPH_TRACER
+ la.pcrel t1, ftrace_stub
+ la.pcrel t3, ftrace_graph_return
+ PTR_L t3, t3, 0
+ bne t1, t3, ftrace_graph_caller
+ la.pcrel t1, ftrace_graph_entry_stub
+ la.pcrel t3, ftrace_graph_entry
+ PTR_L t3, t3, 0
+ bne t1, t3, ftrace_graph_caller
+#endif
+
+SYM_INNER_LABEL(ftrace_stub, SYM_L_GLOBAL)
+ jr ra
+#ifdef CONFIG_FUNCTION_GRAPH_TRACER
+SYM_INNER_LABEL(ftrace_graph_func, SYM_L_GLOBAL)
+ bl ftrace_stub
+#endif
+SYM_FUNC_END(_mcount)
+EXPORT_SYMBOL(_mcount)
+
+#ifdef CONFIG_FUNCTION_GRAPH_TRACER
+SYM_FUNC_START(ftrace_graph_caller)
+ MCOUNT_SAVE_REGS
+
+ PTR_ADDI a0, ra, -4 /* arg0: Callsite self return addr */
+ PTR_ADDI a1, sp, MCOUNT_STACK_SIZE /* arg1: Callsite sp */
+ move a2, s0 /* arg2: Callsite parent ra */
+ bl prepare_ftrace_return
+
+ MCOUNT_RESTORE_REGS
+ jr ra
+SYM_FUNC_END(ftrace_graph_caller)
+
+SYM_FUNC_START(return_to_handler)
+ PTR_ADDI sp, sp, -2 * SZREG
+ PTR_S a0, sp, 0
+ PTR_S a1, sp, SZREG
+
+ bl ftrace_return_to_handler
+
+ /* Restore the real parent address: a0 -> ra */
+ move ra, a0
+
+ PTR_L a0, sp, 0
+ PTR_L a1, sp, SZREG
+ PTR_ADDI sp, sp, 2 * SZREG
+ jr ra
+SYM_FUNC_END(return_to_handler)
+#endif /* CONFIG_FUNCTION_GRAPH_TRACER */
diff --git a/arch/loongarch/kernel/mcount_dyn.S b/arch/loongarch/kernel/mcount_dyn.S
new file mode 100644
index 0000000..bbabf06
--- /dev/null
+++ b/arch/loongarch/kernel/mcount_dyn.S
@@ -0,0 +1,149 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (C) 2022 Loongson Technology Corporation Limited
+ */
+
+#include <asm/export.h>
+#include <asm/ftrace.h>
+#include <asm/regdef.h>
+#include <asm/stackframe.h>
+
+ .text
+/*
+ * Due to -fpatchable-function-entry=2: the compiler inserted 2 NOPs before the
+ * regular C function prologue. When PC arrived here, the last 2 instructions
+ * are as follows:
+ * move t0, ra
+ * bl callsite (for modules, callsite is a tramplione)
+ *
+ * modules trampoline is as follows:
+ * lu12i.w t1, callsite[31:12]
+ * lu32i.d t1, callsite[51:32]
+ * lu52i.d t1, t1, callsite[63:52]
+ * jirl zero, t1, callsite[11:0] >> 2
+ *
+ * See arch/loongarch/kernel/ftrace_dyn.c for details. Here, pay attention to
+ * that the T series regs are available and safe because each C functions
+ * follows the LoongArch's psABI as well.
+ */
+
+ .macro ftrace_regs_entry allregs=0
+ PTR_ADDI sp, sp, -PT_SIZE
+ PTR_S t0, sp, PT_R1 /* Save parent ra at PT_R1(RA) */
+ PTR_S a0, sp, PT_R4
+ PTR_S a1, sp, PT_R5
+ PTR_S a2, sp, PT_R6
+ PTR_S a3, sp, PT_R7
+ PTR_S a4, sp, PT_R8
+ PTR_S a5, sp, PT_R9
+ PTR_S a6, sp, PT_R10
+ PTR_S a7, sp, PT_R11
+ PTR_S fp, sp, PT_R22
+ .if \allregs
+ PTR_S tp, sp, PT_R2
+ PTR_S t0, sp, PT_R12
+ PTR_S t1, sp, PT_R13
+ PTR_S t2, sp, PT_R14
+ PTR_S t3, sp, PT_R15
+ PTR_S t4, sp, PT_R16
+ PTR_S t5, sp, PT_R17
+ PTR_S t6, sp, PT_R18
+ PTR_S t7, sp, PT_R19
+ PTR_S t8, sp, PT_R20
+ PTR_S u0, sp, PT_R21
+ PTR_S s0, sp, PT_R23
+ PTR_S s1, sp, PT_R24
+ PTR_S s2, sp, PT_R25
+ PTR_S s3, sp, PT_R26
+ PTR_S s4, sp, PT_R27
+ PTR_S s5, sp, PT_R28
+ PTR_S s6, sp, PT_R29
+ PTR_S s7, sp, PT_R30
+ PTR_S s8, sp, PT_R31
+ /* Clear it for later use as a flag sometimes. */
+ PTR_S zero, sp, PT_R0
+ .endif
+ PTR_S ra, sp, PT_ERA /* Save trace function ra at PT_ERA */
+ PTR_ADDI t8, sp, PT_SIZE
+ PTR_S t8, sp, PT_R3
+ .endm
+
+SYM_FUNC_START(ftrace_stub)
+ jr ra
+SYM_FUNC_END(ftrace_stub)
+
+SYM_CODE_START(ftrace_common)
+ PTR_ADDI a0, ra, -8 /* arg0: ip */
+ move a1, t0 /* arg1: parent_ip */
+ la.pcrel t1, function_trace_op
+ PTR_L a2, t1, 0 /* arg2: op */
+ move a3, sp /* arg3: regs */
+
+SYM_INNER_LABEL(ftrace_call, SYM_L_GLOBAL)
+ bl ftrace_stub
+#ifdef CONFIG_FUNCTION_GRAPH_TRACER
+SYM_INNER_LABEL(ftrace_graph_call, SYM_L_GLOBAL)
+ nop /* b ftrace_graph_caller */
+#endif
+
+/*
+ * As we didn't use S series regs in this assmembly code and all calls
+ * are C function which will save S series regs by themselves, there is
+ * no need to restore S series regs. The T series is available and safe
+ * at the callsite, so there is no need to restore the T series regs.
+ */
+ftrace_common_return:
+ PTR_L ra, sp, PT_R1
+ PTR_L a0, sp, PT_R4
+ PTR_L a1, sp, PT_R5
+ PTR_L a2, sp, PT_R6
+ PTR_L a3, sp, PT_R7
+ PTR_L a4, sp, PT_R8
+ PTR_L a5, sp, PT_R9
+ PTR_L a6, sp, PT_R10
+ PTR_L a7, sp, PT_R11
+ PTR_L fp, sp, PT_R22
+ PTR_L t0, sp, PT_ERA
+ PTR_ADDI sp, sp, PT_SIZE
+ jr t0
+SYM_CODE_END(ftrace_common)
+
+SYM_CODE_START(ftrace_caller)
+ ftrace_regs_entry allregs=0
+ b ftrace_common
+SYM_CODE_END(ftrace_caller)
+
+#ifdef CONFIG_DYNAMIC_FTRACE_WITH_REGS
+SYM_CODE_START(ftrace_regs_caller)
+ ftrace_regs_entry allregs=1
+ b ftrace_common
+SYM_CODE_END(ftrace_regs_caller)
+#endif
+
+#ifdef CONFIG_FUNCTION_GRAPH_TRACER
+SYM_CODE_START(ftrace_graph_caller)
+ PTR_L a0, sp, PT_ERA
+ PTR_ADDI a0, a0, -8 /* arg0: self_addr */
+ PTR_ADDI a1, sp, PT_R1 /* arg1: parent */
+ bl prepare_ftrace_return
+ b ftrace_common_return
+SYM_CODE_END(ftrace_graph_caller)
+
+SYM_CODE_START(return_to_handler)
+ /* Save return value regs */
+ PTR_ADDI sp, sp, -2 * SZREG
+ PTR_S a0, sp, 0
+ PTR_S a1, sp, SZREG
+
+ move a0, zero
+ bl ftrace_return_to_handler
+ move ra, a0
+
+ /* Restore return value regs */
+ PTR_L a0, sp, 0
+ PTR_L a1, sp, SZREG
+ PTR_ADDI sp, sp, 2 * SZREG
+
+ jr ra
+SYM_CODE_END(return_to_handler)
+#endif
diff --git a/arch/loongarch/kernel/module-sections.c b/arch/loongarch/kernel/module-sections.c
index d296a70..d4dbcda 100644
--- a/arch/loongarch/kernel/module-sections.c
+++ b/arch/loongarch/kernel/module-sections.c
@@ -6,18 +6,19 @@
#include <linux/elf.h>
#include <linux/kernel.h>
#include <linux/module.h>
+#include <linux/ftrace.h>
-Elf_Addr module_emit_got_entry(struct module *mod, Elf_Addr val)
+Elf_Addr module_emit_got_entry(struct module *mod, Elf_Shdr *sechdrs, Elf_Addr val)
{
struct mod_section *got_sec = &mod->arch.got;
int i = got_sec->num_entries;
- struct got_entry *got = get_got_entry(val, got_sec);
+ struct got_entry *got = get_got_entry(val, sechdrs, got_sec);
if (got)
return (Elf_Addr)got;
/* There is no GOT entry for val yet, create a new one. */
- got = (struct got_entry *)got_sec->shdr->sh_addr;
+ got = (struct got_entry *)sechdrs[got_sec->shndx].sh_addr;
got[i] = emit_got_entry(val);
got_sec->num_entries++;
@@ -33,12 +34,12 @@ Elf_Addr module_emit_got_entry(struct module *mod, Elf_Addr val)
return (Elf_Addr)&got[i];
}
-Elf_Addr module_emit_plt_entry(struct module *mod, Elf_Addr val)
+Elf_Addr module_emit_plt_entry(struct module *mod, Elf_Shdr *sechdrs, Elf_Addr val)
{
int nr;
struct mod_section *plt_sec = &mod->arch.plt;
struct mod_section *plt_idx_sec = &mod->arch.plt_idx;
- struct plt_entry *plt = get_plt_entry(val, plt_sec, plt_idx_sec);
+ struct plt_entry *plt = get_plt_entry(val, sechdrs, plt_sec, plt_idx_sec);
struct plt_idx_entry *plt_idx;
if (plt)
@@ -47,9 +48,9 @@ Elf_Addr module_emit_plt_entry(struct module *mod, Elf_Addr val)
nr = plt_sec->num_entries;
/* There is no duplicate entry, create a new one */
- plt = (struct plt_entry *)plt_sec->shdr->sh_addr;
+ plt = (struct plt_entry *)sechdrs[plt_sec->shndx].sh_addr;
plt[nr] = emit_plt_entry(val);
- plt_idx = (struct plt_idx_entry *)plt_idx_sec->shdr->sh_addr;
+ plt_idx = (struct plt_idx_entry *)sechdrs[plt_idx_sec->shndx].sh_addr;
plt_idx[nr] = emit_plt_idx_entry(val);
plt_sec->num_entries++;
@@ -103,28 +104,31 @@ int module_frob_arch_sections(Elf_Ehdr *ehdr, Elf_Shdr *sechdrs,
char *secstrings, struct module *mod)
{
unsigned int i, num_plts = 0, num_gots = 0;
+ Elf_Shdr *got_sec, *plt_sec, *plt_idx_sec, *tramp = NULL;
/*
* Find the empty .plt sections.
*/
for (i = 0; i < ehdr->e_shnum; i++) {
if (!strcmp(secstrings + sechdrs[i].sh_name, ".got"))
- mod->arch.got.shdr = sechdrs + i;
+ mod->arch.got.shndx = i;
else if (!strcmp(secstrings + sechdrs[i].sh_name, ".plt"))
- mod->arch.plt.shdr = sechdrs + i;
+ mod->arch.plt.shndx = i;
else if (!strcmp(secstrings + sechdrs[i].sh_name, ".plt.idx"))
- mod->arch.plt_idx.shdr = sechdrs + i;
+ mod->arch.plt_idx.shndx = i;
+ else if (!strcmp(secstrings + sechdrs[i].sh_name, ".ftrace_trampoline"))
+ tramp = sechdrs + i;
}
- if (!mod->arch.got.shdr) {
+ if (!mod->arch.got.shndx) {
pr_err("%s: module GOT section(s) missing\n", mod->name);
return -ENOEXEC;
}
- if (!mod->arch.plt.shdr) {
+ if (!mod->arch.plt.shndx) {
pr_err("%s: module PLT section(s) missing\n", mod->name);
return -ENOEXEC;
}
- if (!mod->arch.plt_idx.shdr) {
+ if (!mod->arch.plt_idx.shndx) {
pr_err("%s: module PLT.IDX section(s) missing\n", mod->name);
return -ENOEXEC;
}
@@ -145,26 +149,36 @@ int module_frob_arch_sections(Elf_Ehdr *ehdr, Elf_Shdr *sechdrs,
count_max_entries(relas, num_rela, &num_plts, &num_gots);
}
- mod->arch.got.shdr->sh_type = SHT_NOBITS;
- mod->arch.got.shdr->sh_flags = SHF_ALLOC;
- mod->arch.got.shdr->sh_addralign = L1_CACHE_BYTES;
- mod->arch.got.shdr->sh_size = (num_gots + 1) * sizeof(struct got_entry);
+ got_sec = sechdrs + mod->arch.got.shndx;
+ got_sec->sh_type = SHT_NOBITS;
+ got_sec->sh_flags = SHF_ALLOC;
+ got_sec->sh_addralign = L1_CACHE_BYTES;
+ got_sec->sh_size = (num_gots + 1) * sizeof(struct got_entry);
mod->arch.got.num_entries = 0;
mod->arch.got.max_entries = num_gots;
- mod->arch.plt.shdr->sh_type = SHT_NOBITS;
- mod->arch.plt.shdr->sh_flags = SHF_EXECINSTR | SHF_ALLOC;
- mod->arch.plt.shdr->sh_addralign = L1_CACHE_BYTES;
- mod->arch.plt.shdr->sh_size = (num_plts + 1) * sizeof(struct plt_entry);
+ plt_sec = sechdrs + mod->arch.plt.shndx;
+ plt_sec->sh_type = SHT_NOBITS;
+ plt_sec->sh_flags = SHF_EXECINSTR | SHF_ALLOC;
+ plt_sec->sh_addralign = L1_CACHE_BYTES;
+ plt_sec->sh_size = (num_plts + 1) * sizeof(struct plt_entry);
mod->arch.plt.num_entries = 0;
mod->arch.plt.max_entries = num_plts;
- mod->arch.plt_idx.shdr->sh_type = SHT_NOBITS;
- mod->arch.plt_idx.shdr->sh_flags = SHF_ALLOC;
- mod->arch.plt_idx.shdr->sh_addralign = L1_CACHE_BYTES;
- mod->arch.plt_idx.shdr->sh_size = (num_plts + 1) * sizeof(struct plt_idx_entry);
+ plt_idx_sec = sechdrs + mod->arch.plt_idx.shndx;
+ plt_idx_sec->sh_type = SHT_NOBITS;
+ plt_idx_sec->sh_flags = SHF_ALLOC;
+ plt_idx_sec->sh_addralign = L1_CACHE_BYTES;
+ plt_idx_sec->sh_size = (num_plts + 1) * sizeof(struct plt_idx_entry);
mod->arch.plt_idx.num_entries = 0;
mod->arch.plt_idx.max_entries = num_plts;
+ if (tramp) {
+ tramp->sh_type = SHT_NOBITS;
+ tramp->sh_flags = SHF_EXECINSTR | SHF_ALLOC;
+ tramp->sh_addralign = __alignof__(struct plt_entry);
+ tramp->sh_size = NR_FTRACE_PLTS * sizeof(struct plt_entry);
+ }
+
return 0;
}
diff --git a/arch/loongarch/kernel/module.c b/arch/loongarch/kernel/module.c
index 097595b..b8b8608 100644
--- a/arch/loongarch/kernel/module.c
+++ b/arch/loongarch/kernel/module.c
@@ -15,8 +15,11 @@
#include <linux/vmalloc.h>
#include <linux/slab.h>
#include <linux/fs.h>
+#include <linux/ftrace.h>
#include <linux/string.h>
#include <linux/kernel.h>
+#include <asm/alternative.h>
+#include <asm/inst.h>
static int rela_stack_push(s64 stack_value, s64 *rela_stack, size_t *rela_stack_top)
{
@@ -98,16 +101,17 @@ static int apply_r_larch_sop_push_dup(struct module *mod, u32 *location, Elf_Add
return 0;
}
-static int apply_r_larch_sop_push_plt_pcrel(struct module *mod, u32 *location, Elf_Addr v,
+static int apply_r_larch_sop_push_plt_pcrel(struct module *mod,
+ Elf_Shdr *sechdrs, u32 *location, Elf_Addr v,
s64 *rela_stack, size_t *rela_stack_top, unsigned int type)
{
ptrdiff_t offset = (void *)v - (void *)location;
if (offset >= SZ_128M)
- v = module_emit_plt_entry(mod, v);
+ v = module_emit_plt_entry(mod, sechdrs, v);
if (offset < -SZ_128M)
- v = module_emit_plt_entry(mod, v);
+ v = module_emit_plt_entry(mod, sechdrs, v);
return apply_r_larch_sop_push_pcrel(mod, location, v, rela_stack, rela_stack_top, type);
}
@@ -271,17 +275,18 @@ static int apply_r_larch_add_sub(struct module *mod, u32 *location, Elf_Addr v,
}
}
-static int apply_r_larch_b26(struct module *mod, u32 *location, Elf_Addr v,
+static int apply_r_larch_b26(struct module *mod,
+ Elf_Shdr *sechdrs, u32 *location, Elf_Addr v,
s64 *rela_stack, size_t *rela_stack_top, unsigned int type)
{
ptrdiff_t offset = (void *)v - (void *)location;
union loongarch_instruction *insn = (union loongarch_instruction *)location;
if (offset >= SZ_128M)
- v = module_emit_plt_entry(mod, v);
+ v = module_emit_plt_entry(mod, sechdrs, v);
if (offset < -SZ_128M)
- v = module_emit_plt_entry(mod, v);
+ v = module_emit_plt_entry(mod, sechdrs, v);
offset = (void *)v - (void *)location;
@@ -338,10 +343,11 @@ static int apply_r_larch_pcala(struct module *mod, u32 *location, Elf_Addr v,
return 0;
}
-static int apply_r_larch_got_pc(struct module *mod, u32 *location, Elf_Addr v,
+static int apply_r_larch_got_pc(struct module *mod,
+ Elf_Shdr *sechdrs, u32 *location, Elf_Addr v,
s64 *rela_stack, size_t *rela_stack_top, unsigned int type)
{
- Elf_Addr got = module_emit_got_entry(mod, v);
+ Elf_Addr got = module_emit_got_entry(mod, sechdrs, v);
if (!got)
return -EINVAL;
@@ -386,13 +392,10 @@ static reloc_rela_handler reloc_rela_handlers[] = {
[R_LARCH_SOP_PUSH_PCREL] = apply_r_larch_sop_push_pcrel,
[R_LARCH_SOP_PUSH_ABSOLUTE] = apply_r_larch_sop_push_absolute,
[R_LARCH_SOP_PUSH_DUP] = apply_r_larch_sop_push_dup,
- [R_LARCH_SOP_PUSH_PLT_PCREL] = apply_r_larch_sop_push_plt_pcrel,
[R_LARCH_SOP_SUB ... R_LARCH_SOP_IF_ELSE] = apply_r_larch_sop,
[R_LARCH_SOP_POP_32_S_10_5 ... R_LARCH_SOP_POP_32_U] = apply_r_larch_sop_imm_field,
[R_LARCH_ADD32 ... R_LARCH_SUB64] = apply_r_larch_add_sub,
- [R_LARCH_B26] = apply_r_larch_b26,
[R_LARCH_PCALA_HI20...R_LARCH_PCALA64_HI12] = apply_r_larch_pcala,
- [R_LARCH_GOT_PC_HI20...R_LARCH_GOT_PC_LO12] = apply_r_larch_got_pc,
};
int apply_relocate_add(Elf_Shdr *sechdrs, const char *strtab,
@@ -443,7 +446,22 @@ int apply_relocate_add(Elf_Shdr *sechdrs, const char *strtab,
sym->st_value, rel[i].r_addend, (u64)location);
v = sym->st_value + rel[i].r_addend;
- err = handler(mod, location, v, rela_stack, &rela_stack_top, type);
+ switch (type) {
+ case R_LARCH_B26:
+ err = apply_r_larch_b26(mod, sechdrs, location,
+ v, rela_stack, &rela_stack_top, type);
+ break;
+ case R_LARCH_GOT_PC_HI20...R_LARCH_GOT_PC_LO12:
+ err = apply_r_larch_got_pc(mod, sechdrs, location,
+ v, rela_stack, &rela_stack_top, type);
+ break;
+ case R_LARCH_SOP_PUSH_PLT_PCREL:
+ err = apply_r_larch_sop_push_plt_pcrel(mod, sechdrs, location,
+ v, rela_stack, &rela_stack_top, type);
+ break;
+ default:
+ err = handler(mod, location, v, rela_stack, &rela_stack_top, type);
+ }
if (err)
return err;
}
@@ -456,3 +474,36 @@ void *module_alloc(unsigned long size)
return __vmalloc_node_range(size, 1, MODULES_VADDR, MODULES_END,
GFP_KERNEL, PAGE_KERNEL, 0, NUMA_NO_NODE, __builtin_return_address(0));
}
+
+static void module_init_ftrace_plt(const Elf_Ehdr *hdr,
+ const Elf_Shdr *sechdrs, struct module *mod)
+{
+#ifdef CONFIG_DYNAMIC_FTRACE
+ struct plt_entry *ftrace_plts;
+
+ ftrace_plts = (void *)sechdrs->sh_addr;
+
+ ftrace_plts[FTRACE_PLT_IDX] = emit_plt_entry(FTRACE_ADDR);
+
+ if (IS_ENABLED(CONFIG_DYNAMIC_FTRACE_WITH_REGS))
+ ftrace_plts[FTRACE_REGS_PLT_IDX] = emit_plt_entry(FTRACE_REGS_ADDR);
+
+ mod->arch.ftrace_trampolines = ftrace_plts;
+#endif
+}
+
+int module_finalize(const Elf_Ehdr *hdr,
+ const Elf_Shdr *sechdrs, struct module *mod)
+{
+ const Elf_Shdr *s, *se;
+ const char *secstrs = (void *)hdr + sechdrs[hdr->e_shstrndx].sh_offset;
+
+ for (s = sechdrs, se = sechdrs + hdr->e_shnum; s < se; s++) {
+ if (!strcmp(".altinstructions", secstrs + s->sh_name))
+ apply_alternatives((void *)s->sh_addr, (void *)s->sh_addr + s->sh_size);
+ if (!strcmp(".ftrace_trampoline", secstrs + s->sh_name))
+ module_init_ftrace_plt(hdr, s, mod);
+ }
+
+ return 0;
+}
diff --git a/arch/loongarch/kernel/numa.c b/arch/loongarch/kernel/numa.c
index eb5d3a4..7086658 100644
--- a/arch/loongarch/kernel/numa.c
+++ b/arch/loongarch/kernel/numa.c
@@ -388,6 +388,21 @@ static void __init numa_default_distance(void)
}
}
+/*
+ * fake_numa_init() - For Non-ACPI systems
+ * Return: 0 on success, -errno on failure.
+ */
+static int __init fake_numa_init(void)
+{
+ phys_addr_t start = memblock_start_of_DRAM();
+ phys_addr_t end = memblock_end_of_DRAM() - 1;
+
+ node_set(0, numa_nodes_parsed);
+ pr_info("Faking a node at [mem %pap-%pap]\n", &start, &end);
+
+ return numa_add_memblk(0, start, end + 1);
+}
+
int __init init_numa_memory(void)
{
int i;
@@ -404,7 +419,7 @@ int __init init_numa_memory(void)
memset(&numa_meminfo, 0, sizeof(numa_meminfo));
/* Parse SRAT and SLIT if provided by firmware. */
- ret = acpi_numa_init();
+ ret = acpi_disabled ? fake_numa_init() : acpi_numa_init();
if (ret < 0)
return ret;
diff --git a/arch/loongarch/kernel/process.c b/arch/loongarch/kernel/process.c
index d61c9f4..c583b1e 100644
--- a/arch/loongarch/kernel/process.c
+++ b/arch/loongarch/kernel/process.c
@@ -47,6 +47,12 @@
#include <asm/unwind.h>
#include <asm/vdso.h>
+#ifdef CONFIG_STACKPROTECTOR
+#include <linux/stackprotector.h>
+unsigned long __stack_chk_guard __read_mostly;
+EXPORT_SYMBOL(__stack_chk_guard);
+#endif
+
/*
* Idle related variables and functions
*/
diff --git a/arch/loongarch/kernel/reset.c b/arch/loongarch/kernel/reset.c
index 8c82021..1ef8c63 100644
--- a/arch/loongarch/kernel/reset.c
+++ b/arch/loongarch/kernel/reset.c
@@ -15,6 +15,7 @@
#include <acpi/reboot.h>
#include <asm/idle.h>
#include <asm/loongarch.h>
+#include <asm/loongson.h>
void (*pm_power_off)(void);
EXPORT_SYMBOL(pm_power_off);
@@ -42,6 +43,10 @@ void machine_power_off(void)
preempt_disable();
smp_send_stop();
#endif
+#ifdef CONFIG_PM
+ if (!acpi_disabled)
+ enable_pci_wakeup();
+#endif
do_kernel_power_off();
#ifdef CONFIG_EFI
efi.reset_system(EFI_RESET_SHUTDOWN, EFI_SUCCESS, 0, NULL);
diff --git a/arch/loongarch/kernel/setup.c b/arch/loongarch/kernel/setup.c
index ae436de..4344502 100644
--- a/arch/loongarch/kernel/setup.c
+++ b/arch/loongarch/kernel/setup.c
@@ -28,10 +28,16 @@
#include <linux/sizes.h>
#include <linux/device.h>
#include <linux/dma-map-ops.h>
+#include <linux/libfdt.h>
+#include <linux/of_fdt.h>
+#include <linux/of_address.h>
+#include <linux/suspend.h>
#include <linux/swiotlb.h>
#include <asm/addrspace.h>
+#include <asm/alternative.h>
#include <asm/bootinfo.h>
+#include <asm/bugs.h>
#include <asm/cache.h>
#include <asm/cpu.h>
#include <asm/dma.h>
@@ -67,6 +73,7 @@ static const char dmi_empty_string[] = " ";
*
* These are initialized so they are in the .data section
*/
+char init_command_line[COMMAND_LINE_SIZE] __initdata;
static int num_standard_resources;
static struct resource *standard_resources;
@@ -80,6 +87,11 @@ const char *get_system_type(void)
return "generic-loongson-machine";
}
+void __init check_bugs(void)
+{
+ alternative_instructions();
+}
+
static const char *dmi_string_parse(const struct dmi_header *dm, u8 s)
{
const u8 *bp = ((u8 *) dm) + dm->length;
@@ -246,6 +258,58 @@ static void __init arch_parse_crashkernel(void)
#endif
}
+static void __init fdt_setup(void)
+{
+#ifdef CONFIG_OF_EARLY_FLATTREE
+ void *fdt_pointer;
+
+ /* ACPI-based systems do not require parsing fdt */
+ if (acpi_os_get_root_pointer())
+ return;
+
+ /* Look for a device tree configuration table entry */
+ fdt_pointer = efi_fdt_pointer();
+ if (!fdt_pointer || fdt_check_header(fdt_pointer))
+ return;
+
+ early_init_dt_scan(fdt_pointer);
+ early_init_fdt_reserve_self();
+
+ max_low_pfn = PFN_PHYS(memblock_end_of_DRAM());
+#endif
+}
+
+static void __init bootcmdline_init(char **cmdline_p)
+{
+ /*
+ * If CONFIG_CMDLINE_FORCE is enabled then initializing the command line
+ * is trivial - we simply use the built-in command line unconditionally &
+ * unmodified.
+ */
+ if (IS_ENABLED(CONFIG_CMDLINE_FORCE)) {
+ strscpy(boot_command_line, CONFIG_CMDLINE, COMMAND_LINE_SIZE);
+ goto out;
+ }
+
+#ifdef CONFIG_OF_FLATTREE
+ /*
+ * If CONFIG_CMDLINE_BOOTLOADER is enabled and we are in FDT-based system,
+ * the boot_command_line will be overwritten by early_init_dt_scan_chosen().
+ * So we need to append init_command_line (the original copy of boot_command_line)
+ * to boot_command_line.
+ */
+ if (initial_boot_params) {
+ if (boot_command_line[0])
+ strlcat(boot_command_line, " ", COMMAND_LINE_SIZE);
+
+ strlcat(boot_command_line, init_command_line, COMMAND_LINE_SIZE);
+ }
+#endif
+
+out:
+ *cmdline_p = boot_command_line;
+}
+
void __init platform_init(void)
{
arch_reserve_vmcore();
@@ -258,6 +322,7 @@ void __init platform_init(void)
acpi_gbl_use_default_register_widths = false;
acpi_boot_table_init();
#endif
+ unflatten_and_copy_device_tree();
#ifdef CONFIG_NUMA
init_numa_memory();
@@ -290,6 +355,8 @@ static void __init arch_mem_init(char **cmdline_p)
check_kernel_sections_mem();
+ early_init_fdt_scan_reserved_mem();
+
/*
* In order to reduce the possibility of kernel panic when failed to
* get IO TLB memory under CONFIG_SWIOTLB, it is better to allocate
@@ -304,6 +371,10 @@ static void __init arch_mem_init(char **cmdline_p)
dma_contiguous_reserve(PFN_PHYS(max_low_pfn));
+ /* Reserve for hibernation. */
+ register_nosave_region(PFN_DOWN(__pa_symbol(&__nosave_begin)),
+ PFN_UP(__pa_symbol(&__nosave_end)));
+
memblock_dump_all();
early_memtest(PFN_PHYS(ARCH_PFN_OFFSET), PFN_PHYS(max_low_pfn));
@@ -363,6 +434,81 @@ static void __init resource_init(void)
#endif
}
+static int __init add_legacy_isa_io(struct fwnode_handle *fwnode,
+ resource_size_t hw_start, resource_size_t size)
+{
+ int ret = 0;
+ unsigned long vaddr;
+ struct logic_pio_hwaddr *range;
+
+ range = kzalloc(sizeof(*range), GFP_ATOMIC);
+ if (!range)
+ return -ENOMEM;
+
+ range->fwnode = fwnode;
+ range->size = size = round_up(size, PAGE_SIZE);
+ range->hw_start = hw_start;
+ range->flags = LOGIC_PIO_CPU_MMIO;
+
+ ret = logic_pio_register_range(range);
+ if (ret) {
+ kfree(range);
+ return ret;
+ }
+
+ /* Legacy ISA must placed at the start of PCI_IOBASE */
+ if (range->io_start != 0) {
+ logic_pio_unregister_range(range);
+ kfree(range);
+ return -EINVAL;
+ }
+
+ vaddr = (unsigned long)(PCI_IOBASE + range->io_start);
+ ioremap_page_range(vaddr, vaddr + size, hw_start, pgprot_device(PAGE_KERNEL));
+
+ return 0;
+}
+
+static __init int arch_reserve_pio_range(void)
+{
+ struct device_node *np;
+
+ for_each_node_by_name(np, "isa") {
+ struct of_range range;
+ struct of_range_parser parser;
+
+ pr_info("ISA Bridge: %pOF\n", np);
+
+ if (of_range_parser_init(&parser, np)) {
+ pr_info("Failed to parse resources.\n");
+ of_node_put(np);
+ break;
+ }
+
+ for_each_of_range(&parser, &range) {
+ switch (range.flags & IORESOURCE_TYPE_BITS) {
+ case IORESOURCE_IO:
+ pr_info(" IO 0x%016llx..0x%016llx -> 0x%016llx\n",
+ range.cpu_addr,
+ range.cpu_addr + range.size - 1,
+ range.bus_addr);
+ if (add_legacy_isa_io(&np->fwnode, range.cpu_addr, range.size))
+ pr_warn("Failed to reserve legacy IO in Logic PIO\n");
+ break;
+ case IORESOURCE_MEM:
+ pr_info(" MEM 0x%016llx..0x%016llx -> 0x%016llx\n",
+ range.cpu_addr,
+ range.cpu_addr + range.size - 1,
+ range.bus_addr);
+ break;
+ }
+ }
+ }
+
+ return 0;
+}
+arch_initcall(arch_reserve_pio_range);
+
static int __init reserve_memblock_reserved_regions(void)
{
u64 i, j;
@@ -415,12 +561,13 @@ static void __init prefill_possible_map(void)
void __init setup_arch(char **cmdline_p)
{
cpu_probe();
- *cmdline_p = boot_command_line;
init_environ();
efi_init();
+ fdt_setup();
memblock_init();
pagetable_init();
+ bootcmdline_init(cmdline_p);
parse_early_param();
reserve_initrd_mem();
diff --git a/arch/loongarch/kernel/smp.c b/arch/loongarch/kernel/smp.c
index 14508d4..8c6e227 100644
--- a/arch/loongarch/kernel/smp.c
+++ b/arch/loongarch/kernel/smp.c
@@ -16,6 +16,7 @@
#include <linux/smp.h>
#include <linux/threads.h>
#include <linux/export.h>
+#include <linux/syscore_ops.h>
#include <linux/time.h>
#include <linux/tracepoint.h>
#include <linux/sched/hotplug.h>
@@ -180,8 +181,42 @@ irqreturn_t loongson_ipi_interrupt(int irq, void *dev)
return IRQ_HANDLED;
}
+static void __init fdt_smp_setup(void)
+{
+#ifdef CONFIG_OF
+ unsigned int cpu, cpuid;
+ struct device_node *node = NULL;
+
+ for_each_of_cpu_node(node) {
+ if (!of_device_is_available(node))
+ continue;
+
+ cpuid = of_get_cpu_hwid(node, 0);
+ if (cpuid >= nr_cpu_ids)
+ continue;
+
+ if (cpuid == loongson_sysconf.boot_cpu_id) {
+ cpu = 0;
+ numa_add_cpu(cpu);
+ } else {
+ cpu = cpumask_next_zero(-1, cpu_present_mask);
+ }
+
+ num_processors++;
+ set_cpu_possible(cpu, true);
+ set_cpu_present(cpu, true);
+ __cpu_number_map[cpuid] = cpu;
+ __cpu_logical_map[cpu] = cpuid;
+ }
+
+ loongson_sysconf.nr_cpus = num_processors;
+#endif
+}
+
void __init loongson_smp_setup(void)
{
+ fdt_smp_setup();
+
cpu_data[0].core = cpu_logical_map(0) % loongson_sysconf.cores_per_package;
cpu_data[0].package = cpu_logical_map(0) / loongson_sysconf.cores_per_package;
diff --git a/arch/loongarch/kernel/switch.S b/arch/loongarch/kernel/switch.S
index 202a163..31dd819 100644
--- a/arch/loongarch/kernel/switch.S
+++ b/arch/loongarch/kernel/switch.S
@@ -23,6 +23,11 @@
stptr.d ra, a0, THREAD_REG01
stptr.d a3, a0, THREAD_SCHED_RA
stptr.d a4, a0, THREAD_SCHED_CFA
+#if defined(CONFIG_STACKPROTECTOR) && !defined(CONFIG_SMP)
+ la t7, __stack_chk_guard
+ LONG_L t8, a1, TASK_STACK_CANARY
+ LONG_S t8, t7, 0
+#endif
move tp, a2
cpu_restore_nonscratch a1
diff --git a/arch/loongarch/kernel/time.c b/arch/loongarch/kernel/time.c
index 786735d..a6576de 100644
--- a/arch/loongarch/kernel/time.c
+++ b/arch/loongarch/kernel/time.c
@@ -115,12 +115,17 @@ static unsigned long __init get_loops_per_jiffy(void)
return lpj;
}
-static long init_timeval;
+static long init_offset __nosavedata;
+
+void save_counter(void)
+{
+ init_offset = drdtime();
+}
void sync_counter(void)
{
/* Ensure counter begin at 0 */
- csr_write64(-init_timeval, LOONGARCH_CSR_CNTC);
+ csr_write64(init_offset, LOONGARCH_CSR_CNTC);
}
static int get_timer_irq(void)
@@ -219,7 +224,7 @@ void __init time_init(void)
else
const_clock_freq = calc_const_freq();
- init_timeval = drdtime() - csr_read64(LOONGARCH_CSR_CNTC);
+ init_offset = -(drdtime() - csr_read64(LOONGARCH_CSR_CNTC));
constant_clockevent_init();
constant_clocksource_init();
diff --git a/arch/loongarch/kernel/traps.c b/arch/loongarch/kernel/traps.c
index 1a4dce8..7ea62fa 100644
--- a/arch/loongarch/kernel/traps.c
+++ b/arch/loongarch/kernel/traps.c
@@ -368,13 +368,40 @@ asmlinkage void noinstr do_ade(struct pt_regs *regs)
irqentry_exit(regs, state);
}
+/* sysctl hooks */
+int unaligned_enabled __read_mostly = 1; /* Enabled by default */
+int no_unaligned_warning __read_mostly = 1; /* Only 1 warning by default */
+
asmlinkage void noinstr do_ale(struct pt_regs *regs)
{
+ unsigned int *pc;
irqentry_state_t state = irqentry_enter(regs);
+ perf_sw_event(PERF_COUNT_SW_ALIGNMENT_FAULTS, 1, regs, regs->csr_badvaddr);
+
+ /*
+ * Did we catch a fault trying to load an instruction?
+ */
+ if (regs->csr_badvaddr == regs->csr_era)
+ goto sigbus;
+ if (user_mode(regs) && !test_thread_flag(TIF_FIXADE))
+ goto sigbus;
+ if (!unaligned_enabled)
+ goto sigbus;
+ if (!no_unaligned_warning)
+ show_registers(regs);
+
+ pc = (unsigned int *)exception_era(regs);
+
+ emulate_load_store_insn(regs, (void __user *)regs->csr_badvaddr, pc);
+
+ goto out;
+
+sigbus:
die_if_kernel("Kernel ale access", regs);
force_sig_fault(SIGBUS, BUS_ADRALN, (void __user *)regs->csr_badvaddr);
+out:
irqentry_exit(regs, state);
}
diff --git a/arch/loongarch/kernel/unaligned.c b/arch/loongarch/kernel/unaligned.c
new file mode 100644
index 0000000..bdff825
--- /dev/null
+++ b/arch/loongarch/kernel/unaligned.c
@@ -0,0 +1,499 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Handle unaligned accesses by emulation.
+ *
+ * Copyright (C) 2020-2022 Loongson Technology Corporation Limited
+ *
+ * Derived from MIPS:
+ * Copyright (C) 1996, 1998, 1999, 2002 by Ralf Baechle
+ * Copyright (C) 1999 Silicon Graphics, Inc.
+ * Copyright (C) 2014 Imagination Technologies Ltd.
+ */
+#include <linux/mm.h>
+#include <linux/sched.h>
+#include <linux/signal.h>
+#include <linux/debugfs.h>
+#include <linux/perf_event.h>
+
+#include <asm/asm.h>
+#include <asm/branch.h>
+#include <asm/fpu.h>
+#include <asm/inst.h>
+
+#include "access-helper.h"
+
+#ifdef CONFIG_DEBUG_FS
+static u32 unaligned_instructions_user;
+static u32 unaligned_instructions_kernel;
+#endif
+
+static inline unsigned long read_fpr(unsigned int idx)
+{
+#define READ_FPR(idx, __value) \
+ __asm__ __volatile__("movfr2gr.d %0, $f"#idx"\n\t" : "=r"(__value));
+
+ unsigned long __value;
+
+ switch (idx) {
+ case 0:
+ READ_FPR(0, __value);
+ break;
+ case 1:
+ READ_FPR(1, __value);
+ break;
+ case 2:
+ READ_FPR(2, __value);
+ break;
+ case 3:
+ READ_FPR(3, __value);
+ break;
+ case 4:
+ READ_FPR(4, __value);
+ break;
+ case 5:
+ READ_FPR(5, __value);
+ break;
+ case 6:
+ READ_FPR(6, __value);
+ break;
+ case 7:
+ READ_FPR(7, __value);
+ break;
+ case 8:
+ READ_FPR(8, __value);
+ break;
+ case 9:
+ READ_FPR(9, __value);
+ break;
+ case 10:
+ READ_FPR(10, __value);
+ break;
+ case 11:
+ READ_FPR(11, __value);
+ break;
+ case 12:
+ READ_FPR(12, __value);
+ break;
+ case 13:
+ READ_FPR(13, __value);
+ break;
+ case 14:
+ READ_FPR(14, __value);
+ break;
+ case 15:
+ READ_FPR(15, __value);
+ break;
+ case 16:
+ READ_FPR(16, __value);
+ break;
+ case 17:
+ READ_FPR(17, __value);
+ break;
+ case 18:
+ READ_FPR(18, __value);
+ break;
+ case 19:
+ READ_FPR(19, __value);
+ break;
+ case 20:
+ READ_FPR(20, __value);
+ break;
+ case 21:
+ READ_FPR(21, __value);
+ break;
+ case 22:
+ READ_FPR(22, __value);
+ break;
+ case 23:
+ READ_FPR(23, __value);
+ break;
+ case 24:
+ READ_FPR(24, __value);
+ break;
+ case 25:
+ READ_FPR(25, __value);
+ break;
+ case 26:
+ READ_FPR(26, __value);
+ break;
+ case 27:
+ READ_FPR(27, __value);
+ break;
+ case 28:
+ READ_FPR(28, __value);
+ break;
+ case 29:
+ READ_FPR(29, __value);
+ break;
+ case 30:
+ READ_FPR(30, __value);
+ break;
+ case 31:
+ READ_FPR(31, __value);
+ break;
+ default:
+ panic("unexpected idx '%d'", idx);
+ }
+#undef READ_FPR
+ return __value;
+}
+
+static inline void write_fpr(unsigned int idx, unsigned long value)
+{
+#define WRITE_FPR(idx, value) \
+ __asm__ __volatile__("movgr2fr.d $f"#idx", %0\n\t" :: "r"(value));
+
+ switch (idx) {
+ case 0:
+ WRITE_FPR(0, value);
+ break;
+ case 1:
+ WRITE_FPR(1, value);
+ break;
+ case 2:
+ WRITE_FPR(2, value);
+ break;
+ case 3:
+ WRITE_FPR(3, value);
+ break;
+ case 4:
+ WRITE_FPR(4, value);
+ break;
+ case 5:
+ WRITE_FPR(5, value);
+ break;
+ case 6:
+ WRITE_FPR(6, value);
+ break;
+ case 7:
+ WRITE_FPR(7, value);
+ break;
+ case 8:
+ WRITE_FPR(8, value);
+ break;
+ case 9:
+ WRITE_FPR(9, value);
+ break;
+ case 10:
+ WRITE_FPR(10, value);
+ break;
+ case 11:
+ WRITE_FPR(11, value);
+ break;
+ case 12:
+ WRITE_FPR(12, value);
+ break;
+ case 13:
+ WRITE_FPR(13, value);
+ break;
+ case 14:
+ WRITE_FPR(14, value);
+ break;
+ case 15:
+ WRITE_FPR(15, value);
+ break;
+ case 16:
+ WRITE_FPR(16, value);
+ break;
+ case 17:
+ WRITE_FPR(17, value);
+ break;
+ case 18:
+ WRITE_FPR(18, value);
+ break;
+ case 19:
+ WRITE_FPR(19, value);
+ break;
+ case 20:
+ WRITE_FPR(20, value);
+ break;
+ case 21:
+ WRITE_FPR(21, value);
+ break;
+ case 22:
+ WRITE_FPR(22, value);
+ break;
+ case 23:
+ WRITE_FPR(23, value);
+ break;
+ case 24:
+ WRITE_FPR(24, value);
+ break;
+ case 25:
+ WRITE_FPR(25, value);
+ break;
+ case 26:
+ WRITE_FPR(26, value);
+ break;
+ case 27:
+ WRITE_FPR(27, value);
+ break;
+ case 28:
+ WRITE_FPR(28, value);
+ break;
+ case 29:
+ WRITE_FPR(29, value);
+ break;
+ case 30:
+ WRITE_FPR(30, value);
+ break;
+ case 31:
+ WRITE_FPR(31, value);
+ break;
+ default:
+ panic("unexpected idx '%d'", idx);
+ }
+#undef WRITE_FPR
+}
+
+void emulate_load_store_insn(struct pt_regs *regs, void __user *addr, unsigned int *pc)
+{
+ bool fp = false;
+ bool sign, write;
+ bool user = user_mode(regs);
+ unsigned int res, size = 0;
+ unsigned long value = 0;
+ union loongarch_instruction insn;
+
+ perf_sw_event(PERF_COUNT_SW_EMULATION_FAULTS, 1, regs, 0);
+
+ __get_inst(&insn.word, pc, user);
+
+ switch (insn.reg2i12_format.opcode) {
+ case ldh_op:
+ size = 2;
+ sign = true;
+ write = false;
+ break;
+ case ldhu_op:
+ size = 2;
+ sign = false;
+ write = false;
+ break;
+ case sth_op:
+ size = 2;
+ sign = true;
+ write = true;
+ break;
+ case ldw_op:
+ size = 4;
+ sign = true;
+ write = false;
+ break;
+ case ldwu_op:
+ size = 4;
+ sign = false;
+ write = false;
+ break;
+ case stw_op:
+ size = 4;
+ sign = true;
+ write = true;
+ break;
+ case ldd_op:
+ size = 8;
+ sign = true;
+ write = false;
+ break;
+ case std_op:
+ size = 8;
+ sign = true;
+ write = true;
+ break;
+ case flds_op:
+ size = 4;
+ fp = true;
+ sign = true;
+ write = false;
+ break;
+ case fsts_op:
+ size = 4;
+ fp = true;
+ sign = true;
+ write = true;
+ break;
+ case fldd_op:
+ size = 8;
+ fp = true;
+ sign = true;
+ write = false;
+ break;
+ case fstd_op:
+ size = 8;
+ fp = true;
+ sign = true;
+ write = true;
+ break;
+ }
+
+ switch (insn.reg2i14_format.opcode) {
+ case ldptrw_op:
+ size = 4;
+ sign = true;
+ write = false;
+ break;
+ case stptrw_op:
+ size = 4;
+ sign = true;
+ write = true;
+ break;
+ case ldptrd_op:
+ size = 8;
+ sign = true;
+ write = false;
+ break;
+ case stptrd_op:
+ size = 8;
+ sign = true;
+ write = true;
+ break;
+ }
+
+ switch (insn.reg3_format.opcode) {
+ case ldxh_op:
+ size = 2;
+ sign = true;
+ write = false;
+ break;
+ case ldxhu_op:
+ size = 2;
+ sign = false;
+ write = false;
+ break;
+ case stxh_op:
+ size = 2;
+ sign = true;
+ write = true;
+ break;
+ case ldxw_op:
+ size = 4;
+ sign = true;
+ write = false;
+ break;
+ case ldxwu_op:
+ size = 4;
+ sign = false;
+ write = false;
+ break;
+ case stxw_op:
+ size = 4;
+ sign = true;
+ write = true;
+ break;
+ case ldxd_op:
+ size = 8;
+ sign = true;
+ write = false;
+ break;
+ case stxd_op:
+ size = 8;
+ sign = true;
+ write = true;
+ break;
+ case fldxs_op:
+ size = 4;
+ fp = true;
+ sign = true;
+ write = false;
+ break;
+ case fstxs_op:
+ size = 4;
+ fp = true;
+ sign = true;
+ write = true;
+ break;
+ case fldxd_op:
+ size = 8;
+ fp = true;
+ sign = true;
+ write = false;
+ break;
+ case fstxd_op:
+ size = 8;
+ fp = true;
+ sign = true;
+ write = true;
+ break;
+ }
+
+ if (!size)
+ goto sigbus;
+ if (user && !access_ok(addr, size))
+ goto sigbus;
+
+ if (!write) {
+ res = unaligned_read(addr, &value, size, sign);
+ if (res)
+ goto fault;
+
+ /* Rd is the same field in any formats */
+ if (!fp)
+ regs->regs[insn.reg3_format.rd] = value;
+ else {
+ if (is_fpu_owner())
+ write_fpr(insn.reg3_format.rd, value);
+ else
+ set_fpr64(¤t->thread.fpu.fpr[insn.reg3_format.rd], 0, value);
+ }
+ } else {
+ /* Rd is the same field in any formats */
+ if (!fp)
+ value = regs->regs[insn.reg3_format.rd];
+ else {
+ if (is_fpu_owner())
+ value = read_fpr(insn.reg3_format.rd);
+ else
+ value = get_fpr64(¤t->thread.fpu.fpr[insn.reg3_format.rd], 0);
+ }
+
+ res = unaligned_write(addr, value, size);
+ if (res)
+ goto fault;
+ }
+
+#ifdef CONFIG_DEBUG_FS
+ if (user)
+ unaligned_instructions_user++;
+ else
+ unaligned_instructions_kernel++;
+#endif
+
+ compute_return_era(regs);
+
+ return;
+
+fault:
+ /* Did we have an exception handler installed? */
+ if (fixup_exception(regs))
+ return;
+
+ die_if_kernel("Unhandled kernel unaligned access", regs);
+ force_sig(SIGSEGV);
+
+ return;
+
+sigbus:
+ die_if_kernel("Unhandled kernel unaligned access", regs);
+ force_sig(SIGBUS);
+
+ return;
+}
+
+#ifdef CONFIG_DEBUG_FS
+static int __init debugfs_unaligned(void)
+{
+ struct dentry *d;
+
+ d = debugfs_create_dir("loongarch", NULL);
+ if (!d)
+ return -ENOMEM;
+
+ debugfs_create_u32("unaligned_instructions_user",
+ S_IRUGO, d, &unaligned_instructions_user);
+ debugfs_create_u32("unaligned_instructions_kernel",
+ S_IRUGO, d, &unaligned_instructions_kernel);
+
+ return 0;
+}
+arch_initcall(debugfs_unaligned);
+#endif
diff --git a/arch/loongarch/kernel/unwind_guess.c b/arch/loongarch/kernel/unwind_guess.c
index 5afa606..e2d2e4f 100644
--- a/arch/loongarch/kernel/unwind_guess.c
+++ b/arch/loongarch/kernel/unwind_guess.c
@@ -3,6 +3,7 @@
* Copyright (C) 2022 Loongson Technology Corporation Limited
*/
#include <linux/kernel.h>
+#include <linux/ftrace.h>
#include <asm/unwind.h>
@@ -53,7 +54,8 @@ bool unwind_next_frame(struct unwind_state *state)
state->sp < info->end;
state->sp += sizeof(unsigned long)) {
addr = *(unsigned long *)(state->sp);
-
+ state->pc = ftrace_graph_ret_addr(state->task, &state->graph_idx,
+ addr, (unsigned long *)(state->sp - GRAPH_FAKE_OFFSET));
if (__kernel_text_address(addr))
return true;
}
diff --git a/arch/loongarch/kernel/unwind_prologue.c b/arch/loongarch/kernel/unwind_prologue.c
index 4571c3c..0f8d145 100644
--- a/arch/loongarch/kernel/unwind_prologue.c
+++ b/arch/loongarch/kernel/unwind_prologue.c
@@ -2,12 +2,23 @@
/*
* Copyright (C) 2022 Loongson Technology Corporation Limited
*/
+#include <linux/ftrace.h>
#include <linux/kallsyms.h>
#include <asm/inst.h>
#include <asm/ptrace.h>
#include <asm/unwind.h>
+static inline void unwind_state_fixup(struct unwind_state *state)
+{
+#ifdef CONFIG_DYNAMIC_FTRACE
+ static unsigned long ftrace = (unsigned long)ftrace_call + 4;
+
+ if (state->pc == ftrace)
+ state->is_ftrace = true;
+#endif
+}
+
unsigned long unwind_get_return_address(struct unwind_state *state)
{
@@ -32,6 +43,8 @@ static bool unwind_by_guess(struct unwind_state *state)
state->sp < info->end;
state->sp += sizeof(unsigned long)) {
addr = *(unsigned long *)(state->sp);
+ state->pc = ftrace_graph_ret_addr(state->task, &state->graph_idx,
+ addr, (unsigned long *)(state->sp - GRAPH_FAKE_OFFSET));
if (__kernel_text_address(addr))
return true;
}
@@ -41,15 +54,30 @@ static bool unwind_by_guess(struct unwind_state *state)
static bool unwind_by_prologue(struct unwind_state *state)
{
- struct stack_info *info = &state->stack_info;
- union loongarch_instruction *ip, *ip_end;
long frame_ra = -1;
unsigned long frame_size = 0;
unsigned long size, offset, pc = state->pc;
+ struct pt_regs *regs;
+ struct stack_info *info = &state->stack_info;
+ union loongarch_instruction *ip, *ip_end;
if (state->sp >= info->end || state->sp < info->begin)
return false;
+ if (state->is_ftrace) {
+ /*
+ * As we meet ftrace_regs_entry, reset first flag like first doing
+ * tracing. Prologue analysis will stop soon because PC is at entry.
+ */
+ regs = (struct pt_regs *)state->sp;
+ state->first = true;
+ state->is_ftrace = false;
+ state->pc = regs->csr_era;
+ state->ra = regs->regs[1];
+ state->sp = regs->regs[3];
+ return true;
+ }
+
if (!kallsyms_lookup_size_offset(pc, &size, &offset))
return false;
@@ -95,7 +123,7 @@ static bool unwind_by_prologue(struct unwind_state *state)
state->pc = *(unsigned long *)(state->sp + frame_ra);
state->sp = state->sp + frame_size;
- return !!__kernel_text_address(state->pc);
+ goto out;
first:
state->first = false;
@@ -104,7 +132,9 @@ static bool unwind_by_prologue(struct unwind_state *state)
state->pc = state->ra;
- return !!__kernel_text_address(state->ra);
+out:
+ unwind_state_fixup(state);
+ return !!__kernel_text_address(state->pc);
}
void unwind_start(struct unwind_state *state, struct task_struct *task,
@@ -147,8 +177,11 @@ bool unwind_next_frame(struct unwind_state *state)
break;
case UNWINDER_PROLOGUE:
- if (unwind_by_prologue(state))
+ if (unwind_by_prologue(state)) {
+ state->pc = ftrace_graph_ret_addr(state->task, &state->graph_idx,
+ state->pc, (unsigned long *)(state->sp - GRAPH_FAKE_OFFSET));
return true;
+ }
if (info->type == STACK_TYPE_IRQ &&
info->end == state->sp) {
@@ -158,10 +191,11 @@ bool unwind_next_frame(struct unwind_state *state)
if (user_mode(regs) || !__kernel_text_address(pc))
return false;
- state->pc = pc;
- state->sp = regs->regs[3];
- state->ra = regs->regs[1];
state->first = true;
+ state->ra = regs->regs[1];
+ state->sp = regs->regs[3];
+ state->pc = ftrace_graph_ret_addr(state->task, &state->graph_idx,
+ pc, (unsigned long *)(state->sp - GRAPH_FAKE_OFFSET));
get_stack_info(state->sp, state->task, info);
return true;
diff --git a/arch/loongarch/kernel/vmlinux.lds.S b/arch/loongarch/kernel/vmlinux.lds.S
index b3309a5..733b16e 100644
--- a/arch/loongarch/kernel/vmlinux.lds.S
+++ b/arch/loongarch/kernel/vmlinux.lds.S
@@ -4,6 +4,7 @@
#include <asm/thread_info.h>
#define PAGE_SIZE _PAGE_SIZE
+#define RO_EXCEPTION_TABLE_ALIGN 4
/*
* Put .bss..swapper_pg_dir as the first thing in .bss. This will
@@ -53,7 +54,17 @@
. = ALIGN(PECOFF_SEGMENT_ALIGN);
_etext = .;
- EXCEPTION_TABLE(16)
+ /*
+ * struct alt_inst entries. From the header (alternative.h):
+ * "Alternative instructions for different CPU types or capabilities"
+ * Think locking instructions on spinlocks.
+ */
+ . = ALIGN(4);
+ .altinstructions : AT(ADDR(.altinstructions) - LOAD_OFFSET) {
+ __alt_instructions = .;
+ *(.altinstructions)
+ __alt_instructions_end = .;
+ }
.got : ALIGN(16) { *(.got) }
.plt : ALIGN(16) { *(.plt) }
diff --git a/arch/loongarch/lib/Makefile b/arch/loongarch/lib/Makefile
index e36635f..40bde63 100644
--- a/arch/loongarch/lib/Makefile
+++ b/arch/loongarch/lib/Makefile
@@ -3,4 +3,5 @@
# Makefile for LoongArch-specific library files.
#
-lib-y += delay.o clear_user.o copy_user.o dump_tlb.o
+lib-y += delay.o memset.o memcpy.o memmove.o \
+ clear_user.o copy_user.o dump_tlb.o unaligned.o
diff --git a/arch/loongarch/lib/clear_user.S b/arch/loongarch/lib/clear_user.S
index 16ba2b8..2dc48e6 100644
--- a/arch/loongarch/lib/clear_user.S
+++ b/arch/loongarch/lib/clear_user.S
@@ -3,30 +3,37 @@
* Copyright (C) 2020-2022 Loongson Technology Corporation Limited
*/
+#include <asm/alternative-asm.h>
#include <asm/asm.h>
#include <asm/asmmacro.h>
+#include <asm/asm-extable.h>
+#include <asm/cpu.h>
#include <asm/export.h>
#include <asm/regdef.h>
-.macro fixup_ex from, to, offset, fix
-.if \fix
- .section .fixup, "ax"
-\to: addi.d a0, a1, \offset
+.irp to, 0, 1, 2, 3, 4, 5, 6, 7
+.L_fixup_handle_\to\():
+ addi.d a0, a1, (\to) * (-8)
jr ra
- .previous
-.endif
- .section __ex_table, "a"
- PTR \from\()b, \to\()b
- .previous
-.endm
+.endr
+
+SYM_FUNC_START(__clear_user)
+ /*
+ * Some CPUs support hardware unaligned access
+ */
+ ALTERNATIVE "b __clear_user_generic", \
+ "b __clear_user_fast", CPU_FEATURE_UAL
+SYM_FUNC_END(__clear_user)
+
+EXPORT_SYMBOL(__clear_user)
/*
- * unsigned long __clear_user(void *addr, size_t size)
+ * unsigned long __clear_user_generic(void *addr, size_t size)
*
* a0: addr
* a1: size
*/
-SYM_FUNC_START(__clear_user)
+SYM_FUNC_START(__clear_user_generic)
beqz a1, 2f
1: st.b zero, a0, 0
@@ -37,7 +44,55 @@
2: move a0, a1
jr ra
- fixup_ex 1, 3, 0, 1
-SYM_FUNC_END(__clear_user)
+ _asm_extable 1b, .L_fixup_handle_0
+SYM_FUNC_END(__clear_user_generic)
-EXPORT_SYMBOL(__clear_user)
+/*
+ * unsigned long __clear_user_fast(void *addr, unsigned long size)
+ *
+ * a0: addr
+ * a1: size
+ */
+SYM_FUNC_START(__clear_user_fast)
+ beqz a1, 10f
+
+ ori a2, zero, 64
+ blt a1, a2, 9f
+
+ /* set 64 bytes at a time */
+1: st.d zero, a0, 0
+2: st.d zero, a0, 8
+3: st.d zero, a0, 16
+4: st.d zero, a0, 24
+5: st.d zero, a0, 32
+6: st.d zero, a0, 40
+7: st.d zero, a0, 48
+8: st.d zero, a0, 56
+
+ addi.d a0, a0, 64
+ addi.d a1, a1, -64
+ bge a1, a2, 1b
+
+ beqz a1, 10f
+
+ /* set the remaining bytes */
+9: st.b zero, a0, 0
+ addi.d a0, a0, 1
+ addi.d a1, a1, -1
+ bgt a1, zero, 9b
+
+ /* return */
+10: move a0, a1
+ jr ra
+
+ /* fixup and ex_table */
+ _asm_extable 1b, .L_fixup_handle_0
+ _asm_extable 2b, .L_fixup_handle_1
+ _asm_extable 3b, .L_fixup_handle_2
+ _asm_extable 4b, .L_fixup_handle_3
+ _asm_extable 5b, .L_fixup_handle_4
+ _asm_extable 6b, .L_fixup_handle_5
+ _asm_extable 7b, .L_fixup_handle_6
+ _asm_extable 8b, .L_fixup_handle_7
+ _asm_extable 9b, .L_fixup_handle_0
+SYM_FUNC_END(__clear_user_fast)
diff --git a/arch/loongarch/lib/copy_user.S b/arch/loongarch/lib/copy_user.S
index 97d2032..55ac602 100644
--- a/arch/loongarch/lib/copy_user.S
+++ b/arch/loongarch/lib/copy_user.S
@@ -3,31 +3,38 @@
* Copyright (C) 2020-2022 Loongson Technology Corporation Limited
*/
+#include <asm/alternative-asm.h>
#include <asm/asm.h>
#include <asm/asmmacro.h>
+#include <asm/asm-extable.h>
+#include <asm/cpu.h>
#include <asm/export.h>
#include <asm/regdef.h>
-.macro fixup_ex from, to, offset, fix
-.if \fix
- .section .fixup, "ax"
-\to: addi.d a0, a2, \offset
+.irp to, 0, 1, 2, 3, 4, 5, 6, 7
+.L_fixup_handle_\to\():
+ addi.d a0, a2, (\to) * (-8)
jr ra
- .previous
-.endif
- .section __ex_table, "a"
- PTR \from\()b, \to\()b
- .previous
-.endm
+.endr
+
+SYM_FUNC_START(__copy_user)
+ /*
+ * Some CPUs support hardware unaligned access
+ */
+ ALTERNATIVE "b __copy_user_generic", \
+ "b __copy_user_fast", CPU_FEATURE_UAL
+SYM_FUNC_END(__copy_user)
+
+EXPORT_SYMBOL(__copy_user)
/*
- * unsigned long __copy_user(void *to, const void *from, size_t n)
+ * unsigned long __copy_user_generic(void *to, const void *from, size_t n)
*
* a0: to
* a1: from
* a2: n
*/
-SYM_FUNC_START(__copy_user)
+SYM_FUNC_START(__copy_user_generic)
beqz a2, 3f
1: ld.b t0, a1, 0
@@ -40,8 +47,77 @@
3: move a0, a2
jr ra
- fixup_ex 1, 4, 0, 1
- fixup_ex 2, 4, 0, 0
-SYM_FUNC_END(__copy_user)
+ _asm_extable 1b, .L_fixup_handle_0
+ _asm_extable 2b, .L_fixup_handle_0
+SYM_FUNC_END(__copy_user_generic)
-EXPORT_SYMBOL(__copy_user)
+/*
+ * unsigned long __copy_user_fast(void *to, const void *from, unsigned long n)
+ *
+ * a0: to
+ * a1: from
+ * a2: n
+ */
+SYM_FUNC_START(__copy_user_fast)
+ beqz a2, 19f
+
+ ori a3, zero, 64
+ blt a2, a3, 17f
+
+ /* copy 64 bytes at a time */
+1: ld.d t0, a1, 0
+2: ld.d t1, a1, 8
+3: ld.d t2, a1, 16
+4: ld.d t3, a1, 24
+5: ld.d t4, a1, 32
+6: ld.d t5, a1, 40
+7: ld.d t6, a1, 48
+8: ld.d t7, a1, 56
+9: st.d t0, a0, 0
+10: st.d t1, a0, 8
+11: st.d t2, a0, 16
+12: st.d t3, a0, 24
+13: st.d t4, a0, 32
+14: st.d t5, a0, 40
+15: st.d t6, a0, 48
+16: st.d t7, a0, 56
+
+ addi.d a0, a0, 64
+ addi.d a1, a1, 64
+ addi.d a2, a2, -64
+ bge a2, a3, 1b
+
+ beqz a2, 19f
+
+ /* copy the remaining bytes */
+17: ld.b t0, a1, 0
+18: st.b t0, a0, 0
+ addi.d a0, a0, 1
+ addi.d a1, a1, 1
+ addi.d a2, a2, -1
+ bgt a2, zero, 17b
+
+ /* return */
+19: move a0, a2
+ jr ra
+
+ /* fixup and ex_table */
+ _asm_extable 1b, .L_fixup_handle_0
+ _asm_extable 2b, .L_fixup_handle_1
+ _asm_extable 3b, .L_fixup_handle_2
+ _asm_extable 4b, .L_fixup_handle_3
+ _asm_extable 5b, .L_fixup_handle_4
+ _asm_extable 6b, .L_fixup_handle_5
+ _asm_extable 7b, .L_fixup_handle_6
+ _asm_extable 8b, .L_fixup_handle_7
+ _asm_extable 9b, .L_fixup_handle_0
+ _asm_extable 10b, .L_fixup_handle_1
+ _asm_extable 11b, .L_fixup_handle_2
+ _asm_extable 12b, .L_fixup_handle_3
+ _asm_extable 13b, .L_fixup_handle_4
+ _asm_extable 14b, .L_fixup_handle_5
+ _asm_extable 15b, .L_fixup_handle_6
+ _asm_extable 16b, .L_fixup_handle_7
+ _asm_extable 17b, .L_fixup_handle_0
+ _asm_extable 18b, .L_fixup_handle_0
+SYM_FUNC_END(__copy_user_fast)
diff --git a/arch/loongarch/lib/memcpy.S b/arch/loongarch/lib/memcpy.S
new file mode 100644
index 0000000..7c07d59
--- /dev/null
+++ b/arch/loongarch/lib/memcpy.S
@@ -0,0 +1,95 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (C) 2020-2022 Loongson Technology Corporation Limited
+ */
+
+#include <asm/alternative-asm.h>
+#include <asm/asm.h>
+#include <asm/asmmacro.h>
+#include <asm/cpu.h>
+#include <asm/export.h>
+#include <asm/regdef.h>
+
+SYM_FUNC_START(memcpy)
+ /*
+ * Some CPUs support hardware unaligned access
+ */
+ ALTERNATIVE "b __memcpy_generic", \
+ "b __memcpy_fast", CPU_FEATURE_UAL
+SYM_FUNC_END(memcpy)
+
+EXPORT_SYMBOL(memcpy)
+
+/*
+ * void *__memcpy_generic(void *dst, const void *src, size_t n)
+ *
+ * a0: dst
+ * a1: src
+ * a2: n
+ */
+SYM_FUNC_START(__memcpy_generic)
+ move a3, a0
+ beqz a2, 2f
+
+1: ld.b t0, a1, 0
+ st.b t0, a0, 0
+ addi.d a0, a0, 1
+ addi.d a1, a1, 1
+ addi.d a2, a2, -1
+ bgt a2, zero, 1b
+
+2: move a0, a3
+ jr ra
+SYM_FUNC_END(__memcpy_generic)
+
+/*
+ * void *__memcpy_fast(void *dst, const void *src, size_t n)
+ *
+ * a0: dst
+ * a1: src
+ * a2: n
+ */
+SYM_FUNC_START(__memcpy_fast)
+ move a3, a0
+ beqz a2, 3f
+
+ ori a4, zero, 64
+ blt a2, a4, 2f
+
+ /* copy 64 bytes at a time */
+1: ld.d t0, a1, 0
+ ld.d t1, a1, 8
+ ld.d t2, a1, 16
+ ld.d t3, a1, 24
+ ld.d t4, a1, 32
+ ld.d t5, a1, 40
+ ld.d t6, a1, 48
+ ld.d t7, a1, 56
+ st.d t0, a0, 0
+ st.d t1, a0, 8
+ st.d t2, a0, 16
+ st.d t3, a0, 24
+ st.d t4, a0, 32
+ st.d t5, a0, 40
+ st.d t6, a0, 48
+ st.d t7, a0, 56
+
+ addi.d a0, a0, 64
+ addi.d a1, a1, 64
+ addi.d a2, a2, -64
+ bge a2, a4, 1b
+
+ beqz a2, 3f
+
+ /* copy the remaining bytes */
+2: ld.b t0, a1, 0
+ st.b t0, a0, 0
+ addi.d a0, a0, 1
+ addi.d a1, a1, 1
+ addi.d a2, a2, -1
+ bgt a2, zero, 2b
+
+ /* return */
+3: move a0, a3
+ jr ra
+SYM_FUNC_END(__memcpy_fast)
diff --git a/arch/loongarch/lib/memmove.S b/arch/loongarch/lib/memmove.S
new file mode 100644
index 0000000..6ffdb46
--- /dev/null
+++ b/arch/loongarch/lib/memmove.S
@@ -0,0 +1,121 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (C) 2020-2022 Loongson Technology Corporation Limited
+ */
+
+#include <asm/alternative-asm.h>
+#include <asm/asm.h>
+#include <asm/asmmacro.h>
+#include <asm/cpu.h>
+#include <asm/export.h>
+#include <asm/regdef.h>
+
+SYM_FUNC_START(memmove)
+ blt a0, a1, 1f /* dst < src, memcpy */
+ blt a1, a0, 3f /* src < dst, rmemcpy */
+ jr ra /* dst == src, return */
+
+ /* if (src - dst) < 64, copy 1 byte at a time */
+1: ori a3, zero, 64
+ sub.d t0, a1, a0
+ blt t0, a3, 2f
+ b memcpy
+2: b __memcpy_generic
+
+ /* if (dst - src) < 64, copy 1 byte at a time */
+3: ori a3, zero, 64
+ sub.d t0, a0, a1
+ blt t0, a3, 4f
+ b rmemcpy
+4: b __rmemcpy_generic
+SYM_FUNC_END(memmove)
+
+EXPORT_SYMBOL(memmove)
+
+SYM_FUNC_START(rmemcpy)
+ /*
+ * Some CPUs support hardware unaligned access
+ */
+ ALTERNATIVE "b __rmemcpy_generic", \
+ "b __rmemcpy_fast", CPU_FEATURE_UAL
+SYM_FUNC_END(rmemcpy)
+
+/*
+ * void *__rmemcpy_generic(void *dst, const void *src, size_t n)
+ *
+ * a0: dst
+ * a1: src
+ * a2: n
+ */
+SYM_FUNC_START(__rmemcpy_generic)
+ move a3, a0
+ beqz a2, 2f
+
+ add.d a0, a0, a2
+ add.d a1, a1, a2
+
+1: ld.b t0, a1, -1
+ st.b t0, a0, -1
+ addi.d a0, a0, -1
+ addi.d a1, a1, -1
+ addi.d a2, a2, -1
+ bgt a2, zero, 1b
+
+2: move a0, a3
+ jr ra
+SYM_FUNC_END(__rmemcpy_generic)
+
+/*
+ * void *__rmemcpy_fast(void *dst, const void *src, size_t n)
+ *
+ * a0: dst
+ * a1: src
+ * a2: n
+ */
+SYM_FUNC_START(__rmemcpy_fast)
+ move a3, a0
+ beqz a2, 3f
+
+ add.d a0, a0, a2
+ add.d a1, a1, a2
+
+ ori a4, zero, 64
+ blt a2, a4, 2f
+
+ /* copy 64 bytes at a time */
+1: ld.d t0, a1, -8
+ ld.d t1, a1, -16
+ ld.d t2, a1, -24
+ ld.d t3, a1, -32
+ ld.d t4, a1, -40
+ ld.d t5, a1, -48
+ ld.d t6, a1, -56
+ ld.d t7, a1, -64
+ st.d t0, a0, -8
+ st.d t1, a0, -16
+ st.d t2, a0, -24
+ st.d t3, a0, -32
+ st.d t4, a0, -40
+ st.d t5, a0, -48
+ st.d t6, a0, -56
+ st.d t7, a0, -64
+
+ addi.d a0, a0, -64
+ addi.d a1, a1, -64
+ addi.d a2, a2, -64
+ bge a2, a4, 1b
+
+ beqz a2, 3f
+
+ /* copy the remaining bytes */
+2: ld.b t0, a1, -1
+ st.b t0, a0, -1
+ addi.d a0, a0, -1
+ addi.d a1, a1, -1
+ addi.d a2, a2, -1
+ bgt a2, zero, 2b
+
+ /* return */
+3: move a0, a3
+ jr ra
+SYM_FUNC_END(__rmemcpy_fast)
diff --git a/arch/loongarch/lib/memset.S b/arch/loongarch/lib/memset.S
new file mode 100644
index 0000000..e7cb4ea3
--- /dev/null
+++ b/arch/loongarch/lib/memset.S
@@ -0,0 +1,91 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (C) 2020-2022 Loongson Technology Corporation Limited
+ */
+
+#include <asm/alternative-asm.h>
+#include <asm/asm.h>
+#include <asm/asmmacro.h>
+#include <asm/cpu.h>
+#include <asm/export.h>
+#include <asm/regdef.h>
+
+.macro fill_to_64 r0
+ bstrins.d \r0, \r0, 15, 8
+ bstrins.d \r0, \r0, 31, 16
+ bstrins.d \r0, \r0, 63, 32
+.endm
+
+SYM_FUNC_START(memset)
+ /*
+ * Some CPUs support hardware unaligned access
+ */
+ ALTERNATIVE "b __memset_generic", \
+ "b __memset_fast", CPU_FEATURE_UAL
+SYM_FUNC_END(memset)
+
+EXPORT_SYMBOL(memset)
+
+/*
+ * void *__memset_generic(void *s, int c, size_t n)
+ *
+ * a0: s
+ * a1: c
+ * a2: n
+ */
+SYM_FUNC_START(__memset_generic)
+ move a3, a0
+ beqz a2, 2f
+
+1: st.b a1, a0, 0
+ addi.d a0, a0, 1
+ addi.d a2, a2, -1
+ bgt a2, zero, 1b
+
+2: move a0, a3
+ jr ra
+SYM_FUNC_END(__memset_generic)
+
+/*
+ * void *__memset_fast(void *s, int c, size_t n)
+ *
+ * a0: s
+ * a1: c
+ * a2: n
+ */
+SYM_FUNC_START(__memset_fast)
+ move a3, a0
+ beqz a2, 3f
+
+ ori a4, zero, 64
+ blt a2, a4, 2f
+
+ /* fill a1 to 64 bits */
+ fill_to_64 a1
+
+ /* set 64 bytes at a time */
+1: st.d a1, a0, 0
+ st.d a1, a0, 8
+ st.d a1, a0, 16
+ st.d a1, a0, 24
+ st.d a1, a0, 32
+ st.d a1, a0, 40
+ st.d a1, a0, 48
+ st.d a1, a0, 56
+
+ addi.d a0, a0, 64
+ addi.d a2, a2, -64
+ bge a2, a4, 1b
+
+ beqz a2, 3f
+
+ /* set the remaining bytes */
+2: st.b a1, a0, 0
+ addi.d a0, a0, 1
+ addi.d a2, a2, -1
+ bgt a2, zero, 2b
+
+ /* return */
+3: move a0, a3
+ jr ra
+SYM_FUNC_END(__memset_fast)
diff --git a/arch/loongarch/lib/unaligned.S b/arch/loongarch/lib/unaligned.S
new file mode 100644
index 0000000..9177fd6
--- /dev/null
+++ b/arch/loongarch/lib/unaligned.S
@@ -0,0 +1,84 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (C) 2020-2022 Loongson Technology Corporation Limited
+ */
+
+#include <linux/linkage.h>
+
+#include <asm/asm.h>
+#include <asm/asmmacro.h>
+#include <asm/asm-extable.h>
+#include <asm/errno.h>
+#include <asm/export.h>
+#include <asm/regdef.h>
+
+.L_fixup_handle_unaligned:
+ li.w a0, -EFAULT
+ jr ra
+
+/*
+ * unsigned long unaligned_read(void *addr, void *value, unsigned long n, bool sign)
+ *
+ * a0: addr
+ * a1: value
+ * a2: n
+ * a3: sign
+ */
+SYM_FUNC_START(unaligned_read)
+ beqz a2, 5f
+
+ li.w t2, 0
+ addi.d t0, a2, -1
+ slli.d t1, t0, 3
+ add.d a0, a0, t0
+
+ beqz a3, 2f
+1: ld.b t3, a0, 0
+ b 3f
+
+2: ld.bu t3, a0, 0
+3: sll.d t3, t3, t1
+ or t2, t2, t3
+ addi.d t1, t1, -8
+ addi.d a0, a0, -1
+ addi.d a2, a2, -1
+ bgtz a2, 2b
+4: st.d t2, a1, 0
+
+ move a0, a2
+ jr ra
+
+5: li.w a0, -EFAULT
+ jr ra
+
+ _asm_extable 1b, .L_fixup_handle_unaligned
+ _asm_extable 2b, .L_fixup_handle_unaligned
+ _asm_extable 4b, .L_fixup_handle_unaligned
+SYM_FUNC_END(unaligned_read)
+
+/*
+ * unsigned long unaligned_write(void *addr, unsigned long value, unsigned long n)
+ *
+ * a0: addr
+ * a1: value
+ * a2: n
+ */
+SYM_FUNC_START(unaligned_write)
+ beqz a2, 3f
+
+ li.w t0, 0
+1: srl.d t1, a1, t0
+2: st.b t1, a0, 0
+ addi.d t0, t0, 8
+ addi.d a2, a2, -1
+ addi.d a0, a0, 1
+ bgtz a2, 1b
+
+ move a0, a2
+ jr ra
+
+3: li.w a0, -EFAULT
+ jr ra
+
+ _asm_extable 2b, .L_fixup_handle_unaligned
+SYM_FUNC_END(unaligned_write)
diff --git a/arch/loongarch/mm/extable.c b/arch/loongarch/mm/extable.c
index bc20988..9ab6987 100644
--- a/arch/loongarch/mm/extable.c
+++ b/arch/loongarch/mm/extable.c
@@ -2,21 +2,62 @@
/*
* Copyright (C) 2020-2022 Loongson Technology Corporation Limited
*/
+#include <linux/bitfield.h>
#include <linux/extable.h>
-#include <linux/spinlock.h>
-#include <asm/branch.h>
#include <linux/uaccess.h>
+#include <asm/asm-extable.h>
+#include <asm/branch.h>
-int fixup_exception(struct pt_regs *regs)
+static inline unsigned long
+get_ex_fixup(const struct exception_table_entry *ex)
{
- const struct exception_table_entry *fixup;
+ return ((unsigned long)&ex->fixup + ex->fixup);
+}
- fixup = search_exception_tables(exception_era(regs));
- if (fixup) {
- regs->csr_era = fixup->fixup;
+static inline void regs_set_gpr(struct pt_regs *regs,
+ unsigned int offset, unsigned long val)
+{
+ if (offset && offset <= MAX_REG_OFFSET)
+ *(unsigned long *)((unsigned long)regs + offset) = val;
+}
- return 1;
+static bool ex_handler_fixup(const struct exception_table_entry *ex,
+ struct pt_regs *regs)
+{
+ regs->csr_era = get_ex_fixup(ex);
+
+ return true;
+}
+
+static bool ex_handler_uaccess_err_zero(const struct exception_table_entry *ex,
+ struct pt_regs *regs)
+{
+ int reg_err = FIELD_GET(EX_DATA_REG_ERR, ex->data);
+ int reg_zero = FIELD_GET(EX_DATA_REG_ZERO, ex->data);
+
+ regs_set_gpr(regs, reg_err * sizeof(unsigned long), -EFAULT);
+ regs_set_gpr(regs, reg_zero * sizeof(unsigned long), 0);
+ regs->csr_era = get_ex_fixup(ex);
+
+ return true;
+}
+
+bool fixup_exception(struct pt_regs *regs)
+{
+ const struct exception_table_entry *ex;
+
+ ex = search_exception_tables(exception_era(regs));
+ if (!ex)
+ return false;
+
+ switch (ex->type) {
+ case EX_TYPE_FIXUP:
+ return ex_handler_fixup(ex, regs);
+ case EX_TYPE_UACCESS_ERR_ZERO:
+ return ex_handler_uaccess_err_zero(ex, regs);
+ case EX_TYPE_BPF:
+ return ex_handler_bpf(ex, regs);
}
- return 0;
+ BUG();
}
diff --git a/arch/loongarch/net/bpf_jit.c b/arch/loongarch/net/bpf_jit.c
index bdcd0c7..c4b1947 100644
--- a/arch/loongarch/net/bpf_jit.c
+++ b/arch/loongarch/net/bpf_jit.c
@@ -387,6 +387,65 @@ static bool is_signed_bpf_cond(u8 cond)
cond == BPF_JSGE || cond == BPF_JSLE;
}
+#define BPF_FIXUP_REG_MASK GENMASK(31, 27)
+#define BPF_FIXUP_OFFSET_MASK GENMASK(26, 0)
+
+bool ex_handler_bpf(const struct exception_table_entry *ex,
+ struct pt_regs *regs)
+{
+ int dst_reg = FIELD_GET(BPF_FIXUP_REG_MASK, ex->fixup);
+ off_t offset = FIELD_GET(BPF_FIXUP_OFFSET_MASK, ex->fixup);
+
+ regs->regs[dst_reg] = 0;
+ regs->csr_era = (unsigned long)&ex->fixup - offset;
+
+ return true;
+}
+
+/* For accesses to BTF pointers, add an entry to the exception table */
+static int add_exception_handler(const struct bpf_insn *insn,
+ struct jit_ctx *ctx,
+ int dst_reg)
+{
+ unsigned long pc;
+ off_t offset;
+ struct exception_table_entry *ex;
+
+ if (!ctx->image || !ctx->prog->aux->extable || BPF_MODE(insn->code) != BPF_PROBE_MEM)
+ return 0;
+
+ if (WARN_ON_ONCE(ctx->num_exentries >= ctx->prog->aux->num_exentries))
+ return -EINVAL;
+
+ ex = &ctx->prog->aux->extable[ctx->num_exentries];
+ pc = (unsigned long)&ctx->image[ctx->idx - 1];
+
+ offset = pc - (long)&ex->insn;
+ if (WARN_ON_ONCE(offset >= 0 || offset < INT_MIN))
+ return -ERANGE;
+
+ ex->insn = offset;
+
+ /*
+ * Since the extable follows the program, the fixup offset is always
+ * negative and limited to BPF_JIT_REGION_SIZE. Store a positive value
+ * to keep things simple, and put the destination register in the upper
+ * bits. We don't need to worry about buildtime or runtime sort
+ * modifying the upper bits because the table is already sorted, and
+ * isn't part of the main exception table.
+ */
+ offset = (long)&ex->fixup - (pc + LOONGARCH_INSN_SIZE);
+ if (!FIELD_FIT(BPF_FIXUP_OFFSET_MASK, offset))
+ return -ERANGE;
+
+ ex->type = EX_TYPE_BPF;
+ ex->fixup = FIELD_PREP(BPF_FIXUP_OFFSET_MASK, offset) | FIELD_PREP(BPF_FIXUP_REG_MASK, dst_reg);
+
+ ctx->num_exentries++;
+
+ return 0;
+}
+
static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, bool extra_pass)
{
u8 tm = -1;
@@ -816,6 +875,10 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, bool ext
case BPF_LDX | BPF_MEM | BPF_H:
case BPF_LDX | BPF_MEM | BPF_W:
case BPF_LDX | BPF_MEM | BPF_DW:
+ case BPF_LDX | BPF_PROBE_MEM | BPF_DW:
+ case BPF_LDX | BPF_PROBE_MEM | BPF_W:
+ case BPF_LDX | BPF_PROBE_MEM | BPF_H:
+ case BPF_LDX | BPF_PROBE_MEM | BPF_B:
switch (BPF_SIZE(code)) {
case BPF_B:
if (is_signed_imm12(off)) {
@@ -854,6 +917,10 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, bool ext
}
break;
}
+
+ ret = add_exception_handler(insn, ctx, dst);
+ if (ret)
+ return ret;
break;
/* *(size *)(dst + off) = imm */
@@ -1018,6 +1085,9 @@ static int validate_code(struct jit_ctx *ctx)
return -1;
}
+ if (WARN_ON_ONCE(ctx->num_exentries != ctx->prog->aux->num_exentries))
+ return -1;
+
return 0;
}
@@ -1025,7 +1095,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
{
bool tmp_blinded = false, extra_pass = false;
u8 *image_ptr;
- int image_size;
+ int image_size, prog_size, extable_size;
struct jit_ctx ctx;
struct jit_data *jit_data;
struct bpf_binary_header *header;
@@ -1066,7 +1136,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
image_ptr = jit_data->image;
header = jit_data->header;
extra_pass = true;
- image_size = sizeof(u32) * ctx.idx;
+ prog_size = sizeof(u32) * ctx.idx;
goto skip_init_ctx;
}
@@ -1088,12 +1158,15 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
ctx.epilogue_offset = ctx.idx;
build_epilogue(&ctx);
+ extable_size = prog->aux->num_exentries * sizeof(struct exception_table_entry);
+
/* Now we know the actual image size.
* As each LoongArch instruction is of length 32bit,
* we are translating number of JITed intructions into
* the size required to store these JITed code.
*/
- image_size = sizeof(u32) * ctx.idx;
+ prog_size = sizeof(u32) * ctx.idx;
+ image_size = prog_size + extable_size;
/* Now we know the size of the structure to make */
header = bpf_jit_binary_alloc(image_size, &image_ptr,
sizeof(u32), jit_fill_hole);
@@ -1104,9 +1177,12 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
/* 2. Now, the actual pass to generate final JIT code */
ctx.image = (union loongarch_instruction *)image_ptr;
+ if (extable_size)
+ prog->aux->extable = (void *)image_ptr + prog_size;
skip_init_ctx:
ctx.idx = 0;
+ ctx.num_exentries = 0;
build_prologue(&ctx);
if (build_body(&ctx, extra_pass)) {
@@ -1125,7 +1201,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
/* And we're done */
if (bpf_jit_enable > 1)
- bpf_jit_dump(prog->len, image_size, 2, ctx.image);
+ bpf_jit_dump(prog->len, prog_size, 2, ctx.image);
/* Update the icache */
flush_icache_range((unsigned long)header, (unsigned long)(ctx.image + ctx.idx));
@@ -1147,7 +1223,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
jit_data->header = header;
}
prog->jited = 1;
- prog->jited_len = image_size;
+ prog->jited_len = prog_size;
prog->bpf_func = (void *)ctx.image;
if (!prog->is_func || extra_pass) {
diff --git a/arch/loongarch/net/bpf_jit.h b/arch/loongarch/net/bpf_jit.h
index e665ddb..ca70802 100644
--- a/arch/loongarch/net/bpf_jit.h
+++ b/arch/loongarch/net/bpf_jit.h
@@ -4,6 +4,7 @@
*
* Copyright (C) 2022 Loongson Technology Corporation Limited
*/
+#include <linux/bitfield.h>
#include <linux/bpf.h>
#include <linux/filter.h>
#include <asm/cacheflush.h>
@@ -15,6 +16,7 @@ struct jit_ctx {
unsigned int flags;
unsigned int epilogue_offset;
u32 *offset;
+ int num_exentries;
union loongarch_instruction *image;
u32 stack_size;
};
diff --git a/arch/loongarch/pci/acpi.c b/arch/loongarch/pci/acpi.c
index 8235ec9..365f7de 100644
--- a/arch/loongarch/pci/acpi.c
+++ b/arch/loongarch/pci/acpi.c
@@ -26,9 +26,12 @@ void pcibios_add_bus(struct pci_bus *bus)
int pcibios_root_bridge_prepare(struct pci_host_bridge *bridge)
{
- struct pci_config_window *cfg = bridge->bus->sysdata;
- struct acpi_device *adev = to_acpi_device(cfg->parent);
+ struct acpi_device *adev = NULL;
struct device *bus_dev = &bridge->bus->dev;
+ struct pci_config_window *cfg = bridge->bus->sysdata;
+
+ if (!acpi_disabled)
+ adev = to_acpi_device(cfg->parent);
ACPI_COMPANION_SET(&bridge->dev, adev);
set_dev_node(bus_dev, pa_to_nid(cfg->res.start));
diff --git a/arch/loongarch/power/Makefile b/arch/loongarch/power/Makefile
new file mode 100644
index 0000000..58151d0
--- /dev/null
+++ b/arch/loongarch/power/Makefile
@@ -0,0 +1,4 @@
+obj-y += platform.o
+
+obj-$(CONFIG_SUSPEND) += suspend.o suspend_asm.o
+obj-$(CONFIG_HIBERNATION) += hibernate.o hibernate_asm.o
diff --git a/arch/loongarch/power/hibernate.c b/arch/loongarch/power/hibernate.c
new file mode 100644
index 0000000..1e05905
--- /dev/null
+++ b/arch/loongarch/power/hibernate.c
@@ -0,0 +1,62 @@
+// SPDX-License-Identifier: GPL-2.0
+#include <asm/fpu.h>
+#include <asm/loongson.h>
+#include <asm/sections.h>
+#include <asm/tlbflush.h>
+#include <linux/suspend.h>
+
+static u32 saved_crmd;
+static u32 saved_prmd;
+static u32 saved_euen;
+static u32 saved_ecfg;
+static u64 saved_pcpu_base;
+struct pt_regs saved_regs;
+
+void save_processor_state(void)
+{
+ saved_crmd = csr_read32(LOONGARCH_CSR_CRMD);
+ saved_prmd = csr_read32(LOONGARCH_CSR_PRMD);
+ saved_euen = csr_read32(LOONGARCH_CSR_EUEN);
+ saved_ecfg = csr_read32(LOONGARCH_CSR_ECFG);
+ saved_pcpu_base = csr_read64(PERCPU_BASE_KS);
+
+ if (is_fpu_owner())
+ save_fp(current);
+}
+
+void restore_processor_state(void)
+{
+ csr_write32(saved_crmd, LOONGARCH_CSR_CRMD);
+ csr_write32(saved_prmd, LOONGARCH_CSR_PRMD);
+ csr_write32(saved_euen, LOONGARCH_CSR_EUEN);
+ csr_write32(saved_ecfg, LOONGARCH_CSR_ECFG);
+ csr_write64(saved_pcpu_base, PERCPU_BASE_KS);
+
+ if (is_fpu_owner())
+ restore_fp(current);
+}
+
+int pfn_is_nosave(unsigned long pfn)
+{
+ unsigned long nosave_begin_pfn = PFN_DOWN(__pa(&__nosave_begin));
+ unsigned long nosave_end_pfn = PFN_UP(__pa(&__nosave_end));
+
+ return (pfn >= nosave_begin_pfn) && (pfn < nosave_end_pfn);
+}
+
+extern int swsusp_asm_suspend(void);
+
+int swsusp_arch_suspend(void)
+{
+ enable_pci_wakeup();
+ return swsusp_asm_suspend();
+}
+
+extern int swsusp_asm_resume(void);
+
+int swsusp_arch_resume(void)
+{
+ /* Avoid TLB mismatch during and after kernel resume */
+ local_flush_tlb_all();
+ return swsusp_asm_resume();
+}
diff --git a/arch/loongarch/power/hibernate_asm.S b/arch/loongarch/power/hibernate_asm.S
new file mode 100644
index 0000000..3c747c0
--- /dev/null
+++ b/arch/loongarch/power/hibernate_asm.S
@@ -0,0 +1,66 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Hibernation support specific for LoongArch
+ *
+ * Author: Huacai Chen <chenhuacai@loongson.cn>
+ * Copyright (C) 2020-2022 Loongson Technology Corporation Limited
+ */
+#include <linux/linkage.h>
+#include <asm/asm.h>
+#include <asm/asm-offsets.h>
+#include <asm/regdef.h>
+
+.text
+SYM_FUNC_START(swsusp_asm_suspend)
+ la.pcrel t0, saved_regs
+ PTR_S ra, t0, PT_R1
+ PTR_S tp, t0, PT_R2
+ PTR_S sp, t0, PT_R3
+ PTR_S u0, t0, PT_R21
+ PTR_S fp, t0, PT_R22
+ PTR_S s0, t0, PT_R23
+ PTR_S s1, t0, PT_R24
+ PTR_S s2, t0, PT_R25
+ PTR_S s3, t0, PT_R26
+ PTR_S s4, t0, PT_R27
+ PTR_S s5, t0, PT_R28
+ PTR_S s6, t0, PT_R29
+ PTR_S s7, t0, PT_R30
+ PTR_S s8, t0, PT_R31
+ b swsusp_save
+SYM_FUNC_END(swsusp_asm_suspend)
+
+SYM_FUNC_START(swsusp_asm_resume)
+ la.pcrel t0, restore_pblist
+ PTR_L t0, t0, 0
+0:
+ PTR_L t1, t0, PBE_ADDRESS /* source */
+ PTR_L t2, t0, PBE_ORIG_ADDRESS /* destination */
+ PTR_LI t3, _PAGE_SIZE
+ PTR_ADD t3, t3, t1
+1:
+ REG_L t8, t1, 0
+ REG_S t8, t2, 0
+ PTR_ADDI t1, t1, SZREG
+ PTR_ADDI t2, t2, SZREG
+ bne t1, t3, 1b
+ PTR_L t0, t0, PBE_NEXT
+ bnez t0, 0b
+ la.pcrel t0, saved_regs
+ PTR_L ra, t0, PT_R1
+ PTR_L tp, t0, PT_R2
+ PTR_L sp, t0, PT_R3
+ PTR_L u0, t0, PT_R21
+ PTR_L fp, t0, PT_R22
+ PTR_L s0, t0, PT_R23
+ PTR_L s1, t0, PT_R24
+ PTR_L s2, t0, PT_R25
+ PTR_L s3, t0, PT_R26
+ PTR_L s4, t0, PT_R27
+ PTR_L s5, t0, PT_R28
+ PTR_L s6, t0, PT_R29
+ PTR_L s7, t0, PT_R30
+ PTR_L s8, t0, PT_R31
+ PTR_LI a0, 0x0
+ jirl zero, ra, 0
+SYM_FUNC_END(swsusp_asm_resume)
diff --git a/arch/loongarch/power/platform.c b/arch/loongarch/power/platform.c
new file mode 100644
index 0000000..3ea8e07
--- /dev/null
+++ b/arch/loongarch/power/platform.c
@@ -0,0 +1,57 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Author: Huacai Chen <chenhuacai@loongson.cn>
+ * Copyright (C) 2020-2022 Loongson Technology Corporation Limited
+ */
+#include <linux/acpi.h>
+#include <linux/platform_device.h>
+
+#include <asm/bootinfo.h>
+#include <asm/loongson.h>
+
+void enable_gpe_wakeup(void)
+{
+ if (acpi_disabled)
+ return;
+
+ if (acpi_gbl_reduced_hardware)
+ return;
+
+ acpi_enable_all_wakeup_gpes();
+}
+
+void enable_pci_wakeup(void)
+{
+ if (acpi_disabled)
+ return;
+
+ if (acpi_gbl_reduced_hardware)
+ return;
+
+ acpi_write_bit_register(ACPI_BITREG_PCIEXP_WAKE_STATUS, 1);
+
+ if (acpi_gbl_FADT.flags & ACPI_FADT_PCI_EXPRESS_WAKE)
+ acpi_write_bit_register(ACPI_BITREG_PCIEXP_WAKE_DISABLE, 0);
+}
+
+static int __init loongson3_acpi_suspend_init(void)
+{
+#ifdef CONFIG_ACPI
+ acpi_status status;
+ uint64_t suspend_addr = 0;
+
+ if (acpi_disabled || acpi_gbl_reduced_hardware)
+ return 0;
+
+ acpi_write_bit_register(ACPI_BITREG_SCI_ENABLE, 1);
+ status = acpi_evaluate_integer(NULL, "\\SADR", NULL, &suspend_addr);
+ if (ACPI_FAILURE(status) || !suspend_addr) {
+ pr_err("ACPI S3 is not support!\n");
+ return -1;
+ }
+ loongson_sysconf.suspend_addr = (u64)phys_to_virt(PHYSADDR(suspend_addr));
+#endif
+ return 0;
+}
+
+device_initcall(loongson3_acpi_suspend_init);
diff --git a/arch/loongarch/power/suspend.c b/arch/loongarch/power/suspend.c
new file mode 100644
index 0000000..5e19733
--- /dev/null
+++ b/arch/loongarch/power/suspend.c
@@ -0,0 +1,73 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * loongson-specific suspend support
+ *
+ * Author: Huacai Chen <chenhuacai@loongson.cn>
+ * Copyright (C) 2020-2022 Loongson Technology Corporation Limited
+ */
+#include <linux/acpi.h>
+#include <linux/pm.h>
+#include <linux/suspend.h>
+
+#include <asm/loongarch.h>
+#include <asm/loongson.h>
+#include <asm/setup.h>
+#include <asm/time.h>
+#include <asm/tlbflush.h>
+
+u64 loongarch_suspend_addr;
+
+struct saved_registers {
+ u32 ecfg;
+ u32 euen;
+ u64 pgd;
+ u64 kpgd;
+ u32 pwctl0;
+ u32 pwctl1;
+};
+static struct saved_registers saved_regs;
+
+static void arch_common_suspend(void)
+{
+ save_counter();
+ saved_regs.pgd = csr_read64(LOONGARCH_CSR_PGDL);
+ saved_regs.kpgd = csr_read64(LOONGARCH_CSR_PGDH);
+ saved_regs.pwctl0 = csr_read32(LOONGARCH_CSR_PWCTL0);
+ saved_regs.pwctl1 = csr_read32(LOONGARCH_CSR_PWCTL1);
+ saved_regs.ecfg = csr_read32(LOONGARCH_CSR_ECFG);
+ saved_regs.euen = csr_read32(LOONGARCH_CSR_EUEN);
+
+ loongarch_suspend_addr = loongson_sysconf.suspend_addr;
+}
+
+static void arch_common_resume(void)
+{
+ sync_counter();
+ local_flush_tlb_all();
+ csr_write64(per_cpu_offset(0), PERCPU_BASE_KS);
+ csr_write64(eentry, LOONGARCH_CSR_EENTRY);
+ csr_write64(eentry, LOONGARCH_CSR_MERRENTRY);
+ csr_write64(tlbrentry, LOONGARCH_CSR_TLBRENTRY);
+
+ csr_write64(saved_regs.pgd, LOONGARCH_CSR_PGDL);
+ csr_write64(saved_regs.kpgd, LOONGARCH_CSR_PGDH);
+ csr_write32(saved_regs.pwctl0, LOONGARCH_CSR_PWCTL0);
+ csr_write32(saved_regs.pwctl1, LOONGARCH_CSR_PWCTL1);
+ csr_write32(saved_regs.ecfg, LOONGARCH_CSR_ECFG);
+ csr_write32(saved_regs.euen, LOONGARCH_CSR_EUEN);
+}
+
+int loongarch_acpi_suspend(void)
+{
+ enable_gpe_wakeup();
+ enable_pci_wakeup();
+
+ arch_common_suspend();
+
+ /* processor specific suspend */
+ loongarch_suspend_enter();
+
+ arch_common_resume();
+
+ return 0;
+}
diff --git a/arch/loongarch/power/suspend_asm.S b/arch/loongarch/power/suspend_asm.S
new file mode 100644
index 0000000..eb26756
--- /dev/null
+++ b/arch/loongarch/power/suspend_asm.S
@@ -0,0 +1,89 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Sleep helper for Loongson-3 sleep mode.
+ *
+ * Author: Huacai Chen <chenhuacai@loongson.cn>
+ * Copyright (C) 2020-2022 Loongson Technology Corporation Limited
+ */
+
+#include <asm/asm.h>
+#include <asm/asmmacro.h>
+#include <asm/addrspace.h>
+#include <asm/loongarch.h>
+#include <asm/stackframe.h>
+
+/* preparatory stuff */
+.macro SETUP_SLEEP
+ addi.d sp, sp, -PT_SIZE
+ st.d $r1, sp, PT_R1
+ st.d $r2, sp, PT_R2
+ st.d $r3, sp, PT_R3
+ st.d $r4, sp, PT_R4
+ st.d $r21, sp, PT_R21
+ st.d $r22, sp, PT_R22
+ st.d $r23, sp, PT_R23
+ st.d $r24, sp, PT_R24
+ st.d $r25, sp, PT_R25
+ st.d $r26, sp, PT_R26
+ st.d $r27, sp, PT_R27
+ st.d $r28, sp, PT_R28
+ st.d $r29, sp, PT_R29
+ st.d $r30, sp, PT_R30
+ st.d $r31, sp, PT_R31
+
+ la.pcrel t0, acpi_saved_sp
+ st.d sp, t0, 0
+.endm
+
+.macro SETUP_WAKEUP
+ ld.d $r1, sp, PT_R1
+ ld.d $r2, sp, PT_R2
+ ld.d $r3, sp, PT_R3
+ ld.d $r4, sp, PT_R4
+ ld.d $r21, sp, PT_R21
+ ld.d $r22, sp, PT_R22
+ ld.d $r23, sp, PT_R23
+ ld.d $r24, sp, PT_R24
+ ld.d $r25, sp, PT_R25
+ ld.d $r26, sp, PT_R26
+ ld.d $r27, sp, PT_R27
+ ld.d $r28, sp, PT_R28
+ ld.d $r29, sp, PT_R29
+ ld.d $r30, sp, PT_R30
+ ld.d $r31, sp, PT_R31
+.endm
+
+ .text
+ .align 12
+
+/* Sleep/wakeup code for Loongson-3 */
+SYM_FUNC_START(loongarch_suspend_enter)
+ SETUP_SLEEP
+ bl __flush_cache_all
+
+ /* Pass RA and SP to BIOS */
+ addi.d a1, sp, 0
+ la.pcrel a0, loongarch_wakeup_start
+ la.pcrel t0, loongarch_suspend_addr
+ ld.d t0, t0, 0
+ jirl a0, t0, 0 /* Call BIOS's STR sleep routine */
+
+ /*
+ * This is where we return upon wakeup.
+ * Reload all of the registers and return.
+ */
+SYM_INNER_LABEL(loongarch_wakeup_start, SYM_L_GLOBAL)
+ li.d t0, CSR_DMW0_INIT # UC, PLV0
+ csrwr t0, LOONGARCH_CSR_DMWIN0
+ li.d t0, CSR_DMW1_INIT # CA, PLV0
+ csrwr t0, LOONGARCH_CSR_DMWIN1
+
+ la.abs t0, 0f
+ jr t0
+0:
+ la.pcrel t0, acpi_saved_sp
+ ld.d sp, t0, 0
+ SETUP_WAKEUP
+ addi.d sp, sp, PT_SIZE
+ jr ra
+SYM_FUNC_END(loongarch_suspend_enter)
diff --git a/arch/mips/Kconfig b/arch/mips/Kconfig
index b26b776..15cb692 100644
--- a/arch/mips/Kconfig
+++ b/arch/mips/Kconfig
@@ -46,7 +46,7 @@
select GENERIC_SCHED_CLOCK if !CAVIUM_OCTEON_SOC
select GENERIC_SMP_IDLE_THREAD
select GENERIC_TIME_VSYSCALL
- select GUP_GET_PTE_LOW_HIGH if CPU_MIPS32 && PHYS_ADDR_T_64BIT
+ select GUP_GET_PXX_LOW_HIGH if CPU_MIPS32 && PHYS_ADDR_T_64BIT
select HAVE_ARCH_COMPILER_H
select HAVE_ARCH_JUMP_LABEL
select HAVE_ARCH_KGDB if MIPS_FP_SUPPORT
diff --git a/arch/parisc/include/asm/pdc.h b/arch/parisc/include/asm/pdc.h
index fcbcf9a..40793be 100644
--- a/arch/parisc/include/asm/pdc.h
+++ b/arch/parisc/include/asm/pdc.h
@@ -37,7 +37,7 @@ int pdc_system_map_find_mods(struct pdc_system_map_mod_info *pdc_mod_info,
int pdc_system_map_find_addrs(struct pdc_system_map_addr_info *pdc_addr_info,
long mod_index, long addr_index);
int pdc_model_info(struct pdc_model *model);
-int pdc_model_sysmodel(char *name);
+int pdc_model_sysmodel(unsigned int os_id, char *name);
int pdc_model_cpuid(unsigned long *cpu_id);
int pdc_model_versions(unsigned long *versions, int id);
int pdc_model_capabilities(unsigned long *capabilities);
diff --git a/arch/parisc/include/asm/pgtable.h b/arch/parisc/include/asm/pgtable.h
index bd09a44..ea35743 100644
--- a/arch/parisc/include/asm/pgtable.h
+++ b/arch/parisc/include/asm/pgtable.h
@@ -151,8 +151,8 @@ extern void __update_cache(pte_t pte);
/* This calculates the number of initial pages we need for the initial
* page tables */
-#if (KERNEL_INITIAL_ORDER) >= (PMD_SHIFT)
-# define PT_INITIAL (1 << (KERNEL_INITIAL_ORDER - PMD_SHIFT))
+#if (KERNEL_INITIAL_ORDER) >= (PLD_SHIFT + BITS_PER_PTE)
+# define PT_INITIAL (1 << (KERNEL_INITIAL_ORDER - PLD_SHIFT - BITS_PER_PTE))
#else
# define PT_INITIAL (1) /* all initial PTEs fit into one page */
#endif
diff --git a/arch/parisc/include/uapi/asm/mman.h b/arch/parisc/include/uapi/asm/mman.h
index 22133a6..68c44f9 100644
--- a/arch/parisc/include/uapi/asm/mman.h
+++ b/arch/parisc/include/uapi/asm/mman.h
@@ -49,6 +49,19 @@
#define MADV_DONTFORK 10 /* don't inherit across fork */
#define MADV_DOFORK 11 /* do inherit across fork */
+#define MADV_MERGEABLE 12 /* KSM may merge identical pages */
+#define MADV_UNMERGEABLE 13 /* KSM may not merge identical pages */
+
+#define MADV_HUGEPAGE 14 /* Worth backing with hugepages */
+#define MADV_NOHUGEPAGE 15 /* Not worth backing with hugepages */
+
+#define MADV_DONTDUMP 16 /* Explicity exclude from the core dump,
+ overrides the coredump filter bits */
+#define MADV_DODUMP 17 /* Clear the MADV_NODUMP flag */
+
+#define MADV_WIPEONFORK 18 /* Zero memory on fork, child only */
+#define MADV_KEEPONFORK 19 /* Undo MADV_WIPEONFORK */
+
#define MADV_COLD 20 /* deactivate these pages */
#define MADV_PAGEOUT 21 /* reclaim these pages */
@@ -57,27 +70,13 @@
#define MADV_DONTNEED_LOCKED 24 /* like DONTNEED, but drop locked pages too */
-#define MADV_MERGEABLE 65 /* KSM may merge identical pages */
-#define MADV_UNMERGEABLE 66 /* KSM may not merge identical pages */
-
-#define MADV_HUGEPAGE 67 /* Worth backing with hugepages */
-#define MADV_NOHUGEPAGE 68 /* Not worth backing with hugepages */
-
-#define MADV_DONTDUMP 69 /* Explicity exclude from the core dump,
- overrides the coredump filter bits */
-#define MADV_DODUMP 70 /* Clear the MADV_NODUMP flag */
-
-#define MADV_WIPEONFORK 71 /* Zero memory on fork, child only */
-#define MADV_KEEPONFORK 72 /* Undo MADV_WIPEONFORK */
-
-#define MADV_COLLAPSE 73 /* Synchronous hugepage collapse */
+#define MADV_COLLAPSE 25 /* Synchronous hugepage collapse */
#define MADV_HWPOISON 100 /* poison a page for testing */
#define MADV_SOFT_OFFLINE 101 /* soft offline page for testing */
/* compatibility flags */
#define MAP_FILE 0
-#define MAP_VARIABLE 0
#define PKEY_DISABLE_ACCESS 0x1
#define PKEY_DISABLE_WRITE 0x2
diff --git a/arch/parisc/kernel/firmware.c b/arch/parisc/kernel/firmware.c
index 6a7e315..4dfe1f4 100644
--- a/arch/parisc/kernel/firmware.c
+++ b/arch/parisc/kernel/firmware.c
@@ -74,8 +74,8 @@
static DEFINE_SPINLOCK(pdc_lock);
#endif
-extern unsigned long pdc_result[NUM_PDC_RESULT];
-extern unsigned long pdc_result2[NUM_PDC_RESULT];
+unsigned long pdc_result[NUM_PDC_RESULT] __aligned(8);
+unsigned long pdc_result2[NUM_PDC_RESULT] __aligned(8);
#ifdef CONFIG_64BIT
#define WIDE_FIRMWARE 0x1
@@ -527,14 +527,14 @@ int pdc_model_info(struct pdc_model *model)
* Using OS_ID_HPUX will return the equivalent of the 'modelname' command
* on HP/UX.
*/
-int pdc_model_sysmodel(char *name)
+int pdc_model_sysmodel(unsigned int os_id, char *name)
{
int retval;
unsigned long flags;
spin_lock_irqsave(&pdc_lock, flags);
retval = mem_pdc_call(PDC_MODEL, PDC_MODEL_SYSMODEL, __pa(pdc_result),
- OS_ID_HPUX, __pa(name));
+ os_id, __pa(name));
convert_to_wide(pdc_result);
if (retval == PDC_OK) {
@@ -1288,9 +1288,8 @@ void pdc_io_reset_devices(void)
#endif /* defined(BOOTLOADER) */
-/* locked by pdc_console_lock */
-static int __attribute__((aligned(8))) iodc_retbuf[32];
-static char __attribute__((aligned(64))) iodc_dbuf[4096];
+/* locked by pdc_lock */
+static char iodc_dbuf[4096] __page_aligned_bss;
/**
* pdc_iodc_print - Console print using IODC.
@@ -1307,6 +1306,9 @@ int pdc_iodc_print(const unsigned char *str, unsigned count)
unsigned int i;
unsigned long flags;
+ count = min_t(unsigned int, count, sizeof(iodc_dbuf));
+
+ spin_lock_irqsave(&pdc_lock, flags);
for (i = 0; i < count;) {
switch(str[i]) {
case '\n':
@@ -1322,12 +1324,11 @@ int pdc_iodc_print(const unsigned char *str, unsigned count)
}
print:
- spin_lock_irqsave(&pdc_lock, flags);
- real32_call(PAGE0->mem_cons.iodc_io,
- (unsigned long)PAGE0->mem_cons.hpa, ENTRY_IO_COUT,
- PAGE0->mem_cons.spa, __pa(PAGE0->mem_cons.dp.layers),
- __pa(iodc_retbuf), 0, __pa(iodc_dbuf), i, 0);
- spin_unlock_irqrestore(&pdc_lock, flags);
+ real32_call(PAGE0->mem_cons.iodc_io,
+ (unsigned long)PAGE0->mem_cons.hpa, ENTRY_IO_COUT,
+ PAGE0->mem_cons.spa, __pa(PAGE0->mem_cons.dp.layers),
+ __pa(pdc_result), 0, __pa(iodc_dbuf), i, 0);
+ spin_unlock_irqrestore(&pdc_lock, flags);
return i;
}
@@ -1354,10 +1355,11 @@ int pdc_iodc_getc(void)
real32_call(PAGE0->mem_kbd.iodc_io,
(unsigned long)PAGE0->mem_kbd.hpa, ENTRY_IO_CIN,
PAGE0->mem_kbd.spa, __pa(PAGE0->mem_kbd.dp.layers),
- __pa(iodc_retbuf), 0, __pa(iodc_dbuf), 1, 0);
+ __pa(pdc_result), 0, __pa(iodc_dbuf), 1, 0);
ch = *iodc_dbuf;
- status = *iodc_retbuf;
+ /* like convert_to_wide() but for first return value only: */
+ status = *(int *)&pdc_result;
spin_unlock_irqrestore(&pdc_lock, flags);
if (status == 0)
diff --git a/arch/parisc/kernel/kgdb.c b/arch/parisc/kernel/kgdb.c
index ab7620f..b16fa9b 100644
--- a/arch/parisc/kernel/kgdb.c
+++ b/arch/parisc/kernel/kgdb.c
@@ -208,23 +208,3 @@ int kgdb_arch_handle_exception(int trap, int signo,
}
return -1;
}
-
-/* KGDB console driver which uses PDC to read chars from keyboard */
-
-static void kgdb_pdc_write_char(u8 chr)
-{
- /* no need to print char. kgdb will do it. */
-}
-
-static struct kgdb_io kgdb_pdc_io_ops = {
- .name = "kgdb_pdc",
- .read_char = pdc_iodc_getc,
- .write_char = kgdb_pdc_write_char,
-};
-
-static int __init kgdb_pdc_init(void)
-{
- kgdb_register_io_module(&kgdb_pdc_io_ops);
- return 0;
-}
-early_initcall(kgdb_pdc_init);
diff --git a/arch/parisc/kernel/pdc_cons.c b/arch/parisc/kernel/pdc_cons.c
index 7d0989f..cf3bf82 100644
--- a/arch/parisc/kernel/pdc_cons.c
+++ b/arch/parisc/kernel/pdc_cons.c
@@ -12,37 +12,27 @@
#include <asm/page.h> /* for PAGE0 */
#include <asm/pdc.h> /* for iodc_call() proto and friends */
-static DEFINE_SPINLOCK(pdc_console_lock);
-
static void pdc_console_write(struct console *co, const char *s, unsigned count)
{
int i = 0;
- unsigned long flags;
- spin_lock_irqsave(&pdc_console_lock, flags);
do {
i += pdc_iodc_print(s + i, count - i);
} while (i < count);
- spin_unlock_irqrestore(&pdc_console_lock, flags);
}
#ifdef CONFIG_KGDB
static int kgdb_pdc_read_char(void)
{
- int c;
- unsigned long flags;
-
- spin_lock_irqsave(&pdc_console_lock, flags);
- c = pdc_iodc_getc();
- spin_unlock_irqrestore(&pdc_console_lock, flags);
+ int c = pdc_iodc_getc();
return (c <= 0) ? NO_POLL_CHAR : c;
}
static void kgdb_pdc_write_char(u8 chr)
{
- if (PAGE0->mem_cons.cl_class != CL_DUPLEX)
- pdc_console_write(NULL, &chr, 1);
+ /* no need to print char as it's shown on standard console */
+ /* pdc_iodc_print(&chr, 1); */
}
static struct kgdb_io kgdb_pdc_io_ops = {
diff --git a/arch/parisc/kernel/processor.c b/arch/parisc/kernel/processor.c
index dddaaa6..ba07e76 100644
--- a/arch/parisc/kernel/processor.c
+++ b/arch/parisc/kernel/processor.c
@@ -272,10 +272,15 @@ void __init collect_boot_cpu_data(void)
printk(KERN_INFO "capabilities 0x%lx\n",
boot_cpu_data.pdc.capabilities);
- if (pdc_model_sysmodel(boot_cpu_data.pdc.sys_model_name) == PDC_OK)
- printk(KERN_INFO "model %s\n",
+ if (pdc_model_sysmodel(OS_ID_HPUX, boot_cpu_data.pdc.sys_model_name) == PDC_OK)
+ pr_info("HP-UX model name: %s\n",
boot_cpu_data.pdc.sys_model_name);
+ serial_no[0] = 0;
+ if (pdc_model_sysmodel(OS_ID_MPEXL, serial_no) == PDC_OK &&
+ serial_no[0])
+ pr_info("MPE/iX model name: %s\n", serial_no);
+
dump_stack_set_arch_desc("%s", boot_cpu_data.pdc.sys_model_name);
boot_cpu_data.hversion = boot_cpu_data.pdc.model.hversion;
diff --git a/arch/parisc/kernel/real2.S b/arch/parisc/kernel/real2.S
index 2b16d8d..4dc12c4 100644
--- a/arch/parisc/kernel/real2.S
+++ b/arch/parisc/kernel/real2.S
@@ -15,28 +15,15 @@
#include <linux/linkage.h>
-
- .section .bss
-
- .export pdc_result
- .export pdc_result2
- .align 8
-pdc_result:
- .block ASM_PDC_RESULT_SIZE
-pdc_result2:
- .block ASM_PDC_RESULT_SIZE
-
.export real_stack
- .export real32_stack
.export real64_stack
- .align 64
+ __PAGE_ALIGNED_BSS
real_stack:
-real32_stack:
real64_stack:
.block 8192
#define N_SAVED_REGS 9
-
+ .section .bss
save_cr_space:
.block REG_SZ * N_SAVED_REGS
save_cr_end:
diff --git a/arch/parisc/kernel/setup.c b/arch/parisc/kernel/setup.c
index 375f38d..0797db6 100644
--- a/arch/parisc/kernel/setup.c
+++ b/arch/parisc/kernel/setup.c
@@ -50,15 +50,15 @@ void __init setup_cmdline(char **cmdline_p)
extern unsigned int boot_args[];
char *p;
- /* Collect stuff passed in from the boot loader */
+ *cmdline_p = command_line;
/* boot_args[0] is free-mem start, boot_args[1] is ptr to command line */
- if (boot_args[0] < 64) {
- /* called from hpux boot loader */
- boot_command_line[0] = '\0';
- } else {
- strscpy(boot_command_line, (char *)__va(boot_args[1]),
- COMMAND_LINE_SIZE);
+ if (boot_args[0] < 64)
+ return; /* return if called from hpux boot loader */
+
+ /* Collect stuff passed in from the boot loader */
+ strscpy(boot_command_line, (char *)__va(boot_args[1]),
+ COMMAND_LINE_SIZE);
/* autodetect console type (if not done by palo yet) */
p = boot_command_line;
@@ -75,16 +75,14 @@ void __init setup_cmdline(char **cmdline_p)
strlcat(p, " earlycon=pdc", COMMAND_LINE_SIZE);
#ifdef CONFIG_BLK_DEV_INITRD
- if (boot_args[2] != 0) /* did palo pass us a ramdisk? */
- {
- initrd_start = (unsigned long)__va(boot_args[2]);
- initrd_end = (unsigned long)__va(boot_args[3]);
- }
-#endif
+ /* did palo pass us a ramdisk? */
+ if (boot_args[2] != 0) {
+ initrd_start = (unsigned long)__va(boot_args[2]);
+ initrd_end = (unsigned long)__va(boot_args[3]);
}
+#endif
strscpy(command_line, boot_command_line, COMMAND_LINE_SIZE);
- *cmdline_p = command_line;
}
#ifdef CONFIG_PA11
diff --git a/arch/parisc/kernel/sys_parisc.c b/arch/parisc/kernel/sys_parisc.c
index 848b0702..09a34b0 100644
--- a/arch/parisc/kernel/sys_parisc.c
+++ b/arch/parisc/kernel/sys_parisc.c
@@ -465,3 +465,31 @@ asmlinkage long parisc_inotify_init1(int flags)
flags = FIX_O_NONBLOCK(flags);
return sys_inotify_init1(flags);
}
+
+/*
+ * madvise() wrapper
+ *
+ * Up to kernel v6.1 parisc has different values than all other
+ * platforms for the MADV_xxx flags listed below.
+ * To keep binary compatibility with existing userspace programs
+ * translate the former values to the new values.
+ *
+ * XXX: Remove this wrapper in year 2025 (or later)
+ */
+
+asmlinkage notrace long parisc_madvise(unsigned long start, size_t len_in, int behavior)
+{
+ switch (behavior) {
+ case 65: behavior = MADV_MERGEABLE; break;
+ case 66: behavior = MADV_UNMERGEABLE; break;
+ case 67: behavior = MADV_HUGEPAGE; break;
+ case 68: behavior = MADV_NOHUGEPAGE; break;
+ case 69: behavior = MADV_DONTDUMP; break;
+ case 70: behavior = MADV_DODUMP; break;
+ case 71: behavior = MADV_WIPEONFORK; break;
+ case 72: behavior = MADV_KEEPONFORK; break;
+ case 73: behavior = MADV_COLLAPSE; break;
+ }
+
+ return sys_madvise(start, len_in, behavior);
+}
diff --git a/arch/parisc/kernel/syscalls/syscall.tbl b/arch/parisc/kernel/syscalls/syscall.tbl
index 8a99c99..0e42fce 100644
--- a/arch/parisc/kernel/syscalls/syscall.tbl
+++ b/arch/parisc/kernel/syscalls/syscall.tbl
@@ -131,7 +131,7 @@
116 common sysinfo sys_sysinfo compat_sys_sysinfo
117 common shutdown sys_shutdown
118 common fsync sys_fsync
-119 common madvise sys_madvise
+119 common madvise parisc_madvise
120 common clone sys_clone_wrapper
121 common setdomainname sys_setdomainname
122 common sendfile sys_sendfile compat_sys_sendfile
diff --git a/arch/parisc/kernel/vdso32/Makefile b/arch/parisc/kernel/vdso32/Makefile
index 85b1c6d..4459a48 100644
--- a/arch/parisc/kernel/vdso32/Makefile
+++ b/arch/parisc/kernel/vdso32/Makefile
@@ -26,7 +26,7 @@
# Force dependency (incbin is bad)
# link rule for the .so file, .lds has to be first
-$(obj)/vdso32.so: $(src)/vdso32.lds $(obj-vdso32) $(obj-cvdso32) $(VDSO_LIBGCC)
+$(obj)/vdso32.so: $(src)/vdso32.lds $(obj-vdso32) $(obj-cvdso32) $(VDSO_LIBGCC) FORCE
$(call if_changed,vdso32ld)
# assembly rules for the .S files
@@ -38,7 +38,7 @@
# actual build commands
quiet_cmd_vdso32ld = VDSO32L $@
- cmd_vdso32ld = $(CROSS32CC) $(c_flags) -Wl,-T $^ -o $@
+ cmd_vdso32ld = $(CROSS32CC) $(c_flags) -Wl,-T $(filter-out FORCE, $^) -o $@
quiet_cmd_vdso32as = VDSO32A $@
cmd_vdso32as = $(CROSS32CC) $(a_flags) -c -o $@ $<
quiet_cmd_vdso32cc = VDSO32C $@
diff --git a/arch/parisc/kernel/vdso64/Makefile b/arch/parisc/kernel/vdso64/Makefile
index a30f5ec..f3d6045 100644
--- a/arch/parisc/kernel/vdso64/Makefile
+++ b/arch/parisc/kernel/vdso64/Makefile
@@ -26,7 +26,7 @@
# Force dependency (incbin is bad)
# link rule for the .so file, .lds has to be first
-$(obj)/vdso64.so: $(src)/vdso64.lds $(obj-vdso64) $(VDSO_LIBGCC)
+$(obj)/vdso64.so: $(src)/vdso64.lds $(obj-vdso64) $(VDSO_LIBGCC) FORCE
$(call if_changed,vdso64ld)
# assembly rules for the .S files
@@ -35,7 +35,7 @@
# actual build commands
quiet_cmd_vdso64ld = VDSO64L $@
- cmd_vdso64ld = $(CC) $(c_flags) -Wl,-T $^ -o $@
+ cmd_vdso64ld = $(CC) $(c_flags) -Wl,-T $(filter-out FORCE, $^) -o $@
quiet_cmd_vdso64as = VDSO64A $@
cmd_vdso64as = $(CC) $(a_flags) -c -o $@ $<
diff --git a/arch/powerpc/Kconfig b/arch/powerpc/Kconfig
index 2ca5418..b8c4ac5 100644
--- a/arch/powerpc/Kconfig
+++ b/arch/powerpc/Kconfig
@@ -1,6 +1,9 @@
# SPDX-License-Identifier: GPL-2.0
source "arch/powerpc/platforms/Kconfig.cputype"
+config CC_HAS_ELFV2
+ def_bool PPC64 && $(cc-option, -mabi=elfv2)
+
config 32BIT
bool
default y if PPC32
@@ -96,7 +99,7 @@
config GENERIC_LOCKBREAK
bool
default y
- depends on SMP && PREEMPTION
+ depends on SMP && PREEMPTION && !PPC_QUEUED_SPINLOCKS
config GENERIC_HWEIGHT
bool
@@ -155,7 +158,6 @@
select ARCH_USE_CMPXCHG_LOCKREF if PPC64
select ARCH_USE_MEMTEST
select ARCH_USE_QUEUED_RWLOCKS if PPC_QUEUED_SPINLOCKS
- select ARCH_USE_QUEUED_SPINLOCKS if PPC_QUEUED_SPINLOCKS
select ARCH_WANT_DEFAULT_TOPDOWN_MMAP_LAYOUT
select ARCH_WANT_IPC_PARSE_VERSION
select ARCH_WANT_IRQS_OFF_ACTIVATE_MM
@@ -239,6 +241,8 @@
select HAVE_MOD_ARCH_SPECIFIC
select HAVE_NMI if PERF_EVENTS || (PPC64 && PPC_BOOK3S)
select HAVE_OPTPROBES
+ select HAVE_OBJTOOL if PPC32 || MPROFILE_KERNEL
+ select HAVE_OBJTOOL_MCOUNT if HAVE_OBJTOOL
select HAVE_PERF_EVENTS
select HAVE_PERF_EVENTS_NMI if PPC64
select HAVE_PERF_REGS
@@ -294,6 +298,9 @@
default y
depends on PPC_BOOK3S_64 || PPC_E500
+config PPC_HAS_LBARX_LHARX
+ bool
+
config EARLY_PRINTK
bool
default y
@@ -529,6 +536,15 @@
Say N if you are unsure.
+config INTERRUPT_SANITIZE_REGISTERS
+ bool "Clear gprs on interrupt arrival"
+ depends on PPC64 && ARCH_HAS_SYSCALL_WRAPPER
+ default PPC_BOOK3E_64 || PPC_PSERIES || PPC_POWERNV
+ help
+ Reduce the influence of user register state on interrupt handlers and
+ syscalls through clearing user state from registers before handling
+ the exception.
+
config PPC_QUEUED_SPINLOCKS
bool "Queued spinlocks" if EXPERT
depends on SMP
@@ -583,6 +599,24 @@
config ARCH_HAS_KEXEC_PURGATORY
def_bool KEXEC_FILE
+config PPC64_BIG_ENDIAN_ELF_ABI_V2
+ bool "Build big-endian kernel using ELF ABI V2 (EXPERIMENTAL)"
+ depends on PPC64 && CPU_BIG_ENDIAN
+ depends on CC_HAS_ELFV2
+ depends on LD_IS_BFD && LD_VERSION >= 22400
+ default n
+ help
+ This builds the kernel image using the "Power Architecture 64-Bit ELF
+ V2 ABI Specification", which has a reduced stack overhead and faster
+ function calls. This internal kernel ABI option does not affect
+ userspace compatibility.
+
+ The V2 ABI is standard for 64-bit little-endian, but for big-endian
+ it is less well tested by kernel and toolchain. However some distros
+ build userspace this way, and it can produce a functioning kernel.
+
+ This requires GCC and binutils 2.24 or newer.
+
config RELOCATABLE
bool "Build a relocatable kernel"
depends on PPC64 || (FLATMEM && (44x || PPC_85xx))
@@ -1012,19 +1046,6 @@
read/write operations on these variables. Say Y if you have
secure boot enabled and want to expose variables to userspace.
-config PPC_RTAS_FILTER
- bool "Enable filtering of RTAS syscalls"
- default y
- depends on PPC_RTAS
- help
- The RTAS syscall API has security issues that could be used to
- compromise system integrity. This option enforces restrictions on the
- RTAS calls and arguments passed by userspace programs to mitigate
- these issues.
-
- Say Y unless you know what you are doing and the filter is causing
- problems for you.
-
endmenu
config ISA_DMA_API
diff --git a/arch/powerpc/boot/dts/fsl/t1024qds.dts b/arch/powerpc/boot/dts/fsl/t1024qds.dts
index d6858b7..9ea7942 100644
--- a/arch/powerpc/boot/dts/fsl/t1024qds.dts
+++ b/arch/powerpc/boot/dts/fsl/t1024qds.dts
@@ -151,7 +151,7 @@ slic@3 {
};
i2c@118000 {
- pca9547@77 {
+ i2c-mux@77 {
compatible = "nxp,pca9547";
reg = <0x77>;
#address-cells = <1>;
diff --git a/arch/powerpc/boot/dts/fsl/t1024rdb.dts b/arch/powerpc/boot/dts/fsl/t1024rdb.dts
index dbcd31c..270aaf6 100644
--- a/arch/powerpc/boot/dts/fsl/t1024rdb.dts
+++ b/arch/powerpc/boot/dts/fsl/t1024rdb.dts
@@ -165,7 +165,7 @@ rtc@68 {
};
i2c@118100 {
- pca9546@77 {
+ i2c-mux@77 {
compatible = "nxp,pca9546";
reg = <0x77>;
#address-cells = <1>;
diff --git a/arch/powerpc/boot/dts/fsl/t104xqds.dtsi b/arch/powerpc/boot/dts/fsl/t104xqds.dtsi
index 6154797..1c329f0 100644
--- a/arch/powerpc/boot/dts/fsl/t104xqds.dtsi
+++ b/arch/powerpc/boot/dts/fsl/t104xqds.dtsi
@@ -268,7 +268,7 @@ flash@0 {
};
i2c@118000 {
- pca9547@77 {
+ i2c-mux@77 {
compatible = "nxp,pca9547";
reg = <0x77>;
};
diff --git a/arch/powerpc/boot/dts/fsl/t104xrdb.dtsi b/arch/powerpc/boot/dts/fsl/t104xrdb.dtsi
index bfe1ed5..fc7bec5 100644
--- a/arch/powerpc/boot/dts/fsl/t104xrdb.dtsi
+++ b/arch/powerpc/boot/dts/fsl/t104xrdb.dtsi
@@ -128,7 +128,7 @@ adt7461@4c {
};
i2c@118100 {
- pca9546@77 {
+ i2c-mux@77 {
compatible = "nxp,pca9546";
reg = <0x77>;
#address-cells = <1>;
diff --git a/arch/powerpc/boot/dts/fsl/t208xqds.dtsi b/arch/powerpc/boot/dts/fsl/t208xqds.dtsi
index db41399..962c999 100644
--- a/arch/powerpc/boot/dts/fsl/t208xqds.dtsi
+++ b/arch/powerpc/boot/dts/fsl/t208xqds.dtsi
@@ -135,7 +135,7 @@ flash@2 {
};
i2c@118000 {
- pca9547@77 {
+ i2c-mux@77 {
compatible = "nxp,pca9547";
reg = <0x77>;
#address-cells = <1>;
diff --git a/arch/powerpc/boot/dts/fsl/t208xrdb.dtsi b/arch/powerpc/boot/dts/fsl/t208xrdb.dtsi
index ff87e67..ecc3e8c 100644
--- a/arch/powerpc/boot/dts/fsl/t208xrdb.dtsi
+++ b/arch/powerpc/boot/dts/fsl/t208xrdb.dtsi
@@ -138,7 +138,7 @@ eeprom@50 {
};
i2c@118100 {
- pca9546@77 {
+ i2c-mux@77 {
compatible = "nxp,pca9546";
reg = <0x77>;
};
diff --git a/arch/powerpc/boot/dts/microwatt.dts b/arch/powerpc/boot/dts/microwatt.dts
index b69db1d..269e930 100644
--- a/arch/powerpc/boot/dts/microwatt.dts
+++ b/arch/powerpc/boot/dts/microwatt.dts
@@ -21,6 +21,14 @@ memory@0 {
reg = <0x00000000 0x00000000 0x00000000 0x10000000>;
};
+ clocks {
+ sys_clk: litex_sys_clk {
+ #clock-cells = <0>;
+ compatible = "fixed-clock";
+ clock-frequency = <100000000>;
+ };
+ };
+
cpus {
#size-cells = <0x00>;
#address-cells = <0x01>;
@@ -141,6 +149,20 @@ ethernet@8020000 {
litex,slot-size = <0x800>;
interrupts = <0x11 0x1>;
};
+
+ mmc@8040000 {
+ compatible = "litex,mmc";
+ reg = <0x8042800 0x800
+ 0x8041000 0x800
+ 0x8040800 0x800
+ 0x8042000 0x800
+ 0x8041800 0x800>;
+ reg-names = "phy", "core", "reader", "writer", "irq";
+ bus-width = <4>;
+ interrupts = <0x13 1>;
+ cap-sd-highspeed;
+ clocks = <&sys_clk>;
+ };
};
chosen {
diff --git a/arch/powerpc/boot/dts/turris1x.dts b/arch/powerpc/boot/dts/turris1x.dts
index 045af66..e9cda34 100644
--- a/arch/powerpc/boot/dts/turris1x.dts
+++ b/arch/powerpc/boot/dts/turris1x.dts
@@ -69,6 +69,20 @@ temperature-sensor@4c {
interrupt-parent = <&gpio>;
interrupts = <12 IRQ_TYPE_LEVEL_LOW>, /* GPIO12 - ALERT pin */
<13 IRQ_TYPE_LEVEL_LOW>; /* GPIO13 - CRIT pin */
+ #address-cells = <1>;
+ #size-cells = <0>;
+
+ /* Local temperature sensor (SA56004ED internal) */
+ channel@0 {
+ reg = <0>;
+ label = "board";
+ };
+
+ /* Remote temperature sensor (D+/D- connected to P2020 CPU Temperature Diode) */
+ channel@1 {
+ reg = <1>;
+ label = "cpu";
+ };
};
/* DDR3 SPD/EEPROM */
diff --git a/arch/powerpc/boot/dts/warp.dts b/arch/powerpc/boot/dts/warp.dts
index b4f3274..aa62d08 100644
--- a/arch/powerpc/boot/dts/warp.dts
+++ b/arch/powerpc/boot/dts/warp.dts
@@ -258,14 +258,12 @@ GPIO1: gpio@ef600c00 {
};
power-leds {
- compatible = "gpio-leds";
+ compatible = "warp-power-leds";
green {
gpios = <&GPIO1 0 0>;
- default-state = "keep";
};
red {
gpios = <&GPIO1 1 0>;
- default-state = "keep";
};
};
diff --git a/arch/powerpc/boot/wrapper b/arch/powerpc/boot/wrapper
index 5bdd4dd..af04cea 100755
--- a/arch/powerpc/boot/wrapper
+++ b/arch/powerpc/boot/wrapper
@@ -215,6 +215,11 @@
}'
}
+ld_is_lld()
+{
+ ${CROSS}ld -V 2>&1 | grep -q LLD
+}
+
# Do not include PT_INTERP segment when linking pie. Non-pie linking
# just ignores this option.
LD_VERSION=$(${CROSS}ld --version | ld_version)
@@ -223,6 +228,14 @@
nodl="--no-dynamic-linker"
fi
+# suppress some warnings in recent ld versions
+nowarn="-z noexecstack"
+if ! ld_is_lld; then
+ if [ "$LD_VERSION" -ge "$(echo 2.39 | ld_version)" ]; then
+ nowarn="$nowarn --no-warn-rwx-segments"
+ fi
+fi
+
platformo=$object/"$platform".o
lds=$object/zImage.lds
ext=strip
@@ -504,7 +517,7 @@
text_start="-Ttext $link_address"
fi
#link everything
- ${CROSS}ld -m $format -T $lds $text_start $pie $nodl $rodynamic $notext -o "$ofile" $map \
+ ${CROSS}ld -m $format -T $lds $text_start $pie $nodl $nowarn $rodynamic $notext -o "$ofile" $map \
$platformo $tmp $object/wrapper.a
rm $tmp
fi
@@ -581,7 +594,7 @@
# reached, then enter the system reset vector of the partially decompressed
# image. No warning is issued.
rm -f "$odir"/{otheros,otheros-too-big}.bld
- size=$(${CROSS}nm --no-sort --radix=d "$ofile" | egrep ' _end$' | cut -d' ' -f1)
+ size=$(${CROSS}nm --no-sort --radix=d "$ofile" | grep -E ' _end$' | cut -d' ' -f1)
bld="otheros.bld"
if [ $size -gt $((0x1000000)) ]; then
bld="otheros-too-big.bld"
diff --git a/arch/powerpc/include/asm/asm.h b/arch/powerpc/include/asm/asm.h
new file mode 100644
index 0000000..86f46b6
--- /dev/null
+++ b/arch/powerpc/include/asm/asm.h
@@ -0,0 +1,7 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _ASM_POWERPC_ASM_H
+#define _ASM_POWERPC_ASM_H
+
+#define _ASM_PTR " .long "
+
+#endif /* _ASM_POWERPC_ASM_H */
diff --git a/arch/powerpc/include/asm/book3s/32/tlbflush.h b/arch/powerpc/include/asm/book3s/32/tlbflush.h
index ba1743c..4be5729 100644
--- a/arch/powerpc/include/asm/book3s/32/tlbflush.h
+++ b/arch/powerpc/include/asm/book3s/32/tlbflush.h
@@ -2,6 +2,8 @@
#ifndef _ASM_POWERPC_BOOK3S_32_TLBFLUSH_H
#define _ASM_POWERPC_BOOK3S_32_TLBFLUSH_H
+#include <linux/build_bug.h>
+
#define MMU_NO_CONTEXT (0)
/*
* TLB flushing for "classic" hash-MMU 32-bit CPUs, 6xx, 7xx, 7xxx
@@ -74,6 +76,13 @@ static inline void local_flush_tlb_page(struct vm_area_struct *vma,
{
flush_tlb_page(vma, vmaddr);
}
+
+static inline void local_flush_tlb_page_psize(struct mm_struct *mm,
+ unsigned long vmaddr, int psize)
+{
+ BUILD_BUG();
+}
+
static inline void local_flush_tlb_mm(struct mm_struct *mm)
{
flush_tlb_mm(mm);
diff --git a/arch/powerpc/include/asm/book3s/64/tlbflush-hash.h b/arch/powerpc/include/asm/book3s/64/tlbflush-hash.h
index 751921f..146287d 100644
--- a/arch/powerpc/include/asm/book3s/64/tlbflush-hash.h
+++ b/arch/powerpc/include/asm/book3s/64/tlbflush-hash.h
@@ -65,56 +65,6 @@ extern void flush_hash_range(unsigned long number, int local);
extern void flush_hash_hugepage(unsigned long vsid, unsigned long addr,
pmd_t *pmdp, unsigned int psize, int ssize,
unsigned long flags);
-static inline void hash__local_flush_tlb_mm(struct mm_struct *mm)
-{
-}
-
-static inline void hash__flush_tlb_mm(struct mm_struct *mm)
-{
-}
-
-static inline void hash__local_flush_all_mm(struct mm_struct *mm)
-{
- /*
- * There's no Page Walk Cache for hash, so what is needed is
- * the same as flush_tlb_mm(), which doesn't really make sense
- * with hash. So the only thing we could do is flush the
- * entire LPID! Punt for now, as it's not being used.
- */
- WARN_ON_ONCE(1);
-}
-
-static inline void hash__flush_all_mm(struct mm_struct *mm)
-{
- /*
- * There's no Page Walk Cache for hash, so what is needed is
- * the same as flush_tlb_mm(), which doesn't really make sense
- * with hash. So the only thing we could do is flush the
- * entire LPID! Punt for now, as it's not being used.
- */
- WARN_ON_ONCE(1);
-}
-
-static inline void hash__local_flush_tlb_page(struct vm_area_struct *vma,
- unsigned long vmaddr)
-{
-}
-
-static inline void hash__flush_tlb_page(struct vm_area_struct *vma,
- unsigned long vmaddr)
-{
-}
-
-static inline void hash__flush_tlb_range(struct vm_area_struct *vma,
- unsigned long start, unsigned long end)
-{
-}
-
-static inline void hash__flush_tlb_kernel_range(unsigned long start,
- unsigned long end)
-{
-}
-
struct mmu_gather;
extern void hash__tlb_flush(struct mmu_gather *tlb);
diff --git a/arch/powerpc/include/asm/book3s/64/tlbflush.h b/arch/powerpc/include/asm/book3s/64/tlbflush.h
index 67655cd6..dd39313 100644
--- a/arch/powerpc/include/asm/book3s/64/tlbflush.h
+++ b/arch/powerpc/include/asm/book3s/64/tlbflush.h
@@ -47,8 +47,7 @@ static inline void flush_pmd_tlb_range(struct vm_area_struct *vma,
unsigned long start, unsigned long end)
{
if (radix_enabled())
- return radix__flush_pmd_tlb_range(vma, start, end);
- return hash__flush_tlb_range(vma, start, end);
+ radix__flush_pmd_tlb_range(vma, start, end);
}
#define __HAVE_ARCH_FLUSH_HUGETLB_TLB_RANGE
@@ -57,81 +56,65 @@ static inline void flush_hugetlb_tlb_range(struct vm_area_struct *vma,
unsigned long end)
{
if (radix_enabled())
- return radix__flush_hugetlb_tlb_range(vma, start, end);
- return hash__flush_tlb_range(vma, start, end);
+ radix__flush_hugetlb_tlb_range(vma, start, end);
}
static inline void flush_tlb_range(struct vm_area_struct *vma,
unsigned long start, unsigned long end)
{
if (radix_enabled())
- return radix__flush_tlb_range(vma, start, end);
- return hash__flush_tlb_range(vma, start, end);
+ radix__flush_tlb_range(vma, start, end);
}
static inline void flush_tlb_kernel_range(unsigned long start,
unsigned long end)
{
if (radix_enabled())
- return radix__flush_tlb_kernel_range(start, end);
- return hash__flush_tlb_kernel_range(start, end);
+ radix__flush_tlb_kernel_range(start, end);
}
static inline void local_flush_tlb_mm(struct mm_struct *mm)
{
if (radix_enabled())
- return radix__local_flush_tlb_mm(mm);
- return hash__local_flush_tlb_mm(mm);
+ radix__local_flush_tlb_mm(mm);
}
static inline void local_flush_tlb_page(struct vm_area_struct *vma,
unsigned long vmaddr)
{
if (radix_enabled())
- return radix__local_flush_tlb_page(vma, vmaddr);
- return hash__local_flush_tlb_page(vma, vmaddr);
+ radix__local_flush_tlb_page(vma, vmaddr);
}
-static inline void local_flush_all_mm(struct mm_struct *mm)
+static inline void local_flush_tlb_page_psize(struct mm_struct *mm,
+ unsigned long vmaddr, int psize)
{
if (radix_enabled())
- return radix__local_flush_all_mm(mm);
- return hash__local_flush_all_mm(mm);
+ radix__local_flush_tlb_page_psize(mm, vmaddr, psize);
}
static inline void tlb_flush(struct mmu_gather *tlb)
{
if (radix_enabled())
- return radix__tlb_flush(tlb);
- return hash__tlb_flush(tlb);
+ radix__tlb_flush(tlb);
}
#ifdef CONFIG_SMP
static inline void flush_tlb_mm(struct mm_struct *mm)
{
if (radix_enabled())
- return radix__flush_tlb_mm(mm);
- return hash__flush_tlb_mm(mm);
+ radix__flush_tlb_mm(mm);
}
static inline void flush_tlb_page(struct vm_area_struct *vma,
unsigned long vmaddr)
{
if (radix_enabled())
- return radix__flush_tlb_page(vma, vmaddr);
- return hash__flush_tlb_page(vma, vmaddr);
-}
-
-static inline void flush_all_mm(struct mm_struct *mm)
-{
- if (radix_enabled())
- return radix__flush_all_mm(mm);
- return hash__flush_all_mm(mm);
+ radix__flush_tlb_page(vma, vmaddr);
}
#else
#define flush_tlb_mm(mm) local_flush_tlb_mm(mm)
#define flush_tlb_page(vma, addr) local_flush_tlb_page(vma, addr)
-#define flush_all_mm(mm) local_flush_all_mm(mm)
#endif /* CONFIG_SMP */
#define flush_tlb_fix_spurious_fault flush_tlb_fix_spurious_fault
diff --git a/arch/powerpc/include/asm/bug.h b/arch/powerpc/include/asm/bug.h
index 61a4736..ef42adb 100644
--- a/arch/powerpc/include/asm/bug.h
+++ b/arch/powerpc/include/asm/bug.h
@@ -99,7 +99,8 @@
__label__ __label_warn_on; \
\
WARN_ENTRY("twi 31, 0, 0", BUGFLAG_WARNING | (flags), __label_warn_on); \
- unreachable(); \
+ barrier_before_unreachable(); \
+ __builtin_unreachable(); \
\
__label_warn_on: \
break; \
diff --git a/arch/powerpc/include/asm/cmpxchg.h b/arch/powerpc/include/asm/cmpxchg.h
index 05f246c..d0ea057 100644
--- a/arch/powerpc/include/asm/cmpxchg.h
+++ b/arch/powerpc/include/asm/cmpxchg.h
@@ -77,10 +77,76 @@ u32 __cmpxchg_##type##sfx(volatile void *p, u32 old, u32 new) \
* the previous value stored there.
*/
+#ifndef CONFIG_PPC_HAS_LBARX_LHARX
XCHG_GEN(u8, _local, "memory");
XCHG_GEN(u8, _relaxed, "cc");
XCHG_GEN(u16, _local, "memory");
XCHG_GEN(u16, _relaxed, "cc");
+#else
+static __always_inline unsigned long
+__xchg_u8_local(volatile void *p, unsigned long val)
+{
+ unsigned long prev;
+
+ __asm__ __volatile__(
+"1: lbarx %0,0,%2 # __xchg_u8_local\n"
+" stbcx. %3,0,%2 \n"
+" bne- 1b"
+ : "=&r" (prev), "+m" (*(volatile unsigned char *)p)
+ : "r" (p), "r" (val)
+ : "cc", "memory");
+
+ return prev;
+}
+
+static __always_inline unsigned long
+__xchg_u8_relaxed(u8 *p, unsigned long val)
+{
+ unsigned long prev;
+
+ __asm__ __volatile__(
+"1: lbarx %0,0,%2 # __xchg_u8_relaxed\n"
+" stbcx. %3,0,%2\n"
+" bne- 1b"
+ : "=&r" (prev), "+m" (*p)
+ : "r" (p), "r" (val)
+ : "cc");
+
+ return prev;
+}
+
+static __always_inline unsigned long
+__xchg_u16_local(volatile void *p, unsigned long val)
+{
+ unsigned long prev;
+
+ __asm__ __volatile__(
+"1: lharx %0,0,%2 # __xchg_u16_local\n"
+" sthcx. %3,0,%2\n"
+" bne- 1b"
+ : "=&r" (prev), "+m" (*(volatile unsigned short *)p)
+ : "r" (p), "r" (val)
+ : "cc", "memory");
+
+ return prev;
+}
+
+static __always_inline unsigned long
+__xchg_u16_relaxed(u16 *p, unsigned long val)
+{
+ unsigned long prev;
+
+ __asm__ __volatile__(
+"1: lharx %0,0,%2 # __xchg_u16_relaxed\n"
+" sthcx. %3,0,%2\n"
+" bne- 1b"
+ : "=&r" (prev), "+m" (*p)
+ : "r" (p), "r" (val)
+ : "cc");
+
+ return prev;
+}
+#endif
static __always_inline unsigned long
__xchg_u32_local(volatile void *p, unsigned long val)
@@ -198,11 +264,12 @@ __xchg_relaxed(void *ptr, unsigned long x, unsigned int size)
(__typeof__(*(ptr))) __xchg_relaxed((ptr), \
(unsigned long)_x_, sizeof(*(ptr))); \
})
+
/*
* Compare and exchange - if *p == old, set it to new,
* and return the old value of *p.
*/
-
+#ifndef CONFIG_PPC_HAS_LBARX_LHARX
CMPXCHG_GEN(u8, , PPC_ATOMIC_ENTRY_BARRIER, PPC_ATOMIC_EXIT_BARRIER, "memory");
CMPXCHG_GEN(u8, _local, , , "memory");
CMPXCHG_GEN(u8, _acquire, , PPC_ACQUIRE_BARRIER, "memory");
@@ -211,6 +278,168 @@ CMPXCHG_GEN(u16, , PPC_ATOMIC_ENTRY_BARRIER, PPC_ATOMIC_EXIT_BARRIER, "memory");
CMPXCHG_GEN(u16, _local, , , "memory");
CMPXCHG_GEN(u16, _acquire, , PPC_ACQUIRE_BARRIER, "memory");
CMPXCHG_GEN(u16, _relaxed, , , "cc");
+#else
+static __always_inline unsigned long
+__cmpxchg_u8(volatile unsigned char *p, unsigned long old, unsigned long new)
+{
+ unsigned int prev;
+
+ __asm__ __volatile__ (
+ PPC_ATOMIC_ENTRY_BARRIER
+"1: lbarx %0,0,%2 # __cmpxchg_u8\n"
+" cmpw 0,%0,%3\n"
+" bne- 2f\n"
+" stbcx. %4,0,%2\n"
+" bne- 1b"
+ PPC_ATOMIC_EXIT_BARRIER
+ "\n\
+2:"
+ : "=&r" (prev), "+m" (*p)
+ : "r" (p), "r" (old), "r" (new)
+ : "cc", "memory");
+
+ return prev;
+}
+
+static __always_inline unsigned long
+__cmpxchg_u8_local(volatile unsigned char *p, unsigned long old,
+ unsigned long new)
+{
+ unsigned int prev;
+
+ __asm__ __volatile__ (
+"1: lbarx %0,0,%2 # __cmpxchg_u8_local\n"
+" cmpw 0,%0,%3\n"
+" bne- 2f\n"
+" stbcx. %4,0,%2\n"
+" bne- 1b\n"
+"2:"
+ : "=&r" (prev), "+m" (*p)
+ : "r" (p), "r" (old), "r" (new)
+ : "cc", "memory");
+
+ return prev;
+}
+
+static __always_inline unsigned long
+__cmpxchg_u8_relaxed(u8 *p, unsigned long old, unsigned long new)
+{
+ unsigned long prev;
+
+ __asm__ __volatile__ (
+"1: lbarx %0,0,%2 # __cmpxchg_u8_relaxed\n"
+" cmpw 0,%0,%3\n"
+" bne- 2f\n"
+" stbcx. %4,0,%2\n"
+" bne- 1b\n"
+"2:"
+ : "=&r" (prev), "+m" (*p)
+ : "r" (p), "r" (old), "r" (new)
+ : "cc");
+
+ return prev;
+}
+
+static __always_inline unsigned long
+__cmpxchg_u8_acquire(u8 *p, unsigned long old, unsigned long new)
+{
+ unsigned long prev;
+
+ __asm__ __volatile__ (
+"1: lbarx %0,0,%2 # __cmpxchg_u8_acquire\n"
+" cmpw 0,%0,%3\n"
+" bne- 2f\n"
+" stbcx. %4,0,%2\n"
+" bne- 1b\n"
+ PPC_ACQUIRE_BARRIER
+"2:"
+ : "=&r" (prev), "+m" (*p)
+ : "r" (p), "r" (old), "r" (new)
+ : "cc", "memory");
+
+ return prev;
+}
+
+static __always_inline unsigned long
+__cmpxchg_u16(volatile unsigned short *p, unsigned long old, unsigned long new)
+{
+ unsigned int prev;
+
+ __asm__ __volatile__ (
+ PPC_ATOMIC_ENTRY_BARRIER
+"1: lharx %0,0,%2 # __cmpxchg_u16\n"
+" cmpw 0,%0,%3\n"
+" bne- 2f\n"
+" sthcx. %4,0,%2\n"
+" bne- 1b\n"
+ PPC_ATOMIC_EXIT_BARRIER
+"2:"
+ : "=&r" (prev), "+m" (*p)
+ : "r" (p), "r" (old), "r" (new)
+ : "cc", "memory");
+
+ return prev;
+}
+
+static __always_inline unsigned long
+__cmpxchg_u16_local(volatile unsigned short *p, unsigned long old,
+ unsigned long new)
+{
+ unsigned int prev;
+
+ __asm__ __volatile__ (
+"1: lharx %0,0,%2 # __cmpxchg_u16_local\n"
+" cmpw 0,%0,%3\n"
+" bne- 2f\n"
+" sthcx. %4,0,%2\n"
+" bne- 1b"
+"2:"
+ : "=&r" (prev), "+m" (*p)
+ : "r" (p), "r" (old), "r" (new)
+ : "cc", "memory");
+
+ return prev;
+}
+
+static __always_inline unsigned long
+__cmpxchg_u16_relaxed(u16 *p, unsigned long old, unsigned long new)
+{
+ unsigned long prev;
+
+ __asm__ __volatile__ (
+"1: lharx %0,0,%2 # __cmpxchg_u16_relaxed\n"
+" cmpw 0,%0,%3\n"
+" bne- 2f\n"
+" sthcx. %4,0,%2\n"
+" bne- 1b\n"
+"2:"
+ : "=&r" (prev), "+m" (*p)
+ : "r" (p), "r" (old), "r" (new)
+ : "cc");
+
+ return prev;
+}
+
+static __always_inline unsigned long
+__cmpxchg_u16_acquire(u16 *p, unsigned long old, unsigned long new)
+{
+ unsigned long prev;
+
+ __asm__ __volatile__ (
+"1: lharx %0,0,%2 # __cmpxchg_u16_acquire\n"
+" cmpw 0,%0,%3\n"
+" bne- 2f\n"
+" sthcx. %4,0,%2\n"
+" bne- 1b\n"
+ PPC_ACQUIRE_BARRIER
+"2:"
+ : "=&r" (prev), "+m" (*p)
+ : "r" (p), "r" (old), "r" (new)
+ : "cc", "memory");
+
+ return prev;
+}
+#endif
static __always_inline unsigned long
__cmpxchg_u32(volatile unsigned int *p, unsigned long old, unsigned long new)
diff --git a/arch/powerpc/include/asm/code-patching.h b/arch/powerpc/include/asm/code-patching.h
index 1c6316e..3f88154 100644
--- a/arch/powerpc/include/asm/code-patching.h
+++ b/arch/powerpc/include/asm/code-patching.h
@@ -22,8 +22,6 @@
#define BRANCH_SET_LINK 0x1
#define BRANCH_ABSOLUTE 0x2
-DECLARE_STATIC_KEY_FALSE(init_mem_is_free);
-
/*
* Powerpc branch instruction is :
*
diff --git a/arch/powerpc/include/asm/cputime.h b/arch/powerpc/include/asm/cputime.h
index 431ae23..4961fb3 100644
--- a/arch/powerpc/include/asm/cputime.h
+++ b/arch/powerpc/include/asm/cputime.h
@@ -21,23 +21,8 @@
#include <asm/param.h>
#include <asm/firmware.h>
-typedef u64 __nocast cputime_t;
-typedef u64 __nocast cputime64_t;
-
-#define cmpxchg_cputime(ptr, old, new) cmpxchg(ptr, old, new)
-
#ifdef __KERNEL__
-/*
- * Convert cputime <-> microseconds
- */
-extern u64 __cputime_usec_factor;
-
-static inline unsigned long cputime_to_usecs(const cputime_t ct)
-{
- return mulhdu((__force u64) ct, __cputime_usec_factor);
-}
-
-#define cputime_to_nsecs(cputime) tb_to_ns((__force u64)cputime)
+#define cputime_to_nsecs(cputime) tb_to_ns(cputime)
/*
* PPC64 uses PACA which is task independent for storing accounting data while
diff --git a/arch/powerpc/include/asm/debug.h b/arch/powerpc/include/asm/debug.h
index 86a1473..51c74460 100644
--- a/arch/powerpc/include/asm/debug.h
+++ b/arch/powerpc/include/asm/debug.h
@@ -46,6 +46,8 @@ static inline int debugger_fault_handler(struct pt_regs *regs) { return 0; }
#endif
void __set_breakpoint(int nr, struct arch_hw_breakpoint *brk);
+void suspend_breakpoints(void);
+void restore_breakpoints(void);
bool ppc_breakpoint_available(void);
#ifdef CONFIG_PPC_ADV_DEBUG_REGS
extern void do_send_trap(struct pt_regs *regs, unsigned long address,
diff --git a/arch/powerpc/include/asm/ftrace.h b/arch/powerpc/include/asm/ftrace.h
index 259b9dd..91c049d 100644
--- a/arch/powerpc/include/asm/ftrace.h
+++ b/arch/powerpc/include/asm/ftrace.h
@@ -10,6 +10,13 @@
#define HAVE_FUNCTION_GRAPH_RET_ADDR_PTR
+/* Ignore unused weak functions which will have larger offsets */
+#ifdef CONFIG_MPROFILE_KERNEL
+#define FTRACE_MCOUNT_MAX_OFFSET 12
+#elif defined(CONFIG_PPC32)
+#define FTRACE_MCOUNT_MAX_OFFSET 8
+#endif
+
#ifndef __ASSEMBLY__
extern void _mcount(void);
@@ -84,17 +91,6 @@ void ftrace_graph_func(unsigned long ip, unsigned long parent_ip,
* those.
*/
#define ARCH_HAS_SYSCALL_MATCH_SYM_NAME
-#ifdef CONFIG_PPC64_ELF_ABI_V1
-static inline bool arch_syscall_match_sym_name(const char *sym, const char *name)
-{
- /* We need to skip past the initial dot, and the __se_sys alias */
- return !strcmp(sym + 1, name) ||
- (!strncmp(sym, ".__se_sys", 9) && !strcmp(sym + 6, name)) ||
- (!strncmp(sym, ".ppc_", 5) && !strcmp(sym + 5, name + 4)) ||
- (!strncmp(sym, ".ppc32_", 7) && !strcmp(sym + 7, name + 4)) ||
- (!strncmp(sym, ".ppc64_", 7) && !strcmp(sym + 7, name + 4));
-}
-#else
static inline bool arch_syscall_match_sym_name(const char *sym, const char *name)
{
return !strcmp(sym, name) ||
@@ -103,7 +99,6 @@ static inline bool arch_syscall_match_sym_name(const char *sym, const char *name
(!strncmp(sym, "ppc32_", 6) && !strcmp(sym + 6, name + 4)) ||
(!strncmp(sym, "ppc64_", 6) && !strcmp(sym + 6, name + 4));
}
-#endif /* CONFIG_PPC64_ELF_ABI_V1 */
#endif /* CONFIG_FTRACE_SYSCALLS */
#if defined(CONFIG_PPC64) && defined(CONFIG_FUNCTION_TRACER)
diff --git a/arch/powerpc/include/asm/hvcall.h b/arch/powerpc/include/asm/hvcall.h
index 8abae46..95fd7f9 100644
--- a/arch/powerpc/include/asm/hvcall.h
+++ b/arch/powerpc/include/asm/hvcall.h
@@ -79,7 +79,7 @@
#define H_NOT_ENOUGH_RESOURCES -44
#define H_R_STATE -45
#define H_RESCINDED -46
-#define H_P1 -54
+#define H_ABORTED -54
#define H_P2 -55
#define H_P3 -56
#define H_P4 -57
@@ -100,7 +100,6 @@
#define H_COP_HW -74
#define H_STATE -75
#define H_IN_USE -77
-#define H_ABORTED -78
#define H_UNSUPPORTED_FLAG_START -256
#define H_UNSUPPORTED_FLAG_END -511
#define H_MULTI_THREADS_ACTIVE -9005
diff --git a/arch/powerpc/include/asm/irqflags.h b/arch/powerpc/include/asm/irqflags.h
index 1a6c1ce..47d4671 100644
--- a/arch/powerpc/include/asm/irqflags.h
+++ b/arch/powerpc/include/asm/irqflags.h
@@ -11,64 +11,6 @@
*/
#include <asm/hw_irq.h>
-#else
-#ifdef CONFIG_TRACE_IRQFLAGS
-#ifdef CONFIG_IRQSOFF_TRACER
-/*
- * Since the ftrace irqsoff latency trace checks CALLER_ADDR1,
- * which is the stack frame here, we need to force a stack frame
- * in case we came from user space.
- */
-#define TRACE_WITH_FRAME_BUFFER(func) \
- mflr r0; \
- stdu r1, -STACK_FRAME_OVERHEAD(r1); \
- std r0, 16(r1); \
- stdu r1, -STACK_FRAME_OVERHEAD(r1); \
- bl func; \
- ld r1, 0(r1); \
- ld r1, 0(r1);
-#else
-#define TRACE_WITH_FRAME_BUFFER(func) \
- bl func;
-#endif
-
-/*
- * These are calls to C code, so the caller must be prepared for volatiles to
- * be clobbered.
- */
-#define TRACE_ENABLE_INTS TRACE_WITH_FRAME_BUFFER(trace_hardirqs_on)
-#define TRACE_DISABLE_INTS TRACE_WITH_FRAME_BUFFER(trace_hardirqs_off)
-
-/*
- * This is used by assembly code to soft-disable interrupts first and
- * reconcile irq state.
- *
- * NB: This may call C code, so the caller must be prepared for volatiles to
- * be clobbered.
- */
-#define RECONCILE_IRQ_STATE(__rA, __rB) \
- lbz __rA,PACAIRQSOFTMASK(r13); \
- lbz __rB,PACAIRQHAPPENED(r13); \
- andi. __rA,__rA,IRQS_DISABLED; \
- li __rA,IRQS_DISABLED; \
- ori __rB,__rB,PACA_IRQ_HARD_DIS; \
- stb __rB,PACAIRQHAPPENED(r13); \
- bne 44f; \
- stb __rA,PACAIRQSOFTMASK(r13); \
- TRACE_DISABLE_INTS; \
-44:
-
-#else
-#define TRACE_ENABLE_INTS
-#define TRACE_DISABLE_INTS
-
-#define RECONCILE_IRQ_STATE(__rA, __rB) \
- lbz __rA,PACAIRQHAPPENED(r13); \
- li __rB,IRQS_DISABLED; \
- ori __rA,__rA,PACA_IRQ_HARD_DIS; \
- stb __rB,PACAIRQSOFTMASK(r13); \
- stb __rA,PACAIRQHAPPENED(r13)
-#endif
#endif
#endif
diff --git a/arch/powerpc/include/asm/kvm_book3s_asm.h b/arch/powerpc/include/asm/kvm_book3s_asm.h
index c8882d9..a367979 100644
--- a/arch/powerpc/include/asm/kvm_book3s_asm.h
+++ b/arch/powerpc/include/asm/kvm_book3s_asm.h
@@ -105,7 +105,7 @@ struct kvmppc_host_state {
void __iomem *xive_tima_virt;
u32 saved_xirr;
u64 dabr;
- u64 host_mmcr[10]; /* MMCR 0,1,A, SIAR, SDAR, MMCR2, SIER, MMCR3, SIER2/3 */
+ u64 host_mmcr[7]; /* MMCR 0,1,A, SIAR, SDAR, MMCR2, SIER */
u32 host_pmc[8];
u64 host_purr;
u64 host_spurr;
diff --git a/arch/powerpc/include/asm/kvm_ppc.h b/arch/powerpc/include/asm/kvm_ppc.h
index bfacf12..eae9619 100644
--- a/arch/powerpc/include/asm/kvm_ppc.h
+++ b/arch/powerpc/include/asm/kvm_ppc.h
@@ -1014,6 +1014,18 @@ static inline void kvmppc_fix_ee_before_entry(void)
#endif
}
+static inline void kvmppc_fix_ee_after_exit(void)
+{
+#ifdef CONFIG_PPC64
+ /* Only need to enable IRQs by hard enabling them after this */
+ local_paca->irq_happened = PACA_IRQ_HARD_DIS;
+ irq_soft_mask_set(IRQS_ALL_DISABLED);
+#endif
+
+ trace_hardirqs_off();
+}
+
+
static inline ulong kvmppc_get_ea_indexed(struct kvm_vcpu *vcpu, int ra, int rb)
{
ulong ea;
diff --git a/arch/powerpc/include/asm/linkage.h b/arch/powerpc/include/asm/linkage.h
index b71b958..b88d1d2 100644
--- a/arch/powerpc/include/asm/linkage.h
+++ b/arch/powerpc/include/asm/linkage.h
@@ -4,6 +4,9 @@
#include <asm/types.h>
+#define __ALIGN .align 2
+#define __ALIGN_STR ".align 2"
+
#ifdef CONFIG_PPC64_ELF_ABI_V1
#define cond_syscall(x) \
asm ("\t.weak " #x "\n\t.set " #x ", sys_ni_syscall\n" \
diff --git a/arch/powerpc/include/asm/mmu_context.h b/arch/powerpc/include/asm/mmu_context.h
index c1ea270..57f5017 100644
--- a/arch/powerpc/include/asm/mmu_context.h
+++ b/arch/powerpc/include/asm/mmu_context.h
@@ -151,8 +151,8 @@ static inline void mm_context_remove_copro(struct mm_struct *mm)
* nMMU and/or PSL need to be cleaned up.
*
* Both the 'copros' and 'active_cpus' counts are looked at in
- * flush_all_mm() to determine the scope (local/global) of the
- * TLBIs, so we need to flush first before decrementing
+ * radix__flush_all_mm() to determine the scope (local/global)
+ * of the TLBIs, so we need to flush first before decrementing
* 'copros'. If this API is used by several callers for the
* same context, it can lead to over-flushing. It's hopefully
* not common enough to be a problem.
@@ -164,7 +164,7 @@ static inline void mm_context_remove_copro(struct mm_struct *mm)
* in-between.
*/
if (radix_enabled()) {
- flush_all_mm(mm);
+ radix__flush_all_mm(mm);
c = atomic_dec_if_positive(&mm->context.copros);
/* Detect imbalance between add and remove */
diff --git a/arch/powerpc/include/asm/nohash/32/pgtable.h b/arch/powerpc/include/asm/nohash/32/pgtable.h
index 0d40b33..70edad4 100644
--- a/arch/powerpc/include/asm/nohash/32/pgtable.h
+++ b/arch/powerpc/include/asm/nohash/32/pgtable.h
@@ -256,14 +256,20 @@ static inline pte_basic_t pte_update(struct mm_struct *mm, unsigned long addr, p
num = number_of_cells_per_pte(pmd, new, huge);
- for (i = 0; i < num; i++, entry++, new += SZ_4K)
- *entry = new;
+ for (i = 0; i < num; i += PAGE_SIZE / SZ_4K, new += PAGE_SIZE) {
+ *entry++ = new;
+ if (IS_ENABLED(CONFIG_PPC_16K_PAGES) && num != 1) {
+ *entry++ = new;
+ *entry++ = new;
+ *entry++ = new;
+ }
+ }
return old;
}
#ifdef CONFIG_PPC_16K_PAGES
-#define __HAVE_ARCH_PTEP_GET
+#define ptep_get ptep_get
static inline pte_t ptep_get(pte_t *ptep)
{
pte_basic_t val = READ_ONCE(ptep->pte);
diff --git a/arch/powerpc/include/asm/nohash/pgtable.h b/arch/powerpc/include/asm/nohash/pgtable.h
index d9067df..69c3a05 100644
--- a/arch/powerpc/include/asm/nohash/pgtable.h
+++ b/arch/powerpc/include/asm/nohash/pgtable.h
@@ -183,7 +183,7 @@ static inline void __set_pte_at(struct mm_struct *mm, unsigned long addr,
* cases, and 32-bit non-hash with 32-bit PTEs.
*/
#if defined(CONFIG_PPC_8xx) && defined(CONFIG_PPC_16K_PAGES)
- ptep->pte = ptep->pte1 = ptep->pte2 = ptep->pte3 = pte_val(pte);
+ ptep->pte3 = ptep->pte2 = ptep->pte1 = ptep->pte = pte_val(pte);
#else
*ptep = pte;
#endif
diff --git a/arch/powerpc/include/asm/nohash/tlbflush.h b/arch/powerpc/include/asm/nohash/tlbflush.h
index bdaf34a..9a2cf83 100644
--- a/arch/powerpc/include/asm/nohash/tlbflush.h
+++ b/arch/powerpc/include/asm/nohash/tlbflush.h
@@ -45,6 +45,12 @@ static inline void local_flush_tlb_page(struct vm_area_struct *vma, unsigned lon
asm volatile ("tlbie %0; sync" : : "r" (vmaddr) : "memory");
}
+static inline void local_flush_tlb_page_psize(struct mm_struct *mm,
+ unsigned long vmaddr, int psize)
+{
+ asm volatile ("tlbie %0; sync" : : "r" (vmaddr) : "memory");
+}
+
static inline void flush_tlb_kernel_range(unsigned long start, unsigned long end)
{
start &= PAGE_MASK;
@@ -58,6 +64,7 @@ static inline void flush_tlb_kernel_range(unsigned long start, unsigned long end
extern void flush_tlb_kernel_range(unsigned long start, unsigned long end);
extern void local_flush_tlb_mm(struct mm_struct *mm);
extern void local_flush_tlb_page(struct vm_area_struct *vma, unsigned long vmaddr);
+void local_flush_tlb_page_psize(struct mm_struct *mm, unsigned long vmaddr, int psize);
extern void __local_flush_tlb_page(struct mm_struct *mm, unsigned long vmaddr,
int tsize, int ind);
diff --git a/arch/powerpc/include/asm/ppc_asm.h b/arch/powerpc/include/asm/ppc_asm.h
index 753a275..d2f4461 100644
--- a/arch/powerpc/include/asm/ppc_asm.h
+++ b/arch/powerpc/include/asm/ppc_asm.h
@@ -74,6 +74,25 @@
#define SAVE_GPR(n, base) SAVE_GPRS(n, n, base)
#define REST_GPR(n, base) REST_GPRS(n, n, base)
+/* macros for handling user register sanitisation */
+#ifdef CONFIG_INTERRUPT_SANITIZE_REGISTERS
+#define SANITIZE_SYSCALL_GPRS() ZEROIZE_GPR(0); \
+ ZEROIZE_GPRS(5, 12); \
+ ZEROIZE_NVGPRS()
+#define SANITIZE_GPR(n) ZEROIZE_GPR(n)
+#define SANITIZE_GPRS(start, end) ZEROIZE_GPRS(start, end)
+#define SANITIZE_NVGPRS() ZEROIZE_NVGPRS()
+#define SANITIZE_RESTORE_NVGPRS() REST_NVGPRS(r1)
+#define HANDLER_RESTORE_NVGPRS()
+#else
+#define SANITIZE_SYSCALL_GPRS()
+#define SANITIZE_GPR(n)
+#define SANITIZE_GPRS(start, end)
+#define SANITIZE_NVGPRS()
+#define SANITIZE_RESTORE_NVGPRS()
+#define HANDLER_RESTORE_NVGPRS() REST_NVGPRS(r1)
+#endif /* CONFIG_INTERRUPT_SANITIZE_REGISTERS */
+
#define SAVE_FPR(n, base) stfd n,8*TS_FPRWIDTH*(n)(base)
#define SAVE_2FPRS(n, base) SAVE_FPR(n, base); SAVE_FPR(n+1, base)
#define SAVE_4FPRS(n, base) SAVE_2FPRS(n, base); SAVE_2FPRS(n+2, base)
diff --git a/arch/powerpc/include/asm/processor.h b/arch/powerpc/include/asm/processor.h
index 6318029..e96c9b8 100644
--- a/arch/powerpc/include/asm/processor.h
+++ b/arch/powerpc/include/asm/processor.h
@@ -374,9 +374,18 @@ static inline unsigned long __pack_fe01(unsigned int fpmode)
#endif
-/* Check that a certain kernel stack pointer is valid in task_struct p */
-int validate_sp(unsigned long sp, struct task_struct *p,
- unsigned long nbytes);
+/*
+ * Check that a certain kernel stack pointer is a valid (minimum sized)
+ * stack frame in task_struct p.
+ */
+int validate_sp(unsigned long sp, struct task_struct *p);
+
+/*
+ * validate the stack frame of a particular minimum size, used for when we are
+ * looking at a certain object in the stack beyond the minimum.
+ */
+int validate_sp_size(unsigned long sp, struct task_struct *p,
+ unsigned long nbytes);
/*
* Prefetch macros.
diff --git a/arch/powerpc/include/asm/prom.h b/arch/powerpc/include/asm/prom.h
index 2e82820fb..c0107d8 100644
--- a/arch/powerpc/include/asm/prom.h
+++ b/arch/powerpc/include/asm/prom.h
@@ -85,6 +85,7 @@ struct of_drc_info {
extern int of_read_drc_info_cell(struct property **prop,
const __be32 **curval, struct of_drc_info *data);
+extern unsigned int boot_cpu_node_count;
/*
* There are two methods for telling firmware what our capabilities are.
diff --git a/arch/powerpc/include/asm/ps3.h b/arch/powerpc/include/asm/ps3.h
index 8a0d8fb..d503dbd 100644
--- a/arch/powerpc/include/asm/ps3.h
+++ b/arch/powerpc/include/asm/ps3.h
@@ -425,10 +425,6 @@ static inline void *ps3_system_bus_get_drvdata(
return dev_get_drvdata(&dev->core);
}
-/* These two need global scope for get_arch_dma_ops(). */
-
-extern struct bus_type ps3_system_bus_type;
-
/* system manager */
struct ps3_sys_manager_ops {
diff --git a/arch/powerpc/include/asm/pte-walk.h b/arch/powerpc/include/asm/pte-walk.h
index 714a35f..73c22c5 100644
--- a/arch/powerpc/include/asm/pte-walk.h
+++ b/arch/powerpc/include/asm/pte-walk.h
@@ -60,29 +60,4 @@ static inline phys_addr_t ppc_find_vmap_phys(unsigned long addr)
return pa;
}
-/*
- * This is what we should always use. Any other lockless page table lookup needs
- * careful audit against THP split.
- */
-static inline pte_t *find_current_mm_pte(pgd_t *pgdir, unsigned long ea,
- bool *is_thp, unsigned *hshift)
-{
- pte_t *pte;
-
- VM_WARN(!arch_irqs_disabled(), "%s called with irq enabled\n", __func__);
- VM_WARN(pgdir != current->mm->pgd,
- "%s lock less page table lookup called on wrong mm\n", __func__);
- pte = __find_linux_pte(pgdir, ea, is_thp, hshift);
-
-#if defined(CONFIG_DEBUG_VM) && \
- !(defined(CONFIG_HUGETLB_PAGE) || defined(CONFIG_TRANSPARENT_HUGEPAGE))
- /*
- * We should not find huge page if these configs are not enabled.
- */
- if (hshift)
- WARN_ON(*hshift);
-#endif
- return pte;
-}
-
#endif /* _ASM_POWERPC_PTE_WALK_H */
diff --git a/arch/powerpc/include/asm/ptrace.h b/arch/powerpc/include/asm/ptrace.h
index 2efec6d..0eb90a0 100644
--- a/arch/powerpc/include/asm/ptrace.h
+++ b/arch/powerpc/include/asm/ptrace.h
@@ -97,8 +97,6 @@ struct pt_regs
#endif
-#define STACK_FRAME_WITH_PT_REGS (STACK_FRAME_OVERHEAD + sizeof(struct pt_regs))
-
// Always displays as "REGS" in memory dumps
#ifdef CONFIG_CPU_BIG_ENDIAN
#define STACK_FRAME_REGS_MARKER ASM_CONST(0x52454753)
@@ -120,16 +118,27 @@ struct pt_regs
#define USER_REDZONE_SIZE 512
#define KERNEL_REDZONE_SIZE 288
-#define STACK_FRAME_OVERHEAD 112 /* size of minimum stack frame */
#define STACK_FRAME_LR_SAVE 2 /* Location of LR in stack frame */
-#define STACK_INT_FRAME_SIZE (sizeof(struct pt_regs) + \
- STACK_FRAME_OVERHEAD + KERNEL_REDZONE_SIZE)
-#define STACK_FRAME_MARKER 12
#ifdef CONFIG_PPC64_ELF_ABI_V2
#define STACK_FRAME_MIN_SIZE 32
+#define STACK_USER_INT_FRAME_SIZE (sizeof(struct pt_regs) + STACK_FRAME_MIN_SIZE + 16)
+#define STACK_INT_FRAME_REGS (STACK_FRAME_MIN_SIZE + 16)
+#define STACK_INT_FRAME_MARKER STACK_FRAME_MIN_SIZE
+#define STACK_SWITCH_FRAME_SIZE (sizeof(struct pt_regs) + STACK_FRAME_MIN_SIZE + 16)
+#define STACK_SWITCH_FRAME_REGS (STACK_FRAME_MIN_SIZE + 16)
#else
-#define STACK_FRAME_MIN_SIZE STACK_FRAME_OVERHEAD
+/*
+ * The ELFv1 ABI specifies 48 bytes plus a minimum 64 byte parameter save
+ * area. This parameter area is not used by calls to C from interrupt entry,
+ * so the second from last one of those is used for the frame marker.
+ */
+#define STACK_FRAME_MIN_SIZE 112
+#define STACK_USER_INT_FRAME_SIZE (sizeof(struct pt_regs) + STACK_FRAME_MIN_SIZE)
+#define STACK_INT_FRAME_REGS STACK_FRAME_MIN_SIZE
+#define STACK_INT_FRAME_MARKER (STACK_FRAME_MIN_SIZE - 16)
+#define STACK_SWITCH_FRAME_SIZE (sizeof(struct pt_regs) + STACK_FRAME_MIN_SIZE)
+#define STACK_SWITCH_FRAME_REGS STACK_FRAME_MIN_SIZE
#endif
/* Size of dummy stack frame allocated when calling signal handler. */
@@ -140,17 +149,22 @@ struct pt_regs
#define USER_REDZONE_SIZE 0
#define KERNEL_REDZONE_SIZE 0
-#define STACK_FRAME_OVERHEAD 16 /* size of minimum stack frame */
+#define STACK_FRAME_MIN_SIZE 16
#define STACK_FRAME_LR_SAVE 1 /* Location of LR in stack frame */
-#define STACK_INT_FRAME_SIZE (sizeof(struct pt_regs) + STACK_FRAME_OVERHEAD)
-#define STACK_FRAME_MARKER 2
-#define STACK_FRAME_MIN_SIZE STACK_FRAME_OVERHEAD
+#define STACK_USER_INT_FRAME_SIZE (sizeof(struct pt_regs) + STACK_FRAME_MIN_SIZE)
+#define STACK_INT_FRAME_REGS STACK_FRAME_MIN_SIZE
+#define STACK_INT_FRAME_MARKER (STACK_FRAME_MIN_SIZE - 8)
+#define STACK_SWITCH_FRAME_SIZE (sizeof(struct pt_regs) + STACK_FRAME_MIN_SIZE)
+#define STACK_SWITCH_FRAME_REGS STACK_FRAME_MIN_SIZE
/* Size of stack frame allocated when calling signal handler. */
#define __SIGNAL_FRAMESIZE 64
#endif /* __powerpc64__ */
+#define STACK_INT_FRAME_SIZE (KERNEL_REDZONE_SIZE + STACK_USER_INT_FRAME_SIZE)
+#define STACK_INT_FRAME_MARKER_LONGS (STACK_INT_FRAME_MARKER/sizeof(long))
+
#ifndef __ASSEMBLY__
#include <asm/paca.h>
diff --git a/arch/powerpc/include/asm/qspinlock.h b/arch/powerpc/include/asm/qspinlock.h
index b676c4f..28a53fb 100644
--- a/arch/powerpc/include/asm/qspinlock.h
+++ b/arch/powerpc/include/asm/qspinlock.h
@@ -2,83 +2,173 @@
#ifndef _ASM_POWERPC_QSPINLOCK_H
#define _ASM_POWERPC_QSPINLOCK_H
-#include <asm-generic/qspinlock_types.h>
+#include <linux/compiler.h>
+#include <asm/qspinlock_types.h>
#include <asm/paravirt.h>
-#define _Q_PENDING_LOOPS (1 << 9) /* not tuned */
-
-#ifdef CONFIG_PARAVIRT_SPINLOCKS
-extern void native_queued_spin_lock_slowpath(struct qspinlock *lock, u32 val);
-extern void __pv_queued_spin_lock_slowpath(struct qspinlock *lock, u32 val);
-extern void __pv_queued_spin_unlock(struct qspinlock *lock);
-
-static __always_inline void queued_spin_lock_slowpath(struct qspinlock *lock, u32 val)
-{
- if (!is_shared_processor())
- native_queued_spin_lock_slowpath(lock, val);
- else
- __pv_queued_spin_lock_slowpath(lock, val);
-}
-
-#define queued_spin_unlock queued_spin_unlock
-static inline void queued_spin_unlock(struct qspinlock *lock)
-{
- if (!is_shared_processor())
- smp_store_release(&lock->locked, 0);
- else
- __pv_queued_spin_unlock(lock);
-}
-
+#ifdef CONFIG_PPC64
+/*
+ * Use the EH=1 hint for accesses that result in the lock being acquired.
+ * The hardware is supposed to optimise this pattern by holding the lock
+ * cacheline longer, and releasing when a store to the same memory (the
+ * unlock) is performed.
+ */
+#define _Q_SPIN_EH_HINT 1
#else
-extern void queued_spin_lock_slowpath(struct qspinlock *lock, u32 val);
-#endif
-
-static __always_inline void queued_spin_lock(struct qspinlock *lock)
-{
- u32 val = 0;
-
- if (likely(arch_atomic_try_cmpxchg_lock(&lock->val, &val, _Q_LOCKED_VAL)))
- return;
-
- queued_spin_lock_slowpath(lock, val);
-}
-#define queued_spin_lock queued_spin_lock
-
-#ifdef CONFIG_PARAVIRT_SPINLOCKS
-#define SPIN_THRESHOLD (1<<15) /* not tuned */
-
-static __always_inline void pv_wait(u8 *ptr, u8 val)
-{
- if (*ptr != val)
- return;
- yield_to_any();
- /*
- * We could pass in a CPU here if waiting in the queue and yield to
- * the previous CPU in the queue.
- */
-}
-
-static __always_inline void pv_kick(int cpu)
-{
- prod_cpu(cpu);
-}
-
-extern void __pv_init_lock_hash(void);
-
-static inline void pv_spinlocks_init(void)
-{
- __pv_init_lock_hash();
-}
-
+#define _Q_SPIN_EH_HINT 0
#endif
/*
- * Queued spinlocks rely heavily on smp_cond_load_relaxed() to busy-wait,
- * which was found to have performance problems if implemented with
- * the preferred spin_begin()/spin_end() SMT priority pattern. Use the
- * generic version instead.
+ * The trylock itself may steal. This makes trylocks slightly stronger, and
+ * makes locks slightly more efficient when stealing.
+ *
+ * This is compile-time, so if true then there may always be stealers, so the
+ * nosteal paths become unused.
*/
+#define _Q_SPIN_TRY_LOCK_STEAL 1
-#include <asm-generic/qspinlock.h>
+/*
+ * Put a speculation barrier after testing the lock/node and finding it
+ * busy. Try to prevent pointless speculation in slow paths.
+ *
+ * Slows down the lockstorm microbenchmark with no stealing, where locking
+ * is purely FIFO through the queue. May have more benefit in real workload
+ * where speculating into the wrong place could have a greater cost.
+ */
+#define _Q_SPIN_SPEC_BARRIER 0
+
+#ifdef CONFIG_PPC64
+/*
+ * Execute a miso instruction after passing the MCS lock ownership to the
+ * queue head. Miso is intended to make stores visible to other CPUs sooner.
+ *
+ * This seems to make the lockstorm microbenchmark nospin test go slightly
+ * faster on POWER10, but disable for now.
+ */
+#define _Q_SPIN_MISO 0
+#else
+#define _Q_SPIN_MISO 0
+#endif
+
+#ifdef CONFIG_PPC64
+/*
+ * This executes miso after an unlock of the lock word, having ownership
+ * pass to the next CPU sooner. This will slow the uncontended path to some
+ * degree. Not evidence it helps yet.
+ */
+#define _Q_SPIN_MISO_UNLOCK 0
+#else
+#define _Q_SPIN_MISO_UNLOCK 0
+#endif
+
+/*
+ * Seems to slow down lockstorm microbenchmark, suspect queue node just
+ * has to become shared again right afterwards when its waiter spins on
+ * the lock field.
+ */
+#define _Q_SPIN_PREFETCH_NEXT 0
+
+static __always_inline int queued_spin_is_locked(struct qspinlock *lock)
+{
+ return READ_ONCE(lock->val);
+}
+
+static __always_inline int queued_spin_value_unlocked(struct qspinlock lock)
+{
+ return !lock.val;
+}
+
+static __always_inline int queued_spin_is_contended(struct qspinlock *lock)
+{
+ return !!(READ_ONCE(lock->val) & _Q_TAIL_CPU_MASK);
+}
+
+static __always_inline u32 queued_spin_encode_locked_val(void)
+{
+ /* XXX: make this use lock value in paca like simple spinlocks? */
+ return _Q_LOCKED_VAL | (smp_processor_id() << _Q_OWNER_CPU_OFFSET);
+}
+
+static __always_inline int __queued_spin_trylock_nosteal(struct qspinlock *lock)
+{
+ u32 new = queued_spin_encode_locked_val();
+ u32 prev;
+
+ /* Trylock succeeds only when unlocked and no queued nodes */
+ asm volatile(
+"1: lwarx %0,0,%1,%3 # __queued_spin_trylock_nosteal \n"
+" cmpwi 0,%0,0 \n"
+" bne- 2f \n"
+" stwcx. %2,0,%1 \n"
+" bne- 1b \n"
+"\t" PPC_ACQUIRE_BARRIER " \n"
+"2: \n"
+ : "=&r" (prev)
+ : "r" (&lock->val), "r" (new),
+ "i" (_Q_SPIN_EH_HINT)
+ : "cr0", "memory");
+
+ return likely(prev == 0);
+}
+
+static __always_inline int __queued_spin_trylock_steal(struct qspinlock *lock)
+{
+ u32 new = queued_spin_encode_locked_val();
+ u32 prev, tmp;
+
+ /* Trylock may get ahead of queued nodes if it finds unlocked */
+ asm volatile(
+"1: lwarx %0,0,%2,%5 # __queued_spin_trylock_steal \n"
+" andc. %1,%0,%4 \n"
+" bne- 2f \n"
+" and %1,%0,%4 \n"
+" or %1,%1,%3 \n"
+" stwcx. %1,0,%2 \n"
+" bne- 1b \n"
+"\t" PPC_ACQUIRE_BARRIER " \n"
+"2: \n"
+ : "=&r" (prev), "=&r" (tmp)
+ : "r" (&lock->val), "r" (new), "r" (_Q_TAIL_CPU_MASK),
+ "i" (_Q_SPIN_EH_HINT)
+ : "cr0", "memory");
+
+ return likely(!(prev & ~_Q_TAIL_CPU_MASK));
+}
+
+static __always_inline int queued_spin_trylock(struct qspinlock *lock)
+{
+ if (!_Q_SPIN_TRY_LOCK_STEAL)
+ return __queued_spin_trylock_nosteal(lock);
+ else
+ return __queued_spin_trylock_steal(lock);
+}
+
+void queued_spin_lock_slowpath(struct qspinlock *lock);
+
+static __always_inline void queued_spin_lock(struct qspinlock *lock)
+{
+ if (!queued_spin_trylock(lock))
+ queued_spin_lock_slowpath(lock);
+}
+
+static inline void queued_spin_unlock(struct qspinlock *lock)
+{
+ smp_store_release(&lock->locked, 0);
+ if (_Q_SPIN_MISO_UNLOCK)
+ asm volatile("miso" ::: "memory");
+}
+
+#define arch_spin_is_locked(l) queued_spin_is_locked(l)
+#define arch_spin_is_contended(l) queued_spin_is_contended(l)
+#define arch_spin_value_unlocked(l) queued_spin_value_unlocked(l)
+#define arch_spin_lock(l) queued_spin_lock(l)
+#define arch_spin_trylock(l) queued_spin_trylock(l)
+#define arch_spin_unlock(l) queued_spin_unlock(l)
+
+#ifdef CONFIG_PARAVIRT_SPINLOCKS
+void pv_spinlocks_init(void);
+#else
+static inline void pv_spinlocks_init(void) { }
+#endif
#endif /* _ASM_POWERPC_QSPINLOCK_H */
diff --git a/arch/powerpc/include/asm/qspinlock_paravirt.h b/arch/powerpc/include/asm/qspinlock_paravirt.h
deleted file mode 100644
index 6b60e77..0000000
--- a/arch/powerpc/include/asm/qspinlock_paravirt.h
+++ /dev/null
@@ -1,7 +0,0 @@
-/* SPDX-License-Identifier: GPL-2.0-or-later */
-#ifndef _ASM_POWERPC_QSPINLOCK_PARAVIRT_H
-#define _ASM_POWERPC_QSPINLOCK_PARAVIRT_H
-
-EXPORT_SYMBOL(__pv_queued_spin_unlock);
-
-#endif /* _ASM_POWERPC_QSPINLOCK_PARAVIRT_H */
diff --git a/arch/powerpc/include/asm/qspinlock_types.h b/arch/powerpc/include/asm/qspinlock_types.h
new file mode 100644
index 0000000..4766a7aa
--- /dev/null
+++ b/arch/powerpc/include/asm/qspinlock_types.h
@@ -0,0 +1,72 @@
+/* SPDX-License-Identifier: GPL-2.0-or-later */
+#ifndef _ASM_POWERPC_QSPINLOCK_TYPES_H
+#define _ASM_POWERPC_QSPINLOCK_TYPES_H
+
+#include <linux/types.h>
+#include <asm/byteorder.h>
+
+typedef struct qspinlock {
+ union {
+ u32 val;
+
+#ifdef __LITTLE_ENDIAN
+ struct {
+ u16 locked;
+ u8 reserved[2];
+ };
+#else
+ struct {
+ u8 reserved[2];
+ u16 locked;
+ };
+#endif
+ };
+} arch_spinlock_t;
+
+#define __ARCH_SPIN_LOCK_UNLOCKED { { .val = 0 } }
+
+/*
+ * Bitfields in the lock word:
+ *
+ * 0: locked bit
+ * 1-14: lock holder cpu
+ * 15: lock owner or queuer vcpus observed to be preempted bit
+ * 16: must queue bit
+ * 17-31: tail cpu (+1)
+ */
+#define _Q_SET_MASK(type) (((1U << _Q_ ## type ## _BITS) - 1)\
+ << _Q_ ## type ## _OFFSET)
+/* 0x00000001 */
+#define _Q_LOCKED_OFFSET 0
+#define _Q_LOCKED_BITS 1
+#define _Q_LOCKED_VAL (1U << _Q_LOCKED_OFFSET)
+
+/* 0x00007ffe */
+#define _Q_OWNER_CPU_OFFSET 1
+#define _Q_OWNER_CPU_BITS 14
+#define _Q_OWNER_CPU_MASK _Q_SET_MASK(OWNER_CPU)
+
+#if CONFIG_NR_CPUS > (1U << _Q_OWNER_CPU_BITS)
+#error "qspinlock does not support such large CONFIG_NR_CPUS"
+#endif
+
+/* 0x00008000 */
+#define _Q_SLEEPY_OFFSET 15
+#define _Q_SLEEPY_BITS 1
+#define _Q_SLEEPY_VAL (1U << _Q_SLEEPY_OFFSET)
+
+/* 0x00010000 */
+#define _Q_MUST_Q_OFFSET 16
+#define _Q_MUST_Q_BITS 1
+#define _Q_MUST_Q_VAL (1U << _Q_MUST_Q_OFFSET)
+
+/* 0xfffe0000 */
+#define _Q_TAIL_CPU_OFFSET 17
+#define _Q_TAIL_CPU_BITS 15
+#define _Q_TAIL_CPU_MASK _Q_SET_MASK(TAIL_CPU)
+
+#if CONFIG_NR_CPUS >= (1U << _Q_TAIL_CPU_BITS)
+#error "qspinlock does not support such large CONFIG_NR_CPUS"
+#endif
+
+#endif /* _ASM_POWERPC_QSPINLOCK_TYPES_H */
diff --git a/arch/powerpc/include/asm/rtas.h b/arch/powerpc/include/asm/rtas.h
index 56319ae..479a95c 100644
--- a/arch/powerpc/include/asm/rtas.h
+++ b/arch/powerpc/include/asm/rtas.h
@@ -33,21 +33,6 @@
#define RTAS_THREADS_ACTIVE -9005 /* Multiple processor threads active */
#define RTAS_OUTSTANDING_COPROC -9006 /* Outstanding coprocessor operations */
-/*
- * In general to call RTAS use rtas_token("string") to lookup
- * an RTAS token for the given string (e.g. "event-scan").
- * To actually perform the call use
- * ret = rtas_call(token, n_in, n_out, ...)
- * Where n_in is the number of input parameters and
- * n_out is the number of output parameters
- *
- * If the "string" is invalid on this system, RTAS_UNKNOWN_SERVICE
- * will be returned as a token. rtas_call() does look for this
- * token and error out gracefully so rtas_call(rtas_token("str"), ...)
- * may be safely used for one-shot calls to RTAS.
- *
- */
-
/* RTAS event classes */
#define RTAS_INTERNAL_ERROR 0x80000000 /* set bit 0 */
#define RTAS_EPOW_WARNING 0x40000000 /* set bit 1 */
diff --git a/arch/powerpc/include/asm/spinlock.h b/arch/powerpc/include/asm/spinlock.h
index bd75872..7dafca8e 100644
--- a/arch/powerpc/include/asm/spinlock.h
+++ b/arch/powerpc/include/asm/spinlock.h
@@ -13,7 +13,7 @@
/* See include/linux/spinlock.h */
#define smp_mb__after_spinlock() smp_mb()
-#ifndef CONFIG_PARAVIRT_SPINLOCKS
+#ifndef CONFIG_PPC_QUEUED_SPINLOCKS
static inline void pv_spinlocks_init(void) { }
#endif
diff --git a/arch/powerpc/include/asm/spinlock_types.h b/arch/powerpc/include/asm/spinlock_types.h
index d5f8a74..40b0144 100644
--- a/arch/powerpc/include/asm/spinlock_types.h
+++ b/arch/powerpc/include/asm/spinlock_types.h
@@ -7,7 +7,7 @@
#endif
#ifdef CONFIG_PPC_QUEUED_SPINLOCKS
-#include <asm-generic/qspinlock_types.h>
+#include <asm/qspinlock_types.h>
#include <asm-generic/qrwlock_types.h>
#else
#include <asm/simple_spinlock_types.h>
diff --git a/arch/powerpc/kernel/asm-offsets.c b/arch/powerpc/kernel/asm-offsets.c
index 4ce2a4a..d24a59a 100644
--- a/arch/powerpc/kernel/asm-offsets.c
+++ b/arch/powerpc/kernel/asm-offsets.c
@@ -72,7 +72,7 @@
#endif
#define STACK_PT_REGS_OFFSET(sym, val) \
- DEFINE(sym, STACK_FRAME_OVERHEAD + offsetof(struct pt_regs, val))
+ DEFINE(sym, STACK_INT_FRAME_REGS + offsetof(struct pt_regs, val))
int main(void)
{
@@ -167,9 +167,8 @@ int main(void)
OFFSET(THREAD_CKVRSTATE, thread_struct, ckvr_state.vr);
OFFSET(THREAD_CKVRSAVE, thread_struct, ckvrsave);
OFFSET(THREAD_CKFPSTATE, thread_struct, ckfp_state.fpr);
- /* Local pt_regs on stack for Transactional Memory funcs. */
- DEFINE(TM_FRAME_SIZE, STACK_FRAME_OVERHEAD +
- sizeof(struct pt_regs) + 16);
+ /* Local pt_regs on stack in int frame form, plus 16 bytes for TM */
+ DEFINE(TM_FRAME_SIZE, STACK_INT_FRAME_SIZE + 16);
#endif /* CONFIG_PPC_TRANSACTIONAL_MEM */
OFFSET(TI_LOCAL_FLAGS, thread_info, local_flags);
@@ -261,7 +260,7 @@ int main(void)
/* Interrupt register frame */
DEFINE(INT_FRAME_SIZE, STACK_INT_FRAME_SIZE);
- DEFINE(SWITCH_FRAME_SIZE, STACK_FRAME_WITH_PT_REGS);
+ DEFINE(SWITCH_FRAME_SIZE, STACK_SWITCH_FRAME_SIZE);
STACK_PT_REGS_OFFSET(GPR0, gpr[0]);
STACK_PT_REGS_OFFSET(GPR1, gpr[1]);
STACK_PT_REGS_OFFSET(GPR2, gpr[2]);
@@ -418,21 +417,18 @@ int main(void)
/* book3s */
#ifdef CONFIG_KVM_BOOK3S_HV_POSSIBLE
- OFFSET(KVM_TLB_SETS, kvm, arch.tlb_sets);
OFFSET(KVM_SDR1, kvm, arch.sdr1);
OFFSET(KVM_HOST_LPID, kvm, arch.host_lpid);
OFFSET(KVM_HOST_LPCR, kvm, arch.host_lpcr);
OFFSET(KVM_HOST_SDR1, kvm, arch.host_sdr1);
OFFSET(KVM_ENABLED_HCALLS, kvm, arch.enabled_hcalls);
OFFSET(KVM_VRMA_SLB_V, kvm, arch.vrma_slb_v);
- OFFSET(KVM_RADIX, kvm, arch.radix);
OFFSET(KVM_SECURE_GUEST, kvm, arch.secure_guest);
OFFSET(VCPU_DSISR, kvm_vcpu, arch.shregs.dsisr);
OFFSET(VCPU_DAR, kvm_vcpu, arch.shregs.dar);
OFFSET(VCPU_VPA, kvm_vcpu, arch.vpa.pinned_addr);
OFFSET(VCPU_VPA_DIRTY, kvm_vcpu, arch.vpa.dirty);
OFFSET(VCPU_HEIR, kvm_vcpu, arch.emul_inst);
- OFFSET(VCPU_NESTED, kvm_vcpu, arch.nested);
OFFSET(VCPU_CPU, kvm_vcpu, cpu);
OFFSET(VCPU_THREAD_CPU, kvm_vcpu, arch.thread_cpu);
#endif
@@ -449,16 +445,12 @@ int main(void)
OFFSET(VCPU_DABRX, kvm_vcpu, arch.dabrx);
OFFSET(VCPU_DAWR0, kvm_vcpu, arch.dawr0);
OFFSET(VCPU_DAWRX0, kvm_vcpu, arch.dawrx0);
- OFFSET(VCPU_DAWR1, kvm_vcpu, arch.dawr1);
- OFFSET(VCPU_DAWRX1, kvm_vcpu, arch.dawrx1);
OFFSET(VCPU_CIABR, kvm_vcpu, arch.ciabr);
OFFSET(VCPU_HFLAGS, kvm_vcpu, arch.hflags);
OFFSET(VCPU_DEC_EXPIRES, kvm_vcpu, arch.dec_expires);
OFFSET(VCPU_PENDING_EXC, kvm_vcpu, arch.pending_exceptions);
OFFSET(VCPU_CEDED, kvm_vcpu, arch.ceded);
OFFSET(VCPU_PRODDED, kvm_vcpu, arch.prodded);
- OFFSET(VCPU_IRQ_PENDING, kvm_vcpu, arch.irq_pending);
- OFFSET(VCPU_DBELL_REQ, kvm_vcpu, arch.doorbell_request);
OFFSET(VCPU_MMCR, kvm_vcpu, arch.mmcr);
OFFSET(VCPU_MMCRA, kvm_vcpu, arch.mmcra);
OFFSET(VCPU_MMCRS, kvm_vcpu, arch.mmcrs);
@@ -486,8 +478,6 @@ int main(void)
OFFSET(VCPU_TCSCR, kvm_vcpu, arch.tcscr);
OFFSET(VCPU_ACOP, kvm_vcpu, arch.acop);
OFFSET(VCPU_WORT, kvm_vcpu, arch.wort);
- OFFSET(VCPU_TID, kvm_vcpu, arch.tid);
- OFFSET(VCPU_PSSCR, kvm_vcpu, arch.psscr);
OFFSET(VCPU_HFSCR, kvm_vcpu, arch.hfscr);
OFFSET(VCORE_ENTRY_EXIT, kvmppc_vcore, entry_exit_map);
OFFSET(VCORE_IN_GUEST, kvmppc_vcore, in_guest);
@@ -582,8 +572,6 @@ int main(void)
HSTATE_FIELD(HSTATE_HWTHREAD_STATE, hwthread_state);
HSTATE_FIELD(HSTATE_KVM_VCPU, kvm_vcpu);
HSTATE_FIELD(HSTATE_KVM_VCORE, kvm_vcore);
- HSTATE_FIELD(HSTATE_XIVE_TIMA_PHYS, xive_tima_phys);
- HSTATE_FIELD(HSTATE_XIVE_TIMA_VIRT, xive_tima_virt);
HSTATE_FIELD(HSTATE_HOST_IPI, host_ipi);
HSTATE_FIELD(HSTATE_PTID, ptid);
HSTATE_FIELD(HSTATE_FAKE_SUSPEND, fake_suspend);
@@ -594,9 +582,6 @@ int main(void)
HSTATE_FIELD(HSTATE_SDAR, host_mmcr[4]);
HSTATE_FIELD(HSTATE_MMCR2, host_mmcr[5]);
HSTATE_FIELD(HSTATE_SIER, host_mmcr[6]);
- HSTATE_FIELD(HSTATE_MMCR3, host_mmcr[7]);
- HSTATE_FIELD(HSTATE_SIER2, host_mmcr[8]);
- HSTATE_FIELD(HSTATE_SIER3, host_mmcr[9]);
HSTATE_FIELD(HSTATE_PMC1, host_pmc[0]);
HSTATE_FIELD(HSTATE_PMC2, host_pmc[1]);
HSTATE_FIELD(HSTATE_PMC3, host_pmc[2]);
@@ -672,17 +657,6 @@ int main(void)
OFFSET(VCPU_HOST_MAS6, kvm_vcpu, arch.host_mas6);
#endif
-#ifdef CONFIG_KVM_XICS
- DEFINE(VCPU_XIVE_SAVED_STATE, offsetof(struct kvm_vcpu,
- arch.xive_saved_state));
- DEFINE(VCPU_XIVE_CAM_WORD, offsetof(struct kvm_vcpu,
- arch.xive_cam_word));
- DEFINE(VCPU_XIVE_PUSHED, offsetof(struct kvm_vcpu, arch.xive_pushed));
- DEFINE(VCPU_XIVE_ESC_ON, offsetof(struct kvm_vcpu, arch.xive_esc_on));
- DEFINE(VCPU_XIVE_ESC_RADDR, offsetof(struct kvm_vcpu, arch.xive_esc_raddr));
- DEFINE(VCPU_XIVE_ESC_VADDR, offsetof(struct kvm_vcpu, arch.xive_esc_vaddr));
-#endif
-
#ifdef CONFIG_KVM_EXIT_TIMING
OFFSET(VCPU_TIMING_EXIT_TBU, kvm_vcpu, arch.timing_exit.tv32.tbu);
OFFSET(VCPU_TIMING_EXIT_TBL, kvm_vcpu, arch.timing_exit.tv32.tbl);
diff --git a/arch/powerpc/kernel/cpu_setup_6xx.S b/arch/powerpc/kernel/cpu_setup_6xx.S
index f8b5ff6..f29ce3d 100644
--- a/arch/powerpc/kernel/cpu_setup_6xx.S
+++ b/arch/powerpc/kernel/cpu_setup_6xx.S
@@ -4,6 +4,8 @@
* Copyright (C) 2003 Benjamin Herrenschmidt (benh@kernel.crashing.org)
*/
+#include <linux/linkage.h>
+
#include <asm/processor.h>
#include <asm/page.h>
#include <asm/cputable.h>
@@ -81,7 +83,7 @@
blr
/* Enable caches for 603's, 604, 750 & 7400 */
-setup_common_caches:
+SYM_FUNC_START_LOCAL(setup_common_caches)
mfspr r11,SPRN_HID0
andi. r0,r11,HID0_DCE
ori r11,r11,HID0_ICE|HID0_DCE
@@ -95,11 +97,12 @@
sync
isync
blr
+SYM_FUNC_END(setup_common_caches)
/* 604, 604e, 604ev, ...
* Enable superscalar execution & branch history table
*/
-setup_604_hid0:
+SYM_FUNC_START_LOCAL(setup_604_hid0)
mfspr r11,SPRN_HID0
ori r11,r11,HID0_SIED|HID0_BHTE
ori r8,r11,HID0_BTCD
@@ -110,6 +113,7 @@
sync
isync
blr
+SYM_FUNC_END(setup_604_hid0)
/* 7400 <= rev 2.7 and 7410 rev = 1.0 suffer from some
* erratas we work around here.
@@ -125,13 +129,14 @@
* needed once we have applied workaround #5 (though it's
* not set by Apple's firmware at least).
*/
-setup_7400_workarounds:
+SYM_FUNC_START_LOCAL(setup_7400_workarounds)
mfpvr r3
rlwinm r3,r3,0,20,31
cmpwi 0,r3,0x0207
ble 1f
blr
-setup_7410_workarounds:
+SYM_FUNC_END(setup_7400_workarounds)
+SYM_FUNC_START_LOCAL(setup_7410_workarounds)
mfpvr r3
rlwinm r3,r3,0,20,31
cmpwi 0,r3,0x0100
@@ -151,6 +156,7 @@
sync
isync
blr
+SYM_FUNC_END(setup_7410_workarounds)
/* 740/750/7400/7410
* Enable Store Gathering (SGE), Address Broadcast (ABE),
@@ -158,7 +164,7 @@
* Dynamic Power Management (DPM), Speculative (SPD)
* Clear Instruction cache throttling (ICTC)
*/
-setup_750_7400_hid0:
+SYM_FUNC_START_LOCAL(setup_750_7400_hid0)
mfspr r11,SPRN_HID0
ori r11,r11,HID0_SGE | HID0_ABE | HID0_BHTE | HID0_BTIC
oris r11,r11,HID0_DPM@h
@@ -177,12 +183,13 @@
sync
isync
blr
+SYM_FUNC_END(setup_750_7400_hid0)
/* 750cx specific
* Looks like we have to disable NAP feature for some PLL settings...
* (waiting for confirmation)
*/
-setup_750cx:
+SYM_FUNC_START_LOCAL(setup_750cx)
mfspr r10, SPRN_HID1
rlwinm r10,r10,4,28,31
cmpwi cr0,r10,7
@@ -196,11 +203,13 @@
andc r6,r6,r7
stw r6,CPU_SPEC_FEATURES(r4)
blr
+SYM_FUNC_END(setup_750cx)
/* 750fx specific
*/
-setup_750fx:
+SYM_FUNC_START_LOCAL(setup_750fx)
blr
+SYM_FUNC_END(setup_750fx)
/* MPC 745x
* Enable Store Gathering (SGE), Branch Folding (FOLD)
@@ -212,7 +221,7 @@
* Clear Instruction cache throttling (ICTC)
* Enable L2 HW prefetch
*/
-setup_745x_specifics:
+SYM_FUNC_START_LOCAL(setup_745x_specifics)
/* We check for the presence of an L3 cache setup by
* the firmware. If any, we disable NAP capability as
* it's known to be bogus on rev 2.1 and earlier
@@ -270,6 +279,7 @@
sync
isync
blr
+SYM_FUNC_END(setup_745x_specifics)
/*
* Initialize the FPU registers. This is needed to work around an errata
diff --git a/arch/powerpc/kernel/cpu_setup_e500.S b/arch/powerpc/kernel/cpu_setup_e500.S
index 2ab2516..077cfcc 100644
--- a/arch/powerpc/kernel/cpu_setup_e500.S
+++ b/arch/powerpc/kernel/cpu_setup_e500.S
@@ -8,6 +8,8 @@
* Benjamin Herrenschmidt <benh@kernel.crashing.org>
*/
+#include <linux/linkage.h>
+
#include <asm/page.h>
#include <asm/processor.h>
#include <asm/cputable.h>
@@ -274,7 +276,7 @@
blr
-has_L2_cache:
+SYM_FUNC_START_LOCAL(has_L2_cache)
/* skip L2 cache on P2040/P2040E as they have no L2 cache */
mfspr r3, SPRN_SVR
/* shift right by 8 bits and clear E bit of SVR */
@@ -290,9 +292,10 @@
1:
li r3, 0
blr
+SYM_FUNC_END(has_L2_cache)
/* flush backside L2 cache */
-flush_backside_L2_cache:
+SYM_FUNC_START_LOCAL(flush_backside_L2_cache)
mflr r10
bl has_L2_cache
mtlr r10
@@ -313,6 +316,7 @@
bne 1b
2:
blr
+SYM_FUNC_END(flush_backside_L2_cache)
_GLOBAL(cpu_down_flush_e500v2)
mflr r0
diff --git a/arch/powerpc/kernel/entry_32.S b/arch/powerpc/kernel/entry_32.S
index 3fc7c98..5604c9a 100644
--- a/arch/powerpc/kernel/entry_32.S
+++ b/arch/powerpc/kernel/entry_32.S
@@ -18,6 +18,8 @@
#include <linux/err.h>
#include <linux/sys.h>
#include <linux/threads.h>
+#include <linux/linkage.h>
+
#include <asm/reg.h>
#include <asm/page.h>
#include <asm/mmu.h>
@@ -74,17 +76,18 @@
#endif /* CONFIG_PPC_BOOK3S_32 || CONFIG_PPC_E500 */
#if defined(CONFIG_PPC_KUEP) && defined(CONFIG_PPC_BOOK3S_32)
- .globl __kuep_lock
-__kuep_lock:
+SYM_FUNC_START(__kuep_lock)
lwz r9, THREAD+THSR0(r2)
update_user_segments_by_4 r9, r10, r11, r12
blr
+SYM_FUNC_END(__kuep_lock)
-__kuep_unlock:
+SYM_FUNC_START_LOCAL(__kuep_unlock)
lwz r9, THREAD+THSR0(r2)
rlwinm r9,r9,0,~SR_NX
update_user_segments_by_4 r9, r10, r11, r12
blr
+SYM_FUNC_END(__kuep_unlock)
.macro kuep_lock
bl __kuep_lock
@@ -114,7 +117,7 @@
addi r12,r12,STACK_FRAME_REGS_MARKER@l
stw r9,_MSR(r1)
li r2, INTERRUPT_SYSCALL
- stw r12,8(r1)
+ stw r12,STACK_INT_FRAME_MARKER(r1)
stw r2,_TRAP(r1)
SAVE_GPR(0, r1)
SAVE_GPRS(3, 8, r1)
@@ -123,12 +126,12 @@
kuep_lock
/* Calling convention has r3 = regs, r4 = orig r0 */
- addi r3,r1,STACK_FRAME_OVERHEAD
+ addi r3,r1,STACK_INT_FRAME_REGS
mr r4,r0
bl system_call_exception
ret_from_syscall:
- addi r4,r1,STACK_FRAME_OVERHEAD
+ addi r4,r1,STACK_INT_FRAME_REGS
li r5,0
bl syscall_exit_prepare
#ifdef CONFIG_PPC_47x
@@ -215,9 +218,9 @@
* in arch/ppc/kernel/process.c
*/
_GLOBAL(_switch)
- stwu r1,-INT_FRAME_SIZE(r1)
+ stwu r1,-SWITCH_FRAME_SIZE(r1)
mflr r0
- stw r0,INT_FRAME_SIZE+4(r1)
+ stw r0,SWITCH_FRAME_SIZE+4(r1)
/* r3-r12 are caller saved -- Cort */
SAVE_NVGPRS(r1)
stw r0,_NIP(r1) /* Return to switch caller */
@@ -248,7 +251,7 @@
lwz r4,_NIP(r1) /* Return to _switch caller in new task */
mtlr r4
- addi r1,r1,INT_FRAME_SIZE
+ addi r1,r1,SWITCH_FRAME_SIZE
blr
.globl fast_exception_return
@@ -293,7 +296,7 @@
.globl interrupt_return
interrupt_return:
lwz r4,_MSR(r1)
- addi r3,r1,STACK_FRAME_OVERHEAD
+ addi r3,r1,STACK_INT_FRAME_REGS
andi. r0,r4,MSR_PR
beq .Lkernel_interrupt_return
bl interrupt_exit_user_prepare
diff --git a/arch/powerpc/kernel/entry_64.S b/arch/powerpc/kernel/entry_64.S
index 3e2e37e..1bf1121 100644
--- a/arch/powerpc/kernel/entry_64.S
+++ b/arch/powerpc/kernel/entry_64.S
@@ -14,6 +14,7 @@
* code, and exception/interrupt return code for PowerPC.
*/
+#include <linux/objtool.h>
#include <linux/errno.h>
#include <linux/err.h>
#include <asm/cache.h>
@@ -73,6 +74,7 @@
// Flush the link stack
.rept 64
+ ANNOTATE_INTRA_FUNCTION_CALL
bl .+4
.endr
b 1f
diff --git a/arch/powerpc/kernel/exceptions-64e.S b/arch/powerpc/kernel/exceptions-64e.S
index 2f68fb2..3f86091 100644
--- a/arch/powerpc/kernel/exceptions-64e.S
+++ b/arch/powerpc/kernel/exceptions-64e.S
@@ -358,7 +358,6 @@
std r14,PACA_EXMC+EX_R14(r13); \
std r15,PACA_EXMC+EX_R15(r13)
-
/* Core exception code for all exceptions except TLB misses. */
#define EXCEPTION_COMMON_LVL(n, scratch, excf) \
exc_##n##_common: \
@@ -391,10 +390,11 @@
std r10,_CCR(r1); /* store orig CR in stackframe */ \
std r9,GPR1(r1); /* store stack frame back link */ \
std r11,SOFTE(r1); /* and save it to stackframe */ \
- std r12,STACK_FRAME_OVERHEAD-16(r1); /* mark the frame */ \
+ std r12,STACK_INT_FRAME_MARKER(r1); /* mark the frame */ \
std r3,_TRAP(r1); /* set trap number */ \
std r0,RESULT(r1); /* clear regs->result */ \
- SAVE_NVGPRS(r1);
+ SAVE_NVGPRS(r1); \
+ SANITIZE_NVGPRS(); /* minimise speculation influence */
#define EXCEPTION_COMMON(n) \
EXCEPTION_COMMON_LVL(n, SPRN_SPRG_GEN_SCRATCH, PACA_EXGEN)
@@ -455,7 +455,7 @@
EXCEPTION_COMMON(trapnum) \
ack(r8); \
CHECK_NAPPING(); \
- addi r3,r1,STACK_FRAME_OVERHEAD; \
+ addi r3,r1,STACK_INT_FRAME_REGS; \
bl hdlr; \
b interrupt_return
@@ -504,7 +504,7 @@
EXCEPTION_COMMON_CRIT(0x100)
bl special_reg_save
CHECK_NAPPING();
- addi r3,r1,STACK_FRAME_OVERHEAD
+ addi r3,r1,STACK_INT_FRAME_REGS
bl unknown_nmi_exception
b ret_from_crit_except
@@ -515,7 +515,7 @@
EXCEPTION_COMMON_MC(0x000)
bl special_reg_save
CHECK_NAPPING();
- addi r3,r1,STACK_FRAME_OVERHEAD
+ addi r3,r1,STACK_INT_FRAME_REGS
bl machine_check_exception
b ret_from_mc_except
@@ -570,7 +570,7 @@
std r14,_ESR(r1)
ld r14,PACA_EXGEN+EX_R14(r13)
EXCEPTION_COMMON(0x700)
- addi r3,r1,STACK_FRAME_OVERHEAD
+ addi r3,r1,STACK_INT_FRAME_REGS
bl program_check_exception
REST_NVGPRS(r1)
b interrupt_return
@@ -586,7 +586,7 @@
beq- 1f
bl load_up_fpu
b fast_interrupt_return
-1: addi r3,r1,STACK_FRAME_OVERHEAD
+1: addi r3,r1,STACK_INT_FRAME_REGS
bl kernel_fp_unavailable_exception
b interrupt_return
@@ -606,7 +606,7 @@
1:
END_FTR_SECTION_IFSET(CPU_FTR_ALTIVEC)
#endif
- addi r3,r1,STACK_FRAME_OVERHEAD
+ addi r3,r1,STACK_INT_FRAME_REGS
bl altivec_unavailable_exception
b interrupt_return
@@ -616,7 +616,7 @@
BOOKE_INTERRUPT_ALTIVEC_ASSIST,
PROLOG_ADDITION_NONE)
EXCEPTION_COMMON(0x220)
- addi r3,r1,STACK_FRAME_OVERHEAD
+ addi r3,r1,STACK_INT_FRAME_REGS
#ifdef CONFIG_ALTIVEC
BEGIN_FTR_SECTION
bl altivec_assist_exception
@@ -643,7 +643,7 @@
EXCEPTION_COMMON_CRIT(0x9f0)
bl special_reg_save
CHECK_NAPPING();
- addi r3,r1,STACK_FRAME_OVERHEAD
+ addi r3,r1,STACK_INT_FRAME_REGS
#ifdef CONFIG_BOOKE_WDT
bl WatchdogException
#else
@@ -664,7 +664,7 @@
NORMAL_EXCEPTION_PROLOG(0xf20, BOOKE_INTERRUPT_AP_UNAVAIL,
PROLOG_ADDITION_NONE)
EXCEPTION_COMMON(0xf20)
- addi r3,r1,STACK_FRAME_OVERHEAD
+ addi r3,r1,STACK_INT_FRAME_REGS
bl unknown_exception
b interrupt_return
@@ -731,7 +731,7 @@
ld r14,PACA_EXCRIT+EX_R14(r13)
ld r15,PACA_EXCRIT+EX_R15(r13)
EXCEPTION_COMMON_CRIT(0xd00)
- addi r3,r1,STACK_FRAME_OVERHEAD
+ addi r3,r1,STACK_INT_FRAME_REGS
bl DebugException
REST_NVGPRS(r1)
b interrupt_return
@@ -802,7 +802,7 @@
ld r14,PACA_EXDBG+EX_R14(r13)
ld r15,PACA_EXDBG+EX_R15(r13)
EXCEPTION_COMMON_DBG(0xd08)
- addi r3,r1,STACK_FRAME_OVERHEAD
+ addi r3,r1,STACK_INT_FRAME_REGS
bl DebugException
REST_NVGPRS(r1)
b interrupt_return
@@ -812,7 +812,7 @@
PROLOG_ADDITION_NONE)
EXCEPTION_COMMON(0x260)
CHECK_NAPPING()
- addi r3,r1,STACK_FRAME_OVERHEAD
+ addi r3,r1,STACK_INT_FRAME_REGS
/*
* XXX: Returning from performance_monitor_exception taken as a
* soft-NMI (Linux irqs disabled) may be risky to use interrupt_return
@@ -834,7 +834,7 @@
EXCEPTION_COMMON_CRIT(0x2a0)
bl special_reg_save
CHECK_NAPPING();
- addi r3,r1,STACK_FRAME_OVERHEAD
+ addi r3,r1,STACK_INT_FRAME_REGS
bl unknown_nmi_exception
b ret_from_crit_except
@@ -846,7 +846,7 @@
GDBELL_EXCEPTION_PROLOG(0x2c0, BOOKE_INTERRUPT_GUEST_DBELL,
PROLOG_ADDITION_NONE)
EXCEPTION_COMMON(0x2c0)
- addi r3,r1,STACK_FRAME_OVERHEAD
+ addi r3,r1,STACK_INT_FRAME_REGS
bl unknown_exception
b interrupt_return
@@ -857,7 +857,7 @@
EXCEPTION_COMMON_CRIT(0x2e0)
bl special_reg_save
CHECK_NAPPING();
- addi r3,r1,STACK_FRAME_OVERHEAD
+ addi r3,r1,STACK_INT_FRAME_REGS
bl unknown_nmi_exception
b ret_from_crit_except
@@ -866,7 +866,7 @@
NORMAL_EXCEPTION_PROLOG(0x310, BOOKE_INTERRUPT_HV_SYSCALL,
PROLOG_ADDITION_NONE)
EXCEPTION_COMMON(0x310)
- addi r3,r1,STACK_FRAME_OVERHEAD
+ addi r3,r1,STACK_INT_FRAME_REGS
bl unknown_exception
b interrupt_return
@@ -875,7 +875,7 @@
NORMAL_EXCEPTION_PROLOG(0x320, BOOKE_INTERRUPT_HV_PRIV,
PROLOG_ADDITION_NONE)
EXCEPTION_COMMON(0x320)
- addi r3,r1,STACK_FRAME_OVERHEAD
+ addi r3,r1,STACK_INT_FRAME_REGS
bl unknown_exception
b interrupt_return
@@ -884,7 +884,7 @@
NORMAL_EXCEPTION_PROLOG(0x340, BOOKE_INTERRUPT_LRAT_ERROR,
PROLOG_ADDITION_NONE)
EXCEPTION_COMMON(0x340)
- addi r3,r1,STACK_FRAME_OVERHEAD
+ addi r3,r1,STACK_INT_FRAME_REGS
bl unknown_exception
b interrupt_return
@@ -979,7 +979,7 @@
* original values stashed away in the PACA
*/
storage_fault_common:
- addi r3,r1,STACK_FRAME_OVERHEAD
+ addi r3,r1,STACK_INT_FRAME_REGS
bl do_page_fault
b interrupt_return
@@ -988,7 +988,7 @@
* continues here.
*/
alignment_more:
- addi r3,r1,STACK_FRAME_OVERHEAD
+ addi r3,r1,STACK_INT_FRAME_REGS
bl alignment_exception
REST_NVGPRS(r1)
b interrupt_return
@@ -1069,7 +1069,7 @@
ZEROIZE_GPR(12)
std r12,0(r11)
LOAD_PACA_TOC()
-1: addi r3,r1,STACK_FRAME_OVERHEAD
+1: addi r3,r1,STACK_INT_FRAME_REGS
bl kernel_bad_stack
b 1b
diff --git a/arch/powerpc/kernel/exceptions-64s.S b/arch/powerpc/kernel/exceptions-64s.S
index 651c36b..6441a1b 100644
--- a/arch/powerpc/kernel/exceptions-64s.S
+++ b/arch/powerpc/kernel/exceptions-64s.S
@@ -13,6 +13,7 @@
*
*/
+#include <linux/linkage.h>
#include <asm/hw_irq.h>
#include <asm/exception-64s.h>
#include <asm/ptrace.h>
@@ -111,6 +112,7 @@
#define ISTACK .L_ISTACK_\name\() /* Set regular kernel stack */
#define __ISTACK(name) .L_ISTACK_ ## name
#define IKUAP .L_IKUAP_\name\() /* Do KUAP lock */
+#define IMSR_R12 .L_IMSR_R12_\name\() /* Assumes MSR saved to r12 */
#define INT_DEFINE_BEGIN(n) \
.macro int_define_ ## n name
@@ -176,6 +178,9 @@
.ifndef IKUAP
IKUAP=1
.endif
+ .ifndef IMSR_R12
+ IMSR_R12=0
+ .endif
.endm
/*
@@ -502,6 +507,7 @@
std r10,0(r1) /* make stack chain pointer */
std r0,GPR0(r1) /* save r0 in stackframe */
std r10,GPR1(r1) /* save r1 in stackframe */
+ SANITIZE_GPR(0)
/* Mark our [H]SRRs valid for return */
li r10,1
@@ -544,8 +550,14 @@
std r9,GPR11(r1)
std r10,GPR12(r1)
std r11,GPR13(r1)
+ .if !IMSR_R12
+ SANITIZE_GPRS(9, 12)
+ .else
+ SANITIZE_GPRS(9, 11)
+ .endif
SAVE_NVGPRS(r1)
+ SANITIZE_NVGPRS()
.if IDAR
.if IISIDE
@@ -577,8 +589,8 @@
END_FTR_SECTION_IFSET(CPU_FTR_CFAR)
ld r10,IAREA+EX_CTR(r13)
std r10,_CTR(r1)
- std r2,GPR2(r1) /* save r2 in stackframe */
- SAVE_GPRS(3, 8, r1) /* save r3 - r8 in stackframe */
+ SAVE_GPRS(2, 8, r1) /* save r2 - r8 in stackframe */
+ SANITIZE_GPRS(2, 8)
mflr r9 /* Get LR, later save to stack */
LOAD_PACA_TOC() /* get kernel TOC into r2 */
std r9,_LINK(r1)
@@ -591,7 +603,7 @@
li r10,0
LOAD_REG_IMMEDIATE(r11, STACK_FRAME_REGS_MARKER)
std r10,RESULT(r1) /* clear regs->result */
- std r11,STACK_FRAME_OVERHEAD-16(r1) /* mark the frame */
+ std r11,STACK_INT_FRAME_MARKER(r1) /* mark the frame */
.endm
/*
@@ -696,6 +708,7 @@
mtlr r9
ld r9,_CCR(r1)
mtcr r9
+ SANITIZE_RESTORE_NVGPRS()
REST_GPRS(2, 13, r1)
REST_GPR(0, r1)
/* restore original r1. */
@@ -1061,7 +1074,7 @@
subi r1,r1,INT_FRAME_SIZE
__GEN_COMMON_BODY system_reset
- addi r3,r1,STACK_FRAME_OVERHEAD
+ addi r3,r1,STACK_INT_FRAME_REGS
bl system_reset_exception
/* Clear MSR_RI before setting SRR0 and SRR1. */
@@ -1208,7 +1221,7 @@
BEGIN_FTR_SECTION
bl enable_machine_check
END_FTR_SECTION_IFSET(CPU_FTR_HVMODE)
- addi r3,r1,STACK_FRAME_OVERHEAD
+ addi r3,r1,STACK_INT_FRAME_REGS
BEGIN_FTR_SECTION
bl machine_check_early_boot
END_FTR_SECTION(0, 1) // nop out after boot
@@ -1298,7 +1311,7 @@
* save area: PACA_EXMC instead of PACA_EXGEN.
*/
GEN_COMMON machine_check
- addi r3,r1,STACK_FRAME_OVERHEAD
+ addi r3,r1,STACK_INT_FRAME_REGS
bl machine_check_exception_async
b interrupt_return_srr
@@ -1364,14 +1377,14 @@
* This is the NMI version of the handler because we are called from
* the early handler which is a true NMI.
*/
- addi r3,r1,STACK_FRAME_OVERHEAD
+ addi r3,r1,STACK_INT_FRAME_REGS
bl machine_check_exception
/*
* We will not reach here. Even if we did, there is no way out.
* Call unrecoverable_exception and die.
*/
- addi r3,r1,STACK_FRAME_OVERHEAD
+ addi r3,r1,STACK_INT_FRAME_REGS
bl unrecoverable_exception
b .
@@ -1422,7 +1435,7 @@
EXC_COMMON_BEGIN(data_access_common)
GEN_COMMON data_access
ld r4,_DSISR(r1)
- addi r3,r1,STACK_FRAME_OVERHEAD
+ addi r3,r1,STACK_INT_FRAME_REGS
andis. r0,r4,DSISR_DABRMATCH@h
bne- 1f
#ifdef CONFIG_PPC_64S_HASH_MMU
@@ -1441,7 +1454,7 @@
* do_break() may have changed the NV GPRS while handling a breakpoint.
* If so, we need to restore them with their updated values.
*/
- REST_NVGPRS(r1)
+ HANDLER_RESTORE_NVGPRS()
b interrupt_return_srr
@@ -1479,7 +1492,7 @@
#ifdef CONFIG_PPC_64S_HASH_MMU
BEGIN_MMU_FTR_SECTION
/* HPT case, do SLB fault */
- addi r3,r1,STACK_FRAME_OVERHEAD
+ addi r3,r1,STACK_INT_FRAME_REGS
bl do_slb_fault
cmpdi r3,0
bne- 1f
@@ -1493,7 +1506,7 @@
li r3,-EFAULT
#endif
std r3,RESULT(r1)
- addi r3,r1,STACK_FRAME_OVERHEAD
+ addi r3,r1,STACK_INT_FRAME_REGS
bl do_bad_segment_interrupt
b interrupt_return_srr
@@ -1525,7 +1538,7 @@
EXC_VIRT_END(instruction_access, 0x4400, 0x80)
EXC_COMMON_BEGIN(instruction_access_common)
GEN_COMMON instruction_access
- addi r3,r1,STACK_FRAME_OVERHEAD
+ addi r3,r1,STACK_INT_FRAME_REGS
#ifdef CONFIG_PPC_64S_HASH_MMU
BEGIN_MMU_FTR_SECTION
bl do_hash_fault
@@ -1567,7 +1580,7 @@
#ifdef CONFIG_PPC_64S_HASH_MMU
BEGIN_MMU_FTR_SECTION
/* HPT case, do SLB fault */
- addi r3,r1,STACK_FRAME_OVERHEAD
+ addi r3,r1,STACK_INT_FRAME_REGS
bl do_slb_fault
cmpdi r3,0
bne- 1f
@@ -1581,7 +1594,7 @@
li r3,-EFAULT
#endif
std r3,RESULT(r1)
- addi r3,r1,STACK_FRAME_OVERHEAD
+ addi r3,r1,STACK_INT_FRAME_REGS
bl do_bad_segment_interrupt
b interrupt_return_srr
@@ -1635,7 +1648,7 @@
EXC_VIRT_END(hardware_interrupt, 0x4500, 0x100)
EXC_COMMON_BEGIN(hardware_interrupt_common)
GEN_COMMON hardware_interrupt
- addi r3,r1,STACK_FRAME_OVERHEAD
+ addi r3,r1,STACK_INT_FRAME_REGS
bl do_IRQ
BEGIN_FTR_SECTION
b interrupt_return_hsrr
@@ -1665,9 +1678,9 @@
EXC_VIRT_END(alignment, 0x4600, 0x100)
EXC_COMMON_BEGIN(alignment_common)
GEN_COMMON alignment
- addi r3,r1,STACK_FRAME_OVERHEAD
+ addi r3,r1,STACK_INT_FRAME_REGS
bl alignment_exception
- REST_NVGPRS(r1) /* instruction emulation may change GPRs */
+ HANDLER_RESTORE_NVGPRS() /* instruction emulation may change GPRs */
b interrupt_return_srr
@@ -1731,9 +1744,9 @@
__GEN_COMMON_BODY program_check
.Ldo_program_check:
- addi r3,r1,STACK_FRAME_OVERHEAD
+ addi r3,r1,STACK_INT_FRAME_REGS
bl program_check_exception
- REST_NVGPRS(r1) /* instruction emulation may change GPRs */
+ HANDLER_RESTORE_NVGPRS() /* instruction emulation may change GPRs */
b interrupt_return_srr
@@ -1751,6 +1764,7 @@
#ifdef CONFIG_KVM_BOOK3S_PR_POSSIBLE
IKVM_REAL=1
#endif
+ IMSR_R12=1
INT_DEFINE_END(fp_unavailable)
EXC_REAL_BEGIN(fp_unavailable, 0x800, 0x100)
@@ -1762,7 +1776,7 @@
EXC_COMMON_BEGIN(fp_unavailable_common)
GEN_COMMON fp_unavailable
bne 1f /* if from user, just load it up */
- addi r3,r1,STACK_FRAME_OVERHEAD
+ addi r3,r1,STACK_INT_FRAME_REGS
bl kernel_fp_unavailable_exception
0: trap
EMIT_BUG_ENTRY 0b, __FILE__, __LINE__, 0
@@ -1780,7 +1794,7 @@
b fast_interrupt_return_srr
#ifdef CONFIG_PPC_TRANSACTIONAL_MEM
2: /* User process was in a transaction */
- addi r3,r1,STACK_FRAME_OVERHEAD
+ addi r3,r1,STACK_INT_FRAME_REGS
bl fp_unavailable_tm
b interrupt_return_srr
#endif
@@ -1824,7 +1838,7 @@
EXC_VIRT_END(decrementer, 0x4900, 0x80)
EXC_COMMON_BEGIN(decrementer_common)
GEN_COMMON decrementer
- addi r3,r1,STACK_FRAME_OVERHEAD
+ addi r3,r1,STACK_INT_FRAME_REGS
bl timer_interrupt
b interrupt_return_srr
@@ -1909,7 +1923,7 @@
EXC_VIRT_END(doorbell_super, 0x4a00, 0x100)
EXC_COMMON_BEGIN(doorbell_super_common)
GEN_COMMON doorbell_super
- addi r3,r1,STACK_FRAME_OVERHEAD
+ addi r3,r1,STACK_INT_FRAME_REGS
#ifdef CONFIG_PPC_DOORBELL
bl doorbell_exception
#else
@@ -2076,7 +2090,7 @@
EXC_VIRT_END(single_step, 0x4d00, 0x100)
EXC_COMMON_BEGIN(single_step_common)
GEN_COMMON single_step
- addi r3,r1,STACK_FRAME_OVERHEAD
+ addi r3,r1,STACK_INT_FRAME_REGS
bl single_step_exception
b interrupt_return_srr
@@ -2110,7 +2124,7 @@
EXC_VIRT_END(h_data_storage, 0x4e00, 0x20)
EXC_COMMON_BEGIN(h_data_storage_common)
GEN_COMMON h_data_storage
- addi r3,r1,STACK_FRAME_OVERHEAD
+ addi r3,r1,STACK_INT_FRAME_REGS
BEGIN_MMU_FTR_SECTION
bl do_bad_page_fault_segv
MMU_FTR_SECTION_ELSE
@@ -2139,7 +2153,7 @@
EXC_VIRT_END(h_instr_storage, 0x4e20, 0x20)
EXC_COMMON_BEGIN(h_instr_storage_common)
GEN_COMMON h_instr_storage
- addi r3,r1,STACK_FRAME_OVERHEAD
+ addi r3,r1,STACK_INT_FRAME_REGS
bl unknown_exception
b interrupt_return_hsrr
@@ -2162,9 +2176,9 @@
EXC_VIRT_END(emulation_assist, 0x4e40, 0x20)
EXC_COMMON_BEGIN(emulation_assist_common)
GEN_COMMON emulation_assist
- addi r3,r1,STACK_FRAME_OVERHEAD
+ addi r3,r1,STACK_INT_FRAME_REGS
bl emulation_assist_interrupt
- REST_NVGPRS(r1) /* instruction emulation may change GPRs */
+ HANDLER_RESTORE_NVGPRS() /* instruction emulation may change GPRs */
b interrupt_return_hsrr
@@ -2222,7 +2236,7 @@
__GEN_COMMON_BODY hmi_exception_early
- addi r3,r1,STACK_FRAME_OVERHEAD
+ addi r3,r1,STACK_INT_FRAME_REGS
bl hmi_exception_realmode
cmpdi cr0,r3,0
bne 1f
@@ -2240,7 +2254,7 @@
EXC_COMMON_BEGIN(hmi_exception_common)
GEN_COMMON hmi_exception
- addi r3,r1,STACK_FRAME_OVERHEAD
+ addi r3,r1,STACK_INT_FRAME_REGS
bl handle_hmi_exception
b interrupt_return_hsrr
@@ -2274,7 +2288,7 @@
EXC_VIRT_END(h_doorbell, 0x4e80, 0x20)
EXC_COMMON_BEGIN(h_doorbell_common)
GEN_COMMON h_doorbell
- addi r3,r1,STACK_FRAME_OVERHEAD
+ addi r3,r1,STACK_INT_FRAME_REGS
#ifdef CONFIG_PPC_DOORBELL
bl doorbell_exception
#else
@@ -2310,7 +2324,7 @@
EXC_VIRT_END(h_virt_irq, 0x4ea0, 0x20)
EXC_COMMON_BEGIN(h_virt_irq_common)
GEN_COMMON h_virt_irq
- addi r3,r1,STACK_FRAME_OVERHEAD
+ addi r3,r1,STACK_INT_FRAME_REGS
bl do_IRQ
b interrupt_return_hsrr
@@ -2356,7 +2370,7 @@
EXC_VIRT_END(performance_monitor, 0x4f00, 0x20)
EXC_COMMON_BEGIN(performance_monitor_common)
GEN_COMMON performance_monitor
- addi r3,r1,STACK_FRAME_OVERHEAD
+ addi r3,r1,STACK_INT_FRAME_REGS
lbz r4,PACAIRQSOFTMASK(r13)
cmpdi r4,IRQS_ENABLED
bne 1f
@@ -2384,6 +2398,7 @@
#ifdef CONFIG_KVM_BOOK3S_PR_POSSIBLE
IKVM_REAL=1
#endif
+ IMSR_R12=1
INT_DEFINE_END(altivec_unavailable)
EXC_REAL_BEGIN(altivec_unavailable, 0xf20, 0x20)
@@ -2410,14 +2425,14 @@
b fast_interrupt_return_srr
#ifdef CONFIG_PPC_TRANSACTIONAL_MEM
2: /* User process was in a transaction */
- addi r3,r1,STACK_FRAME_OVERHEAD
+ addi r3,r1,STACK_INT_FRAME_REGS
bl altivec_unavailable_tm
b interrupt_return_srr
#endif
1:
END_FTR_SECTION_IFSET(CPU_FTR_ALTIVEC)
#endif
- addi r3,r1,STACK_FRAME_OVERHEAD
+ addi r3,r1,STACK_INT_FRAME_REGS
bl altivec_unavailable_exception
b interrupt_return_srr
@@ -2433,6 +2448,7 @@
#ifdef CONFIG_KVM_BOOK3S_PR_POSSIBLE
IKVM_REAL=1
#endif
+ IMSR_R12=1
INT_DEFINE_END(vsx_unavailable)
EXC_REAL_BEGIN(vsx_unavailable, 0xf40, 0x20)
@@ -2458,14 +2474,14 @@
b load_up_vsx
#ifdef CONFIG_PPC_TRANSACTIONAL_MEM
2: /* User process was in a transaction */
- addi r3,r1,STACK_FRAME_OVERHEAD
+ addi r3,r1,STACK_INT_FRAME_REGS
bl vsx_unavailable_tm
b interrupt_return_srr
#endif
1:
END_FTR_SECTION_IFSET(CPU_FTR_VSX)
#endif
- addi r3,r1,STACK_FRAME_OVERHEAD
+ addi r3,r1,STACK_INT_FRAME_REGS
bl vsx_unavailable_exception
b interrupt_return_srr
@@ -2492,9 +2508,9 @@
EXC_VIRT_END(facility_unavailable, 0x4f60, 0x20)
EXC_COMMON_BEGIN(facility_unavailable_common)
GEN_COMMON facility_unavailable
- addi r3,r1,STACK_FRAME_OVERHEAD
+ addi r3,r1,STACK_INT_FRAME_REGS
bl facility_unavailable_exception
- REST_NVGPRS(r1) /* instruction emulation may change GPRs */
+ HANDLER_RESTORE_NVGPRS() /* instruction emulation may change GPRs */
b interrupt_return_srr
@@ -2520,9 +2536,10 @@
EXC_VIRT_END(h_facility_unavailable, 0x4f80, 0x20)
EXC_COMMON_BEGIN(h_facility_unavailable_common)
GEN_COMMON h_facility_unavailable
- addi r3,r1,STACK_FRAME_OVERHEAD
+ addi r3,r1,STACK_INT_FRAME_REGS
bl facility_unavailable_exception
- REST_NVGPRS(r1) /* XXX Shouldn't be necessary in practice */
+ /* XXX Shouldn't be necessary in practice */
+ HANDLER_RESTORE_NVGPRS()
b interrupt_return_hsrr
@@ -2550,7 +2567,7 @@
EXC_VIRT_NONE(0x5200, 0x100)
EXC_COMMON_BEGIN(cbe_system_error_common)
GEN_COMMON cbe_system_error
- addi r3,r1,STACK_FRAME_OVERHEAD
+ addi r3,r1,STACK_INT_FRAME_REGS
bl cbe_system_error_exception
b interrupt_return_hsrr
@@ -2581,7 +2598,7 @@
EXC_VIRT_END(instruction_breakpoint, 0x5300, 0x100)
EXC_COMMON_BEGIN(instruction_breakpoint_common)
GEN_COMMON instruction_breakpoint
- addi r3,r1,STACK_FRAME_OVERHEAD
+ addi r3,r1,STACK_INT_FRAME_REGS
bl instruction_breakpoint_exception
b interrupt_return_srr
@@ -2703,7 +2720,7 @@
EXC_COMMON_BEGIN(denorm_exception_common)
GEN_COMMON denorm_exception
- addi r3,r1,STACK_FRAME_OVERHEAD
+ addi r3,r1,STACK_INT_FRAME_REGS
bl unknown_exception
b interrupt_return_hsrr
@@ -2720,7 +2737,7 @@
EXC_VIRT_NONE(0x5600, 0x100)
EXC_COMMON_BEGIN(cbe_maintenance_common)
GEN_COMMON cbe_maintenance
- addi r3,r1,STACK_FRAME_OVERHEAD
+ addi r3,r1,STACK_INT_FRAME_REGS
bl cbe_maintenance_exception
b interrupt_return_hsrr
@@ -2745,10 +2762,10 @@
EXC_VIRT_END(altivec_assist, 0x5700, 0x100)
EXC_COMMON_BEGIN(altivec_assist_common)
GEN_COMMON altivec_assist
- addi r3,r1,STACK_FRAME_OVERHEAD
+ addi r3,r1,STACK_INT_FRAME_REGS
#ifdef CONFIG_ALTIVEC
bl altivec_assist_exception
- REST_NVGPRS(r1) /* instruction emulation may change GPRs */
+ HANDLER_RESTORE_NVGPRS() /* instruction emulation may change GPRs */
#else
bl unknown_exception
#endif
@@ -2767,7 +2784,7 @@
EXC_VIRT_NONE(0x5800, 0x100)
EXC_COMMON_BEGIN(cbe_thermal_common)
GEN_COMMON cbe_thermal
- addi r3,r1,STACK_FRAME_OVERHEAD
+ addi r3,r1,STACK_INT_FRAME_REGS
bl cbe_thermal_exception
b interrupt_return_hsrr
@@ -2800,7 +2817,7 @@
subi r1,r1,INT_FRAME_SIZE
__GEN_COMMON_BODY soft_nmi
- addi r3,r1,STACK_FRAME_OVERHEAD
+ addi r3,r1,STACK_INT_FRAME_REGS
bl soft_nmi_interrupt
/* Clear MSR_RI before setting SRR0 and SRR1. */
@@ -3124,7 +3141,7 @@
blr
/* MSR[RI] should be clear because this uses SRR[01] */
-disable_machine_check:
+SYM_FUNC_START_LOCAL(disable_machine_check)
mflr r0
bcl 20,31,$+4
0: mflr r3
@@ -3137,3 +3154,4 @@
RFI_TO_KERNEL
1: mtlr r0
blr
+SYM_FUNC_END(disable_machine_check)
diff --git a/arch/powerpc/kernel/head_32.h b/arch/powerpc/kernel/head_32.h
index c328626..f8e2911 100644
--- a/arch/powerpc/kernel/head_32.h
+++ b/arch/powerpc/kernel/head_32.h
@@ -112,7 +112,7 @@ _ASM_NOKPROBE_SYMBOL(\name\()_virt)
stw r0,GPR0(r1)
lis r10,STACK_FRAME_REGS_MARKER@ha /* exception frame marker */
addi r10,r10,STACK_FRAME_REGS_MARKER@l
- stw r10,8(r1)
+ stw r10,STACK_INT_FRAME_MARKER(r1)
li r10, \trapno
stw r10,_TRAP(r1)
SAVE_GPRS(3, 8, r1)
@@ -127,7 +127,7 @@ _ASM_NOKPROBE_SYMBOL(\name\()_virt)
mfspr r10,SPRN_XER
addi r2, r2, -THREAD
stw r10,_XER(r1)
- addi r3,r1,STACK_FRAME_OVERHEAD
+ addi r3,r1,STACK_INT_FRAME_REGS
.endm
.macro prepare_transfer_to_handler
diff --git a/arch/powerpc/kernel/head_40x.S b/arch/powerpc/kernel/head_40x.S
index 088f500..3f68a16 100644
--- a/arch/powerpc/kernel/head_40x.S
+++ b/arch/powerpc/kernel/head_40x.S
@@ -28,6 +28,8 @@
#include <linux/init.h>
#include <linux/pgtable.h>
#include <linux/sizes.h>
+#include <linux/linkage.h>
+
#include <asm/processor.h>
#include <asm/page.h>
#include <asm/mmu.h>
@@ -602,7 +604,7 @@
lis r1,init_thread_union@ha
addi r1,r1,init_thread_union@l
li r0,0
- stwu r0,THREAD_SIZE-STACK_FRAME_OVERHEAD(r1)
+ stwu r0,THREAD_SIZE-STACK_FRAME_MIN_SIZE(r1)
bl early_init /* We have to do this with MMU on */
@@ -662,7 +664,7 @@
* kernel initialization. This maps the first 32 MBytes of memory 1:1
* virtual to physical and more importantly sets the cache mode.
*/
-initial_mmu:
+SYM_FUNC_START_LOCAL(initial_mmu)
tlbia /* Invalidate all TLB entries */
isync
@@ -711,6 +713,7 @@
mtspr SPRN_EVPR,r0
blr
+SYM_FUNC_END(initial_mmu)
_GLOBAL(abort)
mfspr r13,SPRN_DBCR0
diff --git a/arch/powerpc/kernel/head_44x.S b/arch/powerpc/kernel/head_44x.S
index f15cb9f..63a85c1 100644
--- a/arch/powerpc/kernel/head_44x.S
+++ b/arch/powerpc/kernel/head_44x.S
@@ -109,7 +109,7 @@
lis r1,init_thread_union@h
ori r1,r1,init_thread_union@l
li r0,0
- stwu r0,THREAD_SIZE-STACK_FRAME_OVERHEAD(r1)
+ stwu r0,THREAD_SIZE-STACK_FRAME_MIN_SIZE(r1)
bl early_init
@@ -1012,7 +1012,7 @@
*/
lis r1,temp_boot_stack@h
ori r1,r1,temp_boot_stack@l
- addi r1,r1,1024-STACK_FRAME_OVERHEAD
+ addi r1,r1,1024-STACK_FRAME_MIN_SIZE
li r0,0
stw r0,0(r1)
bl mmu_init_secondary
@@ -1025,7 +1025,7 @@
lwz r1,TASK_STACK(r2)
/* Current stack pointer */
- addi r1,r1,THREAD_SIZE-STACK_FRAME_OVERHEAD
+ addi r1,r1,THREAD_SIZE-STACK_FRAME_MIN_SIZE
li r0,0
stw r0,0(r1)
diff --git a/arch/powerpc/kernel/head_64.S b/arch/powerpc/kernel/head_64.S
index dedcc6f..7558ba4 100644
--- a/arch/powerpc/kernel/head_64.S
+++ b/arch/powerpc/kernel/head_64.S
@@ -18,6 +18,7 @@
* variants.
*/
+#include <linux/linkage.h>
#include <linux/threads.h>
#include <linux/init.h>
#include <asm/reg.h>
@@ -424,7 +425,7 @@
/* Create a temp kernel stack for use before relocation is on. */
ld r1,PACAEMERGSP(r13)
- subi r1,r1,STACK_FRAME_OVERHEAD
+ subi r1,r1,STACK_FRAME_MIN_SIZE
/* See if we need to call a cpu state restore handler */
LOAD_REG_ADDR(r23, cur_cpu_spec)
@@ -462,7 +463,7 @@
* Assumes we're mapped EA == RA if the MMU is on.
*/
#ifdef CONFIG_PPC_BOOK3S
-__mmu_off:
+SYM_FUNC_START_LOCAL(__mmu_off)
mfmsr r3
andi. r0,r3,MSR_IR|MSR_DR
beqlr
@@ -473,6 +474,7 @@
sync
rfid
b . /* prevent speculative execution */
+SYM_FUNC_END(__mmu_off)
#endif
@@ -780,7 +782,7 @@
/* Create a temp kernel stack for use before relocation is on. */
ld r1,PACAEMERGSP(r13)
- subi r1,r1,STACK_FRAME_OVERHEAD
+ subi r1,r1,STACK_FRAME_MIN_SIZE
b __secondary_start
@@ -869,7 +871,7 @@
/*
* This subroutine clobbers r11 and r12
*/
-enable_64b_mode:
+SYM_FUNC_START_LOCAL(enable_64b_mode)
mfmsr r11 /* grab the current MSR */
#ifdef CONFIG_PPC_BOOK3E_64
oris r11,r11,0x8000 /* CM bit set, we'll set ICM later */
@@ -881,6 +883,7 @@
isync
#endif
blr
+SYM_FUNC_END(enable_64b_mode)
/*
* This puts the TOC pointer into r2, offset by 0x8000 (as expected
@@ -958,7 +961,7 @@
LOAD_REG_IMMEDIATE(r1,THREAD_SIZE)
add r1,r3,r1
li r0,0
- stdu r0,-STACK_FRAME_OVERHEAD(r1)
+ stdu r0,-STACK_FRAME_MIN_SIZE(r1)
/*
* Do very early kernel initializations, including initial hash table
diff --git a/arch/powerpc/kernel/head_85xx.S b/arch/powerpc/kernel/head_85xx.S
index 52c0ab4..d438ca7 100644
--- a/arch/powerpc/kernel/head_85xx.S
+++ b/arch/powerpc/kernel/head_85xx.S
@@ -29,6 +29,8 @@
#include <linux/init.h>
#include <linux/threads.h>
#include <linux/pgtable.h>
+#include <linux/linkage.h>
+
#include <asm/processor.h>
#include <asm/page.h>
#include <asm/mmu.h>
@@ -229,7 +231,7 @@
lis r1,init_thread_union@h
ori r1,r1,init_thread_union@l
li r0,0
- stwu r0,THREAD_SIZE-STACK_FRAME_OVERHEAD(r1)
+ stwu r0,THREAD_SIZE-STACK_FRAME_MIN_SIZE(r1)
#ifdef CONFIG_SMP
stw r24, TASK_CPU(r2)
@@ -885,7 +887,7 @@
* Translate the effec addr in r3 to phys addr. The phys addr will be put
* into r3(higher 32bit) and r4(lower 32bit)
*/
-get_phys_addr:
+SYM_FUNC_START_LOCAL(get_phys_addr)
mfmsr r8
mfspr r9,SPRN_PID
rlwinm r9,r9,16,0x3fff0000 /* turn PID into MAS6[SPID] */
@@ -907,6 +909,7 @@
mfspr r3,SPRN_MAS7
#endif
blr
+SYM_FUNC_END(get_phys_addr)
/*
* Global functions
@@ -972,10 +975,10 @@
li r4,THREAD_ACC
evstddx evr6, r4, r3 /* save off accumulator */
beq 1f
- lwz r4,_MSR-STACK_FRAME_OVERHEAD(r5)
+ lwz r4,_MSR-STACK_INT_FRAME_REGS(r5)
lis r3,MSR_SPE@h
andc r4,r4,r3 /* disable SPE for previous task */
- stw r4,_MSR-STACK_FRAME_OVERHEAD(r5)
+ stw r4,_MSR-STACK_INT_FRAME_REGS(r5)
1:
blr
#endif /* CONFIG_SPE */
@@ -1044,7 +1047,7 @@
lwz r1,TASK_STACK(r2)
/* stack */
- addi r1,r1,THREAD_SIZE-STACK_FRAME_OVERHEAD
+ addi r1,r1,THREAD_SIZE-STACK_FRAME_MIN_SIZE
li r0,0
stw r0,0(r1)
diff --git a/arch/powerpc/kernel/head_8xx.S b/arch/powerpc/kernel/head_8xx.S
index 0b05f2b..a79751e 100644
--- a/arch/powerpc/kernel/head_8xx.S
+++ b/arch/powerpc/kernel/head_8xx.S
@@ -18,6 +18,8 @@
#include <linux/magic.h>
#include <linux/pgtable.h>
#include <linux/sizes.h>
+#include <linux/linkage.h>
+
#include <asm/processor.h>
#include <asm/page.h>
#include <asm/mmu.h>
@@ -537,7 +539,7 @@
ori r0, r0, STACK_END_MAGIC@l
stw r0, 0(r1)
li r0,0
- stwu r0,THREAD_SIZE-STACK_FRAME_OVERHEAD(r1)
+ stwu r0,THREAD_SIZE-STACK_FRAME_MIN_SIZE(r1)
lis r6, swapper_pg_dir@ha
tophys(r6,r6)
@@ -625,7 +627,7 @@
* 24 Mbytes of data, and the 512k IMMR space. Anything not covered by
* these mappings is mapped by page tables.
*/
-initial_mmu:
+SYM_FUNC_START_LOCAL(initial_mmu)
li r8, 0
mtspr SPRN_MI_CTR, r8 /* remove PINNED ITLB entries */
lis r10, MD_TWAM@h
@@ -686,6 +688,7 @@
#endif
mtspr SPRN_DER, r8
blr
+SYM_FUNC_END(initial_mmu)
_GLOBAL(mmu_pin_tlb)
lis r9, (1f - PAGE_OFFSET)@h
diff --git a/arch/powerpc/kernel/head_book3s_32.S b/arch/powerpc/kernel/head_book3s_32.S
index 519b606..c51f28b 100644
--- a/arch/powerpc/kernel/head_book3s_32.S
+++ b/arch/powerpc/kernel/head_book3s_32.S
@@ -18,6 +18,8 @@
#include <linux/init.h>
#include <linux/pgtable.h>
+#include <linux/linkage.h>
+
#include <asm/reg.h>
#include <asm/page.h>
#include <asm/mmu.h>
@@ -840,7 +842,7 @@
lwz r1,TASK_STACK(r1)
/* stack */
- addi r1,r1,THREAD_SIZE-STACK_FRAME_OVERHEAD
+ addi r1,r1,THREAD_SIZE-STACK_FRAME_MIN_SIZE
li r0,0
tophys(r3,r1)
stw r0,0(r3)
@@ -877,7 +879,7 @@
* Load stuff into the MMU. Intended to be called with
* IR=0 and DR=0.
*/
-early_hash_table:
+SYM_FUNC_START_LOCAL(early_hash_table)
sync /* Force all PTE updates to finish */
isync
tlbia /* Clear all TLB entries */
@@ -888,8 +890,9 @@
ori r6, r6, 3 /* 256kB table */
mtspr SPRN_SDR1, r6
blr
+SYM_FUNC_END(early_hash_table)
-load_up_mmu:
+SYM_FUNC_START_LOCAL(load_up_mmu)
sync /* Force all PTE updates to finish */
isync
tlbia /* Clear all TLB entries */
@@ -918,6 +921,7 @@
LOAD_BAT(7,r3,r4,r5)
END_MMU_FTR_SECTION_IFSET(MMU_FTR_USE_HIGH_BATS)
blr
+SYM_FUNC_END(load_up_mmu)
_GLOBAL(load_segment_registers)
li r0, NUM_USER_SEGMENTS /* load up user segment register values */
@@ -966,7 +970,7 @@
lis r1,init_thread_union@ha
addi r1,r1,init_thread_union@l
li r0,0
- stwu r0,THREAD_SIZE-STACK_FRAME_OVERHEAD(r1)
+ stwu r0,THREAD_SIZE-STACK_FRAME_MIN_SIZE(r1)
/*
* Do early platform-specific initialization,
* and set up the MMU.
@@ -1028,7 +1032,7 @@
* this makes sure it's done.
* -- Cort
*/
-clear_bats:
+SYM_FUNC_START_LOCAL(clear_bats)
li r10,0
mtspr SPRN_DBAT0U,r10
@@ -1072,6 +1076,7 @@
mtspr SPRN_IBAT7L,r10
END_MMU_FTR_SECTION_IFSET(MMU_FTR_USE_HIGH_BATS)
blr
+SYM_FUNC_END(clear_bats)
_GLOBAL(update_bats)
lis r4, 1f@h
@@ -1108,15 +1113,16 @@
mtspr SPRN_SRR1, r6
rfi
-flush_tlbs:
+SYM_FUNC_START_LOCAL(flush_tlbs)
lis r10, 0x40
1: addic. r10, r10, -0x1000
tlbie r10
bgt 1b
sync
blr
+SYM_FUNC_END(flush_tlbs)
-mmu_off:
+SYM_FUNC_START_LOCAL(mmu_off)
addi r4, r3, __after_mmu_off - _start
mfmsr r3
andi. r0,r3,MSR_DR|MSR_IR /* MMU enabled? */
@@ -1128,9 +1134,10 @@
mtspr SPRN_SRR1,r3
sync
rfi
+SYM_FUNC_END(mmu_off)
/* We use one BAT to map up to 256M of RAM at _PAGE_OFFSET */
-initial_bats:
+SYM_FUNC_START_LOCAL(initial_bats)
lis r11,PAGE_OFFSET@h
tophys(r8,r11)
#ifdef CONFIG_SMP
@@ -1146,9 +1153,10 @@
mtspr SPRN_IBAT0U,r11
isync
blr
+SYM_FUNC_END(initial_bats)
#ifdef CONFIG_BOOTX_TEXT
-setup_disp_bat:
+SYM_FUNC_START_LOCAL(setup_disp_bat)
/*
* setup the display bat prepared for us in prom.c
*/
@@ -1164,10 +1172,11 @@
mtspr SPRN_DBAT3L,r8
mtspr SPRN_DBAT3U,r11
blr
+SYM_FUNC_END(setup_disp_bat)
#endif /* CONFIG_BOOTX_TEXT */
#ifdef CONFIG_PPC_EARLY_DEBUG_CPM
-setup_cpm_bat:
+SYM_FUNC_START_LOCAL(setup_cpm_bat)
lis r8, 0xf000
ori r8, r8, 0x002a
mtspr SPRN_DBAT1L, r8
@@ -1177,10 +1186,11 @@
mtspr SPRN_DBAT1U, r11
blr
+SYM_FUNC_END(setup_cpm_bat)
#endif
#ifdef CONFIG_PPC_EARLY_DEBUG_USBGECKO
-setup_usbgecko_bat:
+SYM_FUNC_START_LOCAL(setup_usbgecko_bat)
/* prepare a BAT for early io */
#if defined(CONFIG_GAMECUBE)
lis r8, 0x0c00
@@ -1199,6 +1209,7 @@
mtspr SPRN_DBAT1L, r8
mtspr SPRN_DBAT1U, r11
blr
+SYM_FUNC_END(setup_usbgecko_bat)
#endif
.data
diff --git a/arch/powerpc/kernel/head_booke.h b/arch/powerpc/kernel/head_booke.h
index 1cb9d0f..37d43c1 100644
--- a/arch/powerpc/kernel/head_booke.h
+++ b/arch/powerpc/kernel/head_booke.h
@@ -84,7 +84,7 @@ END_BTB_FLUSH_SECTION
stw r0,GPR0(r1)
lis r10, STACK_FRAME_REGS_MARKER@ha /* exception frame marker */
addi r10, r10, STACK_FRAME_REGS_MARKER@l
- stw r10, 8(r1)
+ stw r10, STACK_INT_FRAME_MARKER(r1)
li r10, \trapno
stw r10,_TRAP(r1)
SAVE_GPRS(3, 8, r1)
@@ -99,7 +99,7 @@ END_BTB_FLUSH_SECTION
mfspr r10,SPRN_XER
addi r2, r2, -THREAD
stw r10,_XER(r1)
- addi r3,r1,STACK_FRAME_OVERHEAD
+ addi r3,r1,STACK_INT_FRAME_REGS
.endm
.macro prepare_transfer_to_handler
diff --git a/arch/powerpc/kernel/hw_breakpoint.c b/arch/powerpc/kernel/hw_breakpoint.c
index 8db1a15..e1b4e70 100644
--- a/arch/powerpc/kernel/hw_breakpoint.c
+++ b/arch/powerpc/kernel/hw_breakpoint.c
@@ -646,7 +646,7 @@ int hw_breakpoint_handler(struct die_args *args)
ppc_inst_t instr = ppc_inst(0);
int type = 0;
int size = 0;
- unsigned long ea;
+ unsigned long ea = 0;
/* Disable breakpoints during exception handling */
hw_breakpoint_disable();
diff --git a/arch/powerpc/kernel/interrupt_64.S b/arch/powerpc/kernel/interrupt_64.S
index a019ed6..fccc344 100644
--- a/arch/powerpc/kernel/interrupt_64.S
+++ b/arch/powerpc/kernel/interrupt_64.S
@@ -77,11 +77,11 @@
std r11,_TRAP(r1)
std r12,_CCR(r1)
std r3,ORIG_GPR3(r1)
- /* Calling convention has r3 = regs, r4 = orig r0 */
- addi r3,r1,STACK_FRAME_OVERHEAD
- mr r4,r0
LOAD_REG_IMMEDIATE(r11, STACK_FRAME_REGS_MARKER)
- std r11,-16(r3) /* "regshere" marker */
+ std r11,STACK_INT_FRAME_MARKER(r1) /* "regs" marker */
+ /* Calling convention has r3 = regs, r4 = orig r0 */
+ addi r3,r1,STACK_INT_FRAME_REGS
+ mr r4,r0
BEGIN_FTR_SECTION
HMT_MEDIUM
@@ -96,10 +96,15 @@
* but this is the best we can do.
*/
+ /*
+ * Zero user registers to prevent influencing speculative execution
+ * state of kernel code.
+ */
+ SANITIZE_SYSCALL_GPRS()
bl system_call_exception
.Lsyscall_vectored_\name\()_exit:
- addi r4,r1,STACK_FRAME_OVERHEAD
+ addi r4,r1,STACK_INT_FRAME_REGS
li r5,1 /* scv */
bl syscall_exit_prepare
std r1,PACA_EXIT_SAVE_R1(r13) /* save r1 for restart */
@@ -124,6 +129,7 @@
HMT_MEDIUM_LOW
END_FTR_SECTION_IFSET(CPU_FTR_HAS_PPR)
+ SANITIZE_RESTORE_NVGPRS()
cmpdi r3,0
bne .Lsyscall_vectored_\name\()_restore_regs
@@ -159,7 +165,7 @@
ld r4,_LINK(r1)
ld r5,_XER(r1)
- REST_NVGPRS(r1)
+ HANDLER_RESTORE_NVGPRS()
REST_GPR(0, r1)
mtcr r2
mtctr r3
@@ -176,7 +182,7 @@
ld r1,PACA_EXIT_SAVE_R1(r13)
LOAD_PACA_TOC()
ld r3,RESULT(r1)
- addi r4,r1,STACK_FRAME_OVERHEAD
+ addi r4,r1,STACK_INT_FRAME_REGS
li r11,IRQS_ALL_DISABLED
stb r11,PACAIRQSOFTMASK(r13)
bl syscall_exit_restart
@@ -250,11 +256,11 @@
std r11,_TRAP(r1)
std r12,_CCR(r1)
std r3,ORIG_GPR3(r1)
- /* Calling convention has r3 = regs, r4 = orig r0 */
- addi r3,r1,STACK_FRAME_OVERHEAD
- mr r4,r0
LOAD_REG_IMMEDIATE(r11, STACK_FRAME_REGS_MARKER)
- std r11,-16(r3) /* "regshere" marker */
+ std r11,STACK_INT_FRAME_MARKER(r1) /* "regs" marker */
+ /* Calling convention has r3 = regs, r4 = orig r0 */
+ addi r3,r1,STACK_INT_FRAME_REGS
+ mr r4,r0
#ifdef CONFIG_PPC_BOOK3S
li r11,1
@@ -275,10 +281,15 @@
wrteei 1
#endif
+ /*
+ * Zero user registers to prevent influencing speculative execution
+ * state of kernel code.
+ */
+ SANITIZE_SYSCALL_GPRS()
bl system_call_exception
.Lsyscall_exit:
- addi r4,r1,STACK_FRAME_OVERHEAD
+ addi r4,r1,STACK_INT_FRAME_REGS
li r5,0 /* !scv */
bl syscall_exit_prepare
std r1,PACA_EXIT_SAVE_R1(r13) /* save r1 for restart */
@@ -315,6 +326,7 @@
stdcx. r0,0,r1 /* to clear the reservation */
END_FTR_SECTION_IFCLR(CPU_FTR_STCX_CHECKS_ADDRESS)
+ SANITIZE_RESTORE_NVGPRS()
cmpdi r3,0
bne .Lsyscall_restore_regs
/* Zero volatile regs that may contain sensitive kernel data */
@@ -342,7 +354,7 @@
.Lsyscall_restore_regs:
ld r3,_CTR(r1)
ld r4,_XER(r1)
- REST_NVGPRS(r1)
+ HANDLER_RESTORE_NVGPRS()
mtctr r3
mtspr SPRN_XER,r4
REST_GPR(0, r1)
@@ -357,7 +369,7 @@
ld r1,PACA_EXIT_SAVE_R1(r13)
LOAD_PACA_TOC()
ld r3,RESULT(r1)
- addi r4,r1,STACK_FRAME_OVERHEAD
+ addi r4,r1,STACK_INT_FRAME_REGS
li r11,IRQS_ALL_DISABLED
stb r11,PACAIRQSOFTMASK(r13)
bl syscall_exit_restart
@@ -388,7 +400,7 @@
andi. r0,r5,MSR_RI
li r3,0 /* 0 return value, no EMULATE_STACK_STORE */
bne+ .Lfast_kernel_interrupt_return_srr
- addi r3,r1,STACK_FRAME_OVERHEAD
+ addi r3,r1,STACK_INT_FRAME_REGS
bl unrecoverable_exception
b . /* should not get here */
#else
@@ -406,11 +418,13 @@
beq interrupt_return_\srr\()_kernel
interrupt_return_\srr\()_user: /* make backtraces match the _kernel variant */
_ASM_NOKPROBE_SYMBOL(interrupt_return_\srr\()_user)
- addi r3,r1,STACK_FRAME_OVERHEAD
+ addi r3,r1,STACK_INT_FRAME_REGS
bl interrupt_exit_user_prepare
+#ifndef CONFIG_INTERRUPT_SANITIZE_REGISTERS
cmpdi r3,0
bne- .Lrestore_nvgprs_\srr
.Lrestore_nvgprs_\srr\()_cont:
+#endif
std r1,PACA_EXIT_SAVE_R1(r13) /* save r1 for restart */
#ifdef CONFIG_PPC_BOOK3S
.Linterrupt_return_\srr\()_user_rst_start:
@@ -424,6 +438,7 @@
stb r11,PACAIRQHAPPENED(r13) # clear out possible HARD_DIS
.Lfast_user_interrupt_return_\srr\():
+ SANITIZE_RESTORE_NVGPRS()
#ifdef CONFIG_PPC_BOOK3S
.ifc \srr,srr
lbz r4,PACASRR_VALID(r13)
@@ -493,9 +508,11 @@
b . /* prevent speculative execution */
.Linterrupt_return_\srr\()_user_rst_end:
+#ifndef CONFIG_INTERRUPT_SANITIZE_REGISTERS
.Lrestore_nvgprs_\srr\():
REST_NVGPRS(r1)
b .Lrestore_nvgprs_\srr\()_cont
+#endif
#ifdef CONFIG_PPC_BOOK3S
interrupt_return_\srr\()_user_restart:
@@ -503,7 +520,7 @@
GET_PACA(r13)
ld r1,PACA_EXIT_SAVE_R1(r13)
LOAD_PACA_TOC()
- addi r3,r1,STACK_FRAME_OVERHEAD
+ addi r3,r1,STACK_INT_FRAME_REGS
li r11,IRQS_ALL_DISABLED
stb r11,PACAIRQSOFTMASK(r13)
bl interrupt_exit_user_restart
@@ -518,7 +535,7 @@
.balign IFETCH_ALIGN_BYTES
interrupt_return_\srr\()_kernel:
_ASM_NOKPROBE_SYMBOL(interrupt_return_\srr\()_kernel)
- addi r3,r1,STACK_FRAME_OVERHEAD
+ addi r3,r1,STACK_INT_FRAME_REGS
bl interrupt_exit_kernel_prepare
std r1,PACA_EXIT_SAVE_R1(r13) /* save r1 for restart */
@@ -585,6 +602,7 @@
stb r11,PACAIRQHAPPENED(r13) // clear the possible HARD_DIS
.Lfast_kernel_interrupt_return_\srr\():
+ SANITIZE_RESTORE_NVGPRS()
cmpdi cr1,r3,0
#ifdef CONFIG_PPC_BOOK3S
.ifc \srr,srr
@@ -637,7 +655,7 @@
* Leaving a stale STACK_FRAME_REGS_MARKER on the stack can confuse
* the reliable stack unwinder later on. Clear it.
*/
- std r0,STACK_FRAME_OVERHEAD-16(r1)
+ std r0,STACK_INT_FRAME_MARKER(r1)
REST_GPRS(2, 5, r1)
@@ -684,7 +702,7 @@
GET_PACA(r13)
ld r1,PACA_EXIT_SAVE_R1(r13)
LOAD_PACA_TOC()
- addi r3,r1,STACK_FRAME_OVERHEAD
+ addi r3,r1,STACK_INT_FRAME_REGS
li r11,IRQS_ALL_DISABLED
stb r11,PACAIRQSOFTMASK(r13)
bl interrupt_exit_kernel_restart
diff --git a/arch/powerpc/kernel/irq.c b/arch/powerpc/kernel/irq.c
index 9ede61a5..c5b9ce8 100644
--- a/arch/powerpc/kernel/irq.c
+++ b/arch/powerpc/kernel/irq.c
@@ -210,7 +210,7 @@ static __always_inline void call_do_softirq(const void *sp)
PPC_LL " %%r1, 0(%%r1) ;"
: // Outputs
: // Inputs
- [sp] "b" (sp), [offset] "i" (THREAD_SIZE - STACK_FRAME_OVERHEAD),
+ [sp] "b" (sp), [offset] "i" (THREAD_SIZE - STACK_FRAME_MIN_SIZE),
[callee] "i" (__do_softirq)
: // Clobbers
"lr", "xer", "ctr", "memory", "cr0", "cr1", "cr5", "cr6",
@@ -264,7 +264,7 @@ static __always_inline void call_do_irq(struct pt_regs *regs, void *sp)
: // Outputs
"+r" (r3)
: // Inputs
- [sp] "b" (sp), [offset] "i" (THREAD_SIZE - STACK_FRAME_OVERHEAD),
+ [sp] "b" (sp), [offset] "i" (THREAD_SIZE - STACK_FRAME_MIN_SIZE),
[callee] "i" (__do_irq)
: // Clobbers
"lr", "xer", "ctr", "memory", "cr0", "cr1", "cr5", "cr6",
diff --git a/arch/powerpc/kernel/kgdb.c b/arch/powerpc/kernel/kgdb.c
index 1a1e999..ebe4d16 100644
--- a/arch/powerpc/kernel/kgdb.c
+++ b/arch/powerpc/kernel/kgdb.c
@@ -191,7 +191,7 @@ static int kgdb_break_match(struct pt_regs *regs)
void sleeping_thread_to_gdb_regs(unsigned long *gdb_regs, struct task_struct *p)
{
struct pt_regs *regs = (struct pt_regs *)(p->thread.ksp +
- STACK_FRAME_OVERHEAD);
+ STACK_INT_FRAME_REGS);
unsigned long *ptr = gdb_regs;
int reg;
diff --git a/arch/powerpc/kernel/kprobes.c b/arch/powerpc/kernel/kprobes.c
index bd7b1a0..b20ee72 100644
--- a/arch/powerpc/kernel/kprobes.c
+++ b/arch/powerpc/kernel/kprobes.c
@@ -20,12 +20,12 @@
#include <linux/kdebug.h>
#include <linux/slab.h>
#include <linux/moduleloader.h>
+#include <linux/set_memory.h>
#include <asm/code-patching.h>
#include <asm/cacheflush.h>
#include <asm/sstep.h>
#include <asm/sections.h>
#include <asm/inst.h>
-#include <asm/set_memory.h>
#include <linux/uaccess.h>
DEFINE_PER_CPU(struct kprobe *, current_kprobe) = NULL;
@@ -134,10 +134,9 @@ void *alloc_insn_page(void)
if (!page)
return NULL;
- if (strict_module_rwx_enabled()) {
- set_memory_ro((unsigned long)page, 1);
- set_memory_x((unsigned long)page, 1);
- }
+ if (strict_module_rwx_enabled())
+ set_memory_rox((unsigned long)page, 1);
+
return page;
}
@@ -158,9 +157,7 @@ int arch_prepare_kprobe(struct kprobe *p)
printk("Cannot register a kprobe on the second word of prefixed instruction\n");
ret = -EINVAL;
}
- preempt_disable();
prev = get_kprobe(p->addr - 1);
- preempt_enable_no_resched();
/*
* When prev is a ftrace-based kprobe, we don't have an insn, and it
@@ -371,7 +368,7 @@ int kprobe_handler(struct pt_regs *regs)
if (ret > 0) {
restore_previous_kprobe(kcb);
- preempt_enable_no_resched();
+ preempt_enable();
return 1;
}
}
@@ -384,7 +381,7 @@ int kprobe_handler(struct pt_regs *regs)
if (p->pre_handler && p->pre_handler(p, regs)) {
/* handler changed execution path, so skip ss setup */
reset_current_kprobe();
- preempt_enable_no_resched();
+ preempt_enable();
return 1;
}
@@ -397,7 +394,7 @@ int kprobe_handler(struct pt_regs *regs)
kcb->kprobe_status = KPROBE_HIT_SSDONE;
reset_current_kprobe();
- preempt_enable_no_resched();
+ preempt_enable();
return 1;
}
}
@@ -406,7 +403,7 @@ int kprobe_handler(struct pt_regs *regs)
return 1;
no_kprobe:
- preempt_enable_no_resched();
+ preempt_enable();
return ret;
}
NOKPROBE_SYMBOL(kprobe_handler);
@@ -492,7 +489,7 @@ int kprobe_post_handler(struct pt_regs *regs)
}
reset_current_kprobe();
out:
- preempt_enable_no_resched();
+ preempt_enable();
/*
* if somebody else is singlestepping across a probe point, msr
@@ -531,7 +528,7 @@ int kprobe_fault_handler(struct pt_regs *regs, int trapnr)
restore_previous_kprobe(kcb);
else
reset_current_kprobe();
- preempt_enable_no_resched();
+ preempt_enable();
break;
case KPROBE_HIT_ACTIVE:
case KPROBE_HIT_SSDONE:
diff --git a/arch/powerpc/kernel/misc_32.S b/arch/powerpc/kernel/misc_32.S
index e5127b1..daf8f87 100644
--- a/arch/powerpc/kernel/misc_32.S
+++ b/arch/powerpc/kernel/misc_32.S
@@ -382,7 +382,7 @@
_GLOBAL(start_secondary_resume)
/* Reset stack */
rlwinm r1, r1, 0, 0, 31 - THREAD_SHIFT
- addi r1,r1,THREAD_SIZE-STACK_FRAME_OVERHEAD
+ addi r1,r1,THREAD_SIZE-STACK_FRAME_MIN_SIZE
li r3,0
stw r3,0(r1) /* Zero the stack frame pointer */
bl start_secondary
diff --git a/arch/powerpc/kernel/misc_64.S b/arch/powerpc/kernel/misc_64.S
index 36184ca..c39c07a 100644
--- a/arch/powerpc/kernel/misc_64.S
+++ b/arch/powerpc/kernel/misc_64.S
@@ -9,6 +9,7 @@
* PPC64 updates by Dave Engebretsen (engebret@us.ibm.com)
*/
+#include <linux/linkage.h>
#include <linux/sys.h>
#include <asm/unistd.h>
#include <asm/errno.h>
@@ -353,7 +354,7 @@
*
* don't overwrite r3 here, it is live for kexec_wait above.
*/
-real_mode: /* assume normal blr return */
+SYM_FUNC_START_LOCAL(real_mode) /* assume normal blr return */
#ifdef CONFIG_PPC_BOOK3E_64
/* Create an identity mapping. */
b kexec_create_tlb
@@ -370,6 +371,7 @@
mtspr SPRN_SRR0,r11
rfid
#endif
+SYM_FUNC_END(real_mode)
/*
* kexec_sequence(newstack, start, image, control, clear_all(),
@@ -384,7 +386,7 @@
std r0,16(r1)
/* switch stacks to newstack -- &kexec_stack.stack */
- stdu r1,THREAD_SIZE-STACK_FRAME_OVERHEAD(r3)
+ stdu r1,THREAD_SIZE-STACK_FRAME_MIN_SIZE(r3)
mr r1,r3
li r0,0
@@ -401,7 +403,7 @@
std r26,-48(r1)
std r25,-56(r1)
- stdu r1,-STACK_FRAME_OVERHEAD-64(r1)
+ stdu r1,-STACK_FRAME_MIN_SIZE-64(r1)
/* save args into preserved regs */
mr r31,r3 /* newstack (both) */
diff --git a/arch/powerpc/kernel/module_64.c b/arch/powerpc/kernel/module_64.c
index 7e45dc9..ff04564 100644
--- a/arch/powerpc/kernel/module_64.c
+++ b/arch/powerpc/kernel/module_64.c
@@ -31,6 +31,16 @@
this, and makes other things simpler. Anton?
--RR. */
+bool module_elf_check_arch(Elf_Ehdr *hdr)
+{
+ unsigned long abi_level = hdr->e_flags & 0x3;
+
+ if (IS_ENABLED(CONFIG_PPC64_ELF_ABI_V2))
+ return abi_level == 2;
+ else
+ return abi_level < 2;
+}
+
#ifdef CONFIG_PPC64_ELF_ABI_V2
static func_desc_t func_desc(unsigned long addr)
diff --git a/arch/powerpc/kernel/optprobes.c b/arch/powerpc/kernel/optprobes.c
index 3b1c223..004fae2 100644
--- a/arch/powerpc/kernel/optprobes.c
+++ b/arch/powerpc/kernel/optprobes.c
@@ -112,7 +112,7 @@ static void optimized_callback(struct optimized_kprobe *op,
__this_cpu_write(current_kprobe, NULL);
}
- preempt_enable_no_resched();
+ preempt_enable();
}
NOKPROBE_SYMBOL(optimized_callback);
diff --git a/arch/powerpc/kernel/optprobes_head.S b/arch/powerpc/kernel/optprobes_head.S
index cd4e7bc..35932f4 100644
--- a/arch/powerpc/kernel/optprobes_head.S
+++ b/arch/powerpc/kernel/optprobes_head.S
@@ -85,7 +85,7 @@
TEMPLATE_FOR_IMM_LOAD_INSNS
/* 2. pt_regs pointer in r4 */
- addi r4,r1,STACK_FRAME_OVERHEAD
+ addi r4,r1,STACK_INT_FRAME_REGS
.global optprobe_template_call_handler
optprobe_template_call_handler:
@@ -96,7 +96,7 @@
* Parameters for instruction emulation:
* 1. Pass SP in register r3.
*/
- addi r3,r1,STACK_FRAME_OVERHEAD
+ addi r3,r1,STACK_INT_FRAME_REGS
.global optprobe_template_insn
optprobe_template_insn:
diff --git a/arch/powerpc/kernel/ppc_save_regs.S b/arch/powerpc/kernel/ppc_save_regs.S
index 2d4d21b..49813f9 100644
--- a/arch/powerpc/kernel/ppc_save_regs.S
+++ b/arch/powerpc/kernel/ppc_save_regs.S
@@ -21,60 +21,33 @@
* different ABIs, though).
*/
_GLOBAL(ppc_save_regs)
- PPC_STL r0,0*SZL(r3)
+ /* This allows stack frame accessor macros and offsets to be used */
+ subi r3,r3,STACK_INT_FRAME_REGS
+ PPC_STL r0,GPR0(r3)
#ifdef CONFIG_PPC32
- stmw r2, 2*SZL(r3)
+ stmw r2,GPR2(r3)
#else
- PPC_STL r2,2*SZL(r3)
- PPC_STL r3,3*SZL(r3)
- PPC_STL r4,4*SZL(r3)
- PPC_STL r5,5*SZL(r3)
- PPC_STL r6,6*SZL(r3)
- PPC_STL r7,7*SZL(r3)
- PPC_STL r8,8*SZL(r3)
- PPC_STL r9,9*SZL(r3)
- PPC_STL r10,10*SZL(r3)
- PPC_STL r11,11*SZL(r3)
- PPC_STL r12,12*SZL(r3)
- PPC_STL r13,13*SZL(r3)
- PPC_STL r14,14*SZL(r3)
- PPC_STL r15,15*SZL(r3)
- PPC_STL r16,16*SZL(r3)
- PPC_STL r17,17*SZL(r3)
- PPC_STL r18,18*SZL(r3)
- PPC_STL r19,19*SZL(r3)
- PPC_STL r20,20*SZL(r3)
- PPC_STL r21,21*SZL(r3)
- PPC_STL r22,22*SZL(r3)
- PPC_STL r23,23*SZL(r3)
- PPC_STL r24,24*SZL(r3)
- PPC_STL r25,25*SZL(r3)
- PPC_STL r26,26*SZL(r3)
- PPC_STL r27,27*SZL(r3)
- PPC_STL r28,28*SZL(r3)
- PPC_STL r29,29*SZL(r3)
- PPC_STL r30,30*SZL(r3)
- PPC_STL r31,31*SZL(r3)
+ SAVE_GPRS(2, 31, r3)
lbz r0,PACAIRQSOFTMASK(r13)
- PPC_STL r0,SOFTE-STACK_FRAME_OVERHEAD(r3)
+ PPC_STL r0,SOFTE(r3)
#endif
/* go up one stack frame for SP */
PPC_LL r4,0(r1)
- PPC_STL r4,1*SZL(r3)
+ PPC_STL r4,GPR1(r3)
/* get caller's LR */
PPC_LL r0,LRSAVE(r4)
- PPC_STL r0,_LINK-STACK_FRAME_OVERHEAD(r3)
+ PPC_STL r0,_LINK(r3)
mflr r0
- PPC_STL r0,_NIP-STACK_FRAME_OVERHEAD(r3)
+ PPC_STL r0,_NIP(r3)
mfmsr r0
- PPC_STL r0,_MSR-STACK_FRAME_OVERHEAD(r3)
+ PPC_STL r0,_MSR(r3)
mfctr r0
- PPC_STL r0,_CTR-STACK_FRAME_OVERHEAD(r3)
+ PPC_STL r0,_CTR(r3)
mfxer r0
- PPC_STL r0,_XER-STACK_FRAME_OVERHEAD(r3)
+ PPC_STL r0,_XER(r3)
mfcr r0
- PPC_STL r0,_CCR-STACK_FRAME_OVERHEAD(r3)
+ PPC_STL r0,_CCR(r3)
li r0,0
- PPC_STL r0,_TRAP-STACK_FRAME_OVERHEAD(r3)
- PPC_STL r0,ORIG_GPR3-STACK_FRAME_OVERHEAD(r3)
+ PPC_STL r0,_TRAP(r3)
+ PPC_STL r0,ORIG_GPR3(r3)
blr
diff --git a/arch/powerpc/kernel/process.c b/arch/powerpc/kernel/process.c
index fcf6043..c22cc23 100644
--- a/arch/powerpc/kernel/process.c
+++ b/arch/powerpc/kernel/process.c
@@ -862,10 +862,8 @@ static inline int set_breakpoint_8xx(struct arch_hw_breakpoint *brk)
return 0;
}
-void __set_breakpoint(int nr, struct arch_hw_breakpoint *brk)
+static void set_hw_breakpoint(int nr, struct arch_hw_breakpoint *brk)
{
- memcpy(this_cpu_ptr(¤t_brk[nr]), brk, sizeof(*brk));
-
if (dawr_enabled())
// Power8 or later
set_dawr(nr, brk);
@@ -879,6 +877,12 @@ void __set_breakpoint(int nr, struct arch_hw_breakpoint *brk)
WARN_ON_ONCE(1);
}
+void __set_breakpoint(int nr, struct arch_hw_breakpoint *brk)
+{
+ memcpy(this_cpu_ptr(¤t_brk[nr]), brk, sizeof(*brk));
+ set_hw_breakpoint(nr, brk);
+}
+
/* Check if we have DAWR or DABR hardware */
bool ppc_breakpoint_available(void)
{
@@ -891,6 +895,34 @@ bool ppc_breakpoint_available(void)
}
EXPORT_SYMBOL_GPL(ppc_breakpoint_available);
+/* Disable the breakpoint in hardware without touching current_brk[] */
+void suspend_breakpoints(void)
+{
+ struct arch_hw_breakpoint brk = {0};
+ int i;
+
+ if (!ppc_breakpoint_available())
+ return;
+
+ for (i = 0; i < nr_wp_slots(); i++)
+ set_hw_breakpoint(i, &brk);
+}
+
+/*
+ * Re-enable breakpoints suspended by suspend_breakpoints() in hardware
+ * from current_brk[]
+ */
+void restore_breakpoints(void)
+{
+ int i;
+
+ if (!ppc_breakpoint_available())
+ return;
+
+ for (i = 0; i < nr_wp_slots(); i++)
+ set_hw_breakpoint(i, this_cpu_ptr(¤t_brk[i]));
+}
+
#ifdef CONFIG_PPC_TRANSACTIONAL_MEM
static inline bool tm_enabled(struct task_struct *tsk)
@@ -1359,7 +1391,7 @@ static void show_instructions(struct pt_regs *regs)
unsigned long nip = regs->nip;
unsigned long pc = regs->nip - (NR_INSN_TO_PRINT * 3 / 4 * sizeof(int));
- printk("Instruction dump:");
+ printk("Code: ");
/*
* If we were executing with the MMU off for instructions, adjust pc
@@ -1373,9 +1405,6 @@ static void show_instructions(struct pt_regs *regs)
for (i = 0; i < NR_INSN_TO_PRINT; i++) {
int instr;
- if (!(i % 8))
- pr_cont("\n");
-
if (!__kernel_text_address(pc) ||
get_kernel_nofault(instr, (const void *)pc)) {
pr_cont("XXXXXXXX ");
@@ -1726,13 +1755,17 @@ int copy_thread(struct task_struct *p, const struct kernel_clone_args *args)
klp_init_thread_info(p);
+ /* Create initial stack frame. */
+ sp -= STACK_USER_INT_FRAME_SIZE;
+ *(unsigned long *)(sp + STACK_INT_FRAME_MARKER) = STACK_FRAME_REGS_MARKER;
+
/* Copy registers */
- sp -= sizeof(struct pt_regs);
- childregs = (struct pt_regs *) sp;
+ childregs = (struct pt_regs *)(sp + STACK_INT_FRAME_REGS);
if (unlikely(args->fn)) {
/* kernel thread */
+ ((unsigned long *)sp)[0] = 0;
memset(childregs, 0, sizeof(struct pt_regs));
- childregs->gpr[1] = sp + sizeof(struct pt_regs);
+ childregs->gpr[1] = sp + STACK_USER_INT_FRAME_SIZE;
/* function */
if (args->fn)
childregs->gpr[14] = ppc_function_entry((void *)args->fn);
@@ -1750,6 +1783,7 @@ int copy_thread(struct task_struct *p, const struct kernel_clone_args *args)
*childregs = *regs;
if (usp)
childregs->gpr[1] = usp;
+ ((unsigned long *)sp)[0] = childregs->gpr[1];
p->thread.regs = childregs;
/* 64s sets this in ret_from_fork */
if (!IS_ENABLED(CONFIG_PPC_BOOK3S_64))
@@ -1767,7 +1801,6 @@ int copy_thread(struct task_struct *p, const struct kernel_clone_args *args)
f = ret_from_fork;
}
childregs->msr &= ~(MSR_FP|MSR_VEC|MSR_VSX);
- sp -= STACK_FRAME_OVERHEAD;
/*
* The way this works is that at some point in the future
@@ -1777,11 +1810,12 @@ int copy_thread(struct task_struct *p, const struct kernel_clone_args *args)
* do some house keeping and then return from the fork or clone
* system call, using the stack frame created above.
*/
- ((unsigned long *)sp)[0] = 0;
- sp -= sizeof(struct pt_regs);
- kregs = (struct pt_regs *) sp;
- sp -= STACK_FRAME_OVERHEAD;
+ ((unsigned long *)sp)[STACK_FRAME_LR_SAVE] = (unsigned long)f;
+ sp -= STACK_SWITCH_FRAME_SIZE;
+ ((unsigned long *)sp)[0] = sp + STACK_SWITCH_FRAME_SIZE;
+ kregs = (struct pt_regs *)(sp + STACK_SWITCH_FRAME_REGS);
p->thread.ksp = sp;
+
#ifdef CONFIG_HAVE_HW_BREAKPOINT
for (i = 0; i < nr_wp_slots(); i++)
p->thread.ptrace_bps[i] = NULL;
@@ -2123,9 +2157,12 @@ static inline int valid_emergency_stack(unsigned long sp, struct task_struct *p,
return 0;
}
-
-int validate_sp(unsigned long sp, struct task_struct *p,
- unsigned long nbytes)
+/*
+ * validate the stack frame of a particular minimum size, used for when we are
+ * looking at a certain object in the stack beyond the minimum.
+ */
+int validate_sp_size(unsigned long sp, struct task_struct *p,
+ unsigned long nbytes)
{
unsigned long stack_page = (unsigned long)task_stack_page(p);
@@ -2141,7 +2178,10 @@ int validate_sp(unsigned long sp, struct task_struct *p,
return valid_emergency_stack(sp, p, nbytes);
}
-EXPORT_SYMBOL(validate_sp);
+int validate_sp(unsigned long sp, struct task_struct *p)
+{
+ return validate_sp_size(sp, p, STACK_FRAME_MIN_SIZE);
+}
static unsigned long ___get_wchan(struct task_struct *p)
{
@@ -2149,13 +2189,12 @@ static unsigned long ___get_wchan(struct task_struct *p)
int count = 0;
sp = p->thread.ksp;
- if (!validate_sp(sp, p, STACK_FRAME_OVERHEAD))
+ if (!validate_sp(sp, p))
return 0;
do {
sp = READ_ONCE_NOCHECK(*(unsigned long *)sp);
- if (!validate_sp(sp, p, STACK_FRAME_OVERHEAD) ||
- task_is_running(p))
+ if (!validate_sp(sp, p) || task_is_running(p))
return 0;
if (count > 0) {
ip = READ_ONCE_NOCHECK(((unsigned long *)sp)[STACK_FRAME_LR_SAVE]);
@@ -2209,7 +2248,7 @@ void __no_sanitize_address show_stack(struct task_struct *tsk,
lr = 0;
printk("%sCall Trace:\n", loglvl);
do {
- if (!validate_sp(sp, tsk, STACK_FRAME_OVERHEAD))
+ if (!validate_sp(sp, tsk))
break;
stack = (unsigned long *) sp;
@@ -2230,12 +2269,16 @@ void __no_sanitize_address show_stack(struct task_struct *tsk,
/*
* See if this is an exception frame.
- * We look for the "regshere" marker in the current frame.
+ * We look for the "regs" marker in the current frame.
+ *
+ * STACK_SWITCH_FRAME_SIZE being the smallest frame that
+ * could hold a pt_regs, if that does not fit then it can't
+ * have regs.
*/
- if (validate_sp(sp, tsk, STACK_FRAME_WITH_PT_REGS)
- && stack[STACK_FRAME_MARKER] == STACK_FRAME_REGS_MARKER) {
+ if (validate_sp_size(sp, tsk, STACK_SWITCH_FRAME_SIZE)
+ && stack[STACK_INT_FRAME_MARKER_LONGS] == STACK_FRAME_REGS_MARKER) {
struct pt_regs *regs = (struct pt_regs *)
- (sp + STACK_FRAME_OVERHEAD);
+ (sp + STACK_INT_FRAME_REGS);
lr = regs->link;
printk("%s--- interrupt: %lx at %pS\n",
diff --git a/arch/powerpc/kernel/prom.c b/arch/powerpc/kernel/prom.c
index 1eed87d..4f1c920 100644
--- a/arch/powerpc/kernel/prom.c
+++ b/arch/powerpc/kernel/prom.c
@@ -72,6 +72,7 @@ int __initdata iommu_is_off;
int __initdata iommu_force_on;
unsigned long tce_alloc_start, tce_alloc_end;
u64 ppc64_rma_size;
+unsigned int boot_cpu_node_count __ro_after_init;
#endif
static phys_addr_t first_memblock_size;
static int __initdata boot_cpu_count;
@@ -335,6 +336,9 @@ static int __init early_init_dt_scan_cpus(unsigned long node,
if (type == NULL || strcmp(type, "cpu") != 0)
return 0;
+ if (IS_ENABLED(CONFIG_PPC64))
+ boot_cpu_node_count++;
+
/* Get physical cpuid */
intserv = of_get_flat_dt_prop(node, "ibm,ppc-interrupt-server#s", &len);
if (!intserv)
diff --git a/arch/powerpc/kernel/rtas.c b/arch/powerpc/kernel/rtas.c
index e847f9b..deded51 100644
--- a/arch/powerpc/kernel/rtas.c
+++ b/arch/powerpc/kernel/rtas.c
@@ -7,43 +7,35 @@
* Copyright (C) 2001 IBM.
*/
-#include <linux/stdarg.h>
-#include <linux/kernel.h>
-#include <linux/types.h>
-#include <linux/spinlock.h>
-#include <linux/export.h>
-#include <linux/init.h>
+#define pr_fmt(fmt) "rtas: " fmt
+
#include <linux/capability.h>
#include <linux/delay.h>
-#include <linux/cpu.h>
-#include <linux/sched.h>
-#include <linux/smp.h>
-#include <linux/completion.h>
-#include <linux/cpumask.h>
+#include <linux/export.h>
+#include <linux/init.h>
+#include <linux/kernel.h>
#include <linux/memblock.h>
-#include <linux/slab.h>
-#include <linux/reboot.h>
-#include <linux/security.h>
-#include <linux/syscalls.h>
#include <linux/of.h>
#include <linux/of_fdt.h>
-
-#include <asm/interrupt.h>
-#include <asm/rtas.h>
-#include <asm/hvcall.h>
-#include <asm/machdep.h>
-#include <asm/firmware.h>
-#include <asm/page.h>
-#include <asm/param.h>
-#include <asm/delay.h>
+#include <linux/reboot.h>
+#include <linux/sched.h>
+#include <linux/security.h>
+#include <linux/slab.h>
+#include <linux/spinlock.h>
+#include <linux/stdarg.h>
+#include <linux/syscalls.h>
+#include <linux/types.h>
#include <linux/uaccess.h>
-#include <asm/udbg.h>
-#include <asm/syscalls.h>
-#include <asm/smp.h>
-#include <linux/atomic.h>
-#include <asm/time.h>
+
+#include <asm/delay.h>
+#include <asm/firmware.h>
+#include <asm/interrupt.h>
+#include <asm/machdep.h>
#include <asm/mmu.h>
-#include <asm/topology.h>
+#include <asm/page.h>
+#include <asm/rtas.h>
+#include <asm/time.h>
+#include <asm/udbg.h>
/* This is here deliberately so it's only used in this file */
void enter_rtas(unsigned long);
@@ -353,6 +345,9 @@ int rtas_service_present(const char *service)
EXPORT_SYMBOL(rtas_service_present);
#ifdef CONFIG_RTAS_ERROR_LOGGING
+
+static u32 rtas_error_log_max __ro_after_init = RTAS_ERROR_LOG_MAX;
+
/*
* Return the firmware-specified size of the error log buffer
* for all rtas calls that require an error buffer argument.
@@ -360,21 +355,30 @@ EXPORT_SYMBOL(rtas_service_present);
*/
int rtas_get_error_log_max(void)
{
- static int rtas_error_log_max;
- if (rtas_error_log_max)
- return rtas_error_log_max;
-
- rtas_error_log_max = rtas_token ("rtas-error-log-max");
- if ((rtas_error_log_max == RTAS_UNKNOWN_SERVICE) ||
- (rtas_error_log_max > RTAS_ERROR_LOG_MAX)) {
- printk (KERN_WARNING "RTAS: bad log buffer size %d\n",
- rtas_error_log_max);
- rtas_error_log_max = RTAS_ERROR_LOG_MAX;
- }
return rtas_error_log_max;
}
EXPORT_SYMBOL(rtas_get_error_log_max);
+static void __init init_error_log_max(void)
+{
+ static const char propname[] __initconst = "rtas-error-log-max";
+ u32 max;
+
+ if (of_property_read_u32(rtas.dev, propname, &max)) {
+ pr_warn("%s not found, using default of %u\n",
+ propname, RTAS_ERROR_LOG_MAX);
+ max = RTAS_ERROR_LOG_MAX;
+ }
+
+ if (max > RTAS_ERROR_LOG_MAX) {
+ pr_warn("%s = %u, clamping max error log size to %u\n",
+ propname, max, RTAS_ERROR_LOG_MAX);
+ max = RTAS_ERROR_LOG_MAX;
+ }
+
+ rtas_error_log_max = max;
+}
+
static char rtas_err_buf[RTAS_ERROR_LOG_MAX];
static int rtas_last_error_token;
@@ -432,6 +436,7 @@ static char *__fetch_rtas_last_error(char *altbuf)
#else /* CONFIG_RTAS_ERROR_LOGGING */
#define __fetch_rtas_last_error(x) NULL
#define get_errorlog_buffer() NULL
+static void __init init_error_log_max(void) {}
#endif
@@ -467,6 +472,64 @@ void rtas_call_unlocked(struct rtas_args *args, int token, int nargs, int nret,
static int ibm_open_errinjct_token;
static int ibm_errinjct_token;
+/**
+ * rtas_call() - Invoke an RTAS firmware function.
+ * @token: Identifies the function being invoked.
+ * @nargs: Number of input parameters. Does not include token.
+ * @nret: Number of output parameters, including the call status.
+ * @outputs: Array of @nret output words.
+ * @....: List of @nargs input parameters.
+ *
+ * Invokes the RTAS function indicated by @token, which the caller
+ * should obtain via rtas_token().
+ *
+ * The @nargs and @nret arguments must match the number of input and
+ * output parameters specified for the RTAS function.
+ *
+ * rtas_call() returns RTAS status codes, not conventional Linux errno
+ * values. Callers must translate any failure to an appropriate errno
+ * in syscall context. Most callers of RTAS functions that can return
+ * -2 or 990x should use rtas_busy_delay() to correctly handle those
+ * statuses before calling again.
+ *
+ * The return value descriptions are adapted from 7.2.8 [RTAS] Return
+ * Codes of the PAPR and CHRP specifications.
+ *
+ * Context: Process context preferably, interrupt context if
+ * necessary. Acquires an internal spinlock and may perform
+ * GFP_ATOMIC slab allocation in error path. Unsafe for NMI
+ * context.
+ * Return:
+ * * 0 - RTAS function call succeeded.
+ * * -1 - RTAS function encountered a hardware or
+ * platform error, or the token is invalid,
+ * or the function is restricted by kernel policy.
+ * * -2 - Specs say "A necessary hardware device was busy,
+ * and the requested function could not be
+ * performed. The operation should be retried at
+ * a later time." This is misleading, at least with
+ * respect to current RTAS implementations. What it
+ * usually means in practice is that the function
+ * could not be completed while meeting RTAS's
+ * deadline for returning control to the OS (250us
+ * for PAPR/PowerVM, typically), but the call may be
+ * immediately reattempted to resume work on it.
+ * * -3 - Parameter error.
+ * * -7 - Unexpected state change.
+ * * 9000...9899 - Vendor-specific success codes.
+ * * 9900...9905 - Advisory extended delay. Caller should try
+ * again after ~10^x ms has elapsed, where x is
+ * the last digit of the status [0-5]. Again going
+ * beyond the PAPR text, 990x on PowerVM indicates
+ * contention for RTAS-internal resources. Other
+ * RTAS call sequences in progress should be
+ * allowed to complete before reattempting the
+ * call.
+ * * -9000 - Multi-level isolation error.
+ * * -9999...-9004 - Vendor-specific error codes.
+ * * Additional negative values - Function-specific error.
+ * * Additional positive values - Function-specific success.
+ */
int rtas_call(int token, int nargs, int nret, int *outputs, ...)
{
va_list list;
@@ -657,8 +720,7 @@ static int rtas_error_rc(int rtas_rc)
rc = -ENODEV;
break;
default:
- printk(KERN_ERR "%s: unexpected RTAS error %d\n",
- __func__, rtas_rc);
+ pr_err("%s: unexpected error %d\n", __func__, rtas_rc);
rc = -ERANGE;
break;
}
@@ -862,8 +924,8 @@ void __noreturn rtas_restart(char *cmd)
{
if (rtas_flash_term_hook)
rtas_flash_term_hook(SYS_RESTART);
- printk("RTAS system-reboot returned %d\n",
- rtas_call(rtas_token("system-reboot"), 0, 1, NULL));
+ pr_emerg("system-reboot returned %d\n",
+ rtas_call(rtas_token("system-reboot"), 0, 1, NULL));
for (;;);
}
@@ -872,8 +934,8 @@ void rtas_power_off(void)
if (rtas_flash_term_hook)
rtas_flash_term_hook(SYS_POWER_OFF);
/* allow power on only with power button press */
- printk("RTAS power-off returned %d\n",
- rtas_call(rtas_token("power-off"), 2, 1, NULL, -1, -1));
+ pr_emerg("power-off returned %d\n",
+ rtas_call(rtas_token("power-off"), 2, 1, NULL, -1, -1));
for (;;);
}
@@ -882,13 +944,14 @@ void __noreturn rtas_halt(void)
if (rtas_flash_term_hook)
rtas_flash_term_hook(SYS_HALT);
/* allow power on only with power button press */
- printk("RTAS power-off returned %d\n",
- rtas_call(rtas_token("power-off"), 2, 1, NULL, -1, -1));
+ pr_emerg("power-off returned %d\n",
+ rtas_call(rtas_token("power-off"), 2, 1, NULL, -1, -1));
for (;;);
}
/* Must be in the RMO region, so we place it here */
static char rtas_os_term_buf[2048];
+static s32 ibm_os_term_token = RTAS_UNKNOWN_SERVICE;
void rtas_os_term(char *str)
{
@@ -900,19 +963,23 @@ void rtas_os_term(char *str)
* this property may terminate the partition which we want to avoid
* since it interferes with panic_timeout.
*/
- if (RTAS_UNKNOWN_SERVICE == rtas_token("ibm,os-term") ||
- RTAS_UNKNOWN_SERVICE == rtas_token("ibm,extended-os-term"))
+ if (ibm_os_term_token == RTAS_UNKNOWN_SERVICE)
return;
snprintf(rtas_os_term_buf, 2048, "OS panic: %s", str);
+ /*
+ * Keep calling as long as RTAS returns a "try again" status,
+ * but don't use rtas_busy_delay(), which potentially
+ * schedules.
+ */
do {
- status = rtas_call(rtas_token("ibm,os-term"), 1, 1, NULL,
+ status = rtas_call(ibm_os_term_token, 1, 1, NULL,
__pa(rtas_os_term_buf));
- } while (rtas_busy_delay(status));
+ } while (rtas_busy_delay_time(status));
if (status != 0)
- printk(KERN_EMERG "ibm,os-term call failed %d\n", status);
+ pr_emerg("ibm,os-term call failed %d\n", status);
}
/**
@@ -983,8 +1050,6 @@ noinstr struct pseries_errorlog *get_pseries_errorlog(struct rtas_error_log *log
return NULL;
}
-#ifdef CONFIG_PPC_RTAS_FILTER
-
/*
* The sys_rtas syscall, as originally designed, allows root to pass
* arbitrary physical addresses to RTAS calls. A number of RTAS calls
@@ -1133,20 +1198,6 @@ static void __init rtas_syscall_filter_init(void)
rtas_filters[i].token = rtas_token(rtas_filters[i].name);
}
-#else
-
-static bool block_rtas_call(int token, int nargs,
- struct rtas_args *args)
-{
- return false;
-}
-
-static void __init rtas_syscall_filter_init(void)
-{
-}
-
-#endif /* CONFIG_PPC_RTAS_FILTER */
-
/* We assume to be passed big endian arguments */
SYSCALL_DEFINE1(rtas, struct rtas_args __user *, uargs)
{
@@ -1277,6 +1328,15 @@ void __init rtas_initialize(void)
no_entry = of_property_read_u32(rtas.dev, "linux,rtas-entry", &entry);
rtas.entry = no_entry ? rtas.base : entry;
+ init_error_log_max();
+
+ /*
+ * Discover these now to avoid device tree lookups in the
+ * panic path.
+ */
+ if (of_property_read_bool(rtas.dev, "ibm,extended-os-term"))
+ ibm_os_term_token = rtas_token("ibm,os-term");
+
/* If RTAS was found, allocate the RMO buffer for it and look for
* the stop-self token if any
*/
diff --git a/arch/powerpc/kernel/rtasd.c b/arch/powerpc/kernel/rtasd.c
index 5270b45..cc56ac6 100644
--- a/arch/powerpc/kernel/rtasd.c
+++ b/arch/powerpc/kernel/rtasd.c
@@ -9,6 +9,7 @@
#include <linux/errno.h>
#include <linux/sched.h>
#include <linux/kernel.h>
+#include <linux/of.h>
#include <linux/poll.h>
#include <linux/proc_fs.h>
#include <linux/init.h>
@@ -499,6 +500,8 @@ EXPORT_SYMBOL_GPL(rtas_cancel_event_scan);
static int __init rtas_event_scan_init(void)
{
+ int err;
+
if (!machine_is(pseries) && !machine_is(chrp))
return 0;
@@ -509,8 +512,8 @@ static int __init rtas_event_scan_init(void)
return -ENODEV;
}
- rtas_event_scan_rate = rtas_token("rtas-event-scan-rate");
- if (rtas_event_scan_rate == RTAS_UNKNOWN_SERVICE) {
+ err = of_property_read_u32(rtas.dev, "rtas-event-scan-rate", &rtas_event_scan_rate);
+ if (err) {
printk(KERN_ERR "rtasd: no rtas-event-scan-rate on system\n");
return -ENODEV;
}
diff --git a/arch/powerpc/kernel/smp.c b/arch/powerpc/kernel/smp.c
index 0da6e59..6b90f10 100644
--- a/arch/powerpc/kernel/smp.c
+++ b/arch/powerpc/kernel/smp.c
@@ -1249,7 +1249,7 @@ static void cpu_idle_thread_init(unsigned int cpu, struct task_struct *idle)
#ifdef CONFIG_PPC64
paca_ptrs[cpu]->__current = idle;
paca_ptrs[cpu]->kstack = (unsigned long)task_stack_page(idle) +
- THREAD_SIZE - STACK_FRAME_OVERHEAD;
+ THREAD_SIZE - STACK_FRAME_MIN_SIZE;
#endif
task_thread_info(idle)->cpu = cpu;
secondary_current = current_set[cpu] = idle;
diff --git a/arch/powerpc/kernel/stacktrace.c b/arch/powerpc/kernel/stacktrace.c
index a2443d6..5de8597 100644
--- a/arch/powerpc/kernel/stacktrace.c
+++ b/arch/powerpc/kernel/stacktrace.c
@@ -43,7 +43,7 @@ void __no_sanitize_address arch_stack_walk(stack_trace_consume_fn consume_entry,
unsigned long *stack = (unsigned long *) sp;
unsigned long newsp, ip;
- if (!validate_sp(sp, task, STACK_FRAME_OVERHEAD))
+ if (!validate_sp(sp, task))
return;
newsp = stack[0];
@@ -77,7 +77,7 @@ int __no_sanitize_address arch_stack_walk_reliable(stack_trace_consume_fn consum
/*
* For user tasks, this is the SP value loaded on
* kernel entry, see "PACAKSAVE(r13)" in _switch() and
- * system_call_common()/EXCEPTION_PROLOG_COMMON().
+ * system_call_common().
*
* Likewise for non-swapper kernel threads,
* this also happens to be the top of the stack
@@ -88,13 +88,13 @@ int __no_sanitize_address arch_stack_walk_reliable(stack_trace_consume_fn consum
* an unreliable stack trace until it's been
* _switch()'ed to for the first time.
*/
- stack_end -= STACK_FRAME_OVERHEAD + sizeof(struct pt_regs);
+ stack_end -= STACK_USER_INT_FRAME_SIZE;
} else {
/*
* idle tasks have a custom stack layout,
* c.f. cpu_idle_thread_init().
*/
- stack_end -= STACK_FRAME_OVERHEAD;
+ stack_end -= STACK_FRAME_MIN_SIZE;
}
if (task == current)
@@ -136,7 +136,7 @@ int __no_sanitize_address arch_stack_walk_reliable(stack_trace_consume_fn consum
/* Mark stacktraces with exception frames as unreliable. */
if (sp <= stack_end - STACK_INT_FRAME_SIZE &&
- stack[STACK_FRAME_MARKER] == STACK_FRAME_REGS_MARKER) {
+ stack[STACK_INT_FRAME_MARKER_LONGS] == STACK_FRAME_REGS_MARKER) {
return -EINVAL;
}
diff --git a/arch/powerpc/kernel/swsusp_32.S b/arch/powerpc/kernel/swsusp_32.S
index e0cbd63..ffb7932 100644
--- a/arch/powerpc/kernel/swsusp_32.S
+++ b/arch/powerpc/kernel/swsusp_32.S
@@ -1,5 +1,7 @@
/* SPDX-License-Identifier: GPL-2.0 */
#include <linux/threads.h>
+#include <linux/linkage.h>
+
#include <asm/processor.h>
#include <asm/page.h>
#include <asm/cputable.h>
@@ -400,7 +402,7 @@
/* FIXME:This construct is actually not useful since we don't shut
* down the instruction MMU, we could just flip back MSR-DR on.
*/
-turn_on_mmu:
+SYM_FUNC_START_LOCAL(turn_on_mmu)
mflr r4
mtsrr0 r4
mtsrr1 r3
@@ -408,4 +410,5 @@
isync
rfi
_ASM_NOKPROBE_SYMBOL(turn_on_mmu)
+SYM_FUNC_END(turn_on_mmu)
diff --git a/arch/powerpc/kernel/time.c b/arch/powerpc/kernel/time.c
index a2ab397..d68de36 100644
--- a/arch/powerpc/kernel/time.c
+++ b/arch/powerpc/kernel/time.c
@@ -130,7 +130,7 @@ unsigned long tb_ticks_per_jiffy;
unsigned long tb_ticks_per_usec = 100; /* sane default */
EXPORT_SYMBOL(tb_ticks_per_usec);
unsigned long tb_ticks_per_sec;
-EXPORT_SYMBOL(tb_ticks_per_sec); /* for cputime_t conversions */
+EXPORT_SYMBOL(tb_ticks_per_sec); /* for cputime conversions */
DEFINE_SPINLOCK(rtc_lock);
EXPORT_SYMBOL_GPL(rtc_lock);
@@ -151,21 +151,6 @@ bool tb_invalid;
#ifdef CONFIG_VIRT_CPU_ACCOUNTING_NATIVE
/*
- * Factor for converting from cputime_t (timebase ticks) to
- * microseconds. This is stored as 0.64 fixed-point binary fraction.
- */
-u64 __cputime_usec_factor;
-EXPORT_SYMBOL(__cputime_usec_factor);
-
-static void calc_cputime_factors(void)
-{
- struct div_result res;
-
- div128_by_32(1000000, 0, tb_ticks_per_sec, &res);
- __cputime_usec_factor = res.result_low;
-}
-
-/*
* Read the SPURR on systems that have it, otherwise the PURR,
* or if that doesn't exist return the timebase value passed in.
*/
@@ -369,10 +354,7 @@ void vtime_flush(struct task_struct *tsk)
acct->hardirq_time = 0;
acct->softirq_time = 0;
}
-
-#else /* ! CONFIG_VIRT_CPU_ACCOUNTING_NATIVE */
-#define calc_cputime_factors()
-#endif
+#endif /* CONFIG_VIRT_CPU_ACCOUNTING_NATIVE */
void __delay(unsigned long loops)
{
@@ -914,7 +896,6 @@ void __init time_init(void)
tb_ticks_per_jiffy = ppc_tb_freq / HZ;
tb_ticks_per_sec = ppc_tb_freq;
tb_ticks_per_usec = ppc_tb_freq / 1000000;
- calc_cputime_factors();
/*
* Compute scale factor for sched_clock.
diff --git a/arch/powerpc/kernel/tm.S b/arch/powerpc/kernel/tm.S
index 5a0f023..9feab5e 100644
--- a/arch/powerpc/kernel/tm.S
+++ b/arch/powerpc/kernel/tm.S
@@ -117,7 +117,7 @@
std r2, STK_GOT(r1)
stdu r1, -TM_FRAME_SIZE(r1)
- /* We've a struct pt_regs at [r1+STACK_FRAME_OVERHEAD]. */
+ /* We've a struct pt_regs at [r1+STACK_INT_FRAME_REGS]. */
std r3, STK_PARAM(R3)(r1)
SAVE_NVGPRS(r1)
@@ -222,7 +222,7 @@
* Make r7 look like an exception frame so that we can use the neat
* GPRx(n) macros. r7 is NOT a pt_regs ptr!
*/
- subi r7, r7, STACK_FRAME_OVERHEAD
+ subi r7, r7, STACK_INT_FRAME_REGS
/* Sync the userland GPRs 2-12, 14-31 to thread->regs: */
SAVE_GPR(0, r7) /* user r0 */
@@ -359,7 +359,7 @@
stdu r1, -TM_FRAME_SIZE(r1)
/*
- * We've a struct pt_regs at [r1+STACK_FRAME_OVERHEAD].
+ * We've a struct pt_regs at [r1+STACK_INT_FRAME_REGS].
* This is used for backing up the NVGPRs:
*/
SAVE_NVGPRS(r1)
@@ -379,7 +379,7 @@
* Make r7 look like an exception frame so that we can use the neat
* GPRx(n) macros. r7 is now NOT a pt_regs ptr!
*/
- subi r7, r7, STACK_FRAME_OVERHEAD
+ subi r7, r7, STACK_INT_FRAME_REGS
/* We need to setup MSR for FP/VMX/VSX register save instructions. */
mfmsr r6
diff --git a/arch/powerpc/kernel/trace/ftrace_mprofile.S b/arch/powerpc/kernel/trace/ftrace_mprofile.S
index d031093..ffb1db3 100644
--- a/arch/powerpc/kernel/trace/ftrace_mprofile.S
+++ b/arch/powerpc/kernel/trace/ftrace_mprofile.S
@@ -110,7 +110,7 @@
.endif
/* Load &pt_regs in r6 for call below */
- addi r6, r1, STACK_FRAME_OVERHEAD
+ addi r6, r1, STACK_INT_FRAME_REGS
.endm
.macro ftrace_regs_exit allregs
diff --git a/arch/powerpc/kernel/vdso/Makefile b/arch/powerpc/kernel/vdso/Makefile
index a2e7b0c..6a977b0 100644
--- a/arch/powerpc/kernel/vdso/Makefile
+++ b/arch/powerpc/kernel/vdso/Makefile
@@ -102,3 +102,5 @@
cmd_vdso64ld_and_check = $(VDSOCC) $(c_flags) $(CC64FLAGS) -o $@ -Wl,-T$(filter %.lds,$^) $(filter %.o,$^) -z noexecstack ; $(cmd_vdso_check)
quiet_cmd_vdso64as = VDSO64A $@
cmd_vdso64as = $(VDSOCC) $(a_flags) $(CC64FLAGS) $(AS64FLAGS) -c -o $@ $<
+
+OBJECT_FILES_NON_STANDARD := y
diff --git a/arch/powerpc/kernel/vector.S b/arch/powerpc/kernel/vector.S
index 5cf64740..ffe5d90 100644
--- a/arch/powerpc/kernel/vector.S
+++ b/arch/powerpc/kernel/vector.S
@@ -1,4 +1,5 @@
/* SPDX-License-Identifier: GPL-2.0 */
+#include <linux/linkage.h>
#include <asm/processor.h>
#include <asm/ppc_asm.h>
#include <asm/reg.h>
@@ -185,7 +186,7 @@
* Internal routine to enable floating point and set FPSCR to 0.
* Don't call it from C; it doesn't use the normal calling convention.
*/
-fpenable:
+SYM_FUNC_START_LOCAL(fpenable)
#ifdef CONFIG_PPC32
stwu r1,-64(r1)
#else
@@ -202,6 +203,7 @@
mffs fr31
MTFSF_L(fr1)
blr
+SYM_FUNC_END(fpenable)
fpdisable:
mtlr r12
diff --git a/arch/powerpc/kexec/file_load_64.c b/arch/powerpc/kexec/file_load_64.c
index 60e12b7..af8854f 100644
--- a/arch/powerpc/kexec/file_load_64.c
+++ b/arch/powerpc/kexec/file_load_64.c
@@ -26,6 +26,7 @@
#include <asm/firmware.h>
#include <asm/kexec_ranges.h>
#include <asm/crashdump-ppc64.h>
+#include <asm/prom.h>
struct umem_info {
u64 *buf; /* data buffer for usable-memory property */
@@ -929,6 +930,45 @@ int setup_purgatory_ppc64(struct kimage *image, const void *slave_code,
}
/**
+ * get_cpu_node_size - Compute the size of a CPU node in the FDT.
+ * This should be done only once and the value is stored in
+ * a static variable.
+ * Returns the max size of a CPU node in the FDT.
+ */
+static unsigned int cpu_node_size(void)
+{
+ static unsigned int size;
+ struct device_node *dn;
+ struct property *pp;
+
+ /*
+ * Don't compute it twice, we are assuming that the per CPU node size
+ * doesn't change during the system's life.
+ */
+ if (size)
+ return size;
+
+ dn = of_find_node_by_type(NULL, "cpu");
+ if (WARN_ON_ONCE(!dn)) {
+ // Unlikely to happen
+ return 0;
+ }
+
+ /*
+ * We compute the sub node size for a CPU node, assuming it
+ * will be the same for all.
+ */
+ size += strlen(dn->name) + 5;
+ for_each_property_of_node(dn, pp) {
+ size += strlen(pp->name);
+ size += pp->length;
+ }
+
+ of_node_put(dn);
+ return size;
+}
+
+/**
* kexec_extra_fdt_size_ppc64 - Return the estimated additional size needed to
* setup FDT for kexec/kdump kernel.
* @image: kexec image being loaded.
@@ -937,6 +977,8 @@ int setup_purgatory_ppc64(struct kimage *image, const void *slave_code,
*/
unsigned int kexec_extra_fdt_size_ppc64(struct kimage *image)
{
+ unsigned int cpu_nodes, extra_size;
+ struct device_node *dn;
u64 usm_entries;
if (image->type != KEXEC_TYPE_CRASH)
@@ -949,7 +991,22 @@ unsigned int kexec_extra_fdt_size_ppc64(struct kimage *image)
*/
usm_entries = ((memblock_end_of_DRAM() / drmem_lmb_size()) +
(2 * (resource_size(&crashk_res) / drmem_lmb_size())));
- return (unsigned int)(usm_entries * sizeof(u64));
+
+ extra_size = (unsigned int)(usm_entries * sizeof(u64));
+
+ /*
+ * Get the number of CPU nodes in the current DT. This allows to
+ * reserve places for CPU nodes added since the boot time.
+ */
+ cpu_nodes = 0;
+ for_each_node_by_type(dn, "cpu") {
+ cpu_nodes++;
+ }
+
+ if (cpu_nodes > boot_cpu_node_count)
+ extra_size += (cpu_nodes - boot_cpu_node_count) * cpu_node_size();
+
+ return extra_size;
}
/**
diff --git a/arch/powerpc/kvm/book3s_64_mmu_hv.c b/arch/powerpc/kvm/book3s_64_mmu_hv.c
index 4939f57..7006bcb 100644
--- a/arch/powerpc/kvm/book3s_64_mmu_hv.c
+++ b/arch/powerpc/kvm/book3s_64_mmu_hv.c
@@ -1202,7 +1202,7 @@ static int resize_hpt_allocate(struct kvm_resize_hpt *resize)
if (rc < 0)
return rc;
- resize_hpt_debug(resize, "resize_hpt_allocate(): HPT @ 0x%lx\n",
+ resize_hpt_debug(resize, "%s(): HPT @ 0x%lx\n", __func__,
resize->hpt.virt);
return 0;
@@ -1443,7 +1443,7 @@ static void resize_hpt_prepare_work(struct work_struct *work)
*/
mutex_unlock(&kvm->arch.mmu_setup_lock);
- resize_hpt_debug(resize, "resize_hpt_prepare_work(): order = %d\n",
+ resize_hpt_debug(resize, "%s(): order = %d\n", __func__,
resize->order);
err = resize_hpt_allocate(resize);
@@ -1887,8 +1887,7 @@ static ssize_t kvm_htab_write(struct file *file, const char __user *buf,
ret = kvmppc_virtmode_do_h_enter(kvm, H_EXACT, i, v, r,
tmp);
if (ret != H_SUCCESS) {
- pr_err("kvm_htab_write ret %ld i=%ld v=%lx "
- "r=%lx\n", ret, i, v, r);
+ pr_err("%s ret %ld i=%ld v=%lx r=%lx\n", __func__, ret, i, v, r);
goto out;
}
if (!mmu_ready && is_vrma_hpte(v)) {
diff --git a/arch/powerpc/kvm/book3s_64_vio.c b/arch/powerpc/kvm/book3s_64_vio.c
index 4086437..95e738e 100644
--- a/arch/powerpc/kvm/book3s_64_vio.c
+++ b/arch/powerpc/kvm/book3s_64_vio.c
@@ -294,14 +294,14 @@ long kvm_vm_ioctl_create_spapr_tce(struct kvm *kvm,
struct kvmppc_spapr_tce_table *stt = NULL;
struct kvmppc_spapr_tce_table *siter;
struct mm_struct *mm = kvm->mm;
- unsigned long npages, size = args->size;
+ unsigned long npages;
int ret;
if (!args->size || args->page_shift < 12 || args->page_shift > 34 ||
(args->offset + args->size > (ULLONG_MAX >> args->page_shift)))
return -EINVAL;
- npages = kvmppc_tce_pages(size);
+ npages = kvmppc_tce_pages(args->size);
ret = account_locked_vm(mm, kvmppc_stt_pages(npages), true);
if (ret)
return ret;
@@ -314,7 +314,7 @@ long kvm_vm_ioctl_create_spapr_tce(struct kvm *kvm,
stt->liobn = args->liobn;
stt->page_shift = args->page_shift;
stt->offset = args->offset;
- stt->size = size;
+ stt->size = args->size;
stt->kvm = kvm;
mutex_init(&stt->alloc_lock);
INIT_LIST_HEAD_RCU(&stt->iommu_tables);
diff --git a/arch/powerpc/kvm/book3s_hv_interrupts.S b/arch/powerpc/kvm/book3s_hv_interrupts.S
index 59d89e4..c0deeea 100644
--- a/arch/powerpc/kvm/book3s_hv_interrupts.S
+++ b/arch/powerpc/kvm/book3s_hv_interrupts.S
@@ -9,6 +9,7 @@
* Authors: Alexander Graf <agraf@suse.de>
*/
+#include <linux/linkage.h>
#include <asm/ppc_asm.h>
#include <asm/kvm_asm.h>
#include <asm/reg.h>
@@ -107,7 +108,7 @@
/*
* void kvmhv_save_host_pmu(void)
*/
-kvmhv_save_host_pmu:
+SYM_FUNC_START_LOCAL(kvmhv_save_host_pmu)
BEGIN_FTR_SECTION
/* Work around P8 PMAE bug */
li r3, -1
@@ -154,3 +155,4 @@
stw r8, HSTATE_PMC5(r13)
stw r9, HSTATE_PMC6(r13)
31: blr
+SYM_FUNC_END(kvmhv_save_host_pmu)
diff --git a/arch/powerpc/kvm/book3s_hv_rmhandlers.S b/arch/powerpc/kvm/book3s_hv_rmhandlers.S
index 37f50861d..acf8091 100644
--- a/arch/powerpc/kvm/book3s_hv_rmhandlers.S
+++ b/arch/powerpc/kvm/book3s_hv_rmhandlers.S
@@ -10,6 +10,8 @@
* Authors: Alexander Graf <agraf@suse.de>
*/
+#include <linux/linkage.h>
+#include <linux/objtool.h>
#include <asm/ppc_asm.h>
#include <asm/code-patching-asm.h>
#include <asm/kvm_asm.h>
@@ -1522,12 +1524,14 @@
/* Flush the link stack. On Power8 it's up to 32 entries in size. */
.rept 32
+ ANNOTATE_INTRA_FUNCTION_CALL
bl .+4
.endr
/* And on Power9 it's up to 64. */
BEGIN_FTR_SECTION
.rept 32
+ ANNOTATE_INTRA_FUNCTION_CALL
bl .+4
.endr
END_FTR_SECTION_IFSET(CPU_FTR_ARCH_300)
@@ -2358,7 +2362,7 @@
* This routine calls kvmppc_read_intr, a C function, if an external
* interrupt is pending.
*/
-kvmppc_check_wake_reason:
+SYM_FUNC_START_LOCAL(kvmppc_check_wake_reason)
mfspr r6, SPRN_SRR1
BEGIN_FTR_SECTION
rlwinm r6, r6, 45-31, 0xf /* extract wake reason field (P8) */
@@ -2427,6 +2431,7 @@
addi r1, r1, PPC_MIN_STKFRM
mtlr r0
blr
+SYM_FUNC_END(kvmppc_check_wake_reason)
/*
* Save away FP, VMX and VSX registers.
@@ -2434,7 +2439,7 @@
* N.B. r30 and r31 are volatile across this function,
* thus it is not callable from C.
*/
-kvmppc_save_fp:
+SYM_FUNC_START_LOCAL(kvmppc_save_fp)
mflr r30
mr r31,r3
mfmsr r5
@@ -2462,6 +2467,7 @@
stw r6,VCPU_VRSAVE(r31)
mtlr r30
blr
+SYM_FUNC_END(kvmppc_save_fp)
/*
* Load up FP, VMX and VSX registers
@@ -2469,7 +2475,7 @@
* N.B. r30 and r31 are volatile across this function,
* thus it is not callable from C.
*/
-kvmppc_load_fp:
+SYM_FUNC_START_LOCAL(kvmppc_load_fp)
mflr r30
mr r31,r4
mfmsr r9
@@ -2498,6 +2504,7 @@
mtlr r30
mr r4,r31
blr
+SYM_FUNC_END(kvmppc_load_fp)
#ifdef CONFIG_PPC_TRANSACTIONAL_MEM
/*
@@ -2729,7 +2736,7 @@
std r6, SOFTE(r1)
LOAD_PACA_TOC()
LOAD_REG_IMMEDIATE(3, STACK_FRAME_REGS_MARKER)
- std r3, STACK_FRAME_OVERHEAD-16(r1)
+ std r3, STACK_INT_FRAME_MARKER(r1)
/*
* XXX On POWER7 and POWER8, we just spin here since we don't
@@ -2746,7 +2753,7 @@
* r9 has a vcpu pointer (in)
* r0 is used as a scratch register
*/
-kvmppc_msr_interrupt:
+SYM_FUNC_START_LOCAL(kvmppc_msr_interrupt)
rldicl r0, r11, 64 - MSR_TS_S_LG, 62
cmpwi r0, 2 /* Check if we are in transactional state.. */
ld r11, VCPU_INTR_MSR(r9)
@@ -2755,13 +2762,14 @@
li r0, 1
1: rldimi r11, r0, MSR_TS_S_LG, 63 - MSR_TS_T_LG
blr
+SYM_FUNC_END(kvmppc_msr_interrupt)
/*
* void kvmhv_load_guest_pmu(struct kvm_vcpu *vcpu)
*
* Load up guest PMU state. R3 points to the vcpu struct.
*/
-kvmhv_load_guest_pmu:
+SYM_FUNC_START_LOCAL(kvmhv_load_guest_pmu)
mr r4, r3
mflr r0
li r3, 1
@@ -2811,13 +2819,14 @@
isync
mtlr r0
blr
+SYM_FUNC_END(kvmhv_load_guest_pmu)
/*
* void kvmhv_load_host_pmu(void)
*
* Reload host PMU state saved in the PACA by kvmhv_save_host_pmu.
*/
-kvmhv_load_host_pmu:
+SYM_FUNC_START_LOCAL(kvmhv_load_host_pmu)
mflr r0
lbz r4, PACA_PMCINUSE(r13) /* is the host using the PMU? */
cmpwi r4, 0
@@ -2859,6 +2868,7 @@
isync
mtlr r0
23: blr
+SYM_FUNC_END(kvmhv_load_host_pmu)
/*
* void kvmhv_save_guest_pmu(struct kvm_vcpu *vcpu, bool pmu_in_use)
@@ -2866,7 +2876,7 @@
* Save guest PMU state into the vcpu struct.
* r3 = vcpu, r4 = full save flag (PMU in use flag set in VPA)
*/
-kvmhv_save_guest_pmu:
+SYM_FUNC_START_LOCAL(kvmhv_save_guest_pmu)
mr r9, r3
mr r8, r4
BEGIN_FTR_SECTION
@@ -2942,6 +2952,7 @@
mtspr SPRN_MMCRS, r4
END_FTR_SECTION_IFSET(CPU_FTR_ARCH_207S)
22: blr
+SYM_FUNC_END(kvmhv_save_guest_pmu)
/*
* This works around a hardware bug on POWER8E processors, where
diff --git a/arch/powerpc/kvm/book3s_hv_uvmem.c b/arch/powerpc/kvm/book3s_hv_uvmem.c
index e2f11f9..1d67baa 100644
--- a/arch/powerpc/kvm/book3s_hv_uvmem.c
+++ b/arch/powerpc/kvm/book3s_hv_uvmem.c
@@ -1190,8 +1190,7 @@ int kvmppc_uvmem_init(void)
pfn_first = res->start >> PAGE_SHIFT;
pfn_last = pfn_first + (resource_size(res) >> PAGE_SHIFT);
- kvmppc_uvmem_bitmap = kcalloc(BITS_TO_LONGS(pfn_last - pfn_first),
- sizeof(unsigned long), GFP_KERNEL);
+ kvmppc_uvmem_bitmap = bitmap_zalloc(pfn_last - pfn_first, GFP_KERNEL);
if (!kvmppc_uvmem_bitmap) {
ret = -ENOMEM;
goto out_unmap;
@@ -1215,5 +1214,5 @@ void kvmppc_uvmem_free(void)
memunmap_pages(&kvmppc_uvmem_pgmap);
release_mem_region(kvmppc_uvmem_pgmap.range.start,
range_len(&kvmppc_uvmem_pgmap.range));
- kfree(kvmppc_uvmem_bitmap);
+ bitmap_free(kvmppc_uvmem_bitmap);
}
diff --git a/arch/powerpc/kvm/book3s_xive.c b/arch/powerpc/kvm/book3s_xive.c
index 4ca2364..f411581 100644
--- a/arch/powerpc/kvm/book3s_xive.c
+++ b/arch/powerpc/kvm/book3s_xive.c
@@ -539,7 +539,7 @@ static int xive_vm_h_eoi(struct kvm_vcpu *vcpu, unsigned long xirr)
if (irq == XICS_IPI || irq == 0) {
/*
* This barrier orders the setting of xc->cppr vs.
- * subsquent test of xc->mfrr done inside
+ * subsequent test of xc->mfrr done inside
* scan_interrupts and push_pending_to_hw
*/
smp_mb();
@@ -563,7 +563,7 @@ static int xive_vm_h_eoi(struct kvm_vcpu *vcpu, unsigned long xirr)
/*
* This barrier orders both setting of in_eoi above vs,
* subsequent test of guest_priority, and the setting
- * of xc->cppr vs. subsquent test of xc->mfrr done inside
+ * of xc->cppr vs. subsequent test of xc->mfrr done inside
* scan_interrupts and push_pending_to_hw
*/
smp_mb();
@@ -1785,8 +1785,7 @@ void kvmppc_xive_disable_vcpu_interrupts(struct kvm_vcpu *vcpu)
* stale_p (because it has no easy way to address it). Hence we have
* to adjust stale_p before shutting down the interrupt.
*/
-void xive_cleanup_single_escalation(struct kvm_vcpu *vcpu,
- struct kvmppc_xive_vcpu *xc, int irq)
+void xive_cleanup_single_escalation(struct kvm_vcpu *vcpu, int irq)
{
struct irq_data *d = irq_get_irq_data(irq);
struct xive_irq_data *xd = irq_data_get_irq_handler_data(d);
@@ -1827,8 +1826,7 @@ void kvmppc_xive_cleanup_vcpu(struct kvm_vcpu *vcpu)
for (i = 0; i < KVMPPC_XIVE_Q_COUNT; i++) {
if (xc->esc_virq[i]) {
if (kvmppc_xive_has_single_escalation(xc->xive))
- xive_cleanup_single_escalation(vcpu, xc,
- xc->esc_virq[i]);
+ xive_cleanup_single_escalation(vcpu, xc->esc_virq[i]);
free_irq(xc->esc_virq[i], vcpu);
irq_dispose_mapping(xc->esc_virq[i]);
kfree(xc->esc_virq_names[i]);
@@ -2392,7 +2390,7 @@ static int xive_set_source(struct kvmppc_xive *xive, long irq, u64 addr)
/*
* Now, we select a target if we have one. If we don't we
* leave the interrupt untargetted. It means that an interrupt
- * can become "untargetted" accross migration if it was masked
+ * can become "untargetted" across migration if it was masked
* by set_xive() but there is little we can do about it.
*/
diff --git a/arch/powerpc/kvm/book3s_xive.h b/arch/powerpc/kvm/book3s_xive.h
index 1e48f72..62bf39f 100644
--- a/arch/powerpc/kvm/book3s_xive.h
+++ b/arch/powerpc/kvm/book3s_xive.h
@@ -299,8 +299,7 @@ int kvmppc_xive_select_target(struct kvm *kvm, u32 *server, u8 prio);
int kvmppc_xive_attach_escalation(struct kvm_vcpu *vcpu, u8 prio,
bool single_escalation);
struct kvmppc_xive *kvmppc_xive_get_device(struct kvm *kvm, u32 type);
-void xive_cleanup_single_escalation(struct kvm_vcpu *vcpu,
- struct kvmppc_xive_vcpu *xc, int irq);
+void xive_cleanup_single_escalation(struct kvm_vcpu *vcpu, int irq);
int kvmppc_xive_compute_vp_id(struct kvmppc_xive *xive, u32 cpu, u32 *vp);
int kvmppc_xive_set_nr_servers(struct kvmppc_xive *xive, u64 addr);
bool kvmppc_xive_check_save_restore(struct kvm_vcpu *vcpu);
diff --git a/arch/powerpc/kvm/book3s_xive_native.c b/arch/powerpc/kvm/book3s_xive_native.c
index 5271c33..4f566be 100644
--- a/arch/powerpc/kvm/book3s_xive_native.c
+++ b/arch/powerpc/kvm/book3s_xive_native.c
@@ -93,8 +93,7 @@ void kvmppc_xive_native_cleanup_vcpu(struct kvm_vcpu *vcpu)
/* Free the escalation irq */
if (xc->esc_virq[i]) {
if (kvmppc_xive_has_single_escalation(xc->xive))
- xive_cleanup_single_escalation(vcpu, xc,
- xc->esc_virq[i]);
+ xive_cleanup_single_escalation(vcpu, xc->esc_virq[i]);
free_irq(xc->esc_virq[i], vcpu);
irq_dispose_mapping(xc->esc_virq[i]);
kfree(xc->esc_virq_names[i]);
diff --git a/arch/powerpc/kvm/booke.c b/arch/powerpc/kvm/booke.c
index 7b4920e..0dce93c 100644
--- a/arch/powerpc/kvm/booke.c
+++ b/arch/powerpc/kvm/booke.c
@@ -1015,6 +1015,9 @@ int kvmppc_handle_exit(struct kvm_vcpu *vcpu, unsigned int exit_nr)
u32 last_inst = KVM_INST_FETCH_FAILED;
enum emulation_result emulated = EMULATE_DONE;
+ /* Fix irq state (pairs with kvmppc_fix_ee_before_entry()) */
+ kvmppc_fix_ee_after_exit();
+
/* update before a new last_exit_type is rewritten */
kvmppc_update_timing_stats(vcpu);
diff --git a/arch/powerpc/kvm/bookehv_interrupts.S b/arch/powerpc/kvm/bookehv_interrupts.S
index 8262c14..b5fe6fb 100644
--- a/arch/powerpc/kvm/bookehv_interrupts.S
+++ b/arch/powerpc/kvm/bookehv_interrupts.S
@@ -424,15 +424,6 @@
mtspr SPRN_EPCR, r3
isync
-#ifdef CONFIG_64BIT
- /*
- * We enter with interrupts disabled in hardware, but
- * we need to call RECONCILE_IRQ_STATE to ensure
- * that the software state is kept in sync.
- */
- RECONCILE_IRQ_STATE(r3,r5)
-#endif
-
/* Switch to kernel stack and jump to handler. */
mr r3, r4
mr r5, r14 /* intno */
diff --git a/arch/powerpc/kvm/fpu.S b/arch/powerpc/kvm/fpu.S
index 315c949..b68e7f2 100644
--- a/arch/powerpc/kvm/fpu.S
+++ b/arch/powerpc/kvm/fpu.S
@@ -6,6 +6,8 @@
*/
#include <linux/pgtable.h>
+#include <linux/linkage.h>
+
#include <asm/reg.h>
#include <asm/page.h>
#include <asm/mmu.h>
@@ -110,18 +112,22 @@
* R8 = (double*)¶m3 [load_three]
* LR = instruction call function
*/
-fpd_load_three:
+SYM_FUNC_START_LOCAL(fpd_load_three)
lfd 2,0(r8) /* load param3 */
-fpd_load_two:
+SYM_FUNC_START_LOCAL(fpd_load_two)
lfd 1,0(r7) /* load param2 */
-fpd_load_one:
+SYM_FUNC_START_LOCAL(fpd_load_one)
lfd 0,0(r6) /* load param1 */
-fpd_load_none:
+SYM_FUNC_START_LOCAL(fpd_load_none)
lfd 3,0(r3) /* load up fpscr value */
MTFSF_L(3)
lwz r6, 0(r4) /* load cr */
mtcr r6
blr
+SYM_FUNC_END(fpd_load_none)
+SYM_FUNC_END(fpd_load_one)
+SYM_FUNC_END(fpd_load_two)
+SYM_FUNC_END(fpd_load_three)
/*
* End of double instruction processing
@@ -131,13 +137,14 @@
* R5 = (double*)&result
* LR = caller of instruction call function
*/
-fpd_return:
+SYM_FUNC_START_LOCAL(fpd_return)
mfcr r6
stfd 0,0(r5) /* save result */
mffs 0
stfd 0,0(r3) /* save new fpscr value */
stw r6,0(r4) /* save new cr value */
blr
+SYM_FUNC_END(fpd_return)
/*
* Double operation with no input operand
diff --git a/arch/powerpc/lib/Makefile b/arch/powerpc/lib/Makefile
index 8560c91..4de71cb 100644
--- a/arch/powerpc/lib/Makefile
+++ b/arch/powerpc/lib/Makefile
@@ -52,7 +52,9 @@
obj64-y += copypage_64.o copyuser_64.o mem_64.o hweight_64.o \
memcpy_64.o copy_mc_64.o
-ifndef CONFIG_PPC_QUEUED_SPINLOCKS
+ifdef CONFIG_PPC_QUEUED_SPINLOCKS
+obj-$(CONFIG_SMP) += qspinlock.o
+else
obj64-$(CONFIG_SMP) += locks.o
endif
diff --git a/arch/powerpc/lib/code-patching.c b/arch/powerpc/lib/code-patching.c
index ad0cf31..b00112d 100644
--- a/arch/powerpc/lib/code-patching.c
+++ b/arch/powerpc/lib/code-patching.c
@@ -4,12 +4,17 @@
*/
#include <linux/kprobes.h>
+#include <linux/mmu_context.h>
+#include <linux/random.h>
#include <linux/vmalloc.h>
#include <linux/init.h>
#include <linux/cpuhotplug.h>
#include <linux/uaccess.h>
#include <linux/jump_label.h>
+#include <asm/debug.h>
+#include <asm/pgalloc.h>
+#include <asm/tlb.h>
#include <asm/tlbflush.h>
#include <asm/page.h>
#include <asm/code-patching.h>
@@ -41,12 +46,59 @@ int raw_patch_instruction(u32 *addr, ppc_inst_t instr)
return __patch_instruction(addr, instr, addr);
}
-#ifdef CONFIG_STRICT_KERNEL_RWX
-static DEFINE_PER_CPU(struct vm_struct *, text_poke_area);
+struct patch_context {
+ union {
+ struct vm_struct *area;
+ struct mm_struct *mm;
+ };
+ unsigned long addr;
+ pte_t *pte;
+};
+
+static DEFINE_PER_CPU(struct patch_context, cpu_patching_context);
static int map_patch_area(void *addr, unsigned long text_poke_addr);
static void unmap_patch_area(unsigned long addr);
+static bool mm_patch_enabled(void)
+{
+ return IS_ENABLED(CONFIG_SMP) && radix_enabled();
+}
+
+/*
+ * The following applies for Radix MMU. Hash MMU has different requirements,
+ * and so is not supported.
+ *
+ * Changing mm requires context synchronising instructions on both sides of
+ * the context switch, as well as a hwsync between the last instruction for
+ * which the address of an associated storage access was translated using
+ * the current context.
+ *
+ * switch_mm_irqs_off() performs an isync after the context switch. It is
+ * the responsibility of the caller to perform the CSI and hwsync before
+ * starting/stopping the temp mm.
+ */
+static struct mm_struct *start_using_temp_mm(struct mm_struct *temp_mm)
+{
+ struct mm_struct *orig_mm = current->active_mm;
+
+ lockdep_assert_irqs_disabled();
+ switch_mm_irqs_off(orig_mm, temp_mm, current);
+
+ WARN_ON(!mm_is_thread_local(temp_mm));
+
+ suspend_breakpoints();
+ return orig_mm;
+}
+
+static void stop_using_temp_mm(struct mm_struct *temp_mm,
+ struct mm_struct *orig_mm)
+{
+ lockdep_assert_irqs_disabled();
+ switch_mm_irqs_off(temp_mm, orig_mm, current);
+ restore_breakpoints();
+}
+
static int text_area_cpu_up(unsigned int cpu)
{
struct vm_struct *area;
@@ -68,29 +120,108 @@ static int text_area_cpu_up(unsigned int cpu)
unmap_patch_area(addr);
- this_cpu_write(text_poke_area, area);
+ this_cpu_write(cpu_patching_context.area, area);
+ this_cpu_write(cpu_patching_context.addr, addr);
+ this_cpu_write(cpu_patching_context.pte, virt_to_kpte(addr));
return 0;
}
static int text_area_cpu_down(unsigned int cpu)
{
- free_vm_area(this_cpu_read(text_poke_area));
+ free_vm_area(this_cpu_read(cpu_patching_context.area));
+ this_cpu_write(cpu_patching_context.area, NULL);
+ this_cpu_write(cpu_patching_context.addr, 0);
+ this_cpu_write(cpu_patching_context.pte, NULL);
+ return 0;
+}
+
+static void put_patching_mm(struct mm_struct *mm, unsigned long patching_addr)
+{
+ struct mmu_gather tlb;
+
+ tlb_gather_mmu(&tlb, mm);
+ free_pgd_range(&tlb, patching_addr, patching_addr + PAGE_SIZE, 0, 0);
+ mmput(mm);
+}
+
+static int text_area_cpu_up_mm(unsigned int cpu)
+{
+ struct mm_struct *mm;
+ unsigned long addr;
+ pte_t *pte;
+ spinlock_t *ptl;
+
+ mm = mm_alloc();
+ if (WARN_ON(!mm))
+ goto fail_no_mm;
+
+ /*
+ * Choose a random page-aligned address from the interval
+ * [PAGE_SIZE .. DEFAULT_MAP_WINDOW - PAGE_SIZE].
+ * The lower address bound is PAGE_SIZE to avoid the zero-page.
+ */
+ addr = (1 + (get_random_long() % (DEFAULT_MAP_WINDOW / PAGE_SIZE - 2))) << PAGE_SHIFT;
+
+ /*
+ * PTE allocation uses GFP_KERNEL which means we need to
+ * pre-allocate the PTE here because we cannot do the
+ * allocation during patching when IRQs are disabled.
+ *
+ * Using get_locked_pte() to avoid open coding, the lock
+ * is unnecessary.
+ */
+ pte = get_locked_pte(mm, addr, &ptl);
+ if (!pte)
+ goto fail_no_pte;
+ pte_unmap_unlock(pte, ptl);
+
+ this_cpu_write(cpu_patching_context.mm, mm);
+ this_cpu_write(cpu_patching_context.addr, addr);
+
+ return 0;
+
+fail_no_pte:
+ put_patching_mm(mm, addr);
+fail_no_mm:
+ return -ENOMEM;
+}
+
+static int text_area_cpu_down_mm(unsigned int cpu)
+{
+ put_patching_mm(this_cpu_read(cpu_patching_context.mm),
+ this_cpu_read(cpu_patching_context.addr));
+
+ this_cpu_write(cpu_patching_context.mm, NULL);
+ this_cpu_write(cpu_patching_context.addr, 0);
+
return 0;
}
static __ro_after_init DEFINE_STATIC_KEY_FALSE(poking_init_done);
-/*
- * Although BUG_ON() is rude, in this case it should only happen if ENOMEM, and
- * we judge it as being preferable to a kernel that will crash later when
- * someone tries to use patch_instruction().
- */
void __init poking_init(void)
{
- BUG_ON(!cpuhp_setup_state(CPUHP_AP_ONLINE_DYN,
- "powerpc/text_poke:online", text_area_cpu_up,
- text_area_cpu_down));
+ int ret;
+
+ if (!IS_ENABLED(CONFIG_STRICT_KERNEL_RWX))
+ return;
+
+ if (mm_patch_enabled())
+ ret = cpuhp_setup_state(CPUHP_AP_ONLINE_DYN,
+ "powerpc/text_poke_mm:online",
+ text_area_cpu_up_mm,
+ text_area_cpu_down_mm);
+ else
+ ret = cpuhp_setup_state(CPUHP_AP_ONLINE_DYN,
+ "powerpc/text_poke:online",
+ text_area_cpu_up,
+ text_area_cpu_down);
+
+ /* cpuhp_setup_state returns >= 0 on success */
+ if (WARN_ON(ret < 0))
+ return;
+
static_branch_enable(&poking_init_done);
}
@@ -147,6 +278,56 @@ static void unmap_patch_area(unsigned long addr)
flush_tlb_kernel_range(addr, addr + PAGE_SIZE);
}
+static int __do_patch_instruction_mm(u32 *addr, ppc_inst_t instr)
+{
+ int err;
+ u32 *patch_addr;
+ unsigned long text_poke_addr;
+ pte_t *pte;
+ unsigned long pfn = get_patch_pfn(addr);
+ struct mm_struct *patching_mm;
+ struct mm_struct *orig_mm;
+ spinlock_t *ptl;
+
+ patching_mm = __this_cpu_read(cpu_patching_context.mm);
+ text_poke_addr = __this_cpu_read(cpu_patching_context.addr);
+ patch_addr = (u32 *)(text_poke_addr + offset_in_page(addr));
+
+ pte = get_locked_pte(patching_mm, text_poke_addr, &ptl);
+ if (!pte)
+ return -ENOMEM;
+
+ __set_pte_at(patching_mm, text_poke_addr, pte, pfn_pte(pfn, PAGE_KERNEL), 0);
+
+ /* order PTE update before use, also serves as the hwsync */
+ asm volatile("ptesync": : :"memory");
+
+ /* order context switch after arbitrary prior code */
+ isync();
+
+ orig_mm = start_using_temp_mm(patching_mm);
+
+ err = __patch_instruction(addr, instr, patch_addr);
+
+ /* hwsync performed by __patch_instruction (sync) if successful */
+ if (err)
+ mb(); /* sync */
+
+ /* context synchronisation performed by __patch_instruction (isync or exception) */
+ stop_using_temp_mm(patching_mm, orig_mm);
+
+ pte_clear(patching_mm, text_poke_addr, pte);
+ /*
+ * ptesync to order PTE update before TLB invalidation done
+ * by radix__local_flush_tlb_page_psize (in _tlbiel_va)
+ */
+ local_flush_tlb_page_psize(patching_mm, text_poke_addr, mmu_virtual_psize);
+
+ pte_unmap_unlock(pte, ptl);
+
+ return err;
+}
+
static int __do_patch_instruction(u32 *addr, ppc_inst_t instr)
{
int err;
@@ -155,10 +336,10 @@ static int __do_patch_instruction(u32 *addr, ppc_inst_t instr)
pte_t *pte;
unsigned long pfn = get_patch_pfn(addr);
- text_poke_addr = (unsigned long)__this_cpu_read(text_poke_area)->addr & PAGE_MASK;
+ text_poke_addr = (unsigned long)__this_cpu_read(cpu_patching_context.addr) & PAGE_MASK;
patch_addr = (u32 *)(text_poke_addr + offset_in_page(addr));
- pte = virt_to_kpte(text_poke_addr);
+ pte = __this_cpu_read(cpu_patching_context.pte);
__set_pte_at(&init_mm, text_poke_addr, pte, pfn_pte(pfn, PAGE_KERNEL), 0);
/* See ptesync comment in radix__set_pte_at() */
if (radix_enabled())
@@ -172,7 +353,7 @@ static int __do_patch_instruction(u32 *addr, ppc_inst_t instr)
return err;
}
-static int do_patch_instruction(u32 *addr, ppc_inst_t instr)
+int patch_instruction(u32 *addr, ppc_inst_t instr)
{
int err;
unsigned long flags;
@@ -182,34 +363,19 @@ static int do_patch_instruction(u32 *addr, ppc_inst_t instr)
* when text_poke_area is not ready, but we still need
* to allow patching. We just do the plain old patching
*/
- if (!static_branch_likely(&poking_init_done))
+ if (!IS_ENABLED(CONFIG_STRICT_KERNEL_RWX) ||
+ !static_branch_likely(&poking_init_done))
return raw_patch_instruction(addr, instr);
local_irq_save(flags);
- err = __do_patch_instruction(addr, instr);
+ if (mm_patch_enabled())
+ err = __do_patch_instruction_mm(addr, instr);
+ else
+ err = __do_patch_instruction(addr, instr);
local_irq_restore(flags);
return err;
}
-#else /* !CONFIG_STRICT_KERNEL_RWX */
-
-static int do_patch_instruction(u32 *addr, ppc_inst_t instr)
-{
- return raw_patch_instruction(addr, instr);
-}
-
-#endif /* CONFIG_STRICT_KERNEL_RWX */
-
-__ro_after_init DEFINE_STATIC_KEY_FALSE(init_mem_is_free);
-
-int patch_instruction(u32 *addr, ppc_inst_t instr)
-{
- /* Make sure we aren't patching a freed init section */
- if (static_branch_likely(&init_mem_is_free) && init_section_contains(addr, 4))
- return 0;
-
- return do_patch_instruction(addr, instr);
-}
NOKPROBE_SYMBOL(patch_instruction);
int patch_branch(u32 *addr, unsigned long target, int flags)
diff --git a/arch/powerpc/lib/feature-fixups.c b/arch/powerpc/lib/feature-fixups.c
index 31f40f5..80def1c 100644
--- a/arch/powerpc/lib/feature-fixups.c
+++ b/arch/powerpc/lib/feature-fixups.c
@@ -117,10 +117,64 @@ void do_feature_fixups(unsigned long value, void *fixup_start, void *fixup_end)
}
}
+#ifdef CONFIG_PPC_BARRIER_NOSPEC
+static bool is_fixup_addr_valid(void *dest, size_t size)
+{
+ return system_state < SYSTEM_FREEING_INITMEM ||
+ !init_section_contains(dest, size);
+}
+
+static int do_patch_fixups(long *start, long *end, unsigned int *instrs, int num)
+{
+ int i;
+
+ for (i = 0; start < end; start++, i++) {
+ int j;
+ unsigned int *dest = (void *)start + *start;
+
+ if (!is_fixup_addr_valid(dest, sizeof(*instrs) * num))
+ continue;
+
+ pr_devel("patching dest %lx\n", (unsigned long)dest);
+
+ for (j = 0; j < num; j++)
+ patch_instruction(dest + j, ppc_inst(instrs[j]));
+ }
+ return i;
+}
+#endif
+
#ifdef CONFIG_PPC_BOOK3S_64
+static int do_patch_entry_fixups(long *start, long *end, unsigned int *instrs,
+ bool do_fallback, void *fallback)
+{
+ int i;
+
+ for (i = 0; start < end; start++, i++) {
+ unsigned int *dest = (void *)start + *start;
+
+ if (!is_fixup_addr_valid(dest, sizeof(*instrs) * 3))
+ continue;
+
+ pr_devel("patching dest %lx\n", (unsigned long)dest);
+
+ // See comment in do_entry_flush_fixups() RE order of patching
+ if (do_fallback) {
+ patch_instruction(dest, ppc_inst(instrs[0]));
+ patch_instruction(dest + 2, ppc_inst(instrs[2]));
+ patch_branch(dest + 1, (unsigned long)fallback, BRANCH_SET_LINK);
+ } else {
+ patch_instruction(dest + 1, ppc_inst(instrs[1]));
+ patch_instruction(dest + 2, ppc_inst(instrs[2]));
+ patch_instruction(dest, ppc_inst(instrs[0]));
+ }
+ }
+ return i;
+}
+
static void do_stf_entry_barrier_fixups(enum stf_barrier_type types)
{
- unsigned int instrs[3], *dest;
+ unsigned int instrs[3];
long *start, *end;
int i;
@@ -144,23 +198,8 @@ static void do_stf_entry_barrier_fixups(enum stf_barrier_type types)
instrs[i++] = PPC_RAW_ORI(_R31, _R31, 0); /* speculation barrier */
}
- for (i = 0; start < end; start++, i++) {
- dest = (void *)start + *start;
-
- pr_devel("patching dest %lx\n", (unsigned long)dest);
-
- // See comment in do_entry_flush_fixups() RE order of patching
- if (types & STF_BARRIER_FALLBACK) {
- patch_instruction(dest, ppc_inst(instrs[0]));
- patch_instruction(dest + 2, ppc_inst(instrs[2]));
- patch_branch(dest + 1,
- (unsigned long)&stf_barrier_fallback, BRANCH_SET_LINK);
- } else {
- patch_instruction(dest + 1, ppc_inst(instrs[1]));
- patch_instruction(dest + 2, ppc_inst(instrs[2]));
- patch_instruction(dest, ppc_inst(instrs[0]));
- }
- }
+ i = do_patch_entry_fixups(start, end, instrs, types & STF_BARRIER_FALLBACK,
+ &stf_barrier_fallback);
printk(KERN_DEBUG "stf-barrier: patched %d entry locations (%s barrier)\n", i,
(types == STF_BARRIER_NONE) ? "no" :
@@ -172,7 +211,7 @@ static void do_stf_entry_barrier_fixups(enum stf_barrier_type types)
static void do_stf_exit_barrier_fixups(enum stf_barrier_type types)
{
- unsigned int instrs[6], *dest;
+ unsigned int instrs[6];
long *start, *end;
int i;
@@ -206,18 +245,8 @@ static void do_stf_exit_barrier_fixups(enum stf_barrier_type types)
instrs[i++] = PPC_RAW_EIEIO() | 0x02000000; /* eieio + bit 6 hint */
}
- for (i = 0; start < end; start++, i++) {
- dest = (void *)start + *start;
+ i = do_patch_fixups(start, end, instrs, ARRAY_SIZE(instrs));
- pr_devel("patching dest %lx\n", (unsigned long)dest);
-
- patch_instruction(dest, ppc_inst(instrs[0]));
- patch_instruction(dest + 1, ppc_inst(instrs[1]));
- patch_instruction(dest + 2, ppc_inst(instrs[2]));
- patch_instruction(dest + 3, ppc_inst(instrs[3]));
- patch_instruction(dest + 4, ppc_inst(instrs[4]));
- patch_instruction(dest + 5, ppc_inst(instrs[5]));
- }
printk(KERN_DEBUG "stf-barrier: patched %d exit locations (%s barrier)\n", i,
(types == STF_BARRIER_NONE) ? "no" :
(types == STF_BARRIER_FALLBACK) ? "fallback" :
@@ -274,7 +303,7 @@ void do_stf_barrier_fixups(enum stf_barrier_type types)
void do_uaccess_flush_fixups(enum l1d_flush_type types)
{
- unsigned int instrs[4], *dest;
+ unsigned int instrs[4];
long *start, *end;
int i;
@@ -300,17 +329,7 @@ void do_uaccess_flush_fixups(enum l1d_flush_type types)
if (types & L1D_FLUSH_MTTRIG)
instrs[i++] = PPC_RAW_MTSPR(SPRN_TRIG2, _R0);
- for (i = 0; start < end; start++, i++) {
- dest = (void *)start + *start;
-
- pr_devel("patching dest %lx\n", (unsigned long)dest);
-
- patch_instruction(dest, ppc_inst(instrs[0]));
-
- patch_instruction(dest + 1, ppc_inst(instrs[1]));
- patch_instruction(dest + 2, ppc_inst(instrs[2]));
- patch_instruction(dest + 3, ppc_inst(instrs[3]));
- }
+ i = do_patch_fixups(start, end, instrs, ARRAY_SIZE(instrs));
printk(KERN_DEBUG "uaccess-flush: patched %d locations (%s flush)\n", i,
(types == L1D_FLUSH_NONE) ? "no" :
@@ -325,7 +344,7 @@ void do_uaccess_flush_fixups(enum l1d_flush_type types)
static int __do_entry_flush_fixups(void *data)
{
enum l1d_flush_type types = *(enum l1d_flush_type *)data;
- unsigned int instrs[3], *dest;
+ unsigned int instrs[3];
long *start, *end;
int i;
@@ -375,42 +394,13 @@ static int __do_entry_flush_fixups(void *data)
start = PTRRELOC(&__start___entry_flush_fixup);
end = PTRRELOC(&__stop___entry_flush_fixup);
- for (i = 0; start < end; start++, i++) {
- dest = (void *)start + *start;
-
- pr_devel("patching dest %lx\n", (unsigned long)dest);
-
- if (types == L1D_FLUSH_FALLBACK) {
- patch_instruction(dest, ppc_inst(instrs[0]));
- patch_instruction(dest + 2, ppc_inst(instrs[2]));
- patch_branch(dest + 1,
- (unsigned long)&entry_flush_fallback, BRANCH_SET_LINK);
- } else {
- patch_instruction(dest + 1, ppc_inst(instrs[1]));
- patch_instruction(dest + 2, ppc_inst(instrs[2]));
- patch_instruction(dest, ppc_inst(instrs[0]));
- }
- }
+ i = do_patch_entry_fixups(start, end, instrs, types == L1D_FLUSH_FALLBACK,
+ &entry_flush_fallback);
start = PTRRELOC(&__start___scv_entry_flush_fixup);
end = PTRRELOC(&__stop___scv_entry_flush_fixup);
- for (; start < end; start++, i++) {
- dest = (void *)start + *start;
-
- pr_devel("patching dest %lx\n", (unsigned long)dest);
-
- if (types == L1D_FLUSH_FALLBACK) {
- patch_instruction(dest, ppc_inst(instrs[0]));
- patch_instruction(dest + 2, ppc_inst(instrs[2]));
- patch_branch(dest + 1,
- (unsigned long)&scv_entry_flush_fallback, BRANCH_SET_LINK);
- } else {
- patch_instruction(dest + 1, ppc_inst(instrs[1]));
- patch_instruction(dest + 2, ppc_inst(instrs[2]));
- patch_instruction(dest, ppc_inst(instrs[0]));
- }
- }
-
+ i += do_patch_entry_fixups(start, end, instrs, types == L1D_FLUSH_FALLBACK,
+ &scv_entry_flush_fallback);
printk(KERN_DEBUG "entry-flush: patched %d locations (%s flush)\n", i,
(types == L1D_FLUSH_NONE) ? "no" :
@@ -438,7 +428,7 @@ void do_entry_flush_fixups(enum l1d_flush_type types)
static int __do_rfi_flush_fixups(void *data)
{
enum l1d_flush_type types = *(enum l1d_flush_type *)data;
- unsigned int instrs[3], *dest;
+ unsigned int instrs[3];
long *start, *end;
int i;
@@ -462,15 +452,7 @@ static int __do_rfi_flush_fixups(void *data)
if (types & L1D_FLUSH_MTTRIG)
instrs[i++] = PPC_RAW_MTSPR(SPRN_TRIG2, _R0);
- for (i = 0; start < end; start++, i++) {
- dest = (void *)start + *start;
-
- pr_devel("patching dest %lx\n", (unsigned long)dest);
-
- patch_instruction(dest, ppc_inst(instrs[0]));
- patch_instruction(dest + 1, ppc_inst(instrs[1]));
- patch_instruction(dest + 2, ppc_inst(instrs[2]));
- }
+ i = do_patch_fixups(start, end, instrs, ARRAY_SIZE(instrs));
printk(KERN_DEBUG "rfi-flush: patched %d locations (%s flush)\n", i,
(types == L1D_FLUSH_NONE) ? "no" :
@@ -512,7 +494,7 @@ void do_rfi_flush_fixups(enum l1d_flush_type types)
void do_barrier_nospec_fixups_range(bool enable, void *fixup_start, void *fixup_end)
{
- unsigned int instr, *dest;
+ unsigned int instr;
long *start, *end;
int i;
@@ -526,12 +508,7 @@ void do_barrier_nospec_fixups_range(bool enable, void *fixup_start, void *fixup_
instr = PPC_RAW_ORI(_R31, _R31, 0); /* speculation barrier */
}
- for (i = 0; start < end; start++, i++) {
- dest = (void *)start + *start;
-
- pr_devel("patching dest %lx\n", (unsigned long)dest);
- patch_instruction(dest, ppc_inst(instr));
- }
+ i = do_patch_fixups(start, end, &instr, 1);
printk(KERN_DEBUG "barrier-nospec: patched %d locations\n", i);
}
@@ -553,7 +530,7 @@ void do_barrier_nospec_fixups(bool enable)
#ifdef CONFIG_PPC_E500
void do_barrier_nospec_fixups_range(bool enable, void *fixup_start, void *fixup_end)
{
- unsigned int instr[2], *dest;
+ unsigned int instr[2];
long *start, *end;
int i;
@@ -569,13 +546,7 @@ void do_barrier_nospec_fixups_range(bool enable, void *fixup_start, void *fixup_
instr[1] = PPC_RAW_SYNC();
}
- for (i = 0; start < end; start++, i++) {
- dest = (void *)start + *start;
-
- pr_devel("patching dest %lx\n", (unsigned long)dest);
- patch_instruction(dest, ppc_inst(instr[0]));
- patch_instruction(dest + 1, ppc_inst(instr[1]));
- }
+ i = do_patch_fixups(start, end, instr, ARRAY_SIZE(instr));
printk(KERN_DEBUG "barrier-nospec: patched %d locations\n", i);
}
diff --git a/arch/powerpc/lib/qspinlock.c b/arch/powerpc/lib/qspinlock.c
new file mode 100644
index 0000000..e4bd145
--- /dev/null
+++ b/arch/powerpc/lib/qspinlock.c
@@ -0,0 +1,997 @@
+// SPDX-License-Identifier: GPL-2.0-or-later
+#include <linux/bug.h>
+#include <linux/compiler.h>
+#include <linux/export.h>
+#include <linux/percpu.h>
+#include <linux/processor.h>
+#include <linux/smp.h>
+#include <linux/topology.h>
+#include <linux/sched/clock.h>
+#include <asm/qspinlock.h>
+#include <asm/paravirt.h>
+
+#define MAX_NODES 4
+
+struct qnode {
+ struct qnode *next;
+ struct qspinlock *lock;
+ int cpu;
+ int yield_cpu;
+ u8 locked; /* 1 if lock acquired */
+};
+
+struct qnodes {
+ int count;
+ struct qnode nodes[MAX_NODES];
+};
+
+/* Tuning parameters */
+static int steal_spins __read_mostly = (1 << 5);
+static int remote_steal_spins __read_mostly = (1 << 2);
+#if _Q_SPIN_TRY_LOCK_STEAL == 1
+static const bool maybe_stealers = true;
+#else
+static bool maybe_stealers __read_mostly = true;
+#endif
+static int head_spins __read_mostly = (1 << 8);
+
+static bool pv_yield_owner __read_mostly = true;
+static bool pv_yield_allow_steal __read_mostly = false;
+static bool pv_spin_on_preempted_owner __read_mostly = false;
+static bool pv_sleepy_lock __read_mostly = true;
+static bool pv_sleepy_lock_sticky __read_mostly = false;
+static u64 pv_sleepy_lock_interval_ns __read_mostly = 0;
+static int pv_sleepy_lock_factor __read_mostly = 256;
+static bool pv_yield_prev __read_mostly = true;
+static bool pv_yield_propagate_owner __read_mostly = true;
+static bool pv_prod_head __read_mostly = false;
+
+static DEFINE_PER_CPU_ALIGNED(struct qnodes, qnodes);
+static DEFINE_PER_CPU_ALIGNED(u64, sleepy_lock_seen_clock);
+
+#if _Q_SPIN_SPEC_BARRIER == 1
+#define spec_barrier() do { asm volatile("ori 31,31,0" ::: "memory"); } while (0)
+#else
+#define spec_barrier() do { } while (0)
+#endif
+
+static __always_inline bool recently_sleepy(void)
+{
+ /* pv_sleepy_lock is true when this is called */
+ if (pv_sleepy_lock_interval_ns) {
+ u64 seen = this_cpu_read(sleepy_lock_seen_clock);
+
+ if (seen) {
+ u64 delta = sched_clock() - seen;
+ if (delta < pv_sleepy_lock_interval_ns)
+ return true;
+ this_cpu_write(sleepy_lock_seen_clock, 0);
+ }
+ }
+
+ return false;
+}
+
+static __always_inline int get_steal_spins(bool paravirt, bool sleepy)
+{
+ if (paravirt && sleepy)
+ return steal_spins * pv_sleepy_lock_factor;
+ else
+ return steal_spins;
+}
+
+static __always_inline int get_remote_steal_spins(bool paravirt, bool sleepy)
+{
+ if (paravirt && sleepy)
+ return remote_steal_spins * pv_sleepy_lock_factor;
+ else
+ return remote_steal_spins;
+}
+
+static __always_inline int get_head_spins(bool paravirt, bool sleepy)
+{
+ if (paravirt && sleepy)
+ return head_spins * pv_sleepy_lock_factor;
+ else
+ return head_spins;
+}
+
+static inline u32 encode_tail_cpu(int cpu)
+{
+ return (cpu + 1) << _Q_TAIL_CPU_OFFSET;
+}
+
+static inline int decode_tail_cpu(u32 val)
+{
+ return (val >> _Q_TAIL_CPU_OFFSET) - 1;
+}
+
+static inline int get_owner_cpu(u32 val)
+{
+ return (val & _Q_OWNER_CPU_MASK) >> _Q_OWNER_CPU_OFFSET;
+}
+
+/*
+ * Try to acquire the lock if it was not already locked. If the tail matches
+ * mytail then clear it, otherwise leave it unchnaged. Return previous value.
+ *
+ * This is used by the head of the queue to acquire the lock and clean up
+ * its tail if it was the last one queued.
+ */
+static __always_inline u32 trylock_clean_tail(struct qspinlock *lock, u32 tail)
+{
+ u32 newval = queued_spin_encode_locked_val();
+ u32 prev, tmp;
+
+ asm volatile(
+"1: lwarx %0,0,%2,%7 # trylock_clean_tail \n"
+ /* This test is necessary if there could be stealers */
+" andi. %1,%0,%5 \n"
+" bne 3f \n"
+ /* Test whether the lock tail == mytail */
+" and %1,%0,%6 \n"
+" cmpw 0,%1,%3 \n"
+ /* Merge the new locked value */
+" or %1,%1,%4 \n"
+" bne 2f \n"
+ /* If the lock tail matched, then clear it, otherwise leave it. */
+" andc %1,%1,%6 \n"
+"2: stwcx. %1,0,%2 \n"
+" bne- 1b \n"
+"\t" PPC_ACQUIRE_BARRIER " \n"
+"3: \n"
+ : "=&r" (prev), "=&r" (tmp)
+ : "r" (&lock->val), "r"(tail), "r" (newval),
+ "i" (_Q_LOCKED_VAL),
+ "r" (_Q_TAIL_CPU_MASK),
+ "i" (_Q_SPIN_EH_HINT)
+ : "cr0", "memory");
+
+ return prev;
+}
+
+/*
+ * Publish our tail, replacing previous tail. Return previous value.
+ *
+ * This provides a release barrier for publishing node, this pairs with the
+ * acquire barrier in get_tail_qnode() when the next CPU finds this tail
+ * value.
+ */
+static __always_inline u32 publish_tail_cpu(struct qspinlock *lock, u32 tail)
+{
+ u32 prev, tmp;
+
+ asm volatile(
+"\t" PPC_RELEASE_BARRIER " \n"
+"1: lwarx %0,0,%2 # publish_tail_cpu \n"
+" andc %1,%0,%4 \n"
+" or %1,%1,%3 \n"
+" stwcx. %1,0,%2 \n"
+" bne- 1b \n"
+ : "=&r" (prev), "=&r"(tmp)
+ : "r" (&lock->val), "r" (tail), "r"(_Q_TAIL_CPU_MASK)
+ : "cr0", "memory");
+
+ return prev;
+}
+
+static __always_inline u32 set_mustq(struct qspinlock *lock)
+{
+ u32 prev;
+
+ asm volatile(
+"1: lwarx %0,0,%1 # set_mustq \n"
+" or %0,%0,%2 \n"
+" stwcx. %0,0,%1 \n"
+" bne- 1b \n"
+ : "=&r" (prev)
+ : "r" (&lock->val), "r" (_Q_MUST_Q_VAL)
+ : "cr0", "memory");
+
+ return prev;
+}
+
+static __always_inline u32 clear_mustq(struct qspinlock *lock)
+{
+ u32 prev;
+
+ asm volatile(
+"1: lwarx %0,0,%1 # clear_mustq \n"
+" andc %0,%0,%2 \n"
+" stwcx. %0,0,%1 \n"
+" bne- 1b \n"
+ : "=&r" (prev)
+ : "r" (&lock->val), "r" (_Q_MUST_Q_VAL)
+ : "cr0", "memory");
+
+ return prev;
+}
+
+static __always_inline bool try_set_sleepy(struct qspinlock *lock, u32 old)
+{
+ u32 prev;
+ u32 new = old | _Q_SLEEPY_VAL;
+
+ BUG_ON(!(old & _Q_LOCKED_VAL));
+ BUG_ON(old & _Q_SLEEPY_VAL);
+
+ asm volatile(
+"1: lwarx %0,0,%1 # try_set_sleepy \n"
+" cmpw 0,%0,%2 \n"
+" bne- 2f \n"
+" stwcx. %3,0,%1 \n"
+" bne- 1b \n"
+"2: \n"
+ : "=&r" (prev)
+ : "r" (&lock->val), "r"(old), "r" (new)
+ : "cr0", "memory");
+
+ return likely(prev == old);
+}
+
+static __always_inline void seen_sleepy_owner(struct qspinlock *lock, u32 val)
+{
+ if (pv_sleepy_lock) {
+ if (pv_sleepy_lock_interval_ns)
+ this_cpu_write(sleepy_lock_seen_clock, sched_clock());
+ if (!(val & _Q_SLEEPY_VAL))
+ try_set_sleepy(lock, val);
+ }
+}
+
+static __always_inline void seen_sleepy_lock(void)
+{
+ if (pv_sleepy_lock && pv_sleepy_lock_interval_ns)
+ this_cpu_write(sleepy_lock_seen_clock, sched_clock());
+}
+
+static __always_inline void seen_sleepy_node(struct qspinlock *lock, u32 val)
+{
+ if (pv_sleepy_lock) {
+ if (pv_sleepy_lock_interval_ns)
+ this_cpu_write(sleepy_lock_seen_clock, sched_clock());
+ if (val & _Q_LOCKED_VAL) {
+ if (!(val & _Q_SLEEPY_VAL))
+ try_set_sleepy(lock, val);
+ }
+ }
+}
+
+static struct qnode *get_tail_qnode(struct qspinlock *lock, u32 val)
+{
+ int cpu = decode_tail_cpu(val);
+ struct qnodes *qnodesp = per_cpu_ptr(&qnodes, cpu);
+ int idx;
+
+ /*
+ * After publishing the new tail and finding a previous tail in the
+ * previous val (which is the control dependency), this barrier
+ * orders the release barrier in publish_tail_cpu performed by the
+ * last CPU, with subsequently looking at its qnode structures
+ * after the barrier.
+ */
+ smp_acquire__after_ctrl_dep();
+
+ for (idx = 0; idx < MAX_NODES; idx++) {
+ struct qnode *qnode = &qnodesp->nodes[idx];
+ if (qnode->lock == lock)
+ return qnode;
+ }
+
+ BUG();
+}
+
+/* Called inside spin_begin(). Returns whether or not the vCPU was preempted. */
+static __always_inline bool __yield_to_locked_owner(struct qspinlock *lock, u32 val, bool paravirt, bool mustq)
+{
+ int owner;
+ u32 yield_count;
+ bool preempted = false;
+
+ BUG_ON(!(val & _Q_LOCKED_VAL));
+
+ if (!paravirt)
+ goto relax;
+
+ if (!pv_yield_owner)
+ goto relax;
+
+ owner = get_owner_cpu(val);
+ yield_count = yield_count_of(owner);
+
+ if ((yield_count & 1) == 0)
+ goto relax; /* owner vcpu is running */
+
+ spin_end();
+
+ seen_sleepy_owner(lock, val);
+ preempted = true;
+
+ /*
+ * Read the lock word after sampling the yield count. On the other side
+ * there may a wmb because the yield count update is done by the
+ * hypervisor preemption and the value update by the OS, however this
+ * ordering might reduce the chance of out of order accesses and
+ * improve the heuristic.
+ */
+ smp_rmb();
+
+ if (READ_ONCE(lock->val) == val) {
+ if (mustq)
+ clear_mustq(lock);
+ yield_to_preempted(owner, yield_count);
+ if (mustq)
+ set_mustq(lock);
+ spin_begin();
+
+ /* Don't relax if we yielded. Maybe we should? */
+ return preempted;
+ }
+ spin_begin();
+relax:
+ spin_cpu_relax();
+
+ return preempted;
+}
+
+/* Called inside spin_begin(). Returns whether or not the vCPU was preempted. */
+static __always_inline bool yield_to_locked_owner(struct qspinlock *lock, u32 val, bool paravirt)
+{
+ return __yield_to_locked_owner(lock, val, paravirt, false);
+}
+
+/* Called inside spin_begin(). Returns whether or not the vCPU was preempted. */
+static __always_inline bool yield_head_to_locked_owner(struct qspinlock *lock, u32 val, bool paravirt)
+{
+ bool mustq = false;
+
+ if ((val & _Q_MUST_Q_VAL) && pv_yield_allow_steal)
+ mustq = true;
+
+ return __yield_to_locked_owner(lock, val, paravirt, mustq);
+}
+
+static __always_inline void propagate_yield_cpu(struct qnode *node, u32 val, int *set_yield_cpu, bool paravirt)
+{
+ struct qnode *next;
+ int owner;
+
+ if (!paravirt)
+ return;
+ if (!pv_yield_propagate_owner)
+ return;
+
+ owner = get_owner_cpu(val);
+ if (*set_yield_cpu == owner)
+ return;
+
+ next = READ_ONCE(node->next);
+ if (!next)
+ return;
+
+ if (vcpu_is_preempted(owner)) {
+ next->yield_cpu = owner;
+ *set_yield_cpu = owner;
+ } else if (*set_yield_cpu != -1) {
+ next->yield_cpu = owner;
+ *set_yield_cpu = owner;
+ }
+}
+
+/* Called inside spin_begin() */
+static __always_inline bool yield_to_prev(struct qspinlock *lock, struct qnode *node, u32 val, bool paravirt)
+{
+ int prev_cpu = decode_tail_cpu(val);
+ u32 yield_count;
+ int yield_cpu;
+ bool preempted = false;
+
+ if (!paravirt)
+ goto relax;
+
+ if (!pv_yield_propagate_owner)
+ goto yield_prev;
+
+ yield_cpu = READ_ONCE(node->yield_cpu);
+ if (yield_cpu == -1) {
+ /* Propagate back the -1 CPU */
+ if (node->next && node->next->yield_cpu != -1)
+ node->next->yield_cpu = yield_cpu;
+ goto yield_prev;
+ }
+
+ yield_count = yield_count_of(yield_cpu);
+ if ((yield_count & 1) == 0)
+ goto yield_prev; /* owner vcpu is running */
+
+ spin_end();
+
+ preempted = true;
+ seen_sleepy_node(lock, val);
+
+ smp_rmb();
+
+ if (yield_cpu == node->yield_cpu) {
+ if (node->next && node->next->yield_cpu != yield_cpu)
+ node->next->yield_cpu = yield_cpu;
+ yield_to_preempted(yield_cpu, yield_count);
+ spin_begin();
+ return preempted;
+ }
+ spin_begin();
+
+yield_prev:
+ if (!pv_yield_prev)
+ goto relax;
+
+ yield_count = yield_count_of(prev_cpu);
+ if ((yield_count & 1) == 0)
+ goto relax; /* owner vcpu is running */
+
+ spin_end();
+
+ preempted = true;
+ seen_sleepy_node(lock, val);
+
+ smp_rmb(); /* See __yield_to_locked_owner comment */
+
+ if (!node->locked) {
+ yield_to_preempted(prev_cpu, yield_count);
+ spin_begin();
+ return preempted;
+ }
+ spin_begin();
+
+relax:
+ spin_cpu_relax();
+
+ return preempted;
+}
+
+static __always_inline bool steal_break(u32 val, int iters, bool paravirt, bool sleepy)
+{
+ if (iters >= get_steal_spins(paravirt, sleepy))
+ return true;
+
+ if (IS_ENABLED(CONFIG_NUMA) &&
+ (iters >= get_remote_steal_spins(paravirt, sleepy))) {
+ int cpu = get_owner_cpu(val);
+ if (numa_node_id() != cpu_to_node(cpu))
+ return true;
+ }
+ return false;
+}
+
+static __always_inline bool try_to_steal_lock(struct qspinlock *lock, bool paravirt)
+{
+ bool seen_preempted = false;
+ bool sleepy = false;
+ int iters = 0;
+ u32 val;
+
+ if (!steal_spins) {
+ /* XXX: should spin_on_preempted_owner do anything here? */
+ return false;
+ }
+
+ /* Attempt to steal the lock */
+ spin_begin();
+ do {
+ bool preempted = false;
+
+ val = READ_ONCE(lock->val);
+ if (val & _Q_MUST_Q_VAL)
+ break;
+ spec_barrier();
+
+ if (unlikely(!(val & _Q_LOCKED_VAL))) {
+ spin_end();
+ if (__queued_spin_trylock_steal(lock))
+ return true;
+ spin_begin();
+ } else {
+ preempted = yield_to_locked_owner(lock, val, paravirt);
+ }
+
+ if (paravirt && pv_sleepy_lock) {
+ if (!sleepy) {
+ if (val & _Q_SLEEPY_VAL) {
+ seen_sleepy_lock();
+ sleepy = true;
+ } else if (recently_sleepy()) {
+ sleepy = true;
+ }
+ }
+ if (pv_sleepy_lock_sticky && seen_preempted &&
+ !(val & _Q_SLEEPY_VAL)) {
+ if (try_set_sleepy(lock, val))
+ val |= _Q_SLEEPY_VAL;
+ }
+ }
+
+ if (preempted) {
+ seen_preempted = true;
+ sleepy = true;
+ if (!pv_spin_on_preempted_owner)
+ iters++;
+ /*
+ * pv_spin_on_preempted_owner don't increase iters
+ * while the owner is preempted -- we won't interfere
+ * with it by definition. This could introduce some
+ * latency issue if we continually observe preempted
+ * owners, but hopefully that's a rare corner case of
+ * a badly oversubscribed system.
+ */
+ } else {
+ iters++;
+ }
+ } while (!steal_break(val, iters, paravirt, sleepy));
+
+ spin_end();
+
+ return false;
+}
+
+static __always_inline void queued_spin_lock_mcs_queue(struct qspinlock *lock, bool paravirt)
+{
+ struct qnodes *qnodesp;
+ struct qnode *next, *node;
+ u32 val, old, tail;
+ bool seen_preempted = false;
+ bool sleepy = false;
+ bool mustq = false;
+ int idx;
+ int set_yield_cpu = -1;
+ int iters = 0;
+
+ BUILD_BUG_ON(CONFIG_NR_CPUS >= (1U << _Q_TAIL_CPU_BITS));
+
+ qnodesp = this_cpu_ptr(&qnodes);
+ if (unlikely(qnodesp->count >= MAX_NODES)) {
+ spec_barrier();
+ while (!queued_spin_trylock(lock))
+ cpu_relax();
+ return;
+ }
+
+ idx = qnodesp->count++;
+ /*
+ * Ensure that we increment the head node->count before initialising
+ * the actual node. If the compiler is kind enough to reorder these
+ * stores, then an IRQ could overwrite our assignments.
+ */
+ barrier();
+ node = &qnodesp->nodes[idx];
+ node->next = NULL;
+ node->lock = lock;
+ node->cpu = smp_processor_id();
+ node->yield_cpu = -1;
+ node->locked = 0;
+
+ tail = encode_tail_cpu(node->cpu);
+
+ old = publish_tail_cpu(lock, tail);
+
+ /*
+ * If there was a previous node; link it and wait until reaching the
+ * head of the waitqueue.
+ */
+ if (old & _Q_TAIL_CPU_MASK) {
+ struct qnode *prev = get_tail_qnode(lock, old);
+
+ /* Link @node into the waitqueue. */
+ WRITE_ONCE(prev->next, node);
+
+ /* Wait for mcs node lock to be released */
+ spin_begin();
+ while (!node->locked) {
+ spec_barrier();
+
+ if (yield_to_prev(lock, node, old, paravirt))
+ seen_preempted = true;
+ }
+ spec_barrier();
+ spin_end();
+
+ /* Clear out stale propagated yield_cpu */
+ if (paravirt && pv_yield_propagate_owner && node->yield_cpu != -1)
+ node->yield_cpu = -1;
+
+ smp_rmb(); /* acquire barrier for the mcs lock */
+
+ /*
+ * Generic qspinlocks have this prefetch here, but it seems
+ * like it could cause additional line transitions because
+ * the waiter will keep loading from it.
+ */
+ if (_Q_SPIN_PREFETCH_NEXT) {
+ next = READ_ONCE(node->next);
+ if (next)
+ prefetchw(next);
+ }
+ }
+
+ /* We're at the head of the waitqueue, wait for the lock. */
+again:
+ spin_begin();
+ for (;;) {
+ bool preempted;
+
+ val = READ_ONCE(lock->val);
+ if (!(val & _Q_LOCKED_VAL))
+ break;
+ spec_barrier();
+
+ if (paravirt && pv_sleepy_lock && maybe_stealers) {
+ if (!sleepy) {
+ if (val & _Q_SLEEPY_VAL) {
+ seen_sleepy_lock();
+ sleepy = true;
+ } else if (recently_sleepy()) {
+ sleepy = true;
+ }
+ }
+ if (pv_sleepy_lock_sticky && seen_preempted &&
+ !(val & _Q_SLEEPY_VAL)) {
+ if (try_set_sleepy(lock, val))
+ val |= _Q_SLEEPY_VAL;
+ }
+ }
+
+ propagate_yield_cpu(node, val, &set_yield_cpu, paravirt);
+ preempted = yield_head_to_locked_owner(lock, val, paravirt);
+ if (!maybe_stealers)
+ continue;
+
+ if (preempted)
+ seen_preempted = true;
+
+ if (paravirt && preempted) {
+ sleepy = true;
+
+ if (!pv_spin_on_preempted_owner)
+ iters++;
+ } else {
+ iters++;
+ }
+
+ if (!mustq && iters >= get_head_spins(paravirt, sleepy)) {
+ mustq = true;
+ set_mustq(lock);
+ val |= _Q_MUST_Q_VAL;
+ }
+ }
+ spec_barrier();
+ spin_end();
+
+ /* If we're the last queued, must clean up the tail. */
+ old = trylock_clean_tail(lock, tail);
+ if (unlikely(old & _Q_LOCKED_VAL)) {
+ BUG_ON(!maybe_stealers);
+ goto again; /* Can only be true if maybe_stealers. */
+ }
+
+ if ((old & _Q_TAIL_CPU_MASK) == tail)
+ goto release; /* We were the tail, no next. */
+
+ /* There is a next, must wait for node->next != NULL (MCS protocol) */
+ next = READ_ONCE(node->next);
+ if (!next) {
+ spin_begin();
+ while (!(next = READ_ONCE(node->next)))
+ cpu_relax();
+ spin_end();
+ }
+ spec_barrier();
+
+ /*
+ * Unlock the next mcs waiter node. Release barrier is not required
+ * here because the acquirer is only accessing the lock word, and
+ * the acquire barrier we took the lock with orders that update vs
+ * this store to locked. The corresponding barrier is the smp_rmb()
+ * acquire barrier for mcs lock, above.
+ */
+ if (paravirt && pv_prod_head) {
+ int next_cpu = next->cpu;
+ WRITE_ONCE(next->locked, 1);
+ if (_Q_SPIN_MISO)
+ asm volatile("miso" ::: "memory");
+ if (vcpu_is_preempted(next_cpu))
+ prod_cpu(next_cpu);
+ } else {
+ WRITE_ONCE(next->locked, 1);
+ if (_Q_SPIN_MISO)
+ asm volatile("miso" ::: "memory");
+ }
+
+release:
+ qnodesp->count--; /* release the node */
+}
+
+void queued_spin_lock_slowpath(struct qspinlock *lock)
+{
+ /*
+ * This looks funny, but it induces the compiler to inline both
+ * sides of the branch rather than share code as when the condition
+ * is passed as the paravirt argument to the functions.
+ */
+ if (IS_ENABLED(CONFIG_PARAVIRT_SPINLOCKS) && is_shared_processor()) {
+ if (try_to_steal_lock(lock, true)) {
+ spec_barrier();
+ return;
+ }
+ queued_spin_lock_mcs_queue(lock, true);
+ } else {
+ if (try_to_steal_lock(lock, false)) {
+ spec_barrier();
+ return;
+ }
+ queued_spin_lock_mcs_queue(lock, false);
+ }
+}
+EXPORT_SYMBOL(queued_spin_lock_slowpath);
+
+#ifdef CONFIG_PARAVIRT_SPINLOCKS
+void pv_spinlocks_init(void)
+{
+}
+#endif
+
+#include <linux/debugfs.h>
+static int steal_spins_set(void *data, u64 val)
+{
+#if _Q_SPIN_TRY_LOCK_STEAL == 1
+ /* MAYBE_STEAL remains true */
+ steal_spins = val;
+#else
+ static DEFINE_MUTEX(lock);
+
+ /*
+ * The lock slow path has a !maybe_stealers case that can assume
+ * the head of queue will not see concurrent waiters. That waiter
+ * is unsafe in the presence of stealers, so must keep them away
+ * from one another.
+ */
+
+ mutex_lock(&lock);
+ if (val && !steal_spins) {
+ maybe_stealers = true;
+ /* wait for queue head waiter to go away */
+ synchronize_rcu();
+ steal_spins = val;
+ } else if (!val && steal_spins) {
+ steal_spins = val;
+ /* wait for all possible stealers to go away */
+ synchronize_rcu();
+ maybe_stealers = false;
+ } else {
+ steal_spins = val;
+ }
+ mutex_unlock(&lock);
+#endif
+
+ return 0;
+}
+
+static int steal_spins_get(void *data, u64 *val)
+{
+ *val = steal_spins;
+
+ return 0;
+}
+
+DEFINE_SIMPLE_ATTRIBUTE(fops_steal_spins, steal_spins_get, steal_spins_set, "%llu\n");
+
+static int remote_steal_spins_set(void *data, u64 val)
+{
+ remote_steal_spins = val;
+
+ return 0;
+}
+
+static int remote_steal_spins_get(void *data, u64 *val)
+{
+ *val = remote_steal_spins;
+
+ return 0;
+}
+
+DEFINE_SIMPLE_ATTRIBUTE(fops_remote_steal_spins, remote_steal_spins_get, remote_steal_spins_set, "%llu\n");
+
+static int head_spins_set(void *data, u64 val)
+{
+ head_spins = val;
+
+ return 0;
+}
+
+static int head_spins_get(void *data, u64 *val)
+{
+ *val = head_spins;
+
+ return 0;
+}
+
+DEFINE_SIMPLE_ATTRIBUTE(fops_head_spins, head_spins_get, head_spins_set, "%llu\n");
+
+static int pv_yield_owner_set(void *data, u64 val)
+{
+ pv_yield_owner = !!val;
+
+ return 0;
+}
+
+static int pv_yield_owner_get(void *data, u64 *val)
+{
+ *val = pv_yield_owner;
+
+ return 0;
+}
+
+DEFINE_SIMPLE_ATTRIBUTE(fops_pv_yield_owner, pv_yield_owner_get, pv_yield_owner_set, "%llu\n");
+
+static int pv_yield_allow_steal_set(void *data, u64 val)
+{
+ pv_yield_allow_steal = !!val;
+
+ return 0;
+}
+
+static int pv_yield_allow_steal_get(void *data, u64 *val)
+{
+ *val = pv_yield_allow_steal;
+
+ return 0;
+}
+
+DEFINE_SIMPLE_ATTRIBUTE(fops_pv_yield_allow_steal, pv_yield_allow_steal_get, pv_yield_allow_steal_set, "%llu\n");
+
+static int pv_spin_on_preempted_owner_set(void *data, u64 val)
+{
+ pv_spin_on_preempted_owner = !!val;
+
+ return 0;
+}
+
+static int pv_spin_on_preempted_owner_get(void *data, u64 *val)
+{
+ *val = pv_spin_on_preempted_owner;
+
+ return 0;
+}
+
+DEFINE_SIMPLE_ATTRIBUTE(fops_pv_spin_on_preempted_owner, pv_spin_on_preempted_owner_get, pv_spin_on_preempted_owner_set, "%llu\n");
+
+static int pv_sleepy_lock_set(void *data, u64 val)
+{
+ pv_sleepy_lock = !!val;
+
+ return 0;
+}
+
+static int pv_sleepy_lock_get(void *data, u64 *val)
+{
+ *val = pv_sleepy_lock;
+
+ return 0;
+}
+
+DEFINE_SIMPLE_ATTRIBUTE(fops_pv_sleepy_lock, pv_sleepy_lock_get, pv_sleepy_lock_set, "%llu\n");
+
+static int pv_sleepy_lock_sticky_set(void *data, u64 val)
+{
+ pv_sleepy_lock_sticky = !!val;
+
+ return 0;
+}
+
+static int pv_sleepy_lock_sticky_get(void *data, u64 *val)
+{
+ *val = pv_sleepy_lock_sticky;
+
+ return 0;
+}
+
+DEFINE_SIMPLE_ATTRIBUTE(fops_pv_sleepy_lock_sticky, pv_sleepy_lock_sticky_get, pv_sleepy_lock_sticky_set, "%llu\n");
+
+static int pv_sleepy_lock_interval_ns_set(void *data, u64 val)
+{
+ pv_sleepy_lock_interval_ns = val;
+
+ return 0;
+}
+
+static int pv_sleepy_lock_interval_ns_get(void *data, u64 *val)
+{
+ *val = pv_sleepy_lock_interval_ns;
+
+ return 0;
+}
+
+DEFINE_SIMPLE_ATTRIBUTE(fops_pv_sleepy_lock_interval_ns, pv_sleepy_lock_interval_ns_get, pv_sleepy_lock_interval_ns_set, "%llu\n");
+
+static int pv_sleepy_lock_factor_set(void *data, u64 val)
+{
+ pv_sleepy_lock_factor = val;
+
+ return 0;
+}
+
+static int pv_sleepy_lock_factor_get(void *data, u64 *val)
+{
+ *val = pv_sleepy_lock_factor;
+
+ return 0;
+}
+
+DEFINE_SIMPLE_ATTRIBUTE(fops_pv_sleepy_lock_factor, pv_sleepy_lock_factor_get, pv_sleepy_lock_factor_set, "%llu\n");
+
+static int pv_yield_prev_set(void *data, u64 val)
+{
+ pv_yield_prev = !!val;
+
+ return 0;
+}
+
+static int pv_yield_prev_get(void *data, u64 *val)
+{
+ *val = pv_yield_prev;
+
+ return 0;
+}
+
+DEFINE_SIMPLE_ATTRIBUTE(fops_pv_yield_prev, pv_yield_prev_get, pv_yield_prev_set, "%llu\n");
+
+static int pv_yield_propagate_owner_set(void *data, u64 val)
+{
+ pv_yield_propagate_owner = !!val;
+
+ return 0;
+}
+
+static int pv_yield_propagate_owner_get(void *data, u64 *val)
+{
+ *val = pv_yield_propagate_owner;
+
+ return 0;
+}
+
+DEFINE_SIMPLE_ATTRIBUTE(fops_pv_yield_propagate_owner, pv_yield_propagate_owner_get, pv_yield_propagate_owner_set, "%llu\n");
+
+static int pv_prod_head_set(void *data, u64 val)
+{
+ pv_prod_head = !!val;
+
+ return 0;
+}
+
+static int pv_prod_head_get(void *data, u64 *val)
+{
+ *val = pv_prod_head;
+
+ return 0;
+}
+
+DEFINE_SIMPLE_ATTRIBUTE(fops_pv_prod_head, pv_prod_head_get, pv_prod_head_set, "%llu\n");
+
+static __init int spinlock_debugfs_init(void)
+{
+ debugfs_create_file("qspl_steal_spins", 0600, arch_debugfs_dir, NULL, &fops_steal_spins);
+ debugfs_create_file("qspl_remote_steal_spins", 0600, arch_debugfs_dir, NULL, &fops_remote_steal_spins);
+ debugfs_create_file("qspl_head_spins", 0600, arch_debugfs_dir, NULL, &fops_head_spins);
+ if (is_shared_processor()) {
+ debugfs_create_file("qspl_pv_yield_owner", 0600, arch_debugfs_dir, NULL, &fops_pv_yield_owner);
+ debugfs_create_file("qspl_pv_yield_allow_steal", 0600, arch_debugfs_dir, NULL, &fops_pv_yield_allow_steal);
+ debugfs_create_file("qspl_pv_spin_on_preempted_owner", 0600, arch_debugfs_dir, NULL, &fops_pv_spin_on_preempted_owner);
+ debugfs_create_file("qspl_pv_sleepy_lock", 0600, arch_debugfs_dir, NULL, &fops_pv_sleepy_lock);
+ debugfs_create_file("qspl_pv_sleepy_lock_sticky", 0600, arch_debugfs_dir, NULL, &fops_pv_sleepy_lock_sticky);
+ debugfs_create_file("qspl_pv_sleepy_lock_interval_ns", 0600, arch_debugfs_dir, NULL, &fops_pv_sleepy_lock_interval_ns);
+ debugfs_create_file("qspl_pv_sleepy_lock_factor", 0600, arch_debugfs_dir, NULL, &fops_pv_sleepy_lock_factor);
+ debugfs_create_file("qspl_pv_yield_prev", 0600, arch_debugfs_dir, NULL, &fops_pv_yield_prev);
+ debugfs_create_file("qspl_pv_yield_propagate_owner", 0600, arch_debugfs_dir, NULL, &fops_pv_yield_propagate_owner);
+ debugfs_create_file("qspl_pv_prod_head", 0600, arch_debugfs_dir, NULL, &fops_pv_prod_head);
+ }
+
+ return 0;
+}
+device_initcall(spinlock_debugfs_init);
diff --git a/arch/powerpc/lib/sstep.c b/arch/powerpc/lib/sstep.c
index 398b569..38158b7 100644
--- a/arch/powerpc/lib/sstep.c
+++ b/arch/powerpc/lib/sstep.c
@@ -2284,15 +2284,7 @@ int analyse_instr(struct instruction_op *op, const struct pt_regs *regs,
op->type = MKOP(STCX, 0, 4);
break;
-#ifdef __powerpc64__
- case 84: /* ldarx */
- op->type = MKOP(LARX, 0, 8);
- break;
-
- case 214: /* stdcx. */
- op->type = MKOP(STCX, 0, 8);
- break;
-
+#ifdef CONFIG_PPC_HAS_LBARX_LHARX
case 52: /* lbarx */
op->type = MKOP(LARX, 0, 1);
break;
@@ -2308,6 +2300,15 @@ int analyse_instr(struct instruction_op *op, const struct pt_regs *regs,
case 726: /* sthcx. */
op->type = MKOP(STCX, 0, 2);
break;
+#endif
+#ifdef __powerpc64__
+ case 84: /* ldarx */
+ op->type = MKOP(LARX, 0, 8);
+ break;
+
+ case 214: /* stdcx. */
+ op->type = MKOP(STCX, 0, 8);
+ break;
case 276: /* lqarx */
if (!((rd & 1) || rd == ra || rd == rb))
@@ -3334,7 +3335,7 @@ int emulate_loadstore(struct pt_regs *regs, struct instruction_op *op)
err = 0;
val = 0;
switch (size) {
-#ifdef __powerpc64__
+#ifdef CONFIG_PPC_HAS_LBARX_LHARX
case 1:
__get_user_asmx(val, ea, err, "lbarx");
break;
diff --git a/arch/powerpc/lib/test_emulate_step_exec_instr.S b/arch/powerpc/lib/test_emulate_step_exec_instr.S
index 5473f9d..e2b646a 100644
--- a/arch/powerpc/lib/test_emulate_step_exec_instr.S
+++ b/arch/powerpc/lib/test_emulate_step_exec_instr.S
@@ -16,7 +16,7 @@
/*
* Stack frame layout (INT_FRAME_SIZE bytes)
- * In-memory pt_regs (SP + STACK_FRAME_OVERHEAD)
+ * In-memory pt_regs (SP + STACK_INT_FRAME_REGS)
* Scratch space (SP + 8)
* Back chain (SP + 0)
*/
diff --git a/arch/powerpc/mm/book3s64/hash_4k.c b/arch/powerpc/mm/book3s64/hash_4k.c
index 7de1a8a..02acbfd 100644
--- a/arch/powerpc/mm/book3s64/hash_4k.c
+++ b/arch/powerpc/mm/book3s64/hash_4k.c
@@ -16,6 +16,8 @@
#include <asm/machdep.h>
#include <asm/mmu.h>
+#include "internal.h"
+
int __hash_page_4K(unsigned long ea, unsigned long access, unsigned long vsid,
pte_t *ptep, unsigned long trap, unsigned long flags,
int ssize, int subpg_prot)
@@ -118,6 +120,9 @@ int __hash_page_4K(unsigned long ea, unsigned long access, unsigned long vsid,
}
new_pte = (new_pte & ~_PAGE_HPTEFLAGS) | H_PAGE_HASHPTE;
new_pte |= pte_set_hidx(ptep, rpte, 0, slot, PTRS_PER_PTE);
+
+ if (stress_hpt())
+ hpt_do_stress(ea, hpte_group);
}
*ptep = __pte(new_pte & ~H_PAGE_BUSY);
return 0;
diff --git a/arch/powerpc/mm/book3s64/hash_64k.c b/arch/powerpc/mm/book3s64/hash_64k.c
index 998c681..954af42 100644
--- a/arch/powerpc/mm/book3s64/hash_64k.c
+++ b/arch/powerpc/mm/book3s64/hash_64k.c
@@ -16,6 +16,8 @@
#include <asm/machdep.h>
#include <asm/mmu.h>
+#include "internal.h"
+
/*
* Return true, if the entry has a slot value which
* the software considers as invalid.
@@ -216,6 +218,9 @@ int __hash_page_4K(unsigned long ea, unsigned long access, unsigned long vsid,
new_pte |= pte_set_hidx(ptep, rpte, subpg_index, slot, PTRS_PER_PTE);
new_pte |= H_PAGE_HASHPTE;
+ if (stress_hpt())
+ hpt_do_stress(ea, hpte_group);
+
*ptep = __pte(new_pte & ~H_PAGE_BUSY);
return 0;
}
@@ -327,7 +332,12 @@ int __hash_page_64K(unsigned long ea, unsigned long access,
new_pte = (new_pte & ~_PAGE_HPTEFLAGS) | H_PAGE_HASHPTE;
new_pte |= pte_set_hidx(ptep, rpte, 0, slot, PTRS_PER_PTE);
+
+ if (stress_hpt())
+ hpt_do_stress(ea, hpte_group);
}
+
*ptep = __pte(new_pte & ~H_PAGE_BUSY);
+
return 0;
}
diff --git a/arch/powerpc/mm/book3s64/hash_utils.c b/arch/powerpc/mm/book3s64/hash_utils.c
index 6df4c6d..80a148c 100644
--- a/arch/powerpc/mm/book3s64/hash_utils.c
+++ b/arch/powerpc/mm/book3s64/hash_utils.c
@@ -471,7 +471,7 @@ int htab_remove_mapping(unsigned long vstart, unsigned long vend,
return ret;
}
-static bool disable_1tb_segments = false;
+static bool disable_1tb_segments __ro_after_init;
static int __init parse_disable_1tb_segments(char *p)
{
@@ -480,6 +480,40 @@ static int __init parse_disable_1tb_segments(char *p)
}
early_param("disable_1tb_segments", parse_disable_1tb_segments);
+bool stress_hpt_enabled __initdata;
+
+static int __init parse_stress_hpt(char *p)
+{
+ stress_hpt_enabled = true;
+ return 0;
+}
+early_param("stress_hpt", parse_stress_hpt);
+
+__ro_after_init DEFINE_STATIC_KEY_FALSE(stress_hpt_key);
+
+/*
+ * per-CPU array allocated if we enable stress_hpt.
+ */
+#define STRESS_MAX_GROUPS 16
+struct stress_hpt_struct {
+ unsigned long last_group[STRESS_MAX_GROUPS];
+};
+
+static inline int stress_nr_groups(void)
+{
+ /*
+ * LPAR H_REMOVE flushes TLB, so need some number > 1 of entries
+ * to allow practical forward progress. Bare metal returns 1, which
+ * seems to help uncover more bugs.
+ */
+ if (firmware_has_feature(FW_FEATURE_LPAR))
+ return STRESS_MAX_GROUPS;
+ else
+ return 1;
+}
+
+static struct stress_hpt_struct *stress_hpt_struct;
+
static int __init htab_dt_scan_seg_sizes(unsigned long node,
const char *uname, int depth,
void *data)
@@ -976,6 +1010,23 @@ static void __init hash_init_partition_table(phys_addr_t hash_table,
pr_info("Partition table %p\n", partition_tb);
}
+void hpt_clear_stress(void);
+static struct timer_list stress_hpt_timer;
+void stress_hpt_timer_fn(struct timer_list *timer)
+{
+ int next_cpu;
+
+ hpt_clear_stress();
+ if (!firmware_has_feature(FW_FEATURE_LPAR))
+ tlbiel_all();
+
+ next_cpu = cpumask_next(raw_smp_processor_id(), cpu_online_mask);
+ if (next_cpu >= nr_cpu_ids)
+ next_cpu = cpumask_first(cpu_online_mask);
+ stress_hpt_timer.expires = jiffies + msecs_to_jiffies(10);
+ add_timer_on(&stress_hpt_timer, next_cpu);
+}
+
static void __init htab_initialize(void)
{
unsigned long table;
@@ -995,6 +1046,20 @@ static void __init htab_initialize(void)
if (stress_slb_enabled)
static_branch_enable(&stress_slb_key);
+ if (stress_hpt_enabled) {
+ unsigned long tmp;
+ static_branch_enable(&stress_hpt_key);
+ // Too early to use nr_cpu_ids, so use NR_CPUS
+ tmp = memblock_phys_alloc_range(sizeof(struct stress_hpt_struct) * NR_CPUS,
+ 0, 0, MEMBLOCK_ALLOC_ANYWHERE);
+ memset((void *)tmp, 0xff, sizeof(struct stress_hpt_struct) * NR_CPUS);
+ stress_hpt_struct = __va(tmp);
+
+ timer_setup(&stress_hpt_timer, stress_hpt_timer_fn, 0);
+ stress_hpt_timer.expires = jiffies + msecs_to_jiffies(10);
+ add_timer(&stress_hpt_timer);
+ }
+
/*
* Calculate the required size of the htab. We want the number of
* PTEGs to equal one half the number of real pages.
@@ -1980,6 +2045,69 @@ long hpte_insert_repeating(unsigned long hash, unsigned long vpn,
return slot;
}
+void hpt_clear_stress(void)
+{
+ int cpu = raw_smp_processor_id();
+ int g;
+
+ for (g = 0; g < stress_nr_groups(); g++) {
+ unsigned long last_group;
+ last_group = stress_hpt_struct[cpu].last_group[g];
+
+ if (last_group != -1UL) {
+ int i;
+ for (i = 0; i < HPTES_PER_GROUP; i++) {
+ if (mmu_hash_ops.hpte_remove(last_group) == -1)
+ break;
+ }
+ stress_hpt_struct[cpu].last_group[g] = -1;
+ }
+ }
+}
+
+void hpt_do_stress(unsigned long ea, unsigned long hpte_group)
+{
+ unsigned long last_group;
+ int cpu = raw_smp_processor_id();
+
+ last_group = stress_hpt_struct[cpu].last_group[stress_nr_groups() - 1];
+ if (hpte_group == last_group)
+ return;
+
+ if (last_group != -1UL) {
+ int i;
+ /*
+ * Concurrent CPUs might be inserting into this group, so
+ * give up after a number of iterations, to prevent a live
+ * lock.
+ */
+ for (i = 0; i < HPTES_PER_GROUP; i++) {
+ if (mmu_hash_ops.hpte_remove(last_group) == -1)
+ break;
+ }
+ stress_hpt_struct[cpu].last_group[stress_nr_groups() - 1] = -1;
+ }
+
+ if (ea >= PAGE_OFFSET) {
+ /*
+ * We would really like to prefetch to get the TLB loaded, then
+ * remove the PTE before returning from fault interrupt, to
+ * increase the hash fault rate.
+ *
+ * Unfortunately QEMU TCG does not model the TLB in a way that
+ * makes this possible, and systemsim (mambo) emulator does not
+ * bring in TLBs with prefetches (although loads/stores do
+ * work for non-CI PTEs).
+ *
+ * So remember this PTE and clear it on the next hash fault.
+ */
+ memmove(&stress_hpt_struct[cpu].last_group[1],
+ &stress_hpt_struct[cpu].last_group[0],
+ (stress_nr_groups() - 1) * sizeof(unsigned long));
+ stress_hpt_struct[cpu].last_group[0] = hpte_group;
+ }
+}
+
#if defined(CONFIG_DEBUG_PAGEALLOC) || defined(CONFIG_KFENCE)
static DEFINE_RAW_SPINLOCK(linear_map_hash_lock);
diff --git a/arch/powerpc/mm/book3s64/internal.h b/arch/powerpc/mm/book3s64/internal.h
index 5045048..a57a25f 100644
--- a/arch/powerpc/mm/book3s64/internal.h
+++ b/arch/powerpc/mm/book3s64/internal.h
@@ -13,6 +13,17 @@ static inline bool stress_slb(void)
return static_branch_unlikely(&stress_slb_key);
}
+extern bool stress_hpt_enabled;
+
+DECLARE_STATIC_KEY_FALSE(stress_hpt_key);
+
+static inline bool stress_hpt(void)
+{
+ return static_branch_unlikely(&stress_hpt_key);
+}
+
+void hpt_do_stress(unsigned long ea, unsigned long hpte_group);
+
void slb_setup_new_exec(void);
void exit_lazy_flush_tlb(struct mm_struct *mm, bool always_flush);
diff --git a/arch/powerpc/mm/book3s64/pgtable.c b/arch/powerpc/mm/book3s64/pgtable.c
index f6151a5..85c84e8 100644
--- a/arch/powerpc/mm/book3s64/pgtable.c
+++ b/arch/powerpc/mm/book3s64/pgtable.c
@@ -100,14 +100,14 @@ static void do_serialize(void *arg)
}
/*
- * Serialize against find_current_mm_pte which does lock-less
+ * Serialize against __find_linux_pte() which does lock-less
* lookup in page tables with local interrupts disabled. For huge pages
* it casts pmd_t to pte_t. Since format of pte_t is different from
* pmd_t we want to prevent transit from pmd pointing to page table
* to pmd pointing to huge page (and back) while interrupts are disabled.
* We clear pmd to possibly replace it with page table pointer in
* different code paths. So make sure we wait for the parallel
- * find_current_mm_pte to finish.
+ * __find_linux_pte() to finish.
*/
void serialize_against_pte_lookup(struct mm_struct *mm)
{
diff --git a/arch/powerpc/mm/mem.c b/arch/powerpc/mm/mem.c
index 84d17195..8b121df 100644
--- a/arch/powerpc/mm/mem.c
+++ b/arch/powerpc/mm/mem.c
@@ -344,7 +344,6 @@ void free_initmem(void)
{
ppc_md.progress = ppc_printk_progress;
mark_initmem_nx();
- static_branch_enable(&init_mem_is_free);
free_initmem_default(POISON_FREE_INITMEM);
ftrace_free_init_tramp();
}
diff --git a/arch/powerpc/mm/nohash/kaslr_booke.c b/arch/powerpc/mm/nohash/kaslr_booke.c
index 0d04f9d..2fb3eda 100644
--- a/arch/powerpc/mm/nohash/kaslr_booke.c
+++ b/arch/powerpc/mm/nohash/kaslr_booke.c
@@ -19,7 +19,6 @@
#include <asm/cacheflush.h>
#include <asm/kdump.h>
#include <mm/mmu_decl.h>
-#include <generated/utsrelease.h>
struct regions {
unsigned long pa_start;
diff --git a/arch/powerpc/mm/nohash/tlb.c b/arch/powerpc/mm/nohash/tlb.c
index 2c15c86..a903b30 100644
--- a/arch/powerpc/mm/nohash/tlb.c
+++ b/arch/powerpc/mm/nohash/tlb.c
@@ -184,6 +184,14 @@ void local_flush_tlb_page(struct vm_area_struct *vma, unsigned long vmaddr)
mmu_get_tsize(mmu_virtual_psize), 0);
}
EXPORT_SYMBOL(local_flush_tlb_page);
+
+void local_flush_tlb_page_psize(struct mm_struct *mm,
+ unsigned long vmaddr, int psize)
+{
+ __local_flush_tlb_page(mm, vmaddr, mmu_get_tsize(psize), 0);
+}
+EXPORT_SYMBOL(local_flush_tlb_page_psize);
+
#endif
/*
diff --git a/arch/powerpc/perf/callchain.c b/arch/powerpc/perf/callchain.c
index 082f6d0..6b4434d 100644
--- a/arch/powerpc/perf/callchain.c
+++ b/arch/powerpc/perf/callchain.c
@@ -27,7 +27,7 @@ static int valid_next_sp(unsigned long sp, unsigned long prev_sp)
{
if (sp & 0xf)
return 0; /* must be 16-byte aligned */
- if (!validate_sp(sp, current, STACK_FRAME_OVERHEAD))
+ if (!validate_sp(sp, current))
return 0;
if (sp >= prev_sp + STACK_FRAME_MIN_SIZE)
return 1;
@@ -53,7 +53,7 @@ perf_callchain_kernel(struct perf_callchain_entry_ctx *entry, struct pt_regs *re
sp = regs->gpr[1];
perf_callchain_store(entry, perf_instruction_pointer(regs));
- if (!validate_sp(sp, current, STACK_FRAME_OVERHEAD))
+ if (!validate_sp(sp, current))
return;
for (;;) {
@@ -61,12 +61,13 @@ perf_callchain_kernel(struct perf_callchain_entry_ctx *entry, struct pt_regs *re
next_sp = fp[0];
if (next_sp == sp + STACK_INT_FRAME_SIZE &&
- fp[STACK_FRAME_MARKER] == STACK_FRAME_REGS_MARKER) {
+ validate_sp_size(sp, current, STACK_INT_FRAME_SIZE) &&
+ fp[STACK_INT_FRAME_MARKER_LONGS] == STACK_FRAME_REGS_MARKER) {
/*
* This looks like an interrupt frame for an
* interrupt that occurred in the kernel
*/
- regs = (struct pt_regs *)(sp + STACK_FRAME_OVERHEAD);
+ regs = (struct pt_regs *)(sp + STACK_INT_FRAME_REGS);
next_ip = regs->nip;
lr = regs->link;
level = 0;
diff --git a/arch/powerpc/perf/hv-gpci-requests.h b/arch/powerpc/perf/hv-gpci-requests.h
index 8965b44..5e86371 100644
--- a/arch/powerpc/perf/hv-gpci-requests.h
+++ b/arch/powerpc/perf/hv-gpci-requests.h
@@ -79,6 +79,7 @@ REQUEST(__field(0, 8, partition_id)
)
#include I(REQUEST_END)
+#ifdef ENABLE_EVENTS_COUNTERINFO_V6
/*
* Not available for counter_info_version >= 0x8, use
* run_instruction_cycles_by_partition(0x100) instead.
@@ -92,6 +93,7 @@ REQUEST(__field(0, 8, partition_id)
__count(0x10, 8, cycles)
)
#include I(REQUEST_END)
+#endif
#define REQUEST_NAME system_performance_capabilities
#define REQUEST_NUM 0x40
@@ -103,6 +105,7 @@ REQUEST(__field(0, 1, perf_collect_privileged)
)
#include I(REQUEST_END)
+#ifdef ENABLE_EVENTS_COUNTERINFO_V6
#define REQUEST_NAME processor_bus_utilization_abc_links
#define REQUEST_NUM 0x50
#define REQUEST_IDX_KIND "hw_chip_id=?"
@@ -194,6 +197,7 @@ REQUEST(__field(0, 4, phys_processor_idx)
__count(0x28, 8, instructions_completed)
)
#include I(REQUEST_END)
+#endif
/* Processor_core_power_mode (0x95) skipped, no counters */
/* Affinity_domain_information_by_virtual_processor (0xA0) skipped,
diff --git a/arch/powerpc/perf/hv-gpci.c b/arch/powerpc/perf/hv-gpci.c
index 5eb60ed..7ff8ff3 100644
--- a/arch/powerpc/perf/hv-gpci.c
+++ b/arch/powerpc/perf/hv-gpci.c
@@ -70,9 +70,9 @@ static const struct attribute_group format_group = {
.attrs = format_attrs,
};
-static const struct attribute_group event_group = {
+static struct attribute_group event_group = {
.name = "events",
- .attrs = hv_gpci_event_attrs,
+ /* .attrs is set in init */
};
#define HV_CAPS_ATTR(_name, _format) \
@@ -330,6 +330,7 @@ static int hv_gpci_init(void)
int r;
unsigned long hret;
struct hv_perf_caps caps;
+ struct hv_gpci_request_buffer *arg;
hv_gpci_assert_offsets_correct();
@@ -353,6 +354,36 @@ static int hv_gpci_init(void)
/* sampling not supported */
h_gpci_pmu.capabilities |= PERF_PMU_CAP_NO_INTERRUPT;
+ arg = (void *)get_cpu_var(hv_gpci_reqb);
+ memset(arg, 0, HGPCI_REQ_BUFFER_SIZE);
+
+ /*
+ * hcall H_GET_PERF_COUNTER_INFO populates the output
+ * counter_info_version value based on the system hypervisor.
+ * Pass the counter request 0x10 corresponds to request type
+ * 'Dispatch_timebase_by_processor', to get the supported
+ * counter_info_version.
+ */
+ arg->params.counter_request = cpu_to_be32(0x10);
+
+ r = plpar_hcall_norets(H_GET_PERF_COUNTER_INFO,
+ virt_to_phys(arg), HGPCI_REQ_BUFFER_SIZE);
+ if (r) {
+ pr_devel("hcall failed, can't get supported counter_info_version: 0x%x\n", r);
+ arg->params.counter_info_version_out = 0x8;
+ }
+
+ /*
+ * Use counter_info_version_out value to assign
+ * required hv-gpci event list.
+ */
+ if (arg->params.counter_info_version_out >= 0x8)
+ event_group.attrs = hv_gpci_event_attrs;
+ else
+ event_group.attrs = hv_gpci_event_attrs_v6;
+
+ put_cpu_var(hv_gpci_reqb);
+
r = perf_pmu_register(&h_gpci_pmu, h_gpci_pmu.name, -1);
if (r)
return r;
diff --git a/arch/powerpc/perf/hv-gpci.h b/arch/powerpc/perf/hv-gpci.h
index 4d10826..c720209 100644
--- a/arch/powerpc/perf/hv-gpci.h
+++ b/arch/powerpc/perf/hv-gpci.h
@@ -26,6 +26,7 @@ enum {
#define REQUEST_FILE "../hv-gpci-requests.h"
#define NAME_LOWER hv_gpci
#define NAME_UPPER HV_GPCI
+#define ENABLE_EVENTS_COUNTERINFO_V6
#include "req-gen/perf.h"
#undef REQUEST_FILE
#undef NAME_LOWER
diff --git a/arch/powerpc/perf/req-gen/perf.h b/arch/powerpc/perf/req-gen/perf.h
index fa9bc80..6b2a59f 100644
--- a/arch/powerpc/perf/req-gen/perf.h
+++ b/arch/powerpc/perf/req-gen/perf.h
@@ -139,6 +139,26 @@ PMU_EVENT_ATTR_STRING( \
#define REQUEST_(r_name, r_value, r_idx_1, r_fields) \
r_fields
+/* Generate event list for platforms with counter_info_version 0x6 or below */
+static __maybe_unused struct attribute *hv_gpci_event_attrs_v6[] = {
+#include REQUEST_FILE
+ NULL
+};
+
+/*
+ * Based on getPerfCountInfo v1.018 documentation, some of the hv-gpci
+ * events were deprecated for platform firmware that supports
+ * counter_info_version 0x8 or above.
+ * Those deprecated events are still part of platform firmware that
+ * support counter_info_version 0x6 and below. As per the getPerfCountInfo
+ * v1.018 documentation there is no counter_info_version 0x7.
+ * Undefining macro ENABLE_EVENTS_COUNTERINFO_V6, to disable the addition of
+ * deprecated events in "hv_gpci_event_attrs" attribute group, for platforms
+ * that supports counter_info_version 0x8 or above.
+ */
+#undef ENABLE_EVENTS_COUNTERINFO_V6
+
+/* Generate event list for platforms with counter_info_version 0x8 or above*/
static __maybe_unused struct attribute *hv_gpci_event_attrs[] = {
#include REQUEST_FILE
NULL
diff --git a/arch/powerpc/platforms/44x/warp.c b/arch/powerpc/platforms/44x/warp.c
index f03432e..cefa313 100644
--- a/arch/powerpc/platforms/44x/warp.c
+++ b/arch/powerpc/platforms/44x/warp.c
@@ -5,15 +5,17 @@
* Copyright (c) 2008-2009 PIKA Technologies
* Sean MacLennan <smaclennan@pikatech.com>
*/
+#include <linux/err.h>
#include <linux/init.h>
#include <linux/of_platform.h>
#include <linux/kthread.h>
+#include <linux/leds.h>
#include <linux/i2c.h>
#include <linux/interrupt.h>
#include <linux/delay.h>
#include <linux/of_address.h>
#include <linux/of_irq.h>
-#include <linux/of_gpio.h>
+#include <linux/gpio/consumer.h>
#include <linux/slab.h>
#include <linux/export.h>
@@ -92,8 +94,6 @@ static int __init warp_post_info(void)
static LIST_HEAD(dtm_shutdown_list);
static void __iomem *dtm_fpga;
-static unsigned green_led, red_led;
-
struct dtm_shutdown {
struct list_head list;
@@ -101,7 +101,6 @@ struct dtm_shutdown {
void *arg;
};
-
int pika_dtm_register_shutdown(void (*func)(void *arg), void *arg)
{
struct dtm_shutdown *shutdown;
@@ -132,6 +131,35 @@ int pika_dtm_unregister_shutdown(void (*func)(void *arg), void *arg)
return -EINVAL;
}
+#define WARP_GREEN_LED 0
+#define WARP_RED_LED 1
+
+static struct gpio_led warp_gpio_led_pins[] = {
+ [WARP_GREEN_LED] = {
+ .name = "green",
+ .default_state = LEDS_DEFSTATE_KEEP,
+ .gpiod = NULL, /* to be filled by pika_setup_leds() */
+ },
+ [WARP_RED_LED] = {
+ .name = "red",
+ .default_state = LEDS_DEFSTATE_KEEP,
+ .gpiod = NULL, /* to be filled by pika_setup_leds() */
+ },
+};
+
+static struct gpio_led_platform_data warp_gpio_led_data = {
+ .leds = warp_gpio_led_pins,
+ .num_leds = ARRAY_SIZE(warp_gpio_led_pins),
+};
+
+static struct platform_device warp_gpio_leds = {
+ .name = "leds-gpio",
+ .id = -1,
+ .dev = {
+ .platform_data = &warp_gpio_led_data,
+ },
+};
+
static irqreturn_t temp_isr(int irq, void *context)
{
struct dtm_shutdown *shutdown;
@@ -139,7 +167,7 @@ static irqreturn_t temp_isr(int irq, void *context)
local_irq_disable();
- gpio_set_value(green_led, 0);
+ gpiod_set_value(warp_gpio_led_pins[WARP_GREEN_LED].gpiod, 0);
/* Run through the shutdown list. */
list_for_each_entry(shutdown, &dtm_shutdown_list, list)
@@ -153,7 +181,7 @@ static irqreturn_t temp_isr(int irq, void *context)
out_be32(dtm_fpga + 0x14, reset);
}
- gpio_set_value(red_led, value);
+ gpiod_set_value(warp_gpio_led_pins[WARP_RED_LED].gpiod, value);
value ^= 1;
mdelay(500);
}
@@ -162,25 +190,78 @@ static irqreturn_t temp_isr(int irq, void *context)
return IRQ_HANDLED;
}
+/*
+ * Because green and red power LEDs are normally driven by leds-gpio driver,
+ * but in case of critical temperature shutdown we want to drive them
+ * ourselves, we acquire both and then create leds-gpio platform device
+ * ourselves, instead of doing it through device tree. This way we can still
+ * keep access to the gpios and use them when needed.
+ */
static int pika_setup_leds(void)
{
struct device_node *np, *child;
+ struct gpio_desc *gpio;
+ struct gpio_led *led;
+ int led_count = 0;
+ int error;
+ int i;
- np = of_find_compatible_node(NULL, NULL, "gpio-leds");
+ np = of_find_compatible_node(NULL, NULL, "warp-power-leds");
if (!np) {
printk(KERN_ERR __FILE__ ": Unable to find leds\n");
return -ENOENT;
}
- for_each_child_of_node(np, child)
- if (of_node_name_eq(child, "green"))
- green_led = of_get_gpio(child, 0);
- else if (of_node_name_eq(child, "red"))
- red_led = of_get_gpio(child, 0);
+ for_each_child_of_node(np, child) {
+ for (i = 0; i < ARRAY_SIZE(warp_gpio_led_pins); i++) {
+ led = &warp_gpio_led_pins[i];
+
+ if (!of_node_name_eq(child, led->name))
+ continue;
+
+ if (led->gpiod) {
+ printk(KERN_ERR __FILE__ ": %s led has already been defined\n",
+ led->name);
+ continue;
+ }
+
+ gpio = fwnode_gpiod_get_index(of_fwnode_handle(child),
+ NULL, 0, GPIOD_ASIS,
+ led->name);
+ error = PTR_ERR_OR_ZERO(gpio);
+ if (error) {
+ printk(KERN_ERR __FILE__ ": Failed to get %s led gpio: %d\n",
+ led->name, error);
+ of_node_put(child);
+ goto err_cleanup_pins;
+ }
+
+ led->gpiod = gpio;
+ led_count++;
+ }
+ }
of_node_put(np);
+ /* Skip device registration if no leds have been defined */
+ if (led_count) {
+ error = platform_device_register(&warp_gpio_leds);
+ if (error) {
+ printk(KERN_ERR __FILE__ ": Unable to add leds-gpio: %d\n",
+ error);
+ goto err_cleanup_pins;
+ }
+ }
+
return 0;
+
+err_cleanup_pins:
+ for (i = 0; i < ARRAY_SIZE(warp_gpio_led_pins); i++) {
+ led = &warp_gpio_led_pins[i];
+ gpiod_put(led->gpiod);
+ led->gpiod = NULL;
+ }
+ return error;
}
static void pika_setup_critical_temp(struct device_node *np,
diff --git a/arch/powerpc/platforms/4xx/hsta_msi.c b/arch/powerpc/platforms/4xx/hsta_msi.c
index d4f7fff..e11b57a 100644
--- a/arch/powerpc/platforms/4xx/hsta_msi.c
+++ b/arch/powerpc/platforms/4xx/hsta_msi.c
@@ -115,6 +115,7 @@ static void hsta_teardown_msi_irqs(struct pci_dev *dev)
msi_bitmap_free_hwirqs(&ppc4xx_hsta_msi.bmp, irq, 1);
pr_debug("%s: Teardown IRQ %u (index %u)\n", __func__,
entry->irq, irq);
+ entry->irq = 0;
}
}
diff --git a/arch/powerpc/platforms/52xx/lite5200_sleep.S b/arch/powerpc/platforms/52xx/lite5200_sleep.S
index afee8b1..0b12647 100644
--- a/arch/powerpc/platforms/52xx/lite5200_sleep.S
+++ b/arch/powerpc/platforms/52xx/lite5200_sleep.S
@@ -1,4 +1,6 @@
/* SPDX-License-Identifier: GPL-2.0 */
+#include <linux/linkage.h>
+
#include <asm/reg.h>
#include <asm/ppc_asm.h>
#include <asm/processor.h>
@@ -178,7 +180,8 @@
/* local udelay in sram is needed */
- udelay: /* r11 - tb_ticks_per_usec, r12 - usecs, overwrites r13 */
+SYM_FUNC_START_LOCAL(udelay)
+ /* r11 - tb_ticks_per_usec, r12 - usecs, overwrites r13 */
mullw r12, r12, r11
mftb r13 /* start */
add r12, r13, r12 /* end */
@@ -187,6 +190,7 @@
cmp cr0, r13, r12
blt 1b
blr
+SYM_FUNC_END(udelay)
sram_code_end:
@@ -271,7 +275,7 @@
SAVE_SR(n+2, addr+2); \
SAVE_SR(n+3, addr+3);
-save_regs:
+SYM_FUNC_START_LOCAL(save_regs)
stw r0, 0(r4)
stw r1, 0x4(r4)
stw r2, 0x8(r4)
@@ -317,6 +321,7 @@
SAVE_SPRN(TBRU, 0x5b)
blr
+SYM_FUNC_END(save_regs)
/* restore registers */
@@ -336,7 +341,7 @@
LOAD_SR(n+2, addr+2); \
LOAD_SR(n+3, addr+3);
-restore_regs:
+SYM_FUNC_START_LOCAL(restore_regs)
lis r4, registers@h
ori r4, r4, registers@l
@@ -393,6 +398,7 @@
blr
_ASM_NOKPROBE_SYMBOL(restore_regs)
+SYM_FUNC_END(restore_regs)
@@ -403,7 +409,7 @@
* Flush data cache
* Do this by just reading lots of stuff into the cache.
*/
-flush_data_cache:
+SYM_FUNC_START_LOCAL(flush_data_cache)
lis r3,CONFIG_KERNEL_START@h
ori r3,r3,CONFIG_KERNEL_START@l
li r4,NUM_CACHE_LINES
@@ -413,3 +419,4 @@
addi r3,r3,L1_CACHE_BYTES /* Next line, please */
bdnz 1b
blr
+SYM_FUNC_END(flush_data_cache)
diff --git a/arch/powerpc/platforms/52xx/mpc52xx_lpbfifo.c b/arch/powerpc/platforms/52xx/mpc52xx_lpbfifo.c
index 48038aa..6d1dd6e 100644
--- a/arch/powerpc/platforms/52xx/mpc52xx_lpbfifo.c
+++ b/arch/powerpc/platforms/52xx/mpc52xx_lpbfifo.c
@@ -59,6 +59,8 @@ static struct mpc52xx_lpbfifo lpbfifo;
/**
* mpc52xx_lpbfifo_kick - Trigger the next block of data to be transferred
+ *
+ * @req: Pointer to request structure
*/
static void mpc52xx_lpbfifo_kick(struct mpc52xx_lpbfifo_request *req)
{
@@ -178,6 +180,8 @@ static void mpc52xx_lpbfifo_kick(struct mpc52xx_lpbfifo_request *req)
/**
* mpc52xx_lpbfifo_irq - IRQ handler for LPB FIFO
+ * @irq: IRQ number to be handled
+ * @dev_id: device ID cookie
*
* On transmit, the dma completion irq triggers before the fifo completion
* triggers. Handle the dma completion here instead of the LPB FIFO Bestcomm
@@ -216,6 +220,8 @@ static void mpc52xx_lpbfifo_kick(struct mpc52xx_lpbfifo_request *req)
* or nested spinlock condition. The out path is non-trivial, so
* extra fiddling is done to make sure all paths lead to the same
* outbound code.
+ *
+ * Return: irqreturn code (%IRQ_HANDLED)
*/
static irqreturn_t mpc52xx_lpbfifo_irq(int irq, void *dev_id)
{
@@ -320,8 +326,12 @@ static irqreturn_t mpc52xx_lpbfifo_irq(int irq, void *dev_id)
/**
* mpc52xx_lpbfifo_bcom_irq - IRQ handler for LPB FIFO Bestcomm task
+ * @irq: IRQ number to be handled
+ * @dev_id: device ID cookie
*
* Only used when receiving data.
+ *
+ * Return: irqreturn code (%IRQ_HANDLED)
*/
static irqreturn_t mpc52xx_lpbfifo_bcom_irq(int irq, void *dev_id)
{
@@ -372,7 +382,7 @@ static irqreturn_t mpc52xx_lpbfifo_bcom_irq(int irq, void *dev_id)
}
/**
- * mpc52xx_lpbfifo_bcom_poll - Poll for DMA completion
+ * mpc52xx_lpbfifo_poll - Poll for DMA completion
*/
void mpc52xx_lpbfifo_poll(void)
{
@@ -393,6 +403,8 @@ EXPORT_SYMBOL(mpc52xx_lpbfifo_poll);
/**
* mpc52xx_lpbfifo_submit - Submit an LPB FIFO transfer request.
* @req: Pointer to request structure
+ *
+ * Return: %0 on success, -errno code on error
*/
int mpc52xx_lpbfifo_submit(struct mpc52xx_lpbfifo_request *req)
{
@@ -531,6 +543,7 @@ static int mpc52xx_lpbfifo_probe(struct platform_device *op)
err_bcom_rx_irq:
bcom_gen_bd_rx_release(lpbfifo.bcom_rx_task);
err_bcom_rx:
+ free_irq(lpbfifo.irq, &lpbfifo);
err_irq:
iounmap(lpbfifo.regs);
lpbfifo.regs = NULL;
diff --git a/arch/powerpc/platforms/83xx/mpc832x_rdb.c b/arch/powerpc/platforms/83xx/mpc832x_rdb.c
index e12cb44..caa96ed 100644
--- a/arch/powerpc/platforms/83xx/mpc832x_rdb.c
+++ b/arch/powerpc/platforms/83xx/mpc832x_rdb.c
@@ -107,7 +107,7 @@ static int __init of_fsl_spi_probe(char *type, char *compatible, u32 sysclk,
goto next;
unreg:
- platform_device_del(pdev);
+ platform_device_put(pdev);
err:
pr_err("%pOF: registration failed\n", np);
next:
diff --git a/arch/powerpc/platforms/85xx/sgy_cts1000.c b/arch/powerpc/platforms/85xx/sgy_cts1000.c
index e14d1b7..751395c 100644
--- a/arch/powerpc/platforms/85xx/sgy_cts1000.c
+++ b/arch/powerpc/platforms/85xx/sgy_cts1000.c
@@ -7,10 +7,13 @@
* Copyright 2012 by Servergy, Inc.
*/
+#define pr_fmt(fmt) "gpio-halt: " fmt
+
+#include <linux/err.h>
#include <linux/platform_device.h>
#include <linux/device.h>
+#include <linux/gpio/consumer.h>
#include <linux/module.h>
-#include <linux/of_gpio.h>
#include <linux/of_irq.h>
#include <linux/workqueue.h>
#include <linux/reboot.h>
@@ -18,7 +21,8 @@
#include <asm/machdep.h>
-static struct device_node *halt_node;
+static struct gpio_desc *halt_gpio;
+static int halt_irq;
static const struct of_device_id child_match[] = {
{
@@ -36,23 +40,10 @@ static DECLARE_WORK(gpio_halt_wq, gpio_halt_wfn);
static void __noreturn gpio_halt_cb(void)
{
- enum of_gpio_flags flags;
- int trigger, gpio;
-
- if (!halt_node)
- panic("No reset GPIO information was provided in DT\n");
-
- gpio = of_get_gpio_flags(halt_node, 0, &flags);
-
- if (!gpio_is_valid(gpio))
- panic("Provided GPIO is invalid\n");
-
- trigger = (flags == OF_GPIO_ACTIVE_LOW);
-
- printk(KERN_INFO "gpio-halt: triggering GPIO.\n");
+ pr_info("triggering GPIO.\n");
/* Probably wont return */
- gpio_set_value(gpio, trigger);
+ gpiod_set_value(halt_gpio, 1);
panic("Halt failed\n");
}
@@ -61,95 +52,78 @@ static void __noreturn gpio_halt_cb(void)
* to handle the shutdown/poweroff. */
static irqreturn_t gpio_halt_irq(int irq, void *__data)
{
- printk(KERN_INFO "gpio-halt: shutdown due to power button IRQ.\n");
+ struct platform_device *pdev = __data;
+
+ dev_info(&pdev->dev, "scheduling shutdown due to power button IRQ\n");
schedule_work(&gpio_halt_wq);
return IRQ_HANDLED;
};
-static int gpio_halt_probe(struct platform_device *pdev)
+static int __gpio_halt_probe(struct platform_device *pdev,
+ struct device_node *halt_node)
{
- enum of_gpio_flags flags;
- struct device_node *node = pdev->dev.of_node;
- struct device_node *child_node;
- int gpio, err, irq;
- int trigger;
+ int err;
- if (!node)
- return -ENODEV;
-
- /* If there's no matching child, this isn't really an error */
- child_node = of_find_matching_node(node, child_match);
- if (!child_node)
- return 0;
-
- /* Technically we could just read the first one, but punish
- * DT writers for invalid form. */
- if (of_gpio_count(child_node) != 1) {
- err = -EINVAL;
- goto err_put;
- }
-
- /* Get the gpio number relative to the dynamic base. */
- gpio = of_get_gpio_flags(child_node, 0, &flags);
- if (!gpio_is_valid(gpio)) {
- err = -EINVAL;
- goto err_put;
- }
-
- err = gpio_request(gpio, "gpio-halt");
+ halt_gpio = fwnode_gpiod_get_index(of_fwnode_handle(halt_node),
+ NULL, 0, GPIOD_OUT_LOW, "gpio-halt");
+ err = PTR_ERR_OR_ZERO(halt_gpio);
if (err) {
- printk(KERN_ERR "gpio-halt: error requesting GPIO %d.\n",
- gpio);
- goto err_put;
+ dev_err(&pdev->dev, "failed to request halt GPIO: %d\n", err);
+ return err;
}
- trigger = (flags == OF_GPIO_ACTIVE_LOW);
-
- gpio_direction_output(gpio, !trigger);
-
/* Now get the IRQ which tells us when the power button is hit */
- irq = irq_of_parse_and_map(child_node, 0);
- err = request_irq(irq, gpio_halt_irq, IRQF_TRIGGER_RISING |
- IRQF_TRIGGER_FALLING, "gpio-halt", child_node);
+ halt_irq = irq_of_parse_and_map(halt_node, 0);
+ err = request_irq(halt_irq, gpio_halt_irq,
+ IRQF_TRIGGER_RISING | IRQF_TRIGGER_FALLING,
+ "gpio-halt", pdev);
if (err) {
- printk(KERN_ERR "gpio-halt: error requesting IRQ %d for "
- "GPIO %d.\n", irq, gpio);
- gpio_free(gpio);
- goto err_put;
+ dev_err(&pdev->dev, "failed to request IRQ %d: %d\n",
+ halt_irq, err);
+ gpiod_put(halt_gpio);
+ halt_gpio = NULL;
+ return err;
}
/* Register our halt function */
ppc_md.halt = gpio_halt_cb;
pm_power_off = gpio_halt_cb;
- printk(KERN_INFO "gpio-halt: registered GPIO %d (%d trigger, %d"
- " irq).\n", gpio, trigger, irq);
+ dev_info(&pdev->dev, "registered halt GPIO, irq: %d\n", halt_irq);
- halt_node = child_node;
return 0;
+}
-err_put:
- of_node_put(child_node);
- return err;
+static int gpio_halt_probe(struct platform_device *pdev)
+{
+ struct device_node *halt_node;
+ int ret;
+
+ if (!pdev->dev.of_node)
+ return -ENODEV;
+
+ /* If there's no matching child, this isn't really an error */
+ halt_node = of_find_matching_node(pdev->dev.of_node, child_match);
+ if (!halt_node)
+ return -ENODEV;
+
+ ret = __gpio_halt_probe(pdev, halt_node);
+ of_node_put(halt_node);
+
+ return ret;
}
static int gpio_halt_remove(struct platform_device *pdev)
{
- if (halt_node) {
- int gpio = of_get_gpio(halt_node, 0);
- int irq = irq_of_parse_and_map(halt_node, 0);
+ free_irq(halt_irq, pdev);
+ cancel_work_sync(&gpio_halt_wq);
- free_irq(irq, halt_node);
+ ppc_md.halt = NULL;
+ pm_power_off = NULL;
- ppc_md.halt = NULL;
- pm_power_off = NULL;
-
- gpio_free(gpio);
-
- of_node_put(halt_node);
- halt_node = NULL;
- }
+ gpiod_put(halt_gpio);
+ halt_gpio = NULL;
return 0;
}
diff --git a/arch/powerpc/platforms/Kconfig.cputype b/arch/powerpc/platforms/Kconfig.cputype
index 0c4eed9..9563336 100644
--- a/arch/powerpc/platforms/Kconfig.cputype
+++ b/arch/powerpc/platforms/Kconfig.cputype
@@ -135,6 +135,7 @@
depends on PPC_BOOK3S_64 && CPU_LITTLE_ENDIAN
select ARCH_HAS_FAST_MULTIPLIER
select PPC_64S_HASH_MMU
+ select PPC_HAS_LBARX_LHARX
config POWERPC_CPU
bool "Generic 32 bits powerpc"
@@ -160,17 +161,20 @@
depends on PPC_BOOK3S_64
select ARCH_HAS_FAST_MULTIPLIER
select PPC_64S_HASH_MMU
+ select PPC_HAS_LBARX_LHARX
config POWER8_CPU
bool "POWER8"
depends on PPC_BOOK3S_64
select ARCH_HAS_FAST_MULTIPLIER
select PPC_64S_HASH_MMU
+ select PPC_HAS_LBARX_LHARX
config POWER9_CPU
bool "POWER9"
depends on PPC_BOOK3S_64
select ARCH_HAS_FAST_MULTIPLIER
+ select PPC_HAS_LBARX_LHARX
config POWER10_CPU
bool "POWER10"
@@ -184,6 +188,7 @@
config E6500_CPU
bool "Freescale e6500"
depends on PPC64 && PPC_E500
+ select PPC_HAS_LBARX_LHARX
config 405_CPU
bool "40x family"
@@ -575,10 +580,10 @@
endchoice
config PPC64_ELF_ABI_V1
- def_bool PPC64 && CPU_BIG_ENDIAN
+ def_bool PPC64 && (CPU_BIG_ENDIAN && !PPC64_BIG_ENDIAN_ELF_ABI_V2)
config PPC64_ELF_ABI_V2
- def_bool PPC64 && CPU_LITTLE_ENDIAN
+ def_bool PPC64 && !PPC64_ELF_ABI_V1
config PPC64_BOOT_WRAPPER
def_bool n
diff --git a/arch/powerpc/platforms/cell/axon_msi.c b/arch/powerpc/platforms/cell/axon_msi.c
index 5b012ab..0c11aad 100644
--- a/arch/powerpc/platforms/cell/axon_msi.c
+++ b/arch/powerpc/platforms/cell/axon_msi.c
@@ -289,6 +289,7 @@ static void axon_msi_teardown_msi_irqs(struct pci_dev *dev)
msi_for_each_desc(entry, &dev->dev, MSI_DESC_ASSOCIATED) {
irq_set_msi_desc(entry->irq, NULL);
irq_dispose_mapping(entry->irq);
+ entry->irq = 0;
}
}
diff --git a/arch/powerpc/platforms/pasemi/gpio_mdio.c b/arch/powerpc/platforms/pasemi/gpio_mdio.c
index bf30016..913b77b 100644
--- a/arch/powerpc/platforms/pasemi/gpio_mdio.c
+++ b/arch/powerpc/platforms/pasemi/gpio_mdio.c
@@ -294,7 +294,7 @@ static struct platform_driver gpio_mdio_driver =
},
};
-static int gpio_mdio_init(void)
+static int __init gpio_mdio_init(void)
{
struct device_node *np;
@@ -314,7 +314,7 @@ static int gpio_mdio_init(void)
}
module_init(gpio_mdio_init);
-static void gpio_mdio_exit(void)
+static void __exit gpio_mdio_exit(void)
{
platform_driver_unregister(&gpio_mdio_driver);
if (gpio_regs)
diff --git a/arch/powerpc/platforms/pasemi/msi.c b/arch/powerpc/platforms/pasemi/msi.c
index dc18466..166c97f 100644
--- a/arch/powerpc/platforms/pasemi/msi.c
+++ b/arch/powerpc/platforms/pasemi/msi.c
@@ -66,6 +66,7 @@ static void pasemi_msi_teardown_msi_irqs(struct pci_dev *pdev)
hwirq = virq_to_hw(entry->irq);
irq_set_msi_desc(entry->irq, NULL);
irq_dispose_mapping(entry->irq);
+ entry->irq = 0;
msi_bitmap_free_hwirqs(&msi_mpic->msi_bitmap, hwirq, ALLOC_CHUNK);
}
}
diff --git a/arch/powerpc/platforms/powermac/setup.c b/arch/powerpc/platforms/powermac/setup.c
index 04daa7f..4f7ee88 100644
--- a/arch/powerpc/platforms/powermac/setup.c
+++ b/arch/powerpc/platforms/powermac/setup.c
@@ -70,9 +70,7 @@
#undef SHOW_GATWICK_IRQS
-int ppc_override_l2cr = 0;
-int ppc_override_l2cr_value;
-int has_l2cache = 0;
+static int has_l2cache;
int pmac_newworld;
@@ -236,22 +234,16 @@ static void __init l2cr_init(void)
const unsigned int *l2cr =
of_get_property(np, "l2cr-value", NULL);
if (l2cr) {
- ppc_override_l2cr = 1;
- ppc_override_l2cr_value = *l2cr;
_set_L2CR(0);
- _set_L2CR(ppc_override_l2cr_value);
+ _set_L2CR(*l2cr);
+ pr_info("L2CR overridden (0x%x), backside cache is %s\n",
+ *l2cr, ((*l2cr) & 0x80000000) ?
+ "enabled" : "disabled");
}
of_node_put(np);
break;
}
}
-
- if (ppc_override_l2cr)
- printk(KERN_INFO "L2CR overridden (0x%x), "
- "backside cache is %s\n",
- ppc_override_l2cr_value,
- (ppc_override_l2cr_value & 0x80000000)
- ? "enabled" : "disabled");
}
#endif
diff --git a/arch/powerpc/platforms/ps3/system-bus.c b/arch/powerpc/platforms/ps3/system-bus.c
index 2502e9b..38a7e02 100644
--- a/arch/powerpc/platforms/ps3/system-bus.c
+++ b/arch/powerpc/platforms/ps3/system-bus.c
@@ -466,7 +466,7 @@ static struct attribute *ps3_system_bus_dev_attrs[] = {
};
ATTRIBUTE_GROUPS(ps3_system_bus_dev);
-struct bus_type ps3_system_bus_type = {
+static struct bus_type ps3_system_bus_type = {
.name = "ps3_system_bus",
.match = ps3_system_bus_match,
.uevent = ps3_system_bus_uevent,
diff --git a/arch/powerpc/platforms/pseries/eeh_pseries.c b/arch/powerpc/platforms/pseries/eeh_pseries.c
index 8e40cca..6b507b6 100644
--- a/arch/powerpc/platforms/pseries/eeh_pseries.c
+++ b/arch/powerpc/platforms/pseries/eeh_pseries.c
@@ -154,7 +154,7 @@ static int pseries_eeh_get_pe_config_addr(struct pci_dn *pdn)
/**
* pseries_eeh_phb_reset - Reset the specified PHB
* @phb: PCI controller
- * @config_adddr: the associated config address
+ * @config_addr: the associated config address
* @option: reset option
*
* Reset the specified PHB/PE
@@ -188,7 +188,7 @@ static int pseries_eeh_phb_reset(struct pci_controller *phb, int config_addr, in
/**
* pseries_eeh_phb_configure_bridge - Configure PCI bridges in the indicated PE
* @phb: PCI controller
- * @config_adddr: the associated config address
+ * @config_addr: the associated config address
*
* The function will be called to reconfigure the bridges included
* in the specified PE so that the mulfunctional PE would be recovered
@@ -848,16 +848,7 @@ static int __init eeh_pseries_init(void)
}
/* Initialize error log size */
- eeh_error_buf_size = rtas_token("rtas-error-log-max");
- if (eeh_error_buf_size == RTAS_UNKNOWN_SERVICE) {
- pr_info("%s: unknown EEH error log size\n",
- __func__);
- eeh_error_buf_size = 1024;
- } else if (eeh_error_buf_size > RTAS_ERROR_LOG_MAX) {
- pr_info("%s: EEH error log size %d exceeds the maximal %d\n",
- __func__, eeh_error_buf_size, RTAS_ERROR_LOG_MAX);
- eeh_error_buf_size = RTAS_ERROR_LOG_MAX;
- }
+ eeh_error_buf_size = rtas_get_error_log_max();
/* Set EEH probe mode */
eeh_add_flag(EEH_PROBE_MODE_DEVTREE | EEH_ENABLE_IO_FOR_LOG);
diff --git a/arch/powerpc/platforms/pseries/hotplug-cpu.c b/arch/powerpc/platforms/pseries/hotplug-cpu.c
index e0a7ac5..090ae5a 100644
--- a/arch/powerpc/platforms/pseries/hotplug-cpu.c
+++ b/arch/powerpc/platforms/pseries/hotplug-cpu.c
@@ -70,6 +70,7 @@ static void pseries_cpu_offline_self(void)
xics_teardown_cpu();
unregister_slb_shadow(hwcpu);
+ unregister_vpa(hwcpu);
rtas_stop_self();
/* Should never get here... */
diff --git a/arch/powerpc/platforms/pseries/hvCall.S b/arch/powerpc/platforms/pseries/hvCall.S
index 762eb15..783c16a 100644
--- a/arch/powerpc/platforms/pseries/hvCall.S
+++ b/arch/powerpc/platforms/pseries/hvCall.S
@@ -27,7 +27,9 @@
/*
* precall must preserve all registers. use unused STK_PARAM()
- * areas to save snapshots and opcode.
+ * areas to save snapshots and opcode. STK_PARAM() in the caller's
+ * frame will be available even on ELFv2 because these are all
+ * variadic functions.
*/
#define HCALL_INST_PRECALL(FIRST_REG) \
mflr r0; \
@@ -41,29 +43,29 @@
std r10,STK_PARAM(R10)(r1); \
std r0,16(r1); \
addi r4,r1,STK_PARAM(FIRST_REG); \
- stdu r1,-STACK_FRAME_OVERHEAD(r1); \
+ stdu r1,-STACK_FRAME_MIN_SIZE(r1); \
bl __trace_hcall_entry; \
- ld r3,STACK_FRAME_OVERHEAD+STK_PARAM(R3)(r1); \
- ld r4,STACK_FRAME_OVERHEAD+STK_PARAM(R4)(r1); \
- ld r5,STACK_FRAME_OVERHEAD+STK_PARAM(R5)(r1); \
- ld r6,STACK_FRAME_OVERHEAD+STK_PARAM(R6)(r1); \
- ld r7,STACK_FRAME_OVERHEAD+STK_PARAM(R7)(r1); \
- ld r8,STACK_FRAME_OVERHEAD+STK_PARAM(R8)(r1); \
- ld r9,STACK_FRAME_OVERHEAD+STK_PARAM(R9)(r1); \
- ld r10,STACK_FRAME_OVERHEAD+STK_PARAM(R10)(r1)
+ ld r3,STACK_FRAME_MIN_SIZE+STK_PARAM(R3)(r1); \
+ ld r4,STACK_FRAME_MIN_SIZE+STK_PARAM(R4)(r1); \
+ ld r5,STACK_FRAME_MIN_SIZE+STK_PARAM(R5)(r1); \
+ ld r6,STACK_FRAME_MIN_SIZE+STK_PARAM(R6)(r1); \
+ ld r7,STACK_FRAME_MIN_SIZE+STK_PARAM(R7)(r1); \
+ ld r8,STACK_FRAME_MIN_SIZE+STK_PARAM(R8)(r1); \
+ ld r9,STACK_FRAME_MIN_SIZE+STK_PARAM(R9)(r1); \
+ ld r10,STACK_FRAME_MIN_SIZE+STK_PARAM(R10)(r1)
/*
* postcall is performed immediately before function return which
* allows liberal use of volatile registers.
*/
#define __HCALL_INST_POSTCALL \
- ld r0,STACK_FRAME_OVERHEAD+STK_PARAM(R3)(r1); \
- std r3,STACK_FRAME_OVERHEAD+STK_PARAM(R3)(r1); \
+ ld r0,STACK_FRAME_MIN_SIZE+STK_PARAM(R3)(r1); \
+ std r3,STACK_FRAME_MIN_SIZE+STK_PARAM(R3)(r1); \
mr r4,r3; \
mr r3,r0; \
bl __trace_hcall_exit; \
- ld r0,STACK_FRAME_OVERHEAD+16(r1); \
- addi r1,r1,STACK_FRAME_OVERHEAD; \
+ ld r0,STACK_FRAME_MIN_SIZE+16(r1); \
+ addi r1,r1,STACK_FRAME_MIN_SIZE; \
ld r3,STK_PARAM(R3)(r1); \
mtlr r0
@@ -303,14 +305,14 @@
mr r7,r8
mr r8,r9
mr r9,r10
- ld r10,STACK_FRAME_OVERHEAD+STK_PARAM(R11)(r1)
- ld r11,STACK_FRAME_OVERHEAD+STK_PARAM(R12)(r1)
- ld r12,STACK_FRAME_OVERHEAD+STK_PARAM(R13)(r1)
+ ld r10,STACK_FRAME_MIN_SIZE+STK_PARAM(R11)(r1)
+ ld r11,STACK_FRAME_MIN_SIZE+STK_PARAM(R12)(r1)
+ ld r12,STACK_FRAME_MIN_SIZE+STK_PARAM(R13)(r1)
HVSC
mr r0,r12
- ld r12,STACK_FRAME_OVERHEAD+STK_PARAM(R4)(r1)
+ ld r12,STACK_FRAME_MIN_SIZE+STK_PARAM(R4)(r1)
std r4,0(r12)
std r5,8(r12)
std r6,16(r12)
diff --git a/arch/powerpc/platforms/pseries/iommu.c b/arch/powerpc/platforms/pseries/iommu.c
index 561adac..c74b71d 100644
--- a/arch/powerpc/platforms/pseries/iommu.c
+++ b/arch/powerpc/platforms/pseries/iommu.c
@@ -248,7 +248,7 @@ static int tce_buildmulti_pSeriesLP(struct iommu_table *tbl, long tcenum,
* Set up the page with TCE data, looping through and setting
* the values.
*/
- limit = min_t(long, npages, 4096/TCE_ENTRY_SIZE);
+ limit = min_t(long, npages, 4096 / TCE_ENTRY_SIZE);
for (l = 0; l < limit; l++) {
tcep[l] = cpu_to_be64(proto_tce | rpn << tceshift);
diff --git a/arch/powerpc/platforms/pseries/mobility.c b/arch/powerpc/platforms/pseries/mobility.c
index 634fac5..4cea71a 100644
--- a/arch/powerpc/platforms/pseries/mobility.c
+++ b/arch/powerpc/platforms/pseries/mobility.c
@@ -635,10 +635,13 @@ static int do_join(void *arg)
prod_others();
}
/*
- * Execution may have been suspended for several seconds, so
- * reset the watchdog.
+ * Execution may have been suspended for several seconds, so reset
+ * the watchdogs. touch_nmi_watchdog() also touches the soft lockup
+ * watchdog.
*/
+ rcu_cpu_stall_reset();
touch_nmi_watchdog();
+
return ret;
}
diff --git a/arch/powerpc/platforms/pseries/plpks.c b/arch/powerpc/platforms/pseries/plpks.c
index f4b5b5a..4edd158 100644
--- a/arch/powerpc/platforms/pseries/plpks.c
+++ b/arch/powerpc/platforms/pseries/plpks.c
@@ -75,7 +75,7 @@ static int pseries_status_to_err(int rc)
case H_FUNCTION:
err = -ENXIO;
break;
- case H_P1:
+ case H_PARAMETER:
case H_P2:
case H_P3:
case H_P4:
@@ -111,7 +111,7 @@ static int pseries_status_to_err(int rc)
err = -EEXIST;
break;
case H_ABORTED:
- err = -EINTR;
+ err = -EIO;
break;
default:
err = -EINVAL;
@@ -162,19 +162,15 @@ static struct plpks_auth *construct_auth(u8 consumer)
if (consumer > PKS_OS_OWNER)
return ERR_PTR(-EINVAL);
- auth = kmalloc(struct_size(auth, password, maxpwsize), GFP_KERNEL);
+ auth = kzalloc(struct_size(auth, password, maxpwsize), GFP_KERNEL);
if (!auth)
return ERR_PTR(-ENOMEM);
auth->version = 1;
auth->consumer = consumer;
- auth->rsvd0 = 0;
- auth->rsvd1 = 0;
- if (consumer == PKS_FW_OWNER || consumer == PKS_BOOTLOADER_OWNER) {
- auth->passwordlength = 0;
+ if (consumer == PKS_FW_OWNER || consumer == PKS_BOOTLOADER_OWNER)
return auth;
- }
memcpy(auth->password, ospassword, ospasswordlength);
@@ -312,10 +308,6 @@ int plpks_write_var(struct plpks_var var)
if (!rc)
rc = plpks_confirm_object_flushed(label, auth);
- if (rc)
- pr_err("Failed to write variable %s for component %s with error %d\n",
- var.name, var.component, rc);
-
rc = pseries_status_to_err(rc);
kfree(label);
out:
@@ -350,10 +342,6 @@ int plpks_remove_var(char *component, u8 varos, struct plpks_var_name vname)
if (!rc)
rc = plpks_confirm_object_flushed(label, auth);
- if (rc)
- pr_err("Failed to remove variable %s for component %s with error %d\n",
- vname.name, component, rc);
-
rc = pseries_status_to_err(rc);
kfree(label);
out:
@@ -366,22 +354,24 @@ static int plpks_read_var(u8 consumer, struct plpks_var *var)
{
unsigned long retbuf[PLPAR_HCALL_BUFSIZE] = { 0 };
struct plpks_auth *auth;
- struct label *label;
+ struct label *label = NULL;
u8 *output;
int rc;
if (var->namelen > MAX_NAME_SIZE)
return -EINVAL;
- auth = construct_auth(PKS_OS_OWNER);
+ auth = construct_auth(consumer);
if (IS_ERR(auth))
return PTR_ERR(auth);
- label = construct_label(var->component, var->os, var->name,
- var->namelen);
- if (IS_ERR(label)) {
- rc = PTR_ERR(label);
- goto out_free_auth;
+ if (consumer == PKS_OS_OWNER) {
+ label = construct_label(var->component, var->os, var->name,
+ var->namelen);
+ if (IS_ERR(label)) {
+ rc = PTR_ERR(label);
+ goto out_free_auth;
+ }
}
output = kzalloc(maxobjsize, GFP_KERNEL);
@@ -390,13 +380,17 @@ static int plpks_read_var(u8 consumer, struct plpks_var *var)
goto out_free_label;
}
- rc = plpar_hcall(H_PKS_READ_OBJECT, retbuf, virt_to_phys(auth),
- virt_to_phys(label), label->size, virt_to_phys(output),
- maxobjsize);
+ if (consumer == PKS_OS_OWNER)
+ rc = plpar_hcall(H_PKS_READ_OBJECT, retbuf, virt_to_phys(auth),
+ virt_to_phys(label), label->size, virt_to_phys(output),
+ maxobjsize);
+ else
+ rc = plpar_hcall(H_PKS_READ_OBJECT, retbuf, virt_to_phys(auth),
+ virt_to_phys(var->name), var->namelen, virt_to_phys(output),
+ maxobjsize);
+
if (rc != H_SUCCESS) {
- pr_err("Failed to read variable %s for component %s with error %d\n",
- var->name, var->component, rc);
rc = pseries_status_to_err(rc);
goto out_free_output;
}
diff --git a/arch/powerpc/platforms/pseries/plpks.h b/arch/powerpc/platforms/pseries/plpks.h
index c6a2913..275ccd8 100644
--- a/arch/powerpc/platforms/pseries/plpks.h
+++ b/arch/powerpc/platforms/pseries/plpks.h
@@ -17,7 +17,7 @@
#define WORLDREADABLE 0x08000000
#define SIGNEDUPDATE 0x01000000
-#define PLPKS_VAR_LINUX 0x01
+#define PLPKS_VAR_LINUX 0x02
#define PLPKS_VAR_COMMON 0x04
struct plpks_var {
diff --git a/arch/powerpc/sysdev/fsl_msi.c b/arch/powerpc/sysdev/fsl_msi.c
index 73c2d707..57978a4 100644
--- a/arch/powerpc/sysdev/fsl_msi.c
+++ b/arch/powerpc/sysdev/fsl_msi.c
@@ -132,6 +132,7 @@ static void fsl_teardown_msi_irqs(struct pci_dev *pdev)
msi_data = irq_get_chip_data(entry->irq);
irq_set_msi_desc(entry->irq, NULL);
irq_dispose_mapping(entry->irq);
+ entry->irq = 0;
msi_bitmap_free_hwirqs(&msi_data->bitmap, hwirq, 1);
}
}
diff --git a/arch/powerpc/sysdev/fsl_pci.c b/arch/powerpc/sysdev/fsl_pci.c
index 974d3db..b7232c4 100644
--- a/arch/powerpc/sysdev/fsl_pci.c
+++ b/arch/powerpc/sysdev/fsl_pci.c
@@ -1139,6 +1139,19 @@ void __init fsl_pci_assign_primary(void)
}
/*
+ * If there's no PCI host bridge with ISA then check for
+ * PCI host bridge with alias "pci0" (first PCI host bridge).
+ */
+ np = of_find_node_by_path("pci0");
+ if (np && of_match_node(pci_ids, np) && of_device_is_available(np)) {
+ fsl_pci_primary = np;
+ of_node_put(np);
+ return;
+ }
+ if (np)
+ of_node_put(np);
+
+ /*
* If there's no PCI host bridge with ISA, arbitrarily
* designate one as primary. This can go away once
* various bugs with primary-less systems are fixed.
diff --git a/arch/powerpc/sysdev/mpic_msgr.c b/arch/powerpc/sysdev/mpic_msgr.c
index a439e33..d75064f 100644
--- a/arch/powerpc/sysdev/mpic_msgr.c
+++ b/arch/powerpc/sysdev/mpic_msgr.c
@@ -20,7 +20,7 @@
#define MPIC_MSGR_REGISTERS_PER_BLOCK 4
#define MPIC_MSGR_STRIDE 0x10
-#define MPIC_MSGR_MER_OFFSET 0x100
+#define MPIC_MSGR_MER_OFFSET (0x100 / sizeof(u32))
#define MSGR_INUSE 0
#define MSGR_FREE 1
@@ -234,7 +234,7 @@ static int mpic_msgr_probe(struct platform_device *dev)
reg_number = block_number * MPIC_MSGR_REGISTERS_PER_BLOCK + i;
msgr->base = msgr_block_addr + i * MPIC_MSGR_STRIDE;
- msgr->mer = (u32 *)((u8 *)msgr->base + MPIC_MSGR_MER_OFFSET);
+ msgr->mer = msgr->base + MPIC_MSGR_MER_OFFSET;
msgr->in_use = MSGR_FREE;
msgr->num = i;
raw_spin_lock_init(&msgr->lock);
diff --git a/arch/powerpc/sysdev/mpic_u3msi.c b/arch/powerpc/sysdev/mpic_u3msi.c
index 1d8cfdf..492cb03 100644
--- a/arch/powerpc/sysdev/mpic_u3msi.c
+++ b/arch/powerpc/sysdev/mpic_u3msi.c
@@ -108,6 +108,7 @@ static void u3msi_teardown_msi_irqs(struct pci_dev *pdev)
hwirq = virq_to_hw(entry->irq);
irq_set_msi_desc(entry->irq, NULL);
irq_dispose_mapping(entry->irq);
+ entry->irq = 0;
msi_bitmap_free_hwirqs(&msi_mpic->msi_bitmap, hwirq, 1);
}
}
diff --git a/arch/powerpc/sysdev/xive/native.c b/arch/powerpc/sysdev/xive/native.c
index 3925825..19d880e 100644
--- a/arch/powerpc/sysdev/xive/native.c
+++ b/arch/powerpc/sysdev/xive/native.c
@@ -535,13 +535,13 @@ static bool __init xive_parse_provisioning(struct device_node *np)
static void __init xive_native_setup_pools(void)
{
/* Allocate a pool big enough */
- pr_debug("XIVE: Allocating VP block for pool size %u\n", nr_cpu_ids);
+ pr_debug("Allocating VP block for pool size %u\n", nr_cpu_ids);
xive_pool_vps = xive_native_alloc_vp_block(nr_cpu_ids);
if (WARN_ON(xive_pool_vps == XIVE_INVALID_VP))
- pr_err("XIVE: Failed to allocate pool VP, KVM might not function\n");
+ pr_err("Failed to allocate pool VP, KVM might not function\n");
- pr_debug("XIVE: Pool VPs allocated at 0x%x for %u max CPUs\n",
+ pr_debug("Pool VPs allocated at 0x%x for %u max CPUs\n",
xive_pool_vps, nr_cpu_ids);
}
diff --git a/arch/powerpc/sysdev/xive/spapr.c b/arch/powerpc/sysdev/xive/spapr.c
index e2c8f93..e454192 100644
--- a/arch/powerpc/sysdev/xive/spapr.c
+++ b/arch/powerpc/sysdev/xive/spapr.c
@@ -439,6 +439,7 @@ static int xive_spapr_populate_irq_data(u32 hw_irq, struct xive_irq_data *data)
data->trig_mmio = ioremap(data->trig_page, 1u << data->esb_shift);
if (!data->trig_mmio) {
+ iounmap(data->eoi_mmio);
pr_err("Failed to map trigger page for irq 0x%x\n", hw_irq);
return -ENOMEM;
}
diff --git a/arch/powerpc/xmon/xmon.c b/arch/powerpc/xmon/xmon.c
index f51c882..0da66bc 100644
--- a/arch/powerpc/xmon/xmon.c
+++ b/arch/powerpc/xmon/xmon.c
@@ -1525,9 +1525,9 @@ bpt_cmds(void)
cmd = inchar();
switch (cmd) {
- static const char badaddr[] = "Only kernel addresses are permitted for breakpoints\n";
- int mode;
- case 'd': /* bd - hardware data breakpoint */
+ case 'd': { /* bd - hardware data breakpoint */
+ static const char badaddr[] = "Only kernel addresses are permitted for breakpoints\n";
+ int mode;
if (xmon_is_ro) {
printf(xmon_ro_msg);
break;
@@ -1560,6 +1560,7 @@ bpt_cmds(void)
force_enable_xmon();
break;
+ }
case 'i': /* bi - hardware instr breakpoint */
if (xmon_is_ro) {
@@ -1720,7 +1721,6 @@ static void get_function_bounds(unsigned long pc, unsigned long *startp,
}
#define LRSAVE_OFFSET (STACK_FRAME_LR_SAVE * sizeof(unsigned long))
-#define MARKER_OFFSET (STACK_FRAME_MARKER * sizeof(unsigned long))
static void xmon_show_stack(unsigned long sp, unsigned long lr,
unsigned long pc)
@@ -1781,14 +1781,13 @@ static void xmon_show_stack(unsigned long sp, unsigned long lr,
xmon_print_symbol(ip, " ", "\n");
}
- /* Look for "regshere" marker to see if this is
+ /* Look for "regs" marker to see if this is
an exception frame. */
- if (mread(sp + MARKER_OFFSET, &marker, sizeof(unsigned long))
+ if (mread(sp + STACK_INT_FRAME_MARKER, &marker, sizeof(unsigned long))
&& marker == STACK_FRAME_REGS_MARKER) {
- if (mread(sp + STACK_FRAME_OVERHEAD, ®s, sizeof(regs))
- != sizeof(regs)) {
+ if (mread(sp + STACK_INT_FRAME_REGS, ®s, sizeof(regs)) != sizeof(regs)) {
printf("Couldn't read registers at %lx\n",
- sp + STACK_FRAME_OVERHEAD);
+ sp + STACK_INT_FRAME_REGS);
break;
}
printf("--- Exception: %lx %s at ", regs.trap,
diff --git a/arch/riscv/Makefile b/arch/riscv/Makefile
index 0d13b59..faf2c21 100644
--- a/arch/riscv/Makefile
+++ b/arch/riscv/Makefile
@@ -37,7 +37,7 @@
endif
ifeq ($(CONFIG_LD_IS_LLD),y)
-ifeq ($(shell test $(CONFIG_LLD_VERSION) -lt 150000; echo $$?),0)
+ifeq ($(call test-lt, $(CONFIG_LLD_VERSION), 150000),y)
KBUILD_CFLAGS += -mno-relax
KBUILD_AFLAGS += -mno-relax
ifndef CONFIG_AS_IS_LLVM
diff --git a/arch/s390/include/asm/pci.h b/arch/s390/include/asm/pci.h
index 108e732..b248694 100644
--- a/arch/s390/include/asm/pci.h
+++ b/arch/s390/include/asm/pci.h
@@ -117,7 +117,9 @@ struct zpci_bus {
struct zpci_dev {
struct zpci_bus *zbus;
struct list_head entry; /* list of all zpci_devices, needed for hotplug, etc. */
+ struct list_head iommu_list;
struct kref kref;
+ struct rcu_head rcu;
struct hotplug_slot hotplug_slot;
enum zpci_state state;
@@ -155,7 +157,6 @@ struct zpci_dev {
/* DMA stuff */
unsigned long *dma_table;
- spinlock_t dma_table_lock;
int tlb_refresh;
spinlock_t iommu_bitmap_lock;
@@ -220,7 +221,7 @@ void zpci_device_reserved(struct zpci_dev *zdev);
bool zpci_is_device_configured(struct zpci_dev *zdev);
int zpci_hot_reset_device(struct zpci_dev *zdev);
-int zpci_register_ioat(struct zpci_dev *, u8, u64, u64, u64);
+int zpci_register_ioat(struct zpci_dev *, u8, u64, u64, u64, u8 *);
int zpci_unregister_ioat(struct zpci_dev *, u8);
void zpci_remove_reserved_devices(void);
void zpci_update_fh(struct zpci_dev *zdev, u32 fh);
diff --git a/arch/s390/kvm/pci.c b/arch/s390/kvm/pci.c
index ded1af2..ec51e81 100644
--- a/arch/s390/kvm/pci.c
+++ b/arch/s390/kvm/pci.c
@@ -434,6 +434,7 @@ static void kvm_s390_pci_dev_release(struct zpci_dev *zdev)
static int kvm_s390_pci_register_kvm(void *opaque, struct kvm *kvm)
{
struct zpci_dev *zdev = opaque;
+ u8 status;
int rc;
if (!zdev)
@@ -486,7 +487,7 @@ static int kvm_s390_pci_register_kvm(void *opaque, struct kvm *kvm)
/* Re-register the IOMMU that was already created */
rc = zpci_register_ioat(zdev, 0, zdev->start_dma, zdev->end_dma,
- virt_to_phys(zdev->dma_table));
+ virt_to_phys(zdev->dma_table), &status);
if (rc)
goto clear_gisa;
@@ -516,6 +517,7 @@ static void kvm_s390_pci_unregister_kvm(void *opaque)
{
struct zpci_dev *zdev = opaque;
struct kvm *kvm;
+ u8 status;
if (!zdev)
return;
@@ -554,7 +556,7 @@ static void kvm_s390_pci_unregister_kvm(void *opaque)
/* Re-register the IOMMU that was already created */
zpci_register_ioat(zdev, 0, zdev->start_dma, zdev->end_dma,
- virt_to_phys(zdev->dma_table));
+ virt_to_phys(zdev->dma_table), &status);
out:
spin_lock(&kvm->arch.kzdev_list_lock);
diff --git a/arch/s390/pci/pci.c b/arch/s390/pci/pci.c
index 73cdc55..ef38b15 100644
--- a/arch/s390/pci/pci.c
+++ b/arch/s390/pci/pci.c
@@ -116,20 +116,20 @@ EXPORT_SYMBOL_GPL(pci_proc_domain);
/* Modify PCI: Register I/O address translation parameters */
int zpci_register_ioat(struct zpci_dev *zdev, u8 dmaas,
- u64 base, u64 limit, u64 iota)
+ u64 base, u64 limit, u64 iota, u8 *status)
{
u64 req = ZPCI_CREATE_REQ(zdev->fh, dmaas, ZPCI_MOD_FC_REG_IOAT);
struct zpci_fib fib = {0};
- u8 cc, status;
+ u8 cc;
WARN_ON_ONCE(iota & 0x3fff);
fib.pba = base;
fib.pal = limit;
fib.iota = iota | ZPCI_IOTA_RTTO_FLAG;
fib.gd = zdev->gisa;
- cc = zpci_mod_fc(req, &fib, &status);
+ cc = zpci_mod_fc(req, &fib, status);
if (cc)
- zpci_dbg(3, "reg ioat fid:%x, cc:%d, status:%d\n", zdev->fid, cc, status);
+ zpci_dbg(3, "reg ioat fid:%x, cc:%d, status:%d\n", zdev->fid, cc, *status);
return cc;
}
EXPORT_SYMBOL_GPL(zpci_register_ioat);
@@ -764,6 +764,7 @@ EXPORT_SYMBOL_GPL(zpci_disable_device);
*/
int zpci_hot_reset_device(struct zpci_dev *zdev)
{
+ u8 status;
int rc;
zpci_dbg(3, "rst fid:%x, fh:%x\n", zdev->fid, zdev->fh);
@@ -787,7 +788,7 @@ int zpci_hot_reset_device(struct zpci_dev *zdev)
if (zdev->dma_table)
rc = zpci_register_ioat(zdev, 0, zdev->start_dma, zdev->end_dma,
- virt_to_phys(zdev->dma_table));
+ virt_to_phys(zdev->dma_table), &status);
else
rc = zpci_dma_init_device(zdev);
if (rc) {
@@ -995,7 +996,7 @@ void zpci_release_device(struct kref *kref)
break;
}
zpci_dbg(3, "rem fid:%x\n", zdev->fid);
- kfree(zdev);
+ kfree_rcu(zdev, rcu);
}
int zpci_report_error(struct pci_dev *pdev,
diff --git a/arch/s390/pci/pci_dma.c b/arch/s390/pci/pci_dma.c
index 227cf0a..ea478d1 100644
--- a/arch/s390/pci/pci_dma.c
+++ b/arch/s390/pci/pci_dma.c
@@ -63,37 +63,55 @@ static void dma_free_page_table(void *table)
kmem_cache_free(dma_page_table_cache, table);
}
-static unsigned long *dma_get_seg_table_origin(unsigned long *entry)
+static unsigned long *dma_get_seg_table_origin(unsigned long *rtep)
{
+ unsigned long old_rte, rte;
unsigned long *sto;
- if (reg_entry_isvalid(*entry))
- sto = get_rt_sto(*entry);
- else {
+ rte = READ_ONCE(*rtep);
+ if (reg_entry_isvalid(rte)) {
+ sto = get_rt_sto(rte);
+ } else {
sto = dma_alloc_cpu_table();
if (!sto)
return NULL;
- set_rt_sto(entry, virt_to_phys(sto));
- validate_rt_entry(entry);
- entry_clr_protected(entry);
+ set_rt_sto(&rte, virt_to_phys(sto));
+ validate_rt_entry(&rte);
+ entry_clr_protected(&rte);
+
+ old_rte = cmpxchg(rtep, ZPCI_TABLE_INVALID, rte);
+ if (old_rte != ZPCI_TABLE_INVALID) {
+ /* Somone else was faster, use theirs */
+ dma_free_cpu_table(sto);
+ sto = get_rt_sto(old_rte);
+ }
}
return sto;
}
-static unsigned long *dma_get_page_table_origin(unsigned long *entry)
+static unsigned long *dma_get_page_table_origin(unsigned long *step)
{
+ unsigned long old_ste, ste;
unsigned long *pto;
- if (reg_entry_isvalid(*entry))
- pto = get_st_pto(*entry);
- else {
+ ste = READ_ONCE(*step);
+ if (reg_entry_isvalid(ste)) {
+ pto = get_st_pto(ste);
+ } else {
pto = dma_alloc_page_table();
if (!pto)
return NULL;
- set_st_pto(entry, virt_to_phys(pto));
- validate_st_entry(entry);
- entry_clr_protected(entry);
+ set_st_pto(&ste, virt_to_phys(pto));
+ validate_st_entry(&ste);
+ entry_clr_protected(&ste);
+
+ old_ste = cmpxchg(step, ZPCI_TABLE_INVALID, ste);
+ if (old_ste != ZPCI_TABLE_INVALID) {
+ /* Somone else was faster, use theirs */
+ dma_free_page_table(pto);
+ pto = get_st_pto(old_ste);
+ }
}
return pto;
}
@@ -117,19 +135,24 @@ unsigned long *dma_walk_cpu_trans(unsigned long *rto, dma_addr_t dma_addr)
return &pto[px];
}
-void dma_update_cpu_trans(unsigned long *entry, phys_addr_t page_addr, int flags)
+void dma_update_cpu_trans(unsigned long *ptep, phys_addr_t page_addr, int flags)
{
+ unsigned long pte;
+
+ pte = READ_ONCE(*ptep);
if (flags & ZPCI_PTE_INVALID) {
- invalidate_pt_entry(entry);
+ invalidate_pt_entry(&pte);
} else {
- set_pt_pfaa(entry, page_addr);
- validate_pt_entry(entry);
+ set_pt_pfaa(&pte, page_addr);
+ validate_pt_entry(&pte);
}
if (flags & ZPCI_TABLE_PROTECTED)
- entry_set_protected(entry);
+ entry_set_protected(&pte);
else
- entry_clr_protected(entry);
+ entry_clr_protected(&pte);
+
+ xchg(ptep, pte);
}
static int __dma_update_trans(struct zpci_dev *zdev, phys_addr_t pa,
@@ -137,18 +160,14 @@ static int __dma_update_trans(struct zpci_dev *zdev, phys_addr_t pa,
{
unsigned int nr_pages = PAGE_ALIGN(size) >> PAGE_SHIFT;
phys_addr_t page_addr = (pa & PAGE_MASK);
- unsigned long irq_flags;
unsigned long *entry;
int i, rc = 0;
if (!nr_pages)
return -EINVAL;
- spin_lock_irqsave(&zdev->dma_table_lock, irq_flags);
- if (!zdev->dma_table) {
- rc = -EINVAL;
- goto out_unlock;
- }
+ if (!zdev->dma_table)
+ return -EINVAL;
for (i = 0; i < nr_pages; i++) {
entry = dma_walk_cpu_trans(zdev->dma_table, dma_addr);
@@ -173,8 +192,6 @@ static int __dma_update_trans(struct zpci_dev *zdev, phys_addr_t pa,
dma_update_cpu_trans(entry, page_addr, flags);
}
}
-out_unlock:
- spin_unlock_irqrestore(&zdev->dma_table_lock, irq_flags);
return rc;
}
@@ -547,6 +564,7 @@ static void s390_dma_unmap_sg(struct device *dev, struct scatterlist *sg,
int zpci_dma_init_device(struct zpci_dev *zdev)
{
+ u8 status;
int rc;
/*
@@ -557,7 +575,6 @@ int zpci_dma_init_device(struct zpci_dev *zdev)
WARN_ON(zdev->s390_domain);
spin_lock_init(&zdev->iommu_bitmap_lock);
- spin_lock_init(&zdev->dma_table_lock);
zdev->dma_table = dma_alloc_cpu_table();
if (!zdev->dma_table) {
@@ -598,7 +615,7 @@ int zpci_dma_init_device(struct zpci_dev *zdev)
}
if (zpci_register_ioat(zdev, 0, zdev->start_dma, zdev->end_dma,
- virt_to_phys(zdev->dma_table))) {
+ virt_to_phys(zdev->dma_table), &status)) {
rc = -EIO;
goto free_bitmap;
}
diff --git a/arch/sh/Kconfig b/arch/sh/Kconfig
index 5f220e9..0665ac0 100644
--- a/arch/sh/Kconfig
+++ b/arch/sh/Kconfig
@@ -24,7 +24,7 @@
select GENERIC_PCI_IOMAP if PCI
select GENERIC_SCHED_CLOCK
select GENERIC_SMP_IDLE_THREAD
- select GUP_GET_PTE_LOW_HIGH if X2TLB
+ select GUP_GET_PXX_LOW_HIGH if X2TLB
select HAVE_ARCH_AUDITSYSCALL
select HAVE_ARCH_KGDB
select HAVE_ARCH_SECCOMP_FILTER
diff --git a/arch/sh/include/asm/pgtable-3level.h b/arch/sh/include/asm/pgtable-3level.h
index cdced80..a889a3a 100644
--- a/arch/sh/include/asm/pgtable-3level.h
+++ b/arch/sh/include/asm/pgtable-3level.h
@@ -28,9 +28,15 @@
#define pmd_ERROR(e) \
printk("%s:%d: bad pmd %016llx.\n", __FILE__, __LINE__, pmd_val(e))
-typedef struct { unsigned long long pmd; } pmd_t;
+typedef struct {
+ struct {
+ unsigned long pmd_low;
+ unsigned long pmd_high;
+ };
+ unsigned long long pmd;
+} pmd_t;
#define pmd_val(x) ((x).pmd)
-#define __pmd(x) ((pmd_t) { (x) } )
+#define __pmd(x) ((pmd_t) { .pmd = (x) } )
static inline pmd_t *pud_pgtable(pud_t pud)
{
diff --git a/arch/um/include/asm/pgtable-3level.h b/arch/um/include/asm/pgtable-3level.h
index cb896e6..8a5032e 100644
--- a/arch/um/include/asm/pgtable-3level.h
+++ b/arch/um/include/asm/pgtable-3level.h
@@ -58,11 +58,7 @@
#define pud_populate(mm, pud, pmd) \
set_pud(pud, __pud(_PAGE_TABLE + __pa(pmd)))
-#ifdef CONFIG_64BIT
-#define set_pud(pudptr, pudval) set_64bit((u64 *) (pudptr), pud_val(pudval))
-#else
#define set_pud(pudptr, pudval) (*(pudptr) = (pudval))
-#endif
static inline int pgd_newpage(pgd_t pgd)
{
@@ -71,11 +67,7 @@ static inline int pgd_newpage(pgd_t pgd)
static inline void pgd_mkuptodate(pgd_t pgd) { pgd_val(pgd) &= ~_PAGE_NEWPAGE; }
-#ifdef CONFIG_64BIT
-#define set_pmd(pmdptr, pmdval) set_64bit((u64 *) (pmdptr), pmd_val(pmdval))
-#else
#define set_pmd(pmdptr, pmdval) (*(pmdptr) = (pmdval))
-#endif
static inline void pud_clear (pud_t *pud)
{
diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig
index 7577c33..3604074 100644
--- a/arch/x86/Kconfig
+++ b/arch/x86/Kconfig
@@ -159,7 +159,7 @@
select GENERIC_TIME_VSYSCALL
select GENERIC_GETTIMEOFDAY
select GENERIC_VDSO_TIME_NS
- select GUP_GET_PTE_LOW_HIGH if X86_PAE
+ select GUP_GET_PXX_LOW_HIGH if X86_PAE
select HARDIRQS_SW_RESEND
select HARDLOCKUP_CHECK_TIMESTAMP if X86_64
select HAVE_ACPI_APEI if ACPI
@@ -197,6 +197,7 @@
select HAVE_CONTEXT_TRACKING_USER_OFFSTACK if HAVE_CONTEXT_TRACKING_USER
select HAVE_C_RECORDMCOUNT
select HAVE_OBJTOOL_MCOUNT if HAVE_OBJTOOL
+ select HAVE_OBJTOOL_NOP_MCOUNT if HAVE_OBJTOOL_MCOUNT
select HAVE_BUILDTIME_MCOUNT_SORT
select HAVE_DEBUG_KMEMLEAK
select HAVE_DMA_CONTIGUOUS
diff --git a/arch/x86/Makefile b/arch/x86/Makefile
index a3a07df8..9cf0732 100644
--- a/arch/x86/Makefile
+++ b/arch/x86/Makefile
@@ -217,7 +217,7 @@
KBUILD_LDFLAGS += -m elf_$(UTS_MACHINE)
ifdef CONFIG_LTO_CLANG
-ifeq ($(shell test $(CONFIG_LLD_VERSION) -lt 130000; echo $$?),0)
+ifeq ($(call test-lt, $(CONFIG_LLD_VERSION), 130000),y)
KBUILD_LDFLAGS += -plugin-opt=-stack-alignment=$(if $(CONFIG_X86_32),4,8)
endif
endif
diff --git a/arch/x86/include/asm/cmpxchg_32.h b/arch/x86/include/asm/cmpxchg_32.h
index 215f5a6..6ba80ce 100644
--- a/arch/x86/include/asm/cmpxchg_32.h
+++ b/arch/x86/include/asm/cmpxchg_32.h
@@ -7,34 +7,6 @@
* you need to test for the feature in boot_cpu_data.
*/
-/*
- * CMPXCHG8B only writes to the target if we had the previous
- * value in registers, otherwise it acts as a read and gives us the
- * "new previous" value. That is why there is a loop. Preloading
- * EDX:EAX is a performance optimization: in the common case it means
- * we need only one locked operation.
- *
- * A SIMD/3DNOW!/MMX/FPU 64-bit store here would require at the very
- * least an FPU save and/or %cr0.ts manipulation.
- *
- * cmpxchg8b must be used with the lock prefix here to allow the
- * instruction to be executed atomically. We need to have the reader
- * side to see the coherent 64bit value.
- */
-static inline void set_64bit(volatile u64 *ptr, u64 value)
-{
- u32 low = value;
- u32 high = value >> 32;
- u64 prev = *ptr;
-
- asm volatile("\n1:\t"
- LOCK_PREFIX "cmpxchg8b %0\n\t"
- "jnz 1b"
- : "=m" (*ptr), "+A" (prev)
- : "b" (low), "c" (high)
- : "memory");
-}
-
#ifdef CONFIG_X86_CMPXCHG64
#define arch_cmpxchg64(ptr, o, n) \
((__typeof__(*(ptr)))__cmpxchg64((ptr), (unsigned long long)(o), \
diff --git a/arch/x86/include/asm/cmpxchg_64.h b/arch/x86/include/asm/cmpxchg_64.h
index 250187a..0d3beb2 100644
--- a/arch/x86/include/asm/cmpxchg_64.h
+++ b/arch/x86/include/asm/cmpxchg_64.h
@@ -2,11 +2,6 @@
#ifndef _ASM_X86_CMPXCHG_64_H
#define _ASM_X86_CMPXCHG_64_H
-static inline void set_64bit(volatile u64 *ptr, u64 val)
-{
- *ptr = val;
-}
-
#define arch_cmpxchg64(ptr, o, n) \
({ \
BUILD_BUG_ON(sizeof(*(ptr)) != 8); \
diff --git a/arch/x86/include/asm/cpu_entry_area.h b/arch/x86/include/asm/cpu_entry_area.h
index 75efc4c..462fc34 100644
--- a/arch/x86/include/asm/cpu_entry_area.h
+++ b/arch/x86/include/asm/cpu_entry_area.h
@@ -130,10 +130,6 @@ struct cpu_entry_area {
};
#define CPU_ENTRY_AREA_SIZE (sizeof(struct cpu_entry_area))
-#define CPU_ENTRY_AREA_ARRAY_SIZE (CPU_ENTRY_AREA_SIZE * NR_CPUS)
-
-/* Total size includes the readonly IDT mapping page as well: */
-#define CPU_ENTRY_AREA_TOTAL_SIZE (CPU_ENTRY_AREA_ARRAY_SIZE + PAGE_SIZE)
DECLARE_PER_CPU(struct cpu_entry_area *, cpu_entry_area);
DECLARE_PER_CPU(struct cea_exception_stacks *, cea_exception_stacks);
diff --git a/arch/x86/include/asm/kasan.h b/arch/x86/include/asm/kasan.h
index 13e70da..de75306 100644
--- a/arch/x86/include/asm/kasan.h
+++ b/arch/x86/include/asm/kasan.h
@@ -28,9 +28,12 @@
#ifdef CONFIG_KASAN
void __init kasan_early_init(void);
void __init kasan_init(void);
+void __init kasan_populate_shadow_for_vaddr(void *va, size_t size, int nid);
#else
static inline void kasan_early_init(void) { }
static inline void kasan_init(void) { }
+static inline void kasan_populate_shadow_for_vaddr(void *va, size_t size,
+ int nid) { }
#endif
#endif
diff --git a/arch/x86/include/asm/page_types.h b/arch/x86/include/asm/page_types.h
index a506a41..86bd431 100644
--- a/arch/x86/include/asm/page_types.h
+++ b/arch/x86/include/asm/page_types.h
@@ -11,20 +11,14 @@
#define PAGE_SIZE (_AC(1,UL) << PAGE_SHIFT)
#define PAGE_MASK (~(PAGE_SIZE-1))
-#define PMD_PAGE_SIZE (_AC(1, UL) << PMD_SHIFT)
-#define PMD_PAGE_MASK (~(PMD_PAGE_SIZE-1))
-
-#define PUD_PAGE_SIZE (_AC(1, UL) << PUD_SHIFT)
-#define PUD_PAGE_MASK (~(PUD_PAGE_SIZE-1))
-
#define __VIRTUAL_MASK ((1UL << __VIRTUAL_MASK_SHIFT) - 1)
-/* Cast *PAGE_MASK to a signed type so that it is sign-extended if
+/* Cast P*D_MASK to a signed type so that it is sign-extended if
virtual addresses are 32-bits but physical addresses are larger
(ie, 32-bit PAE). */
#define PHYSICAL_PAGE_MASK (((signed long)PAGE_MASK) & __PHYSICAL_MASK)
-#define PHYSICAL_PMD_PAGE_MASK (((signed long)PMD_PAGE_MASK) & __PHYSICAL_MASK)
-#define PHYSICAL_PUD_PAGE_MASK (((signed long)PUD_PAGE_MASK) & __PHYSICAL_MASK)
+#define PHYSICAL_PMD_PAGE_MASK (((signed long)PMD_MASK) & __PHYSICAL_MASK)
+#define PHYSICAL_PUD_PAGE_MASK (((signed long)PUD_MASK) & __PHYSICAL_MASK)
#define HPAGE_SHIFT PMD_SHIFT
#define HPAGE_SIZE (_AC(1,UL) << HPAGE_SHIFT)
diff --git a/arch/x86/include/asm/pgtable-3level.h b/arch/x86/include/asm/pgtable-3level.h
index 28421a8..967b135 100644
--- a/arch/x86/include/asm/pgtable-3level.h
+++ b/arch/x86/include/asm/pgtable-3level.h
@@ -2,8 +2,6 @@
#ifndef _ASM_X86_PGTABLE_3LEVEL_H
#define _ASM_X86_PGTABLE_3LEVEL_H
-#include <asm/atomic64_32.h>
-
/*
* Intel Physical Address Extension (PAE) Mode - three-level page
* tables on PPro+ CPUs.
@@ -21,7 +19,15 @@
pr_err("%s:%d: bad pgd %p(%016Lx)\n", \
__FILE__, __LINE__, &(e), pgd_val(e))
-/* Rules for using set_pte: the pte being assigned *must* be
+#define pxx_xchg64(_pxx, _ptr, _val) ({ \
+ _pxx##val_t *_p = (_pxx##val_t *)_ptr; \
+ _pxx##val_t _o = *_p; \
+ do { } while (!try_cmpxchg64(_p, &_o, (_val))); \
+ native_make_##_pxx(_o); \
+})
+
+/*
+ * Rules for using set_pte: the pte being assigned *must* be
* either not present or in a state where the hardware will
* not attempt to update the pte. In places where this is
* not possible, use pte_get_and_clear to obtain the old pte
@@ -29,75 +35,19 @@
*/
static inline void native_set_pte(pte_t *ptep, pte_t pte)
{
- ptep->pte_high = pte.pte_high;
+ WRITE_ONCE(ptep->pte_high, pte.pte_high);
smp_wmb();
- ptep->pte_low = pte.pte_low;
-}
-
-#define pmd_read_atomic pmd_read_atomic
-/*
- * pte_offset_map_lock() on 32-bit PAE kernels was reading the pmd_t with
- * a "*pmdp" dereference done by GCC. Problem is, in certain places
- * where pte_offset_map_lock() is called, concurrent page faults are
- * allowed, if the mmap_lock is hold for reading. An example is mincore
- * vs page faults vs MADV_DONTNEED. On the page fault side
- * pmd_populate() rightfully does a set_64bit(), but if we're reading the
- * pmd_t with a "*pmdp" on the mincore side, a SMP race can happen
- * because GCC will not read the 64-bit value of the pmd atomically.
- *
- * To fix this all places running pte_offset_map_lock() while holding the
- * mmap_lock in read mode, shall read the pmdp pointer using this
- * function to know if the pmd is null or not, and in turn to know if
- * they can run pte_offset_map_lock() or pmd_trans_huge() or other pmd
- * operations.
- *
- * Without THP if the mmap_lock is held for reading, the pmd can only
- * transition from null to not null while pmd_read_atomic() runs. So
- * we can always return atomic pmd values with this function.
- *
- * With THP if the mmap_lock is held for reading, the pmd can become
- * trans_huge or none or point to a pte (and in turn become "stable")
- * at any time under pmd_read_atomic(). We could read it truly
- * atomically here with an atomic64_read() for the THP enabled case (and
- * it would be a whole lot simpler), but to avoid using cmpxchg8b we
- * only return an atomic pmdval if the low part of the pmdval is later
- * found to be stable (i.e. pointing to a pte). We are also returning a
- * 'none' (zero) pmdval if the low part of the pmd is zero.
- *
- * In some cases the high and low part of the pmdval returned may not be
- * consistent if THP is enabled (the low part may point to previously
- * mapped hugepage, while the high part may point to a more recently
- * mapped hugepage), but pmd_none_or_trans_huge_or_clear_bad() only
- * needs the low part of the pmd to be read atomically to decide if the
- * pmd is unstable or not, with the only exception when the low part
- * of the pmd is zero, in which case we return a 'none' pmd.
- */
-static inline pmd_t pmd_read_atomic(pmd_t *pmdp)
-{
- pmdval_t ret;
- u32 *tmp = (u32 *)pmdp;
-
- ret = (pmdval_t) (*tmp);
- if (ret) {
- /*
- * If the low part is null, we must not read the high part
- * or we can end up with a partial pmd.
- */
- smp_rmb();
- ret |= ((pmdval_t)*(tmp + 1)) << 32;
- }
-
- return (pmd_t) { ret };
+ WRITE_ONCE(ptep->pte_low, pte.pte_low);
}
static inline void native_set_pte_atomic(pte_t *ptep, pte_t pte)
{
- set_64bit((unsigned long long *)(ptep), native_pte_val(pte));
+ pxx_xchg64(pte, ptep, native_pte_val(pte));
}
static inline void native_set_pmd(pmd_t *pmdp, pmd_t pmd)
{
- set_64bit((unsigned long long *)(pmdp), native_pmd_val(pmd));
+ pxx_xchg64(pmd, pmdp, native_pmd_val(pmd));
}
static inline void native_set_pud(pud_t *pudp, pud_t pud)
@@ -105,7 +55,7 @@ static inline void native_set_pud(pud_t *pudp, pud_t pud)
#ifdef CONFIG_PAGE_TABLE_ISOLATION
pud.p4d.pgd = pti_set_user_pgtbl(&pudp->p4d.pgd, pud.p4d.pgd);
#endif
- set_64bit((unsigned long long *)(pudp), native_pud_val(pud));
+ pxx_xchg64(pud, pudp, native_pud_val(pud));
}
/*
@@ -116,17 +66,16 @@ static inline void native_set_pud(pud_t *pudp, pud_t pud)
static inline void native_pte_clear(struct mm_struct *mm, unsigned long addr,
pte_t *ptep)
{
- ptep->pte_low = 0;
+ WRITE_ONCE(ptep->pte_low, 0);
smp_wmb();
- ptep->pte_high = 0;
+ WRITE_ONCE(ptep->pte_high, 0);
}
-static inline void native_pmd_clear(pmd_t *pmd)
+static inline void native_pmd_clear(pmd_t *pmdp)
{
- u32 *tmp = (u32 *)pmd;
- *tmp = 0;
+ WRITE_ONCE(pmdp->pmd_low, 0);
smp_wmb();
- *(tmp + 1) = 0;
+ WRITE_ONCE(pmdp->pmd_high, 0);
}
static inline void native_pud_clear(pud_t *pudp)
@@ -149,41 +98,26 @@ static inline void pud_clear(pud_t *pudp)
*/
}
+
#ifdef CONFIG_SMP
static inline pte_t native_ptep_get_and_clear(pte_t *ptep)
{
- pte_t res;
+ return pxx_xchg64(pte, ptep, 0ULL);
+}
- res.pte = (pteval_t)arch_atomic64_xchg((atomic64_t *)ptep, 0);
+static inline pmd_t native_pmdp_get_and_clear(pmd_t *pmdp)
+{
+ return pxx_xchg64(pmd, pmdp, 0ULL);
+}
- return res;
+static inline pud_t native_pudp_get_and_clear(pud_t *pudp)
+{
+ return pxx_xchg64(pud, pudp, 0ULL);
}
#else
#define native_ptep_get_and_clear(xp) native_local_ptep_get_and_clear(xp)
-#endif
-
-union split_pmd {
- struct {
- u32 pmd_low;
- u32 pmd_high;
- };
- pmd_t pmd;
-};
-
-#ifdef CONFIG_SMP
-static inline pmd_t native_pmdp_get_and_clear(pmd_t *pmdp)
-{
- union split_pmd res, *orig = (union split_pmd *)pmdp;
-
- /* xchg acts as a barrier before setting of the high bits */
- res.pmd_low = xchg(&orig->pmd_low, 0);
- res.pmd_high = orig->pmd_high;
- orig->pmd_high = 0;
-
- return res.pmd;
-}
-#else
#define native_pmdp_get_and_clear(xp) native_local_pmdp_get_and_clear(xp)
+#define native_pudp_get_and_clear(xp) native_local_pudp_get_and_clear(xp)
#endif
#ifndef pmdp_establish
@@ -199,55 +133,18 @@ static inline pmd_t pmdp_establish(struct vm_area_struct *vma,
* anybody.
*/
if (!(pmd_val(pmd) & _PAGE_PRESENT)) {
- union split_pmd old, new, *ptr;
-
- ptr = (union split_pmd *)pmdp;
-
- new.pmd = pmd;
-
/* xchg acts as a barrier before setting of the high bits */
- old.pmd_low = xchg(&ptr->pmd_low, new.pmd_low);
- old.pmd_high = ptr->pmd_high;
- ptr->pmd_high = new.pmd_high;
- return old.pmd;
+ old.pmd_low = xchg(&pmdp->pmd_low, pmd.pmd_low);
+ old.pmd_high = READ_ONCE(pmdp->pmd_high);
+ WRITE_ONCE(pmdp->pmd_high, pmd.pmd_high);
+
+ return old;
}
- do {
- old = *pmdp;
- } while (cmpxchg64(&pmdp->pmd, old.pmd, pmd.pmd) != old.pmd);
-
- return old;
+ return pxx_xchg64(pmd, pmdp, pmd.pmd);
}
#endif
-#ifdef CONFIG_SMP
-union split_pud {
- struct {
- u32 pud_low;
- u32 pud_high;
- };
- pud_t pud;
-};
-
-static inline pud_t native_pudp_get_and_clear(pud_t *pudp)
-{
- union split_pud res, *orig = (union split_pud *)pudp;
-
-#ifdef CONFIG_PAGE_TABLE_ISOLATION
- pti_set_user_pgtbl(&pudp->p4d.pgd, __pgd(0));
-#endif
-
- /* xchg acts as a barrier before setting of the high bits */
- res.pud_low = xchg(&orig->pud_low, 0);
- res.pud_high = orig->pud_high;
- orig->pud_high = 0;
-
- return res.pud;
-}
-#else
-#define native_pudp_get_and_clear(xp) native_local_pudp_get_and_clear(xp)
-#endif
-
/* Encode and de-code a swap entry */
#define SWP_TYPE_BITS 5
diff --git a/arch/x86/include/asm/pgtable-3level_types.h b/arch/x86/include/asm/pgtable-3level_types.h
index 56baf43..8091134 100644
--- a/arch/x86/include/asm/pgtable-3level_types.h
+++ b/arch/x86/include/asm/pgtable-3level_types.h
@@ -18,6 +18,13 @@ typedef union {
};
pteval_t pte;
} pte_t;
+
+typedef union {
+ struct {
+ unsigned long pmd_low, pmd_high;
+ };
+ pmdval_t pmd;
+} pmd_t;
#endif /* !__ASSEMBLY__ */
#define SHARED_KERNEL_PMD (!static_cpu_has(X86_FEATURE_PTI))
diff --git a/arch/x86/include/asm/pgtable_64_types.h b/arch/x86/include/asm/pgtable_64_types.h
index 04f3606..38bf837 100644
--- a/arch/x86/include/asm/pgtable_64_types.h
+++ b/arch/x86/include/asm/pgtable_64_types.h
@@ -19,6 +19,7 @@ typedef unsigned long pgdval_t;
typedef unsigned long pgprotval_t;
typedef struct { pteval_t pte; } pte_t;
+typedef struct { pmdval_t pmd; } pmd_t;
#ifdef CONFIG_X86_5LEVEL
extern unsigned int __pgtable_l5_enabled;
diff --git a/arch/x86/include/asm/pgtable_areas.h b/arch/x86/include/asm/pgtable_areas.h
index d34cce1..4f056fb 100644
--- a/arch/x86/include/asm/pgtable_areas.h
+++ b/arch/x86/include/asm/pgtable_areas.h
@@ -11,6 +11,12 @@
#define CPU_ENTRY_AREA_RO_IDT_VADDR ((void *)CPU_ENTRY_AREA_RO_IDT)
-#define CPU_ENTRY_AREA_MAP_SIZE (CPU_ENTRY_AREA_PER_CPU + CPU_ENTRY_AREA_ARRAY_SIZE - CPU_ENTRY_AREA_BASE)
+#ifdef CONFIG_X86_32
+#define CPU_ENTRY_AREA_MAP_SIZE (CPU_ENTRY_AREA_PER_CPU + \
+ (CPU_ENTRY_AREA_SIZE * NR_CPUS) - \
+ CPU_ENTRY_AREA_BASE)
+#else
+#define CPU_ENTRY_AREA_MAP_SIZE P4D_SIZE
+#endif
#endif /* _ASM_X86_PGTABLE_AREAS_H */
diff --git a/arch/x86/include/asm/pgtable_types.h b/arch/x86/include/asm/pgtable_types.h
index aa174fe..447d4be 100644
--- a/arch/x86/include/asm/pgtable_types.h
+++ b/arch/x86/include/asm/pgtable_types.h
@@ -361,11 +361,9 @@ static inline pudval_t native_pud_val(pud_t pud)
#endif
#if CONFIG_PGTABLE_LEVELS > 2
-typedef struct { pmdval_t pmd; } pmd_t;
-
static inline pmd_t native_make_pmd(pmdval_t val)
{
- return (pmd_t) { val };
+ return (pmd_t) { .pmd = val };
}
static inline pmdval_t native_pmd_val(pmd_t pmd)
diff --git a/arch/x86/include/asm/processor-flags.h b/arch/x86/include/asm/processor-flags.h
index 02c2cbd..a7f3d91 100644
--- a/arch/x86/include/asm/processor-flags.h
+++ b/arch/x86/include/asm/processor-flags.h
@@ -35,7 +35,7 @@
*/
#ifdef CONFIG_X86_64
/* Mask off the address space ID and SME encryption bits. */
-#define CR3_ADDR_MASK __sme_clr(0x7FFFFFFFFFFFF000ull)
+#define CR3_ADDR_MASK __sme_clr(PHYSICAL_PAGE_MASK)
#define CR3_PCID_MASK 0xFFFull
#define CR3_NOFLUSH BIT_ULL(63)
diff --git a/arch/x86/include/asm/set_memory.h b/arch/x86/include/asm/set_memory.h
index b45c4d2..a5e8964 100644
--- a/arch/x86/include/asm/set_memory.h
+++ b/arch/x86/include/asm/set_memory.h
@@ -6,6 +6,9 @@
#include <asm/page.h>
#include <asm-generic/set_memory.h>
+#define set_memory_rox set_memory_rox
+int set_memory_rox(unsigned long addr, int numpages);
+
/*
* The set_memory_* API can be used to change various attributes of a virtual
* address range. The attributes include:
diff --git a/arch/x86/kernel/alternative.c b/arch/x86/kernel/alternative.c
index 23cbfa8..7d8c3cb 100644
--- a/arch/x86/kernel/alternative.c
+++ b/arch/x86/kernel/alternative.c
@@ -2142,11 +2142,6 @@ void __ref text_poke_queue(void *addr, const void *opcode, size_t len, const voi
{
struct text_poke_loc *tp;
- if (unlikely(system_state == SYSTEM_BOOTING)) {
- text_poke_early(addr, opcode, len);
- return;
- }
-
text_poke_flush(addr);
tp = &tp_vec[tp_vec_nr++];
@@ -2168,11 +2163,6 @@ void __ref text_poke_bp(void *addr, const void *opcode, size_t len, const void *
{
struct text_poke_loc tp;
- if (unlikely(system_state == SYSTEM_BOOTING)) {
- text_poke_early(addr, opcode, len);
- return;
- }
-
text_poke_loc_init(&tp, addr, opcode, len, emulate);
text_poke_bp_batch(&tp, 1);
}
diff --git a/arch/x86/kernel/amd_gart_64.c b/arch/x86/kernel/amd_gart_64.c
index 19a0207..56a917d 100644
--- a/arch/x86/kernel/amd_gart_64.c
+++ b/arch/x86/kernel/amd_gart_64.c
@@ -504,7 +504,7 @@ static __init unsigned long check_iommu_size(unsigned long aper, u64 aper_size)
}
a = aper + iommu_size;
- iommu_size -= round_up(a, PMD_PAGE_SIZE) - a;
+ iommu_size -= round_up(a, PMD_SIZE) - a;
if (iommu_size < 64*1024*1024) {
pr_warn("PCI-DMA: Warning: Small IOMMU %luMB."
diff --git a/arch/x86/kernel/ftrace.c b/arch/x86/kernel/ftrace.c
index ae0a8e5..5e7ead5 100644
--- a/arch/x86/kernel/ftrace.c
+++ b/arch/x86/kernel/ftrace.c
@@ -24,10 +24,10 @@
#include <linux/module.h>
#include <linux/memory.h>
#include <linux/vmalloc.h>
+#include <linux/set_memory.h>
#include <trace/syscall.h>
-#include <asm/set_memory.h>
#include <asm/kprobes.h>
#include <asm/ftrace.h>
#include <asm/nops.h>
@@ -423,9 +423,7 @@ create_trampoline(struct ftrace_ops *ops, unsigned int *tramp_size)
/* ALLOC_TRAMP flags lets us know we created it */
ops->flags |= FTRACE_OPS_FL_ALLOC_TRAMP;
- if (likely(system_state != SYSTEM_BOOTING))
- set_memory_ro((unsigned long)trampoline, npages);
- set_memory_x((unsigned long)trampoline, npages);
+ set_memory_rox((unsigned long)trampoline, npages);
return (unsigned long)trampoline;
fail:
tramp_free(trampoline);
diff --git a/arch/x86/kernel/head64.c b/arch/x86/kernel/head64.c
index 6a3cfaf..387e4b1 100644
--- a/arch/x86/kernel/head64.c
+++ b/arch/x86/kernel/head64.c
@@ -203,7 +203,7 @@ unsigned long __head __startup_64(unsigned long physaddr,
load_delta = physaddr - (unsigned long)(_text - __START_KERNEL_map);
/* Is the address not 2M aligned? */
- if (load_delta & ~PMD_PAGE_MASK)
+ if (load_delta & ~PMD_MASK)
for (;;);
/* Include the SME encryption mask in the fixup value */
diff --git a/arch/x86/kernel/hw_breakpoint.c b/arch/x86/kernel/hw_breakpoint.c
index 668a4a6..bbb0f73 100644
--- a/arch/x86/kernel/hw_breakpoint.c
+++ b/arch/x86/kernel/hw_breakpoint.c
@@ -266,7 +266,7 @@ static inline bool within_cpu_entry(unsigned long addr, unsigned long end)
/* CPU entry erea is always used for CPU entry */
if (within_area(addr, end, CPU_ENTRY_AREA_BASE,
- CPU_ENTRY_AREA_TOTAL_SIZE))
+ CPU_ENTRY_AREA_MAP_SIZE))
return true;
/*
diff --git a/arch/x86/kernel/kprobes/core.c b/arch/x86/kernel/kprobes/core.c
index 01b8d95..6629968 100644
--- a/arch/x86/kernel/kprobes/core.c
+++ b/arch/x86/kernel/kprobes/core.c
@@ -43,6 +43,7 @@
#include <linux/objtool.h>
#include <linux/vmalloc.h>
#include <linux/pgtable.h>
+#include <linux/set_memory.h>
#include <asm/text-patching.h>
#include <asm/cacheflush.h>
@@ -51,7 +52,6 @@
#include <asm/alternative.h>
#include <asm/insn.h>
#include <asm/debugreg.h>
-#include <asm/set_memory.h>
#include <asm/ibt.h>
#include "common.h"
@@ -415,16 +415,10 @@ void *alloc_insn_page(void)
return NULL;
/*
- * First make the page read-only, and only then make it executable to
- * prevent it from being W+X in between.
- */
- set_memory_ro((unsigned long)page, 1);
-
- /*
* TODO: Once additional kernel code protection mechanisms are set, ensure
* that the page was not maliciously altered and it is still zeroed.
*/
- set_memory_x((unsigned long)page, 1);
+ set_memory_rox((unsigned long)page, 1);
return page;
}
diff --git a/arch/x86/mm/cpu_entry_area.c b/arch/x86/mm/cpu_entry_area.c
index 42cd96e..8bb1aa6 100644
--- a/arch/x86/mm/cpu_entry_area.c
+++ b/arch/x86/mm/cpu_entry_area.c
@@ -9,22 +9,60 @@
#include <asm/cpu_entry_area.h>
#include <asm/fixmap.h>
#include <asm/desc.h>
+#include <asm/kasan.h>
static DEFINE_PER_CPU_PAGE_ALIGNED(struct entry_stack_page, entry_stack_storage);
#ifdef CONFIG_X86_64
static DEFINE_PER_CPU_PAGE_ALIGNED(struct exception_stacks, exception_stacks);
DEFINE_PER_CPU(struct cea_exception_stacks*, cea_exception_stacks);
-#endif
-#ifdef CONFIG_X86_32
+static DEFINE_PER_CPU_READ_MOSTLY(unsigned long, _cea_offset);
+
+static __always_inline unsigned int cea_offset(unsigned int cpu)
+{
+ return per_cpu(_cea_offset, cpu);
+}
+
+static __init void init_cea_offsets(void)
+{
+ unsigned int max_cea;
+ unsigned int i, j;
+
+ max_cea = (CPU_ENTRY_AREA_MAP_SIZE - PAGE_SIZE) / CPU_ENTRY_AREA_SIZE;
+
+ /* O(sodding terrible) */
+ for_each_possible_cpu(i) {
+ unsigned int cea;
+
+again:
+ cea = prandom_u32_max(max_cea);
+
+ for_each_possible_cpu(j) {
+ if (cea_offset(j) == cea)
+ goto again;
+
+ if (i == j)
+ break;
+ }
+
+ per_cpu(_cea_offset, i) = cea;
+ }
+}
+#else /* !X86_64 */
DECLARE_PER_CPU_PAGE_ALIGNED(struct doublefault_stack, doublefault_stack);
+
+static __always_inline unsigned int cea_offset(unsigned int cpu)
+{
+ return cpu;
+}
+static inline void init_cea_offsets(void) { }
#endif
/* Is called from entry code, so must be noinstr */
noinstr struct cpu_entry_area *get_cpu_entry_area(int cpu)
{
- unsigned long va = CPU_ENTRY_AREA_PER_CPU + cpu * CPU_ENTRY_AREA_SIZE;
+ unsigned long va = CPU_ENTRY_AREA_PER_CPU + cea_offset(cpu) * CPU_ENTRY_AREA_SIZE;
BUILD_BUG_ON(sizeof(struct cpu_entry_area) % PAGE_SIZE != 0);
return (struct cpu_entry_area *) va;
@@ -148,6 +186,9 @@ static void __init setup_cpu_entry_area(unsigned int cpu)
pgprot_t tss_prot = PAGE_KERNEL;
#endif
+ kasan_populate_shadow_for_vaddr(cea, CPU_ENTRY_AREA_SIZE,
+ early_cpu_to_node(cpu));
+
cea_set_pte(&cea->gdt, get_cpu_gdt_paddr(cpu), gdt_prot);
cea_map_percpu_pages(&cea->entry_stack_page,
@@ -201,7 +242,6 @@ static __init void setup_cpu_entry_area_ptes(void)
/* The +1 is for the readonly IDT: */
BUILD_BUG_ON((CPU_ENTRY_AREA_PAGES+1)*PAGE_SIZE != CPU_ENTRY_AREA_MAP_SIZE);
- BUILD_BUG_ON(CPU_ENTRY_AREA_TOTAL_SIZE != CPU_ENTRY_AREA_MAP_SIZE);
BUG_ON(CPU_ENTRY_AREA_BASE & ~PMD_MASK);
start = CPU_ENTRY_AREA_BASE;
@@ -217,6 +257,8 @@ void __init setup_cpu_entry_areas(void)
{
unsigned int cpu;
+ init_cea_offsets();
+
setup_cpu_entry_area_ptes();
for_each_possible_cpu(cpu)
diff --git a/arch/x86/mm/init.c b/arch/x86/mm/init.c
index 9121bc1..d398735 100644
--- a/arch/x86/mm/init.c
+++ b/arch/x86/mm/init.c
@@ -801,7 +801,7 @@ void __init poking_init(void)
spinlock_t *ptl;
pte_t *ptep;
- poking_mm = copy_init_mm();
+ poking_mm = mm_alloc();
BUG_ON(!poking_mm);
/*
diff --git a/arch/x86/mm/kasan_init_64.c b/arch/x86/mm/kasan_init_64.c
index e7b9b46..0302491 100644
--- a/arch/x86/mm/kasan_init_64.c
+++ b/arch/x86/mm/kasan_init_64.c
@@ -316,10 +316,33 @@ void __init kasan_early_init(void)
kasan_map_early_shadow(init_top_pgt);
}
+static unsigned long kasan_mem_to_shadow_align_down(unsigned long va)
+{
+ unsigned long shadow = (unsigned long)kasan_mem_to_shadow((void *)va);
+
+ return round_down(shadow, PAGE_SIZE);
+}
+
+static unsigned long kasan_mem_to_shadow_align_up(unsigned long va)
+{
+ unsigned long shadow = (unsigned long)kasan_mem_to_shadow((void *)va);
+
+ return round_up(shadow, PAGE_SIZE);
+}
+
+void __init kasan_populate_shadow_for_vaddr(void *va, size_t size, int nid)
+{
+ unsigned long shadow_start, shadow_end;
+
+ shadow_start = kasan_mem_to_shadow_align_down((unsigned long)va);
+ shadow_end = kasan_mem_to_shadow_align_up((unsigned long)va + size);
+ kasan_populate_shadow(shadow_start, shadow_end, nid);
+}
+
void __init kasan_init(void)
{
+ unsigned long shadow_cea_begin, shadow_cea_per_cpu_begin, shadow_cea_end;
int i;
- void *shadow_cpu_entry_begin, *shadow_cpu_entry_end;
memcpy(early_top_pgt, init_top_pgt, sizeof(early_top_pgt));
@@ -360,16 +383,10 @@ void __init kasan_init(void)
map_range(&pfn_mapped[i]);
}
- shadow_cpu_entry_begin = (void *)CPU_ENTRY_AREA_BASE;
- shadow_cpu_entry_begin = kasan_mem_to_shadow(shadow_cpu_entry_begin);
- shadow_cpu_entry_begin = (void *)round_down(
- (unsigned long)shadow_cpu_entry_begin, PAGE_SIZE);
-
- shadow_cpu_entry_end = (void *)(CPU_ENTRY_AREA_BASE +
- CPU_ENTRY_AREA_MAP_SIZE);
- shadow_cpu_entry_end = kasan_mem_to_shadow(shadow_cpu_entry_end);
- shadow_cpu_entry_end = (void *)round_up(
- (unsigned long)shadow_cpu_entry_end, PAGE_SIZE);
+ shadow_cea_begin = kasan_mem_to_shadow_align_down(CPU_ENTRY_AREA_BASE);
+ shadow_cea_per_cpu_begin = kasan_mem_to_shadow_align_up(CPU_ENTRY_AREA_PER_CPU);
+ shadow_cea_end = kasan_mem_to_shadow_align_up(CPU_ENTRY_AREA_BASE +
+ CPU_ENTRY_AREA_MAP_SIZE);
kasan_populate_early_shadow(
kasan_mem_to_shadow((void *)PAGE_OFFSET + MAXMEM),
@@ -391,12 +408,18 @@ void __init kasan_init(void)
kasan_populate_early_shadow(
kasan_mem_to_shadow((void *)VMALLOC_END + 1),
- shadow_cpu_entry_begin);
+ (void *)shadow_cea_begin);
- kasan_populate_shadow((unsigned long)shadow_cpu_entry_begin,
- (unsigned long)shadow_cpu_entry_end, 0);
+ /*
+ * Populate the shadow for the shared portion of the CPU entry area.
+ * Shadows for the per-CPU areas are mapped on-demand, as each CPU's
+ * area is randomly placed somewhere in the 512GiB range and mapping
+ * the entire 512GiB range is prohibitively expensive.
+ */
+ kasan_populate_shadow(shadow_cea_begin,
+ shadow_cea_per_cpu_begin, 0);
- kasan_populate_early_shadow(shadow_cpu_entry_end,
+ kasan_populate_early_shadow((void *)shadow_cea_end,
kasan_mem_to_shadow((void *)__START_KERNEL_map));
kasan_populate_shadow((unsigned long)kasan_mem_to_shadow(_stext),
diff --git a/arch/x86/mm/mem_encrypt_boot.S b/arch/x86/mm/mem_encrypt_boot.S
index 9de3d90..e25288e 100644
--- a/arch/x86/mm/mem_encrypt_boot.S
+++ b/arch/x86/mm/mem_encrypt_boot.S
@@ -26,7 +26,7 @@
* RCX - virtual address of the encryption workarea, including:
* - stack page (PAGE_SIZE)
* - encryption routine page (PAGE_SIZE)
- * - intermediate copy buffer (PMD_PAGE_SIZE)
+ * - intermediate copy buffer (PMD_SIZE)
* R8 - physical address of the pagetables to use for encryption
*/
@@ -123,7 +123,7 @@
wbinvd /* Invalidate any cache entries */
/* Copy/encrypt up to 2MB at a time */
- movq $PMD_PAGE_SIZE, %r12
+ movq $PMD_SIZE, %r12
1:
cmpq %r12, %r9
jnb 2f
diff --git a/arch/x86/mm/mem_encrypt_identity.c b/arch/x86/mm/mem_encrypt_identity.c
index f415498..88cccd6 100644
--- a/arch/x86/mm/mem_encrypt_identity.c
+++ b/arch/x86/mm/mem_encrypt_identity.c
@@ -93,7 +93,7 @@ struct sme_populate_pgd_data {
* section is 2MB aligned to allow for simple pagetable setup using only
* PMD entries (see vmlinux.lds.S).
*/
-static char sme_workarea[2 * PMD_PAGE_SIZE] __section(".init.scratch");
+static char sme_workarea[2 * PMD_SIZE] __section(".init.scratch");
static char sme_cmdline_arg[] __initdata = "mem_encrypt";
static char sme_cmdline_on[] __initdata = "on";
@@ -198,8 +198,8 @@ static void __init __sme_map_range_pmd(struct sme_populate_pgd_data *ppd)
while (ppd->vaddr < ppd->vaddr_end) {
sme_populate_pgd_large(ppd);
- ppd->vaddr += PMD_PAGE_SIZE;
- ppd->paddr += PMD_PAGE_SIZE;
+ ppd->vaddr += PMD_SIZE;
+ ppd->paddr += PMD_SIZE;
}
}
@@ -225,11 +225,11 @@ static void __init __sme_map_range(struct sme_populate_pgd_data *ppd,
vaddr_end = ppd->vaddr_end;
/* If start is not 2MB aligned, create PTE entries */
- ppd->vaddr_end = ALIGN(ppd->vaddr, PMD_PAGE_SIZE);
+ ppd->vaddr_end = ALIGN(ppd->vaddr, PMD_SIZE);
__sme_map_range_pte(ppd);
/* Create PMD entries */
- ppd->vaddr_end = vaddr_end & PMD_PAGE_MASK;
+ ppd->vaddr_end = vaddr_end & PMD_MASK;
__sme_map_range_pmd(ppd);
/* If end is not 2MB aligned, create PTE entries */
@@ -325,7 +325,7 @@ void __init sme_encrypt_kernel(struct boot_params *bp)
/* Physical addresses gives us the identity mapped virtual addresses */
kernel_start = __pa_symbol(_text);
- kernel_end = ALIGN(__pa_symbol(_end), PMD_PAGE_SIZE);
+ kernel_end = ALIGN(__pa_symbol(_end), PMD_SIZE);
kernel_len = kernel_end - kernel_start;
initrd_start = 0;
@@ -355,12 +355,12 @@ void __init sme_encrypt_kernel(struct boot_params *bp)
* executable encryption area size:
* stack page (PAGE_SIZE)
* encryption routine page (PAGE_SIZE)
- * intermediate copy buffer (PMD_PAGE_SIZE)
+ * intermediate copy buffer (PMD_SIZE)
* pagetable structures for the encryption of the kernel
* pagetable structures for workarea (in case not currently mapped)
*/
execute_start = workarea_start;
- execute_end = execute_start + (PAGE_SIZE * 2) + PMD_PAGE_SIZE;
+ execute_end = execute_start + (PAGE_SIZE * 2) + PMD_SIZE;
execute_len = execute_end - execute_start;
/*
@@ -383,7 +383,7 @@ void __init sme_encrypt_kernel(struct boot_params *bp)
* before it is mapped.
*/
workarea_len = execute_len + pgtable_area_len;
- workarea_end = ALIGN(workarea_start + workarea_len, PMD_PAGE_SIZE);
+ workarea_end = ALIGN(workarea_start + workarea_len, PMD_SIZE);
/*
* Set the address to the start of where newly created pagetable
diff --git a/arch/x86/mm/pat/set_memory.c b/arch/x86/mm/pat/set_memory.c
index ef34ba2..356758b 100644
--- a/arch/x86/mm/pat/set_memory.c
+++ b/arch/x86/mm/pat/set_memory.c
@@ -220,6 +220,23 @@ within_inclusive(unsigned long addr, unsigned long start, unsigned long end)
#ifdef CONFIG_X86_64
+/*
+ * The kernel image is mapped into two places in the virtual address space
+ * (addresses without KASLR, of course):
+ *
+ * 1. The kernel direct map (0xffff880000000000)
+ * 2. The "high kernel map" (0xffffffff81000000)
+ *
+ * We actually execute out of #2. If we get the address of a kernel symbol, it
+ * points to #2, but almost all physical-to-virtual translations point to #1.
+ *
+ * This is so that we can have both a directmap of all physical memory *and*
+ * take full advantage of the the limited (s32) immediate addressing range (2G)
+ * of x86_64.
+ *
+ * See Documentation/x86/x86_64/mm.rst for more detail.
+ */
+
static inline unsigned long highmap_start_pfn(void)
{
return __pa_symbol(_text) >> PAGE_SHIFT;
@@ -605,10 +622,6 @@ static inline pgprot_t verify_rwx(pgprot_t old, pgprot_t new, unsigned long star
{
unsigned long end;
- /* Kernel text is rw at boot up */
- if (system_state == SYSTEM_BOOTING)
- return new;
-
/*
* 32-bit has some unfixable W+X issues, like EFI code
* and writeable data being in the same page. Disable
@@ -765,11 +778,11 @@ phys_addr_t slow_virt_to_phys(void *__virt_addr)
switch (level) {
case PG_LEVEL_1G:
phys_addr = (phys_addr_t)pud_pfn(*(pud_t *)pte) << PAGE_SHIFT;
- offset = virt_addr & ~PUD_PAGE_MASK;
+ offset = virt_addr & ~PUD_MASK;
break;
case PG_LEVEL_2M:
phys_addr = (phys_addr_t)pmd_pfn(*(pmd_t *)pte) << PAGE_SHIFT;
- offset = virt_addr & ~PMD_PAGE_MASK;
+ offset = virt_addr & ~PMD_MASK;
break;
default:
phys_addr = (phys_addr_t)pte_pfn(*pte) << PAGE_SHIFT;
@@ -1059,7 +1072,7 @@ __split_large_page(struct cpa_data *cpa, pte_t *kpte, unsigned long address,
case PG_LEVEL_1G:
ref_prot = pud_pgprot(*(pud_t *)kpte);
ref_pfn = pud_pfn(*(pud_t *)kpte);
- pfninc = PMD_PAGE_SIZE >> PAGE_SHIFT;
+ pfninc = PMD_SIZE >> PAGE_SHIFT;
lpaddr = address & PUD_MASK;
lpinc = PMD_SIZE;
/*
@@ -1646,8 +1659,11 @@ static int __change_page_attr(struct cpa_data *cpa, int primary)
return err;
}
-static int __change_page_attr_set_clr(struct cpa_data *cpa, int checkalias);
+static int __change_page_attr_set_clr(struct cpa_data *cpa, int primary);
+/*
+ * Check the directmap and "high kernel map" 'aliases'.
+ */
static int cpa_process_alias(struct cpa_data *cpa)
{
struct cpa_data alias_cpa;
@@ -1671,6 +1687,12 @@ static int cpa_process_alias(struct cpa_data *cpa)
alias_cpa.flags &= ~(CPA_PAGES_ARRAY | CPA_ARRAY);
alias_cpa.curpage = 0;
+ /* Directmap always has NX set, do not modify. */
+ if (__supported_pte_mask & _PAGE_NX) {
+ alias_cpa.mask_clr.pgprot &= ~_PAGE_NX;
+ alias_cpa.mask_set.pgprot &= ~_PAGE_NX;
+ }
+
cpa->force_flush_all = 1;
ret = __change_page_attr_set_clr(&alias_cpa, 0);
@@ -1693,6 +1715,15 @@ static int cpa_process_alias(struct cpa_data *cpa)
alias_cpa.flags &= ~(CPA_PAGES_ARRAY | CPA_ARRAY);
alias_cpa.curpage = 0;
+ /*
+ * [_text, _brk_end) also covers data, do not modify NX except
+ * in cases where the highmap is the primary target.
+ */
+ if (__supported_pte_mask & _PAGE_NX) {
+ alias_cpa.mask_clr.pgprot &= ~_PAGE_NX;
+ alias_cpa.mask_set.pgprot &= ~_PAGE_NX;
+ }
+
cpa->force_flush_all = 1;
/*
* The high mapping range is imprecise, so ignore the
@@ -1705,12 +1736,19 @@ static int cpa_process_alias(struct cpa_data *cpa)
return 0;
}
-static int __change_page_attr_set_clr(struct cpa_data *cpa, int checkalias)
+static int __change_page_attr_set_clr(struct cpa_data *cpa, int primary)
{
unsigned long numpages = cpa->numpages;
unsigned long rempages = numpages;
int ret = 0;
+ /*
+ * No changes, easy!
+ */
+ if (!(pgprot_val(cpa->mask_set) | pgprot_val(cpa->mask_clr)) &&
+ !cpa->force_split)
+ return ret;
+
while (rempages) {
/*
* Store the remaining nr of pages for the large page
@@ -1723,13 +1761,13 @@ static int __change_page_attr_set_clr(struct cpa_data *cpa, int checkalias)
if (!debug_pagealloc_enabled())
spin_lock(&cpa_lock);
- ret = __change_page_attr(cpa, checkalias);
+ ret = __change_page_attr(cpa, primary);
if (!debug_pagealloc_enabled())
spin_unlock(&cpa_lock);
if (ret)
goto out;
- if (checkalias) {
+ if (primary && !(cpa->flags & CPA_NO_CHECK_ALIAS)) {
ret = cpa_process_alias(cpa);
if (ret)
goto out;
@@ -1757,7 +1795,7 @@ static int change_page_attr_set_clr(unsigned long *addr, int numpages,
struct page **pages)
{
struct cpa_data cpa;
- int ret, cache, checkalias;
+ int ret, cache;
memset(&cpa, 0, sizeof(cpa));
@@ -1803,20 +1841,11 @@ static int change_page_attr_set_clr(unsigned long *addr, int numpages,
cpa.numpages = numpages;
cpa.mask_set = mask_set;
cpa.mask_clr = mask_clr;
- cpa.flags = 0;
+ cpa.flags = in_flag;
cpa.curpage = 0;
cpa.force_split = force_split;
- if (in_flag & (CPA_ARRAY | CPA_PAGES_ARRAY))
- cpa.flags |= in_flag;
-
- /* No alias checking for _NX bit modifications */
- checkalias = (pgprot_val(mask_set) | pgprot_val(mask_clr)) != _PAGE_NX;
- /* Has caller explicitly disabled alias checking? */
- if (in_flag & CPA_NO_CHECK_ALIAS)
- checkalias = 0;
-
- ret = __change_page_attr_set_clr(&cpa, checkalias);
+ ret = __change_page_attr_set_clr(&cpa, 1);
/*
* Check whether we really changed something:
@@ -2047,6 +2076,16 @@ int set_memory_ro(unsigned long addr, int numpages)
return change_page_attr_clear(&addr, numpages, __pgprot(_PAGE_RW), 0);
}
+int set_memory_rox(unsigned long addr, int numpages)
+{
+ pgprot_t clr = __pgprot(_PAGE_RW);
+
+ if (__supported_pte_mask & _PAGE_NX)
+ clr.pgprot |= _PAGE_NX;
+
+ return change_page_attr_clear(&addr, numpages, clr, 0);
+}
+
int set_memory_rw(unsigned long addr, int numpages)
{
return change_page_attr_set(&addr, numpages, __pgprot(_PAGE_RW), 0);
@@ -2059,11 +2098,9 @@ int set_memory_np(unsigned long addr, int numpages)
int set_memory_np_noalias(unsigned long addr, int numpages)
{
- int cpa_flags = CPA_NO_CHECK_ALIAS;
-
return change_page_attr_set_clr(&addr, numpages, __pgprot(0),
__pgprot(_PAGE_PRESENT), 0,
- cpa_flags, NULL);
+ CPA_NO_CHECK_ALIAS, NULL);
}
int set_memory_4k(unsigned long addr, int numpages)
@@ -2280,7 +2317,7 @@ static int __set_pages_p(struct page *page, int numpages)
.numpages = numpages,
.mask_set = __pgprot(_PAGE_PRESENT | _PAGE_RW),
.mask_clr = __pgprot(0),
- .flags = 0};
+ .flags = CPA_NO_CHECK_ALIAS };
/*
* No alias checking needed for setting present flag. otherwise,
@@ -2288,7 +2325,7 @@ static int __set_pages_p(struct page *page, int numpages)
* mappings (this adds to complexity if we want to do this from
* atomic context especially). Let's keep it simple!
*/
- return __change_page_attr_set_clr(&cpa, 0);
+ return __change_page_attr_set_clr(&cpa, 1);
}
static int __set_pages_np(struct page *page, int numpages)
@@ -2299,7 +2336,7 @@ static int __set_pages_np(struct page *page, int numpages)
.numpages = numpages,
.mask_set = __pgprot(0),
.mask_clr = __pgprot(_PAGE_PRESENT | _PAGE_RW),
- .flags = 0};
+ .flags = CPA_NO_CHECK_ALIAS };
/*
* No alias checking needed for setting not present flag. otherwise,
@@ -2307,7 +2344,7 @@ static int __set_pages_np(struct page *page, int numpages)
* mappings (this adds to complexity if we want to do this from
* atomic context especially). Let's keep it simple!
*/
- return __change_page_attr_set_clr(&cpa, 0);
+ return __change_page_attr_set_clr(&cpa, 1);
}
int set_direct_map_invalid_noflush(struct page *page)
@@ -2378,7 +2415,7 @@ int __init kernel_map_pages_in_pgd(pgd_t *pgd, u64 pfn, unsigned long address,
.numpages = numpages,
.mask_set = __pgprot(0),
.mask_clr = __pgprot(~page_flags & (_PAGE_NX|_PAGE_RW)),
- .flags = 0,
+ .flags = CPA_NO_CHECK_ALIAS,
};
WARN_ONCE(num_online_cpus() > 1, "Don't call after initializing SMP");
@@ -2391,7 +2428,7 @@ int __init kernel_map_pages_in_pgd(pgd_t *pgd, u64 pfn, unsigned long address,
cpa.mask_set = __pgprot(_PAGE_PRESENT | page_flags);
- retval = __change_page_attr_set_clr(&cpa, 0);
+ retval = __change_page_attr_set_clr(&cpa, 1);
__flush_tlb_all();
out:
@@ -2421,12 +2458,12 @@ int __init kernel_unmap_pages_in_pgd(pgd_t *pgd, unsigned long address,
.numpages = numpages,
.mask_set = __pgprot(0),
.mask_clr = __pgprot(_PAGE_PRESENT | _PAGE_RW),
- .flags = 0,
+ .flags = CPA_NO_CHECK_ALIAS,
};
WARN_ONCE(num_online_cpus() > 1, "Don't call after initializing SMP");
- retval = __change_page_attr_set_clr(&cpa, 0);
+ retval = __change_page_attr_set_clr(&cpa, 1);
__flush_tlb_all();
return retval;
diff --git a/arch/x86/mm/pti.c b/arch/x86/mm/pti.c
index ffe3b3a..78414c6 100644
--- a/arch/x86/mm/pti.c
+++ b/arch/x86/mm/pti.c
@@ -592,7 +592,7 @@ static void pti_set_kernel_image_nonglobal(void)
* of the image.
*/
unsigned long start = PFN_ALIGN(_text);
- unsigned long end = ALIGN((unsigned long)_end, PMD_PAGE_SIZE);
+ unsigned long end = ALIGN((unsigned long)_end, PMD_SIZE);
/*
* This clears _PAGE_GLOBAL from the entire kernel image.
diff --git a/drivers/accel/drm_accel.c b/drivers/accel/drm_accel.c
index a5ee84a..1b69824 100644
--- a/drivers/accel/drm_accel.c
+++ b/drivers/accel/drm_accel.c
@@ -27,7 +27,7 @@ static struct device_type accel_sysfs_device_minor = {
.name = "accel_minor"
};
-static char *accel_devnode(struct device *dev, umode_t *mode)
+static char *accel_devnode(const struct device *dev, umode_t *mode)
{
return kasprintf(GFP_KERNEL, "accel/%s", dev_name(dev));
}
diff --git a/drivers/base/firmware_loader/firmware.h b/drivers/base/firmware_loader/firmware.h
index fe77e91..bf549d6 100644
--- a/drivers/base/firmware_loader/firmware.h
+++ b/drivers/base/firmware_loader/firmware.h
@@ -9,8 +9,6 @@
#include <linux/list.h>
#include <linux/completion.h>
-#include <generated/utsrelease.h>
-
/**
* enum fw_opt - options to control firmware loading behaviour
*
diff --git a/drivers/cpuidle/cpuidle-powernv.c b/drivers/cpuidle/cpuidle-powernv.c
index 0b5461b..9ebedd9 100644
--- a/drivers/cpuidle/cpuidle-powernv.c
+++ b/drivers/cpuidle/cpuidle-powernv.c
@@ -76,6 +76,7 @@ static int snooze_loop(struct cpuidle_device *dev,
local_irq_enable();
snooze_exit_time = get_tb() + get_snooze_timeout(dev, drv, index);
+ dev->poll_time_limit = false;
ppc64_runlatch_off();
HMT_very_low();
while (!need_resched()) {
@@ -86,6 +87,7 @@ static int snooze_loop(struct cpuidle_device *dev,
* cleared to order subsequent test of need_resched().
*/
clear_thread_flag(TIF_POLLING_NRFLAG);
+ dev->poll_time_limit = true;
smp_mb();
break;
}
@@ -155,7 +157,8 @@ static struct cpuidle_state powernv_states[CPUIDLE_STATE_MAX] = {
.desc = "snooze",
.exit_latency = 0,
.target_residency = 0,
- .enter = snooze_loop },
+ .enter = snooze_loop,
+ .flags = CPUIDLE_FLAG_POLLING },
};
static int powernv_cpuidle_cpu_online(unsigned int cpu)
diff --git a/drivers/cpuidle/cpuidle-pseries.c b/drivers/cpuidle/cpuidle-pseries.c
index 7e7ab55..1bad4d2 100644
--- a/drivers/cpuidle/cpuidle-pseries.c
+++ b/drivers/cpuidle/cpuidle-pseries.c
@@ -44,6 +44,7 @@ static int snooze_loop(struct cpuidle_device *dev,
pseries_idle_prolog();
local_irq_enable();
snooze_exit_time = get_tb() + snooze_timeout;
+ dev->poll_time_limit = false;
while (!need_resched()) {
HMT_low();
@@ -54,6 +55,7 @@ static int snooze_loop(struct cpuidle_device *dev,
* loop anyway. Require a barrier after polling is
* cleared to order subsequent test of need_resched().
*/
+ dev->poll_time_limit = true;
clear_thread_flag(TIF_POLLING_NRFLAG);
smp_mb();
break;
@@ -268,7 +270,8 @@ static struct cpuidle_state dedicated_states[NR_DEDICATED_STATES] = {
.desc = "snooze",
.exit_latency = 0,
.target_residency = 0,
- .enter = &snooze_loop },
+ .enter = &snooze_loop,
+ .flags = CPUIDLE_FLAG_POLLING },
{ /* CEDE */
.name = "CEDE",
.desc = "CEDE",
@@ -286,7 +289,8 @@ static struct cpuidle_state shared_states[] = {
.desc = "snooze",
.exit_latency = 0,
.target_residency = 0,
- .enter = &snooze_loop },
+ .enter = &snooze_loop,
+ .flags = CPUIDLE_FLAG_POLLING },
{ /* Shared Cede */
.name = "Shared Cede",
.desc = "Shared Cede",
diff --git a/drivers/crypto/vmx/Makefile b/drivers/crypto/vmx/Makefile
index 2560cfea..7257b8c 100644
--- a/drivers/crypto/vmx/Makefile
+++ b/drivers/crypto/vmx/Makefile
@@ -2,10 +2,22 @@
obj-$(CONFIG_CRYPTO_DEV_VMX_ENCRYPT) += vmx-crypto.o
vmx-crypto-objs := vmx.o aesp8-ppc.o ghashp8-ppc.o aes.o aes_cbc.o aes_ctr.o aes_xts.o ghash.o
+ifeq ($(CONFIG_CPU_LITTLE_ENDIAN),y)
+override flavour := linux-ppc64le
+else
+ifdef CONFIG_PPC64_ELF_ABI_V2
+override flavour := linux-ppc64-elfv2
+else
+override flavour := linux-ppc64
+endif
+endif
+
quiet_cmd_perl = PERL $@
- cmd_perl = $(PERL) $< $(if $(CONFIG_CPU_LITTLE_ENDIAN), linux-ppc64le, linux-ppc64) > $@
+ cmd_perl = $(PERL) $< $(flavour) > $@
targets += aesp8-ppc.S ghashp8-ppc.S
$(obj)/aesp8-ppc.S $(obj)/ghashp8-ppc.S: $(obj)/%.S: $(src)/%.pl FORCE
$(call if_changed,perl)
+
+OBJECT_FILES_NON_STANDARD_aesp8-ppc.o := y
diff --git a/drivers/crypto/vmx/ppc-xlate.pl b/drivers/crypto/vmx/ppc-xlate.pl
index 36db2ef..b583898 100644
--- a/drivers/crypto/vmx/ppc-xlate.pl
+++ b/drivers/crypto/vmx/ppc-xlate.pl
@@ -9,6 +9,8 @@
my %GLOBALS;
my $dotinlocallabels=($flavour=~/linux/)?1:0;
+my $elfv2abi=(($flavour =~ /linux-ppc64le/) or ($flavour =~ /linux-ppc64-elfv2/))?1:0;
+my $dotfunctions=($elfv2abi=~1)?0:1;
################################################################
# directives which need special treatment on different platforms
@@ -40,7 +42,7 @@
};
my $text = sub {
my $ret = ($flavour =~ /aix/) ? ".csect\t.text[PR],7" : ".text";
- $ret = ".abiversion 2\n".$ret if ($flavour =~ /linux.*64le/);
+ $ret = ".abiversion 2\n".$ret if ($elfv2abi);
$ret;
};
my $machine = sub {
@@ -56,8 +58,8 @@
if ($flavour =~ /linux/)
{ shift;
my $name = shift; $name =~ s|^[\.\_]||;
- my $ret = ".size $name,.-".($flavour=~/64$/?".":"").$name;
- $ret .= "\n.size .$name,.-.$name" if ($flavour=~/64$/);
+ my $ret = ".size $name,.-".($dotfunctions?".":"").$name;
+ $ret .= "\n.size .$name,.-.$name" if ($dotfunctions);
$ret;
}
else
@@ -142,7 +144,7 @@
# Some ABIs specify vrsave, special-purpose register #256, as reserved
# for system use.
-my $no_vrsave = ($flavour =~ /linux-ppc64le/);
+my $no_vrsave = ($elfv2abi);
my $mtspr = sub {
my ($f,$idx,$ra) = @_;
if ($idx == 256 && $no_vrsave) {
diff --git a/drivers/dma/Kconfig b/drivers/dma/Kconfig
index 25e111a..b6d48d5 100644
--- a/drivers/dma/Kconfig
+++ b/drivers/dma/Kconfig
@@ -97,6 +97,7 @@
tristate "Atmel AHB DMA support"
depends on ARCH_AT91
select DMA_ENGINE
+ select DMA_VIRTUAL_CHANNELS
help
Support the Atmel AHB DMA controller.
@@ -357,14 +358,6 @@
If unsure, say N.
-config INTEL_IOP_ADMA
- tristate "Intel IOP32x ADMA support"
- depends on ARCH_IOP32X || COMPILE_TEST
- select DMA_ENGINE
- select ASYNC_TX_ENABLE_CHANNEL_SWITCH
- help
- Enable support for the Intel(R) IOP Series RAID engines.
-
config K3_DMA
tristate "Hisilicon K3 DMA support"
depends on ARCH_HI3xxx || ARCH_HISI || COMPILE_TEST
diff --git a/drivers/dma/Makefile b/drivers/dma/Makefile
index 10f7d42..5b55ada 100644
--- a/drivers/dma/Makefile
+++ b/drivers/dma/Makefile
@@ -44,7 +44,6 @@
obj-$(CONFIG_INTEL_IDMA64) += idma64.o
obj-$(CONFIG_INTEL_IOATDMA) += ioat/
obj-y += idxd/
-obj-$(CONFIG_INTEL_IOP_ADMA) += iop-adma.o
obj-$(CONFIG_K3_DMA) += k3dma.o
obj-$(CONFIG_LPC18XX_DMAMUX) += lpc18xx-dmamux.o
obj-$(CONFIG_MILBEAUT_HDMAC) += milbeaut-hdmac.o
diff --git a/drivers/dma/apple-admac.c b/drivers/dma/apple-admac.c
index a2cc520..90f28bd 100644
--- a/drivers/dma/apple-admac.c
+++ b/drivers/dma/apple-admac.c
@@ -21,6 +21,12 @@
#define NCHANNELS_MAX 64
#define IRQ_NOUTPUTS 4
+/*
+ * For allocation purposes we split the cache
+ * memory into blocks of fixed size (given in bytes).
+ */
+#define SRAM_BLOCK 2048
+
#define RING_WRITE_SLOT GENMASK(1, 0)
#define RING_READ_SLOT GENMASK(5, 4)
#define RING_FULL BIT(9)
@@ -36,6 +42,9 @@
#define REG_TX_STOP 0x0004
#define REG_RX_START 0x0008
#define REG_RX_STOP 0x000c
+#define REG_IMPRINT 0x0090
+#define REG_TX_SRAM_SIZE 0x0094
+#define REG_RX_SRAM_SIZE 0x0098
#define REG_CHAN_CTL(ch) (0x8000 + (ch) * 0x200)
#define REG_CHAN_CTL_RST_RINGS BIT(0)
@@ -53,7 +62,9 @@
#define BUS_WIDTH_FRAME_2_WORDS 0x10
#define BUS_WIDTH_FRAME_4_WORDS 0x20
-#define CHAN_BUFSIZE 0x8000
+#define REG_CHAN_SRAM_CARVEOUT(ch) (0x8050 + (ch) * 0x200)
+#define CHAN_SRAM_CARVEOUT_SIZE GENMASK(31, 16)
+#define CHAN_SRAM_CARVEOUT_BASE GENMASK(15, 0)
#define REG_CHAN_FIFOCTL(ch) (0x8054 + (ch) * 0x200)
#define CHAN_FIFOCTL_LIMIT GENMASK(31, 16)
@@ -76,6 +87,8 @@ struct admac_chan {
struct dma_chan chan;
struct tasklet_struct tasklet;
+ u32 carveout;
+
spinlock_t lock;
struct admac_tx *current_tx;
int nperiod_acks;
@@ -92,12 +105,24 @@ struct admac_chan {
struct list_head to_free;
};
+struct admac_sram {
+ u32 size;
+ /*
+ * SRAM_CARVEOUT has 16-bit fields, so the SRAM cannot be larger than
+ * 64K and a 32-bit bitfield over 2K blocks covers it.
+ */
+ u32 allocated;
+};
+
struct admac_data {
struct dma_device dma;
struct device *dev;
__iomem void *base;
struct reset_control *rstc;
+ struct mutex cache_alloc_lock;
+ struct admac_sram txcache, rxcache;
+
int irq;
int irq_index;
int nchannels;
@@ -118,6 +143,60 @@ struct admac_tx {
struct list_head node;
};
+static int admac_alloc_sram_carveout(struct admac_data *ad,
+ enum dma_transfer_direction dir,
+ u32 *out)
+{
+ struct admac_sram *sram;
+ int i, ret = 0, nblocks;
+
+ if (dir == DMA_MEM_TO_DEV)
+ sram = &ad->txcache;
+ else
+ sram = &ad->rxcache;
+
+ mutex_lock(&ad->cache_alloc_lock);
+
+ nblocks = sram->size / SRAM_BLOCK;
+ for (i = 0; i < nblocks; i++)
+ if (!(sram->allocated & BIT(i)))
+ break;
+
+ if (i < nblocks) {
+ *out = FIELD_PREP(CHAN_SRAM_CARVEOUT_BASE, i * SRAM_BLOCK) |
+ FIELD_PREP(CHAN_SRAM_CARVEOUT_SIZE, SRAM_BLOCK);
+ sram->allocated |= BIT(i);
+ } else {
+ ret = -EBUSY;
+ }
+
+ mutex_unlock(&ad->cache_alloc_lock);
+
+ return ret;
+}
+
+static void admac_free_sram_carveout(struct admac_data *ad,
+ enum dma_transfer_direction dir,
+ u32 carveout)
+{
+ struct admac_sram *sram;
+ u32 base = FIELD_GET(CHAN_SRAM_CARVEOUT_BASE, carveout);
+ int i;
+
+ if (dir == DMA_MEM_TO_DEV)
+ sram = &ad->txcache;
+ else
+ sram = &ad->rxcache;
+
+ if (WARN_ON(base >= sram->size))
+ return;
+
+ mutex_lock(&ad->cache_alloc_lock);
+ i = base / SRAM_BLOCK;
+ sram->allocated &= ~BIT(i);
+ mutex_unlock(&ad->cache_alloc_lock);
+}
+
static void admac_modify(struct admac_data *ad, int reg, u32 mask, u32 val)
{
void __iomem *addr = ad->base + reg;
@@ -466,15 +545,28 @@ static void admac_synchronize(struct dma_chan *chan)
static int admac_alloc_chan_resources(struct dma_chan *chan)
{
struct admac_chan *adchan = to_admac_chan(chan);
+ struct admac_data *ad = adchan->host;
+ int ret;
dma_cookie_init(&adchan->chan);
+ ret = admac_alloc_sram_carveout(ad, admac_chan_direction(adchan->no),
+ &adchan->carveout);
+ if (ret < 0)
+ return ret;
+
+ writel_relaxed(adchan->carveout,
+ ad->base + REG_CHAN_SRAM_CARVEOUT(adchan->no));
return 0;
}
static void admac_free_chan_resources(struct dma_chan *chan)
{
+ struct admac_chan *adchan = to_admac_chan(chan);
+
admac_terminate_all(chan);
admac_synchronize(chan);
+ admac_free_sram_carveout(adchan->host, admac_chan_direction(adchan->no),
+ adchan->carveout);
}
static struct dma_chan *admac_dma_of_xlate(struct of_phandle_args *dma_spec,
@@ -712,6 +804,7 @@ static int admac_probe(struct platform_device *pdev)
platform_set_drvdata(pdev, ad);
ad->dev = &pdev->dev;
ad->nchannels = nchannels;
+ mutex_init(&ad->cache_alloc_lock);
/*
* The controller has 4 IRQ outputs. Try them all until
@@ -801,6 +894,13 @@ static int admac_probe(struct platform_device *pdev)
goto free_irq;
}
+ ad->txcache.size = readl_relaxed(ad->base + REG_TX_SRAM_SIZE);
+ ad->rxcache.size = readl_relaxed(ad->base + REG_RX_SRAM_SIZE);
+
+ dev_info(&pdev->dev, "Audio DMA Controller\n");
+ dev_info(&pdev->dev, "imprint %x TX cache %u RX cache %u\n",
+ readl_relaxed(ad->base + REG_IMPRINT), ad->txcache.size, ad->rxcache.size);
+
return 0;
free_irq:
diff --git a/drivers/dma/at_hdmac.c b/drivers/dma/at_hdmac.c
index 858bd64..8858470 100644
--- a/drivers/dma/at_hdmac.c
+++ b/drivers/dma/at_hdmac.c
@@ -3,6 +3,7 @@
* Driver for the Atmel AHB DMA Controller (aka HDMA or DMAC on AT91 systems)
*
* Copyright (C) 2008 Atmel Corporation
+ * Copyright (C) 2022 Microchip Technology, Inc. and its subsidiaries
*
* This supports the Atmel AHB DMA Controller found in several Atmel SoCs.
* The only Atmel DMA Controller that is not covered by this driver is the one
@@ -10,20 +11,22 @@
*/
#include <dt-bindings/dma/at91.h>
+#include <linux/bitfield.h>
#include <linux/clk.h>
#include <linux/dmaengine.h>
-#include <linux/dma-mapping.h>
#include <linux/dmapool.h>
+#include <linux/dma-mapping.h>
#include <linux/interrupt.h>
#include <linux/module.h>
-#include <linux/platform_device.h>
-#include <linux/slab.h>
#include <linux/of.h>
+#include <linux/overflow.h>
#include <linux/of_device.h>
#include <linux/of_dma.h>
+#include <linux/platform_device.h>
+#include <linux/slab.h>
-#include "at_hdmac_regs.h"
#include "dmaengine.h"
+#include "virt-dma.h"
/*
* Glossary
@@ -34,9 +37,449 @@
* atc_ / atchan : ATmel DMA Channel entity related
*/
-#define ATC_DEFAULT_CFG (ATC_FIFOCFG_HALFFIFO)
-#define ATC_DEFAULT_CTRLB (ATC_SIF(AT_DMA_MEM_IF) \
- |ATC_DIF(AT_DMA_MEM_IF))
+#define AT_DMA_MAX_NR_CHANNELS 8
+
+/* Global Configuration Register */
+#define AT_DMA_GCFG 0x00
+#define AT_DMA_IF_BIGEND(i) BIT((i)) /* AHB-Lite Interface i in Big-endian mode */
+#define AT_DMA_ARB_CFG BIT(4) /* Arbiter mode. */
+
+/* Controller Enable Register */
+#define AT_DMA_EN 0x04
+#define AT_DMA_ENABLE BIT(0)
+
+/* Software Single Request Register */
+#define AT_DMA_SREQ 0x08
+#define AT_DMA_SSREQ(x) BIT((x) << 1) /* Request a source single transfer on channel x */
+#define AT_DMA_DSREQ(x) BIT(1 + ((x) << 1)) /* Request a destination single transfer on channel x */
+
+/* Software Chunk Transfer Request Register */
+#define AT_DMA_CREQ 0x0c
+#define AT_DMA_SCREQ(x) BIT((x) << 1) /* Request a source chunk transfer on channel x */
+#define AT_DMA_DCREQ(x) BIT(1 + ((x) << 1)) /* Request a destination chunk transfer on channel x */
+
+/* Software Last Transfer Flag Register */
+#define AT_DMA_LAST 0x10
+#define AT_DMA_SLAST(x) BIT((x) << 1) /* This src rq is last tx of buffer on channel x */
+#define AT_DMA_DLAST(x) BIT(1 + ((x) << 1)) /* This dst rq is last tx of buffer on channel x */
+
+/* Request Synchronization Register */
+#define AT_DMA_SYNC 0x14
+#define AT_DMA_SYR(h) BIT((h)) /* Synchronize handshake line h */
+
+/* Error, Chained Buffer transfer completed and Buffer transfer completed Interrupt registers */
+#define AT_DMA_EBCIER 0x18 /* Enable register */
+#define AT_DMA_EBCIDR 0x1c /* Disable register */
+#define AT_DMA_EBCIMR 0x20 /* Mask Register */
+#define AT_DMA_EBCISR 0x24 /* Status Register */
+#define AT_DMA_CBTC_OFFSET 8
+#define AT_DMA_ERR_OFFSET 16
+#define AT_DMA_BTC(x) BIT((x))
+#define AT_DMA_CBTC(x) BIT(AT_DMA_CBTC_OFFSET + (x))
+#define AT_DMA_ERR(x) BIT(AT_DMA_ERR_OFFSET + (x))
+
+/* Channel Handler Enable Register */
+#define AT_DMA_CHER 0x28
+#define AT_DMA_ENA(x) BIT((x))
+#define AT_DMA_SUSP(x) BIT(8 + (x))
+#define AT_DMA_KEEP(x) BIT(24 + (x))
+
+/* Channel Handler Disable Register */
+#define AT_DMA_CHDR 0x2c
+#define AT_DMA_DIS(x) BIT(x)
+#define AT_DMA_RES(x) BIT(8 + (x))
+
+/* Channel Handler Status Register */
+#define AT_DMA_CHSR 0x30
+#define AT_DMA_EMPT(x) BIT(16 + (x))
+#define AT_DMA_STAL(x) BIT(24 + (x))
+
+/* Channel registers base address */
+#define AT_DMA_CH_REGS_BASE 0x3c
+#define ch_regs(x) (AT_DMA_CH_REGS_BASE + (x) * 0x28) /* Channel x base addr */
+
+/* Hardware register offset for each channel */
+#define ATC_SADDR_OFFSET 0x00 /* Source Address Register */
+#define ATC_DADDR_OFFSET 0x04 /* Destination Address Register */
+#define ATC_DSCR_OFFSET 0x08 /* Descriptor Address Register */
+#define ATC_CTRLA_OFFSET 0x0c /* Control A Register */
+#define ATC_CTRLB_OFFSET 0x10 /* Control B Register */
+#define ATC_CFG_OFFSET 0x14 /* Configuration Register */
+#define ATC_SPIP_OFFSET 0x18 /* Src PIP Configuration Register */
+#define ATC_DPIP_OFFSET 0x1c /* Dst PIP Configuration Register */
+
+
+/* Bitfield definitions */
+
+/* Bitfields in DSCR */
+#define ATC_DSCR_IF GENMASK(1, 0) /* Dsc feched via AHB-Lite Interface */
+
+/* Bitfields in CTRLA */
+#define ATC_BTSIZE_MAX GENMASK(15, 0) /* Maximum Buffer Transfer Size */
+#define ATC_BTSIZE GENMASK(15, 0) /* Buffer Transfer Size */
+#define ATC_SCSIZE GENMASK(18, 16) /* Source Chunk Transfer Size */
+#define ATC_DCSIZE GENMASK(22, 20) /* Destination Chunk Transfer Size */
+#define ATC_SRC_WIDTH GENMASK(25, 24) /* Source Single Transfer Size */
+#define ATC_DST_WIDTH GENMASK(29, 28) /* Destination Single Transfer Size */
+#define ATC_DONE BIT(31) /* Tx Done (only written back in descriptor) */
+
+/* Bitfields in CTRLB */
+#define ATC_SIF GENMASK(1, 0) /* Src tx done via AHB-Lite Interface i */
+#define ATC_DIF GENMASK(5, 4) /* Dst tx done via AHB-Lite Interface i */
+#define AT_DMA_MEM_IF 0x0 /* interface 0 as memory interface */
+#define AT_DMA_PER_IF 0x1 /* interface 1 as peripheral interface */
+#define ATC_SRC_PIP BIT(8) /* Source Picture-in-Picture enabled */
+#define ATC_DST_PIP BIT(12) /* Destination Picture-in-Picture enabled */
+#define ATC_SRC_DSCR_DIS BIT(16) /* Src Descriptor fetch disable */
+#define ATC_DST_DSCR_DIS BIT(20) /* Dst Descriptor fetch disable */
+#define ATC_FC GENMASK(22, 21) /* Choose Flow Controller */
+#define ATC_FC_MEM2MEM 0x0 /* Mem-to-Mem (DMA) */
+#define ATC_FC_MEM2PER 0x1 /* Mem-to-Periph (DMA) */
+#define ATC_FC_PER2MEM 0x2 /* Periph-to-Mem (DMA) */
+#define ATC_FC_PER2PER 0x3 /* Periph-to-Periph (DMA) */
+#define ATC_FC_PER2MEM_PER 0x4 /* Periph-to-Mem (Peripheral) */
+#define ATC_FC_MEM2PER_PER 0x5 /* Mem-to-Periph (Peripheral) */
+#define ATC_FC_PER2PER_SRCPER 0x6 /* Periph-to-Periph (Src Peripheral) */
+#define ATC_FC_PER2PER_DSTPER 0x7 /* Periph-to-Periph (Dst Peripheral) */
+#define ATC_SRC_ADDR_MODE GENMASK(25, 24)
+#define ATC_SRC_ADDR_MODE_INCR 0x0 /* Incrementing Mode */
+#define ATC_SRC_ADDR_MODE_DECR 0x1 /* Decrementing Mode */
+#define ATC_SRC_ADDR_MODE_FIXED 0x2 /* Fixed Mode */
+#define ATC_DST_ADDR_MODE GENMASK(29, 28)
+#define ATC_DST_ADDR_MODE_INCR 0x0 /* Incrementing Mode */
+#define ATC_DST_ADDR_MODE_DECR 0x1 /* Decrementing Mode */
+#define ATC_DST_ADDR_MODE_FIXED 0x2 /* Fixed Mode */
+#define ATC_IEN BIT(30) /* BTC interrupt enable (active low) */
+#define ATC_AUTO BIT(31) /* Auto multiple buffer tx enable */
+
+/* Bitfields in CFG */
+#define ATC_PER_MSB(h) ((0x30U & (h)) >> 4) /* Extract most significant bits of a handshaking identifier */
+
+#define ATC_SRC_PER GENMASK(3, 0) /* Channel src rq associated with periph handshaking ifc h */
+#define ATC_DST_PER GENMASK(7, 4) /* Channel dst rq associated with periph handshaking ifc h */
+#define ATC_SRC_REP BIT(8) /* Source Replay Mod */
+#define ATC_SRC_H2SEL BIT(9) /* Source Handshaking Mod */
+#define ATC_SRC_PER_MSB GENMASK(11, 10) /* Channel src rq (most significant bits) */
+#define ATC_DST_REP BIT(12) /* Destination Replay Mod */
+#define ATC_DST_H2SEL BIT(13) /* Destination Handshaking Mod */
+#define ATC_DST_PER_MSB GENMASK(15, 14) /* Channel dst rq (most significant bits) */
+#define ATC_SOD BIT(16) /* Stop On Done */
+#define ATC_LOCK_IF BIT(20) /* Interface Lock */
+#define ATC_LOCK_B BIT(21) /* AHB Bus Lock */
+#define ATC_LOCK_IF_L BIT(22) /* Master Interface Arbiter Lock */
+#define ATC_AHB_PROT GENMASK(26, 24) /* AHB Protection */
+#define ATC_FIFOCFG GENMASK(29, 28) /* FIFO Request Configuration */
+#define ATC_FIFOCFG_LARGESTBURST 0x0
+#define ATC_FIFOCFG_HALFFIFO 0x1
+#define ATC_FIFOCFG_ENOUGHSPACE 0x2
+
+/* Bitfields in SPIP */
+#define ATC_SPIP_HOLE GENMASK(15, 0)
+#define ATC_SPIP_BOUNDARY GENMASK(25, 16)
+
+/* Bitfields in DPIP */
+#define ATC_DPIP_HOLE GENMASK(15, 0)
+#define ATC_DPIP_BOUNDARY GENMASK(25, 16)
+
+#define ATC_SRC_PER_ID(id) (FIELD_PREP(ATC_SRC_PER_MSB, (id)) | \
+ FIELD_PREP(ATC_SRC_PER, (id)))
+#define ATC_DST_PER_ID(id) (FIELD_PREP(ATC_DST_PER_MSB, (id)) | \
+ FIELD_PREP(ATC_DST_PER, (id)))
+
+
+
+/*-- descriptors -----------------------------------------------------*/
+
+/* LLI == Linked List Item; aka DMA buffer descriptor */
+struct at_lli {
+ /* values that are not changed by hardware */
+ u32 saddr;
+ u32 daddr;
+ /* value that may get written back: */
+ u32 ctrla;
+ /* more values that are not changed by hardware */
+ u32 ctrlb;
+ u32 dscr; /* chain to next lli */
+};
+
+/**
+ * struct atdma_sg - atdma scatter gather entry
+ * @len: length of the current Linked List Item.
+ * @lli: linked list item that is passed to the DMA controller
+ * @lli_phys: physical address of the LLI.
+ */
+struct atdma_sg {
+ unsigned int len;
+ struct at_lli *lli;
+ dma_addr_t lli_phys;
+};
+
+/**
+ * struct at_desc - software descriptor
+ * @vd: pointer to the virtual dma descriptor.
+ * @atchan: pointer to the atmel dma channel.
+ * @total_len: total transaction byte count
+ * @sg_len: number of sg entries.
+ * @sg: array of sgs.
+ */
+struct at_desc {
+ struct virt_dma_desc vd;
+ struct at_dma_chan *atchan;
+ size_t total_len;
+ unsigned int sglen;
+ /* Interleaved data */
+ size_t boundary;
+ size_t dst_hole;
+ size_t src_hole;
+
+ /* Memset temporary buffer */
+ bool memset_buffer;
+ dma_addr_t memset_paddr;
+ int *memset_vaddr;
+ struct atdma_sg sg[];
+};
+
+/*-- Channels --------------------------------------------------------*/
+
+/**
+ * atc_status - information bits stored in channel status flag
+ *
+ * Manipulated with atomic operations.
+ */
+enum atc_status {
+ ATC_IS_PAUSED = 1,
+ ATC_IS_CYCLIC = 24,
+};
+
+/**
+ * struct at_dma_chan - internal representation of an Atmel HDMAC channel
+ * @vc: virtual dma channel entry.
+ * @atdma: pointer to the driver data.
+ * @ch_regs: memory mapped register base
+ * @mask: channel index in a mask
+ * @per_if: peripheral interface
+ * @mem_if: memory interface
+ * @status: transmit status information from irq/prep* functions
+ * to tasklet (use atomic operations)
+ * @save_cfg: configuration register that is saved on suspend/resume cycle
+ * @save_dscr: for cyclic operations, preserve next descriptor address in
+ * the cyclic list on suspend/resume cycle
+ * @dma_sconfig: configuration for slave transfers, passed via
+ * .device_config
+ * @desc: pointer to the atmel dma descriptor.
+ */
+struct at_dma_chan {
+ struct virt_dma_chan vc;
+ struct at_dma *atdma;
+ void __iomem *ch_regs;
+ u8 mask;
+ u8 per_if;
+ u8 mem_if;
+ unsigned long status;
+ u32 save_cfg;
+ u32 save_dscr;
+ struct dma_slave_config dma_sconfig;
+ bool cyclic;
+ struct at_desc *desc;
+};
+
+#define channel_readl(atchan, name) \
+ __raw_readl((atchan)->ch_regs + ATC_##name##_OFFSET)
+
+#define channel_writel(atchan, name, val) \
+ __raw_writel((val), (atchan)->ch_regs + ATC_##name##_OFFSET)
+
+/*
+ * Fix sconfig's burst size according to at_hdmac. We need to convert them as:
+ * 1 -> 0, 4 -> 1, 8 -> 2, 16 -> 3, 32 -> 4, 64 -> 5, 128 -> 6, 256 -> 7.
+ *
+ * This can be done by finding most significant bit set.
+ */
+static inline void convert_burst(u32 *maxburst)
+{
+ if (*maxburst > 1)
+ *maxburst = fls(*maxburst) - 2;
+ else
+ *maxburst = 0;
+}
+
+/*
+ * Fix sconfig's bus width according to at_hdmac.
+ * 1 byte -> 0, 2 bytes -> 1, 4 bytes -> 2.
+ */
+static inline u8 convert_buswidth(enum dma_slave_buswidth addr_width)
+{
+ switch (addr_width) {
+ case DMA_SLAVE_BUSWIDTH_2_BYTES:
+ return 1;
+ case DMA_SLAVE_BUSWIDTH_4_BYTES:
+ return 2;
+ default:
+ /* For 1 byte width or fallback */
+ return 0;
+ }
+}
+
+/*-- Controller ------------------------------------------------------*/
+
+/**
+ * struct at_dma - internal representation of an Atmel HDMA Controller
+ * @dma_device: dmaengine dma_device object members
+ * @atdma_devtype: identifier of DMA controller compatibility
+ * @ch_regs: memory mapped register base
+ * @clk: dma controller clock
+ * @save_imr: interrupt mask register that is saved on suspend/resume cycle
+ * @all_chan_mask: all channels availlable in a mask
+ * @lli_pool: hw lli table
+ * @chan: channels table to store at_dma_chan structures
+ */
+struct at_dma {
+ struct dma_device dma_device;
+ void __iomem *regs;
+ struct clk *clk;
+ u32 save_imr;
+
+ u8 all_chan_mask;
+
+ struct dma_pool *lli_pool;
+ struct dma_pool *memset_pool;
+ /* AT THE END channels table */
+ struct at_dma_chan chan[];
+};
+
+#define dma_readl(atdma, name) \
+ __raw_readl((atdma)->regs + AT_DMA_##name)
+#define dma_writel(atdma, name, val) \
+ __raw_writel((val), (atdma)->regs + AT_DMA_##name)
+
+static inline struct at_desc *to_atdma_desc(struct dma_async_tx_descriptor *t)
+{
+ return container_of(t, struct at_desc, vd.tx);
+}
+
+static inline struct at_dma_chan *to_at_dma_chan(struct dma_chan *chan)
+{
+ return container_of(chan, struct at_dma_chan, vc.chan);
+}
+
+static inline struct at_dma *to_at_dma(struct dma_device *ddev)
+{
+ return container_of(ddev, struct at_dma, dma_device);
+}
+
+
+/*-- Helper functions ------------------------------------------------*/
+
+static struct device *chan2dev(struct dma_chan *chan)
+{
+ return &chan->dev->device;
+}
+
+#if defined(VERBOSE_DEBUG)
+static void vdbg_dump_regs(struct at_dma_chan *atchan)
+{
+ struct at_dma *atdma = to_at_dma(atchan->vc.chan.device);
+
+ dev_err(chan2dev(&atchan->vc.chan),
+ " channel %d : imr = 0x%x, chsr = 0x%x\n",
+ atchan->vc.chan.chan_id,
+ dma_readl(atdma, EBCIMR),
+ dma_readl(atdma, CHSR));
+
+ dev_err(chan2dev(&atchan->vc.chan),
+ " channel: s0x%x d0x%x ctrl0x%x:0x%x cfg0x%x l0x%x\n",
+ channel_readl(atchan, SADDR),
+ channel_readl(atchan, DADDR),
+ channel_readl(atchan, CTRLA),
+ channel_readl(atchan, CTRLB),
+ channel_readl(atchan, CFG),
+ channel_readl(atchan, DSCR));
+}
+#else
+static void vdbg_dump_regs(struct at_dma_chan *atchan) {}
+#endif
+
+static void atc_dump_lli(struct at_dma_chan *atchan, struct at_lli *lli)
+{
+ dev_crit(chan2dev(&atchan->vc.chan),
+ "desc: s%pad d%pad ctrl0x%x:0x%x l%pad\n",
+ &lli->saddr, &lli->daddr,
+ lli->ctrla, lli->ctrlb, &lli->dscr);
+}
+
+
+static void atc_setup_irq(struct at_dma *atdma, int chan_id, int on)
+{
+ u32 ebci;
+
+ /* enable interrupts on buffer transfer completion & error */
+ ebci = AT_DMA_BTC(chan_id)
+ | AT_DMA_ERR(chan_id);
+ if (on)
+ dma_writel(atdma, EBCIER, ebci);
+ else
+ dma_writel(atdma, EBCIDR, ebci);
+}
+
+static void atc_enable_chan_irq(struct at_dma *atdma, int chan_id)
+{
+ atc_setup_irq(atdma, chan_id, 1);
+}
+
+static void atc_disable_chan_irq(struct at_dma *atdma, int chan_id)
+{
+ atc_setup_irq(atdma, chan_id, 0);
+}
+
+
+/**
+ * atc_chan_is_enabled - test if given channel is enabled
+ * @atchan: channel we want to test status
+ */
+static inline int atc_chan_is_enabled(struct at_dma_chan *atchan)
+{
+ struct at_dma *atdma = to_at_dma(atchan->vc.chan.device);
+
+ return !!(dma_readl(atdma, CHSR) & atchan->mask);
+}
+
+/**
+ * atc_chan_is_paused - test channel pause/resume status
+ * @atchan: channel we want to test status
+ */
+static inline int atc_chan_is_paused(struct at_dma_chan *atchan)
+{
+ return test_bit(ATC_IS_PAUSED, &atchan->status);
+}
+
+/**
+ * atc_chan_is_cyclic - test if given channel has cyclic property set
+ * @atchan: channel we want to test status
+ */
+static inline int atc_chan_is_cyclic(struct at_dma_chan *atchan)
+{
+ return test_bit(ATC_IS_CYCLIC, &atchan->status);
+}
+
+/**
+ * set_lli_eol - set end-of-link to descriptor so it will end transfer
+ * @desc: descriptor, signle or at the end of a chain, to end chain on
+ * @i: index of the atmel scatter gather entry that is at the end of the chain.
+ */
+static void set_lli_eol(struct at_desc *desc, unsigned int i)
+{
+ u32 ctrlb = desc->sg[i].lli->ctrlb;
+
+ ctrlb &= ~ATC_IEN;
+ ctrlb |= ATC_SRC_DSCR_DIS | ATC_DST_DSCR_DIS;
+
+ desc->sg[i].lli->ctrlb = ctrlb;
+ desc->sg[i].lli->dscr = 0;
+}
+
+#define ATC_DEFAULT_CFG FIELD_PREP(ATC_FIFOCFG, ATC_FIFOCFG_HALFFIFO)
+#define ATC_DEFAULT_CTRLB (FIELD_PREP(ATC_SIF, AT_DMA_MEM_IF) | \
+ FIELD_PREP(ATC_DIF, AT_DMA_MEM_IF))
#define ATC_DMA_BUSWIDTHS\
(BIT(DMA_SLAVE_BUSWIDTH_UNDEFINED) |\
BIT(DMA_SLAVE_BUSWIDTH_1_BYTE) |\
@@ -74,13 +517,6 @@ struct at_dma_slave {
u32 cfg;
};
-/* prototypes */
-static dma_cookie_t atc_tx_submit(struct dma_async_tx_descriptor *tx);
-static void atc_issue_pending(struct dma_chan *chan);
-
-
-/*----------------------------------------------------------------------*/
-
static inline unsigned int atc_get_xfer_width(dma_addr_t src, dma_addr_t dst,
size_t len)
{
@@ -96,194 +532,72 @@ static inline unsigned int atc_get_xfer_width(dma_addr_t src, dma_addr_t dst,
return width;
}
-static struct at_desc *atc_first_active(struct at_dma_chan *atchan)
+static void atdma_lli_chain(struct at_desc *desc, unsigned int i)
{
- return list_first_entry(&atchan->active_list,
- struct at_desc, desc_node);
-}
+ struct atdma_sg *atdma_sg = &desc->sg[i];
-static struct at_desc *atc_first_queued(struct at_dma_chan *atchan)
-{
- return list_first_entry(&atchan->queue,
- struct at_desc, desc_node);
-}
-
-/**
- * atc_alloc_descriptor - allocate and return an initialized descriptor
- * @chan: the channel to allocate descriptors for
- * @gfp_flags: GFP allocation flags
- *
- * Note: The ack-bit is positioned in the descriptor flag at creation time
- * to make initial allocation more convenient. This bit will be cleared
- * and control will be given to client at usage time (during
- * preparation functions).
- */
-static struct at_desc *atc_alloc_descriptor(struct dma_chan *chan,
- gfp_t gfp_flags)
-{
- struct at_desc *desc = NULL;
- struct at_dma *atdma = to_at_dma(chan->device);
- dma_addr_t phys;
-
- desc = dma_pool_zalloc(atdma->dma_desc_pool, gfp_flags, &phys);
- if (desc) {
- INIT_LIST_HEAD(&desc->tx_list);
- dma_async_tx_descriptor_init(&desc->txd, chan);
- /* txd.flags will be overwritten in prep functions */
- desc->txd.flags = DMA_CTRL_ACK;
- desc->txd.tx_submit = atc_tx_submit;
- desc->txd.phys = phys;
- }
-
- return desc;
-}
-
-/**
- * atc_desc_get - get an unused descriptor from free_list
- * @atchan: channel we want a new descriptor for
- */
-static struct at_desc *atc_desc_get(struct at_dma_chan *atchan)
-{
- struct at_desc *desc, *_desc;
- struct at_desc *ret = NULL;
- unsigned long flags;
- unsigned int i = 0;
-
- spin_lock_irqsave(&atchan->lock, flags);
- list_for_each_entry_safe(desc, _desc, &atchan->free_list, desc_node) {
- i++;
- if (async_tx_test_ack(&desc->txd)) {
- list_del(&desc->desc_node);
- ret = desc;
- break;
- }
- dev_dbg(chan2dev(&atchan->chan_common),
- "desc %p not ACKed\n", desc);
- }
- spin_unlock_irqrestore(&atchan->lock, flags);
- dev_vdbg(chan2dev(&atchan->chan_common),
- "scanned %u descriptors on freelist\n", i);
-
- /* no more descriptor available in initial pool: create one more */
- if (!ret)
- ret = atc_alloc_descriptor(&atchan->chan_common, GFP_NOWAIT);
-
- return ret;
-}
-
-/**
- * atc_desc_put - move a descriptor, including any children, to the free list
- * @atchan: channel we work on
- * @desc: descriptor, at the head of a chain, to move to free list
- */
-static void atc_desc_put(struct at_dma_chan *atchan, struct at_desc *desc)
-{
- if (desc) {
- struct at_desc *child;
- unsigned long flags;
-
- spin_lock_irqsave(&atchan->lock, flags);
- list_for_each_entry(child, &desc->tx_list, desc_node)
- dev_vdbg(chan2dev(&atchan->chan_common),
- "moving child desc %p to freelist\n",
- child);
- list_splice_init(&desc->tx_list, &atchan->free_list);
- dev_vdbg(chan2dev(&atchan->chan_common),
- "moving desc %p to freelist\n", desc);
- list_add(&desc->desc_node, &atchan->free_list);
- spin_unlock_irqrestore(&atchan->lock, flags);
- }
-}
-
-/**
- * atc_desc_chain - build chain adding a descriptor
- * @first: address of first descriptor of the chain
- * @prev: address of previous descriptor of the chain
- * @desc: descriptor to queue
- *
- * Called from prep_* functions
- */
-static void atc_desc_chain(struct at_desc **first, struct at_desc **prev,
- struct at_desc *desc)
-{
- if (!(*first)) {
- *first = desc;
- } else {
- /* inform the HW lli about chaining */
- (*prev)->lli.dscr = desc->txd.phys;
- /* insert the link descriptor to the LD ring */
- list_add_tail(&desc->desc_node,
- &(*first)->tx_list);
- }
- *prev = desc;
+ if (i)
+ desc->sg[i - 1].lli->dscr = atdma_sg->lli_phys;
}
/**
* atc_dostart - starts the DMA engine for real
* @atchan: the channel we want to start
- * @first: first descriptor in the list we want to begin with
- *
- * Called with atchan->lock held and bh disabled
*/
-static void atc_dostart(struct at_dma_chan *atchan, struct at_desc *first)
+static void atc_dostart(struct at_dma_chan *atchan)
{
- struct at_dma *atdma = to_at_dma(atchan->chan_common.device);
+ struct virt_dma_desc *vd = vchan_next_desc(&atchan->vc);
+ struct at_desc *desc;
- /* ASSERT: channel is idle */
- if (atc_chan_is_enabled(atchan)) {
- dev_err(chan2dev(&atchan->chan_common),
- "BUG: Attempted to start non-idle channel\n");
- dev_err(chan2dev(&atchan->chan_common),
- " channel: s0x%x d0x%x ctrl0x%x:0x%x l0x%x\n",
- channel_readl(atchan, SADDR),
- channel_readl(atchan, DADDR),
- channel_readl(atchan, CTRLA),
- channel_readl(atchan, CTRLB),
- channel_readl(atchan, DSCR));
-
- /* The tasklet will hopefully advance the queue... */
+ if (!vd) {
+ atchan->desc = NULL;
return;
}
vdbg_dump_regs(atchan);
+ list_del(&vd->node);
+ atchan->desc = desc = to_atdma_desc(&vd->tx);
+
channel_writel(atchan, SADDR, 0);
channel_writel(atchan, DADDR, 0);
channel_writel(atchan, CTRLA, 0);
channel_writel(atchan, CTRLB, 0);
- channel_writel(atchan, DSCR, first->txd.phys);
- channel_writel(atchan, SPIP, ATC_SPIP_HOLE(first->src_hole) |
- ATC_SPIP_BOUNDARY(first->boundary));
- channel_writel(atchan, DPIP, ATC_DPIP_HOLE(first->dst_hole) |
- ATC_DPIP_BOUNDARY(first->boundary));
+ channel_writel(atchan, DSCR, desc->sg[0].lli_phys);
+ channel_writel(atchan, SPIP,
+ FIELD_PREP(ATC_SPIP_HOLE, desc->src_hole) |
+ FIELD_PREP(ATC_SPIP_BOUNDARY, desc->boundary));
+ channel_writel(atchan, DPIP,
+ FIELD_PREP(ATC_DPIP_HOLE, desc->dst_hole) |
+ FIELD_PREP(ATC_DPIP_BOUNDARY, desc->boundary));
+
/* Don't allow CPU to reorder channel enable. */
wmb();
- dma_writel(atdma, CHER, atchan->mask);
+ dma_writel(atchan->atdma, CHER, atchan->mask);
vdbg_dump_regs(atchan);
}
-/*
- * atc_get_desc_by_cookie - get the descriptor of a cookie
- * @atchan: the DMA channel
- * @cookie: the cookie to get the descriptor for
- */
-static struct at_desc *atc_get_desc_by_cookie(struct at_dma_chan *atchan,
- dma_cookie_t cookie)
+static void atdma_desc_free(struct virt_dma_desc *vd)
{
- struct at_desc *desc, *_desc;
+ struct at_dma *atdma = to_at_dma(vd->tx.chan->device);
+ struct at_desc *desc = to_atdma_desc(&vd->tx);
+ unsigned int i;
- list_for_each_entry_safe(desc, _desc, &atchan->queue, desc_node) {
- if (desc->txd.cookie == cookie)
- return desc;
+ for (i = 0; i < desc->sglen; i++) {
+ if (desc->sg[i].lli)
+ dma_pool_free(atdma->lli_pool, desc->sg[i].lli,
+ desc->sg[i].lli_phys);
}
- list_for_each_entry_safe(desc, _desc, &atchan->active_list, desc_node) {
- if (desc->txd.cookie == cookie)
- return desc;
+ /* If the transfer was a memset, free our temporary buffer */
+ if (desc->memset_buffer) {
+ dma_pool_free(atdma->memset_pool, desc->memset_vaddr,
+ desc->memset_paddr);
+ desc->memset_buffer = false;
}
- return NULL;
+ kfree(desc);
}
/**
@@ -293,10 +607,10 @@ static struct at_desc *atc_get_desc_by_cookie(struct at_dma_chan *atchan,
* @current_len: the number of bytes left before reading CTRLA
* @ctrla: the value of CTRLA
*/
-static inline int atc_calc_bytes_left(int current_len, u32 ctrla)
+static inline u32 atc_calc_bytes_left(u32 current_len, u32 ctrla)
{
- u32 btsize = (ctrla & ATC_BTSIZE_MAX);
- u32 src_width = ATC_REG_TO_SRC_WIDTH(ctrla);
+ u32 btsize = FIELD_GET(ATC_BTSIZE, ctrla);
+ u32 src_width = FIELD_GET(ATC_SRC_WIDTH, ctrla);
/*
* According to the datasheet, when reading the Control A Register
@@ -308,246 +622,153 @@ static inline int atc_calc_bytes_left(int current_len, u32 ctrla)
}
/**
- * atc_get_bytes_left - get the number of bytes residue for a cookie
- * @chan: DMA channel
- * @cookie: transaction identifier to check status of
+ * atc_get_llis_residue - Get residue for a hardware linked list transfer
+ *
+ * Calculate the residue by removing the length of the Linked List Item (LLI)
+ * already transferred from the total length. To get the current LLI we can use
+ * the value of the channel's DSCR register and compare it against the DSCR
+ * value of each LLI.
+ *
+ * The CTRLA register provides us with the amount of data already read from the
+ * source for the LLI. So we can compute a more accurate residue by also
+ * removing the number of bytes corresponding to this amount of data.
+ *
+ * However, the DSCR and CTRLA registers cannot be read both atomically. Hence a
+ * race condition may occur: the first read register may refer to one LLI
+ * whereas the second read may refer to a later LLI in the list because of the
+ * DMA transfer progression inbetween the two reads.
+ *
+ * One solution could have been to pause the DMA transfer, read the DSCR and
+ * CTRLA then resume the DMA transfer. Nonetheless, this approach presents some
+ * drawbacks:
+ * - If the DMA transfer is paused, RX overruns or TX underruns are more likey
+ * to occur depending on the system latency. Taking the USART driver as an
+ * example, it uses a cyclic DMA transfer to read data from the Receive
+ * Holding Register (RHR) to avoid RX overruns since the RHR is not protected
+ * by any FIFO on most Atmel SoCs. So pausing the DMA transfer to compute the
+ * residue would break the USART driver design.
+ * - The atc_pause() function masks interrupts but we'd rather avoid to do so
+ * for system latency purpose.
+ *
+ * Then we'd rather use another solution: the DSCR is read a first time, the
+ * CTRLA is read in turn, next the DSCR is read a second time. If the two
+ * consecutive read values of the DSCR are the same then we assume both refers
+ * to the very same LLI as well as the CTRLA value read inbetween does. For
+ * cyclic tranfers, the assumption is that a full loop is "not so fast". If the
+ * two DSCR values are different, we read again the CTRLA then the DSCR till two
+ * consecutive read values from DSCR are equal or till the maximum trials is
+ * reach. This algorithm is very unlikely not to find a stable value for DSCR.
+ * @atchan: pointer to an atmel hdmac channel.
+ * @desc: pointer to the descriptor for which the residue is calculated.
+ * @residue: residue to be set to dma_tx_state.
+ * Returns 0 on success, -errno otherwise.
*/
-static int atc_get_bytes_left(struct dma_chan *chan, dma_cookie_t cookie)
+static int atc_get_llis_residue(struct at_dma_chan *atchan,
+ struct at_desc *desc, u32 *residue)
{
- struct at_dma_chan *atchan = to_at_dma_chan(chan);
- struct at_desc *desc_first = atc_first_active(atchan);
- struct at_desc *desc;
- int ret;
- u32 ctrla, dscr;
+ u32 len, ctrla, dscr;
unsigned int i;
- /*
- * If the cookie doesn't match to the currently running transfer then
- * we can return the total length of the associated DMA transfer,
- * because it is still queued.
- */
- desc = atc_get_desc_by_cookie(atchan, cookie);
- if (desc == NULL)
- return -EINVAL;
- else if (desc != desc_first)
- return desc->total_len;
+ len = desc->total_len;
+ dscr = channel_readl(atchan, DSCR);
+ rmb(); /* ensure DSCR is read before CTRLA */
+ ctrla = channel_readl(atchan, CTRLA);
+ for (i = 0; i < ATC_MAX_DSCR_TRIALS; ++i) {
+ u32 new_dscr;
- /* cookie matches to the currently running transfer */
- ret = desc_first->total_len;
-
- if (desc_first->lli.dscr) {
- /* hardware linked list transfer */
+ rmb(); /* ensure DSCR is read after CTRLA */
+ new_dscr = channel_readl(atchan, DSCR);
/*
- * Calculate the residue by removing the length of the child
- * descriptors already transferred from the total length.
- * To get the current child descriptor we can use the value of
- * the channel's DSCR register and compare it against the value
- * of the hardware linked list structure of each child
- * descriptor.
- *
- * The CTRLA register provides us with the amount of data
- * already read from the source for the current child
- * descriptor. So we can compute a more accurate residue by also
- * removing the number of bytes corresponding to this amount of
- * data.
- *
- * However, the DSCR and CTRLA registers cannot be read both
- * atomically. Hence a race condition may occur: the first read
- * register may refer to one child descriptor whereas the second
- * read may refer to a later child descriptor in the list
- * because of the DMA transfer progression inbetween the two
- * reads.
- *
- * One solution could have been to pause the DMA transfer, read
- * the DSCR and CTRLA then resume the DMA transfer. Nonetheless,
- * this approach presents some drawbacks:
- * - If the DMA transfer is paused, RX overruns or TX underruns
- * are more likey to occur depending on the system latency.
- * Taking the USART driver as an example, it uses a cyclic DMA
- * transfer to read data from the Receive Holding Register
- * (RHR) to avoid RX overruns since the RHR is not protected
- * by any FIFO on most Atmel SoCs. So pausing the DMA transfer
- * to compute the residue would break the USART driver design.
- * - The atc_pause() function masks interrupts but we'd rather
- * avoid to do so for system latency purpose.
- *
- * Then we'd rather use another solution: the DSCR is read a
- * first time, the CTRLA is read in turn, next the DSCR is read
- * a second time. If the two consecutive read values of the DSCR
- * are the same then we assume both refers to the very same
- * child descriptor as well as the CTRLA value read inbetween
- * does. For cyclic tranfers, the assumption is that a full loop
- * is "not so fast".
- * If the two DSCR values are different, we read again the CTRLA
- * then the DSCR till two consecutive read values from DSCR are
- * equal or till the maxium trials is reach.
- * This algorithm is very unlikely not to find a stable value for
- * DSCR.
+ * If the DSCR register value has not changed inside the DMA
+ * controller since the previous read, we assume that both the
+ * dscr and ctrla values refers to the very same descriptor.
*/
+ if (likely(new_dscr == dscr))
+ break;
- dscr = channel_readl(atchan, DSCR);
+ /*
+ * DSCR has changed inside the DMA controller, so the previouly
+ * read value of CTRLA may refer to an already processed
+ * descriptor hence could be outdated. We need to update ctrla
+ * to match the current descriptor.
+ */
+ dscr = new_dscr;
rmb(); /* ensure DSCR is read before CTRLA */
ctrla = channel_readl(atchan, CTRLA);
- for (i = 0; i < ATC_MAX_DSCR_TRIALS; ++i) {
- u32 new_dscr;
+ }
+ if (unlikely(i == ATC_MAX_DSCR_TRIALS))
+ return -ETIMEDOUT;
- rmb(); /* ensure DSCR is read after CTRLA */
- new_dscr = channel_readl(atchan, DSCR);
+ /* For the first descriptor we can be more accurate. */
+ if (desc->sg[0].lli->dscr == dscr) {
+ *residue = atc_calc_bytes_left(len, ctrla);
+ return 0;
+ }
+ len -= desc->sg[0].len;
- /*
- * If the DSCR register value has not changed inside the
- * DMA controller since the previous read, we assume
- * that both the dscr and ctrla values refers to the
- * very same descriptor.
- */
- if (likely(new_dscr == dscr))
- break;
-
- /*
- * DSCR has changed inside the DMA controller, so the
- * previouly read value of CTRLA may refer to an already
- * processed descriptor hence could be outdated.
- * We need to update ctrla to match the current
- * descriptor.
- */
- dscr = new_dscr;
- rmb(); /* ensure DSCR is read before CTRLA */
- ctrla = channel_readl(atchan, CTRLA);
- }
- if (unlikely(i == ATC_MAX_DSCR_TRIALS))
- return -ETIMEDOUT;
-
- /* for the first descriptor we can be more accurate */
- if (desc_first->lli.dscr == dscr)
- return atc_calc_bytes_left(ret, ctrla);
-
- ret -= desc_first->len;
- list_for_each_entry(desc, &desc_first->tx_list, desc_node) {
- if (desc->lli.dscr == dscr)
- break;
-
- ret -= desc->len;
- }
-
- /*
- * For the current descriptor in the chain we can calculate
- * the remaining bytes using the channel's register.
- */
- ret = atc_calc_bytes_left(ret, ctrla);
- } else {
- /* single transfer */
- ctrla = channel_readl(atchan, CTRLA);
- ret = atc_calc_bytes_left(ret, ctrla);
+ for (i = 1; i < desc->sglen; i++) {
+ if (desc->sg[i].lli && desc->sg[i].lli->dscr == dscr)
+ break;
+ len -= desc->sg[i].len;
}
- return ret;
+ /*
+ * For the current LLI in the chain we can calculate the remaining bytes
+ * using the channel's CTRLA register.
+ */
+ *residue = atc_calc_bytes_left(len, ctrla);
+ return 0;
+
}
/**
- * atc_chain_complete - finish work for one transaction chain
- * @atchan: channel we work on
- * @desc: descriptor at the head of the chain we want do complete
+ * atc_get_residue - get the number of bytes residue for a cookie.
+ * The residue is passed by address and updated on success.
+ * @chan: DMA channel
+ * @cookie: transaction identifier to check status of
+ * @residue: residue to be updated.
+ * Return 0 on success, -errono otherwise.
*/
-static void
-atc_chain_complete(struct at_dma_chan *atchan, struct at_desc *desc)
+static int atc_get_residue(struct dma_chan *chan, dma_cookie_t cookie,
+ u32 *residue)
{
- struct dma_async_tx_descriptor *txd = &desc->txd;
- struct at_dma *atdma = to_at_dma(atchan->chan_common.device);
- unsigned long flags;
+ struct at_dma_chan *atchan = to_at_dma_chan(chan);
+ struct virt_dma_desc *vd;
+ struct at_desc *desc = NULL;
+ u32 len, ctrla;
- dev_vdbg(chan2dev(&atchan->chan_common),
- "descriptor %u complete\n", txd->cookie);
+ vd = vchan_find_desc(&atchan->vc, cookie);
+ if (vd)
+ desc = to_atdma_desc(&vd->tx);
+ else if (atchan->desc && atchan->desc->vd.tx.cookie == cookie)
+ desc = atchan->desc;
- spin_lock_irqsave(&atchan->lock, flags);
+ if (!desc)
+ return -EINVAL;
- /* mark the descriptor as complete for non cyclic cases only */
- if (!atc_chan_is_cyclic(atchan))
- dma_cookie_complete(txd);
+ if (desc->sg[0].lli->dscr)
+ /* hardware linked list transfer */
+ return atc_get_llis_residue(atchan, desc, residue);
- spin_unlock_irqrestore(&atchan->lock, flags);
-
- dma_descriptor_unmap(txd);
- /* for cyclic transfers,
- * no need to replay callback function while stopping */
- if (!atc_chan_is_cyclic(atchan))
- dmaengine_desc_get_callback_invoke(txd, NULL);
-
- dma_run_dependencies(txd);
-
- spin_lock_irqsave(&atchan->lock, flags);
- /* move children to free_list */
- list_splice_init(&desc->tx_list, &atchan->free_list);
- /* add myself to free_list */
- list_add(&desc->desc_node, &atchan->free_list);
- spin_unlock_irqrestore(&atchan->lock, flags);
-
- /* If the transfer was a memset, free our temporary buffer */
- if (desc->memset_buffer) {
- dma_pool_free(atdma->memset_pool, desc->memset_vaddr,
- desc->memset_paddr);
- desc->memset_buffer = false;
- }
+ /* single transfer */
+ len = desc->total_len;
+ ctrla = channel_readl(atchan, CTRLA);
+ *residue = atc_calc_bytes_left(len, ctrla);
+ return 0;
}
/**
- * atc_advance_work - at the end of a transaction, move forward
- * @atchan: channel where the transaction ended
- */
-static void atc_advance_work(struct at_dma_chan *atchan)
-{
- struct at_desc *desc;
- unsigned long flags;
-
- dev_vdbg(chan2dev(&atchan->chan_common), "advance_work\n");
-
- spin_lock_irqsave(&atchan->lock, flags);
- if (atc_chan_is_enabled(atchan) || list_empty(&atchan->active_list))
- return spin_unlock_irqrestore(&atchan->lock, flags);
-
- desc = atc_first_active(atchan);
- /* Remove the transfer node from the active list. */
- list_del_init(&desc->desc_node);
- spin_unlock_irqrestore(&atchan->lock, flags);
- atc_chain_complete(atchan, desc);
-
- /* advance work */
- spin_lock_irqsave(&atchan->lock, flags);
- if (!list_empty(&atchan->active_list)) {
- desc = atc_first_queued(atchan);
- list_move_tail(&desc->desc_node, &atchan->active_list);
- atc_dostart(atchan, desc);
- }
- spin_unlock_irqrestore(&atchan->lock, flags);
-}
-
-
-/**
* atc_handle_error - handle errors reported by DMA controller
- * @atchan: channel where error occurs
+ * @atchan: channel where error occurs.
+ * @i: channel index
*/
-static void atc_handle_error(struct at_dma_chan *atchan)
+static void atc_handle_error(struct at_dma_chan *atchan, unsigned int i)
{
- struct at_desc *bad_desc;
- struct at_desc *desc;
- struct at_desc *child;
- unsigned long flags;
+ struct at_desc *desc = atchan->desc;
- spin_lock_irqsave(&atchan->lock, flags);
- /*
- * The descriptor currently at the head of the active list is
- * broked. Since we don't have any way to report errors, we'll
- * just have to scream loudly and try to carry on.
- */
- bad_desc = atc_first_active(atchan);
- list_del_init(&bad_desc->desc_node);
-
- /* Try to restart the controller */
- if (!list_empty(&atchan->active_list)) {
- desc = atc_first_queued(atchan);
- list_move_tail(&desc->desc_node, &atchan->active_list);
- atc_dostart(atchan, desc);
- }
+ /* Disable channel on AHB error */
+ dma_writel(atchan->atdma, CHDR, AT_DMA_RES(i) | atchan->mask);
/*
* KERN_CRITICAL may seem harsh, but since this only happens
@@ -556,54 +777,42 @@ static void atc_handle_error(struct at_dma_chan *atchan)
* controller flagged an error instead of scribbling over
* random memory locations.
*/
- dev_crit(chan2dev(&atchan->chan_common),
- "Bad descriptor submitted for DMA!\n");
- dev_crit(chan2dev(&atchan->chan_common),
- " cookie: %d\n", bad_desc->txd.cookie);
- atc_dump_lli(atchan, &bad_desc->lli);
- list_for_each_entry(child, &bad_desc->tx_list, desc_node)
- atc_dump_lli(atchan, &child->lli);
-
- spin_unlock_irqrestore(&atchan->lock, flags);
-
- /* Pretend the descriptor completed successfully */
- atc_chain_complete(atchan, bad_desc);
+ dev_crit(chan2dev(&atchan->vc.chan), "Bad descriptor submitted for DMA!\n");
+ dev_crit(chan2dev(&atchan->vc.chan), "cookie: %d\n",
+ desc->vd.tx.cookie);
+ for (i = 0; i < desc->sglen; i++)
+ atc_dump_lli(atchan, desc->sg[i].lli);
}
-/**
- * atc_handle_cyclic - at the end of a period, run callback function
- * @atchan: channel used for cyclic operations
- */
-static void atc_handle_cyclic(struct at_dma_chan *atchan)
+static void atdma_handle_chan_done(struct at_dma_chan *atchan, u32 pending,
+ unsigned int i)
{
- struct at_desc *first = atc_first_active(atchan);
- struct dma_async_tx_descriptor *txd = &first->txd;
+ struct at_desc *desc;
- dev_vdbg(chan2dev(&atchan->chan_common),
- "new cyclic period llp 0x%08x\n",
- channel_readl(atchan, DSCR));
+ spin_lock(&atchan->vc.lock);
+ desc = atchan->desc;
- dmaengine_desc_get_callback_invoke(txd, NULL);
-}
+ if (desc) {
+ if (pending & AT_DMA_ERR(i)) {
+ atc_handle_error(atchan, i);
+ /* Pretend the descriptor completed successfully */
+ }
-/*-- IRQ & Tasklet ---------------------------------------------------*/
-
-static void atc_tasklet(struct tasklet_struct *t)
-{
- struct at_dma_chan *atchan = from_tasklet(atchan, t, tasklet);
-
- if (test_and_clear_bit(ATC_IS_ERROR, &atchan->status))
- return atc_handle_error(atchan);
-
- if (atc_chan_is_cyclic(atchan))
- return atc_handle_cyclic(atchan);
-
- atc_advance_work(atchan);
+ if (atc_chan_is_cyclic(atchan)) {
+ vchan_cyclic_callback(&desc->vd);
+ } else {
+ vchan_cookie_complete(&desc->vd);
+ atchan->desc = NULL;
+ if (!(atc_chan_is_enabled(atchan)))
+ atc_dostart(atchan);
+ }
+ }
+ spin_unlock(&atchan->vc.lock);
}
static irqreturn_t at_dma_interrupt(int irq, void *dev_id)
{
- struct at_dma *atdma = (struct at_dma *)dev_id;
+ struct at_dma *atdma = dev_id;
struct at_dma_chan *atchan;
int i;
u32 status, pending, imr;
@@ -617,23 +826,16 @@ static irqreturn_t at_dma_interrupt(int irq, void *dev_id)
if (!pending)
break;
- dev_vdbg(atdma->dma_common.dev,
+ dev_vdbg(atdma->dma_device.dev,
"interrupt: status = 0x%08x, 0x%08x, 0x%08x\n",
status, imr, pending);
- for (i = 0; i < atdma->dma_common.chancnt; i++) {
+ for (i = 0; i < atdma->dma_device.chancnt; i++) {
atchan = &atdma->chan[i];
- if (pending & (AT_DMA_BTC(i) | AT_DMA_ERR(i))) {
- if (pending & AT_DMA_ERR(i)) {
- /* Disable channel on AHB error */
- dma_writel(atdma, CHDR,
- AT_DMA_RES(i) | atchan->mask);
- /* Give information to tasklet */
- set_bit(ATC_IS_ERROR, &atchan->status);
- }
- tasklet_schedule(&atchan->tasklet);
- ret = IRQ_HANDLED;
- }
+ if (!(pending & (AT_DMA_BTC(i) | AT_DMA_ERR(i))))
+ continue;
+ atdma_handle_chan_done(atchan, pending, i);
+ ret = IRQ_HANDLED;
}
} while (pending);
@@ -641,35 +843,7 @@ static irqreturn_t at_dma_interrupt(int irq, void *dev_id)
return ret;
}
-
/*-- DMA Engine API --------------------------------------------------*/
-
-/**
- * atc_tx_submit - set the prepared descriptor(s) to be executed by the engine
- * @tx: descriptor at the head of the transaction chain
- *
- * Queue chain if DMA engine is working already
- *
- * Cookie increment and adding to active_list or queue must be atomic
- */
-static dma_cookie_t atc_tx_submit(struct dma_async_tx_descriptor *tx)
-{
- struct at_desc *desc = txd_to_at_desc(tx);
- struct at_dma_chan *atchan = to_at_dma_chan(tx->chan);
- dma_cookie_t cookie;
- unsigned long flags;
-
- spin_lock_irqsave(&atchan->lock, flags);
- cookie = dma_cookie_assign(tx);
-
- list_add_tail(&desc->desc_node, &atchan->queue);
- spin_unlock_irqrestore(&atchan->lock, flags);
-
- dev_vdbg(chan2dev(tx->chan), "tx_submit: queued %u\n",
- desc->txd.cookie);
- return cookie;
-}
-
/**
* atc_prep_dma_interleaved - prepare memory to memory interleaved operation
* @chan: the channel to prepare operation on
@@ -681,9 +855,12 @@ atc_prep_dma_interleaved(struct dma_chan *chan,
struct dma_interleaved_template *xt,
unsigned long flags)
{
+ struct at_dma *atdma = to_at_dma(chan->device);
struct at_dma_chan *atchan = to_at_dma_chan(chan);
struct data_chunk *first;
- struct at_desc *desc = NULL;
+ struct atdma_sg *atdma_sg;
+ struct at_desc *desc;
+ struct at_lli *lli;
size_t xfer_count;
unsigned int dwidth;
u32 ctrla;
@@ -722,8 +899,7 @@ atc_prep_dma_interleaved(struct dma_chan *chan,
len += chunk->size;
}
- dwidth = atc_get_xfer_width(xt->src_start,
- xt->dst_start, len);
+ dwidth = atc_get_xfer_width(xt->src_start, xt->dst_start, len);
xfer_count = len >> dwidth;
if (xfer_count > ATC_BTSIZE_MAX) {
@@ -731,42 +907,43 @@ atc_prep_dma_interleaved(struct dma_chan *chan,
return NULL;
}
- ctrla = ATC_SRC_WIDTH(dwidth) |
- ATC_DST_WIDTH(dwidth);
+ ctrla = FIELD_PREP(ATC_SRC_WIDTH, dwidth) |
+ FIELD_PREP(ATC_DST_WIDTH, dwidth);
- ctrlb = ATC_DEFAULT_CTRLB | ATC_IEN
- | ATC_SRC_ADDR_MODE_INCR
- | ATC_DST_ADDR_MODE_INCR
- | ATC_SRC_PIP
- | ATC_DST_PIP
- | ATC_FC_MEM2MEM;
+ ctrlb = ATC_DEFAULT_CTRLB | ATC_IEN |
+ FIELD_PREP(ATC_SRC_ADDR_MODE, ATC_SRC_ADDR_MODE_INCR) |
+ FIELD_PREP(ATC_DST_ADDR_MODE, ATC_DST_ADDR_MODE_INCR) |
+ ATC_SRC_PIP | ATC_DST_PIP |
+ FIELD_PREP(ATC_FC, ATC_FC_MEM2MEM);
- /* create the transfer */
- desc = atc_desc_get(atchan);
- if (!desc) {
- dev_err(chan2dev(chan),
- "%s: couldn't allocate our descriptor\n", __func__);
+ desc = kzalloc(struct_size(desc, sg, 1), GFP_ATOMIC);
+ if (!desc)
+ return NULL;
+ desc->sglen = 1;
+
+ atdma_sg = desc->sg;
+ atdma_sg->lli = dma_pool_alloc(atdma->lli_pool, GFP_NOWAIT,
+ &atdma_sg->lli_phys);
+ if (!atdma_sg->lli) {
+ kfree(desc);
return NULL;
}
+ lli = atdma_sg->lli;
- desc->lli.saddr = xt->src_start;
- desc->lli.daddr = xt->dst_start;
- desc->lli.ctrla = ctrla | xfer_count;
- desc->lli.ctrlb = ctrlb;
+ lli->saddr = xt->src_start;
+ lli->daddr = xt->dst_start;
+ lli->ctrla = ctrla | xfer_count;
+ lli->ctrlb = ctrlb;
desc->boundary = first->size >> dwidth;
desc->dst_hole = (dmaengine_get_dst_icg(xt, first) >> dwidth) + 1;
desc->src_hole = (dmaengine_get_src_icg(xt, first) >> dwidth) + 1;
- desc->txd.cookie = -EBUSY;
- desc->total_len = desc->len = len;
+ atdma_sg->len = len;
+ desc->total_len = len;
- /* set end-of-link to the last link descriptor of list*/
- set_desc_eol(desc);
-
- desc->txd.flags = flags; /* client is in control of this ack */
-
- return &desc->txd;
+ set_lli_eol(desc, 0);
+ return vchan_tx_prep(&atchan->vc, &desc->vd, flags);
}
/**
@@ -781,29 +958,36 @@ static struct dma_async_tx_descriptor *
atc_prep_dma_memcpy(struct dma_chan *chan, dma_addr_t dest, dma_addr_t src,
size_t len, unsigned long flags)
{
+ struct at_dma *atdma = to_at_dma(chan->device);
struct at_dma_chan *atchan = to_at_dma_chan(chan);
struct at_desc *desc = NULL;
- struct at_desc *first = NULL;
- struct at_desc *prev = NULL;
size_t xfer_count;
size_t offset;
+ size_t sg_len;
unsigned int src_width;
unsigned int dst_width;
+ unsigned int i;
u32 ctrla;
u32 ctrlb;
- dev_vdbg(chan2dev(chan), "prep_dma_memcpy: d%pad s%pad l0x%zx f0x%lx\n",
- &dest, &src, len, flags);
+ dev_dbg(chan2dev(chan), "prep_dma_memcpy: d%pad s%pad l0x%zx f0x%lx\n",
+ &dest, &src, len, flags);
if (unlikely(!len)) {
- dev_dbg(chan2dev(chan), "prep_dma_memcpy: length is zero!\n");
+ dev_err(chan2dev(chan), "prep_dma_memcpy: length is zero!\n");
return NULL;
}
- ctrlb = ATC_DEFAULT_CTRLB | ATC_IEN
- | ATC_SRC_ADDR_MODE_INCR
- | ATC_DST_ADDR_MODE_INCR
- | ATC_FC_MEM2MEM;
+ sg_len = DIV_ROUND_UP(len, ATC_BTSIZE_MAX);
+ desc = kzalloc(struct_size(desc, sg, sg_len), GFP_ATOMIC);
+ if (!desc)
+ return NULL;
+ desc->sglen = sg_len;
+
+ ctrlb = ATC_DEFAULT_CTRLB | ATC_IEN |
+ FIELD_PREP(ATC_SRC_ADDR_MODE, ATC_SRC_ADDR_MODE_INCR) |
+ FIELD_PREP(ATC_DST_ADDR_MODE, ATC_DST_ADDR_MODE_INCR) |
+ FIELD_PREP(ATC_FC, ATC_FC_MEM2MEM);
/*
* We can be a lot more clever here, but this should take care
@@ -811,82 +995,78 @@ atc_prep_dma_memcpy(struct dma_chan *chan, dma_addr_t dest, dma_addr_t src,
*/
src_width = dst_width = atc_get_xfer_width(src, dest, len);
- ctrla = ATC_SRC_WIDTH(src_width) |
- ATC_DST_WIDTH(dst_width);
+ ctrla = FIELD_PREP(ATC_SRC_WIDTH, src_width) |
+ FIELD_PREP(ATC_DST_WIDTH, dst_width);
- for (offset = 0; offset < len; offset += xfer_count << src_width) {
- xfer_count = min_t(size_t, (len - offset) >> src_width,
- ATC_BTSIZE_MAX);
+ for (offset = 0, i = 0; offset < len;
+ offset += xfer_count << src_width, i++) {
+ struct atdma_sg *atdma_sg = &desc->sg[i];
+ struct at_lli *lli;
- desc = atc_desc_get(atchan);
- if (!desc)
+ atdma_sg->lli = dma_pool_alloc(atdma->lli_pool, GFP_NOWAIT,
+ &atdma_sg->lli_phys);
+ if (!atdma_sg->lli)
goto err_desc_get;
+ lli = atdma_sg->lli;
- desc->lli.saddr = src + offset;
- desc->lli.daddr = dest + offset;
- desc->lli.ctrla = ctrla | xfer_count;
- desc->lli.ctrlb = ctrlb;
+ xfer_count = min_t(size_t, (len - offset) >> src_width,
+ ATC_BTSIZE_MAX);
- desc->txd.cookie = 0;
- desc->len = xfer_count << src_width;
+ lli->saddr = src + offset;
+ lli->daddr = dest + offset;
+ lli->ctrla = ctrla | xfer_count;
+ lli->ctrlb = ctrlb;
- atc_desc_chain(&first, &prev, desc);
+ desc->sg[i].len = xfer_count << src_width;
+
+ atdma_lli_chain(desc, i);
}
- /* First descriptor of the chain embedds additional information */
- first->txd.cookie = -EBUSY;
- first->total_len = len;
+ desc->total_len = len;
/* set end-of-link to the last link descriptor of list*/
- set_desc_eol(desc);
+ set_lli_eol(desc, i - 1);
- first->txd.flags = flags; /* client is in control of this ack */
-
- return &first->txd;
+ return vchan_tx_prep(&atchan->vc, &desc->vd, flags);
err_desc_get:
- atc_desc_put(atchan, first);
+ atdma_desc_free(&desc->vd);
return NULL;
}
-static struct at_desc *atc_create_memset_desc(struct dma_chan *chan,
- dma_addr_t psrc,
- dma_addr_t pdst,
- size_t len)
+static int atdma_create_memset_lli(struct dma_chan *chan,
+ struct atdma_sg *atdma_sg,
+ dma_addr_t psrc, dma_addr_t pdst, size_t len)
{
- struct at_dma_chan *atchan = to_at_dma_chan(chan);
- struct at_desc *desc;
+ struct at_dma *atdma = to_at_dma(chan->device);
+ struct at_lli *lli;
size_t xfer_count;
-
- u32 ctrla = ATC_SRC_WIDTH(2) | ATC_DST_WIDTH(2);
+ u32 ctrla = FIELD_PREP(ATC_SRC_WIDTH, 2) | FIELD_PREP(ATC_DST_WIDTH, 2);
u32 ctrlb = ATC_DEFAULT_CTRLB | ATC_IEN |
- ATC_SRC_ADDR_MODE_FIXED |
- ATC_DST_ADDR_MODE_INCR |
- ATC_FC_MEM2MEM;
+ FIELD_PREP(ATC_SRC_ADDR_MODE, ATC_SRC_ADDR_MODE_FIXED) |
+ FIELD_PREP(ATC_DST_ADDR_MODE, ATC_DST_ADDR_MODE_INCR) |
+ FIELD_PREP(ATC_FC, ATC_FC_MEM2MEM);
xfer_count = len >> 2;
if (xfer_count > ATC_BTSIZE_MAX) {
- dev_err(chan2dev(chan), "%s: buffer is too big\n",
- __func__);
- return NULL;
+ dev_err(chan2dev(chan), "%s: buffer is too big\n", __func__);
+ return -EINVAL;
}
- desc = atc_desc_get(atchan);
- if (!desc) {
- dev_err(chan2dev(chan), "%s: can't get a descriptor\n",
- __func__);
- return NULL;
- }
+ atdma_sg->lli = dma_pool_alloc(atdma->lli_pool, GFP_NOWAIT,
+ &atdma_sg->lli_phys);
+ if (!atdma_sg->lli)
+ return -ENOMEM;
+ lli = atdma_sg->lli;
- desc->lli.saddr = psrc;
- desc->lli.daddr = pdst;
- desc->lli.ctrla = ctrla | xfer_count;
- desc->lli.ctrlb = ctrlb;
+ lli->saddr = psrc;
+ lli->daddr = pdst;
+ lli->ctrla = ctrla | xfer_count;
+ lli->ctrlb = ctrlb;
- desc->txd.cookie = 0;
- desc->len = len;
+ atdma_sg->len = len;
- return desc;
+ return 0;
}
/**
@@ -901,11 +1081,13 @@ static struct dma_async_tx_descriptor *
atc_prep_dma_memset(struct dma_chan *chan, dma_addr_t dest, int value,
size_t len, unsigned long flags)
{
+ struct at_dma_chan *atchan = to_at_dma_chan(chan);
struct at_dma *atdma = to_at_dma(chan->device);
struct at_desc *desc;
void __iomem *vaddr;
dma_addr_t paddr;
char fill_pattern;
+ int ret;
dev_vdbg(chan2dev(chan), "%s: d%pad v0x%x l0x%zx f0x%lx\n", __func__,
&dest, value, len, flags);
@@ -936,27 +1118,28 @@ atc_prep_dma_memset(struct dma_chan *chan, dma_addr_t dest, int value,
(fill_pattern << 8) |
fill_pattern;
- desc = atc_create_memset_desc(chan, paddr, dest, len);
- if (!desc) {
- dev_err(chan2dev(chan), "%s: couldn't get a descriptor\n",
- __func__);
+ desc = kzalloc(struct_size(desc, sg, 1), GFP_ATOMIC);
+ if (!desc)
goto err_free_buffer;
- }
+ desc->sglen = 1;
+
+ ret = atdma_create_memset_lli(chan, desc->sg, paddr, dest, len);
+ if (ret)
+ goto err_free_desc;
desc->memset_paddr = paddr;
desc->memset_vaddr = vaddr;
desc->memset_buffer = true;
- desc->txd.cookie = -EBUSY;
desc->total_len = len;
/* set end-of-link on the descriptor */
- set_desc_eol(desc);
+ set_lli_eol(desc, 0);
- desc->txd.flags = flags;
+ return vchan_tx_prep(&atchan->vc, &desc->vd, flags);
- return &desc->txd;
-
+err_free_desc:
+ kfree(desc);
err_free_buffer:
dma_pool_free(atdma->memset_pool, vaddr, paddr);
return NULL;
@@ -970,12 +1153,13 @@ atc_prep_dma_memset_sg(struct dma_chan *chan,
{
struct at_dma_chan *atchan = to_at_dma_chan(chan);
struct at_dma *atdma = to_at_dma(chan->device);
- struct at_desc *desc = NULL, *first = NULL, *prev = NULL;
+ struct at_desc *desc;
struct scatterlist *sg;
void __iomem *vaddr;
dma_addr_t paddr;
size_t total_len = 0;
int i;
+ int ret;
dev_vdbg(chan2dev(chan), "%s: v0x%x l0x%zx f0x%lx\n", __func__,
value, sg_len, flags);
@@ -994,6 +1178,11 @@ atc_prep_dma_memset_sg(struct dma_chan *chan,
}
*(u32*)vaddr = value;
+ desc = kzalloc(struct_size(desc, sg, sg_len), GFP_ATOMIC);
+ if (!desc)
+ goto err_free_dma_buf;
+ desc->sglen = sg_len;
+
for_each_sg(sgl, sg, sg_len, i) {
dma_addr_t dest = sg_dma_address(sg);
size_t len = sg_dma_len(sg);
@@ -1004,38 +1193,33 @@ atc_prep_dma_memset_sg(struct dma_chan *chan,
if (!is_dma_fill_aligned(chan->device, dest, 0, len)) {
dev_err(chan2dev(chan), "%s: buffer is not aligned\n",
__func__);
- goto err_put_desc;
+ goto err_free_desc;
}
- desc = atc_create_memset_desc(chan, paddr, dest, len);
- if (!desc)
- goto err_put_desc;
+ ret = atdma_create_memset_lli(chan, &desc->sg[i], paddr, dest,
+ len);
+ if (ret)
+ goto err_free_desc;
- atc_desc_chain(&first, &prev, desc);
-
+ atdma_lli_chain(desc, i);
total_len += len;
}
- /*
- * Only set the buffer pointers on the last descriptor to
- * avoid free'ing while we have our transfer still going
- */
desc->memset_paddr = paddr;
desc->memset_vaddr = vaddr;
desc->memset_buffer = true;
- first->txd.cookie = -EBUSY;
- first->total_len = total_len;
+ desc->total_len = total_len;
/* set end-of-link on the descriptor */
- set_desc_eol(desc);
+ set_lli_eol(desc, i - 1);
- first->txd.flags = flags;
+ return vchan_tx_prep(&atchan->vc, &desc->vd, flags);
- return &first->txd;
-
-err_put_desc:
- atc_desc_put(atchan, first);
+err_free_desc:
+ atdma_desc_free(&desc->vd);
+err_free_dma_buf:
+ dma_pool_free(atdma->memset_pool, vaddr, paddr);
return NULL;
}
@@ -1053,11 +1237,11 @@ atc_prep_slave_sg(struct dma_chan *chan, struct scatterlist *sgl,
unsigned int sg_len, enum dma_transfer_direction direction,
unsigned long flags, void *context)
{
+ struct at_dma *atdma = to_at_dma(chan->device);
struct at_dma_chan *atchan = to_at_dma_chan(chan);
struct at_dma_slave *atslave = chan->private;
struct dma_slave_config *sconfig = &atchan->dma_sconfig;
- struct at_desc *first = NULL;
- struct at_desc *prev = NULL;
+ struct at_desc *desc;
u32 ctrla;
u32 ctrlb;
dma_addr_t reg;
@@ -1077,27 +1261,38 @@ atc_prep_slave_sg(struct dma_chan *chan, struct scatterlist *sgl,
return NULL;
}
- ctrla = ATC_SCSIZE(sconfig->src_maxburst)
- | ATC_DCSIZE(sconfig->dst_maxburst);
+ desc = kzalloc(struct_size(desc, sg, sg_len), GFP_ATOMIC);
+ if (!desc)
+ return NULL;
+ desc->sglen = sg_len;
+
+ ctrla = FIELD_PREP(ATC_SCSIZE, sconfig->src_maxburst) |
+ FIELD_PREP(ATC_DCSIZE, sconfig->dst_maxburst);
ctrlb = ATC_IEN;
switch (direction) {
case DMA_MEM_TO_DEV:
reg_width = convert_buswidth(sconfig->dst_addr_width);
- ctrla |= ATC_DST_WIDTH(reg_width);
- ctrlb |= ATC_DST_ADDR_MODE_FIXED
- | ATC_SRC_ADDR_MODE_INCR
- | ATC_FC_MEM2PER
- | ATC_SIF(atchan->mem_if) | ATC_DIF(atchan->per_if);
+ ctrla |= FIELD_PREP(ATC_DST_WIDTH, reg_width);
+ ctrlb |= FIELD_PREP(ATC_DST_ADDR_MODE,
+ ATC_DST_ADDR_MODE_FIXED) |
+ FIELD_PREP(ATC_SRC_ADDR_MODE, ATC_SRC_ADDR_MODE_INCR) |
+ FIELD_PREP(ATC_FC, ATC_FC_MEM2PER) |
+ FIELD_PREP(ATC_SIF, atchan->mem_if) |
+ FIELD_PREP(ATC_DIF, atchan->per_if);
reg = sconfig->dst_addr;
for_each_sg(sgl, sg, sg_len, i) {
- struct at_desc *desc;
+ struct atdma_sg *atdma_sg = &desc->sg[i];
+ struct at_lli *lli;
u32 len;
u32 mem;
- desc = atc_desc_get(atchan);
- if (!desc)
+ atdma_sg->lli = dma_pool_alloc(atdma->lli_pool,
+ GFP_NOWAIT,
+ &atdma_sg->lli_phys);
+ if (!atdma_sg->lli)
goto err_desc_get;
+ lli = atdma_sg->lli;
mem = sg_dma_address(sg);
len = sg_dma_len(sg);
@@ -1110,35 +1305,43 @@ atc_prep_slave_sg(struct dma_chan *chan, struct scatterlist *sgl,
if (unlikely(mem & 3 || len & 3))
mem_width = 0;
- desc->lli.saddr = mem;
- desc->lli.daddr = reg;
- desc->lli.ctrla = ctrla
- | ATC_SRC_WIDTH(mem_width)
- | len >> mem_width;
- desc->lli.ctrlb = ctrlb;
- desc->len = len;
+ lli->saddr = mem;
+ lli->daddr = reg;
+ lli->ctrla = ctrla |
+ FIELD_PREP(ATC_SRC_WIDTH, mem_width) |
+ len >> mem_width;
+ lli->ctrlb = ctrlb;
- atc_desc_chain(&first, &prev, desc);
+ atdma_sg->len = len;
total_len += len;
+
+ desc->sg[i].len = len;
+ atdma_lli_chain(desc, i);
}
break;
case DMA_DEV_TO_MEM:
reg_width = convert_buswidth(sconfig->src_addr_width);
- ctrla |= ATC_SRC_WIDTH(reg_width);
- ctrlb |= ATC_DST_ADDR_MODE_INCR
- | ATC_SRC_ADDR_MODE_FIXED
- | ATC_FC_PER2MEM
- | ATC_SIF(atchan->per_if) | ATC_DIF(atchan->mem_if);
+ ctrla |= FIELD_PREP(ATC_SRC_WIDTH, reg_width);
+ ctrlb |= FIELD_PREP(ATC_DST_ADDR_MODE, ATC_DST_ADDR_MODE_INCR) |
+ FIELD_PREP(ATC_SRC_ADDR_MODE,
+ ATC_SRC_ADDR_MODE_FIXED) |
+ FIELD_PREP(ATC_FC, ATC_FC_PER2MEM) |
+ FIELD_PREP(ATC_SIF, atchan->per_if) |
+ FIELD_PREP(ATC_DIF, atchan->mem_if);
reg = sconfig->src_addr;
for_each_sg(sgl, sg, sg_len, i) {
- struct at_desc *desc;
+ struct atdma_sg *atdma_sg = &desc->sg[i];
+ struct at_lli *lli;
u32 len;
u32 mem;
- desc = atc_desc_get(atchan);
- if (!desc)
+ atdma_sg->lli = dma_pool_alloc(atdma->lli_pool,
+ GFP_NOWAIT,
+ &atdma_sg->lli_phys);
+ if (!atdma_sg->lli)
goto err_desc_get;
+ lli = atdma_sg->lli;
mem = sg_dma_address(sg);
len = sg_dma_len(sg);
@@ -1151,16 +1354,17 @@ atc_prep_slave_sg(struct dma_chan *chan, struct scatterlist *sgl,
if (unlikely(mem & 3 || len & 3))
mem_width = 0;
- desc->lli.saddr = reg;
- desc->lli.daddr = mem;
- desc->lli.ctrla = ctrla
- | ATC_DST_WIDTH(mem_width)
- | len >> reg_width;
- desc->lli.ctrlb = ctrlb;
- desc->len = len;
+ lli->saddr = reg;
+ lli->daddr = mem;
+ lli->ctrla = ctrla |
+ FIELD_PREP(ATC_DST_WIDTH, mem_width) |
+ len >> reg_width;
+ lli->ctrlb = ctrlb;
- atc_desc_chain(&first, &prev, desc);
+ desc->sg[i].len = len;
total_len += len;
+
+ atdma_lli_chain(desc, i);
}
break;
default:
@@ -1168,21 +1372,16 @@ atc_prep_slave_sg(struct dma_chan *chan, struct scatterlist *sgl,
}
/* set end-of-link to the last link descriptor of list*/
- set_desc_eol(prev);
+ set_lli_eol(desc, i - 1);
- /* First descriptor of the chain embedds additional information */
- first->txd.cookie = -EBUSY;
- first->total_len = total_len;
+ desc->total_len = total_len;
- /* first link descriptor of list is responsible of flags */
- first->txd.flags = flags; /* client is in control of this ack */
-
- return &first->txd;
+ return vchan_tx_prep(&atchan->vc, &desc->vd, flags);
err_desc_get:
dev_err(chan2dev(chan), "not enough descriptors available\n");
err:
- atc_desc_put(atchan, first);
+ atdma_desc_free(&desc->vd);
return NULL;
}
@@ -1212,50 +1411,59 @@ atc_dma_cyclic_check_values(unsigned int reg_width, dma_addr_t buf_addr,
*/
static int
atc_dma_cyclic_fill_desc(struct dma_chan *chan, struct at_desc *desc,
- unsigned int period_index, dma_addr_t buf_addr,
+ unsigned int i, dma_addr_t buf_addr,
unsigned int reg_width, size_t period_len,
enum dma_transfer_direction direction)
{
+ struct at_dma *atdma = to_at_dma(chan->device);
struct at_dma_chan *atchan = to_at_dma_chan(chan);
struct dma_slave_config *sconfig = &atchan->dma_sconfig;
- u32 ctrla;
+ struct atdma_sg *atdma_sg = &desc->sg[i];
+ struct at_lli *lli;
- /* prepare common CRTLA value */
- ctrla = ATC_SCSIZE(sconfig->src_maxburst)
- | ATC_DCSIZE(sconfig->dst_maxburst)
- | ATC_DST_WIDTH(reg_width)
- | ATC_SRC_WIDTH(reg_width)
- | period_len >> reg_width;
+ atdma_sg->lli = dma_pool_alloc(atdma->lli_pool, GFP_ATOMIC,
+ &atdma_sg->lli_phys);
+ if (!atdma_sg->lli)
+ return -ENOMEM;
+ lli = atdma_sg->lli;
switch (direction) {
case DMA_MEM_TO_DEV:
- desc->lli.saddr = buf_addr + (period_len * period_index);
- desc->lli.daddr = sconfig->dst_addr;
- desc->lli.ctrla = ctrla;
- desc->lli.ctrlb = ATC_DST_ADDR_MODE_FIXED
- | ATC_SRC_ADDR_MODE_INCR
- | ATC_FC_MEM2PER
- | ATC_SIF(atchan->mem_if)
- | ATC_DIF(atchan->per_if);
- desc->len = period_len;
+ lli->saddr = buf_addr + (period_len * i);
+ lli->daddr = sconfig->dst_addr;
+ lli->ctrlb = FIELD_PREP(ATC_DST_ADDR_MODE,
+ ATC_DST_ADDR_MODE_FIXED) |
+ FIELD_PREP(ATC_SRC_ADDR_MODE,
+ ATC_SRC_ADDR_MODE_INCR) |
+ FIELD_PREP(ATC_FC, ATC_FC_MEM2PER) |
+ FIELD_PREP(ATC_SIF, atchan->mem_if) |
+ FIELD_PREP(ATC_DIF, atchan->per_if);
+
break;
case DMA_DEV_TO_MEM:
- desc->lli.saddr = sconfig->src_addr;
- desc->lli.daddr = buf_addr + (period_len * period_index);
- desc->lli.ctrla = ctrla;
- desc->lli.ctrlb = ATC_DST_ADDR_MODE_INCR
- | ATC_SRC_ADDR_MODE_FIXED
- | ATC_FC_PER2MEM
- | ATC_SIF(atchan->per_if)
- | ATC_DIF(atchan->mem_if);
- desc->len = period_len;
+ lli->saddr = sconfig->src_addr;
+ lli->daddr = buf_addr + (period_len * i);
+ lli->ctrlb = FIELD_PREP(ATC_DST_ADDR_MODE,
+ ATC_DST_ADDR_MODE_INCR) |
+ FIELD_PREP(ATC_SRC_ADDR_MODE,
+ ATC_SRC_ADDR_MODE_FIXED) |
+ FIELD_PREP(ATC_FC, ATC_FC_PER2MEM) |
+ FIELD_PREP(ATC_SIF, atchan->per_if) |
+ FIELD_PREP(ATC_DIF, atchan->mem_if);
break;
default:
return -EINVAL;
}
+ lli->ctrla = FIELD_PREP(ATC_SCSIZE, sconfig->src_maxburst) |
+ FIELD_PREP(ATC_DCSIZE, sconfig->dst_maxburst) |
+ FIELD_PREP(ATC_DST_WIDTH, reg_width) |
+ FIELD_PREP(ATC_SRC_WIDTH, reg_width) |
+ period_len >> reg_width;
+ desc->sg[i].len = period_len;
+
return 0;
}
@@ -1276,8 +1484,7 @@ atc_prep_dma_cyclic(struct dma_chan *chan, dma_addr_t buf_addr, size_t buf_len,
struct at_dma_chan *atchan = to_at_dma_chan(chan);
struct at_dma_slave *atslave = chan->private;
struct dma_slave_config *sconfig = &atchan->dma_sconfig;
- struct at_desc *first = NULL;
- struct at_desc *prev = NULL;
+ struct at_desc *desc;
unsigned long was_cyclic;
unsigned int reg_width;
unsigned int periods = buf_len / period_len;
@@ -1311,33 +1518,26 @@ atc_prep_dma_cyclic(struct dma_chan *chan, dma_addr_t buf_addr, size_t buf_len,
if (atc_dma_cyclic_check_values(reg_width, buf_addr, period_len))
goto err_out;
+ desc = kzalloc(struct_size(desc, sg, periods), GFP_ATOMIC);
+ if (!desc)
+ goto err_out;
+ desc->sglen = periods;
+
/* build cyclic linked list */
for (i = 0; i < periods; i++) {
- struct at_desc *desc;
-
- desc = atc_desc_get(atchan);
- if (!desc)
- goto err_desc_get;
-
if (atc_dma_cyclic_fill_desc(chan, desc, i, buf_addr,
reg_width, period_len, direction))
- goto err_desc_get;
-
- atc_desc_chain(&first, &prev, desc);
+ goto err_fill_desc;
+ atdma_lli_chain(desc, i);
}
-
+ desc->total_len = buf_len;
/* lets make a cyclic list */
- prev->lli.dscr = first->txd.phys;
+ desc->sg[i - 1].lli->dscr = desc->sg[0].lli_phys;
- /* First descriptor of the chain embedds additional information */
- first->txd.cookie = -EBUSY;
- first->total_len = buf_len;
+ return vchan_tx_prep(&atchan->vc, &desc->vd, flags);
- return &first->txd;
-
-err_desc_get:
- dev_err(chan2dev(chan), "not enough descriptors available\n");
- atc_desc_put(atchan, first);
+err_fill_desc:
+ atdma_desc_free(&desc->vd);
err_out:
clear_bit(ATC_IS_CYCLIC, &atchan->status);
return NULL;
@@ -1366,17 +1566,17 @@ static int atc_pause(struct dma_chan *chan)
{
struct at_dma_chan *atchan = to_at_dma_chan(chan);
struct at_dma *atdma = to_at_dma(chan->device);
- int chan_id = atchan->chan_common.chan_id;
+ int chan_id = atchan->vc.chan.chan_id;
unsigned long flags;
dev_vdbg(chan2dev(chan), "%s\n", __func__);
- spin_lock_irqsave(&atchan->lock, flags);
+ spin_lock_irqsave(&atchan->vc.lock, flags);
dma_writel(atdma, CHER, AT_DMA_SUSP(chan_id));
set_bit(ATC_IS_PAUSED, &atchan->status);
- spin_unlock_irqrestore(&atchan->lock, flags);
+ spin_unlock_irqrestore(&atchan->vc.lock, flags);
return 0;
}
@@ -1385,7 +1585,7 @@ static int atc_resume(struct dma_chan *chan)
{
struct at_dma_chan *atchan = to_at_dma_chan(chan);
struct at_dma *atdma = to_at_dma(chan->device);
- int chan_id = atchan->chan_common.chan_id;
+ int chan_id = atchan->vc.chan.chan_id;
unsigned long flags;
dev_vdbg(chan2dev(chan), "%s\n", __func__);
@@ -1393,12 +1593,12 @@ static int atc_resume(struct dma_chan *chan)
if (!atc_chan_is_paused(atchan))
return 0;
- spin_lock_irqsave(&atchan->lock, flags);
+ spin_lock_irqsave(&atchan->vc.lock, flags);
dma_writel(atdma, CHDR, AT_DMA_RES(chan_id));
clear_bit(ATC_IS_PAUSED, &atchan->status);
- spin_unlock_irqrestore(&atchan->lock, flags);
+ spin_unlock_irqrestore(&atchan->vc.lock, flags);
return 0;
}
@@ -1407,9 +1607,11 @@ static int atc_terminate_all(struct dma_chan *chan)
{
struct at_dma_chan *atchan = to_at_dma_chan(chan);
struct at_dma *atdma = to_at_dma(chan->device);
- int chan_id = atchan->chan_common.chan_id;
+ int chan_id = atchan->vc.chan.chan_id;
unsigned long flags;
+ LIST_HEAD(list);
+
dev_vdbg(chan2dev(chan), "%s\n", __func__);
/*
@@ -1418,7 +1620,7 @@ static int atc_terminate_all(struct dma_chan *chan)
* channel. We still have to poll the channel enable bit due
* to AHB/HSB limitations.
*/
- spin_lock_irqsave(&atchan->lock, flags);
+ spin_lock_irqsave(&atchan->vc.lock, flags);
/* disabling channel: must also remove suspend state */
dma_writel(atdma, CHDR, AT_DMA_RES(chan_id) | atchan->mask);
@@ -1427,15 +1629,20 @@ static int atc_terminate_all(struct dma_chan *chan)
while (dma_readl(atdma, CHSR) & atchan->mask)
cpu_relax();
- /* active_list entries will end up before queued entries */
- list_splice_tail_init(&atchan->queue, &atchan->free_list);
- list_splice_tail_init(&atchan->active_list, &atchan->free_list);
+ if (atchan->desc) {
+ vchan_terminate_vdesc(&atchan->desc->vd);
+ atchan->desc = NULL;
+ }
+
+ vchan_get_all_descriptors(&atchan->vc, &list);
clear_bit(ATC_IS_PAUSED, &atchan->status);
/* if channel dedicated to cyclic operations, free it */
clear_bit(ATC_IS_CYCLIC, &atchan->status);
- spin_unlock_irqrestore(&atchan->lock, flags);
+ spin_unlock_irqrestore(&atchan->vc.lock, flags);
+
+ vchan_dma_desc_free_list(&atchan->vc, &list);
return 0;
}
@@ -1457,60 +1664,43 @@ atc_tx_status(struct dma_chan *chan,
{
struct at_dma_chan *atchan = to_at_dma_chan(chan);
unsigned long flags;
- enum dma_status ret;
- int bytes = 0;
+ enum dma_status dma_status;
+ u32 residue;
+ int ret;
- ret = dma_cookie_status(chan, cookie, txstate);
- if (ret == DMA_COMPLETE)
- return ret;
- /*
- * There's no point calculating the residue if there's
- * no txstate to store the value.
- */
- if (!txstate)
- return DMA_ERROR;
+ dma_status = dma_cookie_status(chan, cookie, txstate);
+ if (dma_status == DMA_COMPLETE || !txstate)
+ return dma_status;
- spin_lock_irqsave(&atchan->lock, flags);
-
+ spin_lock_irqsave(&atchan->vc.lock, flags);
/* Get number of bytes left in the active transactions */
- bytes = atc_get_bytes_left(chan, cookie);
+ ret = atc_get_residue(chan, cookie, &residue);
+ spin_unlock_irqrestore(&atchan->vc.lock, flags);
- spin_unlock_irqrestore(&atchan->lock, flags);
-
- if (unlikely(bytes < 0)) {
+ if (unlikely(ret < 0)) {
dev_vdbg(chan2dev(chan), "get residual bytes error\n");
return DMA_ERROR;
} else {
- dma_set_residue(txstate, bytes);
+ dma_set_residue(txstate, residue);
}
- dev_vdbg(chan2dev(chan), "tx_status %d: cookie = %d residue = %d\n",
- ret, cookie, bytes);
+ dev_vdbg(chan2dev(chan), "tx_status %d: cookie = %d residue = %u\n",
+ dma_status, cookie, residue);
- return ret;
+ return dma_status;
}
-/**
- * atc_issue_pending - takes the first transaction descriptor in the pending
- * queue and starts the transfer.
- * @chan: target DMA channel
- */
static void atc_issue_pending(struct dma_chan *chan)
{
struct at_dma_chan *atchan = to_at_dma_chan(chan);
- struct at_desc *desc;
unsigned long flags;
- dev_vdbg(chan2dev(chan), "issue_pending\n");
-
- spin_lock_irqsave(&atchan->lock, flags);
- if (atc_chan_is_enabled(atchan) || list_empty(&atchan->queue))
- return spin_unlock_irqrestore(&atchan->lock, flags);
-
- desc = atc_first_queued(atchan);
- list_move_tail(&desc->desc_node, &atchan->active_list);
- atc_dostart(atchan, desc);
- spin_unlock_irqrestore(&atchan->lock, flags);
+ spin_lock_irqsave(&atchan->vc.lock, flags);
+ if (vchan_issue_pending(&atchan->vc) && !atchan->desc) {
+ if (!(atc_chan_is_enabled(atchan)))
+ atc_dostart(atchan);
+ }
+ spin_unlock_irqrestore(&atchan->vc.lock, flags);
}
/**
@@ -1523,9 +1713,7 @@ static int atc_alloc_chan_resources(struct dma_chan *chan)
{
struct at_dma_chan *atchan = to_at_dma_chan(chan);
struct at_dma *atdma = to_at_dma(chan->device);
- struct at_desc *desc;
struct at_dma_slave *atslave;
- int i;
u32 cfg;
dev_vdbg(chan2dev(chan), "alloc_chan_resources\n");
@@ -1536,11 +1724,6 @@ static int atc_alloc_chan_resources(struct dma_chan *chan)
return -EIO;
}
- if (!list_empty(&atchan->free_list)) {
- dev_dbg(chan2dev(chan), "can't allocate channel resources (channel not freed from a previous use)\n");
- return -EIO;
- }
-
cfg = ATC_DEFAULT_CFG;
atslave = chan->private;
@@ -1549,33 +1732,17 @@ static int atc_alloc_chan_resources(struct dma_chan *chan)
* We need controller-specific data to set up slave
* transfers.
*/
- BUG_ON(!atslave->dma_dev || atslave->dma_dev != atdma->dma_common.dev);
+ BUG_ON(!atslave->dma_dev || atslave->dma_dev != atdma->dma_device.dev);
/* if cfg configuration specified take it instead of default */
if (atslave->cfg)
cfg = atslave->cfg;
}
- /* Allocate initial pool of descriptors */
- for (i = 0; i < init_nr_desc_per_channel; i++) {
- desc = atc_alloc_descriptor(chan, GFP_KERNEL);
- if (!desc) {
- dev_err(atdma->dma_common.dev,
- "Only %d initial descriptors\n", i);
- break;
- }
- list_add_tail(&desc->desc_node, &atchan->free_list);
- }
-
- dma_cookie_init(chan);
-
/* channel parameters */
channel_writel(atchan, CFG, cfg);
- dev_dbg(chan2dev(chan),
- "alloc_chan_resources: allocated %d descriptors\n", i);
-
- return i;
+ return 0;
}
/**
@@ -1585,22 +1752,10 @@ static int atc_alloc_chan_resources(struct dma_chan *chan)
static void atc_free_chan_resources(struct dma_chan *chan)
{
struct at_dma_chan *atchan = to_at_dma_chan(chan);
- struct at_dma *atdma = to_at_dma(chan->device);
- struct at_desc *desc, *_desc;
- LIST_HEAD(list);
- /* ASSERT: channel is idle */
- BUG_ON(!list_empty(&atchan->active_list));
- BUG_ON(!list_empty(&atchan->queue));
BUG_ON(atc_chan_is_enabled(atchan));
- list_for_each_entry_safe(desc, _desc, &atchan->free_list, desc_node) {
- dev_vdbg(chan2dev(chan), " freeing descriptor %p\n", desc);
- list_del(&desc->desc_node);
- /* free link descriptor */
- dma_pool_free(atdma->dma_desc_pool, desc, desc->txd.phys);
- }
- list_splice_init(&atchan->free_list, &list);
+ vchan_free_chan_resources(to_virt_chan(chan));
atchan->status = 0;
/*
@@ -1651,14 +1806,13 @@ static struct dma_chan *at_dma_xlate(struct of_phandle_args *dma_spec,
return NULL;
}
- atslave->cfg = ATC_DST_H2SEL_HW | ATC_SRC_H2SEL_HW;
+ atslave->cfg = ATC_DST_H2SEL | ATC_SRC_H2SEL;
/*
* We can fill both SRC_PER and DST_PER, one of these fields will be
* ignored depending on DMA transfer direction.
*/
per_id = dma_spec->args[1] & AT91_DMA_CFG_PER_ID_MASK;
- atslave->cfg |= ATC_DST_PER_MSB(per_id) | ATC_DST_PER(per_id)
- | ATC_SRC_PER_MSB(per_id) | ATC_SRC_PER(per_id);
+ atslave->cfg |= ATC_DST_PER_ID(per_id) | ATC_SRC_PER_ID(per_id);
/*
* We have to translate the value we get from the device tree since
* the half FIFO configuration value had to be 0 to keep backward
@@ -1666,14 +1820,16 @@ static struct dma_chan *at_dma_xlate(struct of_phandle_args *dma_spec,
*/
switch (dma_spec->args[1] & AT91_DMA_CFG_FIFOCFG_MASK) {
case AT91_DMA_CFG_FIFOCFG_ALAP:
- atslave->cfg |= ATC_FIFOCFG_LARGESTBURST;
+ atslave->cfg |= FIELD_PREP(ATC_FIFOCFG,
+ ATC_FIFOCFG_LARGESTBURST);
break;
case AT91_DMA_CFG_FIFOCFG_ASAP:
- atslave->cfg |= ATC_FIFOCFG_ENOUGHSPACE;
+ atslave->cfg |= FIELD_PREP(ATC_FIFOCFG,
+ ATC_FIFOCFG_ENOUGHSPACE);
break;
case AT91_DMA_CFG_FIFOCFG_HALF:
default:
- atslave->cfg |= ATC_FIFOCFG_HALFFIFO;
+ atslave->cfg |= FIELD_PREP(ATC_FIFOCFG, ATC_FIFOCFG_HALFFIFO);
}
atslave->dma_dev = &dmac_pdev->dev;
@@ -1768,9 +1924,7 @@ static void at_dma_off(struct at_dma *atdma)
static int __init at_dma_probe(struct platform_device *pdev)
{
- struct resource *io;
struct at_dma *atdma;
- size_t size;
int irq;
int err;
int i;
@@ -1790,44 +1944,31 @@ static int __init at_dma_probe(struct platform_device *pdev)
if (!plat_dat)
return -ENODEV;
- io = platform_get_resource(pdev, IORESOURCE_MEM, 0);
- if (!io)
- return -EINVAL;
+ atdma = devm_kzalloc(&pdev->dev,
+ struct_size(atdma, chan, plat_dat->nr_channels),
+ GFP_KERNEL);
+ if (!atdma)
+ return -ENOMEM;
+
+ atdma->regs = devm_platform_ioremap_resource(pdev, 0);
+ if (IS_ERR(atdma->regs))
+ return PTR_ERR(atdma->regs);
irq = platform_get_irq(pdev, 0);
if (irq < 0)
return irq;
- size = sizeof(struct at_dma);
- size += plat_dat->nr_channels * sizeof(struct at_dma_chan);
- atdma = kzalloc(size, GFP_KERNEL);
- if (!atdma)
- return -ENOMEM;
-
/* discover transaction capabilities */
- atdma->dma_common.cap_mask = plat_dat->cap_mask;
+ atdma->dma_device.cap_mask = plat_dat->cap_mask;
atdma->all_chan_mask = (1 << plat_dat->nr_channels) - 1;
- size = resource_size(io);
- if (!request_mem_region(io->start, size, pdev->dev.driver->name)) {
- err = -EBUSY;
- goto err_kfree;
- }
+ atdma->clk = devm_clk_get(&pdev->dev, "dma_clk");
+ if (IS_ERR(atdma->clk))
+ return PTR_ERR(atdma->clk);
- atdma->regs = ioremap(io->start, size);
- if (!atdma->regs) {
- err = -ENOMEM;
- goto err_release_r;
- }
-
- atdma->clk = clk_get(&pdev->dev, "dma_clk");
- if (IS_ERR(atdma->clk)) {
- err = PTR_ERR(atdma->clk);
- goto err_clk;
- }
err = clk_prepare_enable(atdma->clk);
if (err)
- goto err_clk_prepare;
+ return err;
/* force dma off, just in case */
at_dma_off(atdma);
@@ -1839,11 +1980,11 @@ static int __init at_dma_probe(struct platform_device *pdev)
platform_set_drvdata(pdev, atdma);
/* create a pool of consistent memory blocks for hardware descriptors */
- atdma->dma_desc_pool = dma_pool_create("at_hdmac_desc_pool",
- &pdev->dev, sizeof(struct at_desc),
- 4 /* word alignment */, 0);
- if (!atdma->dma_desc_pool) {
- dev_err(&pdev->dev, "No memory for descriptors dma pool\n");
+ atdma->lli_pool = dma_pool_create("at_hdmac_lli_pool",
+ &pdev->dev, sizeof(struct at_lli),
+ 4 /* word alignment */, 0);
+ if (!atdma->lli_pool) {
+ dev_err(&pdev->dev, "Unable to allocate DMA LLI descriptor pool\n");
err = -ENOMEM;
goto err_desc_pool_create;
}
@@ -1862,73 +2003,66 @@ static int __init at_dma_probe(struct platform_device *pdev)
cpu_relax();
/* initialize channels related values */
- INIT_LIST_HEAD(&atdma->dma_common.channels);
+ INIT_LIST_HEAD(&atdma->dma_device.channels);
for (i = 0; i < plat_dat->nr_channels; i++) {
struct at_dma_chan *atchan = &atdma->chan[i];
atchan->mem_if = AT_DMA_MEM_IF;
atchan->per_if = AT_DMA_PER_IF;
- atchan->chan_common.device = &atdma->dma_common;
- dma_cookie_init(&atchan->chan_common);
- list_add_tail(&atchan->chan_common.device_node,
- &atdma->dma_common.channels);
atchan->ch_regs = atdma->regs + ch_regs(i);
- spin_lock_init(&atchan->lock);
atchan->mask = 1 << i;
- INIT_LIST_HEAD(&atchan->active_list);
- INIT_LIST_HEAD(&atchan->queue);
- INIT_LIST_HEAD(&atchan->free_list);
-
- tasklet_setup(&atchan->tasklet, atc_tasklet);
+ atchan->atdma = atdma;
+ atchan->vc.desc_free = atdma_desc_free;
+ vchan_init(&atchan->vc, &atdma->dma_device);
atc_enable_chan_irq(atdma, i);
}
/* set base routines */
- atdma->dma_common.device_alloc_chan_resources = atc_alloc_chan_resources;
- atdma->dma_common.device_free_chan_resources = atc_free_chan_resources;
- atdma->dma_common.device_tx_status = atc_tx_status;
- atdma->dma_common.device_issue_pending = atc_issue_pending;
- atdma->dma_common.dev = &pdev->dev;
+ atdma->dma_device.device_alloc_chan_resources = atc_alloc_chan_resources;
+ atdma->dma_device.device_free_chan_resources = atc_free_chan_resources;
+ atdma->dma_device.device_tx_status = atc_tx_status;
+ atdma->dma_device.device_issue_pending = atc_issue_pending;
+ atdma->dma_device.dev = &pdev->dev;
/* set prep routines based on capability */
- if (dma_has_cap(DMA_INTERLEAVE, atdma->dma_common.cap_mask))
- atdma->dma_common.device_prep_interleaved_dma = atc_prep_dma_interleaved;
+ if (dma_has_cap(DMA_INTERLEAVE, atdma->dma_device.cap_mask))
+ atdma->dma_device.device_prep_interleaved_dma = atc_prep_dma_interleaved;
- if (dma_has_cap(DMA_MEMCPY, atdma->dma_common.cap_mask))
- atdma->dma_common.device_prep_dma_memcpy = atc_prep_dma_memcpy;
+ if (dma_has_cap(DMA_MEMCPY, atdma->dma_device.cap_mask))
+ atdma->dma_device.device_prep_dma_memcpy = atc_prep_dma_memcpy;
- if (dma_has_cap(DMA_MEMSET, atdma->dma_common.cap_mask)) {
- atdma->dma_common.device_prep_dma_memset = atc_prep_dma_memset;
- atdma->dma_common.device_prep_dma_memset_sg = atc_prep_dma_memset_sg;
- atdma->dma_common.fill_align = DMAENGINE_ALIGN_4_BYTES;
+ if (dma_has_cap(DMA_MEMSET, atdma->dma_device.cap_mask)) {
+ atdma->dma_device.device_prep_dma_memset = atc_prep_dma_memset;
+ atdma->dma_device.device_prep_dma_memset_sg = atc_prep_dma_memset_sg;
+ atdma->dma_device.fill_align = DMAENGINE_ALIGN_4_BYTES;
}
- if (dma_has_cap(DMA_SLAVE, atdma->dma_common.cap_mask)) {
- atdma->dma_common.device_prep_slave_sg = atc_prep_slave_sg;
+ if (dma_has_cap(DMA_SLAVE, atdma->dma_device.cap_mask)) {
+ atdma->dma_device.device_prep_slave_sg = atc_prep_slave_sg;
/* controller can do slave DMA: can trigger cyclic transfers */
- dma_cap_set(DMA_CYCLIC, atdma->dma_common.cap_mask);
- atdma->dma_common.device_prep_dma_cyclic = atc_prep_dma_cyclic;
- atdma->dma_common.device_config = atc_config;
- atdma->dma_common.device_pause = atc_pause;
- atdma->dma_common.device_resume = atc_resume;
- atdma->dma_common.device_terminate_all = atc_terminate_all;
- atdma->dma_common.src_addr_widths = ATC_DMA_BUSWIDTHS;
- atdma->dma_common.dst_addr_widths = ATC_DMA_BUSWIDTHS;
- atdma->dma_common.directions = BIT(DMA_DEV_TO_MEM) | BIT(DMA_MEM_TO_DEV);
- atdma->dma_common.residue_granularity = DMA_RESIDUE_GRANULARITY_BURST;
+ dma_cap_set(DMA_CYCLIC, atdma->dma_device.cap_mask);
+ atdma->dma_device.device_prep_dma_cyclic = atc_prep_dma_cyclic;
+ atdma->dma_device.device_config = atc_config;
+ atdma->dma_device.device_pause = atc_pause;
+ atdma->dma_device.device_resume = atc_resume;
+ atdma->dma_device.device_terminate_all = atc_terminate_all;
+ atdma->dma_device.src_addr_widths = ATC_DMA_BUSWIDTHS;
+ atdma->dma_device.dst_addr_widths = ATC_DMA_BUSWIDTHS;
+ atdma->dma_device.directions = BIT(DMA_DEV_TO_MEM) | BIT(DMA_MEM_TO_DEV);
+ atdma->dma_device.residue_granularity = DMA_RESIDUE_GRANULARITY_BURST;
}
dma_writel(atdma, EN, AT_DMA_ENABLE);
dev_info(&pdev->dev, "Atmel AHB DMA Controller ( %s%s%s), %d channels\n",
- dma_has_cap(DMA_MEMCPY, atdma->dma_common.cap_mask) ? "cpy " : "",
- dma_has_cap(DMA_MEMSET, atdma->dma_common.cap_mask) ? "set " : "",
- dma_has_cap(DMA_SLAVE, atdma->dma_common.cap_mask) ? "slave " : "",
+ dma_has_cap(DMA_MEMCPY, atdma->dma_device.cap_mask) ? "cpy " : "",
+ dma_has_cap(DMA_MEMSET, atdma->dma_device.cap_mask) ? "set " : "",
+ dma_has_cap(DMA_SLAVE, atdma->dma_device.cap_mask) ? "slave " : "",
plat_dat->nr_channels);
- err = dma_async_device_register(&atdma->dma_common);
+ err = dma_async_device_register(&atdma->dma_device);
if (err) {
dev_err(&pdev->dev, "Unable to register: %d.\n", err);
goto err_dma_async_device_register;
@@ -1951,24 +2085,15 @@ static int __init at_dma_probe(struct platform_device *pdev)
return 0;
err_of_dma_controller_register:
- dma_async_device_unregister(&atdma->dma_common);
+ dma_async_device_unregister(&atdma->dma_device);
err_dma_async_device_register:
dma_pool_destroy(atdma->memset_pool);
err_memset_pool_create:
- dma_pool_destroy(atdma->dma_desc_pool);
+ dma_pool_destroy(atdma->lli_pool);
err_desc_pool_create:
free_irq(platform_get_irq(pdev, 0), atdma);
err_irq:
clk_disable_unprepare(atdma->clk);
-err_clk_prepare:
- clk_put(atdma->clk);
-err_clk:
- iounmap(atdma->regs);
- atdma->regs = NULL;
-err_release_r:
- release_mem_region(io->start, size);
-err_kfree:
- kfree(atdma);
return err;
}
@@ -1976,38 +2101,24 @@ static int at_dma_remove(struct platform_device *pdev)
{
struct at_dma *atdma = platform_get_drvdata(pdev);
struct dma_chan *chan, *_chan;
- struct resource *io;
at_dma_off(atdma);
if (pdev->dev.of_node)
of_dma_controller_free(pdev->dev.of_node);
- dma_async_device_unregister(&atdma->dma_common);
+ dma_async_device_unregister(&atdma->dma_device);
dma_pool_destroy(atdma->memset_pool);
- dma_pool_destroy(atdma->dma_desc_pool);
+ dma_pool_destroy(atdma->lli_pool);
free_irq(platform_get_irq(pdev, 0), atdma);
- list_for_each_entry_safe(chan, _chan, &atdma->dma_common.channels,
+ list_for_each_entry_safe(chan, _chan, &atdma->dma_device.channels,
device_node) {
- struct at_dma_chan *atchan = to_at_dma_chan(chan);
-
/* Disable interrupts */
atc_disable_chan_irq(atdma, chan->chan_id);
-
- tasklet_kill(&atchan->tasklet);
list_del(&chan->device_node);
}
clk_disable_unprepare(atdma->clk);
- clk_put(atdma->clk);
-
- iounmap(atdma->regs);
- atdma->regs = NULL;
-
- io = platform_get_resource(pdev, IORESOURCE_MEM, 0);
- release_mem_region(io->start, resource_size(io));
-
- kfree(atdma);
return 0;
}
@@ -2025,7 +2136,7 @@ static int at_dma_prepare(struct device *dev)
struct at_dma *atdma = dev_get_drvdata(dev);
struct dma_chan *chan, *_chan;
- list_for_each_entry_safe(chan, _chan, &atdma->dma_common.channels,
+ list_for_each_entry_safe(chan, _chan, &atdma->dma_device.channels,
device_node) {
struct at_dma_chan *atchan = to_at_dma_chan(chan);
/* wait for transaction completion (except in cyclic case) */
@@ -2037,7 +2148,7 @@ static int at_dma_prepare(struct device *dev)
static void atc_suspend_cyclic(struct at_dma_chan *atchan)
{
- struct dma_chan *chan = &atchan->chan_common;
+ struct dma_chan *chan = &atchan->vc.chan;
/* Channel should be paused by user
* do it anyway even if it is not done already */
@@ -2060,7 +2171,7 @@ static int at_dma_suspend_noirq(struct device *dev)
struct dma_chan *chan, *_chan;
/* preserve data */
- list_for_each_entry_safe(chan, _chan, &atdma->dma_common.channels,
+ list_for_each_entry_safe(chan, _chan, &atdma->dma_device.channels,
device_node) {
struct at_dma_chan *atchan = to_at_dma_chan(chan);
@@ -2078,7 +2189,7 @@ static int at_dma_suspend_noirq(struct device *dev)
static void atc_resume_cyclic(struct at_dma_chan *atchan)
{
- struct at_dma *atdma = to_at_dma(atchan->chan_common.device);
+ struct at_dma *atdma = to_at_dma(atchan->vc.chan.device);
/* restore channel status for cyclic descriptors list:
* next descriptor in the cyclic list at the time of suspend */
@@ -2110,7 +2221,7 @@ static int at_dma_resume_noirq(struct device *dev)
/* restore saved data */
dma_writel(atdma, EBCIER, atdma->save_imr);
- list_for_each_entry_safe(chan, _chan, &atdma->dma_common.channels,
+ list_for_each_entry_safe(chan, _chan, &atdma->dma_device.channels,
device_node) {
struct at_dma_chan *atchan = to_at_dma_chan(chan);
@@ -2121,7 +2232,7 @@ static int at_dma_resume_noirq(struct device *dev)
return 0;
}
-static const struct dev_pm_ops at_dma_dev_pm_ops = {
+static const struct dev_pm_ops __maybe_unused at_dma_dev_pm_ops = {
.prepare = at_dma_prepare,
.suspend_noirq = at_dma_suspend_noirq,
.resume_noirq = at_dma_resume_noirq,
@@ -2133,7 +2244,7 @@ static struct platform_driver at_dma_driver = {
.id_table = atdma_devtypes,
.driver = {
.name = "at_hdmac",
- .pm = &at_dma_dev_pm_ops,
+ .pm = pm_ptr(&at_dma_dev_pm_ops),
.of_match_table = of_match_ptr(atmel_dma_dt_ids),
},
};
@@ -2152,5 +2263,6 @@ module_exit(at_dma_exit);
MODULE_DESCRIPTION("Atmel AHB DMA Controller driver");
MODULE_AUTHOR("Nicolas Ferre <nicolas.ferre@atmel.com>");
+MODULE_AUTHOR("Tudor Ambarus <tudor.ambarus@microchip.com>");
MODULE_LICENSE("GPL");
MODULE_ALIAS("platform:at_hdmac");
diff --git a/drivers/dma/at_hdmac_regs.h b/drivers/dma/at_hdmac_regs.h
deleted file mode 100644
index d4d382d..0000000
--- a/drivers/dma/at_hdmac_regs.h
+++ /dev/null
@@ -1,478 +0,0 @@
-/* SPDX-License-Identifier: GPL-2.0-or-later */
-/*
- * Header file for the Atmel AHB DMA Controller driver
- *
- * Copyright (C) 2008 Atmel Corporation
- */
-#ifndef AT_HDMAC_REGS_H
-#define AT_HDMAC_REGS_H
-
-#define AT_DMA_MAX_NR_CHANNELS 8
-
-
-#define AT_DMA_GCFG 0x00 /* Global Configuration Register */
-#define AT_DMA_IF_BIGEND(i) (0x1 << (i)) /* AHB-Lite Interface i in Big-endian mode */
-#define AT_DMA_ARB_CFG (0x1 << 4) /* Arbiter mode. */
-#define AT_DMA_ARB_CFG_FIXED (0x0 << 4)
-#define AT_DMA_ARB_CFG_ROUND_ROBIN (0x1 << 4)
-
-#define AT_DMA_EN 0x04 /* Controller Enable Register */
-#define AT_DMA_ENABLE (0x1 << 0)
-
-#define AT_DMA_SREQ 0x08 /* Software Single Request Register */
-#define AT_DMA_SSREQ(x) (0x1 << ((x) << 1)) /* Request a source single transfer on channel x */
-#define AT_DMA_DSREQ(x) (0x1 << (1 + ((x) << 1))) /* Request a destination single transfer on channel x */
-
-#define AT_DMA_CREQ 0x0C /* Software Chunk Transfer Request Register */
-#define AT_DMA_SCREQ(x) (0x1 << ((x) << 1)) /* Request a source chunk transfer on channel x */
-#define AT_DMA_DCREQ(x) (0x1 << (1 + ((x) << 1))) /* Request a destination chunk transfer on channel x */
-
-#define AT_DMA_LAST 0x10 /* Software Last Transfer Flag Register */
-#define AT_DMA_SLAST(x) (0x1 << ((x) << 1)) /* This src rq is last tx of buffer on channel x */
-#define AT_DMA_DLAST(x) (0x1 << (1 + ((x) << 1))) /* This dst rq is last tx of buffer on channel x */
-
-#define AT_DMA_SYNC 0x14 /* Request Synchronization Register */
-#define AT_DMA_SYR(h) (0x1 << (h)) /* Synchronize handshake line h */
-
-/* Error, Chained Buffer transfer completed and Buffer transfer completed Interrupt registers */
-#define AT_DMA_EBCIER 0x18 /* Enable register */
-#define AT_DMA_EBCIDR 0x1C /* Disable register */
-#define AT_DMA_EBCIMR 0x20 /* Mask Register */
-#define AT_DMA_EBCISR 0x24 /* Status Register */
-#define AT_DMA_CBTC_OFFSET 8
-#define AT_DMA_ERR_OFFSET 16
-#define AT_DMA_BTC(x) (0x1 << (x))
-#define AT_DMA_CBTC(x) (0x1 << (AT_DMA_CBTC_OFFSET + (x)))
-#define AT_DMA_ERR(x) (0x1 << (AT_DMA_ERR_OFFSET + (x)))
-
-#define AT_DMA_CHER 0x28 /* Channel Handler Enable Register */
-#define AT_DMA_ENA(x) (0x1 << (x))
-#define AT_DMA_SUSP(x) (0x1 << ( 8 + (x)))
-#define AT_DMA_KEEP(x) (0x1 << (24 + (x)))
-
-#define AT_DMA_CHDR 0x2C /* Channel Handler Disable Register */
-#define AT_DMA_DIS(x) (0x1 << (x))
-#define AT_DMA_RES(x) (0x1 << ( 8 + (x)))
-
-#define AT_DMA_CHSR 0x30 /* Channel Handler Status Register */
-#define AT_DMA_EMPT(x) (0x1 << (16 + (x)))
-#define AT_DMA_STAL(x) (0x1 << (24 + (x)))
-
-
-#define AT_DMA_CH_REGS_BASE 0x3C /* Channel registers base address */
-#define ch_regs(x) (AT_DMA_CH_REGS_BASE + (x) * 0x28) /* Channel x base addr */
-
-/* Hardware register offset for each channel */
-#define ATC_SADDR_OFFSET 0x00 /* Source Address Register */
-#define ATC_DADDR_OFFSET 0x04 /* Destination Address Register */
-#define ATC_DSCR_OFFSET 0x08 /* Descriptor Address Register */
-#define ATC_CTRLA_OFFSET 0x0C /* Control A Register */
-#define ATC_CTRLB_OFFSET 0x10 /* Control B Register */
-#define ATC_CFG_OFFSET 0x14 /* Configuration Register */
-#define ATC_SPIP_OFFSET 0x18 /* Src PIP Configuration Register */
-#define ATC_DPIP_OFFSET 0x1C /* Dst PIP Configuration Register */
-
-
-/* Bitfield definitions */
-
-/* Bitfields in DSCR */
-#define ATC_DSCR_IF(i) (0x3 & (i)) /* Dsc feched via AHB-Lite Interface i */
-
-/* Bitfields in CTRLA */
-#define ATC_BTSIZE_MAX 0xFFFFUL /* Maximum Buffer Transfer Size */
-#define ATC_BTSIZE(x) (ATC_BTSIZE_MAX & (x)) /* Buffer Transfer Size */
-#define ATC_SCSIZE_MASK (0x7 << 16) /* Source Chunk Transfer Size */
-#define ATC_SCSIZE(x) (ATC_SCSIZE_MASK & ((x) << 16))
-#define ATC_SCSIZE_1 (0x0 << 16)
-#define ATC_SCSIZE_4 (0x1 << 16)
-#define ATC_SCSIZE_8 (0x2 << 16)
-#define ATC_SCSIZE_16 (0x3 << 16)
-#define ATC_SCSIZE_32 (0x4 << 16)
-#define ATC_SCSIZE_64 (0x5 << 16)
-#define ATC_SCSIZE_128 (0x6 << 16)
-#define ATC_SCSIZE_256 (0x7 << 16)
-#define ATC_DCSIZE_MASK (0x7 << 20) /* Destination Chunk Transfer Size */
-#define ATC_DCSIZE(x) (ATC_DCSIZE_MASK & ((x) << 20))
-#define ATC_DCSIZE_1 (0x0 << 20)
-#define ATC_DCSIZE_4 (0x1 << 20)
-#define ATC_DCSIZE_8 (0x2 << 20)
-#define ATC_DCSIZE_16 (0x3 << 20)
-#define ATC_DCSIZE_32 (0x4 << 20)
-#define ATC_DCSIZE_64 (0x5 << 20)
-#define ATC_DCSIZE_128 (0x6 << 20)
-#define ATC_DCSIZE_256 (0x7 << 20)
-#define ATC_SRC_WIDTH_MASK (0x3 << 24) /* Source Single Transfer Size */
-#define ATC_SRC_WIDTH(x) ((x) << 24)
-#define ATC_SRC_WIDTH_BYTE (0x0 << 24)
-#define ATC_SRC_WIDTH_HALFWORD (0x1 << 24)
-#define ATC_SRC_WIDTH_WORD (0x2 << 24)
-#define ATC_REG_TO_SRC_WIDTH(r) (((r) >> 24) & 0x3)
-#define ATC_DST_WIDTH_MASK (0x3 << 28) /* Destination Single Transfer Size */
-#define ATC_DST_WIDTH(x) ((x) << 28)
-#define ATC_DST_WIDTH_BYTE (0x0 << 28)
-#define ATC_DST_WIDTH_HALFWORD (0x1 << 28)
-#define ATC_DST_WIDTH_WORD (0x2 << 28)
-#define ATC_DONE (0x1 << 31) /* Tx Done (only written back in descriptor) */
-
-/* Bitfields in CTRLB */
-#define ATC_SIF(i) (0x3 & (i)) /* Src tx done via AHB-Lite Interface i */
-#define ATC_DIF(i) ((0x3 & (i)) << 4) /* Dst tx done via AHB-Lite Interface i */
- /* Specify AHB interfaces */
-#define AT_DMA_MEM_IF 0 /* interface 0 as memory interface */
-#define AT_DMA_PER_IF 1 /* interface 1 as peripheral interface */
-
-#define ATC_SRC_PIP (0x1 << 8) /* Source Picture-in-Picture enabled */
-#define ATC_DST_PIP (0x1 << 12) /* Destination Picture-in-Picture enabled */
-#define ATC_SRC_DSCR_DIS (0x1 << 16) /* Src Descriptor fetch disable */
-#define ATC_DST_DSCR_DIS (0x1 << 20) /* Dst Descriptor fetch disable */
-#define ATC_FC_MASK (0x7 << 21) /* Choose Flow Controller */
-#define ATC_FC_MEM2MEM (0x0 << 21) /* Mem-to-Mem (DMA) */
-#define ATC_FC_MEM2PER (0x1 << 21) /* Mem-to-Periph (DMA) */
-#define ATC_FC_PER2MEM (0x2 << 21) /* Periph-to-Mem (DMA) */
-#define ATC_FC_PER2PER (0x3 << 21) /* Periph-to-Periph (DMA) */
-#define ATC_FC_PER2MEM_PER (0x4 << 21) /* Periph-to-Mem (Peripheral) */
-#define ATC_FC_MEM2PER_PER (0x5 << 21) /* Mem-to-Periph (Peripheral) */
-#define ATC_FC_PER2PER_SRCPER (0x6 << 21) /* Periph-to-Periph (Src Peripheral) */
-#define ATC_FC_PER2PER_DSTPER (0x7 << 21) /* Periph-to-Periph (Dst Peripheral) */
-#define ATC_SRC_ADDR_MODE_MASK (0x3 << 24)
-#define ATC_SRC_ADDR_MODE_INCR (0x0 << 24) /* Incrementing Mode */
-#define ATC_SRC_ADDR_MODE_DECR (0x1 << 24) /* Decrementing Mode */
-#define ATC_SRC_ADDR_MODE_FIXED (0x2 << 24) /* Fixed Mode */
-#define ATC_DST_ADDR_MODE_MASK (0x3 << 28)
-#define ATC_DST_ADDR_MODE_INCR (0x0 << 28) /* Incrementing Mode */
-#define ATC_DST_ADDR_MODE_DECR (0x1 << 28) /* Decrementing Mode */
-#define ATC_DST_ADDR_MODE_FIXED (0x2 << 28) /* Fixed Mode */
-#define ATC_IEN (0x1 << 30) /* BTC interrupt enable (active low) */
-#define ATC_AUTO (0x1 << 31) /* Auto multiple buffer tx enable */
-
-/* Bitfields in CFG */
-#define ATC_PER_MSB(h) ((0x30U & (h)) >> 4) /* Extract most significant bits of a handshaking identifier */
-
-#define ATC_SRC_PER(h) (0xFU & (h)) /* Channel src rq associated with periph handshaking ifc h */
-#define ATC_DST_PER(h) ((0xFU & (h)) << 4) /* Channel dst rq associated with periph handshaking ifc h */
-#define ATC_SRC_REP (0x1 << 8) /* Source Replay Mod */
-#define ATC_SRC_H2SEL (0x1 << 9) /* Source Handshaking Mod */
-#define ATC_SRC_H2SEL_SW (0x0 << 9)
-#define ATC_SRC_H2SEL_HW (0x1 << 9)
-#define ATC_SRC_PER_MSB(h) (ATC_PER_MSB(h) << 10) /* Channel src rq (most significant bits) */
-#define ATC_DST_REP (0x1 << 12) /* Destination Replay Mod */
-#define ATC_DST_H2SEL (0x1 << 13) /* Destination Handshaking Mod */
-#define ATC_DST_H2SEL_SW (0x0 << 13)
-#define ATC_DST_H2SEL_HW (0x1 << 13)
-#define ATC_DST_PER_MSB(h) (ATC_PER_MSB(h) << 14) /* Channel dst rq (most significant bits) */
-#define ATC_SOD (0x1 << 16) /* Stop On Done */
-#define ATC_LOCK_IF (0x1 << 20) /* Interface Lock */
-#define ATC_LOCK_B (0x1 << 21) /* AHB Bus Lock */
-#define ATC_LOCK_IF_L (0x1 << 22) /* Master Interface Arbiter Lock */
-#define ATC_LOCK_IF_L_CHUNK (0x0 << 22)
-#define ATC_LOCK_IF_L_BUFFER (0x1 << 22)
-#define ATC_AHB_PROT_MASK (0x7 << 24) /* AHB Protection */
-#define ATC_FIFOCFG_MASK (0x3 << 28) /* FIFO Request Configuration */
-#define ATC_FIFOCFG_LARGESTBURST (0x0 << 28)
-#define ATC_FIFOCFG_HALFFIFO (0x1 << 28)
-#define ATC_FIFOCFG_ENOUGHSPACE (0x2 << 28)
-
-/* Bitfields in SPIP */
-#define ATC_SPIP_HOLE(x) (0xFFFFU & (x))
-#define ATC_SPIP_BOUNDARY(x) ((0x3FF & (x)) << 16)
-
-/* Bitfields in DPIP */
-#define ATC_DPIP_HOLE(x) (0xFFFFU & (x))
-#define ATC_DPIP_BOUNDARY(x) ((0x3FF & (x)) << 16)
-
-
-/*-- descriptors -----------------------------------------------------*/
-
-/* LLI == Linked List Item; aka DMA buffer descriptor */
-struct at_lli {
- /* values that are not changed by hardware */
- u32 saddr;
- u32 daddr;
- /* value that may get written back: */
- u32 ctrla;
- /* more values that are not changed by hardware */
- u32 ctrlb;
- u32 dscr; /* chain to next lli */
-};
-
-/**
- * struct at_desc - software descriptor
- * @at_lli: hardware lli structure
- * @txd: support for the async_tx api
- * @desc_node: node on the channed descriptors list
- * @len: descriptor byte count
- * @total_len: total transaction byte count
- */
-struct at_desc {
- /* FIRST values the hardware uses */
- struct at_lli lli;
-
- /* THEN values for driver housekeeping */
- struct list_head tx_list;
- struct dma_async_tx_descriptor txd;
- struct list_head desc_node;
- size_t len;
- size_t total_len;
-
- /* Interleaved data */
- size_t boundary;
- size_t dst_hole;
- size_t src_hole;
-
- /* Memset temporary buffer */
- bool memset_buffer;
- dma_addr_t memset_paddr;
- int *memset_vaddr;
-};
-
-static inline struct at_desc *
-txd_to_at_desc(struct dma_async_tx_descriptor *txd)
-{
- return container_of(txd, struct at_desc, txd);
-}
-
-
-/*-- Channels --------------------------------------------------------*/
-
-/**
- * atc_status - information bits stored in channel status flag
- *
- * Manipulated with atomic operations.
- */
-enum atc_status {
- ATC_IS_ERROR = 0,
- ATC_IS_PAUSED = 1,
- ATC_IS_CYCLIC = 24,
-};
-
-/**
- * struct at_dma_chan - internal representation of an Atmel HDMAC channel
- * @chan_common: common dmaengine channel object members
- * @device: parent device
- * @ch_regs: memory mapped register base
- * @mask: channel index in a mask
- * @per_if: peripheral interface
- * @mem_if: memory interface
- * @status: transmit status information from irq/prep* functions
- * to tasklet (use atomic operations)
- * @tasklet: bottom half to finish transaction work
- * @save_cfg: configuration register that is saved on suspend/resume cycle
- * @save_dscr: for cyclic operations, preserve next descriptor address in
- * the cyclic list on suspend/resume cycle
- * @dma_sconfig: configuration for slave transfers, passed via
- * .device_config
- * @lock: serializes enqueue/dequeue operations to descriptors lists
- * @active_list: list of descriptors dmaengine is being running on
- * @queue: list of descriptors ready to be submitted to engine
- * @free_list: list of descriptors usable by the channel
- */
-struct at_dma_chan {
- struct dma_chan chan_common;
- struct at_dma *device;
- void __iomem *ch_regs;
- u8 mask;
- u8 per_if;
- u8 mem_if;
- unsigned long status;
- struct tasklet_struct tasklet;
- u32 save_cfg;
- u32 save_dscr;
- struct dma_slave_config dma_sconfig;
-
- spinlock_t lock;
-
- /* these other elements are all protected by lock */
- struct list_head active_list;
- struct list_head queue;
- struct list_head free_list;
-};
-
-#define channel_readl(atchan, name) \
- __raw_readl((atchan)->ch_regs + ATC_##name##_OFFSET)
-
-#define channel_writel(atchan, name, val) \
- __raw_writel((val), (atchan)->ch_regs + ATC_##name##_OFFSET)
-
-static inline struct at_dma_chan *to_at_dma_chan(struct dma_chan *dchan)
-{
- return container_of(dchan, struct at_dma_chan, chan_common);
-}
-
-/*
- * Fix sconfig's burst size according to at_hdmac. We need to convert them as:
- * 1 -> 0, 4 -> 1, 8 -> 2, 16 -> 3, 32 -> 4, 64 -> 5, 128 -> 6, 256 -> 7.
- *
- * This can be done by finding most significant bit set.
- */
-static inline void convert_burst(u32 *maxburst)
-{
- if (*maxburst > 1)
- *maxburst = fls(*maxburst) - 2;
- else
- *maxburst = 0;
-}
-
-/*
- * Fix sconfig's bus width according to at_hdmac.
- * 1 byte -> 0, 2 bytes -> 1, 4 bytes -> 2.
- */
-static inline u8 convert_buswidth(enum dma_slave_buswidth addr_width)
-{
- switch (addr_width) {
- case DMA_SLAVE_BUSWIDTH_2_BYTES:
- return 1;
- case DMA_SLAVE_BUSWIDTH_4_BYTES:
- return 2;
- default:
- /* For 1 byte width or fallback */
- return 0;
- }
-}
-
-/*-- Controller ------------------------------------------------------*/
-
-/**
- * struct at_dma - internal representation of an Atmel HDMA Controller
- * @chan_common: common dmaengine dma_device object members
- * @atdma_devtype: identifier of DMA controller compatibility
- * @ch_regs: memory mapped register base
- * @clk: dma controller clock
- * @save_imr: interrupt mask register that is saved on suspend/resume cycle
- * @all_chan_mask: all channels availlable in a mask
- * @dma_desc_pool: base of DMA descriptor region (DMA address)
- * @chan: channels table to store at_dma_chan structures
- */
-struct at_dma {
- struct dma_device dma_common;
- void __iomem *regs;
- struct clk *clk;
- u32 save_imr;
-
- u8 all_chan_mask;
-
- struct dma_pool *dma_desc_pool;
- struct dma_pool *memset_pool;
- /* AT THE END channels table */
- struct at_dma_chan chan[];
-};
-
-#define dma_readl(atdma, name) \
- __raw_readl((atdma)->regs + AT_DMA_##name)
-#define dma_writel(atdma, name, val) \
- __raw_writel((val), (atdma)->regs + AT_DMA_##name)
-
-static inline struct at_dma *to_at_dma(struct dma_device *ddev)
-{
- return container_of(ddev, struct at_dma, dma_common);
-}
-
-
-/*-- Helper functions ------------------------------------------------*/
-
-static struct device *chan2dev(struct dma_chan *chan)
-{
- return &chan->dev->device;
-}
-
-#if defined(VERBOSE_DEBUG)
-static void vdbg_dump_regs(struct at_dma_chan *atchan)
-{
- struct at_dma *atdma = to_at_dma(atchan->chan_common.device);
-
- dev_err(chan2dev(&atchan->chan_common),
- " channel %d : imr = 0x%x, chsr = 0x%x\n",
- atchan->chan_common.chan_id,
- dma_readl(atdma, EBCIMR),
- dma_readl(atdma, CHSR));
-
- dev_err(chan2dev(&atchan->chan_common),
- " channel: s0x%x d0x%x ctrl0x%x:0x%x cfg0x%x l0x%x\n",
- channel_readl(atchan, SADDR),
- channel_readl(atchan, DADDR),
- channel_readl(atchan, CTRLA),
- channel_readl(atchan, CTRLB),
- channel_readl(atchan, CFG),
- channel_readl(atchan, DSCR));
-}
-#else
-static void vdbg_dump_regs(struct at_dma_chan *atchan) {}
-#endif
-
-static void atc_dump_lli(struct at_dma_chan *atchan, struct at_lli *lli)
-{
- dev_crit(chan2dev(&atchan->chan_common),
- "desc: s%pad d%pad ctrl0x%x:0x%x l%pad\n",
- &lli->saddr, &lli->daddr,
- lli->ctrla, lli->ctrlb, &lli->dscr);
-}
-
-
-static void atc_setup_irq(struct at_dma *atdma, int chan_id, int on)
-{
- u32 ebci;
-
- /* enable interrupts on buffer transfer completion & error */
- ebci = AT_DMA_BTC(chan_id)
- | AT_DMA_ERR(chan_id);
- if (on)
- dma_writel(atdma, EBCIER, ebci);
- else
- dma_writel(atdma, EBCIDR, ebci);
-}
-
-static void atc_enable_chan_irq(struct at_dma *atdma, int chan_id)
-{
- atc_setup_irq(atdma, chan_id, 1);
-}
-
-static void atc_disable_chan_irq(struct at_dma *atdma, int chan_id)
-{
- atc_setup_irq(atdma, chan_id, 0);
-}
-
-
-/**
- * atc_chan_is_enabled - test if given channel is enabled
- * @atchan: channel we want to test status
- */
-static inline int atc_chan_is_enabled(struct at_dma_chan *atchan)
-{
- struct at_dma *atdma = to_at_dma(atchan->chan_common.device);
-
- return !!(dma_readl(atdma, CHSR) & atchan->mask);
-}
-
-/**
- * atc_chan_is_paused - test channel pause/resume status
- * @atchan: channel we want to test status
- */
-static inline int atc_chan_is_paused(struct at_dma_chan *atchan)
-{
- return test_bit(ATC_IS_PAUSED, &atchan->status);
-}
-
-/**
- * atc_chan_is_cyclic - test if given channel has cyclic property set
- * @atchan: channel we want to test status
- */
-static inline int atc_chan_is_cyclic(struct at_dma_chan *atchan)
-{
- return test_bit(ATC_IS_CYCLIC, &atchan->status);
-}
-
-/**
- * set_desc_eol - set end-of-link to descriptor so it will end transfer
- * @desc: descriptor, signle or at the end of a chain, to end chain on
- */
-static void set_desc_eol(struct at_desc *desc)
-{
- u32 ctrlb = desc->lli.ctrlb;
-
- ctrlb &= ~ATC_IEN;
- ctrlb |= ATC_SRC_DSCR_DIS | ATC_DST_DSCR_DIS;
-
- desc->lli.ctrlb = ctrlb;
- desc->lli.dscr = 0;
-}
-
-#endif /* AT_HDMAC_REGS_H */
diff --git a/drivers/dma/dma-jz4780.c b/drivers/dma/dma-jz4780.c
index 2a48380..9c1a6e9 100644
--- a/drivers/dma/dma-jz4780.c
+++ b/drivers/dma/dma-jz4780.c
@@ -1038,6 +1038,13 @@ static const struct jz4780_dma_soc_data jz4725b_dma_soc_data = {
JZ_SOC_DATA_BREAK_LINKS,
};
+static const struct jz4780_dma_soc_data jz4755_dma_soc_data = {
+ .nb_channels = 4,
+ .transfer_ord_max = 5,
+ .flags = JZ_SOC_DATA_PER_CHAN_PM | JZ_SOC_DATA_NO_DCKES_DCKEC |
+ JZ_SOC_DATA_BREAK_LINKS,
+};
+
static const struct jz4780_dma_soc_data jz4760_dma_soc_data = {
.nb_channels = 5,
.transfer_ord_max = 6,
@@ -1101,6 +1108,7 @@ static const struct jz4780_dma_soc_data x1830_dma_soc_data = {
static const struct of_device_id jz4780_dma_dt_match[] = {
{ .compatible = "ingenic,jz4740-dma", .data = &jz4740_dma_soc_data },
{ .compatible = "ingenic,jz4725b-dma", .data = &jz4725b_dma_soc_data },
+ { .compatible = "ingenic,jz4755-dma", .data = &jz4755_dma_soc_data },
{ .compatible = "ingenic,jz4760-dma", .data = &jz4760_dma_soc_data },
{ .compatible = "ingenic,jz4760-mdma", .data = &jz4760_mdma_soc_data },
{ .compatible = "ingenic,jz4760-bdma", .data = &jz4760_bdma_soc_data },
diff --git a/drivers/dma/idma64.c b/drivers/dma/idma64.c
index f4c07ad..c33087c 100644
--- a/drivers/dma/idma64.c
+++ b/drivers/dma/idma64.c
@@ -600,7 +600,7 @@ static int idma64_probe(struct idma64_chip *chip)
return 0;
}
-static int idma64_remove(struct idma64_chip *chip)
+static void idma64_remove(struct idma64_chip *chip)
{
struct idma64 *idma64 = chip->idma64;
unsigned short i;
@@ -618,8 +618,6 @@ static int idma64_remove(struct idma64_chip *chip)
tasklet_kill(&idma64c->vchan.task);
}
-
- return 0;
}
/* ---------------------------------------------------------------------- */
@@ -664,7 +662,9 @@ static int idma64_platform_remove(struct platform_device *pdev)
{
struct idma64_chip *chip = platform_get_drvdata(pdev);
- return idma64_remove(chip);
+ idma64_remove(chip);
+
+ return 0;
}
static int __maybe_unused idma64_pm_suspend(struct device *dev)
diff --git a/drivers/dma/idxd/device.c b/drivers/dma/idxd/device.c
index 6f44fa8..06f5d37 100644
--- a/drivers/dma/idxd/device.c
+++ b/drivers/dma/idxd/device.c
@@ -7,7 +7,6 @@
#include <linux/io-64-nonatomic-lo-hi.h>
#include <linux/dmaengine.h>
#include <linux/irq.h>
-#include <linux/msi.h>
#include <uapi/linux/idxd.h>
#include "../dmaengine.h"
#include "idxd.h"
diff --git a/drivers/dma/idxd/sysfs.c b/drivers/dma/idxd/sysfs.c
index 7269bd5..3229dfc 100644
--- a/drivers/dma/idxd/sysfs.c
+++ b/drivers/dma/idxd/sysfs.c
@@ -528,6 +528,22 @@ static bool idxd_group_attr_progress_limit_invisible(struct attribute *attr,
!idxd->hw.group_cap.progress_limit;
}
+static bool idxd_group_attr_read_buffers_invisible(struct attribute *attr,
+ struct idxd_device *idxd)
+{
+ /*
+ * Intel IAA does not support Read Buffer allocation control,
+ * make these attributes invisible.
+ */
+ return (attr == &dev_attr_group_use_token_limit.attr ||
+ attr == &dev_attr_group_use_read_buffer_limit.attr ||
+ attr == &dev_attr_group_tokens_allowed.attr ||
+ attr == &dev_attr_group_read_buffers_allowed.attr ||
+ attr == &dev_attr_group_tokens_reserved.attr ||
+ attr == &dev_attr_group_read_buffers_reserved.attr) &&
+ idxd->data->type == IDXD_TYPE_IAX;
+}
+
static umode_t idxd_group_attr_visible(struct kobject *kobj,
struct attribute *attr, int n)
{
@@ -538,6 +554,9 @@ static umode_t idxd_group_attr_visible(struct kobject *kobj,
if (idxd_group_attr_progress_limit_invisible(attr, idxd))
return 0;
+ if (idxd_group_attr_read_buffers_invisible(attr, idxd))
+ return 0;
+
return attr->mode;
}
@@ -1233,6 +1252,14 @@ static bool idxd_wq_attr_op_config_invisible(struct attribute *attr,
!idxd->hw.wq_cap.op_config;
}
+static bool idxd_wq_attr_max_batch_size_invisible(struct attribute *attr,
+ struct idxd_device *idxd)
+{
+ /* Intel IAA does not support batch processing, make it invisible */
+ return attr == &dev_attr_wq_max_batch_size.attr &&
+ idxd->data->type == IDXD_TYPE_IAX;
+}
+
static umode_t idxd_wq_attr_visible(struct kobject *kobj,
struct attribute *attr, int n)
{
@@ -1243,6 +1270,9 @@ static umode_t idxd_wq_attr_visible(struct kobject *kobj,
if (idxd_wq_attr_op_config_invisible(attr, idxd))
return 0;
+ if (idxd_wq_attr_max_batch_size_invisible(attr, idxd))
+ return 0;
+
return attr->mode;
}
@@ -1533,6 +1563,43 @@ static ssize_t cmd_status_store(struct device *dev, struct device_attribute *att
}
static DEVICE_ATTR_RW(cmd_status);
+static bool idxd_device_attr_max_batch_size_invisible(struct attribute *attr,
+ struct idxd_device *idxd)
+{
+ /* Intel IAA does not support batch processing, make it invisible */
+ return attr == &dev_attr_max_batch_size.attr &&
+ idxd->data->type == IDXD_TYPE_IAX;
+}
+
+static bool idxd_device_attr_read_buffers_invisible(struct attribute *attr,
+ struct idxd_device *idxd)
+{
+ /*
+ * Intel IAA does not support Read Buffer allocation control,
+ * make these attributes invisible.
+ */
+ return (attr == &dev_attr_max_tokens.attr ||
+ attr == &dev_attr_max_read_buffers.attr ||
+ attr == &dev_attr_token_limit.attr ||
+ attr == &dev_attr_read_buffer_limit.attr) &&
+ idxd->data->type == IDXD_TYPE_IAX;
+}
+
+static umode_t idxd_device_attr_visible(struct kobject *kobj,
+ struct attribute *attr, int n)
+{
+ struct device *dev = container_of(kobj, struct device, kobj);
+ struct idxd_device *idxd = confdev_to_idxd(dev);
+
+ if (idxd_device_attr_max_batch_size_invisible(attr, idxd))
+ return 0;
+
+ if (idxd_device_attr_read_buffers_invisible(attr, idxd))
+ return 0;
+
+ return attr->mode;
+}
+
static struct attribute *idxd_device_attributes[] = {
&dev_attr_version.attr,
&dev_attr_max_groups.attr,
@@ -1560,6 +1627,7 @@ static struct attribute *idxd_device_attributes[] = {
static const struct attribute_group idxd_device_attribute_group = {
.attrs = idxd_device_attributes,
+ .is_visible = idxd_device_attr_visible,
};
static const struct attribute_group *idxd_attribute_groups[] = {
diff --git a/drivers/dma/ioat/dma.c b/drivers/dma/ioat/dma.c
index e2070df..79d2440 100644
--- a/drivers/dma/ioat/dma.c
+++ b/drivers/dma/ioat/dma.c
@@ -33,7 +33,7 @@ MODULE_PARM_DESC(completion_timeout,
static int idle_timeout = 2000;
module_param(idle_timeout, int, 0644);
MODULE_PARM_DESC(idle_timeout,
- "set ioat idel timeout [msec] (default 2000 [msec])");
+ "set ioat idle timeout [msec] (default 2000 [msec])");
#define IDLE_TIMEOUT msecs_to_jiffies(idle_timeout)
#define COMPLETION_TIMEOUT msecs_to_jiffies(completion_timeout)
diff --git a/drivers/dma/iop-adma.c b/drivers/dma/iop-adma.c
deleted file mode 100644
index 310b899..0000000
--- a/drivers/dma/iop-adma.c
+++ /dev/null
@@ -1,1554 +0,0 @@
-// SPDX-License-Identifier: GPL-2.0-only
-/*
- * offload engine driver for the Intel Xscale series of i/o processors
- * Copyright © 2006, Intel Corporation.
- */
-
-/*
- * This driver supports the asynchrounous DMA copy and RAID engines available
- * on the Intel Xscale(R) family of I/O Processors (IOP 32x, 33x, 134x)
- */
-
-#include <linux/init.h>
-#include <linux/module.h>
-#include <linux/delay.h>
-#include <linux/dma-mapping.h>
-#include <linux/spinlock.h>
-#include <linux/interrupt.h>
-#include <linux/platform_device.h>
-#include <linux/prefetch.h>
-#include <linux/memory.h>
-#include <linux/ioport.h>
-#include <linux/raid/pq.h>
-#include <linux/slab.h>
-
-#include "iop-adma.h"
-#include "dmaengine.h"
-
-#define to_iop_adma_chan(chan) container_of(chan, struct iop_adma_chan, common)
-#define to_iop_adma_device(dev) \
- container_of(dev, struct iop_adma_device, common)
-#define tx_to_iop_adma_slot(tx) \
- container_of(tx, struct iop_adma_desc_slot, async_tx)
-
-/**
- * iop_adma_free_slots - flags descriptor slots for reuse
- * @slot: Slot to free
- * Caller must hold &iop_chan->lock while calling this function
- */
-static void iop_adma_free_slots(struct iop_adma_desc_slot *slot)
-{
- int stride = slot->slots_per_op;
-
- while (stride--) {
- slot->slots_per_op = 0;
- slot = list_entry(slot->slot_node.next,
- struct iop_adma_desc_slot,
- slot_node);
- }
-}
-
-static dma_cookie_t
-iop_adma_run_tx_complete_actions(struct iop_adma_desc_slot *desc,
- struct iop_adma_chan *iop_chan, dma_cookie_t cookie)
-{
- struct dma_async_tx_descriptor *tx = &desc->async_tx;
-
- BUG_ON(tx->cookie < 0);
- if (tx->cookie > 0) {
- cookie = tx->cookie;
- tx->cookie = 0;
-
- /* call the callback (must not sleep or submit new
- * operations to this channel)
- */
- dmaengine_desc_get_callback_invoke(tx, NULL);
-
- dma_descriptor_unmap(tx);
- if (desc->group_head)
- desc->group_head = NULL;
- }
-
- /* run dependent operations */
- dma_run_dependencies(tx);
-
- return cookie;
-}
-
-static int
-iop_adma_clean_slot(struct iop_adma_desc_slot *desc,
- struct iop_adma_chan *iop_chan)
-{
- /* the client is allowed to attach dependent operations
- * until 'ack' is set
- */
- if (!async_tx_test_ack(&desc->async_tx))
- return 0;
-
- /* leave the last descriptor in the chain
- * so we can append to it
- */
- if (desc->chain_node.next == &iop_chan->chain)
- return 1;
-
- dev_dbg(iop_chan->device->common.dev,
- "\tfree slot: %d slots_per_op: %d\n",
- desc->idx, desc->slots_per_op);
-
- list_del(&desc->chain_node);
- iop_adma_free_slots(desc);
-
- return 0;
-}
-
-static void __iop_adma_slot_cleanup(struct iop_adma_chan *iop_chan)
-{
- struct iop_adma_desc_slot *iter, *_iter, *grp_start = NULL;
- dma_cookie_t cookie = 0;
- u32 current_desc = iop_chan_get_current_descriptor(iop_chan);
- int busy = iop_chan_is_busy(iop_chan);
- int seen_current = 0, slot_cnt = 0, slots_per_op = 0;
-
- dev_dbg(iop_chan->device->common.dev, "%s\n", __func__);
- /* free completed slots from the chain starting with
- * the oldest descriptor
- */
- list_for_each_entry_safe(iter, _iter, &iop_chan->chain,
- chain_node) {
- pr_debug("\tcookie: %d slot: %d busy: %d "
- "this_desc: %pad next_desc: %#llx ack: %d\n",
- iter->async_tx.cookie, iter->idx, busy,
- &iter->async_tx.phys, (u64)iop_desc_get_next_desc(iter),
- async_tx_test_ack(&iter->async_tx));
- prefetch(_iter);
- prefetch(&_iter->async_tx);
-
- /* do not advance past the current descriptor loaded into the
- * hardware channel, subsequent descriptors are either in
- * process or have not been submitted
- */
- if (seen_current)
- break;
-
- /* stop the search if we reach the current descriptor and the
- * channel is busy, or if it appears that the current descriptor
- * needs to be re-read (i.e. has been appended to)
- */
- if (iter->async_tx.phys == current_desc) {
- BUG_ON(seen_current++);
- if (busy || iop_desc_get_next_desc(iter))
- break;
- }
-
- /* detect the start of a group transaction */
- if (!slot_cnt && !slots_per_op) {
- slot_cnt = iter->slot_cnt;
- slots_per_op = iter->slots_per_op;
- if (slot_cnt <= slots_per_op) {
- slot_cnt = 0;
- slots_per_op = 0;
- }
- }
-
- if (slot_cnt) {
- pr_debug("\tgroup++\n");
- if (!grp_start)
- grp_start = iter;
- slot_cnt -= slots_per_op;
- }
-
- /* all the members of a group are complete */
- if (slots_per_op != 0 && slot_cnt == 0) {
- struct iop_adma_desc_slot *grp_iter, *_grp_iter;
- int end_of_chain = 0;
- pr_debug("\tgroup end\n");
-
- /* collect the total results */
- if (grp_start->xor_check_result) {
- u32 zero_sum_result = 0;
- slot_cnt = grp_start->slot_cnt;
- grp_iter = grp_start;
-
- list_for_each_entry_from(grp_iter,
- &iop_chan->chain, chain_node) {
- zero_sum_result |=
- iop_desc_get_zero_result(grp_iter);
- pr_debug("\titer%d result: %d\n",
- grp_iter->idx, zero_sum_result);
- slot_cnt -= slots_per_op;
- if (slot_cnt == 0)
- break;
- }
- pr_debug("\tgrp_start->xor_check_result: %p\n",
- grp_start->xor_check_result);
- *grp_start->xor_check_result = zero_sum_result;
- }
-
- /* clean up the group */
- slot_cnt = grp_start->slot_cnt;
- grp_iter = grp_start;
- list_for_each_entry_safe_from(grp_iter, _grp_iter,
- &iop_chan->chain, chain_node) {
- cookie = iop_adma_run_tx_complete_actions(
- grp_iter, iop_chan, cookie);
-
- slot_cnt -= slots_per_op;
- end_of_chain = iop_adma_clean_slot(grp_iter,
- iop_chan);
-
- if (slot_cnt == 0 || end_of_chain)
- break;
- }
-
- /* the group should be complete at this point */
- BUG_ON(slot_cnt);
-
- slots_per_op = 0;
- grp_start = NULL;
- if (end_of_chain)
- break;
- else
- continue;
- } else if (slots_per_op) /* wait for group completion */
- continue;
-
- /* write back zero sum results (single descriptor case) */
- if (iter->xor_check_result && iter->async_tx.cookie)
- *iter->xor_check_result =
- iop_desc_get_zero_result(iter);
-
- cookie = iop_adma_run_tx_complete_actions(
- iter, iop_chan, cookie);
-
- if (iop_adma_clean_slot(iter, iop_chan))
- break;
- }
-
- if (cookie > 0) {
- iop_chan->common.completed_cookie = cookie;
- pr_debug("\tcompleted cookie %d\n", cookie);
- }
-}
-
-static void
-iop_adma_slot_cleanup(struct iop_adma_chan *iop_chan)
-{
- spin_lock_bh(&iop_chan->lock);
- __iop_adma_slot_cleanup(iop_chan);
- spin_unlock_bh(&iop_chan->lock);
-}
-
-static void iop_adma_tasklet(struct tasklet_struct *t)
-{
- struct iop_adma_chan *iop_chan = from_tasklet(iop_chan, t,
- irq_tasklet);
-
- /* lockdep will flag depedency submissions as potentially
- * recursive locking, this is not the case as a dependency
- * submission will never recurse a channels submit routine.
- * There are checks in async_tx.c to prevent this.
- */
- spin_lock_nested(&iop_chan->lock, SINGLE_DEPTH_NESTING);
- __iop_adma_slot_cleanup(iop_chan);
- spin_unlock(&iop_chan->lock);
-}
-
-static struct iop_adma_desc_slot *
-iop_adma_alloc_slots(struct iop_adma_chan *iop_chan, int num_slots,
- int slots_per_op)
-{
- struct iop_adma_desc_slot *iter, *_iter, *alloc_start = NULL;
- LIST_HEAD(chain);
- int slots_found, retry = 0;
-
- /* start search from the last allocated descrtiptor
- * if a contiguous allocation can not be found start searching
- * from the beginning of the list
- */
-retry:
- slots_found = 0;
- if (retry == 0)
- iter = iop_chan->last_used;
- else
- iter = list_entry(&iop_chan->all_slots,
- struct iop_adma_desc_slot,
- slot_node);
-
- list_for_each_entry_safe_continue(
- iter, _iter, &iop_chan->all_slots, slot_node) {
- prefetch(_iter);
- prefetch(&_iter->async_tx);
- if (iter->slots_per_op) {
- /* give up after finding the first busy slot
- * on the second pass through the list
- */
- if (retry)
- break;
-
- slots_found = 0;
- continue;
- }
-
- /* start the allocation if the slot is correctly aligned */
- if (!slots_found++) {
- if (iop_desc_is_aligned(iter, slots_per_op))
- alloc_start = iter;
- else {
- slots_found = 0;
- continue;
- }
- }
-
- if (slots_found == num_slots) {
- struct iop_adma_desc_slot *alloc_tail = NULL;
- struct iop_adma_desc_slot *last_used = NULL;
- iter = alloc_start;
- while (num_slots) {
- int i;
- dev_dbg(iop_chan->device->common.dev,
- "allocated slot: %d "
- "(desc %p phys: %#llx) slots_per_op %d\n",
- iter->idx, iter->hw_desc,
- (u64)iter->async_tx.phys, slots_per_op);
-
- /* pre-ack all but the last descriptor */
- if (num_slots != slots_per_op)
- async_tx_ack(&iter->async_tx);
-
- list_add_tail(&iter->chain_node, &chain);
- alloc_tail = iter;
- iter->async_tx.cookie = 0;
- iter->slot_cnt = num_slots;
- iter->xor_check_result = NULL;
- for (i = 0; i < slots_per_op; i++) {
- iter->slots_per_op = slots_per_op - i;
- last_used = iter;
- iter = list_entry(iter->slot_node.next,
- struct iop_adma_desc_slot,
- slot_node);
- }
- num_slots -= slots_per_op;
- }
- alloc_tail->group_head = alloc_start;
- alloc_tail->async_tx.cookie = -EBUSY;
- list_splice(&chain, &alloc_tail->tx_list);
- iop_chan->last_used = last_used;
- iop_desc_clear_next_desc(alloc_start);
- iop_desc_clear_next_desc(alloc_tail);
- return alloc_tail;
- }
- }
- if (!retry++)
- goto retry;
-
- /* perform direct reclaim if the allocation fails */
- __iop_adma_slot_cleanup(iop_chan);
-
- return NULL;
-}
-
-static void iop_adma_check_threshold(struct iop_adma_chan *iop_chan)
-{
- dev_dbg(iop_chan->device->common.dev, "pending: %d\n",
- iop_chan->pending);
-
- if (iop_chan->pending >= IOP_ADMA_THRESHOLD) {
- iop_chan->pending = 0;
- iop_chan_append(iop_chan);
- }
-}
-
-static dma_cookie_t
-iop_adma_tx_submit(struct dma_async_tx_descriptor *tx)
-{
- struct iop_adma_desc_slot *sw_desc = tx_to_iop_adma_slot(tx);
- struct iop_adma_chan *iop_chan = to_iop_adma_chan(tx->chan);
- struct iop_adma_desc_slot *grp_start, *old_chain_tail;
- int slot_cnt;
- dma_cookie_t cookie;
- dma_addr_t next_dma;
-
- grp_start = sw_desc->group_head;
- slot_cnt = grp_start->slot_cnt;
-
- spin_lock_bh(&iop_chan->lock);
- cookie = dma_cookie_assign(tx);
-
- old_chain_tail = list_entry(iop_chan->chain.prev,
- struct iop_adma_desc_slot, chain_node);
- list_splice_init(&sw_desc->tx_list,
- &old_chain_tail->chain_node);
-
- /* fix up the hardware chain */
- next_dma = grp_start->async_tx.phys;
- iop_desc_set_next_desc(old_chain_tail, next_dma);
- BUG_ON(iop_desc_get_next_desc(old_chain_tail) != next_dma); /* flush */
-
- /* check for pre-chained descriptors */
- iop_paranoia(iop_desc_get_next_desc(sw_desc));
-
- /* increment the pending count by the number of slots
- * memcpy operations have a 1:1 (slot:operation) relation
- * other operations are heavier and will pop the threshold
- * more often.
- */
- iop_chan->pending += slot_cnt;
- iop_adma_check_threshold(iop_chan);
- spin_unlock_bh(&iop_chan->lock);
-
- dev_dbg(iop_chan->device->common.dev, "%s cookie: %d slot: %d\n",
- __func__, sw_desc->async_tx.cookie, sw_desc->idx);
-
- return cookie;
-}
-
-static void iop_chan_start_null_memcpy(struct iop_adma_chan *iop_chan);
-static void iop_chan_start_null_xor(struct iop_adma_chan *iop_chan);
-
-/**
- * iop_adma_alloc_chan_resources - returns the number of allocated descriptors
- * @chan: allocate descriptor resources for this channel
- *
- * Note: We keep the slots for 1 operation on iop_chan->chain at all times. To
- * avoid deadlock, via async_xor, num_descs_in_pool must at a minimum be
- * greater than 2x the number slots needed to satisfy a device->max_xor
- * request.
- * */
-static int iop_adma_alloc_chan_resources(struct dma_chan *chan)
-{
- char *hw_desc;
- dma_addr_t dma_desc;
- int idx;
- struct iop_adma_chan *iop_chan = to_iop_adma_chan(chan);
- struct iop_adma_desc_slot *slot = NULL;
- int init = iop_chan->slots_allocated ? 0 : 1;
- struct iop_adma_platform_data *plat_data =
- dev_get_platdata(&iop_chan->device->pdev->dev);
- int num_descs_in_pool = plat_data->pool_size/IOP_ADMA_SLOT_SIZE;
-
- /* Allocate descriptor slots */
- do {
- idx = iop_chan->slots_allocated;
- if (idx == num_descs_in_pool)
- break;
-
- slot = kzalloc(sizeof(*slot), GFP_KERNEL);
- if (!slot) {
- printk(KERN_INFO "IOP ADMA Channel only initialized"
- " %d descriptor slots", idx);
- break;
- }
- hw_desc = (char *) iop_chan->device->dma_desc_pool_virt;
- slot->hw_desc = (void *) &hw_desc[idx * IOP_ADMA_SLOT_SIZE];
-
- dma_async_tx_descriptor_init(&slot->async_tx, chan);
- slot->async_tx.tx_submit = iop_adma_tx_submit;
- INIT_LIST_HEAD(&slot->tx_list);
- INIT_LIST_HEAD(&slot->chain_node);
- INIT_LIST_HEAD(&slot->slot_node);
- dma_desc = iop_chan->device->dma_desc_pool;
- slot->async_tx.phys = dma_desc + idx * IOP_ADMA_SLOT_SIZE;
- slot->idx = idx;
-
- spin_lock_bh(&iop_chan->lock);
- iop_chan->slots_allocated++;
- list_add_tail(&slot->slot_node, &iop_chan->all_slots);
- spin_unlock_bh(&iop_chan->lock);
- } while (iop_chan->slots_allocated < num_descs_in_pool);
-
- if (idx && !iop_chan->last_used)
- iop_chan->last_used = list_entry(iop_chan->all_slots.next,
- struct iop_adma_desc_slot,
- slot_node);
-
- dev_dbg(iop_chan->device->common.dev,
- "allocated %d descriptor slots last_used: %p\n",
- iop_chan->slots_allocated, iop_chan->last_used);
-
- /* initialize the channel and the chain with a null operation */
- if (init) {
- if (dma_has_cap(DMA_MEMCPY,
- iop_chan->device->common.cap_mask))
- iop_chan_start_null_memcpy(iop_chan);
- else if (dma_has_cap(DMA_XOR,
- iop_chan->device->common.cap_mask))
- iop_chan_start_null_xor(iop_chan);
- else
- BUG();
- }
-
- return (idx > 0) ? idx : -ENOMEM;
-}
-
-static struct dma_async_tx_descriptor *
-iop_adma_prep_dma_interrupt(struct dma_chan *chan, unsigned long flags)
-{
- struct iop_adma_chan *iop_chan = to_iop_adma_chan(chan);
- struct iop_adma_desc_slot *sw_desc, *grp_start;
- int slot_cnt, slots_per_op;
-
- dev_dbg(iop_chan->device->common.dev, "%s\n", __func__);
-
- spin_lock_bh(&iop_chan->lock);
- slot_cnt = iop_chan_interrupt_slot_count(&slots_per_op, iop_chan);
- sw_desc = iop_adma_alloc_slots(iop_chan, slot_cnt, slots_per_op);
- if (sw_desc) {
- grp_start = sw_desc->group_head;
- iop_desc_init_interrupt(grp_start, iop_chan);
- sw_desc->async_tx.flags = flags;
- }
- spin_unlock_bh(&iop_chan->lock);
-
- return sw_desc ? &sw_desc->async_tx : NULL;
-}
-
-static struct dma_async_tx_descriptor *
-iop_adma_prep_dma_memcpy(struct dma_chan *chan, dma_addr_t dma_dest,
- dma_addr_t dma_src, size_t len, unsigned long flags)
-{
- struct iop_adma_chan *iop_chan = to_iop_adma_chan(chan);
- struct iop_adma_desc_slot *sw_desc, *grp_start;
- int slot_cnt, slots_per_op;
-
- if (unlikely(!len))
- return NULL;
- BUG_ON(len > IOP_ADMA_MAX_BYTE_COUNT);
-
- dev_dbg(iop_chan->device->common.dev, "%s len: %zu\n",
- __func__, len);
-
- spin_lock_bh(&iop_chan->lock);
- slot_cnt = iop_chan_memcpy_slot_count(len, &slots_per_op);
- sw_desc = iop_adma_alloc_slots(iop_chan, slot_cnt, slots_per_op);
- if (sw_desc) {
- grp_start = sw_desc->group_head;
- iop_desc_init_memcpy(grp_start, flags);
- iop_desc_set_byte_count(grp_start, iop_chan, len);
- iop_desc_set_dest_addr(grp_start, iop_chan, dma_dest);
- iop_desc_set_memcpy_src_addr(grp_start, dma_src);
- sw_desc->async_tx.flags = flags;
- }
- spin_unlock_bh(&iop_chan->lock);
-
- return sw_desc ? &sw_desc->async_tx : NULL;
-}
-
-static struct dma_async_tx_descriptor *
-iop_adma_prep_dma_xor(struct dma_chan *chan, dma_addr_t dma_dest,
- dma_addr_t *dma_src, unsigned int src_cnt, size_t len,
- unsigned long flags)
-{
- struct iop_adma_chan *iop_chan = to_iop_adma_chan(chan);
- struct iop_adma_desc_slot *sw_desc, *grp_start;
- int slot_cnt, slots_per_op;
-
- if (unlikely(!len))
- return NULL;
- BUG_ON(len > IOP_ADMA_XOR_MAX_BYTE_COUNT);
-
- dev_dbg(iop_chan->device->common.dev,
- "%s src_cnt: %d len: %zu flags: %lx\n",
- __func__, src_cnt, len, flags);
-
- spin_lock_bh(&iop_chan->lock);
- slot_cnt = iop_chan_xor_slot_count(len, src_cnt, &slots_per_op);
- sw_desc = iop_adma_alloc_slots(iop_chan, slot_cnt, slots_per_op);
- if (sw_desc) {
- grp_start = sw_desc->group_head;
- iop_desc_init_xor(grp_start, src_cnt, flags);
- iop_desc_set_byte_count(grp_start, iop_chan, len);
- iop_desc_set_dest_addr(grp_start, iop_chan, dma_dest);
- sw_desc->async_tx.flags = flags;
- while (src_cnt--)
- iop_desc_set_xor_src_addr(grp_start, src_cnt,
- dma_src[src_cnt]);
- }
- spin_unlock_bh(&iop_chan->lock);
-
- return sw_desc ? &sw_desc->async_tx : NULL;
-}
-
-static struct dma_async_tx_descriptor *
-iop_adma_prep_dma_xor_val(struct dma_chan *chan, dma_addr_t *dma_src,
- unsigned int src_cnt, size_t len, u32 *result,
- unsigned long flags)
-{
- struct iop_adma_chan *iop_chan = to_iop_adma_chan(chan);
- struct iop_adma_desc_slot *sw_desc, *grp_start;
- int slot_cnt, slots_per_op;
-
- if (unlikely(!len))
- return NULL;
-
- dev_dbg(iop_chan->device->common.dev, "%s src_cnt: %d len: %zu\n",
- __func__, src_cnt, len);
-
- spin_lock_bh(&iop_chan->lock);
- slot_cnt = iop_chan_zero_sum_slot_count(len, src_cnt, &slots_per_op);
- sw_desc = iop_adma_alloc_slots(iop_chan, slot_cnt, slots_per_op);
- if (sw_desc) {
- grp_start = sw_desc->group_head;
- iop_desc_init_zero_sum(grp_start, src_cnt, flags);
- iop_desc_set_zero_sum_byte_count(grp_start, len);
- grp_start->xor_check_result = result;
- pr_debug("\t%s: grp_start->xor_check_result: %p\n",
- __func__, grp_start->xor_check_result);
- sw_desc->async_tx.flags = flags;
- while (src_cnt--)
- iop_desc_set_zero_sum_src_addr(grp_start, src_cnt,
- dma_src[src_cnt]);
- }
- spin_unlock_bh(&iop_chan->lock);
-
- return sw_desc ? &sw_desc->async_tx : NULL;
-}
-
-static struct dma_async_tx_descriptor *
-iop_adma_prep_dma_pq(struct dma_chan *chan, dma_addr_t *dst, dma_addr_t *src,
- unsigned int src_cnt, const unsigned char *scf, size_t len,
- unsigned long flags)
-{
- struct iop_adma_chan *iop_chan = to_iop_adma_chan(chan);
- struct iop_adma_desc_slot *sw_desc, *g;
- int slot_cnt, slots_per_op;
- int continue_srcs;
-
- if (unlikely(!len))
- return NULL;
- BUG_ON(len > IOP_ADMA_XOR_MAX_BYTE_COUNT);
-
- dev_dbg(iop_chan->device->common.dev,
- "%s src_cnt: %d len: %zu flags: %lx\n",
- __func__, src_cnt, len, flags);
-
- if (dmaf_p_disabled_continue(flags))
- continue_srcs = 1+src_cnt;
- else if (dmaf_continue(flags))
- continue_srcs = 3+src_cnt;
- else
- continue_srcs = 0+src_cnt;
-
- spin_lock_bh(&iop_chan->lock);
- slot_cnt = iop_chan_pq_slot_count(len, continue_srcs, &slots_per_op);
- sw_desc = iop_adma_alloc_slots(iop_chan, slot_cnt, slots_per_op);
- if (sw_desc) {
- int i;
-
- g = sw_desc->group_head;
- iop_desc_set_byte_count(g, iop_chan, len);
-
- /* even if P is disabled its destination address (bits
- * [3:0]) must match Q. It is ok if P points to an
- * invalid address, it won't be written.
- */
- if (flags & DMA_PREP_PQ_DISABLE_P)
- dst[0] = dst[1] & 0x7;
-
- iop_desc_set_pq_addr(g, dst);
- sw_desc->async_tx.flags = flags;
- for (i = 0; i < src_cnt; i++)
- iop_desc_set_pq_src_addr(g, i, src[i], scf[i]);
-
- /* if we are continuing a previous operation factor in
- * the old p and q values, see the comment for dma_maxpq
- * in include/linux/dmaengine.h
- */
- if (dmaf_p_disabled_continue(flags))
- iop_desc_set_pq_src_addr(g, i++, dst[1], 1);
- else if (dmaf_continue(flags)) {
- iop_desc_set_pq_src_addr(g, i++, dst[0], 0);
- iop_desc_set_pq_src_addr(g, i++, dst[1], 1);
- iop_desc_set_pq_src_addr(g, i++, dst[1], 0);
- }
- iop_desc_init_pq(g, i, flags);
- }
- spin_unlock_bh(&iop_chan->lock);
-
- return sw_desc ? &sw_desc->async_tx : NULL;
-}
-
-static struct dma_async_tx_descriptor *
-iop_adma_prep_dma_pq_val(struct dma_chan *chan, dma_addr_t *pq, dma_addr_t *src,
- unsigned int src_cnt, const unsigned char *scf,
- size_t len, enum sum_check_flags *pqres,
- unsigned long flags)
-{
- struct iop_adma_chan *iop_chan = to_iop_adma_chan(chan);
- struct iop_adma_desc_slot *sw_desc, *g;
- int slot_cnt, slots_per_op;
-
- if (unlikely(!len))
- return NULL;
- BUG_ON(len > IOP_ADMA_XOR_MAX_BYTE_COUNT);
-
- dev_dbg(iop_chan->device->common.dev, "%s src_cnt: %d len: %zu\n",
- __func__, src_cnt, len);
-
- spin_lock_bh(&iop_chan->lock);
- slot_cnt = iop_chan_pq_zero_sum_slot_count(len, src_cnt + 2, &slots_per_op);
- sw_desc = iop_adma_alloc_slots(iop_chan, slot_cnt, slots_per_op);
- if (sw_desc) {
- /* for validate operations p and q are tagged onto the
- * end of the source list
- */
- int pq_idx = src_cnt;
-
- g = sw_desc->group_head;
- iop_desc_init_pq_zero_sum(g, src_cnt+2, flags);
- iop_desc_set_pq_zero_sum_byte_count(g, len);
- g->pq_check_result = pqres;
- pr_debug("\t%s: g->pq_check_result: %p\n",
- __func__, g->pq_check_result);
- sw_desc->async_tx.flags = flags;
- while (src_cnt--)
- iop_desc_set_pq_zero_sum_src_addr(g, src_cnt,
- src[src_cnt],
- scf[src_cnt]);
- iop_desc_set_pq_zero_sum_addr(g, pq_idx, src);
- }
- spin_unlock_bh(&iop_chan->lock);
-
- return sw_desc ? &sw_desc->async_tx : NULL;
-}
-
-static void iop_adma_free_chan_resources(struct dma_chan *chan)
-{
- struct iop_adma_chan *iop_chan = to_iop_adma_chan(chan);
- struct iop_adma_desc_slot *iter, *_iter;
- int in_use_descs = 0;
-
- iop_adma_slot_cleanup(iop_chan);
-
- spin_lock_bh(&iop_chan->lock);
- list_for_each_entry_safe(iter, _iter, &iop_chan->chain,
- chain_node) {
- in_use_descs++;
- list_del(&iter->chain_node);
- }
- list_for_each_entry_safe_reverse(
- iter, _iter, &iop_chan->all_slots, slot_node) {
- list_del(&iter->slot_node);
- kfree(iter);
- iop_chan->slots_allocated--;
- }
- iop_chan->last_used = NULL;
-
- dev_dbg(iop_chan->device->common.dev, "%s slots_allocated %d\n",
- __func__, iop_chan->slots_allocated);
- spin_unlock_bh(&iop_chan->lock);
-
- /* one is ok since we left it on there on purpose */
- if (in_use_descs > 1)
- printk(KERN_ERR "IOP: Freeing %d in use descriptors!\n",
- in_use_descs - 1);
-}
-
-/**
- * iop_adma_status - poll the status of an ADMA transaction
- * @chan: ADMA channel handle
- * @cookie: ADMA transaction identifier
- * @txstate: a holder for the current state of the channel or NULL
- */
-static enum dma_status iop_adma_status(struct dma_chan *chan,
- dma_cookie_t cookie,
- struct dma_tx_state *txstate)
-{
- struct iop_adma_chan *iop_chan = to_iop_adma_chan(chan);
- int ret;
-
- ret = dma_cookie_status(chan, cookie, txstate);
- if (ret == DMA_COMPLETE)
- return ret;
-
- iop_adma_slot_cleanup(iop_chan);
-
- return dma_cookie_status(chan, cookie, txstate);
-}
-
-static irqreturn_t iop_adma_eot_handler(int irq, void *data)
-{
- struct iop_adma_chan *chan = data;
-
- dev_dbg(chan->device->common.dev, "%s\n", __func__);
-
- tasklet_schedule(&chan->irq_tasklet);
-
- iop_adma_device_clear_eot_status(chan);
-
- return IRQ_HANDLED;
-}
-
-static irqreturn_t iop_adma_eoc_handler(int irq, void *data)
-{
- struct iop_adma_chan *chan = data;
-
- dev_dbg(chan->device->common.dev, "%s\n", __func__);
-
- tasklet_schedule(&chan->irq_tasklet);
-
- iop_adma_device_clear_eoc_status(chan);
-
- return IRQ_HANDLED;
-}
-
-static irqreturn_t iop_adma_err_handler(int irq, void *data)
-{
- struct iop_adma_chan *chan = data;
- unsigned long status = iop_chan_get_status(chan);
-
- dev_err(chan->device->common.dev,
- "error ( %s%s%s%s%s%s%s)\n",
- iop_is_err_int_parity(status, chan) ? "int_parity " : "",
- iop_is_err_mcu_abort(status, chan) ? "mcu_abort " : "",
- iop_is_err_int_tabort(status, chan) ? "int_tabort " : "",
- iop_is_err_int_mabort(status, chan) ? "int_mabort " : "",
- iop_is_err_pci_tabort(status, chan) ? "pci_tabort " : "",
- iop_is_err_pci_mabort(status, chan) ? "pci_mabort " : "",
- iop_is_err_split_tx(status, chan) ? "split_tx " : "");
-
- iop_adma_device_clear_err_status(chan);
-
- BUG();
-
- return IRQ_HANDLED;
-}
-
-static void iop_adma_issue_pending(struct dma_chan *chan)
-{
- struct iop_adma_chan *iop_chan = to_iop_adma_chan(chan);
-
- if (iop_chan->pending) {
- iop_chan->pending = 0;
- iop_chan_append(iop_chan);
- }
-}
-
-/*
- * Perform a transaction to verify the HW works.
- */
-#define IOP_ADMA_TEST_SIZE 2000
-
-static int iop_adma_memcpy_self_test(struct iop_adma_device *device)
-{
- int i;
- void *src, *dest;
- dma_addr_t src_dma, dest_dma;
- struct dma_chan *dma_chan;
- dma_cookie_t cookie;
- struct dma_async_tx_descriptor *tx;
- int err = 0;
- struct iop_adma_chan *iop_chan;
-
- dev_dbg(device->common.dev, "%s\n", __func__);
-
- src = kmalloc(IOP_ADMA_TEST_SIZE, GFP_KERNEL);
- if (!src)
- return -ENOMEM;
- dest = kzalloc(IOP_ADMA_TEST_SIZE, GFP_KERNEL);
- if (!dest) {
- kfree(src);
- return -ENOMEM;
- }
-
- /* Fill in src buffer */
- for (i = 0; i < IOP_ADMA_TEST_SIZE; i++)
- ((u8 *) src)[i] = (u8)i;
-
- /* Start copy, using first DMA channel */
- dma_chan = container_of(device->common.channels.next,
- struct dma_chan,
- device_node);
- if (iop_adma_alloc_chan_resources(dma_chan) < 1) {
- err = -ENODEV;
- goto out;
- }
-
- dest_dma = dma_map_single(dma_chan->device->dev, dest,
- IOP_ADMA_TEST_SIZE, DMA_FROM_DEVICE);
- src_dma = dma_map_single(dma_chan->device->dev, src,
- IOP_ADMA_TEST_SIZE, DMA_TO_DEVICE);
- tx = iop_adma_prep_dma_memcpy(dma_chan, dest_dma, src_dma,
- IOP_ADMA_TEST_SIZE,
- DMA_PREP_INTERRUPT | DMA_CTRL_ACK);
-
- cookie = iop_adma_tx_submit(tx);
- iop_adma_issue_pending(dma_chan);
- msleep(1);
-
- if (iop_adma_status(dma_chan, cookie, NULL) !=
- DMA_COMPLETE) {
- dev_err(dma_chan->device->dev,
- "Self-test copy timed out, disabling\n");
- err = -ENODEV;
- goto free_resources;
- }
-
- iop_chan = to_iop_adma_chan(dma_chan);
- dma_sync_single_for_cpu(&iop_chan->device->pdev->dev, dest_dma,
- IOP_ADMA_TEST_SIZE, DMA_FROM_DEVICE);
- if (memcmp(src, dest, IOP_ADMA_TEST_SIZE)) {
- dev_err(dma_chan->device->dev,
- "Self-test copy failed compare, disabling\n");
- err = -ENODEV;
- goto free_resources;
- }
-
-free_resources:
- iop_adma_free_chan_resources(dma_chan);
-out:
- kfree(src);
- kfree(dest);
- return err;
-}
-
-#define IOP_ADMA_NUM_SRC_TEST 4 /* must be <= 15 */
-static int
-iop_adma_xor_val_self_test(struct iop_adma_device *device)
-{
- int i, src_idx;
- struct page *dest;
- struct page *xor_srcs[IOP_ADMA_NUM_SRC_TEST];
- struct page *zero_sum_srcs[IOP_ADMA_NUM_SRC_TEST + 1];
- dma_addr_t dma_srcs[IOP_ADMA_NUM_SRC_TEST + 1];
- dma_addr_t dest_dma;
- struct dma_async_tx_descriptor *tx;
- struct dma_chan *dma_chan;
- dma_cookie_t cookie;
- u8 cmp_byte = 0;
- u32 cmp_word;
- u32 zero_sum_result;
- int err = 0;
- struct iop_adma_chan *iop_chan;
-
- dev_dbg(device->common.dev, "%s\n", __func__);
-
- for (src_idx = 0; src_idx < IOP_ADMA_NUM_SRC_TEST; src_idx++) {
- xor_srcs[src_idx] = alloc_page(GFP_KERNEL);
- if (!xor_srcs[src_idx]) {
- while (src_idx--)
- __free_page(xor_srcs[src_idx]);
- return -ENOMEM;
- }
- }
-
- dest = alloc_page(GFP_KERNEL);
- if (!dest) {
- while (src_idx--)
- __free_page(xor_srcs[src_idx]);
- return -ENOMEM;
- }
-
- /* Fill in src buffers */
- for (src_idx = 0; src_idx < IOP_ADMA_NUM_SRC_TEST; src_idx++) {
- u8 *ptr = page_address(xor_srcs[src_idx]);
- for (i = 0; i < PAGE_SIZE; i++)
- ptr[i] = (1 << src_idx);
- }
-
- for (src_idx = 0; src_idx < IOP_ADMA_NUM_SRC_TEST; src_idx++)
- cmp_byte ^= (u8) (1 << src_idx);
-
- cmp_word = (cmp_byte << 24) | (cmp_byte << 16) |
- (cmp_byte << 8) | cmp_byte;
-
- memset(page_address(dest), 0, PAGE_SIZE);
-
- dma_chan = container_of(device->common.channels.next,
- struct dma_chan,
- device_node);
- if (iop_adma_alloc_chan_resources(dma_chan) < 1) {
- err = -ENODEV;
- goto out;
- }
-
- /* test xor */
- dest_dma = dma_map_page(dma_chan->device->dev, dest, 0,
- PAGE_SIZE, DMA_FROM_DEVICE);
- for (i = 0; i < IOP_ADMA_NUM_SRC_TEST; i++)
- dma_srcs[i] = dma_map_page(dma_chan->device->dev, xor_srcs[i],
- 0, PAGE_SIZE, DMA_TO_DEVICE);
- tx = iop_adma_prep_dma_xor(dma_chan, dest_dma, dma_srcs,
- IOP_ADMA_NUM_SRC_TEST, PAGE_SIZE,
- DMA_PREP_INTERRUPT | DMA_CTRL_ACK);
-
- cookie = iop_adma_tx_submit(tx);
- iop_adma_issue_pending(dma_chan);
- msleep(8);
-
- if (iop_adma_status(dma_chan, cookie, NULL) !=
- DMA_COMPLETE) {
- dev_err(dma_chan->device->dev,
- "Self-test xor timed out, disabling\n");
- err = -ENODEV;
- goto free_resources;
- }
-
- iop_chan = to_iop_adma_chan(dma_chan);
- dma_sync_single_for_cpu(&iop_chan->device->pdev->dev, dest_dma,
- PAGE_SIZE, DMA_FROM_DEVICE);
- for (i = 0; i < (PAGE_SIZE / sizeof(u32)); i++) {
- u32 *ptr = page_address(dest);
- if (ptr[i] != cmp_word) {
- dev_err(dma_chan->device->dev,
- "Self-test xor failed compare, disabling\n");
- err = -ENODEV;
- goto free_resources;
- }
- }
- dma_sync_single_for_device(&iop_chan->device->pdev->dev, dest_dma,
- PAGE_SIZE, DMA_TO_DEVICE);
-
- /* skip zero sum if the capability is not present */
- if (!dma_has_cap(DMA_XOR_VAL, dma_chan->device->cap_mask))
- goto free_resources;
-
- /* zero sum the sources with the destintation page */
- for (i = 0; i < IOP_ADMA_NUM_SRC_TEST; i++)
- zero_sum_srcs[i] = xor_srcs[i];
- zero_sum_srcs[i] = dest;
-
- zero_sum_result = 1;
-
- for (i = 0; i < IOP_ADMA_NUM_SRC_TEST + 1; i++)
- dma_srcs[i] = dma_map_page(dma_chan->device->dev,
- zero_sum_srcs[i], 0, PAGE_SIZE,
- DMA_TO_DEVICE);
- tx = iop_adma_prep_dma_xor_val(dma_chan, dma_srcs,
- IOP_ADMA_NUM_SRC_TEST + 1, PAGE_SIZE,
- &zero_sum_result,
- DMA_PREP_INTERRUPT | DMA_CTRL_ACK);
-
- cookie = iop_adma_tx_submit(tx);
- iop_adma_issue_pending(dma_chan);
- msleep(8);
-
- if (iop_adma_status(dma_chan, cookie, NULL) != DMA_COMPLETE) {
- dev_err(dma_chan->device->dev,
- "Self-test zero sum timed out, disabling\n");
- err = -ENODEV;
- goto free_resources;
- }
-
- if (zero_sum_result != 0) {
- dev_err(dma_chan->device->dev,
- "Self-test zero sum failed compare, disabling\n");
- err = -ENODEV;
- goto free_resources;
- }
-
- /* test for non-zero parity sum */
- zero_sum_result = 0;
- for (i = 0; i < IOP_ADMA_NUM_SRC_TEST + 1; i++)
- dma_srcs[i] = dma_map_page(dma_chan->device->dev,
- zero_sum_srcs[i], 0, PAGE_SIZE,
- DMA_TO_DEVICE);
- tx = iop_adma_prep_dma_xor_val(dma_chan, dma_srcs,
- IOP_ADMA_NUM_SRC_TEST + 1, PAGE_SIZE,
- &zero_sum_result,
- DMA_PREP_INTERRUPT | DMA_CTRL_ACK);
-
- cookie = iop_adma_tx_submit(tx);
- iop_adma_issue_pending(dma_chan);
- msleep(8);
-
- if (iop_adma_status(dma_chan, cookie, NULL) != DMA_COMPLETE) {
- dev_err(dma_chan->device->dev,
- "Self-test non-zero sum timed out, disabling\n");
- err = -ENODEV;
- goto free_resources;
- }
-
- if (zero_sum_result != 1) {
- dev_err(dma_chan->device->dev,
- "Self-test non-zero sum failed compare, disabling\n");
- err = -ENODEV;
- goto free_resources;
- }
-
-free_resources:
- iop_adma_free_chan_resources(dma_chan);
-out:
- src_idx = IOP_ADMA_NUM_SRC_TEST;
- while (src_idx--)
- __free_page(xor_srcs[src_idx]);
- __free_page(dest);
- return err;
-}
-
-#ifdef CONFIG_RAID6_PQ
-static int
-iop_adma_pq_zero_sum_self_test(struct iop_adma_device *device)
-{
- /* combined sources, software pq results, and extra hw pq results */
- struct page *pq[IOP_ADMA_NUM_SRC_TEST+2+2];
- /* ptr to the extra hw pq buffers defined above */
- struct page **pq_hw = &pq[IOP_ADMA_NUM_SRC_TEST+2];
- /* address conversion buffers (dma_map / page_address) */
- void *pq_sw[IOP_ADMA_NUM_SRC_TEST+2];
- dma_addr_t pq_src[IOP_ADMA_NUM_SRC_TEST+2];
- dma_addr_t *pq_dest = &pq_src[IOP_ADMA_NUM_SRC_TEST];
-
- int i;
- struct dma_async_tx_descriptor *tx;
- struct dma_chan *dma_chan;
- dma_cookie_t cookie;
- u32 zero_sum_result;
- int err = 0;
- struct device *dev;
-
- dev_dbg(device->common.dev, "%s\n", __func__);
-
- for (i = 0; i < ARRAY_SIZE(pq); i++) {
- pq[i] = alloc_page(GFP_KERNEL);
- if (!pq[i]) {
- while (i--)
- __free_page(pq[i]);
- return -ENOMEM;
- }
- }
-
- /* Fill in src buffers */
- for (i = 0; i < IOP_ADMA_NUM_SRC_TEST; i++) {
- pq_sw[i] = page_address(pq[i]);
- memset(pq_sw[i], 0x11111111 * (1<<i), PAGE_SIZE);
- }
- pq_sw[i] = page_address(pq[i]);
- pq_sw[i+1] = page_address(pq[i+1]);
-
- dma_chan = container_of(device->common.channels.next,
- struct dma_chan,
- device_node);
- if (iop_adma_alloc_chan_resources(dma_chan) < 1) {
- err = -ENODEV;
- goto out;
- }
-
- dev = dma_chan->device->dev;
-
- /* initialize the dests */
- memset(page_address(pq_hw[0]), 0 , PAGE_SIZE);
- memset(page_address(pq_hw[1]), 0 , PAGE_SIZE);
-
- /* test pq */
- pq_dest[0] = dma_map_page(dev, pq_hw[0], 0, PAGE_SIZE, DMA_FROM_DEVICE);
- pq_dest[1] = dma_map_page(dev, pq_hw[1], 0, PAGE_SIZE, DMA_FROM_DEVICE);
- for (i = 0; i < IOP_ADMA_NUM_SRC_TEST; i++)
- pq_src[i] = dma_map_page(dev, pq[i], 0, PAGE_SIZE,
- DMA_TO_DEVICE);
-
- tx = iop_adma_prep_dma_pq(dma_chan, pq_dest, pq_src,
- IOP_ADMA_NUM_SRC_TEST, (u8 *)raid6_gfexp,
- PAGE_SIZE,
- DMA_PREP_INTERRUPT |
- DMA_CTRL_ACK);
-
- cookie = iop_adma_tx_submit(tx);
- iop_adma_issue_pending(dma_chan);
- msleep(8);
-
- if (iop_adma_status(dma_chan, cookie, NULL) !=
- DMA_COMPLETE) {
- dev_err(dev, "Self-test pq timed out, disabling\n");
- err = -ENODEV;
- goto free_resources;
- }
-
- raid6_call.gen_syndrome(IOP_ADMA_NUM_SRC_TEST+2, PAGE_SIZE, pq_sw);
-
- if (memcmp(pq_sw[IOP_ADMA_NUM_SRC_TEST],
- page_address(pq_hw[0]), PAGE_SIZE) != 0) {
- dev_err(dev, "Self-test p failed compare, disabling\n");
- err = -ENODEV;
- goto free_resources;
- }
- if (memcmp(pq_sw[IOP_ADMA_NUM_SRC_TEST+1],
- page_address(pq_hw[1]), PAGE_SIZE) != 0) {
- dev_err(dev, "Self-test q failed compare, disabling\n");
- err = -ENODEV;
- goto free_resources;
- }
-
- /* test correct zero sum using the software generated pq values */
- for (i = 0; i < IOP_ADMA_NUM_SRC_TEST + 2; i++)
- pq_src[i] = dma_map_page(dev, pq[i], 0, PAGE_SIZE,
- DMA_TO_DEVICE);
-
- zero_sum_result = ~0;
- tx = iop_adma_prep_dma_pq_val(dma_chan, &pq_src[IOP_ADMA_NUM_SRC_TEST],
- pq_src, IOP_ADMA_NUM_SRC_TEST,
- raid6_gfexp, PAGE_SIZE, &zero_sum_result,
- DMA_PREP_INTERRUPT|DMA_CTRL_ACK);
-
- cookie = iop_adma_tx_submit(tx);
- iop_adma_issue_pending(dma_chan);
- msleep(8);
-
- if (iop_adma_status(dma_chan, cookie, NULL) !=
- DMA_COMPLETE) {
- dev_err(dev, "Self-test pq-zero-sum timed out, disabling\n");
- err = -ENODEV;
- goto free_resources;
- }
-
- if (zero_sum_result != 0) {
- dev_err(dev, "Self-test pq-zero-sum failed to validate: %x\n",
- zero_sum_result);
- err = -ENODEV;
- goto free_resources;
- }
-
- /* test incorrect zero sum */
- i = IOP_ADMA_NUM_SRC_TEST;
- memset(pq_sw[i] + 100, 0, 100);
- memset(pq_sw[i+1] + 200, 0, 200);
- for (i = 0; i < IOP_ADMA_NUM_SRC_TEST + 2; i++)
- pq_src[i] = dma_map_page(dev, pq[i], 0, PAGE_SIZE,
- DMA_TO_DEVICE);
-
- zero_sum_result = 0;
- tx = iop_adma_prep_dma_pq_val(dma_chan, &pq_src[IOP_ADMA_NUM_SRC_TEST],
- pq_src, IOP_ADMA_NUM_SRC_TEST,
- raid6_gfexp, PAGE_SIZE, &zero_sum_result,
- DMA_PREP_INTERRUPT|DMA_CTRL_ACK);
-
- cookie = iop_adma_tx_submit(tx);
- iop_adma_issue_pending(dma_chan);
- msleep(8);
-
- if (iop_adma_status(dma_chan, cookie, NULL) !=
- DMA_COMPLETE) {
- dev_err(dev, "Self-test !pq-zero-sum timed out, disabling\n");
- err = -ENODEV;
- goto free_resources;
- }
-
- if (zero_sum_result != (SUM_CHECK_P_RESULT | SUM_CHECK_Q_RESULT)) {
- dev_err(dev, "Self-test !pq-zero-sum failed to validate: %x\n",
- zero_sum_result);
- err = -ENODEV;
- goto free_resources;
- }
-
-free_resources:
- iop_adma_free_chan_resources(dma_chan);
-out:
- i = ARRAY_SIZE(pq);
- while (i--)
- __free_page(pq[i]);
- return err;
-}
-#endif
-
-static int iop_adma_remove(struct platform_device *dev)
-{
- struct iop_adma_device *device = platform_get_drvdata(dev);
- struct dma_chan *chan, *_chan;
- struct iop_adma_chan *iop_chan;
- struct iop_adma_platform_data *plat_data = dev_get_platdata(&dev->dev);
-
- dma_async_device_unregister(&device->common);
-
- dma_free_coherent(&dev->dev, plat_data->pool_size,
- device->dma_desc_pool_virt, device->dma_desc_pool);
-
- list_for_each_entry_safe(chan, _chan, &device->common.channels,
- device_node) {
- iop_chan = to_iop_adma_chan(chan);
- list_del(&chan->device_node);
- kfree(iop_chan);
- }
- kfree(device);
-
- return 0;
-}
-
-static int iop_adma_probe(struct platform_device *pdev)
-{
- struct resource *res;
- int ret = 0, i;
- struct iop_adma_device *adev;
- struct iop_adma_chan *iop_chan;
- struct dma_device *dma_dev;
- struct iop_adma_platform_data *plat_data = dev_get_platdata(&pdev->dev);
-
- res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
- if (!res)
- return -ENODEV;
-
- if (!devm_request_mem_region(&pdev->dev, res->start,
- resource_size(res), pdev->name))
- return -EBUSY;
-
- adev = kzalloc(sizeof(*adev), GFP_KERNEL);
- if (!adev)
- return -ENOMEM;
- dma_dev = &adev->common;
-
- /* allocate coherent memory for hardware descriptors
- * note: writecombine gives slightly better performance, but
- * requires that we explicitly flush the writes
- */
- adev->dma_desc_pool_virt = dma_alloc_wc(&pdev->dev,
- plat_data->pool_size,
- &adev->dma_desc_pool,
- GFP_KERNEL);
- if (!adev->dma_desc_pool_virt) {
- ret = -ENOMEM;
- goto err_free_adev;
- }
-
- dev_dbg(&pdev->dev, "%s: allocated descriptor pool virt %p phys %pad\n",
- __func__, adev->dma_desc_pool_virt, &adev->dma_desc_pool);
-
- adev->id = plat_data->hw_id;
-
- /* discover transaction capabilites from the platform data */
- dma_dev->cap_mask = plat_data->cap_mask;
-
- adev->pdev = pdev;
- platform_set_drvdata(pdev, adev);
-
- INIT_LIST_HEAD(&dma_dev->channels);
-
- /* set base routines */
- dma_dev->device_alloc_chan_resources = iop_adma_alloc_chan_resources;
- dma_dev->device_free_chan_resources = iop_adma_free_chan_resources;
- dma_dev->device_tx_status = iop_adma_status;
- dma_dev->device_issue_pending = iop_adma_issue_pending;
- dma_dev->dev = &pdev->dev;
-
- /* set prep routines based on capability */
- if (dma_has_cap(DMA_MEMCPY, dma_dev->cap_mask))
- dma_dev->device_prep_dma_memcpy = iop_adma_prep_dma_memcpy;
- if (dma_has_cap(DMA_XOR, dma_dev->cap_mask)) {
- dma_dev->max_xor = iop_adma_get_max_xor();
- dma_dev->device_prep_dma_xor = iop_adma_prep_dma_xor;
- }
- if (dma_has_cap(DMA_XOR_VAL, dma_dev->cap_mask))
- dma_dev->device_prep_dma_xor_val =
- iop_adma_prep_dma_xor_val;
- if (dma_has_cap(DMA_PQ, dma_dev->cap_mask)) {
- dma_set_maxpq(dma_dev, iop_adma_get_max_pq(), 0);
- dma_dev->device_prep_dma_pq = iop_adma_prep_dma_pq;
- }
- if (dma_has_cap(DMA_PQ_VAL, dma_dev->cap_mask))
- dma_dev->device_prep_dma_pq_val =
- iop_adma_prep_dma_pq_val;
- if (dma_has_cap(DMA_INTERRUPT, dma_dev->cap_mask))
- dma_dev->device_prep_dma_interrupt =
- iop_adma_prep_dma_interrupt;
-
- iop_chan = kzalloc(sizeof(*iop_chan), GFP_KERNEL);
- if (!iop_chan) {
- ret = -ENOMEM;
- goto err_free_dma;
- }
- iop_chan->device = adev;
-
- iop_chan->mmr_base = devm_ioremap(&pdev->dev, res->start,
- resource_size(res));
- if (!iop_chan->mmr_base) {
- ret = -ENOMEM;
- goto err_free_iop_chan;
- }
- tasklet_setup(&iop_chan->irq_tasklet, iop_adma_tasklet);
-
- /* clear errors before enabling interrupts */
- iop_adma_device_clear_err_status(iop_chan);
-
- for (i = 0; i < 3; i++) {
- static const irq_handler_t handler[] = {
- iop_adma_eot_handler,
- iop_adma_eoc_handler,
- iop_adma_err_handler
- };
- int irq = platform_get_irq(pdev, i);
- if (irq < 0) {
- ret = -ENXIO;
- goto err_free_iop_chan;
- } else {
- ret = devm_request_irq(&pdev->dev, irq,
- handler[i], 0, pdev->name, iop_chan);
- if (ret)
- goto err_free_iop_chan;
- }
- }
-
- spin_lock_init(&iop_chan->lock);
- INIT_LIST_HEAD(&iop_chan->chain);
- INIT_LIST_HEAD(&iop_chan->all_slots);
- iop_chan->common.device = dma_dev;
- dma_cookie_init(&iop_chan->common);
- list_add_tail(&iop_chan->common.device_node, &dma_dev->channels);
-
- if (dma_has_cap(DMA_MEMCPY, dma_dev->cap_mask)) {
- ret = iop_adma_memcpy_self_test(adev);
- dev_dbg(&pdev->dev, "memcpy self test returned %d\n", ret);
- if (ret)
- goto err_free_iop_chan;
- }
-
- if (dma_has_cap(DMA_XOR, dma_dev->cap_mask)) {
- ret = iop_adma_xor_val_self_test(adev);
- dev_dbg(&pdev->dev, "xor self test returned %d\n", ret);
- if (ret)
- goto err_free_iop_chan;
- }
-
- if (dma_has_cap(DMA_PQ, dma_dev->cap_mask) &&
- dma_has_cap(DMA_PQ_VAL, dma_dev->cap_mask)) {
- #ifdef CONFIG_RAID6_PQ
- ret = iop_adma_pq_zero_sum_self_test(adev);
- dev_dbg(&pdev->dev, "pq self test returned %d\n", ret);
- #else
- /* can not test raid6, so do not publish capability */
- dma_cap_clear(DMA_PQ, dma_dev->cap_mask);
- dma_cap_clear(DMA_PQ_VAL, dma_dev->cap_mask);
- ret = 0;
- #endif
- if (ret)
- goto err_free_iop_chan;
- }
-
- dev_info(&pdev->dev, "Intel(R) IOP: ( %s%s%s%s%s%s)\n",
- dma_has_cap(DMA_PQ, dma_dev->cap_mask) ? "pq " : "",
- dma_has_cap(DMA_PQ_VAL, dma_dev->cap_mask) ? "pq_val " : "",
- dma_has_cap(DMA_XOR, dma_dev->cap_mask) ? "xor " : "",
- dma_has_cap(DMA_XOR_VAL, dma_dev->cap_mask) ? "xor_val " : "",
- dma_has_cap(DMA_MEMCPY, dma_dev->cap_mask) ? "cpy " : "",
- dma_has_cap(DMA_INTERRUPT, dma_dev->cap_mask) ? "intr " : "");
-
- dma_async_device_register(dma_dev);
- goto out;
-
- err_free_iop_chan:
- kfree(iop_chan);
- err_free_dma:
- dma_free_coherent(&adev->pdev->dev, plat_data->pool_size,
- adev->dma_desc_pool_virt, adev->dma_desc_pool);
- err_free_adev:
- kfree(adev);
- out:
- return ret;
-}
-
-static void iop_chan_start_null_memcpy(struct iop_adma_chan *iop_chan)
-{
- struct iop_adma_desc_slot *sw_desc, *grp_start;
- dma_cookie_t cookie;
- int slot_cnt, slots_per_op;
-
- dev_dbg(iop_chan->device->common.dev, "%s\n", __func__);
-
- spin_lock_bh(&iop_chan->lock);
- slot_cnt = iop_chan_memcpy_slot_count(0, &slots_per_op);
- sw_desc = iop_adma_alloc_slots(iop_chan, slot_cnt, slots_per_op);
- if (sw_desc) {
- grp_start = sw_desc->group_head;
-
- list_splice_init(&sw_desc->tx_list, &iop_chan->chain);
- async_tx_ack(&sw_desc->async_tx);
- iop_desc_init_memcpy(grp_start, 0);
- iop_desc_set_byte_count(grp_start, iop_chan, 0);
- iop_desc_set_dest_addr(grp_start, iop_chan, 0);
- iop_desc_set_memcpy_src_addr(grp_start, 0);
-
- cookie = dma_cookie_assign(&sw_desc->async_tx);
-
- /* initialize the completed cookie to be less than
- * the most recently used cookie
- */
- iop_chan->common.completed_cookie = cookie - 1;
-
- /* channel should not be busy */
- BUG_ON(iop_chan_is_busy(iop_chan));
-
- /* clear any prior error-status bits */
- iop_adma_device_clear_err_status(iop_chan);
-
- /* disable operation */
- iop_chan_disable(iop_chan);
-
- /* set the descriptor address */
- iop_chan_set_next_descriptor(iop_chan, sw_desc->async_tx.phys);
-
- /* 1/ don't add pre-chained descriptors
- * 2/ dummy read to flush next_desc write
- */
- BUG_ON(iop_desc_get_next_desc(sw_desc));
-
- /* run the descriptor */
- iop_chan_enable(iop_chan);
- } else
- dev_err(iop_chan->device->common.dev,
- "failed to allocate null descriptor\n");
- spin_unlock_bh(&iop_chan->lock);
-}
-
-static void iop_chan_start_null_xor(struct iop_adma_chan *iop_chan)
-{
- struct iop_adma_desc_slot *sw_desc, *grp_start;
- dma_cookie_t cookie;
- int slot_cnt, slots_per_op;
-
- dev_dbg(iop_chan->device->common.dev, "%s\n", __func__);
-
- spin_lock_bh(&iop_chan->lock);
- slot_cnt = iop_chan_xor_slot_count(0, 2, &slots_per_op);
- sw_desc = iop_adma_alloc_slots(iop_chan, slot_cnt, slots_per_op);
- if (sw_desc) {
- grp_start = sw_desc->group_head;
- list_splice_init(&sw_desc->tx_list, &iop_chan->chain);
- async_tx_ack(&sw_desc->async_tx);
- iop_desc_init_null_xor(grp_start, 2, 0);
- iop_desc_set_byte_count(grp_start, iop_chan, 0);
- iop_desc_set_dest_addr(grp_start, iop_chan, 0);
- iop_desc_set_xor_src_addr(grp_start, 0, 0);
- iop_desc_set_xor_src_addr(grp_start, 1, 0);
-
- cookie = dma_cookie_assign(&sw_desc->async_tx);
-
- /* initialize the completed cookie to be less than
- * the most recently used cookie
- */
- iop_chan->common.completed_cookie = cookie - 1;
-
- /* channel should not be busy */
- BUG_ON(iop_chan_is_busy(iop_chan));
-
- /* clear any prior error-status bits */
- iop_adma_device_clear_err_status(iop_chan);
-
- /* disable operation */
- iop_chan_disable(iop_chan);
-
- /* set the descriptor address */
- iop_chan_set_next_descriptor(iop_chan, sw_desc->async_tx.phys);
-
- /* 1/ don't add pre-chained descriptors
- * 2/ dummy read to flush next_desc write
- */
- BUG_ON(iop_desc_get_next_desc(sw_desc));
-
- /* run the descriptor */
- iop_chan_enable(iop_chan);
- } else
- dev_err(iop_chan->device->common.dev,
- "failed to allocate null descriptor\n");
- spin_unlock_bh(&iop_chan->lock);
-}
-
-static struct platform_driver iop_adma_driver = {
- .probe = iop_adma_probe,
- .remove = iop_adma_remove,
- .driver = {
- .name = "iop-adma",
- },
-};
-
-module_platform_driver(iop_adma_driver);
-
-MODULE_AUTHOR("Intel Corporation");
-MODULE_DESCRIPTION("IOP ADMA Engine Driver");
-MODULE_LICENSE("GPL");
-MODULE_ALIAS("platform:iop-adma");
diff --git a/drivers/dma/iop-adma.h b/drivers/dma/iop-adma.h
deleted file mode 100644
index d44eabb..0000000
--- a/drivers/dma/iop-adma.h
+++ /dev/null
@@ -1,914 +0,0 @@
-/* SPDX-License-Identifier: GPL-2.0-only */
-/*
- * Copyright © 2006, Intel Corporation.
- */
-#ifndef _ADMA_H
-#define _ADMA_H
-#include <linux/types.h>
-#include <linux/io.h>
-#include <linux/platform_data/dma-iop32x.h>
-
-/* Memory copy units */
-#define DMA_CCR(chan) (chan->mmr_base + 0x0)
-#define DMA_CSR(chan) (chan->mmr_base + 0x4)
-#define DMA_DAR(chan) (chan->mmr_base + 0xc)
-#define DMA_NDAR(chan) (chan->mmr_base + 0x10)
-#define DMA_PADR(chan) (chan->mmr_base + 0x14)
-#define DMA_PUADR(chan) (chan->mmr_base + 0x18)
-#define DMA_LADR(chan) (chan->mmr_base + 0x1c)
-#define DMA_BCR(chan) (chan->mmr_base + 0x20)
-#define DMA_DCR(chan) (chan->mmr_base + 0x24)
-
-/* Application accelerator unit */
-#define AAU_ACR(chan) (chan->mmr_base + 0x0)
-#define AAU_ASR(chan) (chan->mmr_base + 0x4)
-#define AAU_ADAR(chan) (chan->mmr_base + 0x8)
-#define AAU_ANDAR(chan) (chan->mmr_base + 0xc)
-#define AAU_SAR(src, chan) (chan->mmr_base + (0x10 + ((src) << 2)))
-#define AAU_DAR(chan) (chan->mmr_base + 0x20)
-#define AAU_ABCR(chan) (chan->mmr_base + 0x24)
-#define AAU_ADCR(chan) (chan->mmr_base + 0x28)
-#define AAU_SAR_EDCR(src_edc) (chan->mmr_base + (0x02c + ((src_edc-4) << 2)))
-#define AAU_EDCR0_IDX 8
-#define AAU_EDCR1_IDX 17
-#define AAU_EDCR2_IDX 26
-
-struct iop3xx_aau_desc_ctrl {
- unsigned int int_en:1;
- unsigned int blk1_cmd_ctrl:3;
- unsigned int blk2_cmd_ctrl:3;
- unsigned int blk3_cmd_ctrl:3;
- unsigned int blk4_cmd_ctrl:3;
- unsigned int blk5_cmd_ctrl:3;
- unsigned int blk6_cmd_ctrl:3;
- unsigned int blk7_cmd_ctrl:3;
- unsigned int blk8_cmd_ctrl:3;
- unsigned int blk_ctrl:2;
- unsigned int dual_xor_en:1;
- unsigned int tx_complete:1;
- unsigned int zero_result_err:1;
- unsigned int zero_result_en:1;
- unsigned int dest_write_en:1;
-};
-
-struct iop3xx_aau_e_desc_ctrl {
- unsigned int reserved:1;
- unsigned int blk1_cmd_ctrl:3;
- unsigned int blk2_cmd_ctrl:3;
- unsigned int blk3_cmd_ctrl:3;
- unsigned int blk4_cmd_ctrl:3;
- unsigned int blk5_cmd_ctrl:3;
- unsigned int blk6_cmd_ctrl:3;
- unsigned int blk7_cmd_ctrl:3;
- unsigned int blk8_cmd_ctrl:3;
- unsigned int reserved2:7;
-};
-
-struct iop3xx_dma_desc_ctrl {
- unsigned int pci_transaction:4;
- unsigned int int_en:1;
- unsigned int dac_cycle_en:1;
- unsigned int mem_to_mem_en:1;
- unsigned int crc_data_tx_en:1;
- unsigned int crc_gen_en:1;
- unsigned int crc_seed_dis:1;
- unsigned int reserved:21;
- unsigned int crc_tx_complete:1;
-};
-
-struct iop3xx_desc_dma {
- u32 next_desc;
- union {
- u32 pci_src_addr;
- u32 pci_dest_addr;
- u32 src_addr;
- };
- union {
- u32 upper_pci_src_addr;
- u32 upper_pci_dest_addr;
- };
- union {
- u32 local_pci_src_addr;
- u32 local_pci_dest_addr;
- u32 dest_addr;
- };
- u32 byte_count;
- union {
- u32 desc_ctrl;
- struct iop3xx_dma_desc_ctrl desc_ctrl_field;
- };
- u32 crc_addr;
-};
-
-struct iop3xx_desc_aau {
- u32 next_desc;
- u32 src[4];
- u32 dest_addr;
- u32 byte_count;
- union {
- u32 desc_ctrl;
- struct iop3xx_aau_desc_ctrl desc_ctrl_field;
- };
- union {
- u32 src_addr;
- u32 e_desc_ctrl;
- struct iop3xx_aau_e_desc_ctrl e_desc_ctrl_field;
- } src_edc[31];
-};
-
-struct iop3xx_aau_gfmr {
- unsigned int gfmr1:8;
- unsigned int gfmr2:8;
- unsigned int gfmr3:8;
- unsigned int gfmr4:8;
-};
-
-struct iop3xx_desc_pq_xor {
- u32 next_desc;
- u32 src[3];
- union {
- u32 data_mult1;
- struct iop3xx_aau_gfmr data_mult1_field;
- };
- u32 dest_addr;
- u32 byte_count;
- union {
- u32 desc_ctrl;
- struct iop3xx_aau_desc_ctrl desc_ctrl_field;
- };
- union {
- u32 src_addr;
- u32 e_desc_ctrl;
- struct iop3xx_aau_e_desc_ctrl e_desc_ctrl_field;
- u32 data_multiplier;
- struct iop3xx_aau_gfmr data_mult_field;
- u32 reserved;
- } src_edc_gfmr[19];
-};
-
-struct iop3xx_desc_dual_xor {
- u32 next_desc;
- u32 src0_addr;
- u32 src1_addr;
- u32 h_src_addr;
- u32 d_src_addr;
- u32 h_dest_addr;
- u32 byte_count;
- union {
- u32 desc_ctrl;
- struct iop3xx_aau_desc_ctrl desc_ctrl_field;
- };
- u32 d_dest_addr;
-};
-
-union iop3xx_desc {
- struct iop3xx_desc_aau *aau;
- struct iop3xx_desc_dma *dma;
- struct iop3xx_desc_pq_xor *pq_xor;
- struct iop3xx_desc_dual_xor *dual_xor;
- void *ptr;
-};
-
-/* No support for p+q operations */
-static inline int
-iop_chan_pq_slot_count(size_t len, int src_cnt, int *slots_per_op)
-{
- BUG();
- return 0;
-}
-
-static inline void
-iop_desc_init_pq(struct iop_adma_desc_slot *desc, int src_cnt,
- unsigned long flags)
-{
- BUG();
-}
-
-static inline void
-iop_desc_set_pq_addr(struct iop_adma_desc_slot *desc, dma_addr_t *addr)
-{
- BUG();
-}
-
-static inline void
-iop_desc_set_pq_src_addr(struct iop_adma_desc_slot *desc, int src_idx,
- dma_addr_t addr, unsigned char coef)
-{
- BUG();
-}
-
-static inline int
-iop_chan_pq_zero_sum_slot_count(size_t len, int src_cnt, int *slots_per_op)
-{
- BUG();
- return 0;
-}
-
-static inline void
-iop_desc_init_pq_zero_sum(struct iop_adma_desc_slot *desc, int src_cnt,
- unsigned long flags)
-{
- BUG();
-}
-
-static inline void
-iop_desc_set_pq_zero_sum_byte_count(struct iop_adma_desc_slot *desc, u32 len)
-{
- BUG();
-}
-
-#define iop_desc_set_pq_zero_sum_src_addr iop_desc_set_pq_src_addr
-
-static inline void
-iop_desc_set_pq_zero_sum_addr(struct iop_adma_desc_slot *desc, int pq_idx,
- dma_addr_t *src)
-{
- BUG();
-}
-
-static inline int iop_adma_get_max_xor(void)
-{
- return 32;
-}
-
-static inline int iop_adma_get_max_pq(void)
-{
- BUG();
- return 0;
-}
-
-static inline u32 iop_chan_get_current_descriptor(struct iop_adma_chan *chan)
-{
- int id = chan->device->id;
-
- switch (id) {
- case DMA0_ID:
- case DMA1_ID:
- return __raw_readl(DMA_DAR(chan));
- case AAU_ID:
- return __raw_readl(AAU_ADAR(chan));
- default:
- BUG();
- }
- return 0;
-}
-
-static inline void iop_chan_set_next_descriptor(struct iop_adma_chan *chan,
- u32 next_desc_addr)
-{
- int id = chan->device->id;
-
- switch (id) {
- case DMA0_ID:
- case DMA1_ID:
- __raw_writel(next_desc_addr, DMA_NDAR(chan));
- break;
- case AAU_ID:
- __raw_writel(next_desc_addr, AAU_ANDAR(chan));
- break;
- }
-
-}
-
-#define IOP_ADMA_STATUS_BUSY (1 << 10)
-#define IOP_ADMA_ZERO_SUM_MAX_BYTE_COUNT (1024)
-#define IOP_ADMA_XOR_MAX_BYTE_COUNT (16 * 1024 * 1024)
-#define IOP_ADMA_MAX_BYTE_COUNT (16 * 1024 * 1024)
-
-static inline int iop_chan_is_busy(struct iop_adma_chan *chan)
-{
- u32 status = __raw_readl(DMA_CSR(chan));
- return (status & IOP_ADMA_STATUS_BUSY) ? 1 : 0;
-}
-
-static inline int iop_desc_is_aligned(struct iop_adma_desc_slot *desc,
- int num_slots)
-{
- /* num_slots will only ever be 1, 2, 4, or 8 */
- return (desc->idx & (num_slots - 1)) ? 0 : 1;
-}
-
-/* to do: support large (i.e. > hw max) buffer sizes */
-static inline int iop_chan_memcpy_slot_count(size_t len, int *slots_per_op)
-{
- *slots_per_op = 1;
- return 1;
-}
-
-/* to do: support large (i.e. > hw max) buffer sizes */
-static inline int iop_chan_memset_slot_count(size_t len, int *slots_per_op)
-{
- *slots_per_op = 1;
- return 1;
-}
-
-static inline int iop3xx_aau_xor_slot_count(size_t len, int src_cnt,
- int *slots_per_op)
-{
- static const char slot_count_table[] = {
- 1, 1, 1, 1, /* 01 - 04 */
- 2, 2, 2, 2, /* 05 - 08 */
- 4, 4, 4, 4, /* 09 - 12 */
- 4, 4, 4, 4, /* 13 - 16 */
- 8, 8, 8, 8, /* 17 - 20 */
- 8, 8, 8, 8, /* 21 - 24 */
- 8, 8, 8, 8, /* 25 - 28 */
- 8, 8, 8, 8, /* 29 - 32 */
- };
- *slots_per_op = slot_count_table[src_cnt - 1];
- return *slots_per_op;
-}
-
-static inline int
-iop_chan_interrupt_slot_count(int *slots_per_op, struct iop_adma_chan *chan)
-{
- switch (chan->device->id) {
- case DMA0_ID:
- case DMA1_ID:
- return iop_chan_memcpy_slot_count(0, slots_per_op);
- case AAU_ID:
- return iop3xx_aau_xor_slot_count(0, 2, slots_per_op);
- default:
- BUG();
- }
- return 0;
-}
-
-static inline int iop_chan_xor_slot_count(size_t len, int src_cnt,
- int *slots_per_op)
-{
- int slot_cnt = iop3xx_aau_xor_slot_count(len, src_cnt, slots_per_op);
-
- if (len <= IOP_ADMA_XOR_MAX_BYTE_COUNT)
- return slot_cnt;
-
- len -= IOP_ADMA_XOR_MAX_BYTE_COUNT;
- while (len > IOP_ADMA_XOR_MAX_BYTE_COUNT) {
- len -= IOP_ADMA_XOR_MAX_BYTE_COUNT;
- slot_cnt += *slots_per_op;
- }
-
- slot_cnt += *slots_per_op;
-
- return slot_cnt;
-}
-
-/* zero sum on iop3xx is limited to 1k at a time so it requires multiple
- * descriptors
- */
-static inline int iop_chan_zero_sum_slot_count(size_t len, int src_cnt,
- int *slots_per_op)
-{
- int slot_cnt = iop3xx_aau_xor_slot_count(len, src_cnt, slots_per_op);
-
- if (len <= IOP_ADMA_ZERO_SUM_MAX_BYTE_COUNT)
- return slot_cnt;
-
- len -= IOP_ADMA_ZERO_SUM_MAX_BYTE_COUNT;
- while (len > IOP_ADMA_ZERO_SUM_MAX_BYTE_COUNT) {
- len -= IOP_ADMA_ZERO_SUM_MAX_BYTE_COUNT;
- slot_cnt += *slots_per_op;
- }
-
- slot_cnt += *slots_per_op;
-
- return slot_cnt;
-}
-
-static inline u32 iop_desc_get_byte_count(struct iop_adma_desc_slot *desc,
- struct iop_adma_chan *chan)
-{
- union iop3xx_desc hw_desc = { .ptr = desc->hw_desc, };
-
- switch (chan->device->id) {
- case DMA0_ID:
- case DMA1_ID:
- return hw_desc.dma->byte_count;
- case AAU_ID:
- return hw_desc.aau->byte_count;
- default:
- BUG();
- }
- return 0;
-}
-
-/* translate the src_idx to a descriptor word index */
-static inline int __desc_idx(int src_idx)
-{
- static const int desc_idx_table[] = { 0, 0, 0, 0,
- 0, 1, 2, 3,
- 5, 6, 7, 8,
- 9, 10, 11, 12,
- 14, 15, 16, 17,
- 18, 19, 20, 21,
- 23, 24, 25, 26,
- 27, 28, 29, 30,
- };
-
- return desc_idx_table[src_idx];
-}
-
-static inline u32 iop_desc_get_src_addr(struct iop_adma_desc_slot *desc,
- struct iop_adma_chan *chan,
- int src_idx)
-{
- union iop3xx_desc hw_desc = { .ptr = desc->hw_desc, };
-
- switch (chan->device->id) {
- case DMA0_ID:
- case DMA1_ID:
- return hw_desc.dma->src_addr;
- case AAU_ID:
- break;
- default:
- BUG();
- }
-
- if (src_idx < 4)
- return hw_desc.aau->src[src_idx];
- else
- return hw_desc.aau->src_edc[__desc_idx(src_idx)].src_addr;
-}
-
-static inline void iop3xx_aau_desc_set_src_addr(struct iop3xx_desc_aau *hw_desc,
- int src_idx, dma_addr_t addr)
-{
- if (src_idx < 4)
- hw_desc->src[src_idx] = addr;
- else
- hw_desc->src_edc[__desc_idx(src_idx)].src_addr = addr;
-}
-
-static inline void
-iop_desc_init_memcpy(struct iop_adma_desc_slot *desc, unsigned long flags)
-{
- struct iop3xx_desc_dma *hw_desc = desc->hw_desc;
- union {
- u32 value;
- struct iop3xx_dma_desc_ctrl field;
- } u_desc_ctrl;
-
- u_desc_ctrl.value = 0;
- u_desc_ctrl.field.mem_to_mem_en = 1;
- u_desc_ctrl.field.pci_transaction = 0xe; /* memory read block */
- u_desc_ctrl.field.int_en = flags & DMA_PREP_INTERRUPT;
- hw_desc->desc_ctrl = u_desc_ctrl.value;
- hw_desc->upper_pci_src_addr = 0;
- hw_desc->crc_addr = 0;
-}
-
-static inline void
-iop_desc_init_memset(struct iop_adma_desc_slot *desc, unsigned long flags)
-{
- struct iop3xx_desc_aau *hw_desc = desc->hw_desc;
- union {
- u32 value;
- struct iop3xx_aau_desc_ctrl field;
- } u_desc_ctrl;
-
- u_desc_ctrl.value = 0;
- u_desc_ctrl.field.blk1_cmd_ctrl = 0x2; /* memory block fill */
- u_desc_ctrl.field.dest_write_en = 1;
- u_desc_ctrl.field.int_en = flags & DMA_PREP_INTERRUPT;
- hw_desc->desc_ctrl = u_desc_ctrl.value;
-}
-
-static inline u32
-iop3xx_desc_init_xor(struct iop3xx_desc_aau *hw_desc, int src_cnt,
- unsigned long flags)
-{
- int i, shift;
- u32 edcr;
- union {
- u32 value;
- struct iop3xx_aau_desc_ctrl field;
- } u_desc_ctrl;
-
- u_desc_ctrl.value = 0;
- switch (src_cnt) {
- case 25 ... 32:
- u_desc_ctrl.field.blk_ctrl = 0x3; /* use EDCR[2:0] */
- edcr = 0;
- shift = 1;
- for (i = 24; i < src_cnt; i++) {
- edcr |= (1 << shift);
- shift += 3;
- }
- hw_desc->src_edc[AAU_EDCR2_IDX].e_desc_ctrl = edcr;
- src_cnt = 24;
- fallthrough;
- case 17 ... 24:
- if (!u_desc_ctrl.field.blk_ctrl) {
- hw_desc->src_edc[AAU_EDCR2_IDX].e_desc_ctrl = 0;
- u_desc_ctrl.field.blk_ctrl = 0x3; /* use EDCR[2:0] */
- }
- edcr = 0;
- shift = 1;
- for (i = 16; i < src_cnt; i++) {
- edcr |= (1 << shift);
- shift += 3;
- }
- hw_desc->src_edc[AAU_EDCR1_IDX].e_desc_ctrl = edcr;
- src_cnt = 16;
- fallthrough;
- case 9 ... 16:
- if (!u_desc_ctrl.field.blk_ctrl)
- u_desc_ctrl.field.blk_ctrl = 0x2; /* use EDCR0 */
- edcr = 0;
- shift = 1;
- for (i = 8; i < src_cnt; i++) {
- edcr |= (1 << shift);
- shift += 3;
- }
- hw_desc->src_edc[AAU_EDCR0_IDX].e_desc_ctrl = edcr;
- src_cnt = 8;
- fallthrough;
- case 2 ... 8:
- shift = 1;
- for (i = 0; i < src_cnt; i++) {
- u_desc_ctrl.value |= (1 << shift);
- shift += 3;
- }
-
- if (!u_desc_ctrl.field.blk_ctrl && src_cnt > 4)
- u_desc_ctrl.field.blk_ctrl = 0x1; /* use mini-desc */
- }
-
- u_desc_ctrl.field.dest_write_en = 1;
- u_desc_ctrl.field.blk1_cmd_ctrl = 0x7; /* direct fill */
- u_desc_ctrl.field.int_en = flags & DMA_PREP_INTERRUPT;
- hw_desc->desc_ctrl = u_desc_ctrl.value;
-
- return u_desc_ctrl.value;
-}
-
-static inline void
-iop_desc_init_xor(struct iop_adma_desc_slot *desc, int src_cnt,
- unsigned long flags)
-{
- iop3xx_desc_init_xor(desc->hw_desc, src_cnt, flags);
-}
-
-/* return the number of operations */
-static inline int
-iop_desc_init_zero_sum(struct iop_adma_desc_slot *desc, int src_cnt,
- unsigned long flags)
-{
- int slot_cnt = desc->slot_cnt, slots_per_op = desc->slots_per_op;
- struct iop3xx_desc_aau *hw_desc, *prev_hw_desc, *iter;
- union {
- u32 value;
- struct iop3xx_aau_desc_ctrl field;
- } u_desc_ctrl;
- int i, j;
-
- hw_desc = desc->hw_desc;
-
- for (i = 0, j = 0; (slot_cnt -= slots_per_op) >= 0;
- i += slots_per_op, j++) {
- iter = iop_hw_desc_slot_idx(hw_desc, i);
- u_desc_ctrl.value = iop3xx_desc_init_xor(iter, src_cnt, flags);
- u_desc_ctrl.field.dest_write_en = 0;
- u_desc_ctrl.field.zero_result_en = 1;
- u_desc_ctrl.field.int_en = flags & DMA_PREP_INTERRUPT;
- iter->desc_ctrl = u_desc_ctrl.value;
-
- /* for the subsequent descriptors preserve the store queue
- * and chain them together
- */
- if (i) {
- prev_hw_desc =
- iop_hw_desc_slot_idx(hw_desc, i - slots_per_op);
- prev_hw_desc->next_desc =
- (u32) (desc->async_tx.phys + (i << 5));
- }
- }
-
- return j;
-}
-
-static inline void
-iop_desc_init_null_xor(struct iop_adma_desc_slot *desc, int src_cnt,
- unsigned long flags)
-{
- struct iop3xx_desc_aau *hw_desc = desc->hw_desc;
- union {
- u32 value;
- struct iop3xx_aau_desc_ctrl field;
- } u_desc_ctrl;
-
- u_desc_ctrl.value = 0;
- switch (src_cnt) {
- case 25 ... 32:
- u_desc_ctrl.field.blk_ctrl = 0x3; /* use EDCR[2:0] */
- hw_desc->src_edc[AAU_EDCR2_IDX].e_desc_ctrl = 0;
- fallthrough;
- case 17 ... 24:
- if (!u_desc_ctrl.field.blk_ctrl) {
- hw_desc->src_edc[AAU_EDCR2_IDX].e_desc_ctrl = 0;
- u_desc_ctrl.field.blk_ctrl = 0x3; /* use EDCR[2:0] */
- }
- hw_desc->src_edc[AAU_EDCR1_IDX].e_desc_ctrl = 0;
- fallthrough;
- case 9 ... 16:
- if (!u_desc_ctrl.field.blk_ctrl)
- u_desc_ctrl.field.blk_ctrl = 0x2; /* use EDCR0 */
- hw_desc->src_edc[AAU_EDCR0_IDX].e_desc_ctrl = 0;
- fallthrough;
- case 1 ... 8:
- if (!u_desc_ctrl.field.blk_ctrl && src_cnt > 4)
- u_desc_ctrl.field.blk_ctrl = 0x1; /* use mini-desc */
- }
-
- u_desc_ctrl.field.dest_write_en = 0;
- u_desc_ctrl.field.int_en = flags & DMA_PREP_INTERRUPT;
- hw_desc->desc_ctrl = u_desc_ctrl.value;
-}
-
-static inline void iop_desc_set_byte_count(struct iop_adma_desc_slot *desc,
- struct iop_adma_chan *chan,
- u32 byte_count)
-{
- union iop3xx_desc hw_desc = { .ptr = desc->hw_desc, };
-
- switch (chan->device->id) {
- case DMA0_ID:
- case DMA1_ID:
- hw_desc.dma->byte_count = byte_count;
- break;
- case AAU_ID:
- hw_desc.aau->byte_count = byte_count;
- break;
- default:
- BUG();
- }
-}
-
-static inline void
-iop_desc_init_interrupt(struct iop_adma_desc_slot *desc,
- struct iop_adma_chan *chan)
-{
- union iop3xx_desc hw_desc = { .ptr = desc->hw_desc, };
-
- switch (chan->device->id) {
- case DMA0_ID:
- case DMA1_ID:
- iop_desc_init_memcpy(desc, 1);
- hw_desc.dma->byte_count = 0;
- hw_desc.dma->dest_addr = 0;
- hw_desc.dma->src_addr = 0;
- break;
- case AAU_ID:
- iop_desc_init_null_xor(desc, 2, 1);
- hw_desc.aau->byte_count = 0;
- hw_desc.aau->dest_addr = 0;
- hw_desc.aau->src[0] = 0;
- hw_desc.aau->src[1] = 0;
- break;
- default:
- BUG();
- }
-}
-
-static inline void
-iop_desc_set_zero_sum_byte_count(struct iop_adma_desc_slot *desc, u32 len)
-{
- int slots_per_op = desc->slots_per_op;
- struct iop3xx_desc_aau *hw_desc = desc->hw_desc, *iter;
- int i = 0;
-
- if (len <= IOP_ADMA_ZERO_SUM_MAX_BYTE_COUNT) {
- hw_desc->byte_count = len;
- } else {
- do {
- iter = iop_hw_desc_slot_idx(hw_desc, i);
- iter->byte_count = IOP_ADMA_ZERO_SUM_MAX_BYTE_COUNT;
- len -= IOP_ADMA_ZERO_SUM_MAX_BYTE_COUNT;
- i += slots_per_op;
- } while (len > IOP_ADMA_ZERO_SUM_MAX_BYTE_COUNT);
-
- iter = iop_hw_desc_slot_idx(hw_desc, i);
- iter->byte_count = len;
- }
-}
-
-static inline void iop_desc_set_dest_addr(struct iop_adma_desc_slot *desc,
- struct iop_adma_chan *chan,
- dma_addr_t addr)
-{
- union iop3xx_desc hw_desc = { .ptr = desc->hw_desc, };
-
- switch (chan->device->id) {
- case DMA0_ID:
- case DMA1_ID:
- hw_desc.dma->dest_addr = addr;
- break;
- case AAU_ID:
- hw_desc.aau->dest_addr = addr;
- break;
- default:
- BUG();
- }
-}
-
-static inline void iop_desc_set_memcpy_src_addr(struct iop_adma_desc_slot *desc,
- dma_addr_t addr)
-{
- struct iop3xx_desc_dma *hw_desc = desc->hw_desc;
- hw_desc->src_addr = addr;
-}
-
-static inline void
-iop_desc_set_zero_sum_src_addr(struct iop_adma_desc_slot *desc, int src_idx,
- dma_addr_t addr)
-{
-
- struct iop3xx_desc_aau *hw_desc = desc->hw_desc, *iter;
- int slot_cnt = desc->slot_cnt, slots_per_op = desc->slots_per_op;
- int i;
-
- for (i = 0; (slot_cnt -= slots_per_op) >= 0;
- i += slots_per_op, addr += IOP_ADMA_ZERO_SUM_MAX_BYTE_COUNT) {
- iter = iop_hw_desc_slot_idx(hw_desc, i);
- iop3xx_aau_desc_set_src_addr(iter, src_idx, addr);
- }
-}
-
-static inline void iop_desc_set_xor_src_addr(struct iop_adma_desc_slot *desc,
- int src_idx, dma_addr_t addr)
-{
-
- struct iop3xx_desc_aau *hw_desc = desc->hw_desc, *iter;
- int slot_cnt = desc->slot_cnt, slots_per_op = desc->slots_per_op;
- int i;
-
- for (i = 0; (slot_cnt -= slots_per_op) >= 0;
- i += slots_per_op, addr += IOP_ADMA_XOR_MAX_BYTE_COUNT) {
- iter = iop_hw_desc_slot_idx(hw_desc, i);
- iop3xx_aau_desc_set_src_addr(iter, src_idx, addr);
- }
-}
-
-static inline void iop_desc_set_next_desc(struct iop_adma_desc_slot *desc,
- u32 next_desc_addr)
-{
- /* hw_desc->next_desc is the same location for all channels */
- union iop3xx_desc hw_desc = { .ptr = desc->hw_desc, };
-
- iop_paranoia(hw_desc.dma->next_desc);
- hw_desc.dma->next_desc = next_desc_addr;
-}
-
-static inline u32 iop_desc_get_next_desc(struct iop_adma_desc_slot *desc)
-{
- /* hw_desc->next_desc is the same location for all channels */
- union iop3xx_desc hw_desc = { .ptr = desc->hw_desc, };
- return hw_desc.dma->next_desc;
-}
-
-static inline void iop_desc_clear_next_desc(struct iop_adma_desc_slot *desc)
-{
- /* hw_desc->next_desc is the same location for all channels */
- union iop3xx_desc hw_desc = { .ptr = desc->hw_desc, };
- hw_desc.dma->next_desc = 0;
-}
-
-static inline void iop_desc_set_block_fill_val(struct iop_adma_desc_slot *desc,
- u32 val)
-{
- struct iop3xx_desc_aau *hw_desc = desc->hw_desc;
- hw_desc->src[0] = val;
-}
-
-static inline enum sum_check_flags
-iop_desc_get_zero_result(struct iop_adma_desc_slot *desc)
-{
- struct iop3xx_desc_aau *hw_desc = desc->hw_desc;
- struct iop3xx_aau_desc_ctrl desc_ctrl = hw_desc->desc_ctrl_field;
-
- iop_paranoia(!(desc_ctrl.tx_complete && desc_ctrl.zero_result_en));
- return desc_ctrl.zero_result_err << SUM_CHECK_P;
-}
-
-static inline void iop_chan_append(struct iop_adma_chan *chan)
-{
- u32 dma_chan_ctrl;
-
- dma_chan_ctrl = __raw_readl(DMA_CCR(chan));
- dma_chan_ctrl |= 0x2;
- __raw_writel(dma_chan_ctrl, DMA_CCR(chan));
-}
-
-static inline u32 iop_chan_get_status(struct iop_adma_chan *chan)
-{
- return __raw_readl(DMA_CSR(chan));
-}
-
-static inline void iop_chan_disable(struct iop_adma_chan *chan)
-{
- u32 dma_chan_ctrl = __raw_readl(DMA_CCR(chan));
- dma_chan_ctrl &= ~1;
- __raw_writel(dma_chan_ctrl, DMA_CCR(chan));
-}
-
-static inline void iop_chan_enable(struct iop_adma_chan *chan)
-{
- u32 dma_chan_ctrl = __raw_readl(DMA_CCR(chan));
-
- dma_chan_ctrl |= 1;
- __raw_writel(dma_chan_ctrl, DMA_CCR(chan));
-}
-
-static inline void iop_adma_device_clear_eot_status(struct iop_adma_chan *chan)
-{
- u32 status = __raw_readl(DMA_CSR(chan));
- status &= (1 << 9);
- __raw_writel(status, DMA_CSR(chan));
-}
-
-static inline void iop_adma_device_clear_eoc_status(struct iop_adma_chan *chan)
-{
- u32 status = __raw_readl(DMA_CSR(chan));
- status &= (1 << 8);
- __raw_writel(status, DMA_CSR(chan));
-}
-
-static inline void iop_adma_device_clear_err_status(struct iop_adma_chan *chan)
-{
- u32 status = __raw_readl(DMA_CSR(chan));
-
- switch (chan->device->id) {
- case DMA0_ID:
- case DMA1_ID:
- status &= (1 << 5) | (1 << 3) | (1 << 2) | (1 << 1);
- break;
- case AAU_ID:
- status &= (1 << 5);
- break;
- default:
- BUG();
- }
-
- __raw_writel(status, DMA_CSR(chan));
-}
-
-static inline int
-iop_is_err_int_parity(unsigned long status, struct iop_adma_chan *chan)
-{
- return 0;
-}
-
-static inline int
-iop_is_err_mcu_abort(unsigned long status, struct iop_adma_chan *chan)
-{
- return 0;
-}
-
-static inline int
-iop_is_err_int_tabort(unsigned long status, struct iop_adma_chan *chan)
-{
- return 0;
-}
-
-static inline int
-iop_is_err_int_mabort(unsigned long status, struct iop_adma_chan *chan)
-{
- return test_bit(5, &status);
-}
-
-static inline int
-iop_is_err_pci_tabort(unsigned long status, struct iop_adma_chan *chan)
-{
- switch (chan->device->id) {
- case DMA0_ID:
- case DMA1_ID:
- return test_bit(2, &status);
- default:
- return 0;
- }
-}
-
-static inline int
-iop_is_err_pci_mabort(unsigned long status, struct iop_adma_chan *chan)
-{
- switch (chan->device->id) {
- case DMA0_ID:
- case DMA1_ID:
- return test_bit(3, &status);
- default:
- return 0;
- }
-}
-
-static inline int
-iop_is_err_split_tx(unsigned long status, struct iop_adma_chan *chan)
-{
- switch (chan->device->id) {
- case DMA0_ID:
- case DMA1_ID:
- return test_bit(1, &status);
- default:
- return 0;
- }
-}
-#endif /* _ADMA_H */
diff --git a/drivers/dma/qcom/gpi.c b/drivers/dma/qcom/gpi.c
index 3f56514..061add8 100644
--- a/drivers/dma/qcom/gpi.c
+++ b/drivers/dma/qcom/gpi.c
@@ -2286,9 +2286,14 @@ static int gpi_probe(struct platform_device *pdev)
}
static const struct of_device_id gpi_of_match[] = {
- { .compatible = "qcom,sc7280-gpi-dma", .data = (void *)0x10000 },
{ .compatible = "qcom,sdm845-gpi-dma", .data = (void *)0x0 },
{ .compatible = "qcom,sm6350-gpi-dma", .data = (void *)0x10000 },
+ /*
+ * Do not grow the list for compatible devices. Instead use
+ * qcom,sdm845-gpi-dma (for ee_offset = 0x0) or qcom,sm6350-gpi-dma
+ * (for ee_offset = 0x10000).
+ */
+ { .compatible = "qcom,sc7280-gpi-dma", .data = (void *)0x10000 },
{ .compatible = "qcom,sm8150-gpi-dma", .data = (void *)0x0 },
{ .compatible = "qcom,sm8250-gpi-dma", .data = (void *)0x0 },
{ .compatible = "qcom,sm8350-gpi-dma", .data = (void *)0x10000 },
diff --git a/drivers/dma/sh/shdma-arm.h b/drivers/dma/sh/shdma-arm.h
deleted file mode 100644
index 7459f9a..0000000
--- a/drivers/dma/sh/shdma-arm.h
+++ /dev/null
@@ -1,48 +0,0 @@
-/* SPDX-License-Identifier: GPL-2.0 */
-/*
- * Renesas SuperH DMA Engine support
- *
- * Copyright (C) 2013 Renesas Electronics, Inc.
- */
-
-#ifndef SHDMA_ARM_H
-#define SHDMA_ARM_H
-
-#include "shdma.h"
-
-/* Transmit sizes and respective CHCR register values */
-enum {
- XMIT_SZ_8BIT = 0,
- XMIT_SZ_16BIT = 1,
- XMIT_SZ_32BIT = 2,
- XMIT_SZ_64BIT = 7,
- XMIT_SZ_128BIT = 3,
- XMIT_SZ_256BIT = 4,
- XMIT_SZ_512BIT = 5,
-};
-
-/* log2(size / 8) - used to calculate number of transfers */
-#define SH_DMAE_TS_SHIFT { \
- [XMIT_SZ_8BIT] = 0, \
- [XMIT_SZ_16BIT] = 1, \
- [XMIT_SZ_32BIT] = 2, \
- [XMIT_SZ_64BIT] = 3, \
- [XMIT_SZ_128BIT] = 4, \
- [XMIT_SZ_256BIT] = 5, \
- [XMIT_SZ_512BIT] = 6, \
-}
-
-#define TS_LOW_BIT 0x3 /* --xx */
-#define TS_HI_BIT 0xc /* xx-- */
-
-#define TS_LOW_SHIFT (3)
-#define TS_HI_SHIFT (20 - 2) /* 2 bits for shifted low TS */
-
-#define TS_INDEX2VAL(i) \
- ((((i) & TS_LOW_BIT) << TS_LOW_SHIFT) |\
- (((i) & TS_HI_BIT) << TS_HI_SHIFT))
-
-#define CHCR_TX(xmit_sz) (DM_FIX | SM_INC | RS_ERS | TS_INDEX2VAL((xmit_sz)))
-#define CHCR_RX(xmit_sz) (DM_INC | SM_FIX | RS_ERS | TS_INDEX2VAL((xmit_sz)))
-
-#endif
diff --git a/drivers/dma/tegra186-gpc-dma.c b/drivers/dma/tegra186-gpc-dma.c
index fa9bda4..1d1180d 100644
--- a/drivers/dma/tegra186-gpc-dma.c
+++ b/drivers/dma/tegra186-gpc-dma.c
@@ -161,7 +161,10 @@
#define TEGRA_GPCDMA_BURST_COMPLETION_TIMEOUT 5000 /* 5 msec */
/* Channel base address offset from GPCDMA base address */
-#define TEGRA_GPCDMA_CHANNEL_BASE_ADD_OFFSET 0x20000
+#define TEGRA_GPCDMA_CHANNEL_BASE_ADDR_OFFSET 0x10000
+
+/* Default channel mask reserving channel0 */
+#define TEGRA_GPCDMA_DEFAULT_CHANNEL_MASK 0xfffffffe
struct tegra_dma;
struct tegra_dma_channel;
@@ -246,6 +249,7 @@ struct tegra_dma {
const struct tegra_dma_chip_data *chip_data;
unsigned long sid_m2d_reserved;
unsigned long sid_d2m_reserved;
+ u32 chan_mask;
void __iomem *base_addr;
struct device *dev;
struct dma_device dma_dev;
@@ -1288,7 +1292,7 @@ static struct dma_chan *tegra_dma_of_xlate(struct of_phandle_args *dma_spec,
}
static const struct tegra_dma_chip_data tegra186_dma_chip_data = {
- .nr_channels = 31,
+ .nr_channels = 32,
.channel_reg_size = SZ_64K,
.max_dma_count = SZ_1G,
.hw_support_pause = false,
@@ -1296,7 +1300,7 @@ static const struct tegra_dma_chip_data tegra186_dma_chip_data = {
};
static const struct tegra_dma_chip_data tegra194_dma_chip_data = {
- .nr_channels = 31,
+ .nr_channels = 32,
.channel_reg_size = SZ_64K,
.max_dma_count = SZ_1G,
.hw_support_pause = true,
@@ -1304,7 +1308,7 @@ static const struct tegra_dma_chip_data tegra194_dma_chip_data = {
};
static const struct tegra_dma_chip_data tegra234_dma_chip_data = {
- .nr_channels = 31,
+ .nr_channels = 32,
.channel_reg_size = SZ_64K,
.max_dma_count = SZ_1G,
.hw_support_pause = true,
@@ -1380,15 +1384,28 @@ static int tegra_dma_probe(struct platform_device *pdev)
}
stream_id = iommu_spec->ids[0] & 0xffff;
+ ret = device_property_read_u32(&pdev->dev, "dma-channel-mask",
+ &tdma->chan_mask);
+ if (ret) {
+ dev_warn(&pdev->dev,
+ "Missing dma-channel-mask property, using default channel mask %#x\n",
+ TEGRA_GPCDMA_DEFAULT_CHANNEL_MASK);
+ tdma->chan_mask = TEGRA_GPCDMA_DEFAULT_CHANNEL_MASK;
+ }
+
INIT_LIST_HEAD(&tdma->dma_dev.channels);
for (i = 0; i < cdata->nr_channels; i++) {
struct tegra_dma_channel *tdc = &tdma->channels[i];
+ /* Check for channel mask */
+ if (!(tdma->chan_mask & BIT(i)))
+ continue;
+
tdc->irq = platform_get_irq(pdev, i);
if (tdc->irq < 0)
return tdc->irq;
- tdc->chan_base_offset = TEGRA_GPCDMA_CHANNEL_BASE_ADD_OFFSET +
+ tdc->chan_base_offset = TEGRA_GPCDMA_CHANNEL_BASE_ADDR_OFFSET +
i * cdata->channel_reg_size;
snprintf(tdc->name, sizeof(tdc->name), "gpcdma.%d", i);
tdc->tdma = tdma;
@@ -1449,8 +1466,8 @@ static int tegra_dma_probe(struct platform_device *pdev)
return ret;
}
- dev_info(&pdev->dev, "GPC DMA driver register %d channels\n",
- cdata->nr_channels);
+ dev_info(&pdev->dev, "GPC DMA driver register %lu channels\n",
+ hweight_long(tdma->chan_mask));
return 0;
}
@@ -1473,6 +1490,9 @@ static int __maybe_unused tegra_dma_pm_suspend(struct device *dev)
for (i = 0; i < tdma->chip_data->nr_channels; i++) {
struct tegra_dma_channel *tdc = &tdma->channels[i];
+ if (!(tdma->chan_mask & BIT(i)))
+ continue;
+
if (tdc->dma_desc) {
dev_err(tdma->dev, "channel %u busy\n", i);
return -EBUSY;
@@ -1492,6 +1512,9 @@ static int __maybe_unused tegra_dma_pm_resume(struct device *dev)
for (i = 0; i < tdma->chip_data->nr_channels; i++) {
struct tegra_dma_channel *tdc = &tdma->channels[i];
+ if (!(tdma->chan_mask & BIT(i)))
+ continue;
+
tegra_dma_program_sid(tdc, tdc->stream_id);
}
diff --git a/drivers/dma/ti/Kconfig b/drivers/dma/ti/Kconfig
index 79618fa..2adc2cc 100644
--- a/drivers/dma/ti/Kconfig
+++ b/drivers/dma/ti/Kconfig
@@ -35,7 +35,7 @@
DMA engine is found on OMAP and DRA7xx parts.
config TI_K3_UDMA
- bool "Texas Instruments UDMA support"
+ tristate "Texas Instruments UDMA support"
depends on ARCH_K3
depends on TI_SCI_PROTOCOL
depends on TI_SCI_INTA_IRQCHIP
@@ -48,7 +48,7 @@
DMA engine is used in AM65x and j721e.
config TI_K3_UDMA_GLUE_LAYER
- bool "Texas Instruments UDMA Glue layer for non DMAengine users"
+ tristate "Texas Instruments UDMA Glue layer for non DMAengine users"
depends on ARCH_K3
depends on TI_K3_UDMA
help
@@ -56,7 +56,8 @@
If unsure, say N.
config TI_K3_PSIL
- bool
+ tristate
+ default TI_K3_UDMA
config TI_DMA_CROSSBAR
bool
diff --git a/drivers/dma/ti/Makefile b/drivers/dma/ti/Makefile
index d3a303f..b53d05b 100644
--- a/drivers/dma/ti/Makefile
+++ b/drivers/dma/ti/Makefile
@@ -4,11 +4,12 @@
obj-$(CONFIG_DMA_OMAP) += omap-dma.o
obj-$(CONFIG_TI_K3_UDMA) += k3-udma.o
obj-$(CONFIG_TI_K3_UDMA_GLUE_LAYER) += k3-udma-glue.o
-obj-$(CONFIG_TI_K3_PSIL) += k3-psil.o \
- k3-psil-am654.o \
- k3-psil-j721e.o \
- k3-psil-j7200.o \
- k3-psil-am64.o \
- k3-psil-j721s2.o \
- k3-psil-am62.o
+k3-psil-lib-objs := k3-psil.o \
+ k3-psil-am654.o \
+ k3-psil-j721e.o \
+ k3-psil-j7200.o \
+ k3-psil-am64.o \
+ k3-psil-j721s2.o \
+ k3-psil-am62.o
+obj-$(CONFIG_TI_K3_PSIL) += k3-psil-lib.o
obj-$(CONFIG_TI_DMA_CROSSBAR) += dma-crossbar.o
diff --git a/drivers/dma/ti/k3-psil.c b/drivers/dma/ti/k3-psil.c
index 761a384..8b6533a 100644
--- a/drivers/dma/ti/k3-psil.c
+++ b/drivers/dma/ti/k3-psil.c
@@ -5,6 +5,7 @@
*/
#include <linux/kernel.h>
+#include <linux/module.h>
#include <linux/device.h>
#include <linux/init.h>
#include <linux/mutex.h>
@@ -101,3 +102,4 @@ int psil_set_new_ep_config(struct device *dev, const char *name,
return 0;
}
EXPORT_SYMBOL_GPL(psil_set_new_ep_config);
+MODULE_LICENSE("GPL v2");
diff --git a/drivers/dma/ti/k3-udma-glue.c b/drivers/dma/ti/k3-udma-glue.c
index 4f1aeb81..789193e 100644
--- a/drivers/dma/ti/k3-udma-glue.c
+++ b/drivers/dma/ti/k3-udma-glue.c
@@ -6,6 +6,7 @@
*
*/
+#include <linux/module.h>
#include <linux/atomic.h>
#include <linux/delay.h>
#include <linux/dma-mapping.h>
@@ -1436,4 +1437,6 @@ static int __init k3_udma_glue_class_init(void)
{
return class_register(&k3_udma_glue_devclass);
}
-arch_initcall(k3_udma_glue_class_init);
+
+module_init(k3_udma_glue_class_init);
+MODULE_LICENSE("GPL v2");
diff --git a/drivers/dma/ti/k3-udma.c b/drivers/dma/ti/k3-udma.c
index 7b50819..ce8b80b 100644
--- a/drivers/dma/ti/k3-udma.c
+++ b/drivers/dma/ti/k3-udma.c
@@ -5,6 +5,7 @@
*/
#include <linux/kernel.h>
+#include <linux/module.h>
#include <linux/delay.h>
#include <linux/dmaengine.h>
#include <linux/dma-mapping.h>
@@ -4335,18 +4336,10 @@ static const struct of_device_id udma_of_match[] = {
.compatible = "ti,j721e-navss-mcu-udmap",
.data = &j721e_mcu_data,
},
- { /* Sentinel */ },
-};
-
-static const struct of_device_id bcdma_of_match[] = {
{
.compatible = "ti,am64-dmss-bcdma",
.data = &am64_bcdma_data,
},
- { /* Sentinel */ },
-};
-
-static const struct of_device_id pktdma_of_match[] = {
{
.compatible = "ti,am64-dmss-pktdma",
.data = &am64_pktdma_data,
@@ -5271,14 +5264,9 @@ static int udma_probe(struct platform_device *pdev)
return -ENOMEM;
match = of_match_node(udma_of_match, dev->of_node);
- if (!match)
- match = of_match_node(bcdma_of_match, dev->of_node);
if (!match) {
- match = of_match_node(pktdma_of_match, dev->of_node);
- if (!match) {
- dev_err(dev, "No compatible match found\n");
- return -ENODEV;
- }
+ dev_err(dev, "No compatible match found\n");
+ return -ENODEV;
}
ud->match_data = match->data;
@@ -5511,27 +5499,9 @@ static struct platform_driver udma_driver = {
},
.probe = udma_probe,
};
-builtin_platform_driver(udma_driver);
-static struct platform_driver bcdma_driver = {
- .driver = {
- .name = "ti-bcdma",
- .of_match_table = bcdma_of_match,
- .suppress_bind_attrs = true,
- },
- .probe = udma_probe,
-};
-builtin_platform_driver(bcdma_driver);
-
-static struct platform_driver pktdma_driver = {
- .driver = {
- .name = "ti-pktdma",
- .of_match_table = pktdma_of_match,
- .suppress_bind_attrs = true,
- },
- .probe = udma_probe,
-};
-builtin_platform_driver(pktdma_driver);
+module_platform_driver(udma_driver);
+MODULE_LICENSE("GPL v2");
/* Private interfaces to UDMA */
#include "k3-udma-private.c"
diff --git a/drivers/dma/xilinx/xilinx_dma.c b/drivers/dma/xilinx/xilinx_dma.c
index 8cd4e69..a8d23cd 100644
--- a/drivers/dma/xilinx/xilinx_dma.c
+++ b/drivers/dma/xilinx/xilinx_dma.c
@@ -1659,6 +1659,8 @@ static void xilinx_dma_issue_pending(struct dma_chan *dchan)
* xilinx_dma_device_config - Configure the DMA channel
* @dchan: DMA channel
* @config: channel configuration
+ *
+ * Return: 0 always.
*/
static int xilinx_dma_device_config(struct dma_chan *dchan,
struct dma_slave_config *config)
@@ -2924,7 +2926,7 @@ static int xilinx_dma_chan_probe(struct xilinx_dma_device *xdev,
* @xdev: Driver specific device structure
* @node: Device node
*
- * Return: 0 always.
+ * Return: '0' on success and failure value on error.
*/
static int xilinx_dma_child_probe(struct xilinx_dma_device *xdev,
struct device_node *node)
diff --git a/drivers/hsi/clients/ssi_protocol.c b/drivers/hsi/clients/ssi_protocol.c
index 274ad84..38e572f 100644
--- a/drivers/hsi/clients/ssi_protocol.c
+++ b/drivers/hsi/clients/ssi_protocol.c
@@ -968,7 +968,7 @@ static void ssip_xmit_work(struct work_struct *work)
ssip_xmit(cl);
}
-static int ssip_pn_xmit(struct sk_buff *skb, struct net_device *dev)
+static netdev_tx_t ssip_pn_xmit(struct sk_buff *skb, struct net_device *dev)
{
struct hsi_client *cl = to_hsi_client(dev->dev.parent);
struct ssi_protocol *ssi = hsi_client_drvdata(cl);
@@ -1027,7 +1027,7 @@ static int ssip_pn_xmit(struct sk_buff *skb, struct net_device *dev)
dev->stats.tx_packets++;
dev->stats.tx_bytes += skb->len;
- return 0;
+ return NETDEV_TX_OK;
drop2:
hsi_free_msg(msg);
drop:
@@ -1035,7 +1035,7 @@ static int ssip_pn_xmit(struct sk_buff *skb, struct net_device *dev)
inc_dropped:
dev->stats.tx_dropped++;
- return 0;
+ return NETDEV_TX_OK;
}
/* CMT reset event handler */
diff --git a/drivers/hsi/controllers/omap_ssi_core.c b/drivers/hsi/controllers/omap_ssi_core.c
index eb98201..26f2c3c 100644
--- a/drivers/hsi/controllers/omap_ssi_core.c
+++ b/drivers/hsi/controllers/omap_ssi_core.c
@@ -502,8 +502,10 @@ static int ssi_probe(struct platform_device *pd)
platform_set_drvdata(pd, ssi);
err = ssi_add_controller(ssi, pd);
- if (err < 0)
+ if (err < 0) {
+ hsi_put_controller(ssi);
goto out1;
+ }
pm_runtime_enable(&pd->dev);
@@ -536,9 +538,9 @@ static int ssi_probe(struct platform_device *pd)
device_for_each_child(&pd->dev, NULL, ssi_remove_ports);
out2:
ssi_remove_controller(ssi);
+ pm_runtime_disable(&pd->dev);
out1:
platform_set_drvdata(pd, NULL);
- pm_runtime_disable(&pd->dev);
return err;
}
@@ -629,7 +631,13 @@ static int __init ssi_init(void) {
if (ret)
return ret;
- return platform_driver_register(&ssi_port_pdriver);
+ ret = platform_driver_register(&ssi_port_pdriver);
+ if (ret) {
+ platform_driver_unregister(&ssi_pdriver);
+ return ret;
+ }
+
+ return 0;
}
module_init(ssi_init);
diff --git a/drivers/infiniband/sw/rxe/rxe_resp.c b/drivers/infiniband/sw/rxe/rxe_resp.c
index 7a60c77..c749722 100644
--- a/drivers/infiniband/sw/rxe/rxe_resp.c
+++ b/drivers/infiniband/sw/rxe/rxe_resp.c
@@ -785,53 +785,61 @@ static enum resp_states atomic_reply(struct rxe_qp *qp,
return ret;
}
-static enum resp_states atomic_write_reply(struct rxe_qp *qp,
- struct rxe_pkt_info *pkt)
+#ifdef CONFIG_64BIT
+static enum resp_states do_atomic_write(struct rxe_qp *qp,
+ struct rxe_pkt_info *pkt)
{
- u64 src, *dst;
- struct resp_res *res = qp->resp.res;
struct rxe_mr *mr = qp->resp.mr;
int payload = payload_size(pkt);
+ u64 src, *dst;
+
+ if (mr->state != RXE_MR_STATE_VALID)
+ return RESPST_ERR_RKEY_VIOLATION;
+
+ memcpy(&src, payload_addr(pkt), payload);
+
+ dst = iova_to_vaddr(mr, qp->resp.va + qp->resp.offset, payload);
+ /* check vaddr is 8 bytes aligned. */
+ if (!dst || (uintptr_t)dst & 7)
+ return RESPST_ERR_MISALIGNED_ATOMIC;
+
+ /* Do atomic write after all prior operations have completed */
+ smp_store_release(dst, src);
+
+ /* decrease resp.resid to zero */
+ qp->resp.resid -= sizeof(payload);
+
+ qp->resp.msn++;
+
+ /* next expected psn, read handles this separately */
+ qp->resp.psn = (pkt->psn + 1) & BTH_PSN_MASK;
+ qp->resp.ack_psn = qp->resp.psn;
+
+ qp->resp.opcode = pkt->opcode;
+ qp->resp.status = IB_WC_SUCCESS;
+ return RESPST_ACKNOWLEDGE;
+}
+#else
+static enum resp_states do_atomic_write(struct rxe_qp *qp,
+ struct rxe_pkt_info *pkt)
+{
+ return RESPST_ERR_UNSUPPORTED_OPCODE;
+}
+#endif /* CONFIG_64BIT */
+
+static enum resp_states atomic_write_reply(struct rxe_qp *qp,
+ struct rxe_pkt_info *pkt)
+{
+ struct resp_res *res = qp->resp.res;
if (!res) {
res = rxe_prepare_res(qp, pkt, RXE_ATOMIC_WRITE_MASK);
qp->resp.res = res;
}
- if (!res->replay) {
-#ifdef CONFIG_64BIT
- if (mr->state != RXE_MR_STATE_VALID)
- return RESPST_ERR_RKEY_VIOLATION;
-
- memcpy(&src, payload_addr(pkt), payload);
-
- dst = iova_to_vaddr(mr, qp->resp.va + qp->resp.offset, payload);
- /* check vaddr is 8 bytes aligned. */
- if (!dst || (uintptr_t)dst & 7)
- return RESPST_ERR_MISALIGNED_ATOMIC;
-
- /* Do atomic write after all prior operations have completed */
- smp_store_release(dst, src);
-
- /* decrease resp.resid to zero */
- qp->resp.resid -= sizeof(payload);
-
- qp->resp.msn++;
-
- /* next expected psn, read handles this separately */
- qp->resp.psn = (pkt->psn + 1) & BTH_PSN_MASK;
- qp->resp.ack_psn = qp->resp.psn;
-
- qp->resp.opcode = pkt->opcode;
- qp->resp.status = IB_WC_SUCCESS;
-
+ if (res->replay)
return RESPST_ACKNOWLEDGE;
-#else
- return RESPST_ERR_UNSUPPORTED_OPCODE;
-#endif /* CONFIG_64BIT */
- }
-
- return RESPST_ACKNOWLEDGE;
+ return do_atomic_write(qp, pkt);
}
static struct sk_buff *prepare_ack_packet(struct rxe_qp *qp,
diff --git a/drivers/infiniband/sw/siw/siw_qp_tx.c b/drivers/infiniband/sw/siw/siw_qp_tx.c
index 7d47b52..05052b4 100644
--- a/drivers/infiniband/sw/siw/siw_qp_tx.c
+++ b/drivers/infiniband/sw/siw/siw_qp_tx.c
@@ -29,7 +29,7 @@ static struct page *siw_get_pblpage(struct siw_mem *mem, u64 addr, int *idx)
dma_addr_t paddr = siw_pbl_get_buffer(pbl, offset, NULL, idx);
if (paddr)
- return virt_to_page((void *)paddr);
+ return virt_to_page((void *)(uintptr_t)paddr);
return NULL;
}
diff --git a/drivers/iommu/amd/init.c b/drivers/iommu/amd/init.c
index 1a2d425..467b194 100644
--- a/drivers/iommu/amd/init.c
+++ b/drivers/iommu/amd/init.c
@@ -85,7 +85,7 @@
#define LOOP_TIMEOUT 2000000
-#define IVRS_GET_SBDF_ID(seg, bus, dev, fd) (((seg & 0xffff) << 16) | ((bus & 0xff) << 8) \
+#define IVRS_GET_SBDF_ID(seg, bus, dev, fn) (((seg & 0xffff) << 16) | ((bus & 0xff) << 8) \
| ((dev & 0x1f) << 3) | (fn & 0x7))
/*
@@ -3402,18 +3402,24 @@ static int __init parse_amd_iommu_options(char *str)
static int __init parse_ivrs_ioapic(char *str)
{
u32 seg = 0, bus, dev, fn;
- int ret, id, i;
+ int id, i;
u32 devid;
- ret = sscanf(str, "[%d]=%x:%x.%x", &id, &bus, &dev, &fn);
- if (ret != 4) {
- ret = sscanf(str, "[%d]=%x:%x:%x.%x", &id, &seg, &bus, &dev, &fn);
- if (ret != 5) {
- pr_err("Invalid command line: ivrs_ioapic%s\n", str);
- return 1;
- }
+ if (sscanf(str, "=%d@%x:%x.%x", &id, &bus, &dev, &fn) == 4 ||
+ sscanf(str, "=%d@%x:%x:%x.%x", &id, &seg, &bus, &dev, &fn) == 5)
+ goto found;
+
+ if (sscanf(str, "[%d]=%x:%x.%x", &id, &bus, &dev, &fn) == 4 ||
+ sscanf(str, "[%d]=%x:%x:%x.%x", &id, &seg, &bus, &dev, &fn) == 5) {
+ pr_warn("ivrs_ioapic%s option format deprecated; use ivrs_ioapic=%d@%04x:%02x:%02x.%d instead\n",
+ str, id, seg, bus, dev, fn);
+ goto found;
}
+ pr_err("Invalid command line: ivrs_ioapic%s\n", str);
+ return 1;
+
+found:
if (early_ioapic_map_size == EARLY_MAP_SIZE) {
pr_err("Early IOAPIC map overflow - ignoring ivrs_ioapic%s\n",
str);
@@ -3434,18 +3440,24 @@ static int __init parse_ivrs_ioapic(char *str)
static int __init parse_ivrs_hpet(char *str)
{
u32 seg = 0, bus, dev, fn;
- int ret, id, i;
+ int id, i;
u32 devid;
- ret = sscanf(str, "[%d]=%x:%x.%x", &id, &bus, &dev, &fn);
- if (ret != 4) {
- ret = sscanf(str, "[%d]=%x:%x:%x.%x", &id, &seg, &bus, &dev, &fn);
- if (ret != 5) {
- pr_err("Invalid command line: ivrs_hpet%s\n", str);
- return 1;
- }
+ if (sscanf(str, "=%d@%x:%x.%x", &id, &bus, &dev, &fn) == 4 ||
+ sscanf(str, "=%d@%x:%x:%x.%x", &id, &seg, &bus, &dev, &fn) == 5)
+ goto found;
+
+ if (sscanf(str, "[%d]=%x:%x.%x", &id, &bus, &dev, &fn) == 4 ||
+ sscanf(str, "[%d]=%x:%x:%x.%x", &id, &seg, &bus, &dev, &fn) == 5) {
+ pr_warn("ivrs_hpet%s option format deprecated; use ivrs_hpet=%d@%04x:%02x:%02x.%d instead\n",
+ str, id, seg, bus, dev, fn);
+ goto found;
}
+ pr_err("Invalid command line: ivrs_hpet%s\n", str);
+ return 1;
+
+found:
if (early_hpet_map_size == EARLY_MAP_SIZE) {
pr_err("Early HPET map overflow - ignoring ivrs_hpet%s\n",
str);
@@ -3466,19 +3478,36 @@ static int __init parse_ivrs_hpet(char *str)
static int __init parse_ivrs_acpihid(char *str)
{
u32 seg = 0, bus, dev, fn;
- char *hid, *uid, *p;
+ char *hid, *uid, *p, *addr;
char acpiid[ACPIHID_UID_LEN + ACPIHID_HID_LEN] = {0};
- int ret, i;
+ int i;
- ret = sscanf(str, "[%x:%x.%x]=%s", &bus, &dev, &fn, acpiid);
- if (ret != 4) {
- ret = sscanf(str, "[%x:%x:%x.%x]=%s", &seg, &bus, &dev, &fn, acpiid);
- if (ret != 5) {
- pr_err("Invalid command line: ivrs_acpihid(%s)\n", str);
- return 1;
+ addr = strchr(str, '@');
+ if (!addr) {
+ if (sscanf(str, "[%x:%x.%x]=%s", &bus, &dev, &fn, acpiid) == 4 ||
+ sscanf(str, "[%x:%x:%x.%x]=%s", &seg, &bus, &dev, &fn, acpiid) == 5) {
+ pr_warn("ivrs_acpihid%s option format deprecated; use ivrs_acpihid=%s@%04x:%02x:%02x.%d instead\n",
+ str, acpiid, seg, bus, dev, fn);
+ goto found;
}
+ goto not_found;
}
+ /* We have the '@', make it the terminator to get just the acpiid */
+ *addr++ = 0;
+
+ if (sscanf(str, "=%s", acpiid) != 1)
+ goto not_found;
+
+ if (sscanf(addr, "%x:%x.%x", &bus, &dev, &fn) == 3 ||
+ sscanf(addr, "%x:%x:%x.%x", &seg, &bus, &dev, &fn) == 4)
+ goto found;
+
+not_found:
+ pr_err("Invalid command line: ivrs_acpihid%s\n", str);
+ return 1;
+
+found:
p = acpiid;
hid = strsep(&p, ":");
uid = p;
@@ -3488,6 +3517,13 @@ static int __init parse_ivrs_acpihid(char *str)
return 1;
}
+ /*
+ * Ignore leading zeroes after ':', so e.g., AMDI0095:00
+ * will match AMDI0095:0 in the second strcmp in acpi_dev_hid_uid_match
+ */
+ while (*uid == '0' && *(uid + 1))
+ uid++;
+
i = early_acpihid_map_size++;
memcpy(early_acpihid_map[i].hid, hid, strlen(hid));
memcpy(early_acpihid_map[i].uid, uid, strlen(uid));
diff --git a/drivers/iommu/amd/iommu.c b/drivers/iommu/amd/iommu.c
index 8d37d90..cbeaab5 100644
--- a/drivers/iommu/amd/iommu.c
+++ b/drivers/iommu/amd/iommu.c
@@ -767,7 +767,7 @@ EXPORT_SYMBOL(amd_iommu_register_ga_log_notifier);
static void iommu_poll_ga_log(struct amd_iommu *iommu)
{
- u32 head, tail, cnt = 0;
+ u32 head, tail;
if (iommu->ga_log == NULL)
return;
@@ -780,7 +780,6 @@ static void iommu_poll_ga_log(struct amd_iommu *iommu)
u64 log_entry;
raw = (u64 *)(iommu->ga_log + head);
- cnt++;
/* Avoid memcpy function-call overhead */
log_entry = *raw;
diff --git a/drivers/iommu/amd/iommu_v2.c b/drivers/iommu/amd/iommu_v2.c
index 6a1f02c..864e4ff 100644
--- a/drivers/iommu/amd/iommu_v2.c
+++ b/drivers/iommu/amd/iommu_v2.c
@@ -587,6 +587,7 @@ static int ppr_notifier(struct notifier_block *nb, unsigned long e, void *data)
put_device_state(dev_state);
out:
+ pci_dev_put(pdev);
return ret;
}
@@ -639,7 +640,9 @@ int amd_iommu_bind_pasid(struct pci_dev *pdev, u32 pasid,
if (pasid_state->mm == NULL)
goto out_free;
- mmu_notifier_register(&pasid_state->mn, mm);
+ ret = mmu_notifier_register(&pasid_state->mn, mm);
+ if (ret)
+ goto out_free;
ret = set_pasid_state(dev_state, pasid_state, pasid);
if (ret)
diff --git a/drivers/iommu/arm/arm-smmu/arm-smmu-impl.c b/drivers/iommu/arm/arm-smmu/arm-smmu-impl.c
index 658f3cc..9dc772f 100644
--- a/drivers/iommu/arm/arm-smmu/arm-smmu-impl.c
+++ b/drivers/iommu/arm/arm-smmu/arm-smmu-impl.c
@@ -136,6 +136,9 @@ int arm_mmu500_reset(struct arm_smmu_device *smmu)
reg = arm_smmu_cb_read(smmu, i, ARM_SMMU_CB_ACTLR);
reg &= ~ARM_MMU500_ACTLR_CPRE;
arm_smmu_cb_write(smmu, i, ARM_SMMU_CB_ACTLR, reg);
+ reg = arm_smmu_cb_read(smmu, i, ARM_SMMU_CB_ACTLR);
+ if (reg & ARM_MMU500_ACTLR_CPRE)
+ dev_warn_once(smmu->dev, "Failed to disable prefetcher [errata #841119 and #826419], check ACR.CACHE_LOCK\n");
}
return 0;
diff --git a/drivers/iommu/arm/arm-smmu/arm-smmu-qcom-debug.c b/drivers/iommu/arm/arm-smmu/arm-smmu-qcom-debug.c
index 6eed8e6..74e9ef2 100644
--- a/drivers/iommu/arm/arm-smmu/arm-smmu-qcom-debug.c
+++ b/drivers/iommu/arm/arm-smmu/arm-smmu-qcom-debug.c
@@ -10,16 +10,6 @@
#include "arm-smmu.h"
#include "arm-smmu-qcom.h"
-enum qcom_smmu_impl_reg_offset {
- QCOM_SMMU_TBU_PWR_STATUS,
- QCOM_SMMU_STATS_SYNC_INV_TBU_ACK,
- QCOM_SMMU_MMU2QSS_AND_SAFE_WAIT_CNTR,
-};
-
-struct qcom_smmu_config {
- const u32 *reg_offset;
-};
-
void qcom_smmu_tlb_sync_debug(struct arm_smmu_device *smmu)
{
int ret;
@@ -59,84 +49,3 @@ void qcom_smmu_tlb_sync_debug(struct arm_smmu_device *smmu)
tbu_pwr_status, sync_inv_ack, sync_inv_progress);
}
}
-
-/* Implementation Defined Register Space 0 register offsets */
-static const u32 qcom_smmu_impl0_reg_offset[] = {
- [QCOM_SMMU_TBU_PWR_STATUS] = 0x2204,
- [QCOM_SMMU_STATS_SYNC_INV_TBU_ACK] = 0x25dc,
- [QCOM_SMMU_MMU2QSS_AND_SAFE_WAIT_CNTR] = 0x2670,
-};
-
-static const struct qcom_smmu_config qcm2290_smmu_cfg = {
- .reg_offset = qcom_smmu_impl0_reg_offset,
-};
-
-static const struct qcom_smmu_config sc7180_smmu_cfg = {
- .reg_offset = qcom_smmu_impl0_reg_offset,
-};
-
-static const struct qcom_smmu_config sc7280_smmu_cfg = {
- .reg_offset = qcom_smmu_impl0_reg_offset,
-};
-
-static const struct qcom_smmu_config sc8180x_smmu_cfg = {
- .reg_offset = qcom_smmu_impl0_reg_offset,
-};
-
-static const struct qcom_smmu_config sc8280xp_smmu_cfg = {
- .reg_offset = qcom_smmu_impl0_reg_offset,
-};
-
-static const struct qcom_smmu_config sm6125_smmu_cfg = {
- .reg_offset = qcom_smmu_impl0_reg_offset,
-};
-
-static const struct qcom_smmu_config sm6350_smmu_cfg = {
- .reg_offset = qcom_smmu_impl0_reg_offset,
-};
-
-static const struct qcom_smmu_config sm8150_smmu_cfg = {
- .reg_offset = qcom_smmu_impl0_reg_offset,
-};
-
-static const struct qcom_smmu_config sm8250_smmu_cfg = {
- .reg_offset = qcom_smmu_impl0_reg_offset,
-};
-
-static const struct qcom_smmu_config sm8350_smmu_cfg = {
- .reg_offset = qcom_smmu_impl0_reg_offset,
-};
-
-static const struct qcom_smmu_config sm8450_smmu_cfg = {
- .reg_offset = qcom_smmu_impl0_reg_offset,
-};
-
-static const struct of_device_id __maybe_unused qcom_smmu_impl_debug_match[] = {
- { .compatible = "qcom,msm8998-smmu-v2" },
- { .compatible = "qcom,qcm2290-smmu-500", .data = &qcm2290_smmu_cfg },
- { .compatible = "qcom,sc7180-smmu-500", .data = &sc7180_smmu_cfg },
- { .compatible = "qcom,sc7280-smmu-500", .data = &sc7280_smmu_cfg},
- { .compatible = "qcom,sc8180x-smmu-500", .data = &sc8180x_smmu_cfg },
- { .compatible = "qcom,sc8280xp-smmu-500", .data = &sc8280xp_smmu_cfg },
- { .compatible = "qcom,sdm630-smmu-v2" },
- { .compatible = "qcom,sdm845-smmu-500" },
- { .compatible = "qcom,sm6125-smmu-500", .data = &sm6125_smmu_cfg},
- { .compatible = "qcom,sm6350-smmu-500", .data = &sm6350_smmu_cfg},
- { .compatible = "qcom,sm8150-smmu-500", .data = &sm8150_smmu_cfg },
- { .compatible = "qcom,sm8250-smmu-500", .data = &sm8250_smmu_cfg },
- { .compatible = "qcom,sm8350-smmu-500", .data = &sm8350_smmu_cfg },
- { .compatible = "qcom,sm8450-smmu-500", .data = &sm8450_smmu_cfg },
- { }
-};
-
-const void *qcom_smmu_impl_data(struct arm_smmu_device *smmu)
-{
- const struct of_device_id *match;
- const struct device_node *np = smmu->dev->of_node;
-
- match = of_match_node(qcom_smmu_impl_debug_match, np);
- if (!match)
- return NULL;
-
- return match->data;
-}
diff --git a/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c b/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c
index b2708de..91d404d 100644
--- a/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c
+++ b/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c
@@ -361,6 +361,8 @@ static int qcom_sdm845_smmu500_reset(struct arm_smmu_device *smmu)
{
int ret;
+ arm_mmu500_reset(smmu);
+
/*
* To address performance degradation in non-real time clients,
* such as USB and UFS, turn off wait-for-safe on sdm845 based boards,
@@ -374,41 +376,67 @@ static int qcom_sdm845_smmu500_reset(struct arm_smmu_device *smmu)
return ret;
}
-static int qcom_smmu500_reset(struct arm_smmu_device *smmu)
-{
- const struct device_node *np = smmu->dev->of_node;
-
- arm_mmu500_reset(smmu);
-
- if (of_device_is_compatible(np, "qcom,sdm845-smmu-500"))
- return qcom_sdm845_smmu500_reset(smmu);
-
- return 0;
-}
-
-static const struct arm_smmu_impl qcom_smmu_impl = {
+static const struct arm_smmu_impl qcom_smmu_v2_impl = {
.init_context = qcom_smmu_init_context,
.cfg_probe = qcom_smmu_cfg_probe,
.def_domain_type = qcom_smmu_def_domain_type,
- .reset = qcom_smmu500_reset,
.write_s2cr = qcom_smmu_write_s2cr,
.tlb_sync = qcom_smmu_tlb_sync,
};
-static const struct arm_smmu_impl qcom_adreno_smmu_impl = {
+static const struct arm_smmu_impl qcom_smmu_500_impl = {
+ .init_context = qcom_smmu_init_context,
+ .cfg_probe = qcom_smmu_cfg_probe,
+ .def_domain_type = qcom_smmu_def_domain_type,
+ .reset = arm_mmu500_reset,
+ .write_s2cr = qcom_smmu_write_s2cr,
+ .tlb_sync = qcom_smmu_tlb_sync,
+};
+
+static const struct arm_smmu_impl sdm845_smmu_500_impl = {
+ .init_context = qcom_smmu_init_context,
+ .cfg_probe = qcom_smmu_cfg_probe,
+ .def_domain_type = qcom_smmu_def_domain_type,
+ .reset = qcom_sdm845_smmu500_reset,
+ .write_s2cr = qcom_smmu_write_s2cr,
+ .tlb_sync = qcom_smmu_tlb_sync,
+};
+
+static const struct arm_smmu_impl qcom_adreno_smmu_v2_impl = {
.init_context = qcom_adreno_smmu_init_context,
.def_domain_type = qcom_smmu_def_domain_type,
- .reset = qcom_smmu500_reset,
+ .alloc_context_bank = qcom_adreno_smmu_alloc_context_bank,
+ .write_sctlr = qcom_adreno_smmu_write_sctlr,
+ .tlb_sync = qcom_smmu_tlb_sync,
+};
+
+static const struct arm_smmu_impl qcom_adreno_smmu_500_impl = {
+ .init_context = qcom_adreno_smmu_init_context,
+ .def_domain_type = qcom_smmu_def_domain_type,
+ .reset = arm_mmu500_reset,
.alloc_context_bank = qcom_adreno_smmu_alloc_context_bank,
.write_sctlr = qcom_adreno_smmu_write_sctlr,
.tlb_sync = qcom_smmu_tlb_sync,
};
static struct arm_smmu_device *qcom_smmu_create(struct arm_smmu_device *smmu,
- const struct arm_smmu_impl *impl)
+ const struct qcom_smmu_match_data *data)
{
+ const struct device_node *np = smmu->dev->of_node;
+ const struct arm_smmu_impl *impl;
struct qcom_smmu *qsmmu;
+ if (!data)
+ return ERR_PTR(-EINVAL);
+
+ if (np && of_device_is_compatible(np, "qcom,adreno-smmu"))
+ impl = data->adreno_impl;
+ else
+ impl = data->impl;
+
+ if (!impl)
+ return smmu;
+
/* Check to make sure qcom_scm has finished probing */
if (!qcom_scm_is_available())
return ERR_PTR(-EPROBE_DEFER);
@@ -418,27 +446,77 @@ static struct arm_smmu_device *qcom_smmu_create(struct arm_smmu_device *smmu,
return ERR_PTR(-ENOMEM);
qsmmu->smmu.impl = impl;
- qsmmu->cfg = qcom_smmu_impl_data(smmu);
+ qsmmu->cfg = data->cfg;
return &qsmmu->smmu;
}
+/* Implementation Defined Register Space 0 register offsets */
+static const u32 qcom_smmu_impl0_reg_offset[] = {
+ [QCOM_SMMU_TBU_PWR_STATUS] = 0x2204,
+ [QCOM_SMMU_STATS_SYNC_INV_TBU_ACK] = 0x25dc,
+ [QCOM_SMMU_MMU2QSS_AND_SAFE_WAIT_CNTR] = 0x2670,
+};
+
+static const struct qcom_smmu_config qcom_smmu_impl0_cfg = {
+ .reg_offset = qcom_smmu_impl0_reg_offset,
+};
+
+/*
+ * It is not yet possible to use MDP SMMU with the bypass quirk on the msm8996,
+ * there are not enough context banks.
+ */
+static const struct qcom_smmu_match_data msm8996_smmu_data = {
+ .impl = NULL,
+ .adreno_impl = &qcom_adreno_smmu_v2_impl,
+};
+
+static const struct qcom_smmu_match_data qcom_smmu_v2_data = {
+ .impl = &qcom_smmu_v2_impl,
+ .adreno_impl = &qcom_adreno_smmu_v2_impl,
+};
+
+static const struct qcom_smmu_match_data sdm845_smmu_500_data = {
+ .impl = &sdm845_smmu_500_impl,
+ /*
+ * No need for adreno impl here. On sdm845 the Adreno SMMU is handled
+ * by the separate sdm845-smmu-v2 device.
+ */
+ /* Also no debug configuration. */
+};
+
+static const struct qcom_smmu_match_data qcom_smmu_500_impl0_data = {
+ .impl = &qcom_smmu_500_impl,
+ .adreno_impl = &qcom_adreno_smmu_500_impl,
+ .cfg = &qcom_smmu_impl0_cfg,
+};
+
+/*
+ * Do not add any more qcom,SOC-smmu-500 entries to this list, unless they need
+ * special handling and can not be covered by the qcom,smmu-500 entry.
+ */
static const struct of_device_id __maybe_unused qcom_smmu_impl_of_match[] = {
- { .compatible = "qcom,msm8998-smmu-v2" },
- { .compatible = "qcom,qcm2290-smmu-500" },
- { .compatible = "qcom,sc7180-smmu-500" },
- { .compatible = "qcom,sc7280-smmu-500" },
- { .compatible = "qcom,sc8180x-smmu-500" },
- { .compatible = "qcom,sc8280xp-smmu-500" },
- { .compatible = "qcom,sdm630-smmu-v2" },
- { .compatible = "qcom,sdm845-smmu-500" },
- { .compatible = "qcom,sm6125-smmu-500" },
- { .compatible = "qcom,sm6350-smmu-500" },
- { .compatible = "qcom,sm6375-smmu-500" },
- { .compatible = "qcom,sm8150-smmu-500" },
- { .compatible = "qcom,sm8250-smmu-500" },
- { .compatible = "qcom,sm8350-smmu-500" },
- { .compatible = "qcom,sm8450-smmu-500" },
+ { .compatible = "qcom,msm8996-smmu-v2", .data = &msm8996_smmu_data },
+ { .compatible = "qcom,msm8998-smmu-v2", .data = &qcom_smmu_v2_data },
+ { .compatible = "qcom,qcm2290-smmu-500", .data = &qcom_smmu_500_impl0_data },
+ { .compatible = "qcom,qdu1000-smmu-500", .data = &qcom_smmu_500_impl0_data },
+ { .compatible = "qcom,sc7180-smmu-500", .data = &qcom_smmu_500_impl0_data },
+ { .compatible = "qcom,sc7280-smmu-500", .data = &qcom_smmu_500_impl0_data },
+ { .compatible = "qcom,sc8180x-smmu-500", .data = &qcom_smmu_500_impl0_data },
+ { .compatible = "qcom,sc8280xp-smmu-500", .data = &qcom_smmu_500_impl0_data },
+ { .compatible = "qcom,sdm630-smmu-v2", .data = &qcom_smmu_v2_data },
+ { .compatible = "qcom,sdm845-smmu-v2", .data = &qcom_smmu_v2_data },
+ { .compatible = "qcom,sdm845-smmu-500", .data = &sdm845_smmu_500_data },
+ { .compatible = "qcom,sm6115-smmu-500", .data = &qcom_smmu_500_impl0_data},
+ { .compatible = "qcom,sm6125-smmu-500", .data = &qcom_smmu_500_impl0_data },
+ { .compatible = "qcom,sm6350-smmu-v2", .data = &qcom_smmu_v2_data },
+ { .compatible = "qcom,sm6350-smmu-500", .data = &qcom_smmu_500_impl0_data },
+ { .compatible = "qcom,sm6375-smmu-500", .data = &qcom_smmu_500_impl0_data },
+ { .compatible = "qcom,sm8150-smmu-500", .data = &qcom_smmu_500_impl0_data },
+ { .compatible = "qcom,sm8250-smmu-500", .data = &qcom_smmu_500_impl0_data },
+ { .compatible = "qcom,sm8350-smmu-500", .data = &qcom_smmu_500_impl0_data },
+ { .compatible = "qcom,sm8450-smmu-500", .data = &qcom_smmu_500_impl0_data },
+ { .compatible = "qcom,smmu-500", .data = &qcom_smmu_500_impl0_data },
{ }
};
@@ -453,26 +531,19 @@ static struct acpi_platform_list qcom_acpi_platlist[] = {
struct arm_smmu_device *qcom_smmu_impl_init(struct arm_smmu_device *smmu)
{
const struct device_node *np = smmu->dev->of_node;
+ const struct of_device_id *match;
#ifdef CONFIG_ACPI
if (np == NULL) {
/* Match platform for ACPI boot */
if (acpi_match_platform_list(qcom_acpi_platlist) >= 0)
- return qcom_smmu_create(smmu, &qcom_smmu_impl);
+ return qcom_smmu_create(smmu, &qcom_smmu_500_impl0_data);
}
#endif
- /*
- * Do not change this order of implementation, i.e., first adreno
- * smmu impl and then apss smmu since we can have both implementing
- * arm,mmu-500 in which case we will miss setting adreno smmu specific
- * features if the order is changed.
- */
- if (of_device_is_compatible(np, "qcom,adreno-smmu"))
- return qcom_smmu_create(smmu, &qcom_adreno_smmu_impl);
-
- if (of_match_node(qcom_smmu_impl_of_match, np))
- return qcom_smmu_create(smmu, &qcom_smmu_impl);
+ match = of_match_node(qcom_smmu_impl_of_match, np);
+ if (match)
+ return qcom_smmu_create(smmu, match->data);
return smmu;
}
diff --git a/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.h b/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.h
index 99ec8f8..5939105 100644
--- a/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.h
+++ b/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.h
@@ -14,15 +14,26 @@ struct qcom_smmu {
u32 stall_enabled;
};
+enum qcom_smmu_impl_reg_offset {
+ QCOM_SMMU_TBU_PWR_STATUS,
+ QCOM_SMMU_STATS_SYNC_INV_TBU_ACK,
+ QCOM_SMMU_MMU2QSS_AND_SAFE_WAIT_CNTR,
+};
+
+struct qcom_smmu_config {
+ const u32 *reg_offset;
+};
+
+struct qcom_smmu_match_data {
+ const struct qcom_smmu_config *cfg;
+ const struct arm_smmu_impl *impl;
+ const struct arm_smmu_impl *adreno_impl;
+};
+
#ifdef CONFIG_ARM_SMMU_QCOM_DEBUG
void qcom_smmu_tlb_sync_debug(struct arm_smmu_device *smmu);
-const void *qcom_smmu_impl_data(struct arm_smmu_device *smmu);
#else
static inline void qcom_smmu_tlb_sync_debug(struct arm_smmu_device *smmu) { }
-static inline const void *qcom_smmu_impl_data(struct arm_smmu_device *smmu)
-{
- return NULL;
-}
#endif
#endif /* _ARM_SMMU_QCOM_H */
diff --git a/drivers/iommu/arm/arm-smmu/qcom_iommu.c b/drivers/iommu/arm/arm-smmu/qcom_iommu.c
index bfd7b51..270c3d9 100644
--- a/drivers/iommu/arm/arm-smmu/qcom_iommu.c
+++ b/drivers/iommu/arm/arm-smmu/qcom_iommu.c
@@ -410,7 +410,8 @@ static void qcom_iommu_detach_dev(struct iommu_domain *domain, struct device *de
}
static int qcom_iommu_map(struct iommu_domain *domain, unsigned long iova,
- phys_addr_t paddr, size_t size, int prot, gfp_t gfp)
+ phys_addr_t paddr, size_t pgsize, size_t pgcount,
+ int prot, gfp_t gfp, size_t *mapped)
{
int ret;
unsigned long flags;
@@ -421,13 +422,14 @@ static int qcom_iommu_map(struct iommu_domain *domain, unsigned long iova,
return -ENODEV;
spin_lock_irqsave(&qcom_domain->pgtbl_lock, flags);
- ret = ops->map(ops, iova, paddr, size, prot, GFP_ATOMIC);
+ ret = ops->map_pages(ops, iova, paddr, pgsize, pgcount, prot, GFP_ATOMIC, mapped);
spin_unlock_irqrestore(&qcom_domain->pgtbl_lock, flags);
return ret;
}
static size_t qcom_iommu_unmap(struct iommu_domain *domain, unsigned long iova,
- size_t size, struct iommu_iotlb_gather *gather)
+ size_t pgsize, size_t pgcount,
+ struct iommu_iotlb_gather *gather)
{
size_t ret;
unsigned long flags;
@@ -444,7 +446,7 @@ static size_t qcom_iommu_unmap(struct iommu_domain *domain, unsigned long iova,
*/
pm_runtime_get_sync(qcom_domain->iommu->dev);
spin_lock_irqsave(&qcom_domain->pgtbl_lock, flags);
- ret = ops->unmap(ops, iova, size, gather);
+ ret = ops->unmap_pages(ops, iova, pgsize, pgcount, gather);
spin_unlock_irqrestore(&qcom_domain->pgtbl_lock, flags);
pm_runtime_put_sync(qcom_domain->iommu->dev);
@@ -582,8 +584,8 @@ static const struct iommu_ops qcom_iommu_ops = {
.default_domain_ops = &(const struct iommu_domain_ops) {
.attach_dev = qcom_iommu_attach_dev,
.detach_dev = qcom_iommu_detach_dev,
- .map = qcom_iommu_map,
- .unmap = qcom_iommu_unmap,
+ .map_pages = qcom_iommu_map,
+ .unmap_pages = qcom_iommu_unmap,
.flush_iotlb_all = qcom_iommu_flush_iotlb_all,
.iotlb_sync = qcom_iommu_iotlb_sync,
.iova_to_phys = qcom_iommu_iova_to_phys,
diff --git a/drivers/iommu/exynos-iommu.c b/drivers/iommu/exynos-iommu.c
index 45fd485..b0cde22 100644
--- a/drivers/iommu/exynos-iommu.c
+++ b/drivers/iommu/exynos-iommu.c
@@ -708,10 +708,6 @@ static int exynos_sysmmu_probe(struct platform_device *pdev)
if (ret)
return ret;
- ret = iommu_device_register(&data->iommu, &exynos_iommu_ops, dev);
- if (ret)
- goto err_iommu_register;
-
platform_set_drvdata(pdev, data);
if (PG_ENT_SHIFT < 0) {
@@ -743,11 +739,13 @@ static int exynos_sysmmu_probe(struct platform_device *pdev)
pm_runtime_enable(dev);
+ ret = iommu_device_register(&data->iommu, &exynos_iommu_ops, dev);
+ if (ret)
+ goto err_dma_set_mask;
+
return 0;
err_dma_set_mask:
- iommu_device_unregister(&data->iommu);
-err_iommu_register:
iommu_device_sysfs_remove(&data->iommu);
return ret;
}
@@ -1432,12 +1430,6 @@ static int __init exynos_iommu_init(void)
return -ENOMEM;
}
- ret = platform_driver_register(&exynos_sysmmu_driver);
- if (ret) {
- pr_err("%s: Failed to register driver\n", __func__);
- goto err_reg_driver;
- }
-
zero_lv2_table = kmem_cache_zalloc(lv2table_kmem_cache, GFP_KERNEL);
if (zero_lv2_table == NULL) {
pr_err("%s: Failed to allocate zero level2 page table\n",
@@ -1446,10 +1438,16 @@ static int __init exynos_iommu_init(void)
goto err_zero_lv2;
}
+ ret = platform_driver_register(&exynos_sysmmu_driver);
+ if (ret) {
+ pr_err("%s: Failed to register driver\n", __func__);
+ goto err_reg_driver;
+ }
+
return 0;
-err_zero_lv2:
- platform_driver_unregister(&exynos_sysmmu_driver);
err_reg_driver:
+ platform_driver_unregister(&exynos_sysmmu_driver);
+err_zero_lv2:
kmem_cache_destroy(lv2table_kmem_cache);
return ret;
}
diff --git a/drivers/iommu/fsl_pamu.c b/drivers/iommu/fsl_pamu.c
index 2eb3211..05d820f 100644
--- a/drivers/iommu/fsl_pamu.c
+++ b/drivers/iommu/fsl_pamu.c
@@ -779,7 +779,7 @@ static int fsl_pamu_probe(struct platform_device *pdev)
of_get_address(dev->of_node, 0, &size, NULL);
irq = irq_of_parse_and_map(dev->of_node, 0);
- if (irq == NO_IRQ) {
+ if (!irq) {
dev_warn(dev, "no interrupts listed in PAMU node\n");
goto error;
}
@@ -868,7 +868,7 @@ static int fsl_pamu_probe(struct platform_device *pdev)
ret = create_csd(ppaact_phys, mem_size, csd_port_id);
if (ret) {
dev_err(dev, "could not create coherence subdomain\n");
- return ret;
+ goto error;
}
}
@@ -903,7 +903,7 @@ static int fsl_pamu_probe(struct platform_device *pdev)
return 0;
error:
- if (irq != NO_IRQ)
+ if (irq)
free_irq(irq, data);
kfree_sensitive(data);
diff --git a/drivers/iommu/intel/iommu.c b/drivers/iommu/intel/iommu.c
index bef8e8f..59df7e4 100644
--- a/drivers/iommu/intel/iommu.c
+++ b/drivers/iommu/intel/iommu.c
@@ -277,7 +277,8 @@ static LIST_HEAD(dmar_satc_units);
#define for_each_rmrr_units(rmrr) \
list_for_each_entry(rmrr, &dmar_rmrr_units, list)
-static void dmar_remove_one_dev_info(struct device *dev);
+static void device_block_translation(struct device *dev);
+static void intel_iommu_domain_free(struct iommu_domain *domain);
int dmar_disabled = !IS_ENABLED(CONFIG_INTEL_IOMMU_DEFAULT_ON);
int intel_iommu_sm = IS_ENABLED(CONFIG_INTEL_IOMMU_SCALABLE_MODE_DEFAULT_ON);
@@ -382,11 +383,6 @@ static inline int domain_type_is_si(struct dmar_domain *domain)
return domain->domain.type == IOMMU_DOMAIN_IDENTITY;
}
-static inline bool domain_use_first_level(struct dmar_domain *domain)
-{
- return domain->flags & DOMAIN_FLAG_USE_FIRST_LEVEL;
-}
-
static inline int domain_pfn_supported(struct dmar_domain *domain,
unsigned long pfn)
{
@@ -500,7 +496,7 @@ static int domain_update_iommu_superpage(struct dmar_domain *domain,
rcu_read_lock();
for_each_active_iommu(iommu, drhd) {
if (iommu != skip) {
- if (domain && domain_use_first_level(domain)) {
+ if (domain && domain->use_first_level) {
if (!cap_fl1gp_support(iommu->cap))
mask = 0x1;
} else {
@@ -578,7 +574,7 @@ static void domain_update_iommu_cap(struct dmar_domain *domain)
* paging and 57-bits with 5-level paging). Hence, skip bit
* [N-1].
*/
- if (domain_use_first_level(domain))
+ if (domain->use_first_level)
domain->domain.geometry.aperture_end = __DOMAIN_MAX_ADDR(domain->gaw - 1);
else
domain->domain.geometry.aperture_end = __DOMAIN_MAX_ADDR(domain->gaw);
@@ -779,19 +775,6 @@ static void domain_flush_cache(struct dmar_domain *domain,
clflush_cache_range(addr, size);
}
-static int device_context_mapped(struct intel_iommu *iommu, u8 bus, u8 devfn)
-{
- struct context_entry *context;
- int ret = 0;
-
- spin_lock(&iommu->lock);
- context = iommu_context_addr(iommu, bus, devfn, 0);
- if (context)
- ret = context_present(context);
- spin_unlock(&iommu->lock);
- return ret;
-}
-
static void free_context_table(struct intel_iommu *iommu)
{
struct context_entry *context;
@@ -959,7 +942,7 @@ static struct dma_pte *pfn_to_dma_pte(struct dmar_domain *domain,
domain_flush_cache(domain, tmp_page, VTD_PAGE_SIZE);
pteval = ((uint64_t)virt_to_dma_pfn(tmp_page) << VTD_PAGE_SHIFT) | DMA_PTE_READ | DMA_PTE_WRITE;
- if (domain_use_first_level(domain))
+ if (domain->use_first_level)
pteval |= DMA_FL_PTE_XD | DMA_FL_PTE_US | DMA_FL_PTE_ACCESS;
if (cmpxchg64(&pte->val, 0ULL, pteval))
@@ -1418,7 +1401,7 @@ static void iommu_enable_pci_caps(struct device_domain_info *info)
{
struct pci_dev *pdev;
- if (!info || !dev_is_pci(info->dev))
+ if (!dev_is_pci(info->dev))
return;
pdev = to_pci_dev(info->dev);
@@ -1458,7 +1441,7 @@ static void iommu_enable_pci_caps(struct device_domain_info *info)
}
}
-static void iommu_disable_dev_iotlb(struct device_domain_info *info)
+static void iommu_disable_pci_caps(struct device_domain_info *info)
{
struct pci_dev *pdev;
@@ -1529,7 +1512,7 @@ static void iommu_flush_iotlb_psi(struct intel_iommu *iommu,
if (ih)
ih = 1 << 6;
- if (domain_use_first_level(domain)) {
+ if (domain->use_first_level) {
qi_flush_piotlb(iommu, did, PASID_RID2PASID, addr, pages, ih);
} else {
unsigned long bitmask = aligned_pages - 1;
@@ -1583,7 +1566,7 @@ static inline void __mapping_notify_one(struct intel_iommu *iommu,
* It's a non-present to present mapping. Only flush if caching mode
* and second level.
*/
- if (cap_caching_mode(iommu->cap) && !domain_use_first_level(domain))
+ if (cap_caching_mode(iommu->cap) && !domain->use_first_level)
iommu_flush_iotlb_psi(iommu, domain, pfn, pages, 0, 1);
else
iommu_flush_write_buffer(iommu);
@@ -1599,7 +1582,7 @@ static void intel_flush_iotlb_all(struct iommu_domain *domain)
struct intel_iommu *iommu = info->iommu;
u16 did = domain_id_iommu(dmar_domain, iommu);
- if (domain_use_first_level(dmar_domain))
+ if (dmar_domain->use_first_level)
qi_flush_piotlb(iommu, did, PASID_RID2PASID, 0, -1, 0);
else
iommu->flush.flush_iotlb(iommu, did, 0, 0,
@@ -1772,7 +1755,7 @@ static struct dmar_domain *alloc_domain(unsigned int type)
domain->nid = NUMA_NO_NODE;
if (first_level_by_default(type))
- domain->flags |= DOMAIN_FLAG_USE_FIRST_LEVEL;
+ domain->use_first_level = true;
domain->has_iotlb_device = false;
INIT_LIST_HEAD(&domain->devices);
spin_lock_init(&domain->lock);
@@ -2064,7 +2047,6 @@ static int domain_context_mapping_one(struct dmar_domain *domain,
} else {
iommu_flush_write_buffer(iommu);
}
- iommu_enable_pci_caps(info);
ret = 0;
@@ -2116,30 +2098,6 @@ domain_context_mapping(struct dmar_domain *domain, struct device *dev)
&domain_context_mapping_cb, &data);
}
-static int domain_context_mapped_cb(struct pci_dev *pdev,
- u16 alias, void *opaque)
-{
- struct intel_iommu *iommu = opaque;
-
- return !device_context_mapped(iommu, PCI_BUS_NUM(alias), alias & 0xff);
-}
-
-static int domain_context_mapped(struct device *dev)
-{
- struct intel_iommu *iommu;
- u8 bus, devfn;
-
- iommu = device_to_iommu(dev, &bus, &devfn);
- if (!iommu)
- return -ENODEV;
-
- if (!dev_is_pci(dev))
- return device_context_mapped(iommu, bus, devfn);
-
- return !pci_for_each_dma_alias(to_pci_dev(dev),
- domain_context_mapped_cb, iommu);
-}
-
/* Returns a number of VTD pages, but aligned to MM page size */
static inline unsigned long aligned_nrpages(unsigned long host_addr,
size_t size)
@@ -2229,7 +2187,7 @@ __domain_mapping(struct dmar_domain *domain, unsigned long iov_pfn,
attr = prot & (DMA_PTE_READ | DMA_PTE_WRITE | DMA_PTE_SNP);
attr |= DMA_FL_PTE_PRESENT;
- if (domain_use_first_level(domain)) {
+ if (domain->use_first_level) {
attr |= DMA_FL_PTE_XD | DMA_FL_PTE_US | DMA_FL_PTE_ACCESS;
if (prot & DMA_PTE_WRITE)
attr |= DMA_FL_PTE_DIRTY;
@@ -2472,7 +2430,8 @@ static int __init si_domain_init(int hw)
return 0;
}
-static int domain_add_dev_info(struct dmar_domain *domain, struct device *dev)
+static int dmar_domain_attach_device(struct dmar_domain *domain,
+ struct device *dev)
{
struct device_domain_info *info = dev_iommu_priv_get(dev);
struct intel_iommu *iommu;
@@ -2494,18 +2453,11 @@ static int domain_add_dev_info(struct dmar_domain *domain, struct device *dev)
/* PASID table is mandatory for a PCI device in scalable mode. */
if (sm_supported(iommu) && !dev_is_real_dma_subdevice(dev)) {
- ret = intel_pasid_alloc_table(dev);
- if (ret) {
- dev_err(dev, "PASID table allocation failed\n");
- dmar_remove_one_dev_info(dev);
- return ret;
- }
-
/* Setup the PASID entry for requests without PASID: */
if (hw_pass_through && domain_type_is_si(domain))
ret = intel_pasid_setup_pass_through(iommu, domain,
dev, PASID_RID2PASID);
- else if (domain_use_first_level(domain))
+ else if (domain->use_first_level)
ret = domain_setup_first_level(iommu, domain, dev,
PASID_RID2PASID);
else
@@ -2513,7 +2465,7 @@ static int domain_add_dev_info(struct dmar_domain *domain, struct device *dev)
dev, PASID_RID2PASID);
if (ret) {
dev_err(dev, "Setup RID2PASID failed\n");
- dmar_remove_one_dev_info(dev);
+ device_block_translation(dev);
return ret;
}
}
@@ -2521,10 +2473,12 @@ static int domain_add_dev_info(struct dmar_domain *domain, struct device *dev)
ret = domain_context_mapping(domain, dev);
if (ret) {
dev_err(dev, "Domain context map failed\n");
- dmar_remove_one_dev_info(dev);
+ device_block_translation(dev);
return ret;
}
+ iommu_enable_pci_caps(info);
+
return 0;
}
@@ -4125,9 +4079,8 @@ static void dmar_remove_one_dev_info(struct device *dev)
intel_pasid_tear_down_entry(iommu, info->dev,
PASID_RID2PASID, false);
- iommu_disable_dev_iotlb(info);
+ iommu_disable_pci_caps(info);
domain_context_clear(info);
- intel_pasid_free_table(info->dev);
}
spin_lock_irqsave(&domain->lock, flags);
@@ -4138,6 +4091,37 @@ static void dmar_remove_one_dev_info(struct device *dev)
info->domain = NULL;
}
+/*
+ * Clear the page table pointer in context or pasid table entries so that
+ * all DMA requests without PASID from the device are blocked. If the page
+ * table has been set, clean up the data structures.
+ */
+static void device_block_translation(struct device *dev)
+{
+ struct device_domain_info *info = dev_iommu_priv_get(dev);
+ struct intel_iommu *iommu = info->iommu;
+ unsigned long flags;
+
+ iommu_disable_pci_caps(info);
+ if (!dev_is_real_dma_subdevice(dev)) {
+ if (sm_supported(iommu))
+ intel_pasid_tear_down_entry(iommu, dev,
+ PASID_RID2PASID, false);
+ else
+ domain_context_clear(info);
+ }
+
+ if (!info->domain)
+ return;
+
+ spin_lock_irqsave(&info->domain->lock, flags);
+ list_del(&info->link);
+ spin_unlock_irqrestore(&info->domain->lock, flags);
+
+ domain_detach_iommu(info->domain, iommu);
+ info->domain = NULL;
+}
+
static int md_domain_init(struct dmar_domain *domain, int guest_width)
{
int adjust_width;
@@ -4159,12 +4143,28 @@ static int md_domain_init(struct dmar_domain *domain, int guest_width)
return 0;
}
+static int blocking_domain_attach_dev(struct iommu_domain *domain,
+ struct device *dev)
+{
+ device_block_translation(dev);
+ return 0;
+}
+
+static struct iommu_domain blocking_domain = {
+ .ops = &(const struct iommu_domain_ops) {
+ .attach_dev = blocking_domain_attach_dev,
+ .free = intel_iommu_domain_free
+ }
+};
+
static struct iommu_domain *intel_iommu_domain_alloc(unsigned type)
{
struct dmar_domain *dmar_domain;
struct iommu_domain *domain;
switch (type) {
+ case IOMMU_DOMAIN_BLOCKED:
+ return &blocking_domain;
case IOMMU_DOMAIN_DMA:
case IOMMU_DOMAIN_DMA_FQ:
case IOMMU_DOMAIN_UNMANAGED:
@@ -4199,7 +4199,7 @@ static struct iommu_domain *intel_iommu_domain_alloc(unsigned type)
static void intel_iommu_domain_free(struct iommu_domain *domain)
{
- if (domain != &si_domain->domain)
+ if (domain != &si_domain->domain && domain != &blocking_domain)
domain_exit(to_dmar_domain(domain));
}
@@ -4246,6 +4246,7 @@ static int prepare_domain_attach_device(struct iommu_domain *domain,
static int intel_iommu_attach_device(struct iommu_domain *domain,
struct device *dev)
{
+ struct device_domain_info *info = dev_iommu_priv_get(dev);
int ret;
if (domain->type == IOMMU_DOMAIN_UNMANAGED &&
@@ -4254,25 +4255,14 @@ static int intel_iommu_attach_device(struct iommu_domain *domain,
return -EPERM;
}
- /* normally dev is not mapped */
- if (unlikely(domain_context_mapped(dev))) {
- struct device_domain_info *info = dev_iommu_priv_get(dev);
-
- if (info->domain)
- dmar_remove_one_dev_info(dev);
- }
+ if (info->domain)
+ device_block_translation(dev);
ret = prepare_domain_attach_device(domain, dev);
if (ret)
return ret;
- return domain_add_dev_info(to_dmar_domain(domain), dev);
-}
-
-static void intel_iommu_detach_device(struct iommu_domain *domain,
- struct device *dev)
-{
- dmar_remove_one_dev_info(dev);
+ return dmar_domain_attach_device(to_dmar_domain(domain), dev);
}
static int intel_iommu_map(struct iommu_domain *domain,
@@ -4436,7 +4426,7 @@ static void domain_set_force_snooping(struct dmar_domain *domain)
* Second level page table supports per-PTE snoop control. The
* iommu_map() interface will handle this by setting SNP bit.
*/
- if (!domain_use_first_level(domain)) {
+ if (!domain->use_first_level) {
domain->set_pte_snp = true;
return;
}
@@ -4491,6 +4481,7 @@ static struct iommu_device *intel_iommu_probe_device(struct device *dev)
struct device_domain_info *info;
struct intel_iommu *iommu;
u8 bus, devfn;
+ int ret;
iommu = device_to_iommu(dev, &bus, &devfn);
if (!iommu || !iommu->iommu.ops)
@@ -4535,6 +4526,16 @@ static struct iommu_device *intel_iommu_probe_device(struct device *dev)
dev_iommu_priv_set(dev, info);
+ if (sm_supported(iommu) && !dev_is_real_dma_subdevice(dev)) {
+ ret = intel_pasid_alloc_table(dev);
+ if (ret) {
+ dev_err(dev, "PASID table allocation failed\n");
+ dev_iommu_priv_set(dev, NULL);
+ kfree(info);
+ return ERR_PTR(ret);
+ }
+ }
+
return &iommu->iommu;
}
@@ -4543,6 +4544,7 @@ static void intel_iommu_release_device(struct device *dev)
struct device_domain_info *info = dev_iommu_priv_get(dev);
dmar_remove_one_dev_info(dev);
+ intel_pasid_free_table(dev);
dev_iommu_priv_set(dev, NULL);
kfree(info);
set_dma_ops(dev, NULL);
@@ -4777,7 +4779,6 @@ const struct iommu_ops intel_iommu_ops = {
#endif
.default_domain_ops = &(const struct iommu_domain_ops) {
.attach_dev = intel_iommu_attach_device,
- .detach_dev = intel_iommu_detach_device,
.map_pages = intel_iommu_map_pages,
.unmap_pages = intel_iommu_unmap_pages,
.iotlb_sync_map = intel_iommu_iotlb_sync_map,
diff --git a/drivers/iommu/intel/iommu.h b/drivers/iommu/intel/iommu.h
index f83ad8d..06e61e4 100644
--- a/drivers/iommu/intel/iommu.h
+++ b/drivers/iommu/intel/iommu.h
@@ -515,14 +515,6 @@ struct context_entry {
u64 hi;
};
-/*
- * When VT-d works in the scalable mode, it allows DMA translation to
- * happen through either first level or second level page table. This
- * bit marks that the DMA translation for the domain goes through the
- * first level page table, otherwise, it goes through the second level.
- */
-#define DOMAIN_FLAG_USE_FIRST_LEVEL BIT(1)
-
struct iommu_domain_info {
struct intel_iommu *iommu;
unsigned int refcnt; /* Refcount of devices per iommu */
@@ -539,6 +531,11 @@ struct dmar_domain {
u8 iommu_coherency: 1; /* indicate coherency of iommu access */
u8 force_snooping : 1; /* Create IOPTEs with snoop control */
u8 set_pte_snp:1;
+ u8 use_first_level:1; /* DMA translation for the domain goes
+ * through the first level page table,
+ * otherwise, goes through the second
+ * level.
+ */
spinlock_t lock; /* Protect device tracking lists */
struct list_head devices; /* all devices' list */
@@ -548,8 +545,6 @@ struct dmar_domain {
/* adjusted guest address width, 0 is level 2 30-bit */
int agaw;
-
- int flags; /* flags to find out type of domain */
int iommu_superpage;/* Level of superpages supported:
0 == 4KiB (no superpages), 1 == 2MiB,
2 == 1GiB, 3 == 512GiB, 4 == 1TiB */
diff --git a/drivers/iommu/intel/irq_remapping.c b/drivers/iommu/intel/irq_remapping.c
index a723f53..f58f5f5 100644
--- a/drivers/iommu/intel/irq_remapping.c
+++ b/drivers/iommu/intel/irq_remapping.c
@@ -174,7 +174,6 @@ static int modify_irte(struct irq_2_iommu *irq_iommu,
index = irq_iommu->irte_index + irq_iommu->sub_handle;
irte = &iommu->ir_table->base[index];
-#if defined(CONFIG_HAVE_CMPXCHG_DOUBLE)
if ((irte->pst == 1) || (irte_modified->pst == 1)) {
bool ret;
@@ -188,11 +187,9 @@ static int modify_irte(struct irq_2_iommu *irq_iommu,
* same as the old value.
*/
WARN_ON(!ret);
- } else
-#endif
- {
- set_64bit(&irte->low, irte_modified->low);
- set_64bit(&irte->high, irte_modified->high);
+ } else {
+ WRITE_ONCE(irte->low, irte_modified->low);
+ WRITE_ONCE(irte->high, irte_modified->high);
}
__iommu_flush_cache(iommu, irte, sizeof(*irte));
@@ -250,8 +247,8 @@ static int clear_entries(struct irq_2_iommu *irq_iommu)
end = start + (1 << irq_iommu->irte_mask);
for (entry = start; entry < end; entry++) {
- set_64bit(&entry->low, 0);
- set_64bit(&entry->high, 0);
+ WRITE_ONCE(entry->low, 0);
+ WRITE_ONCE(entry->high, 0);
}
bitmap_release_region(iommu->ir_table->bitmap, index,
irq_iommu->irte_mask);
diff --git a/drivers/iommu/io-pgtable-arm-v7s.c b/drivers/iommu/io-pgtable-arm-v7s.c
index ba3115f..75f244a 100644
--- a/drivers/iommu/io-pgtable-arm-v7s.c
+++ b/drivers/iommu/io-pgtable-arm-v7s.c
@@ -564,8 +564,7 @@ static int arm_v7s_map_pages(struct io_pgtable_ops *ops, unsigned long iova,
iova += pgsize;
paddr += pgsize;
- if (mapped)
- *mapped += pgsize;
+ *mapped += pgsize;
}
/*
* Synchronise all PTE updates for the new mapping before there's
@@ -576,12 +575,6 @@ static int arm_v7s_map_pages(struct io_pgtable_ops *ops, unsigned long iova,
return ret;
}
-static int arm_v7s_map(struct io_pgtable_ops *ops, unsigned long iova,
- phys_addr_t paddr, size_t size, int prot, gfp_t gfp)
-{
- return arm_v7s_map_pages(ops, iova, paddr, size, 1, prot, gfp, NULL);
-}
-
static void arm_v7s_free_pgtable(struct io_pgtable *iop)
{
struct arm_v7s_io_pgtable *data = io_pgtable_to_data(iop);
@@ -764,12 +757,6 @@ static size_t arm_v7s_unmap_pages(struct io_pgtable_ops *ops, unsigned long iova
return unmapped;
}
-static size_t arm_v7s_unmap(struct io_pgtable_ops *ops, unsigned long iova,
- size_t size, struct iommu_iotlb_gather *gather)
-{
- return arm_v7s_unmap_pages(ops, iova, size, 1, gather);
-}
-
static phys_addr_t arm_v7s_iova_to_phys(struct io_pgtable_ops *ops,
unsigned long iova)
{
@@ -842,9 +829,7 @@ static struct io_pgtable *arm_v7s_alloc_pgtable(struct io_pgtable_cfg *cfg,
goto out_free_data;
data->iop.ops = (struct io_pgtable_ops) {
- .map = arm_v7s_map,
.map_pages = arm_v7s_map_pages,
- .unmap = arm_v7s_unmap,
.unmap_pages = arm_v7s_unmap_pages,
.iova_to_phys = arm_v7s_iova_to_phys,
};
@@ -954,6 +939,7 @@ static int __init arm_v7s_do_selftests(void)
};
unsigned int iova, size, iova_start;
unsigned int i, loopnr = 0;
+ size_t mapped;
selftest_running = true;
@@ -984,15 +970,16 @@ static int __init arm_v7s_do_selftests(void)
iova = 0;
for_each_set_bit(i, &cfg.pgsize_bitmap, BITS_PER_LONG) {
size = 1UL << i;
- if (ops->map(ops, iova, iova, size, IOMMU_READ |
- IOMMU_WRITE |
- IOMMU_NOEXEC |
- IOMMU_CACHE, GFP_KERNEL))
+ if (ops->map_pages(ops, iova, iova, size, 1,
+ IOMMU_READ | IOMMU_WRITE |
+ IOMMU_NOEXEC | IOMMU_CACHE,
+ GFP_KERNEL, &mapped))
return __FAIL(ops);
/* Overlapping mappings */
- if (!ops->map(ops, iova, iova + size, size,
- IOMMU_READ | IOMMU_NOEXEC, GFP_KERNEL))
+ if (!ops->map_pages(ops, iova, iova + size, size, 1,
+ IOMMU_READ | IOMMU_NOEXEC, GFP_KERNEL,
+ &mapped))
return __FAIL(ops);
if (ops->iova_to_phys(ops, iova + 42) != (iova + 42))
@@ -1007,11 +994,12 @@ static int __init arm_v7s_do_selftests(void)
size = 1UL << __ffs(cfg.pgsize_bitmap);
while (i < loopnr) {
iova_start = i * SZ_16M;
- if (ops->unmap(ops, iova_start + size, size, NULL) != size)
+ if (ops->unmap_pages(ops, iova_start + size, size, 1, NULL) != size)
return __FAIL(ops);
/* Remap of partial unmap */
- if (ops->map(ops, iova_start + size, size, size, IOMMU_READ, GFP_KERNEL))
+ if (ops->map_pages(ops, iova_start + size, size, size, 1,
+ IOMMU_READ, GFP_KERNEL, &mapped))
return __FAIL(ops);
if (ops->iova_to_phys(ops, iova_start + size + 42)
@@ -1025,14 +1013,15 @@ static int __init arm_v7s_do_selftests(void)
for_each_set_bit(i, &cfg.pgsize_bitmap, BITS_PER_LONG) {
size = 1UL << i;
- if (ops->unmap(ops, iova, size, NULL) != size)
+ if (ops->unmap_pages(ops, iova, size, 1, NULL) != size)
return __FAIL(ops);
if (ops->iova_to_phys(ops, iova + 42))
return __FAIL(ops);
/* Remap full block */
- if (ops->map(ops, iova, iova, size, IOMMU_WRITE, GFP_KERNEL))
+ if (ops->map_pages(ops, iova, iova, size, 1, IOMMU_WRITE,
+ GFP_KERNEL, &mapped))
return __FAIL(ops);
if (ops->iova_to_phys(ops, iova + 42) != (iova + 42))
diff --git a/drivers/iommu/io-pgtable-arm.c b/drivers/iommu/io-pgtable-arm.c
index 0ba817e..72dcdd4 100644
--- a/drivers/iommu/io-pgtable-arm.c
+++ b/drivers/iommu/io-pgtable-arm.c
@@ -360,7 +360,7 @@ static int __arm_lpae_map(struct arm_lpae_io_pgtable *data, unsigned long iova,
max_entries = ARM_LPAE_PTES_PER_TABLE(data) - map_idx_start;
num_entries = min_t(int, pgcount, max_entries);
ret = arm_lpae_init_pte(data, iova, paddr, prot, lvl, num_entries, ptep);
- if (!ret && mapped)
+ if (!ret)
*mapped += num_entries * size;
return ret;
@@ -496,13 +496,6 @@ static int arm_lpae_map_pages(struct io_pgtable_ops *ops, unsigned long iova,
return ret;
}
-static int arm_lpae_map(struct io_pgtable_ops *ops, unsigned long iova,
- phys_addr_t paddr, size_t size, int iommu_prot, gfp_t gfp)
-{
- return arm_lpae_map_pages(ops, iova, paddr, size, 1, iommu_prot, gfp,
- NULL);
-}
-
static void __arm_lpae_free_pgtable(struct arm_lpae_io_pgtable *data, int lvl,
arm_lpae_iopte *ptep)
{
@@ -682,12 +675,6 @@ static size_t arm_lpae_unmap_pages(struct io_pgtable_ops *ops, unsigned long iov
data->start_level, ptep);
}
-static size_t arm_lpae_unmap(struct io_pgtable_ops *ops, unsigned long iova,
- size_t size, struct iommu_iotlb_gather *gather)
-{
- return arm_lpae_unmap_pages(ops, iova, size, 1, gather);
-}
-
static phys_addr_t arm_lpae_iova_to_phys(struct io_pgtable_ops *ops,
unsigned long iova)
{
@@ -799,9 +786,7 @@ arm_lpae_alloc_pgtable(struct io_pgtable_cfg *cfg)
data->pgd_bits = va_bits - (data->bits_per_level * (levels - 1));
data->iop.ops = (struct io_pgtable_ops) {
- .map = arm_lpae_map,
.map_pages = arm_lpae_map_pages,
- .unmap = arm_lpae_unmap,
.unmap_pages = arm_lpae_unmap_pages,
.iova_to_phys = arm_lpae_iova_to_phys,
};
@@ -1176,7 +1161,7 @@ static int __init arm_lpae_run_tests(struct io_pgtable_cfg *cfg)
int i, j;
unsigned long iova;
- size_t size;
+ size_t size, mapped;
struct io_pgtable_ops *ops;
selftest_running = true;
@@ -1209,15 +1194,16 @@ static int __init arm_lpae_run_tests(struct io_pgtable_cfg *cfg)
for_each_set_bit(j, &cfg->pgsize_bitmap, BITS_PER_LONG) {
size = 1UL << j;
- if (ops->map(ops, iova, iova, size, IOMMU_READ |
- IOMMU_WRITE |
- IOMMU_NOEXEC |
- IOMMU_CACHE, GFP_KERNEL))
+ if (ops->map_pages(ops, iova, iova, size, 1,
+ IOMMU_READ | IOMMU_WRITE |
+ IOMMU_NOEXEC | IOMMU_CACHE,
+ GFP_KERNEL, &mapped))
return __FAIL(ops, i);
/* Overlapping mappings */
- if (!ops->map(ops, iova, iova + size, size,
- IOMMU_READ | IOMMU_NOEXEC, GFP_KERNEL))
+ if (!ops->map_pages(ops, iova, iova + size, size, 1,
+ IOMMU_READ | IOMMU_NOEXEC,
+ GFP_KERNEL, &mapped))
return __FAIL(ops, i);
if (ops->iova_to_phys(ops, iova + 42) != (iova + 42))
@@ -1228,11 +1214,12 @@ static int __init arm_lpae_run_tests(struct io_pgtable_cfg *cfg)
/* Partial unmap */
size = 1UL << __ffs(cfg->pgsize_bitmap);
- if (ops->unmap(ops, SZ_1G + size, size, NULL) != size)
+ if (ops->unmap_pages(ops, SZ_1G + size, size, 1, NULL) != size)
return __FAIL(ops, i);
/* Remap of partial unmap */
- if (ops->map(ops, SZ_1G + size, size, size, IOMMU_READ, GFP_KERNEL))
+ if (ops->map_pages(ops, SZ_1G + size, size, size, 1,
+ IOMMU_READ, GFP_KERNEL, &mapped))
return __FAIL(ops, i);
if (ops->iova_to_phys(ops, SZ_1G + size + 42) != (size + 42))
@@ -1243,14 +1230,15 @@ static int __init arm_lpae_run_tests(struct io_pgtable_cfg *cfg)
for_each_set_bit(j, &cfg->pgsize_bitmap, BITS_PER_LONG) {
size = 1UL << j;
- if (ops->unmap(ops, iova, size, NULL) != size)
+ if (ops->unmap_pages(ops, iova, size, 1, NULL) != size)
return __FAIL(ops, i);
if (ops->iova_to_phys(ops, iova + 42))
return __FAIL(ops, i);
/* Remap full block */
- if (ops->map(ops, iova, iova, size, IOMMU_WRITE, GFP_KERNEL))
+ if (ops->map_pages(ops, iova, iova, size, 1,
+ IOMMU_WRITE, GFP_KERNEL, &mapped))
return __FAIL(ops, i);
if (ops->iova_to_phys(ops, iova + 42) != (iova + 42))
diff --git a/drivers/iommu/iommu.c b/drivers/iommu/iommu.c
index d69ebba..de91dd8 100644
--- a/drivers/iommu/iommu.c
+++ b/drivers/iommu/iommu.c
@@ -306,13 +306,23 @@ static int __iommu_probe_device(struct device *dev, struct list_head *group_list
const struct iommu_ops *ops = dev->bus->iommu_ops;
struct iommu_device *iommu_dev;
struct iommu_group *group;
+ static DEFINE_MUTEX(iommu_probe_device_lock);
int ret;
if (!ops)
return -ENODEV;
-
- if (!dev_iommu_get(dev))
- return -ENOMEM;
+ /*
+ * Serialise to avoid races between IOMMU drivers registering in
+ * parallel and/or the "replay" calls from ACPI/OF code via client
+ * driver probe. Once the latter have been cleaned up we should
+ * probably be able to use device_lock() here to minimise the scope,
+ * but for now enforcing a simple global ordering is fine.
+ */
+ mutex_lock(&iommu_probe_device_lock);
+ if (!dev_iommu_get(dev)) {
+ ret = -ENOMEM;
+ goto err_unlock;
+ }
if (!try_module_get(ops->owner)) {
ret = -EINVAL;
@@ -333,11 +343,14 @@ static int __iommu_probe_device(struct device *dev, struct list_head *group_list
ret = PTR_ERR(group);
goto out_release;
}
- iommu_group_put(group);
+ mutex_lock(&group->mutex);
if (group_list && !group->default_domain && list_empty(&group->entry))
list_add_tail(&group->entry, group_list);
+ mutex_unlock(&group->mutex);
+ iommu_group_put(group);
+ mutex_unlock(&iommu_probe_device_lock);
iommu_device_link(iommu_dev, dev);
return 0;
@@ -352,6 +365,9 @@ static int __iommu_probe_device(struct device *dev, struct list_head *group_list
err_free:
dev_iommu_free(dev);
+err_unlock:
+ mutex_unlock(&iommu_probe_device_lock);
+
return ret;
}
@@ -1824,11 +1840,11 @@ int bus_iommu_probe(struct bus_type *bus)
return ret;
list_for_each_entry_safe(group, next, &group_list, entry) {
+ mutex_lock(&group->mutex);
+
/* Remove item from the list */
list_del_init(&group->entry);
- mutex_lock(&group->mutex);
-
/* Try to allocate default domain */
probe_alloc_default_domain(bus, group);
diff --git a/drivers/iommu/ipmmu-vmsa.c b/drivers/iommu/ipmmu-vmsa.c
index 22230cc..a003bd5 100644
--- a/drivers/iommu/ipmmu-vmsa.c
+++ b/drivers/iommu/ipmmu-vmsa.c
@@ -659,22 +659,22 @@ static void ipmmu_detach_device(struct iommu_domain *io_domain,
}
static int ipmmu_map(struct iommu_domain *io_domain, unsigned long iova,
- phys_addr_t paddr, size_t size, int prot, gfp_t gfp)
+ phys_addr_t paddr, size_t pgsize, size_t pgcount,
+ int prot, gfp_t gfp, size_t *mapped)
{
struct ipmmu_vmsa_domain *domain = to_vmsa_domain(io_domain);
- if (!domain)
- return -ENODEV;
-
- return domain->iop->map(domain->iop, iova, paddr, size, prot, gfp);
+ return domain->iop->map_pages(domain->iop, iova, paddr, pgsize, pgcount,
+ prot, gfp, mapped);
}
static size_t ipmmu_unmap(struct iommu_domain *io_domain, unsigned long iova,
- size_t size, struct iommu_iotlb_gather *gather)
+ size_t pgsize, size_t pgcount,
+ struct iommu_iotlb_gather *gather)
{
struct ipmmu_vmsa_domain *domain = to_vmsa_domain(io_domain);
- return domain->iop->unmap(domain->iop, iova, size, gather);
+ return domain->iop->unmap_pages(domain->iop, iova, pgsize, pgcount, gather);
}
static void ipmmu_flush_iotlb_all(struct iommu_domain *io_domain)
@@ -877,8 +877,8 @@ static const struct iommu_ops ipmmu_ops = {
.default_domain_ops = &(const struct iommu_domain_ops) {
.attach_dev = ipmmu_attach_device,
.detach_dev = ipmmu_detach_device,
- .map = ipmmu_map,
- .unmap = ipmmu_unmap,
+ .map_pages = ipmmu_map,
+ .unmap_pages = ipmmu_unmap,
.flush_iotlb_all = ipmmu_flush_iotlb_all,
.iotlb_sync = ipmmu_iotlb_sync,
.iova_to_phys = ipmmu_iova_to_phys,
diff --git a/drivers/iommu/msm_iommu.c b/drivers/iommu/msm_iommu.c
index 16179a9..c606249 100644
--- a/drivers/iommu/msm_iommu.c
+++ b/drivers/iommu/msm_iommu.c
@@ -471,14 +471,16 @@ static void msm_iommu_detach_dev(struct iommu_domain *domain,
}
static int msm_iommu_map(struct iommu_domain *domain, unsigned long iova,
- phys_addr_t pa, size_t len, int prot, gfp_t gfp)
+ phys_addr_t pa, size_t pgsize, size_t pgcount,
+ int prot, gfp_t gfp, size_t *mapped)
{
struct msm_priv *priv = to_msm_priv(domain);
unsigned long flags;
int ret;
spin_lock_irqsave(&priv->pgtlock, flags);
- ret = priv->iop->map(priv->iop, iova, pa, len, prot, GFP_ATOMIC);
+ ret = priv->iop->map_pages(priv->iop, iova, pa, pgsize, pgcount, prot,
+ GFP_ATOMIC, mapped);
spin_unlock_irqrestore(&priv->pgtlock, flags);
return ret;
@@ -493,16 +495,18 @@ static void msm_iommu_sync_map(struct iommu_domain *domain, unsigned long iova,
}
static size_t msm_iommu_unmap(struct iommu_domain *domain, unsigned long iova,
- size_t len, struct iommu_iotlb_gather *gather)
+ size_t pgsize, size_t pgcount,
+ struct iommu_iotlb_gather *gather)
{
struct msm_priv *priv = to_msm_priv(domain);
unsigned long flags;
+ size_t ret;
spin_lock_irqsave(&priv->pgtlock, flags);
- len = priv->iop->unmap(priv->iop, iova, len, gather);
+ ret = priv->iop->unmap_pages(priv->iop, iova, pgsize, pgcount, gather);
spin_unlock_irqrestore(&priv->pgtlock, flags);
- return len;
+ return ret;
}
static phys_addr_t msm_iommu_iova_to_phys(struct iommu_domain *domain,
@@ -679,8 +683,8 @@ static struct iommu_ops msm_iommu_ops = {
.default_domain_ops = &(const struct iommu_domain_ops) {
.attach_dev = msm_iommu_attach_dev,
.detach_dev = msm_iommu_detach_dev,
- .map = msm_iommu_map,
- .unmap = msm_iommu_unmap,
+ .map_pages = msm_iommu_map,
+ .unmap_pages = msm_iommu_unmap,
/*
* Nothing is needed here, the barrier to guarantee
* completion of the tlb sync operation is implicitly
diff --git a/drivers/iommu/mtk_iommu.c b/drivers/iommu/mtk_iommu.c
index b383c83..2badd6a 100644
--- a/drivers/iommu/mtk_iommu.c
+++ b/drivers/iommu/mtk_iommu.c
@@ -108,8 +108,12 @@
#define F_MMU_INT_ID_SUB_COMM_ID(a) (((a) >> 7) & 0x3)
#define F_MMU_INT_ID_COMM_ID_EXT(a) (((a) >> 10) & 0x7)
#define F_MMU_INT_ID_SUB_COMM_ID_EXT(a) (((a) >> 7) & 0x7)
+/* Macro for 5 bits length port ID field (default) */
#define F_MMU_INT_ID_LARB_ID(a) (((a) >> 7) & 0x7)
#define F_MMU_INT_ID_PORT_ID(a) (((a) >> 2) & 0x1f)
+/* Macro for 6 bits length port ID field */
+#define F_MMU_INT_ID_LARB_ID_WID_6(a) (((a) >> 8) & 0x7)
+#define F_MMU_INT_ID_PORT_ID_WID_6(a) (((a) >> 2) & 0x3f)
#define MTK_PROTECT_PA_ALIGN 256
#define MTK_IOMMU_BANK_SZ 0x1000
@@ -139,6 +143,7 @@
#define IFA_IOMMU_PCIE_SUPPORT BIT(16)
#define PGTABLE_PA_35_EN BIT(17)
#define TF_PORT_TO_ADDR_MT8173 BIT(18)
+#define INT_ID_PORT_WIDTH_6 BIT(19)
#define MTK_IOMMU_HAS_FLAG_MASK(pdata, _x, mask) \
((((pdata)->flags) & (mask)) == (_x))
@@ -165,6 +170,7 @@ enum mtk_iommu_plat {
M4U_MT8186,
M4U_MT8192,
M4U_MT8195,
+ M4U_MT8365,
};
struct mtk_iommu_iova_region {
@@ -223,10 +229,7 @@ struct mtk_iommu_data {
struct device *smicomm_dev;
struct mtk_iommu_bank_data *bank;
-
- struct dma_iommu_mapping *mapping; /* For mtk_iommu_v1.c */
struct regmap *pericfg;
-
struct mutex mutex; /* Protect m4u_group/m4u_dom above */
/*
@@ -441,20 +444,25 @@ static irqreturn_t mtk_iommu_isr(int irq, void *dev_id)
fault_pa |= (u64)pa34_32 << 32;
if (MTK_IOMMU_IS_TYPE(plat_data, MTK_IOMMU_TYPE_MM)) {
- fault_port = F_MMU_INT_ID_PORT_ID(regval);
if (MTK_IOMMU_HAS_FLAG(plat_data, HAS_SUB_COMM_2BITS)) {
fault_larb = F_MMU_INT_ID_COMM_ID(regval);
sub_comm = F_MMU_INT_ID_SUB_COMM_ID(regval);
+ fault_port = F_MMU_INT_ID_PORT_ID(regval);
} else if (MTK_IOMMU_HAS_FLAG(plat_data, HAS_SUB_COMM_3BITS)) {
fault_larb = F_MMU_INT_ID_COMM_ID_EXT(regval);
sub_comm = F_MMU_INT_ID_SUB_COMM_ID_EXT(regval);
+ fault_port = F_MMU_INT_ID_PORT_ID(regval);
+ } else if (MTK_IOMMU_HAS_FLAG(plat_data, INT_ID_PORT_WIDTH_6)) {
+ fault_port = F_MMU_INT_ID_PORT_ID_WID_6(regval);
+ fault_larb = F_MMU_INT_ID_LARB_ID_WID_6(regval);
} else {
+ fault_port = F_MMU_INT_ID_PORT_ID(regval);
fault_larb = F_MMU_INT_ID_LARB_ID(regval);
}
fault_larb = data->plat_data->larbid_remap[fault_larb][sub_comm];
}
- if (report_iommu_fault(&dom->domain, bank->parent_dev, fault_iova,
+ if (!dom || report_iommu_fault(&dom->domain, bank->parent_dev, fault_iova,
write ? IOMMU_FAULT_WRITE : IOMMU_FAULT_READ)) {
dev_err_ratelimited(
bank->parent_dev,
@@ -711,7 +719,8 @@ static void mtk_iommu_detach_device(struct iommu_domain *domain,
}
static int mtk_iommu_map(struct iommu_domain *domain, unsigned long iova,
- phys_addr_t paddr, size_t size, int prot, gfp_t gfp)
+ phys_addr_t paddr, size_t pgsize, size_t pgcount,
+ int prot, gfp_t gfp, size_t *mapped)
{
struct mtk_iommu_domain *dom = to_mtk_domain(domain);
@@ -720,17 +729,17 @@ static int mtk_iommu_map(struct iommu_domain *domain, unsigned long iova,
paddr |= BIT_ULL(32);
/* Synchronize with the tlb_lock */
- return dom->iop->map(dom->iop, iova, paddr, size, prot, gfp);
+ return dom->iop->map_pages(dom->iop, iova, paddr, pgsize, pgcount, prot, gfp, mapped);
}
static size_t mtk_iommu_unmap(struct iommu_domain *domain,
- unsigned long iova, size_t size,
+ unsigned long iova, size_t pgsize, size_t pgcount,
struct iommu_iotlb_gather *gather)
{
struct mtk_iommu_domain *dom = to_mtk_domain(domain);
- iommu_iotlb_gather_add_range(gather, iova, size);
- return dom->iop->unmap(dom->iop, iova, size, gather);
+ iommu_iotlb_gather_add_range(gather, iova, pgsize * pgcount);
+ return dom->iop->unmap_pages(dom->iop, iova, pgsize, pgcount, gather);
}
static void mtk_iommu_flush_iotlb_all(struct iommu_domain *domain)
@@ -938,8 +947,8 @@ static const struct iommu_ops mtk_iommu_ops = {
.default_domain_ops = &(const struct iommu_domain_ops) {
.attach_dev = mtk_iommu_attach_device,
.detach_dev = mtk_iommu_detach_device,
- .map = mtk_iommu_map,
- .unmap = mtk_iommu_unmap,
+ .map_pages = mtk_iommu_map,
+ .unmap_pages = mtk_iommu_unmap,
.flush_iotlb_all = mtk_iommu_flush_iotlb_all,
.iotlb_sync = mtk_iommu_iotlb_sync,
.iotlb_sync_map = mtk_iommu_sync_map,
@@ -1043,21 +1052,26 @@ static const struct component_master_ops mtk_iommu_com_ops = {
static int mtk_iommu_mm_dts_parse(struct device *dev, struct component_match **match,
struct mtk_iommu_data *data)
{
- struct device_node *larbnode, *smicomm_node, *smi_subcomm_node;
- struct platform_device *plarbdev;
+ struct device_node *larbnode, *frst_avail_smicomm_node = NULL;
+ struct platform_device *plarbdev, *pcommdev;
struct device_link *link;
int i, larb_nr, ret;
larb_nr = of_count_phandle_with_args(dev->of_node, "mediatek,larbs", NULL);
if (larb_nr < 0)
return larb_nr;
+ if (larb_nr == 0 || larb_nr > MTK_LARB_NR_MAX)
+ return -EINVAL;
for (i = 0; i < larb_nr; i++) {
+ struct device_node *smicomm_node, *smi_subcomm_node;
u32 id;
larbnode = of_parse_phandle(dev->of_node, "mediatek,larbs", i);
- if (!larbnode)
- return -EINVAL;
+ if (!larbnode) {
+ ret = -EINVAL;
+ goto err_larbdev_put;
+ }
if (!of_device_is_available(larbnode)) {
of_node_put(larbnode);
@@ -1067,48 +1081,91 @@ static int mtk_iommu_mm_dts_parse(struct device *dev, struct component_match **m
ret = of_property_read_u32(larbnode, "mediatek,larb-id", &id);
if (ret)/* The id is consecutive if there is no this property */
id = i;
+ if (id >= MTK_LARB_NR_MAX) {
+ of_node_put(larbnode);
+ ret = -EINVAL;
+ goto err_larbdev_put;
+ }
plarbdev = of_find_device_by_node(larbnode);
+ of_node_put(larbnode);
if (!plarbdev) {
- of_node_put(larbnode);
- return -ENODEV;
+ ret = -ENODEV;
+ goto err_larbdev_put;
}
- if (!plarbdev->dev.driver) {
- of_node_put(larbnode);
- return -EPROBE_DEFER;
+ if (data->larb_imu[id].dev) {
+ platform_device_put(plarbdev);
+ ret = -EEXIST;
+ goto err_larbdev_put;
}
data->larb_imu[id].dev = &plarbdev->dev;
- component_match_add_release(dev, match, component_release_of,
- component_compare_of, larbnode);
+ if (!plarbdev->dev.driver) {
+ ret = -EPROBE_DEFER;
+ goto err_larbdev_put;
+ }
+
+ /* Get smi-(sub)-common dev from the last larb. */
+ smi_subcomm_node = of_parse_phandle(larbnode, "mediatek,smi", 0);
+ if (!smi_subcomm_node) {
+ ret = -EINVAL;
+ goto err_larbdev_put;
+ }
+
+ /*
+ * It may have two level smi-common. the node is smi-sub-common if it
+ * has a new mediatek,smi property. otherwise it is smi-commmon.
+ */
+ smicomm_node = of_parse_phandle(smi_subcomm_node, "mediatek,smi", 0);
+ if (smicomm_node)
+ of_node_put(smi_subcomm_node);
+ else
+ smicomm_node = smi_subcomm_node;
+
+ /*
+ * All the larbs that connect to one IOMMU must connect with the same
+ * smi-common.
+ */
+ if (!frst_avail_smicomm_node) {
+ frst_avail_smicomm_node = smicomm_node;
+ } else if (frst_avail_smicomm_node != smicomm_node) {
+ dev_err(dev, "mediatek,smi property is not right @larb%d.", id);
+ of_node_put(smicomm_node);
+ ret = -EINVAL;
+ goto err_larbdev_put;
+ } else {
+ of_node_put(smicomm_node);
+ }
+
+ component_match_add(dev, match, component_compare_dev, &plarbdev->dev);
+ platform_device_put(plarbdev);
}
- /* Get smi-(sub)-common dev from the last larb. */
- smi_subcomm_node = of_parse_phandle(larbnode, "mediatek,smi", 0);
- if (!smi_subcomm_node)
+ if (!frst_avail_smicomm_node)
return -EINVAL;
- /*
- * It may have two level smi-common. the node is smi-sub-common if it
- * has a new mediatek,smi property. otherwise it is smi-commmon.
- */
- smicomm_node = of_parse_phandle(smi_subcomm_node, "mediatek,smi", 0);
- if (smicomm_node)
- of_node_put(smi_subcomm_node);
- else
- smicomm_node = smi_subcomm_node;
-
- plarbdev = of_find_device_by_node(smicomm_node);
- of_node_put(smicomm_node);
- data->smicomm_dev = &plarbdev->dev;
+ pcommdev = of_find_device_by_node(frst_avail_smicomm_node);
+ of_node_put(frst_avail_smicomm_node);
+ if (!pcommdev)
+ return -ENODEV;
+ data->smicomm_dev = &pcommdev->dev;
link = device_link_add(data->smicomm_dev, dev,
DL_FLAG_STATELESS | DL_FLAG_PM_RUNTIME);
+ platform_device_put(pcommdev);
if (!link) {
dev_err(dev, "Unable to link %s.\n", dev_name(data->smicomm_dev));
return -EINVAL;
}
return 0;
+
+err_larbdev_put:
+ for (i = MTK_LARB_NR_MAX - 1; i >= 0; i--) {
+ if (!data->larb_imu[i].dev)
+ continue;
+ put_device(data->larb_imu[i].dev);
+ }
+ return ret;
}
static int mtk_iommu_probe(struct platform_device *pdev)
@@ -1173,6 +1230,8 @@ static int mtk_iommu_probe(struct platform_device *pdev)
banks_num = data->plat_data->banks_num;
res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
+ if (!res)
+ return -EINVAL;
if (resource_size(res) < banks_num * MTK_IOMMU_BANK_SZ) {
dev_err(dev, "banknr %d. res %pR is not enough.\n", banks_num, res);
return -EINVAL;
@@ -1516,6 +1575,17 @@ static const struct mtk_iommu_plat_data mt8195_data_vpp = {
{4, MTK_INVALID_LARBID, MTK_INVALID_LARBID, MTK_INVALID_LARBID, 6}},
};
+static const struct mtk_iommu_plat_data mt8365_data = {
+ .m4u_plat = M4U_MT8365,
+ .flags = RESET_AXI | INT_ID_PORT_WIDTH_6,
+ .inv_sel_reg = REG_MMU_INV_SEL_GEN1,
+ .banks_num = 1,
+ .banks_enable = {true},
+ .iova_region = single_domain,
+ .iova_region_nr = ARRAY_SIZE(single_domain),
+ .larbid_remap = {{0}, {1}, {2}, {3}, {4}, {5}}, /* Linear mapping. */
+};
+
static const struct of_device_id mtk_iommu_of_ids[] = {
{ .compatible = "mediatek,mt2712-m4u", .data = &mt2712_data},
{ .compatible = "mediatek,mt6779-m4u", .data = &mt6779_data},
@@ -1528,6 +1598,7 @@ static const struct of_device_id mtk_iommu_of_ids[] = {
{ .compatible = "mediatek,mt8195-iommu-infra", .data = &mt8195_data_infra},
{ .compatible = "mediatek,mt8195-iommu-vdo", .data = &mt8195_data_vdo},
{ .compatible = "mediatek,mt8195-iommu-vpp", .data = &mt8195_data_vpp},
+ { .compatible = "mediatek,mt8365-m4u", .data = &mt8365_data},
{}
};
diff --git a/drivers/iommu/mtk_iommu_v1.c b/drivers/iommu/mtk_iommu_v1.c
index 6e0e658..69682ee 100644
--- a/drivers/iommu/mtk_iommu_v1.c
+++ b/drivers/iommu/mtk_iommu_v1.c
@@ -327,44 +327,42 @@ static void mtk_iommu_v1_detach_device(struct iommu_domain *domain, struct devic
}
static int mtk_iommu_v1_map(struct iommu_domain *domain, unsigned long iova,
- phys_addr_t paddr, size_t size, int prot, gfp_t gfp)
+ phys_addr_t paddr, size_t pgsize, size_t pgcount,
+ int prot, gfp_t gfp, size_t *mapped)
{
struct mtk_iommu_v1_domain *dom = to_mtk_domain(domain);
- unsigned int page_num = size >> MT2701_IOMMU_PAGE_SHIFT;
unsigned long flags;
unsigned int i;
u32 *pgt_base_iova = dom->pgt_va + (iova >> MT2701_IOMMU_PAGE_SHIFT);
u32 pabase = (u32)paddr;
- int map_size = 0;
spin_lock_irqsave(&dom->pgtlock, flags);
- for (i = 0; i < page_num; i++) {
- if (pgt_base_iova[i]) {
- memset(pgt_base_iova, 0, i * sizeof(u32));
+ for (i = 0; i < pgcount; i++) {
+ if (pgt_base_iova[i])
break;
- }
pgt_base_iova[i] = pabase | F_DESC_VALID | F_DESC_NONSEC;
pabase += MT2701_IOMMU_PAGE_SIZE;
- map_size += MT2701_IOMMU_PAGE_SIZE;
}
spin_unlock_irqrestore(&dom->pgtlock, flags);
- mtk_iommu_v1_tlb_flush_range(dom->data, iova, size);
+ *mapped = i * MT2701_IOMMU_PAGE_SIZE;
+ mtk_iommu_v1_tlb_flush_range(dom->data, iova, *mapped);
- return map_size == size ? 0 : -EEXIST;
+ return i == pgcount ? 0 : -EEXIST;
}
static size_t mtk_iommu_v1_unmap(struct iommu_domain *domain, unsigned long iova,
- size_t size, struct iommu_iotlb_gather *gather)
+ size_t pgsize, size_t pgcount,
+ struct iommu_iotlb_gather *gather)
{
struct mtk_iommu_v1_domain *dom = to_mtk_domain(domain);
unsigned long flags;
u32 *pgt_base_iova = dom->pgt_va + (iova >> MT2701_IOMMU_PAGE_SHIFT);
- unsigned int page_num = size >> MT2701_IOMMU_PAGE_SHIFT;
+ size_t size = pgcount * MT2701_IOMMU_PAGE_SIZE;
spin_lock_irqsave(&dom->pgtlock, flags);
- memset(pgt_base_iova, 0, page_num * sizeof(u32));
+ memset(pgt_base_iova, 0, pgcount * sizeof(u32));
spin_unlock_irqrestore(&dom->pgtlock, flags);
mtk_iommu_v1_tlb_flush_range(dom->data, iova, size);
@@ -586,13 +584,13 @@ static const struct iommu_ops mtk_iommu_v1_ops = {
.release_device = mtk_iommu_v1_release_device,
.def_domain_type = mtk_iommu_v1_def_domain_type,
.device_group = generic_device_group,
- .pgsize_bitmap = ~0UL << MT2701_IOMMU_PAGE_SHIFT,
+ .pgsize_bitmap = MT2701_IOMMU_PAGE_SIZE,
.owner = THIS_MODULE,
.default_domain_ops = &(const struct iommu_domain_ops) {
.attach_dev = mtk_iommu_v1_attach_device,
.detach_dev = mtk_iommu_v1_detach_device,
- .map = mtk_iommu_v1_map,
- .unmap = mtk_iommu_v1_unmap,
+ .map_pages = mtk_iommu_v1_map,
+ .unmap_pages = mtk_iommu_v1_unmap,
.iova_to_phys = mtk_iommu_v1_iova_to_phys,
.free = mtk_iommu_v1_domain_free,
}
diff --git a/drivers/iommu/rockchip-iommu.c b/drivers/iommu/rockchip-iommu.c
index a3fc59b..a68eadd 100644
--- a/drivers/iommu/rockchip-iommu.c
+++ b/drivers/iommu/rockchip-iommu.c
@@ -280,19 +280,17 @@ static u32 rk_mk_pte(phys_addr_t page, int prot)
* 11:9 - Page address bit 34:32
* 8:4 - Page address bit 39:35
* 3 - Security
- * 2 - Readable
- * 1 - Writable
+ * 2 - Writable
+ * 1 - Readable
* 0 - 1 if Page @ Page address is valid
*/
-#define RK_PTE_PAGE_READABLE_V2 BIT(2)
-#define RK_PTE_PAGE_WRITABLE_V2 BIT(1)
static u32 rk_mk_pte_v2(phys_addr_t page, int prot)
{
u32 flags = 0;
- flags |= (prot & IOMMU_READ) ? RK_PTE_PAGE_READABLE_V2 : 0;
- flags |= (prot & IOMMU_WRITE) ? RK_PTE_PAGE_WRITABLE_V2 : 0;
+ flags |= (prot & IOMMU_READ) ? RK_PTE_PAGE_READABLE : 0;
+ flags |= (prot & IOMMU_WRITE) ? RK_PTE_PAGE_WRITABLE : 0;
return rk_mk_dte_v2(page) | flags;
}
diff --git a/drivers/iommu/s390-iommu.c b/drivers/iommu/s390-iommu.c
index 3c07178..ed33c6c 100644
--- a/drivers/iommu/s390-iommu.c
+++ b/drivers/iommu/s390-iommu.c
@@ -10,28 +10,18 @@
#include <linux/iommu.h>
#include <linux/iommu-helper.h>
#include <linux/sizes.h>
+#include <linux/rculist.h>
+#include <linux/rcupdate.h>
#include <asm/pci_dma.h>
-/*
- * Physically contiguous memory regions can be mapped with 4 KiB alignment,
- * we allow all page sizes that are an order of 4KiB (no special large page
- * support so far).
- */
-#define S390_IOMMU_PGSIZES (~0xFFFUL)
-
static const struct iommu_ops s390_iommu_ops;
struct s390_domain {
struct iommu_domain domain;
struct list_head devices;
unsigned long *dma_table;
- spinlock_t dma_table_lock;
spinlock_t list_lock;
-};
-
-struct s390_domain_device {
- struct list_head list;
- struct zpci_dev *zdev;
+ struct rcu_head rcu;
};
static struct s390_domain *to_s390_domain(struct iommu_domain *dom)
@@ -67,20 +57,50 @@ static struct iommu_domain *s390_domain_alloc(unsigned domain_type)
kfree(s390_domain);
return NULL;
}
+ s390_domain->domain.geometry.force_aperture = true;
+ s390_domain->domain.geometry.aperture_start = 0;
+ s390_domain->domain.geometry.aperture_end = ZPCI_TABLE_SIZE_RT - 1;
- spin_lock_init(&s390_domain->dma_table_lock);
spin_lock_init(&s390_domain->list_lock);
- INIT_LIST_HEAD(&s390_domain->devices);
+ INIT_LIST_HEAD_RCU(&s390_domain->devices);
return &s390_domain->domain;
}
+static void s390_iommu_rcu_free_domain(struct rcu_head *head)
+{
+ struct s390_domain *s390_domain = container_of(head, struct s390_domain, rcu);
+
+ dma_cleanup_tables(s390_domain->dma_table);
+ kfree(s390_domain);
+}
+
static void s390_domain_free(struct iommu_domain *domain)
{
struct s390_domain *s390_domain = to_s390_domain(domain);
- dma_cleanup_tables(s390_domain->dma_table);
- kfree(s390_domain);
+ rcu_read_lock();
+ WARN_ON(!list_empty(&s390_domain->devices));
+ rcu_read_unlock();
+
+ call_rcu(&s390_domain->rcu, s390_iommu_rcu_free_domain);
+}
+
+static void __s390_iommu_detach_device(struct zpci_dev *zdev)
+{
+ struct s390_domain *s390_domain = zdev->s390_domain;
+ unsigned long flags;
+
+ if (!s390_domain)
+ return;
+
+ spin_lock_irqsave(&s390_domain->list_lock, flags);
+ list_del_rcu(&zdev->iommu_list);
+ spin_unlock_irqrestore(&s390_domain->list_lock, flags);
+
+ zpci_unregister_ioat(zdev, 0);
+ zdev->s390_domain = NULL;
+ zdev->dma_table = NULL;
}
static int s390_iommu_attach_device(struct iommu_domain *domain,
@@ -88,98 +108,74 @@ static int s390_iommu_attach_device(struct iommu_domain *domain,
{
struct s390_domain *s390_domain = to_s390_domain(domain);
struct zpci_dev *zdev = to_zpci_dev(dev);
- struct s390_domain_device *domain_device;
unsigned long flags;
- int cc, rc;
+ u8 status;
+ int cc;
if (!zdev)
return -ENODEV;
- domain_device = kzalloc(sizeof(*domain_device), GFP_KERNEL);
- if (!domain_device)
- return -ENOMEM;
-
- if (zdev->dma_table && !zdev->s390_domain) {
- cc = zpci_dma_exit_device(zdev);
- if (cc) {
- rc = -EIO;
- goto out_free;
- }
- }
+ if (WARN_ON(domain->geometry.aperture_start > zdev->end_dma ||
+ domain->geometry.aperture_end < zdev->start_dma))
+ return -EINVAL;
if (zdev->s390_domain)
- zpci_unregister_ioat(zdev, 0);
+ __s390_iommu_detach_device(zdev);
+ else if (zdev->dma_table)
+ zpci_dma_exit_device(zdev);
+
+ cc = zpci_register_ioat(zdev, 0, zdev->start_dma, zdev->end_dma,
+ virt_to_phys(s390_domain->dma_table), &status);
+ /*
+ * If the device is undergoing error recovery the reset code
+ * will re-establish the new domain.
+ */
+ if (cc && status != ZPCI_PCI_ST_FUNC_NOT_AVAIL)
+ return -EIO;
+ zdev->dma_table = s390_domain->dma_table;
zdev->dma_table = s390_domain->dma_table;
- cc = zpci_register_ioat(zdev, 0, zdev->start_dma, zdev->end_dma,
- virt_to_phys(zdev->dma_table));
- if (cc) {
- rc = -EIO;
- goto out_restore;
- }
+ zdev->s390_domain = s390_domain;
spin_lock_irqsave(&s390_domain->list_lock, flags);
- /* First device defines the DMA range limits */
- if (list_empty(&s390_domain->devices)) {
- domain->geometry.aperture_start = zdev->start_dma;
- domain->geometry.aperture_end = zdev->end_dma;
- domain->geometry.force_aperture = true;
- /* Allow only devices with identical DMA range limits */
- } else if (domain->geometry.aperture_start != zdev->start_dma ||
- domain->geometry.aperture_end != zdev->end_dma) {
- rc = -EINVAL;
- spin_unlock_irqrestore(&s390_domain->list_lock, flags);
- goto out_restore;
- }
- domain_device->zdev = zdev;
- zdev->s390_domain = s390_domain;
- list_add(&domain_device->list, &s390_domain->devices);
+ list_add_rcu(&zdev->iommu_list, &s390_domain->devices);
spin_unlock_irqrestore(&s390_domain->list_lock, flags);
return 0;
-
-out_restore:
- if (!zdev->s390_domain) {
- zpci_dma_init_device(zdev);
- } else {
- zdev->dma_table = zdev->s390_domain->dma_table;
- zpci_register_ioat(zdev, 0, zdev->start_dma, zdev->end_dma,
- virt_to_phys(zdev->dma_table));
- }
-out_free:
- kfree(domain_device);
-
- return rc;
}
static void s390_iommu_detach_device(struct iommu_domain *domain,
struct device *dev)
{
- struct s390_domain *s390_domain = to_s390_domain(domain);
struct zpci_dev *zdev = to_zpci_dev(dev);
- struct s390_domain_device *domain_device, *tmp;
- unsigned long flags;
- int found = 0;
- if (!zdev)
- return;
+ WARN_ON(zdev->s390_domain != to_s390_domain(domain));
- spin_lock_irqsave(&s390_domain->list_lock, flags);
- list_for_each_entry_safe(domain_device, tmp, &s390_domain->devices,
- list) {
- if (domain_device->zdev == zdev) {
- list_del(&domain_device->list);
- kfree(domain_device);
- found = 1;
- break;
- }
+ __s390_iommu_detach_device(zdev);
+ zpci_dma_init_device(zdev);
+}
+
+static void s390_iommu_get_resv_regions(struct device *dev,
+ struct list_head *list)
+{
+ struct zpci_dev *zdev = to_zpci_dev(dev);
+ struct iommu_resv_region *region;
+
+ if (zdev->start_dma) {
+ region = iommu_alloc_resv_region(0, zdev->start_dma, 0,
+ IOMMU_RESV_RESERVED, GFP_KERNEL);
+ if (!region)
+ return;
+ list_add_tail(®ion->list, list);
}
- spin_unlock_irqrestore(&s390_domain->list_lock, flags);
- if (found && (zdev->s390_domain == s390_domain)) {
- zdev->s390_domain = NULL;
- zpci_unregister_ioat(zdev, 0);
- zpci_dma_init_device(zdev);
+ if (zdev->end_dma < ZPCI_TABLE_SIZE_RT - 1) {
+ region = iommu_alloc_resv_region(zdev->end_dma + 1,
+ ZPCI_TABLE_SIZE_RT - zdev->end_dma - 1,
+ 0, IOMMU_RESV_RESERVED, GFP_KERNEL);
+ if (!region)
+ return;
+ list_add_tail(®ion->list, list);
}
}
@@ -192,55 +188,88 @@ static struct iommu_device *s390_iommu_probe_device(struct device *dev)
zdev = to_zpci_dev(dev);
+ if (zdev->start_dma > zdev->end_dma ||
+ zdev->start_dma > ZPCI_TABLE_SIZE_RT - 1)
+ return ERR_PTR(-EINVAL);
+
+ if (zdev->end_dma > ZPCI_TABLE_SIZE_RT - 1)
+ zdev->end_dma = ZPCI_TABLE_SIZE_RT - 1;
+
return &zdev->iommu_dev;
}
static void s390_iommu_release_device(struct device *dev)
{
struct zpci_dev *zdev = to_zpci_dev(dev);
- struct iommu_domain *domain;
/*
- * This is a workaround for a scenario where the IOMMU API common code
- * "forgets" to call the detach_dev callback: After binding a device
- * to vfio-pci and completing the VFIO_SET_IOMMU ioctl (which triggers
- * the attach_dev), removing the device via
- * "echo 1 > /sys/bus/pci/devices/.../remove" won't trigger detach_dev,
- * only release_device will be called via the BUS_NOTIFY_REMOVED_DEVICE
- * notifier.
- *
- * So let's call detach_dev from here if it hasn't been called before.
+ * release_device is expected to detach any domain currently attached
+ * to the device, but keep it attached to other devices in the group.
*/
- if (zdev && zdev->s390_domain) {
- domain = iommu_get_domain_for_dev(dev);
- if (domain)
- s390_iommu_detach_device(domain, dev);
- }
+ if (zdev)
+ __s390_iommu_detach_device(zdev);
}
-static int s390_iommu_update_trans(struct s390_domain *s390_domain,
- phys_addr_t pa, dma_addr_t dma_addr,
- size_t size, int flags)
+static void s390_iommu_flush_iotlb_all(struct iommu_domain *domain)
{
- struct s390_domain_device *domain_device;
+ struct s390_domain *s390_domain = to_s390_domain(domain);
+ struct zpci_dev *zdev;
+
+ rcu_read_lock();
+ list_for_each_entry_rcu(zdev, &s390_domain->devices, iommu_list) {
+ zpci_refresh_trans((u64)zdev->fh << 32, zdev->start_dma,
+ zdev->end_dma - zdev->start_dma + 1);
+ }
+ rcu_read_unlock();
+}
+
+static void s390_iommu_iotlb_sync(struct iommu_domain *domain,
+ struct iommu_iotlb_gather *gather)
+{
+ struct s390_domain *s390_domain = to_s390_domain(domain);
+ size_t size = gather->end - gather->start + 1;
+ struct zpci_dev *zdev;
+
+ /* If gather was never added to there is nothing to flush */
+ if (!gather->end)
+ return;
+
+ rcu_read_lock();
+ list_for_each_entry_rcu(zdev, &s390_domain->devices, iommu_list) {
+ zpci_refresh_trans((u64)zdev->fh << 32, gather->start,
+ size);
+ }
+ rcu_read_unlock();
+}
+
+static void s390_iommu_iotlb_sync_map(struct iommu_domain *domain,
+ unsigned long iova, size_t size)
+{
+ struct s390_domain *s390_domain = to_s390_domain(domain);
+ struct zpci_dev *zdev;
+
+ rcu_read_lock();
+ list_for_each_entry_rcu(zdev, &s390_domain->devices, iommu_list) {
+ if (!zdev->tlb_refresh)
+ continue;
+ zpci_refresh_trans((u64)zdev->fh << 32,
+ iova, size);
+ }
+ rcu_read_unlock();
+}
+
+static int s390_iommu_validate_trans(struct s390_domain *s390_domain,
+ phys_addr_t pa, dma_addr_t dma_addr,
+ unsigned long nr_pages, int flags)
+{
phys_addr_t page_addr = pa & PAGE_MASK;
- dma_addr_t start_dma_addr = dma_addr;
- unsigned long irq_flags, nr_pages, i;
unsigned long *entry;
- int rc = 0;
+ unsigned long i;
+ int rc;
- if (dma_addr < s390_domain->domain.geometry.aperture_start ||
- dma_addr + size > s390_domain->domain.geometry.aperture_end)
- return -EINVAL;
-
- nr_pages = PAGE_ALIGN(size) >> PAGE_SHIFT;
- if (!nr_pages)
- return 0;
-
- spin_lock_irqsave(&s390_domain->dma_table_lock, irq_flags);
for (i = 0; i < nr_pages; i++) {
entry = dma_walk_cpu_trans(s390_domain->dma_table, dma_addr);
- if (!entry) {
+ if (unlikely(!entry)) {
rc = -ENOMEM;
goto undo_cpu_trans;
}
@@ -249,47 +278,70 @@ static int s390_iommu_update_trans(struct s390_domain *s390_domain,
dma_addr += PAGE_SIZE;
}
- spin_lock(&s390_domain->list_lock);
- list_for_each_entry(domain_device, &s390_domain->devices, list) {
- rc = zpci_refresh_trans((u64) domain_device->zdev->fh << 32,
- start_dma_addr, nr_pages * PAGE_SIZE);
- if (rc)
- break;
- }
- spin_unlock(&s390_domain->list_lock);
+ return 0;
undo_cpu_trans:
- if (rc && ((flags & ZPCI_PTE_VALID_MASK) == ZPCI_PTE_VALID)) {
- flags = ZPCI_PTE_INVALID;
- while (i-- > 0) {
- page_addr -= PAGE_SIZE;
- dma_addr -= PAGE_SIZE;
- entry = dma_walk_cpu_trans(s390_domain->dma_table,
- dma_addr);
- if (!entry)
- break;
- dma_update_cpu_trans(entry, page_addr, flags);
- }
+ while (i-- > 0) {
+ dma_addr -= PAGE_SIZE;
+ entry = dma_walk_cpu_trans(s390_domain->dma_table,
+ dma_addr);
+ if (!entry)
+ break;
+ dma_update_cpu_trans(entry, 0, ZPCI_PTE_INVALID);
}
- spin_unlock_irqrestore(&s390_domain->dma_table_lock, irq_flags);
return rc;
}
-static int s390_iommu_map(struct iommu_domain *domain, unsigned long iova,
- phys_addr_t paddr, size_t size, int prot, gfp_t gfp)
+static int s390_iommu_invalidate_trans(struct s390_domain *s390_domain,
+ dma_addr_t dma_addr, unsigned long nr_pages)
+{
+ unsigned long *entry;
+ unsigned long i;
+ int rc = 0;
+
+ for (i = 0; i < nr_pages; i++) {
+ entry = dma_walk_cpu_trans(s390_domain->dma_table, dma_addr);
+ if (unlikely(!entry)) {
+ rc = -EINVAL;
+ break;
+ }
+ dma_update_cpu_trans(entry, 0, ZPCI_PTE_INVALID);
+ dma_addr += PAGE_SIZE;
+ }
+
+ return rc;
+}
+
+static int s390_iommu_map_pages(struct iommu_domain *domain,
+ unsigned long iova, phys_addr_t paddr,
+ size_t pgsize, size_t pgcount,
+ int prot, gfp_t gfp, size_t *mapped)
{
struct s390_domain *s390_domain = to_s390_domain(domain);
+ size_t size = pgcount << __ffs(pgsize);
int flags = ZPCI_PTE_VALID, rc = 0;
+ if (pgsize != SZ_4K)
+ return -EINVAL;
+
+ if (iova < s390_domain->domain.geometry.aperture_start ||
+ (iova + size - 1) > s390_domain->domain.geometry.aperture_end)
+ return -EINVAL;
+
+ if (!IS_ALIGNED(iova | paddr, pgsize))
+ return -EINVAL;
+
if (!(prot & IOMMU_READ))
return -EINVAL;
if (!(prot & IOMMU_WRITE))
flags |= ZPCI_TABLE_PROTECTED;
- rc = s390_iommu_update_trans(s390_domain, paddr, iova,
- size, flags);
+ rc = s390_iommu_validate_trans(s390_domain, paddr, iova,
+ pgcount, flags);
+ if (!rc)
+ *mapped = size;
return rc;
}
@@ -298,7 +350,8 @@ static phys_addr_t s390_iommu_iova_to_phys(struct iommu_domain *domain,
dma_addr_t iova)
{
struct s390_domain *s390_domain = to_s390_domain(domain);
- unsigned long *sto, *pto, *rto, flags;
+ unsigned long *rto, *sto, *pto;
+ unsigned long ste, pte, rte;
unsigned int rtx, sx, px;
phys_addr_t phys = 0;
@@ -311,38 +364,40 @@ static phys_addr_t s390_iommu_iova_to_phys(struct iommu_domain *domain,
px = calc_px(iova);
rto = s390_domain->dma_table;
- spin_lock_irqsave(&s390_domain->dma_table_lock, flags);
- if (rto && reg_entry_isvalid(rto[rtx])) {
- sto = get_rt_sto(rto[rtx]);
- if (sto && reg_entry_isvalid(sto[sx])) {
- pto = get_st_pto(sto[sx]);
- if (pto && pt_entry_isvalid(pto[px]))
- phys = pto[px] & ZPCI_PTE_ADDR_MASK;
+ rte = READ_ONCE(rto[rtx]);
+ if (reg_entry_isvalid(rte)) {
+ sto = get_rt_sto(rte);
+ ste = READ_ONCE(sto[sx]);
+ if (reg_entry_isvalid(ste)) {
+ pto = get_st_pto(ste);
+ pte = READ_ONCE(pto[px]);
+ if (pt_entry_isvalid(pte))
+ phys = pte & ZPCI_PTE_ADDR_MASK;
}
}
- spin_unlock_irqrestore(&s390_domain->dma_table_lock, flags);
return phys;
}
-static size_t s390_iommu_unmap(struct iommu_domain *domain,
- unsigned long iova, size_t size,
- struct iommu_iotlb_gather *gather)
+static size_t s390_iommu_unmap_pages(struct iommu_domain *domain,
+ unsigned long iova,
+ size_t pgsize, size_t pgcount,
+ struct iommu_iotlb_gather *gather)
{
struct s390_domain *s390_domain = to_s390_domain(domain);
- int flags = ZPCI_PTE_INVALID;
- phys_addr_t paddr;
+ size_t size = pgcount << __ffs(pgsize);
int rc;
- paddr = s390_iommu_iova_to_phys(domain, iova);
- if (!paddr)
+ if (WARN_ON(iova < s390_domain->domain.geometry.aperture_start ||
+ (iova + size - 1) > s390_domain->domain.geometry.aperture_end))
return 0;
- rc = s390_iommu_update_trans(s390_domain, paddr, iova,
- size, flags);
+ rc = s390_iommu_invalidate_trans(s390_domain, iova, pgcount);
if (rc)
return 0;
+ iommu_iotlb_gather_add_range(gather, iova, size);
+
return size;
}
@@ -380,12 +435,16 @@ static const struct iommu_ops s390_iommu_ops = {
.probe_device = s390_iommu_probe_device,
.release_device = s390_iommu_release_device,
.device_group = generic_device_group,
- .pgsize_bitmap = S390_IOMMU_PGSIZES,
+ .pgsize_bitmap = SZ_4K,
+ .get_resv_regions = s390_iommu_get_resv_regions,
.default_domain_ops = &(const struct iommu_domain_ops) {
.attach_dev = s390_iommu_attach_device,
.detach_dev = s390_iommu_detach_device,
- .map = s390_iommu_map,
- .unmap = s390_iommu_unmap,
+ .map_pages = s390_iommu_map_pages,
+ .unmap_pages = s390_iommu_unmap_pages,
+ .flush_iotlb_all = s390_iommu_flush_iotlb_all,
+ .iotlb_sync = s390_iommu_iotlb_sync,
+ .iotlb_sync_map = s390_iommu_iotlb_sync_map,
.iova_to_phys = s390_iommu_iova_to_phys,
.free = s390_domain_free,
}
diff --git a/drivers/iommu/sprd-iommu.c b/drivers/iommu/sprd-iommu.c
index e027933..219bfa1 100644
--- a/drivers/iommu/sprd-iommu.c
+++ b/drivers/iommu/sprd-iommu.c
@@ -271,10 +271,11 @@ static void sprd_iommu_detach_device(struct iommu_domain *domain,
}
static int sprd_iommu_map(struct iommu_domain *domain, unsigned long iova,
- phys_addr_t paddr, size_t size, int prot, gfp_t gfp)
+ phys_addr_t paddr, size_t pgsize, size_t pgcount,
+ int prot, gfp_t gfp, size_t *mapped)
{
struct sprd_iommu_domain *dom = to_sprd_domain(domain);
- unsigned int page_num = size >> SPRD_IOMMU_PAGE_SHIFT;
+ size_t size = pgcount * SPRD_IOMMU_PAGE_SIZE;
unsigned long flags;
unsigned int i;
u32 *pgt_base_iova;
@@ -296,35 +297,37 @@ static int sprd_iommu_map(struct iommu_domain *domain, unsigned long iova,
pgt_base_iova = dom->pgt_va + ((iova - start) >> SPRD_IOMMU_PAGE_SHIFT);
spin_lock_irqsave(&dom->pgtlock, flags);
- for (i = 0; i < page_num; i++) {
+ for (i = 0; i < pgcount; i++) {
pgt_base_iova[i] = pabase >> SPRD_IOMMU_PAGE_SHIFT;
pabase += SPRD_IOMMU_PAGE_SIZE;
}
spin_unlock_irqrestore(&dom->pgtlock, flags);
+ *mapped = size;
return 0;
}
static size_t sprd_iommu_unmap(struct iommu_domain *domain, unsigned long iova,
- size_t size, struct iommu_iotlb_gather *iotlb_gather)
+ size_t pgsize, size_t pgcount,
+ struct iommu_iotlb_gather *iotlb_gather)
{
struct sprd_iommu_domain *dom = to_sprd_domain(domain);
unsigned long flags;
u32 *pgt_base_iova;
- unsigned int page_num = size >> SPRD_IOMMU_PAGE_SHIFT;
+ size_t size = pgcount * SPRD_IOMMU_PAGE_SIZE;
unsigned long start = domain->geometry.aperture_start;
unsigned long end = domain->geometry.aperture_end;
if (iova < start || (iova + size) > (end + 1))
- return -EINVAL;
+ return 0;
pgt_base_iova = dom->pgt_va + ((iova - start) >> SPRD_IOMMU_PAGE_SHIFT);
spin_lock_irqsave(&dom->pgtlock, flags);
- memset(pgt_base_iova, 0, page_num * sizeof(u32));
+ memset(pgt_base_iova, 0, pgcount * sizeof(u32));
spin_unlock_irqrestore(&dom->pgtlock, flags);
- return 0;
+ return size;
}
static void sprd_iommu_sync_map(struct iommu_domain *domain,
@@ -407,13 +410,13 @@ static const struct iommu_ops sprd_iommu_ops = {
.probe_device = sprd_iommu_probe_device,
.device_group = sprd_iommu_device_group,
.of_xlate = sprd_iommu_of_xlate,
- .pgsize_bitmap = ~0UL << SPRD_IOMMU_PAGE_SHIFT,
+ .pgsize_bitmap = SPRD_IOMMU_PAGE_SIZE,
.owner = THIS_MODULE,
.default_domain_ops = &(const struct iommu_domain_ops) {
.attach_dev = sprd_iommu_attach_device,
.detach_dev = sprd_iommu_detach_device,
- .map = sprd_iommu_map,
- .unmap = sprd_iommu_unmap,
+ .map_pages = sprd_iommu_map,
+ .unmap_pages = sprd_iommu_unmap,
.iotlb_sync_map = sprd_iommu_sync_map,
.iotlb_sync = sprd_iommu_sync,
.iova_to_phys = sprd_iommu_iova_to_phys,
diff --git a/drivers/iommu/sun50i-iommu.c b/drivers/iommu/sun50i-iommu.c
index cd9b74e..5b585ea 100644
--- a/drivers/iommu/sun50i-iommu.c
+++ b/drivers/iommu/sun50i-iommu.c
@@ -27,6 +27,7 @@
#include <linux/types.h>
#define IOMMU_RESET_REG 0x010
+#define IOMMU_RESET_RELEASE_ALL 0xffffffff
#define IOMMU_ENABLE_REG 0x020
#define IOMMU_ENABLE_ENABLE BIT(0)
@@ -92,6 +93,8 @@
#define NUM_PT_ENTRIES 256
#define PT_SIZE (NUM_PT_ENTRIES * PT_ENTRY_SIZE)
+#define SPAGE_SIZE 4096
+
struct sun50i_iommu {
struct iommu_device iommu;
@@ -270,7 +273,7 @@ static u32 sun50i_mk_pte(phys_addr_t page, int prot)
enum sun50i_iommu_aci aci;
u32 flags = 0;
- if (prot & (IOMMU_READ | IOMMU_WRITE))
+ if ((prot & (IOMMU_READ | IOMMU_WRITE)) == (IOMMU_READ | IOMMU_WRITE))
aci = SUN50I_IOMMU_ACI_RD_WR;
else if (prot & IOMMU_READ)
aci = SUN50I_IOMMU_ACI_RD;
@@ -294,6 +297,62 @@ static void sun50i_table_flush(struct sun50i_iommu_domain *sun50i_domain,
dma_sync_single_for_device(iommu->dev, dma, size, DMA_TO_DEVICE);
}
+static void sun50i_iommu_zap_iova(struct sun50i_iommu *iommu,
+ unsigned long iova)
+{
+ u32 reg;
+ int ret;
+
+ iommu_write(iommu, IOMMU_TLB_IVLD_ADDR_REG, iova);
+ iommu_write(iommu, IOMMU_TLB_IVLD_ADDR_MASK_REG, GENMASK(31, 12));
+ iommu_write(iommu, IOMMU_TLB_IVLD_ENABLE_REG,
+ IOMMU_TLB_IVLD_ENABLE_ENABLE);
+
+ ret = readl_poll_timeout_atomic(iommu->base + IOMMU_TLB_IVLD_ENABLE_REG,
+ reg, !reg, 1, 2000);
+ if (ret)
+ dev_warn(iommu->dev, "TLB invalidation timed out!\n");
+}
+
+static void sun50i_iommu_zap_ptw_cache(struct sun50i_iommu *iommu,
+ unsigned long iova)
+{
+ u32 reg;
+ int ret;
+
+ iommu_write(iommu, IOMMU_PC_IVLD_ADDR_REG, iova);
+ iommu_write(iommu, IOMMU_PC_IVLD_ENABLE_REG,
+ IOMMU_PC_IVLD_ENABLE_ENABLE);
+
+ ret = readl_poll_timeout_atomic(iommu->base + IOMMU_PC_IVLD_ENABLE_REG,
+ reg, !reg, 1, 2000);
+ if (ret)
+ dev_warn(iommu->dev, "PTW cache invalidation timed out!\n");
+}
+
+static void sun50i_iommu_zap_range(struct sun50i_iommu *iommu,
+ unsigned long iova, size_t size)
+{
+ assert_spin_locked(&iommu->iommu_lock);
+
+ iommu_write(iommu, IOMMU_AUTO_GATING_REG, 0);
+
+ sun50i_iommu_zap_iova(iommu, iova);
+ sun50i_iommu_zap_iova(iommu, iova + SPAGE_SIZE);
+ if (size > SPAGE_SIZE) {
+ sun50i_iommu_zap_iova(iommu, iova + size);
+ sun50i_iommu_zap_iova(iommu, iova + size + SPAGE_SIZE);
+ }
+ sun50i_iommu_zap_ptw_cache(iommu, iova);
+ sun50i_iommu_zap_ptw_cache(iommu, iova + SZ_1M);
+ if (size > SZ_1M) {
+ sun50i_iommu_zap_ptw_cache(iommu, iova + size);
+ sun50i_iommu_zap_ptw_cache(iommu, iova + size + SZ_1M);
+ }
+
+ iommu_write(iommu, IOMMU_AUTO_GATING_REG, IOMMU_AUTO_GATING_ENABLE);
+}
+
static int sun50i_iommu_flush_all_tlb(struct sun50i_iommu *iommu)
{
u32 reg;
@@ -343,6 +402,18 @@ static void sun50i_iommu_flush_iotlb_all(struct iommu_domain *domain)
spin_unlock_irqrestore(&iommu->iommu_lock, flags);
}
+static void sun50i_iommu_iotlb_sync_map(struct iommu_domain *domain,
+ unsigned long iova, size_t size)
+{
+ struct sun50i_iommu_domain *sun50i_domain = to_sun50i_domain(domain);
+ struct sun50i_iommu *iommu = sun50i_domain->iommu;
+ unsigned long flags;
+
+ spin_lock_irqsave(&iommu->iommu_lock, flags);
+ sun50i_iommu_zap_range(iommu, iova, size);
+ spin_unlock_irqrestore(&iommu->iommu_lock, flags);
+}
+
static void sun50i_iommu_iotlb_sync(struct iommu_domain *domain,
struct iommu_iotlb_gather *gather)
{
@@ -511,7 +582,7 @@ static u32 *sun50i_dte_get_page_table(struct sun50i_iommu_domain *sun50i_domain,
sun50i_iommu_free_page_table(iommu, drop_pt);
}
- sun50i_table_flush(sun50i_domain, page_table, PT_SIZE);
+ sun50i_table_flush(sun50i_domain, page_table, NUM_PT_ENTRIES);
sun50i_table_flush(sun50i_domain, dte_addr, 1);
return page_table;
@@ -601,7 +672,6 @@ static struct iommu_domain *sun50i_iommu_domain_alloc(unsigned type)
struct sun50i_iommu_domain *sun50i_domain;
if (type != IOMMU_DOMAIN_DMA &&
- type != IOMMU_DOMAIN_IDENTITY &&
type != IOMMU_DOMAIN_UNMANAGED)
return NULL;
@@ -766,6 +836,7 @@ static const struct iommu_ops sun50i_iommu_ops = {
.attach_dev = sun50i_iommu_attach_device,
.detach_dev = sun50i_iommu_detach_device,
.flush_iotlb_all = sun50i_iommu_flush_iotlb_all,
+ .iotlb_sync_map = sun50i_iommu_iotlb_sync_map,
.iotlb_sync = sun50i_iommu_iotlb_sync,
.iova_to_phys = sun50i_iommu_iova_to_phys,
.map = sun50i_iommu_map,
@@ -785,6 +856,8 @@ static void sun50i_iommu_report_fault(struct sun50i_iommu *iommu,
report_iommu_fault(iommu->domain, iommu->dev, iova, prot);
else
dev_err(iommu->dev, "Page fault while iommu not attached to any domain?\n");
+
+ sun50i_iommu_zap_range(iommu, iova, SPAGE_SIZE);
}
static phys_addr_t sun50i_iommu_handle_pt_irq(struct sun50i_iommu *iommu,
@@ -868,8 +941,8 @@ static phys_addr_t sun50i_iommu_handle_perm_irq(struct sun50i_iommu *iommu)
static irqreturn_t sun50i_iommu_irq(int irq, void *dev_id)
{
+ u32 status, l1_status, l2_status, resets;
struct sun50i_iommu *iommu = dev_id;
- u32 status;
spin_lock(&iommu->iommu_lock);
@@ -879,6 +952,9 @@ static irqreturn_t sun50i_iommu_irq(int irq, void *dev_id)
return IRQ_NONE;
}
+ l1_status = iommu_read(iommu, IOMMU_L1PG_INT_REG);
+ l2_status = iommu_read(iommu, IOMMU_L2PG_INT_REG);
+
if (status & IOMMU_INT_INVALID_L2PG)
sun50i_iommu_handle_pt_irq(iommu,
IOMMU_INT_ERR_ADDR_L2_REG,
@@ -892,8 +968,9 @@ static irqreturn_t sun50i_iommu_irq(int irq, void *dev_id)
iommu_write(iommu, IOMMU_INT_CLR_REG, status);
- iommu_write(iommu, IOMMU_RESET_REG, ~status);
- iommu_write(iommu, IOMMU_RESET_REG, status);
+ resets = (status | l1_status | l2_status) & IOMMU_INT_MASTER_MASK;
+ iommu_write(iommu, IOMMU_RESET_REG, ~resets);
+ iommu_write(iommu, IOMMU_RESET_REG, IOMMU_RESET_RELEASE_ALL);
spin_unlock(&iommu->iommu_lock);
diff --git a/drivers/leds/leds-blinkm.c b/drivers/leds/leds-blinkm.c
index 3fb6a2f..e19cc8a 100644
--- a/drivers/leds/leds-blinkm.c
+++ b/drivers/leds/leds-blinkm.c
@@ -139,11 +139,11 @@ static ssize_t show_color_common(struct device *dev, char *buf, int color)
return ret;
switch (color) {
case RED:
- return scnprintf(buf, PAGE_SIZE, "%02X\n", data->red);
+ return sysfs_emit(buf, "%02X\n", data->red);
case GREEN:
- return scnprintf(buf, PAGE_SIZE, "%02X\n", data->green);
+ return sysfs_emit(buf, "%02X\n", data->green);
case BLUE:
- return scnprintf(buf, PAGE_SIZE, "%02X\n", data->blue);
+ return sysfs_emit(buf, "%02X\n", data->blue);
default:
return -EINVAL;
}
@@ -253,7 +253,7 @@ static DEVICE_ATTR_RW(blue);
static ssize_t test_show(struct device *dev, struct device_attribute *attr,
char *buf)
{
- return scnprintf(buf, PAGE_SIZE,
+ return sysfs_emit(buf,
"#Write into test to start test sequence!#\n");
}
diff --git a/drivers/leds/leds-is31fl319x.c b/drivers/leds/leds-is31fl319x.c
index 52b59b6..b2f4c4e 100644
--- a/drivers/leds/leds-is31fl319x.c
+++ b/drivers/leds/leds-is31fl319x.c
@@ -38,6 +38,7 @@
#define IS31FL3190_CURRENT_uA_MIN 5000
#define IS31FL3190_CURRENT_uA_DEFAULT 42000
#define IS31FL3190_CURRENT_uA_MAX 42000
+#define IS31FL3190_CURRENT_SHIFT 2
#define IS31FL3190_CURRENT_MASK GENMASK(4, 2)
#define IS31FL3190_CURRENT_5_mA 0x02
#define IS31FL3190_CURRENT_10_mA 0x01
@@ -553,7 +554,7 @@ static int is31fl319x_probe(struct i2c_client *client)
is31fl3196_db_to_gain(is31->audio_gain_db));
else
regmap_update_bits(is31->regmap, IS31FL3190_CURRENT, IS31FL3190_CURRENT_MASK,
- is31fl3190_microamp_to_cs(dev, aggregated_led_microamp));
+ is31fl3190_microamp_to_cs(dev, aggregated_led_microamp) << IS31FL3190_CURRENT_SHIFT);
for (i = 0; i < is31->cdef->num_leds; i++) {
struct is31fl319x_led *led = &is31->leds[i];
diff --git a/drivers/leds/leds-lm3533.c b/drivers/leds/leds-lm3533.c
index 43d5970..bcd414e 100644
--- a/drivers/leds/leds-lm3533.c
+++ b/drivers/leds/leds-lm3533.c
@@ -314,7 +314,7 @@ static ssize_t show_id(struct device *dev,
struct led_classdev *led_cdev = dev_get_drvdata(dev);
struct lm3533_led *led = to_lm3533_led(led_cdev);
- return scnprintf(buf, PAGE_SIZE, "%d\n", led->id);
+ return sysfs_emit(buf, "%d\n", led->id);
}
/*
@@ -344,7 +344,7 @@ static ssize_t show_risefalltime(struct device *dev,
if (ret)
return ret;
- return scnprintf(buf, PAGE_SIZE, "%x\n", val);
+ return sysfs_emit(buf, "%x\n", val);
}
static ssize_t show_risetime(struct device *dev,
@@ -415,7 +415,7 @@ static ssize_t show_als_channel(struct device *dev,
channel = (val & LM3533_REG_CTRLBANK_BCONF_ALS_CHANNEL_MASK) + 1;
- return scnprintf(buf, PAGE_SIZE, "%u\n", channel);
+ return sysfs_emit(buf, "%u\n", channel);
}
static ssize_t store_als_channel(struct device *dev,
@@ -465,7 +465,7 @@ static ssize_t show_als_en(struct device *dev,
enable = val & LM3533_REG_CTRLBANK_BCONF_ALS_EN_MASK;
- return scnprintf(buf, PAGE_SIZE, "%d\n", enable);
+ return sysfs_emit(buf, "%d\n", enable);
}
static ssize_t store_als_en(struct device *dev,
@@ -518,7 +518,7 @@ static ssize_t show_linear(struct device *dev,
else
linear = 0;
- return scnprintf(buf, PAGE_SIZE, "%x\n", linear);
+ return sysfs_emit(buf, "%x\n", linear);
}
static ssize_t store_linear(struct device *dev,
@@ -564,7 +564,7 @@ static ssize_t show_pwm(struct device *dev,
if (ret)
return ret;
- return scnprintf(buf, PAGE_SIZE, "%u\n", val);
+ return sysfs_emit(buf, "%u\n", val);
}
static ssize_t store_pwm(struct device *dev,
diff --git a/drivers/leds/leds-lp5521.c b/drivers/leds/leds-lp5521.c
index 7ff20c2..19478d9 100644
--- a/drivers/leds/leds-lp5521.c
+++ b/drivers/leds/leds-lp5521.c
@@ -469,7 +469,7 @@ static ssize_t lp5521_selftest(struct device *dev,
ret = lp5521_run_selftest(chip, buf);
mutex_unlock(&chip->lock);
- return scnprintf(buf, PAGE_SIZE, "%s\n", ret ? "FAIL" : "OK");
+ return sysfs_emit(buf, "%s\n", ret ? "FAIL" : "OK");
}
/* device attributes */
diff --git a/drivers/leds/leds-lp5523.c b/drivers/leds/leds-lp5523.c
index 369d40b..e08e3de 100644
--- a/drivers/leds/leds-lp5523.c
+++ b/drivers/leds/leds-lp5523.c
@@ -581,8 +581,8 @@ static ssize_t lp5523_selftest(struct device *dev,
struct lp55xx_led *led = i2c_get_clientdata(to_i2c_client(dev));
struct lp55xx_chip *chip = led->chip;
struct lp55xx_platform_data *pdata = chip->pdata;
- int i, ret, pos = 0;
- u8 status, adc, vdd;
+ int ret, pos = 0;
+ u8 status, adc, vdd, i;
mutex_lock(&chip->lock);
@@ -612,20 +612,21 @@ static ssize_t lp5523_selftest(struct device *dev,
vdd--; /* There may be some fluctuation in measurement */
- for (i = 0; i < LP5523_MAX_LEDS; i++) {
- /* Skip non-existing channels */
+ for (i = 0; i < pdata->num_channels; i++) {
+ /* Skip disabled channels */
if (pdata->led_config[i].led_current == 0)
continue;
/* Set default current */
- lp55xx_write(chip, LP5523_REG_LED_CURRENT_BASE + i,
+ lp55xx_write(chip, LP5523_REG_LED_CURRENT_BASE + led->chan_nr,
pdata->led_config[i].led_current);
- lp55xx_write(chip, LP5523_REG_LED_PWM_BASE + i, 0xff);
+ lp55xx_write(chip, LP5523_REG_LED_PWM_BASE + led->chan_nr,
+ 0xff);
/* let current stabilize 2 - 4ms before measurements start */
usleep_range(2000, 4000);
lp55xx_write(chip, LP5523_REG_LED_TEST_CTRL,
- LP5523_EN_LEDTEST | i);
+ LP5523_EN_LEDTEST | led->chan_nr);
/* ADC conversion time is 2.7 ms typically */
usleep_range(3000, 6000);
ret = lp55xx_read(chip, LP5523_REG_STATUS, &status);
@@ -633,20 +634,22 @@ static ssize_t lp5523_selftest(struct device *dev,
goto fail;
if (!(status & LP5523_LEDTEST_DONE))
- usleep_range(3000, 6000);/* Was not ready. Wait. */
+ usleep_range(3000, 6000); /* Was not ready. Wait. */
ret = lp55xx_read(chip, LP5523_REG_LED_TEST_ADC, &adc);
if (ret < 0)
goto fail;
if (adc >= vdd || adc < LP5523_ADC_SHORTCIRC_LIM)
- pos += sprintf(buf + pos, "LED %d FAIL\n", i);
+ pos += sprintf(buf + pos, "LED %d FAIL\n",
+ led->chan_nr);
- lp55xx_write(chip, LP5523_REG_LED_PWM_BASE + i, 0x00);
+ lp55xx_write(chip, LP5523_REG_LED_PWM_BASE + led->chan_nr,
+ 0x00);
/* Restore current */
- lp55xx_write(chip, LP5523_REG_LED_CURRENT_BASE + i,
- led->led_current);
+ lp55xx_write(chip, LP5523_REG_LED_CURRENT_BASE + led->chan_nr,
+ led->led_current);
led++;
}
if (pos == 0)
diff --git a/drivers/leds/leds-lp55xx-common.c b/drivers/leds/leds-lp55xx-common.c
index 9fdfc1b..c194096 100644
--- a/drivers/leds/leds-lp55xx-common.c
+++ b/drivers/leds/leds-lp55xx-common.c
@@ -88,7 +88,7 @@ static ssize_t led_current_show(struct device *dev,
{
struct lp55xx_led *led = dev_to_lp55xx_led(dev);
- return scnprintf(buf, PAGE_SIZE, "%d\n", led->led_current);
+ return sysfs_emit(buf, "%d\n", led->led_current);
}
static ssize_t led_current_store(struct device *dev,
@@ -121,7 +121,7 @@ static ssize_t max_current_show(struct device *dev,
{
struct lp55xx_led *led = dev_to_lp55xx_led(dev);
- return scnprintf(buf, PAGE_SIZE, "%d\n", led->max_current);
+ return sysfs_emit(buf, "%d\n", led->max_current);
}
static DEVICE_ATTR_RW(led_current);
@@ -166,7 +166,7 @@ static int lp55xx_init_led(struct lp55xx_led *led,
struct mc_subled *mc_led_info;
struct led_classdev *led_cdev;
char name[32];
- int i, j = 0;
+ int i;
int ret;
if (chan >= max_channel) {
@@ -201,7 +201,6 @@ static int lp55xx_init_led(struct lp55xx_led *led,
pdata->led_config[chan].color_id[i];
mc_led_info[i].channel =
pdata->led_config[chan].output_num[i];
- j++;
}
led->mc_cdev.subled_info = mc_led_info;
diff --git a/drivers/leds/leds-max8997.c b/drivers/leds/leds-max8997.c
index c0bddb3..c8d7f55 100644
--- a/drivers/leds/leds-max8997.c
+++ b/drivers/leds/leds-max8997.c
@@ -238,11 +238,6 @@ static int max8997_led_probe(struct platform_device *pdev)
char name[20];
int ret = 0;
- if (pdata == NULL) {
- dev_err(&pdev->dev, "no platform data\n");
- return -ENODEV;
- }
-
led = devm_kzalloc(&pdev->dev, sizeof(*led), GFP_KERNEL);
if (led == NULL)
return -ENOMEM;
@@ -258,7 +253,7 @@ static int max8997_led_probe(struct platform_device *pdev)
led->iodev = iodev;
/* initialize mode and brightness according to platform_data */
- if (pdata->led_pdata) {
+ if (pdata && pdata->led_pdata) {
u8 mode = 0, brightness = 0;
mode = pdata->led_pdata->mode[led->id];
diff --git a/drivers/leds/leds-pca955x.c b/drivers/leds/leds-pca955x.c
index 81aaf21..33ec454 100644
--- a/drivers/leds/leds-pca955x.c
+++ b/drivers/leds/leds-pca955x.c
@@ -145,12 +145,6 @@ static inline int pca95xx_num_input_regs(int bits)
return (bits + 7) / 8;
}
-/* 4 bits per LED selector register */
-static inline int pca95xx_num_led_regs(int bits)
-{
- return (bits + 3) / 4;
-}
-
/*
* Return an LED selector register value based on an existing one, with
* the appropriate 2-bit state value set for the given LED number (0-3).
diff --git a/drivers/leds/rgb/leds-qcom-lpg.c b/drivers/leds/rgb/leds-qcom-lpg.c
index 02f51cc..c1a5625 100644
--- a/drivers/leds/rgb/leds-qcom-lpg.c
+++ b/drivers/leds/rgb/leds-qcom-lpg.c
@@ -602,8 +602,8 @@ static void lpg_brightness_set(struct lpg_led *led, struct led_classdev *cdev,
lpg_lut_sync(lpg, lut_mask);
}
-static void lpg_brightness_single_set(struct led_classdev *cdev,
- enum led_brightness value)
+static int lpg_brightness_single_set(struct led_classdev *cdev,
+ enum led_brightness value)
{
struct lpg_led *led = container_of(cdev, struct lpg_led, cdev);
struct mc_subled info;
@@ -614,10 +614,12 @@ static void lpg_brightness_single_set(struct led_classdev *cdev,
lpg_brightness_set(led, cdev, &info);
mutex_unlock(&led->lpg->lock);
+
+ return 0;
}
-static void lpg_brightness_mc_set(struct led_classdev *cdev,
- enum led_brightness value)
+static int lpg_brightness_mc_set(struct led_classdev *cdev,
+ enum led_brightness value)
{
struct led_classdev_mc *mc = lcdev_to_mccdev(cdev);
struct lpg_led *led = container_of(mc, struct lpg_led, mcdev);
@@ -628,6 +630,8 @@ static void lpg_brightness_mc_set(struct led_classdev *cdev,
lpg_brightness_set(led, cdev, mc->subled_info);
mutex_unlock(&led->lpg->lock);
+
+ return 0;
}
static int lpg_blink_set(struct lpg_led *led,
@@ -1118,7 +1122,7 @@ static int lpg_add_led(struct lpg *lpg, struct device_node *np)
led->mcdev.num_colors = num_channels;
cdev = &led->mcdev.led_cdev;
- cdev->brightness_set = lpg_brightness_mc_set;
+ cdev->brightness_set_blocking = lpg_brightness_mc_set;
cdev->blink_set = lpg_blink_mc_set;
/* Register pattern accessors only if we have a LUT block */
@@ -1132,7 +1136,7 @@ static int lpg_add_led(struct lpg *lpg, struct device_node *np)
return ret;
cdev = &led->cdev;
- cdev->brightness_set = lpg_brightness_single_set;
+ cdev->brightness_set_blocking = lpg_brightness_single_set;
cdev->blink_set = lpg_blink_single_set;
/* Register pattern accessors only if we have a LUT block */
@@ -1151,7 +1155,7 @@ static int lpg_add_led(struct lpg *lpg, struct device_node *np)
else
cdev->brightness = LED_OFF;
- cdev->brightness_set(cdev, cdev->brightness);
+ cdev->brightness_set_blocking(cdev, cdev->brightness);
init_data.fwnode = of_fwnode_handle(np);
diff --git a/drivers/leds/trigger/ledtrig-pattern.c b/drivers/leds/trigger/ledtrig-pattern.c
index 43a265d..885ca63 100644
--- a/drivers/leds/trigger/ledtrig-pattern.c
+++ b/drivers/leds/trigger/ledtrig-pattern.c
@@ -155,7 +155,7 @@ static ssize_t repeat_show(struct device *dev, struct device_attribute *attr,
mutex_unlock(&data->lock);
- return scnprintf(buf, PAGE_SIZE, "%d\n", repeat);
+ return sysfs_emit(buf, "%d\n", repeat);
}
static ssize_t repeat_store(struct device *dev, struct device_attribute *attr,
diff --git a/drivers/macintosh/adb.c b/drivers/macintosh/adb.c
index 1bbb9ca..23bd0c7 100644
--- a/drivers/macintosh/adb.c
+++ b/drivers/macintosh/adb.c
@@ -478,7 +478,7 @@ adb_register(int default_id, int handler_id, struct adb_ids *ids,
if ((adb_handler[i].original_address == default_id) &&
(!handler_id || (handler_id == adb_handler[i].handler_id) ||
try_handler_change(i, handler_id))) {
- if (adb_handler[i].handler != 0) {
+ if (adb_handler[i].handler) {
pr_err("Two handlers for ADB device %d\n",
default_id);
continue;
@@ -673,7 +673,7 @@ static int adb_open(struct inode *inode, struct file *file)
goto out;
}
state = kmalloc(sizeof(struct adbdev_state), GFP_KERNEL);
- if (state == 0) {
+ if (!state) {
ret = -ENOMEM;
goto out;
}
diff --git a/drivers/macintosh/ams/ams-i2c.c b/drivers/macintosh/ams/ams-i2c.c
index 3ded340..a4a1035 100644
--- a/drivers/macintosh/ams/ams-i2c.c
+++ b/drivers/macintosh/ams/ams-i2c.c
@@ -56,8 +56,7 @@ enum ams_i2c_cmd {
AMS_CMD_START,
};
-static int ams_i2c_probe(struct i2c_client *client,
- const struct i2c_device_id *id);
+static int ams_i2c_probe(struct i2c_client *client);
static void ams_i2c_remove(struct i2c_client *client);
static const struct i2c_device_id ams_id[] = {
@@ -70,7 +69,7 @@ static struct i2c_driver ams_i2c_driver = {
.driver = {
.name = "ams",
},
- .probe = ams_i2c_probe,
+ .probe_new = ams_i2c_probe,
.remove = ams_i2c_remove,
.id_table = ams_id,
};
@@ -155,8 +154,7 @@ static void ams_i2c_get_xyz(s8 *x, s8 *y, s8 *z)
*z = ams_i2c_read(AMS_DATAZ);
}
-static int ams_i2c_probe(struct i2c_client *client,
- const struct i2c_device_id *id)
+static int ams_i2c_probe(struct i2c_client *client)
{
int vmaj, vmin;
int result;
diff --git a/drivers/macintosh/ams/ams.h b/drivers/macintosh/ams/ams.h
index 935bdd9..2c159c8 100644
--- a/drivers/macintosh/ams/ams.h
+++ b/drivers/macintosh/ams/ams.h
@@ -1,4 +1,7 @@
/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _AMS_H
+#define _AMS_H
+
#include <linux/i2c.h>
#include <linux/input.h>
#include <linux/kthread.h>
@@ -69,3 +72,5 @@ extern int ams_i2c_init(struct device_node *np);
extern int ams_input_init(void);
extern void ams_input_exit(void);
+
+#endif /* _AMS_H */
diff --git a/drivers/macintosh/macio-adb.c b/drivers/macintosh/macio-adb.c
index 9b63bd2..55a9f8c 100644
--- a/drivers/macintosh/macio-adb.c
+++ b/drivers/macintosh/macio-adb.c
@@ -100,7 +100,7 @@ int macio_init(void)
unsigned int irq;
adbs = of_find_compatible_node(NULL, "adb", "chrp,adb0");
- if (adbs == 0)
+ if (!adbs)
return -ENXIO;
if (of_address_to_resource(adbs, 0, &r)) {
@@ -108,6 +108,10 @@ int macio_init(void)
return -ENXIO;
}
adb = ioremap(r.start, sizeof(struct adb_regs));
+ if (!adb) {
+ of_node_put(adbs);
+ return -ENOMEM;
+ }
out_8(&adb->ctrl.r, 0);
out_8(&adb->intr.r, 0);
@@ -183,7 +187,7 @@ static int macio_send_request(struct adb_request *req, int sync)
req->reply_len = 0;
spin_lock_irqsave(&macio_lock, flags);
- if (current_req != 0) {
+ if (current_req) {
last_req->next = req;
last_req = req;
} else {
@@ -213,7 +217,8 @@ static irqreturn_t macio_adb_interrupt(int irq, void *arg)
spin_lock(&macio_lock);
if (in_8(&adb->intr.r) & TAG) {
handled = 1;
- if ((req = current_req) != 0) {
+ req = current_req;
+ if (req) {
/* put the current request in */
for (i = 0; i < req->nbytes; ++i)
out_8(&adb->data[i].r, req->data[i]);
diff --git a/drivers/macintosh/macio_asic.c b/drivers/macintosh/macio_asic.c
index 1ec1e59..3bc1f37 100644
--- a/drivers/macintosh/macio_asic.c
+++ b/drivers/macintosh/macio_asic.c
@@ -424,7 +424,7 @@ static struct macio_dev * macio_add_one_device(struct macio_chip *chip,
if (of_device_register(&dev->ofdev) != 0) {
printk(KERN_DEBUG"macio: device registration error for %s!\n",
dev_name(&dev->ofdev.dev));
- kfree(dev);
+ put_device(&dev->ofdev.dev);
return NULL;
}
diff --git a/drivers/macintosh/therm_adt746x.c b/drivers/macintosh/therm_adt746x.c
index b004ea2..8f5db90 100644
--- a/drivers/macintosh/therm_adt746x.c
+++ b/drivers/macintosh/therm_adt746x.c
@@ -464,9 +464,9 @@ static void thermostat_remove_files(struct thermostat *th)
}
-static int probe_thermostat(struct i2c_client *client,
- const struct i2c_device_id *id)
+static int probe_thermostat(struct i2c_client *client)
{
+ const struct i2c_device_id *id = i2c_client_get_device_id(client);
struct device_node *np = client->dev.of_node;
struct thermostat* th;
const __be32 *prop;
@@ -598,7 +598,7 @@ static struct i2c_driver thermostat_driver = {
.driver = {
.name = "therm_adt746x",
},
- .probe = probe_thermostat,
+ .probe_new = probe_thermostat,
.remove = remove_thermostat,
.id_table = therm_adt746x_id,
};
diff --git a/drivers/macintosh/therm_windtunnel.c b/drivers/macintosh/therm_windtunnel.c
index b8228ca..22b15ef 100644
--- a/drivers/macintosh/therm_windtunnel.c
+++ b/drivers/macintosh/therm_windtunnel.c
@@ -411,8 +411,9 @@ static const struct i2c_device_id therm_windtunnel_id[] = {
MODULE_DEVICE_TABLE(i2c, therm_windtunnel_id);
static int
-do_probe(struct i2c_client *cl, const struct i2c_device_id *id)
+do_probe(struct i2c_client *cl)
{
+ const struct i2c_device_id *id = i2c_client_get_device_id(cl);
struct i2c_adapter *adapter = cl->adapter;
int ret = 0;
@@ -441,7 +442,7 @@ static struct i2c_driver g4fan_driver = {
.driver = {
.name = "therm_windtunnel",
},
- .probe = do_probe,
+ .probe_new = do_probe,
.remove = do_remove,
.id_table = therm_windtunnel_id,
};
diff --git a/drivers/macintosh/via-pmu-backlight.c b/drivers/macintosh/via-pmu-backlight.c
index 2194016..c2d87e7 100644
--- a/drivers/macintosh/via-pmu-backlight.c
+++ b/drivers/macintosh/via-pmu-backlight.c
@@ -71,12 +71,7 @@ static int pmu_backlight_get_level_brightness(int level)
static int __pmu_backlight_update_status(struct backlight_device *bd)
{
struct adb_request req;
- int level = bd->props.brightness;
-
-
- if (bd->props.power != FB_BLANK_UNBLANK ||
- bd->props.fb_blank != FB_BLANK_UNBLANK)
- level = 0;
+ int level = backlight_get_brightness(bd);
if (level > 0) {
int pmulevel = pmu_backlight_get_level_brightness(level);
diff --git a/drivers/macintosh/via-pmu.c b/drivers/macintosh/via-pmu.c
index 4965796..e0cb8da 100644
--- a/drivers/macintosh/via-pmu.c
+++ b/drivers/macintosh/via-pmu.c
@@ -203,9 +203,11 @@ static int init_pmu(void);
static void pmu_start(void);
static irqreturn_t via_pmu_interrupt(int irq, void *arg);
static irqreturn_t gpio1_interrupt(int irq, void *arg);
+#ifdef CONFIG_PROC_FS
static int pmu_info_proc_show(struct seq_file *m, void *v);
static int pmu_irqstats_proc_show(struct seq_file *m, void *v);
static int pmu_battery_proc_show(struct seq_file *m, void *v);
+#endif
static void pmu_pass_intr(unsigned char *data, int len);
static const struct proc_ops pmu_options_proc_ops;
@@ -852,6 +854,7 @@ query_battery_state(void)
2, PMU_SMART_BATTERY_STATE, pmu_cur_battery+1);
}
+#ifdef CONFIG_PROC_FS
static int pmu_info_proc_show(struct seq_file *m, void *v)
{
seq_printf(m, "PMU driver version : %d\n", PMU_DRIVER_VERSION);
@@ -972,6 +975,7 @@ static const struct proc_ops pmu_options_proc_ops = {
.proc_release = single_release,
.proc_write = pmu_options_proc_write,
};
+#endif
#ifdef CONFIG_ADB
/* Send an ADB command */
diff --git a/drivers/macintosh/windfarm_ad7417_sensor.c b/drivers/macintosh/windfarm_ad7417_sensor.c
index c5c54a4..33b4723 100644
--- a/drivers/macintosh/windfarm_ad7417_sensor.c
+++ b/drivers/macintosh/windfarm_ad7417_sensor.c
@@ -229,8 +229,7 @@ static void wf_ad7417_init_chip(struct wf_ad7417_priv *pv)
pv->config = config;
}
-static int wf_ad7417_probe(struct i2c_client *client,
- const struct i2c_device_id *id)
+static int wf_ad7417_probe(struct i2c_client *client)
{
struct wf_ad7417_priv *pv;
const struct mpu_data *mpu;
@@ -321,7 +320,7 @@ static struct i2c_driver wf_ad7417_driver = {
.name = "wf_ad7417",
.of_match_table = wf_ad7417_of_id,
},
- .probe = wf_ad7417_probe,
+ .probe_new = wf_ad7417_probe,
.remove = wf_ad7417_remove,
.id_table = wf_ad7417_id,
};
diff --git a/drivers/macintosh/windfarm_fcu_controls.c b/drivers/macintosh/windfarm_fcu_controls.c
index c5b1ca5..e027d88 100644
--- a/drivers/macintosh/windfarm_fcu_controls.c
+++ b/drivers/macintosh/windfarm_fcu_controls.c
@@ -514,8 +514,7 @@ static int wf_fcu_init_chip(struct wf_fcu_priv *pv)
return 0;
}
-static int wf_fcu_probe(struct i2c_client *client,
- const struct i2c_device_id *id)
+static int wf_fcu_probe(struct i2c_client *client)
{
struct wf_fcu_priv *pv;
@@ -590,7 +589,7 @@ static struct i2c_driver wf_fcu_driver = {
.name = "wf_fcu",
.of_match_table = wf_fcu_of_id,
},
- .probe = wf_fcu_probe,
+ .probe_new = wf_fcu_probe,
.remove = wf_fcu_remove,
.id_table = wf_fcu_id,
};
diff --git a/drivers/macintosh/windfarm_lm75_sensor.c b/drivers/macintosh/windfarm_lm75_sensor.c
index 204661c..24f0a444 100644
--- a/drivers/macintosh/windfarm_lm75_sensor.c
+++ b/drivers/macintosh/windfarm_lm75_sensor.c
@@ -87,9 +87,9 @@ static const struct wf_sensor_ops wf_lm75_ops = {
.owner = THIS_MODULE,
};
-static int wf_lm75_probe(struct i2c_client *client,
- const struct i2c_device_id *id)
-{
+static int wf_lm75_probe(struct i2c_client *client)
+{
+ const struct i2c_device_id *id = i2c_client_get_device_id(client);
struct wf_lm75_sensor *lm;
int rc, ds1775;
const char *name, *loc;
@@ -177,7 +177,7 @@ static struct i2c_driver wf_lm75_driver = {
.name = "wf_lm75",
.of_match_table = wf_lm75_of_id,
},
- .probe = wf_lm75_probe,
+ .probe_new = wf_lm75_probe,
.remove = wf_lm75_remove,
.id_table = wf_lm75_id,
};
diff --git a/drivers/macintosh/windfarm_lm87_sensor.c b/drivers/macintosh/windfarm_lm87_sensor.c
index 40d2546..f37a32c 100644
--- a/drivers/macintosh/windfarm_lm87_sensor.c
+++ b/drivers/macintosh/windfarm_lm87_sensor.c
@@ -95,8 +95,7 @@ static const struct wf_sensor_ops wf_lm87_ops = {
.owner = THIS_MODULE,
};
-static int wf_lm87_probe(struct i2c_client *client,
- const struct i2c_device_id *id)
+static int wf_lm87_probe(struct i2c_client *client)
{
struct wf_lm87_sensor *lm;
const char *name = NULL, *loc;
@@ -173,7 +172,7 @@ static struct i2c_driver wf_lm87_driver = {
.name = "wf_lm87",
.of_match_table = wf_lm87_of_id,
},
- .probe = wf_lm87_probe,
+ .probe_new = wf_lm87_probe,
.remove = wf_lm87_remove,
.id_table = wf_lm87_id,
};
diff --git a/drivers/macintosh/windfarm_max6690_sensor.c b/drivers/macintosh/windfarm_max6690_sensor.c
index c0d404e..6c5ab657 100644
--- a/drivers/macintosh/windfarm_max6690_sensor.c
+++ b/drivers/macintosh/windfarm_max6690_sensor.c
@@ -60,8 +60,7 @@ static const struct wf_sensor_ops wf_max6690_ops = {
.owner = THIS_MODULE,
};
-static int wf_max6690_probe(struct i2c_client *client,
- const struct i2c_device_id *id)
+static int wf_max6690_probe(struct i2c_client *client)
{
const char *name, *loc;
struct wf_6690_sensor *max;
@@ -129,7 +128,7 @@ static struct i2c_driver wf_max6690_driver = {
.name = "wf_max6690",
.of_match_table = wf_max6690_of_id,
},
- .probe = wf_max6690_probe,
+ .probe_new = wf_max6690_probe,
.remove = wf_max6690_remove,
.id_table = wf_max6690_id,
};
diff --git a/drivers/macintosh/windfarm_pid.h b/drivers/macintosh/windfarm_pid.h
index 83f747d..335613a2 100644
--- a/drivers/macintosh/windfarm_pid.h
+++ b/drivers/macintosh/windfarm_pid.h
@@ -1,4 +1,7 @@
/* SPDX-License-Identifier: GPL-2.0-only */
+#ifndef _WINDFARM_PID_H
+#define _WINDFARM_PID_H
+
/*
* Windfarm PowerMac thermal control. Generic PID helpers
*
@@ -82,3 +85,5 @@ struct wf_cpu_pid_state {
extern void wf_cpu_pid_init(struct wf_cpu_pid_state *st,
struct wf_cpu_pid_param *param);
extern s32 wf_cpu_pid_run(struct wf_cpu_pid_state *st, s32 power, s32 temp);
+
+#endif /* _WINDFARM_PID_H */
diff --git a/drivers/macintosh/windfarm_pm121.c b/drivers/macintosh/windfarm_pm121.c
index 36312f1..8250041 100644
--- a/drivers/macintosh/windfarm_pm121.c
+++ b/drivers/macintosh/windfarm_pm121.c
@@ -651,7 +651,7 @@ static void pm121_create_cpu_fans(void)
/* First, locate the PID params in SMU SBD */
hdr = smu_get_sdb_partition(SMU_SDB_CPUPIDDATA_ID, NULL);
- if (hdr == 0) {
+ if (!hdr) {
printk(KERN_WARNING "pm121: CPU PID fan config not found.\n");
goto fail;
}
@@ -970,7 +970,7 @@ static int pm121_init_pm(void)
const struct smu_sdbp_header *hdr;
hdr = smu_get_sdb_partition(SMU_SDB_SENSORTREE_ID, NULL);
- if (hdr != 0) {
+ if (hdr) {
struct smu_sdbp_sensortree *st =
(struct smu_sdbp_sensortree *)&hdr[1];
pm121_mach_model = st->model_id;
diff --git a/drivers/macintosh/windfarm_pm81.c b/drivers/macintosh/windfarm_pm81.c
index e0f4743..257fb2c 100644
--- a/drivers/macintosh/windfarm_pm81.c
+++ b/drivers/macintosh/windfarm_pm81.c
@@ -401,7 +401,7 @@ static void wf_smu_create_cpu_fans(void)
/* First, locate the PID params in SMU SBD */
hdr = smu_get_sdb_partition(SMU_SDB_CPUPIDDATA_ID, NULL);
- if (hdr == 0) {
+ if (!hdr) {
printk(KERN_WARNING "windfarm: CPU PID fan config not found "
"max fan speed\n");
goto fail;
@@ -705,7 +705,7 @@ static int wf_init_pm(void)
const struct smu_sdbp_header *hdr;
hdr = smu_get_sdb_partition(SMU_SDB_SENSORTREE_ID, NULL);
- if (hdr != 0) {
+ if (hdr) {
struct smu_sdbp_sensortree *st =
(struct smu_sdbp_sensortree *)&hdr[1];
wf_smu_mach_model = st->model_id;
diff --git a/drivers/macintosh/windfarm_pm91.c b/drivers/macintosh/windfarm_pm91.c
index c853585..120a9cf 100644
--- a/drivers/macintosh/windfarm_pm91.c
+++ b/drivers/macintosh/windfarm_pm91.c
@@ -150,7 +150,7 @@ static void wf_smu_create_cpu_fans(void)
/* First, locate the PID params in SMU SBD */
hdr = smu_get_sdb_partition(SMU_SDB_CPUPIDDATA_ID, NULL);
- if (hdr == 0) {
+ if (!hdr) {
printk(KERN_WARNING "windfarm: CPU PID fan config not found "
"max fan speed\n");
goto fail;
diff --git a/drivers/macintosh/windfarm_smu_controls.c b/drivers/macintosh/windfarm_smu_controls.c
index e9957ad..bdd92b2 100644
--- a/drivers/macintosh/windfarm_smu_controls.c
+++ b/drivers/macintosh/windfarm_smu_controls.c
@@ -266,12 +266,11 @@ static int __init smu_controls_init(void)
return -ENODEV;
/* Look for RPM fans */
- for (fans = NULL; (fans = of_get_next_child(smu, fans)) != NULL;)
+ for_each_child_of_node(smu, fans)
if (of_node_name_eq(fans, "rpm-fans") ||
of_device_is_compatible(fans, "smu-rpm-fans"))
break;
- for (fan = NULL;
- fans && (fan = of_get_next_child(fans, fan)) != NULL;) {
+ for_each_child_of_node(fans, fan) {
struct smu_fan_control *fct;
fct = smu_fan_create(fan, 0);
@@ -286,11 +285,10 @@ static int __init smu_controls_init(void)
/* Look for PWM fans */
- for (fans = NULL; (fans = of_get_next_child(smu, fans)) != NULL;)
+ for_each_child_of_node(smu, fans)
if (of_node_name_eq(fans, "pwm-fans"))
break;
- for (fan = NULL;
- fans && (fan = of_get_next_child(fans, fan)) != NULL;) {
+ for_each_child_of_node(fans, fan) {
struct smu_fan_control *fct;
fct = smu_fan_create(fan, 1);
diff --git a/drivers/macintosh/windfarm_smu_sat.c b/drivers/macintosh/windfarm_smu_sat.c
index be5d459..ebc4256 100644
--- a/drivers/macintosh/windfarm_smu_sat.c
+++ b/drivers/macintosh/windfarm_smu_sat.c
@@ -189,8 +189,7 @@ static const struct wf_sensor_ops wf_sat_ops = {
.owner = THIS_MODULE,
};
-static int wf_sat_probe(struct i2c_client *client,
- const struct i2c_device_id *id)
+static int wf_sat_probe(struct i2c_client *client)
{
struct device_node *dev = client->dev.of_node;
struct wf_sat *sat;
@@ -349,7 +348,7 @@ static struct i2c_driver wf_sat_driver = {
.name = "wf_smu_sat",
.of_match_table = wf_sat_of_id,
},
- .probe = wf_sat_probe,
+ .probe_new = wf_sat_probe,
.remove = wf_sat_remove,
.id_table = wf_sat_id,
};
diff --git a/drivers/misc/cxl/pci.c b/drivers/misc/cxl/pci.c
index 6d495d6..0ff9448 100644
--- a/drivers/misc/cxl/pci.c
+++ b/drivers/misc/cxl/pci.c
@@ -387,6 +387,7 @@ int cxl_calc_capp_routing(struct pci_dev *dev, u64 *chipid,
rc = get_phb_index(np, phb_index);
if (rc) {
pr_err("cxl: invalid phb index\n");
+ of_node_put(np);
return rc;
}
diff --git a/drivers/misc/cxl/vphb.c b/drivers/misc/cxl/vphb.c
index 1264253..6332db8 100644
--- a/drivers/misc/cxl/vphb.c
+++ b/drivers/misc/cxl/vphb.c
@@ -67,12 +67,6 @@ static void cxl_pci_disable_device(struct pci_dev *dev)
}
}
-static resource_size_t cxl_pci_window_alignment(struct pci_bus *bus,
- unsigned long type)
-{
- return 1;
-}
-
static void cxl_pci_reset_secondary_bus(struct pci_dev *dev)
{
/* Should we do an AFU reset here ? */
@@ -200,7 +194,6 @@ static struct pci_controller_ops cxl_pci_controller_ops =
.enable_device_hook = cxl_pci_enable_device_hook,
.disable_device = cxl_pci_disable_device,
.release_device = cxl_pci_disable_device,
- .window_alignment = cxl_pci_window_alignment,
.reset_secondary_bus = cxl_pci_reset_secondary_bus,
.setup_msi_irqs = cxl_setup_msi_irqs,
.teardown_msi_irqs = cxl_teardown_msi_irqs,
diff --git a/drivers/misc/sram-exec.c b/drivers/misc/sram-exec.c
index a948e95..b71dbbd 100644
--- a/drivers/misc/sram-exec.c
+++ b/drivers/misc/sram-exec.c
@@ -10,9 +10,9 @@
#include <linux/genalloc.h>
#include <linux/mm.h>
#include <linux/sram.h>
+#include <linux/set_memory.h>
#include <asm/fncpy.h>
-#include <asm/set_memory.h>
#include "sram.h"
@@ -106,10 +106,7 @@ void *sram_exec_copy(struct gen_pool *pool, void *dst, void *src,
dst_cpy = fncpy(dst, src, size);
- ret = set_memory_ro((unsigned long)base, pages);
- if (ret)
- goto error_out;
- ret = set_memory_x((unsigned long)base, pages);
+ ret = set_memory_rox((unsigned long)base, pages);
if (ret)
goto error_out;
diff --git a/drivers/of/irq.c b/drivers/of/irq.c
index 2bac44f..e9bf523 100644
--- a/drivers/of/irq.c
+++ b/drivers/of/irq.c
@@ -730,6 +730,7 @@ struct irq_domain *of_msi_get_domain(struct device *dev,
return NULL;
}
+EXPORT_SYMBOL_GPL(of_msi_get_domain);
/**
* of_msi_configure - Set the msi_domain field of a device
diff --git a/drivers/parisc/led.c b/drivers/parisc/led.c
index d4be9d2..8bdc5e0 100644
--- a/drivers/parisc/led.c
+++ b/drivers/parisc/led.c
@@ -137,6 +137,9 @@ static int start_task(void)
/* Create the work queue and queue the LED task */
led_wq = create_singlethread_workqueue("led_wq");
+ if (!led_wq)
+ return -ENOMEM;
+
queue_delayed_work(led_wq, &led_task, 0);
return 0;
diff --git a/drivers/pci/controller/dwc/pcie-qcom-ep.c b/drivers/pci/controller/dwc/pcie-qcom-ep.c
index 6d0d1b7..19b3283 100644
--- a/drivers/pci/controller/dwc/pcie-qcom-ep.c
+++ b/drivers/pci/controller/dwc/pcie-qcom-ep.c
@@ -14,6 +14,7 @@
#include <linux/delay.h>
#include <linux/gpio/consumer.h>
#include <linux/mfd/syscon.h>
+#include <linux/phy/pcie.h>
#include <linux/phy/phy.h>
#include <linux/platform_device.h>
#include <linux/pm_domain.h>
@@ -268,6 +269,10 @@ static int qcom_pcie_enable_resources(struct qcom_pcie_ep *pcie_ep)
if (ret)
goto err_disable_clk;
+ ret = phy_set_mode_ext(pcie_ep->phy, PHY_MODE_PCIE, PHY_MODE_PCIE_EP);
+ if (ret)
+ goto err_phy_exit;
+
ret = phy_power_on(pcie_ep->phy);
if (ret)
goto err_phy_exit;
diff --git a/drivers/pci/controller/dwc/pcie-qcom.c b/drivers/pci/controller/dwc/pcie-qcom.c
index 38d5d46..77e5dc7 100644
--- a/drivers/pci/controller/dwc/pcie-qcom.c
+++ b/drivers/pci/controller/dwc/pcie-qcom.c
@@ -23,6 +23,7 @@
#include <linux/pci.h>
#include <linux/pm_runtime.h>
#include <linux/platform_device.h>
+#include <linux/phy/pcie.h>
#include <linux/phy/phy.h>
#include <linux/regulator/consumer.h>
#include <linux/reset.h>
@@ -1499,6 +1500,10 @@ static int qcom_pcie_host_init(struct dw_pcie_rp *pp)
if (ret)
return ret;
+ ret = phy_set_mode_ext(pcie->phy, PHY_MODE_PCIE, PHY_MODE_PCIE_RC);
+ if (ret)
+ goto err_deinit;
+
ret = phy_power_on(pcie->phy);
if (ret)
goto err_deinit;
diff --git a/drivers/phy/allwinner/phy-sun4i-usb.c b/drivers/phy/allwinner/phy-sun4i-usb.c
index 3a3831f..5472db9 100644
--- a/drivers/phy/allwinner/phy-sun4i-usb.c
+++ b/drivers/phy/allwinner/phy-sun4i-usb.c
@@ -120,6 +120,7 @@ struct sun4i_usb_phy_cfg {
u8 phyctl_offset;
bool dedicated_clocks;
bool phy0_dual_route;
+ bool needs_phy2_siddq;
int missing_phys;
};
@@ -289,6 +290,50 @@ static int sun4i_usb_phy_init(struct phy *_phy)
return ret;
}
+ /* Some PHYs on some SoCs need the help of PHY2 to work. */
+ if (data->cfg->needs_phy2_siddq && phy->index != 2) {
+ struct sun4i_usb_phy *phy2 = &data->phys[2];
+
+ ret = clk_prepare_enable(phy2->clk);
+ if (ret) {
+ reset_control_assert(phy->reset);
+ clk_disable_unprepare(phy->clk2);
+ clk_disable_unprepare(phy->clk);
+ return ret;
+ }
+
+ ret = reset_control_deassert(phy2->reset);
+ if (ret) {
+ clk_disable_unprepare(phy2->clk);
+ reset_control_assert(phy->reset);
+ clk_disable_unprepare(phy->clk2);
+ clk_disable_unprepare(phy->clk);
+ return ret;
+ }
+
+ /*
+ * This extra clock is just needed to access the
+ * REG_HCI_PHY_CTL PMU register for PHY2.
+ */
+ ret = clk_prepare_enable(phy2->clk2);
+ if (ret) {
+ reset_control_assert(phy2->reset);
+ clk_disable_unprepare(phy2->clk);
+ reset_control_assert(phy->reset);
+ clk_disable_unprepare(phy->clk2);
+ clk_disable_unprepare(phy->clk);
+ return ret;
+ }
+
+ if (phy2->pmu && data->cfg->hci_phy_ctl_clear) {
+ val = readl(phy2->pmu + REG_HCI_PHY_CTL);
+ val &= ~data->cfg->hci_phy_ctl_clear;
+ writel(val, phy2->pmu + REG_HCI_PHY_CTL);
+ }
+
+ clk_disable_unprepare(phy->clk2);
+ }
+
if (phy->pmu && data->cfg->hci_phy_ctl_clear) {
val = readl(phy->pmu + REG_HCI_PHY_CTL);
val &= ~data->cfg->hci_phy_ctl_clear;
@@ -354,6 +399,13 @@ static int sun4i_usb_phy_exit(struct phy *_phy)
data->phy0_init = false;
}
+ if (data->cfg->needs_phy2_siddq && phy->index != 2) {
+ struct sun4i_usb_phy *phy2 = &data->phys[2];
+
+ clk_disable_unprepare(phy2->clk);
+ reset_control_assert(phy2->reset);
+ }
+
sun4i_usb_phy_passby(phy, 0);
reset_control_assert(phy->reset);
clk_disable_unprepare(phy->clk2);
@@ -785,6 +837,13 @@ static int sun4i_usb_phy_probe(struct platform_device *pdev)
dev_err(dev, "failed to get clock %s\n", name);
return PTR_ERR(phy->clk2);
}
+ } else {
+ snprintf(name, sizeof(name), "pmu%d_clk", i);
+ phy->clk2 = devm_clk_get_optional(dev, name);
+ if (IS_ERR(phy->clk2)) {
+ dev_err(dev, "failed to get clock %s\n", name);
+ return PTR_ERR(phy->clk2);
+ }
}
snprintf(name, sizeof(name), "usb%d_reset", i);
@@ -973,6 +1032,17 @@ static const struct sun4i_usb_phy_cfg sun50i_h6_cfg = {
.missing_phys = BIT(1) | BIT(2),
};
+static const struct sun4i_usb_phy_cfg sun50i_h616_cfg = {
+ .num_phys = 4,
+ .type = sun50i_h6_phy,
+ .disc_thresh = 3,
+ .phyctl_offset = REG_PHYCTL_A33,
+ .dedicated_clocks = true,
+ .phy0_dual_route = true,
+ .hci_phy_ctl_clear = PHY_CTL_SIDDQ,
+ .needs_phy2_siddq = true,
+};
+
static const struct of_device_id sun4i_usb_phy_of_match[] = {
{ .compatible = "allwinner,sun4i-a10-usb-phy", .data = &sun4i_a10_cfg },
{ .compatible = "allwinner,sun5i-a13-usb-phy", .data = &sun5i_a13_cfg },
@@ -988,6 +1058,7 @@ static const struct of_device_id sun4i_usb_phy_of_match[] = {
{ .compatible = "allwinner,sun50i-a64-usb-phy",
.data = &sun50i_a64_cfg},
{ .compatible = "allwinner,sun50i-h6-usb-phy", .data = &sun50i_h6_cfg },
+ { .compatible = "allwinner,sun50i-h616-usb-phy", .data = &sun50i_h616_cfg },
{ },
};
MODULE_DEVICE_TABLE(of, sun4i_usb_phy_of_match);
diff --git a/drivers/phy/allwinner/phy-sun6i-mipi-dphy.c b/drivers/phy/allwinner/phy-sun6i-mipi-dphy.c
index 3900f16..36eab952 100644
--- a/drivers/phy/allwinner/phy-sun6i-mipi-dphy.c
+++ b/drivers/phy/allwinner/phy-sun6i-mipi-dphy.c
@@ -70,11 +70,19 @@
#define SUN6I_DPHY_ANA0_REG 0x4c
#define SUN6I_DPHY_ANA0_REG_PWS BIT(31)
+#define SUN6I_DPHY_ANA0_REG_PWEND BIT(30)
+#define SUN6I_DPHY_ANA0_REG_PWENC BIT(29)
#define SUN6I_DPHY_ANA0_REG_DMPC BIT(28)
#define SUN6I_DPHY_ANA0_REG_DMPD(n) (((n) & 0xf) << 24)
+#define SUN6I_DPHY_ANA0_REG_SRXDT(n) (((n) & 0xf) << 20)
+#define SUN6I_DPHY_ANA0_REG_SRXCK(n) (((n) & 0xf) << 16)
+#define SUN6I_DPHY_ANA0_REG_SDIV2 BIT(15)
#define SUN6I_DPHY_ANA0_REG_SLV(n) (((n) & 7) << 12)
#define SUN6I_DPHY_ANA0_REG_DEN(n) (((n) & 0xf) << 8)
+#define SUN6I_DPHY_ANA0_REG_PLR(n) (((n) & 0xf) << 4)
#define SUN6I_DPHY_ANA0_REG_SFB(n) (((n) & 3) << 2)
+#define SUN6I_DPHY_ANA0_REG_RSD BIT(1)
+#define SUN6I_DPHY_ANA0_REG_SELSCK BIT(0)
#define SUN6I_DPHY_ANA1_REG 0x50
#define SUN6I_DPHY_ANA1_REG_VTTMODE BIT(31)
@@ -97,8 +105,13 @@
#define SUN6I_DPHY_ANA3_EN_LDOR BIT(18)
#define SUN6I_DPHY_ANA4_REG 0x5c
+#define SUN6I_DPHY_ANA4_REG_EN_MIPI BIT(31)
+#define SUN6I_DPHY_ANA4_REG_EN_COMTEST BIT(30)
+#define SUN6I_DPHY_ANA4_REG_COMTEST(n) (((n) & 3) << 28)
+#define SUN6I_DPHY_ANA4_REG_IB(n) (((n) & 3) << 25)
#define SUN6I_DPHY_ANA4_REG_DMPLVC BIT(24)
#define SUN6I_DPHY_ANA4_REG_DMPLVD(n) (((n) & 0xf) << 20)
+#define SUN6I_DPHY_ANA4_REG_VTT_SET(n) (((n) & 0x7) << 17)
#define SUN6I_DPHY_ANA4_REG_CKDV(n) (((n) & 0x1f) << 12)
#define SUN6I_DPHY_ANA4_REG_TMSC(n) (((n) & 3) << 10)
#define SUN6I_DPHY_ANA4_REG_TMSD(n) (((n) & 3) << 8)
@@ -109,11 +122,68 @@
#define SUN6I_DPHY_DBG5_REG 0xf4
+#define SUN50I_DPHY_TX_SLEW_REG0 0xf8
+#define SUN50I_DPHY_TX_SLEW_REG1 0xfc
+#define SUN50I_DPHY_TX_SLEW_REG2 0x100
+
+#define SUN50I_DPHY_PLL_REG0 0x104
+#define SUN50I_DPHY_PLL_REG0_CP36_EN BIT(23)
+#define SUN50I_DPHY_PLL_REG0_LDO_EN BIT(22)
+#define SUN50I_DPHY_PLL_REG0_EN_LVS BIT(21)
+#define SUN50I_DPHY_PLL_REG0_PLL_EN BIT(20)
+#define SUN50I_DPHY_PLL_REG0_P(n) (((n) & 0xf) << 16)
+#define SUN50I_DPHY_PLL_REG0_N(n) (((n) & 0xff) << 8)
+#define SUN50I_DPHY_PLL_REG0_NDET BIT(7)
+#define SUN50I_DPHY_PLL_REG0_TDIV BIT(6)
+#define SUN50I_DPHY_PLL_REG0_M0(n) (((n) & 3) << 4)
+#define SUN50I_DPHY_PLL_REG0_M1(n) ((n) & 0xf)
+
+#define SUN50I_DPHY_PLL_REG1 0x108
+#define SUN50I_DPHY_PLL_REG1_UNLOCK_MDSEL(n) (((n) & 3) << 14)
+#define SUN50I_DPHY_PLL_REG1_LOCKMDSEL BIT(13)
+#define SUN50I_DPHY_PLL_REG1_LOCKDET_EN BIT(12)
+#define SUN50I_DPHY_PLL_REG1_VSETA(n) (((n) & 0x7) << 9)
+#define SUN50I_DPHY_PLL_REG1_VSETD(n) (((n) & 0x7) << 6)
+#define SUN50I_DPHY_PLL_REG1_LPF_SW BIT(5)
+#define SUN50I_DPHY_PLL_REG1_ICP_SEL(n) (((n) & 3) << 3)
+#define SUN50I_DPHY_PLL_REG1_ATEST_SEL(n) (((n) & 3) << 1)
+#define SUN50I_DPHY_PLL_REG1_TEST_EN BIT(0)
+
+#define SUN50I_DPHY_PLL_REG2 0x10c
+#define SUN50I_DPHY_PLL_REG2_SDM_EN BIT(31)
+#define SUN50I_DPHY_PLL_REG2_FF_EN BIT(30)
+#define SUN50I_DPHY_PLL_REG2_SS_EN BIT(29)
+#define SUN50I_DPHY_PLL_REG2_SS_FRAC(n) (((n) & 0x1ff) << 20)
+#define SUN50I_DPHY_PLL_REG2_SS_INT(n) (((n) & 0xff) << 12)
+#define SUN50I_DPHY_PLL_REG2_FRAC(n) ((n) & 0xfff)
+
+#define SUN50I_COMBO_PHY_REG0 0x110
+#define SUN50I_COMBO_PHY_REG0_EN_TEST_COMBOLDO BIT(5)
+#define SUN50I_COMBO_PHY_REG0_EN_TEST_0P8 BIT(4)
+#define SUN50I_COMBO_PHY_REG0_EN_MIPI BIT(3)
+#define SUN50I_COMBO_PHY_REG0_EN_LVDS BIT(2)
+#define SUN50I_COMBO_PHY_REG0_EN_COMBOLDO BIT(1)
+#define SUN50I_COMBO_PHY_REG0_EN_CP BIT(0)
+
+#define SUN50I_COMBO_PHY_REG1 0x114
+#define SUN50I_COMBO_PHY_REG2_REG_VREF1P6(n) (((n) & 0x7) << 4)
+#define SUN50I_COMBO_PHY_REG2_REG_VREF0P8(n) ((n) & 0x7)
+
+#define SUN50I_COMBO_PHY_REG2 0x118
+#define SUN50I_COMBO_PHY_REG2_HS_STOP_DLY(n) ((n) & 0xff)
+
enum sun6i_dphy_direction {
SUN6I_DPHY_DIRECTION_TX,
SUN6I_DPHY_DIRECTION_RX,
};
+struct sun6i_dphy;
+
+struct sun6i_dphy_variant {
+ void (*tx_power_on)(struct sun6i_dphy *dphy);
+ bool rx_supported;
+};
+
struct sun6i_dphy {
struct clk *bus_clk;
struct clk *mod_clk;
@@ -123,6 +193,7 @@ struct sun6i_dphy {
struct phy *phy;
struct phy_configure_opts_mipi_dphy config;
+ const struct sun6i_dphy_variant *variant;
enum sun6i_dphy_direction direction;
};
@@ -151,37 +222,10 @@ static int sun6i_dphy_configure(struct phy *phy, union phy_configure_opts *opts)
return 0;
}
-static int sun6i_dphy_tx_power_on(struct sun6i_dphy *dphy)
+static void sun6i_a31_mipi_dphy_tx_power_on(struct sun6i_dphy *dphy)
{
u8 lanes_mask = GENMASK(dphy->config.lanes - 1, 0);
- regmap_write(dphy->regs, SUN6I_DPHY_TX_CTL_REG,
- SUN6I_DPHY_TX_CTL_HS_TX_CLK_CONT);
-
- regmap_write(dphy->regs, SUN6I_DPHY_TX_TIME0_REG,
- SUN6I_DPHY_TX_TIME0_LP_CLK_DIV(14) |
- SUN6I_DPHY_TX_TIME0_HS_PREPARE(6) |
- SUN6I_DPHY_TX_TIME0_HS_TRAIL(10));
-
- regmap_write(dphy->regs, SUN6I_DPHY_TX_TIME1_REG,
- SUN6I_DPHY_TX_TIME1_CLK_PREPARE(7) |
- SUN6I_DPHY_TX_TIME1_CLK_ZERO(50) |
- SUN6I_DPHY_TX_TIME1_CLK_PRE(3) |
- SUN6I_DPHY_TX_TIME1_CLK_POST(10));
-
- regmap_write(dphy->regs, SUN6I_DPHY_TX_TIME2_REG,
- SUN6I_DPHY_TX_TIME2_CLK_TRAIL(30));
-
- regmap_write(dphy->regs, SUN6I_DPHY_TX_TIME3_REG, 0);
-
- regmap_write(dphy->regs, SUN6I_DPHY_TX_TIME4_REG,
- SUN6I_DPHY_TX_TIME4_HS_TX_ANA0(3) |
- SUN6I_DPHY_TX_TIME4_HS_TX_ANA1(3));
-
- regmap_write(dphy->regs, SUN6I_DPHY_GCTL_REG,
- SUN6I_DPHY_GCTL_LANE_NUM(dphy->config.lanes) |
- SUN6I_DPHY_GCTL_EN);
-
regmap_write(dphy->regs, SUN6I_DPHY_ANA0_REG,
SUN6I_DPHY_ANA0_REG_PWS |
SUN6I_DPHY_ANA0_REG_DMPC |
@@ -213,6 +257,106 @@ static int sun6i_dphy_tx_power_on(struct sun6i_dphy *dphy)
SUN6I_DPHY_ANA3_EN_LDOC |
SUN6I_DPHY_ANA3_EN_LDOD);
udelay(1);
+}
+
+static void sun50i_a100_mipi_dphy_tx_power_on(struct sun6i_dphy *dphy)
+{
+ unsigned long mipi_symbol_rate = dphy->config.hs_clk_rate;
+ unsigned int div, n;
+
+ regmap_write(dphy->regs, SUN6I_DPHY_ANA4_REG,
+ SUN6I_DPHY_ANA4_REG_IB(2) |
+ SUN6I_DPHY_ANA4_REG_DMPLVD(4) |
+ SUN6I_DPHY_ANA4_REG_VTT_SET(3) |
+ SUN6I_DPHY_ANA4_REG_CKDV(3) |
+ SUN6I_DPHY_ANA4_REG_TMSD(1) |
+ SUN6I_DPHY_ANA4_REG_TMSC(1) |
+ SUN6I_DPHY_ANA4_REG_TXPUSD(2) |
+ SUN6I_DPHY_ANA4_REG_TXPUSC(3) |
+ SUN6I_DPHY_ANA4_REG_TXDNSD(2) |
+ SUN6I_DPHY_ANA4_REG_TXDNSC(3));
+
+ regmap_update_bits(dphy->regs, SUN6I_DPHY_ANA2_REG,
+ SUN6I_DPHY_ANA2_EN_CK_CPU,
+ SUN6I_DPHY_ANA2_EN_CK_CPU);
+
+ regmap_update_bits(dphy->regs, SUN6I_DPHY_ANA2_REG,
+ SUN6I_DPHY_ANA2_REG_ENIB,
+ SUN6I_DPHY_ANA2_REG_ENIB);
+
+ regmap_write(dphy->regs, SUN6I_DPHY_ANA3_REG,
+ SUN6I_DPHY_ANA3_EN_LDOR |
+ SUN6I_DPHY_ANA3_EN_LDOC |
+ SUN6I_DPHY_ANA3_EN_LDOD);
+
+ regmap_write(dphy->regs, SUN6I_DPHY_ANA0_REG,
+ SUN6I_DPHY_ANA0_REG_PLR(4) |
+ SUN6I_DPHY_ANA0_REG_SFB(1));
+
+ regmap_write(dphy->regs, SUN50I_COMBO_PHY_REG0,
+ SUN50I_COMBO_PHY_REG0_EN_CP);
+
+ /* Choose a divider to limit the VCO frequency to around 2 GHz. */
+ div = 16 >> order_base_2(DIV_ROUND_UP(mipi_symbol_rate, 264000000));
+ n = mipi_symbol_rate * div / 24000000;
+
+ regmap_write(dphy->regs, SUN50I_DPHY_PLL_REG0,
+ SUN50I_DPHY_PLL_REG0_CP36_EN |
+ SUN50I_DPHY_PLL_REG0_LDO_EN |
+ SUN50I_DPHY_PLL_REG0_EN_LVS |
+ SUN50I_DPHY_PLL_REG0_PLL_EN |
+ SUN50I_DPHY_PLL_REG0_NDET |
+ SUN50I_DPHY_PLL_REG0_P((div - 1) % 8) |
+ SUN50I_DPHY_PLL_REG0_N(n) |
+ SUN50I_DPHY_PLL_REG0_M0((div - 1) / 8) |
+ SUN50I_DPHY_PLL_REG0_M1(2));
+
+ /* Disable sigma-delta modulation. */
+ regmap_write(dphy->regs, SUN50I_DPHY_PLL_REG2, 0);
+
+ regmap_update_bits(dphy->regs, SUN6I_DPHY_ANA4_REG,
+ SUN6I_DPHY_ANA4_REG_EN_MIPI,
+ SUN6I_DPHY_ANA4_REG_EN_MIPI);
+
+ regmap_update_bits(dphy->regs, SUN50I_COMBO_PHY_REG0,
+ SUN50I_COMBO_PHY_REG0_EN_MIPI |
+ SUN50I_COMBO_PHY_REG0_EN_COMBOLDO,
+ SUN50I_COMBO_PHY_REG0_EN_MIPI |
+ SUN50I_COMBO_PHY_REG0_EN_COMBOLDO);
+
+ regmap_write(dphy->regs, SUN50I_COMBO_PHY_REG2,
+ SUN50I_COMBO_PHY_REG2_HS_STOP_DLY(20));
+ udelay(1);
+}
+
+static int sun6i_dphy_tx_power_on(struct sun6i_dphy *dphy)
+{
+ u8 lanes_mask = GENMASK(dphy->config.lanes - 1, 0);
+
+ regmap_write(dphy->regs, SUN6I_DPHY_TX_CTL_REG,
+ SUN6I_DPHY_TX_CTL_HS_TX_CLK_CONT);
+
+ regmap_write(dphy->regs, SUN6I_DPHY_TX_TIME0_REG,
+ SUN6I_DPHY_TX_TIME0_LP_CLK_DIV(14) |
+ SUN6I_DPHY_TX_TIME0_HS_PREPARE(6) |
+ SUN6I_DPHY_TX_TIME0_HS_TRAIL(10));
+
+ regmap_write(dphy->regs, SUN6I_DPHY_TX_TIME1_REG,
+ SUN6I_DPHY_TX_TIME1_CLK_PREPARE(7) |
+ SUN6I_DPHY_TX_TIME1_CLK_ZERO(50) |
+ SUN6I_DPHY_TX_TIME1_CLK_PRE(3) |
+ SUN6I_DPHY_TX_TIME1_CLK_POST(10));
+
+ regmap_write(dphy->regs, SUN6I_DPHY_TX_TIME2_REG,
+ SUN6I_DPHY_TX_TIME2_CLK_TRAIL(30));
+
+ regmap_write(dphy->regs, SUN6I_DPHY_TX_TIME3_REG, 0);
+
+ regmap_write(dphy->regs, SUN6I_DPHY_TX_TIME4_REG,
+ SUN6I_DPHY_TX_TIME4_HS_TX_ANA0(3) |
+ SUN6I_DPHY_TX_TIME4_HS_TX_ANA1(3));
+
+ dphy->variant->tx_power_on(dphy);
regmap_update_bits(dphy->regs, SUN6I_DPHY_ANA3_REG,
SUN6I_DPHY_ANA3_EN_VTTC |
@@ -239,6 +383,10 @@ static int sun6i_dphy_tx_power_on(struct sun6i_dphy *dphy)
SUN6I_DPHY_ANA2_EN_P2S_CPU_MASK,
SUN6I_DPHY_ANA2_EN_P2S_CPU(lanes_mask));
+ regmap_write(dphy->regs, SUN6I_DPHY_GCTL_REG,
+ SUN6I_DPHY_GCTL_LANE_NUM(dphy->config.lanes) |
+ SUN6I_DPHY_GCTL_EN);
+
return 0;
}
@@ -393,7 +541,7 @@ static const struct regmap_config sun6i_dphy_regmap_config = {
.reg_bits = 32,
.val_bits = 32,
.reg_stride = 4,
- .max_register = SUN6I_DPHY_DBG5_REG,
+ .max_register = SUN50I_COMBO_PHY_REG2,
.name = "mipi-dphy",
};
@@ -409,6 +557,10 @@ static int sun6i_dphy_probe(struct platform_device *pdev)
if (!dphy)
return -ENOMEM;
+ dphy->variant = device_get_match_data(&pdev->dev);
+ if (!dphy->variant)
+ return -EINVAL;
+
regs = devm_platform_ioremap_resource(pdev, 0);
if (IS_ERR(regs)) {
dev_err(&pdev->dev, "Couldn't map the DPHY encoder registers\n");
@@ -445,8 +597,14 @@ static int sun6i_dphy_probe(struct platform_device *pdev)
ret = of_property_read_string(pdev->dev.of_node, "allwinner,direction",
&direction);
- if (!ret && !strncmp(direction, "rx", 2))
+ if (!ret && !strncmp(direction, "rx", 2)) {
+ if (!dphy->variant->rx_supported) {
+ dev_err(&pdev->dev, "RX not supported on this variant\n");
+ return -EOPNOTSUPP;
+ }
+
dphy->direction = SUN6I_DPHY_DIRECTION_RX;
+ }
phy_set_drvdata(dphy->phy, dphy);
phy_provider = devm_of_phy_provider_register(&pdev->dev, of_phy_simple_xlate);
@@ -454,8 +612,24 @@ static int sun6i_dphy_probe(struct platform_device *pdev)
return PTR_ERR_OR_ZERO(phy_provider);
}
+static const struct sun6i_dphy_variant sun6i_a31_mipi_dphy_variant = {
+ .tx_power_on = sun6i_a31_mipi_dphy_tx_power_on,
+ .rx_supported = true,
+};
+
+static const struct sun6i_dphy_variant sun50i_a100_mipi_dphy_variant = {
+ .tx_power_on = sun50i_a100_mipi_dphy_tx_power_on,
+};
+
static const struct of_device_id sun6i_dphy_of_table[] = {
- { .compatible = "allwinner,sun6i-a31-mipi-dphy" },
+ {
+ .compatible = "allwinner,sun6i-a31-mipi-dphy",
+ .data = &sun6i_a31_mipi_dphy_variant,
+ },
+ {
+ .compatible = "allwinner,sun50i-a100-mipi-dphy",
+ .data = &sun50i_a100_mipi_dphy_variant,
+ },
{ }
};
MODULE_DEVICE_TABLE(of, sun6i_dphy_of_table);
diff --git a/drivers/phy/broadcom/phy-brcm-usb-init-synopsys.c b/drivers/phy/broadcom/phy-brcm-usb-init-synopsys.c
index d2524b7..76cf428 100644
--- a/drivers/phy/broadcom/phy-brcm-usb-init-synopsys.c
+++ b/drivers/phy/broadcom/phy-brcm-usb-init-synopsys.c
@@ -18,14 +18,14 @@
#define PIARBCTL_CAM 0x00
#define PIARBCTL_SPLITTER 0x04
#define PIARBCTL_MISC 0x08
-#define PIARBCTL_MISC_SECURE_MASK 0x80000000
-#define PIARBCTL_MISC_USB_SELECT_MASK 0x40000000
-#define PIARBCTL_MISC_USB_4G_SDRAM_MASK 0x20000000
-#define PIARBCTL_MISC_USB_PRIORITY_MASK 0x000f0000
-#define PIARBCTL_MISC_USB_MEM_PAGE_MASK 0x0000f000
-#define PIARBCTL_MISC_CAM1_MEM_PAGE_MASK 0x00000f00
-#define PIARBCTL_MISC_CAM0_MEM_PAGE_MASK 0x000000f0
-#define PIARBCTL_MISC_SATA_PRIORITY_MASK 0x0000000f
+#define PIARBCTL_MISC_SATA_PRIORITY_MASK GENMASK(3, 0)
+#define PIARBCTL_MISC_CAM0_MEM_PAGE_MASK GENMASK(7, 4)
+#define PIARBCTL_MISC_CAM1_MEM_PAGE_MASK GENMASK(11, 8)
+#define PIARBCTL_MISC_USB_MEM_PAGE_MASK GENMASK(15, 12)
+#define PIARBCTL_MISC_USB_PRIORITY_MASK GENMASK(19, 16)
+#define PIARBCTL_MISC_USB_4G_SDRAM_MASK BIT(29)
+#define PIARBCTL_MISC_USB_SELECT_MASK BIT(30)
+#define PIARBCTL_MISC_SECURE_MASK BIT(31)
#define PIARBCTL_MISC_USB_ONLY_MASK \
(PIARBCTL_MISC_USB_SELECT_MASK | \
@@ -35,46 +35,47 @@
/* Register definitions for the USB CTRL block */
#define USB_CTRL_SETUP 0x00
-#define USB_CTRL_SETUP_STRAP_IPP_SEL_MASK 0x02000000
-#define USB_CTRL_SETUP_SCB2_EN_MASK 0x00008000
-#define USB_CTRL_SETUP_tca_drv_sel_MASK 0x01000000
-#define USB_CTRL_SETUP_SCB1_EN_MASK 0x00004000
-#define USB_CTRL_SETUP_SOFT_SHUTDOWN_MASK 0x00000200
-#define USB_CTRL_SETUP_IPP_MASK 0x00000020
-#define USB_CTRL_SETUP_IOC_MASK 0x00000010
+#define USB_CTRL_SETUP_IOC_MASK BIT(4)
+#define USB_CTRL_SETUP_IPP_MASK BIT(5)
+#define USB_CTRL_SETUP_SOFT_SHUTDOWN_MASK BIT(9)
+#define USB_CTRL_SETUP_SCB1_EN_MASK BIT(14)
+#define USB_CTRL_SETUP_SCB2_EN_MASK BIT(15)
+#define USB_CTRL_SETUP_tca_drv_sel_MASK BIT(24)
+#define USB_CTRL_SETUP_STRAP_IPP_SEL_MASK BIT(25)
#define USB_CTRL_USB_PM 0x04
-#define USB_CTRL_USB_PM_USB_PWRDN_MASK 0x80000000
-#define USB_CTRL_USB_PM_SOFT_RESET_MASK 0x40000000
-#define USB_CTRL_USB_PM_BDC_SOFT_RESETB_MASK 0x00800000
-#define USB_CTRL_USB_PM_XHC_SOFT_RESETB_MASK 0x00400000
-#define USB_CTRL_USB_PM_XHC_PME_EN_MASK 0x00000010
-#define USB_CTRL_USB_PM_XHC_S2_CLK_SWITCH_EN_MASK 0x00000008
+#define USB_CTRL_USB_PM_XHC_S2_CLK_SWITCH_EN_MASK BIT(3)
+#define USB_CTRL_USB_PM_XHC_PME_EN_MASK BIT(4)
+#define USB_CTRL_USB_PM_XHC_SOFT_RESETB_MASK BIT(22)
+#define USB_CTRL_USB_PM_BDC_SOFT_RESETB_MASK BIT(23)
+#define USB_CTRL_USB_PM_SOFT_RESET_MASK BIT(30)
+#define USB_CTRL_USB_PM_USB_PWRDN_MASK BIT(31)
#define USB_CTRL_USB_PM_STATUS 0x08
#define USB_CTRL_USB_DEVICE_CTL1 0x10
-#define USB_CTRL_USB_DEVICE_CTL1_PORT_MODE_MASK 0x00000003
+#define USB_CTRL_USB_DEVICE_CTL1_PORT_MODE_MASK GENMASK(1, 0)
#define USB_CTRL_TEST_PORT_CTL 0x30
-#define USB_CTRL_TEST_PORT_CTL_TPOUT_SEL_MASK 0x000000ff
+#define USB_CTRL_TEST_PORT_CTL_TPOUT_SEL_MASK GENMASK(7, 0)
#define USB_CTRL_TEST_PORT_CTL_TPOUT_SEL_PME_GEN_MASK 0x0000002e
#define USB_CTRL_TP_DIAG1 0x34
-#define USB_CTLR_TP_DIAG1_wake_MASK 0x00000002
+#define USB_CTLR_TP_DIAG1_wake_MASK BIT(1)
#define USB_CTRL_CTLR_CSHCR 0x50
-#define USB_CTRL_CTLR_CSHCR_ctl_pme_en_MASK 0x00040000
+#define USB_CTRL_CTLR_CSHCR_ctl_pme_en_MASK BIT(18)
/* Register definitions for the USB_PHY block in 7211b0 */
#define USB_PHY_PLL_CTL 0x00
-#define USB_PHY_PLL_CTL_PLL_RESETB_MASK 0x40000000
+#define USB_PHY_PLL_CTL_PLL_SUSPEND_MASK BIT(27)
+#define USB_PHY_PLL_CTL_PLL_RESETB_MASK BIT(30)
#define USB_PHY_PLL_LDO_CTL 0x08
-#define USB_PHY_PLL_LDO_CTL_AFE_CORERDY_MASK 0x00000004
-#define USB_PHY_PLL_LDO_CTL_AFE_LDO_PWRDWNB_MASK 0x00000002
-#define USB_PHY_PLL_LDO_CTL_AFE_BG_PWRDWNB_MASK 0x00000001
+#define USB_PHY_PLL_LDO_CTL_AFE_BG_PWRDWNB_MASK BIT(0)
+#define USB_PHY_PLL_LDO_CTL_AFE_LDO_PWRDWNB_MASK BIT(1)
+#define USB_PHY_PLL_LDO_CTL_AFE_CORERDY_MASK BIT(2)
#define USB_PHY_UTMI_CTL_1 0x04
-#define USB_PHY_UTMI_CTL_1_POWER_UP_FSM_EN_MASK 0x00000800
-#define USB_PHY_UTMI_CTL_1_PHY_MODE_MASK 0x0000000c
+#define USB_PHY_UTMI_CTL_1_PHY_MODE_MASK GENMASK(3, 2)
#define USB_PHY_UTMI_CTL_1_PHY_MODE_SHIFT 2
+#define USB_PHY_UTMI_CTL_1_POWER_UP_FSM_EN_MASK BIT(11)
#define USB_PHY_IDDQ 0x1c
-#define USB_PHY_IDDQ_phy_iddq_MASK 0x00000001
+#define USB_PHY_IDDQ_phy_iddq_MASK BIT(0)
#define USB_PHY_STATUS 0x20
-#define USB_PHY_STATUS_pll_lock_MASK 0x00000001
+#define USB_PHY_STATUS_pll_lock_MASK BIT(0)
/* Register definitions for the MDIO registers in the DWC2 block of
* the 7211b0.
@@ -86,7 +87,7 @@
/* Register definitions for the BDC EC block in 7211b0 */
#define BDC_EC_AXIRDA 0x0c
-#define BDC_EC_AXIRDA_RTS_MASK 0xf0000000
+#define BDC_EC_AXIRDA_RTS_MASK GENMASK(31, 28)
#define BDC_EC_AXIRDA_RTS_SHIFT 28
@@ -195,10 +196,10 @@ static void usb_init_common(struct brcm_usb_init_params *params)
if (USB_CTRL_MASK(USB_DEVICE_CTL1, PORT_MODE)) {
reg = brcm_usb_readl(USB_CTRL_REG(ctrl, USB_DEVICE_CTL1));
reg &= ~USB_CTRL_MASK(USB_DEVICE_CTL1, PORT_MODE);
- reg |= params->mode;
+ reg |= params->port_mode;
brcm_usb_writel(reg, USB_CTRL_REG(ctrl, USB_DEVICE_CTL1));
}
- switch (params->mode) {
+ switch (params->supported_port_modes) {
case USB_CTLR_MODE_HOST:
USB_CTRL_UNSET(ctrl, USB_PM, BDC_SOFT_RESETB);
break;
@@ -259,6 +260,11 @@ static void usb_init_common_7211b0(struct brcm_usb_init_params *params)
brcm_usb_writel(reg, usb_phy + USB_PHY_UTMI_CTL_1);
}
+ /* Disable PLL auto suspend */
+ reg = brcm_usb_readl(usb_phy + USB_PHY_PLL_CTL);
+ reg |= USB_PHY_PLL_CTL_PLL_SUSPEND_MASK;
+ brcm_usb_writel(reg, usb_phy + USB_PHY_PLL_CTL);
+
/* Init the PHY */
reg = USB_PHY_PLL_LDO_CTL_AFE_CORERDY_MASK |
USB_PHY_PLL_LDO_CTL_AFE_LDO_PWRDWNB_MASK |
@@ -276,7 +282,7 @@ static void usb_init_common_7211b0(struct brcm_usb_init_params *params)
/* Set the PHY_MODE */
reg = brcm_usb_readl(usb_phy + USB_PHY_UTMI_CTL_1);
reg &= ~USB_PHY_UTMI_CTL_1_PHY_MODE_MASK;
- reg |= params->mode << USB_PHY_UTMI_CTL_1_PHY_MODE_SHIFT;
+ reg |= params->supported_port_modes << USB_PHY_UTMI_CTL_1_PHY_MODE_SHIFT;
brcm_usb_writel(reg, usb_phy + USB_PHY_UTMI_CTL_1);
usb_init_common(params);
@@ -286,7 +292,7 @@ static void usb_init_common_7211b0(struct brcm_usb_init_params *params)
* the default "Read Transaction Size" of 6 (1024 bytes).
* Set it to 4 (256 bytes).
*/
- if ((params->mode != USB_CTLR_MODE_HOST) && bdc_ec) {
+ if ((params->supported_port_modes != USB_CTLR_MODE_HOST) && bdc_ec) {
reg = brcm_usb_readl(bdc_ec + BDC_EC_AXIRDA);
reg &= ~BDC_EC_AXIRDA_RTS_MASK;
reg |= (0x4 << BDC_EC_AXIRDA_RTS_SHIFT);
@@ -331,13 +337,12 @@ static void usb_uninit_common_7216(struct brcm_usb_init_params *params)
pr_debug("%s\n", __func__);
- if (!params->wake_enabled) {
- USB_CTRL_SET(ctrl, USB_PM, USB_PWRDN);
-
+ if (params->wake_enabled) {
/* Switch to using slower clock during suspend to save power */
USB_CTRL_SET(ctrl, USB_PM, XHC_S2_CLK_SWITCH_EN);
- } else {
usb_wake_enable_7216(params, true);
+ } else {
+ USB_CTRL_SET(ctrl, USB_PM, USB_PWRDN);
}
}
@@ -385,7 +390,7 @@ static int usb_get_dual_select(struct brcm_usb_init_params *params)
return reg;
}
-static void usb_set_dual_select(struct brcm_usb_init_params *params, int mode)
+static void usb_set_dual_select(struct brcm_usb_init_params *params)
{
void __iomem *ctrl = params->regs[BRCM_REGS_CTRL];
u32 reg;
@@ -394,7 +399,7 @@ static void usb_set_dual_select(struct brcm_usb_init_params *params, int mode)
reg = brcm_usb_readl(USB_CTRL_REG(ctrl, USB_DEVICE_CTL1));
reg &= ~USB_CTRL_MASK(USB_DEVICE_CTL1, PORT_MODE);
- reg |= mode;
+ reg |= params->port_mode;
brcm_usb_writel(reg, USB_CTRL_REG(ctrl, USB_DEVICE_CTL1));
}
@@ -425,7 +430,6 @@ void brcm_usb_dvr_init_7216(struct brcm_usb_init_params *params)
params->family_name = "7216";
params->ops = &bcm7216_ops;
- params->suspend_with_clocks = true;
}
void brcm_usb_dvr_init_7211b0(struct brcm_usb_init_params *params)
@@ -435,5 +439,4 @@ void brcm_usb_dvr_init_7211b0(struct brcm_usb_init_params *params)
params->family_name = "7211";
params->ops = &bcm7211b0_ops;
- params->suspend_with_clocks = true;
}
diff --git a/drivers/phy/broadcom/phy-brcm-usb-init.c b/drivers/phy/broadcom/phy-brcm-usb-init.c
index dddcbd3..a1ca833 100644
--- a/drivers/phy/broadcom/phy-brcm-usb-init.c
+++ b/drivers/phy/broadcom/phy-brcm-usb-init.c
@@ -21,57 +21,57 @@
/* Register definitions for the USB CTRL block */
#define USB_CTRL_SETUP 0x00
-#define USB_CTRL_SETUP_IOC_MASK 0x00000010
-#define USB_CTRL_SETUP_IPP_MASK 0x00000020
-#define USB_CTRL_SETUP_BABO_MASK 0x00000001
-#define USB_CTRL_SETUP_FNHW_MASK 0x00000002
-#define USB_CTRL_SETUP_FNBO_MASK 0x00000004
-#define USB_CTRL_SETUP_WABO_MASK 0x00000008
-#define USB_CTRL_SETUP_SCB_CLIENT_SWAP_MASK 0x00002000 /* option */
-#define USB_CTRL_SETUP_SCB1_EN_MASK 0x00004000 /* option */
-#define USB_CTRL_SETUP_SCB2_EN_MASK 0x00008000 /* option */
-#define USB_CTRL_SETUP_SS_EHCI64BIT_EN_MASK 0X00020000 /* option */
-#define USB_CTRL_SETUP_SS_EHCI64BIT_EN_VAR_MASK 0x00010000 /* option */
-#define USB_CTRL_SETUP_STRAP_IPP_SEL_MASK 0x02000000 /* option */
-#define USB_CTRL_SETUP_CC_DRD_MODE_ENABLE_MASK 0x04000000 /* option */
-#define USB_CTRL_SETUP_STRAP_CC_DRD_MODE_ENABLE_SEL_MASK 0x08000000 /* opt */
-#define USB_CTRL_SETUP_OC3_DISABLE_MASK 0xc0000000 /* option */
+#define USB_CTRL_SETUP_BABO_MASK BIT(0)
+#define USB_CTRL_SETUP_FNHW_MASK BIT(1)
+#define USB_CTRL_SETUP_FNBO_MASK BIT(2)
+#define USB_CTRL_SETUP_WABO_MASK BIT(3)
+#define USB_CTRL_SETUP_IOC_MASK BIT(4)
+#define USB_CTRL_SETUP_IPP_MASK BIT(5)
+#define USB_CTRL_SETUP_SCB_CLIENT_SWAP_MASK BIT(13) /* option */
+#define USB_CTRL_SETUP_SCB1_EN_MASK BIT(14) /* option */
+#define USB_CTRL_SETUP_SCB2_EN_MASK BIT(15) /* option */
+#define USB_CTRL_SETUP_SS_EHCI64BIT_EN_MASK BIT(17) /* option */
+#define USB_CTRL_SETUP_SS_EHCI64BIT_EN_VAR_MASK BIT(16) /* option */
+#define USB_CTRL_SETUP_STRAP_IPP_SEL_MASK BIT(25) /* option */
+#define USB_CTRL_SETUP_CC_DRD_MODE_ENABLE_MASK BIT(26) /* option */
+#define USB_CTRL_SETUP_STRAP_CC_DRD_MODE_ENABLE_SEL_MASK BIT(27) /* opt */
+#define USB_CTRL_SETUP_OC3_DISABLE_MASK GENMASK(31, 30) /* option */
#define USB_CTRL_PLL_CTL 0x04
-#define USB_CTRL_PLL_CTL_PLL_SUSPEND_EN_MASK 0x08000000
-#define USB_CTRL_PLL_CTL_PLL_RESETB_MASK 0x40000000
-#define USB_CTRL_PLL_CTL_PLL_IDDQ_PWRDN_MASK 0x80000000 /* option */
+#define USB_CTRL_PLL_CTL_PLL_SUSPEND_EN_MASK BIT(27)
+#define USB_CTRL_PLL_CTL_PLL_RESETB_MASK BIT(30)
+#define USB_CTRL_PLL_CTL_PLL_IDDQ_PWRDN_MASK BIT(31) /* option */
#define USB_CTRL_EBRIDGE 0x0c
-#define USB_CTRL_EBRIDGE_ESTOP_SCB_REQ_MASK 0x00020000 /* option */
-#define USB_CTRL_EBRIDGE_EBR_SCB_SIZE_MASK 0x00000f80 /* option */
+#define USB_CTRL_EBRIDGE_EBR_SCB_SIZE_MASK GENMASK(11, 7) /* option */
+#define USB_CTRL_EBRIDGE_ESTOP_SCB_REQ_MASK BIT(17) /* option */
#define USB_CTRL_OBRIDGE 0x10
-#define USB_CTRL_OBRIDGE_LS_KEEP_ALIVE_MASK 0x08000000
+#define USB_CTRL_OBRIDGE_LS_KEEP_ALIVE_MASK BIT(27)
#define USB_CTRL_MDIO 0x14
#define USB_CTRL_MDIO2 0x18
#define USB_CTRL_UTMI_CTL_1 0x2c
-#define USB_CTRL_UTMI_CTL_1_POWER_UP_FSM_EN_MASK 0x00000800
-#define USB_CTRL_UTMI_CTL_1_POWER_UP_FSM_EN_P1_MASK 0x08000000
+#define USB_CTRL_UTMI_CTL_1_POWER_UP_FSM_EN_MASK BIT(11)
+#define USB_CTRL_UTMI_CTL_1_POWER_UP_FSM_EN_P1_MASK BIT(27)
#define USB_CTRL_USB_PM 0x34
-#define USB_CTRL_USB_PM_BDC_SOFT_RESETB_MASK 0x00800000 /* option */
-#define USB_CTRL_USB_PM_XHC_SOFT_RESETB_MASK 0x00400000 /* option */
-#define USB_CTRL_USB_PM_XHC_SOFT_RESETB_VAR_MASK 0x40000000 /* option */
-#define USB_CTRL_USB_PM_USB_PWRDN_MASK 0x80000000 /* option */
-#define USB_CTRL_USB_PM_SOFT_RESET_MASK 0x40000000 /* option */
-#define USB_CTRL_USB_PM_USB20_HC_RESETB_MASK 0x30000000 /* option */
-#define USB_CTRL_USB_PM_USB20_HC_RESETB_VAR_MASK 0x00300000 /* option */
-#define USB_CTRL_USB_PM_RMTWKUP_EN_MASK 0x00000001
+#define USB_CTRL_USB_PM_RMTWKUP_EN_MASK BIT(0)
+#define USB_CTRL_USB_PM_USB20_HC_RESETB_VAR_MASK GENMASK(21, 20) /* option */
+#define USB_CTRL_USB_PM_XHC_SOFT_RESETB_MASK BIT(22) /* option */
+#define USB_CTRL_USB_PM_BDC_SOFT_RESETB_MASK BIT(23) /* option */
+#define USB_CTRL_USB_PM_USB20_HC_RESETB_MASK GENMASK(29, 28) /* option */
+#define USB_CTRL_USB_PM_XHC_SOFT_RESETB_VAR_MASK BIT(30) /* option */
+#define USB_CTRL_USB_PM_SOFT_RESET_MASK BIT(30) /* option */
+#define USB_CTRL_USB_PM_USB_PWRDN_MASK BIT(31) /* option */
#define USB_CTRL_USB_PM_STATUS 0x38
#define USB_CTRL_USB30_CTL1 0x60
-#define USB_CTRL_USB30_CTL1_PHY3_PLL_SEQ_START_MASK 0x00000010
-#define USB_CTRL_USB30_CTL1_PHY3_RESETB_MASK 0x00010000
-#define USB_CTRL_USB30_CTL1_XHC_SOFT_RESETB_MASK 0x00020000 /* option */
-#define USB_CTRL_USB30_CTL1_USB3_IOC_MASK 0x10000000 /* option */
-#define USB_CTRL_USB30_CTL1_USB3_IPP_MASK 0x20000000 /* option */
+#define USB_CTRL_USB30_CTL1_PHY3_PLL_SEQ_START_MASK BIT(4)
+#define USB_CTRL_USB30_CTL1_PHY3_RESETB_MASK BIT(16)
+#define USB_CTRL_USB30_CTL1_XHC_SOFT_RESETB_MASK BIT(17) /* option */
+#define USB_CTRL_USB30_CTL1_USB3_IOC_MASK BIT(28) /* option */
+#define USB_CTRL_USB30_CTL1_USB3_IPP_MASK BIT(29) /* option */
#define USB_CTRL_USB30_PCTL 0x70
-#define USB_CTRL_USB30_PCTL_PHY3_SOFT_RESETB_MASK 0x00000002
-#define USB_CTRL_USB30_PCTL_PHY3_IDDQ_OVERRIDE_MASK 0x00008000
-#define USB_CTRL_USB30_PCTL_PHY3_SOFT_RESETB_P1_MASK 0x00020000
+#define USB_CTRL_USB30_PCTL_PHY3_SOFT_RESETB_MASK BIT(1)
+#define USB_CTRL_USB30_PCTL_PHY3_IDDQ_OVERRIDE_MASK BIT(15)
+#define USB_CTRL_USB30_PCTL_PHY3_SOFT_RESETB_P1_MASK BIT(17)
#define USB_CTRL_USB_DEVICE_CTL1 0x90
-#define USB_CTRL_USB_DEVICE_CTL1_PORT_MODE_MASK 0x00000003 /* option */
+#define USB_CTRL_USB_DEVICE_CTL1_PORT_MODE_MASK GENMASK(1, 0) /* option */
/* Register definitions for the XHCI EC block */
#define USB_XHCI_EC_IRAADR 0x658
@@ -876,11 +876,11 @@ static void usb_init_common(struct brcm_usb_init_params *params)
reg = brcm_usb_readl(USB_CTRL_REG(ctrl, USB_DEVICE_CTL1));
reg &= ~USB_CTRL_MASK_FAMILY(params, USB_DEVICE_CTL1,
PORT_MODE);
- reg |= params->mode;
+ reg |= params->port_mode;
brcm_usb_writel(reg, USB_CTRL_REG(ctrl, USB_DEVICE_CTL1));
}
if (USB_CTRL_MASK_FAMILY(params, USB_PM, BDC_SOFT_RESETB)) {
- switch (params->mode) {
+ switch (params->supported_port_modes) {
case USB_CTLR_MODE_HOST:
USB_CTRL_UNSET_FAMILY(params, USB_PM, BDC_SOFT_RESETB);
break;
@@ -891,7 +891,7 @@ static void usb_init_common(struct brcm_usb_init_params *params)
}
}
if (USB_CTRL_MASK_FAMILY(params, SETUP, CC_DRD_MODE_ENABLE)) {
- if (params->mode == USB_CTLR_MODE_TYPEC_PD)
+ if (params->supported_port_modes == USB_CTLR_MODE_TYPEC_PD)
USB_CTRL_SET_FAMILY(params, SETUP, CC_DRD_MODE_ENABLE);
else
USB_CTRL_UNSET_FAMILY(params, SETUP,
@@ -1000,7 +1000,7 @@ static int usb_get_dual_select(struct brcm_usb_init_params *params)
return reg;
}
-static void usb_set_dual_select(struct brcm_usb_init_params *params, int mode)
+static void usb_set_dual_select(struct brcm_usb_init_params *params)
{
void __iomem *ctrl = params->regs[BRCM_REGS_CTRL];
u32 reg;
@@ -1011,7 +1011,7 @@ static void usb_set_dual_select(struct brcm_usb_init_params *params, int mode)
reg = brcm_usb_readl(USB_CTRL_REG(ctrl, USB_DEVICE_CTL1));
reg &= ~USB_CTRL_MASK_FAMILY(params, USB_DEVICE_CTL1,
PORT_MODE);
- reg |= mode;
+ reg |= params->port_mode;
brcm_usb_writel(reg, USB_CTRL_REG(ctrl, USB_DEVICE_CTL1));
}
}
diff --git a/drivers/phy/broadcom/phy-brcm-usb-init.h b/drivers/phy/broadcom/phy-brcm-usb-init.h
index 1ccb5dd..f9fbf8f 100644
--- a/drivers/phy/broadcom/phy-brcm-usb-init.h
+++ b/drivers/phy/broadcom/phy-brcm-usb-init.h
@@ -45,14 +45,15 @@ struct brcm_usb_init_ops {
void (*uninit_eohci)(struct brcm_usb_init_params *params);
void (*uninit_xhci)(struct brcm_usb_init_params *params);
int (*get_dual_select)(struct brcm_usb_init_params *params);
- void (*set_dual_select)(struct brcm_usb_init_params *params, int mode);
+ void (*set_dual_select)(struct brcm_usb_init_params *params);
};
struct brcm_usb_init_params {
void __iomem *regs[BRCM_REGS_MAX];
int ioc;
int ipp;
- int mode;
+ int supported_port_modes;
+ int port_mode;
u32 family_id;
u32 product_id;
int selected_family;
@@ -61,7 +62,6 @@ struct brcm_usb_init_params {
const struct brcm_usb_init_ops *ops;
struct regmap *syscon_piarbctl;
bool wake_enabled;
- bool suspend_with_clocks;
};
void brcm_usb_dvr_init_4908(struct brcm_usb_init_params *params);
@@ -153,11 +153,10 @@ static inline int brcm_usb_get_dual_select(struct brcm_usb_init_params *ini)
return 0;
}
-static inline void brcm_usb_set_dual_select(struct brcm_usb_init_params *ini,
- int mode)
+static inline void brcm_usb_set_dual_select(struct brcm_usb_init_params *ini)
{
if (ini->ops->set_dual_select)
- ini->ops->set_dual_select(ini, mode);
+ ini->ops->set_dual_select(ini);
}
#endif /* _USB_BRCM_COMMON_INIT_H */
diff --git a/drivers/phy/broadcom/phy-brcm-usb.c b/drivers/phy/broadcom/phy-brcm-usb.c
index 2cb3779..4de3999 100644
--- a/drivers/phy/broadcom/phy-brcm-usb.c
+++ b/drivers/phy/broadcom/phy-brcm-usb.c
@@ -102,9 +102,9 @@ static int brcm_pm_notifier(struct notifier_block *notifier,
static irqreturn_t brcm_usb_phy_wake_isr(int irq, void *dev_id)
{
- struct phy *gphy = dev_id;
+ struct device *dev = dev_id;
- pm_wakeup_event(&gphy->dev, 0);
+ pm_wakeup_event(dev, 0);
return IRQ_HANDLED;
}
@@ -233,7 +233,7 @@ static ssize_t dr_mode_show(struct device *dev,
return sprintf(buf, "%s\n",
value_to_name(&brcm_dr_mode_to_name[0],
ARRAY_SIZE(brcm_dr_mode_to_name),
- priv->ini.mode));
+ priv->ini.supported_port_modes));
}
static DEVICE_ATTR_RO(dr_mode);
@@ -249,7 +249,8 @@ static ssize_t dual_select_store(struct device *dev,
res = name_to_value(&brcm_dual_mode_to_name[0],
ARRAY_SIZE(brcm_dual_mode_to_name), buf, &value);
if (!res) {
- brcm_usb_set_dual_select(&priv->ini, value);
+ priv->ini.port_mode = value;
+ brcm_usb_set_dual_select(&priv->ini);
res = len;
}
mutex_unlock(&sysfs_lock);
@@ -445,13 +446,13 @@ static int brcm_usb_phy_dvr_init(struct platform_device *pdev,
priv->suspend_clk = NULL;
}
- priv->wake_irq = platform_get_irq_byname(pdev, "wake");
+ priv->wake_irq = platform_get_irq_byname_optional(pdev, "wake");
if (priv->wake_irq < 0)
- priv->wake_irq = platform_get_irq_byname(pdev, "wakeup");
+ priv->wake_irq = platform_get_irq_byname_optional(pdev, "wakeup");
if (priv->wake_irq >= 0) {
err = devm_request_irq(dev, priv->wake_irq,
brcm_usb_phy_wake_isr, 0,
- dev_name(dev), gphy);
+ dev_name(dev), dev);
if (err < 0)
return err;
device_set_wakeup_capable(dev, 1);
@@ -495,13 +496,16 @@ static int brcm_usb_phy_probe(struct platform_device *pdev)
of_property_read_u32(dn, "brcm,ipp", &priv->ini.ipp);
of_property_read_u32(dn, "brcm,ioc", &priv->ini.ioc);
- priv->ini.mode = USB_CTLR_MODE_HOST;
+ priv->ini.supported_port_modes = USB_CTLR_MODE_HOST;
err = of_property_read_string(dn, "dr_mode", &mode);
if (err == 0) {
name_to_value(&brcm_dr_mode_to_name[0],
ARRAY_SIZE(brcm_dr_mode_to_name),
- mode, &priv->ini.mode);
+ mode, &priv->ini.supported_port_modes);
}
+ /* Default port_mode to supported port_modes */
+ priv->ini.port_mode = priv->ini.supported_port_modes;
+
if (of_property_read_bool(dn, "brcm,has-xhci"))
priv->has_xhci = true;
if (of_property_read_bool(dn, "brcm,has-eohci"))
@@ -539,7 +543,7 @@ static int brcm_usb_phy_probe(struct platform_device *pdev)
* Create sysfs entries for mode.
* Remove "dual_select" attribute if not in dual mode
*/
- if (priv->ini.mode != USB_CTLR_MODE_DRD)
+ if (priv->ini.supported_port_modes != USB_CTLR_MODE_DRD)
brcm_usb_phy_attrs[1] = NULL;
err = sysfs_create_group(&dev->kobj, &brcm_usb_phy_group);
if (err)
@@ -598,7 +602,7 @@ static int brcm_usb_phy_suspend(struct device *dev)
* and newer XHCI->2.0-clks/3.0-clks.
*/
- if (!priv->ini.suspend_with_clocks) {
+ if (!priv->ini.wake_enabled) {
if (priv->phys[BRCM_USB_PHY_3_0].inited)
clk_disable_unprepare(priv->usb_30_clk);
if (priv->phys[BRCM_USB_PHY_2_0].inited ||
@@ -615,8 +619,10 @@ static int brcm_usb_phy_resume(struct device *dev)
{
struct brcm_usb_phy_data *priv = dev_get_drvdata(dev);
- clk_prepare_enable(priv->usb_20_clk);
- clk_prepare_enable(priv->usb_30_clk);
+ if (!priv->ini.wake_enabled) {
+ clk_prepare_enable(priv->usb_20_clk);
+ clk_prepare_enable(priv->usb_30_clk);
+ }
brcm_usb_init_ipp(&priv->ini);
/*
diff --git a/drivers/phy/freescale/phy-fsl-imx8m-pcie.c b/drivers/phy/freescale/phy-fsl-imx8m-pcie.c
index c932864..7585e80 100644
--- a/drivers/phy/freescale/phy-fsl-imx8m-pcie.c
+++ b/drivers/phy/freescale/phy-fsl-imx8m-pcie.c
@@ -11,6 +11,7 @@
#include <linux/mfd/syscon.h>
#include <linux/mfd/syscon/imx7-iomuxc-gpr.h>
#include <linux/module.h>
+#include <linux/of_device.h>
#include <linux/phy/phy.h>
#include <linux/platform_device.h>
#include <linux/regmap.h>
@@ -31,12 +32,10 @@
#define IMX8MM_PCIE_PHY_CMN_REG065 0x194
#define ANA_AUX_RX_TERM (BIT(7) | BIT(4))
#define ANA_AUX_TX_LVL GENMASK(3, 0)
-#define IMX8MM_PCIE_PHY_CMN_REG75 0x1D4
-#define PCIE_PHY_CMN_REG75_PLL_DONE 0x3
+#define IMX8MM_PCIE_PHY_CMN_REG075 0x1D4
+#define ANA_PLL_DONE 0x3
#define PCIE_PHY_TRSV_REG5 0x414
-#define PCIE_PHY_TRSV_REG5_GEN1_DEEMP 0x2D
#define PCIE_PHY_TRSV_REG6 0x418
-#define PCIE_PHY_TRSV_REG6_GEN2_DEEMP 0xF
#define IMX8MM_GPR_PCIE_REF_CLK_SEL GENMASK(25, 24)
#define IMX8MM_GPR_PCIE_REF_CLK_PLL FIELD_PREP(IMX8MM_GPR_PCIE_REF_CLK_SEL, 0x3)
@@ -47,16 +46,28 @@
#define IMX8MM_GPR_PCIE_SSC_EN BIT(16)
#define IMX8MM_GPR_PCIE_AUX_EN_OVERRIDE BIT(9)
+enum imx8_pcie_phy_type {
+ IMX8MM,
+ IMX8MP,
+};
+
+struct imx8_pcie_phy_drvdata {
+ const char *gpr;
+ enum imx8_pcie_phy_type variant;
+};
+
struct imx8_pcie_phy {
void __iomem *base;
struct clk *clk;
struct phy *phy;
struct regmap *iomuxc_gpr;
+ struct reset_control *perst;
struct reset_control *reset;
u32 refclk_pad_mode;
u32 tx_deemph_gen1;
u32 tx_deemph_gen2;
bool clkreq_unused;
+ const struct imx8_pcie_phy_drvdata *drvdata;
};
static int imx8_pcie_phy_power_on(struct phy *phy)
@@ -65,34 +76,22 @@ static int imx8_pcie_phy_power_on(struct phy *phy)
u32 val, pad_mode;
struct imx8_pcie_phy *imx8_phy = phy_get_drvdata(phy);
- reset_control_assert(imx8_phy->reset);
-
pad_mode = imx8_phy->refclk_pad_mode;
- /* Set AUX_EN_OVERRIDE 1'b0, when the CLKREQ# isn't hooked */
- regmap_update_bits(imx8_phy->iomuxc_gpr, IOMUXC_GPR14,
- IMX8MM_GPR_PCIE_AUX_EN_OVERRIDE,
- imx8_phy->clkreq_unused ?
- 0 : IMX8MM_GPR_PCIE_AUX_EN_OVERRIDE);
- regmap_update_bits(imx8_phy->iomuxc_gpr, IOMUXC_GPR14,
- IMX8MM_GPR_PCIE_AUX_EN,
- IMX8MM_GPR_PCIE_AUX_EN);
- regmap_update_bits(imx8_phy->iomuxc_gpr, IOMUXC_GPR14,
- IMX8MM_GPR_PCIE_POWER_OFF, 0);
- regmap_update_bits(imx8_phy->iomuxc_gpr, IOMUXC_GPR14,
- IMX8MM_GPR_PCIE_SSC_EN, 0);
+ switch (imx8_phy->drvdata->variant) {
+ case IMX8MM:
+ reset_control_assert(imx8_phy->reset);
- regmap_update_bits(imx8_phy->iomuxc_gpr, IOMUXC_GPR14,
- IMX8MM_GPR_PCIE_REF_CLK_SEL,
- pad_mode == IMX8_PCIE_REFCLK_PAD_INPUT ?
- IMX8MM_GPR_PCIE_REF_CLK_EXT :
- IMX8MM_GPR_PCIE_REF_CLK_PLL);
- usleep_range(100, 200);
-
- /* Do the PHY common block reset */
- regmap_update_bits(imx8_phy->iomuxc_gpr, IOMUXC_GPR14,
- IMX8MM_GPR_PCIE_CMN_RST,
- IMX8MM_GPR_PCIE_CMN_RST);
- usleep_range(200, 500);
+ /* Tune PHY de-emphasis setting to pass PCIe compliance. */
+ if (imx8_phy->tx_deemph_gen1)
+ writel(imx8_phy->tx_deemph_gen1,
+ imx8_phy->base + PCIE_PHY_TRSV_REG5);
+ if (imx8_phy->tx_deemph_gen2)
+ writel(imx8_phy->tx_deemph_gen2,
+ imx8_phy->base + PCIE_PHY_TRSV_REG6);
+ break;
+ case IMX8MP: /* Do nothing. */
+ break;
+ }
if (pad_mode == IMX8_PCIE_REFCLK_PAD_INPUT ||
pad_mode == IMX8_PCIE_REFCLK_PAD_UNUSED) {
@@ -120,20 +119,44 @@ static int imx8_pcie_phy_power_on(struct phy *phy)
imx8_phy->base + IMX8MM_PCIE_PHY_CMN_REG065);
}
- /* Tune PHY de-emphasis setting to pass PCIe compliance. */
- if (imx8_phy->tx_deemph_gen1)
- writel(imx8_phy->tx_deemph_gen1,
- imx8_phy->base + PCIE_PHY_TRSV_REG5);
- if (imx8_phy->tx_deemph_gen2)
- writel(imx8_phy->tx_deemph_gen2,
- imx8_phy->base + PCIE_PHY_TRSV_REG6);
+ /* Set AUX_EN_OVERRIDE 1'b0, when the CLKREQ# isn't hooked */
+ regmap_update_bits(imx8_phy->iomuxc_gpr, IOMUXC_GPR14,
+ IMX8MM_GPR_PCIE_AUX_EN_OVERRIDE,
+ imx8_phy->clkreq_unused ?
+ 0 : IMX8MM_GPR_PCIE_AUX_EN_OVERRIDE);
+ regmap_update_bits(imx8_phy->iomuxc_gpr, IOMUXC_GPR14,
+ IMX8MM_GPR_PCIE_AUX_EN,
+ IMX8MM_GPR_PCIE_AUX_EN);
+ regmap_update_bits(imx8_phy->iomuxc_gpr, IOMUXC_GPR14,
+ IMX8MM_GPR_PCIE_POWER_OFF, 0);
+ regmap_update_bits(imx8_phy->iomuxc_gpr, IOMUXC_GPR14,
+ IMX8MM_GPR_PCIE_SSC_EN, 0);
- reset_control_deassert(imx8_phy->reset);
+ regmap_update_bits(imx8_phy->iomuxc_gpr, IOMUXC_GPR14,
+ IMX8MM_GPR_PCIE_REF_CLK_SEL,
+ pad_mode == IMX8_PCIE_REFCLK_PAD_INPUT ?
+ IMX8MM_GPR_PCIE_REF_CLK_EXT :
+ IMX8MM_GPR_PCIE_REF_CLK_PLL);
+ usleep_range(100, 200);
+
+ /* Do the PHY common block reset */
+ regmap_update_bits(imx8_phy->iomuxc_gpr, IOMUXC_GPR14,
+ IMX8MM_GPR_PCIE_CMN_RST,
+ IMX8MM_GPR_PCIE_CMN_RST);
+
+ switch (imx8_phy->drvdata->variant) {
+ case IMX8MP:
+ reset_control_deassert(imx8_phy->perst);
+ fallthrough;
+ case IMX8MM:
+ reset_control_deassert(imx8_phy->reset);
+ usleep_range(200, 500);
+ break;
+ }
/* Polling to check the phy is ready or not. */
- ret = readl_poll_timeout(imx8_phy->base + IMX8MM_PCIE_PHY_CMN_REG75,
- val, val == PCIE_PHY_CMN_REG75_PLL_DONE,
- 10, 20000);
+ ret = readl_poll_timeout(imx8_phy->base + IMX8MM_PCIE_PHY_CMN_REG075,
+ val, val == ANA_PLL_DONE, 10, 20000);
return ret;
}
@@ -160,6 +183,23 @@ static const struct phy_ops imx8_pcie_phy_ops = {
.owner = THIS_MODULE,
};
+static const struct imx8_pcie_phy_drvdata imx8mm_drvdata = {
+ .gpr = "fsl,imx8mm-iomuxc-gpr",
+ .variant = IMX8MM,
+};
+
+static const struct imx8_pcie_phy_drvdata imx8mp_drvdata = {
+ .gpr = "fsl,imx8mp-iomuxc-gpr",
+ .variant = IMX8MP,
+};
+
+static const struct of_device_id imx8_pcie_phy_of_match[] = {
+ {.compatible = "fsl,imx8mm-pcie-phy", .data = &imx8mm_drvdata, },
+ {.compatible = "fsl,imx8mp-pcie-phy", .data = &imx8mp_drvdata, },
+ { },
+};
+MODULE_DEVICE_TABLE(of, imx8_pcie_phy_of_match);
+
static int imx8_pcie_phy_probe(struct platform_device *pdev)
{
struct phy_provider *phy_provider;
@@ -172,6 +212,8 @@ static int imx8_pcie_phy_probe(struct platform_device *pdev)
if (!imx8_phy)
return -ENOMEM;
+ imx8_phy->drvdata = of_device_get_match_data(dev);
+
/* get PHY refclk pad mode */
of_property_read_u32(np, "fsl,refclk-pad-mode",
&imx8_phy->refclk_pad_mode);
@@ -197,7 +239,7 @@ static int imx8_pcie_phy_probe(struct platform_device *pdev)
/* Grab GPR config register range */
imx8_phy->iomuxc_gpr =
- syscon_regmap_lookup_by_compatible("fsl,imx6q-iomuxc-gpr");
+ syscon_regmap_lookup_by_compatible(imx8_phy->drvdata->gpr);
if (IS_ERR(imx8_phy->iomuxc_gpr)) {
dev_err(dev, "unable to find iomuxc registers\n");
return PTR_ERR(imx8_phy->iomuxc_gpr);
@@ -209,6 +251,14 @@ static int imx8_pcie_phy_probe(struct platform_device *pdev)
return PTR_ERR(imx8_phy->reset);
}
+ if (imx8_phy->drvdata->variant == IMX8MP) {
+ imx8_phy->perst =
+ devm_reset_control_get_exclusive(dev, "perst");
+ if (IS_ERR(imx8_phy->perst))
+ dev_err_probe(dev, PTR_ERR(imx8_phy->perst),
+ "Failed to get PCIE PHY PERST control\n");
+ }
+
res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
imx8_phy->base = devm_ioremap_resource(dev, res);
if (IS_ERR(imx8_phy->base))
@@ -225,12 +275,6 @@ static int imx8_pcie_phy_probe(struct platform_device *pdev)
return PTR_ERR_OR_ZERO(phy_provider);
}
-static const struct of_device_id imx8_pcie_phy_of_match[] = {
- {.compatible = "fsl,imx8mm-pcie-phy",},
- { },
-};
-MODULE_DEVICE_TABLE(of, imx8_pcie_phy_of_match);
-
static struct platform_driver imx8_pcie_phy_driver = {
.probe = imx8_pcie_phy_probe,
.driver = {
diff --git a/drivers/phy/marvell/phy-mmp3-hsic.c b/drivers/phy/marvell/phy-mmp3-hsic.c
index 7cccf01..f2537fd 100644
--- a/drivers/phy/marvell/phy-mmp3-hsic.c
+++ b/drivers/phy/marvell/phy-mmp3-hsic.c
@@ -41,12 +41,10 @@ static int mmp3_hsic_phy_probe(struct platform_device *pdev)
{
struct device *dev = &pdev->dev;
struct phy_provider *provider;
- struct resource *resource;
void __iomem *base;
struct phy *phy;
- resource = platform_get_resource(pdev, IORESOURCE_MEM, 0);
- base = devm_ioremap_resource(dev, resource);
+ base = devm_platform_get_and_ioremap_resource(pdev, 0, NULL);
if (IS_ERR(base))
return PTR_ERR(base);
diff --git a/drivers/phy/marvell/phy-mvebu-a3700-comphy.c b/drivers/phy/marvell/phy-mvebu-a3700-comphy.c
index 67712c7..d641b34 100644
--- a/drivers/phy/marvell/phy-mvebu-a3700-comphy.c
+++ b/drivers/phy/marvell/phy-mvebu-a3700-comphy.c
@@ -826,6 +826,9 @@ mvebu_a3700_comphy_usb3_power_on(struct mvebu_a3700_comphy_lane *lane)
if (ret)
return ret;
+ /* COMPHY register reset (cleared automatically) */
+ comphy_lane_reg_set(lane, COMPHY_SFT_RESET, SFT_RST, SFT_RST);
+
/*
* 0. Set PHY OTG Control(0x5d034), bit 4, Power up OTG module The
* register belong to UTMI module, so it is set in UTMI phy driver.
diff --git a/drivers/phy/qualcomm/Kconfig b/drivers/phy/qualcomm/Kconfig
index 5c98850..eb9ddc6 100644
--- a/drivers/phy/qualcomm/Kconfig
+++ b/drivers/phy/qualcomm/Kconfig
@@ -54,6 +54,7 @@
tristate "Qualcomm QMP PHY Driver"
depends on OF && COMMON_CLK && (ARCH_QCOM || COMPILE_TEST)
select GENERIC_PHY
+ select MFD_SYSCON
help
Enable this to support the QMP PHY transceiver that is used
with controllers such as PCIe, UFS, and USB on Qualcomm chips.
diff --git a/drivers/phy/qualcomm/phy-qcom-qmp-combo.c b/drivers/phy/qualcomm/phy-qcom-qmp-combo.c
index ba9d761..77052c6 100644
--- a/drivers/phy/qualcomm/phy-qcom-qmp-combo.c
+++ b/drivers/phy/qualcomm/phy-qcom-qmp-combo.c
@@ -20,7 +20,7 @@
#include <linux/reset.h>
#include <linux/slab.h>
-#include <dt-bindings/phy/phy.h>
+#include <dt-bindings/phy/phy-qcom-qmp.h>
#include "phy-qcom-qmp.h"
@@ -63,18 +63,11 @@
#define CLAMP_EN BIT(0) /* enables i/o clamp_n */
#define PHY_INIT_COMPLETE_TIMEOUT 10000
-#define POWER_DOWN_DELAY_US_MIN 10
-#define POWER_DOWN_DELAY_US_MAX 11
struct qmp_phy_init_tbl {
unsigned int offset;
unsigned int val;
/*
- * register part of layout ?
- * if yes, then offset gives index in the reg-layout
- */
- bool in_layout;
- /*
* mask of lanes for which this register is written
* for cases when second lane needs different values
*/
@@ -88,14 +81,6 @@ struct qmp_phy_init_tbl {
.lane_mask = 0xff, \
}
-#define QMP_PHY_INIT_CFG_L(o, v) \
- { \
- .offset = o, \
- .val = v, \
- .in_layout = true, \
- .lane_mask = 0xff, \
- }
-
#define QMP_PHY_INIT_CFG_LANE(o, v, l) \
{ \
.offset = o, \
@@ -121,6 +106,7 @@ static const unsigned int qmp_v3_usb3phy_regs_layout[QPHY_LAYOUT_SIZE] = {
[QPHY_SW_RESET] = 0x00,
[QPHY_START_CTRL] = 0x08,
[QPHY_PCS_STATUS] = 0x174,
+ [QPHY_PCS_POWER_DOWN_CONTROL] = 0x04,
[QPHY_PCS_AUTONOMOUS_MODE_CTRL] = 0x0d8,
[QPHY_PCS_LFPS_RXTERM_IRQ_CLEAR] = 0x0dc,
[QPHY_PCS_LFPS_RXTERM_IRQ_STATUS] = 0x170,
@@ -810,13 +796,24 @@ static const u8 qmp_dp_v5_voltage_swing_hbr_rbr[4][4] = {
{ 0x3f, 0xff, 0xff, 0xff }
};
-struct qmp_phy;
+struct qmp_combo;
-/* struct qmp_phy_cfg - per-PHY initialization config */
+struct qmp_combo_offsets {
+ u16 com;
+ u16 txa;
+ u16 rxa;
+ u16 txb;
+ u16 rxb;
+ u16 usb3_serdes;
+ u16 usb3_pcs_misc;
+ u16 usb3_pcs;
+ u16 usb3_pcs_usb;
+ u16 dp_serdes;
+ u16 dp_dp_phy;
+};
+
struct qmp_phy_cfg {
- /* phy-type - PCIE/UFS/USB */
- unsigned int type;
- int lanes;
+ const struct qmp_combo_offsets *offsets;
/* Init sequence for PHY blocks - serdes, tx, rx, pcs */
const struct qmp_phy_init_tbl *serdes_tbl;
@@ -830,6 +827,11 @@ struct qmp_phy_cfg {
const struct qmp_phy_init_tbl *pcs_usb_tbl;
int pcs_usb_tbl_num;
+ const struct qmp_phy_init_tbl *dp_serdes_tbl;
+ int dp_serdes_tbl_num;
+ const struct qmp_phy_init_tbl *dp_tx_tbl;
+ int dp_tx_tbl_num;
+
/* Init sequence for DP PHY block link rates */
const struct qmp_phy_init_tbl *serdes_tbl_rbr;
int serdes_tbl_rbr_num;
@@ -847,10 +849,10 @@ struct qmp_phy_cfg {
const u8 (*pre_emphasis_hbr3_hbr2)[4][4];
/* DP PHY callbacks */
- int (*configure_dp_phy)(struct qmp_phy *qphy);
- void (*configure_dp_tx)(struct qmp_phy *qphy);
- int (*calibrate_dp_phy)(struct qmp_phy *qphy);
- void (*dp_aux_init)(struct qmp_phy *qphy);
+ int (*configure_dp_phy)(struct qmp_combo *qmp);
+ void (*configure_dp_tx)(struct qmp_combo *qmp);
+ int (*calibrate_dp_phy)(struct qmp_combo *qmp);
+ void (*dp_aux_init)(struct qmp_combo *qmp);
/* clock ids to be requested */
const char * const *clk_list;
@@ -865,50 +867,21 @@ struct qmp_phy_cfg {
/* array of registers with different offsets */
const unsigned int *regs;
- unsigned int start_ctrl;
- unsigned int pwrdn_ctrl;
- /* bit offset of PHYSTATUS in QPHY_PCS_STATUS register */
- unsigned int phy_status;
-
/* true, if PHY needs delay after POWER_DOWN */
bool has_pwrdn_delay;
- /* power_down delay in usec */
- int pwrdn_delay_min;
- int pwrdn_delay_max;
/* Offset from PCS to PCS_USB region */
unsigned int pcs_usb_offset;
};
-struct qmp_phy_combo_cfg {
- const struct qmp_phy_cfg *usb_cfg;
- const struct qmp_phy_cfg *dp_cfg;
-};
+struct qmp_combo {
+ struct device *dev;
-/**
- * struct qmp_phy - per-lane phy descriptor
- *
- * @phy: generic phy
- * @cfg: phy specific configuration
- * @serdes: iomapped memory space for phy's serdes (i.e. PLL)
- * @tx: iomapped memory space for lane's tx
- * @rx: iomapped memory space for lane's rx
- * @pcs: iomapped memory space for lane's pcs
- * @tx2: iomapped memory space for second lane's tx (in dual lane PHYs)
- * @rx2: iomapped memory space for second lane's rx (in dual lane PHYs)
- * @pcs_misc: iomapped memory space for lane's pcs_misc
- * @pcs_usb: iomapped memory space for lane's pcs_usb
- * @pipe_clk: pipe clock
- * @qmp: QMP phy to which this lane belongs
- * @mode: current PHY mode
- * @dp_aux_cfg: Display port aux config
- * @dp_opts: Display port optional config
- * @dp_clks: Display port clocks
- */
-struct qmp_phy {
- struct phy *phy;
const struct qmp_phy_cfg *cfg;
+
+ void __iomem *com;
+
void __iomem *serdes;
void __iomem *tx;
void __iomem *rx;
@@ -917,62 +890,43 @@ struct qmp_phy {
void __iomem *rx2;
void __iomem *pcs_misc;
void __iomem *pcs_usb;
+
+ void __iomem *dp_serdes;
+ void __iomem *dp_tx;
+ void __iomem *dp_tx2;
+ void __iomem *dp_dp_phy;
+
struct clk *pipe_clk;
- struct qcom_qmp *qmp;
- enum phy_mode mode;
- unsigned int dp_aux_cfg;
- struct phy_configure_opts_dp dp_opts;
- struct qmp_phy_dp_clks *dp_clks;
-};
-
-struct qmp_phy_dp_clks {
- struct qmp_phy *qphy;
- struct clk_hw dp_link_hw;
- struct clk_hw dp_pixel_hw;
-};
-
-/**
- * struct qcom_qmp - structure holding QMP phy block attributes
- *
- * @dev: device
- * @dp_com: iomapped memory space for phy's dp_com control block
- *
- * @clks: array of clocks required by phy
- * @resets: array of resets required by phy
- * @vregs: regulator supplies bulk data
- *
- * @phys: array of per-lane phy descriptors
- * @phy_mutex: mutex lock for PHY common block initialization
- * @init_count: phy common block initialization count
- * @ufs_reset: optional UFS PHY reset handle
- */
-struct qcom_qmp {
- struct device *dev;
- void __iomem *dp_com;
-
struct clk_bulk_data *clks;
struct reset_control_bulk_data *resets;
struct regulator_bulk_data *vregs;
- struct qmp_phy **phys;
-
struct mutex phy_mutex;
int init_count;
- struct reset_control *ufs_reset;
+ struct phy *usb_phy;
+ enum phy_mode mode;
+
+ struct phy *dp_phy;
+ unsigned int dp_aux_cfg;
+ struct phy_configure_opts_dp dp_opts;
+
+ struct clk_fixed_rate pipe_clk_fixed;
+ struct clk_hw dp_link_hw;
+ struct clk_hw dp_pixel_hw;
};
-static void qcom_qmp_v3_phy_dp_aux_init(struct qmp_phy *qphy);
-static void qcom_qmp_v3_phy_configure_dp_tx(struct qmp_phy *qphy);
-static int qcom_qmp_v3_phy_configure_dp_phy(struct qmp_phy *qphy);
-static int qcom_qmp_v3_dp_phy_calibrate(struct qmp_phy *qphy);
+static void qmp_v3_dp_aux_init(struct qmp_combo *qmp);
+static void qmp_v3_configure_dp_tx(struct qmp_combo *qmp);
+static int qmp_v3_configure_dp_phy(struct qmp_combo *qmp);
+static int qmp_v3_calibrate_dp_phy(struct qmp_combo *qmp);
-static void qcom_qmp_v4_phy_dp_aux_init(struct qmp_phy *qphy);
-static void qcom_qmp_v4_phy_configure_dp_tx(struct qmp_phy *qphy);
-static int qcom_qmp_v4_phy_configure_dp_phy(struct qmp_phy *qphy);
-static int qcom_qmp_v4_dp_phy_calibrate(struct qmp_phy *qphy);
+static void qmp_v4_dp_aux_init(struct qmp_combo *qmp);
+static void qmp_v4_configure_dp_tx(struct qmp_combo *qmp);
+static int qmp_v4_configure_dp_phy(struct qmp_combo *qmp);
+static int qmp_v4_calibrate_dp_phy(struct qmp_combo *qmp);
-static int qcom_qmp_v5_phy_configure_dp_phy(struct qmp_phy *qphy);
+static int qmp_v5_configure_dp_phy(struct qmp_combo *qmp);
static inline void qphy_setbits(void __iomem *base, u32 offset, u32 val)
{
@@ -1004,7 +958,7 @@ static const char * const qmp_v3_phy_clk_l[] = {
};
static const char * const qmp_v4_phy_clk_l[] = {
- "aux", "ref_clk_src", "ref", "com_aux",
+ "aux", "ref", "com_aux",
};
/* the primary usb3 phy on sm8250 doesn't have a ref clock */
@@ -1021,10 +975,21 @@ static const char * const sc7180_usb3phy_reset_l[] = {
"phy",
};
-static const struct qmp_phy_cfg sc7180_usb3phy_cfg = {
- .type = PHY_TYPE_USB3,
- .lanes = 2,
+static const struct qmp_combo_offsets qmp_combo_offsets_v5 = {
+ .com = 0x0000,
+ .txa = 0x0400,
+ .rxa = 0x0600,
+ .txb = 0x0a00,
+ .rxb = 0x0c00,
+ .usb3_serdes = 0x1000,
+ .usb3_pcs_misc = 0x1200,
+ .usb3_pcs = 0x1400,
+ .usb3_pcs_usb = 0x1700,
+ .dp_serdes = 0x2000,
+ .dp_dp_phy = 0x2200,
+};
+static const struct qmp_phy_cfg sc7180_usb3dpphy_cfg = {
.serdes_tbl = qmp_v3_usb3_serdes_tbl,
.serdes_tbl_num = ARRAY_SIZE(qmp_v3_usb3_serdes_tbl),
.tx_tbl = qmp_v3_usb3_tx_tbl,
@@ -1033,31 +998,11 @@ static const struct qmp_phy_cfg sc7180_usb3phy_cfg = {
.rx_tbl_num = ARRAY_SIZE(qmp_v3_usb3_rx_tbl),
.pcs_tbl = qmp_v3_usb3_pcs_tbl,
.pcs_tbl_num = ARRAY_SIZE(qmp_v3_usb3_pcs_tbl),
- .clk_list = qmp_v3_phy_clk_l,
- .num_clks = ARRAY_SIZE(qmp_v3_phy_clk_l),
- .reset_list = sc7180_usb3phy_reset_l,
- .num_resets = ARRAY_SIZE(sc7180_usb3phy_reset_l),
- .vreg_list = qmp_phy_vreg_l,
- .num_vregs = ARRAY_SIZE(qmp_phy_vreg_l),
- .regs = qmp_v3_usb3phy_regs_layout,
- .start_ctrl = SERDES_START | PCS_START,
- .pwrdn_ctrl = SW_PWRDN,
- .phy_status = PHYSTATUS,
-
- .has_pwrdn_delay = true,
- .pwrdn_delay_min = POWER_DOWN_DELAY_US_MIN,
- .pwrdn_delay_max = POWER_DOWN_DELAY_US_MAX,
-};
-
-static const struct qmp_phy_cfg sc7180_dpphy_cfg = {
- .type = PHY_TYPE_DP,
- .lanes = 2,
-
- .serdes_tbl = qmp_v3_dp_serdes_tbl,
- .serdes_tbl_num = ARRAY_SIZE(qmp_v3_dp_serdes_tbl),
- .tx_tbl = qmp_v3_dp_tx_tbl,
- .tx_tbl_num = ARRAY_SIZE(qmp_v3_dp_tx_tbl),
+ .dp_serdes_tbl = qmp_v3_dp_serdes_tbl,
+ .dp_serdes_tbl_num = ARRAY_SIZE(qmp_v3_dp_serdes_tbl),
+ .dp_tx_tbl = qmp_v3_dp_tx_tbl,
+ .dp_tx_tbl_num = ARRAY_SIZE(qmp_v3_dp_tx_tbl),
.serdes_tbl_rbr = qmp_v3_dp_serdes_tbl_rbr,
.serdes_tbl_rbr_num = ARRAY_SIZE(qmp_v3_dp_serdes_tbl_rbr),
@@ -1073,6 +1018,11 @@ static const struct qmp_phy_cfg sc7180_dpphy_cfg = {
.swing_hbr3_hbr2 = &qmp_dp_v3_voltage_swing_hbr3_hbr2,
.pre_emphasis_hbr3_hbr2 = &qmp_dp_v3_pre_emphasis_hbr3_hbr2,
+ .dp_aux_init = qmp_v3_dp_aux_init,
+ .configure_dp_tx = qmp_v3_configure_dp_tx,
+ .configure_dp_phy = qmp_v3_configure_dp_phy,
+ .calibrate_dp_phy = qmp_v3_calibrate_dp_phy,
+
.clk_list = qmp_v3_phy_clk_l,
.num_clks = ARRAY_SIZE(qmp_v3_phy_clk_l),
.reset_list = sc7180_usb3phy_reset_l,
@@ -1081,21 +1031,10 @@ static const struct qmp_phy_cfg sc7180_dpphy_cfg = {
.num_vregs = ARRAY_SIZE(qmp_phy_vreg_l),
.regs = qmp_v3_usb3phy_regs_layout,
- .dp_aux_init = qcom_qmp_v3_phy_dp_aux_init,
- .configure_dp_tx = qcom_qmp_v3_phy_configure_dp_tx,
- .configure_dp_phy = qcom_qmp_v3_phy_configure_dp_phy,
- .calibrate_dp_phy = qcom_qmp_v3_dp_phy_calibrate,
+ .has_pwrdn_delay = true,
};
-static const struct qmp_phy_combo_cfg sc7180_usb3dpphy_cfg = {
- .usb_cfg = &sc7180_usb3phy_cfg,
- .dp_cfg = &sc7180_dpphy_cfg,
-};
-
-static const struct qmp_phy_cfg sdm845_usb3phy_cfg = {
- .type = PHY_TYPE_USB3,
- .lanes = 2,
-
+static const struct qmp_phy_cfg sdm845_usb3dpphy_cfg = {
.serdes_tbl = qmp_v3_usb3_serdes_tbl,
.serdes_tbl_num = ARRAY_SIZE(qmp_v3_usb3_serdes_tbl),
.tx_tbl = qmp_v3_usb3_tx_tbl,
@@ -1104,6 +1043,31 @@ static const struct qmp_phy_cfg sdm845_usb3phy_cfg = {
.rx_tbl_num = ARRAY_SIZE(qmp_v3_usb3_rx_tbl),
.pcs_tbl = qmp_v3_usb3_pcs_tbl,
.pcs_tbl_num = ARRAY_SIZE(qmp_v3_usb3_pcs_tbl),
+
+ .dp_serdes_tbl = qmp_v3_dp_serdes_tbl,
+ .dp_serdes_tbl_num = ARRAY_SIZE(qmp_v3_dp_serdes_tbl),
+ .dp_tx_tbl = qmp_v3_dp_tx_tbl,
+ .dp_tx_tbl_num = ARRAY_SIZE(qmp_v3_dp_tx_tbl),
+
+ .serdes_tbl_rbr = qmp_v3_dp_serdes_tbl_rbr,
+ .serdes_tbl_rbr_num = ARRAY_SIZE(qmp_v3_dp_serdes_tbl_rbr),
+ .serdes_tbl_hbr = qmp_v3_dp_serdes_tbl_hbr,
+ .serdes_tbl_hbr_num = ARRAY_SIZE(qmp_v3_dp_serdes_tbl_hbr),
+ .serdes_tbl_hbr2 = qmp_v3_dp_serdes_tbl_hbr2,
+ .serdes_tbl_hbr2_num = ARRAY_SIZE(qmp_v3_dp_serdes_tbl_hbr2),
+ .serdes_tbl_hbr3 = qmp_v3_dp_serdes_tbl_hbr3,
+ .serdes_tbl_hbr3_num = ARRAY_SIZE(qmp_v3_dp_serdes_tbl_hbr3),
+
+ .swing_hbr_rbr = &qmp_dp_v3_voltage_swing_hbr_rbr,
+ .pre_emphasis_hbr_rbr = &qmp_dp_v3_pre_emphasis_hbr_rbr,
+ .swing_hbr3_hbr2 = &qmp_dp_v3_voltage_swing_hbr3_hbr2,
+ .pre_emphasis_hbr3_hbr2 = &qmp_dp_v3_pre_emphasis_hbr3_hbr2,
+
+ .dp_aux_init = qmp_v3_dp_aux_init,
+ .configure_dp_tx = qmp_v3_configure_dp_tx,
+ .configure_dp_phy = qmp_v3_configure_dp_phy,
+ .calibrate_dp_phy = qmp_v3_calibrate_dp_phy,
+
.clk_list = qmp_v3_phy_clk_l,
.num_clks = ARRAY_SIZE(qmp_v3_phy_clk_l),
.reset_list = msm8996_usb3phy_reset_l,
@@ -1112,24 +1076,10 @@ static const struct qmp_phy_cfg sdm845_usb3phy_cfg = {
.num_vregs = ARRAY_SIZE(qmp_phy_vreg_l),
.regs = qmp_v3_usb3phy_regs_layout,
- .start_ctrl = SERDES_START | PCS_START,
- .pwrdn_ctrl = SW_PWRDN,
- .phy_status = PHYSTATUS,
-
.has_pwrdn_delay = true,
- .pwrdn_delay_min = POWER_DOWN_DELAY_US_MIN,
- .pwrdn_delay_max = POWER_DOWN_DELAY_US_MAX,
};
-static const struct qmp_phy_combo_cfg sdm845_usb3dpphy_cfg = {
- .usb_cfg = &sdm845_usb3phy_cfg,
- .dp_cfg = &sc7180_dpphy_cfg,
-};
-
-static const struct qmp_phy_cfg sm8150_usb3phy_cfg = {
- .type = PHY_TYPE_USB3,
- .lanes = 2,
-
+static const struct qmp_phy_cfg sc8180x_usb3dpphy_cfg = {
.serdes_tbl = sm8150_usb3_serdes_tbl,
.serdes_tbl_num = ARRAY_SIZE(sm8150_usb3_serdes_tbl),
.tx_tbl = sm8150_usb3_tx_tbl,
@@ -1140,33 +1090,11 @@ static const struct qmp_phy_cfg sm8150_usb3phy_cfg = {
.pcs_tbl_num = ARRAY_SIZE(sm8150_usb3_pcs_tbl),
.pcs_usb_tbl = sm8150_usb3_pcs_usb_tbl,
.pcs_usb_tbl_num = ARRAY_SIZE(sm8150_usb3_pcs_usb_tbl),
- .clk_list = qmp_v4_phy_clk_l,
- .num_clks = ARRAY_SIZE(qmp_v4_phy_clk_l),
- .reset_list = msm8996_usb3phy_reset_l,
- .num_resets = ARRAY_SIZE(msm8996_usb3phy_reset_l),
- .vreg_list = qmp_phy_vreg_l,
- .num_vregs = ARRAY_SIZE(qmp_phy_vreg_l),
- .regs = qmp_v4_usb3phy_regs_layout,
- .pcs_usb_offset = 0x300,
- .start_ctrl = SERDES_START | PCS_START,
- .pwrdn_ctrl = SW_PWRDN,
- .phy_status = PHYSTATUS,
-
-
- .has_pwrdn_delay = true,
- .pwrdn_delay_min = POWER_DOWN_DELAY_US_MIN,
- .pwrdn_delay_max = POWER_DOWN_DELAY_US_MAX,
-};
-
-static const struct qmp_phy_cfg sc8180x_dpphy_cfg = {
- .type = PHY_TYPE_DP,
- .lanes = 2,
-
- .serdes_tbl = qmp_v4_dp_serdes_tbl,
- .serdes_tbl_num = ARRAY_SIZE(qmp_v4_dp_serdes_tbl),
- .tx_tbl = qmp_v4_dp_tx_tbl,
- .tx_tbl_num = ARRAY_SIZE(qmp_v4_dp_tx_tbl),
+ .dp_serdes_tbl = qmp_v4_dp_serdes_tbl,
+ .dp_serdes_tbl_num = ARRAY_SIZE(qmp_v4_dp_serdes_tbl),
+ .dp_tx_tbl = qmp_v4_dp_tx_tbl,
+ .dp_tx_tbl_num = ARRAY_SIZE(qmp_v4_dp_tx_tbl),
.serdes_tbl_rbr = qmp_v4_dp_serdes_tbl_rbr,
.serdes_tbl_rbr_num = ARRAY_SIZE(qmp_v4_dp_serdes_tbl_rbr),
@@ -1182,37 +1110,11 @@ static const struct qmp_phy_cfg sc8180x_dpphy_cfg = {
.swing_hbr3_hbr2 = &qmp_dp_v3_voltage_swing_hbr3_hbr2,
.pre_emphasis_hbr3_hbr2 = &qmp_dp_v3_pre_emphasis_hbr3_hbr2,
- .clk_list = qmp_v3_phy_clk_l,
- .num_clks = ARRAY_SIZE(qmp_v3_phy_clk_l),
- .reset_list = sc7180_usb3phy_reset_l,
- .num_resets = ARRAY_SIZE(sc7180_usb3phy_reset_l),
- .vreg_list = qmp_phy_vreg_l,
- .num_vregs = ARRAY_SIZE(qmp_phy_vreg_l),
- .regs = qmp_v3_usb3phy_regs_layout,
+ .dp_aux_init = qmp_v4_dp_aux_init,
+ .configure_dp_tx = qmp_v4_configure_dp_tx,
+ .configure_dp_phy = qmp_v4_configure_dp_phy,
+ .calibrate_dp_phy = qmp_v4_calibrate_dp_phy,
- .dp_aux_init = qcom_qmp_v4_phy_dp_aux_init,
- .configure_dp_tx = qcom_qmp_v4_phy_configure_dp_tx,
- .configure_dp_phy = qcom_qmp_v4_phy_configure_dp_phy,
- .calibrate_dp_phy = qcom_qmp_v4_dp_phy_calibrate,
-};
-
-static const struct qmp_phy_combo_cfg sc8180x_usb3dpphy_cfg = {
- .usb_cfg = &sm8150_usb3phy_cfg,
- .dp_cfg = &sc8180x_dpphy_cfg,
-};
-
-static const struct qmp_phy_cfg sc8280xp_usb43dp_usb_cfg = {
- .type = PHY_TYPE_USB3,
- .lanes = 2,
-
- .serdes_tbl = sc8280xp_usb43dp_serdes_tbl,
- .serdes_tbl_num = ARRAY_SIZE(sc8280xp_usb43dp_serdes_tbl),
- .tx_tbl = sc8280xp_usb43dp_tx_tbl,
- .tx_tbl_num = ARRAY_SIZE(sc8280xp_usb43dp_tx_tbl),
- .rx_tbl = sc8280xp_usb43dp_rx_tbl,
- .rx_tbl_num = ARRAY_SIZE(sc8280xp_usb43dp_rx_tbl),
- .pcs_tbl = sc8280xp_usb43dp_pcs_tbl,
- .pcs_tbl_num = ARRAY_SIZE(sc8280xp_usb43dp_pcs_tbl),
.clk_list = qmp_v4_phy_clk_l,
.num_clks = ARRAY_SIZE(qmp_v4_phy_clk_l),
.reset_list = msm8996_usb3phy_reset_l,
@@ -1222,23 +1124,25 @@ static const struct qmp_phy_cfg sc8280xp_usb43dp_usb_cfg = {
.regs = qmp_v4_usb3phy_regs_layout,
.pcs_usb_offset = 0x300,
- .start_ctrl = SERDES_START | PCS_START,
- .pwrdn_ctrl = SW_PWRDN,
- .phy_status = PHYSTATUS,
-
.has_pwrdn_delay = true,
- .pwrdn_delay_min = POWER_DOWN_DELAY_US_MIN,
- .pwrdn_delay_max = POWER_DOWN_DELAY_US_MAX,
};
-static const struct qmp_phy_cfg sc8280xp_usb43dp_dp_cfg = {
- .type = PHY_TYPE_DP,
- .lanes = 2,
+static const struct qmp_phy_cfg sc8280xp_usb43dpphy_cfg = {
+ .offsets = &qmp_combo_offsets_v5,
- .serdes_tbl = qmp_v5_dp_serdes_tbl,
- .serdes_tbl_num = ARRAY_SIZE(qmp_v5_dp_serdes_tbl),
- .tx_tbl = qmp_v5_5nm_dp_tx_tbl,
- .tx_tbl_num = ARRAY_SIZE(qmp_v5_5nm_dp_tx_tbl),
+ .serdes_tbl = sc8280xp_usb43dp_serdes_tbl,
+ .serdes_tbl_num = ARRAY_SIZE(sc8280xp_usb43dp_serdes_tbl),
+ .tx_tbl = sc8280xp_usb43dp_tx_tbl,
+ .tx_tbl_num = ARRAY_SIZE(sc8280xp_usb43dp_tx_tbl),
+ .rx_tbl = sc8280xp_usb43dp_rx_tbl,
+ .rx_tbl_num = ARRAY_SIZE(sc8280xp_usb43dp_rx_tbl),
+ .pcs_tbl = sc8280xp_usb43dp_pcs_tbl,
+ .pcs_tbl_num = ARRAY_SIZE(sc8280xp_usb43dp_pcs_tbl),
+
+ .dp_serdes_tbl = qmp_v5_dp_serdes_tbl,
+ .dp_serdes_tbl_num = ARRAY_SIZE(qmp_v5_dp_serdes_tbl),
+ .dp_tx_tbl = qmp_v5_5nm_dp_tx_tbl,
+ .dp_tx_tbl_num = ARRAY_SIZE(qmp_v5_5nm_dp_tx_tbl),
.serdes_tbl_rbr = qmp_v4_dp_serdes_tbl_rbr,
.serdes_tbl_rbr_num = ARRAY_SIZE(qmp_v4_dp_serdes_tbl_rbr),
@@ -1254,6 +1158,11 @@ static const struct qmp_phy_cfg sc8280xp_usb43dp_dp_cfg = {
.swing_hbr3_hbr2 = &qmp_dp_v5_voltage_swing_hbr3_hbr2,
.pre_emphasis_hbr3_hbr2 = &qmp_dp_v5_pre_emphasis_hbr3_hbr2,
+ .dp_aux_init = qmp_v4_dp_aux_init,
+ .configure_dp_tx = qmp_v4_configure_dp_tx,
+ .configure_dp_phy = qmp_v5_configure_dp_phy,
+ .calibrate_dp_phy = qmp_v4_calibrate_dp_phy,
+
.clk_list = qmp_v4_phy_clk_l,
.num_clks = ARRAY_SIZE(qmp_v4_phy_clk_l),
.reset_list = msm8996_usb3phy_reset_l,
@@ -1261,22 +1170,9 @@ static const struct qmp_phy_cfg sc8280xp_usb43dp_dp_cfg = {
.vreg_list = qmp_phy_vreg_l,
.num_vregs = ARRAY_SIZE(qmp_phy_vreg_l),
.regs = qmp_v4_usb3phy_regs_layout,
-
- .dp_aux_init = qcom_qmp_v4_phy_dp_aux_init,
- .configure_dp_tx = qcom_qmp_v4_phy_configure_dp_tx,
- .configure_dp_phy = qcom_qmp_v5_phy_configure_dp_phy,
- .calibrate_dp_phy = qcom_qmp_v4_dp_phy_calibrate,
};
-static const struct qmp_phy_combo_cfg sc8280xp_usb43dpphy_combo_cfg = {
- .usb_cfg = &sc8280xp_usb43dp_usb_cfg,
- .dp_cfg = &sc8280xp_usb43dp_dp_cfg,
-};
-
-static const struct qmp_phy_cfg sm8250_usb3phy_cfg = {
- .type = PHY_TYPE_USB3,
- .lanes = 2,
-
+static const struct qmp_phy_cfg sm8250_usb3dpphy_cfg = {
.serdes_tbl = sm8150_usb3_serdes_tbl,
.serdes_tbl_num = ARRAY_SIZE(sm8150_usb3_serdes_tbl),
.tx_tbl = sm8250_usb3_tx_tbl,
@@ -1287,32 +1183,11 @@ static const struct qmp_phy_cfg sm8250_usb3phy_cfg = {
.pcs_tbl_num = ARRAY_SIZE(sm8250_usb3_pcs_tbl),
.pcs_usb_tbl = sm8250_usb3_pcs_usb_tbl,
.pcs_usb_tbl_num = ARRAY_SIZE(sm8250_usb3_pcs_usb_tbl),
- .clk_list = qmp_v4_sm8250_usbphy_clk_l,
- .num_clks = ARRAY_SIZE(qmp_v4_sm8250_usbphy_clk_l),
- .reset_list = msm8996_usb3phy_reset_l,
- .num_resets = ARRAY_SIZE(msm8996_usb3phy_reset_l),
- .vreg_list = qmp_phy_vreg_l,
- .num_vregs = ARRAY_SIZE(qmp_phy_vreg_l),
- .regs = qmp_v4_usb3phy_regs_layout,
- .pcs_usb_offset = 0x300,
- .start_ctrl = SERDES_START | PCS_START,
- .pwrdn_ctrl = SW_PWRDN,
- .phy_status = PHYSTATUS,
-
- .has_pwrdn_delay = true,
- .pwrdn_delay_min = POWER_DOWN_DELAY_US_MIN,
- .pwrdn_delay_max = POWER_DOWN_DELAY_US_MAX,
-};
-
-static const struct qmp_phy_cfg sm8250_dpphy_cfg = {
- .type = PHY_TYPE_DP,
- .lanes = 2,
-
- .serdes_tbl = qmp_v4_dp_serdes_tbl,
- .serdes_tbl_num = ARRAY_SIZE(qmp_v4_dp_serdes_tbl),
- .tx_tbl = qmp_v4_dp_tx_tbl,
- .tx_tbl_num = ARRAY_SIZE(qmp_v4_dp_tx_tbl),
+ .dp_serdes_tbl = qmp_v4_dp_serdes_tbl,
+ .dp_serdes_tbl_num = ARRAY_SIZE(qmp_v4_dp_serdes_tbl),
+ .dp_tx_tbl = qmp_v4_dp_tx_tbl,
+ .dp_tx_tbl_num = ARRAY_SIZE(qmp_v4_dp_tx_tbl),
.serdes_tbl_rbr = qmp_v4_dp_serdes_tbl_rbr,
.serdes_tbl_rbr_num = ARRAY_SIZE(qmp_v4_dp_serdes_tbl_rbr),
@@ -1328,27 +1203,24 @@ static const struct qmp_phy_cfg sm8250_dpphy_cfg = {
.swing_hbr3_hbr2 = &qmp_dp_v3_voltage_swing_hbr3_hbr2,
.pre_emphasis_hbr3_hbr2 = &qmp_dp_v3_pre_emphasis_hbr3_hbr2,
- .clk_list = qmp_v4_phy_clk_l,
- .num_clks = ARRAY_SIZE(qmp_v4_phy_clk_l),
+ .dp_aux_init = qmp_v4_dp_aux_init,
+ .configure_dp_tx = qmp_v4_configure_dp_tx,
+ .configure_dp_phy = qmp_v4_configure_dp_phy,
+ .calibrate_dp_phy = qmp_v4_calibrate_dp_phy,
+
+ .clk_list = qmp_v4_sm8250_usbphy_clk_l,
+ .num_clks = ARRAY_SIZE(qmp_v4_sm8250_usbphy_clk_l),
.reset_list = msm8996_usb3phy_reset_l,
.num_resets = ARRAY_SIZE(msm8996_usb3phy_reset_l),
.vreg_list = qmp_phy_vreg_l,
.num_vregs = ARRAY_SIZE(qmp_phy_vreg_l),
.regs = qmp_v4_usb3phy_regs_layout,
+ .pcs_usb_offset = 0x300,
- .dp_aux_init = qcom_qmp_v4_phy_dp_aux_init,
- .configure_dp_tx = qcom_qmp_v4_phy_configure_dp_tx,
- .configure_dp_phy = qcom_qmp_v4_phy_configure_dp_phy,
- .calibrate_dp_phy = qcom_qmp_v4_dp_phy_calibrate,
-};
-
-static const struct qmp_phy_combo_cfg sm8250_usb3dpphy_cfg = {
- .usb_cfg = &sm8250_usb3phy_cfg,
- .dp_cfg = &sm8250_dpphy_cfg,
+ .has_pwrdn_delay = true,
};
static void qmp_combo_configure_lane(void __iomem *base,
- const unsigned int *regs,
const struct qmp_phy_init_tbl tbl[],
int num,
u8 lane_mask)
@@ -1363,110 +1235,98 @@ static void qmp_combo_configure_lane(void __iomem *base,
if (!(t->lane_mask & lane_mask))
continue;
- if (t->in_layout)
- writel(t->val, base + regs[t->offset]);
- else
- writel(t->val, base + t->offset);
+ writel(t->val, base + t->offset);
}
}
static void qmp_combo_configure(void __iomem *base,
- const unsigned int *regs,
const struct qmp_phy_init_tbl tbl[],
int num)
{
- qmp_combo_configure_lane(base, regs, tbl, num, 0xff);
+ qmp_combo_configure_lane(base, tbl, num, 0xff);
}
-static int qmp_combo_serdes_init(struct qmp_phy *qphy)
+static int qmp_combo_dp_serdes_init(struct qmp_combo *qmp)
{
- const struct qmp_phy_cfg *cfg = qphy->cfg;
- void __iomem *serdes = qphy->serdes;
- const struct phy_configure_opts_dp *dp_opts = &qphy->dp_opts;
- const struct qmp_phy_init_tbl *serdes_tbl = cfg->serdes_tbl;
- int serdes_tbl_num = cfg->serdes_tbl_num;
+ const struct qmp_phy_cfg *cfg = qmp->cfg;
+ void __iomem *serdes = qmp->dp_serdes;
+ const struct phy_configure_opts_dp *dp_opts = &qmp->dp_opts;
- qmp_combo_configure(serdes, cfg->regs, serdes_tbl, serdes_tbl_num);
+ qmp_combo_configure(serdes, cfg->dp_serdes_tbl, cfg->dp_serdes_tbl_num);
- if (cfg->type == PHY_TYPE_DP) {
- switch (dp_opts->link_rate) {
- case 1620:
- qmp_combo_configure(serdes, cfg->regs,
- cfg->serdes_tbl_rbr,
- cfg->serdes_tbl_rbr_num);
- break;
- case 2700:
- qmp_combo_configure(serdes, cfg->regs,
- cfg->serdes_tbl_hbr,
- cfg->serdes_tbl_hbr_num);
- break;
- case 5400:
- qmp_combo_configure(serdes, cfg->regs,
- cfg->serdes_tbl_hbr2,
- cfg->serdes_tbl_hbr2_num);
- break;
- case 8100:
- qmp_combo_configure(serdes, cfg->regs,
- cfg->serdes_tbl_hbr3,
- cfg->serdes_tbl_hbr3_num);
- break;
- default:
- /* Other link rates aren't supported */
- return -EINVAL;
- }
+ switch (dp_opts->link_rate) {
+ case 1620:
+ qmp_combo_configure(serdes, cfg->serdes_tbl_rbr,
+ cfg->serdes_tbl_rbr_num);
+ break;
+ case 2700:
+ qmp_combo_configure(serdes, cfg->serdes_tbl_hbr,
+ cfg->serdes_tbl_hbr_num);
+ break;
+ case 5400:
+ qmp_combo_configure(serdes, cfg->serdes_tbl_hbr2,
+ cfg->serdes_tbl_hbr2_num);
+ break;
+ case 8100:
+ qmp_combo_configure(serdes, cfg->serdes_tbl_hbr3,
+ cfg->serdes_tbl_hbr3_num);
+ break;
+ default:
+ /* Other link rates aren't supported */
+ return -EINVAL;
}
return 0;
}
-static void qcom_qmp_v3_phy_dp_aux_init(struct qmp_phy *qphy)
+static void qmp_v3_dp_aux_init(struct qmp_combo *qmp)
{
writel(DP_PHY_PD_CTL_PWRDN | DP_PHY_PD_CTL_AUX_PWRDN |
DP_PHY_PD_CTL_PLL_PWRDN | DP_PHY_PD_CTL_DP_CLAMP_EN,
- qphy->pcs + QSERDES_DP_PHY_PD_CTL);
+ qmp->dp_dp_phy + QSERDES_DP_PHY_PD_CTL);
/* Turn on BIAS current for PHY/PLL */
writel(QSERDES_V3_COM_BIAS_EN | QSERDES_V3_COM_BIAS_EN_MUX |
QSERDES_V3_COM_CLKBUF_L_EN | QSERDES_V3_COM_EN_SYSCLK_TX_SEL,
- qphy->serdes + QSERDES_V3_COM_BIAS_EN_CLKBUFLR_EN);
+ qmp->dp_serdes + QSERDES_V3_COM_BIAS_EN_CLKBUFLR_EN);
- writel(DP_PHY_PD_CTL_PSR_PWRDN, qphy->pcs + QSERDES_DP_PHY_PD_CTL);
+ writel(DP_PHY_PD_CTL_PSR_PWRDN, qmp->dp_dp_phy + QSERDES_DP_PHY_PD_CTL);
writel(DP_PHY_PD_CTL_PWRDN | DP_PHY_PD_CTL_AUX_PWRDN |
DP_PHY_PD_CTL_LANE_0_1_PWRDN |
DP_PHY_PD_CTL_LANE_2_3_PWRDN | DP_PHY_PD_CTL_PLL_PWRDN |
DP_PHY_PD_CTL_DP_CLAMP_EN,
- qphy->pcs + QSERDES_DP_PHY_PD_CTL);
+ qmp->dp_dp_phy + QSERDES_DP_PHY_PD_CTL);
writel(QSERDES_V3_COM_BIAS_EN |
QSERDES_V3_COM_BIAS_EN_MUX | QSERDES_V3_COM_CLKBUF_R_EN |
QSERDES_V3_COM_CLKBUF_L_EN | QSERDES_V3_COM_EN_SYSCLK_TX_SEL |
QSERDES_V3_COM_CLKBUF_RX_DRIVE_L,
- qphy->serdes + QSERDES_V3_COM_BIAS_EN_CLKBUFLR_EN);
+ qmp->dp_serdes + QSERDES_V3_COM_BIAS_EN_CLKBUFLR_EN);
- writel(0x00, qphy->pcs + QSERDES_DP_PHY_AUX_CFG0);
- writel(0x13, qphy->pcs + QSERDES_DP_PHY_AUX_CFG1);
- writel(0x24, qphy->pcs + QSERDES_DP_PHY_AUX_CFG2);
- writel(0x00, qphy->pcs + QSERDES_DP_PHY_AUX_CFG3);
- writel(0x0a, qphy->pcs + QSERDES_DP_PHY_AUX_CFG4);
- writel(0x26, qphy->pcs + QSERDES_DP_PHY_AUX_CFG5);
- writel(0x0a, qphy->pcs + QSERDES_DP_PHY_AUX_CFG6);
- writel(0x03, qphy->pcs + QSERDES_DP_PHY_AUX_CFG7);
- writel(0xbb, qphy->pcs + QSERDES_DP_PHY_AUX_CFG8);
- writel(0x03, qphy->pcs + QSERDES_DP_PHY_AUX_CFG9);
- qphy->dp_aux_cfg = 0;
+ writel(0x00, qmp->dp_dp_phy + QSERDES_DP_PHY_AUX_CFG0);
+ writel(0x13, qmp->dp_dp_phy + QSERDES_DP_PHY_AUX_CFG1);
+ writel(0x24, qmp->dp_dp_phy + QSERDES_DP_PHY_AUX_CFG2);
+ writel(0x00, qmp->dp_dp_phy + QSERDES_DP_PHY_AUX_CFG3);
+ writel(0x0a, qmp->dp_dp_phy + QSERDES_DP_PHY_AUX_CFG4);
+ writel(0x26, qmp->dp_dp_phy + QSERDES_DP_PHY_AUX_CFG5);
+ writel(0x0a, qmp->dp_dp_phy + QSERDES_DP_PHY_AUX_CFG6);
+ writel(0x03, qmp->dp_dp_phy + QSERDES_DP_PHY_AUX_CFG7);
+ writel(0xbb, qmp->dp_dp_phy + QSERDES_DP_PHY_AUX_CFG8);
+ writel(0x03, qmp->dp_dp_phy + QSERDES_DP_PHY_AUX_CFG9);
+ qmp->dp_aux_cfg = 0;
writel(PHY_AUX_STOP_ERR_MASK | PHY_AUX_DEC_ERR_MASK |
PHY_AUX_SYNC_ERR_MASK | PHY_AUX_ALIGN_ERR_MASK |
PHY_AUX_REQ_ERR_MASK,
- qphy->pcs + QSERDES_V3_DP_PHY_AUX_INTERRUPT_MASK);
+ qmp->dp_dp_phy + QSERDES_V3_DP_PHY_AUX_INTERRUPT_MASK);
}
-static int qmp_combo_configure_dp_swing(struct qmp_phy *qphy,
+static int qmp_combo_configure_dp_swing(struct qmp_combo *qmp,
unsigned int drv_lvl_reg, unsigned int emp_post_reg)
{
- const struct phy_configure_opts_dp *dp_opts = &qphy->dp_opts;
- const struct qmp_phy_cfg *cfg = qphy->cfg;
+ const struct phy_configure_opts_dp *dp_opts = &qmp->dp_opts;
+ const struct qmp_phy_cfg *cfg = qmp->cfg;
unsigned int v_level = 0, p_level = 0;
u8 voltage_swing_cfg, pre_emphasis_cfg;
int i;
@@ -1492,20 +1352,20 @@ static int qmp_combo_configure_dp_swing(struct qmp_phy *qphy,
voltage_swing_cfg |= DP_PHY_TXn_TX_DRV_LVL_MUX_EN;
pre_emphasis_cfg |= DP_PHY_TXn_TX_EMP_POST1_LVL_MUX_EN;
- writel(voltage_swing_cfg, qphy->tx + drv_lvl_reg);
- writel(pre_emphasis_cfg, qphy->tx + emp_post_reg);
- writel(voltage_swing_cfg, qphy->tx2 + drv_lvl_reg);
- writel(pre_emphasis_cfg, qphy->tx2 + emp_post_reg);
+ writel(voltage_swing_cfg, qmp->dp_tx + drv_lvl_reg);
+ writel(pre_emphasis_cfg, qmp->dp_tx + emp_post_reg);
+ writel(voltage_swing_cfg, qmp->dp_tx2 + drv_lvl_reg);
+ writel(pre_emphasis_cfg, qmp->dp_tx2 + emp_post_reg);
return 0;
}
-static void qcom_qmp_v3_phy_configure_dp_tx(struct qmp_phy *qphy)
+static void qmp_v3_configure_dp_tx(struct qmp_combo *qmp)
{
- const struct phy_configure_opts_dp *dp_opts = &qphy->dp_opts;
+ const struct phy_configure_opts_dp *dp_opts = &qmp->dp_opts;
u32 bias_en, drvr_en;
- if (qmp_combo_configure_dp_swing(qphy, QSERDES_V3_TX_TX_DRV_LVL,
+ if (qmp_combo_configure_dp_swing(qmp, QSERDES_V3_TX_TX_DRV_LVL,
QSERDES_V3_TX_TX_EMP_POST1_LVL) < 0)
return;
@@ -1517,13 +1377,13 @@ static void qcom_qmp_v3_phy_configure_dp_tx(struct qmp_phy *qphy)
drvr_en = 0x10;
}
- writel(drvr_en, qphy->tx + QSERDES_V3_TX_HIGHZ_DRVR_EN);
- writel(bias_en, qphy->tx + QSERDES_V3_TX_TRANSCEIVER_BIAS_EN);
- writel(drvr_en, qphy->tx2 + QSERDES_V3_TX_HIGHZ_DRVR_EN);
- writel(bias_en, qphy->tx2 + QSERDES_V3_TX_TRANSCEIVER_BIAS_EN);
+ writel(drvr_en, qmp->dp_tx + QSERDES_V3_TX_HIGHZ_DRVR_EN);
+ writel(bias_en, qmp->dp_tx + QSERDES_V3_TX_TRANSCEIVER_BIAS_EN);
+ writel(drvr_en, qmp->dp_tx2 + QSERDES_V3_TX_HIGHZ_DRVR_EN);
+ writel(bias_en, qmp->dp_tx2 + QSERDES_V3_TX_TRANSCEIVER_BIAS_EN);
}
-static bool qmp_combo_configure_dp_mode(struct qmp_phy *qphy)
+static bool qmp_combo_configure_dp_mode(struct qmp_combo *qmp)
{
u32 val;
bool reverse = false;
@@ -1543,27 +1403,26 @@ static bool qmp_combo_configure_dp_mode(struct qmp_phy *qphy)
* if (lane_cnt == 4 || orientation == ORIENTATION_CC1)
* val |= DP_PHY_PD_CTL_LANE_2_3_PWRDN;
* if (orientation == ORIENTATION_CC2)
- * writel(0x4c, qphy->pcs + QSERDES_V3_DP_PHY_MODE);
+ * writel(0x4c, qmp->dp_dp_phy + QSERDES_V3_DP_PHY_MODE);
*/
val |= DP_PHY_PD_CTL_LANE_2_3_PWRDN;
- writel(val, qphy->pcs + QSERDES_DP_PHY_PD_CTL);
+ writel(val, qmp->dp_dp_phy + QSERDES_DP_PHY_PD_CTL);
- writel(0x5c, qphy->pcs + QSERDES_DP_PHY_MODE);
+ writel(0x5c, qmp->dp_dp_phy + QSERDES_DP_PHY_MODE);
return reverse;
}
-static int qcom_qmp_v3_phy_configure_dp_phy(struct qmp_phy *qphy)
+static int qmp_v3_configure_dp_phy(struct qmp_combo *qmp)
{
- const struct qmp_phy_dp_clks *dp_clks = qphy->dp_clks;
- const struct phy_configure_opts_dp *dp_opts = &qphy->dp_opts;
+ const struct phy_configure_opts_dp *dp_opts = &qmp->dp_opts;
u32 phy_vco_div, status;
unsigned long pixel_freq;
- qmp_combo_configure_dp_mode(qphy);
+ qmp_combo_configure_dp_mode(qmp);
- writel(0x05, qphy->pcs + QSERDES_V3_DP_PHY_TX0_TX1_LANE_CTL);
- writel(0x05, qphy->pcs + QSERDES_V3_DP_PHY_TX2_TX3_LANE_CTL);
+ writel(0x05, qmp->dp_dp_phy + QSERDES_V3_DP_PHY_TX0_TX1_LANE_CTL);
+ writel(0x05, qmp->dp_dp_phy + QSERDES_V3_DP_PHY_TX2_TX3_LANE_CTL);
switch (dp_opts->link_rate) {
case 1620:
@@ -1586,40 +1445,40 @@ static int qcom_qmp_v3_phy_configure_dp_phy(struct qmp_phy *qphy)
/* Other link rates aren't supported */
return -EINVAL;
}
- writel(phy_vco_div, qphy->pcs + QSERDES_V3_DP_PHY_VCO_DIV);
+ writel(phy_vco_div, qmp->dp_dp_phy + QSERDES_V3_DP_PHY_VCO_DIV);
- clk_set_rate(dp_clks->dp_link_hw.clk, dp_opts->link_rate * 100000);
- clk_set_rate(dp_clks->dp_pixel_hw.clk, pixel_freq);
+ clk_set_rate(qmp->dp_link_hw.clk, dp_opts->link_rate * 100000);
+ clk_set_rate(qmp->dp_pixel_hw.clk, pixel_freq);
- writel(0x04, qphy->pcs + QSERDES_DP_PHY_AUX_CFG2);
- writel(0x01, qphy->pcs + QSERDES_DP_PHY_CFG);
- writel(0x05, qphy->pcs + QSERDES_DP_PHY_CFG);
- writel(0x01, qphy->pcs + QSERDES_DP_PHY_CFG);
- writel(0x09, qphy->pcs + QSERDES_DP_PHY_CFG);
+ writel(0x04, qmp->dp_dp_phy + QSERDES_DP_PHY_AUX_CFG2);
+ writel(0x01, qmp->dp_dp_phy + QSERDES_DP_PHY_CFG);
+ writel(0x05, qmp->dp_dp_phy + QSERDES_DP_PHY_CFG);
+ writel(0x01, qmp->dp_dp_phy + QSERDES_DP_PHY_CFG);
+ writel(0x09, qmp->dp_dp_phy + QSERDES_DP_PHY_CFG);
- writel(0x20, qphy->serdes + QSERDES_V3_COM_RESETSM_CNTRL);
+ writel(0x20, qmp->dp_serdes + QSERDES_V3_COM_RESETSM_CNTRL);
- if (readl_poll_timeout(qphy->serdes + QSERDES_V3_COM_C_READY_STATUS,
+ if (readl_poll_timeout(qmp->dp_serdes + QSERDES_V3_COM_C_READY_STATUS,
status,
((status & BIT(0)) > 0),
500,
10000))
return -ETIMEDOUT;
- writel(0x19, qphy->pcs + QSERDES_DP_PHY_CFG);
+ writel(0x19, qmp->dp_dp_phy + QSERDES_DP_PHY_CFG);
- if (readl_poll_timeout(qphy->pcs + QSERDES_V3_DP_PHY_STATUS,
+ if (readl_poll_timeout(qmp->dp_dp_phy + QSERDES_V3_DP_PHY_STATUS,
status,
((status & BIT(1)) > 0),
500,
10000))
return -ETIMEDOUT;
- writel(0x18, qphy->pcs + QSERDES_DP_PHY_CFG);
+ writel(0x18, qmp->dp_dp_phy + QSERDES_DP_PHY_CFG);
udelay(2000);
- writel(0x19, qphy->pcs + QSERDES_DP_PHY_CFG);
+ writel(0x19, qmp->dp_dp_phy + QSERDES_DP_PHY_CFG);
- return readl_poll_timeout(qphy->pcs + QSERDES_V3_DP_PHY_STATUS,
+ return readl_poll_timeout(qmp->dp_dp_phy + QSERDES_V3_DP_PHY_STATUS,
status,
((status & BIT(1)) > 0),
500,
@@ -1630,76 +1489,75 @@ static int qcom_qmp_v3_phy_configure_dp_phy(struct qmp_phy *qphy)
* We need to calibrate the aux setting here as many times
* as the caller tries
*/
-static int qcom_qmp_v3_dp_phy_calibrate(struct qmp_phy *qphy)
+static int qmp_v3_calibrate_dp_phy(struct qmp_combo *qmp)
{
static const u8 cfg1_settings[] = { 0x13, 0x23, 0x1d };
u8 val;
- qphy->dp_aux_cfg++;
- qphy->dp_aux_cfg %= ARRAY_SIZE(cfg1_settings);
- val = cfg1_settings[qphy->dp_aux_cfg];
+ qmp->dp_aux_cfg++;
+ qmp->dp_aux_cfg %= ARRAY_SIZE(cfg1_settings);
+ val = cfg1_settings[qmp->dp_aux_cfg];
- writel(val, qphy->pcs + QSERDES_DP_PHY_AUX_CFG1);
+ writel(val, qmp->dp_dp_phy + QSERDES_DP_PHY_AUX_CFG1);
return 0;
}
-static void qcom_qmp_v4_phy_dp_aux_init(struct qmp_phy *qphy)
+static void qmp_v4_dp_aux_init(struct qmp_combo *qmp)
{
writel(DP_PHY_PD_CTL_PWRDN | DP_PHY_PD_CTL_PSR_PWRDN | DP_PHY_PD_CTL_AUX_PWRDN |
DP_PHY_PD_CTL_PLL_PWRDN | DP_PHY_PD_CTL_DP_CLAMP_EN,
- qphy->pcs + QSERDES_DP_PHY_PD_CTL);
+ qmp->dp_dp_phy + QSERDES_DP_PHY_PD_CTL);
/* Turn on BIAS current for PHY/PLL */
- writel(0x17, qphy->serdes + QSERDES_V4_COM_BIAS_EN_CLKBUFLR_EN);
+ writel(0x17, qmp->dp_serdes + QSERDES_V4_COM_BIAS_EN_CLKBUFLR_EN);
- writel(0x00, qphy->pcs + QSERDES_DP_PHY_AUX_CFG0);
- writel(0x13, qphy->pcs + QSERDES_DP_PHY_AUX_CFG1);
- writel(0xa4, qphy->pcs + QSERDES_DP_PHY_AUX_CFG2);
- writel(0x00, qphy->pcs + QSERDES_DP_PHY_AUX_CFG3);
- writel(0x0a, qphy->pcs + QSERDES_DP_PHY_AUX_CFG4);
- writel(0x26, qphy->pcs + QSERDES_DP_PHY_AUX_CFG5);
- writel(0x0a, qphy->pcs + QSERDES_DP_PHY_AUX_CFG6);
- writel(0x03, qphy->pcs + QSERDES_DP_PHY_AUX_CFG7);
- writel(0xb7, qphy->pcs + QSERDES_DP_PHY_AUX_CFG8);
- writel(0x03, qphy->pcs + QSERDES_DP_PHY_AUX_CFG9);
- qphy->dp_aux_cfg = 0;
+ writel(0x00, qmp->dp_dp_phy + QSERDES_DP_PHY_AUX_CFG0);
+ writel(0x13, qmp->dp_dp_phy + QSERDES_DP_PHY_AUX_CFG1);
+ writel(0xa4, qmp->dp_dp_phy + QSERDES_DP_PHY_AUX_CFG2);
+ writel(0x00, qmp->dp_dp_phy + QSERDES_DP_PHY_AUX_CFG3);
+ writel(0x0a, qmp->dp_dp_phy + QSERDES_DP_PHY_AUX_CFG4);
+ writel(0x26, qmp->dp_dp_phy + QSERDES_DP_PHY_AUX_CFG5);
+ writel(0x0a, qmp->dp_dp_phy + QSERDES_DP_PHY_AUX_CFG6);
+ writel(0x03, qmp->dp_dp_phy + QSERDES_DP_PHY_AUX_CFG7);
+ writel(0xb7, qmp->dp_dp_phy + QSERDES_DP_PHY_AUX_CFG8);
+ writel(0x03, qmp->dp_dp_phy + QSERDES_DP_PHY_AUX_CFG9);
+ qmp->dp_aux_cfg = 0;
writel(PHY_AUX_STOP_ERR_MASK | PHY_AUX_DEC_ERR_MASK |
PHY_AUX_SYNC_ERR_MASK | PHY_AUX_ALIGN_ERR_MASK |
PHY_AUX_REQ_ERR_MASK,
- qphy->pcs + QSERDES_V4_DP_PHY_AUX_INTERRUPT_MASK);
+ qmp->dp_dp_phy + QSERDES_V4_DP_PHY_AUX_INTERRUPT_MASK);
}
-static void qcom_qmp_v4_phy_configure_dp_tx(struct qmp_phy *qphy)
+static void qmp_v4_configure_dp_tx(struct qmp_combo *qmp)
{
/* Program default values before writing proper values */
- writel(0x27, qphy->tx + QSERDES_V4_TX_TX_DRV_LVL);
- writel(0x27, qphy->tx2 + QSERDES_V4_TX_TX_DRV_LVL);
+ writel(0x27, qmp->dp_tx + QSERDES_V4_TX_TX_DRV_LVL);
+ writel(0x27, qmp->dp_tx2 + QSERDES_V4_TX_TX_DRV_LVL);
- writel(0x20, qphy->tx + QSERDES_V4_TX_TX_EMP_POST1_LVL);
- writel(0x20, qphy->tx2 + QSERDES_V4_TX_TX_EMP_POST1_LVL);
+ writel(0x20, qmp->dp_tx + QSERDES_V4_TX_TX_EMP_POST1_LVL);
+ writel(0x20, qmp->dp_tx2 + QSERDES_V4_TX_TX_EMP_POST1_LVL);
- qmp_combo_configure_dp_swing(qphy, QSERDES_V4_TX_TX_DRV_LVL,
+ qmp_combo_configure_dp_swing(qmp, QSERDES_V4_TX_TX_DRV_LVL,
QSERDES_V4_TX_TX_EMP_POST1_LVL);
}
-static int qcom_qmp_v45_phy_configure_dp_phy(struct qmp_phy *qphy)
+static int qmp_v45_configure_dp_phy(struct qmp_combo *qmp)
{
- const struct qmp_phy_dp_clks *dp_clks = qphy->dp_clks;
- const struct phy_configure_opts_dp *dp_opts = &qphy->dp_opts;
+ const struct phy_configure_opts_dp *dp_opts = &qmp->dp_opts;
u32 phy_vco_div, status;
unsigned long pixel_freq;
- writel(0x0f, qphy->pcs + QSERDES_V4_DP_PHY_CFG_1);
+ writel(0x0f, qmp->dp_dp_phy + QSERDES_V4_DP_PHY_CFG_1);
- qmp_combo_configure_dp_mode(qphy);
+ qmp_combo_configure_dp_mode(qmp);
- writel(0x13, qphy->pcs + QSERDES_DP_PHY_AUX_CFG1);
- writel(0xa4, qphy->pcs + QSERDES_DP_PHY_AUX_CFG2);
+ writel(0x13, qmp->dp_dp_phy + QSERDES_DP_PHY_AUX_CFG1);
+ writel(0xa4, qmp->dp_dp_phy + QSERDES_DP_PHY_AUX_CFG2);
- writel(0x05, qphy->pcs + QSERDES_V4_DP_PHY_TX0_TX1_LANE_CTL);
- writel(0x05, qphy->pcs + QSERDES_V4_DP_PHY_TX2_TX3_LANE_CTL);
+ writel(0x05, qmp->dp_dp_phy + QSERDES_V4_DP_PHY_TX0_TX1_LANE_CTL);
+ writel(0x05, qmp->dp_dp_phy + QSERDES_V4_DP_PHY_TX2_TX3_LANE_CTL);
switch (dp_opts->link_rate) {
case 1620:
@@ -1722,49 +1580,49 @@ static int qcom_qmp_v45_phy_configure_dp_phy(struct qmp_phy *qphy)
/* Other link rates aren't supported */
return -EINVAL;
}
- writel(phy_vco_div, qphy->pcs + QSERDES_V4_DP_PHY_VCO_DIV);
+ writel(phy_vco_div, qmp->dp_dp_phy + QSERDES_V4_DP_PHY_VCO_DIV);
- clk_set_rate(dp_clks->dp_link_hw.clk, dp_opts->link_rate * 100000);
- clk_set_rate(dp_clks->dp_pixel_hw.clk, pixel_freq);
+ clk_set_rate(qmp->dp_link_hw.clk, dp_opts->link_rate * 100000);
+ clk_set_rate(qmp->dp_pixel_hw.clk, pixel_freq);
- writel(0x01, qphy->pcs + QSERDES_DP_PHY_CFG);
- writel(0x05, qphy->pcs + QSERDES_DP_PHY_CFG);
- writel(0x01, qphy->pcs + QSERDES_DP_PHY_CFG);
- writel(0x09, qphy->pcs + QSERDES_DP_PHY_CFG);
+ writel(0x01, qmp->dp_dp_phy + QSERDES_DP_PHY_CFG);
+ writel(0x05, qmp->dp_dp_phy + QSERDES_DP_PHY_CFG);
+ writel(0x01, qmp->dp_dp_phy + QSERDES_DP_PHY_CFG);
+ writel(0x09, qmp->dp_dp_phy + QSERDES_DP_PHY_CFG);
- writel(0x20, qphy->serdes + QSERDES_V4_COM_RESETSM_CNTRL);
+ writel(0x20, qmp->dp_serdes + QSERDES_V4_COM_RESETSM_CNTRL);
- if (readl_poll_timeout(qphy->serdes + QSERDES_V4_COM_C_READY_STATUS,
+ if (readl_poll_timeout(qmp->dp_serdes + QSERDES_V4_COM_C_READY_STATUS,
status,
((status & BIT(0)) > 0),
500,
10000))
return -ETIMEDOUT;
- if (readl_poll_timeout(qphy->serdes + QSERDES_V4_COM_CMN_STATUS,
+ if (readl_poll_timeout(qmp->dp_serdes + QSERDES_V4_COM_CMN_STATUS,
status,
((status & BIT(0)) > 0),
500,
10000))
return -ETIMEDOUT;
- if (readl_poll_timeout(qphy->serdes + QSERDES_V4_COM_CMN_STATUS,
+ if (readl_poll_timeout(qmp->dp_serdes + QSERDES_V4_COM_CMN_STATUS,
status,
((status & BIT(1)) > 0),
500,
10000))
return -ETIMEDOUT;
- writel(0x19, qphy->pcs + QSERDES_DP_PHY_CFG);
+ writel(0x19, qmp->dp_dp_phy + QSERDES_DP_PHY_CFG);
- if (readl_poll_timeout(qphy->pcs + QSERDES_V4_DP_PHY_STATUS,
+ if (readl_poll_timeout(qmp->dp_dp_phy + QSERDES_V4_DP_PHY_STATUS,
status,
((status & BIT(0)) > 0),
500,
10000))
return -ETIMEDOUT;
- if (readl_poll_timeout(qphy->pcs + QSERDES_V4_DP_PHY_STATUS,
+ if (readl_poll_timeout(qmp->dp_dp_phy + QSERDES_V4_DP_PHY_STATUS,
status,
((status & BIT(1)) > 0),
500,
@@ -1774,15 +1632,15 @@ static int qcom_qmp_v45_phy_configure_dp_phy(struct qmp_phy *qphy)
return 0;
}
-static int qcom_qmp_v4_phy_configure_dp_phy(struct qmp_phy *qphy)
+static int qmp_v4_configure_dp_phy(struct qmp_combo *qmp)
{
- const struct phy_configure_opts_dp *dp_opts = &qphy->dp_opts;
+ const struct phy_configure_opts_dp *dp_opts = &qmp->dp_opts;
u32 bias0_en, drvr0_en, bias1_en, drvr1_en;
bool reverse = false;
u32 status;
int ret;
- ret = qcom_qmp_v45_phy_configure_dp_phy(qphy);
+ ret = qmp_v45_configure_dp_phy(qmp);
if (ret < 0)
return ret;
@@ -1808,43 +1666,43 @@ static int qcom_qmp_v4_phy_configure_dp_phy(struct qmp_phy *qphy)
drvr1_en = 0x10;
}
- writel(drvr0_en, qphy->tx + QSERDES_V4_TX_HIGHZ_DRVR_EN);
- writel(bias0_en, qphy->tx + QSERDES_V4_TX_TRANSCEIVER_BIAS_EN);
- writel(drvr1_en, qphy->tx2 + QSERDES_V4_TX_HIGHZ_DRVR_EN);
- writel(bias1_en, qphy->tx2 + QSERDES_V4_TX_TRANSCEIVER_BIAS_EN);
+ writel(drvr0_en, qmp->dp_tx + QSERDES_V4_TX_HIGHZ_DRVR_EN);
+ writel(bias0_en, qmp->dp_tx + QSERDES_V4_TX_TRANSCEIVER_BIAS_EN);
+ writel(drvr1_en, qmp->dp_tx2 + QSERDES_V4_TX_HIGHZ_DRVR_EN);
+ writel(bias1_en, qmp->dp_tx2 + QSERDES_V4_TX_TRANSCEIVER_BIAS_EN);
- writel(0x18, qphy->pcs + QSERDES_DP_PHY_CFG);
+ writel(0x18, qmp->dp_dp_phy + QSERDES_DP_PHY_CFG);
udelay(2000);
- writel(0x19, qphy->pcs + QSERDES_DP_PHY_CFG);
+ writel(0x19, qmp->dp_dp_phy + QSERDES_DP_PHY_CFG);
- if (readl_poll_timeout(qphy->pcs + QSERDES_V4_DP_PHY_STATUS,
+ if (readl_poll_timeout(qmp->dp_dp_phy + QSERDES_V4_DP_PHY_STATUS,
status,
((status & BIT(1)) > 0),
500,
10000))
return -ETIMEDOUT;
- writel(0x0a, qphy->tx + QSERDES_V4_TX_TX_POL_INV);
- writel(0x0a, qphy->tx2 + QSERDES_V4_TX_TX_POL_INV);
+ writel(0x0a, qmp->dp_tx + QSERDES_V4_TX_TX_POL_INV);
+ writel(0x0a, qmp->dp_tx2 + QSERDES_V4_TX_TX_POL_INV);
- writel(0x27, qphy->tx + QSERDES_V4_TX_TX_DRV_LVL);
- writel(0x27, qphy->tx2 + QSERDES_V4_TX_TX_DRV_LVL);
+ writel(0x27, qmp->dp_tx + QSERDES_V4_TX_TX_DRV_LVL);
+ writel(0x27, qmp->dp_tx2 + QSERDES_V4_TX_TX_DRV_LVL);
- writel(0x20, qphy->tx + QSERDES_V4_TX_TX_EMP_POST1_LVL);
- writel(0x20, qphy->tx2 + QSERDES_V4_TX_TX_EMP_POST1_LVL);
+ writel(0x20, qmp->dp_tx + QSERDES_V4_TX_TX_EMP_POST1_LVL);
+ writel(0x20, qmp->dp_tx2 + QSERDES_V4_TX_TX_EMP_POST1_LVL);
return 0;
}
-static int qcom_qmp_v5_phy_configure_dp_phy(struct qmp_phy *qphy)
+static int qmp_v5_configure_dp_phy(struct qmp_combo *qmp)
{
- const struct phy_configure_opts_dp *dp_opts = &qphy->dp_opts;
+ const struct phy_configure_opts_dp *dp_opts = &qmp->dp_opts;
u32 bias0_en, drvr0_en, bias1_en, drvr1_en;
bool reverse = false;
u32 status;
int ret;
- ret = qcom_qmp_v45_phy_configure_dp_phy(qphy);
+ ret = qmp_v45_configure_dp_phy(qmp);
if (ret < 0)
return ret;
@@ -1865,30 +1723,30 @@ static int qcom_qmp_v5_phy_configure_dp_phy(struct qmp_phy *qphy)
drvr1_en = 0x10;
}
- writel(drvr0_en, qphy->tx + QSERDES_V5_5NM_TX_HIGHZ_DRVR_EN);
- writel(bias0_en, qphy->tx + QSERDES_V5_5NM_TX_TRANSCEIVER_BIAS_EN);
- writel(drvr1_en, qphy->tx2 + QSERDES_V5_5NM_TX_HIGHZ_DRVR_EN);
- writel(bias1_en, qphy->tx2 + QSERDES_V5_5NM_TX_TRANSCEIVER_BIAS_EN);
+ writel(drvr0_en, qmp->dp_tx + QSERDES_V5_5NM_TX_HIGHZ_DRVR_EN);
+ writel(bias0_en, qmp->dp_tx + QSERDES_V5_5NM_TX_TRANSCEIVER_BIAS_EN);
+ writel(drvr1_en, qmp->dp_tx2 + QSERDES_V5_5NM_TX_HIGHZ_DRVR_EN);
+ writel(bias1_en, qmp->dp_tx2 + QSERDES_V5_5NM_TX_TRANSCEIVER_BIAS_EN);
- writel(0x18, qphy->pcs + QSERDES_DP_PHY_CFG);
+ writel(0x18, qmp->dp_dp_phy + QSERDES_DP_PHY_CFG);
udelay(2000);
- writel(0x19, qphy->pcs + QSERDES_DP_PHY_CFG);
+ writel(0x19, qmp->dp_dp_phy + QSERDES_DP_PHY_CFG);
- if (readl_poll_timeout(qphy->pcs + QSERDES_V4_DP_PHY_STATUS,
+ if (readl_poll_timeout(qmp->dp_dp_phy + QSERDES_V4_DP_PHY_STATUS,
status,
((status & BIT(1)) > 0),
500,
10000))
return -ETIMEDOUT;
- writel(0x0a, qphy->tx + QSERDES_V5_5NM_TX_TX_POL_INV);
- writel(0x0a, qphy->tx2 + QSERDES_V5_5NM_TX_TX_POL_INV);
+ writel(0x0a, qmp->dp_tx + QSERDES_V5_5NM_TX_TX_POL_INV);
+ writel(0x0a, qmp->dp_tx2 + QSERDES_V5_5NM_TX_TX_POL_INV);
- writel(0x27, qphy->tx + QSERDES_V5_5NM_TX_TX_DRV_LVL);
- writel(0x27, qphy->tx2 + QSERDES_V5_5NM_TX_TX_DRV_LVL);
+ writel(0x27, qmp->dp_tx + QSERDES_V5_5NM_TX_TX_DRV_LVL);
+ writel(0x27, qmp->dp_tx2 + QSERDES_V5_5NM_TX_TX_DRV_LVL);
- writel(0x20, qphy->tx + QSERDES_V5_5NM_TX_TX_EMP_POST1_LVL);
- writel(0x20, qphy->tx2 + QSERDES_V5_5NM_TX_TX_EMP_POST1_LVL);
+ writel(0x20, qmp->dp_tx + QSERDES_V5_5NM_TX_TX_EMP_POST1_LVL);
+ writel(0x20, qmp->dp_tx2 + QSERDES_V5_5NM_TX_TX_EMP_POST1_LVL);
return 0;
}
@@ -1897,52 +1755,50 @@ static int qcom_qmp_v5_phy_configure_dp_phy(struct qmp_phy *qphy)
* We need to calibrate the aux setting here as many times
* as the caller tries
*/
-static int qcom_qmp_v4_dp_phy_calibrate(struct qmp_phy *qphy)
+static int qmp_v4_calibrate_dp_phy(struct qmp_combo *qmp)
{
static const u8 cfg1_settings[] = { 0x20, 0x13, 0x23, 0x1d };
u8 val;
- qphy->dp_aux_cfg++;
- qphy->dp_aux_cfg %= ARRAY_SIZE(cfg1_settings);
- val = cfg1_settings[qphy->dp_aux_cfg];
+ qmp->dp_aux_cfg++;
+ qmp->dp_aux_cfg %= ARRAY_SIZE(cfg1_settings);
+ val = cfg1_settings[qmp->dp_aux_cfg];
- writel(val, qphy->pcs + QSERDES_DP_PHY_AUX_CFG1);
+ writel(val, qmp->dp_dp_phy + QSERDES_DP_PHY_AUX_CFG1);
return 0;
}
-static int qcom_qmp_dp_phy_configure(struct phy *phy, union phy_configure_opts *opts)
+static int qmp_combo_dp_configure(struct phy *phy, union phy_configure_opts *opts)
{
const struct phy_configure_opts_dp *dp_opts = &opts->dp;
- struct qmp_phy *qphy = phy_get_drvdata(phy);
- const struct qmp_phy_cfg *cfg = qphy->cfg;
+ struct qmp_combo *qmp = phy_get_drvdata(phy);
+ const struct qmp_phy_cfg *cfg = qmp->cfg;
- memcpy(&qphy->dp_opts, dp_opts, sizeof(*dp_opts));
- if (qphy->dp_opts.set_voltages) {
- cfg->configure_dp_tx(qphy);
- qphy->dp_opts.set_voltages = 0;
+ memcpy(&qmp->dp_opts, dp_opts, sizeof(*dp_opts));
+ if (qmp->dp_opts.set_voltages) {
+ cfg->configure_dp_tx(qmp);
+ qmp->dp_opts.set_voltages = 0;
}
return 0;
}
-static int qcom_qmp_dp_phy_calibrate(struct phy *phy)
+static int qmp_combo_dp_calibrate(struct phy *phy)
{
- struct qmp_phy *qphy = phy_get_drvdata(phy);
- const struct qmp_phy_cfg *cfg = qphy->cfg;
+ struct qmp_combo *qmp = phy_get_drvdata(phy);
+ const struct qmp_phy_cfg *cfg = qmp->cfg;
if (cfg->calibrate_dp_phy)
- return cfg->calibrate_dp_phy(qphy);
+ return cfg->calibrate_dp_phy(qmp);
return 0;
}
-static int qmp_combo_com_init(struct qmp_phy *qphy)
+static int qmp_combo_com_init(struct qmp_combo *qmp)
{
- struct qcom_qmp *qmp = qphy->qmp;
- const struct qmp_phy_cfg *cfg = qphy->cfg;
- void __iomem *pcs = qphy->pcs;
- void __iomem *dp_com = qmp->dp_com;
+ const struct qmp_phy_cfg *cfg = qmp->cfg;
+ void __iomem *com = qmp->com;
int ret;
mutex_lock(&qmp->phy_mutex);
@@ -1951,7 +1807,6 @@ static int qmp_combo_com_init(struct qmp_phy *qphy)
return 0;
}
- /* turn on regulator supplies */
ret = regulator_bulk_enable(cfg->num_vregs, qmp->vregs);
if (ret) {
dev_err(qmp->dev, "failed to enable regulators, err=%d\n", ret);
@@ -1974,33 +1829,28 @@ static int qmp_combo_com_init(struct qmp_phy *qphy)
if (ret)
goto err_assert_reset;
- qphy_setbits(dp_com, QPHY_V3_DP_COM_POWER_DOWN_CTRL, SW_PWRDN);
+ qphy_setbits(com, QPHY_V3_DP_COM_POWER_DOWN_CTRL, SW_PWRDN);
/* override hardware control for reset of qmp phy */
- qphy_setbits(dp_com, QPHY_V3_DP_COM_RESET_OVRD_CTRL,
+ qphy_setbits(com, QPHY_V3_DP_COM_RESET_OVRD_CTRL,
SW_DPPHY_RESET_MUX | SW_DPPHY_RESET |
SW_USB3PHY_RESET_MUX | SW_USB3PHY_RESET);
/* Default type-c orientation, i.e CC1 */
- qphy_setbits(dp_com, QPHY_V3_DP_COM_TYPEC_CTRL, 0x02);
+ qphy_setbits(com, QPHY_V3_DP_COM_TYPEC_CTRL, 0x02);
- qphy_setbits(dp_com, QPHY_V3_DP_COM_PHY_MODE_CTRL, USB3_MODE | DP_MODE);
+ qphy_setbits(com, QPHY_V3_DP_COM_PHY_MODE_CTRL, USB3_MODE | DP_MODE);
/* bring both QMP USB and QMP DP PHYs PCS block out of reset */
- qphy_clrbits(dp_com, QPHY_V3_DP_COM_RESET_OVRD_CTRL,
+ qphy_clrbits(com, QPHY_V3_DP_COM_RESET_OVRD_CTRL,
SW_DPPHY_RESET_MUX | SW_DPPHY_RESET |
SW_USB3PHY_RESET_MUX | SW_USB3PHY_RESET);
- qphy_clrbits(dp_com, QPHY_V3_DP_COM_SWI_CTRL, 0x03);
- qphy_clrbits(dp_com, QPHY_V3_DP_COM_SW_RESET, SW_RESET);
+ qphy_clrbits(com, QPHY_V3_DP_COM_SWI_CTRL, 0x03);
+ qphy_clrbits(com, QPHY_V3_DP_COM_SW_RESET, SW_RESET);
- if (cfg->regs[QPHY_PCS_POWER_DOWN_CONTROL])
- qphy_setbits(pcs,
- cfg->regs[QPHY_PCS_POWER_DOWN_CONTROL],
- cfg->pwrdn_ctrl);
- else
- qphy_setbits(pcs, QPHY_V2_PCS_POWER_DOWN_CONTROL,
- cfg->pwrdn_ctrl);
+ qphy_setbits(qmp->pcs, cfg->regs[QPHY_PCS_POWER_DOWN_CONTROL],
+ SW_PWRDN);
mutex_unlock(&qmp->phy_mutex);
@@ -2016,10 +1866,9 @@ static int qmp_combo_com_init(struct qmp_phy *qphy)
return ret;
}
-static int qmp_combo_com_exit(struct qmp_phy *qphy)
+static int qmp_combo_com_exit(struct qmp_combo *qmp)
{
- struct qcom_qmp *qmp = qphy->qmp;
- const struct qmp_phy_cfg *cfg = qphy->cfg;
+ const struct qmp_phy_cfg *cfg = qmp->cfg;
mutex_lock(&qmp->phy_mutex);
if (--qmp->init_count) {
@@ -2027,8 +1876,6 @@ static int qmp_combo_com_exit(struct qmp_phy *qphy)
return 0;
}
- reset_control_assert(qmp->ufs_reset);
-
reset_control_bulk_assert(cfg->num_resets, qmp->resets);
clk_bulk_disable_unprepare(cfg->num_clks, qmp->clks);
@@ -2040,183 +1887,201 @@ static int qmp_combo_com_exit(struct qmp_phy *qphy)
return 0;
}
-static int qmp_combo_init(struct phy *phy)
+static int qmp_combo_dp_init(struct phy *phy)
{
- struct qmp_phy *qphy = phy_get_drvdata(phy);
- struct qcom_qmp *qmp = qphy->qmp;
- const struct qmp_phy_cfg *cfg = qphy->cfg;
+ struct qmp_combo *qmp = phy_get_drvdata(phy);
+ const struct qmp_phy_cfg *cfg = qmp->cfg;
int ret;
- dev_vdbg(qmp->dev, "Initializing QMP phy\n");
- ret = qmp_combo_com_init(qphy);
+ ret = qmp_combo_com_init(qmp);
if (ret)
return ret;
- if (cfg->type == PHY_TYPE_DP)
- cfg->dp_aux_init(qphy);
+ cfg->dp_aux_init(qmp);
return 0;
}
-static int qmp_combo_power_on(struct phy *phy)
+static int qmp_combo_dp_exit(struct phy *phy)
{
- struct qmp_phy *qphy = phy_get_drvdata(phy);
- struct qcom_qmp *qmp = qphy->qmp;
- const struct qmp_phy_cfg *cfg = qphy->cfg;
- void __iomem *tx = qphy->tx;
- void __iomem *rx = qphy->rx;
- void __iomem *pcs = qphy->pcs;
+ struct qmp_combo *qmp = phy_get_drvdata(phy);
+
+ qmp_combo_com_exit(qmp);
+
+ return 0;
+}
+
+static int qmp_combo_dp_power_on(struct phy *phy)
+{
+ struct qmp_combo *qmp = phy_get_drvdata(phy);
+ const struct qmp_phy_cfg *cfg = qmp->cfg;
+ void __iomem *tx = qmp->dp_tx;
+ void __iomem *tx2 = qmp->dp_tx2;
+
+ qmp_combo_dp_serdes_init(qmp);
+
+ qmp_combo_configure_lane(tx, cfg->dp_tx_tbl, cfg->dp_tx_tbl_num, 1);
+ qmp_combo_configure_lane(tx2, cfg->dp_tx_tbl, cfg->dp_tx_tbl_num, 2);
+
+ /* Configure special DP tx tunings */
+ cfg->configure_dp_tx(qmp);
+
+ /* Configure link rate, swing, etc. */
+ cfg->configure_dp_phy(qmp);
+
+ return 0;
+}
+
+static int qmp_combo_dp_power_off(struct phy *phy)
+{
+ struct qmp_combo *qmp = phy_get_drvdata(phy);
+
+ /* Assert DP PHY power down */
+ writel(DP_PHY_PD_CTL_PSR_PWRDN, qmp->dp_dp_phy + QSERDES_DP_PHY_PD_CTL);
+
+ return 0;
+}
+
+static int qmp_combo_usb_power_on(struct phy *phy)
+{
+ struct qmp_combo *qmp = phy_get_drvdata(phy);
+ const struct qmp_phy_cfg *cfg = qmp->cfg;
+ void __iomem *serdes = qmp->serdes;
+ void __iomem *tx = qmp->tx;
+ void __iomem *rx = qmp->rx;
+ void __iomem *tx2 = qmp->tx2;
+ void __iomem *rx2 = qmp->rx2;
+ void __iomem *pcs = qmp->pcs;
void __iomem *status;
- unsigned int mask, val, ready;
+ unsigned int val;
int ret;
- qmp_combo_serdes_init(qphy);
+ qmp_combo_configure(serdes, cfg->serdes_tbl, cfg->serdes_tbl_num);
- ret = clk_prepare_enable(qphy->pipe_clk);
+ ret = clk_prepare_enable(qmp->pipe_clk);
if (ret) {
dev_err(qmp->dev, "pipe_clk enable failed err=%d\n", ret);
return ret;
}
/* Tx, Rx, and PCS configurations */
- qmp_combo_configure_lane(tx, cfg->regs, cfg->tx_tbl, cfg->tx_tbl_num, 1);
+ qmp_combo_configure_lane(tx, cfg->tx_tbl, cfg->tx_tbl_num, 1);
+ qmp_combo_configure_lane(tx2, cfg->tx_tbl, cfg->tx_tbl_num, 2);
- if (cfg->lanes >= 2) {
- qmp_combo_configure_lane(qphy->tx2, cfg->regs, cfg->tx_tbl,
- cfg->tx_tbl_num, 2);
- }
+ qmp_combo_configure_lane(rx, cfg->rx_tbl, cfg->rx_tbl_num, 1);
+ qmp_combo_configure_lane(rx2, cfg->rx_tbl, cfg->rx_tbl_num, 2);
- /* Configure special DP tx tunings */
- if (cfg->type == PHY_TYPE_DP)
- cfg->configure_dp_tx(qphy);
-
- qmp_combo_configure_lane(rx, cfg->regs, cfg->rx_tbl, cfg->rx_tbl_num, 1);
-
- if (cfg->lanes >= 2) {
- qmp_combo_configure_lane(qphy->rx2, cfg->regs, cfg->rx_tbl,
- cfg->rx_tbl_num, 2);
- }
-
- /* Configure link rate, swing, etc. */
- if (cfg->type == PHY_TYPE_DP)
- cfg->configure_dp_phy(qphy);
- else
- qmp_combo_configure(pcs, cfg->regs, cfg->pcs_tbl, cfg->pcs_tbl_num);
-
- ret = reset_control_deassert(qmp->ufs_reset);
- if (ret)
- goto err_disable_pipe_clk;
+ qmp_combo_configure(pcs, cfg->pcs_tbl, cfg->pcs_tbl_num);
if (cfg->has_pwrdn_delay)
- usleep_range(cfg->pwrdn_delay_min, cfg->pwrdn_delay_max);
+ usleep_range(10, 20);
- if (cfg->type != PHY_TYPE_DP) {
- /* Pull PHY out of reset state */
- qphy_clrbits(pcs, cfg->regs[QPHY_SW_RESET], SW_RESET);
- /* start SerDes and Phy-Coding-Sublayer */
- qphy_setbits(pcs, cfg->regs[QPHY_START_CTRL], cfg->start_ctrl);
+ /* Pull PHY out of reset state */
+ qphy_clrbits(pcs, cfg->regs[QPHY_SW_RESET], SW_RESET);
- status = pcs + cfg->regs[QPHY_PCS_STATUS];
- mask = cfg->phy_status;
- ready = 0;
+ /* start SerDes and Phy-Coding-Sublayer */
+ qphy_setbits(pcs, cfg->regs[QPHY_START_CTRL], SERDES_START | PCS_START);
- ret = readl_poll_timeout(status, val, (val & mask) == ready, 10,
- PHY_INIT_COMPLETE_TIMEOUT);
- if (ret) {
- dev_err(qmp->dev, "phy initialization timed-out\n");
- goto err_disable_pipe_clk;
- }
+ status = pcs + cfg->regs[QPHY_PCS_STATUS];
+ ret = readl_poll_timeout(status, val, !(val & PHYSTATUS), 200,
+ PHY_INIT_COMPLETE_TIMEOUT);
+ if (ret) {
+ dev_err(qmp->dev, "phy initialization timed-out\n");
+ goto err_disable_pipe_clk;
}
+
return 0;
err_disable_pipe_clk:
- clk_disable_unprepare(qphy->pipe_clk);
+ clk_disable_unprepare(qmp->pipe_clk);
return ret;
}
-static int qmp_combo_power_off(struct phy *phy)
+static int qmp_combo_usb_power_off(struct phy *phy)
{
- struct qmp_phy *qphy = phy_get_drvdata(phy);
- const struct qmp_phy_cfg *cfg = qphy->cfg;
+ struct qmp_combo *qmp = phy_get_drvdata(phy);
+ const struct qmp_phy_cfg *cfg = qmp->cfg;
- clk_disable_unprepare(qphy->pipe_clk);
+ clk_disable_unprepare(qmp->pipe_clk);
- if (cfg->type == PHY_TYPE_DP) {
- /* Assert DP PHY power down */
- writel(DP_PHY_PD_CTL_PSR_PWRDN, qphy->pcs + QSERDES_DP_PHY_PD_CTL);
- } else {
- /* PHY reset */
- qphy_setbits(qphy->pcs, cfg->regs[QPHY_SW_RESET], SW_RESET);
+ /* PHY reset */
+ qphy_setbits(qmp->pcs, cfg->regs[QPHY_SW_RESET], SW_RESET);
- /* stop SerDes and Phy-Coding-Sublayer */
- qphy_clrbits(qphy->pcs, cfg->regs[QPHY_START_CTRL], cfg->start_ctrl);
+ /* stop SerDes and Phy-Coding-Sublayer */
+ qphy_clrbits(qmp->pcs, cfg->regs[QPHY_START_CTRL],
+ SERDES_START | PCS_START);
- /* Put PHY into POWER DOWN state: active low */
- if (cfg->regs[QPHY_PCS_POWER_DOWN_CONTROL]) {
- qphy_clrbits(qphy->pcs, cfg->regs[QPHY_PCS_POWER_DOWN_CONTROL],
- cfg->pwrdn_ctrl);
- } else {
- qphy_clrbits(qphy->pcs, QPHY_V2_PCS_POWER_DOWN_CONTROL,
- cfg->pwrdn_ctrl);
- }
- }
+ /* Put PHY into POWER DOWN state: active low */
+ qphy_clrbits(qmp->pcs, cfg->regs[QPHY_PCS_POWER_DOWN_CONTROL],
+ SW_PWRDN);
return 0;
}
-static int qmp_combo_exit(struct phy *phy)
+static int qmp_combo_usb_init(struct phy *phy)
{
- struct qmp_phy *qphy = phy_get_drvdata(phy);
-
- qmp_combo_com_exit(qphy);
-
- return 0;
-}
-
-static int qmp_combo_enable(struct phy *phy)
-{
+ struct qmp_combo *qmp = phy_get_drvdata(phy);
int ret;
- ret = qmp_combo_init(phy);
+ ret = qmp_combo_com_init(qmp);
if (ret)
return ret;
- ret = qmp_combo_power_on(phy);
+ ret = qmp_combo_usb_power_on(phy);
if (ret)
- qmp_combo_exit(phy);
+ qmp_combo_com_exit(qmp);
return ret;
}
-static int qmp_combo_disable(struct phy *phy)
+static int qmp_combo_usb_exit(struct phy *phy)
{
+ struct qmp_combo *qmp = phy_get_drvdata(phy);
int ret;
- ret = qmp_combo_power_off(phy);
+ ret = qmp_combo_usb_power_off(phy);
if (ret)
return ret;
- return qmp_combo_exit(phy);
+
+ return qmp_combo_com_exit(qmp);
}
-static int qmp_combo_set_mode(struct phy *phy, enum phy_mode mode, int submode)
+static int qmp_combo_usb_set_mode(struct phy *phy, enum phy_mode mode, int submode)
{
- struct qmp_phy *qphy = phy_get_drvdata(phy);
+ struct qmp_combo *qmp = phy_get_drvdata(phy);
- qphy->mode = mode;
+ qmp->mode = mode;
return 0;
}
-static void qmp_combo_enable_autonomous_mode(struct qmp_phy *qphy)
+static const struct phy_ops qmp_combo_usb_phy_ops = {
+ .init = qmp_combo_usb_init,
+ .exit = qmp_combo_usb_exit,
+ .set_mode = qmp_combo_usb_set_mode,
+ .owner = THIS_MODULE,
+};
+
+static const struct phy_ops qmp_combo_dp_phy_ops = {
+ .init = qmp_combo_dp_init,
+ .configure = qmp_combo_dp_configure,
+ .power_on = qmp_combo_dp_power_on,
+ .calibrate = qmp_combo_dp_calibrate,
+ .power_off = qmp_combo_dp_power_off,
+ .exit = qmp_combo_dp_exit,
+ .owner = THIS_MODULE,
+};
+
+static void qmp_combo_enable_autonomous_mode(struct qmp_combo *qmp)
{
- const struct qmp_phy_cfg *cfg = qphy->cfg;
- void __iomem *pcs_usb = qphy->pcs_usb ?: qphy->pcs;
- void __iomem *pcs_misc = qphy->pcs_misc;
+ const struct qmp_phy_cfg *cfg = qmp->cfg;
+ void __iomem *pcs_usb = qmp->pcs_usb ?: qmp->pcs;
+ void __iomem *pcs_misc = qmp->pcs_misc;
u32 intr_mask;
- if (qphy->mode == PHY_MODE_USB_HOST_SS ||
- qphy->mode == PHY_MODE_USB_DEVICE_SS)
+ if (qmp->mode == PHY_MODE_USB_HOST_SS ||
+ qmp->mode == PHY_MODE_USB_DEVICE_SS)
intr_mask = ARCVR_DTCT_EN | ALFPS_DTCT_EN;
else
intr_mask = ARCVR_DTCT_EN | ARCVR_DTCT_EVENT_SEL;
@@ -2237,11 +2102,11 @@ static void qmp_combo_enable_autonomous_mode(struct qmp_phy *qphy)
qphy_clrbits(pcs_misc, QPHY_V3_PCS_MISC_CLAMP_ENABLE, CLAMP_EN);
}
-static void qmp_combo_disable_autonomous_mode(struct qmp_phy *qphy)
+static void qmp_combo_disable_autonomous_mode(struct qmp_combo *qmp)
{
- const struct qmp_phy_cfg *cfg = qphy->cfg;
- void __iomem *pcs_usb = qphy->pcs_usb ?: qphy->pcs;
- void __iomem *pcs_misc = qphy->pcs_misc;
+ const struct qmp_phy_cfg *cfg = qmp->cfg;
+ void __iomem *pcs_usb = qmp->pcs_usb ?: qmp->pcs;
+ void __iomem *pcs_misc = qmp->pcs_misc;
/* Disable i/o clamp_n on resume for normal mode */
if (pcs_misc)
@@ -2257,24 +2122,19 @@ static void qmp_combo_disable_autonomous_mode(struct qmp_phy *qphy)
static int __maybe_unused qmp_combo_runtime_suspend(struct device *dev)
{
- struct qcom_qmp *qmp = dev_get_drvdata(dev);
- struct qmp_phy *qphy = qmp->phys[0];
- const struct qmp_phy_cfg *cfg = qphy->cfg;
+ struct qmp_combo *qmp = dev_get_drvdata(dev);
+ const struct qmp_phy_cfg *cfg = qmp->cfg;
- dev_vdbg(dev, "Suspending QMP phy, mode:%d\n", qphy->mode);
-
- /* Supported only for USB3 PHY and luckily USB3 is the first phy */
- if (cfg->type != PHY_TYPE_USB3)
- return 0;
+ dev_vdbg(dev, "Suspending QMP phy, mode:%d\n", qmp->mode);
if (!qmp->init_count) {
dev_vdbg(dev, "PHY not initialized, bailing out\n");
return 0;
}
- qmp_combo_enable_autonomous_mode(qphy);
+ qmp_combo_enable_autonomous_mode(qmp);
- clk_disable_unprepare(qphy->pipe_clk);
+ clk_disable_unprepare(qmp->pipe_clk);
clk_bulk_disable_unprepare(cfg->num_clks, qmp->clks);
return 0;
@@ -2282,16 +2142,11 @@ static int __maybe_unused qmp_combo_runtime_suspend(struct device *dev)
static int __maybe_unused qmp_combo_runtime_resume(struct device *dev)
{
- struct qcom_qmp *qmp = dev_get_drvdata(dev);
- struct qmp_phy *qphy = qmp->phys[0];
- const struct qmp_phy_cfg *cfg = qphy->cfg;
+ struct qmp_combo *qmp = dev_get_drvdata(dev);
+ const struct qmp_phy_cfg *cfg = qmp->cfg;
int ret = 0;
- dev_vdbg(dev, "Resuming QMP phy, mode:%d\n", qphy->mode);
-
- /* Supported only for USB3 PHY and luckily USB3 is the first phy */
- if (cfg->type != PHY_TYPE_USB3)
- return 0;
+ dev_vdbg(dev, "Resuming QMP phy, mode:%d\n", qmp->mode);
if (!qmp->init_count) {
dev_vdbg(dev, "PHY not initialized, bailing out\n");
@@ -2302,21 +2157,27 @@ static int __maybe_unused qmp_combo_runtime_resume(struct device *dev)
if (ret)
return ret;
- ret = clk_prepare_enable(qphy->pipe_clk);
+ ret = clk_prepare_enable(qmp->pipe_clk);
if (ret) {
dev_err(dev, "pipe_clk enable failed, err=%d\n", ret);
clk_bulk_disable_unprepare(cfg->num_clks, qmp->clks);
return ret;
}
- qmp_combo_disable_autonomous_mode(qphy);
+ qmp_combo_disable_autonomous_mode(qmp);
return 0;
}
-static int qmp_combo_vreg_init(struct device *dev, const struct qmp_phy_cfg *cfg)
+static const struct dev_pm_ops qmp_combo_pm_ops = {
+ SET_RUNTIME_PM_OPS(qmp_combo_runtime_suspend,
+ qmp_combo_runtime_resume, NULL)
+};
+
+static int qmp_combo_vreg_init(struct qmp_combo *qmp)
{
- struct qcom_qmp *qmp = dev_get_drvdata(dev);
+ const struct qmp_phy_cfg *cfg = qmp->cfg;
+ struct device *dev = qmp->dev;
int num = cfg->num_vregs;
int ret, i;
@@ -2346,9 +2207,10 @@ static int qmp_combo_vreg_init(struct device *dev, const struct qmp_phy_cfg *cfg
return 0;
}
-static int qmp_combo_reset_init(struct device *dev, const struct qmp_phy_cfg *cfg)
+static int qmp_combo_reset_init(struct qmp_combo *qmp)
{
- struct qcom_qmp *qmp = dev_get_drvdata(dev);
+ const struct qmp_phy_cfg *cfg = qmp->cfg;
+ struct device *dev = qmp->dev;
int i;
int ret;
@@ -2367,9 +2229,10 @@ static int qmp_combo_reset_init(struct device *dev, const struct qmp_phy_cfg *cf
return 0;
}
-static int qmp_combo_clk_init(struct device *dev, const struct qmp_phy_cfg *cfg)
+static int qmp_combo_clk_init(struct qmp_combo *qmp)
{
- struct qcom_qmp *qmp = dev_get_drvdata(dev);
+ const struct qmp_phy_cfg *cfg = qmp->cfg;
+ struct device *dev = qmp->dev;
int num = cfg->num_clks;
int i;
@@ -2406,41 +2269,21 @@ static void phy_clk_release_provider(void *res)
* clk | +-------+ | +-----+
* +---------------+
*/
-static int phy_pipe_clk_register(struct qcom_qmp *qmp, struct device_node *np)
+static int phy_pipe_clk_register(struct qmp_combo *qmp, struct device_node *np)
{
- struct clk_fixed_rate *fixed;
+ struct clk_fixed_rate *fixed = &qmp->pipe_clk_fixed;
struct clk_init_data init = { };
- int ret;
+ char name[64];
- ret = of_property_read_string(np, "clock-output-names", &init.name);
- if (ret) {
- dev_err(qmp->dev, "%pOFn: No clock-output-names\n", np);
- return ret;
- }
-
- fixed = devm_kzalloc(qmp->dev, sizeof(*fixed), GFP_KERNEL);
- if (!fixed)
- return -ENOMEM;
-
+ snprintf(name, sizeof(name), "%s::pipe_clk", dev_name(qmp->dev));
+ init.name = name;
init.ops = &clk_fixed_rate_ops;
/* controllers using QMP phys use 125MHz pipe clock interface */
fixed->fixed_rate = 125000000;
fixed->hw.init = &init;
- ret = devm_clk_hw_register(qmp->dev, &fixed->hw);
- if (ret)
- return ret;
-
- ret = of_clk_add_hw_provider(np, of_clk_hw_simple_get, &fixed->hw);
- if (ret)
- return ret;
-
- /*
- * Roll a devm action because the clock provider is the child node, but
- * the child node is not actually a device.
- */
- return devm_add_action_or_reset(qmp->dev, phy_clk_release_provider, np);
+ return devm_clk_hw_register(qmp->dev, &fixed->hw);
}
/*
@@ -2492,8 +2335,7 @@ static int phy_pipe_clk_register(struct qcom_qmp *qmp, struct device_node *np)
* for DP pixel clock
*
*/
-static int qcom_qmp_dp_pixel_clk_determine_rate(struct clk_hw *hw,
- struct clk_rate_request *req)
+static int qmp_dp_pixel_clk_determine_rate(struct clk_hw *hw, struct clk_rate_request *req)
{
switch (req->rate) {
case 1620000000UL / 2:
@@ -2505,16 +2347,13 @@ static int qcom_qmp_dp_pixel_clk_determine_rate(struct clk_hw *hw,
}
}
-static unsigned long
-qcom_qmp_dp_pixel_clk_recalc_rate(struct clk_hw *hw, unsigned long parent_rate)
+static unsigned long qmp_dp_pixel_clk_recalc_rate(struct clk_hw *hw, unsigned long parent_rate)
{
- const struct qmp_phy_dp_clks *dp_clks;
- const struct qmp_phy *qphy;
+ const struct qmp_combo *qmp;
const struct phy_configure_opts_dp *dp_opts;
- dp_clks = container_of(hw, struct qmp_phy_dp_clks, dp_pixel_hw);
- qphy = dp_clks->qphy;
- dp_opts = &qphy->dp_opts;
+ qmp = container_of(hw, struct qmp_combo, dp_pixel_hw);
+ dp_opts = &qmp->dp_opts;
switch (dp_opts->link_rate) {
case 1620:
@@ -2530,13 +2369,12 @@ qcom_qmp_dp_pixel_clk_recalc_rate(struct clk_hw *hw, unsigned long parent_rate)
}
}
-static const struct clk_ops qcom_qmp_dp_pixel_clk_ops = {
- .determine_rate = qcom_qmp_dp_pixel_clk_determine_rate,
- .recalc_rate = qcom_qmp_dp_pixel_clk_recalc_rate,
+static const struct clk_ops qmp_dp_pixel_clk_ops = {
+ .determine_rate = qmp_dp_pixel_clk_determine_rate,
+ .recalc_rate = qmp_dp_pixel_clk_recalc_rate,
};
-static int qcom_qmp_dp_link_clk_determine_rate(struct clk_hw *hw,
- struct clk_rate_request *req)
+static int qmp_dp_link_clk_determine_rate(struct clk_hw *hw, struct clk_rate_request *req)
{
switch (req->rate) {
case 162000000:
@@ -2549,16 +2387,13 @@ static int qcom_qmp_dp_link_clk_determine_rate(struct clk_hw *hw,
}
}
-static unsigned long
-qcom_qmp_dp_link_clk_recalc_rate(struct clk_hw *hw, unsigned long parent_rate)
+static unsigned long qmp_dp_link_clk_recalc_rate(struct clk_hw *hw, unsigned long parent_rate)
{
- const struct qmp_phy_dp_clks *dp_clks;
- const struct qmp_phy *qphy;
+ const struct qmp_combo *qmp;
const struct phy_configure_opts_dp *dp_opts;
- dp_clks = container_of(hw, struct qmp_phy_dp_clks, dp_link_hw);
- qphy = dp_clks->qphy;
- dp_opts = &qphy->dp_opts;
+ qmp = container_of(hw, struct qmp_combo, dp_link_hw);
+ dp_opts = &qmp->dp_opts;
switch (dp_opts->link_rate) {
case 1620:
@@ -2571,15 +2406,14 @@ qcom_qmp_dp_link_clk_recalc_rate(struct clk_hw *hw, unsigned long parent_rate)
}
}
-static const struct clk_ops qcom_qmp_dp_link_clk_ops = {
- .determine_rate = qcom_qmp_dp_link_clk_determine_rate,
- .recalc_rate = qcom_qmp_dp_link_clk_recalc_rate,
+static const struct clk_ops qmp_dp_link_clk_ops = {
+ .determine_rate = qmp_dp_link_clk_determine_rate,
+ .recalc_rate = qmp_dp_link_clk_recalc_rate,
};
-static struct clk_hw *
-qcom_qmp_dp_clks_hw_get(struct of_phandle_args *clkspec, void *data)
+static struct clk_hw *qmp_dp_clks_hw_get(struct of_phandle_args *clkspec, void *data)
{
- struct qmp_phy_dp_clks *dp_clks = data;
+ struct qmp_combo *qmp = data;
unsigned int idx = clkspec->args[0];
if (idx >= 2) {
@@ -2588,43 +2422,76 @@ qcom_qmp_dp_clks_hw_get(struct of_phandle_args *clkspec, void *data)
}
if (idx == 0)
- return &dp_clks->dp_link_hw;
+ return &qmp->dp_link_hw;
- return &dp_clks->dp_pixel_hw;
+ return &qmp->dp_pixel_hw;
}
-static int phy_dp_clks_register(struct qcom_qmp *qmp, struct qmp_phy *qphy,
- struct device_node *np)
+static int phy_dp_clks_register(struct qmp_combo *qmp, struct device_node *np)
{
struct clk_init_data init = { };
- struct qmp_phy_dp_clks *dp_clks;
char name[64];
int ret;
- dp_clks = devm_kzalloc(qmp->dev, sizeof(*dp_clks), GFP_KERNEL);
- if (!dp_clks)
- return -ENOMEM;
-
- dp_clks->qphy = qphy;
- qphy->dp_clks = dp_clks;
-
snprintf(name, sizeof(name), "%s::link_clk", dev_name(qmp->dev));
- init.ops = &qcom_qmp_dp_link_clk_ops;
+ init.ops = &qmp_dp_link_clk_ops;
init.name = name;
- dp_clks->dp_link_hw.init = &init;
- ret = devm_clk_hw_register(qmp->dev, &dp_clks->dp_link_hw);
+ qmp->dp_link_hw.init = &init;
+ ret = devm_clk_hw_register(qmp->dev, &qmp->dp_link_hw);
if (ret)
return ret;
snprintf(name, sizeof(name), "%s::vco_div_clk", dev_name(qmp->dev));
- init.ops = &qcom_qmp_dp_pixel_clk_ops;
+ init.ops = &qmp_dp_pixel_clk_ops;
init.name = name;
- dp_clks->dp_pixel_hw.init = &init;
- ret = devm_clk_hw_register(qmp->dev, &dp_clks->dp_pixel_hw);
+ qmp->dp_pixel_hw.init = &init;
+ ret = devm_clk_hw_register(qmp->dev, &qmp->dp_pixel_hw);
if (ret)
return ret;
- ret = of_clk_add_hw_provider(np, qcom_qmp_dp_clks_hw_get, dp_clks);
+ return 0;
+}
+
+static struct clk_hw *qmp_combo_clk_hw_get(struct of_phandle_args *clkspec, void *data)
+{
+ struct qmp_combo *qmp = data;
+
+ switch (clkspec->args[0]) {
+ case QMP_USB43DP_USB3_PIPE_CLK:
+ return &qmp->pipe_clk_fixed.hw;
+ case QMP_USB43DP_DP_LINK_CLK:
+ return &qmp->dp_link_hw;
+ case QMP_USB43DP_DP_VCO_DIV_CLK:
+ return &qmp->dp_pixel_hw;
+ }
+
+ return ERR_PTR(-EINVAL);
+}
+
+static int qmp_combo_register_clocks(struct qmp_combo *qmp, struct device_node *usb_np,
+ struct device_node *dp_np)
+{
+ int ret;
+
+ ret = phy_pipe_clk_register(qmp, usb_np);
+ if (ret)
+ return ret;
+
+ ret = phy_dp_clks_register(qmp, dp_np);
+ if (ret)
+ return ret;
+
+ /*
+ * Register a single provider for bindings without child nodes.
+ */
+ if (usb_np == qmp->dev->of_node)
+ return devm_of_clk_add_hw_provider(qmp->dev, qmp_combo_clk_hw_get, qmp);
+
+ /*
+ * Register multiple providers for legacy bindings with child nodes.
+ */
+ ret = of_clk_add_hw_provider(usb_np, of_clk_hw_simple_get,
+ &qmp->pipe_clk_fixed.hw);
if (ret)
return ret;
@@ -2632,124 +2499,293 @@ static int phy_dp_clks_register(struct qcom_qmp *qmp, struct qmp_phy *qphy,
* Roll a devm action because the clock provider is the child node, but
* the child node is not actually a device.
*/
- return devm_add_action_or_reset(qmp->dev, phy_clk_release_provider, np);
+ ret = devm_add_action_or_reset(qmp->dev, phy_clk_release_provider, usb_np);
+ if (ret)
+ return ret;
+
+ ret = of_clk_add_hw_provider(dp_np, qmp_dp_clks_hw_get, qmp);
+ if (ret)
+ return ret;
+
+ return devm_add_action_or_reset(qmp->dev, phy_clk_release_provider, dp_np);
}
-static const struct phy_ops qmp_combo_usb_ops = {
- .init = qmp_combo_enable,
- .exit = qmp_combo_disable,
- .set_mode = qmp_combo_set_mode,
- .owner = THIS_MODULE,
-};
-
-static const struct phy_ops qmp_combo_dp_ops = {
- .init = qmp_combo_init,
- .configure = qcom_qmp_dp_phy_configure,
- .power_on = qmp_combo_power_on,
- .calibrate = qcom_qmp_dp_phy_calibrate,
- .power_off = qmp_combo_power_off,
- .exit = qmp_combo_exit,
- .set_mode = qmp_combo_set_mode,
- .owner = THIS_MODULE,
-};
-
-static int qmp_combo_create(struct device *dev, struct device_node *np, int id,
- void __iomem *serdes, const struct qmp_phy_cfg *cfg)
+static int qmp_combo_parse_dt_lecacy_dp(struct qmp_combo *qmp, struct device_node *np)
{
- struct qcom_qmp *qmp = dev_get_drvdata(dev);
- struct phy *generic_phy;
- struct qmp_phy *qphy;
- const struct phy_ops *ops;
- int ret;
-
- qphy = devm_kzalloc(dev, sizeof(*qphy), GFP_KERNEL);
- if (!qphy)
- return -ENOMEM;
-
- qphy->cfg = cfg;
- qphy->serdes = serdes;
- /*
- * Get memory resources for each phy lane:
- * Resources are indexed as: tx -> 0; rx -> 1; pcs -> 2.
- * For dual lane PHYs: tx2 -> 3, rx2 -> 4, pcs_misc (optional) -> 5
- * For single lane PHYs: pcs_misc (optional) -> 3.
- */
- qphy->tx = devm_of_iomap(dev, np, 0, NULL);
- if (IS_ERR(qphy->tx))
- return PTR_ERR(qphy->tx);
-
- qphy->rx = devm_of_iomap(dev, np, 1, NULL);
- if (IS_ERR(qphy->rx))
- return PTR_ERR(qphy->rx);
-
- qphy->pcs = devm_of_iomap(dev, np, 2, NULL);
- if (IS_ERR(qphy->pcs))
- return PTR_ERR(qphy->pcs);
-
- if (cfg->pcs_usb_offset)
- qphy->pcs_usb = qphy->pcs + cfg->pcs_usb_offset;
-
- if (cfg->lanes >= 2) {
- qphy->tx2 = devm_of_iomap(dev, np, 3, NULL);
- if (IS_ERR(qphy->tx2))
- return PTR_ERR(qphy->tx2);
-
- qphy->rx2 = devm_of_iomap(dev, np, 4, NULL);
- if (IS_ERR(qphy->rx2))
- return PTR_ERR(qphy->rx2);
-
- qphy->pcs_misc = devm_of_iomap(dev, np, 5, NULL);
- } else {
- qphy->pcs_misc = devm_of_iomap(dev, np, 3, NULL);
- }
-
- if (IS_ERR(qphy->pcs_misc)) {
- dev_vdbg(dev, "PHY pcs_misc-reg not used\n");
- qphy->pcs_misc = NULL;
- }
+ struct device *dev = qmp->dev;
/*
- * Get PHY's Pipe clock, if any. USB3 and PCIe are PIPE3
- * based phys, so they essentially have pipe clock. So,
- * we return error in case phy is USB3 or PIPE type.
- * Otherwise, we initialize pipe clock to NULL for
- * all phys that don't need this.
+ * Get memory resources from the DP child node:
+ * Resources are indexed as: tx -> 0; rx -> 1; pcs -> 2;
+ * tx2 -> 3; rx2 -> 4
+ *
+ * Note that only tx/tx2 and pcs (dp_phy) are used by the DP
+ * implementation.
*/
- qphy->pipe_clk = devm_get_clk_from_child(dev, np, NULL);
- if (IS_ERR(qphy->pipe_clk)) {
- if (cfg->type == PHY_TYPE_USB3)
- return dev_err_probe(dev, PTR_ERR(qphy->pipe_clk),
- "failed to get lane%d pipe_clk\n",
- id);
- qphy->pipe_clk = NULL;
- }
+ qmp->dp_tx = devm_of_iomap(dev, np, 0, NULL);
+ if (IS_ERR(qmp->dp_tx))
+ return PTR_ERR(qmp->dp_tx);
- if (cfg->type == PHY_TYPE_DP)
- ops = &qmp_combo_dp_ops;
- else
- ops = &qmp_combo_usb_ops;
+ qmp->dp_dp_phy = devm_of_iomap(dev, np, 2, NULL);
+ if (IS_ERR(qmp->dp_dp_phy))
+ return PTR_ERR(qmp->dp_dp_phy);
- generic_phy = devm_phy_create(dev, np, ops);
- if (IS_ERR(generic_phy)) {
- ret = PTR_ERR(generic_phy);
- dev_err(dev, "failed to create qphy %d\n", ret);
- return ret;
- }
-
- qphy->phy = generic_phy;
- qphy->qmp = qmp;
- qmp->phys[id] = qphy;
- phy_set_drvdata(generic_phy, qphy);
+ qmp->dp_tx2 = devm_of_iomap(dev, np, 3, NULL);
+ if (IS_ERR(qmp->dp_tx2))
+ return PTR_ERR(qmp->dp_tx2);
return 0;
}
+static int qmp_combo_parse_dt_lecacy_usb(struct qmp_combo *qmp, struct device_node *np)
+{
+ const struct qmp_phy_cfg *cfg = qmp->cfg;
+ struct device *dev = qmp->dev;
+
+ /*
+ * Get memory resources from the USB child node:
+ * Resources are indexed as: tx -> 0; rx -> 1; pcs -> 2;
+ * tx2 -> 3; rx2 -> 4; pcs_misc (optional) -> 5
+ */
+ qmp->tx = devm_of_iomap(dev, np, 0, NULL);
+ if (IS_ERR(qmp->tx))
+ return PTR_ERR(qmp->tx);
+
+ qmp->rx = devm_of_iomap(dev, np, 1, NULL);
+ if (IS_ERR(qmp->rx))
+ return PTR_ERR(qmp->rx);
+
+ qmp->pcs = devm_of_iomap(dev, np, 2, NULL);
+ if (IS_ERR(qmp->pcs))
+ return PTR_ERR(qmp->pcs);
+
+ if (cfg->pcs_usb_offset)
+ qmp->pcs_usb = qmp->pcs + cfg->pcs_usb_offset;
+
+ qmp->tx2 = devm_of_iomap(dev, np, 3, NULL);
+ if (IS_ERR(qmp->tx2))
+ return PTR_ERR(qmp->tx2);
+
+ qmp->rx2 = devm_of_iomap(dev, np, 4, NULL);
+ if (IS_ERR(qmp->rx2))
+ return PTR_ERR(qmp->rx2);
+
+ qmp->pcs_misc = devm_of_iomap(dev, np, 5, NULL);
+ if (IS_ERR(qmp->pcs_misc)) {
+ dev_vdbg(dev, "PHY pcs_misc-reg not used\n");
+ qmp->pcs_misc = NULL;
+ }
+
+ qmp->pipe_clk = devm_get_clk_from_child(dev, np, NULL);
+ if (IS_ERR(qmp->pipe_clk)) {
+ return dev_err_probe(dev, PTR_ERR(qmp->pipe_clk),
+ "failed to get pipe clock\n");
+ }
+
+ return 0;
+}
+
+static int qmp_combo_parse_dt_legacy(struct qmp_combo *qmp, struct device_node *usb_np,
+ struct device_node *dp_np)
+{
+ struct platform_device *pdev = to_platform_device(qmp->dev);
+ int ret;
+
+ qmp->serdes = devm_platform_ioremap_resource(pdev, 0);
+ if (IS_ERR(qmp->serdes))
+ return PTR_ERR(qmp->serdes);
+
+ qmp->com = devm_platform_ioremap_resource(pdev, 1);
+ if (IS_ERR(qmp->com))
+ return PTR_ERR(qmp->com);
+
+ qmp->dp_serdes = devm_platform_ioremap_resource(pdev, 2);
+ if (IS_ERR(qmp->dp_serdes))
+ return PTR_ERR(qmp->dp_serdes);
+
+ ret = qmp_combo_parse_dt_lecacy_usb(qmp, usb_np);
+ if (ret)
+ return ret;
+
+ ret = qmp_combo_parse_dt_lecacy_dp(qmp, dp_np);
+ if (ret)
+ return ret;
+
+ return 0;
+}
+
+static int qmp_combo_parse_dt(struct qmp_combo *qmp)
+{
+ struct platform_device *pdev = to_platform_device(qmp->dev);
+ const struct qmp_phy_cfg *cfg = qmp->cfg;
+ const struct qmp_combo_offsets *offs = cfg->offsets;
+ struct device *dev = qmp->dev;
+ void __iomem *base;
+
+ if (!offs)
+ return -EINVAL;
+
+ base = devm_platform_ioremap_resource(pdev, 0);
+ if (IS_ERR(base))
+ return PTR_ERR(base);
+
+ qmp->com = base + offs->com;
+ qmp->tx = base + offs->txa;
+ qmp->rx = base + offs->rxa;
+ qmp->tx2 = base + offs->txb;
+ qmp->rx2 = base + offs->rxb;
+
+ qmp->serdes = base + offs->usb3_serdes;
+ qmp->pcs_misc = base + offs->usb3_pcs_misc;
+ qmp->pcs = base + offs->usb3_pcs;
+ qmp->pcs_usb = base + offs->usb3_pcs_usb;
+
+ qmp->dp_serdes = base + offs->dp_serdes;
+ qmp->dp_tx = base + offs->txa;
+ qmp->dp_tx2 = base + offs->txb;
+ qmp->dp_dp_phy = base + offs->dp_dp_phy;
+
+ qmp->pipe_clk = devm_clk_get(dev, "usb3_pipe");
+ if (IS_ERR(qmp->pipe_clk)) {
+ return dev_err_probe(dev, PTR_ERR(qmp->pipe_clk),
+ "failed to get usb3_pipe clock\n");
+ }
+
+ return 0;
+}
+
+static struct phy *qmp_combo_phy_xlate(struct device *dev, struct of_phandle_args *args)
+{
+ struct qmp_combo *qmp = dev_get_drvdata(dev);
+
+ if (args->args_count == 0)
+ return ERR_PTR(-EINVAL);
+
+ switch (args->args[0]) {
+ case QMP_USB43DP_USB3_PHY:
+ return qmp->usb_phy;
+ case QMP_USB43DP_DP_PHY:
+ return qmp->dp_phy;
+ }
+
+ return ERR_PTR(-EINVAL);
+}
+
+static int qmp_combo_probe(struct platform_device *pdev)
+{
+ struct qmp_combo *qmp;
+ struct device *dev = &pdev->dev;
+ struct device_node *dp_np, *usb_np;
+ struct phy_provider *phy_provider;
+ int ret;
+
+ qmp = devm_kzalloc(dev, sizeof(*qmp), GFP_KERNEL);
+ if (!qmp)
+ return -ENOMEM;
+
+ qmp->dev = dev;
+
+ qmp->cfg = of_device_get_match_data(dev);
+ if (!qmp->cfg)
+ return -EINVAL;
+
+ mutex_init(&qmp->phy_mutex);
+
+ ret = qmp_combo_clk_init(qmp);
+ if (ret)
+ return ret;
+
+ ret = qmp_combo_reset_init(qmp);
+ if (ret)
+ return ret;
+
+ ret = qmp_combo_vreg_init(qmp);
+ if (ret)
+ return ret;
+
+ /* Check for legacy binding with child nodes. */
+ usb_np = of_get_child_by_name(dev->of_node, "usb3-phy");
+ if (usb_np) {
+ dp_np = of_get_child_by_name(dev->of_node, "dp-phy");
+ if (!dp_np) {
+ of_node_put(usb_np);
+ return -EINVAL;
+ }
+
+ ret = qmp_combo_parse_dt_legacy(qmp, usb_np, dp_np);
+ } else {
+ usb_np = of_node_get(dev->of_node);
+ dp_np = of_node_get(dev->of_node);
+
+ ret = qmp_combo_parse_dt(qmp);
+ }
+ if (ret)
+ goto err_node_put;
+
+ pm_runtime_set_active(dev);
+ ret = devm_pm_runtime_enable(dev);
+ if (ret)
+ goto err_node_put;
+ /*
+ * Prevent runtime pm from being ON by default. Users can enable
+ * it using power/control in sysfs.
+ */
+ pm_runtime_forbid(dev);
+
+ ret = qmp_combo_register_clocks(qmp, usb_np, dp_np);
+ if (ret)
+ goto err_node_put;
+
+ qmp->usb_phy = devm_phy_create(dev, usb_np, &qmp_combo_usb_phy_ops);
+ if (IS_ERR(qmp->usb_phy)) {
+ ret = PTR_ERR(qmp->usb_phy);
+ dev_err(dev, "failed to create USB PHY: %d\n", ret);
+ goto err_node_put;
+ }
+
+ phy_set_drvdata(qmp->usb_phy, qmp);
+
+ qmp->dp_phy = devm_phy_create(dev, dp_np, &qmp_combo_dp_phy_ops);
+ if (IS_ERR(qmp->dp_phy)) {
+ ret = PTR_ERR(qmp->dp_phy);
+ dev_err(dev, "failed to create DP PHY: %d\n", ret);
+ goto err_node_put;
+ }
+
+ phy_set_drvdata(qmp->dp_phy, qmp);
+
+ dev_set_drvdata(dev, qmp);
+
+ if (usb_np == dev->of_node)
+ phy_provider = devm_of_phy_provider_register(dev, qmp_combo_phy_xlate);
+ else
+ phy_provider = devm_of_phy_provider_register(dev, of_phy_simple_xlate);
+
+ of_node_put(usb_np);
+ of_node_put(dp_np);
+
+ return PTR_ERR_OR_ZERO(phy_provider);
+
+err_node_put:
+ of_node_put(usb_np);
+ of_node_put(dp_np);
+ return ret;
+}
+
static const struct of_device_id qmp_combo_of_match_table[] = {
{
.compatible = "qcom,sc7180-qmp-usb3-dp-phy",
.data = &sc7180_usb3dpphy_cfg,
},
{
+ .compatible = "qcom,sc8180x-qmp-usb3-dp-phy",
+ .data = &sc8180x_usb3dpphy_cfg,
+ },
+ {
+ .compatible = "qcom,sc8280xp-qmp-usb43dp-phy",
+ .data = &sc8280xp_usb43dpphy_cfg,
+ },
+ {
.compatible = "qcom,sdm845-qmp-usb3-dp-phy",
.data = &sdm845_usb3dpphy_cfg,
},
@@ -2757,161 +2793,10 @@ static const struct of_device_id qmp_combo_of_match_table[] = {
.compatible = "qcom,sm8250-qmp-usb3-dp-phy",
.data = &sm8250_usb3dpphy_cfg,
},
- {
- .compatible = "qcom,sc8180x-qmp-usb3-dp-phy",
- .data = &sc8180x_usb3dpphy_cfg,
- },
- {
- .compatible = "qcom,sc8280xp-qmp-usb43dp-phy",
- .data = &sc8280xp_usb43dpphy_combo_cfg,
- },
{ }
};
MODULE_DEVICE_TABLE(of, qmp_combo_of_match_table);
-static const struct dev_pm_ops qmp_combo_pm_ops = {
- SET_RUNTIME_PM_OPS(qmp_combo_runtime_suspend,
- qmp_combo_runtime_resume, NULL)
-};
-
-static int qmp_combo_probe(struct platform_device *pdev)
-{
- struct qcom_qmp *qmp;
- struct device *dev = &pdev->dev;
- struct device_node *child;
- struct phy_provider *phy_provider;
- void __iomem *serdes;
- void __iomem *usb_serdes;
- void __iomem *dp_serdes = NULL;
- const struct qmp_phy_combo_cfg *combo_cfg = NULL;
- const struct qmp_phy_cfg *cfg = NULL;
- const struct qmp_phy_cfg *usb_cfg = NULL;
- const struct qmp_phy_cfg *dp_cfg = NULL;
- int num, id, expected_phys;
- int ret;
-
- qmp = devm_kzalloc(dev, sizeof(*qmp), GFP_KERNEL);
- if (!qmp)
- return -ENOMEM;
-
- qmp->dev = dev;
- dev_set_drvdata(dev, qmp);
-
- /* Get the specific init parameters of QMP phy */
- combo_cfg = of_device_get_match_data(dev);
- if (!combo_cfg)
- return -EINVAL;
-
- usb_cfg = combo_cfg->usb_cfg;
- cfg = usb_cfg; /* Setup clks and regulators */
-
- /* per PHY serdes; usually located at base address */
- usb_serdes = serdes = devm_platform_ioremap_resource(pdev, 0);
- if (IS_ERR(serdes))
- return PTR_ERR(serdes);
-
- qmp->dp_com = devm_platform_ioremap_resource(pdev, 1);
- if (IS_ERR(qmp->dp_com))
- return PTR_ERR(qmp->dp_com);
-
- /* Only two serdes for combo PHY */
- dp_serdes = devm_platform_ioremap_resource(pdev, 2);
- if (IS_ERR(dp_serdes))
- return PTR_ERR(dp_serdes);
-
- dp_cfg = combo_cfg->dp_cfg;
- expected_phys = 2;
-
- mutex_init(&qmp->phy_mutex);
-
- ret = qmp_combo_clk_init(dev, cfg);
- if (ret)
- return ret;
-
- ret = qmp_combo_reset_init(dev, cfg);
- if (ret)
- return ret;
-
- ret = qmp_combo_vreg_init(dev, cfg);
- if (ret)
- return dev_err_probe(dev, ret,
- "failed to get regulator supplies\n");
-
- num = of_get_available_child_count(dev->of_node);
- /* do we have a rogue child node ? */
- if (num > expected_phys)
- return -EINVAL;
-
- qmp->phys = devm_kcalloc(dev, num, sizeof(*qmp->phys), GFP_KERNEL);
- if (!qmp->phys)
- return -ENOMEM;
-
- pm_runtime_set_active(dev);
- ret = devm_pm_runtime_enable(dev);
- if (ret)
- return ret;
- /*
- * Prevent runtime pm from being ON by default. Users can enable
- * it using power/control in sysfs.
- */
- pm_runtime_forbid(dev);
-
- id = 0;
- for_each_available_child_of_node(dev->of_node, child) {
- if (of_node_name_eq(child, "dp-phy")) {
- cfg = dp_cfg;
- serdes = dp_serdes;
-
- /* Create per-lane phy */
- ret = qmp_combo_create(dev, child, id, serdes, cfg);
- if (ret) {
- dev_err(dev, "failed to create lane%d phy, %d\n",
- id, ret);
- goto err_node_put;
- }
-
- ret = phy_dp_clks_register(qmp, qmp->phys[id], child);
- if (ret) {
- dev_err(qmp->dev,
- "failed to register DP clock source\n");
- goto err_node_put;
- }
- } else if (of_node_name_eq(child, "usb3-phy")) {
- cfg = usb_cfg;
- serdes = usb_serdes;
-
- /* Create per-lane phy */
- ret = qmp_combo_create(dev, child, id, serdes, cfg);
- if (ret) {
- dev_err(dev, "failed to create lane%d phy, %d\n",
- id, ret);
- goto err_node_put;
- }
-
- /*
- * Register the pipe clock provided by phy.
- * See function description to see details of this pipe clock.
- */
- ret = phy_pipe_clk_register(qmp, child);
- if (ret) {
- dev_err(qmp->dev,
- "failed to register pipe clock source\n");
- goto err_node_put;
- }
- }
-
- id++;
- }
-
- phy_provider = devm_of_phy_provider_register(dev, of_phy_simple_xlate);
-
- return PTR_ERR_OR_ZERO(phy_provider);
-
-err_node_put:
- of_node_put(child);
- return ret;
-}
-
static struct platform_driver qmp_combo_driver = {
.probe = qmp_combo_probe,
.driver = {
diff --git a/drivers/phy/qualcomm/phy-qcom-qmp-pcie-msm8996.c b/drivers/phy/qualcomm/phy-qcom-qmp-pcie-msm8996.c
index 461f0b5..a088477 100644
--- a/drivers/phy/qualcomm/phy-qcom-qmp-pcie-msm8996.c
+++ b/drivers/phy/qualcomm/phy-qcom-qmp-pcie-msm8996.c
@@ -20,8 +20,6 @@
#include <linux/reset.h>
#include <linux/slab.h>
-#include <dt-bindings/phy/phy.h>
-
#include "phy-qcom-qmp.h"
/* QPHY_SW_RESET bit */
@@ -35,23 +33,17 @@
#define PLL_READY_GATE_EN BIT(3)
/* QPHY_PCS_STATUS bit */
#define PHYSTATUS BIT(6)
-#define PHYSTATUS_4_20 BIT(7)
/* QPHY_COM_PCS_READY_STATUS bit */
#define PCS_READY BIT(0)
#define PHY_INIT_COMPLETE_TIMEOUT 10000
#define POWER_DOWN_DELAY_US_MIN 10
-#define POWER_DOWN_DELAY_US_MAX 11
+#define POWER_DOWN_DELAY_US_MAX 20
struct qmp_phy_init_tbl {
unsigned int offset;
unsigned int val;
/*
- * register part of layout ?
- * if yes, then offset gives index in the reg-layout
- */
- bool in_layout;
- /*
* mask of lanes for which this register is written
* for cases when second lane needs different values
*/
@@ -65,14 +57,6 @@ struct qmp_phy_init_tbl {
.lane_mask = 0xff, \
}
-#define QMP_PHY_INIT_CFG_L(o, v) \
- { \
- .offset = o, \
- .val = v, \
- .in_layout = true, \
- .lane_mask = 0xff, \
- }
-
#define QMP_PHY_INIT_CFG_LANE(o, v, l) \
{ \
.offset = o, \
@@ -91,7 +75,6 @@ enum qphy_reg_layout {
QPHY_SW_RESET,
QPHY_START_CTRL,
QPHY_PCS_STATUS,
- QPHY_PCS_POWER_DOWN_CONTROL,
/* Keep last to ensure regs_layout arrays are properly initialized */
QPHY_LAYOUT_SIZE
};
@@ -211,18 +194,6 @@ struct qmp_phy_cfg {
/* array of registers with different offsets */
const unsigned int *regs;
-
- unsigned int start_ctrl;
- unsigned int pwrdn_ctrl;
- unsigned int mask_com_pcs_ready;
- /* bit offset of PHYSTATUS in QPHY_PCS_STATUS register */
- unsigned int phy_status;
-
- /* true, if PHY needs delay after POWER_DOWN */
- bool has_pwrdn_delay;
- /* power_down delay in usec */
- int pwrdn_delay_min;
- int pwrdn_delay_max;
};
/**
@@ -335,19 +306,9 @@ static const struct qmp_phy_cfg msm8996_pciephy_cfg = {
.vreg_list = qmp_phy_vreg_l,
.num_vregs = ARRAY_SIZE(qmp_phy_vreg_l),
.regs = pciephy_regs_layout,
-
- .start_ctrl = PCS_START | PLL_READY_GATE_EN,
- .pwrdn_ctrl = SW_PWRDN | REFCLK_DRV_DSBL,
- .mask_com_pcs_ready = PCS_READY,
- .phy_status = PHYSTATUS,
-
- .has_pwrdn_delay = true,
- .pwrdn_delay_min = POWER_DOWN_DELAY_US_MIN,
- .pwrdn_delay_max = POWER_DOWN_DELAY_US_MAX,
};
static void qmp_pcie_msm8996_configure_lane(void __iomem *base,
- const unsigned int *regs,
const struct qmp_phy_init_tbl tbl[],
int num,
u8 lane_mask)
@@ -362,19 +323,15 @@ static void qmp_pcie_msm8996_configure_lane(void __iomem *base,
if (!(t->lane_mask & lane_mask))
continue;
- if (t->in_layout)
- writel(t->val, base + regs[t->offset]);
- else
- writel(t->val, base + t->offset);
+ writel(t->val, base + t->offset);
}
}
static void qmp_pcie_msm8996_configure(void __iomem *base,
- const unsigned int *regs,
const struct qmp_phy_init_tbl tbl[],
int num)
{
- qmp_pcie_msm8996_configure_lane(base, regs, tbl, num, 0xff);
+ qmp_pcie_msm8996_configure_lane(base, tbl, num, 0xff);
}
static int qmp_pcie_msm8996_serdes_init(struct qmp_phy *qphy)
@@ -385,19 +342,17 @@ static int qmp_pcie_msm8996_serdes_init(struct qmp_phy *qphy)
const struct qmp_phy_init_tbl *serdes_tbl = cfg->serdes_tbl;
int serdes_tbl_num = cfg->serdes_tbl_num;
void __iomem *status;
- unsigned int mask, val;
+ unsigned int val;
int ret;
- qmp_pcie_msm8996_configure(serdes, cfg->regs, serdes_tbl, serdes_tbl_num);
+ qmp_pcie_msm8996_configure(serdes, serdes_tbl, serdes_tbl_num);
qphy_clrbits(serdes, cfg->regs[QPHY_COM_SW_RESET], SW_RESET);
qphy_setbits(serdes, cfg->regs[QPHY_COM_START_CONTROL],
SERDES_START | PCS_START);
status = serdes + cfg->regs[QPHY_COM_PCS_READY_STATUS];
- mask = cfg->mask_com_pcs_ready;
-
- ret = readl_poll_timeout(status, val, (val & mask), 10,
+ ret = readl_poll_timeout(status, val, (val & PCS_READY), 200,
PHY_INIT_COMPLETE_TIMEOUT);
if (ret) {
dev_err(qmp->dev,
@@ -421,7 +376,6 @@ static int qmp_pcie_msm8996_com_init(struct qmp_phy *qphy)
return 0;
}
- /* turn on regulator supplies */
ret = regulator_bulk_enable(cfg->num_vregs, qmp->vregs);
if (ret) {
dev_err(qmp->dev, "failed to enable regulators, err=%d\n", ret);
@@ -514,7 +468,7 @@ static int qmp_pcie_msm8996_power_on(struct phy *phy)
void __iomem *rx = qphy->rx;
void __iomem *pcs = qphy->pcs;
void __iomem *status;
- unsigned int mask, val, ready;
+ unsigned int val;
int ret;
qmp_pcie_msm8996_serdes_init(qphy);
@@ -533,34 +487,28 @@ static int qmp_pcie_msm8996_power_on(struct phy *phy)
}
/* Tx, Rx, and PCS configurations */
- qmp_pcie_msm8996_configure_lane(tx, cfg->regs, cfg->tx_tbl,
- cfg->tx_tbl_num, 1);
-
- qmp_pcie_msm8996_configure_lane(rx, cfg->regs, cfg->rx_tbl,
- cfg->rx_tbl_num, 1);
-
- qmp_pcie_msm8996_configure(pcs, cfg->regs, cfg->pcs_tbl, cfg->pcs_tbl_num);
+ qmp_pcie_msm8996_configure_lane(tx, cfg->tx_tbl, cfg->tx_tbl_num, 1);
+ qmp_pcie_msm8996_configure_lane(rx, cfg->rx_tbl, cfg->rx_tbl_num, 1);
+ qmp_pcie_msm8996_configure(pcs, cfg->pcs_tbl, cfg->pcs_tbl_num);
/*
* Pull out PHY from POWER DOWN state.
* This is active low enable signal to power-down PHY.
*/
- qphy_setbits(pcs, QPHY_V2_PCS_POWER_DOWN_CONTROL, cfg->pwrdn_ctrl);
+ qphy_setbits(pcs, QPHY_V2_PCS_POWER_DOWN_CONTROL,
+ SW_PWRDN | REFCLK_DRV_DSBL);
- if (cfg->has_pwrdn_delay)
- usleep_range(cfg->pwrdn_delay_min, cfg->pwrdn_delay_max);
+ usleep_range(POWER_DOWN_DELAY_US_MIN, POWER_DOWN_DELAY_US_MAX);
/* Pull PHY out of reset state */
qphy_clrbits(pcs, cfg->regs[QPHY_SW_RESET], SW_RESET);
/* start SerDes and Phy-Coding-Sublayer */
- qphy_setbits(pcs, cfg->regs[QPHY_START_CTRL], cfg->start_ctrl);
+ qphy_setbits(pcs, cfg->regs[QPHY_START_CTRL],
+ PCS_START | PLL_READY_GATE_EN);
status = pcs + cfg->regs[QPHY_PCS_STATUS];
- mask = cfg->phy_status;
- ready = 0;
-
- ret = readl_poll_timeout(status, val, (val & mask) == ready, 10,
+ ret = readl_poll_timeout(status, val, !(val & PHYSTATUS), 200,
PHY_INIT_COMPLETE_TIMEOUT);
if (ret) {
dev_err(qmp->dev, "phy initialization timed-out\n");
@@ -588,16 +536,12 @@ static int qmp_pcie_msm8996_power_off(struct phy *phy)
qphy_setbits(qphy->pcs, cfg->regs[QPHY_SW_RESET], SW_RESET);
/* stop SerDes and Phy-Coding-Sublayer */
- qphy_clrbits(qphy->pcs, cfg->regs[QPHY_START_CTRL], cfg->start_ctrl);
+ qphy_clrbits(qphy->pcs, cfg->regs[QPHY_START_CTRL],
+ SERDES_START | PCS_START);
/* Put PHY into POWER DOWN state: active low */
- if (cfg->regs[QPHY_PCS_POWER_DOWN_CONTROL]) {
- qphy_clrbits(qphy->pcs, cfg->regs[QPHY_PCS_POWER_DOWN_CONTROL],
- cfg->pwrdn_ctrl);
- } else {
- qphy_clrbits(qphy->pcs, QPHY_V2_PCS_POWER_DOWN_CONTROL,
- cfg->pwrdn_ctrl);
- }
+ qphy_clrbits(qphy->pcs, QPHY_V2_PCS_POWER_DOWN_CONTROL,
+ SW_PWRDN | REFCLK_DRV_DSBL);
return 0;
}
@@ -777,7 +721,7 @@ static int qmp_pcie_msm8996_create(struct device *dev, struct device_node *np, i
qphy->cfg = cfg;
qphy->serdes = serdes;
/*
- * Get memory resources for each phy lane:
+ * Get memory resources for each PHY:
* Resources are indexed as: tx -> 0; rx -> 1; pcs -> 2.
*/
qphy->tx = devm_of_iomap(dev, np, 0, NULL);
@@ -851,12 +795,10 @@ static int qmp_pcie_msm8996_probe(struct platform_device *pdev)
qmp->dev = dev;
dev_set_drvdata(dev, qmp);
- /* Get the specific init parameters of QMP phy */
cfg = of_device_get_match_data(dev);
if (!cfg)
return -EINVAL;
- /* per PHY serdes; usually located at base address */
serdes = devm_platform_ioremap_resource(pdev, 0);
if (IS_ERR(serdes))
return PTR_ERR(serdes);
@@ -875,8 +817,7 @@ static int qmp_pcie_msm8996_probe(struct platform_device *pdev)
ret = qmp_pcie_msm8996_vreg_init(dev, cfg);
if (ret)
- return dev_err_probe(dev, ret,
- "failed to get regulator supplies\n");
+ return ret;
num = of_get_available_child_count(dev->of_node);
/* do we have a rogue child node ? */
diff --git a/drivers/phy/qualcomm/phy-qcom-qmp-pcie.c b/drivers/phy/qualcomm/phy-qcom-qmp-pcie.c
index 5be5348..1b136a8 100644
--- a/drivers/phy/qualcomm/phy-qcom-qmp-pcie.c
+++ b/drivers/phy/qualcomm/phy-qcom-qmp-pcie.c
@@ -10,18 +10,19 @@
#include <linux/io.h>
#include <linux/iopoll.h>
#include <linux/kernel.h>
+#include <linux/mfd/syscon.h>
#include <linux/module.h>
#include <linux/of.h>
#include <linux/of_device.h>
#include <linux/of_address.h>
+#include <linux/phy/pcie.h>
#include <linux/phy/phy.h>
#include <linux/platform_device.h>
+#include <linux/regmap.h>
#include <linux/regulator/consumer.h>
#include <linux/reset.h>
#include <linux/slab.h>
-#include <dt-bindings/phy/phy.h>
-
#include "phy-qcom-qmp.h"
/* QPHY_SW_RESET bit */
@@ -42,11 +43,6 @@ struct qmp_phy_init_tbl {
unsigned int offset;
unsigned int val;
/*
- * register part of layout ?
- * if yes, then offset gives index in the reg-layout
- */
- bool in_layout;
- /*
* mask of lanes for which this register is written
* for cases when second lane needs different values
*/
@@ -60,14 +56,6 @@ struct qmp_phy_init_tbl {
.lane_mask = 0xff, \
}
-#define QMP_PHY_INIT_CFG_L(o, v) \
- { \
- .offset = o, \
- .val = v, \
- .in_layout = true, \
- .lane_mask = 0xff, \
- }
-
#define QMP_PHY_INIT_CFG_LANE(o, v, l) \
{ \
.offset = o, \
@@ -77,11 +65,6 @@ struct qmp_phy_init_tbl {
/* set of registers with offsets different per-PHY */
enum qphy_reg_layout {
- /* Common block control registers */
- QPHY_COM_SW_RESET,
- QPHY_COM_POWER_DOWN_CONTROL,
- QPHY_COM_START_CONTROL,
- QPHY_COM_PCS_READY_STATUS,
/* PCS registers */
QPHY_SW_RESET,
QPHY_START_CTRL,
@@ -99,25 +82,24 @@ static const unsigned int ipq_pciephy_gen3_regs_layout[QPHY_LAYOUT_SIZE] = {
};
static const unsigned int pciephy_regs_layout[QPHY_LAYOUT_SIZE] = {
- [QPHY_COM_SW_RESET] = 0x400,
- [QPHY_COM_POWER_DOWN_CONTROL] = 0x404,
- [QPHY_COM_START_CONTROL] = 0x408,
- [QPHY_COM_PCS_READY_STATUS] = 0x448,
[QPHY_SW_RESET] = 0x00,
[QPHY_START_CTRL] = 0x08,
[QPHY_PCS_STATUS] = 0x174,
+ [QPHY_PCS_POWER_DOWN_CONTROL] = 0x04,
};
static const unsigned int sdm845_qmp_pciephy_regs_layout[QPHY_LAYOUT_SIZE] = {
[QPHY_SW_RESET] = 0x00,
[QPHY_START_CTRL] = 0x08,
[QPHY_PCS_STATUS] = 0x174,
+ [QPHY_PCS_POWER_DOWN_CONTROL] = 0x04,
};
static const unsigned int sdm845_qhp_pciephy_regs_layout[QPHY_LAYOUT_SIZE] = {
[QPHY_SW_RESET] = 0x00,
[QPHY_START_CTRL] = 0x08,
[QPHY_PCS_STATUS] = 0x2ac,
+ [QPHY_PCS_POWER_DOWN_CONTROL] = 0x04,
};
static const unsigned int sm8250_pcie_regs_layout[QPHY_LAYOUT_SIZE] = {
@@ -393,8 +375,6 @@ static const struct qmp_phy_init_tbl ipq8074_pcie_pcs_tbl[] = {
QMP_PHY_INIT_CFG(QPHY_V2_PCS_RX_SIGDET_LVL, 0x99),
QMP_PHY_INIT_CFG(QPHY_V2_PCS_TXDEEMPH_M6DB_V0, 0x15),
QMP_PHY_INIT_CFG(QPHY_V2_PCS_TXDEEMPH_M3P5DB_V0, 0xe),
- QMP_PHY_INIT_CFG_L(QPHY_SW_RESET, 0x0),
- QMP_PHY_INIT_CFG_L(QPHY_START_CTRL, 0x3),
};
static const struct qmp_phy_init_tbl ipq8074_pcie_gen3_serdes_tbl[] = {
@@ -505,6 +485,13 @@ static const struct qmp_phy_init_tbl ipq8074_pcie_gen3_pcs_tbl[] = {
QMP_PHY_INIT_CFG(QPHY_V4_PCS_FLL_CNTRL1, 0x01),
QMP_PHY_INIT_CFG(QPHY_V4_PCS_P2U3_WAKEUP_DLY_TIME_AUXCLK_H, 0x0),
QMP_PHY_INIT_CFG(QPHY_V4_PCS_P2U3_WAKEUP_DLY_TIME_AUXCLK_L, 0x1),
+ QMP_PHY_INIT_CFG(QPHY_V4_PCS_G12S1_TXDEEMPH_M3P5DB, 0x10),
+ QMP_PHY_INIT_CFG(QPHY_V4_PCS_RX_DCC_CAL_CONFIG, 0x01),
+ QMP_PHY_INIT_CFG(QPHY_V4_PCS_RX_SIGDET_LVL, 0xaa),
+ QMP_PHY_INIT_CFG(QPHY_V4_PCS_REFGEN_REQ_CONFIG1, 0x0d),
+};
+
+static const struct qmp_phy_init_tbl ipq8074_pcie_gen3_pcs_misc_tbl[] = {
QMP_PHY_INIT_CFG(QPHY_V4_PCS_PCIE_OSC_DTCT_ACTIONS, 0x0),
QMP_PHY_INIT_CFG(QPHY_V4_PCS_PCIE_L1P1_WAKEUP_DLY_TIME_AUXCLK_H, 0x00),
QMP_PHY_INIT_CFG(QPHY_V4_PCS_PCIE_L1P1_WAKEUP_DLY_TIME_AUXCLK_L, 0x01),
@@ -517,11 +504,7 @@ static const struct qmp_phy_init_tbl ipq8074_pcie_gen3_pcs_tbl[] = {
QMP_PHY_INIT_CFG(QPHY_V4_PCS_PCIE_OSC_DTCT_MODE2_CONFIG2, 0x50),
QMP_PHY_INIT_CFG(QPHY_V4_PCS_PCIE_OSC_DTCT_MODE2_CONFIG4, 0x1a),
QMP_PHY_INIT_CFG(QPHY_V4_PCS_PCIE_OSC_DTCT_MODE2_CONFIG5, 0x6),
- QMP_PHY_INIT_CFG(QPHY_V4_PCS_G12S1_TXDEEMPH_M3P5DB, 0x10),
QMP_PHY_INIT_CFG(QPHY_V4_PCS_PCIE_ENDPOINT_REFCLK_DRIVE, 0xc1),
- QMP_PHY_INIT_CFG(QPHY_V4_PCS_RX_DCC_CAL_CONFIG, 0x01),
- QMP_PHY_INIT_CFG(QPHY_V4_PCS_RX_SIGDET_LVL, 0xaa),
- QMP_PHY_INIT_CFG(QPHY_V4_PCS_REFGEN_REQ_CONFIG1, 0x0d),
};
static const struct qmp_phy_init_tbl sdm845_qmp_pcie_serdes_tbl[] = {
@@ -854,6 +837,147 @@ static const struct qmp_phy_init_tbl sc8180x_qmp_pcie_pcs_misc_tbl[] = {
QMP_PHY_INIT_CFG(QPHY_V4_PCS_PCIE_ENDPOINT_REFCLK_DRIVE, 0xc1),
};
+static const struct qmp_phy_init_tbl sc8280xp_qmp_pcie_serdes_tbl[] = {
+ QMP_PHY_INIT_CFG(QSERDES_V5_COM_SSC_EN_CENTER, 0x00),
+ QMP_PHY_INIT_CFG(QSERDES_V5_COM_SSC_PER1, 0x31),
+ QMP_PHY_INIT_CFG(QSERDES_V5_COM_SSC_PER2, 0x01),
+ QMP_PHY_INIT_CFG(QSERDES_V5_COM_SSC_STEP_SIZE1_MODE0, 0xde),
+ QMP_PHY_INIT_CFG(QSERDES_V5_COM_SSC_STEP_SIZE2_MODE0, 0x07),
+ QMP_PHY_INIT_CFG(QSERDES_V5_COM_SSC_STEP_SIZE1_MODE1, 0x4c),
+ QMP_PHY_INIT_CFG(QSERDES_V5_COM_SSC_STEP_SIZE2_MODE1, 0x06),
+ QMP_PHY_INIT_CFG(QSERDES_V5_COM_CLK_ENABLE1, 0x90),
+ QMP_PHY_INIT_CFG(QSERDES_V5_COM_PLL_IVCO, 0x0f),
+ QMP_PHY_INIT_CFG(QSERDES_V5_COM_CP_CTRL_MODE0, 0x06),
+ QMP_PHY_INIT_CFG(QSERDES_V5_COM_CP_CTRL_MODE1, 0x06),
+ QMP_PHY_INIT_CFG(QSERDES_V5_COM_PLL_RCTRL_MODE0, 0x16),
+ QMP_PHY_INIT_CFG(QSERDES_V5_COM_PLL_RCTRL_MODE1, 0x16),
+ QMP_PHY_INIT_CFG(QSERDES_V5_COM_PLL_CCTRL_MODE0, 0x36),
+ QMP_PHY_INIT_CFG(QSERDES_V5_COM_PLL_CCTRL_MODE1, 0x36),
+ QMP_PHY_INIT_CFG(QSERDES_V5_COM_SYSCLK_EN_SEL, 0x08),
+ QMP_PHY_INIT_CFG(QSERDES_V5_COM_LOCK_CMP_EN, 0x42),
+ QMP_PHY_INIT_CFG(QSERDES_V5_COM_LOCK_CMP1_MODE0, 0x0a),
+ QMP_PHY_INIT_CFG(QSERDES_V5_COM_LOCK_CMP2_MODE0, 0x1a),
+ QMP_PHY_INIT_CFG(QSERDES_V5_COM_LOCK_CMP1_MODE1, 0x14),
+ QMP_PHY_INIT_CFG(QSERDES_V5_COM_LOCK_CMP2_MODE1, 0x34),
+ QMP_PHY_INIT_CFG(QSERDES_V5_COM_DEC_START_MODE0, 0x82),
+ QMP_PHY_INIT_CFG(QSERDES_V5_COM_DEC_START_MODE1, 0x68),
+ QMP_PHY_INIT_CFG(QSERDES_V5_COM_DIV_FRAC_START1_MODE0, 0x55),
+ QMP_PHY_INIT_CFG(QSERDES_V5_COM_DIV_FRAC_START2_MODE0, 0x55),
+ QMP_PHY_INIT_CFG(QSERDES_V5_COM_DIV_FRAC_START3_MODE0, 0x03),
+ QMP_PHY_INIT_CFG(QSERDES_V5_COM_DIV_FRAC_START1_MODE1, 0xab),
+ QMP_PHY_INIT_CFG(QSERDES_V5_COM_DIV_FRAC_START2_MODE1, 0xaa),
+ QMP_PHY_INIT_CFG(QSERDES_V5_COM_DIV_FRAC_START3_MODE1, 0x02),
+ QMP_PHY_INIT_CFG(QSERDES_V5_COM_VCO_TUNE_MAP, 0x02),
+ QMP_PHY_INIT_CFG(QSERDES_V5_COM_VCO_TUNE1_MODE0, 0x24),
+ QMP_PHY_INIT_CFG(QSERDES_V5_COM_VCO_TUNE1_MODE1, 0xb4),
+ QMP_PHY_INIT_CFG(QSERDES_V5_COM_VCO_TUNE2_MODE1, 0x03),
+ QMP_PHY_INIT_CFG(QSERDES_V5_COM_CLK_SELECT, 0x34),
+ QMP_PHY_INIT_CFG(QSERDES_V5_COM_HSCLK_SEL, 0x01),
+ QMP_PHY_INIT_CFG(QSERDES_V5_COM_CORECLK_DIV_MODE1, 0x08),
+ QMP_PHY_INIT_CFG(QSERDES_V5_COM_BIN_VCOCAL_CMP_CODE1_MODE0, 0xb9),
+ QMP_PHY_INIT_CFG(QSERDES_V5_COM_BIN_VCOCAL_CMP_CODE2_MODE0, 0x1e),
+ QMP_PHY_INIT_CFG(QSERDES_V5_COM_BIN_VCOCAL_CMP_CODE1_MODE1, 0x94),
+ QMP_PHY_INIT_CFG(QSERDES_V5_COM_BIN_VCOCAL_CMP_CODE2_MODE1, 0x18),
+ QMP_PHY_INIT_CFG(QSERDES_V5_COM_BIN_VCOCAL_HSCLK_SEL, 0x11),
+};
+
+static const struct qmp_phy_init_tbl sc8280xp_qmp_gen3x1_pcie_rc_serdes_tbl[] = {
+ QMP_PHY_INIT_CFG(QSERDES_V5_COM_SYSCLK_BUF_ENABLE, 0x07),
+};
+
+static const struct qmp_phy_init_tbl sc8280xp_qmp_gen3x2_pcie_rc_serdes_tbl[] = {
+ QMP_PHY_INIT_CFG(QSERDES_V5_COM_BIAS_EN_CLKBUFLR_EN, 0x14),
+};
+
+static const struct qmp_phy_init_tbl sc8280xp_qmp_gen3x4_pcie_serdes_4ln_tbl[] = {
+ QMP_PHY_INIT_CFG(QSERDES_V5_COM_BIAS_EN_CLKBUFLR_EN, 0x1c),
+};
+
+static const struct qmp_phy_init_tbl sc8280xp_qmp_gen3x1_pcie_tx_tbl[] = {
+ QMP_PHY_INIT_CFG(QSERDES_V5_TX_PI_QEC_CTRL, 0x20),
+ QMP_PHY_INIT_CFG(QSERDES_V5_TX_LANE_MODE_1, 0x75),
+ QMP_PHY_INIT_CFG(QSERDES_V5_TX_LANE_MODE_4, 0x3f),
+ QMP_PHY_INIT_CFG(QSERDES_V5_TX_RES_CODE_LANE_OFFSET_TX, 0x1d),
+ QMP_PHY_INIT_CFG(QSERDES_V5_TX_RES_CODE_LANE_OFFSET_RX, 0x0c),
+};
+
+static const struct qmp_phy_init_tbl sc8280xp_qmp_gen3x1_pcie_rx_tbl[] = {
+ QMP_PHY_INIT_CFG(QSERDES_V5_RX_RX_MODE_00_LOW, 0x7f),
+ QMP_PHY_INIT_CFG(QSERDES_V5_RX_RX_MODE_00_HIGH, 0xff),
+ QMP_PHY_INIT_CFG(QSERDES_V5_RX_RX_MODE_00_HIGH2, 0xbf),
+ QMP_PHY_INIT_CFG(QSERDES_V5_RX_RX_MODE_00_HIGH3, 0x3f),
+ QMP_PHY_INIT_CFG(QSERDES_V5_RX_RX_MODE_00_HIGH4, 0xd8),
+ QMP_PHY_INIT_CFG(QSERDES_V5_RX_RX_MODE_01_LOW, 0xdc),
+ QMP_PHY_INIT_CFG(QSERDES_V5_RX_RX_MODE_01_HIGH, 0xdc),
+ QMP_PHY_INIT_CFG(QSERDES_V5_RX_RX_MODE_01_HIGH2, 0x5c),
+ QMP_PHY_INIT_CFG(QSERDES_V5_RX_RX_MODE_01_HIGH3, 0x34),
+ QMP_PHY_INIT_CFG(QSERDES_V5_RX_RX_MODE_01_HIGH4, 0xa6),
+ QMP_PHY_INIT_CFG(QSERDES_V5_RX_TX_ADAPT_POST_THRESH, 0xf0),
+ QMP_PHY_INIT_CFG(QSERDES_V5_RX_RX_MODE_10_HIGH3, 0x34),
+ QMP_PHY_INIT_CFG(QSERDES_V5_RX_VGA_CAL_CNTRL2, 0x07),
+ QMP_PHY_INIT_CFG(QSERDES_V5_RX_GM_CAL, 0x00),
+ QMP_PHY_INIT_CFG(QSERDES_V5_RX_UCDR_SB2_THRESH1, 0x08),
+ QMP_PHY_INIT_CFG(QSERDES_V5_RX_UCDR_SB2_THRESH2, 0x08),
+ QMP_PHY_INIT_CFG(QSERDES_V5_RX_UCDR_PI_CONTROLS, 0xf0),
+ QMP_PHY_INIT_CFG(QSERDES_V5_RX_DFE_CTLE_POST_CAL_OFFSET, 0x38),
+};
+
+static const struct qmp_phy_init_tbl sc8280xp_qmp_gen3x1_pcie_pcs_tbl[] = {
+ QMP_PHY_INIT_CFG(QPHY_V5_PCS_REFGEN_REQ_CONFIG1, 0x05),
+ QMP_PHY_INIT_CFG(QPHY_V5_PCS_RX_SIGDET_LVL, 0x77),
+ QMP_PHY_INIT_CFG(QPHY_V5_PCS_RATE_SLEW_CNTRL1, 0x0b),
+};
+
+static const struct qmp_phy_init_tbl sc8280xp_qmp_gen3x1_pcie_pcs_misc_tbl[] = {
+ QMP_PHY_INIT_CFG(QPHY_V5_PCS_PCIE_OSC_DTCT_ACTIONS, 0x00),
+ QMP_PHY_INIT_CFG(QPHY_V5_PCS_PCIE_INT_AUX_CLK_CONFIG1, 0x00),
+ QMP_PHY_INIT_CFG(QPHY_V5_PCS_PCIE_EQ_CONFIG2, 0x0f),
+ QMP_PHY_INIT_CFG(QPHY_V5_PCS_PCIE_ENDPOINT_REFCLK_DRIVE, 0xc1),
+};
+
+static const struct qmp_phy_init_tbl sc8280xp_qmp_gen3x2_pcie_tx_tbl[] = {
+ QMP_PHY_INIT_CFG_LANE(QSERDES_V5_TX_PI_QEC_CTRL, 0x02, 1),
+ QMP_PHY_INIT_CFG_LANE(QSERDES_V5_TX_PI_QEC_CTRL, 0x04, 2),
+ QMP_PHY_INIT_CFG(QSERDES_V5_TX_LANE_MODE_1, 0xd5),
+ QMP_PHY_INIT_CFG(QSERDES_V5_TX_LANE_MODE_4, 0x3f),
+ QMP_PHY_INIT_CFG(QSERDES_V5_TX_RES_CODE_LANE_OFFSET_TX, 0x11),
+ QMP_PHY_INIT_CFG(QSERDES_V5_TX_RES_CODE_LANE_OFFSET_RX, 0x0c),
+};
+
+static const struct qmp_phy_init_tbl sc8280xp_qmp_gen3x2_pcie_rx_tbl[] = {
+ QMP_PHY_INIT_CFG(QSERDES_V5_RX_RX_MODE_00_LOW, 0x7f),
+ QMP_PHY_INIT_CFG(QSERDES_V5_RX_RX_MODE_00_HIGH, 0xff),
+ QMP_PHY_INIT_CFG(QSERDES_V5_RX_RX_MODE_00_HIGH2, 0x7f),
+ QMP_PHY_INIT_CFG(QSERDES_V5_RX_RX_MODE_00_HIGH3, 0x34),
+ QMP_PHY_INIT_CFG(QSERDES_V5_RX_RX_MODE_00_HIGH4, 0xd8),
+ QMP_PHY_INIT_CFG(QSERDES_V5_RX_RX_MODE_01_LOW, 0xdc),
+ QMP_PHY_INIT_CFG(QSERDES_V5_RX_RX_MODE_01_HIGH, 0xdc),
+ QMP_PHY_INIT_CFG(QSERDES_V5_RX_RX_MODE_01_HIGH2, 0x5c),
+ QMP_PHY_INIT_CFG(QSERDES_V5_RX_RX_MODE_01_HIGH3, 0x34),
+ QMP_PHY_INIT_CFG(QSERDES_V5_RX_RX_MODE_01_HIGH4, 0xa6),
+ QMP_PHY_INIT_CFG(QSERDES_V5_RX_RX_MODE_10_HIGH3, 0x34),
+ QMP_PHY_INIT_CFG(QSERDES_V5_RX_VGA_CAL_CNTRL2, 0x0f),
+ QMP_PHY_INIT_CFG(QSERDES_V5_RX_GM_CAL, 0x00),
+ QMP_PHY_INIT_CFG(QSERDES_V5_RX_UCDR_SB2_THRESH1, 0x08),
+ QMP_PHY_INIT_CFG(QSERDES_V5_RX_UCDR_SB2_THRESH2, 0x08),
+ QMP_PHY_INIT_CFG(QSERDES_V5_RX_UCDR_PI_CONTROLS, 0xf0),
+ QMP_PHY_INIT_CFG(QSERDES_V5_RX_DFE_CTLE_POST_CAL_OFFSET, 0x38),
+};
+
+static const struct qmp_phy_init_tbl sc8280xp_qmp_gen3x2_pcie_pcs_tbl[] = {
+ QMP_PHY_INIT_CFG(QPHY_V5_PCS_REFGEN_REQ_CONFIG1, 0x05),
+ QMP_PHY_INIT_CFG(QPHY_V5_PCS_RX_SIGDET_LVL, 0x88),
+ QMP_PHY_INIT_CFG(QPHY_V5_PCS_RATE_SLEW_CNTRL1, 0x0b),
+ QMP_PHY_INIT_CFG(QPHY_V5_PCS_EQ_CONFIG3, 0x0f),
+};
+
+static const struct qmp_phy_init_tbl sc8280xp_qmp_gen3x2_pcie_pcs_misc_tbl[] = {
+ QMP_PHY_INIT_CFG(QPHY_V5_PCS_PCIE_POWER_STATE_CONFIG2, 0x1d),
+ QMP_PHY_INIT_CFG(QPHY_V5_PCS_PCIE_POWER_STATE_CONFIG4, 0x07),
+ QMP_PHY_INIT_CFG(QPHY_V5_PCS_PCIE_ENDPOINT_REFCLK_DRIVE, 0xc1),
+ QMP_PHY_INIT_CFG(QPHY_V5_PCS_PCIE_OSC_DTCT_ACTIONS, 0x00),
+};
+
static const struct qmp_phy_init_tbl sm8250_qmp_pcie_serdes_tbl[] = {
QMP_PHY_INIT_CFG(QSERDES_V4_COM_SYSCLK_EN_SEL, 0x08),
QMP_PHY_INIT_CFG(QSERDES_V4_COM_CLK_SELECT, 0x34),
@@ -1184,15 +1308,29 @@ static const struct qmp_phy_init_tbl sm8450_qmp_gen3x1_pcie_pcs_misc_tbl[] = {
};
static const struct qmp_phy_init_tbl sm8450_qmp_gen4x2_pcie_serdes_tbl[] = {
+ QMP_PHY_INIT_CFG(QSERDES_V5_COM_BIAS_EN_CLKBUFLR_EN, 0x14),
+ QMP_PHY_INIT_CFG(QSERDES_V5_COM_PLL_IVCO, 0x0f),
+ QMP_PHY_INIT_CFG(QSERDES_V5_COM_LOCK_CMP_EN, 0x46),
+ QMP_PHY_INIT_CFG(QSERDES_V5_COM_LOCK_CMP_CFG, 0x04),
+ QMP_PHY_INIT_CFG(QSERDES_V5_COM_VCO_TUNE_MAP, 0x02),
+ QMP_PHY_INIT_CFG(QSERDES_V5_COM_HSCLK_SEL, 0x12),
+ QMP_PHY_INIT_CFG(QSERDES_V5_COM_HSCLK_HS_SWITCH_SEL, 0x00),
+ QMP_PHY_INIT_CFG(QSERDES_V5_COM_CORECLK_DIV_MODE0, 0x0a),
+ QMP_PHY_INIT_CFG(QSERDES_V5_COM_CORECLK_DIV_MODE1, 0x04),
+ QMP_PHY_INIT_CFG(QSERDES_V5_COM_CMN_MISC1, 0x88),
+ QMP_PHY_INIT_CFG(QSERDES_V5_COM_CMN_CONFIG, 0x06),
+ QMP_PHY_INIT_CFG(QSERDES_V5_COM_CMN_MODE, 0x14),
+ QMP_PHY_INIT_CFG(QSERDES_V5_COM_VCO_DC_LEVEL_CTRL, 0x0f),
+};
+
+static const struct qmp_phy_init_tbl sm8450_qmp_gen4x2_pcie_rc_serdes_tbl[] = {
QMP_PHY_INIT_CFG(QSERDES_V5_COM_SSC_PER1, 0x31),
QMP_PHY_INIT_CFG(QSERDES_V5_COM_SSC_PER2, 0x01),
QMP_PHY_INIT_CFG(QSERDES_V5_COM_SSC_STEP_SIZE1_MODE0, 0xde),
QMP_PHY_INIT_CFG(QSERDES_V5_COM_SSC_STEP_SIZE2_MODE0, 0x07),
QMP_PHY_INIT_CFG(QSERDES_V5_COM_SSC_STEP_SIZE1_MODE1, 0x97),
QMP_PHY_INIT_CFG(QSERDES_V5_COM_SSC_STEP_SIZE2_MODE1, 0x0c),
- QMP_PHY_INIT_CFG(QSERDES_V5_COM_BIAS_EN_CLKBUFLR_EN, 0x14),
QMP_PHY_INIT_CFG(QSERDES_V5_COM_CLK_ENABLE1, 0x90),
- QMP_PHY_INIT_CFG(QSERDES_V5_COM_PLL_IVCO, 0x0f),
QMP_PHY_INIT_CFG(QSERDES_V5_COM_CP_CTRL_MODE0, 0x06),
QMP_PHY_INIT_CFG(QSERDES_V5_COM_CP_CTRL_MODE1, 0x06),
QMP_PHY_INIT_CFG(QSERDES_V5_COM_PLL_RCTRL_MODE0, 0x16),
@@ -1200,8 +1338,6 @@ static const struct qmp_phy_init_tbl sm8450_qmp_gen4x2_pcie_serdes_tbl[] = {
QMP_PHY_INIT_CFG(QSERDES_V5_COM_PLL_CCTRL_MODE0, 0x36),
QMP_PHY_INIT_CFG(QSERDES_V5_COM_PLL_CCTRL_MODE1, 0x36),
QMP_PHY_INIT_CFG(QSERDES_V5_COM_SYSCLK_EN_SEL, 0x08),
- QMP_PHY_INIT_CFG(QSERDES_V5_COM_LOCK_CMP_EN, 0x46),
- QMP_PHY_INIT_CFG(QSERDES_V5_COM_LOCK_CMP_CFG, 0x04),
QMP_PHY_INIT_CFG(QSERDES_V5_COM_LOCK_CMP1_MODE0, 0x0a),
QMP_PHY_INIT_CFG(QSERDES_V5_COM_LOCK_CMP2_MODE0, 0x1a),
QMP_PHY_INIT_CFG(QSERDES_V5_COM_LOCK_CMP1_MODE1, 0x14),
@@ -1214,17 +1350,8 @@ static const struct qmp_phy_init_tbl sm8450_qmp_gen4x2_pcie_serdes_tbl[] = {
QMP_PHY_INIT_CFG(QSERDES_V5_COM_DIV_FRAC_START1_MODE1, 0x55),
QMP_PHY_INIT_CFG(QSERDES_V5_COM_DIV_FRAC_START2_MODE1, 0x55),
QMP_PHY_INIT_CFG(QSERDES_V5_COM_DIV_FRAC_START3_MODE1, 0x05),
- QMP_PHY_INIT_CFG(QSERDES_V5_COM_VCO_TUNE_MAP, 0x02),
QMP_PHY_INIT_CFG(QSERDES_V5_COM_CLK_SELECT, 0x34),
- QMP_PHY_INIT_CFG(QSERDES_V5_COM_HSCLK_SEL, 0x12),
- QMP_PHY_INIT_CFG(QSERDES_V5_COM_HSCLK_HS_SWITCH_SEL, 0x00),
- QMP_PHY_INIT_CFG(QSERDES_V5_COM_CORECLK_DIV_MODE0, 0x0a),
- QMP_PHY_INIT_CFG(QSERDES_V5_COM_CORECLK_DIV_MODE1, 0x04),
- QMP_PHY_INIT_CFG(QSERDES_V5_COM_CMN_MISC1, 0x88),
QMP_PHY_INIT_CFG(QSERDES_V5_COM_CORE_CLK_EN, 0x20),
- QMP_PHY_INIT_CFG(QSERDES_V5_COM_CMN_CONFIG, 0x06),
- QMP_PHY_INIT_CFG(QSERDES_V5_COM_CMN_MODE, 0x14),
- QMP_PHY_INIT_CFG(QSERDES_V5_COM_VCO_DC_LEVEL_CTRL, 0x0f),
};
static const struct qmp_phy_init_tbl sm8450_qmp_gen4x2_pcie_tx_tbl[] = {
@@ -1285,46 +1412,95 @@ static const struct qmp_phy_init_tbl sm8450_qmp_gen4x2_pcie_rx_tbl[] = {
};
static const struct qmp_phy_init_tbl sm8450_qmp_gen4x2_pcie_pcs_tbl[] = {
- QMP_PHY_INIT_CFG(QPHY_V5_PCS_EQ_CONFIG2, 0x16),
- QMP_PHY_INIT_CFG(QPHY_V5_PCS_EQ_CONFIG3, 0x22),
- QMP_PHY_INIT_CFG(QPHY_V5_PCS_G3S2_PRE_GAIN, 0x2e),
- QMP_PHY_INIT_CFG(QPHY_V5_PCS_RX_SIGDET_LVL, 0x99),
+ QMP_PHY_INIT_CFG(QPHY_V5_20_PCS_EQ_CONFIG4, 0x16),
+ QMP_PHY_INIT_CFG(QPHY_V5_20_PCS_EQ_CONFIG5, 0x22),
+ QMP_PHY_INIT_CFG(QPHY_V5_20_PCS_G3S2_PRE_GAIN, 0x2e),
+ QMP_PHY_INIT_CFG(QPHY_V5_20_PCS_RX_SIGDET_LVL, 0x99),
};
static const struct qmp_phy_init_tbl sm8450_qmp_gen4x2_pcie_pcs_misc_tbl[] = {
- QMP_PHY_INIT_CFG(QPHY_V5_20_PCS_PCIE_ENDPOINT_REFCLK_DRIVE, 0xc1),
- QMP_PHY_INIT_CFG(QPHY_V5_20_PCS_PCIE_OSC_DTCT_ACTIONS, 0x00),
QMP_PHY_INIT_CFG(QPHY_V5_20_PCS_PCIE_G4_EQ_CONFIG5, 0x02),
QMP_PHY_INIT_CFG(QPHY_V5_20_PCS_PCIE_EQ_CONFIG1, 0x16),
QMP_PHY_INIT_CFG(QPHY_V5_20_PCS_PCIE_RX_MARGINING_CONFIG3, 0x28),
QMP_PHY_INIT_CFG(QPHY_V5_20_PCS_PCIE_G4_PRE_GAIN, 0x2e),
};
+static const struct qmp_phy_init_tbl sm8450_qmp_gen4x2_pcie_rc_pcs_misc_tbl[] = {
+ QMP_PHY_INIT_CFG(QPHY_V5_20_PCS_PCIE_ENDPOINT_REFCLK_DRIVE, 0xc1),
+ QMP_PHY_INIT_CFG(QPHY_V5_20_PCS_PCIE_OSC_DTCT_ACTIONS, 0x00),
+ QMP_PHY_INIT_CFG(QPHY_V5_20_PCS_PCIE_PRESET_P10_POST, 0x00),
+};
+
+static const struct qmp_phy_init_tbl sm8450_qmp_gen4x2_pcie_ep_serdes_tbl[] = {
+ QMP_PHY_INIT_CFG(QSERDES_V5_COM_BG_TIMER, 0x02),
+ QMP_PHY_INIT_CFG(QSERDES_V5_COM_SYS_CLK_CTRL, 0x07),
+ QMP_PHY_INIT_CFG(QSERDES_V5_COM_CP_CTRL_MODE0, 0x27),
+ QMP_PHY_INIT_CFG(QSERDES_V5_COM_CP_CTRL_MODE1, 0x0a),
+ QMP_PHY_INIT_CFG(QSERDES_V5_COM_PLL_RCTRL_MODE0, 0x17),
+ QMP_PHY_INIT_CFG(QSERDES_V5_COM_PLL_RCTRL_MODE1, 0x19),
+ QMP_PHY_INIT_CFG(QSERDES_V5_COM_PLL_CCTRL_MODE0, 0x00),
+ QMP_PHY_INIT_CFG(QSERDES_V5_COM_PLL_CCTRL_MODE1, 0x03),
+ QMP_PHY_INIT_CFG(QSERDES_V5_COM_SYSCLK_EN_SEL, 0x00),
+ QMP_PHY_INIT_CFG(QSERDES_V5_COM_LOCK_CMP1_MODE0, 0xff),
+ QMP_PHY_INIT_CFG(QSERDES_V5_COM_LOCK_CMP2_MODE0, 0x04),
+ QMP_PHY_INIT_CFG(QSERDES_V5_COM_LOCK_CMP1_MODE1, 0xff),
+ QMP_PHY_INIT_CFG(QSERDES_V5_COM_LOCK_CMP2_MODE1, 0x09),
+ QMP_PHY_INIT_CFG(QSERDES_V5_COM_DEC_START_MODE0, 0x19),
+ QMP_PHY_INIT_CFG(QSERDES_V5_COM_DEC_START_MODE1, 0x28),
+ QMP_PHY_INIT_CFG(QSERDES_V5_COM_INTEGLOOP_GAIN0_MODE0, 0xfb),
+ QMP_PHY_INIT_CFG(QSERDES_V5_COM_INTEGLOOP_GAIN1_MODE0, 0x01),
+ QMP_PHY_INIT_CFG(QSERDES_V5_COM_INTEGLOOP_GAIN0_MODE1, 0xfb),
+ QMP_PHY_INIT_CFG(QSERDES_V5_COM_INTEGLOOP_GAIN1_MODE1, 0x01),
+ QMP_PHY_INIT_CFG(QSERDES_V5_COM_CORE_CLK_EN, 0x60),
+};
+
+static const struct qmp_phy_init_tbl sm8450_qmp_gen4x2_pcie_ep_pcs_misc_tbl[] = {
+ QMP_PHY_INIT_CFG(QPHY_V5_20_PCS_PCIE_OSC_DTCT_MODE2_CONFIG5, 0x08),
+};
+
+struct qmp_pcie_offsets {
+ u16 serdes;
+ u16 pcs;
+ u16 pcs_misc;
+ u16 tx;
+ u16 rx;
+ u16 tx2;
+ u16 rx2;
+};
+
+struct qmp_phy_cfg_tbls {
+ const struct qmp_phy_init_tbl *serdes;
+ int serdes_num;
+ const struct qmp_phy_init_tbl *tx;
+ int tx_num;
+ const struct qmp_phy_init_tbl *rx;
+ int rx_num;
+ const struct qmp_phy_init_tbl *pcs;
+ int pcs_num;
+ const struct qmp_phy_init_tbl *pcs_misc;
+ int pcs_misc_num;
+};
+
/* struct qmp_phy_cfg - per-PHY initialization config */
struct qmp_phy_cfg {
int lanes;
- /* Init sequence for PHY blocks - serdes, tx, rx, pcs */
- const struct qmp_phy_init_tbl *serdes_tbl;
- int serdes_tbl_num;
- const struct qmp_phy_init_tbl *serdes_tbl_sec;
- int serdes_tbl_num_sec;
- const struct qmp_phy_init_tbl *tx_tbl;
- int tx_tbl_num;
- const struct qmp_phy_init_tbl *tx_tbl_sec;
- int tx_tbl_num_sec;
- const struct qmp_phy_init_tbl *rx_tbl;
- int rx_tbl_num;
- const struct qmp_phy_init_tbl *rx_tbl_sec;
- int rx_tbl_num_sec;
- const struct qmp_phy_init_tbl *pcs_tbl;
- int pcs_tbl_num;
- const struct qmp_phy_init_tbl *pcs_tbl_sec;
- int pcs_tbl_num_sec;
- const struct qmp_phy_init_tbl *pcs_misc_tbl;
- int pcs_misc_tbl_num;
- const struct qmp_phy_init_tbl *pcs_misc_tbl_sec;
- int pcs_misc_tbl_num_sec;
+ const struct qmp_pcie_offsets *offsets;
+
+ /* Main init sequence for PHY blocks - serdes, tx, rx, pcs */
+ const struct qmp_phy_cfg_tbls tbls;
+ /*
+ * Additional init sequences for PHY blocks, providing additional
+ * register programming. They are used for providing separate sequences
+ * for the Root Complex and End Point use cases.
+ *
+ * If EP mode is not supported, both tables can be left unset.
+ */
+ const struct qmp_phy_cfg_tbls *tbls_rc;
+ const struct qmp_phy_cfg_tbls *tbls_ep;
+
+ const struct qmp_phy_init_tbl *serdes_4ln_tbl;
+ int serdes_4ln_num;
/* clock ids to be requested */
const char * const *clk_list;
@@ -1339,69 +1515,43 @@ struct qmp_phy_cfg {
/* array of registers with different offsets */
const unsigned int *regs;
- unsigned int start_ctrl;
unsigned int pwrdn_ctrl;
/* bit offset of PHYSTATUS in QPHY_PCS_STATUS register */
unsigned int phy_status;
- /* true, if PHY needs delay after POWER_DOWN */
- bool has_pwrdn_delay;
- /* power_down delay in usec */
- int pwrdn_delay_min;
- int pwrdn_delay_max;
+ bool skip_start_delay;
/* QMP PHY pipe clock interface rate */
unsigned long pipe_clock_rate;
};
-/**
- * struct qmp_phy - per-lane phy descriptor
- *
- * @phy: generic phy
- * @cfg: phy specific configuration
- * @serdes: iomapped memory space for phy's serdes (i.e. PLL)
- * @tx: iomapped memory space for lane's tx
- * @rx: iomapped memory space for lane's rx
- * @pcs: iomapped memory space for lane's pcs
- * @tx2: iomapped memory space for second lane's tx (in dual lane PHYs)
- * @rx2: iomapped memory space for second lane's rx (in dual lane PHYs)
- * @pcs_misc: iomapped memory space for lane's pcs_misc
- * @pipe_clk: pipe clock
- * @qmp: QMP phy to which this lane belongs
- */
-struct qmp_phy {
- struct phy *phy;
- const struct qmp_phy_cfg *cfg;
- void __iomem *serdes;
- void __iomem *tx;
- void __iomem *rx;
- void __iomem *pcs;
- void __iomem *tx2;
- void __iomem *rx2;
- void __iomem *pcs_misc;
- struct clk *pipe_clk;
- struct qcom_qmp *qmp;
-};
-
-/**
- * struct qcom_qmp - structure holding QMP phy block attributes
- *
- * @dev: device
- *
- * @clks: array of clocks required by phy
- * @resets: array of resets required by phy
- * @vregs: regulator supplies bulk data
- *
- * @phys: array of per-lane phy descriptors
- */
-struct qcom_qmp {
+struct qmp_pcie {
struct device *dev;
+ const struct qmp_phy_cfg *cfg;
+ bool tcsr_4ln_config;
+
+ void __iomem *serdes;
+ void __iomem *pcs;
+ void __iomem *pcs_misc;
+ void __iomem *tx;
+ void __iomem *rx;
+ void __iomem *tx2;
+ void __iomem *rx2;
+
+ void __iomem *port_b;
+
struct clk_bulk_data *clks;
+ struct clk_bulk_data pipe_clks[2];
+ int num_pipe_clks;
+
struct reset_control_bulk_data *resets;
struct regulator_bulk_data *vregs;
- struct qmp_phy **phys;
+ struct phy *phy;
+ int mode;
+
+ struct clk_fixed_rate pipe_clk_fixed;
};
static inline void qphy_setbits(void __iomem *base, u32 offset, u32 val)
@@ -1429,10 +1579,17 @@ static inline void qphy_clrbits(void __iomem *base, u32 offset, u32 val)
}
/* list of clocks required by phy */
+static const char * const ipq8074_pciephy_clk_l[] = {
+ "aux", "cfg_ahb",
+};
+
static const char * const msm8996_phy_clk_l[] = {
"aux", "cfg_ahb", "ref",
};
+static const char * const sc8280xp_pciephy_clk_l[] = {
+ "aux", "cfg_ahb", "ref", "rchng",
+};
static const char * const sdm845_pciephy_clk_l[] = {
"aux", "cfg_ahb", "ref", "refgen",
@@ -1443,10 +1600,6 @@ static const char * const qmp_phy_vreg_l[] = {
"vdda-phy", "vdda-pll",
};
-static const char * const ipq8074_pciephy_clk_l[] = {
- "aux", "cfg_ahb",
-};
-
/* list of resets */
static const char * const ipq8074_pciephy_reset_l[] = {
"phy", "common",
@@ -1456,17 +1609,29 @@ static const char * const sdm845_pciephy_reset_l[] = {
"phy",
};
+static const struct qmp_pcie_offsets qmp_pcie_offsets_v5 = {
+ .serdes = 0,
+ .pcs = 0x0200,
+ .pcs_misc = 0x0600,
+ .tx = 0x0e00,
+ .rx = 0x1000,
+ .tx2 = 0x1600,
+ .rx2 = 0x1800,
+};
+
static const struct qmp_phy_cfg ipq8074_pciephy_cfg = {
.lanes = 1,
- .serdes_tbl = ipq8074_pcie_serdes_tbl,
- .serdes_tbl_num = ARRAY_SIZE(ipq8074_pcie_serdes_tbl),
- .tx_tbl = ipq8074_pcie_tx_tbl,
- .tx_tbl_num = ARRAY_SIZE(ipq8074_pcie_tx_tbl),
- .rx_tbl = ipq8074_pcie_rx_tbl,
- .rx_tbl_num = ARRAY_SIZE(ipq8074_pcie_rx_tbl),
- .pcs_tbl = ipq8074_pcie_pcs_tbl,
- .pcs_tbl_num = ARRAY_SIZE(ipq8074_pcie_pcs_tbl),
+ .tbls = {
+ .serdes = ipq8074_pcie_serdes_tbl,
+ .serdes_num = ARRAY_SIZE(ipq8074_pcie_serdes_tbl),
+ .tx = ipq8074_pcie_tx_tbl,
+ .tx_num = ARRAY_SIZE(ipq8074_pcie_tx_tbl),
+ .rx = ipq8074_pcie_rx_tbl,
+ .rx_num = ARRAY_SIZE(ipq8074_pcie_rx_tbl),
+ .pcs = ipq8074_pcie_pcs_tbl,
+ .pcs_num = ARRAY_SIZE(ipq8074_pcie_pcs_tbl),
+ },
.clk_list = ipq8074_pciephy_clk_l,
.num_clks = ARRAY_SIZE(ipq8074_pciephy_clk_l),
.reset_list = ipq8074_pciephy_reset_l,
@@ -1475,26 +1640,25 @@ static const struct qmp_phy_cfg ipq8074_pciephy_cfg = {
.num_vregs = 0,
.regs = pciephy_regs_layout,
- .start_ctrl = SERDES_START | PCS_START,
.pwrdn_ctrl = SW_PWRDN | REFCLK_DRV_DSBL,
.phy_status = PHYSTATUS,
-
- .has_pwrdn_delay = true,
- .pwrdn_delay_min = 995, /* us */
- .pwrdn_delay_max = 1005, /* us */
};
static const struct qmp_phy_cfg ipq8074_pciephy_gen3_cfg = {
.lanes = 1,
- .serdes_tbl = ipq8074_pcie_gen3_serdes_tbl,
- .serdes_tbl_num = ARRAY_SIZE(ipq8074_pcie_gen3_serdes_tbl),
- .tx_tbl = ipq8074_pcie_gen3_tx_tbl,
- .tx_tbl_num = ARRAY_SIZE(ipq8074_pcie_gen3_tx_tbl),
- .rx_tbl = ipq8074_pcie_gen3_rx_tbl,
- .rx_tbl_num = ARRAY_SIZE(ipq8074_pcie_gen3_rx_tbl),
- .pcs_tbl = ipq8074_pcie_gen3_pcs_tbl,
- .pcs_tbl_num = ARRAY_SIZE(ipq8074_pcie_gen3_pcs_tbl),
+ .tbls = {
+ .serdes = ipq8074_pcie_gen3_serdes_tbl,
+ .serdes_num = ARRAY_SIZE(ipq8074_pcie_gen3_serdes_tbl),
+ .tx = ipq8074_pcie_gen3_tx_tbl,
+ .tx_num = ARRAY_SIZE(ipq8074_pcie_gen3_tx_tbl),
+ .rx = ipq8074_pcie_gen3_rx_tbl,
+ .rx_num = ARRAY_SIZE(ipq8074_pcie_gen3_rx_tbl),
+ .pcs = ipq8074_pcie_gen3_pcs_tbl,
+ .pcs_num = ARRAY_SIZE(ipq8074_pcie_gen3_pcs_tbl),
+ .pcs_misc = ipq8074_pcie_gen3_pcs_misc_tbl,
+ .pcs_misc_num = ARRAY_SIZE(ipq8074_pcie_gen3_pcs_misc_tbl),
+ },
.clk_list = ipq8074_pciephy_clk_l,
.num_clks = ARRAY_SIZE(ipq8074_pciephy_clk_l),
.reset_list = ipq8074_pciephy_reset_l,
@@ -1503,12 +1667,8 @@ static const struct qmp_phy_cfg ipq8074_pciephy_gen3_cfg = {
.num_vregs = 0,
.regs = ipq_pciephy_gen3_regs_layout,
- .start_ctrl = SERDES_START | PCS_START,
.pwrdn_ctrl = SW_PWRDN | REFCLK_DRV_DSBL,
-
- .has_pwrdn_delay = true,
- .pwrdn_delay_min = 995, /* us */
- .pwrdn_delay_max = 1005, /* us */
+ .phy_status = PHYSTATUS,
.pipe_clock_rate = 250000000,
};
@@ -1516,16 +1676,18 @@ static const struct qmp_phy_cfg ipq8074_pciephy_gen3_cfg = {
static const struct qmp_phy_cfg ipq6018_pciephy_cfg = {
.lanes = 1,
- .serdes_tbl = ipq6018_pcie_serdes_tbl,
- .serdes_tbl_num = ARRAY_SIZE(ipq6018_pcie_serdes_tbl),
- .tx_tbl = ipq6018_pcie_tx_tbl,
- .tx_tbl_num = ARRAY_SIZE(ipq6018_pcie_tx_tbl),
- .rx_tbl = ipq6018_pcie_rx_tbl,
- .rx_tbl_num = ARRAY_SIZE(ipq6018_pcie_rx_tbl),
- .pcs_tbl = ipq6018_pcie_pcs_tbl,
- .pcs_tbl_num = ARRAY_SIZE(ipq6018_pcie_pcs_tbl),
- .pcs_misc_tbl = ipq6018_pcie_pcs_misc_tbl,
- .pcs_misc_tbl_num = ARRAY_SIZE(ipq6018_pcie_pcs_misc_tbl),
+ .tbls = {
+ .serdes = ipq6018_pcie_serdes_tbl,
+ .serdes_num = ARRAY_SIZE(ipq6018_pcie_serdes_tbl),
+ .tx = ipq6018_pcie_tx_tbl,
+ .tx_num = ARRAY_SIZE(ipq6018_pcie_tx_tbl),
+ .rx = ipq6018_pcie_rx_tbl,
+ .rx_num = ARRAY_SIZE(ipq6018_pcie_rx_tbl),
+ .pcs = ipq6018_pcie_pcs_tbl,
+ .pcs_num = ARRAY_SIZE(ipq6018_pcie_pcs_tbl),
+ .pcs_misc = ipq6018_pcie_pcs_misc_tbl,
+ .pcs_misc_num = ARRAY_SIZE(ipq6018_pcie_pcs_misc_tbl),
+ },
.clk_list = ipq8074_pciephy_clk_l,
.num_clks = ARRAY_SIZE(ipq8074_pciephy_clk_l),
.reset_list = ipq8074_pciephy_reset_l,
@@ -1534,27 +1696,25 @@ static const struct qmp_phy_cfg ipq6018_pciephy_cfg = {
.num_vregs = 0,
.regs = ipq_pciephy_gen3_regs_layout,
- .start_ctrl = SERDES_START | PCS_START,
.pwrdn_ctrl = SW_PWRDN | REFCLK_DRV_DSBL,
-
- .has_pwrdn_delay = true,
- .pwrdn_delay_min = 995, /* us */
- .pwrdn_delay_max = 1005, /* us */
+ .phy_status = PHYSTATUS,
};
static const struct qmp_phy_cfg sdm845_qmp_pciephy_cfg = {
.lanes = 1,
- .serdes_tbl = sdm845_qmp_pcie_serdes_tbl,
- .serdes_tbl_num = ARRAY_SIZE(sdm845_qmp_pcie_serdes_tbl),
- .tx_tbl = sdm845_qmp_pcie_tx_tbl,
- .tx_tbl_num = ARRAY_SIZE(sdm845_qmp_pcie_tx_tbl),
- .rx_tbl = sdm845_qmp_pcie_rx_tbl,
- .rx_tbl_num = ARRAY_SIZE(sdm845_qmp_pcie_rx_tbl),
- .pcs_tbl = sdm845_qmp_pcie_pcs_tbl,
- .pcs_tbl_num = ARRAY_SIZE(sdm845_qmp_pcie_pcs_tbl),
- .pcs_misc_tbl = sdm845_qmp_pcie_pcs_misc_tbl,
- .pcs_misc_tbl_num = ARRAY_SIZE(sdm845_qmp_pcie_pcs_misc_tbl),
+ .tbls = {
+ .serdes = sdm845_qmp_pcie_serdes_tbl,
+ .serdes_num = ARRAY_SIZE(sdm845_qmp_pcie_serdes_tbl),
+ .tx = sdm845_qmp_pcie_tx_tbl,
+ .tx_num = ARRAY_SIZE(sdm845_qmp_pcie_tx_tbl),
+ .rx = sdm845_qmp_pcie_rx_tbl,
+ .rx_num = ARRAY_SIZE(sdm845_qmp_pcie_rx_tbl),
+ .pcs = sdm845_qmp_pcie_pcs_tbl,
+ .pcs_num = ARRAY_SIZE(sdm845_qmp_pcie_pcs_tbl),
+ .pcs_misc = sdm845_qmp_pcie_pcs_misc_tbl,
+ .pcs_misc_num = ARRAY_SIZE(sdm845_qmp_pcie_pcs_misc_tbl),
+ },
.clk_list = sdm845_pciephy_clk_l,
.num_clks = ARRAY_SIZE(sdm845_pciephy_clk_l),
.reset_list = sdm845_pciephy_reset_l,
@@ -1563,26 +1723,23 @@ static const struct qmp_phy_cfg sdm845_qmp_pciephy_cfg = {
.num_vregs = ARRAY_SIZE(qmp_phy_vreg_l),
.regs = sdm845_qmp_pciephy_regs_layout,
- .start_ctrl = PCS_START | SERDES_START,
.pwrdn_ctrl = SW_PWRDN | REFCLK_DRV_DSBL,
.phy_status = PHYSTATUS,
-
- .has_pwrdn_delay = true,
- .pwrdn_delay_min = 995, /* us */
- .pwrdn_delay_max = 1005, /* us */
};
static const struct qmp_phy_cfg sdm845_qhp_pciephy_cfg = {
.lanes = 1,
- .serdes_tbl = sdm845_qhp_pcie_serdes_tbl,
- .serdes_tbl_num = ARRAY_SIZE(sdm845_qhp_pcie_serdes_tbl),
- .tx_tbl = sdm845_qhp_pcie_tx_tbl,
- .tx_tbl_num = ARRAY_SIZE(sdm845_qhp_pcie_tx_tbl),
- .rx_tbl = sdm845_qhp_pcie_rx_tbl,
- .rx_tbl_num = ARRAY_SIZE(sdm845_qhp_pcie_rx_tbl),
- .pcs_tbl = sdm845_qhp_pcie_pcs_tbl,
- .pcs_tbl_num = ARRAY_SIZE(sdm845_qhp_pcie_pcs_tbl),
+ .tbls = {
+ .serdes = sdm845_qhp_pcie_serdes_tbl,
+ .serdes_num = ARRAY_SIZE(sdm845_qhp_pcie_serdes_tbl),
+ .tx = sdm845_qhp_pcie_tx_tbl,
+ .tx_num = ARRAY_SIZE(sdm845_qhp_pcie_tx_tbl),
+ .rx = sdm845_qhp_pcie_rx_tbl,
+ .rx_num = ARRAY_SIZE(sdm845_qhp_pcie_rx_tbl),
+ .pcs = sdm845_qhp_pcie_pcs_tbl,
+ .pcs_num = ARRAY_SIZE(sdm845_qhp_pcie_pcs_tbl),
+ },
.clk_list = sdm845_pciephy_clk_l,
.num_clks = ARRAY_SIZE(sdm845_pciephy_clk_l),
.reset_list = sdm845_pciephy_reset_l,
@@ -1591,36 +1748,35 @@ static const struct qmp_phy_cfg sdm845_qhp_pciephy_cfg = {
.num_vregs = ARRAY_SIZE(qmp_phy_vreg_l),
.regs = sdm845_qhp_pciephy_regs_layout,
- .start_ctrl = PCS_START | SERDES_START,
.pwrdn_ctrl = SW_PWRDN | REFCLK_DRV_DSBL,
.phy_status = PHYSTATUS,
-
- .has_pwrdn_delay = true,
- .pwrdn_delay_min = 995, /* us */
- .pwrdn_delay_max = 1005, /* us */
};
static const struct qmp_phy_cfg sm8250_qmp_gen3x1_pciephy_cfg = {
.lanes = 1,
- .serdes_tbl = sm8250_qmp_pcie_serdes_tbl,
- .serdes_tbl_num = ARRAY_SIZE(sm8250_qmp_pcie_serdes_tbl),
- .serdes_tbl_sec = sm8250_qmp_gen3x1_pcie_serdes_tbl,
- .serdes_tbl_num_sec = ARRAY_SIZE(sm8250_qmp_gen3x1_pcie_serdes_tbl),
- .tx_tbl = sm8250_qmp_pcie_tx_tbl,
- .tx_tbl_num = ARRAY_SIZE(sm8250_qmp_pcie_tx_tbl),
- .rx_tbl = sm8250_qmp_pcie_rx_tbl,
- .rx_tbl_num = ARRAY_SIZE(sm8250_qmp_pcie_rx_tbl),
- .rx_tbl_sec = sm8250_qmp_gen3x1_pcie_rx_tbl,
- .rx_tbl_num_sec = ARRAY_SIZE(sm8250_qmp_gen3x1_pcie_rx_tbl),
- .pcs_tbl = sm8250_qmp_pcie_pcs_tbl,
- .pcs_tbl_num = ARRAY_SIZE(sm8250_qmp_pcie_pcs_tbl),
- .pcs_tbl_sec = sm8250_qmp_gen3x1_pcie_pcs_tbl,
- .pcs_tbl_num_sec = ARRAY_SIZE(sm8250_qmp_gen3x1_pcie_pcs_tbl),
- .pcs_misc_tbl = sm8250_qmp_pcie_pcs_misc_tbl,
- .pcs_misc_tbl_num = ARRAY_SIZE(sm8250_qmp_pcie_pcs_misc_tbl),
- .pcs_misc_tbl_sec = sm8250_qmp_gen3x1_pcie_pcs_misc_tbl,
- .pcs_misc_tbl_num_sec = ARRAY_SIZE(sm8250_qmp_gen3x1_pcie_pcs_misc_tbl),
+ .tbls = {
+ .serdes = sm8250_qmp_pcie_serdes_tbl,
+ .serdes_num = ARRAY_SIZE(sm8250_qmp_pcie_serdes_tbl),
+ .tx = sm8250_qmp_pcie_tx_tbl,
+ .tx_num = ARRAY_SIZE(sm8250_qmp_pcie_tx_tbl),
+ .rx = sm8250_qmp_pcie_rx_tbl,
+ .rx_num = ARRAY_SIZE(sm8250_qmp_pcie_rx_tbl),
+ .pcs = sm8250_qmp_pcie_pcs_tbl,
+ .pcs_num = ARRAY_SIZE(sm8250_qmp_pcie_pcs_tbl),
+ .pcs_misc = sm8250_qmp_pcie_pcs_misc_tbl,
+ .pcs_misc_num = ARRAY_SIZE(sm8250_qmp_pcie_pcs_misc_tbl),
+ },
+ .tbls_rc = &(const struct qmp_phy_cfg_tbls) {
+ .serdes = sm8250_qmp_gen3x1_pcie_serdes_tbl,
+ .serdes_num = ARRAY_SIZE(sm8250_qmp_gen3x1_pcie_serdes_tbl),
+ .rx = sm8250_qmp_gen3x1_pcie_rx_tbl,
+ .rx_num = ARRAY_SIZE(sm8250_qmp_gen3x1_pcie_rx_tbl),
+ .pcs = sm8250_qmp_gen3x1_pcie_pcs_tbl,
+ .pcs_num = ARRAY_SIZE(sm8250_qmp_gen3x1_pcie_pcs_tbl),
+ .pcs_misc = sm8250_qmp_gen3x1_pcie_pcs_misc_tbl,
+ .pcs_misc_num = ARRAY_SIZE(sm8250_qmp_gen3x1_pcie_pcs_misc_tbl),
+ },
.clk_list = sdm845_pciephy_clk_l,
.num_clks = ARRAY_SIZE(sdm845_pciephy_clk_l),
.reset_list = sdm845_pciephy_reset_l,
@@ -1629,36 +1785,35 @@ static const struct qmp_phy_cfg sm8250_qmp_gen3x1_pciephy_cfg = {
.num_vregs = ARRAY_SIZE(qmp_phy_vreg_l),
.regs = sm8250_pcie_regs_layout,
- .start_ctrl = PCS_START | SERDES_START,
.pwrdn_ctrl = SW_PWRDN | REFCLK_DRV_DSBL,
.phy_status = PHYSTATUS,
-
- .has_pwrdn_delay = true,
- .pwrdn_delay_min = 995, /* us */
- .pwrdn_delay_max = 1005, /* us */
};
static const struct qmp_phy_cfg sm8250_qmp_gen3x2_pciephy_cfg = {
.lanes = 2,
- .serdes_tbl = sm8250_qmp_pcie_serdes_tbl,
- .serdes_tbl_num = ARRAY_SIZE(sm8250_qmp_pcie_serdes_tbl),
- .tx_tbl = sm8250_qmp_pcie_tx_tbl,
- .tx_tbl_num = ARRAY_SIZE(sm8250_qmp_pcie_tx_tbl),
- .tx_tbl_sec = sm8250_qmp_gen3x2_pcie_tx_tbl,
- .tx_tbl_num_sec = ARRAY_SIZE(sm8250_qmp_gen3x2_pcie_tx_tbl),
- .rx_tbl = sm8250_qmp_pcie_rx_tbl,
- .rx_tbl_num = ARRAY_SIZE(sm8250_qmp_pcie_rx_tbl),
- .rx_tbl_sec = sm8250_qmp_gen3x2_pcie_rx_tbl,
- .rx_tbl_num_sec = ARRAY_SIZE(sm8250_qmp_gen3x2_pcie_rx_tbl),
- .pcs_tbl = sm8250_qmp_pcie_pcs_tbl,
- .pcs_tbl_num = ARRAY_SIZE(sm8250_qmp_pcie_pcs_tbl),
- .pcs_tbl_sec = sm8250_qmp_gen3x2_pcie_pcs_tbl,
- .pcs_tbl_num_sec = ARRAY_SIZE(sm8250_qmp_gen3x2_pcie_pcs_tbl),
- .pcs_misc_tbl = sm8250_qmp_pcie_pcs_misc_tbl,
- .pcs_misc_tbl_num = ARRAY_SIZE(sm8250_qmp_pcie_pcs_misc_tbl),
- .pcs_misc_tbl_sec = sm8250_qmp_gen3x2_pcie_pcs_misc_tbl,
- .pcs_misc_tbl_num_sec = ARRAY_SIZE(sm8250_qmp_gen3x2_pcie_pcs_misc_tbl),
+ .tbls = {
+ .serdes = sm8250_qmp_pcie_serdes_tbl,
+ .serdes_num = ARRAY_SIZE(sm8250_qmp_pcie_serdes_tbl),
+ .tx = sm8250_qmp_pcie_tx_tbl,
+ .tx_num = ARRAY_SIZE(sm8250_qmp_pcie_tx_tbl),
+ .rx = sm8250_qmp_pcie_rx_tbl,
+ .rx_num = ARRAY_SIZE(sm8250_qmp_pcie_rx_tbl),
+ .pcs = sm8250_qmp_pcie_pcs_tbl,
+ .pcs_num = ARRAY_SIZE(sm8250_qmp_pcie_pcs_tbl),
+ .pcs_misc = sm8250_qmp_pcie_pcs_misc_tbl,
+ .pcs_misc_num = ARRAY_SIZE(sm8250_qmp_pcie_pcs_misc_tbl),
+ },
+ .tbls_rc = &(const struct qmp_phy_cfg_tbls) {
+ .tx = sm8250_qmp_gen3x2_pcie_tx_tbl,
+ .tx_num = ARRAY_SIZE(sm8250_qmp_gen3x2_pcie_tx_tbl),
+ .rx = sm8250_qmp_gen3x2_pcie_rx_tbl,
+ .rx_num = ARRAY_SIZE(sm8250_qmp_gen3x2_pcie_rx_tbl),
+ .pcs = sm8250_qmp_gen3x2_pcie_pcs_tbl,
+ .pcs_num = ARRAY_SIZE(sm8250_qmp_gen3x2_pcie_pcs_tbl),
+ .pcs_misc = sm8250_qmp_gen3x2_pcie_pcs_misc_tbl,
+ .pcs_misc_num = ARRAY_SIZE(sm8250_qmp_gen3x2_pcie_pcs_misc_tbl),
+ },
.clk_list = sdm845_pciephy_clk_l,
.num_clks = ARRAY_SIZE(sdm845_pciephy_clk_l),
.reset_list = sdm845_pciephy_reset_l,
@@ -1667,26 +1822,23 @@ static const struct qmp_phy_cfg sm8250_qmp_gen3x2_pciephy_cfg = {
.num_vregs = ARRAY_SIZE(qmp_phy_vreg_l),
.regs = sm8250_pcie_regs_layout,
- .start_ctrl = PCS_START | SERDES_START,
.pwrdn_ctrl = SW_PWRDN | REFCLK_DRV_DSBL,
.phy_status = PHYSTATUS,
-
- .has_pwrdn_delay = true,
- .pwrdn_delay_min = 995, /* us */
- .pwrdn_delay_max = 1005, /* us */
};
static const struct qmp_phy_cfg msm8998_pciephy_cfg = {
.lanes = 1,
- .serdes_tbl = msm8998_pcie_serdes_tbl,
- .serdes_tbl_num = ARRAY_SIZE(msm8998_pcie_serdes_tbl),
- .tx_tbl = msm8998_pcie_tx_tbl,
- .tx_tbl_num = ARRAY_SIZE(msm8998_pcie_tx_tbl),
- .rx_tbl = msm8998_pcie_rx_tbl,
- .rx_tbl_num = ARRAY_SIZE(msm8998_pcie_rx_tbl),
- .pcs_tbl = msm8998_pcie_pcs_tbl,
- .pcs_tbl_num = ARRAY_SIZE(msm8998_pcie_pcs_tbl),
+ .tbls = {
+ .serdes = msm8998_pcie_serdes_tbl,
+ .serdes_num = ARRAY_SIZE(msm8998_pcie_serdes_tbl),
+ .tx = msm8998_pcie_tx_tbl,
+ .tx_num = ARRAY_SIZE(msm8998_pcie_tx_tbl),
+ .rx = msm8998_pcie_rx_tbl,
+ .rx_num = ARRAY_SIZE(msm8998_pcie_rx_tbl),
+ .pcs = msm8998_pcie_pcs_tbl,
+ .pcs_num = ARRAY_SIZE(msm8998_pcie_pcs_tbl),
+ },
.clk_list = msm8996_phy_clk_l,
.num_clks = ARRAY_SIZE(msm8996_phy_clk_l),
.reset_list = ipq8074_pciephy_reset_l,
@@ -1695,24 +1847,27 @@ static const struct qmp_phy_cfg msm8998_pciephy_cfg = {
.num_vregs = ARRAY_SIZE(qmp_phy_vreg_l),
.regs = pciephy_regs_layout,
- .start_ctrl = SERDES_START | PCS_START,
.pwrdn_ctrl = SW_PWRDN | REFCLK_DRV_DSBL,
.phy_status = PHYSTATUS,
+
+ .skip_start_delay = true,
};
static const struct qmp_phy_cfg sc8180x_pciephy_cfg = {
.lanes = 1,
- .serdes_tbl = sc8180x_qmp_pcie_serdes_tbl,
- .serdes_tbl_num = ARRAY_SIZE(sc8180x_qmp_pcie_serdes_tbl),
- .tx_tbl = sc8180x_qmp_pcie_tx_tbl,
- .tx_tbl_num = ARRAY_SIZE(sc8180x_qmp_pcie_tx_tbl),
- .rx_tbl = sc8180x_qmp_pcie_rx_tbl,
- .rx_tbl_num = ARRAY_SIZE(sc8180x_qmp_pcie_rx_tbl),
- .pcs_tbl = sc8180x_qmp_pcie_pcs_tbl,
- .pcs_tbl_num = ARRAY_SIZE(sc8180x_qmp_pcie_pcs_tbl),
- .pcs_misc_tbl = sc8180x_qmp_pcie_pcs_misc_tbl,
- .pcs_misc_tbl_num = ARRAY_SIZE(sc8180x_qmp_pcie_pcs_misc_tbl),
+ .tbls = {
+ .serdes = sc8180x_qmp_pcie_serdes_tbl,
+ .serdes_num = ARRAY_SIZE(sc8180x_qmp_pcie_serdes_tbl),
+ .tx = sc8180x_qmp_pcie_tx_tbl,
+ .tx_num = ARRAY_SIZE(sc8180x_qmp_pcie_tx_tbl),
+ .rx = sc8180x_qmp_pcie_rx_tbl,
+ .rx_num = ARRAY_SIZE(sc8180x_qmp_pcie_rx_tbl),
+ .pcs = sc8180x_qmp_pcie_pcs_tbl,
+ .pcs_num = ARRAY_SIZE(sc8180x_qmp_pcie_pcs_tbl),
+ .pcs_misc = sc8180x_qmp_pcie_pcs_misc_tbl,
+ .pcs_misc_num = ARRAY_SIZE(sc8180x_qmp_pcie_pcs_misc_tbl),
+ },
.clk_list = sdm845_pciephy_clk_l,
.num_clks = ARRAY_SIZE(sdm845_pciephy_clk_l),
.reset_list = sdm845_pciephy_reset_l,
@@ -1721,27 +1876,133 @@ static const struct qmp_phy_cfg sc8180x_pciephy_cfg = {
.num_vregs = ARRAY_SIZE(qmp_phy_vreg_l),
.regs = sm8250_pcie_regs_layout,
- .start_ctrl = PCS_START | SERDES_START,
.pwrdn_ctrl = SW_PWRDN | REFCLK_DRV_DSBL,
+ .phy_status = PHYSTATUS,
+};
- .has_pwrdn_delay = true,
- .pwrdn_delay_min = 995, /* us */
- .pwrdn_delay_max = 1005, /* us */
+static const struct qmp_phy_cfg sc8280xp_qmp_gen3x1_pciephy_cfg = {
+ .lanes = 1,
+
+ .offsets = &qmp_pcie_offsets_v5,
+
+ .tbls = {
+ .serdes = sc8280xp_qmp_pcie_serdes_tbl,
+ .serdes_num = ARRAY_SIZE(sc8280xp_qmp_pcie_serdes_tbl),
+ .tx = sc8280xp_qmp_gen3x1_pcie_tx_tbl,
+ .tx_num = ARRAY_SIZE(sc8280xp_qmp_gen3x1_pcie_tx_tbl),
+ .rx = sc8280xp_qmp_gen3x1_pcie_rx_tbl,
+ .rx_num = ARRAY_SIZE(sc8280xp_qmp_gen3x1_pcie_rx_tbl),
+ .pcs = sc8280xp_qmp_gen3x1_pcie_pcs_tbl,
+ .pcs_num = ARRAY_SIZE(sc8280xp_qmp_gen3x1_pcie_pcs_tbl),
+ .pcs_misc = sc8280xp_qmp_gen3x1_pcie_pcs_misc_tbl,
+ .pcs_misc_num = ARRAY_SIZE(sc8280xp_qmp_gen3x1_pcie_pcs_misc_tbl),
+ },
+
+ .tbls_rc = &(const struct qmp_phy_cfg_tbls) {
+ .serdes = sc8280xp_qmp_gen3x1_pcie_rc_serdes_tbl,
+ .serdes_num = ARRAY_SIZE(sc8280xp_qmp_gen3x1_pcie_rc_serdes_tbl),
+ },
+
+ .clk_list = sc8280xp_pciephy_clk_l,
+ .num_clks = ARRAY_SIZE(sc8280xp_pciephy_clk_l),
+ .reset_list = sdm845_pciephy_reset_l,
+ .num_resets = ARRAY_SIZE(sdm845_pciephy_reset_l),
+ .vreg_list = qmp_phy_vreg_l,
+ .num_vregs = ARRAY_SIZE(qmp_phy_vreg_l),
+ .regs = sm8250_pcie_regs_layout,
+
+ .pwrdn_ctrl = SW_PWRDN | REFCLK_DRV_DSBL,
+ .phy_status = PHYSTATUS,
+};
+
+static const struct qmp_phy_cfg sc8280xp_qmp_gen3x2_pciephy_cfg = {
+ .lanes = 2,
+
+ .offsets = &qmp_pcie_offsets_v5,
+
+ .tbls = {
+ .serdes = sc8280xp_qmp_pcie_serdes_tbl,
+ .serdes_num = ARRAY_SIZE(sc8280xp_qmp_pcie_serdes_tbl),
+ .tx = sc8280xp_qmp_gen3x2_pcie_tx_tbl,
+ .tx_num = ARRAY_SIZE(sc8280xp_qmp_gen3x2_pcie_tx_tbl),
+ .rx = sc8280xp_qmp_gen3x2_pcie_rx_tbl,
+ .rx_num = ARRAY_SIZE(sc8280xp_qmp_gen3x2_pcie_rx_tbl),
+ .pcs = sc8280xp_qmp_gen3x2_pcie_pcs_tbl,
+ .pcs_num = ARRAY_SIZE(sc8280xp_qmp_gen3x2_pcie_pcs_tbl),
+ .pcs_misc = sc8280xp_qmp_gen3x2_pcie_pcs_misc_tbl,
+ .pcs_misc_num = ARRAY_SIZE(sc8280xp_qmp_gen3x2_pcie_pcs_misc_tbl),
+ },
+
+ .tbls_rc = &(const struct qmp_phy_cfg_tbls) {
+ .serdes = sc8280xp_qmp_gen3x2_pcie_rc_serdes_tbl,
+ .serdes_num = ARRAY_SIZE(sc8280xp_qmp_gen3x2_pcie_rc_serdes_tbl),
+ },
+
+ .clk_list = sc8280xp_pciephy_clk_l,
+ .num_clks = ARRAY_SIZE(sc8280xp_pciephy_clk_l),
+ .reset_list = sdm845_pciephy_reset_l,
+ .num_resets = ARRAY_SIZE(sdm845_pciephy_reset_l),
+ .vreg_list = qmp_phy_vreg_l,
+ .num_vregs = ARRAY_SIZE(qmp_phy_vreg_l),
+ .regs = sm8250_pcie_regs_layout,
+
+ .pwrdn_ctrl = SW_PWRDN | REFCLK_DRV_DSBL,
+ .phy_status = PHYSTATUS,
+};
+
+static const struct qmp_phy_cfg sc8280xp_qmp_gen3x4_pciephy_cfg = {
+ .lanes = 4,
+
+ .offsets = &qmp_pcie_offsets_v5,
+
+ .tbls = {
+ .serdes = sc8280xp_qmp_pcie_serdes_tbl,
+ .serdes_num = ARRAY_SIZE(sc8280xp_qmp_pcie_serdes_tbl),
+ .tx = sc8280xp_qmp_gen3x2_pcie_tx_tbl,
+ .tx_num = ARRAY_SIZE(sc8280xp_qmp_gen3x2_pcie_tx_tbl),
+ .rx = sc8280xp_qmp_gen3x2_pcie_rx_tbl,
+ .rx_num = ARRAY_SIZE(sc8280xp_qmp_gen3x2_pcie_rx_tbl),
+ .pcs = sc8280xp_qmp_gen3x2_pcie_pcs_tbl,
+ .pcs_num = ARRAY_SIZE(sc8280xp_qmp_gen3x2_pcie_pcs_tbl),
+ .pcs_misc = sc8280xp_qmp_gen3x2_pcie_pcs_misc_tbl,
+ .pcs_misc_num = ARRAY_SIZE(sc8280xp_qmp_gen3x2_pcie_pcs_misc_tbl),
+ },
+
+ .tbls_rc = &(const struct qmp_phy_cfg_tbls) {
+ .serdes = sc8280xp_qmp_gen3x2_pcie_rc_serdes_tbl,
+ .serdes_num = ARRAY_SIZE(sc8280xp_qmp_gen3x2_pcie_rc_serdes_tbl),
+ },
+
+ .serdes_4ln_tbl = sc8280xp_qmp_gen3x4_pcie_serdes_4ln_tbl,
+ .serdes_4ln_num = ARRAY_SIZE(sc8280xp_qmp_gen3x4_pcie_serdes_4ln_tbl),
+
+ .clk_list = sc8280xp_pciephy_clk_l,
+ .num_clks = ARRAY_SIZE(sc8280xp_pciephy_clk_l),
+ .reset_list = sdm845_pciephy_reset_l,
+ .num_resets = ARRAY_SIZE(sdm845_pciephy_reset_l),
+ .vreg_list = qmp_phy_vreg_l,
+ .num_vregs = ARRAY_SIZE(qmp_phy_vreg_l),
+ .regs = sm8250_pcie_regs_layout,
+
+ .pwrdn_ctrl = SW_PWRDN | REFCLK_DRV_DSBL,
+ .phy_status = PHYSTATUS,
};
static const struct qmp_phy_cfg sdx55_qmp_pciephy_cfg = {
.lanes = 2,
- .serdes_tbl = sdx55_qmp_pcie_serdes_tbl,
- .serdes_tbl_num = ARRAY_SIZE(sdx55_qmp_pcie_serdes_tbl),
- .tx_tbl = sdx55_qmp_pcie_tx_tbl,
- .tx_tbl_num = ARRAY_SIZE(sdx55_qmp_pcie_tx_tbl),
- .rx_tbl = sdx55_qmp_pcie_rx_tbl,
- .rx_tbl_num = ARRAY_SIZE(sdx55_qmp_pcie_rx_tbl),
- .pcs_tbl = sdx55_qmp_pcie_pcs_tbl,
- .pcs_tbl_num = ARRAY_SIZE(sdx55_qmp_pcie_pcs_tbl),
- .pcs_misc_tbl = sdx55_qmp_pcie_pcs_misc_tbl,
- .pcs_misc_tbl_num = ARRAY_SIZE(sdx55_qmp_pcie_pcs_misc_tbl),
+ .tbls = {
+ .serdes = sdx55_qmp_pcie_serdes_tbl,
+ .serdes_num = ARRAY_SIZE(sdx55_qmp_pcie_serdes_tbl),
+ .tx = sdx55_qmp_pcie_tx_tbl,
+ .tx_num = ARRAY_SIZE(sdx55_qmp_pcie_tx_tbl),
+ .rx = sdx55_qmp_pcie_rx_tbl,
+ .rx_num = ARRAY_SIZE(sdx55_qmp_pcie_rx_tbl),
+ .pcs = sdx55_qmp_pcie_pcs_tbl,
+ .pcs_num = ARRAY_SIZE(sdx55_qmp_pcie_pcs_tbl),
+ .pcs_misc = sdx55_qmp_pcie_pcs_misc_tbl,
+ .pcs_misc_num = ARRAY_SIZE(sdx55_qmp_pcie_pcs_misc_tbl),
+ },
.clk_list = sdm845_pciephy_clk_l,
.num_clks = ARRAY_SIZE(sdm845_pciephy_clk_l),
.reset_list = sdm845_pciephy_reset_l,
@@ -1750,28 +2011,25 @@ static const struct qmp_phy_cfg sdx55_qmp_pciephy_cfg = {
.num_vregs = ARRAY_SIZE(qmp_phy_vreg_l),
.regs = sm8250_pcie_regs_layout,
- .start_ctrl = PCS_START | SERDES_START,
.pwrdn_ctrl = SW_PWRDN,
.phy_status = PHYSTATUS_4_20,
-
- .has_pwrdn_delay = true,
- .pwrdn_delay_min = 995, /* us */
- .pwrdn_delay_max = 1005, /* us */
};
static const struct qmp_phy_cfg sm8450_qmp_gen3x1_pciephy_cfg = {
.lanes = 1,
- .serdes_tbl = sm8450_qmp_gen3x1_pcie_serdes_tbl,
- .serdes_tbl_num = ARRAY_SIZE(sm8450_qmp_gen3x1_pcie_serdes_tbl),
- .tx_tbl = sm8450_qmp_gen3x1_pcie_tx_tbl,
- .tx_tbl_num = ARRAY_SIZE(sm8450_qmp_gen3x1_pcie_tx_tbl),
- .rx_tbl = sm8450_qmp_gen3x1_pcie_rx_tbl,
- .rx_tbl_num = ARRAY_SIZE(sm8450_qmp_gen3x1_pcie_rx_tbl),
- .pcs_tbl = sm8450_qmp_gen3x1_pcie_pcs_tbl,
- .pcs_tbl_num = ARRAY_SIZE(sm8450_qmp_gen3x1_pcie_pcs_tbl),
- .pcs_misc_tbl = sm8450_qmp_gen3x1_pcie_pcs_misc_tbl,
- .pcs_misc_tbl_num = ARRAY_SIZE(sm8450_qmp_gen3x1_pcie_pcs_misc_tbl),
+ .tbls = {
+ .serdes = sm8450_qmp_gen3x1_pcie_serdes_tbl,
+ .serdes_num = ARRAY_SIZE(sm8450_qmp_gen3x1_pcie_serdes_tbl),
+ .tx = sm8450_qmp_gen3x1_pcie_tx_tbl,
+ .tx_num = ARRAY_SIZE(sm8450_qmp_gen3x1_pcie_tx_tbl),
+ .rx = sm8450_qmp_gen3x1_pcie_rx_tbl,
+ .rx_num = ARRAY_SIZE(sm8450_qmp_gen3x1_pcie_rx_tbl),
+ .pcs = sm8450_qmp_gen3x1_pcie_pcs_tbl,
+ .pcs_num = ARRAY_SIZE(sm8450_qmp_gen3x1_pcie_pcs_tbl),
+ .pcs_misc = sm8450_qmp_gen3x1_pcie_pcs_misc_tbl,
+ .pcs_misc_num = ARRAY_SIZE(sm8450_qmp_gen3x1_pcie_pcs_misc_tbl),
+ },
.clk_list = sdm845_pciephy_clk_l,
.num_clks = ARRAY_SIZE(sdm845_pciephy_clk_l),
.reset_list = sdm845_pciephy_reset_l,
@@ -1780,28 +2038,40 @@ static const struct qmp_phy_cfg sm8450_qmp_gen3x1_pciephy_cfg = {
.num_vregs = ARRAY_SIZE(qmp_phy_vreg_l),
.regs = sm8250_pcie_regs_layout,
- .start_ctrl = SERDES_START | PCS_START,
.pwrdn_ctrl = SW_PWRDN | REFCLK_DRV_DSBL,
.phy_status = PHYSTATUS,
-
- .has_pwrdn_delay = true,
- .pwrdn_delay_min = 995, /* us */
- .pwrdn_delay_max = 1005, /* us */
};
static const struct qmp_phy_cfg sm8450_qmp_gen4x2_pciephy_cfg = {
.lanes = 2,
- .serdes_tbl = sm8450_qmp_gen4x2_pcie_serdes_tbl,
- .serdes_tbl_num = ARRAY_SIZE(sm8450_qmp_gen4x2_pcie_serdes_tbl),
- .tx_tbl = sm8450_qmp_gen4x2_pcie_tx_tbl,
- .tx_tbl_num = ARRAY_SIZE(sm8450_qmp_gen4x2_pcie_tx_tbl),
- .rx_tbl = sm8450_qmp_gen4x2_pcie_rx_tbl,
- .rx_tbl_num = ARRAY_SIZE(sm8450_qmp_gen4x2_pcie_rx_tbl),
- .pcs_tbl = sm8450_qmp_gen4x2_pcie_pcs_tbl,
- .pcs_tbl_num = ARRAY_SIZE(sm8450_qmp_gen4x2_pcie_pcs_tbl),
- .pcs_misc_tbl = sm8450_qmp_gen4x2_pcie_pcs_misc_tbl,
- .pcs_misc_tbl_num = ARRAY_SIZE(sm8450_qmp_gen4x2_pcie_pcs_misc_tbl),
+ .tbls = {
+ .serdes = sm8450_qmp_gen4x2_pcie_serdes_tbl,
+ .serdes_num = ARRAY_SIZE(sm8450_qmp_gen4x2_pcie_serdes_tbl),
+ .tx = sm8450_qmp_gen4x2_pcie_tx_tbl,
+ .tx_num = ARRAY_SIZE(sm8450_qmp_gen4x2_pcie_tx_tbl),
+ .rx = sm8450_qmp_gen4x2_pcie_rx_tbl,
+ .rx_num = ARRAY_SIZE(sm8450_qmp_gen4x2_pcie_rx_tbl),
+ .pcs = sm8450_qmp_gen4x2_pcie_pcs_tbl,
+ .pcs_num = ARRAY_SIZE(sm8450_qmp_gen4x2_pcie_pcs_tbl),
+ .pcs_misc = sm8450_qmp_gen4x2_pcie_pcs_misc_tbl,
+ .pcs_misc_num = ARRAY_SIZE(sm8450_qmp_gen4x2_pcie_pcs_misc_tbl),
+ },
+
+ .tbls_rc = &(const struct qmp_phy_cfg_tbls) {
+ .serdes = sm8450_qmp_gen4x2_pcie_rc_serdes_tbl,
+ .serdes_num = ARRAY_SIZE(sm8450_qmp_gen4x2_pcie_rc_serdes_tbl),
+ .pcs_misc = sm8450_qmp_gen4x2_pcie_rc_pcs_misc_tbl,
+ .pcs_misc_num = ARRAY_SIZE(sm8450_qmp_gen4x2_pcie_rc_pcs_misc_tbl),
+ },
+
+ .tbls_ep = &(const struct qmp_phy_cfg_tbls) {
+ .serdes = sm8450_qmp_gen4x2_pcie_ep_serdes_tbl,
+ .serdes_num = ARRAY_SIZE(sm8450_qmp_gen4x2_pcie_ep_serdes_tbl),
+ .pcs_misc = sm8450_qmp_gen4x2_pcie_ep_pcs_misc_tbl,
+ .pcs_misc_num = ARRAY_SIZE(sm8450_qmp_gen4x2_pcie_ep_pcs_misc_tbl),
+ },
+
.clk_list = sdm845_pciephy_clk_l,
.num_clks = ARRAY_SIZE(sdm845_pciephy_clk_l),
.reset_list = sdm845_pciephy_reset_l,
@@ -1810,17 +2080,11 @@ static const struct qmp_phy_cfg sm8450_qmp_gen4x2_pciephy_cfg = {
.num_vregs = ARRAY_SIZE(qmp_phy_vreg_l),
.regs = sm8250_pcie_regs_layout,
- .start_ctrl = SERDES_START | PCS_START,
.pwrdn_ctrl = SW_PWRDN | REFCLK_DRV_DSBL,
.phy_status = PHYSTATUS_4_20,
-
- .has_pwrdn_delay = true,
- .pwrdn_delay_min = 995, /* us */
- .pwrdn_delay_max = 1005, /* us */
};
static void qmp_pcie_configure_lane(void __iomem *base,
- const unsigned int *regs,
const struct qmp_phy_init_tbl tbl[],
int num,
u8 lane_mask)
@@ -1835,43 +2099,74 @@ static void qmp_pcie_configure_lane(void __iomem *base,
if (!(t->lane_mask & lane_mask))
continue;
- if (t->in_layout)
- writel(t->val, base + regs[t->offset]);
- else
- writel(t->val, base + t->offset);
+ writel(t->val, base + t->offset);
}
}
static void qmp_pcie_configure(void __iomem *base,
- const unsigned int *regs,
const struct qmp_phy_init_tbl tbl[],
int num)
{
- qmp_pcie_configure_lane(base, regs, tbl, num, 0xff);
+ qmp_pcie_configure_lane(base, tbl, num, 0xff);
}
-static int qmp_pcie_serdes_init(struct qmp_phy *qphy)
+static void qmp_pcie_init_port_b(struct qmp_pcie *qmp, const struct qmp_phy_cfg_tbls *tbls)
{
- const struct qmp_phy_cfg *cfg = qphy->cfg;
- void __iomem *serdes = qphy->serdes;
- const struct qmp_phy_init_tbl *serdes_tbl = cfg->serdes_tbl;
- int serdes_tbl_num = cfg->serdes_tbl_num;
+ const struct qmp_phy_cfg *cfg = qmp->cfg;
+ const struct qmp_pcie_offsets *offs = cfg->offsets;
+ void __iomem *tx3, *rx3, *tx4, *rx4;
- qmp_pcie_configure(serdes, cfg->regs, serdes_tbl, serdes_tbl_num);
- qmp_pcie_configure(serdes, cfg->regs, cfg->serdes_tbl_sec, cfg->serdes_tbl_num_sec);
+ tx3 = qmp->port_b + offs->tx;
+ rx3 = qmp->port_b + offs->rx;
+ tx4 = qmp->port_b + offs->tx2;
+ rx4 = qmp->port_b + offs->rx2;
- return 0;
+ qmp_pcie_configure_lane(tx3, tbls->tx, tbls->tx_num, 1);
+ qmp_pcie_configure_lane(rx3, tbls->rx, tbls->rx_num, 1);
+
+ qmp_pcie_configure_lane(tx4, tbls->tx, tbls->tx_num, 2);
+ qmp_pcie_configure_lane(rx4, tbls->rx, tbls->rx_num, 2);
+}
+
+static void qmp_pcie_init_registers(struct qmp_pcie *qmp, const struct qmp_phy_cfg_tbls *tbls)
+{
+ const struct qmp_phy_cfg *cfg = qmp->cfg;
+ void __iomem *serdes = qmp->serdes;
+ void __iomem *tx = qmp->tx;
+ void __iomem *rx = qmp->rx;
+ void __iomem *tx2 = qmp->tx2;
+ void __iomem *rx2 = qmp->rx2;
+ void __iomem *pcs = qmp->pcs;
+ void __iomem *pcs_misc = qmp->pcs_misc;
+
+ if (!tbls)
+ return;
+
+ qmp_pcie_configure(serdes, tbls->serdes, tbls->serdes_num);
+
+ qmp_pcie_configure_lane(tx, tbls->tx, tbls->tx_num, 1);
+ qmp_pcie_configure_lane(rx, tbls->rx, tbls->rx_num, 1);
+
+ if (cfg->lanes >= 2) {
+ qmp_pcie_configure_lane(tx2, tbls->tx, tbls->tx_num, 2);
+ qmp_pcie_configure_lane(rx2, tbls->rx, tbls->rx_num, 2);
+ }
+
+ qmp_pcie_configure(pcs, tbls->pcs, tbls->pcs_num);
+ qmp_pcie_configure(pcs_misc, tbls->pcs_misc, tbls->pcs_misc_num);
+
+ if (cfg->lanes >= 4 && qmp->tcsr_4ln_config) {
+ qmp_pcie_configure(serdes, cfg->serdes_4ln_tbl, cfg->serdes_4ln_num);
+ qmp_pcie_init_port_b(qmp, tbls);
+ }
}
static int qmp_pcie_init(struct phy *phy)
{
- struct qmp_phy *qphy = phy_get_drvdata(phy);
- struct qcom_qmp *qmp = qphy->qmp;
- const struct qmp_phy_cfg *cfg = qphy->cfg;
- void __iomem *pcs = qphy->pcs;
+ struct qmp_pcie *qmp = phy_get_drvdata(phy);
+ const struct qmp_phy_cfg *cfg = qmp->cfg;
int ret;
- /* turn on regulator supplies */
ret = regulator_bulk_enable(cfg->num_vregs, qmp->vregs);
if (ret) {
dev_err(qmp->dev, "failed to enable regulators, err=%d\n", ret);
@@ -1884,6 +2179,8 @@ static int qmp_pcie_init(struct phy *phy)
goto err_disable_regulators;
}
+ usleep_range(200, 300);
+
ret = reset_control_bulk_deassert(cfg->num_resets, qmp->resets);
if (ret) {
dev_err(qmp->dev, "reset deassert failed\n");
@@ -1894,14 +2191,6 @@ static int qmp_pcie_init(struct phy *phy)
if (ret)
goto err_assert_reset;
- if (cfg->regs[QPHY_PCS_POWER_DOWN_CONTROL])
- qphy_setbits(pcs,
- cfg->regs[QPHY_PCS_POWER_DOWN_CONTROL],
- cfg->pwrdn_ctrl);
- else
- qphy_setbits(pcs, QPHY_V2_PCS_POWER_DOWN_CONTROL,
- cfg->pwrdn_ctrl);
-
return 0;
err_assert_reset:
@@ -1914,9 +2203,8 @@ static int qmp_pcie_init(struct phy *phy)
static int qmp_pcie_exit(struct phy *phy)
{
- struct qmp_phy *qphy = phy_get_drvdata(phy);
- struct qcom_qmp *qmp = qphy->qmp;
- const struct qmp_phy_cfg *cfg = qphy->cfg;
+ struct qmp_pcie *qmp = phy_get_drvdata(phy);
+ const struct qmp_phy_cfg *cfg = qmp->cfg;
reset_control_bulk_assert(cfg->num_resets, qmp->resets);
@@ -1929,72 +2217,41 @@ static int qmp_pcie_exit(struct phy *phy)
static int qmp_pcie_power_on(struct phy *phy)
{
- struct qmp_phy *qphy = phy_get_drvdata(phy);
- struct qcom_qmp *qmp = qphy->qmp;
- const struct qmp_phy_cfg *cfg = qphy->cfg;
- void __iomem *tx = qphy->tx;
- void __iomem *rx = qphy->rx;
- void __iomem *pcs = qphy->pcs;
- void __iomem *pcs_misc = qphy->pcs_misc;
+ struct qmp_pcie *qmp = phy_get_drvdata(phy);
+ const struct qmp_phy_cfg *cfg = qmp->cfg;
+ const struct qmp_phy_cfg_tbls *mode_tbls;
+ void __iomem *pcs = qmp->pcs;
void __iomem *status;
- unsigned int mask, val, ready;
+ unsigned int mask, val;
int ret;
- qmp_pcie_serdes_init(qphy);
+ qphy_setbits(pcs, cfg->regs[QPHY_PCS_POWER_DOWN_CONTROL],
+ cfg->pwrdn_ctrl);
- ret = clk_prepare_enable(qphy->pipe_clk);
- if (ret) {
- dev_err(qmp->dev, "pipe_clk enable failed err=%d\n", ret);
+ if (qmp->mode == PHY_MODE_PCIE_RC)
+ mode_tbls = cfg->tbls_rc;
+ else
+ mode_tbls = cfg->tbls_ep;
+
+ qmp_pcie_init_registers(qmp, &cfg->tbls);
+ qmp_pcie_init_registers(qmp, mode_tbls);
+
+ ret = clk_bulk_prepare_enable(qmp->num_pipe_clks, qmp->pipe_clks);
+ if (ret)
return ret;
- }
-
- /* Tx, Rx, and PCS configurations */
- qmp_pcie_configure_lane(tx, cfg->regs, cfg->tx_tbl, cfg->tx_tbl_num, 1);
- qmp_pcie_configure_lane(tx, cfg->regs, cfg->tx_tbl_sec, cfg->tx_tbl_num_sec, 1);
-
- if (cfg->lanes >= 2) {
- qmp_pcie_configure_lane(qphy->tx2, cfg->regs, cfg->tx_tbl,
- cfg->tx_tbl_num, 2);
- qmp_pcie_configure_lane(qphy->tx2, cfg->regs, cfg->tx_tbl_sec,
- cfg->tx_tbl_num_sec, 2);
- }
-
- qmp_pcie_configure_lane(rx, cfg->regs, cfg->rx_tbl, cfg->rx_tbl_num, 1);
- qmp_pcie_configure_lane(rx, cfg->regs, cfg->rx_tbl_sec, cfg->rx_tbl_num_sec, 1);
-
- if (cfg->lanes >= 2) {
- qmp_pcie_configure_lane(qphy->rx2, cfg->regs, cfg->rx_tbl,
- cfg->rx_tbl_num, 2);
- qmp_pcie_configure_lane(qphy->rx2, cfg->regs, cfg->rx_tbl_sec,
- cfg->rx_tbl_num_sec, 2);
- }
-
- qmp_pcie_configure(pcs, cfg->regs, cfg->pcs_tbl, cfg->pcs_tbl_num);
- qmp_pcie_configure(pcs, cfg->regs, cfg->pcs_tbl_sec, cfg->pcs_tbl_num_sec);
-
- qmp_pcie_configure(pcs_misc, cfg->regs, cfg->pcs_misc_tbl, cfg->pcs_misc_tbl_num);
- qmp_pcie_configure(pcs_misc, cfg->regs, cfg->pcs_misc_tbl_sec, cfg->pcs_misc_tbl_num_sec);
-
- /*
- * Pull out PHY from POWER DOWN state.
- * This is active low enable signal to power-down PHY.
- */
- qphy_setbits(pcs, QPHY_V2_PCS_POWER_DOWN_CONTROL, cfg->pwrdn_ctrl);
-
- if (cfg->has_pwrdn_delay)
- usleep_range(cfg->pwrdn_delay_min, cfg->pwrdn_delay_max);
/* Pull PHY out of reset state */
qphy_clrbits(pcs, cfg->regs[QPHY_SW_RESET], SW_RESET);
/* start SerDes and Phy-Coding-Sublayer */
- qphy_setbits(pcs, cfg->regs[QPHY_START_CTRL], cfg->start_ctrl);
+ qphy_setbits(pcs, cfg->regs[QPHY_START_CTRL], SERDES_START | PCS_START);
+
+ if (!cfg->skip_start_delay)
+ usleep_range(1000, 1200);
status = pcs + cfg->regs[QPHY_PCS_STATUS];
mask = cfg->phy_status;
- ready = 0;
-
- ret = readl_poll_timeout(status, val, (val & mask) == ready, 10,
+ ret = readl_poll_timeout(status, val, !(val & mask), 200,
PHY_INIT_COMPLETE_TIMEOUT);
if (ret) {
dev_err(qmp->dev, "phy initialization timed-out\n");
@@ -2004,32 +2261,28 @@ static int qmp_pcie_power_on(struct phy *phy)
return 0;
err_disable_pipe_clk:
- clk_disable_unprepare(qphy->pipe_clk);
+ clk_bulk_disable_unprepare(qmp->num_pipe_clks, qmp->pipe_clks);
return ret;
}
static int qmp_pcie_power_off(struct phy *phy)
{
- struct qmp_phy *qphy = phy_get_drvdata(phy);
- const struct qmp_phy_cfg *cfg = qphy->cfg;
+ struct qmp_pcie *qmp = phy_get_drvdata(phy);
+ const struct qmp_phy_cfg *cfg = qmp->cfg;
- clk_disable_unprepare(qphy->pipe_clk);
+ clk_bulk_disable_unprepare(qmp->num_pipe_clks, qmp->pipe_clks);
/* PHY reset */
- qphy_setbits(qphy->pcs, cfg->regs[QPHY_SW_RESET], SW_RESET);
+ qphy_setbits(qmp->pcs, cfg->regs[QPHY_SW_RESET], SW_RESET);
/* stop SerDes and Phy-Coding-Sublayer */
- qphy_clrbits(qphy->pcs, cfg->regs[QPHY_START_CTRL], cfg->start_ctrl);
+ qphy_clrbits(qmp->pcs, cfg->regs[QPHY_START_CTRL],
+ SERDES_START | PCS_START);
/* Put PHY into POWER DOWN state: active low */
- if (cfg->regs[QPHY_PCS_POWER_DOWN_CONTROL]) {
- qphy_clrbits(qphy->pcs, cfg->regs[QPHY_PCS_POWER_DOWN_CONTROL],
- cfg->pwrdn_ctrl);
- } else {
- qphy_clrbits(qphy->pcs, QPHY_V2_PCS_POWER_DOWN_CONTROL,
- cfg->pwrdn_ctrl);
- }
+ qphy_clrbits(qmp->pcs, cfg->regs[QPHY_PCS_POWER_DOWN_CONTROL],
+ cfg->pwrdn_ctrl);
return 0;
}
@@ -2060,9 +2313,34 @@ static int qmp_pcie_disable(struct phy *phy)
return qmp_pcie_exit(phy);
}
-static int qmp_pcie_vreg_init(struct device *dev, const struct qmp_phy_cfg *cfg)
+static int qmp_pcie_set_mode(struct phy *phy, enum phy_mode mode, int submode)
{
- struct qcom_qmp *qmp = dev_get_drvdata(dev);
+ struct qmp_pcie *qmp = phy_get_drvdata(phy);
+
+ switch (submode) {
+ case PHY_MODE_PCIE_RC:
+ case PHY_MODE_PCIE_EP:
+ qmp->mode = submode;
+ break;
+ default:
+ dev_err(&phy->dev, "Unsupported submode %d\n", submode);
+ return -EINVAL;
+ }
+
+ return 0;
+}
+
+static const struct phy_ops qmp_pcie_phy_ops = {
+ .power_on = qmp_pcie_enable,
+ .power_off = qmp_pcie_disable,
+ .set_mode = qmp_pcie_set_mode,
+ .owner = THIS_MODULE,
+};
+
+static int qmp_pcie_vreg_init(struct qmp_pcie *qmp)
+{
+ const struct qmp_phy_cfg *cfg = qmp->cfg;
+ struct device *dev = qmp->dev;
int num = cfg->num_vregs;
int i;
@@ -2076,9 +2354,10 @@ static int qmp_pcie_vreg_init(struct device *dev, const struct qmp_phy_cfg *cfg)
return devm_regulator_bulk_get(dev, num, qmp->vregs);
}
-static int qmp_pcie_reset_init(struct device *dev, const struct qmp_phy_cfg *cfg)
+static int qmp_pcie_reset_init(struct qmp_pcie *qmp)
{
- struct qcom_qmp *qmp = dev_get_drvdata(dev);
+ const struct qmp_phy_cfg *cfg = qmp->cfg;
+ struct device *dev = qmp->dev;
int i;
int ret;
@@ -2097,9 +2376,10 @@ static int qmp_pcie_reset_init(struct device *dev, const struct qmp_phy_cfg *cfg
return 0;
}
-static int qmp_pcie_clk_init(struct device *dev, const struct qmp_phy_cfg *cfg)
+static int qmp_pcie_clk_init(struct qmp_pcie *qmp)
{
- struct qcom_qmp *qmp = dev_get_drvdata(dev);
+ const struct qmp_phy_cfg *cfg = qmp->cfg;
+ struct device *dev = qmp->dev;
int num = cfg->num_clks;
int i;
@@ -2136,9 +2416,9 @@ static void phy_clk_release_provider(void *res)
* clk | +-------+ | +-----+
* +---------------+
*/
-static int phy_pipe_clk_register(struct qcom_qmp *qmp, struct device_node *np)
+static int phy_pipe_clk_register(struct qmp_pcie *qmp, struct device_node *np)
{
- struct clk_fixed_rate *fixed;
+ struct clk_fixed_rate *fixed = &qmp->pipe_clk_fixed;
struct clk_init_data init = { };
int ret;
@@ -2148,18 +2428,14 @@ static int phy_pipe_clk_register(struct qcom_qmp *qmp, struct device_node *np)
return ret;
}
- fixed = devm_kzalloc(qmp->dev, sizeof(*fixed), GFP_KERNEL);
- if (!fixed)
- return -ENOMEM;
-
init.ops = &clk_fixed_rate_ops;
/*
* Controllers using QMP PHY-s use 125MHz pipe clock interface
* unless other frequency is specified in the PHY config.
*/
- if (qmp->phys[0]->cfg->pipe_clock_rate)
- fixed->fixed_rate = qmp->phys[0]->cfg->pipe_clock_rate;
+ if (qmp->cfg->pipe_clock_rate)
+ fixed->fixed_rate = qmp->cfg->pipe_clock_rate;
else
fixed->fixed_rate = 125000000;
@@ -2180,114 +2456,261 @@ static int phy_pipe_clk_register(struct qcom_qmp *qmp, struct device_node *np)
return devm_add_action_or_reset(qmp->dev, phy_clk_release_provider, np);
}
-static const struct phy_ops qmp_pcie_ops = {
- .power_on = qmp_pcie_enable,
- .power_off = qmp_pcie_disable,
- .owner = THIS_MODULE,
-};
-
-static int qmp_pcie_create(struct device *dev, struct device_node *np, int id,
- void __iomem *serdes, const struct qmp_phy_cfg *cfg)
+static int qmp_pcie_parse_dt_legacy(struct qmp_pcie *qmp, struct device_node *np)
{
- struct qcom_qmp *qmp = dev_get_drvdata(dev);
- struct phy *generic_phy;
- struct qmp_phy *qphy;
- int ret;
+ struct platform_device *pdev = to_platform_device(qmp->dev);
+ const struct qmp_phy_cfg *cfg = qmp->cfg;
+ struct device *dev = qmp->dev;
+ struct clk *clk;
- qphy = devm_kzalloc(dev, sizeof(*qphy), GFP_KERNEL);
- if (!qphy)
- return -ENOMEM;
+ qmp->serdes = devm_platform_ioremap_resource(pdev, 0);
+ if (IS_ERR(qmp->serdes))
+ return PTR_ERR(qmp->serdes);
- qphy->cfg = cfg;
- qphy->serdes = serdes;
/*
- * Get memory resources for each phy lane:
+ * Get memory resources for the PHY:
* Resources are indexed as: tx -> 0; rx -> 1; pcs -> 2.
* For dual lane PHYs: tx2 -> 3, rx2 -> 4, pcs_misc (optional) -> 5
* For single lane PHYs: pcs_misc (optional) -> 3.
*/
- qphy->tx = devm_of_iomap(dev, np, 0, NULL);
- if (IS_ERR(qphy->tx))
- return PTR_ERR(qphy->tx);
+ qmp->tx = devm_of_iomap(dev, np, 0, NULL);
+ if (IS_ERR(qmp->tx))
+ return PTR_ERR(qmp->tx);
if (of_device_is_compatible(dev->of_node, "qcom,sdm845-qhp-pcie-phy"))
- qphy->rx = qphy->tx;
+ qmp->rx = qmp->tx;
else
- qphy->rx = devm_of_iomap(dev, np, 1, NULL);
- if (IS_ERR(qphy->rx))
- return PTR_ERR(qphy->rx);
+ qmp->rx = devm_of_iomap(dev, np, 1, NULL);
+ if (IS_ERR(qmp->rx))
+ return PTR_ERR(qmp->rx);
- qphy->pcs = devm_of_iomap(dev, np, 2, NULL);
- if (IS_ERR(qphy->pcs))
- return PTR_ERR(qphy->pcs);
+ qmp->pcs = devm_of_iomap(dev, np, 2, NULL);
+ if (IS_ERR(qmp->pcs))
+ return PTR_ERR(qmp->pcs);
if (cfg->lanes >= 2) {
- qphy->tx2 = devm_of_iomap(dev, np, 3, NULL);
- if (IS_ERR(qphy->tx2))
- return PTR_ERR(qphy->tx2);
+ qmp->tx2 = devm_of_iomap(dev, np, 3, NULL);
+ if (IS_ERR(qmp->tx2))
+ return PTR_ERR(qmp->tx2);
- qphy->rx2 = devm_of_iomap(dev, np, 4, NULL);
- if (IS_ERR(qphy->rx2))
- return PTR_ERR(qphy->rx2);
+ qmp->rx2 = devm_of_iomap(dev, np, 4, NULL);
+ if (IS_ERR(qmp->rx2))
+ return PTR_ERR(qmp->rx2);
- qphy->pcs_misc = devm_of_iomap(dev, np, 5, NULL);
+ qmp->pcs_misc = devm_of_iomap(dev, np, 5, NULL);
} else {
- qphy->pcs_misc = devm_of_iomap(dev, np, 3, NULL);
+ qmp->pcs_misc = devm_of_iomap(dev, np, 3, NULL);
}
- if (IS_ERR(qphy->pcs_misc) &&
+ if (IS_ERR(qmp->pcs_misc) &&
of_device_is_compatible(dev->of_node, "qcom,ipq6018-qmp-pcie-phy"))
- qphy->pcs_misc = qphy->pcs + 0x400;
+ qmp->pcs_misc = qmp->pcs + 0x400;
- if (IS_ERR(qphy->pcs_misc)) {
- if (cfg->pcs_misc_tbl || cfg->pcs_misc_tbl_sec)
- return PTR_ERR(qphy->pcs_misc);
+ if (IS_ERR(qmp->pcs_misc)) {
+ if (cfg->tbls.pcs_misc ||
+ (cfg->tbls_rc && cfg->tbls_rc->pcs_misc) ||
+ (cfg->tbls_ep && cfg->tbls_ep->pcs_misc)) {
+ return PTR_ERR(qmp->pcs_misc);
+ }
}
- qphy->pipe_clk = devm_get_clk_from_child(dev, np, NULL);
- if (IS_ERR(qphy->pipe_clk)) {
- return dev_err_probe(dev, PTR_ERR(qphy->pipe_clk),
- "failed to get lane%d pipe clock\n", id);
+ clk = devm_get_clk_from_child(dev, np, NULL);
+ if (IS_ERR(clk)) {
+ return dev_err_probe(dev, PTR_ERR(clk),
+ "failed to get pipe clock\n");
}
- generic_phy = devm_phy_create(dev, np, &qmp_pcie_ops);
- if (IS_ERR(generic_phy)) {
- ret = PTR_ERR(generic_phy);
- dev_err(dev, "failed to create qphy %d\n", ret);
- return ret;
- }
-
- qphy->phy = generic_phy;
- qphy->qmp = qmp;
- qmp->phys[id] = qphy;
- phy_set_drvdata(generic_phy, qphy);
+ qmp->num_pipe_clks = 1;
+ qmp->pipe_clks[0].id = "pipe";
+ qmp->pipe_clks[0].clk = clk;
return 0;
}
+static int qmp_pcie_get_4ln_config(struct qmp_pcie *qmp)
+{
+ struct regmap *tcsr;
+ unsigned int args[2];
+ int ret;
+
+ tcsr = syscon_regmap_lookup_by_phandle_args(qmp->dev->of_node,
+ "qcom,4ln-config-sel",
+ ARRAY_SIZE(args), args);
+ if (IS_ERR(tcsr)) {
+ ret = PTR_ERR(tcsr);
+ if (ret == -ENOENT)
+ return 0;
+
+ dev_err(qmp->dev, "failed to lookup syscon: %d\n", ret);
+ return ret;
+ }
+
+ ret = regmap_test_bits(tcsr, args[0], BIT(args[1]));
+ if (ret < 0) {
+ dev_err(qmp->dev, "failed to read tcsr: %d\n", ret);
+ return ret;
+ }
+
+ qmp->tcsr_4ln_config = ret;
+
+ dev_dbg(qmp->dev, "4ln_config_sel = %d\n", qmp->tcsr_4ln_config);
+
+ return 0;
+}
+
+static int qmp_pcie_parse_dt(struct qmp_pcie *qmp)
+{
+ struct platform_device *pdev = to_platform_device(qmp->dev);
+ const struct qmp_phy_cfg *cfg = qmp->cfg;
+ const struct qmp_pcie_offsets *offs = cfg->offsets;
+ struct device *dev = qmp->dev;
+ void __iomem *base;
+ int ret;
+
+ if (!offs)
+ return -EINVAL;
+
+ ret = qmp_pcie_get_4ln_config(qmp);
+ if (ret)
+ return ret;
+
+ base = devm_platform_ioremap_resource(pdev, 0);
+ if (IS_ERR(base))
+ return PTR_ERR(base);
+
+ qmp->serdes = base + offs->serdes;
+ qmp->pcs = base + offs->pcs;
+ qmp->pcs_misc = base + offs->pcs_misc;
+ qmp->tx = base + offs->tx;
+ qmp->rx = base + offs->rx;
+
+ if (cfg->lanes >= 2) {
+ qmp->tx2 = base + offs->tx2;
+ qmp->rx2 = base + offs->rx2;
+ }
+
+ if (qmp->cfg->lanes >= 4 && qmp->tcsr_4ln_config) {
+ qmp->port_b = devm_platform_ioremap_resource(pdev, 1);
+ if (IS_ERR(qmp->port_b))
+ return PTR_ERR(qmp->port_b);
+ }
+
+ qmp->num_pipe_clks = 2;
+ qmp->pipe_clks[0].id = "pipe";
+ qmp->pipe_clks[1].id = "pipediv2";
+
+ ret = devm_clk_bulk_get(dev, qmp->num_pipe_clks, qmp->pipe_clks);
+ if (ret)
+ return ret;
+
+ return 0;
+}
+
+static int qmp_pcie_probe(struct platform_device *pdev)
+{
+ struct device *dev = &pdev->dev;
+ struct phy_provider *phy_provider;
+ struct device_node *np;
+ struct qmp_pcie *qmp;
+ int ret;
+
+ qmp = devm_kzalloc(dev, sizeof(*qmp), GFP_KERNEL);
+ if (!qmp)
+ return -ENOMEM;
+
+ qmp->dev = dev;
+
+ qmp->cfg = of_device_get_match_data(dev);
+ if (!qmp->cfg)
+ return -EINVAL;
+
+ WARN_ON_ONCE(!qmp->cfg->pwrdn_ctrl);
+ WARN_ON_ONCE(!qmp->cfg->phy_status);
+
+ ret = qmp_pcie_clk_init(qmp);
+ if (ret)
+ return ret;
+
+ ret = qmp_pcie_reset_init(qmp);
+ if (ret)
+ return ret;
+
+ ret = qmp_pcie_vreg_init(qmp);
+ if (ret)
+ return ret;
+
+ /* Check for legacy binding with child node. */
+ np = of_get_next_available_child(dev->of_node, NULL);
+ if (np) {
+ ret = qmp_pcie_parse_dt_legacy(qmp, np);
+ } else {
+ np = of_node_get(dev->of_node);
+ ret = qmp_pcie_parse_dt(qmp);
+ }
+ if (ret)
+ goto err_node_put;
+
+ ret = phy_pipe_clk_register(qmp, np);
+ if (ret)
+ goto err_node_put;
+
+ qmp->mode = PHY_MODE_PCIE_RC;
+
+ qmp->phy = devm_phy_create(dev, np, &qmp_pcie_phy_ops);
+ if (IS_ERR(qmp->phy)) {
+ ret = PTR_ERR(qmp->phy);
+ dev_err(dev, "failed to create PHY: %d\n", ret);
+ goto err_node_put;
+ }
+
+ phy_set_drvdata(qmp->phy, qmp);
+
+ of_node_put(np);
+
+ phy_provider = devm_of_phy_provider_register(dev, of_phy_simple_xlate);
+
+ return PTR_ERR_OR_ZERO(phy_provider);
+
+err_node_put:
+ of_node_put(np);
+ return ret;
+}
+
static const struct of_device_id qmp_pcie_of_match_table[] = {
{
- .compatible = "qcom,msm8998-qmp-pcie-phy",
- .data = &msm8998_pciephy_cfg,
- }, {
- .compatible = "qcom,ipq8074-qmp-pcie-phy",
- .data = &ipq8074_pciephy_cfg,
+ .compatible = "qcom,ipq6018-qmp-pcie-phy",
+ .data = &ipq6018_pciephy_cfg,
}, {
.compatible = "qcom,ipq8074-qmp-gen3-pcie-phy",
.data = &ipq8074_pciephy_gen3_cfg,
}, {
- .compatible = "qcom,ipq6018-qmp-pcie-phy",
- .data = &ipq6018_pciephy_cfg,
+ .compatible = "qcom,ipq8074-qmp-pcie-phy",
+ .data = &ipq8074_pciephy_cfg,
+ }, {
+ .compatible = "qcom,msm8998-qmp-pcie-phy",
+ .data = &msm8998_pciephy_cfg,
}, {
.compatible = "qcom,sc8180x-qmp-pcie-phy",
.data = &sc8180x_pciephy_cfg,
}, {
+ .compatible = "qcom,sc8280xp-qmp-gen3x1-pcie-phy",
+ .data = &sc8280xp_qmp_gen3x1_pciephy_cfg,
+ }, {
+ .compatible = "qcom,sc8280xp-qmp-gen3x2-pcie-phy",
+ .data = &sc8280xp_qmp_gen3x2_pciephy_cfg,
+ }, {
+ .compatible = "qcom,sc8280xp-qmp-gen3x4-pcie-phy",
+ .data = &sc8280xp_qmp_gen3x4_pciephy_cfg,
+ }, {
.compatible = "qcom,sdm845-qhp-pcie-phy",
.data = &sdm845_qhp_pciephy_cfg,
}, {
.compatible = "qcom,sdm845-qmp-pcie-phy",
.data = &sdm845_qmp_pciephy_cfg,
}, {
+ .compatible = "qcom,sdx55-qmp-pcie-phy",
+ .data = &sdx55_qmp_pciephy_cfg,
+ }, {
.compatible = "qcom,sm8250-qmp-gen3x1-pcie-phy",
.data = &sm8250_qmp_gen3x1_pciephy_cfg,
}, {
@@ -2297,9 +2720,6 @@ static const struct of_device_id qmp_pcie_of_match_table[] = {
.compatible = "qcom,sm8250-qmp-modem-pcie-phy",
.data = &sm8250_qmp_gen3x2_pciephy_cfg,
}, {
- .compatible = "qcom,sdx55-qmp-pcie-phy",
- .data = &sdx55_qmp_pciephy_cfg,
- }, {
.compatible = "qcom,sm8450-qmp-gen3x1-pcie-phy",
.data = &sm8450_qmp_gen3x1_pciephy_cfg,
}, {
@@ -2310,89 +2730,6 @@ static const struct of_device_id qmp_pcie_of_match_table[] = {
};
MODULE_DEVICE_TABLE(of, qmp_pcie_of_match_table);
-static int qmp_pcie_probe(struct platform_device *pdev)
-{
- struct qcom_qmp *qmp;
- struct device *dev = &pdev->dev;
- struct device_node *child;
- struct phy_provider *phy_provider;
- void __iomem *serdes;
- const struct qmp_phy_cfg *cfg = NULL;
- int num, id;
- int ret;
-
- qmp = devm_kzalloc(dev, sizeof(*qmp), GFP_KERNEL);
- if (!qmp)
- return -ENOMEM;
-
- qmp->dev = dev;
- dev_set_drvdata(dev, qmp);
-
- /* Get the specific init parameters of QMP phy */
- cfg = of_device_get_match_data(dev);
- if (!cfg)
- return -EINVAL;
-
- /* per PHY serdes; usually located at base address */
- serdes = devm_platform_ioremap_resource(pdev, 0);
- if (IS_ERR(serdes))
- return PTR_ERR(serdes);
-
- ret = qmp_pcie_clk_init(dev, cfg);
- if (ret)
- return ret;
-
- ret = qmp_pcie_reset_init(dev, cfg);
- if (ret)
- return ret;
-
- ret = qmp_pcie_vreg_init(dev, cfg);
- if (ret)
- return dev_err_probe(dev, ret,
- "failed to get regulator supplies\n");
-
- num = of_get_available_child_count(dev->of_node);
- /* do we have a rogue child node ? */
- if (num > 1)
- return -EINVAL;
-
- qmp->phys = devm_kcalloc(dev, num, sizeof(*qmp->phys), GFP_KERNEL);
- if (!qmp->phys)
- return -ENOMEM;
-
- id = 0;
- for_each_available_child_of_node(dev->of_node, child) {
- /* Create per-lane phy */
- ret = qmp_pcie_create(dev, child, id, serdes, cfg);
- if (ret) {
- dev_err(dev, "failed to create lane%d phy, %d\n",
- id, ret);
- goto err_node_put;
- }
-
- /*
- * Register the pipe clock provided by phy.
- * See function description to see details of this pipe clock.
- */
- ret = phy_pipe_clk_register(qmp, child);
- if (ret) {
- dev_err(qmp->dev,
- "failed to register pipe clock source\n");
- goto err_node_put;
- }
-
- id++;
- }
-
- phy_provider = devm_of_phy_provider_register(dev, of_phy_simple_xlate);
-
- return PTR_ERR_OR_ZERO(phy_provider);
-
-err_node_put:
- of_node_put(child);
- return ret;
-}
-
static struct platform_driver qmp_pcie_driver = {
.probe = qmp_pcie_probe,
.driver = {
diff --git a/drivers/phy/qualcomm/phy-qcom-qmp-pcs-pcie-v5.h b/drivers/phy/qualcomm/phy-qcom-qmp-pcs-pcie-v5.h
index 2e19fb3..a469ae2 100644
--- a/drivers/phy/qualcomm/phy-qcom-qmp-pcs-pcie-v5.h
+++ b/drivers/phy/qualcomm/phy-qcom-qmp-pcs-pcie-v5.h
@@ -8,6 +8,8 @@
#define QCOM_PHY_QMP_PCS_PCIE_V5_H_
/* Only for QMP V5 PHY - PCS_PCIE registers */
+#define QPHY_V5_PCS_PCIE_POWER_STATE_CONFIG2 0x0c
+#define QPHY_V5_PCS_PCIE_POWER_STATE_CONFIG4 0x14
#define QPHY_V5_PCS_PCIE_ENDPOINT_REFCLK_DRIVE 0x20
#define QPHY_V5_PCS_PCIE_INT_AUX_CLK_CONFIG1 0x54
#define QPHY_V5_PCS_PCIE_OSC_DTCT_ACTIONS 0x94
diff --git a/drivers/phy/qualcomm/phy-qcom-qmp-pcs-pcie-v5_20.h b/drivers/phy/qualcomm/phy-qcom-qmp-pcs-pcie-v5_20.h
index 1eedf50..3d9713d 100644
--- a/drivers/phy/qualcomm/phy-qcom-qmp-pcs-pcie-v5_20.h
+++ b/drivers/phy/qualcomm/phy-qcom-qmp-pcs-pcie-v5_20.h
@@ -8,8 +8,10 @@
/* Only for QMP V5_20 PHY - PCIe PCS registers */
#define QPHY_V5_20_PCS_PCIE_ENDPOINT_REFCLK_DRIVE 0x01c
+#define QPHY_V5_20_PCS_PCIE_OSC_DTCT_MODE2_CONFIG5 0x084
#define QPHY_V5_20_PCS_PCIE_OSC_DTCT_ACTIONS 0x090
#define QPHY_V5_20_PCS_PCIE_EQ_CONFIG1 0x0a0
+#define QPHY_V5_20_PCS_PCIE_PRESET_P10_POST 0x0e0
#define QPHY_V5_20_PCS_PCIE_G4_EQ_CONFIG5 0x108
#define QPHY_V5_20_PCS_PCIE_G4_PRE_GAIN 0x15c
#define QPHY_V5_20_PCS_PCIE_RX_MARGINING_CONFIG3 0x184
diff --git a/drivers/phy/qualcomm/phy-qcom-qmp-pcs-v5_20.h b/drivers/phy/qualcomm/phy-qcom-qmp-pcs-v5_20.h
new file mode 100644
index 0000000..9a5a20d
--- /dev/null
+++ b/drivers/phy/qualcomm/phy-qcom-qmp-pcs-v5_20.h
@@ -0,0 +1,14 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (c) 2022, Linaro Ltd.
+ */
+
+#ifndef QCOM_PHY_QMP_PCS_V5_20_H_
+#define QCOM_PHY_QMP_PCS_V5_20_H_
+
+#define QPHY_V5_20_PCS_G3S2_PRE_GAIN 0x170
+#define QPHY_V5_20_PCS_RX_SIGDET_LVL 0x188
+#define QPHY_V5_20_PCS_EQ_CONFIG4 0x1e0
+#define QPHY_V5_20_PCS_EQ_CONFIG5 0x1e4
+
+#endif
diff --git a/drivers/phy/qualcomm/phy-qcom-qmp-ufs.c b/drivers/phy/qualcomm/phy-qcom-qmp-ufs.c
index c08d34a..318eea3 100644
--- a/drivers/phy/qualcomm/phy-qcom-qmp-ufs.c
+++ b/drivers/phy/qualcomm/phy-qcom-qmp-ufs.c
@@ -20,8 +20,6 @@
#include <linux/reset.h>
#include <linux/slab.h>
-#include <dt-bindings/phy/phy.h>
-
#include "phy-qcom-qmp.h"
/* QPHY_SW_RESET bit */
@@ -31,8 +29,6 @@
/* QPHY_START_CONTROL bits */
#define SERDES_START BIT(0)
#define PCS_START BIT(1)
-/* QPHY_PCS_STATUS bit */
-#define PHYSTATUS BIT(6)
/* QPHY_PCS_READY_STATUS bit */
#define PCS_READY BIT(0)
@@ -42,11 +38,6 @@ struct qmp_phy_init_tbl {
unsigned int offset;
unsigned int val;
/*
- * register part of layout ?
- * if yes, then offset gives index in the reg-layout
- */
- bool in_layout;
- /*
* mask of lanes for which this register is written
* for cases when second lane needs different values
*/
@@ -60,14 +51,6 @@ struct qmp_phy_init_tbl {
.lane_mask = 0xff, \
}
-#define QMP_PHY_INIT_CFG_L(o, v) \
- { \
- .offset = o, \
- .val = v, \
- .in_layout = true, \
- .lane_mask = 0xff, \
- }
-
#define QMP_PHY_INIT_CFG_LANE(o, v, l) \
{ \
.offset = o, \
@@ -89,22 +72,26 @@ enum qphy_reg_layout {
static const unsigned int msm8996_ufsphy_regs_layout[QPHY_LAYOUT_SIZE] = {
[QPHY_START_CTRL] = 0x00,
[QPHY_PCS_READY_STATUS] = 0x168,
+ [QPHY_PCS_POWER_DOWN_CONTROL] = 0x04,
};
static const unsigned int sdm845_ufsphy_regs_layout[QPHY_LAYOUT_SIZE] = {
[QPHY_START_CTRL] = 0x00,
[QPHY_PCS_READY_STATUS] = 0x160,
+ [QPHY_PCS_POWER_DOWN_CONTROL] = 0x04,
};
static const unsigned int sm6115_ufsphy_regs_layout[QPHY_LAYOUT_SIZE] = {
[QPHY_START_CTRL] = 0x00,
[QPHY_PCS_READY_STATUS] = 0x168,
+ [QPHY_PCS_POWER_DOWN_CONTROL] = 0x04,
};
static const unsigned int sm8150_ufsphy_regs_layout[QPHY_LAYOUT_SIZE] = {
[QPHY_START_CTRL] = QPHY_V4_PCS_UFS_PHY_START,
[QPHY_PCS_READY_STATUS] = QPHY_V4_PCS_UFS_READY_STATUS,
[QPHY_SW_RESET] = QPHY_V4_PCS_UFS_SW_RESET,
+ [QPHY_PCS_POWER_DOWN_CONTROL] = QPHY_V4_PCS_UFS_POWER_DOWN_CONTROL,
};
static const struct qmp_phy_init_tbl msm8996_ufs_serdes_tbl[] = {
@@ -531,10 +518,21 @@ static const struct qmp_phy_init_tbl sm8350_ufsphy_pcs_tbl[] = {
QMP_PHY_INIT_CFG(QPHY_V5_PCS_UFS_MULTI_LANE_CTRL1, 0x02),
};
+struct qmp_ufs_offsets {
+ u16 serdes;
+ u16 pcs;
+ u16 tx;
+ u16 rx;
+ u16 tx2;
+ u16 rx2;
+};
+
/* struct qmp_phy_cfg - per-PHY initialization config */
struct qmp_phy_cfg {
int lanes;
+ const struct qmp_ufs_offsets *offsets;
+
/* Init sequence for PHY blocks - serdes, tx, rx, pcs */
const struct qmp_phy_init_tbl *serdes_tbl;
int serdes_tbl_num;
@@ -555,63 +553,28 @@ struct qmp_phy_cfg {
/* array of registers with different offsets */
const unsigned int *regs;
- unsigned int start_ctrl;
- unsigned int pwrdn_ctrl;
- /* bit offset of PHYSTATUS in QPHY_PCS_STATUS register */
- unsigned int phy_status;
-
/* true, if PCS block has no separate SW_RESET register */
bool no_pcs_sw_reset;
};
-/**
- * struct qmp_phy - per-lane phy descriptor
- *
- * @phy: generic phy
- * @cfg: phy specific configuration
- * @serdes: iomapped memory space for phy's serdes (i.e. PLL)
- * @tx: iomapped memory space for lane's tx
- * @rx: iomapped memory space for lane's rx
- * @pcs: iomapped memory space for lane's pcs
- * @tx2: iomapped memory space for second lane's tx (in dual lane PHYs)
- * @rx2: iomapped memory space for second lane's rx (in dual lane PHYs)
- * @pcs_misc: iomapped memory space for lane's pcs_misc
- * @qmp: QMP phy to which this lane belongs
- */
-struct qmp_phy {
- struct phy *phy;
+struct qmp_ufs {
+ struct device *dev;
+
const struct qmp_phy_cfg *cfg;
+
void __iomem *serdes;
+ void __iomem *pcs;
+ void __iomem *pcs_misc;
void __iomem *tx;
void __iomem *rx;
- void __iomem *pcs;
void __iomem *tx2;
void __iomem *rx2;
- void __iomem *pcs_misc;
- struct qcom_qmp *qmp;
-};
-
-/**
- * struct qcom_qmp - structure holding QMP phy block attributes
- *
- * @dev: device
- *
- * @clks: array of clocks required by phy
- * @resets: array of resets required by phy
- * @vregs: regulator supplies bulk data
- *
- * @phys: array of per-lane phy descriptors
- * @ufs_reset: optional UFS PHY reset handle
- */
-struct qcom_qmp {
- struct device *dev;
struct clk_bulk_data *clks;
struct regulator_bulk_data *vregs;
-
- struct qmp_phy **phys;
-
struct reset_control *ufs_reset;
+
+ struct phy *phy;
};
static inline void qphy_setbits(void __iomem *base, u32 offset, u32 val)
@@ -657,6 +620,15 @@ static const char * const qmp_phy_vreg_l[] = {
"vdda-phy", "vdda-pll",
};
+static const struct qmp_ufs_offsets qmp_ufs_offsets_v5 = {
+ .serdes = 0,
+ .pcs = 0xc00,
+ .tx = 0x400,
+ .rx = 0x600,
+ .tx2 = 0x800,
+ .rx2 = 0xa00,
+};
+
static const struct qmp_phy_cfg msm8996_ufs_cfg = {
.lanes = 1,
@@ -675,13 +647,29 @@ static const struct qmp_phy_cfg msm8996_ufs_cfg = {
.regs = msm8996_ufsphy_regs_layout,
- .start_ctrl = SERDES_START,
- .pwrdn_ctrl = SW_PWRDN,
- .phy_status = PHYSTATUS,
-
.no_pcs_sw_reset = true,
};
+static const struct qmp_phy_cfg sc8280xp_ufsphy_cfg = {
+ .lanes = 2,
+
+ .offsets = &qmp_ufs_offsets_v5,
+
+ .serdes_tbl = sm8350_ufsphy_serdes_tbl,
+ .serdes_tbl_num = ARRAY_SIZE(sm8350_ufsphy_serdes_tbl),
+ .tx_tbl = sm8350_ufsphy_tx_tbl,
+ .tx_tbl_num = ARRAY_SIZE(sm8350_ufsphy_tx_tbl),
+ .rx_tbl = sm8350_ufsphy_rx_tbl,
+ .rx_tbl_num = ARRAY_SIZE(sm8350_ufsphy_rx_tbl),
+ .pcs_tbl = sm8350_ufsphy_pcs_tbl,
+ .pcs_tbl_num = ARRAY_SIZE(sm8350_ufsphy_pcs_tbl),
+ .clk_list = sdm845_ufs_phy_clk_l,
+ .num_clks = ARRAY_SIZE(sdm845_ufs_phy_clk_l),
+ .vreg_list = qmp_phy_vreg_l,
+ .num_vregs = ARRAY_SIZE(qmp_phy_vreg_l),
+ .regs = sm8150_ufsphy_regs_layout,
+};
+
static const struct qmp_phy_cfg sdm845_ufsphy_cfg = {
.lanes = 2,
@@ -699,10 +687,6 @@ static const struct qmp_phy_cfg sdm845_ufsphy_cfg = {
.num_vregs = ARRAY_SIZE(qmp_phy_vreg_l),
.regs = sdm845_ufsphy_regs_layout,
- .start_ctrl = SERDES_START,
- .pwrdn_ctrl = SW_PWRDN,
- .phy_status = PHYSTATUS,
-
.no_pcs_sw_reset = true,
};
@@ -723,9 +707,6 @@ static const struct qmp_phy_cfg sm6115_ufsphy_cfg = {
.num_vregs = ARRAY_SIZE(qmp_phy_vreg_l),
.regs = sm6115_ufsphy_regs_layout,
- .start_ctrl = SERDES_START,
- .pwrdn_ctrl = SW_PWRDN,
-
.no_pcs_sw_reset = true,
};
@@ -745,10 +726,6 @@ static const struct qmp_phy_cfg sm8150_ufsphy_cfg = {
.vreg_list = qmp_phy_vreg_l,
.num_vregs = ARRAY_SIZE(qmp_phy_vreg_l),
.regs = sm8150_ufsphy_regs_layout,
-
- .start_ctrl = SERDES_START,
- .pwrdn_ctrl = SW_PWRDN,
- .phy_status = PHYSTATUS,
};
static const struct qmp_phy_cfg sm8350_ufsphy_cfg = {
@@ -767,10 +744,6 @@ static const struct qmp_phy_cfg sm8350_ufsphy_cfg = {
.vreg_list = qmp_phy_vreg_l,
.num_vregs = ARRAY_SIZE(qmp_phy_vreg_l),
.regs = sm8150_ufsphy_regs_layout,
-
- .start_ctrl = SERDES_START,
- .pwrdn_ctrl = SW_PWRDN,
- .phy_status = PHYSTATUS,
};
static const struct qmp_phy_cfg sm8450_ufsphy_cfg = {
@@ -789,14 +762,9 @@ static const struct qmp_phy_cfg sm8450_ufsphy_cfg = {
.vreg_list = qmp_phy_vreg_l,
.num_vregs = ARRAY_SIZE(qmp_phy_vreg_l),
.regs = sm8150_ufsphy_regs_layout,
-
- .start_ctrl = SERDES_START,
- .pwrdn_ctrl = SW_PWRDN,
- .phy_status = PHYSTATUS,
};
static void qmp_ufs_configure_lane(void __iomem *base,
- const unsigned int *regs,
const struct qmp_phy_init_tbl tbl[],
int num,
u8 lane_mask)
@@ -811,41 +779,35 @@ static void qmp_ufs_configure_lane(void __iomem *base,
if (!(t->lane_mask & lane_mask))
continue;
- if (t->in_layout)
- writel(t->val, base + regs[t->offset]);
- else
- writel(t->val, base + t->offset);
+ writel(t->val, base + t->offset);
}
}
static void qmp_ufs_configure(void __iomem *base,
- const unsigned int *regs,
const struct qmp_phy_init_tbl tbl[],
int num)
{
- qmp_ufs_configure_lane(base, regs, tbl, num, 0xff);
+ qmp_ufs_configure_lane(base, tbl, num, 0xff);
}
-static int qmp_ufs_serdes_init(struct qmp_phy *qphy)
+static int qmp_ufs_serdes_init(struct qmp_ufs *qmp)
{
- const struct qmp_phy_cfg *cfg = qphy->cfg;
- void __iomem *serdes = qphy->serdes;
+ const struct qmp_phy_cfg *cfg = qmp->cfg;
+ void __iomem *serdes = qmp->serdes;
const struct qmp_phy_init_tbl *serdes_tbl = cfg->serdes_tbl;
int serdes_tbl_num = cfg->serdes_tbl_num;
- qmp_ufs_configure(serdes, cfg->regs, serdes_tbl, serdes_tbl_num);
+ qmp_ufs_configure(serdes, serdes_tbl, serdes_tbl_num);
return 0;
}
-static int qmp_ufs_com_init(struct qmp_phy *qphy)
+static int qmp_ufs_com_init(struct qmp_ufs *qmp)
{
- struct qcom_qmp *qmp = qphy->qmp;
- const struct qmp_phy_cfg *cfg = qphy->cfg;
- void __iomem *pcs = qphy->pcs;
+ const struct qmp_phy_cfg *cfg = qmp->cfg;
+ void __iomem *pcs = qmp->pcs;
int ret;
- /* turn on regulator supplies */
ret = regulator_bulk_enable(cfg->num_vregs, qmp->vregs);
if (ret) {
dev_err(qmp->dev, "failed to enable regulators, err=%d\n", ret);
@@ -856,13 +818,7 @@ static int qmp_ufs_com_init(struct qmp_phy *qphy)
if (ret)
goto err_disable_regulators;
- if (cfg->regs[QPHY_PCS_POWER_DOWN_CONTROL])
- qphy_setbits(pcs,
- cfg->regs[QPHY_PCS_POWER_DOWN_CONTROL],
- cfg->pwrdn_ctrl);
- else
- qphy_setbits(pcs, QPHY_V2_PCS_POWER_DOWN_CONTROL,
- cfg->pwrdn_ctrl);
+ qphy_setbits(pcs, cfg->regs[QPHY_PCS_POWER_DOWN_CONTROL], SW_PWRDN);
return 0;
@@ -872,10 +828,9 @@ static int qmp_ufs_com_init(struct qmp_phy *qphy)
return ret;
}
-static int qmp_ufs_com_exit(struct qmp_phy *qphy)
+static int qmp_ufs_com_exit(struct qmp_ufs *qmp)
{
- struct qcom_qmp *qmp = qphy->qmp;
- const struct qmp_phy_cfg *cfg = qphy->cfg;
+ const struct qmp_phy_cfg *cfg = qmp->cfg;
reset_control_assert(qmp->ufs_reset);
@@ -888,9 +843,8 @@ static int qmp_ufs_com_exit(struct qmp_phy *qphy)
static int qmp_ufs_init(struct phy *phy)
{
- struct qmp_phy *qphy = phy_get_drvdata(phy);
- struct qcom_qmp *qmp = qphy->qmp;
- const struct qmp_phy_cfg *cfg = qphy->cfg;
+ struct qmp_ufs *qmp = phy_get_drvdata(phy);
+ const struct qmp_phy_cfg *cfg = qmp->cfg;
int ret;
dev_vdbg(qmp->dev, "Initializing QMP phy\n");
@@ -921,7 +875,7 @@ static int qmp_ufs_init(struct phy *phy)
return ret;
}
- ret = qmp_ufs_com_init(qphy);
+ ret = qmp_ufs_com_init(qmp);
if (ret)
return ret;
@@ -930,34 +884,27 @@ static int qmp_ufs_init(struct phy *phy)
static int qmp_ufs_power_on(struct phy *phy)
{
- struct qmp_phy *qphy = phy_get_drvdata(phy);
- struct qcom_qmp *qmp = qphy->qmp;
- const struct qmp_phy_cfg *cfg = qphy->cfg;
- void __iomem *tx = qphy->tx;
- void __iomem *rx = qphy->rx;
- void __iomem *pcs = qphy->pcs;
+ struct qmp_ufs *qmp = phy_get_drvdata(phy);
+ const struct qmp_phy_cfg *cfg = qmp->cfg;
+ void __iomem *tx = qmp->tx;
+ void __iomem *rx = qmp->rx;
+ void __iomem *pcs = qmp->pcs;
void __iomem *status;
- unsigned int mask, val, ready;
+ unsigned int val;
int ret;
- qmp_ufs_serdes_init(qphy);
+ qmp_ufs_serdes_init(qmp);
/* Tx, Rx, and PCS configurations */
- qmp_ufs_configure_lane(tx, cfg->regs, cfg->tx_tbl, cfg->tx_tbl_num, 1);
+ qmp_ufs_configure_lane(tx, cfg->tx_tbl, cfg->tx_tbl_num, 1);
+ qmp_ufs_configure_lane(rx, cfg->rx_tbl, cfg->rx_tbl_num, 1);
if (cfg->lanes >= 2) {
- qmp_ufs_configure_lane(qphy->tx2, cfg->regs,
- cfg->tx_tbl, cfg->tx_tbl_num, 2);
+ qmp_ufs_configure_lane(qmp->tx2, cfg->tx_tbl, cfg->tx_tbl_num, 2);
+ qmp_ufs_configure_lane(qmp->rx2, cfg->rx_tbl, cfg->rx_tbl_num, 2);
}
- qmp_ufs_configure_lane(rx, cfg->regs, cfg->rx_tbl, cfg->rx_tbl_num, 1);
-
- if (cfg->lanes >= 2) {
- qmp_ufs_configure_lane(qphy->rx2, cfg->regs,
- cfg->rx_tbl, cfg->rx_tbl_num, 2);
- }
-
- qmp_ufs_configure(pcs, cfg->regs, cfg->pcs_tbl, cfg->pcs_tbl_num);
+ qmp_ufs_configure(pcs, cfg->pcs_tbl, cfg->pcs_tbl_num);
ret = reset_control_deassert(qmp->ufs_reset);
if (ret)
@@ -966,14 +913,12 @@ static int qmp_ufs_power_on(struct phy *phy)
/* Pull PHY out of reset state */
if (!cfg->no_pcs_sw_reset)
qphy_clrbits(pcs, cfg->regs[QPHY_SW_RESET], SW_RESET);
- /* start SerDes and Phy-Coding-Sublayer */
- qphy_setbits(pcs, cfg->regs[QPHY_START_CTRL], cfg->start_ctrl);
+
+ /* start SerDes */
+ qphy_setbits(pcs, cfg->regs[QPHY_START_CTRL], SERDES_START);
status = pcs + cfg->regs[QPHY_PCS_READY_STATUS];
- mask = PCS_READY;
- ready = PCS_READY;
-
- ret = readl_poll_timeout(status, val, (val & mask) == ready, 10,
+ ret = readl_poll_timeout(status, val, (val & PCS_READY), 200,
PHY_INIT_COMPLETE_TIMEOUT);
if (ret) {
dev_err(qmp->dev, "phy initialization timed-out\n");
@@ -985,33 +930,28 @@ static int qmp_ufs_power_on(struct phy *phy)
static int qmp_ufs_power_off(struct phy *phy)
{
- struct qmp_phy *qphy = phy_get_drvdata(phy);
- const struct qmp_phy_cfg *cfg = qphy->cfg;
+ struct qmp_ufs *qmp = phy_get_drvdata(phy);
+ const struct qmp_phy_cfg *cfg = qmp->cfg;
/* PHY reset */
if (!cfg->no_pcs_sw_reset)
- qphy_setbits(qphy->pcs, cfg->regs[QPHY_SW_RESET], SW_RESET);
+ qphy_setbits(qmp->pcs, cfg->regs[QPHY_SW_RESET], SW_RESET);
- /* stop SerDes and Phy-Coding-Sublayer */
- qphy_clrbits(qphy->pcs, cfg->regs[QPHY_START_CTRL], cfg->start_ctrl);
+ /* stop SerDes */
+ qphy_clrbits(qmp->pcs, cfg->regs[QPHY_START_CTRL], SERDES_START);
/* Put PHY into POWER DOWN state: active low */
- if (cfg->regs[QPHY_PCS_POWER_DOWN_CONTROL]) {
- qphy_clrbits(qphy->pcs, cfg->regs[QPHY_PCS_POWER_DOWN_CONTROL],
- cfg->pwrdn_ctrl);
- } else {
- qphy_clrbits(qphy->pcs, QPHY_V2_PCS_POWER_DOWN_CONTROL,
- cfg->pwrdn_ctrl);
- }
+ qphy_clrbits(qmp->pcs, cfg->regs[QPHY_PCS_POWER_DOWN_CONTROL],
+ SW_PWRDN);
return 0;
}
static int qmp_ufs_exit(struct phy *phy)
{
- struct qmp_phy *qphy = phy_get_drvdata(phy);
+ struct qmp_ufs *qmp = phy_get_drvdata(phy);
- qmp_ufs_com_exit(qphy);
+ qmp_ufs_com_exit(qmp);
return 0;
}
@@ -1041,9 +981,16 @@ static int qmp_ufs_disable(struct phy *phy)
return qmp_ufs_exit(phy);
}
-static int qmp_ufs_vreg_init(struct device *dev, const struct qmp_phy_cfg *cfg)
+static const struct phy_ops qcom_qmp_ufs_phy_ops = {
+ .power_on = qmp_ufs_enable,
+ .power_off = qmp_ufs_disable,
+ .owner = THIS_MODULE,
+};
+
+static int qmp_ufs_vreg_init(struct qmp_ufs *qmp)
{
- struct qcom_qmp *qmp = dev_get_drvdata(dev);
+ const struct qmp_phy_cfg *cfg = qmp->cfg;
+ struct device *dev = qmp->dev;
int num = cfg->num_vregs;
int i;
@@ -1057,9 +1004,10 @@ static int qmp_ufs_vreg_init(struct device *dev, const struct qmp_phy_cfg *cfg)
return devm_regulator_bulk_get(dev, num, qmp->vregs);
}
-static int qmp_ufs_clk_init(struct device *dev, const struct qmp_phy_cfg *cfg)
+static int qmp_ufs_clk_init(struct qmp_ufs *qmp)
{
- struct qcom_qmp *qmp = dev_get_drvdata(dev);
+ const struct qmp_phy_cfg *cfg = qmp->cfg;
+ struct device *dev = qmp->dev;
int num = cfg->num_clks;
int i;
@@ -1073,76 +1021,138 @@ static int qmp_ufs_clk_init(struct device *dev, const struct qmp_phy_cfg *cfg)
return devm_clk_bulk_get(dev, num, qmp->clks);
}
-static const struct phy_ops qcom_qmp_ufs_ops = {
- .power_on = qmp_ufs_enable,
- .power_off = qmp_ufs_disable,
- .owner = THIS_MODULE,
-};
-
-static int qmp_ufs_create(struct device *dev, struct device_node *np, int id,
- void __iomem *serdes, const struct qmp_phy_cfg *cfg)
+static int qmp_ufs_parse_dt_legacy(struct qmp_ufs *qmp, struct device_node *np)
{
- struct qcom_qmp *qmp = dev_get_drvdata(dev);
- struct phy *generic_phy;
- struct qmp_phy *qphy;
- int ret;
+ struct platform_device *pdev = to_platform_device(qmp->dev);
+ const struct qmp_phy_cfg *cfg = qmp->cfg;
+ struct device *dev = qmp->dev;
- qphy = devm_kzalloc(dev, sizeof(*qphy), GFP_KERNEL);
- if (!qphy)
- return -ENOMEM;
+ qmp->serdes = devm_platform_ioremap_resource(pdev, 0);
+ if (IS_ERR(qmp->serdes))
+ return PTR_ERR(qmp->serdes);
- qphy->cfg = cfg;
- qphy->serdes = serdes;
/*
- * Get memory resources for each phy lane:
+ * Get memory resources for the PHY:
* Resources are indexed as: tx -> 0; rx -> 1; pcs -> 2.
* For dual lane PHYs: tx2 -> 3, rx2 -> 4, pcs_misc (optional) -> 5
* For single lane PHYs: pcs_misc (optional) -> 3.
*/
- qphy->tx = devm_of_iomap(dev, np, 0, NULL);
- if (IS_ERR(qphy->tx))
- return PTR_ERR(qphy->tx);
+ qmp->tx = devm_of_iomap(dev, np, 0, NULL);
+ if (IS_ERR(qmp->tx))
+ return PTR_ERR(qmp->tx);
- qphy->rx = devm_of_iomap(dev, np, 1, NULL);
- if (IS_ERR(qphy->rx))
- return PTR_ERR(qphy->rx);
+ qmp->rx = devm_of_iomap(dev, np, 1, NULL);
+ if (IS_ERR(qmp->rx))
+ return PTR_ERR(qmp->rx);
- qphy->pcs = devm_of_iomap(dev, np, 2, NULL);
- if (IS_ERR(qphy->pcs))
- return PTR_ERR(qphy->pcs);
+ qmp->pcs = devm_of_iomap(dev, np, 2, NULL);
+ if (IS_ERR(qmp->pcs))
+ return PTR_ERR(qmp->pcs);
if (cfg->lanes >= 2) {
- qphy->tx2 = devm_of_iomap(dev, np, 3, NULL);
- if (IS_ERR(qphy->tx2))
- return PTR_ERR(qphy->tx2);
+ qmp->tx2 = devm_of_iomap(dev, np, 3, NULL);
+ if (IS_ERR(qmp->tx2))
+ return PTR_ERR(qmp->tx2);
- qphy->rx2 = devm_of_iomap(dev, np, 4, NULL);
- if (IS_ERR(qphy->rx2))
- return PTR_ERR(qphy->rx2);
+ qmp->rx2 = devm_of_iomap(dev, np, 4, NULL);
+ if (IS_ERR(qmp->rx2))
+ return PTR_ERR(qmp->rx2);
- qphy->pcs_misc = devm_of_iomap(dev, np, 5, NULL);
+ qmp->pcs_misc = devm_of_iomap(dev, np, 5, NULL);
} else {
- qphy->pcs_misc = devm_of_iomap(dev, np, 3, NULL);
+ qmp->pcs_misc = devm_of_iomap(dev, np, 3, NULL);
}
- if (IS_ERR(qphy->pcs_misc))
+ if (IS_ERR(qmp->pcs_misc))
dev_vdbg(dev, "PHY pcs_misc-reg not used\n");
- generic_phy = devm_phy_create(dev, np, &qcom_qmp_ufs_ops);
- if (IS_ERR(generic_phy)) {
- ret = PTR_ERR(generic_phy);
- dev_err(dev, "failed to create qphy %d\n", ret);
- return ret;
+ return 0;
+}
+
+static int qmp_ufs_parse_dt(struct qmp_ufs *qmp)
+{
+ struct platform_device *pdev = to_platform_device(qmp->dev);
+ const struct qmp_phy_cfg *cfg = qmp->cfg;
+ const struct qmp_ufs_offsets *offs = cfg->offsets;
+ void __iomem *base;
+
+ if (!offs)
+ return -EINVAL;
+
+ base = devm_platform_ioremap_resource(pdev, 0);
+ if (IS_ERR(base))
+ return PTR_ERR(base);
+
+ qmp->serdes = base + offs->serdes;
+ qmp->pcs = base + offs->pcs;
+ qmp->tx = base + offs->tx;
+ qmp->rx = base + offs->rx;
+
+ if (cfg->lanes >= 2) {
+ qmp->tx2 = base + offs->tx2;
+ qmp->rx2 = base + offs->rx2;
}
- qphy->phy = generic_phy;
- qphy->qmp = qmp;
- qmp->phys[id] = qphy;
- phy_set_drvdata(generic_phy, qphy);
-
return 0;
}
+static int qmp_ufs_probe(struct platform_device *pdev)
+{
+ struct device *dev = &pdev->dev;
+ struct phy_provider *phy_provider;
+ struct device_node *np;
+ struct qmp_ufs *qmp;
+ int ret;
+
+ qmp = devm_kzalloc(dev, sizeof(*qmp), GFP_KERNEL);
+ if (!qmp)
+ return -ENOMEM;
+
+ qmp->dev = dev;
+
+ qmp->cfg = of_device_get_match_data(dev);
+ if (!qmp->cfg)
+ return -EINVAL;
+
+ ret = qmp_ufs_clk_init(qmp);
+ if (ret)
+ return ret;
+
+ ret = qmp_ufs_vreg_init(qmp);
+ if (ret)
+ return ret;
+
+ /* Check for legacy binding with child node. */
+ np = of_get_next_available_child(dev->of_node, NULL);
+ if (np) {
+ ret = qmp_ufs_parse_dt_legacy(qmp, np);
+ } else {
+ np = of_node_get(dev->of_node);
+ ret = qmp_ufs_parse_dt(qmp);
+ }
+ if (ret)
+ goto err_node_put;
+
+ qmp->phy = devm_phy_create(dev, np, &qcom_qmp_ufs_phy_ops);
+ if (IS_ERR(qmp->phy)) {
+ ret = PTR_ERR(qmp->phy);
+ dev_err(dev, "failed to create PHY: %d\n", ret);
+ goto err_node_put;
+ }
+
+ phy_set_drvdata(qmp->phy, qmp);
+
+ of_node_put(np);
+
+ phy_provider = devm_of_phy_provider_register(dev, of_phy_simple_xlate);
+
+ return PTR_ERR_OR_ZERO(phy_provider);
+
+err_node_put:
+ of_node_put(np);
+ return ret;
+}
+
static const struct of_device_id qmp_ufs_of_match_table[] = {
{
.compatible = "qcom,msm8996-qmp-ufs-phy",
@@ -1155,7 +1165,7 @@ static const struct of_device_id qmp_ufs_of_match_table[] = {
.data = &sm8150_ufsphy_cfg,
}, {
.compatible = "qcom,sc8280xp-qmp-ufs-phy",
- .data = &sm8350_ufsphy_cfg,
+ .data = &sc8280xp_ufsphy_cfg,
}, {
.compatible = "qcom,sdm845-qmp-ufs-phy",
.data = &sdm845_ufsphy_cfg,
@@ -1182,74 +1192,6 @@ static const struct of_device_id qmp_ufs_of_match_table[] = {
};
MODULE_DEVICE_TABLE(of, qmp_ufs_of_match_table);
-static int qmp_ufs_probe(struct platform_device *pdev)
-{
- struct qcom_qmp *qmp;
- struct device *dev = &pdev->dev;
- struct device_node *child;
- struct phy_provider *phy_provider;
- void __iomem *serdes;
- const struct qmp_phy_cfg *cfg = NULL;
- int num, id;
- int ret;
-
- qmp = devm_kzalloc(dev, sizeof(*qmp), GFP_KERNEL);
- if (!qmp)
- return -ENOMEM;
-
- qmp->dev = dev;
- dev_set_drvdata(dev, qmp);
-
- /* Get the specific init parameters of QMP phy */
- cfg = of_device_get_match_data(dev);
- if (!cfg)
- return -EINVAL;
-
- /* per PHY serdes; usually located at base address */
- serdes = devm_platform_ioremap_resource(pdev, 0);
- if (IS_ERR(serdes))
- return PTR_ERR(serdes);
-
- ret = qmp_ufs_clk_init(dev, cfg);
- if (ret)
- return ret;
-
- ret = qmp_ufs_vreg_init(dev, cfg);
- if (ret)
- return dev_err_probe(dev, ret,
- "failed to get regulator supplies\n");
-
- num = of_get_available_child_count(dev->of_node);
- /* do we have a rogue child node ? */
- if (num > 1)
- return -EINVAL;
-
- qmp->phys = devm_kcalloc(dev, num, sizeof(*qmp->phys), GFP_KERNEL);
- if (!qmp->phys)
- return -ENOMEM;
-
- id = 0;
- for_each_available_child_of_node(dev->of_node, child) {
- /* Create per-lane phy */
- ret = qmp_ufs_create(dev, child, id, serdes, cfg);
- if (ret) {
- dev_err(dev, "failed to create lane%d phy, %d\n",
- id, ret);
- goto err_node_put;
- }
-
- id++;
- }
-
- phy_provider = devm_of_phy_provider_register(dev, of_phy_simple_xlate);
-
- return PTR_ERR_OR_ZERO(phy_provider);
-
-err_node_put:
- of_node_put(child);
- return ret;
-}
-
static struct platform_driver qmp_ufs_driver = {
.probe = qmp_ufs_probe,
.driver = {
diff --git a/drivers/phy/qualcomm/phy-qcom-qmp-usb.c b/drivers/phy/qualcomm/phy-qcom-qmp-usb.c
index b84c0d4..4aa338f 100644
--- a/drivers/phy/qualcomm/phy-qcom-qmp-usb.c
+++ b/drivers/phy/qualcomm/phy-qcom-qmp-usb.c
@@ -20,8 +20,6 @@
#include <linux/reset.h>
#include <linux/slab.h>
-#include <dt-bindings/phy/phy.h>
-
#include "phy-qcom-qmp.h"
/* QPHY_SW_RESET bit */
@@ -63,18 +61,11 @@
#define CLAMP_EN BIT(0) /* enables i/o clamp_n */
#define PHY_INIT_COMPLETE_TIMEOUT 10000
-#define POWER_DOWN_DELAY_US_MIN 10
-#define POWER_DOWN_DELAY_US_MAX 11
struct qmp_phy_init_tbl {
unsigned int offset;
unsigned int val;
/*
- * register part of layout ?
- * if yes, then offset gives index in the reg-layout
- */
- bool in_layout;
- /*
* mask of lanes for which this register is written
* for cases when second lane needs different values
*/
@@ -88,14 +79,6 @@ struct qmp_phy_init_tbl {
.lane_mask = 0xff, \
}
-#define QMP_PHY_INIT_CFG_L(o, v) \
- { \
- .offset = o, \
- .val = v, \
- .in_layout = true, \
- .lane_mask = 0xff, \
- }
-
#define QMP_PHY_INIT_CFG_LANE(o, v, l) \
{ \
.offset = o, \
@@ -126,6 +109,7 @@ static const unsigned int usb3phy_regs_layout[QPHY_LAYOUT_SIZE] = {
[QPHY_PCS_AUTONOMOUS_MODE_CTRL] = 0x0d4,
[QPHY_PCS_LFPS_RXTERM_IRQ_CLEAR] = 0x0d8,
[QPHY_PCS_LFPS_RXTERM_IRQ_STATUS] = 0x178,
+ [QPHY_PCS_POWER_DOWN_CONTROL] = 0x04,
};
static const unsigned int qmp_v3_usb3phy_regs_layout[QPHY_LAYOUT_SIZE] = {
@@ -135,6 +119,7 @@ static const unsigned int qmp_v3_usb3phy_regs_layout[QPHY_LAYOUT_SIZE] = {
[QPHY_PCS_AUTONOMOUS_MODE_CTRL] = 0x0d8,
[QPHY_PCS_LFPS_RXTERM_IRQ_CLEAR] = 0x0dc,
[QPHY_PCS_LFPS_RXTERM_IRQ_STATUS] = 0x170,
+ [QPHY_PCS_POWER_DOWN_CONTROL] = 0x04,
};
static const unsigned int qmp_v4_usb3phy_regs_layout[QPHY_LAYOUT_SIZE] = {
@@ -1427,10 +1412,20 @@ static const struct qmp_phy_init_tbl sc8280xp_usb3_uniphy_pcs_tbl[] = {
QMP_PHY_INIT_CFG(QPHY_V5_PCS_REFGEN_REQ_CONFIG1, 0x21),
};
+struct qmp_usb_offsets {
+ u16 serdes;
+ u16 pcs;
+ u16 pcs_usb;
+ u16 tx;
+ u16 rx;
+};
+
/* struct qmp_phy_cfg - per-PHY initialization config */
struct qmp_phy_cfg {
int lanes;
+ const struct qmp_usb_offsets *offsets;
+
/* Init sequence for PHY blocks - serdes, tx, rx, pcs */
const struct qmp_phy_init_tbl *serdes_tbl;
int serdes_tbl_num;
@@ -1456,16 +1451,8 @@ struct qmp_phy_cfg {
/* array of registers with different offsets */
const unsigned int *regs;
- unsigned int start_ctrl;
- unsigned int pwrdn_ctrl;
- /* bit offset of PHYSTATUS in QPHY_PCS_STATUS register */
- unsigned int phy_status;
-
/* true, if PHY needs delay after POWER_DOWN */
bool has_pwrdn_delay;
- /* power_down delay in usec */
- int pwrdn_delay_min;
- int pwrdn_delay_max;
/* true, if PHY has a separate DP_COM control block */
bool has_phy_dp_com_ctrl;
@@ -1474,60 +1461,32 @@ struct qmp_phy_cfg {
unsigned int pcs_usb_offset;
};
-/**
- * struct qmp_phy - per-lane phy descriptor
- *
- * @phy: generic phy
- * @cfg: phy specific configuration
- * @serdes: iomapped memory space for phy's serdes (i.e. PLL)
- * @tx: iomapped memory space for lane's tx
- * @rx: iomapped memory space for lane's rx
- * @pcs: iomapped memory space for lane's pcs
- * @tx2: iomapped memory space for second lane's tx (in dual lane PHYs)
- * @rx2: iomapped memory space for second lane's rx (in dual lane PHYs)
- * @pcs_misc: iomapped memory space for lane's pcs_misc
- * @pcs_usb: iomapped memory space for lane's pcs_usb
- * @pipe_clk: pipe clock
- * @qmp: QMP phy to which this lane belongs
- * @mode: current PHY mode
- */
-struct qmp_phy {
- struct phy *phy;
+struct qmp_usb {
+ struct device *dev;
+
const struct qmp_phy_cfg *cfg;
+
void __iomem *serdes;
- void __iomem *tx;
- void __iomem *rx;
void __iomem *pcs;
- void __iomem *tx2;
- void __iomem *rx2;
void __iomem *pcs_misc;
void __iomem *pcs_usb;
- struct clk *pipe_clk;
- struct qcom_qmp *qmp;
- enum phy_mode mode;
-};
+ void __iomem *tx;
+ void __iomem *rx;
+ void __iomem *tx2;
+ void __iomem *rx2;
-/**
- * struct qcom_qmp - structure holding QMP phy block attributes
- *
- * @dev: device
- * @dp_com: iomapped memory space for phy's dp_com control block
- *
- * @clks: array of clocks required by phy
- * @resets: array of resets required by phy
- * @vregs: regulator supplies bulk data
- *
- * @phys: array of per-lane phy descriptors
- */
-struct qcom_qmp {
- struct device *dev;
void __iomem *dp_com;
+ struct clk *pipe_clk;
struct clk_bulk_data *clks;
struct reset_control_bulk_data *resets;
struct regulator_bulk_data *vregs;
- struct qmp_phy **phys;
+ enum phy_mode mode;
+
+ struct phy *phy;
+
+ struct clk_fixed_rate pipe_clk_fixed;
};
static inline void qphy_setbits(void __iomem *base, u32 offset, u32 val)
@@ -1564,6 +1523,10 @@ static const char * const qmp_v3_phy_clk_l[] = {
};
static const char * const qmp_v4_phy_clk_l[] = {
+ "aux", "ref", "com_aux",
+};
+
+static const char * const qmp_v4_ref_phy_clk_l[] = {
"aux", "ref_clk_src", "ref", "com_aux",
};
@@ -1599,6 +1562,14 @@ static const char * const qmp_phy_vreg_l[] = {
"vdda-phy", "vdda-pll",
};
+static const struct qmp_usb_offsets qmp_usb_offsets_v5 = {
+ .serdes = 0,
+ .pcs = 0x0200,
+ .pcs_usb = 0x1200,
+ .tx = 0x0e00,
+ .rx = 0x1000,
+};
+
static const struct qmp_phy_cfg ipq8074_usb3phy_cfg = {
.lanes = 1,
@@ -1616,11 +1587,7 @@ static const struct qmp_phy_cfg ipq8074_usb3phy_cfg = {
.num_resets = ARRAY_SIZE(msm8996_usb3phy_reset_l),
.vreg_list = qmp_phy_vreg_l,
.num_vregs = ARRAY_SIZE(qmp_phy_vreg_l),
- .regs = usb3phy_regs_layout,
-
- .start_ctrl = SERDES_START | PCS_START,
- .pwrdn_ctrl = SW_PWRDN,
- .phy_status = PHYSTATUS,
+ .regs = qmp_v3_usb3phy_regs_layout,
};
static const struct qmp_phy_cfg msm8996_usb3phy_cfg = {
@@ -1641,10 +1608,6 @@ static const struct qmp_phy_cfg msm8996_usb3phy_cfg = {
.vreg_list = qmp_phy_vreg_l,
.num_vregs = ARRAY_SIZE(qmp_phy_vreg_l),
.regs = usb3phy_regs_layout,
-
- .start_ctrl = SERDES_START | PCS_START,
- .pwrdn_ctrl = SW_PWRDN,
- .phy_status = PHYSTATUS,
};
static const struct qmp_phy_cfg qmp_v3_usb3phy_cfg = {
@@ -1666,14 +1629,7 @@ static const struct qmp_phy_cfg qmp_v3_usb3phy_cfg = {
.num_vregs = ARRAY_SIZE(qmp_phy_vreg_l),
.regs = qmp_v3_usb3phy_regs_layout,
- .start_ctrl = SERDES_START | PCS_START,
- .pwrdn_ctrl = SW_PWRDN,
- .phy_status = PHYSTATUS,
-
.has_pwrdn_delay = true,
- .pwrdn_delay_min = POWER_DOWN_DELAY_US_MIN,
- .pwrdn_delay_max = POWER_DOWN_DELAY_US_MAX,
-
.has_phy_dp_com_ctrl = true,
};
@@ -1696,20 +1652,15 @@ static const struct qmp_phy_cfg sc7180_usb3phy_cfg = {
.num_vregs = ARRAY_SIZE(qmp_phy_vreg_l),
.regs = qmp_v3_usb3phy_regs_layout,
- .start_ctrl = SERDES_START | PCS_START,
- .pwrdn_ctrl = SW_PWRDN,
- .phy_status = PHYSTATUS,
-
.has_pwrdn_delay = true,
- .pwrdn_delay_min = POWER_DOWN_DELAY_US_MIN,
- .pwrdn_delay_max = POWER_DOWN_DELAY_US_MAX,
-
.has_phy_dp_com_ctrl = true,
};
static const struct qmp_phy_cfg sc8280xp_usb3_uniphy_cfg = {
.lanes = 1,
+ .offsets = &qmp_usb_offsets_v5,
+
.serdes_tbl = sc8280xp_usb3_uniphy_serdes_tbl,
.serdes_tbl_num = ARRAY_SIZE(sc8280xp_usb3_uniphy_serdes_tbl),
.tx_tbl = sc8280xp_usb3_uniphy_tx_tbl,
@@ -1720,19 +1671,11 @@ static const struct qmp_phy_cfg sc8280xp_usb3_uniphy_cfg = {
.pcs_tbl_num = ARRAY_SIZE(sc8280xp_usb3_uniphy_pcs_tbl),
.clk_list = qmp_v4_phy_clk_l,
.num_clks = ARRAY_SIZE(qmp_v4_phy_clk_l),
- .reset_list = msm8996_usb3phy_reset_l,
- .num_resets = ARRAY_SIZE(msm8996_usb3phy_reset_l),
+ .reset_list = qcm2290_usb3phy_reset_l,
+ .num_resets = ARRAY_SIZE(qcm2290_usb3phy_reset_l),
.vreg_list = qmp_phy_vreg_l,
.num_vregs = ARRAY_SIZE(qmp_phy_vreg_l),
.regs = qmp_v4_usb3phy_regs_layout,
-
- .start_ctrl = SERDES_START | PCS_START,
- .pwrdn_ctrl = SW_PWRDN,
- .phy_status = PHYSTATUS,
-
- .has_pwrdn_delay = true,
- .pwrdn_delay_min = POWER_DOWN_DELAY_US_MIN,
- .pwrdn_delay_max = POWER_DOWN_DELAY_US_MAX,
};
static const struct qmp_phy_cfg qmp_v3_usb3_uniphy_cfg = {
@@ -1754,13 +1697,7 @@ static const struct qmp_phy_cfg qmp_v3_usb3_uniphy_cfg = {
.num_vregs = ARRAY_SIZE(qmp_phy_vreg_l),
.regs = qmp_v3_usb3phy_regs_layout,
- .start_ctrl = SERDES_START | PCS_START,
- .pwrdn_ctrl = SW_PWRDN,
- .phy_status = PHYSTATUS,
-
.has_pwrdn_delay = true,
- .pwrdn_delay_min = POWER_DOWN_DELAY_US_MIN,
- .pwrdn_delay_max = POWER_DOWN_DELAY_US_MAX,
};
static const struct qmp_phy_cfg msm8998_usb3phy_cfg = {
@@ -1781,10 +1718,6 @@ static const struct qmp_phy_cfg msm8998_usb3phy_cfg = {
.vreg_list = qmp_phy_vreg_l,
.num_vregs = ARRAY_SIZE(qmp_phy_vreg_l),
.regs = qmp_v3_usb3phy_regs_layout,
-
- .start_ctrl = SERDES_START | PCS_START,
- .pwrdn_ctrl = SW_PWRDN,
- .phy_status = PHYSTATUS,
};
static const struct qmp_phy_cfg sm8150_usb3phy_cfg = {
@@ -1800,8 +1733,8 @@ static const struct qmp_phy_cfg sm8150_usb3phy_cfg = {
.pcs_tbl_num = ARRAY_SIZE(sm8150_usb3_pcs_tbl),
.pcs_usb_tbl = sm8150_usb3_pcs_usb_tbl,
.pcs_usb_tbl_num = ARRAY_SIZE(sm8150_usb3_pcs_usb_tbl),
- .clk_list = qmp_v4_phy_clk_l,
- .num_clks = ARRAY_SIZE(qmp_v4_phy_clk_l),
+ .clk_list = qmp_v4_ref_phy_clk_l,
+ .num_clks = ARRAY_SIZE(qmp_v4_ref_phy_clk_l),
.reset_list = msm8996_usb3phy_reset_l,
.num_resets = ARRAY_SIZE(msm8996_usb3phy_reset_l),
.vreg_list = qmp_phy_vreg_l,
@@ -1809,15 +1742,7 @@ static const struct qmp_phy_cfg sm8150_usb3phy_cfg = {
.regs = qmp_v4_usb3phy_regs_layout,
.pcs_usb_offset = 0x300,
- .start_ctrl = SERDES_START | PCS_START,
- .pwrdn_ctrl = SW_PWRDN,
- .phy_status = PHYSTATUS,
-
-
.has_pwrdn_delay = true,
- .pwrdn_delay_min = POWER_DOWN_DELAY_US_MIN,
- .pwrdn_delay_max = POWER_DOWN_DELAY_US_MAX,
-
.has_phy_dp_com_ctrl = true,
};
@@ -1834,8 +1759,8 @@ static const struct qmp_phy_cfg sm8150_usb3_uniphy_cfg = {
.pcs_tbl_num = ARRAY_SIZE(sm8150_usb3_uniphy_pcs_tbl),
.pcs_usb_tbl = sm8150_usb3_uniphy_pcs_usb_tbl,
.pcs_usb_tbl_num = ARRAY_SIZE(sm8150_usb3_uniphy_pcs_usb_tbl),
- .clk_list = qmp_v4_phy_clk_l,
- .num_clks = ARRAY_SIZE(qmp_v4_phy_clk_l),
+ .clk_list = qmp_v4_ref_phy_clk_l,
+ .num_clks = ARRAY_SIZE(qmp_v4_ref_phy_clk_l),
.reset_list = msm8996_usb3phy_reset_l,
.num_resets = ARRAY_SIZE(msm8996_usb3phy_reset_l),
.vreg_list = qmp_phy_vreg_l,
@@ -1843,13 +1768,7 @@ static const struct qmp_phy_cfg sm8150_usb3_uniphy_cfg = {
.regs = qmp_v4_usb3phy_regs_layout,
.pcs_usb_offset = 0x600,
- .start_ctrl = SERDES_START | PCS_START,
- .pwrdn_ctrl = SW_PWRDN,
- .phy_status = PHYSTATUS,
-
.has_pwrdn_delay = true,
- .pwrdn_delay_min = POWER_DOWN_DELAY_US_MIN,
- .pwrdn_delay_max = POWER_DOWN_DELAY_US_MAX,
};
static const struct qmp_phy_cfg sm8250_usb3phy_cfg = {
@@ -1874,14 +1793,7 @@ static const struct qmp_phy_cfg sm8250_usb3phy_cfg = {
.regs = qmp_v4_usb3phy_regs_layout,
.pcs_usb_offset = 0x300,
- .start_ctrl = SERDES_START | PCS_START,
- .pwrdn_ctrl = SW_PWRDN,
- .phy_status = PHYSTATUS,
-
.has_pwrdn_delay = true,
- .pwrdn_delay_min = POWER_DOWN_DELAY_US_MIN,
- .pwrdn_delay_max = POWER_DOWN_DELAY_US_MAX,
-
.has_phy_dp_com_ctrl = true,
};
@@ -1898,8 +1810,8 @@ static const struct qmp_phy_cfg sm8250_usb3_uniphy_cfg = {
.pcs_tbl_num = ARRAY_SIZE(sm8250_usb3_uniphy_pcs_tbl),
.pcs_usb_tbl = sm8250_usb3_uniphy_pcs_usb_tbl,
.pcs_usb_tbl_num = ARRAY_SIZE(sm8250_usb3_uniphy_pcs_usb_tbl),
- .clk_list = qmp_v4_phy_clk_l,
- .num_clks = ARRAY_SIZE(qmp_v4_phy_clk_l),
+ .clk_list = qmp_v4_ref_phy_clk_l,
+ .num_clks = ARRAY_SIZE(qmp_v4_ref_phy_clk_l),
.reset_list = msm8996_usb3phy_reset_l,
.num_resets = ARRAY_SIZE(msm8996_usb3phy_reset_l),
.vreg_list = qmp_phy_vreg_l,
@@ -1907,13 +1819,7 @@ static const struct qmp_phy_cfg sm8250_usb3_uniphy_cfg = {
.regs = qmp_v4_usb3phy_regs_layout,
.pcs_usb_offset = 0x600,
- .start_ctrl = SERDES_START | PCS_START,
- .pwrdn_ctrl = SW_PWRDN,
- .phy_status = PHYSTATUS,
-
.has_pwrdn_delay = true,
- .pwrdn_delay_min = POWER_DOWN_DELAY_US_MIN,
- .pwrdn_delay_max = POWER_DOWN_DELAY_US_MAX,
};
static const struct qmp_phy_cfg sdx55_usb3_uniphy_cfg = {
@@ -1938,13 +1844,7 @@ static const struct qmp_phy_cfg sdx55_usb3_uniphy_cfg = {
.regs = qmp_v4_usb3phy_regs_layout,
.pcs_usb_offset = 0x600,
- .start_ctrl = SERDES_START | PCS_START,
- .pwrdn_ctrl = SW_PWRDN,
- .phy_status = PHYSTATUS,
-
.has_pwrdn_delay = true,
- .pwrdn_delay_min = POWER_DOWN_DELAY_US_MIN,
- .pwrdn_delay_max = POWER_DOWN_DELAY_US_MAX,
};
static const struct qmp_phy_cfg sdx65_usb3_uniphy_cfg = {
@@ -1969,13 +1869,7 @@ static const struct qmp_phy_cfg sdx65_usb3_uniphy_cfg = {
.regs = qmp_v4_usb3phy_regs_layout,
.pcs_usb_offset = 0x1000,
- .start_ctrl = SERDES_START | PCS_START,
- .pwrdn_ctrl = SW_PWRDN,
- .phy_status = PHYSTATUS,
-
.has_pwrdn_delay = true,
- .pwrdn_delay_min = POWER_DOWN_DELAY_US_MIN,
- .pwrdn_delay_max = POWER_DOWN_DELAY_US_MAX,
};
static const struct qmp_phy_cfg sm8350_usb3phy_cfg = {
@@ -2000,14 +1894,7 @@ static const struct qmp_phy_cfg sm8350_usb3phy_cfg = {
.regs = qmp_v4_usb3phy_regs_layout,
.pcs_usb_offset = 0x300,
- .start_ctrl = SERDES_START | PCS_START,
- .pwrdn_ctrl = SW_PWRDN,
- .phy_status = PHYSTATUS,
-
.has_pwrdn_delay = true,
- .pwrdn_delay_min = POWER_DOWN_DELAY_US_MIN,
- .pwrdn_delay_max = POWER_DOWN_DELAY_US_MAX,
-
.has_phy_dp_com_ctrl = true,
};
@@ -2024,8 +1911,8 @@ static const struct qmp_phy_cfg sm8350_usb3_uniphy_cfg = {
.pcs_tbl_num = ARRAY_SIZE(sm8350_usb3_uniphy_pcs_tbl),
.pcs_usb_tbl = sm8350_usb3_uniphy_pcs_usb_tbl,
.pcs_usb_tbl_num = ARRAY_SIZE(sm8350_usb3_uniphy_pcs_usb_tbl),
- .clk_list = qmp_v4_phy_clk_l,
- .num_clks = ARRAY_SIZE(qmp_v4_phy_clk_l),
+ .clk_list = qmp_v4_ref_phy_clk_l,
+ .num_clks = ARRAY_SIZE(qmp_v4_ref_phy_clk_l),
.reset_list = msm8996_usb3phy_reset_l,
.num_resets = ARRAY_SIZE(msm8996_usb3phy_reset_l),
.vreg_list = qmp_phy_vreg_l,
@@ -2033,13 +1920,7 @@ static const struct qmp_phy_cfg sm8350_usb3_uniphy_cfg = {
.regs = qmp_v4_usb3phy_regs_layout,
.pcs_usb_offset = 0x1000,
- .start_ctrl = SERDES_START | PCS_START,
- .pwrdn_ctrl = SW_PWRDN,
- .phy_status = PHYSTATUS,
-
.has_pwrdn_delay = true,
- .pwrdn_delay_min = POWER_DOWN_DELAY_US_MIN,
- .pwrdn_delay_max = POWER_DOWN_DELAY_US_MAX,
};
static const struct qmp_phy_cfg qcm2290_usb3phy_cfg = {
@@ -2060,14 +1941,9 @@ static const struct qmp_phy_cfg qcm2290_usb3phy_cfg = {
.vreg_list = qmp_phy_vreg_l,
.num_vregs = ARRAY_SIZE(qmp_phy_vreg_l),
.regs = qcm2290_usb3phy_regs_layout,
-
- .start_ctrl = SERDES_START | PCS_START,
- .pwrdn_ctrl = SW_PWRDN,
- .phy_status = PHYSTATUS,
};
static void qmp_usb_configure_lane(void __iomem *base,
- const unsigned int *regs,
const struct qmp_phy_init_tbl tbl[],
int num,
u8 lane_mask)
@@ -2082,43 +1958,37 @@ static void qmp_usb_configure_lane(void __iomem *base,
if (!(t->lane_mask & lane_mask))
continue;
- if (t->in_layout)
- writel(t->val, base + regs[t->offset]);
- else
- writel(t->val, base + t->offset);
+ writel(t->val, base + t->offset);
}
}
static void qmp_usb_configure(void __iomem *base,
- const unsigned int *regs,
const struct qmp_phy_init_tbl tbl[],
int num)
{
- qmp_usb_configure_lane(base, regs, tbl, num, 0xff);
+ qmp_usb_configure_lane(base, tbl, num, 0xff);
}
-static int qmp_usb_serdes_init(struct qmp_phy *qphy)
+static int qmp_usb_serdes_init(struct qmp_usb *qmp)
{
- const struct qmp_phy_cfg *cfg = qphy->cfg;
- void __iomem *serdes = qphy->serdes;
+ const struct qmp_phy_cfg *cfg = qmp->cfg;
+ void __iomem *serdes = qmp->serdes;
const struct qmp_phy_init_tbl *serdes_tbl = cfg->serdes_tbl;
int serdes_tbl_num = cfg->serdes_tbl_num;
- qmp_usb_configure(serdes, cfg->regs, serdes_tbl, serdes_tbl_num);
+ qmp_usb_configure(serdes, serdes_tbl, serdes_tbl_num);
return 0;
}
static int qmp_usb_init(struct phy *phy)
{
- struct qmp_phy *qphy = phy_get_drvdata(phy);
- struct qcom_qmp *qmp = qphy->qmp;
- const struct qmp_phy_cfg *cfg = qphy->cfg;
- void __iomem *pcs = qphy->pcs;
+ struct qmp_usb *qmp = phy_get_drvdata(phy);
+ const struct qmp_phy_cfg *cfg = qmp->cfg;
+ void __iomem *pcs = qmp->pcs;
void __iomem *dp_com = qmp->dp_com;
int ret;
- /* turn on regulator supplies */
ret = regulator_bulk_enable(cfg->num_vregs, qmp->vregs);
if (ret) {
dev_err(qmp->dev, "failed to enable regulators, err=%d\n", ret);
@@ -2164,13 +2034,7 @@ static int qmp_usb_init(struct phy *phy)
qphy_clrbits(dp_com, QPHY_V3_DP_COM_SW_RESET, SW_RESET);
}
- if (cfg->regs[QPHY_PCS_POWER_DOWN_CONTROL])
- qphy_setbits(pcs,
- cfg->regs[QPHY_PCS_POWER_DOWN_CONTROL],
- cfg->pwrdn_ctrl);
- else
- qphy_setbits(pcs, QPHY_V2_PCS_POWER_DOWN_CONTROL,
- cfg->pwrdn_ctrl);
+ qphy_setbits(pcs, cfg->regs[QPHY_PCS_POWER_DOWN_CONTROL], SW_PWRDN);
return 0;
@@ -2184,9 +2048,8 @@ static int qmp_usb_init(struct phy *phy)
static int qmp_usb_exit(struct phy *phy)
{
- struct qmp_phy *qphy = phy_get_drvdata(phy);
- struct qcom_qmp *qmp = qphy->qmp;
- const struct qmp_phy_cfg *cfg = qphy->cfg;
+ struct qmp_usb *qmp = phy_get_drvdata(phy);
+ const struct qmp_phy_cfg *cfg = qmp->cfg;
reset_control_bulk_assert(cfg->num_resets, qmp->resets);
@@ -2199,56 +2062,45 @@ static int qmp_usb_exit(struct phy *phy)
static int qmp_usb_power_on(struct phy *phy)
{
- struct qmp_phy *qphy = phy_get_drvdata(phy);
- struct qcom_qmp *qmp = qphy->qmp;
- const struct qmp_phy_cfg *cfg = qphy->cfg;
- void __iomem *tx = qphy->tx;
- void __iomem *rx = qphy->rx;
- void __iomem *pcs = qphy->pcs;
+ struct qmp_usb *qmp = phy_get_drvdata(phy);
+ const struct qmp_phy_cfg *cfg = qmp->cfg;
+ void __iomem *tx = qmp->tx;
+ void __iomem *rx = qmp->rx;
+ void __iomem *pcs = qmp->pcs;
void __iomem *status;
- unsigned int mask, val, ready;
+ unsigned int val;
int ret;
- qmp_usb_serdes_init(qphy);
+ qmp_usb_serdes_init(qmp);
- ret = clk_prepare_enable(qphy->pipe_clk);
+ ret = clk_prepare_enable(qmp->pipe_clk);
if (ret) {
dev_err(qmp->dev, "pipe_clk enable failed err=%d\n", ret);
return ret;
}
/* Tx, Rx, and PCS configurations */
- qmp_usb_configure_lane(tx, cfg->regs, cfg->tx_tbl, cfg->tx_tbl_num, 1);
+ qmp_usb_configure_lane(tx, cfg->tx_tbl, cfg->tx_tbl_num, 1);
+ qmp_usb_configure_lane(rx, cfg->rx_tbl, cfg->rx_tbl_num, 1);
if (cfg->lanes >= 2) {
- qmp_usb_configure_lane(qphy->tx2, cfg->regs,
- cfg->tx_tbl, cfg->tx_tbl_num, 2);
+ qmp_usb_configure_lane(qmp->tx2, cfg->tx_tbl, cfg->tx_tbl_num, 2);
+ qmp_usb_configure_lane(qmp->rx2, cfg->rx_tbl, cfg->rx_tbl_num, 2);
}
- qmp_usb_configure_lane(rx, cfg->regs, cfg->rx_tbl, cfg->rx_tbl_num, 1);
-
- if (cfg->lanes >= 2) {
- qmp_usb_configure_lane(qphy->rx2, cfg->regs,
- cfg->rx_tbl, cfg->rx_tbl_num, 2);
- }
-
- /* Configure link rate, swing, etc. */
- qmp_usb_configure(pcs, cfg->regs, cfg->pcs_tbl, cfg->pcs_tbl_num);
+ qmp_usb_configure(pcs, cfg->pcs_tbl, cfg->pcs_tbl_num);
if (cfg->has_pwrdn_delay)
- usleep_range(cfg->pwrdn_delay_min, cfg->pwrdn_delay_max);
+ usleep_range(10, 20);
/* Pull PHY out of reset state */
qphy_clrbits(pcs, cfg->regs[QPHY_SW_RESET], SW_RESET);
/* start SerDes and Phy-Coding-Sublayer */
- qphy_setbits(pcs, cfg->regs[QPHY_START_CTRL], cfg->start_ctrl);
+ qphy_setbits(pcs, cfg->regs[QPHY_START_CTRL], SERDES_START | PCS_START);
status = pcs + cfg->regs[QPHY_PCS_STATUS];
- mask = cfg->phy_status;
- ready = 0;
-
- ret = readl_poll_timeout(status, val, (val & mask) == ready, 10,
+ ret = readl_poll_timeout(status, val, !(val & PHYSTATUS), 200,
PHY_INIT_COMPLETE_TIMEOUT);
if (ret) {
dev_err(qmp->dev, "phy initialization timed-out\n");
@@ -2258,32 +2110,28 @@ static int qmp_usb_power_on(struct phy *phy)
return 0;
err_disable_pipe_clk:
- clk_disable_unprepare(qphy->pipe_clk);
+ clk_disable_unprepare(qmp->pipe_clk);
return ret;
}
static int qmp_usb_power_off(struct phy *phy)
{
- struct qmp_phy *qphy = phy_get_drvdata(phy);
- const struct qmp_phy_cfg *cfg = qphy->cfg;
+ struct qmp_usb *qmp = phy_get_drvdata(phy);
+ const struct qmp_phy_cfg *cfg = qmp->cfg;
- clk_disable_unprepare(qphy->pipe_clk);
+ clk_disable_unprepare(qmp->pipe_clk);
/* PHY reset */
- qphy_setbits(qphy->pcs, cfg->regs[QPHY_SW_RESET], SW_RESET);
+ qphy_setbits(qmp->pcs, cfg->regs[QPHY_SW_RESET], SW_RESET);
/* stop SerDes and Phy-Coding-Sublayer */
- qphy_clrbits(qphy->pcs, cfg->regs[QPHY_START_CTRL], cfg->start_ctrl);
+ qphy_clrbits(qmp->pcs, cfg->regs[QPHY_START_CTRL],
+ SERDES_START | PCS_START);
/* Put PHY into POWER DOWN state: active low */
- if (cfg->regs[QPHY_PCS_POWER_DOWN_CONTROL]) {
- qphy_clrbits(qphy->pcs, cfg->regs[QPHY_PCS_POWER_DOWN_CONTROL],
- cfg->pwrdn_ctrl);
- } else {
- qphy_clrbits(qphy->pcs, QPHY_V2_PCS_POWER_DOWN_CONTROL,
- cfg->pwrdn_ctrl);
- }
+ qphy_clrbits(qmp->pcs, cfg->regs[QPHY_PCS_POWER_DOWN_CONTROL],
+ SW_PWRDN);
return 0;
}
@@ -2315,22 +2163,29 @@ static int qmp_usb_disable(struct phy *phy)
static int qmp_usb_set_mode(struct phy *phy, enum phy_mode mode, int submode)
{
- struct qmp_phy *qphy = phy_get_drvdata(phy);
+ struct qmp_usb *qmp = phy_get_drvdata(phy);
- qphy->mode = mode;
+ qmp->mode = mode;
return 0;
}
-static void qmp_usb_enable_autonomous_mode(struct qmp_phy *qphy)
+static const struct phy_ops qmp_usb_phy_ops = {
+ .init = qmp_usb_enable,
+ .exit = qmp_usb_disable,
+ .set_mode = qmp_usb_set_mode,
+ .owner = THIS_MODULE,
+};
+
+static void qmp_usb_enable_autonomous_mode(struct qmp_usb *qmp)
{
- const struct qmp_phy_cfg *cfg = qphy->cfg;
- void __iomem *pcs_usb = qphy->pcs_usb ?: qphy->pcs;
- void __iomem *pcs_misc = qphy->pcs_misc;
+ const struct qmp_phy_cfg *cfg = qmp->cfg;
+ void __iomem *pcs_usb = qmp->pcs_usb ?: qmp->pcs;
+ void __iomem *pcs_misc = qmp->pcs_misc;
u32 intr_mask;
- if (qphy->mode == PHY_MODE_USB_HOST_SS ||
- qphy->mode == PHY_MODE_USB_DEVICE_SS)
+ if (qmp->mode == PHY_MODE_USB_HOST_SS ||
+ qmp->mode == PHY_MODE_USB_DEVICE_SS)
intr_mask = ARCVR_DTCT_EN | ALFPS_DTCT_EN;
else
intr_mask = ARCVR_DTCT_EN | ARCVR_DTCT_EVENT_SEL;
@@ -2351,11 +2206,11 @@ static void qmp_usb_enable_autonomous_mode(struct qmp_phy *qphy)
qphy_clrbits(pcs_misc, QPHY_V3_PCS_MISC_CLAMP_ENABLE, CLAMP_EN);
}
-static void qmp_usb_disable_autonomous_mode(struct qmp_phy *qphy)
+static void qmp_usb_disable_autonomous_mode(struct qmp_usb *qmp)
{
- const struct qmp_phy_cfg *cfg = qphy->cfg;
- void __iomem *pcs_usb = qphy->pcs_usb ?: qphy->pcs;
- void __iomem *pcs_misc = qphy->pcs_misc;
+ const struct qmp_phy_cfg *cfg = qmp->cfg;
+ void __iomem *pcs_usb = qmp->pcs_usb ?: qmp->pcs;
+ void __iomem *pcs_misc = qmp->pcs_misc;
/* Disable i/o clamp_n on resume for normal mode */
if (pcs_misc)
@@ -2371,20 +2226,19 @@ static void qmp_usb_disable_autonomous_mode(struct qmp_phy *qphy)
static int __maybe_unused qmp_usb_runtime_suspend(struct device *dev)
{
- struct qcom_qmp *qmp = dev_get_drvdata(dev);
- struct qmp_phy *qphy = qmp->phys[0];
- const struct qmp_phy_cfg *cfg = qphy->cfg;
+ struct qmp_usb *qmp = dev_get_drvdata(dev);
+ const struct qmp_phy_cfg *cfg = qmp->cfg;
- dev_vdbg(dev, "Suspending QMP phy, mode:%d\n", qphy->mode);
+ dev_vdbg(dev, "Suspending QMP phy, mode:%d\n", qmp->mode);
- if (!qphy->phy->init_count) {
+ if (!qmp->phy->init_count) {
dev_vdbg(dev, "PHY not initialized, bailing out\n");
return 0;
}
- qmp_usb_enable_autonomous_mode(qphy);
+ qmp_usb_enable_autonomous_mode(qmp);
- clk_disable_unprepare(qphy->pipe_clk);
+ clk_disable_unprepare(qmp->pipe_clk);
clk_bulk_disable_unprepare(cfg->num_clks, qmp->clks);
return 0;
@@ -2392,14 +2246,13 @@ static int __maybe_unused qmp_usb_runtime_suspend(struct device *dev)
static int __maybe_unused qmp_usb_runtime_resume(struct device *dev)
{
- struct qcom_qmp *qmp = dev_get_drvdata(dev);
- struct qmp_phy *qphy = qmp->phys[0];
- const struct qmp_phy_cfg *cfg = qphy->cfg;
+ struct qmp_usb *qmp = dev_get_drvdata(dev);
+ const struct qmp_phy_cfg *cfg = qmp->cfg;
int ret = 0;
- dev_vdbg(dev, "Resuming QMP phy, mode:%d\n", qphy->mode);
+ dev_vdbg(dev, "Resuming QMP phy, mode:%d\n", qmp->mode);
- if (!qphy->phy->init_count) {
+ if (!qmp->phy->init_count) {
dev_vdbg(dev, "PHY not initialized, bailing out\n");
return 0;
}
@@ -2408,21 +2261,27 @@ static int __maybe_unused qmp_usb_runtime_resume(struct device *dev)
if (ret)
return ret;
- ret = clk_prepare_enable(qphy->pipe_clk);
+ ret = clk_prepare_enable(qmp->pipe_clk);
if (ret) {
dev_err(dev, "pipe_clk enable failed, err=%d\n", ret);
clk_bulk_disable_unprepare(cfg->num_clks, qmp->clks);
return ret;
}
- qmp_usb_disable_autonomous_mode(qphy);
+ qmp_usb_disable_autonomous_mode(qmp);
return 0;
}
-static int qmp_usb_vreg_init(struct device *dev, const struct qmp_phy_cfg *cfg)
+static const struct dev_pm_ops qmp_usb_pm_ops = {
+ SET_RUNTIME_PM_OPS(qmp_usb_runtime_suspend,
+ qmp_usb_runtime_resume, NULL)
+};
+
+static int qmp_usb_vreg_init(struct qmp_usb *qmp)
{
- struct qcom_qmp *qmp = dev_get_drvdata(dev);
+ const struct qmp_phy_cfg *cfg = qmp->cfg;
+ struct device *dev = qmp->dev;
int num = cfg->num_vregs;
int i;
@@ -2436,9 +2295,10 @@ static int qmp_usb_vreg_init(struct device *dev, const struct qmp_phy_cfg *cfg)
return devm_regulator_bulk_get(dev, num, qmp->vregs);
}
-static int qmp_usb_reset_init(struct device *dev, const struct qmp_phy_cfg *cfg)
+static int qmp_usb_reset_init(struct qmp_usb *qmp)
{
- struct qcom_qmp *qmp = dev_get_drvdata(dev);
+ const struct qmp_phy_cfg *cfg = qmp->cfg;
+ struct device *dev = qmp->dev;
int i;
int ret;
@@ -2457,9 +2317,10 @@ static int qmp_usb_reset_init(struct device *dev, const struct qmp_phy_cfg *cfg)
return 0;
}
-static int qmp_usb_clk_init(struct device *dev, const struct qmp_phy_cfg *cfg)
+static int qmp_usb_clk_init(struct qmp_usb *qmp)
{
- struct qcom_qmp *qmp = dev_get_drvdata(dev);
+ const struct qmp_phy_cfg *cfg = qmp->cfg;
+ struct device *dev = qmp->dev;
int num = cfg->num_clks;
int i;
@@ -2496,9 +2357,9 @@ static void phy_clk_release_provider(void *res)
* clk | +-------+ | +-----+
* +---------------+
*/
-static int phy_pipe_clk_register(struct qcom_qmp *qmp, struct device_node *np)
+static int phy_pipe_clk_register(struct qmp_usb *qmp, struct device_node *np)
{
- struct clk_fixed_rate *fixed;
+ struct clk_fixed_rate *fixed = &qmp->pipe_clk_fixed;
struct clk_init_data init = { };
int ret;
@@ -2508,10 +2369,6 @@ static int phy_pipe_clk_register(struct qcom_qmp *qmp, struct device_node *np)
return ret;
}
- fixed = devm_kzalloc(qmp->dev, sizeof(*fixed), GFP_KERNEL);
- if (!fixed)
- return -ENOMEM;
-
init.ops = &clk_fixed_rate_ops;
/* controllers using QMP phys use 125MHz pipe clock interface */
@@ -2533,13 +2390,6 @@ static int phy_pipe_clk_register(struct qcom_qmp *qmp, struct device_node *np)
return devm_add_action_or_reset(qmp->dev, phy_clk_release_provider, np);
}
-static const struct phy_ops qmp_usb_ops = {
- .init = qmp_usb_enable,
- .exit = qmp_usb_disable,
- .set_mode = qmp_usb_set_mode,
- .owner = THIS_MODULE,
-};
-
static void __iomem *qmp_usb_iomap(struct device *dev, struct device_node *np,
int index, bool exclusive)
{
@@ -2555,15 +2405,22 @@ static void __iomem *qmp_usb_iomap(struct device *dev, struct device_node *np,
return devm_of_iomap(dev, np, index, NULL);
}
-static
-int qmp_usb_create(struct device *dev, struct device_node *np, int id,
- void __iomem *serdes, const struct qmp_phy_cfg *cfg)
+static int qmp_usb_parse_dt_legacy(struct qmp_usb *qmp, struct device_node *np)
{
- struct qcom_qmp *qmp = dev_get_drvdata(dev);
- struct phy *generic_phy;
- struct qmp_phy *qphy;
+ struct platform_device *pdev = to_platform_device(qmp->dev);
+ const struct qmp_phy_cfg *cfg = qmp->cfg;
+ struct device *dev = qmp->dev;
bool exclusive = true;
- int ret;
+
+ qmp->serdes = devm_platform_ioremap_resource(pdev, 0);
+ if (IS_ERR(qmp->serdes))
+ return PTR_ERR(qmp->serdes);
+
+ if (cfg->has_phy_dp_com_ctrl) {
+ qmp->dp_com = devm_platform_ioremap_resource(pdev, 1);
+ if (IS_ERR(qmp->dp_com))
+ return PTR_ERR(qmp->dp_com);
+ }
/*
* FIXME: These bindings should be fixed to not rely on overlapping
@@ -2574,83 +2431,176 @@ int qmp_usb_create(struct device *dev, struct device_node *np, int id,
if (of_device_is_compatible(dev->of_node, "qcom,sm8350-qmp-usb3-uni-phy"))
exclusive = false;
- qphy = devm_kzalloc(dev, sizeof(*qphy), GFP_KERNEL);
- if (!qphy)
- return -ENOMEM;
-
- qphy->cfg = cfg;
- qphy->serdes = serdes;
/*
- * Get memory resources for each phy lane:
+ * Get memory resources for the PHY:
* Resources are indexed as: tx -> 0; rx -> 1; pcs -> 2.
* For dual lane PHYs: tx2 -> 3, rx2 -> 4, pcs_misc (optional) -> 5
* For single lane PHYs: pcs_misc (optional) -> 3.
*/
- qphy->tx = devm_of_iomap(dev, np, 0, NULL);
- if (IS_ERR(qphy->tx))
- return PTR_ERR(qphy->tx);
+ qmp->tx = devm_of_iomap(dev, np, 0, NULL);
+ if (IS_ERR(qmp->tx))
+ return PTR_ERR(qmp->tx);
- qphy->rx = devm_of_iomap(dev, np, 1, NULL);
- if (IS_ERR(qphy->rx))
- return PTR_ERR(qphy->rx);
+ qmp->rx = devm_of_iomap(dev, np, 1, NULL);
+ if (IS_ERR(qmp->rx))
+ return PTR_ERR(qmp->rx);
- qphy->pcs = qmp_usb_iomap(dev, np, 2, exclusive);
- if (IS_ERR(qphy->pcs))
- return PTR_ERR(qphy->pcs);
+ qmp->pcs = qmp_usb_iomap(dev, np, 2, exclusive);
+ if (IS_ERR(qmp->pcs))
+ return PTR_ERR(qmp->pcs);
if (cfg->pcs_usb_offset)
- qphy->pcs_usb = qphy->pcs + cfg->pcs_usb_offset;
+ qmp->pcs_usb = qmp->pcs + cfg->pcs_usb_offset;
if (cfg->lanes >= 2) {
- qphy->tx2 = devm_of_iomap(dev, np, 3, NULL);
- if (IS_ERR(qphy->tx2))
- return PTR_ERR(qphy->tx2);
+ qmp->tx2 = devm_of_iomap(dev, np, 3, NULL);
+ if (IS_ERR(qmp->tx2))
+ return PTR_ERR(qmp->tx2);
- qphy->rx2 = devm_of_iomap(dev, np, 4, NULL);
- if (IS_ERR(qphy->rx2))
- return PTR_ERR(qphy->rx2);
+ qmp->rx2 = devm_of_iomap(dev, np, 4, NULL);
+ if (IS_ERR(qmp->rx2))
+ return PTR_ERR(qmp->rx2);
- qphy->pcs_misc = devm_of_iomap(dev, np, 5, NULL);
+ qmp->pcs_misc = devm_of_iomap(dev, np, 5, NULL);
} else {
- qphy->pcs_misc = devm_of_iomap(dev, np, 3, NULL);
+ qmp->pcs_misc = devm_of_iomap(dev, np, 3, NULL);
}
- if (IS_ERR(qphy->pcs_misc)) {
+ if (IS_ERR(qmp->pcs_misc)) {
dev_vdbg(dev, "PHY pcs_misc-reg not used\n");
- qphy->pcs_misc = NULL;
+ qmp->pcs_misc = NULL;
}
- qphy->pipe_clk = devm_get_clk_from_child(dev, np, NULL);
- if (IS_ERR(qphy->pipe_clk)) {
- return dev_err_probe(dev, PTR_ERR(qphy->pipe_clk),
- "failed to get lane%d pipe clock\n", id);
+ qmp->pipe_clk = devm_get_clk_from_child(dev, np, NULL);
+ if (IS_ERR(qmp->pipe_clk)) {
+ return dev_err_probe(dev, PTR_ERR(qmp->pipe_clk),
+ "failed to get pipe clock\n");
}
- generic_phy = devm_phy_create(dev, np, &qmp_usb_ops);
- if (IS_ERR(generic_phy)) {
- ret = PTR_ERR(generic_phy);
- dev_err(dev, "failed to create qphy %d\n", ret);
- return ret;
- }
-
- qphy->phy = generic_phy;
- qphy->qmp = qmp;
- qmp->phys[id] = qphy;
- phy_set_drvdata(generic_phy, qphy);
-
return 0;
}
+static int qmp_usb_parse_dt(struct qmp_usb *qmp)
+{
+ struct platform_device *pdev = to_platform_device(qmp->dev);
+ const struct qmp_phy_cfg *cfg = qmp->cfg;
+ const struct qmp_usb_offsets *offs = cfg->offsets;
+ struct device *dev = qmp->dev;
+ void __iomem *base;
+
+ if (!offs)
+ return -EINVAL;
+
+ base = devm_platform_ioremap_resource(pdev, 0);
+ if (IS_ERR(base))
+ return PTR_ERR(base);
+
+ qmp->serdes = base + offs->serdes;
+ qmp->pcs = base + offs->pcs;
+ qmp->pcs_usb = base + offs->pcs_usb;
+ qmp->tx = base + offs->tx;
+ qmp->rx = base + offs->rx;
+
+ qmp->pipe_clk = devm_clk_get(dev, "pipe");
+ if (IS_ERR(qmp->pipe_clk)) {
+ return dev_err_probe(dev, PTR_ERR(qmp->pipe_clk),
+ "failed to get pipe clock\n");
+ }
+
+ return 0;
+}
+
+static int qmp_usb_probe(struct platform_device *pdev)
+{
+ struct device *dev = &pdev->dev;
+ struct phy_provider *phy_provider;
+ struct device_node *np;
+ struct qmp_usb *qmp;
+ int ret;
+
+ qmp = devm_kzalloc(dev, sizeof(*qmp), GFP_KERNEL);
+ if (!qmp)
+ return -ENOMEM;
+
+ qmp->dev = dev;
+
+ qmp->cfg = of_device_get_match_data(dev);
+ if (!qmp->cfg)
+ return -EINVAL;
+
+ ret = qmp_usb_clk_init(qmp);
+ if (ret)
+ return ret;
+
+ ret = qmp_usb_reset_init(qmp);
+ if (ret)
+ return ret;
+
+ ret = qmp_usb_vreg_init(qmp);
+ if (ret)
+ return ret;
+
+ /* Check for legacy binding with child node. */
+ np = of_get_next_available_child(dev->of_node, NULL);
+ if (np) {
+ ret = qmp_usb_parse_dt_legacy(qmp, np);
+ } else {
+ np = of_node_get(dev->of_node);
+ ret = qmp_usb_parse_dt(qmp);
+ }
+ if (ret)
+ goto err_node_put;
+
+ pm_runtime_set_active(dev);
+ ret = devm_pm_runtime_enable(dev);
+ if (ret)
+ goto err_node_put;
+ /*
+ * Prevent runtime pm from being ON by default. Users can enable
+ * it using power/control in sysfs.
+ */
+ pm_runtime_forbid(dev);
+
+ ret = phy_pipe_clk_register(qmp, np);
+ if (ret)
+ goto err_node_put;
+
+ qmp->phy = devm_phy_create(dev, np, &qmp_usb_phy_ops);
+ if (IS_ERR(qmp->phy)) {
+ ret = PTR_ERR(qmp->phy);
+ dev_err(dev, "failed to create PHY: %d\n", ret);
+ goto err_node_put;
+ }
+
+ phy_set_drvdata(qmp->phy, qmp);
+
+ of_node_put(np);
+
+ phy_provider = devm_of_phy_provider_register(dev, of_phy_simple_xlate);
+
+ return PTR_ERR_OR_ZERO(phy_provider);
+
+err_node_put:
+ of_node_put(np);
+ return ret;
+}
+
static const struct of_device_id qmp_usb_of_match_table[] = {
{
+ .compatible = "qcom,ipq6018-qmp-usb3-phy",
+ .data = &ipq8074_usb3phy_cfg,
+ }, {
.compatible = "qcom,ipq8074-qmp-usb3-phy",
.data = &ipq8074_usb3phy_cfg,
}, {
.compatible = "qcom,msm8996-qmp-usb3-phy",
.data = &msm8996_usb3phy_cfg,
}, {
- .compatible = "qcom,ipq6018-qmp-usb3-phy",
- .data = &ipq8074_usb3phy_cfg,
+ .compatible = "qcom,msm8998-qmp-usb3-phy",
+ .data = &msm8998_usb3phy_cfg,
+ }, {
+ .compatible = "qcom,qcm2290-qmp-usb3-phy",
+ .data = &qcm2290_usb3phy_cfg,
}, {
.compatible = "qcom,sc7180-qmp-usb3-phy",
.data = &sc7180_usb3phy_cfg,
@@ -2667,8 +2617,11 @@ static const struct of_device_id qmp_usb_of_match_table[] = {
.compatible = "qcom,sdm845-qmp-usb3-uni-phy",
.data = &qmp_v3_usb3_uniphy_cfg,
}, {
- .compatible = "qcom,msm8998-qmp-usb3-phy",
- .data = &msm8998_usb3phy_cfg,
+ .compatible = "qcom,sdx55-qmp-usb3-uni-phy",
+ .data = &sdx55_usb3_uniphy_cfg,
+ }, {
+ .compatible = "qcom,sdx65-qmp-usb3-uni-phy",
+ .data = &sdx65_usb3_uniphy_cfg,
}, {
.compatible = "qcom,sm8150-qmp-usb3-phy",
.data = &sm8150_usb3phy_cfg,
@@ -2682,12 +2635,6 @@ static const struct of_device_id qmp_usb_of_match_table[] = {
.compatible = "qcom,sm8250-qmp-usb3-uni-phy",
.data = &sm8250_usb3_uniphy_cfg,
}, {
- .compatible = "qcom,sdx55-qmp-usb3-uni-phy",
- .data = &sdx55_usb3_uniphy_cfg,
- }, {
- .compatible = "qcom,sdx65-qmp-usb3-uni-phy",
- .data = &sdx65_usb3_uniphy_cfg,
- }, {
.compatible = "qcom,sm8350-qmp-usb3-phy",
.data = &sm8350_usb3phy_cfg,
}, {
@@ -2696,119 +2643,11 @@ static const struct of_device_id qmp_usb_of_match_table[] = {
}, {
.compatible = "qcom,sm8450-qmp-usb3-phy",
.data = &sm8350_usb3phy_cfg,
- }, {
- .compatible = "qcom,qcm2290-qmp-usb3-phy",
- .data = &qcm2290_usb3phy_cfg,
},
{ },
};
MODULE_DEVICE_TABLE(of, qmp_usb_of_match_table);
-static const struct dev_pm_ops qmp_usb_pm_ops = {
- SET_RUNTIME_PM_OPS(qmp_usb_runtime_suspend,
- qmp_usb_runtime_resume, NULL)
-};
-
-static int qmp_usb_probe(struct platform_device *pdev)
-{
- struct qcom_qmp *qmp;
- struct device *dev = &pdev->dev;
- struct device_node *child;
- struct phy_provider *phy_provider;
- void __iomem *serdes;
- const struct qmp_phy_cfg *cfg = NULL;
- int num, id;
- int ret;
-
- qmp = devm_kzalloc(dev, sizeof(*qmp), GFP_KERNEL);
- if (!qmp)
- return -ENOMEM;
-
- qmp->dev = dev;
- dev_set_drvdata(dev, qmp);
-
- /* Get the specific init parameters of QMP phy */
- cfg = of_device_get_match_data(dev);
- if (!cfg)
- return -EINVAL;
-
- /* per PHY serdes; usually located at base address */
- serdes = devm_platform_ioremap_resource(pdev, 0);
- if (IS_ERR(serdes))
- return PTR_ERR(serdes);
-
- /* per PHY dp_com; if PHY has dp_com control block */
- if (cfg->has_phy_dp_com_ctrl) {
- qmp->dp_com = devm_platform_ioremap_resource(pdev, 1);
- if (IS_ERR(qmp->dp_com))
- return PTR_ERR(qmp->dp_com);
- }
-
- ret = qmp_usb_clk_init(dev, cfg);
- if (ret)
- return ret;
-
- ret = qmp_usb_reset_init(dev, cfg);
- if (ret)
- return ret;
-
- ret = qmp_usb_vreg_init(dev, cfg);
- if (ret)
- return dev_err_probe(dev, ret,
- "failed to get regulator supplies\n");
-
- num = of_get_available_child_count(dev->of_node);
- /* do we have a rogue child node ? */
- if (num > 1)
- return -EINVAL;
-
- qmp->phys = devm_kcalloc(dev, num, sizeof(*qmp->phys), GFP_KERNEL);
- if (!qmp->phys)
- return -ENOMEM;
-
- pm_runtime_set_active(dev);
- ret = devm_pm_runtime_enable(dev);
- if (ret)
- return ret;
- /*
- * Prevent runtime pm from being ON by default. Users can enable
- * it using power/control in sysfs.
- */
- pm_runtime_forbid(dev);
-
- id = 0;
- for_each_available_child_of_node(dev->of_node, child) {
- /* Create per-lane phy */
- ret = qmp_usb_create(dev, child, id, serdes, cfg);
- if (ret) {
- dev_err(dev, "failed to create lane%d phy, %d\n",
- id, ret);
- goto err_node_put;
- }
-
- /*
- * Register the pipe clock provided by phy.
- * See function description to see details of this pipe clock.
- */
- ret = phy_pipe_clk_register(qmp, child);
- if (ret) {
- dev_err(qmp->dev,
- "failed to register pipe clock source\n");
- goto err_node_put;
- }
-
- id++;
- }
-
- phy_provider = devm_of_phy_provider_register(dev, of_phy_simple_xlate);
-
- return PTR_ERR_OR_ZERO(phy_provider);
-
-err_node_put:
- of_node_put(child);
- return ret;
-}
-
static struct platform_driver qmp_usb_driver = {
.probe = qmp_usb_probe,
.driver = {
diff --git a/drivers/phy/qualcomm/phy-qcom-qmp.h b/drivers/phy/qualcomm/phy-qcom-qmp.h
index 26274e3..29a48f04 100644
--- a/drivers/phy/qualcomm/phy-qcom-qmp.h
+++ b/drivers/phy/qualcomm/phy-qcom-qmp.h
@@ -38,6 +38,7 @@
#include "phy-qcom-qmp-pcs-pcie-v4_20.h"
#include "phy-qcom-qmp-pcs-v5.h"
+#include "phy-qcom-qmp-pcs-v5_20.h"
#include "phy-qcom-qmp-pcs-pcie-v5.h"
#include "phy-qcom-qmp-pcs-usb-v5.h"
#include "phy-qcom-qmp-pcs-ufs-v5.h"
diff --git a/drivers/phy/renesas/Kconfig b/drivers/phy/renesas/Kconfig
index 111bdca..36505fc 100644
--- a/drivers/phy/renesas/Kconfig
+++ b/drivers/phy/renesas/Kconfig
@@ -2,6 +2,14 @@
#
# Phy drivers for Renesas platforms
#
+# NOTE: Please sorted config names alphabetically.
+config PHY_R8A779F0_ETHERNET_SERDES
+ tristate "Renesas R-Car S4-8 Ethernet SERDES driver"
+ depends on ARCH_RENESAS || COMPILE_TEST
+ select GENERIC_PHY
+ help
+ Support for Ethernet SERDES found on Renesas R-Car S4-8 SoCs.
+
config PHY_RCAR_GEN2
tristate "Renesas R-Car generation 2 USB PHY driver"
depends on ARCH_RENESAS
diff --git a/drivers/phy/renesas/Makefile b/drivers/phy/renesas/Makefile
index b599ff8..8896d19 100644
--- a/drivers/phy/renesas/Makefile
+++ b/drivers/phy/renesas/Makefile
@@ -1,4 +1,5 @@
# SPDX-License-Identifier: GPL-2.0
+obj-$(CONFIG_PHY_R8A779F0_ETHERNET_SERDES) += r8a779f0-ether-serdes.o
obj-$(CONFIG_PHY_RCAR_GEN2) += phy-rcar-gen2.o
obj-$(CONFIG_PHY_RCAR_GEN3_PCIE) += phy-rcar-gen3-pcie.o
obj-$(CONFIG_PHY_RCAR_GEN3_USB2) += phy-rcar-gen3-usb2.o
diff --git a/drivers/phy/renesas/r8a779f0-ether-serdes.c b/drivers/phy/renesas/r8a779f0-ether-serdes.c
new file mode 100644
index 0000000..ec6594e
--- /dev/null
+++ b/drivers/phy/renesas/r8a779f0-ether-serdes.c
@@ -0,0 +1,417 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Renesas Ethernet SERDES device driver
+ *
+ * Copyright (C) 2022 Renesas Electronics Corporation
+ */
+
+#include <linux/delay.h>
+#include <linux/err.h>
+#include <linux/iopoll.h>
+#include <linux/kernel.h>
+#include <linux/phy.h>
+#include <linux/phy/phy.h>
+#include <linux/platform_device.h>
+#include <linux/reset.h>
+
+#define R8A779F0_ETH_SERDES_NUM 3
+#define R8A779F0_ETH_SERDES_OFFSET 0x0400
+#define R8A779F0_ETH_SERDES_BANK_SELECT 0x03fc
+#define R8A779F0_ETH_SERDES_TIMEOUT_US 100000
+#define R8A779F0_ETH_SERDES_NUM_RETRY_LINKUP 3
+#define R8A779F0_ETH_SERDES_NUM_RETRY_INIT 3
+
+struct r8a779f0_eth_serdes_drv_data;
+struct r8a779f0_eth_serdes_channel {
+ struct r8a779f0_eth_serdes_drv_data *dd;
+ struct phy *phy;
+ void __iomem *addr;
+ phy_interface_t phy_interface;
+ int speed;
+ int index;
+};
+
+struct r8a779f0_eth_serdes_drv_data {
+ void __iomem *addr;
+ struct platform_device *pdev;
+ struct reset_control *reset;
+ struct r8a779f0_eth_serdes_channel channel[R8A779F0_ETH_SERDES_NUM];
+ bool initialized;
+};
+
+/*
+ * The datasheet describes initialization procedure without any information
+ * about registers' name/bits. So, this is all black magic to initialize
+ * the hardware.
+ */
+static void r8a779f0_eth_serdes_write32(void __iomem *addr, u32 offs, u32 bank, u32 data)
+{
+ iowrite32(bank, addr + R8A779F0_ETH_SERDES_BANK_SELECT);
+ iowrite32(data, addr + offs);
+}
+
+static int
+r8a779f0_eth_serdes_reg_wait(struct r8a779f0_eth_serdes_channel *channel,
+ u32 offs, u32 bank, u32 mask, u32 expected)
+{
+ int ret;
+ u32 val;
+
+ iowrite32(bank, channel->addr + R8A779F0_ETH_SERDES_BANK_SELECT);
+
+ ret = readl_poll_timeout_atomic(channel->addr + offs, val,
+ (val & mask) == expected,
+ 1, R8A779F0_ETH_SERDES_TIMEOUT_US);
+ if (ret)
+ dev_dbg(&channel->phy->dev,
+ "%s: index %d, offs %x, bank %x, mask %x, expected %x\n",
+ __func__, channel->index, offs, bank, mask, expected);
+
+ return ret;
+}
+
+static int
+r8a779f0_eth_serdes_common_init_ram(struct r8a779f0_eth_serdes_drv_data *dd)
+{
+ struct r8a779f0_eth_serdes_channel *channel;
+ int i, ret;
+
+ for (i = 0; i < R8A779F0_ETH_SERDES_NUM; i++) {
+ channel = &dd->channel[i];
+ ret = r8a779f0_eth_serdes_reg_wait(channel, 0x026c, 0x180, BIT(0), 0x01);
+ if (ret)
+ return ret;
+ }
+
+ r8a779f0_eth_serdes_write32(dd->addr, 0x026c, 0x180, 0x03);
+
+ return ret;
+}
+
+static int
+r8a779f0_eth_serdes_common_setting(struct r8a779f0_eth_serdes_channel *channel)
+{
+ struct r8a779f0_eth_serdes_drv_data *dd = channel->dd;
+
+ switch (channel->phy_interface) {
+ case PHY_INTERFACE_MODE_SGMII:
+ r8a779f0_eth_serdes_write32(dd->addr, 0x0244, 0x180, 0x0097);
+ r8a779f0_eth_serdes_write32(dd->addr, 0x01d0, 0x180, 0x0060);
+ r8a779f0_eth_serdes_write32(dd->addr, 0x01d8, 0x180, 0x2200);
+ r8a779f0_eth_serdes_write32(dd->addr, 0x01d4, 0x180, 0x0000);
+ r8a779f0_eth_serdes_write32(dd->addr, 0x01e0, 0x180, 0x003d);
+ return 0;
+ default:
+ return -EOPNOTSUPP;
+ }
+}
+
+static int
+r8a779f0_eth_serdes_chan_setting(struct r8a779f0_eth_serdes_channel *channel)
+{
+ int ret;
+
+ switch (channel->phy_interface) {
+ case PHY_INTERFACE_MODE_SGMII:
+ r8a779f0_eth_serdes_write32(channel->addr, 0x0000, 0x380, 0x2000);
+ r8a779f0_eth_serdes_write32(channel->addr, 0x01c0, 0x180, 0x0011);
+ r8a779f0_eth_serdes_write32(channel->addr, 0x0248, 0x180, 0x0540);
+ r8a779f0_eth_serdes_write32(channel->addr, 0x0258, 0x180, 0x0015);
+ r8a779f0_eth_serdes_write32(channel->addr, 0x0144, 0x180, 0x0100);
+ r8a779f0_eth_serdes_write32(channel->addr, 0x01a0, 0x180, 0x0000);
+ r8a779f0_eth_serdes_write32(channel->addr, 0x00d0, 0x180, 0x0002);
+ r8a779f0_eth_serdes_write32(channel->addr, 0x0150, 0x180, 0x0003);
+ r8a779f0_eth_serdes_write32(channel->addr, 0x00c8, 0x180, 0x0100);
+ r8a779f0_eth_serdes_write32(channel->addr, 0x0148, 0x180, 0x0100);
+ r8a779f0_eth_serdes_write32(channel->addr, 0x0174, 0x180, 0x0000);
+ r8a779f0_eth_serdes_write32(channel->addr, 0x0160, 0x180, 0x0007);
+ r8a779f0_eth_serdes_write32(channel->addr, 0x01ac, 0x180, 0x0000);
+ r8a779f0_eth_serdes_write32(channel->addr, 0x00c4, 0x180, 0x0310);
+ r8a779f0_eth_serdes_write32(channel->addr, 0x00c8, 0x380, 0x0101);
+ ret = r8a779f0_eth_serdes_reg_wait(channel, 0x00c8, 0x0180, BIT(0), 0);
+ if (ret)
+ return ret;
+
+ r8a779f0_eth_serdes_write32(channel->addr, 0x0148, 0x180, 0x0101);
+ ret = r8a779f0_eth_serdes_reg_wait(channel, 0x0148, 0x0180, BIT(0), 0);
+ if (ret)
+ return ret;
+
+ r8a779f0_eth_serdes_write32(channel->addr, 0x00c4, 0x180, 0x1310);
+ r8a779f0_eth_serdes_write32(channel->addr, 0x00d8, 0x180, 0x1800);
+ r8a779f0_eth_serdes_write32(channel->addr, 0x00dc, 0x180, 0x0000);
+ r8a779f0_eth_serdes_write32(channel->addr, 0x001c, 0x300, 0x0001);
+ r8a779f0_eth_serdes_write32(channel->addr, 0x0000, 0x380, 0x2100);
+ ret = r8a779f0_eth_serdes_reg_wait(channel, 0x0000, 0x0380, BIT(8), 0);
+ if (ret)
+ return ret;
+
+ if (channel->speed == 1000)
+ r8a779f0_eth_serdes_write32(channel->addr, 0x0000, 0x1f00, 0x0140);
+ else if (channel->speed == 100)
+ r8a779f0_eth_serdes_write32(channel->addr, 0x0000, 0x1f00, 0x2100);
+
+ /* For AN_ON */
+ r8a779f0_eth_serdes_write32(channel->addr, 0x0004, 0x1f80, 0x0005);
+ r8a779f0_eth_serdes_write32(channel->addr, 0x0028, 0x1f80, 0x07a1);
+ r8a779f0_eth_serdes_write32(channel->addr, 0x0000, 0x1f80, 0x0208);
+ break;
+ default:
+ return -EOPNOTSUPP;
+ }
+
+ return 0;
+}
+
+static int
+r8a779f0_eth_serdes_chan_speed(struct r8a779f0_eth_serdes_channel *channel)
+{
+ int ret;
+
+ switch (channel->phy_interface) {
+ case PHY_INTERFACE_MODE_SGMII:
+ /* For AN_ON */
+ if (channel->speed == 1000)
+ r8a779f0_eth_serdes_write32(channel->addr, 0x0000, 0x1f00, 0x1140);
+ else if (channel->speed == 100)
+ r8a779f0_eth_serdes_write32(channel->addr, 0x0000, 0x1f00, 0x3100);
+ ret = r8a779f0_eth_serdes_reg_wait(channel, 0x0008, 0x1f80, BIT(0), 1);
+ if (ret)
+ return ret;
+ r8a779f0_eth_serdes_write32(channel->addr, 0x0008, 0x1f80, 0x0000);
+ break;
+ default:
+ return -EOPNOTSUPP;
+ }
+
+ return 0;
+}
+
+
+static int r8a779f0_eth_serdes_monitor_linkup(struct r8a779f0_eth_serdes_channel *channel)
+{
+ int i, ret;
+
+ for (i = 0; i < R8A779F0_ETH_SERDES_NUM_RETRY_LINKUP; i++) {
+ ret = r8a779f0_eth_serdes_reg_wait(channel, 0x0004, 0x300,
+ BIT(2), BIT(2));
+ if (!ret)
+ break;
+
+ /* restart */
+ r8a779f0_eth_serdes_write32(channel->addr, 0x0144, 0x180, 0x0100);
+ udelay(1);
+ r8a779f0_eth_serdes_write32(channel->addr, 0x0144, 0x180, 0x0000);
+ }
+
+ return ret;
+}
+
+static int r8a779f0_eth_serdes_hw_init(struct r8a779f0_eth_serdes_channel *channel)
+{
+ struct r8a779f0_eth_serdes_drv_data *dd = channel->dd;
+ int i, ret;
+
+ if (dd->initialized)
+ return 0;
+
+ ret = r8a779f0_eth_serdes_common_init_ram(dd);
+ if (ret)
+ return ret;
+
+ for (i = 0; i < R8A779F0_ETH_SERDES_NUM; i++) {
+ ret = r8a779f0_eth_serdes_reg_wait(&dd->channel[i], 0x0000,
+ 0x300, BIT(15), 0);
+ if (ret)
+ return ret;
+ }
+
+ for (i = 0; i < R8A779F0_ETH_SERDES_NUM; i++)
+ r8a779f0_eth_serdes_write32(dd->channel[i].addr, 0x03d4, 0x380, 0x0443);
+
+ ret = r8a779f0_eth_serdes_common_setting(channel);
+ if (ret)
+ return ret;
+
+ for (i = 0; i < R8A779F0_ETH_SERDES_NUM; i++)
+ r8a779f0_eth_serdes_write32(dd->channel[i].addr, 0x03d0, 0x380, 0x0001);
+
+
+ r8a779f0_eth_serdes_write32(dd->addr, 0x0000, 0x380, 0x8000);
+
+ ret = r8a779f0_eth_serdes_common_init_ram(dd);
+ if (ret)
+ return ret;
+
+ ret = r8a779f0_eth_serdes_reg_wait(&dd->channel[0], 0x0000, 0x380, BIT(15), 0);
+ if (ret)
+ return ret;
+
+ for (i = 0; i < R8A779F0_ETH_SERDES_NUM; i++) {
+ ret = r8a779f0_eth_serdes_chan_setting(&dd->channel[i]);
+ if (ret)
+ return ret;
+ }
+
+ for (i = 0; i < R8A779F0_ETH_SERDES_NUM; i++) {
+ ret = r8a779f0_eth_serdes_chan_speed(&dd->channel[i]);
+ if (ret)
+ return ret;
+ }
+
+ for (i = 0; i < R8A779F0_ETH_SERDES_NUM; i++)
+ r8a779f0_eth_serdes_write32(dd->channel[i].addr, 0x03c0, 0x380, 0x0000);
+ for (i = 0; i < R8A779F0_ETH_SERDES_NUM; i++)
+ r8a779f0_eth_serdes_write32(dd->channel[i].addr, 0x03d0, 0x380, 0x0000);
+
+ for (i = 0; i < R8A779F0_ETH_SERDES_NUM; i++) {
+ ret = r8a779f0_eth_serdes_monitor_linkup(&dd->channel[i]);
+ if (ret)
+ return ret;
+ }
+
+ return 0;
+}
+
+static int r8a779f0_eth_serdes_init(struct phy *p)
+{
+ struct r8a779f0_eth_serdes_channel *channel = phy_get_drvdata(p);
+ int i, ret;
+
+ for (i = 0; i < R8A779F0_ETH_SERDES_NUM_RETRY_INIT; i++) {
+ ret = r8a779f0_eth_serdes_hw_init(channel);
+ if (!ret) {
+ channel->dd->initialized = true;
+ break;
+ }
+ usleep_range(1000, 2000);
+ }
+
+ return ret;
+}
+
+static int r8a779f0_eth_serdes_set_mode(struct phy *p, enum phy_mode mode,
+ int submode)
+{
+ struct r8a779f0_eth_serdes_channel *channel = phy_get_drvdata(p);
+
+ if (mode != PHY_MODE_ETHERNET)
+ return -EOPNOTSUPP;
+
+ switch (submode) {
+ case PHY_INTERFACE_MODE_GMII:
+ case PHY_INTERFACE_MODE_SGMII:
+ case PHY_INTERFACE_MODE_USXGMII:
+ channel->phy_interface = submode;
+ return 0;
+ default:
+ return -EOPNOTSUPP;
+ }
+}
+
+static int r8a779f0_eth_serdes_set_speed(struct phy *p, int speed)
+{
+ struct r8a779f0_eth_serdes_channel *channel = phy_get_drvdata(p);
+
+ channel->speed = speed;
+
+ return 0;
+}
+
+static const struct phy_ops r8a779f0_eth_serdes_ops = {
+ .init = r8a779f0_eth_serdes_init,
+ .set_mode = r8a779f0_eth_serdes_set_mode,
+ .set_speed = r8a779f0_eth_serdes_set_speed,
+};
+
+static struct phy *r8a779f0_eth_serdes_xlate(struct device *dev,
+ struct of_phandle_args *args)
+{
+ struct r8a779f0_eth_serdes_drv_data *dd = dev_get_drvdata(dev);
+
+ if (args->args[0] >= R8A779F0_ETH_SERDES_NUM)
+ return ERR_PTR(-ENODEV);
+
+ return dd->channel[args->args[0]].phy;
+}
+
+static const struct of_device_id r8a779f0_eth_serdes_of_table[] = {
+ { .compatible = "renesas,r8a779f0-ether-serdes", },
+ { }
+};
+MODULE_DEVICE_TABLE(of, r8a779f0_eth_serdes_of_table);
+
+static int r8a779f0_eth_serdes_probe(struct platform_device *pdev)
+{
+ struct r8a779f0_eth_serdes_drv_data *dd;
+ struct phy_provider *provider;
+ struct resource *res;
+ int i;
+
+ res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
+ if (!res) {
+ dev_err(&pdev->dev, "invalid resource\n");
+ return -EINVAL;
+ }
+
+ dd = devm_kzalloc(&pdev->dev, sizeof(*dd), GFP_KERNEL);
+ if (!dd)
+ return -ENOMEM;
+
+ platform_set_drvdata(pdev, dd);
+ dd->pdev = pdev;
+ dd->addr = devm_ioremap_resource(&pdev->dev, res);
+ if (IS_ERR(dd->addr))
+ return PTR_ERR(dd->addr);
+
+ dd->reset = devm_reset_control_get(&pdev->dev, NULL);
+ if (IS_ERR(dd->reset))
+ return PTR_ERR(dd->reset);
+
+ reset_control_reset(dd->reset);
+
+ for (i = 0; i < R8A779F0_ETH_SERDES_NUM; i++) {
+ struct r8a779f0_eth_serdes_channel *channel = &dd->channel[i];
+
+ channel->phy = devm_phy_create(&pdev->dev, NULL,
+ &r8a779f0_eth_serdes_ops);
+ if (IS_ERR(channel->phy))
+ return PTR_ERR(channel->phy);
+ channel->addr = dd->addr + R8A779F0_ETH_SERDES_OFFSET * i;
+ channel->dd = dd;
+ channel->index = i;
+ phy_set_drvdata(channel->phy, channel);
+ }
+
+ provider = devm_of_phy_provider_register(&pdev->dev,
+ r8a779f0_eth_serdes_xlate);
+ if (IS_ERR(provider))
+ return PTR_ERR(provider);
+
+ pm_runtime_enable(&pdev->dev);
+ pm_runtime_get_sync(&pdev->dev);
+
+ return 0;
+}
+
+static int r8a779f0_eth_serdes_remove(struct platform_device *pdev)
+{
+ pm_runtime_put(&pdev->dev);
+ pm_runtime_disable(&pdev->dev);
+
+ platform_set_drvdata(pdev, NULL);
+
+ return 0;
+}
+
+static struct platform_driver r8a779f0_eth_serdes_driver_platform = {
+ .probe = r8a779f0_eth_serdes_probe,
+ .remove = r8a779f0_eth_serdes_remove,
+ .driver = {
+ .name = "r8a779f0_eth_serdes",
+ .of_match_table = r8a779f0_eth_serdes_of_table,
+ }
+};
+module_platform_driver(r8a779f0_eth_serdes_driver_platform);
+MODULE_AUTHOR("Yoshihiro Shimoda");
+MODULE_DESCRIPTION("Renesas Ethernet SERDES device driver");
+MODULE_LICENSE("GPL");
diff --git a/drivers/phy/tegra/phy-tegra194-p2u.c b/drivers/phy/tegra/phy-tegra194-p2u.c
index 1415ca7..633e6b7 100644
--- a/drivers/phy/tegra/phy-tegra194-p2u.c
+++ b/drivers/phy/tegra/phy-tegra194-p2u.c
@@ -15,6 +15,7 @@
#include <linux/phy/phy.h>
#define P2U_CONTROL_CMN 0x74
+#define P2U_CONTROL_CMN_ENABLE_L2_EXIT_RATE_CHANGE BIT(13)
#define P2U_CONTROL_CMN_SKP_SIZE_PROTECTION_EN BIT(20)
#define P2U_PERIODIC_EQ_CTRL_GEN3 0xc0
@@ -85,8 +86,21 @@ static int tegra_p2u_power_on(struct phy *x)
return 0;
}
+static int tegra_p2u_calibrate(struct phy *x)
+{
+ struct tegra_p2u *phy = phy_get_drvdata(x);
+ u32 val;
+
+ val = p2u_readl(phy, P2U_CONTROL_CMN);
+ val |= P2U_CONTROL_CMN_ENABLE_L2_EXIT_RATE_CHANGE;
+ p2u_writel(phy, val, P2U_CONTROL_CMN);
+
+ return 0;
+}
+
static const struct phy_ops ops = {
.power_on = tegra_p2u_power_on,
+ .calibrate = tegra_p2u_calibrate,
.owner = THIS_MODULE,
};
diff --git a/drivers/phy/tegra/xusb-tegra124.c b/drivers/phy/tegra/xusb-tegra124.c
index db56c7f..f4f75ea 100644
--- a/drivers/phy/tegra/xusb-tegra124.c
+++ b/drivers/phy/tegra/xusb-tegra124.c
@@ -1652,7 +1652,6 @@ tegra124_usb3_port_map(struct tegra_xusb_port *port)
static const struct tegra_xusb_port_ops tegra124_usb3_port_ops = {
.release = tegra_xusb_usb3_port_release,
- .remove = tegra_xusb_usb3_port_remove,
.enable = tegra124_usb3_port_enable,
.disable = tegra124_usb3_port_disable,
.map = tegra124_usb3_port_map,
diff --git a/drivers/phy/tegra/xusb-tegra186.c b/drivers/phy/tegra/xusb-tegra186.c
index 0996ede..6a8bd87 100644
--- a/drivers/phy/tegra/xusb-tegra186.c
+++ b/drivers/phy/tegra/xusb-tegra186.c
@@ -1185,7 +1185,6 @@ tegra186_usb3_port_map(struct tegra_xusb_port *port)
static const struct tegra_xusb_port_ops tegra186_usb3_port_ops = {
.release = tegra_xusb_usb3_port_release,
- .remove = tegra_xusb_usb3_port_remove,
.enable = tegra186_usb3_port_enable,
.disable = tegra186_usb3_port_disable,
.map = tegra186_usb3_port_map,
diff --git a/drivers/phy/tegra/xusb-tegra210.c b/drivers/phy/tegra/xusb-tegra210.c
index eedfc7c..ebc8a7e 100644
--- a/drivers/phy/tegra/xusb-tegra210.c
+++ b/drivers/phy/tegra/xusb-tegra210.c
@@ -3078,7 +3078,6 @@ tegra210_usb3_port_map(struct tegra_xusb_port *port)
static const struct tegra_xusb_port_ops tegra210_usb3_port_ops = {
.release = tegra_xusb_usb3_port_release,
- .remove = tegra_xusb_usb3_port_remove,
.enable = tegra210_usb3_port_enable,
.disable = tegra210_usb3_port_disable,
.map = tegra210_usb3_port_map,
diff --git a/drivers/phy/tegra/xusb.c b/drivers/phy/tegra/xusb.c
index dce45fb..ff4b930 100644
--- a/drivers/phy/tegra/xusb.c
+++ b/drivers/phy/tegra/xusb.c
@@ -954,8 +954,7 @@ static int tegra_xusb_usb3_port_parse_dt(struct tegra_xusb_usb3_port *usb3)
return -EINVAL;
}
- usb3->supply = regulator_get(&port->dev, "vbus");
- return PTR_ERR_OR_ZERO(usb3->supply);
+ return 0;
}
static int tegra_xusb_add_usb3_port(struct tegra_xusb_padctl *padctl,
@@ -1012,13 +1011,6 @@ void tegra_xusb_usb3_port_release(struct tegra_xusb_port *port)
kfree(usb3);
}
-void tegra_xusb_usb3_port_remove(struct tegra_xusb_port *port)
-{
- struct tegra_xusb_usb3_port *usb3 = to_usb3_port(port);
-
- regulator_put(usb3->supply);
-}
-
static void __tegra_xusb_remove_ports(struct tegra_xusb_padctl *padctl)
{
struct tegra_xusb_port *port, *tmp;
diff --git a/drivers/phy/tegra/xusb.h b/drivers/phy/tegra/xusb.h
index 8cfbbdb..c384734 100644
--- a/drivers/phy/tegra/xusb.h
+++ b/drivers/phy/tegra/xusb.h
@@ -359,7 +359,6 @@ void tegra_xusb_hsic_port_release(struct tegra_xusb_port *port);
struct tegra_xusb_usb3_port {
struct tegra_xusb_port base;
- struct regulator *supply;
bool context_saved;
unsigned int port;
bool internal;
@@ -381,7 +380,6 @@ struct tegra_xusb_usb3_port *
tegra_xusb_find_usb3_port(struct tegra_xusb_padctl *padctl,
unsigned int index);
void tegra_xusb_usb3_port_release(struct tegra_xusb_port *port);
-void tegra_xusb_usb3_port_remove(struct tegra_xusb_port *port);
struct tegra_xusb_port_ops {
void (*release)(struct tegra_xusb_port *port);
diff --git a/drivers/phy/ti/phy-gmii-sel.c b/drivers/phy/ti/phy-gmii-sel.c
index 0bcfd6d..8c66781 100644
--- a/drivers/phy/ti/phy-gmii-sel.c
+++ b/drivers/phy/ti/phy-gmii-sel.c
@@ -50,6 +50,7 @@ struct phy_gmii_sel_soc_data {
const struct reg_field (*regfields)[PHY_GMII_SEL_LAST];
bool use_of_data;
u64 extra_modes;
+ u32 num_qsgmii_main_ports;
};
struct phy_gmii_sel_priv {
@@ -213,6 +214,17 @@ struct phy_gmii_sel_soc_data phy_gmii_sel_cpsw5g_soc_j7200 = {
.use_of_data = true,
.regfields = phy_gmii_sel_fields_am654,
.extra_modes = BIT(PHY_INTERFACE_MODE_QSGMII),
+ .num_ports = 4,
+ .num_qsgmii_main_ports = 1,
+};
+
+static const
+struct phy_gmii_sel_soc_data phy_gmii_sel_cpsw9g_soc_j721e = {
+ .use_of_data = true,
+ .regfields = phy_gmii_sel_fields_am654,
+ .extra_modes = BIT(PHY_INTERFACE_MODE_QSGMII),
+ .num_ports = 8,
+ .num_qsgmii_main_ports = 2,
};
static const struct of_device_id phy_gmii_sel_id_table[] = {
@@ -240,6 +252,10 @@ static const struct of_device_id phy_gmii_sel_id_table[] = {
.compatible = "ti,j7200-cpsw5g-phy-gmii-sel",
.data = &phy_gmii_sel_cpsw5g_soc_j7200,
},
+ {
+ .compatible = "ti,j721e-cpsw9g-phy-gmii-sel",
+ .data = &phy_gmii_sel_cpsw9g_soc_j721e,
+ },
{}
};
MODULE_DEVICE_TABLE(of, phy_gmii_sel_id_table);
@@ -378,11 +394,13 @@ static int phy_gmii_sel_init_ports(struct phy_gmii_sel_priv *priv)
static int phy_gmii_sel_probe(struct platform_device *pdev)
{
struct device *dev = &pdev->dev;
+ const struct phy_gmii_sel_soc_data *soc_data;
struct device_node *node = dev->of_node;
const struct of_device_id *of_id;
struct phy_gmii_sel_priv *priv;
u32 main_ports = 1;
int ret;
+ u32 i;
of_id = of_match_node(phy_gmii_sel_id_table, pdev->dev.of_node);
if (!of_id)
@@ -394,16 +412,26 @@ static int phy_gmii_sel_probe(struct platform_device *pdev)
priv->dev = &pdev->dev;
priv->soc_data = of_id->data;
+ soc_data = priv->soc_data;
priv->num_ports = priv->soc_data->num_ports;
- of_property_read_u32(node, "ti,qsgmii-main-ports", &main_ports);
+ priv->qsgmii_main_ports = 0;
+
/*
- * Ensure that main_ports is within bounds. If the property
- * ti,qsgmii-main-ports is not mentioned, or the value mentioned
- * is out of bounds, default to 1.
+ * Based on the compatible, try to read the appropriate number of
+ * QSGMII main ports from the "ti,qsgmii-main-ports" property from
+ * the device-tree node.
*/
- if (main_ports < 1 || main_ports > 4)
- main_ports = 1;
- priv->qsgmii_main_ports = PHY_GMII_PORT(main_ports);
+ for (i = 0; i < soc_data->num_qsgmii_main_ports; i++) {
+ of_property_read_u32_index(node, "ti,qsgmii-main-ports", i, &main_ports);
+ /*
+ * Ensure that main_ports is within bounds.
+ */
+ if (main_ports < 1 || main_ports > soc_data->num_ports) {
+ dev_err(dev, "Invalid qsgmii main port provided\n");
+ return -EINVAL;
+ }
+ priv->qsgmii_main_ports |= PHY_GMII_PORT(main_ports);
+ }
priv->regmap = syscon_node_to_regmap(node->parent);
if (IS_ERR(priv->regmap)) {
diff --git a/drivers/phy/ti/phy-j721e-wiz.c b/drivers/phy/ti/phy-j721e-wiz.c
index 41725c6..ddce5ef 100644
--- a/drivers/phy/ti/phy-j721e-wiz.c
+++ b/drivers/phy/ti/phy-j721e-wiz.c
@@ -81,14 +81,20 @@ static const struct reg_field phy_reset_n = REG_FIELD(WIZ_SERDES_RST, 31, 31);
static const struct reg_field phy_en_refclk = REG_FIELD(WIZ_SERDES_RST, 30, 30);
static const struct reg_field pll1_refclk_mux_sel =
REG_FIELD(WIZ_SERDES_RST, 29, 29);
+static const struct reg_field pll1_refclk_mux_sel_2 =
+ REG_FIELD(WIZ_SERDES_RST, 22, 23);
static const struct reg_field pll0_refclk_mux_sel =
REG_FIELD(WIZ_SERDES_RST, 28, 28);
+static const struct reg_field pll0_refclk_mux_sel_2 =
+ REG_FIELD(WIZ_SERDES_RST, 28, 29);
static const struct reg_field refclk_dig_sel_16g =
REG_FIELD(WIZ_SERDES_RST, 24, 25);
static const struct reg_field refclk_dig_sel_10g =
REG_FIELD(WIZ_SERDES_RST, 24, 24);
static const struct reg_field pma_cmn_refclk_int_mode =
REG_FIELD(WIZ_SERDES_TOP_CTRL, 28, 29);
+static const struct reg_field pma_cmn_refclk1_int_mode =
+ REG_FIELD(WIZ_SERDES_TOP_CTRL, 20, 21);
static const struct reg_field pma_cmn_refclk_mode =
REG_FIELD(WIZ_SERDES_TOP_CTRL, 30, 31);
static const struct reg_field pma_cmn_refclk_dig_div =
@@ -315,6 +321,8 @@ enum wiz_type {
J721E_WIZ_10G, /* Also for J7200 SR1.0 */
AM64_WIZ_10G,
J7200_WIZ_10G, /* J7200 SR2.0 */
+ J784S4_WIZ_10G,
+ J721S2_WIZ_10G,
};
struct wiz_data {
@@ -992,6 +1000,8 @@ static void wiz_clock_cleanup(struct wiz *wiz, struct device_node *node)
switch (wiz->type) {
case AM64_WIZ_10G:
case J7200_WIZ_10G:
+ case J784S4_WIZ_10G:
+ case J721S2_WIZ_10G:
of_clk_del_provider(dev->of_node);
return;
default:
@@ -1123,6 +1133,8 @@ static int wiz_clock_init(struct wiz *wiz, struct device_node *node)
switch (wiz->type) {
case AM64_WIZ_10G:
case J7200_WIZ_10G:
+ case J784S4_WIZ_10G:
+ case J721S2_WIZ_10G:
ret = wiz_clock_register(wiz);
if (ret)
dev_err(dev, "Failed to register wiz clocks\n");
@@ -1205,6 +1217,7 @@ static int wiz_phy_fullrt_div(struct wiz *wiz, int lane)
break;
case J721E_WIZ_10G:
case J7200_WIZ_10G:
+ case J721S2_WIZ_10G:
if (wiz->lane_phy_type[lane] == PHY_TYPE_SGMII)
return regmap_field_write(wiz->p0_fullrt_div[lane], 0x2);
break;
@@ -1299,6 +1312,25 @@ static struct wiz_data j7200_pg2_10g_data = {
.clk_div_sel_num = WIZ_DIV_NUM_CLOCKS_10G,
};
+static struct wiz_data j784s4_10g_data = {
+ .type = J784S4_WIZ_10G,
+ .pll0_refclk_mux_sel = &pll0_refclk_mux_sel_2,
+ .pll1_refclk_mux_sel = &pll1_refclk_mux_sel_2,
+ .refclk_dig_sel = &refclk_dig_sel_16g,
+ .pma_cmn_refclk1_int_mode = &pma_cmn_refclk1_int_mode,
+ .clk_mux_sel = clk_mux_sel_10g_2_refclk,
+ .clk_div_sel_num = WIZ_DIV_NUM_CLOCKS_10G,
+};
+
+static struct wiz_data j721s2_10g_data = {
+ .type = J721S2_WIZ_10G,
+ .pll0_refclk_mux_sel = &pll0_refclk_mux_sel,
+ .pll1_refclk_mux_sel = &pll1_refclk_mux_sel,
+ .refclk_dig_sel = &refclk_dig_sel_10g,
+ .clk_mux_sel = clk_mux_sel_10g,
+ .clk_div_sel_num = WIZ_DIV_NUM_CLOCKS_10G,
+};
+
static const struct of_device_id wiz_id_table[] = {
{
.compatible = "ti,j721e-wiz-16g", .data = &j721e_16g_data,
@@ -1312,6 +1344,12 @@ static const struct of_device_id wiz_id_table[] = {
{
.compatible = "ti,j7200-wiz-10g", .data = &j7200_pg2_10g_data,
},
+ {
+ .compatible = "ti,j784s4-wiz-10g", .data = &j784s4_10g_data,
+ },
+ {
+ .compatible = "ti,j721s2-wiz-10g", .data = &j721s2_10g_data,
+ },
{}
};
MODULE_DEVICE_TABLE(of, wiz_id_table);
diff --git a/drivers/power/supply/88pm860x_charger.c b/drivers/power/supply/88pm860x_charger.c
index f21ce52f..2b9fcb7 100644
--- a/drivers/power/supply/88pm860x_charger.c
+++ b/drivers/power/supply/88pm860x_charger.c
@@ -690,8 +690,7 @@ static int pm860x_charger_probe(struct platform_device *pdev)
(chip->id == CHIP_PM8607) ? chip->companion : chip->client;
if (!info->i2c_8606) {
dev_err(&pdev->dev, "Missed I2C address of 88PM8606!\n");
- ret = -EINVAL;
- goto out;
+ return -EINVAL;
}
info->dev = &pdev->dev;
@@ -704,44 +703,26 @@ static int pm860x_charger_probe(struct platform_device *pdev)
psy_cfg.drv_data = info;
psy_cfg.supplied_to = pm860x_supplied_to;
psy_cfg.num_supplicants = ARRAY_SIZE(pm860x_supplied_to);
- info->usb = power_supply_register(&pdev->dev, &pm860x_charger_desc,
- &psy_cfg);
+ info->usb = devm_power_supply_register(&pdev->dev, &pm860x_charger_desc,
+ &psy_cfg);
if (IS_ERR(info->usb)) {
- ret = PTR_ERR(info->usb);
- goto out;
+ return PTR_ERR(info->usb);
}
pm860x_init_charger(info);
for (i = 0; i < ARRAY_SIZE(info->irq); i++) {
- ret = request_threaded_irq(info->irq[i], NULL,
- pm860x_irq_descs[i].handler,
- IRQF_ONESHOT, pm860x_irq_descs[i].name, info);
+ ret = devm_request_threaded_irq(&pdev->dev, info->irq[i], NULL,
+ pm860x_irq_descs[i].handler,
+ IRQF_ONESHOT,
+ pm860x_irq_descs[i].name, info);
if (ret < 0) {
dev_err(chip->dev, "Failed to request IRQ: #%d: %d\n",
info->irq[i], ret);
- goto out_irq;
+ return ret;
}
}
return 0;
-
-out_irq:
- power_supply_unregister(info->usb);
- while (--i >= 0)
- free_irq(info->irq[i], info);
-out:
- return ret;
-}
-
-static int pm860x_charger_remove(struct platform_device *pdev)
-{
- struct pm860x_charger_info *info = platform_get_drvdata(pdev);
- int i;
-
- power_supply_unregister(info->usb);
- for (i = 0; i < info->irq_nums; i++)
- free_irq(info->irq[i], info);
- return 0;
}
static struct platform_driver pm860x_charger_driver = {
@@ -749,7 +730,6 @@ static struct platform_driver pm860x_charger_driver = {
.name = "88pm860x-charger",
},
.probe = pm860x_charger_probe,
- .remove = pm860x_charger_remove,
};
module_platform_driver(pm860x_charger_driver);
diff --git a/drivers/power/supply/ab8500_charger.c b/drivers/power/supply/ab8500_charger.c
index c19c504..308e685 100644
--- a/drivers/power/supply/ab8500_charger.c
+++ b/drivers/power/supply/ab8500_charger.c
@@ -1940,7 +1940,7 @@ static int ab8500_charger_get_ext_psy_data(struct device *dev, void *data)
*
* Due to a asic bug it is necessary to lower the input current to the vbus
* charger when charging with at some specific levels. This issue is only valid
- * for below a certain battery voltage. This function makes sure that the
+ * for below a certain battery voltage. This function makes sure that
* the allowed current limit isn't exceeded.
*/
static void ab8500_charger_check_vbat_work(struct work_struct *work)
@@ -3719,7 +3719,14 @@ static int __init ab8500_charger_init(void)
if (ret)
return ret;
- return platform_driver_register(&ab8500_charger_driver);
+ ret = platform_driver_register(&ab8500_charger_driver);
+ if (ret) {
+ platform_unregister_drivers(ab8500_charger_component_drivers,
+ ARRAY_SIZE(ab8500_charger_component_drivers));
+ return ret;
+ }
+
+ return 0;
}
static void __exit ab8500_charger_exit(void)
diff --git a/drivers/power/supply/adp5061.c b/drivers/power/supply/adp5061.c
index fcf8ff0..840db629 100644
--- a/drivers/power/supply/adp5061.c
+++ b/drivers/power/supply/adp5061.c
@@ -694,8 +694,7 @@ static const struct power_supply_desc adp5061_desc = {
.num_properties = ARRAY_SIZE(adp5061_props),
};
-static int adp5061_probe(struct i2c_client *client,
- const struct i2c_device_id *id)
+static int adp5061_probe(struct i2c_client *client)
{
struct power_supply_config psy_cfg = {};
struct adp5061_state *st;
@@ -737,7 +736,7 @@ static struct i2c_driver adp5061_driver = {
.driver = {
.name = KBUILD_MODNAME,
},
- .probe = adp5061_probe,
+ .probe_new = adp5061_probe,
.id_table = adp5061_id,
};
module_i2c_driver(adp5061_driver);
diff --git a/drivers/power/supply/bd99954-charger.c b/drivers/power/supply/bd99954-charger.c
index 96e93e1..250362e 100644
--- a/drivers/power/supply/bd99954-charger.c
+++ b/drivers/power/supply/bd99954-charger.c
@@ -768,27 +768,13 @@ static const struct power_supply_desc bd9995x_power_supply_desc = {
* Describe the setting in linear_range table.
*/
static const struct linear_range input_current_limit_ranges[] = {
- {
- .min = 0,
- .step = 32000,
- .min_sel = 0x0,
- .max_sel = 0x1ff,
- },
+ LINEAR_RANGE(0, 0x0, 0x1ff, 32000),
};
/* Possible trickle, pre-charging and termination current values */
static const struct linear_range charging_current_ranges[] = {
- {
- .min = 0,
- .step = 64000,
- .min_sel = 0x0,
- .max_sel = 0x10,
- }, {
- .min = 1024000,
- .step = 0,
- .min_sel = 0x11,
- .max_sel = 0x1f,
- },
+ LINEAR_RANGE(0, 0x0, 0x10, 64000),
+ LINEAR_RANGE(1024000, 0x11, 0x1f, 0),
};
/*
@@ -796,72 +782,28 @@ static const struct linear_range charging_current_ranges[] = {
* and battery over voltage protection have same possible values
*/
static const struct linear_range charge_voltage_regulation_ranges[] = {
- {
- .min = 2560000,
- .step = 0,
- .min_sel = 0,
- .max_sel = 0xA0,
- }, {
- .min = 2560000,
- .step = 16000,
- .min_sel = 0xA0,
- .max_sel = 0x4B0,
- }, {
- .min = 19200000,
- .step = 0,
- .min_sel = 0x4B0,
- .max_sel = 0x7FF,
- },
+ LINEAR_RANGE(2560000, 0, 0xA0, 0),
+ LINEAR_RANGE(2560000, 0xA0, 0x4B0, 16000),
+ LINEAR_RANGE(19200000, 0x4B0, 0x7FF, 0),
};
/* Possible VSYS voltage regulation values */
static const struct linear_range vsys_voltage_regulation_ranges[] = {
- {
- .min = 2560000,
- .step = 0,
- .min_sel = 0,
- .max_sel = 0x28,
- }, {
- .min = 2560000,
- .step = 64000,
- .min_sel = 0x28,
- .max_sel = 0x12C,
- }, {
- .min = 19200000,
- .step = 0,
- .min_sel = 0x12C,
- .max_sel = 0x1FF,
- },
+ LINEAR_RANGE(2560000, 0, 0x28, 0),
+ LINEAR_RANGE(2560000, 0x28, 0x12C, 64000),
+ LINEAR_RANGE(19200000, 0x12C, 0x1FF, 0),
};
/* Possible settings for switching from trickle to pre-charging limits */
static const struct linear_range trickle_to_pre_threshold_ranges[] = {
- {
- .min = 2048000,
- .step = 0,
- .min_sel = 0,
- .max_sel = 0x20,
- }, {
- .min = 2048000,
- .step = 64000,
- .min_sel = 0x20,
- .max_sel = 0x12C,
- }, {
- .min = 19200000,
- .step = 0,
- .min_sel = 0x12C,
- .max_sel = 0x1FF
- }
+ LINEAR_RANGE(2048000, 0, 0x20, 0),
+ LINEAR_RANGE(2048000, 0x20, 0x12C, 64000),
+ LINEAR_RANGE(19200000, 0x12C, 0x1FF, 0),
};
/* Possible current values for fast-charging constant current phase */
static const struct linear_range fast_charge_current_ranges[] = {
- {
- .min = 0,
- .step = 64000,
- .min_sel = 0,
- .max_sel = 0xFF,
- }
+ LINEAR_RANGE(0, 0, 0xFF, 64000),
};
struct battery_init {
diff --git a/drivers/power/supply/bq2415x_charger.c b/drivers/power/supply/bq2415x_charger.c
index 6b99e1c..d2cb743 100644
--- a/drivers/power/supply/bq2415x_charger.c
+++ b/drivers/power/supply/bq2415x_charger.c
@@ -1520,9 +1520,9 @@ static int bq2415x_power_supply_init(struct bq2415x_device *bq)
}
/* main bq2415x probe function */
-static int bq2415x_probe(struct i2c_client *client,
- const struct i2c_device_id *id)
+static int bq2415x_probe(struct i2c_client *client)
{
+ const struct i2c_device_id *id = i2c_client_get_device_id(client);
int ret;
int num;
char *name = NULL;
@@ -1780,7 +1780,7 @@ static struct i2c_driver bq2415x_driver = {
.of_match_table = of_match_ptr(bq2415x_of_match_table),
.acpi_match_table = ACPI_PTR(bq2415x_i2c_acpi_match),
},
- .probe = bq2415x_probe,
+ .probe_new = bq2415x_probe,
.remove = bq2415x_remove,
.id_table = bq2415x_i2c_id_table,
};
diff --git a/drivers/power/supply/bq24190_charger.c b/drivers/power/supply/bq24190_charger.c
index 2274679..2b2c3a4 100644
--- a/drivers/power/supply/bq24190_charger.c
+++ b/drivers/power/supply/bq24190_charger.c
@@ -1767,9 +1767,9 @@ static int bq24190_get_config(struct bq24190_dev_info *bdi)
return 0;
}
-static int bq24190_probe(struct i2c_client *client,
- const struct i2c_device_id *id)
+static int bq24190_probe(struct i2c_client *client)
{
+ const struct i2c_device_id *id = i2c_client_get_device_id(client);
struct i2c_adapter *adapter = client->adapter;
struct device *dev = &client->dev;
struct power_supply_config charger_cfg = {}, battery_cfg = {};
@@ -2032,7 +2032,7 @@ static const struct of_device_id bq24190_of_match[] = {
MODULE_DEVICE_TABLE(of, bq24190_of_match);
static struct i2c_driver bq24190_driver = {
- .probe = bq24190_probe,
+ .probe_new = bq24190_probe,
.remove = bq24190_remove,
.shutdown = bq24190_shutdown,
.id_table = bq24190_i2c_ids,
diff --git a/drivers/power/supply/bq24257_charger.c b/drivers/power/supply/bq24257_charger.c
index a309bbe..ab4c497 100644
--- a/drivers/power/supply/bq24257_charger.c
+++ b/drivers/power/supply/bq24257_charger.c
@@ -947,9 +947,9 @@ static int bq24257_fw_probe(struct bq24257_device *bq)
return 0;
}
-static int bq24257_probe(struct i2c_client *client,
- const struct i2c_device_id *id)
+static int bq24257_probe(struct i2c_client *client)
{
+ const struct i2c_device_id *id = i2c_client_get_device_id(client);
struct i2c_adapter *adapter = client->adapter;
struct device *dev = &client->dev;
const struct acpi_device_id *acpi_id;
@@ -1167,7 +1167,7 @@ static struct i2c_driver bq24257_driver = {
.acpi_match_table = ACPI_PTR(bq24257_acpi_match),
.pm = &bq24257_pm,
},
- .probe = bq24257_probe,
+ .probe_new = bq24257_probe,
.remove = bq24257_remove,
.id_table = bq24257_i2c_ids,
};
diff --git a/drivers/power/supply/bq24735-charger.c b/drivers/power/supply/bq24735-charger.c
index 3ce36d0..cfca3a8 100644
--- a/drivers/power/supply/bq24735-charger.c
+++ b/drivers/power/supply/bq24735-charger.c
@@ -352,8 +352,7 @@ static struct bq24735_platform *bq24735_parse_dt_data(struct i2c_client *client)
return pdata;
}
-static int bq24735_charger_probe(struct i2c_client *client,
- const struct i2c_device_id *id)
+static int bq24735_charger_probe(struct i2c_client *client)
{
int ret;
struct bq24735 *charger;
@@ -506,7 +505,7 @@ static struct i2c_driver bq24735_charger_driver = {
.name = "bq24735-charger",
.of_match_table = bq24735_match_ids,
},
- .probe = bq24735_charger_probe,
+ .probe_new = bq24735_charger_probe,
.id_table = bq24735_charger_id,
};
diff --git a/drivers/power/supply/bq2515x_charger.c b/drivers/power/supply/bq2515x_charger.c
index 4f76ad9..da224ae 100644
--- a/drivers/power/supply/bq2515x_charger.c
+++ b/drivers/power/supply/bq2515x_charger.c
@@ -1078,9 +1078,9 @@ static const struct regmap_config bq25155_regmap_config = {
.volatile_reg = bq2515x_volatile_register,
};
-static int bq2515x_probe(struct i2c_client *client,
- const struct i2c_device_id *id)
+static int bq2515x_probe(struct i2c_client *client)
{
+ const struct i2c_device_id *id = i2c_client_get_device_id(client);
struct device *dev = &client->dev;
struct bq2515x_device *bq2515x;
struct power_supply_config charger_cfg = {};
@@ -1158,7 +1158,7 @@ static struct i2c_driver bq2515x_driver = {
.name = "bq2515x-charger",
.of_match_table = bq2515x_of_match,
},
- .probe = bq2515x_probe,
+ .probe_new = bq2515x_probe,
.id_table = bq2515x_i2c_ids,
};
module_i2c_driver(bq2515x_driver);
diff --git a/drivers/power/supply/bq256xx_charger.c b/drivers/power/supply/bq256xx_charger.c
index 01ad84f..db13e28 100644
--- a/drivers/power/supply/bq256xx_charger.c
+++ b/drivers/power/supply/bq256xx_charger.c
@@ -1619,9 +1619,9 @@ static int bq256xx_parse_dt(struct bq256xx_device *bq,
return 0;
}
-static int bq256xx_probe(struct i2c_client *client,
- const struct i2c_device_id *id)
+static int bq256xx_probe(struct i2c_client *client)
{
+ const struct i2c_device_id *id = i2c_client_get_device_id(client);
struct device *dev = &client->dev;
struct bq256xx_device *bq;
struct power_supply_config psy_cfg = { };
@@ -1744,7 +1744,7 @@ static struct i2c_driver bq256xx_driver = {
.of_match_table = bq256xx_of_match,
.acpi_match_table = bq256xx_acpi_match,
},
- .probe = bq256xx_probe,
+ .probe_new = bq256xx_probe,
.id_table = bq256xx_i2c_ids,
};
module_i2c_driver(bq256xx_driver);
diff --git a/drivers/power/supply/bq25890_charger.c b/drivers/power/supply/bq25890_charger.c
index 6020b58..2d731ea 100644
--- a/drivers/power/supply/bq25890_charger.c
+++ b/drivers/power/supply/bq25890_charger.c
@@ -529,40 +529,6 @@ static int bq25890_power_supply_get_property(struct power_supply *psy,
val->intval = POWER_SUPPLY_HEALTH_UNSPEC_FAILURE;
break;
- case POWER_SUPPLY_PROP_CONSTANT_CHARGE_CURRENT_MAX:
- val->intval = bq25890_find_val(bq->init_data.ichg, TBL_ICHG);
-
- /* When temperature is too low, charge current is decreased */
- if (bq->state.ntc_fault == NTC_FAULT_COOL) {
- ret = bq25890_field_read(bq, F_JEITA_ISET);
- if (ret < 0)
- return ret;
-
- if (ret)
- val->intval /= 5;
- else
- val->intval /= 2;
- }
- break;
-
- case POWER_SUPPLY_PROP_CONSTANT_CHARGE_VOLTAGE:
- if (!state.online) {
- val->intval = 0;
- break;
- }
-
- ret = bq25890_field_read(bq, F_BATV); /* read measured value */
- if (ret < 0)
- return ret;
-
- /* converted_val = 2.304V + ADC_val * 20mV (table 10.3.15) */
- val->intval = 2304000 + ret * 20000;
- break;
-
- case POWER_SUPPLY_PROP_CONSTANT_CHARGE_VOLTAGE_MAX:
- val->intval = bq25890_find_val(bq->init_data.vreg, TBL_VREG);
- break;
-
case POWER_SUPPLY_PROP_PRECHARGE_CURRENT:
val->intval = bq25890_find_val(bq->init_data.iprechg, TBL_ITERM);
break;
@@ -579,8 +545,71 @@ static int bq25890_power_supply_get_property(struct power_supply *psy,
val->intval = bq25890_find_val(ret, TBL_IINLIM);
break;
- case POWER_SUPPLY_PROP_VOLTAGE_NOW:
- ret = bq25890_field_read(bq, F_SYSV); /* read measured value */
+ case POWER_SUPPLY_PROP_CURRENT_NOW: /* I_BAT now */
+ /*
+ * This is ADC-sampled immediate charge current supplied
+ * from charger to battery. The property name is confusing,
+ * for clarification refer to:
+ * Documentation/ABI/testing/sysfs-class-power
+ * /sys/class/power_supply/<supply_name>/current_now
+ */
+ ret = bq25890_field_read(bq, F_ICHGR); /* read measured value */
+ if (ret < 0)
+ return ret;
+
+ /* converted_val = ADC_val * 50mA (table 10.3.19) */
+ val->intval = ret * -50000;
+ break;
+
+ case POWER_SUPPLY_PROP_CONSTANT_CHARGE_CURRENT: /* I_BAT user limit */
+ /*
+ * This is user-configured constant charge current supplied
+ * from charger to battery in first phase of charging, when
+ * battery voltage is below constant charge voltage.
+ *
+ * This value reflects the current hardware setting.
+ *
+ * The POWER_SUPPLY_PROP_CONSTANT_CHARGE_CURRENT_MAX is the
+ * maximum value of this property.
+ */
+ ret = bq25890_field_read(bq, F_ICHG);
+ if (ret < 0)
+ return ret;
+ val->intval = bq25890_find_val(ret, TBL_ICHG);
+
+ /* When temperature is too low, charge current is decreased */
+ if (bq->state.ntc_fault == NTC_FAULT_COOL) {
+ ret = bq25890_field_read(bq, F_JEITA_ISET);
+ if (ret < 0)
+ return ret;
+
+ if (ret)
+ val->intval /= 5;
+ else
+ val->intval /= 2;
+ }
+ break;
+
+ case POWER_SUPPLY_PROP_CONSTANT_CHARGE_CURRENT_MAX: /* I_BAT max */
+ /*
+ * This is maximum allowed constant charge current supplied
+ * from charger to battery in first phase of charging, when
+ * battery voltage is below constant charge voltage.
+ *
+ * This value is constant for each battery and set from DT.
+ */
+ val->intval = bq25890_find_val(bq->init_data.ichg, TBL_ICHG);
+ break;
+
+ case POWER_SUPPLY_PROP_VOLTAGE_NOW: /* V_BAT now */
+ /*
+ * This is ADC-sampled immediate charge voltage supplied
+ * from charger to battery. The property name is confusing,
+ * for clarification refer to:
+ * Documentation/ABI/testing/sysfs-class-power
+ * /sys/class/power_supply/<supply_name>/voltage_now
+ */
+ ret = bq25890_field_read(bq, F_BATV); /* read measured value */
if (ret < 0)
return ret;
@@ -588,13 +617,33 @@ static int bq25890_power_supply_get_property(struct power_supply *psy,
val->intval = 2304000 + ret * 20000;
break;
- case POWER_SUPPLY_PROP_CURRENT_NOW:
- ret = bq25890_field_read(bq, F_ICHGR); /* read measured value */
+ case POWER_SUPPLY_PROP_CONSTANT_CHARGE_VOLTAGE: /* V_BAT user limit */
+ /*
+ * This is user-configured constant charge voltage supplied
+ * from charger to battery in second phase of charging, when
+ * battery voltage reached constant charge voltage.
+ *
+ * This value reflects the current hardware setting.
+ *
+ * The POWER_SUPPLY_PROP_CONSTANT_CHARGE_VOLTAGE_MAX is the
+ * maximum value of this property.
+ */
+ ret = bq25890_field_read(bq, F_VREG);
if (ret < 0)
return ret;
- /* converted_val = ADC_val * 50mA (table 10.3.19) */
- val->intval = ret * -50000;
+ val->intval = bq25890_find_val(ret, TBL_VREG);
+ break;
+
+ case POWER_SUPPLY_PROP_CONSTANT_CHARGE_VOLTAGE_MAX: /* V_BAT max */
+ /*
+ * This is maximum allowed constant charge voltage supplied
+ * from charger to battery in second phase of charging, when
+ * battery voltage reached constant charge voltage.
+ *
+ * This value is constant for each battery and set from DT.
+ */
+ val->intval = bq25890_find_val(bq->init_data.vreg, TBL_VREG);
break;
case POWER_SUPPLY_PROP_TEMP:
@@ -618,9 +667,18 @@ static int bq25890_power_supply_set_property(struct power_supply *psy,
const union power_supply_propval *val)
{
struct bq25890_device *bq = power_supply_get_drvdata(psy);
+ int maxval;
u8 lval;
switch (psp) {
+ case POWER_SUPPLY_PROP_CONSTANT_CHARGE_CURRENT:
+ maxval = bq25890_find_val(bq->init_data.ichg, TBL_ICHG);
+ lval = bq25890_find_idx(min(val->intval, maxval), TBL_ICHG);
+ return bq25890_field_write(bq, F_ICHG, lval);
+ case POWER_SUPPLY_PROP_CONSTANT_CHARGE_VOLTAGE:
+ maxval = bq25890_find_val(bq->init_data.vreg, TBL_VREG);
+ lval = bq25890_find_idx(min(val->intval, maxval), TBL_VREG);
+ return bq25890_field_write(bq, F_VREG, lval);
case POWER_SUPPLY_PROP_INPUT_CURRENT_LIMIT:
lval = bq25890_find_idx(val->intval, TBL_IINLIM);
return bq25890_field_write(bq, F_IINLIM, lval);
@@ -633,6 +691,8 @@ static int bq25890_power_supply_property_is_writeable(struct power_supply *psy,
enum power_supply_property psp)
{
switch (psp) {
+ case POWER_SUPPLY_PROP_CONSTANT_CHARGE_CURRENT:
+ case POWER_SUPPLY_PROP_CONSTANT_CHARGE_VOLTAGE:
case POWER_SUPPLY_PROP_INPUT_CURRENT_LIMIT:
return true;
default:
@@ -880,6 +940,7 @@ static const enum power_supply_property bq25890_power_supply_props[] = {
POWER_SUPPLY_PROP_CHARGE_TYPE,
POWER_SUPPLY_PROP_ONLINE,
POWER_SUPPLY_PROP_HEALTH,
+ POWER_SUPPLY_PROP_CONSTANT_CHARGE_CURRENT,
POWER_SUPPLY_PROP_CONSTANT_CHARGE_CURRENT_MAX,
POWER_SUPPLY_PROP_CONSTANT_CHARGE_VOLTAGE,
POWER_SUPPLY_PROP_CONSTANT_CHARGE_VOLTAGE_MAX,
@@ -1034,10 +1095,32 @@ static int bq25890_vbus_is_enabled(struct regulator_dev *rdev)
return bq25890_field_read(bq, F_OTG_CFG);
}
+static int bq25890_vbus_get_voltage(struct regulator_dev *rdev)
+{
+ struct bq25890_device *bq = rdev_get_drvdata(rdev);
+
+ return bq25890_get_vbus_voltage(bq);
+}
+
+static int bq25890_vsys_get_voltage(struct regulator_dev *rdev)
+{
+ struct bq25890_device *bq = rdev_get_drvdata(rdev);
+ int ret;
+
+ /* Should be some output voltage ? */
+ ret = bq25890_field_read(bq, F_SYSV); /* read measured value */
+ if (ret < 0)
+ return ret;
+
+ /* converted_val = 2.304V + ADC_val * 20mV (table 10.3.15) */
+ return 2304000 + ret * 20000;
+}
+
static const struct regulator_ops bq25890_vbus_ops = {
.enable = bq25890_vbus_enable,
.disable = bq25890_vbus_disable,
.is_enabled = bq25890_vbus_is_enabled,
+ .get_voltage = bq25890_vbus_get_voltage,
};
static const struct regulator_desc bq25890_vbus_desc = {
@@ -1046,9 +1129,54 @@ static const struct regulator_desc bq25890_vbus_desc = {
.type = REGULATOR_VOLTAGE,
.owner = THIS_MODULE,
.ops = &bq25890_vbus_ops,
- .fixed_uV = 5000000,
- .n_voltages = 1,
};
+
+static const struct regulator_ops bq25890_vsys_ops = {
+ .get_voltage = bq25890_vsys_get_voltage,
+};
+
+static const struct regulator_desc bq25890_vsys_desc = {
+ .name = "vsys",
+ .of_match = "vsys",
+ .type = REGULATOR_VOLTAGE,
+ .owner = THIS_MODULE,
+ .ops = &bq25890_vsys_ops,
+};
+
+static int bq25890_register_regulator(struct bq25890_device *bq)
+{
+ struct bq25890_platform_data *pdata = dev_get_platdata(bq->dev);
+ struct regulator_config cfg = {
+ .dev = bq->dev,
+ .driver_data = bq,
+ };
+ struct regulator_dev *reg;
+
+ if (pdata)
+ cfg.init_data = pdata->regulator_init_data;
+
+ reg = devm_regulator_register(bq->dev, &bq25890_vbus_desc, &cfg);
+ if (IS_ERR(reg)) {
+ return dev_err_probe(bq->dev, PTR_ERR(reg),
+ "registering vbus regulator");
+ }
+
+ /* pdata->regulator_init_data is for vbus only */
+ cfg.init_data = NULL;
+ reg = devm_regulator_register(bq->dev, &bq25890_vsys_desc, &cfg);
+ if (IS_ERR(reg)) {
+ return dev_err_probe(bq->dev, PTR_ERR(reg),
+ "registering vsys regulator");
+ }
+
+ return 0;
+}
+#else
+static inline int
+bq25890_register_regulator(struct bq25890_device *bq)
+{
+ return 0;
+}
#endif
static int bq25890_get_chip_version(struct bq25890_device *bq)
@@ -1189,8 +1317,14 @@ static int bq25890_fw_probe(struct bq25890_device *bq)
return 0;
}
-static int bq25890_probe(struct i2c_client *client,
- const struct i2c_device_id *id)
+static void bq25890_non_devm_cleanup(void *data)
+{
+ struct bq25890_device *bq = data;
+
+ cancel_delayed_work_sync(&bq->pump_express_work);
+}
+
+static int bq25890_probe(struct i2c_client *client)
{
struct device *dev = &client->dev;
struct bq25890_device *bq;
@@ -1244,56 +1378,47 @@ static int bq25890_probe(struct i2c_client *client,
/* OTG reporting */
bq->usb_phy = devm_usb_get_phy(dev, USB_PHY_TYPE_USB2);
- if (!IS_ERR_OR_NULL(bq->usb_phy)) {
- INIT_WORK(&bq->usb_work, bq25890_usb_work);
- bq->usb_nb.notifier_call = bq25890_usb_notifier;
- usb_register_notifier(bq->usb_phy, &bq->usb_nb);
- }
-#ifdef CONFIG_REGULATOR
- else {
- struct bq25890_platform_data *pdata = dev_get_platdata(dev);
- struct regulator_config cfg = { };
- struct regulator_dev *reg;
- cfg.dev = dev;
- cfg.driver_data = bq;
- if (pdata)
- cfg.init_data = pdata->regulator_init_data;
+ /*
+ * This must be before bq25890_power_supply_init(), so that it runs
+ * after devm unregisters the power_supply.
+ */
+ ret = devm_add_action_or_reset(dev, bq25890_non_devm_cleanup, bq);
+ if (ret)
+ return ret;
- reg = devm_regulator_register(dev, &bq25890_vbus_desc, &cfg);
- if (IS_ERR(reg))
- return dev_err_probe(dev, PTR_ERR(reg), "registering regulator");
- }
-#endif
+ ret = bq25890_register_regulator(bq);
+ if (ret)
+ return ret;
ret = bq25890_power_supply_init(bq);
- if (ret < 0) {
- dev_err(dev, "Failed to register power supply\n");
- goto err_unregister_usb_notifier;
- }
+ if (ret < 0)
+ return dev_err_probe(dev, ret, "registering power supply\n");
ret = devm_request_threaded_irq(dev, client->irq, NULL,
bq25890_irq_handler_thread,
IRQF_TRIGGER_FALLING | IRQF_ONESHOT,
BQ25890_IRQ_PIN, bq);
if (ret)
- goto err_unregister_usb_notifier;
+ return ret;
+
+ if (!IS_ERR_OR_NULL(bq->usb_phy)) {
+ INIT_WORK(&bq->usb_work, bq25890_usb_work);
+ bq->usb_nb.notifier_call = bq25890_usb_notifier;
+ usb_register_notifier(bq->usb_phy, &bq->usb_nb);
+ }
return 0;
-
-err_unregister_usb_notifier:
- if (!IS_ERR_OR_NULL(bq->usb_phy))
- usb_unregister_notifier(bq->usb_phy, &bq->usb_nb);
-
- return ret;
}
static void bq25890_remove(struct i2c_client *client)
{
struct bq25890_device *bq = i2c_get_clientdata(client);
- if (!IS_ERR_OR_NULL(bq->usb_phy))
+ if (!IS_ERR_OR_NULL(bq->usb_phy)) {
usb_unregister_notifier(bq->usb_phy, &bq->usb_nb);
+ cancel_work_sync(&bq->usb_work);
+ }
if (!bq->skip_reset) {
/* reset all registers to default values */
@@ -1400,7 +1525,7 @@ static struct i2c_driver bq25890_driver = {
.acpi_match_table = ACPI_PTR(bq25890_acpi_match),
.pm = &bq25890_pm,
},
- .probe = bq25890_probe,
+ .probe_new = bq25890_probe,
.remove = bq25890_remove,
.shutdown = bq25890_shutdown,
.id_table = bq25890_i2c_ids,
diff --git a/drivers/power/supply/bq25980_charger.c b/drivers/power/supply/bq25980_charger.c
index 9339f56..a59d976 100644
--- a/drivers/power/supply/bq25980_charger.c
+++ b/drivers/power/supply/bq25980_charger.c
@@ -1207,9 +1207,9 @@ static int bq25980_parse_dt(struct bq25980_device *bq)
return 0;
}
-static int bq25980_probe(struct i2c_client *client,
- const struct i2c_device_id *id)
+static int bq25980_probe(struct i2c_client *client)
{
+ const struct i2c_device_id *id = i2c_client_get_device_id(client);
struct device *dev = &client->dev;
struct bq25980_device *bq;
int ret;
@@ -1287,7 +1287,7 @@ static struct i2c_driver bq25980_driver = {
.name = "bq25980-charger",
.of_match_table = bq25980_of_match,
},
- .probe = bq25980_probe,
+ .probe_new = bq25980_probe,
.id_table = bq25980_i2c_ids,
};
module_i2c_driver(bq25980_driver);
diff --git a/drivers/power/supply/bq27xxx_battery_i2c.c b/drivers/power/supply/bq27xxx_battery_i2c.c
index 94b00bb..f876899 100644
--- a/drivers/power/supply/bq27xxx_battery_i2c.c
+++ b/drivers/power/supply/bq27xxx_battery_i2c.c
@@ -136,9 +136,9 @@ static int bq27xxx_battery_i2c_bulk_write(struct bq27xxx_device_info *di,
return 0;
}
-static int bq27xxx_battery_i2c_probe(struct i2c_client *client,
- const struct i2c_device_id *id)
+static int bq27xxx_battery_i2c_probe(struct i2c_client *client)
{
+ const struct i2c_device_id *id = i2c_client_get_device_id(client);
struct bq27xxx_device_info *di;
int ret;
char *name;
@@ -295,7 +295,7 @@ static struct i2c_driver bq27xxx_battery_i2c_driver = {
.name = "bq27xxx-battery",
.of_match_table = of_match_ptr(bq27xxx_battery_i2c_of_match_table),
},
- .probe = bq27xxx_battery_i2c_probe,
+ .probe_new = bq27xxx_battery_i2c_probe,
.remove = bq27xxx_battery_i2c_remove,
.id_table = bq27xxx_i2c_id_table,
};
diff --git a/drivers/power/supply/cw2015_battery.c b/drivers/power/supply/cw2015_battery.c
index 6d52641..473522b 100644
--- a/drivers/power/supply/cw2015_battery.c
+++ b/drivers/power/supply/cw2015_battery.c
@@ -699,6 +699,9 @@ static int cw_bat_probe(struct i2c_client *client)
}
cw_bat->battery_workqueue = create_singlethread_workqueue("rk_battery");
+ if (!cw_bat->battery_workqueue)
+ return -ENOMEM;
+
devm_delayed_work_autocancel(&client->dev,
&cw_bat->battery_delay_work, cw_bat_work);
queue_delayed_work(cw_bat->battery_workqueue,
diff --git a/drivers/power/supply/ds2782_battery.c b/drivers/power/supply/ds2782_battery.c
index d78cd05..9b96192 100644
--- a/drivers/power/supply/ds2782_battery.c
+++ b/drivers/power/supply/ds2782_battery.c
@@ -368,9 +368,9 @@ static const struct ds278x_battery_ops ds278x_ops[] = {
}
};
-static int ds278x_battery_probe(struct i2c_client *client,
- const struct i2c_device_id *id)
+static int ds278x_battery_probe(struct i2c_client *client)
{
+ const struct i2c_device_id *id = i2c_client_get_device_id(client);
struct ds278x_platform_data *pdata = client->dev.platform_data;
struct power_supply_config psy_cfg = {};
struct ds278x_info *info;
@@ -458,7 +458,7 @@ static struct i2c_driver ds278x_battery_driver = {
.name = "ds2782-battery",
.pm = &ds278x_battery_pm_ops,
},
- .probe = ds278x_battery_probe,
+ .probe_new = ds278x_battery_probe,
.remove = ds278x_battery_remove,
.id_table = ds278x_id,
};
diff --git a/drivers/power/supply/lp8727_charger.c b/drivers/power/supply/lp8727_charger.c
index 384a374..e6c2137 100644
--- a/drivers/power/supply/lp8727_charger.c
+++ b/drivers/power/supply/lp8727_charger.c
@@ -540,7 +540,7 @@ static struct lp8727_platform_data *lp8727_parse_dt(struct device *dev)
}
#endif
-static int lp8727_probe(struct i2c_client *cl, const struct i2c_device_id *id)
+static int lp8727_probe(struct i2c_client *cl)
{
struct lp8727_chg *pchg;
struct lp8727_platform_data *pdata;
@@ -615,7 +615,7 @@ static struct i2c_driver lp8727_driver = {
.name = "lp8727",
.of_match_table = of_match_ptr(lp8727_dt_ids),
},
- .probe = lp8727_probe,
+ .probe_new = lp8727_probe,
.remove = lp8727_remove,
.id_table = lp8727_ids,
};
diff --git a/drivers/power/supply/lp8788-charger.c b/drivers/power/supply/lp8788-charger.c
index 56c5752..f5f47a0 100644
--- a/drivers/power/supply/lp8788-charger.c
+++ b/drivers/power/supply/lp8788-charger.c
@@ -520,7 +520,7 @@ static int lp8788_set_irqs(struct platform_device *pdev,
static int lp8788_irq_register(struct platform_device *pdev,
struct lp8788_charger *pchg)
{
- const char *name[] = {
+ static const char * const name[] = {
LP8788_CHG_IRQ, LP8788_PRSW_IRQ, LP8788_BATT_IRQ
};
int i;
diff --git a/drivers/power/supply/ltc2941-battery-gauge.c b/drivers/power/supply/ltc2941-battery-gauge.c
index 6573052..d3fb428 100644
--- a/drivers/power/supply/ltc2941-battery-gauge.c
+++ b/drivers/power/supply/ltc2941-battery-gauge.c
@@ -439,8 +439,7 @@ static enum power_supply_property ltc294x_properties[] = {
POWER_SUPPLY_PROP_CURRENT_NOW,
};
-static int ltc294x_i2c_probe(struct i2c_client *client,
- const struct i2c_device_id *id)
+static int ltc294x_i2c_probe(struct i2c_client *client)
{
struct power_supply_config psy_cfg = {};
struct ltc294x_info *info;
@@ -636,7 +635,7 @@ static struct i2c_driver ltc294x_driver = {
.of_match_table = ltc294x_i2c_of_match,
.pm = LTC294X_PM_OPS,
},
- .probe = ltc294x_i2c_probe,
+ .probe_new = ltc294x_i2c_probe,
.shutdown = ltc294x_i2c_shutdown,
.id_table = ltc294x_i2c_id,
};
diff --git a/drivers/power/supply/ltc4162-l-charger.c b/drivers/power/supply/ltc4162-l-charger.c
index 1a5cb44..db2bb52 100644
--- a/drivers/power/supply/ltc4162-l-charger.c
+++ b/drivers/power/supply/ltc4162-l-charger.c
@@ -819,8 +819,7 @@ static void ltc4162l_clear_interrupts(struct ltc4162l_info *info)
regmap_write(info->regmap, LTC4162L_CHARGE_STATUS_ALERTS_REG, 0);
}
-static int ltc4162l_probe(struct i2c_client *client,
- const struct i2c_device_id *id)
+static int ltc4162l_probe(struct i2c_client *client)
{
struct i2c_adapter *adapter = client->adapter;
struct device *dev = &client->dev;
@@ -916,7 +915,7 @@ static const struct of_device_id ltc4162l_of_match[] = {
MODULE_DEVICE_TABLE(of, ltc4162l_of_match);
static struct i2c_driver ltc4162l_driver = {
- .probe = ltc4162l_probe,
+ .probe_new = ltc4162l_probe,
.alert = ltc4162l_alert,
.id_table = ltc4162l_i2c_id_table,
.driver = {
diff --git a/drivers/power/supply/max14656_charger_detector.c b/drivers/power/supply/max14656_charger_detector.c
index fc36828..0d0180f 100644
--- a/drivers/power/supply/max14656_charger_detector.c
+++ b/drivers/power/supply/max14656_charger_detector.c
@@ -234,8 +234,7 @@ static enum power_supply_property max14656_battery_props[] = {
POWER_SUPPLY_PROP_MANUFACTURER,
};
-static int max14656_probe(struct i2c_client *client,
- const struct i2c_device_id *id)
+static int max14656_probe(struct i2c_client *client)
{
struct i2c_adapter *adapter = client->adapter;
struct device *dev = &client->dev;
@@ -317,7 +316,7 @@ static struct i2c_driver max14656_i2c_driver = {
.name = "max14656",
.of_match_table = max14656_match_table,
},
- .probe = max14656_probe,
+ .probe_new = max14656_probe,
.id_table = max14656_id,
};
module_i2c_driver(max14656_i2c_driver);
diff --git a/drivers/power/supply/max17040_battery.c b/drivers/power/supply/max17040_battery.c
index a9aef1e..d107595 100644
--- a/drivers/power/supply/max17040_battery.c
+++ b/drivers/power/supply/max17040_battery.c
@@ -430,9 +430,9 @@ static const struct power_supply_desc max17040_battery_desc = {
.num_properties = ARRAY_SIZE(max17040_battery_props),
};
-static int max17040_probe(struct i2c_client *client,
- const struct i2c_device_id *id)
+static int max17040_probe(struct i2c_client *client)
{
+ const struct i2c_device_id *id = i2c_client_get_device_id(client);
struct i2c_adapter *adapter = client->adapter;
struct power_supply_config psy_cfg = {};
struct max17040_chip *chip;
@@ -599,7 +599,7 @@ static struct i2c_driver max17040_i2c_driver = {
.of_match_table = max17040_of_match,
.pm = MAX17040_PM_OPS,
},
- .probe = max17040_probe,
+ .probe_new = max17040_probe,
.id_table = max17040_id,
};
module_i2c_driver(max17040_i2c_driver);
diff --git a/drivers/power/supply/max17042_battery.c b/drivers/power/supply/max17042_battery.c
index ab031bb..89cabe8 100644
--- a/drivers/power/supply/max17042_battery.c
+++ b/drivers/power/supply/max17042_battery.c
@@ -1031,9 +1031,9 @@ static const struct power_supply_desc max17042_no_current_sense_psy_desc = {
.num_properties = ARRAY_SIZE(max17042_battery_props) - 2,
};
-static int max17042_probe(struct i2c_client *client,
- const struct i2c_device_id *id)
+static int max17042_probe(struct i2c_client *client)
{
+ const struct i2c_device_id *id = i2c_client_get_device_id(client);
struct i2c_adapter *adapter = client->adapter;
const struct power_supply_desc *max17042_desc = &max17042_psy_desc;
struct power_supply_config psy_cfg = {};
@@ -1220,7 +1220,7 @@ static struct i2c_driver max17042_i2c_driver = {
.of_match_table = of_match_ptr(max17042_dt_match),
.pm = &max17042_pm_ops,
},
- .probe = max17042_probe,
+ .probe_new = max17042_probe,
.id_table = max17042_id,
};
module_i2c_driver(max17042_i2c_driver);
diff --git a/drivers/power/supply/mt6360_charger.c b/drivers/power/supply/mt6360_charger.c
index 3abaa72e..92e48e3 100644
--- a/drivers/power/supply/mt6360_charger.c
+++ b/drivers/power/supply/mt6360_charger.c
@@ -113,16 +113,13 @@ enum {
MT6360_RANGE_MAX,
};
-#define MT6360_LINEAR_RANGE(idx, _min, _min_sel, _max_sel, _step) \
- [idx] = REGULATOR_LINEAR_RANGE(_min, _min_sel, _max_sel, _step)
-
static const struct linear_range mt6360_chg_range[MT6360_RANGE_MAX] = {
- MT6360_LINEAR_RANGE(MT6360_RANGE_VMIVR, 3900000, 0, 0x5F, 100000),
- MT6360_LINEAR_RANGE(MT6360_RANGE_ICHG, 100000, 0, 0x31, 100000),
- MT6360_LINEAR_RANGE(MT6360_RANGE_VOREG, 3900000, 0, 0x51, 10000),
- MT6360_LINEAR_RANGE(MT6360_RANGE_AICR, 100000, 0, 0x3F, 50000),
- MT6360_LINEAR_RANGE(MT6360_RANGE_IPREC, 100000, 0, 0x0F, 50000),
- MT6360_LINEAR_RANGE(MT6360_RANGE_IEOC, 100000, 0, 0x0F, 50000),
+ LINEAR_RANGE_IDX(MT6360_RANGE_VMIVR, 3900000, 0, 0x5F, 100000),
+ LINEAR_RANGE_IDX(MT6360_RANGE_ICHG, 100000, 0, 0x31, 100000),
+ LINEAR_RANGE_IDX(MT6360_RANGE_VOREG, 3900000, 0, 0x51, 10000),
+ LINEAR_RANGE_IDX(MT6360_RANGE_AICR, 100000, 0, 0x3F, 50000),
+ LINEAR_RANGE_IDX(MT6360_RANGE_IPREC, 100000, 0, 0x0F, 50000),
+ LINEAR_RANGE_IDX(MT6360_RANGE_IEOC, 100000, 0, 0x0F, 50000),
};
struct mt6360_chg_info {
diff --git a/drivers/power/supply/power_supply_core.c b/drivers/power/supply/power_supply_core.c
index 4b5fb17..7c790c4 100644
--- a/drivers/power/supply/power_supply_core.c
+++ b/drivers/power/supply/power_supply_core.c
@@ -750,6 +750,11 @@ int power_supply_get_battery_info(struct power_supply *psy,
int i, tab_len, size;
propname = kasprintf(GFP_KERNEL, "ocv-capacity-table-%d", index);
+ if (!propname) {
+ power_supply_put_battery_info(psy, info);
+ err = -ENOMEM;
+ goto out_put_node;
+ }
list = of_get_property(battery_np, propname, &size);
if (!list || !size) {
dev_err(&psy->dev, "failed to get %s\n", propname);
@@ -870,7 +875,6 @@ EXPORT_SYMBOL_GPL(power_supply_temp2resist_simple);
* power_supply_vbat2ri() - find the battery internal resistance
* from the battery voltage
* @info: The battery information container
- * @table: Pointer to battery resistance temperature table
* @vbat_uv: The battery voltage in microvolt
* @charging: If we are charging (true) or not (false)
*
@@ -1387,8 +1391,8 @@ __power_supply_register(struct device *parent,
register_cooler_failed:
psy_unregister_thermal(psy);
register_thermal_failed:
- device_del(dev);
wakeup_init_failed:
+ device_del(dev);
device_add_failed:
check_supplies_failed:
dev_set_name_failed:
diff --git a/drivers/power/supply/rk817_charger.c b/drivers/power/supply/rk817_charger.c
index f20a6ac..4f9c1c4 100644
--- a/drivers/power/supply/rk817_charger.c
+++ b/drivers/power/supply/rk817_charger.c
@@ -1060,8 +1060,10 @@ static int rk817_charger_probe(struct platform_device *pdev)
return -ENODEV;
charger = devm_kzalloc(&pdev->dev, sizeof(*charger), GFP_KERNEL);
- if (!charger)
+ if (!charger) {
+ of_node_put(node);
return -ENOMEM;
+ }
charger->rk808 = rk808;
diff --git a/drivers/power/supply/rt5033_battery.c b/drivers/power/supply/rt5033_battery.c
index 736dec6..5c04cf3 100644
--- a/drivers/power/supply/rt5033_battery.c
+++ b/drivers/power/supply/rt5033_battery.c
@@ -112,8 +112,7 @@ static const struct power_supply_desc rt5033_battery_desc = {
.num_properties = ARRAY_SIZE(rt5033_battery_props),
};
-static int rt5033_battery_probe(struct i2c_client *client,
- const struct i2c_device_id *id)
+static int rt5033_battery_probe(struct i2c_client *client)
{
struct i2c_adapter *adapter = client->adapter;
struct power_supply_config psy_cfg = {};
@@ -173,7 +172,7 @@ static struct i2c_driver rt5033_battery_driver = {
.name = "rt5033-battery",
.of_match_table = rt5033_battery_of_match,
},
- .probe = rt5033_battery_probe,
+ .probe_new = rt5033_battery_probe,
.remove = rt5033_battery_remove,
.id_table = rt5033_battery_id,
};
diff --git a/drivers/power/supply/rt9455_charger.c b/drivers/power/supply/rt9455_charger.c
index 7296228..31fb652 100644
--- a/drivers/power/supply/rt9455_charger.c
+++ b/drivers/power/supply/rt9455_charger.c
@@ -1581,8 +1581,7 @@ static const struct regmap_config rt9455_regmap_config = {
.cache_type = REGCACHE_RBTREE,
};
-static int rt9455_probe(struct i2c_client *client,
- const struct i2c_device_id *id)
+static int rt9455_probe(struct i2c_client *client)
{
struct i2c_adapter *adapter = client->adapter;
struct device *dev = &client->dev;
@@ -1738,7 +1737,7 @@ MODULE_DEVICE_TABLE(acpi, rt9455_i2c_acpi_match);
#endif
static struct i2c_driver rt9455_driver = {
- .probe = rt9455_probe,
+ .probe_new = rt9455_probe,
.remove = rt9455_remove,
.id_table = rt9455_i2c_id_table,
.driver = {
diff --git a/drivers/power/supply/sbs-charger.c b/drivers/power/supply/sbs-charger.c
index b08f7d0..75ebcbf 100644
--- a/drivers/power/supply/sbs-charger.c
+++ b/drivers/power/supply/sbs-charger.c
@@ -162,8 +162,7 @@ static const struct power_supply_desc sbs_desc = {
.get_property = sbs_get_property,
};
-static int sbs_probe(struct i2c_client *client,
- const struct i2c_device_id *id)
+static int sbs_probe(struct i2c_client *client)
{
struct power_supply_config psy_cfg = {};
struct sbs_info *chip;
@@ -241,7 +240,7 @@ static const struct i2c_device_id sbs_id[] = {
MODULE_DEVICE_TABLE(i2c, sbs_id);
static struct i2c_driver sbs_driver = {
- .probe = sbs_probe,
+ .probe_new = sbs_probe,
.id_table = sbs_id,
.driver = {
.name = "sbs-charger",
diff --git a/drivers/power/supply/sbs-manager.c b/drivers/power/supply/sbs-manager.c
index 71ec8f7..bde9773 100644
--- a/drivers/power/supply/sbs-manager.c
+++ b/drivers/power/supply/sbs-manager.c
@@ -315,9 +315,9 @@ static void sbsm_del_mux_adapter(void *data)
i2c_mux_del_adapters(sbsm->muxc);
}
-static int sbsm_probe(struct i2c_client *client,
- const struct i2c_device_id *id)
+static int sbsm_probe(struct i2c_client *client)
{
+ const struct i2c_device_id *id = i2c_client_get_device_id(client);
struct i2c_adapter *adapter = client->adapter;
struct sbsm_data *data;
struct device *dev = &client->dev;
@@ -409,7 +409,7 @@ static struct i2c_driver sbsm_driver = {
.name = "sbsm",
.of_match_table = of_match_ptr(sbsm_dt_ids),
},
- .probe = sbsm_probe,
+ .probe_new = sbsm_probe,
.alert = sbsm_alert,
.id_table = sbsm_ids
};
diff --git a/drivers/power/supply/smb347-charger.c b/drivers/power/supply/smb347-charger.c
index 996a82f8..b5f03831 100644
--- a/drivers/power/supply/smb347-charger.c
+++ b/drivers/power/supply/smb347-charger.c
@@ -1528,9 +1528,9 @@ static const struct regulator_desc smb347_usb_vbus_regulator_desc = {
.n_voltages = 1,
};
-static int smb347_probe(struct i2c_client *client,
- const struct i2c_device_id *id)
+static int smb347_probe(struct i2c_client *client)
{
+ const struct i2c_device_id *id = i2c_client_get_device_id(client);
struct power_supply_config mains_usb_cfg = {};
struct regulator_config usb_rdev_cfg = {};
struct device *dev = &client->dev;
@@ -1629,7 +1629,7 @@ static struct i2c_driver smb347_driver = {
.name = "smb347",
.of_match_table = smb3xx_of_match,
},
- .probe = smb347_probe,
+ .probe_new = smb347_probe,
.remove = smb347_remove,
.shutdown = smb347_shutdown,
.id_table = smb347_id,
diff --git a/drivers/power/supply/ucs1002_power.c b/drivers/power/supply/ucs1002_power.c
index ef673ec..836d44c 100644
--- a/drivers/power/supply/ucs1002_power.c
+++ b/drivers/power/supply/ucs1002_power.c
@@ -532,8 +532,7 @@ static const struct regulator_desc ucs1002_regulator_descriptor = {
.n_voltages = 1,
};
-static int ucs1002_probe(struct i2c_client *client,
- const struct i2c_device_id *dev_id)
+static int ucs1002_probe(struct i2c_client *client)
{
struct device *dev = &client->dev;
struct power_supply_config charger_config = {};
@@ -681,7 +680,7 @@ static struct i2c_driver ucs1002_driver = {
.name = "ucs1002",
.of_match_table = ucs1002_of_match,
},
- .probe = ucs1002_probe,
+ .probe_new = ucs1002_probe,
};
module_i2c_driver(ucs1002_driver);
diff --git a/drivers/power/supply/z2_battery.c b/drivers/power/supply/z2_battery.c
index 1897c29..0ba4a59 100644
--- a/drivers/power/supply/z2_battery.c
+++ b/drivers/power/supply/z2_battery.c
@@ -176,8 +176,7 @@ static int z2_batt_ps_init(struct z2_charger *charger, int props)
return 0;
}
-static int z2_batt_probe(struct i2c_client *client,
- const struct i2c_device_id *id)
+static int z2_batt_probe(struct i2c_client *client)
{
int ret = 0;
int props = 1; /* POWER_SUPPLY_PROP_PRESENT */
@@ -206,10 +205,12 @@ static int z2_batt_probe(struct i2c_client *client,
charger->charge_gpiod = devm_gpiod_get_optional(&client->dev,
NULL, GPIOD_IN);
- if (IS_ERR(charger->charge_gpiod))
- return dev_err_probe(&client->dev,
+ if (IS_ERR(charger->charge_gpiod)) {
+ ret = dev_err_probe(&client->dev,
PTR_ERR(charger->charge_gpiod),
"failed to get charge GPIO\n");
+ goto err;
+ }
if (charger->charge_gpiod) {
gpiod_set_consumer_name(charger->charge_gpiod, "BATT CHRG");
@@ -306,7 +307,7 @@ static struct i2c_driver z2_batt_driver = {
.name = "z2-battery",
.pm = Z2_BATTERY_PM_OPS
},
- .probe = z2_batt_probe,
+ .probe_new = z2_batt_probe,
.remove = z2_batt_remove,
.id_table = z2_batt_id,
};
diff --git a/drivers/ps3/ps3-lpm.c b/drivers/ps3/ps3-lpm.c
index 65512b6..200ad87 100644
--- a/drivers/ps3/ps3-lpm.c
+++ b/drivers/ps3/ps3-lpm.c
@@ -1066,7 +1066,7 @@ EXPORT_SYMBOL_GPL(ps3_disable_pm_interrupts);
* instance, specified by one of enum ps3_lpm_tb_type.
* @tb_cache: Optional user supplied buffer to use as the trace buffer cache.
* If NULL, the driver will allocate and manage an internal buffer.
- * Unused when when @tb_type is PS3_LPM_TB_TYPE_NONE.
+ * Unused when @tb_type is PS3_LPM_TB_TYPE_NONE.
* @tb_cache_size: The size in bytes of the user supplied @tb_cache buffer.
* Unused when @tb_cache is NULL or @tb_type is PS3_LPM_TB_TYPE_NONE.
*/
diff --git a/drivers/rtc/Kconfig b/drivers/rtc/Kconfig
index bb63edb..2bb640d 100644
--- a/drivers/rtc/Kconfig
+++ b/drivers/rtc/Kconfig
@@ -433,7 +433,7 @@
config RTC_DRV_ISL12026
tristate "Intersil ISL12026"
- depends on OF || COMPILE_TEST
+ depends on OF
help
If you say yes here you get support for the
Intersil ISL12026 RTC chip.
@@ -1351,16 +1351,6 @@
This driver can also be built as a module. If so, the module
will be called rtc-asm9260.
-config RTC_DRV_DAVINCI
- tristate "TI DaVinci RTC"
- depends on ARCH_DAVINCI_DM365 || COMPILE_TEST
- help
- If you say yes here you get support for the RTC on the
- DaVinci platforms (DM365).
-
- This driver can also be built as a module. If so, the module
- will be called rtc-davinci.
-
config RTC_DRV_DIGICOLOR
tristate "Conexant Digicolor RTC"
depends on ARCH_DIGICOLOR || COMPILE_TEST
diff --git a/drivers/rtc/Makefile b/drivers/rtc/Makefile
index aab22bc..791994e 100644
--- a/drivers/rtc/Makefile
+++ b/drivers/rtc/Makefile
@@ -44,7 +44,6 @@
obj-$(CONFIG_RTC_DRV_DA9052) += rtc-da9052.o
obj-$(CONFIG_RTC_DRV_DA9055) += rtc-da9055.o
obj-$(CONFIG_RTC_DRV_DA9063) += rtc-da9063.o
-obj-$(CONFIG_RTC_DRV_DAVINCI) += rtc-davinci.o
obj-$(CONFIG_RTC_DRV_DIGICOLOR) += rtc-digicolor.o
obj-$(CONFIG_RTC_DRV_DM355EVM) += rtc-dm355evm.o
obj-$(CONFIG_RTC_DRV_DS1216) += rtc-ds1216.o
diff --git a/drivers/rtc/class.c b/drivers/rtc/class.c
index e48223c..e5b7b48 100644
--- a/drivers/rtc/class.c
+++ b/drivers/rtc/class.c
@@ -374,11 +374,11 @@ struct rtc_device *devm_rtc_allocate_device(struct device *dev)
rtc->id = id;
rtc->dev.parent = dev;
- err = dev_set_name(&rtc->dev, "rtc%d", id);
+ err = devm_add_action_or_reset(dev, devm_rtc_release_device, rtc);
if (err)
return ERR_PTR(err);
- err = devm_add_action_or_reset(dev, devm_rtc_release_device, rtc);
+ err = dev_set_name(&rtc->dev, "rtc%d", id);
if (err)
return ERR_PTR(err);
diff --git a/drivers/rtc/interface.c b/drivers/rtc/interface.c
index 9edd662c..7c30cb3 100644
--- a/drivers/rtc/interface.c
+++ b/drivers/rtc/interface.c
@@ -256,7 +256,7 @@ int __rtc_read_alarm(struct rtc_device *rtc, struct rtc_wkalrm *alarm)
*
* This could all instead be done in the lower level driver,
* but since more than one lower level RTC implementation needs it,
- * then it's probably best best to do it here instead of there..
+ * then it's probably best to do it here instead of there..
*/
/* Get the "before" timestamp */
diff --git a/drivers/rtc/rtc-abx80x.c b/drivers/rtc/rtc-abx80x.c
index 9b0138d..2e0e643 100644
--- a/drivers/rtc/rtc-abx80x.c
+++ b/drivers/rtc/rtc-abx80x.c
@@ -12,6 +12,7 @@
#include <linux/bcd.h>
#include <linux/i2c.h>
+#include <linux/kstrtox.h>
#include <linux/module.h>
#include <linux/of_device.h>
#include <linux/rtc.h>
@@ -673,13 +674,28 @@ static int abx80x_setup_watchdog(struct abx80x_priv *priv)
}
#endif
-static int abx80x_probe(struct i2c_client *client,
- const struct i2c_device_id *id)
+static const struct i2c_device_id abx80x_id[] = {
+ { "abx80x", ABX80X },
+ { "ab0801", AB0801 },
+ { "ab0803", AB0803 },
+ { "ab0804", AB0804 },
+ { "ab0805", AB0805 },
+ { "ab1801", AB1801 },
+ { "ab1803", AB1803 },
+ { "ab1804", AB1804 },
+ { "ab1805", AB1805 },
+ { "rv1805", RV1805 },
+ { }
+};
+MODULE_DEVICE_TABLE(i2c, abx80x_id);
+
+static int abx80x_probe(struct i2c_client *client)
{
struct device_node *np = client->dev.of_node;
struct abx80x_priv *priv;
int i, data, err, trickle_cfg = -EINVAL;
char buf[7];
+ const struct i2c_device_id *id = i2c_match_id(abx80x_id, client);
unsigned int part = id->driver_data;
unsigned int partnumber;
unsigned int majrev, minrev;
@@ -847,21 +863,6 @@ static int abx80x_probe(struct i2c_client *client,
return devm_rtc_register_device(priv->rtc);
}
-static const struct i2c_device_id abx80x_id[] = {
- { "abx80x", ABX80X },
- { "ab0801", AB0801 },
- { "ab0803", AB0803 },
- { "ab0804", AB0804 },
- { "ab0805", AB0805 },
- { "ab1801", AB1801 },
- { "ab1803", AB1803 },
- { "ab1804", AB1804 },
- { "ab1805", AB1805 },
- { "rv1805", RV1805 },
- { }
-};
-MODULE_DEVICE_TABLE(i2c, abx80x_id);
-
#ifdef CONFIG_OF
static const struct of_device_id abx80x_of_match[] = {
{
@@ -914,7 +915,7 @@ static struct i2c_driver abx80x_driver = {
.name = "rtc-abx80x",
.of_match_table = of_match_ptr(abx80x_of_match),
},
- .probe = abx80x_probe,
+ .probe_new = abx80x_probe,
.id_table = abx80x_id,
};
diff --git a/drivers/rtc/rtc-at91rm9200.c b/drivers/rtc/rtc-at91rm9200.c
index fe396d2..e9d1723 100644
--- a/drivers/rtc/rtc-at91rm9200.c
+++ b/drivers/rtc/rtc-at91rm9200.c
@@ -130,7 +130,7 @@ static void at91_rtc_write_idr(u32 mask)
*
* Note that there is still a possibility that the mask is updated
* before interrupts have actually been disabled in hardware. The only
- * way to be certain would be to poll the IMR-register, which is is
+ * way to be certain would be to poll the IMR-register, which is
* the very register we are trying to emulate. The register read back
* is a reasonable heuristic.
*/
diff --git a/drivers/rtc/rtc-bq32k.c b/drivers/rtc/rtc-bq32k.c
index 6d6a55ef..967ddc6 100644
--- a/drivers/rtc/rtc-bq32k.c
+++ b/drivers/rtc/rtc-bq32k.c
@@ -13,6 +13,7 @@
#include <linux/i2c.h>
#include <linux/rtc.h>
#include <linux/init.h>
+#include <linux/kstrtox.h>
#include <linux/errno.h>
#include <linux/bcd.h>
diff --git a/drivers/rtc/rtc-cmos.c b/drivers/rtc/rtc-cmos.c
index 58cc2ba..00e2ca7 100644
--- a/drivers/rtc/rtc-cmos.c
+++ b/drivers/rtc/rtc-cmos.c
@@ -744,6 +744,168 @@ static irqreturn_t cmos_interrupt(int irq, void *p)
return IRQ_NONE;
}
+#ifdef CONFIG_ACPI
+
+#include <linux/acpi.h>
+
+static u32 rtc_handler(void *context)
+{
+ struct device *dev = context;
+ struct cmos_rtc *cmos = dev_get_drvdata(dev);
+ unsigned char rtc_control = 0;
+ unsigned char rtc_intr;
+ unsigned long flags;
+
+
+ /*
+ * Always update rtc irq when ACPI is used as RTC Alarm.
+ * Or else, ACPI SCI is enabled during suspend/resume only,
+ * update rtc irq in that case.
+ */
+ if (cmos_use_acpi_alarm())
+ cmos_interrupt(0, (void *)cmos->rtc);
+ else {
+ /* Fix me: can we use cmos_interrupt() here as well? */
+ spin_lock_irqsave(&rtc_lock, flags);
+ if (cmos_rtc.suspend_ctrl)
+ rtc_control = CMOS_READ(RTC_CONTROL);
+ if (rtc_control & RTC_AIE) {
+ cmos_rtc.suspend_ctrl &= ~RTC_AIE;
+ CMOS_WRITE(rtc_control, RTC_CONTROL);
+ rtc_intr = CMOS_READ(RTC_INTR_FLAGS);
+ rtc_update_irq(cmos->rtc, 1, rtc_intr);
+ }
+ spin_unlock_irqrestore(&rtc_lock, flags);
+ }
+
+ pm_wakeup_hard_event(dev);
+ acpi_clear_event(ACPI_EVENT_RTC);
+ acpi_disable_event(ACPI_EVENT_RTC, 0);
+ return ACPI_INTERRUPT_HANDLED;
+}
+
+static void acpi_rtc_event_setup(struct device *dev)
+{
+ if (acpi_disabled)
+ return;
+
+ acpi_install_fixed_event_handler(ACPI_EVENT_RTC, rtc_handler, dev);
+ /*
+ * After the RTC handler is installed, the Fixed_RTC event should
+ * be disabled. Only when the RTC alarm is set will it be enabled.
+ */
+ acpi_clear_event(ACPI_EVENT_RTC);
+ acpi_disable_event(ACPI_EVENT_RTC, 0);
+}
+
+static void acpi_rtc_event_cleanup(void)
+{
+ if (acpi_disabled)
+ return;
+
+ acpi_remove_fixed_event_handler(ACPI_EVENT_RTC, rtc_handler);
+}
+
+static void rtc_wake_on(struct device *dev)
+{
+ acpi_clear_event(ACPI_EVENT_RTC);
+ acpi_enable_event(ACPI_EVENT_RTC, 0);
+}
+
+static void rtc_wake_off(struct device *dev)
+{
+ acpi_disable_event(ACPI_EVENT_RTC, 0);
+}
+
+#ifdef CONFIG_X86
+/* Enable use_acpi_alarm mode for Intel platforms no earlier than 2015 */
+static void use_acpi_alarm_quirks(void)
+{
+ if (boot_cpu_data.x86_vendor != X86_VENDOR_INTEL)
+ return;
+
+ if (!is_hpet_enabled())
+ return;
+
+ if (dmi_get_bios_year() < 2015)
+ return;
+
+ use_acpi_alarm = true;
+}
+#else
+static inline void use_acpi_alarm_quirks(void) { }
+#endif
+
+static void acpi_cmos_wake_setup(struct device *dev)
+{
+ if (acpi_disabled)
+ return;
+
+ use_acpi_alarm_quirks();
+
+ cmos_rtc.wake_on = rtc_wake_on;
+ cmos_rtc.wake_off = rtc_wake_off;
+
+ /* ACPI tables bug workaround. */
+ if (acpi_gbl_FADT.month_alarm && !acpi_gbl_FADT.day_alarm) {
+ dev_dbg(dev, "bogus FADT month_alarm (%d)\n",
+ acpi_gbl_FADT.month_alarm);
+ acpi_gbl_FADT.month_alarm = 0;
+ }
+
+ cmos_rtc.day_alrm = acpi_gbl_FADT.day_alarm;
+ cmos_rtc.mon_alrm = acpi_gbl_FADT.month_alarm;
+ cmos_rtc.century = acpi_gbl_FADT.century;
+
+ if (acpi_gbl_FADT.flags & ACPI_FADT_S4_RTC_WAKE)
+ dev_info(dev, "RTC can wake from S4\n");
+
+ /* RTC always wakes from S1/S2/S3, and often S4/STD */
+ device_init_wakeup(dev, 1);
+}
+
+static void cmos_check_acpi_rtc_status(struct device *dev,
+ unsigned char *rtc_control)
+{
+ struct cmos_rtc *cmos = dev_get_drvdata(dev);
+ acpi_event_status rtc_status;
+ acpi_status status;
+
+ if (acpi_gbl_FADT.flags & ACPI_FADT_FIXED_RTC)
+ return;
+
+ status = acpi_get_event_status(ACPI_EVENT_RTC, &rtc_status);
+ if (ACPI_FAILURE(status)) {
+ dev_err(dev, "Could not get RTC status\n");
+ } else if (rtc_status & ACPI_EVENT_FLAG_SET) {
+ unsigned char mask;
+ *rtc_control &= ~RTC_AIE;
+ CMOS_WRITE(*rtc_control, RTC_CONTROL);
+ mask = CMOS_READ(RTC_INTR_FLAGS);
+ rtc_update_irq(cmos->rtc, 1, mask);
+ }
+}
+
+#else /* !CONFIG_ACPI */
+
+static inline void acpi_rtc_event_setup(struct device *dev)
+{
+}
+
+static inline void acpi_rtc_event_cleanup(void)
+{
+}
+
+static inline void acpi_cmos_wake_setup(struct device *dev)
+{
+}
+
+static inline void cmos_check_acpi_rtc_status(struct device *dev,
+ unsigned char *rtc_control)
+{
+}
+#endif /* CONFIG_ACPI */
+
#ifdef CONFIG_PNP
#define INITSECTION
@@ -827,19 +989,27 @@ cmos_do_probe(struct device *dev, struct resource *ports, int rtc_irq)
if (info->address_space)
address_space = info->address_space;
- if (info->rtc_day_alarm && info->rtc_day_alarm < 128)
- cmos_rtc.day_alrm = info->rtc_day_alarm;
- if (info->rtc_mon_alarm && info->rtc_mon_alarm < 128)
- cmos_rtc.mon_alrm = info->rtc_mon_alarm;
- if (info->rtc_century && info->rtc_century < 128)
- cmos_rtc.century = info->rtc_century;
+ cmos_rtc.day_alrm = info->rtc_day_alarm;
+ cmos_rtc.mon_alrm = info->rtc_mon_alarm;
+ cmos_rtc.century = info->rtc_century;
if (info->wake_on && info->wake_off) {
cmos_rtc.wake_on = info->wake_on;
cmos_rtc.wake_off = info->wake_off;
}
+ } else {
+ acpi_cmos_wake_setup(dev);
}
+ if (cmos_rtc.day_alrm >= 128)
+ cmos_rtc.day_alrm = 0;
+
+ if (cmos_rtc.mon_alrm >= 128)
+ cmos_rtc.mon_alrm = 0;
+
+ if (cmos_rtc.century >= 128)
+ cmos_rtc.century = 0;
+
cmos_rtc.dev = dev;
dev_set_drvdata(dev, &cmos_rtc);
@@ -928,6 +1098,13 @@ cmos_do_probe(struct device *dev, struct resource *ports, int rtc_irq)
nvmem_cfg.size = address_space - NVRAM_OFFSET;
devm_rtc_nvmem_register(cmos_rtc.rtc, &nvmem_cfg);
+ /*
+ * Everything has gone well so far, so by default register a handler for
+ * the ACPI RTC fixed event.
+ */
+ if (!info)
+ acpi_rtc_event_setup(dev);
+
dev_info(dev, "%s%s, %d bytes nvram%s\n",
!is_valid_irq(rtc_irq) ? "no alarms" :
cmos_rtc.mon_alrm ? "alarms up to one year" :
@@ -973,6 +1150,9 @@ static void cmos_do_remove(struct device *dev)
hpet_unregister_irq_handler(cmos_interrupt);
}
+ if (!dev_get_platdata(dev))
+ acpi_rtc_event_cleanup();
+
cmos->rtc = NULL;
ports = cmos->iomem;
@@ -1122,9 +1302,6 @@ static void cmos_check_wkalrm(struct device *dev)
}
}
-static void cmos_check_acpi_rtc_status(struct device *dev,
- unsigned char *rtc_control);
-
static int __maybe_unused cmos_resume(struct device *dev)
{
struct cmos_rtc *cmos = dev_get_drvdata(dev);
@@ -1191,175 +1368,13 @@ static SIMPLE_DEV_PM_OPS(cmos_pm_ops, cmos_suspend, cmos_resume);
* predate even PNPBIOS should set up platform_bus devices.
*/
-#ifdef CONFIG_ACPI
-
-#include <linux/acpi.h>
-
-static u32 rtc_handler(void *context)
-{
- struct device *dev = context;
- struct cmos_rtc *cmos = dev_get_drvdata(dev);
- unsigned char rtc_control = 0;
- unsigned char rtc_intr;
- unsigned long flags;
-
-
- /*
- * Always update rtc irq when ACPI is used as RTC Alarm.
- * Or else, ACPI SCI is enabled during suspend/resume only,
- * update rtc irq in that case.
- */
- if (cmos_use_acpi_alarm())
- cmos_interrupt(0, (void *)cmos->rtc);
- else {
- /* Fix me: can we use cmos_interrupt() here as well? */
- spin_lock_irqsave(&rtc_lock, flags);
- if (cmos_rtc.suspend_ctrl)
- rtc_control = CMOS_READ(RTC_CONTROL);
- if (rtc_control & RTC_AIE) {
- cmos_rtc.suspend_ctrl &= ~RTC_AIE;
- CMOS_WRITE(rtc_control, RTC_CONTROL);
- rtc_intr = CMOS_READ(RTC_INTR_FLAGS);
- rtc_update_irq(cmos->rtc, 1, rtc_intr);
- }
- spin_unlock_irqrestore(&rtc_lock, flags);
- }
-
- pm_wakeup_hard_event(dev);
- acpi_clear_event(ACPI_EVENT_RTC);
- acpi_disable_event(ACPI_EVENT_RTC, 0);
- return ACPI_INTERRUPT_HANDLED;
-}
-
-static inline void rtc_wake_setup(struct device *dev)
-{
- if (acpi_disabled)
- return;
-
- acpi_install_fixed_event_handler(ACPI_EVENT_RTC, rtc_handler, dev);
- /*
- * After the RTC handler is installed, the Fixed_RTC event should
- * be disabled. Only when the RTC alarm is set will it be enabled.
- */
- acpi_clear_event(ACPI_EVENT_RTC);
- acpi_disable_event(ACPI_EVENT_RTC, 0);
-}
-
-static void rtc_wake_on(struct device *dev)
-{
- acpi_clear_event(ACPI_EVENT_RTC);
- acpi_enable_event(ACPI_EVENT_RTC, 0);
-}
-
-static void rtc_wake_off(struct device *dev)
-{
- acpi_disable_event(ACPI_EVENT_RTC, 0);
-}
-
-#ifdef CONFIG_X86
-/* Enable use_acpi_alarm mode for Intel platforms no earlier than 2015 */
-static void use_acpi_alarm_quirks(void)
-{
- if (boot_cpu_data.x86_vendor != X86_VENDOR_INTEL)
- return;
-
- if (!is_hpet_enabled())
- return;
-
- if (dmi_get_bios_year() < 2015)
- return;
-
- use_acpi_alarm = true;
-}
-#else
-static inline void use_acpi_alarm_quirks(void) { }
-#endif
-
-/* Every ACPI platform has a mc146818 compatible "cmos rtc". Here we find
- * its device node and pass extra config data. This helps its driver use
- * capabilities that the now-obsolete mc146818 didn't have, and informs it
- * that this board's RTC is wakeup-capable (per ACPI spec).
- */
-static struct cmos_rtc_board_info acpi_rtc_info;
-
-static void cmos_wake_setup(struct device *dev)
-{
- if (acpi_disabled)
- return;
-
- use_acpi_alarm_quirks();
-
- acpi_rtc_info.wake_on = rtc_wake_on;
- acpi_rtc_info.wake_off = rtc_wake_off;
-
- /* workaround bug in some ACPI tables */
- if (acpi_gbl_FADT.month_alarm && !acpi_gbl_FADT.day_alarm) {
- dev_dbg(dev, "bogus FADT month_alarm (%d)\n",
- acpi_gbl_FADT.month_alarm);
- acpi_gbl_FADT.month_alarm = 0;
- }
-
- acpi_rtc_info.rtc_day_alarm = acpi_gbl_FADT.day_alarm;
- acpi_rtc_info.rtc_mon_alarm = acpi_gbl_FADT.month_alarm;
- acpi_rtc_info.rtc_century = acpi_gbl_FADT.century;
-
- /* NOTE: S4_RTC_WAKE is NOT currently useful to Linux */
- if (acpi_gbl_FADT.flags & ACPI_FADT_S4_RTC_WAKE)
- dev_info(dev, "RTC can wake from S4\n");
-
- dev->platform_data = &acpi_rtc_info;
-
- /* RTC always wakes from S1/S2/S3, and often S4/STD */
- device_init_wakeup(dev, 1);
-}
-
-static void cmos_check_acpi_rtc_status(struct device *dev,
- unsigned char *rtc_control)
-{
- struct cmos_rtc *cmos = dev_get_drvdata(dev);
- acpi_event_status rtc_status;
- acpi_status status;
-
- if (acpi_gbl_FADT.flags & ACPI_FADT_FIXED_RTC)
- return;
-
- status = acpi_get_event_status(ACPI_EVENT_RTC, &rtc_status);
- if (ACPI_FAILURE(status)) {
- dev_err(dev, "Could not get RTC status\n");
- } else if (rtc_status & ACPI_EVENT_FLAG_SET) {
- unsigned char mask;
- *rtc_control &= ~RTC_AIE;
- CMOS_WRITE(*rtc_control, RTC_CONTROL);
- mask = CMOS_READ(RTC_INTR_FLAGS);
- rtc_update_irq(cmos->rtc, 1, mask);
- }
-}
-
-#else
-
-static void cmos_wake_setup(struct device *dev)
-{
-}
-
-static void cmos_check_acpi_rtc_status(struct device *dev,
- unsigned char *rtc_control)
-{
-}
-
-static void rtc_wake_setup(struct device *dev)
-{
-}
-#endif
-
#ifdef CONFIG_PNP
#include <linux/pnp.h>
static int cmos_pnp_probe(struct pnp_dev *pnp, const struct pnp_device_id *id)
{
- int irq, ret;
-
- cmos_wake_setup(&pnp->dev);
+ int irq;
if (pnp_port_start(pnp, 0) == 0x70 && !pnp_irq_valid(pnp, 0)) {
irq = 0;
@@ -1375,13 +1390,7 @@ static int cmos_pnp_probe(struct pnp_dev *pnp, const struct pnp_device_id *id)
irq = pnp_irq(pnp, 0);
}
- ret = cmos_do_probe(&pnp->dev, pnp_get_resource(pnp, IORESOURCE_IO, 0), irq);
- if (ret)
- return ret;
-
- rtc_wake_setup(&pnp->dev);
-
- return 0;
+ return cmos_do_probe(&pnp->dev, pnp_get_resource(pnp, IORESOURCE_IO, 0), irq);
}
static void cmos_pnp_remove(struct pnp_dev *pnp)
@@ -1465,10 +1474,9 @@ static inline void cmos_of_init(struct platform_device *pdev) {}
static int __init cmos_platform_probe(struct platform_device *pdev)
{
struct resource *resource;
- int irq, ret;
+ int irq;
cmos_of_init(pdev);
- cmos_wake_setup(&pdev->dev);
if (RTC_IOMAPPED)
resource = platform_get_resource(pdev, IORESOURCE_IO, 0);
@@ -1478,13 +1486,7 @@ static int __init cmos_platform_probe(struct platform_device *pdev)
if (irq < 0)
irq = -1;
- ret = cmos_do_probe(&pdev->dev, resource, irq);
- if (ret)
- return ret;
-
- rtc_wake_setup(&pdev->dev);
-
- return 0;
+ return cmos_do_probe(&pdev->dev, resource, irq);
}
static int cmos_platform_remove(struct platform_device *pdev)
diff --git a/drivers/rtc/rtc-cros-ec.c b/drivers/rtc/rtc-cros-ec.c
index 887f519..a3ec066 100644
--- a/drivers/rtc/rtc-cros-ec.c
+++ b/drivers/rtc/rtc-cros-ec.c
@@ -14,6 +14,8 @@
#define DRV_NAME "cros-ec-rtc"
+#define SECS_PER_DAY (24 * 60 * 60)
+
/**
* struct cros_ec_rtc - Driver data for EC RTC
*
@@ -43,13 +45,8 @@ static int cros_ec_rtc_get(struct cros_ec_device *cros_ec, u32 command,
msg.msg.insize = sizeof(msg.data);
ret = cros_ec_cmd_xfer_status(cros_ec, &msg.msg);
- if (ret < 0) {
- dev_err(cros_ec->dev,
- "error getting %s from EC: %d\n",
- command == EC_CMD_RTC_GET_VALUE ? "time" : "alarm",
- ret);
+ if (ret < 0)
return ret;
- }
*response = msg.data.time;
@@ -59,7 +56,7 @@ static int cros_ec_rtc_get(struct cros_ec_device *cros_ec, u32 command,
static int cros_ec_rtc_set(struct cros_ec_device *cros_ec, u32 command,
u32 param)
{
- int ret = 0;
+ int ret;
struct {
struct cros_ec_command msg;
struct ec_response_rtc data;
@@ -71,13 +68,8 @@ static int cros_ec_rtc_set(struct cros_ec_device *cros_ec, u32 command,
msg.data.time = param;
ret = cros_ec_cmd_xfer_status(cros_ec, &msg.msg);
- if (ret < 0) {
- dev_err(cros_ec->dev, "error setting %s on EC: %d\n",
- command == EC_CMD_RTC_SET_VALUE ? "time" : "alarm",
- ret);
+ if (ret < 0)
return ret;
- }
-
return 0;
}
@@ -190,8 +182,21 @@ static int cros_ec_rtc_set_alarm(struct device *dev, struct rtc_wkalrm *alrm)
ret = cros_ec_rtc_set(cros_ec, EC_CMD_RTC_SET_ALARM, alarm_offset);
if (ret < 0) {
- dev_err(dev, "error setting alarm: %d\n", ret);
- return ret;
+ if (ret == -EINVAL && alarm_offset >= SECS_PER_DAY) {
+ /*
+ * RTC chips on some older Chromebooks can only handle
+ * alarms up to 24h in the future. Try to set an alarm
+ * below that limit to avoid suspend failures.
+ */
+ ret = cros_ec_rtc_set(cros_ec, EC_CMD_RTC_SET_ALARM,
+ SECS_PER_DAY - 1);
+ }
+
+ if (ret < 0) {
+ dev_err(dev, "error setting alarm in %u seconds: %d\n",
+ alarm_offset, ret);
+ return ret;
+ }
}
return 0;
diff --git a/drivers/rtc/rtc-davinci.c b/drivers/rtc/rtc-davinci.c
deleted file mode 100644
index 6bef0f2..0000000
--- a/drivers/rtc/rtc-davinci.c
+++ /dev/null
@@ -1,512 +0,0 @@
-// SPDX-License-Identifier: GPL-2.0-or-later
-/*
- * DaVinci Power Management and Real Time Clock Driver for TI platforms
- *
- * Copyright (C) 2009 Texas Instruments, Inc
- *
- * Author: Miguel Aguilar <miguel.aguilar@ridgerun.com>
- */
-#include <linux/kernel.h>
-#include <linux/init.h>
-#include <linux/module.h>
-#include <linux/ioport.h>
-#include <linux/delay.h>
-#include <linux/spinlock.h>
-#include <linux/rtc.h>
-#include <linux/bcd.h>
-#include <linux/platform_device.h>
-#include <linux/io.h>
-#include <linux/slab.h>
-
-/*
- * The DaVinci RTC is a simple RTC with the following
- * Sec: 0 - 59 : BCD count
- * Min: 0 - 59 : BCD count
- * Hour: 0 - 23 : BCD count
- * Day: 0 - 0x7FFF(32767) : Binary count ( Over 89 years )
- */
-
-/* PRTC interface registers */
-#define DAVINCI_PRTCIF_PID 0x00
-#define PRTCIF_CTLR 0x04
-#define PRTCIF_LDATA 0x08
-#define PRTCIF_UDATA 0x0C
-#define PRTCIF_INTEN 0x10
-#define PRTCIF_INTFLG 0x14
-
-/* PRTCIF_CTLR bit fields */
-#define PRTCIF_CTLR_BUSY BIT(31)
-#define PRTCIF_CTLR_SIZE BIT(25)
-#define PRTCIF_CTLR_DIR BIT(24)
-#define PRTCIF_CTLR_BENU_MSB BIT(23)
-#define PRTCIF_CTLR_BENU_3RD_BYTE BIT(22)
-#define PRTCIF_CTLR_BENU_2ND_BYTE BIT(21)
-#define PRTCIF_CTLR_BENU_LSB BIT(20)
-#define PRTCIF_CTLR_BENU_MASK (0x00F00000)
-#define PRTCIF_CTLR_BENL_MSB BIT(19)
-#define PRTCIF_CTLR_BENL_3RD_BYTE BIT(18)
-#define PRTCIF_CTLR_BENL_2ND_BYTE BIT(17)
-#define PRTCIF_CTLR_BENL_LSB BIT(16)
-#define PRTCIF_CTLR_BENL_MASK (0x000F0000)
-
-/* PRTCIF_INTEN bit fields */
-#define PRTCIF_INTEN_RTCSS BIT(1)
-#define PRTCIF_INTEN_RTCIF BIT(0)
-#define PRTCIF_INTEN_MASK (PRTCIF_INTEN_RTCSS \
- | PRTCIF_INTEN_RTCIF)
-
-/* PRTCIF_INTFLG bit fields */
-#define PRTCIF_INTFLG_RTCSS BIT(1)
-#define PRTCIF_INTFLG_RTCIF BIT(0)
-#define PRTCIF_INTFLG_MASK (PRTCIF_INTFLG_RTCSS \
- | PRTCIF_INTFLG_RTCIF)
-
-/* PRTC subsystem registers */
-#define PRTCSS_RTC_INTC_EXTENA1 (0x0C)
-#define PRTCSS_RTC_CTRL (0x10)
-#define PRTCSS_RTC_WDT (0x11)
-#define PRTCSS_RTC_TMR0 (0x12)
-#define PRTCSS_RTC_TMR1 (0x13)
-#define PRTCSS_RTC_CCTRL (0x14)
-#define PRTCSS_RTC_SEC (0x15)
-#define PRTCSS_RTC_MIN (0x16)
-#define PRTCSS_RTC_HOUR (0x17)
-#define PRTCSS_RTC_DAY0 (0x18)
-#define PRTCSS_RTC_DAY1 (0x19)
-#define PRTCSS_RTC_AMIN (0x1A)
-#define PRTCSS_RTC_AHOUR (0x1B)
-#define PRTCSS_RTC_ADAY0 (0x1C)
-#define PRTCSS_RTC_ADAY1 (0x1D)
-#define PRTCSS_RTC_CLKC_CNT (0x20)
-
-/* PRTCSS_RTC_INTC_EXTENA1 */
-#define PRTCSS_RTC_INTC_EXTENA1_MASK (0x07)
-
-/* PRTCSS_RTC_CTRL bit fields */
-#define PRTCSS_RTC_CTRL_WDTBUS BIT(7)
-#define PRTCSS_RTC_CTRL_WEN BIT(6)
-#define PRTCSS_RTC_CTRL_WDRT BIT(5)
-#define PRTCSS_RTC_CTRL_WDTFLG BIT(4)
-#define PRTCSS_RTC_CTRL_TE BIT(3)
-#define PRTCSS_RTC_CTRL_TIEN BIT(2)
-#define PRTCSS_RTC_CTRL_TMRFLG BIT(1)
-#define PRTCSS_RTC_CTRL_TMMD BIT(0)
-
-/* PRTCSS_RTC_CCTRL bit fields */
-#define PRTCSS_RTC_CCTRL_CALBUSY BIT(7)
-#define PRTCSS_RTC_CCTRL_DAEN BIT(5)
-#define PRTCSS_RTC_CCTRL_HAEN BIT(4)
-#define PRTCSS_RTC_CCTRL_MAEN BIT(3)
-#define PRTCSS_RTC_CCTRL_ALMFLG BIT(2)
-#define PRTCSS_RTC_CCTRL_AIEN BIT(1)
-#define PRTCSS_RTC_CCTRL_CAEN BIT(0)
-
-static DEFINE_SPINLOCK(davinci_rtc_lock);
-
-struct davinci_rtc {
- struct rtc_device *rtc;
- void __iomem *base;
- int irq;
-};
-
-static inline void rtcif_write(struct davinci_rtc *davinci_rtc,
- u32 val, u32 addr)
-{
- writel(val, davinci_rtc->base + addr);
-}
-
-static inline u32 rtcif_read(struct davinci_rtc *davinci_rtc, u32 addr)
-{
- return readl(davinci_rtc->base + addr);
-}
-
-static inline void rtcif_wait(struct davinci_rtc *davinci_rtc)
-{
- while (rtcif_read(davinci_rtc, PRTCIF_CTLR) & PRTCIF_CTLR_BUSY)
- cpu_relax();
-}
-
-static inline void rtcss_write(struct davinci_rtc *davinci_rtc,
- unsigned long val, u8 addr)
-{
- rtcif_wait(davinci_rtc);
-
- rtcif_write(davinci_rtc, PRTCIF_CTLR_BENL_LSB | addr, PRTCIF_CTLR);
- rtcif_write(davinci_rtc, val, PRTCIF_LDATA);
-
- rtcif_wait(davinci_rtc);
-}
-
-static inline u8 rtcss_read(struct davinci_rtc *davinci_rtc, u8 addr)
-{
- rtcif_wait(davinci_rtc);
-
- rtcif_write(davinci_rtc, PRTCIF_CTLR_DIR | PRTCIF_CTLR_BENL_LSB | addr,
- PRTCIF_CTLR);
-
- rtcif_wait(davinci_rtc);
-
- return rtcif_read(davinci_rtc, PRTCIF_LDATA);
-}
-
-static inline void davinci_rtcss_calendar_wait(struct davinci_rtc *davinci_rtc)
-{
- while (rtcss_read(davinci_rtc, PRTCSS_RTC_CCTRL) &
- PRTCSS_RTC_CCTRL_CALBUSY)
- cpu_relax();
-}
-
-static irqreturn_t davinci_rtc_interrupt(int irq, void *class_dev)
-{
- struct davinci_rtc *davinci_rtc = class_dev;
- unsigned long events = 0;
- u32 irq_flg;
- u8 alm_irq, tmr_irq;
- u8 rtc_ctrl, rtc_cctrl;
- int ret = IRQ_NONE;
-
- irq_flg = rtcif_read(davinci_rtc, PRTCIF_INTFLG) &
- PRTCIF_INTFLG_RTCSS;
-
- alm_irq = rtcss_read(davinci_rtc, PRTCSS_RTC_CCTRL) &
- PRTCSS_RTC_CCTRL_ALMFLG;
-
- tmr_irq = rtcss_read(davinci_rtc, PRTCSS_RTC_CTRL) &
- PRTCSS_RTC_CTRL_TMRFLG;
-
- if (irq_flg) {
- if (alm_irq) {
- events |= RTC_IRQF | RTC_AF;
- rtc_cctrl = rtcss_read(davinci_rtc, PRTCSS_RTC_CCTRL);
- rtc_cctrl |= PRTCSS_RTC_CCTRL_ALMFLG;
- rtcss_write(davinci_rtc, rtc_cctrl, PRTCSS_RTC_CCTRL);
- } else if (tmr_irq) {
- events |= RTC_IRQF | RTC_PF;
- rtc_ctrl = rtcss_read(davinci_rtc, PRTCSS_RTC_CTRL);
- rtc_ctrl |= PRTCSS_RTC_CTRL_TMRFLG;
- rtcss_write(davinci_rtc, rtc_ctrl, PRTCSS_RTC_CTRL);
- }
-
- rtcif_write(davinci_rtc, PRTCIF_INTFLG_RTCSS,
- PRTCIF_INTFLG);
- rtc_update_irq(davinci_rtc->rtc, 1, events);
-
- ret = IRQ_HANDLED;
- }
-
- return ret;
-}
-
-static int
-davinci_rtc_ioctl(struct device *dev, unsigned int cmd, unsigned long arg)
-{
- struct davinci_rtc *davinci_rtc = dev_get_drvdata(dev);
- u8 rtc_ctrl;
- unsigned long flags;
- int ret = 0;
-
- spin_lock_irqsave(&davinci_rtc_lock, flags);
-
- rtc_ctrl = rtcss_read(davinci_rtc, PRTCSS_RTC_CTRL);
-
- switch (cmd) {
- case RTC_WIE_ON:
- rtc_ctrl |= PRTCSS_RTC_CTRL_WEN | PRTCSS_RTC_CTRL_WDTFLG;
- break;
- case RTC_WIE_OFF:
- rtc_ctrl &= ~PRTCSS_RTC_CTRL_WEN;
- break;
- default:
- ret = -ENOIOCTLCMD;
- }
-
- rtcss_write(davinci_rtc, rtc_ctrl, PRTCSS_RTC_CTRL);
-
- spin_unlock_irqrestore(&davinci_rtc_lock, flags);
-
- return ret;
-}
-
-static void convertfromdays(u16 days, struct rtc_time *tm)
-{
- int tmp_days, year, mon;
-
- for (year = 2000;; year++) {
- tmp_days = rtc_year_days(1, 12, year);
- if (days >= tmp_days)
- days -= tmp_days;
- else {
- for (mon = 0;; mon++) {
- tmp_days = rtc_month_days(mon, year);
- if (days >= tmp_days) {
- days -= tmp_days;
- } else {
- tm->tm_year = year - 1900;
- tm->tm_mon = mon;
- tm->tm_mday = days + 1;
- break;
- }
- }
- break;
- }
- }
-}
-
-static void convert2days(u16 *days, struct rtc_time *tm)
-{
- int i;
- *days = 0;
-
- for (i = 2000; i < 1900 + tm->tm_year; i++)
- *days += rtc_year_days(1, 12, i);
-
- *days += rtc_year_days(tm->tm_mday, tm->tm_mon, 1900 + tm->tm_year);
-}
-
-static int davinci_rtc_read_time(struct device *dev, struct rtc_time *tm)
-{
- struct davinci_rtc *davinci_rtc = dev_get_drvdata(dev);
- u16 days = 0;
- u8 day0, day1;
- unsigned long flags;
-
- spin_lock_irqsave(&davinci_rtc_lock, flags);
-
- davinci_rtcss_calendar_wait(davinci_rtc);
- tm->tm_sec = bcd2bin(rtcss_read(davinci_rtc, PRTCSS_RTC_SEC));
-
- davinci_rtcss_calendar_wait(davinci_rtc);
- tm->tm_min = bcd2bin(rtcss_read(davinci_rtc, PRTCSS_RTC_MIN));
-
- davinci_rtcss_calendar_wait(davinci_rtc);
- tm->tm_hour = bcd2bin(rtcss_read(davinci_rtc, PRTCSS_RTC_HOUR));
-
- davinci_rtcss_calendar_wait(davinci_rtc);
- day0 = rtcss_read(davinci_rtc, PRTCSS_RTC_DAY0);
-
- davinci_rtcss_calendar_wait(davinci_rtc);
- day1 = rtcss_read(davinci_rtc, PRTCSS_RTC_DAY1);
-
- spin_unlock_irqrestore(&davinci_rtc_lock, flags);
-
- days |= day1;
- days <<= 8;
- days |= day0;
-
- convertfromdays(days, tm);
-
- return 0;
-}
-
-static int davinci_rtc_set_time(struct device *dev, struct rtc_time *tm)
-{
- struct davinci_rtc *davinci_rtc = dev_get_drvdata(dev);
- u16 days;
- u8 rtc_cctrl;
- unsigned long flags;
-
- convert2days(&days, tm);
-
- spin_lock_irqsave(&davinci_rtc_lock, flags);
-
- davinci_rtcss_calendar_wait(davinci_rtc);
- rtcss_write(davinci_rtc, bin2bcd(tm->tm_sec), PRTCSS_RTC_SEC);
-
- davinci_rtcss_calendar_wait(davinci_rtc);
- rtcss_write(davinci_rtc, bin2bcd(tm->tm_min), PRTCSS_RTC_MIN);
-
- davinci_rtcss_calendar_wait(davinci_rtc);
- rtcss_write(davinci_rtc, bin2bcd(tm->tm_hour), PRTCSS_RTC_HOUR);
-
- davinci_rtcss_calendar_wait(davinci_rtc);
- rtcss_write(davinci_rtc, days & 0xFF, PRTCSS_RTC_DAY0);
-
- davinci_rtcss_calendar_wait(davinci_rtc);
- rtcss_write(davinci_rtc, (days & 0xFF00) >> 8, PRTCSS_RTC_DAY1);
-
- rtc_cctrl = rtcss_read(davinci_rtc, PRTCSS_RTC_CCTRL);
- rtc_cctrl |= PRTCSS_RTC_CCTRL_CAEN;
- rtcss_write(davinci_rtc, rtc_cctrl, PRTCSS_RTC_CCTRL);
-
- spin_unlock_irqrestore(&davinci_rtc_lock, flags);
-
- return 0;
-}
-
-static int davinci_rtc_alarm_irq_enable(struct device *dev,
- unsigned int enabled)
-{
- struct davinci_rtc *davinci_rtc = dev_get_drvdata(dev);
- unsigned long flags;
- u8 rtc_cctrl = rtcss_read(davinci_rtc, PRTCSS_RTC_CCTRL);
-
- spin_lock_irqsave(&davinci_rtc_lock, flags);
-
- if (enabled)
- rtc_cctrl |= PRTCSS_RTC_CCTRL_DAEN |
- PRTCSS_RTC_CCTRL_HAEN |
- PRTCSS_RTC_CCTRL_MAEN |
- PRTCSS_RTC_CCTRL_ALMFLG |
- PRTCSS_RTC_CCTRL_AIEN;
- else
- rtc_cctrl &= ~PRTCSS_RTC_CCTRL_AIEN;
-
- davinci_rtcss_calendar_wait(davinci_rtc);
- rtcss_write(davinci_rtc, rtc_cctrl, PRTCSS_RTC_CCTRL);
-
- spin_unlock_irqrestore(&davinci_rtc_lock, flags);
-
- return 0;
-}
-
-static int davinci_rtc_read_alarm(struct device *dev, struct rtc_wkalrm *alm)
-{
- struct davinci_rtc *davinci_rtc = dev_get_drvdata(dev);
- u16 days = 0;
- u8 day0, day1;
- unsigned long flags;
-
- alm->time.tm_sec = 0;
-
- spin_lock_irqsave(&davinci_rtc_lock, flags);
-
- davinci_rtcss_calendar_wait(davinci_rtc);
- alm->time.tm_min = bcd2bin(rtcss_read(davinci_rtc, PRTCSS_RTC_AMIN));
-
- davinci_rtcss_calendar_wait(davinci_rtc);
- alm->time.tm_hour = bcd2bin(rtcss_read(davinci_rtc, PRTCSS_RTC_AHOUR));
-
- davinci_rtcss_calendar_wait(davinci_rtc);
- day0 = rtcss_read(davinci_rtc, PRTCSS_RTC_ADAY0);
-
- davinci_rtcss_calendar_wait(davinci_rtc);
- day1 = rtcss_read(davinci_rtc, PRTCSS_RTC_ADAY1);
-
- spin_unlock_irqrestore(&davinci_rtc_lock, flags);
- days |= day1;
- days <<= 8;
- days |= day0;
-
- convertfromdays(days, &alm->time);
-
- alm->pending = !!(rtcss_read(davinci_rtc,
- PRTCSS_RTC_CCTRL) &
- PRTCSS_RTC_CCTRL_AIEN);
- alm->enabled = alm->pending && device_may_wakeup(dev);
-
- return 0;
-}
-
-static int davinci_rtc_set_alarm(struct device *dev, struct rtc_wkalrm *alm)
-{
- struct davinci_rtc *davinci_rtc = dev_get_drvdata(dev);
- unsigned long flags;
- u16 days;
-
- convert2days(&days, &alm->time);
-
- spin_lock_irqsave(&davinci_rtc_lock, flags);
-
- davinci_rtcss_calendar_wait(davinci_rtc);
- rtcss_write(davinci_rtc, bin2bcd(alm->time.tm_min), PRTCSS_RTC_AMIN);
-
- davinci_rtcss_calendar_wait(davinci_rtc);
- rtcss_write(davinci_rtc, bin2bcd(alm->time.tm_hour), PRTCSS_RTC_AHOUR);
-
- davinci_rtcss_calendar_wait(davinci_rtc);
- rtcss_write(davinci_rtc, days & 0xFF, PRTCSS_RTC_ADAY0);
-
- davinci_rtcss_calendar_wait(davinci_rtc);
- rtcss_write(davinci_rtc, (days & 0xFF00) >> 8, PRTCSS_RTC_ADAY1);
-
- spin_unlock_irqrestore(&davinci_rtc_lock, flags);
-
- return 0;
-}
-
-static const struct rtc_class_ops davinci_rtc_ops = {
- .ioctl = davinci_rtc_ioctl,
- .read_time = davinci_rtc_read_time,
- .set_time = davinci_rtc_set_time,
- .alarm_irq_enable = davinci_rtc_alarm_irq_enable,
- .read_alarm = davinci_rtc_read_alarm,
- .set_alarm = davinci_rtc_set_alarm,
-};
-
-static int __init davinci_rtc_probe(struct platform_device *pdev)
-{
- struct device *dev = &pdev->dev;
- struct davinci_rtc *davinci_rtc;
- int ret = 0;
-
- davinci_rtc = devm_kzalloc(&pdev->dev, sizeof(struct davinci_rtc), GFP_KERNEL);
- if (!davinci_rtc)
- return -ENOMEM;
-
- davinci_rtc->irq = platform_get_irq(pdev, 0);
- if (davinci_rtc->irq < 0)
- return davinci_rtc->irq;
-
- davinci_rtc->base = devm_platform_ioremap_resource(pdev, 0);
- if (IS_ERR(davinci_rtc->base))
- return PTR_ERR(davinci_rtc->base);
-
- platform_set_drvdata(pdev, davinci_rtc);
-
- davinci_rtc->rtc = devm_rtc_allocate_device(&pdev->dev);
- if (IS_ERR(davinci_rtc->rtc))
- return PTR_ERR(davinci_rtc->rtc);
-
- davinci_rtc->rtc->ops = &davinci_rtc_ops;
- davinci_rtc->rtc->range_min = RTC_TIMESTAMP_BEGIN_2000;
- davinci_rtc->rtc->range_max = RTC_TIMESTAMP_BEGIN_2000 + (1 << 16) * 86400ULL - 1;
-
- rtcif_write(davinci_rtc, PRTCIF_INTFLG_RTCSS, PRTCIF_INTFLG);
- rtcif_write(davinci_rtc, 0, PRTCIF_INTEN);
- rtcss_write(davinci_rtc, 0, PRTCSS_RTC_INTC_EXTENA1);
-
- rtcss_write(davinci_rtc, 0, PRTCSS_RTC_CTRL);
- rtcss_write(davinci_rtc, 0, PRTCSS_RTC_CCTRL);
-
- ret = devm_request_irq(dev, davinci_rtc->irq, davinci_rtc_interrupt,
- 0, "davinci_rtc", davinci_rtc);
- if (ret < 0) {
- dev_err(dev, "unable to register davinci RTC interrupt\n");
- return ret;
- }
-
- /* Enable interrupts */
- rtcif_write(davinci_rtc, PRTCIF_INTEN_RTCSS, PRTCIF_INTEN);
- rtcss_write(davinci_rtc, PRTCSS_RTC_INTC_EXTENA1_MASK,
- PRTCSS_RTC_INTC_EXTENA1);
-
- rtcss_write(davinci_rtc, PRTCSS_RTC_CCTRL_CAEN, PRTCSS_RTC_CCTRL);
-
- device_init_wakeup(&pdev->dev, 0);
-
- return devm_rtc_register_device(davinci_rtc->rtc);
-}
-
-static int __exit davinci_rtc_remove(struct platform_device *pdev)
-{
- struct davinci_rtc *davinci_rtc = platform_get_drvdata(pdev);
-
- device_init_wakeup(&pdev->dev, 0);
-
- rtcif_write(davinci_rtc, 0, PRTCIF_INTEN);
-
- return 0;
-}
-
-static struct platform_driver davinci_rtc_driver = {
- .remove = __exit_p(davinci_rtc_remove),
- .driver = {
- .name = "rtc_davinci",
- },
-};
-
-module_platform_driver_probe(davinci_rtc_driver, davinci_rtc_probe);
-
-MODULE_AUTHOR("Miguel Aguilar <miguel.aguilar@ridgerun.com>");
-MODULE_DESCRIPTION("Texas Instruments DaVinci PRTC Driver");
-MODULE_LICENSE("GPL");
diff --git a/drivers/rtc/rtc-ds1302.c b/drivers/rtc/rtc-ds1302.c
index 6d66ab5..ecc7d03 100644
--- a/drivers/rtc/rtc-ds1302.c
+++ b/drivers/rtc/rtc-ds1302.c
@@ -185,11 +185,6 @@ static int ds1302_probe(struct spi_device *spi)
return 0;
}
-static void ds1302_remove(struct spi_device *spi)
-{
- spi_set_drvdata(spi, NULL);
-}
-
#ifdef CONFIG_OF
static const struct of_device_id ds1302_dt_ids[] = {
{ .compatible = "maxim,ds1302", },
@@ -208,7 +203,6 @@ static struct spi_driver ds1302_driver = {
.driver.name = "rtc-ds1302",
.driver.of_match_table = of_match_ptr(ds1302_dt_ids),
.probe = ds1302_probe,
- .remove = ds1302_remove,
.id_table = ds1302_spi_ids,
};
diff --git a/drivers/rtc/rtc-ds1307.c b/drivers/rtc/rtc-ds1307.c
index d51565b..def9b7f 100644
--- a/drivers/rtc/rtc-ds1307.c
+++ b/drivers/rtc/rtc-ds1307.c
@@ -11,6 +11,7 @@
#include <linux/bcd.h>
#include <linux/i2c.h>
#include <linux/init.h>
+#include <linux/kstrtox.h>
#include <linux/mod_devicetable.h>
#include <linux/module.h>
#include <linux/property.h>
@@ -1218,8 +1219,7 @@ static ssize_t frequency_test_show(struct device *dev,
regmap_read(ds1307->regmap, M41TXX_REG_CONTROL, &ctrl_reg);
- return scnprintf(buf, PAGE_SIZE, (ctrl_reg & M41TXX_BIT_FT) ? "on\n" :
- "off\n");
+ return sysfs_emit(buf, (ctrl_reg & M41TXX_BIT_FT) ? "on\n" : "off\n");
}
static DEVICE_ATTR_RW(frequency_test);
diff --git a/drivers/rtc/rtc-ds1347.c b/drivers/rtc/rtc-ds1347.c
index 157bf52..a40c1a5 100644
--- a/drivers/rtc/rtc-ds1347.c
+++ b/drivers/rtc/rtc-ds1347.c
@@ -112,7 +112,7 @@ static int ds1347_set_time(struct device *dev, struct rtc_time *dt)
return err;
century = (dt->tm_year / 100) + 19;
- err = regmap_write(map, DS1347_CENTURY_REG, century);
+ err = regmap_write(map, DS1347_CENTURY_REG, bin2bcd(century));
if (err)
return err;
diff --git a/drivers/rtc/rtc-ds1742.c b/drivers/rtc/rtc-ds1742.c
index 13d45c6..a5026b0 100644
--- a/drivers/rtc/rtc-ds1742.c
+++ b/drivers/rtc/rtc-ds1742.c
@@ -158,8 +158,7 @@ static int ds1742_rtc_probe(struct platform_device *pdev)
if (!pdata)
return -ENOMEM;
- res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
- ioaddr = devm_ioremap_resource(&pdev->dev, res);
+ ioaddr = devm_platform_get_and_ioremap_resource(pdev, 0, &res);
if (IS_ERR(ioaddr))
return PTR_ERR(ioaddr);
diff --git a/drivers/rtc/rtc-efi.c b/drivers/rtc/rtc-efi.c
index 11850c2..e991ccc 100644
--- a/drivers/rtc/rtc-efi.c
+++ b/drivers/rtc/rtc-efi.c
@@ -271,6 +271,8 @@ static int __init efi_rtc_probe(struct platform_device *dev)
clear_bit(RTC_FEATURE_UPDATE_INTERRUPT, rtc->features);
set_bit(RTC_FEATURE_ALARM_WAKEUP_ONLY, rtc->features);
+ device_init_wakeup(&dev->dev, true);
+
return devm_rtc_register_device(rtc);
}
diff --git a/drivers/rtc/rtc-fsl-ftm-alarm.c b/drivers/rtc/rtc-fsl-ftm-alarm.c
index c0df49f..3d7c407 100644
--- a/drivers/rtc/rtc-fsl-ftm-alarm.c
+++ b/drivers/rtc/rtc-fsl-ftm-alarm.c
@@ -327,12 +327,7 @@ static struct platform_driver ftm_rtc_driver = {
},
};
-static int __init ftm_alarm_init(void)
-{
- return platform_driver_register(&ftm_rtc_driver);
-}
-
-device_initcall(ftm_alarm_init);
+module_platform_driver(ftm_rtc_driver);
MODULE_DESCRIPTION("NXP/Freescale FlexTimer alarm driver");
MODULE_AUTHOR("Biwen Li <biwen.li@nxp.com>");
diff --git a/drivers/rtc/rtc-isl12022.c b/drivers/rtc/rtc-isl12022.c
index ca677c4..a3b0de3 100644
--- a/drivers/rtc/rtc-isl12022.c
+++ b/drivers/rtc/rtc-isl12022.c
@@ -17,6 +17,7 @@
#include <linux/of.h>
#include <linux/of_device.h>
#include <linux/regmap.h>
+#include <linux/hwmon.h>
/* ISL register offsets */
#define ISL12022_REG_SC 0x00
@@ -30,6 +31,9 @@
#define ISL12022_REG_SR 0x07
#define ISL12022_REG_INT 0x08
+#define ISL12022_REG_BETA 0x0d
+#define ISL12022_REG_TEMP_L 0x28
+
/* ISL register bits */
#define ISL12022_HR_MIL (1 << 7) /* military or 24 hour time */
@@ -38,6 +42,7 @@
#define ISL12022_INT_WRTC (1 << 6)
+#define ISL12022_BETA_TSE (1 << 7)
static struct i2c_driver isl12022_driver;
@@ -46,6 +51,93 @@ struct isl12022 {
struct regmap *regmap;
};
+static umode_t isl12022_hwmon_is_visible(const void *data,
+ enum hwmon_sensor_types type,
+ u32 attr, int channel)
+{
+ if (type == hwmon_temp && attr == hwmon_temp_input)
+ return 0444;
+
+ return 0;
+}
+
+/*
+ * A user-initiated temperature conversion is not started by this function,
+ * so the temperature is updated once every ~60 seconds.
+ */
+static int isl12022_hwmon_read_temp(struct device *dev, long *mC)
+{
+ struct isl12022 *isl12022 = dev_get_drvdata(dev);
+ struct regmap *regmap = isl12022->regmap;
+ u8 temp_buf[2];
+ int temp, ret;
+
+ ret = regmap_bulk_read(regmap, ISL12022_REG_TEMP_L,
+ temp_buf, sizeof(temp_buf));
+ if (ret)
+ return ret;
+ /*
+ * Temperature is represented as a 10-bit number, unit half-Kelvins.
+ */
+ temp = (temp_buf[1] << 8) | temp_buf[0];
+ temp *= 500;
+ temp -= 273000;
+
+ *mC = temp;
+
+ return 0;
+}
+
+static int isl12022_hwmon_read(struct device *dev,
+ enum hwmon_sensor_types type,
+ u32 attr, int channel, long *val)
+{
+ if (type == hwmon_temp && attr == hwmon_temp_input)
+ return isl12022_hwmon_read_temp(dev, val);
+
+ return -EOPNOTSUPP;
+}
+
+static const struct hwmon_channel_info *isl12022_hwmon_info[] = {
+ HWMON_CHANNEL_INFO(temp, HWMON_T_INPUT),
+ NULL
+};
+
+static const struct hwmon_ops isl12022_hwmon_ops = {
+ .is_visible = isl12022_hwmon_is_visible,
+ .read = isl12022_hwmon_read,
+};
+
+static const struct hwmon_chip_info isl12022_hwmon_chip_info = {
+ .ops = &isl12022_hwmon_ops,
+ .info = isl12022_hwmon_info,
+};
+
+static void isl12022_hwmon_register(struct device *dev)
+{
+ struct isl12022 *isl12022;
+ struct device *hwmon;
+ int ret;
+
+ if (!IS_REACHABLE(CONFIG_HWMON))
+ return;
+
+ isl12022 = dev_get_drvdata(dev);
+
+ ret = regmap_update_bits(isl12022->regmap, ISL12022_REG_BETA,
+ ISL12022_BETA_TSE, ISL12022_BETA_TSE);
+ if (ret) {
+ dev_warn(dev, "unable to enable temperature sensor\n");
+ return;
+ }
+
+ hwmon = devm_hwmon_device_register_with_info(dev, "isl12022", isl12022,
+ &isl12022_hwmon_chip_info,
+ NULL);
+ if (IS_ERR(hwmon))
+ dev_warn(dev, "unable to register hwmon device: %pe\n", hwmon);
+}
+
/*
* In the routines that deal directly with the isl12022 hardware, we use
* rtc_time -- month 0-11, hour 0-23, yr = calendar year-epoch.
@@ -160,6 +252,8 @@ static int isl12022_probe(struct i2c_client *client)
return PTR_ERR(isl12022->regmap);
}
+ isl12022_hwmon_register(&client->dev);
+
isl12022->rtc = devm_rtc_allocate_device(&client->dev);
if (IS_ERR(isl12022->rtc))
return PTR_ERR(isl12022->rtc);
diff --git a/drivers/rtc/rtc-isl1208.c b/drivers/rtc/rtc-isl1208.c
index f448a52..73cc6aa 100644
--- a/drivers/rtc/rtc-isl1208.c
+++ b/drivers/rtc/rtc-isl1208.c
@@ -797,7 +797,7 @@ static int isl1208_setup_irq(struct i2c_client *client, int irq)
}
static int
-isl1208_probe(struct i2c_client *client, const struct i2c_device_id *id)
+isl1208_probe(struct i2c_client *client)
{
int rc = 0;
struct isl1208_state *isl1208;
@@ -821,6 +821,8 @@ isl1208_probe(struct i2c_client *client, const struct i2c_device_id *id)
if (!isl1208->config)
return -ENODEV;
} else {
+ const struct i2c_device_id *id = i2c_match_id(isl1208_id, client);
+
if (id->driver_data >= ISL_LAST_ID)
return -ENODEV;
isl1208->config = &isl1208_configs[id->driver_data];
@@ -906,7 +908,7 @@ static struct i2c_driver isl1208_driver = {
.name = "rtc-isl1208",
.of_match_table = of_match_ptr(isl1208_of_match),
},
- .probe = isl1208_probe,
+ .probe_new = isl1208_probe,
.id_table = isl1208_id,
};
diff --git a/drivers/rtc/rtc-m41t80.c b/drivers/rtc/rtc-m41t80.c
index e0b4d37..494052d 100644
--- a/drivers/rtc/rtc-m41t80.c
+++ b/drivers/rtc/rtc-m41t80.c
@@ -692,7 +692,7 @@ static void wdt_disable(void)
* @ppos: pointer to the position to write. No seeks allowed
*
* A write to a watchdog device is defined as a keepalive signal. Any
- * write of data will do, as we we don't define content meaning.
+ * write of data will do, as we don't define content meaning.
*/
static ssize_t wdt_write(struct file *file, const char __user *buf,
size_t count, loff_t *ppos)
@@ -876,8 +876,7 @@ static struct notifier_block wdt_notifier = {
*****************************************************************************
*/
-static int m41t80_probe(struct i2c_client *client,
- const struct i2c_device_id *id)
+static int m41t80_probe(struct i2c_client *client)
{
struct i2c_adapter *adapter = client->adapter;
int rc = 0;
@@ -897,11 +896,13 @@ static int m41t80_probe(struct i2c_client *client,
return -ENOMEM;
m41t80_data->client = client;
- if (client->dev.of_node)
+ if (client->dev.of_node) {
m41t80_data->features = (unsigned long)
of_device_get_match_data(&client->dev);
- else
+ } else {
+ const struct i2c_device_id *id = i2c_match_id(m41t80_id, client);
m41t80_data->features = id->driver_data;
+ }
i2c_set_clientdata(client, m41t80_data);
m41t80_data->rtc = devm_rtc_allocate_device(&client->dev);
@@ -1007,7 +1008,7 @@ static struct i2c_driver m41t80_driver = {
.of_match_table = of_match_ptr(m41t80_of_match),
.pm = &m41t80_pm,
},
- .probe = m41t80_probe,
+ .probe_new = m41t80_probe,
.remove = m41t80_remove,
.id_table = m41t80_id,
};
diff --git a/drivers/rtc/rtc-msc313.c b/drivers/rtc/rtc-msc313.c
index f3fde01..8d7737e 100644
--- a/drivers/rtc/rtc-msc313.c
+++ b/drivers/rtc/rtc-msc313.c
@@ -212,22 +212,12 @@ static int msc313_rtc_probe(struct platform_device *pdev)
return ret;
}
- clk = devm_clk_get(dev, NULL);
+ clk = devm_clk_get_enabled(dev, NULL);
if (IS_ERR(clk)) {
dev_err(dev, "No input reference clock\n");
return PTR_ERR(clk);
}
- ret = clk_prepare_enable(clk);
- if (ret) {
- dev_err(dev, "Failed to enable the reference clock, %d\n", ret);
- return ret;
- }
-
- ret = devm_add_action_or_reset(dev, (void (*) (void *))clk_disable_unprepare, clk);
- if (ret)
- return ret;
-
rate = clk_get_rate(clk);
writew(rate & 0xFFFF, priv->rtc_base + REG_RTC_FREQ_CW_L);
writew((rate >> 16) & 0xFFFF, priv->rtc_base + REG_RTC_FREQ_CW_H);
diff --git a/drivers/rtc/rtc-mxc_v2.c b/drivers/rtc/rtc-mxc_v2.c
index 5e03834..f6d2ad9 100644
--- a/drivers/rtc/rtc-mxc_v2.c
+++ b/drivers/rtc/rtc-mxc_v2.c
@@ -336,8 +336,10 @@ static int mxc_rtc_probe(struct platform_device *pdev)
}
pdata->rtc = devm_rtc_allocate_device(&pdev->dev);
- if (IS_ERR(pdata->rtc))
+ if (IS_ERR(pdata->rtc)) {
+ clk_disable_unprepare(pdata->clk);
return PTR_ERR(pdata->rtc);
+ }
pdata->rtc->ops = &mxc_rtc_ops;
pdata->rtc->range_max = U32_MAX;
diff --git a/drivers/rtc/rtc-nct3018y.c b/drivers/rtc/rtc-nct3018y.c
index d43acd3..0a3b14c 100644
--- a/drivers/rtc/rtc-nct3018y.c
+++ b/drivers/rtc/rtc-nct3018y.c
@@ -452,8 +452,7 @@ static const struct rtc_class_ops nct3018y_rtc_ops = {
.ioctl = nct3018y_ioctl,
};
-static int nct3018y_probe(struct i2c_client *client,
- const struct i2c_device_id *id)
+static int nct3018y_probe(struct i2c_client *client)
{
struct nct3018y *nct3018y;
int err, flags;
@@ -541,7 +540,7 @@ static struct i2c_driver nct3018y_driver = {
.name = "rtc-nct3018y",
.of_match_table = of_match_ptr(nct3018y_of_match),
},
- .probe = nct3018y_probe,
+ .probe_new = nct3018y_probe,
.id_table = nct3018y_id,
};
diff --git a/drivers/rtc/rtc-pcf2127.c b/drivers/rtc/rtc-pcf2127.c
index 63b275b..87f4fc9 100644
--- a/drivers/rtc/rtc-pcf2127.c
+++ b/drivers/rtc/rtc-pcf2127.c
@@ -885,9 +885,17 @@ static const struct regmap_bus pcf2127_i2c_regmap = {
static struct i2c_driver pcf2127_i2c_driver;
-static int pcf2127_i2c_probe(struct i2c_client *client,
- const struct i2c_device_id *id)
+static const struct i2c_device_id pcf2127_i2c_id[] = {
+ { "pcf2127", 1 },
+ { "pcf2129", 0 },
+ { "pca2129", 0 },
+ { }
+};
+MODULE_DEVICE_TABLE(i2c, pcf2127_i2c_id);
+
+static int pcf2127_i2c_probe(struct i2c_client *client)
{
+ const struct i2c_device_id *id = i2c_match_id(pcf2127_i2c_id, client);
struct regmap *regmap;
static const struct regmap_config config = {
.reg_bits = 8,
@@ -910,20 +918,12 @@ static int pcf2127_i2c_probe(struct i2c_client *client,
pcf2127_i2c_driver.driver.name, id->driver_data);
}
-static const struct i2c_device_id pcf2127_i2c_id[] = {
- { "pcf2127", 1 },
- { "pcf2129", 0 },
- { "pca2129", 0 },
- { }
-};
-MODULE_DEVICE_TABLE(i2c, pcf2127_i2c_id);
-
static struct i2c_driver pcf2127_i2c_driver = {
.driver = {
.name = "rtc-pcf2127-i2c",
.of_match_table = of_match_ptr(pcf2127_of_match),
},
- .probe = pcf2127_i2c_probe,
+ .probe_new = pcf2127_i2c_probe,
.id_table = pcf2127_i2c_id,
};
diff --git a/drivers/rtc/rtc-pcf85063.c b/drivers/rtc/rtc-pcf85063.c
index 0958919..754e039 100644
--- a/drivers/rtc/rtc-pcf85063.c
+++ b/drivers/rtc/rtc-pcf85063.c
@@ -169,10 +169,10 @@ static int pcf85063_rtc_read_alarm(struct device *dev, struct rtc_wkalrm *alrm)
if (ret)
return ret;
- alrm->time.tm_sec = bcd2bin(buf[0]);
- alrm->time.tm_min = bcd2bin(buf[1]);
- alrm->time.tm_hour = bcd2bin(buf[2]);
- alrm->time.tm_mday = bcd2bin(buf[3]);
+ alrm->time.tm_sec = bcd2bin(buf[0] & 0x7f);
+ alrm->time.tm_min = bcd2bin(buf[1] & 0x7f);
+ alrm->time.tm_hour = bcd2bin(buf[2] & 0x3f);
+ alrm->time.tm_mday = bcd2bin(buf[3] & 0x3f);
ret = regmap_read(pcf85063->regmap, PCF85063_REG_CTRL2, &val);
if (ret)
@@ -424,7 +424,7 @@ static int pcf85063_clkout_control(struct clk_hw *hw, bool enable)
unsigned int buf;
int ret;
- ret = regmap_read(pcf85063->regmap, PCF85063_REG_OFFSET, &buf);
+ ret = regmap_read(pcf85063->regmap, PCF85063_REG_CTRL2, &buf);
if (ret < 0)
return ret;
buf &= PCF85063_REG_CLKO_F_MASK;
diff --git a/drivers/rtc/rtc-pcf8523.c b/drivers/rtc/rtc-pcf8523.c
index 6174b3f..92de99f 100644
--- a/drivers/rtc/rtc-pcf8523.c
+++ b/drivers/rtc/rtc-pcf8523.c
@@ -99,24 +99,24 @@ static irqreturn_t pcf8523_irq(int irq, void *dev_id)
static int pcf8523_rtc_read_time(struct device *dev, struct rtc_time *tm)
{
struct pcf8523 *pcf8523 = dev_get_drvdata(dev);
- u8 regs[7];
+ u8 regs[10];
int err;
- err = regmap_bulk_read(pcf8523->regmap, PCF8523_REG_SECONDS, regs,
+ err = regmap_bulk_read(pcf8523->regmap, PCF8523_REG_CONTROL1, regs,
sizeof(regs));
if (err < 0)
return err;
- if (regs[0] & PCF8523_SECONDS_OS)
+ if ((regs[0] & PCF8523_CONTROL1_STOP) || (regs[3] & PCF8523_SECONDS_OS))
return -EINVAL;
- tm->tm_sec = bcd2bin(regs[0] & 0x7f);
- tm->tm_min = bcd2bin(regs[1] & 0x7f);
- tm->tm_hour = bcd2bin(regs[2] & 0x3f);
- tm->tm_mday = bcd2bin(regs[3] & 0x3f);
- tm->tm_wday = regs[4] & 0x7;
- tm->tm_mon = bcd2bin(regs[5] & 0x1f) - 1;
- tm->tm_year = bcd2bin(regs[6]) + 100;
+ tm->tm_sec = bcd2bin(regs[3] & 0x7f);
+ tm->tm_min = bcd2bin(regs[4] & 0x7f);
+ tm->tm_hour = bcd2bin(regs[5] & 0x3f);
+ tm->tm_mday = bcd2bin(regs[6] & 0x3f);
+ tm->tm_wday = regs[7] & 0x7;
+ tm->tm_mon = bcd2bin(regs[8] & 0x1f) - 1;
+ tm->tm_year = bcd2bin(regs[9]) + 100;
return 0;
}
diff --git a/drivers/rtc/rtc-pcf8563.c b/drivers/rtc/rtc-pcf8563.c
index 11fa978..0a7fd94 100644
--- a/drivers/rtc/rtc-pcf8563.c
+++ b/drivers/rtc/rtc-pcf8563.c
@@ -567,6 +567,8 @@ static int pcf8563_probe(struct i2c_client *client)
client->irq);
return err;
}
+ } else {
+ clear_bit(RTC_FEATURE_ALARM, pcf8563->rtc->features);
}
err = devm_rtc_register_device(pcf8563->rtc);
diff --git a/drivers/rtc/rtc-pic32.c b/drivers/rtc/rtc-pic32.c
index 7fb9145..fa351ac 100644
--- a/drivers/rtc/rtc-pic32.c
+++ b/drivers/rtc/rtc-pic32.c
@@ -324,16 +324,16 @@ static int pic32_rtc_probe(struct platform_device *pdev)
spin_lock_init(&pdata->alarm_lock);
+ pdata->rtc = devm_rtc_allocate_device(&pdev->dev);
+ if (IS_ERR(pdata->rtc))
+ return PTR_ERR(pdata->rtc);
+
clk_prepare_enable(pdata->clk);
pic32_rtc_enable(pdata, 1);
device_init_wakeup(&pdev->dev, 1);
- pdata->rtc = devm_rtc_allocate_device(&pdev->dev);
- if (IS_ERR(pdata->rtc))
- return PTR_ERR(pdata->rtc);
-
pdata->rtc->ops = &pic32_rtcops;
pdata->rtc->range_min = RTC_TIMESTAMP_BEGIN_2000;
pdata->rtc->range_max = RTC_TIMESTAMP_END_2099;
diff --git a/drivers/rtc/rtc-pm8xxx.c b/drivers/rtc/rtc-pm8xxx.c
index dc6d147..716e5d9 100644
--- a/drivers/rtc/rtc-pm8xxx.c
+++ b/drivers/rtc/rtc-pm8xxx.c
@@ -461,7 +461,6 @@ static const struct pm8xxx_rtc_regs pmk8350_regs = {
*/
static const struct of_device_id pm8xxx_id_table[] = {
{ .compatible = "qcom,pm8921-rtc", .data = &pm8921_regs },
- { .compatible = "qcom,pm8018-rtc", .data = &pm8921_regs },
{ .compatible = "qcom,pm8058-rtc", .data = &pm8058_regs },
{ .compatible = "qcom,pm8941-rtc", .data = &pm8941_regs },
{ .compatible = "qcom,pmk8350-rtc", .data = &pmk8350_regs },
diff --git a/drivers/rtc/rtc-rk808.c b/drivers/rtc/rtc-rk808.c
index e920da8..2d9bcb3 100644
--- a/drivers/rtc/rtc-rk808.c
+++ b/drivers/rtc/rtc-rk808.c
@@ -14,7 +14,6 @@
#include <linux/bcd.h>
#include <linux/mfd/rk808.h>
#include <linux/platform_device.h>
-#include <linux/i2c.h>
/* RTC_CTRL_REG bitfields */
#define BIT_RTC_CTRL_REG_STOP_RTC_M BIT(0)
@@ -51,7 +50,7 @@ struct rk_rtc_compat_reg {
};
struct rk808_rtc {
- struct rk808 *rk808;
+ struct regmap *regmap;
struct rtc_device *rtc;
struct rk_rtc_compat_reg *creg;
int irq;
@@ -97,12 +96,11 @@ static void gregorian_to_rockchip(struct rtc_time *tm)
static int rk808_rtc_readtime(struct device *dev, struct rtc_time *tm)
{
struct rk808_rtc *rk808_rtc = dev_get_drvdata(dev);
- struct rk808 *rk808 = rk808_rtc->rk808;
u8 rtc_data[NUM_TIME_REGS];
int ret;
/* Force an update of the shadowed registers right now */
- ret = regmap_update_bits(rk808->regmap, rk808_rtc->creg->ctrl_reg,
+ ret = regmap_update_bits(rk808_rtc->regmap, rk808_rtc->creg->ctrl_reg,
BIT_RTC_CTRL_REG_RTC_GET_TIME,
BIT_RTC_CTRL_REG_RTC_GET_TIME);
if (ret) {
@@ -116,7 +114,7 @@ static int rk808_rtc_readtime(struct device *dev, struct rtc_time *tm)
* 32khz. If we clear the GET_TIME bit here, the time of i2c transfer
* certainly more than 31.25us: 16 * 2.5us at 400kHz bus frequency.
*/
- ret = regmap_update_bits(rk808->regmap, rk808_rtc->creg->ctrl_reg,
+ ret = regmap_update_bits(rk808_rtc->regmap, rk808_rtc->creg->ctrl_reg,
BIT_RTC_CTRL_REG_RTC_GET_TIME,
0);
if (ret) {
@@ -124,7 +122,7 @@ static int rk808_rtc_readtime(struct device *dev, struct rtc_time *tm)
return ret;
}
- ret = regmap_bulk_read(rk808->regmap, rk808_rtc->creg->seconds_reg,
+ ret = regmap_bulk_read(rk808_rtc->regmap, rk808_rtc->creg->seconds_reg,
rtc_data, NUM_TIME_REGS);
if (ret) {
dev_err(dev, "Failed to bulk read rtc_data: %d\n", ret);
@@ -148,7 +146,6 @@ static int rk808_rtc_readtime(struct device *dev, struct rtc_time *tm)
static int rk808_rtc_set_time(struct device *dev, struct rtc_time *tm)
{
struct rk808_rtc *rk808_rtc = dev_get_drvdata(dev);
- struct rk808 *rk808 = rk808_rtc->rk808;
u8 rtc_data[NUM_TIME_REGS];
int ret;
@@ -163,7 +160,7 @@ static int rk808_rtc_set_time(struct device *dev, struct rtc_time *tm)
rtc_data[6] = bin2bcd(tm->tm_wday);
/* Stop RTC while updating the RTC registers */
- ret = regmap_update_bits(rk808->regmap, rk808_rtc->creg->ctrl_reg,
+ ret = regmap_update_bits(rk808_rtc->regmap, rk808_rtc->creg->ctrl_reg,
BIT_RTC_CTRL_REG_STOP_RTC_M,
BIT_RTC_CTRL_REG_STOP_RTC_M);
if (ret) {
@@ -171,14 +168,14 @@ static int rk808_rtc_set_time(struct device *dev, struct rtc_time *tm)
return ret;
}
- ret = regmap_bulk_write(rk808->regmap, rk808_rtc->creg->seconds_reg,
+ ret = regmap_bulk_write(rk808_rtc->regmap, rk808_rtc->creg->seconds_reg,
rtc_data, NUM_TIME_REGS);
if (ret) {
dev_err(dev, "Failed to bull write rtc_data: %d\n", ret);
return ret;
}
/* Start RTC again */
- ret = regmap_update_bits(rk808->regmap, rk808_rtc->creg->ctrl_reg,
+ ret = regmap_update_bits(rk808_rtc->regmap, rk808_rtc->creg->ctrl_reg,
BIT_RTC_CTRL_REG_STOP_RTC_M, 0);
if (ret) {
dev_err(dev, "Failed to update RTC control: %d\n", ret);
@@ -191,12 +188,11 @@ static int rk808_rtc_set_time(struct device *dev, struct rtc_time *tm)
static int rk808_rtc_readalarm(struct device *dev, struct rtc_wkalrm *alrm)
{
struct rk808_rtc *rk808_rtc = dev_get_drvdata(dev);
- struct rk808 *rk808 = rk808_rtc->rk808;
u8 alrm_data[NUM_ALARM_REGS];
uint32_t int_reg;
int ret;
- ret = regmap_bulk_read(rk808->regmap,
+ ret = regmap_bulk_read(rk808_rtc->regmap,
rk808_rtc->creg->alarm_seconds_reg,
alrm_data, NUM_ALARM_REGS);
if (ret) {
@@ -212,7 +208,7 @@ static int rk808_rtc_readalarm(struct device *dev, struct rtc_wkalrm *alrm)
alrm->time.tm_year = (bcd2bin(alrm_data[5] & YEARS_REG_MSK)) + 100;
rockchip_to_gregorian(&alrm->time);
- ret = regmap_read(rk808->regmap, rk808_rtc->creg->int_reg, &int_reg);
+ ret = regmap_read(rk808_rtc->regmap, rk808_rtc->creg->int_reg, &int_reg);
if (ret) {
dev_err(dev, "Failed to read RTC INT REG: %d\n", ret);
return ret;
@@ -228,10 +224,9 @@ static int rk808_rtc_readalarm(struct device *dev, struct rtc_wkalrm *alrm)
static int rk808_rtc_stop_alarm(struct rk808_rtc *rk808_rtc)
{
- struct rk808 *rk808 = rk808_rtc->rk808;
int ret;
- ret = regmap_update_bits(rk808->regmap, rk808_rtc->creg->int_reg,
+ ret = regmap_update_bits(rk808_rtc->regmap, rk808_rtc->creg->int_reg,
BIT_RTC_INTERRUPTS_REG_IT_ALARM_M, 0);
return ret;
@@ -239,10 +234,9 @@ static int rk808_rtc_stop_alarm(struct rk808_rtc *rk808_rtc)
static int rk808_rtc_start_alarm(struct rk808_rtc *rk808_rtc)
{
- struct rk808 *rk808 = rk808_rtc->rk808;
int ret;
- ret = regmap_update_bits(rk808->regmap, rk808_rtc->creg->int_reg,
+ ret = regmap_update_bits(rk808_rtc->regmap, rk808_rtc->creg->int_reg,
BIT_RTC_INTERRUPTS_REG_IT_ALARM_M,
BIT_RTC_INTERRUPTS_REG_IT_ALARM_M);
@@ -252,7 +246,6 @@ static int rk808_rtc_start_alarm(struct rk808_rtc *rk808_rtc)
static int rk808_rtc_setalarm(struct device *dev, struct rtc_wkalrm *alrm)
{
struct rk808_rtc *rk808_rtc = dev_get_drvdata(dev);
- struct rk808 *rk808 = rk808_rtc->rk808;
u8 alrm_data[NUM_ALARM_REGS];
int ret;
@@ -272,7 +265,7 @@ static int rk808_rtc_setalarm(struct device *dev, struct rtc_wkalrm *alrm)
alrm_data[4] = bin2bcd(alrm->time.tm_mon + 1);
alrm_data[5] = bin2bcd(alrm->time.tm_year - 100);
- ret = regmap_bulk_write(rk808->regmap,
+ ret = regmap_bulk_write(rk808_rtc->regmap,
rk808_rtc->creg->alarm_seconds_reg,
alrm_data, NUM_ALARM_REGS);
if (ret) {
@@ -313,20 +306,18 @@ static int rk808_rtc_alarm_irq_enable(struct device *dev,
static irqreturn_t rk808_alarm_irq(int irq, void *data)
{
struct rk808_rtc *rk808_rtc = data;
- struct rk808 *rk808 = rk808_rtc->rk808;
- struct i2c_client *client = rk808->i2c;
int ret;
- ret = regmap_write(rk808->regmap, rk808_rtc->creg->status_reg,
+ ret = regmap_write(rk808_rtc->regmap, rk808_rtc->creg->status_reg,
RTC_STATUS_MASK);
if (ret) {
- dev_err(&client->dev,
+ dev_err(&rk808_rtc->rtc->dev,
"%s:Failed to update RTC status: %d\n", __func__, ret);
return ret;
}
rtc_update_irq(rk808_rtc->rtc, 1, RTC_IRQF | RTC_AF);
- dev_dbg(&client->dev,
+ dev_dbg(&rk808_rtc->rtc->dev,
"%s:irq=%d\n", __func__, irq);
return IRQ_HANDLED;
}
@@ -404,10 +395,12 @@ static int rk808_rtc_probe(struct platform_device *pdev)
break;
}
platform_set_drvdata(pdev, rk808_rtc);
- rk808_rtc->rk808 = rk808;
+ rk808_rtc->regmap = dev_get_regmap(pdev->dev.parent, NULL);
+ if (!rk808_rtc->regmap)
+ return -ENODEV;
/* start rtc running by default, and use shadowed timer. */
- ret = regmap_update_bits(rk808->regmap, rk808_rtc->creg->ctrl_reg,
+ ret = regmap_update_bits(rk808_rtc->regmap, rk808_rtc->creg->ctrl_reg,
BIT_RTC_CTRL_REG_STOP_RTC_M |
BIT_RTC_CTRL_REG_RTC_READSEL_M,
BIT_RTC_CTRL_REG_RTC_READSEL_M);
@@ -417,7 +410,7 @@ static int rk808_rtc_probe(struct platform_device *pdev)
return ret;
}
- ret = regmap_write(rk808->regmap, rk808_rtc->creg->status_reg,
+ ret = regmap_write(rk808_rtc->regmap, rk808_rtc->creg->status_reg,
RTC_STATUS_MASK);
if (ret) {
dev_err(&pdev->dev,
diff --git a/drivers/rtc/rtc-rs5c313.c b/drivers/rtc/rtc-rs5c313.c
index e98f85f..712a08e 100644
--- a/drivers/rtc/rtc-rs5c313.c
+++ b/drivers/rtc/rtc-rs5c313.c
@@ -2,7 +2,7 @@
* Ricoh RS5C313 RTC device/driver
* Copyright (C) 2007 Nobuhiro Iwamatsu
*
- * 2005-09-19 modifed by kogiidena
+ * 2005-09-19 modified by kogiidena
*
* Based on the old drivers/char/rs5c313_rtc.c by:
* Copyright (C) 2000 Philipp Rumpf <prumpf@tux.org>
@@ -36,7 +36,7 @@
* 1.11a Daniele Bellucci: Audit create_proc_read_entry in rtc_init
* 1.12 Venkatesh Pallipadi: Hooks for emulating rtc on HPET base-timer
* CONFIG_HPET_EMULATE_RTC
- * 1.13 Nobuhiro Iwamatsu: Updata driver.
+ * 1.13 Nobuhiro Iwamatsu: Update driver.
*/
#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
@@ -280,7 +280,7 @@ static int rs5c313_rtc_set_time(struct device *dev, struct rtc_time *tm)
while (1) {
RS5C313_CEENABLE; /* CE:H */
- /* Initiatlize control reg. 24 hour */
+ /* Initialize control reg. 24 hour */
rs5c313_write_cntreg(0x04);
if (!(rs5c313_read_cntreg() & RS5C313_CNTREG_ADJ_BSY))
diff --git a/drivers/rtc/rtc-rs5c372.c b/drivers/rtc/rtc-rs5c372.c
index 9562c47..b4c5d01 100644
--- a/drivers/rtc/rtc-rs5c372.c
+++ b/drivers/rtc/rtc-rs5c372.c
@@ -150,7 +150,7 @@ static int rs5c_get_regs(struct rs5c372 *rs5c)
* least 80219 chips; this works around that bug.
*
* The third method on the other hand doesn't work for the SMBus-only
- * configurations, so we use the the first method there, stripping off
+ * configurations, so we use the first method there, stripping off
* the extra register in the process.
*/
if (rs5c->smbus) {
@@ -791,8 +791,7 @@ static int rs5c_oscillator_setup(struct rs5c372 *rs5c372)
return 0;
}
-static int rs5c372_probe(struct i2c_client *client,
- const struct i2c_device_id *id)
+static int rs5c372_probe(struct i2c_client *client)
{
int err = 0;
int smbus_mode = 0;
@@ -826,11 +825,13 @@ static int rs5c372_probe(struct i2c_client *client,
rs5c372->client = client;
i2c_set_clientdata(client, rs5c372);
- if (client->dev.of_node)
+ if (client->dev.of_node) {
rs5c372->type = (enum rtc_type)
of_device_get_match_data(&client->dev);
- else
+ } else {
+ const struct i2c_device_id *id = i2c_match_id(rs5c372_id, client);
rs5c372->type = id->driver_data;
+ }
/* we read registers 0x0f then 0x00-0x0f; skip the first one */
rs5c372->regs = &rs5c372->buf[1];
@@ -920,7 +921,7 @@ static struct i2c_driver rs5c372_driver = {
.name = "rtc-rs5c372",
.of_match_table = of_match_ptr(rs5c372_of_match),
},
- .probe = rs5c372_probe,
+ .probe_new = rs5c372_probe,
.remove = rs5c372_remove,
.id_table = rs5c372_id,
};
diff --git a/drivers/rtc/rtc-rv3028.c b/drivers/rtc/rtc-rv3028.c
index dd170e3..b0099e2 100644
--- a/drivers/rtc/rtc-rv3028.c
+++ b/drivers/rtc/rtc-rv3028.c
@@ -902,9 +902,20 @@ static int rv3028_probe(struct i2c_client *client)
return PTR_ERR(rv3028->rtc);
if (client->irq > 0) {
+ unsigned long flags;
+
+ /*
+ * If flags = 0, devm_request_threaded_irq() will use IRQ flags
+ * obtained from device tree.
+ */
+ if (dev_fwnode(&client->dev))
+ flags = 0;
+ else
+ flags = IRQF_TRIGGER_LOW;
+
ret = devm_request_threaded_irq(&client->dev, client->irq,
NULL, rv3028_handle_irq,
- IRQF_TRIGGER_LOW | IRQF_ONESHOT,
+ flags | IRQF_ONESHOT,
"rv3028", rv3028);
if (ret) {
dev_warn(&client->dev, "unable to request IRQ, alarms disabled\n");
diff --git a/drivers/rtc/rtc-rv3029c2.c b/drivers/rtc/rtc-rv3029c2.c
index eb483a3..e4fdd47 100644
--- a/drivers/rtc/rtc-rv3029c2.c
+++ b/drivers/rtc/rtc-rv3029c2.c
@@ -17,6 +17,7 @@
#include <linux/of.h>
#include <linux/hwmon.h>
#include <linux/hwmon-sysfs.h>
+#include <linux/kstrtox.h>
#include <linux/regmap.h>
/* Register map */
diff --git a/drivers/rtc/rtc-rv8803.c b/drivers/rtc/rtc-rv8803.c
index 3527a05..b581b6d 100644
--- a/drivers/rtc/rtc-rv8803.c
+++ b/drivers/rtc/rtc-rv8803.c
@@ -576,8 +576,16 @@ static int rv8803_regs_configure(struct rv8803_data *rv8803)
return 0;
}
-static int rv8803_probe(struct i2c_client *client,
- const struct i2c_device_id *id)
+static const struct i2c_device_id rv8803_id[] = {
+ { "rv8803", rv_8803 },
+ { "rv8804", rx_8804 },
+ { "rx8803", rx_8803 },
+ { "rx8900", rx_8900 },
+ { }
+};
+MODULE_DEVICE_TABLE(i2c, rv8803_id);
+
+static int rv8803_probe(struct i2c_client *client)
{
struct i2c_adapter *adapter = client->adapter;
struct rv8803_data *rv8803;
@@ -605,11 +613,14 @@ static int rv8803_probe(struct i2c_client *client,
mutex_init(&rv8803->flags_lock);
rv8803->client = client;
- if (client->dev.of_node)
+ if (client->dev.of_node) {
rv8803->type = (enum rv8803_type)
of_device_get_match_data(&client->dev);
- else
+ } else {
+ const struct i2c_device_id *id = i2c_match_id(rv8803_id, client);
+
rv8803->type = id->driver_data;
+ }
i2c_set_clientdata(client, rv8803);
flags = rv8803_read_reg(client, RV8803_FLAG);
@@ -666,15 +677,6 @@ static int rv8803_probe(struct i2c_client *client,
return 0;
}
-static const struct i2c_device_id rv8803_id[] = {
- { "rv8803", rv_8803 },
- { "rv8804", rx_8804 },
- { "rx8803", rx_8803 },
- { "rx8900", rx_8900 },
- { }
-};
-MODULE_DEVICE_TABLE(i2c, rv8803_id);
-
static const __maybe_unused struct of_device_id rv8803_of_match[] = {
{
.compatible = "microcrystal,rv8803",
@@ -701,7 +703,7 @@ static struct i2c_driver rv8803_driver = {
.name = "rtc-rv8803",
.of_match_table = of_match_ptr(rv8803_of_match),
},
- .probe = rv8803_probe,
+ .probe_new = rv8803_probe,
.id_table = rv8803_id,
};
module_i2c_driver(rv8803_driver);
diff --git a/drivers/rtc/rtc-rx6110.c b/drivers/rtc/rtc-rx6110.c
index cc63455..76a4983 100644
--- a/drivers/rtc/rtc-rx6110.c
+++ b/drivers/rtc/rtc-rx6110.c
@@ -376,7 +376,7 @@ static const struct spi_device_id rx6110_spi_id[] = {
};
MODULE_DEVICE_TABLE(spi, rx6110_spi_id);
-static const struct of_device_id rx6110_spi_of_match[] = {
+static const __maybe_unused struct of_device_id rx6110_spi_of_match[] = {
{ .compatible = "epson,rx6110" },
{ },
};
diff --git a/drivers/rtc/rtc-rx8025.c b/drivers/rtc/rtc-rx8025.c
index dde86f3..331c20d 100644
--- a/drivers/rtc/rtc-rx8025.c
+++ b/drivers/rtc/rtc-rx8025.c
@@ -19,6 +19,7 @@
#include <linux/bitops.h>
#include <linux/i2c.h>
#include <linux/kernel.h>
+#include <linux/kstrtox.h>
#include <linux/module.h>
#include <linux/rtc.h>
@@ -519,9 +520,9 @@ static const struct attribute_group rx8025_attr_group = {
.attrs = rx8025_attrs,
};
-static int rx8025_probe(struct i2c_client *client,
- const struct i2c_device_id *id)
+static int rx8025_probe(struct i2c_client *client)
{
+ const struct i2c_device_id *id = i2c_match_id(rx8025_id, client);
struct i2c_adapter *adapter = client->adapter;
struct rx8025_data *rx8025;
int err = 0;
@@ -580,7 +581,7 @@ static struct i2c_driver rx8025_driver = {
.driver = {
.name = "rtc-rx8025",
},
- .probe = rx8025_probe,
+ .probe_new = rx8025_probe,
.id_table = rx8025_id,
};
diff --git a/drivers/rtc/rtc-rzn1.c b/drivers/rtc/rtc-rzn1.c
index ac78879..0d36bc5 100644
--- a/drivers/rtc/rtc-rzn1.c
+++ b/drivers/rtc/rtc-rzn1.c
@@ -355,7 +355,9 @@ static int rzn1_rtc_probe(struct platform_device *pdev)
set_bit(RTC_FEATURE_ALARM_RES_MINUTE, rtc->rtcdev->features);
clear_bit(RTC_FEATURE_UPDATE_INTERRUPT, rtc->rtcdev->features);
- devm_pm_runtime_enable(&pdev->dev);
+ ret = devm_pm_runtime_enable(&pdev->dev);
+ if (ret < 0)
+ return ret;
ret = pm_runtime_resume_and_get(&pdev->dev);
if (ret < 0)
return ret;
diff --git a/drivers/rtc/rtc-s35390a.c b/drivers/rtc/rtc-s35390a.c
index 81d97b1..b18daaf 100644
--- a/drivers/rtc/rtc-s35390a.c
+++ b/drivers/rtc/rtc-s35390a.c
@@ -211,7 +211,7 @@ static int s35390a_rtc_set_time(struct device *dev, struct rtc_time *tm)
{
struct i2c_client *client = to_i2c_client(dev);
struct s35390a *s35390a = i2c_get_clientdata(client);
- int i, err;
+ int i;
char buf[7], status;
dev_dbg(&client->dev, "%s: tm is secs=%d, mins=%d, hours=%d mday=%d, "
@@ -234,9 +234,7 @@ static int s35390a_rtc_set_time(struct device *dev, struct rtc_time *tm)
for (i = 0; i < 7; ++i)
buf[i] = bitrev8(buf[i]);
- err = s35390a_set_reg(s35390a, S35390A_CMD_TIME1, buf, sizeof(buf));
-
- return err;
+ return s35390a_set_reg(s35390a, S35390A_CMD_TIME1, buf, sizeof(buf));
}
static int s35390a_rtc_read_time(struct device *dev, struct rtc_time *tm)
diff --git a/drivers/rtc/rtc-s3c.c b/drivers/rtc/rtc-s3c.c
index db529733..8fc5efd 100644
--- a/drivers/rtc/rtc-s3c.c
+++ b/drivers/rtc/rtc-s3c.c
@@ -429,14 +429,9 @@ static int s3c_rtc_probe(struct platform_device *pdev)
return PTR_ERR(info->base);
info->rtc_clk = devm_clk_get(&pdev->dev, "rtc");
- if (IS_ERR(info->rtc_clk)) {
- ret = PTR_ERR(info->rtc_clk);
- if (ret != -EPROBE_DEFER)
- dev_err(&pdev->dev, "failed to find rtc clock\n");
- else
- dev_dbg(&pdev->dev, "probe deferred due to missing rtc clk\n");
- return ret;
- }
+ if (IS_ERR(info->rtc_clk))
+ return dev_err_probe(&pdev->dev, PTR_ERR(info->rtc_clk),
+ "failed to find rtc clock\n");
ret = clk_prepare_enable(info->rtc_clk);
if (ret)
return ret;
diff --git a/drivers/rtc/rtc-snvs.c b/drivers/rtc/rtc-snvs.c
index bd929b0..d82acf1 100644
--- a/drivers/rtc/rtc-snvs.c
+++ b/drivers/rtc/rtc-snvs.c
@@ -32,6 +32,14 @@
#define SNVS_LPPGDR_INIT 0x41736166
#define CNTR_TO_SECS_SH 15
+/* The maximum RTC clock cycles that are allowed to pass between two
+ * consecutive clock counter register reads. If the values are corrupted a
+ * bigger difference is expected. The RTC frequency is 32kHz. With 320 cycles
+ * we end at 10ms which should be enough for most cases. If it once takes
+ * longer than expected we do a retry.
+ */
+#define MAX_RTC_READ_DIFF_CYCLES 320
+
struct snvs_rtc_data {
struct rtc_device *rtc;
struct regmap *regmap;
@@ -56,6 +64,7 @@ static u64 rtc_read_lpsrt(struct snvs_rtc_data *data)
static u32 rtc_read_lp_counter(struct snvs_rtc_data *data)
{
u64 read1, read2;
+ s64 diff;
unsigned int timeout = 100;
/* As expected, the registers might update between the read of the LSB
@@ -66,7 +75,8 @@ static u32 rtc_read_lp_counter(struct snvs_rtc_data *data)
do {
read2 = read1;
read1 = rtc_read_lpsrt(data);
- } while (read1 != read2 && --timeout);
+ diff = read1 - read2;
+ } while (((diff < 0) || (diff > MAX_RTC_READ_DIFF_CYCLES)) && --timeout);
if (!timeout)
dev_err(&data->rtc->dev, "Timeout trying to get valid LPSRT Counter read\n");
@@ -78,13 +88,15 @@ static u32 rtc_read_lp_counter(struct snvs_rtc_data *data)
static int rtc_read_lp_counter_lsb(struct snvs_rtc_data *data, u32 *lsb)
{
u32 count1, count2;
+ s32 diff;
unsigned int timeout = 100;
regmap_read(data->regmap, data->offset + SNVS_LPSRTCLR, &count1);
do {
count2 = count1;
regmap_read(data->regmap, data->offset + SNVS_LPSRTCLR, &count1);
- } while (count1 != count2 && --timeout);
+ diff = count1 - count2;
+ } while (((diff < 0) || (diff > MAX_RTC_READ_DIFF_CYCLES)) && --timeout);
if (!timeout) {
dev_err(&data->rtc->dev, "Timeout trying to get valid LPSRT Counter read\n");
return -ETIMEDOUT;
diff --git a/drivers/rtc/rtc-st-lpc.c b/drivers/rtc/rtc-st-lpc.c
index bdb20f6..0f8e423 100644
--- a/drivers/rtc/rtc-st-lpc.c
+++ b/drivers/rtc/rtc-st-lpc.c
@@ -238,6 +238,7 @@ static int st_rtc_probe(struct platform_device *pdev)
rtc->clkrate = clk_get_rate(rtc->clk);
if (!rtc->clkrate) {
+ clk_disable_unprepare(rtc->clk);
dev_err(&pdev->dev, "Unable to fetch clock rate\n");
return -EINVAL;
}
diff --git a/drivers/rtc/sysfs.c b/drivers/rtc/sysfs.c
index 00f1945b..e3062c4 100644
--- a/drivers/rtc/sysfs.c
+++ b/drivers/rtc/sysfs.c
@@ -6,6 +6,7 @@
* Author: Alessandro Zummo <a.zummo@towertech.it>
*/
+#include <linux/kstrtox.h>
#include <linux/module.h>
#include <linux/rtc.h>
diff --git a/drivers/soc/mediatek/mtk-pm-domains.c b/drivers/soc/mediatek/mtk-pm-domains.c
index 09e3c38..474b272 100644
--- a/drivers/soc/mediatek/mtk-pm-domains.c
+++ b/drivers/soc/mediatek/mtk-pm-domains.c
@@ -275,9 +275,9 @@ static int scpsys_power_off(struct generic_pm_domain *genpd)
clk_bulk_disable_unprepare(pd->num_subsys_clks, pd->subsys_clks);
/* subsys power off */
- regmap_clear_bits(scpsys->base, pd->data->ctl_offs, PWR_RST_B_BIT);
regmap_set_bits(scpsys->base, pd->data->ctl_offs, PWR_ISO_BIT);
regmap_set_bits(scpsys->base, pd->data->ctl_offs, PWR_CLK_DIS_BIT);
+ regmap_clear_bits(scpsys->base, pd->data->ctl_offs, PWR_RST_B_BIT);
regmap_clear_bits(scpsys->base, pd->data->ctl_offs, PWR_ON_2ND_BIT);
regmap_clear_bits(scpsys->base, pd->data->ctl_offs, PWR_ON_BIT);
diff --git a/drivers/soc/tegra/Kconfig b/drivers/soc/tegra/Kconfig
index 5f53242..3658fb0 100644
--- a/drivers/soc/tegra/Kconfig
+++ b/drivers/soc/tegra/Kconfig
@@ -95,6 +95,7 @@
config ARCH_TEGRA_186_SOC
bool "NVIDIA Tegra186 SoC"
+ depends on !CPU_BIG_ENDIAN
select MAILBOX
select TEGRA_BPMP
select TEGRA_HSP_MBOX
@@ -110,6 +111,7 @@
config ARCH_TEGRA_194_SOC
bool "NVIDIA Tegra194 SoC"
+ depends on !CPU_BIG_ENDIAN
select MAILBOX
select PINCTRL_TEGRA194
select TEGRA_BPMP
@@ -121,6 +123,7 @@
config ARCH_TEGRA_234_SOC
bool "NVIDIA Tegra234 SoC"
+ depends on !CPU_BIG_ENDIAN
select MAILBOX
select TEGRA_BPMP
select TEGRA_HSP_MBOX
diff --git a/drivers/soundwire/Makefile b/drivers/soundwire/Makefile
index 9867767..ca97414 100644
--- a/drivers/soundwire/Makefile
+++ b/drivers/soundwire/Makefile
@@ -20,7 +20,7 @@
obj-$(CONFIG_SOUNDWIRE_CADENCE) += soundwire-cadence.o
#Intel driver
-soundwire-intel-y := intel.o intel_init.o dmi-quirks.o
+soundwire-intel-y := intel.o intel_auxdevice.o intel_init.o dmi-quirks.o
obj-$(CONFIG_SOUNDWIRE_INTEL) += soundwire-intel.o
#Qualcomm driver
diff --git a/drivers/soundwire/cadence_master.c b/drivers/soundwire/cadence_master.c
index 93929f1..a1de363 100644
--- a/drivers/soundwire/cadence_master.c
+++ b/drivers/soundwire/cadence_master.c
@@ -1707,47 +1707,45 @@ int cdns_set_sdw_stream(struct snd_soc_dai *dai,
void *stream, int direction)
{
struct sdw_cdns *cdns = snd_soc_dai_get_drvdata(dai);
- struct sdw_cdns_dma_data *dma;
+ struct sdw_cdns_dai_runtime *dai_runtime;
+
+ dai_runtime = cdns->dai_runtime_array[dai->id];
if (stream) {
/* first paranoia check */
- if (direction == SNDRV_PCM_STREAM_PLAYBACK)
- dma = dai->playback_dma_data;
- else
- dma = dai->capture_dma_data;
-
- if (dma) {
+ if (dai_runtime) {
dev_err(dai->dev,
- "dma_data already allocated for dai %s\n",
+ "dai_runtime already allocated for dai %s\n",
dai->name);
return -EINVAL;
}
- /* allocate and set dma info */
- dma = kzalloc(sizeof(*dma), GFP_KERNEL);
- if (!dma)
+ /* allocate and set dai_runtime info */
+ dai_runtime = kzalloc(sizeof(*dai_runtime), GFP_KERNEL);
+ if (!dai_runtime)
return -ENOMEM;
- dma->stream_type = SDW_STREAM_PCM;
+ dai_runtime->stream_type = SDW_STREAM_PCM;
- dma->bus = &cdns->bus;
- dma->link_id = cdns->instance;
+ dai_runtime->bus = &cdns->bus;
+ dai_runtime->link_id = cdns->instance;
- dma->stream = stream;
+ dai_runtime->stream = stream;
+ dai_runtime->direction = direction;
- if (direction == SNDRV_PCM_STREAM_PLAYBACK)
- dai->playback_dma_data = dma;
- else
- dai->capture_dma_data = dma;
+ cdns->dai_runtime_array[dai->id] = dai_runtime;
} else {
- /* for NULL stream we release allocated dma_data */
- if (direction == SNDRV_PCM_STREAM_PLAYBACK) {
- kfree(dai->playback_dma_data);
- dai->playback_dma_data = NULL;
- } else {
- kfree(dai->capture_dma_data);
- dai->capture_dma_data = NULL;
+ /* second paranoia check */
+ if (!dai_runtime) {
+ dev_err(dai->dev,
+ "dai_runtime not allocated for dai %s\n",
+ dai->name);
+ return -EINVAL;
}
+
+ /* for NULL stream we release allocated dai_runtime */
+ kfree(dai_runtime);
+ cdns->dai_runtime_array[dai->id] = NULL;
}
return 0;
}
diff --git a/drivers/soundwire/cadence_master.h b/drivers/soundwire/cadence_master.h
index ca9e805..0434d70 100644
--- a/drivers/soundwire/cadence_master.h
+++ b/drivers/soundwire/cadence_master.h
@@ -70,7 +70,7 @@ struct sdw_cdns_stream_config {
};
/**
- * struct sdw_cdns_dma_data: Cadence DMA data
+ * struct sdw_cdns_dai_runtime: Cadence DAI runtime data
*
* @name: SoundWire stream name
* @stream: stream runtime
@@ -81,8 +81,9 @@ struct sdw_cdns_stream_config {
* @hw_params: hw_params to be applied in .prepare step
* @suspended: status set when suspended, to be used in .prepare
* @paused: status set in .trigger, to be used in suspend
+ * @direction: stream direction
*/
-struct sdw_cdns_dma_data {
+struct sdw_cdns_dai_runtime {
char *name;
struct sdw_stream_runtime *stream;
struct sdw_cdns_pdi *pdi;
@@ -92,6 +93,7 @@ struct sdw_cdns_dma_data {
struct snd_pcm_hw_params *hw_params;
bool suspended;
bool paused;
+ int direction;
};
/**
@@ -108,6 +110,7 @@ struct sdw_cdns_dma_data {
* @registers: Cadence registers
* @link_up: Link status
* @msg_count: Messages sent on bus
+ * @dai_runtime_array: runtime context for each allocated DAI.
*/
struct sdw_cdns {
struct device *dev;
@@ -135,6 +138,8 @@ struct sdw_cdns {
struct work_struct work;
struct list_head list;
+
+ struct sdw_cdns_dai_runtime **dai_runtime_array;
};
#define bus_to_cdns(_bus) container_of(_bus, struct sdw_cdns, bus)
diff --git a/drivers/soundwire/dmi-quirks.c b/drivers/soundwire/dmi-quirks.c
index f81cdd8..7969881 100644
--- a/drivers/soundwire/dmi-quirks.c
+++ b/drivers/soundwire/dmi-quirks.c
@@ -91,6 +91,14 @@ static const struct dmi_system_id adr_remap_quirk_table[] = {
.driver_data = (void *)intel_tgl_bios,
},
{
+ /* quirk used for NUC15 LAPBC710 skew */
+ .matches = {
+ DMI_MATCH(DMI_BOARD_VENDOR, "Intel Corporation"),
+ DMI_MATCH(DMI_BOARD_NAME, "LAPBC710"),
+ },
+ .driver_data = (void *)intel_tgl_bios,
+ },
+ {
.matches = {
DMI_MATCH(DMI_SYS_VENDOR, "Dell Inc"),
DMI_EXACT_MATCH(DMI_PRODUCT_SKU, "0A3E")
diff --git a/drivers/soundwire/intel.c b/drivers/soundwire/intel.c
index 8c76541..bc9c50b 100644
--- a/drivers/soundwire/intel.c
+++ b/drivers/soundwire/intel.c
@@ -8,10 +8,7 @@
#include <linux/acpi.h>
#include <linux/debugfs.h>
#include <linux/delay.h>
-#include <linux/module.h>
-#include <linux/interrupt.h>
#include <linux/io.h>
-#include <linux/auxiliary_bus.h>
#include <sound/pcm_params.h>
#include <linux/pm_runtime.h>
#include <sound/soc.h>
@@ -22,27 +19,6 @@
#include "bus.h"
#include "intel.h"
-/* IDA min selected to avoid conflicts with HDaudio/iDISP SDI values */
-#define INTEL_DEV_NUM_IDA_MIN 4
-
-#define INTEL_MASTER_SUSPEND_DELAY_MS 3000
-#define INTEL_MASTER_RESET_ITERATIONS 10
-
-/*
- * debug/config flags for the Intel SoundWire Master.
- *
- * Since we may have multiple masters active, we can have up to 8
- * flags reused in each byte, with master0 using the ls-byte, etc.
- */
-
-#define SDW_INTEL_MASTER_DISABLE_PM_RUNTIME BIT(0)
-#define SDW_INTEL_MASTER_DISABLE_CLOCK_STOP BIT(1)
-#define SDW_INTEL_MASTER_DISABLE_PM_RUNTIME_IDLE BIT(2)
-#define SDW_INTEL_MASTER_DISABLE_MULTI_LINK BIT(3)
-
-static int md_flags;
-module_param_named(sdw_md_flags, md_flags, int, 0444);
-MODULE_PARM_DESC(sdw_md_flags, "SoundWire Intel Master device flags (0x0 all off)");
enum intel_pdi_type {
INTEL_PDI_IN = 0,
@@ -745,10 +721,10 @@ static int intel_free_stream(struct sdw_intel *sdw,
* bank switch routines
*/
-static int intel_pre_bank_switch(struct sdw_bus *bus)
+static int intel_pre_bank_switch(struct sdw_intel *sdw)
{
- struct sdw_cdns *cdns = bus_to_cdns(bus);
- struct sdw_intel *sdw = cdns_to_intel(cdns);
+ struct sdw_cdns *cdns = &sdw->cdns;
+ struct sdw_bus *bus = &cdns->bus;
/* Write to register only for multi-link */
if (!bus->multi_link)
@@ -759,10 +735,10 @@ static int intel_pre_bank_switch(struct sdw_bus *bus)
return 0;
}
-static int intel_post_bank_switch(struct sdw_bus *bus)
+static int intel_post_bank_switch(struct sdw_intel *sdw)
{
- struct sdw_cdns *cdns = bus_to_cdns(bus);
- struct sdw_intel *sdw = cdns_to_intel(cdns);
+ struct sdw_cdns *cdns = &sdw->cdns;
+ struct sdw_bus *bus = &cdns->bus;
void __iomem *shim = sdw->link_res->shim;
int sync_reg, ret;
@@ -824,15 +800,15 @@ static int intel_hw_params(struct snd_pcm_substream *substream,
{
struct sdw_cdns *cdns = snd_soc_dai_get_drvdata(dai);
struct sdw_intel *sdw = cdns_to_intel(cdns);
- struct sdw_cdns_dma_data *dma;
+ struct sdw_cdns_dai_runtime *dai_runtime;
struct sdw_cdns_pdi *pdi;
struct sdw_stream_config sconfig;
struct sdw_port_config *pconfig;
int ch, dir;
int ret;
- dma = snd_soc_dai_get_dma_data(dai, substream);
- if (!dma)
+ dai_runtime = cdns->dai_runtime_array[dai->id];
+ if (!dai_runtime)
return -EIO;
ch = params_channels(params);
@@ -854,10 +830,10 @@ static int intel_hw_params(struct snd_pcm_substream *substream,
sdw_cdns_config_stream(cdns, ch, dir, pdi);
/* store pdi and hw_params, may be needed in prepare step */
- dma->paused = false;
- dma->suspended = false;
- dma->pdi = pdi;
- dma->hw_params = params;
+ dai_runtime->paused = false;
+ dai_runtime->suspended = false;
+ dai_runtime->pdi = pdi;
+ dai_runtime->hw_params = params;
/* Inform DSP about PDI stream number */
ret = intel_params_stream(sdw, substream->stream, dai, params,
@@ -869,7 +845,7 @@ static int intel_hw_params(struct snd_pcm_substream *substream,
sconfig.direction = dir;
sconfig.ch_count = ch;
sconfig.frame_rate = params_rate(params);
- sconfig.type = dma->stream_type;
+ sconfig.type = dai_runtime->stream_type;
sconfig.bps = snd_pcm_format_width(params_format(params));
@@ -884,7 +860,7 @@ static int intel_hw_params(struct snd_pcm_substream *substream,
pconfig->ch_mask = (1 << ch) - 1;
ret = sdw_stream_add_master(&cdns->bus, &sconfig,
- pconfig, 1, dma->stream);
+ pconfig, 1, dai_runtime->stream);
if (ret)
dev_err(cdns->dev, "add master to stream failed:%d\n", ret);
@@ -898,19 +874,19 @@ static int intel_prepare(struct snd_pcm_substream *substream,
{
struct sdw_cdns *cdns = snd_soc_dai_get_drvdata(dai);
struct sdw_intel *sdw = cdns_to_intel(cdns);
- struct sdw_cdns_dma_data *dma;
+ struct sdw_cdns_dai_runtime *dai_runtime;
int ch, dir;
int ret = 0;
- dma = snd_soc_dai_get_dma_data(dai, substream);
- if (!dma) {
- dev_err(dai->dev, "failed to get dma data in %s\n",
+ dai_runtime = cdns->dai_runtime_array[dai->id];
+ if (!dai_runtime) {
+ dev_err(dai->dev, "failed to get dai runtime in %s\n",
__func__);
return -EIO;
}
- if (dma->suspended) {
- dma->suspended = false;
+ if (dai_runtime->suspended) {
+ dai_runtime->suspended = false;
/*
* .prepare() is called after system resume, where we
@@ -921,21 +897,21 @@ static int intel_prepare(struct snd_pcm_substream *substream,
*/
/* configure stream */
- ch = params_channels(dma->hw_params);
+ ch = params_channels(dai_runtime->hw_params);
if (substream->stream == SNDRV_PCM_STREAM_CAPTURE)
dir = SDW_DATA_DIR_RX;
else
dir = SDW_DATA_DIR_TX;
- intel_pdi_shim_configure(sdw, dma->pdi);
- intel_pdi_alh_configure(sdw, dma->pdi);
- sdw_cdns_config_stream(cdns, ch, dir, dma->pdi);
+ intel_pdi_shim_configure(sdw, dai_runtime->pdi);
+ intel_pdi_alh_configure(sdw, dai_runtime->pdi);
+ sdw_cdns_config_stream(cdns, ch, dir, dai_runtime->pdi);
/* Inform DSP about PDI stream number */
ret = intel_params_stream(sdw, substream->stream, dai,
- dma->hw_params,
+ dai_runtime->hw_params,
sdw->instance,
- dma->pdi->intel_alh_id);
+ dai_runtime->pdi->intel_alh_id);
}
return ret;
@@ -946,11 +922,11 @@ intel_hw_free(struct snd_pcm_substream *substream, struct snd_soc_dai *dai)
{
struct sdw_cdns *cdns = snd_soc_dai_get_drvdata(dai);
struct sdw_intel *sdw = cdns_to_intel(cdns);
- struct sdw_cdns_dma_data *dma;
+ struct sdw_cdns_dai_runtime *dai_runtime;
int ret;
- dma = snd_soc_dai_get_dma_data(dai, substream);
- if (!dma)
+ dai_runtime = cdns->dai_runtime_array[dai->id];
+ if (!dai_runtime)
return -EIO;
/*
@@ -959,10 +935,10 @@ intel_hw_free(struct snd_pcm_substream *substream, struct snd_soc_dai *dai)
* DEPREPARED for the first cpu-dai and to RELEASED for the last
* cpu-dai.
*/
- ret = sdw_stream_remove_master(&cdns->bus, dma->stream);
+ ret = sdw_stream_remove_master(&cdns->bus, dai_runtime->stream);
if (ret < 0) {
dev_err(dai->dev, "remove master from stream %s failed: %d\n",
- dma->stream->name, ret);
+ dai_runtime->stream->name, ret);
return ret;
}
@@ -972,8 +948,8 @@ intel_hw_free(struct snd_pcm_substream *substream, struct snd_soc_dai *dai)
return ret;
}
- dma->hw_params = NULL;
- dma->pdi = NULL;
+ dai_runtime->hw_params = NULL;
+ dai_runtime->pdi = NULL;
return 0;
}
@@ -996,17 +972,14 @@ static int intel_pcm_set_sdw_stream(struct snd_soc_dai *dai,
static void *intel_get_sdw_stream(struct snd_soc_dai *dai,
int direction)
{
- struct sdw_cdns_dma_data *dma;
+ struct sdw_cdns *cdns = snd_soc_dai_get_drvdata(dai);
+ struct sdw_cdns_dai_runtime *dai_runtime;
- if (direction == SNDRV_PCM_STREAM_PLAYBACK)
- dma = dai->playback_dma_data;
- else
- dma = dai->capture_dma_data;
-
- if (!dma)
+ dai_runtime = cdns->dai_runtime_array[dai->id];
+ if (!dai_runtime)
return ERR_PTR(-EINVAL);
- return dma->stream;
+ return dai_runtime->stream;
}
static int intel_trigger(struct snd_pcm_substream *substream, int cmd, struct snd_soc_dai *dai)
@@ -1014,7 +987,7 @@ static int intel_trigger(struct snd_pcm_substream *substream, int cmd, struct sn
struct sdw_cdns *cdns = snd_soc_dai_get_drvdata(dai);
struct sdw_intel *sdw = cdns_to_intel(cdns);
struct sdw_intel_link_res *res = sdw->link_res;
- struct sdw_cdns_dma_data *dma;
+ struct sdw_cdns_dai_runtime *dai_runtime;
int ret = 0;
/*
@@ -1025,9 +998,9 @@ static int intel_trigger(struct snd_pcm_substream *substream, int cmd, struct sn
if (res->ops && res->ops->trigger)
res->ops->trigger(dai, cmd, substream->stream);
- dma = snd_soc_dai_get_dma_data(dai, substream);
- if (!dma) {
- dev_err(dai->dev, "failed to get dma data in %s\n",
+ dai_runtime = cdns->dai_runtime_array[dai->id];
+ if (!dai_runtime) {
+ dev_err(dai->dev, "failed to get dai runtime in %s\n",
__func__);
return -EIO;
}
@@ -1042,17 +1015,17 @@ static int intel_trigger(struct snd_pcm_substream *substream, int cmd, struct sn
* the .trigger callback is used to track the suspend case only.
*/
- dma->suspended = true;
+ dai_runtime->suspended = true;
ret = intel_free_stream(sdw, substream->stream, dai, sdw->instance);
break;
case SNDRV_PCM_TRIGGER_PAUSE_PUSH:
- dma->paused = true;
+ dai_runtime->paused = true;
break;
case SNDRV_PCM_TRIGGER_STOP:
case SNDRV_PCM_TRIGGER_PAUSE_RELEASE:
- dma->paused = false;
+ dai_runtime->paused = false;
break;
default:
break;
@@ -1091,27 +1064,21 @@ static int intel_component_dais_suspend(struct snd_soc_component *component)
for_each_component_dais(component, dai) {
struct sdw_cdns *cdns = snd_soc_dai_get_drvdata(dai);
struct sdw_intel *sdw = cdns_to_intel(cdns);
- struct sdw_cdns_dma_data *dma;
- int stream;
+ struct sdw_cdns_dai_runtime *dai_runtime;
int ret;
- dma = dai->playback_dma_data;
- stream = SNDRV_PCM_STREAM_PLAYBACK;
- if (!dma) {
- dma = dai->capture_dma_data;
- stream = SNDRV_PCM_STREAM_CAPTURE;
- }
+ dai_runtime = cdns->dai_runtime_array[dai->id];
- if (!dma)
+ if (!dai_runtime)
continue;
- if (dma->suspended)
+ if (dai_runtime->suspended)
continue;
- if (dma->paused) {
- dma->suspended = true;
+ if (dai_runtime->paused) {
+ dai_runtime->suspended = true;
- ret = intel_free_stream(sdw, stream, dai, sdw->instance);
+ ret = intel_free_stream(sdw, dai_runtime->direction, dai, sdw->instance);
if (ret < 0)
return ret;
}
@@ -1178,6 +1145,7 @@ static int intel_create_dai(struct sdw_cdns *cdns,
static int intel_register_dai(struct sdw_intel *sdw)
{
+ struct sdw_cdns_dai_runtime **dai_runtime_array;
struct sdw_cdns_stream_config config;
struct sdw_cdns *cdns = &sdw->cdns;
struct sdw_cdns_streams *stream;
@@ -1195,6 +1163,13 @@ static int intel_register_dai(struct sdw_intel *sdw)
/* DAIs are created based on total number of PDIs supported */
num_dai = cdns->pcm.num_pdi;
+ dai_runtime_array = devm_kcalloc(cdns->dev, num_dai,
+ sizeof(struct sdw_cdns_dai_runtime *),
+ GFP_KERNEL);
+ if (!dai_runtime_array)
+ return -ENOMEM;
+ cdns->dai_runtime_array = dai_runtime_array;
+
dais = devm_kcalloc(cdns->dev, num_dai, sizeof(*dais), GFP_KERNEL);
if (!dais)
return -ENOMEM;
@@ -1423,620 +1398,26 @@ static int intel_stop_bus(struct sdw_intel *sdw, bool clock_stop)
return 0;
}
-static int sdw_master_read_intel_prop(struct sdw_bus *bus)
-{
- struct sdw_master_prop *prop = &bus->prop;
- struct fwnode_handle *link;
- char name[32];
- u32 quirk_mask;
+const struct sdw_intel_hw_ops sdw_intel_cnl_hw_ops = {
+ .debugfs_init = intel_debugfs_init,
+ .debugfs_exit = intel_debugfs_exit,
- /* Find master handle */
- snprintf(name, sizeof(name),
- "mipi-sdw-link-%d-subproperties", bus->link_id);
+ .register_dai = intel_register_dai,
- link = device_get_named_child_node(bus->dev, name);
- if (!link) {
- dev_err(bus->dev, "Master node %s not found\n", name);
- return -EIO;
- }
+ .check_clock_stop = intel_check_clock_stop,
+ .start_bus = intel_start_bus,
+ .start_bus_after_reset = intel_start_bus_after_reset,
+ .start_bus_after_clock_stop = intel_start_bus_after_clock_stop,
+ .stop_bus = intel_stop_bus,
- fwnode_property_read_u32(link,
- "intel-sdw-ip-clock",
- &prop->mclk_freq);
+ .link_power_up = intel_link_power_up,
+ .link_power_down = intel_link_power_down,
- /* the values reported by BIOS are the 2x clock, not the bus clock */
- prop->mclk_freq /= 2;
+ .shim_check_wake = intel_shim_check_wake,
+ .shim_wake = intel_shim_wake,
- fwnode_property_read_u32(link,
- "intel-quirk-mask",
- &quirk_mask);
-
- if (quirk_mask & SDW_INTEL_QUIRK_MASK_BUS_DISABLE)
- prop->hw_disabled = true;
-
- prop->quirks = SDW_MASTER_QUIRKS_CLEAR_INITIAL_CLASH |
- SDW_MASTER_QUIRKS_CLEAR_INITIAL_PARITY;
-
- return 0;
-}
-
-static int intel_prop_read(struct sdw_bus *bus)
-{
- /* Initialize with default handler to read all DisCo properties */
- sdw_master_read_prop(bus);
-
- /* read Intel-specific properties */
- sdw_master_read_intel_prop(bus);
-
- return 0;
-}
-
-static struct sdw_master_ops sdw_intel_ops = {
- .read_prop = intel_prop_read,
- .override_adr = sdw_dmi_override_adr,
- .xfer_msg = cdns_xfer_msg,
- .xfer_msg_defer = cdns_xfer_msg_defer,
- .reset_page_addr = cdns_reset_page_addr,
- .set_bus_conf = cdns_bus_conf,
.pre_bank_switch = intel_pre_bank_switch,
.post_bank_switch = intel_post_bank_switch,
- .read_ping_status = cdns_read_ping_status,
};
-
-/*
- * probe and init (aux_dev_id argument is required by function prototype but not used)
- */
-static int intel_link_probe(struct auxiliary_device *auxdev,
- const struct auxiliary_device_id *aux_dev_id)
-
-{
- struct device *dev = &auxdev->dev;
- struct sdw_intel_link_dev *ldev = auxiliary_dev_to_sdw_intel_link_dev(auxdev);
- struct sdw_intel *sdw;
- struct sdw_cdns *cdns;
- struct sdw_bus *bus;
- int ret;
-
- sdw = devm_kzalloc(dev, sizeof(*sdw), GFP_KERNEL);
- if (!sdw)
- return -ENOMEM;
-
- cdns = &sdw->cdns;
- bus = &cdns->bus;
-
- sdw->instance = auxdev->id;
- sdw->link_res = &ldev->link_res;
- cdns->dev = dev;
- cdns->registers = sdw->link_res->registers;
- cdns->instance = sdw->instance;
- cdns->msg_count = 0;
-
- bus->link_id = auxdev->id;
- bus->dev_num_ida_min = INTEL_DEV_NUM_IDA_MIN;
- bus->clk_stop_timeout = 1;
-
- sdw_cdns_probe(cdns);
-
- /* Set ops */
- bus->ops = &sdw_intel_ops;
-
- /* set driver data, accessed by snd_soc_dai_get_drvdata() */
- auxiliary_set_drvdata(auxdev, cdns);
-
- /* use generic bandwidth allocation algorithm */
- sdw->cdns.bus.compute_params = sdw_compute_params;
-
- /* avoid resuming from pm_runtime suspend if it's not required */
- dev_pm_set_driver_flags(dev, DPM_FLAG_SMART_SUSPEND);
-
- ret = sdw_bus_master_add(bus, dev, dev->fwnode);
- if (ret) {
- dev_err(dev, "sdw_bus_master_add fail: %d\n", ret);
- return ret;
- }
-
- if (bus->prop.hw_disabled)
- dev_info(dev,
- "SoundWire master %d is disabled, will be ignored\n",
- bus->link_id);
- /*
- * Ignore BIOS err_threshold, it's a really bad idea when dealing
- * with multiple hardware synchronized links
- */
- bus->prop.err_threshold = 0;
-
- return 0;
-}
-
-int intel_link_startup(struct auxiliary_device *auxdev)
-{
- struct device *dev = &auxdev->dev;
- struct sdw_cdns *cdns = auxiliary_get_drvdata(auxdev);
- struct sdw_intel *sdw = cdns_to_intel(cdns);
- struct sdw_bus *bus = &cdns->bus;
- int link_flags;
- bool multi_link;
- u32 clock_stop_quirks;
- int ret;
-
- if (bus->prop.hw_disabled) {
- dev_info(dev,
- "SoundWire master %d is disabled, ignoring\n",
- sdw->instance);
- return 0;
- }
-
- link_flags = md_flags >> (bus->link_id * 8);
- multi_link = !(link_flags & SDW_INTEL_MASTER_DISABLE_MULTI_LINK);
- if (!multi_link) {
- dev_dbg(dev, "Multi-link is disabled\n");
- } else {
- /*
- * hardware-based synchronization is required regardless
- * of the number of segments used by a stream: SSP-based
- * synchronization is gated by gsync when the multi-master
- * mode is set.
- */
- bus->hw_sync_min_links = 1;
- }
- bus->multi_link = multi_link;
-
- /* Initialize shim, controller */
- ret = intel_link_power_up(sdw);
- if (ret)
- goto err_init;
-
- /* Register DAIs */
- ret = intel_register_dai(sdw);
- if (ret) {
- dev_err(dev, "DAI registration failed: %d\n", ret);
- goto err_power_up;
- }
-
- intel_debugfs_init(sdw);
-
- /* start bus */
- ret = intel_start_bus(sdw);
- if (ret) {
- dev_err(dev, "bus start failed: %d\n", ret);
- goto err_power_up;
- }
-
- /* Enable runtime PM */
- if (!(link_flags & SDW_INTEL_MASTER_DISABLE_PM_RUNTIME)) {
- pm_runtime_set_autosuspend_delay(dev,
- INTEL_MASTER_SUSPEND_DELAY_MS);
- pm_runtime_use_autosuspend(dev);
- pm_runtime_mark_last_busy(dev);
-
- pm_runtime_set_active(dev);
- pm_runtime_enable(dev);
- }
-
- clock_stop_quirks = sdw->link_res->clock_stop_quirks;
- if (clock_stop_quirks & SDW_INTEL_CLK_STOP_NOT_ALLOWED) {
- /*
- * To keep the clock running we need to prevent
- * pm_runtime suspend from happening by increasing the
- * reference count.
- * This quirk is specified by the parent PCI device in
- * case of specific latency requirements. It will have
- * no effect if pm_runtime is disabled by the user via
- * a module parameter for testing purposes.
- */
- pm_runtime_get_noresume(dev);
- }
-
- /*
- * The runtime PM status of Slave devices is "Unsupported"
- * until they report as ATTACHED. If they don't, e.g. because
- * there are no Slave devices populated or if the power-on is
- * delayed or dependent on a power switch, the Master will
- * remain active and prevent its parent from suspending.
- *
- * Conditionally force the pm_runtime core to re-evaluate the
- * Master status in the absence of any Slave activity. A quirk
- * is provided to e.g. deal with Slaves that may be powered on
- * with a delay. A more complete solution would require the
- * definition of Master properties.
- */
- if (!(link_flags & SDW_INTEL_MASTER_DISABLE_PM_RUNTIME_IDLE))
- pm_runtime_idle(dev);
-
- sdw->startup_done = true;
- return 0;
-
-err_power_up:
- intel_link_power_down(sdw);
-err_init:
- return ret;
-}
-
-static void intel_link_remove(struct auxiliary_device *auxdev)
-{
- struct sdw_cdns *cdns = auxiliary_get_drvdata(auxdev);
- struct sdw_intel *sdw = cdns_to_intel(cdns);
- struct sdw_bus *bus = &cdns->bus;
-
- /*
- * Since pm_runtime is already disabled, we don't decrease
- * the refcount when the clock_stop_quirk is
- * SDW_INTEL_CLK_STOP_NOT_ALLOWED
- */
- if (!bus->prop.hw_disabled) {
- intel_debugfs_exit(sdw);
- sdw_cdns_enable_interrupt(cdns, false);
- }
- sdw_bus_master_delete(bus);
-}
-
-int intel_link_process_wakeen_event(struct auxiliary_device *auxdev)
-{
- struct device *dev = &auxdev->dev;
- struct sdw_intel *sdw;
- struct sdw_bus *bus;
-
- sdw = auxiliary_get_drvdata(auxdev);
- bus = &sdw->cdns.bus;
-
- if (bus->prop.hw_disabled || !sdw->startup_done) {
- dev_dbg(dev, "SoundWire master %d is disabled or not-started, ignoring\n",
- bus->link_id);
- return 0;
- }
-
- if (!intel_shim_check_wake(sdw))
- return 0;
-
- /* disable WAKEEN interrupt ASAP to prevent interrupt flood */
- intel_shim_wake(sdw, false);
-
- /*
- * resume the Master, which will generate a bus reset and result in
- * Slaves re-attaching and be re-enumerated. The SoundWire physical
- * device which generated the wake will trigger an interrupt, which
- * will in turn cause the corresponding Linux Slave device to be
- * resumed and the Slave codec driver to check the status.
- */
- pm_request_resume(dev);
-
- return 0;
-}
-
-/*
- * PM calls
- */
-
-static int intel_resume_child_device(struct device *dev, void *data)
-{
- int ret;
- struct sdw_slave *slave = dev_to_sdw_dev(dev);
-
- if (!slave->probed) {
- dev_dbg(dev, "skipping device, no probed driver\n");
- return 0;
- }
- if (!slave->dev_num_sticky) {
- dev_dbg(dev, "skipping device, never detected on bus\n");
- return 0;
- }
-
- ret = pm_request_resume(dev);
- if (ret < 0)
- dev_err(dev, "%s: pm_request_resume failed: %d\n", __func__, ret);
-
- return ret;
-}
-
-static int __maybe_unused intel_pm_prepare(struct device *dev)
-{
- struct sdw_cdns *cdns = dev_get_drvdata(dev);
- struct sdw_intel *sdw = cdns_to_intel(cdns);
- struct sdw_bus *bus = &cdns->bus;
- u32 clock_stop_quirks;
- int ret;
-
- if (bus->prop.hw_disabled || !sdw->startup_done) {
- dev_dbg(dev, "SoundWire master %d is disabled or not-started, ignoring\n",
- bus->link_id);
- return 0;
- }
-
- clock_stop_quirks = sdw->link_res->clock_stop_quirks;
-
- if (pm_runtime_suspended(dev) &&
- pm_runtime_suspended(dev->parent) &&
- ((clock_stop_quirks & SDW_INTEL_CLK_STOP_BUS_RESET) ||
- !clock_stop_quirks)) {
- /*
- * if we've enabled clock stop, and the parent is suspended, the SHIM registers
- * are not accessible and the shim wake cannot be disabled.
- * The only solution is to resume the entire bus to full power
- */
-
- /*
- * If any operation in this block fails, we keep going since we don't want
- * to prevent system suspend from happening and errors should be recoverable
- * on resume.
- */
-
- /*
- * first resume the device for this link. This will also by construction
- * resume the PCI parent device.
- */
- ret = pm_request_resume(dev);
- if (ret < 0) {
- dev_err(dev, "%s: pm_request_resume failed: %d\n", __func__, ret);
- return 0;
- }
-
- /*
- * Continue resuming the entire bus (parent + child devices) to exit
- * the clock stop mode. If there are no devices connected on this link
- * this is a no-op.
- * The resume to full power could have been implemented with a .prepare
- * step in SoundWire codec drivers. This would however require a lot
- * of code to handle an Intel-specific corner case. It is simpler in
- * practice to add a loop at the link level.
- */
- ret = device_for_each_child(bus->dev, NULL, intel_resume_child_device);
-
- if (ret < 0)
- dev_err(dev, "%s: intel_resume_child_device failed: %d\n", __func__, ret);
- }
-
- return 0;
-}
-
-static int __maybe_unused intel_suspend(struct device *dev)
-{
- struct sdw_cdns *cdns = dev_get_drvdata(dev);
- struct sdw_intel *sdw = cdns_to_intel(cdns);
- struct sdw_bus *bus = &cdns->bus;
- u32 clock_stop_quirks;
- int ret;
-
- if (bus->prop.hw_disabled || !sdw->startup_done) {
- dev_dbg(dev, "SoundWire master %d is disabled or not-started, ignoring\n",
- bus->link_id);
- return 0;
- }
-
- if (pm_runtime_suspended(dev)) {
- dev_dbg(dev, "pm_runtime status: suspended\n");
-
- clock_stop_quirks = sdw->link_res->clock_stop_quirks;
-
- if ((clock_stop_quirks & SDW_INTEL_CLK_STOP_BUS_RESET) ||
- !clock_stop_quirks) {
-
- if (pm_runtime_suspended(dev->parent)) {
- /*
- * paranoia check: this should not happen with the .prepare
- * resume to full power
- */
- dev_err(dev, "%s: invalid config: parent is suspended\n", __func__);
- } else {
- intel_shim_wake(sdw, false);
- }
- }
-
- return 0;
- }
-
- ret = intel_stop_bus(sdw, false);
- if (ret < 0) {
- dev_err(dev, "%s: cannot stop bus: %d\n", __func__, ret);
- return ret;
- }
-
- return 0;
-}
-
-static int __maybe_unused intel_suspend_runtime(struct device *dev)
-{
- struct sdw_cdns *cdns = dev_get_drvdata(dev);
- struct sdw_intel *sdw = cdns_to_intel(cdns);
- struct sdw_bus *bus = &cdns->bus;
- u32 clock_stop_quirks;
- int ret;
-
- if (bus->prop.hw_disabled || !sdw->startup_done) {
- dev_dbg(dev, "SoundWire master %d is disabled or not-started, ignoring\n",
- bus->link_id);
- return 0;
- }
-
- clock_stop_quirks = sdw->link_res->clock_stop_quirks;
-
- if (clock_stop_quirks & SDW_INTEL_CLK_STOP_TEARDOWN) {
- ret = intel_stop_bus(sdw, false);
- if (ret < 0) {
- dev_err(dev, "%s: cannot stop bus during teardown: %d\n",
- __func__, ret);
- return ret;
- }
- } else if (clock_stop_quirks & SDW_INTEL_CLK_STOP_BUS_RESET || !clock_stop_quirks) {
- ret = intel_stop_bus(sdw, true);
- if (ret < 0) {
- dev_err(dev, "%s: cannot stop bus during clock_stop: %d\n",
- __func__, ret);
- return ret;
- }
- } else {
- dev_err(dev, "%s clock_stop_quirks %x unsupported\n",
- __func__, clock_stop_quirks);
- ret = -EINVAL;
- }
-
- return ret;
-}
-
-static int __maybe_unused intel_resume(struct device *dev)
-{
- struct sdw_cdns *cdns = dev_get_drvdata(dev);
- struct sdw_intel *sdw = cdns_to_intel(cdns);
- struct sdw_bus *bus = &cdns->bus;
- int link_flags;
- int ret;
-
- if (bus->prop.hw_disabled || !sdw->startup_done) {
- dev_dbg(dev, "SoundWire master %d is disabled or not-started, ignoring\n",
- bus->link_id);
- return 0;
- }
-
- link_flags = md_flags >> (bus->link_id * 8);
-
- if (pm_runtime_suspended(dev)) {
- dev_dbg(dev, "pm_runtime status was suspended, forcing active\n");
-
- /* follow required sequence from runtime_pm.rst */
- pm_runtime_disable(dev);
- pm_runtime_set_active(dev);
- pm_runtime_mark_last_busy(dev);
- pm_runtime_enable(dev);
-
- link_flags = md_flags >> (bus->link_id * 8);
-
- if (!(link_flags & SDW_INTEL_MASTER_DISABLE_PM_RUNTIME_IDLE))
- pm_runtime_idle(dev);
- }
-
- ret = intel_link_power_up(sdw);
- if (ret) {
- dev_err(dev, "%s failed: %d\n", __func__, ret);
- return ret;
- }
-
- /*
- * make sure all Slaves are tagged as UNATTACHED and provide
- * reason for reinitialization
- */
- sdw_clear_slave_status(bus, SDW_UNATTACH_REQUEST_MASTER_RESET);
-
- ret = intel_start_bus(sdw);
- if (ret < 0) {
- dev_err(dev, "cannot start bus during resume\n");
- intel_link_power_down(sdw);
- return ret;
- }
-
- /*
- * after system resume, the pm_runtime suspend() may kick in
- * during the enumeration, before any children device force the
- * master device to remain active. Using pm_runtime_get()
- * routines is not really possible, since it'd prevent the
- * master from suspending.
- * A reasonable compromise is to update the pm_runtime
- * counters and delay the pm_runtime suspend by several
- * seconds, by when all enumeration should be complete.
- */
- pm_runtime_mark_last_busy(dev);
-
- return 0;
-}
-
-static int __maybe_unused intel_resume_runtime(struct device *dev)
-{
- struct sdw_cdns *cdns = dev_get_drvdata(dev);
- struct sdw_intel *sdw = cdns_to_intel(cdns);
- struct sdw_bus *bus = &cdns->bus;
- u32 clock_stop_quirks;
- int ret;
-
- if (bus->prop.hw_disabled || !sdw->startup_done) {
- dev_dbg(dev, "SoundWire master %d is disabled or not-started, ignoring\n",
- bus->link_id);
- return 0;
- }
-
- /* unconditionally disable WAKEEN interrupt */
- intel_shim_wake(sdw, false);
-
- clock_stop_quirks = sdw->link_res->clock_stop_quirks;
-
- if (clock_stop_quirks & SDW_INTEL_CLK_STOP_TEARDOWN) {
- ret = intel_link_power_up(sdw);
- if (ret) {
- dev_err(dev, "%s: power_up failed after teardown: %d\n", __func__, ret);
- return ret;
- }
-
- /*
- * make sure all Slaves are tagged as UNATTACHED and provide
- * reason for reinitialization
- */
- sdw_clear_slave_status(bus, SDW_UNATTACH_REQUEST_MASTER_RESET);
-
- ret = intel_start_bus(sdw);
- if (ret < 0) {
- dev_err(dev, "%s: cannot start bus after teardown: %d\n", __func__, ret);
- intel_link_power_down(sdw);
- return ret;
- }
-
-
- } else if (clock_stop_quirks & SDW_INTEL_CLK_STOP_BUS_RESET) {
- ret = intel_link_power_up(sdw);
- if (ret) {
- dev_err(dev, "%s: power_up failed after bus reset: %d\n", __func__, ret);
- return ret;
- }
-
- ret = intel_start_bus_after_reset(sdw);
- if (ret < 0) {
- dev_err(dev, "%s: cannot start bus after reset: %d\n", __func__, ret);
- intel_link_power_down(sdw);
- return ret;
- }
- } else if (!clock_stop_quirks) {
-
- intel_check_clock_stop(sdw);
-
- ret = intel_link_power_up(sdw);
- if (ret) {
- dev_err(dev, "%s: power_up failed: %d\n", __func__, ret);
- return ret;
- }
-
- ret = intel_start_bus_after_clock_stop(sdw);
- if (ret < 0) {
- dev_err(dev, "%s: cannot start bus after clock stop: %d\n", __func__, ret);
- intel_link_power_down(sdw);
- return ret;
- }
- } else {
- dev_err(dev, "%s: clock_stop_quirks %x unsupported\n",
- __func__, clock_stop_quirks);
- ret = -EINVAL;
- }
-
- return ret;
-}
-
-static const struct dev_pm_ops intel_pm = {
- .prepare = intel_pm_prepare,
- SET_SYSTEM_SLEEP_PM_OPS(intel_suspend, intel_resume)
- SET_RUNTIME_PM_OPS(intel_suspend_runtime, intel_resume_runtime, NULL)
-};
-
-static const struct auxiliary_device_id intel_link_id_table[] = {
- { .name = "soundwire_intel.link" },
- {},
-};
-MODULE_DEVICE_TABLE(auxiliary, intel_link_id_table);
-
-static struct auxiliary_driver sdw_intel_drv = {
- .probe = intel_link_probe,
- .remove = intel_link_remove,
- .driver = {
- /* auxiliary_driver_register() sets .name to be the modname */
- .pm = &intel_pm,
- },
- .id_table = intel_link_id_table
-};
-module_auxiliary_driver(sdw_intel_drv);
+EXPORT_SYMBOL_NS(sdw_intel_cnl_hw_ops, SOUNDWIRE_INTEL);
-MODULE_LICENSE("Dual BSD/GPL");
-MODULE_DESCRIPTION("Intel Soundwire Link Driver");
diff --git a/drivers/soundwire/intel.h b/drivers/soundwire/intel.h
index cd93a44..de98833 100644
--- a/drivers/soundwire/intel.h
+++ b/drivers/soundwire/intel.h
@@ -7,6 +7,7 @@
/**
* struct sdw_intel_link_res - Soundwire Intel link resource structure,
* typically populated by the controller driver.
+ * @hw_ops: platform-specific ops
* @mmio_base: mmio base of SoundWire registers
* @registers: Link IO registers base
* @shim: Audio shim pointer
@@ -22,6 +23,8 @@
* @list: used to walk-through all masters exposed by the same controller
*/
struct sdw_intel_link_res {
+ const struct sdw_intel_hw_ops *hw_ops;
+
void __iomem *mmio_base; /* not strictly needed, useful for debug */
void __iomem *registers;
void __iomem *shim;
@@ -47,15 +50,92 @@ struct sdw_intel {
#endif
};
-int intel_link_startup(struct auxiliary_device *auxdev);
-int intel_link_process_wakeen_event(struct auxiliary_device *auxdev);
+#define cdns_to_intel(_cdns) container_of(_cdns, struct sdw_intel, cdns)
-struct sdw_intel_link_dev {
- struct auxiliary_device auxdev;
- struct sdw_intel_link_res link_res;
-};
+#define INTEL_MASTER_RESET_ITERATIONS 10
-#define auxiliary_dev_to_sdw_intel_link_dev(auxiliary_dev) \
- container_of(auxiliary_dev, struct sdw_intel_link_dev, auxdev)
+#define SDW_INTEL_CHECK_OPS(sdw, cb) ((sdw) && (sdw)->link_res && (sdw)->link_res->hw_ops && \
+ (sdw)->link_res->hw_ops->cb)
+#define SDW_INTEL_OPS(sdw, cb) ((sdw)->link_res->hw_ops->cb)
+
+static inline void sdw_intel_debugfs_init(struct sdw_intel *sdw)
+{
+ if (SDW_INTEL_CHECK_OPS(sdw, debugfs_init))
+ SDW_INTEL_OPS(sdw, debugfs_init)(sdw);
+}
+
+static inline void sdw_intel_debugfs_exit(struct sdw_intel *sdw)
+{
+ if (SDW_INTEL_CHECK_OPS(sdw, debugfs_exit))
+ SDW_INTEL_OPS(sdw, debugfs_exit)(sdw);
+}
+
+static inline int sdw_intel_register_dai(struct sdw_intel *sdw)
+{
+ if (SDW_INTEL_CHECK_OPS(sdw, register_dai))
+ return SDW_INTEL_OPS(sdw, register_dai)(sdw);
+ return -ENOTSUPP;
+}
+
+static inline void sdw_intel_check_clock_stop(struct sdw_intel *sdw)
+{
+ if (SDW_INTEL_CHECK_OPS(sdw, check_clock_stop))
+ SDW_INTEL_OPS(sdw, check_clock_stop)(sdw);
+}
+
+static inline int sdw_intel_start_bus(struct sdw_intel *sdw)
+{
+ if (SDW_INTEL_CHECK_OPS(sdw, start_bus))
+ return SDW_INTEL_OPS(sdw, start_bus)(sdw);
+ return -ENOTSUPP;
+}
+
+static inline int sdw_intel_start_bus_after_reset(struct sdw_intel *sdw)
+{
+ if (SDW_INTEL_CHECK_OPS(sdw, start_bus_after_reset))
+ return SDW_INTEL_OPS(sdw, start_bus_after_reset)(sdw);
+ return -ENOTSUPP;
+}
+
+static inline int sdw_intel_start_bus_after_clock_stop(struct sdw_intel *sdw)
+{
+ if (SDW_INTEL_CHECK_OPS(sdw, start_bus_after_clock_stop))
+ return SDW_INTEL_OPS(sdw, start_bus_after_clock_stop)(sdw);
+ return -ENOTSUPP;
+}
+
+static inline int sdw_intel_stop_bus(struct sdw_intel *sdw, bool clock_stop)
+{
+ if (SDW_INTEL_CHECK_OPS(sdw, stop_bus))
+ return SDW_INTEL_OPS(sdw, stop_bus)(sdw, clock_stop);
+ return -ENOTSUPP;
+}
+
+static inline int sdw_intel_link_power_up(struct sdw_intel *sdw)
+{
+ if (SDW_INTEL_CHECK_OPS(sdw, link_power_up))
+ return SDW_INTEL_OPS(sdw, link_power_up)(sdw);
+ return -ENOTSUPP;
+}
+
+static inline int sdw_intel_link_power_down(struct sdw_intel *sdw)
+{
+ if (SDW_INTEL_CHECK_OPS(sdw, link_power_down))
+ return SDW_INTEL_OPS(sdw, link_power_down)(sdw);
+ return -ENOTSUPP;
+}
+
+static inline int sdw_intel_shim_check_wake(struct sdw_intel *sdw)
+{
+ if (SDW_INTEL_CHECK_OPS(sdw, shim_check_wake))
+ return SDW_INTEL_OPS(sdw, shim_check_wake)(sdw);
+ return -ENOTSUPP;
+}
+
+static inline void sdw_intel_shim_wake(struct sdw_intel *sdw, bool wake_enable)
+{
+ if (SDW_INTEL_CHECK_OPS(sdw, shim_wake))
+ SDW_INTEL_OPS(sdw, shim_wake)(sdw, wake_enable);
+}
#endif /* __SDW_INTEL_LOCAL_H */
diff --git a/drivers/soundwire/intel_auxdevice.c b/drivers/soundwire/intel_auxdevice.c
new file mode 100644
index 0000000..96c6b211
--- /dev/null
+++ b/drivers/soundwire/intel_auxdevice.c
@@ -0,0 +1,678 @@
+// SPDX-License-Identifier: (GPL-2.0 OR BSD-3-Clause)
+// Copyright(c) 2015-22 Intel Corporation.
+
+/*
+ * Soundwire Intel Manager Driver
+ */
+
+#include <linux/acpi.h>
+#include <linux/debugfs.h>
+#include <linux/delay.h>
+#include <linux/module.h>
+#include <linux/interrupt.h>
+#include <linux/io.h>
+#include <linux/auxiliary_bus.h>
+#include <sound/pcm_params.h>
+#include <linux/pm_runtime.h>
+#include <sound/soc.h>
+#include <linux/soundwire/sdw_registers.h>
+#include <linux/soundwire/sdw.h>
+#include <linux/soundwire/sdw_intel.h>
+#include "cadence_master.h"
+#include "bus.h"
+#include "intel.h"
+#include "intel_auxdevice.h"
+
+/* IDA min selected to avoid conflicts with HDaudio/iDISP SDI values */
+#define INTEL_DEV_NUM_IDA_MIN 4
+
+#define INTEL_MASTER_SUSPEND_DELAY_MS 3000
+
+/*
+ * debug/config flags for the Intel SoundWire Master.
+ *
+ * Since we may have multiple masters active, we can have up to 8
+ * flags reused in each byte, with master0 using the ls-byte, etc.
+ */
+
+#define SDW_INTEL_MASTER_DISABLE_PM_RUNTIME BIT(0)
+#define SDW_INTEL_MASTER_DISABLE_CLOCK_STOP BIT(1)
+#define SDW_INTEL_MASTER_DISABLE_PM_RUNTIME_IDLE BIT(2)
+#define SDW_INTEL_MASTER_DISABLE_MULTI_LINK BIT(3)
+
+static int md_flags;
+module_param_named(sdw_md_flags, md_flags, int, 0444);
+MODULE_PARM_DESC(sdw_md_flags, "SoundWire Intel Master device flags (0x0 all off)");
+
+static int generic_pre_bank_switch(struct sdw_bus *bus)
+{
+ struct sdw_cdns *cdns = bus_to_cdns(bus);
+ struct sdw_intel *sdw = cdns_to_intel(cdns);
+
+ return sdw->link_res->hw_ops->pre_bank_switch(sdw);
+}
+
+static int generic_post_bank_switch(struct sdw_bus *bus)
+{
+ struct sdw_cdns *cdns = bus_to_cdns(bus);
+ struct sdw_intel *sdw = cdns_to_intel(cdns);
+
+ return sdw->link_res->hw_ops->post_bank_switch(sdw);
+}
+
+static int sdw_master_read_intel_prop(struct sdw_bus *bus)
+{
+ struct sdw_master_prop *prop = &bus->prop;
+ struct fwnode_handle *link;
+ char name[32];
+ u32 quirk_mask;
+
+ /* Find master handle */
+ snprintf(name, sizeof(name),
+ "mipi-sdw-link-%d-subproperties", bus->link_id);
+
+ link = device_get_named_child_node(bus->dev, name);
+ if (!link) {
+ dev_err(bus->dev, "Master node %s not found\n", name);
+ return -EIO;
+ }
+
+ fwnode_property_read_u32(link,
+ "intel-sdw-ip-clock",
+ &prop->mclk_freq);
+
+ /* the values reported by BIOS are the 2x clock, not the bus clock */
+ prop->mclk_freq /= 2;
+
+ fwnode_property_read_u32(link,
+ "intel-quirk-mask",
+ &quirk_mask);
+
+ if (quirk_mask & SDW_INTEL_QUIRK_MASK_BUS_DISABLE)
+ prop->hw_disabled = true;
+
+ prop->quirks = SDW_MASTER_QUIRKS_CLEAR_INITIAL_CLASH |
+ SDW_MASTER_QUIRKS_CLEAR_INITIAL_PARITY;
+
+ return 0;
+}
+
+static int intel_prop_read(struct sdw_bus *bus)
+{
+ /* Initialize with default handler to read all DisCo properties */
+ sdw_master_read_prop(bus);
+
+ /* read Intel-specific properties */
+ sdw_master_read_intel_prop(bus);
+
+ return 0;
+}
+
+static struct sdw_master_ops sdw_intel_ops = {
+ .read_prop = intel_prop_read,
+ .override_adr = sdw_dmi_override_adr,
+ .xfer_msg = cdns_xfer_msg,
+ .xfer_msg_defer = cdns_xfer_msg_defer,
+ .reset_page_addr = cdns_reset_page_addr,
+ .set_bus_conf = cdns_bus_conf,
+ .pre_bank_switch = generic_pre_bank_switch,
+ .post_bank_switch = generic_post_bank_switch,
+ .read_ping_status = cdns_read_ping_status,
+};
+
+/*
+ * probe and init (aux_dev_id argument is required by function prototype but not used)
+ */
+static int intel_link_probe(struct auxiliary_device *auxdev,
+ const struct auxiliary_device_id *aux_dev_id)
+
+{
+ struct device *dev = &auxdev->dev;
+ struct sdw_intel_link_dev *ldev = auxiliary_dev_to_sdw_intel_link_dev(auxdev);
+ struct sdw_intel *sdw;
+ struct sdw_cdns *cdns;
+ struct sdw_bus *bus;
+ int ret;
+
+ sdw = devm_kzalloc(dev, sizeof(*sdw), GFP_KERNEL);
+ if (!sdw)
+ return -ENOMEM;
+
+ cdns = &sdw->cdns;
+ bus = &cdns->bus;
+
+ sdw->instance = auxdev->id;
+ sdw->link_res = &ldev->link_res;
+ cdns->dev = dev;
+ cdns->registers = sdw->link_res->registers;
+ cdns->instance = sdw->instance;
+ cdns->msg_count = 0;
+
+ bus->link_id = auxdev->id;
+ bus->dev_num_ida_min = INTEL_DEV_NUM_IDA_MIN;
+ bus->clk_stop_timeout = 1;
+
+ sdw_cdns_probe(cdns);
+
+ /* Set ops */
+ bus->ops = &sdw_intel_ops;
+
+ /* set driver data, accessed by snd_soc_dai_get_drvdata() */
+ auxiliary_set_drvdata(auxdev, cdns);
+
+ /* use generic bandwidth allocation algorithm */
+ sdw->cdns.bus.compute_params = sdw_compute_params;
+
+ /* avoid resuming from pm_runtime suspend if it's not required */
+ dev_pm_set_driver_flags(dev, DPM_FLAG_SMART_SUSPEND);
+
+ ret = sdw_bus_master_add(bus, dev, dev->fwnode);
+ if (ret) {
+ dev_err(dev, "sdw_bus_master_add fail: %d\n", ret);
+ return ret;
+ }
+
+ if (bus->prop.hw_disabled)
+ dev_info(dev,
+ "SoundWire master %d is disabled, will be ignored\n",
+ bus->link_id);
+ /*
+ * Ignore BIOS err_threshold, it's a really bad idea when dealing
+ * with multiple hardware synchronized links
+ */
+ bus->prop.err_threshold = 0;
+
+ return 0;
+}
+
+int intel_link_startup(struct auxiliary_device *auxdev)
+{
+ struct device *dev = &auxdev->dev;
+ struct sdw_cdns *cdns = auxiliary_get_drvdata(auxdev);
+ struct sdw_intel *sdw = cdns_to_intel(cdns);
+ struct sdw_bus *bus = &cdns->bus;
+ int link_flags;
+ bool multi_link;
+ u32 clock_stop_quirks;
+ int ret;
+
+ if (bus->prop.hw_disabled) {
+ dev_info(dev,
+ "SoundWire master %d is disabled, ignoring\n",
+ sdw->instance);
+ return 0;
+ }
+
+ link_flags = md_flags >> (bus->link_id * 8);
+ multi_link = !(link_flags & SDW_INTEL_MASTER_DISABLE_MULTI_LINK);
+ if (!multi_link) {
+ dev_dbg(dev, "Multi-link is disabled\n");
+ } else {
+ /*
+ * hardware-based synchronization is required regardless
+ * of the number of segments used by a stream: SSP-based
+ * synchronization is gated by gsync when the multi-master
+ * mode is set.
+ */
+ bus->hw_sync_min_links = 1;
+ }
+ bus->multi_link = multi_link;
+
+ /* Initialize shim, controller */
+ ret = sdw_intel_link_power_up(sdw);
+ if (ret)
+ goto err_init;
+
+ /* Register DAIs */
+ ret = sdw_intel_register_dai(sdw);
+ if (ret) {
+ dev_err(dev, "DAI registration failed: %d\n", ret);
+ goto err_power_up;
+ }
+
+ sdw_intel_debugfs_init(sdw);
+
+ /* start bus */
+ ret = sdw_intel_start_bus(sdw);
+ if (ret) {
+ dev_err(dev, "bus start failed: %d\n", ret);
+ goto err_power_up;
+ }
+
+ /* Enable runtime PM */
+ if (!(link_flags & SDW_INTEL_MASTER_DISABLE_PM_RUNTIME)) {
+ pm_runtime_set_autosuspend_delay(dev,
+ INTEL_MASTER_SUSPEND_DELAY_MS);
+ pm_runtime_use_autosuspend(dev);
+ pm_runtime_mark_last_busy(dev);
+
+ pm_runtime_set_active(dev);
+ pm_runtime_enable(dev);
+ }
+
+ clock_stop_quirks = sdw->link_res->clock_stop_quirks;
+ if (clock_stop_quirks & SDW_INTEL_CLK_STOP_NOT_ALLOWED) {
+ /*
+ * To keep the clock running we need to prevent
+ * pm_runtime suspend from happening by increasing the
+ * reference count.
+ * This quirk is specified by the parent PCI device in
+ * case of specific latency requirements. It will have
+ * no effect if pm_runtime is disabled by the user via
+ * a module parameter for testing purposes.
+ */
+ pm_runtime_get_noresume(dev);
+ }
+
+ /*
+ * The runtime PM status of Slave devices is "Unsupported"
+ * until they report as ATTACHED. If they don't, e.g. because
+ * there are no Slave devices populated or if the power-on is
+ * delayed or dependent on a power switch, the Master will
+ * remain active and prevent its parent from suspending.
+ *
+ * Conditionally force the pm_runtime core to re-evaluate the
+ * Master status in the absence of any Slave activity. A quirk
+ * is provided to e.g. deal with Slaves that may be powered on
+ * with a delay. A more complete solution would require the
+ * definition of Master properties.
+ */
+ if (!(link_flags & SDW_INTEL_MASTER_DISABLE_PM_RUNTIME_IDLE))
+ pm_runtime_idle(dev);
+
+ sdw->startup_done = true;
+ return 0;
+
+err_power_up:
+ sdw_intel_link_power_down(sdw);
+err_init:
+ return ret;
+}
+
+static void intel_link_remove(struct auxiliary_device *auxdev)
+{
+ struct sdw_cdns *cdns = auxiliary_get_drvdata(auxdev);
+ struct sdw_intel *sdw = cdns_to_intel(cdns);
+ struct sdw_bus *bus = &cdns->bus;
+
+ /*
+ * Since pm_runtime is already disabled, we don't decrease
+ * the refcount when the clock_stop_quirk is
+ * SDW_INTEL_CLK_STOP_NOT_ALLOWED
+ */
+ if (!bus->prop.hw_disabled) {
+ sdw_intel_debugfs_exit(sdw);
+ sdw_cdns_enable_interrupt(cdns, false);
+ }
+ sdw_bus_master_delete(bus);
+}
+
+int intel_link_process_wakeen_event(struct auxiliary_device *auxdev)
+{
+ struct device *dev = &auxdev->dev;
+ struct sdw_intel *sdw;
+ struct sdw_bus *bus;
+
+ sdw = auxiliary_get_drvdata(auxdev);
+ bus = &sdw->cdns.bus;
+
+ if (bus->prop.hw_disabled || !sdw->startup_done) {
+ dev_dbg(dev, "SoundWire master %d is disabled or not-started, ignoring\n",
+ bus->link_id);
+ return 0;
+ }
+
+ if (!sdw_intel_shim_check_wake(sdw))
+ return 0;
+
+ /* disable WAKEEN interrupt ASAP to prevent interrupt flood */
+ sdw_intel_shim_wake(sdw, false);
+
+ /*
+ * resume the Master, which will generate a bus reset and result in
+ * Slaves re-attaching and be re-enumerated. The SoundWire physical
+ * device which generated the wake will trigger an interrupt, which
+ * will in turn cause the corresponding Linux Slave device to be
+ * resumed and the Slave codec driver to check the status.
+ */
+ pm_request_resume(dev);
+
+ return 0;
+}
+
+/*
+ * PM calls
+ */
+
+static int intel_resume_child_device(struct device *dev, void *data)
+{
+ int ret;
+ struct sdw_slave *slave = dev_to_sdw_dev(dev);
+
+ if (!slave->probed) {
+ dev_dbg(dev, "skipping device, no probed driver\n");
+ return 0;
+ }
+ if (!slave->dev_num_sticky) {
+ dev_dbg(dev, "skipping device, never detected on bus\n");
+ return 0;
+ }
+
+ ret = pm_request_resume(dev);
+ if (ret < 0)
+ dev_err(dev, "%s: pm_request_resume failed: %d\n", __func__, ret);
+
+ return ret;
+}
+
+static int __maybe_unused intel_pm_prepare(struct device *dev)
+{
+ struct sdw_cdns *cdns = dev_get_drvdata(dev);
+ struct sdw_intel *sdw = cdns_to_intel(cdns);
+ struct sdw_bus *bus = &cdns->bus;
+ u32 clock_stop_quirks;
+ int ret;
+
+ if (bus->prop.hw_disabled || !sdw->startup_done) {
+ dev_dbg(dev, "SoundWire master %d is disabled or not-started, ignoring\n",
+ bus->link_id);
+ return 0;
+ }
+
+ clock_stop_quirks = sdw->link_res->clock_stop_quirks;
+
+ if (pm_runtime_suspended(dev) &&
+ pm_runtime_suspended(dev->parent) &&
+ ((clock_stop_quirks & SDW_INTEL_CLK_STOP_BUS_RESET) ||
+ !clock_stop_quirks)) {
+ /*
+ * if we've enabled clock stop, and the parent is suspended, the SHIM registers
+ * are not accessible and the shim wake cannot be disabled.
+ * The only solution is to resume the entire bus to full power
+ */
+
+ /*
+ * If any operation in this block fails, we keep going since we don't want
+ * to prevent system suspend from happening and errors should be recoverable
+ * on resume.
+ */
+
+ /*
+ * first resume the device for this link. This will also by construction
+ * resume the PCI parent device.
+ */
+ ret = pm_request_resume(dev);
+ if (ret < 0) {
+ dev_err(dev, "%s: pm_request_resume failed: %d\n", __func__, ret);
+ return 0;
+ }
+
+ /*
+ * Continue resuming the entire bus (parent + child devices) to exit
+ * the clock stop mode. If there are no devices connected on this link
+ * this is a no-op.
+ * The resume to full power could have been implemented with a .prepare
+ * step in SoundWire codec drivers. This would however require a lot
+ * of code to handle an Intel-specific corner case. It is simpler in
+ * practice to add a loop at the link level.
+ */
+ ret = device_for_each_child(bus->dev, NULL, intel_resume_child_device);
+
+ if (ret < 0)
+ dev_err(dev, "%s: intel_resume_child_device failed: %d\n", __func__, ret);
+ }
+
+ return 0;
+}
+
+static int __maybe_unused intel_suspend(struct device *dev)
+{
+ struct sdw_cdns *cdns = dev_get_drvdata(dev);
+ struct sdw_intel *sdw = cdns_to_intel(cdns);
+ struct sdw_bus *bus = &cdns->bus;
+ u32 clock_stop_quirks;
+ int ret;
+
+ if (bus->prop.hw_disabled || !sdw->startup_done) {
+ dev_dbg(dev, "SoundWire master %d is disabled or not-started, ignoring\n",
+ bus->link_id);
+ return 0;
+ }
+
+ if (pm_runtime_suspended(dev)) {
+ dev_dbg(dev, "pm_runtime status: suspended\n");
+
+ clock_stop_quirks = sdw->link_res->clock_stop_quirks;
+
+ if ((clock_stop_quirks & SDW_INTEL_CLK_STOP_BUS_RESET) ||
+ !clock_stop_quirks) {
+
+ if (pm_runtime_suspended(dev->parent)) {
+ /*
+ * paranoia check: this should not happen with the .prepare
+ * resume to full power
+ */
+ dev_err(dev, "%s: invalid config: parent is suspended\n", __func__);
+ } else {
+ sdw_intel_shim_wake(sdw, false);
+ }
+ }
+
+ return 0;
+ }
+
+ ret = sdw_intel_stop_bus(sdw, false);
+ if (ret < 0) {
+ dev_err(dev, "%s: cannot stop bus: %d\n", __func__, ret);
+ return ret;
+ }
+
+ return 0;
+}
+
+static int __maybe_unused intel_suspend_runtime(struct device *dev)
+{
+ struct sdw_cdns *cdns = dev_get_drvdata(dev);
+ struct sdw_intel *sdw = cdns_to_intel(cdns);
+ struct sdw_bus *bus = &cdns->bus;
+ u32 clock_stop_quirks;
+ int ret;
+
+ if (bus->prop.hw_disabled || !sdw->startup_done) {
+ dev_dbg(dev, "SoundWire master %d is disabled or not-started, ignoring\n",
+ bus->link_id);
+ return 0;
+ }
+
+ clock_stop_quirks = sdw->link_res->clock_stop_quirks;
+
+ if (clock_stop_quirks & SDW_INTEL_CLK_STOP_TEARDOWN) {
+ ret = sdw_intel_stop_bus(sdw, false);
+ if (ret < 0) {
+ dev_err(dev, "%s: cannot stop bus during teardown: %d\n",
+ __func__, ret);
+ return ret;
+ }
+ } else if (clock_stop_quirks & SDW_INTEL_CLK_STOP_BUS_RESET || !clock_stop_quirks) {
+ ret = sdw_intel_stop_bus(sdw, true);
+ if (ret < 0) {
+ dev_err(dev, "%s: cannot stop bus during clock_stop: %d\n",
+ __func__, ret);
+ return ret;
+ }
+ } else {
+ dev_err(dev, "%s clock_stop_quirks %x unsupported\n",
+ __func__, clock_stop_quirks);
+ ret = -EINVAL;
+ }
+
+ return ret;
+}
+
+static int __maybe_unused intel_resume(struct device *dev)
+{
+ struct sdw_cdns *cdns = dev_get_drvdata(dev);
+ struct sdw_intel *sdw = cdns_to_intel(cdns);
+ struct sdw_bus *bus = &cdns->bus;
+ int link_flags;
+ int ret;
+
+ if (bus->prop.hw_disabled || !sdw->startup_done) {
+ dev_dbg(dev, "SoundWire master %d is disabled or not-started, ignoring\n",
+ bus->link_id);
+ return 0;
+ }
+
+ link_flags = md_flags >> (bus->link_id * 8);
+
+ if (pm_runtime_suspended(dev)) {
+ dev_dbg(dev, "pm_runtime status was suspended, forcing active\n");
+
+ /* follow required sequence from runtime_pm.rst */
+ pm_runtime_disable(dev);
+ pm_runtime_set_active(dev);
+ pm_runtime_mark_last_busy(dev);
+ pm_runtime_enable(dev);
+
+ link_flags = md_flags >> (bus->link_id * 8);
+
+ if (!(link_flags & SDW_INTEL_MASTER_DISABLE_PM_RUNTIME_IDLE))
+ pm_runtime_idle(dev);
+ }
+
+ ret = sdw_intel_link_power_up(sdw);
+ if (ret) {
+ dev_err(dev, "%s failed: %d\n", __func__, ret);
+ return ret;
+ }
+
+ /*
+ * make sure all Slaves are tagged as UNATTACHED and provide
+ * reason for reinitialization
+ */
+ sdw_clear_slave_status(bus, SDW_UNATTACH_REQUEST_MASTER_RESET);
+
+ ret = sdw_intel_start_bus(sdw);
+ if (ret < 0) {
+ dev_err(dev, "cannot start bus during resume\n");
+ sdw_intel_link_power_down(sdw);
+ return ret;
+ }
+
+ /*
+ * after system resume, the pm_runtime suspend() may kick in
+ * during the enumeration, before any children device force the
+ * master device to remain active. Using pm_runtime_get()
+ * routines is not really possible, since it'd prevent the
+ * master from suspending.
+ * A reasonable compromise is to update the pm_runtime
+ * counters and delay the pm_runtime suspend by several
+ * seconds, by when all enumeration should be complete.
+ */
+ pm_runtime_mark_last_busy(dev);
+
+ return 0;
+}
+
+static int __maybe_unused intel_resume_runtime(struct device *dev)
+{
+ struct sdw_cdns *cdns = dev_get_drvdata(dev);
+ struct sdw_intel *sdw = cdns_to_intel(cdns);
+ struct sdw_bus *bus = &cdns->bus;
+ u32 clock_stop_quirks;
+ int ret;
+
+ if (bus->prop.hw_disabled || !sdw->startup_done) {
+ dev_dbg(dev, "SoundWire master %d is disabled or not-started, ignoring\n",
+ bus->link_id);
+ return 0;
+ }
+
+ /* unconditionally disable WAKEEN interrupt */
+ sdw_intel_shim_wake(sdw, false);
+
+ clock_stop_quirks = sdw->link_res->clock_stop_quirks;
+
+ if (clock_stop_quirks & SDW_INTEL_CLK_STOP_TEARDOWN) {
+ ret = sdw_intel_link_power_up(sdw);
+ if (ret) {
+ dev_err(dev, "%s: power_up failed after teardown: %d\n", __func__, ret);
+ return ret;
+ }
+
+ /*
+ * make sure all Slaves are tagged as UNATTACHED and provide
+ * reason for reinitialization
+ */
+ sdw_clear_slave_status(bus, SDW_UNATTACH_REQUEST_MASTER_RESET);
+
+ ret = sdw_intel_start_bus(sdw);
+ if (ret < 0) {
+ dev_err(dev, "%s: cannot start bus after teardown: %d\n", __func__, ret);
+ sdw_intel_link_power_down(sdw);
+ return ret;
+ }
+
+ } else if (clock_stop_quirks & SDW_INTEL_CLK_STOP_BUS_RESET) {
+ ret = sdw_intel_link_power_up(sdw);
+ if (ret) {
+ dev_err(dev, "%s: power_up failed after bus reset: %d\n", __func__, ret);
+ return ret;
+ }
+
+ ret = sdw_intel_start_bus_after_reset(sdw);
+ if (ret < 0) {
+ dev_err(dev, "%s: cannot start bus after reset: %d\n", __func__, ret);
+ sdw_intel_link_power_down(sdw);
+ return ret;
+ }
+ } else if (!clock_stop_quirks) {
+
+ sdw_intel_check_clock_stop(sdw);
+
+ ret = sdw_intel_link_power_up(sdw);
+ if (ret) {
+ dev_err(dev, "%s: power_up failed: %d\n", __func__, ret);
+ return ret;
+ }
+
+ ret = sdw_intel_start_bus_after_clock_stop(sdw);
+ if (ret < 0) {
+ dev_err(dev, "%s: cannot start bus after clock stop: %d\n", __func__, ret);
+ sdw_intel_link_power_down(sdw);
+ return ret;
+ }
+ } else {
+ dev_err(dev, "%s: clock_stop_quirks %x unsupported\n",
+ __func__, clock_stop_quirks);
+ ret = -EINVAL;
+ }
+
+ return ret;
+}
+
+static const struct dev_pm_ops intel_pm = {
+ .prepare = intel_pm_prepare,
+ SET_SYSTEM_SLEEP_PM_OPS(intel_suspend, intel_resume)
+ SET_RUNTIME_PM_OPS(intel_suspend_runtime, intel_resume_runtime, NULL)
+};
+
+static const struct auxiliary_device_id intel_link_id_table[] = {
+ { .name = "soundwire_intel.link" },
+ {},
+};
+MODULE_DEVICE_TABLE(auxiliary, intel_link_id_table);
+
+static struct auxiliary_driver sdw_intel_drv = {
+ .probe = intel_link_probe,
+ .remove = intel_link_remove,
+ .driver = {
+ /* auxiliary_driver_register() sets .name to be the modname */
+ .pm = &intel_pm,
+ },
+ .id_table = intel_link_id_table
+};
+module_auxiliary_driver(sdw_intel_drv);
+
+MODULE_LICENSE("Dual BSD/GPL");
+MODULE_DESCRIPTION("Intel Soundwire Link Driver");
diff --git a/drivers/soundwire/intel_auxdevice.h b/drivers/soundwire/intel_auxdevice.h
new file mode 100644
index 0000000..a00ecde
--- /dev/null
+++ b/drivers/soundwire/intel_auxdevice.h
@@ -0,0 +1,18 @@
+/* SPDX-License-Identifier: (GPL-2.0 OR BSD-3-Clause) */
+/* Copyright(c) 2015-2022 Intel Corporation. */
+
+#ifndef __SDW_INTEL_AUXDEVICE_H
+#define __SDW_INTEL_AUXDEVICE_H
+
+int intel_link_startup(struct auxiliary_device *auxdev);
+int intel_link_process_wakeen_event(struct auxiliary_device *auxdev);
+
+struct sdw_intel_link_dev {
+ struct auxiliary_device auxdev;
+ struct sdw_intel_link_res link_res;
+};
+
+#define auxiliary_dev_to_sdw_intel_link_dev(auxiliary_dev) \
+ container_of(auxiliary_dev, struct sdw_intel_link_dev, auxdev)
+
+#endif /* __SDW_INTEL_AUXDEVICE_H */
diff --git a/drivers/soundwire/intel_init.c b/drivers/soundwire/intel_init.c
index d684292..cbe56b9 100644
--- a/drivers/soundwire/intel_init.c
+++ b/drivers/soundwire/intel_init.c
@@ -17,6 +17,7 @@
#include <linux/soundwire/sdw_intel.h>
#include "cadence_master.h"
#include "intel.h"
+#include "intel_auxdevice.h"
static void intel_link_dev_release(struct device *dev)
{
@@ -60,6 +61,7 @@ static struct sdw_intel_link_dev *intel_link_dev_register(struct sdw_intel_res *
/* Add link information used in the driver probe */
link = &ldev->link_res;
+ link->hw_ops = res->hw_ops;
link->mmio_base = res->mmio_base;
link->registers = res->mmio_base + SDW_LINK_BASE
+ (SDW_LINK_SIZE * link_id);
diff --git a/drivers/soundwire/qcom.c b/drivers/soundwire/qcom.c
index cee2b22..3354248 100644
--- a/drivers/soundwire/qcom.c
+++ b/drivers/soundwire/qcom.c
@@ -25,6 +25,8 @@
#define SWRM_COMP_SW_RESET 0x008
#define SWRM_COMP_STATUS 0x014
+#define SWRM_LINK_MANAGER_EE 0x018
+#define SWRM_EE_CPU 1
#define SWRM_FRM_GEN_ENABLED BIT(0)
#define SWRM_COMP_HW_VERSION 0x00
#define SWRM_COMP_CFG_ADDR 0x04
@@ -104,7 +106,6 @@
#define SWRM_REG_VAL_PACK(data, dev, id, reg) \
((reg) | ((id) << 16) | ((dev) << 20) | ((data) << 24))
-#define SWRM_SPECIAL_CMD_ID 0xF
#define MAX_FREQ_NUM 1
#define TIMEOUT_MS 100
#define QCOM_SWRM_MAX_RD_LEN 0x1
@@ -694,7 +695,14 @@ static int qcom_swrm_init(struct qcom_swrm_ctrl *ctrl)
u32p_replace_bits(&val, SWRM_DEF_CMD_NO_PINGS, SWRM_MCP_CFG_MAX_NUM_OF_CMD_NO_PINGS_BMSK);
ctrl->reg_write(ctrl, SWRM_MCP_CFG_ADDR, val);
- ctrl->reg_write(ctrl, SWRM_MCP_BUS_CTRL, SWRM_MCP_BUS_CLK_START);
+ if (ctrl->version >= 0x01070000) {
+ ctrl->reg_write(ctrl, SWRM_LINK_MANAGER_EE, SWRM_EE_CPU);
+ ctrl->reg_write(ctrl, SWRM_MCP_BUS_CTRL,
+ SWRM_MCP_BUS_CLK_START << SWRM_EE_CPU);
+ } else {
+ ctrl->reg_write(ctrl, SWRM_MCP_BUS_CTRL, SWRM_MCP_BUS_CLK_START);
+ }
+
/* Configure number of retries of a read/write cmd */
if (ctrl->version > 0x01050001) {
/* Only for versions >= 1.5.1 */
@@ -1331,8 +1339,8 @@ static int qcom_swrm_probe(struct platform_device *pdev)
}
if (data->sw_clk_gate_required) {
- ctrl->audio_cgcr = devm_reset_control_get_exclusive(dev, "swr_audio_cgcr");
- if (IS_ERR_OR_NULL(ctrl->audio_cgcr)) {
+ ctrl->audio_cgcr = devm_reset_control_get_optional_exclusive(dev, "swr_audio_cgcr");
+ if (IS_ERR(ctrl->audio_cgcr)) {
dev_err(dev, "Failed to get cgcr reset ctrl required for SW gating\n");
ret = PTR_ERR(ctrl->audio_cgcr);
goto err_init;
@@ -1519,7 +1527,13 @@ static int __maybe_unused swrm_runtime_resume(struct device *dev)
} else {
reset_control_reset(ctrl->audio_cgcr);
- ctrl->reg_write(ctrl, SWRM_MCP_BUS_CTRL, SWRM_MCP_BUS_CLK_START);
+ if (ctrl->version >= 0x01070000) {
+ ctrl->reg_write(ctrl, SWRM_LINK_MANAGER_EE, SWRM_EE_CPU);
+ ctrl->reg_write(ctrl, SWRM_MCP_BUS_CTRL,
+ SWRM_MCP_BUS_CLK_START << SWRM_EE_CPU);
+ } else {
+ ctrl->reg_write(ctrl, SWRM_MCP_BUS_CTRL, SWRM_MCP_BUS_CLK_START);
+ }
ctrl->reg_write(ctrl, SWRM_INTERRUPT_CLEAR,
SWRM_INTERRUPT_STATUS_MASTER_CLASH_DET);
@@ -1583,6 +1597,7 @@ static const struct of_device_id qcom_swrm_of_match[] = {
{ .compatible = "qcom,soundwire-v1.3.0", .data = &swrm_v1_3_data },
{ .compatible = "qcom,soundwire-v1.5.1", .data = &swrm_v1_5_data },
{ .compatible = "qcom,soundwire-v1.6.0", .data = &swrm_v1_6_data },
+ { .compatible = "qcom,soundwire-v1.7.0", .data = &swrm_v1_5_data },
{/* sentinel */},
};
diff --git a/drivers/watchdog/Kconfig b/drivers/watchdog/Kconfig
index b64bc49..0bc40b7 100644
--- a/drivers/watchdog/Kconfig
+++ b/drivers/watchdog/Kconfig
@@ -1055,6 +1055,13 @@
feature. More information can be found at
<https://www.advantech.com.tw/products/>
+config ADVANTECH_EC_WDT
+ tristate "Advantech Embedded Controller Watchdog Timer"
+ depends on X86
+ help
+ This driver supports Advantech products with ITE based Embedded Controller.
+ It does not support Advantech products with other ECs or without EC.
+
config ALIM1535_WDT
tristate "ALi M1535 PMU Watchdog Timer"
depends on X86 && PCI
diff --git a/drivers/watchdog/Makefile b/drivers/watchdog/Makefile
index d41e5f8..9cbf658 100644
--- a/drivers/watchdog/Makefile
+++ b/drivers/watchdog/Makefile
@@ -102,6 +102,7 @@
# X86 (i386 + ia64 + x86_64) Architecture
obj-$(CONFIG_ACQUIRE_WDT) += acquirewdt.o
obj-$(CONFIG_ADVANTECH_WDT) += advantechwdt.o
+obj-$(CONFIG_ADVANTECH_EC_WDT) += advantech_ec_wdt.o
obj-$(CONFIG_ALIM1535_WDT) += alim1535_wdt.o
obj-$(CONFIG_ALIM7101_WDT) += alim7101_wdt.o
obj-$(CONFIG_EBC_C384_WDT) += ebc-c384_wdt.o
diff --git a/drivers/watchdog/advantech_ec_wdt.c b/drivers/watchdog/advantech_ec_wdt.c
new file mode 100644
index 0000000..7c380f9
--- /dev/null
+++ b/drivers/watchdog/advantech_ec_wdt.c
@@ -0,0 +1,205 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Advantech Embedded Controller Watchdog Driver
+ *
+ * This driver supports Advantech products with ITE based Embedded Controller.
+ * It does not support Advantech products with other ECs or without EC.
+ *
+ * Copyright (C) 2022 Advantech Europe B.V.
+ */
+
+#include <linux/delay.h>
+#include <linux/io.h>
+#include <linux/isa.h>
+#include <linux/kernel.h>
+#include <linux/module.h>
+#include <linux/moduleparam.h>
+#include <linux/watchdog.h>
+
+#define DRIVER_NAME "advantech_ec_wdt"
+
+/* EC IO region */
+#define EC_BASE_ADDR 0x299
+#define EC_ADDR_EXTENT 2
+
+/* EC minimum IO access delay in ms */
+#define EC_MIN_DELAY 10
+
+/* EC interface definitions */
+#define EC_ADDR_CMD (EC_BASE_ADDR + 1)
+#define EC_ADDR_DATA EC_BASE_ADDR
+#define EC_CMD_EC_PROBE 0x30
+#define EC_CMD_COMM 0x89
+#define EC_CMD_WDT_START 0x28
+#define EC_CMD_WDT_STOP 0x29
+#define EC_CMD_WDT_RESET 0x2A
+#define EC_DAT_EN_DLY_H 0x58
+#define EC_DAT_EN_DLY_L 0x59
+#define EC_DAT_RST_DLY_H 0x5E
+#define EC_DAT_RST_DLY_L 0x5F
+#define EC_MAGIC 0x95
+
+/* module parameters */
+#define MIN_TIME 1
+#define MAX_TIME 6000 /* 100 minutes */
+#define DEFAULT_TIME 60
+
+static unsigned int timeout;
+static ktime_t ec_timestamp;
+
+module_param(timeout, uint, 0);
+MODULE_PARM_DESC(timeout,
+ "Default Watchdog timer setting (" __MODULE_STRING(DEFAULT_TIME) "s). The range is from " __MODULE_STRING(MIN_TIME) " to " __MODULE_STRING(MAX_TIME) ".");
+
+static void adv_ec_wdt_timing_gate(void)
+{
+ ktime_t time_cur, time_delta;
+
+ /* ensure minimum delay between IO accesses*/
+ time_cur = ktime_get();
+ time_delta = ktime_to_ms(ktime_sub(time_cur, ec_timestamp));
+ if (time_delta < EC_MIN_DELAY) {
+ time_delta = EC_MIN_DELAY - time_delta;
+ usleep_range(time_delta * 1000, (time_delta + 1) * 1000);
+ }
+ ec_timestamp = ktime_get();
+}
+
+static void adv_ec_wdt_outb(unsigned char value, unsigned short port)
+{
+ adv_ec_wdt_timing_gate();
+ outb(value, port);
+}
+
+static unsigned char adv_ec_wdt_inb(unsigned short port)
+{
+ adv_ec_wdt_timing_gate();
+ return inb(port);
+}
+
+static int adv_ec_wdt_ping(struct watchdog_device *wdd)
+{
+ adv_ec_wdt_outb(EC_CMD_WDT_RESET, EC_ADDR_CMD);
+ return 0;
+}
+
+static int adv_ec_wdt_set_timeout(struct watchdog_device *wdd, unsigned int t)
+{
+ unsigned int val;
+
+ /* scale time to EC 100 ms base */
+ val = t * 10;
+
+ /* reset enable delay, just in case it was set by BIOS etc. */
+ adv_ec_wdt_outb(EC_CMD_COMM, EC_ADDR_CMD);
+ adv_ec_wdt_outb(EC_DAT_EN_DLY_H, EC_ADDR_DATA);
+ adv_ec_wdt_outb(0, EC_ADDR_DATA);
+
+ adv_ec_wdt_outb(EC_CMD_COMM, EC_ADDR_CMD);
+ adv_ec_wdt_outb(EC_DAT_EN_DLY_L, EC_ADDR_DATA);
+ adv_ec_wdt_outb(0, EC_ADDR_DATA);
+
+ /* set reset delay */
+ adv_ec_wdt_outb(EC_CMD_COMM, EC_ADDR_CMD);
+ adv_ec_wdt_outb(EC_DAT_RST_DLY_H, EC_ADDR_DATA);
+ adv_ec_wdt_outb(val >> 8, EC_ADDR_DATA);
+
+ adv_ec_wdt_outb(EC_CMD_COMM, EC_ADDR_CMD);
+ adv_ec_wdt_outb(EC_DAT_RST_DLY_L, EC_ADDR_DATA);
+ adv_ec_wdt_outb(val & 0xFF, EC_ADDR_DATA);
+
+ wdd->timeout = t;
+ return 0;
+}
+
+static int adv_ec_wdt_start(struct watchdog_device *wdd)
+{
+ adv_ec_wdt_set_timeout(wdd, wdd->timeout);
+ adv_ec_wdt_outb(EC_CMD_WDT_START, EC_ADDR_CMD);
+
+ return 0;
+}
+
+static int adv_ec_wdt_stop(struct watchdog_device *wdd)
+{
+ adv_ec_wdt_outb(EC_CMD_WDT_STOP, EC_ADDR_CMD);
+
+ return 0;
+}
+
+static const struct watchdog_info adv_ec_wdt_info = {
+ .identity = DRIVER_NAME,
+ .options = WDIOF_SETTIMEOUT |
+ WDIOF_MAGICCLOSE |
+ WDIOF_KEEPALIVEPING,
+};
+
+static const struct watchdog_ops adv_ec_wdt_ops = {
+ .owner = THIS_MODULE,
+ .start = adv_ec_wdt_start,
+ .stop = adv_ec_wdt_stop,
+ .ping = adv_ec_wdt_ping,
+ .set_timeout = adv_ec_wdt_set_timeout,
+};
+
+static struct watchdog_device adv_ec_wdt_dev = {
+ .info = &adv_ec_wdt_info,
+ .ops = &adv_ec_wdt_ops,
+ .min_timeout = MIN_TIME,
+ .max_timeout = MAX_TIME,
+ .timeout = DEFAULT_TIME,
+};
+
+static int adv_ec_wdt_probe(struct device *dev, unsigned int id)
+{
+ if (!devm_request_region(dev, EC_BASE_ADDR, EC_ADDR_EXTENT, dev_name(dev))) {
+ dev_err(dev, "Unable to lock port addresses (0x%X-0x%X)\n",
+ EC_BASE_ADDR, EC_BASE_ADDR + EC_ADDR_EXTENT);
+ return -EBUSY;
+ }
+
+ watchdog_init_timeout(&adv_ec_wdt_dev, timeout, dev);
+ watchdog_stop_on_reboot(&adv_ec_wdt_dev);
+ watchdog_stop_on_unregister(&adv_ec_wdt_dev);
+
+ return devm_watchdog_register_device(dev, &adv_ec_wdt_dev);
+}
+
+static struct isa_driver adv_ec_wdt_driver = {
+ .probe = adv_ec_wdt_probe,
+ .driver = {
+ .name = DRIVER_NAME,
+ },
+};
+
+static int __init adv_ec_wdt_init(void)
+{
+ unsigned int val;
+
+ /* quick probe for EC */
+ if (!request_region(EC_BASE_ADDR, EC_ADDR_EXTENT, DRIVER_NAME))
+ return -EBUSY;
+
+ adv_ec_wdt_outb(EC_CMD_EC_PROBE, EC_ADDR_CMD);
+ val = adv_ec_wdt_inb(EC_ADDR_DATA);
+ release_region(EC_BASE_ADDR, EC_ADDR_EXTENT);
+
+ if (val != EC_MAGIC)
+ return -ENODEV;
+
+ return isa_register_driver(&adv_ec_wdt_driver, 1);
+}
+
+static void __exit adv_ec_wdt_exit(void)
+{
+ isa_unregister_driver(&adv_ec_wdt_driver);
+}
+
+module_init(adv_ec_wdt_init);
+module_exit(adv_ec_wdt_exit);
+
+MODULE_AUTHOR("Thomas Kastner <thomas.kastner@advantech.com>");
+MODULE_DESCRIPTION("Advantech Embedded Controller Watchdog Device Driver");
+MODULE_LICENSE("GPL");
+MODULE_VERSION("20221019");
+MODULE_ALIAS("isa:" DRIVER_NAME);
diff --git a/drivers/watchdog/aspeed_wdt.c b/drivers/watchdog/aspeed_wdt.c
index 0cff2ad..86b5331 100644
--- a/drivers/watchdog/aspeed_wdt.c
+++ b/drivers/watchdog/aspeed_wdt.c
@@ -5,11 +5,14 @@
* Joel Stanley <joel@jms.id.au>
*/
+#include <linux/bits.h>
#include <linux/delay.h>
+#include <linux/interrupt.h>
#include <linux/io.h>
#include <linux/kernel.h>
#include <linux/module.h>
#include <linux/of.h>
+#include <linux/of_irq.h>
#include <linux/platform_device.h>
#include <linux/watchdog.h>
@@ -18,28 +21,41 @@ module_param(nowayout, bool, 0);
MODULE_PARM_DESC(nowayout, "Watchdog cannot be stopped once started (default="
__MODULE_STRING(WATCHDOG_NOWAYOUT) ")");
+struct aspeed_wdt_config {
+ u32 ext_pulse_width_mask;
+ u32 irq_shift;
+ u32 irq_mask;
+};
+
struct aspeed_wdt {
struct watchdog_device wdd;
void __iomem *base;
u32 ctrl;
-};
-
-struct aspeed_wdt_config {
- u32 ext_pulse_width_mask;
+ const struct aspeed_wdt_config *cfg;
};
static const struct aspeed_wdt_config ast2400_config = {
.ext_pulse_width_mask = 0xff,
+ .irq_shift = 0,
+ .irq_mask = 0,
};
static const struct aspeed_wdt_config ast2500_config = {
.ext_pulse_width_mask = 0xfffff,
+ .irq_shift = 12,
+ .irq_mask = GENMASK(31, 12),
+};
+
+static const struct aspeed_wdt_config ast2600_config = {
+ .ext_pulse_width_mask = 0xfffff,
+ .irq_shift = 0,
+ .irq_mask = GENMASK(31, 10),
};
static const struct of_device_id aspeed_wdt_of_table[] = {
{ .compatible = "aspeed,ast2400-wdt", .data = &ast2400_config },
{ .compatible = "aspeed,ast2500-wdt", .data = &ast2500_config },
- { .compatible = "aspeed,ast2600-wdt", .data = &ast2500_config },
+ { .compatible = "aspeed,ast2600-wdt", .data = &ast2600_config },
{ },
};
MODULE_DEVICE_TABLE(of, aspeed_wdt_of_table);
@@ -58,6 +74,7 @@ MODULE_DEVICE_TABLE(of, aspeed_wdt_of_table);
#define WDT_CTRL_RESET_SYSTEM BIT(1)
#define WDT_CTRL_ENABLE BIT(0)
#define WDT_TIMEOUT_STATUS 0x10
+#define WDT_TIMEOUT_STATUS_IRQ BIT(2)
#define WDT_TIMEOUT_STATUS_BOOT_SECONDARY BIT(1)
#define WDT_CLEAR_TIMEOUT_STATUS 0x14
#define WDT_CLEAR_TIMEOUT_AND_BOOT_CODE_SELECTION BIT(0)
@@ -160,6 +177,26 @@ static int aspeed_wdt_set_timeout(struct watchdog_device *wdd,
return 0;
}
+static int aspeed_wdt_set_pretimeout(struct watchdog_device *wdd,
+ unsigned int pretimeout)
+{
+ struct aspeed_wdt *wdt = to_aspeed_wdt(wdd);
+ u32 actual = pretimeout * WDT_RATE_1MHZ;
+ u32 s = wdt->cfg->irq_shift;
+ u32 m = wdt->cfg->irq_mask;
+
+ wdd->pretimeout = pretimeout;
+ wdt->ctrl &= ~m;
+ if (pretimeout)
+ wdt->ctrl |= ((actual << s) & m) | WDT_CTRL_WDT_INTR;
+ else
+ wdt->ctrl &= ~WDT_CTRL_WDT_INTR;
+
+ writel(wdt->ctrl, wdt->base + WDT_CTRL);
+
+ return 0;
+}
+
static int aspeed_wdt_restart(struct watchdog_device *wdd,
unsigned long action, void *data)
{
@@ -232,6 +269,7 @@ static const struct watchdog_ops aspeed_wdt_ops = {
.stop = aspeed_wdt_stop,
.ping = aspeed_wdt_ping,
.set_timeout = aspeed_wdt_set_timeout,
+ .set_pretimeout = aspeed_wdt_set_pretimeout,
.restart = aspeed_wdt_restart,
.owner = THIS_MODULE,
};
@@ -243,10 +281,29 @@ static const struct watchdog_info aspeed_wdt_info = {
.identity = KBUILD_MODNAME,
};
+static const struct watchdog_info aspeed_wdt_pretimeout_info = {
+ .options = WDIOF_KEEPALIVEPING
+ | WDIOF_PRETIMEOUT
+ | WDIOF_MAGICCLOSE
+ | WDIOF_SETTIMEOUT,
+ .identity = KBUILD_MODNAME,
+};
+
+static irqreturn_t aspeed_wdt_irq(int irq, void *arg)
+{
+ struct watchdog_device *wdd = arg;
+ struct aspeed_wdt *wdt = to_aspeed_wdt(wdd);
+ u32 status = readl(wdt->base + WDT_TIMEOUT_STATUS);
+
+ if (status & WDT_TIMEOUT_STATUS_IRQ)
+ watchdog_notify_pretimeout(wdd);
+
+ return IRQ_HANDLED;
+}
+
static int aspeed_wdt_probe(struct platform_device *pdev)
{
struct device *dev = &pdev->dev;
- const struct aspeed_wdt_config *config;
const struct of_device_id *ofdid;
struct aspeed_wdt *wdt;
struct device_node *np;
@@ -259,11 +316,33 @@ static int aspeed_wdt_probe(struct platform_device *pdev)
if (!wdt)
return -ENOMEM;
+ np = dev->of_node;
+
+ ofdid = of_match_node(aspeed_wdt_of_table, np);
+ if (!ofdid)
+ return -EINVAL;
+ wdt->cfg = ofdid->data;
+
wdt->base = devm_platform_ioremap_resource(pdev, 0);
if (IS_ERR(wdt->base))
return PTR_ERR(wdt->base);
wdt->wdd.info = &aspeed_wdt_info;
+
+ if (wdt->cfg->irq_mask) {
+ int irq = platform_get_irq_optional(pdev, 0);
+
+ if (irq > 0) {
+ ret = devm_request_irq(dev, irq, aspeed_wdt_irq,
+ IRQF_SHARED, dev_name(dev),
+ wdt);
+ if (ret)
+ return ret;
+
+ wdt->wdd.info = &aspeed_wdt_pretimeout_info;
+ }
+ }
+
wdt->wdd.ops = &aspeed_wdt_ops;
wdt->wdd.max_hw_heartbeat_ms = WDT_MAX_TIMEOUT_MS;
wdt->wdd.parent = dev;
@@ -273,13 +352,6 @@ static int aspeed_wdt_probe(struct platform_device *pdev)
watchdog_set_nowayout(&wdt->wdd, nowayout);
- np = dev->of_node;
-
- ofdid = of_match_node(aspeed_wdt_of_table, np);
- if (!ofdid)
- return -EINVAL;
- config = ofdid->data;
-
/*
* On clock rates:
* - ast2400 wdt can run at PCLK, or 1MHz
@@ -331,7 +403,7 @@ static int aspeed_wdt_probe(struct platform_device *pdev)
(of_device_is_compatible(np, "aspeed,ast2600-wdt"))) {
u32 reg = readl(wdt->base + WDT_RESET_WIDTH);
- reg &= config->ext_pulse_width_mask;
+ reg &= wdt->cfg->ext_pulse_width_mask;
if (of_property_read_bool(np, "aspeed,ext-active-high"))
reg |= WDT_ACTIVE_HIGH_MAGIC;
else
@@ -339,7 +411,7 @@ static int aspeed_wdt_probe(struct platform_device *pdev)
writel(reg, wdt->base + WDT_RESET_WIDTH);
- reg &= config->ext_pulse_width_mask;
+ reg &= wdt->cfg->ext_pulse_width_mask;
if (of_property_read_bool(np, "aspeed,ext-push-pull"))
reg |= WDT_PUSH_PULL_MAGIC;
else
@@ -349,7 +421,7 @@ static int aspeed_wdt_probe(struct platform_device *pdev)
}
if (!of_property_read_u32(np, "aspeed,ext-pulse-duration", &duration)) {
- u32 max_duration = config->ext_pulse_width_mask + 1;
+ u32 max_duration = wdt->cfg->ext_pulse_width_mask + 1;
if (duration == 0 || duration > max_duration) {
dev_err(dev, "Invalid pulse duration: %uus\n",
diff --git a/drivers/watchdog/at91rm9200_wdt.c b/drivers/watchdog/at91rm9200_wdt.c
index 6d751eb..5126454 100644
--- a/drivers/watchdog/at91rm9200_wdt.c
+++ b/drivers/watchdog/at91rm9200_wdt.c
@@ -278,8 +278,6 @@ static void at91wdt_shutdown(struct platform_device *pdev)
at91_wdt_stop();
}
-#ifdef CONFIG_PM
-
static int at91wdt_suspend(struct platform_device *pdev, pm_message_t message)
{
at91_wdt_stop();
@@ -293,11 +291,6 @@ static int at91wdt_resume(struct platform_device *pdev)
return 0;
}
-#else
-#define at91wdt_suspend NULL
-#define at91wdt_resume NULL
-#endif
-
static const struct of_device_id at91_wdt_dt_ids[] = {
{ .compatible = "atmel,at91rm9200-wdt" },
{ /* sentinel */ }
@@ -308,8 +301,8 @@ static struct platform_driver at91wdt_driver = {
.probe = at91wdt_probe,
.remove = at91wdt_remove,
.shutdown = at91wdt_shutdown,
- .suspend = at91wdt_suspend,
- .resume = at91wdt_resume,
+ .suspend = pm_ptr(at91wdt_suspend),
+ .resume = pm_ptr(at91wdt_resume),
.driver = {
.name = "atmel_st_watchdog",
.of_match_table = at91_wdt_dt_ids,
diff --git a/drivers/watchdog/db8500_wdt.c b/drivers/watchdog/db8500_wdt.c
index 6ed8b63..97148ac 100644
--- a/drivers/watchdog/db8500_wdt.c
+++ b/drivers/watchdog/db8500_wdt.c
@@ -105,7 +105,6 @@ static int db8500_wdt_probe(struct platform_device *pdev)
return 0;
}
-#ifdef CONFIG_PM
static int db8500_wdt_suspend(struct platform_device *pdev,
pm_message_t state)
{
@@ -130,15 +129,11 @@ static int db8500_wdt_resume(struct platform_device *pdev)
}
return 0;
}
-#else
-#define db8500_wdt_suspend NULL
-#define db8500_wdt_resume NULL
-#endif
static struct platform_driver db8500_wdt_driver = {
.probe = db8500_wdt_probe,
- .suspend = db8500_wdt_suspend,
- .resume = db8500_wdt_resume,
+ .suspend = pm_ptr(db8500_wdt_suspend),
+ .resume = pm_ptr(db8500_wdt_resume),
.driver = {
.name = "db8500_wdt",
},
diff --git a/drivers/watchdog/iTCO_wdt.c b/drivers/watchdog/iTCO_wdt.c
index 34693f1..e937b4d 100644
--- a/drivers/watchdog/iTCO_wdt.c
+++ b/drivers/watchdog/iTCO_wdt.c
@@ -423,14 +423,18 @@ static unsigned int iTCO_wdt_get_timeleft(struct watchdog_device *wd_dev)
return time_left;
}
-static void iTCO_wdt_set_running(struct iTCO_wdt_private *p)
+/* Returns true if the watchdog was running */
+static bool iTCO_wdt_set_running(struct iTCO_wdt_private *p)
{
u16 val;
- /* Bit 11: TCO Timer Halt -> 0 = The TCO timer is * enabled */
+ /* Bit 11: TCO Timer Halt -> 0 = The TCO timer is enabled */
val = inw(TCO1_CNT(p));
- if (!(val & BIT(11)))
+ if (!(val & BIT(11))) {
set_bit(WDOG_HW_RUNNING, &p->wddev.status);
+ return true;
+ }
+ return false;
}
/*
@@ -518,9 +522,6 @@ static int iTCO_wdt_probe(struct platform_device *pdev)
return -ENODEV; /* Cannot reset NO_REBOOT bit */
}
- /* Set the NO_REBOOT bit to prevent later reboots, just for sure */
- p->update_no_reboot_bit(p->no_reboot_priv, true);
-
if (turn_SMI_watchdog_clear_off >= p->iTCO_version) {
/*
* Bit 13: TCO_EN -> 0
@@ -572,7 +573,13 @@ static int iTCO_wdt_probe(struct platform_device *pdev)
watchdog_set_drvdata(&p->wddev, p);
platform_set_drvdata(pdev, p);
- iTCO_wdt_set_running(p);
+ if (!iTCO_wdt_set_running(p)) {
+ /*
+ * If the watchdog was not running set NO_REBOOT now to
+ * prevent later reboots.
+ */
+ p->update_no_reboot_bit(p->no_reboot_priv, true);
+ }
/* Check that the heartbeat value is within it's range;
if not reset to the default */
diff --git a/drivers/watchdog/kempld_wdt.c b/drivers/watchdog/kempld_wdt.c
index 40bd518..e6c7a29 100644
--- a/drivers/watchdog/kempld_wdt.c
+++ b/drivers/watchdog/kempld_wdt.c
@@ -75,9 +75,7 @@ struct kempld_wdt_data {
struct watchdog_device wdd;
unsigned int pretimeout;
struct kempld_wdt_stage stage[KEMPLD_WDT_MAX_STAGES];
-#ifdef CONFIG_PM
u8 pm_status_store;
-#endif
};
#define DEFAULT_TIMEOUT 30 /* seconds */
@@ -495,7 +493,6 @@ static int kempld_wdt_probe(struct platform_device *pdev)
return 0;
}
-#ifdef CONFIG_PM
/* Disable watchdog if it is active during suspend */
static int kempld_wdt_suspend(struct platform_device *pdev,
pm_message_t message)
@@ -531,18 +528,14 @@ static int kempld_wdt_resume(struct platform_device *pdev)
else
return kempld_wdt_stop(wdd);
}
-#else
-#define kempld_wdt_suspend NULL
-#define kempld_wdt_resume NULL
-#endif
static struct platform_driver kempld_wdt_driver = {
.driver = {
.name = "kempld-wdt",
},
.probe = kempld_wdt_probe,
- .suspend = kempld_wdt_suspend,
- .resume = kempld_wdt_resume,
+ .suspend = pm_ptr(kempld_wdt_suspend),
+ .resume = pm_ptr(kempld_wdt_resume),
};
module_platform_driver(kempld_wdt_driver);
diff --git a/drivers/watchdog/mtk_wdt.c b/drivers/watchdog/mtk_wdt.c
index e977875..3e62125 100644
--- a/drivers/watchdog/mtk_wdt.c
+++ b/drivers/watchdog/mtk_wdt.c
@@ -10,9 +10,11 @@
*/
#include <dt-bindings/reset/mt2712-resets.h>
+#include <dt-bindings/reset/mediatek,mt6795-resets.h>
#include <dt-bindings/reset/mt7986-resets.h>
#include <dt-bindings/reset/mt8183-resets.h>
#include <dt-bindings/reset/mt8186-resets.h>
+#include <dt-bindings/reset/mt8188-resets.h>
#include <dt-bindings/reset/mt8192-resets.h>
#include <dt-bindings/reset/mt8195-resets.h>
#include <linux/delay.h>
@@ -78,6 +80,10 @@ static const struct mtk_wdt_data mt2712_data = {
.toprgu_sw_rst_num = MT2712_TOPRGU_SW_RST_NUM,
};
+static const struct mtk_wdt_data mt6795_data = {
+ .toprgu_sw_rst_num = MT6795_TOPRGU_SW_RST_NUM,
+};
+
static const struct mtk_wdt_data mt7986_data = {
.toprgu_sw_rst_num = MT7986_TOPRGU_SW_RST_NUM,
};
@@ -90,6 +96,10 @@ static const struct mtk_wdt_data mt8186_data = {
.toprgu_sw_rst_num = MT8186_TOPRGU_SW_RST_NUM,
};
+static const struct mtk_wdt_data mt8188_data = {
+ .toprgu_sw_rst_num = MT8188_TOPRGU_SW_RST_NUM,
+};
+
static const struct mtk_wdt_data mt8192_data = {
.toprgu_sw_rst_num = MT8192_TOPRGU_SW_RST_NUM,
};
@@ -426,9 +436,11 @@ static int mtk_wdt_resume(struct device *dev)
static const struct of_device_id mtk_wdt_dt_ids[] = {
{ .compatible = "mediatek,mt2712-wdt", .data = &mt2712_data },
{ .compatible = "mediatek,mt6589-wdt" },
+ { .compatible = "mediatek,mt6795-wdt", .data = &mt6795_data },
{ .compatible = "mediatek,mt7986-wdt", .data = &mt7986_data },
{ .compatible = "mediatek,mt8183-wdt", .data = &mt8183_data },
{ .compatible = "mediatek,mt8186-wdt", .data = &mt8186_data },
+ { .compatible = "mediatek,mt8188-wdt", .data = &mt8188_data },
{ .compatible = "mediatek,mt8192-wdt", .data = &mt8192_data },
{ .compatible = "mediatek,mt8195-wdt", .data = &mt8195_data },
{ /* sentinel */ }
diff --git a/drivers/watchdog/omap_wdt.c b/drivers/watchdog/omap_wdt.c
index 74d785b..e75aa86 100644
--- a/drivers/watchdog/omap_wdt.c
+++ b/drivers/watchdog/omap_wdt.c
@@ -316,8 +316,6 @@ static int omap_wdt_remove(struct platform_device *pdev)
return 0;
}
-#ifdef CONFIG_PM
-
/* REVISIT ... not clear this is the best way to handle system suspend; and
* it's very inappropriate for selective device suspend (e.g. suspending this
* through sysfs rather than by stopping the watchdog daemon). Also, this
@@ -353,11 +351,6 @@ static int omap_wdt_resume(struct platform_device *pdev)
return 0;
}
-#else
-#define omap_wdt_suspend NULL
-#define omap_wdt_resume NULL
-#endif
-
static const struct of_device_id omap_wdt_of_match[] = {
{ .compatible = "ti,omap3-wdt", },
{},
@@ -368,8 +361,8 @@ static struct platform_driver omap_wdt_driver = {
.probe = omap_wdt_probe,
.remove = omap_wdt_remove,
.shutdown = omap_wdt_shutdown,
- .suspend = omap_wdt_suspend,
- .resume = omap_wdt_resume,
+ .suspend = pm_ptr(omap_wdt_suspend),
+ .resume = pm_ptr(omap_wdt_resume),
.driver = {
.name = "omap_wdt",
.of_match_table = omap_wdt_of_match,
diff --git a/drivers/watchdog/rn5t618_wdt.c b/drivers/watchdog/rn5t618_wdt.c
index 6e524c8..40d8ebd 100644
--- a/drivers/watchdog/rn5t618_wdt.c
+++ b/drivers/watchdog/rn5t618_wdt.c
@@ -144,6 +144,8 @@ static int rn5t618_wdt_probe(struct platform_device *pdev)
struct rn5t618 *rn5t618 = dev_get_drvdata(dev->parent);
struct rn5t618_wdt *wdt;
int min_timeout, max_timeout;
+ int ret;
+ unsigned int val;
wdt = devm_kzalloc(dev, sizeof(struct rn5t618_wdt), GFP_KERNEL);
if (!wdt)
@@ -160,6 +162,16 @@ static int rn5t618_wdt_probe(struct platform_device *pdev)
wdt->wdt_dev.timeout = max_timeout;
wdt->wdt_dev.parent = dev;
+ /* Read out previous power-off factor */
+ ret = regmap_read(wdt->rn5t618->regmap, RN5T618_POFFHIS, &val);
+ if (ret)
+ return ret;
+
+ if (val & RN5T618_POFFHIS_VINDET)
+ wdt->wdt_dev.bootstatus = WDIOF_POWERUNDER;
+ else if (val & RN5T618_POFFHIS_WDG)
+ wdt->wdt_dev.bootstatus = WDIOF_CARDRESET;
+
watchdog_set_drvdata(&wdt->wdt_dev, wdt);
watchdog_init_timeout(&wdt->wdt_dev, timeout, dev);
watchdog_set_nowayout(&wdt->wdt_dev, nowayout);
diff --git a/drivers/watchdog/twl4030_wdt.c b/drivers/watchdog/twl4030_wdt.c
index 36b4a66..09d17e2 100644
--- a/drivers/watchdog/twl4030_wdt.c
+++ b/drivers/watchdog/twl4030_wdt.c
@@ -81,7 +81,6 @@ static int twl4030_wdt_probe(struct platform_device *pdev)
return devm_watchdog_register_device(dev, wdt);
}
-#ifdef CONFIG_PM
static int twl4030_wdt_suspend(struct platform_device *pdev, pm_message_t state)
{
struct watchdog_device *wdt = platform_get_drvdata(pdev);
@@ -99,10 +98,6 @@ static int twl4030_wdt_resume(struct platform_device *pdev)
return 0;
}
-#else
-#define twl4030_wdt_suspend NULL
-#define twl4030_wdt_resume NULL
-#endif
static const struct of_device_id twl_wdt_of_match[] = {
{ .compatible = "ti,twl4030-wdt", },
@@ -112,8 +107,8 @@ MODULE_DEVICE_TABLE(of, twl_wdt_of_match);
static struct platform_driver twl4030_wdt_driver = {
.probe = twl4030_wdt_probe,
- .suspend = twl4030_wdt_suspend,
- .resume = twl4030_wdt_resume,
+ .suspend = pm_ptr(twl4030_wdt_suspend),
+ .resume = pm_ptr(twl4030_wdt_resume),
.driver = {
.name = "twl4030_wdt",
.of_match_table = twl_wdt_of_match,
diff --git a/fs/gfs2/aops.c b/fs/gfs2/aops.c
index 05bee80..e782b4f 100644
--- a/fs/gfs2/aops.c
+++ b/fs/gfs2/aops.c
@@ -427,8 +427,6 @@ static int stuffed_readpage(struct gfs2_inode *ip, struct page *page)
return error;
kaddr = kmap_atomic(page);
- if (dsize > gfs2_max_stuffed_size(ip))
- dsize = gfs2_max_stuffed_size(ip);
memcpy(kaddr, dibh->b_data + sizeof(struct gfs2_dinode), dsize);
memset(kaddr + dsize, 0, PAGE_SIZE - dsize);
kunmap_atomic(kaddr);
diff --git a/fs/gfs2/bmap.c b/fs/gfs2/bmap.c
index 3bdb2c6..e7537fd 100644
--- a/fs/gfs2/bmap.c
+++ b/fs/gfs2/bmap.c
@@ -61,9 +61,6 @@ static int gfs2_unstuffer_page(struct gfs2_inode *ip, struct buffer_head *dibh,
void *kaddr = kmap(page);
u64 dsize = i_size_read(inode);
- if (dsize > gfs2_max_stuffed_size(ip))
- dsize = gfs2_max_stuffed_size(ip);
-
memcpy(kaddr, dibh->b_data + sizeof(struct gfs2_dinode), dsize);
memset(kaddr + dsize, 0, PAGE_SIZE - dsize);
kunmap(page);
diff --git a/fs/gfs2/file.c b/fs/gfs2/file.c
index 60c6fb9..eea5be4 100644
--- a/fs/gfs2/file.c
+++ b/fs/gfs2/file.c
@@ -1445,14 +1445,13 @@ static int gfs2_lock(struct file *file, int cmd, struct file_lock *fl)
static void __flock_holder_uninit(struct file *file, struct gfs2_holder *fl_gh)
{
- struct gfs2_glock *gl = fl_gh->gh_gl;
+ struct gfs2_glock *gl = gfs2_glock_hold(fl_gh->gh_gl);
/*
* Make sure gfs2_glock_put() won't sleep under the file->f_lock
* spinlock.
*/
- gfs2_glock_hold(gl);
spin_lock(&file->f_lock);
gfs2_holder_uninit(fl_gh);
spin_unlock(&file->f_lock);
diff --git a/fs/gfs2/glock.c b/fs/gfs2/glock.c
index df335c2..524f3c9 100644
--- a/fs/gfs2/glock.c
+++ b/fs/gfs2/glock.c
@@ -186,10 +186,11 @@ void gfs2_glock_free(struct gfs2_glock *gl)
*
*/
-void gfs2_glock_hold(struct gfs2_glock *gl)
+struct gfs2_glock *gfs2_glock_hold(struct gfs2_glock *gl)
{
GLOCK_BUG_ON(gl, __lockref_is_dead(&gl->gl_lockref));
lockref_get(&gl->gl_lockref);
+ return gl;
}
/**
@@ -205,12 +206,6 @@ static int demote_ok(const struct gfs2_glock *gl)
if (gl->gl_state == LM_ST_UNLOCKED)
return 0;
- /*
- * Note that demote_ok is used for the lru process of disposing of
- * glocks. For this purpose, we don't care if the glock's holders
- * have the HIF_MAY_DEMOTE flag set or not. If someone is using
- * them, don't demote.
- */
if (!list_empty(&gl->gl_holders))
return 0;
if (glops->go_demote_ok)
@@ -393,7 +388,7 @@ static void do_error(struct gfs2_glock *gl, const int ret)
struct gfs2_holder *gh, *tmp;
list_for_each_entry_safe(gh, tmp, &gl->gl_holders, gh_list) {
- if (!test_bit(HIF_WAIT, &gh->gh_iflags))
+ if (test_bit(HIF_HOLDER, &gh->gh_iflags))
continue;
if (ret & LM_OUT_ERROR)
gh->gh_error = -EIO;
@@ -408,45 +403,6 @@ static void do_error(struct gfs2_glock *gl, const int ret)
}
/**
- * demote_incompat_holders - demote incompatible demoteable holders
- * @gl: the glock we want to promote
- * @current_gh: the newly promoted holder
- *
- * We're passing the newly promoted holder in @current_gh, but actually, any of
- * the strong holders would do.
- */
-static void demote_incompat_holders(struct gfs2_glock *gl,
- struct gfs2_holder *current_gh)
-{
- struct gfs2_holder *gh, *tmp;
-
- /*
- * Demote incompatible holders before we make ourselves eligible.
- * (This holder may or may not allow auto-demoting, but we don't want
- * to demote the new holder before it's even granted.)
- */
- list_for_each_entry_safe(gh, tmp, &gl->gl_holders, gh_list) {
- /*
- * Since holders are at the front of the list, we stop when we
- * find the first non-holder.
- */
- if (!test_bit(HIF_HOLDER, &gh->gh_iflags))
- return;
- if (gh == current_gh)
- continue;
- if (test_bit(HIF_MAY_DEMOTE, &gh->gh_iflags) &&
- !may_grant(gl, current_gh, gh)) {
- /*
- * We should not recurse into do_promote because
- * __gfs2_glock_dq only calls handle_callback,
- * gfs2_glock_add_to_lru and __gfs2_glock_queue_work.
- */
- __gfs2_glock_dq(gh);
- }
- }
-}
-
-/**
* find_first_holder - find the first "holder" gh
* @gl: the glock
*/
@@ -464,26 +420,6 @@ static inline struct gfs2_holder *find_first_holder(const struct gfs2_glock *gl)
return NULL;
}
-/**
- * find_first_strong_holder - find the first non-demoteable holder
- * @gl: the glock
- *
- * Find the first holder that doesn't have the HIF_MAY_DEMOTE flag set.
- */
-static inline struct gfs2_holder *
-find_first_strong_holder(struct gfs2_glock *gl)
-{
- struct gfs2_holder *gh;
-
- list_for_each_entry(gh, &gl->gl_holders, gh_list) {
- if (!test_bit(HIF_HOLDER, &gh->gh_iflags))
- return NULL;
- if (!test_bit(HIF_MAY_DEMOTE, &gh->gh_iflags))
- return gh;
- }
- return NULL;
-}
-
/*
* gfs2_instantiate - Call the glops instantiate function
* @gh: The glock holder
@@ -540,9 +476,8 @@ int gfs2_instantiate(struct gfs2_holder *gh)
static int do_promote(struct gfs2_glock *gl)
{
struct gfs2_holder *gh, *current_gh;
- bool incompat_holders_demoted = false;
- current_gh = find_first_strong_holder(gl);
+ current_gh = find_first_holder(gl);
list_for_each_entry(gh, &gl->gl_holders, gh_list) {
if (test_bit(HIF_HOLDER, &gh->gh_iflags))
continue;
@@ -561,11 +496,8 @@ static int do_promote(struct gfs2_glock *gl)
set_bit(HIF_HOLDER, &gh->gh_iflags);
trace_gfs2_promote(gh);
gfs2_holder_wake(gh);
- if (!incompat_holders_demoted) {
+ if (!current_gh)
current_gh = gh;
- demote_incompat_holders(gl, current_gh);
- incompat_holders_demoted = true;
- }
}
return 0;
}
@@ -927,6 +859,48 @@ __acquires(&gl->gl_lockref.lock)
return;
}
+/**
+ * glock_set_object - set the gl_object field of a glock
+ * @gl: the glock
+ * @object: the object
+ */
+void glock_set_object(struct gfs2_glock *gl, void *object)
+{
+ void *prev_object;
+
+ spin_lock(&gl->gl_lockref.lock);
+ prev_object = gl->gl_object;
+ gl->gl_object = object;
+ spin_unlock(&gl->gl_lockref.lock);
+ if (gfs2_assert_warn(gl->gl_name.ln_sbd, prev_object == NULL)) {
+ pr_warn("glock=%u/%llx\n",
+ gl->gl_name.ln_type,
+ (unsigned long long)gl->gl_name.ln_number);
+ gfs2_dump_glock(NULL, gl, true);
+ }
+}
+
+/**
+ * glock_clear_object - clear the gl_object field of a glock
+ * @gl: the glock
+ */
+void glock_clear_object(struct gfs2_glock *gl, void *object)
+{
+ void *prev_object;
+
+ spin_lock(&gl->gl_lockref.lock);
+ prev_object = gl->gl_object;
+ gl->gl_object = NULL;
+ spin_unlock(&gl->gl_lockref.lock);
+ if (gfs2_assert_warn(gl->gl_name.ln_sbd,
+ prev_object == object || prev_object == NULL)) {
+ pr_warn("glock=%u/%llx\n",
+ gl->gl_name.ln_type,
+ (unsigned long long)gl->gl_name.ln_number);
+ gfs2_dump_glock(NULL, gl, true);
+ }
+}
+
void gfs2_inode_remember_delete(struct gfs2_glock *gl, u64 generation)
{
struct gfs2_inode_lvb *ri = (void *)gl->gl_lksb.sb_lvbptr;
@@ -980,8 +954,6 @@ static bool gfs2_try_evict(struct gfs2_glock *gl)
ip = NULL;
spin_unlock(&gl->gl_lockref.lock);
if (ip) {
- struct gfs2_glock *inode_gl = NULL;
-
gl->gl_no_formal_ino = ip->i_no_formal_ino;
set_bit(GIF_DEFERRED_DELETE, &ip->i_flags);
d_prune_aliases(&ip->i_inode);
@@ -991,14 +963,14 @@ static bool gfs2_try_evict(struct gfs2_glock *gl)
spin_lock(&gl->gl_lockref.lock);
ip = gl->gl_object;
if (ip) {
- inode_gl = ip->i_gl;
- lockref_get(&inode_gl->gl_lockref);
clear_bit(GIF_DEFERRED_DELETE, &ip->i_flags);
+ if (!igrab(&ip->i_inode))
+ ip = NULL;
}
spin_unlock(&gl->gl_lockref.lock);
- if (inode_gl) {
- gfs2_glock_poke(inode_gl);
- gfs2_glock_put(inode_gl);
+ if (ip) {
+ gfs2_glock_poke(ip->i_gl);
+ iput(&ip->i_inode);
}
evicted = !ip;
}
@@ -1039,6 +1011,7 @@ static void delete_work_func(struct work_struct *work)
if (gfs2_queue_delete_work(gl, 5 * HZ))
return;
}
+ goto out;
}
inode = gfs2_lookup_by_inum(sdp, no_addr, gl->gl_no_formal_ino,
@@ -1051,6 +1024,7 @@ static void delete_work_func(struct work_struct *work)
d_prune_aliases(inode);
iput(inode);
}
+out:
gfs2_glock_put(gl);
}
@@ -1256,13 +1230,12 @@ void __gfs2_holder_init(struct gfs2_glock *gl, unsigned int state, u16 flags,
struct gfs2_holder *gh, unsigned long ip)
{
INIT_LIST_HEAD(&gh->gh_list);
- gh->gh_gl = gl;
+ gh->gh_gl = gfs2_glock_hold(gl);
gh->gh_ip = ip;
gh->gh_owner_pid = get_pid(task_pid(current));
gh->gh_state = state;
gh->gh_flags = flags;
gh->gh_iflags = 0;
- gfs2_glock_hold(gl);
}
/**
@@ -1496,7 +1469,7 @@ __acquires(&gl->gl_lockref.lock)
if (test_bit(GLF_LOCK, &gl->gl_flags)) {
struct gfs2_holder *current_gh;
- current_gh = find_first_strong_holder(gl);
+ current_gh = find_first_holder(gl);
try_futile = !may_grant(gl, current_gh, gh);
}
if (test_bit(GLF_INVALIDATE_IN_PROGRESS, &gl->gl_flags))
@@ -1508,8 +1481,6 @@ __acquires(&gl->gl_lockref.lock)
continue;
if (gh->gh_gl->gl_ops->go_type == LM_TYPE_FLOCK)
continue;
- if (test_bit(HIF_MAY_DEMOTE, &gh2->gh_iflags))
- continue;
if (!pid_is_meaningful(gh2))
continue;
goto trap_recursive;
@@ -1619,69 +1590,28 @@ static inline bool needs_demote(struct gfs2_glock *gl)
static void __gfs2_glock_dq(struct gfs2_holder *gh)
{
struct gfs2_glock *gl = gh->gh_gl;
- struct gfs2_sbd *sdp = gl->gl_name.ln_sbd;
unsigned delay = 0;
int fast_path = 0;
/*
- * This while loop is similar to function demote_incompat_holders:
- * If the glock is due to be demoted (which may be from another node
- * or even if this holder is GL_NOCACHE), the weak holders are
- * demoted as well, allowing the glock to be demoted.
+ * This holder should not be cached, so mark it for demote.
+ * Note: this should be done before the check for needs_demote
+ * below.
*/
- while (gh) {
- /*
- * If we're in the process of file system withdraw, we cannot
- * just dequeue any glocks until our journal is recovered, lest
- * we introduce file system corruption. We need two exceptions
- * to this rule: We need to allow unlocking of nondisk glocks
- * and the glock for our own journal that needs recovery.
- */
- if (test_bit(SDF_WITHDRAW_RECOVERY, &sdp->sd_flags) &&
- glock_blocked_by_withdraw(gl) &&
- gh->gh_gl != sdp->sd_jinode_gl) {
- sdp->sd_glock_dqs_held++;
- spin_unlock(&gl->gl_lockref.lock);
- might_sleep();
- wait_on_bit(&sdp->sd_flags, SDF_WITHDRAW_RECOVERY,
- TASK_UNINTERRUPTIBLE);
- spin_lock(&gl->gl_lockref.lock);
- }
+ if (gh->gh_flags & GL_NOCACHE)
+ handle_callback(gl, LM_ST_UNLOCKED, 0, false);
- /*
- * This holder should not be cached, so mark it for demote.
- * Note: this should be done before the check for needs_demote
- * below.
- */
- if (gh->gh_flags & GL_NOCACHE)
- handle_callback(gl, LM_ST_UNLOCKED, 0, false);
+ list_del_init(&gh->gh_list);
+ clear_bit(HIF_HOLDER, &gh->gh_iflags);
+ trace_gfs2_glock_queue(gh, 0);
- list_del_init(&gh->gh_list);
- clear_bit(HIF_HOLDER, &gh->gh_iflags);
- trace_gfs2_glock_queue(gh, 0);
-
- /*
- * If there hasn't been a demote request we are done.
- * (Let the remaining holders, if any, keep holding it.)
- */
- if (!needs_demote(gl)) {
- if (list_empty(&gl->gl_holders))
- fast_path = 1;
- break;
- }
- /*
- * If we have another strong holder (we cannot auto-demote)
- * we are done. It keeps holding it until it is done.
- */
- if (find_first_strong_holder(gl))
- break;
-
- /*
- * If we have a weak holder at the head of the list, it
- * (and all others like it) must be auto-demoted. If there
- * are no more weak holders, we exit the while loop.
- */
- gh = find_first_holder(gl);
+ /*
+ * If there hasn't been a demote request we are done.
+ * (Let the remaining holders, if any, keep holding it.)
+ */
+ if (!needs_demote(gl)) {
+ if (list_empty(&gl->gl_holders))
+ fast_path = 1;
}
if (!test_bit(GLF_LFLUSH, &gl->gl_flags) && demote_ok(gl))
@@ -1705,8 +1635,17 @@ static void __gfs2_glock_dq(struct gfs2_holder *gh)
void gfs2_glock_dq(struct gfs2_holder *gh)
{
struct gfs2_glock *gl = gh->gh_gl;
+ struct gfs2_sbd *sdp = gl->gl_name.ln_sbd;
spin_lock(&gl->gl_lockref.lock);
+ if (!gfs2_holder_queued(gh)) {
+ /*
+ * May have already been dequeued because the locking request
+ * was GL_ASYNC and it has failed in the meantime.
+ */
+ goto out;
+ }
+
if (list_is_first(&gh->gh_list, &gl->gl_holders) &&
!test_bit(HIF_HOLDER, &gh->gh_iflags)) {
spin_unlock(&gl->gl_lockref.lock);
@@ -1715,7 +1654,26 @@ void gfs2_glock_dq(struct gfs2_holder *gh)
spin_lock(&gl->gl_lockref.lock);
}
+ /*
+ * If we're in the process of file system withdraw, we cannot just
+ * dequeue any glocks until our journal is recovered, lest we introduce
+ * file system corruption. We need two exceptions to this rule: We need
+ * to allow unlocking of nondisk glocks and the glock for our own
+ * journal that needs recovery.
+ */
+ if (test_bit(SDF_WITHDRAW_RECOVERY, &sdp->sd_flags) &&
+ glock_blocked_by_withdraw(gl) &&
+ gh->gh_gl != sdp->sd_jinode_gl) {
+ sdp->sd_glock_dqs_held++;
+ spin_unlock(&gl->gl_lockref.lock);
+ might_sleep();
+ wait_on_bit(&sdp->sd_flags, SDF_WITHDRAW_RECOVERY,
+ TASK_UNINTERRUPTIBLE);
+ spin_lock(&gl->gl_lockref.lock);
+ }
+
__gfs2_glock_dq(gh);
+out:
spin_unlock(&gl->gl_lockref.lock);
}
@@ -1888,33 +1846,6 @@ void gfs2_glock_cb(struct gfs2_glock *gl, unsigned int state)
if (test_bit(GLF_REPLY_PENDING, &gl->gl_flags))
delay = gl->gl_hold_time;
}
- /*
- * Note 1: We cannot call demote_incompat_holders from handle_callback
- * or gfs2_set_demote due to recursion problems like: gfs2_glock_dq ->
- * handle_callback -> demote_incompat_holders -> gfs2_glock_dq
- * Plus, we only want to demote the holders if the request comes from
- * a remote cluster node because local holder conflicts are resolved
- * elsewhere.
- *
- * Note 2: if a remote node wants this glock in EX mode, lock_dlm will
- * request that we set our state to UNLOCKED. Here we mock up a holder
- * to make it look like someone wants the lock EX locally. Any SH
- * and DF requests should be able to share the lock without demoting.
- *
- * Note 3: We only want to demote the demoteable holders when there
- * are no more strong holders. The demoteable holders might as well
- * keep the glock until the last strong holder is done with it.
- */
- if (!find_first_strong_holder(gl)) {
- struct gfs2_holder mock_gh = {
- .gh_gl = gl,
- .gh_state = (state == LM_ST_UNLOCKED) ?
- LM_ST_EXCLUSIVE : state,
- .gh_iflags = BIT(HIF_HOLDER)
- };
-
- demote_incompat_holders(gl, &mock_gh);
- }
handle_callback(gl, state, delay, true);
__gfs2_glock_queue_work(gl, delay);
spin_unlock(&gl->gl_lockref.lock);
@@ -2306,8 +2237,6 @@ static const char *hflags2str(char *buf, u16 flags, unsigned long iflags)
*p++ = 'H';
if (test_bit(HIF_WAIT, &iflags))
*p++ = 'W';
- if (test_bit(HIF_MAY_DEMOTE, &iflags))
- *p++ = 'D';
if (flags & GL_SKIP)
*p++ = 's';
*p = 0;
diff --git a/fs/gfs2/glock.h b/fs/gfs2/glock.h
index 0d068f4..f37ac08 100644
--- a/fs/gfs2/glock.h
+++ b/fs/gfs2/glock.h
@@ -156,8 +156,6 @@ static inline struct gfs2_holder *gfs2_glock_is_locked_by_me(struct gfs2_glock *
list_for_each_entry(gh, &gl->gl_holders, gh_list) {
if (!test_bit(HIF_HOLDER, &gh->gh_iflags))
break;
- if (test_bit(HIF_MAY_DEMOTE, &gh->gh_iflags))
- continue;
if (gh->gh_owner_pid == pid)
goto out;
}
@@ -196,7 +194,7 @@ static inline struct address_space *gfs2_glock2aspace(struct gfs2_glock *gl)
extern int gfs2_glock_get(struct gfs2_sbd *sdp, u64 number,
const struct gfs2_glock_operations *glops,
int create, struct gfs2_glock **glp);
-extern void gfs2_glock_hold(struct gfs2_glock *gl);
+extern struct gfs2_glock *gfs2_glock_hold(struct gfs2_glock *gl);
extern void gfs2_glock_put(struct gfs2_glock *gl);
extern void gfs2_glock_queue_put(struct gfs2_glock *gl);
@@ -288,6 +286,9 @@ extern void gfs2_delete_debugfs_file(struct gfs2_sbd *sdp);
extern void gfs2_register_debugfs(void);
extern void gfs2_unregister_debugfs(void);
+extern void glock_set_object(struct gfs2_glock *gl, void *object);
+extern void glock_clear_object(struct gfs2_glock *gl, void *object);
+
extern const struct lm_lockops gfs2_dlm_ops;
static inline void gfs2_holder_mark_uninitialized(struct gfs2_holder *gh)
@@ -305,64 +306,6 @@ static inline bool gfs2_holder_queued(struct gfs2_holder *gh)
return !list_empty(&gh->gh_list);
}
-/**
- * glock_set_object - set the gl_object field of a glock
- * @gl: the glock
- * @object: the object
- */
-static inline void glock_set_object(struct gfs2_glock *gl, void *object)
-{
- spin_lock(&gl->gl_lockref.lock);
- if (gfs2_assert_warn(gl->gl_name.ln_sbd, gl->gl_object == NULL))
- gfs2_dump_glock(NULL, gl, true);
- gl->gl_object = object;
- spin_unlock(&gl->gl_lockref.lock);
-}
-
-/**
- * glock_clear_object - clear the gl_object field of a glock
- * @gl: the glock
- * @object: the object
- *
- * I'd love to similarly add this:
- * else if (gfs2_assert_warn(gl->gl_sbd, gl->gl_object == object))
- * gfs2_dump_glock(NULL, gl, true);
- * Unfortunately, that's not possible because as soon as gfs2_delete_inode
- * frees the block in the rgrp, another process can reassign it for an I_NEW
- * inode in gfs2_create_inode because that calls new_inode, not gfs2_iget.
- * That means gfs2_delete_inode may subsequently try to call this function
- * for a glock that's already pointing to a brand new inode. If we clear the
- * new inode's gl_object, we'll introduce metadata corruption. Function
- * gfs2_delete_inode calls clear_inode which calls gfs2_clear_inode which also
- * tries to clear gl_object, so it's more than just gfs2_delete_inode.
- *
- */
-static inline void glock_clear_object(struct gfs2_glock *gl, void *object)
-{
- spin_lock(&gl->gl_lockref.lock);
- if (gl->gl_object == object)
- gl->gl_object = NULL;
- spin_unlock(&gl->gl_lockref.lock);
-}
-
-static inline void gfs2_holder_allow_demote(struct gfs2_holder *gh)
-{
- struct gfs2_glock *gl = gh->gh_gl;
-
- spin_lock(&gl->gl_lockref.lock);
- set_bit(HIF_MAY_DEMOTE, &gh->gh_iflags);
- spin_unlock(&gl->gl_lockref.lock);
-}
-
-static inline void gfs2_holder_disallow_demote(struct gfs2_holder *gh)
-{
- struct gfs2_glock *gl = gh->gh_gl;
-
- spin_lock(&gl->gl_lockref.lock);
- clear_bit(HIF_MAY_DEMOTE, &gh->gh_iflags);
- spin_unlock(&gl->gl_lockref.lock);
-}
-
extern void gfs2_inode_remember_delete(struct gfs2_glock *gl, u64 generation);
extern bool gfs2_inode_already_deleted(struct gfs2_glock *gl, u64 generation);
diff --git a/fs/gfs2/glops.c b/fs/gfs2/glops.c
index 49210a2..d78b61e 100644
--- a/fs/gfs2/glops.c
+++ b/fs/gfs2/glops.c
@@ -397,38 +397,39 @@ static int gfs2_dinode_in(struct gfs2_inode *ip, const void *buf)
struct timespec64 atime;
u16 height, depth;
umode_t mode = be32_to_cpu(str->di_mode);
- bool is_new = ip->i_inode.i_state & I_NEW;
+ struct inode *inode = &ip->i_inode;
+ bool is_new = inode->i_state & I_NEW;
if (unlikely(ip->i_no_addr != be64_to_cpu(str->di_num.no_addr)))
goto corrupt;
- if (unlikely(!is_new && inode_wrong_type(&ip->i_inode, mode)))
+ if (unlikely(!is_new && inode_wrong_type(inode, mode)))
goto corrupt;
ip->i_no_formal_ino = be64_to_cpu(str->di_num.no_formal_ino);
- ip->i_inode.i_mode = mode;
+ inode->i_mode = mode;
if (is_new) {
- ip->i_inode.i_rdev = 0;
+ inode->i_rdev = 0;
switch (mode & S_IFMT) {
case S_IFBLK:
case S_IFCHR:
- ip->i_inode.i_rdev = MKDEV(be32_to_cpu(str->di_major),
- be32_to_cpu(str->di_minor));
+ inode->i_rdev = MKDEV(be32_to_cpu(str->di_major),
+ be32_to_cpu(str->di_minor));
break;
}
}
- i_uid_write(&ip->i_inode, be32_to_cpu(str->di_uid));
- i_gid_write(&ip->i_inode, be32_to_cpu(str->di_gid));
- set_nlink(&ip->i_inode, be32_to_cpu(str->di_nlink));
- i_size_write(&ip->i_inode, be64_to_cpu(str->di_size));
- gfs2_set_inode_blocks(&ip->i_inode, be64_to_cpu(str->di_blocks));
+ i_uid_write(inode, be32_to_cpu(str->di_uid));
+ i_gid_write(inode, be32_to_cpu(str->di_gid));
+ set_nlink(inode, be32_to_cpu(str->di_nlink));
+ i_size_write(inode, be64_to_cpu(str->di_size));
+ gfs2_set_inode_blocks(inode, be64_to_cpu(str->di_blocks));
atime.tv_sec = be64_to_cpu(str->di_atime);
atime.tv_nsec = be32_to_cpu(str->di_atime_nsec);
- if (timespec64_compare(&ip->i_inode.i_atime, &atime) < 0)
- ip->i_inode.i_atime = atime;
- ip->i_inode.i_mtime.tv_sec = be64_to_cpu(str->di_mtime);
- ip->i_inode.i_mtime.tv_nsec = be32_to_cpu(str->di_mtime_nsec);
- ip->i_inode.i_ctime.tv_sec = be64_to_cpu(str->di_ctime);
- ip->i_inode.i_ctime.tv_nsec = be32_to_cpu(str->di_ctime_nsec);
+ if (timespec64_compare(&inode->i_atime, &atime) < 0)
+ inode->i_atime = atime;
+ inode->i_mtime.tv_sec = be64_to_cpu(str->di_mtime);
+ inode->i_mtime.tv_nsec = be32_to_cpu(str->di_mtime_nsec);
+ inode->i_ctime.tv_sec = be64_to_cpu(str->di_ctime);
+ inode->i_ctime.tv_nsec = be32_to_cpu(str->di_ctime_nsec);
ip->i_goal = be64_to_cpu(str->di_goal_meta);
ip->i_generation = be64_to_cpu(str->di_generation);
@@ -436,7 +437,7 @@ static int gfs2_dinode_in(struct gfs2_inode *ip, const void *buf)
ip->i_diskflags = be32_to_cpu(str->di_flags);
ip->i_eattr = be64_to_cpu(str->di_eattr);
/* i_diskflags and i_eattr must be set before gfs2_set_inode_flags() */
- gfs2_set_inode_flags(&ip->i_inode);
+ gfs2_set_inode_flags(inode);
height = be16_to_cpu(str->di_height);
if (unlikely(height > GFS2_MAX_META_HEIGHT))
goto corrupt;
@@ -448,8 +449,11 @@ static int gfs2_dinode_in(struct gfs2_inode *ip, const void *buf)
ip->i_depth = (u8)depth;
ip->i_entries = be32_to_cpu(str->di_entries);
- if (S_ISREG(ip->i_inode.i_mode))
- gfs2_set_aops(&ip->i_inode);
+ if (gfs2_is_stuffed(ip) && inode->i_size > gfs2_max_stuffed_size(ip))
+ goto corrupt;
+
+ if (S_ISREG(inode->i_mode))
+ gfs2_set_aops(inode);
return 0;
corrupt:
diff --git a/fs/gfs2/incore.h b/fs/gfs2/incore.h
index d09d989..c267650 100644
--- a/fs/gfs2/incore.h
+++ b/fs/gfs2/incore.h
@@ -252,7 +252,6 @@ struct gfs2_lkstats {
enum {
/* States */
- HIF_MAY_DEMOTE = 1,
HIF_HOLDER = 6, /* Set for gh that "holds" the glock */
HIF_WAIT = 10,
};
diff --git a/fs/gfs2/inode.c b/fs/gfs2/inode.c
index 1371e06..614db30 100644
--- a/fs/gfs2/inode.c
+++ b/fs/gfs2/inode.c
@@ -142,6 +142,11 @@ struct inode *gfs2_inode_lookup(struct super_block *sb, unsigned int type,
if (unlikely(error))
goto fail;
+ /*
+ * The only caller that sets @blktype to GFS2_BLKST_UNLINKED is
+ * delete_work_func(). Make sure not to cancel the delete work
+ * from within itself here.
+ */
if (blktype == GFS2_BLKST_UNLINKED)
extra_flags |= LM_FLAG_TRY;
else
@@ -403,12 +408,17 @@ static int alloc_dinode(struct gfs2_inode *ip, u32 flags, unsigned *dblocks)
goto out_ipreserv;
error = gfs2_alloc_blocks(ip, &ip->i_no_addr, dblocks, 1, &ip->i_generation);
+ if (error)
+ goto out_trans_end;
+
ip->i_no_formal_ino = ip->i_generation;
ip->i_inode.i_ino = ip->i_no_addr;
ip->i_goal = ip->i_no_addr;
+ if (*dblocks > 1)
+ ip->i_eattr = ip->i_no_addr + 1;
+out_trans_end:
gfs2_trans_end(sdp);
-
out_ipreserv:
gfs2_inplace_release(ip);
out_quota:
@@ -586,6 +596,12 @@ static int gfs2_initxattrs(struct inode *inode, const struct xattr *xattr_array,
* @size: The initial size of the inode (ignored for directories)
* @excl: Force fail if inode exists
*
+ * FIXME: Change to allocate the disk blocks and write them out in the same
+ * transaction. That way, we can no longer end up in a situation in which an
+ * inode is allocated, the node crashes, and the block looks like a valid
+ * inode. (With atomic creates in place, we will also no longer need to zero
+ * the link count and dirty the inode here on failure.)
+ *
* Returns: 0 on success, or error code
*/
@@ -596,12 +612,12 @@ static int gfs2_create_inode(struct inode *dir, struct dentry *dentry,
{
const struct qstr *name = &dentry->d_name;
struct posix_acl *default_acl, *acl;
- struct gfs2_holder ghs[2];
+ struct gfs2_holder d_gh, gh;
struct inode *inode = NULL;
struct gfs2_inode *dip = GFS2_I(dir), *ip;
struct gfs2_sbd *sdp = GFS2_SB(&dip->i_inode);
struct gfs2_glock *io_gl;
- int error, free_vfs_inode = 1;
+ int error;
u32 aflags = 0;
unsigned blocks = 1;
struct gfs2_diradd da = { .bh = NULL, .save_loc = 1, };
@@ -617,10 +633,10 @@ static int gfs2_create_inode(struct inode *dir, struct dentry *dentry,
if (error)
goto fail;
- error = gfs2_glock_nq_init(dip->i_gl, LM_ST_EXCLUSIVE, 0, ghs);
+ error = gfs2_glock_nq_init(dip->i_gl, LM_ST_EXCLUSIVE, 0, &d_gh);
if (error)
goto fail;
- gfs2_holder_mark_uninitialized(ghs + 1);
+ gfs2_holder_mark_uninitialized(&gh);
error = create_ok(dip, name, mode);
if (error)
@@ -642,7 +658,7 @@ static int gfs2_create_inode(struct inode *dir, struct dentry *dentry,
else
error = finish_no_open(file, NULL);
}
- gfs2_glock_dq_uninit(ghs);
+ gfs2_glock_dq_uninit(&d_gh);
goto fail;
} else if (error != -ENOENT) {
goto fail_gunlock;
@@ -656,12 +672,12 @@ static int gfs2_create_inode(struct inode *dir, struct dentry *dentry,
error = -ENOMEM;
if (!inode)
goto fail_gunlock;
+ ip = GFS2_I(inode);
error = posix_acl_create(dir, &mode, &default_acl, &acl);
if (error)
goto fail_gunlock;
- ip = GFS2_I(inode);
error = gfs2_qa_get(ip);
if (error)
goto fail_free_acls;
@@ -723,15 +739,19 @@ static int gfs2_create_inode(struct inode *dir, struct dentry *dentry,
goto fail_free_inode;
gfs2_cancel_delete_work(io_gl);
+retry:
error = insert_inode_locked4(inode, ip->i_no_addr, iget_test, &ip->i_no_addr);
- BUG_ON(error);
+ if (error == -EBUSY)
+ goto retry;
+ if (error)
+ goto fail_gunlock2;
error = gfs2_glock_nq_init(io_gl, LM_ST_SHARED, GL_EXACT | GL_NOPID,
&ip->i_iopen_gh);
if (error)
goto fail_gunlock2;
- error = gfs2_glock_nq_init(ip->i_gl, LM_ST_EXCLUSIVE, GL_SKIP, ghs + 1);
+ error = gfs2_glock_nq_init(ip->i_gl, LM_ST_EXCLUSIVE, GL_SKIP, &gh);
if (error)
goto fail_gunlock3;
@@ -739,10 +759,8 @@ static int gfs2_create_inode(struct inode *dir, struct dentry *dentry,
if (error)
goto fail_gunlock3;
- if (blocks > 1) {
- ip->i_eattr = ip->i_no_addr + 1;
+ if (blocks > 1)
gfs2_init_xattr(ip);
- }
init_dinode(dip, ip, symname);
gfs2_trans_end(sdp);
@@ -750,9 +768,6 @@ static int gfs2_create_inode(struct inode *dir, struct dentry *dentry,
glock_set_object(io_gl, ip);
gfs2_set_iop(inode);
- free_vfs_inode = 0; /* After this point, the inode is no longer
- considered free. Any failures need to undo
- the gfs2 structures. */
if (default_acl) {
error = __gfs2_set_acl(inode, default_acl, ACL_TYPE_DEFAULT);
if (error)
@@ -785,9 +800,9 @@ static int gfs2_create_inode(struct inode *dir, struct dentry *dentry,
file->f_mode |= FMODE_CREATED;
error = finish_open(file, dentry, gfs2_open_common);
}
- gfs2_glock_dq_uninit(ghs);
+ gfs2_glock_dq_uninit(&d_gh);
gfs2_qa_put(ip);
- gfs2_glock_dq_uninit(ghs + 1);
+ gfs2_glock_dq_uninit(&gh);
gfs2_glock_put(io_gl);
gfs2_qa_put(dip);
unlock_new_inode(inode);
@@ -801,10 +816,6 @@ static int gfs2_create_inode(struct inode *dir, struct dentry *dentry,
fail_gunlock2:
gfs2_glock_put(io_gl);
fail_free_inode:
- if (ip->i_gl) {
- if (free_vfs_inode) /* else evict will do the put for us */
- gfs2_glock_put(ip->i_gl);
- }
gfs2_rs_deltree(&ip->i_res);
gfs2_qa_put(ip);
fail_free_acls:
@@ -812,20 +823,19 @@ static int gfs2_create_inode(struct inode *dir, struct dentry *dentry,
posix_acl_release(acl);
fail_gunlock:
gfs2_dir_no_add(&da);
- gfs2_glock_dq_uninit(ghs);
+ gfs2_glock_dq_uninit(&d_gh);
if (!IS_ERR_OR_NULL(inode)) {
+ set_bit(GIF_ALLOC_FAILED, &ip->i_flags);
clear_nlink(inode);
- if (!free_vfs_inode)
+ if (ip->i_no_addr)
mark_inode_dirty(inode);
- set_bit(free_vfs_inode ? GIF_FREE_VFS_INODE : GIF_ALLOC_FAILED,
- &GFS2_I(inode)->i_flags);
if (inode->i_state & I_NEW)
iget_failed(inode);
else
iput(inode);
}
- if (gfs2_holder_initialized(ghs + 1))
- gfs2_glock_dq_uninit(ghs + 1);
+ if (gfs2_holder_initialized(&gh))
+ gfs2_glock_dq_uninit(&gh);
fail:
gfs2_qa_put(dip);
return error;
diff --git a/fs/gfs2/meta_io.c b/fs/gfs2/meta_io.c
index 6ed728a..3c41b86 100644
--- a/fs/gfs2/meta_io.c
+++ b/fs/gfs2/meta_io.c
@@ -442,6 +442,12 @@ void gfs2_journal_wipe(struct gfs2_inode *ip, u64 bstart, u32 blen)
struct buffer_head *bh;
int ty;
+ if (!ip->i_gl) {
+ /* This can only happen during incomplete inode creation. */
+ BUG_ON(!test_bit(GIF_ALLOC_FAILED, &ip->i_flags));
+ return;
+ }
+
gfs2_ail1_wipe(sdp, bstart, blen);
while (blen) {
ty = REMOVE_META;
diff --git a/fs/gfs2/super.c b/fs/gfs2/super.c
index b018957..999cc14 100644
--- a/fs/gfs2/super.c
+++ b/fs/gfs2/super.c
@@ -379,6 +379,7 @@ static int gfs2_lock_fs_check_clean(struct gfs2_sbd *sdp)
void gfs2_dinode_out(const struct gfs2_inode *ip, void *buf)
{
+ const struct inode *inode = &ip->i_inode;
struct gfs2_dinode *str = buf;
str->di_header.mh_magic = cpu_to_be32(GFS2_MAGIC);
@@ -386,15 +387,15 @@ void gfs2_dinode_out(const struct gfs2_inode *ip, void *buf)
str->di_header.mh_format = cpu_to_be32(GFS2_FORMAT_DI);
str->di_num.no_addr = cpu_to_be64(ip->i_no_addr);
str->di_num.no_formal_ino = cpu_to_be64(ip->i_no_formal_ino);
- str->di_mode = cpu_to_be32(ip->i_inode.i_mode);
- str->di_uid = cpu_to_be32(i_uid_read(&ip->i_inode));
- str->di_gid = cpu_to_be32(i_gid_read(&ip->i_inode));
- str->di_nlink = cpu_to_be32(ip->i_inode.i_nlink);
- str->di_size = cpu_to_be64(i_size_read(&ip->i_inode));
- str->di_blocks = cpu_to_be64(gfs2_get_inode_blocks(&ip->i_inode));
- str->di_atime = cpu_to_be64(ip->i_inode.i_atime.tv_sec);
- str->di_mtime = cpu_to_be64(ip->i_inode.i_mtime.tv_sec);
- str->di_ctime = cpu_to_be64(ip->i_inode.i_ctime.tv_sec);
+ str->di_mode = cpu_to_be32(inode->i_mode);
+ str->di_uid = cpu_to_be32(i_uid_read(inode));
+ str->di_gid = cpu_to_be32(i_gid_read(inode));
+ str->di_nlink = cpu_to_be32(inode->i_nlink);
+ str->di_size = cpu_to_be64(i_size_read(inode));
+ str->di_blocks = cpu_to_be64(gfs2_get_inode_blocks(inode));
+ str->di_atime = cpu_to_be64(inode->i_atime.tv_sec);
+ str->di_mtime = cpu_to_be64(inode->i_mtime.tv_sec);
+ str->di_ctime = cpu_to_be64(inode->i_ctime.tv_sec);
str->di_goal_meta = cpu_to_be64(ip->i_goal);
str->di_goal_data = cpu_to_be64(ip->i_goal);
@@ -402,16 +403,16 @@ void gfs2_dinode_out(const struct gfs2_inode *ip, void *buf)
str->di_flags = cpu_to_be32(ip->i_diskflags);
str->di_height = cpu_to_be16(ip->i_height);
- str->di_payload_format = cpu_to_be32(S_ISDIR(ip->i_inode.i_mode) &&
+ str->di_payload_format = cpu_to_be32(S_ISDIR(inode->i_mode) &&
!(ip->i_diskflags & GFS2_DIF_EXHASH) ?
GFS2_FORMAT_DE : 0);
str->di_depth = cpu_to_be16(ip->i_depth);
str->di_entries = cpu_to_be32(ip->i_entries);
str->di_eattr = cpu_to_be64(ip->i_eattr);
- str->di_atime_nsec = cpu_to_be32(ip->i_inode.i_atime.tv_nsec);
- str->di_mtime_nsec = cpu_to_be32(ip->i_inode.i_mtime.tv_nsec);
- str->di_ctime_nsec = cpu_to_be32(ip->i_inode.i_ctime.tv_nsec);
+ str->di_atime_nsec = cpu_to_be32(inode->i_atime.tv_nsec);
+ str->di_mtime_nsec = cpu_to_be32(inode->i_mtime.tv_nsec);
+ str->di_ctime_nsec = cpu_to_be32(inode->i_ctime.tv_nsec);
}
/**
@@ -475,6 +476,12 @@ static void gfs2_dirty_inode(struct inode *inode, int flags)
int need_endtrans = 0;
int ret;
+ if (unlikely(!ip->i_gl)) {
+ /* This can only happen during incomplete inode creation. */
+ BUG_ON(!test_bit(GIF_ALLOC_FAILED, &ip->i_flags));
+ return;
+ }
+
if (unlikely(gfs2_withdrawn(sdp)))
return;
if (!gfs2_glock_is_locked_by_me(ip->i_gl)) {
@@ -927,8 +934,7 @@ static int gfs2_drop_inode(struct inode *inode)
{
struct gfs2_inode *ip = GFS2_I(inode);
- if (!test_bit(GIF_FREE_VFS_INODE, &ip->i_flags) &&
- inode->i_nlink &&
+ if (inode->i_nlink &&
gfs2_holder_initialized(&ip->i_iopen_gh)) {
struct gfs2_glock *gl = ip->i_iopen_gh.gh_gl;
if (test_bit(GLF_DEMOTE, &gl->gl_flags))
@@ -1076,7 +1082,13 @@ static void gfs2_final_release_pages(struct gfs2_inode *ip)
struct inode *inode = &ip->i_inode;
struct gfs2_glock *gl = ip->i_gl;
- truncate_inode_pages(gfs2_glock2aspace(ip->i_gl), 0);
+ if (unlikely(!gl)) {
+ /* This can only happen during incomplete inode creation. */
+ BUG_ON(!test_bit(GIF_ALLOC_FAILED, &ip->i_flags));
+ return;
+ }
+
+ truncate_inode_pages(gfs2_glock2aspace(gl), 0);
truncate_inode_pages(&inode->i_data, 0);
if (atomic_read(&gl->gl_revokes) == 0) {
@@ -1218,10 +1230,8 @@ static enum dinode_demise evict_should_delete(struct inode *inode,
struct gfs2_sbd *sdp = sb->s_fs_info;
int ret;
- if (test_bit(GIF_ALLOC_FAILED, &ip->i_flags)) {
- BUG_ON(!gfs2_glock_is_locked_by_me(ip->i_gl));
+ if (unlikely(test_bit(GIF_ALLOC_FAILED, &ip->i_flags)))
goto should_delete;
- }
if (test_bit(GIF_DEFERRED_DELETE, &ip->i_flags))
return SHOULD_DEFER_EVICTION;
@@ -1294,13 +1304,22 @@ static int evict_unlinked_inode(struct inode *inode)
goto out;
}
- /* We're about to clear the bitmap for the dinode, but as soon as we
- do, gfs2_create_inode can create another inode at the same block
- location and try to set gl_object again. We clear gl_object here so
- that subsequent inode creates don't see an old gl_object. */
- glock_clear_object(ip->i_gl, ip);
+ if (ip->i_gl)
+ gfs2_inode_remember_delete(ip->i_gl, ip->i_no_formal_ino);
+
+ /*
+ * As soon as we clear the bitmap for the dinode, gfs2_create_inode()
+ * can get called to recreate it, or even gfs2_inode_lookup() if the
+ * inode was recreated on another node in the meantime.
+ *
+ * However, inserting the new inode into the inode hash table will not
+ * succeed until the old inode is removed, and that only happens after
+ * ->evict_inode() returns. The new inode is attached to its inode and
+ * iopen glocks after inserting it into the inode hash table, so at
+ * that point we can be sure that both glocks are unused.
+ */
+
ret = gfs2_dinode_dealloc(ip);
- gfs2_inode_remember_delete(ip->i_gl, ip->i_no_formal_ino);
out:
return ret;
}
@@ -1367,12 +1386,7 @@ static void gfs2_evict_inode(struct inode *inode)
struct gfs2_holder gh;
int ret;
- if (test_bit(GIF_FREE_VFS_INODE, &ip->i_flags)) {
- clear_inode(inode);
- return;
- }
-
- if (inode->i_nlink || sb_rdonly(sb))
+ if (inode->i_nlink || sb_rdonly(sb) || !ip->i_no_addr)
goto out;
gfs2_holder_mark_uninitialized(&gh);
@@ -1405,12 +1419,9 @@ static void gfs2_evict_inode(struct inode *inode)
struct gfs2_glock *gl = ip->i_iopen_gh.gh_gl;
glock_clear_object(gl, ip);
- if (test_bit(HIF_HOLDER, &ip->i_iopen_gh.gh_iflags)) {
- ip->i_iopen_gh.gh_flags |= GL_NOCACHE;
- gfs2_glock_dq(&ip->i_iopen_gh);
- }
gfs2_glock_hold(gl);
- gfs2_holder_uninit(&ip->i_iopen_gh);
+ ip->i_iopen_gh.gh_flags |= GL_NOCACHE;
+ gfs2_glock_dq_uninit(&ip->i_iopen_gh);
gfs2_glock_put_eventually(gl);
}
if (ip->i_gl) {
@@ -1429,6 +1440,7 @@ static struct inode *gfs2_alloc_inode(struct super_block *sb)
ip = alloc_inode_sb(sb, gfs2_inode_cachep, GFP_KERNEL);
if (!ip)
return NULL;
+ ip->i_no_addr = 0;
ip->i_flags = 0;
ip->i_gl = NULL;
gfs2_holder_mark_uninitialized(&ip->i_iopen_gh);
diff --git a/fs/gfs2/xattr.c b/fs/gfs2/xattr.c
index f6a6605..518c067 100644
--- a/fs/gfs2/xattr.c
+++ b/fs/gfs2/xattr.c
@@ -1412,11 +1412,13 @@ static int ea_dealloc_block(struct gfs2_inode *ip)
ip->i_eattr = 0;
gfs2_add_inode_blocks(&ip->i_inode, -1);
- error = gfs2_meta_inode_buffer(ip, &dibh);
- if (!error) {
- gfs2_trans_add_meta(ip->i_gl, dibh);
- gfs2_dinode_out(ip, dibh->b_data);
- brelse(dibh);
+ if (likely(!test_bit(GIF_ALLOC_FAILED, &ip->i_flags))) {
+ error = gfs2_meta_inode_buffer(ip, &dibh);
+ if (!error) {
+ gfs2_trans_add_meta(ip->i_gl, dibh);
+ gfs2_dinode_out(ip, dibh->b_data);
+ brelse(dibh);
+ }
}
gfs2_trans_end(sdp);
@@ -1445,14 +1447,16 @@ int gfs2_ea_dealloc(struct gfs2_inode *ip)
if (error)
return error;
- error = ea_foreach(ip, ea_dealloc_unstuffed, NULL);
- if (error)
- goto out_quota;
-
- if (ip->i_diskflags & GFS2_DIF_EA_INDIRECT) {
- error = ea_dealloc_indirect(ip);
+ if (likely(!test_bit(GIF_ALLOC_FAILED, &ip->i_flags))) {
+ error = ea_foreach(ip, ea_dealloc_unstuffed, NULL);
if (error)
goto out_quota;
+
+ if (ip->i_diskflags & GFS2_DIF_EA_INDIRECT) {
+ error = ea_dealloc_indirect(ip);
+ if (error)
+ goto out_quota;
+ }
}
error = ea_dealloc_block(ip);
diff --git a/fs/nfsd/filecache.c b/fs/nfsd/filecache.c
index 1998b4d..45b2c9e 100644
--- a/fs/nfsd/filecache.c
+++ b/fs/nfsd/filecache.c
@@ -324,8 +324,7 @@ nfsd_file_alloc(struct nfsd_file_lookup_key *key, unsigned int may)
if (key->gc)
__set_bit(NFSD_FILE_GC, &nf->nf_flags);
nf->nf_inode = key->inode;
- /* nf_ref is pre-incremented for hash table */
- refcount_set(&nf->nf_ref, 2);
+ refcount_set(&nf->nf_ref, 1);
nf->nf_may = key->need;
nf->nf_mark = NULL;
}
@@ -377,24 +376,35 @@ nfsd_file_unhash(struct nfsd_file *nf)
return false;
}
-static bool
+static void
nfsd_file_free(struct nfsd_file *nf)
{
s64 age = ktime_to_ms(ktime_sub(ktime_get(), nf->nf_birthtime));
- bool flush = false;
trace_nfsd_file_free(nf);
this_cpu_inc(nfsd_file_releases);
this_cpu_add(nfsd_file_total_age, age);
+ nfsd_file_unhash(nf);
+
+ /*
+ * We call fsync here in order to catch writeback errors. It's not
+ * strictly required by the protocol, but an nfsd_file could get
+ * evicted from the cache before a COMMIT comes in. If another
+ * task were to open that file in the interim and scrape the error,
+ * then the client may never see it. By calling fsync here, we ensure
+ * that writeback happens before the entry is freed, and that any
+ * errors reported result in the write verifier changing.
+ */
+ nfsd_file_fsync(nf);
+
if (nf->nf_mark)
nfsd_file_mark_put(nf->nf_mark);
if (nf->nf_file) {
get_file(nf->nf_file);
filp_close(nf->nf_file, NULL);
fput(nf->nf_file);
- flush = true;
}
/*
@@ -402,10 +412,9 @@ nfsd_file_free(struct nfsd_file *nf)
* WARN and leak it to preserve system stability.
*/
if (WARN_ON_ONCE(!list_empty(&nf->nf_lru)))
- return flush;
+ return;
call_rcu(&nf->nf_rcu, nfsd_file_slab_free);
- return flush;
}
static bool
@@ -421,17 +430,23 @@ nfsd_file_check_writeback(struct nfsd_file *nf)
mapping_tagged(mapping, PAGECACHE_TAG_WRITEBACK);
}
-static void nfsd_file_lru_add(struct nfsd_file *nf)
+static bool nfsd_file_lru_add(struct nfsd_file *nf)
{
set_bit(NFSD_FILE_REFERENCED, &nf->nf_flags);
- if (list_lru_add(&nfsd_file_lru, &nf->nf_lru))
+ if (list_lru_add(&nfsd_file_lru, &nf->nf_lru)) {
trace_nfsd_file_lru_add(nf);
+ return true;
+ }
+ return false;
}
-static void nfsd_file_lru_remove(struct nfsd_file *nf)
+static bool nfsd_file_lru_remove(struct nfsd_file *nf)
{
- if (list_lru_del(&nfsd_file_lru, &nf->nf_lru))
+ if (list_lru_del(&nfsd_file_lru, &nf->nf_lru)) {
trace_nfsd_file_lru_del(nf);
+ return true;
+ }
+ return false;
}
struct nfsd_file *
@@ -442,54 +457,48 @@ nfsd_file_get(struct nfsd_file *nf)
return NULL;
}
-static void
-nfsd_file_unhash_and_queue(struct nfsd_file *nf, struct list_head *dispose)
-{
- trace_nfsd_file_unhash_and_queue(nf);
- if (nfsd_file_unhash(nf)) {
- /* caller must call nfsd_file_dispose_list() later */
- nfsd_file_lru_remove(nf);
- list_add(&nf->nf_lru, dispose);
- }
-}
-
-static void
-nfsd_file_put_noref(struct nfsd_file *nf)
-{
- trace_nfsd_file_put(nf);
-
- if (refcount_dec_and_test(&nf->nf_ref)) {
- WARN_ON(test_bit(NFSD_FILE_HASHED, &nf->nf_flags));
- nfsd_file_lru_remove(nf);
- nfsd_file_free(nf);
- }
-}
-
-static void
-nfsd_file_unhash_and_put(struct nfsd_file *nf)
-{
- if (nfsd_file_unhash(nf))
- nfsd_file_put_noref(nf);
-}
-
+/**
+ * nfsd_file_put - put the reference to a nfsd_file
+ * @nf: nfsd_file of which to put the reference
+ *
+ * Put a reference to a nfsd_file. In the non-GC case, we just put the
+ * reference immediately. In the GC case, if the reference would be
+ * the last one, the put it on the LRU instead to be cleaned up later.
+ */
void
nfsd_file_put(struct nfsd_file *nf)
{
might_sleep();
+ trace_nfsd_file_put(nf);
- if (test_bit(NFSD_FILE_GC, &nf->nf_flags))
- nfsd_file_lru_add(nf);
- else if (refcount_read(&nf->nf_ref) == 2)
- nfsd_file_unhash_and_put(nf);
+ if (test_bit(NFSD_FILE_GC, &nf->nf_flags) &&
+ test_bit(NFSD_FILE_HASHED, &nf->nf_flags)) {
+ /*
+ * If this is the last reference (nf_ref == 1), then try to
+ * transfer it to the LRU.
+ */
+ if (refcount_dec_not_one(&nf->nf_ref))
+ return;
- if (!test_bit(NFSD_FILE_HASHED, &nf->nf_flags)) {
- nfsd_file_fsync(nf);
- nfsd_file_put_noref(nf);
- } else if (nf->nf_file && test_bit(NFSD_FILE_GC, &nf->nf_flags)) {
- nfsd_file_put_noref(nf);
- nfsd_file_schedule_laundrette();
- } else
- nfsd_file_put_noref(nf);
+ /* Try to add it to the LRU. If that fails, decrement. */
+ if (nfsd_file_lru_add(nf)) {
+ /* If it's still hashed, we're done */
+ if (test_bit(NFSD_FILE_HASHED, &nf->nf_flags)) {
+ nfsd_file_schedule_laundrette();
+ return;
+ }
+
+ /*
+ * We're racing with unhashing, so try to remove it from
+ * the LRU. If removal fails, then someone else already
+ * has our reference.
+ */
+ if (!nfsd_file_lru_remove(nf))
+ return;
+ }
+ }
+ if (refcount_dec_and_test(&nf->nf_ref))
+ nfsd_file_free(nf);
}
static void
@@ -497,34 +506,14 @@ nfsd_file_dispose_list(struct list_head *dispose)
{
struct nfsd_file *nf;
- while(!list_empty(dispose)) {
+ while (!list_empty(dispose)) {
nf = list_first_entry(dispose, struct nfsd_file, nf_lru);
list_del_init(&nf->nf_lru);
- nfsd_file_fsync(nf);
- nfsd_file_put_noref(nf);
+ nfsd_file_free(nf);
}
}
static void
-nfsd_file_dispose_list_sync(struct list_head *dispose)
-{
- bool flush = false;
- struct nfsd_file *nf;
-
- while(!list_empty(dispose)) {
- nf = list_first_entry(dispose, struct nfsd_file, nf_lru);
- list_del_init(&nf->nf_lru);
- nfsd_file_fsync(nf);
- if (!refcount_dec_and_test(&nf->nf_ref))
- continue;
- if (nfsd_file_free(nf))
- flush = true;
- }
- if (flush)
- flush_delayed_fput();
-}
-
-static void
nfsd_file_list_remove_disposal(struct list_head *dst,
struct nfsd_fcache_disposal *l)
{
@@ -591,21 +580,8 @@ nfsd_file_lru_cb(struct list_head *item, struct list_lru_one *lru,
struct list_head *head = arg;
struct nfsd_file *nf = list_entry(item, struct nfsd_file, nf_lru);
- /*
- * Do a lockless refcount check. The hashtable holds one reference, so
- * we look to see if anything else has a reference, or if any have
- * been put since the shrinker last ran. Those don't get unhashed and
- * released.
- *
- * Note that in the put path, we set the flag and then decrement the
- * counter. Here we check the counter and then test and clear the flag.
- * That order is deliberate to ensure that we can do this locklessly.
- */
- if (refcount_read(&nf->nf_ref) > 1) {
- list_lru_isolate(lru, &nf->nf_lru);
- trace_nfsd_file_gc_in_use(nf);
- return LRU_REMOVED;
- }
+ /* We should only be dealing with GC entries here */
+ WARN_ON_ONCE(!test_bit(NFSD_FILE_GC, &nf->nf_flags));
/*
* Don't throw out files that are still undergoing I/O or
@@ -616,40 +592,30 @@ nfsd_file_lru_cb(struct list_head *item, struct list_lru_one *lru,
return LRU_SKIP;
}
+ /* If it was recently added to the list, skip it */
if (test_and_clear_bit(NFSD_FILE_REFERENCED, &nf->nf_flags)) {
trace_nfsd_file_gc_referenced(nf);
return LRU_ROTATE;
}
- if (!test_and_clear_bit(NFSD_FILE_HASHED, &nf->nf_flags)) {
- trace_nfsd_file_gc_hashed(nf);
- return LRU_SKIP;
+ /*
+ * Put the reference held on behalf of the LRU. If it wasn't the last
+ * one, then just remove it from the LRU and ignore it.
+ */
+ if (!refcount_dec_and_test(&nf->nf_ref)) {
+ trace_nfsd_file_gc_in_use(nf);
+ list_lru_isolate(lru, &nf->nf_lru);
+ return LRU_REMOVED;
}
+ /* Refcount went to zero. Unhash it and queue it to the dispose list */
+ nfsd_file_unhash(nf);
list_lru_isolate_move(lru, &nf->nf_lru, head);
this_cpu_inc(nfsd_file_evictions);
trace_nfsd_file_gc_disposed(nf);
return LRU_REMOVED;
}
-/*
- * Unhash items on @dispose immediately, then queue them on the
- * disposal workqueue to finish releasing them in the background.
- *
- * cel: Note that between the time list_lru_shrink_walk runs and
- * now, these items are in the hash table but marked unhashed.
- * Why release these outside of lru_cb ? There's no lock ordering
- * problem since lru_cb currently takes no lock.
- */
-static void nfsd_file_gc_dispose_list(struct list_head *dispose)
-{
- struct nfsd_file *nf;
-
- list_for_each_entry(nf, dispose, nf_lru)
- nfsd_file_hash_remove(nf);
- nfsd_file_dispose_list_delayed(dispose);
-}
-
static void
nfsd_file_gc(void)
{
@@ -659,7 +625,7 @@ nfsd_file_gc(void)
ret = list_lru_walk(&nfsd_file_lru, nfsd_file_lru_cb,
&dispose, list_lru_count(&nfsd_file_lru));
trace_nfsd_file_gc_removed(ret, list_lru_count(&nfsd_file_lru));
- nfsd_file_gc_dispose_list(&dispose);
+ nfsd_file_dispose_list_delayed(&dispose);
}
static void
@@ -685,7 +651,7 @@ nfsd_file_lru_scan(struct shrinker *s, struct shrink_control *sc)
ret = list_lru_shrink_walk(&nfsd_file_lru, sc,
nfsd_file_lru_cb, &dispose);
trace_nfsd_file_shrinker_removed(ret, list_lru_count(&nfsd_file_lru));
- nfsd_file_gc_dispose_list(&dispose);
+ nfsd_file_dispose_list_delayed(&dispose);
return ret;
}
@@ -695,72 +661,111 @@ static struct shrinker nfsd_file_shrinker = {
.seeks = 1,
};
-/*
- * Find all cache items across all net namespaces that match @inode and
- * move them to @dispose. The lookup is atomic wrt nfsd_file_acquire().
+/**
+ * nfsd_file_queue_for_close: try to close out any open nfsd_files for an inode
+ * @inode: inode on which to close out nfsd_files
+ * @dispose: list on which to gather nfsd_files to close out
+ *
+ * An nfsd_file represents a struct file being held open on behalf of nfsd. An
+ * open file however can block other activity (such as leases), or cause
+ * undesirable behavior (e.g. spurious silly-renames when reexporting NFS).
+ *
+ * This function is intended to find open nfsd_files when this sort of
+ * conflicting access occurs and then attempt to close those files out.
+ *
+ * Populates the dispose list with entries that have already had their
+ * refcounts go to zero. The actual free of an nfsd_file can be expensive,
+ * so we leave it up to the caller whether it wants to wait or not.
*/
-static unsigned int
-__nfsd_file_close_inode(struct inode *inode, struct list_head *dispose)
+static void
+nfsd_file_queue_for_close(struct inode *inode, struct list_head *dispose)
{
struct nfsd_file_lookup_key key = {
.type = NFSD_FILE_KEY_INODE,
.inode = inode,
};
- unsigned int count = 0;
struct nfsd_file *nf;
rcu_read_lock();
do {
+ int decrement = 1;
+
nf = rhashtable_lookup(&nfsd_file_rhash_tbl, &key,
nfsd_file_rhash_params);
if (!nf)
break;
- nfsd_file_unhash_and_queue(nf, dispose);
- count++;
+
+ /* If we raced with someone else unhashing, ignore it */
+ if (!nfsd_file_unhash(nf))
+ continue;
+
+ /* If we can't get a reference, ignore it */
+ if (!nfsd_file_get(nf))
+ continue;
+
+ /* Extra decrement if we remove from the LRU */
+ if (nfsd_file_lru_remove(nf))
+ ++decrement;
+
+ /* If refcount goes to 0, then put on the dispose list */
+ if (refcount_sub_and_test(decrement, &nf->nf_ref)) {
+ list_add(&nf->nf_lru, dispose);
+ trace_nfsd_file_closing(nf);
+ }
} while (1);
rcu_read_unlock();
- return count;
-}
-
-/**
- * nfsd_file_close_inode_sync - attempt to forcibly close a nfsd_file
- * @inode: inode of the file to attempt to remove
- *
- * Unhash and put, then flush and fput all cache items associated with @inode.
- */
-void
-nfsd_file_close_inode_sync(struct inode *inode)
-{
- LIST_HEAD(dispose);
- unsigned int count;
-
- count = __nfsd_file_close_inode(inode, &dispose);
- trace_nfsd_file_close_inode_sync(inode, count);
- nfsd_file_dispose_list_sync(&dispose);
}
/**
* nfsd_file_close_inode - attempt a delayed close of a nfsd_file
* @inode: inode of the file to attempt to remove
*
- * Unhash and put all cache item associated with @inode.
+ * Close out any open nfsd_files that can be reaped for @inode. The
+ * actual freeing is deferred to the dispose_list_delayed infrastructure.
+ *
+ * This is used by the fsnotify callbacks and setlease notifier.
*/
static void
nfsd_file_close_inode(struct inode *inode)
{
LIST_HEAD(dispose);
- unsigned int count;
- count = __nfsd_file_close_inode(inode, &dispose);
- trace_nfsd_file_close_inode(inode, count);
+ nfsd_file_queue_for_close(inode, &dispose);
nfsd_file_dispose_list_delayed(&dispose);
}
/**
+ * nfsd_file_close_inode_sync - attempt to forcibly close a nfsd_file
+ * @inode: inode of the file to attempt to remove
+ *
+ * Close out any open nfsd_files that can be reaped for @inode. The
+ * nfsd_files are closed out synchronously.
+ *
+ * This is called from nfsd_rename and nfsd_unlink to avoid silly-renames
+ * when reexporting NFS.
+ */
+void
+nfsd_file_close_inode_sync(struct inode *inode)
+{
+ struct nfsd_file *nf;
+ LIST_HEAD(dispose);
+
+ trace_nfsd_file_close(inode);
+
+ nfsd_file_queue_for_close(inode, &dispose);
+ while (!list_empty(&dispose)) {
+ nf = list_first_entry(&dispose, struct nfsd_file, nf_lru);
+ list_del_init(&nf->nf_lru);
+ nfsd_file_free(nf);
+ }
+ flush_delayed_fput();
+}
+
+/**
* nfsd_file_delayed_close - close unused nfsd_files
* @work: dummy
*
- * Walk the LRU list and close any entries that have not been used since
+ * Walk the LRU list and destroy any entries that have not been used since
* the last scan.
*/
static void
@@ -782,7 +787,7 @@ nfsd_file_lease_notifier_call(struct notifier_block *nb, unsigned long arg,
/* Only close files for F_SETLEASE leases */
if (fl->fl_flags & FL_LEASE)
- nfsd_file_close_inode_sync(file_inode(fl->fl_file));
+ nfsd_file_close_inode(file_inode(fl->fl_file));
return 0;
}
@@ -903,6 +908,13 @@ nfsd_file_cache_init(void)
goto out;
}
+/**
+ * __nfsd_file_cache_purge: clean out the cache for shutdown
+ * @net: net-namespace to shut down the cache (may be NULL)
+ *
+ * Walk the nfsd_file cache and close out any that match @net. If @net is NULL,
+ * then close out everything. Called when an nfsd instance is being shut down.
+ */
static void
__nfsd_file_cache_purge(struct net *net)
{
@@ -916,8 +928,11 @@ __nfsd_file_cache_purge(struct net *net)
nf = rhashtable_walk_next(&iter);
while (!IS_ERR_OR_NULL(nf)) {
- if (!net || nf->nf_net == net)
- nfsd_file_unhash_and_queue(nf, &dispose);
+ if (!net || nf->nf_net == net) {
+ nfsd_file_unhash(nf);
+ nfsd_file_lru_remove(nf);
+ list_add(&nf->nf_lru, &dispose);
+ }
nf = rhashtable_walk_next(&iter);
}
@@ -1084,8 +1099,12 @@ nfsd_file_do_acquire(struct svc_rqst *rqstp, struct svc_fh *fhp,
if (nf)
nf = nfsd_file_get(nf);
rcu_read_unlock();
- if (nf)
+
+ if (nf) {
+ if (nfsd_file_lru_remove(nf))
+ WARN_ON_ONCE(refcount_dec_and_test(&nf->nf_ref));
goto wait_for_construction;
+ }
nf = nfsd_file_alloc(&key, may_flags);
if (!nf) {
@@ -1118,11 +1137,11 @@ nfsd_file_do_acquire(struct svc_rqst *rqstp, struct svc_fh *fhp,
goto out;
}
open_retry = false;
- nfsd_file_put_noref(nf);
+ if (refcount_dec_and_test(&nf->nf_ref))
+ nfsd_file_free(nf);
goto retry;
}
- nfsd_file_lru_remove(nf);
this_cpu_inc(nfsd_file_cache_hits);
status = nfserrno(nfsd_open_break_lease(file_inode(nf->nf_file), may_flags));
@@ -1132,7 +1151,8 @@ nfsd_file_do_acquire(struct svc_rqst *rqstp, struct svc_fh *fhp,
this_cpu_inc(nfsd_file_acquisitions);
*pnf = nf;
} else {
- nfsd_file_put(nf);
+ if (refcount_dec_and_test(&nf->nf_ref))
+ nfsd_file_free(nf);
nf = NULL;
}
@@ -1158,8 +1178,10 @@ nfsd_file_do_acquire(struct svc_rqst *rqstp, struct svc_fh *fhp,
* If construction failed, or we raced with a call to unlink()
* then unhash.
*/
- if (status != nfs_ok || key.inode->i_nlink == 0)
- nfsd_file_unhash_and_put(nf);
+ if (status == nfs_ok && key.inode->i_nlink == 0)
+ status = nfserr_jukebox;
+ if (status != nfs_ok)
+ nfsd_file_unhash(nf);
clear_bit_unlock(NFSD_FILE_PENDING, &nf->nf_flags);
smp_mb__after_atomic();
wake_up_bit(&nf->nf_flags, NFSD_FILE_PENDING);
diff --git a/fs/nfsd/nfs4callback.c b/fs/nfsd/nfs4callback.c
index d6e1d38..2a815f5 100644
--- a/fs/nfsd/nfs4callback.c
+++ b/fs/nfsd/nfs4callback.c
@@ -988,7 +988,6 @@ static int setup_callback_client(struct nfs4_client *clp, struct nfs4_cb_conn *c
} else {
if (!conn->cb_xprt)
return -EINVAL;
- clp->cl_cb_conn.cb_xprt = conn->cb_xprt;
clp->cl_cb_session = ses;
args.bc_xprt = conn->cb_xprt;
args.prognumber = clp->cl_cb_session->se_cb_prog;
@@ -1008,6 +1007,9 @@ static int setup_callback_client(struct nfs4_client *clp, struct nfs4_cb_conn *c
rpc_shutdown_client(client);
return -ENOMEM;
}
+
+ if (clp->cl_minorversion != 0)
+ clp->cl_cb_conn.cb_xprt = conn->cb_xprt;
clp->cl_cb_client = client;
clp->cl_cb_cred = cred;
rcu_read_lock();
diff --git a/fs/nfsd/nfs4proc.c b/fs/nfsd/nfs4proc.c
index 73ed32a..bd880d5 100644
--- a/fs/nfsd/nfs4proc.c
+++ b/fs/nfsd/nfs4proc.c
@@ -1461,13 +1461,6 @@ nfsd4_interssc_connect(struct nl4_server *nss, struct svc_rqst *rqstp,
return status;
}
-static void
-nfsd4_interssc_disconnect(struct vfsmount *ss_mnt)
-{
- nfs_do_sb_deactive(ss_mnt->mnt_sb);
- mntput(ss_mnt);
-}
-
/*
* Verify COPY destination stateid.
*
@@ -1570,11 +1563,6 @@ nfsd4_cleanup_inter_ssc(struct vfsmount *ss_mnt, struct file *filp,
{
}
-static void
-nfsd4_interssc_disconnect(struct vfsmount *ss_mnt)
-{
-}
-
static struct file *nfs42_ssc_open(struct vfsmount *ss_mnt,
struct nfs_fh *src_fh,
nfs4_stateid *stateid)
@@ -1770,7 +1758,7 @@ static int nfsd4_do_async_copy(void *data)
default:
nfserr = nfserr_offload_denied;
}
- nfsd4_interssc_disconnect(copy->ss_mnt);
+ /* ss_mnt will be unmounted by the laundromat */
goto do_callback;
}
nfserr = nfsd4_do_copy(copy, filp, copy->nf_dst->nf_file,
@@ -1851,8 +1839,10 @@ nfsd4_copy(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate,
if (async_copy)
cleanup_async_copy(async_copy);
status = nfserrno(-ENOMEM);
- if (nfsd4_ssc_is_inter(copy))
- nfsd4_interssc_disconnect(copy->ss_mnt);
+ /*
+ * source's vfsmount of inter-copy will be unmounted
+ * by the laundromat
+ */
goto out;
}
diff --git a/fs/nfsd/trace.h b/fs/nfsd/trace.h
index 46b8f68..c852ae8e 100644
--- a/fs/nfsd/trace.h
+++ b/fs/nfsd/trace.h
@@ -876,8 +876,8 @@ DEFINE_CLID_EVENT(confirmed_r);
__print_flags(val, "|", \
{ 1 << NFSD_FILE_HASHED, "HASHED" }, \
{ 1 << NFSD_FILE_PENDING, "PENDING" }, \
- { 1 << NFSD_FILE_REFERENCED, "REFERENCED"}, \
- { 1 << NFSD_FILE_GC, "GC"})
+ { 1 << NFSD_FILE_REFERENCED, "REFERENCED" }, \
+ { 1 << NFSD_FILE_GC, "GC" })
DECLARE_EVENT_CLASS(nfsd_file_class,
TP_PROTO(struct nfsd_file *nf),
@@ -912,6 +912,7 @@ DEFINE_EVENT(nfsd_file_class, name, \
DEFINE_NFSD_FILE_EVENT(nfsd_file_free);
DEFINE_NFSD_FILE_EVENT(nfsd_file_unhash);
DEFINE_NFSD_FILE_EVENT(nfsd_file_put);
+DEFINE_NFSD_FILE_EVENT(nfsd_file_closing);
DEFINE_NFSD_FILE_EVENT(nfsd_file_unhash_and_queue);
TRACE_EVENT(nfsd_file_alloc,
@@ -1103,35 +1104,6 @@ TRACE_EVENT(nfsd_file_open,
__entry->nf_file)
)
-DECLARE_EVENT_CLASS(nfsd_file_search_class,
- TP_PROTO(
- const struct inode *inode,
- unsigned int count
- ),
- TP_ARGS(inode, count),
- TP_STRUCT__entry(
- __field(const struct inode *, inode)
- __field(unsigned int, count)
- ),
- TP_fast_assign(
- __entry->inode = inode;
- __entry->count = count;
- ),
- TP_printk("inode=%p count=%u",
- __entry->inode, __entry->count)
-);
-
-#define DEFINE_NFSD_FILE_SEARCH_EVENT(name) \
-DEFINE_EVENT(nfsd_file_search_class, name, \
- TP_PROTO( \
- const struct inode *inode, \
- unsigned int count \
- ), \
- TP_ARGS(inode, count))
-
-DEFINE_NFSD_FILE_SEARCH_EVENT(nfsd_file_close_inode_sync);
-DEFINE_NFSD_FILE_SEARCH_EVENT(nfsd_file_close_inode);
-
TRACE_EVENT(nfsd_file_is_cached,
TP_PROTO(
const struct inode *inode,
@@ -1209,7 +1181,6 @@ DEFINE_NFSD_FILE_GC_EVENT(nfsd_file_lru_del_disposed);
DEFINE_NFSD_FILE_GC_EVENT(nfsd_file_gc_in_use);
DEFINE_NFSD_FILE_GC_EVENT(nfsd_file_gc_writeback);
DEFINE_NFSD_FILE_GC_EVENT(nfsd_file_gc_referenced);
-DEFINE_NFSD_FILE_GC_EVENT(nfsd_file_gc_hashed);
DEFINE_NFSD_FILE_GC_EVENT(nfsd_file_gc_disposed);
DECLARE_EVENT_CLASS(nfsd_file_lruwalk_class,
@@ -1241,6 +1212,22 @@ DEFINE_EVENT(nfsd_file_lruwalk_class, name, \
DEFINE_NFSD_FILE_LRUWALK_EVENT(nfsd_file_gc_removed);
DEFINE_NFSD_FILE_LRUWALK_EVENT(nfsd_file_shrinker_removed);
+TRACE_EVENT(nfsd_file_close,
+ TP_PROTO(
+ const struct inode *inode
+ ),
+ TP_ARGS(inode),
+ TP_STRUCT__entry(
+ __field(const void *, inode)
+ ),
+ TP_fast_assign(
+ __entry->inode = inode;
+ ),
+ TP_printk("inode=%p",
+ __entry->inode
+ )
+);
+
TRACE_EVENT(nfsd_file_fsync,
TP_PROTO(
const struct nfsd_file *nf,
diff --git a/include/asm-generic/io.h b/include/asm-generic/io.h
index a68f8fb..4c44a29 100644
--- a/include/asm-generic/io.h
+++ b/include/asm-generic/io.h
@@ -80,24 +80,24 @@ DECLARE_TRACEPOINT(rwmmio_read);
DECLARE_TRACEPOINT(rwmmio_post_read);
void log_write_mmio(u64 val, u8 width, volatile void __iomem *addr,
- unsigned long caller_addr);
+ unsigned long caller_addr, unsigned long caller_addr0);
void log_post_write_mmio(u64 val, u8 width, volatile void __iomem *addr,
- unsigned long caller_addr);
+ unsigned long caller_addr, unsigned long caller_addr0);
void log_read_mmio(u8 width, const volatile void __iomem *addr,
- unsigned long caller_addr);
+ unsigned long caller_addr, unsigned long caller_addr0);
void log_post_read_mmio(u64 val, u8 width, const volatile void __iomem *addr,
- unsigned long caller_addr);
+ unsigned long caller_addr, unsigned long caller_addr0);
#else
static inline void log_write_mmio(u64 val, u8 width, volatile void __iomem *addr,
- unsigned long caller_addr) {}
+ unsigned long caller_addr, unsigned long caller_addr0) {}
static inline void log_post_write_mmio(u64 val, u8 width, volatile void __iomem *addr,
- unsigned long caller_addr) {}
+ unsigned long caller_addr, unsigned long caller_addr0) {}
static inline void log_read_mmio(u8 width, const volatile void __iomem *addr,
- unsigned long caller_addr) {}
+ unsigned long caller_addr, unsigned long caller_addr0) {}
static inline void log_post_read_mmio(u64 val, u8 width, const volatile void __iomem *addr,
- unsigned long caller_addr) {}
+ unsigned long caller_addr, unsigned long caller_addr0) {}
#endif /* CONFIG_TRACE_MMIO_ACCESS */
@@ -188,11 +188,11 @@ static inline u8 readb(const volatile void __iomem *addr)
{
u8 val;
- log_read_mmio(8, addr, _THIS_IP_);
+ log_read_mmio(8, addr, _THIS_IP_, _RET_IP_);
__io_br();
val = __raw_readb(addr);
__io_ar(val);
- log_post_read_mmio(val, 8, addr, _THIS_IP_);
+ log_post_read_mmio(val, 8, addr, _THIS_IP_, _RET_IP_);
return val;
}
#endif
@@ -203,11 +203,11 @@ static inline u16 readw(const volatile void __iomem *addr)
{
u16 val;
- log_read_mmio(16, addr, _THIS_IP_);
+ log_read_mmio(16, addr, _THIS_IP_, _RET_IP_);
__io_br();
val = __le16_to_cpu((__le16 __force)__raw_readw(addr));
__io_ar(val);
- log_post_read_mmio(val, 16, addr, _THIS_IP_);
+ log_post_read_mmio(val, 16, addr, _THIS_IP_, _RET_IP_);
return val;
}
#endif
@@ -218,11 +218,11 @@ static inline u32 readl(const volatile void __iomem *addr)
{
u32 val;
- log_read_mmio(32, addr, _THIS_IP_);
+ log_read_mmio(32, addr, _THIS_IP_, _RET_IP_);
__io_br();
val = __le32_to_cpu((__le32 __force)__raw_readl(addr));
__io_ar(val);
- log_post_read_mmio(val, 32, addr, _THIS_IP_);
+ log_post_read_mmio(val, 32, addr, _THIS_IP_, _RET_IP_);
return val;
}
#endif
@@ -234,11 +234,11 @@ static inline u64 readq(const volatile void __iomem *addr)
{
u64 val;
- log_read_mmio(64, addr, _THIS_IP_);
+ log_read_mmio(64, addr, _THIS_IP_, _RET_IP_);
__io_br();
val = __le64_to_cpu(__raw_readq(addr));
__io_ar(val);
- log_post_read_mmio(val, 64, addr, _THIS_IP_);
+ log_post_read_mmio(val, 64, addr, _THIS_IP_, _RET_IP_);
return val;
}
#endif
@@ -248,11 +248,11 @@ static inline u64 readq(const volatile void __iomem *addr)
#define writeb writeb
static inline void writeb(u8 value, volatile void __iomem *addr)
{
- log_write_mmio(value, 8, addr, _THIS_IP_);
+ log_write_mmio(value, 8, addr, _THIS_IP_, _RET_IP_);
__io_bw();
__raw_writeb(value, addr);
__io_aw();
- log_post_write_mmio(value, 8, addr, _THIS_IP_);
+ log_post_write_mmio(value, 8, addr, _THIS_IP_, _RET_IP_);
}
#endif
@@ -260,11 +260,11 @@ static inline void writeb(u8 value, volatile void __iomem *addr)
#define writew writew
static inline void writew(u16 value, volatile void __iomem *addr)
{
- log_write_mmio(value, 16, addr, _THIS_IP_);
+ log_write_mmio(value, 16, addr, _THIS_IP_, _RET_IP_);
__io_bw();
__raw_writew((u16 __force)cpu_to_le16(value), addr);
__io_aw();
- log_post_write_mmio(value, 16, addr, _THIS_IP_);
+ log_post_write_mmio(value, 16, addr, _THIS_IP_, _RET_IP_);
}
#endif
@@ -272,11 +272,11 @@ static inline void writew(u16 value, volatile void __iomem *addr)
#define writel writel
static inline void writel(u32 value, volatile void __iomem *addr)
{
- log_write_mmio(value, 32, addr, _THIS_IP_);
+ log_write_mmio(value, 32, addr, _THIS_IP_, _RET_IP_);
__io_bw();
__raw_writel((u32 __force)__cpu_to_le32(value), addr);
__io_aw();
- log_post_write_mmio(value, 32, addr, _THIS_IP_);
+ log_post_write_mmio(value, 32, addr, _THIS_IP_, _RET_IP_);
}
#endif
@@ -285,11 +285,11 @@ static inline void writel(u32 value, volatile void __iomem *addr)
#define writeq writeq
static inline void writeq(u64 value, volatile void __iomem *addr)
{
- log_write_mmio(value, 64, addr, _THIS_IP_);
+ log_write_mmio(value, 64, addr, _THIS_IP_, _RET_IP_);
__io_bw();
__raw_writeq(__cpu_to_le64(value), addr);
__io_aw();
- log_post_write_mmio(value, 64, addr, _THIS_IP_);
+ log_post_write_mmio(value, 64, addr, _THIS_IP_, _RET_IP_);
}
#endif
#endif /* CONFIG_64BIT */
@@ -305,9 +305,9 @@ static inline u8 readb_relaxed(const volatile void __iomem *addr)
{
u8 val;
- log_read_mmio(8, addr, _THIS_IP_);
+ log_read_mmio(8, addr, _THIS_IP_, _RET_IP_);
val = __raw_readb(addr);
- log_post_read_mmio(val, 8, addr, _THIS_IP_);
+ log_post_read_mmio(val, 8, addr, _THIS_IP_, _RET_IP_);
return val;
}
#endif
@@ -318,9 +318,9 @@ static inline u16 readw_relaxed(const volatile void __iomem *addr)
{
u16 val;
- log_read_mmio(16, addr, _THIS_IP_);
+ log_read_mmio(16, addr, _THIS_IP_, _RET_IP_);
val = __le16_to_cpu(__raw_readw(addr));
- log_post_read_mmio(val, 16, addr, _THIS_IP_);
+ log_post_read_mmio(val, 16, addr, _THIS_IP_, _RET_IP_);
return val;
}
#endif
@@ -331,9 +331,9 @@ static inline u32 readl_relaxed(const volatile void __iomem *addr)
{
u32 val;
- log_read_mmio(32, addr, _THIS_IP_);
+ log_read_mmio(32, addr, _THIS_IP_, _RET_IP_);
val = __le32_to_cpu(__raw_readl(addr));
- log_post_read_mmio(val, 32, addr, _THIS_IP_);
+ log_post_read_mmio(val, 32, addr, _THIS_IP_, _RET_IP_);
return val;
}
#endif
@@ -344,9 +344,9 @@ static inline u64 readq_relaxed(const volatile void __iomem *addr)
{
u64 val;
- log_read_mmio(64, addr, _THIS_IP_);
+ log_read_mmio(64, addr, _THIS_IP_, _RET_IP_);
val = __le64_to_cpu(__raw_readq(addr));
- log_post_read_mmio(val, 64, addr, _THIS_IP_);
+ log_post_read_mmio(val, 64, addr, _THIS_IP_, _RET_IP_);
return val;
}
#endif
@@ -355,9 +355,9 @@ static inline u64 readq_relaxed(const volatile void __iomem *addr)
#define writeb_relaxed writeb_relaxed
static inline void writeb_relaxed(u8 value, volatile void __iomem *addr)
{
- log_write_mmio(value, 8, addr, _THIS_IP_);
+ log_write_mmio(value, 8, addr, _THIS_IP_, _RET_IP_);
__raw_writeb(value, addr);
- log_post_write_mmio(value, 8, addr, _THIS_IP_);
+ log_post_write_mmio(value, 8, addr, _THIS_IP_, _RET_IP_);
}
#endif
@@ -365,9 +365,9 @@ static inline void writeb_relaxed(u8 value, volatile void __iomem *addr)
#define writew_relaxed writew_relaxed
static inline void writew_relaxed(u16 value, volatile void __iomem *addr)
{
- log_write_mmio(value, 16, addr, _THIS_IP_);
+ log_write_mmio(value, 16, addr, _THIS_IP_, _RET_IP_);
__raw_writew(cpu_to_le16(value), addr);
- log_post_write_mmio(value, 16, addr, _THIS_IP_);
+ log_post_write_mmio(value, 16, addr, _THIS_IP_, _RET_IP_);
}
#endif
@@ -375,9 +375,9 @@ static inline void writew_relaxed(u16 value, volatile void __iomem *addr)
#define writel_relaxed writel_relaxed
static inline void writel_relaxed(u32 value, volatile void __iomem *addr)
{
- log_write_mmio(value, 32, addr, _THIS_IP_);
+ log_write_mmio(value, 32, addr, _THIS_IP_, _RET_IP_);
__raw_writel(__cpu_to_le32(value), addr);
- log_post_write_mmio(value, 32, addr, _THIS_IP_);
+ log_post_write_mmio(value, 32, addr, _THIS_IP_, _RET_IP_);
}
#endif
@@ -385,9 +385,9 @@ static inline void writel_relaxed(u32 value, volatile void __iomem *addr)
#define writeq_relaxed writeq_relaxed
static inline void writeq_relaxed(u64 value, volatile void __iomem *addr)
{
- log_write_mmio(value, 64, addr, _THIS_IP_);
+ log_write_mmio(value, 64, addr, _THIS_IP_, _RET_IP_);
__raw_writeq(__cpu_to_le64(value), addr);
- log_post_write_mmio(value, 64, addr, _THIS_IP_);
+ log_post_write_mmio(value, 64, addr, _THIS_IP_, _RET_IP_);
}
#endif
diff --git a/include/dt-bindings/memory/mediatek,mt8365-larb-port.h b/include/dt-bindings/memory/mediatek,mt8365-larb-port.h
new file mode 100644
index 0000000..56d5a5d
--- /dev/null
+++ b/include/dt-bindings/memory/mediatek,mt8365-larb-port.h
@@ -0,0 +1,90 @@
+/* SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause) */
+/*
+ * Copyright (c) 2022 MediaTek Inc.
+ * Author: Yong Wu <yong.wu@mediatek.com>
+ */
+#ifndef _DT_BINDINGS_MEMORY_MT8365_LARB_PORT_H_
+#define _DT_BINDINGS_MEMORY_MT8365_LARB_PORT_H_
+
+#include <dt-bindings/memory/mtk-memory-port.h>
+
+#define M4U_LARB0_ID 0
+#define M4U_LARB1_ID 1
+#define M4U_LARB2_ID 2
+#define M4U_LARB3_ID 3
+
+/* larb0 */
+#define M4U_PORT_DISP_OVL0 MTK_M4U_ID(M4U_LARB0_ID, 0)
+#define M4U_PORT_DISP_OVL0_2L MTK_M4U_ID(M4U_LARB0_ID, 1)
+#define M4U_PORT_DISP_RDMA0 MTK_M4U_ID(M4U_LARB0_ID, 2)
+#define M4U_PORT_DISP_WDMA0 MTK_M4U_ID(M4U_LARB0_ID, 3)
+#define M4U_PORT_DISP_RDMA1 MTK_M4U_ID(M4U_LARB0_ID, 4)
+#define M4U_PORT_MDP_RDMA0 MTK_M4U_ID(M4U_LARB0_ID, 5)
+#define M4U_PORT_MDP_WROT1 MTK_M4U_ID(M4U_LARB0_ID, 6)
+#define M4U_PORT_MDP_WROT0 MTK_M4U_ID(M4U_LARB0_ID, 7)
+#define M4U_PORT_MDP_RDMA1 MTK_M4U_ID(M4U_LARB0_ID, 8)
+#define M4U_PORT_DISP_FAKE0 MTK_M4U_ID(M4U_LARB0_ID, 9)
+#define M4U_PORT_APU_READ MTK_M4U_ID(M4U_LARB0_ID, 10)
+#define M4U_PORT_APU_WRITE MTK_M4U_ID(M4U_LARB0_ID, 11)
+
+/* larb1 */
+#define M4U_PORT_VENC_RCPU MTK_M4U_ID(M4U_LARB1_ID, 0)
+#define M4U_PORT_VENC_REC MTK_M4U_ID(M4U_LARB1_ID, 1)
+#define M4U_PORT_VENC_BSDMA MTK_M4U_ID(M4U_LARB1_ID, 2)
+#define M4U_PORT_VENC_SV_COMV MTK_M4U_ID(M4U_LARB1_ID, 3)
+#define M4U_PORT_VENC_RD_COMV MTK_M4U_ID(M4U_LARB1_ID, 4)
+#define M4U_PORT_VENC_NBM_RDMA MTK_M4U_ID(M4U_LARB1_ID, 5)
+#define M4U_PORT_VENC_NBM_RDMA_LITE MTK_M4U_ID(M4U_LARB1_ID, 6)
+#define M4U_PORT_JPGENC_Y_RDMA MTK_M4U_ID(M4U_LARB1_ID, 7)
+#define M4U_PORT_JPGENC_C_RDMA MTK_M4U_ID(M4U_LARB1_ID, 8)
+#define M4U_PORT_JPGENC_Q_TABLE MTK_M4U_ID(M4U_LARB1_ID, 9)
+#define M4U_PORT_JPGENC_BSDMA MTK_M4U_ID(M4U_LARB1_ID, 10)
+#define M4U_PORT_JPGDEC_WDMA MTK_M4U_ID(M4U_LARB1_ID, 11)
+#define M4U_PORT_JPGDEC_BSDMA MTK_M4U_ID(M4U_LARB1_ID, 12)
+#define M4U_PORT_VENC_NBM_WDMA MTK_M4U_ID(M4U_LARB1_ID, 13)
+#define M4U_PORT_VENC_NBM_WDMA_LITE MTK_M4U_ID(M4U_LARB1_ID, 14)
+#define M4U_PORT_VENC_CUR_LUMA MTK_M4U_ID(M4U_LARB1_ID, 15)
+#define M4U_PORT_VENC_CUR_CHROMA MTK_M4U_ID(M4U_LARB1_ID, 16)
+#define M4U_PORT_VENC_REF_LUMA MTK_M4U_ID(M4U_LARB1_ID, 17)
+#define M4U_PORT_VENC_REF_CHROMA MTK_M4U_ID(M4U_LARB1_ID, 18)
+
+/* larb2 */
+#define M4U_PORT_CAM_IMGO MTK_M4U_ID(M4U_LARB2_ID, 0)
+#define M4U_PORT_CAM_RRZO MTK_M4U_ID(M4U_LARB2_ID, 1)
+#define M4U_PORT_CAM_AAO MTK_M4U_ID(M4U_LARB2_ID, 2)
+#define M4U_PORT_CAM_LCS MTK_M4U_ID(M4U_LARB2_ID, 3)
+#define M4U_PORT_CAM_ESFKO MTK_M4U_ID(M4U_LARB2_ID, 4)
+#define M4U_PORT_CAM_CAM_SV0 MTK_M4U_ID(M4U_LARB2_ID, 5)
+#define M4U_PORT_CAM_CAM_SV1 MTK_M4U_ID(M4U_LARB2_ID, 6)
+#define M4U_PORT_CAM_LSCI MTK_M4U_ID(M4U_LARB2_ID, 7)
+#define M4U_PORT_CAM_LSCI_D MTK_M4U_ID(M4U_LARB2_ID, 8)
+#define M4U_PORT_CAM_AFO MTK_M4U_ID(M4U_LARB2_ID, 9)
+#define M4U_PORT_CAM_SPARE MTK_M4U_ID(M4U_LARB2_ID, 10)
+#define M4U_PORT_CAM_BPCI MTK_M4U_ID(M4U_LARB2_ID, 11)
+#define M4U_PORT_CAM_BPCI_D MTK_M4U_ID(M4U_LARB2_ID, 12)
+#define M4U_PORT_CAM_UFDI MTK_M4U_ID(M4U_LARB2_ID, 13)
+#define M4U_PORT_CAM_IMGI MTK_M4U_ID(M4U_LARB2_ID, 14)
+#define M4U_PORT_CAM_IMG2O MTK_M4U_ID(M4U_LARB2_ID, 15)
+#define M4U_PORT_CAM_IMG3O MTK_M4U_ID(M4U_LARB2_ID, 16)
+#define M4U_PORT_CAM_WPE0_I MTK_M4U_ID(M4U_LARB2_ID, 17)
+#define M4U_PORT_CAM_WPE1_I MTK_M4U_ID(M4U_LARB2_ID, 18)
+#define M4U_PORT_CAM_WPE_O MTK_M4U_ID(M4U_LARB2_ID, 19)
+#define M4U_PORT_CAM_FD0_I MTK_M4U_ID(M4U_LARB2_ID, 20)
+#define M4U_PORT_CAM_FD1_I MTK_M4U_ID(M4U_LARB2_ID, 21)
+#define M4U_PORT_CAM_FD0_O MTK_M4U_ID(M4U_LARB2_ID, 22)
+#define M4U_PORT_CAM_FD1_O MTK_M4U_ID(M4U_LARB2_ID, 23)
+
+/* larb3 */
+#define M4U_PORT_HW_VDEC_MC_EXT MTK_M4U_ID(M4U_LARB3_ID, 0)
+#define M4U_PORT_HW_VDEC_UFO_EXT MTK_M4U_ID(M4U_LARB3_ID, 1)
+#define M4U_PORT_HW_VDEC_PP_EXT MTK_M4U_ID(M4U_LARB3_ID, 2)
+#define M4U_PORT_HW_VDEC_PRED_RD_EXT MTK_M4U_ID(M4U_LARB3_ID, 3)
+#define M4U_PORT_HW_VDEC_PRED_WR_EXT MTK_M4U_ID(M4U_LARB3_ID, 4)
+#define M4U_PORT_HW_VDEC_PPWRAP_EXT MTK_M4U_ID(M4U_LARB3_ID, 5)
+#define M4U_PORT_HW_VDEC_TILE_EXT MTK_M4U_ID(M4U_LARB3_ID, 6)
+#define M4U_PORT_HW_VDEC_VLD_EXT MTK_M4U_ID(M4U_LARB3_ID, 7)
+#define M4U_PORT_HW_VDEC_VLD2_EXT MTK_M4U_ID(M4U_LARB3_ID, 8)
+#define M4U_PORT_HW_VDEC_AVC_MV_EXT MTK_M4U_ID(M4U_LARB3_ID, 9)
+#define M4U_PORT_HW_VDEC_RG_CTRL_DMA_EXT MTK_M4U_ID(M4U_LARB3_ID, 10)
+
+#endif
diff --git a/include/dt-bindings/phy/phy-qcom-qmp.h b/include/dt-bindings/phy/phy-qcom-qmp.h
new file mode 100644
index 0000000..4edec4c
--- /dev/null
+++ b/include/dt-bindings/phy/phy-qcom-qmp.h
@@ -0,0 +1,20 @@
+/* SPDX-License-Identifier: (GPL-2.0 OR BSD-3-Clause) */
+/*
+ * Qualcomm QMP PHY constants
+ *
+ * Copyright (C) 2022 Linaro Limited
+ */
+
+#ifndef _DT_BINDINGS_PHY_QMP
+#define _DT_BINDINGS_PHY_QMP
+
+/* QMP USB4-USB3-DP clocks */
+#define QMP_USB43DP_USB3_PIPE_CLK 0
+#define QMP_USB43DP_DP_LINK_CLK 1
+#define QMP_USB43DP_DP_VCO_DIV_CLK 2
+
+/* QMP USB4-USB3-DP PHYs */
+#define QMP_USB43DP_USB3_PHY 0
+#define QMP_USB43DP_DP_PHY 1
+
+#endif /* _DT_BINDINGS_PHY_QMP */
diff --git a/include/dt-bindings/reset/mt8188-resets.h b/include/dt-bindings/reset/mt8188-resets.h
new file mode 100644
index 0000000..377cdfd
--- /dev/null
+++ b/include/dt-bindings/reset/mt8188-resets.h
@@ -0,0 +1,36 @@
+/* SPDX-License-Identifier: (GPL-2.0+ OR BSD-3-Clause)*/
+/*
+ * Copyright (c) 2022 MediaTek Inc.
+ * Author: Runyang Chen <runyang.chen@mediatek.com>
+ */
+
+#ifndef _DT_BINDINGS_RESET_CONTROLLER_MT8188
+#define _DT_BINDINGS_RESET_CONTROLLER_MT8188
+
+#define MT8188_TOPRGU_CONN_MCU_SW_RST 0
+#define MT8188_TOPRGU_INFRA_GRST_SW_RST 1
+#define MT8188_TOPRGU_IPU0_SW_RST 2
+#define MT8188_TOPRGU_IPU1_SW_RST 3
+#define MT8188_TOPRGU_IPU2_SW_RST 4
+#define MT8188_TOPRGU_AUD_ASRC_SW_RST 5
+#define MT8188_TOPRGU_INFRA_SW_RST 6
+#define MT8188_TOPRGU_MMSYS_SW_RST 7
+#define MT8188_TOPRGU_MFG_SW_RST 8
+#define MT8188_TOPRGU_VENC_SW_RST 9
+#define MT8188_TOPRGU_VDEC_SW_RST 10
+#define MT8188_TOPRGU_CAM_VCORE_SW_RST 11
+#define MT8188_TOPRGU_SCP_SW_RST 12
+#define MT8188_TOPRGU_APMIXEDSYS_SW_RST 13
+#define MT8188_TOPRGU_AUDIO_SW_RST 14
+#define MT8188_TOPRGU_CAMSYS_SW_RST 15
+#define MT8188_TOPRGU_MJC_SW_RST 16
+#define MT8188_TOPRGU_PERI_SW_RST 17
+#define MT8188_TOPRGU_PERI_AO_SW_RST 18
+#define MT8188_TOPRGU_PCIE_SW_RST 19
+#define MT8188_TOPRGU_ADSPSYS_SW_RST 21
+#define MT8188_TOPRGU_DPTX_SW_RST 22
+#define MT8188_TOPRGU_SPMI_MST_SW_RST 23
+
+#define MT8188_TOPRGU_SW_RST_NUM 24
+
+#endif /* _DT_BINDINGS_RESET_CONTROLLER_MT8188 */
diff --git a/include/linux/filter.h b/include/linux/filter.h
index bf70197..ccc4a4a 100644
--- a/include/linux/filter.h
+++ b/include/linux/filter.h
@@ -860,8 +860,7 @@ static inline void bpf_prog_lock_ro(struct bpf_prog *fp)
static inline void bpf_jit_binary_lock_ro(struct bpf_binary_header *hdr)
{
set_vm_flush_reset_perms(hdr);
- set_memory_ro((unsigned long)hdr, hdr->size >> PAGE_SHIFT);
- set_memory_x((unsigned long)hdr, hdr->size >> PAGE_SHIFT);
+ set_memory_rox((unsigned long)hdr, hdr->size >> PAGE_SHIFT);
}
int sk_filter_trim_cap(struct sock *sk, struct sk_buff *skb, unsigned int cap);
diff --git a/include/linux/io-pgtable.h b/include/linux/io-pgtable.h
index 1f068df..1b7a44b 100644
--- a/include/linux/io-pgtable.h
+++ b/include/linux/io-pgtable.h
@@ -150,9 +150,7 @@ struct io_pgtable_cfg {
/**
* struct io_pgtable_ops - Page table manipulation API for IOMMU drivers.
*
- * @map: Map a physically contiguous memory region.
* @map_pages: Map a physically contiguous range of pages of the same size.
- * @unmap: Unmap a physically contiguous memory region.
* @unmap_pages: Unmap a range of virtually contiguous pages of the same size.
* @iova_to_phys: Translate iova to physical address.
*
@@ -160,13 +158,9 @@ struct io_pgtable_cfg {
* the same names.
*/
struct io_pgtable_ops {
- int (*map)(struct io_pgtable_ops *ops, unsigned long iova,
- phys_addr_t paddr, size_t size, int prot, gfp_t gfp);
int (*map_pages)(struct io_pgtable_ops *ops, unsigned long iova,
phys_addr_t paddr, size_t pgsize, size_t pgcount,
int prot, gfp_t gfp, size_t *mapped);
- size_t (*unmap)(struct io_pgtable_ops *ops, unsigned long iova,
- size_t size, struct iommu_iotlb_gather *gather);
size_t (*unmap_pages)(struct io_pgtable_ops *ops, unsigned long iova,
size_t pgsize, size_t pgcount,
struct iommu_iotlb_gather *gather);
diff --git a/include/linux/mfd/rn5t618.h b/include/linux/mfd/rn5t618.h
index 8aa0bda..aacb6d5 100644
--- a/include/linux/mfd/rn5t618.h
+++ b/include/linux/mfd/rn5t618.h
@@ -227,6 +227,15 @@
#define RN5T618_WATCHDOG_WDOGTIM_S 0
#define RN5T618_PWRIRQ_IR_WDOG BIT(6)
+#define RN5T618_POFFHIS_PWRON BIT(0)
+#define RN5T618_POFFHIS_TSHUT BIT(1)
+#define RN5T618_POFFHIS_VINDET BIT(2)
+#define RN5T618_POFFHIS_IODET BIT(3)
+#define RN5T618_POFFHIS_CPU BIT(4)
+#define RN5T618_POFFHIS_WDG BIT(5)
+#define RN5T618_POFFHIS_DCLIM BIT(6)
+#define RN5T618_POFFHIS_N_OE BIT(7)
+
enum {
RN5T618_DCDC1,
RN5T618_DCDC2,
diff --git a/include/linux/moduleloader.h b/include/linux/moduleloader.h
index 9e09d11..7b4587a 100644
--- a/include/linux/moduleloader.h
+++ b/include/linux/moduleloader.h
@@ -13,6 +13,9 @@
* must be implemented by each architecture.
*/
+/* arch may override to do additional checking of ELF header architecture */
+bool module_elf_check_arch(Elf_Ehdr *hdr);
+
/* Adjust arch-specific sections. Return 0 on success. */
int module_frob_arch_sections(Elf_Ehdr *hdr,
Elf_Shdr *sechdrs,
diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h
index dfabd54..1159b25 100644
--- a/include/linux/pgtable.h
+++ b/include/linux/pgtable.h
@@ -309,24 +309,28 @@ static inline void ptep_clear(struct mm_struct *mm, unsigned long addr,
ptep_get_and_clear(mm, addr, ptep);
}
-#ifndef __HAVE_ARCH_PTEP_GET
+#ifndef ptep_get
static inline pte_t ptep_get(pte_t *ptep)
{
return READ_ONCE(*ptep);
}
#endif
-#ifdef CONFIG_GUP_GET_PTE_LOW_HIGH
+#ifndef pmdp_get
+static inline pmd_t pmdp_get(pmd_t *pmdp)
+{
+ return READ_ONCE(*pmdp);
+}
+#endif
+
+#ifdef CONFIG_GUP_GET_PXX_LOW_HIGH
/*
- * WARNING: only to be used in the get_user_pages_fast() implementation.
- *
- * With get_user_pages_fast(), we walk down the pagetables without taking any
- * locks. For this we would like to load the pointers atomically, but sometimes
- * that is not possible (e.g. without expensive cmpxchg8b on x86_32 PAE). What
- * we do have is the guarantee that a PTE will only either go from not present
- * to present, or present to not present or both -- it will not switch to a
- * completely different present page without a TLB flush in between; something
- * that we are blocking by holding interrupts off.
+ * For walking the pagetables without holding any locks. Some architectures
+ * (eg x86-32 PAE) cannot load the entries atomically without using expensive
+ * instructions. We are guaranteed that a PTE will only either go from not
+ * present to present, or present to not present -- it will not switch to a
+ * completely different present page without a TLB flush inbetween; which we
+ * are blocking by holding interrupts off.
*
* Setting ptes from not present to present goes:
*
@@ -361,15 +365,42 @@ static inline pte_t ptep_get_lockless(pte_t *ptep)
return pte;
}
-#else /* CONFIG_GUP_GET_PTE_LOW_HIGH */
+#define ptep_get_lockless ptep_get_lockless
+
+#if CONFIG_PGTABLE_LEVELS > 2
+static inline pmd_t pmdp_get_lockless(pmd_t *pmdp)
+{
+ pmd_t pmd;
+
+ do {
+ pmd.pmd_low = pmdp->pmd_low;
+ smp_rmb();
+ pmd.pmd_high = pmdp->pmd_high;
+ smp_rmb();
+ } while (unlikely(pmd.pmd_low != pmdp->pmd_low));
+
+ return pmd;
+}
+#define pmdp_get_lockless pmdp_get_lockless
+#endif /* CONFIG_PGTABLE_LEVELS > 2 */
+#endif /* CONFIG_GUP_GET_PXX_LOW_HIGH */
+
/*
* We require that the PTE can be read atomically.
*/
+#ifndef ptep_get_lockless
static inline pte_t ptep_get_lockless(pte_t *ptep)
{
return ptep_get(ptep);
}
-#endif /* CONFIG_GUP_GET_PTE_LOW_HIGH */
+#endif
+
+#ifndef pmdp_get_lockless
+static inline pmd_t pmdp_get_lockless(pmd_t *pmdp)
+{
+ return pmdp_get(pmdp);
+}
+#endif
#ifdef CONFIG_TRANSPARENT_HUGEPAGE
#ifndef __HAVE_ARCH_PMDP_HUGE_GET_AND_CLEAR
@@ -1313,18 +1344,6 @@ static inline int pud_trans_unstable(pud_t *pud)
#endif
}
-#ifndef pmd_read_atomic
-static inline pmd_t pmd_read_atomic(pmd_t *pmdp)
-{
- /*
- * Depend on compiler for an atomic pmd read. NOTE: this is
- * only going to work, if the pmdval_t isn't larger than
- * an unsigned long.
- */
- return *pmdp;
-}
-#endif
-
#ifndef arch_needs_pgtable_deposit
#define arch_needs_pgtable_deposit() (false)
#endif
@@ -1351,13 +1370,13 @@ static inline pmd_t pmd_read_atomic(pmd_t *pmdp)
*/
static inline int pmd_none_or_trans_huge_or_clear_bad(pmd_t *pmd)
{
- pmd_t pmdval = pmd_read_atomic(pmd);
+ pmd_t pmdval = pmdp_get_lockless(pmd);
/*
* The barrier will stabilize the pmdval in a register or on
* the stack so that it will stop changing under the code.
*
* When CONFIG_TRANSPARENT_HUGEPAGE=y on x86 32bit PAE,
- * pmd_read_atomic is allowed to return a not atomic pmdval
+ * pmdp_get_lockless is allowed to return a not atomic pmdval
* (for example pointing to an hugepage that has never been
* mapped in the pmd). The below checks will only care about
* the low part of the pmd with 32bit PAE x86 anyway, with the
diff --git a/include/linux/sched/task.h b/include/linux/sched/task.h
index d6c4816..357e006 100644
--- a/include/linux/sched/task.h
+++ b/include/linux/sched/task.h
@@ -65,6 +65,7 @@ extern void sched_dead(struct task_struct *p);
void __noreturn do_task_dead(void);
void __noreturn make_task_dead(int signr);
+extern void mm_cache_init(void);
extern void proc_caches_init(void);
extern void fork_init(void);
@@ -90,7 +91,6 @@ extern void exit_itimers(struct task_struct *);
extern pid_t kernel_clone(struct kernel_clone_args *kargs);
struct task_struct *create_io_thread(int (*fn)(void *), void *arg, int node);
struct task_struct *fork_idle(int);
-struct mm_struct *copy_init_mm(void);
extern pid_t kernel_thread(int (*fn)(void *), void *arg, unsigned long flags);
extern pid_t user_mode_thread(int (*fn)(void *), void *arg, unsigned long flags);
extern long kernel_wait4(pid_t, int __user *, int, struct rusage *);
diff --git a/include/linux/set_memory.h b/include/linux/set_memory.h
index 369769c..95ac839 100644
--- a/include/linux/set_memory.h
+++ b/include/linux/set_memory.h
@@ -14,6 +14,16 @@ static inline int set_memory_x(unsigned long addr, int numpages) { return 0; }
static inline int set_memory_nx(unsigned long addr, int numpages) { return 0; }
#endif
+#ifndef set_memory_rox
+static inline int set_memory_rox(unsigned long addr, int numpages)
+{
+ int ret = set_memory_ro(addr, numpages);
+ if (ret)
+ return ret;
+ return set_memory_x(addr, numpages);
+}
+#endif
+
#ifndef CONFIG_ARCH_HAS_SET_DIRECT_MAP
static inline int set_direct_map_invalid_noflush(struct page *page)
{
diff --git a/include/linux/soundwire/sdw_intel.h b/include/linux/soundwire/sdw_intel.h
index d2f581f..91f0dc5 100644
--- a/include/linux/soundwire/sdw_intel.h
+++ b/include/linux/soundwire/sdw_intel.h
@@ -233,6 +233,7 @@ struct sdw_intel_ctx {
* struct sdw_intel_res - Soundwire Intel global resource structure,
* typically populated by the DSP driver
*
+ * @hw_ops: abstraction for platform ops
* @count: link count
* @mmio_base: mmio base of SoundWire registers
* @irq: interrupt number
@@ -249,6 +250,7 @@ struct sdw_intel_ctx {
* @alh_base: sdw alh base.
*/
struct sdw_intel_res {
+ const struct sdw_intel_hw_ops *hw_ops;
int count;
void __iomem *mmio_base;
int irq;
@@ -290,4 +292,46 @@ irqreturn_t sdw_intel_thread(int irq, void *dev_id);
#define SDW_INTEL_QUIRK_MASK_BUS_DISABLE BIT(1)
+struct sdw_intel;
+
+/* struct intel_sdw_hw_ops - SoundWire ops for Intel platforms.
+ * @debugfs_init: initialize all debugfs capabilities
+ * @debugfs_exit: close and cleanup debugfs capabilities
+ * @register_dai: read all PDI information and register DAIs
+ * @check_clock_stop: throw error message if clock is not stopped.
+ * @start_bus: normal start
+ * @start_bus_after_reset: start after reset
+ * @start_bus_after_clock_stop: start after mode0 clock stop
+ * @stop_bus: stop all bus
+ * @link_power_up: power-up using chip-specific helpers
+ * @link_power_down: power-down with chip-specific helpers
+ * @shim_check_wake: check if a wake was received
+ * @shim_wake: enable/disable in-band wake management
+ * @pre_bank_switch: helper for bus management
+ * @post_bank_switch: helper for bus management
+ */
+struct sdw_intel_hw_ops {
+ void (*debugfs_init)(struct sdw_intel *sdw);
+ void (*debugfs_exit)(struct sdw_intel *sdw);
+
+ int (*register_dai)(struct sdw_intel *sdw);
+
+ void (*check_clock_stop)(struct sdw_intel *sdw);
+ int (*start_bus)(struct sdw_intel *sdw);
+ int (*start_bus_after_reset)(struct sdw_intel *sdw);
+ int (*start_bus_after_clock_stop)(struct sdw_intel *sdw);
+ int (*stop_bus)(struct sdw_intel *sdw, bool clock_stop);
+
+ int (*link_power_up)(struct sdw_intel *sdw);
+ int (*link_power_down)(struct sdw_intel *sdw);
+
+ int (*shim_check_wake)(struct sdw_intel *sdw);
+ void (*shim_wake)(struct sdw_intel *sdw, bool wake_enable);
+
+ int (*pre_bank_switch)(struct sdw_intel *sdw);
+ int (*post_bank_switch)(struct sdw_intel *sdw);
+};
+
+extern const struct sdw_intel_hw_ops sdw_intel_cnl_hw_ops;
+
#endif
diff --git a/include/linux/zstd_lib.h b/include/linux/zstd_lib.h
index b8c7dbf9..79d5546 100644
--- a/include/linux/zstd_lib.h
+++ b/include/linux/zstd_lib.h
@@ -17,8 +17,16 @@
/* ===== ZSTDLIB_API : control library symbols visibility ===== */
-#define ZSTDLIB_VISIBILITY
-#define ZSTDLIB_API ZSTDLIB_VISIBILITY
+#ifndef ZSTDLIB_VISIBLE
+# if (__GNUC__ >= 4) && !defined(__MINGW32__)
+# define ZSTDLIB_VISIBLE __attribute__ ((visibility ("default")))
+# define ZSTDLIB_HIDDEN __attribute__ ((visibility ("hidden")))
+# else
+# define ZSTDLIB_VISIBLE
+# define ZSTDLIB_HIDDEN
+# endif
+#endif
+#define ZSTDLIB_API ZSTDLIB_VISIBLE
/* *****************************************************************************
@@ -56,8 +64,8 @@
/*------ Version ------*/
#define ZSTD_VERSION_MAJOR 1
-#define ZSTD_VERSION_MINOR 4
-#define ZSTD_VERSION_RELEASE 10
+#define ZSTD_VERSION_MINOR 5
+#define ZSTD_VERSION_RELEASE 2
#define ZSTD_VERSION_NUMBER (ZSTD_VERSION_MAJOR *100*100 + ZSTD_VERSION_MINOR *100 + ZSTD_VERSION_RELEASE)
/*! ZSTD_versionNumber() :
@@ -94,7 +102,6 @@ ZSTDLIB_API const char* ZSTD_versionString(void);
#define ZSTD_BLOCKSIZE_MAX (1<<ZSTD_BLOCKSIZELOG_MAX)
-
/* *************************************
* Simple API
***************************************/
@@ -151,7 +158,7 @@ ZSTDLIB_API unsigned long long ZSTD_getFrameContentSize(const void *src, size_t
* @return : decompressed size of `src` frame content _if known and not empty_, 0 otherwise. */
ZSTDLIB_API unsigned long long ZSTD_getDecompressedSize(const void* src, size_t srcSize);
-/*! ZSTD_findFrameCompressedSize() :
+/*! ZSTD_findFrameCompressedSize() : Requires v1.4.0+
* `src` should point to the start of a ZSTD frame or skippable frame.
* `srcSize` must be >= first frame size
* @return : the compressed size of the first frame starting at `src`,
@@ -165,8 +172,9 @@ ZSTDLIB_API size_t ZSTD_findFrameCompressedSize(const void* src, size_t srcSize)
ZSTDLIB_API size_t ZSTD_compressBound(size_t srcSize); /*!< maximum compressed size in worst case single-pass scenario */
ZSTDLIB_API unsigned ZSTD_isError(size_t code); /*!< tells if a `size_t` function result is an error code */
ZSTDLIB_API const char* ZSTD_getErrorName(size_t code); /*!< provides readable string from an error code */
-ZSTDLIB_API int ZSTD_minCLevel(void); /*!< minimum negative compression level allowed */
+ZSTDLIB_API int ZSTD_minCLevel(void); /*!< minimum negative compression level allowed, requires v1.4.0+ */
ZSTDLIB_API int ZSTD_maxCLevel(void); /*!< maximum compression level available */
+ZSTDLIB_API int ZSTD_defaultCLevel(void); /*!< default compression level, specified by ZSTD_CLEVEL_DEFAULT, requires v1.5.0+ */
/* *************************************
@@ -219,9 +227,9 @@ ZSTDLIB_API size_t ZSTD_decompressDCtx(ZSTD_DCtx* dctx,
const void* src, size_t srcSize);
-/* *************************************
-* Advanced compression API
-***************************************/
+/* *******************************************
+* Advanced compression API (Requires v1.4.0+)
+**********************************************/
/* API design :
* Parameters are pushed one by one into an existing context,
@@ -232,7 +240,7 @@ ZSTDLIB_API size_t ZSTD_decompressDCtx(ZSTD_DCtx* dctx,
*
* It's possible to reset all parameters to "default" using ZSTD_CCtx_reset().
*
- * This API supercedes all other "advanced" API entry points in the experimental section.
+ * This API supersedes all other "advanced" API entry points in the experimental section.
* In the future, we expect to remove from experimental API entry points which are redundant with this API.
*/
@@ -251,7 +259,6 @@ typedef enum { ZSTD_fast=1,
Only the order (from fast to strong) is guaranteed */
} ZSTD_strategy;
-
typedef enum {
/* compression parameters
@@ -317,7 +324,6 @@ typedef enum {
* The higher the value of selected strategy, the more complex it is,
* resulting in stronger and slower compression.
* Special: value 0 means "use default strategy". */
-
/* LDM mode parameters */
ZSTD_c_enableLongDistanceMatching=160, /* Enable long distance matching.
* This parameter is designed to improve compression ratio
@@ -374,7 +380,7 @@ typedef enum {
ZSTD_c_jobSize=401, /* Size of a compression job. This value is enforced only when nbWorkers >= 1.
* Each compression job is completed in parallel, so this value can indirectly impact the nb of active threads.
* 0 means default, which is dynamically determined based on compression parameters.
- * Job size must be a minimum of overlap size, or 1 MB, whichever is largest.
+ * Job size must be a minimum of overlap size, or ZSTDMT_JOBSIZE_MIN (= 512 KB), whichever is largest.
* The minimum size is automatically and transparently enforced. */
ZSTD_c_overlapLog=402, /* Control the overlap size, as a fraction of window size.
* The overlap size is an amount of data reloaded from previous job at the beginning of a new job.
@@ -404,6 +410,8 @@ typedef enum {
* ZSTD_c_stableOutBuffer
* ZSTD_c_blockDelimiters
* ZSTD_c_validateSequences
+ * ZSTD_c_useBlockSplitter
+ * ZSTD_c_useRowMatchFinder
* Because they are not stable, it's necessary to define ZSTD_STATIC_LINKING_ONLY to access them.
* note : never ever use experimentalParam? names directly;
* also, the enums values themselves are unstable and can still change.
@@ -419,7 +427,10 @@ typedef enum {
ZSTD_c_experimentalParam9=1006,
ZSTD_c_experimentalParam10=1007,
ZSTD_c_experimentalParam11=1008,
- ZSTD_c_experimentalParam12=1009
+ ZSTD_c_experimentalParam12=1009,
+ ZSTD_c_experimentalParam13=1010,
+ ZSTD_c_experimentalParam14=1011,
+ ZSTD_c_experimentalParam15=1012
} ZSTD_cParameter;
typedef struct {
@@ -504,9 +515,9 @@ ZSTDLIB_API size_t ZSTD_compress2( ZSTD_CCtx* cctx,
const void* src, size_t srcSize);
-/* *************************************
-* Advanced decompression API
-***************************************/
+/* *********************************************
+* Advanced decompression API (Requires v1.4.0+)
+************************************************/
/* The advanced API pushes parameters one by one into an existing DCtx context.
* Parameters are sticky, and remain valid for all following frames
@@ -668,7 +679,7 @@ typedef enum {
: note : multithreaded compression will block to flush as much output as possible. */
} ZSTD_EndDirective;
-/*! ZSTD_compressStream2() :
+/*! ZSTD_compressStream2() : Requires v1.4.0+
* Behaves about the same as ZSTD_compressStream, with additional control on end directive.
* - Compression parameters are pushed into CCtx before starting compression, using ZSTD_CCtx_set*()
* - Compression parameters cannot be changed once compression is started (save a list of exceptions in multi-threading mode)
@@ -714,11 +725,11 @@ ZSTDLIB_API size_t ZSTD_CStreamOutSize(void); /*< recommended size for output
/* *****************************************************************************
- * This following is a legacy streaming API.
+ * This following is a legacy streaming API, available since v1.0+ .
* It can be replaced by ZSTD_CCtx_reset() and ZSTD_compressStream2().
* It is redundant, but remains fully supported.
- * Advanced parameters and dictionary compression can only be used through the
- * new API.
+ * Streaming in combination with advanced parameters and dictionary compression
+ * can only be used through the new API.
******************************************************************************/
/*!
@@ -796,7 +807,7 @@ ZSTDLIB_API size_t ZSTD_DStreamOutSize(void); /*!< recommended size for output
/*! ZSTD_compress_usingDict() :
* Compression at an explicit compression level using a Dictionary.
* A dictionary can be any arbitrary data segment (also called a prefix),
- * or a buffer with specified information (see dictBuilder/zdict.h).
+ * or a buffer with specified information (see zdict.h).
* Note : This function loads the dictionary, resulting in significant startup delay.
* It's intended for a dictionary used only once.
* Note 2 : When `dict == NULL || dictSize < 8` no dictionary is used. */
@@ -879,19 +890,25 @@ ZSTDLIB_API size_t ZSTD_decompress_usingDDict(ZSTD_DCtx* dctx,
* Dictionary helper functions
*******************************/
-/*! ZSTD_getDictID_fromDict() :
+/*! ZSTD_getDictID_fromDict() : Requires v1.4.0+
* Provides the dictID stored within dictionary.
* if @return == 0, the dictionary is not conformant with Zstandard specification.
* It can still be loaded, but as a content-only dictionary. */
ZSTDLIB_API unsigned ZSTD_getDictID_fromDict(const void* dict, size_t dictSize);
-/*! ZSTD_getDictID_fromDDict() :
+/*! ZSTD_getDictID_fromCDict() : Requires v1.5.0+
+ * Provides the dictID of the dictionary loaded into `cdict`.
+ * If @return == 0, the dictionary is not conformant to Zstandard specification, or empty.
+ * Non-conformant dictionaries can still be loaded, but as content-only dictionaries. */
+ZSTDLIB_API unsigned ZSTD_getDictID_fromCDict(const ZSTD_CDict* cdict);
+
+/*! ZSTD_getDictID_fromDDict() : Requires v1.4.0+
* Provides the dictID of the dictionary loaded into `ddict`.
* If @return == 0, the dictionary is not conformant to Zstandard specification, or empty.
* Non-conformant dictionaries can still be loaded, but as content-only dictionaries. */
ZSTDLIB_API unsigned ZSTD_getDictID_fromDDict(const ZSTD_DDict* ddict);
-/*! ZSTD_getDictID_fromFrame() :
+/*! ZSTD_getDictID_fromFrame() : Requires v1.4.0+
* Provides the dictID required to decompressed the frame stored within `src`.
* If @return == 0, the dictID could not be decoded.
* This could for one of the following reasons :
@@ -905,16 +922,16 @@ ZSTDLIB_API unsigned ZSTD_getDictID_fromFrame(const void* src, size_t srcSize);
/* *****************************************************************************
- * Advanced dictionary and prefix API
+ * Advanced dictionary and prefix API (Requires v1.4.0+)
*
* This API allows dictionaries to be used with ZSTD_compress2(),
- * ZSTD_compressStream2(), and ZSTD_decompress(). Dictionaries are sticky, and
+ * ZSTD_compressStream2(), and ZSTD_decompressDCtx(). Dictionaries are sticky, and
* only reset with the context is reset with ZSTD_reset_parameters or
* ZSTD_reset_session_and_parameters. Prefixes are single-use.
******************************************************************************/
-/*! ZSTD_CCtx_loadDictionary() :
+/*! ZSTD_CCtx_loadDictionary() : Requires v1.4.0+
* Create an internal CDict from `dict` buffer.
* Decompression will have to use same dictionary.
* @result : 0, or an error code (which can be tested with ZSTD_isError()).
@@ -933,7 +950,7 @@ ZSTDLIB_API unsigned ZSTD_getDictID_fromFrame(const void* src, size_t srcSize);
* to precisely select how dictionary content must be interpreted. */
ZSTDLIB_API size_t ZSTD_CCtx_loadDictionary(ZSTD_CCtx* cctx, const void* dict, size_t dictSize);
-/*! ZSTD_CCtx_refCDict() :
+/*! ZSTD_CCtx_refCDict() : Requires v1.4.0+
* Reference a prepared dictionary, to be used for all next compressed frames.
* Note that compression parameters are enforced from within CDict,
* and supersede any compression parameter previously set within CCtx.
@@ -947,7 +964,7 @@ ZSTDLIB_API size_t ZSTD_CCtx_loadDictionary(ZSTD_CCtx* cctx, const void* dict, s
* Note 2 : CDict is just referenced, its lifetime must outlive its usage within CCtx. */
ZSTDLIB_API size_t ZSTD_CCtx_refCDict(ZSTD_CCtx* cctx, const ZSTD_CDict* cdict);
-/*! ZSTD_CCtx_refPrefix() :
+/*! ZSTD_CCtx_refPrefix() : Requires v1.4.0+
* Reference a prefix (single-usage dictionary) for next compressed frame.
* A prefix is **only used once**. Tables are discarded at end of frame (ZSTD_e_end).
* Decompression will need same prefix to properly regenerate data.
@@ -968,7 +985,7 @@ ZSTDLIB_API size_t ZSTD_CCtx_refCDict(ZSTD_CCtx* cctx, const ZSTD_CDict* cdict);
ZSTDLIB_API size_t ZSTD_CCtx_refPrefix(ZSTD_CCtx* cctx,
const void* prefix, size_t prefixSize);
-/*! ZSTD_DCtx_loadDictionary() :
+/*! ZSTD_DCtx_loadDictionary() : Requires v1.4.0+
* Create an internal DDict from dict buffer,
* to be used to decompress next frames.
* The dictionary remains valid for all future frames, until explicitly invalidated.
@@ -985,7 +1002,7 @@ ZSTDLIB_API size_t ZSTD_CCtx_refPrefix(ZSTD_CCtx* cctx,
*/
ZSTDLIB_API size_t ZSTD_DCtx_loadDictionary(ZSTD_DCtx* dctx, const void* dict, size_t dictSize);
-/*! ZSTD_DCtx_refDDict() :
+/*! ZSTD_DCtx_refDDict() : Requires v1.4.0+
* Reference a prepared dictionary, to be used to decompress next frames.
* The dictionary remains active for decompression of future frames using same DCtx.
*
@@ -1003,7 +1020,7 @@ ZSTDLIB_API size_t ZSTD_DCtx_loadDictionary(ZSTD_DCtx* dctx, const void* dict, s
*/
ZSTDLIB_API size_t ZSTD_DCtx_refDDict(ZSTD_DCtx* dctx, const ZSTD_DDict* ddict);
-/*! ZSTD_DCtx_refPrefix() :
+/*! ZSTD_DCtx_refPrefix() : Requires v1.4.0+
* Reference a prefix (single-usage dictionary) to decompress next frame.
* This is the reverse operation of ZSTD_CCtx_refPrefix(),
* and must use the same prefix as the one used during compression.
@@ -1024,7 +1041,7 @@ ZSTDLIB_API size_t ZSTD_DCtx_refPrefix(ZSTD_DCtx* dctx,
/* === Memory management === */
-/*! ZSTD_sizeof_*() :
+/*! ZSTD_sizeof_*() : Requires v1.4.0+
* These functions give the _current_ memory usage of selected object.
* Note that object memory usage can evolve (increase or decrease) over time. */
ZSTDLIB_API size_t ZSTD_sizeof_CCtx(const ZSTD_CCtx* cctx);
@@ -1049,6 +1066,29 @@ ZSTDLIB_API size_t ZSTD_sizeof_DDict(const ZSTD_DDict* ddict);
#if !defined(ZSTD_H_ZSTD_STATIC_LINKING_ONLY)
#define ZSTD_H_ZSTD_STATIC_LINKING_ONLY
+/* This can be overridden externally to hide static symbols. */
+#ifndef ZSTDLIB_STATIC_API
+#define ZSTDLIB_STATIC_API ZSTDLIB_VISIBLE
+#endif
+
+/* Deprecation warnings :
+ * Should these warnings be a problem, it is generally possible to disable them,
+ * typically with -Wno-deprecated-declarations for gcc or _CRT_SECURE_NO_WARNINGS in Visual.
+ * Otherwise, it's also possible to define ZSTD_DISABLE_DEPRECATE_WARNINGS.
+ */
+#ifdef ZSTD_DISABLE_DEPRECATE_WARNINGS
+# define ZSTD_DEPRECATED(message) ZSTDLIB_STATIC_API /* disable deprecation warnings */
+#else
+# if (defined(GNUC) && (GNUC > 4 || (GNUC == 4 && GNUC_MINOR >= 5))) || defined(__clang__)
+# define ZSTD_DEPRECATED(message) ZSTDLIB_STATIC_API __attribute__((deprecated(message)))
+# elif (__GNUC__ >= 3)
+# define ZSTD_DEPRECATED(message) ZSTDLIB_STATIC_API __attribute__((deprecated))
+# else
+# pragma message("WARNING: You need to implement ZSTD_DEPRECATED for this compiler")
+# define ZSTD_DEPRECATED(message) ZSTDLIB_STATIC_API
+# endif
+#endif /* ZSTD_DISABLE_DEPRECATE_WARNINGS */
+
/* **************************************************************************************
* experimental API (static linking only)
****************************************************************************************
@@ -1111,9 +1151,6 @@ ZSTDLIB_API size_t ZSTD_sizeof_DDict(const ZSTD_DDict* ddict);
#define ZSTD_SRCSIZEHINT_MIN 0
#define ZSTD_SRCSIZEHINT_MAX INT_MAX
-/* internal */
-#define ZSTD_HASHLOG3_MAX 17
-
/* --- Advanced types --- */
@@ -1255,6 +1292,15 @@ typedef enum {
ZSTD_lcm_uncompressed = 2 /*< Always emit uncompressed literals. */
} ZSTD_literalCompressionMode_e;
+typedef enum {
+ /* Note: This enum controls features which are conditionally beneficial. Zstd typically will make a final
+ * decision on whether or not to enable the feature (ZSTD_ps_auto), but setting the switch to ZSTD_ps_enable
+ * or ZSTD_ps_disable allow for a force enable/disable the feature.
+ */
+ ZSTD_ps_auto = 0, /* Let the library automatically determine whether the feature shall be enabled */
+ ZSTD_ps_enable = 1, /* Force-enable the feature */
+ ZSTD_ps_disable = 2 /* Do not use the feature */
+} ZSTD_paramSwitch_e;
/* *************************************
* Frame size functions
@@ -1281,7 +1327,7 @@ typedef enum {
* note 5 : ZSTD_findDecompressedSize handles multiple frames, and so it must traverse the input to
* read each contained frame header. This is fast as most of the data is skipped,
* however it does mean that all frame data must be present and valid. */
-ZSTDLIB_API unsigned long long ZSTD_findDecompressedSize(const void* src, size_t srcSize);
+ZSTDLIB_STATIC_API unsigned long long ZSTD_findDecompressedSize(const void* src, size_t srcSize);
/*! ZSTD_decompressBound() :
* `src` should point to the start of a series of ZSTD encoded and/or skippable frames
@@ -1296,13 +1342,13 @@ ZSTDLIB_API unsigned long long ZSTD_findDecompressedSize(const void* src, size_t
* note 3 : when the decompressed size field isn't available, the upper-bound for that frame is calculated by:
* upper-bound = # blocks * min(128 KB, Window_Size)
*/
-ZSTDLIB_API unsigned long long ZSTD_decompressBound(const void* src, size_t srcSize);
+ZSTDLIB_STATIC_API unsigned long long ZSTD_decompressBound(const void* src, size_t srcSize);
/*! ZSTD_frameHeaderSize() :
* srcSize must be >= ZSTD_FRAMEHEADERSIZE_PREFIX.
* @return : size of the Frame Header,
* or an error code (if srcSize is too small) */
-ZSTDLIB_API size_t ZSTD_frameHeaderSize(const void* src, size_t srcSize);
+ZSTDLIB_STATIC_API size_t ZSTD_frameHeaderSize(const void* src, size_t srcSize);
typedef enum {
ZSTD_sf_noBlockDelimiters = 0, /* Representation of ZSTD_Sequence has no block delimiters, sequences only */
@@ -1325,12 +1371,12 @@ typedef enum {
* @return : number of sequences generated
*/
-ZSTDLIB_API size_t ZSTD_generateSequences(ZSTD_CCtx* zc, ZSTD_Sequence* outSeqs,
+ZSTDLIB_STATIC_API size_t ZSTD_generateSequences(ZSTD_CCtx* zc, ZSTD_Sequence* outSeqs,
size_t outSeqsSize, const void* src, size_t srcSize);
/*! ZSTD_mergeBlockDelimiters() :
* Given an array of ZSTD_Sequence, remove all sequences that represent block delimiters/last literals
- * by merging them into into the literals of the next sequence.
+ * by merging them into the literals of the next sequence.
*
* As such, the final generated result has no explicit representation of block boundaries,
* and the final last literals segment is not represented in the sequences.
@@ -1339,7 +1385,7 @@ ZSTDLIB_API size_t ZSTD_generateSequences(ZSTD_CCtx* zc, ZSTD_Sequence* outSeqs,
* setting of ZSTD_c_blockDelimiters as ZSTD_sf_noBlockDelimiters
* @return : number of sequences left after merging
*/
-ZSTDLIB_API size_t ZSTD_mergeBlockDelimiters(ZSTD_Sequence* sequences, size_t seqsSize);
+ZSTDLIB_STATIC_API size_t ZSTD_mergeBlockDelimiters(ZSTD_Sequence* sequences, size_t seqsSize);
/*! ZSTD_compressSequences() :
* Compress an array of ZSTD_Sequence, generated from the original source buffer, into dst.
@@ -1369,7 +1415,7 @@ ZSTDLIB_API size_t ZSTD_mergeBlockDelimiters(ZSTD_Sequence* sequences, size_t se
* and cannot emit an RLE block that disagrees with the repcode history
* @return : final compressed size or a ZSTD error.
*/
-ZSTDLIB_API size_t ZSTD_compressSequences(ZSTD_CCtx* const cctx, void* dst, size_t dstSize,
+ZSTDLIB_STATIC_API size_t ZSTD_compressSequences(ZSTD_CCtx* const cctx, void* dst, size_t dstSize,
const ZSTD_Sequence* inSeqs, size_t inSeqsSize,
const void* src, size_t srcSize);
@@ -1377,7 +1423,7 @@ ZSTDLIB_API size_t ZSTD_compressSequences(ZSTD_CCtx* const cctx, void* dst, size
/*! ZSTD_writeSkippableFrame() :
* Generates a zstd skippable frame containing data given by src, and writes it to dst buffer.
*
- * Skippable frames begin with a a 4-byte magic number. There are 16 possible choices of magic number,
+ * Skippable frames begin with a 4-byte magic number. There are 16 possible choices of magic number,
* ranging from ZSTD_MAGIC_SKIPPABLE_START to ZSTD_MAGIC_SKIPPABLE_START+15.
* As such, the parameter magicVariant controls the exact skippable frame magic number variant used, so
* the magic number used will be ZSTD_MAGIC_SKIPPABLE_START + magicVariant.
@@ -1387,9 +1433,29 @@ ZSTDLIB_API size_t ZSTD_compressSequences(ZSTD_CCtx* const cctx, void* dst, size
*
* @return : number of bytes written or a ZSTD error.
*/
-ZSTDLIB_API size_t ZSTD_writeSkippableFrame(void* dst, size_t dstCapacity,
+ZSTDLIB_STATIC_API size_t ZSTD_writeSkippableFrame(void* dst, size_t dstCapacity,
const void* src, size_t srcSize, unsigned magicVariant);
+/*! ZSTD_readSkippableFrame() :
+ * Retrieves a zstd skippable frame containing data given by src, and writes it to dst buffer.
+ *
+ * The parameter magicVariant will receive the magicVariant that was supplied when the frame was written,
+ * i.e. magicNumber - ZSTD_MAGIC_SKIPPABLE_START. This can be NULL if the caller is not interested
+ * in the magicVariant.
+ *
+ * Returns an error if destination buffer is not large enough, or if the frame is not skippable.
+ *
+ * @return : number of bytes written or a ZSTD error.
+ */
+ZSTDLIB_API size_t ZSTD_readSkippableFrame(void* dst, size_t dstCapacity, unsigned* magicVariant,
+ const void* src, size_t srcSize);
+
+/*! ZSTD_isSkippableFrame() :
+ * Tells if the content of `buffer` starts with a valid Frame Identifier for a skippable frame.
+ */
+ZSTDLIB_API unsigned ZSTD_isSkippableFrame(const void* buffer, size_t size);
+
+
/* *************************************
* Memory management
@@ -1418,10 +1484,10 @@ ZSTDLIB_API size_t ZSTD_writeSkippableFrame(void* dst, size_t dstCapacity,
* Note 2 : only single-threaded compression is supported.
* ZSTD_estimateCCtxSize_usingCCtxParams() will return an error code if ZSTD_c_nbWorkers is >= 1.
*/
-ZSTDLIB_API size_t ZSTD_estimateCCtxSize(int compressionLevel);
-ZSTDLIB_API size_t ZSTD_estimateCCtxSize_usingCParams(ZSTD_compressionParameters cParams);
-ZSTDLIB_API size_t ZSTD_estimateCCtxSize_usingCCtxParams(const ZSTD_CCtx_params* params);
-ZSTDLIB_API size_t ZSTD_estimateDCtxSize(void);
+ZSTDLIB_STATIC_API size_t ZSTD_estimateCCtxSize(int compressionLevel);
+ZSTDLIB_STATIC_API size_t ZSTD_estimateCCtxSize_usingCParams(ZSTD_compressionParameters cParams);
+ZSTDLIB_STATIC_API size_t ZSTD_estimateCCtxSize_usingCCtxParams(const ZSTD_CCtx_params* params);
+ZSTDLIB_STATIC_API size_t ZSTD_estimateDCtxSize(void);
/*! ZSTD_estimateCStreamSize() :
* ZSTD_estimateCStreamSize() will provide a budget large enough for any compression level up to selected one.
@@ -1436,20 +1502,20 @@ ZSTDLIB_API size_t ZSTD_estimateDCtxSize(void);
* Note : if streaming is init with function ZSTD_init?Stream_usingDict(),
* an internal ?Dict will be created, which additional size is not estimated here.
* In this case, get total size by adding ZSTD_estimate?DictSize */
-ZSTDLIB_API size_t ZSTD_estimateCStreamSize(int compressionLevel);
-ZSTDLIB_API size_t ZSTD_estimateCStreamSize_usingCParams(ZSTD_compressionParameters cParams);
-ZSTDLIB_API size_t ZSTD_estimateCStreamSize_usingCCtxParams(const ZSTD_CCtx_params* params);
-ZSTDLIB_API size_t ZSTD_estimateDStreamSize(size_t windowSize);
-ZSTDLIB_API size_t ZSTD_estimateDStreamSize_fromFrame(const void* src, size_t srcSize);
+ZSTDLIB_STATIC_API size_t ZSTD_estimateCStreamSize(int compressionLevel);
+ZSTDLIB_STATIC_API size_t ZSTD_estimateCStreamSize_usingCParams(ZSTD_compressionParameters cParams);
+ZSTDLIB_STATIC_API size_t ZSTD_estimateCStreamSize_usingCCtxParams(const ZSTD_CCtx_params* params);
+ZSTDLIB_STATIC_API size_t ZSTD_estimateDStreamSize(size_t windowSize);
+ZSTDLIB_STATIC_API size_t ZSTD_estimateDStreamSize_fromFrame(const void* src, size_t srcSize);
/*! ZSTD_estimate?DictSize() :
* ZSTD_estimateCDictSize() will bet that src size is relatively "small", and content is copied, like ZSTD_createCDict().
* ZSTD_estimateCDictSize_advanced() makes it possible to control compression parameters precisely, like ZSTD_createCDict_advanced().
* Note : dictionaries created by reference (`ZSTD_dlm_byRef`) are logically smaller.
*/
-ZSTDLIB_API size_t ZSTD_estimateCDictSize(size_t dictSize, int compressionLevel);
-ZSTDLIB_API size_t ZSTD_estimateCDictSize_advanced(size_t dictSize, ZSTD_compressionParameters cParams, ZSTD_dictLoadMethod_e dictLoadMethod);
-ZSTDLIB_API size_t ZSTD_estimateDDictSize(size_t dictSize, ZSTD_dictLoadMethod_e dictLoadMethod);
+ZSTDLIB_STATIC_API size_t ZSTD_estimateCDictSize(size_t dictSize, int compressionLevel);
+ZSTDLIB_STATIC_API size_t ZSTD_estimateCDictSize_advanced(size_t dictSize, ZSTD_compressionParameters cParams, ZSTD_dictLoadMethod_e dictLoadMethod);
+ZSTDLIB_STATIC_API size_t ZSTD_estimateDDictSize(size_t dictSize, ZSTD_dictLoadMethod_e dictLoadMethod);
/*! ZSTD_initStatic*() :
* Initialize an object using a pre-allocated fixed-size buffer.
@@ -1472,20 +1538,20 @@ ZSTDLIB_API size_t ZSTD_estimateDDictSize(size_t dictSize, ZSTD_dictLoadMethod_e
* Limitation 2 : static cctx currently not compatible with multi-threading.
* Limitation 3 : static dctx is incompatible with legacy support.
*/
-ZSTDLIB_API ZSTD_CCtx* ZSTD_initStaticCCtx(void* workspace, size_t workspaceSize);
-ZSTDLIB_API ZSTD_CStream* ZSTD_initStaticCStream(void* workspace, size_t workspaceSize); /*< same as ZSTD_initStaticCCtx() */
+ZSTDLIB_STATIC_API ZSTD_CCtx* ZSTD_initStaticCCtx(void* workspace, size_t workspaceSize);
+ZSTDLIB_STATIC_API ZSTD_CStream* ZSTD_initStaticCStream(void* workspace, size_t workspaceSize); /*< same as ZSTD_initStaticCCtx() */
-ZSTDLIB_API ZSTD_DCtx* ZSTD_initStaticDCtx(void* workspace, size_t workspaceSize);
-ZSTDLIB_API ZSTD_DStream* ZSTD_initStaticDStream(void* workspace, size_t workspaceSize); /*< same as ZSTD_initStaticDCtx() */
+ZSTDLIB_STATIC_API ZSTD_DCtx* ZSTD_initStaticDCtx(void* workspace, size_t workspaceSize);
+ZSTDLIB_STATIC_API ZSTD_DStream* ZSTD_initStaticDStream(void* workspace, size_t workspaceSize); /*< same as ZSTD_initStaticDCtx() */
-ZSTDLIB_API const ZSTD_CDict* ZSTD_initStaticCDict(
+ZSTDLIB_STATIC_API const ZSTD_CDict* ZSTD_initStaticCDict(
void* workspace, size_t workspaceSize,
const void* dict, size_t dictSize,
ZSTD_dictLoadMethod_e dictLoadMethod,
ZSTD_dictContentType_e dictContentType,
ZSTD_compressionParameters cParams);
-ZSTDLIB_API const ZSTD_DDict* ZSTD_initStaticDDict(
+ZSTDLIB_STATIC_API const ZSTD_DDict* ZSTD_initStaticDDict(
void* workspace, size_t workspaceSize,
const void* dict, size_t dictSize,
ZSTD_dictLoadMethod_e dictLoadMethod,
@@ -1504,44 +1570,44 @@ static
__attribute__((__unused__))
ZSTD_customMem const ZSTD_defaultCMem = { NULL, NULL, NULL }; /*< this constant defers to stdlib's functions */
-ZSTDLIB_API ZSTD_CCtx* ZSTD_createCCtx_advanced(ZSTD_customMem customMem);
-ZSTDLIB_API ZSTD_CStream* ZSTD_createCStream_advanced(ZSTD_customMem customMem);
-ZSTDLIB_API ZSTD_DCtx* ZSTD_createDCtx_advanced(ZSTD_customMem customMem);
-ZSTDLIB_API ZSTD_DStream* ZSTD_createDStream_advanced(ZSTD_customMem customMem);
+ZSTDLIB_STATIC_API ZSTD_CCtx* ZSTD_createCCtx_advanced(ZSTD_customMem customMem);
+ZSTDLIB_STATIC_API ZSTD_CStream* ZSTD_createCStream_advanced(ZSTD_customMem customMem);
+ZSTDLIB_STATIC_API ZSTD_DCtx* ZSTD_createDCtx_advanced(ZSTD_customMem customMem);
+ZSTDLIB_STATIC_API ZSTD_DStream* ZSTD_createDStream_advanced(ZSTD_customMem customMem);
-ZSTDLIB_API ZSTD_CDict* ZSTD_createCDict_advanced(const void* dict, size_t dictSize,
+ZSTDLIB_STATIC_API ZSTD_CDict* ZSTD_createCDict_advanced(const void* dict, size_t dictSize,
ZSTD_dictLoadMethod_e dictLoadMethod,
ZSTD_dictContentType_e dictContentType,
ZSTD_compressionParameters cParams,
ZSTD_customMem customMem);
-/* ! Thread pool :
- * These prototypes make it possible to share a thread pool among multiple compression contexts.
- * This can limit resources for applications with multiple threads where each one uses
- * a threaded compression mode (via ZSTD_c_nbWorkers parameter).
- * ZSTD_createThreadPool creates a new thread pool with a given number of threads.
- * Note that the lifetime of such pool must exist while being used.
- * ZSTD_CCtx_refThreadPool assigns a thread pool to a context (use NULL argument value
- * to use an internal thread pool).
- * ZSTD_freeThreadPool frees a thread pool, accepts NULL pointer.
+/*! Thread pool :
+ * These prototypes make it possible to share a thread pool among multiple compression contexts.
+ * This can limit resources for applications with multiple threads where each one uses
+ * a threaded compression mode (via ZSTD_c_nbWorkers parameter).
+ * ZSTD_createThreadPool creates a new thread pool with a given number of threads.
+ * Note that the lifetime of such pool must exist while being used.
+ * ZSTD_CCtx_refThreadPool assigns a thread pool to a context (use NULL argument value
+ * to use an internal thread pool).
+ * ZSTD_freeThreadPool frees a thread pool, accepts NULL pointer.
*/
typedef struct POOL_ctx_s ZSTD_threadPool;
-ZSTDLIB_API ZSTD_threadPool* ZSTD_createThreadPool(size_t numThreads);
-ZSTDLIB_API void ZSTD_freeThreadPool (ZSTD_threadPool* pool); /* accept NULL pointer */
-ZSTDLIB_API size_t ZSTD_CCtx_refThreadPool(ZSTD_CCtx* cctx, ZSTD_threadPool* pool);
+ZSTDLIB_STATIC_API ZSTD_threadPool* ZSTD_createThreadPool(size_t numThreads);
+ZSTDLIB_STATIC_API void ZSTD_freeThreadPool (ZSTD_threadPool* pool); /* accept NULL pointer */
+ZSTDLIB_STATIC_API size_t ZSTD_CCtx_refThreadPool(ZSTD_CCtx* cctx, ZSTD_threadPool* pool);
/*
* This API is temporary and is expected to change or disappear in the future!
*/
-ZSTDLIB_API ZSTD_CDict* ZSTD_createCDict_advanced2(
+ZSTDLIB_STATIC_API ZSTD_CDict* ZSTD_createCDict_advanced2(
const void* dict, size_t dictSize,
ZSTD_dictLoadMethod_e dictLoadMethod,
ZSTD_dictContentType_e dictContentType,
const ZSTD_CCtx_params* cctxParams,
ZSTD_customMem customMem);
-ZSTDLIB_API ZSTD_DDict* ZSTD_createDDict_advanced(
+ZSTDLIB_STATIC_API ZSTD_DDict* ZSTD_createDDict_advanced(
const void* dict, size_t dictSize,
ZSTD_dictLoadMethod_e dictLoadMethod,
ZSTD_dictContentType_e dictContentType,
@@ -1558,28 +1624,22 @@ ZSTDLIB_API ZSTD_DDict* ZSTD_createDDict_advanced(
* As a consequence, `dictBuffer` **must** outlive CDict,
* and its content must remain unmodified throughout the lifetime of CDict.
* note: equivalent to ZSTD_createCDict_advanced(), with dictLoadMethod==ZSTD_dlm_byRef */
-ZSTDLIB_API ZSTD_CDict* ZSTD_createCDict_byReference(const void* dictBuffer, size_t dictSize, int compressionLevel);
-
-/*! ZSTD_getDictID_fromCDict() :
- * Provides the dictID of the dictionary loaded into `cdict`.
- * If @return == 0, the dictionary is not conformant to Zstandard specification, or empty.
- * Non-conformant dictionaries can still be loaded, but as content-only dictionaries. */
-ZSTDLIB_API unsigned ZSTD_getDictID_fromCDict(const ZSTD_CDict* cdict);
+ZSTDLIB_STATIC_API ZSTD_CDict* ZSTD_createCDict_byReference(const void* dictBuffer, size_t dictSize, int compressionLevel);
/*! ZSTD_getCParams() :
* @return ZSTD_compressionParameters structure for a selected compression level and estimated srcSize.
* `estimatedSrcSize` value is optional, select 0 if not known */
-ZSTDLIB_API ZSTD_compressionParameters ZSTD_getCParams(int compressionLevel, unsigned long long estimatedSrcSize, size_t dictSize);
+ZSTDLIB_STATIC_API ZSTD_compressionParameters ZSTD_getCParams(int compressionLevel, unsigned long long estimatedSrcSize, size_t dictSize);
/*! ZSTD_getParams() :
* same as ZSTD_getCParams(), but @return a full `ZSTD_parameters` object instead of sub-component `ZSTD_compressionParameters`.
* All fields of `ZSTD_frameParameters` are set to default : contentSize=1, checksum=0, noDictID=0 */
-ZSTDLIB_API ZSTD_parameters ZSTD_getParams(int compressionLevel, unsigned long long estimatedSrcSize, size_t dictSize);
+ZSTDLIB_STATIC_API ZSTD_parameters ZSTD_getParams(int compressionLevel, unsigned long long estimatedSrcSize, size_t dictSize);
/*! ZSTD_checkCParams() :
* Ensure param values remain within authorized range.
* @return 0 on success, or an error code (can be checked with ZSTD_isError()) */
-ZSTDLIB_API size_t ZSTD_checkCParams(ZSTD_compressionParameters params);
+ZSTDLIB_STATIC_API size_t ZSTD_checkCParams(ZSTD_compressionParameters params);
/*! ZSTD_adjustCParams() :
* optimize params for a given `srcSize` and `dictSize`.
@@ -1587,23 +1647,25 @@ ZSTDLIB_API size_t ZSTD_checkCParams(ZSTD_compressionParameters params);
* `dictSize` must be `0` when there is no dictionary.
* cPar can be invalid : all parameters will be clamped within valid range in the @return struct.
* This function never fails (wide contract) */
-ZSTDLIB_API ZSTD_compressionParameters ZSTD_adjustCParams(ZSTD_compressionParameters cPar, unsigned long long srcSize, size_t dictSize);
+ZSTDLIB_STATIC_API ZSTD_compressionParameters ZSTD_adjustCParams(ZSTD_compressionParameters cPar, unsigned long long srcSize, size_t dictSize);
/*! ZSTD_compress_advanced() :
* Note : this function is now DEPRECATED.
* It can be replaced by ZSTD_compress2(), in combination with ZSTD_CCtx_setParameter() and other parameter setters.
- * This prototype will be marked as deprecated and generate compilation warning on reaching v1.5.x */
-ZSTDLIB_API size_t ZSTD_compress_advanced(ZSTD_CCtx* cctx,
+ * This prototype will generate compilation warnings. */
+ZSTD_DEPRECATED("use ZSTD_compress2")
+size_t ZSTD_compress_advanced(ZSTD_CCtx* cctx,
void* dst, size_t dstCapacity,
const void* src, size_t srcSize,
const void* dict,size_t dictSize,
ZSTD_parameters params);
/*! ZSTD_compress_usingCDict_advanced() :
- * Note : this function is now REDUNDANT.
+ * Note : this function is now DEPRECATED.
* It can be replaced by ZSTD_compress2(), in combination with ZSTD_CCtx_loadDictionary() and other parameter setters.
- * This prototype will be marked as deprecated and generate compilation warning in some future version */
-ZSTDLIB_API size_t ZSTD_compress_usingCDict_advanced(ZSTD_CCtx* cctx,
+ * This prototype will generate compilation warnings. */
+ZSTD_DEPRECATED("use ZSTD_compress2 with ZSTD_CCtx_loadDictionary")
+size_t ZSTD_compress_usingCDict_advanced(ZSTD_CCtx* cctx,
void* dst, size_t dstCapacity,
const void* src, size_t srcSize,
const ZSTD_CDict* cdict,
@@ -1613,18 +1675,18 @@ ZSTDLIB_API size_t ZSTD_compress_usingCDict_advanced(ZSTD_CCtx* cctx,
/*! ZSTD_CCtx_loadDictionary_byReference() :
* Same as ZSTD_CCtx_loadDictionary(), but dictionary content is referenced, instead of being copied into CCtx.
* It saves some memory, but also requires that `dict` outlives its usage within `cctx` */
-ZSTDLIB_API size_t ZSTD_CCtx_loadDictionary_byReference(ZSTD_CCtx* cctx, const void* dict, size_t dictSize);
+ZSTDLIB_STATIC_API size_t ZSTD_CCtx_loadDictionary_byReference(ZSTD_CCtx* cctx, const void* dict, size_t dictSize);
/*! ZSTD_CCtx_loadDictionary_advanced() :
* Same as ZSTD_CCtx_loadDictionary(), but gives finer control over
* how to load the dictionary (by copy ? by reference ?)
* and how to interpret it (automatic ? force raw mode ? full mode only ?) */
-ZSTDLIB_API size_t ZSTD_CCtx_loadDictionary_advanced(ZSTD_CCtx* cctx, const void* dict, size_t dictSize, ZSTD_dictLoadMethod_e dictLoadMethod, ZSTD_dictContentType_e dictContentType);
+ZSTDLIB_STATIC_API size_t ZSTD_CCtx_loadDictionary_advanced(ZSTD_CCtx* cctx, const void* dict, size_t dictSize, ZSTD_dictLoadMethod_e dictLoadMethod, ZSTD_dictContentType_e dictContentType);
/*! ZSTD_CCtx_refPrefix_advanced() :
* Same as ZSTD_CCtx_refPrefix(), but gives finer control over
* how to interpret prefix content (automatic ? force raw mode (default) ? full mode only ?) */
-ZSTDLIB_API size_t ZSTD_CCtx_refPrefix_advanced(ZSTD_CCtx* cctx, const void* prefix, size_t prefixSize, ZSTD_dictContentType_e dictContentType);
+ZSTDLIB_STATIC_API size_t ZSTD_CCtx_refPrefix_advanced(ZSTD_CCtx* cctx, const void* prefix, size_t prefixSize, ZSTD_dictContentType_e dictContentType);
/* === experimental parameters === */
/* these parameters can be used with ZSTD_setParameter()
@@ -1663,9 +1725,15 @@ ZSTDLIB_API size_t ZSTD_CCtx_refPrefix_advanced(ZSTD_CCtx* cctx, const void* pre
* See the comments on that enum for an explanation of the feature. */
#define ZSTD_c_forceAttachDict ZSTD_c_experimentalParam4
-/* Controls how the literals are compressed (default is auto).
- * The value must be of type ZSTD_literalCompressionMode_e.
- * See ZSTD_literalCompressionMode_t enum definition for details.
+/* Controlled with ZSTD_paramSwitch_e enum.
+ * Default is ZSTD_ps_auto.
+ * Set to ZSTD_ps_disable to never compress literals.
+ * Set to ZSTD_ps_enable to always compress literals. (Note: uncompressed literals
+ * may still be emitted if huffman is not beneficial to use.)
+ *
+ * By default, in ZSTD_ps_auto, the library will decide at runtime whether to use
+ * literals compression based on the compression parameters - specifically,
+ * negative compression levels do not use literal compression.
*/
#define ZSTD_c_literalCompressionMode ZSTD_c_experimentalParam5
@@ -1728,7 +1796,7 @@ ZSTDLIB_API size_t ZSTD_CCtx_refPrefix_advanced(ZSTD_CCtx* cctx, const void* pre
*
* Note that this means that the CDict tables can no longer be copied into the
* CCtx, so the dict attachment mode ZSTD_dictForceCopy will no longer be
- * useable. The dictionary can only be attached or reloaded.
+ * usable. The dictionary can only be attached or reloaded.
*
* In general, you should expect compression to be faster--sometimes very much
* so--and CDict creation to be slightly slower. Eventually, we will probably
@@ -1817,12 +1885,55 @@ ZSTDLIB_API size_t ZSTD_CCtx_refPrefix_advanced(ZSTD_CCtx* cctx, const void* pre
*/
#define ZSTD_c_validateSequences ZSTD_c_experimentalParam12
+/* ZSTD_c_useBlockSplitter
+ * Controlled with ZSTD_paramSwitch_e enum.
+ * Default is ZSTD_ps_auto.
+ * Set to ZSTD_ps_disable to never use block splitter.
+ * Set to ZSTD_ps_enable to always use block splitter.
+ *
+ * By default, in ZSTD_ps_auto, the library will decide at runtime whether to use
+ * block splitting based on the compression parameters.
+ */
+#define ZSTD_c_useBlockSplitter ZSTD_c_experimentalParam13
+
+/* ZSTD_c_useRowMatchFinder
+ * Controlled with ZSTD_paramSwitch_e enum.
+ * Default is ZSTD_ps_auto.
+ * Set to ZSTD_ps_disable to never use row-based matchfinder.
+ * Set to ZSTD_ps_enable to force usage of row-based matchfinder.
+ *
+ * By default, in ZSTD_ps_auto, the library will decide at runtime whether to use
+ * the row-based matchfinder based on support for SIMD instructions and the window log.
+ * Note that this only pertains to compression strategies: greedy, lazy, and lazy2
+ */
+#define ZSTD_c_useRowMatchFinder ZSTD_c_experimentalParam14
+
+/* ZSTD_c_deterministicRefPrefix
+ * Default is 0 == disabled. Set to 1 to enable.
+ *
+ * Zstd produces different results for prefix compression when the prefix is
+ * directly adjacent to the data about to be compressed vs. when it isn't.
+ * This is because zstd detects that the two buffers are contiguous and it can
+ * use a more efficient match finding algorithm. However, this produces different
+ * results than when the two buffers are non-contiguous. This flag forces zstd
+ * to always load the prefix in non-contiguous mode, even if it happens to be
+ * adjacent to the data, to guarantee determinism.
+ *
+ * If you really care about determinism when using a dictionary or prefix,
+ * like when doing delta compression, you should select this option. It comes
+ * at a speed penalty of about ~2.5% if the dictionary and data happened to be
+ * contiguous, and is free if they weren't contiguous. We don't expect that
+ * intentionally making the dictionary and data contiguous will be worth the
+ * cost to memcpy() the data.
+ */
+#define ZSTD_c_deterministicRefPrefix ZSTD_c_experimentalParam15
+
/*! ZSTD_CCtx_getParameter() :
* Get the requested compression parameter value, selected by enum ZSTD_cParameter,
* and store it into int* value.
* @return : 0, or an error code (which can be tested with ZSTD_isError()).
*/
-ZSTDLIB_API size_t ZSTD_CCtx_getParameter(const ZSTD_CCtx* cctx, ZSTD_cParameter param, int* value);
+ZSTDLIB_STATIC_API size_t ZSTD_CCtx_getParameter(const ZSTD_CCtx* cctx, ZSTD_cParameter param, int* value);
/*! ZSTD_CCtx_params :
@@ -1842,27 +1953,27 @@ ZSTDLIB_API size_t ZSTD_CCtx_getParameter(const ZSTD_CCtx* cctx, ZSTD_cParameter
* This can be used with ZSTD_estimateCCtxSize_advanced_usingCCtxParams()
* for static allocation of CCtx for single-threaded compression.
*/
-ZSTDLIB_API ZSTD_CCtx_params* ZSTD_createCCtxParams(void);
-ZSTDLIB_API size_t ZSTD_freeCCtxParams(ZSTD_CCtx_params* params); /* accept NULL pointer */
+ZSTDLIB_STATIC_API ZSTD_CCtx_params* ZSTD_createCCtxParams(void);
+ZSTDLIB_STATIC_API size_t ZSTD_freeCCtxParams(ZSTD_CCtx_params* params); /* accept NULL pointer */
/*! ZSTD_CCtxParams_reset() :
* Reset params to default values.
*/
-ZSTDLIB_API size_t ZSTD_CCtxParams_reset(ZSTD_CCtx_params* params);
+ZSTDLIB_STATIC_API size_t ZSTD_CCtxParams_reset(ZSTD_CCtx_params* params);
/*! ZSTD_CCtxParams_init() :
* Initializes the compression parameters of cctxParams according to
* compression level. All other parameters are reset to their default values.
*/
-ZSTDLIB_API size_t ZSTD_CCtxParams_init(ZSTD_CCtx_params* cctxParams, int compressionLevel);
+ZSTDLIB_STATIC_API size_t ZSTD_CCtxParams_init(ZSTD_CCtx_params* cctxParams, int compressionLevel);
/*! ZSTD_CCtxParams_init_advanced() :
* Initializes the compression and frame parameters of cctxParams according to
* params. All other parameters are reset to their default values.
*/
-ZSTDLIB_API size_t ZSTD_CCtxParams_init_advanced(ZSTD_CCtx_params* cctxParams, ZSTD_parameters params);
+ZSTDLIB_STATIC_API size_t ZSTD_CCtxParams_init_advanced(ZSTD_CCtx_params* cctxParams, ZSTD_parameters params);
-/*! ZSTD_CCtxParams_setParameter() :
+/*! ZSTD_CCtxParams_setParameter() : Requires v1.4.0+
* Similar to ZSTD_CCtx_setParameter.
* Set one compression parameter, selected by enum ZSTD_cParameter.
* Parameters must be applied to a ZSTD_CCtx using
@@ -1870,14 +1981,14 @@ ZSTDLIB_API size_t ZSTD_CCtxParams_init_advanced(ZSTD_CCtx_params* cctxParams, Z
* @result : a code representing success or failure (which can be tested with
* ZSTD_isError()).
*/
-ZSTDLIB_API size_t ZSTD_CCtxParams_setParameter(ZSTD_CCtx_params* params, ZSTD_cParameter param, int value);
+ZSTDLIB_STATIC_API size_t ZSTD_CCtxParams_setParameter(ZSTD_CCtx_params* params, ZSTD_cParameter param, int value);
/*! ZSTD_CCtxParams_getParameter() :
* Similar to ZSTD_CCtx_getParameter.
* Get the requested value of one compression parameter, selected by enum ZSTD_cParameter.
* @result : 0, or an error code (which can be tested with ZSTD_isError()).
*/
-ZSTDLIB_API size_t ZSTD_CCtxParams_getParameter(const ZSTD_CCtx_params* params, ZSTD_cParameter param, int* value);
+ZSTDLIB_STATIC_API size_t ZSTD_CCtxParams_getParameter(const ZSTD_CCtx_params* params, ZSTD_cParameter param, int* value);
/*! ZSTD_CCtx_setParametersUsingCCtxParams() :
* Apply a set of ZSTD_CCtx_params to the compression context.
@@ -1886,7 +1997,7 @@ ZSTDLIB_API size_t ZSTD_CCtxParams_getParameter(const ZSTD_CCtx_params* params,
* if nbWorkers>=1, new parameters will be picked up at next job,
* with a few restrictions (windowLog, pledgedSrcSize, nbWorkers, jobSize, and overlapLog are not updated).
*/
-ZSTDLIB_API size_t ZSTD_CCtx_setParametersUsingCCtxParams(
+ZSTDLIB_STATIC_API size_t ZSTD_CCtx_setParametersUsingCCtxParams(
ZSTD_CCtx* cctx, const ZSTD_CCtx_params* params);
/*! ZSTD_compressStream2_simpleArgs() :
@@ -1895,7 +2006,7 @@ ZSTDLIB_API size_t ZSTD_CCtx_setParametersUsingCCtxParams(
* This variant might be helpful for binders from dynamic languages
* which have troubles handling structures containing memory pointers.
*/
-ZSTDLIB_API size_t ZSTD_compressStream2_simpleArgs (
+ZSTDLIB_STATIC_API size_t ZSTD_compressStream2_simpleArgs (
ZSTD_CCtx* cctx,
void* dst, size_t dstCapacity, size_t* dstPos,
const void* src, size_t srcSize, size_t* srcPos,
@@ -1911,33 +2022,33 @@ ZSTDLIB_API size_t ZSTD_compressStream2_simpleArgs (
* Note : Frame Identifier is 4 bytes. If `size < 4`, @return will always be 0.
* Note 2 : Legacy Frame Identifiers are considered valid only if Legacy Support is enabled.
* Note 3 : Skippable Frame Identifiers are considered valid. */
-ZSTDLIB_API unsigned ZSTD_isFrame(const void* buffer, size_t size);
+ZSTDLIB_STATIC_API unsigned ZSTD_isFrame(const void* buffer, size_t size);
/*! ZSTD_createDDict_byReference() :
* Create a digested dictionary, ready to start decompression operation without startup delay.
* Dictionary content is referenced, and therefore stays in dictBuffer.
* It is important that dictBuffer outlives DDict,
* it must remain read accessible throughout the lifetime of DDict */
-ZSTDLIB_API ZSTD_DDict* ZSTD_createDDict_byReference(const void* dictBuffer, size_t dictSize);
+ZSTDLIB_STATIC_API ZSTD_DDict* ZSTD_createDDict_byReference(const void* dictBuffer, size_t dictSize);
/*! ZSTD_DCtx_loadDictionary_byReference() :
* Same as ZSTD_DCtx_loadDictionary(),
* but references `dict` content instead of copying it into `dctx`.
* This saves memory if `dict` remains around.,
* However, it's imperative that `dict` remains accessible (and unmodified) while being used, so it must outlive decompression. */
-ZSTDLIB_API size_t ZSTD_DCtx_loadDictionary_byReference(ZSTD_DCtx* dctx, const void* dict, size_t dictSize);
+ZSTDLIB_STATIC_API size_t ZSTD_DCtx_loadDictionary_byReference(ZSTD_DCtx* dctx, const void* dict, size_t dictSize);
/*! ZSTD_DCtx_loadDictionary_advanced() :
* Same as ZSTD_DCtx_loadDictionary(),
* but gives direct control over
* how to load the dictionary (by copy ? by reference ?)
* and how to interpret it (automatic ? force raw mode ? full mode only ?). */
-ZSTDLIB_API size_t ZSTD_DCtx_loadDictionary_advanced(ZSTD_DCtx* dctx, const void* dict, size_t dictSize, ZSTD_dictLoadMethod_e dictLoadMethod, ZSTD_dictContentType_e dictContentType);
+ZSTDLIB_STATIC_API size_t ZSTD_DCtx_loadDictionary_advanced(ZSTD_DCtx* dctx, const void* dict, size_t dictSize, ZSTD_dictLoadMethod_e dictLoadMethod, ZSTD_dictContentType_e dictContentType);
/*! ZSTD_DCtx_refPrefix_advanced() :
* Same as ZSTD_DCtx_refPrefix(), but gives finer control over
* how to interpret prefix content (automatic ? force raw mode (default) ? full mode only ?) */
-ZSTDLIB_API size_t ZSTD_DCtx_refPrefix_advanced(ZSTD_DCtx* dctx, const void* prefix, size_t prefixSize, ZSTD_dictContentType_e dictContentType);
+ZSTDLIB_STATIC_API size_t ZSTD_DCtx_refPrefix_advanced(ZSTD_DCtx* dctx, const void* prefix, size_t prefixSize, ZSTD_dictContentType_e dictContentType);
/*! ZSTD_DCtx_setMaxWindowSize() :
* Refuses allocating internal buffers for frames requiring a window size larger than provided limit.
@@ -1946,14 +2057,14 @@ ZSTDLIB_API size_t ZSTD_DCtx_refPrefix_advanced(ZSTD_DCtx* dctx, const void* pre
* By default, a decompression context accepts all window sizes <= (1 << ZSTD_WINDOWLOG_LIMIT_DEFAULT)
* @return : 0, or an error code (which can be tested using ZSTD_isError()).
*/
-ZSTDLIB_API size_t ZSTD_DCtx_setMaxWindowSize(ZSTD_DCtx* dctx, size_t maxWindowSize);
+ZSTDLIB_STATIC_API size_t ZSTD_DCtx_setMaxWindowSize(ZSTD_DCtx* dctx, size_t maxWindowSize);
/*! ZSTD_DCtx_getParameter() :
* Get the requested decompression parameter value, selected by enum ZSTD_dParameter,
* and store it into int* value.
* @return : 0, or an error code (which can be tested with ZSTD_isError()).
*/
-ZSTDLIB_API size_t ZSTD_DCtx_getParameter(ZSTD_DCtx* dctx, ZSTD_dParameter param, int* value);
+ZSTDLIB_STATIC_API size_t ZSTD_DCtx_getParameter(ZSTD_DCtx* dctx, ZSTD_dParameter param, int* value);
/* ZSTD_d_format
* experimental parameter,
@@ -2028,11 +2139,13 @@ ZSTDLIB_API size_t ZSTD_DCtx_getParameter(ZSTD_DCtx* dctx, ZSTD_dParameter param
/*! ZSTD_DCtx_setFormat() :
+ * This function is REDUNDANT. Prefer ZSTD_DCtx_setParameter().
* Instruct the decoder context about what kind of data to decode next.
* This instruction is mandatory to decode data without a fully-formed header,
* such ZSTD_f_zstd1_magicless for example.
* @return : 0, or an error code (which can be tested using ZSTD_isError()). */
-ZSTDLIB_API size_t ZSTD_DCtx_setFormat(ZSTD_DCtx* dctx, ZSTD_format_e format);
+ZSTD_DEPRECATED("use ZSTD_DCtx_setParameter() instead")
+size_t ZSTD_DCtx_setFormat(ZSTD_DCtx* dctx, ZSTD_format_e format);
/*! ZSTD_decompressStream_simpleArgs() :
* Same as ZSTD_decompressStream(),
@@ -2040,7 +2153,7 @@ ZSTDLIB_API size_t ZSTD_DCtx_setFormat(ZSTD_DCtx* dctx, ZSTD_format_e format);
* This can be helpful for binders from dynamic languages
* which have troubles handling structures containing memory pointers.
*/
-ZSTDLIB_API size_t ZSTD_decompressStream_simpleArgs (
+ZSTDLIB_STATIC_API size_t ZSTD_decompressStream_simpleArgs (
ZSTD_DCtx* dctx,
void* dst, size_t dstCapacity, size_t* dstPos,
const void* src, size_t srcSize, size_t* srcPos);
@@ -2056,7 +2169,7 @@ ZSTDLIB_API size_t ZSTD_decompressStream_simpleArgs (
/*===== Advanced Streaming compression functions =====*/
/*! ZSTD_initCStream_srcSize() :
- * This function is deprecated, and equivalent to:
+ * This function is DEPRECATED, and equivalent to:
* ZSTD_CCtx_reset(zcs, ZSTD_reset_session_only);
* ZSTD_CCtx_refCDict(zcs, NULL); // clear the dictionary (if any)
* ZSTD_CCtx_setParameter(zcs, ZSTD_c_compressionLevel, compressionLevel);
@@ -2065,15 +2178,15 @@ ZSTDLIB_API size_t ZSTD_decompressStream_simpleArgs (
* pledgedSrcSize must be correct. If it is not known at init time, use
* ZSTD_CONTENTSIZE_UNKNOWN. Note that, for compatibility with older programs,
* "0" also disables frame content size field. It may be enabled in the future.
- * Note : this prototype will be marked as deprecated and generate compilation warnings on reaching v1.5.x
+ * This prototype will generate compilation warnings.
*/
-ZSTDLIB_API size_t
-ZSTD_initCStream_srcSize(ZSTD_CStream* zcs,
+ZSTD_DEPRECATED("use ZSTD_CCtx_reset, see zstd.h for detailed instructions")
+size_t ZSTD_initCStream_srcSize(ZSTD_CStream* zcs,
int compressionLevel,
unsigned long long pledgedSrcSize);
/*! ZSTD_initCStream_usingDict() :
- * This function is deprecated, and is equivalent to:
+ * This function is DEPRECATED, and is equivalent to:
* ZSTD_CCtx_reset(zcs, ZSTD_reset_session_only);
* ZSTD_CCtx_setParameter(zcs, ZSTD_c_compressionLevel, compressionLevel);
* ZSTD_CCtx_loadDictionary(zcs, dict, dictSize);
@@ -2082,15 +2195,15 @@ ZSTD_initCStream_srcSize(ZSTD_CStream* zcs,
* dict == NULL or dictSize < 8, in which case no dict is used.
* Note: dict is loaded with ZSTD_dct_auto (treated as a full zstd dictionary if
* it begins with ZSTD_MAGIC_DICTIONARY, else as raw content) and ZSTD_dlm_byCopy.
- * Note : this prototype will be marked as deprecated and generate compilation warnings on reaching v1.5.x
+ * This prototype will generate compilation warnings.
*/
-ZSTDLIB_API size_t
-ZSTD_initCStream_usingDict(ZSTD_CStream* zcs,
+ZSTD_DEPRECATED("use ZSTD_CCtx_reset, see zstd.h for detailed instructions")
+size_t ZSTD_initCStream_usingDict(ZSTD_CStream* zcs,
const void* dict, size_t dictSize,
int compressionLevel);
/*! ZSTD_initCStream_advanced() :
- * This function is deprecated, and is approximately equivalent to:
+ * This function is DEPRECATED, and is approximately equivalent to:
* ZSTD_CCtx_reset(zcs, ZSTD_reset_session_only);
* // Pseudocode: Set each zstd parameter and leave the rest as-is.
* for ((param, value) : params) {
@@ -2102,23 +2215,24 @@ ZSTD_initCStream_usingDict(ZSTD_CStream* zcs,
* dict is loaded with ZSTD_dct_auto and ZSTD_dlm_byCopy.
* pledgedSrcSize must be correct.
* If srcSize is not known at init time, use value ZSTD_CONTENTSIZE_UNKNOWN.
- * Note : this prototype will be marked as deprecated and generate compilation warnings on reaching v1.5.x
+ * This prototype will generate compilation warnings.
*/
-ZSTDLIB_API size_t
-ZSTD_initCStream_advanced(ZSTD_CStream* zcs,
+ZSTD_DEPRECATED("use ZSTD_CCtx_reset, see zstd.h for detailed instructions")
+size_t ZSTD_initCStream_advanced(ZSTD_CStream* zcs,
const void* dict, size_t dictSize,
ZSTD_parameters params,
unsigned long long pledgedSrcSize);
/*! ZSTD_initCStream_usingCDict() :
- * This function is deprecated, and equivalent to:
+ * This function is DEPRECATED, and equivalent to:
* ZSTD_CCtx_reset(zcs, ZSTD_reset_session_only);
* ZSTD_CCtx_refCDict(zcs, cdict);
*
* note : cdict will just be referenced, and must outlive compression session
- * Note : this prototype will be marked as deprecated and generate compilation warnings on reaching v1.5.x
+ * This prototype will generate compilation warnings.
*/
-ZSTDLIB_API size_t ZSTD_initCStream_usingCDict(ZSTD_CStream* zcs, const ZSTD_CDict* cdict);
+ZSTD_DEPRECATED("use ZSTD_CCtx_reset and ZSTD_CCtx_refCDict, see zstd.h for detailed instructions")
+size_t ZSTD_initCStream_usingCDict(ZSTD_CStream* zcs, const ZSTD_CDict* cdict);
/*! ZSTD_initCStream_usingCDict_advanced() :
* This function is DEPRECATED, and is approximately equivalent to:
@@ -2133,18 +2247,21 @@ ZSTDLIB_API size_t ZSTD_initCStream_usingCDict(ZSTD_CStream* zcs, const ZSTD_CDi
* same as ZSTD_initCStream_usingCDict(), with control over frame parameters.
* pledgedSrcSize must be correct. If srcSize is not known at init time, use
* value ZSTD_CONTENTSIZE_UNKNOWN.
- * Note : this prototype will be marked as deprecated and generate compilation warnings on reaching v1.5.x
+ * This prototype will generate compilation warnings.
*/
-ZSTDLIB_API size_t
-ZSTD_initCStream_usingCDict_advanced(ZSTD_CStream* zcs,
+ZSTD_DEPRECATED("use ZSTD_CCtx_reset and ZSTD_CCtx_refCDict, see zstd.h for detailed instructions")
+size_t ZSTD_initCStream_usingCDict_advanced(ZSTD_CStream* zcs,
const ZSTD_CDict* cdict,
ZSTD_frameParameters fParams,
unsigned long long pledgedSrcSize);
/*! ZSTD_resetCStream() :
- * This function is deprecated, and is equivalent to:
+ * This function is DEPRECATED, and is equivalent to:
* ZSTD_CCtx_reset(zcs, ZSTD_reset_session_only);
* ZSTD_CCtx_setPledgedSrcSize(zcs, pledgedSrcSize);
+ * Note: ZSTD_resetCStream() interprets pledgedSrcSize == 0 as ZSTD_CONTENTSIZE_UNKNOWN, but
+ * ZSTD_CCtx_setPledgedSrcSize() does not do the same, so ZSTD_CONTENTSIZE_UNKNOWN must be
+ * explicitly specified.
*
* start a new frame, using same parameters from previous frame.
* This is typically useful to skip dictionary loading stage, since it will re-use it in-place.
@@ -2154,9 +2271,10 @@ ZSTD_initCStream_usingCDict_advanced(ZSTD_CStream* zcs,
* For the time being, pledgedSrcSize==0 is interpreted as "srcSize unknown" for compatibility with older programs,
* but it will change to mean "empty" in future version, so use macro ZSTD_CONTENTSIZE_UNKNOWN instead.
* @return : 0, or an error code (which can be tested using ZSTD_isError())
- * Note : this prototype will be marked as deprecated and generate compilation warnings on reaching v1.5.x
+ * This prototype will generate compilation warnings.
*/
-ZSTDLIB_API size_t ZSTD_resetCStream(ZSTD_CStream* zcs, unsigned long long pledgedSrcSize);
+ZSTD_DEPRECATED("use ZSTD_CCtx_reset, see zstd.h for detailed instructions")
+size_t ZSTD_resetCStream(ZSTD_CStream* zcs, unsigned long long pledgedSrcSize);
typedef struct {
@@ -2174,7 +2292,7 @@ typedef struct {
* Note : (ingested - consumed) is amount of input data buffered internally, not yet compressed.
* Aggregates progression inside active worker threads.
*/
-ZSTDLIB_API ZSTD_frameProgression ZSTD_getFrameProgression(const ZSTD_CCtx* cctx);
+ZSTDLIB_STATIC_API ZSTD_frameProgression ZSTD_getFrameProgression(const ZSTD_CCtx* cctx);
/*! ZSTD_toFlushNow() :
* Tell how many bytes are ready to be flushed immediately.
@@ -2189,7 +2307,7 @@ ZSTDLIB_API ZSTD_frameProgression ZSTD_getFrameProgression(const ZSTD_CCtx* cctx
* therefore flush speed is limited by production speed of oldest job
* irrespective of the speed of concurrent (and newer) jobs.
*/
-ZSTDLIB_API size_t ZSTD_toFlushNow(ZSTD_CCtx* cctx);
+ZSTDLIB_STATIC_API size_t ZSTD_toFlushNow(ZSTD_CCtx* cctx);
/*===== Advanced Streaming decompression functions =====*/
@@ -2203,7 +2321,7 @@ ZSTDLIB_API size_t ZSTD_toFlushNow(ZSTD_CCtx* cctx);
* note: no dictionary will be used if dict == NULL or dictSize < 8
* Note : this prototype will be marked as deprecated and generate compilation warnings on reaching v1.5.x
*/
-ZSTDLIB_API size_t ZSTD_initDStream_usingDict(ZSTD_DStream* zds, const void* dict, size_t dictSize);
+ZSTDLIB_STATIC_API size_t ZSTD_initDStream_usingDict(ZSTD_DStream* zds, const void* dict, size_t dictSize);
/*!
* This function is deprecated, and is equivalent to:
@@ -2214,7 +2332,7 @@ ZSTDLIB_API size_t ZSTD_initDStream_usingDict(ZSTD_DStream* zds, const void* dic
* note : ddict is referenced, it must outlive decompression session
* Note : this prototype will be marked as deprecated and generate compilation warnings on reaching v1.5.x
*/
-ZSTDLIB_API size_t ZSTD_initDStream_usingDDict(ZSTD_DStream* zds, const ZSTD_DDict* ddict);
+ZSTDLIB_STATIC_API size_t ZSTD_initDStream_usingDDict(ZSTD_DStream* zds, const ZSTD_DDict* ddict);
/*!
* This function is deprecated, and is equivalent to:
@@ -2224,7 +2342,7 @@ ZSTDLIB_API size_t ZSTD_initDStream_usingDDict(ZSTD_DStream* zds, const ZSTD_DDi
* re-use decompression parameters from previous init; saves dictionary loading
* Note : this prototype will be marked as deprecated and generate compilation warnings on reaching v1.5.x
*/
-ZSTDLIB_API size_t ZSTD_resetDStream(ZSTD_DStream* zds);
+ZSTDLIB_STATIC_API size_t ZSTD_resetDStream(ZSTD_DStream* zds);
/* *******************************************************************
@@ -2243,8 +2361,7 @@ ZSTDLIB_API size_t ZSTD_resetDStream(ZSTD_DStream* zds);
ZSTD_CCtx object can be re-used multiple times within successive compression operations.
Start by initializing a context.
- Use ZSTD_compressBegin(), or ZSTD_compressBegin_usingDict() for dictionary compression,
- or ZSTD_compressBegin_advanced(), for finer parameter control.
+ Use ZSTD_compressBegin(), or ZSTD_compressBegin_usingDict() for dictionary compression.
It's also possible to duplicate a reference context which has already been initialized, using ZSTD_copyCCtx()
Then, consume your input using ZSTD_compressContinue().
@@ -2267,17 +2384,19 @@ ZSTDLIB_API size_t ZSTD_resetDStream(ZSTD_DStream* zds);
*/
/*===== Buffer-less streaming compression functions =====*/
-ZSTDLIB_API size_t ZSTD_compressBegin(ZSTD_CCtx* cctx, int compressionLevel);
-ZSTDLIB_API size_t ZSTD_compressBegin_usingDict(ZSTD_CCtx* cctx, const void* dict, size_t dictSize, int compressionLevel);
-ZSTDLIB_API size_t ZSTD_compressBegin_advanced(ZSTD_CCtx* cctx, const void* dict, size_t dictSize, ZSTD_parameters params, unsigned long long pledgedSrcSize); /*< pledgedSrcSize : If srcSize is not known at init time, use ZSTD_CONTENTSIZE_UNKNOWN */
-ZSTDLIB_API size_t ZSTD_compressBegin_usingCDict(ZSTD_CCtx* cctx, const ZSTD_CDict* cdict); /*< note: fails if cdict==NULL */
-ZSTDLIB_API size_t ZSTD_compressBegin_usingCDict_advanced(ZSTD_CCtx* const cctx, const ZSTD_CDict* const cdict, ZSTD_frameParameters const fParams, unsigned long long const pledgedSrcSize); /* compression parameters are already set within cdict. pledgedSrcSize must be correct. If srcSize is not known, use macro ZSTD_CONTENTSIZE_UNKNOWN */
-ZSTDLIB_API size_t ZSTD_copyCCtx(ZSTD_CCtx* cctx, const ZSTD_CCtx* preparedCCtx, unsigned long long pledgedSrcSize); /*< note: if pledgedSrcSize is not known, use ZSTD_CONTENTSIZE_UNKNOWN */
+ZSTDLIB_STATIC_API size_t ZSTD_compressBegin(ZSTD_CCtx* cctx, int compressionLevel);
+ZSTDLIB_STATIC_API size_t ZSTD_compressBegin_usingDict(ZSTD_CCtx* cctx, const void* dict, size_t dictSize, int compressionLevel);
+ZSTDLIB_STATIC_API size_t ZSTD_compressBegin_usingCDict(ZSTD_CCtx* cctx, const ZSTD_CDict* cdict); /*< note: fails if cdict==NULL */
+ZSTDLIB_STATIC_API size_t ZSTD_copyCCtx(ZSTD_CCtx* cctx, const ZSTD_CCtx* preparedCCtx, unsigned long long pledgedSrcSize); /*< note: if pledgedSrcSize is not known, use ZSTD_CONTENTSIZE_UNKNOWN */
-ZSTDLIB_API size_t ZSTD_compressContinue(ZSTD_CCtx* cctx, void* dst, size_t dstCapacity, const void* src, size_t srcSize);
-ZSTDLIB_API size_t ZSTD_compressEnd(ZSTD_CCtx* cctx, void* dst, size_t dstCapacity, const void* src, size_t srcSize);
+ZSTDLIB_STATIC_API size_t ZSTD_compressContinue(ZSTD_CCtx* cctx, void* dst, size_t dstCapacity, const void* src, size_t srcSize);
+ZSTDLIB_STATIC_API size_t ZSTD_compressEnd(ZSTD_CCtx* cctx, void* dst, size_t dstCapacity, const void* src, size_t srcSize);
-
+/* The ZSTD_compressBegin_advanced() and ZSTD_compressBegin_usingCDict_advanced() are now DEPRECATED and will generate a compiler warning */
+ZSTD_DEPRECATED("use advanced API to access custom parameters")
+size_t ZSTD_compressBegin_advanced(ZSTD_CCtx* cctx, const void* dict, size_t dictSize, ZSTD_parameters params, unsigned long long pledgedSrcSize); /*< pledgedSrcSize : If srcSize is not known at init time, use ZSTD_CONTENTSIZE_UNKNOWN */
+ZSTD_DEPRECATED("use advanced API to access custom parameters")
+size_t ZSTD_compressBegin_usingCDict_advanced(ZSTD_CCtx* const cctx, const ZSTD_CDict* const cdict, ZSTD_frameParameters const fParams, unsigned long long const pledgedSrcSize); /* compression parameters are already set within cdict. pledgedSrcSize must be correct. If srcSize is not known, use macro ZSTD_CONTENTSIZE_UNKNOWN */
/*
Buffer-less streaming decompression (synchronous mode)
@@ -2368,24 +2487,24 @@ typedef struct {
* @return : 0, `zfhPtr` is correctly filled,
* >0, `srcSize` is too small, value is wanted `srcSize` amount,
* or an error code, which can be tested using ZSTD_isError() */
-ZSTDLIB_API size_t ZSTD_getFrameHeader(ZSTD_frameHeader* zfhPtr, const void* src, size_t srcSize); /*< doesn't consume input */
+ZSTDLIB_STATIC_API size_t ZSTD_getFrameHeader(ZSTD_frameHeader* zfhPtr, const void* src, size_t srcSize); /*< doesn't consume input */
/*! ZSTD_getFrameHeader_advanced() :
* same as ZSTD_getFrameHeader(),
* with added capability to select a format (like ZSTD_f_zstd1_magicless) */
-ZSTDLIB_API size_t ZSTD_getFrameHeader_advanced(ZSTD_frameHeader* zfhPtr, const void* src, size_t srcSize, ZSTD_format_e format);
-ZSTDLIB_API size_t ZSTD_decodingBufferSize_min(unsigned long long windowSize, unsigned long long frameContentSize); /*< when frame content size is not known, pass in frameContentSize == ZSTD_CONTENTSIZE_UNKNOWN */
+ZSTDLIB_STATIC_API size_t ZSTD_getFrameHeader_advanced(ZSTD_frameHeader* zfhPtr, const void* src, size_t srcSize, ZSTD_format_e format);
+ZSTDLIB_STATIC_API size_t ZSTD_decodingBufferSize_min(unsigned long long windowSize, unsigned long long frameContentSize); /*< when frame content size is not known, pass in frameContentSize == ZSTD_CONTENTSIZE_UNKNOWN */
-ZSTDLIB_API size_t ZSTD_decompressBegin(ZSTD_DCtx* dctx);
-ZSTDLIB_API size_t ZSTD_decompressBegin_usingDict(ZSTD_DCtx* dctx, const void* dict, size_t dictSize);
-ZSTDLIB_API size_t ZSTD_decompressBegin_usingDDict(ZSTD_DCtx* dctx, const ZSTD_DDict* ddict);
+ZSTDLIB_STATIC_API size_t ZSTD_decompressBegin(ZSTD_DCtx* dctx);
+ZSTDLIB_STATIC_API size_t ZSTD_decompressBegin_usingDict(ZSTD_DCtx* dctx, const void* dict, size_t dictSize);
+ZSTDLIB_STATIC_API size_t ZSTD_decompressBegin_usingDDict(ZSTD_DCtx* dctx, const ZSTD_DDict* ddict);
-ZSTDLIB_API size_t ZSTD_nextSrcSizeToDecompress(ZSTD_DCtx* dctx);
-ZSTDLIB_API size_t ZSTD_decompressContinue(ZSTD_DCtx* dctx, void* dst, size_t dstCapacity, const void* src, size_t srcSize);
+ZSTDLIB_STATIC_API size_t ZSTD_nextSrcSizeToDecompress(ZSTD_DCtx* dctx);
+ZSTDLIB_STATIC_API size_t ZSTD_decompressContinue(ZSTD_DCtx* dctx, void* dst, size_t dstCapacity, const void* src, size_t srcSize);
/* misc */
-ZSTDLIB_API void ZSTD_copyDCtx(ZSTD_DCtx* dctx, const ZSTD_DCtx* preparedDCtx);
+ZSTDLIB_STATIC_API void ZSTD_copyDCtx(ZSTD_DCtx* dctx, const ZSTD_DCtx* preparedDCtx);
typedef enum { ZSTDnit_frameHeader, ZSTDnit_blockHeader, ZSTDnit_block, ZSTDnit_lastBlock, ZSTDnit_checksum, ZSTDnit_skippableFrame } ZSTD_nextInputType_e;
-ZSTDLIB_API ZSTD_nextInputType_e ZSTD_nextInputType(ZSTD_DCtx* dctx);
+ZSTDLIB_STATIC_API ZSTD_nextInputType_e ZSTD_nextInputType(ZSTD_DCtx* dctx);
@@ -2422,10 +2541,10 @@ ZSTDLIB_API ZSTD_nextInputType_e ZSTD_nextInputType(ZSTD_DCtx* dctx);
*/
/*===== Raw zstd block functions =====*/
-ZSTDLIB_API size_t ZSTD_getBlockSize (const ZSTD_CCtx* cctx);
-ZSTDLIB_API size_t ZSTD_compressBlock (ZSTD_CCtx* cctx, void* dst, size_t dstCapacity, const void* src, size_t srcSize);
-ZSTDLIB_API size_t ZSTD_decompressBlock(ZSTD_DCtx* dctx, void* dst, size_t dstCapacity, const void* src, size_t srcSize);
-ZSTDLIB_API size_t ZSTD_insertBlock (ZSTD_DCtx* dctx, const void* blockStart, size_t blockSize); /*< insert uncompressed block into `dctx` history. Useful for multi-blocks decompression. */
+ZSTDLIB_STATIC_API size_t ZSTD_getBlockSize (const ZSTD_CCtx* cctx);
+ZSTDLIB_STATIC_API size_t ZSTD_compressBlock (ZSTD_CCtx* cctx, void* dst, size_t dstCapacity, const void* src, size_t srcSize);
+ZSTDLIB_STATIC_API size_t ZSTD_decompressBlock(ZSTD_DCtx* dctx, void* dst, size_t dstCapacity, const void* src, size_t srcSize);
+ZSTDLIB_STATIC_API size_t ZSTD_insertBlock (ZSTD_DCtx* dctx, const void* blockStart, size_t blockSize); /*< insert uncompressed block into `dctx` history. Useful for multi-blocks decompression. */
#endif /* ZSTD_H_ZSTD_STATIC_LINKING_ONLY */
diff --git a/include/misc/cxl.h b/include/misc/cxl.h
index 0410412..d804429 100644
--- a/include/misc/cxl.h
+++ b/include/misc/cxl.h
@@ -30,7 +30,7 @@ unsigned int cxl_pci_to_cfg_record(struct pci_dev *dev);
/*
* Context lifetime overview:
*
- * An AFU context may be inited and then started and stoppped multiple times
+ * An AFU context may be inited and then started and stopped multiple times
* before it's released. ie.
* - cxl_dev_context_init()
* - cxl_start_context()
diff --git a/include/trace/events/rwmmio.h b/include/trace/events/rwmmio.h
index de41159..a43e5dd 100644
--- a/include/trace/events/rwmmio.h
+++ b/include/trace/events/rwmmio.h
@@ -12,12 +12,14 @@
DECLARE_EVENT_CLASS(rwmmio_rw_template,
- TP_PROTO(unsigned long caller, u64 val, u8 width, volatile void __iomem *addr),
+ TP_PROTO(unsigned long caller, unsigned long caller0, u64 val, u8 width,
+ volatile void __iomem *addr),
- TP_ARGS(caller, val, width, addr),
+ TP_ARGS(caller, caller0, val, width, addr),
TP_STRUCT__entry(
__field(unsigned long, caller)
+ __field(unsigned long, caller0)
__field(unsigned long, addr)
__field(u64, val)
__field(u8, width)
@@ -25,56 +27,64 @@ DECLARE_EVENT_CLASS(rwmmio_rw_template,
TP_fast_assign(
__entry->caller = caller;
+ __entry->caller0 = caller0;
__entry->val = val;
__entry->addr = (unsigned long)addr;
__entry->width = width;
),
- TP_printk("%pS width=%d val=%#llx addr=%#lx",
- (void *)__entry->caller, __entry->width,
+ TP_printk("%pS -> %pS width=%d val=%#llx addr=%#lx",
+ (void *)__entry->caller0, (void *)__entry->caller, __entry->width,
__entry->val, __entry->addr)
);
DEFINE_EVENT(rwmmio_rw_template, rwmmio_write,
- TP_PROTO(unsigned long caller, u64 val, u8 width, volatile void __iomem *addr),
- TP_ARGS(caller, val, width, addr)
+ TP_PROTO(unsigned long caller, unsigned long caller0, u64 val, u8 width,
+ volatile void __iomem *addr),
+ TP_ARGS(caller, caller0, val, width, addr)
);
DEFINE_EVENT(rwmmio_rw_template, rwmmio_post_write,
- TP_PROTO(unsigned long caller, u64 val, u8 width, volatile void __iomem *addr),
- TP_ARGS(caller, val, width, addr)
+ TP_PROTO(unsigned long caller, unsigned long caller0, u64 val, u8 width,
+ volatile void __iomem *addr),
+ TP_ARGS(caller, caller0, val, width, addr)
);
TRACE_EVENT(rwmmio_read,
- TP_PROTO(unsigned long caller, u8 width, const volatile void __iomem *addr),
+ TP_PROTO(unsigned long caller, unsigned long caller0, u8 width,
+ const volatile void __iomem *addr),
- TP_ARGS(caller, width, addr),
+ TP_ARGS(caller, caller0, width, addr),
TP_STRUCT__entry(
__field(unsigned long, caller)
+ __field(unsigned long, caller0)
__field(unsigned long, addr)
__field(u8, width)
),
TP_fast_assign(
__entry->caller = caller;
+ __entry->caller0 = caller0;
__entry->addr = (unsigned long)addr;
__entry->width = width;
),
- TP_printk("%pS width=%d addr=%#lx",
- (void *)__entry->caller, __entry->width, __entry->addr)
+ TP_printk("%pS -> %pS width=%d addr=%#lx",
+ (void *)__entry->caller0, (void *)__entry->caller, __entry->width, __entry->addr)
);
TRACE_EVENT(rwmmio_post_read,
- TP_PROTO(unsigned long caller, u64 val, u8 width, const volatile void __iomem *addr),
+ TP_PROTO(unsigned long caller, unsigned long caller0, u64 val, u8 width,
+ const volatile void __iomem *addr),
- TP_ARGS(caller, val, width, addr),
+ TP_ARGS(caller, caller0, val, width, addr),
TP_STRUCT__entry(
__field(unsigned long, caller)
+ __field(unsigned long, caller0)
__field(unsigned long, addr)
__field(u64, val)
__field(u8, width)
@@ -82,13 +92,14 @@ TRACE_EVENT(rwmmio_post_read,
TP_fast_assign(
__entry->caller = caller;
+ __entry->caller0 = caller0;
__entry->val = val;
__entry->addr = (unsigned long)addr;
__entry->width = width;
),
- TP_printk("%pS width=%d val=%#llx addr=%#lx",
- (void *)__entry->caller, __entry->width,
+ TP_printk("%pS -> %pS width=%d val=%#llx addr=%#lx",
+ (void *)__entry->caller0, (void *)__entry->caller, __entry->width,
__entry->val, __entry->addr)
);
diff --git a/include/uapi/asm-generic/types.h b/include/uapi/asm-generic/types.h
index dfaa50d..7ad4dd0 100644
--- a/include/uapi/asm-generic/types.h
+++ b/include/uapi/asm-generic/types.h
@@ -1,9 +1,9 @@
/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */
-#ifndef _ASM_GENERIC_TYPES_H
-#define _ASM_GENERIC_TYPES_H
+#ifndef _UAPI_ASM_GENERIC_TYPES_H
+#define _UAPI_ASM_GENERIC_TYPES_H
/*
* int-ll64 is used everywhere now.
*/
#include <asm-generic/int-ll64.h>
-#endif /* _ASM_GENERIC_TYPES_H */
+#endif /* _UAPI_ASM_GENERIC_TYPES_H */
diff --git a/include/uapi/linux/hsi/cs-protocol.h b/include/uapi/linux/hsi/cs-protocol.h
index c7f6e76..07c3bfb 100644
--- a/include/uapi/linux/hsi/cs-protocol.h
+++ b/include/uapi/linux/hsi/cs-protocol.h
@@ -6,20 +6,6 @@
*
* Contact: Kai Vehmanen <kai.vehmanen@nokia.com>
* Original author: Peter Ujfalusi <peter.ujfalusi@nokia.com>
- *
- * This program is free software; you can redistribute it and/or
- * modify it under the terms of the GNU General Public License
- * version 2 as published by the Free Software Foundation.
- *
- * This program is distributed in the hope that it will be useful, but
- * WITHOUT ANY WARRANTY; without even the implied warranty of
- * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
- * General Public License for more details.
- *
- * You should have received a copy of the GNU General Public License
- * along with this program; if not, write to the Free Software
- * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA
- * 02110-1301 USA
*/
#ifndef _CS_PROTOCOL_H
diff --git a/include/uapi/linux/hsi/hsi_char.h b/include/uapi/linux/hsi/hsi_char.h
index 91623b03..5ef72f0 100644
--- a/include/uapi/linux/hsi/hsi_char.h
+++ b/include/uapi/linux/hsi/hsi_char.h
@@ -5,20 +5,6 @@
* Copyright (C) 2010 Nokia Corporation. All rights reserved.
*
* Contact: Andras Domokos <andras.domokos at nokia.com>
- *
- * This program is free software; you can redistribute it and/or
- * modify it under the terms of the GNU General Public License
- * version 2 as published by the Free Software Foundation.
- *
- * This program is distributed in the hope that it will be useful, but
- * WITHOUT ANY WARRANTY; without even the implied warranty of
- * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
- * General Public License for more details.
- *
- * You should have received a copy of the GNU General Public License
- * along with this program; if not, write to the Free Software
- * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA
- * 02110-1301 USA
*/
#ifndef __HSI_CHAR_H
diff --git a/include/uapi/linux/idxd.h b/include/uapi/linux/idxd.h
index 2b9e7fe..1d553be 100644
--- a/include/uapi/linux/idxd.h
+++ b/include/uapi/linux/idxd.h
@@ -295,7 +295,7 @@ struct dsa_completion_record {
};
uint32_t delta_rec_size;
- uint32_t crc_val;
+ uint64_t crc_val;
/* DIF check & strip */
struct {
diff --git a/include/uapi/linux/swab.h b/include/uapi/linux/swab.h
index 0723a9c..0171718 100644
--- a/include/uapi/linux/swab.h
+++ b/include/uapi/linux/swab.h
@@ -3,7 +3,7 @@
#define _UAPI_LINUX_SWAB_H
#include <linux/types.h>
-#include <linux/compiler.h>
+#include <linux/stddef.h>
#include <asm/bitsperlong.h>
#include <asm/swab.h>
diff --git a/init/main.c b/init/main.c
index d213371c..e1c3911 100644
--- a/init/main.c
+++ b/init/main.c
@@ -863,6 +863,7 @@ static void __init mm_init(void)
/* Should be run after espfix64 is set up. */
pti_init();
kmsan_init_runtime();
+ mm_cache_init();
}
#ifdef CONFIG_RANDOMIZE_KSTACK_OFFSET
@@ -998,7 +999,7 @@ asmlinkage __visible void __init __no_sanitize_address start_kernel(void)
sort_main_extable();
trap_init();
mm_init();
-
+ poking_init();
ftrace_init();
/* trace_printk can be enabled here */
@@ -1137,7 +1138,6 @@ asmlinkage __visible void __init __no_sanitize_address start_kernel(void)
taskstats_init_early();
delayacct_init();
- poking_init();
check_bugs();
acpi_subsystem_init();
diff --git a/init/version.c b/init/version.c
index 01d4ab0..f117921 100644
--- a/init/version.c
+++ b/init/version.c
@@ -15,7 +15,6 @@
#include <linux/printk.h>
#include <linux/uts.h>
#include <linux/utsname.h>
-#include <generated/utsrelease.h>
#include <linux/proc_ns.h>
static int __init early_hostname(char *arg)
diff --git a/kernel/bpf/bpf_struct_ops.c b/kernel/bpf/bpf_struct_ops.c
index 84b2d9d..ece9870 100644
--- a/kernel/bpf/bpf_struct_ops.c
+++ b/kernel/bpf/bpf_struct_ops.c
@@ -494,8 +494,7 @@ static int bpf_struct_ops_map_update_elem(struct bpf_map *map, void *key,
refcount_set(&kvalue->refcnt, 1);
bpf_map_inc(map);
- set_memory_ro((long)st_map->image, 1);
- set_memory_x((long)st_map->image, 1);
+ set_memory_rox((long)st_map->image, 1);
err = st_ops->reg(kdata);
if (likely(!err)) {
/* Pair with smp_load_acquire() during lookup_elem().
diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c
index 7f98dec..6cca66b 100644
--- a/kernel/bpf/core.c
+++ b/kernel/bpf/core.c
@@ -868,8 +868,7 @@ static struct bpf_prog_pack *alloc_new_pack(bpf_jit_fill_hole_t bpf_fill_ill_ins
list_add_tail(&pack->list, &pack_list);
set_vm_flush_reset_perms(pack->ptr);
- set_memory_ro((unsigned long)pack->ptr, BPF_PROG_PACK_SIZE / PAGE_SIZE);
- set_memory_x((unsigned long)pack->ptr, BPF_PROG_PACK_SIZE / PAGE_SIZE);
+ set_memory_rox((unsigned long)pack->ptr, BPF_PROG_PACK_SIZE / PAGE_SIZE);
return pack;
}
@@ -887,8 +886,7 @@ void *bpf_prog_pack_alloc(u32 size, bpf_jit_fill_hole_t bpf_fill_ill_insns)
if (ptr) {
bpf_fill_ill_insns(ptr, size);
set_vm_flush_reset_perms(ptr);
- set_memory_ro((unsigned long)ptr, size / PAGE_SIZE);
- set_memory_x((unsigned long)ptr, size / PAGE_SIZE);
+ set_memory_rox((unsigned long)ptr, size / PAGE_SIZE);
}
goto out;
}
diff --git a/kernel/bpf/trampoline.c b/kernel/bpf/trampoline.c
index d639521..11f5ec0 100644
--- a/kernel/bpf/trampoline.c
+++ b/kernel/bpf/trampoline.c
@@ -468,8 +468,7 @@ static int bpf_trampoline_update(struct bpf_trampoline *tr, bool lock_direct_mut
if (err < 0)
goto out;
- set_memory_ro((long)im->image, 1);
- set_memory_x((long)im->image, 1);
+ set_memory_rox((long)im->image, 1);
WARN_ON(tr->cur_image && tr->selector == 0);
WARN_ON(!tr->cur_image && tr->selector);
diff --git a/kernel/events/core.c b/kernel/events/core.c
index e47914a..eacc370 100644
--- a/kernel/events/core.c
+++ b/kernel/events/core.c
@@ -7493,7 +7493,7 @@ static u64 perf_get_pgtable_size(struct mm_struct *mm, unsigned long addr)
return pud_leaf_size(pud);
pmdp = pmd_offset_lockless(pudp, pud, addr);
- pmd = READ_ONCE(*pmdp);
+ pmd = pmdp_get_lockless(pmdp);
if (!pmd_present(pmd))
return 0;
diff --git a/kernel/fork.c b/kernel/fork.c
index 7a08025..9f7fe35 100644
--- a/kernel/fork.c
+++ b/kernel/fork.c
@@ -2607,11 +2607,6 @@ struct task_struct * __init fork_idle(int cpu)
return task;
}
-struct mm_struct *copy_init_mm(void)
-{
- return dup_mm(NULL, &init_mm);
-}
-
/*
* This is like kernel_clone(), but shaved down and tailored to just
* creating io_uring workers. It returns a created task, or an error pointer.
@@ -3030,10 +3025,27 @@ static void sighand_ctor(void *data)
init_waitqueue_head(&sighand->signalfd_wqh);
}
-void __init proc_caches_init(void)
+void __init mm_cache_init(void)
{
unsigned int mm_size;
+ /*
+ * The mm_cpumask is located at the end of mm_struct, and is
+ * dynamically sized based on the maximum CPU number this system
+ * can have, taking hotplug into account (nr_cpu_ids).
+ */
+ mm_size = sizeof(struct mm_struct) + cpumask_size();
+
+ mm_cachep = kmem_cache_create_usercopy("mm_struct",
+ mm_size, ARCH_MIN_MMSTRUCT_ALIGN,
+ SLAB_HWCACHE_ALIGN|SLAB_PANIC|SLAB_ACCOUNT,
+ offsetof(struct mm_struct, saved_auxv),
+ sizeof_field(struct mm_struct, saved_auxv),
+ NULL);
+}
+
+void __init proc_caches_init(void)
+{
sighand_cachep = kmem_cache_create("sighand_cache",
sizeof(struct sighand_struct), 0,
SLAB_HWCACHE_ALIGN|SLAB_PANIC|SLAB_TYPESAFE_BY_RCU|
@@ -3051,19 +3063,6 @@ void __init proc_caches_init(void)
SLAB_HWCACHE_ALIGN|SLAB_PANIC|SLAB_ACCOUNT,
NULL);
- /*
- * The mm_cpumask is located at the end of mm_struct, and is
- * dynamically sized based on the maximum CPU number this system
- * can have, taking hotplug into account (nr_cpu_ids).
- */
- mm_size = sizeof(struct mm_struct) + cpumask_size();
-
- mm_cachep = kmem_cache_create_usercopy("mm_struct",
- mm_size, ARCH_MIN_MMSTRUCT_ALIGN,
- SLAB_HWCACHE_ALIGN|SLAB_PANIC|SLAB_ACCOUNT,
- offsetof(struct mm_struct, saved_auxv),
- sizeof_field(struct mm_struct, saved_auxv),
- NULL);
vm_area_cachep = KMEM_CACHE(vm_area_struct, SLAB_PANIC|SLAB_ACCOUNT);
mmap_init();
nsproxy_cache_init();
diff --git a/kernel/irq/msi.c b/kernel/irq/msi.c
index bd4d4dd..955267b 100644
--- a/kernel/irq/msi.c
+++ b/kernel/irq/msi.c
@@ -165,7 +165,8 @@ static bool msi_ctrl_valid(struct device *dev, struct msi_ctrl *ctrl)
unsigned int hwsize;
if (WARN_ON_ONCE(ctrl->domid >= MSI_MAX_DEVICE_IRQDOMAINS ||
- !dev->msi.data->__domains[ctrl->domid].domain))
+ (dev->msi.domain &&
+ !dev->msi.data->__domains[ctrl->domid].domain)))
return false;
hwsize = msi_domain_get_hwsize(dev, ctrl->domid);
@@ -609,8 +610,8 @@ static unsigned int msi_domain_get_hwsize(struct device *dev, unsigned int domid
info = domain->host_data;
return info->hwsize;
}
- /* No domain, no size... */
- return 0;
+ /* No domain, default to MSI_XA_DOMAIN_SIZE */
+ return MSI_XA_DOMAIN_SIZE;
}
static inline void irq_chip_write_msi_msg(struct irq_data *data,
diff --git a/kernel/module/main.c b/kernel/module/main.c
index ff2dfd1..48568a0 100644
--- a/kernel/module/main.c
+++ b/kernel/module/main.c
@@ -1671,6 +1671,11 @@ static int elf_validity_check(struct load_info *info)
info->hdr->e_machine);
goto no_exec;
}
+ if (!module_elf_check_arch(info->hdr)) {
+ pr_err("Invalid module architecture in ELF header: %u\n",
+ info->hdr->e_machine);
+ goto no_exec;
+ }
if (info->hdr->e_shentsize != sizeof(Elf_Shdr)) {
pr_err("Invalid ELF section header size\n");
goto no_exec;
@@ -2244,6 +2249,11 @@ static void flush_module_icache(const struct module *mod)
(unsigned long)mod->core_layout.base + mod->core_layout.size);
}
+bool __weak module_elf_check_arch(Elf_Ehdr *hdr)
+{
+ return true;
+}
+
int __weak module_frob_arch_sections(Elf_Ehdr *hdr,
Elf_Shdr *sechdrs,
char *secstrings,
diff --git a/kernel/padata.c b/kernel/padata.c
index de90af5..e007b8a 100644
--- a/kernel/padata.c
+++ b/kernel/padata.c
@@ -83,8 +83,16 @@ static struct padata_work *padata_work_alloc(void)
return pw;
}
-static void padata_work_init(struct padata_work *pw, work_func_t work_fn,
- void *data, int flags)
+/*
+ * This function is marked __ref because this function may be optimized in such
+ * a way that it directly refers to work_fn's address, which causes modpost to
+ * complain when work_fn is marked __init. This scenario was observed with clang
+ * LTO, where padata_work_init() was optimized to refer directly to
+ * padata_mt_helper() because the calls to padata_work_init() with other work_fn
+ * values were eliminated or inlined.
+ */
+static void __ref padata_work_init(struct padata_work *pw, work_func_t work_fn,
+ void *data, int flags)
{
if (flags & PADATA_WORK_ONSTACK)
INIT_WORK_ONSTACK(&pw->pw_work, work_fn);
diff --git a/kernel/trace/Kconfig b/kernel/trace/Kconfig
index e95c8fe..1975452 100644
--- a/kernel/trace/Kconfig
+++ b/kernel/trace/Kconfig
@@ -82,6 +82,13 @@
help
Arch supports objtool --mcount
+config HAVE_OBJTOOL_NOP_MCOUNT
+ bool
+ help
+ Arch supports the objtool options --mcount with --mnop.
+ An architecture can select this if it wants to enable nop'ing
+ of ftrace locations.
+
config HAVE_C_RECORDMCOUNT
bool
help
diff --git a/lib/Kconfig.debug b/lib/Kconfig.debug
index a93c45e..881c3f8 100644
--- a/lib/Kconfig.debug
+++ b/lib/Kconfig.debug
@@ -312,8 +312,21 @@
DEBUG_INFO build and compile times are reduced too.
Only works with newer gcc versions.
-config DEBUG_INFO_COMPRESSED
- bool "Compressed debugging information"
+choice
+ prompt "Compressed Debug information"
+ help
+ Compress the resulting debug info. Results in smaller debug info sections,
+ but requires that consumers are able to decompress the results.
+
+ If unsure, choose DEBUG_INFO_COMPRESSED_NONE.
+
+config DEBUG_INFO_COMPRESSED_NONE
+ bool "Don't compress debug information"
+ help
+ Don't compress debug info sections.
+
+config DEBUG_INFO_COMPRESSED_ZLIB
+ bool "Compress debugging information with zlib"
depends on $(cc-option,-gz=zlib)
depends on $(ld-option,--compress-debug-sections=zlib)
help
@@ -327,6 +340,18 @@
preferable to setting $KDEB_COMPRESS to "none" which would be even
larger.
+config DEBUG_INFO_COMPRESSED_ZSTD
+ bool "Compress debugging information with zstd"
+ depends on $(cc-option,-gz=zstd)
+ depends on $(ld-option,--compress-debug-sections=zstd)
+ help
+ Compress the debug information using zstd. This may provide better
+ compression than zlib, for about the same time costs, but requires newer
+ toolchain support. Requires GCC 13.0+ or Clang 16.0+, binutils 2.40+, and
+ zstd.
+
+endchoice # "Compressed Debug information"
+
config DEBUG_INFO_SPLIT
bool "Produce split debuginfo in .dwo files"
depends on $(cc-option,-gsplit-dwarf)
@@ -728,6 +753,7 @@
select STACKTRACE if STACKTRACE_SUPPORT
select KALLSYMS
select CRC32
+ select STACKDEPOT
help
Say Y here if you want to enable the memory leak
detector. The memory allocation/freeing is traced in a way
@@ -1982,7 +2008,6 @@
config FAULT_INJECTION_STACKTRACE_FILTER
bool "stacktrace filter for fault-injection capabilities"
depends on FAULT_INJECTION_DEBUG_FS && STACKTRACE_SUPPORT
- depends on !X86_64
select STACKTRACE
depends on FRAME_POINTER || MIPS || PPC || S390 || MICROBLAZE || ARM || ARC || X86
help
diff --git a/lib/fault-inject.c b/lib/fault-inject.c
index 1421818..6cff320 100644
--- a/lib/fault-inject.c
+++ b/lib/fault-inject.c
@@ -71,7 +71,7 @@ static bool fail_stacktrace(struct fault_attr *attr)
int n, nr_entries;
bool found = (attr->require_start == 0 && attr->require_end == ULONG_MAX);
- if (depth == 0)
+ if (depth == 0 || (found && !attr->reject_start && !attr->reject_end))
return found;
nr_entries = stack_trace_save(entries, depth, 1);
@@ -102,10 +102,16 @@ static inline bool fail_stacktrace(struct fault_attr *attr)
bool should_fail_ex(struct fault_attr *attr, ssize_t size, int flags)
{
+ bool stack_checked = false;
+
if (in_task()) {
unsigned int fail_nth = READ_ONCE(current->fail_nth);
if (fail_nth) {
+ if (!fail_stacktrace(attr))
+ return false;
+
+ stack_checked = true;
fail_nth--;
WRITE_ONCE(current->fail_nth, fail_nth);
if (!fail_nth)
@@ -125,6 +131,9 @@ bool should_fail_ex(struct fault_attr *attr, ssize_t size, int flags)
if (atomic_read(&attr->times) == 0)
return false;
+ if (!stack_checked && !fail_stacktrace(attr))
+ return false;
+
if (atomic_read(&attr->space) > size) {
atomic_sub(size, &attr->space);
return false;
@@ -139,9 +148,6 @@ bool should_fail_ex(struct fault_attr *attr, ssize_t size, int flags)
if (attr->probability <= get_random_u32_below(100))
return false;
- if (!fail_stacktrace(attr))
- return false;
-
fail:
if (!(flags & FAULT_NOWARN))
fail_dump(attr);
@@ -226,10 +232,10 @@ struct dentry *fault_create_debugfs_attr(const char *name,
#ifdef CONFIG_FAULT_INJECTION_STACKTRACE_FILTER
debugfs_create_stacktrace_depth("stacktrace-depth", mode, dir,
&attr->stacktrace_depth);
- debugfs_create_ul("require-start", mode, dir, &attr->require_start);
- debugfs_create_ul("require-end", mode, dir, &attr->require_end);
- debugfs_create_ul("reject-start", mode, dir, &attr->reject_start);
- debugfs_create_ul("reject-end", mode, dir, &attr->reject_end);
+ debugfs_create_xul("require-start", mode, dir, &attr->require_start);
+ debugfs_create_xul("require-end", mode, dir, &attr->require_end);
+ debugfs_create_xul("reject-start", mode, dir, &attr->reject_start);
+ debugfs_create_xul("reject-end", mode, dir, &attr->reject_end);
#endif /* CONFIG_FAULT_INJECTION_STACKTRACE_FILTER */
attr->dname = dget(dir);
diff --git a/lib/maple_tree.c b/lib/maple_tree.c
index 3fe1491..fe3947b 100644
--- a/lib/maple_tree.c
+++ b/lib/maple_tree.c
@@ -6062,7 +6062,7 @@ void *mas_find_rev(struct ma_state *mas, unsigned long min)
if (mas->index < min)
return NULL;
- /* Retries on dead nodes handled by mas_next_entry */
+ /* Retries on dead nodes handled by mas_prev_entry */
return mas_prev_entry(mas, min);
}
EXPORT_SYMBOL_GPL(mas_find_rev);
diff --git a/lib/trace_readwrite.c b/lib/trace_readwrite.c
index 8863703..62b4e8b 100644
--- a/lib/trace_readwrite.c
+++ b/lib/trace_readwrite.c
@@ -14,33 +14,33 @@
#ifdef CONFIG_TRACE_MMIO_ACCESS
void log_write_mmio(u64 val, u8 width, volatile void __iomem *addr,
- unsigned long caller_addr)
+ unsigned long caller_addr, unsigned long caller_addr0)
{
- trace_rwmmio_write(caller_addr, val, width, addr);
+ trace_rwmmio_write(caller_addr, caller_addr0, val, width, addr);
}
EXPORT_SYMBOL_GPL(log_write_mmio);
EXPORT_TRACEPOINT_SYMBOL_GPL(rwmmio_write);
void log_post_write_mmio(u64 val, u8 width, volatile void __iomem *addr,
- unsigned long caller_addr)
+ unsigned long caller_addr, unsigned long caller_addr0)
{
- trace_rwmmio_post_write(caller_addr, val, width, addr);
+ trace_rwmmio_post_write(caller_addr, caller_addr0, val, width, addr);
}
EXPORT_SYMBOL_GPL(log_post_write_mmio);
EXPORT_TRACEPOINT_SYMBOL_GPL(rwmmio_post_write);
void log_read_mmio(u8 width, const volatile void __iomem *addr,
- unsigned long caller_addr)
+ unsigned long caller_addr, unsigned long caller_addr0)
{
- trace_rwmmio_read(caller_addr, width, addr);
+ trace_rwmmio_read(caller_addr, caller_addr0, width, addr);
}
EXPORT_SYMBOL_GPL(log_read_mmio);
EXPORT_TRACEPOINT_SYMBOL_GPL(rwmmio_read);
void log_post_read_mmio(u64 val, u8 width, const volatile void __iomem *addr,
- unsigned long caller_addr)
+ unsigned long caller_addr, unsigned long caller_addr0)
{
- trace_rwmmio_post_read(caller_addr, val, width, addr);
+ trace_rwmmio_post_read(caller_addr, caller_addr0, val, width, addr);
}
EXPORT_SYMBOL_GPL(log_post_read_mmio);
EXPORT_TRACEPOINT_SYMBOL_GPL(rwmmio_post_read);
diff --git a/lib/zstd/Makefile b/lib/zstd/Makefile
index 440bd00..20f08c6 100644
--- a/lib/zstd/Makefile
+++ b/lib/zstd/Makefile
@@ -35,6 +35,7 @@
decompress/zstd_decompress_block.o \
zstd_common-y := \
+ zstd_common_module.o \
common/debug.o \
common/entropy_common.o \
common/error_private.o \
diff --git a/lib/zstd/common/bitstream.h b/lib/zstd/common/bitstream.h
index 28248ab..feef3a1b 100644
--- a/lib/zstd/common/bitstream.h
+++ b/lib/zstd/common/bitstream.h
@@ -313,7 +313,16 @@ MEM_STATIC FORCE_INLINE_ATTR size_t BIT_getMiddleBits(size_t bitContainer, U32 c
U32 const regMask = sizeof(bitContainer)*8 - 1;
/* if start > regMask, bitstream is corrupted, and result is undefined */
assert(nbBits < BIT_MASK_SIZE);
+ /* x86 transform & ((1 << nbBits) - 1) to bzhi instruction, it is better
+ * than accessing memory. When bmi2 instruction is not present, we consider
+ * such cpus old (pre-Haswell, 2013) and their performance is not of that
+ * importance.
+ */
+#if defined(__x86_64__) || defined(_M_X86)
+ return (bitContainer >> (start & regMask)) & ((((U64)1) << nbBits) - 1);
+#else
return (bitContainer >> (start & regMask)) & BIT_mask[nbBits];
+#endif
}
MEM_STATIC FORCE_INLINE_ATTR size_t BIT_getLowerBits(size_t bitContainer, U32 const nbBits)
diff --git a/lib/zstd/common/compiler.h b/lib/zstd/common/compiler.h
index f5a9c70..c42d39f 100644
--- a/lib/zstd/common/compiler.h
+++ b/lib/zstd/common/compiler.h
@@ -11,6 +11,8 @@
#ifndef ZSTD_COMPILER_H
#define ZSTD_COMPILER_H
+#include "portability_macros.h"
+
/*-*******************************************************
* Compiler specifics
*********************************************************/
@@ -34,7 +36,7 @@
/*
On MSVC qsort requires that functions passed into it use the __cdecl calling conversion(CC).
- This explictly marks such functions as __cdecl so that the code will still compile
+ This explicitly marks such functions as __cdecl so that the code will still compile
if a CC other than __cdecl has been made the default.
*/
#define WIN_CDECL
@@ -70,25 +72,13 @@
/* target attribute */
-#ifndef __has_attribute
- #define __has_attribute(x) 0 /* Compatibility with non-clang compilers. */
-#endif
#define TARGET_ATTRIBUTE(target) __attribute__((__target__(target)))
-/* Enable runtime BMI2 dispatch based on the CPU.
- * Enabled for clang & gcc >=4.8 on x86 when BMI2 isn't enabled by default.
+/* Target attribute for BMI2 dynamic dispatch.
+ * Enable lzcnt, bmi, and bmi2.
+ * We test for bmi1 & bmi2. lzcnt is included in bmi1.
*/
-#ifndef DYNAMIC_BMI2
- #if ((defined(__clang__) && __has_attribute(__target__)) \
- || (defined(__GNUC__) \
- && (__GNUC__ >= 5 || (__GNUC__ == 4 && __GNUC_MINOR__ >= 8)))) \
- && (defined(__x86_64__) || defined(_M_X86)) \
- && !defined(__BMI2__)
- # define DYNAMIC_BMI2 1
- #else
- # define DYNAMIC_BMI2 0
- #endif
-#endif
+#define BMI2_TARGET_ATTRIBUTE TARGET_ATTRIBUTE("lzcnt,bmi,bmi2")
/* prefetch
* can be disabled, by declaring NO_PREFETCH build macro */
@@ -115,8 +105,9 @@
}
/* vectorization
- * older GCC (pre gcc-4.3 picked as the cutoff) uses a different syntax */
-#if !defined(__INTEL_COMPILER) && !defined(__clang__) && defined(__GNUC__)
+ * older GCC (pre gcc-4.3 picked as the cutoff) uses a different syntax,
+ * and some compilers, like Intel ICC and MCST LCC, do not support it at all. */
+#if !defined(__INTEL_COMPILER) && !defined(__clang__) && defined(__GNUC__) && !defined(__LCC__)
# if (__GNUC__ == 4 && __GNUC_MINOR__ > 3) || (__GNUC__ >= 5)
# define DONT_VECTORIZE __attribute__((optimize("no-tree-vectorize")))
# else
@@ -134,20 +125,18 @@
#define LIKELY(x) (__builtin_expect((x), 1))
#define UNLIKELY(x) (__builtin_expect((x), 0))
+#if __has_builtin(__builtin_unreachable) || (defined(__GNUC__) && (__GNUC__ > 4 || (__GNUC__ == 4 && __GNUC_MINOR__ >= 5)))
+# define ZSTD_UNREACHABLE { assert(0), __builtin_unreachable(); }
+#else
+# define ZSTD_UNREACHABLE { assert(0); }
+#endif
+
/* disable warnings */
/*Like DYNAMIC_BMI2 but for compile time determination of BMI2 support*/
-/* compat. with non-clang compilers */
-#ifndef __has_builtin
-# define __has_builtin(x) 0
-#endif
-
-/* compat. with non-clang compilers */
-#ifndef __has_feature
-# define __has_feature(x) 0
-#endif
+/* compile time determination of SIMD support */
/* C-language Attributes are added in C23. */
#if defined(__STDC_VERSION__) && (__STDC_VERSION__ > 201710L) && defined(__has_c_attribute)
@@ -168,10 +157,28 @@
*/
#define ZSTD_FALLTHROUGH fallthrough
-/* detects whether we are being compiled under msan */
+/*-**************************************************************
+* Alignment check
+*****************************************************************/
+/* this test was initially positioned in mem.h,
+ * but this file is removed (or replaced) for linux kernel
+ * so it's now hosted in compiler.h,
+ * which remains valid for both user & kernel spaces.
+ */
-/* detects whether we are being compiled under asan */
+#ifndef ZSTD_ALIGNOF
+/* covers gcc, clang & MSVC */
+/* note : this section must come first, before C11,
+ * due to a limitation in the kernel source generator */
+# define ZSTD_ALIGNOF(T) __alignof(T)
+
+#endif /* ZSTD_ALIGNOF */
+
+/*-**************************************************************
+* Sanitizer
+*****************************************************************/
+
#endif /* ZSTD_COMPILER_H */
diff --git a/lib/zstd/common/entropy_common.c b/lib/zstd/common/entropy_common.c
index a311808..fef6705 100644
--- a/lib/zstd/common/entropy_common.c
+++ b/lib/zstd/common/entropy_common.c
@@ -15,7 +15,6 @@
/* *************************************
* Dependencies
***************************************/
-#include <linux/module.h>
#include "mem.h"
#include "error_private.h" /* ERR_*, ERROR */
#define FSE_STATIC_LINKING_ONLY /* FSE_MIN_TABLELOG */
@@ -213,7 +212,7 @@ static size_t FSE_readNCount_body_default(
}
#if DYNAMIC_BMI2
-TARGET_ATTRIBUTE("bmi2") static size_t FSE_readNCount_body_bmi2(
+BMI2_TARGET_ATTRIBUTE static size_t FSE_readNCount_body_bmi2(
short* normalizedCounter, unsigned* maxSVPtr, unsigned* tableLogPtr,
const void* headerBuffer, size_t hbSize)
{
@@ -240,7 +239,7 @@ size_t FSE_readNCount(
{
return FSE_readNCount_bmi2(normalizedCounter, maxSVPtr, tableLogPtr, headerBuffer, hbSize, /* bmi2 */ 0);
}
-EXPORT_SYMBOL_GPL(FSE_readNCount);
+
/*! HUF_readStats() :
Read compact Huffman tree, saved by HUF_writeCTable().
@@ -256,7 +255,6 @@ size_t HUF_readStats(BYTE* huffWeight, size_t hwSize, U32* rankStats,
U32 wksp[HUF_READ_STATS_WORKSPACE_SIZE_U32];
return HUF_readStats_wksp(huffWeight, hwSize, rankStats, nbSymbolsPtr, tableLogPtr, src, srcSize, wksp, sizeof(wksp), /* bmi2 */ 0);
}
-EXPORT_SYMBOL_GPL(HUF_readStats);
FORCE_INLINE_TEMPLATE size_t
HUF_readStats_body(BYTE* huffWeight, size_t hwSize, U32* rankStats,
@@ -296,7 +294,7 @@ HUF_readStats_body(BYTE* huffWeight, size_t hwSize, U32* rankStats,
ZSTD_memset(rankStats, 0, (HUF_TABLELOG_MAX + 1) * sizeof(U32));
weightTotal = 0;
{ U32 n; for (n=0; n<oSize; n++) {
- if (huffWeight[n] >= HUF_TABLELOG_MAX) return ERROR(corruption_detected);
+ if (huffWeight[n] > HUF_TABLELOG_MAX) return ERROR(corruption_detected);
rankStats[huffWeight[n]]++;
weightTotal += (1 << huffWeight[n]) >> 1;
} }
@@ -334,7 +332,7 @@ static size_t HUF_readStats_body_default(BYTE* huffWeight, size_t hwSize, U32* r
}
#if DYNAMIC_BMI2
-static TARGET_ATTRIBUTE("bmi2") size_t HUF_readStats_body_bmi2(BYTE* huffWeight, size_t hwSize, U32* rankStats,
+static BMI2_TARGET_ATTRIBUTE size_t HUF_readStats_body_bmi2(BYTE* huffWeight, size_t hwSize, U32* rankStats,
U32* nbSymbolsPtr, U32* tableLogPtr,
const void* src, size_t srcSize,
void* workSpace, size_t wkspSize)
@@ -357,4 +355,3 @@ size_t HUF_readStats_wksp(BYTE* huffWeight, size_t hwSize, U32* rankStats,
(void)bmi2;
return HUF_readStats_body_default(huffWeight, hwSize, rankStats, nbSymbolsPtr, tableLogPtr, src, srcSize, workSpace, wkspSize);
}
-EXPORT_SYMBOL_GPL(HUF_readStats_wksp);
diff --git a/lib/zstd/common/error_private.h b/lib/zstd/common/error_private.h
index d14e686..ca5101e 100644
--- a/lib/zstd/common/error_private.h
+++ b/lib/zstd/common/error_private.h
@@ -18,8 +18,10 @@
/* ****************************************
* Dependencies
******************************************/
-#include "zstd_deps.h" /* size_t */
#include <linux/zstd_errors.h> /* enum list */
+#include "compiler.h"
+#include "debug.h"
+#include "zstd_deps.h" /* size_t */
/* ****************************************
@@ -62,5 +64,82 @@ ERR_STATIC const char* ERR_getErrorName(size_t code)
return ERR_getErrorString(ERR_getErrorCode(code));
}
+/*
+ * Ignore: this is an internal helper.
+ *
+ * This is a helper function to help force C99-correctness during compilation.
+ * Under strict compilation modes, variadic macro arguments can't be empty.
+ * However, variadic function arguments can be. Using a function therefore lets
+ * us statically check that at least one (string) argument was passed,
+ * independent of the compilation flags.
+ */
+static INLINE_KEYWORD UNUSED_ATTR
+void _force_has_format_string(const char *format, ...) {
+ (void)format;
+}
+
+/*
+ * Ignore: this is an internal helper.
+ *
+ * We want to force this function invocation to be syntactically correct, but
+ * we don't want to force runtime evaluation of its arguments.
+ */
+#define _FORCE_HAS_FORMAT_STRING(...) \
+ if (0) { \
+ _force_has_format_string(__VA_ARGS__); \
+ }
+
+#define ERR_QUOTE(str) #str
+
+/*
+ * Return the specified error if the condition evaluates to true.
+ *
+ * In debug modes, prints additional information.
+ * In order to do that (particularly, printing the conditional that failed),
+ * this can't just wrap RETURN_ERROR().
+ */
+#define RETURN_ERROR_IF(cond, err, ...) \
+ if (cond) { \
+ RAWLOG(3, "%s:%d: ERROR!: check %s failed, returning %s", \
+ __FILE__, __LINE__, ERR_QUOTE(cond), ERR_QUOTE(ERROR(err))); \
+ _FORCE_HAS_FORMAT_STRING(__VA_ARGS__); \
+ RAWLOG(3, ": " __VA_ARGS__); \
+ RAWLOG(3, "\n"); \
+ return ERROR(err); \
+ }
+
+/*
+ * Unconditionally return the specified error.
+ *
+ * In debug modes, prints additional information.
+ */
+#define RETURN_ERROR(err, ...) \
+ do { \
+ RAWLOG(3, "%s:%d: ERROR!: unconditional check failed, returning %s", \
+ __FILE__, __LINE__, ERR_QUOTE(ERROR(err))); \
+ _FORCE_HAS_FORMAT_STRING(__VA_ARGS__); \
+ RAWLOG(3, ": " __VA_ARGS__); \
+ RAWLOG(3, "\n"); \
+ return ERROR(err); \
+ } while(0);
+
+/*
+ * If the provided expression evaluates to an error code, returns that error code.
+ *
+ * In debug modes, prints additional information.
+ */
+#define FORWARD_IF_ERROR(err, ...) \
+ do { \
+ size_t const err_code = (err); \
+ if (ERR_isError(err_code)) { \
+ RAWLOG(3, "%s:%d: ERROR!: forwarding error in %s: %s", \
+ __FILE__, __LINE__, ERR_QUOTE(err), ERR_getErrorName(err_code)); \
+ _FORCE_HAS_FORMAT_STRING(__VA_ARGS__); \
+ RAWLOG(3, ": " __VA_ARGS__); \
+ RAWLOG(3, "\n"); \
+ return err_code; \
+ } \
+ } while(0);
+
#endif /* ERROR_H_MODULE */
diff --git a/lib/zstd/common/fse.h b/lib/zstd/common/fse.h
index 0bb174c..4507043b 100644
--- a/lib/zstd/common/fse.h
+++ b/lib/zstd/common/fse.h
@@ -333,8 +333,9 @@ size_t FSE_buildCTable_rle (FSE_CTable* ct, unsigned char symbolValue);
/* FSE_buildCTable_wksp() :
* Same as FSE_buildCTable(), but using an externally allocated scratch buffer (`workSpace`).
* `wkspSize` must be >= `FSE_BUILD_CTABLE_WORKSPACE_SIZE_U32(maxSymbolValue, tableLog)` of `unsigned`.
+ * See FSE_buildCTable_wksp() for breakdown of workspace usage.
*/
-#define FSE_BUILD_CTABLE_WORKSPACE_SIZE_U32(maxSymbolValue, tableLog) (maxSymbolValue + 2 + (1ull << (tableLog - 2)))
+#define FSE_BUILD_CTABLE_WORKSPACE_SIZE_U32(maxSymbolValue, tableLog) (((maxSymbolValue + 2) + (1ull << (tableLog)))/2 + sizeof(U64)/sizeof(U32) /* additional 8 bytes for potential table overwrite */)
#define FSE_BUILD_CTABLE_WORKSPACE_SIZE(maxSymbolValue, tableLog) (sizeof(unsigned) * FSE_BUILD_CTABLE_WORKSPACE_SIZE_U32(maxSymbolValue, tableLog))
size_t FSE_buildCTable_wksp(FSE_CTable* ct, const short* normalizedCounter, unsigned maxSymbolValue, unsigned tableLog, void* workSpace, size_t wkspSize);
diff --git a/lib/zstd/common/fse_decompress.c b/lib/zstd/common/fse_decompress.c
index 2c8bbe3..a0d0609 100644
--- a/lib/zstd/common/fse_decompress.c
+++ b/lib/zstd/common/fse_decompress.c
@@ -365,7 +365,7 @@ static size_t FSE_decompress_wksp_body_default(void* dst, size_t dstCapacity, co
}
#if DYNAMIC_BMI2
-TARGET_ATTRIBUTE("bmi2") static size_t FSE_decompress_wksp_body_bmi2(void* dst, size_t dstCapacity, const void* cSrc, size_t cSrcSize, unsigned maxLog, void* workSpace, size_t wkspSize)
+BMI2_TARGET_ATTRIBUTE static size_t FSE_decompress_wksp_body_bmi2(void* dst, size_t dstCapacity, const void* cSrc, size_t cSrcSize, unsigned maxLog, void* workSpace, size_t wkspSize)
{
return FSE_decompress_wksp_body(dst, dstCapacity, cSrc, cSrcSize, maxLog, workSpace, wkspSize, 1);
}
diff --git a/lib/zstd/common/huf.h b/lib/zstd/common/huf.h
index 88c5586..5042ff87 100644
--- a/lib/zstd/common/huf.h
+++ b/lib/zstd/common/huf.h
@@ -86,9 +86,9 @@ HUF_PUBLIC_API size_t HUF_compress2 (void* dst, size_t dstCapacity,
/* HUF_compress4X_wksp() :
* Same as HUF_compress2(), but uses externally allocated `workSpace`.
- * `workspace` must have minimum alignment of 4, and be at least as large as HUF_WORKSPACE_SIZE */
-#define HUF_WORKSPACE_SIZE ((6 << 10) + 256)
-#define HUF_WORKSPACE_SIZE_U32 (HUF_WORKSPACE_SIZE / sizeof(U32))
+ * `workspace` must be at least as large as HUF_WORKSPACE_SIZE */
+#define HUF_WORKSPACE_SIZE ((8 << 10) + 512 /* sorting scratch space */)
+#define HUF_WORKSPACE_SIZE_U64 (HUF_WORKSPACE_SIZE / sizeof(U64))
HUF_PUBLIC_API size_t HUF_compress4X_wksp (void* dst, size_t dstCapacity,
const void* src, size_t srcSize,
unsigned maxSymbolValue, unsigned tableLog,
@@ -113,11 +113,11 @@ HUF_PUBLIC_API size_t HUF_compress4X_wksp (void* dst, size_t dstCapacity,
/* *** Constants *** */
-#define HUF_TABLELOG_MAX 12 /* max runtime value of tableLog (due to static allocation); can be modified up to HUF_ABSOLUTEMAX_TABLELOG */
+#define HUF_TABLELOG_MAX 12 /* max runtime value of tableLog (due to static allocation); can be modified up to HUF_TABLELOG_ABSOLUTEMAX */
#define HUF_TABLELOG_DEFAULT 11 /* default tableLog value when none specified */
#define HUF_SYMBOLVALUE_MAX 255
-#define HUF_TABLELOG_ABSOLUTEMAX 15 /* absolute limit of HUF_MAX_TABLELOG. Beyond that value, code does not work */
+#define HUF_TABLELOG_ABSOLUTEMAX 12 /* absolute limit of HUF_MAX_TABLELOG. Beyond that value, code does not work */
#if (HUF_TABLELOG_MAX > HUF_TABLELOG_ABSOLUTEMAX)
# error "HUF_TABLELOG_MAX is too large !"
#endif
@@ -133,15 +133,11 @@ HUF_PUBLIC_API size_t HUF_compress4X_wksp (void* dst, size_t dstCapacity,
/* static allocation of HUF's Compression Table */
/* this is a private definition, just exposed for allocation and strict aliasing purpose. never EVER access its members directly */
-struct HUF_CElt_s {
- U16 val;
- BYTE nbBits;
-}; /* typedef'd to HUF_CElt */
-typedef struct HUF_CElt_s HUF_CElt; /* consider it an incomplete type */
-#define HUF_CTABLE_SIZE_U32(maxSymbolValue) ((maxSymbolValue)+1) /* Use tables of U32, for proper alignment */
-#define HUF_CTABLE_SIZE(maxSymbolValue) (HUF_CTABLE_SIZE_U32(maxSymbolValue) * sizeof(U32))
+typedef size_t HUF_CElt; /* consider it an incomplete type */
+#define HUF_CTABLE_SIZE_ST(maxSymbolValue) ((maxSymbolValue)+2) /* Use tables of size_t, for proper alignment */
+#define HUF_CTABLE_SIZE(maxSymbolValue) (HUF_CTABLE_SIZE_ST(maxSymbolValue) * sizeof(size_t))
#define HUF_CREATE_STATIC_CTABLE(name, maxSymbolValue) \
- HUF_CElt name[HUF_CTABLE_SIZE_U32(maxSymbolValue)] /* no final ; */
+ HUF_CElt name[HUF_CTABLE_SIZE_ST(maxSymbolValue)] /* no final ; */
/* static allocation of HUF's DTable */
typedef U32 HUF_DTable;
@@ -191,6 +187,7 @@ size_t HUF_buildCTable (HUF_CElt* CTable, const unsigned* count, unsigned maxSym
size_t HUF_writeCTable (void* dst, size_t maxDstSize, const HUF_CElt* CTable, unsigned maxSymbolValue, unsigned huffLog);
size_t HUF_writeCTable_wksp(void* dst, size_t maxDstSize, const HUF_CElt* CTable, unsigned maxSymbolValue, unsigned huffLog, void* workspace, size_t workspaceSize);
size_t HUF_compress4X_usingCTable(void* dst, size_t dstSize, const void* src, size_t srcSize, const HUF_CElt* CTable);
+size_t HUF_compress4X_usingCTable_bmi2(void* dst, size_t dstSize, const void* src, size_t srcSize, const HUF_CElt* CTable, int bmi2);
size_t HUF_estimateCompressedSize(const HUF_CElt* CTable, const unsigned* count, unsigned maxSymbolValue);
int HUF_validateCTable(const HUF_CElt* CTable, const unsigned* count, unsigned maxSymbolValue);
@@ -203,12 +200,13 @@ typedef enum {
* Same as HUF_compress4X_wksp(), but considers using hufTable if *repeat != HUF_repeat_none.
* If it uses hufTable it does not modify hufTable or repeat.
* If it doesn't, it sets *repeat = HUF_repeat_none, and it sets hufTable to the table used.
- * If preferRepeat then the old table will always be used if valid. */
+ * If preferRepeat then the old table will always be used if valid.
+ * If suspectUncompressible then some sampling checks will be run to potentially skip huffman coding */
size_t HUF_compress4X_repeat(void* dst, size_t dstSize,
const void* src, size_t srcSize,
unsigned maxSymbolValue, unsigned tableLog,
void* workSpace, size_t wkspSize, /*< `workSpace` must be aligned on 4-bytes boundaries, `wkspSize` must be >= HUF_WORKSPACE_SIZE */
- HUF_CElt* hufTable, HUF_repeat* repeat, int preferRepeat, int bmi2);
+ HUF_CElt* hufTable, HUF_repeat* repeat, int preferRepeat, int bmi2, unsigned suspectUncompressible);
/* HUF_buildCTable_wksp() :
* Same as HUF_buildCTable(), but using externally allocated scratch buffer.
@@ -246,11 +244,10 @@ size_t HUF_readStats_wksp(BYTE* huffWeight, size_t hwSize,
* Loading a CTable saved with HUF_writeCTable() */
size_t HUF_readCTable (HUF_CElt* CTable, unsigned* maxSymbolValuePtr, const void* src, size_t srcSize, unsigned *hasZeroWeights);
-/* HUF_getNbBits() :
+/* HUF_getNbBitsFromCTable() :
* Read nbBits from CTable symbolTable, for symbol `symbolValue` presumed <= HUF_SYMBOLVALUE_MAX
- * Note 1 : is not inlined, as HUF_CElt definition is private
- * Note 2 : const void* used, so that it can provide a statically allocated table as argument (which uses type U32) */
-U32 HUF_getNbBits(const void* symbolTable, U32 symbolValue);
+ * Note 1 : is not inlined, as HUF_CElt definition is private */
+U32 HUF_getNbBitsFromCTable(const HUF_CElt* symbolTable, U32 symbolValue);
/*
* HUF_decompress() does the following:
@@ -302,18 +299,20 @@ size_t HUF_decompress4X2_usingDTable(void* dst, size_t maxDstSize, const void* c
/* ====================== */
size_t HUF_compress1X (void* dst, size_t dstSize, const void* src, size_t srcSize, unsigned maxSymbolValue, unsigned tableLog);
-size_t HUF_compress1X_wksp (void* dst, size_t dstSize, const void* src, size_t srcSize, unsigned maxSymbolValue, unsigned tableLog, void* workSpace, size_t wkspSize); /*< `workSpace` must be a table of at least HUF_WORKSPACE_SIZE_U32 unsigned */
+size_t HUF_compress1X_wksp (void* dst, size_t dstSize, const void* src, size_t srcSize, unsigned maxSymbolValue, unsigned tableLog, void* workSpace, size_t wkspSize); /*< `workSpace` must be a table of at least HUF_WORKSPACE_SIZE_U64 U64 */
size_t HUF_compress1X_usingCTable(void* dst, size_t dstSize, const void* src, size_t srcSize, const HUF_CElt* CTable);
+size_t HUF_compress1X_usingCTable_bmi2(void* dst, size_t dstSize, const void* src, size_t srcSize, const HUF_CElt* CTable, int bmi2);
/* HUF_compress1X_repeat() :
* Same as HUF_compress1X_wksp(), but considers using hufTable if *repeat != HUF_repeat_none.
* If it uses hufTable it does not modify hufTable or repeat.
* If it doesn't, it sets *repeat = HUF_repeat_none, and it sets hufTable to the table used.
- * If preferRepeat then the old table will always be used if valid. */
+ * If preferRepeat then the old table will always be used if valid.
+ * If suspectUncompressible then some sampling checks will be run to potentially skip huffman coding */
size_t HUF_compress1X_repeat(void* dst, size_t dstSize,
const void* src, size_t srcSize,
unsigned maxSymbolValue, unsigned tableLog,
void* workSpace, size_t wkspSize, /*< `workSpace` must be aligned on 4-bytes boundaries, `wkspSize` must be >= HUF_WORKSPACE_SIZE */
- HUF_CElt* hufTable, HUF_repeat* repeat, int preferRepeat, int bmi2);
+ HUF_CElt* hufTable, HUF_repeat* repeat, int preferRepeat, int bmi2, unsigned suspectUncompressible);
size_t HUF_decompress1X1 (void* dst, size_t dstSize, const void* cSrc, size_t cSrcSize); /* single-symbol decoder */
#ifndef HUF_FORCE_DECOMPRESS_X1
@@ -351,6 +350,9 @@ size_t HUF_decompress4X_hufOnly_wksp_bmi2(HUF_DTable* dctx, void* dst, size_t ds
#ifndef HUF_FORCE_DECOMPRESS_X2
size_t HUF_readDTableX1_wksp_bmi2(HUF_DTable* DTable, const void* src, size_t srcSize, void* workSpace, size_t wkspSize, int bmi2);
#endif
+#ifndef HUF_FORCE_DECOMPRESS_X1
+size_t HUF_readDTableX2_wksp_bmi2(HUF_DTable* DTable, const void* src, size_t srcSize, void* workSpace, size_t wkspSize, int bmi2);
+#endif
#endif /* HUF_STATIC_LINKING_ONLY */
diff --git a/lib/zstd/common/mem.h b/lib/zstd/common/mem.h
index dcdd586..1d9cc03 100644
--- a/lib/zstd/common/mem.h
+++ b/lib/zstd/common/mem.h
@@ -30,6 +30,8 @@
* Basic Types
*****************************************************************/
typedef uint8_t BYTE;
+typedef uint8_t U8;
+typedef int8_t S8;
typedef uint16_t U16;
typedef int16_t S16;
typedef uint32_t U32;
diff --git a/lib/zstd/common/portability_macros.h b/lib/zstd/common/portability_macros.h
new file mode 100644
index 0000000..0e3b2c0
--- /dev/null
+++ b/lib/zstd/common/portability_macros.h
@@ -0,0 +1,93 @@
+/*
+ * Copyright (c) Facebook, Inc.
+ * All rights reserved.
+ *
+ * This source code is licensed under both the BSD-style license (found in the
+ * LICENSE file in the root directory of this source tree) and the GPLv2 (found
+ * in the COPYING file in the root directory of this source tree).
+ * You may select, at your option, one of the above-listed licenses.
+ */
+
+#ifndef ZSTD_PORTABILITY_MACROS_H
+#define ZSTD_PORTABILITY_MACROS_H
+
+/*
+ * This header file contains macro defintions to support portability.
+ * This header is shared between C and ASM code, so it MUST only
+ * contain macro definitions. It MUST not contain any C code.
+ *
+ * This header ONLY defines macros to detect platforms/feature support.
+ *
+ */
+
+
+/* compat. with non-clang compilers */
+#ifndef __has_attribute
+ #define __has_attribute(x) 0
+#endif
+
+/* compat. with non-clang compilers */
+#ifndef __has_builtin
+# define __has_builtin(x) 0
+#endif
+
+/* compat. with non-clang compilers */
+#ifndef __has_feature
+# define __has_feature(x) 0
+#endif
+
+/* detects whether we are being compiled under msan */
+
+/* detects whether we are being compiled under asan */
+
+/* detects whether we are being compiled under dfsan */
+
+/* Mark the internal assembly functions as hidden */
+#ifdef __ELF__
+# define ZSTD_HIDE_ASM_FUNCTION(func) .hidden func
+#else
+# define ZSTD_HIDE_ASM_FUNCTION(func)
+#endif
+
+/* Enable runtime BMI2 dispatch based on the CPU.
+ * Enabled for clang & gcc >=4.8 on x86 when BMI2 isn't enabled by default.
+ */
+#ifndef DYNAMIC_BMI2
+ #if ((defined(__clang__) && __has_attribute(__target__)) \
+ || (defined(__GNUC__) \
+ && (__GNUC__ >= 5 || (__GNUC__ == 4 && __GNUC_MINOR__ >= 8)))) \
+ && (defined(__x86_64__) || defined(_M_X64)) \
+ && !defined(__BMI2__)
+ # define DYNAMIC_BMI2 1
+ #else
+ # define DYNAMIC_BMI2 0
+ #endif
+#endif
+
+/*
+ * Only enable assembly for GNUC comptabile compilers,
+ * because other platforms may not support GAS assembly syntax.
+ *
+ * Only enable assembly for Linux / MacOS, other platforms may
+ * work, but they haven't been tested. This could likely be
+ * extended to BSD systems.
+ *
+ * Disable assembly when MSAN is enabled, because MSAN requires
+ * 100% of code to be instrumented to work.
+ */
+#define ZSTD_ASM_SUPPORTED 1
+
+/*
+ * Determines whether we should enable assembly for x86-64
+ * with BMI2.
+ *
+ * Enable if all of the following conditions hold:
+ * - ASM hasn't been explicitly disabled by defining ZSTD_DISABLE_ASM
+ * - Assembly is supported
+ * - We are compiling for x86-64 and either:
+ * - DYNAMIC_BMI2 is enabled
+ * - BMI2 is supported at compile time
+ */
+#define ZSTD_ENABLE_ASM_X86_64_BMI2 0
+
+#endif /* ZSTD_PORTABILITY_MACROS_H */
diff --git a/lib/zstd/common/zstd_common.c b/lib/zstd/common/zstd_common.c
index 0f1f63b..3d7e35b 100644
--- a/lib/zstd/common/zstd_common.c
+++ b/lib/zstd/common/zstd_common.c
@@ -13,7 +13,6 @@
/*-*************************************
* Dependencies
***************************************/
-#include <linux/module.h>
#define ZSTD_DEPS_NEED_MALLOC
#include "zstd_deps.h" /* ZSTD_malloc, ZSTD_calloc, ZSTD_free, ZSTD_memset */
#include "error_private.h"
@@ -36,17 +35,14 @@ const char* ZSTD_versionString(void) { return ZSTD_VERSION_STRING; }
* tells if a return value is an error code
* symbol is required for external callers */
unsigned ZSTD_isError(size_t code) { return ERR_isError(code); }
-EXPORT_SYMBOL_GPL(ZSTD_isError);
/*! ZSTD_getErrorName() :
* provides error code string from function result (useful for debugging) */
const char* ZSTD_getErrorName(size_t code) { return ERR_getErrorName(code); }
-EXPORT_SYMBOL_GPL(ZSTD_getErrorName);
/*! ZSTD_getError() :
* convert a `size_t` function result into a proper ZSTD_errorCode enum */
ZSTD_ErrorCode ZSTD_getErrorCode(size_t code) { return ERR_getErrorCode(code); }
-EXPORT_SYMBOL_GPL(ZSTD_getErrorCode);
/*! ZSTD_getErrorString() :
* provides error code string from enum */
@@ -63,7 +59,6 @@ void* ZSTD_customMalloc(size_t size, ZSTD_customMem customMem)
return customMem.customAlloc(customMem.opaque, size);
return ZSTD_malloc(size);
}
-EXPORT_SYMBOL_GPL(ZSTD_customMalloc);
void* ZSTD_customCalloc(size_t size, ZSTD_customMem customMem)
{
@@ -76,7 +71,6 @@ void* ZSTD_customCalloc(size_t size, ZSTD_customMem customMem)
}
return ZSTD_calloc(1, size);
}
-EXPORT_SYMBOL_GPL(ZSTD_customCalloc);
void ZSTD_customFree(void* ptr, ZSTD_customMem customMem)
{
@@ -87,7 +81,3 @@ void ZSTD_customFree(void* ptr, ZSTD_customMem customMem)
ZSTD_free(ptr);
}
}
-EXPORT_SYMBOL_GPL(ZSTD_customFree);
-
-MODULE_LICENSE("Dual BSD/GPL");
-MODULE_DESCRIPTION("Zstd Common");
diff --git a/lib/zstd/common/zstd_internal.h b/lib/zstd/common/zstd_internal.h
index fc6f3a9..93305d9 100644
--- a/lib/zstd/common/zstd_internal.h
+++ b/lib/zstd/common/zstd_internal.h
@@ -20,6 +20,7 @@
* Dependencies
***************************************/
#include "compiler.h"
+#include "cpu.h"
#include "mem.h"
#include "debug.h" /* assert, DEBUGLOG, RAWLOG, g_debuglevel */
#include "error_private.h"
@@ -47,81 +48,7 @@
#undef MAX
#define MIN(a,b) ((a)<(b) ? (a) : (b))
#define MAX(a,b) ((a)>(b) ? (a) : (b))
-
-/*
- * Ignore: this is an internal helper.
- *
- * This is a helper function to help force C99-correctness during compilation.
- * Under strict compilation modes, variadic macro arguments can't be empty.
- * However, variadic function arguments can be. Using a function therefore lets
- * us statically check that at least one (string) argument was passed,
- * independent of the compilation flags.
- */
-static INLINE_KEYWORD UNUSED_ATTR
-void _force_has_format_string(const char *format, ...) {
- (void)format;
-}
-
-/*
- * Ignore: this is an internal helper.
- *
- * We want to force this function invocation to be syntactically correct, but
- * we don't want to force runtime evaluation of its arguments.
- */
-#define _FORCE_HAS_FORMAT_STRING(...) \
- if (0) { \
- _force_has_format_string(__VA_ARGS__); \
- }
-
-/*
- * Return the specified error if the condition evaluates to true.
- *
- * In debug modes, prints additional information.
- * In order to do that (particularly, printing the conditional that failed),
- * this can't just wrap RETURN_ERROR().
- */
-#define RETURN_ERROR_IF(cond, err, ...) \
- if (cond) { \
- RAWLOG(3, "%s:%d: ERROR!: check %s failed, returning %s", \
- __FILE__, __LINE__, ZSTD_QUOTE(cond), ZSTD_QUOTE(ERROR(err))); \
- _FORCE_HAS_FORMAT_STRING(__VA_ARGS__); \
- RAWLOG(3, ": " __VA_ARGS__); \
- RAWLOG(3, "\n"); \
- return ERROR(err); \
- }
-
-/*
- * Unconditionally return the specified error.
- *
- * In debug modes, prints additional information.
- */
-#define RETURN_ERROR(err, ...) \
- do { \
- RAWLOG(3, "%s:%d: ERROR!: unconditional check failed, returning %s", \
- __FILE__, __LINE__, ZSTD_QUOTE(ERROR(err))); \
- _FORCE_HAS_FORMAT_STRING(__VA_ARGS__); \
- RAWLOG(3, ": " __VA_ARGS__); \
- RAWLOG(3, "\n"); \
- return ERROR(err); \
- } while(0);
-
-/*
- * If the provided expression evaluates to an error code, returns that error code.
- *
- * In debug modes, prints additional information.
- */
-#define FORWARD_IF_ERROR(err, ...) \
- do { \
- size_t const err_code = (err); \
- if (ERR_isError(err_code)) { \
- RAWLOG(3, "%s:%d: ERROR!: forwarding error in %s: %s", \
- __FILE__, __LINE__, ZSTD_QUOTE(err), ERR_getErrorName(err_code)); \
- _FORCE_HAS_FORMAT_STRING(__VA_ARGS__); \
- RAWLOG(3, ": " __VA_ARGS__); \
- RAWLOG(3, "\n"); \
- return err_code; \
- } \
- } while(0);
+#define BOUNDED(min,val,max) (MAX(min,MIN(val,max)))
/*-*************************************
@@ -130,7 +57,6 @@ void _force_has_format_string(const char *format, ...) {
#define ZSTD_OPT_NUM (1<<12)
#define ZSTD_REP_NUM 3 /* number of repcodes */
-#define ZSTD_REP_MOVE (ZSTD_REP_NUM-1)
static UNUSED_ATTR const U32 repStartValue[ZSTD_REP_NUM] = { 1, 4, 8 };
#define KB *(1 <<10)
@@ -182,7 +108,7 @@ typedef enum { set_basic, set_rle, set_compressed, set_repeat } symbolEncodingTy
/* Each table cannot take more than #symbols * FSELog bits */
#define ZSTD_MAX_FSE_HEADERS_SIZE (((MaxML + 1) * MLFSELog + (MaxLL + 1) * LLFSELog + (MaxOff + 1) * OffFSELog + 7) / 8)
-static UNUSED_ATTR const U32 LL_bits[MaxLL+1] = {
+static UNUSED_ATTR const U8 LL_bits[MaxLL+1] = {
0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0,
1, 1, 1, 1, 2, 2, 3, 3,
@@ -199,7 +125,7 @@ static UNUSED_ATTR const S16 LL_defaultNorm[MaxLL+1] = {
#define LL_DEFAULTNORMLOG 6 /* for static allocation */
static UNUSED_ATTR const U32 LL_defaultNormLog = LL_DEFAULTNORMLOG;
-static UNUSED_ATTR const U32 ML_bits[MaxML+1] = {
+static UNUSED_ATTR const U8 ML_bits[MaxML+1] = {
0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0,
@@ -234,12 +160,31 @@ static UNUSED_ATTR const U32 OF_defaultNormLog = OF_DEFAULTNORMLOG;
* Shared functions to include for inlining
*********************************************/
static void ZSTD_copy8(void* dst, const void* src) {
+#if defined(ZSTD_ARCH_ARM_NEON)
+ vst1_u8((uint8_t*)dst, vld1_u8((const uint8_t*)src));
+#else
ZSTD_memcpy(dst, src, 8);
+#endif
}
-
#define COPY8(d,s) { ZSTD_copy8(d,s); d+=8; s+=8; }
+
+/* Need to use memmove here since the literal buffer can now be located within
+ the dst buffer. In circumstances where the op "catches up" to where the
+ literal buffer is, there can be partial overlaps in this call on the final
+ copy if the literal is being shifted by less than 16 bytes. */
static void ZSTD_copy16(void* dst, const void* src) {
- ZSTD_memcpy(dst, src, 16);
+#if defined(ZSTD_ARCH_ARM_NEON)
+ vst1q_u8((uint8_t*)dst, vld1q_u8((const uint8_t*)src));
+#elif defined(ZSTD_ARCH_X86_SSE2)
+ _mm_storeu_si128((__m128i*)dst, _mm_loadu_si128((const __m128i*)src));
+#elif defined(__clang__)
+ ZSTD_memmove(dst, src, 16);
+#else
+ /* ZSTD_memmove is not inlined properly by gcc */
+ BYTE copy16_buf[16];
+ ZSTD_memcpy(copy16_buf, src, 16);
+ ZSTD_memcpy(dst, copy16_buf, 16);
+#endif
}
#define COPY16(d,s) { ZSTD_copy16(d,s); d+=16; s+=16; }
@@ -267,8 +212,6 @@ void ZSTD_wildcopy(void* dst, const void* src, ptrdiff_t length, ZSTD_overlap_e
BYTE* op = (BYTE*)dst;
BYTE* const oend = op + length;
- assert(diff >= 8 || (ovtype == ZSTD_no_overlap && diff <= -WILDCOPY_VECLEN));
-
if (ovtype == ZSTD_overlap_src_before_dst && diff < WILDCOPY_VECLEN) {
/* Handle short offset copies. */
do {
@@ -331,11 +274,18 @@ typedef enum {
* Private declarations
*********************************************/
typedef struct seqDef_s {
- U32 offset; /* Offset code of the sequence */
+ U32 offBase; /* offBase == Offset + ZSTD_REP_NUM, or repcode 1,2,3 */
U16 litLength;
- U16 matchLength;
+ U16 mlBase; /* mlBase == matchLength - MINMATCH */
} seqDef;
+/* Controls whether seqStore has a single "long" litLength or matchLength. See seqStore_t. */
+typedef enum {
+ ZSTD_llt_none = 0, /* no longLengthType */
+ ZSTD_llt_literalLength = 1, /* represents a long literal */
+ ZSTD_llt_matchLength = 2 /* represents a long match */
+} ZSTD_longLengthType_e;
+
typedef struct {
seqDef* sequencesStart;
seqDef* sequences; /* ptr to end of sequences */
@@ -347,12 +297,12 @@ typedef struct {
size_t maxNbSeq;
size_t maxNbLit;
- /* longLengthPos and longLengthID to allow us to represent either a single litLength or matchLength
+ /* longLengthPos and longLengthType to allow us to represent either a single litLength or matchLength
* in the seqStore that has a value larger than U16 (if it exists). To do so, we increment
* the existing value of the litLength or matchLength by 0x10000.
*/
- U32 longLengthID; /* 0 == no longLength; 1 == Represent the long literal; 2 == Represent the long match; */
- U32 longLengthPos; /* Index of the sequence to apply long length modification to */
+ ZSTD_longLengthType_e longLengthType;
+ U32 longLengthPos; /* Index of the sequence to apply long length modification to */
} seqStore_t;
typedef struct {
@@ -362,18 +312,18 @@ typedef struct {
/*
* Returns the ZSTD_sequenceLength for the given sequences. It handles the decoding of long sequences
- * indicated by longLengthPos and longLengthID, and adds MINMATCH back to matchLength.
+ * indicated by longLengthPos and longLengthType, and adds MINMATCH back to matchLength.
*/
MEM_STATIC ZSTD_sequenceLength ZSTD_getSequenceLength(seqStore_t const* seqStore, seqDef const* seq)
{
ZSTD_sequenceLength seqLen;
seqLen.litLength = seq->litLength;
- seqLen.matchLength = seq->matchLength + MINMATCH;
+ seqLen.matchLength = seq->mlBase + MINMATCH;
if (seqStore->longLengthPos == (U32)(seq - seqStore->sequencesStart)) {
- if (seqStore->longLengthID == 1) {
+ if (seqStore->longLengthType == ZSTD_llt_literalLength) {
seqLen.litLength += 0xFFFF;
}
- if (seqStore->longLengthID == 2) {
+ if (seqStore->longLengthType == ZSTD_llt_matchLength) {
seqLen.matchLength += 0xFFFF;
}
}
@@ -419,6 +369,41 @@ MEM_STATIC U32 ZSTD_highbit32(U32 val) /* compress, dictBuilder, decodeCorpus
}
}
+/*
+ * Counts the number of trailing zeros of a `size_t`.
+ * Most compilers should support CTZ as a builtin. A backup
+ * implementation is provided if the builtin isn't supported, but
+ * it may not be terribly efficient.
+ */
+MEM_STATIC unsigned ZSTD_countTrailingZeros(size_t val)
+{
+ if (MEM_64bits()) {
+# if (__GNUC__ >= 4)
+ return __builtin_ctzll((U64)val);
+# else
+ static const int DeBruijnBytePos[64] = { 0, 1, 2, 7, 3, 13, 8, 19,
+ 4, 25, 14, 28, 9, 34, 20, 56,
+ 5, 17, 26, 54, 15, 41, 29, 43,
+ 10, 31, 38, 35, 21, 45, 49, 57,
+ 63, 6, 12, 18, 24, 27, 33, 55,
+ 16, 53, 40, 42, 30, 37, 44, 48,
+ 62, 11, 23, 32, 52, 39, 36, 47,
+ 61, 22, 51, 46, 60, 50, 59, 58 };
+ return DeBruijnBytePos[((U64)((val & -(long long)val) * 0x0218A392CDABBD3FULL)) >> 58];
+# endif
+ } else { /* 32 bits */
+# if (__GNUC__ >= 3)
+ return __builtin_ctz((U32)val);
+# else
+ static const int DeBruijnBytePos[32] = { 0, 1, 28, 2, 29, 14, 24, 3,
+ 30, 22, 20, 15, 25, 17, 4, 8,
+ 31, 27, 13, 23, 21, 19, 16, 7,
+ 26, 12, 18, 6, 11, 5, 10, 9 };
+ return DeBruijnBytePos[((U32)((val & -(S32)val) * 0x077CB531U)) >> 27];
+# endif
+ }
+}
+
/* ZSTD_invalidateRepCodes() :
* ensures next compression will not use repcodes from previous block.
@@ -445,6 +430,14 @@ size_t ZSTD_getcBlockSize(const void* src, size_t srcSize,
size_t ZSTD_decodeSeqHeaders(ZSTD_DCtx* dctx, int* nbSeqPtr,
const void* src, size_t srcSize);
+/*
+ * @returns true iff the CPU supports dynamic BMI2 dispatch.
+ */
+MEM_STATIC int ZSTD_cpuSupportsBmi2(void)
+{
+ ZSTD_cpuid_t cpuid = ZSTD_cpuid();
+ return ZSTD_cpuid_bmi1(cpuid) && ZSTD_cpuid_bmi2(cpuid);
+}
#endif /* ZSTD_CCOMMON_H_MODULE */
diff --git a/lib/zstd/compress/clevels.h b/lib/zstd/compress/clevels.h
new file mode 100644
index 0000000..d9a7611
--- /dev/null
+++ b/lib/zstd/compress/clevels.h
@@ -0,0 +1,132 @@
+/*
+ * Copyright (c) Yann Collet, Facebook, Inc.
+ * All rights reserved.
+ *
+ * This source code is licensed under both the BSD-style license (found in the
+ * LICENSE file in the root directory of this source tree) and the GPLv2 (found
+ * in the COPYING file in the root directory of this source tree).
+ * You may select, at your option, one of the above-listed licenses.
+ */
+
+#ifndef ZSTD_CLEVELS_H
+#define ZSTD_CLEVELS_H
+
+#define ZSTD_STATIC_LINKING_ONLY /* ZSTD_compressionParameters */
+#include <linux/zstd.h>
+
+/*-===== Pre-defined compression levels =====-*/
+
+#define ZSTD_MAX_CLEVEL 22
+
+__attribute__((__unused__))
+
+static const ZSTD_compressionParameters ZSTD_defaultCParameters[4][ZSTD_MAX_CLEVEL+1] = {
+{ /* "default" - for any srcSize > 256 KB */
+ /* W, C, H, S, L, TL, strat */
+ { 19, 12, 13, 1, 6, 1, ZSTD_fast }, /* base for negative levels */
+ { 19, 13, 14, 1, 7, 0, ZSTD_fast }, /* level 1 */
+ { 20, 15, 16, 1, 6, 0, ZSTD_fast }, /* level 2 */
+ { 21, 16, 17, 1, 5, 0, ZSTD_dfast }, /* level 3 */
+ { 21, 18, 18, 1, 5, 0, ZSTD_dfast }, /* level 4 */
+ { 21, 18, 19, 3, 5, 2, ZSTD_greedy }, /* level 5 */
+ { 21, 18, 19, 3, 5, 4, ZSTD_lazy }, /* level 6 */
+ { 21, 19, 20, 4, 5, 8, ZSTD_lazy }, /* level 7 */
+ { 21, 19, 20, 4, 5, 16, ZSTD_lazy2 }, /* level 8 */
+ { 22, 20, 21, 4, 5, 16, ZSTD_lazy2 }, /* level 9 */
+ { 22, 21, 22, 5, 5, 16, ZSTD_lazy2 }, /* level 10 */
+ { 22, 21, 22, 6, 5, 16, ZSTD_lazy2 }, /* level 11 */
+ { 22, 22, 23, 6, 5, 32, ZSTD_lazy2 }, /* level 12 */
+ { 22, 22, 22, 4, 5, 32, ZSTD_btlazy2 }, /* level 13 */
+ { 22, 22, 23, 5, 5, 32, ZSTD_btlazy2 }, /* level 14 */
+ { 22, 23, 23, 6, 5, 32, ZSTD_btlazy2 }, /* level 15 */
+ { 22, 22, 22, 5, 5, 48, ZSTD_btopt }, /* level 16 */
+ { 23, 23, 22, 5, 4, 64, ZSTD_btopt }, /* level 17 */
+ { 23, 23, 22, 6, 3, 64, ZSTD_btultra }, /* level 18 */
+ { 23, 24, 22, 7, 3,256, ZSTD_btultra2}, /* level 19 */
+ { 25, 25, 23, 7, 3,256, ZSTD_btultra2}, /* level 20 */
+ { 26, 26, 24, 7, 3,512, ZSTD_btultra2}, /* level 21 */
+ { 27, 27, 25, 9, 3,999, ZSTD_btultra2}, /* level 22 */
+},
+{ /* for srcSize <= 256 KB */
+ /* W, C, H, S, L, T, strat */
+ { 18, 12, 13, 1, 5, 1, ZSTD_fast }, /* base for negative levels */
+ { 18, 13, 14, 1, 6, 0, ZSTD_fast }, /* level 1 */
+ { 18, 14, 14, 1, 5, 0, ZSTD_dfast }, /* level 2 */
+ { 18, 16, 16, 1, 4, 0, ZSTD_dfast }, /* level 3 */
+ { 18, 16, 17, 3, 5, 2, ZSTD_greedy }, /* level 4.*/
+ { 18, 17, 18, 5, 5, 2, ZSTD_greedy }, /* level 5.*/
+ { 18, 18, 19, 3, 5, 4, ZSTD_lazy }, /* level 6.*/
+ { 18, 18, 19, 4, 4, 4, ZSTD_lazy }, /* level 7 */
+ { 18, 18, 19, 4, 4, 8, ZSTD_lazy2 }, /* level 8 */
+ { 18, 18, 19, 5, 4, 8, ZSTD_lazy2 }, /* level 9 */
+ { 18, 18, 19, 6, 4, 8, ZSTD_lazy2 }, /* level 10 */
+ { 18, 18, 19, 5, 4, 12, ZSTD_btlazy2 }, /* level 11.*/
+ { 18, 19, 19, 7, 4, 12, ZSTD_btlazy2 }, /* level 12.*/
+ { 18, 18, 19, 4, 4, 16, ZSTD_btopt }, /* level 13 */
+ { 18, 18, 19, 4, 3, 32, ZSTD_btopt }, /* level 14.*/
+ { 18, 18, 19, 6, 3,128, ZSTD_btopt }, /* level 15.*/
+ { 18, 19, 19, 6, 3,128, ZSTD_btultra }, /* level 16.*/
+ { 18, 19, 19, 8, 3,256, ZSTD_btultra }, /* level 17.*/
+ { 18, 19, 19, 6, 3,128, ZSTD_btultra2}, /* level 18.*/
+ { 18, 19, 19, 8, 3,256, ZSTD_btultra2}, /* level 19.*/
+ { 18, 19, 19, 10, 3,512, ZSTD_btultra2}, /* level 20.*/
+ { 18, 19, 19, 12, 3,512, ZSTD_btultra2}, /* level 21.*/
+ { 18, 19, 19, 13, 3,999, ZSTD_btultra2}, /* level 22.*/
+},
+{ /* for srcSize <= 128 KB */
+ /* W, C, H, S, L, T, strat */
+ { 17, 12, 12, 1, 5, 1, ZSTD_fast }, /* base for negative levels */
+ { 17, 12, 13, 1, 6, 0, ZSTD_fast }, /* level 1 */
+ { 17, 13, 15, 1, 5, 0, ZSTD_fast }, /* level 2 */
+ { 17, 15, 16, 2, 5, 0, ZSTD_dfast }, /* level 3 */
+ { 17, 17, 17, 2, 4, 0, ZSTD_dfast }, /* level 4 */
+ { 17, 16, 17, 3, 4, 2, ZSTD_greedy }, /* level 5 */
+ { 17, 16, 17, 3, 4, 4, ZSTD_lazy }, /* level 6 */
+ { 17, 16, 17, 3, 4, 8, ZSTD_lazy2 }, /* level 7 */
+ { 17, 16, 17, 4, 4, 8, ZSTD_lazy2 }, /* level 8 */
+ { 17, 16, 17, 5, 4, 8, ZSTD_lazy2 }, /* level 9 */
+ { 17, 16, 17, 6, 4, 8, ZSTD_lazy2 }, /* level 10 */
+ { 17, 17, 17, 5, 4, 8, ZSTD_btlazy2 }, /* level 11 */
+ { 17, 18, 17, 7, 4, 12, ZSTD_btlazy2 }, /* level 12 */
+ { 17, 18, 17, 3, 4, 12, ZSTD_btopt }, /* level 13.*/
+ { 17, 18, 17, 4, 3, 32, ZSTD_btopt }, /* level 14.*/
+ { 17, 18, 17, 6, 3,256, ZSTD_btopt }, /* level 15.*/
+ { 17, 18, 17, 6, 3,128, ZSTD_btultra }, /* level 16.*/
+ { 17, 18, 17, 8, 3,256, ZSTD_btultra }, /* level 17.*/
+ { 17, 18, 17, 10, 3,512, ZSTD_btultra }, /* level 18.*/
+ { 17, 18, 17, 5, 3,256, ZSTD_btultra2}, /* level 19.*/
+ { 17, 18, 17, 7, 3,512, ZSTD_btultra2}, /* level 20.*/
+ { 17, 18, 17, 9, 3,512, ZSTD_btultra2}, /* level 21.*/
+ { 17, 18, 17, 11, 3,999, ZSTD_btultra2}, /* level 22.*/
+},
+{ /* for srcSize <= 16 KB */
+ /* W, C, H, S, L, T, strat */
+ { 14, 12, 13, 1, 5, 1, ZSTD_fast }, /* base for negative levels */
+ { 14, 14, 15, 1, 5, 0, ZSTD_fast }, /* level 1 */
+ { 14, 14, 15, 1, 4, 0, ZSTD_fast }, /* level 2 */
+ { 14, 14, 15, 2, 4, 0, ZSTD_dfast }, /* level 3 */
+ { 14, 14, 14, 4, 4, 2, ZSTD_greedy }, /* level 4 */
+ { 14, 14, 14, 3, 4, 4, ZSTD_lazy }, /* level 5.*/
+ { 14, 14, 14, 4, 4, 8, ZSTD_lazy2 }, /* level 6 */
+ { 14, 14, 14, 6, 4, 8, ZSTD_lazy2 }, /* level 7 */
+ { 14, 14, 14, 8, 4, 8, ZSTD_lazy2 }, /* level 8.*/
+ { 14, 15, 14, 5, 4, 8, ZSTD_btlazy2 }, /* level 9.*/
+ { 14, 15, 14, 9, 4, 8, ZSTD_btlazy2 }, /* level 10.*/
+ { 14, 15, 14, 3, 4, 12, ZSTD_btopt }, /* level 11.*/
+ { 14, 15, 14, 4, 3, 24, ZSTD_btopt }, /* level 12.*/
+ { 14, 15, 14, 5, 3, 32, ZSTD_btultra }, /* level 13.*/
+ { 14, 15, 15, 6, 3, 64, ZSTD_btultra }, /* level 14.*/
+ { 14, 15, 15, 7, 3,256, ZSTD_btultra }, /* level 15.*/
+ { 14, 15, 15, 5, 3, 48, ZSTD_btultra2}, /* level 16.*/
+ { 14, 15, 15, 6, 3,128, ZSTD_btultra2}, /* level 17.*/
+ { 14, 15, 15, 7, 3,256, ZSTD_btultra2}, /* level 18.*/
+ { 14, 15, 15, 8, 3,256, ZSTD_btultra2}, /* level 19.*/
+ { 14, 15, 15, 8, 3,512, ZSTD_btultra2}, /* level 20.*/
+ { 14, 15, 15, 9, 3,512, ZSTD_btultra2}, /* level 21.*/
+ { 14, 15, 15, 10, 3,999, ZSTD_btultra2}, /* level 22.*/
+},
+};
+
+
+
+#endif /* ZSTD_CLEVELS_H */
diff --git a/lib/zstd/compress/fse_compress.c b/lib/zstd/compress/fse_compress.c
index 436985b..ec5b1ca 100644
--- a/lib/zstd/compress/fse_compress.c
+++ b/lib/zstd/compress/fse_compress.c
@@ -75,13 +75,14 @@ size_t FSE_buildCTable_wksp(FSE_CTable* ct,
void* const FSCT = ((U32*)ptr) + 1 /* header */ + (tableLog ? tableSize>>1 : 1) ;
FSE_symbolCompressionTransform* const symbolTT = (FSE_symbolCompressionTransform*) (FSCT);
U32 const step = FSE_TABLESTEP(tableSize);
+ U32 const maxSV1 = maxSymbolValue+1;
- U32* cumul = (U32*)workSpace;
- FSE_FUNCTION_TYPE* tableSymbol = (FSE_FUNCTION_TYPE*)(cumul + (maxSymbolValue + 2));
+ U16* cumul = (U16*)workSpace; /* size = maxSV1 */
+ FSE_FUNCTION_TYPE* const tableSymbol = (FSE_FUNCTION_TYPE*)(cumul + (maxSV1+1)); /* size = tableSize */
U32 highThreshold = tableSize-1;
- if ((size_t)workSpace & 3) return ERROR(GENERIC); /* Must be 4 byte aligned */
+ assert(((size_t)workSpace & 1) == 0); /* Must be 2 bytes-aligned */
if (FSE_BUILD_CTABLE_WORKSPACE_SIZE(maxSymbolValue, tableLog) > wkspSize) return ERROR(tableLog_tooLarge);
/* CTable header */
tableU16[-2] = (U16) tableLog;
@@ -98,20 +99,61 @@ size_t FSE_buildCTable_wksp(FSE_CTable* ct,
/* symbol start positions */
{ U32 u;
cumul[0] = 0;
- for (u=1; u <= maxSymbolValue+1; u++) {
+ for (u=1; u <= maxSV1; u++) {
if (normalizedCounter[u-1]==-1) { /* Low proba symbol */
cumul[u] = cumul[u-1] + 1;
tableSymbol[highThreshold--] = (FSE_FUNCTION_TYPE)(u-1);
} else {
- cumul[u] = cumul[u-1] + normalizedCounter[u-1];
+ assert(normalizedCounter[u-1] >= 0);
+ cumul[u] = cumul[u-1] + (U16)normalizedCounter[u-1];
+ assert(cumul[u] >= cumul[u-1]); /* no overflow */
} }
- cumul[maxSymbolValue+1] = tableSize+1;
+ cumul[maxSV1] = (U16)(tableSize+1);
}
/* Spread symbols */
- { U32 position = 0;
+ if (highThreshold == tableSize - 1) {
+ /* Case for no low prob count symbols. Lay down 8 bytes at a time
+ * to reduce branch misses since we are operating on a small block
+ */
+ BYTE* const spread = tableSymbol + tableSize; /* size = tableSize + 8 (may write beyond tableSize) */
+ { U64 const add = 0x0101010101010101ull;
+ size_t pos = 0;
+ U64 sv = 0;
+ U32 s;
+ for (s=0; s<maxSV1; ++s, sv += add) {
+ int i;
+ int const n = normalizedCounter[s];
+ MEM_write64(spread + pos, sv);
+ for (i = 8; i < n; i += 8) {
+ MEM_write64(spread + pos + i, sv);
+ }
+ assert(n>=0);
+ pos += (size_t)n;
+ }
+ }
+ /* Spread symbols across the table. Lack of lowprob symbols means that
+ * we don't need variable sized inner loop, so we can unroll the loop and
+ * reduce branch misses.
+ */
+ { size_t position = 0;
+ size_t s;
+ size_t const unroll = 2; /* Experimentally determined optimal unroll */
+ assert(tableSize % unroll == 0); /* FSE_MIN_TABLELOG is 5 */
+ for (s = 0; s < (size_t)tableSize; s += unroll) {
+ size_t u;
+ for (u = 0; u < unroll; ++u) {
+ size_t const uPosition = (position + (u * step)) & tableMask;
+ tableSymbol[uPosition] = spread[s + u];
+ }
+ position = (position + (unroll * step)) & tableMask;
+ }
+ assert(position == 0); /* Must have initialized all positions */
+ }
+ } else {
+ U32 position = 0;
U32 symbol;
- for (symbol=0; symbol<=maxSymbolValue; symbol++) {
+ for (symbol=0; symbol<maxSV1; symbol++) {
int nbOccurrences;
int const freq = normalizedCounter[symbol];
for (nbOccurrences=0; nbOccurrences<freq; nbOccurrences++) {
@@ -120,7 +162,6 @@ size_t FSE_buildCTable_wksp(FSE_CTable* ct,
while (position > highThreshold)
position = (position + step) & tableMask; /* Low proba area */
} }
-
assert(position==0); /* Must have initialized all positions */
}
@@ -144,16 +185,17 @@ size_t FSE_buildCTable_wksp(FSE_CTable* ct,
case -1:
case 1:
symbolTT[s].deltaNbBits = (tableLog << 16) - (1<<tableLog);
- symbolTT[s].deltaFindState = total - 1;
+ assert(total <= INT_MAX);
+ symbolTT[s].deltaFindState = (int)(total - 1);
total ++;
break;
default :
- {
- U32 const maxBitsOut = tableLog - BIT_highbit32 (normalizedCounter[s]-1);
- U32 const minStatePlus = normalizedCounter[s] << maxBitsOut;
+ assert(normalizedCounter[s] > 1);
+ { U32 const maxBitsOut = tableLog - BIT_highbit32 ((U32)normalizedCounter[s]-1);
+ U32 const minStatePlus = (U32)normalizedCounter[s] << maxBitsOut;
symbolTT[s].deltaNbBits = (maxBitsOut << 16) - minStatePlus;
- symbolTT[s].deltaFindState = total - normalizedCounter[s];
- total += normalizedCounter[s];
+ symbolTT[s].deltaFindState = (int)(total - (unsigned)normalizedCounter[s]);
+ total += (unsigned)normalizedCounter[s];
} } } }
#if 0 /* debug : symbol costs */
@@ -164,8 +206,7 @@ size_t FSE_buildCTable_wksp(FSE_CTable* ct,
symbol, normalizedCounter[symbol],
FSE_getMaxNbBits(symbolTT, symbol),
(double)FSE_bitCost(symbolTT, tableLog, symbol, 8) / 256);
- }
- }
+ } }
#endif
return 0;
@@ -173,16 +214,18 @@ size_t FSE_buildCTable_wksp(FSE_CTable* ct,
-
#ifndef FSE_COMMONDEFS_ONLY
-
/*-**************************************************************
* FSE NCount encoding
****************************************************************/
size_t FSE_NCountWriteBound(unsigned maxSymbolValue, unsigned tableLog)
{
- size_t const maxHeaderSize = (((maxSymbolValue+1) * tableLog) >> 3) + 3;
+ size_t const maxHeaderSize = (((maxSymbolValue+1) * tableLog
+ + 4 /* bitCount initialized at 4 */
+ + 2 /* first two symbols may use one additional bit each */) / 8)
+ + 1 /* round up to whole nb bytes */
+ + 2 /* additional two bytes for bitstream flush */;
return maxSymbolValue ? maxHeaderSize : FSE_NCOUNTBOUND; /* maxSymbolValue==0 ? use default */
}
diff --git a/lib/zstd/compress/huf_compress.c b/lib/zstd/compress/huf_compress.c
index f76a526..74ef0db 100644
--- a/lib/zstd/compress/huf_compress.c
+++ b/lib/zstd/compress/huf_compress.c
@@ -50,6 +50,28 @@ unsigned HUF_optimalTableLog(unsigned maxTableLog, size_t srcSize, unsigned maxS
/* *******************************************************
* HUF : Huffman block compression
*********************************************************/
+#define HUF_WORKSPACE_MAX_ALIGNMENT 8
+
+static void* HUF_alignUpWorkspace(void* workspace, size_t* workspaceSizePtr, size_t align)
+{
+ size_t const mask = align - 1;
+ size_t const rem = (size_t)workspace & mask;
+ size_t const add = (align - rem) & mask;
+ BYTE* const aligned = (BYTE*)workspace + add;
+ assert((align & (align - 1)) == 0); /* pow 2 */
+ assert(align <= HUF_WORKSPACE_MAX_ALIGNMENT);
+ if (*workspaceSizePtr >= add) {
+ assert(add < align);
+ assert(((size_t)aligned & mask) == 0);
+ *workspaceSizePtr -= add;
+ return aligned;
+ } else {
+ *workspaceSizePtr = 0;
+ return NULL;
+ }
+}
+
+
/* HUF_compressWeights() :
* Same as FSE_compress(), but dedicated to huff0's weights compression.
* The use case needs much less stack memory.
@@ -72,7 +94,7 @@ static size_t HUF_compressWeights(void* dst, size_t dstSize, const void* weightT
unsigned maxSymbolValue = HUF_TABLELOG_MAX;
U32 tableLog = MAX_FSE_TABLELOG_FOR_HUFF_HEADER;
- HUF_CompressWeightsWksp* wksp = (HUF_CompressWeightsWksp*)workspace;
+ HUF_CompressWeightsWksp* wksp = (HUF_CompressWeightsWksp*)HUF_alignUpWorkspace(workspace, &workspaceSize, ZSTD_ALIGNOF(U32));
if (workspaceSize < sizeof(HUF_CompressWeightsWksp)) return ERROR(GENERIC);
@@ -103,6 +125,40 @@ static size_t HUF_compressWeights(void* dst, size_t dstSize, const void* weightT
return (size_t)(op-ostart);
}
+static size_t HUF_getNbBits(HUF_CElt elt)
+{
+ return elt & 0xFF;
+}
+
+static size_t HUF_getNbBitsFast(HUF_CElt elt)
+{
+ return elt;
+}
+
+static size_t HUF_getValue(HUF_CElt elt)
+{
+ return elt & ~0xFF;
+}
+
+static size_t HUF_getValueFast(HUF_CElt elt)
+{
+ return elt;
+}
+
+static void HUF_setNbBits(HUF_CElt* elt, size_t nbBits)
+{
+ assert(nbBits <= HUF_TABLELOG_ABSOLUTEMAX);
+ *elt = nbBits;
+}
+
+static void HUF_setValue(HUF_CElt* elt, size_t value)
+{
+ size_t const nbBits = HUF_getNbBits(*elt);
+ if (nbBits > 0) {
+ assert((value >> nbBits) == 0);
+ *elt |= value << (sizeof(HUF_CElt) * 8 - nbBits);
+ }
+}
typedef struct {
HUF_CompressWeightsWksp wksp;
@@ -114,9 +170,10 @@ size_t HUF_writeCTable_wksp(void* dst, size_t maxDstSize,
const HUF_CElt* CTable, unsigned maxSymbolValue, unsigned huffLog,
void* workspace, size_t workspaceSize)
{
+ HUF_CElt const* const ct = CTable + 1;
BYTE* op = (BYTE*)dst;
U32 n;
- HUF_WriteCTableWksp* wksp = (HUF_WriteCTableWksp*)workspace;
+ HUF_WriteCTableWksp* wksp = (HUF_WriteCTableWksp*)HUF_alignUpWorkspace(workspace, &workspaceSize, ZSTD_ALIGNOF(U32));
/* check conditions */
if (workspaceSize < sizeof(HUF_WriteCTableWksp)) return ERROR(GENERIC);
@@ -127,9 +184,10 @@ size_t HUF_writeCTable_wksp(void* dst, size_t maxDstSize,
for (n=1; n<huffLog+1; n++)
wksp->bitsToWeight[n] = (BYTE)(huffLog + 1 - n);
for (n=0; n<maxSymbolValue; n++)
- wksp->huffWeight[n] = wksp->bitsToWeight[CTable[n].nbBits];
+ wksp->huffWeight[n] = wksp->bitsToWeight[HUF_getNbBits(ct[n])];
/* attempt weights compression by FSE */
+ if (maxDstSize < 1) return ERROR(dstSize_tooSmall);
{ CHECK_V_F(hSize, HUF_compressWeights(op+1, maxDstSize-1, wksp->huffWeight, maxSymbolValue, &wksp->wksp, sizeof(wksp->wksp)) );
if ((hSize>1) & (hSize < maxSymbolValue/2)) { /* FSE compressed */
op[0] = (BYTE)hSize;
@@ -163,6 +221,7 @@ size_t HUF_readCTable (HUF_CElt* CTable, unsigned* maxSymbolValuePtr, const void
U32 rankVal[HUF_TABLELOG_ABSOLUTEMAX + 1]; /* large enough for values from 0 to 16 */
U32 tableLog = 0;
U32 nbSymbols = 0;
+ HUF_CElt* const ct = CTable + 1;
/* get symbol weights */
CHECK_V_F(readSize, HUF_readStats(huffWeight, HUF_SYMBOLVALUE_MAX+1, rankVal, &nbSymbols, &tableLog, src, srcSize));
@@ -172,6 +231,8 @@ size_t HUF_readCTable (HUF_CElt* CTable, unsigned* maxSymbolValuePtr, const void
if (tableLog > HUF_TABLELOG_MAX) return ERROR(tableLog_tooLarge);
if (nbSymbols > *maxSymbolValuePtr+1) return ERROR(maxSymbolValue_tooSmall);
+ CTable[0] = tableLog;
+
/* Prepare base value per rank */
{ U32 n, nextRankStart = 0;
for (n=1; n<=tableLog; n++) {
@@ -183,13 +244,13 @@ size_t HUF_readCTable (HUF_CElt* CTable, unsigned* maxSymbolValuePtr, const void
/* fill nbBits */
{ U32 n; for (n=0; n<nbSymbols; n++) {
const U32 w = huffWeight[n];
- CTable[n].nbBits = (BYTE)(tableLog + 1 - w) & -(w != 0);
+ HUF_setNbBits(ct + n, (BYTE)(tableLog + 1 - w) & -(w != 0));
} }
/* fill val */
{ U16 nbPerRank[HUF_TABLELOG_MAX+2] = {0}; /* support w=0=>n=tableLog+1 */
U16 valPerRank[HUF_TABLELOG_MAX+2] = {0};
- { U32 n; for (n=0; n<nbSymbols; n++) nbPerRank[CTable[n].nbBits]++; }
+ { U32 n; for (n=0; n<nbSymbols; n++) nbPerRank[HUF_getNbBits(ct[n])]++; }
/* determine stating value per rank */
valPerRank[tableLog+1] = 0; /* for w==0 */
{ U16 min = 0;
@@ -199,18 +260,18 @@ size_t HUF_readCTable (HUF_CElt* CTable, unsigned* maxSymbolValuePtr, const void
min >>= 1;
} }
/* assign value within rank, symbol order */
- { U32 n; for (n=0; n<nbSymbols; n++) CTable[n].val = valPerRank[CTable[n].nbBits]++; }
+ { U32 n; for (n=0; n<nbSymbols; n++) HUF_setValue(ct + n, valPerRank[HUF_getNbBits(ct[n])]++); }
}
*maxSymbolValuePtr = nbSymbols - 1;
return readSize;
}
-U32 HUF_getNbBits(const void* symbolTable, U32 symbolValue)
+U32 HUF_getNbBitsFromCTable(HUF_CElt const* CTable, U32 symbolValue)
{
- const HUF_CElt* table = (const HUF_CElt*)symbolTable;
+ const HUF_CElt* ct = CTable + 1;
assert(symbolValue <= HUF_SYMBOLVALUE_MAX);
- return table[symbolValue].nbBits;
+ return (U32)HUF_getNbBits(ct[symbolValue]);
}
@@ -364,22 +425,118 @@ static U32 HUF_setMaxHeight(nodeElt* huffNode, U32 lastNonNull, U32 maxNbBits)
}
typedef struct {
- U32 base;
- U32 curr;
+ U16 base;
+ U16 curr;
} rankPos;
typedef nodeElt huffNodeTable[HUF_CTABLE_WORKSPACE_SIZE_U32];
-#define RANK_POSITION_TABLE_SIZE 32
+/* Number of buckets available for HUF_sort() */
+#define RANK_POSITION_TABLE_SIZE 192
typedef struct {
huffNodeTable huffNodeTbl;
rankPos rankPosition[RANK_POSITION_TABLE_SIZE];
} HUF_buildCTable_wksp_tables;
+/* RANK_POSITION_DISTINCT_COUNT_CUTOFF == Cutoff point in HUF_sort() buckets for which we use log2 bucketing.
+ * Strategy is to use as many buckets as possible for representing distinct
+ * counts while using the remainder to represent all "large" counts.
+ *
+ * To satisfy this requirement for 192 buckets, we can do the following:
+ * Let buckets 0-166 represent distinct counts of [0, 166]
+ * Let buckets 166 to 192 represent all remaining counts up to RANK_POSITION_MAX_COUNT_LOG using log2 bucketing.
+ */
+#define RANK_POSITION_MAX_COUNT_LOG 32
+#define RANK_POSITION_LOG_BUCKETS_BEGIN (RANK_POSITION_TABLE_SIZE - 1) - RANK_POSITION_MAX_COUNT_LOG - 1 /* == 158 */
+#define RANK_POSITION_DISTINCT_COUNT_CUTOFF RANK_POSITION_LOG_BUCKETS_BEGIN + BIT_highbit32(RANK_POSITION_LOG_BUCKETS_BEGIN) /* == 166 */
+
+/* Return the appropriate bucket index for a given count. See definition of
+ * RANK_POSITION_DISTINCT_COUNT_CUTOFF for explanation of bucketing strategy.
+ */
+static U32 HUF_getIndex(U32 const count) {
+ return (count < RANK_POSITION_DISTINCT_COUNT_CUTOFF)
+ ? count
+ : BIT_highbit32(count) + RANK_POSITION_LOG_BUCKETS_BEGIN;
+}
+
+/* Helper swap function for HUF_quickSortPartition() */
+static void HUF_swapNodes(nodeElt* a, nodeElt* b) {
+ nodeElt tmp = *a;
+ *a = *b;
+ *b = tmp;
+}
+
+/* Returns 0 if the huffNode array is not sorted by descending count */
+MEM_STATIC int HUF_isSorted(nodeElt huffNode[], U32 const maxSymbolValue1) {
+ U32 i;
+ for (i = 1; i < maxSymbolValue1; ++i) {
+ if (huffNode[i].count > huffNode[i-1].count) {
+ return 0;
+ }
+ }
+ return 1;
+}
+
+/* Insertion sort by descending order */
+HINT_INLINE void HUF_insertionSort(nodeElt huffNode[], int const low, int const high) {
+ int i;
+ int const size = high-low+1;
+ huffNode += low;
+ for (i = 1; i < size; ++i) {
+ nodeElt const key = huffNode[i];
+ int j = i - 1;
+ while (j >= 0 && huffNode[j].count < key.count) {
+ huffNode[j + 1] = huffNode[j];
+ j--;
+ }
+ huffNode[j + 1] = key;
+ }
+}
+
+/* Pivot helper function for quicksort. */
+static int HUF_quickSortPartition(nodeElt arr[], int const low, int const high) {
+ /* Simply select rightmost element as pivot. "Better" selectors like
+ * median-of-three don't experimentally appear to have any benefit.
+ */
+ U32 const pivot = arr[high].count;
+ int i = low - 1;
+ int j = low;
+ for ( ; j < high; j++) {
+ if (arr[j].count > pivot) {
+ i++;
+ HUF_swapNodes(&arr[i], &arr[j]);
+ }
+ }
+ HUF_swapNodes(&arr[i + 1], &arr[high]);
+ return i + 1;
+}
+
+/* Classic quicksort by descending with partially iterative calls
+ * to reduce worst case callstack size.
+ */
+static void HUF_simpleQuickSort(nodeElt arr[], int low, int high) {
+ int const kInsertionSortThreshold = 8;
+ if (high - low < kInsertionSortThreshold) {
+ HUF_insertionSort(arr, low, high);
+ return;
+ }
+ while (low < high) {
+ int const idx = HUF_quickSortPartition(arr, low, high);
+ if (idx - low < high - idx) {
+ HUF_simpleQuickSort(arr, low, idx - 1);
+ low = idx + 1;
+ } else {
+ HUF_simpleQuickSort(arr, idx + 1, high);
+ high = idx - 1;
+ }
+ }
+}
+
/*
* HUF_sort():
* Sorts the symbols [0, maxSymbolValue] by count[symbol] in decreasing order.
+ * This is a typical bucket sorting strategy that uses either quicksort or insertion sort to sort each bucket.
*
* @param[out] huffNode Sorted symbols by decreasing count. Only members `.count` and `.byte` are filled.
* Must have (maxSymbolValue + 1) entries.
@@ -387,44 +544,52 @@ typedef struct {
* @param[in] maxSymbolValue Maximum symbol value.
* @param rankPosition This is a scratch workspace. Must have RANK_POSITION_TABLE_SIZE entries.
*/
-static void HUF_sort(nodeElt* huffNode, const unsigned* count, U32 maxSymbolValue, rankPos* rankPosition)
-{
- int n;
- int const maxSymbolValue1 = (int)maxSymbolValue + 1;
+static void HUF_sort(nodeElt huffNode[], const unsigned count[], U32 const maxSymbolValue, rankPos rankPosition[]) {
+ U32 n;
+ U32 const maxSymbolValue1 = maxSymbolValue+1;
/* Compute base and set curr to base.
- * For symbol s let lowerRank = BIT_highbit32(count[n]+1) and rank = lowerRank + 1.
- * Then 2^lowerRank <= count[n]+1 <= 2^rank.
+ * For symbol s let lowerRank = HUF_getIndex(count[n]) and rank = lowerRank + 1.
+ * See HUF_getIndex to see bucketing strategy.
* We attribute each symbol to lowerRank's base value, because we want to know where
* each rank begins in the output, so for rank R we want to count ranks R+1 and above.
*/
ZSTD_memset(rankPosition, 0, sizeof(*rankPosition) * RANK_POSITION_TABLE_SIZE);
for (n = 0; n < maxSymbolValue1; ++n) {
- U32 lowerRank = BIT_highbit32(count[n] + 1);
+ U32 lowerRank = HUF_getIndex(count[n]);
+ assert(lowerRank < RANK_POSITION_TABLE_SIZE - 1);
rankPosition[lowerRank].base++;
}
+
assert(rankPosition[RANK_POSITION_TABLE_SIZE - 1].base == 0);
+ /* Set up the rankPosition table */
for (n = RANK_POSITION_TABLE_SIZE - 1; n > 0; --n) {
rankPosition[n-1].base += rankPosition[n].base;
rankPosition[n-1].curr = rankPosition[n-1].base;
}
- /* Sort */
+
+ /* Insert each symbol into their appropriate bucket, setting up rankPosition table. */
for (n = 0; n < maxSymbolValue1; ++n) {
U32 const c = count[n];
- U32 const r = BIT_highbit32(c+1) + 1;
- U32 pos = rankPosition[r].curr++;
- /* Insert into the correct position in the rank.
- * We have at most 256 symbols, so this insertion should be fine.
- */
- while ((pos > rankPosition[r].base) && (c > huffNode[pos-1].count)) {
- huffNode[pos] = huffNode[pos-1];
- pos--;
- }
+ U32 const r = HUF_getIndex(c) + 1;
+ U32 const pos = rankPosition[r].curr++;
+ assert(pos < maxSymbolValue1);
huffNode[pos].count = c;
huffNode[pos].byte = (BYTE)n;
}
-}
+ /* Sort each bucket. */
+ for (n = RANK_POSITION_DISTINCT_COUNT_CUTOFF; n < RANK_POSITION_TABLE_SIZE - 1; ++n) {
+ U32 const bucketSize = rankPosition[n].curr-rankPosition[n].base;
+ U32 const bucketStartIdx = rankPosition[n].base;
+ if (bucketSize > 1) {
+ assert(bucketStartIdx < maxSymbolValue1);
+ HUF_simpleQuickSort(huffNode + bucketStartIdx, 0, bucketSize-1);
+ }
+ }
+
+ assert(HUF_isSorted(huffNode, maxSymbolValue1));
+}
/* HUF_buildCTable_wksp() :
* Same as HUF_buildCTable(), but using externally allocated scratch buffer.
@@ -487,6 +652,7 @@ static int HUF_buildTree(nodeElt* huffNode, U32 maxSymbolValue)
*/
static void HUF_buildCTableFromTree(HUF_CElt* CTable, nodeElt const* huffNode, int nonNullRank, U32 maxSymbolValue, U32 maxNbBits)
{
+ HUF_CElt* const ct = CTable + 1;
/* fill result into ctable (val, nbBits) */
int n;
U16 nbPerRank[HUF_TABLELOG_MAX+1] = {0};
@@ -502,20 +668,20 @@ static void HUF_buildCTableFromTree(HUF_CElt* CTable, nodeElt const* huffNode, i
min >>= 1;
} }
for (n=0; n<alphabetSize; n++)
- CTable[huffNode[n].byte].nbBits = huffNode[n].nbBits; /* push nbBits per symbol, symbol order */
+ HUF_setNbBits(ct + huffNode[n].byte, huffNode[n].nbBits); /* push nbBits per symbol, symbol order */
for (n=0; n<alphabetSize; n++)
- CTable[n].val = valPerRank[CTable[n].nbBits]++; /* assign value within rank, symbol order */
+ HUF_setValue(ct + n, valPerRank[HUF_getNbBits(ct[n])]++); /* assign value within rank, symbol order */
+ CTable[0] = maxNbBits;
}
-size_t HUF_buildCTable_wksp (HUF_CElt* tree, const unsigned* count, U32 maxSymbolValue, U32 maxNbBits, void* workSpace, size_t wkspSize)
+size_t HUF_buildCTable_wksp (HUF_CElt* CTable, const unsigned* count, U32 maxSymbolValue, U32 maxNbBits, void* workSpace, size_t wkspSize)
{
- HUF_buildCTable_wksp_tables* const wksp_tables = (HUF_buildCTable_wksp_tables*)workSpace;
+ HUF_buildCTable_wksp_tables* const wksp_tables = (HUF_buildCTable_wksp_tables*)HUF_alignUpWorkspace(workSpace, &wkspSize, ZSTD_ALIGNOF(U32));
nodeElt* const huffNode0 = wksp_tables->huffNodeTbl;
nodeElt* const huffNode = huffNode0+1;
int nonNullRank;
/* safety checks */
- if (((size_t)workSpace & 3) != 0) return ERROR(GENERIC); /* must be aligned on 4-bytes boundaries */
if (wkspSize < sizeof(HUF_buildCTable_wksp_tables))
return ERROR(workSpace_tooSmall);
if (maxNbBits == 0) maxNbBits = HUF_TABLELOG_DEFAULT;
@@ -533,99 +699,334 @@ size_t HUF_buildCTable_wksp (HUF_CElt* tree, const unsigned* count, U32 maxSymbo
maxNbBits = HUF_setMaxHeight(huffNode, (U32)nonNullRank, maxNbBits);
if (maxNbBits > HUF_TABLELOG_MAX) return ERROR(GENERIC); /* check fit into table */
- HUF_buildCTableFromTree(tree, huffNode, nonNullRank, maxSymbolValue, maxNbBits);
+ HUF_buildCTableFromTree(CTable, huffNode, nonNullRank, maxSymbolValue, maxNbBits);
return maxNbBits;
}
size_t HUF_estimateCompressedSize(const HUF_CElt* CTable, const unsigned* count, unsigned maxSymbolValue)
{
+ HUF_CElt const* ct = CTable + 1;
size_t nbBits = 0;
int s;
for (s = 0; s <= (int)maxSymbolValue; ++s) {
- nbBits += CTable[s].nbBits * count[s];
+ nbBits += HUF_getNbBits(ct[s]) * count[s];
}
return nbBits >> 3;
}
int HUF_validateCTable(const HUF_CElt* CTable, const unsigned* count, unsigned maxSymbolValue) {
+ HUF_CElt const* ct = CTable + 1;
int bad = 0;
int s;
for (s = 0; s <= (int)maxSymbolValue; ++s) {
- bad |= (count[s] != 0) & (CTable[s].nbBits == 0);
+ bad |= (count[s] != 0) & (HUF_getNbBits(ct[s]) == 0);
}
return !bad;
}
size_t HUF_compressBound(size_t size) { return HUF_COMPRESSBOUND(size); }
-FORCE_INLINE_TEMPLATE void
-HUF_encodeSymbol(BIT_CStream_t* bitCPtr, U32 symbol, const HUF_CElt* CTable)
+/* HUF_CStream_t:
+ * Huffman uses its own BIT_CStream_t implementation.
+ * There are three major differences from BIT_CStream_t:
+ * 1. HUF_addBits() takes a HUF_CElt (size_t) which is
+ * the pair (nbBits, value) in the format:
+ * format:
+ * - Bits [0, 4) = nbBits
+ * - Bits [4, 64 - nbBits) = 0
+ * - Bits [64 - nbBits, 64) = value
+ * 2. The bitContainer is built from the upper bits and
+ * right shifted. E.g. to add a new value of N bits
+ * you right shift the bitContainer by N, then or in
+ * the new value into the N upper bits.
+ * 3. The bitstream has two bit containers. You can add
+ * bits to the second container and merge them into
+ * the first container.
+ */
+
+#define HUF_BITS_IN_CONTAINER (sizeof(size_t) * 8)
+
+typedef struct {
+ size_t bitContainer[2];
+ size_t bitPos[2];
+
+ BYTE* startPtr;
+ BYTE* ptr;
+ BYTE* endPtr;
+} HUF_CStream_t;
+
+/*! HUF_initCStream():
+ * Initializes the bitstream.
+ * @returns 0 or an error code.
+ */
+static size_t HUF_initCStream(HUF_CStream_t* bitC,
+ void* startPtr, size_t dstCapacity)
{
- BIT_addBitsFast(bitCPtr, CTable[symbol].val, CTable[symbol].nbBits);
+ ZSTD_memset(bitC, 0, sizeof(*bitC));
+ bitC->startPtr = (BYTE*)startPtr;
+ bitC->ptr = bitC->startPtr;
+ bitC->endPtr = bitC->startPtr + dstCapacity - sizeof(bitC->bitContainer[0]);
+ if (dstCapacity <= sizeof(bitC->bitContainer[0])) return ERROR(dstSize_tooSmall);
+ return 0;
}
-#define HUF_FLUSHBITS(s) BIT_flushBits(s)
+/*! HUF_addBits():
+ * Adds the symbol stored in HUF_CElt elt to the bitstream.
+ *
+ * @param elt The element we're adding. This is a (nbBits, value) pair.
+ * See the HUF_CStream_t docs for the format.
+ * @param idx Insert into the bitstream at this idx.
+ * @param kFast This is a template parameter. If the bitstream is guaranteed
+ * to have at least 4 unused bits after this call it may be 1,
+ * otherwise it must be 0. HUF_addBits() is faster when fast is set.
+ */
+FORCE_INLINE_TEMPLATE void HUF_addBits(HUF_CStream_t* bitC, HUF_CElt elt, int idx, int kFast)
+{
+ assert(idx <= 1);
+ assert(HUF_getNbBits(elt) <= HUF_TABLELOG_ABSOLUTEMAX);
+ /* This is efficient on x86-64 with BMI2 because shrx
+ * only reads the low 6 bits of the register. The compiler
+ * knows this and elides the mask. When fast is set,
+ * every operation can use the same value loaded from elt.
+ */
+ bitC->bitContainer[idx] >>= HUF_getNbBits(elt);
+ bitC->bitContainer[idx] |= kFast ? HUF_getValueFast(elt) : HUF_getValue(elt);
+ /* We only read the low 8 bits of bitC->bitPos[idx] so it
+ * doesn't matter that the high bits have noise from the value.
+ */
+ bitC->bitPos[idx] += HUF_getNbBitsFast(elt);
+ assert((bitC->bitPos[idx] & 0xFF) <= HUF_BITS_IN_CONTAINER);
+ /* The last 4-bits of elt are dirty if fast is set,
+ * so we must not be overwriting bits that have already been
+ * inserted into the bit container.
+ */
+#if DEBUGLEVEL >= 1
+ {
+ size_t const nbBits = HUF_getNbBits(elt);
+ size_t const dirtyBits = nbBits == 0 ? 0 : BIT_highbit32((U32)nbBits) + 1;
+ (void)dirtyBits;
+ /* Middle bits are 0. */
+ assert(((elt >> dirtyBits) << (dirtyBits + nbBits)) == 0);
+ /* We didn't overwrite any bits in the bit container. */
+ assert(!kFast || (bitC->bitPos[idx] & 0xFF) <= HUF_BITS_IN_CONTAINER);
+ (void)dirtyBits;
+ }
+#endif
+}
-#define HUF_FLUSHBITS_1(stream) \
- if (sizeof((stream)->bitContainer)*8 < HUF_TABLELOG_MAX*2+7) HUF_FLUSHBITS(stream)
+FORCE_INLINE_TEMPLATE void HUF_zeroIndex1(HUF_CStream_t* bitC)
+{
+ bitC->bitContainer[1] = 0;
+ bitC->bitPos[1] = 0;
+}
-#define HUF_FLUSHBITS_2(stream) \
- if (sizeof((stream)->bitContainer)*8 < HUF_TABLELOG_MAX*4+7) HUF_FLUSHBITS(stream)
+/*! HUF_mergeIndex1() :
+ * Merges the bit container @ index 1 into the bit container @ index 0
+ * and zeros the bit container @ index 1.
+ */
+FORCE_INLINE_TEMPLATE void HUF_mergeIndex1(HUF_CStream_t* bitC)
+{
+ assert((bitC->bitPos[1] & 0xFF) < HUF_BITS_IN_CONTAINER);
+ bitC->bitContainer[0] >>= (bitC->bitPos[1] & 0xFF);
+ bitC->bitContainer[0] |= bitC->bitContainer[1];
+ bitC->bitPos[0] += bitC->bitPos[1];
+ assert((bitC->bitPos[0] & 0xFF) <= HUF_BITS_IN_CONTAINER);
+}
+
+/*! HUF_flushBits() :
+* Flushes the bits in the bit container @ index 0.
+*
+* @post bitPos will be < 8.
+* @param kFast If kFast is set then we must know a-priori that
+* the bit container will not overflow.
+*/
+FORCE_INLINE_TEMPLATE void HUF_flushBits(HUF_CStream_t* bitC, int kFast)
+{
+ /* The upper bits of bitPos are noisy, so we must mask by 0xFF. */
+ size_t const nbBits = bitC->bitPos[0] & 0xFF;
+ size_t const nbBytes = nbBits >> 3;
+ /* The top nbBits bits of bitContainer are the ones we need. */
+ size_t const bitContainer = bitC->bitContainer[0] >> (HUF_BITS_IN_CONTAINER - nbBits);
+ /* Mask bitPos to account for the bytes we consumed. */
+ bitC->bitPos[0] &= 7;
+ assert(nbBits > 0);
+ assert(nbBits <= sizeof(bitC->bitContainer[0]) * 8);
+ assert(bitC->ptr <= bitC->endPtr);
+ MEM_writeLEST(bitC->ptr, bitContainer);
+ bitC->ptr += nbBytes;
+ assert(!kFast || bitC->ptr <= bitC->endPtr);
+ if (!kFast && bitC->ptr > bitC->endPtr) bitC->ptr = bitC->endPtr;
+ /* bitContainer doesn't need to be modified because the leftover
+ * bits are already the top bitPos bits. And we don't care about
+ * noise in the lower values.
+ */
+}
+
+/*! HUF_endMark()
+ * @returns The Huffman stream end mark: A 1-bit value = 1.
+ */
+static HUF_CElt HUF_endMark(void)
+{
+ HUF_CElt endMark;
+ HUF_setNbBits(&endMark, 1);
+ HUF_setValue(&endMark, 1);
+ return endMark;
+}
+
+/*! HUF_closeCStream() :
+ * @return Size of CStream, in bytes,
+ * or 0 if it could not fit into dstBuffer */
+static size_t HUF_closeCStream(HUF_CStream_t* bitC)
+{
+ HUF_addBits(bitC, HUF_endMark(), /* idx */ 0, /* kFast */ 0);
+ HUF_flushBits(bitC, /* kFast */ 0);
+ {
+ size_t const nbBits = bitC->bitPos[0] & 0xFF;
+ if (bitC->ptr >= bitC->endPtr) return 0; /* overflow detected */
+ return (bitC->ptr - bitC->startPtr) + (nbBits > 0);
+ }
+}
+
+FORCE_INLINE_TEMPLATE void
+HUF_encodeSymbol(HUF_CStream_t* bitCPtr, U32 symbol, const HUF_CElt* CTable, int idx, int fast)
+{
+ HUF_addBits(bitCPtr, CTable[symbol], idx, fast);
+}
+
+FORCE_INLINE_TEMPLATE void
+HUF_compress1X_usingCTable_internal_body_loop(HUF_CStream_t* bitC,
+ const BYTE* ip, size_t srcSize,
+ const HUF_CElt* ct,
+ int kUnroll, int kFastFlush, int kLastFast)
+{
+ /* Join to kUnroll */
+ int n = (int)srcSize;
+ int rem = n % kUnroll;
+ if (rem > 0) {
+ for (; rem > 0; --rem) {
+ HUF_encodeSymbol(bitC, ip[--n], ct, 0, /* fast */ 0);
+ }
+ HUF_flushBits(bitC, kFastFlush);
+ }
+ assert(n % kUnroll == 0);
+
+ /* Join to 2 * kUnroll */
+ if (n % (2 * kUnroll)) {
+ int u;
+ for (u = 1; u < kUnroll; ++u) {
+ HUF_encodeSymbol(bitC, ip[n - u], ct, 0, 1);
+ }
+ HUF_encodeSymbol(bitC, ip[n - kUnroll], ct, 0, kLastFast);
+ HUF_flushBits(bitC, kFastFlush);
+ n -= kUnroll;
+ }
+ assert(n % (2 * kUnroll) == 0);
+
+ for (; n>0; n-= 2 * kUnroll) {
+ /* Encode kUnroll symbols into the bitstream @ index 0. */
+ int u;
+ for (u = 1; u < kUnroll; ++u) {
+ HUF_encodeSymbol(bitC, ip[n - u], ct, /* idx */ 0, /* fast */ 1);
+ }
+ HUF_encodeSymbol(bitC, ip[n - kUnroll], ct, /* idx */ 0, /* fast */ kLastFast);
+ HUF_flushBits(bitC, kFastFlush);
+ /* Encode kUnroll symbols into the bitstream @ index 1.
+ * This allows us to start filling the bit container
+ * without any data dependencies.
+ */
+ HUF_zeroIndex1(bitC);
+ for (u = 1; u < kUnroll; ++u) {
+ HUF_encodeSymbol(bitC, ip[n - kUnroll - u], ct, /* idx */ 1, /* fast */ 1);
+ }
+ HUF_encodeSymbol(bitC, ip[n - kUnroll - kUnroll], ct, /* idx */ 1, /* fast */ kLastFast);
+ /* Merge bitstream @ index 1 into the bitstream @ index 0 */
+ HUF_mergeIndex1(bitC);
+ HUF_flushBits(bitC, kFastFlush);
+ }
+ assert(n == 0);
+
+}
+
+/*
+ * Returns a tight upper bound on the output space needed by Huffman
+ * with 8 bytes buffer to handle over-writes. If the output is at least
+ * this large we don't need to do bounds checks during Huffman encoding.
+ */
+static size_t HUF_tightCompressBound(size_t srcSize, size_t tableLog)
+{
+ return ((srcSize * tableLog) >> 3) + 8;
+}
+
FORCE_INLINE_TEMPLATE size_t
HUF_compress1X_usingCTable_internal_body(void* dst, size_t dstSize,
const void* src, size_t srcSize,
const HUF_CElt* CTable)
{
+ U32 const tableLog = (U32)CTable[0];
+ HUF_CElt const* ct = CTable + 1;
const BYTE* ip = (const BYTE*) src;
BYTE* const ostart = (BYTE*)dst;
BYTE* const oend = ostart + dstSize;
BYTE* op = ostart;
- size_t n;
- BIT_CStream_t bitC;
+ HUF_CStream_t bitC;
/* init */
if (dstSize < 8) return 0; /* not enough space to compress */
- { size_t const initErr = BIT_initCStream(&bitC, op, (size_t)(oend-op));
+ { size_t const initErr = HUF_initCStream(&bitC, op, (size_t)(oend-op));
if (HUF_isError(initErr)) return 0; }
- n = srcSize & ~3; /* join to mod 4 */
- switch (srcSize & 3)
- {
- case 3:
- HUF_encodeSymbol(&bitC, ip[n+ 2], CTable);
- HUF_FLUSHBITS_2(&bitC);
- ZSTD_FALLTHROUGH;
- case 2:
- HUF_encodeSymbol(&bitC, ip[n+ 1], CTable);
- HUF_FLUSHBITS_1(&bitC);
- ZSTD_FALLTHROUGH;
- case 1:
- HUF_encodeSymbol(&bitC, ip[n+ 0], CTable);
- HUF_FLUSHBITS(&bitC);
- ZSTD_FALLTHROUGH;
- case 0: ZSTD_FALLTHROUGH;
- default: break;
+ if (dstSize < HUF_tightCompressBound(srcSize, (size_t)tableLog) || tableLog > 11)
+ HUF_compress1X_usingCTable_internal_body_loop(&bitC, ip, srcSize, ct, /* kUnroll */ MEM_32bits() ? 2 : 4, /* kFast */ 0, /* kLastFast */ 0);
+ else {
+ if (MEM_32bits()) {
+ switch (tableLog) {
+ case 11:
+ HUF_compress1X_usingCTable_internal_body_loop(&bitC, ip, srcSize, ct, /* kUnroll */ 2, /* kFastFlush */ 1, /* kLastFast */ 0);
+ break;
+ case 10: ZSTD_FALLTHROUGH;
+ case 9: ZSTD_FALLTHROUGH;
+ case 8:
+ HUF_compress1X_usingCTable_internal_body_loop(&bitC, ip, srcSize, ct, /* kUnroll */ 2, /* kFastFlush */ 1, /* kLastFast */ 1);
+ break;
+ case 7: ZSTD_FALLTHROUGH;
+ default:
+ HUF_compress1X_usingCTable_internal_body_loop(&bitC, ip, srcSize, ct, /* kUnroll */ 3, /* kFastFlush */ 1, /* kLastFast */ 1);
+ break;
+ }
+ } else {
+ switch (tableLog) {
+ case 11:
+ HUF_compress1X_usingCTable_internal_body_loop(&bitC, ip, srcSize, ct, /* kUnroll */ 5, /* kFastFlush */ 1, /* kLastFast */ 0);
+ break;
+ case 10:
+ HUF_compress1X_usingCTable_internal_body_loop(&bitC, ip, srcSize, ct, /* kUnroll */ 5, /* kFastFlush */ 1, /* kLastFast */ 1);
+ break;
+ case 9:
+ HUF_compress1X_usingCTable_internal_body_loop(&bitC, ip, srcSize, ct, /* kUnroll */ 6, /* kFastFlush */ 1, /* kLastFast */ 0);
+ break;
+ case 8:
+ HUF_compress1X_usingCTable_internal_body_loop(&bitC, ip, srcSize, ct, /* kUnroll */ 7, /* kFastFlush */ 1, /* kLastFast */ 0);
+ break;
+ case 7:
+ HUF_compress1X_usingCTable_internal_body_loop(&bitC, ip, srcSize, ct, /* kUnroll */ 8, /* kFastFlush */ 1, /* kLastFast */ 0);
+ break;
+ case 6: ZSTD_FALLTHROUGH;
+ default:
+ HUF_compress1X_usingCTable_internal_body_loop(&bitC, ip, srcSize, ct, /* kUnroll */ 9, /* kFastFlush */ 1, /* kLastFast */ 1);
+ break;
+ }
+ }
}
+ assert(bitC.ptr <= bitC.endPtr);
- for (; n>0; n-=4) { /* note : n&3==0 at this stage */
- HUF_encodeSymbol(&bitC, ip[n- 1], CTable);
- HUF_FLUSHBITS_1(&bitC);
- HUF_encodeSymbol(&bitC, ip[n- 2], CTable);
- HUF_FLUSHBITS_2(&bitC);
- HUF_encodeSymbol(&bitC, ip[n- 3], CTable);
- HUF_FLUSHBITS_1(&bitC);
- HUF_encodeSymbol(&bitC, ip[n- 4], CTable);
- HUF_FLUSHBITS(&bitC);
- }
-
- return BIT_closeCStream(&bitC);
+ return HUF_closeCStream(&bitC);
}
#if DYNAMIC_BMI2
-static TARGET_ATTRIBUTE("bmi2") size_t
+static BMI2_TARGET_ATTRIBUTE size_t
HUF_compress1X_usingCTable_internal_bmi2(void* dst, size_t dstSize,
const void* src, size_t srcSize,
const HUF_CElt* CTable)
@@ -667,9 +1068,13 @@ HUF_compress1X_usingCTable_internal(void* dst, size_t dstSize,
size_t HUF_compress1X_usingCTable(void* dst, size_t dstSize, const void* src, size_t srcSize, const HUF_CElt* CTable)
{
- return HUF_compress1X_usingCTable_internal(dst, dstSize, src, srcSize, CTable, /* bmi2 */ 0);
+ return HUF_compress1X_usingCTable_bmi2(dst, dstSize, src, srcSize, CTable, /* bmi2 */ 0);
}
+size_t HUF_compress1X_usingCTable_bmi2(void* dst, size_t dstSize, const void* src, size_t srcSize, const HUF_CElt* CTable, int bmi2)
+{
+ return HUF_compress1X_usingCTable_internal(dst, dstSize, src, srcSize, CTable, bmi2);
+}
static size_t
HUF_compress4X_usingCTable_internal(void* dst, size_t dstSize,
@@ -689,8 +1094,7 @@ HUF_compress4X_usingCTable_internal(void* dst, size_t dstSize,
assert(op <= oend);
{ CHECK_V_F(cSize, HUF_compress1X_usingCTable_internal(op, (size_t)(oend-op), ip, segmentSize, CTable, bmi2) );
- if (cSize==0) return 0;
- assert(cSize <= 65535);
+ if (cSize == 0 || cSize > 65535) return 0;
MEM_writeLE16(ostart, (U16)cSize);
op += cSize;
}
@@ -698,8 +1102,7 @@ HUF_compress4X_usingCTable_internal(void* dst, size_t dstSize,
ip += segmentSize;
assert(op <= oend);
{ CHECK_V_F(cSize, HUF_compress1X_usingCTable_internal(op, (size_t)(oend-op), ip, segmentSize, CTable, bmi2) );
- if (cSize==0) return 0;
- assert(cSize <= 65535);
+ if (cSize == 0 || cSize > 65535) return 0;
MEM_writeLE16(ostart+2, (U16)cSize);
op += cSize;
}
@@ -707,8 +1110,7 @@ HUF_compress4X_usingCTable_internal(void* dst, size_t dstSize,
ip += segmentSize;
assert(op <= oend);
{ CHECK_V_F(cSize, HUF_compress1X_usingCTable_internal(op, (size_t)(oend-op), ip, segmentSize, CTable, bmi2) );
- if (cSize==0) return 0;
- assert(cSize <= 65535);
+ if (cSize == 0 || cSize > 65535) return 0;
MEM_writeLE16(ostart+4, (U16)cSize);
op += cSize;
}
@@ -717,7 +1119,7 @@ HUF_compress4X_usingCTable_internal(void* dst, size_t dstSize,
assert(op <= oend);
assert(ip <= iend);
{ CHECK_V_F(cSize, HUF_compress1X_usingCTable_internal(op, (size_t)(oend-op), ip, (size_t)(iend-ip), CTable, bmi2) );
- if (cSize==0) return 0;
+ if (cSize == 0 || cSize > 65535) return 0;
op += cSize;
}
@@ -726,7 +1128,12 @@ HUF_compress4X_usingCTable_internal(void* dst, size_t dstSize,
size_t HUF_compress4X_usingCTable(void* dst, size_t dstSize, const void* src, size_t srcSize, const HUF_CElt* CTable)
{
- return HUF_compress4X_usingCTable_internal(dst, dstSize, src, srcSize, CTable, /* bmi2 */ 0);
+ return HUF_compress4X_usingCTable_bmi2(dst, dstSize, src, srcSize, CTable, /* bmi2 */ 0);
+}
+
+size_t HUF_compress4X_usingCTable_bmi2(void* dst, size_t dstSize, const void* src, size_t srcSize, const HUF_CElt* CTable, int bmi2)
+{
+ return HUF_compress4X_usingCTable_internal(dst, dstSize, src, srcSize, CTable, bmi2);
}
typedef enum { HUF_singleStream, HUF_fourStreams } HUF_nbStreams_e;
@@ -750,35 +1157,38 @@ static size_t HUF_compressCTable_internal(
typedef struct {
unsigned count[HUF_SYMBOLVALUE_MAX + 1];
- HUF_CElt CTable[HUF_SYMBOLVALUE_MAX + 1];
+ HUF_CElt CTable[HUF_CTABLE_SIZE_ST(HUF_SYMBOLVALUE_MAX)];
union {
HUF_buildCTable_wksp_tables buildCTable_wksp;
HUF_WriteCTableWksp writeCTable_wksp;
+ U32 hist_wksp[HIST_WKSP_SIZE_U32];
} wksps;
} HUF_compress_tables_t;
+#define SUSPECT_INCOMPRESSIBLE_SAMPLE_SIZE 4096
+#define SUSPECT_INCOMPRESSIBLE_SAMPLE_RATIO 10 /* Must be >= 2 */
+
/* HUF_compress_internal() :
* `workSpace_align4` must be aligned on 4-bytes boundaries,
- * and occupies the same space as a table of HUF_WORKSPACE_SIZE_U32 unsigned */
+ * and occupies the same space as a table of HUF_WORKSPACE_SIZE_U64 unsigned */
static size_t
HUF_compress_internal (void* dst, size_t dstSize,
const void* src, size_t srcSize,
unsigned maxSymbolValue, unsigned huffLog,
HUF_nbStreams_e nbStreams,
- void* workSpace_align4, size_t wkspSize,
+ void* workSpace, size_t wkspSize,
HUF_CElt* oldHufTable, HUF_repeat* repeat, int preferRepeat,
- const int bmi2)
+ const int bmi2, unsigned suspectUncompressible)
{
- HUF_compress_tables_t* const table = (HUF_compress_tables_t*)workSpace_align4;
+ HUF_compress_tables_t* const table = (HUF_compress_tables_t*)HUF_alignUpWorkspace(workSpace, &wkspSize, ZSTD_ALIGNOF(size_t));
BYTE* const ostart = (BYTE*)dst;
BYTE* const oend = ostart + dstSize;
BYTE* op = ostart;
- HUF_STATIC_ASSERT(sizeof(*table) <= HUF_WORKSPACE_SIZE);
- assert(((size_t)workSpace_align4 & 3) == 0); /* must be aligned on 4-bytes boundaries */
+ HUF_STATIC_ASSERT(sizeof(*table) + HUF_WORKSPACE_MAX_ALIGNMENT <= HUF_WORKSPACE_SIZE);
/* checks & inits */
- if (wkspSize < HUF_WORKSPACE_SIZE) return ERROR(workSpace_tooSmall);
+ if (wkspSize < sizeof(*table)) return ERROR(workSpace_tooSmall);
if (!srcSize) return 0; /* Uncompressed */
if (!dstSize) return 0; /* cannot fit anything within dst budget */
if (srcSize > HUF_BLOCKSIZE_MAX) return ERROR(srcSize_wrong); /* current block size limit */
@@ -794,8 +1204,23 @@ HUF_compress_internal (void* dst, size_t dstSize,
nbStreams, oldHufTable, bmi2);
}
+ /* If uncompressible data is suspected, do a smaller sampling first */
+ DEBUG_STATIC_ASSERT(SUSPECT_INCOMPRESSIBLE_SAMPLE_RATIO >= 2);
+ if (suspectUncompressible && srcSize >= (SUSPECT_INCOMPRESSIBLE_SAMPLE_SIZE * SUSPECT_INCOMPRESSIBLE_SAMPLE_RATIO)) {
+ size_t largestTotal = 0;
+ { unsigned maxSymbolValueBegin = maxSymbolValue;
+ CHECK_V_F(largestBegin, HIST_count_simple (table->count, &maxSymbolValueBegin, (const BYTE*)src, SUSPECT_INCOMPRESSIBLE_SAMPLE_SIZE) );
+ largestTotal += largestBegin;
+ }
+ { unsigned maxSymbolValueEnd = maxSymbolValue;
+ CHECK_V_F(largestEnd, HIST_count_simple (table->count, &maxSymbolValueEnd, (const BYTE*)src + srcSize - SUSPECT_INCOMPRESSIBLE_SAMPLE_SIZE, SUSPECT_INCOMPRESSIBLE_SAMPLE_SIZE) );
+ largestTotal += largestEnd;
+ }
+ if (largestTotal <= ((2 * SUSPECT_INCOMPRESSIBLE_SAMPLE_SIZE) >> 7)+4) return 0; /* heuristic : probably not compressible enough */
+ }
+
/* Scan input and build symbol stats */
- { CHECK_V_F(largest, HIST_count_wksp (table->count, &maxSymbolValue, (const BYTE*)src, srcSize, workSpace_align4, wkspSize) );
+ { CHECK_V_F(largest, HIST_count_wksp (table->count, &maxSymbolValue, (const BYTE*)src, srcSize, table->wksps.hist_wksp, sizeof(table->wksps.hist_wksp)) );
if (largest == srcSize) { *ostart = ((const BYTE*)src)[0]; return 1; } /* single symbol, rle */
if (largest <= (srcSize >> 7)+4) return 0; /* heuristic : probably not compressible enough */
}
@@ -820,9 +1245,12 @@ HUF_compress_internal (void* dst, size_t dstSize,
&table->wksps.buildCTable_wksp, sizeof(table->wksps.buildCTable_wksp));
CHECK_F(maxBits);
huffLog = (U32)maxBits;
- /* Zero unused symbols in CTable, so we can check it for validity */
- ZSTD_memset(table->CTable + (maxSymbolValue + 1), 0,
- sizeof(table->CTable) - ((maxSymbolValue + 1) * sizeof(HUF_CElt)));
+ }
+ /* Zero unused symbols in CTable, so we can check it for validity */
+ {
+ size_t const ctableSize = HUF_CTABLE_SIZE_ST(maxSymbolValue);
+ size_t const unusedSize = sizeof(table->CTable) - ctableSize * sizeof(HUF_CElt);
+ ZSTD_memset(table->CTable + ctableSize, 0, unusedSize);
}
/* Write table description header */
@@ -859,19 +1287,20 @@ size_t HUF_compress1X_wksp (void* dst, size_t dstSize,
return HUF_compress_internal(dst, dstSize, src, srcSize,
maxSymbolValue, huffLog, HUF_singleStream,
workSpace, wkspSize,
- NULL, NULL, 0, 0 /*bmi2*/);
+ NULL, NULL, 0, 0 /*bmi2*/, 0);
}
size_t HUF_compress1X_repeat (void* dst, size_t dstSize,
const void* src, size_t srcSize,
unsigned maxSymbolValue, unsigned huffLog,
void* workSpace, size_t wkspSize,
- HUF_CElt* hufTable, HUF_repeat* repeat, int preferRepeat, int bmi2)
+ HUF_CElt* hufTable, HUF_repeat* repeat, int preferRepeat,
+ int bmi2, unsigned suspectUncompressible)
{
return HUF_compress_internal(dst, dstSize, src, srcSize,
maxSymbolValue, huffLog, HUF_singleStream,
workSpace, wkspSize, hufTable,
- repeat, preferRepeat, bmi2);
+ repeat, preferRepeat, bmi2, suspectUncompressible);
}
/* HUF_compress4X_repeat():
@@ -885,21 +1314,22 @@ size_t HUF_compress4X_wksp (void* dst, size_t dstSize,
return HUF_compress_internal(dst, dstSize, src, srcSize,
maxSymbolValue, huffLog, HUF_fourStreams,
workSpace, wkspSize,
- NULL, NULL, 0, 0 /*bmi2*/);
+ NULL, NULL, 0, 0 /*bmi2*/, 0);
}
/* HUF_compress4X_repeat():
* compress input using 4 streams.
+ * consider skipping quickly
* re-use an existing huffman compression table */
size_t HUF_compress4X_repeat (void* dst, size_t dstSize,
const void* src, size_t srcSize,
unsigned maxSymbolValue, unsigned huffLog,
void* workSpace, size_t wkspSize,
- HUF_CElt* hufTable, HUF_repeat* repeat, int preferRepeat, int bmi2)
+ HUF_CElt* hufTable, HUF_repeat* repeat, int preferRepeat, int bmi2, unsigned suspectUncompressible)
{
return HUF_compress_internal(dst, dstSize, src, srcSize,
maxSymbolValue, huffLog, HUF_fourStreams,
workSpace, wkspSize,
- hufTable, repeat, preferRepeat, bmi2);
+ hufTable, repeat, preferRepeat, bmi2, suspectUncompressible);
}
diff --git a/lib/zstd/compress/zstd_compress.c b/lib/zstd/compress/zstd_compress.c
index a4e9160..f620caf 100644
--- a/lib/zstd/compress/zstd_compress.c
+++ b/lib/zstd/compress/zstd_compress.c
@@ -12,7 +12,6 @@
* Dependencies
***************************************/
#include "../common/zstd_deps.h" /* INT_MAX, ZSTD_memset, ZSTD_memcpy */
-#include "../common/cpu.h"
#include "../common/mem.h"
#include "hist.h" /* HIST_countFast_wksp */
#define FSE_STATIC_LINKING_ONLY /* FSE_encodeSymbol */
@@ -39,6 +38,18 @@
* Note that functions with explicit context such as ZSTD_compressCCtx() are unaffected.
*/
+/*!
+ * ZSTD_HASHLOG3_MAX :
+ * Maximum size of the hash table dedicated to find 3-bytes matches,
+ * in log format, aka 17 => 1 << 17 == 128Ki positions.
+ * This structure is only used in zstd_opt.
+ * Since allocation is centralized for all strategies, it has to be known here.
+ * The actual (selected) size of the hash table is then stored in ZSTD_matchState_t.hashLog3,
+ * so that zstd_opt.c doesn't need to know about this constant.
+ */
+#ifndef ZSTD_HASHLOG3_MAX
+# define ZSTD_HASHLOG3_MAX 17
+#endif
/*-*************************************
* Helper functions
@@ -69,6 +80,10 @@ struct ZSTD_CDict_s {
ZSTD_customMem customMem;
U32 dictID;
int compressionLevel; /* 0 indicates that advanced API was used to select CDict params */
+ ZSTD_paramSwitch_e useRowMatchFinder; /* Indicates whether the CDict was created with params that would use
+ * row-based matchfinder. Unless the cdict is reloaded, we will use
+ * the same greedy/lazy matchfinder at compression time.
+ */
}; /* typedef'd to ZSTD_CDict within "zstd.h" */
ZSTD_CCtx* ZSTD_createCCtx(void)
@@ -81,7 +96,7 @@ static void ZSTD_initCCtx(ZSTD_CCtx* cctx, ZSTD_customMem memManager)
assert(cctx != NULL);
ZSTD_memset(cctx, 0, sizeof(*cctx));
cctx->customMem = memManager;
- cctx->bmi2 = ZSTD_cpuid_bmi2(ZSTD_cpuid());
+ cctx->bmi2 = ZSTD_cpuSupportsBmi2();
{ size_t const err = ZSTD_CCtx_reset(cctx, ZSTD_reset_parameters);
assert(!ZSTD_isError(err));
(void)err;
@@ -192,12 +207,64 @@ size_t ZSTD_sizeof_CStream(const ZSTD_CStream* zcs)
/* private API call, for dictBuilder only */
const seqStore_t* ZSTD_getSeqStore(const ZSTD_CCtx* ctx) { return &(ctx->seqStore); }
+/* Returns true if the strategy supports using a row based matchfinder */
+static int ZSTD_rowMatchFinderSupported(const ZSTD_strategy strategy) {
+ return (strategy >= ZSTD_greedy && strategy <= ZSTD_lazy2);
+}
+
+/* Returns true if the strategy and useRowMatchFinder mode indicate that we will use the row based matchfinder
+ * for this compression.
+ */
+static int ZSTD_rowMatchFinderUsed(const ZSTD_strategy strategy, const ZSTD_paramSwitch_e mode) {
+ assert(mode != ZSTD_ps_auto);
+ return ZSTD_rowMatchFinderSupported(strategy) && (mode == ZSTD_ps_enable);
+}
+
+/* Returns row matchfinder usage given an initial mode and cParams */
+static ZSTD_paramSwitch_e ZSTD_resolveRowMatchFinderMode(ZSTD_paramSwitch_e mode,
+ const ZSTD_compressionParameters* const cParams) {
+#if defined(ZSTD_ARCH_X86_SSE2) || defined(ZSTD_ARCH_ARM_NEON)
+ int const kHasSIMD128 = 1;
+#else
+ int const kHasSIMD128 = 0;
+#endif
+ if (mode != ZSTD_ps_auto) return mode; /* if requested enabled, but no SIMD, we still will use row matchfinder */
+ mode = ZSTD_ps_disable;
+ if (!ZSTD_rowMatchFinderSupported(cParams->strategy)) return mode;
+ if (kHasSIMD128) {
+ if (cParams->windowLog > 14) mode = ZSTD_ps_enable;
+ } else {
+ if (cParams->windowLog > 17) mode = ZSTD_ps_enable;
+ }
+ return mode;
+}
+
+/* Returns block splitter usage (generally speaking, when using slower/stronger compression modes) */
+static ZSTD_paramSwitch_e ZSTD_resolveBlockSplitterMode(ZSTD_paramSwitch_e mode,
+ const ZSTD_compressionParameters* const cParams) {
+ if (mode != ZSTD_ps_auto) return mode;
+ return (cParams->strategy >= ZSTD_btopt && cParams->windowLog >= 17) ? ZSTD_ps_enable : ZSTD_ps_disable;
+}
+
+/* Returns 1 if the arguments indicate that we should allocate a chainTable, 0 otherwise */
+static int ZSTD_allocateChainTable(const ZSTD_strategy strategy,
+ const ZSTD_paramSwitch_e useRowMatchFinder,
+ const U32 forDDSDict) {
+ assert(useRowMatchFinder != ZSTD_ps_auto);
+ /* We always should allocate a chaintable if we are allocating a matchstate for a DDS dictionary matchstate.
+ * We do not allocate a chaintable if we are using ZSTD_fast, or are using the row-based matchfinder.
+ */
+ return forDDSDict || ((strategy != ZSTD_fast) && !ZSTD_rowMatchFinderUsed(strategy, useRowMatchFinder));
+}
+
/* Returns 1 if compression parameters are such that we should
* enable long distance matching (wlog >= 27, strategy >= btopt).
* Returns 0 otherwise.
*/
-static U32 ZSTD_CParams_shouldEnableLdm(const ZSTD_compressionParameters* const cParams) {
- return cParams->strategy >= ZSTD_btopt && cParams->windowLog >= 27;
+static ZSTD_paramSwitch_e ZSTD_resolveEnableLdm(ZSTD_paramSwitch_e mode,
+ const ZSTD_compressionParameters* const cParams) {
+ if (mode != ZSTD_ps_auto) return mode;
+ return (cParams->strategy >= ZSTD_btopt && cParams->windowLog >= 27) ? ZSTD_ps_enable : ZSTD_ps_disable;
}
static ZSTD_CCtx_params ZSTD_makeCCtxParamsFromCParams(
@@ -208,15 +275,15 @@ static ZSTD_CCtx_params ZSTD_makeCCtxParamsFromCParams(
ZSTD_CCtxParams_init(&cctxParams, ZSTD_CLEVEL_DEFAULT);
cctxParams.cParams = cParams;
- if (ZSTD_CParams_shouldEnableLdm(&cParams)) {
- DEBUGLOG(4, "ZSTD_makeCCtxParamsFromCParams(): Including LDM into cctx params");
- cctxParams.ldmParams.enableLdm = 1;
- /* LDM is enabled by default for optimal parser and window size >= 128MB */
+ /* Adjust advanced params according to cParams */
+ cctxParams.ldmParams.enableLdm = ZSTD_resolveEnableLdm(cctxParams.ldmParams.enableLdm, &cParams);
+ if (cctxParams.ldmParams.enableLdm == ZSTD_ps_enable) {
ZSTD_ldm_adjustParameters(&cctxParams.ldmParams, &cParams);
assert(cctxParams.ldmParams.hashLog >= cctxParams.ldmParams.bucketSizeLog);
assert(cctxParams.ldmParams.hashRateLog < 32);
}
-
+ cctxParams.useBlockSplitter = ZSTD_resolveBlockSplitterMode(cctxParams.useBlockSplitter, &cParams);
+ cctxParams.useRowMatchFinder = ZSTD_resolveRowMatchFinderMode(cctxParams.useRowMatchFinder, &cParams);
assert(!ZSTD_checkCParams(cParams));
return cctxParams;
}
@@ -275,6 +342,11 @@ static void ZSTD_CCtxParams_init_internal(ZSTD_CCtx_params* cctxParams, ZSTD_par
* But, set it for tracing anyway.
*/
cctxParams->compressionLevel = compressionLevel;
+ cctxParams->useRowMatchFinder = ZSTD_resolveRowMatchFinderMode(cctxParams->useRowMatchFinder, ¶ms->cParams);
+ cctxParams->useBlockSplitter = ZSTD_resolveBlockSplitterMode(cctxParams->useBlockSplitter, ¶ms->cParams);
+ cctxParams->ldmParams.enableLdm = ZSTD_resolveEnableLdm(cctxParams->ldmParams.enableLdm, ¶ms->cParams);
+ DEBUGLOG(4, "ZSTD_CCtxParams_init_internal: useRowMatchFinder=%d, useBlockSplitter=%d ldm=%d",
+ cctxParams->useRowMatchFinder, cctxParams->useBlockSplitter, cctxParams->ldmParams.enableLdm);
}
size_t ZSTD_CCtxParams_init_advanced(ZSTD_CCtx_params* cctxParams, ZSTD_parameters params)
@@ -431,9 +503,9 @@ ZSTD_bounds ZSTD_cParam_getBounds(ZSTD_cParameter param)
return bounds;
case ZSTD_c_literalCompressionMode:
- ZSTD_STATIC_ASSERT(ZSTD_lcm_auto < ZSTD_lcm_huffman && ZSTD_lcm_huffman < ZSTD_lcm_uncompressed);
- bounds.lowerBound = ZSTD_lcm_auto;
- bounds.upperBound = ZSTD_lcm_uncompressed;
+ ZSTD_STATIC_ASSERT(ZSTD_ps_auto < ZSTD_ps_enable && ZSTD_ps_enable < ZSTD_ps_disable);
+ bounds.lowerBound = (int)ZSTD_ps_auto;
+ bounds.upperBound = (int)ZSTD_ps_disable;
return bounds;
case ZSTD_c_targetCBlockSize:
@@ -462,6 +534,21 @@ ZSTD_bounds ZSTD_cParam_getBounds(ZSTD_cParameter param)
bounds.upperBound = 1;
return bounds;
+ case ZSTD_c_useBlockSplitter:
+ bounds.lowerBound = (int)ZSTD_ps_auto;
+ bounds.upperBound = (int)ZSTD_ps_disable;
+ return bounds;
+
+ case ZSTD_c_useRowMatchFinder:
+ bounds.lowerBound = (int)ZSTD_ps_auto;
+ bounds.upperBound = (int)ZSTD_ps_disable;
+ return bounds;
+
+ case ZSTD_c_deterministicRefPrefix:
+ bounds.lowerBound = 0;
+ bounds.upperBound = 1;
+ return bounds;
+
default:
bounds.error = ERROR(parameter_unsupported);
return bounds;
@@ -523,6 +610,9 @@ static int ZSTD_isUpdateAuthorized(ZSTD_cParameter param)
case ZSTD_c_stableOutBuffer:
case ZSTD_c_blockDelimiters:
case ZSTD_c_validateSequences:
+ case ZSTD_c_useBlockSplitter:
+ case ZSTD_c_useRowMatchFinder:
+ case ZSTD_c_deterministicRefPrefix:
default:
return 0;
}
@@ -575,6 +665,9 @@ size_t ZSTD_CCtx_setParameter(ZSTD_CCtx* cctx, ZSTD_cParameter param, int value)
case ZSTD_c_stableOutBuffer:
case ZSTD_c_blockDelimiters:
case ZSTD_c_validateSequences:
+ case ZSTD_c_useBlockSplitter:
+ case ZSTD_c_useRowMatchFinder:
+ case ZSTD_c_deterministicRefPrefix:
break;
default: RETURN_ERROR(parameter_unsupported, "unknown parameter");
@@ -672,7 +765,7 @@ size_t ZSTD_CCtxParams_setParameter(ZSTD_CCtx_params* CCtxParams,
}
case ZSTD_c_literalCompressionMode : {
- const ZSTD_literalCompressionMode_e lcm = (ZSTD_literalCompressionMode_e)value;
+ const ZSTD_paramSwitch_e lcm = (ZSTD_paramSwitch_e)value;
BOUNDCHECK(ZSTD_c_literalCompressionMode, lcm);
CCtxParams->literalCompressionMode = lcm;
return CCtxParams->literalCompressionMode;
@@ -699,7 +792,7 @@ size_t ZSTD_CCtxParams_setParameter(ZSTD_CCtx_params* CCtxParams,
return CCtxParams->enableDedicatedDictSearch;
case ZSTD_c_enableLongDistanceMatching :
- CCtxParams->ldmParams.enableLdm = (value!=0);
+ CCtxParams->ldmParams.enableLdm = (ZSTD_paramSwitch_e)value;
return CCtxParams->ldmParams.enableLdm;
case ZSTD_c_ldmHashLog :
@@ -758,6 +851,21 @@ size_t ZSTD_CCtxParams_setParameter(ZSTD_CCtx_params* CCtxParams,
CCtxParams->validateSequences = value;
return CCtxParams->validateSequences;
+ case ZSTD_c_useBlockSplitter:
+ BOUNDCHECK(ZSTD_c_useBlockSplitter, value);
+ CCtxParams->useBlockSplitter = (ZSTD_paramSwitch_e)value;
+ return CCtxParams->useBlockSplitter;
+
+ case ZSTD_c_useRowMatchFinder:
+ BOUNDCHECK(ZSTD_c_useRowMatchFinder, value);
+ CCtxParams->useRowMatchFinder = (ZSTD_paramSwitch_e)value;
+ return CCtxParams->useRowMatchFinder;
+
+ case ZSTD_c_deterministicRefPrefix:
+ BOUNDCHECK(ZSTD_c_deterministicRefPrefix, value);
+ CCtxParams->deterministicRefPrefix = !!value;
+ return CCtxParams->deterministicRefPrefix;
+
default: RETURN_ERROR(parameter_unsupported, "unknown parameter");
}
}
@@ -863,6 +971,15 @@ size_t ZSTD_CCtxParams_getParameter(
case ZSTD_c_validateSequences :
*value = (int)CCtxParams->validateSequences;
break;
+ case ZSTD_c_useBlockSplitter :
+ *value = (int)CCtxParams->useBlockSplitter;
+ break;
+ case ZSTD_c_useRowMatchFinder :
+ *value = (int)CCtxParams->useRowMatchFinder;
+ break;
+ case ZSTD_c_deterministicRefPrefix:
+ *value = (int)CCtxParams->deterministicRefPrefix;
+ break;
default: RETURN_ERROR(parameter_unsupported, "unknown parameter");
}
return 0;
@@ -889,7 +1006,7 @@ size_t ZSTD_CCtx_setParametersUsingCCtxParams(
return 0;
}
-ZSTDLIB_API size_t ZSTD_CCtx_setPledgedSrcSize(ZSTD_CCtx* cctx, unsigned long long pledgedSrcSize)
+size_t ZSTD_CCtx_setPledgedSrcSize(ZSTD_CCtx* cctx, unsigned long long pledgedSrcSize)
{
DEBUGLOG(4, "ZSTD_CCtx_setPledgedSrcSize to %u bytes", (U32)pledgedSrcSize);
RETURN_ERROR_IF(cctx->streamStage != zcss_init, stage_wrong,
@@ -969,14 +1086,14 @@ size_t ZSTD_CCtx_loadDictionary_advanced(
return 0;
}
-ZSTDLIB_API size_t ZSTD_CCtx_loadDictionary_byReference(
+size_t ZSTD_CCtx_loadDictionary_byReference(
ZSTD_CCtx* cctx, const void* dict, size_t dictSize)
{
return ZSTD_CCtx_loadDictionary_advanced(
cctx, dict, dictSize, ZSTD_dlm_byRef, ZSTD_dct_auto);
}
-ZSTDLIB_API size_t ZSTD_CCtx_loadDictionary(ZSTD_CCtx* cctx, const void* dict, size_t dictSize)
+size_t ZSTD_CCtx_loadDictionary(ZSTD_CCtx* cctx, const void* dict, size_t dictSize)
{
return ZSTD_CCtx_loadDictionary_advanced(
cctx, dict, dictSize, ZSTD_dlm_byCopy, ZSTD_dct_auto);
@@ -1146,7 +1263,7 @@ ZSTD_adjustCParams_internal(ZSTD_compressionParameters cPar,
break;
case ZSTD_cpm_createCDict:
/* Assume a small source size when creating a dictionary
- * with an unkown source size.
+ * with an unknown source size.
*/
if (dictSize && srcSize == ZSTD_CONTENTSIZE_UNKNOWN)
srcSize = minSrcSize;
@@ -1220,7 +1337,7 @@ ZSTD_compressionParameters ZSTD_getCParamsFromCCtxParams(
srcSizeHint = CCtxParams->srcSizeHint;
}
cParams = ZSTD_getCParams_internal(CCtxParams->compressionLevel, srcSizeHint, dictSize, mode);
- if (CCtxParams->ldmParams.enableLdm) cParams.windowLog = ZSTD_LDM_DEFAULT_WINDOW_LOG;
+ if (CCtxParams->ldmParams.enableLdm == ZSTD_ps_enable) cParams.windowLog = ZSTD_LDM_DEFAULT_WINDOW_LOG;
ZSTD_overrideCParams(&cParams, &CCtxParams->cParams);
assert(!ZSTD_checkCParams(cParams));
/* srcSizeHint == 0 means 0 */
@@ -1229,9 +1346,14 @@ ZSTD_compressionParameters ZSTD_getCParamsFromCCtxParams(
static size_t
ZSTD_sizeof_matchState(const ZSTD_compressionParameters* const cParams,
+ const ZSTD_paramSwitch_e useRowMatchFinder,
+ const U32 enableDedicatedDictSearch,
const U32 forCCtx)
{
- size_t const chainSize = (cParams->strategy == ZSTD_fast) ? 0 : ((size_t)1 << cParams->chainLog);
+ /* chain table size should be 0 for fast or row-hash strategies */
+ size_t const chainSize = ZSTD_allocateChainTable(cParams->strategy, useRowMatchFinder, enableDedicatedDictSearch && !forCCtx)
+ ? ((size_t)1 << cParams->chainLog)
+ : 0;
size_t const hSize = ((size_t)1) << cParams->hashLog;
U32 const hashLog3 = (forCCtx && cParams->minMatch==3) ? MIN(ZSTD_HASHLOG3_MAX, cParams->windowLog) : 0;
size_t const h3Size = hashLog3 ? ((size_t)1) << hashLog3 : 0;
@@ -1241,43 +1363,53 @@ ZSTD_sizeof_matchState(const ZSTD_compressionParameters* const cParams,
+ hSize * sizeof(U32)
+ h3Size * sizeof(U32);
size_t const optPotentialSpace =
- ZSTD_cwksp_alloc_size((MaxML+1) * sizeof(U32))
- + ZSTD_cwksp_alloc_size((MaxLL+1) * sizeof(U32))
- + ZSTD_cwksp_alloc_size((MaxOff+1) * sizeof(U32))
- + ZSTD_cwksp_alloc_size((1<<Litbits) * sizeof(U32))
- + ZSTD_cwksp_alloc_size((ZSTD_OPT_NUM+1) * sizeof(ZSTD_match_t))
- + ZSTD_cwksp_alloc_size((ZSTD_OPT_NUM+1) * sizeof(ZSTD_optimal_t));
+ ZSTD_cwksp_aligned_alloc_size((MaxML+1) * sizeof(U32))
+ + ZSTD_cwksp_aligned_alloc_size((MaxLL+1) * sizeof(U32))
+ + ZSTD_cwksp_aligned_alloc_size((MaxOff+1) * sizeof(U32))
+ + ZSTD_cwksp_aligned_alloc_size((1<<Litbits) * sizeof(U32))
+ + ZSTD_cwksp_aligned_alloc_size((ZSTD_OPT_NUM+1) * sizeof(ZSTD_match_t))
+ + ZSTD_cwksp_aligned_alloc_size((ZSTD_OPT_NUM+1) * sizeof(ZSTD_optimal_t));
+ size_t const lazyAdditionalSpace = ZSTD_rowMatchFinderUsed(cParams->strategy, useRowMatchFinder)
+ ? ZSTD_cwksp_aligned_alloc_size(hSize*sizeof(U16))
+ : 0;
size_t const optSpace = (forCCtx && (cParams->strategy >= ZSTD_btopt))
? optPotentialSpace
: 0;
+ size_t const slackSpace = ZSTD_cwksp_slack_space_required();
+
+ /* tables are guaranteed to be sized in multiples of 64 bytes (or 16 uint32_t) */
+ ZSTD_STATIC_ASSERT(ZSTD_HASHLOG_MIN >= 4 && ZSTD_WINDOWLOG_MIN >= 4 && ZSTD_CHAINLOG_MIN >= 4);
+ assert(useRowMatchFinder != ZSTD_ps_auto);
+
DEBUGLOG(4, "chainSize: %u - hSize: %u - h3Size: %u",
(U32)chainSize, (U32)hSize, (U32)h3Size);
- return tableSpace + optSpace;
+ return tableSpace + optSpace + slackSpace + lazyAdditionalSpace;
}
static size_t ZSTD_estimateCCtxSize_usingCCtxParams_internal(
const ZSTD_compressionParameters* cParams,
const ldmParams_t* ldmParams,
const int isStatic,
+ const ZSTD_paramSwitch_e useRowMatchFinder,
const size_t buffInSize,
const size_t buffOutSize,
const U64 pledgedSrcSize)
{
- size_t const windowSize = MAX(1, (size_t)MIN(((U64)1 << cParams->windowLog), pledgedSrcSize));
+ size_t const windowSize = (size_t) BOUNDED(1ULL, 1ULL << cParams->windowLog, pledgedSrcSize);
size_t const blockSize = MIN(ZSTD_BLOCKSIZE_MAX, windowSize);
U32 const divider = (cParams->minMatch==3) ? 3 : 4;
size_t const maxNbSeq = blockSize / divider;
size_t const tokenSpace = ZSTD_cwksp_alloc_size(WILDCOPY_OVERLENGTH + blockSize)
- + ZSTD_cwksp_alloc_size(maxNbSeq * sizeof(seqDef))
+ + ZSTD_cwksp_aligned_alloc_size(maxNbSeq * sizeof(seqDef))
+ 3 * ZSTD_cwksp_alloc_size(maxNbSeq * sizeof(BYTE));
size_t const entropySpace = ZSTD_cwksp_alloc_size(ENTROPY_WORKSPACE_SIZE);
size_t const blockStateSpace = 2 * ZSTD_cwksp_alloc_size(sizeof(ZSTD_compressedBlockState_t));
- size_t const matchStateSize = ZSTD_sizeof_matchState(cParams, /* forCCtx */ 1);
+ size_t const matchStateSize = ZSTD_sizeof_matchState(cParams, useRowMatchFinder, /* enableDedicatedDictSearch */ 0, /* forCCtx */ 1);
size_t const ldmSpace = ZSTD_ldm_getTableSize(*ldmParams);
size_t const maxNbLdmSeq = ZSTD_ldm_getMaxNbSeq(*ldmParams, blockSize);
- size_t const ldmSeqSpace = ldmParams->enableLdm ?
- ZSTD_cwksp_alloc_size(maxNbLdmSeq * sizeof(rawSeq)) : 0;
+ size_t const ldmSeqSpace = ldmParams->enableLdm == ZSTD_ps_enable ?
+ ZSTD_cwksp_aligned_alloc_size(maxNbLdmSeq * sizeof(rawSeq)) : 0;
size_t const bufferSpace = ZSTD_cwksp_alloc_size(buffInSize)
@@ -1303,19 +1435,32 @@ size_t ZSTD_estimateCCtxSize_usingCCtxParams(const ZSTD_CCtx_params* params)
{
ZSTD_compressionParameters const cParams =
ZSTD_getCParamsFromCCtxParams(params, ZSTD_CONTENTSIZE_UNKNOWN, 0, ZSTD_cpm_noAttachDict);
+ ZSTD_paramSwitch_e const useRowMatchFinder = ZSTD_resolveRowMatchFinderMode(params->useRowMatchFinder,
+ &cParams);
RETURN_ERROR_IF(params->nbWorkers > 0, GENERIC, "Estimate CCtx size is supported for single-threaded compression only.");
/* estimateCCtxSize is for one-shot compression. So no buffers should
* be needed. However, we still allocate two 0-sized buffers, which can
* take space under ASAN. */
return ZSTD_estimateCCtxSize_usingCCtxParams_internal(
- &cParams, ¶ms->ldmParams, 1, 0, 0, ZSTD_CONTENTSIZE_UNKNOWN);
+ &cParams, ¶ms->ldmParams, 1, useRowMatchFinder, 0, 0, ZSTD_CONTENTSIZE_UNKNOWN);
}
size_t ZSTD_estimateCCtxSize_usingCParams(ZSTD_compressionParameters cParams)
{
- ZSTD_CCtx_params const params = ZSTD_makeCCtxParamsFromCParams(cParams);
- return ZSTD_estimateCCtxSize_usingCCtxParams(¶ms);
+ ZSTD_CCtx_params initialParams = ZSTD_makeCCtxParamsFromCParams(cParams);
+ if (ZSTD_rowMatchFinderSupported(cParams.strategy)) {
+ /* Pick bigger of not using and using row-based matchfinder for greedy and lazy strategies */
+ size_t noRowCCtxSize;
+ size_t rowCCtxSize;
+ initialParams.useRowMatchFinder = ZSTD_ps_disable;
+ noRowCCtxSize = ZSTD_estimateCCtxSize_usingCCtxParams(&initialParams);
+ initialParams.useRowMatchFinder = ZSTD_ps_enable;
+ rowCCtxSize = ZSTD_estimateCCtxSize_usingCCtxParams(&initialParams);
+ return MAX(noRowCCtxSize, rowCCtxSize);
+ } else {
+ return ZSTD_estimateCCtxSize_usingCCtxParams(&initialParams);
+ }
}
static size_t ZSTD_estimateCCtxSize_internal(int compressionLevel)
@@ -1355,17 +1500,29 @@ size_t ZSTD_estimateCStreamSize_usingCCtxParams(const ZSTD_CCtx_params* params)
size_t const outBuffSize = (params->outBufferMode == ZSTD_bm_buffered)
? ZSTD_compressBound(blockSize) + 1
: 0;
+ ZSTD_paramSwitch_e const useRowMatchFinder = ZSTD_resolveRowMatchFinderMode(params->useRowMatchFinder, ¶ms->cParams);
return ZSTD_estimateCCtxSize_usingCCtxParams_internal(
- &cParams, ¶ms->ldmParams, 1, inBuffSize, outBuffSize,
+ &cParams, ¶ms->ldmParams, 1, useRowMatchFinder, inBuffSize, outBuffSize,
ZSTD_CONTENTSIZE_UNKNOWN);
}
}
size_t ZSTD_estimateCStreamSize_usingCParams(ZSTD_compressionParameters cParams)
{
- ZSTD_CCtx_params const params = ZSTD_makeCCtxParamsFromCParams(cParams);
- return ZSTD_estimateCStreamSize_usingCCtxParams(¶ms);
+ ZSTD_CCtx_params initialParams = ZSTD_makeCCtxParamsFromCParams(cParams);
+ if (ZSTD_rowMatchFinderSupported(cParams.strategy)) {
+ /* Pick bigger of not using and using row-based matchfinder for greedy and lazy strategies */
+ size_t noRowCCtxSize;
+ size_t rowCCtxSize;
+ initialParams.useRowMatchFinder = ZSTD_ps_disable;
+ noRowCCtxSize = ZSTD_estimateCStreamSize_usingCCtxParams(&initialParams);
+ initialParams.useRowMatchFinder = ZSTD_ps_enable;
+ rowCCtxSize = ZSTD_estimateCStreamSize_usingCCtxParams(&initialParams);
+ return MAX(noRowCCtxSize, rowCCtxSize);
+ } else {
+ return ZSTD_estimateCStreamSize_usingCCtxParams(&initialParams);
+ }
}
static size_t ZSTD_estimateCStreamSize_internal(int compressionLevel)
@@ -1480,20 +1637,27 @@ typedef enum {
ZSTD_resetTarget_CCtx
} ZSTD_resetTarget_e;
+
static size_t
ZSTD_reset_matchState(ZSTD_matchState_t* ms,
ZSTD_cwksp* ws,
const ZSTD_compressionParameters* cParams,
+ const ZSTD_paramSwitch_e useRowMatchFinder,
const ZSTD_compResetPolicy_e crp,
const ZSTD_indexResetPolicy_e forceResetIndex,
const ZSTD_resetTarget_e forWho)
{
- size_t const chainSize = (cParams->strategy == ZSTD_fast) ? 0 : ((size_t)1 << cParams->chainLog);
+ /* disable chain table allocation for fast or row-based strategies */
+ size_t const chainSize = ZSTD_allocateChainTable(cParams->strategy, useRowMatchFinder,
+ ms->dedicatedDictSearch && (forWho == ZSTD_resetTarget_CDict))
+ ? ((size_t)1 << cParams->chainLog)
+ : 0;
size_t const hSize = ((size_t)1) << cParams->hashLog;
U32 const hashLog3 = ((forWho == ZSTD_resetTarget_CCtx) && cParams->minMatch==3) ? MIN(ZSTD_HASHLOG3_MAX, cParams->windowLog) : 0;
size_t const h3Size = hashLog3 ? ((size_t)1) << hashLog3 : 0;
DEBUGLOG(4, "reset indices : %u", forceResetIndex == ZSTDirp_reset);
+ assert(useRowMatchFinder != ZSTD_ps_auto);
if (forceResetIndex == ZSTDirp_reset) {
ZSTD_window_init(&ms->window);
ZSTD_cwksp_mark_tables_dirty(ws);
@@ -1532,11 +1696,23 @@ ZSTD_reset_matchState(ZSTD_matchState_t* ms,
ms->opt.priceTable = (ZSTD_optimal_t*)ZSTD_cwksp_reserve_aligned(ws, (ZSTD_OPT_NUM+1) * sizeof(ZSTD_optimal_t));
}
+ if (ZSTD_rowMatchFinderUsed(cParams->strategy, useRowMatchFinder)) {
+ { /* Row match finder needs an additional table of hashes ("tags") */
+ size_t const tagTableSize = hSize*sizeof(U16);
+ ms->tagTable = (U16*)ZSTD_cwksp_reserve_aligned(ws, tagTableSize);
+ if (ms->tagTable) ZSTD_memset(ms->tagTable, 0, tagTableSize);
+ }
+ { /* Switch to 32-entry rows if searchLog is 5 (or more) */
+ U32 const rowLog = BOUNDED(4, cParams->searchLog, 6);
+ assert(cParams->hashLog >= rowLog);
+ ms->rowHashLog = cParams->hashLog - rowLog;
+ }
+ }
+
ms->cParams = *cParams;
RETURN_ERROR_IF(ZSTD_cwksp_reserve_failed(ws), memory_allocation,
"failed a workspace allocation in ZSTD_reset_matchState");
-
return 0;
}
@@ -1553,61 +1729,87 @@ static int ZSTD_indexTooCloseToMax(ZSTD_window_t w)
return (size_t)(w.nextSrc - w.base) > (ZSTD_CURRENT_MAX - ZSTD_INDEXOVERFLOW_MARGIN);
}
+/* ZSTD_dictTooBig():
+ * When dictionaries are larger than ZSTD_CHUNKSIZE_MAX they can't be loaded in
+ * one go generically. So we ensure that in that case we reset the tables to zero,
+ * so that we can load as much of the dictionary as possible.
+ */
+static int ZSTD_dictTooBig(size_t const loadedDictSize)
+{
+ return loadedDictSize > ZSTD_CHUNKSIZE_MAX;
+}
+
/*! ZSTD_resetCCtx_internal() :
- note : `params` are assumed fully validated at this stage */
+ * @param loadedDictSize The size of the dictionary to be loaded
+ * into the context, if any. If no dictionary is used, or the
+ * dictionary is being attached / copied, then pass 0.
+ * note : `params` are assumed fully validated at this stage.
+ */
static size_t ZSTD_resetCCtx_internal(ZSTD_CCtx* zc,
- ZSTD_CCtx_params params,
+ ZSTD_CCtx_params const* params,
U64 const pledgedSrcSize,
+ size_t const loadedDictSize,
ZSTD_compResetPolicy_e const crp,
ZSTD_buffered_policy_e const zbuff)
{
ZSTD_cwksp* const ws = &zc->workspace;
- DEBUGLOG(4, "ZSTD_resetCCtx_internal: pledgedSrcSize=%u, wlog=%u",
- (U32)pledgedSrcSize, params.cParams.windowLog);
- assert(!ZSTD_isError(ZSTD_checkCParams(params.cParams)));
+ DEBUGLOG(4, "ZSTD_resetCCtx_internal: pledgedSrcSize=%u, wlog=%u, useRowMatchFinder=%d useBlockSplitter=%d",
+ (U32)pledgedSrcSize, params->cParams.windowLog, (int)params->useRowMatchFinder, (int)params->useBlockSplitter);
+ assert(!ZSTD_isError(ZSTD_checkCParams(params->cParams)));
zc->isFirstBlock = 1;
- if (params.ldmParams.enableLdm) {
+ /* Set applied params early so we can modify them for LDM,
+ * and point params at the applied params.
+ */
+ zc->appliedParams = *params;
+ params = &zc->appliedParams;
+
+ assert(params->useRowMatchFinder != ZSTD_ps_auto);
+ assert(params->useBlockSplitter != ZSTD_ps_auto);
+ assert(params->ldmParams.enableLdm != ZSTD_ps_auto);
+ if (params->ldmParams.enableLdm == ZSTD_ps_enable) {
/* Adjust long distance matching parameters */
- ZSTD_ldm_adjustParameters(¶ms.ldmParams, ¶ms.cParams);
- assert(params.ldmParams.hashLog >= params.ldmParams.bucketSizeLog);
- assert(params.ldmParams.hashRateLog < 32);
+ ZSTD_ldm_adjustParameters(&zc->appliedParams.ldmParams, ¶ms->cParams);
+ assert(params->ldmParams.hashLog >= params->ldmParams.bucketSizeLog);
+ assert(params->ldmParams.hashRateLog < 32);
}
- { size_t const windowSize = MAX(1, (size_t)MIN(((U64)1 << params.cParams.windowLog), pledgedSrcSize));
+ { size_t const windowSize = MAX(1, (size_t)MIN(((U64)1 << params->cParams.windowLog), pledgedSrcSize));
size_t const blockSize = MIN(ZSTD_BLOCKSIZE_MAX, windowSize);
- U32 const divider = (params.cParams.minMatch==3) ? 3 : 4;
+ U32 const divider = (params->cParams.minMatch==3) ? 3 : 4;
size_t const maxNbSeq = blockSize / divider;
- size_t const buffOutSize = (zbuff == ZSTDb_buffered && params.outBufferMode == ZSTD_bm_buffered)
+ size_t const buffOutSize = (zbuff == ZSTDb_buffered && params->outBufferMode == ZSTD_bm_buffered)
? ZSTD_compressBound(blockSize) + 1
: 0;
- size_t const buffInSize = (zbuff == ZSTDb_buffered && params.inBufferMode == ZSTD_bm_buffered)
+ size_t const buffInSize = (zbuff == ZSTDb_buffered && params->inBufferMode == ZSTD_bm_buffered)
? windowSize + blockSize
: 0;
- size_t const maxNbLdmSeq = ZSTD_ldm_getMaxNbSeq(params.ldmParams, blockSize);
+ size_t const maxNbLdmSeq = ZSTD_ldm_getMaxNbSeq(params->ldmParams, blockSize);
int const indexTooClose = ZSTD_indexTooCloseToMax(zc->blockState.matchState.window);
+ int const dictTooBig = ZSTD_dictTooBig(loadedDictSize);
ZSTD_indexResetPolicy_e needsIndexReset =
- (!indexTooClose && zc->initialized) ? ZSTDirp_continue : ZSTDirp_reset;
+ (indexTooClose || dictTooBig || !zc->initialized) ? ZSTDirp_reset : ZSTDirp_continue;
size_t const neededSpace =
ZSTD_estimateCCtxSize_usingCCtxParams_internal(
- ¶ms.cParams, ¶ms.ldmParams, zc->staticSize != 0,
+ ¶ms->cParams, ¶ms->ldmParams, zc->staticSize != 0, params->useRowMatchFinder,
buffInSize, buffOutSize, pledgedSrcSize);
+ int resizeWorkspace;
+
FORWARD_IF_ERROR(neededSpace, "cctx size estimate failed!");
if (!zc->staticSize) ZSTD_cwksp_bump_oversized_duration(ws, 0);
- /* Check if workspace is large enough, alloc a new one if needed */
- {
+ { /* Check if workspace is large enough, alloc a new one if needed */
int const workspaceTooSmall = ZSTD_cwksp_sizeof(ws) < neededSpace;
int const workspaceWasteful = ZSTD_cwksp_check_wasteful(ws, neededSpace);
-
+ resizeWorkspace = workspaceTooSmall || workspaceWasteful;
DEBUGLOG(4, "Need %zu B workspace", neededSpace);
DEBUGLOG(4, "windowSize: %zu - blockSize: %zu", windowSize, blockSize);
- if (workspaceTooSmall || workspaceWasteful) {
+ if (resizeWorkspace) {
DEBUGLOG(4, "Resize workspaceSize from %zuKB to %zuKB",
ZSTD_cwksp_sizeof(ws) >> 10,
neededSpace >> 10);
@@ -1629,14 +1831,13 @@ static size_t ZSTD_resetCCtx_internal(ZSTD_CCtx* zc,
zc->blockState.nextCBlock = (ZSTD_compressedBlockState_t*) ZSTD_cwksp_reserve_object(ws, sizeof(ZSTD_compressedBlockState_t));
RETURN_ERROR_IF(zc->blockState.nextCBlock == NULL, memory_allocation, "couldn't allocate nextCBlock");
zc->entropyWorkspace = (U32*) ZSTD_cwksp_reserve_object(ws, ENTROPY_WORKSPACE_SIZE);
- RETURN_ERROR_IF(zc->blockState.nextCBlock == NULL, memory_allocation, "couldn't allocate entropyWorkspace");
+ RETURN_ERROR_IF(zc->entropyWorkspace == NULL, memory_allocation, "couldn't allocate entropyWorkspace");
} }
ZSTD_cwksp_clear(ws);
/* init params */
- zc->appliedParams = params;
- zc->blockState.matchState.cParams = params.cParams;
+ zc->blockState.matchState.cParams = params->cParams;
zc->pledgedSrcSizePlusOne = pledgedSrcSize+1;
zc->consumedSrcSize = 0;
zc->producedCSize = 0;
@@ -1667,11 +1868,11 @@ static size_t ZSTD_resetCCtx_internal(ZSTD_CCtx* zc,
zc->outBuff = (char*)ZSTD_cwksp_reserve_buffer(ws, buffOutSize);
/* ldm bucketOffsets table */
- if (params.ldmParams.enableLdm) {
+ if (params->ldmParams.enableLdm == ZSTD_ps_enable) {
/* TODO: avoid memset? */
size_t const numBuckets =
- ((size_t)1) << (params.ldmParams.hashLog -
- params.ldmParams.bucketSizeLog);
+ ((size_t)1) << (params->ldmParams.hashLog -
+ params->ldmParams.bucketSizeLog);
zc->ldmState.bucketOffsets = ZSTD_cwksp_reserve_buffer(ws, numBuckets);
ZSTD_memset(zc->ldmState.bucketOffsets, 0, numBuckets);
}
@@ -1687,32 +1888,28 @@ static size_t ZSTD_resetCCtx_internal(ZSTD_CCtx* zc,
FORWARD_IF_ERROR(ZSTD_reset_matchState(
&zc->blockState.matchState,
ws,
- ¶ms.cParams,
+ ¶ms->cParams,
+ params->useRowMatchFinder,
crp,
needsIndexReset,
ZSTD_resetTarget_CCtx), "");
/* ldm hash table */
- if (params.ldmParams.enableLdm) {
+ if (params->ldmParams.enableLdm == ZSTD_ps_enable) {
/* TODO: avoid memset? */
- size_t const ldmHSize = ((size_t)1) << params.ldmParams.hashLog;
+ size_t const ldmHSize = ((size_t)1) << params->ldmParams.hashLog;
zc->ldmState.hashTable = (ldmEntry_t*)ZSTD_cwksp_reserve_aligned(ws, ldmHSize * sizeof(ldmEntry_t));
ZSTD_memset(zc->ldmState.hashTable, 0, ldmHSize * sizeof(ldmEntry_t));
zc->ldmSequences = (rawSeq*)ZSTD_cwksp_reserve_aligned(ws, maxNbLdmSeq * sizeof(rawSeq));
zc->maxNbLdmSequences = maxNbLdmSeq;
ZSTD_window_init(&zc->ldmState.window);
- ZSTD_window_clear(&zc->ldmState.window);
zc->ldmState.loadedDictEnd = 0;
}
- /* Due to alignment, when reusing a workspace, we can actually consume
- * up to 3 extra bytes for alignment. See the comments in zstd_cwksp.h
- */
- assert(ZSTD_cwksp_used(ws) >= neededSpace &&
- ZSTD_cwksp_used(ws) <= neededSpace + 3);
-
DEBUGLOG(3, "wksp: finished allocating, %zd bytes remain available", ZSTD_cwksp_available_space(ws));
+ assert(ZSTD_cwksp_estimated_space_within_bounds(ws, neededSpace, resizeWorkspace));
+
zc->initialized = 1;
return 0;
@@ -1768,6 +1965,8 @@ ZSTD_resetCCtx_byAttachingCDict(ZSTD_CCtx* cctx,
U64 pledgedSrcSize,
ZSTD_buffered_policy_e zbuff)
{
+ DEBUGLOG(4, "ZSTD_resetCCtx_byAttachingCDict() pledgedSrcSize=%llu",
+ (unsigned long long)pledgedSrcSize);
{
ZSTD_compressionParameters adjusted_cdict_cParams = cdict->matchState.cParams;
unsigned const windowLog = params.cParams.windowLog;
@@ -1783,7 +1982,9 @@ ZSTD_resetCCtx_byAttachingCDict(ZSTD_CCtx* cctx,
params.cParams = ZSTD_adjustCParams_internal(adjusted_cdict_cParams, pledgedSrcSize,
cdict->dictContentSize, ZSTD_cpm_attachDict);
params.cParams.windowLog = windowLog;
- FORWARD_IF_ERROR(ZSTD_resetCCtx_internal(cctx, params, pledgedSrcSize,
+ params.useRowMatchFinder = cdict->useRowMatchFinder; /* cdict overrides */
+ FORWARD_IF_ERROR(ZSTD_resetCCtx_internal(cctx, ¶ms, pledgedSrcSize,
+ /* loadedDictSize */ 0,
ZSTDcrp_makeClean, zbuff), "");
assert(cctx->appliedParams.cParams.strategy == adjusted_cdict_cParams.strategy);
}
@@ -1827,15 +2028,17 @@ static size_t ZSTD_resetCCtx_byCopyingCDict(ZSTD_CCtx* cctx,
const ZSTD_compressionParameters *cdict_cParams = &cdict->matchState.cParams;
assert(!cdict->matchState.dedicatedDictSearch);
-
- DEBUGLOG(4, "copying dictionary into context");
+ DEBUGLOG(4, "ZSTD_resetCCtx_byCopyingCDict() pledgedSrcSize=%llu",
+ (unsigned long long)pledgedSrcSize);
{ unsigned const windowLog = params.cParams.windowLog;
assert(windowLog != 0);
/* Copy only compression parameters related to tables. */
params.cParams = *cdict_cParams;
params.cParams.windowLog = windowLog;
- FORWARD_IF_ERROR(ZSTD_resetCCtx_internal(cctx, params, pledgedSrcSize,
+ params.useRowMatchFinder = cdict->useRowMatchFinder;
+ FORWARD_IF_ERROR(ZSTD_resetCCtx_internal(cctx, ¶ms, pledgedSrcSize,
+ /* loadedDictSize */ 0,
ZSTDcrp_leaveDirty, zbuff), "");
assert(cctx->appliedParams.cParams.strategy == cdict_cParams->strategy);
assert(cctx->appliedParams.cParams.hashLog == cdict_cParams->hashLog);
@@ -1843,17 +2046,30 @@ static size_t ZSTD_resetCCtx_byCopyingCDict(ZSTD_CCtx* cctx,
}
ZSTD_cwksp_mark_tables_dirty(&cctx->workspace);
+ assert(params.useRowMatchFinder != ZSTD_ps_auto);
/* copy tables */
- { size_t const chainSize = (cdict_cParams->strategy == ZSTD_fast) ? 0 : ((size_t)1 << cdict_cParams->chainLog);
+ { size_t const chainSize = ZSTD_allocateChainTable(cdict_cParams->strategy, cdict->useRowMatchFinder, 0 /* DDS guaranteed disabled */)
+ ? ((size_t)1 << cdict_cParams->chainLog)
+ : 0;
size_t const hSize = (size_t)1 << cdict_cParams->hashLog;
ZSTD_memcpy(cctx->blockState.matchState.hashTable,
cdict->matchState.hashTable,
hSize * sizeof(U32));
- ZSTD_memcpy(cctx->blockState.matchState.chainTable,
+ /* Do not copy cdict's chainTable if cctx has parameters such that it would not use chainTable */
+ if (ZSTD_allocateChainTable(cctx->appliedParams.cParams.strategy, cctx->appliedParams.useRowMatchFinder, 0 /* forDDSDict */)) {
+ ZSTD_memcpy(cctx->blockState.matchState.chainTable,
cdict->matchState.chainTable,
chainSize * sizeof(U32));
+ }
+ /* copy tag table */
+ if (ZSTD_rowMatchFinderUsed(cdict_cParams->strategy, cdict->useRowMatchFinder)) {
+ size_t const tagTableSize = hSize*sizeof(U16);
+ ZSTD_memcpy(cctx->blockState.matchState.tagTable,
+ cdict->matchState.tagTable,
+ tagTableSize);
+ }
}
/* Zero the hashTable3, since the cdict never fills it */
@@ -1917,16 +2133,22 @@ static size_t ZSTD_copyCCtx_internal(ZSTD_CCtx* dstCCtx,
U64 pledgedSrcSize,
ZSTD_buffered_policy_e zbuff)
{
- DEBUGLOG(5, "ZSTD_copyCCtx_internal");
RETURN_ERROR_IF(srcCCtx->stage!=ZSTDcs_init, stage_wrong,
"Can't copy a ctx that's not in init stage.");
-
+ DEBUGLOG(5, "ZSTD_copyCCtx_internal");
ZSTD_memcpy(&dstCCtx->customMem, &srcCCtx->customMem, sizeof(ZSTD_customMem));
{ ZSTD_CCtx_params params = dstCCtx->requestedParams;
/* Copy only compression parameters related to tables. */
params.cParams = srcCCtx->appliedParams.cParams;
+ assert(srcCCtx->appliedParams.useRowMatchFinder != ZSTD_ps_auto);
+ assert(srcCCtx->appliedParams.useBlockSplitter != ZSTD_ps_auto);
+ assert(srcCCtx->appliedParams.ldmParams.enableLdm != ZSTD_ps_auto);
+ params.useRowMatchFinder = srcCCtx->appliedParams.useRowMatchFinder;
+ params.useBlockSplitter = srcCCtx->appliedParams.useBlockSplitter;
+ params.ldmParams = srcCCtx->appliedParams.ldmParams;
params.fParams = fParams;
- ZSTD_resetCCtx_internal(dstCCtx, params, pledgedSrcSize,
+ ZSTD_resetCCtx_internal(dstCCtx, ¶ms, pledgedSrcSize,
+ /* loadedDictSize */ 0,
ZSTDcrp_leaveDirty, zbuff);
assert(dstCCtx->appliedParams.cParams.windowLog == srcCCtx->appliedParams.cParams.windowLog);
assert(dstCCtx->appliedParams.cParams.strategy == srcCCtx->appliedParams.cParams.strategy);
@@ -1938,7 +2160,11 @@ static size_t ZSTD_copyCCtx_internal(ZSTD_CCtx* dstCCtx,
ZSTD_cwksp_mark_tables_dirty(&dstCCtx->workspace);
/* copy tables */
- { size_t const chainSize = (srcCCtx->appliedParams.cParams.strategy == ZSTD_fast) ? 0 : ((size_t)1 << srcCCtx->appliedParams.cParams.chainLog);
+ { size_t const chainSize = ZSTD_allocateChainTable(srcCCtx->appliedParams.cParams.strategy,
+ srcCCtx->appliedParams.useRowMatchFinder,
+ 0 /* forDDSDict */)
+ ? ((size_t)1 << srcCCtx->appliedParams.cParams.chainLog)
+ : 0;
size_t const hSize = (size_t)1 << srcCCtx->appliedParams.cParams.hashLog;
int const h3log = srcCCtx->blockState.matchState.hashLog3;
size_t const h3Size = h3log ? ((size_t)1 << h3log) : 0;
@@ -2005,6 +2231,8 @@ ZSTD_reduceTable_internal (U32* const table, U32 const size, U32 const reducerVa
int const nbRows = (int)size / ZSTD_ROWSIZE;
int cellNb = 0;
int rowNb;
+ /* Protect special index values < ZSTD_WINDOW_START_INDEX. */
+ U32 const reducerThreshold = reducerValue + ZSTD_WINDOW_START_INDEX;
assert((size & (ZSTD_ROWSIZE-1)) == 0); /* multiple of ZSTD_ROWSIZE */
assert(size < (1U<<31)); /* can be casted to int */
@@ -2012,12 +2240,17 @@ ZSTD_reduceTable_internal (U32* const table, U32 const size, U32 const reducerVa
for (rowNb=0 ; rowNb < nbRows ; rowNb++) {
int column;
for (column=0; column<ZSTD_ROWSIZE; column++) {
- if (preserveMark) {
- U32 const adder = (table[cellNb] == ZSTD_DUBT_UNSORTED_MARK) ? reducerValue : 0;
- table[cellNb] += adder;
+ U32 newVal;
+ if (preserveMark && table[cellNb] == ZSTD_DUBT_UNSORTED_MARK) {
+ /* This write is pointless, but is required(?) for the compiler
+ * to auto-vectorize the loop. */
+ newVal = ZSTD_DUBT_UNSORTED_MARK;
+ } else if (table[cellNb] < reducerThreshold) {
+ newVal = 0;
+ } else {
+ newVal = table[cellNb] - reducerValue;
}
- if (table[cellNb] < reducerValue) table[cellNb] = 0;
- else table[cellNb] -= reducerValue;
+ table[cellNb] = newVal;
cellNb++;
} }
}
@@ -2040,7 +2273,7 @@ static void ZSTD_reduceIndex (ZSTD_matchState_t* ms, ZSTD_CCtx_params const* par
ZSTD_reduceTable(ms->hashTable, hSize, reducerValue);
}
- if (params->cParams.strategy != ZSTD_fast) {
+ if (ZSTD_allocateChainTable(params->cParams.strategy, params->useRowMatchFinder, (U32)ms->dedicatedDictSearch)) {
U32 const chainSize = (U32)1 << params->cParams.chainLog;
if (params->cParams.strategy == ZSTD_btlazy2)
ZSTD_reduceTable_btlazy2(ms->chainTable, chainSize, reducerValue);
@@ -2072,14 +2305,14 @@ void ZSTD_seqToCodes(const seqStore_t* seqStorePtr)
assert(nbSeq <= seqStorePtr->maxNbSeq);
for (u=0; u<nbSeq; u++) {
U32 const llv = sequences[u].litLength;
- U32 const mlv = sequences[u].matchLength;
+ U32 const mlv = sequences[u].mlBase;
llCodeTable[u] = (BYTE)ZSTD_LLcode(llv);
- ofCodeTable[u] = (BYTE)ZSTD_highbit32(sequences[u].offset);
+ ofCodeTable[u] = (BYTE)ZSTD_highbit32(sequences[u].offBase);
mlCodeTable[u] = (BYTE)ZSTD_MLcode(mlv);
}
- if (seqStorePtr->longLengthID==1)
+ if (seqStorePtr->longLengthType==ZSTD_llt_literalLength)
llCodeTable[seqStorePtr->longLengthPos] = MaxLL;
- if (seqStorePtr->longLengthID==2)
+ if (seqStorePtr->longLengthType==ZSTD_llt_matchLength)
mlCodeTable[seqStorePtr->longLengthPos] = MaxML;
}
@@ -2093,10 +2326,161 @@ static int ZSTD_useTargetCBlockSize(const ZSTD_CCtx_params* cctxParams)
return (cctxParams->targetCBlockSize != 0);
}
-/* ZSTD_entropyCompressSequences_internal():
- * actually compresses both literals and sequences */
+/* ZSTD_blockSplitterEnabled():
+ * Returns if block splitting param is being used
+ * If used, compression will do best effort to split a block in order to improve compression ratio.
+ * At the time this function is called, the parameter must be finalized.
+ * Returns 1 if true, 0 otherwise. */
+static int ZSTD_blockSplitterEnabled(ZSTD_CCtx_params* cctxParams)
+{
+ DEBUGLOG(5, "ZSTD_blockSplitterEnabled (useBlockSplitter=%d)", cctxParams->useBlockSplitter);
+ assert(cctxParams->useBlockSplitter != ZSTD_ps_auto);
+ return (cctxParams->useBlockSplitter == ZSTD_ps_enable);
+}
+
+/* Type returned by ZSTD_buildSequencesStatistics containing finalized symbol encoding types
+ * and size of the sequences statistics
+ */
+typedef struct {
+ U32 LLtype;
+ U32 Offtype;
+ U32 MLtype;
+ size_t size;
+ size_t lastCountSize; /* Accounts for bug in 1.3.4. More detail in ZSTD_entropyCompressSeqStore_internal() */
+} ZSTD_symbolEncodingTypeStats_t;
+
+/* ZSTD_buildSequencesStatistics():
+ * Returns a ZSTD_symbolEncodingTypeStats_t, or a zstd error code in the `size` field.
+ * Modifies `nextEntropy` to have the appropriate values as a side effect.
+ * nbSeq must be greater than 0.
+ *
+ * entropyWkspSize must be of size at least ENTROPY_WORKSPACE_SIZE - (MaxSeq + 1)*sizeof(U32)
+ */
+static ZSTD_symbolEncodingTypeStats_t
+ZSTD_buildSequencesStatistics(seqStore_t* seqStorePtr, size_t nbSeq,
+ const ZSTD_fseCTables_t* prevEntropy, ZSTD_fseCTables_t* nextEntropy,
+ BYTE* dst, const BYTE* const dstEnd,
+ ZSTD_strategy strategy, unsigned* countWorkspace,
+ void* entropyWorkspace, size_t entropyWkspSize) {
+ BYTE* const ostart = dst;
+ const BYTE* const oend = dstEnd;
+ BYTE* op = ostart;
+ FSE_CTable* CTable_LitLength = nextEntropy->litlengthCTable;
+ FSE_CTable* CTable_OffsetBits = nextEntropy->offcodeCTable;
+ FSE_CTable* CTable_MatchLength = nextEntropy->matchlengthCTable;
+ const BYTE* const ofCodeTable = seqStorePtr->ofCode;
+ const BYTE* const llCodeTable = seqStorePtr->llCode;
+ const BYTE* const mlCodeTable = seqStorePtr->mlCode;
+ ZSTD_symbolEncodingTypeStats_t stats;
+
+ stats.lastCountSize = 0;
+ /* convert length/distances into codes */
+ ZSTD_seqToCodes(seqStorePtr);
+ assert(op <= oend);
+ assert(nbSeq != 0); /* ZSTD_selectEncodingType() divides by nbSeq */
+ /* build CTable for Literal Lengths */
+ { unsigned max = MaxLL;
+ size_t const mostFrequent = HIST_countFast_wksp(countWorkspace, &max, llCodeTable, nbSeq, entropyWorkspace, entropyWkspSize); /* can't fail */
+ DEBUGLOG(5, "Building LL table");
+ nextEntropy->litlength_repeatMode = prevEntropy->litlength_repeatMode;
+ stats.LLtype = ZSTD_selectEncodingType(&nextEntropy->litlength_repeatMode,
+ countWorkspace, max, mostFrequent, nbSeq,
+ LLFSELog, prevEntropy->litlengthCTable,
+ LL_defaultNorm, LL_defaultNormLog,
+ ZSTD_defaultAllowed, strategy);
+ assert(set_basic < set_compressed && set_rle < set_compressed);
+ assert(!(stats.LLtype < set_compressed && nextEntropy->litlength_repeatMode != FSE_repeat_none)); /* We don't copy tables */
+ { size_t const countSize = ZSTD_buildCTable(
+ op, (size_t)(oend - op),
+ CTable_LitLength, LLFSELog, (symbolEncodingType_e)stats.LLtype,
+ countWorkspace, max, llCodeTable, nbSeq,
+ LL_defaultNorm, LL_defaultNormLog, MaxLL,
+ prevEntropy->litlengthCTable,
+ sizeof(prevEntropy->litlengthCTable),
+ entropyWorkspace, entropyWkspSize);
+ if (ZSTD_isError(countSize)) {
+ DEBUGLOG(3, "ZSTD_buildCTable for LitLens failed");
+ stats.size = countSize;
+ return stats;
+ }
+ if (stats.LLtype == set_compressed)
+ stats.lastCountSize = countSize;
+ op += countSize;
+ assert(op <= oend);
+ } }
+ /* build CTable for Offsets */
+ { unsigned max = MaxOff;
+ size_t const mostFrequent = HIST_countFast_wksp(
+ countWorkspace, &max, ofCodeTable, nbSeq, entropyWorkspace, entropyWkspSize); /* can't fail */
+ /* We can only use the basic table if max <= DefaultMaxOff, otherwise the offsets are too large */
+ ZSTD_defaultPolicy_e const defaultPolicy = (max <= DefaultMaxOff) ? ZSTD_defaultAllowed : ZSTD_defaultDisallowed;
+ DEBUGLOG(5, "Building OF table");
+ nextEntropy->offcode_repeatMode = prevEntropy->offcode_repeatMode;
+ stats.Offtype = ZSTD_selectEncodingType(&nextEntropy->offcode_repeatMode,
+ countWorkspace, max, mostFrequent, nbSeq,
+ OffFSELog, prevEntropy->offcodeCTable,
+ OF_defaultNorm, OF_defaultNormLog,
+ defaultPolicy, strategy);
+ assert(!(stats.Offtype < set_compressed && nextEntropy->offcode_repeatMode != FSE_repeat_none)); /* We don't copy tables */
+ { size_t const countSize = ZSTD_buildCTable(
+ op, (size_t)(oend - op),
+ CTable_OffsetBits, OffFSELog, (symbolEncodingType_e)stats.Offtype,
+ countWorkspace, max, ofCodeTable, nbSeq,
+ OF_defaultNorm, OF_defaultNormLog, DefaultMaxOff,
+ prevEntropy->offcodeCTable,
+ sizeof(prevEntropy->offcodeCTable),
+ entropyWorkspace, entropyWkspSize);
+ if (ZSTD_isError(countSize)) {
+ DEBUGLOG(3, "ZSTD_buildCTable for Offsets failed");
+ stats.size = countSize;
+ return stats;
+ }
+ if (stats.Offtype == set_compressed)
+ stats.lastCountSize = countSize;
+ op += countSize;
+ assert(op <= oend);
+ } }
+ /* build CTable for MatchLengths */
+ { unsigned max = MaxML;
+ size_t const mostFrequent = HIST_countFast_wksp(
+ countWorkspace, &max, mlCodeTable, nbSeq, entropyWorkspace, entropyWkspSize); /* can't fail */
+ DEBUGLOG(5, "Building ML table (remaining space : %i)", (int)(oend-op));
+ nextEntropy->matchlength_repeatMode = prevEntropy->matchlength_repeatMode;
+ stats.MLtype = ZSTD_selectEncodingType(&nextEntropy->matchlength_repeatMode,
+ countWorkspace, max, mostFrequent, nbSeq,
+ MLFSELog, prevEntropy->matchlengthCTable,
+ ML_defaultNorm, ML_defaultNormLog,
+ ZSTD_defaultAllowed, strategy);
+ assert(!(stats.MLtype < set_compressed && nextEntropy->matchlength_repeatMode != FSE_repeat_none)); /* We don't copy tables */
+ { size_t const countSize = ZSTD_buildCTable(
+ op, (size_t)(oend - op),
+ CTable_MatchLength, MLFSELog, (symbolEncodingType_e)stats.MLtype,
+ countWorkspace, max, mlCodeTable, nbSeq,
+ ML_defaultNorm, ML_defaultNormLog, MaxML,
+ prevEntropy->matchlengthCTable,
+ sizeof(prevEntropy->matchlengthCTable),
+ entropyWorkspace, entropyWkspSize);
+ if (ZSTD_isError(countSize)) {
+ DEBUGLOG(3, "ZSTD_buildCTable for MatchLengths failed");
+ stats.size = countSize;
+ return stats;
+ }
+ if (stats.MLtype == set_compressed)
+ stats.lastCountSize = countSize;
+ op += countSize;
+ assert(op <= oend);
+ } }
+ stats.size = (size_t)(op-ostart);
+ return stats;
+}
+
+/* ZSTD_entropyCompressSeqStore_internal():
+ * compresses both literals and sequences
+ * Returns compressed size of block, or a zstd error.
+ */
+#define SUSPECT_UNCOMPRESSIBLE_LITERAL_RATIO 20
MEM_STATIC size_t
-ZSTD_entropyCompressSequences_internal(seqStore_t* seqStorePtr,
+ZSTD_entropyCompressSeqStore_internal(seqStore_t* seqStorePtr,
const ZSTD_entropyCTables_t* prevEntropy,
ZSTD_entropyCTables_t* nextEntropy,
const ZSTD_CCtx_params* cctxParams,
@@ -2110,36 +2494,38 @@ ZSTD_entropyCompressSequences_internal(seqStore_t* seqStorePtr,
FSE_CTable* CTable_LitLength = nextEntropy->fse.litlengthCTable;
FSE_CTable* CTable_OffsetBits = nextEntropy->fse.offcodeCTable;
FSE_CTable* CTable_MatchLength = nextEntropy->fse.matchlengthCTable;
- U32 LLtype, Offtype, MLtype; /* compressed, raw or rle */
const seqDef* const sequences = seqStorePtr->sequencesStart;
+ const size_t nbSeq = seqStorePtr->sequences - seqStorePtr->sequencesStart;
const BYTE* const ofCodeTable = seqStorePtr->ofCode;
const BYTE* const llCodeTable = seqStorePtr->llCode;
const BYTE* const mlCodeTable = seqStorePtr->mlCode;
BYTE* const ostart = (BYTE*)dst;
BYTE* const oend = ostart + dstCapacity;
BYTE* op = ostart;
- size_t const nbSeq = (size_t)(seqStorePtr->sequences - seqStorePtr->sequencesStart);
- BYTE* seqHead;
- BYTE* lastNCount = NULL;
+ size_t lastCountSize;
entropyWorkspace = count + (MaxSeq + 1);
entropyWkspSize -= (MaxSeq + 1) * sizeof(*count);
- DEBUGLOG(4, "ZSTD_entropyCompressSequences_internal (nbSeq=%zu)", nbSeq);
+ DEBUGLOG(4, "ZSTD_entropyCompressSeqStore_internal (nbSeq=%zu)", nbSeq);
ZSTD_STATIC_ASSERT(HUF_WORKSPACE_SIZE >= (1<<MAX(MLFSELog,LLFSELog)));
assert(entropyWkspSize >= HUF_WORKSPACE_SIZE);
/* Compress literals */
{ const BYTE* const literals = seqStorePtr->litStart;
+ size_t const numSequences = seqStorePtr->sequences - seqStorePtr->sequencesStart;
+ size_t const numLiterals = seqStorePtr->lit - seqStorePtr->litStart;
+ /* Base suspicion of uncompressibility on ratio of literals to sequences */
+ unsigned const suspectUncompressible = (numSequences == 0) || (numLiterals / numSequences >= SUSPECT_UNCOMPRESSIBLE_LITERAL_RATIO);
size_t const litSize = (size_t)(seqStorePtr->lit - literals);
size_t const cSize = ZSTD_compressLiterals(
&prevEntropy->huf, &nextEntropy->huf,
cctxParams->cParams.strategy,
- ZSTD_disableLiteralsCompression(cctxParams),
+ ZSTD_literalsCompressionIsDisabled(cctxParams),
op, dstCapacity,
literals, litSize,
entropyWorkspace, entropyWkspSize,
- bmi2);
+ bmi2, suspectUncompressible);
FORWARD_IF_ERROR(cSize, "ZSTD_compressLiterals failed");
assert(cSize <= dstCapacity);
op += cSize;
@@ -2165,95 +2551,20 @@ ZSTD_entropyCompressSequences_internal(seqStore_t* seqStorePtr,
ZSTD_memcpy(&nextEntropy->fse, &prevEntropy->fse, sizeof(prevEntropy->fse));
return (size_t)(op - ostart);
}
-
- /* seqHead : flags for FSE encoding type */
- seqHead = op++;
- assert(op <= oend);
-
- /* convert length/distances into codes */
- ZSTD_seqToCodes(seqStorePtr);
- /* build CTable for Literal Lengths */
- { unsigned max = MaxLL;
- size_t const mostFrequent = HIST_countFast_wksp(count, &max, llCodeTable, nbSeq, entropyWorkspace, entropyWkspSize); /* can't fail */
- DEBUGLOG(5, "Building LL table");
- nextEntropy->fse.litlength_repeatMode = prevEntropy->fse.litlength_repeatMode;
- LLtype = ZSTD_selectEncodingType(&nextEntropy->fse.litlength_repeatMode,
- count, max, mostFrequent, nbSeq,
- LLFSELog, prevEntropy->fse.litlengthCTable,
- LL_defaultNorm, LL_defaultNormLog,
- ZSTD_defaultAllowed, strategy);
- assert(set_basic < set_compressed && set_rle < set_compressed);
- assert(!(LLtype < set_compressed && nextEntropy->fse.litlength_repeatMode != FSE_repeat_none)); /* We don't copy tables */
- { size_t const countSize = ZSTD_buildCTable(
- op, (size_t)(oend - op),
- CTable_LitLength, LLFSELog, (symbolEncodingType_e)LLtype,
- count, max, llCodeTable, nbSeq,
- LL_defaultNorm, LL_defaultNormLog, MaxLL,
- prevEntropy->fse.litlengthCTable,
- sizeof(prevEntropy->fse.litlengthCTable),
- entropyWorkspace, entropyWkspSize);
- FORWARD_IF_ERROR(countSize, "ZSTD_buildCTable for LitLens failed");
- if (LLtype == set_compressed)
- lastNCount = op;
- op += countSize;
- assert(op <= oend);
- } }
- /* build CTable for Offsets */
- { unsigned max = MaxOff;
- size_t const mostFrequent = HIST_countFast_wksp(
- count, &max, ofCodeTable, nbSeq, entropyWorkspace, entropyWkspSize); /* can't fail */
- /* We can only use the basic table if max <= DefaultMaxOff, otherwise the offsets are too large */
- ZSTD_defaultPolicy_e const defaultPolicy = (max <= DefaultMaxOff) ? ZSTD_defaultAllowed : ZSTD_defaultDisallowed;
- DEBUGLOG(5, "Building OF table");
- nextEntropy->fse.offcode_repeatMode = prevEntropy->fse.offcode_repeatMode;
- Offtype = ZSTD_selectEncodingType(&nextEntropy->fse.offcode_repeatMode,
- count, max, mostFrequent, nbSeq,
- OffFSELog, prevEntropy->fse.offcodeCTable,
- OF_defaultNorm, OF_defaultNormLog,
- defaultPolicy, strategy);
- assert(!(Offtype < set_compressed && nextEntropy->fse.offcode_repeatMode != FSE_repeat_none)); /* We don't copy tables */
- { size_t const countSize = ZSTD_buildCTable(
- op, (size_t)(oend - op),
- CTable_OffsetBits, OffFSELog, (symbolEncodingType_e)Offtype,
- count, max, ofCodeTable, nbSeq,
- OF_defaultNorm, OF_defaultNormLog, DefaultMaxOff,
- prevEntropy->fse.offcodeCTable,
- sizeof(prevEntropy->fse.offcodeCTable),
- entropyWorkspace, entropyWkspSize);
- FORWARD_IF_ERROR(countSize, "ZSTD_buildCTable for Offsets failed");
- if (Offtype == set_compressed)
- lastNCount = op;
- op += countSize;
- assert(op <= oend);
- } }
- /* build CTable for MatchLengths */
- { unsigned max = MaxML;
- size_t const mostFrequent = HIST_countFast_wksp(
- count, &max, mlCodeTable, nbSeq, entropyWorkspace, entropyWkspSize); /* can't fail */
- DEBUGLOG(5, "Building ML table (remaining space : %i)", (int)(oend-op));
- nextEntropy->fse.matchlength_repeatMode = prevEntropy->fse.matchlength_repeatMode;
- MLtype = ZSTD_selectEncodingType(&nextEntropy->fse.matchlength_repeatMode,
- count, max, mostFrequent, nbSeq,
- MLFSELog, prevEntropy->fse.matchlengthCTable,
- ML_defaultNorm, ML_defaultNormLog,
- ZSTD_defaultAllowed, strategy);
- assert(!(MLtype < set_compressed && nextEntropy->fse.matchlength_repeatMode != FSE_repeat_none)); /* We don't copy tables */
- { size_t const countSize = ZSTD_buildCTable(
- op, (size_t)(oend - op),
- CTable_MatchLength, MLFSELog, (symbolEncodingType_e)MLtype,
- count, max, mlCodeTable, nbSeq,
- ML_defaultNorm, ML_defaultNormLog, MaxML,
- prevEntropy->fse.matchlengthCTable,
- sizeof(prevEntropy->fse.matchlengthCTable),
- entropyWorkspace, entropyWkspSize);
- FORWARD_IF_ERROR(countSize, "ZSTD_buildCTable for MatchLengths failed");
- if (MLtype == set_compressed)
- lastNCount = op;
- op += countSize;
- assert(op <= oend);
- } }
-
- *seqHead = (BYTE)((LLtype<<6) + (Offtype<<4) + (MLtype<<2));
+ {
+ ZSTD_symbolEncodingTypeStats_t stats;
+ BYTE* seqHead = op++;
+ /* build stats for sequences */
+ stats = ZSTD_buildSequencesStatistics(seqStorePtr, nbSeq,
+ &prevEntropy->fse, &nextEntropy->fse,
+ op, oend,
+ strategy, count,
+ entropyWorkspace, entropyWkspSize);
+ FORWARD_IF_ERROR(stats.size, "ZSTD_buildSequencesStatistics failed!");
+ *seqHead = (BYTE)((stats.LLtype<<6) + (stats.Offtype<<4) + (stats.MLtype<<2));
+ lastCountSize = stats.lastCountSize;
+ op += stats.size;
+ }
{ size_t const bitstreamSize = ZSTD_encodeSequences(
op, (size_t)(oend - op),
@@ -2273,9 +2584,9 @@ ZSTD_entropyCompressSequences_internal(seqStore_t* seqStorePtr,
* In this exceedingly rare case, we will simply emit an uncompressed
* block, since it isn't worth optimizing.
*/
- if (lastNCount && (op - lastNCount) < 4) {
- /* NCountSize >= 2 && bitstreamSize > 0 ==> lastCountSize == 3 */
- assert(op - lastNCount == 3);
+ if (lastCountSize && (lastCountSize + bitstreamSize) < 4) {
+ /* lastCountSize >= 2 && bitstreamSize > 0 ==> lastCountSize == 3 */
+ assert(lastCountSize + bitstreamSize == 3);
DEBUGLOG(5, "Avoiding bug in zstd decoder in versions <= 1.3.4 by "
"emitting an uncompressed block.");
return 0;
@@ -2287,7 +2598,7 @@ ZSTD_entropyCompressSequences_internal(seqStore_t* seqStorePtr,
}
MEM_STATIC size_t
-ZSTD_entropyCompressSequences(seqStore_t* seqStorePtr,
+ZSTD_entropyCompressSeqStore(seqStore_t* seqStorePtr,
const ZSTD_entropyCTables_t* prevEntropy,
ZSTD_entropyCTables_t* nextEntropy,
const ZSTD_CCtx_params* cctxParams,
@@ -2296,7 +2607,7 @@ ZSTD_entropyCompressSequences(seqStore_t* seqStorePtr,
void* entropyWorkspace, size_t entropyWkspSize,
int bmi2)
{
- size_t const cSize = ZSTD_entropyCompressSequences_internal(
+ size_t const cSize = ZSTD_entropyCompressSeqStore_internal(
seqStorePtr, prevEntropy, nextEntropy, cctxParams,
dst, dstCapacity,
entropyWorkspace, entropyWkspSize, bmi2);
@@ -2306,20 +2617,20 @@ ZSTD_entropyCompressSequences(seqStore_t* seqStorePtr,
*/
if ((cSize == ERROR(dstSize_tooSmall)) & (srcSize <= dstCapacity))
return 0; /* block not compressed */
- FORWARD_IF_ERROR(cSize, "ZSTD_entropyCompressSequences_internal failed");
+ FORWARD_IF_ERROR(cSize, "ZSTD_entropyCompressSeqStore_internal failed");
/* Check compressibility */
{ size_t const maxCSize = srcSize - ZSTD_minGain(srcSize, cctxParams->cParams.strategy);
if (cSize >= maxCSize) return 0; /* block not compressed */
}
- DEBUGLOG(4, "ZSTD_entropyCompressSequences() cSize: %zu\n", cSize);
+ DEBUGLOG(4, "ZSTD_entropyCompressSeqStore() cSize: %zu", cSize);
return cSize;
}
/* ZSTD_selectBlockCompressor() :
* Not static, but internal use only (used by long distance matcher)
* assumption : strat is a valid strategy */
-ZSTD_blockCompressor ZSTD_selectBlockCompressor(ZSTD_strategy strat, ZSTD_dictMode_e dictMode)
+ZSTD_blockCompressor ZSTD_selectBlockCompressor(ZSTD_strategy strat, ZSTD_paramSwitch_e useRowMatchFinder, ZSTD_dictMode_e dictMode)
{
static const ZSTD_blockCompressor blockCompressor[4][ZSTD_STRATEGY_MAX+1] = {
{ ZSTD_compressBlock_fast /* default for 0 */,
@@ -2367,7 +2678,28 @@ ZSTD_blockCompressor ZSTD_selectBlockCompressor(ZSTD_strategy strat, ZSTD_dictMo
ZSTD_STATIC_ASSERT((unsigned)ZSTD_fast == 1);
assert(ZSTD_cParam_withinBounds(ZSTD_c_strategy, strat));
- selectedCompressor = blockCompressor[(int)dictMode][(int)strat];
+ DEBUGLOG(4, "Selected block compressor: dictMode=%d strat=%d rowMatchfinder=%d", (int)dictMode, (int)strat, (int)useRowMatchFinder);
+ if (ZSTD_rowMatchFinderUsed(strat, useRowMatchFinder)) {
+ static const ZSTD_blockCompressor rowBasedBlockCompressors[4][3] = {
+ { ZSTD_compressBlock_greedy_row,
+ ZSTD_compressBlock_lazy_row,
+ ZSTD_compressBlock_lazy2_row },
+ { ZSTD_compressBlock_greedy_extDict_row,
+ ZSTD_compressBlock_lazy_extDict_row,
+ ZSTD_compressBlock_lazy2_extDict_row },
+ { ZSTD_compressBlock_greedy_dictMatchState_row,
+ ZSTD_compressBlock_lazy_dictMatchState_row,
+ ZSTD_compressBlock_lazy2_dictMatchState_row },
+ { ZSTD_compressBlock_greedy_dedicatedDictSearch_row,
+ ZSTD_compressBlock_lazy_dedicatedDictSearch_row,
+ ZSTD_compressBlock_lazy2_dedicatedDictSearch_row }
+ };
+ DEBUGLOG(4, "Selecting a row-based matchfinder");
+ assert(useRowMatchFinder != ZSTD_ps_auto);
+ selectedCompressor = rowBasedBlockCompressors[(int)dictMode][(int)strat - (int)ZSTD_greedy];
+ } else {
+ selectedCompressor = blockCompressor[(int)dictMode][(int)strat];
+ }
assert(selectedCompressor != NULL);
return selectedCompressor;
}
@@ -2383,7 +2715,7 @@ void ZSTD_resetSeqStore(seqStore_t* ssPtr)
{
ssPtr->lit = ssPtr->litStart;
ssPtr->sequences = ssPtr->sequencesStart;
- ssPtr->longLengthID = 0;
+ ssPtr->longLengthType = ZSTD_llt_none;
}
typedef enum { ZSTDbss_compress, ZSTDbss_noCompress } ZSTD_buildSeqStore_e;
@@ -2430,15 +2762,16 @@ static size_t ZSTD_buildSeqStore(ZSTD_CCtx* zc, const void* src, size_t srcSize)
zc->blockState.nextCBlock->rep[i] = zc->blockState.prevCBlock->rep[i];
}
if (zc->externSeqStore.pos < zc->externSeqStore.size) {
- assert(!zc->appliedParams.ldmParams.enableLdm);
+ assert(zc->appliedParams.ldmParams.enableLdm == ZSTD_ps_disable);
/* Updates ldmSeqStore.pos */
lastLLSize =
ZSTD_ldm_blockCompress(&zc->externSeqStore,
ms, &zc->seqStore,
zc->blockState.nextCBlock->rep,
+ zc->appliedParams.useRowMatchFinder,
src, srcSize);
assert(zc->externSeqStore.pos <= zc->externSeqStore.size);
- } else if (zc->appliedParams.ldmParams.enableLdm) {
+ } else if (zc->appliedParams.ldmParams.enableLdm == ZSTD_ps_enable) {
rawSeqStore_t ldmSeqStore = kNullRawSeqStore;
ldmSeqStore.seq = zc->ldmSequences;
@@ -2452,10 +2785,13 @@ static size_t ZSTD_buildSeqStore(ZSTD_CCtx* zc, const void* src, size_t srcSize)
ZSTD_ldm_blockCompress(&ldmSeqStore,
ms, &zc->seqStore,
zc->blockState.nextCBlock->rep,
+ zc->appliedParams.useRowMatchFinder,
src, srcSize);
assert(ldmSeqStore.pos == ldmSeqStore.size);
} else { /* not long range mode */
- ZSTD_blockCompressor const blockCompressor = ZSTD_selectBlockCompressor(zc->appliedParams.cParams.strategy, dictMode);
+ ZSTD_blockCompressor const blockCompressor = ZSTD_selectBlockCompressor(zc->appliedParams.cParams.strategy,
+ zc->appliedParams.useRowMatchFinder,
+ dictMode);
ms->ldmSeqStore = NULL;
lastLLSize = blockCompressor(ms, &zc->seqStore, zc->blockState.nextCBlock->rep, src, srcSize);
}
@@ -2483,22 +2819,22 @@ static void ZSTD_copyBlockSequences(ZSTD_CCtx* zc)
assert(zc->seqCollector.maxSequences >= seqStoreSeqSize + 1);
ZSTD_memcpy(updatedRepcodes.rep, zc->blockState.prevCBlock->rep, sizeof(repcodes_t));
for (i = 0; i < seqStoreSeqSize; ++i) {
- U32 rawOffset = seqStoreSeqs[i].offset - ZSTD_REP_NUM;
+ U32 rawOffset = seqStoreSeqs[i].offBase - ZSTD_REP_NUM;
outSeqs[i].litLength = seqStoreSeqs[i].litLength;
- outSeqs[i].matchLength = seqStoreSeqs[i].matchLength + MINMATCH;
+ outSeqs[i].matchLength = seqStoreSeqs[i].mlBase + MINMATCH;
outSeqs[i].rep = 0;
if (i == seqStore->longLengthPos) {
- if (seqStore->longLengthID == 1) {
+ if (seqStore->longLengthType == ZSTD_llt_literalLength) {
outSeqs[i].litLength += 0x10000;
- } else if (seqStore->longLengthID == 2) {
+ } else if (seqStore->longLengthType == ZSTD_llt_matchLength) {
outSeqs[i].matchLength += 0x10000;
}
}
- if (seqStoreSeqs[i].offset <= ZSTD_REP_NUM) {
+ if (seqStoreSeqs[i].offBase <= ZSTD_REP_NUM) {
/* Derive the correct offset corresponding to a repcode */
- outSeqs[i].rep = seqStoreSeqs[i].offset;
+ outSeqs[i].rep = seqStoreSeqs[i].offBase;
if (outSeqs[i].litLength != 0) {
rawOffset = updatedRepcodes.rep[outSeqs[i].rep - 1];
} else {
@@ -2512,9 +2848,9 @@ static void ZSTD_copyBlockSequences(ZSTD_CCtx* zc)
outSeqs[i].offset = rawOffset;
/* seqStoreSeqs[i].offset == offCode+1, and ZSTD_updateRep() expects offCode
so we provide seqStoreSeqs[i].offset - 1 */
- updatedRepcodes = ZSTD_updateRep(updatedRepcodes.rep,
- seqStoreSeqs[i].offset - 1,
- seqStoreSeqs[i].litLength == 0);
+ ZSTD_updateRep(updatedRepcodes.rep,
+ seqStoreSeqs[i].offBase - 1,
+ seqStoreSeqs[i].litLength == 0);
literalsRead += outSeqs[i].litLength;
}
/* Insert last literals (if any exist) in the block as a sequence with ml == off == 0.
@@ -2602,16 +2938,740 @@ static int ZSTD_maybeRLE(seqStore_t const* seqStore)
return nbSeqs < 4 && nbLits < 10;
}
-static void ZSTD_confirmRepcodesAndEntropyTables(ZSTD_CCtx* zc)
+static void ZSTD_blockState_confirmRepcodesAndEntropyTables(ZSTD_blockState_t* const bs)
{
- ZSTD_compressedBlockState_t* const tmp = zc->blockState.prevCBlock;
- zc->blockState.prevCBlock = zc->blockState.nextCBlock;
- zc->blockState.nextCBlock = tmp;
+ ZSTD_compressedBlockState_t* const tmp = bs->prevCBlock;
+ bs->prevCBlock = bs->nextCBlock;
+ bs->nextCBlock = tmp;
}
-static size_t ZSTD_compressBlock_internal(ZSTD_CCtx* zc,
- void* dst, size_t dstCapacity,
- const void* src, size_t srcSize, U32 frame)
+/* Writes the block header */
+static void writeBlockHeader(void* op, size_t cSize, size_t blockSize, U32 lastBlock) {
+ U32 const cBlockHeader = cSize == 1 ?
+ lastBlock + (((U32)bt_rle)<<1) + (U32)(blockSize << 3) :
+ lastBlock + (((U32)bt_compressed)<<1) + (U32)(cSize << 3);
+ MEM_writeLE24(op, cBlockHeader);
+ DEBUGLOG(3, "writeBlockHeader: cSize: %zu blockSize: %zu lastBlock: %u", cSize, blockSize, lastBlock);
+}
+
+/* ZSTD_buildBlockEntropyStats_literals() :
+ * Builds entropy for the literals.
+ * Stores literals block type (raw, rle, compressed, repeat) and
+ * huffman description table to hufMetadata.
+ * Requires ENTROPY_WORKSPACE_SIZE workspace
+ * @return : size of huffman description table or error code */
+static size_t ZSTD_buildBlockEntropyStats_literals(void* const src, size_t srcSize,
+ const ZSTD_hufCTables_t* prevHuf,
+ ZSTD_hufCTables_t* nextHuf,
+ ZSTD_hufCTablesMetadata_t* hufMetadata,
+ const int literalsCompressionIsDisabled,
+ void* workspace, size_t wkspSize)
+{
+ BYTE* const wkspStart = (BYTE*)workspace;
+ BYTE* const wkspEnd = wkspStart + wkspSize;
+ BYTE* const countWkspStart = wkspStart;
+ unsigned* const countWksp = (unsigned*)workspace;
+ const size_t countWkspSize = (HUF_SYMBOLVALUE_MAX + 1) * sizeof(unsigned);
+ BYTE* const nodeWksp = countWkspStart + countWkspSize;
+ const size_t nodeWkspSize = wkspEnd-nodeWksp;
+ unsigned maxSymbolValue = HUF_SYMBOLVALUE_MAX;
+ unsigned huffLog = HUF_TABLELOG_DEFAULT;
+ HUF_repeat repeat = prevHuf->repeatMode;
+ DEBUGLOG(5, "ZSTD_buildBlockEntropyStats_literals (srcSize=%zu)", srcSize);
+
+ /* Prepare nextEntropy assuming reusing the existing table */
+ ZSTD_memcpy(nextHuf, prevHuf, sizeof(*prevHuf));
+
+ if (literalsCompressionIsDisabled) {
+ DEBUGLOG(5, "set_basic - disabled");
+ hufMetadata->hType = set_basic;
+ return 0;
+ }
+
+ /* small ? don't even attempt compression (speed opt) */
+#ifndef COMPRESS_LITERALS_SIZE_MIN
+#define COMPRESS_LITERALS_SIZE_MIN 63
+#endif
+ { size_t const minLitSize = (prevHuf->repeatMode == HUF_repeat_valid) ? 6 : COMPRESS_LITERALS_SIZE_MIN;
+ if (srcSize <= minLitSize) {
+ DEBUGLOG(5, "set_basic - too small");
+ hufMetadata->hType = set_basic;
+ return 0;
+ }
+ }
+
+ /* Scan input and build symbol stats */
+ { size_t const largest = HIST_count_wksp (countWksp, &maxSymbolValue, (const BYTE*)src, srcSize, workspace, wkspSize);
+ FORWARD_IF_ERROR(largest, "HIST_count_wksp failed");
+ if (largest == srcSize) {
+ DEBUGLOG(5, "set_rle");
+ hufMetadata->hType = set_rle;
+ return 0;
+ }
+ if (largest <= (srcSize >> 7)+4) {
+ DEBUGLOG(5, "set_basic - no gain");
+ hufMetadata->hType = set_basic;
+ return 0;
+ }
+ }
+
+ /* Validate the previous Huffman table */
+ if (repeat == HUF_repeat_check && !HUF_validateCTable((HUF_CElt const*)prevHuf->CTable, countWksp, maxSymbolValue)) {
+ repeat = HUF_repeat_none;
+ }
+
+ /* Build Huffman Tree */
+ ZSTD_memset(nextHuf->CTable, 0, sizeof(nextHuf->CTable));
+ huffLog = HUF_optimalTableLog(huffLog, srcSize, maxSymbolValue);
+ { size_t const maxBits = HUF_buildCTable_wksp((HUF_CElt*)nextHuf->CTable, countWksp,
+ maxSymbolValue, huffLog,
+ nodeWksp, nodeWkspSize);
+ FORWARD_IF_ERROR(maxBits, "HUF_buildCTable_wksp");
+ huffLog = (U32)maxBits;
+ { /* Build and write the CTable */
+ size_t const newCSize = HUF_estimateCompressedSize(
+ (HUF_CElt*)nextHuf->CTable, countWksp, maxSymbolValue);
+ size_t const hSize = HUF_writeCTable_wksp(
+ hufMetadata->hufDesBuffer, sizeof(hufMetadata->hufDesBuffer),
+ (HUF_CElt*)nextHuf->CTable, maxSymbolValue, huffLog,
+ nodeWksp, nodeWkspSize);
+ /* Check against repeating the previous CTable */
+ if (repeat != HUF_repeat_none) {
+ size_t const oldCSize = HUF_estimateCompressedSize(
+ (HUF_CElt const*)prevHuf->CTable, countWksp, maxSymbolValue);
+ if (oldCSize < srcSize && (oldCSize <= hSize + newCSize || hSize + 12 >= srcSize)) {
+ DEBUGLOG(5, "set_repeat - smaller");
+ ZSTD_memcpy(nextHuf, prevHuf, sizeof(*prevHuf));
+ hufMetadata->hType = set_repeat;
+ return 0;
+ }
+ }
+ if (newCSize + hSize >= srcSize) {
+ DEBUGLOG(5, "set_basic - no gains");
+ ZSTD_memcpy(nextHuf, prevHuf, sizeof(*prevHuf));
+ hufMetadata->hType = set_basic;
+ return 0;
+ }
+ DEBUGLOG(5, "set_compressed (hSize=%u)", (U32)hSize);
+ hufMetadata->hType = set_compressed;
+ nextHuf->repeatMode = HUF_repeat_check;
+ return hSize;
+ }
+ }
+}
+
+
+/* ZSTD_buildDummySequencesStatistics():
+ * Returns a ZSTD_symbolEncodingTypeStats_t with all encoding types as set_basic,
+ * and updates nextEntropy to the appropriate repeatMode.
+ */
+static ZSTD_symbolEncodingTypeStats_t
+ZSTD_buildDummySequencesStatistics(ZSTD_fseCTables_t* nextEntropy) {
+ ZSTD_symbolEncodingTypeStats_t stats = {set_basic, set_basic, set_basic, 0, 0};
+ nextEntropy->litlength_repeatMode = FSE_repeat_none;
+ nextEntropy->offcode_repeatMode = FSE_repeat_none;
+ nextEntropy->matchlength_repeatMode = FSE_repeat_none;
+ return stats;
+}
+
+/* ZSTD_buildBlockEntropyStats_sequences() :
+ * Builds entropy for the sequences.
+ * Stores symbol compression modes and fse table to fseMetadata.
+ * Requires ENTROPY_WORKSPACE_SIZE wksp.
+ * @return : size of fse tables or error code */
+static size_t ZSTD_buildBlockEntropyStats_sequences(seqStore_t* seqStorePtr,
+ const ZSTD_fseCTables_t* prevEntropy,
+ ZSTD_fseCTables_t* nextEntropy,
+ const ZSTD_CCtx_params* cctxParams,
+ ZSTD_fseCTablesMetadata_t* fseMetadata,
+ void* workspace, size_t wkspSize)
+{
+ ZSTD_strategy const strategy = cctxParams->cParams.strategy;
+ size_t const nbSeq = seqStorePtr->sequences - seqStorePtr->sequencesStart;
+ BYTE* const ostart = fseMetadata->fseTablesBuffer;
+ BYTE* const oend = ostart + sizeof(fseMetadata->fseTablesBuffer);
+ BYTE* op = ostart;
+ unsigned* countWorkspace = (unsigned*)workspace;
+ unsigned* entropyWorkspace = countWorkspace + (MaxSeq + 1);
+ size_t entropyWorkspaceSize = wkspSize - (MaxSeq + 1) * sizeof(*countWorkspace);
+ ZSTD_symbolEncodingTypeStats_t stats;
+
+ DEBUGLOG(5, "ZSTD_buildBlockEntropyStats_sequences (nbSeq=%zu)", nbSeq);
+ stats = nbSeq != 0 ? ZSTD_buildSequencesStatistics(seqStorePtr, nbSeq,
+ prevEntropy, nextEntropy, op, oend,
+ strategy, countWorkspace,
+ entropyWorkspace, entropyWorkspaceSize)
+ : ZSTD_buildDummySequencesStatistics(nextEntropy);
+ FORWARD_IF_ERROR(stats.size, "ZSTD_buildSequencesStatistics failed!");
+ fseMetadata->llType = (symbolEncodingType_e) stats.LLtype;
+ fseMetadata->ofType = (symbolEncodingType_e) stats.Offtype;
+ fseMetadata->mlType = (symbolEncodingType_e) stats.MLtype;
+ fseMetadata->lastCountSize = stats.lastCountSize;
+ return stats.size;
+}
+
+
+/* ZSTD_buildBlockEntropyStats() :
+ * Builds entropy for the block.
+ * Requires workspace size ENTROPY_WORKSPACE_SIZE
+ *
+ * @return : 0 on success or error code
+ */
+size_t ZSTD_buildBlockEntropyStats(seqStore_t* seqStorePtr,
+ const ZSTD_entropyCTables_t* prevEntropy,
+ ZSTD_entropyCTables_t* nextEntropy,
+ const ZSTD_CCtx_params* cctxParams,
+ ZSTD_entropyCTablesMetadata_t* entropyMetadata,
+ void* workspace, size_t wkspSize)
+{
+ size_t const litSize = seqStorePtr->lit - seqStorePtr->litStart;
+ entropyMetadata->hufMetadata.hufDesSize =
+ ZSTD_buildBlockEntropyStats_literals(seqStorePtr->litStart, litSize,
+ &prevEntropy->huf, &nextEntropy->huf,
+ &entropyMetadata->hufMetadata,
+ ZSTD_literalsCompressionIsDisabled(cctxParams),
+ workspace, wkspSize);
+ FORWARD_IF_ERROR(entropyMetadata->hufMetadata.hufDesSize, "ZSTD_buildBlockEntropyStats_literals failed");
+ entropyMetadata->fseMetadata.fseTablesSize =
+ ZSTD_buildBlockEntropyStats_sequences(seqStorePtr,
+ &prevEntropy->fse, &nextEntropy->fse,
+ cctxParams,
+ &entropyMetadata->fseMetadata,
+ workspace, wkspSize);
+ FORWARD_IF_ERROR(entropyMetadata->fseMetadata.fseTablesSize, "ZSTD_buildBlockEntropyStats_sequences failed");
+ return 0;
+}
+
+/* Returns the size estimate for the literals section (header + content) of a block */
+static size_t ZSTD_estimateBlockSize_literal(const BYTE* literals, size_t litSize,
+ const ZSTD_hufCTables_t* huf,
+ const ZSTD_hufCTablesMetadata_t* hufMetadata,
+ void* workspace, size_t wkspSize,
+ int writeEntropy)
+{
+ unsigned* const countWksp = (unsigned*)workspace;
+ unsigned maxSymbolValue = HUF_SYMBOLVALUE_MAX;
+ size_t literalSectionHeaderSize = 3 + (litSize >= 1 KB) + (litSize >= 16 KB);
+ U32 singleStream = litSize < 256;
+
+ if (hufMetadata->hType == set_basic) return litSize;
+ else if (hufMetadata->hType == set_rle) return 1;
+ else if (hufMetadata->hType == set_compressed || hufMetadata->hType == set_repeat) {
+ size_t const largest = HIST_count_wksp (countWksp, &maxSymbolValue, (const BYTE*)literals, litSize, workspace, wkspSize);
+ if (ZSTD_isError(largest)) return litSize;
+ { size_t cLitSizeEstimate = HUF_estimateCompressedSize((const HUF_CElt*)huf->CTable, countWksp, maxSymbolValue);
+ if (writeEntropy) cLitSizeEstimate += hufMetadata->hufDesSize;
+ if (!singleStream) cLitSizeEstimate += 6; /* multi-stream huffman uses 6-byte jump table */
+ return cLitSizeEstimate + literalSectionHeaderSize;
+ } }
+ assert(0); /* impossible */
+ return 0;
+}
+
+/* Returns the size estimate for the FSE-compressed symbols (of, ml, ll) of a block */
+static size_t ZSTD_estimateBlockSize_symbolType(symbolEncodingType_e type,
+ const BYTE* codeTable, size_t nbSeq, unsigned maxCode,
+ const FSE_CTable* fseCTable,
+ const U8* additionalBits,
+ short const* defaultNorm, U32 defaultNormLog, U32 defaultMax,
+ void* workspace, size_t wkspSize)
+{
+ unsigned* const countWksp = (unsigned*)workspace;
+ const BYTE* ctp = codeTable;
+ const BYTE* const ctStart = ctp;
+ const BYTE* const ctEnd = ctStart + nbSeq;
+ size_t cSymbolTypeSizeEstimateInBits = 0;
+ unsigned max = maxCode;
+
+ HIST_countFast_wksp(countWksp, &max, codeTable, nbSeq, workspace, wkspSize); /* can't fail */
+ if (type == set_basic) {
+ /* We selected this encoding type, so it must be valid. */
+ assert(max <= defaultMax);
+ (void)defaultMax;
+ cSymbolTypeSizeEstimateInBits = ZSTD_crossEntropyCost(defaultNorm, defaultNormLog, countWksp, max);
+ } else if (type == set_rle) {
+ cSymbolTypeSizeEstimateInBits = 0;
+ } else if (type == set_compressed || type == set_repeat) {
+ cSymbolTypeSizeEstimateInBits = ZSTD_fseBitCost(fseCTable, countWksp, max);
+ }
+ if (ZSTD_isError(cSymbolTypeSizeEstimateInBits)) {
+ return nbSeq * 10;
+ }
+ while (ctp < ctEnd) {
+ if (additionalBits) cSymbolTypeSizeEstimateInBits += additionalBits[*ctp];
+ else cSymbolTypeSizeEstimateInBits += *ctp; /* for offset, offset code is also the number of additional bits */
+ ctp++;
+ }
+ return cSymbolTypeSizeEstimateInBits >> 3;
+}
+
+/* Returns the size estimate for the sequences section (header + content) of a block */
+static size_t ZSTD_estimateBlockSize_sequences(const BYTE* ofCodeTable,
+ const BYTE* llCodeTable,
+ const BYTE* mlCodeTable,
+ size_t nbSeq,
+ const ZSTD_fseCTables_t* fseTables,
+ const ZSTD_fseCTablesMetadata_t* fseMetadata,
+ void* workspace, size_t wkspSize,
+ int writeEntropy)
+{
+ size_t sequencesSectionHeaderSize = 1 /* seqHead */ + 1 /* min seqSize size */ + (nbSeq >= 128) + (nbSeq >= LONGNBSEQ);
+ size_t cSeqSizeEstimate = 0;
+ cSeqSizeEstimate += ZSTD_estimateBlockSize_symbolType(fseMetadata->ofType, ofCodeTable, nbSeq, MaxOff,
+ fseTables->offcodeCTable, NULL,
+ OF_defaultNorm, OF_defaultNormLog, DefaultMaxOff,
+ workspace, wkspSize);
+ cSeqSizeEstimate += ZSTD_estimateBlockSize_symbolType(fseMetadata->llType, llCodeTable, nbSeq, MaxLL,
+ fseTables->litlengthCTable, LL_bits,
+ LL_defaultNorm, LL_defaultNormLog, MaxLL,
+ workspace, wkspSize);
+ cSeqSizeEstimate += ZSTD_estimateBlockSize_symbolType(fseMetadata->mlType, mlCodeTable, nbSeq, MaxML,
+ fseTables->matchlengthCTable, ML_bits,
+ ML_defaultNorm, ML_defaultNormLog, MaxML,
+ workspace, wkspSize);
+ if (writeEntropy) cSeqSizeEstimate += fseMetadata->fseTablesSize;
+ return cSeqSizeEstimate + sequencesSectionHeaderSize;
+}
+
+/* Returns the size estimate for a given stream of literals, of, ll, ml */
+static size_t ZSTD_estimateBlockSize(const BYTE* literals, size_t litSize,
+ const BYTE* ofCodeTable,
+ const BYTE* llCodeTable,
+ const BYTE* mlCodeTable,
+ size_t nbSeq,
+ const ZSTD_entropyCTables_t* entropy,
+ const ZSTD_entropyCTablesMetadata_t* entropyMetadata,
+ void* workspace, size_t wkspSize,
+ int writeLitEntropy, int writeSeqEntropy) {
+ size_t const literalsSize = ZSTD_estimateBlockSize_literal(literals, litSize,
+ &entropy->huf, &entropyMetadata->hufMetadata,
+ workspace, wkspSize, writeLitEntropy);
+ size_t const seqSize = ZSTD_estimateBlockSize_sequences(ofCodeTable, llCodeTable, mlCodeTable,
+ nbSeq, &entropy->fse, &entropyMetadata->fseMetadata,
+ workspace, wkspSize, writeSeqEntropy);
+ return seqSize + literalsSize + ZSTD_blockHeaderSize;
+}
+
+/* Builds entropy statistics and uses them for blocksize estimation.
+ *
+ * Returns the estimated compressed size of the seqStore, or a zstd error.
+ */
+static size_t ZSTD_buildEntropyStatisticsAndEstimateSubBlockSize(seqStore_t* seqStore, ZSTD_CCtx* zc) {
+ ZSTD_entropyCTablesMetadata_t* entropyMetadata = &zc->blockSplitCtx.entropyMetadata;
+ DEBUGLOG(6, "ZSTD_buildEntropyStatisticsAndEstimateSubBlockSize()");
+ FORWARD_IF_ERROR(ZSTD_buildBlockEntropyStats(seqStore,
+ &zc->blockState.prevCBlock->entropy,
+ &zc->blockState.nextCBlock->entropy,
+ &zc->appliedParams,
+ entropyMetadata,
+ zc->entropyWorkspace, ENTROPY_WORKSPACE_SIZE /* statically allocated in resetCCtx */), "");
+ return ZSTD_estimateBlockSize(seqStore->litStart, (size_t)(seqStore->lit - seqStore->litStart),
+ seqStore->ofCode, seqStore->llCode, seqStore->mlCode,
+ (size_t)(seqStore->sequences - seqStore->sequencesStart),
+ &zc->blockState.nextCBlock->entropy, entropyMetadata, zc->entropyWorkspace, ENTROPY_WORKSPACE_SIZE,
+ (int)(entropyMetadata->hufMetadata.hType == set_compressed), 1);
+}
+
+/* Returns literals bytes represented in a seqStore */
+static size_t ZSTD_countSeqStoreLiteralsBytes(const seqStore_t* const seqStore) {
+ size_t literalsBytes = 0;
+ size_t const nbSeqs = seqStore->sequences - seqStore->sequencesStart;
+ size_t i;
+ for (i = 0; i < nbSeqs; ++i) {
+ seqDef seq = seqStore->sequencesStart[i];
+ literalsBytes += seq.litLength;
+ if (i == seqStore->longLengthPos && seqStore->longLengthType == ZSTD_llt_literalLength) {
+ literalsBytes += 0x10000;
+ }
+ }
+ return literalsBytes;
+}
+
+/* Returns match bytes represented in a seqStore */
+static size_t ZSTD_countSeqStoreMatchBytes(const seqStore_t* const seqStore) {
+ size_t matchBytes = 0;
+ size_t const nbSeqs = seqStore->sequences - seqStore->sequencesStart;
+ size_t i;
+ for (i = 0; i < nbSeqs; ++i) {
+ seqDef seq = seqStore->sequencesStart[i];
+ matchBytes += seq.mlBase + MINMATCH;
+ if (i == seqStore->longLengthPos && seqStore->longLengthType == ZSTD_llt_matchLength) {
+ matchBytes += 0x10000;
+ }
+ }
+ return matchBytes;
+}
+
+/* Derives the seqStore that is a chunk of the originalSeqStore from [startIdx, endIdx).
+ * Stores the result in resultSeqStore.
+ */
+static void ZSTD_deriveSeqStoreChunk(seqStore_t* resultSeqStore,
+ const seqStore_t* originalSeqStore,
+ size_t startIdx, size_t endIdx) {
+ BYTE* const litEnd = originalSeqStore->lit;
+ size_t literalsBytes;
+ size_t literalsBytesPreceding = 0;
+
+ *resultSeqStore = *originalSeqStore;
+ if (startIdx > 0) {
+ resultSeqStore->sequences = originalSeqStore->sequencesStart + startIdx;
+ literalsBytesPreceding = ZSTD_countSeqStoreLiteralsBytes(resultSeqStore);
+ }
+
+ /* Move longLengthPos into the correct position if necessary */
+ if (originalSeqStore->longLengthType != ZSTD_llt_none) {
+ if (originalSeqStore->longLengthPos < startIdx || originalSeqStore->longLengthPos > endIdx) {
+ resultSeqStore->longLengthType = ZSTD_llt_none;
+ } else {
+ resultSeqStore->longLengthPos -= (U32)startIdx;
+ }
+ }
+ resultSeqStore->sequencesStart = originalSeqStore->sequencesStart + startIdx;
+ resultSeqStore->sequences = originalSeqStore->sequencesStart + endIdx;
+ literalsBytes = ZSTD_countSeqStoreLiteralsBytes(resultSeqStore);
+ resultSeqStore->litStart += literalsBytesPreceding;
+ if (endIdx == (size_t)(originalSeqStore->sequences - originalSeqStore->sequencesStart)) {
+ /* This accounts for possible last literals if the derived chunk reaches the end of the block */
+ resultSeqStore->lit = litEnd;
+ } else {
+ resultSeqStore->lit = resultSeqStore->litStart+literalsBytes;
+ }
+ resultSeqStore->llCode += startIdx;
+ resultSeqStore->mlCode += startIdx;
+ resultSeqStore->ofCode += startIdx;
+}
+
+/*
+ * Returns the raw offset represented by the combination of offCode, ll0, and repcode history.
+ * offCode must represent a repcode in the numeric representation of ZSTD_storeSeq().
+ */
+static U32
+ZSTD_resolveRepcodeToRawOffset(const U32 rep[ZSTD_REP_NUM], const U32 offCode, const U32 ll0)
+{
+ U32 const adjustedOffCode = STORED_REPCODE(offCode) - 1 + ll0; /* [ 0 - 3 ] */
+ assert(STORED_IS_REPCODE(offCode));
+ if (adjustedOffCode == ZSTD_REP_NUM) {
+ /* litlength == 0 and offCode == 2 implies selection of first repcode - 1 */
+ assert(rep[0] > 0);
+ return rep[0] - 1;
+ }
+ return rep[adjustedOffCode];
+}
+
+/*
+ * ZSTD_seqStore_resolveOffCodes() reconciles any possible divergences in offset history that may arise
+ * due to emission of RLE/raw blocks that disturb the offset history,
+ * and replaces any repcodes within the seqStore that may be invalid.
+ *
+ * dRepcodes are updated as would be on the decompression side.
+ * cRepcodes are updated exactly in accordance with the seqStore.
+ *
+ * Note : this function assumes seq->offBase respects the following numbering scheme :
+ * 0 : invalid
+ * 1-3 : repcode 1-3
+ * 4+ : real_offset+3
+ */
+static void ZSTD_seqStore_resolveOffCodes(repcodes_t* const dRepcodes, repcodes_t* const cRepcodes,
+ seqStore_t* const seqStore, U32 const nbSeq) {
+ U32 idx = 0;
+ for (; idx < nbSeq; ++idx) {
+ seqDef* const seq = seqStore->sequencesStart + idx;
+ U32 const ll0 = (seq->litLength == 0);
+ U32 const offCode = OFFBASE_TO_STORED(seq->offBase);
+ assert(seq->offBase > 0);
+ if (STORED_IS_REPCODE(offCode)) {
+ U32 const dRawOffset = ZSTD_resolveRepcodeToRawOffset(dRepcodes->rep, offCode, ll0);
+ U32 const cRawOffset = ZSTD_resolveRepcodeToRawOffset(cRepcodes->rep, offCode, ll0);
+ /* Adjust simulated decompression repcode history if we come across a mismatch. Replace
+ * the repcode with the offset it actually references, determined by the compression
+ * repcode history.
+ */
+ if (dRawOffset != cRawOffset) {
+ seq->offBase = cRawOffset + ZSTD_REP_NUM;
+ }
+ }
+ /* Compression repcode history is always updated with values directly from the unmodified seqStore.
+ * Decompression repcode history may use modified seq->offset value taken from compression repcode history.
+ */
+ ZSTD_updateRep(dRepcodes->rep, OFFBASE_TO_STORED(seq->offBase), ll0);
+ ZSTD_updateRep(cRepcodes->rep, offCode, ll0);
+ }
+}
+
+/* ZSTD_compressSeqStore_singleBlock():
+ * Compresses a seqStore into a block with a block header, into the buffer dst.
+ *
+ * Returns the total size of that block (including header) or a ZSTD error code.
+ */
+static size_t
+ZSTD_compressSeqStore_singleBlock(ZSTD_CCtx* zc, seqStore_t* const seqStore,
+ repcodes_t* const dRep, repcodes_t* const cRep,
+ void* dst, size_t dstCapacity,
+ const void* src, size_t srcSize,
+ U32 lastBlock, U32 isPartition)
+{
+ const U32 rleMaxLength = 25;
+ BYTE* op = (BYTE*)dst;
+ const BYTE* ip = (const BYTE*)src;
+ size_t cSize;
+ size_t cSeqsSize;
+
+ /* In case of an RLE or raw block, the simulated decompression repcode history must be reset */
+ repcodes_t const dRepOriginal = *dRep;
+ DEBUGLOG(5, "ZSTD_compressSeqStore_singleBlock");
+ if (isPartition)
+ ZSTD_seqStore_resolveOffCodes(dRep, cRep, seqStore, (U32)(seqStore->sequences - seqStore->sequencesStart));
+
+ RETURN_ERROR_IF(dstCapacity < ZSTD_blockHeaderSize, dstSize_tooSmall, "Block header doesn't fit");
+ cSeqsSize = ZSTD_entropyCompressSeqStore(seqStore,
+ &zc->blockState.prevCBlock->entropy, &zc->blockState.nextCBlock->entropy,
+ &zc->appliedParams,
+ op + ZSTD_blockHeaderSize, dstCapacity - ZSTD_blockHeaderSize,
+ srcSize,
+ zc->entropyWorkspace, ENTROPY_WORKSPACE_SIZE /* statically allocated in resetCCtx */,
+ zc->bmi2);
+ FORWARD_IF_ERROR(cSeqsSize, "ZSTD_entropyCompressSeqStore failed!");
+
+ if (!zc->isFirstBlock &&
+ cSeqsSize < rleMaxLength &&
+ ZSTD_isRLE((BYTE const*)src, srcSize)) {
+ /* We don't want to emit our first block as a RLE even if it qualifies because
+ * doing so will cause the decoder (cli only) to throw a "should consume all input error."
+ * This is only an issue for zstd <= v1.4.3
+ */
+ cSeqsSize = 1;
+ }
+
+ if (zc->seqCollector.collectSequences) {
+ ZSTD_copyBlockSequences(zc);
+ ZSTD_blockState_confirmRepcodesAndEntropyTables(&zc->blockState);
+ return 0;
+ }
+
+ if (cSeqsSize == 0) {
+ cSize = ZSTD_noCompressBlock(op, dstCapacity, ip, srcSize, lastBlock);
+ FORWARD_IF_ERROR(cSize, "Nocompress block failed");
+ DEBUGLOG(4, "Writing out nocompress block, size: %zu", cSize);
+ *dRep = dRepOriginal; /* reset simulated decompression repcode history */
+ } else if (cSeqsSize == 1) {
+ cSize = ZSTD_rleCompressBlock(op, dstCapacity, *ip, srcSize, lastBlock);
+ FORWARD_IF_ERROR(cSize, "RLE compress block failed");
+ DEBUGLOG(4, "Writing out RLE block, size: %zu", cSize);
+ *dRep = dRepOriginal; /* reset simulated decompression repcode history */
+ } else {
+ ZSTD_blockState_confirmRepcodesAndEntropyTables(&zc->blockState);
+ writeBlockHeader(op, cSeqsSize, srcSize, lastBlock);
+ cSize = ZSTD_blockHeaderSize + cSeqsSize;
+ DEBUGLOG(4, "Writing out compressed block, size: %zu", cSize);
+ }
+
+ if (zc->blockState.prevCBlock->entropy.fse.offcode_repeatMode == FSE_repeat_valid)
+ zc->blockState.prevCBlock->entropy.fse.offcode_repeatMode = FSE_repeat_check;
+
+ return cSize;
+}
+
+/* Struct to keep track of where we are in our recursive calls. */
+typedef struct {
+ U32* splitLocations; /* Array of split indices */
+ size_t idx; /* The current index within splitLocations being worked on */
+} seqStoreSplits;
+
+#define MIN_SEQUENCES_BLOCK_SPLITTING 300
+
+/* Helper function to perform the recursive search for block splits.
+ * Estimates the cost of seqStore prior to split, and estimates the cost of splitting the sequences in half.
+ * If advantageous to split, then we recurse down the two sub-blocks. If not, or if an error occurred in estimation, then
+ * we do not recurse.
+ *
+ * Note: The recursion depth is capped by a heuristic minimum number of sequences, defined by MIN_SEQUENCES_BLOCK_SPLITTING.
+ * In theory, this means the absolute largest recursion depth is 10 == log2(maxNbSeqInBlock/MIN_SEQUENCES_BLOCK_SPLITTING).
+ * In practice, recursion depth usually doesn't go beyond 4.
+ *
+ * Furthermore, the number of splits is capped by ZSTD_MAX_NB_BLOCK_SPLITS. At ZSTD_MAX_NB_BLOCK_SPLITS == 196 with the current existing blockSize
+ * maximum of 128 KB, this value is actually impossible to reach.
+ */
+static void
+ZSTD_deriveBlockSplitsHelper(seqStoreSplits* splits, size_t startIdx, size_t endIdx,
+ ZSTD_CCtx* zc, const seqStore_t* origSeqStore)
+{
+ seqStore_t* fullSeqStoreChunk = &zc->blockSplitCtx.fullSeqStoreChunk;
+ seqStore_t* firstHalfSeqStore = &zc->blockSplitCtx.firstHalfSeqStore;
+ seqStore_t* secondHalfSeqStore = &zc->blockSplitCtx.secondHalfSeqStore;
+ size_t estimatedOriginalSize;
+ size_t estimatedFirstHalfSize;
+ size_t estimatedSecondHalfSize;
+ size_t midIdx = (startIdx + endIdx)/2;
+
+ if (endIdx - startIdx < MIN_SEQUENCES_BLOCK_SPLITTING || splits->idx >= ZSTD_MAX_NB_BLOCK_SPLITS) {
+ DEBUGLOG(6, "ZSTD_deriveBlockSplitsHelper: Too few sequences");
+ return;
+ }
+ DEBUGLOG(4, "ZSTD_deriveBlockSplitsHelper: startIdx=%zu endIdx=%zu", startIdx, endIdx);
+ ZSTD_deriveSeqStoreChunk(fullSeqStoreChunk, origSeqStore, startIdx, endIdx);
+ ZSTD_deriveSeqStoreChunk(firstHalfSeqStore, origSeqStore, startIdx, midIdx);
+ ZSTD_deriveSeqStoreChunk(secondHalfSeqStore, origSeqStore, midIdx, endIdx);
+ estimatedOriginalSize = ZSTD_buildEntropyStatisticsAndEstimateSubBlockSize(fullSeqStoreChunk, zc);
+ estimatedFirstHalfSize = ZSTD_buildEntropyStatisticsAndEstimateSubBlockSize(firstHalfSeqStore, zc);
+ estimatedSecondHalfSize = ZSTD_buildEntropyStatisticsAndEstimateSubBlockSize(secondHalfSeqStore, zc);
+ DEBUGLOG(4, "Estimated original block size: %zu -- First half split: %zu -- Second half split: %zu",
+ estimatedOriginalSize, estimatedFirstHalfSize, estimatedSecondHalfSize);
+ if (ZSTD_isError(estimatedOriginalSize) || ZSTD_isError(estimatedFirstHalfSize) || ZSTD_isError(estimatedSecondHalfSize)) {
+ return;
+ }
+ if (estimatedFirstHalfSize + estimatedSecondHalfSize < estimatedOriginalSize) {
+ ZSTD_deriveBlockSplitsHelper(splits, startIdx, midIdx, zc, origSeqStore);
+ splits->splitLocations[splits->idx] = (U32)midIdx;
+ splits->idx++;
+ ZSTD_deriveBlockSplitsHelper(splits, midIdx, endIdx, zc, origSeqStore);
+ }
+}
+
+/* Base recursive function. Populates a table with intra-block partition indices that can improve compression ratio.
+ *
+ * Returns the number of splits made (which equals the size of the partition table - 1).
+ */
+static size_t ZSTD_deriveBlockSplits(ZSTD_CCtx* zc, U32 partitions[], U32 nbSeq) {
+ seqStoreSplits splits = {partitions, 0};
+ if (nbSeq <= 4) {
+ DEBUGLOG(4, "ZSTD_deriveBlockSplits: Too few sequences to split");
+ /* Refuse to try and split anything with less than 4 sequences */
+ return 0;
+ }
+ ZSTD_deriveBlockSplitsHelper(&splits, 0, nbSeq, zc, &zc->seqStore);
+ splits.splitLocations[splits.idx] = nbSeq;
+ DEBUGLOG(5, "ZSTD_deriveBlockSplits: final nb partitions: %zu", splits.idx+1);
+ return splits.idx;
+}
+
+/* ZSTD_compressBlock_splitBlock():
+ * Attempts to split a given block into multiple blocks to improve compression ratio.
+ *
+ * Returns combined size of all blocks (which includes headers), or a ZSTD error code.
+ */
+static size_t
+ZSTD_compressBlock_splitBlock_internal(ZSTD_CCtx* zc, void* dst, size_t dstCapacity,
+ const void* src, size_t blockSize, U32 lastBlock, U32 nbSeq)
+{
+ size_t cSize = 0;
+ const BYTE* ip = (const BYTE*)src;
+ BYTE* op = (BYTE*)dst;
+ size_t i = 0;
+ size_t srcBytesTotal = 0;
+ U32* partitions = zc->blockSplitCtx.partitions; /* size == ZSTD_MAX_NB_BLOCK_SPLITS */
+ seqStore_t* nextSeqStore = &zc->blockSplitCtx.nextSeqStore;
+ seqStore_t* currSeqStore = &zc->blockSplitCtx.currSeqStore;
+ size_t numSplits = ZSTD_deriveBlockSplits(zc, partitions, nbSeq);
+
+ /* If a block is split and some partitions are emitted as RLE/uncompressed, then repcode history
+ * may become invalid. In order to reconcile potentially invalid repcodes, we keep track of two
+ * separate repcode histories that simulate repcode history on compression and decompression side,
+ * and use the histories to determine whether we must replace a particular repcode with its raw offset.
+ *
+ * 1) cRep gets updated for each partition, regardless of whether the block was emitted as uncompressed
+ * or RLE. This allows us to retrieve the offset value that an invalid repcode references within
+ * a nocompress/RLE block.
+ * 2) dRep gets updated only for compressed partitions, and when a repcode gets replaced, will use
+ * the replacement offset value rather than the original repcode to update the repcode history.
+ * dRep also will be the final repcode history sent to the next block.
+ *
+ * See ZSTD_seqStore_resolveOffCodes() for more details.
+ */
+ repcodes_t dRep;
+ repcodes_t cRep;
+ ZSTD_memcpy(dRep.rep, zc->blockState.prevCBlock->rep, sizeof(repcodes_t));
+ ZSTD_memcpy(cRep.rep, zc->blockState.prevCBlock->rep, sizeof(repcodes_t));
+ ZSTD_memset(nextSeqStore, 0, sizeof(seqStore_t));
+
+ DEBUGLOG(4, "ZSTD_compressBlock_splitBlock_internal (dstCapacity=%u, dictLimit=%u, nextToUpdate=%u)",
+ (unsigned)dstCapacity, (unsigned)zc->blockState.matchState.window.dictLimit,
+ (unsigned)zc->blockState.matchState.nextToUpdate);
+
+ if (numSplits == 0) {
+ size_t cSizeSingleBlock = ZSTD_compressSeqStore_singleBlock(zc, &zc->seqStore,
+ &dRep, &cRep,
+ op, dstCapacity,
+ ip, blockSize,
+ lastBlock, 0 /* isPartition */);
+ FORWARD_IF_ERROR(cSizeSingleBlock, "Compressing single block from splitBlock_internal() failed!");
+ DEBUGLOG(5, "ZSTD_compressBlock_splitBlock_internal: No splits");
+ assert(cSizeSingleBlock <= ZSTD_BLOCKSIZE_MAX + ZSTD_blockHeaderSize);
+ return cSizeSingleBlock;
+ }
+
+ ZSTD_deriveSeqStoreChunk(currSeqStore, &zc->seqStore, 0, partitions[0]);
+ for (i = 0; i <= numSplits; ++i) {
+ size_t srcBytes;
+ size_t cSizeChunk;
+ U32 const lastPartition = (i == numSplits);
+ U32 lastBlockEntireSrc = 0;
+
+ srcBytes = ZSTD_countSeqStoreLiteralsBytes(currSeqStore) + ZSTD_countSeqStoreMatchBytes(currSeqStore);
+ srcBytesTotal += srcBytes;
+ if (lastPartition) {
+ /* This is the final partition, need to account for possible last literals */
+ srcBytes += blockSize - srcBytesTotal;
+ lastBlockEntireSrc = lastBlock;
+ } else {
+ ZSTD_deriveSeqStoreChunk(nextSeqStore, &zc->seqStore, partitions[i], partitions[i+1]);
+ }
+
+ cSizeChunk = ZSTD_compressSeqStore_singleBlock(zc, currSeqStore,
+ &dRep, &cRep,
+ op, dstCapacity,
+ ip, srcBytes,
+ lastBlockEntireSrc, 1 /* isPartition */);
+ DEBUGLOG(5, "Estimated size: %zu actual size: %zu", ZSTD_buildEntropyStatisticsAndEstimateSubBlockSize(currSeqStore, zc), cSizeChunk);
+ FORWARD_IF_ERROR(cSizeChunk, "Compressing chunk failed!");
+
+ ip += srcBytes;
+ op += cSizeChunk;
+ dstCapacity -= cSizeChunk;
+ cSize += cSizeChunk;
+ *currSeqStore = *nextSeqStore;
+ assert(cSizeChunk <= ZSTD_BLOCKSIZE_MAX + ZSTD_blockHeaderSize);
+ }
+ /* cRep and dRep may have diverged during the compression. If so, we use the dRep repcodes
+ * for the next block.
+ */
+ ZSTD_memcpy(zc->blockState.prevCBlock->rep, dRep.rep, sizeof(repcodes_t));
+ return cSize;
+}
+
+static size_t
+ZSTD_compressBlock_splitBlock(ZSTD_CCtx* zc,
+ void* dst, size_t dstCapacity,
+ const void* src, size_t srcSize, U32 lastBlock)
+{
+ const BYTE* ip = (const BYTE*)src;
+ BYTE* op = (BYTE*)dst;
+ U32 nbSeq;
+ size_t cSize;
+ DEBUGLOG(4, "ZSTD_compressBlock_splitBlock");
+ assert(zc->appliedParams.useBlockSplitter == ZSTD_ps_enable);
+
+ { const size_t bss = ZSTD_buildSeqStore(zc, src, srcSize);
+ FORWARD_IF_ERROR(bss, "ZSTD_buildSeqStore failed");
+ if (bss == ZSTDbss_noCompress) {
+ if (zc->blockState.prevCBlock->entropy.fse.offcode_repeatMode == FSE_repeat_valid)
+ zc->blockState.prevCBlock->entropy.fse.offcode_repeatMode = FSE_repeat_check;
+ cSize = ZSTD_noCompressBlock(op, dstCapacity, ip, srcSize, lastBlock);
+ FORWARD_IF_ERROR(cSize, "ZSTD_noCompressBlock failed");
+ DEBUGLOG(4, "ZSTD_compressBlock_splitBlock: Nocompress block");
+ return cSize;
+ }
+ nbSeq = (U32)(zc->seqStore.sequences - zc->seqStore.sequencesStart);
+ }
+
+ cSize = ZSTD_compressBlock_splitBlock_internal(zc, dst, dstCapacity, src, srcSize, lastBlock, nbSeq);
+ FORWARD_IF_ERROR(cSize, "Splitting blocks failed!");
+ return cSize;
+}
+
+static size_t
+ZSTD_compressBlock_internal(ZSTD_CCtx* zc,
+ void* dst, size_t dstCapacity,
+ const void* src, size_t srcSize, U32 frame)
{
/* This the upper bound for the length of an rle block.
* This isn't the actual upper bound. Finding the real threshold
@@ -2632,12 +3692,12 @@ static size_t ZSTD_compressBlock_internal(ZSTD_CCtx* zc,
if (zc->seqCollector.collectSequences) {
ZSTD_copyBlockSequences(zc);
- ZSTD_confirmRepcodesAndEntropyTables(zc);
+ ZSTD_blockState_confirmRepcodesAndEntropyTables(&zc->blockState);
return 0;
}
/* encode sequences and literals */
- cSize = ZSTD_entropyCompressSequences(&zc->seqStore,
+ cSize = ZSTD_entropyCompressSeqStore(&zc->seqStore,
&zc->blockState.prevCBlock->entropy, &zc->blockState.nextCBlock->entropy,
&zc->appliedParams,
dst, dstCapacity,
@@ -2645,12 +3705,6 @@ static size_t ZSTD_compressBlock_internal(ZSTD_CCtx* zc,
zc->entropyWorkspace, ENTROPY_WORKSPACE_SIZE /* statically allocated in resetCCtx */,
zc->bmi2);
- if (zc->seqCollector.collectSequences) {
- ZSTD_copyBlockSequences(zc);
- return 0;
- }
-
-
if (frame &&
/* We don't want to emit our first block as a RLE even if it qualifies because
* doing so will cause the decoder (cli only) to throw a "should consume all input error."
@@ -2666,7 +3720,7 @@ static size_t ZSTD_compressBlock_internal(ZSTD_CCtx* zc,
out:
if (!ZSTD_isError(cSize) && cSize > 1) {
- ZSTD_confirmRepcodesAndEntropyTables(zc);
+ ZSTD_blockState_confirmRepcodesAndEntropyTables(&zc->blockState);
}
/* We check that dictionaries have offset codes available for the first
* block. After the first block, the offcode table might not have large
@@ -2719,7 +3773,7 @@ static size_t ZSTD_compressBlock_targetCBlockSize_body(ZSTD_CCtx* zc,
size_t const maxCSize = srcSize - ZSTD_minGain(srcSize, zc->appliedParams.cParams.strategy);
FORWARD_IF_ERROR(cSize, "ZSTD_compressSuperBlock failed");
if (cSize != 0 && cSize < maxCSize + ZSTD_blockHeaderSize) {
- ZSTD_confirmRepcodesAndEntropyTables(zc);
+ ZSTD_blockState_confirmRepcodesAndEntropyTables(&zc->blockState);
return cSize;
}
}
@@ -2759,9 +3813,9 @@ static void ZSTD_overflowCorrectIfNeeded(ZSTD_matchState_t* ms,
void const* ip,
void const* iend)
{
- if (ZSTD_window_needOverflowCorrection(ms->window, iend)) {
- U32 const maxDist = (U32)1 << params->cParams.windowLog;
- U32 const cycleLog = ZSTD_cycleLog(params->cParams.chainLog, params->cParams.strategy);
+ U32 const cycleLog = ZSTD_cycleLog(params->cParams.chainLog, params->cParams.strategy);
+ U32 const maxDist = (U32)1 << params->cParams.windowLog;
+ if (ZSTD_window_needOverflowCorrection(ms->window, cycleLog, maxDist, ms->loadedDictEnd, ip, iend)) {
U32 const correction = ZSTD_window_correctOverflow(&ms->window, cycleLog, maxDist, ip);
ZSTD_STATIC_ASSERT(ZSTD_CHAINLOG_MAX <= 30);
ZSTD_STATIC_ASSERT(ZSTD_WINDOWLOG_MAX_32 <= 30);
@@ -2784,7 +3838,7 @@ static void ZSTD_overflowCorrectIfNeeded(ZSTD_matchState_t* ms,
* Frame is supposed already started (header already produced)
* @return : compressed size, or an error code
*/
-static size_t ZSTD_compress_frameChunk (ZSTD_CCtx* cctx,
+static size_t ZSTD_compress_frameChunk(ZSTD_CCtx* cctx,
void* dst, size_t dstCapacity,
const void* src, size_t srcSize,
U32 lastFrameChunk)
@@ -2814,6 +3868,7 @@ static size_t ZSTD_compress_frameChunk (ZSTD_CCtx* cctx,
ZSTD_overflowCorrectIfNeeded(
ms, &cctx->workspace, &cctx->appliedParams, ip, ip + blockSize);
ZSTD_checkDictValidity(&ms->window, ip + blockSize, maxDist, &ms->loadedDictEnd, &ms->dictMatchState);
+ ZSTD_window_enforceMaxDist(&ms->window, ip, maxDist, &ms->loadedDictEnd, &ms->dictMatchState);
/* Ensure hash/chain table insertion resumes no sooner than lowlimit */
if (ms->nextToUpdate < ms->window.lowLimit) ms->nextToUpdate = ms->window.lowLimit;
@@ -2824,6 +3879,10 @@ static size_t ZSTD_compress_frameChunk (ZSTD_CCtx* cctx,
FORWARD_IF_ERROR(cSize, "ZSTD_compressBlock_targetCBlockSize failed");
assert(cSize > 0);
assert(cSize <= blockSize + ZSTD_blockHeaderSize);
+ } else if (ZSTD_blockSplitterEnabled(&cctx->appliedParams)) {
+ cSize = ZSTD_compressBlock_splitBlock(cctx, op, dstCapacity, ip, blockSize, lastBlock);
+ FORWARD_IF_ERROR(cSize, "ZSTD_compressBlock_splitBlock failed");
+ assert(cSize > 0 || cctx->seqCollector.collectSequences == 1);
} else {
cSize = ZSTD_compressBlock_internal(cctx,
op+ZSTD_blockHeaderSize, dstCapacity-ZSTD_blockHeaderSize,
@@ -2946,7 +4005,7 @@ size_t ZSTD_referenceExternalSequences(ZSTD_CCtx* cctx, rawSeq* seq, size_t nbSe
{
RETURN_ERROR_IF(cctx->stage != ZSTDcs_init, stage_wrong,
"wrong cctx stage");
- RETURN_ERROR_IF(cctx->appliedParams.ldmParams.enableLdm,
+ RETURN_ERROR_IF(cctx->appliedParams.ldmParams.enableLdm == ZSTD_ps_enable,
parameter_unsupported,
"incompatible with ldm");
cctx->externSeqStore.seq = seq;
@@ -2983,11 +4042,12 @@ static size_t ZSTD_compressContinue_internal (ZSTD_CCtx* cctx,
if (!srcSize) return fhSize; /* do not generate an empty block if no input */
- if (!ZSTD_window_update(&ms->window, src, srcSize)) {
+ if (!ZSTD_window_update(&ms->window, src, srcSize, ms->forceNonContiguous)) {
+ ms->forceNonContiguous = 0;
ms->nextToUpdate = ms->window.dictLimit;
}
- if (cctx->appliedParams.ldmParams.enableLdm) {
- ZSTD_window_update(&cctx->ldmState.window, src, srcSize);
+ if (cctx->appliedParams.ldmParams.enableLdm == ZSTD_ps_enable) {
+ ZSTD_window_update(&cctx->ldmState.window, src, srcSize, /* forceNonContiguous */ 0);
}
if (!frame) {
@@ -3055,63 +4115,86 @@ static size_t ZSTD_loadDictionaryContent(ZSTD_matchState_t* ms,
{
const BYTE* ip = (const BYTE*) src;
const BYTE* const iend = ip + srcSize;
-
- ZSTD_window_update(&ms->window, src, srcSize);
- ms->loadedDictEnd = params->forceWindow ? 0 : (U32)(iend - ms->window.base);
-
- if (params->ldmParams.enableLdm && ls != NULL) {
- ZSTD_window_update(&ls->window, src, srcSize);
- ls->loadedDictEnd = params->forceWindow ? 0 : (U32)(iend - ls->window.base);
- }
+ int const loadLdmDict = params->ldmParams.enableLdm == ZSTD_ps_enable && ls != NULL;
/* Assert that we the ms params match the params we're being given */
ZSTD_assertEqualCParams(params->cParams, ms->cParams);
+ if (srcSize > ZSTD_CHUNKSIZE_MAX) {
+ /* Allow the dictionary to set indices up to exactly ZSTD_CURRENT_MAX.
+ * Dictionaries right at the edge will immediately trigger overflow
+ * correction, but I don't want to insert extra constraints here.
+ */
+ U32 const maxDictSize = ZSTD_CURRENT_MAX - 1;
+ /* We must have cleared our windows when our source is this large. */
+ assert(ZSTD_window_isEmpty(ms->window));
+ if (loadLdmDict)
+ assert(ZSTD_window_isEmpty(ls->window));
+ /* If the dictionary is too large, only load the suffix of the dictionary. */
+ if (srcSize > maxDictSize) {
+ ip = iend - maxDictSize;
+ src = ip;
+ srcSize = maxDictSize;
+ }
+ }
+
+ DEBUGLOG(4, "ZSTD_loadDictionaryContent(): useRowMatchFinder=%d", (int)params->useRowMatchFinder);
+ ZSTD_window_update(&ms->window, src, srcSize, /* forceNonContiguous */ 0);
+ ms->loadedDictEnd = params->forceWindow ? 0 : (U32)(iend - ms->window.base);
+ ms->forceNonContiguous = params->deterministicRefPrefix;
+
+ if (loadLdmDict) {
+ ZSTD_window_update(&ls->window, src, srcSize, /* forceNonContiguous */ 0);
+ ls->loadedDictEnd = params->forceWindow ? 0 : (U32)(iend - ls->window.base);
+ }
+
if (srcSize <= HASH_READ_SIZE) return 0;
- while (iend - ip > HASH_READ_SIZE) {
- size_t const remaining = (size_t)(iend - ip);
- size_t const chunk = MIN(remaining, ZSTD_CHUNKSIZE_MAX);
- const BYTE* const ichunk = ip + chunk;
+ ZSTD_overflowCorrectIfNeeded(ms, ws, params, ip, iend);
- ZSTD_overflowCorrectIfNeeded(ms, ws, params, ip, ichunk);
+ if (loadLdmDict)
+ ZSTD_ldm_fillHashTable(ls, ip, iend, ¶ms->ldmParams);
- if (params->ldmParams.enableLdm && ls != NULL)
- ZSTD_ldm_fillHashTable(ls, (const BYTE*)src, (const BYTE*)src + srcSize, ¶ms->ldmParams);
+ switch(params->cParams.strategy)
+ {
+ case ZSTD_fast:
+ ZSTD_fillHashTable(ms, iend, dtlm);
+ break;
+ case ZSTD_dfast:
+ ZSTD_fillDoubleHashTable(ms, iend, dtlm);
+ break;
- switch(params->cParams.strategy)
- {
- case ZSTD_fast:
- ZSTD_fillHashTable(ms, ichunk, dtlm);
- break;
- case ZSTD_dfast:
- ZSTD_fillDoubleHashTable(ms, ichunk, dtlm);
- break;
-
- case ZSTD_greedy:
- case ZSTD_lazy:
- case ZSTD_lazy2:
- if (chunk >= HASH_READ_SIZE && ms->dedicatedDictSearch) {
- assert(chunk == remaining); /* must load everything in one go */
- ZSTD_dedicatedDictSearch_lazy_loadDictionary(ms, ichunk-HASH_READ_SIZE);
- } else if (chunk >= HASH_READ_SIZE) {
- ZSTD_insertAndFindFirstIndex(ms, ichunk-HASH_READ_SIZE);
+ case ZSTD_greedy:
+ case ZSTD_lazy:
+ case ZSTD_lazy2:
+ assert(srcSize >= HASH_READ_SIZE);
+ if (ms->dedicatedDictSearch) {
+ assert(ms->chainTable != NULL);
+ ZSTD_dedicatedDictSearch_lazy_loadDictionary(ms, iend-HASH_READ_SIZE);
+ } else {
+ assert(params->useRowMatchFinder != ZSTD_ps_auto);
+ if (params->useRowMatchFinder == ZSTD_ps_enable) {
+ size_t const tagTableSize = ((size_t)1 << params->cParams.hashLog) * sizeof(U16);
+ ZSTD_memset(ms->tagTable, 0, tagTableSize);
+ ZSTD_row_update(ms, iend-HASH_READ_SIZE);
+ DEBUGLOG(4, "Using row-based hash table for lazy dict");
+ } else {
+ ZSTD_insertAndFindFirstIndex(ms, iend-HASH_READ_SIZE);
+ DEBUGLOG(4, "Using chain-based hash table for lazy dict");
}
- break;
-
- case ZSTD_btlazy2: /* we want the dictionary table fully sorted */
- case ZSTD_btopt:
- case ZSTD_btultra:
- case ZSTD_btultra2:
- if (chunk >= HASH_READ_SIZE)
- ZSTD_updateTree(ms, ichunk-HASH_READ_SIZE, ichunk);
- break;
-
- default:
- assert(0); /* not possible : not a valid strategy id */
}
+ break;
- ip = ichunk;
+ case ZSTD_btlazy2: /* we want the dictionary table fully sorted */
+ case ZSTD_btopt:
+ case ZSTD_btultra:
+ case ZSTD_btultra2:
+ assert(srcSize >= HASH_READ_SIZE);
+ ZSTD_updateTree(ms, iend-HASH_READ_SIZE, iend);
+ break;
+
+ default:
+ assert(0); /* not possible : not a valid strategy id */
}
ms->nextToUpdate = (U32)(iend - ms->window.base);
@@ -3250,7 +4333,6 @@ static size_t ZSTD_loadZstdDictionary(ZSTD_compressedBlockState_t* bs,
const BYTE* const dictEnd = dictPtr + dictSize;
size_t dictID;
size_t eSize;
-
ZSTD_STATIC_ASSERT(HUF_WORKSPACE_SIZE >= (1<<MAX(MLFSELog,LLFSELog)));
assert(dictSize >= 8);
assert(MEM_readLE32(dictPtr) == ZSTD_MAGIC_DICTIONARY);
@@ -3321,6 +4403,7 @@ static size_t ZSTD_compressBegin_internal(ZSTD_CCtx* cctx,
const ZSTD_CCtx_params* params, U64 pledgedSrcSize,
ZSTD_buffered_policy_e zbuff)
{
+ size_t const dictContentSize = cdict ? cdict->dictContentSize : dictSize;
DEBUGLOG(4, "ZSTD_compressBegin_internal: wlog=%u", params->cParams.windowLog);
/* params are supposed to be fully validated at this point */
assert(!ZSTD_isError(ZSTD_checkCParams(params->cParams)));
@@ -3335,7 +4418,8 @@ static size_t ZSTD_compressBegin_internal(ZSTD_CCtx* cctx,
return ZSTD_resetCCtx_usingCDict(cctx, cdict, params, pledgedSrcSize, zbuff);
}
- FORWARD_IF_ERROR( ZSTD_resetCCtx_internal(cctx, *params, pledgedSrcSize,
+ FORWARD_IF_ERROR( ZSTD_resetCCtx_internal(cctx, params, pledgedSrcSize,
+ dictContentSize,
ZSTDcrp_makeClean, zbuff) , "");
{ size_t const dictID = cdict ?
ZSTD_compress_insertDictionary(
@@ -3350,7 +4434,7 @@ static size_t ZSTD_compressBegin_internal(ZSTD_CCtx* cctx,
FORWARD_IF_ERROR(dictID, "ZSTD_compress_insertDictionary failed");
assert(dictID <= UINT_MAX);
cctx->dictID = (U32)dictID;
- cctx->dictContentSize = cdict ? cdict->dictContentSize : dictSize;
+ cctx->dictContentSize = dictContentSize;
}
return 0;
}
@@ -3485,15 +4569,14 @@ size_t ZSTD_compress_advanced (ZSTD_CCtx* cctx,
const void* dict,size_t dictSize,
ZSTD_parameters params)
{
- ZSTD_CCtx_params cctxParams;
DEBUGLOG(4, "ZSTD_compress_advanced");
FORWARD_IF_ERROR(ZSTD_checkCParams(params.cParams), "");
- ZSTD_CCtxParams_init_internal(&cctxParams, ¶ms, ZSTD_NO_CLEVEL);
+ ZSTD_CCtxParams_init_internal(&cctx->simpleApiParams, ¶ms, ZSTD_NO_CLEVEL);
return ZSTD_compress_advanced_internal(cctx,
dst, dstCapacity,
src, srcSize,
dict, dictSize,
- &cctxParams);
+ &cctx->simpleApiParams);
}
/* Internal */
@@ -3517,14 +4600,13 @@ size_t ZSTD_compress_usingDict(ZSTD_CCtx* cctx,
const void* dict, size_t dictSize,
int compressionLevel)
{
- ZSTD_CCtx_params cctxParams;
{
ZSTD_parameters const params = ZSTD_getParams_internal(compressionLevel, srcSize, dict ? dictSize : 0, ZSTD_cpm_noAttachDict);
assert(params.fParams.contentSizeFlag == 1);
- ZSTD_CCtxParams_init_internal(&cctxParams, ¶ms, (compressionLevel == 0) ? ZSTD_CLEVEL_DEFAULT: compressionLevel);
+ ZSTD_CCtxParams_init_internal(&cctx->simpleApiParams, ¶ms, (compressionLevel == 0) ? ZSTD_CLEVEL_DEFAULT: compressionLevel);
}
DEBUGLOG(4, "ZSTD_compress_usingDict (srcSize=%u)", (unsigned)srcSize);
- return ZSTD_compress_advanced_internal(cctx, dst, dstCapacity, src, srcSize, dict, dictSize, &cctxParams);
+ return ZSTD_compress_advanced_internal(cctx, dst, dstCapacity, src, srcSize, dict, dictSize, &cctx->simpleApiParams);
}
size_t ZSTD_compressCCtx(ZSTD_CCtx* cctx,
@@ -3561,7 +4643,10 @@ size_t ZSTD_estimateCDictSize_advanced(
DEBUGLOG(5, "sizeof(ZSTD_CDict) : %u", (unsigned)sizeof(ZSTD_CDict));
return ZSTD_cwksp_alloc_size(sizeof(ZSTD_CDict))
+ ZSTD_cwksp_alloc_size(HUF_WORKSPACE_SIZE)
- + ZSTD_sizeof_matchState(&cParams, /* forCCtx */ 0)
+ /* enableDedicatedDictSearch == 1 ensures that CDict estimation will not be too small
+ * in case we are using DDS with row-hash. */
+ + ZSTD_sizeof_matchState(&cParams, ZSTD_resolveRowMatchFinderMode(ZSTD_ps_auto, &cParams),
+ /* enableDedicatedDictSearch */ 1, /* forCCtx */ 0)
+ (dictLoadMethod == ZSTD_dlm_byRef ? 0
: ZSTD_cwksp_alloc_size(ZSTD_cwksp_align(dictSize, sizeof(void *))));
}
@@ -3592,9 +4677,6 @@ static size_t ZSTD_initCDict_internal(
assert(!ZSTD_checkCParams(params.cParams));
cdict->matchState.cParams = params.cParams;
cdict->matchState.dedicatedDictSearch = params.enableDedicatedDictSearch;
- if (cdict->matchState.dedicatedDictSearch && dictSize > ZSTD_CHUNKSIZE_MAX) {
- cdict->matchState.dedicatedDictSearch = 0;
- }
if ((dictLoadMethod == ZSTD_dlm_byRef) || (!dictBuffer) || (!dictSize)) {
cdict->dictContent = dictBuffer;
} else {
@@ -3615,6 +4697,7 @@ static size_t ZSTD_initCDict_internal(
&cdict->matchState,
&cdict->workspace,
¶ms.cParams,
+ params.useRowMatchFinder,
ZSTDcrp_makeClean,
ZSTDirp_reset,
ZSTD_resetTarget_CDict), "");
@@ -3638,14 +4721,17 @@ static size_t ZSTD_initCDict_internal(
static ZSTD_CDict* ZSTD_createCDict_advanced_internal(size_t dictSize,
ZSTD_dictLoadMethod_e dictLoadMethod,
- ZSTD_compressionParameters cParams, ZSTD_customMem customMem)
+ ZSTD_compressionParameters cParams,
+ ZSTD_paramSwitch_e useRowMatchFinder,
+ U32 enableDedicatedDictSearch,
+ ZSTD_customMem customMem)
{
if ((!customMem.customAlloc) ^ (!customMem.customFree)) return NULL;
{ size_t const workspaceSize =
ZSTD_cwksp_alloc_size(sizeof(ZSTD_CDict)) +
ZSTD_cwksp_alloc_size(HUF_WORKSPACE_SIZE) +
- ZSTD_sizeof_matchState(&cParams, /* forCCtx */ 0) +
+ ZSTD_sizeof_matchState(&cParams, useRowMatchFinder, enableDedicatedDictSearch, /* forCCtx */ 0) +
(dictLoadMethod == ZSTD_dlm_byRef ? 0
: ZSTD_cwksp_alloc_size(ZSTD_cwksp_align(dictSize, sizeof(void*))));
void* const workspace = ZSTD_customMalloc(workspaceSize, customMem);
@@ -3664,7 +4750,7 @@ static ZSTD_CDict* ZSTD_createCDict_advanced_internal(size_t dictSize,
ZSTD_cwksp_move(&cdict->workspace, &ws);
cdict->customMem = customMem;
cdict->compressionLevel = ZSTD_NO_CLEVEL; /* signals advanced API usage */
-
+ cdict->useRowMatchFinder = useRowMatchFinder;
return cdict;
}
}
@@ -3686,7 +4772,7 @@ ZSTD_CDict* ZSTD_createCDict_advanced(const void* dictBuffer, size_t dictSize,
&cctxParams, customMem);
}
-ZSTDLIB_API ZSTD_CDict* ZSTD_createCDict_advanced2(
+ZSTD_CDict* ZSTD_createCDict_advanced2(
const void* dict, size_t dictSize,
ZSTD_dictLoadMethod_e dictLoadMethod,
ZSTD_dictContentType_e dictContentType,
@@ -3716,10 +4802,13 @@ ZSTDLIB_API ZSTD_CDict* ZSTD_createCDict_advanced2(
&cctxParams, ZSTD_CONTENTSIZE_UNKNOWN, dictSize, ZSTD_cpm_createCDict);
}
+ DEBUGLOG(3, "ZSTD_createCDict_advanced2: DDS: %u", cctxParams.enableDedicatedDictSearch);
cctxParams.cParams = cParams;
+ cctxParams.useRowMatchFinder = ZSTD_resolveRowMatchFinderMode(cctxParams.useRowMatchFinder, &cParams);
cdict = ZSTD_createCDict_advanced_internal(dictSize,
dictLoadMethod, cctxParams.cParams,
+ cctxParams.useRowMatchFinder, cctxParams.enableDedicatedDictSearch,
customMem);
if (ZSTD_isError( ZSTD_initCDict_internal(cdict,
@@ -3788,7 +4877,9 @@ const ZSTD_CDict* ZSTD_initStaticCDict(
ZSTD_dictContentType_e dictContentType,
ZSTD_compressionParameters cParams)
{
- size_t const matchStateSize = ZSTD_sizeof_matchState(&cParams, /* forCCtx */ 0);
+ ZSTD_paramSwitch_e const useRowMatchFinder = ZSTD_resolveRowMatchFinderMode(ZSTD_ps_auto, &cParams);
+ /* enableDedicatedDictSearch == 1 ensures matchstate is not too small in case this CDict will be used for DDS + row hash */
+ size_t const matchStateSize = ZSTD_sizeof_matchState(&cParams, useRowMatchFinder, /* enableDedicatedDictSearch */ 1, /* forCCtx */ 0);
size_t const neededSize = ZSTD_cwksp_alloc_size(sizeof(ZSTD_CDict))
+ (dictLoadMethod == ZSTD_dlm_byRef ? 0
: ZSTD_cwksp_alloc_size(ZSTD_cwksp_align(dictSize, sizeof(void*))))
@@ -3813,6 +4904,8 @@ const ZSTD_CDict* ZSTD_initStaticCDict(
ZSTD_CCtxParams_init(¶ms, 0);
params.cParams = cParams;
+ params.useRowMatchFinder = useRowMatchFinder;
+ cdict->useRowMatchFinder = useRowMatchFinder;
if (ZSTD_isError( ZSTD_initCDict_internal(cdict,
dict, dictSize,
@@ -3839,15 +4932,15 @@ unsigned ZSTD_getDictID_fromCDict(const ZSTD_CDict* cdict)
return cdict->dictID;
}
-
-/* ZSTD_compressBegin_usingCDict_advanced() :
- * cdict must be != NULL */
-size_t ZSTD_compressBegin_usingCDict_advanced(
+/* ZSTD_compressBegin_usingCDict_internal() :
+ * Implementation of various ZSTD_compressBegin_usingCDict* functions.
+ */
+static size_t ZSTD_compressBegin_usingCDict_internal(
ZSTD_CCtx* const cctx, const ZSTD_CDict* const cdict,
ZSTD_frameParameters const fParams, unsigned long long const pledgedSrcSize)
{
ZSTD_CCtx_params cctxParams;
- DEBUGLOG(4, "ZSTD_compressBegin_usingCDict_advanced");
+ DEBUGLOG(4, "ZSTD_compressBegin_usingCDict_internal");
RETURN_ERROR_IF(cdict==NULL, dictionary_wrong, "NULL pointer!");
/* Initialize the cctxParams from the cdict */
{
@@ -3879,23 +4972,46 @@ size_t ZSTD_compressBegin_usingCDict_advanced(
ZSTDb_not_buffered);
}
+
+/* ZSTD_compressBegin_usingCDict_advanced() :
+ * This function is DEPRECATED.
+ * cdict must be != NULL */
+size_t ZSTD_compressBegin_usingCDict_advanced(
+ ZSTD_CCtx* const cctx, const ZSTD_CDict* const cdict,
+ ZSTD_frameParameters const fParams, unsigned long long const pledgedSrcSize)
+{
+ return ZSTD_compressBegin_usingCDict_internal(cctx, cdict, fParams, pledgedSrcSize);
+}
+
/* ZSTD_compressBegin_usingCDict() :
- * pledgedSrcSize=0 means "unknown"
- * if pledgedSrcSize>0, it will enable contentSizeFlag */
+ * cdict must be != NULL */
size_t ZSTD_compressBegin_usingCDict(ZSTD_CCtx* cctx, const ZSTD_CDict* cdict)
{
ZSTD_frameParameters const fParams = { 0 /*content*/, 0 /*checksum*/, 0 /*noDictID*/ };
- DEBUGLOG(4, "ZSTD_compressBegin_usingCDict : dictIDFlag == %u", !fParams.noDictIDFlag);
- return ZSTD_compressBegin_usingCDict_advanced(cctx, cdict, fParams, ZSTD_CONTENTSIZE_UNKNOWN);
+ return ZSTD_compressBegin_usingCDict_internal(cctx, cdict, fParams, ZSTD_CONTENTSIZE_UNKNOWN);
}
+/*! ZSTD_compress_usingCDict_internal():
+ * Implementation of various ZSTD_compress_usingCDict* functions.
+ */
+static size_t ZSTD_compress_usingCDict_internal(ZSTD_CCtx* cctx,
+ void* dst, size_t dstCapacity,
+ const void* src, size_t srcSize,
+ const ZSTD_CDict* cdict, ZSTD_frameParameters fParams)
+{
+ FORWARD_IF_ERROR(ZSTD_compressBegin_usingCDict_internal(cctx, cdict, fParams, srcSize), ""); /* will check if cdict != NULL */
+ return ZSTD_compressEnd(cctx, dst, dstCapacity, src, srcSize);
+}
+
+/*! ZSTD_compress_usingCDict_advanced():
+ * This function is DEPRECATED.
+ */
size_t ZSTD_compress_usingCDict_advanced(ZSTD_CCtx* cctx,
void* dst, size_t dstCapacity,
const void* src, size_t srcSize,
const ZSTD_CDict* cdict, ZSTD_frameParameters fParams)
{
- FORWARD_IF_ERROR(ZSTD_compressBegin_usingCDict_advanced(cctx, cdict, fParams, srcSize), ""); /* will check if cdict != NULL */
- return ZSTD_compressEnd(cctx, dst, dstCapacity, src, srcSize);
+ return ZSTD_compress_usingCDict_internal(cctx, dst, dstCapacity, src, srcSize, cdict, fParams);
}
/*! ZSTD_compress_usingCDict() :
@@ -3909,7 +5025,7 @@ size_t ZSTD_compress_usingCDict(ZSTD_CCtx* cctx,
const ZSTD_CDict* cdict)
{
ZSTD_frameParameters const fParams = { 1 /*content*/, 0 /*checksum*/, 0 /*noDictID*/ };
- return ZSTD_compress_usingCDict_advanced(cctx, dst, dstCapacity, src, srcSize, cdict, fParams);
+ return ZSTD_compress_usingCDict_internal(cctx, dst, dstCapacity, src, srcSize, cdict, fParams);
}
@@ -4313,8 +5429,13 @@ static size_t ZSTD_CCtx_init_compressStream2(ZSTD_CCtx* cctx,
FORWARD_IF_ERROR( ZSTD_initLocalDict(cctx) , ""); /* Init the local dict if present. */
ZSTD_memset(&cctx->prefixDict, 0, sizeof(cctx->prefixDict)); /* single usage */
assert(prefixDict.dict==NULL || cctx->cdict==NULL); /* only one can be set */
- if (cctx->cdict)
- params.compressionLevel = cctx->cdict->compressionLevel; /* let cdict take priority in terms of compression level */
+ if (cctx->cdict && !cctx->localDict.cdict) {
+ /* Let the cdict's compression level take priority over the requested params.
+ * But do not take the cdict's compression level if the "cdict" is actually a localDict
+ * generated from ZSTD_initLocalDict().
+ */
+ params.compressionLevel = cctx->cdict->compressionLevel;
+ }
DEBUGLOG(4, "ZSTD_compressStream2 : transparent init stage");
if (endOp == ZSTD_e_end) cctx->pledgedSrcSizePlusOne = inSize + 1; /* auto-fix pledgedSrcSize */
{
@@ -4327,11 +5448,9 @@ static size_t ZSTD_CCtx_init_compressStream2(ZSTD_CCtx* cctx,
dictSize, mode);
}
- if (ZSTD_CParams_shouldEnableLdm(¶ms.cParams)) {
- /* Enable LDM by default for optimal parser and window size >= 128MB */
- DEBUGLOG(4, "LDM enabled by default (window size >= 128MB, strategy >= btopt)");
- params.ldmParams.enableLdm = 1;
- }
+ params.useBlockSplitter = ZSTD_resolveBlockSplitterMode(params.useBlockSplitter, ¶ms.cParams);
+ params.ldmParams.enableLdm = ZSTD_resolveEnableLdm(params.ldmParams.enableLdm, ¶ms.cParams);
+ params.useRowMatchFinder = ZSTD_resolveRowMatchFinderMode(params.useRowMatchFinder, ¶ms.cParams);
{ U64 const pledgedSrcSize = cctx->pledgedSrcSizePlusOne - 1;
assert(!ZSTD_isError(ZSTD_checkCParams(params.cParams)));
@@ -4436,39 +5555,39 @@ typedef struct {
size_t posInSrc; /* Number of bytes given by sequences provided so far */
} ZSTD_sequencePosition;
-/* Returns a ZSTD error code if sequence is not valid */
-static size_t ZSTD_validateSequence(U32 offCode, U32 matchLength,
- size_t posInSrc, U32 windowLog, size_t dictSize, U32 minMatch) {
- size_t offsetBound;
- U32 windowSize = 1 << windowLog;
- /* posInSrc represents the amount of data the the decoder would decode up to this point.
+/* ZSTD_validateSequence() :
+ * @offCode : is presumed to follow format required by ZSTD_storeSeq()
+ * @returns a ZSTD error code if sequence is not valid
+ */
+static size_t
+ZSTD_validateSequence(U32 offCode, U32 matchLength,
+ size_t posInSrc, U32 windowLog, size_t dictSize)
+{
+ U32 const windowSize = 1 << windowLog;
+ /* posInSrc represents the amount of data the decoder would decode up to this point.
* As long as the amount of data decoded is less than or equal to window size, offsets may be
* larger than the total length of output decoded in order to reference the dict, even larger than
* window size. After output surpasses windowSize, we're limited to windowSize offsets again.
*/
- offsetBound = posInSrc > windowSize ? (size_t)windowSize : posInSrc + (size_t)dictSize;
- RETURN_ERROR_IF(offCode > offsetBound + ZSTD_REP_MOVE, corruption_detected, "Offset too large!");
- RETURN_ERROR_IF(matchLength < minMatch, corruption_detected, "Matchlength too small");
+ size_t const offsetBound = posInSrc > windowSize ? (size_t)windowSize : posInSrc + (size_t)dictSize;
+ RETURN_ERROR_IF(offCode > STORE_OFFSET(offsetBound), corruption_detected, "Offset too large!");
+ RETURN_ERROR_IF(matchLength < MINMATCH, corruption_detected, "Matchlength too small");
return 0;
}
/* Returns an offset code, given a sequence's raw offset, the ongoing repcode array, and whether litLength == 0 */
-static U32 ZSTD_finalizeOffCode(U32 rawOffset, const U32 rep[ZSTD_REP_NUM], U32 ll0) {
- U32 offCode = rawOffset + ZSTD_REP_MOVE;
- U32 repCode = 0;
+static U32 ZSTD_finalizeOffCode(U32 rawOffset, const U32 rep[ZSTD_REP_NUM], U32 ll0)
+{
+ U32 offCode = STORE_OFFSET(rawOffset);
if (!ll0 && rawOffset == rep[0]) {
- repCode = 1;
+ offCode = STORE_REPCODE_1;
} else if (rawOffset == rep[1]) {
- repCode = 2 - ll0;
+ offCode = STORE_REPCODE(2 - ll0);
} else if (rawOffset == rep[2]) {
- repCode = 3 - ll0;
+ offCode = STORE_REPCODE(3 - ll0);
} else if (ll0 && rawOffset == rep[0] - 1) {
- repCode = 3;
- }
- if (repCode) {
- /* ZSTD_storeSeq expects a number in the range [0, 2] to represent a repcode */
- offCode = repCode - 1;
+ offCode = STORE_REPCODE_3;
}
return offCode;
}
@@ -4476,18 +5595,17 @@ static U32 ZSTD_finalizeOffCode(U32 rawOffset, const U32 rep[ZSTD_REP_NUM], U32
/* Returns 0 on success, and a ZSTD_error otherwise. This function scans through an array of
* ZSTD_Sequence, storing the sequences it finds, until it reaches a block delimiter.
*/
-static size_t ZSTD_copySequencesToSeqStoreExplicitBlockDelim(ZSTD_CCtx* cctx, ZSTD_sequencePosition* seqPos,
- const ZSTD_Sequence* const inSeqs, size_t inSeqsSize,
- const void* src, size_t blockSize) {
+static size_t
+ZSTD_copySequencesToSeqStoreExplicitBlockDelim(ZSTD_CCtx* cctx,
+ ZSTD_sequencePosition* seqPos,
+ const ZSTD_Sequence* const inSeqs, size_t inSeqsSize,
+ const void* src, size_t blockSize)
+{
U32 idx = seqPos->idx;
BYTE const* ip = (BYTE const*)(src);
const BYTE* const iend = ip + blockSize;
repcodes_t updatedRepcodes;
U32 dictSize;
- U32 litLength;
- U32 matchLength;
- U32 ll0;
- U32 offCode;
if (cctx->cdict) {
dictSize = (U32)cctx->cdict->dictContentSize;
@@ -4498,23 +5616,22 @@ static size_t ZSTD_copySequencesToSeqStoreExplicitBlockDelim(ZSTD_CCtx* cctx, ZS
}
ZSTD_memcpy(updatedRepcodes.rep, cctx->blockState.prevCBlock->rep, sizeof(repcodes_t));
for (; (inSeqs[idx].matchLength != 0 || inSeqs[idx].offset != 0) && idx < inSeqsSize; ++idx) {
- litLength = inSeqs[idx].litLength;
- matchLength = inSeqs[idx].matchLength;
- ll0 = litLength == 0;
- offCode = ZSTD_finalizeOffCode(inSeqs[idx].offset, updatedRepcodes.rep, ll0);
- updatedRepcodes = ZSTD_updateRep(updatedRepcodes.rep, offCode, ll0);
+ U32 const litLength = inSeqs[idx].litLength;
+ U32 const ll0 = (litLength == 0);
+ U32 const matchLength = inSeqs[idx].matchLength;
+ U32 const offCode = ZSTD_finalizeOffCode(inSeqs[idx].offset, updatedRepcodes.rep, ll0);
+ ZSTD_updateRep(updatedRepcodes.rep, offCode, ll0);
DEBUGLOG(6, "Storing sequence: (of: %u, ml: %u, ll: %u)", offCode, matchLength, litLength);
if (cctx->appliedParams.validateSequences) {
seqPos->posInSrc += litLength + matchLength;
FORWARD_IF_ERROR(ZSTD_validateSequence(offCode, matchLength, seqPos->posInSrc,
- cctx->appliedParams.cParams.windowLog, dictSize,
- cctx->appliedParams.cParams.minMatch),
+ cctx->appliedParams.cParams.windowLog, dictSize),
"Sequence validation failed");
}
RETURN_ERROR_IF(idx - seqPos->idx > cctx->seqStore.maxNbSeq, memory_allocation,
"Not enough memory allocated. Try adjusting ZSTD_c_minMatch.");
- ZSTD_storeSeq(&cctx->seqStore, litLength, ip, iend, offCode, matchLength - MINMATCH);
+ ZSTD_storeSeq(&cctx->seqStore, litLength, ip, iend, offCode, matchLength);
ip += matchLength + litLength;
}
ZSTD_memcpy(cctx->blockState.nextCBlock->rep, updatedRepcodes.rep, sizeof(repcodes_t));
@@ -4541,9 +5658,11 @@ static size_t ZSTD_copySequencesToSeqStoreExplicitBlockDelim(ZSTD_CCtx* cctx, ZS
* avoid splitting a match, or to avoid splitting a match such that it would produce a match
* smaller than MINMATCH. In this case, we return the number of bytes that we didn't read from this block.
*/
-static size_t ZSTD_copySequencesToSeqStoreNoBlockDelim(ZSTD_CCtx* cctx, ZSTD_sequencePosition* seqPos,
- const ZSTD_Sequence* const inSeqs, size_t inSeqsSize,
- const void* src, size_t blockSize) {
+static size_t
+ZSTD_copySequencesToSeqStoreNoBlockDelim(ZSTD_CCtx* cctx, ZSTD_sequencePosition* seqPos,
+ const ZSTD_Sequence* const inSeqs, size_t inSeqsSize,
+ const void* src, size_t blockSize)
+{
U32 idx = seqPos->idx;
U32 startPosInSequence = seqPos->posInSequence;
U32 endPosInSequence = seqPos->posInSequence + (U32)blockSize;
@@ -4553,10 +5672,6 @@ static size_t ZSTD_copySequencesToSeqStoreNoBlockDelim(ZSTD_CCtx* cctx, ZSTD_seq
repcodes_t updatedRepcodes;
U32 bytesAdjustment = 0;
U32 finalMatchSplit = 0;
- U32 litLength;
- U32 matchLength;
- U32 rawOffset;
- U32 offCode;
if (cctx->cdict) {
dictSize = cctx->cdict->dictContentSize;
@@ -4570,9 +5685,10 @@ static size_t ZSTD_copySequencesToSeqStoreNoBlockDelim(ZSTD_CCtx* cctx, ZSTD_seq
ZSTD_memcpy(updatedRepcodes.rep, cctx->blockState.prevCBlock->rep, sizeof(repcodes_t));
while (endPosInSequence && idx < inSeqsSize && !finalMatchSplit) {
const ZSTD_Sequence currSeq = inSeqs[idx];
- litLength = currSeq.litLength;
- matchLength = currSeq.matchLength;
- rawOffset = currSeq.offset;
+ U32 litLength = currSeq.litLength;
+ U32 matchLength = currSeq.matchLength;
+ U32 const rawOffset = currSeq.offset;
+ U32 offCode;
/* Modify the sequence depending on where endPosInSequence lies */
if (endPosInSequence >= currSeq.litLength + currSeq.matchLength) {
@@ -4625,22 +5741,21 @@ static size_t ZSTD_copySequencesToSeqStoreNoBlockDelim(ZSTD_CCtx* cctx, ZSTD_seq
}
}
/* Check if this offset can be represented with a repcode */
- { U32 ll0 = (litLength == 0);
+ { U32 const ll0 = (litLength == 0);
offCode = ZSTD_finalizeOffCode(rawOffset, updatedRepcodes.rep, ll0);
- updatedRepcodes = ZSTD_updateRep(updatedRepcodes.rep, offCode, ll0);
+ ZSTD_updateRep(updatedRepcodes.rep, offCode, ll0);
}
if (cctx->appliedParams.validateSequences) {
seqPos->posInSrc += litLength + matchLength;
FORWARD_IF_ERROR(ZSTD_validateSequence(offCode, matchLength, seqPos->posInSrc,
- cctx->appliedParams.cParams.windowLog, dictSize,
- cctx->appliedParams.cParams.minMatch),
+ cctx->appliedParams.cParams.windowLog, dictSize),
"Sequence validation failed");
}
DEBUGLOG(6, "Storing sequence: (of: %u, ml: %u, ll: %u)", offCode, matchLength, litLength);
RETURN_ERROR_IF(idx - seqPos->idx > cctx->seqStore.maxNbSeq, memory_allocation,
"Not enough memory allocated. Try adjusting ZSTD_c_minMatch.");
- ZSTD_storeSeq(&cctx->seqStore, litLength, ip, iend, offCode, matchLength - MINMATCH);
+ ZSTD_storeSeq(&cctx->seqStore, litLength, ip, iend, offCode, matchLength);
ip += matchLength + litLength;
}
DEBUGLOG(5, "Ending seq: idx: %u (of: %u ml: %u ll: %u)", idx, inSeqs[idx].offset, inSeqs[idx].matchLength, inSeqs[idx].litLength);
@@ -4665,7 +5780,8 @@ static size_t ZSTD_copySequencesToSeqStoreNoBlockDelim(ZSTD_CCtx* cctx, ZSTD_seq
typedef size_t (*ZSTD_sequenceCopier) (ZSTD_CCtx* cctx, ZSTD_sequencePosition* seqPos,
const ZSTD_Sequence* const inSeqs, size_t inSeqsSize,
const void* src, size_t blockSize);
-static ZSTD_sequenceCopier ZSTD_selectSequenceCopier(ZSTD_sequenceFormat_e mode) {
+static ZSTD_sequenceCopier ZSTD_selectSequenceCopier(ZSTD_sequenceFormat_e mode)
+{
ZSTD_sequenceCopier sequenceCopier = NULL;
assert(ZSTD_cParam_withinBounds(ZSTD_c_blockDelimiters, mode));
if (mode == ZSTD_sf_explicitBlockDelimiters) {
@@ -4679,12 +5795,15 @@ static ZSTD_sequenceCopier ZSTD_selectSequenceCopier(ZSTD_sequenceFormat_e mode)
/* Compress, block-by-block, all of the sequences given.
*
- * Returns the cumulative size of all compressed blocks (including their headers), otherwise a ZSTD error.
+ * Returns the cumulative size of all compressed blocks (including their headers),
+ * otherwise a ZSTD error.
*/
-static size_t ZSTD_compressSequences_internal(ZSTD_CCtx* cctx,
- void* dst, size_t dstCapacity,
- const ZSTD_Sequence* inSeqs, size_t inSeqsSize,
- const void* src, size_t srcSize) {
+static size_t
+ZSTD_compressSequences_internal(ZSTD_CCtx* cctx,
+ void* dst, size_t dstCapacity,
+ const ZSTD_Sequence* inSeqs, size_t inSeqsSize,
+ const void* src, size_t srcSize)
+{
size_t cSize = 0;
U32 lastBlock;
size_t blockSize;
@@ -4694,7 +5813,7 @@ static size_t ZSTD_compressSequences_internal(ZSTD_CCtx* cctx,
BYTE const* ip = (BYTE const*)src;
BYTE* op = (BYTE*)dst;
- ZSTD_sequenceCopier sequenceCopier = ZSTD_selectSequenceCopier(cctx->appliedParams.blockDelimiters);
+ ZSTD_sequenceCopier const sequenceCopier = ZSTD_selectSequenceCopier(cctx->appliedParams.blockDelimiters);
DEBUGLOG(4, "ZSTD_compressSequences_internal srcSize: %zu, inSeqsSize: %zu", srcSize, inSeqsSize);
/* Special case: empty frame */
@@ -4732,7 +5851,7 @@ static size_t ZSTD_compressSequences_internal(ZSTD_CCtx* cctx,
continue;
}
- compressedSeqsSize = ZSTD_entropyCompressSequences(&cctx->seqStore,
+ compressedSeqsSize = ZSTD_entropyCompressSeqStore(&cctx->seqStore,
&cctx->blockState.prevCBlock->entropy, &cctx->blockState.nextCBlock->entropy,
&cctx->appliedParams,
op + ZSTD_blockHeaderSize /* Leave space for block header */, dstCapacity - ZSTD_blockHeaderSize,
@@ -4764,7 +5883,7 @@ static size_t ZSTD_compressSequences_internal(ZSTD_CCtx* cctx,
} else {
U32 cBlockHeader;
/* Error checking and repcodes update */
- ZSTD_confirmRepcodesAndEntropyTables(cctx);
+ ZSTD_blockState_confirmRepcodesAndEntropyTables(&cctx->blockState);
if (cctx->blockState.prevCBlock->entropy.fse.offcode_repeatMode == FSE_repeat_valid)
cctx->blockState.prevCBlock->entropy.fse.offcode_repeatMode = FSE_repeat_check;
@@ -4794,7 +5913,8 @@ static size_t ZSTD_compressSequences_internal(ZSTD_CCtx* cctx,
size_t ZSTD_compressSequences(ZSTD_CCtx* const cctx, void* dst, size_t dstCapacity,
const ZSTD_Sequence* inSeqs, size_t inSeqsSize,
- const void* src, size_t srcSize) {
+ const void* src, size_t srcSize)
+{
BYTE* op = (BYTE*)dst;
size_t cSize = 0;
size_t compressedBlocksSize = 0;
@@ -4861,117 +5981,11 @@ size_t ZSTD_endStream(ZSTD_CStream* zcs, ZSTD_outBuffer* output)
/*-===== Pre-defined compression levels =====-*/
+#include "clevels.h"
-#define ZSTD_MAX_CLEVEL 22
int ZSTD_maxCLevel(void) { return ZSTD_MAX_CLEVEL; }
int ZSTD_minCLevel(void) { return (int)-ZSTD_TARGETLENGTH_MAX; }
-
-static const ZSTD_compressionParameters ZSTD_defaultCParameters[4][ZSTD_MAX_CLEVEL+1] = {
-{ /* "default" - for any srcSize > 256 KB */
- /* W, C, H, S, L, TL, strat */
- { 19, 12, 13, 1, 6, 1, ZSTD_fast }, /* base for negative levels */
- { 19, 13, 14, 1, 7, 0, ZSTD_fast }, /* level 1 */
- { 20, 15, 16, 1, 6, 0, ZSTD_fast }, /* level 2 */
- { 21, 16, 17, 1, 5, 0, ZSTD_dfast }, /* level 3 */
- { 21, 18, 18, 1, 5, 0, ZSTD_dfast }, /* level 4 */
- { 21, 18, 19, 2, 5, 2, ZSTD_greedy }, /* level 5 */
- { 21, 19, 19, 3, 5, 4, ZSTD_greedy }, /* level 6 */
- { 21, 19, 19, 3, 5, 8, ZSTD_lazy }, /* level 7 */
- { 21, 19, 19, 3, 5, 16, ZSTD_lazy2 }, /* level 8 */
- { 21, 19, 20, 4, 5, 16, ZSTD_lazy2 }, /* level 9 */
- { 22, 20, 21, 4, 5, 16, ZSTD_lazy2 }, /* level 10 */
- { 22, 21, 22, 4, 5, 16, ZSTD_lazy2 }, /* level 11 */
- { 22, 21, 22, 5, 5, 16, ZSTD_lazy2 }, /* level 12 */
- { 22, 21, 22, 5, 5, 32, ZSTD_btlazy2 }, /* level 13 */
- { 22, 22, 23, 5, 5, 32, ZSTD_btlazy2 }, /* level 14 */
- { 22, 23, 23, 6, 5, 32, ZSTD_btlazy2 }, /* level 15 */
- { 22, 22, 22, 5, 5, 48, ZSTD_btopt }, /* level 16 */
- { 23, 23, 22, 5, 4, 64, ZSTD_btopt }, /* level 17 */
- { 23, 23, 22, 6, 3, 64, ZSTD_btultra }, /* level 18 */
- { 23, 24, 22, 7, 3,256, ZSTD_btultra2}, /* level 19 */
- { 25, 25, 23, 7, 3,256, ZSTD_btultra2}, /* level 20 */
- { 26, 26, 24, 7, 3,512, ZSTD_btultra2}, /* level 21 */
- { 27, 27, 25, 9, 3,999, ZSTD_btultra2}, /* level 22 */
-},
-{ /* for srcSize <= 256 KB */
- /* W, C, H, S, L, T, strat */
- { 18, 12, 13, 1, 5, 1, ZSTD_fast }, /* base for negative levels */
- { 18, 13, 14, 1, 6, 0, ZSTD_fast }, /* level 1 */
- { 18, 14, 14, 1, 5, 0, ZSTD_dfast }, /* level 2 */
- { 18, 16, 16, 1, 4, 0, ZSTD_dfast }, /* level 3 */
- { 18, 16, 17, 2, 5, 2, ZSTD_greedy }, /* level 4.*/
- { 18, 18, 18, 3, 5, 2, ZSTD_greedy }, /* level 5.*/
- { 18, 18, 19, 3, 5, 4, ZSTD_lazy }, /* level 6.*/
- { 18, 18, 19, 4, 4, 4, ZSTD_lazy }, /* level 7 */
- { 18, 18, 19, 4, 4, 8, ZSTD_lazy2 }, /* level 8 */
- { 18, 18, 19, 5, 4, 8, ZSTD_lazy2 }, /* level 9 */
- { 18, 18, 19, 6, 4, 8, ZSTD_lazy2 }, /* level 10 */
- { 18, 18, 19, 5, 4, 12, ZSTD_btlazy2 }, /* level 11.*/
- { 18, 19, 19, 7, 4, 12, ZSTD_btlazy2 }, /* level 12.*/
- { 18, 18, 19, 4, 4, 16, ZSTD_btopt }, /* level 13 */
- { 18, 18, 19, 4, 3, 32, ZSTD_btopt }, /* level 14.*/
- { 18, 18, 19, 6, 3,128, ZSTD_btopt }, /* level 15.*/
- { 18, 19, 19, 6, 3,128, ZSTD_btultra }, /* level 16.*/
- { 18, 19, 19, 8, 3,256, ZSTD_btultra }, /* level 17.*/
- { 18, 19, 19, 6, 3,128, ZSTD_btultra2}, /* level 18.*/
- { 18, 19, 19, 8, 3,256, ZSTD_btultra2}, /* level 19.*/
- { 18, 19, 19, 10, 3,512, ZSTD_btultra2}, /* level 20.*/
- { 18, 19, 19, 12, 3,512, ZSTD_btultra2}, /* level 21.*/
- { 18, 19, 19, 13, 3,999, ZSTD_btultra2}, /* level 22.*/
-},
-{ /* for srcSize <= 128 KB */
- /* W, C, H, S, L, T, strat */
- { 17, 12, 12, 1, 5, 1, ZSTD_fast }, /* base for negative levels */
- { 17, 12, 13, 1, 6, 0, ZSTD_fast }, /* level 1 */
- { 17, 13, 15, 1, 5, 0, ZSTD_fast }, /* level 2 */
- { 17, 15, 16, 2, 5, 0, ZSTD_dfast }, /* level 3 */
- { 17, 17, 17, 2, 4, 0, ZSTD_dfast }, /* level 4 */
- { 17, 16, 17, 3, 4, 2, ZSTD_greedy }, /* level 5 */
- { 17, 17, 17, 3, 4, 4, ZSTD_lazy }, /* level 6 */
- { 17, 17, 17, 3, 4, 8, ZSTD_lazy2 }, /* level 7 */
- { 17, 17, 17, 4, 4, 8, ZSTD_lazy2 }, /* level 8 */
- { 17, 17, 17, 5, 4, 8, ZSTD_lazy2 }, /* level 9 */
- { 17, 17, 17, 6, 4, 8, ZSTD_lazy2 }, /* level 10 */
- { 17, 17, 17, 5, 4, 8, ZSTD_btlazy2 }, /* level 11 */
- { 17, 18, 17, 7, 4, 12, ZSTD_btlazy2 }, /* level 12 */
- { 17, 18, 17, 3, 4, 12, ZSTD_btopt }, /* level 13.*/
- { 17, 18, 17, 4, 3, 32, ZSTD_btopt }, /* level 14.*/
- { 17, 18, 17, 6, 3,256, ZSTD_btopt }, /* level 15.*/
- { 17, 18, 17, 6, 3,128, ZSTD_btultra }, /* level 16.*/
- { 17, 18, 17, 8, 3,256, ZSTD_btultra }, /* level 17.*/
- { 17, 18, 17, 10, 3,512, ZSTD_btultra }, /* level 18.*/
- { 17, 18, 17, 5, 3,256, ZSTD_btultra2}, /* level 19.*/
- { 17, 18, 17, 7, 3,512, ZSTD_btultra2}, /* level 20.*/
- { 17, 18, 17, 9, 3,512, ZSTD_btultra2}, /* level 21.*/
- { 17, 18, 17, 11, 3,999, ZSTD_btultra2}, /* level 22.*/
-},
-{ /* for srcSize <= 16 KB */
- /* W, C, H, S, L, T, strat */
- { 14, 12, 13, 1, 5, 1, ZSTD_fast }, /* base for negative levels */
- { 14, 14, 15, 1, 5, 0, ZSTD_fast }, /* level 1 */
- { 14, 14, 15, 1, 4, 0, ZSTD_fast }, /* level 2 */
- { 14, 14, 15, 2, 4, 0, ZSTD_dfast }, /* level 3 */
- { 14, 14, 14, 4, 4, 2, ZSTD_greedy }, /* level 4 */
- { 14, 14, 14, 3, 4, 4, ZSTD_lazy }, /* level 5.*/
- { 14, 14, 14, 4, 4, 8, ZSTD_lazy2 }, /* level 6 */
- { 14, 14, 14, 6, 4, 8, ZSTD_lazy2 }, /* level 7 */
- { 14, 14, 14, 8, 4, 8, ZSTD_lazy2 }, /* level 8.*/
- { 14, 15, 14, 5, 4, 8, ZSTD_btlazy2 }, /* level 9.*/
- { 14, 15, 14, 9, 4, 8, ZSTD_btlazy2 }, /* level 10.*/
- { 14, 15, 14, 3, 4, 12, ZSTD_btopt }, /* level 11.*/
- { 14, 15, 14, 4, 3, 24, ZSTD_btopt }, /* level 12.*/
- { 14, 15, 14, 5, 3, 32, ZSTD_btultra }, /* level 13.*/
- { 14, 15, 15, 6, 3, 64, ZSTD_btultra }, /* level 14.*/
- { 14, 15, 15, 7, 3,256, ZSTD_btultra }, /* level 15.*/
- { 14, 15, 15, 5, 3, 48, ZSTD_btultra2}, /* level 16.*/
- { 14, 15, 15, 6, 3,128, ZSTD_btultra2}, /* level 17.*/
- { 14, 15, 15, 7, 3,256, ZSTD_btultra2}, /* level 18.*/
- { 14, 15, 15, 8, 3,256, ZSTD_btultra2}, /* level 19.*/
- { 14, 15, 15, 8, 3,512, ZSTD_btultra2}, /* level 20.*/
- { 14, 15, 15, 9, 3,512, ZSTD_btultra2}, /* level 21.*/
- { 14, 15, 15, 10, 3,999, ZSTD_btultra2}, /* level 22.*/
-},
-};
+int ZSTD_defaultCLevel(void) { return ZSTD_CLEVEL_DEFAULT; }
static ZSTD_compressionParameters ZSTD_dedicatedDictSearch_getCParams(int const compressionLevel, size_t const dictSize)
{
@@ -4999,7 +6013,7 @@ static int ZSTD_dedicatedDictSearch_isSupported(
{
return (cParams->strategy >= ZSTD_greedy)
&& (cParams->strategy <= ZSTD_lazy2)
- && (cParams->hashLog >= cParams->chainLog)
+ && (cParams->hashLog > cParams->chainLog)
&& (cParams->chainLog <= 24);
}
@@ -5018,6 +6032,9 @@ static void ZSTD_dedicatedDictSearch_revertCParams(
case ZSTD_lazy:
case ZSTD_lazy2:
cParams->hashLog -= ZSTD_LAZY_DDSS_BUCKET_LOG;
+ if (cParams->hashLog < ZSTD_HASHLOG_MIN) {
+ cParams->hashLog = ZSTD_HASHLOG_MIN;
+ }
break;
case ZSTD_btlazy2:
case ZSTD_btopt:
@@ -5066,6 +6083,7 @@ static ZSTD_compressionParameters ZSTD_getCParams_internal(int compressionLevel,
else row = compressionLevel;
{ ZSTD_compressionParameters cp = ZSTD_defaultCParameters[tableID][row];
+ DEBUGLOG(5, "ZSTD_getCParams_internal selected tableID: %u row: %u strat: %u", tableID, row, (U32)cp.strategy);
/* acceleration factor */
if (compressionLevel < 0) {
int const clampedCompressionLevel = MAX(ZSTD_minCLevel(), compressionLevel);
diff --git a/lib/zstd/compress/zstd_compress_internal.h b/lib/zstd/compress/zstd_compress_internal.h
index 685d2f9..71697a1 100644
--- a/lib/zstd/compress/zstd_compress_internal.h
+++ b/lib/zstd/compress/zstd_compress_internal.h
@@ -57,7 +57,7 @@ typedef struct {
} ZSTD_localDict;
typedef struct {
- HUF_CElt CTable[HUF_CTABLE_SIZE_U32(255)];
+ HUF_CElt CTable[HUF_CTABLE_SIZE_ST(255)];
HUF_repeat repeatMode;
} ZSTD_hufCTables_t;
@@ -75,8 +75,55 @@ typedef struct {
ZSTD_fseCTables_t fse;
} ZSTD_entropyCTables_t;
+/* *********************************************
+* Entropy buffer statistics structs and funcs *
+***********************************************/
+/* ZSTD_hufCTablesMetadata_t :
+ * Stores Literals Block Type for a super-block in hType, and
+ * huffman tree description in hufDesBuffer.
+ * hufDesSize refers to the size of huffman tree description in bytes.
+ * This metadata is populated in ZSTD_buildBlockEntropyStats_literals() */
typedef struct {
- U32 off; /* Offset code (offset + ZSTD_REP_MOVE) for the match */
+ symbolEncodingType_e hType;
+ BYTE hufDesBuffer[ZSTD_MAX_HUF_HEADER_SIZE];
+ size_t hufDesSize;
+} ZSTD_hufCTablesMetadata_t;
+
+/* ZSTD_fseCTablesMetadata_t :
+ * Stores symbol compression modes for a super-block in {ll, ol, ml}Type, and
+ * fse tables in fseTablesBuffer.
+ * fseTablesSize refers to the size of fse tables in bytes.
+ * This metadata is populated in ZSTD_buildBlockEntropyStats_sequences() */
+typedef struct {
+ symbolEncodingType_e llType;
+ symbolEncodingType_e ofType;
+ symbolEncodingType_e mlType;
+ BYTE fseTablesBuffer[ZSTD_MAX_FSE_HEADERS_SIZE];
+ size_t fseTablesSize;
+ size_t lastCountSize; /* This is to account for bug in 1.3.4. More detail in ZSTD_entropyCompressSeqStore_internal() */
+} ZSTD_fseCTablesMetadata_t;
+
+typedef struct {
+ ZSTD_hufCTablesMetadata_t hufMetadata;
+ ZSTD_fseCTablesMetadata_t fseMetadata;
+} ZSTD_entropyCTablesMetadata_t;
+
+/* ZSTD_buildBlockEntropyStats() :
+ * Builds entropy for the block.
+ * @return : 0 on success or error code */
+size_t ZSTD_buildBlockEntropyStats(seqStore_t* seqStorePtr,
+ const ZSTD_entropyCTables_t* prevEntropy,
+ ZSTD_entropyCTables_t* nextEntropy,
+ const ZSTD_CCtx_params* cctxParams,
+ ZSTD_entropyCTablesMetadata_t* entropyMetadata,
+ void* workspace, size_t wkspSize);
+
+/* *******************************
+* Compression internals structs *
+*********************************/
+
+typedef struct {
+ U32 off; /* Offset sumtype code for the match, using ZSTD_storeSeq() format */
U32 len; /* Raw length of match */
} ZSTD_match_t;
@@ -126,7 +173,7 @@ typedef struct {
U32 offCodeSumBasePrice; /* to compare to log2(offreq) */
ZSTD_OptPrice_e priceType; /* prices can be determined dynamically, or follow a pre-defined cost structure */
const ZSTD_entropyCTables_t* symbolCosts; /* pre-calculated dictionary statistics */
- ZSTD_literalCompressionMode_e literalCompressionMode;
+ ZSTD_paramSwitch_e literalCompressionMode;
} optState_t;
typedef struct {
@@ -135,14 +182,23 @@ typedef struct {
} ZSTD_compressedBlockState_t;
typedef struct {
- BYTE const* nextSrc; /* next block here to continue on current prefix */
- BYTE const* base; /* All regular indexes relative to this position */
- BYTE const* dictBase; /* extDict indexes relative to this position */
- U32 dictLimit; /* below that point, need extDict */
- U32 lowLimit; /* below that point, no more valid data */
+ BYTE const* nextSrc; /* next block here to continue on current prefix */
+ BYTE const* base; /* All regular indexes relative to this position */
+ BYTE const* dictBase; /* extDict indexes relative to this position */
+ U32 dictLimit; /* below that point, need extDict */
+ U32 lowLimit; /* below that point, no more valid data */
+ U32 nbOverflowCorrections; /* Number of times overflow correction has run since
+ * ZSTD_window_init(). Useful for debugging coredumps
+ * and for ZSTD_WINDOW_OVERFLOW_CORRECT_FREQUENTLY.
+ */
} ZSTD_window_t;
+#define ZSTD_WINDOW_START_INDEX 2
+
typedef struct ZSTD_matchState_t ZSTD_matchState_t;
+
+#define ZSTD_ROW_HASH_CACHE_SIZE 8 /* Size of prefetching hash cache for row-based matchfinder */
+
struct ZSTD_matchState_t {
ZSTD_window_t window; /* State for window round buffer management */
U32 loadedDictEnd; /* index of end of dictionary, within context's referential.
@@ -154,9 +210,17 @@ struct ZSTD_matchState_t {
*/
U32 nextToUpdate; /* index from which to continue table update */
U32 hashLog3; /* dispatch table for matches of len==3 : larger == faster, more memory */
+
+ U32 rowHashLog; /* For row-based matchfinder: Hashlog based on nb of rows in the hashTable.*/
+ U16* tagTable; /* For row-based matchFinder: A row-based table containing the hashes and head index. */
+ U32 hashCache[ZSTD_ROW_HASH_CACHE_SIZE]; /* For row-based matchFinder: a cache of hashes to improve speed */
+
U32* hashTable;
U32* hashTable3;
U32* chainTable;
+
+ U32 forceNonContiguous; /* Non-zero if we should force non-contiguous load for the next window update. */
+
int dedicatedDictSearch; /* Indicates whether this matchState is using the
* dedicated dictionary search structure.
*/
@@ -196,7 +260,7 @@ typedef struct {
} ldmState_t;
typedef struct {
- U32 enableLdm; /* 1 if enable long distance matching */
+ ZSTD_paramSwitch_e enableLdm; /* ZSTD_ps_enable to enable LDM. ZSTD_ps_auto by default */
U32 hashLog; /* Log size of hashTable */
U32 bucketSizeLog; /* Log bucket size for collision resolution, at most 8 */
U32 minMatchLength; /* Minimum match length */
@@ -227,7 +291,7 @@ struct ZSTD_CCtx_params_s {
* There is no guarantee that hint is close to actual source size */
ZSTD_dictAttachPref_e attachDictPref;
- ZSTD_literalCompressionMode_e literalCompressionMode;
+ ZSTD_paramSwitch_e literalCompressionMode;
/* Multithreading: used to pass parameters to mtctx */
int nbWorkers;
@@ -249,6 +313,15 @@ struct ZSTD_CCtx_params_s {
ZSTD_sequenceFormat_e blockDelimiters;
int validateSequences;
+ /* Block splitting */
+ ZSTD_paramSwitch_e useBlockSplitter;
+
+ /* Param for deciding whether to use row-based matchfinder */
+ ZSTD_paramSwitch_e useRowMatchFinder;
+
+ /* Always load a dictionary in ext-dict mode (not prefix mode)? */
+ int deterministicRefPrefix;
+
/* Internal use, for createCCtxParams() and freeCCtxParams() only */
ZSTD_customMem customMem;
}; /* typedef'd to ZSTD_CCtx_params within "zstd.h" */
@@ -266,12 +339,29 @@ typedef enum {
ZSTDb_buffered
} ZSTD_buffered_policy_e;
+/*
+ * Struct that contains all elements of block splitter that should be allocated
+ * in a wksp.
+ */
+#define ZSTD_MAX_NB_BLOCK_SPLITS 196
+typedef struct {
+ seqStore_t fullSeqStoreChunk;
+ seqStore_t firstHalfSeqStore;
+ seqStore_t secondHalfSeqStore;
+ seqStore_t currSeqStore;
+ seqStore_t nextSeqStore;
+
+ U32 partitions[ZSTD_MAX_NB_BLOCK_SPLITS];
+ ZSTD_entropyCTablesMetadata_t entropyMetadata;
+} ZSTD_blockSplitCtx;
+
struct ZSTD_CCtx_s {
ZSTD_compressionStage_e stage;
int cParamsChanged; /* == 1 if cParams(except wlog) or compression level are changed in requestedParams. Triggers transmission of new params to ZSTDMT (if available) then reset to 0. */
int bmi2; /* == 1 if the CPU supports BMI2 and 0 otherwise. CPU support is determined dynamically once per context lifetime. */
ZSTD_CCtx_params requestedParams;
ZSTD_CCtx_params appliedParams;
+ ZSTD_CCtx_params simpleApiParams; /* Param storage used by the simple API - not sticky. Must only be used in top-level simple API functions for storage. */
U32 dictID;
size_t dictContentSize;
@@ -296,7 +386,7 @@ struct ZSTD_CCtx_s {
ZSTD_blockState_t blockState;
U32* entropyWorkspace; /* entropy workspace of ENTROPY_WORKSPACE_SIZE bytes */
- /* Wether we are streaming or not */
+ /* Whether we are streaming or not */
ZSTD_buffered_policy_e bufferedPolicy;
/* streaming */
@@ -324,6 +414,9 @@ struct ZSTD_CCtx_s {
/* Multi-threading */
/* Tracing */
+
+ /* Workspace for block splitter */
+ ZSTD_blockSplitCtx blockSplitCtx;
};
typedef enum { ZSTD_dtlm_fast, ZSTD_dtlm_full } ZSTD_dictTableLoadMethod_e;
@@ -358,7 +451,7 @@ typedef enum {
typedef size_t (*ZSTD_blockCompressor) (
ZSTD_matchState_t* bs, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM],
void const* src, size_t srcSize);
-ZSTD_blockCompressor ZSTD_selectBlockCompressor(ZSTD_strategy strat, ZSTD_dictMode_e dictMode);
+ZSTD_blockCompressor ZSTD_selectBlockCompressor(ZSTD_strategy strat, ZSTD_paramSwitch_e rowMatchfinderMode, ZSTD_dictMode_e dictMode);
MEM_STATIC U32 ZSTD_LLcode(U32 litLength)
@@ -392,31 +485,6 @@ MEM_STATIC U32 ZSTD_MLcode(U32 mlBase)
return (mlBase > 127) ? ZSTD_highbit32(mlBase) + ML_deltaCode : ML_Code[mlBase];
}
-typedef struct repcodes_s {
- U32 rep[3];
-} repcodes_t;
-
-MEM_STATIC repcodes_t ZSTD_updateRep(U32 const rep[3], U32 const offset, U32 const ll0)
-{
- repcodes_t newReps;
- if (offset >= ZSTD_REP_NUM) { /* full offset */
- newReps.rep[2] = rep[1];
- newReps.rep[1] = rep[0];
- newReps.rep[0] = offset - ZSTD_REP_MOVE;
- } else { /* repcode */
- U32 const repCode = offset + ll0;
- if (repCode > 0) { /* note : if repCode==0, no change */
- U32 const currentOffset = (repCode==ZSTD_REP_NUM) ? (rep[0] - 1) : rep[repCode];
- newReps.rep[2] = (repCode >= 2) ? rep[1] : rep[2];
- newReps.rep[1] = rep[0];
- newReps.rep[0] = currentOffset;
- } else { /* repCode == 0 */
- ZSTD_memcpy(&newReps, rep, sizeof(newReps));
- }
- }
- return newReps;
-}
-
/* ZSTD_cParam_withinBounds:
* @return 1 if value is within cParam bounds,
* 0 otherwise */
@@ -465,17 +533,17 @@ MEM_STATIC size_t ZSTD_minGain(size_t srcSize, ZSTD_strategy strat)
return (srcSize >> minlog) + 2;
}
-MEM_STATIC int ZSTD_disableLiteralsCompression(const ZSTD_CCtx_params* cctxParams)
+MEM_STATIC int ZSTD_literalsCompressionIsDisabled(const ZSTD_CCtx_params* cctxParams)
{
switch (cctxParams->literalCompressionMode) {
- case ZSTD_lcm_huffman:
+ case ZSTD_ps_enable:
return 0;
- case ZSTD_lcm_uncompressed:
+ case ZSTD_ps_disable:
return 1;
default:
assert(0 /* impossible: pre-validated */);
ZSTD_FALLTHROUGH;
- case ZSTD_lcm_auto:
+ case ZSTD_ps_auto:
return (cctxParams->cParams.strategy == ZSTD_fast) && (cctxParams->cParams.targetLength > 0);
}
}
@@ -485,7 +553,9 @@ MEM_STATIC int ZSTD_disableLiteralsCompression(const ZSTD_CCtx_params* cctxParam
* Only called when the sequence ends past ilimit_w, so it only needs to be optimized for single
* large copies.
*/
-static void ZSTD_safecopyLiterals(BYTE* op, BYTE const* ip, BYTE const* const iend, BYTE const* ilimit_w) {
+static void
+ZSTD_safecopyLiterals(BYTE* op, BYTE const* ip, BYTE const* const iend, BYTE const* ilimit_w)
+{
assert(iend > ilimit_w);
if (ip <= ilimit_w) {
ZSTD_wildcopy(op, ip, ilimit_w - ip, ZSTD_no_overlap);
@@ -495,14 +565,30 @@ static void ZSTD_safecopyLiterals(BYTE* op, BYTE const* ip, BYTE const* const ie
while (ip < iend) *op++ = *ip++;
}
+#define ZSTD_REP_MOVE (ZSTD_REP_NUM-1)
+#define STORE_REPCODE_1 STORE_REPCODE(1)
+#define STORE_REPCODE_2 STORE_REPCODE(2)
+#define STORE_REPCODE_3 STORE_REPCODE(3)
+#define STORE_REPCODE(r) (assert((r)>=1), assert((r)<=3), (r)-1)
+#define STORE_OFFSET(o) (assert((o)>0), o + ZSTD_REP_MOVE)
+#define STORED_IS_OFFSET(o) ((o) > ZSTD_REP_MOVE)
+#define STORED_IS_REPCODE(o) ((o) <= ZSTD_REP_MOVE)
+#define STORED_OFFSET(o) (assert(STORED_IS_OFFSET(o)), (o)-ZSTD_REP_MOVE)
+#define STORED_REPCODE(o) (assert(STORED_IS_REPCODE(o)), (o)+1) /* returns ID 1,2,3 */
+#define STORED_TO_OFFBASE(o) ((o)+1)
+#define OFFBASE_TO_STORED(o) ((o)-1)
+
/*! ZSTD_storeSeq() :
- * Store a sequence (litlen, litPtr, offCode and mlBase) into seqStore_t.
- * `offCode` : distance to match + ZSTD_REP_MOVE (values <= ZSTD_REP_MOVE are repCodes).
- * `mlBase` : matchLength - MINMATCH
+ * Store a sequence (litlen, litPtr, offCode and matchLength) into seqStore_t.
+ * @offBase_minus1 : Users should use employ macros STORE_REPCODE_X and STORE_OFFSET().
+ * @matchLength : must be >= MINMATCH
* Allowed to overread literals up to litLimit.
*/
-HINT_INLINE UNUSED_ATTR
-void ZSTD_storeSeq(seqStore_t* seqStorePtr, size_t litLength, const BYTE* literals, const BYTE* litLimit, U32 offCode, size_t mlBase)
+HINT_INLINE UNUSED_ATTR void
+ZSTD_storeSeq(seqStore_t* seqStorePtr,
+ size_t litLength, const BYTE* literals, const BYTE* litLimit,
+ U32 offBase_minus1,
+ size_t matchLength)
{
BYTE const* const litLimit_w = litLimit - WILDCOPY_OVERLENGTH;
BYTE const* const litEnd = literals + litLength;
@@ -511,7 +597,7 @@ void ZSTD_storeSeq(seqStore_t* seqStorePtr, size_t litLength, const BYTE* litera
if (g_start==NULL) g_start = (const BYTE*)literals; /* note : index only works for compression within a single segment */
{ U32 const pos = (U32)((const BYTE*)literals - g_start);
DEBUGLOG(6, "Cpos%7u :%3u literals, match%4u bytes at offCode%7u",
- pos, (U32)litLength, (U32)mlBase+MINMATCH, (U32)offCode);
+ pos, (U32)litLength, (U32)matchLength, (U32)offBase_minus1);
}
#endif
assert((size_t)(seqStorePtr->sequences - seqStorePtr->sequencesStart) < seqStorePtr->maxNbSeq);
@@ -535,26 +621,66 @@ void ZSTD_storeSeq(seqStore_t* seqStorePtr, size_t litLength, const BYTE* litera
/* literal Length */
if (litLength>0xFFFF) {
- assert(seqStorePtr->longLengthID == 0); /* there can only be a single long length */
- seqStorePtr->longLengthID = 1;
+ assert(seqStorePtr->longLengthType == ZSTD_llt_none); /* there can only be a single long length */
+ seqStorePtr->longLengthType = ZSTD_llt_literalLength;
seqStorePtr->longLengthPos = (U32)(seqStorePtr->sequences - seqStorePtr->sequencesStart);
}
seqStorePtr->sequences[0].litLength = (U16)litLength;
/* match offset */
- seqStorePtr->sequences[0].offset = offCode + 1;
+ seqStorePtr->sequences[0].offBase = STORED_TO_OFFBASE(offBase_minus1);
/* match Length */
- if (mlBase>0xFFFF) {
- assert(seqStorePtr->longLengthID == 0); /* there can only be a single long length */
- seqStorePtr->longLengthID = 2;
- seqStorePtr->longLengthPos = (U32)(seqStorePtr->sequences - seqStorePtr->sequencesStart);
+ assert(matchLength >= MINMATCH);
+ { size_t const mlBase = matchLength - MINMATCH;
+ if (mlBase>0xFFFF) {
+ assert(seqStorePtr->longLengthType == ZSTD_llt_none); /* there can only be a single long length */
+ seqStorePtr->longLengthType = ZSTD_llt_matchLength;
+ seqStorePtr->longLengthPos = (U32)(seqStorePtr->sequences - seqStorePtr->sequencesStart);
+ }
+ seqStorePtr->sequences[0].mlBase = (U16)mlBase;
}
- seqStorePtr->sequences[0].matchLength = (U16)mlBase;
seqStorePtr->sequences++;
}
+/* ZSTD_updateRep() :
+ * updates in-place @rep (array of repeat offsets)
+ * @offBase_minus1 : sum-type, with same numeric representation as ZSTD_storeSeq()
+ */
+MEM_STATIC void
+ZSTD_updateRep(U32 rep[ZSTD_REP_NUM], U32 const offBase_minus1, U32 const ll0)
+{
+ if (STORED_IS_OFFSET(offBase_minus1)) { /* full offset */
+ rep[2] = rep[1];
+ rep[1] = rep[0];
+ rep[0] = STORED_OFFSET(offBase_minus1);
+ } else { /* repcode */
+ U32 const repCode = STORED_REPCODE(offBase_minus1) - 1 + ll0;
+ if (repCode > 0) { /* note : if repCode==0, no change */
+ U32 const currentOffset = (repCode==ZSTD_REP_NUM) ? (rep[0] - 1) : rep[repCode];
+ rep[2] = (repCode >= 2) ? rep[1] : rep[2];
+ rep[1] = rep[0];
+ rep[0] = currentOffset;
+ } else { /* repCode == 0 */
+ /* nothing to do */
+ }
+ }
+}
+
+typedef struct repcodes_s {
+ U32 rep[3];
+} repcodes_t;
+
+MEM_STATIC repcodes_t
+ZSTD_newRep(U32 const rep[ZSTD_REP_NUM], U32 const offBase_minus1, U32 const ll0)
+{
+ repcodes_t newReps;
+ ZSTD_memcpy(&newReps, rep, sizeof(newReps));
+ ZSTD_updateRep(newReps.rep, offBase_minus1, ll0);
+ return newReps;
+}
+
/*-*************************************
* Match length counter
@@ -778,6 +904,13 @@ MEM_STATIC void ZSTD_window_clear(ZSTD_window_t* window)
window->dictLimit = end;
}
+MEM_STATIC U32 ZSTD_window_isEmpty(ZSTD_window_t const window)
+{
+ return window.dictLimit == ZSTD_WINDOW_START_INDEX &&
+ window.lowLimit == ZSTD_WINDOW_START_INDEX &&
+ (window.nextSrc - window.base) == ZSTD_WINDOW_START_INDEX;
+}
+
/*
* ZSTD_window_hasExtDict():
* Returns non-zero if the window has a non-empty extDict.
@@ -801,15 +934,71 @@ MEM_STATIC ZSTD_dictMode_e ZSTD_matchState_dictMode(const ZSTD_matchState_t *ms)
ZSTD_noDict;
}
+/* Defining this macro to non-zero tells zstd to run the overflow correction
+ * code much more frequently. This is very inefficient, and should only be
+ * used for tests and fuzzers.
+ */
+#ifndef ZSTD_WINDOW_OVERFLOW_CORRECT_FREQUENTLY
+# ifdef FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION
+# define ZSTD_WINDOW_OVERFLOW_CORRECT_FREQUENTLY 1
+# else
+# define ZSTD_WINDOW_OVERFLOW_CORRECT_FREQUENTLY 0
+# endif
+#endif
+
+/*
+ * ZSTD_window_canOverflowCorrect():
+ * Returns non-zero if the indices are large enough for overflow correction
+ * to work correctly without impacting compression ratio.
+ */
+MEM_STATIC U32 ZSTD_window_canOverflowCorrect(ZSTD_window_t const window,
+ U32 cycleLog,
+ U32 maxDist,
+ U32 loadedDictEnd,
+ void const* src)
+{
+ U32 const cycleSize = 1u << cycleLog;
+ U32 const curr = (U32)((BYTE const*)src - window.base);
+ U32 const minIndexToOverflowCorrect = cycleSize
+ + MAX(maxDist, cycleSize)
+ + ZSTD_WINDOW_START_INDEX;
+
+ /* Adjust the min index to backoff the overflow correction frequency,
+ * so we don't waste too much CPU in overflow correction. If this
+ * computation overflows we don't really care, we just need to make
+ * sure it is at least minIndexToOverflowCorrect.
+ */
+ U32 const adjustment = window.nbOverflowCorrections + 1;
+ U32 const adjustedIndex = MAX(minIndexToOverflowCorrect * adjustment,
+ minIndexToOverflowCorrect);
+ U32 const indexLargeEnough = curr > adjustedIndex;
+
+ /* Only overflow correct early if the dictionary is invalidated already,
+ * so we don't hurt compression ratio.
+ */
+ U32 const dictionaryInvalidated = curr > maxDist + loadedDictEnd;
+
+ return indexLargeEnough && dictionaryInvalidated;
+}
+
/*
* ZSTD_window_needOverflowCorrection():
* Returns non-zero if the indices are getting too large and need overflow
* protection.
*/
MEM_STATIC U32 ZSTD_window_needOverflowCorrection(ZSTD_window_t const window,
+ U32 cycleLog,
+ U32 maxDist,
+ U32 loadedDictEnd,
+ void const* src,
void const* srcEnd)
{
U32 const curr = (U32)((BYTE const*)srcEnd - window.base);
+ if (ZSTD_WINDOW_OVERFLOW_CORRECT_FREQUENTLY) {
+ if (ZSTD_window_canOverflowCorrect(window, cycleLog, maxDist, loadedDictEnd, src)) {
+ return 1;
+ }
+ }
return curr > ZSTD_CURRENT_MAX;
}
@@ -821,7 +1010,6 @@ MEM_STATIC U32 ZSTD_window_needOverflowCorrection(ZSTD_window_t const window,
*
* The least significant cycleLog bits of the indices must remain the same,
* which may be 0. Every index up to maxDist in the past must be valid.
- * NOTE: (maxDist & cycleMask) must be zero.
*/
MEM_STATIC U32 ZSTD_window_correctOverflow(ZSTD_window_t* window, U32 cycleLog,
U32 maxDist, void const* src)
@@ -845,32 +1033,52 @@ MEM_STATIC U32 ZSTD_window_correctOverflow(ZSTD_window_t* window, U32 cycleLog,
* 3. (cctx->lowLimit + 1<<windowLog) < 1<<32:
* windowLog <= 31 ==> 3<<29 + 1<<windowLog < 7<<29 < 1<<32.
*/
- U32 const cycleMask = (1U << cycleLog) - 1;
+ U32 const cycleSize = 1u << cycleLog;
+ U32 const cycleMask = cycleSize - 1;
U32 const curr = (U32)((BYTE const*)src - window->base);
- U32 const currentCycle0 = curr & cycleMask;
- /* Exclude zero so that newCurrent - maxDist >= 1. */
- U32 const currentCycle1 = currentCycle0 == 0 ? (1U << cycleLog) : currentCycle0;
- U32 const newCurrent = currentCycle1 + maxDist;
+ U32 const currentCycle = curr & cycleMask;
+ /* Ensure newCurrent - maxDist >= ZSTD_WINDOW_START_INDEX. */
+ U32 const currentCycleCorrection = currentCycle < ZSTD_WINDOW_START_INDEX
+ ? MAX(cycleSize, ZSTD_WINDOW_START_INDEX)
+ : 0;
+ U32 const newCurrent = currentCycle
+ + currentCycleCorrection
+ + MAX(maxDist, cycleSize);
U32 const correction = curr - newCurrent;
- assert((maxDist & cycleMask) == 0);
+ /* maxDist must be a power of two so that:
+ * (newCurrent & cycleMask) == (curr & cycleMask)
+ * This is required to not corrupt the chains / binary tree.
+ */
+ assert((maxDist & (maxDist - 1)) == 0);
+ assert((curr & cycleMask) == (newCurrent & cycleMask));
assert(curr > newCurrent);
- /* Loose bound, should be around 1<<29 (see above) */
- assert(correction > 1<<28);
+ if (!ZSTD_WINDOW_OVERFLOW_CORRECT_FREQUENTLY) {
+ /* Loose bound, should be around 1<<29 (see above) */
+ assert(correction > 1<<28);
+ }
window->base += correction;
window->dictBase += correction;
- if (window->lowLimit <= correction) window->lowLimit = 1;
- else window->lowLimit -= correction;
- if (window->dictLimit <= correction) window->dictLimit = 1;
- else window->dictLimit -= correction;
+ if (window->lowLimit < correction + ZSTD_WINDOW_START_INDEX) {
+ window->lowLimit = ZSTD_WINDOW_START_INDEX;
+ } else {
+ window->lowLimit -= correction;
+ }
+ if (window->dictLimit < correction + ZSTD_WINDOW_START_INDEX) {
+ window->dictLimit = ZSTD_WINDOW_START_INDEX;
+ } else {
+ window->dictLimit -= correction;
+ }
/* Ensure we can still reference the full window. */
assert(newCurrent >= maxDist);
- assert(newCurrent - maxDist >= 1);
+ assert(newCurrent - maxDist >= ZSTD_WINDOW_START_INDEX);
/* Ensure that lowLimit and dictLimit didn't underflow. */
assert(window->lowLimit <= newCurrent);
assert(window->dictLimit <= newCurrent);
+ ++window->nbOverflowCorrections;
+
DEBUGLOG(4, "Correction of 0x%x bytes to lowLimit=0x%x", correction,
window->lowLimit);
return correction;
@@ -975,11 +1183,13 @@ ZSTD_checkDictValidity(const ZSTD_window_t* window,
MEM_STATIC void ZSTD_window_init(ZSTD_window_t* window) {
ZSTD_memset(window, 0, sizeof(*window));
- window->base = (BYTE const*)"";
- window->dictBase = (BYTE const*)"";
- window->dictLimit = 1; /* start from 1, so that 1st position is valid */
- window->lowLimit = 1; /* it ensures first and later CCtx usages compress the same */
- window->nextSrc = window->base + 1; /* see issue #1241 */
+ window->base = (BYTE const*)" ";
+ window->dictBase = (BYTE const*)" ";
+ ZSTD_STATIC_ASSERT(ZSTD_DUBT_UNSORTED_MARK < ZSTD_WINDOW_START_INDEX); /* Start above ZSTD_DUBT_UNSORTED_MARK */
+ window->dictLimit = ZSTD_WINDOW_START_INDEX; /* start from >0, so that 1st position is valid */
+ window->lowLimit = ZSTD_WINDOW_START_INDEX; /* it ensures first and later CCtx usages compress the same */
+ window->nextSrc = window->base + ZSTD_WINDOW_START_INDEX; /* see issue #1241 */
+ window->nbOverflowCorrections = 0;
}
/*
@@ -990,7 +1200,8 @@ MEM_STATIC void ZSTD_window_init(ZSTD_window_t* window) {
* Returns non-zero if the segment is contiguous.
*/
MEM_STATIC U32 ZSTD_window_update(ZSTD_window_t* window,
- void const* src, size_t srcSize)
+ void const* src, size_t srcSize,
+ int forceNonContiguous)
{
BYTE const* const ip = (BYTE const*)src;
U32 contiguous = 1;
@@ -1000,7 +1211,7 @@ MEM_STATIC U32 ZSTD_window_update(ZSTD_window_t* window,
assert(window->base != NULL);
assert(window->dictBase != NULL);
/* Check if blocks follow each other */
- if (src != window->nextSrc) {
+ if (src != window->nextSrc || forceNonContiguous) {
/* not contiguous */
size_t const distanceFromBase = (size_t)(window->nextSrc - window->base);
DEBUGLOG(5, "Non contiguous blocks, new segment starts at %u", window->dictLimit);
@@ -1030,15 +1241,15 @@ MEM_STATIC U32 ZSTD_window_update(ZSTD_window_t* window,
*/
MEM_STATIC U32 ZSTD_getLowestMatchIndex(const ZSTD_matchState_t* ms, U32 curr, unsigned windowLog)
{
- U32 const maxDistance = 1U << windowLog;
- U32 const lowestValid = ms->window.lowLimit;
- U32 const withinWindow = (curr - lowestValid > maxDistance) ? curr - maxDistance : lowestValid;
- U32 const isDictionary = (ms->loadedDictEnd != 0);
+ U32 const maxDistance = 1U << windowLog;
+ U32 const lowestValid = ms->window.lowLimit;
+ U32 const withinWindow = (curr - lowestValid > maxDistance) ? curr - maxDistance : lowestValid;
+ U32 const isDictionary = (ms->loadedDictEnd != 0);
/* When using a dictionary the entire dictionary is valid if a single byte of the dictionary
* is within the window. We invalidate the dictionary (and set loadedDictEnd to 0) when it isn't
* valid for the entire block. So this check is sufficient to find the lowest valid match index.
*/
- U32 const matchLowest = isDictionary ? lowestValid : withinWindow;
+ U32 const matchLowest = isDictionary ? lowestValid : withinWindow;
return matchLowest;
}
diff --git a/lib/zstd/compress/zstd_compress_literals.c b/lib/zstd/compress/zstd_compress_literals.c
index 655bcda..52b0a80 100644
--- a/lib/zstd/compress/zstd_compress_literals.c
+++ b/lib/zstd/compress/zstd_compress_literals.c
@@ -73,7 +73,8 @@ size_t ZSTD_compressLiterals (ZSTD_hufCTables_t const* prevHuf,
void* dst, size_t dstCapacity,
const void* src, size_t srcSize,
void* entropyWorkspace, size_t entropyWorkspaceSize,
- const int bmi2)
+ const int bmi2,
+ unsigned suspectUncompressible)
{
size_t const minGain = ZSTD_minGain(srcSize, strategy);
size_t const lhSize = 3 + (srcSize >= 1 KB) + (srcSize >= 16 KB);
@@ -105,11 +106,11 @@ size_t ZSTD_compressLiterals (ZSTD_hufCTables_t const* prevHuf,
HUF_compress1X_repeat(
ostart+lhSize, dstCapacity-lhSize, src, srcSize,
HUF_SYMBOLVALUE_MAX, HUF_TABLELOG_DEFAULT, entropyWorkspace, entropyWorkspaceSize,
- (HUF_CElt*)nextHuf->CTable, &repeat, preferRepeat, bmi2) :
+ (HUF_CElt*)nextHuf->CTable, &repeat, preferRepeat, bmi2, suspectUncompressible) :
HUF_compress4X_repeat(
ostart+lhSize, dstCapacity-lhSize, src, srcSize,
HUF_SYMBOLVALUE_MAX, HUF_TABLELOG_DEFAULT, entropyWorkspace, entropyWorkspaceSize,
- (HUF_CElt*)nextHuf->CTable, &repeat, preferRepeat, bmi2);
+ (HUF_CElt*)nextHuf->CTable, &repeat, preferRepeat, bmi2, suspectUncompressible);
if (repeat != HUF_repeat_none) {
/* reused the existing table */
DEBUGLOG(5, "Reusing previous huffman table");
@@ -117,7 +118,7 @@ size_t ZSTD_compressLiterals (ZSTD_hufCTables_t const* prevHuf,
}
}
- if ((cLitSize==0) | (cLitSize >= srcSize - minGain) | ERR_isError(cLitSize)) {
+ if ((cLitSize==0) || (cLitSize >= srcSize - minGain) || ERR_isError(cLitSize)) {
ZSTD_memcpy(nextHuf, prevHuf, sizeof(*prevHuf));
return ZSTD_noCompressLiterals(dst, dstCapacity, src, srcSize);
}
diff --git a/lib/zstd/compress/zstd_compress_literals.h b/lib/zstd/compress/zstd_compress_literals.h
index 9904c0c..9775fb9 100644
--- a/lib/zstd/compress/zstd_compress_literals.h
+++ b/lib/zstd/compress/zstd_compress_literals.h
@@ -18,12 +18,14 @@ size_t ZSTD_noCompressLiterals (void* dst, size_t dstCapacity, const void* src,
size_t ZSTD_compressRleLiteralsBlock (void* dst, size_t dstCapacity, const void* src, size_t srcSize);
+/* If suspectUncompressible then some sampling checks will be run to potentially skip huffman coding */
size_t ZSTD_compressLiterals (ZSTD_hufCTables_t const* prevHuf,
ZSTD_hufCTables_t* nextHuf,
ZSTD_strategy strategy, int disableLiteralCompression,
void* dst, size_t dstCapacity,
const void* src, size_t srcSize,
void* entropyWorkspace, size_t entropyWorkspaceSize,
- const int bmi2);
+ const int bmi2,
+ unsigned suspectUncompressible);
#endif /* ZSTD_COMPRESS_LITERALS_H */
diff --git a/lib/zstd/compress/zstd_compress_sequences.c b/lib/zstd/compress/zstd_compress_sequences.c
index dcfcdc9..21ddc1b 100644
--- a/lib/zstd/compress/zstd_compress_sequences.c
+++ b/lib/zstd/compress/zstd_compress_sequences.c
@@ -85,6 +85,8 @@ static size_t ZSTD_entropyCost(unsigned const* count, unsigned const max, size_t
{
unsigned cost = 0;
unsigned s;
+
+ assert(total > 0);
for (s = 0; s <= max; ++s) {
unsigned norm = (unsigned)((256 * count[s]) / total);
if (count[s] != 0 && norm == 0)
@@ -273,10 +275,11 @@ ZSTD_buildCTable(void* dst, size_t dstCapacity,
assert(nbSeq_1 > 1);
assert(entropyWorkspaceSize >= sizeof(ZSTD_BuildCTableWksp));
(void)entropyWorkspaceSize;
- FORWARD_IF_ERROR(FSE_normalizeCount(wksp->norm, tableLog, count, nbSeq_1, max, ZSTD_useLowProbCount(nbSeq_1)), "");
- { size_t const NCountSize = FSE_writeNCount(op, oend - op, wksp->norm, max, tableLog); /* overflow protected */
+ FORWARD_IF_ERROR(FSE_normalizeCount(wksp->norm, tableLog, count, nbSeq_1, max, ZSTD_useLowProbCount(nbSeq_1)), "FSE_normalizeCount failed");
+ assert(oend >= op);
+ { size_t const NCountSize = FSE_writeNCount(op, (size_t)(oend - op), wksp->norm, max, tableLog); /* overflow protected */
FORWARD_IF_ERROR(NCountSize, "FSE_writeNCount failed");
- FORWARD_IF_ERROR(FSE_buildCTable_wksp(nextCTable, wksp->norm, max, tableLog, wksp->wksp, sizeof(wksp->wksp)), "");
+ FORWARD_IF_ERROR(FSE_buildCTable_wksp(nextCTable, wksp->norm, max, tableLog, wksp->wksp, sizeof(wksp->wksp)), "FSE_buildCTable_wksp failed");
return NCountSize;
}
}
@@ -310,19 +313,19 @@ ZSTD_encodeSequences_body(
FSE_initCState2(&stateLitLength, CTable_LitLength, llCodeTable[nbSeq-1]);
BIT_addBits(&blockStream, sequences[nbSeq-1].litLength, LL_bits[llCodeTable[nbSeq-1]]);
if (MEM_32bits()) BIT_flushBits(&blockStream);
- BIT_addBits(&blockStream, sequences[nbSeq-1].matchLength, ML_bits[mlCodeTable[nbSeq-1]]);
+ BIT_addBits(&blockStream, sequences[nbSeq-1].mlBase, ML_bits[mlCodeTable[nbSeq-1]]);
if (MEM_32bits()) BIT_flushBits(&blockStream);
if (longOffsets) {
U32 const ofBits = ofCodeTable[nbSeq-1];
unsigned const extraBits = ofBits - MIN(ofBits, STREAM_ACCUMULATOR_MIN-1);
if (extraBits) {
- BIT_addBits(&blockStream, sequences[nbSeq-1].offset, extraBits);
+ BIT_addBits(&blockStream, sequences[nbSeq-1].offBase, extraBits);
BIT_flushBits(&blockStream);
}
- BIT_addBits(&blockStream, sequences[nbSeq-1].offset >> extraBits,
+ BIT_addBits(&blockStream, sequences[nbSeq-1].offBase >> extraBits,
ofBits - extraBits);
} else {
- BIT_addBits(&blockStream, sequences[nbSeq-1].offset, ofCodeTable[nbSeq-1]);
+ BIT_addBits(&blockStream, sequences[nbSeq-1].offBase, ofCodeTable[nbSeq-1]);
}
BIT_flushBits(&blockStream);
@@ -336,8 +339,8 @@ ZSTD_encodeSequences_body(
U32 const mlBits = ML_bits[mlCode];
DEBUGLOG(6, "encoding: litlen:%2u - matchlen:%2u - offCode:%7u",
(unsigned)sequences[n].litLength,
- (unsigned)sequences[n].matchLength + MINMATCH,
- (unsigned)sequences[n].offset);
+ (unsigned)sequences[n].mlBase + MINMATCH,
+ (unsigned)sequences[n].offBase);
/* 32b*/ /* 64b*/
/* (7)*/ /* (7)*/
FSE_encodeSymbol(&blockStream, &stateOffsetBits, ofCode); /* 15 */ /* 15 */
@@ -348,18 +351,18 @@ ZSTD_encodeSequences_body(
BIT_flushBits(&blockStream); /* (7)*/
BIT_addBits(&blockStream, sequences[n].litLength, llBits);
if (MEM_32bits() && ((llBits+mlBits)>24)) BIT_flushBits(&blockStream);
- BIT_addBits(&blockStream, sequences[n].matchLength, mlBits);
+ BIT_addBits(&blockStream, sequences[n].mlBase, mlBits);
if (MEM_32bits() || (ofBits+mlBits+llBits > 56)) BIT_flushBits(&blockStream);
if (longOffsets) {
unsigned const extraBits = ofBits - MIN(ofBits, STREAM_ACCUMULATOR_MIN-1);
if (extraBits) {
- BIT_addBits(&blockStream, sequences[n].offset, extraBits);
+ BIT_addBits(&blockStream, sequences[n].offBase, extraBits);
BIT_flushBits(&blockStream); /* (7)*/
}
- BIT_addBits(&blockStream, sequences[n].offset >> extraBits,
+ BIT_addBits(&blockStream, sequences[n].offBase >> extraBits,
ofBits - extraBits); /* 31 */
} else {
- BIT_addBits(&blockStream, sequences[n].offset, ofBits); /* 31 */
+ BIT_addBits(&blockStream, sequences[n].offBase, ofBits); /* 31 */
}
BIT_flushBits(&blockStream); /* (7)*/
DEBUGLOG(7, "remaining space : %i", (int)(blockStream.endPtr - blockStream.ptr));
@@ -396,7 +399,7 @@ ZSTD_encodeSequences_default(
#if DYNAMIC_BMI2
-static TARGET_ATTRIBUTE("bmi2") size_t
+static BMI2_TARGET_ATTRIBUTE size_t
ZSTD_encodeSequences_bmi2(
void* dst, size_t dstCapacity,
FSE_CTable const* CTable_MatchLength, BYTE const* mlCodeTable,
diff --git a/lib/zstd/compress/zstd_compress_superblock.c b/lib/zstd/compress/zstd_compress_superblock.c
index b0610b2..17d836c 100644
--- a/lib/zstd/compress/zstd_compress_superblock.c
+++ b/lib/zstd/compress/zstd_compress_superblock.c
@@ -15,289 +15,10 @@
#include "../common/zstd_internal.h" /* ZSTD_getSequenceLength */
#include "hist.h" /* HIST_countFast_wksp */
-#include "zstd_compress_internal.h"
+#include "zstd_compress_internal.h" /* ZSTD_[huf|fse|entropy]CTablesMetadata_t */
#include "zstd_compress_sequences.h"
#include "zstd_compress_literals.h"
-/*-*************************************
-* Superblock entropy buffer structs
-***************************************/
-/* ZSTD_hufCTablesMetadata_t :
- * Stores Literals Block Type for a super-block in hType, and
- * huffman tree description in hufDesBuffer.
- * hufDesSize refers to the size of huffman tree description in bytes.
- * This metadata is populated in ZSTD_buildSuperBlockEntropy_literal() */
-typedef struct {
- symbolEncodingType_e hType;
- BYTE hufDesBuffer[ZSTD_MAX_HUF_HEADER_SIZE];
- size_t hufDesSize;
-} ZSTD_hufCTablesMetadata_t;
-
-/* ZSTD_fseCTablesMetadata_t :
- * Stores symbol compression modes for a super-block in {ll, ol, ml}Type, and
- * fse tables in fseTablesBuffer.
- * fseTablesSize refers to the size of fse tables in bytes.
- * This metadata is populated in ZSTD_buildSuperBlockEntropy_sequences() */
-typedef struct {
- symbolEncodingType_e llType;
- symbolEncodingType_e ofType;
- symbolEncodingType_e mlType;
- BYTE fseTablesBuffer[ZSTD_MAX_FSE_HEADERS_SIZE];
- size_t fseTablesSize;
- size_t lastCountSize; /* This is to account for bug in 1.3.4. More detail in ZSTD_compressSubBlock_sequences() */
-} ZSTD_fseCTablesMetadata_t;
-
-typedef struct {
- ZSTD_hufCTablesMetadata_t hufMetadata;
- ZSTD_fseCTablesMetadata_t fseMetadata;
-} ZSTD_entropyCTablesMetadata_t;
-
-
-/* ZSTD_buildSuperBlockEntropy_literal() :
- * Builds entropy for the super-block literals.
- * Stores literals block type (raw, rle, compressed, repeat) and
- * huffman description table to hufMetadata.
- * @return : size of huffman description table or error code */
-static size_t ZSTD_buildSuperBlockEntropy_literal(void* const src, size_t srcSize,
- const ZSTD_hufCTables_t* prevHuf,
- ZSTD_hufCTables_t* nextHuf,
- ZSTD_hufCTablesMetadata_t* hufMetadata,
- const int disableLiteralsCompression,
- void* workspace, size_t wkspSize)
-{
- BYTE* const wkspStart = (BYTE*)workspace;
- BYTE* const wkspEnd = wkspStart + wkspSize;
- BYTE* const countWkspStart = wkspStart;
- unsigned* const countWksp = (unsigned*)workspace;
- const size_t countWkspSize = (HUF_SYMBOLVALUE_MAX + 1) * sizeof(unsigned);
- BYTE* const nodeWksp = countWkspStart + countWkspSize;
- const size_t nodeWkspSize = wkspEnd-nodeWksp;
- unsigned maxSymbolValue = 255;
- unsigned huffLog = HUF_TABLELOG_DEFAULT;
- HUF_repeat repeat = prevHuf->repeatMode;
-
- DEBUGLOG(5, "ZSTD_buildSuperBlockEntropy_literal (srcSize=%zu)", srcSize);
-
- /* Prepare nextEntropy assuming reusing the existing table */
- ZSTD_memcpy(nextHuf, prevHuf, sizeof(*prevHuf));
-
- if (disableLiteralsCompression) {
- DEBUGLOG(5, "set_basic - disabled");
- hufMetadata->hType = set_basic;
- return 0;
- }
-
- /* small ? don't even attempt compression (speed opt) */
-# define COMPRESS_LITERALS_SIZE_MIN 63
- { size_t const minLitSize = (prevHuf->repeatMode == HUF_repeat_valid) ? 6 : COMPRESS_LITERALS_SIZE_MIN;
- if (srcSize <= minLitSize) {
- DEBUGLOG(5, "set_basic - too small");
- hufMetadata->hType = set_basic;
- return 0;
- }
- }
-
- /* Scan input and build symbol stats */
- { size_t const largest = HIST_count_wksp (countWksp, &maxSymbolValue, (const BYTE*)src, srcSize, workspace, wkspSize);
- FORWARD_IF_ERROR(largest, "HIST_count_wksp failed");
- if (largest == srcSize) {
- DEBUGLOG(5, "set_rle");
- hufMetadata->hType = set_rle;
- return 0;
- }
- if (largest <= (srcSize >> 7)+4) {
- DEBUGLOG(5, "set_basic - no gain");
- hufMetadata->hType = set_basic;
- return 0;
- }
- }
-
- /* Validate the previous Huffman table */
- if (repeat == HUF_repeat_check && !HUF_validateCTable((HUF_CElt const*)prevHuf->CTable, countWksp, maxSymbolValue)) {
- repeat = HUF_repeat_none;
- }
-
- /* Build Huffman Tree */
- ZSTD_memset(nextHuf->CTable, 0, sizeof(nextHuf->CTable));
- huffLog = HUF_optimalTableLog(huffLog, srcSize, maxSymbolValue);
- { size_t const maxBits = HUF_buildCTable_wksp((HUF_CElt*)nextHuf->CTable, countWksp,
- maxSymbolValue, huffLog,
- nodeWksp, nodeWkspSize);
- FORWARD_IF_ERROR(maxBits, "HUF_buildCTable_wksp");
- huffLog = (U32)maxBits;
- { /* Build and write the CTable */
- size_t const newCSize = HUF_estimateCompressedSize(
- (HUF_CElt*)nextHuf->CTable, countWksp, maxSymbolValue);
- size_t const hSize = HUF_writeCTable_wksp(
- hufMetadata->hufDesBuffer, sizeof(hufMetadata->hufDesBuffer),
- (HUF_CElt*)nextHuf->CTable, maxSymbolValue, huffLog,
- nodeWksp, nodeWkspSize);
- /* Check against repeating the previous CTable */
- if (repeat != HUF_repeat_none) {
- size_t const oldCSize = HUF_estimateCompressedSize(
- (HUF_CElt const*)prevHuf->CTable, countWksp, maxSymbolValue);
- if (oldCSize < srcSize && (oldCSize <= hSize + newCSize || hSize + 12 >= srcSize)) {
- DEBUGLOG(5, "set_repeat - smaller");
- ZSTD_memcpy(nextHuf, prevHuf, sizeof(*prevHuf));
- hufMetadata->hType = set_repeat;
- return 0;
- }
- }
- if (newCSize + hSize >= srcSize) {
- DEBUGLOG(5, "set_basic - no gains");
- ZSTD_memcpy(nextHuf, prevHuf, sizeof(*prevHuf));
- hufMetadata->hType = set_basic;
- return 0;
- }
- DEBUGLOG(5, "set_compressed (hSize=%u)", (U32)hSize);
- hufMetadata->hType = set_compressed;
- nextHuf->repeatMode = HUF_repeat_check;
- return hSize;
- }
- }
-}
-
-/* ZSTD_buildSuperBlockEntropy_sequences() :
- * Builds entropy for the super-block sequences.
- * Stores symbol compression modes and fse table to fseMetadata.
- * @return : size of fse tables or error code */
-static size_t ZSTD_buildSuperBlockEntropy_sequences(seqStore_t* seqStorePtr,
- const ZSTD_fseCTables_t* prevEntropy,
- ZSTD_fseCTables_t* nextEntropy,
- const ZSTD_CCtx_params* cctxParams,
- ZSTD_fseCTablesMetadata_t* fseMetadata,
- void* workspace, size_t wkspSize)
-{
- BYTE* const wkspStart = (BYTE*)workspace;
- BYTE* const wkspEnd = wkspStart + wkspSize;
- BYTE* const countWkspStart = wkspStart;
- unsigned* const countWksp = (unsigned*)workspace;
- const size_t countWkspSize = (MaxSeq + 1) * sizeof(unsigned);
- BYTE* const cTableWksp = countWkspStart + countWkspSize;
- const size_t cTableWkspSize = wkspEnd-cTableWksp;
- ZSTD_strategy const strategy = cctxParams->cParams.strategy;
- FSE_CTable* CTable_LitLength = nextEntropy->litlengthCTable;
- FSE_CTable* CTable_OffsetBits = nextEntropy->offcodeCTable;
- FSE_CTable* CTable_MatchLength = nextEntropy->matchlengthCTable;
- const BYTE* const ofCodeTable = seqStorePtr->ofCode;
- const BYTE* const llCodeTable = seqStorePtr->llCode;
- const BYTE* const mlCodeTable = seqStorePtr->mlCode;
- size_t const nbSeq = seqStorePtr->sequences - seqStorePtr->sequencesStart;
- BYTE* const ostart = fseMetadata->fseTablesBuffer;
- BYTE* const oend = ostart + sizeof(fseMetadata->fseTablesBuffer);
- BYTE* op = ostart;
-
- assert(cTableWkspSize >= (1 << MaxFSELog) * sizeof(FSE_FUNCTION_TYPE));
- DEBUGLOG(5, "ZSTD_buildSuperBlockEntropy_sequences (nbSeq=%zu)", nbSeq);
- ZSTD_memset(workspace, 0, wkspSize);
-
- fseMetadata->lastCountSize = 0;
- /* convert length/distances into codes */
- ZSTD_seqToCodes(seqStorePtr);
- /* build CTable for Literal Lengths */
- { U32 LLtype;
- unsigned max = MaxLL;
- size_t const mostFrequent = HIST_countFast_wksp(countWksp, &max, llCodeTable, nbSeq, workspace, wkspSize); /* can't fail */
- DEBUGLOG(5, "Building LL table");
- nextEntropy->litlength_repeatMode = prevEntropy->litlength_repeatMode;
- LLtype = ZSTD_selectEncodingType(&nextEntropy->litlength_repeatMode,
- countWksp, max, mostFrequent, nbSeq,
- LLFSELog, prevEntropy->litlengthCTable,
- LL_defaultNorm, LL_defaultNormLog,
- ZSTD_defaultAllowed, strategy);
- assert(set_basic < set_compressed && set_rle < set_compressed);
- assert(!(LLtype < set_compressed && nextEntropy->litlength_repeatMode != FSE_repeat_none)); /* We don't copy tables */
- { size_t const countSize = ZSTD_buildCTable(op, oend - op, CTable_LitLength, LLFSELog, (symbolEncodingType_e)LLtype,
- countWksp, max, llCodeTable, nbSeq, LL_defaultNorm, LL_defaultNormLog, MaxLL,
- prevEntropy->litlengthCTable, sizeof(prevEntropy->litlengthCTable),
- cTableWksp, cTableWkspSize);
- FORWARD_IF_ERROR(countSize, "ZSTD_buildCTable for LitLens failed");
- if (LLtype == set_compressed)
- fseMetadata->lastCountSize = countSize;
- op += countSize;
- fseMetadata->llType = (symbolEncodingType_e) LLtype;
- } }
- /* build CTable for Offsets */
- { U32 Offtype;
- unsigned max = MaxOff;
- size_t const mostFrequent = HIST_countFast_wksp(countWksp, &max, ofCodeTable, nbSeq, workspace, wkspSize); /* can't fail */
- /* We can only use the basic table if max <= DefaultMaxOff, otherwise the offsets are too large */
- ZSTD_defaultPolicy_e const defaultPolicy = (max <= DefaultMaxOff) ? ZSTD_defaultAllowed : ZSTD_defaultDisallowed;
- DEBUGLOG(5, "Building OF table");
- nextEntropy->offcode_repeatMode = prevEntropy->offcode_repeatMode;
- Offtype = ZSTD_selectEncodingType(&nextEntropy->offcode_repeatMode,
- countWksp, max, mostFrequent, nbSeq,
- OffFSELog, prevEntropy->offcodeCTable,
- OF_defaultNorm, OF_defaultNormLog,
- defaultPolicy, strategy);
- assert(!(Offtype < set_compressed && nextEntropy->offcode_repeatMode != FSE_repeat_none)); /* We don't copy tables */
- { size_t const countSize = ZSTD_buildCTable(op, oend - op, CTable_OffsetBits, OffFSELog, (symbolEncodingType_e)Offtype,
- countWksp, max, ofCodeTable, nbSeq, OF_defaultNorm, OF_defaultNormLog, DefaultMaxOff,
- prevEntropy->offcodeCTable, sizeof(prevEntropy->offcodeCTable),
- cTableWksp, cTableWkspSize);
- FORWARD_IF_ERROR(countSize, "ZSTD_buildCTable for Offsets failed");
- if (Offtype == set_compressed)
- fseMetadata->lastCountSize = countSize;
- op += countSize;
- fseMetadata->ofType = (symbolEncodingType_e) Offtype;
- } }
- /* build CTable for MatchLengths */
- { U32 MLtype;
- unsigned max = MaxML;
- size_t const mostFrequent = HIST_countFast_wksp(countWksp, &max, mlCodeTable, nbSeq, workspace, wkspSize); /* can't fail */
- DEBUGLOG(5, "Building ML table (remaining space : %i)", (int)(oend-op));
- nextEntropy->matchlength_repeatMode = prevEntropy->matchlength_repeatMode;
- MLtype = ZSTD_selectEncodingType(&nextEntropy->matchlength_repeatMode,
- countWksp, max, mostFrequent, nbSeq,
- MLFSELog, prevEntropy->matchlengthCTable,
- ML_defaultNorm, ML_defaultNormLog,
- ZSTD_defaultAllowed, strategy);
- assert(!(MLtype < set_compressed && nextEntropy->matchlength_repeatMode != FSE_repeat_none)); /* We don't copy tables */
- { size_t const countSize = ZSTD_buildCTable(op, oend - op, CTable_MatchLength, MLFSELog, (symbolEncodingType_e)MLtype,
- countWksp, max, mlCodeTable, nbSeq, ML_defaultNorm, ML_defaultNormLog, MaxML,
- prevEntropy->matchlengthCTable, sizeof(prevEntropy->matchlengthCTable),
- cTableWksp, cTableWkspSize);
- FORWARD_IF_ERROR(countSize, "ZSTD_buildCTable for MatchLengths failed");
- if (MLtype == set_compressed)
- fseMetadata->lastCountSize = countSize;
- op += countSize;
- fseMetadata->mlType = (symbolEncodingType_e) MLtype;
- } }
- assert((size_t) (op-ostart) <= sizeof(fseMetadata->fseTablesBuffer));
- return op-ostart;
-}
-
-
-/* ZSTD_buildSuperBlockEntropy() :
- * Builds entropy for the super-block.
- * @return : 0 on success or error code */
-static size_t
-ZSTD_buildSuperBlockEntropy(seqStore_t* seqStorePtr,
- const ZSTD_entropyCTables_t* prevEntropy,
- ZSTD_entropyCTables_t* nextEntropy,
- const ZSTD_CCtx_params* cctxParams,
- ZSTD_entropyCTablesMetadata_t* entropyMetadata,
- void* workspace, size_t wkspSize)
-{
- size_t const litSize = seqStorePtr->lit - seqStorePtr->litStart;
- DEBUGLOG(5, "ZSTD_buildSuperBlockEntropy");
- entropyMetadata->hufMetadata.hufDesSize =
- ZSTD_buildSuperBlockEntropy_literal(seqStorePtr->litStart, litSize,
- &prevEntropy->huf, &nextEntropy->huf,
- &entropyMetadata->hufMetadata,
- ZSTD_disableLiteralsCompression(cctxParams),
- workspace, wkspSize);
- FORWARD_IF_ERROR(entropyMetadata->hufMetadata.hufDesSize, "ZSTD_buildSuperBlockEntropy_literal failed");
- entropyMetadata->fseMetadata.fseTablesSize =
- ZSTD_buildSuperBlockEntropy_sequences(seqStorePtr,
- &prevEntropy->fse, &nextEntropy->fse,
- cctxParams,
- &entropyMetadata->fseMetadata,
- workspace, wkspSize);
- FORWARD_IF_ERROR(entropyMetadata->fseMetadata.fseTablesSize, "ZSTD_buildSuperBlockEntropy_sequences failed");
- return 0;
-}
-
/* ZSTD_compressSubBlock_literal() :
* Compresses literals section for a sub-block.
* When we have to write the Huffman table we will sometimes choose a header
@@ -411,8 +132,7 @@ static size_t ZSTD_seqDecompressedSize(seqStore_t const* seqStore, const seqDef*
const seqDef* sp = sstart;
size_t matchLengthSum = 0;
size_t litLengthSum = 0;
- /* Only used by assert(), suppress unused variable warnings in production. */
- (void)litLengthSum;
+ (void)(litLengthSum); /* suppress unused variable warning on some environments */
while (send-sp > 0) {
ZSTD_sequenceLength const seqLen = ZSTD_getSequenceLength(seqStore, sp);
litLengthSum += seqLen.litLength;
@@ -605,7 +325,7 @@ static size_t ZSTD_estimateSubBlockSize_literal(const BYTE* literals, size_t lit
static size_t ZSTD_estimateSubBlockSize_symbolType(symbolEncodingType_e type,
const BYTE* codeTable, unsigned maxCode,
size_t nbSeq, const FSE_CTable* fseCTable,
- const U32* additionalBits,
+ const U8* additionalBits,
short const* defaultNorm, U32 defaultNormLog, U32 defaultMax,
void* workspace, size_t wkspSize)
{
@@ -646,8 +366,9 @@ static size_t ZSTD_estimateSubBlockSize_sequences(const BYTE* ofCodeTable,
void* workspace, size_t wkspSize,
int writeEntropy)
{
- size_t sequencesSectionHeaderSize = 3; /* Use hard coded size of 3 bytes */
+ size_t const sequencesSectionHeaderSize = 3; /* Use hard coded size of 3 bytes */
size_t cSeqSizeEstimate = 0;
+ if (nbSeq == 0) return sequencesSectionHeaderSize;
cSeqSizeEstimate += ZSTD_estimateSubBlockSize_symbolType(fseMetadata->ofType, ofCodeTable, MaxOff,
nbSeq, fseTables->offcodeCTable, NULL,
OF_defaultNorm, OF_defaultNormLog, DefaultMaxOff,
@@ -754,7 +475,7 @@ static size_t ZSTD_compressSubBlock_multi(const seqStore_t* seqStorePtr,
/* I think there is an optimization opportunity here.
* Calling ZSTD_estimateSubBlockSize for every sequence can be wasteful
* since it recalculates estimate from scratch.
- * For example, it would recount literal distribution and symbol codes everytime.
+ * For example, it would recount literal distribution and symbol codes every time.
*/
cBlockSizeEstimate = ZSTD_estimateSubBlockSize(lp, litSize, ofCodePtr, llCodePtr, mlCodePtr, seqCount,
&nextCBlock->entropy, entropyMetadata,
@@ -818,7 +539,7 @@ static size_t ZSTD_compressSubBlock_multi(const seqStore_t* seqStorePtr,
repcodes_t rep;
ZSTD_memcpy(&rep, prevCBlock->rep, sizeof(rep));
for (seq = sstart; seq < sp; ++seq) {
- rep = ZSTD_updateRep(rep.rep, seq->offset - 1, ZSTD_getSequenceLength(seqStorePtr, seq).litLength == 0);
+ ZSTD_updateRep(rep.rep, seq->offBase - 1, ZSTD_getSequenceLength(seqStorePtr, seq).litLength == 0);
}
ZSTD_memcpy(nextCBlock->rep, &rep, sizeof(rep));
}
@@ -833,7 +554,7 @@ size_t ZSTD_compressSuperBlock(ZSTD_CCtx* zc,
unsigned lastBlock) {
ZSTD_entropyCTablesMetadata_t entropyMetadata;
- FORWARD_IF_ERROR(ZSTD_buildSuperBlockEntropy(&zc->seqStore,
+ FORWARD_IF_ERROR(ZSTD_buildBlockEntropyStats(&zc->seqStore,
&zc->blockState.prevCBlock->entropy,
&zc->blockState.nextCBlock->entropy,
&zc->appliedParams,
diff --git a/lib/zstd/compress/zstd_cwksp.h b/lib/zstd/compress/zstd_cwksp.h
index 98e359a..349fc92 100644
--- a/lib/zstd/compress/zstd_cwksp.h
+++ b/lib/zstd/compress/zstd_cwksp.h
@@ -32,6 +32,10 @@
#define ZSTD_CWKSP_ASAN_REDZONE_SIZE 128
#endif
+
+/* Set our tables and aligneds to align by 64 bytes */
+#define ZSTD_CWKSP_ALIGNMENT_BYTES 64
+
/*-*************************************
* Structures
***************************************/
@@ -114,10 +118,11 @@ typedef enum {
* - Tables: these are any of several different datastructures (hash tables,
* chain tables, binary trees) that all respect a common format: they are
* uint32_t arrays, all of whose values are between 0 and (nextSrc - base).
- * Their sizes depend on the cparams.
+ * Their sizes depend on the cparams. These tables are 64-byte aligned.
*
* - Aligned: these buffers are used for various purposes that require 4 byte
- * alignment, but don't require any initialization before they're used.
+ * alignment, but don't require any initialization before they're used. These
+ * buffers are each aligned to 64 bytes.
*
* - Buffers: these buffers are used for various purposes that don't require
* any alignment or initialization before they're used. This means they can
@@ -130,8 +135,7 @@ typedef enum {
*
* 1. Objects
* 2. Buffers
- * 3. Aligned
- * 4. Tables
+ * 3. Aligned/Tables
*
* Attempts to reserve objects of different types out of order will fail.
*/
@@ -184,6 +188,8 @@ MEM_STATIC size_t ZSTD_cwksp_align(size_t size, size_t const align) {
* Since tables aren't currently redzoned, you don't need to call through this
* to figure out how much space you need for the matchState tables. Everything
* else is though.
+ *
+ * Do not use for sizing aligned buffers. Instead, use ZSTD_cwksp_aligned_alloc_size().
*/
MEM_STATIC size_t ZSTD_cwksp_alloc_size(size_t size) {
if (size == 0)
@@ -191,53 +197,55 @@ MEM_STATIC size_t ZSTD_cwksp_alloc_size(size_t size) {
return size;
}
-MEM_STATIC void ZSTD_cwksp_internal_advance_phase(
- ZSTD_cwksp* ws, ZSTD_cwksp_alloc_phase_e phase) {
- assert(phase >= ws->phase);
- if (phase > ws->phase) {
- if (ws->phase < ZSTD_cwksp_alloc_buffers &&
- phase >= ZSTD_cwksp_alloc_buffers) {
- ws->tableValidEnd = ws->objectEnd;
- }
- if (ws->phase < ZSTD_cwksp_alloc_aligned &&
- phase >= ZSTD_cwksp_alloc_aligned) {
- /* If unaligned allocations down from a too-large top have left us
- * unaligned, we need to realign our alloc ptr. Technically, this
- * can consume space that is unaccounted for in the neededSpace
- * calculation. However, I believe this can only happen when the
- * workspace is too large, and specifically when it is too large
- * by a larger margin than the space that will be consumed. */
- /* TODO: cleaner, compiler warning friendly way to do this??? */
- ws->allocStart = (BYTE*)ws->allocStart - ((size_t)ws->allocStart & (sizeof(U32)-1));
- if (ws->allocStart < ws->tableValidEnd) {
- ws->tableValidEnd = ws->allocStart;
- }
- }
- ws->phase = phase;
- }
+/*
+ * Returns an adjusted alloc size that is the nearest larger multiple of 64 bytes.
+ * Used to determine the number of bytes required for a given "aligned".
+ */
+MEM_STATIC size_t ZSTD_cwksp_aligned_alloc_size(size_t size) {
+ return ZSTD_cwksp_alloc_size(ZSTD_cwksp_align(size, ZSTD_CWKSP_ALIGNMENT_BYTES));
}
/*
- * Returns whether this object/buffer/etc was allocated in this workspace.
+ * Returns the amount of additional space the cwksp must allocate
+ * for internal purposes (currently only alignment).
*/
-MEM_STATIC int ZSTD_cwksp_owns_buffer(const ZSTD_cwksp* ws, const void* ptr) {
- return (ptr != NULL) && (ws->workspace <= ptr) && (ptr <= ws->workspaceEnd);
+MEM_STATIC size_t ZSTD_cwksp_slack_space_required(void) {
+ /* For alignment, the wksp will always allocate an additional n_1=[1, 64] bytes
+ * to align the beginning of tables section, as well as another n_2=[0, 63] bytes
+ * to align the beginning of the aligned section.
+ *
+ * n_1 + n_2 == 64 bytes if the cwksp is freshly allocated, due to tables and
+ * aligneds being sized in multiples of 64 bytes.
+ */
+ size_t const slackSpace = ZSTD_CWKSP_ALIGNMENT_BYTES;
+ return slackSpace;
+}
+
+
+/*
+ * Return the number of additional bytes required to align a pointer to the given number of bytes.
+ * alignBytes must be a power of two.
+ */
+MEM_STATIC size_t ZSTD_cwksp_bytes_to_align_ptr(void* ptr, const size_t alignBytes) {
+ size_t const alignBytesMask = alignBytes - 1;
+ size_t const bytes = (alignBytes - ((size_t)ptr & (alignBytesMask))) & alignBytesMask;
+ assert((alignBytes & alignBytesMask) == 0);
+ assert(bytes != ZSTD_CWKSP_ALIGNMENT_BYTES);
+ return bytes;
}
/*
* Internal function. Do not use directly.
+ * Reserves the given number of bytes within the aligned/buffer segment of the wksp,
+ * which counts from the end of the wksp (as opposed to the object/table segment).
+ *
+ * Returns a pointer to the beginning of that space.
*/
-MEM_STATIC void* ZSTD_cwksp_reserve_internal(
- ZSTD_cwksp* ws, size_t bytes, ZSTD_cwksp_alloc_phase_e phase) {
- void* alloc;
- void* bottom = ws->tableEnd;
- ZSTD_cwksp_internal_advance_phase(ws, phase);
- alloc = (BYTE *)ws->allocStart - bytes;
-
- if (bytes == 0)
- return NULL;
-
-
+MEM_STATIC void*
+ZSTD_cwksp_reserve_internal_buffer_space(ZSTD_cwksp* ws, size_t const bytes)
+{
+ void* const alloc = (BYTE*)ws->allocStart - bytes;
+ void* const bottom = ws->tableEnd;
DEBUGLOG(5, "cwksp: reserving %p %zd bytes, %zd bytes remaining",
alloc, bytes, ZSTD_cwksp_available_space(ws) - bytes);
ZSTD_cwksp_assert_internal_consistency(ws);
@@ -247,10 +255,81 @@ MEM_STATIC void* ZSTD_cwksp_reserve_internal(
ws->allocFailed = 1;
return NULL;
}
+ /* the area is reserved from the end of wksp.
+ * If it overlaps with tableValidEnd, it voids guarantees on values' range */
if (alloc < ws->tableValidEnd) {
ws->tableValidEnd = alloc;
}
ws->allocStart = alloc;
+ return alloc;
+}
+
+/*
+ * Moves the cwksp to the next phase, and does any necessary allocations.
+ * cwksp initialization must necessarily go through each phase in order.
+ * Returns a 0 on success, or zstd error
+ */
+MEM_STATIC size_t
+ZSTD_cwksp_internal_advance_phase(ZSTD_cwksp* ws, ZSTD_cwksp_alloc_phase_e phase)
+{
+ assert(phase >= ws->phase);
+ if (phase > ws->phase) {
+ /* Going from allocating objects to allocating buffers */
+ if (ws->phase < ZSTD_cwksp_alloc_buffers &&
+ phase >= ZSTD_cwksp_alloc_buffers) {
+ ws->tableValidEnd = ws->objectEnd;
+ }
+
+ /* Going from allocating buffers to allocating aligneds/tables */
+ if (ws->phase < ZSTD_cwksp_alloc_aligned &&
+ phase >= ZSTD_cwksp_alloc_aligned) {
+ { /* Align the start of the "aligned" to 64 bytes. Use [1, 64] bytes. */
+ size_t const bytesToAlign =
+ ZSTD_CWKSP_ALIGNMENT_BYTES - ZSTD_cwksp_bytes_to_align_ptr(ws->allocStart, ZSTD_CWKSP_ALIGNMENT_BYTES);
+ DEBUGLOG(5, "reserving aligned alignment addtl space: %zu", bytesToAlign);
+ ZSTD_STATIC_ASSERT((ZSTD_CWKSP_ALIGNMENT_BYTES & (ZSTD_CWKSP_ALIGNMENT_BYTES - 1)) == 0); /* power of 2 */
+ RETURN_ERROR_IF(!ZSTD_cwksp_reserve_internal_buffer_space(ws, bytesToAlign),
+ memory_allocation, "aligned phase - alignment initial allocation failed!");
+ }
+ { /* Align the start of the tables to 64 bytes. Use [0, 63] bytes */
+ void* const alloc = ws->objectEnd;
+ size_t const bytesToAlign = ZSTD_cwksp_bytes_to_align_ptr(alloc, ZSTD_CWKSP_ALIGNMENT_BYTES);
+ void* const objectEnd = (BYTE*)alloc + bytesToAlign;
+ DEBUGLOG(5, "reserving table alignment addtl space: %zu", bytesToAlign);
+ RETURN_ERROR_IF(objectEnd > ws->workspaceEnd, memory_allocation,
+ "table phase - alignment initial allocation failed!");
+ ws->objectEnd = objectEnd;
+ ws->tableEnd = objectEnd; /* table area starts being empty */
+ if (ws->tableValidEnd < ws->tableEnd) {
+ ws->tableValidEnd = ws->tableEnd;
+ } } }
+ ws->phase = phase;
+ ZSTD_cwksp_assert_internal_consistency(ws);
+ }
+ return 0;
+}
+
+/*
+ * Returns whether this object/buffer/etc was allocated in this workspace.
+ */
+MEM_STATIC int ZSTD_cwksp_owns_buffer(const ZSTD_cwksp* ws, const void* ptr)
+{
+ return (ptr != NULL) && (ws->workspace <= ptr) && (ptr <= ws->workspaceEnd);
+}
+
+/*
+ * Internal function. Do not use directly.
+ */
+MEM_STATIC void*
+ZSTD_cwksp_reserve_internal(ZSTD_cwksp* ws, size_t bytes, ZSTD_cwksp_alloc_phase_e phase)
+{
+ void* alloc;
+ if (ZSTD_isError(ZSTD_cwksp_internal_advance_phase(ws, phase)) || bytes == 0) {
+ return NULL;
+ }
+
+
+ alloc = ZSTD_cwksp_reserve_internal_buffer_space(ws, bytes);
return alloc;
@@ -259,33 +338,44 @@ MEM_STATIC void* ZSTD_cwksp_reserve_internal(
/*
* Reserves and returns unaligned memory.
*/
-MEM_STATIC BYTE* ZSTD_cwksp_reserve_buffer(ZSTD_cwksp* ws, size_t bytes) {
+MEM_STATIC BYTE* ZSTD_cwksp_reserve_buffer(ZSTD_cwksp* ws, size_t bytes)
+{
return (BYTE*)ZSTD_cwksp_reserve_internal(ws, bytes, ZSTD_cwksp_alloc_buffers);
}
/*
- * Reserves and returns memory sized on and aligned on sizeof(unsigned).
+ * Reserves and returns memory sized on and aligned on ZSTD_CWKSP_ALIGNMENT_BYTES (64 bytes).
*/
-MEM_STATIC void* ZSTD_cwksp_reserve_aligned(ZSTD_cwksp* ws, size_t bytes) {
- assert((bytes & (sizeof(U32)-1)) == 0);
- return ZSTD_cwksp_reserve_internal(ws, ZSTD_cwksp_align(bytes, sizeof(U32)), ZSTD_cwksp_alloc_aligned);
+MEM_STATIC void* ZSTD_cwksp_reserve_aligned(ZSTD_cwksp* ws, size_t bytes)
+{
+ void* ptr = ZSTD_cwksp_reserve_internal(ws, ZSTD_cwksp_align(bytes, ZSTD_CWKSP_ALIGNMENT_BYTES),
+ ZSTD_cwksp_alloc_aligned);
+ assert(((size_t)ptr & (ZSTD_CWKSP_ALIGNMENT_BYTES-1))== 0);
+ return ptr;
}
/*
- * Aligned on sizeof(unsigned). These buffers have the special property that
+ * Aligned on 64 bytes. These buffers have the special property that
* their values remain constrained, allowing us to re-use them without
* memset()-ing them.
*/
-MEM_STATIC void* ZSTD_cwksp_reserve_table(ZSTD_cwksp* ws, size_t bytes) {
+MEM_STATIC void* ZSTD_cwksp_reserve_table(ZSTD_cwksp* ws, size_t bytes)
+{
const ZSTD_cwksp_alloc_phase_e phase = ZSTD_cwksp_alloc_aligned;
- void* alloc = ws->tableEnd;
- void* end = (BYTE *)alloc + bytes;
- void* top = ws->allocStart;
+ void* alloc;
+ void* end;
+ void* top;
+
+ if (ZSTD_isError(ZSTD_cwksp_internal_advance_phase(ws, phase))) {
+ return NULL;
+ }
+ alloc = ws->tableEnd;
+ end = (BYTE *)alloc + bytes;
+ top = ws->allocStart;
DEBUGLOG(5, "cwksp: reserving %p table %zd bytes, %zd bytes remaining",
alloc, bytes, ZSTD_cwksp_available_space(ws) - bytes);
assert((bytes & (sizeof(U32)-1)) == 0);
- ZSTD_cwksp_internal_advance_phase(ws, phase);
ZSTD_cwksp_assert_internal_consistency(ws);
assert(end <= top);
if (end > top) {
@@ -296,27 +386,31 @@ MEM_STATIC void* ZSTD_cwksp_reserve_table(ZSTD_cwksp* ws, size_t bytes) {
ws->tableEnd = end;
+ assert((bytes & (ZSTD_CWKSP_ALIGNMENT_BYTES-1)) == 0);
+ assert(((size_t)alloc & (ZSTD_CWKSP_ALIGNMENT_BYTES-1))== 0);
return alloc;
}
/*
* Aligned on sizeof(void*).
+ * Note : should happen only once, at workspace first initialization
*/
-MEM_STATIC void* ZSTD_cwksp_reserve_object(ZSTD_cwksp* ws, size_t bytes) {
- size_t roundedBytes = ZSTD_cwksp_align(bytes, sizeof(void*));
+MEM_STATIC void* ZSTD_cwksp_reserve_object(ZSTD_cwksp* ws, size_t bytes)
+{
+ size_t const roundedBytes = ZSTD_cwksp_align(bytes, sizeof(void*));
void* alloc = ws->objectEnd;
void* end = (BYTE*)alloc + roundedBytes;
- DEBUGLOG(5,
+ DEBUGLOG(4,
"cwksp: reserving %p object %zd bytes (rounded to %zd), %zd bytes remaining",
alloc, bytes, roundedBytes, ZSTD_cwksp_available_space(ws) - roundedBytes);
- assert(((size_t)alloc & (sizeof(void*)-1)) == 0);
- assert((bytes & (sizeof(void*)-1)) == 0);
+ assert((size_t)alloc % ZSTD_ALIGNOF(void*) == 0);
+ assert(bytes % ZSTD_ALIGNOF(void*) == 0);
ZSTD_cwksp_assert_internal_consistency(ws);
/* we must be in the first phase, no advance is possible */
if (ws->phase != ZSTD_cwksp_alloc_objects || end > ws->workspaceEnd) {
- DEBUGLOG(4, "cwksp: object alloc failed!");
+ DEBUGLOG(3, "cwksp: object alloc failed!");
ws->allocFailed = 1;
return NULL;
}
@@ -328,7 +422,8 @@ MEM_STATIC void* ZSTD_cwksp_reserve_object(ZSTD_cwksp* ws, size_t bytes) {
return alloc;
}
-MEM_STATIC void ZSTD_cwksp_mark_tables_dirty(ZSTD_cwksp* ws) {
+MEM_STATIC void ZSTD_cwksp_mark_tables_dirty(ZSTD_cwksp* ws)
+{
DEBUGLOG(4, "cwksp: ZSTD_cwksp_mark_tables_dirty");
@@ -451,6 +546,24 @@ MEM_STATIC int ZSTD_cwksp_reserve_failed(const ZSTD_cwksp* ws) {
* Functions Checking Free Space
***************************************/
+/* ZSTD_alignmentSpaceWithinBounds() :
+ * Returns if the estimated space needed for a wksp is within an acceptable limit of the
+ * actual amount of space used.
+ */
+MEM_STATIC int ZSTD_cwksp_estimated_space_within_bounds(const ZSTD_cwksp* const ws,
+ size_t const estimatedSpace, int resizedWorkspace) {
+ if (resizedWorkspace) {
+ /* Resized/newly allocated wksp should have exact bounds */
+ return ZSTD_cwksp_used(ws) == estimatedSpace;
+ } else {
+ /* Due to alignment, when reusing a workspace, we can actually consume 63 fewer or more bytes
+ * than estimatedSpace. See the comments in zstd_cwksp.h for details.
+ */
+ return (ZSTD_cwksp_used(ws) >= estimatedSpace - 63) && (ZSTD_cwksp_used(ws) <= estimatedSpace + 63);
+ }
+}
+
+
MEM_STATIC size_t ZSTD_cwksp_available_space(ZSTD_cwksp* ws) {
return (size_t)((BYTE*)ws->allocStart - (BYTE*)ws->tableEnd);
}
diff --git a/lib/zstd/compress/zstd_double_fast.c b/lib/zstd/compress/zstd_double_fast.c
index b0424d2..76933de 100644
--- a/lib/zstd/compress/zstd_double_fast.c
+++ b/lib/zstd/compress/zstd_double_fast.c
@@ -48,10 +48,216 @@ void ZSTD_fillDoubleHashTable(ZSTD_matchState_t* ms,
FORCE_INLINE_TEMPLATE
-size_t ZSTD_compressBlock_doubleFast_generic(
+size_t ZSTD_compressBlock_doubleFast_noDict_generic(
+ ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM],
+ void const* src, size_t srcSize, U32 const mls /* template */)
+{
+ ZSTD_compressionParameters const* cParams = &ms->cParams;
+ U32* const hashLong = ms->hashTable;
+ const U32 hBitsL = cParams->hashLog;
+ U32* const hashSmall = ms->chainTable;
+ const U32 hBitsS = cParams->chainLog;
+ const BYTE* const base = ms->window.base;
+ const BYTE* const istart = (const BYTE*)src;
+ const BYTE* anchor = istart;
+ const U32 endIndex = (U32)((size_t)(istart - base) + srcSize);
+ /* presumes that, if there is a dictionary, it must be using Attach mode */
+ const U32 prefixLowestIndex = ZSTD_getLowestPrefixIndex(ms, endIndex, cParams->windowLog);
+ const BYTE* const prefixLowest = base + prefixLowestIndex;
+ const BYTE* const iend = istart + srcSize;
+ const BYTE* const ilimit = iend - HASH_READ_SIZE;
+ U32 offset_1=rep[0], offset_2=rep[1];
+ U32 offsetSaved = 0;
+
+ size_t mLength;
+ U32 offset;
+ U32 curr;
+
+ /* how many positions to search before increasing step size */
+ const size_t kStepIncr = 1 << kSearchStrength;
+ /* the position at which to increment the step size if no match is found */
+ const BYTE* nextStep;
+ size_t step; /* the current step size */
+
+ size_t hl0; /* the long hash at ip */
+ size_t hl1; /* the long hash at ip1 */
+
+ U32 idxl0; /* the long match index for ip */
+ U32 idxl1; /* the long match index for ip1 */
+
+ const BYTE* matchl0; /* the long match for ip */
+ const BYTE* matchs0; /* the short match for ip */
+ const BYTE* matchl1; /* the long match for ip1 */
+
+ const BYTE* ip = istart; /* the current position */
+ const BYTE* ip1; /* the next position */
+
+ DEBUGLOG(5, "ZSTD_compressBlock_doubleFast_noDict_generic");
+
+ /* init */
+ ip += ((ip - prefixLowest) == 0);
+ {
+ U32 const current = (U32)(ip - base);
+ U32 const windowLow = ZSTD_getLowestPrefixIndex(ms, current, cParams->windowLog);
+ U32 const maxRep = current - windowLow;
+ if (offset_2 > maxRep) offsetSaved = offset_2, offset_2 = 0;
+ if (offset_1 > maxRep) offsetSaved = offset_1, offset_1 = 0;
+ }
+
+ /* Outer Loop: one iteration per match found and stored */
+ while (1) {
+ step = 1;
+ nextStep = ip + kStepIncr;
+ ip1 = ip + step;
+
+ if (ip1 > ilimit) {
+ goto _cleanup;
+ }
+
+ hl0 = ZSTD_hashPtr(ip, hBitsL, 8);
+ idxl0 = hashLong[hl0];
+ matchl0 = base + idxl0;
+
+ /* Inner Loop: one iteration per search / position */
+ do {
+ const size_t hs0 = ZSTD_hashPtr(ip, hBitsS, mls);
+ const U32 idxs0 = hashSmall[hs0];
+ curr = (U32)(ip-base);
+ matchs0 = base + idxs0;
+
+ hashLong[hl0] = hashSmall[hs0] = curr; /* update hash tables */
+
+ /* check noDict repcode */
+ if ((offset_1 > 0) & (MEM_read32(ip+1-offset_1) == MEM_read32(ip+1))) {
+ mLength = ZSTD_count(ip+1+4, ip+1+4-offset_1, iend) + 4;
+ ip++;
+ ZSTD_storeSeq(seqStore, (size_t)(ip-anchor), anchor, iend, STORE_REPCODE_1, mLength);
+ goto _match_stored;
+ }
+
+ hl1 = ZSTD_hashPtr(ip1, hBitsL, 8);
+
+ if (idxl0 > prefixLowestIndex) {
+ /* check prefix long match */
+ if (MEM_read64(matchl0) == MEM_read64(ip)) {
+ mLength = ZSTD_count(ip+8, matchl0+8, iend) + 8;
+ offset = (U32)(ip-matchl0);
+ while (((ip>anchor) & (matchl0>prefixLowest)) && (ip[-1] == matchl0[-1])) { ip--; matchl0--; mLength++; } /* catch up */
+ goto _match_found;
+ }
+ }
+
+ idxl1 = hashLong[hl1];
+ matchl1 = base + idxl1;
+
+ if (idxs0 > prefixLowestIndex) {
+ /* check prefix short match */
+ if (MEM_read32(matchs0) == MEM_read32(ip)) {
+ goto _search_next_long;
+ }
+ }
+
+ if (ip1 >= nextStep) {
+ PREFETCH_L1(ip1 + 64);
+ PREFETCH_L1(ip1 + 128);
+ step++;
+ nextStep += kStepIncr;
+ }
+ ip = ip1;
+ ip1 += step;
+
+ hl0 = hl1;
+ idxl0 = idxl1;
+ matchl0 = matchl1;
+ #if defined(__aarch64__)
+ PREFETCH_L1(ip+256);
+ #endif
+ } while (ip1 <= ilimit);
+
+_cleanup:
+ /* save reps for next block */
+ rep[0] = offset_1 ? offset_1 : offsetSaved;
+ rep[1] = offset_2 ? offset_2 : offsetSaved;
+
+ /* Return the last literals size */
+ return (size_t)(iend - anchor);
+
+_search_next_long:
+
+ /* check prefix long +1 match */
+ if (idxl1 > prefixLowestIndex) {
+ if (MEM_read64(matchl1) == MEM_read64(ip1)) {
+ ip = ip1;
+ mLength = ZSTD_count(ip+8, matchl1+8, iend) + 8;
+ offset = (U32)(ip-matchl1);
+ while (((ip>anchor) & (matchl1>prefixLowest)) && (ip[-1] == matchl1[-1])) { ip--; matchl1--; mLength++; } /* catch up */
+ goto _match_found;
+ }
+ }
+
+ /* if no long +1 match, explore the short match we found */
+ mLength = ZSTD_count(ip+4, matchs0+4, iend) + 4;
+ offset = (U32)(ip - matchs0);
+ while (((ip>anchor) & (matchs0>prefixLowest)) && (ip[-1] == matchs0[-1])) { ip--; matchs0--; mLength++; } /* catch up */
+
+ /* fall-through */
+
+_match_found: /* requires ip, offset, mLength */
+ offset_2 = offset_1;
+ offset_1 = offset;
+
+ if (step < 4) {
+ /* It is unsafe to write this value back to the hashtable when ip1 is
+ * greater than or equal to the new ip we will have after we're done
+ * processing this match. Rather than perform that test directly
+ * (ip1 >= ip + mLength), which costs speed in practice, we do a simpler
+ * more predictable test. The minmatch even if we take a short match is
+ * 4 bytes, so as long as step, the distance between ip and ip1
+ * (initially) is less than 4, we know ip1 < new ip. */
+ hashLong[hl1] = (U32)(ip1 - base);
+ }
+
+ ZSTD_storeSeq(seqStore, (size_t)(ip-anchor), anchor, iend, STORE_OFFSET(offset), mLength);
+
+_match_stored:
+ /* match found */
+ ip += mLength;
+ anchor = ip;
+
+ if (ip <= ilimit) {
+ /* Complementary insertion */
+ /* done after iLimit test, as candidates could be > iend-8 */
+ { U32 const indexToInsert = curr+2;
+ hashLong[ZSTD_hashPtr(base+indexToInsert, hBitsL, 8)] = indexToInsert;
+ hashLong[ZSTD_hashPtr(ip-2, hBitsL, 8)] = (U32)(ip-2-base);
+ hashSmall[ZSTD_hashPtr(base+indexToInsert, hBitsS, mls)] = indexToInsert;
+ hashSmall[ZSTD_hashPtr(ip-1, hBitsS, mls)] = (U32)(ip-1-base);
+ }
+
+ /* check immediate repcode */
+ while ( (ip <= ilimit)
+ && ( (offset_2>0)
+ & (MEM_read32(ip) == MEM_read32(ip - offset_2)) )) {
+ /* store sequence */
+ size_t const rLength = ZSTD_count(ip+4, ip+4-offset_2, iend) + 4;
+ U32 const tmpOff = offset_2; offset_2 = offset_1; offset_1 = tmpOff; /* swap offset_2 <=> offset_1 */
+ hashSmall[ZSTD_hashPtr(ip, hBitsS, mls)] = (U32)(ip-base);
+ hashLong[ZSTD_hashPtr(ip, hBitsL, 8)] = (U32)(ip-base);
+ ZSTD_storeSeq(seqStore, 0, anchor, iend, STORE_REPCODE_1, rLength);
+ ip += rLength;
+ anchor = ip;
+ continue; /* faster when present ... (?) */
+ }
+ }
+ }
+}
+
+
+FORCE_INLINE_TEMPLATE
+size_t ZSTD_compressBlock_doubleFast_dictMatchState_generic(
ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM],
void const* src, size_t srcSize,
- U32 const mls /* template */, ZSTD_dictMode_e const dictMode)
+ U32 const mls /* template */)
{
ZSTD_compressionParameters const* cParams = &ms->cParams;
U32* const hashLong = ms->hashTable;
@@ -72,54 +278,30 @@ size_t ZSTD_compressBlock_doubleFast_generic(
U32 offsetSaved = 0;
const ZSTD_matchState_t* const dms = ms->dictMatchState;
- const ZSTD_compressionParameters* const dictCParams =
- dictMode == ZSTD_dictMatchState ?
- &dms->cParams : NULL;
- const U32* const dictHashLong = dictMode == ZSTD_dictMatchState ?
- dms->hashTable : NULL;
- const U32* const dictHashSmall = dictMode == ZSTD_dictMatchState ?
- dms->chainTable : NULL;
- const U32 dictStartIndex = dictMode == ZSTD_dictMatchState ?
- dms->window.dictLimit : 0;
- const BYTE* const dictBase = dictMode == ZSTD_dictMatchState ?
- dms->window.base : NULL;
- const BYTE* const dictStart = dictMode == ZSTD_dictMatchState ?
- dictBase + dictStartIndex : NULL;
- const BYTE* const dictEnd = dictMode == ZSTD_dictMatchState ?
- dms->window.nextSrc : NULL;
- const U32 dictIndexDelta = dictMode == ZSTD_dictMatchState ?
- prefixLowestIndex - (U32)(dictEnd - dictBase) :
- 0;
- const U32 dictHBitsL = dictMode == ZSTD_dictMatchState ?
- dictCParams->hashLog : hBitsL;
- const U32 dictHBitsS = dictMode == ZSTD_dictMatchState ?
- dictCParams->chainLog : hBitsS;
+ const ZSTD_compressionParameters* const dictCParams = &dms->cParams;
+ const U32* const dictHashLong = dms->hashTable;
+ const U32* const dictHashSmall = dms->chainTable;
+ const U32 dictStartIndex = dms->window.dictLimit;
+ const BYTE* const dictBase = dms->window.base;
+ const BYTE* const dictStart = dictBase + dictStartIndex;
+ const BYTE* const dictEnd = dms->window.nextSrc;
+ const U32 dictIndexDelta = prefixLowestIndex - (U32)(dictEnd - dictBase);
+ const U32 dictHBitsL = dictCParams->hashLog;
+ const U32 dictHBitsS = dictCParams->chainLog;
const U32 dictAndPrefixLength = (U32)((ip - prefixLowest) + (dictEnd - dictStart));
- DEBUGLOG(5, "ZSTD_compressBlock_doubleFast_generic");
-
- assert(dictMode == ZSTD_noDict || dictMode == ZSTD_dictMatchState);
+ DEBUGLOG(5, "ZSTD_compressBlock_doubleFast_dictMatchState_generic");
/* if a dictionary is attached, it must be within window range */
- if (dictMode == ZSTD_dictMatchState) {
- assert(ms->window.dictLimit + (1U << cParams->windowLog) >= endIndex);
- }
+ assert(ms->window.dictLimit + (1U << cParams->windowLog) >= endIndex);
/* init */
ip += (dictAndPrefixLength == 0);
- if (dictMode == ZSTD_noDict) {
- U32 const curr = (U32)(ip - base);
- U32 const windowLow = ZSTD_getLowestPrefixIndex(ms, curr, cParams->windowLog);
- U32 const maxRep = curr - windowLow;
- if (offset_2 > maxRep) offsetSaved = offset_2, offset_2 = 0;
- if (offset_1 > maxRep) offsetSaved = offset_1, offset_1 = 0;
- }
- if (dictMode == ZSTD_dictMatchState) {
- /* dictMatchState repCode checks don't currently handle repCode == 0
- * disabling. */
- assert(offset_1 <= dictAndPrefixLength);
- assert(offset_2 <= dictAndPrefixLength);
- }
+
+ /* dictMatchState repCode checks don't currently handle repCode == 0
+ * disabling. */
+ assert(offset_1 <= dictAndPrefixLength);
+ assert(offset_2 <= dictAndPrefixLength);
/* Main Search Loop */
while (ip < ilimit) { /* < instead of <=, because repcode check at (ip+1) */
@@ -135,29 +317,18 @@ size_t ZSTD_compressBlock_doubleFast_generic(
const BYTE* matchLong = base + matchIndexL;
const BYTE* match = base + matchIndexS;
const U32 repIndex = curr + 1 - offset_1;
- const BYTE* repMatch = (dictMode == ZSTD_dictMatchState
- && repIndex < prefixLowestIndex) ?
+ const BYTE* repMatch = (repIndex < prefixLowestIndex) ?
dictBase + (repIndex - dictIndexDelta) :
base + repIndex;
hashLong[h2] = hashSmall[h] = curr; /* update hash tables */
- /* check dictMatchState repcode */
- if (dictMode == ZSTD_dictMatchState
- && ((U32)((prefixLowestIndex-1) - repIndex) >= 3 /* intentional underflow */)
+ /* check repcode */
+ if (((U32)((prefixLowestIndex-1) - repIndex) >= 3 /* intentional underflow */)
&& (MEM_read32(repMatch) == MEM_read32(ip+1)) ) {
const BYTE* repMatchEnd = repIndex < prefixLowestIndex ? dictEnd : iend;
mLength = ZSTD_count_2segments(ip+1+4, repMatch+4, iend, repMatchEnd, prefixLowest) + 4;
ip++;
- ZSTD_storeSeq(seqStore, (size_t)(ip-anchor), anchor, iend, 0, mLength-MINMATCH);
- goto _match_stored;
- }
-
- /* check noDict repcode */
- if ( dictMode == ZSTD_noDict
- && ((offset_1 > 0) & (MEM_read32(ip+1-offset_1) == MEM_read32(ip+1)))) {
- mLength = ZSTD_count(ip+1+4, ip+1+4-offset_1, iend) + 4;
- ip++;
- ZSTD_storeSeq(seqStore, (size_t)(ip-anchor), anchor, iend, 0, mLength-MINMATCH);
+ ZSTD_storeSeq(seqStore, (size_t)(ip-anchor), anchor, iend, STORE_REPCODE_1, mLength);
goto _match_stored;
}
@@ -169,7 +340,7 @@ size_t ZSTD_compressBlock_doubleFast_generic(
while (((ip>anchor) & (matchLong>prefixLowest)) && (ip[-1] == matchLong[-1])) { ip--; matchLong--; mLength++; } /* catch up */
goto _match_found;
}
- } else if (dictMode == ZSTD_dictMatchState) {
+ } else {
/* check dictMatchState long match */
U32 const dictMatchIndexL = dictHashLong[dictHL];
const BYTE* dictMatchL = dictBase + dictMatchIndexL;
@@ -187,7 +358,7 @@ size_t ZSTD_compressBlock_doubleFast_generic(
if (MEM_read32(match) == MEM_read32(ip)) {
goto _search_next_long;
}
- } else if (dictMode == ZSTD_dictMatchState) {
+ } else {
/* check dictMatchState short match */
U32 const dictMatchIndexS = dictHashSmall[dictHS];
match = dictBase + dictMatchIndexS;
@@ -220,7 +391,7 @@ size_t ZSTD_compressBlock_doubleFast_generic(
while (((ip>anchor) & (matchL3>prefixLowest)) && (ip[-1] == matchL3[-1])) { ip--; matchL3--; mLength++; } /* catch up */
goto _match_found;
}
- } else if (dictMode == ZSTD_dictMatchState) {
+ } else {
/* check dict long +1 match */
U32 const dictMatchIndexL3 = dictHashLong[dictHLNext];
const BYTE* dictMatchL3 = dictBase + dictMatchIndexL3;
@@ -234,7 +405,7 @@ size_t ZSTD_compressBlock_doubleFast_generic(
} } }
/* if no long +1 match, explore the short match we found */
- if (dictMode == ZSTD_dictMatchState && matchIndexS < prefixLowestIndex) {
+ if (matchIndexS < prefixLowestIndex) {
mLength = ZSTD_count_2segments(ip+4, match+4, iend, dictEnd, prefixLowest) + 4;
offset = (U32)(curr - matchIndexS);
while (((ip>anchor) & (match>dictStart)) && (ip[-1] == match[-1])) { ip--; match--; mLength++; } /* catch up */
@@ -248,7 +419,7 @@ size_t ZSTD_compressBlock_doubleFast_generic(
offset_2 = offset_1;
offset_1 = offset;
- ZSTD_storeSeq(seqStore, (size_t)(ip-anchor), anchor, iend, offset + ZSTD_REP_MOVE, mLength-MINMATCH);
+ ZSTD_storeSeq(seqStore, (size_t)(ip-anchor), anchor, iend, STORE_OFFSET(offset), mLength);
_match_stored:
/* match found */
@@ -266,43 +437,27 @@ size_t ZSTD_compressBlock_doubleFast_generic(
}
/* check immediate repcode */
- if (dictMode == ZSTD_dictMatchState) {
- while (ip <= ilimit) {
- U32 const current2 = (U32)(ip-base);
- U32 const repIndex2 = current2 - offset_2;
- const BYTE* repMatch2 = dictMode == ZSTD_dictMatchState
- && repIndex2 < prefixLowestIndex ?
- dictBase + repIndex2 - dictIndexDelta :
- base + repIndex2;
- if ( ((U32)((prefixLowestIndex-1) - (U32)repIndex2) >= 3 /* intentional overflow */)
- && (MEM_read32(repMatch2) == MEM_read32(ip)) ) {
- const BYTE* const repEnd2 = repIndex2 < prefixLowestIndex ? dictEnd : iend;
- size_t const repLength2 = ZSTD_count_2segments(ip+4, repMatch2+4, iend, repEnd2, prefixLowest) + 4;
- U32 tmpOffset = offset_2; offset_2 = offset_1; offset_1 = tmpOffset; /* swap offset_2 <=> offset_1 */
- ZSTD_storeSeq(seqStore, 0, anchor, iend, 0, repLength2-MINMATCH);
- hashSmall[ZSTD_hashPtr(ip, hBitsS, mls)] = current2;
- hashLong[ZSTD_hashPtr(ip, hBitsL, 8)] = current2;
- ip += repLength2;
- anchor = ip;
- continue;
- }
- break;
- } }
-
- if (dictMode == ZSTD_noDict) {
- while ( (ip <= ilimit)
- && ( (offset_2>0)
- & (MEM_read32(ip) == MEM_read32(ip - offset_2)) )) {
- /* store sequence */
- size_t const rLength = ZSTD_count(ip+4, ip+4-offset_2, iend) + 4;
- U32 const tmpOff = offset_2; offset_2 = offset_1; offset_1 = tmpOff; /* swap offset_2 <=> offset_1 */
- hashSmall[ZSTD_hashPtr(ip, hBitsS, mls)] = (U32)(ip-base);
- hashLong[ZSTD_hashPtr(ip, hBitsL, 8)] = (U32)(ip-base);
- ZSTD_storeSeq(seqStore, 0, anchor, iend, 0, rLength-MINMATCH);
- ip += rLength;
+ while (ip <= ilimit) {
+ U32 const current2 = (U32)(ip-base);
+ U32 const repIndex2 = current2 - offset_2;
+ const BYTE* repMatch2 = repIndex2 < prefixLowestIndex ?
+ dictBase + repIndex2 - dictIndexDelta :
+ base + repIndex2;
+ if ( ((U32)((prefixLowestIndex-1) - (U32)repIndex2) >= 3 /* intentional overflow */)
+ && (MEM_read32(repMatch2) == MEM_read32(ip)) ) {
+ const BYTE* const repEnd2 = repIndex2 < prefixLowestIndex ? dictEnd : iend;
+ size_t const repLength2 = ZSTD_count_2segments(ip+4, repMatch2+4, iend, repEnd2, prefixLowest) + 4;
+ U32 tmpOffset = offset_2; offset_2 = offset_1; offset_1 = tmpOffset; /* swap offset_2 <=> offset_1 */
+ ZSTD_storeSeq(seqStore, 0, anchor, iend, STORE_REPCODE_1, repLength2);
+ hashSmall[ZSTD_hashPtr(ip, hBitsS, mls)] = current2;
+ hashLong[ZSTD_hashPtr(ip, hBitsL, 8)] = current2;
+ ip += repLength2;
anchor = ip;
- continue; /* faster when present ... (?) */
- } } }
+ continue;
+ }
+ break;
+ }
+ }
} /* while (ip < ilimit) */
/* save reps for next block */
@@ -313,6 +468,24 @@ size_t ZSTD_compressBlock_doubleFast_generic(
return (size_t)(iend - anchor);
}
+#define ZSTD_GEN_DFAST_FN(dictMode, mls) \
+ static size_t ZSTD_compressBlock_doubleFast_##dictMode##_##mls( \
+ ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM], \
+ void const* src, size_t srcSize) \
+ { \
+ return ZSTD_compressBlock_doubleFast_##dictMode##_generic(ms, seqStore, rep, src, srcSize, mls); \
+ }
+
+ZSTD_GEN_DFAST_FN(noDict, 4)
+ZSTD_GEN_DFAST_FN(noDict, 5)
+ZSTD_GEN_DFAST_FN(noDict, 6)
+ZSTD_GEN_DFAST_FN(noDict, 7)
+
+ZSTD_GEN_DFAST_FN(dictMatchState, 4)
+ZSTD_GEN_DFAST_FN(dictMatchState, 5)
+ZSTD_GEN_DFAST_FN(dictMatchState, 6)
+ZSTD_GEN_DFAST_FN(dictMatchState, 7)
+
size_t ZSTD_compressBlock_doubleFast(
ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM],
@@ -323,13 +496,13 @@ size_t ZSTD_compressBlock_doubleFast(
{
default: /* includes case 3 */
case 4 :
- return ZSTD_compressBlock_doubleFast_generic(ms, seqStore, rep, src, srcSize, 4, ZSTD_noDict);
+ return ZSTD_compressBlock_doubleFast_noDict_4(ms, seqStore, rep, src, srcSize);
case 5 :
- return ZSTD_compressBlock_doubleFast_generic(ms, seqStore, rep, src, srcSize, 5, ZSTD_noDict);
+ return ZSTD_compressBlock_doubleFast_noDict_5(ms, seqStore, rep, src, srcSize);
case 6 :
- return ZSTD_compressBlock_doubleFast_generic(ms, seqStore, rep, src, srcSize, 6, ZSTD_noDict);
+ return ZSTD_compressBlock_doubleFast_noDict_6(ms, seqStore, rep, src, srcSize);
case 7 :
- return ZSTD_compressBlock_doubleFast_generic(ms, seqStore, rep, src, srcSize, 7, ZSTD_noDict);
+ return ZSTD_compressBlock_doubleFast_noDict_7(ms, seqStore, rep, src, srcSize);
}
}
@@ -343,13 +516,13 @@ size_t ZSTD_compressBlock_doubleFast_dictMatchState(
{
default: /* includes case 3 */
case 4 :
- return ZSTD_compressBlock_doubleFast_generic(ms, seqStore, rep, src, srcSize, 4, ZSTD_dictMatchState);
+ return ZSTD_compressBlock_doubleFast_dictMatchState_4(ms, seqStore, rep, src, srcSize);
case 5 :
- return ZSTD_compressBlock_doubleFast_generic(ms, seqStore, rep, src, srcSize, 5, ZSTD_dictMatchState);
+ return ZSTD_compressBlock_doubleFast_dictMatchState_5(ms, seqStore, rep, src, srcSize);
case 6 :
- return ZSTD_compressBlock_doubleFast_generic(ms, seqStore, rep, src, srcSize, 6, ZSTD_dictMatchState);
+ return ZSTD_compressBlock_doubleFast_dictMatchState_6(ms, seqStore, rep, src, srcSize);
case 7 :
- return ZSTD_compressBlock_doubleFast_generic(ms, seqStore, rep, src, srcSize, 7, ZSTD_dictMatchState);
+ return ZSTD_compressBlock_doubleFast_dictMatchState_7(ms, seqStore, rep, src, srcSize);
}
}
@@ -385,7 +558,7 @@ static size_t ZSTD_compressBlock_doubleFast_extDict_generic(
/* if extDict is invalidated due to maxDistance, switch to "regular" variant */
if (prefixStartIndex == dictStartIndex)
- return ZSTD_compressBlock_doubleFast_generic(ms, seqStore, rep, src, srcSize, mls, ZSTD_noDict);
+ return ZSTD_compressBlock_doubleFast(ms, seqStore, rep, src, srcSize);
/* Search Loop */
while (ip < ilimit) { /* < instead of <=, because (ip+1) */
@@ -407,12 +580,12 @@ static size_t ZSTD_compressBlock_doubleFast_extDict_generic(
hashSmall[hSmall] = hashLong[hLong] = curr; /* update hash table */
if ((((U32)((prefixStartIndex-1) - repIndex) >= 3) /* intentional underflow : ensure repIndex doesn't overlap dict + prefix */
- & (repIndex > dictStartIndex))
+ & (offset_1 <= curr+1 - dictStartIndex)) /* note: we are searching at curr+1 */
&& (MEM_read32(repMatch) == MEM_read32(ip+1)) ) {
const BYTE* repMatchEnd = repIndex < prefixStartIndex ? dictEnd : iend;
mLength = ZSTD_count_2segments(ip+1+4, repMatch+4, iend, repMatchEnd, prefixStart) + 4;
ip++;
- ZSTD_storeSeq(seqStore, (size_t)(ip-anchor), anchor, iend, 0, mLength-MINMATCH);
+ ZSTD_storeSeq(seqStore, (size_t)(ip-anchor), anchor, iend, STORE_REPCODE_1, mLength);
} else {
if ((matchLongIndex > dictStartIndex) && (MEM_read64(matchLong) == MEM_read64(ip))) {
const BYTE* const matchEnd = matchLongIndex < prefixStartIndex ? dictEnd : iend;
@@ -423,7 +596,7 @@ static size_t ZSTD_compressBlock_doubleFast_extDict_generic(
while (((ip>anchor) & (matchLong>lowMatchPtr)) && (ip[-1] == matchLong[-1])) { ip--; matchLong--; mLength++; } /* catch up */
offset_2 = offset_1;
offset_1 = offset;
- ZSTD_storeSeq(seqStore, (size_t)(ip-anchor), anchor, iend, offset + ZSTD_REP_MOVE, mLength-MINMATCH);
+ ZSTD_storeSeq(seqStore, (size_t)(ip-anchor), anchor, iend, STORE_OFFSET(offset), mLength);
} else if ((matchIndex > dictStartIndex) && (MEM_read32(match) == MEM_read32(ip))) {
size_t const h3 = ZSTD_hashPtr(ip+1, hBitsL, 8);
@@ -448,7 +621,7 @@ static size_t ZSTD_compressBlock_doubleFast_extDict_generic(
}
offset_2 = offset_1;
offset_1 = offset;
- ZSTD_storeSeq(seqStore, (size_t)(ip-anchor), anchor, iend, offset + ZSTD_REP_MOVE, mLength-MINMATCH);
+ ZSTD_storeSeq(seqStore, (size_t)(ip-anchor), anchor, iend, STORE_OFFSET(offset), mLength);
} else {
ip += ((ip-anchor) >> kSearchStrength) + 1;
@@ -475,12 +648,12 @@ static size_t ZSTD_compressBlock_doubleFast_extDict_generic(
U32 const repIndex2 = current2 - offset_2;
const BYTE* repMatch2 = repIndex2 < prefixStartIndex ? dictBase + repIndex2 : base + repIndex2;
if ( (((U32)((prefixStartIndex-1) - repIndex2) >= 3) /* intentional overflow : ensure repIndex2 doesn't overlap dict + prefix */
- & (repIndex2 > dictStartIndex))
+ & (offset_2 <= current2 - dictStartIndex))
&& (MEM_read32(repMatch2) == MEM_read32(ip)) ) {
const BYTE* const repEnd2 = repIndex2 < prefixStartIndex ? dictEnd : iend;
size_t const repLength2 = ZSTD_count_2segments(ip+4, repMatch2+4, iend, repEnd2, prefixStart) + 4;
U32 const tmpOffset = offset_2; offset_2 = offset_1; offset_1 = tmpOffset; /* swap offset_2 <=> offset_1 */
- ZSTD_storeSeq(seqStore, 0, anchor, iend, 0, repLength2-MINMATCH);
+ ZSTD_storeSeq(seqStore, 0, anchor, iend, STORE_REPCODE_1, repLength2);
hashSmall[ZSTD_hashPtr(ip, hBitsS, mls)] = current2;
hashLong[ZSTD_hashPtr(ip, hBitsL, 8)] = current2;
ip += repLength2;
@@ -498,6 +671,10 @@ static size_t ZSTD_compressBlock_doubleFast_extDict_generic(
return (size_t)(iend - anchor);
}
+ZSTD_GEN_DFAST_FN(extDict, 4)
+ZSTD_GEN_DFAST_FN(extDict, 5)
+ZSTD_GEN_DFAST_FN(extDict, 6)
+ZSTD_GEN_DFAST_FN(extDict, 7)
size_t ZSTD_compressBlock_doubleFast_extDict(
ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM],
@@ -508,12 +685,12 @@ size_t ZSTD_compressBlock_doubleFast_extDict(
{
default: /* includes case 3 */
case 4 :
- return ZSTD_compressBlock_doubleFast_extDict_generic(ms, seqStore, rep, src, srcSize, 4);
+ return ZSTD_compressBlock_doubleFast_extDict_4(ms, seqStore, rep, src, srcSize);
case 5 :
- return ZSTD_compressBlock_doubleFast_extDict_generic(ms, seqStore, rep, src, srcSize, 5);
+ return ZSTD_compressBlock_doubleFast_extDict_5(ms, seqStore, rep, src, srcSize);
case 6 :
- return ZSTD_compressBlock_doubleFast_extDict_generic(ms, seqStore, rep, src, srcSize, 6);
+ return ZSTD_compressBlock_doubleFast_extDict_6(ms, seqStore, rep, src, srcSize);
case 7 :
- return ZSTD_compressBlock_doubleFast_extDict_generic(ms, seqStore, rep, src, srcSize, 7);
+ return ZSTD_compressBlock_doubleFast_extDict_7(ms, seqStore, rep, src, srcSize);
}
}
diff --git a/lib/zstd/compress/zstd_fast.c b/lib/zstd/compress/zstd_fast.c
index 96b7d48..a752e6b 100644
--- a/lib/zstd/compress/zstd_fast.c
+++ b/lib/zstd/compress/zstd_fast.c
@@ -43,145 +43,294 @@ void ZSTD_fillHashTable(ZSTD_matchState_t* ms,
}
+/*
+ * If you squint hard enough (and ignore repcodes), the search operation at any
+ * given position is broken into 4 stages:
+ *
+ * 1. Hash (map position to hash value via input read)
+ * 2. Lookup (map hash val to index via hashtable read)
+ * 3. Load (map index to value at that position via input read)
+ * 4. Compare
+ *
+ * Each of these steps involves a memory read at an address which is computed
+ * from the previous step. This means these steps must be sequenced and their
+ * latencies are cumulative.
+ *
+ * Rather than do 1->2->3->4 sequentially for a single position before moving
+ * onto the next, this implementation interleaves these operations across the
+ * next few positions:
+ *
+ * R = Repcode Read & Compare
+ * H = Hash
+ * T = Table Lookup
+ * M = Match Read & Compare
+ *
+ * Pos | Time -->
+ * ----+-------------------
+ * N | ... M
+ * N+1 | ... TM
+ * N+2 | R H T M
+ * N+3 | H TM
+ * N+4 | R H T M
+ * N+5 | H ...
+ * N+6 | R ...
+ *
+ * This is very much analogous to the pipelining of execution in a CPU. And just
+ * like a CPU, we have to dump the pipeline when we find a match (i.e., take a
+ * branch).
+ *
+ * When this happens, we throw away our current state, and do the following prep
+ * to re-enter the loop:
+ *
+ * Pos | Time -->
+ * ----+-------------------
+ * N | H T
+ * N+1 | H
+ *
+ * This is also the work we do at the beginning to enter the loop initially.
+ */
FORCE_INLINE_TEMPLATE size_t
-ZSTD_compressBlock_fast_generic(
+ZSTD_compressBlock_fast_noDict_generic(
ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM],
void const* src, size_t srcSize,
- U32 const mls)
+ U32 const mls, U32 const hasStep)
{
const ZSTD_compressionParameters* const cParams = &ms->cParams;
U32* const hashTable = ms->hashTable;
U32 const hlog = cParams->hashLog;
/* support stepSize of 0 */
- size_t const stepSize = cParams->targetLength + !(cParams->targetLength) + 1;
+ size_t const stepSize = hasStep ? (cParams->targetLength + !(cParams->targetLength) + 1) : 2;
const BYTE* const base = ms->window.base;
const BYTE* const istart = (const BYTE*)src;
- /* We check ip0 (ip + 0) and ip1 (ip + 1) each loop */
- const BYTE* ip0 = istart;
- const BYTE* ip1;
- const BYTE* anchor = istart;
const U32 endIndex = (U32)((size_t)(istart - base) + srcSize);
const U32 prefixStartIndex = ZSTD_getLowestPrefixIndex(ms, endIndex, cParams->windowLog);
const BYTE* const prefixStart = base + prefixStartIndex;
const BYTE* const iend = istart + srcSize;
const BYTE* const ilimit = iend - HASH_READ_SIZE;
- U32 offset_1=rep[0], offset_2=rep[1];
+
+ const BYTE* anchor = istart;
+ const BYTE* ip0 = istart;
+ const BYTE* ip1;
+ const BYTE* ip2;
+ const BYTE* ip3;
+ U32 current0;
+
+ U32 rep_offset1 = rep[0];
+ U32 rep_offset2 = rep[1];
U32 offsetSaved = 0;
- /* init */
+ size_t hash0; /* hash for ip0 */
+ size_t hash1; /* hash for ip1 */
+ U32 idx; /* match idx for ip0 */
+ U32 mval; /* src value at match idx */
+
+ U32 offcode;
+ const BYTE* match0;
+ size_t mLength;
+
+ /* ip0 and ip1 are always adjacent. The targetLength skipping and
+ * uncompressibility acceleration is applied to every other position,
+ * matching the behavior of #1562. step therefore represents the gap
+ * between pairs of positions, from ip0 to ip2 or ip1 to ip3. */
+ size_t step;
+ const BYTE* nextStep;
+ const size_t kStepIncr = (1 << (kSearchStrength - 1));
+
DEBUGLOG(5, "ZSTD_compressBlock_fast_generic");
ip0 += (ip0 == prefixStart);
- ip1 = ip0 + 1;
{ U32 const curr = (U32)(ip0 - base);
U32 const windowLow = ZSTD_getLowestPrefixIndex(ms, curr, cParams->windowLog);
U32 const maxRep = curr - windowLow;
- if (offset_2 > maxRep) offsetSaved = offset_2, offset_2 = 0;
- if (offset_1 > maxRep) offsetSaved = offset_1, offset_1 = 0;
+ if (rep_offset2 > maxRep) offsetSaved = rep_offset2, rep_offset2 = 0;
+ if (rep_offset1 > maxRep) offsetSaved = rep_offset1, rep_offset1 = 0;
}
- /* Main Search Loop */
-#ifdef __INTEL_COMPILER
- /* From intel 'The vector pragma indicates that the loop should be
- * vectorized if it is legal to do so'. Can be used together with
- * #pragma ivdep (but have opted to exclude that because intel
- * warns against using it).*/
- #pragma vector always
-#endif
- while (ip1 < ilimit) { /* < instead of <=, because check at ip0+2 */
- size_t mLength;
- BYTE const* ip2 = ip0 + 2;
- size_t const h0 = ZSTD_hashPtr(ip0, hlog, mls);
- U32 const val0 = MEM_read32(ip0);
- size_t const h1 = ZSTD_hashPtr(ip1, hlog, mls);
- U32 const val1 = MEM_read32(ip1);
- U32 const current0 = (U32)(ip0-base);
- U32 const current1 = (U32)(ip1-base);
- U32 const matchIndex0 = hashTable[h0];
- U32 const matchIndex1 = hashTable[h1];
- BYTE const* repMatch = ip2 - offset_1;
- const BYTE* match0 = base + matchIndex0;
- const BYTE* match1 = base + matchIndex1;
- U32 offcode;
+ /* start each op */
+_start: /* Requires: ip0 */
-#if defined(__aarch64__)
- PREFETCH_L1(ip0+256);
-#endif
+ step = stepSize;
+ nextStep = ip0 + kStepIncr;
- hashTable[h0] = current0; /* update hash table */
- hashTable[h1] = current1; /* update hash table */
+ /* calculate positions, ip0 - anchor == 0, so we skip step calc */
+ ip1 = ip0 + 1;
+ ip2 = ip0 + step;
+ ip3 = ip2 + 1;
- assert(ip0 + 1 == ip1);
+ if (ip3 >= ilimit) {
+ goto _cleanup;
+ }
- if ((offset_1 > 0) & (MEM_read32(repMatch) == MEM_read32(ip2))) {
- mLength = (ip2[-1] == repMatch[-1]) ? 1 : 0;
- ip0 = ip2 - mLength;
- match0 = repMatch - mLength;
+ hash0 = ZSTD_hashPtr(ip0, hlog, mls);
+ hash1 = ZSTD_hashPtr(ip1, hlog, mls);
+
+ idx = hashTable[hash0];
+
+ do {
+ /* load repcode match for ip[2]*/
+ const U32 rval = MEM_read32(ip2 - rep_offset1);
+
+ /* write back hash table entry */
+ current0 = (U32)(ip0 - base);
+ hashTable[hash0] = current0;
+
+ /* check repcode at ip[2] */
+ if ((MEM_read32(ip2) == rval) & (rep_offset1 > 0)) {
+ ip0 = ip2;
+ match0 = ip0 - rep_offset1;
+ mLength = ip0[-1] == match0[-1];
+ ip0 -= mLength;
+ match0 -= mLength;
+ offcode = STORE_REPCODE_1;
mLength += 4;
- offcode = 0;
goto _match;
}
- if ((matchIndex0 > prefixStartIndex) && MEM_read32(match0) == val0) {
- /* found a regular match */
+
+ /* load match for ip[0] */
+ if (idx >= prefixStartIndex) {
+ mval = MEM_read32(base + idx);
+ } else {
+ mval = MEM_read32(ip0) ^ 1; /* guaranteed to not match. */
+ }
+
+ /* check match at ip[0] */
+ if (MEM_read32(ip0) == mval) {
+ /* found a match! */
goto _offset;
}
- if ((matchIndex1 > prefixStartIndex) && MEM_read32(match1) == val1) {
- /* found a regular match after one literal */
- ip0 = ip1;
- match0 = match1;
+
+ /* lookup ip[1] */
+ idx = hashTable[hash1];
+
+ /* hash ip[2] */
+ hash0 = hash1;
+ hash1 = ZSTD_hashPtr(ip2, hlog, mls);
+
+ /* advance to next positions */
+ ip0 = ip1;
+ ip1 = ip2;
+ ip2 = ip3;
+
+ /* write back hash table entry */
+ current0 = (U32)(ip0 - base);
+ hashTable[hash0] = current0;
+
+ /* load match for ip[0] */
+ if (idx >= prefixStartIndex) {
+ mval = MEM_read32(base + idx);
+ } else {
+ mval = MEM_read32(ip0) ^ 1; /* guaranteed to not match. */
+ }
+
+ /* check match at ip[0] */
+ if (MEM_read32(ip0) == mval) {
+ /* found a match! */
goto _offset;
}
- { size_t const step = ((size_t)(ip0-anchor) >> (kSearchStrength - 1)) + stepSize;
- assert(step >= 2);
- ip0 += step;
- ip1 += step;
- continue;
+
+ /* lookup ip[1] */
+ idx = hashTable[hash1];
+
+ /* hash ip[2] */
+ hash0 = hash1;
+ hash1 = ZSTD_hashPtr(ip2, hlog, mls);
+
+ /* advance to next positions */
+ ip0 = ip1;
+ ip1 = ip2;
+ ip2 = ip0 + step;
+ ip3 = ip1 + step;
+
+ /* calculate step */
+ if (ip2 >= nextStep) {
+ step++;
+ PREFETCH_L1(ip1 + 64);
+ PREFETCH_L1(ip1 + 128);
+ nextStep += kStepIncr;
}
-_offset: /* Requires: ip0, match0 */
- /* Compute the offset code */
- offset_2 = offset_1;
- offset_1 = (U32)(ip0-match0);
- offcode = offset_1 + ZSTD_REP_MOVE;
- mLength = 4;
- /* Count the backwards match length */
- while (((ip0>anchor) & (match0>prefixStart))
- && (ip0[-1] == match0[-1])) { ip0--; match0--; mLength++; } /* catch up */
+ } while (ip3 < ilimit);
-_match: /* Requires: ip0, match0, offcode */
- /* Count the forward length */
- mLength += ZSTD_count(ip0+mLength, match0+mLength, iend);
- ZSTD_storeSeq(seqStore, (size_t)(ip0-anchor), anchor, iend, offcode, mLength-MINMATCH);
- /* match found */
- ip0 += mLength;
- anchor = ip0;
-
- if (ip0 <= ilimit) {
- /* Fill Table */
- assert(base+current0+2 > istart); /* check base overflow */
- hashTable[ZSTD_hashPtr(base+current0+2, hlog, mls)] = current0+2; /* here because current+2 could be > iend-8 */
- hashTable[ZSTD_hashPtr(ip0-2, hlog, mls)] = (U32)(ip0-2-base);
-
- if (offset_2 > 0) { /* offset_2==0 means offset_2 is invalidated */
- while ( (ip0 <= ilimit) && (MEM_read32(ip0) == MEM_read32(ip0 - offset_2)) ) {
- /* store sequence */
- size_t const rLength = ZSTD_count(ip0+4, ip0+4-offset_2, iend) + 4;
- { U32 const tmpOff = offset_2; offset_2 = offset_1; offset_1 = tmpOff; } /* swap offset_2 <=> offset_1 */
- hashTable[ZSTD_hashPtr(ip0, hlog, mls)] = (U32)(ip0-base);
- ip0 += rLength;
- ZSTD_storeSeq(seqStore, 0 /*litLen*/, anchor, iend, 0 /*offCode*/, rLength-MINMATCH);
- anchor = ip0;
- continue; /* faster when present (confirmed on gcc-8) ... (?) */
- } } }
- ip1 = ip0 + 1;
- }
+_cleanup:
+ /* Note that there are probably still a couple positions we could search.
+ * However, it seems to be a meaningful performance hit to try to search
+ * them. So let's not. */
/* save reps for next block */
- rep[0] = offset_1 ? offset_1 : offsetSaved;
- rep[1] = offset_2 ? offset_2 : offsetSaved;
+ rep[0] = rep_offset1 ? rep_offset1 : offsetSaved;
+ rep[1] = rep_offset2 ? rep_offset2 : offsetSaved;
/* Return the last literals size */
return (size_t)(iend - anchor);
+
+_offset: /* Requires: ip0, idx */
+
+ /* Compute the offset code. */
+ match0 = base + idx;
+ rep_offset2 = rep_offset1;
+ rep_offset1 = (U32)(ip0-match0);
+ offcode = STORE_OFFSET(rep_offset1);
+ mLength = 4;
+
+ /* Count the backwards match length. */
+ while (((ip0>anchor) & (match0>prefixStart)) && (ip0[-1] == match0[-1])) {
+ ip0--;
+ match0--;
+ mLength++;
+ }
+
+_match: /* Requires: ip0, match0, offcode */
+
+ /* Count the forward length. */
+ mLength += ZSTD_count(ip0 + mLength, match0 + mLength, iend);
+
+ ZSTD_storeSeq(seqStore, (size_t)(ip0 - anchor), anchor, iend, offcode, mLength);
+
+ ip0 += mLength;
+ anchor = ip0;
+
+ /* write next hash table entry */
+ if (ip1 < ip0) {
+ hashTable[hash1] = (U32)(ip1 - base);
+ }
+
+ /* Fill table and check for immediate repcode. */
+ if (ip0 <= ilimit) {
+ /* Fill Table */
+ assert(base+current0+2 > istart); /* check base overflow */
+ hashTable[ZSTD_hashPtr(base+current0+2, hlog, mls)] = current0+2; /* here because current+2 could be > iend-8 */
+ hashTable[ZSTD_hashPtr(ip0-2, hlog, mls)] = (U32)(ip0-2-base);
+
+ if (rep_offset2 > 0) { /* rep_offset2==0 means rep_offset2 is invalidated */
+ while ( (ip0 <= ilimit) && (MEM_read32(ip0) == MEM_read32(ip0 - rep_offset2)) ) {
+ /* store sequence */
+ size_t const rLength = ZSTD_count(ip0+4, ip0+4-rep_offset2, iend) + 4;
+ { U32 const tmpOff = rep_offset2; rep_offset2 = rep_offset1; rep_offset1 = tmpOff; } /* swap rep_offset2 <=> rep_offset1 */
+ hashTable[ZSTD_hashPtr(ip0, hlog, mls)] = (U32)(ip0-base);
+ ip0 += rLength;
+ ZSTD_storeSeq(seqStore, 0 /*litLen*/, anchor, iend, STORE_REPCODE_1, rLength);
+ anchor = ip0;
+ continue; /* faster when present (confirmed on gcc-8) ... (?) */
+ } } }
+
+ goto _start;
}
+#define ZSTD_GEN_FAST_FN(dictMode, mls, step) \
+ static size_t ZSTD_compressBlock_fast_##dictMode##_##mls##_##step( \
+ ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM], \
+ void const* src, size_t srcSize) \
+ { \
+ return ZSTD_compressBlock_fast_##dictMode##_generic(ms, seqStore, rep, src, srcSize, mls, step); \
+ }
+
+ZSTD_GEN_FAST_FN(noDict, 4, 1)
+ZSTD_GEN_FAST_FN(noDict, 5, 1)
+ZSTD_GEN_FAST_FN(noDict, 6, 1)
+ZSTD_GEN_FAST_FN(noDict, 7, 1)
+
+ZSTD_GEN_FAST_FN(noDict, 4, 0)
+ZSTD_GEN_FAST_FN(noDict, 5, 0)
+ZSTD_GEN_FAST_FN(noDict, 6, 0)
+ZSTD_GEN_FAST_FN(noDict, 7, 0)
size_t ZSTD_compressBlock_fast(
ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM],
@@ -189,24 +338,40 @@ size_t ZSTD_compressBlock_fast(
{
U32 const mls = ms->cParams.minMatch;
assert(ms->dictMatchState == NULL);
- switch(mls)
- {
- default: /* includes case 3 */
- case 4 :
- return ZSTD_compressBlock_fast_generic(ms, seqStore, rep, src, srcSize, 4);
- case 5 :
- return ZSTD_compressBlock_fast_generic(ms, seqStore, rep, src, srcSize, 5);
- case 6 :
- return ZSTD_compressBlock_fast_generic(ms, seqStore, rep, src, srcSize, 6);
- case 7 :
- return ZSTD_compressBlock_fast_generic(ms, seqStore, rep, src, srcSize, 7);
+ if (ms->cParams.targetLength > 1) {
+ switch(mls)
+ {
+ default: /* includes case 3 */
+ case 4 :
+ return ZSTD_compressBlock_fast_noDict_4_1(ms, seqStore, rep, src, srcSize);
+ case 5 :
+ return ZSTD_compressBlock_fast_noDict_5_1(ms, seqStore, rep, src, srcSize);
+ case 6 :
+ return ZSTD_compressBlock_fast_noDict_6_1(ms, seqStore, rep, src, srcSize);
+ case 7 :
+ return ZSTD_compressBlock_fast_noDict_7_1(ms, seqStore, rep, src, srcSize);
+ }
+ } else {
+ switch(mls)
+ {
+ default: /* includes case 3 */
+ case 4 :
+ return ZSTD_compressBlock_fast_noDict_4_0(ms, seqStore, rep, src, srcSize);
+ case 5 :
+ return ZSTD_compressBlock_fast_noDict_5_0(ms, seqStore, rep, src, srcSize);
+ case 6 :
+ return ZSTD_compressBlock_fast_noDict_6_0(ms, seqStore, rep, src, srcSize);
+ case 7 :
+ return ZSTD_compressBlock_fast_noDict_7_0(ms, seqStore, rep, src, srcSize);
+ }
+
}
}
FORCE_INLINE_TEMPLATE
size_t ZSTD_compressBlock_fast_dictMatchState_generic(
ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM],
- void const* src, size_t srcSize, U32 const mls)
+ void const* src, size_t srcSize, U32 const mls, U32 const hasStep)
{
const ZSTD_compressionParameters* const cParams = &ms->cParams;
U32* const hashTable = ms->hashTable;
@@ -242,6 +407,8 @@ size_t ZSTD_compressBlock_fast_dictMatchState_generic(
assert(endIndex - prefixStartIndex <= maxDistance);
(void)maxDistance; (void)endIndex; /* these variables are not used when assert() is disabled */
+ (void)hasStep; /* not currently specialized on whether it's accelerated */
+
/* ensure there will be no underflow
* when translating a dict index into a local index */
assert(prefixStartIndex >= (U32)(dictEnd - dictBase));
@@ -272,7 +439,7 @@ size_t ZSTD_compressBlock_fast_dictMatchState_generic(
const BYTE* const repMatchEnd = repIndex < prefixStartIndex ? dictEnd : iend;
mLength = ZSTD_count_2segments(ip+1+4, repMatch+4, iend, repMatchEnd, prefixStart) + 4;
ip++;
- ZSTD_storeSeq(seqStore, (size_t)(ip-anchor), anchor, iend, 0, mLength-MINMATCH);
+ ZSTD_storeSeq(seqStore, (size_t)(ip-anchor), anchor, iend, STORE_REPCODE_1, mLength);
} else if ( (matchIndex <= prefixStartIndex) ) {
size_t const dictHash = ZSTD_hashPtr(ip, dictHLog, mls);
U32 const dictMatchIndex = dictHashTable[dictHash];
@@ -292,7 +459,7 @@ size_t ZSTD_compressBlock_fast_dictMatchState_generic(
} /* catch up */
offset_2 = offset_1;
offset_1 = offset;
- ZSTD_storeSeq(seqStore, (size_t)(ip-anchor), anchor, iend, offset + ZSTD_REP_MOVE, mLength-MINMATCH);
+ ZSTD_storeSeq(seqStore, (size_t)(ip-anchor), anchor, iend, STORE_OFFSET(offset), mLength);
}
} else if (MEM_read32(match) != MEM_read32(ip)) {
/* it's not a match, and we're not going to check the dictionary */
@@ -307,7 +474,7 @@ size_t ZSTD_compressBlock_fast_dictMatchState_generic(
&& (ip[-1] == match[-1])) { ip--; match--; mLength++; } /* catch up */
offset_2 = offset_1;
offset_1 = offset;
- ZSTD_storeSeq(seqStore, (size_t)(ip-anchor), anchor, iend, offset + ZSTD_REP_MOVE, mLength-MINMATCH);
+ ZSTD_storeSeq(seqStore, (size_t)(ip-anchor), anchor, iend, STORE_OFFSET(offset), mLength);
}
/* match found */
@@ -332,7 +499,7 @@ size_t ZSTD_compressBlock_fast_dictMatchState_generic(
const BYTE* const repEnd2 = repIndex2 < prefixStartIndex ? dictEnd : iend;
size_t const repLength2 = ZSTD_count_2segments(ip+4, repMatch2+4, iend, repEnd2, prefixStart) + 4;
U32 tmpOffset = offset_2; offset_2 = offset_1; offset_1 = tmpOffset; /* swap offset_2 <=> offset_1 */
- ZSTD_storeSeq(seqStore, 0, anchor, iend, 0, repLength2-MINMATCH);
+ ZSTD_storeSeq(seqStore, 0, anchor, iend, STORE_REPCODE_1, repLength2);
hashTable[ZSTD_hashPtr(ip, hlog, mls)] = current2;
ip += repLength2;
anchor = ip;
@@ -351,6 +518,12 @@ size_t ZSTD_compressBlock_fast_dictMatchState_generic(
return (size_t)(iend - anchor);
}
+
+ZSTD_GEN_FAST_FN(dictMatchState, 4, 0)
+ZSTD_GEN_FAST_FN(dictMatchState, 5, 0)
+ZSTD_GEN_FAST_FN(dictMatchState, 6, 0)
+ZSTD_GEN_FAST_FN(dictMatchState, 7, 0)
+
size_t ZSTD_compressBlock_fast_dictMatchState(
ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM],
void const* src, size_t srcSize)
@@ -361,20 +534,20 @@ size_t ZSTD_compressBlock_fast_dictMatchState(
{
default: /* includes case 3 */
case 4 :
- return ZSTD_compressBlock_fast_dictMatchState_generic(ms, seqStore, rep, src, srcSize, 4);
+ return ZSTD_compressBlock_fast_dictMatchState_4_0(ms, seqStore, rep, src, srcSize);
case 5 :
- return ZSTD_compressBlock_fast_dictMatchState_generic(ms, seqStore, rep, src, srcSize, 5);
+ return ZSTD_compressBlock_fast_dictMatchState_5_0(ms, seqStore, rep, src, srcSize);
case 6 :
- return ZSTD_compressBlock_fast_dictMatchState_generic(ms, seqStore, rep, src, srcSize, 6);
+ return ZSTD_compressBlock_fast_dictMatchState_6_0(ms, seqStore, rep, src, srcSize);
case 7 :
- return ZSTD_compressBlock_fast_dictMatchState_generic(ms, seqStore, rep, src, srcSize, 7);
+ return ZSTD_compressBlock_fast_dictMatchState_7_0(ms, seqStore, rep, src, srcSize);
}
}
static size_t ZSTD_compressBlock_fast_extDict_generic(
ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM],
- void const* src, size_t srcSize, U32 const mls)
+ void const* src, size_t srcSize, U32 const mls, U32 const hasStep)
{
const ZSTD_compressionParameters* const cParams = &ms->cParams;
U32* const hashTable = ms->hashTable;
@@ -398,11 +571,13 @@ static size_t ZSTD_compressBlock_fast_extDict_generic(
const BYTE* const ilimit = iend - 8;
U32 offset_1=rep[0], offset_2=rep[1];
+ (void)hasStep; /* not currently specialized on whether it's accelerated */
+
DEBUGLOG(5, "ZSTD_compressBlock_fast_extDict_generic (offset_1=%u)", offset_1);
/* switch to "regular" variant if extDict is invalidated due to maxDistance */
if (prefixStartIndex == dictStartIndex)
- return ZSTD_compressBlock_fast_generic(ms, seqStore, rep, src, srcSize, mls);
+ return ZSTD_compressBlock_fast(ms, seqStore, rep, src, srcSize);
/* Search Loop */
while (ip < ilimit) { /* < instead of <=, because (ip+1) */
@@ -416,14 +591,14 @@ static size_t ZSTD_compressBlock_fast_extDict_generic(
const BYTE* const repMatch = repBase + repIndex;
hashTable[h] = curr; /* update hash table */
DEBUGLOG(7, "offset_1 = %u , curr = %u", offset_1, curr);
- assert(offset_1 <= curr +1); /* check repIndex */
- if ( (((U32)((prefixStartIndex-1) - repIndex) >= 3) /* intentional underflow */ & (repIndex > dictStartIndex))
+ if ( ( ((U32)((prefixStartIndex-1) - repIndex) >= 3) /* intentional underflow */
+ & (offset_1 <= curr+1 - dictStartIndex) ) /* note: we are searching at curr+1 */
&& (MEM_read32(repMatch) == MEM_read32(ip+1)) ) {
const BYTE* const repMatchEnd = repIndex < prefixStartIndex ? dictEnd : iend;
size_t const rLength = ZSTD_count_2segments(ip+1 +4, repMatch +4, iend, repMatchEnd, prefixStart) + 4;
ip++;
- ZSTD_storeSeq(seqStore, (size_t)(ip-anchor), anchor, iend, 0, rLength-MINMATCH);
+ ZSTD_storeSeq(seqStore, (size_t)(ip-anchor), anchor, iend, STORE_REPCODE_1, rLength);
ip += rLength;
anchor = ip;
} else {
@@ -439,7 +614,7 @@ static size_t ZSTD_compressBlock_fast_extDict_generic(
size_t mLength = ZSTD_count_2segments(ip+4, match+4, iend, matchEnd, prefixStart) + 4;
while (((ip>anchor) & (match>lowMatchPtr)) && (ip[-1] == match[-1])) { ip--; match--; mLength++; } /* catch up */
offset_2 = offset_1; offset_1 = offset; /* update offset history */
- ZSTD_storeSeq(seqStore, (size_t)(ip-anchor), anchor, iend, offset + ZSTD_REP_MOVE, mLength-MINMATCH);
+ ZSTD_storeSeq(seqStore, (size_t)(ip-anchor), anchor, iend, STORE_OFFSET(offset), mLength);
ip += mLength;
anchor = ip;
} }
@@ -453,12 +628,12 @@ static size_t ZSTD_compressBlock_fast_extDict_generic(
U32 const current2 = (U32)(ip-base);
U32 const repIndex2 = current2 - offset_2;
const BYTE* const repMatch2 = repIndex2 < prefixStartIndex ? dictBase + repIndex2 : base + repIndex2;
- if ( (((U32)((prefixStartIndex-1) - repIndex2) >= 3) & (repIndex2 > dictStartIndex)) /* intentional overflow */
+ if ( (((U32)((prefixStartIndex-1) - repIndex2) >= 3) & (offset_2 <= curr - dictStartIndex)) /* intentional overflow */
&& (MEM_read32(repMatch2) == MEM_read32(ip)) ) {
const BYTE* const repEnd2 = repIndex2 < prefixStartIndex ? dictEnd : iend;
size_t const repLength2 = ZSTD_count_2segments(ip+4, repMatch2+4, iend, repEnd2, prefixStart) + 4;
{ U32 const tmpOffset = offset_2; offset_2 = offset_1; offset_1 = tmpOffset; } /* swap offset_2 <=> offset_1 */
- ZSTD_storeSeq(seqStore, 0 /*litlen*/, anchor, iend, 0 /*offcode*/, repLength2-MINMATCH);
+ ZSTD_storeSeq(seqStore, 0 /*litlen*/, anchor, iend, STORE_REPCODE_1, repLength2);
hashTable[ZSTD_hashPtr(ip, hlog, mls)] = current2;
ip += repLength2;
anchor = ip;
@@ -475,6 +650,10 @@ static size_t ZSTD_compressBlock_fast_extDict_generic(
return (size_t)(iend - anchor);
}
+ZSTD_GEN_FAST_FN(extDict, 4, 0)
+ZSTD_GEN_FAST_FN(extDict, 5, 0)
+ZSTD_GEN_FAST_FN(extDict, 6, 0)
+ZSTD_GEN_FAST_FN(extDict, 7, 0)
size_t ZSTD_compressBlock_fast_extDict(
ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM],
@@ -485,12 +664,12 @@ size_t ZSTD_compressBlock_fast_extDict(
{
default: /* includes case 3 */
case 4 :
- return ZSTD_compressBlock_fast_extDict_generic(ms, seqStore, rep, src, srcSize, 4);
+ return ZSTD_compressBlock_fast_extDict_4_0(ms, seqStore, rep, src, srcSize);
case 5 :
- return ZSTD_compressBlock_fast_extDict_generic(ms, seqStore, rep, src, srcSize, 5);
+ return ZSTD_compressBlock_fast_extDict_5_0(ms, seqStore, rep, src, srcSize);
case 6 :
- return ZSTD_compressBlock_fast_extDict_generic(ms, seqStore, rep, src, srcSize, 6);
+ return ZSTD_compressBlock_fast_extDict_6_0(ms, seqStore, rep, src, srcSize);
case 7 :
- return ZSTD_compressBlock_fast_extDict_generic(ms, seqStore, rep, src, srcSize, 7);
+ return ZSTD_compressBlock_fast_extDict_7_0(ms, seqStore, rep, src, srcSize);
}
}
diff --git a/lib/zstd/compress/zstd_lazy.c b/lib/zstd/compress/zstd_lazy.c
index fb54d4e..0298a01 100644
--- a/lib/zstd/compress/zstd_lazy.c
+++ b/lib/zstd/compress/zstd_lazy.c
@@ -61,7 +61,7 @@ ZSTD_updateDUBT(ZSTD_matchState_t* ms,
* assumption : curr >= btlow == (curr - btmask)
* doesn't fail */
static void
-ZSTD_insertDUBT1(ZSTD_matchState_t* ms,
+ZSTD_insertDUBT1(const ZSTD_matchState_t* ms,
U32 curr, const BYTE* inputEnd,
U32 nbCompares, U32 btLow,
const ZSTD_dictMode_e dictMode)
@@ -151,7 +151,7 @@ ZSTD_insertDUBT1(ZSTD_matchState_t* ms,
static size_t
ZSTD_DUBT_findBetterDictMatch (
- ZSTD_matchState_t* ms,
+ const ZSTD_matchState_t* ms,
const BYTE* const ip, const BYTE* const iend,
size_t* offsetPtr,
size_t bestLength,
@@ -197,8 +197,8 @@ ZSTD_DUBT_findBetterDictMatch (
U32 matchIndex = dictMatchIndex + dictIndexDelta;
if ( (4*(int)(matchLength-bestLength)) > (int)(ZSTD_highbit32(curr-matchIndex+1) - ZSTD_highbit32((U32)offsetPtr[0]+1)) ) {
DEBUGLOG(9, "ZSTD_DUBT_findBetterDictMatch(%u) : found better match length %u -> %u and offsetCode %u -> %u (dictMatchIndex %u, matchIndex %u)",
- curr, (U32)bestLength, (U32)matchLength, (U32)*offsetPtr, ZSTD_REP_MOVE + curr - matchIndex, dictMatchIndex, matchIndex);
- bestLength = matchLength, *offsetPtr = ZSTD_REP_MOVE + curr - matchIndex;
+ curr, (U32)bestLength, (U32)matchLength, (U32)*offsetPtr, STORE_OFFSET(curr - matchIndex), dictMatchIndex, matchIndex);
+ bestLength = matchLength, *offsetPtr = STORE_OFFSET(curr - matchIndex);
}
if (ip+matchLength == iend) { /* reached end of input : ip[matchLength] is not valid, no way to know if it's larger or smaller than match */
break; /* drop, to guarantee consistency (miss a little bit of compression) */
@@ -218,7 +218,7 @@ ZSTD_DUBT_findBetterDictMatch (
}
if (bestLength >= MINMATCH) {
- U32 const mIndex = curr - ((U32)*offsetPtr - ZSTD_REP_MOVE); (void)mIndex;
+ U32 const mIndex = curr - (U32)STORED_OFFSET(*offsetPtr); (void)mIndex;
DEBUGLOG(8, "ZSTD_DUBT_findBetterDictMatch(%u) : found match of length %u and offsetCode %u (pos %u)",
curr, (U32)bestLength, (U32)*offsetPtr, mIndex);
}
@@ -328,7 +328,7 @@ ZSTD_DUBT_findBestMatch(ZSTD_matchState_t* ms,
if (matchLength > matchEndIdx - matchIndex)
matchEndIdx = matchIndex + (U32)matchLength;
if ( (4*(int)(matchLength-bestLength)) > (int)(ZSTD_highbit32(curr-matchIndex+1) - ZSTD_highbit32((U32)offsetPtr[0]+1)) )
- bestLength = matchLength, *offsetPtr = ZSTD_REP_MOVE + curr - matchIndex;
+ bestLength = matchLength, *offsetPtr = STORE_OFFSET(curr - matchIndex);
if (ip+matchLength == iend) { /* equal : no way to know if inf or sup */
if (dictMode == ZSTD_dictMatchState) {
nbCompares = 0; /* in addition to avoiding checking any
@@ -368,7 +368,7 @@ ZSTD_DUBT_findBestMatch(ZSTD_matchState_t* ms,
assert(matchEndIdx > curr+8); /* ensure nextToUpdate is increased */
ms->nextToUpdate = matchEndIdx - 8; /* skip repetitive patterns */
if (bestLength >= MINMATCH) {
- U32 const mIndex = curr - ((U32)*offsetPtr - ZSTD_REP_MOVE); (void)mIndex;
+ U32 const mIndex = curr - (U32)STORED_OFFSET(*offsetPtr); (void)mIndex;
DEBUGLOG(8, "ZSTD_DUBT_findBestMatch(%u) : found match of length %u and offsetCode %u (pos %u)",
curr, (U32)bestLength, (U32)*offsetPtr, mIndex);
}
@@ -391,91 +391,9 @@ ZSTD_BtFindBestMatch( ZSTD_matchState_t* ms,
return ZSTD_DUBT_findBestMatch(ms, ip, iLimit, offsetPtr, mls, dictMode);
}
-
-static size_t
-ZSTD_BtFindBestMatch_selectMLS ( ZSTD_matchState_t* ms,
- const BYTE* ip, const BYTE* const iLimit,
- size_t* offsetPtr)
-{
- switch(ms->cParams.minMatch)
- {
- default : /* includes case 3 */
- case 4 : return ZSTD_BtFindBestMatch(ms, ip, iLimit, offsetPtr, 4, ZSTD_noDict);
- case 5 : return ZSTD_BtFindBestMatch(ms, ip, iLimit, offsetPtr, 5, ZSTD_noDict);
- case 7 :
- case 6 : return ZSTD_BtFindBestMatch(ms, ip, iLimit, offsetPtr, 6, ZSTD_noDict);
- }
-}
-
-
-static size_t ZSTD_BtFindBestMatch_dictMatchState_selectMLS (
- ZSTD_matchState_t* ms,
- const BYTE* ip, const BYTE* const iLimit,
- size_t* offsetPtr)
-{
- switch(ms->cParams.minMatch)
- {
- default : /* includes case 3 */
- case 4 : return ZSTD_BtFindBestMatch(ms, ip, iLimit, offsetPtr, 4, ZSTD_dictMatchState);
- case 5 : return ZSTD_BtFindBestMatch(ms, ip, iLimit, offsetPtr, 5, ZSTD_dictMatchState);
- case 7 :
- case 6 : return ZSTD_BtFindBestMatch(ms, ip, iLimit, offsetPtr, 6, ZSTD_dictMatchState);
- }
-}
-
-
-static size_t ZSTD_BtFindBestMatch_extDict_selectMLS (
- ZSTD_matchState_t* ms,
- const BYTE* ip, const BYTE* const iLimit,
- size_t* offsetPtr)
-{
- switch(ms->cParams.minMatch)
- {
- default : /* includes case 3 */
- case 4 : return ZSTD_BtFindBestMatch(ms, ip, iLimit, offsetPtr, 4, ZSTD_extDict);
- case 5 : return ZSTD_BtFindBestMatch(ms, ip, iLimit, offsetPtr, 5, ZSTD_extDict);
- case 7 :
- case 6 : return ZSTD_BtFindBestMatch(ms, ip, iLimit, offsetPtr, 6, ZSTD_extDict);
- }
-}
-
-
-
/* *********************************
-* Hash Chain
+* Dedicated dict search
***********************************/
-#define NEXT_IN_CHAIN(d, mask) chainTable[(d) & (mask)]
-
-/* Update chains up to ip (excluded)
- Assumption : always within prefix (i.e. not within extDict) */
-FORCE_INLINE_TEMPLATE U32 ZSTD_insertAndFindFirstIndex_internal(
- ZSTD_matchState_t* ms,
- const ZSTD_compressionParameters* const cParams,
- const BYTE* ip, U32 const mls)
-{
- U32* const hashTable = ms->hashTable;
- const U32 hashLog = cParams->hashLog;
- U32* const chainTable = ms->chainTable;
- const U32 chainMask = (1 << cParams->chainLog) - 1;
- const BYTE* const base = ms->window.base;
- const U32 target = (U32)(ip - base);
- U32 idx = ms->nextToUpdate;
-
- while(idx < target) { /* catch up */
- size_t const h = ZSTD_hashPtr(base+idx, hashLog, mls);
- NEXT_IN_CHAIN(idx, chainMask) = hashTable[h];
- hashTable[h] = idx;
- idx++;
- }
-
- ms->nextToUpdate = target;
- return hashTable[ZSTD_hashPtr(ip, hashLog, mls)];
-}
-
-U32 ZSTD_insertAndFindFirstIndex(ZSTD_matchState_t* ms, const BYTE* ip) {
- const ZSTD_compressionParameters* const cParams = &ms->cParams;
- return ZSTD_insertAndFindFirstIndex_internal(ms, cParams, ip, ms->cParams.minMatch);
-}
void ZSTD_dedicatedDictSearch_lazy_loadDictionary(ZSTD_matchState_t* ms, const BYTE* const ip)
{
@@ -485,7 +403,7 @@ void ZSTD_dedicatedDictSearch_lazy_loadDictionary(ZSTD_matchState_t* ms, const B
U32* const chainTable = ms->chainTable;
U32 const chainSize = 1 << ms->cParams.chainLog;
U32 idx = ms->nextToUpdate;
- U32 const minChain = chainSize < target ? target - chainSize : idx;
+ U32 const minChain = chainSize < target - idx ? target - chainSize : idx;
U32 const bucketSize = 1 << ZSTD_LAZY_DDSS_BUCKET_LOG;
U32 const cacheSize = bucketSize - 1;
U32 const chainAttempts = (1 << ms->cParams.searchLog) - cacheSize;
@@ -499,13 +417,12 @@ void ZSTD_dedicatedDictSearch_lazy_loadDictionary(ZSTD_matchState_t* ms, const B
U32 const hashLog = ms->cParams.hashLog - ZSTD_LAZY_DDSS_BUCKET_LOG;
U32* const tmpHashTable = hashTable;
U32* const tmpChainTable = hashTable + ((size_t)1 << hashLog);
- U32 const tmpChainSize = ((1 << ZSTD_LAZY_DDSS_BUCKET_LOG) - 1) << hashLog;
+ U32 const tmpChainSize = (U32)((1 << ZSTD_LAZY_DDSS_BUCKET_LOG) - 1) << hashLog;
U32 const tmpMinChain = tmpChainSize < target ? target - tmpChainSize : idx;
-
U32 hashIdx;
assert(ms->cParams.chainLog <= 24);
- assert(ms->cParams.hashLog >= ms->cParams.chainLog);
+ assert(ms->cParams.hashLog > ms->cParams.chainLog);
assert(idx != 0);
assert(tmpMinChain <= minChain);
@@ -536,7 +453,7 @@ void ZSTD_dedicatedDictSearch_lazy_loadDictionary(ZSTD_matchState_t* ms, const B
if (count == cacheSize) {
for (count = 0; count < chainLimit;) {
if (i < minChain) {
- if (!i || countBeyondMinChain++ > cacheSize) {
+ if (!i || ++countBeyondMinChain > cacheSize) {
/* only allow pulling `cacheSize` number of entries
* into the cache or chainTable beyond `minChain`,
* to replace the entries pulled out of the
@@ -592,10 +509,143 @@ void ZSTD_dedicatedDictSearch_lazy_loadDictionary(ZSTD_matchState_t* ms, const B
ms->nextToUpdate = target;
}
+/* Returns the longest match length found in the dedicated dict search structure.
+ * If none are longer than the argument ml, then ml will be returned.
+ */
+FORCE_INLINE_TEMPLATE
+size_t ZSTD_dedicatedDictSearch_lazy_search(size_t* offsetPtr, size_t ml, U32 nbAttempts,
+ const ZSTD_matchState_t* const dms,
+ const BYTE* const ip, const BYTE* const iLimit,
+ const BYTE* const prefixStart, const U32 curr,
+ const U32 dictLimit, const size_t ddsIdx) {
+ const U32 ddsLowestIndex = dms->window.dictLimit;
+ const BYTE* const ddsBase = dms->window.base;
+ const BYTE* const ddsEnd = dms->window.nextSrc;
+ const U32 ddsSize = (U32)(ddsEnd - ddsBase);
+ const U32 ddsIndexDelta = dictLimit - ddsSize;
+ const U32 bucketSize = (1 << ZSTD_LAZY_DDSS_BUCKET_LOG);
+ const U32 bucketLimit = nbAttempts < bucketSize - 1 ? nbAttempts : bucketSize - 1;
+ U32 ddsAttempt;
+ U32 matchIndex;
+
+ for (ddsAttempt = 0; ddsAttempt < bucketSize - 1; ddsAttempt++) {
+ PREFETCH_L1(ddsBase + dms->hashTable[ddsIdx + ddsAttempt]);
+ }
+
+ {
+ U32 const chainPackedPointer = dms->hashTable[ddsIdx + bucketSize - 1];
+ U32 const chainIndex = chainPackedPointer >> 8;
+
+ PREFETCH_L1(&dms->chainTable[chainIndex]);
+ }
+
+ for (ddsAttempt = 0; ddsAttempt < bucketLimit; ddsAttempt++) {
+ size_t currentMl=0;
+ const BYTE* match;
+ matchIndex = dms->hashTable[ddsIdx + ddsAttempt];
+ match = ddsBase + matchIndex;
+
+ if (!matchIndex) {
+ return ml;
+ }
+
+ /* guaranteed by table construction */
+ (void)ddsLowestIndex;
+ assert(matchIndex >= ddsLowestIndex);
+ assert(match+4 <= ddsEnd);
+ if (MEM_read32(match) == MEM_read32(ip)) {
+ /* assumption : matchIndex <= dictLimit-4 (by table construction) */
+ currentMl = ZSTD_count_2segments(ip+4, match+4, iLimit, ddsEnd, prefixStart) + 4;
+ }
+
+ /* save best solution */
+ if (currentMl > ml) {
+ ml = currentMl;
+ *offsetPtr = STORE_OFFSET(curr - (matchIndex + ddsIndexDelta));
+ if (ip+currentMl == iLimit) {
+ /* best possible, avoids read overflow on next attempt */
+ return ml;
+ }
+ }
+ }
+
+ {
+ U32 const chainPackedPointer = dms->hashTable[ddsIdx + bucketSize - 1];
+ U32 chainIndex = chainPackedPointer >> 8;
+ U32 const chainLength = chainPackedPointer & 0xFF;
+ U32 const chainAttempts = nbAttempts - ddsAttempt;
+ U32 const chainLimit = chainAttempts > chainLength ? chainLength : chainAttempts;
+ U32 chainAttempt;
+
+ for (chainAttempt = 0 ; chainAttempt < chainLimit; chainAttempt++) {
+ PREFETCH_L1(ddsBase + dms->chainTable[chainIndex + chainAttempt]);
+ }
+
+ for (chainAttempt = 0 ; chainAttempt < chainLimit; chainAttempt++, chainIndex++) {
+ size_t currentMl=0;
+ const BYTE* match;
+ matchIndex = dms->chainTable[chainIndex];
+ match = ddsBase + matchIndex;
+
+ /* guaranteed by table construction */
+ assert(matchIndex >= ddsLowestIndex);
+ assert(match+4 <= ddsEnd);
+ if (MEM_read32(match) == MEM_read32(ip)) {
+ /* assumption : matchIndex <= dictLimit-4 (by table construction) */
+ currentMl = ZSTD_count_2segments(ip+4, match+4, iLimit, ddsEnd, prefixStart) + 4;
+ }
+
+ /* save best solution */
+ if (currentMl > ml) {
+ ml = currentMl;
+ *offsetPtr = STORE_OFFSET(curr - (matchIndex + ddsIndexDelta));
+ if (ip+currentMl == iLimit) break; /* best possible, avoids read overflow on next attempt */
+ }
+ }
+ }
+ return ml;
+}
+
+
+/* *********************************
+* Hash Chain
+***********************************/
+#define NEXT_IN_CHAIN(d, mask) chainTable[(d) & (mask)]
+
+/* Update chains up to ip (excluded)
+ Assumption : always within prefix (i.e. not within extDict) */
+FORCE_INLINE_TEMPLATE U32 ZSTD_insertAndFindFirstIndex_internal(
+ ZSTD_matchState_t* ms,
+ const ZSTD_compressionParameters* const cParams,
+ const BYTE* ip, U32 const mls)
+{
+ U32* const hashTable = ms->hashTable;
+ const U32 hashLog = cParams->hashLog;
+ U32* const chainTable = ms->chainTable;
+ const U32 chainMask = (1 << cParams->chainLog) - 1;
+ const BYTE* const base = ms->window.base;
+ const U32 target = (U32)(ip - base);
+ U32 idx = ms->nextToUpdate;
+
+ while(idx < target) { /* catch up */
+ size_t const h = ZSTD_hashPtr(base+idx, hashLog, mls);
+ NEXT_IN_CHAIN(idx, chainMask) = hashTable[h];
+ hashTable[h] = idx;
+ idx++;
+ }
+
+ ms->nextToUpdate = target;
+ return hashTable[ZSTD_hashPtr(ip, hashLog, mls)];
+}
+
+U32 ZSTD_insertAndFindFirstIndex(ZSTD_matchState_t* ms, const BYTE* ip) {
+ const ZSTD_compressionParameters* const cParams = &ms->cParams;
+ return ZSTD_insertAndFindFirstIndex_internal(ms, cParams, ip, ms->cParams.minMatch);
+}
/* inlining is important to hardwire a hot branch (template emulation) */
FORCE_INLINE_TEMPLATE
-size_t ZSTD_HcFindBestMatch_generic (
+size_t ZSTD_HcFindBestMatch(
ZSTD_matchState_t* ms,
const BYTE* const ip, const BYTE* const iLimit,
size_t* offsetPtr,
@@ -653,7 +703,7 @@ size_t ZSTD_HcFindBestMatch_generic (
/* save best solution */
if (currentMl > ml) {
ml = currentMl;
- *offsetPtr = curr - matchIndex + ZSTD_REP_MOVE;
+ *offsetPtr = STORE_OFFSET(curr - matchIndex);
if (ip+currentMl == iLimit) break; /* best possible, avoids read overflow on next attempt */
}
@@ -663,90 +713,8 @@ size_t ZSTD_HcFindBestMatch_generic (
assert(nbAttempts <= (1U << ZSTD_SEARCHLOG_MAX)); /* Check we haven't underflowed. */
if (dictMode == ZSTD_dedicatedDictSearch) {
- const U32 ddsLowestIndex = dms->window.dictLimit;
- const BYTE* const ddsBase = dms->window.base;
- const BYTE* const ddsEnd = dms->window.nextSrc;
- const U32 ddsSize = (U32)(ddsEnd - ddsBase);
- const U32 ddsIndexDelta = dictLimit - ddsSize;
- const U32 bucketSize = (1 << ZSTD_LAZY_DDSS_BUCKET_LOG);
- const U32 bucketLimit = nbAttempts < bucketSize - 1 ? nbAttempts : bucketSize - 1;
- U32 ddsAttempt;
-
- for (ddsAttempt = 0; ddsAttempt < bucketSize - 1; ddsAttempt++) {
- PREFETCH_L1(ddsBase + dms->hashTable[ddsIdx + ddsAttempt]);
- }
-
- {
- U32 const chainPackedPointer = dms->hashTable[ddsIdx + bucketSize - 1];
- U32 const chainIndex = chainPackedPointer >> 8;
-
- PREFETCH_L1(&dms->chainTable[chainIndex]);
- }
-
- for (ddsAttempt = 0; ddsAttempt < bucketLimit; ddsAttempt++) {
- size_t currentMl=0;
- const BYTE* match;
- matchIndex = dms->hashTable[ddsIdx + ddsAttempt];
- match = ddsBase + matchIndex;
-
- if (!matchIndex) {
- return ml;
- }
-
- /* guaranteed by table construction */
- (void)ddsLowestIndex;
- assert(matchIndex >= ddsLowestIndex);
- assert(match+4 <= ddsEnd);
- if (MEM_read32(match) == MEM_read32(ip)) {
- /* assumption : matchIndex <= dictLimit-4 (by table construction) */
- currentMl = ZSTD_count_2segments(ip+4, match+4, iLimit, ddsEnd, prefixStart) + 4;
- }
-
- /* save best solution */
- if (currentMl > ml) {
- ml = currentMl;
- *offsetPtr = curr - (matchIndex + ddsIndexDelta) + ZSTD_REP_MOVE;
- if (ip+currentMl == iLimit) {
- /* best possible, avoids read overflow on next attempt */
- return ml;
- }
- }
- }
-
- {
- U32 const chainPackedPointer = dms->hashTable[ddsIdx + bucketSize - 1];
- U32 chainIndex = chainPackedPointer >> 8;
- U32 const chainLength = chainPackedPointer & 0xFF;
- U32 const chainAttempts = nbAttempts - ddsAttempt;
- U32 const chainLimit = chainAttempts > chainLength ? chainLength : chainAttempts;
- U32 chainAttempt;
-
- for (chainAttempt = 0 ; chainAttempt < chainLimit; chainAttempt++) {
- PREFETCH_L1(ddsBase + dms->chainTable[chainIndex + chainAttempt]);
- }
-
- for (chainAttempt = 0 ; chainAttempt < chainLimit; chainAttempt++, chainIndex++) {
- size_t currentMl=0;
- const BYTE* match;
- matchIndex = dms->chainTable[chainIndex];
- match = ddsBase + matchIndex;
-
- /* guaranteed by table construction */
- assert(matchIndex >= ddsLowestIndex);
- assert(match+4 <= ddsEnd);
- if (MEM_read32(match) == MEM_read32(ip)) {
- /* assumption : matchIndex <= dictLimit-4 (by table construction) */
- currentMl = ZSTD_count_2segments(ip+4, match+4, iLimit, ddsEnd, prefixStart) + 4;
- }
-
- /* save best solution */
- if (currentMl > ml) {
- ml = currentMl;
- *offsetPtr = curr - (matchIndex + ddsIndexDelta) + ZSTD_REP_MOVE;
- if (ip+currentMl == iLimit) break; /* best possible, avoids read overflow on next attempt */
- }
- }
- }
+ ml = ZSTD_dedicatedDictSearch_lazy_search(offsetPtr, ml, nbAttempts, dms,
+ ip, iLimit, prefixStart, curr, dictLimit, ddsIdx);
} else if (dictMode == ZSTD_dictMatchState) {
const U32* const dmsChainTable = dms->chainTable;
const U32 dmsChainSize = (1 << dms->cParams.chainLog);
@@ -770,7 +738,8 @@ size_t ZSTD_HcFindBestMatch_generic (
/* save best solution */
if (currentMl > ml) {
ml = currentMl;
- *offsetPtr = curr - (matchIndex + dmsIndexDelta) + ZSTD_REP_MOVE;
+ assert(curr > matchIndex + dmsIndexDelta);
+ *offsetPtr = STORE_OFFSET(curr - (matchIndex + dmsIndexDelta));
if (ip+currentMl == iLimit) break; /* best possible, avoids read overflow on next attempt */
}
@@ -783,75 +752,725 @@ size_t ZSTD_HcFindBestMatch_generic (
return ml;
}
+/* *********************************
+* (SIMD) Row-based matchfinder
+***********************************/
+/* Constants for row-based hash */
+#define ZSTD_ROW_HASH_TAG_OFFSET 16 /* byte offset of hashes in the match state's tagTable from the beginning of a row */
+#define ZSTD_ROW_HASH_TAG_BITS 8 /* nb bits to use for the tag */
+#define ZSTD_ROW_HASH_TAG_MASK ((1u << ZSTD_ROW_HASH_TAG_BITS) - 1)
+#define ZSTD_ROW_HASH_MAX_ENTRIES 64 /* absolute maximum number of entries per row, for all configurations */
-FORCE_INLINE_TEMPLATE size_t ZSTD_HcFindBestMatch_selectMLS (
- ZSTD_matchState_t* ms,
- const BYTE* ip, const BYTE* const iLimit,
- size_t* offsetPtr)
+#define ZSTD_ROW_HASH_CACHE_MASK (ZSTD_ROW_HASH_CACHE_SIZE - 1)
+
+typedef U64 ZSTD_VecMask; /* Clarifies when we are interacting with a U64 representing a mask of matches */
+
+/* ZSTD_VecMask_next():
+ * Starting from the LSB, returns the idx of the next non-zero bit.
+ * Basically counting the nb of trailing zeroes.
+ */
+static U32 ZSTD_VecMask_next(ZSTD_VecMask val) {
+ assert(val != 0);
+# if (defined(__GNUC__) && ((__GNUC__ > 3) || ((__GNUC__ == 3) && (__GNUC_MINOR__ >= 4))))
+ if (sizeof(size_t) == 4) {
+ U32 mostSignificantWord = (U32)(val >> 32);
+ U32 leastSignificantWord = (U32)val;
+ if (leastSignificantWord == 0) {
+ return 32 + (U32)__builtin_ctz(mostSignificantWord);
+ } else {
+ return (U32)__builtin_ctz(leastSignificantWord);
+ }
+ } else {
+ return (U32)__builtin_ctzll(val);
+ }
+# else
+ /* Software ctz version: http://aggregate.org/MAGIC/#Trailing%20Zero%20Count
+ * and: https://stackoverflow.com/questions/2709430/count-number-of-bits-in-a-64-bit-long-big-integer
+ */
+ val = ~val & (val - 1ULL); /* Lowest set bit mask */
+ val = val - ((val >> 1) & 0x5555555555555555);
+ val = (val & 0x3333333333333333ULL) + ((val >> 2) & 0x3333333333333333ULL);
+ return (U32)((((val + (val >> 4)) & 0xF0F0F0F0F0F0F0FULL) * 0x101010101010101ULL) >> 56);
+# endif
+}
+
+/* ZSTD_rotateRight_*():
+ * Rotates a bitfield to the right by "count" bits.
+ * https://en.wikipedia.org/w/index.php?title=Circular_shift&oldid=991635599#Implementing_circular_shifts
+ */
+FORCE_INLINE_TEMPLATE
+U64 ZSTD_rotateRight_U64(U64 const value, U32 count) {
+ assert(count < 64);
+ count &= 0x3F; /* for fickle pattern recognition */
+ return (value >> count) | (U64)(value << ((0U - count) & 0x3F));
+}
+
+FORCE_INLINE_TEMPLATE
+U32 ZSTD_rotateRight_U32(U32 const value, U32 count) {
+ assert(count < 32);
+ count &= 0x1F; /* for fickle pattern recognition */
+ return (value >> count) | (U32)(value << ((0U - count) & 0x1F));
+}
+
+FORCE_INLINE_TEMPLATE
+U16 ZSTD_rotateRight_U16(U16 const value, U32 count) {
+ assert(count < 16);
+ count &= 0x0F; /* for fickle pattern recognition */
+ return (value >> count) | (U16)(value << ((0U - count) & 0x0F));
+}
+
+/* ZSTD_row_nextIndex():
+ * Returns the next index to insert at within a tagTable row, and updates the "head"
+ * value to reflect the update. Essentially cycles backwards from [0, {entries per row})
+ */
+FORCE_INLINE_TEMPLATE U32 ZSTD_row_nextIndex(BYTE* const tagRow, U32 const rowMask) {
+ U32 const next = (*tagRow - 1) & rowMask;
+ *tagRow = (BYTE)next;
+ return next;
+}
+
+/* ZSTD_isAligned():
+ * Checks that a pointer is aligned to "align" bytes which must be a power of 2.
+ */
+MEM_STATIC int ZSTD_isAligned(void const* ptr, size_t align) {
+ assert((align & (align - 1)) == 0);
+ return (((size_t)ptr) & (align - 1)) == 0;
+}
+
+/* ZSTD_row_prefetch():
+ * Performs prefetching for the hashTable and tagTable at a given row.
+ */
+FORCE_INLINE_TEMPLATE void ZSTD_row_prefetch(U32 const* hashTable, U16 const* tagTable, U32 const relRow, U32 const rowLog) {
+ PREFETCH_L1(hashTable + relRow);
+ if (rowLog >= 5) {
+ PREFETCH_L1(hashTable + relRow + 16);
+ /* Note: prefetching more of the hash table does not appear to be beneficial for 128-entry rows */
+ }
+ PREFETCH_L1(tagTable + relRow);
+ if (rowLog == 6) {
+ PREFETCH_L1(tagTable + relRow + 32);
+ }
+ assert(rowLog == 4 || rowLog == 5 || rowLog == 6);
+ assert(ZSTD_isAligned(hashTable + relRow, 64)); /* prefetched hash row always 64-byte aligned */
+ assert(ZSTD_isAligned(tagTable + relRow, (size_t)1 << rowLog)); /* prefetched tagRow sits on correct multiple of bytes (32,64,128) */
+}
+
+/* ZSTD_row_fillHashCache():
+ * Fill up the hash cache starting at idx, prefetching up to ZSTD_ROW_HASH_CACHE_SIZE entries,
+ * but not beyond iLimit.
+ */
+FORCE_INLINE_TEMPLATE void ZSTD_row_fillHashCache(ZSTD_matchState_t* ms, const BYTE* base,
+ U32 const rowLog, U32 const mls,
+ U32 idx, const BYTE* const iLimit)
{
- switch(ms->cParams.minMatch)
- {
- default : /* includes case 3 */
- case 4 : return ZSTD_HcFindBestMatch_generic(ms, ip, iLimit, offsetPtr, 4, ZSTD_noDict);
- case 5 : return ZSTD_HcFindBestMatch_generic(ms, ip, iLimit, offsetPtr, 5, ZSTD_noDict);
- case 7 :
- case 6 : return ZSTD_HcFindBestMatch_generic(ms, ip, iLimit, offsetPtr, 6, ZSTD_noDict);
+ U32 const* const hashTable = ms->hashTable;
+ U16 const* const tagTable = ms->tagTable;
+ U32 const hashLog = ms->rowHashLog;
+ U32 const maxElemsToPrefetch = (base + idx) > iLimit ? 0 : (U32)(iLimit - (base + idx) + 1);
+ U32 const lim = idx + MIN(ZSTD_ROW_HASH_CACHE_SIZE, maxElemsToPrefetch);
+
+ for (; idx < lim; ++idx) {
+ U32 const hash = (U32)ZSTD_hashPtr(base + idx, hashLog + ZSTD_ROW_HASH_TAG_BITS, mls);
+ U32 const row = (hash >> ZSTD_ROW_HASH_TAG_BITS) << rowLog;
+ ZSTD_row_prefetch(hashTable, tagTable, row, rowLog);
+ ms->hashCache[idx & ZSTD_ROW_HASH_CACHE_MASK] = hash;
+ }
+
+ DEBUGLOG(6, "ZSTD_row_fillHashCache(): [%u %u %u %u %u %u %u %u]", ms->hashCache[0], ms->hashCache[1],
+ ms->hashCache[2], ms->hashCache[3], ms->hashCache[4],
+ ms->hashCache[5], ms->hashCache[6], ms->hashCache[7]);
+}
+
+/* ZSTD_row_nextCachedHash():
+ * Returns the hash of base + idx, and replaces the hash in the hash cache with the byte at
+ * base + idx + ZSTD_ROW_HASH_CACHE_SIZE. Also prefetches the appropriate rows from hashTable and tagTable.
+ */
+FORCE_INLINE_TEMPLATE U32 ZSTD_row_nextCachedHash(U32* cache, U32 const* hashTable,
+ U16 const* tagTable, BYTE const* base,
+ U32 idx, U32 const hashLog,
+ U32 const rowLog, U32 const mls)
+{
+ U32 const newHash = (U32)ZSTD_hashPtr(base+idx+ZSTD_ROW_HASH_CACHE_SIZE, hashLog + ZSTD_ROW_HASH_TAG_BITS, mls);
+ U32 const row = (newHash >> ZSTD_ROW_HASH_TAG_BITS) << rowLog;
+ ZSTD_row_prefetch(hashTable, tagTable, row, rowLog);
+ { U32 const hash = cache[idx & ZSTD_ROW_HASH_CACHE_MASK];
+ cache[idx & ZSTD_ROW_HASH_CACHE_MASK] = newHash;
+ return hash;
}
}
-
-static size_t ZSTD_HcFindBestMatch_dictMatchState_selectMLS (
- ZSTD_matchState_t* ms,
- const BYTE* ip, const BYTE* const iLimit,
- size_t* offsetPtr)
+/* ZSTD_row_update_internalImpl():
+ * Updates the hash table with positions starting from updateStartIdx until updateEndIdx.
+ */
+FORCE_INLINE_TEMPLATE void ZSTD_row_update_internalImpl(ZSTD_matchState_t* ms,
+ U32 updateStartIdx, U32 const updateEndIdx,
+ U32 const mls, U32 const rowLog,
+ U32 const rowMask, U32 const useCache)
{
- switch(ms->cParams.minMatch)
- {
- default : /* includes case 3 */
- case 4 : return ZSTD_HcFindBestMatch_generic(ms, ip, iLimit, offsetPtr, 4, ZSTD_dictMatchState);
- case 5 : return ZSTD_HcFindBestMatch_generic(ms, ip, iLimit, offsetPtr, 5, ZSTD_dictMatchState);
- case 7 :
- case 6 : return ZSTD_HcFindBestMatch_generic(ms, ip, iLimit, offsetPtr, 6, ZSTD_dictMatchState);
+ U32* const hashTable = ms->hashTable;
+ U16* const tagTable = ms->tagTable;
+ U32 const hashLog = ms->rowHashLog;
+ const BYTE* const base = ms->window.base;
+
+ DEBUGLOG(6, "ZSTD_row_update_internalImpl(): updateStartIdx=%u, updateEndIdx=%u", updateStartIdx, updateEndIdx);
+ for (; updateStartIdx < updateEndIdx; ++updateStartIdx) {
+ U32 const hash = useCache ? ZSTD_row_nextCachedHash(ms->hashCache, hashTable, tagTable, base, updateStartIdx, hashLog, rowLog, mls)
+ : (U32)ZSTD_hashPtr(base + updateStartIdx, hashLog + ZSTD_ROW_HASH_TAG_BITS, mls);
+ U32 const relRow = (hash >> ZSTD_ROW_HASH_TAG_BITS) << rowLog;
+ U32* const row = hashTable + relRow;
+ BYTE* tagRow = (BYTE*)(tagTable + relRow); /* Though tagTable is laid out as a table of U16, each tag is only 1 byte.
+ Explicit cast allows us to get exact desired position within each row */
+ U32 const pos = ZSTD_row_nextIndex(tagRow, rowMask);
+
+ assert(hash == ZSTD_hashPtr(base + updateStartIdx, hashLog + ZSTD_ROW_HASH_TAG_BITS, mls));
+ ((BYTE*)tagRow)[pos + ZSTD_ROW_HASH_TAG_OFFSET] = hash & ZSTD_ROW_HASH_TAG_MASK;
+ row[pos] = updateStartIdx;
}
}
-
-static size_t ZSTD_HcFindBestMatch_dedicatedDictSearch_selectMLS (
- ZSTD_matchState_t* ms,
- const BYTE* ip, const BYTE* const iLimit,
- size_t* offsetPtr)
+/* ZSTD_row_update_internal():
+ * Inserts the byte at ip into the appropriate position in the hash table, and updates ms->nextToUpdate.
+ * Skips sections of long matches as is necessary.
+ */
+FORCE_INLINE_TEMPLATE void ZSTD_row_update_internal(ZSTD_matchState_t* ms, const BYTE* ip,
+ U32 const mls, U32 const rowLog,
+ U32 const rowMask, U32 const useCache)
{
- switch(ms->cParams.minMatch)
- {
- default : /* includes case 3 */
- case 4 : return ZSTD_HcFindBestMatch_generic(ms, ip, iLimit, offsetPtr, 4, ZSTD_dedicatedDictSearch);
- case 5 : return ZSTD_HcFindBestMatch_generic(ms, ip, iLimit, offsetPtr, 5, ZSTD_dedicatedDictSearch);
- case 7 :
- case 6 : return ZSTD_HcFindBestMatch_generic(ms, ip, iLimit, offsetPtr, 6, ZSTD_dedicatedDictSearch);
+ U32 idx = ms->nextToUpdate;
+ const BYTE* const base = ms->window.base;
+ const U32 target = (U32)(ip - base);
+ const U32 kSkipThreshold = 384;
+ const U32 kMaxMatchStartPositionsToUpdate = 96;
+ const U32 kMaxMatchEndPositionsToUpdate = 32;
+
+ if (useCache) {
+ /* Only skip positions when using hash cache, i.e.
+ * if we are loading a dict, don't skip anything.
+ * If we decide to skip, then we only update a set number
+ * of positions at the beginning and end of the match.
+ */
+ if (UNLIKELY(target - idx > kSkipThreshold)) {
+ U32 const bound = idx + kMaxMatchStartPositionsToUpdate;
+ ZSTD_row_update_internalImpl(ms, idx, bound, mls, rowLog, rowMask, useCache);
+ idx = target - kMaxMatchEndPositionsToUpdate;
+ ZSTD_row_fillHashCache(ms, base, rowLog, mls, idx, ip+1);
+ }
}
+ assert(target >= idx);
+ ZSTD_row_update_internalImpl(ms, idx, target, mls, rowLog, rowMask, useCache);
+ ms->nextToUpdate = target;
+}
+
+/* ZSTD_row_update():
+ * External wrapper for ZSTD_row_update_internal(). Used for filling the hashtable during dictionary
+ * processing.
+ */
+void ZSTD_row_update(ZSTD_matchState_t* const ms, const BYTE* ip) {
+ const U32 rowLog = BOUNDED(4, ms->cParams.searchLog, 6);
+ const U32 rowMask = (1u << rowLog) - 1;
+ const U32 mls = MIN(ms->cParams.minMatch, 6 /* mls caps out at 6 */);
+
+ DEBUGLOG(5, "ZSTD_row_update(), rowLog=%u", rowLog);
+ ZSTD_row_update_internal(ms, ip, mls, rowLog, rowMask, 0 /* dont use cache */);
+}
+
+#if defined(ZSTD_ARCH_X86_SSE2)
+FORCE_INLINE_TEMPLATE ZSTD_VecMask
+ZSTD_row_getSSEMask(int nbChunks, const BYTE* const src, const BYTE tag, const U32 head)
+{
+ const __m128i comparisonMask = _mm_set1_epi8((char)tag);
+ int matches[4] = {0};
+ int i;
+ assert(nbChunks == 1 || nbChunks == 2 || nbChunks == 4);
+ for (i=0; i<nbChunks; i++) {
+ const __m128i chunk = _mm_loadu_si128((const __m128i*)(const void*)(src + 16*i));
+ const __m128i equalMask = _mm_cmpeq_epi8(chunk, comparisonMask);
+ matches[i] = _mm_movemask_epi8(equalMask);
+ }
+ if (nbChunks == 1) return ZSTD_rotateRight_U16((U16)matches[0], head);
+ if (nbChunks == 2) return ZSTD_rotateRight_U32((U32)matches[1] << 16 | (U32)matches[0], head);
+ assert(nbChunks == 4);
+ return ZSTD_rotateRight_U64((U64)matches[3] << 48 | (U64)matches[2] << 32 | (U64)matches[1] << 16 | (U64)matches[0], head);
+}
+#endif
+
+/* Returns a ZSTD_VecMask (U32) that has the nth bit set to 1 if the newly-computed "tag" matches
+ * the hash at the nth position in a row of the tagTable.
+ * Each row is a circular buffer beginning at the value of "head". So we must rotate the "matches" bitfield
+ * to match up with the actual layout of the entries within the hashTable */
+FORCE_INLINE_TEMPLATE ZSTD_VecMask
+ZSTD_row_getMatchMask(const BYTE* const tagRow, const BYTE tag, const U32 head, const U32 rowEntries)
+{
+ const BYTE* const src = tagRow + ZSTD_ROW_HASH_TAG_OFFSET;
+ assert((rowEntries == 16) || (rowEntries == 32) || rowEntries == 64);
+ assert(rowEntries <= ZSTD_ROW_HASH_MAX_ENTRIES);
+
+#if defined(ZSTD_ARCH_X86_SSE2)
+
+ return ZSTD_row_getSSEMask(rowEntries / 16, src, tag, head);
+
+#else /* SW or NEON-LE */
+
+# if defined(ZSTD_ARCH_ARM_NEON)
+ /* This NEON path only works for little endian - otherwise use SWAR below */
+ if (MEM_isLittleEndian()) {
+ if (rowEntries == 16) {
+ const uint8x16_t chunk = vld1q_u8(src);
+ const uint16x8_t equalMask = vreinterpretq_u16_u8(vceqq_u8(chunk, vdupq_n_u8(tag)));
+ const uint16x8_t t0 = vshlq_n_u16(equalMask, 7);
+ const uint32x4_t t1 = vreinterpretq_u32_u16(vsriq_n_u16(t0, t0, 14));
+ const uint64x2_t t2 = vreinterpretq_u64_u32(vshrq_n_u32(t1, 14));
+ const uint8x16_t t3 = vreinterpretq_u8_u64(vsraq_n_u64(t2, t2, 28));
+ const U16 hi = (U16)vgetq_lane_u8(t3, 8);
+ const U16 lo = (U16)vgetq_lane_u8(t3, 0);
+ return ZSTD_rotateRight_U16((hi << 8) | lo, head);
+ } else if (rowEntries == 32) {
+ const uint16x8x2_t chunk = vld2q_u16((const U16*)(const void*)src);
+ const uint8x16_t chunk0 = vreinterpretq_u8_u16(chunk.val[0]);
+ const uint8x16_t chunk1 = vreinterpretq_u8_u16(chunk.val[1]);
+ const uint8x16_t equalMask0 = vceqq_u8(chunk0, vdupq_n_u8(tag));
+ const uint8x16_t equalMask1 = vceqq_u8(chunk1, vdupq_n_u8(tag));
+ const int8x8_t pack0 = vqmovn_s16(vreinterpretq_s16_u8(equalMask0));
+ const int8x8_t pack1 = vqmovn_s16(vreinterpretq_s16_u8(equalMask1));
+ const uint8x8_t t0 = vreinterpret_u8_s8(pack0);
+ const uint8x8_t t1 = vreinterpret_u8_s8(pack1);
+ const uint8x8_t t2 = vsri_n_u8(t1, t0, 2);
+ const uint8x8x2_t t3 = vuzp_u8(t2, t0);
+ const uint8x8_t t4 = vsri_n_u8(t3.val[1], t3.val[0], 4);
+ const U32 matches = vget_lane_u32(vreinterpret_u32_u8(t4), 0);
+ return ZSTD_rotateRight_U32(matches, head);
+ } else { /* rowEntries == 64 */
+ const uint8x16x4_t chunk = vld4q_u8(src);
+ const uint8x16_t dup = vdupq_n_u8(tag);
+ const uint8x16_t cmp0 = vceqq_u8(chunk.val[0], dup);
+ const uint8x16_t cmp1 = vceqq_u8(chunk.val[1], dup);
+ const uint8x16_t cmp2 = vceqq_u8(chunk.val[2], dup);
+ const uint8x16_t cmp3 = vceqq_u8(chunk.val[3], dup);
+
+ const uint8x16_t t0 = vsriq_n_u8(cmp1, cmp0, 1);
+ const uint8x16_t t1 = vsriq_n_u8(cmp3, cmp2, 1);
+ const uint8x16_t t2 = vsriq_n_u8(t1, t0, 2);
+ const uint8x16_t t3 = vsriq_n_u8(t2, t2, 4);
+ const uint8x8_t t4 = vshrn_n_u16(vreinterpretq_u16_u8(t3), 4);
+ const U64 matches = vget_lane_u64(vreinterpret_u64_u8(t4), 0);
+ return ZSTD_rotateRight_U64(matches, head);
+ }
+ }
+# endif /* ZSTD_ARCH_ARM_NEON */
+ /* SWAR */
+ { const size_t chunkSize = sizeof(size_t);
+ const size_t shiftAmount = ((chunkSize * 8) - chunkSize);
+ const size_t xFF = ~((size_t)0);
+ const size_t x01 = xFF / 0xFF;
+ const size_t x80 = x01 << 7;
+ const size_t splatChar = tag * x01;
+ ZSTD_VecMask matches = 0;
+ int i = rowEntries - chunkSize;
+ assert((sizeof(size_t) == 4) || (sizeof(size_t) == 8));
+ if (MEM_isLittleEndian()) { /* runtime check so have two loops */
+ const size_t extractMagic = (xFF / 0x7F) >> chunkSize;
+ do {
+ size_t chunk = MEM_readST(&src[i]);
+ chunk ^= splatChar;
+ chunk = (((chunk | x80) - x01) | chunk) & x80;
+ matches <<= chunkSize;
+ matches |= (chunk * extractMagic) >> shiftAmount;
+ i -= chunkSize;
+ } while (i >= 0);
+ } else { /* big endian: reverse bits during extraction */
+ const size_t msb = xFF ^ (xFF >> 1);
+ const size_t extractMagic = (msb / 0x1FF) | msb;
+ do {
+ size_t chunk = MEM_readST(&src[i]);
+ chunk ^= splatChar;
+ chunk = (((chunk | x80) - x01) | chunk) & x80;
+ matches <<= chunkSize;
+ matches |= ((chunk >> 7) * extractMagic) >> shiftAmount;
+ i -= chunkSize;
+ } while (i >= 0);
+ }
+ matches = ~matches;
+ if (rowEntries == 16) {
+ return ZSTD_rotateRight_U16((U16)matches, head);
+ } else if (rowEntries == 32) {
+ return ZSTD_rotateRight_U32((U32)matches, head);
+ } else {
+ return ZSTD_rotateRight_U64((U64)matches, head);
+ }
+ }
+#endif
+}
+
+/* The high-level approach of the SIMD row based match finder is as follows:
+ * - Figure out where to insert the new entry:
+ * - Generate a hash from a byte along with an additional 1-byte "short hash". The additional byte is our "tag"
+ * - The hashTable is effectively split into groups or "rows" of 16 or 32 entries of U32, and the hash determines
+ * which row to insert into.
+ * - Determine the correct position within the row to insert the entry into. Each row of 16 or 32 can
+ * be considered as a circular buffer with a "head" index that resides in the tagTable.
+ * - Also insert the "tag" into the equivalent row and position in the tagTable.
+ * - Note: The tagTable has 17 or 33 1-byte entries per row, due to 16 or 32 tags, and 1 "head" entry.
+ * The 17 or 33 entry rows are spaced out to occur every 32 or 64 bytes, respectively,
+ * for alignment/performance reasons, leaving some bytes unused.
+ * - Use SIMD to efficiently compare the tags in the tagTable to the 1-byte "short hash" and
+ * generate a bitfield that we can cycle through to check the collisions in the hash table.
+ * - Pick the longest match.
+ */
+FORCE_INLINE_TEMPLATE
+size_t ZSTD_RowFindBestMatch(
+ ZSTD_matchState_t* ms,
+ const BYTE* const ip, const BYTE* const iLimit,
+ size_t* offsetPtr,
+ const U32 mls, const ZSTD_dictMode_e dictMode,
+ const U32 rowLog)
+{
+ U32* const hashTable = ms->hashTable;
+ U16* const tagTable = ms->tagTable;
+ U32* const hashCache = ms->hashCache;
+ const U32 hashLog = ms->rowHashLog;
+ const ZSTD_compressionParameters* const cParams = &ms->cParams;
+ const BYTE* const base = ms->window.base;
+ const BYTE* const dictBase = ms->window.dictBase;
+ const U32 dictLimit = ms->window.dictLimit;
+ const BYTE* const prefixStart = base + dictLimit;
+ const BYTE* const dictEnd = dictBase + dictLimit;
+ const U32 curr = (U32)(ip-base);
+ const U32 maxDistance = 1U << cParams->windowLog;
+ const U32 lowestValid = ms->window.lowLimit;
+ const U32 withinMaxDistance = (curr - lowestValid > maxDistance) ? curr - maxDistance : lowestValid;
+ const U32 isDictionary = (ms->loadedDictEnd != 0);
+ const U32 lowLimit = isDictionary ? lowestValid : withinMaxDistance;
+ const U32 rowEntries = (1U << rowLog);
+ const U32 rowMask = rowEntries - 1;
+ const U32 cappedSearchLog = MIN(cParams->searchLog, rowLog); /* nb of searches is capped at nb entries per row */
+ U32 nbAttempts = 1U << cappedSearchLog;
+ size_t ml=4-1;
+
+ /* DMS/DDS variables that may be referenced laster */
+ const ZSTD_matchState_t* const dms = ms->dictMatchState;
+
+ /* Initialize the following variables to satisfy static analyzer */
+ size_t ddsIdx = 0;
+ U32 ddsExtraAttempts = 0; /* cctx hash tables are limited in searches, but allow extra searches into DDS */
+ U32 dmsTag = 0;
+ U32* dmsRow = NULL;
+ BYTE* dmsTagRow = NULL;
+
+ if (dictMode == ZSTD_dedicatedDictSearch) {
+ const U32 ddsHashLog = dms->cParams.hashLog - ZSTD_LAZY_DDSS_BUCKET_LOG;
+ { /* Prefetch DDS hashtable entry */
+ ddsIdx = ZSTD_hashPtr(ip, ddsHashLog, mls) << ZSTD_LAZY_DDSS_BUCKET_LOG;
+ PREFETCH_L1(&dms->hashTable[ddsIdx]);
+ }
+ ddsExtraAttempts = cParams->searchLog > rowLog ? 1U << (cParams->searchLog - rowLog) : 0;
+ }
+
+ if (dictMode == ZSTD_dictMatchState) {
+ /* Prefetch DMS rows */
+ U32* const dmsHashTable = dms->hashTable;
+ U16* const dmsTagTable = dms->tagTable;
+ U32 const dmsHash = (U32)ZSTD_hashPtr(ip, dms->rowHashLog + ZSTD_ROW_HASH_TAG_BITS, mls);
+ U32 const dmsRelRow = (dmsHash >> ZSTD_ROW_HASH_TAG_BITS) << rowLog;
+ dmsTag = dmsHash & ZSTD_ROW_HASH_TAG_MASK;
+ dmsTagRow = (BYTE*)(dmsTagTable + dmsRelRow);
+ dmsRow = dmsHashTable + dmsRelRow;
+ ZSTD_row_prefetch(dmsHashTable, dmsTagTable, dmsRelRow, rowLog);
+ }
+
+ /* Update the hashTable and tagTable up to (but not including) ip */
+ ZSTD_row_update_internal(ms, ip, mls, rowLog, rowMask, 1 /* useCache */);
+ { /* Get the hash for ip, compute the appropriate row */
+ U32 const hash = ZSTD_row_nextCachedHash(hashCache, hashTable, tagTable, base, curr, hashLog, rowLog, mls);
+ U32 const relRow = (hash >> ZSTD_ROW_HASH_TAG_BITS) << rowLog;
+ U32 const tag = hash & ZSTD_ROW_HASH_TAG_MASK;
+ U32* const row = hashTable + relRow;
+ BYTE* tagRow = (BYTE*)(tagTable + relRow);
+ U32 const head = *tagRow & rowMask;
+ U32 matchBuffer[ZSTD_ROW_HASH_MAX_ENTRIES];
+ size_t numMatches = 0;
+ size_t currMatch = 0;
+ ZSTD_VecMask matches = ZSTD_row_getMatchMask(tagRow, (BYTE)tag, head, rowEntries);
+
+ /* Cycle through the matches and prefetch */
+ for (; (matches > 0) && (nbAttempts > 0); --nbAttempts, matches &= (matches - 1)) {
+ U32 const matchPos = (head + ZSTD_VecMask_next(matches)) & rowMask;
+ U32 const matchIndex = row[matchPos];
+ assert(numMatches < rowEntries);
+ if (matchIndex < lowLimit)
+ break;
+ if ((dictMode != ZSTD_extDict) || matchIndex >= dictLimit) {
+ PREFETCH_L1(base + matchIndex);
+ } else {
+ PREFETCH_L1(dictBase + matchIndex);
+ }
+ matchBuffer[numMatches++] = matchIndex;
+ }
+
+ /* Speed opt: insert current byte into hashtable too. This allows us to avoid one iteration of the loop
+ in ZSTD_row_update_internal() at the next search. */
+ {
+ U32 const pos = ZSTD_row_nextIndex(tagRow, rowMask);
+ tagRow[pos + ZSTD_ROW_HASH_TAG_OFFSET] = (BYTE)tag;
+ row[pos] = ms->nextToUpdate++;
+ }
+
+ /* Return the longest match */
+ for (; currMatch < numMatches; ++currMatch) {
+ U32 const matchIndex = matchBuffer[currMatch];
+ size_t currentMl=0;
+ assert(matchIndex < curr);
+ assert(matchIndex >= lowLimit);
+
+ if ((dictMode != ZSTD_extDict) || matchIndex >= dictLimit) {
+ const BYTE* const match = base + matchIndex;
+ assert(matchIndex >= dictLimit); /* ensures this is true if dictMode != ZSTD_extDict */
+ if (match[ml] == ip[ml]) /* potentially better */
+ currentMl = ZSTD_count(ip, match, iLimit);
+ } else {
+ const BYTE* const match = dictBase + matchIndex;
+ assert(match+4 <= dictEnd);
+ if (MEM_read32(match) == MEM_read32(ip)) /* assumption : matchIndex <= dictLimit-4 (by table construction) */
+ currentMl = ZSTD_count_2segments(ip+4, match+4, iLimit, dictEnd, prefixStart) + 4;
+ }
+
+ /* Save best solution */
+ if (currentMl > ml) {
+ ml = currentMl;
+ *offsetPtr = STORE_OFFSET(curr - matchIndex);
+ if (ip+currentMl == iLimit) break; /* best possible, avoids read overflow on next attempt */
+ }
+ }
+ }
+
+ assert(nbAttempts <= (1U << ZSTD_SEARCHLOG_MAX)); /* Check we haven't underflowed. */
+ if (dictMode == ZSTD_dedicatedDictSearch) {
+ ml = ZSTD_dedicatedDictSearch_lazy_search(offsetPtr, ml, nbAttempts + ddsExtraAttempts, dms,
+ ip, iLimit, prefixStart, curr, dictLimit, ddsIdx);
+ } else if (dictMode == ZSTD_dictMatchState) {
+ /* TODO: Measure and potentially add prefetching to DMS */
+ const U32 dmsLowestIndex = dms->window.dictLimit;
+ const BYTE* const dmsBase = dms->window.base;
+ const BYTE* const dmsEnd = dms->window.nextSrc;
+ const U32 dmsSize = (U32)(dmsEnd - dmsBase);
+ const U32 dmsIndexDelta = dictLimit - dmsSize;
+
+ { U32 const head = *dmsTagRow & rowMask;
+ U32 matchBuffer[ZSTD_ROW_HASH_MAX_ENTRIES];
+ size_t numMatches = 0;
+ size_t currMatch = 0;
+ ZSTD_VecMask matches = ZSTD_row_getMatchMask(dmsTagRow, (BYTE)dmsTag, head, rowEntries);
+
+ for (; (matches > 0) && (nbAttempts > 0); --nbAttempts, matches &= (matches - 1)) {
+ U32 const matchPos = (head + ZSTD_VecMask_next(matches)) & rowMask;
+ U32 const matchIndex = dmsRow[matchPos];
+ if (matchIndex < dmsLowestIndex)
+ break;
+ PREFETCH_L1(dmsBase + matchIndex);
+ matchBuffer[numMatches++] = matchIndex;
+ }
+
+ /* Return the longest match */
+ for (; currMatch < numMatches; ++currMatch) {
+ U32 const matchIndex = matchBuffer[currMatch];
+ size_t currentMl=0;
+ assert(matchIndex >= dmsLowestIndex);
+ assert(matchIndex < curr);
+
+ { const BYTE* const match = dmsBase + matchIndex;
+ assert(match+4 <= dmsEnd);
+ if (MEM_read32(match) == MEM_read32(ip))
+ currentMl = ZSTD_count_2segments(ip+4, match+4, iLimit, dmsEnd, prefixStart) + 4;
+ }
+
+ if (currentMl > ml) {
+ ml = currentMl;
+ assert(curr > matchIndex + dmsIndexDelta);
+ *offsetPtr = STORE_OFFSET(curr - (matchIndex + dmsIndexDelta));
+ if (ip+currentMl == iLimit) break;
+ }
+ }
+ }
+ }
+ return ml;
}
-FORCE_INLINE_TEMPLATE size_t ZSTD_HcFindBestMatch_extDict_selectMLS (
- ZSTD_matchState_t* ms,
- const BYTE* ip, const BYTE* const iLimit,
- size_t* offsetPtr)
-{
- switch(ms->cParams.minMatch)
- {
- default : /* includes case 3 */
- case 4 : return ZSTD_HcFindBestMatch_generic(ms, ip, iLimit, offsetPtr, 4, ZSTD_extDict);
- case 5 : return ZSTD_HcFindBestMatch_generic(ms, ip, iLimit, offsetPtr, 5, ZSTD_extDict);
- case 7 :
- case 6 : return ZSTD_HcFindBestMatch_generic(ms, ip, iLimit, offsetPtr, 6, ZSTD_extDict);
- }
-}
+/*
+ * Generate search functions templated on (dictMode, mls, rowLog).
+ * These functions are outlined for code size & compilation time.
+ * ZSTD_searchMax() dispatches to the correct implementation function.
+ *
+ * TODO: The start of the search function involves loading and calculating a
+ * bunch of constants from the ZSTD_matchState_t. These computations could be
+ * done in an initialization function, and saved somewhere in the match state.
+ * Then we could pass a pointer to the saved state instead of the match state,
+ * and avoid duplicate computations.
+ *
+ * TODO: Move the match re-winding into searchMax. This improves compression
+ * ratio, and unlocks further simplifications with the next TODO.
+ *
+ * TODO: Try moving the repcode search into searchMax. After the re-winding
+ * and repcode search are in searchMax, there is no more logic in the match
+ * finder loop that requires knowledge about the dictMode. So we should be
+ * able to avoid force inlining it, and we can join the extDict loop with
+ * the single segment loop. It should go in searchMax instead of its own
+ * function to avoid having multiple virtual function calls per search.
+ */
+#define ZSTD_BT_SEARCH_FN(dictMode, mls) ZSTD_BtFindBestMatch_##dictMode##_##mls
+#define ZSTD_HC_SEARCH_FN(dictMode, mls) ZSTD_HcFindBestMatch_##dictMode##_##mls
+#define ZSTD_ROW_SEARCH_FN(dictMode, mls, rowLog) ZSTD_RowFindBestMatch_##dictMode##_##mls##_##rowLog
+
+#define ZSTD_SEARCH_FN_ATTRS FORCE_NOINLINE
+
+#define GEN_ZSTD_BT_SEARCH_FN(dictMode, mls) \
+ ZSTD_SEARCH_FN_ATTRS size_t ZSTD_BT_SEARCH_FN(dictMode, mls)( \
+ ZSTD_matchState_t* ms, \
+ const BYTE* ip, const BYTE* const iLimit, \
+ size_t* offBasePtr) \
+ { \
+ assert(MAX(4, MIN(6, ms->cParams.minMatch)) == mls); \
+ return ZSTD_BtFindBestMatch(ms, ip, iLimit, offBasePtr, mls, ZSTD_##dictMode); \
+ } \
+
+#define GEN_ZSTD_HC_SEARCH_FN(dictMode, mls) \
+ ZSTD_SEARCH_FN_ATTRS size_t ZSTD_HC_SEARCH_FN(dictMode, mls)( \
+ ZSTD_matchState_t* ms, \
+ const BYTE* ip, const BYTE* const iLimit, \
+ size_t* offsetPtr) \
+ { \
+ assert(MAX(4, MIN(6, ms->cParams.minMatch)) == mls); \
+ return ZSTD_HcFindBestMatch(ms, ip, iLimit, offsetPtr, mls, ZSTD_##dictMode); \
+ } \
+
+#define GEN_ZSTD_ROW_SEARCH_FN(dictMode, mls, rowLog) \
+ ZSTD_SEARCH_FN_ATTRS size_t ZSTD_ROW_SEARCH_FN(dictMode, mls, rowLog)( \
+ ZSTD_matchState_t* ms, \
+ const BYTE* ip, const BYTE* const iLimit, \
+ size_t* offsetPtr) \
+ { \
+ assert(MAX(4, MIN(6, ms->cParams.minMatch)) == mls); \
+ assert(MAX(4, MIN(6, ms->cParams.searchLog)) == rowLog); \
+ return ZSTD_RowFindBestMatch(ms, ip, iLimit, offsetPtr, mls, ZSTD_##dictMode, rowLog); \
+ } \
+
+#define ZSTD_FOR_EACH_ROWLOG(X, dictMode, mls) \
+ X(dictMode, mls, 4) \
+ X(dictMode, mls, 5) \
+ X(dictMode, mls, 6)
+
+#define ZSTD_FOR_EACH_MLS_ROWLOG(X, dictMode) \
+ ZSTD_FOR_EACH_ROWLOG(X, dictMode, 4) \
+ ZSTD_FOR_EACH_ROWLOG(X, dictMode, 5) \
+ ZSTD_FOR_EACH_ROWLOG(X, dictMode, 6)
+
+#define ZSTD_FOR_EACH_MLS(X, dictMode) \
+ X(dictMode, 4) \
+ X(dictMode, 5) \
+ X(dictMode, 6)
+
+#define ZSTD_FOR_EACH_DICT_MODE(X, ...) \
+ X(__VA_ARGS__, noDict) \
+ X(__VA_ARGS__, extDict) \
+ X(__VA_ARGS__, dictMatchState) \
+ X(__VA_ARGS__, dedicatedDictSearch)
+
+/* Generate row search fns for each combination of (dictMode, mls, rowLog) */
+ZSTD_FOR_EACH_DICT_MODE(ZSTD_FOR_EACH_MLS_ROWLOG, GEN_ZSTD_ROW_SEARCH_FN)
+/* Generate binary Tree search fns for each combination of (dictMode, mls) */
+ZSTD_FOR_EACH_DICT_MODE(ZSTD_FOR_EACH_MLS, GEN_ZSTD_BT_SEARCH_FN)
+/* Generate hash chain search fns for each combination of (dictMode, mls) */
+ZSTD_FOR_EACH_DICT_MODE(ZSTD_FOR_EACH_MLS, GEN_ZSTD_HC_SEARCH_FN)
+
+typedef enum { search_hashChain=0, search_binaryTree=1, search_rowHash=2 } searchMethod_e;
+
+#define GEN_ZSTD_CALL_BT_SEARCH_FN(dictMode, mls) \
+ case mls: \
+ return ZSTD_BT_SEARCH_FN(dictMode, mls)(ms, ip, iend, offsetPtr);
+#define GEN_ZSTD_CALL_HC_SEARCH_FN(dictMode, mls) \
+ case mls: \
+ return ZSTD_HC_SEARCH_FN(dictMode, mls)(ms, ip, iend, offsetPtr);
+#define GEN_ZSTD_CALL_ROW_SEARCH_FN(dictMode, mls, rowLog) \
+ case rowLog: \
+ return ZSTD_ROW_SEARCH_FN(dictMode, mls, rowLog)(ms, ip, iend, offsetPtr);
+
+#define ZSTD_SWITCH_MLS(X, dictMode) \
+ switch (mls) { \
+ ZSTD_FOR_EACH_MLS(X, dictMode) \
+ }
+
+#define ZSTD_SWITCH_ROWLOG(dictMode, mls) \
+ case mls: \
+ switch (rowLog) { \
+ ZSTD_FOR_EACH_ROWLOG(GEN_ZSTD_CALL_ROW_SEARCH_FN, dictMode, mls) \
+ } \
+ ZSTD_UNREACHABLE; \
+ break;
+
+#define ZSTD_SWITCH_SEARCH_METHOD(dictMode) \
+ switch (searchMethod) { \
+ case search_hashChain: \
+ ZSTD_SWITCH_MLS(GEN_ZSTD_CALL_HC_SEARCH_FN, dictMode) \
+ break; \
+ case search_binaryTree: \
+ ZSTD_SWITCH_MLS(GEN_ZSTD_CALL_BT_SEARCH_FN, dictMode) \
+ break; \
+ case search_rowHash: \
+ ZSTD_SWITCH_MLS(ZSTD_SWITCH_ROWLOG, dictMode) \
+ break; \
+ } \
+ ZSTD_UNREACHABLE;
+
+/*
+ * Searches for the longest match at @p ip.
+ * Dispatches to the correct implementation function based on the
+ * (searchMethod, dictMode, mls, rowLog). We use switch statements
+ * here instead of using an indirect function call through a function
+ * pointer because after Spectre and Meltdown mitigations, indirect
+ * function calls can be very costly, especially in the kernel.
+ *
+ * NOTE: dictMode and searchMethod should be templated, so those switch
+ * statements should be optimized out. Only the mls & rowLog switches
+ * should be left.
+ *
+ * @param ms The match state.
+ * @param ip The position to search at.
+ * @param iend The end of the input data.
+ * @param[out] offsetPtr Stores the match offset into this pointer.
+ * @param mls The minimum search length, in the range [4, 6].
+ * @param rowLog The row log (if applicable), in the range [4, 6].
+ * @param searchMethod The search method to use (templated).
+ * @param dictMode The dictMode (templated).
+ *
+ * @returns The length of the longest match found, or < mls if no match is found.
+ * If a match is found its offset is stored in @p offsetPtr.
+ */
+FORCE_INLINE_TEMPLATE size_t ZSTD_searchMax(
+ ZSTD_matchState_t* ms,
+ const BYTE* ip,
+ const BYTE* iend,
+ size_t* offsetPtr,
+ U32 const mls,
+ U32 const rowLog,
+ searchMethod_e const searchMethod,
+ ZSTD_dictMode_e const dictMode)
+{
+ if (dictMode == ZSTD_noDict) {
+ ZSTD_SWITCH_SEARCH_METHOD(noDict)
+ } else if (dictMode == ZSTD_extDict) {
+ ZSTD_SWITCH_SEARCH_METHOD(extDict)
+ } else if (dictMode == ZSTD_dictMatchState) {
+ ZSTD_SWITCH_SEARCH_METHOD(dictMatchState)
+ } else if (dictMode == ZSTD_dedicatedDictSearch) {
+ ZSTD_SWITCH_SEARCH_METHOD(dedicatedDictSearch)
+ }
+ ZSTD_UNREACHABLE;
+ return 0;
+}
/* *******************************
* Common parser - lazy strategy
*********************************/
-typedef enum { search_hashChain, search_binaryTree } searchMethod_e;
FORCE_INLINE_TEMPLATE size_t
ZSTD_compressBlock_lazy_generic(
@@ -865,41 +1484,13 @@ ZSTD_compressBlock_lazy_generic(
const BYTE* ip = istart;
const BYTE* anchor = istart;
const BYTE* const iend = istart + srcSize;
- const BYTE* const ilimit = iend - 8;
+ const BYTE* const ilimit = (searchMethod == search_rowHash) ? iend - 8 - ZSTD_ROW_HASH_CACHE_SIZE : iend - 8;
const BYTE* const base = ms->window.base;
const U32 prefixLowestIndex = ms->window.dictLimit;
const BYTE* const prefixLowest = base + prefixLowestIndex;
+ const U32 mls = BOUNDED(4, ms->cParams.minMatch, 6);
+ const U32 rowLog = BOUNDED(4, ms->cParams.searchLog, 6);
- typedef size_t (*searchMax_f)(
- ZSTD_matchState_t* ms,
- const BYTE* ip, const BYTE* iLimit, size_t* offsetPtr);
-
- /*
- * This table is indexed first by the four ZSTD_dictMode_e values, and then
- * by the two searchMethod_e values. NULLs are placed for configurations
- * that should never occur (extDict modes go to the other implementation
- * below and there is no DDSS for binary tree search yet).
- */
- const searchMax_f searchFuncs[4][2] = {
- {
- ZSTD_HcFindBestMatch_selectMLS,
- ZSTD_BtFindBestMatch_selectMLS
- },
- {
- NULL,
- NULL
- },
- {
- ZSTD_HcFindBestMatch_dictMatchState_selectMLS,
- ZSTD_BtFindBestMatch_dictMatchState_selectMLS
- },
- {
- ZSTD_HcFindBestMatch_dedicatedDictSearch_selectMLS,
- NULL
- }
- };
-
- searchMax_f const searchMax = searchFuncs[dictMode][searchMethod == search_binaryTree];
U32 offset_1 = rep[0], offset_2 = rep[1], savedOffset=0;
const int isDMS = dictMode == ZSTD_dictMatchState;
@@ -915,11 +1506,7 @@ ZSTD_compressBlock_lazy_generic(
0;
const U32 dictAndPrefixLength = (U32)((ip - prefixLowest) + (dictEnd - dictLowest));
- assert(searchMax != NULL);
-
- DEBUGLOG(5, "ZSTD_compressBlock_lazy_generic (dictMode=%u)", (U32)dictMode);
-
- /* init */
+ DEBUGLOG(5, "ZSTD_compressBlock_lazy_generic (dictMode=%u) (searchFunc=%u)", (U32)dictMode, (U32)searchMethod);
ip += (dictAndPrefixLength == 0);
if (dictMode == ZSTD_noDict) {
U32 const curr = (U32)(ip - base);
@@ -935,6 +1522,12 @@ ZSTD_compressBlock_lazy_generic(
assert(offset_2 <= dictAndPrefixLength);
}
+ if (searchMethod == search_rowHash) {
+ ZSTD_row_fillHashCache(ms, base, rowLog,
+ MIN(ms->cParams.minMatch, 6 /* mls caps out at 6 */),
+ ms->nextToUpdate, ilimit);
+ }
+
/* Match Loop */
#if defined(__x86_64__)
/* I've measured random a 5% speed loss on levels 5 & 6 (greedy) when the
@@ -944,8 +1537,9 @@ ZSTD_compressBlock_lazy_generic(
#endif
while (ip < ilimit) {
size_t matchLength=0;
- size_t offset=0;
+ size_t offcode=STORE_REPCODE_1;
const BYTE* start=ip+1;
+ DEBUGLOG(7, "search baseline (depth 0)");
/* check repCode */
if (isDxS) {
@@ -969,9 +1563,9 @@ ZSTD_compressBlock_lazy_generic(
/* first search (depth 0) */
{ size_t offsetFound = 999999999;
- size_t const ml2 = searchMax(ms, ip, iend, &offsetFound);
+ size_t const ml2 = ZSTD_searchMax(ms, ip, iend, &offsetFound, mls, rowLog, searchMethod, dictMode);
if (ml2 > matchLength)
- matchLength = ml2, start = ip, offset=offsetFound;
+ matchLength = ml2, start = ip, offcode=offsetFound;
}
if (matchLength < 4) {
@@ -982,14 +1576,15 @@ ZSTD_compressBlock_lazy_generic(
/* let's try to find a better solution */
if (depth>=1)
while (ip<ilimit) {
+ DEBUGLOG(7, "search depth 1");
ip ++;
if ( (dictMode == ZSTD_noDict)
- && (offset) && ((offset_1>0) & (MEM_read32(ip) == MEM_read32(ip - offset_1)))) {
+ && (offcode) && ((offset_1>0) & (MEM_read32(ip) == MEM_read32(ip - offset_1)))) {
size_t const mlRep = ZSTD_count(ip+4, ip+4-offset_1, iend) + 4;
int const gain2 = (int)(mlRep * 3);
- int const gain1 = (int)(matchLength*3 - ZSTD_highbit32((U32)offset+1) + 1);
+ int const gain1 = (int)(matchLength*3 - ZSTD_highbit32((U32)STORED_TO_OFFBASE(offcode)) + 1);
if ((mlRep >= 4) && (gain2 > gain1))
- matchLength = mlRep, offset = 0, start = ip;
+ matchLength = mlRep, offcode = STORE_REPCODE_1, start = ip;
}
if (isDxS) {
const U32 repIndex = (U32)(ip - base) - offset_1;
@@ -1001,30 +1596,31 @@ ZSTD_compressBlock_lazy_generic(
const BYTE* repMatchEnd = repIndex < prefixLowestIndex ? dictEnd : iend;
size_t const mlRep = ZSTD_count_2segments(ip+4, repMatch+4, iend, repMatchEnd, prefixLowest) + 4;
int const gain2 = (int)(mlRep * 3);
- int const gain1 = (int)(matchLength*3 - ZSTD_highbit32((U32)offset+1) + 1);
+ int const gain1 = (int)(matchLength*3 - ZSTD_highbit32((U32)STORED_TO_OFFBASE(offcode)) + 1);
if ((mlRep >= 4) && (gain2 > gain1))
- matchLength = mlRep, offset = 0, start = ip;
+ matchLength = mlRep, offcode = STORE_REPCODE_1, start = ip;
}
}
{ size_t offset2=999999999;
- size_t const ml2 = searchMax(ms, ip, iend, &offset2);
- int const gain2 = (int)(ml2*4 - ZSTD_highbit32((U32)offset2+1)); /* raw approx */
- int const gain1 = (int)(matchLength*4 - ZSTD_highbit32((U32)offset+1) + 4);
+ size_t const ml2 = ZSTD_searchMax(ms, ip, iend, &offset2, mls, rowLog, searchMethod, dictMode);
+ int const gain2 = (int)(ml2*4 - ZSTD_highbit32((U32)STORED_TO_OFFBASE(offset2))); /* raw approx */
+ int const gain1 = (int)(matchLength*4 - ZSTD_highbit32((U32)STORED_TO_OFFBASE(offcode)) + 4);
if ((ml2 >= 4) && (gain2 > gain1)) {
- matchLength = ml2, offset = offset2, start = ip;
+ matchLength = ml2, offcode = offset2, start = ip;
continue; /* search a better one */
} }
/* let's find an even better one */
if ((depth==2) && (ip<ilimit)) {
+ DEBUGLOG(7, "search depth 2");
ip ++;
if ( (dictMode == ZSTD_noDict)
- && (offset) && ((offset_1>0) & (MEM_read32(ip) == MEM_read32(ip - offset_1)))) {
+ && (offcode) && ((offset_1>0) & (MEM_read32(ip) == MEM_read32(ip - offset_1)))) {
size_t const mlRep = ZSTD_count(ip+4, ip+4-offset_1, iend) + 4;
int const gain2 = (int)(mlRep * 4);
- int const gain1 = (int)(matchLength*4 - ZSTD_highbit32((U32)offset+1) + 1);
+ int const gain1 = (int)(matchLength*4 - ZSTD_highbit32((U32)STORED_TO_OFFBASE(offcode)) + 1);
if ((mlRep >= 4) && (gain2 > gain1))
- matchLength = mlRep, offset = 0, start = ip;
+ matchLength = mlRep, offcode = STORE_REPCODE_1, start = ip;
}
if (isDxS) {
const U32 repIndex = (U32)(ip - base) - offset_1;
@@ -1036,46 +1632,45 @@ ZSTD_compressBlock_lazy_generic(
const BYTE* repMatchEnd = repIndex < prefixLowestIndex ? dictEnd : iend;
size_t const mlRep = ZSTD_count_2segments(ip+4, repMatch+4, iend, repMatchEnd, prefixLowest) + 4;
int const gain2 = (int)(mlRep * 4);
- int const gain1 = (int)(matchLength*4 - ZSTD_highbit32((U32)offset+1) + 1);
+ int const gain1 = (int)(matchLength*4 - ZSTD_highbit32((U32)STORED_TO_OFFBASE(offcode)) + 1);
if ((mlRep >= 4) && (gain2 > gain1))
- matchLength = mlRep, offset = 0, start = ip;
+ matchLength = mlRep, offcode = STORE_REPCODE_1, start = ip;
}
}
{ size_t offset2=999999999;
- size_t const ml2 = searchMax(ms, ip, iend, &offset2);
- int const gain2 = (int)(ml2*4 - ZSTD_highbit32((U32)offset2+1)); /* raw approx */
- int const gain1 = (int)(matchLength*4 - ZSTD_highbit32((U32)offset+1) + 7);
+ size_t const ml2 = ZSTD_searchMax(ms, ip, iend, &offset2, mls, rowLog, searchMethod, dictMode);
+ int const gain2 = (int)(ml2*4 - ZSTD_highbit32((U32)STORED_TO_OFFBASE(offset2))); /* raw approx */
+ int const gain1 = (int)(matchLength*4 - ZSTD_highbit32((U32)STORED_TO_OFFBASE(offcode)) + 7);
if ((ml2 >= 4) && (gain2 > gain1)) {
- matchLength = ml2, offset = offset2, start = ip;
+ matchLength = ml2, offcode = offset2, start = ip;
continue;
} } }
break; /* nothing found : store previous solution */
}
/* NOTE:
- * start[-offset+ZSTD_REP_MOVE-1] is undefined behavior.
- * (-offset+ZSTD_REP_MOVE-1) is unsigned, and is added to start, which
- * overflows the pointer, which is undefined behavior.
+ * Pay attention that `start[-value]` can lead to strange undefined behavior
+ * notably if `value` is unsigned, resulting in a large positive `-value`.
*/
/* catch up */
- if (offset) {
+ if (STORED_IS_OFFSET(offcode)) {
if (dictMode == ZSTD_noDict) {
- while ( ((start > anchor) & (start - (offset-ZSTD_REP_MOVE) > prefixLowest))
- && (start[-1] == (start-(offset-ZSTD_REP_MOVE))[-1]) ) /* only search for offset within prefix */
+ while ( ((start > anchor) & (start - STORED_OFFSET(offcode) > prefixLowest))
+ && (start[-1] == (start-STORED_OFFSET(offcode))[-1]) ) /* only search for offset within prefix */
{ start--; matchLength++; }
}
if (isDxS) {
- U32 const matchIndex = (U32)((start-base) - (offset - ZSTD_REP_MOVE));
+ U32 const matchIndex = (U32)((size_t)(start-base) - STORED_OFFSET(offcode));
const BYTE* match = (matchIndex < prefixLowestIndex) ? dictBase + matchIndex - dictIndexDelta : base + matchIndex;
const BYTE* const mStart = (matchIndex < prefixLowestIndex) ? dictLowest : prefixLowest;
while ((start>anchor) && (match>mStart) && (start[-1] == match[-1])) { start--; match--; matchLength++; } /* catch up */
}
- offset_2 = offset_1; offset_1 = (U32)(offset - ZSTD_REP_MOVE);
+ offset_2 = offset_1; offset_1 = (U32)STORED_OFFSET(offcode);
}
/* store sequence */
_storeSequence:
- { size_t const litLength = start - anchor;
- ZSTD_storeSeq(seqStore, litLength, anchor, iend, (U32)offset, matchLength-MINMATCH);
+ { size_t const litLength = (size_t)(start - anchor);
+ ZSTD_storeSeq(seqStore, litLength, anchor, iend, (U32)offcode, matchLength);
anchor = ip = start + matchLength;
}
@@ -1091,8 +1686,8 @@ ZSTD_compressBlock_lazy_generic(
&& (MEM_read32(repMatch) == MEM_read32(ip)) ) {
const BYTE* const repEnd2 = repIndex < prefixLowestIndex ? dictEnd : iend;
matchLength = ZSTD_count_2segments(ip+4, repMatch+4, iend, repEnd2, prefixLowest) + 4;
- offset = offset_2; offset_2 = offset_1; offset_1 = (U32)offset; /* swap offset_2 <=> offset_1 */
- ZSTD_storeSeq(seqStore, 0, anchor, iend, 0, matchLength-MINMATCH);
+ offcode = offset_2; offset_2 = offset_1; offset_1 = (U32)offcode; /* swap offset_2 <=> offset_1 */
+ ZSTD_storeSeq(seqStore, 0, anchor, iend, STORE_REPCODE_1, matchLength);
ip += matchLength;
anchor = ip;
continue;
@@ -1106,8 +1701,8 @@ ZSTD_compressBlock_lazy_generic(
&& (MEM_read32(ip) == MEM_read32(ip - offset_2)) ) {
/* store sequence */
matchLength = ZSTD_count(ip+4, ip+4-offset_2, iend) + 4;
- offset = offset_2; offset_2 = offset_1; offset_1 = (U32)offset; /* swap repcodes */
- ZSTD_storeSeq(seqStore, 0, anchor, iend, 0, matchLength-MINMATCH);
+ offcode = offset_2; offset_2 = offset_1; offset_1 = (U32)offcode; /* swap repcodes */
+ ZSTD_storeSeq(seqStore, 0, anchor, iend, STORE_REPCODE_1, matchLength);
ip += matchLength;
anchor = ip;
continue; /* faster when present ... (?) */
@@ -1200,6 +1795,70 @@ size_t ZSTD_compressBlock_greedy_dedicatedDictSearch(
return ZSTD_compressBlock_lazy_generic(ms, seqStore, rep, src, srcSize, search_hashChain, 0, ZSTD_dedicatedDictSearch);
}
+/* Row-based matchfinder */
+size_t ZSTD_compressBlock_lazy2_row(
+ ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM],
+ void const* src, size_t srcSize)
+{
+ return ZSTD_compressBlock_lazy_generic(ms, seqStore, rep, src, srcSize, search_rowHash, 2, ZSTD_noDict);
+}
+
+size_t ZSTD_compressBlock_lazy_row(
+ ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM],
+ void const* src, size_t srcSize)
+{
+ return ZSTD_compressBlock_lazy_generic(ms, seqStore, rep, src, srcSize, search_rowHash, 1, ZSTD_noDict);
+}
+
+size_t ZSTD_compressBlock_greedy_row(
+ ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM],
+ void const* src, size_t srcSize)
+{
+ return ZSTD_compressBlock_lazy_generic(ms, seqStore, rep, src, srcSize, search_rowHash, 0, ZSTD_noDict);
+}
+
+size_t ZSTD_compressBlock_lazy2_dictMatchState_row(
+ ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM],
+ void const* src, size_t srcSize)
+{
+ return ZSTD_compressBlock_lazy_generic(ms, seqStore, rep, src, srcSize, search_rowHash, 2, ZSTD_dictMatchState);
+}
+
+size_t ZSTD_compressBlock_lazy_dictMatchState_row(
+ ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM],
+ void const* src, size_t srcSize)
+{
+ return ZSTD_compressBlock_lazy_generic(ms, seqStore, rep, src, srcSize, search_rowHash, 1, ZSTD_dictMatchState);
+}
+
+size_t ZSTD_compressBlock_greedy_dictMatchState_row(
+ ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM],
+ void const* src, size_t srcSize)
+{
+ return ZSTD_compressBlock_lazy_generic(ms, seqStore, rep, src, srcSize, search_rowHash, 0, ZSTD_dictMatchState);
+}
+
+
+size_t ZSTD_compressBlock_lazy2_dedicatedDictSearch_row(
+ ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM],
+ void const* src, size_t srcSize)
+{
+ return ZSTD_compressBlock_lazy_generic(ms, seqStore, rep, src, srcSize, search_rowHash, 2, ZSTD_dedicatedDictSearch);
+}
+
+size_t ZSTD_compressBlock_lazy_dedicatedDictSearch_row(
+ ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM],
+ void const* src, size_t srcSize)
+{
+ return ZSTD_compressBlock_lazy_generic(ms, seqStore, rep, src, srcSize, search_rowHash, 1, ZSTD_dedicatedDictSearch);
+}
+
+size_t ZSTD_compressBlock_greedy_dedicatedDictSearch_row(
+ ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM],
+ void const* src, size_t srcSize)
+{
+ return ZSTD_compressBlock_lazy_generic(ms, seqStore, rep, src, srcSize, search_rowHash, 0, ZSTD_dedicatedDictSearch);
+}
FORCE_INLINE_TEMPLATE
size_t ZSTD_compressBlock_lazy_extDict_generic(
@@ -1212,7 +1871,7 @@ size_t ZSTD_compressBlock_lazy_extDict_generic(
const BYTE* ip = istart;
const BYTE* anchor = istart;
const BYTE* const iend = istart + srcSize;
- const BYTE* const ilimit = iend - 8;
+ const BYTE* const ilimit = searchMethod == search_rowHash ? iend - 8 - ZSTD_ROW_HASH_CACHE_SIZE : iend - 8;
const BYTE* const base = ms->window.base;
const U32 dictLimit = ms->window.dictLimit;
const BYTE* const prefixStart = base + dictLimit;
@@ -1220,18 +1879,20 @@ size_t ZSTD_compressBlock_lazy_extDict_generic(
const BYTE* const dictEnd = dictBase + dictLimit;
const BYTE* const dictStart = dictBase + ms->window.lowLimit;
const U32 windowLog = ms->cParams.windowLog;
-
- typedef size_t (*searchMax_f)(
- ZSTD_matchState_t* ms,
- const BYTE* ip, const BYTE* iLimit, size_t* offsetPtr);
- searchMax_f searchMax = searchMethod==search_binaryTree ? ZSTD_BtFindBestMatch_extDict_selectMLS : ZSTD_HcFindBestMatch_extDict_selectMLS;
+ const U32 mls = BOUNDED(4, ms->cParams.minMatch, 6);
+ const U32 rowLog = BOUNDED(4, ms->cParams.searchLog, 6);
U32 offset_1 = rep[0], offset_2 = rep[1];
- DEBUGLOG(5, "ZSTD_compressBlock_lazy_extDict_generic");
+ DEBUGLOG(5, "ZSTD_compressBlock_lazy_extDict_generic (searchFunc=%u)", (U32)searchMethod);
/* init */
ip += (ip == prefixStart);
+ if (searchMethod == search_rowHash) {
+ ZSTD_row_fillHashCache(ms, base, rowLog,
+ MIN(ms->cParams.minMatch, 6 /* mls caps out at 6 */),
+ ms->nextToUpdate, ilimit);
+ }
/* Match Loop */
#if defined(__x86_64__)
@@ -1242,7 +1903,7 @@ size_t ZSTD_compressBlock_lazy_extDict_generic(
#endif
while (ip < ilimit) {
size_t matchLength=0;
- size_t offset=0;
+ size_t offcode=STORE_REPCODE_1;
const BYTE* start=ip+1;
U32 curr = (U32)(ip-base);
@@ -1251,7 +1912,8 @@ size_t ZSTD_compressBlock_lazy_extDict_generic(
const U32 repIndex = (U32)(curr+1 - offset_1);
const BYTE* const repBase = repIndex < dictLimit ? dictBase : base;
const BYTE* const repMatch = repBase + repIndex;
- if (((U32)((dictLimit-1) - repIndex) >= 3) & (repIndex > windowLow)) /* intentional overflow */
+ if ( ((U32)((dictLimit-1) - repIndex) >= 3) /* intentional overflow */
+ & (offset_1 <= curr+1 - windowLow) ) /* note: we are searching at curr+1 */
if (MEM_read32(ip+1) == MEM_read32(repMatch)) {
/* repcode detected we should take it */
const BYTE* const repEnd = repIndex < dictLimit ? dictEnd : iend;
@@ -1261,9 +1923,9 @@ size_t ZSTD_compressBlock_lazy_extDict_generic(
/* first search (depth 0) */
{ size_t offsetFound = 999999999;
- size_t const ml2 = searchMax(ms, ip, iend, &offsetFound);
+ size_t const ml2 = ZSTD_searchMax(ms, ip, iend, &offsetFound, mls, rowLog, searchMethod, ZSTD_extDict);
if (ml2 > matchLength)
- matchLength = ml2, start = ip, offset=offsetFound;
+ matchLength = ml2, start = ip, offcode=offsetFound;
}
if (matchLength < 4) {
@@ -1277,29 +1939,30 @@ size_t ZSTD_compressBlock_lazy_extDict_generic(
ip ++;
curr++;
/* check repCode */
- if (offset) {
+ if (offcode) {
const U32 windowLow = ZSTD_getLowestMatchIndex(ms, curr, windowLog);
const U32 repIndex = (U32)(curr - offset_1);
const BYTE* const repBase = repIndex < dictLimit ? dictBase : base;
const BYTE* const repMatch = repBase + repIndex;
- if (((U32)((dictLimit-1) - repIndex) >= 3) & (repIndex > windowLow)) /* intentional overflow */
+ if ( ((U32)((dictLimit-1) - repIndex) >= 3) /* intentional overflow : do not test positions overlapping 2 memory segments */
+ & (offset_1 <= curr - windowLow) ) /* equivalent to `curr > repIndex >= windowLow` */
if (MEM_read32(ip) == MEM_read32(repMatch)) {
/* repcode detected */
const BYTE* const repEnd = repIndex < dictLimit ? dictEnd : iend;
size_t const repLength = ZSTD_count_2segments(ip+4, repMatch+4, iend, repEnd, prefixStart) + 4;
int const gain2 = (int)(repLength * 3);
- int const gain1 = (int)(matchLength*3 - ZSTD_highbit32((U32)offset+1) + 1);
+ int const gain1 = (int)(matchLength*3 - ZSTD_highbit32((U32)STORED_TO_OFFBASE(offcode)) + 1);
if ((repLength >= 4) && (gain2 > gain1))
- matchLength = repLength, offset = 0, start = ip;
+ matchLength = repLength, offcode = STORE_REPCODE_1, start = ip;
} }
/* search match, depth 1 */
{ size_t offset2=999999999;
- size_t const ml2 = searchMax(ms, ip, iend, &offset2);
- int const gain2 = (int)(ml2*4 - ZSTD_highbit32((U32)offset2+1)); /* raw approx */
- int const gain1 = (int)(matchLength*4 - ZSTD_highbit32((U32)offset+1) + 4);
+ size_t const ml2 = ZSTD_searchMax(ms, ip, iend, &offset2, mls, rowLog, searchMethod, ZSTD_extDict);
+ int const gain2 = (int)(ml2*4 - ZSTD_highbit32((U32)STORED_TO_OFFBASE(offset2))); /* raw approx */
+ int const gain1 = (int)(matchLength*4 - ZSTD_highbit32((U32)STORED_TO_OFFBASE(offcode)) + 4);
if ((ml2 >= 4) && (gain2 > gain1)) {
- matchLength = ml2, offset = offset2, start = ip;
+ matchLength = ml2, offcode = offset2, start = ip;
continue; /* search a better one */
} }
@@ -1308,47 +1971,48 @@ size_t ZSTD_compressBlock_lazy_extDict_generic(
ip ++;
curr++;
/* check repCode */
- if (offset) {
+ if (offcode) {
const U32 windowLow = ZSTD_getLowestMatchIndex(ms, curr, windowLog);
const U32 repIndex = (U32)(curr - offset_1);
const BYTE* const repBase = repIndex < dictLimit ? dictBase : base;
const BYTE* const repMatch = repBase + repIndex;
- if (((U32)((dictLimit-1) - repIndex) >= 3) & (repIndex > windowLow)) /* intentional overflow */
+ if ( ((U32)((dictLimit-1) - repIndex) >= 3) /* intentional overflow : do not test positions overlapping 2 memory segments */
+ & (offset_1 <= curr - windowLow) ) /* equivalent to `curr > repIndex >= windowLow` */
if (MEM_read32(ip) == MEM_read32(repMatch)) {
/* repcode detected */
const BYTE* const repEnd = repIndex < dictLimit ? dictEnd : iend;
size_t const repLength = ZSTD_count_2segments(ip+4, repMatch+4, iend, repEnd, prefixStart) + 4;
int const gain2 = (int)(repLength * 4);
- int const gain1 = (int)(matchLength*4 - ZSTD_highbit32((U32)offset+1) + 1);
+ int const gain1 = (int)(matchLength*4 - ZSTD_highbit32((U32)STORED_TO_OFFBASE(offcode)) + 1);
if ((repLength >= 4) && (gain2 > gain1))
- matchLength = repLength, offset = 0, start = ip;
+ matchLength = repLength, offcode = STORE_REPCODE_1, start = ip;
} }
/* search match, depth 2 */
{ size_t offset2=999999999;
- size_t const ml2 = searchMax(ms, ip, iend, &offset2);
- int const gain2 = (int)(ml2*4 - ZSTD_highbit32((U32)offset2+1)); /* raw approx */
- int const gain1 = (int)(matchLength*4 - ZSTD_highbit32((U32)offset+1) + 7);
+ size_t const ml2 = ZSTD_searchMax(ms, ip, iend, &offset2, mls, rowLog, searchMethod, ZSTD_extDict);
+ int const gain2 = (int)(ml2*4 - ZSTD_highbit32((U32)STORED_TO_OFFBASE(offset2))); /* raw approx */
+ int const gain1 = (int)(matchLength*4 - ZSTD_highbit32((U32)STORED_TO_OFFBASE(offcode)) + 7);
if ((ml2 >= 4) && (gain2 > gain1)) {
- matchLength = ml2, offset = offset2, start = ip;
+ matchLength = ml2, offcode = offset2, start = ip;
continue;
} } }
break; /* nothing found : store previous solution */
}
/* catch up */
- if (offset) {
- U32 const matchIndex = (U32)((start-base) - (offset - ZSTD_REP_MOVE));
+ if (STORED_IS_OFFSET(offcode)) {
+ U32 const matchIndex = (U32)((size_t)(start-base) - STORED_OFFSET(offcode));
const BYTE* match = (matchIndex < dictLimit) ? dictBase + matchIndex : base + matchIndex;
const BYTE* const mStart = (matchIndex < dictLimit) ? dictStart : prefixStart;
while ((start>anchor) && (match>mStart) && (start[-1] == match[-1])) { start--; match--; matchLength++; } /* catch up */
- offset_2 = offset_1; offset_1 = (U32)(offset - ZSTD_REP_MOVE);
+ offset_2 = offset_1; offset_1 = (U32)STORED_OFFSET(offcode);
}
/* store sequence */
_storeSequence:
- { size_t const litLength = start - anchor;
- ZSTD_storeSeq(seqStore, litLength, anchor, iend, (U32)offset, matchLength-MINMATCH);
+ { size_t const litLength = (size_t)(start - anchor);
+ ZSTD_storeSeq(seqStore, litLength, anchor, iend, (U32)offcode, matchLength);
anchor = ip = start + matchLength;
}
@@ -1359,13 +2023,14 @@ size_t ZSTD_compressBlock_lazy_extDict_generic(
const U32 repIndex = repCurrent - offset_2;
const BYTE* const repBase = repIndex < dictLimit ? dictBase : base;
const BYTE* const repMatch = repBase + repIndex;
- if (((U32)((dictLimit-1) - repIndex) >= 3) & (repIndex > windowLow)) /* intentional overflow */
+ if ( ((U32)((dictLimit-1) - repIndex) >= 3) /* intentional overflow : do not test positions overlapping 2 memory segments */
+ & (offset_2 <= repCurrent - windowLow) ) /* equivalent to `curr > repIndex >= windowLow` */
if (MEM_read32(ip) == MEM_read32(repMatch)) {
/* repcode detected we should take it */
const BYTE* const repEnd = repIndex < dictLimit ? dictEnd : iend;
matchLength = ZSTD_count_2segments(ip+4, repMatch+4, iend, repEnd, prefixStart) + 4;
- offset = offset_2; offset_2 = offset_1; offset_1 = (U32)offset; /* swap offset history */
- ZSTD_storeSeq(seqStore, 0, anchor, iend, 0, matchLength-MINMATCH);
+ offcode = offset_2; offset_2 = offset_1; offset_1 = (U32)offcode; /* swap offset history */
+ ZSTD_storeSeq(seqStore, 0, anchor, iend, STORE_REPCODE_1, matchLength);
ip += matchLength;
anchor = ip;
continue; /* faster when present ... (?) */
@@ -1412,3 +2077,26 @@ size_t ZSTD_compressBlock_btlazy2_extDict(
{
return ZSTD_compressBlock_lazy_extDict_generic(ms, seqStore, rep, src, srcSize, search_binaryTree, 2);
}
+
+size_t ZSTD_compressBlock_greedy_extDict_row(
+ ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM],
+ void const* src, size_t srcSize)
+{
+ return ZSTD_compressBlock_lazy_extDict_generic(ms, seqStore, rep, src, srcSize, search_rowHash, 0);
+}
+
+size_t ZSTD_compressBlock_lazy_extDict_row(
+ ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM],
+ void const* src, size_t srcSize)
+
+{
+ return ZSTD_compressBlock_lazy_extDict_generic(ms, seqStore, rep, src, srcSize, search_rowHash, 1);
+}
+
+size_t ZSTD_compressBlock_lazy2_extDict_row(
+ ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM],
+ void const* src, size_t srcSize)
+
+{
+ return ZSTD_compressBlock_lazy_extDict_generic(ms, seqStore, rep, src, srcSize, search_rowHash, 2);
+}
diff --git a/lib/zstd/compress/zstd_lazy.h b/lib/zstd/compress/zstd_lazy.h
index 2fc5a61..e5bdf4d 100644
--- a/lib/zstd/compress/zstd_lazy.h
+++ b/lib/zstd/compress/zstd_lazy.h
@@ -23,6 +23,7 @@
#define ZSTD_LAZY_DDSS_BUCKET_LOG 2
U32 ZSTD_insertAndFindFirstIndex(ZSTD_matchState_t* ms, const BYTE* ip);
+void ZSTD_row_update(ZSTD_matchState_t* const ms, const BYTE* ip);
void ZSTD_dedicatedDictSearch_lazy_loadDictionary(ZSTD_matchState_t* ms, const BYTE* const ip);
@@ -40,6 +41,15 @@ size_t ZSTD_compressBlock_lazy(
size_t ZSTD_compressBlock_greedy(
ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM],
void const* src, size_t srcSize);
+size_t ZSTD_compressBlock_lazy2_row(
+ ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM],
+ void const* src, size_t srcSize);
+size_t ZSTD_compressBlock_lazy_row(
+ ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM],
+ void const* src, size_t srcSize);
+size_t ZSTD_compressBlock_greedy_row(
+ ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM],
+ void const* src, size_t srcSize);
size_t ZSTD_compressBlock_btlazy2_dictMatchState(
ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM],
@@ -53,6 +63,15 @@ size_t ZSTD_compressBlock_lazy_dictMatchState(
size_t ZSTD_compressBlock_greedy_dictMatchState(
ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM],
void const* src, size_t srcSize);
+size_t ZSTD_compressBlock_lazy2_dictMatchState_row(
+ ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM],
+ void const* src, size_t srcSize);
+size_t ZSTD_compressBlock_lazy_dictMatchState_row(
+ ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM],
+ void const* src, size_t srcSize);
+size_t ZSTD_compressBlock_greedy_dictMatchState_row(
+ ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM],
+ void const* src, size_t srcSize);
size_t ZSTD_compressBlock_lazy2_dedicatedDictSearch(
ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM],
@@ -63,6 +82,15 @@ size_t ZSTD_compressBlock_lazy_dedicatedDictSearch(
size_t ZSTD_compressBlock_greedy_dedicatedDictSearch(
ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM],
void const* src, size_t srcSize);
+size_t ZSTD_compressBlock_lazy2_dedicatedDictSearch_row(
+ ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM],
+ void const* src, size_t srcSize);
+size_t ZSTD_compressBlock_lazy_dedicatedDictSearch_row(
+ ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM],
+ void const* src, size_t srcSize);
+size_t ZSTD_compressBlock_greedy_dedicatedDictSearch_row(
+ ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM],
+ void const* src, size_t srcSize);
size_t ZSTD_compressBlock_greedy_extDict(
ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM],
@@ -73,9 +101,19 @@ size_t ZSTD_compressBlock_lazy_extDict(
size_t ZSTD_compressBlock_lazy2_extDict(
ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM],
void const* src, size_t srcSize);
+size_t ZSTD_compressBlock_greedy_extDict_row(
+ ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM],
+ void const* src, size_t srcSize);
+size_t ZSTD_compressBlock_lazy_extDict_row(
+ ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM],
+ void const* src, size_t srcSize);
+size_t ZSTD_compressBlock_lazy2_extDict_row(
+ ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM],
+ void const* src, size_t srcSize);
size_t ZSTD_compressBlock_btlazy2_extDict(
ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM],
void const* src, size_t srcSize);
+
#endif /* ZSTD_LAZY_H */
diff --git a/lib/zstd/compress/zstd_ldm.c b/lib/zstd/compress/zstd_ldm.c
index 8ef7e88..dd86fc83 100644
--- a/lib/zstd/compress/zstd_ldm.c
+++ b/lib/zstd/compress/zstd_ldm.c
@@ -57,6 +57,33 @@ static void ZSTD_ldm_gear_init(ldmRollingHashState_t* state, ldmParams_t const*
}
}
+/* ZSTD_ldm_gear_reset()
+ * Feeds [data, data + minMatchLength) into the hash without registering any
+ * splits. This effectively resets the hash state. This is used when skipping
+ * over data, either at the beginning of a block, or skipping sections.
+ */
+static void ZSTD_ldm_gear_reset(ldmRollingHashState_t* state,
+ BYTE const* data, size_t minMatchLength)
+{
+ U64 hash = state->rolling;
+ size_t n = 0;
+
+#define GEAR_ITER_ONCE() do { \
+ hash = (hash << 1) + ZSTD_ldm_gearTab[data[n] & 0xff]; \
+ n += 1; \
+ } while (0)
+ while (n + 3 < minMatchLength) {
+ GEAR_ITER_ONCE();
+ GEAR_ITER_ONCE();
+ GEAR_ITER_ONCE();
+ GEAR_ITER_ONCE();
+ }
+ while (n < minMatchLength) {
+ GEAR_ITER_ONCE();
+ }
+#undef GEAR_ITER_ONCE
+}
+
/* ZSTD_ldm_gear_feed():
*
* Registers in the splits array all the split points found in the first
@@ -132,12 +159,12 @@ size_t ZSTD_ldm_getTableSize(ldmParams_t params)
size_t const ldmBucketSize = ((size_t)1) << (params.hashLog - ldmBucketSizeLog);
size_t const totalSize = ZSTD_cwksp_alloc_size(ldmBucketSize)
+ ZSTD_cwksp_alloc_size(ldmHSize * sizeof(ldmEntry_t));
- return params.enableLdm ? totalSize : 0;
+ return params.enableLdm == ZSTD_ps_enable ? totalSize : 0;
}
size_t ZSTD_ldm_getMaxNbSeq(ldmParams_t params, size_t maxChunkSize)
{
- return params.enableLdm ? (maxChunkSize / params.minMatchLength) : 0;
+ return params.enableLdm == ZSTD_ps_enable ? (maxChunkSize / params.minMatchLength) : 0;
}
/* ZSTD_ldm_getBucket() :
@@ -255,7 +282,7 @@ void ZSTD_ldm_fillHashTable(
while (ip < iend) {
size_t hashed;
unsigned n;
-
+
numSplits = 0;
hashed = ZSTD_ldm_gear_feed(&hashState, ip, iend - ip, splits, &numSplits);
@@ -327,16 +354,8 @@ static size_t ZSTD_ldm_generateSequences_internal(
/* Initialize the rolling hash state with the first minMatchLength bytes */
ZSTD_ldm_gear_init(&hashState, params);
- {
- size_t n = 0;
-
- while (n < minMatchLength) {
- numSplits = 0;
- n += ZSTD_ldm_gear_feed(&hashState, ip + n, minMatchLength - n,
- splits, &numSplits);
- }
- ip += minMatchLength;
- }
+ ZSTD_ldm_gear_reset(&hashState, ip, minMatchLength);
+ ip += minMatchLength;
while (ip < ilimit) {
size_t hashed;
@@ -361,6 +380,7 @@ static size_t ZSTD_ldm_generateSequences_internal(
for (n = 0; n < numSplits; n++) {
size_t forwardMatchLength = 0, backwardMatchLength = 0,
bestMatchLength = 0, mLength;
+ U32 offset;
BYTE const* const split = candidates[n].split;
U32 const checksum = candidates[n].checksum;
U32 const hash = candidates[n].hash;
@@ -428,9 +448,9 @@ static size_t ZSTD_ldm_generateSequences_internal(
}
/* Match found */
+ offset = (U32)(split - base) - bestEntry->offset;
mLength = forwardMatchLength + backwardMatchLength;
{
- U32 const offset = (U32)(split - base) - bestEntry->offset;
rawSeq* const seq = rawSeqStore->seq + rawSeqStore->size;
/* Out of sequence storage */
@@ -447,6 +467,21 @@ static size_t ZSTD_ldm_generateSequences_internal(
ZSTD_ldm_insertEntry(ldmState, hash, newEntry, *params);
anchor = split + forwardMatchLength;
+
+ /* If we find a match that ends after the data that we've hashed
+ * then we have a repeating, overlapping, pattern. E.g. all zeros.
+ * If one repetition of the pattern matches our `stopMask` then all
+ * repetitions will. We don't need to insert them all into out table,
+ * only the first one. So skip over overlapping matches.
+ * This is a major speed boost (20x) for compressing a single byte
+ * repeated, when that byte ends up in the table.
+ */
+ if (anchor > ip + hashed) {
+ ZSTD_ldm_gear_reset(&hashState, anchor - minMatchLength, minMatchLength);
+ /* Continue the outer loop at anchor (ip + hashed == anchor). */
+ ip = anchor - hashed;
+ break;
+ }
}
ip += hashed;
@@ -500,7 +535,7 @@ size_t ZSTD_ldm_generateSequences(
assert(chunkStart < iend);
/* 1. Perform overflow correction if necessary. */
- if (ZSTD_window_needOverflowCorrection(ldmState->window, chunkEnd)) {
+ if (ZSTD_window_needOverflowCorrection(ldmState->window, 0, maxDist, ldmState->loadedDictEnd, chunkStart, chunkEnd)) {
U32 const ldmHSize = 1U << params->hashLog;
U32 const correction = ZSTD_window_correctOverflow(
&ldmState->window, /* cycleLog */ 0, maxDist, chunkStart);
@@ -544,7 +579,9 @@ size_t ZSTD_ldm_generateSequences(
return 0;
}
-void ZSTD_ldm_skipSequences(rawSeqStore_t* rawSeqStore, size_t srcSize, U32 const minMatch) {
+void
+ZSTD_ldm_skipSequences(rawSeqStore_t* rawSeqStore, size_t srcSize, U32 const minMatch)
+{
while (srcSize > 0 && rawSeqStore->pos < rawSeqStore->size) {
rawSeq* seq = rawSeqStore->seq + rawSeqStore->pos;
if (srcSize <= seq->litLength) {
@@ -622,12 +659,13 @@ void ZSTD_ldm_skipRawSeqStoreBytes(rawSeqStore_t* rawSeqStore, size_t nbBytes) {
size_t ZSTD_ldm_blockCompress(rawSeqStore_t* rawSeqStore,
ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM],
+ ZSTD_paramSwitch_e useRowMatchFinder,
void const* src, size_t srcSize)
{
const ZSTD_compressionParameters* const cParams = &ms->cParams;
unsigned const minMatch = cParams->minMatch;
ZSTD_blockCompressor const blockCompressor =
- ZSTD_selectBlockCompressor(cParams->strategy, ZSTD_matchState_dictMode(ms));
+ ZSTD_selectBlockCompressor(cParams->strategy, useRowMatchFinder, ZSTD_matchState_dictMode(ms));
/* Input bounds */
BYTE const* const istart = (BYTE const*)src;
BYTE const* const iend = istart + srcSize;
@@ -673,8 +711,8 @@ size_t ZSTD_ldm_blockCompress(rawSeqStore_t* rawSeqStore,
rep[0] = sequence.offset;
/* Store the sequence */
ZSTD_storeSeq(seqStore, newLitLength, ip - newLitLength, iend,
- sequence.offset + ZSTD_REP_MOVE,
- sequence.matchLength - MINMATCH);
+ STORE_OFFSET(sequence.offset),
+ sequence.matchLength);
ip += sequence.matchLength;
}
}
diff --git a/lib/zstd/compress/zstd_ldm.h b/lib/zstd/compress/zstd_ldm.h
index 25b2527..fbc6a5e 100644
--- a/lib/zstd/compress/zstd_ldm.h
+++ b/lib/zstd/compress/zstd_ldm.h
@@ -63,6 +63,7 @@ size_t ZSTD_ldm_generateSequences(
*/
size_t ZSTD_ldm_blockCompress(rawSeqStore_t* rawSeqStore,
ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM],
+ ZSTD_paramSwitch_e useRowMatchFinder,
void const* src, size_t srcSize);
/*
diff --git a/lib/zstd/compress/zstd_ldm_geartab.h b/lib/zstd/compress/zstd_ldm_geartab.h
index e5c24d8..647f865 100644
--- a/lib/zstd/compress/zstd_ldm_geartab.h
+++ b/lib/zstd/compress/zstd_ldm_geartab.h
@@ -11,7 +11,10 @@
#ifndef ZSTD_LDM_GEARTAB_H
#define ZSTD_LDM_GEARTAB_H
-static U64 ZSTD_ldm_gearTab[256] = {
+#include "../common/compiler.h" /* UNUSED_ATTR */
+#include "../common/mem.h" /* U64 */
+
+static UNUSED_ATTR const U64 ZSTD_ldm_gearTab[256] = {
0xf5b8f72c5f77775c, 0x84935f266b7ac412, 0xb647ada9ca730ccc,
0xb065bb4b114fb1de, 0x34584e7e8c3a9fd0, 0x4e97e17c6ae26b05,
0x3a03d743bc99a604, 0xcecd042422c4044f, 0x76de76c58524259e,
diff --git a/lib/zstd/compress/zstd_opt.c b/lib/zstd/compress/zstd_opt.c
index dfc55e3..fd82acf 100644
--- a/lib/zstd/compress/zstd_opt.c
+++ b/lib/zstd/compress/zstd_opt.c
@@ -8,25 +8,12 @@
* You may select, at your option, one of the above-listed licenses.
*/
-/*
- * Disable inlining for the optimal parser for the kernel build.
- * It is unlikely to be used in the kernel, and where it is used
- * latency shouldn't matter because it is very slow to begin with.
- * We prefer a ~180KB binary size win over faster optimal parsing.
- *
- * TODO(https://github.com/facebook/zstd/issues/2862):
- * Improve the code size of the optimal parser in general, so we
- * don't need this hack for the kernel build.
- */
-#define ZSTD_NO_INLINE 1
-
#include "zstd_compress_internal.h"
#include "hist.h"
#include "zstd_opt.h"
#define ZSTD_LITFREQ_ADD 2 /* scaling factor for litFreq, so that frequencies adapt faster to new stats */
-#define ZSTD_FREQ_DIV 4 /* log factor when using previous stats to init next stats */
#define ZSTD_MAX_PRICE (1<<30)
#define ZSTD_PREDEF_THRESHOLD 1024 /* if srcSize < ZSTD_PREDEF_THRESHOLD, symbols' cost is assumed static, directly determined by pre-defined distributions */
@@ -36,11 +23,11 @@
* Price functions for optimal parser
***************************************/
-#if 0 /* approximation at bit level */
+#if 0 /* approximation at bit level (for tests) */
# define BITCOST_ACCURACY 0
# define BITCOST_MULTIPLIER (1 << BITCOST_ACCURACY)
-# define WEIGHT(stat) ((void)opt, ZSTD_bitWeight(stat))
-#elif 0 /* fractional bit accuracy */
+# define WEIGHT(stat, opt) ((void)opt, ZSTD_bitWeight(stat))
+#elif 0 /* fractional bit accuracy (for tests) */
# define BITCOST_ACCURACY 8
# define BITCOST_MULTIPLIER (1 << BITCOST_ACCURACY)
# define WEIGHT(stat,opt) ((void)opt, ZSTD_fracWeight(stat))
@@ -78,7 +65,7 @@ MEM_STATIC double ZSTD_fCost(U32 price)
static int ZSTD_compressedLiterals(optState_t const* const optPtr)
{
- return optPtr->literalCompressionMode != ZSTD_lcm_uncompressed;
+ return optPtr->literalCompressionMode != ZSTD_ps_disable;
}
static void ZSTD_setBasePrices(optState_t* optPtr, int optLevel)
@@ -91,25 +78,46 @@ static void ZSTD_setBasePrices(optState_t* optPtr, int optLevel)
}
-/* ZSTD_downscaleStat() :
- * reduce all elements in table by a factor 2^(ZSTD_FREQ_DIV+malus)
- * return the resulting sum of elements */
-static U32 ZSTD_downscaleStat(unsigned* table, U32 lastEltIndex, int malus)
+static U32 sum_u32(const unsigned table[], size_t nbElts)
+{
+ size_t n;
+ U32 total = 0;
+ for (n=0; n<nbElts; n++) {
+ total += table[n];
+ }
+ return total;
+}
+
+static U32 ZSTD_downscaleStats(unsigned* table, U32 lastEltIndex, U32 shift)
{
U32 s, sum=0;
- DEBUGLOG(5, "ZSTD_downscaleStat (nbElts=%u)", (unsigned)lastEltIndex+1);
- assert(ZSTD_FREQ_DIV+malus > 0 && ZSTD_FREQ_DIV+malus < 31);
+ DEBUGLOG(5, "ZSTD_downscaleStats (nbElts=%u, shift=%u)", (unsigned)lastEltIndex+1, (unsigned)shift);
+ assert(shift < 30);
for (s=0; s<lastEltIndex+1; s++) {
- table[s] = 1 + (table[s] >> (ZSTD_FREQ_DIV+malus));
+ table[s] = 1 + (table[s] >> shift);
sum += table[s];
}
return sum;
}
+/* ZSTD_scaleStats() :
+ * reduce all elements in table is sum too large
+ * return the resulting sum of elements */
+static U32 ZSTD_scaleStats(unsigned* table, U32 lastEltIndex, U32 logTarget)
+{
+ U32 const prevsum = sum_u32(table, lastEltIndex+1);
+ U32 const factor = prevsum >> logTarget;
+ DEBUGLOG(5, "ZSTD_scaleStats (nbElts=%u, target=%u)", (unsigned)lastEltIndex+1, (unsigned)logTarget);
+ assert(logTarget < 30);
+ if (factor <= 1) return prevsum;
+ return ZSTD_downscaleStats(table, lastEltIndex, ZSTD_highbit32(factor));
+}
+
/* ZSTD_rescaleFreqs() :
* if first block (detected by optPtr->litLengthSum == 0) : init statistics
* take hints from dictionary if there is one
- * or init from zero, using src for literals stats, or flat 1 for match symbols
+ * and init from zero if there is none,
+ * using src for literals stats, and baseline stats for sequence symbols
* otherwise downscale existing stats, to be used as seed for next block.
*/
static void
@@ -138,7 +146,7 @@ ZSTD_rescaleFreqs(optState_t* const optPtr,
optPtr->litSum = 0;
for (lit=0; lit<=MaxLit; lit++) {
U32 const scaleLog = 11; /* scale to 2K */
- U32 const bitCost = HUF_getNbBits(optPtr->symbolCosts->huf.CTable, lit);
+ U32 const bitCost = HUF_getNbBitsFromCTable(optPtr->symbolCosts->huf.CTable, lit);
assert(bitCost <= scaleLog);
optPtr->litFreq[lit] = bitCost ? 1 << (scaleLog-bitCost) : 1 /*minimum to calculate cost*/;
optPtr->litSum += optPtr->litFreq[lit];
@@ -186,14 +194,19 @@ ZSTD_rescaleFreqs(optState_t* const optPtr,
if (compressedLiterals) {
unsigned lit = MaxLit;
HIST_count_simple(optPtr->litFreq, &lit, src, srcSize); /* use raw first block to init statistics */
- optPtr->litSum = ZSTD_downscaleStat(optPtr->litFreq, MaxLit, 1);
+ optPtr->litSum = ZSTD_downscaleStats(optPtr->litFreq, MaxLit, 8);
}
- { unsigned ll;
- for (ll=0; ll<=MaxLL; ll++)
- optPtr->litLengthFreq[ll] = 1;
+ { unsigned const baseLLfreqs[MaxLL+1] = {
+ 4, 2, 1, 1, 1, 1, 1, 1,
+ 1, 1, 1, 1, 1, 1, 1, 1,
+ 1, 1, 1, 1, 1, 1, 1, 1,
+ 1, 1, 1, 1, 1, 1, 1, 1,
+ 1, 1, 1, 1
+ };
+ ZSTD_memcpy(optPtr->litLengthFreq, baseLLfreqs, sizeof(baseLLfreqs));
+ optPtr->litLengthSum = sum_u32(baseLLfreqs, MaxLL+1);
}
- optPtr->litLengthSum = MaxLL+1;
{ unsigned ml;
for (ml=0; ml<=MaxML; ml++)
@@ -201,21 +214,26 @@ ZSTD_rescaleFreqs(optState_t* const optPtr,
}
optPtr->matchLengthSum = MaxML+1;
- { unsigned of;
- for (of=0; of<=MaxOff; of++)
- optPtr->offCodeFreq[of] = 1;
+ { unsigned const baseOFCfreqs[MaxOff+1] = {
+ 6, 2, 1, 1, 2, 3, 4, 4,
+ 4, 3, 2, 1, 1, 1, 1, 1,
+ 1, 1, 1, 1, 1, 1, 1, 1,
+ 1, 1, 1, 1, 1, 1, 1, 1
+ };
+ ZSTD_memcpy(optPtr->offCodeFreq, baseOFCfreqs, sizeof(baseOFCfreqs));
+ optPtr->offCodeSum = sum_u32(baseOFCfreqs, MaxOff+1);
}
- optPtr->offCodeSum = MaxOff+1;
+
}
} else { /* new block : re-use previous statistics, scaled down */
if (compressedLiterals)
- optPtr->litSum = ZSTD_downscaleStat(optPtr->litFreq, MaxLit, 1);
- optPtr->litLengthSum = ZSTD_downscaleStat(optPtr->litLengthFreq, MaxLL, 0);
- optPtr->matchLengthSum = ZSTD_downscaleStat(optPtr->matchLengthFreq, MaxML, 0);
- optPtr->offCodeSum = ZSTD_downscaleStat(optPtr->offCodeFreq, MaxOff, 0);
+ optPtr->litSum = ZSTD_scaleStats(optPtr->litFreq, MaxLit, 12);
+ optPtr->litLengthSum = ZSTD_scaleStats(optPtr->litLengthFreq, MaxLL, 11);
+ optPtr->matchLengthSum = ZSTD_scaleStats(optPtr->matchLengthFreq, MaxML, 11);
+ optPtr->offCodeSum = ZSTD_scaleStats(optPtr->offCodeFreq, MaxOff, 11);
}
ZSTD_setBasePrices(optPtr, optLevel);
@@ -251,7 +269,16 @@ static U32 ZSTD_rawLiteralsCost(const BYTE* const literals, U32 const litLength,
* cost of literalLength symbol */
static U32 ZSTD_litLengthPrice(U32 const litLength, const optState_t* const optPtr, int optLevel)
{
- if (optPtr->priceType == zop_predef) return WEIGHT(litLength, optLevel);
+ assert(litLength <= ZSTD_BLOCKSIZE_MAX);
+ if (optPtr->priceType == zop_predef)
+ return WEIGHT(litLength, optLevel);
+ /* We can't compute the litLength price for sizes >= ZSTD_BLOCKSIZE_MAX
+ * because it isn't representable in the zstd format. So instead just
+ * call it 1 bit more than ZSTD_BLOCKSIZE_MAX - 1. In this case the block
+ * would be all literals.
+ */
+ if (litLength == ZSTD_BLOCKSIZE_MAX)
+ return BITCOST_MULTIPLIER + ZSTD_litLengthPrice(ZSTD_BLOCKSIZE_MAX - 1, optPtr, optLevel);
/* dynamic statistics */
{ U32 const llCode = ZSTD_LLcode(litLength);
@@ -264,15 +291,17 @@ static U32 ZSTD_litLengthPrice(U32 const litLength, const optState_t* const optP
/* ZSTD_getMatchPrice() :
* Provides the cost of the match part (offset + matchLength) of a sequence
* Must be combined with ZSTD_fullLiteralsCost() to get the full cost of a sequence.
- * optLevel: when <2, favors small offset for decompression speed (improved cache efficiency) */
+ * @offcode : expects a scale where 0,1,2 are repcodes 1-3, and 3+ are real_offsets+2
+ * @optLevel: when <2, favors small offset for decompression speed (improved cache efficiency)
+ */
FORCE_INLINE_TEMPLATE U32
-ZSTD_getMatchPrice(U32 const offset,
+ZSTD_getMatchPrice(U32 const offcode,
U32 const matchLength,
const optState_t* const optPtr,
int const optLevel)
{
U32 price;
- U32 const offCode = ZSTD_highbit32(offset+1);
+ U32 const offCode = ZSTD_highbit32(STORED_TO_OFFBASE(offcode));
U32 const mlBase = matchLength - MINMATCH;
assert(matchLength >= MINMATCH);
@@ -315,8 +344,8 @@ static void ZSTD_updateStats(optState_t* const optPtr,
optPtr->litLengthSum++;
}
- /* match offset code (0-2=>repCode; 3+=>offset+2) */
- { U32 const offCode = ZSTD_highbit32(offsetCode+1);
+ /* offset code : expected to follow storeSeq() numeric representation */
+ { U32 const offCode = ZSTD_highbit32(STORED_TO_OFFBASE(offsetCode));
assert(offCode <= MaxOff);
optPtr->offCodeFreq[offCode]++;
optPtr->offCodeSum++;
@@ -350,7 +379,7 @@ MEM_STATIC U32 ZSTD_readMINMATCH(const void* memPtr, U32 length)
/* Update hashTable3 up to ip (excluded)
Assumption : always within prefix (i.e. not within extDict) */
-static U32 ZSTD_insertAndFindFirstIndexHash3 (ZSTD_matchState_t* ms,
+static U32 ZSTD_insertAndFindFirstIndexHash3 (const ZSTD_matchState_t* ms,
U32* nextToUpdate3,
const BYTE* const ip)
{
@@ -376,11 +405,13 @@ static U32 ZSTD_insertAndFindFirstIndexHash3 (ZSTD_matchState_t* ms,
* Binary Tree search
***************************************/
/* ZSTD_insertBt1() : add one or multiple positions to tree.
- * ip : assumed <= iend-8 .
+ * @param ip assumed <= iend-8 .
+ * @param target The target of ZSTD_updateTree_internal() - we are filling to this position
* @return : nb of positions added */
static U32 ZSTD_insertBt1(
- ZSTD_matchState_t* ms,
+ const ZSTD_matchState_t* ms,
const BYTE* const ip, const BYTE* const iend,
+ U32 const target,
U32 const mls, const int extDict)
{
const ZSTD_compressionParameters* const cParams = &ms->cParams;
@@ -403,7 +434,10 @@ static U32 ZSTD_insertBt1(
U32* smallerPtr = bt + 2*(curr&btMask);
U32* largerPtr = smallerPtr + 1;
U32 dummy32; /* to be nullified at the end */
- U32 const windowLow = ms->window.lowLimit;
+ /* windowLow is based on target because
+ * we only need positions that will be in the window at the end of the tree update.
+ */
+ U32 const windowLow = ZSTD_getLowestMatchIndex(ms, target, cParams->windowLog);
U32 matchEndIdx = curr+8+1;
size_t bestLength = 8;
U32 nbCompares = 1U << cParams->searchLog;
@@ -416,6 +450,7 @@ static U32 ZSTD_insertBt1(
DEBUGLOG(8, "ZSTD_insertBt1 (%u)", curr);
+ assert(curr <= target);
assert(ip <= iend-8); /* required for h calculation */
hashTable[h] = curr; /* Update Hash Table */
@@ -504,7 +539,7 @@ void ZSTD_updateTree_internal(
idx, target, dictMode);
while(idx < target) {
- U32 const forward = ZSTD_insertBt1(ms, base+idx, iend, mls, dictMode == ZSTD_extDict);
+ U32 const forward = ZSTD_insertBt1(ms, base+idx, iend, target, mls, dictMode == ZSTD_extDict);
assert(idx < (U32)(idx + forward));
idx += forward;
}
@@ -609,7 +644,7 @@ U32 ZSTD_insertBtAndGetAllMatches (
DEBUGLOG(8, "found repCode %u (ll0:%u, offset:%u) of length %u",
repCode, ll0, repOffset, repLen);
bestLength = repLen;
- matches[mnum].off = repCode - ll0;
+ matches[mnum].off = STORE_REPCODE(repCode - ll0 + 1); /* expect value between 1 and 3 */
matches[mnum].len = (U32)repLen;
mnum++;
if ( (repLen > sufficient_len)
@@ -638,7 +673,7 @@ U32 ZSTD_insertBtAndGetAllMatches (
bestLength = mlen;
assert(curr > matchIndex3);
assert(mnum==0); /* no prior solution */
- matches[0].off = (curr - matchIndex3) + ZSTD_REP_MOVE;
+ matches[0].off = STORE_OFFSET(curr - matchIndex3);
matches[0].len = (U32)mlen;
mnum = 1;
if ( (mlen > sufficient_len) |
@@ -647,7 +682,7 @@ U32 ZSTD_insertBtAndGetAllMatches (
return 1;
} } }
/* no dictMatchState lookup: dicts don't have a populated HC3 table */
- }
+ } /* if (mls == 3) */
hashTable[h] = curr; /* Update Hash Table */
@@ -672,20 +707,19 @@ U32 ZSTD_insertBtAndGetAllMatches (
if (matchLength > bestLength) {
DEBUGLOG(8, "found match of length %u at distance %u (offCode=%u)",
- (U32)matchLength, curr - matchIndex, curr - matchIndex + ZSTD_REP_MOVE);
+ (U32)matchLength, curr - matchIndex, STORE_OFFSET(curr - matchIndex));
assert(matchEndIdx > matchIndex);
if (matchLength > matchEndIdx - matchIndex)
matchEndIdx = matchIndex + (U32)matchLength;
bestLength = matchLength;
- matches[mnum].off = (curr - matchIndex) + ZSTD_REP_MOVE;
+ matches[mnum].off = STORE_OFFSET(curr - matchIndex);
matches[mnum].len = (U32)matchLength;
mnum++;
if ( (matchLength > ZSTD_OPT_NUM)
| (ip+matchLength == iLimit) /* equal : no way to know if inf or sup */) {
if (dictMode == ZSTD_dictMatchState) nbCompares = 0; /* break should also skip searching dms */
break; /* drop, to preserve bt consistency (miss a little bit of compression) */
- }
- }
+ } }
if (match[matchLength] < ip[matchLength]) {
/* match smaller than current */
@@ -721,18 +755,17 @@ U32 ZSTD_insertBtAndGetAllMatches (
if (matchLength > bestLength) {
matchIndex = dictMatchIndex + dmsIndexDelta;
DEBUGLOG(8, "found dms match of length %u at distance %u (offCode=%u)",
- (U32)matchLength, curr - matchIndex, curr - matchIndex + ZSTD_REP_MOVE);
+ (U32)matchLength, curr - matchIndex, STORE_OFFSET(curr - matchIndex));
if (matchLength > matchEndIdx - matchIndex)
matchEndIdx = matchIndex + (U32)matchLength;
bestLength = matchLength;
- matches[mnum].off = (curr - matchIndex) + ZSTD_REP_MOVE;
+ matches[mnum].off = STORE_OFFSET(curr - matchIndex);
matches[mnum].len = (U32)matchLength;
mnum++;
if ( (matchLength > ZSTD_OPT_NUM)
| (ip+matchLength == iLimit) /* equal : no way to know if inf or sup */) {
break; /* drop, to guarantee consistency (miss a little bit of compression) */
- }
- }
+ } }
if (dictMatchIndex <= dmsBtLow) { break; } /* beyond tree size, stop the search */
if (match[matchLength] < ip[matchLength]) {
@@ -742,39 +775,91 @@ U32 ZSTD_insertBtAndGetAllMatches (
/* match is larger than current */
commonLengthLarger = matchLength;
dictMatchIndex = nextPtr[0];
- }
- }
- }
+ } } } /* if (dictMode == ZSTD_dictMatchState) */
assert(matchEndIdx > curr+8);
ms->nextToUpdate = matchEndIdx - 8; /* skip repetitive patterns */
return mnum;
}
+typedef U32 (*ZSTD_getAllMatchesFn)(
+ ZSTD_match_t*,
+ ZSTD_matchState_t*,
+ U32*,
+ const BYTE*,
+ const BYTE*,
+ const U32 rep[ZSTD_REP_NUM],
+ U32 const ll0,
+ U32 const lengthToBeat);
-FORCE_INLINE_TEMPLATE U32 ZSTD_BtGetAllMatches (
- ZSTD_match_t* matches, /* store result (match found, increasing size) in this table */
- ZSTD_matchState_t* ms,
- U32* nextToUpdate3,
- const BYTE* ip, const BYTE* const iHighLimit, const ZSTD_dictMode_e dictMode,
- const U32 rep[ZSTD_REP_NUM],
- U32 const ll0,
- U32 const lengthToBeat)
+FORCE_INLINE_TEMPLATE U32 ZSTD_btGetAllMatches_internal(
+ ZSTD_match_t* matches,
+ ZSTD_matchState_t* ms,
+ U32* nextToUpdate3,
+ const BYTE* ip,
+ const BYTE* const iHighLimit,
+ const U32 rep[ZSTD_REP_NUM],
+ U32 const ll0,
+ U32 const lengthToBeat,
+ const ZSTD_dictMode_e dictMode,
+ const U32 mls)
{
- const ZSTD_compressionParameters* const cParams = &ms->cParams;
- U32 const matchLengthSearch = cParams->minMatch;
- DEBUGLOG(8, "ZSTD_BtGetAllMatches");
- if (ip < ms->window.base + ms->nextToUpdate) return 0; /* skipped area */
- ZSTD_updateTree_internal(ms, ip, iHighLimit, matchLengthSearch, dictMode);
- switch(matchLengthSearch)
- {
- case 3 : return ZSTD_insertBtAndGetAllMatches(matches, ms, nextToUpdate3, ip, iHighLimit, dictMode, rep, ll0, lengthToBeat, 3);
- default :
- case 4 : return ZSTD_insertBtAndGetAllMatches(matches, ms, nextToUpdate3, ip, iHighLimit, dictMode, rep, ll0, lengthToBeat, 4);
- case 5 : return ZSTD_insertBtAndGetAllMatches(matches, ms, nextToUpdate3, ip, iHighLimit, dictMode, rep, ll0, lengthToBeat, 5);
- case 7 :
- case 6 : return ZSTD_insertBtAndGetAllMatches(matches, ms, nextToUpdate3, ip, iHighLimit, dictMode, rep, ll0, lengthToBeat, 6);
+ assert(BOUNDED(3, ms->cParams.minMatch, 6) == mls);
+ DEBUGLOG(8, "ZSTD_BtGetAllMatches(dictMode=%d, mls=%u)", (int)dictMode, mls);
+ if (ip < ms->window.base + ms->nextToUpdate)
+ return 0; /* skipped area */
+ ZSTD_updateTree_internal(ms, ip, iHighLimit, mls, dictMode);
+ return ZSTD_insertBtAndGetAllMatches(matches, ms, nextToUpdate3, ip, iHighLimit, dictMode, rep, ll0, lengthToBeat, mls);
+}
+
+#define ZSTD_BT_GET_ALL_MATCHES_FN(dictMode, mls) ZSTD_btGetAllMatches_##dictMode##_##mls
+
+#define GEN_ZSTD_BT_GET_ALL_MATCHES_(dictMode, mls) \
+ static U32 ZSTD_BT_GET_ALL_MATCHES_FN(dictMode, mls)( \
+ ZSTD_match_t* matches, \
+ ZSTD_matchState_t* ms, \
+ U32* nextToUpdate3, \
+ const BYTE* ip, \
+ const BYTE* const iHighLimit, \
+ const U32 rep[ZSTD_REP_NUM], \
+ U32 const ll0, \
+ U32 const lengthToBeat) \
+ { \
+ return ZSTD_btGetAllMatches_internal( \
+ matches, ms, nextToUpdate3, ip, iHighLimit, \
+ rep, ll0, lengthToBeat, ZSTD_##dictMode, mls); \
}
+
+#define GEN_ZSTD_BT_GET_ALL_MATCHES(dictMode) \
+ GEN_ZSTD_BT_GET_ALL_MATCHES_(dictMode, 3) \
+ GEN_ZSTD_BT_GET_ALL_MATCHES_(dictMode, 4) \
+ GEN_ZSTD_BT_GET_ALL_MATCHES_(dictMode, 5) \
+ GEN_ZSTD_BT_GET_ALL_MATCHES_(dictMode, 6)
+
+GEN_ZSTD_BT_GET_ALL_MATCHES(noDict)
+GEN_ZSTD_BT_GET_ALL_MATCHES(extDict)
+GEN_ZSTD_BT_GET_ALL_MATCHES(dictMatchState)
+
+#define ZSTD_BT_GET_ALL_MATCHES_ARRAY(dictMode) \
+ { \
+ ZSTD_BT_GET_ALL_MATCHES_FN(dictMode, 3), \
+ ZSTD_BT_GET_ALL_MATCHES_FN(dictMode, 4), \
+ ZSTD_BT_GET_ALL_MATCHES_FN(dictMode, 5), \
+ ZSTD_BT_GET_ALL_MATCHES_FN(dictMode, 6) \
+ }
+
+static ZSTD_getAllMatchesFn
+ZSTD_selectBtGetAllMatches(ZSTD_matchState_t const* ms, ZSTD_dictMode_e const dictMode)
+{
+ ZSTD_getAllMatchesFn const getAllMatchesFns[3][4] = {
+ ZSTD_BT_GET_ALL_MATCHES_ARRAY(noDict),
+ ZSTD_BT_GET_ALL_MATCHES_ARRAY(extDict),
+ ZSTD_BT_GET_ALL_MATCHES_ARRAY(dictMatchState)
+ };
+ U32 const mls = BOUNDED(3, ms->cParams.minMatch, 6);
+ assert((U32)dictMode < 3);
+ assert(mls - 3 < 4);
+ return getAllMatchesFns[(int)dictMode][mls - 3];
}
/* ***********************
@@ -783,16 +868,18 @@ FORCE_INLINE_TEMPLATE U32 ZSTD_BtGetAllMatches (
/* Struct containing info needed to make decision about ldm inclusion */
typedef struct {
- rawSeqStore_t seqStore; /* External match candidates store for this block */
- U32 startPosInBlock; /* Start position of the current match candidate */
- U32 endPosInBlock; /* End position of the current match candidate */
- U32 offset; /* Offset of the match candidate */
+ rawSeqStore_t seqStore; /* External match candidates store for this block */
+ U32 startPosInBlock; /* Start position of the current match candidate */
+ U32 endPosInBlock; /* End position of the current match candidate */
+ U32 offset; /* Offset of the match candidate */
} ZSTD_optLdm_t;
/* ZSTD_optLdm_skipRawSeqStoreBytes():
- * Moves forward in rawSeqStore by nbBytes, which will update the fields 'pos' and 'posInSequence'.
+ * Moves forward in @rawSeqStore by @nbBytes,
+ * which will update the fields 'pos' and 'posInSequence'.
*/
-static void ZSTD_optLdm_skipRawSeqStoreBytes(rawSeqStore_t* rawSeqStore, size_t nbBytes) {
+static void ZSTD_optLdm_skipRawSeqStoreBytes(rawSeqStore_t* rawSeqStore, size_t nbBytes)
+{
U32 currPos = (U32)(rawSeqStore->posInSequence + nbBytes);
while (currPos && rawSeqStore->pos < rawSeqStore->size) {
rawSeq currSeq = rawSeqStore->seq[rawSeqStore->pos];
@@ -813,8 +900,10 @@ static void ZSTD_optLdm_skipRawSeqStoreBytes(rawSeqStore_t* rawSeqStore, size_t
* Calculates the beginning and end of the next match in the current block.
* Updates 'pos' and 'posInSequence' of the ldmSeqStore.
*/
-static void ZSTD_opt_getNextMatchAndUpdateSeqStore(ZSTD_optLdm_t* optLdm, U32 currPosInBlock,
- U32 blockBytesRemaining) {
+static void
+ZSTD_opt_getNextMatchAndUpdateSeqStore(ZSTD_optLdm_t* optLdm, U32 currPosInBlock,
+ U32 blockBytesRemaining)
+{
rawSeq currSeq;
U32 currBlockEndPos;
U32 literalsBytesRemaining;
@@ -826,8 +915,8 @@ static void ZSTD_opt_getNextMatchAndUpdateSeqStore(ZSTD_optLdm_t* optLdm, U32 cu
optLdm->endPosInBlock = UINT_MAX;
return;
}
- /* Calculate appropriate bytes left in matchLength and litLength after adjusting
- based on ldmSeqStore->posInSequence */
+ /* Calculate appropriate bytes left in matchLength and litLength
+ * after adjusting based on ldmSeqStore->posInSequence */
currSeq = optLdm->seqStore.seq[optLdm->seqStore.pos];
assert(optLdm->seqStore.posInSequence <= currSeq.litLength + currSeq.matchLength);
currBlockEndPos = currPosInBlock + blockBytesRemaining;
@@ -863,15 +952,16 @@ static void ZSTD_opt_getNextMatchAndUpdateSeqStore(ZSTD_optLdm_t* optLdm, U32 cu
}
/* ZSTD_optLdm_maybeAddMatch():
- * Adds a match if it's long enough, based on it's 'matchStartPosInBlock'
- * and 'matchEndPosInBlock', into 'matches'. Maintains the correct ordering of 'matches'
+ * Adds a match if it's long enough,
+ * based on it's 'matchStartPosInBlock' and 'matchEndPosInBlock',
+ * into 'matches'. Maintains the correct ordering of 'matches'.
*/
static void ZSTD_optLdm_maybeAddMatch(ZSTD_match_t* matches, U32* nbMatches,
- ZSTD_optLdm_t* optLdm, U32 currPosInBlock) {
- U32 posDiff = currPosInBlock - optLdm->startPosInBlock;
+ const ZSTD_optLdm_t* optLdm, U32 currPosInBlock)
+{
+ U32 const posDiff = currPosInBlock - optLdm->startPosInBlock;
/* Note: ZSTD_match_t actually contains offCode and matchLength (before subtracting MINMATCH) */
- U32 candidateMatchLength = optLdm->endPosInBlock - optLdm->startPosInBlock - posDiff;
- U32 candidateOffCode = optLdm->offset + ZSTD_REP_MOVE;
+ U32 const candidateMatchLength = optLdm->endPosInBlock - optLdm->startPosInBlock - posDiff;
/* Ensure that current block position is not outside of the match */
if (currPosInBlock < optLdm->startPosInBlock
@@ -881,6 +971,7 @@ static void ZSTD_optLdm_maybeAddMatch(ZSTD_match_t* matches, U32* nbMatches,
}
if (*nbMatches == 0 || ((candidateMatchLength > matches[*nbMatches-1].len) && *nbMatches < ZSTD_OPT_NUM)) {
+ U32 const candidateOffCode = STORE_OFFSET(optLdm->offset);
DEBUGLOG(6, "ZSTD_optLdm_maybeAddMatch(): Adding ldm candidate match (offCode: %u matchLength %u) at block position=%u",
candidateOffCode, candidateMatchLength, currPosInBlock);
matches[*nbMatches].len = candidateMatchLength;
@@ -892,8 +983,11 @@ static void ZSTD_optLdm_maybeAddMatch(ZSTD_match_t* matches, U32* nbMatches,
/* ZSTD_optLdm_processMatchCandidate():
* Wrapper function to update ldm seq store and call ldm functions as necessary.
*/
-static void ZSTD_optLdm_processMatchCandidate(ZSTD_optLdm_t* optLdm, ZSTD_match_t* matches, U32* nbMatches,
- U32 currPosInBlock, U32 remainingBytes) {
+static void
+ZSTD_optLdm_processMatchCandidate(ZSTD_optLdm_t* optLdm,
+ ZSTD_match_t* matches, U32* nbMatches,
+ U32 currPosInBlock, U32 remainingBytes)
+{
if (optLdm->seqStore.size == 0 || optLdm->seqStore.pos >= optLdm->seqStore.size) {
return;
}
@@ -904,19 +998,19 @@ static void ZSTD_optLdm_processMatchCandidate(ZSTD_optLdm_t* optLdm, ZSTD_match_
* at the end of a match from the ldm seq store, and will often be some bytes
* over beyond matchEndPosInBlock. As such, we need to correct for these "overshoots"
*/
- U32 posOvershoot = currPosInBlock - optLdm->endPosInBlock;
+ U32 const posOvershoot = currPosInBlock - optLdm->endPosInBlock;
ZSTD_optLdm_skipRawSeqStoreBytes(&optLdm->seqStore, posOvershoot);
- }
+ }
ZSTD_opt_getNextMatchAndUpdateSeqStore(optLdm, currPosInBlock, remainingBytes);
}
ZSTD_optLdm_maybeAddMatch(matches, nbMatches, optLdm, currPosInBlock);
}
+
/*-*******************************
* Optimal parser
*********************************/
-
static U32 ZSTD_totalLen(ZSTD_optimal_t sol)
{
return sol.litlen + sol.mlen;
@@ -957,6 +1051,8 @@ ZSTD_compressBlock_opt_generic(ZSTD_matchState_t* ms,
const BYTE* const prefixStart = base + ms->window.dictLimit;
const ZSTD_compressionParameters* const cParams = &ms->cParams;
+ ZSTD_getAllMatchesFn getAllMatches = ZSTD_selectBtGetAllMatches(ms, dictMode);
+
U32 const sufficient_len = MIN(cParams->targetLength, ZSTD_OPT_NUM -1);
U32 const minMatch = (cParams->minMatch == 3) ? 3 : 4;
U32 nextToUpdate3 = ms->nextToUpdate;
@@ -984,7 +1080,7 @@ ZSTD_compressBlock_opt_generic(ZSTD_matchState_t* ms,
/* find first match */
{ U32 const litlen = (U32)(ip - anchor);
U32 const ll0 = !litlen;
- U32 nbMatches = ZSTD_BtGetAllMatches(matches, ms, &nextToUpdate3, ip, iend, dictMode, rep, ll0, minMatch);
+ U32 nbMatches = getAllMatches(matches, ms, &nextToUpdate3, ip, iend, rep, ll0, minMatch);
ZSTD_optLdm_processMatchCandidate(&optLdm, matches, &nbMatches,
(U32)(ip-istart), (U32)(iend - ip));
if (!nbMatches) { ip++; continue; }
@@ -998,18 +1094,18 @@ ZSTD_compressBlock_opt_generic(ZSTD_matchState_t* ms,
* in every price. We include the literal length to avoid negative
* prices when we subtract the previous literal length.
*/
- opt[0].price = ZSTD_litLengthPrice(litlen, optStatePtr, optLevel);
+ opt[0].price = (int)ZSTD_litLengthPrice(litlen, optStatePtr, optLevel);
/* large match -> immediate encoding */
{ U32 const maxML = matches[nbMatches-1].len;
- U32 const maxOffset = matches[nbMatches-1].off;
+ U32 const maxOffcode = matches[nbMatches-1].off;
DEBUGLOG(6, "found %u matches of maxLength=%u and maxOffCode=%u at cPos=%u => start new series",
- nbMatches, maxML, maxOffset, (U32)(ip-prefixStart));
+ nbMatches, maxML, maxOffcode, (U32)(ip-prefixStart));
if (maxML > sufficient_len) {
lastSequence.litlen = litlen;
lastSequence.mlen = maxML;
- lastSequence.off = maxOffset;
+ lastSequence.off = maxOffcode;
DEBUGLOG(6, "large match (%u>%u), immediate encoding",
maxML, sufficient_len);
cur = 0;
@@ -1018,24 +1114,25 @@ ZSTD_compressBlock_opt_generic(ZSTD_matchState_t* ms,
} }
/* set prices for first matches starting position == 0 */
- { U32 const literalsPrice = opt[0].price + ZSTD_litLengthPrice(0, optStatePtr, optLevel);
+ assert(opt[0].price >= 0);
+ { U32 const literalsPrice = (U32)opt[0].price + ZSTD_litLengthPrice(0, optStatePtr, optLevel);
U32 pos;
U32 matchNb;
for (pos = 1; pos < minMatch; pos++) {
opt[pos].price = ZSTD_MAX_PRICE; /* mlen, litlen and price will be fixed during forward scanning */
}
for (matchNb = 0; matchNb < nbMatches; matchNb++) {
- U32 const offset = matches[matchNb].off;
+ U32 const offcode = matches[matchNb].off;
U32 const end = matches[matchNb].len;
for ( ; pos <= end ; pos++ ) {
- U32 const matchPrice = ZSTD_getMatchPrice(offset, pos, optStatePtr, optLevel);
+ U32 const matchPrice = ZSTD_getMatchPrice(offcode, pos, optStatePtr, optLevel);
U32 const sequencePrice = literalsPrice + matchPrice;
DEBUGLOG(7, "rPos:%u => set initial price : %.2f",
pos, ZSTD_fCost(sequencePrice));
opt[pos].mlen = pos;
- opt[pos].off = offset;
+ opt[pos].off = offcode;
opt[pos].litlen = litlen;
- opt[pos].price = sequencePrice;
+ opt[pos].price = (int)sequencePrice;
} }
last_pos = pos-1;
}
@@ -1050,9 +1147,9 @@ ZSTD_compressBlock_opt_generic(ZSTD_matchState_t* ms,
/* Fix current position with one literal if cheaper */
{ U32 const litlen = (opt[cur-1].mlen == 0) ? opt[cur-1].litlen + 1 : 1;
int const price = opt[cur-1].price
- + ZSTD_rawLiteralsCost(ip+cur-1, 1, optStatePtr, optLevel)
- + ZSTD_litLengthPrice(litlen, optStatePtr, optLevel)
- - ZSTD_litLengthPrice(litlen-1, optStatePtr, optLevel);
+ + (int)ZSTD_rawLiteralsCost(ip+cur-1, 1, optStatePtr, optLevel)
+ + (int)ZSTD_litLengthPrice(litlen, optStatePtr, optLevel)
+ - (int)ZSTD_litLengthPrice(litlen-1, optStatePtr, optLevel);
assert(price < 1000000000); /* overflow check */
if (price <= opt[cur].price) {
DEBUGLOG(7, "cPos:%zi==rPos:%u : better price (%.2f<=%.2f) using literal (ll==%u) (hist:%u,%u,%u)",
@@ -1078,7 +1175,7 @@ ZSTD_compressBlock_opt_generic(ZSTD_matchState_t* ms,
assert(cur >= opt[cur].mlen);
if (opt[cur].mlen != 0) {
U32 const prev = cur - opt[cur].mlen;
- repcodes_t newReps = ZSTD_updateRep(opt[prev].rep, opt[cur].off, opt[cur].litlen==0);
+ repcodes_t const newReps = ZSTD_newRep(opt[prev].rep, opt[cur].off, opt[cur].litlen==0);
ZSTD_memcpy(opt[cur].rep, &newReps, sizeof(repcodes_t));
} else {
ZSTD_memcpy(opt[cur].rep, opt[cur - 1].rep, sizeof(repcodes_t));
@@ -1095,11 +1192,12 @@ ZSTD_compressBlock_opt_generic(ZSTD_matchState_t* ms,
continue; /* skip unpromising positions; about ~+6% speed, -0.01 ratio */
}
+ assert(opt[cur].price >= 0);
{ U32 const ll0 = (opt[cur].mlen != 0);
U32 const litlen = (opt[cur].mlen == 0) ? opt[cur].litlen : 0;
- U32 const previousPrice = opt[cur].price;
+ U32 const previousPrice = (U32)opt[cur].price;
U32 const basePrice = previousPrice + ZSTD_litLengthPrice(0, optStatePtr, optLevel);
- U32 nbMatches = ZSTD_BtGetAllMatches(matches, ms, &nextToUpdate3, inr, iend, dictMode, opt[cur].rep, ll0, minMatch);
+ U32 nbMatches = getAllMatches(matches, ms, &nextToUpdate3, inr, iend, opt[cur].rep, ll0, minMatch);
U32 matchNb;
ZSTD_optLdm_processMatchCandidate(&optLdm, matches, &nbMatches,
@@ -1137,7 +1235,7 @@ ZSTD_compressBlock_opt_generic(ZSTD_matchState_t* ms,
for (mlen = lastML; mlen >= startML; mlen--) { /* scan downward */
U32 const pos = cur + mlen;
- int const price = basePrice + ZSTD_getMatchPrice(offset, mlen, optStatePtr, optLevel);
+ int const price = (int)basePrice + (int)ZSTD_getMatchPrice(offset, mlen, optStatePtr, optLevel);
if ((pos > last_pos) || (price < opt[pos].price)) {
DEBUGLOG(7, "rPos:%u (ml=%2u) => new better price (%.2f<%.2f)",
@@ -1167,7 +1265,7 @@ ZSTD_compressBlock_opt_generic(ZSTD_matchState_t* ms,
* update them while traversing the sequences.
*/
if (lastSequence.mlen != 0) {
- repcodes_t reps = ZSTD_updateRep(opt[cur].rep, lastSequence.off, lastSequence.litlen==0);
+ repcodes_t const reps = ZSTD_newRep(opt[cur].rep, lastSequence.off, lastSequence.litlen==0);
ZSTD_memcpy(rep, &reps, sizeof(reps));
} else {
ZSTD_memcpy(rep, opt[cur].rep, sizeof(repcodes_t));
@@ -1211,7 +1309,7 @@ ZSTD_compressBlock_opt_generic(ZSTD_matchState_t* ms,
assert(anchor + llen <= iend);
ZSTD_updateStats(optStatePtr, llen, anchor, offCode, mlen);
- ZSTD_storeSeq(seqStore, llen, anchor, iend, offCode, mlen-MINMATCH);
+ ZSTD_storeSeq(seqStore, llen, anchor, iend, offCode, mlen);
anchor += advance;
ip = anchor;
} }
@@ -1223,38 +1321,30 @@ ZSTD_compressBlock_opt_generic(ZSTD_matchState_t* ms,
return (size_t)(iend - anchor);
}
+static size_t ZSTD_compressBlock_opt0(
+ ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM],
+ const void* src, size_t srcSize, const ZSTD_dictMode_e dictMode)
+{
+ return ZSTD_compressBlock_opt_generic(ms, seqStore, rep, src, srcSize, 0 /* optLevel */, dictMode);
+}
+
+static size_t ZSTD_compressBlock_opt2(
+ ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM],
+ const void* src, size_t srcSize, const ZSTD_dictMode_e dictMode)
+{
+ return ZSTD_compressBlock_opt_generic(ms, seqStore, rep, src, srcSize, 2 /* optLevel */, dictMode);
+}
size_t ZSTD_compressBlock_btopt(
ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM],
const void* src, size_t srcSize)
{
DEBUGLOG(5, "ZSTD_compressBlock_btopt");
- return ZSTD_compressBlock_opt_generic(ms, seqStore, rep, src, srcSize, 0 /*optLevel*/, ZSTD_noDict);
+ return ZSTD_compressBlock_opt0(ms, seqStore, rep, src, srcSize, ZSTD_noDict);
}
-/* used in 2-pass strategy */
-static U32 ZSTD_upscaleStat(unsigned* table, U32 lastEltIndex, int bonus)
-{
- U32 s, sum=0;
- assert(ZSTD_FREQ_DIV+bonus >= 0);
- for (s=0; s<lastEltIndex+1; s++) {
- table[s] <<= ZSTD_FREQ_DIV+bonus;
- table[s]--;
- sum += table[s];
- }
- return sum;
-}
-/* used in 2-pass strategy */
-MEM_STATIC void ZSTD_upscaleStats(optState_t* optPtr)
-{
- if (ZSTD_compressedLiterals(optPtr))
- optPtr->litSum = ZSTD_upscaleStat(optPtr->litFreq, MaxLit, 0);
- optPtr->litLengthSum = ZSTD_upscaleStat(optPtr->litLengthFreq, MaxLL, 0);
- optPtr->matchLengthSum = ZSTD_upscaleStat(optPtr->matchLengthFreq, MaxML, 0);
- optPtr->offCodeSum = ZSTD_upscaleStat(optPtr->offCodeFreq, MaxOff, 0);
-}
/* ZSTD_initStats_ultra():
* make a first compression pass, just to seed stats with more accurate starting values.
@@ -1276,7 +1366,7 @@ ZSTD_initStats_ultra(ZSTD_matchState_t* ms,
assert(ms->window.dictLimit == ms->window.lowLimit); /* no dictionary */
assert(ms->window.dictLimit - ms->nextToUpdate <= 1); /* no prefix (note: intentional overflow, defined as 2-complement) */
- ZSTD_compressBlock_opt_generic(ms, seqStore, tmpRep, src, srcSize, 2 /*optLevel*/, ZSTD_noDict); /* generate stats into ms->opt*/
+ ZSTD_compressBlock_opt2(ms, seqStore, tmpRep, src, srcSize, ZSTD_noDict); /* generate stats into ms->opt*/
/* invalidate first scan from history */
ZSTD_resetSeqStore(seqStore);
@@ -1285,8 +1375,6 @@ ZSTD_initStats_ultra(ZSTD_matchState_t* ms,
ms->window.lowLimit = ms->window.dictLimit;
ms->nextToUpdate = ms->window.dictLimit;
- /* re-inforce weight of collected statistics */
- ZSTD_upscaleStats(&ms->opt);
}
size_t ZSTD_compressBlock_btultra(
@@ -1294,7 +1382,7 @@ size_t ZSTD_compressBlock_btultra(
const void* src, size_t srcSize)
{
DEBUGLOG(5, "ZSTD_compressBlock_btultra (srcSize=%zu)", srcSize);
- return ZSTD_compressBlock_opt_generic(ms, seqStore, rep, src, srcSize, 2 /*optLevel*/, ZSTD_noDict);
+ return ZSTD_compressBlock_opt2(ms, seqStore, rep, src, srcSize, ZSTD_noDict);
}
size_t ZSTD_compressBlock_btultra2(
@@ -1322,35 +1410,35 @@ size_t ZSTD_compressBlock_btultra2(
ZSTD_initStats_ultra(ms, seqStore, rep, src, srcSize);
}
- return ZSTD_compressBlock_opt_generic(ms, seqStore, rep, src, srcSize, 2 /*optLevel*/, ZSTD_noDict);
+ return ZSTD_compressBlock_opt2(ms, seqStore, rep, src, srcSize, ZSTD_noDict);
}
size_t ZSTD_compressBlock_btopt_dictMatchState(
ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM],
const void* src, size_t srcSize)
{
- return ZSTD_compressBlock_opt_generic(ms, seqStore, rep, src, srcSize, 0 /*optLevel*/, ZSTD_dictMatchState);
+ return ZSTD_compressBlock_opt0(ms, seqStore, rep, src, srcSize, ZSTD_dictMatchState);
}
size_t ZSTD_compressBlock_btultra_dictMatchState(
ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM],
const void* src, size_t srcSize)
{
- return ZSTD_compressBlock_opt_generic(ms, seqStore, rep, src, srcSize, 2 /*optLevel*/, ZSTD_dictMatchState);
+ return ZSTD_compressBlock_opt2(ms, seqStore, rep, src, srcSize, ZSTD_dictMatchState);
}
size_t ZSTD_compressBlock_btopt_extDict(
ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM],
const void* src, size_t srcSize)
{
- return ZSTD_compressBlock_opt_generic(ms, seqStore, rep, src, srcSize, 0 /*optLevel*/, ZSTD_extDict);
+ return ZSTD_compressBlock_opt0(ms, seqStore, rep, src, srcSize, ZSTD_extDict);
}
size_t ZSTD_compressBlock_btultra_extDict(
ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM],
const void* src, size_t srcSize)
{
- return ZSTD_compressBlock_opt_generic(ms, seqStore, rep, src, srcSize, 2 /*optLevel*/, ZSTD_extDict);
+ return ZSTD_compressBlock_opt2(ms, seqStore, rep, src, srcSize, ZSTD_extDict);
}
/* note : no btultra2 variant for extDict nor dictMatchState,
diff --git a/lib/zstd/decompress/huf_decompress.c b/lib/zstd/decompress/huf_decompress.c
index 5105e59..89b269a 100644
--- a/lib/zstd/decompress/huf_decompress.c
+++ b/lib/zstd/decompress/huf_decompress.c
@@ -22,6 +22,13 @@
#define HUF_STATIC_LINKING_ONLY
#include "../common/huf.h"
#include "../common/error_private.h"
+#include "../common/zstd_internal.h"
+
+/* **************************************************************
+* Constants
+****************************************************************/
+
+#define HUF_DECODER_FAST_TABLELOG 11
/* **************************************************************
* Macros
@@ -36,6 +43,26 @@
#error "Cannot force the use of the X1 and X2 decoders at the same time!"
#endif
+#if ZSTD_ENABLE_ASM_X86_64_BMI2 && DYNAMIC_BMI2
+# define HUF_ASM_X86_64_BMI2_ATTRS BMI2_TARGET_ATTRIBUTE
+#else
+# define HUF_ASM_X86_64_BMI2_ATTRS
+#endif
+
+#define HUF_EXTERN_C
+#define HUF_ASM_DECL HUF_EXTERN_C
+
+#if DYNAMIC_BMI2 || (ZSTD_ENABLE_ASM_X86_64_BMI2 && defined(__BMI2__))
+# define HUF_NEED_BMI2_FUNCTION 1
+#else
+# define HUF_NEED_BMI2_FUNCTION 0
+#endif
+
+#if !(ZSTD_ENABLE_ASM_X86_64_BMI2 && defined(__BMI2__))
+# define HUF_NEED_DEFAULT_FUNCTION 1
+#else
+# define HUF_NEED_DEFAULT_FUNCTION 0
+#endif
/* **************************************************************
* Error Management
@@ -65,7 +92,7 @@
return fn##_body(dst, dstSize, cSrc, cSrcSize, DTable); \
} \
\
- static TARGET_ATTRIBUTE("bmi2") size_t fn##_bmi2( \
+ static BMI2_TARGET_ATTRIBUTE size_t fn##_bmi2( \
void* dst, size_t dstSize, \
const void* cSrc, size_t cSrcSize, \
const HUF_DTable* DTable) \
@@ -107,13 +134,147 @@ static DTableDesc HUF_getDTableDesc(const HUF_DTable* table)
return dtd;
}
+#if ZSTD_ENABLE_ASM_X86_64_BMI2
+
+static size_t HUF_initDStream(BYTE const* ip) {
+ BYTE const lastByte = ip[7];
+ size_t const bitsConsumed = lastByte ? 8 - BIT_highbit32(lastByte) : 0;
+ size_t const value = MEM_readLEST(ip) | 1;
+ assert(bitsConsumed <= 8);
+ return value << bitsConsumed;
+}
+typedef struct {
+ BYTE const* ip[4];
+ BYTE* op[4];
+ U64 bits[4];
+ void const* dt;
+ BYTE const* ilimit;
+ BYTE* oend;
+ BYTE const* iend[4];
+} HUF_DecompressAsmArgs;
+
+/*
+ * Initializes args for the asm decoding loop.
+ * @returns 0 on success
+ * 1 if the fallback implementation should be used.
+ * Or an error code on failure.
+ */
+static size_t HUF_DecompressAsmArgs_init(HUF_DecompressAsmArgs* args, void* dst, size_t dstSize, void const* src, size_t srcSize, const HUF_DTable* DTable)
+{
+ void const* dt = DTable + 1;
+ U32 const dtLog = HUF_getDTableDesc(DTable).tableLog;
+
+ const BYTE* const ilimit = (const BYTE*)src + 6 + 8;
+
+ BYTE* const oend = (BYTE*)dst + dstSize;
+
+ /* The following condition is false on x32 platform,
+ * but HUF_asm is not compatible with this ABI */
+ if (!(MEM_isLittleEndian() && !MEM_32bits())) return 1;
+
+ /* strict minimum : jump table + 1 byte per stream */
+ if (srcSize < 10)
+ return ERROR(corruption_detected);
+
+ /* Must have at least 8 bytes per stream because we don't handle initializing smaller bit containers.
+ * If table log is not correct at this point, fallback to the old decoder.
+ * On small inputs we don't have enough data to trigger the fast loop, so use the old decoder.
+ */
+ if (dtLog != HUF_DECODER_FAST_TABLELOG)
+ return 1;
+
+ /* Read the jump table. */
+ {
+ const BYTE* const istart = (const BYTE*)src;
+ size_t const length1 = MEM_readLE16(istart);
+ size_t const length2 = MEM_readLE16(istart+2);
+ size_t const length3 = MEM_readLE16(istart+4);
+ size_t const length4 = srcSize - (length1 + length2 + length3 + 6);
+ args->iend[0] = istart + 6; /* jumpTable */
+ args->iend[1] = args->iend[0] + length1;
+ args->iend[2] = args->iend[1] + length2;
+ args->iend[3] = args->iend[2] + length3;
+
+ /* HUF_initDStream() requires this, and this small of an input
+ * won't benefit from the ASM loop anyways.
+ * length1 must be >= 16 so that ip[0] >= ilimit before the loop
+ * starts.
+ */
+ if (length1 < 16 || length2 < 8 || length3 < 8 || length4 < 8)
+ return 1;
+ if (length4 > srcSize) return ERROR(corruption_detected); /* overflow */
+ }
+ /* ip[] contains the position that is currently loaded into bits[]. */
+ args->ip[0] = args->iend[1] - sizeof(U64);
+ args->ip[1] = args->iend[2] - sizeof(U64);
+ args->ip[2] = args->iend[3] - sizeof(U64);
+ args->ip[3] = (BYTE const*)src + srcSize - sizeof(U64);
+
+ /* op[] contains the output pointers. */
+ args->op[0] = (BYTE*)dst;
+ args->op[1] = args->op[0] + (dstSize+3)/4;
+ args->op[2] = args->op[1] + (dstSize+3)/4;
+ args->op[3] = args->op[2] + (dstSize+3)/4;
+
+ /* No point to call the ASM loop for tiny outputs. */
+ if (args->op[3] >= oend)
+ return 1;
+
+ /* bits[] is the bit container.
+ * It is read from the MSB down to the LSB.
+ * It is shifted left as it is read, and zeros are
+ * shifted in. After the lowest valid bit a 1 is
+ * set, so that CountTrailingZeros(bits[]) can be used
+ * to count how many bits we've consumed.
+ */
+ args->bits[0] = HUF_initDStream(args->ip[0]);
+ args->bits[1] = HUF_initDStream(args->ip[1]);
+ args->bits[2] = HUF_initDStream(args->ip[2]);
+ args->bits[3] = HUF_initDStream(args->ip[3]);
+
+ /* If ip[] >= ilimit, it is guaranteed to be safe to
+ * reload bits[]. It may be beyond its section, but is
+ * guaranteed to be valid (>= istart).
+ */
+ args->ilimit = ilimit;
+
+ args->oend = oend;
+ args->dt = dt;
+
+ return 0;
+}
+
+static size_t HUF_initRemainingDStream(BIT_DStream_t* bit, HUF_DecompressAsmArgs const* args, int stream, BYTE* segmentEnd)
+{
+ /* Validate that we haven't overwritten. */
+ if (args->op[stream] > segmentEnd)
+ return ERROR(corruption_detected);
+ /* Validate that we haven't read beyond iend[].
+ * Note that ip[] may be < iend[] because the MSB is
+ * the next bit to read, and we may have consumed 100%
+ * of the stream, so down to iend[i] - 8 is valid.
+ */
+ if (args->ip[stream] < args->iend[stream] - 8)
+ return ERROR(corruption_detected);
+
+ /* Construct the BIT_DStream_t. */
+ bit->bitContainer = MEM_readLE64(args->ip[stream]);
+ bit->bitsConsumed = ZSTD_countTrailingZeros((size_t)args->bits[stream]);
+ bit->start = (const char*)args->iend[0];
+ bit->limitPtr = bit->start + sizeof(size_t);
+ bit->ptr = (const char*)args->ip[stream];
+
+ return 0;
+}
+#endif
+
#ifndef HUF_FORCE_DECOMPRESS_X2
/*-***************************/
/* single-symbol decoding */
/*-***************************/
-typedef struct { BYTE byte; BYTE nbBits; } HUF_DEltX1; /* single-symbol decoding */
+typedef struct { BYTE nbBits; BYTE byte; } HUF_DEltX1; /* single-symbol decoding */
/*
* Packs 4 HUF_DEltX1 structs into a U64. This is used to lay down 4 entries at
@@ -122,14 +283,44 @@ typedef struct { BYTE byte; BYTE nbBits; } HUF_DEltX1; /* single-symbol decodi
static U64 HUF_DEltX1_set4(BYTE symbol, BYTE nbBits) {
U64 D4;
if (MEM_isLittleEndian()) {
- D4 = symbol + (nbBits << 8);
- } else {
D4 = (symbol << 8) + nbBits;
+ } else {
+ D4 = symbol + (nbBits << 8);
}
D4 *= 0x0001000100010001ULL;
return D4;
}
+/*
+ * Increase the tableLog to targetTableLog and rescales the stats.
+ * If tableLog > targetTableLog this is a no-op.
+ * @returns New tableLog
+ */
+static U32 HUF_rescaleStats(BYTE* huffWeight, U32* rankVal, U32 nbSymbols, U32 tableLog, U32 targetTableLog)
+{
+ if (tableLog > targetTableLog)
+ return tableLog;
+ if (tableLog < targetTableLog) {
+ U32 const scale = targetTableLog - tableLog;
+ U32 s;
+ /* Increase the weight for all non-zero probability symbols by scale. */
+ for (s = 0; s < nbSymbols; ++s) {
+ huffWeight[s] += (BYTE)((huffWeight[s] == 0) ? 0 : scale);
+ }
+ /* Update rankVal to reflect the new weights.
+ * All weights except 0 get moved to weight + scale.
+ * Weights [1, scale] are empty.
+ */
+ for (s = targetTableLog; s > scale; --s) {
+ rankVal[s] = rankVal[s - scale];
+ }
+ for (s = scale; s > 0; --s) {
+ rankVal[s] = 0;
+ }
+ }
+ return targetTableLog;
+}
+
typedef struct {
U32 rankVal[HUF_TABLELOG_ABSOLUTEMAX + 1];
U32 rankStart[HUF_TABLELOG_ABSOLUTEMAX + 1];
@@ -162,8 +353,12 @@ size_t HUF_readDTableX1_wksp_bmi2(HUF_DTable* DTable, const void* src, size_t sr
iSize = HUF_readStats_wksp(wksp->huffWeight, HUF_SYMBOLVALUE_MAX + 1, wksp->rankVal, &nbSymbols, &tableLog, src, srcSize, wksp->statsWksp, sizeof(wksp->statsWksp), bmi2);
if (HUF_isError(iSize)) return iSize;
+
/* Table header */
{ DTableDesc dtd = HUF_getDTableDesc(DTable);
+ U32 const maxTableLog = dtd.maxTableLog + 1;
+ U32 const targetTableLog = MIN(maxTableLog, HUF_DECODER_FAST_TABLELOG);
+ tableLog = HUF_rescaleStats(wksp->huffWeight, wksp->rankVal, nbSymbols, tableLog, targetTableLog);
if (tableLog > (U32)(dtd.maxTableLog+1)) return ERROR(tableLog_tooLarge); /* DTable too small, Huffman tree cannot fit in */
dtd.tableType = 0;
dtd.tableLog = (BYTE)tableLog;
@@ -207,7 +402,7 @@ size_t HUF_readDTableX1_wksp_bmi2(HUF_DTable* DTable, const void* src, size_t sr
/* fill DTable
* We fill all entries of each weight in order.
- * That way length is a constant for each iteration of the outter loop.
+ * That way length is a constant for each iteration of the outer loop.
* We can switch based on the length to a different inner loop which is
* optimized for that particular case.
*/
@@ -304,11 +499,15 @@ HUF_decodeStreamX1(BYTE* p, BIT_DStream_t* const bitDPtr, BYTE* const pEnd, cons
BYTE* const pStart = p;
/* up to 4 symbols at a time */
- while ((BIT_reloadDStream(bitDPtr) == BIT_DStream_unfinished) & (p < pEnd-3)) {
- HUF_DECODE_SYMBOLX1_2(p, bitDPtr);
- HUF_DECODE_SYMBOLX1_1(p, bitDPtr);
- HUF_DECODE_SYMBOLX1_2(p, bitDPtr);
- HUF_DECODE_SYMBOLX1_0(p, bitDPtr);
+ if ((pEnd - p) > 3) {
+ while ((BIT_reloadDStream(bitDPtr) == BIT_DStream_unfinished) & (p < pEnd-3)) {
+ HUF_DECODE_SYMBOLX1_2(p, bitDPtr);
+ HUF_DECODE_SYMBOLX1_1(p, bitDPtr);
+ HUF_DECODE_SYMBOLX1_2(p, bitDPtr);
+ HUF_DECODE_SYMBOLX1_0(p, bitDPtr);
+ }
+ } else {
+ BIT_reloadDStream(bitDPtr);
}
/* [0-3] symbols remaining */
@@ -388,33 +587,36 @@ HUF_decompress4X1_usingDTable_internal_body(
U32 endSignal = 1;
if (length4 > cSrcSize) return ERROR(corruption_detected); /* overflow */
+ if (opStart4 > oend) return ERROR(corruption_detected); /* overflow */
CHECK_F( BIT_initDStream(&bitD1, istart1, length1) );
CHECK_F( BIT_initDStream(&bitD2, istart2, length2) );
CHECK_F( BIT_initDStream(&bitD3, istart3, length3) );
CHECK_F( BIT_initDStream(&bitD4, istart4, length4) );
/* up to 16 symbols per loop (4 symbols per stream) in 64-bit mode */
- for ( ; (endSignal) & (op4 < olimit) ; ) {
- HUF_DECODE_SYMBOLX1_2(op1, &bitD1);
- HUF_DECODE_SYMBOLX1_2(op2, &bitD2);
- HUF_DECODE_SYMBOLX1_2(op3, &bitD3);
- HUF_DECODE_SYMBOLX1_2(op4, &bitD4);
- HUF_DECODE_SYMBOLX1_1(op1, &bitD1);
- HUF_DECODE_SYMBOLX1_1(op2, &bitD2);
- HUF_DECODE_SYMBOLX1_1(op3, &bitD3);
- HUF_DECODE_SYMBOLX1_1(op4, &bitD4);
- HUF_DECODE_SYMBOLX1_2(op1, &bitD1);
- HUF_DECODE_SYMBOLX1_2(op2, &bitD2);
- HUF_DECODE_SYMBOLX1_2(op3, &bitD3);
- HUF_DECODE_SYMBOLX1_2(op4, &bitD4);
- HUF_DECODE_SYMBOLX1_0(op1, &bitD1);
- HUF_DECODE_SYMBOLX1_0(op2, &bitD2);
- HUF_DECODE_SYMBOLX1_0(op3, &bitD3);
- HUF_DECODE_SYMBOLX1_0(op4, &bitD4);
- endSignal &= BIT_reloadDStreamFast(&bitD1) == BIT_DStream_unfinished;
- endSignal &= BIT_reloadDStreamFast(&bitD2) == BIT_DStream_unfinished;
- endSignal &= BIT_reloadDStreamFast(&bitD3) == BIT_DStream_unfinished;
- endSignal &= BIT_reloadDStreamFast(&bitD4) == BIT_DStream_unfinished;
+ if ((size_t)(oend - op4) >= sizeof(size_t)) {
+ for ( ; (endSignal) & (op4 < olimit) ; ) {
+ HUF_DECODE_SYMBOLX1_2(op1, &bitD1);
+ HUF_DECODE_SYMBOLX1_2(op2, &bitD2);
+ HUF_DECODE_SYMBOLX1_2(op3, &bitD3);
+ HUF_DECODE_SYMBOLX1_2(op4, &bitD4);
+ HUF_DECODE_SYMBOLX1_1(op1, &bitD1);
+ HUF_DECODE_SYMBOLX1_1(op2, &bitD2);
+ HUF_DECODE_SYMBOLX1_1(op3, &bitD3);
+ HUF_DECODE_SYMBOLX1_1(op4, &bitD4);
+ HUF_DECODE_SYMBOLX1_2(op1, &bitD1);
+ HUF_DECODE_SYMBOLX1_2(op2, &bitD2);
+ HUF_DECODE_SYMBOLX1_2(op3, &bitD3);
+ HUF_DECODE_SYMBOLX1_2(op4, &bitD4);
+ HUF_DECODE_SYMBOLX1_0(op1, &bitD1);
+ HUF_DECODE_SYMBOLX1_0(op2, &bitD2);
+ HUF_DECODE_SYMBOLX1_0(op3, &bitD3);
+ HUF_DECODE_SYMBOLX1_0(op4, &bitD4);
+ endSignal &= BIT_reloadDStreamFast(&bitD1) == BIT_DStream_unfinished;
+ endSignal &= BIT_reloadDStreamFast(&bitD2) == BIT_DStream_unfinished;
+ endSignal &= BIT_reloadDStreamFast(&bitD3) == BIT_DStream_unfinished;
+ endSignal &= BIT_reloadDStreamFast(&bitD4) == BIT_DStream_unfinished;
+ }
}
/* check corruption */
@@ -440,6 +642,79 @@ HUF_decompress4X1_usingDTable_internal_body(
}
}
+#if HUF_NEED_BMI2_FUNCTION
+static BMI2_TARGET_ATTRIBUTE
+size_t HUF_decompress4X1_usingDTable_internal_bmi2(void* dst, size_t dstSize, void const* cSrc,
+ size_t cSrcSize, HUF_DTable const* DTable) {
+ return HUF_decompress4X1_usingDTable_internal_body(dst, dstSize, cSrc, cSrcSize, DTable);
+}
+#endif
+
+#if HUF_NEED_DEFAULT_FUNCTION
+static
+size_t HUF_decompress4X1_usingDTable_internal_default(void* dst, size_t dstSize, void const* cSrc,
+ size_t cSrcSize, HUF_DTable const* DTable) {
+ return HUF_decompress4X1_usingDTable_internal_body(dst, dstSize, cSrc, cSrcSize, DTable);
+}
+#endif
+
+#if ZSTD_ENABLE_ASM_X86_64_BMI2
+
+HUF_ASM_DECL void HUF_decompress4X1_usingDTable_internal_bmi2_asm_loop(HUF_DecompressAsmArgs* args) ZSTDLIB_HIDDEN;
+
+static HUF_ASM_X86_64_BMI2_ATTRS
+size_t
+HUF_decompress4X1_usingDTable_internal_bmi2_asm(
+ void* dst, size_t dstSize,
+ const void* cSrc, size_t cSrcSize,
+ const HUF_DTable* DTable)
+{
+ void const* dt = DTable + 1;
+ const BYTE* const iend = (const BYTE*)cSrc + 6;
+ BYTE* const oend = (BYTE*)dst + dstSize;
+ HUF_DecompressAsmArgs args;
+ {
+ size_t const ret = HUF_DecompressAsmArgs_init(&args, dst, dstSize, cSrc, cSrcSize, DTable);
+ FORWARD_IF_ERROR(ret, "Failed to init asm args");
+ if (ret != 0)
+ return HUF_decompress4X1_usingDTable_internal_bmi2(dst, dstSize, cSrc, cSrcSize, DTable);
+ }
+
+ assert(args.ip[0] >= args.ilimit);
+ HUF_decompress4X1_usingDTable_internal_bmi2_asm_loop(&args);
+
+ /* Our loop guarantees that ip[] >= ilimit and that we haven't
+ * overwritten any op[].
+ */
+ assert(args.ip[0] >= iend);
+ assert(args.ip[1] >= iend);
+ assert(args.ip[2] >= iend);
+ assert(args.ip[3] >= iend);
+ assert(args.op[3] <= oend);
+ (void)iend;
+
+ /* finish bit streams one by one. */
+ {
+ size_t const segmentSize = (dstSize+3) / 4;
+ BYTE* segmentEnd = (BYTE*)dst;
+ int i;
+ for (i = 0; i < 4; ++i) {
+ BIT_DStream_t bit;
+ if (segmentSize <= (size_t)(oend - segmentEnd))
+ segmentEnd += segmentSize;
+ else
+ segmentEnd = oend;
+ FORWARD_IF_ERROR(HUF_initRemainingDStream(&bit, &args, i, segmentEnd), "corruption");
+ /* Decompress and validate that we've produced exactly the expected length. */
+ args.op[i] += HUF_decodeStreamX1(args.op[i], &bit, segmentEnd, (HUF_DEltX1 const*)dt, HUF_DECODER_FAST_TABLELOG);
+ if (args.op[i] != segmentEnd) return ERROR(corruption_detected);
+ }
+ }
+
+ /* decoded size */
+ return dstSize;
+}
+#endif /* ZSTD_ENABLE_ASM_X86_64_BMI2 */
typedef size_t (*HUF_decompress_usingDTable_t)(void *dst, size_t dstSize,
const void *cSrc,
@@ -447,8 +722,28 @@ typedef size_t (*HUF_decompress_usingDTable_t)(void *dst, size_t dstSize,
const HUF_DTable *DTable);
HUF_DGEN(HUF_decompress1X1_usingDTable_internal)
-HUF_DGEN(HUF_decompress4X1_usingDTable_internal)
+static size_t HUF_decompress4X1_usingDTable_internal(void* dst, size_t dstSize, void const* cSrc,
+ size_t cSrcSize, HUF_DTable const* DTable, int bmi2)
+{
+#if DYNAMIC_BMI2
+ if (bmi2) {
+# if ZSTD_ENABLE_ASM_X86_64_BMI2
+ return HUF_decompress4X1_usingDTable_internal_bmi2_asm(dst, dstSize, cSrc, cSrcSize, DTable);
+# else
+ return HUF_decompress4X1_usingDTable_internal_bmi2(dst, dstSize, cSrc, cSrcSize, DTable);
+# endif
+ }
+#else
+ (void)bmi2;
+#endif
+
+#if ZSTD_ENABLE_ASM_X86_64_BMI2 && defined(__BMI2__)
+ return HUF_decompress4X1_usingDTable_internal_bmi2_asm(dst, dstSize, cSrc, cSrcSize, DTable);
+#else
+ return HUF_decompress4X1_usingDTable_internal_default(dst, dstSize, cSrc, cSrcSize, DTable);
+#endif
+}
size_t HUF_decompress1X1_usingDTable(
@@ -518,106 +813,226 @@ size_t HUF_decompress4X1_DCtx_wksp(HUF_DTable* dctx, void* dst, size_t dstSize,
/* *************************/
typedef struct { U16 sequence; BYTE nbBits; BYTE length; } HUF_DEltX2; /* double-symbols decoding */
-typedef struct { BYTE symbol; BYTE weight; } sortedSymbol_t;
+typedef struct { BYTE symbol; } sortedSymbol_t;
typedef U32 rankValCol_t[HUF_TABLELOG_MAX + 1];
typedef rankValCol_t rankVal_t[HUF_TABLELOG_MAX];
+/*
+ * Constructs a HUF_DEltX2 in a U32.
+ */
+static U32 HUF_buildDEltX2U32(U32 symbol, U32 nbBits, U32 baseSeq, int level)
+{
+ U32 seq;
+ DEBUG_STATIC_ASSERT(offsetof(HUF_DEltX2, sequence) == 0);
+ DEBUG_STATIC_ASSERT(offsetof(HUF_DEltX2, nbBits) == 2);
+ DEBUG_STATIC_ASSERT(offsetof(HUF_DEltX2, length) == 3);
+ DEBUG_STATIC_ASSERT(sizeof(HUF_DEltX2) == sizeof(U32));
+ if (MEM_isLittleEndian()) {
+ seq = level == 1 ? symbol : (baseSeq + (symbol << 8));
+ return seq + (nbBits << 16) + ((U32)level << 24);
+ } else {
+ seq = level == 1 ? (symbol << 8) : ((baseSeq << 8) + symbol);
+ return (seq << 16) + (nbBits << 8) + (U32)level;
+ }
+}
+
+/*
+ * Constructs a HUF_DEltX2.
+ */
+static HUF_DEltX2 HUF_buildDEltX2(U32 symbol, U32 nbBits, U32 baseSeq, int level)
+{
+ HUF_DEltX2 DElt;
+ U32 const val = HUF_buildDEltX2U32(symbol, nbBits, baseSeq, level);
+ DEBUG_STATIC_ASSERT(sizeof(DElt) == sizeof(val));
+ ZSTD_memcpy(&DElt, &val, sizeof(val));
+ return DElt;
+}
+
+/*
+ * Constructs 2 HUF_DEltX2s and packs them into a U64.
+ */
+static U64 HUF_buildDEltX2U64(U32 symbol, U32 nbBits, U16 baseSeq, int level)
+{
+ U32 DElt = HUF_buildDEltX2U32(symbol, nbBits, baseSeq, level);
+ return (U64)DElt + ((U64)DElt << 32);
+}
+
+/*
+ * Fills the DTable rank with all the symbols from [begin, end) that are each
+ * nbBits long.
+ *
+ * @param DTableRank The start of the rank in the DTable.
+ * @param begin The first symbol to fill (inclusive).
+ * @param end The last symbol to fill (exclusive).
+ * @param nbBits Each symbol is nbBits long.
+ * @param tableLog The table log.
+ * @param baseSeq If level == 1 { 0 } else { the first level symbol }
+ * @param level The level in the table. Must be 1 or 2.
+ */
+static void HUF_fillDTableX2ForWeight(
+ HUF_DEltX2* DTableRank,
+ sortedSymbol_t const* begin, sortedSymbol_t const* end,
+ U32 nbBits, U32 tableLog,
+ U16 baseSeq, int const level)
+{
+ U32 const length = 1U << ((tableLog - nbBits) & 0x1F /* quiet static-analyzer */);
+ const sortedSymbol_t* ptr;
+ assert(level >= 1 && level <= 2);
+ switch (length) {
+ case 1:
+ for (ptr = begin; ptr != end; ++ptr) {
+ HUF_DEltX2 const DElt = HUF_buildDEltX2(ptr->symbol, nbBits, baseSeq, level);
+ *DTableRank++ = DElt;
+ }
+ break;
+ case 2:
+ for (ptr = begin; ptr != end; ++ptr) {
+ HUF_DEltX2 const DElt = HUF_buildDEltX2(ptr->symbol, nbBits, baseSeq, level);
+ DTableRank[0] = DElt;
+ DTableRank[1] = DElt;
+ DTableRank += 2;
+ }
+ break;
+ case 4:
+ for (ptr = begin; ptr != end; ++ptr) {
+ U64 const DEltX2 = HUF_buildDEltX2U64(ptr->symbol, nbBits, baseSeq, level);
+ ZSTD_memcpy(DTableRank + 0, &DEltX2, sizeof(DEltX2));
+ ZSTD_memcpy(DTableRank + 2, &DEltX2, sizeof(DEltX2));
+ DTableRank += 4;
+ }
+ break;
+ case 8:
+ for (ptr = begin; ptr != end; ++ptr) {
+ U64 const DEltX2 = HUF_buildDEltX2U64(ptr->symbol, nbBits, baseSeq, level);
+ ZSTD_memcpy(DTableRank + 0, &DEltX2, sizeof(DEltX2));
+ ZSTD_memcpy(DTableRank + 2, &DEltX2, sizeof(DEltX2));
+ ZSTD_memcpy(DTableRank + 4, &DEltX2, sizeof(DEltX2));
+ ZSTD_memcpy(DTableRank + 6, &DEltX2, sizeof(DEltX2));
+ DTableRank += 8;
+ }
+ break;
+ default:
+ for (ptr = begin; ptr != end; ++ptr) {
+ U64 const DEltX2 = HUF_buildDEltX2U64(ptr->symbol, nbBits, baseSeq, level);
+ HUF_DEltX2* const DTableRankEnd = DTableRank + length;
+ for (; DTableRank != DTableRankEnd; DTableRank += 8) {
+ ZSTD_memcpy(DTableRank + 0, &DEltX2, sizeof(DEltX2));
+ ZSTD_memcpy(DTableRank + 2, &DEltX2, sizeof(DEltX2));
+ ZSTD_memcpy(DTableRank + 4, &DEltX2, sizeof(DEltX2));
+ ZSTD_memcpy(DTableRank + 6, &DEltX2, sizeof(DEltX2));
+ }
+ }
+ break;
+ }
+}
/* HUF_fillDTableX2Level2() :
* `rankValOrigin` must be a table of at least (HUF_TABLELOG_MAX + 1) U32 */
-static void HUF_fillDTableX2Level2(HUF_DEltX2* DTable, U32 sizeLog, const U32 consumed,
- const U32* rankValOrigin, const int minWeight,
- const sortedSymbol_t* sortedSymbols, const U32 sortedListSize,
- U32 nbBitsBaseline, U16 baseSeq, U32* wksp, size_t wkspSize)
+static void HUF_fillDTableX2Level2(HUF_DEltX2* DTable, U32 targetLog, const U32 consumedBits,
+ const U32* rankVal, const int minWeight, const int maxWeight1,
+ const sortedSymbol_t* sortedSymbols, U32 const* rankStart,
+ U32 nbBitsBaseline, U16 baseSeq)
{
- HUF_DEltX2 DElt;
- U32* rankVal = wksp;
-
- assert(wkspSize >= HUF_TABLELOG_MAX + 1);
- (void)wkspSize;
- /* get pre-calculated rankVal */
- ZSTD_memcpy(rankVal, rankValOrigin, sizeof(U32) * (HUF_TABLELOG_MAX + 1));
-
- /* fill skipped values */
+ /* Fill skipped values (all positions up to rankVal[minWeight]).
+ * These are positions only get a single symbol because the combined weight
+ * is too large.
+ */
if (minWeight>1) {
- U32 i, skipSize = rankVal[minWeight];
- MEM_writeLE16(&(DElt.sequence), baseSeq);
- DElt.nbBits = (BYTE)(consumed);
- DElt.length = 1;
- for (i = 0; i < skipSize; i++)
- DTable[i] = DElt;
+ U32 const length = 1U << ((targetLog - consumedBits) & 0x1F /* quiet static-analyzer */);
+ U64 const DEltX2 = HUF_buildDEltX2U64(baseSeq, consumedBits, /* baseSeq */ 0, /* level */ 1);
+ int const skipSize = rankVal[minWeight];
+ assert(length > 1);
+ assert((U32)skipSize < length);
+ switch (length) {
+ case 2:
+ assert(skipSize == 1);
+ ZSTD_memcpy(DTable, &DEltX2, sizeof(DEltX2));
+ break;
+ case 4:
+ assert(skipSize <= 4);
+ ZSTD_memcpy(DTable + 0, &DEltX2, sizeof(DEltX2));
+ ZSTD_memcpy(DTable + 2, &DEltX2, sizeof(DEltX2));
+ break;
+ default:
+ {
+ int i;
+ for (i = 0; i < skipSize; i += 8) {
+ ZSTD_memcpy(DTable + i + 0, &DEltX2, sizeof(DEltX2));
+ ZSTD_memcpy(DTable + i + 2, &DEltX2, sizeof(DEltX2));
+ ZSTD_memcpy(DTable + i + 4, &DEltX2, sizeof(DEltX2));
+ ZSTD_memcpy(DTable + i + 6, &DEltX2, sizeof(DEltX2));
+ }
+ }
+ }
}
- /* fill DTable */
- { U32 s; for (s=0; s<sortedListSize; s++) { /* note : sortedSymbols already skipped */
- const U32 symbol = sortedSymbols[s].symbol;
- const U32 weight = sortedSymbols[s].weight;
- const U32 nbBits = nbBitsBaseline - weight;
- const U32 length = 1 << (sizeLog-nbBits);
- const U32 start = rankVal[weight];
- U32 i = start;
- const U32 end = start + length;
-
- MEM_writeLE16(&(DElt.sequence), (U16)(baseSeq + (symbol << 8)));
- DElt.nbBits = (BYTE)(nbBits + consumed);
- DElt.length = 2;
- do { DTable[i++] = DElt; } while (i<end); /* since length >= 1 */
-
- rankVal[weight] += length;
- } }
+ /* Fill each of the second level symbols by weight. */
+ {
+ int w;
+ for (w = minWeight; w < maxWeight1; ++w) {
+ int const begin = rankStart[w];
+ int const end = rankStart[w+1];
+ U32 const nbBits = nbBitsBaseline - w;
+ U32 const totalBits = nbBits + consumedBits;
+ HUF_fillDTableX2ForWeight(
+ DTable + rankVal[w],
+ sortedSymbols + begin, sortedSymbols + end,
+ totalBits, targetLog,
+ baseSeq, /* level */ 2);
+ }
+ }
}
-
static void HUF_fillDTableX2(HUF_DEltX2* DTable, const U32 targetLog,
- const sortedSymbol_t* sortedList, const U32 sortedListSize,
+ const sortedSymbol_t* sortedList,
const U32* rankStart, rankVal_t rankValOrigin, const U32 maxWeight,
- const U32 nbBitsBaseline, U32* wksp, size_t wkspSize)
+ const U32 nbBitsBaseline)
{
- U32* rankVal = wksp;
+ U32* const rankVal = rankValOrigin[0];
const int scaleLog = nbBitsBaseline - targetLog; /* note : targetLog >= srcLog, hence scaleLog <= 1 */
const U32 minBits = nbBitsBaseline - maxWeight;
- U32 s;
+ int w;
+ int const wEnd = (int)maxWeight + 1;
- assert(wkspSize >= HUF_TABLELOG_MAX + 1);
- wksp += HUF_TABLELOG_MAX + 1;
- wkspSize -= HUF_TABLELOG_MAX + 1;
+ /* Fill DTable in order of weight. */
+ for (w = 1; w < wEnd; ++w) {
+ int const begin = (int)rankStart[w];
+ int const end = (int)rankStart[w+1];
+ U32 const nbBits = nbBitsBaseline - w;
- ZSTD_memcpy(rankVal, rankValOrigin, sizeof(U32) * (HUF_TABLELOG_MAX + 1));
-
- /* fill DTable */
- for (s=0; s<sortedListSize; s++) {
- const U16 symbol = sortedList[s].symbol;
- const U32 weight = sortedList[s].weight;
- const U32 nbBits = nbBitsBaseline - weight;
- const U32 start = rankVal[weight];
- const U32 length = 1 << (targetLog-nbBits);
-
- if (targetLog-nbBits >= minBits) { /* enough room for a second symbol */
- U32 sortedRank;
+ if (targetLog-nbBits >= minBits) {
+ /* Enough room for a second symbol. */
+ int start = rankVal[w];
+ U32 const length = 1U << ((targetLog - nbBits) & 0x1F /* quiet static-analyzer */);
int minWeight = nbBits + scaleLog;
+ int s;
if (minWeight < 1) minWeight = 1;
- sortedRank = rankStart[minWeight];
- HUF_fillDTableX2Level2(DTable+start, targetLog-nbBits, nbBits,
- rankValOrigin[nbBits], minWeight,
- sortedList+sortedRank, sortedListSize-sortedRank,
- nbBitsBaseline, symbol, wksp, wkspSize);
+ /* Fill the DTable for every symbol of weight w.
+ * These symbols get at least 1 second symbol.
+ */
+ for (s = begin; s != end; ++s) {
+ HUF_fillDTableX2Level2(
+ DTable + start, targetLog, nbBits,
+ rankValOrigin[nbBits], minWeight, wEnd,
+ sortedList, rankStart,
+ nbBitsBaseline, sortedList[s].symbol);
+ start += length;
+ }
} else {
- HUF_DEltX2 DElt;
- MEM_writeLE16(&(DElt.sequence), symbol);
- DElt.nbBits = (BYTE)(nbBits);
- DElt.length = 1;
- { U32 const end = start + length;
- U32 u;
- for (u = start; u < end; u++) DTable[u] = DElt;
- } }
- rankVal[weight] += length;
+ /* Only a single symbol. */
+ HUF_fillDTableX2ForWeight(
+ DTable + rankVal[w],
+ sortedList + begin, sortedList + end,
+ nbBits, targetLog,
+ /* baseSeq */ 0, /* level */ 1);
+ }
}
}
typedef struct {
rankValCol_t rankVal[HUF_TABLELOG_MAX];
U32 rankStats[HUF_TABLELOG_MAX + 1];
- U32 rankStart0[HUF_TABLELOG_MAX + 2];
+ U32 rankStart0[HUF_TABLELOG_MAX + 3];
sortedSymbol_t sortedSymbol[HUF_SYMBOLVALUE_MAX + 1];
BYTE weightList[HUF_SYMBOLVALUE_MAX + 1];
U32 calleeWksp[HUF_READ_STATS_WORKSPACE_SIZE_U32];
@@ -627,9 +1042,16 @@ size_t HUF_readDTableX2_wksp(HUF_DTable* DTable,
const void* src, size_t srcSize,
void* workSpace, size_t wkspSize)
{
- U32 tableLog, maxW, sizeOfSort, nbSymbols;
+ return HUF_readDTableX2_wksp_bmi2(DTable, src, srcSize, workSpace, wkspSize, /* bmi2 */ 0);
+}
+
+size_t HUF_readDTableX2_wksp_bmi2(HUF_DTable* DTable,
+ const void* src, size_t srcSize,
+ void* workSpace, size_t wkspSize, int bmi2)
+{
+ U32 tableLog, maxW, nbSymbols;
DTableDesc dtd = HUF_getDTableDesc(DTable);
- U32 const maxTableLog = dtd.maxTableLog;
+ U32 maxTableLog = dtd.maxTableLog;
size_t iSize;
void* dtPtr = DTable+1; /* force compiler to avoid strict-aliasing */
HUF_DEltX2* const dt = (HUF_DEltX2*)dtPtr;
@@ -647,11 +1069,12 @@ size_t HUF_readDTableX2_wksp(HUF_DTable* DTable,
if (maxTableLog > HUF_TABLELOG_MAX) return ERROR(tableLog_tooLarge);
/* ZSTD_memset(weightList, 0, sizeof(weightList)); */ /* is not necessary, even though some analyzer complain ... */
- iSize = HUF_readStats_wksp(wksp->weightList, HUF_SYMBOLVALUE_MAX + 1, wksp->rankStats, &nbSymbols, &tableLog, src, srcSize, wksp->calleeWksp, sizeof(wksp->calleeWksp), /* bmi2 */ 0);
+ iSize = HUF_readStats_wksp(wksp->weightList, HUF_SYMBOLVALUE_MAX + 1, wksp->rankStats, &nbSymbols, &tableLog, src, srcSize, wksp->calleeWksp, sizeof(wksp->calleeWksp), bmi2);
if (HUF_isError(iSize)) return iSize;
/* check result */
if (tableLog > maxTableLog) return ERROR(tableLog_tooLarge); /* DTable can't fit code depth */
+ if (tableLog <= HUF_DECODER_FAST_TABLELOG && maxTableLog > HUF_DECODER_FAST_TABLELOG) maxTableLog = HUF_DECODER_FAST_TABLELOG;
/* find maxWeight */
for (maxW = tableLog; wksp->rankStats[maxW]==0; maxW--) {} /* necessarily finds a solution before 0 */
@@ -664,7 +1087,7 @@ size_t HUF_readDTableX2_wksp(HUF_DTable* DTable,
rankStart[w] = curr;
}
rankStart[0] = nextRankStart; /* put all 0w symbols at the end of sorted list*/
- sizeOfSort = nextRankStart;
+ rankStart[maxW+1] = nextRankStart;
}
/* sort symbols by weight */
@@ -673,7 +1096,6 @@ size_t HUF_readDTableX2_wksp(HUF_DTable* DTable,
U32 const w = wksp->weightList[s];
U32 const r = rankStart[w]++;
wksp->sortedSymbol[r].symbol = (BYTE)s;
- wksp->sortedSymbol[r].weight = (BYTE)w;
}
rankStart[0] = 0; /* forget 0w symbols; this is beginning of weight(1) */
}
@@ -698,10 +1120,9 @@ size_t HUF_readDTableX2_wksp(HUF_DTable* DTable,
} } } }
HUF_fillDTableX2(dt, maxTableLog,
- wksp->sortedSymbol, sizeOfSort,
+ wksp->sortedSymbol,
wksp->rankStart0, wksp->rankVal, maxW,
- tableLog+1,
- wksp->calleeWksp, sizeof(wksp->calleeWksp) / sizeof(U32));
+ tableLog+1);
dtd.tableLog = (BYTE)maxTableLog;
dtd.tableType = 1;
@@ -714,7 +1135,7 @@ FORCE_INLINE_TEMPLATE U32
HUF_decodeSymbolX2(void* op, BIT_DStream_t* DStream, const HUF_DEltX2* dt, const U32 dtLog)
{
size_t const val = BIT_lookBitsFast(DStream, dtLog); /* note : dtLog >= 1 */
- ZSTD_memcpy(op, dt+val, 2);
+ ZSTD_memcpy(op, &dt[val].sequence, 2);
BIT_skipBits(DStream, dt[val].nbBits);
return dt[val].length;
}
@@ -723,15 +1144,17 @@ FORCE_INLINE_TEMPLATE U32
HUF_decodeLastSymbolX2(void* op, BIT_DStream_t* DStream, const HUF_DEltX2* dt, const U32 dtLog)
{
size_t const val = BIT_lookBitsFast(DStream, dtLog); /* note : dtLog >= 1 */
- ZSTD_memcpy(op, dt+val, 1);
- if (dt[val].length==1) BIT_skipBits(DStream, dt[val].nbBits);
- else {
+ ZSTD_memcpy(op, &dt[val].sequence, 1);
+ if (dt[val].length==1) {
+ BIT_skipBits(DStream, dt[val].nbBits);
+ } else {
if (DStream->bitsConsumed < (sizeof(DStream->bitContainer)*8)) {
BIT_skipBits(DStream, dt[val].nbBits);
if (DStream->bitsConsumed > (sizeof(DStream->bitContainer)*8))
/* ugly hack; works only because it's the last symbol. Note : can't easily extract nbBits from just this symbol */
DStream->bitsConsumed = (sizeof(DStream->bitContainer)*8);
- } }
+ }
+ }
return 1;
}
@@ -753,19 +1176,37 @@ HUF_decodeStreamX2(BYTE* p, BIT_DStream_t* bitDPtr, BYTE* const pEnd,
BYTE* const pStart = p;
/* up to 8 symbols at a time */
- while ((BIT_reloadDStream(bitDPtr) == BIT_DStream_unfinished) & (p < pEnd-(sizeof(bitDPtr->bitContainer)-1))) {
- HUF_DECODE_SYMBOLX2_2(p, bitDPtr);
- HUF_DECODE_SYMBOLX2_1(p, bitDPtr);
- HUF_DECODE_SYMBOLX2_2(p, bitDPtr);
- HUF_DECODE_SYMBOLX2_0(p, bitDPtr);
+ if ((size_t)(pEnd - p) >= sizeof(bitDPtr->bitContainer)) {
+ if (dtLog <= 11 && MEM_64bits()) {
+ /* up to 10 symbols at a time */
+ while ((BIT_reloadDStream(bitDPtr) == BIT_DStream_unfinished) & (p < pEnd-9)) {
+ HUF_DECODE_SYMBOLX2_0(p, bitDPtr);
+ HUF_DECODE_SYMBOLX2_0(p, bitDPtr);
+ HUF_DECODE_SYMBOLX2_0(p, bitDPtr);
+ HUF_DECODE_SYMBOLX2_0(p, bitDPtr);
+ HUF_DECODE_SYMBOLX2_0(p, bitDPtr);
+ }
+ } else {
+ /* up to 8 symbols at a time */
+ while ((BIT_reloadDStream(bitDPtr) == BIT_DStream_unfinished) & (p < pEnd-(sizeof(bitDPtr->bitContainer)-1))) {
+ HUF_DECODE_SYMBOLX2_2(p, bitDPtr);
+ HUF_DECODE_SYMBOLX2_1(p, bitDPtr);
+ HUF_DECODE_SYMBOLX2_2(p, bitDPtr);
+ HUF_DECODE_SYMBOLX2_0(p, bitDPtr);
+ }
+ }
+ } else {
+ BIT_reloadDStream(bitDPtr);
}
/* closer to end : up to 2 symbols at a time */
- while ((BIT_reloadDStream(bitDPtr) == BIT_DStream_unfinished) & (p <= pEnd-2))
- HUF_DECODE_SYMBOLX2_0(p, bitDPtr);
+ if ((size_t)(pEnd - p) >= 2) {
+ while ((BIT_reloadDStream(bitDPtr) == BIT_DStream_unfinished) & (p <= pEnd-2))
+ HUF_DECODE_SYMBOLX2_0(p, bitDPtr);
- while (p <= pEnd-2)
- HUF_DECODE_SYMBOLX2_0(p, bitDPtr); /* no need to reload : reached the end of DStream */
+ while (p <= pEnd-2)
+ HUF_DECODE_SYMBOLX2_0(p, bitDPtr); /* no need to reload : reached the end of DStream */
+ }
if (p < pEnd)
p += HUF_decodeLastSymbolX2(p, bitDPtr, dt, dtLog);
@@ -799,7 +1240,6 @@ HUF_decompress1X2_usingDTable_internal_body(
/* decoded size */
return dstSize;
}
-
FORCE_INLINE_TEMPLATE size_t
HUF_decompress4X2_usingDTable_internal_body(
void* dst, size_t dstSize,
@@ -841,57 +1281,60 @@ HUF_decompress4X2_usingDTable_internal_body(
U32 const dtLog = dtd.tableLog;
if (length4 > cSrcSize) return ERROR(corruption_detected); /* overflow */
+ if (opStart4 > oend) return ERROR(corruption_detected); /* overflow */
CHECK_F( BIT_initDStream(&bitD1, istart1, length1) );
CHECK_F( BIT_initDStream(&bitD2, istart2, length2) );
CHECK_F( BIT_initDStream(&bitD3, istart3, length3) );
CHECK_F( BIT_initDStream(&bitD4, istart4, length4) );
/* 16-32 symbols per loop (4-8 symbols per stream) */
- for ( ; (endSignal) & (op4 < olimit); ) {
+ if ((size_t)(oend - op4) >= sizeof(size_t)) {
+ for ( ; (endSignal) & (op4 < olimit); ) {
#if defined(__clang__) && (defined(__x86_64__) || defined(__i386__))
- HUF_DECODE_SYMBOLX2_2(op1, &bitD1);
- HUF_DECODE_SYMBOLX2_1(op1, &bitD1);
- HUF_DECODE_SYMBOLX2_2(op1, &bitD1);
- HUF_DECODE_SYMBOLX2_0(op1, &bitD1);
- HUF_DECODE_SYMBOLX2_2(op2, &bitD2);
- HUF_DECODE_SYMBOLX2_1(op2, &bitD2);
- HUF_DECODE_SYMBOLX2_2(op2, &bitD2);
- HUF_DECODE_SYMBOLX2_0(op2, &bitD2);
- endSignal &= BIT_reloadDStreamFast(&bitD1) == BIT_DStream_unfinished;
- endSignal &= BIT_reloadDStreamFast(&bitD2) == BIT_DStream_unfinished;
- HUF_DECODE_SYMBOLX2_2(op3, &bitD3);
- HUF_DECODE_SYMBOLX2_1(op3, &bitD3);
- HUF_DECODE_SYMBOLX2_2(op3, &bitD3);
- HUF_DECODE_SYMBOLX2_0(op3, &bitD3);
- HUF_DECODE_SYMBOLX2_2(op4, &bitD4);
- HUF_DECODE_SYMBOLX2_1(op4, &bitD4);
- HUF_DECODE_SYMBOLX2_2(op4, &bitD4);
- HUF_DECODE_SYMBOLX2_0(op4, &bitD4);
- endSignal &= BIT_reloadDStreamFast(&bitD3) == BIT_DStream_unfinished;
- endSignal &= BIT_reloadDStreamFast(&bitD4) == BIT_DStream_unfinished;
+ HUF_DECODE_SYMBOLX2_2(op1, &bitD1);
+ HUF_DECODE_SYMBOLX2_1(op1, &bitD1);
+ HUF_DECODE_SYMBOLX2_2(op1, &bitD1);
+ HUF_DECODE_SYMBOLX2_0(op1, &bitD1);
+ HUF_DECODE_SYMBOLX2_2(op2, &bitD2);
+ HUF_DECODE_SYMBOLX2_1(op2, &bitD2);
+ HUF_DECODE_SYMBOLX2_2(op2, &bitD2);
+ HUF_DECODE_SYMBOLX2_0(op2, &bitD2);
+ endSignal &= BIT_reloadDStreamFast(&bitD1) == BIT_DStream_unfinished;
+ endSignal &= BIT_reloadDStreamFast(&bitD2) == BIT_DStream_unfinished;
+ HUF_DECODE_SYMBOLX2_2(op3, &bitD3);
+ HUF_DECODE_SYMBOLX2_1(op3, &bitD3);
+ HUF_DECODE_SYMBOLX2_2(op3, &bitD3);
+ HUF_DECODE_SYMBOLX2_0(op3, &bitD3);
+ HUF_DECODE_SYMBOLX2_2(op4, &bitD4);
+ HUF_DECODE_SYMBOLX2_1(op4, &bitD4);
+ HUF_DECODE_SYMBOLX2_2(op4, &bitD4);
+ HUF_DECODE_SYMBOLX2_0(op4, &bitD4);
+ endSignal &= BIT_reloadDStreamFast(&bitD3) == BIT_DStream_unfinished;
+ endSignal &= BIT_reloadDStreamFast(&bitD4) == BIT_DStream_unfinished;
#else
- HUF_DECODE_SYMBOLX2_2(op1, &bitD1);
- HUF_DECODE_SYMBOLX2_2(op2, &bitD2);
- HUF_DECODE_SYMBOLX2_2(op3, &bitD3);
- HUF_DECODE_SYMBOLX2_2(op4, &bitD4);
- HUF_DECODE_SYMBOLX2_1(op1, &bitD1);
- HUF_DECODE_SYMBOLX2_1(op2, &bitD2);
- HUF_DECODE_SYMBOLX2_1(op3, &bitD3);
- HUF_DECODE_SYMBOLX2_1(op4, &bitD4);
- HUF_DECODE_SYMBOLX2_2(op1, &bitD1);
- HUF_DECODE_SYMBOLX2_2(op2, &bitD2);
- HUF_DECODE_SYMBOLX2_2(op3, &bitD3);
- HUF_DECODE_SYMBOLX2_2(op4, &bitD4);
- HUF_DECODE_SYMBOLX2_0(op1, &bitD1);
- HUF_DECODE_SYMBOLX2_0(op2, &bitD2);
- HUF_DECODE_SYMBOLX2_0(op3, &bitD3);
- HUF_DECODE_SYMBOLX2_0(op4, &bitD4);
- endSignal = (U32)LIKELY((U32)
- (BIT_reloadDStreamFast(&bitD1) == BIT_DStream_unfinished)
- & (BIT_reloadDStreamFast(&bitD2) == BIT_DStream_unfinished)
- & (BIT_reloadDStreamFast(&bitD3) == BIT_DStream_unfinished)
- & (BIT_reloadDStreamFast(&bitD4) == BIT_DStream_unfinished));
+ HUF_DECODE_SYMBOLX2_2(op1, &bitD1);
+ HUF_DECODE_SYMBOLX2_2(op2, &bitD2);
+ HUF_DECODE_SYMBOLX2_2(op3, &bitD3);
+ HUF_DECODE_SYMBOLX2_2(op4, &bitD4);
+ HUF_DECODE_SYMBOLX2_1(op1, &bitD1);
+ HUF_DECODE_SYMBOLX2_1(op2, &bitD2);
+ HUF_DECODE_SYMBOLX2_1(op3, &bitD3);
+ HUF_DECODE_SYMBOLX2_1(op4, &bitD4);
+ HUF_DECODE_SYMBOLX2_2(op1, &bitD1);
+ HUF_DECODE_SYMBOLX2_2(op2, &bitD2);
+ HUF_DECODE_SYMBOLX2_2(op3, &bitD3);
+ HUF_DECODE_SYMBOLX2_2(op4, &bitD4);
+ HUF_DECODE_SYMBOLX2_0(op1, &bitD1);
+ HUF_DECODE_SYMBOLX2_0(op2, &bitD2);
+ HUF_DECODE_SYMBOLX2_0(op3, &bitD3);
+ HUF_DECODE_SYMBOLX2_0(op4, &bitD4);
+ endSignal = (U32)LIKELY((U32)
+ (BIT_reloadDStreamFast(&bitD1) == BIT_DStream_unfinished)
+ & (BIT_reloadDStreamFast(&bitD2) == BIT_DStream_unfinished)
+ & (BIT_reloadDStreamFast(&bitD3) == BIT_DStream_unfinished)
+ & (BIT_reloadDStreamFast(&bitD4) == BIT_DStream_unfinished));
#endif
+ }
}
/* check corruption */
@@ -915,8 +1358,99 @@ HUF_decompress4X2_usingDTable_internal_body(
}
}
+#if HUF_NEED_BMI2_FUNCTION
+static BMI2_TARGET_ATTRIBUTE
+size_t HUF_decompress4X2_usingDTable_internal_bmi2(void* dst, size_t dstSize, void const* cSrc,
+ size_t cSrcSize, HUF_DTable const* DTable) {
+ return HUF_decompress4X2_usingDTable_internal_body(dst, dstSize, cSrc, cSrcSize, DTable);
+}
+#endif
+
+#if HUF_NEED_DEFAULT_FUNCTION
+static
+size_t HUF_decompress4X2_usingDTable_internal_default(void* dst, size_t dstSize, void const* cSrc,
+ size_t cSrcSize, HUF_DTable const* DTable) {
+ return HUF_decompress4X2_usingDTable_internal_body(dst, dstSize, cSrc, cSrcSize, DTable);
+}
+#endif
+
+#if ZSTD_ENABLE_ASM_X86_64_BMI2
+
+HUF_ASM_DECL void HUF_decompress4X2_usingDTable_internal_bmi2_asm_loop(HUF_DecompressAsmArgs* args) ZSTDLIB_HIDDEN;
+
+static HUF_ASM_X86_64_BMI2_ATTRS size_t
+HUF_decompress4X2_usingDTable_internal_bmi2_asm(
+ void* dst, size_t dstSize,
+ const void* cSrc, size_t cSrcSize,
+ const HUF_DTable* DTable) {
+ void const* dt = DTable + 1;
+ const BYTE* const iend = (const BYTE*)cSrc + 6;
+ BYTE* const oend = (BYTE*)dst + dstSize;
+ HUF_DecompressAsmArgs args;
+ {
+ size_t const ret = HUF_DecompressAsmArgs_init(&args, dst, dstSize, cSrc, cSrcSize, DTable);
+ FORWARD_IF_ERROR(ret, "Failed to init asm args");
+ if (ret != 0)
+ return HUF_decompress4X2_usingDTable_internal_bmi2(dst, dstSize, cSrc, cSrcSize, DTable);
+ }
+
+ assert(args.ip[0] >= args.ilimit);
+ HUF_decompress4X2_usingDTable_internal_bmi2_asm_loop(&args);
+
+ /* note : op4 already verified within main loop */
+ assert(args.ip[0] >= iend);
+ assert(args.ip[1] >= iend);
+ assert(args.ip[2] >= iend);
+ assert(args.ip[3] >= iend);
+ assert(args.op[3] <= oend);
+ (void)iend;
+
+ /* finish bitStreams one by one */
+ {
+ size_t const segmentSize = (dstSize+3) / 4;
+ BYTE* segmentEnd = (BYTE*)dst;
+ int i;
+ for (i = 0; i < 4; ++i) {
+ BIT_DStream_t bit;
+ if (segmentSize <= (size_t)(oend - segmentEnd))
+ segmentEnd += segmentSize;
+ else
+ segmentEnd = oend;
+ FORWARD_IF_ERROR(HUF_initRemainingDStream(&bit, &args, i, segmentEnd), "corruption");
+ args.op[i] += HUF_decodeStreamX2(args.op[i], &bit, segmentEnd, (HUF_DEltX2 const*)dt, HUF_DECODER_FAST_TABLELOG);
+ if (args.op[i] != segmentEnd)
+ return ERROR(corruption_detected);
+ }
+ }
+
+ /* decoded size */
+ return dstSize;
+}
+#endif /* ZSTD_ENABLE_ASM_X86_64_BMI2 */
+
+static size_t HUF_decompress4X2_usingDTable_internal(void* dst, size_t dstSize, void const* cSrc,
+ size_t cSrcSize, HUF_DTable const* DTable, int bmi2)
+{
+#if DYNAMIC_BMI2
+ if (bmi2) {
+# if ZSTD_ENABLE_ASM_X86_64_BMI2
+ return HUF_decompress4X2_usingDTable_internal_bmi2_asm(dst, dstSize, cSrc, cSrcSize, DTable);
+# else
+ return HUF_decompress4X2_usingDTable_internal_bmi2(dst, dstSize, cSrc, cSrcSize, DTable);
+# endif
+ }
+#else
+ (void)bmi2;
+#endif
+
+#if ZSTD_ENABLE_ASM_X86_64_BMI2 && defined(__BMI2__)
+ return HUF_decompress4X2_usingDTable_internal_bmi2_asm(dst, dstSize, cSrc, cSrcSize, DTable);
+#else
+ return HUF_decompress4X2_usingDTable_internal_default(dst, dstSize, cSrc, cSrcSize, DTable);
+#endif
+}
+
HUF_DGEN(HUF_decompress1X2_usingDTable_internal)
-HUF_DGEN(HUF_decompress4X2_usingDTable_internal)
size_t HUF_decompress1X2_usingDTable(
void* dst, size_t dstSize,
@@ -1025,25 +1559,25 @@ size_t HUF_decompress4X_usingDTable(void* dst, size_t maxDstSize,
#if !defined(HUF_FORCE_DECOMPRESS_X1) && !defined(HUF_FORCE_DECOMPRESS_X2)
typedef struct { U32 tableTime; U32 decode256Time; } algo_time_t;
-static const algo_time_t algoTime[16 /* Quantization */][3 /* single, double, quad */] =
+static const algo_time_t algoTime[16 /* Quantization */][2 /* single, double */] =
{
/* single, double, quad */
- {{0,0}, {1,1}, {2,2}}, /* Q==0 : impossible */
- {{0,0}, {1,1}, {2,2}}, /* Q==1 : impossible */
- {{ 38,130}, {1313, 74}, {2151, 38}}, /* Q == 2 : 12-18% */
- {{ 448,128}, {1353, 74}, {2238, 41}}, /* Q == 3 : 18-25% */
- {{ 556,128}, {1353, 74}, {2238, 47}}, /* Q == 4 : 25-32% */
- {{ 714,128}, {1418, 74}, {2436, 53}}, /* Q == 5 : 32-38% */
- {{ 883,128}, {1437, 74}, {2464, 61}}, /* Q == 6 : 38-44% */
- {{ 897,128}, {1515, 75}, {2622, 68}}, /* Q == 7 : 44-50% */
- {{ 926,128}, {1613, 75}, {2730, 75}}, /* Q == 8 : 50-56% */
- {{ 947,128}, {1729, 77}, {3359, 77}}, /* Q == 9 : 56-62% */
- {{1107,128}, {2083, 81}, {4006, 84}}, /* Q ==10 : 62-69% */
- {{1177,128}, {2379, 87}, {4785, 88}}, /* Q ==11 : 69-75% */
- {{1242,128}, {2415, 93}, {5155, 84}}, /* Q ==12 : 75-81% */
- {{1349,128}, {2644,106}, {5260,106}}, /* Q ==13 : 81-87% */
- {{1455,128}, {2422,124}, {4174,124}}, /* Q ==14 : 87-93% */
- {{ 722,128}, {1891,145}, {1936,146}}, /* Q ==15 : 93-99% */
+ {{0,0}, {1,1}}, /* Q==0 : impossible */
+ {{0,0}, {1,1}}, /* Q==1 : impossible */
+ {{ 150,216}, { 381,119}}, /* Q == 2 : 12-18% */
+ {{ 170,205}, { 514,112}}, /* Q == 3 : 18-25% */
+ {{ 177,199}, { 539,110}}, /* Q == 4 : 25-32% */
+ {{ 197,194}, { 644,107}}, /* Q == 5 : 32-38% */
+ {{ 221,192}, { 735,107}}, /* Q == 6 : 38-44% */
+ {{ 256,189}, { 881,106}}, /* Q == 7 : 44-50% */
+ {{ 359,188}, {1167,109}}, /* Q == 8 : 50-56% */
+ {{ 582,187}, {1570,114}}, /* Q == 9 : 56-62% */
+ {{ 688,187}, {1712,122}}, /* Q ==10 : 62-69% */
+ {{ 825,186}, {1965,136}}, /* Q ==11 : 69-75% */
+ {{ 976,185}, {2131,150}}, /* Q ==12 : 75-81% */
+ {{1180,186}, {2070,175}}, /* Q ==13 : 81-87% */
+ {{1377,185}, {1731,202}}, /* Q ==14 : 87-93% */
+ {{1412,185}, {1695,202}}, /* Q ==15 : 93-99% */
};
#endif
@@ -1070,7 +1604,7 @@ U32 HUF_selectDecoder (size_t dstSize, size_t cSrcSize)
U32 const D256 = (U32)(dstSize >> 8);
U32 const DTime0 = algoTime[Q][0].tableTime + (algoTime[Q][0].decode256Time * D256);
U32 DTime1 = algoTime[Q][1].tableTime + (algoTime[Q][1].decode256Time * D256);
- DTime1 += DTime1 >> 3; /* advantage to algorithm using less memory, to reduce cache eviction */
+ DTime1 += DTime1 >> 5; /* small advantage to algorithm using less memory, to reduce cache eviction */
return DTime1 < DTime0;
}
#endif
diff --git a/lib/zstd/decompress/zstd_decompress.c b/lib/zstd/decompress/zstd_decompress.c
index b4d81d8..b9b935a 100644
--- a/lib/zstd/decompress/zstd_decompress.c
+++ b/lib/zstd/decompress/zstd_decompress.c
@@ -53,7 +53,6 @@
* Dependencies
*********************************************************/
#include "../common/zstd_deps.h" /* ZSTD_memcpy, ZSTD_memmove, ZSTD_memset */
-#include "../common/cpu.h" /* bmi2 */
#include "../common/mem.h" /* low level memory routines */
#define FSE_STATIC_LINKING_ONLY
#include "../common/fse.h"
@@ -252,11 +251,11 @@ static void ZSTD_initDCtx_internal(ZSTD_DCtx* dctx)
dctx->inBuffSize = 0;
dctx->outBuffSize = 0;
dctx->streamStage = zdss_init;
- dctx->legacyContext = NULL;
- dctx->previousLegacyVersion = 0;
dctx->noForwardProgress = 0;
dctx->oversizedDuration = 0;
- dctx->bmi2 = ZSTD_cpuid_bmi2(ZSTD_cpuid());
+#if DYNAMIC_BMI2
+ dctx->bmi2 = ZSTD_cpuSupportsBmi2();
+#endif
dctx->ddictSet = NULL;
ZSTD_DCtx_resetParameters(dctx);
#ifdef FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION
@@ -277,8 +276,7 @@ ZSTD_DCtx* ZSTD_initStaticDCtx(void *workspace, size_t workspaceSize)
return dctx;
}
-ZSTD_DCtx* ZSTD_createDCtx_advanced(ZSTD_customMem customMem)
-{
+static ZSTD_DCtx* ZSTD_createDCtx_internal(ZSTD_customMem customMem) {
if ((!customMem.customAlloc) ^ (!customMem.customFree)) return NULL;
{ ZSTD_DCtx* const dctx = (ZSTD_DCtx*)ZSTD_customMalloc(sizeof(*dctx), customMem);
@@ -289,10 +287,15 @@ ZSTD_DCtx* ZSTD_createDCtx_advanced(ZSTD_customMem customMem)
}
}
+ZSTD_DCtx* ZSTD_createDCtx_advanced(ZSTD_customMem customMem)
+{
+ return ZSTD_createDCtx_internal(customMem);
+}
+
ZSTD_DCtx* ZSTD_createDCtx(void)
{
DEBUGLOG(3, "ZSTD_createDCtx");
- return ZSTD_createDCtx_advanced(ZSTD_defaultCMem);
+ return ZSTD_createDCtx_internal(ZSTD_defaultCMem);
}
static void ZSTD_clearDict(ZSTD_DCtx* dctx)
@@ -370,6 +373,19 @@ unsigned ZSTD_isFrame(const void* buffer, size_t size)
return 0;
}
+/*! ZSTD_isSkippableFrame() :
+ * Tells if the content of `buffer` starts with a valid Frame Identifier for a skippable frame.
+ * Note : Frame Identifier is 4 bytes. If `size < 4`, @return will always be 0.
+ */
+unsigned ZSTD_isSkippableFrame(const void* buffer, size_t size)
+{
+ if (size < ZSTD_FRAMEIDSIZE) return 0;
+ { U32 const magic = MEM_readLE32(buffer);
+ if ((magic & ZSTD_MAGIC_SKIPPABLE_MASK) == ZSTD_MAGIC_SKIPPABLE_START) return 1;
+ }
+ return 0;
+}
+
/* ZSTD_frameHeaderSize_internal() :
* srcSize must be large enough to reach header size fields.
* note : only works for formats ZSTD_f_zstd1 and ZSTD_f_zstd1_magicless.
@@ -497,7 +513,6 @@ size_t ZSTD_getFrameHeader(ZSTD_frameHeader* zfhPtr, const void* src, size_t src
return ZSTD_getFrameHeader_advanced(zfhPtr, src, srcSize, ZSTD_f_zstd1);
}
-
/* ZSTD_getFrameContentSize() :
* compatible with legacy mode
* @return : decompressed size of the single frame pointed to be `src` if known, otherwise
@@ -532,6 +547,37 @@ static size_t readSkippableFrameSize(void const* src, size_t srcSize)
}
}
+/*! ZSTD_readSkippableFrame() :
+ * Retrieves a zstd skippable frame containing data given by src, and writes it to dst buffer.
+ *
+ * The parameter magicVariant will receive the magicVariant that was supplied when the frame was written,
+ * i.e. magicNumber - ZSTD_MAGIC_SKIPPABLE_START. This can be NULL if the caller is not interested
+ * in the magicVariant.
+ *
+ * Returns an error if destination buffer is not large enough, or if the frame is not skippable.
+ *
+ * @return : number of bytes written or a ZSTD error.
+ */
+ZSTDLIB_API size_t ZSTD_readSkippableFrame(void* dst, size_t dstCapacity, unsigned* magicVariant,
+ const void* src, size_t srcSize)
+{
+ U32 const magicNumber = MEM_readLE32(src);
+ size_t skippableFrameSize = readSkippableFrameSize(src, srcSize);
+ size_t skippableContentSize = skippableFrameSize - ZSTD_SKIPPABLEHEADERSIZE;
+
+ /* check input validity */
+ RETURN_ERROR_IF(!ZSTD_isSkippableFrame(src, srcSize), frameParameter_unsupported, "");
+ RETURN_ERROR_IF(skippableFrameSize < ZSTD_SKIPPABLEHEADERSIZE || skippableFrameSize > srcSize, srcSize_wrong, "");
+ RETURN_ERROR_IF(skippableContentSize > dstCapacity, dstSize_tooSmall, "");
+
+ /* deliver payload */
+ if (skippableContentSize > 0 && dst != NULL)
+ ZSTD_memcpy(dst, (const BYTE *)src + ZSTD_SKIPPABLEHEADERSIZE, skippableContentSize);
+ if (magicVariant != NULL)
+ *magicVariant = magicNumber - ZSTD_MAGIC_SKIPPABLE_START;
+ return skippableContentSize;
+}
+
/* ZSTD_findDecompressedSize() :
* compatible with legacy mode
* `srcSize` must be the exact length of some number of ZSTD compressed and/or
@@ -824,7 +870,7 @@ static size_t ZSTD_decompressFrame(ZSTD_DCtx* dctx,
switch(blockProperties.blockType)
{
case bt_compressed:
- decodedSize = ZSTD_decompressBlock_internal(dctx, op, (size_t)(oend-op), ip, cBlockSize, /* frame */ 1);
+ decodedSize = ZSTD_decompressBlock_internal(dctx, op, (size_t)(oend-op), ip, cBlockSize, /* frame */ 1, not_streaming);
break;
case bt_raw :
decodedSize = ZSTD_copyRawBlock(op, (size_t)(oend-op), ip, cBlockSize);
@@ -976,7 +1022,7 @@ size_t ZSTD_decompress(void* dst, size_t dstCapacity, const void* src, size_t sr
{
#if defined(ZSTD_HEAPMODE) && (ZSTD_HEAPMODE>=1)
size_t regenSize;
- ZSTD_DCtx* const dctx = ZSTD_createDCtx();
+ ZSTD_DCtx* const dctx = ZSTD_createDCtx_internal(ZSTD_defaultCMem);
RETURN_ERROR_IF(dctx==NULL, memory_allocation, "NULL pointer!");
regenSize = ZSTD_decompressDCtx(dctx, dst, dstCapacity, src, srcSize);
ZSTD_freeDCtx(dctx);
@@ -996,7 +1042,7 @@ size_t ZSTD_decompress(void* dst, size_t dstCapacity, const void* src, size_t sr
size_t ZSTD_nextSrcSizeToDecompress(ZSTD_DCtx* dctx) { return dctx->expected; }
/*
- * Similar to ZSTD_nextSrcSizeToDecompress(), but when when a block input can be streamed,
+ * Similar to ZSTD_nextSrcSizeToDecompress(), but when a block input can be streamed,
* we allow taking a partial block as the input. Currently only raw uncompressed blocks can
* be streamed.
*
@@ -1010,7 +1056,7 @@ static size_t ZSTD_nextSrcSizeToDecompressWithInputSize(ZSTD_DCtx* dctx, size_t
return dctx->expected;
if (dctx->bType != bt_raw)
return dctx->expected;
- return MIN(MAX(inputSize, 1), dctx->expected);
+ return BOUNDED(1, inputSize, dctx->expected);
}
ZSTD_nextInputType_e ZSTD_nextInputType(ZSTD_DCtx* dctx) {
@@ -1116,7 +1162,7 @@ size_t ZSTD_decompressContinue(ZSTD_DCtx* dctx, void* dst, size_t dstCapacity, c
{
case bt_compressed:
DEBUGLOG(5, "ZSTD_decompressContinue: case bt_compressed");
- rSize = ZSTD_decompressBlock_internal(dctx, dst, dstCapacity, src, srcSize, /* frame */ 1);
+ rSize = ZSTD_decompressBlock_internal(dctx, dst, dstCapacity, src, srcSize, /* frame */ 1, is_streaming);
dctx->expected = 0; /* Streaming not supported */
break;
case bt_raw :
@@ -1438,7 +1484,7 @@ size_t ZSTD_decompress_usingDDict(ZSTD_DCtx* dctx,
ZSTD_DStream* ZSTD_createDStream(void)
{
DEBUGLOG(3, "ZSTD_createDStream");
- return ZSTD_createDStream_advanced(ZSTD_defaultCMem);
+ return ZSTD_createDCtx_internal(ZSTD_defaultCMem);
}
ZSTD_DStream* ZSTD_initStaticDStream(void *workspace, size_t workspaceSize)
@@ -1448,7 +1494,7 @@ ZSTD_DStream* ZSTD_initStaticDStream(void *workspace, size_t workspaceSize)
ZSTD_DStream* ZSTD_createDStream_advanced(ZSTD_customMem customMem)
{
- return ZSTD_createDCtx_advanced(customMem);
+ return ZSTD_createDCtx_internal(customMem);
}
size_t ZSTD_freeDStream(ZSTD_DStream* zds)
@@ -1708,7 +1754,8 @@ size_t ZSTD_sizeof_DStream(const ZSTD_DStream* dctx)
size_t ZSTD_decodingBufferSize_min(unsigned long long windowSize, unsigned long long frameContentSize)
{
size_t const blockSize = (size_t) MIN(windowSize, ZSTD_BLOCKSIZE_MAX);
- unsigned long long const neededRBSize = windowSize + blockSize + (WILDCOPY_OVERLENGTH * 2);
+ /* space is needed to store the litbuffer after the output of a given block without stomping the extDict of a previous run, as well as to cover both windows against wildcopy*/
+ unsigned long long const neededRBSize = windowSize + blockSize + ZSTD_BLOCKSIZE_MAX + (WILDCOPY_OVERLENGTH * 2);
unsigned long long const neededSize = MIN(frameContentSize, neededRBSize);
size_t const minRBSize = (size_t) neededSize;
RETURN_ERROR_IF((unsigned long long)minRBSize != neededSize,
@@ -1842,7 +1889,6 @@ size_t ZSTD_decompressStream(ZSTD_DStream* zds, ZSTD_outBuffer* output, ZSTD_inB
DEBUGLOG(5, "stage zdss_init => transparent reset ");
zds->streamStage = zdss_loadHeader;
zds->lhSize = zds->inPos = zds->outStart = zds->outEnd = 0;
- zds->legacyVersion = 0;
zds->hostageByte = 0;
zds->expectedOutBuffer = *output;
ZSTD_FALLTHROUGH;
diff --git a/lib/zstd/decompress/zstd_decompress_block.c b/lib/zstd/decompress/zstd_decompress_block.c
index 2d101d9..c1913b8 100644
--- a/lib/zstd/decompress/zstd_decompress_block.c
+++ b/lib/zstd/decompress/zstd_decompress_block.c
@@ -69,15 +69,56 @@ size_t ZSTD_getcBlockSize(const void* src, size_t srcSize,
}
}
+/* Allocate buffer for literals, either overlapping current dst, or split between dst and litExtraBuffer, or stored entirely within litExtraBuffer */
+static void ZSTD_allocateLiteralsBuffer(ZSTD_DCtx* dctx, void* const dst, const size_t dstCapacity, const size_t litSize,
+ const streaming_operation streaming, const size_t expectedWriteSize, const unsigned splitImmediately)
+{
+ if (streaming == not_streaming && dstCapacity > ZSTD_BLOCKSIZE_MAX + WILDCOPY_OVERLENGTH + litSize + WILDCOPY_OVERLENGTH)
+ {
+ /* room for litbuffer to fit without read faulting */
+ dctx->litBuffer = (BYTE*)dst + ZSTD_BLOCKSIZE_MAX + WILDCOPY_OVERLENGTH;
+ dctx->litBufferEnd = dctx->litBuffer + litSize;
+ dctx->litBufferLocation = ZSTD_in_dst;
+ }
+ else if (litSize > ZSTD_LITBUFFEREXTRASIZE)
+ {
+ /* won't fit in litExtraBuffer, so it will be split between end of dst and extra buffer */
+ if (splitImmediately) {
+ /* won't fit in litExtraBuffer, so it will be split between end of dst and extra buffer */
+ dctx->litBuffer = (BYTE*)dst + expectedWriteSize - litSize + ZSTD_LITBUFFEREXTRASIZE - WILDCOPY_OVERLENGTH;
+ dctx->litBufferEnd = dctx->litBuffer + litSize - ZSTD_LITBUFFEREXTRASIZE;
+ }
+ else {
+ /* initially this will be stored entirely in dst during huffman decoding, it will partially shifted to litExtraBuffer after */
+ dctx->litBuffer = (BYTE*)dst + expectedWriteSize - litSize;
+ dctx->litBufferEnd = (BYTE*)dst + expectedWriteSize;
+ }
+ dctx->litBufferLocation = ZSTD_split;
+ }
+ else
+ {
+ /* fits entirely within litExtraBuffer, so no split is necessary */
+ dctx->litBuffer = dctx->litExtraBuffer;
+ dctx->litBufferEnd = dctx->litBuffer + litSize;
+ dctx->litBufferLocation = ZSTD_not_in_dst;
+ }
+}
/* Hidden declaration for fullbench */
size_t ZSTD_decodeLiteralsBlock(ZSTD_DCtx* dctx,
- const void* src, size_t srcSize);
+ const void* src, size_t srcSize,
+ void* dst, size_t dstCapacity, const streaming_operation streaming);
/*! ZSTD_decodeLiteralsBlock() :
+ * Where it is possible to do so without being stomped by the output during decompression, the literals block will be stored
+ * in the dstBuffer. If there is room to do so, it will be stored in full in the excess dst space after where the current
+ * block will be output. Otherwise it will be stored at the end of the current dst blockspace, with a small portion being
+ * stored in dctx->litExtraBuffer to help keep it "ahead" of the current output write.
+ *
* @return : nb of bytes read from src (< srcSize )
* note : symbol not declared but exposed for fullbench */
size_t ZSTD_decodeLiteralsBlock(ZSTD_DCtx* dctx,
- const void* src, size_t srcSize) /* note : srcSize < BLOCKSIZE */
+ const void* src, size_t srcSize, /* note : srcSize < BLOCKSIZE */
+ void* dst, size_t dstCapacity, const streaming_operation streaming)
{
DEBUGLOG(5, "ZSTD_decodeLiteralsBlock");
RETURN_ERROR_IF(srcSize < MIN_CBLOCK_SIZE, corruption_detected, "");
@@ -99,6 +140,7 @@ size_t ZSTD_decodeLiteralsBlock(ZSTD_DCtx* dctx,
U32 const lhlCode = (istart[0] >> 2) & 3;
U32 const lhc = MEM_readLE32(istart);
size_t hufSuccess;
+ size_t expectedWriteSize = MIN(ZSTD_BLOCKSIZE_MAX, dstCapacity);
switch(lhlCode)
{
case 0: case 1: default: /* note : default is impossible, since lhlCode into [0..3] */
@@ -121,8 +163,11 @@ size_t ZSTD_decodeLiteralsBlock(ZSTD_DCtx* dctx,
litCSize = (lhc >> 22) + ((size_t)istart[4] << 10);
break;
}
+ RETURN_ERROR_IF(litSize > 0 && dst == NULL, dstSize_tooSmall, "NULL not handled");
RETURN_ERROR_IF(litSize > ZSTD_BLOCKSIZE_MAX, corruption_detected, "");
RETURN_ERROR_IF(litCSize + lhSize > srcSize, corruption_detected, "");
+ RETURN_ERROR_IF(expectedWriteSize < litSize , dstSize_tooSmall, "");
+ ZSTD_allocateLiteralsBuffer(dctx, dst, dstCapacity, litSize, streaming, expectedWriteSize, 0);
/* prefetch huffman table if cold */
if (dctx->ddictIsCold && (litSize > 768 /* heuristic */)) {
@@ -133,11 +178,11 @@ size_t ZSTD_decodeLiteralsBlock(ZSTD_DCtx* dctx,
if (singleStream) {
hufSuccess = HUF_decompress1X_usingDTable_bmi2(
dctx->litBuffer, litSize, istart+lhSize, litCSize,
- dctx->HUFptr, dctx->bmi2);
+ dctx->HUFptr, ZSTD_DCtx_get_bmi2(dctx));
} else {
hufSuccess = HUF_decompress4X_usingDTable_bmi2(
dctx->litBuffer, litSize, istart+lhSize, litCSize,
- dctx->HUFptr, dctx->bmi2);
+ dctx->HUFptr, ZSTD_DCtx_get_bmi2(dctx));
}
} else {
if (singleStream) {
@@ -150,15 +195,22 @@ size_t ZSTD_decodeLiteralsBlock(ZSTD_DCtx* dctx,
hufSuccess = HUF_decompress1X1_DCtx_wksp_bmi2(
dctx->entropy.hufTable, dctx->litBuffer, litSize,
istart+lhSize, litCSize, dctx->workspace,
- sizeof(dctx->workspace), dctx->bmi2);
+ sizeof(dctx->workspace), ZSTD_DCtx_get_bmi2(dctx));
#endif
} else {
hufSuccess = HUF_decompress4X_hufOnly_wksp_bmi2(
dctx->entropy.hufTable, dctx->litBuffer, litSize,
istart+lhSize, litCSize, dctx->workspace,
- sizeof(dctx->workspace), dctx->bmi2);
+ sizeof(dctx->workspace), ZSTD_DCtx_get_bmi2(dctx));
}
}
+ if (dctx->litBufferLocation == ZSTD_split)
+ {
+ ZSTD_memcpy(dctx->litExtraBuffer, dctx->litBufferEnd - ZSTD_LITBUFFEREXTRASIZE, ZSTD_LITBUFFEREXTRASIZE);
+ ZSTD_memmove(dctx->litBuffer + ZSTD_LITBUFFEREXTRASIZE - WILDCOPY_OVERLENGTH, dctx->litBuffer, litSize - ZSTD_LITBUFFEREXTRASIZE);
+ dctx->litBuffer += ZSTD_LITBUFFEREXTRASIZE - WILDCOPY_OVERLENGTH;
+ dctx->litBufferEnd -= WILDCOPY_OVERLENGTH;
+ }
RETURN_ERROR_IF(HUF_isError(hufSuccess), corruption_detected, "");
@@ -166,13 +218,13 @@ size_t ZSTD_decodeLiteralsBlock(ZSTD_DCtx* dctx,
dctx->litSize = litSize;
dctx->litEntropy = 1;
if (litEncType==set_compressed) dctx->HUFptr = dctx->entropy.hufTable;
- ZSTD_memset(dctx->litBuffer + dctx->litSize, 0, WILDCOPY_OVERLENGTH);
return litCSize + lhSize;
}
case set_basic:
{ size_t litSize, lhSize;
U32 const lhlCode = ((istart[0]) >> 2) & 3;
+ size_t expectedWriteSize = MIN(ZSTD_BLOCKSIZE_MAX, dstCapacity);
switch(lhlCode)
{
case 0: case 2: default: /* note : default is impossible, since lhlCode into [0..3] */
@@ -189,23 +241,36 @@ size_t ZSTD_decodeLiteralsBlock(ZSTD_DCtx* dctx,
break;
}
+ RETURN_ERROR_IF(litSize > 0 && dst == NULL, dstSize_tooSmall, "NULL not handled");
+ RETURN_ERROR_IF(expectedWriteSize < litSize, dstSize_tooSmall, "");
+ ZSTD_allocateLiteralsBuffer(dctx, dst, dstCapacity, litSize, streaming, expectedWriteSize, 1);
if (lhSize+litSize+WILDCOPY_OVERLENGTH > srcSize) { /* risk reading beyond src buffer with wildcopy */
RETURN_ERROR_IF(litSize+lhSize > srcSize, corruption_detected, "");
- ZSTD_memcpy(dctx->litBuffer, istart+lhSize, litSize);
+ if (dctx->litBufferLocation == ZSTD_split)
+ {
+ ZSTD_memcpy(dctx->litBuffer, istart + lhSize, litSize - ZSTD_LITBUFFEREXTRASIZE);
+ ZSTD_memcpy(dctx->litExtraBuffer, istart + lhSize + litSize - ZSTD_LITBUFFEREXTRASIZE, ZSTD_LITBUFFEREXTRASIZE);
+ }
+ else
+ {
+ ZSTD_memcpy(dctx->litBuffer, istart + lhSize, litSize);
+ }
dctx->litPtr = dctx->litBuffer;
dctx->litSize = litSize;
- ZSTD_memset(dctx->litBuffer + dctx->litSize, 0, WILDCOPY_OVERLENGTH);
return lhSize+litSize;
}
/* direct reference into compressed stream */
dctx->litPtr = istart+lhSize;
dctx->litSize = litSize;
+ dctx->litBufferEnd = dctx->litPtr + litSize;
+ dctx->litBufferLocation = ZSTD_not_in_dst;
return lhSize+litSize;
}
case set_rle:
{ U32 const lhlCode = ((istart[0]) >> 2) & 3;
size_t litSize, lhSize;
+ size_t expectedWriteSize = MIN(ZSTD_BLOCKSIZE_MAX, dstCapacity);
switch(lhlCode)
{
case 0: case 2: default: /* note : default is impossible, since lhlCode into [0..3] */
@@ -222,8 +287,19 @@ size_t ZSTD_decodeLiteralsBlock(ZSTD_DCtx* dctx,
RETURN_ERROR_IF(srcSize<4, corruption_detected, "srcSize >= MIN_CBLOCK_SIZE == 3; here we need lhSize+1 = 4");
break;
}
+ RETURN_ERROR_IF(litSize > 0 && dst == NULL, dstSize_tooSmall, "NULL not handled");
RETURN_ERROR_IF(litSize > ZSTD_BLOCKSIZE_MAX, corruption_detected, "");
- ZSTD_memset(dctx->litBuffer, istart[lhSize], litSize + WILDCOPY_OVERLENGTH);
+ RETURN_ERROR_IF(expectedWriteSize < litSize, dstSize_tooSmall, "");
+ ZSTD_allocateLiteralsBuffer(dctx, dst, dstCapacity, litSize, streaming, expectedWriteSize, 1);
+ if (dctx->litBufferLocation == ZSTD_split)
+ {
+ ZSTD_memset(dctx->litBuffer, istart[lhSize], litSize - ZSTD_LITBUFFEREXTRASIZE);
+ ZSTD_memset(dctx->litExtraBuffer, istart[lhSize], ZSTD_LITBUFFEREXTRASIZE);
+ }
+ else
+ {
+ ZSTD_memset(dctx->litBuffer, istart[lhSize], litSize);
+ }
dctx->litPtr = dctx->litBuffer;
dctx->litSize = litSize;
return lhSize+1;
@@ -343,7 +419,7 @@ static const ZSTD_seqSymbol ML_defaultDTable[(1<<ML_DEFAULTNORMLOG)+1] = {
}; /* ML_defaultDTable */
-static void ZSTD_buildSeqTable_rle(ZSTD_seqSymbol* dt, U32 baseValue, U32 nbAddBits)
+static void ZSTD_buildSeqTable_rle(ZSTD_seqSymbol* dt, U32 baseValue, U8 nbAddBits)
{
void* ptr = dt;
ZSTD_seqSymbol_header* const DTableH = (ZSTD_seqSymbol_header*)ptr;
@@ -355,7 +431,7 @@ static void ZSTD_buildSeqTable_rle(ZSTD_seqSymbol* dt, U32 baseValue, U32 nbAddB
cell->nbBits = 0;
cell->nextState = 0;
assert(nbAddBits < 255);
- cell->nbAdditionalBits = (BYTE)nbAddBits;
+ cell->nbAdditionalBits = nbAddBits;
cell->baseValue = baseValue;
}
@@ -367,7 +443,7 @@ static void ZSTD_buildSeqTable_rle(ZSTD_seqSymbol* dt, U32 baseValue, U32 nbAddB
FORCE_INLINE_TEMPLATE
void ZSTD_buildFSETable_body(ZSTD_seqSymbol* dt,
const short* normalizedCounter, unsigned maxSymbolValue,
- const U32* baseValue, const U32* nbAdditionalBits,
+ const U32* baseValue, const U8* nbAdditionalBits,
unsigned tableLog, void* wksp, size_t wkspSize)
{
ZSTD_seqSymbol* const tableDecode = dt+1;
@@ -478,7 +554,7 @@ void ZSTD_buildFSETable_body(ZSTD_seqSymbol* dt,
tableDecode[u].nbBits = (BYTE) (tableLog - BIT_highbit32(nextState) );
tableDecode[u].nextState = (U16) ( (nextState << tableDecode[u].nbBits) - tableSize);
assert(nbAdditionalBits[symbol] < 255);
- tableDecode[u].nbAdditionalBits = (BYTE)nbAdditionalBits[symbol];
+ tableDecode[u].nbAdditionalBits = nbAdditionalBits[symbol];
tableDecode[u].baseValue = baseValue[symbol];
}
}
@@ -487,7 +563,7 @@ void ZSTD_buildFSETable_body(ZSTD_seqSymbol* dt,
/* Avoids the FORCE_INLINE of the _body() function. */
static void ZSTD_buildFSETable_body_default(ZSTD_seqSymbol* dt,
const short* normalizedCounter, unsigned maxSymbolValue,
- const U32* baseValue, const U32* nbAdditionalBits,
+ const U32* baseValue, const U8* nbAdditionalBits,
unsigned tableLog, void* wksp, size_t wkspSize)
{
ZSTD_buildFSETable_body(dt, normalizedCounter, maxSymbolValue,
@@ -495,9 +571,9 @@ static void ZSTD_buildFSETable_body_default(ZSTD_seqSymbol* dt,
}
#if DYNAMIC_BMI2
-TARGET_ATTRIBUTE("bmi2") static void ZSTD_buildFSETable_body_bmi2(ZSTD_seqSymbol* dt,
+BMI2_TARGET_ATTRIBUTE static void ZSTD_buildFSETable_body_bmi2(ZSTD_seqSymbol* dt,
const short* normalizedCounter, unsigned maxSymbolValue,
- const U32* baseValue, const U32* nbAdditionalBits,
+ const U32* baseValue, const U8* nbAdditionalBits,
unsigned tableLog, void* wksp, size_t wkspSize)
{
ZSTD_buildFSETable_body(dt, normalizedCounter, maxSymbolValue,
@@ -507,7 +583,7 @@ TARGET_ATTRIBUTE("bmi2") static void ZSTD_buildFSETable_body_bmi2(ZSTD_seqSymbol
void ZSTD_buildFSETable(ZSTD_seqSymbol* dt,
const short* normalizedCounter, unsigned maxSymbolValue,
- const U32* baseValue, const U32* nbAdditionalBits,
+ const U32* baseValue, const U8* nbAdditionalBits,
unsigned tableLog, void* wksp, size_t wkspSize, int bmi2)
{
#if DYNAMIC_BMI2
@@ -529,7 +605,7 @@ void ZSTD_buildFSETable(ZSTD_seqSymbol* dt,
static size_t ZSTD_buildSeqTable(ZSTD_seqSymbol* DTableSpace, const ZSTD_seqSymbol** DTablePtr,
symbolEncodingType_e type, unsigned max, U32 maxLog,
const void* src, size_t srcSize,
- const U32* baseValue, const U32* nbAdditionalBits,
+ const U32* baseValue, const U8* nbAdditionalBits,
const ZSTD_seqSymbol* defaultTable, U32 flagRepeatTable,
int ddictIsCold, int nbSeq, U32* wksp, size_t wkspSize,
int bmi2)
@@ -541,7 +617,7 @@ static size_t ZSTD_buildSeqTable(ZSTD_seqSymbol* DTableSpace, const ZSTD_seqSymb
RETURN_ERROR_IF((*(const BYTE*)src) > max, corruption_detected, "");
{ U32 const symbol = *(const BYTE*)src;
U32 const baseline = baseValue[symbol];
- U32 const nbBits = nbAdditionalBits[symbol];
+ U8 const nbBits = nbAdditionalBits[symbol];
ZSTD_buildSeqTable_rle(DTableSpace, baseline, nbBits);
}
*DTablePtr = DTableSpace;
@@ -620,7 +696,7 @@ size_t ZSTD_decodeSeqHeaders(ZSTD_DCtx* dctx, int* nbSeqPtr,
LL_defaultDTable, dctx->fseEntropy,
dctx->ddictIsCold, nbSeq,
dctx->workspace, sizeof(dctx->workspace),
- dctx->bmi2);
+ ZSTD_DCtx_get_bmi2(dctx));
RETURN_ERROR_IF(ZSTD_isError(llhSize), corruption_detected, "ZSTD_buildSeqTable failed");
ip += llhSize;
}
@@ -632,7 +708,7 @@ size_t ZSTD_decodeSeqHeaders(ZSTD_DCtx* dctx, int* nbSeqPtr,
OF_defaultDTable, dctx->fseEntropy,
dctx->ddictIsCold, nbSeq,
dctx->workspace, sizeof(dctx->workspace),
- dctx->bmi2);
+ ZSTD_DCtx_get_bmi2(dctx));
RETURN_ERROR_IF(ZSTD_isError(ofhSize), corruption_detected, "ZSTD_buildSeqTable failed");
ip += ofhSize;
}
@@ -644,7 +720,7 @@ size_t ZSTD_decodeSeqHeaders(ZSTD_DCtx* dctx, int* nbSeqPtr,
ML_defaultDTable, dctx->fseEntropy,
dctx->ddictIsCold, nbSeq,
dctx->workspace, sizeof(dctx->workspace),
- dctx->bmi2);
+ ZSTD_DCtx_get_bmi2(dctx));
RETURN_ERROR_IF(ZSTD_isError(mlhSize), corruption_detected, "ZSTD_buildSeqTable failed");
ip += mlhSize;
}
@@ -658,7 +734,6 @@ typedef struct {
size_t litLength;
size_t matchLength;
size_t offset;
- const BYTE* match;
} seq_t;
typedef struct {
@@ -672,9 +747,6 @@ typedef struct {
ZSTD_fseState stateOffb;
ZSTD_fseState stateML;
size_t prevOffset[ZSTD_REP_NUM];
- const BYTE* prefixStart;
- const BYTE* dictEnd;
- size_t pos;
} seqState_t;
/*! ZSTD_overlapCopy8() :
@@ -717,7 +789,7 @@ HINT_INLINE void ZSTD_overlapCopy8(BYTE** op, BYTE const** ip, size_t offset) {
* - ZSTD_overlap_src_before_dst: The src and dst may overlap and may be any distance apart.
* The src buffer must be before the dst buffer.
*/
-static void ZSTD_safecopy(BYTE* op, BYTE* const oend_w, BYTE const* ip, ptrdiff_t length, ZSTD_overlap_e ovtype) {
+static void ZSTD_safecopy(BYTE* op, const BYTE* const oend_w, BYTE const* ip, ptrdiff_t length, ZSTD_overlap_e ovtype) {
ptrdiff_t const diff = op - ip;
BYTE* const oend = op + length;
@@ -733,6 +805,7 @@ static void ZSTD_safecopy(BYTE* op, BYTE* const oend_w, BYTE const* ip, ptrdiff_
/* Copy 8 bytes and ensure the offset >= 8 when there can be overlap. */
assert(length >= 8);
ZSTD_overlapCopy8(&op, &ip, diff);
+ length -= 8;
assert(op - ip >= 8);
assert(op <= oend);
}
@@ -747,12 +820,35 @@ static void ZSTD_safecopy(BYTE* op, BYTE* const oend_w, BYTE const* ip, ptrdiff_
assert(oend > oend_w);
ZSTD_wildcopy(op, ip, oend_w - op, ovtype);
ip += oend_w - op;
- op = oend_w;
+ op += oend_w - op;
}
/* Handle the leftovers. */
while (op < oend) *op++ = *ip++;
}
+/* ZSTD_safecopyDstBeforeSrc():
+ * This version allows overlap with dst before src, or handles the non-overlap case with dst after src
+ * Kept separate from more common ZSTD_safecopy case to avoid performance impact to the safecopy common case */
+static void ZSTD_safecopyDstBeforeSrc(BYTE* op, BYTE const* ip, ptrdiff_t length) {
+ ptrdiff_t const diff = op - ip;
+ BYTE* const oend = op + length;
+
+ if (length < 8 || diff > -8) {
+ /* Handle short lengths, close overlaps, and dst not before src. */
+ while (op < oend) *op++ = *ip++;
+ return;
+ }
+
+ if (op <= oend - WILDCOPY_OVERLENGTH && diff < -WILDCOPY_VECLEN) {
+ ZSTD_wildcopy(op, ip, oend - WILDCOPY_OVERLENGTH - op, ZSTD_no_overlap);
+ ip += oend - WILDCOPY_OVERLENGTH - op;
+ op += oend - WILDCOPY_OVERLENGTH - op;
+ }
+
+ /* Handle the leftovers. */
+ while (op < oend) *op++ = *ip++;
+}
+
/* ZSTD_execSequenceEnd():
* This version handles cases that are near the end of the output buffer. It requires
* more careful checks to make sure there is no overflow. By separating out these hard
@@ -763,9 +859,9 @@ static void ZSTD_safecopy(BYTE* op, BYTE* const oend_w, BYTE const* ip, ptrdiff_
*/
FORCE_NOINLINE
size_t ZSTD_execSequenceEnd(BYTE* op,
- BYTE* const oend, seq_t sequence,
- const BYTE** litPtr, const BYTE* const litLimit,
- const BYTE* const prefixStart, const BYTE* const virtualStart, const BYTE* const dictEnd)
+ BYTE* const oend, seq_t sequence,
+ const BYTE** litPtr, const BYTE* const litLimit,
+ const BYTE* const prefixStart, const BYTE* const virtualStart, const BYTE* const dictEnd)
{
BYTE* const oLitEnd = op + sequence.litLength;
size_t const sequenceLength = sequence.litLength + sequence.matchLength;
@@ -788,27 +884,76 @@ size_t ZSTD_execSequenceEnd(BYTE* op,
if (sequence.offset > (size_t)(oLitEnd - prefixStart)) {
/* offset beyond prefix */
RETURN_ERROR_IF(sequence.offset > (size_t)(oLitEnd - virtualStart), corruption_detected, "");
- match = dictEnd - (prefixStart-match);
+ match = dictEnd - (prefixStart - match);
if (match + sequence.matchLength <= dictEnd) {
ZSTD_memmove(oLitEnd, match, sequence.matchLength);
return sequenceLength;
}
/* span extDict & currentPrefixSegment */
{ size_t const length1 = dictEnd - match;
- ZSTD_memmove(oLitEnd, match, length1);
- op = oLitEnd + length1;
- sequence.matchLength -= length1;
- match = prefixStart;
- } }
+ ZSTD_memmove(oLitEnd, match, length1);
+ op = oLitEnd + length1;
+ sequence.matchLength -= length1;
+ match = prefixStart;
+ }
+ }
+ ZSTD_safecopy(op, oend_w, match, sequence.matchLength, ZSTD_overlap_src_before_dst);
+ return sequenceLength;
+}
+
+/* ZSTD_execSequenceEndSplitLitBuffer():
+ * This version is intended to be used during instances where the litBuffer is still split. It is kept separate to avoid performance impact for the good case.
+ */
+FORCE_NOINLINE
+size_t ZSTD_execSequenceEndSplitLitBuffer(BYTE* op,
+ BYTE* const oend, const BYTE* const oend_w, seq_t sequence,
+ const BYTE** litPtr, const BYTE* const litLimit,
+ const BYTE* const prefixStart, const BYTE* const virtualStart, const BYTE* const dictEnd)
+{
+ BYTE* const oLitEnd = op + sequence.litLength;
+ size_t const sequenceLength = sequence.litLength + sequence.matchLength;
+ const BYTE* const iLitEnd = *litPtr + sequence.litLength;
+ const BYTE* match = oLitEnd - sequence.offset;
+
+
+ /* bounds checks : careful of address space overflow in 32-bit mode */
+ RETURN_ERROR_IF(sequenceLength > (size_t)(oend - op), dstSize_tooSmall, "last match must fit within dstBuffer");
+ RETURN_ERROR_IF(sequence.litLength > (size_t)(litLimit - *litPtr), corruption_detected, "try to read beyond literal buffer");
+ assert(op < op + sequenceLength);
+ assert(oLitEnd < op + sequenceLength);
+
+ /* copy literals */
+ RETURN_ERROR_IF(op > *litPtr && op < *litPtr + sequence.litLength, dstSize_tooSmall, "output should not catch up to and overwrite literal buffer");
+ ZSTD_safecopyDstBeforeSrc(op, *litPtr, sequence.litLength);
+ op = oLitEnd;
+ *litPtr = iLitEnd;
+
+ /* copy Match */
+ if (sequence.offset > (size_t)(oLitEnd - prefixStart)) {
+ /* offset beyond prefix */
+ RETURN_ERROR_IF(sequence.offset > (size_t)(oLitEnd - virtualStart), corruption_detected, "");
+ match = dictEnd - (prefixStart - match);
+ if (match + sequence.matchLength <= dictEnd) {
+ ZSTD_memmove(oLitEnd, match, sequence.matchLength);
+ return sequenceLength;
+ }
+ /* span extDict & currentPrefixSegment */
+ { size_t const length1 = dictEnd - match;
+ ZSTD_memmove(oLitEnd, match, length1);
+ op = oLitEnd + length1;
+ sequence.matchLength -= length1;
+ match = prefixStart;
+ }
+ }
ZSTD_safecopy(op, oend_w, match, sequence.matchLength, ZSTD_overlap_src_before_dst);
return sequenceLength;
}
HINT_INLINE
size_t ZSTD_execSequence(BYTE* op,
- BYTE* const oend, seq_t sequence,
- const BYTE** litPtr, const BYTE* const litLimit,
- const BYTE* const prefixStart, const BYTE* const virtualStart, const BYTE* const dictEnd)
+ BYTE* const oend, seq_t sequence,
+ const BYTE** litPtr, const BYTE* const litLimit,
+ const BYTE* const prefixStart, const BYTE* const virtualStart, const BYTE* const dictEnd)
{
BYTE* const oLitEnd = op + sequence.litLength;
size_t const sequenceLength = sequence.litLength + sequence.matchLength;
@@ -825,10 +970,102 @@ size_t ZSTD_execSequence(BYTE* op,
* - 32-bit mode and the match length overflows
*/
if (UNLIKELY(
+ iLitEnd > litLimit ||
+ oMatchEnd > oend_w ||
+ (MEM_32bits() && (size_t)(oend - op) < sequenceLength + WILDCOPY_OVERLENGTH)))
+ return ZSTD_execSequenceEnd(op, oend, sequence, litPtr, litLimit, prefixStart, virtualStart, dictEnd);
+
+ /* Assumptions (everything else goes into ZSTD_execSequenceEnd()) */
+ assert(op <= oLitEnd /* No overflow */);
+ assert(oLitEnd < oMatchEnd /* Non-zero match & no overflow */);
+ assert(oMatchEnd <= oend /* No underflow */);
+ assert(iLitEnd <= litLimit /* Literal length is in bounds */);
+ assert(oLitEnd <= oend_w /* Can wildcopy literals */);
+ assert(oMatchEnd <= oend_w /* Can wildcopy matches */);
+
+ /* Copy Literals:
+ * Split out litLength <= 16 since it is nearly always true. +1.6% on gcc-9.
+ * We likely don't need the full 32-byte wildcopy.
+ */
+ assert(WILDCOPY_OVERLENGTH >= 16);
+ ZSTD_copy16(op, (*litPtr));
+ if (UNLIKELY(sequence.litLength > 16)) {
+ ZSTD_wildcopy(op + 16, (*litPtr) + 16, sequence.litLength - 16, ZSTD_no_overlap);
+ }
+ op = oLitEnd;
+ *litPtr = iLitEnd; /* update for next sequence */
+
+ /* Copy Match */
+ if (sequence.offset > (size_t)(oLitEnd - prefixStart)) {
+ /* offset beyond prefix -> go into extDict */
+ RETURN_ERROR_IF(UNLIKELY(sequence.offset > (size_t)(oLitEnd - virtualStart)), corruption_detected, "");
+ match = dictEnd + (match - prefixStart);
+ if (match + sequence.matchLength <= dictEnd) {
+ ZSTD_memmove(oLitEnd, match, sequence.matchLength);
+ return sequenceLength;
+ }
+ /* span extDict & currentPrefixSegment */
+ { size_t const length1 = dictEnd - match;
+ ZSTD_memmove(oLitEnd, match, length1);
+ op = oLitEnd + length1;
+ sequence.matchLength -= length1;
+ match = prefixStart;
+ }
+ }
+ /* Match within prefix of 1 or more bytes */
+ assert(op <= oMatchEnd);
+ assert(oMatchEnd <= oend_w);
+ assert(match >= prefixStart);
+ assert(sequence.matchLength >= 1);
+
+ /* Nearly all offsets are >= WILDCOPY_VECLEN bytes, which means we can use wildcopy
+ * without overlap checking.
+ */
+ if (LIKELY(sequence.offset >= WILDCOPY_VECLEN)) {
+ /* We bet on a full wildcopy for matches, since we expect matches to be
+ * longer than literals (in general). In silesia, ~10% of matches are longer
+ * than 16 bytes.
+ */
+ ZSTD_wildcopy(op, match, (ptrdiff_t)sequence.matchLength, ZSTD_no_overlap);
+ return sequenceLength;
+ }
+ assert(sequence.offset < WILDCOPY_VECLEN);
+
+ /* Copy 8 bytes and spread the offset to be >= 8. */
+ ZSTD_overlapCopy8(&op, &match, sequence.offset);
+
+ /* If the match length is > 8 bytes, then continue with the wildcopy. */
+ if (sequence.matchLength > 8) {
+ assert(op < oMatchEnd);
+ ZSTD_wildcopy(op, match, (ptrdiff_t)sequence.matchLength - 8, ZSTD_overlap_src_before_dst);
+ }
+ return sequenceLength;
+}
+
+HINT_INLINE
+size_t ZSTD_execSequenceSplitLitBuffer(BYTE* op,
+ BYTE* const oend, const BYTE* const oend_w, seq_t sequence,
+ const BYTE** litPtr, const BYTE* const litLimit,
+ const BYTE* const prefixStart, const BYTE* const virtualStart, const BYTE* const dictEnd)
+{
+ BYTE* const oLitEnd = op + sequence.litLength;
+ size_t const sequenceLength = sequence.litLength + sequence.matchLength;
+ BYTE* const oMatchEnd = op + sequenceLength; /* risk : address space overflow (32-bits) */
+ const BYTE* const iLitEnd = *litPtr + sequence.litLength;
+ const BYTE* match = oLitEnd - sequence.offset;
+
+ assert(op != NULL /* Precondition */);
+ assert(oend_w < oend /* No underflow */);
+ /* Handle edge cases in a slow path:
+ * - Read beyond end of literals
+ * - Match end is within WILDCOPY_OVERLIMIT of oend
+ * - 32-bit mode and the match length overflows
+ */
+ if (UNLIKELY(
iLitEnd > litLimit ||
oMatchEnd > oend_w ||
(MEM_32bits() && (size_t)(oend - op) < sequenceLength + WILDCOPY_OVERLENGTH)))
- return ZSTD_execSequenceEnd(op, oend, sequence, litPtr, litLimit, prefixStart, virtualStart, dictEnd);
+ return ZSTD_execSequenceEndSplitLitBuffer(op, oend, oend_w, sequence, litPtr, litLimit, prefixStart, virtualStart, dictEnd);
/* Assumptions (everything else goes into ZSTD_execSequenceEnd()) */
assert(op <= oLitEnd /* No overflow */);
@@ -896,6 +1133,7 @@ size_t ZSTD_execSequence(BYTE* op,
return sequenceLength;
}
+
static void
ZSTD_initFseState(ZSTD_fseState* DStatePtr, BIT_DStream_t* bitD, const ZSTD_seqSymbol* dt)
{
@@ -909,20 +1147,10 @@ ZSTD_initFseState(ZSTD_fseState* DStatePtr, BIT_DStream_t* bitD, const ZSTD_seqS
}
FORCE_INLINE_TEMPLATE void
-ZSTD_updateFseState(ZSTD_fseState* DStatePtr, BIT_DStream_t* bitD)
+ZSTD_updateFseStateWithDInfo(ZSTD_fseState* DStatePtr, BIT_DStream_t* bitD, U16 nextState, U32 nbBits)
{
- ZSTD_seqSymbol const DInfo = DStatePtr->table[DStatePtr->state];
- U32 const nbBits = DInfo.nbBits;
size_t const lowBits = BIT_readBits(bitD, nbBits);
- DStatePtr->state = DInfo.nextState + lowBits;
-}
-
-FORCE_INLINE_TEMPLATE void
-ZSTD_updateFseStateWithDInfo(ZSTD_fseState* DStatePtr, BIT_DStream_t* bitD, ZSTD_seqSymbol const DInfo)
-{
- U32 const nbBits = DInfo.nbBits;
- size_t const lowBits = BIT_readBits(bitD, nbBits);
- DStatePtr->state = DInfo.nextState + lowBits;
+ DStatePtr->state = nextState + lowBits;
}
/* We need to add at most (ZSTD_WINDOWLOG_MAX_32 - 1) bits to read the maximum
@@ -936,116 +1164,105 @@ ZSTD_updateFseStateWithDInfo(ZSTD_fseState* DStatePtr, BIT_DStream_t* bitD, ZSTD
: 0)
typedef enum { ZSTD_lo_isRegularOffset, ZSTD_lo_isLongOffset=1 } ZSTD_longOffset_e;
-typedef enum { ZSTD_p_noPrefetch=0, ZSTD_p_prefetch=1 } ZSTD_prefetch_e;
FORCE_INLINE_TEMPLATE seq_t
-ZSTD_decodeSequence(seqState_t* seqState, const ZSTD_longOffset_e longOffsets, const ZSTD_prefetch_e prefetch)
+ZSTD_decodeSequence(seqState_t* seqState, const ZSTD_longOffset_e longOffsets)
{
seq_t seq;
- ZSTD_seqSymbol const llDInfo = seqState->stateLL.table[seqState->stateLL.state];
- ZSTD_seqSymbol const mlDInfo = seqState->stateML.table[seqState->stateML.state];
- ZSTD_seqSymbol const ofDInfo = seqState->stateOffb.table[seqState->stateOffb.state];
- U32 const llBase = llDInfo.baseValue;
- U32 const mlBase = mlDInfo.baseValue;
- U32 const ofBase = ofDInfo.baseValue;
- BYTE const llBits = llDInfo.nbAdditionalBits;
- BYTE const mlBits = mlDInfo.nbAdditionalBits;
- BYTE const ofBits = ofDInfo.nbAdditionalBits;
- BYTE const totalBits = llBits+mlBits+ofBits;
+ const ZSTD_seqSymbol* const llDInfo = seqState->stateLL.table + seqState->stateLL.state;
+ const ZSTD_seqSymbol* const mlDInfo = seqState->stateML.table + seqState->stateML.state;
+ const ZSTD_seqSymbol* const ofDInfo = seqState->stateOffb.table + seqState->stateOffb.state;
+ seq.matchLength = mlDInfo->baseValue;
+ seq.litLength = llDInfo->baseValue;
+ { U32 const ofBase = ofDInfo->baseValue;
+ BYTE const llBits = llDInfo->nbAdditionalBits;
+ BYTE const mlBits = mlDInfo->nbAdditionalBits;
+ BYTE const ofBits = ofDInfo->nbAdditionalBits;
+ BYTE const totalBits = llBits+mlBits+ofBits;
- /* sequence */
- { size_t offset;
- if (ofBits > 1) {
- ZSTD_STATIC_ASSERT(ZSTD_lo_isLongOffset == 1);
- ZSTD_STATIC_ASSERT(LONG_OFFSETS_MAX_EXTRA_BITS_32 == 5);
- assert(ofBits <= MaxOff);
- if (MEM_32bits() && longOffsets && (ofBits >= STREAM_ACCUMULATOR_MIN_32)) {
- U32 const extraBits = ofBits - MIN(ofBits, 32 - seqState->DStream.bitsConsumed);
- offset = ofBase + (BIT_readBitsFast(&seqState->DStream, ofBits - extraBits) << extraBits);
- BIT_reloadDStream(&seqState->DStream);
- if (extraBits) offset += BIT_readBitsFast(&seqState->DStream, extraBits);
- assert(extraBits <= LONG_OFFSETS_MAX_EXTRA_BITS_32); /* to avoid another reload */
- } else {
- offset = ofBase + BIT_readBitsFast(&seqState->DStream, ofBits/*>0*/); /* <= (ZSTD_WINDOWLOG_MAX-1) bits */
- if (MEM_32bits()) BIT_reloadDStream(&seqState->DStream);
- }
- seqState->prevOffset[2] = seqState->prevOffset[1];
- seqState->prevOffset[1] = seqState->prevOffset[0];
- seqState->prevOffset[0] = offset;
- } else {
- U32 const ll0 = (llBase == 0);
- if (LIKELY((ofBits == 0))) {
- if (LIKELY(!ll0))
- offset = seqState->prevOffset[0];
- else {
- offset = seqState->prevOffset[1];
- seqState->prevOffset[1] = seqState->prevOffset[0];
- seqState->prevOffset[0] = offset;
+ U16 const llNext = llDInfo->nextState;
+ U16 const mlNext = mlDInfo->nextState;
+ U16 const ofNext = ofDInfo->nextState;
+ U32 const llnbBits = llDInfo->nbBits;
+ U32 const mlnbBits = mlDInfo->nbBits;
+ U32 const ofnbBits = ofDInfo->nbBits;
+ /*
+ * As gcc has better branch and block analyzers, sometimes it is only
+ * valuable to mark likelyness for clang, it gives around 3-4% of
+ * performance.
+ */
+
+ /* sequence */
+ { size_t offset;
+ #if defined(__clang__)
+ if (LIKELY(ofBits > 1)) {
+ #else
+ if (ofBits > 1) {
+ #endif
+ ZSTD_STATIC_ASSERT(ZSTD_lo_isLongOffset == 1);
+ ZSTD_STATIC_ASSERT(LONG_OFFSETS_MAX_EXTRA_BITS_32 == 5);
+ assert(ofBits <= MaxOff);
+ if (MEM_32bits() && longOffsets && (ofBits >= STREAM_ACCUMULATOR_MIN_32)) {
+ U32 const extraBits = ofBits - MIN(ofBits, 32 - seqState->DStream.bitsConsumed);
+ offset = ofBase + (BIT_readBitsFast(&seqState->DStream, ofBits - extraBits) << extraBits);
+ BIT_reloadDStream(&seqState->DStream);
+ if (extraBits) offset += BIT_readBitsFast(&seqState->DStream, extraBits);
+ assert(extraBits <= LONG_OFFSETS_MAX_EXTRA_BITS_32); /* to avoid another reload */
+ } else {
+ offset = ofBase + BIT_readBitsFast(&seqState->DStream, ofBits/*>0*/); /* <= (ZSTD_WINDOWLOG_MAX-1) bits */
+ if (MEM_32bits()) BIT_reloadDStream(&seqState->DStream);
}
+ seqState->prevOffset[2] = seqState->prevOffset[1];
+ seqState->prevOffset[1] = seqState->prevOffset[0];
+ seqState->prevOffset[0] = offset;
} else {
- offset = ofBase + ll0 + BIT_readBitsFast(&seqState->DStream, 1);
- { size_t temp = (offset==3) ? seqState->prevOffset[0] - 1 : seqState->prevOffset[offset];
- temp += !temp; /* 0 is not valid; input is corrupted; force offset to 1 */
- if (offset != 1) seqState->prevOffset[2] = seqState->prevOffset[1];
- seqState->prevOffset[1] = seqState->prevOffset[0];
- seqState->prevOffset[0] = offset = temp;
- } } }
- seq.offset = offset;
- }
-
- seq.matchLength = mlBase;
- if (mlBits > 0)
- seq.matchLength += BIT_readBitsFast(&seqState->DStream, mlBits/*>0*/);
-
- if (MEM_32bits() && (mlBits+llBits >= STREAM_ACCUMULATOR_MIN_32-LONG_OFFSETS_MAX_EXTRA_BITS_32))
- BIT_reloadDStream(&seqState->DStream);
- if (MEM_64bits() && UNLIKELY(totalBits >= STREAM_ACCUMULATOR_MIN_64-(LLFSELog+MLFSELog+OffFSELog)))
- BIT_reloadDStream(&seqState->DStream);
- /* Ensure there are enough bits to read the rest of data in 64-bit mode. */
- ZSTD_STATIC_ASSERT(16+LLFSELog+MLFSELog+OffFSELog < STREAM_ACCUMULATOR_MIN_64);
-
- seq.litLength = llBase;
- if (llBits > 0)
- seq.litLength += BIT_readBitsFast(&seqState->DStream, llBits/*>0*/);
-
- if (MEM_32bits())
- BIT_reloadDStream(&seqState->DStream);
-
- DEBUGLOG(6, "seq: litL=%u, matchL=%u, offset=%u",
- (U32)seq.litLength, (U32)seq.matchLength, (U32)seq.offset);
-
- if (prefetch == ZSTD_p_prefetch) {
- size_t const pos = seqState->pos + seq.litLength;
- const BYTE* const matchBase = (seq.offset > pos) ? seqState->dictEnd : seqState->prefixStart;
- seq.match = matchBase + pos - seq.offset; /* note : this operation can overflow when seq.offset is really too large, which can only happen when input is corrupted.
- * No consequence though : no memory access will occur, offset is only used for prefetching */
- seqState->pos = pos + seq.matchLength;
- }
-
- /* ANS state update
- * gcc-9.0.0 does 2.5% worse with ZSTD_updateFseStateWithDInfo().
- * clang-9.2.0 does 7% worse with ZSTD_updateFseState().
- * Naturally it seems like ZSTD_updateFseStateWithDInfo() should be the
- * better option, so it is the default for other compilers. But, if you
- * measure that it is worse, please put up a pull request.
- */
- {
-#if !defined(__clang__)
- const int kUseUpdateFseState = 1;
-#else
- const int kUseUpdateFseState = 0;
-#endif
- if (kUseUpdateFseState) {
- ZSTD_updateFseState(&seqState->stateLL, &seqState->DStream); /* <= 9 bits */
- ZSTD_updateFseState(&seqState->stateML, &seqState->DStream); /* <= 9 bits */
- if (MEM_32bits()) BIT_reloadDStream(&seqState->DStream); /* <= 18 bits */
- ZSTD_updateFseState(&seqState->stateOffb, &seqState->DStream); /* <= 8 bits */
- } else {
- ZSTD_updateFseStateWithDInfo(&seqState->stateLL, &seqState->DStream, llDInfo); /* <= 9 bits */
- ZSTD_updateFseStateWithDInfo(&seqState->stateML, &seqState->DStream, mlDInfo); /* <= 9 bits */
- if (MEM_32bits()) BIT_reloadDStream(&seqState->DStream); /* <= 18 bits */
- ZSTD_updateFseStateWithDInfo(&seqState->stateOffb, &seqState->DStream, ofDInfo); /* <= 8 bits */
+ U32 const ll0 = (llDInfo->baseValue == 0);
+ if (LIKELY((ofBits == 0))) {
+ offset = seqState->prevOffset[ll0];
+ seqState->prevOffset[1] = seqState->prevOffset[!ll0];
+ seqState->prevOffset[0] = offset;
+ } else {
+ offset = ofBase + ll0 + BIT_readBitsFast(&seqState->DStream, 1);
+ { size_t temp = (offset==3) ? seqState->prevOffset[0] - 1 : seqState->prevOffset[offset];
+ temp += !temp; /* 0 is not valid; input is corrupted; force offset to 1 */
+ if (offset != 1) seqState->prevOffset[2] = seqState->prevOffset[1];
+ seqState->prevOffset[1] = seqState->prevOffset[0];
+ seqState->prevOffset[0] = offset = temp;
+ } } }
+ seq.offset = offset;
}
+
+ #if defined(__clang__)
+ if (UNLIKELY(mlBits > 0))
+ #else
+ if (mlBits > 0)
+ #endif
+ seq.matchLength += BIT_readBitsFast(&seqState->DStream, mlBits/*>0*/);
+
+ if (MEM_32bits() && (mlBits+llBits >= STREAM_ACCUMULATOR_MIN_32-LONG_OFFSETS_MAX_EXTRA_BITS_32))
+ BIT_reloadDStream(&seqState->DStream);
+ if (MEM_64bits() && UNLIKELY(totalBits >= STREAM_ACCUMULATOR_MIN_64-(LLFSELog+MLFSELog+OffFSELog)))
+ BIT_reloadDStream(&seqState->DStream);
+ /* Ensure there are enough bits to read the rest of data in 64-bit mode. */
+ ZSTD_STATIC_ASSERT(16+LLFSELog+MLFSELog+OffFSELog < STREAM_ACCUMULATOR_MIN_64);
+
+ #if defined(__clang__)
+ if (UNLIKELY(llBits > 0))
+ #else
+ if (llBits > 0)
+ #endif
+ seq.litLength += BIT_readBitsFast(&seqState->DStream, llBits/*>0*/);
+
+ if (MEM_32bits())
+ BIT_reloadDStream(&seqState->DStream);
+
+ DEBUGLOG(6, "seq: litL=%u, matchL=%u, offset=%u",
+ (U32)seq.litLength, (U32)seq.matchLength, (U32)seq.offset);
+
+ ZSTD_updateFseStateWithDInfo(&seqState->stateLL, &seqState->DStream, llNext, llnbBits); /* <= 9 bits */
+ ZSTD_updateFseStateWithDInfo(&seqState->stateML, &seqState->DStream, mlNext, mlnbBits); /* <= 9 bits */
+ if (MEM_32bits()) BIT_reloadDStream(&seqState->DStream); /* <= 18 bits */
+ ZSTD_updateFseStateWithDInfo(&seqState->stateOffb, &seqState->DStream, ofNext, ofnbBits); /* <= 8 bits */
}
return seq;
@@ -1098,9 +1315,11 @@ MEM_STATIC void ZSTD_assertValidSequence(
#endif
#ifndef ZSTD_FORCE_DECOMPRESS_SEQUENCES_LONG
+
+
FORCE_INLINE_TEMPLATE size_t
DONT_VECTORIZE
-ZSTD_decompressSequences_body( ZSTD_DCtx* dctx,
+ZSTD_decompressSequences_bodySplitLitBuffer( ZSTD_DCtx* dctx,
void* dst, size_t maxDstSize,
const void* seqStart, size_t seqSize, int nbSeq,
const ZSTD_longOffset_e isLongOffset,
@@ -1112,17 +1331,16 @@ ZSTD_decompressSequences_body( ZSTD_DCtx* dctx,
BYTE* const oend = ostart + maxDstSize;
BYTE* op = ostart;
const BYTE* litPtr = dctx->litPtr;
- const BYTE* const litEnd = litPtr + dctx->litSize;
+ const BYTE* litBufferEnd = dctx->litBufferEnd;
const BYTE* const prefixStart = (const BYTE*) (dctx->prefixStart);
const BYTE* const vBase = (const BYTE*) (dctx->virtualStart);
const BYTE* const dictEnd = (const BYTE*) (dctx->dictEnd);
- DEBUGLOG(5, "ZSTD_decompressSequences_body");
+ DEBUGLOG(5, "ZSTD_decompressSequences_bodySplitLitBuffer");
(void)frame;
/* Regen sequences */
if (nbSeq) {
seqState_t seqState;
- size_t error = 0;
dctx->fseEntropy = 1;
{ U32 i; for (i=0; i<ZSTD_REP_NUM; i++) seqState.prevOffset[i] = dctx->entropy.rep[i]; }
RETURN_ERROR_IF(
@@ -1138,70 +1356,255 @@ ZSTD_decompressSequences_body( ZSTD_DCtx* dctx,
BIT_DStream_endOfBuffer < BIT_DStream_completed &&
BIT_DStream_completed < BIT_DStream_overflow);
+ /* decompress without overrunning litPtr begins */
+ {
+ seq_t sequence = ZSTD_decodeSequence(&seqState, isLongOffset);
+ /* Align the decompression loop to 32 + 16 bytes.
+ *
+ * zstd compiled with gcc-9 on an Intel i9-9900k shows 10% decompression
+ * speed swings based on the alignment of the decompression loop. This
+ * performance swing is caused by parts of the decompression loop falling
+ * out of the DSB. The entire decompression loop should fit in the DSB,
+ * when it can't we get much worse performance. You can measure if you've
+ * hit the good case or the bad case with this perf command for some
+ * compressed file test.zst:
+ *
+ * perf stat -e cycles -e instructions -e idq.all_dsb_cycles_any_uops \
+ * -e idq.all_mite_cycles_any_uops -- ./zstd -tq test.zst
+ *
+ * If you see most cycles served out of the MITE you've hit the bad case.
+ * If you see most cycles served out of the DSB you've hit the good case.
+ * If it is pretty even then you may be in an okay case.
+ *
+ * This issue has been reproduced on the following CPUs:
+ * - Kabylake: Macbook Pro (15-inch, 2019) 2.4 GHz Intel Core i9
+ * Use Instruments->Counters to get DSB/MITE cycles.
+ * I never got performance swings, but I was able to
+ * go from the good case of mostly DSB to half of the
+ * cycles served from MITE.
+ * - Coffeelake: Intel i9-9900k
+ * - Coffeelake: Intel i7-9700k
+ *
+ * I haven't been able to reproduce the instability or DSB misses on any
+ * of the following CPUS:
+ * - Haswell
+ * - Broadwell: Intel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GH
+ * - Skylake
+ *
+ * Alignment is done for each of the three major decompression loops:
+ * - ZSTD_decompressSequences_bodySplitLitBuffer - presplit section of the literal buffer
+ * - ZSTD_decompressSequences_bodySplitLitBuffer - postsplit section of the literal buffer
+ * - ZSTD_decompressSequences_body
+ * Alignment choices are made to minimize large swings on bad cases and influence on performance
+ * from changes external to this code, rather than to overoptimize on the current commit.
+ *
+ * If you are seeing performance stability this script can help test.
+ * It tests on 4 commits in zstd where I saw performance change.
+ *
+ * https://gist.github.com/terrelln/9889fc06a423fd5ca6e99351564473f4
+ */
#if defined(__x86_64__)
- /* Align the decompression loop to 32 + 16 bytes.
- *
- * zstd compiled with gcc-9 on an Intel i9-9900k shows 10% decompression
- * speed swings based on the alignment of the decompression loop. This
- * performance swing is caused by parts of the decompression loop falling
- * out of the DSB. The entire decompression loop should fit in the DSB,
- * when it can't we get much worse performance. You can measure if you've
- * hit the good case or the bad case with this perf command for some
- * compressed file test.zst:
- *
- * perf stat -e cycles -e instructions -e idq.all_dsb_cycles_any_uops \
- * -e idq.all_mite_cycles_any_uops -- ./zstd -tq test.zst
- *
- * If you see most cycles served out of the MITE you've hit the bad case.
- * If you see most cycles served out of the DSB you've hit the good case.
- * If it is pretty even then you may be in an okay case.
- *
- * I've been able to reproduce this issue on the following CPUs:
- * - Kabylake: Macbook Pro (15-inch, 2019) 2.4 GHz Intel Core i9
- * Use Instruments->Counters to get DSB/MITE cycles.
- * I never got performance swings, but I was able to
- * go from the good case of mostly DSB to half of the
- * cycles served from MITE.
- * - Coffeelake: Intel i9-9900k
- *
- * I haven't been able to reproduce the instability or DSB misses on any
- * of the following CPUS:
- * - Haswell
- * - Broadwell: Intel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GH
- * - Skylake
- *
- * If you are seeing performance stability this script can help test.
- * It tests on 4 commits in zstd where I saw performance change.
- *
- * https://gist.github.com/terrelln/9889fc06a423fd5ca6e99351564473f4
- */
- __asm__(".p2align 5");
- __asm__("nop");
- __asm__(".p2align 4");
+ __asm__(".p2align 6");
+# if __GNUC__ >= 7
+ /* good for gcc-7, gcc-9, and gcc-11 */
+ __asm__("nop");
+ __asm__(".p2align 5");
+ __asm__("nop");
+ __asm__(".p2align 4");
+# if __GNUC__ == 8 || __GNUC__ == 10
+ /* good for gcc-8 and gcc-10 */
+ __asm__("nop");
+ __asm__(".p2align 3");
+# endif
+# endif
#endif
+
+ /* Handle the initial state where litBuffer is currently split between dst and litExtraBuffer */
+ for (; litPtr + sequence.litLength <= dctx->litBufferEnd; ) {
+ size_t const oneSeqSize = ZSTD_execSequenceSplitLitBuffer(op, oend, litPtr + sequence.litLength - WILDCOPY_OVERLENGTH, sequence, &litPtr, litBufferEnd, prefixStart, vBase, dictEnd);
+#if defined(FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION) && defined(FUZZING_ASSERT_VALID_SEQUENCE)
+ assert(!ZSTD_isError(oneSeqSize));
+ if (frame) ZSTD_assertValidSequence(dctx, op, oend, sequence, prefixStart, vBase);
+#endif
+ if (UNLIKELY(ZSTD_isError(oneSeqSize)))
+ return oneSeqSize;
+ DEBUGLOG(6, "regenerated sequence size : %u", (U32)oneSeqSize);
+ op += oneSeqSize;
+ if (UNLIKELY(!--nbSeq))
+ break;
+ BIT_reloadDStream(&(seqState.DStream));
+ sequence = ZSTD_decodeSequence(&seqState, isLongOffset);
+ }
+
+ /* If there are more sequences, they will need to read literals from litExtraBuffer; copy over the remainder from dst and update litPtr and litEnd */
+ if (nbSeq > 0) {
+ const size_t leftoverLit = dctx->litBufferEnd - litPtr;
+ if (leftoverLit)
+ {
+ RETURN_ERROR_IF(leftoverLit > (size_t)(oend - op), dstSize_tooSmall, "remaining lit must fit within dstBuffer");
+ ZSTD_safecopyDstBeforeSrc(op, litPtr, leftoverLit);
+ sequence.litLength -= leftoverLit;
+ op += leftoverLit;
+ }
+ litPtr = dctx->litExtraBuffer;
+ litBufferEnd = dctx->litExtraBuffer + ZSTD_LITBUFFEREXTRASIZE;
+ dctx->litBufferLocation = ZSTD_not_in_dst;
+ {
+ size_t const oneSeqSize = ZSTD_execSequence(op, oend, sequence, &litPtr, litBufferEnd, prefixStart, vBase, dictEnd);
+#if defined(FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION) && defined(FUZZING_ASSERT_VALID_SEQUENCE)
+ assert(!ZSTD_isError(oneSeqSize));
+ if (frame) ZSTD_assertValidSequence(dctx, op, oend, sequence, prefixStart, vBase);
+#endif
+ if (UNLIKELY(ZSTD_isError(oneSeqSize)))
+ return oneSeqSize;
+ DEBUGLOG(6, "regenerated sequence size : %u", (U32)oneSeqSize);
+ op += oneSeqSize;
+ if (--nbSeq)
+ BIT_reloadDStream(&(seqState.DStream));
+ }
+ }
+ }
+
+ if (nbSeq > 0) /* there is remaining lit from extra buffer */
+ {
+
+#if defined(__x86_64__)
+ __asm__(".p2align 6");
+ __asm__("nop");
+# if __GNUC__ != 7
+ /* worse for gcc-7 better for gcc-8, gcc-9, and gcc-10 and clang */
+ __asm__(".p2align 4");
+ __asm__("nop");
+ __asm__(".p2align 3");
+# elif __GNUC__ >= 11
+ __asm__(".p2align 3");
+# else
+ __asm__(".p2align 5");
+ __asm__("nop");
+ __asm__(".p2align 3");
+# endif
+#endif
+
+ for (; ; ) {
+ seq_t const sequence = ZSTD_decodeSequence(&seqState, isLongOffset);
+ size_t const oneSeqSize = ZSTD_execSequence(op, oend, sequence, &litPtr, litBufferEnd, prefixStart, vBase, dictEnd);
+#if defined(FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION) && defined(FUZZING_ASSERT_VALID_SEQUENCE)
+ assert(!ZSTD_isError(oneSeqSize));
+ if (frame) ZSTD_assertValidSequence(dctx, op, oend, sequence, prefixStart, vBase);
+#endif
+ if (UNLIKELY(ZSTD_isError(oneSeqSize)))
+ return oneSeqSize;
+ DEBUGLOG(6, "regenerated sequence size : %u", (U32)oneSeqSize);
+ op += oneSeqSize;
+ if (UNLIKELY(!--nbSeq))
+ break;
+ BIT_reloadDStream(&(seqState.DStream));
+ }
+ }
+
+ /* check if reached exact end */
+ DEBUGLOG(5, "ZSTD_decompressSequences_bodySplitLitBuffer: after decode loop, remaining nbSeq : %i", nbSeq);
+ RETURN_ERROR_IF(nbSeq, corruption_detected, "");
+ RETURN_ERROR_IF(BIT_reloadDStream(&seqState.DStream) < BIT_DStream_completed, corruption_detected, "");
+ /* save reps for next block */
+ { U32 i; for (i=0; i<ZSTD_REP_NUM; i++) dctx->entropy.rep[i] = (U32)(seqState.prevOffset[i]); }
+ }
+
+ /* last literal segment */
+ if (dctx->litBufferLocation == ZSTD_split) /* split hasn't been reached yet, first get dst then copy litExtraBuffer */
+ {
+ size_t const lastLLSize = litBufferEnd - litPtr;
+ RETURN_ERROR_IF(lastLLSize > (size_t)(oend - op), dstSize_tooSmall, "");
+ if (op != NULL) {
+ ZSTD_memmove(op, litPtr, lastLLSize);
+ op += lastLLSize;
+ }
+ litPtr = dctx->litExtraBuffer;
+ litBufferEnd = dctx->litExtraBuffer + ZSTD_LITBUFFEREXTRASIZE;
+ dctx->litBufferLocation = ZSTD_not_in_dst;
+ }
+ { size_t const lastLLSize = litBufferEnd - litPtr;
+ RETURN_ERROR_IF(lastLLSize > (size_t)(oend-op), dstSize_tooSmall, "");
+ if (op != NULL) {
+ ZSTD_memcpy(op, litPtr, lastLLSize);
+ op += lastLLSize;
+ }
+ }
+
+ return op-ostart;
+}
+
+FORCE_INLINE_TEMPLATE size_t
+DONT_VECTORIZE
+ZSTD_decompressSequences_body(ZSTD_DCtx* dctx,
+ void* dst, size_t maxDstSize,
+ const void* seqStart, size_t seqSize, int nbSeq,
+ const ZSTD_longOffset_e isLongOffset,
+ const int frame)
+{
+ const BYTE* ip = (const BYTE*)seqStart;
+ const BYTE* const iend = ip + seqSize;
+ BYTE* const ostart = (BYTE*)dst;
+ BYTE* const oend = dctx->litBufferLocation == ZSTD_not_in_dst ? ostart + maxDstSize : dctx->litBuffer;
+ BYTE* op = ostart;
+ const BYTE* litPtr = dctx->litPtr;
+ const BYTE* const litEnd = litPtr + dctx->litSize;
+ const BYTE* const prefixStart = (const BYTE*)(dctx->prefixStart);
+ const BYTE* const vBase = (const BYTE*)(dctx->virtualStart);
+ const BYTE* const dictEnd = (const BYTE*)(dctx->dictEnd);
+ DEBUGLOG(5, "ZSTD_decompressSequences_body");
+ (void)frame;
+
+ /* Regen sequences */
+ if (nbSeq) {
+ seqState_t seqState;
+ dctx->fseEntropy = 1;
+ { U32 i; for (i = 0; i < ZSTD_REP_NUM; i++) seqState.prevOffset[i] = dctx->entropy.rep[i]; }
+ RETURN_ERROR_IF(
+ ERR_isError(BIT_initDStream(&seqState.DStream, ip, iend - ip)),
+ corruption_detected, "");
+ ZSTD_initFseState(&seqState.stateLL, &seqState.DStream, dctx->LLTptr);
+ ZSTD_initFseState(&seqState.stateOffb, &seqState.DStream, dctx->OFTptr);
+ ZSTD_initFseState(&seqState.stateML, &seqState.DStream, dctx->MLTptr);
+ assert(dst != NULL);
+
+ ZSTD_STATIC_ASSERT(
+ BIT_DStream_unfinished < BIT_DStream_completed &&
+ BIT_DStream_endOfBuffer < BIT_DStream_completed &&
+ BIT_DStream_completed < BIT_DStream_overflow);
+
+#if defined(__x86_64__)
+ __asm__(".p2align 6");
+ __asm__("nop");
+# if __GNUC__ >= 7
+ __asm__(".p2align 5");
+ __asm__("nop");
+ __asm__(".p2align 3");
+# else
+ __asm__(".p2align 4");
+ __asm__("nop");
+ __asm__(".p2align 3");
+# endif
+#endif
+
for ( ; ; ) {
- seq_t const sequence = ZSTD_decodeSequence(&seqState, isLongOffset, ZSTD_p_noPrefetch);
+ seq_t const sequence = ZSTD_decodeSequence(&seqState, isLongOffset);
size_t const oneSeqSize = ZSTD_execSequence(op, oend, sequence, &litPtr, litEnd, prefixStart, vBase, dictEnd);
#if defined(FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION) && defined(FUZZING_ASSERT_VALID_SEQUENCE)
assert(!ZSTD_isError(oneSeqSize));
if (frame) ZSTD_assertValidSequence(dctx, op, oend, sequence, prefixStart, vBase);
#endif
+ if (UNLIKELY(ZSTD_isError(oneSeqSize)))
+ return oneSeqSize;
DEBUGLOG(6, "regenerated sequence size : %u", (U32)oneSeqSize);
- BIT_reloadDStream(&(seqState.DStream));
op += oneSeqSize;
- /* gcc and clang both don't like early returns in this loop.
- * Instead break and check for an error at the end of the loop.
- */
- if (UNLIKELY(ZSTD_isError(oneSeqSize))) {
- error = oneSeqSize;
+ if (UNLIKELY(!--nbSeq))
break;
- }
- if (UNLIKELY(!--nbSeq)) break;
+ BIT_reloadDStream(&(seqState.DStream));
}
/* check if reached exact end */
DEBUGLOG(5, "ZSTD_decompressSequences_body: after decode loop, remaining nbSeq : %i", nbSeq);
- if (ZSTD_isError(error)) return error;
RETURN_ERROR_IF(nbSeq, corruption_detected, "");
RETURN_ERROR_IF(BIT_reloadDStream(&seqState.DStream) < BIT_DStream_completed, corruption_detected, "");
/* save reps for next block */
@@ -1229,9 +1632,37 @@ ZSTD_decompressSequences_default(ZSTD_DCtx* dctx,
{
return ZSTD_decompressSequences_body(dctx, dst, maxDstSize, seqStart, seqSize, nbSeq, isLongOffset, frame);
}
+
+static size_t
+ZSTD_decompressSequencesSplitLitBuffer_default(ZSTD_DCtx* dctx,
+ void* dst, size_t maxDstSize,
+ const void* seqStart, size_t seqSize, int nbSeq,
+ const ZSTD_longOffset_e isLongOffset,
+ const int frame)
+{
+ return ZSTD_decompressSequences_bodySplitLitBuffer(dctx, dst, maxDstSize, seqStart, seqSize, nbSeq, isLongOffset, frame);
+}
#endif /* ZSTD_FORCE_DECOMPRESS_SEQUENCES_LONG */
#ifndef ZSTD_FORCE_DECOMPRESS_SEQUENCES_SHORT
+
+FORCE_INLINE_TEMPLATE size_t
+ZSTD_prefetchMatch(size_t prefetchPos, seq_t const sequence,
+ const BYTE* const prefixStart, const BYTE* const dictEnd)
+{
+ prefetchPos += sequence.litLength;
+ { const BYTE* const matchBase = (sequence.offset > prefetchPos) ? dictEnd : prefixStart;
+ const BYTE* const match = matchBase + prefetchPos - sequence.offset; /* note : this operation can overflow when seq.offset is really too large, which can only happen when input is corrupted.
+ * No consequence though : memory address is only used for prefetching, not for dereferencing */
+ PREFETCH_L1(match); PREFETCH_L1(match+CACHELINE_SIZE); /* note : it's safe to invoke PREFETCH() on any memory address, including invalid ones */
+ }
+ return prefetchPos + sequence.matchLength;
+}
+
+/* This decoding function employs prefetching
+ * to reduce latency impact of cache misses.
+ * It's generally employed when block contains a significant portion of long-distance matches
+ * or when coupled with a "cold" dictionary */
FORCE_INLINE_TEMPLATE size_t
ZSTD_decompressSequencesLong_body(
ZSTD_DCtx* dctx,
@@ -1243,10 +1674,10 @@ ZSTD_decompressSequencesLong_body(
const BYTE* ip = (const BYTE*)seqStart;
const BYTE* const iend = ip + seqSize;
BYTE* const ostart = (BYTE*)dst;
- BYTE* const oend = ostart + maxDstSize;
+ BYTE* const oend = dctx->litBufferLocation == ZSTD_in_dst ? dctx->litBuffer : ostart + maxDstSize;
BYTE* op = ostart;
const BYTE* litPtr = dctx->litPtr;
- const BYTE* const litEnd = litPtr + dctx->litSize;
+ const BYTE* litBufferEnd = dctx->litBufferEnd;
const BYTE* const prefixStart = (const BYTE*) (dctx->prefixStart);
const BYTE* const dictStart = (const BYTE*) (dctx->virtualStart);
const BYTE* const dictEnd = (const BYTE*) (dctx->dictEnd);
@@ -1254,18 +1685,17 @@ ZSTD_decompressSequencesLong_body(
/* Regen sequences */
if (nbSeq) {
-#define STORED_SEQS 4
+#define STORED_SEQS 8
#define STORED_SEQS_MASK (STORED_SEQS-1)
-#define ADVANCED_SEQS 4
+#define ADVANCED_SEQS STORED_SEQS
seq_t sequences[STORED_SEQS];
int const seqAdvance = MIN(nbSeq, ADVANCED_SEQS);
seqState_t seqState;
int seqNb;
+ size_t prefetchPos = (size_t)(op-prefixStart); /* track position relative to prefixStart */
+
dctx->fseEntropy = 1;
{ int i; for (i=0; i<ZSTD_REP_NUM; i++) seqState.prevOffset[i] = dctx->entropy.rep[i]; }
- seqState.prefixStart = prefixStart;
- seqState.pos = (size_t)(op-prefixStart);
- seqState.dictEnd = dictEnd;
assert(dst != NULL);
assert(iend >= ip);
RETURN_ERROR_IF(
@@ -1277,36 +1707,100 @@ ZSTD_decompressSequencesLong_body(
/* prepare in advance */
for (seqNb=0; (BIT_reloadDStream(&seqState.DStream) <= BIT_DStream_completed) && (seqNb<seqAdvance); seqNb++) {
- sequences[seqNb] = ZSTD_decodeSequence(&seqState, isLongOffset, ZSTD_p_prefetch);
- PREFETCH_L1(sequences[seqNb].match); PREFETCH_L1(sequences[seqNb].match + sequences[seqNb].matchLength - 1); /* note : it's safe to invoke PREFETCH() on any memory address, including invalid ones */
+ seq_t const sequence = ZSTD_decodeSequence(&seqState, isLongOffset);
+ prefetchPos = ZSTD_prefetchMatch(prefetchPos, sequence, prefixStart, dictEnd);
+ sequences[seqNb] = sequence;
}
RETURN_ERROR_IF(seqNb<seqAdvance, corruption_detected, "");
- /* decode and decompress */
- for ( ; (BIT_reloadDStream(&(seqState.DStream)) <= BIT_DStream_completed) && (seqNb<nbSeq) ; seqNb++) {
- seq_t const sequence = ZSTD_decodeSequence(&seqState, isLongOffset, ZSTD_p_prefetch);
- size_t const oneSeqSize = ZSTD_execSequence(op, oend, sequences[(seqNb-ADVANCED_SEQS) & STORED_SEQS_MASK], &litPtr, litEnd, prefixStart, dictStart, dictEnd);
+ /* decompress without stomping litBuffer */
+ for (; (BIT_reloadDStream(&(seqState.DStream)) <= BIT_DStream_completed) && (seqNb < nbSeq); seqNb++) {
+ seq_t sequence = ZSTD_decodeSequence(&seqState, isLongOffset);
+ size_t oneSeqSize;
+
+ if (dctx->litBufferLocation == ZSTD_split && litPtr + sequences[(seqNb - ADVANCED_SEQS) & STORED_SEQS_MASK].litLength > dctx->litBufferEnd)
+ {
+ /* lit buffer is reaching split point, empty out the first buffer and transition to litExtraBuffer */
+ const size_t leftoverLit = dctx->litBufferEnd - litPtr;
+ if (leftoverLit)
+ {
+ RETURN_ERROR_IF(leftoverLit > (size_t)(oend - op), dstSize_tooSmall, "remaining lit must fit within dstBuffer");
+ ZSTD_safecopyDstBeforeSrc(op, litPtr, leftoverLit);
+ sequences[(seqNb - ADVANCED_SEQS) & STORED_SEQS_MASK].litLength -= leftoverLit;
+ op += leftoverLit;
+ }
+ litPtr = dctx->litExtraBuffer;
+ litBufferEnd = dctx->litExtraBuffer + ZSTD_LITBUFFEREXTRASIZE;
+ dctx->litBufferLocation = ZSTD_not_in_dst;
+ oneSeqSize = ZSTD_execSequence(op, oend, sequences[(seqNb - ADVANCED_SEQS) & STORED_SEQS_MASK], &litPtr, litBufferEnd, prefixStart, dictStart, dictEnd);
#if defined(FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION) && defined(FUZZING_ASSERT_VALID_SEQUENCE)
- assert(!ZSTD_isError(oneSeqSize));
- if (frame) ZSTD_assertValidSequence(dctx, op, oend, sequences[(seqNb-ADVANCED_SEQS) & STORED_SEQS_MASK], prefixStart, dictStart);
+ assert(!ZSTD_isError(oneSeqSize));
+ if (frame) ZSTD_assertValidSequence(dctx, op, oend, sequences[(seqNb - ADVANCED_SEQS) & STORED_SEQS_MASK], prefixStart, dictStart);
#endif
- if (ZSTD_isError(oneSeqSize)) return oneSeqSize;
- PREFETCH_L1(sequence.match); PREFETCH_L1(sequence.match + sequence.matchLength - 1); /* note : it's safe to invoke PREFETCH() on any memory address, including invalid ones */
- sequences[seqNb & STORED_SEQS_MASK] = sequence;
- op += oneSeqSize;
+ if (ZSTD_isError(oneSeqSize)) return oneSeqSize;
+
+ prefetchPos = ZSTD_prefetchMatch(prefetchPos, sequence, prefixStart, dictEnd);
+ sequences[seqNb & STORED_SEQS_MASK] = sequence;
+ op += oneSeqSize;
+ }
+ else
+ {
+ /* lit buffer is either wholly contained in first or second split, or not split at all*/
+ oneSeqSize = dctx->litBufferLocation == ZSTD_split ?
+ ZSTD_execSequenceSplitLitBuffer(op, oend, litPtr + sequences[(seqNb - ADVANCED_SEQS) & STORED_SEQS_MASK].litLength - WILDCOPY_OVERLENGTH, sequences[(seqNb - ADVANCED_SEQS) & STORED_SEQS_MASK], &litPtr, litBufferEnd, prefixStart, dictStart, dictEnd) :
+ ZSTD_execSequence(op, oend, sequences[(seqNb - ADVANCED_SEQS) & STORED_SEQS_MASK], &litPtr, litBufferEnd, prefixStart, dictStart, dictEnd);
+#if defined(FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION) && defined(FUZZING_ASSERT_VALID_SEQUENCE)
+ assert(!ZSTD_isError(oneSeqSize));
+ if (frame) ZSTD_assertValidSequence(dctx, op, oend, sequences[(seqNb - ADVANCED_SEQS) & STORED_SEQS_MASK], prefixStart, dictStart);
+#endif
+ if (ZSTD_isError(oneSeqSize)) return oneSeqSize;
+
+ prefetchPos = ZSTD_prefetchMatch(prefetchPos, sequence, prefixStart, dictEnd);
+ sequences[seqNb & STORED_SEQS_MASK] = sequence;
+ op += oneSeqSize;
+ }
}
RETURN_ERROR_IF(seqNb<nbSeq, corruption_detected, "");
/* finish queue */
seqNb -= seqAdvance;
for ( ; seqNb<nbSeq ; seqNb++) {
- size_t const oneSeqSize = ZSTD_execSequence(op, oend, sequences[seqNb&STORED_SEQS_MASK], &litPtr, litEnd, prefixStart, dictStart, dictEnd);
+ seq_t *sequence = &(sequences[seqNb&STORED_SEQS_MASK]);
+ if (dctx->litBufferLocation == ZSTD_split && litPtr + sequence->litLength > dctx->litBufferEnd)
+ {
+ const size_t leftoverLit = dctx->litBufferEnd - litPtr;
+ if (leftoverLit)
+ {
+ RETURN_ERROR_IF(leftoverLit > (size_t)(oend - op), dstSize_tooSmall, "remaining lit must fit within dstBuffer");
+ ZSTD_safecopyDstBeforeSrc(op, litPtr, leftoverLit);
+ sequence->litLength -= leftoverLit;
+ op += leftoverLit;
+ }
+ litPtr = dctx->litExtraBuffer;
+ litBufferEnd = dctx->litExtraBuffer + ZSTD_LITBUFFEREXTRASIZE;
+ dctx->litBufferLocation = ZSTD_not_in_dst;
+ {
+ size_t const oneSeqSize = ZSTD_execSequence(op, oend, *sequence, &litPtr, litBufferEnd, prefixStart, dictStart, dictEnd);
#if defined(FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION) && defined(FUZZING_ASSERT_VALID_SEQUENCE)
- assert(!ZSTD_isError(oneSeqSize));
- if (frame) ZSTD_assertValidSequence(dctx, op, oend, sequences[seqNb&STORED_SEQS_MASK], prefixStart, dictStart);
+ assert(!ZSTD_isError(oneSeqSize));
+ if (frame) ZSTD_assertValidSequence(dctx, op, oend, sequences[seqNb&STORED_SEQS_MASK], prefixStart, dictStart);
#endif
- if (ZSTD_isError(oneSeqSize)) return oneSeqSize;
- op += oneSeqSize;
+ if (ZSTD_isError(oneSeqSize)) return oneSeqSize;
+ op += oneSeqSize;
+ }
+ }
+ else
+ {
+ size_t const oneSeqSize = dctx->litBufferLocation == ZSTD_split ?
+ ZSTD_execSequenceSplitLitBuffer(op, oend, litPtr + sequence->litLength - WILDCOPY_OVERLENGTH, *sequence, &litPtr, litBufferEnd, prefixStart, dictStart, dictEnd) :
+ ZSTD_execSequence(op, oend, *sequence, &litPtr, litBufferEnd, prefixStart, dictStart, dictEnd);
+#if defined(FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION) && defined(FUZZING_ASSERT_VALID_SEQUENCE)
+ assert(!ZSTD_isError(oneSeqSize));
+ if (frame) ZSTD_assertValidSequence(dctx, op, oend, sequences[seqNb&STORED_SEQS_MASK], prefixStart, dictStart);
+#endif
+ if (ZSTD_isError(oneSeqSize)) return oneSeqSize;
+ op += oneSeqSize;
+ }
}
/* save reps for next block */
@@ -1314,10 +1808,21 @@ ZSTD_decompressSequencesLong_body(
}
/* last literal segment */
- { size_t const lastLLSize = litEnd - litPtr;
+ if (dctx->litBufferLocation == ZSTD_split) /* first deplete literal buffer in dst, then copy litExtraBuffer */
+ {
+ size_t const lastLLSize = litBufferEnd - litPtr;
+ RETURN_ERROR_IF(lastLLSize > (size_t)(oend - op), dstSize_tooSmall, "");
+ if (op != NULL) {
+ ZSTD_memmove(op, litPtr, lastLLSize);
+ op += lastLLSize;
+ }
+ litPtr = dctx->litExtraBuffer;
+ litBufferEnd = dctx->litExtraBuffer + ZSTD_LITBUFFEREXTRASIZE;
+ }
+ { size_t const lastLLSize = litBufferEnd - litPtr;
RETURN_ERROR_IF(lastLLSize > (size_t)(oend-op), dstSize_tooSmall, "");
if (op != NULL) {
- ZSTD_memcpy(op, litPtr, lastLLSize);
+ ZSTD_memmove(op, litPtr, lastLLSize);
op += lastLLSize;
}
}
@@ -1341,7 +1846,7 @@ ZSTD_decompressSequencesLong_default(ZSTD_DCtx* dctx,
#if DYNAMIC_BMI2
#ifndef ZSTD_FORCE_DECOMPRESS_SEQUENCES_LONG
-static TARGET_ATTRIBUTE("bmi2") size_t
+static BMI2_TARGET_ATTRIBUTE size_t
DONT_VECTORIZE
ZSTD_decompressSequences_bmi2(ZSTD_DCtx* dctx,
void* dst, size_t maxDstSize,
@@ -1351,10 +1856,20 @@ ZSTD_decompressSequences_bmi2(ZSTD_DCtx* dctx,
{
return ZSTD_decompressSequences_body(dctx, dst, maxDstSize, seqStart, seqSize, nbSeq, isLongOffset, frame);
}
+static BMI2_TARGET_ATTRIBUTE size_t
+DONT_VECTORIZE
+ZSTD_decompressSequencesSplitLitBuffer_bmi2(ZSTD_DCtx* dctx,
+ void* dst, size_t maxDstSize,
+ const void* seqStart, size_t seqSize, int nbSeq,
+ const ZSTD_longOffset_e isLongOffset,
+ const int frame)
+{
+ return ZSTD_decompressSequences_bodySplitLitBuffer(dctx, dst, maxDstSize, seqStart, seqSize, nbSeq, isLongOffset, frame);
+}
#endif /* ZSTD_FORCE_DECOMPRESS_SEQUENCES_LONG */
#ifndef ZSTD_FORCE_DECOMPRESS_SEQUENCES_SHORT
-static TARGET_ATTRIBUTE("bmi2") size_t
+static BMI2_TARGET_ATTRIBUTE size_t
ZSTD_decompressSequencesLong_bmi2(ZSTD_DCtx* dctx,
void* dst, size_t maxDstSize,
const void* seqStart, size_t seqSize, int nbSeq,
@@ -1383,11 +1898,25 @@ ZSTD_decompressSequences(ZSTD_DCtx* dctx, void* dst, size_t maxDstSize,
{
DEBUGLOG(5, "ZSTD_decompressSequences");
#if DYNAMIC_BMI2
- if (dctx->bmi2) {
+ if (ZSTD_DCtx_get_bmi2(dctx)) {
return ZSTD_decompressSequences_bmi2(dctx, dst, maxDstSize, seqStart, seqSize, nbSeq, isLongOffset, frame);
}
#endif
- return ZSTD_decompressSequences_default(dctx, dst, maxDstSize, seqStart, seqSize, nbSeq, isLongOffset, frame);
+ return ZSTD_decompressSequences_default(dctx, dst, maxDstSize, seqStart, seqSize, nbSeq, isLongOffset, frame);
+}
+static size_t
+ZSTD_decompressSequencesSplitLitBuffer(ZSTD_DCtx* dctx, void* dst, size_t maxDstSize,
+ const void* seqStart, size_t seqSize, int nbSeq,
+ const ZSTD_longOffset_e isLongOffset,
+ const int frame)
+{
+ DEBUGLOG(5, "ZSTD_decompressSequencesSplitLitBuffer");
+#if DYNAMIC_BMI2
+ if (ZSTD_DCtx_get_bmi2(dctx)) {
+ return ZSTD_decompressSequencesSplitLitBuffer_bmi2(dctx, dst, maxDstSize, seqStart, seqSize, nbSeq, isLongOffset, frame);
+ }
+#endif
+ return ZSTD_decompressSequencesSplitLitBuffer_default(dctx, dst, maxDstSize, seqStart, seqSize, nbSeq, isLongOffset, frame);
}
#endif /* ZSTD_FORCE_DECOMPRESS_SEQUENCES_LONG */
@@ -1407,7 +1936,7 @@ ZSTD_decompressSequencesLong(ZSTD_DCtx* dctx,
{
DEBUGLOG(5, "ZSTD_decompressSequencesLong");
#if DYNAMIC_BMI2
- if (dctx->bmi2) {
+ if (ZSTD_DCtx_get_bmi2(dctx)) {
return ZSTD_decompressSequencesLong_bmi2(dctx, dst, maxDstSize, seqStart, seqSize, nbSeq, isLongOffset, frame);
}
#endif
@@ -1448,7 +1977,7 @@ ZSTD_getLongOffsetsShare(const ZSTD_seqSymbol* offTable)
size_t
ZSTD_decompressBlock_internal(ZSTD_DCtx* dctx,
void* dst, size_t dstCapacity,
- const void* src, size_t srcSize, const int frame)
+ const void* src, size_t srcSize, const int frame, const streaming_operation streaming)
{ /* blockType == blockCompressed */
const BYTE* ip = (const BYTE*)src;
/* isLongOffset must be true if there are long offsets.
@@ -1463,7 +1992,7 @@ ZSTD_decompressBlock_internal(ZSTD_DCtx* dctx,
RETURN_ERROR_IF(srcSize >= ZSTD_BLOCKSIZE_MAX, srcSize_wrong, "");
/* Decode literals section */
- { size_t const litCSize = ZSTD_decodeLiteralsBlock(dctx, src, srcSize);
+ { size_t const litCSize = ZSTD_decodeLiteralsBlock(dctx, src, srcSize, dst, dstCapacity, streaming);
DEBUGLOG(5, "ZSTD_decodeLiteralsBlock : %u", (U32)litCSize);
if (ZSTD_isError(litCSize)) return litCSize;
ip += litCSize;
@@ -1511,7 +2040,10 @@ ZSTD_decompressBlock_internal(ZSTD_DCtx* dctx,
#ifndef ZSTD_FORCE_DECOMPRESS_SEQUENCES_LONG
/* else */
- return ZSTD_decompressSequences(dctx, dst, dstCapacity, ip, srcSize, nbSeq, isLongOffset, frame);
+ if (dctx->litBufferLocation == ZSTD_split)
+ return ZSTD_decompressSequencesSplitLitBuffer(dctx, dst, dstCapacity, ip, srcSize, nbSeq, isLongOffset, frame);
+ else
+ return ZSTD_decompressSequences(dctx, dst, dstCapacity, ip, srcSize, nbSeq, isLongOffset, frame);
#endif
}
}
@@ -1534,7 +2066,7 @@ size_t ZSTD_decompressBlock(ZSTD_DCtx* dctx,
{
size_t dSize;
ZSTD_checkContinuity(dctx, dst, dstCapacity);
- dSize = ZSTD_decompressBlock_internal(dctx, dst, dstCapacity, src, srcSize, /* frame */ 0);
+ dSize = ZSTD_decompressBlock_internal(dctx, dst, dstCapacity, src, srcSize, /* frame */ 0, not_streaming);
dctx->previousDstEnd = (char*)dst + dSize;
return dSize;
}
diff --git a/lib/zstd/decompress/zstd_decompress_block.h b/lib/zstd/decompress/zstd_decompress_block.h
index e7f5f66..3d2d57a 100644
--- a/lib/zstd/decompress/zstd_decompress_block.h
+++ b/lib/zstd/decompress/zstd_decompress_block.h
@@ -33,6 +33,12 @@
*/
+ /* Streaming state is used to inform allocation of the literal buffer */
+typedef enum {
+ not_streaming = 0,
+ is_streaming = 1
+} streaming_operation;
+
/* ZSTD_decompressBlock_internal() :
* decompress block, starting at `src`,
* into destination buffer `dst`.
@@ -41,7 +47,7 @@
*/
size_t ZSTD_decompressBlock_internal(ZSTD_DCtx* dctx,
void* dst, size_t dstCapacity,
- const void* src, size_t srcSize, const int frame);
+ const void* src, size_t srcSize, const int frame, const streaming_operation streaming);
/* ZSTD_buildFSETable() :
* generate FSE decoding table for one symbol (ll, ml or off)
@@ -54,7 +60,7 @@ size_t ZSTD_decompressBlock_internal(ZSTD_DCtx* dctx,
*/
void ZSTD_buildFSETable(ZSTD_seqSymbol* dt,
const short* normalizedCounter, unsigned maxSymbolValue,
- const U32* baseValue, const U32* nbAdditionalBits,
+ const U32* baseValue, const U8* nbAdditionalBits,
unsigned tableLog, void* wksp, size_t wkspSize,
int bmi2);
diff --git a/lib/zstd/decompress/zstd_decompress_internal.h b/lib/zstd/decompress/zstd_decompress_internal.h
index 4b9052f..98102ed 100644
--- a/lib/zstd/decompress/zstd_decompress_internal.h
+++ b/lib/zstd/decompress/zstd_decompress_internal.h
@@ -20,7 +20,7 @@
* Dependencies
*********************************************************/
#include "../common/mem.h" /* BYTE, U16, U32 */
-#include "../common/zstd_internal.h" /* ZSTD_seqSymbol */
+#include "../common/zstd_internal.h" /* constants : MaxLL, MaxML, MaxOff, LLFSELog, etc. */
@@ -40,7 +40,7 @@ static UNUSED_ATTR const U32 OF_base[MaxOff+1] = {
0xFFFD, 0x1FFFD, 0x3FFFD, 0x7FFFD, 0xFFFFD, 0x1FFFFD, 0x3FFFFD, 0x7FFFFD,
0xFFFFFD, 0x1FFFFFD, 0x3FFFFFD, 0x7FFFFFD, 0xFFFFFFD, 0x1FFFFFFD, 0x3FFFFFFD, 0x7FFFFFFD };
-static UNUSED_ATTR const U32 OF_bits[MaxOff+1] = {
+static UNUSED_ATTR const U8 OF_bits[MaxOff+1] = {
0, 1, 2, 3, 4, 5, 6, 7,
8, 9, 10, 11, 12, 13, 14, 15,
16, 17, 18, 19, 20, 21, 22, 23,
@@ -106,6 +106,22 @@ typedef struct {
size_t ddictPtrCount;
} ZSTD_DDictHashSet;
+#ifndef ZSTD_DECODER_INTERNAL_BUFFER
+# define ZSTD_DECODER_INTERNAL_BUFFER (1 << 16)
+#endif
+
+#define ZSTD_LBMIN 64
+#define ZSTD_LBMAX (128 << 10)
+
+/* extra buffer, compensates when dst is not large enough to store litBuffer */
+#define ZSTD_LITBUFFEREXTRASIZE BOUNDED(ZSTD_LBMIN, ZSTD_DECODER_INTERNAL_BUFFER, ZSTD_LBMAX)
+
+typedef enum {
+ ZSTD_not_in_dst = 0, /* Stored entirely within litExtraBuffer */
+ ZSTD_in_dst = 1, /* Stored entirely within dst (in memory after current output write) */
+ ZSTD_split = 2 /* Split between litExtraBuffer and dst */
+} ZSTD_litLocation_e;
+
struct ZSTD_DCtx_s
{
const ZSTD_seqSymbol* LLTptr;
@@ -136,7 +152,9 @@ struct ZSTD_DCtx_s
size_t litSize;
size_t rleSize;
size_t staticSize;
+#if DYNAMIC_BMI2 != 0
int bmi2; /* == 1 if the CPU supports BMI2 and 0 otherwise. CPU support is determined dynamically once per context lifetime. */
+#endif
/* dictionary */
ZSTD_DDict* ddictLocal;
@@ -158,16 +176,16 @@ struct ZSTD_DCtx_s
size_t outStart;
size_t outEnd;
size_t lhSize;
- void* legacyContext;
- U32 previousLegacyVersion;
- U32 legacyVersion;
U32 hostageByte;
int noForwardProgress;
ZSTD_bufferMode_e outBufferMode;
ZSTD_outBuffer expectedOutBuffer;
/* workspace */
- BYTE litBuffer[ZSTD_BLOCKSIZE_MAX + WILDCOPY_OVERLENGTH];
+ BYTE* litBuffer;
+ const BYTE* litBufferEnd;
+ ZSTD_litLocation_e litBufferLocation;
+ BYTE litExtraBuffer[ZSTD_LITBUFFEREXTRASIZE + WILDCOPY_OVERLENGTH]; /* literal buffer can be split between storage within dst and within this scratch buffer */
BYTE headerBuffer[ZSTD_FRAMEHEADERSIZE_MAX];
size_t oversizedDuration;
@@ -180,6 +198,14 @@ struct ZSTD_DCtx_s
/* Tracing */
}; /* typedef'd to ZSTD_DCtx within "zstd.h" */
+MEM_STATIC int ZSTD_DCtx_get_bmi2(const struct ZSTD_DCtx_s *dctx) {
+#if DYNAMIC_BMI2 != 0
+ return dctx->bmi2;
+#else
+ (void)dctx;
+ return 0;
+#endif
+}
/*-*******************************************************
* Shared internal functions
diff --git a/lib/zstd/decompress_sources.h b/lib/zstd/decompress_sources.h
index 0fbec50..a06ca18 100644
--- a/lib/zstd/decompress_sources.h
+++ b/lib/zstd/decompress_sources.h
@@ -16,6 +16,12 @@
* decompression.
*/
+/*
+ * Disable the ASM Huffman implementation because we need to
+ * include all the sources.
+ */
+#define ZSTD_DISABLE_ASM 1
+
#include "common/debug.c"
#include "common/entropy_common.c"
#include "common/error_private.c"
diff --git a/lib/zstd/zstd_common_module.c b/lib/zstd/zstd_common_module.c
new file mode 100644
index 0000000..22686e3
--- /dev/null
+++ b/lib/zstd/zstd_common_module.c
@@ -0,0 +1,32 @@
+// SPDX-License-Identifier: GPL-2.0+ OR BSD-3-Clause
+/*
+ * Copyright (c) Facebook, Inc.
+ * All rights reserved.
+ *
+ * This source code is licensed under both the BSD-style license (found in the
+ * LICENSE file in the root directory of this source tree) and the GPLv2 (found
+ * in the COPYING file in the root directory of this source tree).
+ * You may select, at your option, one of the above-listed licenses.
+ */
+
+#include <linux/module.h>
+
+#include "common/huf.h"
+#include "common/fse.h"
+#include "common/zstd_internal.h"
+
+// Export symbols shared by compress and decompress into a common module
+
+#undef ZSTD_isError /* defined within zstd_internal.h */
+EXPORT_SYMBOL_GPL(FSE_readNCount);
+EXPORT_SYMBOL_GPL(HUF_readStats);
+EXPORT_SYMBOL_GPL(HUF_readStats_wksp);
+EXPORT_SYMBOL_GPL(ZSTD_isError);
+EXPORT_SYMBOL_GPL(ZSTD_getErrorName);
+EXPORT_SYMBOL_GPL(ZSTD_getErrorCode);
+EXPORT_SYMBOL_GPL(ZSTD_customMalloc);
+EXPORT_SYMBOL_GPL(ZSTD_customCalloc);
+EXPORT_SYMBOL_GPL(ZSTD_customFree);
+
+MODULE_LICENSE("Dual BSD/GPL");
+MODULE_DESCRIPTION("Zstd Common");
diff --git a/lib/zstd/zstd_compress_module.c b/lib/zstd/zstd_compress_module.c
index 65548a4..04e1b5c 100644
--- a/lib/zstd/zstd_compress_module.c
+++ b/lib/zstd/zstd_compress_module.c
@@ -133,7 +133,11 @@ EXPORT_SYMBOL(zstd_init_cstream);
size_t zstd_reset_cstream(zstd_cstream *cstream,
unsigned long long pledged_src_size)
{
- return ZSTD_resetCStream(cstream, pledged_src_size);
+ if (pledged_src_size == 0)
+ pledged_src_size = ZSTD_CONTENTSIZE_UNKNOWN;
+ ZSTD_FORWARD_IF_ERR( ZSTD_CCtx_reset(cstream, ZSTD_reset_session_only) );
+ ZSTD_FORWARD_IF_ERR( ZSTD_CCtx_setPledgedSrcSize(cstream, pledged_src_size) );
+ return 0;
}
EXPORT_SYMBOL(zstd_reset_cstream);
diff --git a/mm/Kconfig b/mm/Kconfig
index 3425708..ff7b209d 100644
--- a/mm/Kconfig
+++ b/mm/Kconfig
@@ -1078,7 +1078,7 @@
comment "GUP_TEST needs to have DEBUG_FS enabled"
depends on !GUP_TEST && !DEBUG_FS
-config GUP_GET_PTE_LOW_HIGH
+config GUP_GET_PXX_LOW_HIGH
bool
config ARCH_HAS_PTE_SPECIAL
diff --git a/mm/gup.c b/mm/gup.c
index 2b45d78..f45a3a5 100644
--- a/mm/gup.c
+++ b/mm/gup.c
@@ -2721,7 +2721,7 @@ static int gup_pmd_range(pud_t *pudp, pud_t pud, unsigned long addr, unsigned lo
pmdp = pmd_offset_lockless(pudp, pud, addr);
do {
- pmd_t pmd = READ_ONCE(*pmdp);
+ pmd_t pmd = pmdp_get_lockless(pmdp);
next = pmd_addr_end(addr, end);
if (!pmd_present(pmd))
diff --git a/mm/gup_test.c b/mm/gup_test.c
index 33f431e..8ae7307 100644
--- a/mm/gup_test.c
+++ b/mm/gup_test.c
@@ -214,7 +214,7 @@ static inline void pin_longterm_test_stop(void)
if (pin_longterm_test_nr_pages)
unpin_user_pages(pin_longterm_test_pages,
pin_longterm_test_nr_pages);
- kfree(pin_longterm_test_pages);
+ kvfree(pin_longterm_test_pages);
pin_longterm_test_pages = NULL;
pin_longterm_test_nr_pages = 0;
}
@@ -255,7 +255,7 @@ static inline int pin_longterm_test_start(unsigned long arg)
fast = !!(args.flags & PIN_LONGTERM_TEST_FLAG_USE_FAST);
if (!fast && mmap_read_lock_killable(current->mm)) {
- kfree(pages);
+ kvfree(pages);
return -EINTR;
}
diff --git a/mm/hmm.c b/mm/hmm.c
index 3850fb6..601a99c 100644
--- a/mm/hmm.c
+++ b/mm/hmm.c
@@ -361,8 +361,7 @@ static int hmm_vma_walk_pmd(pmd_t *pmdp,
* huge or device mapping one and compute corresponding pfn
* values.
*/
- pmd = pmd_read_atomic(pmdp);
- barrier();
+ pmd = pmdp_get_lockless(pmdp);
if (!pmd_devmap(pmd) && !pmd_trans_huge(pmd))
goto again;
diff --git a/mm/khugepaged.c b/mm/khugepaged.c
index 5a7d2d5..5cb401a 100644
--- a/mm/khugepaged.c
+++ b/mm/khugepaged.c
@@ -857,7 +857,7 @@ static int find_pmd_or_thp_or_none(struct mm_struct *mm,
if (!*pmd)
return SCAN_PMD_NULL;
- pmde = pmd_read_atomic(*pmd);
+ pmde = pmdp_get_lockless(*pmd);
#ifdef CONFIG_TRANSPARENT_HUGEPAGE
/* See comments in pmd_none_or_trans_huge_or_clear_bad() */
diff --git a/mm/kmemleak.c b/mm/kmemleak.c
index 2673329..92f670e 100644
--- a/mm/kmemleak.c
+++ b/mm/kmemleak.c
@@ -79,6 +79,7 @@
#include <linux/mutex.h>
#include <linux/rcupdate.h>
#include <linux/stacktrace.h>
+#include <linux/stackdepot.h>
#include <linux/cache.h>
#include <linux/percpu.h>
#include <linux/memblock.h>
@@ -159,8 +160,7 @@ struct kmemleak_object {
u32 checksum;
/* memory ranges to be scanned inside an object (empty for all) */
struct hlist_head area_list;
- unsigned long trace[MAX_TRACE];
- unsigned int trace_len;
+ depot_stack_handle_t trace_handle;
unsigned long jiffies; /* creation timestamp */
pid_t pid; /* pid of the current task */
char comm[TASK_COMM_LEN]; /* executable name */
@@ -346,19 +346,22 @@ static void print_unreferenced(struct seq_file *seq,
struct kmemleak_object *object)
{
int i;
+ unsigned long *entries;
+ unsigned int nr_entries;
unsigned int msecs_age = jiffies_to_msecs(jiffies - object->jiffies);
+ nr_entries = stack_depot_fetch(object->trace_handle, &entries);
warn_or_seq_printf(seq, "unreferenced object 0x%08lx (size %zu):\n",
- object->pointer, object->size);
+ object->pointer, object->size);
warn_or_seq_printf(seq, " comm \"%s\", pid %d, jiffies %lu (age %d.%03ds)\n",
- object->comm, object->pid, object->jiffies,
- msecs_age / 1000, msecs_age % 1000);
+ object->comm, object->pid, object->jiffies,
+ msecs_age / 1000, msecs_age % 1000);
hex_dump_object(seq, object);
warn_or_seq_printf(seq, " backtrace:\n");
- for (i = 0; i < object->trace_len; i++) {
- void *ptr = (void *)object->trace[i];
- warn_or_seq_printf(seq, " [<%p>] %pS\n", ptr, ptr);
+ for (i = 0; i < nr_entries; i++) {
+ void *ptr = (void *)entries[i];
+ warn_or_seq_printf(seq, " [<%pK>] %pS\n", ptr, ptr);
}
}
@@ -370,15 +373,16 @@ static void print_unreferenced(struct seq_file *seq,
static void dump_object_info(struct kmemleak_object *object)
{
pr_notice("Object 0x%08lx (size %zu):\n",
- object->pointer, object->size);
+ object->pointer, object->size);
pr_notice(" comm \"%s\", pid %d, jiffies %lu\n",
- object->comm, object->pid, object->jiffies);
+ object->comm, object->pid, object->jiffies);
pr_notice(" min_count = %d\n", object->min_count);
pr_notice(" count = %d\n", object->count);
pr_notice(" flags = 0x%x\n", object->flags);
pr_notice(" checksum = %u\n", object->checksum);
pr_notice(" backtrace:\n");
- stack_trace_print(object->trace, object->trace_len, 4);
+ if (object->trace_handle)
+ stack_depot_print(object->trace_handle);
}
/*
@@ -591,12 +595,18 @@ static struct kmemleak_object *find_and_remove_object(unsigned long ptr, int ali
return object;
}
-/*
- * Save stack trace to the given array of MAX_TRACE size.
- */
-static int __save_stack_trace(unsigned long *trace)
+static noinline depot_stack_handle_t set_track_prepare(void)
{
- return stack_trace_save(trace, MAX_TRACE, 2);
+ depot_stack_handle_t trace_handle;
+ unsigned long entries[MAX_TRACE];
+ unsigned int nr_entries;
+
+ if (!kmemleak_initialized)
+ return 0;
+ nr_entries = stack_trace_save(entries, ARRAY_SIZE(entries), 3);
+ trace_handle = stack_depot_save(entries, nr_entries, GFP_NOWAIT);
+
+ return trace_handle;
}
/*
@@ -653,7 +663,7 @@ static void __create_object(unsigned long ptr, size_t size,
}
/* kernel backtrace */
- object->trace_len = __save_stack_trace(object->trace);
+ object->trace_handle = set_track_prepare();
raw_spin_lock_irqsave(&kmemleak_lock, flags);
@@ -692,7 +702,6 @@ static void __create_object(unsigned long ptr, size_t size,
rb_link_node(&object->rb_node, rb_parent, link);
rb_insert_color(&object->rb_node, is_phys ? &object_phys_tree_root :
&object_tree_root);
-
list_add_tail_rcu(&object->object_list, &object_list);
out:
raw_spin_unlock_irqrestore(&kmemleak_lock, flags);
@@ -1091,7 +1100,7 @@ void __ref kmemleak_update_trace(const void *ptr)
}
raw_spin_lock_irqsave(&object->lock, flags);
- object->trace_len = __save_stack_trace(object->trace);
+ object->trace_handle = set_track_prepare();
raw_spin_unlock_irqrestore(&object->lock, flags);
put_object(object);
@@ -2084,6 +2093,7 @@ void __init kmemleak_init(void)
if (kmemleak_error)
return;
+ stack_depot_init();
jiffies_min_age = msecs_to_jiffies(MSECS_MIN_AGE);
jiffies_scan_wait = msecs_to_jiffies(SECS_SCAN_WAIT * 1000);
diff --git a/mm/mapping_dirty_helpers.c b/mm/mapping_dirty_helpers.c
index 1b0ab8f..175e424 100644
--- a/mm/mapping_dirty_helpers.c
+++ b/mm/mapping_dirty_helpers.c
@@ -126,7 +126,7 @@ static int clean_record_pte(pte_t *pte, unsigned long addr,
static int wp_clean_pmd_entry(pmd_t *pmd, unsigned long addr, unsigned long end,
struct mm_walk *walk)
{
- pmd_t pmdval = pmd_read_atomic(pmd);
+ pmd_t pmdval = pmdp_get_lockless(pmd);
if (!pmd_trans_unstable(&pmdval))
return 0;
diff --git a/mm/mprotect.c b/mm/mprotect.c
index 093cb50..908df12 100644
--- a/mm/mprotect.c
+++ b/mm/mprotect.c
@@ -297,7 +297,7 @@ static unsigned long change_pte_range(struct mmu_gather *tlb,
*/
static inline int pmd_none_or_clear_bad_unless_trans_huge(pmd_t *pmd)
{
- pmd_t pmdval = pmd_read_atomic(pmd);
+ pmd_t pmdval = pmdp_get_lockless(pmd);
/* See pmd_none_or_trans_huge_or_clear_bad for info on barrier */
#ifdef CONFIG_TRANSPARENT_HUGEPAGE
diff --git a/mm/userfaultfd.c b/mm/userfaultfd.c
index b7a9479..0499907 100644
--- a/mm/userfaultfd.c
+++ b/mm/userfaultfd.c
@@ -632,7 +632,7 @@ static __always_inline ssize_t __mcopy_atomic(struct mm_struct *dst_mm,
break;
}
- dst_pmdval = pmd_read_atomic(dst_pmd);
+ dst_pmdval = pmdp_get_lockless(dst_pmd);
/*
* If the dst_pmd is mapped as THP don't
* override it and just be strict.
diff --git a/mm/vmscan.c b/mm/vmscan.c
index aba991c..bd6637f 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -4084,10 +4084,7 @@ static void walk_pmd_range(pud_t *pud, unsigned long start, unsigned long end,
/* walk_pte_range() may call get_next_vma() */
vma = args->vma;
for (i = pmd_index(start), addr = start; addr != end; i++, addr = next) {
- pmd_t val = pmd_read_atomic(pmd + i);
-
- /* for pmd_read_atomic() */
- barrier();
+ pmd_t val = pmdp_get_lockless(pmd + i);
next = pmd_addr_end(addr, end);
diff --git a/net/bpf/bpf_dummy_struct_ops.c b/net/bpf/bpf_dummy_struct_ops.c
index 2d434c1..1ac4467 100644
--- a/net/bpf/bpf_dummy_struct_ops.c
+++ b/net/bpf/bpf_dummy_struct_ops.c
@@ -124,8 +124,7 @@ int bpf_struct_ops_test_run(struct bpf_prog *prog, const union bpf_attr *kattr,
if (err < 0)
goto out;
- set_memory_ro((long)image, 1);
- set_memory_x((long)image, 1);
+ set_memory_rox((long)image, 1);
prog_ret = dummy_ops_call_op(image, args);
err = dummy_ops_copy_args(args);
diff --git a/scripts/Kbuild.include b/scripts/Kbuild.include
index 2bc08ac..2f7356b 100644
--- a/scripts/Kbuild.include
+++ b/scripts/Kbuild.include
@@ -10,6 +10,26 @@
space := $(empty) $(empty)
space_escape := _-_SPACE_-_
pound := \#
+define newline
+
+
+endef
+
+###
+# Comparison macros.
+# Usage: $(call test-lt, $(CONFIG_LLD_VERSION), 150000)
+#
+# Use $(intcmp ...) if supported. (Make >= 4.4)
+# Otherwise, fall back to the 'test' shell command.
+ifeq ($(intcmp 1,0,,,y),y)
+test-ge = $(intcmp $(strip $1)0, $(strip $2)0,,y,y)
+test-gt = $(intcmp $(strip $1)0, $(strip $2)0,,,y)
+else
+test-ge = $(shell test $(strip $1)0 -ge $(strip $2)0 && echo y)
+test-gt = $(shell test $(strip $1)0 -gt $(strip $2)0 && echo y)
+endif
+test-le = $(call test-ge, $2, $1)
+test-lt = $(call test-gt, $2, $1)
###
# Name of target with a '.' as filename prefix. foo/bar.o => foo/.bar.o
@@ -41,6 +61,21 @@
stringify = $(squote)$(quote)$1$(quote)$(squote)
###
+# The path to Kbuild or Makefile. Kbuild has precedence over Makefile.
+kbuild-dir = $(if $(filter /%,$(src)),$(src),$(srctree)/$(src))
+kbuild-file = $(or $(wildcard $(kbuild-dir)/Kbuild),$(kbuild-dir)/Makefile)
+
+###
+# Read a file, replacing newlines with spaces
+#
+# Make 4.2 or later can read a file by using its builtin function.
+ifneq ($(filter-out 3.% 4.0 4.1, $(MAKE_VERSION)),)
+read-file = $(subst $(newline),$(space),$(file < $1))
+else
+read-file = $(shell cat $1 2>/dev/null)
+endif
+
+###
# Easy method for doing a status message
kecho := :
quiet_kecho := echo
@@ -150,9 +185,6 @@
make-cmd = $(call escsq,$(subst $(pound),$$(pound),$(subst $$,$$$$,$(cmd_$(1)))))
# Find any prerequisites that are newer than target or that do not exist.
-# (This is not true for now; $? should contain any non-existent prerequisites,
-# but it does not work as expected when .SECONDARY is present. This seems a bug
-# of GNU Make.)
# PHONY targets skipped in both cases.
newer-prereqs = $(filter-out $(PHONY),$?)
@@ -228,4 +260,14 @@
.DELETE_ON_ERROR:
# do not delete intermediate files automatically
+#
+# .NOTINTERMEDIATE is more correct, but only available on newer Make versions.
+# Make 4.4 introduced .NOTINTERMEDIATE, and it appears in .FEATURES, but the
+# global .NOTINTERMEDIATE does not work. We can use it on Make > 4.4.
+# Use .SECONDARY for older Make versions, but "newer-prereq" cannot detect
+# deleted files.
+ifneq ($(and $(filter notintermediate, $(.FEATURES)),$(filter-out 4.4,$(MAKE_VERSION))),)
+.NOTINTERMEDIATE:
+else
.SECONDARY:
+endif
diff --git a/scripts/Makefile.asm-generic b/scripts/Makefile.asm-generic
index 1d501c57..8d01b37 100644
--- a/scripts/Makefile.asm-generic
+++ b/scripts/Makefile.asm-generic
@@ -10,15 +10,15 @@
all:
src := $(subst /generated,,$(obj))
--include $(src)/Kbuild
+
+include $(srctree)/scripts/Kbuild.include
+-include $(kbuild-file)
# $(generic)/Kbuild lists mandatory-y. Exclude um since it is a special case.
ifneq ($(SRCARCH),um)
include $(srctree)/$(generic)/Kbuild
endif
-include $(srctree)/scripts/Kbuild.include
-
redundant := $(filter $(mandatory-y) $(generated-y), $(generic-y))
redundant += $(foreach f, $(generic-y), $(if $(wildcard $(srctree)/$(src)/$(f)),$(f)))
redundant := $(sort $(redundant))
diff --git a/scripts/Makefile.build b/scripts/Makefile.build
index 3ceab07..a0d5c6c 100644
--- a/scripts/Makefile.build
+++ b/scripts/Makefile.build
@@ -38,11 +38,7 @@
include $(srctree)/scripts/Kbuild.include
include $(srctree)/scripts/Makefile.compiler
-
-# The filename Kbuild has precedence over Makefile
-kbuild-dir := $(if $(filter /%,$(src)),$(src),$(srctree)/$(src))
-include $(or $(wildcard $(kbuild-dir)/Kbuild),$(kbuild-dir)/Makefile)
-
+include $(kbuild-file)
include $(srctree)/scripts/Makefile.lib
# Do not include hostprogs rules unless needed.
@@ -226,6 +222,10 @@
cmd_check_local_export = $(srctree)/scripts/check-local-export $@
+ifneq ($(findstring 1, $(KBUILD_EXTRA_WARN)),)
+cmd_warn_shared_object = $(if $(word 2, $(modname-multi)),$(warning $(kbuild-file): $*.o is added to multiple modules: $(modname-multi)))
+endif
+
define rule_cc_o_c
$(call cmd_and_fixdep,cc_o_c)
$(call cmd,gen_ksymdeps)
@@ -235,6 +235,7 @@
$(call cmd,gen_objtooldep)
$(call cmd,gen_symversions_c)
$(call cmd,record_mcount)
+ $(call cmd,warn_shared_object)
endef
define rule_as_o_S
@@ -243,6 +244,7 @@
$(call cmd,check_local_export)
$(call cmd,gen_objtooldep)
$(call cmd,gen_symversions_S)
+ $(call cmd,warn_shared_object)
endef
# Built-in and composite module parts
@@ -433,7 +435,7 @@
# modules.order unless contained modules are updated.
cmd_modules_order = { $(foreach m, $(real-prereqs), \
- $(if $(filter %/modules.order, $m), cat $m, echo $(patsubst %.o,%.ko,$m));) :; } \
+ $(if $(filter %/modules.order, $m), cat $m, echo $m);) :; } \
> $@
$(obj)/modules.order: $(obj-m) FORCE
diff --git a/scripts/Makefile.clean b/scripts/Makefile.clean
index 878cec6..3649900 100644
--- a/scripts/Makefile.clean
+++ b/scripts/Makefile.clean
@@ -9,10 +9,7 @@
__clean:
include $(srctree)/scripts/Kbuild.include
-
-# The filename Kbuild has precedence over Makefile
-kbuild-dir := $(if $(filter /%,$(src)),$(src),$(srctree)/$(src))
-include $(or $(wildcard $(kbuild-dir)/Kbuild),$(kbuild-dir)/Makefile)
+include $(kbuild-file)
# Figure out what we need to build from the various variables
# ==========================================================================
diff --git a/scripts/Makefile.compiler b/scripts/Makefile.compiler
index 20d353d..3d8adfd 100644
--- a/scripts/Makefile.compiler
+++ b/scripts/Makefile.compiler
@@ -63,11 +63,11 @@
# gcc-min-version
# Usage: cflags-$(call gcc-min-version, 70100) += -foo
-gcc-min-version = $(shell [ $(CONFIG_GCC_VERSION)0 -ge $(1)0 ] && echo y)
+gcc-min-version = $(call test-ge, $(CONFIG_GCC_VERSION), $1)
# clang-min-version
# Usage: cflags-$(call clang-min-version, 110000) += -foo
-clang-min-version = $(shell [ $(CONFIG_CLANG_VERSION)0 -ge $(1)0 ] && echo y)
+clang-min-version = $(call test-ge, $(CONFIG_CLANG_VERSION), $1)
# ld-option
# Usage: KBUILD_LDFLAGS += $(call ld-option, -X, -Y)
diff --git a/scripts/Makefile.debug b/scripts/Makefile.debug
index 332c486..059ff38 100644
--- a/scripts/Makefile.debug
+++ b/scripts/Makefile.debug
@@ -27,10 +27,14 @@
DEBUG_RUSTFLAGS += -Cdebuginfo=2
endif
-ifdef CONFIG_DEBUG_INFO_COMPRESSED
+ifdef CONFIG_DEBUG_INFO_COMPRESSED_ZLIB
DEBUG_CFLAGS += -gz=zlib
KBUILD_AFLAGS += -gz=zlib
KBUILD_LDFLAGS += --compress-debug-sections=zlib
+else ifdef CONFIG_DEBUG_INFO_COMPRESSED_ZSTD
+DEBUG_CFLAGS += -gz=zstd
+KBUILD_AFLAGS += -gz=zstd
+KBUILD_LDFLAGS += --compress-debug-sections=zstd
endif
KBUILD_CFLAGS += $(DEBUG_CFLAGS)
diff --git a/scripts/Makefile.dtbinst b/scripts/Makefile.dtbinst
index 190d781..2ab936e 100644
--- a/scripts/Makefile.dtbinst
+++ b/scripts/Makefile.dtbinst
@@ -15,7 +15,7 @@
include include/config/auto.conf
include $(srctree)/scripts/Kbuild.include
-include $(src)/Makefile
+include $(kbuild-file)
dtbs := $(addprefix $(dst)/, $(dtb-y) $(if $(CONFIG_OF_ALL_DTBS),$(dtb-)))
subdirs := $(addprefix $(obj)/, $(subdir-y) $(subdir-m))
diff --git a/scripts/Makefile.extrawarn b/scripts/Makefile.extrawarn
index 6bbba36..40cd13e 100644
--- a/scripts/Makefile.extrawarn
+++ b/scripts/Makefile.extrawarn
@@ -38,6 +38,7 @@
KBUILD_CFLAGS += -Wno-type-limits
KBUILD_CFLAGS += -Wno-shift-negative-value
+KBUILD_CPPFLAGS += -Wundef
KBUILD_CPPFLAGS += -DKBUILD_EXTRA_WARN1
else
diff --git a/scripts/Makefile.lib b/scripts/Makefile.lib
index b28ac83..4a4a5f6 100644
--- a/scripts/Makefile.lib
+++ b/scripts/Makefile.lib
@@ -258,6 +258,9 @@
objtool-args-$(CONFIG_X86_KERNEL_IBT) += --ibt
objtool-args-$(CONFIG_FINEIBT) += --cfi
objtool-args-$(CONFIG_FTRACE_MCOUNT_USE_OBJTOOL) += --mcount
+ifdef CONFIG_FTRACE_MCOUNT_USE_OBJTOOL
+objtool-args-$(CONFIG_HAVE_OBJTOOL_NOP_MCOUNT) += --mnop
+endif
objtool-args-$(CONFIG_UNWINDER_ORC) += --orc
objtool-args-$(CONFIG_RETPOLINE) += --retpoline
objtool-args-$(CONFIG_RETHUNK) += --rethunk
diff --git a/scripts/Makefile.modfinal b/scripts/Makefile.modfinal
index 25bedd8..a30d5b0 100644
--- a/scripts/Makefile.modfinal
+++ b/scripts/Makefile.modfinal
@@ -13,9 +13,9 @@
include $(srctree)/scripts/Makefile.lib
# find all modules listed in modules.order
-modules := $(sort $(shell cat $(MODORDER)))
+modules := $(call read-file, $(MODORDER))
-__modfinal: $(modules)
+__modfinal: $(modules:%.o=%.ko)
@:
# modname and part-of-module are set to make c_flags define proper module flags
@@ -57,13 +57,13 @@
printf '%s\n' 'cmd_$@ := $(make-cmd)' > $(dot-target).cmd, @:)
# Re-generate module BTFs if either module's .ko or vmlinux changed
-$(modules): %.ko: %.o %.mod.o scripts/module.lds $(and $(CONFIG_DEBUG_INFO_BTF_MODULES),$(KBUILD_BUILTIN),vmlinux) FORCE
+%.ko: %.o %.mod.o scripts/module.lds $(and $(CONFIG_DEBUG_INFO_BTF_MODULES),$(KBUILD_BUILTIN),vmlinux) FORCE
+$(call if_changed_except,ld_ko_o,vmlinux)
ifdef CONFIG_DEBUG_INFO_BTF_MODULES
+$(if $(newer-prereqs),$(call cmd,btf_ko))
endif
-targets += $(modules) $(modules:.ko=.mod.o)
+targets += $(modules:%.o=%.ko) $(modules:%.o=%.mod.o)
# Add FORCE to the prequisites of a target to force it to be always rebuilt.
# ---------------------------------------------------------------------------
diff --git a/scripts/Makefile.modinst b/scripts/Makefile.modinst
index a4c987c..836391e 100644
--- a/scripts/Makefile.modinst
+++ b/scripts/Makefile.modinst
@@ -9,7 +9,7 @@
include include/config/auto.conf
include $(srctree)/scripts/Kbuild.include
-modules := $(sort $(shell cat $(MODORDER)))
+modules := $(call read-file, $(MODORDER))
ifeq ($(KBUILD_EXTMOD),)
dst := $(MODLIB)/kernel
@@ -26,7 +26,7 @@
suffix-$(CONFIG_MODULE_COMPRESS_XZ) := .xz
suffix-$(CONFIG_MODULE_COMPRESS_ZSTD) := .zst
-modules := $(patsubst $(extmod_prefix)%, $(dst)/%$(suffix-y), $(modules))
+modules := $(patsubst $(extmod_prefix)%.o, $(dst)/%.ko$(suffix-y), $(modules))
__modinst: $(modules)
@:
diff --git a/scripts/Makefile.modpost b/scripts/Makefile.modpost
index e41dee6..5eb5e82 100644
--- a/scripts/Makefile.modpost
+++ b/scripts/Makefile.modpost
@@ -38,6 +38,8 @@
include include/config/auto.conf
include $(srctree)/scripts/Kbuild.include
+MODPOST = scripts/mod/modpost
+
modpost-args = \
$(if $(CONFIG_MODVERSIONS),-m) \
$(if $(CONFIG_MODULE_SRCVERSION_ALL),-a) \
@@ -46,6 +48,8 @@
$(if $(CONFIG_MODULE_ALLOW_MISSING_NAMESPACE_IMPORTS)$(KBUILD_NSDEPS),-N) \
-o $@
+modpost-deps := $(MODPOST)
+
# 'make -i -k' ignores compile errors, and builds as many modules as possible.
ifneq ($(findstring i,$(filter-out --%,$(MAKEFLAGS))),)
modpost-args += -n
@@ -78,12 +82,13 @@
.vmlinux.objs: vmlinux.a $(KBUILD_VMLINUX_LIBS) FORCE
$(call if_changed,vmlinux_objs)
-vmlinux.o-if-present := $(wildcard vmlinux.o)
-output-symdump := vmlinux.symvers
-
-ifdef KBUILD_MODULES
-output-symdump := $(if $(vmlinux.o-if-present), Module.symvers, modules-only.symvers)
-missing-input := $(filter-out $(vmlinux.o-if-present),vmlinux.o)
+ifeq ($(wildcard vmlinux.o),)
+missing-input := vmlinux.o
+output-symdump := modules-only.symvers
+else
+modpost-args += vmlinux.o
+modpost-deps += vmlinux.o
+output-symdump := $(if $(KBUILD_MODULES), Module.symvers, vmlinux.symvers)
endif
else
@@ -93,13 +98,18 @@
src := $(obj)
# Include the module's Makefile to find KBUILD_EXTRA_SYMBOLS
-include $(or $(wildcard $(src)/Kbuild), $(src)/Makefile)
+include $(kbuild-file)
-module.symvers-if-present := $(wildcard Module.symvers)
output-symdump := $(KBUILD_EXTMOD)/Module.symvers
-missing-input := $(filter-out $(module.symvers-if-present), Module.symvers)
-modpost-args += -e $(addprefix -i ,$(module.symvers-if-present) $(KBUILD_EXTRA_SYMBOLS))
+ifeq ($(wildcard Module.symvers),)
+missing-input := Module.symvers
+else
+modpost-args += -i Module.symvers
+modpost-deps += Module.symvers
+endif
+
+modpost-args += -e $(addprefix -i , $(KBUILD_EXTRA_SYMBOLS))
endif # ($(KBUILD_EXTMOD),)
@@ -107,9 +117,10 @@
modpost-args += -w
endif
-modorder-if-needed := $(if $(KBUILD_MODULES), $(MODORDER))
-
-MODPOST = scripts/mod/modpost
+ifdef KBUILD_MODULES
+modpost-args += -T $(MODORDER)
+modpost-deps += $(MODORDER)
+endif
# Read out modules.order to pass in modpost.
# Otherwise, allmodconfig would fail with "Argument list too long".
@@ -119,10 +130,10 @@
echo >&2 "WARNING: $(missing-input) is missing."; \
echo >&2 " Modules may not have dependencies or modversions."; \
echo >&2 " You may get many unresolved symbol warnings.";) \
- sed 's/ko$$/o/' $(or $(modorder-if-needed), /dev/null) | $(MODPOST) $(modpost-args) -T - $(vmlinux.o-if-present)
+ $(MODPOST) $(modpost-args)
targets += $(output-symdump)
-$(output-symdump): $(modorder-if-needed) $(vmlinux.o-if-present) $(module.symvers-if-present) $(MODPOST) FORCE
+$(output-symdump): $(modpost-deps) FORCE
$(call if_changed,modpost)
__modpost: $(output-symdump)
diff --git a/scripts/Makefile.package b/scripts/Makefile.package
index 2a90139..539e9f7 100644
--- a/scripts/Makefile.package
+++ b/scripts/Makefile.package
@@ -62,6 +62,16 @@
+rpmbuild $(RPMOPTS) --target $(UTS_MACHINE)-linux -ta $(KERNELPATH).tar.gz \
--define='_smp_mflags %{nil}'
+# srcrpm-pkg
+# ---------------------------------------------------------------------------
+PHONY += srcrpm-pkg
+srcrpm-pkg:
+ $(MAKE) clean
+ $(CONFIG_SHELL) $(MKSPEC) >$(objtree)/kernel.spec
+ $(call cmd,src_tar,$(KERNELPATH),kernel.spec)
+ +rpmbuild $(RPMOPTS) --target $(UTS_MACHINE)-linux -ts $(KERNELPATH).tar.gz \
+ --define='_smp_mflags %{nil}' --define='_srcrpmdir $(srctree)'
+
# binrpm-pkg
# ---------------------------------------------------------------------------
PHONY += binrpm-pkg
diff --git a/scripts/clang-tools/gen_compile_commands.py b/scripts/clang-tools/gen_compile_commands.py
index d800b2c..0227522 100755
--- a/scripts/clang-tools/gen_compile_commands.py
+++ b/scripts/clang-tools/gen_compile_commands.py
@@ -138,10 +138,10 @@
"""
with open(modorder) as f:
for line in f:
- ko = line.rstrip()
- base, ext = os.path.splitext(ko)
- if ext != '.ko':
- sys.exit('{}: module path must end with .ko'.format(ko))
+ obj = line.rstrip()
+ base, ext = os.path.splitext(obj)
+ if ext != '.o':
+ sys.exit('{}: module path must end with .o'.format(obj))
mod = base + '.mod'
# Read from *.mod, to get a list of objects that compose the module.
with open(mod) as m:
diff --git a/scripts/gen_autoksyms.sh b/scripts/gen_autoksyms.sh
index 653fadb..12bcfae 100755
--- a/scripts/gen_autoksyms.sh
+++ b/scripts/gen_autoksyms.sh
@@ -48,7 +48,7 @@
EOT
{
- [ -n "${read_modorder}" ] && sed 's/ko$/usyms/' modules.order | xargs cat
+ [ -n "${read_modorder}" ] && sed 's/o$/usyms/' modules.order | xargs cat
echo "$needed_symbols"
[ -n "$ksym_wl" ] && cat "$ksym_wl"
} | sed -e 's/ /\n/g' | sed -n -e '/^$/!p' |
diff --git a/scripts/jobserver-exec b/scripts/jobserver-exec
index 8762887..4192855 100755
--- a/scripts/jobserver-exec
+++ b/scripts/jobserver-exec
@@ -23,7 +23,9 @@
opts = [x for x in flags.split(" ") if x.startswith("--jobserver")]
# Parse out R,W file descriptor numbers and set them nonblocking.
- fds = opts[0].split("=", 1)[1]
+ # If the MAKEFLAGS variable contains multiple instances of the
+ # --jobserver-auth= option, the last one is relevant.
+ fds = opts[-1].split("=", 1)[1]
reader, writer = [int(x) for x in fds.split(",", 1)]
# Open a private copy of reader to avoid setting nonblocking
# on an unexpecting process with the same reader fd.
diff --git a/scripts/kconfig/.gitignore b/scripts/kconfig/.gitignore
index 500e742..c8a3f9c 100644
--- a/scripts/kconfig/.gitignore
+++ b/scripts/kconfig/.gitignore
@@ -1,5 +1,7 @@
# SPDX-License-Identifier: GPL-2.0-only
/conf
/[gmnq]conf
-/[gmnq]conf-cfg
+/[gmnq]conf-cflags
+/[gmnq]conf-libs
+/qconf-bin
/qconf-moc.cc
diff --git a/scripts/kconfig/Makefile b/scripts/kconfig/Makefile
index b8ef0fb..0b1d15e 100644
--- a/scripts/kconfig/Makefile
+++ b/scripts/kconfig/Makefile
@@ -159,11 +159,12 @@
hostprogs += nconf
nconf-objs := nconf.o nconf.gui.o $(common-objs)
-HOSTLDLIBS_nconf = $(shell . $(obj)/nconf-cfg && echo $$libs)
-HOSTCFLAGS_nconf.o = $(shell . $(obj)/nconf-cfg && echo $$cflags)
-HOSTCFLAGS_nconf.gui.o = $(shell . $(obj)/nconf-cfg && echo $$cflags)
+HOSTLDLIBS_nconf = $(call read-file, $(obj)/nconf-libs)
+HOSTCFLAGS_nconf.o = $(call read-file, $(obj)/nconf-cflags)
+HOSTCFLAGS_nconf.gui.o = $(call read-file, $(obj)/nconf-cflags)
-$(obj)/nconf.o $(obj)/nconf.gui.o: $(obj)/nconf-cfg
+$(obj)/nconf: | $(obj)/nconf-libs
+$(obj)/nconf.o $(obj)/nconf.gui.o: | $(obj)/nconf-cflags
# mconf: Used for the menuconfig target based on lxdialog
hostprogs += mconf
@@ -171,27 +172,28 @@
checklist.o inputbox.o menubox.o textbox.o util.o yesno.o)
mconf-objs := mconf.o $(lxdialog) $(common-objs)
-HOSTLDLIBS_mconf = $(shell . $(obj)/mconf-cfg && echo $$libs)
+HOSTLDLIBS_mconf = $(call read-file, $(obj)/mconf-libs)
$(foreach f, mconf.o $(lxdialog), \
- $(eval HOSTCFLAGS_$f = $$(shell . $(obj)/mconf-cfg && echo $$$$cflags)))
+ $(eval HOSTCFLAGS_$f = $$(call read-file, $(obj)/mconf-cflags)))
-$(addprefix $(obj)/, mconf.o $(lxdialog)): $(obj)/mconf-cfg
+$(obj)/mconf: | $(obj)/mconf-libs
+$(addprefix $(obj)/, mconf.o $(lxdialog)): | $(obj)/mconf-cflags
# qconf: Used for the xconfig target based on Qt
hostprogs += qconf
qconf-cxxobjs := qconf.o qconf-moc.o
qconf-objs := images.o $(common-objs)
-HOSTLDLIBS_qconf = $(shell . $(obj)/qconf-cfg && echo $$libs)
-HOSTCXXFLAGS_qconf.o = $(shell . $(obj)/qconf-cfg && echo $$cflags)
-HOSTCXXFLAGS_qconf-moc.o = $(shell . $(obj)/qconf-cfg && echo $$cflags)
-
-$(obj)/qconf.o: $(obj)/qconf-cfg
+HOSTLDLIBS_qconf = $(call read-file, $(obj)/qconf-libs)
+HOSTCXXFLAGS_qconf.o = -std=c++11 -fPIC $(call read-file, $(obj)/qconf-cflags)
+HOSTCXXFLAGS_qconf-moc.o = -std=c++11 -fPIC $(call read-file, $(obj)/qconf-cflags)
+$(obj)/qconf: | $(obj)/qconf-libs
+$(obj)/qconf.o $(obj)/qconf-moc.o: | $(obj)/qconf-cflags
quiet_cmd_moc = MOC $@
- cmd_moc = $(shell . $(obj)/qconf-cfg && echo $$moc) $< -o $@
+ cmd_moc = $(call read-file, $(obj)/qconf-bin)/moc $< -o $@
-$(obj)/qconf-moc.cc: $(src)/qconf.h $(obj)/qconf-cfg FORCE
+$(obj)/qconf-moc.cc: $(src)/qconf.h FORCE | $(obj)/qconf-bin
$(call if_changed,moc)
targets += qconf-moc.cc
@@ -200,15 +202,16 @@
hostprogs += gconf
gconf-objs := gconf.o images.o $(common-objs)
-HOSTLDLIBS_gconf = $(shell . $(obj)/gconf-cfg && echo $$libs)
-HOSTCFLAGS_gconf.o = $(shell . $(obj)/gconf-cfg && echo $$cflags)
+HOSTLDLIBS_gconf = $(call read-file, $(obj)/gconf-libs)
+HOSTCFLAGS_gconf.o = $(call read-file, $(obj)/gconf-cflags)
-$(obj)/gconf.o: $(obj)/gconf-cfg
+$(obj)/gconf: | $(obj)/gconf-libs
+$(obj)/gconf.o: | $(obj)/gconf-cflags
# check if necessary packages are available, and configure build flags
-filechk_conf_cfg = $(CONFIG_SHELL) $<
+cmd_conf_cfg = $< $(addprefix $(obj)/$*conf-, cflags libs bin)
-$(obj)/%conf-cfg: $(src)/%conf-cfg.sh FORCE
- $(call filechk,conf_cfg)
+$(obj)/%conf-cflags $(obj)/%conf-libs $(obj)/%conf-bin: $(src)/%conf-cfg.sh
+ $(call cmd,conf_cfg)
-clean-files += *conf-cfg
+clean-files += *conf-cflags *conf-libs *conf-bin
diff --git a/scripts/kconfig/gconf-cfg.sh b/scripts/kconfig/gconf-cfg.sh
index cbd90c2..040d8f3 100755
--- a/scripts/kconfig/gconf-cfg.sh
+++ b/scripts/kconfig/gconf-cfg.sh
@@ -1,6 +1,9 @@
#!/bin/sh
# SPDX-License-Identifier: GPL-2.0
+cflags=$1
+libs=$2
+
PKG="gtk+-2.0 gmodule-2.0 libglade-2.0"
if [ -z "$(command -v ${HOSTPKG_CONFIG})" ]; then
@@ -26,5 +29,5 @@
exit 1
fi
-echo cflags=\"$(${HOSTPKG_CONFIG} --cflags $PKG)\"
-echo libs=\"$(${HOSTPKG_CONFIG} --libs $PKG)\"
+${HOSTPKG_CONFIG} --cflags ${PKG} > ${cflags}
+${HOSTPKG_CONFIG} --libs ${PKG} > ${libs}
diff --git a/scripts/kconfig/lkc.h b/scripts/kconfig/lkc.h
index 6ac2eab..e7118d6 100644
--- a/scripts/kconfig/lkc.h
+++ b/scripts/kconfig/lkc.h
@@ -76,7 +76,7 @@ struct gstr str_new(void);
void str_free(struct gstr *gs);
void str_append(struct gstr *gs, const char *s);
void str_printf(struct gstr *gs, const char *fmt, ...);
-const char *str_get(struct gstr *gs);
+char *str_get(struct gstr *gs);
/* menu.c */
void _menu_init(void);
diff --git a/scripts/kconfig/mconf-cfg.sh b/scripts/kconfig/mconf-cfg.sh
index 025b565..1e61f50 100755
--- a/scripts/kconfig/mconf-cfg.sh
+++ b/scripts/kconfig/mconf-cfg.sh
@@ -1,19 +1,22 @@
#!/bin/sh
# SPDX-License-Identifier: GPL-2.0
+cflags=$1
+libs=$2
+
PKG="ncursesw"
PKG2="ncurses"
if [ -n "$(command -v ${HOSTPKG_CONFIG})" ]; then
if ${HOSTPKG_CONFIG} --exists $PKG; then
- echo cflags=\"$(${HOSTPKG_CONFIG} --cflags $PKG)\"
- echo libs=\"$(${HOSTPKG_CONFIG} --libs $PKG)\"
+ ${HOSTPKG_CONFIG} --cflags ${PKG} > ${cflags}
+ ${HOSTPKG_CONFIG} --libs ${PKG} > ${libs}
exit 0
fi
- if ${HOSTPKG_CONFIG} --exists $PKG2; then
- echo cflags=\"$(${HOSTPKG_CONFIG} --cflags $PKG2)\"
- echo libs=\"$(${HOSTPKG_CONFIG} --libs $PKG2)\"
+ if ${HOSTPKG_CONFIG} --exists ${PKG2}; then
+ ${HOSTPKG_CONFIG} --cflags ${PKG2} > ${cflags}
+ ${HOSTPKG_CONFIG} --libs ${PKG2} > ${libs}
exit 0
fi
fi
@@ -22,22 +25,22 @@
# (Even if it is installed, some distributions such as openSUSE cannot
# find ncurses by pkg-config.)
if [ -f /usr/include/ncursesw/ncurses.h ]; then
- echo cflags=\"-D_GNU_SOURCE -I/usr/include/ncursesw\"
- echo libs=\"-lncursesw\"
+ echo -D_GNU_SOURCE -I/usr/include/ncursesw > ${cflags}
+ echo -lncursesw > ${libs}
exit 0
fi
if [ -f /usr/include/ncurses/ncurses.h ]; then
- echo cflags=\"-D_GNU_SOURCE -I/usr/include/ncurses\"
- echo libs=\"-lncurses\"
+ echo -D_GNU_SOURCE -I/usr/include/ncurses > ${cflags}
+ echo -lncurses > ${libs}
exit 0
fi
# As a final fallback before giving up, check if $HOSTCC knows of a default
# ncurses installation (e.g. from a vendor-specific sysroot).
if echo '#include <ncurses.h>' | ${HOSTCC} -E - >/dev/null 2>&1; then
- echo cflags=\"-D_GNU_SOURCE\"
- echo libs=\"-lncurses\"
+ echo -D_GNU_SOURCE > ${cflags}
+ echo -lncurses > ${libs}
exit 0
fi
diff --git a/scripts/kconfig/mconf.c b/scripts/kconfig/mconf.c
index 9d3cf51..9c54968 100644
--- a/scripts/kconfig/mconf.c
+++ b/scripts/kconfig/mconf.c
@@ -440,9 +440,8 @@ static void search_conf(void)
res = get_relations_str(sym_arr, &head);
set_subtitle();
- dres = show_textbox_ext("Search Results", (char *)
- str_get(&res), 0, 0, keys, &vscroll,
- &hscroll, &update_text, (void *)
+ dres = show_textbox_ext("Search Results", str_get(&res), 0, 0,
+ keys, &vscroll, &hscroll, &update_text,
&data);
again = false;
for (i = 0; i < JUMP_NB && keys[i]; i++)
diff --git a/scripts/kconfig/menu.c b/scripts/kconfig/menu.c
index 109325f..b90fff8 100644
--- a/scripts/kconfig/menu.c
+++ b/scripts/kconfig/menu.c
@@ -724,10 +724,8 @@ static void get_prompt_str(struct gstr *r, struct property *prop,
menu = prop->menu;
for (i = 0; menu != &rootmenu && i < 8; menu = menu->parent) {
- bool accessible = menu_is_visible(menu);
-
submenu[i++] = menu;
- if (location == NULL && accessible)
+ if (location == NULL && menu_is_visible(menu))
location = menu;
}
if (head && location) {
diff --git a/scripts/kconfig/nconf-cfg.sh b/scripts/kconfig/nconf-cfg.sh
index 3a10bac2..f871a21 100755
--- a/scripts/kconfig/nconf-cfg.sh
+++ b/scripts/kconfig/nconf-cfg.sh
@@ -1,19 +1,22 @@
#!/bin/sh
# SPDX-License-Identifier: GPL-2.0
+cflags=$1
+libs=$2
+
PKG="ncursesw menuw panelw"
PKG2="ncurses menu panel"
if [ -n "$(command -v ${HOSTPKG_CONFIG})" ]; then
if ${HOSTPKG_CONFIG} --exists $PKG; then
- echo cflags=\"$(${HOSTPKG_CONFIG} --cflags $PKG)\"
- echo libs=\"$(${HOSTPKG_CONFIG} --libs $PKG)\"
+ ${HOSTPKG_CONFIG} --cflags ${PKG} > ${cflags}
+ ${HOSTPKG_CONFIG} --libs ${PKG} > ${libs}
exit 0
fi
if ${HOSTPKG_CONFIG} --exists $PKG2; then
- echo cflags=\"$(${HOSTPKG_CONFIG} --cflags $PKG2)\"
- echo libs=\"$(${HOSTPKG_CONFIG} --libs $PKG2)\"
+ ${HOSTPKG_CONFIG} --cflags ${PKG2} > ${cflags}
+ ${HOSTPKG_CONFIG} --libs ${PKG2} > ${libs}
exit 0
fi
fi
@@ -22,20 +25,20 @@
# (Even if it is installed, some distributions such as openSUSE cannot
# find ncurses by pkg-config.)
if [ -f /usr/include/ncursesw/ncurses.h ]; then
- echo cflags=\"-D_GNU_SOURCE -I/usr/include/ncursesw\"
- echo libs=\"-lncursesw -lmenuw -lpanelw\"
+ echo -D_GNU_SOURCE -I/usr/include/ncursesw > ${cflags}
+ echo -lncursesw -lmenuw -lpanelw > ${libs}
exit 0
fi
if [ -f /usr/include/ncurses/ncurses.h ]; then
- echo cflags=\"-D_GNU_SOURCE -I/usr/include/ncurses\"
- echo libs=\"-lncurses -lmenu -lpanel\"
+ echo -D_GNU_SOURCE -I/usr/include/ncurses > ${cflags}
+ echo -lncurses -lmenu -lpanel > ${libs}
exit 0
fi
if [ -f /usr/include/ncurses.h ]; then
- echo cflags=\"-D_GNU_SOURCE\"
- echo libs=\"-lncurses -lmenu -lpanel\"
+ echo -D_GNU_SOURCE > ${cflags}
+ echo -lncurses -lmenu -lpanel > ${libs}
exit 0
fi
diff --git a/scripts/kconfig/qconf-cfg.sh b/scripts/kconfig/qconf-cfg.sh
index ad652cb..117f36e 100755
--- a/scripts/kconfig/qconf-cfg.sh
+++ b/scripts/kconfig/qconf-cfg.sh
@@ -1,6 +1,10 @@
#!/bin/sh
# SPDX-License-Identifier: GPL-2.0
+cflags=$1
+libs=$2
+bin=$3
+
PKG="Qt5Core Qt5Gui Qt5Widgets"
if [ -z "$(command -v ${HOSTPKG_CONFIG})" ]; then
@@ -11,9 +15,9 @@
fi
if ${HOSTPKG_CONFIG} --exists $PKG; then
- echo cflags=\"-std=c++11 -fPIC $(${HOSTPKG_CONFIG} --cflags $PKG)\"
- echo libs=\"$(${HOSTPKG_CONFIG} --libs $PKG)\"
- echo moc=\"$(${HOSTPKG_CONFIG} --variable=host_bins Qt5Core)/moc\"
+ ${HOSTPKG_CONFIG} --cflags ${PKG} > ${cflags}
+ ${HOSTPKG_CONFIG} --libs ${PKG} > ${libs}
+ ${HOSTPKG_CONFIG} --variable=host_bins Qt5Core > ${bin}
exit 0
fi
diff --git a/scripts/kconfig/util.c b/scripts/kconfig/util.c
index 2958539..b78f114 100644
--- a/scripts/kconfig/util.c
+++ b/scripts/kconfig/util.c
@@ -74,7 +74,7 @@ void str_printf(struct gstr *gs, const char *fmt, ...)
}
/* Retrieve value of growable string */
-const char *str_get(struct gstr *gs)
+char *str_get(struct gstr *gs)
{
return gs->s;
}
diff --git a/scripts/min-tool-version.sh b/scripts/min-tool-version.sh
index 201bccf..a814f1e 100755
--- a/scripts/min-tool-version.sh
+++ b/scripts/min-tool-version.sh
@@ -14,7 +14,7 @@
case "$1" in
binutils)
- echo 2.23.0
+ echo 2.25.0
;;
gcc)
echo 5.1.0
diff --git a/scripts/mod/file2alias.c b/scripts/mod/file2alias.c
index 80d9731..91c2e7b 100644
--- a/scripts/mod/file2alias.c
+++ b/scripts/mod/file2alias.c
@@ -34,19 +34,23 @@ typedef Elf64_Addr kernel_ulong_t;
typedef uint32_t __u32;
typedef uint16_t __u16;
typedef unsigned char __u8;
+
+/* UUID types for backward compatibility, don't use in new code */
typedef struct {
__u8 b[16];
} guid_t;
-/* backwards compatibility, don't use in new code */
-typedef struct {
- __u8 b[16];
-} uuid_le;
typedef struct {
__u8 b[16];
} uuid_t;
+
#define UUID_STRING_LEN 36
+/* MEI UUID type, don't use anywhere else */
+typedef struct {
+ __u8 b[16];
+} uuid_le;
+
/* Big exception to the "don't include kernel headers into userspace, which
* even potentially has different endianness and word sizes, since
* we handle those differences explicitly below */
@@ -140,25 +144,22 @@ static void device_id_check(const char *modname, const char *device_id,
int i;
if (size % id_size || size < id_size) {
- fatal("%s: sizeof(struct %s_device_id)=%lu is not a modulo "
- "of the size of "
- "section __mod_%s__<identifier>_device_table=%lu.\n"
- "Fix definition of struct %s_device_id "
- "in mod_devicetable.h\n",
+ fatal("%s: sizeof(struct %s_device_id)=%lu is not a modulo of the size of section __mod_%s__<identifier>_device_table=%lu.\n"
+ "Fix definition of struct %s_device_id in mod_devicetable.h\n",
modname, device_id, id_size, device_id, size, device_id);
}
/* Verify last one is a terminator */
for (i = 0; i < id_size; i++ ) {
if (*(uint8_t*)(symval+size-id_size+i)) {
- fprintf(stderr,"%s: struct %s_device_id is %lu bytes. "
- "The last of %lu is:\n",
+ fprintf(stderr,
+ "%s: struct %s_device_id is %lu bytes. The last of %lu is:\n",
modname, device_id, id_size, size / id_size);
for (i = 0; i < id_size; i++ )
fprintf(stderr,"0x%02x ",
*(uint8_t*)(symval+size-id_size+i) );
fprintf(stderr,"\n");
- fatal("%s: struct %s_device_id is not terminated "
- "with a NULL entry!\n", modname, device_id);
+ fatal("%s: struct %s_device_id is not terminated with a NULL entry!\n",
+ modname, device_id);
}
}
}
@@ -1154,8 +1155,7 @@ static int do_amba_entry(const char *filename,
DEF_FIELD(symval, amba_id, mask);
if ((id & mask) != id)
- fatal("%s: Masked-off bit(s) of AMBA device ID are non-zero: "
- "id=0x%08X, mask=0x%08X. Please fix this driver.\n",
+ fatal("%s: Masked-off bit(s) of AMBA device ID are non-zero: id=0x%08X, mask=0x%08X. Please fix this driver.\n",
filename, id, mask);
p += sprintf(alias, "amba:d");
diff --git a/scripts/mod/modpost.c b/scripts/mod/modpost.c
index 2c80da0..efff807 100644
--- a/scripts/mod/modpost.c
+++ b/scripts/mod/modpost.c
@@ -519,9 +519,8 @@ static int parse_elf(struct elf_info *info, const char *filename)
int nobits = sechdrs[i].sh_type == SHT_NOBITS;
if (!nobits && sechdrs[i].sh_offset > info->size) {
- fatal("%s is truncated. sechdrs[i].sh_offset=%lu > "
- "sizeof(*hrd)=%zu\n", filename,
- (unsigned long)sechdrs[i].sh_offset,
+ fatal("%s is truncated. sechdrs[i].sh_offset=%lu > sizeof(*hrd)=%zu\n",
+ filename, (unsigned long)sechdrs[i].sh_offset,
sizeof(*hdr));
return 0;
}
@@ -823,10 +822,10 @@ static void check_section(const char *modname, struct elf_info *elf,
#define ALL_EXIT_SECTIONS EXIT_SECTIONS, ALL_XXXEXIT_SECTIONS
#define DATA_SECTIONS ".data", ".data.rel"
-#define TEXT_SECTIONS ".text", ".text.unlikely", ".sched.text", \
+#define TEXT_SECTIONS ".text", ".text.*", ".sched.text", \
".kprobes.text", ".cpuidle.text", ".noinstr.text"
#define OTHER_TEXT_SECTIONS ".ref.text", ".head.text", ".spinlock.text", \
- ".fixup", ".entry.text", ".exception.text", ".text.*", \
+ ".fixup", ".entry.text", ".exception.text", \
".coldtext", ".softirqentry.text"
#define INIT_SECTIONS ".init.*"
@@ -1355,8 +1354,7 @@ static void report_extable_warnings(const char* modname, struct elf_info* elf,
get_pretty_name(is_function(tosym),
&to_pretty_name, &to_pretty_name_p);
- warn("%s(%s+0x%lx): Section mismatch in reference"
- " from the %s %s%s to the %s %s:%s%s\n",
+ warn("%s(%s+0x%lx): Section mismatch in reference from the %s %s%s to the %s %s:%s%s\n",
modname, fromsec, (long)r->r_offset, from_pretty_name,
fromsym_name, from_pretty_name_p,
to_pretty_name, tosec, tosym_name, to_pretty_name_p);
@@ -1523,6 +1521,14 @@ static int addend_mips_rel(struct elf_info *elf, Elf_Shdr *sechdr, Elf_Rela *r)
#define R_RISCV_SUB32 39
#endif
+#ifndef EM_LOONGARCH
+#define EM_LOONGARCH 258
+#endif
+
+#ifndef R_LARCH_SUB32
+#define R_LARCH_SUB32 55
+#endif
+
static void section_rela(const char *modname, struct elf_info *elf,
Elf_Shdr *sechdr)
{
@@ -1564,6 +1570,11 @@ static void section_rela(const char *modname, struct elf_info *elf,
ELF_R_TYPE(r.r_info) == R_RISCV_SUB32)
continue;
break;
+ case EM_LOONGARCH:
+ if (!strcmp("__ex_table", fromsec) &&
+ ELF_R_TYPE(r.r_info) == R_LARCH_SUB32)
+ continue;
+ break;
}
sym = elf->symtab_start + r_sym;
/* Skip special sections */
@@ -1858,11 +1869,9 @@ static void read_symbols_from_files(const char *filename)
FILE *in = stdin;
char fname[PATH_MAX];
- if (strcmp(filename, "-") != 0) {
- in = fopen(filename, "r");
- if (!in)
- fatal("Can't open filenames file %s: %m", filename);
- }
+ in = fopen(filename, "r");
+ if (!in)
+ fatal("Can't open filenames file %s: %m", filename);
while (fgets(fname, PATH_MAX, in) != NULL) {
if (strends(fname, "\n"))
@@ -1870,8 +1879,7 @@ static void read_symbols_from_files(const char *filename)
read_symbols(fname);
}
- if (in != stdin)
- fclose(in);
+ fclose(in);
}
#define SZ 500
diff --git a/scripts/mod/sumversion.c b/scripts/mod/sumversion.c
index 6bf9cac..31066bf 100644
--- a/scripts/mod/sumversion.c
+++ b/scripts/mod/sumversion.c
@@ -153,7 +153,7 @@ static void md4_transform(uint32_t *hash, uint32_t const *in)
static inline void md4_transform_helper(struct md4_ctx *ctx)
{
- le32_to_cpu_array(ctx->block, sizeof(ctx->block) / sizeof(uint32_t));
+ le32_to_cpu_array(ctx->block, ARRAY_SIZE(ctx->block));
md4_transform(ctx->hash, ctx->block);
}
@@ -216,7 +216,7 @@ static void md4_final_ascii(struct md4_ctx *mctx, char *out, unsigned int len)
le32_to_cpu_array(mctx->block, (sizeof(mctx->block) -
sizeof(uint64_t)) / sizeof(uint32_t));
md4_transform(mctx->hash, mctx->block);
- cpu_to_le32_array(mctx->hash, sizeof(mctx->hash) / sizeof(uint32_t));
+ cpu_to_le32_array(mctx->hash, ARRAY_SIZE(mctx->hash));
snprintf(out, len, "%08X%08X%08X%08X",
mctx->hash[0], mctx->hash[1], mctx->hash[2], mctx->hash[3]);
diff --git a/scripts/modules-check.sh b/scripts/modules-check.sh
index e063277..4c8da90 100755
--- a/scripts/modules-check.sh
+++ b/scripts/modules-check.sh
@@ -16,7 +16,7 @@
for m in $(sed 's:.*/::' "$1" | sort | uniq -d)
do
echo "error: the following would cause module name conflict:" >&2
- sed -n "/\/$m/s:^: :p" "$1" >&2
+ sed -n "/\/$m/s:^\(.*\)\.o\$: \1.ko:p" "$1" >&2
exit_code=1
done
}
diff --git a/scripts/package/buildtar b/scripts/package/buildtar
index cb54c7f..4d6f0b1 100755
--- a/scripts/package/buildtar
+++ b/scripts/package/buildtar
@@ -122,7 +122,7 @@
fi
;;
arm64)
- for i in Image.bz2 Image.gz Image.lz4 Image.lzma Image.lzo ; do
+ for i in Image.bz2 Image.gz Image.lz4 Image.lzma Image.lzo vmlinuz.efi ; do
if [ -f "${objtree}/arch/arm64/boot/${i}" ] ; then
cp -v -- "${objtree}/arch/arm64/boot/${i}" "${tmpdir}/boot/vmlinuz-${KERNELRELEASE}"
break
diff --git a/scripts/package/mkdebian b/scripts/package/mkdebian
index a3ac5a7..6cf3832 100755
--- a/scripts/package/mkdebian
+++ b/scripts/package/mkdebian
@@ -175,7 +175,7 @@
Priority: optional
Maintainer: $maintainer
Rules-Requires-Root: no
-Build-Depends: bc, rsync, kmod, cpio, bison, flex | flex:native $extra_build_depends
+Build-Depends: bc, rsync, kmod, cpio, bison, flex $extra_build_depends
Homepage: https://www.kernel.org/
Package: $packagename-$version
diff --git a/scripts/package/mkspec b/scripts/package/mkspec
index 70392fd..dda00a9 100755
--- a/scripts/package/mkspec
+++ b/scripts/package/mkspec
@@ -33,6 +33,8 @@
--exclude=*.o --exclude=*.ko --exclude=*.cmd --exclude=Documentation \
--exclude=.config.old --exclude=.missing-syscalls.d --exclude=*.s"
+test -n "$LOCALVERSION" && MAKE="$MAKE LOCALVERSION=$LOCALVERSION"
+
# We can label the here-doc lines for conditional output to the spec file
#
# Labels:
@@ -49,6 +51,9 @@
URL: https://www.kernel.org
$S Source: kernel-$__KERNELRELEASE.tar.gz
Provides: $PROVIDES
+$S BuildRequires: bc binutils bison dwarves elfutils-libelf-devel flex
+$S BuildRequires: gcc make openssl openssl-devel perl python3 rsync
+
# $UTS_MACHINE as a fallback of _arch in case
# /usr/lib/rpm/platform/*/macros was not included.
%define _arch %{?_arch:$UTS_MACHINE}
@@ -80,6 +85,8 @@
$S$M
$S %prep
$S %setup -q
+$S rm -f scripts/basic/fixdep scripts/kconfig/conf
+$S rm -f tools/objtool/{fixdep,objtool}
$S
$S %build
$S $MAKE %{?_smp_mflags} KBUILD_BUILD_VERSION=%{release}
diff --git a/scripts/recordmcount.c b/scripts/recordmcount.c
index cce12e1..e302165 100644
--- a/scripts/recordmcount.c
+++ b/scripts/recordmcount.c
@@ -38,6 +38,14 @@
#define R_AARCH64_ABS64 257
#endif
+#ifndef EM_LOONGARCH
+#define EM_LOONGARCH 258
+#define R_LARCH_32 1
+#define R_LARCH_64 2
+#define R_LARCH_MARK_LA 20
+#define R_LARCH_SOP_PUSH_PLT_PCREL 29
+#endif
+
#define R_ARM_PC24 1
#define R_ARM_THM_CALL 10
#define R_ARM_CALL 28
@@ -441,6 +449,28 @@ static int arm64_is_fake_mcount(Elf64_Rel const *rp)
return ELF64_R_TYPE(w8(rp->r_info)) != R_AARCH64_CALL26;
}
+static int LARCH32_is_fake_mcount(Elf32_Rel const *rp)
+{
+ switch (ELF64_R_TYPE(w(rp->r_info))) {
+ case R_LARCH_MARK_LA:
+ case R_LARCH_SOP_PUSH_PLT_PCREL:
+ return 0;
+ }
+
+ return 1;
+}
+
+static int LARCH64_is_fake_mcount(Elf64_Rel const *rp)
+{
+ switch (ELF64_R_TYPE(w(rp->r_info))) {
+ case R_LARCH_MARK_LA:
+ case R_LARCH_SOP_PUSH_PLT_PCREL:
+ return 0;
+ }
+
+ return 1;
+}
+
/* 64-bit EM_MIPS has weird ELF64_Rela.r_info.
* http://techpubs.sgi.com/library/manuals/4000/007-4658-001/pdf/007-4658-001.pdf
* We interpret Table 29 Relocation Operation (Elf64_Rel, Elf64_Rela) [p.40]
@@ -558,6 +588,7 @@ static int do_file(char const *const fname)
break;
case EM_IA_64: reltype = R_IA64_IMM64; break;
case EM_MIPS: /* reltype: e_class */ break;
+ case EM_LOONGARCH: /* reltype: e_class */ break;
case EM_PPC: reltype = R_PPC_ADDR32; break;
case EM_PPC64: reltype = R_PPC64_ADDR64; break;
case EM_S390: /* reltype: e_class */ break;
@@ -589,6 +620,10 @@ static int do_file(char const *const fname)
reltype = R_MIPS_32;
is_fake_mcount32 = MIPS32_is_fake_mcount;
}
+ if (w2(ehdr->e_machine) == EM_LOONGARCH) {
+ reltype = R_LARCH_32;
+ is_fake_mcount32 = LARCH32_is_fake_mcount;
+ }
if (do32(ehdr, fname, reltype) < 0)
goto out;
break;
@@ -610,6 +645,10 @@ static int do_file(char const *const fname)
Elf64_r_info = MIPS64_r_info;
is_fake_mcount64 = MIPS64_is_fake_mcount;
}
+ if (w2(ghdr->e_machine) == EM_LOONGARCH) {
+ reltype = R_LARCH_64;
+ is_fake_mcount64 = LARCH64_is_fake_mcount;
+ }
if (do64(ghdr, fname, reltype) < 0)
goto out;
break;
diff --git a/scripts/remove-stale-files b/scripts/remove-stale-files
index ccadfa3..64b14aa 100755
--- a/scripts/remove-stale-files
+++ b/scripts/remove-stale-files
@@ -47,3 +47,5 @@
rm -f scripts/extract-cert
rm -f arch/x86/purgatory/kexec-purgatory.c
+
+rm -f scripts/kconfig/[gmnq]conf-cfg
diff --git a/scripts/sorttable.c b/scripts/sorttable.c
index fba40e9..83cdb84 100644
--- a/scripts/sorttable.c
+++ b/scripts/sorttable.c
@@ -304,6 +304,7 @@ static int do_file(char const *const fname, void *addr)
switch (r2(&ehdr->e_machine)) {
case EM_386:
case EM_AARCH64:
+ case EM_LOONGARCH:
case EM_RISCV:
case EM_S390:
case EM_X86_64:
@@ -317,7 +318,6 @@ static int do_file(char const *const fname, void *addr)
case EM_ARCOMPACT:
case EM_ARCV2:
case EM_ARM:
- case EM_LOONGARCH:
case EM_MICROBLAZE:
case EM_MIPS:
case EM_XTENSA:
diff --git a/sound/soc/sof/intel/hda.c b/sound/soc/sof/intel/hda.c
index 14a2f87..81c697e 100644
--- a/sound/soc/sof/intel/hda.c
+++ b/sound/soc/sof/intel/hda.c
@@ -206,6 +206,7 @@ static int hda_sdw_probe(struct snd_sof_dev *sdev)
memset(&res, 0, sizeof(res));
+ res.hw_ops = &sdw_intel_cnl_hw_ops;
res.mmio_base = sdev->bar[HDA_DSP_BAR];
res.shim_base = hdev->desc->sdw_shim_base;
res.alh_base = hdev->desc->sdw_alh_base;
@@ -1727,3 +1728,4 @@ MODULE_IMPORT_NS(SND_SOC_SOF_HDA_AUDIO_CODEC_I915);
MODULE_IMPORT_NS(SND_SOC_SOF_XTENSA);
MODULE_IMPORT_NS(SND_INTEL_SOUNDWIRE_ACPI);
MODULE_IMPORT_NS(SOUNDWIRE_INTEL_INIT);
+MODULE_IMPORT_NS(SOUNDWIRE_INTEL);
diff --git a/tools/arch/parisc/include/uapi/asm/mman.h b/tools/arch/parisc/include/uapi/asm/mman.h
index 506c06a..4cc88a6 100644
--- a/tools/arch/parisc/include/uapi/asm/mman.h
+++ b/tools/arch/parisc/include/uapi/asm/mman.h
@@ -1,20 +1,20 @@
/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */
#ifndef TOOLS_ARCH_PARISC_UAPI_ASM_MMAN_FIX_H
#define TOOLS_ARCH_PARISC_UAPI_ASM_MMAN_FIX_H
-#define MADV_DODUMP 70
+#define MADV_DODUMP 17
#define MADV_DOFORK 11
-#define MADV_DONTDUMP 69
+#define MADV_DONTDUMP 16
#define MADV_DONTFORK 10
#define MADV_DONTNEED 4
#define MADV_FREE 8
-#define MADV_HUGEPAGE 67
-#define MADV_MERGEABLE 65
-#define MADV_NOHUGEPAGE 68
+#define MADV_HUGEPAGE 14
+#define MADV_MERGEABLE 12
+#define MADV_NOHUGEPAGE 15
#define MADV_NORMAL 0
#define MADV_RANDOM 1
#define MADV_REMOVE 9
#define MADV_SEQUENTIAL 2
-#define MADV_UNMERGEABLE 66
+#define MADV_UNMERGEABLE 13
#define MADV_WILLNEED 3
#define MAP_ANONYMOUS 0x10
#define MAP_DENYWRITE 0x0800
diff --git a/tools/lib/api/Makefile b/tools/lib/api/Makefile
index e21e1b40..044860a 100644
--- a/tools/lib/api/Makefile
+++ b/tools/lib/api/Makefile
@@ -15,6 +15,16 @@
MAKEFLAGS += --no-print-directory
+INSTALL = install
+
+
+# Use DESTDIR for installing into a different root directory.
+# This is useful for building a package. The program will be
+# installed in this directory as if it was the root directory.
+# Then the build tool can move it later.
+DESTDIR ?=
+DESTDIR_SQ = '$(subst ','\'',$(DESTDIR))'
+
LIBFILE = $(OUTPUT)libapi.a
CFLAGS := $(EXTRA_WARNINGS) $(EXTRA_CFLAGS)
@@ -45,10 +55,23 @@
API_IN := $(OUTPUT)libapi-in.o
+ifeq ($(LP64), 1)
+ libdir_relative = lib64
+else
+ libdir_relative = lib
+endif
+
+prefix ?=
+libdir = $(prefix)/$(libdir_relative)
+
+# Shell quotes
+libdir_SQ = $(subst ','\'',$(libdir))
+
all:
export srctree OUTPUT CC LD CFLAGS V
include $(srctree)/tools/build/Makefile.include
+include $(srctree)/tools/scripts/Makefile.include
all: fixdep $(LIBFILE)
@@ -58,6 +81,49 @@
$(LIBFILE): $(API_IN)
$(QUIET_AR)$(RM) $@ && $(AR) rcs $@ $(API_IN)
+define do_install_mkdir
+ if [ ! -d '$(DESTDIR_SQ)$1' ]; then \
+ $(INSTALL) -d -m 755 '$(DESTDIR_SQ)$1'; \
+ fi
+endef
+
+define do_install
+ if [ ! -d '$2' ]; then \
+ $(INSTALL) -d -m 755 '$2'; \
+ fi; \
+ $(INSTALL) $1 $(if $3,-m $3,) '$2'
+endef
+
+install_lib: $(LIBFILE)
+ $(call QUIET_INSTALL, $(LIBFILE)) \
+ $(call do_install_mkdir,$(libdir_SQ)); \
+ cp -fpR $(LIBFILE) $(DESTDIR)$(libdir_SQ)
+
+HDRS := cpu.h debug.h io.h
+FD_HDRS := fd/array.h
+FS_HDRS := fs/fs.h fs/tracing_path.h
+INSTALL_HDRS_PFX := $(DESTDIR)$(prefix)/include/api
+INSTALL_HDRS := $(addprefix $(INSTALL_HDRS_PFX)/, $(HDRS))
+INSTALL_FD_HDRS := $(addprefix $(INSTALL_HDRS_PFX)/, $(FD_HDRS))
+INSTALL_FS_HDRS := $(addprefix $(INSTALL_HDRS_PFX)/, $(FS_HDRS))
+
+$(INSTALL_HDRS): $(INSTALL_HDRS_PFX)/%.h: %.h
+ $(call QUIET_INSTALL, $@) \
+ $(call do_install,$<,$(INSTALL_HDRS_PFX)/,644)
+
+$(INSTALL_FD_HDRS): $(INSTALL_HDRS_PFX)/fd/%.h: fd/%.h
+ $(call QUIET_INSTALL, $@) \
+ $(call do_install,$<,$(INSTALL_HDRS_PFX)/fd/,644)
+
+$(INSTALL_FS_HDRS): $(INSTALL_HDRS_PFX)/fs/%.h: fs/%.h
+ $(call QUIET_INSTALL, $@) \
+ $(call do_install,$<,$(INSTALL_HDRS_PFX)/fs/,644)
+
+install_headers: $(INSTALL_HDRS) $(INSTALL_FD_HDRS) $(INSTALL_FS_HDRS)
+ $(call QUIET_INSTALL, libapi_headers)
+
+install: install_lib install_headers
+
clean:
$(call QUIET_CLEAN, libapi) $(RM) $(LIBFILE); \
find $(or $(OUTPUT),.) -name \*.o -or -name \*.o.cmd -or -name \*.o.d | xargs $(RM)
diff --git a/tools/lib/api/fs/tracing_path.c b/tools/lib/api/fs/tracing_path.c
index 5afb11b..b8e457c 100644
--- a/tools/lib/api/fs/tracing_path.c
+++ b/tools/lib/api/fs/tracing_path.c
@@ -113,6 +113,22 @@ DIR *tracing_events__opendir(void)
return dir;
}
+int tracing_events__scandir_alphasort(struct dirent ***namelist)
+{
+ char *path = get_tracing_file("events");
+ int ret;
+
+ if (!path) {
+ *namelist = NULL;
+ return 0;
+ }
+
+ ret = scandir(path, namelist, NULL, alphasort);
+ put_events_file(path);
+
+ return ret;
+}
+
int tracing_path__strerror_open_tp(int err, char *buf, size_t size,
const char *sys, const char *name)
{
diff --git a/tools/lib/api/fs/tracing_path.h b/tools/lib/api/fs/tracing_path.h
index a19136b..fc6347c 100644
--- a/tools/lib/api/fs/tracing_path.h
+++ b/tools/lib/api/fs/tracing_path.h
@@ -6,6 +6,7 @@
#include <dirent.h>
DIR *tracing_events__opendir(void);
+int tracing_events__scandir_alphasort(struct dirent ***namelist);
void tracing_path_set(const char *mountpoint);
const char *tracing_path_mount(void);
diff --git a/tools/lib/bpf/Makefile b/tools/lib/bpf/Makefile
index 477666f..cf7f02c 100644
--- a/tools/lib/bpf/Makefile
+++ b/tools/lib/bpf/Makefile
@@ -255,6 +255,7 @@
$(call do_install,$<,$(prefix)/include/bpf,644)
install_headers: $(BPF_GENERATED) $(INSTALL_SRC_HDRS) $(INSTALL_GEN_HDRS)
+ $(call QUIET_INSTALL, libbpf_headers)
install_pkgconfig: $(PC_FILE)
$(call QUIET_INSTALL, $(PC_FILE)) \
diff --git a/tools/lib/perf/Makefile b/tools/lib/perf/Makefile
index 21df023..d8cad12 100644
--- a/tools/lib/perf/Makefile
+++ b/tools/lib/perf/Makefile
@@ -176,10 +176,10 @@
endef
define do_install
- if [ ! -d '$(DESTDIR_SQ)$2' ]; then \
- $(INSTALL) -d -m 755 '$(DESTDIR_SQ)$2'; \
- fi; \
- $(INSTALL) $1 $(if $3,-m $3,) '$(DESTDIR_SQ)$2'
+ if [ ! -d '$2' ]; then \
+ $(INSTALL) -d -m 755 '$2'; \
+ fi; \
+ $(INSTALL) $1 $(if $3,-m $3,) '$2'
endef
install_lib: libs
@@ -187,19 +187,28 @@
$(call do_install_mkdir,$(libdir_SQ)); \
cp -fpR $(LIBPERF_ALL) $(DESTDIR)$(libdir_SQ)
-install_headers:
- $(call QUIET_INSTALL, headers) \
- $(call do_install,include/perf/core.h,$(prefix)/include/perf,644); \
- $(call do_install,include/perf/cpumap.h,$(prefix)/include/perf,644); \
- $(call do_install,include/perf/threadmap.h,$(prefix)/include/perf,644); \
- $(call do_install,include/perf/evlist.h,$(prefix)/include/perf,644); \
- $(call do_install,include/perf/evsel.h,$(prefix)/include/perf,644); \
- $(call do_install,include/perf/event.h,$(prefix)/include/perf,644); \
- $(call do_install,include/perf/mmap.h,$(prefix)/include/perf,644);
+HDRS := bpf_perf.h core.h cpumap.h threadmap.h evlist.h evsel.h event.h mmap.h
+INTERNAL_HDRS := cpumap.h evlist.h evsel.h lib.h mmap.h threadmap.h xyarray.h
+
+INSTALL_HDRS_PFX := $(DESTDIR)$(prefix)/include/perf
+INSTALL_HDRS := $(addprefix $(INSTALL_HDRS_PFX)/, $(HDRS))
+INSTALL_INTERNAL_HDRS_PFX := $(DESTDIR)$(prefix)/include/internal
+INSTALL_INTERNAL_HDRS := $(addprefix $(INSTALL_INTERNAL_HDRS_PFX)/, $(INTERNAL_HDRS))
+
+$(INSTALL_HDRS): $(INSTALL_HDRS_PFX)/%.h: include/perf/%.h
+ $(call QUIET_INSTALL, $@) \
+ $(call do_install,$<,$(INSTALL_HDRS_PFX)/,644)
+
+$(INSTALL_INTERNAL_HDRS): $(INSTALL_INTERNAL_HDRS_PFX)/%.h: include/internal/%.h
+ $(call QUIET_INSTALL, $@) \
+ $(call do_install,$<,$(INSTALL_INTERNAL_HDRS_PFX)/,644)
+
+install_headers: $(INSTALL_HDRS) $(INSTALL_INTERNAL_HDRS)
+ $(call QUIET_INSTALL, libperf_headers)
install_pkgconfig: $(LIBPERF_PC)
$(call QUIET_INSTALL, $(LIBPERF_PC)) \
- $(call do_install,$(LIBPERF_PC),$(libdir_SQ)/pkgconfig,644)
+ $(call do_install,$(LIBPERF_PC),$(DESTDIR_SQ)$(libdir_SQ)/pkgconfig,644)
install_doc:
$(Q)$(MAKE) -C Documentation install-man install-html install-examples
diff --git a/tools/lib/perf/include/perf/cpumap.h b/tools/lib/perf/include/perf/cpumap.h
index 03aceb7..3f43f77 100644
--- a/tools/lib/perf/include/perf/cpumap.h
+++ b/tools/lib/perf/include/perf/cpumap.h
@@ -3,7 +3,6 @@
#define __LIBPERF_CPUMAP_H
#include <perf/core.h>
-#include <perf/cpumap.h>
#include <stdio.h>
#include <stdbool.h>
@@ -12,6 +11,8 @@ struct perf_cpu {
int cpu;
};
+struct perf_cpu_map;
+
LIBPERF_API struct perf_cpu_map *perf_cpu_map__dummy_new(void);
LIBPERF_API struct perf_cpu_map *perf_cpu_map__default_new(void);
LIBPERF_API struct perf_cpu_map *perf_cpu_map__new(const char *cpu_list);
diff --git a/tools/lib/subcmd/Makefile b/tools/lib/subcmd/Makefile
index 8f1a09c..b872132 100644
--- a/tools/lib/subcmd/Makefile
+++ b/tools/lib/subcmd/Makefile
@@ -17,6 +17,15 @@
MAKEFLAGS += --no-print-directory
+INSTALL = install
+
+# Use DESTDIR for installing into a different root directory.
+# This is useful for building a package. The program will be
+# installed in this directory as if it was the root directory.
+# Then the build tool can move it later.
+DESTDIR ?=
+DESTDIR_SQ = '$(subst ','\'',$(DESTDIR))'
+
LIBFILE = $(OUTPUT)libsubcmd.a
CFLAGS := -ggdb3 -Wall -Wextra -std=gnu99 -fPIC
@@ -48,6 +57,18 @@
SUBCMD_IN := $(OUTPUT)libsubcmd-in.o
+ifeq ($(LP64), 1)
+ libdir_relative = lib64
+else
+ libdir_relative = lib
+endif
+
+prefix ?=
+libdir = $(prefix)/$(libdir_relative)
+
+# Shell quotes
+libdir_SQ = $(subst ','\'',$(libdir))
+
all:
export srctree OUTPUT CC LD CFLAGS V
@@ -61,6 +82,37 @@
$(LIBFILE): $(SUBCMD_IN)
$(QUIET_AR)$(RM) $@ && $(AR) rcs $@ $(SUBCMD_IN)
+define do_install_mkdir
+ if [ ! -d '$(DESTDIR_SQ)$1' ]; then \
+ $(INSTALL) -d -m 755 '$(DESTDIR_SQ)$1'; \
+ fi
+endef
+
+define do_install
+ if [ ! -d '$2' ]; then \
+ $(INSTALL) -d -m 755 '$2'; \
+ fi; \
+ $(INSTALL) $1 $(if $3,-m $3,) '$2'
+endef
+
+install_lib: $(LIBFILE)
+ $(call QUIET_INSTALL, $(LIBFILE)) \
+ $(call do_install_mkdir,$(libdir_SQ)); \
+ cp -fpR $(LIBFILE) $(DESTDIR)$(libdir_SQ)
+
+HDRS := exec-cmd.h help.h pager.h parse-options.h run-command.h
+INSTALL_HDRS_PFX := $(DESTDIR)$(prefix)/include/subcmd
+INSTALL_HDRS := $(addprefix $(INSTALL_HDRS_PFX)/, $(HDRS))
+
+$(INSTALL_HDRS): $(INSTALL_HDRS_PFX)/%.h: %.h
+ $(call QUIET_INSTALL, $@) \
+ $(call do_install,$<,$(INSTALL_HDRS_PFX)/,644)
+
+install_headers: $(INSTALL_HDRS)
+ $(call QUIET_INSTALL, libsubcmd_headers)
+
+install: install_lib install_headers
+
clean:
$(call QUIET_CLEAN, libsubcmd) $(RM) $(LIBFILE); \
find $(or $(OUTPUT),.) -name \*.o -or -name \*.o.cmd -or -name \*.o.d | xargs $(RM)
diff --git a/tools/lib/symbol/Build b/tools/lib/symbol/Build
new file mode 100644
index 0000000..9b9a9c7
--- /dev/null
+++ b/tools/lib/symbol/Build
@@ -0,0 +1 @@
+libsymbol-y += kallsyms.o
diff --git a/tools/lib/symbol/Makefile b/tools/lib/symbol/Makefile
new file mode 100644
index 0000000..13d43c6
--- /dev/null
+++ b/tools/lib/symbol/Makefile
@@ -0,0 +1,122 @@
+# SPDX-License-Identifier: GPL-2.0
+include ../../scripts/Makefile.include
+include ../../scripts/utilities.mak # QUIET_CLEAN
+
+ifeq ($(srctree),)
+srctree := $(patsubst %/,%,$(dir $(CURDIR)))
+srctree := $(patsubst %/,%,$(dir $(srctree)))
+srctree := $(patsubst %/,%,$(dir $(srctree)))
+#$(info Determined 'srctree' to be $(srctree))
+endif
+
+CC ?= $(CROSS_COMPILE)gcc
+AR ?= $(CROSS_COMPILE)ar
+LD ?= $(CROSS_COMPILE)ld
+
+MAKEFLAGS += --no-print-directory
+
+INSTALL = install
+
+
+# Use DESTDIR for installing into a different root directory.
+# This is useful for building a package. The program will be
+# installed in this directory as if it was the root directory.
+# Then the build tool can move it later.
+DESTDIR ?=
+DESTDIR_SQ = '$(subst ','\'',$(DESTDIR))'
+
+LIBFILE = $(OUTPUT)libsymbol.a
+
+CFLAGS := $(EXTRA_WARNINGS) $(EXTRA_CFLAGS)
+CFLAGS += -ggdb3 -Wall -Wextra -std=gnu11 -U_FORTIFY_SOURCE -fPIC
+
+ifeq ($(DEBUG),0)
+ifeq ($(CC_NO_CLANG), 0)
+ CFLAGS += -O3
+else
+ CFLAGS += -O6
+endif
+endif
+
+ifeq ($(DEBUG),0)
+ CFLAGS += -D_FORTIFY_SOURCE
+endif
+
+# Treat warnings as errors unless directed not to
+ifneq ($(WERROR),0)
+ CFLAGS += -Werror
+endif
+
+CFLAGS += -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64
+
+CFLAGS += -I$(srctree)/tools/lib
+CFLAGS += -I$(srctree)/tools/include
+
+RM = rm -f
+
+SYMBOL_IN := $(OUTPUT)libsymbol-in.o
+
+ifeq ($(LP64), 1)
+ libdir_relative = lib64
+else
+ libdir_relative = lib
+endif
+
+prefix ?=
+libdir = $(prefix)/$(libdir_relative)
+
+# Shell quotes
+libdir_SQ = $(subst ','\'',$(libdir))
+
+all:
+
+export srctree OUTPUT CC LD CFLAGS V
+include $(srctree)/tools/build/Makefile.include
+include $(srctree)/tools/scripts/Makefile.include
+
+all: fixdep $(LIBFILE)
+
+$(SYMBOL_IN): FORCE
+ @$(MAKE) $(build)=libsymbol
+
+$(LIBFILE): $(SYMBOL_IN)
+ $(QUIET_AR)$(RM) $@ && $(AR) rcs $@ $(SYMBOL_IN)
+
+define do_install_mkdir
+ if [ ! -d '$(DESTDIR_SQ)$1' ]; then \
+ $(INSTALL) -d -m 755 '$(DESTDIR_SQ)$1'; \
+ fi
+endef
+
+define do_install
+ if [ ! -d '$2' ]; then \
+ $(INSTALL) -d -m 755 '$2'; \
+ fi; \
+ $(INSTALL) $1 $(if $3,-m $3,) '$2'
+endef
+
+install_lib: $(LIBFILE)
+ $(call QUIET_INSTALL, $(LIBFILE)) \
+ $(call do_install_mkdir,$(libdir_SQ)); \
+ cp -fpR $(LIBFILE) $(DESTDIR)$(libdir_SQ)
+
+HDRS := kallsyms.h
+INSTALL_HDRS_PFX := $(DESTDIR)$(prefix)/include/symbol
+INSTALL_HDRS := $(addprefix $(INSTALL_HDRS_PFX)/, $(HDRS))
+
+$(INSTALL_HDRS): $(INSTALL_HDRS_PFX)/%.h: %.h
+ $(call QUIET_INSTALL, $@) \
+ $(call do_install,$<,$(INSTALL_HDRS_PFX)/,644)
+
+install_headers: $(INSTALL_HDRS)
+ $(call QUIET_INSTALL, libsymbol_headers)
+
+install: install_lib install_headers
+
+clean:
+ $(call QUIET_CLEAN, libsymbol) $(RM) $(LIBFILE); \
+ find $(or $(OUTPUT),.) -name \*.o -or -name \*.o.cmd -or -name \*.o.d | xargs $(RM)
+
+FORCE:
+
+.PHONY: clean FORCE
diff --git a/tools/lib/traceevent/.gitignore b/tools/lib/traceevent/.gitignore
deleted file mode 100644
index 7123c70..0000000
--- a/tools/lib/traceevent/.gitignore
+++ /dev/null
@@ -1,4 +0,0 @@
-# SPDX-License-Identifier: GPL-2.0-only
-TRACEEVENT-CFLAGS
-libtraceevent-dynamic-list
-libtraceevent.so.*
diff --git a/tools/lib/traceevent/Build b/tools/lib/traceevent/Build
deleted file mode 100644
index f9a5d79..0000000
--- a/tools/lib/traceevent/Build
+++ /dev/null
@@ -1,8 +0,0 @@
-libtraceevent-y += event-parse.o
-libtraceevent-y += event-plugin.o
-libtraceevent-y += trace-seq.o
-libtraceevent-y += parse-filter.o
-libtraceevent-y += parse-utils.o
-libtraceevent-y += kbuffer-parse.o
-libtraceevent-y += tep_strerror.o
-libtraceevent-y += event-parse-api.o
diff --git a/tools/lib/traceevent/Documentation/Makefile b/tools/lib/traceevent/Documentation/Makefile
deleted file mode 100644
index aa72ab9..0000000
--- a/tools/lib/traceevent/Documentation/Makefile
+++ /dev/null
@@ -1,207 +0,0 @@
-include ../../../scripts/Makefile.include
-include ../../../scripts/utilities.mak
-
-# This Makefile and manpage XSL files were taken from tools/perf/Documentation
-# and modified for libtraceevent.
-
-MAN3_TXT= \
- $(wildcard libtraceevent-*.txt) \
- libtraceevent.txt
-
-MAN_TXT = $(MAN3_TXT)
-_MAN_XML=$(patsubst %.txt,%.xml,$(MAN_TXT))
-_MAN_HTML=$(patsubst %.txt,%.html,$(MAN_TXT))
-_DOC_MAN3=$(patsubst %.txt,%.3,$(MAN3_TXT))
-
-MAN_XML=$(addprefix $(OUTPUT),$(_MAN_XML))
-MAN_HTML=$(addprefix $(OUTPUT),$(_MAN_HTML))
-DOC_MAN3=$(addprefix $(OUTPUT),$(_DOC_MAN3))
-
-# Make the path relative to DESTDIR, not prefix
-ifndef DESTDIR
-prefix?=$(HOME)
-endif
-bindir?=$(prefix)/bin
-htmldir?=$(prefix)/share/doc/libtraceevent-doc
-pdfdir?=$(prefix)/share/doc/libtraceevent-doc
-mandir?=$(prefix)/share/man
-man3dir=$(mandir)/man3
-
-ASCIIDOC=asciidoc
-ASCIIDOC_EXTRA = --unsafe -f asciidoc.conf
-ASCIIDOC_HTML = xhtml11
-MANPAGE_XSL = manpage-normal.xsl
-XMLTO_EXTRA =
-INSTALL?=install
-RM ?= rm -f
-
-ifdef USE_ASCIIDOCTOR
-ASCIIDOC = asciidoctor
-ASCIIDOC_EXTRA = -a compat-mode
-ASCIIDOC_EXTRA += -I. -rasciidoctor-extensions
-ASCIIDOC_EXTRA += -a mansource="libtraceevent" -a manmanual="libtraceevent Manual"
-ASCIIDOC_HTML = xhtml5
-endif
-
-XMLTO=xmlto
-
-_tmp_tool_path := $(call get-executable,$(ASCIIDOC))
-ifeq ($(_tmp_tool_path),)
- missing_tools = $(ASCIIDOC)
-endif
-
-ifndef USE_ASCIIDOCTOR
-_tmp_tool_path := $(call get-executable,$(XMLTO))
-ifeq ($(_tmp_tool_path),)
- missing_tools += $(XMLTO)
-endif
-endif
-
-#
-# For asciidoc ...
-# -7.1.2, no extra settings are needed.
-# 8.0-, set ASCIIDOC8.
-#
-
-#
-# For docbook-xsl ...
-# -1.68.1, set ASCIIDOC_NO_ROFF? (based on changelog from 1.73.0)
-# 1.69.0, no extra settings are needed?
-# 1.69.1-1.71.0, set DOCBOOK_SUPPRESS_SP?
-# 1.71.1, no extra settings are needed?
-# 1.72.0, set DOCBOOK_XSL_172.
-# 1.73.0-, set ASCIIDOC_NO_ROFF
-#
-
-#
-# If you had been using DOCBOOK_XSL_172 in an attempt to get rid
-# of 'the ".ft C" problem' in your generated manpages, and you
-# instead ended up with weird characters around callouts, try
-# using ASCIIDOC_NO_ROFF instead (it works fine with ASCIIDOC8).
-#
-
-ifdef ASCIIDOC8
-ASCIIDOC_EXTRA += -a asciidoc7compatible
-endif
-ifdef DOCBOOK_XSL_172
-ASCIIDOC_EXTRA += -a libtraceevent-asciidoc-no-roff
-MANPAGE_XSL = manpage-1.72.xsl
-else
- ifdef ASCIIDOC_NO_ROFF
- # docbook-xsl after 1.72 needs the regular XSL, but will not
- # pass-thru raw roff codes from asciidoc.conf, so turn them off.
- ASCIIDOC_EXTRA += -a libtraceevent-asciidoc-no-roff
- endif
-endif
-ifdef MAN_BOLD_LITERAL
-XMLTO_EXTRA += -m manpage-bold-literal.xsl
-endif
-ifdef DOCBOOK_SUPPRESS_SP
-XMLTO_EXTRA += -m manpage-suppress-sp.xsl
-endif
-
-SHELL_PATH ?= $(SHELL)
-# Shell quote;
-SHELL_PATH_SQ = $(subst ','\'',$(SHELL_PATH))
-
-DESTDIR ?=
-DESTDIR_SQ = '$(subst ','\'',$(DESTDIR))'
-
-export DESTDIR DESTDIR_SQ
-
-#
-# Please note that there is a minor bug in asciidoc.
-# The version after 6.0.3 _will_ include the patch found here:
-# http://marc.theaimsgroup.com/?l=libtraceevent&m=111558757202243&w=2
-#
-# Until that version is released you may have to apply the patch
-# yourself - yes, all 6 characters of it!
-#
-QUIET_SUBDIR0 = +$(MAKE) -C # space to separate -C and subdir
-QUIET_SUBDIR1 =
-
-ifneq ($(findstring $(MAKEFLAGS),w),w)
-PRINT_DIR = --no-print-directory
-else # "make -w"
-NO_SUBDIR = :
-endif
-
-ifneq ($(findstring $(MAKEFLAGS),s),s)
-ifneq ($(V),1)
- QUIET_ASCIIDOC = @echo ' ASCIIDOC '$@;
- QUIET_XMLTO = @echo ' XMLTO '$@;
- QUIET_SUBDIR0 = +@subdir=
- QUIET_SUBDIR1 = ;$(NO_SUBDIR) \
- echo ' SUBDIR ' $$subdir; \
- $(MAKE) $(PRINT_DIR) -C $$subdir
- export V
-endif
-endif
-
-all: html man
-
-man: man3
-man3: $(DOC_MAN3)
-
-html: $(MAN_HTML)
-
-$(MAN_HTML) $(DOC_MAN3): asciidoc.conf
-
-install: install-man
-
-check-man-tools:
-ifdef missing_tools
- $(error "You need to install $(missing_tools) for man pages")
-endif
-
-do-install-man: man
- $(call QUIET_INSTALL, Documentation-man) \
- $(INSTALL) -d -m 755 $(DESTDIR)$(man3dir); \
- $(INSTALL) -m 644 $(DOC_MAN3) $(DESTDIR)$(man3dir);
-
-install-man: check-man-tools man do-install-man
-
-uninstall: uninstall-man
-
-uninstall-man:
- $(call QUIET_UNINST, Documentation-man) \
- $(Q)$(RM) $(addprefix $(DESTDIR)$(man3dir)/,$(DOC_MAN3))
-
-
-ifdef missing_tools
- DO_INSTALL_MAN = $(warning Please install $(missing_tools) to have the man pages installed)
-else
- DO_INSTALL_MAN = do-install-man
-endif
-
-CLEAN_FILES = \
- $(MAN_XML) $(addsuffix +,$(MAN_XML)) \
- $(MAN_HTML) $(addsuffix +,$(MAN_HTML)) \
- $(DOC_MAN3) *.3
-
-clean:
- $(call QUIET_CLEAN, Documentation) $(RM) $(CLEAN_FILES)
-
-ifdef USE_ASCIIDOCTOR
-$(OUTPUT)%.3 : $(OUTPUT)%.txt
- $(QUIET_ASCIIDOC)$(RM) $@+ $@ && \
- $(ASCIIDOC) -b manpage -d manpage \
- $(ASCIIDOC_EXTRA) -alibtraceevent_version=$(EVENT_PARSE_VERSION) -o $@+ $< && \
- mv $@+ $@
-endif
-
-$(OUTPUT)%.3 : $(OUTPUT)%.xml
- $(QUIET_XMLTO)$(RM) $@ && \
- $(XMLTO) -o $(OUTPUT). -m $(MANPAGE_XSL) $(XMLTO_EXTRA) man $<
-
-$(OUTPUT)%.xml : %.txt
- $(QUIET_ASCIIDOC)$(RM) $@+ $@ && \
- $(ASCIIDOC) -b docbook -d manpage \
- $(ASCIIDOC_EXTRA) -alibtraceevent_version=$(EVENT_PARSE_VERSION) -o $@+ $< && \
- mv $@+ $@
-
-$(MAN_HTML): $(OUTPUT)%.html : %.txt
- $(QUIET_ASCIIDOC)$(RM) $@+ $@ && \
- $(ASCIIDOC) -b $(ASCIIDOC_HTML) -d manpage \
- $(ASCIIDOC_EXTRA) -aperf_version=$(EVENT_PARSE_VERSION) -o $@+ $< && \
- mv $@+ $@
diff --git a/tools/lib/traceevent/Documentation/asciidoc.conf b/tools/lib/traceevent/Documentation/asciidoc.conf
deleted file mode 100644
index 0759571..0000000
--- a/tools/lib/traceevent/Documentation/asciidoc.conf
+++ /dev/null
@@ -1,120 +0,0 @@
-## linktep: macro
-#
-# Usage: linktep:command[manpage-section]
-#
-# Note, {0} is the manpage section, while {target} is the command.
-#
-# Show TEP link as: <command>(<section>); if section is defined, else just show
-# the command.
-
-[macros]
-(?su)[\\]?(?P<name>linktep):(?P<target>\S*?)\[(?P<attrlist>.*?)\]=
-
-[attributes]
-asterisk=*
-plus=+
-caret=^
-startsb=[
-endsb=]
-tilde=~
-
-ifdef::backend-docbook[]
-[linktep-inlinemacro]
-{0%{target}}
-{0#<citerefentry>}
-{0#<refentrytitle>{target}</refentrytitle><manvolnum>{0}</manvolnum>}
-{0#</citerefentry>}
-endif::backend-docbook[]
-
-ifdef::backend-docbook[]
-ifndef::tep-asciidoc-no-roff[]
-# "unbreak" docbook-xsl v1.68 for manpages. v1.69 works with or without this.
-# v1.72 breaks with this because it replaces dots not in roff requests.
-[listingblock]
-<example><title>{title}</title>
-<literallayout>
-ifdef::doctype-manpage[]
- .ft C
-endif::doctype-manpage[]
-|
-ifdef::doctype-manpage[]
- .ft
-endif::doctype-manpage[]
-</literallayout>
-{title#}</example>
-endif::tep-asciidoc-no-roff[]
-
-ifdef::tep-asciidoc-no-roff[]
-ifdef::doctype-manpage[]
-# The following two small workarounds insert a simple paragraph after screen
-[listingblock]
-<example><title>{title}</title>
-<literallayout>
-|
-</literallayout><simpara></simpara>
-{title#}</example>
-
-[verseblock]
-<formalpara{id? id="{id}"}><title>{title}</title><para>
-{title%}<literallayout{id? id="{id}"}>
-{title#}<literallayout>
-|
-</literallayout>
-{title#}</para></formalpara>
-{title%}<simpara></simpara>
-endif::doctype-manpage[]
-endif::tep-asciidoc-no-roff[]
-endif::backend-docbook[]
-
-ifdef::doctype-manpage[]
-ifdef::backend-docbook[]
-[header]
-template::[header-declarations]
-<refentry>
-<refmeta>
-<refentrytitle>{mantitle}</refentrytitle>
-<manvolnum>{manvolnum}</manvolnum>
-<refmiscinfo class="source">libtraceevent</refmiscinfo>
-<refmiscinfo class="version">{libtraceevent_version}</refmiscinfo>
-<refmiscinfo class="manual">libtraceevent Manual</refmiscinfo>
-</refmeta>
-<refnamediv>
- <refname>{manname1}</refname>
- <refname>{manname2}</refname>
- <refname>{manname3}</refname>
- <refname>{manname4}</refname>
- <refname>{manname5}</refname>
- <refname>{manname6}</refname>
- <refname>{manname7}</refname>
- <refname>{manname8}</refname>
- <refname>{manname9}</refname>
- <refname>{manname10}</refname>
- <refname>{manname11}</refname>
- <refname>{manname12}</refname>
- <refname>{manname13}</refname>
- <refname>{manname14}</refname>
- <refname>{manname15}</refname>
- <refname>{manname16}</refname>
- <refname>{manname17}</refname>
- <refname>{manname18}</refname>
- <refname>{manname19}</refname>
- <refname>{manname20}</refname>
- <refname>{manname21}</refname>
- <refname>{manname22}</refname>
- <refname>{manname23}</refname>
- <refname>{manname24}</refname>
- <refname>{manname25}</refname>
- <refname>{manname26}</refname>
- <refname>{manname27}</refname>
- <refname>{manname28}</refname>
- <refname>{manname29}</refname>
- <refname>{manname30}</refname>
- <refpurpose>{manpurpose}</refpurpose>
-</refnamediv>
-endif::backend-docbook[]
-endif::doctype-manpage[]
-
-ifdef::backend-xhtml11[]
-[linktep-inlinemacro]
-<a href="{target}.html">{target}{0?({0})}</a>
-endif::backend-xhtml11[]
diff --git a/tools/lib/traceevent/Documentation/libtraceevent-commands.txt b/tools/lib/traceevent/Documentation/libtraceevent-commands.txt
deleted file mode 100644
index bec5520..0000000
--- a/tools/lib/traceevent/Documentation/libtraceevent-commands.txt
+++ /dev/null
@@ -1,153 +0,0 @@
-libtraceevent(3)
-================
-
-NAME
-----
-tep_register_comm, tep_override_comm, tep_pid_is_registered,
-tep_data_comm_from_pid, tep_data_pid_from_comm, tep_cmdline_pid -
-Manage pid to process name mappings.
-
-SYNOPSIS
---------
-[verse]
---
-*#include <event-parse.h>*
-
-int *tep_register_comm*(struct tep_handle pass:[*]_tep_, const char pass:[*]_comm_, int _pid_);
-int *tep_override_comm*(struct tep_handle pass:[*]_tep_, const char pass:[*]_comm_, int _pid_);
-bool *tep_is_pid_registered*(struct tep_handle pass:[*]_tep_, int _pid_);
-const char pass:[*]*tep_data_comm_from_pid*(struct tep_handle pass:[*]_pevent_, int _pid_);
-struct cmdline pass:[*]*tep_data_pid_from_comm*(struct tep_handle pass:[*]_pevent_, const char pass:[*]_comm_, struct cmdline pass:[*]_next_);
-int *tep_cmdline_pid*(struct tep_handle pass:[*]_pevent_, struct cmdline pass:[*]_cmdline_);
---
-
-DESCRIPTION
------------
-These functions can be used to handle the mapping between pid and process name.
-The library builds a cache of these mappings, which is used to display the name
-of the process, instead of its pid. This information can be retrieved from
-tracefs/saved_cmdlines file.
-
-The _tep_register_comm()_ function registers a _pid_ / process name mapping.
-If a command with the same _pid_ is already registered, an error is returned.
-The _pid_ argument is the process ID, the _comm_ argument is the process name,
-_tep_ is the event context. The _comm_ is duplicated internally.
-
-The _tep_override_comm()_ function registers a _pid_ / process name mapping.
-If a process with the same pid is already registered, the process name string is
-udapted with the new one. The _pid_ argument is the process ID, the _comm_
-argument is the process name, _tep_ is the event context. The _comm_ is
-duplicated internally.
-
-The _tep_is_pid_registered()_ function checks if a pid has a process name
-mapping registered. The _pid_ argument is the process ID, _tep_ is the event
-context.
-
-The _tep_data_comm_from_pid()_ function returns the process name for a given
-pid. The _pid_ argument is the process ID, _tep_ is the event context.
-The returned string should not be freed, but will be freed when the _tep_
-handler is closed.
-
-The _tep_data_pid_from_comm()_ function returns a pid for a given process name.
-The _comm_ argument is the process name, _tep_ is the event context.
-The argument _next_ is the cmdline structure to search for the next pid.
-As there may be more than one pid for a given process, the result of this call
-can be passed back into a recurring call in the _next_ parameter, to search for
-the next pid. If _next_ is NULL, it will return the first pid associated with
-the _comm_. The function performs a linear search, so it may be slow.
-
-The _tep_cmdline_pid()_ function returns the pid associated with a given
-_cmdline_. The _tep_ argument is the event context.
-
-RETURN VALUE
-------------
-_tep_register_comm()_ function returns 0 on success. In case of an error -1 is
-returned and errno is set to indicate the cause of the problem: ENOMEM, if there
-is not enough memory to duplicate the _comm_ or EEXIST if a mapping for this
-_pid_ is already registered.
-
-_tep_override_comm()_ function returns 0 on success. In case of an error -1 is
-returned and errno is set to indicate the cause of the problem: ENOMEM, if there
-is not enough memory to duplicate the _comm_.
-
-_tep_is_pid_registered()_ function returns true if the _pid_ has a process name
-mapped to it, false otherwise.
-
-_tep_data_comm_from_pid()_ function returns the process name as string, or the
-string "<...>" if there is no mapping for the given pid.
-
-_tep_data_pid_from_comm()_ function returns a pointer to a struct cmdline, that
-holds a pid for a given process, or NULL if none is found. This result can be
-passed back into a recurring call as the _next_ parameter of the function.
-
-_tep_cmdline_pid()_ functions returns the pid for the give cmdline. If _cmdline_
- is NULL, then -1 is returned.
-
-EXAMPLE
--------
-The following example registers pid for command "ls", in context of event _tep_
-and performs various searches for pid / process name mappings:
-[source,c]
---
-#include <event-parse.h>
-...
-int ret;
-int ls_pid = 1021;
-struct tep_handle *tep = tep_alloc();
-...
- ret = tep_register_comm(tep, "ls", ls_pid);
- if (ret != 0 && errno == EEXIST)
- ret = tep_override_comm(tep, "ls", ls_pid);
- if (ret != 0) {
- /* Failed to register pid / command mapping */
- }
-...
- if (tep_is_pid_registered(tep, ls_pid) == 0) {
- /* Command mapping for ls_pid is not registered */
- }
-...
- const char *comm = tep_data_comm_from_pid(tep, ls_pid);
- if (comm) {
- /* Found process name for ls_pid */
- }
-...
- int pid;
- struct cmdline *cmd = tep_data_pid_from_comm(tep, "ls", NULL);
- while (cmd) {
- pid = tep_cmdline_pid(tep, cmd);
- /* Found pid for process "ls" */
- cmd = tep_data_pid_from_comm(tep, "ls", cmd);
- }
---
-FILES
------
-[verse]
---
-*event-parse.h*
- Header file to include in order to have access to the library APIs.
-*-ltraceevent*
- Linker switch to add when building a program that uses the library.
---
-
-SEE ALSO
---------
-_libtraceevent(3)_, _trace-cmd(1)_
-
-AUTHOR
-------
-[verse]
---
-*Steven Rostedt* <rostedt@goodmis.org>, author of *libtraceevent*.
-*Tzvetomir Stoyanov* <tz.stoyanov@gmail.com>, author of this man page.
---
-REPORTING BUGS
---------------
-Report bugs to <linux-trace-devel@vger.kernel.org>
-
-LICENSE
--------
-libtraceevent is Free Software licensed under the GNU LGPL 2.1
-
-RESOURCES
----------
-https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
diff --git a/tools/lib/traceevent/Documentation/libtraceevent-cpus.txt b/tools/lib/traceevent/Documentation/libtraceevent-cpus.txt
deleted file mode 100644
index 5ad70e4..0000000
--- a/tools/lib/traceevent/Documentation/libtraceevent-cpus.txt
+++ /dev/null
@@ -1,77 +0,0 @@
-libtraceevent(3)
-================
-
-NAME
-----
-tep_get_cpus, tep_set_cpus - Get / set the number of CPUs, which have a tracing
-buffer representing it. Note, the buffer may be empty.
-
-SYNOPSIS
---------
-[verse]
---
-*#include <event-parse.h>*
-
-int *tep_get_cpus*(struct tep_handle pass:[*]_tep_);
-void *tep_set_cpus*(struct tep_handle pass:[*]_tep_, int _cpus_);
---
-
-DESCRIPTION
------------
-The _tep_get_cpus()_ function gets the number of CPUs, which have a tracing
-buffer representing it. The _tep_ argument is trace event parser context.
-
-The _tep_set_cpus()_ function sets the number of CPUs, which have a tracing
-buffer representing it. The _tep_ argument is trace event parser context.
-The _cpu_ argument is the number of CPUs with tracing data.
-
-RETURN VALUE
-------------
-The _tep_get_cpus()_ functions returns the number of CPUs, which have tracing
-data recorded.
-
-EXAMPLE
--------
-[source,c]
---
-#include <event-parse.h>
-...
-struct tep_handle *tep = tep_alloc();
-...
- tep_set_cpus(tep, 5);
-...
- printf("We have tracing data for %d CPUs", tep_get_cpus(tep));
---
-
-FILES
------
-[verse]
---
-*event-parse.h*
- Header file to include in order to have access to the library APIs.
-*-ltraceevent*
- Linker switch to add when building a program that uses the library.
---
-
-SEE ALSO
---------
-_libtraceevent(3)_, _trace-cmd(1)_
-
-AUTHOR
-------
-[verse]
---
-*Steven Rostedt* <rostedt@goodmis.org>, author of *libtraceevent*.
-*Tzvetomir Stoyanov* <tz.stoyanov@gmail.com>, author of this man page.
---
-REPORTING BUGS
---------------
-Report bugs to <linux-trace-devel@vger.kernel.org>
-
-LICENSE
--------
-libtraceevent is Free Software licensed under the GNU LGPL 2.1
-
-RESOURCES
----------
-https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
diff --git a/tools/lib/traceevent/Documentation/libtraceevent-endian_read.txt b/tools/lib/traceevent/Documentation/libtraceevent-endian_read.txt
deleted file mode 100644
index e64851b..0000000
--- a/tools/lib/traceevent/Documentation/libtraceevent-endian_read.txt
+++ /dev/null
@@ -1,78 +0,0 @@
-libtraceevent(3)
-================
-
-NAME
-----
-tep_read_number - Reads a number from raw data.
-
-SYNOPSIS
---------
-[verse]
---
-*#include <event-parse.h>*
-
-unsigned long long *tep_read_number*(struct tep_handle pass:[*]_tep_, const void pass:[*]_ptr_, int _size_);
---
-
-DESCRIPTION
------------
-The _tep_read_number()_ function reads an integer from raw data, taking into
-account the endianness of the raw data and the current host. The _tep_ argument
-is the trace event parser context. The _ptr_ is a pointer to the raw data, where
-the integer is, and the _size_ is the size of the integer.
-
-RETURN VALUE
-------------
-The _tep_read_number()_ function returns the integer in the byte order of
-the current host. In case of an error, 0 is returned.
-
-EXAMPLE
--------
-[source,c]
---
-#include <event-parse.h>
-...
-struct tep_handle *tep = tep_alloc();
-...
-void process_record(struct tep_record *record)
-{
- int offset = 24;
- int data = tep_read_number(tep, record->data + offset, 4);
-
- /* Read the 4 bytes at the offset 24 of data as an integer */
-}
-...
---
-
-FILES
------
-[verse]
---
-*event-parse.h*
- Header file to include in order to have access to the library APIs.
-*-ltraceevent*
- Linker switch to add when building a program that uses the library.
---
-
-SEE ALSO
---------
-_libtraceevent(3)_, _trace-cmd(1)_
-
-AUTHOR
-------
-[verse]
---
-*Steven Rostedt* <rostedt@goodmis.org>, author of *libtraceevent*.
-*Tzvetomir Stoyanov* <tz.stoyanov@gmail.com>, author of this man page.
---
-REPORTING BUGS
---------------
-Report bugs to <linux-trace-devel@vger.kernel.org>
-
-LICENSE
--------
-libtraceevent is Free Software licensed under the GNU LGPL 2.1
-
-RESOURCES
----------
-https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
diff --git a/tools/lib/traceevent/Documentation/libtraceevent-event_find.txt b/tools/lib/traceevent/Documentation/libtraceevent-event_find.txt
deleted file mode 100644
index 7bc062c..0000000
--- a/tools/lib/traceevent/Documentation/libtraceevent-event_find.txt
+++ /dev/null
@@ -1,103 +0,0 @@
-libtraceevent(3)
-================
-
-NAME
-----
-tep_find_event,tep_find_event_by_name,tep_find_event_by_record -
-Find events by given key.
-
-SYNOPSIS
---------
-[verse]
---
-*#include <event-parse.h>*
-
-struct tep_event pass:[*]*tep_find_event*(struct tep_handle pass:[*]_tep_, int _id_);
-struct tep_event pass:[*]*tep_find_event_by_name*(struct tep_handle pass:[*]_tep_, const char pass:[*]_sys_, const char pass:[*]_name_);
-struct tep_event pass:[*]*tep_find_event_by_record*(struct tep_handle pass:[*]_tep_, struct tep_record pass:[*]_record_);
---
-
-DESCRIPTION
------------
-This set of functions can be used to search for an event, based on a given
-criteria. All functions require a pointer to a _tep_, trace event parser
-context.
-
-The _tep_find_event()_ function searches for an event by given event _id_. The
-event ID is assigned dynamically and can be viewed in event's format file,
-"ID" field.
-
-The tep_find_event_by_name()_ function searches for an event by given
-event _name_, under the system _sys_. If the _sys_ is NULL (not specified),
-the first event with _name_ is returned.
-
-The tep_find_event_by_record()_ function searches for an event from a given
-_record_.
-
-RETURN VALUE
-------------
-All these functions return a pointer to the found event, or NULL if there is no
-such event.
-
-EXAMPLE
--------
-[source,c]
---
-#include <event-parse.h>
-...
-struct tep_handle *tep = tep_alloc();
-...
-struct tep_event *event;
-
-event = tep_find_event(tep, 1857);
-if (event == NULL) {
- /* There is no event with ID 1857 */
-}
-
-event = tep_find_event_by_name(tep, "kvm", "kvm_exit");
-if (event == NULL) {
- /* There is no kvm_exit event, from kvm system */
-}
-
-void event_from_record(struct tep_record *record)
-{
- struct tep_event *event = tep_find_event_by_record(tep, record);
- if (event == NULL) {
- /* There is no event from given record */
- }
-}
-...
---
-
-FILES
------
-[verse]
---
-*event-parse.h*
- Header file to include in order to have access to the library APIs.
-*-ltraceevent*
- Linker switch to add when building a program that uses the library.
---
-
-SEE ALSO
---------
-_libtraceevent(3)_, _trace-cmd(1)_
-
-AUTHOR
-------
-[verse]
---
-*Steven Rostedt* <rostedt@goodmis.org>, author of *libtraceevent*.
-*Tzvetomir Stoyanov* <tz.stoyanov@gmail.com>, author of this man page.
---
-REPORTING BUGS
---------------
-Report bugs to <linux-trace-devel@vger.kernel.org>
-
-LICENSE
--------
-libtraceevent is Free Software licensed under the GNU LGPL 2.1
-
-RESOURCES
----------
-https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
diff --git a/tools/lib/traceevent/Documentation/libtraceevent-event_get.txt b/tools/lib/traceevent/Documentation/libtraceevent-event_get.txt
deleted file mode 100644
index 6525092..0000000
--- a/tools/lib/traceevent/Documentation/libtraceevent-event_get.txt
+++ /dev/null
@@ -1,99 +0,0 @@
-libtraceevent(3)
-================
-
-NAME
-----
-tep_get_event, tep_get_first_event, tep_get_events_count - Access events.
-
-SYNOPSIS
---------
-[verse]
---
-*#include <event-parse.h>*
-
-struct tep_event pass:[*]*tep_get_event*(struct tep_handle pass:[*]_tep_, int _index_);
-struct tep_event pass:[*]*tep_get_first_event*(struct tep_handle pass:[*]_tep_);
-int *tep_get_events_count*(struct tep_handle pass:[*]_tep_);
---
-
-DESCRIPTION
------------
-The _tep_get_event()_ function returns a pointer to event at the given _index_.
-The _tep_ argument is trace event parser context, the _index_ is the index of
-the requested event.
-
-The _tep_get_first_event()_ function returns a pointer to the first event.
-As events are stored in an array, this function returns the pointer to the
-beginning of the array. The _tep_ argument is trace event parser context.
-
-The _tep_get_events_count()_ function returns the number of the events
-in the array. The _tep_ argument is trace event parser context.
-
-RETURN VALUE
-------------
-The _tep_get_event()_ returns a pointer to the event located at _index_.
-NULL is returned in case of error, in case there are no events or _index_ is
-out of range.
-
-The _tep_get_first_event()_ returns a pointer to the first event. NULL is
-returned in case of error, or in case there are no events.
-
-The _tep_get_events_count()_ returns the number of the events. 0 is
-returned in case of error, or in case there are no events.
-
-EXAMPLE
--------
-[source,c]
---
-#include <event-parse.h>
-...
-struct tep_handle *tep = tep_alloc();
-...
-int i,count = tep_get_events_count(tep);
-struct tep_event *event, *events = tep_get_first_event(tep);
-
-if (events == NULL) {
- /* There are no events */
-} else {
- for (i = 0; i < count; i++) {
- event = (events+i);
- /* process events[i] */
- }
-
- /* Get the last event */
- event = tep_get_event(tep, count-1);
-}
---
-
-FILES
------
-[verse]
---
-*event-parse.h*
- Header file to include in order to have access to the library APIs.
-*-ltraceevent*
- Linker switch to add when building a program that uses the library.
---
-
-SEE ALSO
---------
-_libtraceevent(3)_, _trace-cmd(1)_
-
-AUTHOR
-------
-[verse]
---
-*Steven Rostedt* <rostedt@goodmis.org>, author of *libtraceevent*.
-*Tzvetomir Stoyanov* <tz.stoyanov@gmail.com>, author of this man page.
---
-REPORTING BUGS
---------------
-Report bugs to <linux-trace-devel@vger.kernel.org>
-
-LICENSE
--------
-libtraceevent is Free Software licensed under the GNU LGPL 2.1
-
-RESOURCES
----------
-https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
diff --git a/tools/lib/traceevent/Documentation/libtraceevent-event_list.txt b/tools/lib/traceevent/Documentation/libtraceevent-event_list.txt
deleted file mode 100644
index fba350e..0000000
--- a/tools/lib/traceevent/Documentation/libtraceevent-event_list.txt
+++ /dev/null
@@ -1,122 +0,0 @@
-libtraceevent(3)
-================
-
-NAME
-----
-tep_list_events, tep_list_events_copy -
-Get list of events, sorted by given criteria.
-
-SYNOPSIS
---------
-[verse]
---
-*#include <event-parse.h>*
-
-enum *tep_event_sort_type* {
- _TEP_EVENT_SORT_ID_,
- _TEP_EVENT_SORT_NAME_,
- _TEP_EVENT_SORT_SYSTEM_,
-};
-
-struct tep_event pass:[*]pass:[*]*tep_list_events*(struct tep_handle pass:[*]_tep_, enum tep_event_sort_type _sort_type_);
-struct tep_event pass:[*]pass:[*]*tep_list_events_copy*(struct tep_handle pass:[*]_tep_, enum tep_event_sort_type _sort_type_);
---
-
-DESCRIPTION
------------
-The _tep_list_events()_ function returns an array of pointers to the events,
-sorted by the _sort_type_ criteria. The last element of the array is NULL.
-The returned memory must not be freed, it is managed by the library.
-The function is not thread safe. The _tep_ argument is trace event parser
-context. The _sort_type_ argument is the required sort criteria:
-[verse]
---
- _TEP_EVENT_SORT_ID_ - sort by the event ID.
- _TEP_EVENT_SORT_NAME_ - sort by the event (name, system, id) triplet.
- _TEP_EVENT_SORT_SYSTEM_ - sort by the event (system, name, id) triplet.
---
-
-The _tep_list_events_copy()_ is a thread safe version of _tep_list_events()_.
-It has the same behavior, but the returned array is allocated internally and
-must be freed by the caller. Note that the content of the array must not be
-freed (see the EXAMPLE below).
-
-RETURN VALUE
-------------
-The _tep_list_events()_ function returns an array of pointers to events.
-In case of an error, NULL is returned. The returned array must not be freed,
-it is managed by the library.
-
-The _tep_list_events_copy()_ function returns an array of pointers to events.
-In case of an error, NULL is returned. The returned array must be freed by
-the caller.
-
-EXAMPLE
--------
-[source,c]
---
-#include <event-parse.h>
-...
-struct tep_handle *tep = tep_alloc();
-...
-int i;
-struct tep_event_format **events;
-
-i=0;
-events = tep_list_events(tep, TEP_EVENT_SORT_ID);
-if (events == NULL) {
- /* Failed to get the events, sorted by ID */
-} else {
- while(events[i]) {
- /* walk through the list of the events, sorted by ID */
- i++;
- }
-}
-
-i=0;
-events = tep_list_events_copy(tep, TEP_EVENT_SORT_NAME);
-if (events == NULL) {
- /* Failed to get the events, sorted by name */
-} else {
- while(events[i]) {
- /* walk through the list of the events, sorted by name */
- i++;
- }
- free(events);
-}
-
-...
---
-
-FILES
------
-[verse]
---
-*event-parse.h*
- Header file to include in order to have access to the library APIs.
-*-ltraceevent*
- Linker switch to add when building a program that uses the library.
---
-
-SEE ALSO
---------
-_libtraceevent(3)_, _trace-cmd(1)_
-
-AUTHOR
-------
-[verse]
---
-*Steven Rostedt* <rostedt@goodmis.org>, author of *libtraceevent*.
-*Tzvetomir Stoyanov* <tz.stoyanov@gmail.com>, author of this man page.
---
-REPORTING BUGS
---------------
-Report bugs to <linux-trace-devel@vger.kernel.org>
-
-LICENSE
--------
-libtraceevent is Free Software licensed under the GNU LGPL 2.1
-
-RESOURCES
----------
-https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
diff --git a/tools/lib/traceevent/Documentation/libtraceevent-event_print.txt b/tools/lib/traceevent/Documentation/libtraceevent-event_print.txt
deleted file mode 100644
index 2c6a618..0000000
--- a/tools/lib/traceevent/Documentation/libtraceevent-event_print.txt
+++ /dev/null
@@ -1,130 +0,0 @@
-libtraceevent(3)
-================
-
-NAME
-----
-tep_print_event - Writes event information into a trace sequence.
-
-SYNOPSIS
---------
-[verse]
---
-*#include <event-parse.h>*
-*#include <trace-seq.h>*
-
-void *tep_print_event*(struct tep_handle pass:[*]_tep_, struct trace_seqpass:[*]_s_, struct tep_record pass:[*]_record_, const char pass:[*]_fmt_, _..._)
---
-
-DESCRIPTION
------------
-
-The _tep_print_event()_ function parses the event information of the given
-_record_ and writes it into the trace sequence _s_, according to the format
-string _fmt_. The desired information is specified after the format string.
-The _fmt_ is printf-like format string, following arguments are supported:
-[verse]
---
- TEP_PRINT_PID, "%d" - PID of the event.
- TEP_PRINT_CPU, "%d" - Event CPU.
- TEP_PRINT_COMM, "%s" - Event command string.
- TEP_PRINT_NAME, "%s" - Event name.
- TEP_PRINT_LATENCY, "%s" - Latency of the event. It prints 4 or more
- fields - interrupt state, scheduling state,
- current context, and preemption count.
- Field 1 is the interrupt enabled state:
- d : Interrupts are disabled
- . : Interrupts are enabled
- X : The architecture does not support this
- information
- Field 2 is the "need resched" state.
- N : The task is set to call the scheduler when
- possible, as another higher priority task
- may need to be scheduled in.
- . : The task is not set to call the scheduler.
- Field 3 is the context state.
- . : Normal context
- s : Soft interrupt context
- h : Hard interrupt context
- H : Hard interrupt context which triggered
- during soft interrupt context.
- z : NMI context
- Z : NMI context which triggered during hard
- interrupt context
- Field 4 is the preemption count.
- . : The preempt count is zero.
- On preemptible kernels (where the task can be scheduled
- out in arbitrary locations while in kernel context), the
- preempt count, when non zero, will prevent the kernel
- from scheduling out the current task. The preempt count
- number is displayed when it is not zero.
- Depending on the kernel, it may show other fields
- (lock depth, or migration disabled, which are unique to
- specialized kernels).
- TEP_PRINT_TIME, %d - event time stamp. A divisor and precision can be
- specified as part of this format string:
- "%precision.divisord". Example:
- "%3.1000d" - divide the time by 1000 and print the first
- 3 digits before the dot. Thus, the time stamp
- "123456000" will be printed as "123.456"
- TEP_PRINT_INFO, "%s" - event information.
- TEP_PRINT_INFO_RAW, "%s" - event information, in raw format.
-
---
-EXAMPLE
--------
-[source,c]
---
-#include <event-parse.h>
-#include <trace-seq.h>
-...
-struct trace_seq seq;
-trace_seq_init(&seq);
-struct tep_handle *tep = tep_alloc();
-...
-void print_my_event(struct tep_record *record)
-{
- trace_seq_reset(&seq);
- tep_print_event(tep, s, record, "%16s-%-5d [%03d] %s %6.1000d %s %s",
- TEP_PRINT_COMM, TEP_PRINT_PID, TEP_PRINT_CPU,
- TEP_PRINT_LATENCY, TEP_PRINT_TIME, TEP_PRINT_NAME,
- TEP_PRINT_INFO);
-}
-...
---
-
-FILES
------
-[verse]
---
-*event-parse.h*
- Header file to include in order to have access to the library APIs.
-*trace-seq.h*
- Header file to include in order to have access to trace sequences related APIs.
- Trace sequences are used to allow a function to call several other functions
- to create a string of data to use.
-*-ltraceevent*
- Linker switch to add when building a program that uses the library.
---
-
-SEE ALSO
---------
-_libtraceevent(3)_, _trace-cmd(1)_
-
-AUTHOR
-------
-[verse]
---
-*Steven Rostedt* <rostedt@goodmis.org>, author of *libtraceevent*.
-*Tzvetomir Stoyanov* <tz.stoyanov@gmail.com>, author of this man page.
---
-REPORTING BUGS
---------------
-Report bugs to <linux-trace-devel@vger.kernel.org>
-
-LICENSE
--------
-libtraceevent is Free Software licensed under the GNU LGPL 2.1
-
-RESOURCES
----------
-https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
diff --git a/tools/lib/traceevent/Documentation/libtraceevent-field_find.txt b/tools/lib/traceevent/Documentation/libtraceevent-field_find.txt
deleted file mode 100644
index 0896af5..0000000
--- a/tools/lib/traceevent/Documentation/libtraceevent-field_find.txt
+++ /dev/null
@@ -1,118 +0,0 @@
-libtraceevent(3)
-================
-
-NAME
-----
-tep_find_common_field, tep_find_field, tep_find_any_field -
-Search for a field in an event.
-
-SYNOPSIS
---------
-[verse]
---
-*#include <event-parse.h>*
-
-struct tep_format_field pass:[*]*tep_find_common_field*(struct tep_event pass:[*]_event_, const char pass:[*]_name_);
-struct tep_format_field pass:[*]*tep_find_field*(struct tep_event_ormat pass:[*]_event_, const char pass:[*]_name_);
-struct tep_format_field pass:[*]*tep_find_any_field*(struct tep_event pass:[*]_event_, const char pass:[*]_name_);
---
-
-DESCRIPTION
------------
-These functions search for a field with given name in an event. The field
-returned can be used to find the field content from within a data record.
-
-The _tep_find_common_field()_ function searches for a common field with _name_
-in the _event_.
-
-The _tep_find_field()_ function searches for an event specific field with
-_name_ in the _event_.
-
-The _tep_find_any_field()_ function searches for any field with _name_ in the
-_event_.
-
-RETURN VALUE
-------------
-The _tep_find_common_field(), _tep_find_field()_ and _tep_find_any_field()_
-functions return a pointer to the found field, or NULL in case there is no field
-with the requested name.
-
-EXAMPLE
--------
-[source,c]
---
-#include <event-parse.h>
-...
-void get_htimer_info(struct tep_handle *tep, struct tep_record *record)
-{
- struct tep_format_field *field;
- struct tep_event *event;
- long long softexpires;
- int mode;
- int pid;
-
- event = tep_find_event_by_name(tep, "timer", "hrtimer_start");
-
- field = tep_find_common_field(event, "common_pid");
- if (field == NULL) {
- /* Cannot find "common_pid" field in the event */
- } else {
- /* Get pid from the data record */
- pid = tep_read_number(tep, record->data + field->offset,
- field->size);
- }
-
- field = tep_find_field(event, "softexpires");
- if (field == NULL) {
- /* Cannot find "softexpires" event specific field in the event */
- } else {
- /* Get softexpires parameter from the data record */
- softexpires = tep_read_number(tep, record->data + field->offset,
- field->size);
- }
-
- field = tep_find_any_field(event, "mode");
- if (field == NULL) {
- /* Cannot find "mode" field in the event */
- } else
- {
- /* Get mode parameter from the data record */
- mode = tep_read_number(tep, record->data + field->offset,
- field->size);
- }
-}
-...
---
-
-FILES
------
-[verse]
---
-*event-parse.h*
- Header file to include in order to have access to the library APIs.
-*-ltraceevent*
- Linker switch to add when building a program that uses the library.
---
-
-SEE ALSO
---------
-_libtraceevent(3)_, _trace-cmd(1)_
-
-AUTHOR
-------
-[verse]
---
-*Steven Rostedt* <rostedt@goodmis.org>, author of *libtraceevent*.
-*Tzvetomir Stoyanov* <tz.stoyanov@gmail.com>, author of this man page.
---
-REPORTING BUGS
---------------
-Report bugs to <linux-trace-devel@vger.kernel.org>
-
-LICENSE
--------
-libtraceevent is Free Software licensed under the GNU LGPL 2.1
-
-RESOURCES
----------
-https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
diff --git a/tools/lib/traceevent/Documentation/libtraceevent-field_get_val.txt b/tools/lib/traceevent/Documentation/libtraceevent-field_get_val.txt
deleted file mode 100644
index 6324f0d..0000000
--- a/tools/lib/traceevent/Documentation/libtraceevent-field_get_val.txt
+++ /dev/null
@@ -1,122 +0,0 @@
-libtraceevent(3)
-================
-
-NAME
-----
-tep_get_any_field_val, tep_get_common_field_val, tep_get_field_val,
-tep_get_field_raw - Get value of a field.
-
-SYNOPSIS
---------
-[verse]
---
-*#include <event-parse.h>*
-*#include <trace-seq.h>*
-
-int *tep_get_any_field_val*(struct trace_seq pass:[*]_s_, struct tep_event pass:[*]_event_, const char pass:[*]_name_, struct tep_record pass:[*]_record_, unsigned long long pass:[*]_val_, int _err_);
-int *tep_get_common_field_val*(struct trace_seq pass:[*]_s_, struct tep_event pass:[*]_event_, const char pass:[*]_name_, struct tep_record pass:[*]_record_, unsigned long long pass:[*]_val_, int _err_);
-int *tep_get_field_val*(struct trace_seq pass:[*]_s_, struct tep_event pass:[*]_event_, const char pass:[*]_name_, struct tep_record pass:[*]_record_, unsigned long long pass:[*]_val_, int _err_);
-void pass:[*]*tep_get_field_raw*(struct trace_seq pass:[*]_s_, struct tep_event pass:[*]_event_, const char pass:[*]_name_, struct tep_record pass:[*]_record_, int pass:[*]_len_, int _err_);
---
-
-DESCRIPTION
------------
-These functions can be used to find a field and retrieve its value.
-
-The _tep_get_any_field_val()_ function searches in the _record_ for a field
-with _name_, part of the _event_. If the field is found, its value is stored in
-_val_. If there is an error and _err_ is not zero, then an error string is
-written into _s_.
-
-The _tep_get_common_field_val()_ function does the same as
-_tep_get_any_field_val()_, but searches only in the common fields. This works
-for any event as all events include the common fields.
-
-The _tep_get_field_val()_ function does the same as _tep_get_any_field_val()_,
-but searches only in the event specific fields.
-
-The _tep_get_field_raw()_ function searches in the _record_ for a field with
-_name_, part of the _event_. If the field is found, a pointer to where the field
-exists in the record's raw data is returned. The size of the data is stored in
-_len_. If there is an error and _err_ is not zero, then an error string is
-written into _s_.
-
-RETURN VALUE
-------------
-The _tep_get_any_field_val()_, _tep_get_common_field_val()_ and
-_tep_get_field_val()_ functions return 0 on success, or -1 in case of an error.
-
-The _tep_get_field_raw()_ function returns a pointer to field's raw data, and
-places the length of this data in _len_. In case of an error NULL is returned.
-
-EXAMPLE
--------
-[source,c]
---
-#include <event-parse.h>
-#include <trace-seq.h>
-...
-struct tep_handle *tep = tep_alloc();
-...
-struct tep_event *event = tep_find_event_by_name(tep, "kvm", "kvm_exit");
-...
-void process_record(struct tep_record *record)
-{
- int len;
- char *comm;
- struct tep_event_format *event;
- unsigned long long val;
-
- event = tep_find_event_by_record(pevent, record);
- if (event != NULL) {
- if (tep_get_common_field_val(NULL, event, "common_type",
- record, &val, 0) == 0) {
- /* Got the value of common type field */
- }
- if (tep_get_field_val(NULL, event, "pid", record, &val, 0) == 0) {
- /* Got the value of pid specific field */
- }
- comm = tep_get_field_raw(NULL, event, "comm", record, &len, 0);
- if (comm != NULL) {
- /* Got a pointer to the comm event specific field */
- }
- }
-}
---
-
-FILES
------
-[verse]
---
-*event-parse.h*
- Header file to include in order to have access to the library APIs.
-*trace-seq.h*
- Header file to include in order to have access to trace sequences
- related APIs. Trace sequences are used to allow a function to call
- several other functions to create a string of data to use.
-*-ltraceevent*
- Linker switch to add when building a program that uses the library.
---
-
-SEE ALSO
---------
-_libtraceevent(3)_, _trace-cmd(1)_
-
-AUTHOR
-------
-[verse]
---
-*Steven Rostedt* <rostedt@goodmis.org>, author of *libtraceevent*.
-*Tzvetomir Stoyanov* <tz.stoyanov@gmail.com>, author of this man page.
---
-REPORTING BUGS
---------------
-Report bugs to <linux-trace-devel@vger.kernel.org>
-
-LICENSE
--------
-libtraceevent is Free Software licensed under the GNU LGPL 2.1
-
-RESOURCES
----------
-https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
diff --git a/tools/lib/traceevent/Documentation/libtraceevent-field_print.txt b/tools/lib/traceevent/Documentation/libtraceevent-field_print.txt
deleted file mode 100644
index 9a9df98..0000000
--- a/tools/lib/traceevent/Documentation/libtraceevent-field_print.txt
+++ /dev/null
@@ -1,126 +0,0 @@
-libtraceevent(3)
-================
-
-NAME
-----
-tep_print_field, tep_print_fields, tep_print_num_field, tep_print_func_field -
-Print the field content.
-
-SYNOPSIS
---------
-[verse]
---
-*#include <event-parse.h>*
-*#include <trace-seq.h>*
-
-void *tep_print_field*(struct trace_seq pass:[*]_s_, void pass:[*]_data_, struct tep_format_field pass:[*]_field_);
-void *tep_print_fields*(struct trace_seq pass:[*]_s_, void pass:[*]_data_, int _size_, struct tep_event pass:[*]_event_);
-int *tep_print_num_field*(struct trace_seq pass:[*]_s_, const char pass:[*]_fmt_, struct tep_event pass:[*]_event_, const char pass:[*]_name_, struct tep_record pass:[*]_record_, int _err_);
-int *tep_print_func_field*(struct trace_seq pass:[*]_s_, const char pass:[*]_fmt_, struct tep_event pass:[*]_event_, const char pass:[*]_name_, struct tep_record pass:[*]_record_, int _err_);
---
-
-DESCRIPTION
------------
-These functions print recorded field's data, according to the field's type.
-
-The _tep_print_field()_ function extracts from the recorded raw _data_ value of
-the _field_ and prints it into _s_, according to the field type.
-
-The _tep_print_fields()_ prints each field name followed by the record's field
-value according to the field's type:
-[verse]
---
-"field1_name=field1_value field2_name=field2_value ..."
---
-It iterates all fields of the _event_, and calls _tep_print_field()_ for each of
-them.
-
-The _tep_print_num_field()_ function prints a numeric field with given format
-string. A search is performed in the _event_ for a field with _name_. If such
-field is found, its value is extracted from the _record_ and is printed in the
-_s_, according to the given format string _fmt_. If the argument _err_ is
-non-zero, and an error occures - it is printed in the _s_.
-
-The _tep_print_func_field()_ function prints a function field with given format
-string. A search is performed in the _event_ for a field with _name_. If such
-field is found, its value is extracted from the _record_. The value is assumed
-to be a function address, and a search is perform to find the name of this
-function. The function name (if found) and its address are printed in the _s_,
-according to the given format string _fmt_. If the argument _err_ is non-zero,
-and an error occures - it is printed in _s_.
-
-RETURN VALUE
-------------
-The _tep_print_num_field()_ and _tep_print_func_field()_ functions return 1
-on success, -1 in case of an error or 0 if the print buffer _s_ is full.
-
-EXAMPLE
--------
-[source,c]
---
-#include <event-parse.h>
-#include <trace-seq.h>
-...
-struct tep_handle *tep = tep_alloc();
-...
-struct trace_seq seq;
-trace_seq_init(&seq);
-struct tep_event *event = tep_find_event_by_name(tep, "timer", "hrtimer_start");
-...
-void process_record(struct tep_record *record)
-{
- struct tep_format_field *field_pid = tep_find_common_field(event, "common_pid");
-
- trace_seq_reset(&seq);
-
- /* Print the value of "common_pid" */
- tep_print_field(&seq, record->data, field_pid);
-
- /* Print all fields of the "hrtimer_start" event */
- tep_print_fields(&seq, record->data, record->size, event);
-
- /* Print the value of "expires" field with custom format string */
- tep_print_num_field(&seq, " timer expires in %llu ", event, "expires", record, 0);
-
- /* Print the address and the name of "function" field with custom format string */
- tep_print_func_field(&seq, " timer function is %s ", event, "function", record, 0);
- }
- ...
---
-
-FILES
------
-[verse]
---
-*event-parse.h*
- Header file to include in order to have access to the library APIs.
-*trace-seq.h*
- Header file to include in order to have access to trace sequences related APIs.
- Trace sequences are used to allow a function to call several other functions
- to create a string of data to use.
-*-ltraceevent*
- Linker switch to add when building a program that uses the library.
---
-
-SEE ALSO
---------
-_libtraceevent(3)_, _trace-cmd(1)_
-
-AUTHOR
-------
-[verse]
---
-*Steven Rostedt* <rostedt@goodmis.org>, author of *libtraceevent*.
-*Tzvetomir Stoyanov* <tz.stoyanov@gmail.com>, author of this man page.
---
-REPORTING BUGS
---------------
-Report bugs to <linux-trace-devel@vger.kernel.org>
-
-LICENSE
--------
-libtraceevent is Free Software licensed under the GNU LGPL 2.1
-
-RESOURCES
----------
-https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
diff --git a/tools/lib/traceevent/Documentation/libtraceevent-field_read.txt b/tools/lib/traceevent/Documentation/libtraceevent-field_read.txt
deleted file mode 100644
index 64e9e25..0000000
--- a/tools/lib/traceevent/Documentation/libtraceevent-field_read.txt
+++ /dev/null
@@ -1,81 +0,0 @@
-libtraceevent(3)
-================
-
-NAME
-----
-tep_read_number_field - Reads a number from raw data.
-
-SYNOPSIS
---------
-[verse]
---
-*#include <event-parse.h>*
-
-int *tep_read_number_field*(struct tep_format_field pass:[*]_field_, const void pass:[*]_data_, unsigned long long pass:[*]_value_);
---
-
-DESCRIPTION
------------
-The _tep_read_number_field()_ function reads the value of the _field_ from the
-raw _data_ and stores it in the _value_. The function sets the _value_ according
-to the endianness of the raw data and the current machine and stores it in
-_value_.
-
-RETURN VALUE
-------------
-The _tep_read_number_field()_ function retunrs 0 in case of success, or -1 in
-case of an error.
-
-EXAMPLE
--------
-[source,c]
---
-#include <event-parse.h>
-...
-struct tep_handle *tep = tep_alloc();
-...
-struct tep_event *event = tep_find_event_by_name(tep, "timer", "hrtimer_start");
-...
-void process_record(struct tep_record *record)
-{
- unsigned long long pid;
- struct tep_format_field *field_pid = tep_find_common_field(event, "common_pid");
-
- if (tep_read_number_field(field_pid, record->data, &pid) != 0) {
- /* Failed to get "common_pid" value */
- }
-}
-...
---
-FILES
------
-[verse]
---
-*event-parse.h*
- Header file to include in order to have access to the library APIs.
-*-ltraceevent*
- Linker switch to add when building a program that uses the library.
---
-
-SEE ALSO
---------
-_libtraceevent(3)_, _trace-cmd(1)_
-
-AUTHOR
-------
-[verse]
---
-*Steven Rostedt* <rostedt@goodmis.org>, author of *libtraceevent*.
-*Tzvetomir Stoyanov* <tz.stoyanov@gmail.com>, author of this man page.
---
-REPORTING BUGS
---------------
-Report bugs to <linux-trace-devel@vger.kernel.org>
-
-LICENSE
--------
-libtraceevent is Free Software licensed under the GNU LGPL 2.1
-
-RESOURCES
----------
-https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
diff --git a/tools/lib/traceevent/Documentation/libtraceevent-fields.txt b/tools/lib/traceevent/Documentation/libtraceevent-fields.txt
deleted file mode 100644
index 1ccb531..0000000
--- a/tools/lib/traceevent/Documentation/libtraceevent-fields.txt
+++ /dev/null
@@ -1,105 +0,0 @@
-libtraceevent(3)
-================
-
-NAME
-----
-tep_event_common_fields, tep_event_fields - Get a list of fields for an event.
-
-SYNOPSIS
---------
-[verse]
---
-*#include <event-parse.h>*
-
-struct tep_format_field pass:[*]pass:[*]*tep_event_common_fields*(struct tep_event pass:[*]_event_);
-struct tep_format_field pass:[*]pass:[*]*tep_event_fields*(struct tep_event pass:[*]_event_);
---
-
-DESCRIPTION
------------
-The _tep_event_common_fields()_ function returns an array of pointers to common
-fields for the _event_. The array is allocated in the function and must be freed
-by free(). The last element of the array is NULL.
-
-The _tep_event_fields()_ function returns an array of pointers to event specific
-fields for the _event_. The array is allocated in the function and must be freed
-by free(). The last element of the array is NULL.
-
-RETURN VALUE
-------------
-Both _tep_event_common_fields()_ and _tep_event_fields()_ functions return
-an array of pointers to tep_format_field structures in case of success, or
-NULL in case of an error.
-
-EXAMPLE
--------
-[source,c]
---
-#include <event-parse.h>
-...
-struct tep_handle *tep = tep_alloc();
-...
-int i;
-struct tep_format_field **fields;
-struct tep_event *event = tep_find_event_by_name(tep, "kvm", "kvm_exit");
-if (event != NULL) {
- fields = tep_event_common_fields(event);
- if (fields != NULL) {
- i = 0;
- while (fields[i]) {
- /*
- walk through the list of the common fields
- of the kvm_exit event
- */
- i++;
- }
- free(fields);
- }
- fields = tep_event_fields(event);
- if (fields != NULL) {
- i = 0;
- while (fields[i]) {
- /*
- walk through the list of the event specific
- fields of the kvm_exit event
- */
- i++;
- }
- free(fields);
- }
-}
-...
---
-
-FILES
------
-[verse]
---
-*event-parse.h*
- Header file to include in order to have access to the library APIs.
-*-ltraceevent*
- Linker switch to add when building a program that uses the library.
---
-
-SEE ALSO
---------
-_libtraceevent(3)_, _trace-cmd(1)_
-
-AUTHOR
-------
-[verse]
---
-*Steven Rostedt* <rostedt@goodmis.org>, author of *libtraceevent*.
-*Tzvetomir Stoyanov* <tz.stoyanov@gmail.com>, author of this man page.
---
-REPORTING BUGS
---------------
-Report bugs to <linux-trace-devel@vger.kernel.org>
-
-LICENSE
--------
-libtraceevent is Free Software licensed under the GNU LGPL 2.1
-
-RESOURCES
----------
-https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
diff --git a/tools/lib/traceevent/Documentation/libtraceevent-file_endian.txt b/tools/lib/traceevent/Documentation/libtraceevent-file_endian.txt
deleted file mode 100644
index f401ad3..0000000
--- a/tools/lib/traceevent/Documentation/libtraceevent-file_endian.txt
+++ /dev/null
@@ -1,91 +0,0 @@
-libtraceevent(3)
-================
-
-NAME
-----
-tep_is_file_bigendian, tep_set_file_bigendian - Get / set the endianness of the
-raw data being accessed by the tep handler.
-
-SYNOPSIS
---------
-[verse]
---
-*#include <event-parse.h>*
-
-enum *tep_endian* {
- TEP_LITTLE_ENDIAN = 0,
- TEP_BIG_ENDIAN
-};
-
-bool *tep_is_file_bigendian*(struct tep_handle pass:[*]_tep_);
-void *tep_set_file_bigendian*(struct tep_handle pass:[*]_tep_, enum tep_endian _endian_);
-
---
-DESCRIPTION
------------
-The _tep_is_file_bigendian()_ function gets the endianness of the raw data,
-being accessed by the tep handler. The _tep_ argument is trace event parser
-context.
-
-The _tep_set_file_bigendian()_ function sets the endianness of raw data being
-accessed by the tep handler. The _tep_ argument is trace event parser context.
-[verse]
---
-The _endian_ argument is the endianness:
- _TEP_LITTLE_ENDIAN_ - the raw data is in little endian format,
- _TEP_BIG_ENDIAN_ - the raw data is in big endian format.
---
-RETURN VALUE
-------------
-The _tep_is_file_bigendian()_ function returns true if the data is in bigendian
-format, false otherwise.
-
-EXAMPLE
--------
-[source,c]
---
-#include <event-parse.h>
-...
-struct tep_handle *tep = tep_alloc();
-...
- tep_set_file_bigendian(tep, TEP_LITTLE_ENDIAN);
-...
- if (tep_is_file_bigendian(tep)) {
- /* The raw data is in big endian */
- } else {
- /* The raw data is in little endian */
- }
---
-
-FILES
------
-[verse]
---
-*event-parse.h*
- Header file to include in order to have access to the library APIs.
-*-ltraceevent*
- Linker switch to add when building a program that uses the library.
---
-
-SEE ALSO
---------
-_libtraceevent(3)_, _trace-cmd(1)_
-
-AUTHOR
-------
-[verse]
---
-*Steven Rostedt* <rostedt@goodmis.org>, author of *libtraceevent*.
-*Tzvetomir Stoyanov* <tz.stoyanov@gmail.com>, author of this man page.
---
-REPORTING BUGS
---------------
-Report bugs to <linux-trace-devel@vger.kernel.org>
-
-LICENSE
--------
-libtraceevent is Free Software licensed under the GNU LGPL 2.1
-
-RESOURCES
----------
-https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
diff --git a/tools/lib/traceevent/Documentation/libtraceevent-filter.txt b/tools/lib/traceevent/Documentation/libtraceevent-filter.txt
deleted file mode 100644
index 4a9962d..0000000
--- a/tools/lib/traceevent/Documentation/libtraceevent-filter.txt
+++ /dev/null
@@ -1,209 +0,0 @@
-libtraceevent(3)
-================
-
-NAME
-----
-tep_filter_alloc, tep_filter_free, tep_filter_reset, tep_filter_make_string,
-tep_filter_copy, tep_filter_compare, tep_filter_match, tep_event_filtered,
-tep_filter_remove_event, tep_filter_strerror, tep_filter_add_filter_str -
-Event filter related APIs.
-
-SYNOPSIS
---------
-[verse]
---
-*#include <event-parse.h>*
-
-struct tep_event_filter pass:[*]*tep_filter_alloc*(struct tep_handle pass:[*]_tep_);
-void *tep_filter_free*(struct tep_event_filter pass:[*]_filter_);
-void *tep_filter_reset*(struct tep_event_filter pass:[*]_filter_);
-enum tep_errno *tep_filter_add_filter_str*(struct tep_event_filter pass:[*]_filter_, const char pass:[*]_filter_str_);
-int *tep_event_filtered*(struct tep_event_filter pass:[*]_filter_, int _event_id_);
-int *tep_filter_remove_event*(struct tep_event_filter pass:[*]_filter_, int _event_id_);
-enum tep_errno *tep_filter_match*(struct tep_event_filter pass:[*]_filter_, struct tep_record pass:[*]_record_);
-int *tep_filter_copy*(struct tep_event_filter pass:[*]_dest_, struct tep_event_filter pass:[*]_source_);
-int *tep_filter_compare*(struct tep_event_filter pass:[*]_filter1_, struct tep_event_filter pass:[*]_filter2_);
-char pass:[*]*tep_filter_make_string*(struct tep_event_filter pass:[*]_filter_, int _event_id_);
-int *tep_filter_strerror*(struct tep_event_filter pass:[*]_filter_, enum tep_errno _err_, char pass:[*]buf, size_t _buflen_);
---
-
-DESCRIPTION
------------
-Filters can be attached to traced events. They can be used to filter out various
-events when outputting them. Each event can be filtered based on its parameters,
-described in the event's format file. This set of functions can be used to
-create, delete, modify and attach event filters.
-
-The _tep_filter_alloc()_ function creates a new event filter. The _tep_ argument
-is the trace event parser context.
-
-The _tep_filter_free()_ function frees an event filter and all resources that it
-had used.
-
-The _tep_filter_reset()_ function removes all rules from an event filter and
-resets it.
-
-The _tep_filter_add_filter_str()_ function adds a new rule to the _filter_. The
-_filter_str_ argument is the filter string, that contains the rule.
-
-The _tep_event_filtered()_ function checks if the event with _event_id_ has
-_filter_.
-
-The _tep_filter_remove_event()_ function removes a _filter_ for an event with
-_event_id_.
-
-The _tep_filter_match()_ function tests if a _record_ matches given _filter_.
-
-The _tep_filter_copy()_ function copies a _source_ filter into a _dest_ filter.
-
-The _tep_filter_compare()_ function compares two filers - _filter1_ and _filter2_.
-
-The _tep_filter_make_string()_ function constructs a string, displaying
-the _filter_ contents for given _event_id_.
-
-The _tep_filter_strerror()_ function copies the _filter_ error buffer into the
-given _buf_ with the size _buflen_. If the error buffer is empty, in the _buf_
-is copied a string, describing the error _err_.
-
-RETURN VALUE
-------------
-The _tep_filter_alloc()_ function returns a pointer to the newly created event
-filter, or NULL in case of an error.
-
-The _tep_filter_add_filter_str()_ function returns 0 if the rule was
-successfully added or a negative error code. Use _tep_filter_strerror()_ to see
-actual error message in case of an error.
-
-The _tep_event_filtered()_ function returns 1 if the filter is found for given
-event, or 0 otherwise.
-
-The _tep_filter_remove_event()_ function returns 1 if the vent was removed, or
-0 if the event was not found.
-
-The _tep_filter_match()_ function returns _tep_errno_, according to the result:
-[verse]
---
-_pass:[TEP_ERRNO__FILTER_MATCH]_ - filter found for event, the record matches.
-_pass:[TEP_ERRNO__FILTER_MISS]_ - filter found for event, the record does not match.
-_pass:[TEP_ERRNO__FILTER_NOT_FOUND]_ - no filter found for record's event.
-_pass:[TEP_ERRNO__NO_FILTER]_ - no rules in the filter.
---
-or any other _tep_errno_, if an error occurred during the test.
-
-The _tep_filter_copy()_ function returns 0 on success or -1 if not all rules
- were copied.
-
-The _tep_filter_compare()_ function returns 1 if the two filters hold the same
-content, or 0 if they do not.
-
-The _tep_filter_make_string()_ function returns a string, which must be freed
-with free(), or NULL in case of an error.
-
-The _tep_filter_strerror()_ function returns 0 if message was filled
-successfully, or -1 in case of an error.
-
-EXAMPLE
--------
-[source,c]
---
-#include <event-parse.h>
-...
-struct tep_handle *tep = tep_alloc();
-...
-char errstr[200];
-int ret;
-
-struct tep_event_filter *filter = tep_filter_alloc(tep);
-struct tep_event_filter *filter1 = tep_filter_alloc(tep);
-ret = tep_filter_add_filter_str(filter, "sched/sched_wakeup:target_cpu==1");
-if(ret < 0) {
- tep_filter_strerror(filter, ret, errstr, sizeof(errstr));
- /* Failed to add a new rule to the filter, the error string is in errstr */
-}
-if (tep_filter_copy(filter1, filter) != 0) {
- /* Failed to copy filter in filter1 */
-}
-...
-if (tep_filter_compare(filter, filter1) != 1) {
- /* Both filters are different */
-}
-...
-void process_record(struct tep_handle *tep, struct tep_record *record)
-{
- struct tep_event *event;
- char *fstring;
-
- event = tep_find_event_by_record(tep, record);
-
- if (tep_event_filtered(filter, event->id) == 1) {
- /* The event has filter */
- fstring = tep_filter_make_string(filter, event->id);
- if (fstring != NULL) {
- /* The filter for the event is in fstring */
- free(fstring);
- }
- }
-
- switch (tep_filter_match(filter, record)) {
- case TEP_ERRNO__FILTER_MATCH:
- /* The filter matches the record */
- break;
- case TEP_ERRNO__FILTER_MISS:
- /* The filter does not match the record */
- break;
- case TEP_ERRNO__FILTER_NOT_FOUND:
- /* No filter found for record's event */
- break;
- case TEP_ERRNO__NO_FILTER:
- /* There are no rules in the filter */
- break
- default:
- /* An error occurred during the test */
- break;
- }
-
- if (tep_filter_remove_event(filter, event->id) == 1) {
- /* The event was removed from the filter */
- }
-}
-
-...
-tep_filter_reset(filter);
-...
-tep_filter_free(filter);
-tep_filter_free(filter1);
-...
---
-
-FILES
------
-[verse]
---
-*event-parse.h*
- Header file to include in order to have access to the library APIs.
-*-ltraceevent*
- Linker switch to add when building a program that uses the library.
---
-
-SEE ALSO
---------
-_libtraceevent(3)_, _trace-cmd(1)_
-
-AUTHOR
-------
-[verse]
---
-*Steven Rostedt* <rostedt@goodmis.org>, author of *libtraceevent*.
-*Tzvetomir Stoyanov* <tz.stoyanov@gmail.com>, author of this man page.
---
-REPORTING BUGS
---------------
-Report bugs to <linux-trace-devel@vger.kernel.org>
-
-LICENSE
--------
-libtraceevent is Free Software licensed under the GNU LGPL 2.1
-
-RESOURCES
----------
-https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
diff --git a/tools/lib/traceevent/Documentation/libtraceevent-func_apis.txt b/tools/lib/traceevent/Documentation/libtraceevent-func_apis.txt
deleted file mode 100644
index f6aca0d..0000000
--- a/tools/lib/traceevent/Documentation/libtraceevent-func_apis.txt
+++ /dev/null
@@ -1,183 +0,0 @@
-libtraceevent(3)
-================
-
-NAME
-----
-tep_find_function, tep_find_function_address, tep_set_function_resolver,
-tep_reset_function_resolver, tep_register_function, tep_register_print_string -
-function related tep APIs
-
-SYNOPSIS
---------
-[verse]
---
-*#include <event-parse.h>*
-
-typedef char pass:[*](*tep_func_resolver_t*)(void pass:[*]_priv_, unsigned long long pass:[*]_addrp_, char pass:[**]_modp_);
-int *tep_set_function_resolver*(struct tep_handle pass:[*]_tep_, tep_func_resolver_t pass:[*]_func_, void pass:[*]_priv_);
-void *tep_reset_function_resolver*(struct tep_handle pass:[*]_tep_);
-const char pass:[*]*tep_find_function*(struct tep_handle pass:[*]_tep_, unsigned long long _addr_);
-unsigned long long *tep_find_function_address*(struct tep_handle pass:[*]_tep_, unsigned long long _addr_);
-int *tep_register_function*(struct tep_handle pass:[*]_tep_, char pass:[*]_name_, unsigned long long _addr_, char pass:[*]_mod_);
-int *tep_register_print_string*(struct tep_handle pass:[*]_tep_, const char pass:[*]_fmt_, unsigned long long _addr_);
---
-
-DESCRIPTION
------------
-Some tools may have already a way to resolve the kernel functions. These APIs
-allow them to keep using it instead of duplicating all the entries inside.
-
-The _tep_func_resolver_t_ type is the prototype of the alternative kernel
-functions resolver. This function receives a pointer to its custom context
-(set with the _tep_set_function_resolver()_ call ) and the address of a kernel
-function, which has to be resolved. In case of success, it should return
-the name of the function and its module (if any) in _modp_.
-
-The _tep_set_function_resolver()_ function registers _func_ as an alternative
-kernel functions resolver. The _tep_ argument is trace event parser context.
-The _priv_ argument is a custom context of the _func_ function. The function
-resolver is used by the APIs _tep_find_function()_,
-_tep_find_function_address()_, and _tep_print_func_field()_ to resolve
-a function address to a function name.
-
-The _tep_reset_function_resolver()_ function resets the kernel functions
-resolver to the default function. The _tep_ argument is trace event parser
-context.
-
-
-These APIs can be used to find function name and start address, by given
-address. The given address does not have to be exact, it will select
-the function that would contain it.
-
-The _tep_find_function()_ function returns the function name, which contains the
-given address _addr_. The _tep_ argument is the trace event parser context.
-
-The _tep_find_function_address()_ function returns the function start address,
-by given address _addr_. The _addr_ does not have to be exact, it will select
-the function that would contain it. The _tep_ argument is the trace event
-parser context.
-
-The _tep_register_function()_ function registers a function name mapped to an
-address and (optional) module. This mapping is used in case the function tracer
-or events have "%pS" parameter in its format string. It is common to pass in
-the kallsyms function names with their corresponding addresses with this
-function. The _tep_ argument is the trace event parser context. The _name_ is
-the name of the function, the string is copied internally. The _addr_ is the
-start address of the function. The _mod_ is the kernel module the function may
-be in (NULL for none).
-
-The _tep_register_print_string()_ function registers a string by the address
-it was stored in the kernel. Some strings internal to the kernel with static
-address are passed to certain events. The "%s" in the event's format field
-which has an address needs to know what string would be at that address. The
-tep_register_print_string() supplies the parsing with the mapping between kernel
-addresses and those strings. The _tep_ argument is the trace event parser
-context. The _fmt_ is the string to register, it is copied internally.
-The _addr_ is the address the string was located at.
-
-
-RETURN VALUE
-------------
-The _tep_set_function_resolver()_ function returns 0 in case of success, or -1
-in case of an error.
-
-The _tep_find_function()_ function returns the function name, or NULL in case
-it cannot be found.
-
-The _tep_find_function_address()_ function returns the function start address,
-or 0 in case it cannot be found.
-
-The _tep_register_function()_ function returns 0 in case of success. In case of
-an error -1 is returned, and errno is set to the appropriate error number.
-
-The _tep_register_print_string()_ function returns 0 in case of success. In case
-of an error -1 is returned, and errno is set to the appropriate error number.
-
-EXAMPLE
--------
-[source,c]
---
-#include <event-parse.h>
-...
-struct tep_handle *tep = tep_alloc();
-...
-char *my_resolve_kernel_addr(void *context,
- unsigned long long *addrp, char **modp)
-{
- struct db *function_database = context;
- struct symbol *sym = sql_lookup(function_database, *addrp);
-
- if (!sym)
- return NULL;
-
- *modp = sym->module_name;
- return sym->name;
-}
-
-void show_function( unsigned long long addr)
-{
- unsigned long long fstart;
- const char *fname;
-
- if (tep_set_function_resolver(tep, my_resolve_kernel_addr,
- function_database) != 0) {
- /* failed to register my_resolve_kernel_addr */
- }
-
- /* These APIs use my_resolve_kernel_addr() to resolve the addr */
- fname = tep_find_function(tep, addr);
- fstart = tep_find_function_address(tep, addr);
-
- /*
- addr is in function named fname, starting at fstart address,
- at offset (addr - fstart)
- */
-
- tep_reset_function_resolver(tep);
-
-}
-...
- if (tep_register_function(tep, "kvm_exit",
- (unsigned long long) 0x12345678, "kvm") != 0) {
- /* Failed to register kvm_exit address mapping */
- }
-...
- if (tep_register_print_string(tep, "print string",
- (unsigned long long) 0x87654321, NULL) != 0) {
- /* Failed to register "print string" address mapping */
- }
-...
---
-
-FILES
------
-[verse]
---
-*event-parse.h*
- Header file to include in order to have access to the library APIs.
-*-ltraceevent*
- Linker switch to add when building a program that uses the library.
---
-
-SEE ALSO
---------
-_libtraceevent(3)_, _trace-cmd(1)_
-
-AUTHOR
-------
-[verse]
---
-*Steven Rostedt* <rostedt@goodmis.org>, author of *libtraceevent*.
-*Tzvetomir Stoyanov* <tz.stoyanov@gmail.com>, author of this man page.
---
-REPORTING BUGS
---------------
-Report bugs to <linux-trace-devel@vger.kernel.org>
-
-LICENSE
--------
-libtraceevent is Free Software licensed under the GNU LGPL 2.1
-
-RESOURCES
----------
-https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
diff --git a/tools/lib/traceevent/Documentation/libtraceevent-func_find.txt b/tools/lib/traceevent/Documentation/libtraceevent-func_find.txt
deleted file mode 100644
index 04840e2..0000000
--- a/tools/lib/traceevent/Documentation/libtraceevent-func_find.txt
+++ /dev/null
@@ -1,88 +0,0 @@
-libtraceevent(3)
-================
-
-NAME
-----
-tep_find_function,tep_find_function_address - Find function name / start address.
-
-SYNOPSIS
---------
-[verse]
---
-*#include <event-parse.h>*
-
-const char pass:[*]*tep_find_function*(struct tep_handle pass:[*]_tep_, unsigned long long _addr_);
-unsigned long long *tep_find_function_address*(struct tep_handle pass:[*]_tep_, unsigned long long _addr_);
---
-
-DESCRIPTION
------------
-These functions can be used to find function name and start address, by given
-address. The given address does not have to be exact, it will select the function
-that would contain it.
-
-The _tep_find_function()_ function returns the function name, which contains the
-given address _addr_. The _tep_ argument is the trace event parser context.
-
-The _tep_find_function_address()_ function returns the function start address,
-by given address _addr_. The _addr_ does not have to be exact, it will select the
-function that would contain it. The _tep_ argument is the trace event parser context.
-
-RETURN VALUE
-------------
-The _tep_find_function()_ function returns the function name, or NULL in case
-it cannot be found.
-
-The _tep_find_function_address()_ function returns the function start address,
-or 0 in case it cannot be found.
-
-EXAMPLE
--------
-[source,c]
---
-#include <event-parse.h>
-...
-struct tep_handle *tep = tep_alloc();
-...
-void show_function( unsigned long long addr)
-{
- const char *fname = tep_find_function(tep, addr);
- unsigned long long fstart = tep_find_function_address(tep, addr);
-
- /* addr is in function named fname, starting at fstart address, at offset (addr - fstart) */
-}
-...
---
-
-FILES
------
-[verse]
---
-*event-parse.h*
- Header file to include in order to have access to the library APIs.
-*-ltraceevent*
- Linker switch to add when building a program that uses the library.
---
-
-SEE ALSO
---------
-_libtraceevent(3)_, _trace-cmd(1)_
-
-AUTHOR
-------
-[verse]
---
-*Steven Rostedt* <rostedt@goodmis.org>, author of *libtraceevent*.
-*Tzvetomir Stoyanov* <tz.stoyanov@gmail.com>, author of this man page.
---
-REPORTING BUGS
---------------
-Report bugs to <linux-trace-devel@vger.kernel.org>
-
-LICENSE
--------
-libtraceevent is Free Software licensed under the GNU LGPL 2.1
-
-RESOURCES
----------
-https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
diff --git a/tools/lib/traceevent/Documentation/libtraceevent-handle.txt b/tools/lib/traceevent/Documentation/libtraceevent-handle.txt
deleted file mode 100644
index 45b20172e..0000000
--- a/tools/lib/traceevent/Documentation/libtraceevent-handle.txt
+++ /dev/null
@@ -1,101 +0,0 @@
-libtraceevent(3)
-================
-
-NAME
-----
-tep_alloc, tep_free,tep_ref, tep_unref,tep_get_ref - Create, destroy, manage
-references of trace event parser context.
-
-SYNOPSIS
---------
-[verse]
---
-*#include <event-parse.h>*
-
-struct tep_handle pass:[*]*tep_alloc*(void);
-void *tep_free*(struct tep_handle pass:[*]_tep_);
-void *tep_ref*(struct tep_handle pass:[*]_tep_);
-void *tep_unref*(struct tep_handle pass:[*]_tep_);
-int *tep_get_ref*(struct tep_handle pass:[*]_tep_);
---
-
-DESCRIPTION
------------
-These are the main functions to create and destroy tep_handle - the main
-structure, representing the trace event parser context. This context is used as
-the input parameter of most library APIs.
-
-The _tep_alloc()_ function allocates and initializes the tep context.
-
-The _tep_free()_ function will decrement the reference of the _tep_ handler.
-When there is no more references, then it will free the handler, as well
-as clean up all its resources that it had used. The argument _tep_ is
-the pointer to the trace event parser context.
-
-The _tep_ref()_ function adds a reference to the _tep_ handler.
-
-The _tep_unref()_ function removes a reference from the _tep_ handler. When
-the last reference is removed, the _tep_ is destroyed, and all resources that
-it had used are cleaned up.
-
-The _tep_ref_get()_ functions gets the current references of the _tep_ handler.
-
-RETURN VALUE
-------------
-_tep_alloc()_ returns a pointer to a newly created tep_handle structure.
-NULL is returned in case there is not enough free memory to allocate it.
-
-_tep_ref_get()_ returns the current references of _tep_.
-If _tep_ is NULL, 0 is returned.
-
-EXAMPLE
--------
-[source,c]
---
-#include <event-parse.h>
-
-...
-struct tep_handle *tep = tep_alloc();
-...
-int ref = tep_get_ref(tep);
-tep_ref(tep);
-if ( (ref+1) != tep_get_ref(tep)) {
- /* Something wrong happened, the counter is not incremented by 1 */
-}
-tep_unref(tep);
-...
-tep_free(tep);
-...
---
-FILES
------
-[verse]
---
-*event-parse.h*
- Header file to include in order to have access to the library APIs.
-*-ltraceevent*
- Linker switch to add when building a program that uses the library.
---
-
-SEE ALSO
---------
-_libtraceevent(3)_, _trace-cmd(1)_
-
-AUTHOR
-------
-[verse]
---
-*Steven Rostedt* <rostedt@goodmis.org>, author of *libtraceevent*.
-*Tzvetomir Stoyanov* <tz.stoyanov@gmail.com>, author of this man page.
---
-REPORTING BUGS
---------------
-Report bugs to <linux-trace-devel@vger.kernel.org>
-
-LICENSE
--------
-libtraceevent is Free Software licensed under the GNU LGPL 2.1
-
-RESOURCES
----------
-https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
diff --git a/tools/lib/traceevent/Documentation/libtraceevent-header_page.txt b/tools/lib/traceevent/Documentation/libtraceevent-header_page.txt
deleted file mode 100644
index 615d117..0000000
--- a/tools/lib/traceevent/Documentation/libtraceevent-header_page.txt
+++ /dev/null
@@ -1,102 +0,0 @@
-libtraceevent(3)
-================
-
-NAME
-----
-tep_get_header_page_size, tep_get_header_timestamp_size, tep_is_old_format -
-Get the data stored in the header page, in kernel context.
-
-SYNOPSIS
---------
-[verse]
---
-*#include <event-parse.h>*
-
-int *tep_get_header_page_size*(struct tep_handle pass:[*]_tep_);
-int *tep_get_header_timestamp_size*(struct tep_handle pass:[*]_tep_);
-bool *tep_is_old_format*(struct tep_handle pass:[*]_tep_);
---
-DESCRIPTION
------------
-These functions retrieve information from kernel context, stored in tracefs
-events/header_page. Old kernels do not have header page info, so default values
-from user space context are used.
-
-The _tep_get_header_page_size()_ function returns the size of a long integer,
-in kernel context. The _tep_ argument is trace event parser context.
-This information is retrieved from tracefs events/header_page, "commit" field.
-
-The _tep_get_header_timestamp_size()_ function returns the size of timestamps,
-in kernel context. The _tep_ argument is trace event parser context. This
-information is retrieved from tracefs events/header_page, "timestamp" field.
-
-The _tep_is_old_format()_ function returns true if the kernel predates
-the addition of events/header_page, otherwise it returns false.
-
-RETURN VALUE
-------------
-The _tep_get_header_page_size()_ function returns the size of a long integer,
-in bytes.
-
-The _tep_get_header_timestamp_size()_ function returns the size of timestamps,
-in bytes.
-
-The _tep_is_old_format()_ function returns true, if an old kernel is used to
-generate the tracing data, which has no event/header_page. If the kernel is new,
-or _tep_ is NULL, false is returned.
-
-EXAMPLE
--------
-[source,c]
---
-#include <event-parse.h>
-...
-struct tep_handle *tep = tep_alloc();
-...
- int longsize;
- int timesize;
- bool old;
-
- longsize = tep_get_header_page_size(tep);
- timesize = tep_get_header_timestamp_size(tep);
- old = tep_is_old_format(tep);
-
- printf ("%s kernel is used to generate the tracing data.\n",
- old?"Old":"New");
- printf("The size of a long integer is %d bytes.\n", longsize);
- printf("The timestamps size is %d bytes.\n", timesize);
-...
---
-
-FILES
------
-[verse]
---
-*event-parse.h*
- Header file to include in order to have access to the library APIs.
-*-ltraceevent*
- Linker switch to add when building a program that uses the library.
---
-
-SEE ALSO
---------
-_libtraceevent(3)_, _trace-cmd(1)_
-
-AUTHOR
-------
-[verse]
---
-*Steven Rostedt* <rostedt@goodmis.org>, author of *libtraceevent*.
-*Tzvetomir Stoyanov* <tz.stoyanov@gmail.com>, author of this man page.
---
-REPORTING BUGS
---------------
-Report bugs to <linux-trace-devel@vger.kernel.org>
-
-LICENSE
--------
-libtraceevent is Free Software licensed under the GNU LGPL 2.1
-
-RESOURCES
----------
-https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
diff --git a/tools/lib/traceevent/Documentation/libtraceevent-host_endian.txt b/tools/lib/traceevent/Documentation/libtraceevent-host_endian.txt
deleted file mode 100644
index d5d375e..0000000
--- a/tools/lib/traceevent/Documentation/libtraceevent-host_endian.txt
+++ /dev/null
@@ -1,104 +0,0 @@
-libtraceevent(3)
-================
-
-NAME
-----
-tep_is_bigendian, tep_is_local_bigendian, tep_set_local_bigendian - Get / set
-the endianness of the local machine.
-
-SYNOPSIS
---------
-[verse]
---
-*#include <event-parse.h>*
-
-enum *tep_endian* {
- TEP_LITTLE_ENDIAN = 0,
- TEP_BIG_ENDIAN
-};
-
-int *tep_is_bigendian*(void);
-bool *tep_is_local_bigendian*(struct tep_handle pass:[*]_tep_);
-void *tep_set_local_bigendian*(struct tep_handle pass:[*]_tep_, enum tep_endian _endian_);
---
-
-DESCRIPTION
------------
-
-The _tep_is_bigendian()_ gets the endianness of the machine, executing
-the function.
-
-The _tep_is_local_bigendian()_ function gets the endianness of the local
-machine, saved in the _tep_ handler. The _tep_ argument is the trace event
-parser context. This API is a bit faster than _tep_is_bigendian()_, as it
-returns cached endianness of the local machine instead of checking it each time.
-
-The _tep_set_local_bigendian()_ function sets the endianness of the local
-machine in the _tep_ handler. The _tep_ argument is trace event parser context.
-The _endian_ argument is the endianness:
-[verse]
---
- _TEP_LITTLE_ENDIAN_ - the machine is little endian,
- _TEP_BIG_ENDIAN_ - the machine is big endian.
---
-
-RETURN VALUE
-------------
-The _tep_is_bigendian()_ function returns non zero if the endianness of the
-machine, executing the code, is big endian and zero otherwise.
-
-The _tep_is_local_bigendian()_ function returns true, if the endianness of the
-local machine, saved in the _tep_ handler, is big endian, or false otherwise.
-
-EXAMPLE
--------
-[source,c]
---
-#include <event-parse.h>
-...
-struct tep_handle *tep = tep_alloc();
-...
- if (tep_is_bigendian())
- tep_set_local_bigendian(tep, TEP_BIG_ENDIAN);
- else
- tep_set_local_bigendian(tep, TEP_LITTLE_ENDIAN);
-...
- if (tep_is_local_bigendian(tep))
- printf("This machine you are running on is bigendian\n");
- else
- printf("This machine you are running on is little endian\n");
-
---
-
-FILES
------
-[verse]
---
-*event-parse.h*
- Header file to include in order to have access to the library APIs.
-*-ltraceevent*
- Linker switch to add when building a program that uses the library.
---
-
-SEE ALSO
---------
-_libtraceevent(3)_, _trace-cmd(1)_
-
-AUTHOR
-------
-[verse]
---
-*Steven Rostedt* <rostedt@goodmis.org>, author of *libtraceevent*.
-*Tzvetomir Stoyanov* <tz.stoyanov@gmail.com>, author of this man page.
---
-REPORTING BUGS
---------------
-Report bugs to <linux-trace-devel@vger.kernel.org>
-
-LICENSE
--------
-libtraceevent is Free Software licensed under the GNU LGPL 2.1
-
-RESOURCES
----------
-https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
diff --git a/tools/lib/traceevent/Documentation/libtraceevent-long_size.txt b/tools/lib/traceevent/Documentation/libtraceevent-long_size.txt
deleted file mode 100644
index 01d78ea..0000000
--- a/tools/lib/traceevent/Documentation/libtraceevent-long_size.txt
+++ /dev/null
@@ -1,78 +0,0 @@
-libtraceevent(3)
-================
-
-NAME
-----
-tep_get_long_size, tep_set_long_size - Get / set the size of a long integer on
-the machine, where the trace is generated, in bytes
-
-SYNOPSIS
---------
-[verse]
---
-*#include <event-parse.h>*
-
-int *tep_get_long_size*(strucqt tep_handle pass:[*]_tep_);
-void *tep_set_long_size*(struct tep_handle pass:[*]_tep_, int _long_size_);
---
-
-DESCRIPTION
------------
-The _tep_get_long_size()_ function returns the size of a long integer on the machine,
-where the trace is generated. The _tep_ argument is trace event parser context.
-
-The _tep_set_long_size()_ function sets the size of a long integer on the machine,
-where the trace is generated. The _tep_ argument is trace event parser context.
-The _long_size_ is the size of a long integer, in bytes.
-
-RETURN VALUE
-------------
-The _tep_get_long_size()_ function returns the size of a long integer on the machine,
-where the trace is generated, in bytes.
-
-EXAMPLE
--------
-[source,c]
---
-#include <event-parse.h>
-...
-struct tep_handle *tep = tep_alloc();
-...
-tep_set_long_size(tep, 4);
-...
-int long_size = tep_get_long_size(tep);
-...
---
-
-FILES
------
-[verse]
---
-*event-parse.h*
- Header file to include in order to have access to the library APIs.
-*-ltraceevent*
- Linker switch to add when building a program that uses the library.
---
-
-SEE ALSO
---------
-_libtraceevent(3)_, _trace-cmd(1)_
-
-AUTHOR
-------
-[verse]
---
-*Steven Rostedt* <rostedt@goodmis.org>, author of *libtraceevent*.
-*Tzvetomir Stoyanov* <tz.stoyanov@gmail.com>, author of this man page.
---
-REPORTING BUGS
---------------
-Report bugs to <linux-trace-devel@vger.kernel.org>
-
-LICENSE
--------
-libtraceevent is Free Software licensed under the GNU LGPL 2.1
-
-RESOURCES
----------
-https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
diff --git a/tools/lib/traceevent/Documentation/libtraceevent-page_size.txt b/tools/lib/traceevent/Documentation/libtraceevent-page_size.txt
deleted file mode 100644
index 452c0cf..0000000
--- a/tools/lib/traceevent/Documentation/libtraceevent-page_size.txt
+++ /dev/null
@@ -1,82 +0,0 @@
-libtraceevent(3)
-================
-
-NAME
-----
-tep_get_page_size, tep_set_page_size - Get / set the size of a memory page on
-the machine, where the trace is generated
-
-SYNOPSIS
---------
-[verse]
---
-*#include <event-parse.h>*
-
-int *tep_get_page_size*(struct tep_handle pass:[*]_tep_);
-void *tep_set_page_size*(struct tep_handle pass:[*]_tep_, int _page_size_);
---
-
-DESCRIPTION
------------
-The _tep_get_page_size()_ function returns the size of a memory page on
-the machine, where the trace is generated. The _tep_ argument is trace
-event parser context.
-
-The _tep_set_page_size()_ function stores in the _tep_ context the size of a
-memory page on the machine, where the trace is generated.
-The _tep_ argument is trace event parser context.
-The _page_size_ argument is the size of a memory page, in bytes.
-
-RETURN VALUE
-------------
-The _tep_get_page_size()_ function returns size of the memory page, in bytes.
-
-EXAMPLE
--------
-[source,c]
---
-#include <unistd.h>
-#include <event-parse.h>
-...
-struct tep_handle *tep = tep_alloc();
-...
- int page_size = getpagesize();
-
- tep_set_page_size(tep, page_size);
-
- printf("The page size for this machine is %d\n", tep_get_page_size(tep));
-
---
-
-FILES
------
-[verse]
---
-*event-parse.h*
- Header file to include in order to have access to the library APIs.
-*-ltraceevent*
- Linker switch to add when building a program that uses the library.
---
-
-SEE ALSO
---------
-_libtraceevent(3)_, _trace-cmd(1)_
-
-AUTHOR
-------
-[verse]
---
-*Steven Rostedt* <rostedt@goodmis.org>, author of *libtraceevent*.
-*Tzvetomir Stoyanov* <tz.stoyanov@gmail.com>, author of this man page.
---
-REPORTING BUGS
---------------
-Report bugs to <linux-trace-devel@vger.kernel.org>
-
-LICENSE
--------
-libtraceevent is Free Software licensed under the GNU LGPL 2.1
-
-RESOURCES
----------
-https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
diff --git a/tools/lib/traceevent/Documentation/libtraceevent-parse_event.txt b/tools/lib/traceevent/Documentation/libtraceevent-parse_event.txt
deleted file mode 100644
index f248114..0000000
--- a/tools/lib/traceevent/Documentation/libtraceevent-parse_event.txt
+++ /dev/null
@@ -1,90 +0,0 @@
-libtraceevent(3)
-================
-
-NAME
-----
-tep_parse_event, tep_parse_format - Parse the event format information
-
-SYNOPSIS
---------
-[verse]
---
-*#include <event-parse.h>*
-
-enum tep_errno *tep_parse_event*(struct tep_handle pass:[*]_tep_, const char pass:[*]_buf_, unsigned long _size_, const char pass:[*]_sys_);
-enum tep_errno *tep_parse_format*(struct tep_handle pass:[*]_tep_, struct tep_event pass:[*]pass:[*]_eventp_, const char pass:[*]_buf_, unsigned long _size_, const char pass:[*]_sys_);
---
-
-DESCRIPTION
------------
-The _tep_parse_event()_ function parses the event format and creates an event
-structure to quickly parse raw data for a given event. The _tep_ argument is
-the trace event parser context. The created event structure is stored in the
-_tep_ context. The _buf_ argument is a buffer with _size_, where the event
-format data is. The event format data can be taken from
-tracefs/events/.../.../format files. The _sys_ argument is the system of
-the event.
-
-The _tep_parse_format()_ function does the same as _tep_parse_event()_. The only
-difference is in the extra _eventp_ argument, where the newly created event
-structure is returned.
-
-RETURN VALUE
-------------
-Both _tep_parse_event()_ and _tep_parse_format()_ functions return 0 on success,
-or TEP_ERRNO__... in case of an error.
-
-EXAMPLE
--------
-[source,c]
---
-#include <event-parse.h>
-...
-struct tep_handle *tep = tep_alloc();
-...
-char *buf;
-int size;
-struct tep_event *event = NULL;
-buf = read_file("/sys/kernel/tracing/events/ftrace/print/format", &size);
-if (tep_parse_event(tep, buf, size, "ftrace") != 0) {
- /* Failed to parse the ftrace print format */
-}
-
-if (tep_parse_format(tep, &event, buf, size, "ftrace") != 0) {
- /* Failed to parse the ftrace print format */
-}
-...
---
-
-FILES
------
-[verse]
---
-*event-parse.h*
- Header file to include in order to have access to the library APIs.
-*-ltraceevent*
- Linker switch to add when building a program that uses the library.
---
-
-SEE ALSO
---------
-_libtraceevent(3)_, _trace-cmd(1)_
-
-AUTHOR
-------
-[verse]
---
-*Steven Rostedt* <rostedt@goodmis.org>, author of *libtraceevent*.
-*Tzvetomir Stoyanov* <tz.stoyanov@gmail.com>, author of this man page.
---
-REPORTING BUGS
---------------
-Report bugs to <linux-trace-devel@vger.kernel.org>
-
-LICENSE
--------
-libtraceevent is Free Software licensed under the GNU LGPL 2.1
-
-RESOURCES
----------
-https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
diff --git a/tools/lib/traceevent/Documentation/libtraceevent-parse_head.txt b/tools/lib/traceevent/Documentation/libtraceevent-parse_head.txt
deleted file mode 100644
index c90f16c..0000000
--- a/tools/lib/traceevent/Documentation/libtraceevent-parse_head.txt
+++ /dev/null
@@ -1,82 +0,0 @@
-libtraceevent(3)
-================
-
-NAME
-----
-tep_parse_header_page - Parses the data stored in the header page.
-
-SYNOPSIS
---------
-[verse]
---
-*#include <event-parse.h>*
-
-int *tep_parse_header_page*(struct tep_handle pass:[*]_tep_, char pass:[*]_buf_, unsigned long _size_, int _long_size_);
---
-
-DESCRIPTION
------------
-The _tep_parse_header_page()_ function parses the header page data from _buf_,
-and initializes the _tep_, trace event parser context, with it. The buffer
-_buf_ is with _size_, and is supposed to be copied from
-tracefs/events/header_page.
-
-Some old kernels do not have header page info, in this case the
-_tep_parse_header_page()_ function can be called with _size_ equal to 0. The
-_tep_ context is initialized with default values. The _long_size_ can be used in
-this use case, to set the size of a long integer to be used.
-
-RETURN VALUE
-------------
-The _tep_parse_header_page()_ function returns 0 in case of success, or -1
-in case of an error.
-
-EXAMPLE
--------
-[source,c]
---
-#include <event-parse.h>
-...
-struct tep_handle *tep = tep_alloc();
-...
-char *buf;
-int size;
-buf = read_file("/sys/kernel/tracing/events/header_page", &size);
-if (tep_parse_header_page(tep, buf, size, sizeof(unsigned long)) != 0) {
- /* Failed to parse the header page */
-}
-...
---
-
-FILES
------
-[verse]
---
-*event-parse.h*
- Header file to include in order to have access to the library APIs.
-*-ltraceevent*
- Linker switch to add when building a program that uses the library.
---
-
-SEE ALSO
---------
-_libtraceevent(3)_, _trace-cmd(1)_
-
-AUTHOR
-------
-[verse]
---
-*Steven Rostedt* <rostedt@goodmis.org>, author of *libtraceevent*.
-*Tzvetomir Stoyanov* <tz.stoyanov@gmail.com>, author of this man page.
---
-REPORTING BUGS
---------------
-Report bugs to <linux-trace-devel@vger.kernel.org>
-
-LICENSE
--------
-libtraceevent is Free Software licensed under the GNU LGPL 2.1
-
-RESOURCES
----------
-https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
diff --git a/tools/lib/traceevent/Documentation/libtraceevent-plugins.txt b/tools/lib/traceevent/Documentation/libtraceevent-plugins.txt
deleted file mode 100644
index 4d63943..0000000
--- a/tools/lib/traceevent/Documentation/libtraceevent-plugins.txt
+++ /dev/null
@@ -1,122 +0,0 @@
-libtraceevent(3)
-================
-
-NAME
-----
-tep_load_plugins, tep_unload_plugins, tep_load_plugins_hook - Load / unload traceevent plugins.
-
-SYNOPSIS
---------
-[verse]
---
-*#include <event-parse.h>*
-
-struct tep_plugin_list pass:[*]*tep_load_plugins*(struct tep_handle pass:[*]_tep_);
-void *tep_unload_plugins*(struct tep_plugin_list pass:[*]_plugin_list_, struct tep_handle pass:[*]_tep_);
-void *tep_load_plugins_hook*(struct tep_handle pass:[*]_tep_, const char pass:[*]_suffix_,
- void (pass:[*]_load_plugin_)(struct tep_handle pass:[*]tep,
- const char pass:[*]path,
- const char pass:[*]name,
- void pass:[*]data),
- void pass:[*]_data_);
---
-
-DESCRIPTION
------------
-The _tep_load_plugins()_ function loads all plugins, located in the plugin
-directories. The _tep_ argument is trace event parser context.
-The plugin directories are :
-[verse]
---
- - Directories, specified in _tep_->plugins_dir with priority TEP_PLUGIN_FIRST
- - System's plugin directory, defined at the library compile time. It
- depends on the library installation prefix and usually is
- _(install_preffix)/lib/traceevent/plugins_
- - Directory, defined by the environment variable _TRACEEVENT_PLUGIN_DIR_
- - User's plugin directory, located at _~/.local/lib/traceevent/plugins_
- - Directories, specified in _tep_->plugins_dir with priority TEP_PLUGIN_LAST
---
-Loading of plugins can be controlled by the _tep_flags_, using the
-_tep_set_flag()_ API:
-[verse]
---
- _TEP_DISABLE_SYS_PLUGINS_ - do not load plugins, located in
- the system's plugin directory.
- _TEP_DISABLE_PLUGINS_ - do not load any plugins.
---
-The _tep_set_flag()_ API needs to be called before _tep_load_plugins()_, if
-loading of all plugins is not the desired case.
-
-The _tep_unload_plugins()_ function unloads the plugins, previously loaded by
-_tep_load_plugins()_. The _tep_ argument is trace event parser context. The
-_plugin_list_ is the list of loaded plugins, returned by
-the _tep_load_plugins()_ function.
-
-The _tep_load_plugins_hook_ function walks through all directories with plugins
-and calls user specified _load_plugin()_ hook for each plugin file. Only files
-with given _suffix_ are considered to be plugins. The _data_ is a user specified
-context, passed to _load_plugin()_. Directories and the walk order are the same
-as in _tep_load_plugins()_ API.
-
-RETURN VALUE
-------------
-The _tep_load_plugins()_ function returns a list of successfully loaded plugins,
-or NULL in case no plugins are loaded.
-
-EXAMPLE
--------
-[source,c]
---
-#include <event-parse.h>
-...
-struct tep_handle *tep = tep_alloc();
-...
-struct tep_plugin_list *plugins = tep_load_plugins(tep);
-if (plugins == NULL) {
- /* no plugins are loaded */
-}
-...
-tep_unload_plugins(plugins, tep);
-...
-void print_plugin(struct tep_handle *tep, const char *path,
- const char *name, void *data)
-{
- pritnf("Found libtraceevent plugin %s/%s\n", path, name);
-}
-...
-tep_load_plugins_hook(tep, ".so", print_plugin, NULL);
-...
---
-
-FILES
------
-[verse]
---
-*event-parse.h*
- Header file to include in order to have access to the library APIs.
-*-ltraceevent*
- Linker switch to add when building a program that uses the library.
---
-
-SEE ALSO
---------
-_libtraceevent(3)_, _trace-cmd(1)_, _tep_set_flag(3)_
-
-AUTHOR
-------
-[verse]
---
-*Steven Rostedt* <rostedt@goodmis.org>, author of *libtraceevent*.
-*Tzvetomir Stoyanov* <tz.stoyanov@gmail.com>, author of this man page.
---
-REPORTING BUGS
---------------
-Report bugs to <linux-trace-devel@vger.kernel.org>
-
-LICENSE
--------
-libtraceevent is Free Software licensed under the GNU LGPL 2.1
-
-RESOURCES
----------
-https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
diff --git a/tools/lib/traceevent/Documentation/libtraceevent-record_parse.txt b/tools/lib/traceevent/Documentation/libtraceevent-record_parse.txt
deleted file mode 100644
index e9a6911..0000000
--- a/tools/lib/traceevent/Documentation/libtraceevent-record_parse.txt
+++ /dev/null
@@ -1,137 +0,0 @@
-libtraceevent(3)
-================
-
-NAME
-----
-tep_data_type, tep_data_pid,tep_data_preempt_count, tep_data_flags -
-Extract common fields from a record.
-
-SYNOPSIS
---------
-[verse]
---
-*#include <event-parse.h>*
-
-enum *trace_flag_type* {
- _TRACE_FLAG_IRQS_OFF_,
- _TRACE_FLAG_IRQS_NOSUPPORT_,
- _TRACE_FLAG_NEED_RESCHED_,
- _TRACE_FLAG_HARDIRQ_,
- _TRACE_FLAG_SOFTIRQ_,
-};
-
-int *tep_data_type*(struct tep_handle pass:[*]_tep_, struct tep_record pass:[*]_rec_);
-int *tep_data_pid*(struct tep_handle pass:[*]_tep_, struct tep_record pass:[*]_rec_);
-int *tep_data_preempt_count*(struct tep_handle pass:[*]_tep_, struct tep_record pass:[*]_rec_);
-int *tep_data_flags*(struct tep_handle pass:[*]_tep_, struct tep_record pass:[*]_rec_);
---
-
-DESCRIPTION
------------
-This set of functions can be used to extract common fields from a record.
-
-The _tep_data_type()_ function gets the event id from the record _rec_.
-It reads the "common_type" field. The _tep_ argument is the trace event parser
-context.
-
-The _tep_data_pid()_ function gets the process id from the record _rec_.
-It reads the "common_pid" field. The _tep_ argument is the trace event parser
-context.
-
-The _tep_data_preempt_count()_ function gets the preemption count from the
-record _rec_. It reads the "common_preempt_count" field. The _tep_ argument is
-the trace event parser context.
-
-The _tep_data_flags()_ function gets the latency flags from the record _rec_.
-It reads the "common_flags" field. The _tep_ argument is the trace event parser
-context. Supported latency flags are:
-[verse]
---
- _TRACE_FLAG_IRQS_OFF_, Interrupts are disabled.
- _TRACE_FLAG_IRQS_NOSUPPORT_, Reading IRQ flag is not supported by the architecture.
- _TRACE_FLAG_NEED_RESCHED_, Task needs rescheduling.
- _TRACE_FLAG_HARDIRQ_, Hard IRQ is running.
- _TRACE_FLAG_SOFTIRQ_, Soft IRQ is running.
---
-
-RETURN VALUE
-------------
-The _tep_data_type()_ function returns an integer, representing the event id.
-
-The _tep_data_pid()_ function returns an integer, representing the process id
-
-The _tep_data_preempt_count()_ function returns an integer, representing the
-preemption count.
-
-The _tep_data_flags()_ function returns an integer, representing the latency
-flags. Look at the _trace_flag_type_ enum for supported flags.
-
-All these functions in case of an error return a negative integer.
-
-EXAMPLE
--------
-[source,c]
---
-#include <event-parse.h>
-...
-struct tep_handle *tep = tep_alloc();
-...
-void process_record(struct tep_record *record)
-{
- int data;
-
- data = tep_data_type(tep, record);
- if (data >= 0) {
- /* Got the ID of the event */
- }
-
- data = tep_data_pid(tep, record);
- if (data >= 0) {
- /* Got the process ID */
- }
-
- data = tep_data_preempt_count(tep, record);
- if (data >= 0) {
- /* Got the preemption count */
- }
-
- data = tep_data_flags(tep, record);
- if (data >= 0) {
- /* Got the latency flags */
- }
-}
-...
---
-
-FILES
------
-[verse]
---
-*event-parse.h*
- Header file to include in order to have access to the library APIs.
-*-ltraceevent*
- Linker switch to add when building a program that uses the library.
---
-
-SEE ALSO
---------
-_libtraceevent(3)_, _trace-cmd(1)_
-
-AUTHOR
-------
-[verse]
---
-*Steven Rostedt* <rostedt@goodmis.org>, author of *libtraceevent*.
-*Tzvetomir Stoyanov* <tz.stoyanov@gmail.com>, author of this man page.
---
-REPORTING BUGS
---------------
-Report bugs to <linux-trace-devel@vger.kernel.org>
-
-LICENSE
--------
-libtraceevent is Free Software licensed under the GNU LGPL 2.1
-
-RESOURCES
----------
-https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
diff --git a/tools/lib/traceevent/Documentation/libtraceevent-reg_event_handler.txt b/tools/lib/traceevent/Documentation/libtraceevent-reg_event_handler.txt
deleted file mode 100644
index 53d37d7..0000000
--- a/tools/lib/traceevent/Documentation/libtraceevent-reg_event_handler.txt
+++ /dev/null
@@ -1,156 +0,0 @@
-libtraceevent(3)
-================
-
-NAME
-----
-tep_register_event_handler, tep_unregister_event_handler - Register /
-unregisters a callback function to parse an event information.
-
-SYNOPSIS
---------
-[verse]
---
-*#include <event-parse.h>*
-
-enum *tep_reg_handler* {
- _TEP_REGISTER_SUCCESS_,
- _TEP_REGISTER_SUCCESS_OVERWRITE_,
-};
-
-int *tep_register_event_handler*(struct tep_handle pass:[*]_tep_, int _id_, const char pass:[*]_sys_name_, const char pass:[*]_event_name_, tep_event_handler_func _func_, void pass:[*]_context_);
-int *tep_unregister_event_handler*(struct tep_handle pass:[*]tep, int id, const char pass:[*]sys_name, const char pass:[*]event_name, tep_event_handler_func func, void pass:[*]_context_);
-
-typedef int (*pass:[*]tep_event_handler_func*)(struct trace_seq pass:[*]s, struct tep_record pass:[*]record, struct tep_event pass:[*]event, void pass:[*]context);
---
-
-DESCRIPTION
------------
-The _tep_register_event_handler()_ function registers a handler function,
-which is going to be called to parse the information for a given event.
-The _tep_ argument is the trace event parser context. The _id_ argument is
-the id of the event. The _sys_name_ argument is the name of the system,
-the event belongs to. The _event_name_ argument is the name of the event.
-If _id_ is >= 0, it is used to find the event, otherwise _sys_name_ and
-_event_name_ are used. The _func_ is a pointer to the function, which is going
-to be called to parse the event information. The _context_ argument is a pointer
-to the context data, which will be passed to the _func_. If a handler function
-for the same event is already registered, it will be overridden with the new
-one. This mechanism allows a developer to override the parsing of a given event.
-If for some reason the default print format is not sufficient, the developer
-can register a function for an event to be used to parse the data instead.
-
-The _tep_unregister_event_handler()_ function unregisters the handler function,
-previously registered with _tep_register_event_handler()_. The _tep_ argument
-is the trace event parser context. The _id_, _sys_name_, _event_name_, _func_,
-and _context_ are the same arguments, as when the callback function _func_ was
-registered.
-
-The _tep_event_handler_func_ is the type of the custom event handler
-function. The _s_ argument is the trace sequence, it can be used to create a
-custom string, describing the event. A _record_ to get the event from is passed
-as input parameter and also the _event_ - the handle to the record's event. The
-_context_ is custom context, set when the custom event handler is registered.
-
-RETURN VALUE
-------------
-The _tep_register_event_handler()_ function returns _TEP_REGISTER_SUCCESS_
-if the new handler is registered successfully or
-_TEP_REGISTER_SUCCESS_OVERWRITE_ if an existing handler is overwritten.
-If there is not enough memory to complete the registration,
-TEP_ERRNO__MEM_ALLOC_FAILED is returned.
-
-The _tep_unregister_event_handler()_ function returns 0 if _func_ was removed
-successful or, -1 if the event was not found.
-
-The _tep_event_handler_func_ should return -1 in case of an error,
-or 0 otherwise.
-
-EXAMPLE
--------
-[source,c]
---
-#include <event-parse.h>
-#include <trace-seq.h>
-...
-struct tep_handle *tep = tep_alloc();
-...
-int timer_expire_handler(struct trace_seq *s, struct tep_record *record,
- struct tep_event *event, void *context)
-{
- trace_seq_printf(s, "hrtimer=");
-
- if (tep_print_num_field(s, "0x%llx", event, "timer", record, 0) == -1)
- tep_print_num_field(s, "0x%llx", event, "hrtimer", record, 1);
-
- trace_seq_printf(s, " now=");
-
- tep_print_num_field(s, "%llu", event, "now", record, 1);
-
- tep_print_func_field(s, " function=%s", event, "function", record, 0);
-
- return 0;
-}
-...
- int ret;
-
- ret = tep_register_event_handler(tep, -1, "timer", "hrtimer_expire_entry",
- timer_expire_handler, NULL);
- if (ret < 0) {
- char buf[32];
-
- tep_strerror(tep, ret, buf, 32)
- printf("Failed to register handler for hrtimer_expire_entry: %s\n", buf);
- } else {
- switch (ret) {
- case TEP_REGISTER_SUCCESS:
- printf ("Registered handler for hrtimer_expire_entry\n");
- break;
- case TEP_REGISTER_SUCCESS_OVERWRITE:
- printf ("Overwrote handler for hrtimer_expire_entry\n");
- break;
- }
- }
-...
- ret = tep_unregister_event_handler(tep, -1, "timer", "hrtimer_expire_entry",
- timer_expire_handler, NULL);
- if ( ret )
- printf ("Failed to unregister handler for hrtimer_expire_entry\n");
-
---
-
-FILES
------
-[verse]
---
-*event-parse.h*
- Header file to include in order to have access to the library APIs.
-*trace-seq.h*
- Header file to include in order to have access to trace sequences
- related APIs. Trace sequences are used to allow a function to call
- several other functions to create a string of data to use.
-*-ltraceevent*
- Linker switch to add when building a program that uses the library.
---
-
-SEE ALSO
---------
-_libtraceevent(3)_, _trace-cmd(1)_
-
-AUTHOR
-------
-[verse]
---
-*Steven Rostedt* <rostedt@goodmis.org>, author of *libtraceevent*.
-*Tzvetomir Stoyanov* <tz.stoyanov@gmail.com>, author of this man page.
---
-REPORTING BUGS
---------------
-Report bugs to <linux-trace-devel@vger.kernel.org>
-
-LICENSE
--------
-libtraceevent is Free Software licensed under the GNU LGPL 2.1
-
-RESOURCES
----------
-https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
diff --git a/tools/lib/traceevent/Documentation/libtraceevent-reg_print_func.txt b/tools/lib/traceevent/Documentation/libtraceevent-reg_print_func.txt
deleted file mode 100644
index 708dce9..0000000
--- a/tools/lib/traceevent/Documentation/libtraceevent-reg_print_func.txt
+++ /dev/null
@@ -1,155 +0,0 @@
-libtraceevent(3)
-================
-
-NAME
-----
-tep_register_print_function,tep_unregister_print_function -
-Registers / Unregisters a helper function.
-
-SYNOPSIS
---------
-[verse]
---
-*#include <event-parse.h>*
-
-enum *tep_func_arg_type* {
- TEP_FUNC_ARG_VOID,
- TEP_FUNC_ARG_INT,
- TEP_FUNC_ARG_LONG,
- TEP_FUNC_ARG_STRING,
- TEP_FUNC_ARG_PTR,
- TEP_FUNC_ARG_MAX_TYPES
-};
-
-typedef unsigned long long (*pass:[*]tep_func_handler*)(struct trace_seq pass:[*]s, unsigned long long pass:[*]args);
-
-int *tep_register_print_function*(struct tep_handle pass:[*]_tep_, tep_func_handler _func_, enum tep_func_arg_type _ret_type_, char pass:[*]_name_, _..._);
-int *tep_unregister_print_function*(struct tep_handle pass:[*]_tep_, tep_func_handler _func_, char pass:[*]_name_);
---
-
-DESCRIPTION
------------
-Some events may have helper functions in the print format arguments.
-This allows a plugin to dynamically create a way to process one of
-these functions.
-
-The _tep_register_print_function()_ registers such helper function. The _tep_
-argument is the trace event parser context. The _func_ argument is a pointer
-to the helper function. The _ret_type_ argument is the return type of the
-helper function, value from the _tep_func_arg_type_ enum. The _name_ is the name
-of the helper function, as seen in the print format arguments. The _..._ is a
-variable list of _tep_func_arg_type_ enums, the _func_ function arguments.
-This list must end with _TEP_FUNC_ARG_VOID_. See 'EXAMPLE' section.
-
-The _tep_unregister_print_function()_ unregisters a helper function, previously
-registered with _tep_register_print_function()_. The _tep_ argument is the
-trace event parser context. The _func_ and _name_ arguments are the same, used
-when the helper function was registered.
-
-The _tep_func_handler_ is the type of the helper function. The _s_ argument is
-the trace sequence, it can be used to create a custom string.
-The _args_ is a list of arguments, defined when the helper function was
-registered.
-
-RETURN VALUE
-------------
-The _tep_register_print_function()_ function returns 0 in case of success.
-In case of an error, TEP_ERRNO_... code is returned.
-
-The _tep_unregister_print_function()_ returns 0 in case of success, or -1 in
-case of an error.
-
-EXAMPLE
--------
-Some events have internal functions calls, that appear in the print format
-output. For example "tracefs/events/i915/g4x_wm/format" has:
-[source,c]
---
-print fmt: "pipe %c, frame=%u, scanline=%u, wm %d/%d/%d, sr %s/%d/%d/%d, hpll %s/%d/%d/%d, fbc %s",
- ((REC->pipe) + 'A'), REC->frame, REC->scanline, REC->primary,
- REC->sprite, REC->cursor, yesno(REC->cxsr), REC->sr_plane,
- REC->sr_cursor, REC->sr_fbc, yesno(REC->hpll), REC->hpll_plane,
- REC->hpll_cursor, REC->hpll_fbc, yesno(REC->fbc)
---
-Notice the call to function _yesno()_ in the print arguments. In the kernel
-context, this function has the following implementation:
-[source,c]
---
-static const char *yesno(int x)
-{
- static const char *yes = "yes";
- static const char *no = "no";
-
- return x ? yes : no;
-}
---
-The user space event parser has no idea how to handle this _yesno()_ function.
-The _tep_register_print_function()_ API can be used to register a user space
-helper function, mapped to the kernel's _yesno()_:
-[source,c]
---
-#include <event-parse.h>
-#include <trace-seq.h>
-...
-struct tep_handle *tep = tep_alloc();
-...
-static const char *yes_no_helper(int x)
-{
- return x ? "yes" : "no";
-}
-...
- if ( tep_register_print_function(tep,
- yes_no_helper,
- TEP_FUNC_ARG_STRING,
- "yesno",
- TEP_FUNC_ARG_INT,
- TEP_FUNC_ARG_VOID) != 0) {
- /* Failed to register yes_no_helper function */
- }
-
-/*
- Now, when the event parser encounters this yesno() function, it will know
- how to handle it.
-*/
-...
- if (tep_unregister_print_function(tep, yes_no_helper, "yesno") != 0) {
- /* Failed to unregister yes_no_helper function */
- }
---
-
-FILES
------
-[verse]
---
-*event-parse.h*
- Header file to include in order to have access to the library APIs.
-*trace-seq.h*
- Header file to include in order to have access to trace sequences
- related APIs. Trace sequences are used to allow a function to call
- several other functions to create a string of data to use.
-*-ltraceevent*
- Linker switch to add when building a program that uses the library.
---
-
-SEE ALSO
---------
-_libtraceevent(3)_, _trace-cmd(1)_
-
-AUTHOR
-------
-[verse]
---
-*Steven Rostedt* <rostedt@goodmis.org>, author of *libtraceevent*.
-*Tzvetomir Stoyanov* <tz.stoyanov@gmail.com>, author of this man page.
---
-REPORTING BUGS
---------------
-Report bugs to <linux-trace-devel@vger.kernel.org>
-
-LICENSE
--------
-libtraceevent is Free Software licensed under the GNU LGPL 2.1
-
-RESOURCES
----------
-https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
diff --git a/tools/lib/traceevent/Documentation/libtraceevent-set_flag.txt b/tools/lib/traceevent/Documentation/libtraceevent-set_flag.txt
deleted file mode 100644
index b059978..0000000
--- a/tools/lib/traceevent/Documentation/libtraceevent-set_flag.txt
+++ /dev/null
@@ -1,104 +0,0 @@
-libtraceevent(3)
-================
-
-NAME
-----
-tep_set_flag, tep_clear_flag, tep_test_flag -
-Manage flags of trace event parser context.
-
-SYNOPSIS
---------
-[verse]
---
-*#include <event-parse.h>*
-
-enum *tep_flag* {
- _TEP_NSEC_OUTPUT_,
- _TEP_DISABLE_SYS_PLUGINS_,
- _TEP_DISABLE_PLUGINS_
-};
-void *tep_set_flag*(struct tep_handle pass:[*]_tep_, enum tep_flag _flag_);
-void *tep_clear_flag*(struct tep_handle pass:[*]_tep_, enum tep_flag _flag_);
-bool *tep_test_flag*(struct tep_handle pass:[*]_tep_, enum tep_flag _flag_);
---
-
-DESCRIPTION
------------
-Trace event parser context flags are defined in *enum tep_flag*:
-[verse]
---
-_TEP_NSEC_OUTPUT_ - print event's timestamp in nano seconds, instead of micro seconds.
-_TEP_DISABLE_SYS_PLUGINS_ - disable plugins, located in system's plugin
- directory. This directory is defined at library compile
- time, and usually depends on library installation
- prefix: (install_preffix)/lib/traceevent/plugins
-_TEP_DISABLE_PLUGINS_ - disable all library plugins:
- - in system's plugin directory
- - in directory, defined by the environment variable _TRACEEVENT_PLUGIN_DIR_
- - in user's home directory, _~/.traceevent/plugins_
---
-Note: plugin related flags must me set before calling _tep_load_plugins()_ API.
-
-The _tep_set_flag()_ function sets _flag_ to _tep_ context.
-
-The _tep_clear_flag()_ function clears _flag_ from _tep_ context.
-
-The _tep_test_flag()_ function tests if _flag_ is set to _tep_ context.
-
-RETURN VALUE
-------------
-_tep_test_flag()_ function returns true if _flag_ is set, false otherwise.
-
-EXAMPLE
--------
-[source,c]
---
-#include <event-parse.h>
-...
-struct tep_handle *tep = tep_alloc();
-...
-/* Print timestamps in nanoseconds */
-tep_set_flag(tep, TEP_NSEC_OUTPUT);
-...
-if (tep_test_flag(tep, TEP_NSEC_OUTPUT)) {
- /* print timestamps in nanoseconds */
-} else {
- /* print timestamps in microseconds */
-}
-...
-/* Print timestamps in microseconds */
-tep_clear_flag(tep, TEP_NSEC_OUTPUT);
-...
---
-FILES
------
-[verse]
---
-*event-parse.h*
- Header file to include in order to have access to the library APIs.
-*-ltraceevent*
- Linker switch to add when building a program that uses the library.
---
-
-SEE ALSO
---------
-_libtraceevent(3)_, _trace-cmd(1)_
-
-AUTHOR
-------
-[verse]
---
-*Steven Rostedt* <rostedt@goodmis.org>, author of *libtraceevent*.
-*Tzvetomir Stoyanov* <tz.stoyanov@gmail.com>, author of this man page.
---
-REPORTING BUGS
---------------
-Report bugs to <linux-trace-devel@vger.kernel.org>
-
-LICENSE
--------
-libtraceevent is Free Software licensed under the GNU LGPL 2.1
-
-RESOURCES
----------
-https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
diff --git a/tools/lib/traceevent/Documentation/libtraceevent-strerror.txt b/tools/lib/traceevent/Documentation/libtraceevent-strerror.txt
deleted file mode 100644
index ee4062a..0000000
--- a/tools/lib/traceevent/Documentation/libtraceevent-strerror.txt
+++ /dev/null
@@ -1,85 +0,0 @@
-libtraceevent(3)
-================
-
-NAME
-----
-tep_strerror - Returns a string describing regular errno and tep error number.
-
-SYNOPSIS
---------
-[verse]
---
-*#include <event-parse.h>*
-
-int *tep_strerror*(struct tep_handle pass:[*]_tep_, enum tep_errno _errnum_, char pass:[*]_buf_, size_t _buflen_);
-
---
-DESCRIPTION
------------
-The _tep_strerror()_ function converts tep error number into a human
-readable string.
-The _tep_ argument is trace event parser context. The _errnum_ is a regular
-errno, defined in errno.h, or a tep error number. The string, describing this
-error number is copied in the _buf_ argument. The _buflen_ argument is
-the size of the _buf_.
-
-It as a thread safe wrapper around strerror_r(). The library function has two
-different behaviors - POSIX and GNU specific. The _tep_strerror()_ API always
-behaves as the POSIX version - the error string is copied in the user supplied
-buffer.
-
-RETURN VALUE
-------------
-The _tep_strerror()_ function returns 0, if a valid _errnum_ is passed and the
-string is copied into _buf_. If _errnum_ is not a valid error number,
--1 is returned and _buf_ is not modified.
-
-EXAMPLE
--------
-[source,c]
---
-#include <event-parse.h>
-...
-struct tep_handle *tep = tep_alloc();
-...
-char buf[32];
-char *pool = calloc(1, 128);
-if (tep == NULL) {
- tep_strerror(tep, TEP_ERRNO__MEM_ALLOC_FAILED, buf, 32);
- printf ("The pool is not initialized, %s", buf);
-}
-...
---
-
-FILES
------
-[verse]
---
-*event-parse.h*
- Header file to include in order to have access to the library APIs.
-*-ltraceevent*
- Linker switch to add when building a program that uses the library.
---
-
-SEE ALSO
---------
-_libtraceevent(3)_, _trace-cmd(1)_
-
-AUTHOR
-------
-[verse]
---
-*Steven Rostedt* <rostedt@goodmis.org>, author of *libtraceevent*.
-*Tzvetomir Stoyanov* <tz.stoyanov@gmail.com>, author of this man page.
---
-REPORTING BUGS
---------------
-Report bugs to <linux-trace-devel@vger.kernel.org>
-
-LICENSE
--------
-libtraceevent is Free Software licensed under the GNU LGPL 2.1
-
-RESOURCES
----------
-https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
diff --git a/tools/lib/traceevent/Documentation/libtraceevent-tseq.txt b/tools/lib/traceevent/Documentation/libtraceevent-tseq.txt
deleted file mode 100644
index 8ac6aa1..0000000
--- a/tools/lib/traceevent/Documentation/libtraceevent-tseq.txt
+++ /dev/null
@@ -1,158 +0,0 @@
-libtraceevent(3)
-================
-
-NAME
-----
-trace_seq_init, trace_seq_destroy, trace_seq_reset, trace_seq_terminate,
-trace_seq_putc, trace_seq_puts, trace_seq_printf, trace_seq_vprintf,
-trace_seq_do_fprintf, trace_seq_do_printf -
-Initialize / destroy a trace sequence.
-
-SYNOPSIS
---------
-[verse]
---
-*#include <event-parse.h>*
-*#include <trace-seq.h>*
-
-void *trace_seq_init*(struct trace_seq pass:[*]_s_);
-void *trace_seq_destroy*(struct trace_seq pass:[*]_s_);
-void *trace_seq_reset*(struct trace_seq pass:[*]_s_);
-void *trace_seq_terminate*(struct trace_seq pass:[*]_s_);
-int *trace_seq_putc*(struct trace_seq pass:[*]_s_, unsigned char _c_);
-int *trace_seq_puts*(struct trace_seq pass:[*]_s_, const char pass:[*]_str_);
-int *trace_seq_printf*(struct trace_seq pass:[*]_s_, const char pass:[*]_fmt_, _..._);
-int *trace_seq_vprintf*(struct trace_seq pass:[*]_s_, const char pass:[*]_fmt_, va_list _args_);
-int *trace_seq_do_printf*(struct trace_seq pass:[*]_s_);
-int *trace_seq_do_fprintf*(struct trace_seq pass:[*]_s_, FILE pass:[*]_fp_);
---
-
-DESCRIPTION
------------
-Trace sequences are used to allow a function to call several other functions
-to create a string of data to use.
-
-The _trace_seq_init()_ function initializes the trace sequence _s_.
-
-The _trace_seq_destroy()_ function destroys the trace sequence _s_ and frees
-all its resources that it had used.
-
-The _trace_seq_reset()_ function re-initializes the trace sequence _s_. All
-characters already written in _s_ will be deleted.
-
-The _trace_seq_terminate()_ function terminates the trace sequence _s_. It puts
-the null character pass:['\0'] at the end of the buffer.
-
-The _trace_seq_putc()_ function puts a single character _c_ in the trace
-sequence _s_.
-
-The _trace_seq_puts()_ function puts a NULL terminated string _str_ in the
-trace sequence _s_.
-
-The _trace_seq_printf()_ function puts a formated string _fmt _with
-variable arguments _..._ in the trace sequence _s_.
-
-The _trace_seq_vprintf()_ function puts a formated string _fmt _with
-list of arguments _args_ in the trace sequence _s_.
-
-The _trace_seq_do_printf()_ function prints the buffer of trace sequence _s_ to
-the standard output stdout.
-
-The _trace_seq_do_fprintf()_ function prints the buffer of trace sequence _s_
-to the given file _fp_.
-
-RETURN VALUE
-------------
-Both _trace_seq_putc()_ and _trace_seq_puts()_ functions return the number of
-characters put in the trace sequence, or 0 in case of an error
-
-Both _trace_seq_printf()_ and _trace_seq_vprintf()_ functions return 0 if the
-trace oversizes the buffer's free space, the number of characters printed, or
-a negative value in case of an error.
-
-Both _trace_seq_do_printf()_ and _trace_seq_do_fprintf()_ functions return the
-number of printed characters, or -1 in case of an error.
-
-EXAMPLE
--------
-[source,c]
---
-#include <event-parse.h>
-#include <trace-seq.h>
-...
-struct trace_seq seq;
-trace_seq_init(&seq);
-...
-void foo_seq_print(struct trace_seq *tseq, char *format, ...)
-{
- va_list ap;
- va_start(ap, format);
- if (trace_seq_vprintf(tseq, format, ap) <= 0) {
- /* Failed to print in the trace sequence */
- }
- va_end(ap);
-}
-
-trace_seq_reset(&seq);
-
-char *str = " MAN page example";
-if (trace_seq_puts(&seq, str) != strlen(str)) {
- /* Failed to put str in the trace sequence */
-}
-if (trace_seq_putc(&seq, ':') != 1) {
- /* Failed to put ':' in the trace sequence */
-}
-if (trace_seq_printf(&seq, " trace sequence: %d", 1) <= 0) {
- /* Failed to print in the trace sequence */
-}
-foo_seq_print( &seq, " %d\n", 2);
-
-trace_seq_terminate(&seq);
-...
-
-if (trace_seq_do_printf(&seq) < 0 ) {
- /* Failed to print the sequence buffer to the standard output */
-}
-FILE *fp = fopen("trace.txt", "w");
-if (trace_seq_do_fprintf(&seq, fp) < 0 ) [
- /* Failed to print the sequence buffer to the trace.txt file */
-}
-
-trace_seq_destroy(&seq);
-...
---
-
-FILES
------
-[verse]
---
-*event-parse.h*
- Header file to include in order to have access to the library APIs.
-*trace-seq.h*
- Header file to include in order to have access to trace sequences related APIs.
-*-ltraceevent*
- Linker switch to add when building a program that uses the library.
---
-
-SEE ALSO
---------
-_libtraceevent(3)_, _trace-cmd(1)_
-
-AUTHOR
-------
-[verse]
---
-*Steven Rostedt* <rostedt@goodmis.org>, author of *libtraceevent*.
-*Tzvetomir Stoyanov* <tz.stoyanov@gmail.com>, author of this man page.
---
-REPORTING BUGS
---------------
-Report bugs to <linux-trace-devel@vger.kernel.org>
-
-LICENSE
--------
-libtraceevent is Free Software licensed under the GNU LGPL 2.1
-
-RESOURCES
----------
-https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
diff --git a/tools/lib/traceevent/Documentation/libtraceevent.txt b/tools/lib/traceevent/Documentation/libtraceevent.txt
deleted file mode 100644
index d530a7c..0000000
--- a/tools/lib/traceevent/Documentation/libtraceevent.txt
+++ /dev/null
@@ -1,192 +0,0 @@
-libtraceevent(3)
-================
-
-NAME
-----
-libtraceevent - Linux kernel trace event library
-
-SYNOPSIS
---------
-[verse]
---
-*#include <event-parse.h>*
-
-Management of tep handler data structure and access of its members:
- struct tep_handle pass:[*]*tep_alloc*(void);
- void *tep_free*(struct tep_handle pass:[*]_tep_);
- void *tep_ref*(struct tep_handle pass:[*]_tep_);
- void *tep_unref*(struct tep_handle pass:[*]_tep_);
- int *tep_get_ref*(struct tep_handle pass:[*]_tep_);
- void *tep_set_flag*(struct tep_handle pass:[*]_tep_, enum tep_flag _flag_);
- void *tep_clear_flag*(struct tep_handle pass:[*]_tep_, enum tep_flag _flag_);
- bool *tep_test_flag*(struct tep_handle pass:[*]_tep_, enum tep_flag _flags_);
- int *tep_get_cpus*(struct tep_handle pass:[*]_tep_);
- void *tep_set_cpus*(struct tep_handle pass:[*]_tep_, int _cpus_);
- int *tep_get_long_size*(strucqt tep_handle pass:[*]_tep_);
- void *tep_set_long_size*(struct tep_handle pass:[*]_tep_, int _long_size_);
- int *tep_get_page_size*(struct tep_handle pass:[*]_tep_);
- void *tep_set_page_size*(struct tep_handle pass:[*]_tep_, int _page_size_);
- int *tep_get_header_page_size*(struct tep_handle pass:[*]_tep_);
- int *tep_get_header_timestamp_size*(struct tep_handle pass:[*]_tep_);
- bool *tep_is_old_format*(struct tep_handle pass:[*]_tep_);
- int *tep_strerror*(struct tep_handle pass:[*]_tep_, enum tep_errno _errnum_, char pass:[*]_buf_, size_t _buflen_);
-
-Register / unregister APIs:
- int *tep_register_function*(struct tep_handle pass:[*]_tep_, char pass:[*]_name_, unsigned long long _addr_, char pass:[*]_mod_);
- int *tep_register_event_handler*(struct tep_handle pass:[*]_tep_, int _id_, const char pass:[*]_sys_name_, const char pass:[*]_event_name_, tep_event_handler_func _func_, void pass:[*]_context_);
- int *tep_unregister_event_handler*(struct tep_handle pass:[*]tep, int id, const char pass:[*]sys_name, const char pass:[*]event_name, tep_event_handler_func func, void pass:[*]_context_);
- int *tep_register_print_string*(struct tep_handle pass:[*]_tep_, const char pass:[*]_fmt_, unsigned long long _addr_);
- int *tep_register_print_function*(struct tep_handle pass:[*]_tep_, tep_func_handler _func_, enum tep_func_arg_type _ret_type_, char pass:[*]_name_, _..._);
- int *tep_unregister_print_function*(struct tep_handle pass:[*]_tep_, tep_func_handler _func_, char pass:[*]_name_);
-
-Plugins management:
- struct tep_plugin_list pass:[*]*tep_load_plugins*(struct tep_handle pass:[*]_tep_);
- void *tep_unload_plugins*(struct tep_plugin_list pass:[*]_plugin_list_, struct tep_handle pass:[*]_tep_);
- char pass:[*]pass:[*]*tep_plugin_list_options*(void);
- void *tep_plugin_free_options_list*(char pass:[*]pass:[*]_list_);
- int *tep_plugin_add_options*(const char pass:[*]_name_, struct tep_plugin_option pass:[*]_options_);
- void *tep_plugin_remove_options*(struct tep_plugin_option pass:[*]_options_);
- void *tep_print_plugins*(struct trace_seq pass:[*]_s_, const char pass:[*]_prefix_, const char pass:[*]_suffix_, const struct tep_plugin_list pass:[*]_list_);
-
-Event related APIs:
- struct tep_event pass:[*]*tep_get_event*(struct tep_handle pass:[*]_tep_, int _index_);
- struct tep_event pass:[*]*tep_get_first_event*(struct tep_handle pass:[*]_tep_);
- int *tep_get_events_count*(struct tep_handle pass:[*]_tep_);
- struct tep_event pass:[*]pass:[*]*tep_list_events*(struct tep_handle pass:[*]_tep_, enum tep_event_sort_type _sort_type_);
- struct tep_event pass:[*]pass:[*]*tep_list_events_copy*(struct tep_handle pass:[*]_tep_, enum tep_event_sort_type _sort_type_);
- void *tep_print_event*(struct tep_handle pass:[*]_tep_, struct trace_seq pass:[*]_s_, struct tep_record pass:[*]_record_, const char pass:[*]_fmt_, _..._);
-
-Event finding:
- struct tep_event pass:[*]*tep_find_event*(struct tep_handle pass:[*]_tep_, int _id_);
- struct tep_event pass:[*]*tep_find_event_by_name*(struct tep_handle pass:[*]_tep_, const char pass:[*]_sys_, const char pass:[*]_name_);
- struct tep_event pass:[*]*tep_find_event_by_record*(struct tep_handle pass:[*]_tep_, struct tep_record pass:[*]_record_);
-
-Parsing of event files:
- int *tep_parse_header_page*(struct tep_handle pass:[*]_tep_, char pass:[*]_buf_, unsigned long _size_, int _long_size_);
- enum tep_errno *tep_parse_event*(struct tep_handle pass:[*]_tep_, const char pass:[*]_buf_, unsigned long _size_, const char pass:[*]_sys_);
- enum tep_errno *tep_parse_format*(struct tep_handle pass:[*]_tep_, struct tep_event pass:[*]pass:[*]_eventp_, const char pass:[*]_buf_, unsigned long _size_, const char pass:[*]_sys_);
-
-APIs related to fields from event's format files:
- struct tep_format_field pass:[*]pass:[*]*tep_event_common_fields*(struct tep_event pass:[*]_event_);
- struct tep_format_field pass:[*]pass:[*]*tep_event_fields*(struct tep_event pass:[*]_event_);
- void pass:[*]*tep_get_field_raw*(struct trace_seq pass:[*]_s_, struct tep_event pass:[*]_event_, const char pass:[*]_name_, struct tep_record pass:[*]_record_, int pass:[*]_len_, int _err_);
- int *tep_get_field_val*(struct trace_seq pass:[*]_s_, struct tep_event pass:[*]_event_, const char pass:[*]_name_, struct tep_record pass:[*]_record_, unsigned long long pass:[*]_val_, int _err_);
- int *tep_get_common_field_val*(struct trace_seq pass:[*]_s_, struct tep_event pass:[*]_event_, const char pass:[*]_name_, struct tep_record pass:[*]_record_, unsigned long long pass:[*]_val_, int _err_);
- int *tep_get_any_field_val*(struct trace_seq pass:[*]_s_, struct tep_event pass:[*]_event_, const char pass:[*]_name_, struct tep_record pass:[*]_record_, unsigned long long pass:[*]_val_, int _err_);
- int *tep_read_number_field*(struct tep_format_field pass:[*]_field_, const void pass:[*]_data_, unsigned long long pass:[*]_value_);
-
-Event fields printing:
- void *tep_print_field*(struct trace_seq pass:[*]_s_, void pass:[*]_data_, struct tep_format_field pass:[*]_field_);
- void *tep_print_fields*(struct trace_seq pass:[*]_s_, void pass:[*]_data_, int _size_, struct tep_event pass:[*]_event_);
- int *tep_print_num_field*(struct trace_seq pass:[*]_s_, const char pass:[*]_fmt_, struct tep_event pass:[*]_event_, const char pass:[*]_name_, struct tep_record pass:[*]_record_, int _err_);
- int *tep_print_func_field*(struct trace_seq pass:[*]_s_, const char pass:[*]_fmt_, struct tep_event pass:[*]_event_, const char pass:[*]_name_, struct tep_record pass:[*]_record_, int _err_);
-
-Event fields finding:
- struct tep_format_field pass:[*]*tep_find_common_field*(struct tep_event pass:[*]_event_, const char pass:[*]_name_);
- struct tep_format_field pass:[*]*tep_find_field*(struct tep_event_ormat pass:[*]_event_, const char pass:[*]_name_);
- struct tep_format_field pass:[*]*tep_find_any_field*(struct tep_event pass:[*]_event_, const char pass:[*]_name_);
-
-Functions resolver:
- int *tep_set_function_resolver*(struct tep_handle pass:[*]_tep_, tep_func_resolver_t pass:[*]_func_, void pass:[*]_priv_);
- void *tep_reset_function_resolver*(struct tep_handle pass:[*]_tep_);
- const char pass:[*]*tep_find_function*(struct tep_handle pass:[*]_tep_, unsigned long long _addr_);
- unsigned long long *tep_find_function_address*(struct tep_handle pass:[*]_tep_, unsigned long long _addr_);
-
-Filter management:
- struct tep_event_filter pass:[*]*tep_filter_alloc*(struct tep_handle pass:[*]_tep_);
- enum tep_errno *tep_filter_add_filter_str*(struct tep_event_filter pass:[*]_filter_, const char pass:[*]_filter_str_);
- enum tep_errno *tep_filter_match*(struct tep_event_filter pass:[*]_filter_, struct tep_record pass:[*]_record_);
- int *tep_filter_strerror*(struct tep_event_filter pass:[*]_filter_, enum tep_errno _err_, char pass:[*]buf, size_t _buflen_);
- int *tep_event_filtered*(struct tep_event_filter pass:[*]_filter_, int _event_id_);
- void *tep_filter_reset*(struct tep_event_filter pass:[*]_filter_);
- void *tep_filter_free*(struct tep_event_filter pass:[*]_filter_);
- char pass:[*]*tep_filter_make_string*(struct tep_event_filter pass:[*]_filter_, int _event_id_);
- int *tep_filter_remove_event*(struct tep_event_filter pass:[*]_filter_, int _event_id_);
- int *tep_filter_copy*(struct tep_event_filter pass:[*]_dest_, struct tep_event_filter pass:[*]_source_);
- int *tep_filter_compare*(struct tep_event_filter pass:[*]_filter1_, struct tep_event_filter pass:[*]_filter2_);
-
-Parsing various data from the records:
- int *tep_data_type*(struct tep_handle pass:[*]_tep_, struct tep_record pass:[*]_rec_);
- int *tep_data_pid*(struct tep_handle pass:[*]_tep_, struct tep_record pass:[*]_rec_);
- int *tep_data_preempt_count*(struct tep_handle pass:[*]_tep_, struct tep_record pass:[*]_rec_);
- int *tep_data_flags*(struct tep_handle pass:[*]_tep_, struct tep_record pass:[*]_rec_);
-
-Command and task related APIs:
- const char pass:[*]*tep_data_comm_from_pid*(struct tep_handle pass:[*]_tep_, int _pid_);
- struct cmdline pass:[*]*tep_data_pid_from_comm*(struct tep_handle pass:[*]_tep_, const char pass:[*]_comm_, struct cmdline pass:[*]_next_);
- int *tep_register_comm*(struct tep_handle pass:[*]_tep_, const char pass:[*]_comm_, int _pid_);
- int *tep_override_comm*(struct tep_handle pass:[*]_tep_, const char pass:[*]_comm_, int _pid_);
- bool *tep_is_pid_registered*(struct tep_handle pass:[*]_tep_, int _pid_);
- int *tep_cmdline_pid*(struct tep_handle pass:[*]_tep_, struct cmdline pass:[*]_cmdline_);
-
-Endian related APIs:
- int *tep_is_bigendian*(void);
- unsigned long long *tep_read_number*(struct tep_handle pass:[*]_tep_, const void pass:[*]_ptr_, int _size_);
- bool *tep_is_file_bigendian*(struct tep_handle pass:[*]_tep_);
- void *tep_set_file_bigendian*(struct tep_handle pass:[*]_tep_, enum tep_endian _endian_);
- bool *tep_is_local_bigendian*(struct tep_handle pass:[*]_tep_);
- void *tep_set_local_bigendian*(struct tep_handle pass:[*]_tep_, enum tep_endian _endian_);
-
-Trace sequences:
-*#include <trace-seq.h>*
- void *trace_seq_init*(struct trace_seq pass:[*]_s_);
- void *trace_seq_reset*(struct trace_seq pass:[*]_s_);
- void *trace_seq_destroy*(struct trace_seq pass:[*]_s_);
- int *trace_seq_printf*(struct trace_seq pass:[*]_s_, const char pass:[*]_fmt_, ...);
- int *trace_seq_vprintf*(struct trace_seq pass:[*]_s_, const char pass:[*]_fmt_, va_list _args_);
- int *trace_seq_puts*(struct trace_seq pass:[*]_s_, const char pass:[*]_str_);
- int *trace_seq_putc*(struct trace_seq pass:[*]_s_, unsigned char _c_);
- void *trace_seq_terminate*(struct trace_seq pass:[*]_s_);
- int *trace_seq_do_fprintf*(struct trace_seq pass:[*]_s_, FILE pass:[*]_fp_);
- int *trace_seq_do_printf*(struct trace_seq pass:[*]_s_);
---
-
-DESCRIPTION
------------
-The libtraceevent(3) library provides APIs to access kernel tracepoint events,
-located in the tracefs file system under the events directory.
-
-ENVIRONMENT
------------
-[verse]
---
-TRACEEVENT_PLUGIN_DIR
- Additional plugin directory. All shared object files, located in this directory will be loaded as traceevent plugins.
---
-
-FILES
------
-[verse]
---
-*event-parse.h*
- Header file to include in order to have access to the library APIs.
-*trace-seq.h*
- Header file to include in order to have access to trace sequences related APIs.
- Trace sequences are used to allow a function to call several other functions
- to create a string of data to use.
-*-ltraceevent*
- Linker switch to add when building a program that uses the library.
---
-
-SEE ALSO
---------
-_trace-cmd(1)_
-
-AUTHOR
-------
-[verse]
---
-*Steven Rostedt* <rostedt@goodmis.org>, author of *libtraceevent*.
-*Tzvetomir Stoyanov* <tz.stoyanov@gmail.com>, author of this man page.
---
-REPORTING BUGS
---------------
-Report bugs to <linux-trace-devel@vger.kernel.org>
-
-LICENSE
--------
-libtraceevent is Free Software licensed under the GNU LGPL 2.1
-
-RESOURCES
----------
-https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
diff --git a/tools/lib/traceevent/Documentation/manpage-1.72.xsl b/tools/lib/traceevent/Documentation/manpage-1.72.xsl
deleted file mode 100644
index b4d315c..0000000
--- a/tools/lib/traceevent/Documentation/manpage-1.72.xsl
+++ /dev/null
@@ -1,14 +0,0 @@
-<!-- manpage-1.72.xsl:
- special settings for manpages rendered from asciidoc+docbook
- handles peculiarities in docbook-xsl 1.72.0 -->
-<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
- version="1.0">
-
-<xsl:import href="manpage-base.xsl"/>
-
-<!-- these are the special values for the roff control characters
- needed for docbook-xsl 1.72.0 -->
-<xsl:param name="git.docbook.backslash">▓</xsl:param>
-<xsl:param name="git.docbook.dot" >⌂</xsl:param>
-
-</xsl:stylesheet>
diff --git a/tools/lib/traceevent/Documentation/manpage-base.xsl b/tools/lib/traceevent/Documentation/manpage-base.xsl
deleted file mode 100644
index a264fa61..0000000
--- a/tools/lib/traceevent/Documentation/manpage-base.xsl
+++ /dev/null
@@ -1,35 +0,0 @@
-<!-- manpage-base.xsl:
- special formatting for manpages rendered from asciidoc+docbook -->
-<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
- version="1.0">
-
-<!-- these params silence some output from xmlto -->
-<xsl:param name="man.output.quietly" select="1"/>
-<xsl:param name="refentry.meta.get.quietly" select="1"/>
-
-<!-- convert asciidoc callouts to man page format;
- git.docbook.backslash and git.docbook.dot params
- must be supplied by another XSL file or other means -->
-<xsl:template match="co">
- <xsl:value-of select="concat(
- $git.docbook.backslash,'fB(',
- substring-after(@id,'-'),')',
- $git.docbook.backslash,'fR')"/>
-</xsl:template>
-<xsl:template match="calloutlist">
- <xsl:value-of select="$git.docbook.dot"/>
- <xsl:text>sp </xsl:text>
- <xsl:apply-templates/>
- <xsl:text> </xsl:text>
-</xsl:template>
-<xsl:template match="callout">
- <xsl:value-of select="concat(
- $git.docbook.backslash,'fB',
- substring-after(@arearefs,'-'),
- '. ',$git.docbook.backslash,'fR')"/>
- <xsl:apply-templates/>
- <xsl:value-of select="$git.docbook.dot"/>
- <xsl:text>br </xsl:text>
-</xsl:template>
-
-</xsl:stylesheet>
diff --git a/tools/lib/traceevent/Documentation/manpage-bold-literal.xsl b/tools/lib/traceevent/Documentation/manpage-bold-literal.xsl
deleted file mode 100644
index 608eb5d..0000000
--- a/tools/lib/traceevent/Documentation/manpage-bold-literal.xsl
+++ /dev/null
@@ -1,17 +0,0 @@
-<!-- manpage-bold-literal.xsl:
- special formatting for manpages rendered from asciidoc+docbook -->
-<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
- version="1.0">
-
-<!-- render literal text as bold (instead of plain or monospace);
- this makes literal text easier to distinguish in manpages
- viewed on a tty -->
-<xsl:template match="literal">
- <xsl:value-of select="$git.docbook.backslash"/>
- <xsl:text>fB</xsl:text>
- <xsl:apply-templates/>
- <xsl:value-of select="$git.docbook.backslash"/>
- <xsl:text>fR</xsl:text>
-</xsl:template>
-
-</xsl:stylesheet>
diff --git a/tools/lib/traceevent/Documentation/manpage-normal.xsl b/tools/lib/traceevent/Documentation/manpage-normal.xsl
deleted file mode 100644
index a48f5b1..0000000
--- a/tools/lib/traceevent/Documentation/manpage-normal.xsl
+++ /dev/null
@@ -1,13 +0,0 @@
-<!-- manpage-normal.xsl:
- special settings for manpages rendered from asciidoc+docbook
- handles anything we want to keep away from docbook-xsl 1.72.0 -->
-<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
- version="1.0">
-
-<xsl:import href="manpage-base.xsl"/>
-
-<!-- these are the normal values for the roff control characters -->
-<xsl:param name="git.docbook.backslash">\</xsl:param>
-<xsl:param name="git.docbook.dot" >.</xsl:param>
-
-</xsl:stylesheet>
diff --git a/tools/lib/traceevent/Documentation/manpage-suppress-sp.xsl b/tools/lib/traceevent/Documentation/manpage-suppress-sp.xsl
deleted file mode 100644
index a63c763..0000000
--- a/tools/lib/traceevent/Documentation/manpage-suppress-sp.xsl
+++ /dev/null
@@ -1,21 +0,0 @@
-<!-- manpage-suppress-sp.xsl:
- special settings for manpages rendered from asciidoc+docbook
- handles erroneous, inline .sp in manpage output of some
- versions of docbook-xsl -->
-<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
- version="1.0">
-
-<!-- attempt to work around spurious .sp at the tail of the line
- that some versions of docbook stylesheets seem to add -->
-<xsl:template match="simpara">
- <xsl:variable name="content">
- <xsl:apply-templates/>
- </xsl:variable>
- <xsl:value-of select="normalize-space($content)"/>
- <xsl:if test="not(ancestor::authorblurb) and
- not(ancestor::personblurb)">
- <xsl:text> </xsl:text>
- </xsl:if>
-</xsl:template>
-
-</xsl:stylesheet>
diff --git a/tools/lib/traceevent/Makefile b/tools/lib/traceevent/Makefile
deleted file mode 100644
index c874c01..0000000
--- a/tools/lib/traceevent/Makefile
+++ /dev/null
@@ -1,300 +0,0 @@
-# SPDX-License-Identifier: GPL-2.0
-# trace-cmd version
-EP_VERSION = 1
-EP_PATCHLEVEL = 1
-EP_EXTRAVERSION = 0
-
-# file format version
-FILE_VERSION = 6
-
-MAKEFLAGS += --no-print-directory
-
-
-# Makefiles suck: This macro sets a default value of $(2) for the
-# variable named by $(1), unless the variable has been set by
-# environment or command line. This is necessary for CC and AR
-# because make sets default values, so the simpler ?= approach
-# won't work as expected.
-define allow-override
- $(if $(or $(findstring environment,$(origin $(1))),\
- $(findstring command line,$(origin $(1)))),,\
- $(eval $(1) = $(2)))
-endef
-
-# Allow setting CC and AR, or setting CROSS_COMPILE as a prefix.
-$(call allow-override,CC,$(CROSS_COMPILE)gcc)
-$(call allow-override,AR,$(CROSS_COMPILE)ar)
-$(call allow-override,NM,$(CROSS_COMPILE)nm)
-$(call allow-override,PKG_CONFIG,pkg-config)
-
-EXT = -std=gnu99
-INSTALL = install
-
-# Use DESTDIR for installing into a different root directory.
-# This is useful for building a package. The program will be
-# installed in this directory as if it was the root directory.
-# Then the build tool can move it later.
-DESTDIR ?=
-DESTDIR_SQ = '$(subst ','\'',$(DESTDIR))'
-
-LP64 := $(shell echo __LP64__ | ${CC} ${CFLAGS} -E -x c - | tail -n 1)
-ifeq ($(LP64), 1)
- libdir_relative_temp = lib64
-else
- libdir_relative_temp = lib
-endif
-
-libdir_relative ?= $(libdir_relative_temp)
-prefix ?= /usr/local
-libdir = $(prefix)/$(libdir_relative)
-man_dir = $(prefix)/share/man
-man_dir_SQ = '$(subst ','\'',$(man_dir))'
-pkgconfig_dir ?= $(word 1,$(shell $(PKG_CONFIG) \
- --variable pc_path pkg-config | tr ":" " "))
-includedir_relative = traceevent
-includedir = $(prefix)/include/$(includedir_relative)
-includedir_SQ = '$(subst ','\'',$(includedir))'
-
-export man_dir man_dir_SQ INSTALL
-export DESTDIR DESTDIR_SQ
-export EVENT_PARSE_VERSION
-
-include ../../scripts/Makefile.include
-
-# copy a bit from Linux kbuild
-
-ifeq ("$(origin V)", "command line")
- VERBOSE = $(V)
-endif
-ifndef VERBOSE
- VERBOSE = 0
-endif
-
-ifeq ($(srctree),)
-srctree := $(patsubst %/,%,$(dir $(CURDIR)))
-srctree := $(patsubst %/,%,$(dir $(srctree)))
-srctree := $(patsubst %/,%,$(dir $(srctree)))
-#$(info Determined 'srctree' to be $(srctree))
-endif
-
-export prefix libdir src obj
-
-# Shell quotes
-libdir_SQ = $(subst ','\'',$(libdir))
-libdir_relative_SQ = $(subst ','\'',$(libdir_relative))
-
-CONFIG_INCLUDES =
-CONFIG_LIBS =
-CONFIG_FLAGS =
-
-VERSION = $(EP_VERSION)
-PATCHLEVEL = $(EP_PATCHLEVEL)
-EXTRAVERSION = $(EP_EXTRAVERSION)
-
-OBJ = $@
-N =
-
-EVENT_PARSE_VERSION = $(EP_VERSION).$(EP_PATCHLEVEL).$(EP_EXTRAVERSION)
-
-LIB_TARGET = libtraceevent.a libtraceevent.so.$(EVENT_PARSE_VERSION)
-LIB_INSTALL = libtraceevent.a libtraceevent.so*
-LIB_INSTALL := $(addprefix $(OUTPUT),$(LIB_INSTALL))
-
-INCLUDES = -I. -I $(srctree)/tools/include $(CONFIG_INCLUDES)
-
-# Set compile option CFLAGS
-ifdef EXTRA_CFLAGS
- CFLAGS := $(EXTRA_CFLAGS)
-else
- CFLAGS := -g -Wall
-endif
-
-# Append required CFLAGS
-override CFLAGS += -fPIC
-override CFLAGS += $(CONFIG_FLAGS) $(INCLUDES) $(PLUGIN_DIR_SQ)
-override CFLAGS += $(udis86-flags) -D_GNU_SOURCE
-
-ifeq ($(VERBOSE),1)
- Q =
-else
- Q = @
-endif
-
-# Disable command line variables (CFLAGS) override from top
-# level Makefile (perf), otherwise build Makefile will get
-# the same command line setup.
-MAKEOVERRIDES=
-
-export srctree OUTPUT CC LD CFLAGS V
-build := -f $(srctree)/tools/build/Makefile.build dir=. obj
-
-TE_IN := $(OUTPUT)libtraceevent-in.o
-LIB_TARGET := $(addprefix $(OUTPUT),$(LIB_TARGET))
-
-CMD_TARGETS = $(LIB_TARGET)
-
-TARGETS = $(CMD_TARGETS)
-
-all: all_cmd plugins
-
-all_cmd: $(CMD_TARGETS)
-
-$(TE_IN): force
- $(Q)$(MAKE) $(build)=libtraceevent
-
-$(OUTPUT)libtraceevent.so.$(EVENT_PARSE_VERSION): $(TE_IN)
- $(QUIET_LINK)$(CC) --shared $(LDFLAGS) $^ -Wl,-soname,libtraceevent.so.$(EP_VERSION) -o $@
- @ln -sf $(@F) $(OUTPUT)libtraceevent.so
- @ln -sf $(@F) $(OUTPUT)libtraceevent.so.$(EP_VERSION)
-
-$(OUTPUT)libtraceevent.a: $(TE_IN)
- $(QUIET_LINK)$(RM) $@; $(AR) rcs $@ $^
-
-$(OUTPUT)%.so: $(OUTPUT)%-in.o
- $(QUIET_LINK)$(CC) $(CFLAGS) -shared $(LDFLAGS) -nostartfiles -o $@ $^
-
-define make_version.h
- (echo '/* This file is automatically generated. Do not modify. */'; \
- echo \#define VERSION_CODE $(shell \
- expr $(VERSION) \* 256 + $(PATCHLEVEL)); \
- echo '#define EXTRAVERSION ' $(EXTRAVERSION); \
- echo '#define VERSION_STRING "'$(VERSION).$(PATCHLEVEL).$(EXTRAVERSION)'"'; \
- echo '#define FILE_VERSION '$(FILE_VERSION); \
- ) > $1
-endef
-
-define update_version.h
- ($(call make_version.h, $@.tmp); \
- if [ -r $@ ] && cmp -s $@ $@.tmp; then \
- rm -f $@.tmp; \
- else \
- echo ' UPDATE $@'; \
- mv -f $@.tmp $@; \
- fi);
-endef
-
-ep_version.h: force
- $(Q)$(N)$(call update_version.h)
-
-VERSION_FILES = ep_version.h
-
-define update_dir
- (echo $1 > $@.tmp; \
- if [ -r $@ ] && cmp -s $@ $@.tmp; then \
- rm -f $@.tmp; \
- else \
- echo ' UPDATE $@'; \
- mv -f $@.tmp $@; \
- fi);
-endef
-
-tags: force
- $(RM) tags
- find . -name '*.[ch]' | xargs ctags --extra=+f --c-kinds=+px \
- --regex-c++='/_PE\(([^,)]*).*/TEP_ERRNO__\1/'
-
-TAGS: force
- $(RM) TAGS
- find . -name '*.[ch]' | xargs etags \
- --regex='/_PE(\([^,)]*\).*/TEP_ERRNO__\1/'
-
-define do_install_mkdir
- if [ ! -d '$(DESTDIR_SQ)$1' ]; then \
- $(INSTALL) -d -m 755 '$(DESTDIR_SQ)$1'; \
- fi
-endef
-
-define do_install
- $(call do_install_mkdir,$2); \
- $(INSTALL) $(if $3,-m $3,) $1 '$(DESTDIR_SQ)$2'
-endef
-
-PKG_CONFIG_SOURCE_FILE = libtraceevent.pc
-PKG_CONFIG_FILE := $(addprefix $(OUTPUT),$(PKG_CONFIG_SOURCE_FILE))
-define do_install_pkgconfig_file
- if [ -n "${pkgconfig_dir}" ]; then \
- cp -f ${PKG_CONFIG_SOURCE_FILE}.template ${PKG_CONFIG_FILE}; \
- sed -i "s|INSTALL_PREFIX|${1}|g" ${PKG_CONFIG_FILE}; \
- sed -i "s|LIB_VERSION|${EVENT_PARSE_VERSION}|g" ${PKG_CONFIG_FILE}; \
- sed -i "s|LIB_DIR|${libdir}|g" ${PKG_CONFIG_FILE}; \
- sed -i "s|HEADER_DIR|$(includedir)|g" ${PKG_CONFIG_FILE}; \
- $(call do_install,$(PKG_CONFIG_FILE),$(pkgconfig_dir),644); \
- else \
- (echo Failed to locate pkg-config directory) 1>&2; \
- fi
-endef
-
-install_lib: all_cmd install_plugins install_headers install_pkgconfig
- $(call QUIET_INSTALL, $(LIB_TARGET)) \
- $(call do_install_mkdir,$(libdir_SQ)); \
- cp -fpR $(LIB_INSTALL) $(DESTDIR)$(libdir_SQ)
-
-install_pkgconfig:
- $(call QUIET_INSTALL, $(PKG_CONFIG_FILE)) \
- $(call do_install_pkgconfig_file,$(prefix))
-
-install_headers:
- $(call QUIET_INSTALL, headers) \
- $(call do_install,event-parse.h,$(includedir_SQ),644); \
- $(call do_install,event-utils.h,$(includedir_SQ),644); \
- $(call do_install,trace-seq.h,$(includedir_SQ),644); \
- $(call do_install,kbuffer.h,$(includedir_SQ),644)
-
-install: install_lib
-
-clean: clean_plugins
- $(call QUIET_CLEAN, libtraceevent) \
- $(RM) *.o *~ $(TARGETS) *.a *.so $(VERSION_FILES) .*.d .*.cmd; \
- $(RM) TRACEEVENT-CFLAGS tags TAGS; \
- $(RM) $(PKG_CONFIG_FILE)
-
-PHONY += doc
-doc:
- $(call descend,Documentation)
-
-PHONY += doc-clean
-doc-clean:
- $(call descend,Documentation,clean)
-
-PHONY += doc-install
-doc-install:
- $(call descend,Documentation,install)
-
-PHONY += doc-uninstall
-doc-uninstall:
- $(call descend,Documentation,uninstall)
-
-PHONY += help
-help:
- @echo 'Possible targets:'
- @echo''
- @echo ' all - default, compile the library and the'\
- 'plugins'
- @echo ' plugins - compile the plugins'
- @echo ' install - install the library, the plugins,'\
- 'the header and pkgconfig files'
- @echo ' clean - clean the library and the plugins object files'
- @echo ' doc - compile the documentation files - man'\
- 'and html pages, in the Documentation directory'
- @echo ' doc-clean - clean the documentation files'
- @echo ' doc-install - install the man pages'
- @echo ' doc-uninstall - uninstall the man pages'
- @echo''
-
-PHONY += plugins
-plugins:
- $(call descend,plugins)
-
-PHONY += install_plugins
-install_plugins:
- $(call descend,plugins,install)
-
-PHONY += clean_plugins
-clean_plugins:
- $(call descend,plugins,clean)
-
-force:
-
-# Declare the contents of the .PHONY variable as phony. We keep that
-# information in a variable so we can use it in if_changed and friends.
-.PHONY: $(PHONY)
diff --git a/tools/lib/traceevent/event-parse-api.c b/tools/lib/traceevent/event-parse-api.c
deleted file mode 100644
index f8361e4..0000000
--- a/tools/lib/traceevent/event-parse-api.c
+++ /dev/null
@@ -1,333 +0,0 @@
-// SPDX-License-Identifier: LGPL-2.1
-/*
- * Copyright (C) 2009, 2010 Red Hat Inc, Steven Rostedt <srostedt@redhat.com>
- *
- */
-
-#include "event-parse.h"
-#include "event-parse-local.h"
-#include "event-utils.h"
-
-/**
- * tep_get_event - returns the event with the given index
- * @tep: a handle to the tep_handle
- * @index: index of the requested event, in the range 0 .. nr_events
- *
- * This returns pointer to the element of the events array with the given index
- * If @tep is NULL, or @index is not in the range 0 .. nr_events, NULL is returned.
- */
-struct tep_event *tep_get_event(struct tep_handle *tep, int index)
-{
- if (tep && tep->events && index < tep->nr_events)
- return tep->events[index];
-
- return NULL;
-}
-
-/**
- * tep_get_first_event - returns the first event in the events array
- * @tep: a handle to the tep_handle
- *
- * This returns pointer to the first element of the events array
- * If @tep is NULL, NULL is returned.
- */
-struct tep_event *tep_get_first_event(struct tep_handle *tep)
-{
- return tep_get_event(tep, 0);
-}
-
-/**
- * tep_get_events_count - get the number of defined events
- * @tep: a handle to the tep_handle
- *
- * This returns number of elements in event array
- * If @tep is NULL, 0 is returned.
- */
-int tep_get_events_count(struct tep_handle *tep)
-{
- if (tep)
- return tep->nr_events;
- return 0;
-}
-
-/**
- * tep_set_flag - set event parser flag
- * @tep: a handle to the tep_handle
- * @flag: flag, or combination of flags to be set
- * can be any combination from enum tep_flag
- *
- * This sets a flag or combination of flags from enum tep_flag
- */
-void tep_set_flag(struct tep_handle *tep, int flag)
-{
- if (tep)
- tep->flags |= flag;
-}
-
-/**
- * tep_clear_flag - clear event parser flag
- * @tep: a handle to the tep_handle
- * @flag: flag to be cleared
- *
- * This clears a tep flag
- */
-void tep_clear_flag(struct tep_handle *tep, enum tep_flag flag)
-{
- if (tep)
- tep->flags &= ~flag;
-}
-
-/**
- * tep_test_flag - check the state of event parser flag
- * @tep: a handle to the tep_handle
- * @flag: flag to be checked
- *
- * This returns the state of the requested tep flag.
- * Returns: true if the flag is set, false otherwise.
- */
-bool tep_test_flag(struct tep_handle *tep, enum tep_flag flag)
-{
- if (tep)
- return tep->flags & flag;
- return false;
-}
-
-__hidden unsigned short data2host2(struct tep_handle *tep, unsigned short data)
-{
- unsigned short swap;
-
- if (!tep || tep->host_bigendian == tep->file_bigendian)
- return data;
-
- swap = ((data & 0xffULL) << 8) |
- ((data & (0xffULL << 8)) >> 8);
-
- return swap;
-}
-
-__hidden unsigned int data2host4(struct tep_handle *tep, unsigned int data)
-{
- unsigned int swap;
-
- if (!tep || tep->host_bigendian == tep->file_bigendian)
- return data;
-
- swap = ((data & 0xffULL) << 24) |
- ((data & (0xffULL << 8)) << 8) |
- ((data & (0xffULL << 16)) >> 8) |
- ((data & (0xffULL << 24)) >> 24);
-
- return swap;
-}
-
-__hidden unsigned long long
-data2host8(struct tep_handle *tep, unsigned long long data)
-{
- unsigned long long swap;
-
- if (!tep || tep->host_bigendian == tep->file_bigendian)
- return data;
-
- swap = ((data & 0xffULL) << 56) |
- ((data & (0xffULL << 8)) << 40) |
- ((data & (0xffULL << 16)) << 24) |
- ((data & (0xffULL << 24)) << 8) |
- ((data & (0xffULL << 32)) >> 8) |
- ((data & (0xffULL << 40)) >> 24) |
- ((data & (0xffULL << 48)) >> 40) |
- ((data & (0xffULL << 56)) >> 56);
-
- return swap;
-}
-
-/**
- * tep_get_header_page_size - get size of the header page
- * @tep: a handle to the tep_handle
- *
- * This returns size of the header page
- * If @tep is NULL, 0 is returned.
- */
-int tep_get_header_page_size(struct tep_handle *tep)
-{
- if (tep)
- return tep->header_page_size_size;
- return 0;
-}
-
-/**
- * tep_get_header_timestamp_size - get size of the timestamp in the header page
- * @tep: a handle to the tep_handle
- *
- * This returns size of the timestamp in the header page
- * If @tep is NULL, 0 is returned.
- */
-int tep_get_header_timestamp_size(struct tep_handle *tep)
-{
- if (tep)
- return tep->header_page_ts_size;
- return 0;
-}
-
-/**
- * tep_get_cpus - get the number of CPUs
- * @tep: a handle to the tep_handle
- *
- * This returns the number of CPUs
- * If @tep is NULL, 0 is returned.
- */
-int tep_get_cpus(struct tep_handle *tep)
-{
- if (tep)
- return tep->cpus;
- return 0;
-}
-
-/**
- * tep_set_cpus - set the number of CPUs
- * @tep: a handle to the tep_handle
- *
- * This sets the number of CPUs
- */
-void tep_set_cpus(struct tep_handle *tep, int cpus)
-{
- if (tep)
- tep->cpus = cpus;
-}
-
-/**
- * tep_get_long_size - get the size of a long integer on the traced machine
- * @tep: a handle to the tep_handle
- *
- * This returns the size of a long integer on the traced machine
- * If @tep is NULL, 0 is returned.
- */
-int tep_get_long_size(struct tep_handle *tep)
-{
- if (tep)
- return tep->long_size;
- return 0;
-}
-
-/**
- * tep_set_long_size - set the size of a long integer on the traced machine
- * @tep: a handle to the tep_handle
- * @size: size, in bytes, of a long integer
- *
- * This sets the size of a long integer on the traced machine
- */
-void tep_set_long_size(struct tep_handle *tep, int long_size)
-{
- if (tep)
- tep->long_size = long_size;
-}
-
-/**
- * tep_get_page_size - get the size of a memory page on the traced machine
- * @tep: a handle to the tep_handle
- *
- * This returns the size of a memory page on the traced machine
- * If @tep is NULL, 0 is returned.
- */
-int tep_get_page_size(struct tep_handle *tep)
-{
- if (tep)
- return tep->page_size;
- return 0;
-}
-
-/**
- * tep_set_page_size - set the size of a memory page on the traced machine
- * @tep: a handle to the tep_handle
- * @_page_size: size of a memory page, in bytes
- *
- * This sets the size of a memory page on the traced machine
- */
-void tep_set_page_size(struct tep_handle *tep, int _page_size)
-{
- if (tep)
- tep->page_size = _page_size;
-}
-
-/**
- * tep_is_file_bigendian - return the endian of the file
- * @tep: a handle to the tep_handle
- *
- * This returns true if the file is in big endian order
- * If @tep is NULL, false is returned.
- */
-bool tep_is_file_bigendian(struct tep_handle *tep)
-{
- if (tep)
- return (tep->file_bigendian == TEP_BIG_ENDIAN);
- return false;
-}
-
-/**
- * tep_set_file_bigendian - set if the file is in big endian order
- * @tep: a handle to the tep_handle
- * @endian: non zero, if the file is in big endian order
- *
- * This sets if the file is in big endian order
- */
-void tep_set_file_bigendian(struct tep_handle *tep, enum tep_endian endian)
-{
- if (tep)
- tep->file_bigendian = endian;
-}
-
-/**
- * tep_is_local_bigendian - return the endian of the saved local machine
- * @tep: a handle to the tep_handle
- *
- * This returns true if the saved local machine in @tep is big endian.
- * If @tep is NULL, false is returned.
- */
-bool tep_is_local_bigendian(struct tep_handle *tep)
-{
- if (tep)
- return (tep->host_bigendian == TEP_BIG_ENDIAN);
- return 0;
-}
-
-/**
- * tep_set_local_bigendian - set the stored local machine endian order
- * @tep: a handle to the tep_handle
- * @endian: non zero, if the local host has big endian order
- *
- * This sets the endian order for the local machine.
- */
-void tep_set_local_bigendian(struct tep_handle *tep, enum tep_endian endian)
-{
- if (tep)
- tep->host_bigendian = endian;
-}
-
-/**
- * tep_is_old_format - get if an old kernel is used
- * @tep: a handle to the tep_handle
- *
- * This returns true, if an old kernel is used to generate the tracing events or
- * false if a new kernel is used. Old kernels did not have header page info.
- * If @tep is NULL, false is returned.
- */
-bool tep_is_old_format(struct tep_handle *tep)
-{
- if (tep)
- return tep->old_format;
- return false;
-}
-
-/**
- * tep_set_test_filters - set a flag to test a filter string
- * @tep: a handle to the tep_handle
- * @test_filters: the new value of the test_filters flag
- *
- * This sets a flag to test a filter string. If this flag is set, when
- * tep_filter_add_filter_str() API as called,it will print the filter string
- * instead of adding it.
- */
-void tep_set_test_filters(struct tep_handle *tep, int test_filters)
-{
- if (tep)
- tep->test_filters = test_filters;
-}
diff --git a/tools/lib/traceevent/event-parse-local.h b/tools/lib/traceevent/event-parse-local.h
deleted file mode 100644
index fd4bbcf..0000000
--- a/tools/lib/traceevent/event-parse-local.h
+++ /dev/null
@@ -1,123 +0,0 @@
-// SPDX-License-Identifier: LGPL-2.1
-/*
- * Copyright (C) 2009, 2010 Red Hat Inc, Steven Rostedt <srostedt@redhat.com>
- *
- */
-
-#ifndef _PARSE_EVENTS_INT_H
-#define _PARSE_EVENTS_INT_H
-
-struct tep_cmdline;
-struct cmdline_list;
-struct func_map;
-struct func_list;
-struct event_handler;
-struct func_resolver;
-struct tep_plugins_dir;
-
-#define __hidden __attribute__((visibility ("hidden")))
-
-struct tep_handle {
- int ref_count;
-
- int header_page_ts_offset;
- int header_page_ts_size;
- int header_page_size_offset;
- int header_page_size_size;
- int header_page_data_offset;
- int header_page_data_size;
- int header_page_overwrite;
-
- enum tep_endian file_bigendian;
- enum tep_endian host_bigendian;
-
- int old_format;
-
- int cpus;
- int long_size;
- int page_size;
-
- struct tep_cmdline *cmdlines;
- struct cmdline_list *cmdlist;
- int cmdline_count;
-
- struct func_map *func_map;
- struct func_resolver *func_resolver;
- struct func_list *funclist;
- unsigned int func_count;
-
- struct printk_map *printk_map;
- struct printk_list *printklist;
- unsigned int printk_count;
-
- struct tep_event **events;
- int nr_events;
- struct tep_event **sort_events;
- enum tep_event_sort_type last_type;
-
- int type_offset;
- int type_size;
-
- int pid_offset;
- int pid_size;
-
- int pc_offset;
- int pc_size;
-
- int flags_offset;
- int flags_size;
-
- int ld_offset;
- int ld_size;
-
- int test_filters;
-
- int flags;
-
- struct tep_format_field *bprint_ip_field;
- struct tep_format_field *bprint_fmt_field;
- struct tep_format_field *bprint_buf_field;
-
- struct event_handler *handlers;
- struct tep_function_handler *func_handlers;
-
- /* cache */
- struct tep_event *last_event;
-
- struct tep_plugins_dir *plugins_dir;
-};
-
-enum tep_print_parse_type {
- PRINT_FMT_STRING,
- PRINT_FMT_ARG_DIGIT,
- PRINT_FMT_ARG_POINTER,
- PRINT_FMT_ARG_STRING,
-};
-
-struct tep_print_parse {
- struct tep_print_parse *next;
-
- char *format;
- int ls;
- enum tep_print_parse_type type;
- struct tep_print_arg *arg;
- struct tep_print_arg *len_as_arg;
-};
-
-void free_tep_event(struct tep_event *event);
-void free_tep_format_field(struct tep_format_field *field);
-void free_tep_plugin_paths(struct tep_handle *tep);
-
-unsigned short data2host2(struct tep_handle *tep, unsigned short data);
-unsigned int data2host4(struct tep_handle *tep, unsigned int data);
-unsigned long long data2host8(struct tep_handle *tep, unsigned long long data);
-
-/* access to the internal parser */
-int peek_char(void);
-void init_input_buf(const char *buf, unsigned long long size);
-unsigned long long get_input_buf_ptr(void);
-const char *get_input_buf(void);
-enum tep_event_type read_token(char **tok);
-void free_token(char *tok);
-
-#endif /* _PARSE_EVENTS_INT_H */
diff --git a/tools/lib/traceevent/event-parse.c b/tools/lib/traceevent/event-parse.c
deleted file mode 100644
index 8e24c4c..0000000
--- a/tools/lib/traceevent/event-parse.c
+++ /dev/null
@@ -1,7624 +0,0 @@
-// SPDX-License-Identifier: LGPL-2.1
-/*
- * Copyright (C) 2009, 2010 Red Hat Inc, Steven Rostedt <srostedt@redhat.com>
- *
- *
- * The parts for function graph printing was taken and modified from the
- * Linux Kernel that were written by
- * - Copyright (C) 2009 Frederic Weisbecker,
- * Frederic Weisbecker gave his permission to relicense the code to
- * the Lesser General Public License.
- */
-#include <inttypes.h>
-#include <stdio.h>
-#include <stdlib.h>
-#include <string.h>
-#include <stdarg.h>
-#include <ctype.h>
-#include <errno.h>
-#include <stdint.h>
-#include <limits.h>
-#include <linux/time64.h>
-
-#include <netinet/in.h>
-#include "event-parse.h"
-
-#include "event-parse-local.h"
-#include "event-utils.h"
-#include "trace-seq.h"
-
-static const char *input_buf;
-static unsigned long long input_buf_ptr;
-static unsigned long long input_buf_siz;
-
-static int is_flag_field;
-static int is_symbolic_field;
-
-static int show_warning = 1;
-
-#define do_warning(fmt, ...) \
- do { \
- if (show_warning) \
- warning(fmt, ##__VA_ARGS__); \
- } while (0)
-
-#define do_warning_event(event, fmt, ...) \
- do { \
- if (!show_warning) \
- continue; \
- \
- if (event) \
- warning("[%s:%s] " fmt, event->system, \
- event->name, ##__VA_ARGS__); \
- else \
- warning(fmt, ##__VA_ARGS__); \
- } while (0)
-
-/**
- * init_input_buf - init buffer for parsing
- * @buf: buffer to parse
- * @size: the size of the buffer
- *
- * Initializes the internal buffer that tep_read_token() will parse.
- */
-__hidden void init_input_buf(const char *buf, unsigned long long size)
-{
- input_buf = buf;
- input_buf_siz = size;
- input_buf_ptr = 0;
-}
-
-__hidden const char *get_input_buf(void)
-{
- return input_buf;
-}
-
-__hidden unsigned long long get_input_buf_ptr(void)
-{
- return input_buf_ptr;
-}
-
-struct event_handler {
- struct event_handler *next;
- int id;
- const char *sys_name;
- const char *event_name;
- tep_event_handler_func func;
- void *context;
-};
-
-struct func_params {
- struct func_params *next;
- enum tep_func_arg_type type;
-};
-
-struct tep_function_handler {
- struct tep_function_handler *next;
- enum tep_func_arg_type ret_type;
- char *name;
- tep_func_handler func;
- struct func_params *params;
- int nr_args;
-};
-
-static unsigned long long
-process_defined_func(struct trace_seq *s, void *data, int size,
- struct tep_event *event, struct tep_print_arg *arg);
-
-static void free_func_handle(struct tep_function_handler *func);
-
-void breakpoint(void)
-{
- static int x;
- x++;
-}
-
-static struct tep_print_arg *alloc_arg(void)
-{
- return calloc(1, sizeof(struct tep_print_arg));
-}
-
-struct tep_cmdline {
- char *comm;
- int pid;
-};
-
-static int cmdline_cmp(const void *a, const void *b)
-{
- const struct tep_cmdline *ca = a;
- const struct tep_cmdline *cb = b;
-
- if (ca->pid < cb->pid)
- return -1;
- if (ca->pid > cb->pid)
- return 1;
-
- return 0;
-}
-
-/* Looking for where to place the key */
-static int cmdline_slot_cmp(const void *a, const void *b)
-{
- const struct tep_cmdline *ca = a;
- const struct tep_cmdline *cb = b;
- const struct tep_cmdline *cb1 = cb + 1;
-
- if (ca->pid < cb->pid)
- return -1;
-
- if (ca->pid > cb->pid) {
- if (ca->pid <= cb1->pid)
- return 0;
- return 1;
- }
-
- return 0;
-}
-
-struct cmdline_list {
- struct cmdline_list *next;
- char *comm;
- int pid;
-};
-
-static int cmdline_init(struct tep_handle *tep)
-{
- struct cmdline_list *cmdlist = tep->cmdlist;
- struct cmdline_list *item;
- struct tep_cmdline *cmdlines;
- int i;
-
- cmdlines = malloc(sizeof(*cmdlines) * tep->cmdline_count);
- if (!cmdlines)
- return -1;
-
- i = 0;
- while (cmdlist) {
- cmdlines[i].pid = cmdlist->pid;
- cmdlines[i].comm = cmdlist->comm;
- i++;
- item = cmdlist;
- cmdlist = cmdlist->next;
- free(item);
- }
-
- qsort(cmdlines, tep->cmdline_count, sizeof(*cmdlines), cmdline_cmp);
-
- tep->cmdlines = cmdlines;
- tep->cmdlist = NULL;
-
- return 0;
-}
-
-static const char *find_cmdline(struct tep_handle *tep, int pid)
-{
- const struct tep_cmdline *comm;
- struct tep_cmdline key;
-
- if (!pid)
- return "<idle>";
-
- if (!tep->cmdlines && cmdline_init(tep))
- return "<not enough memory for cmdlines!>";
-
- key.pid = pid;
-
- comm = bsearch(&key, tep->cmdlines, tep->cmdline_count,
- sizeof(*tep->cmdlines), cmdline_cmp);
-
- if (comm)
- return comm->comm;
- return "<...>";
-}
-
-/**
- * tep_is_pid_registered - return if a pid has a cmdline registered
- * @tep: a handle to the trace event parser context
- * @pid: The pid to check if it has a cmdline registered with.
- *
- * Returns true if the pid has a cmdline mapped to it
- * false otherwise.
- */
-bool tep_is_pid_registered(struct tep_handle *tep, int pid)
-{
- const struct tep_cmdline *comm;
- struct tep_cmdline key;
-
- if (!pid)
- return true;
-
- if (!tep->cmdlines && cmdline_init(tep))
- return false;
-
- key.pid = pid;
-
- comm = bsearch(&key, tep->cmdlines, tep->cmdline_count,
- sizeof(*tep->cmdlines), cmdline_cmp);
-
- if (comm)
- return true;
- return false;
-}
-
-/*
- * If the command lines have been converted to an array, then
- * we must add this pid. This is much slower than when cmdlines
- * are added before the array is initialized.
- */
-static int add_new_comm(struct tep_handle *tep,
- const char *comm, int pid, bool override)
-{
- struct tep_cmdline *cmdlines = tep->cmdlines;
- struct tep_cmdline *cmdline;
- struct tep_cmdline key;
- char *new_comm;
- int cnt;
-
- if (!pid)
- return 0;
-
- /* avoid duplicates */
- key.pid = pid;
-
- cmdline = bsearch(&key, tep->cmdlines, tep->cmdline_count,
- sizeof(*tep->cmdlines), cmdline_cmp);
- if (cmdline) {
- if (!override) {
- errno = EEXIST;
- return -1;
- }
- new_comm = strdup(comm);
- if (!new_comm) {
- errno = ENOMEM;
- return -1;
- }
- free(cmdline->comm);
- cmdline->comm = new_comm;
-
- return 0;
- }
-
- cmdlines = realloc(cmdlines, sizeof(*cmdlines) * (tep->cmdline_count + 1));
- if (!cmdlines) {
- errno = ENOMEM;
- return -1;
- }
- tep->cmdlines = cmdlines;
-
- key.comm = strdup(comm);
- if (!key.comm) {
- errno = ENOMEM;
- return -1;
- }
-
- if (!tep->cmdline_count) {
- /* no entries yet */
- tep->cmdlines[0] = key;
- tep->cmdline_count++;
- return 0;
- }
-
- /* Now find where we want to store the new cmdline */
- cmdline = bsearch(&key, tep->cmdlines, tep->cmdline_count - 1,
- sizeof(*tep->cmdlines), cmdline_slot_cmp);
-
- cnt = tep->cmdline_count;
- if (cmdline) {
- /* cmdline points to the one before the spot we want */
- cmdline++;
- cnt -= cmdline - tep->cmdlines;
-
- } else {
- /* The new entry is either before or after the list */
- if (key.pid > tep->cmdlines[tep->cmdline_count - 1].pid) {
- tep->cmdlines[tep->cmdline_count++] = key;
- return 0;
- }
- cmdline = &tep->cmdlines[0];
- }
- memmove(cmdline + 1, cmdline, (cnt * sizeof(*cmdline)));
- *cmdline = key;
-
- tep->cmdline_count++;
-
- return 0;
-}
-
-static int _tep_register_comm(struct tep_handle *tep,
- const char *comm, int pid, bool override)
-{
- struct cmdline_list *item;
-
- if (tep->cmdlines)
- return add_new_comm(tep, comm, pid, override);
-
- item = malloc(sizeof(*item));
- if (!item)
- return -1;
-
- if (comm)
- item->comm = strdup(comm);
- else
- item->comm = strdup("<...>");
- if (!item->comm) {
- free(item);
- return -1;
- }
- item->pid = pid;
- item->next = tep->cmdlist;
-
- tep->cmdlist = item;
- tep->cmdline_count++;
-
- return 0;
-}
-
-/**
- * tep_register_comm - register a pid / comm mapping
- * @tep: a handle to the trace event parser context
- * @comm: the command line to register
- * @pid: the pid to map the command line to
- *
- * This adds a mapping to search for command line names with
- * a given pid. The comm is duplicated. If a command with the same pid
- * already exist, -1 is returned and errno is set to EEXIST
- */
-int tep_register_comm(struct tep_handle *tep, const char *comm, int pid)
-{
- return _tep_register_comm(tep, comm, pid, false);
-}
-
-/**
- * tep_override_comm - register a pid / comm mapping
- * @tep: a handle to the trace event parser context
- * @comm: the command line to register
- * @pid: the pid to map the command line to
- *
- * This adds a mapping to search for command line names with
- * a given pid. The comm is duplicated. If a command with the same pid
- * already exist, the command string is udapted with the new one
- */
-int tep_override_comm(struct tep_handle *tep, const char *comm, int pid)
-{
- if (!tep->cmdlines && cmdline_init(tep)) {
- errno = ENOMEM;
- return -1;
- }
- return _tep_register_comm(tep, comm, pid, true);
-}
-
-struct func_map {
- unsigned long long addr;
- char *func;
- char *mod;
-};
-
-struct func_list {
- struct func_list *next;
- unsigned long long addr;
- char *func;
- char *mod;
-};
-
-static int func_cmp(const void *a, const void *b)
-{
- const struct func_map *fa = a;
- const struct func_map *fb = b;
-
- if (fa->addr < fb->addr)
- return -1;
- if (fa->addr > fb->addr)
- return 1;
-
- return 0;
-}
-
-/*
- * We are searching for a record in between, not an exact
- * match.
- */
-static int func_bcmp(const void *a, const void *b)
-{
- const struct func_map *fa = a;
- const struct func_map *fb = b;
-
- if ((fa->addr == fb->addr) ||
-
- (fa->addr > fb->addr &&
- fa->addr < (fb+1)->addr))
- return 0;
-
- if (fa->addr < fb->addr)
- return -1;
-
- return 1;
-}
-
-static int func_map_init(struct tep_handle *tep)
-{
- struct func_list *funclist;
- struct func_list *item;
- struct func_map *func_map;
- int i;
-
- func_map = malloc(sizeof(*func_map) * (tep->func_count + 1));
- if (!func_map)
- return -1;
-
- funclist = tep->funclist;
-
- i = 0;
- while (funclist) {
- func_map[i].func = funclist->func;
- func_map[i].addr = funclist->addr;
- func_map[i].mod = funclist->mod;
- i++;
- item = funclist;
- funclist = funclist->next;
- free(item);
- }
-
- qsort(func_map, tep->func_count, sizeof(*func_map), func_cmp);
-
- /*
- * Add a special record at the end.
- */
- func_map[tep->func_count].func = NULL;
- func_map[tep->func_count].addr = 0;
- func_map[tep->func_count].mod = NULL;
-
- tep->func_map = func_map;
- tep->funclist = NULL;
-
- return 0;
-}
-
-static struct func_map *
-__find_func(struct tep_handle *tep, unsigned long long addr)
-{
- struct func_map *func;
- struct func_map key;
-
- if (!tep->func_map)
- func_map_init(tep);
-
- key.addr = addr;
-
- func = bsearch(&key, tep->func_map, tep->func_count,
- sizeof(*tep->func_map), func_bcmp);
-
- return func;
-}
-
-struct func_resolver {
- tep_func_resolver_t *func;
- void *priv;
- struct func_map map;
-};
-
-/**
- * tep_set_function_resolver - set an alternative function resolver
- * @tep: a handle to the trace event parser context
- * @resolver: function to be used
- * @priv: resolver function private state.
- *
- * Some tools may have already a way to resolve kernel functions, allow them to
- * keep using it instead of duplicating all the entries inside tep->funclist.
- */
-int tep_set_function_resolver(struct tep_handle *tep,
- tep_func_resolver_t *func, void *priv)
-{
- struct func_resolver *resolver = malloc(sizeof(*resolver));
-
- if (resolver == NULL)
- return -1;
-
- resolver->func = func;
- resolver->priv = priv;
-
- free(tep->func_resolver);
- tep->func_resolver = resolver;
-
- return 0;
-}
-
-/**
- * tep_reset_function_resolver - reset alternative function resolver
- * @tep: a handle to the trace event parser context
- *
- * Stop using whatever alternative resolver was set, use the default
- * one instead.
- */
-void tep_reset_function_resolver(struct tep_handle *tep)
-{
- free(tep->func_resolver);
- tep->func_resolver = NULL;
-}
-
-static struct func_map *
-find_func(struct tep_handle *tep, unsigned long long addr)
-{
- struct func_map *map;
-
- if (!tep->func_resolver)
- return __find_func(tep, addr);
-
- map = &tep->func_resolver->map;
- map->mod = NULL;
- map->addr = addr;
- map->func = tep->func_resolver->func(tep->func_resolver->priv,
- &map->addr, &map->mod);
- if (map->func == NULL)
- return NULL;
-
- return map;
-}
-
-/**
- * tep_find_function - find a function by a given address
- * @tep: a handle to the trace event parser context
- * @addr: the address to find the function with
- *
- * Returns a pointer to the function stored that has the given
- * address. Note, the address does not have to be exact, it
- * will select the function that would contain the address.
- */
-const char *tep_find_function(struct tep_handle *tep, unsigned long long addr)
-{
- struct func_map *map;
-
- map = find_func(tep, addr);
- if (!map)
- return NULL;
-
- return map->func;
-}
-
-/**
- * tep_find_function_address - find a function address by a given address
- * @tep: a handle to the trace event parser context
- * @addr: the address to find the function with
- *
- * Returns the address the function starts at. This can be used in
- * conjunction with tep_find_function to print both the function
- * name and the function offset.
- */
-unsigned long long
-tep_find_function_address(struct tep_handle *tep, unsigned long long addr)
-{
- struct func_map *map;
-
- map = find_func(tep, addr);
- if (!map)
- return 0;
-
- return map->addr;
-}
-
-/**
- * tep_register_function - register a function with a given address
- * @tep: a handle to the trace event parser context
- * @function: the function name to register
- * @addr: the address the function starts at
- * @mod: the kernel module the function may be in (NULL for none)
- *
- * This registers a function name with an address and module.
- * The @func passed in is duplicated.
- */
-int tep_register_function(struct tep_handle *tep, char *func,
- unsigned long long addr, char *mod)
-{
- struct func_list *item = malloc(sizeof(*item));
-
- if (!item)
- return -1;
-
- item->next = tep->funclist;
- item->func = strdup(func);
- if (!item->func)
- goto out_free;
-
- if (mod) {
- item->mod = strdup(mod);
- if (!item->mod)
- goto out_free_func;
- } else
- item->mod = NULL;
- item->addr = addr;
-
- tep->funclist = item;
- tep->func_count++;
-
- return 0;
-
-out_free_func:
- free(item->func);
- item->func = NULL;
-out_free:
- free(item);
- errno = ENOMEM;
- return -1;
-}
-
-/**
- * tep_print_funcs - print out the stored functions
- * @tep: a handle to the trace event parser context
- *
- * This prints out the stored functions.
- */
-void tep_print_funcs(struct tep_handle *tep)
-{
- int i;
-
- if (!tep->func_map)
- func_map_init(tep);
-
- for (i = 0; i < (int)tep->func_count; i++) {
- printf("%016llx %s",
- tep->func_map[i].addr,
- tep->func_map[i].func);
- if (tep->func_map[i].mod)
- printf(" [%s]\n", tep->func_map[i].mod);
- else
- printf("\n");
- }
-}
-
-struct printk_map {
- unsigned long long addr;
- char *printk;
-};
-
-struct printk_list {
- struct printk_list *next;
- unsigned long long addr;
- char *printk;
-};
-
-static int printk_cmp(const void *a, const void *b)
-{
- const struct printk_map *pa = a;
- const struct printk_map *pb = b;
-
- if (pa->addr < pb->addr)
- return -1;
- if (pa->addr > pb->addr)
- return 1;
-
- return 0;
-}
-
-static int printk_map_init(struct tep_handle *tep)
-{
- struct printk_list *printklist;
- struct printk_list *item;
- struct printk_map *printk_map;
- int i;
-
- printk_map = malloc(sizeof(*printk_map) * (tep->printk_count + 1));
- if (!printk_map)
- return -1;
-
- printklist = tep->printklist;
-
- i = 0;
- while (printklist) {
- printk_map[i].printk = printklist->printk;
- printk_map[i].addr = printklist->addr;
- i++;
- item = printklist;
- printklist = printklist->next;
- free(item);
- }
-
- qsort(printk_map, tep->printk_count, sizeof(*printk_map), printk_cmp);
-
- tep->printk_map = printk_map;
- tep->printklist = NULL;
-
- return 0;
-}
-
-static struct printk_map *
-find_printk(struct tep_handle *tep, unsigned long long addr)
-{
- struct printk_map *printk;
- struct printk_map key;
-
- if (!tep->printk_map && printk_map_init(tep))
- return NULL;
-
- key.addr = addr;
-
- printk = bsearch(&key, tep->printk_map, tep->printk_count,
- sizeof(*tep->printk_map), printk_cmp);
-
- return printk;
-}
-
-/**
- * tep_register_print_string - register a string by its address
- * @tep: a handle to the trace event parser context
- * @fmt: the string format to register
- * @addr: the address the string was located at
- *
- * This registers a string by the address it was stored in the kernel.
- * The @fmt passed in is duplicated.
- */
-int tep_register_print_string(struct tep_handle *tep, const char *fmt,
- unsigned long long addr)
-{
- struct printk_list *item = malloc(sizeof(*item));
- char *p;
-
- if (!item)
- return -1;
-
- item->next = tep->printklist;
- item->addr = addr;
-
- /* Strip off quotes and '\n' from the end */
- if (fmt[0] == '"')
- fmt++;
- item->printk = strdup(fmt);
- if (!item->printk)
- goto out_free;
-
- p = item->printk + strlen(item->printk) - 1;
- if (*p == '"')
- *p = 0;
-
- p -= 2;
- if (strcmp(p, "\\n") == 0)
- *p = 0;
-
- tep->printklist = item;
- tep->printk_count++;
-
- return 0;
-
-out_free:
- free(item);
- errno = ENOMEM;
- return -1;
-}
-
-/**
- * tep_print_printk - print out the stored strings
- * @tep: a handle to the trace event parser context
- *
- * This prints the string formats that were stored.
- */
-void tep_print_printk(struct tep_handle *tep)
-{
- int i;
-
- if (!tep->printk_map)
- printk_map_init(tep);
-
- for (i = 0; i < (int)tep->printk_count; i++) {
- printf("%016llx %s\n",
- tep->printk_map[i].addr,
- tep->printk_map[i].printk);
- }
-}
-
-static struct tep_event *alloc_event(void)
-{
- return calloc(1, sizeof(struct tep_event));
-}
-
-static int add_event(struct tep_handle *tep, struct tep_event *event)
-{
- int i;
- struct tep_event **events = realloc(tep->events, sizeof(event) *
- (tep->nr_events + 1));
- if (!events)
- return -1;
-
- tep->events = events;
-
- for (i = 0; i < tep->nr_events; i++) {
- if (tep->events[i]->id > event->id)
- break;
- }
- if (i < tep->nr_events)
- memmove(&tep->events[i + 1],
- &tep->events[i],
- sizeof(event) * (tep->nr_events - i));
-
- tep->events[i] = event;
- tep->nr_events++;
-
- event->tep = tep;
-
- return 0;
-}
-
-static int event_item_type(enum tep_event_type type)
-{
- switch (type) {
- case TEP_EVENT_ITEM ... TEP_EVENT_SQUOTE:
- return 1;
- case TEP_EVENT_ERROR ... TEP_EVENT_DELIM:
- default:
- return 0;
- }
-}
-
-static void free_flag_sym(struct tep_print_flag_sym *fsym)
-{
- struct tep_print_flag_sym *next;
-
- while (fsym) {
- next = fsym->next;
- free(fsym->value);
- free(fsym->str);
- free(fsym);
- fsym = next;
- }
-}
-
-static void free_arg(struct tep_print_arg *arg)
-{
- struct tep_print_arg *farg;
-
- if (!arg)
- return;
-
- switch (arg->type) {
- case TEP_PRINT_ATOM:
- free(arg->atom.atom);
- break;
- case TEP_PRINT_FIELD:
- free(arg->field.name);
- break;
- case TEP_PRINT_FLAGS:
- free_arg(arg->flags.field);
- free(arg->flags.delim);
- free_flag_sym(arg->flags.flags);
- break;
- case TEP_PRINT_SYMBOL:
- free_arg(arg->symbol.field);
- free_flag_sym(arg->symbol.symbols);
- break;
- case TEP_PRINT_HEX:
- case TEP_PRINT_HEX_STR:
- free_arg(arg->hex.field);
- free_arg(arg->hex.size);
- break;
- case TEP_PRINT_INT_ARRAY:
- free_arg(arg->int_array.field);
- free_arg(arg->int_array.count);
- free_arg(arg->int_array.el_size);
- break;
- case TEP_PRINT_TYPE:
- free(arg->typecast.type);
- free_arg(arg->typecast.item);
- break;
- case TEP_PRINT_STRING:
- case TEP_PRINT_BSTRING:
- free(arg->string.string);
- break;
- case TEP_PRINT_BITMASK:
- free(arg->bitmask.bitmask);
- break;
- case TEP_PRINT_DYNAMIC_ARRAY:
- case TEP_PRINT_DYNAMIC_ARRAY_LEN:
- free(arg->dynarray.index);
- break;
- case TEP_PRINT_OP:
- free(arg->op.op);
- free_arg(arg->op.left);
- free_arg(arg->op.right);
- break;
- case TEP_PRINT_FUNC:
- while (arg->func.args) {
- farg = arg->func.args;
- arg->func.args = farg->next;
- free_arg(farg);
- }
- break;
-
- case TEP_PRINT_NULL:
- default:
- break;
- }
-
- free(arg);
-}
-
-static enum tep_event_type get_type(int ch)
-{
- if (ch == '\n')
- return TEP_EVENT_NEWLINE;
- if (isspace(ch))
- return TEP_EVENT_SPACE;
- if (isalnum(ch) || ch == '_')
- return TEP_EVENT_ITEM;
- if (ch == '\'')
- return TEP_EVENT_SQUOTE;
- if (ch == '"')
- return TEP_EVENT_DQUOTE;
- if (!isprint(ch))
- return TEP_EVENT_NONE;
- if (ch == '(' || ch == ')' || ch == ',')
- return TEP_EVENT_DELIM;
-
- return TEP_EVENT_OP;
-}
-
-static int __read_char(void)
-{
- if (input_buf_ptr >= input_buf_siz)
- return -1;
-
- return input_buf[input_buf_ptr++];
-}
-
-/**
- * peek_char - peek at the next character that will be read
- *
- * Returns the next character read, or -1 if end of buffer.
- */
-__hidden int peek_char(void)
-{
- if (input_buf_ptr >= input_buf_siz)
- return -1;
-
- return input_buf[input_buf_ptr];
-}
-
-static int extend_token(char **tok, char *buf, int size)
-{
- char *newtok = realloc(*tok, size);
-
- if (!newtok) {
- free(*tok);
- *tok = NULL;
- return -1;
- }
-
- if (!*tok)
- strcpy(newtok, buf);
- else
- strcat(newtok, buf);
- *tok = newtok;
-
- return 0;
-}
-
-static enum tep_event_type force_token(const char *str, char **tok);
-
-static enum tep_event_type __read_token(char **tok)
-{
- char buf[BUFSIZ];
- int ch, last_ch, quote_ch, next_ch;
- int i = 0;
- int tok_size = 0;
- enum tep_event_type type;
-
- *tok = NULL;
-
-
- ch = __read_char();
- if (ch < 0)
- return TEP_EVENT_NONE;
-
- type = get_type(ch);
- if (type == TEP_EVENT_NONE)
- return type;
-
- buf[i++] = ch;
-
- switch (type) {
- case TEP_EVENT_NEWLINE:
- case TEP_EVENT_DELIM:
- if (asprintf(tok, "%c", ch) < 0)
- return TEP_EVENT_ERROR;
-
- return type;
-
- case TEP_EVENT_OP:
- switch (ch) {
- case '-':
- next_ch = peek_char();
- if (next_ch == '>') {
- buf[i++] = __read_char();
- break;
- }
- /* fall through */
- case '+':
- case '|':
- case '&':
- case '>':
- case '<':
- last_ch = ch;
- ch = peek_char();
- if (ch != last_ch)
- goto test_equal;
- buf[i++] = __read_char();
- switch (last_ch) {
- case '>':
- case '<':
- goto test_equal;
- default:
- break;
- }
- break;
- case '!':
- case '=':
- goto test_equal;
- default: /* what should we do instead? */
- break;
- }
- buf[i] = 0;
- *tok = strdup(buf);
- return type;
-
- test_equal:
- ch = peek_char();
- if (ch == '=')
- buf[i++] = __read_char();
- goto out;
-
- case TEP_EVENT_DQUOTE:
- case TEP_EVENT_SQUOTE:
- /* don't keep quotes */
- i--;
- quote_ch = ch;
- last_ch = 0;
- concat:
- do {
- if (i == (BUFSIZ - 1)) {
- buf[i] = 0;
- tok_size += BUFSIZ;
-
- if (extend_token(tok, buf, tok_size) < 0)
- return TEP_EVENT_NONE;
- i = 0;
- }
- last_ch = ch;
- ch = __read_char();
- buf[i++] = ch;
- /* the '\' '\' will cancel itself */
- if (ch == '\\' && last_ch == '\\')
- last_ch = 0;
- } while (ch != quote_ch || last_ch == '\\');
- /* remove the last quote */
- i--;
-
- /*
- * For strings (double quotes) check the next token.
- * If it is another string, concatinate the two.
- */
- if (type == TEP_EVENT_DQUOTE) {
- unsigned long long save_input_buf_ptr = input_buf_ptr;
-
- do {
- ch = __read_char();
- } while (isspace(ch));
- if (ch == '"')
- goto concat;
- input_buf_ptr = save_input_buf_ptr;
- }
-
- goto out;
-
- case TEP_EVENT_ERROR ... TEP_EVENT_SPACE:
- case TEP_EVENT_ITEM:
- default:
- break;
- }
-
- while (get_type(peek_char()) == type) {
- if (i == (BUFSIZ - 1)) {
- buf[i] = 0;
- tok_size += BUFSIZ;
-
- if (extend_token(tok, buf, tok_size) < 0)
- return TEP_EVENT_NONE;
- i = 0;
- }
- ch = __read_char();
- buf[i++] = ch;
- }
-
- out:
- buf[i] = 0;
- if (extend_token(tok, buf, tok_size + i + 1) < 0)
- return TEP_EVENT_NONE;
-
- if (type == TEP_EVENT_ITEM) {
- /*
- * Older versions of the kernel has a bug that
- * creates invalid symbols and will break the mac80211
- * parsing. This is a work around to that bug.
- *
- * See Linux kernel commit:
- * 811cb50baf63461ce0bdb234927046131fc7fa8b
- */
- if (strcmp(*tok, "LOCAL_PR_FMT") == 0) {
- free(*tok);
- *tok = NULL;
- return force_token("\"%s\" ", tok);
- } else if (strcmp(*tok, "STA_PR_FMT") == 0) {
- free(*tok);
- *tok = NULL;
- return force_token("\" sta:%pM\" ", tok);
- } else if (strcmp(*tok, "VIF_PR_FMT") == 0) {
- free(*tok);
- *tok = NULL;
- return force_token("\" vif:%p(%d)\" ", tok);
- }
- }
-
- return type;
-}
-
-static enum tep_event_type force_token(const char *str, char **tok)
-{
- const char *save_input_buf;
- unsigned long long save_input_buf_ptr;
- unsigned long long save_input_buf_siz;
- enum tep_event_type type;
-
- /* save off the current input pointers */
- save_input_buf = input_buf;
- save_input_buf_ptr = input_buf_ptr;
- save_input_buf_siz = input_buf_siz;
-
- init_input_buf(str, strlen(str));
-
- type = __read_token(tok);
-
- /* reset back to original token */
- input_buf = save_input_buf;
- input_buf_ptr = save_input_buf_ptr;
- input_buf_siz = save_input_buf_siz;
-
- return type;
-}
-
-/**
- * free_token - free a token returned by tep_read_token
- * @token: the token to free
- */
-__hidden void free_token(char *tok)
-{
- if (tok)
- free(tok);
-}
-
-/**
- * read_token - access to utilities to use the tep parser
- * @tok: The token to return
- *
- * This will parse tokens from the string given by
- * tep_init_data().
- *
- * Returns the token type.
- */
-__hidden enum tep_event_type read_token(char **tok)
-{
- enum tep_event_type type;
-
- for (;;) {
- type = __read_token(tok);
- if (type != TEP_EVENT_SPACE)
- return type;
-
- free_token(*tok);
- }
-
- /* not reached */
- *tok = NULL;
- return TEP_EVENT_NONE;
-}
-
-/* no newline */
-static enum tep_event_type read_token_item(char **tok)
-{
- enum tep_event_type type;
-
- for (;;) {
- type = __read_token(tok);
- if (type != TEP_EVENT_SPACE && type != TEP_EVENT_NEWLINE)
- return type;
- free_token(*tok);
- *tok = NULL;
- }
-
- /* not reached */
- *tok = NULL;
- return TEP_EVENT_NONE;
-}
-
-static int test_type(enum tep_event_type type, enum tep_event_type expect)
-{
- if (type != expect) {
- do_warning("Error: expected type %d but read %d",
- expect, type);
- return -1;
- }
- return 0;
-}
-
-static int test_type_token(enum tep_event_type type, const char *token,
- enum tep_event_type expect, const char *expect_tok)
-{
- if (type != expect) {
- do_warning("Error: expected type %d but read %d",
- expect, type);
- return -1;
- }
-
- if (strcmp(token, expect_tok) != 0) {
- do_warning("Error: expected '%s' but read '%s'",
- expect_tok, token);
- return -1;
- }
- return 0;
-}
-
-static int __read_expect_type(enum tep_event_type expect, char **tok, int newline_ok)
-{
- enum tep_event_type type;
-
- if (newline_ok)
- type = read_token(tok);
- else
- type = read_token_item(tok);
- return test_type(type, expect);
-}
-
-static int read_expect_type(enum tep_event_type expect, char **tok)
-{
- return __read_expect_type(expect, tok, 1);
-}
-
-static int __read_expected(enum tep_event_type expect, const char *str,
- int newline_ok)
-{
- enum tep_event_type type;
- char *token;
- int ret;
-
- if (newline_ok)
- type = read_token(&token);
- else
- type = read_token_item(&token);
-
- ret = test_type_token(type, token, expect, str);
-
- free_token(token);
-
- return ret;
-}
-
-static int read_expected(enum tep_event_type expect, const char *str)
-{
- return __read_expected(expect, str, 1);
-}
-
-static int read_expected_item(enum tep_event_type expect, const char *str)
-{
- return __read_expected(expect, str, 0);
-}
-
-static char *event_read_name(void)
-{
- char *token;
-
- if (read_expected(TEP_EVENT_ITEM, "name") < 0)
- return NULL;
-
- if (read_expected(TEP_EVENT_OP, ":") < 0)
- return NULL;
-
- if (read_expect_type(TEP_EVENT_ITEM, &token) < 0)
- goto fail;
-
- return token;
-
- fail:
- free_token(token);
- return NULL;
-}
-
-static int event_read_id(void)
-{
- char *token;
- int id;
-
- if (read_expected_item(TEP_EVENT_ITEM, "ID") < 0)
- return -1;
-
- if (read_expected(TEP_EVENT_OP, ":") < 0)
- return -1;
-
- if (read_expect_type(TEP_EVENT_ITEM, &token) < 0)
- goto fail;
-
- id = strtoul(token, NULL, 0);
- free_token(token);
- return id;
-
- fail:
- free_token(token);
- return -1;
-}
-
-static int field_is_string(struct tep_format_field *field)
-{
- if ((field->flags & TEP_FIELD_IS_ARRAY) &&
- (strstr(field->type, "char") || strstr(field->type, "u8") ||
- strstr(field->type, "s8")))
- return 1;
-
- return 0;
-}
-
-static int field_is_dynamic(struct tep_format_field *field)
-{
- if (strncmp(field->type, "__data_loc", 10) == 0)
- return 1;
-
- return 0;
-}
-
-static int field_is_relative_dynamic(struct tep_format_field *field)
-{
- if (strncmp(field->type, "__rel_loc", 9) == 0)
- return 1;
-
- return 0;
-}
-
-static int field_is_long(struct tep_format_field *field)
-{
- /* includes long long */
- if (strstr(field->type, "long"))
- return 1;
-
- return 0;
-}
-
-static unsigned int type_size(const char *name)
-{
- /* This covers all TEP_FIELD_IS_STRING types. */
- static struct {
- const char *type;
- unsigned int size;
- } table[] = {
- { "u8", 1 },
- { "u16", 2 },
- { "u32", 4 },
- { "u64", 8 },
- { "s8", 1 },
- { "s16", 2 },
- { "s32", 4 },
- { "s64", 8 },
- { "char", 1 },
- { },
- };
- int i;
-
- for (i = 0; table[i].type; i++) {
- if (!strcmp(table[i].type, name))
- return table[i].size;
- }
-
- return 0;
-}
-
-static int append(char **buf, const char *delim, const char *str)
-{
- char *new_buf;
-
- new_buf = realloc(*buf, strlen(*buf) + strlen(delim) + strlen(str) + 1);
- if (!new_buf)
- return -1;
- strcat(new_buf, delim);
- strcat(new_buf, str);
- *buf = new_buf;
- return 0;
-}
-
-static int event_read_fields(struct tep_event *event, struct tep_format_field **fields)
-{
- struct tep_format_field *field = NULL;
- enum tep_event_type type;
- char *token;
- char *last_token;
- char *delim = " ";
- int count = 0;
- int ret;
-
- do {
- unsigned int size_dynamic = 0;
-
- type = read_token(&token);
- if (type == TEP_EVENT_NEWLINE) {
- free_token(token);
- return count;
- }
-
- count++;
-
- if (test_type_token(type, token, TEP_EVENT_ITEM, "field"))
- goto fail;
- free_token(token);
-
- type = read_token(&token);
- /*
- * The ftrace fields may still use the "special" name.
- * Just ignore it.
- */
- if (event->flags & TEP_EVENT_FL_ISFTRACE &&
- type == TEP_EVENT_ITEM && strcmp(token, "special") == 0) {
- free_token(token);
- type = read_token(&token);
- }
-
- if (test_type_token(type, token, TEP_EVENT_OP, ":") < 0)
- goto fail;
-
- free_token(token);
- if (read_expect_type(TEP_EVENT_ITEM, &token) < 0)
- goto fail;
-
- last_token = token;
-
- field = calloc(1, sizeof(*field));
- if (!field)
- goto fail;
-
- field->event = event;
-
- /* read the rest of the type */
- for (;;) {
- type = read_token(&token);
- if (type == TEP_EVENT_ITEM ||
- (type == TEP_EVENT_OP && strcmp(token, "*") == 0) ||
- /*
- * Some of the ftrace fields are broken and have
- * an illegal "." in them.
- */
- (event->flags & TEP_EVENT_FL_ISFTRACE &&
- type == TEP_EVENT_OP && strcmp(token, ".") == 0)) {
-
- if (strcmp(token, "*") == 0)
- field->flags |= TEP_FIELD_IS_POINTER;
-
- if (field->type) {
- ret = append(&field->type, delim, last_token);
- free(last_token);
- if (ret < 0)
- goto fail;
- } else
- field->type = last_token;
- last_token = token;
- delim = " ";
- continue;
- }
-
- /* Handle __attribute__((user)) */
- if ((type == TEP_EVENT_DELIM) &&
- strcmp("__attribute__", last_token) == 0 &&
- token[0] == '(') {
- int depth = 1;
- int ret;
-
- ret = append(&field->type, " ", last_token);
- ret |= append(&field->type, "", "(");
- if (ret < 0)
- goto fail;
-
- delim = " ";
- while ((type = read_token(&token)) != TEP_EVENT_NONE) {
- if (type == TEP_EVENT_DELIM) {
- if (token[0] == '(')
- depth++;
- else if (token[0] == ')')
- depth--;
- if (!depth)
- break;
- ret = append(&field->type, "", token);
- delim = "";
- } else {
- ret = append(&field->type, delim, token);
- delim = " ";
- }
- if (ret < 0)
- goto fail;
- free(last_token);
- last_token = token;
- }
- continue;
- }
- break;
- }
-
- if (!field->type) {
- do_warning_event(event, "%s: no type found", __func__);
- goto fail;
- }
- field->name = field->alias = last_token;
-
- if (test_type(type, TEP_EVENT_OP))
- goto fail;
-
- if (strcmp(token, "[") == 0) {
- enum tep_event_type last_type = type;
- char *brackets = token;
-
- field->flags |= TEP_FIELD_IS_ARRAY;
-
- type = read_token(&token);
-
- if (type == TEP_EVENT_ITEM)
- field->arraylen = strtoul(token, NULL, 0);
- else
- field->arraylen = 0;
-
- while (strcmp(token, "]") != 0) {
- const char *delim;
-
- if (last_type == TEP_EVENT_ITEM &&
- type == TEP_EVENT_ITEM)
- delim = " ";
- else
- delim = "";
-
- last_type = type;
-
- ret = append(&brackets, delim, token);
- if (ret < 0) {
- free(brackets);
- goto fail;
- }
- /* We only care about the last token */
- field->arraylen = strtoul(token, NULL, 0);
- free_token(token);
- type = read_token(&token);
- if (type == TEP_EVENT_NONE) {
- free(brackets);
- do_warning_event(event, "failed to find token");
- goto fail;
- }
- }
-
- free_token(token);
-
- ret = append(&brackets, "", "]");
- if (ret < 0) {
- free(brackets);
- goto fail;
- }
-
- /* add brackets to type */
-
- type = read_token(&token);
- /*
- * If the next token is not an OP, then it is of
- * the format: type [] item;
- */
- if (type == TEP_EVENT_ITEM) {
- ret = append(&field->type, " ", field->name);
- if (ret < 0) {
- free(brackets);
- goto fail;
- }
- ret = append(&field->type, "", brackets);
-
- size_dynamic = type_size(field->name);
- free_token(field->name);
- field->name = field->alias = token;
- type = read_token(&token);
- } else {
- ret = append(&field->type, "", brackets);
- if (ret < 0) {
- free(brackets);
- goto fail;
- }
- }
- free(brackets);
- }
-
- if (field_is_string(field))
- field->flags |= TEP_FIELD_IS_STRING;
- if (field_is_dynamic(field))
- field->flags |= TEP_FIELD_IS_DYNAMIC;
- if (field_is_relative_dynamic(field))
- field->flags |= TEP_FIELD_IS_DYNAMIC | TEP_FIELD_IS_RELATIVE;
- if (field_is_long(field))
- field->flags |= TEP_FIELD_IS_LONG;
-
- if (test_type_token(type, token, TEP_EVENT_OP, ";"))
- goto fail;
- free_token(token);
-
- if (read_expected(TEP_EVENT_ITEM, "offset") < 0)
- goto fail_expect;
-
- if (read_expected(TEP_EVENT_OP, ":") < 0)
- goto fail_expect;
-
- if (read_expect_type(TEP_EVENT_ITEM, &token))
- goto fail;
- field->offset = strtoul(token, NULL, 0);
- free_token(token);
-
- if (read_expected(TEP_EVENT_OP, ";") < 0)
- goto fail_expect;
-
- if (read_expected(TEP_EVENT_ITEM, "size") < 0)
- goto fail_expect;
-
- if (read_expected(TEP_EVENT_OP, ":") < 0)
- goto fail_expect;
-
- if (read_expect_type(TEP_EVENT_ITEM, &token))
- goto fail;
- field->size = strtoul(token, NULL, 0);
- free_token(token);
-
- if (read_expected(TEP_EVENT_OP, ";") < 0)
- goto fail_expect;
-
- type = read_token(&token);
- if (type != TEP_EVENT_NEWLINE) {
- /* newer versions of the kernel have a "signed" type */
- if (test_type_token(type, token, TEP_EVENT_ITEM, "signed"))
- goto fail;
-
- free_token(token);
-
- if (read_expected(TEP_EVENT_OP, ":") < 0)
- goto fail_expect;
-
- if (read_expect_type(TEP_EVENT_ITEM, &token))
- goto fail;
-
- if (strtoul(token, NULL, 0))
- field->flags |= TEP_FIELD_IS_SIGNED;
-
- free_token(token);
- if (read_expected(TEP_EVENT_OP, ";") < 0)
- goto fail_expect;
-
- if (read_expect_type(TEP_EVENT_NEWLINE, &token))
- goto fail;
- }
-
- free_token(token);
-
- if (field->flags & TEP_FIELD_IS_ARRAY) {
- if (field->arraylen)
- field->elementsize = field->size / field->arraylen;
- else if (field->flags & TEP_FIELD_IS_DYNAMIC)
- field->elementsize = size_dynamic;
- else if (field->flags & TEP_FIELD_IS_STRING)
- field->elementsize = 1;
- else if (field->flags & TEP_FIELD_IS_LONG)
- field->elementsize = event->tep ?
- event->tep->long_size :
- sizeof(long);
- } else
- field->elementsize = field->size;
-
- *fields = field;
- fields = &field->next;
-
- } while (1);
-
- return 0;
-
-fail:
- free_token(token);
-fail_expect:
- if (field) {
- free(field->type);
- free(field->name);
- free(field);
- }
- return -1;
-}
-
-static int event_read_format(struct tep_event *event)
-{
- char *token;
- int ret;
-
- if (read_expected_item(TEP_EVENT_ITEM, "format") < 0)
- return -1;
-
- if (read_expected(TEP_EVENT_OP, ":") < 0)
- return -1;
-
- if (read_expect_type(TEP_EVENT_NEWLINE, &token))
- goto fail;
- free_token(token);
-
- ret = event_read_fields(event, &event->format.common_fields);
- if (ret < 0)
- return ret;
- event->format.nr_common = ret;
-
- ret = event_read_fields(event, &event->format.fields);
- if (ret < 0)
- return ret;
- event->format.nr_fields = ret;
-
- return 0;
-
- fail:
- free_token(token);
- return -1;
-}
-
-static enum tep_event_type
-process_arg_token(struct tep_event *event, struct tep_print_arg *arg,
- char **tok, enum tep_event_type type);
-
-static enum tep_event_type
-process_arg(struct tep_event *event, struct tep_print_arg *arg, char **tok)
-{
- enum tep_event_type type;
- char *token;
-
- type = read_token(&token);
- *tok = token;
-
- return process_arg_token(event, arg, tok, type);
-}
-
-static enum tep_event_type
-process_op(struct tep_event *event, struct tep_print_arg *arg, char **tok);
-
-/*
- * For __print_symbolic() and __print_flags, we need to completely
- * evaluate the first argument, which defines what to print next.
- */
-static enum tep_event_type
-process_field_arg(struct tep_event *event, struct tep_print_arg *arg, char **tok)
-{
- enum tep_event_type type;
-
- type = process_arg(event, arg, tok);
-
- while (type == TEP_EVENT_OP) {
- type = process_op(event, arg, tok);
- }
-
- return type;
-}
-
-static enum tep_event_type
-process_cond(struct tep_event *event, struct tep_print_arg *top, char **tok)
-{
- struct tep_print_arg *arg, *left, *right;
- enum tep_event_type type;
- char *token = NULL;
-
- arg = alloc_arg();
- left = alloc_arg();
- right = alloc_arg();
-
- if (!arg || !left || !right) {
- do_warning_event(event, "%s: not enough memory!", __func__);
- /* arg will be freed at out_free */
- free_arg(left);
- free_arg(right);
- goto out_free;
- }
-
- arg->type = TEP_PRINT_OP;
- arg->op.left = left;
- arg->op.right = right;
-
- *tok = NULL;
- type = process_arg(event, left, &token);
-
- again:
- if (type == TEP_EVENT_ERROR)
- goto out_free;
-
- /* Handle other operations in the arguments */
- if (type == TEP_EVENT_OP && strcmp(token, ":") != 0) {
- type = process_op(event, left, &token);
- goto again;
- }
-
- if (test_type_token(type, token, TEP_EVENT_OP, ":"))
- goto out_free;
-
- arg->op.op = token;
-
- type = process_arg(event, right, &token);
-
- top->op.right = arg;
-
- *tok = token;
- return type;
-
-out_free:
- /* Top may point to itself */
- top->op.right = NULL;
- free_token(token);
- free_arg(arg);
- return TEP_EVENT_ERROR;
-}
-
-static enum tep_event_type
-process_array(struct tep_event *event, struct tep_print_arg *top, char **tok)
-{
- struct tep_print_arg *arg;
- enum tep_event_type type;
- char *token = NULL;
-
- arg = alloc_arg();
- if (!arg) {
- do_warning_event(event, "%s: not enough memory!", __func__);
- /* '*tok' is set to top->op.op. No need to free. */
- *tok = NULL;
- return TEP_EVENT_ERROR;
- }
-
- *tok = NULL;
- type = process_arg(event, arg, &token);
- if (test_type_token(type, token, TEP_EVENT_OP, "]"))
- goto out_free;
-
- top->op.right = arg;
-
- free_token(token);
- type = read_token_item(&token);
- *tok = token;
-
- return type;
-
-out_free:
- free_token(token);
- free_arg(arg);
- return TEP_EVENT_ERROR;
-}
-
-static int get_op_prio(char *op)
-{
- if (!op[1]) {
- switch (op[0]) {
- case '~':
- case '!':
- return 4;
- case '*':
- case '/':
- case '%':
- return 6;
- case '+':
- case '-':
- return 7;
- /* '>>' and '<<' are 8 */
- case '<':
- case '>':
- return 9;
- /* '==' and '!=' are 10 */
- case '&':
- return 11;
- case '^':
- return 12;
- case '|':
- return 13;
- case '?':
- return 16;
- default:
- do_warning("unknown op '%c'", op[0]);
- return -1;
- }
- } else {
- if (strcmp(op, "++") == 0 ||
- strcmp(op, "--") == 0) {
- return 3;
- } else if (strcmp(op, ">>") == 0 ||
- strcmp(op, "<<") == 0) {
- return 8;
- } else if (strcmp(op, ">=") == 0 ||
- strcmp(op, "<=") == 0) {
- return 9;
- } else if (strcmp(op, "==") == 0 ||
- strcmp(op, "!=") == 0) {
- return 10;
- } else if (strcmp(op, "&&") == 0) {
- return 14;
- } else if (strcmp(op, "||") == 0) {
- return 15;
- } else {
- do_warning("unknown op '%s'", op);
- return -1;
- }
- }
-}
-
-static int set_op_prio(struct tep_print_arg *arg)
-{
-
- /* single ops are the greatest */
- if (!arg->op.left || arg->op.left->type == TEP_PRINT_NULL)
- arg->op.prio = 0;
- else
- arg->op.prio = get_op_prio(arg->op.op);
-
- return arg->op.prio;
-}
-
-/* Note, *tok does not get freed, but will most likely be saved */
-static enum tep_event_type
-process_op(struct tep_event *event, struct tep_print_arg *arg, char **tok)
-{
- struct tep_print_arg *left, *right = NULL;
- enum tep_event_type type;
- char *token;
-
- /* the op is passed in via tok */
- token = *tok;
-
- if (arg->type == TEP_PRINT_OP && !arg->op.left) {
- /* handle single op */
- if (token[1]) {
- do_warning_event(event, "bad op token %s", token);
- goto out_free;
- }
- switch (token[0]) {
- case '~':
- case '!':
- case '+':
- case '-':
- break;
- default:
- do_warning_event(event, "bad op token %s", token);
- goto out_free;
-
- }
-
- /* make an empty left */
- left = alloc_arg();
- if (!left)
- goto out_warn_free;
-
- left->type = TEP_PRINT_NULL;
- arg->op.left = left;
-
- right = alloc_arg();
- if (!right)
- goto out_warn_free;
-
- arg->op.right = right;
-
- /* do not free the token, it belongs to an op */
- *tok = NULL;
- type = process_arg(event, right, tok);
-
- } else if (strcmp(token, "?") == 0) {
-
- left = alloc_arg();
- if (!left)
- goto out_warn_free;
-
- /* copy the top arg to the left */
- *left = *arg;
-
- arg->type = TEP_PRINT_OP;
- arg->op.op = token;
- arg->op.left = left;
- arg->op.prio = 0;
-
- /* it will set arg->op.right */
- type = process_cond(event, arg, tok);
-
- } else if (strcmp(token, ">>") == 0 ||
- strcmp(token, "<<") == 0 ||
- strcmp(token, "&") == 0 ||
- strcmp(token, "|") == 0 ||
- strcmp(token, "&&") == 0 ||
- strcmp(token, "||") == 0 ||
- strcmp(token, "-") == 0 ||
- strcmp(token, "+") == 0 ||
- strcmp(token, "*") == 0 ||
- strcmp(token, "^") == 0 ||
- strcmp(token, "/") == 0 ||
- strcmp(token, "%") == 0 ||
- strcmp(token, "<") == 0 ||
- strcmp(token, ">") == 0 ||
- strcmp(token, "<=") == 0 ||
- strcmp(token, ">=") == 0 ||
- strcmp(token, "==") == 0 ||
- strcmp(token, "!=") == 0) {
-
- left = alloc_arg();
- if (!left)
- goto out_warn_free;
-
- /* copy the top arg to the left */
- *left = *arg;
-
- arg->type = TEP_PRINT_OP;
- arg->op.op = token;
- arg->op.left = left;
- arg->op.right = NULL;
-
- if (set_op_prio(arg) == -1) {
- event->flags |= TEP_EVENT_FL_FAILED;
- /* arg->op.op (= token) will be freed at out_free */
- arg->op.op = NULL;
- goto out_free;
- }
-
- type = read_token_item(&token);
- *tok = token;
-
- /* could just be a type pointer */
- if ((strcmp(arg->op.op, "*") == 0) &&
- type == TEP_EVENT_DELIM && (strcmp(token, ")") == 0)) {
- int ret;
-
- if (left->type != TEP_PRINT_ATOM) {
- do_warning_event(event, "bad pointer type");
- goto out_free;
- }
- ret = append(&left->atom.atom, " ", "*");
- if (ret < 0)
- goto out_warn_free;
-
- free(arg->op.op);
- *arg = *left;
- free(left);
-
- return type;
- }
-
- right = alloc_arg();
- if (!right)
- goto out_warn_free;
-
- type = process_arg_token(event, right, tok, type);
- if (type == TEP_EVENT_ERROR) {
- free_arg(right);
- /* token was freed in process_arg_token() via *tok */
- token = NULL;
- goto out_free;
- }
-
- if (right->type == TEP_PRINT_OP &&
- get_op_prio(arg->op.op) < get_op_prio(right->op.op)) {
- struct tep_print_arg tmp;
-
- /* rotate ops according to the priority */
- arg->op.right = right->op.left;
-
- tmp = *arg;
- *arg = *right;
- *right = tmp;
-
- arg->op.left = right;
- } else {
- arg->op.right = right;
- }
-
- } else if (strcmp(token, "[") == 0) {
-
- left = alloc_arg();
- if (!left)
- goto out_warn_free;
-
- *left = *arg;
-
- arg->type = TEP_PRINT_OP;
- arg->op.op = token;
- arg->op.left = left;
-
- arg->op.prio = 0;
-
- /* it will set arg->op.right */
- type = process_array(event, arg, tok);
-
- } else {
- do_warning_event(event, "unknown op '%s'", token);
- event->flags |= TEP_EVENT_FL_FAILED;
- /* the arg is now the left side */
- goto out_free;
- }
-
- if (type == TEP_EVENT_OP && strcmp(*tok, ":") != 0) {
- int prio;
-
- /* higher prios need to be closer to the root */
- prio = get_op_prio(*tok);
-
- if (prio > arg->op.prio)
- return process_op(event, arg, tok);
-
- return process_op(event, right, tok);
- }
-
- return type;
-
-out_warn_free:
- do_warning_event(event, "%s: not enough memory!", __func__);
-out_free:
- free_token(token);
- *tok = NULL;
- return TEP_EVENT_ERROR;
-}
-
-static enum tep_event_type
-process_entry(struct tep_event *event __maybe_unused, struct tep_print_arg *arg,
- char **tok)
-{
- enum tep_event_type type;
- char *field;
- char *token;
-
- if (read_expected(TEP_EVENT_OP, "->") < 0)
- goto out_err;
-
- if (read_expect_type(TEP_EVENT_ITEM, &token) < 0)
- goto out_free;
- field = token;
-
- arg->type = TEP_PRINT_FIELD;
- arg->field.name = field;
-
- if (is_flag_field) {
- arg->field.field = tep_find_any_field(event, arg->field.name);
- arg->field.field->flags |= TEP_FIELD_IS_FLAG;
- is_flag_field = 0;
- } else if (is_symbolic_field) {
- arg->field.field = tep_find_any_field(event, arg->field.name);
- arg->field.field->flags |= TEP_FIELD_IS_SYMBOLIC;
- is_symbolic_field = 0;
- }
-
- type = read_token(&token);
- *tok = token;
-
- return type;
-
- out_free:
- free_token(token);
- out_err:
- *tok = NULL;
- return TEP_EVENT_ERROR;
-}
-
-static int alloc_and_process_delim(struct tep_event *event, char *next_token,
- struct tep_print_arg **print_arg)
-{
- struct tep_print_arg *field;
- enum tep_event_type type;
- char *token;
- int ret = 0;
-
- field = alloc_arg();
- if (!field) {
- do_warning_event(event, "%s: not enough memory!", __func__);
- errno = ENOMEM;
- return -1;
- }
-
- type = process_arg(event, field, &token);
-
- if (test_type_token(type, token, TEP_EVENT_DELIM, next_token)) {
- errno = EINVAL;
- ret = -1;
- free_arg(field);
- goto out_free_token;
- }
-
- *print_arg = field;
-
-out_free_token:
- free_token(token);
-
- return ret;
-}
-
-static char *arg_eval (struct tep_print_arg *arg);
-
-static unsigned long long
-eval_type_str(unsigned long long val, const char *type, int pointer)
-{
- int sign = 0;
- char *ref;
- int len;
-
- len = strlen(type);
-
- if (pointer) {
-
- if (type[len-1] != '*') {
- do_warning("pointer expected with non pointer type");
- return val;
- }
-
- ref = malloc(len);
- if (!ref) {
- do_warning("%s: not enough memory!", __func__);
- return val;
- }
- memcpy(ref, type, len);
-
- /* chop off the " *" */
- ref[len - 2] = 0;
-
- val = eval_type_str(val, ref, 0);
- free(ref);
- return val;
- }
-
- /* check if this is a pointer */
- if (type[len - 1] == '*')
- return val;
-
- /* Try to figure out the arg size*/
- if (strncmp(type, "struct", 6) == 0)
- /* all bets off */
- return val;
-
- if (strcmp(type, "u8") == 0)
- return val & 0xff;
-
- if (strcmp(type, "u16") == 0)
- return val & 0xffff;
-
- if (strcmp(type, "u32") == 0)
- return val & 0xffffffff;
-
- if (strcmp(type, "u64") == 0 ||
- strcmp(type, "s64") == 0)
- return val;
-
- if (strcmp(type, "s8") == 0)
- return (unsigned long long)(char)val & 0xff;
-
- if (strcmp(type, "s16") == 0)
- return (unsigned long long)(short)val & 0xffff;
-
- if (strcmp(type, "s32") == 0)
- return (unsigned long long)(int)val & 0xffffffff;
-
- if (strncmp(type, "unsigned ", 9) == 0) {
- sign = 0;
- type += 9;
- }
-
- if (strcmp(type, "char") == 0) {
- if (sign)
- return (unsigned long long)(char)val & 0xff;
- else
- return val & 0xff;
- }
-
- if (strcmp(type, "short") == 0) {
- if (sign)
- return (unsigned long long)(short)val & 0xffff;
- else
- return val & 0xffff;
- }
-
- if (strcmp(type, "int") == 0) {
- if (sign)
- return (unsigned long long)(int)val & 0xffffffff;
- else
- return val & 0xffffffff;
- }
-
- return val;
-}
-
-/*
- * Try to figure out the type.
- */
-static unsigned long long
-eval_type(unsigned long long val, struct tep_print_arg *arg, int pointer)
-{
- if (arg->type != TEP_PRINT_TYPE) {
- do_warning("expected type argument");
- return 0;
- }
-
- return eval_type_str(val, arg->typecast.type, pointer);
-}
-
-static int arg_num_eval(struct tep_print_arg *arg, long long *val)
-{
- long long left, right;
- int ret = 1;
-
- switch (arg->type) {
- case TEP_PRINT_ATOM:
- *val = strtoll(arg->atom.atom, NULL, 0);
- break;
- case TEP_PRINT_TYPE:
- ret = arg_num_eval(arg->typecast.item, val);
- if (!ret)
- break;
- *val = eval_type(*val, arg, 0);
- break;
- case TEP_PRINT_OP:
- switch (arg->op.op[0]) {
- case '|':
- ret = arg_num_eval(arg->op.left, &left);
- if (!ret)
- break;
- ret = arg_num_eval(arg->op.right, &right);
- if (!ret)
- break;
- if (arg->op.op[1])
- *val = left || right;
- else
- *val = left | right;
- break;
- case '&':
- ret = arg_num_eval(arg->op.left, &left);
- if (!ret)
- break;
- ret = arg_num_eval(arg->op.right, &right);
- if (!ret)
- break;
- if (arg->op.op[1])
- *val = left && right;
- else
- *val = left & right;
- break;
- case '<':
- ret = arg_num_eval(arg->op.left, &left);
- if (!ret)
- break;
- ret = arg_num_eval(arg->op.right, &right);
- if (!ret)
- break;
- switch (arg->op.op[1]) {
- case 0:
- *val = left < right;
- break;
- case '<':
- *val = left << right;
- break;
- case '=':
- *val = left <= right;
- break;
- default:
- do_warning("unknown op '%s'", arg->op.op);
- ret = 0;
- }
- break;
- case '>':
- ret = arg_num_eval(arg->op.left, &left);
- if (!ret)
- break;
- ret = arg_num_eval(arg->op.right, &right);
- if (!ret)
- break;
- switch (arg->op.op[1]) {
- case 0:
- *val = left > right;
- break;
- case '>':
- *val = left >> right;
- break;
- case '=':
- *val = left >= right;
- break;
- default:
- do_warning("unknown op '%s'", arg->op.op);
- ret = 0;
- }
- break;
- case '=':
- ret = arg_num_eval(arg->op.left, &left);
- if (!ret)
- break;
- ret = arg_num_eval(arg->op.right, &right);
- if (!ret)
- break;
-
- if (arg->op.op[1] != '=') {
- do_warning("unknown op '%s'", arg->op.op);
- ret = 0;
- } else
- *val = left == right;
- break;
- case '!':
- ret = arg_num_eval(arg->op.left, &left);
- if (!ret)
- break;
- ret = arg_num_eval(arg->op.right, &right);
- if (!ret)
- break;
-
- switch (arg->op.op[1]) {
- case '=':
- *val = left != right;
- break;
- default:
- do_warning("unknown op '%s'", arg->op.op);
- ret = 0;
- }
- break;
- case '-':
- /* check for negative */
- if (arg->op.left->type == TEP_PRINT_NULL)
- left = 0;
- else
- ret = arg_num_eval(arg->op.left, &left);
- if (!ret)
- break;
- ret = arg_num_eval(arg->op.right, &right);
- if (!ret)
- break;
- *val = left - right;
- break;
- case '+':
- if (arg->op.left->type == TEP_PRINT_NULL)
- left = 0;
- else
- ret = arg_num_eval(arg->op.left, &left);
- if (!ret)
- break;
- ret = arg_num_eval(arg->op.right, &right);
- if (!ret)
- break;
- *val = left + right;
- break;
- case '~':
- ret = arg_num_eval(arg->op.right, &right);
- if (!ret)
- break;
- *val = ~right;
- break;
- default:
- do_warning("unknown op '%s'", arg->op.op);
- ret = 0;
- }
- break;
-
- case TEP_PRINT_NULL:
- case TEP_PRINT_FIELD ... TEP_PRINT_SYMBOL:
- case TEP_PRINT_STRING:
- case TEP_PRINT_BSTRING:
- case TEP_PRINT_BITMASK:
- default:
- do_warning("invalid eval type %d", arg->type);
- ret = 0;
-
- }
- return ret;
-}
-
-static char *arg_eval (struct tep_print_arg *arg)
-{
- long long val;
- static char buf[24];
-
- switch (arg->type) {
- case TEP_PRINT_ATOM:
- return arg->atom.atom;
- case TEP_PRINT_TYPE:
- return arg_eval(arg->typecast.item);
- case TEP_PRINT_OP:
- if (!arg_num_eval(arg, &val))
- break;
- sprintf(buf, "%lld", val);
- return buf;
-
- case TEP_PRINT_NULL:
- case TEP_PRINT_FIELD ... TEP_PRINT_SYMBOL:
- case TEP_PRINT_STRING:
- case TEP_PRINT_BSTRING:
- case TEP_PRINT_BITMASK:
- default:
- do_warning("invalid eval type %d", arg->type);
- break;
- }
-
- return NULL;
-}
-
-static enum tep_event_type
-process_fields(struct tep_event *event, struct tep_print_flag_sym **list, char **tok)
-{
- enum tep_event_type type;
- struct tep_print_arg *arg = NULL;
- struct tep_print_flag_sym *field;
- char *token = *tok;
- char *value;
-
- do {
- free_token(token);
- type = read_token_item(&token);
- if (test_type_token(type, token, TEP_EVENT_OP, "{"))
- break;
-
- arg = alloc_arg();
- if (!arg)
- goto out_free;
-
- free_token(token);
- type = process_arg(event, arg, &token);
-
- if (type == TEP_EVENT_OP)
- type = process_op(event, arg, &token);
-
- if (type == TEP_EVENT_ERROR)
- goto out_free;
-
- if (test_type_token(type, token, TEP_EVENT_DELIM, ","))
- goto out_free;
-
- field = calloc(1, sizeof(*field));
- if (!field)
- goto out_free;
-
- value = arg_eval(arg);
- if (value == NULL)
- goto out_free_field;
- field->value = strdup(value);
- if (field->value == NULL)
- goto out_free_field;
-
- free_arg(arg);
- arg = alloc_arg();
- if (!arg)
- goto out_free;
-
- free_token(token);
- type = process_arg(event, arg, &token);
- if (test_type_token(type, token, TEP_EVENT_OP, "}"))
- goto out_free_field;
-
- value = arg_eval(arg);
- if (value == NULL)
- goto out_free_field;
- field->str = strdup(value);
- if (field->str == NULL)
- goto out_free_field;
- free_arg(arg);
- arg = NULL;
-
- *list = field;
- list = &field->next;
-
- free_token(token);
- type = read_token_item(&token);
- } while (type == TEP_EVENT_DELIM && strcmp(token, ",") == 0);
-
- *tok = token;
- return type;
-
-out_free_field:
- free_flag_sym(field);
-out_free:
- free_arg(arg);
- free_token(token);
- *tok = NULL;
-
- return TEP_EVENT_ERROR;
-}
-
-static enum tep_event_type
-process_flags(struct tep_event *event, struct tep_print_arg *arg, char **tok)
-{
- struct tep_print_arg *field;
- enum tep_event_type type;
- char *token = NULL;
-
- memset(arg, 0, sizeof(*arg));
- arg->type = TEP_PRINT_FLAGS;
-
- field = alloc_arg();
- if (!field) {
- do_warning_event(event, "%s: not enough memory!", __func__);
- goto out_free;
- }
-
- type = process_field_arg(event, field, &token);
-
- /* Handle operations in the first argument */
- while (type == TEP_EVENT_OP)
- type = process_op(event, field, &token);
-
- if (test_type_token(type, token, TEP_EVENT_DELIM, ","))
- goto out_free_field;
- free_token(token);
-
- arg->flags.field = field;
-
- type = read_token_item(&token);
- if (event_item_type(type)) {
- arg->flags.delim = token;
- type = read_token_item(&token);
- }
-
- if (test_type_token(type, token, TEP_EVENT_DELIM, ","))
- goto out_free;
-
- type = process_fields(event, &arg->flags.flags, &token);
- if (test_type_token(type, token, TEP_EVENT_DELIM, ")"))
- goto out_free;
-
- free_token(token);
- type = read_token_item(tok);
- return type;
-
-out_free_field:
- free_arg(field);
-out_free:
- free_token(token);
- *tok = NULL;
- return TEP_EVENT_ERROR;
-}
-
-static enum tep_event_type
-process_symbols(struct tep_event *event, struct tep_print_arg *arg, char **tok)
-{
- struct tep_print_arg *field;
- enum tep_event_type type;
- char *token = NULL;
-
- memset(arg, 0, sizeof(*arg));
- arg->type = TEP_PRINT_SYMBOL;
-
- field = alloc_arg();
- if (!field) {
- do_warning_event(event, "%s: not enough memory!", __func__);
- goto out_free;
- }
-
- type = process_field_arg(event, field, &token);
-
- if (test_type_token(type, token, TEP_EVENT_DELIM, ","))
- goto out_free_field;
-
- arg->symbol.field = field;
-
- type = process_fields(event, &arg->symbol.symbols, &token);
- if (test_type_token(type, token, TEP_EVENT_DELIM, ")"))
- goto out_free;
-
- free_token(token);
- type = read_token_item(tok);
- return type;
-
-out_free_field:
- free_arg(field);
-out_free:
- free_token(token);
- *tok = NULL;
- return TEP_EVENT_ERROR;
-}
-
-static enum tep_event_type
-process_hex_common(struct tep_event *event, struct tep_print_arg *arg,
- char **tok, enum tep_print_arg_type type)
-{
- memset(arg, 0, sizeof(*arg));
- arg->type = type;
-
- if (alloc_and_process_delim(event, ",", &arg->hex.field))
- goto out;
-
- if (alloc_and_process_delim(event, ")", &arg->hex.size))
- goto free_field;
-
- return read_token_item(tok);
-
-free_field:
- free_arg(arg->hex.field);
- arg->hex.field = NULL;
-out:
- *tok = NULL;
- return TEP_EVENT_ERROR;
-}
-
-static enum tep_event_type
-process_hex(struct tep_event *event, struct tep_print_arg *arg, char **tok)
-{
- return process_hex_common(event, arg, tok, TEP_PRINT_HEX);
-}
-
-static enum tep_event_type
-process_hex_str(struct tep_event *event, struct tep_print_arg *arg,
- char **tok)
-{
- return process_hex_common(event, arg, tok, TEP_PRINT_HEX_STR);
-}
-
-static enum tep_event_type
-process_int_array(struct tep_event *event, struct tep_print_arg *arg, char **tok)
-{
- memset(arg, 0, sizeof(*arg));
- arg->type = TEP_PRINT_INT_ARRAY;
-
- if (alloc_and_process_delim(event, ",", &arg->int_array.field))
- goto out;
-
- if (alloc_and_process_delim(event, ",", &arg->int_array.count))
- goto free_field;
-
- if (alloc_and_process_delim(event, ")", &arg->int_array.el_size))
- goto free_size;
-
- return read_token_item(tok);
-
-free_size:
- free_arg(arg->int_array.count);
- arg->int_array.count = NULL;
-free_field:
- free_arg(arg->int_array.field);
- arg->int_array.field = NULL;
-out:
- *tok = NULL;
- return TEP_EVENT_ERROR;
-}
-
-static enum tep_event_type
-process_dynamic_array(struct tep_event *event, struct tep_print_arg *arg, char **tok)
-{
- struct tep_format_field *field;
- enum tep_event_type type;
- char *token;
-
- memset(arg, 0, sizeof(*arg));
- arg->type = TEP_PRINT_DYNAMIC_ARRAY;
-
- /*
- * The item within the parenthesis is another field that holds
- * the index into where the array starts.
- */
- type = read_token(&token);
- *tok = token;
- if (type != TEP_EVENT_ITEM)
- goto out_free;
-
- /* Find the field */
-
- field = tep_find_field(event, token);
- if (!field)
- goto out_free;
-
- arg->dynarray.field = field;
- arg->dynarray.index = 0;
-
- if (read_expected(TEP_EVENT_DELIM, ")") < 0)
- goto out_free;
-
- free_token(token);
- type = read_token_item(&token);
- *tok = token;
- if (type != TEP_EVENT_OP || strcmp(token, "[") != 0)
- return type;
-
- free_token(token);
- arg = alloc_arg();
- if (!arg) {
- do_warning_event(event, "%s: not enough memory!", __func__);
- *tok = NULL;
- return TEP_EVENT_ERROR;
- }
-
- type = process_arg(event, arg, &token);
- if (type == TEP_EVENT_ERROR)
- goto out_free_arg;
-
- if (!test_type_token(type, token, TEP_EVENT_OP, "]"))
- goto out_free_arg;
-
- free_token(token);
- type = read_token_item(tok);
- return type;
-
- out_free_arg:
- free_arg(arg);
- out_free:
- free_token(token);
- *tok = NULL;
- return TEP_EVENT_ERROR;
-}
-
-static enum tep_event_type
-process_dynamic_array_len(struct tep_event *event, struct tep_print_arg *arg,
- char **tok)
-{
- struct tep_format_field *field;
- enum tep_event_type type;
- char *token;
-
- if (read_expect_type(TEP_EVENT_ITEM, &token) < 0)
- goto out_free;
-
- arg->type = TEP_PRINT_DYNAMIC_ARRAY_LEN;
-
- /* Find the field */
- field = tep_find_field(event, token);
- if (!field)
- goto out_free;
-
- arg->dynarray.field = field;
- arg->dynarray.index = 0;
-
- if (read_expected(TEP_EVENT_DELIM, ")") < 0)
- goto out_err;
-
- free_token(token);
- type = read_token(&token);
- *tok = token;
-
- return type;
-
- out_free:
- free_token(token);
- out_err:
- *tok = NULL;
- return TEP_EVENT_ERROR;
-}
-
-static enum tep_event_type
-process_paren(struct tep_event *event, struct tep_print_arg *arg, char **tok)
-{
- struct tep_print_arg *item_arg;
- enum tep_event_type type;
- char *token;
-
- type = process_arg(event, arg, &token);
-
- if (type == TEP_EVENT_ERROR)
- goto out_free;
-
- if (type == TEP_EVENT_OP)
- type = process_op(event, arg, &token);
-
- if (type == TEP_EVENT_ERROR)
- goto out_free;
-
- if (test_type_token(type, token, TEP_EVENT_DELIM, ")"))
- goto out_free;
-
- free_token(token);
- type = read_token_item(&token);
-
- /*
- * If the next token is an item or another open paren, then
- * this was a typecast.
- */
- if (event_item_type(type) ||
- (type == TEP_EVENT_DELIM && strcmp(token, "(") == 0)) {
-
- /* make this a typecast and contine */
-
- /* prevous must be an atom */
- if (arg->type != TEP_PRINT_ATOM) {
- do_warning_event(event, "previous needed to be TEP_PRINT_ATOM");
- goto out_free;
- }
-
- item_arg = alloc_arg();
- if (!item_arg) {
- do_warning_event(event, "%s: not enough memory!",
- __func__);
- goto out_free;
- }
-
- arg->type = TEP_PRINT_TYPE;
- arg->typecast.type = arg->atom.atom;
- arg->typecast.item = item_arg;
- type = process_arg_token(event, item_arg, &token, type);
-
- }
-
- *tok = token;
- return type;
-
- out_free:
- free_token(token);
- *tok = NULL;
- return TEP_EVENT_ERROR;
-}
-
-
-static enum tep_event_type
-process_str(struct tep_event *event __maybe_unused, struct tep_print_arg *arg,
- char **tok)
-{
- enum tep_event_type type;
- char *token;
-
- if (read_expect_type(TEP_EVENT_ITEM, &token) < 0)
- goto out_free;
-
- arg->type = TEP_PRINT_STRING;
- arg->string.string = token;
- arg->string.field = NULL;
-
- if (read_expected(TEP_EVENT_DELIM, ")") < 0)
- goto out_err;
-
- type = read_token(&token);
- *tok = token;
-
- return type;
-
- out_free:
- free_token(token);
- out_err:
- *tok = NULL;
- return TEP_EVENT_ERROR;
-}
-
-static enum tep_event_type
-process_bitmask(struct tep_event *event __maybe_unused, struct tep_print_arg *arg,
- char **tok)
-{
- enum tep_event_type type;
- char *token;
-
- if (read_expect_type(TEP_EVENT_ITEM, &token) < 0)
- goto out_free;
-
- arg->type = TEP_PRINT_BITMASK;
- arg->bitmask.bitmask = token;
- arg->bitmask.field = NULL;
-
- if (read_expected(TEP_EVENT_DELIM, ")") < 0)
- goto out_err;
-
- type = read_token(&token);
- *tok = token;
-
- return type;
-
- out_free:
- free_token(token);
- out_err:
- *tok = NULL;
- return TEP_EVENT_ERROR;
-}
-
-static struct tep_function_handler *
-find_func_handler(struct tep_handle *tep, char *func_name)
-{
- struct tep_function_handler *func;
-
- if (!tep)
- return NULL;
-
- for (func = tep->func_handlers; func; func = func->next) {
- if (strcmp(func->name, func_name) == 0)
- break;
- }
-
- return func;
-}
-
-static void remove_func_handler(struct tep_handle *tep, char *func_name)
-{
- struct tep_function_handler *func;
- struct tep_function_handler **next;
-
- next = &tep->func_handlers;
- while ((func = *next)) {
- if (strcmp(func->name, func_name) == 0) {
- *next = func->next;
- free_func_handle(func);
- break;
- }
- next = &func->next;
- }
-}
-
-static enum tep_event_type
-process_func_handler(struct tep_event *event, struct tep_function_handler *func,
- struct tep_print_arg *arg, char **tok)
-{
- struct tep_print_arg **next_arg;
- struct tep_print_arg *farg;
- enum tep_event_type type;
- char *token;
- int i;
-
- arg->type = TEP_PRINT_FUNC;
- arg->func.func = func;
-
- *tok = NULL;
-
- next_arg = &(arg->func.args);
- for (i = 0; i < func->nr_args; i++) {
- farg = alloc_arg();
- if (!farg) {
- do_warning_event(event, "%s: not enough memory!",
- __func__);
- return TEP_EVENT_ERROR;
- }
-
- type = process_arg(event, farg, &token);
- if (i < (func->nr_args - 1)) {
- if (type != TEP_EVENT_DELIM || strcmp(token, ",") != 0) {
- do_warning_event(event,
- "Error: function '%s()' expects %d arguments but event %s only uses %d",
- func->name, func->nr_args,
- event->name, i + 1);
- goto err;
- }
- } else {
- if (type != TEP_EVENT_DELIM || strcmp(token, ")") != 0) {
- do_warning_event(event,
- "Error: function '%s()' only expects %d arguments but event %s has more",
- func->name, func->nr_args, event->name);
- goto err;
- }
- }
-
- *next_arg = farg;
- next_arg = &(farg->next);
- free_token(token);
- }
-
- type = read_token(&token);
- *tok = token;
-
- return type;
-
-err:
- free_arg(farg);
- free_token(token);
- return TEP_EVENT_ERROR;
-}
-
-static enum tep_event_type
-process_builtin_expect(struct tep_event *event, struct tep_print_arg *arg, char **tok)
-{
- enum tep_event_type type;
- char *token = NULL;
-
- /* Handle __builtin_expect( cond, #) */
- type = process_arg(event, arg, &token);
-
- if (type != TEP_EVENT_DELIM || token[0] != ',')
- goto out_free;
-
- free_token(token);
-
- /* We don't care what the second parameter is of the __builtin_expect() */
- if (read_expect_type(TEP_EVENT_ITEM, &token) < 0)
- goto out_free;
-
- if (read_expected(TEP_EVENT_DELIM, ")") < 0)
- goto out_free;
-
- free_token(token);
- type = read_token_item(tok);
- return type;
-
-out_free:
- free_token(token);
- *tok = NULL;
- return TEP_EVENT_ERROR;
-}
-
-static enum tep_event_type
-process_function(struct tep_event *event, struct tep_print_arg *arg,
- char *token, char **tok)
-{
- struct tep_function_handler *func;
-
- if (strcmp(token, "__print_flags") == 0) {
- free_token(token);
- is_flag_field = 1;
- return process_flags(event, arg, tok);
- }
- if (strcmp(token, "__print_symbolic") == 0) {
- free_token(token);
- is_symbolic_field = 1;
- return process_symbols(event, arg, tok);
- }
- if (strcmp(token, "__print_hex") == 0) {
- free_token(token);
- return process_hex(event, arg, tok);
- }
- if (strcmp(token, "__print_hex_str") == 0) {
- free_token(token);
- return process_hex_str(event, arg, tok);
- }
- if (strcmp(token, "__print_array") == 0) {
- free_token(token);
- return process_int_array(event, arg, tok);
- }
- if (strcmp(token, "__get_str") == 0 ||
- strcmp(token, "__get_rel_str") == 0) {
- free_token(token);
- return process_str(event, arg, tok);
- }
- if (strcmp(token, "__get_bitmask") == 0 ||
- strcmp(token, "__get_rel_bitmask") == 0) {
- free_token(token);
- return process_bitmask(event, arg, tok);
- }
- if (strcmp(token, "__get_dynamic_array") == 0 ||
- strcmp(token, "__get_rel_dynamic_array") == 0) {
- free_token(token);
- return process_dynamic_array(event, arg, tok);
- }
- if (strcmp(token, "__get_dynamic_array_len") == 0 ||
- strcmp(token, "__get_rel_dynamic_array_len") == 0) {
- free_token(token);
- return process_dynamic_array_len(event, arg, tok);
- }
- if (strcmp(token, "__builtin_expect") == 0) {
- free_token(token);
- return process_builtin_expect(event, arg, tok);
- }
-
- func = find_func_handler(event->tep, token);
- if (func) {
- free_token(token);
- return process_func_handler(event, func, arg, tok);
- }
-
- do_warning_event(event, "function %s not defined", token);
- free_token(token);
- return TEP_EVENT_ERROR;
-}
-
-static enum tep_event_type
-process_arg_token(struct tep_event *event, struct tep_print_arg *arg,
- char **tok, enum tep_event_type type)
-{
- char *token;
- char *atom;
-
- token = *tok;
-
- switch (type) {
- case TEP_EVENT_ITEM:
- if (strcmp(token, "REC") == 0) {
- free_token(token);
- type = process_entry(event, arg, &token);
- break;
- }
- atom = token;
- /* test the next token */
- type = read_token_item(&token);
-
- /*
- * If the next token is a parenthesis, then this
- * is a function.
- */
- if (type == TEP_EVENT_DELIM && strcmp(token, "(") == 0) {
- free_token(token);
- token = NULL;
- /* this will free atom. */
- type = process_function(event, arg, atom, &token);
- break;
- }
- /* atoms can be more than one token long */
- while (type == TEP_EVENT_ITEM) {
- int ret;
-
- ret = append(&atom, " ", token);
- if (ret < 0) {
- free(atom);
- *tok = NULL;
- free_token(token);
- return TEP_EVENT_ERROR;
- }
- free_token(token);
- type = read_token_item(&token);
- }
-
- arg->type = TEP_PRINT_ATOM;
- arg->atom.atom = atom;
- break;
-
- case TEP_EVENT_DQUOTE:
- case TEP_EVENT_SQUOTE:
- arg->type = TEP_PRINT_ATOM;
- arg->atom.atom = token;
- type = read_token_item(&token);
- break;
- case TEP_EVENT_DELIM:
- if (strcmp(token, "(") == 0) {
- free_token(token);
- type = process_paren(event, arg, &token);
- break;
- }
- case TEP_EVENT_OP:
- /* handle single ops */
- arg->type = TEP_PRINT_OP;
- arg->op.op = token;
- arg->op.left = NULL;
- type = process_op(event, arg, &token);
-
- /* On error, the op is freed */
- if (type == TEP_EVENT_ERROR)
- arg->op.op = NULL;
-
- /* return error type if errored */
- break;
-
- case TEP_EVENT_ERROR ... TEP_EVENT_NEWLINE:
- default:
- do_warning_event(event, "unexpected type %d", type);
- return TEP_EVENT_ERROR;
- }
- *tok = token;
-
- return type;
-}
-
-static int event_read_print_args(struct tep_event *event, struct tep_print_arg **list)
-{
- enum tep_event_type type = TEP_EVENT_ERROR;
- struct tep_print_arg *arg;
- char *token;
- int args = 0;
-
- do {
- if (type == TEP_EVENT_NEWLINE) {
- type = read_token_item(&token);
- continue;
- }
-
- arg = alloc_arg();
- if (!arg) {
- do_warning_event(event, "%s: not enough memory!",
- __func__);
- return -1;
- }
-
- type = process_arg(event, arg, &token);
-
- if (type == TEP_EVENT_ERROR) {
- free_token(token);
- free_arg(arg);
- return -1;
- }
-
- *list = arg;
- args++;
-
- if (type == TEP_EVENT_OP) {
- type = process_op(event, arg, &token);
- free_token(token);
- if (type == TEP_EVENT_ERROR) {
- *list = NULL;
- free_arg(arg);
- return -1;
- }
- list = &arg->next;
- continue;
- }
-
- if (type == TEP_EVENT_DELIM && strcmp(token, ",") == 0) {
- free_token(token);
- *list = arg;
- list = &arg->next;
- continue;
- }
- break;
- } while (type != TEP_EVENT_NONE);
-
- if (type != TEP_EVENT_NONE && type != TEP_EVENT_ERROR)
- free_token(token);
-
- return args;
-}
-
-static int event_read_print(struct tep_event *event)
-{
- enum tep_event_type type;
- char *token;
- int ret;
-
- if (read_expected_item(TEP_EVENT_ITEM, "print") < 0)
- return -1;
-
- if (read_expected(TEP_EVENT_ITEM, "fmt") < 0)
- return -1;
-
- if (read_expected(TEP_EVENT_OP, ":") < 0)
- return -1;
-
- if (read_expect_type(TEP_EVENT_DQUOTE, &token) < 0)
- goto fail;
-
- concat:
- event->print_fmt.format = token;
- event->print_fmt.args = NULL;
-
- /* ok to have no arg */
- type = read_token_item(&token);
-
- if (type == TEP_EVENT_NONE)
- return 0;
-
- /* Handle concatenation of print lines */
- if (type == TEP_EVENT_DQUOTE) {
- char *cat;
-
- if (asprintf(&cat, "%s%s", event->print_fmt.format, token) < 0)
- goto fail;
- free_token(token);
- free_token(event->print_fmt.format);
- event->print_fmt.format = NULL;
- token = cat;
- goto concat;
- }
-
- if (test_type_token(type, token, TEP_EVENT_DELIM, ","))
- goto fail;
-
- free_token(token);
-
- ret = event_read_print_args(event, &event->print_fmt.args);
- if (ret < 0)
- return -1;
-
- return ret;
-
- fail:
- free_token(token);
- return -1;
-}
-
-/**
- * tep_find_common_field - return a common field by event
- * @event: handle for the event
- * @name: the name of the common field to return
- *
- * Returns a common field from the event by the given @name.
- * This only searches the common fields and not all field.
- */
-struct tep_format_field *
-tep_find_common_field(struct tep_event *event, const char *name)
-{
- struct tep_format_field *format;
-
- for (format = event->format.common_fields;
- format; format = format->next) {
- if (strcmp(format->name, name) == 0)
- break;
- }
-
- return format;
-}
-
-/**
- * tep_find_field - find a non-common field
- * @event: handle for the event
- * @name: the name of the non-common field
- *
- * Returns a non-common field by the given @name.
- * This does not search common fields.
- */
-struct tep_format_field *
-tep_find_field(struct tep_event *event, const char *name)
-{
- struct tep_format_field *format;
-
- for (format = event->format.fields;
- format; format = format->next) {
- if (strcmp(format->name, name) == 0)
- break;
- }
-
- return format;
-}
-
-/**
- * tep_find_any_field - find any field by name
- * @event: handle for the event
- * @name: the name of the field
- *
- * Returns a field by the given @name.
- * This searches the common field names first, then
- * the non-common ones if a common one was not found.
- */
-struct tep_format_field *
-tep_find_any_field(struct tep_event *event, const char *name)
-{
- struct tep_format_field *format;
-
- format = tep_find_common_field(event, name);
- if (format)
- return format;
- return tep_find_field(event, name);
-}
-
-/**
- * tep_read_number - read a number from data
- * @tep: a handle to the trace event parser context
- * @ptr: the raw data
- * @size: the size of the data that holds the number
- *
- * Returns the number (converted to host) from the
- * raw data.
- */
-unsigned long long tep_read_number(struct tep_handle *tep,
- const void *ptr, int size)
-{
- unsigned long long val;
-
- switch (size) {
- case 1:
- return *(unsigned char *)ptr;
- case 2:
- return data2host2(tep, *(unsigned short *)ptr);
- case 4:
- return data2host4(tep, *(unsigned int *)ptr);
- case 8:
- memcpy(&val, (ptr), sizeof(unsigned long long));
- return data2host8(tep, val);
- default:
- /* BUG! */
- return 0;
- }
-}
-
-/**
- * tep_read_number_field - read a number from data
- * @field: a handle to the field
- * @data: the raw data to read
- * @value: the value to place the number in
- *
- * Reads raw data according to a field offset and size,
- * and translates it into @value.
- *
- * Returns 0 on success, -1 otherwise.
- */
-int tep_read_number_field(struct tep_format_field *field, const void *data,
- unsigned long long *value)
-{
- if (!field)
- return -1;
- switch (field->size) {
- case 1:
- case 2:
- case 4:
- case 8:
- *value = tep_read_number(field->event->tep,
- data + field->offset, field->size);
- return 0;
- default:
- return -1;
- }
-}
-
-static int get_common_info(struct tep_handle *tep,
- const char *type, int *offset, int *size)
-{
- struct tep_event *event;
- struct tep_format_field *field;
-
- /*
- * All events should have the same common elements.
- * Pick any event to find where the type is;
- */
- if (!tep->events) {
- do_warning("no event_list!");
- return -1;
- }
-
- event = tep->events[0];
- field = tep_find_common_field(event, type);
- if (!field)
- return -1;
-
- *offset = field->offset;
- *size = field->size;
-
- return 0;
-}
-
-static int __parse_common(struct tep_handle *tep, void *data,
- int *size, int *offset, const char *name)
-{
- int ret;
-
- if (!*size) {
- ret = get_common_info(tep, name, offset, size);
- if (ret < 0)
- return ret;
- }
- return tep_read_number(tep, data + *offset, *size);
-}
-
-static int trace_parse_common_type(struct tep_handle *tep, void *data)
-{
- return __parse_common(tep, data,
- &tep->type_size, &tep->type_offset,
- "common_type");
-}
-
-static int parse_common_pid(struct tep_handle *tep, void *data)
-{
- return __parse_common(tep, data,
- &tep->pid_size, &tep->pid_offset,
- "common_pid");
-}
-
-static int parse_common_pc(struct tep_handle *tep, void *data)
-{
- return __parse_common(tep, data,
- &tep->pc_size, &tep->pc_offset,
- "common_preempt_count");
-}
-
-static int parse_common_flags(struct tep_handle *tep, void *data)
-{
- return __parse_common(tep, data,
- &tep->flags_size, &tep->flags_offset,
- "common_flags");
-}
-
-static int parse_common_lock_depth(struct tep_handle *tep, void *data)
-{
- return __parse_common(tep, data,
- &tep->ld_size, &tep->ld_offset,
- "common_lock_depth");
-}
-
-static int parse_common_migrate_disable(struct tep_handle *tep, void *data)
-{
- return __parse_common(tep, data,
- &tep->ld_size, &tep->ld_offset,
- "common_migrate_disable");
-}
-
-static int events_id_cmp(const void *a, const void *b);
-
-/**
- * tep_find_event - find an event by given id
- * @tep: a handle to the trace event parser context
- * @id: the id of the event
- *
- * Returns an event that has a given @id.
- */
-struct tep_event *tep_find_event(struct tep_handle *tep, int id)
-{
- struct tep_event **eventptr;
- struct tep_event key;
- struct tep_event *pkey = &key;
-
- /* Check cache first */
- if (tep->last_event && tep->last_event->id == id)
- return tep->last_event;
-
- key.id = id;
-
- eventptr = bsearch(&pkey, tep->events, tep->nr_events,
- sizeof(*tep->events), events_id_cmp);
-
- if (eventptr) {
- tep->last_event = *eventptr;
- return *eventptr;
- }
-
- return NULL;
-}
-
-/**
- * tep_find_event_by_name - find an event by given name
- * @tep: a handle to the trace event parser context
- * @sys: the system name to search for
- * @name: the name of the event to search for
- *
- * This returns an event with a given @name and under the system
- * @sys. If @sys is NULL the first event with @name is returned.
- */
-struct tep_event *
-tep_find_event_by_name(struct tep_handle *tep,
- const char *sys, const char *name)
-{
- struct tep_event *event = NULL;
- int i;
-
- if (tep->last_event &&
- strcmp(tep->last_event->name, name) == 0 &&
- (!sys || strcmp(tep->last_event->system, sys) == 0))
- return tep->last_event;
-
- for (i = 0; i < tep->nr_events; i++) {
- event = tep->events[i];
- if (strcmp(event->name, name) == 0) {
- if (!sys)
- break;
- if (strcmp(event->system, sys) == 0)
- break;
- }
- }
- if (i == tep->nr_events)
- event = NULL;
-
- tep->last_event = event;
- return event;
-}
-
-static unsigned long long
-eval_num_arg(void *data, int size, struct tep_event *event, struct tep_print_arg *arg)
-{
- struct tep_handle *tep = event->tep;
- unsigned long long val = 0;
- unsigned long long left, right;
- struct tep_print_arg *typearg = NULL;
- struct tep_print_arg *larg;
- unsigned long offset;
- unsigned int field_size;
-
- switch (arg->type) {
- case TEP_PRINT_NULL:
- /* ?? */
- return 0;
- case TEP_PRINT_ATOM:
- return strtoull(arg->atom.atom, NULL, 0);
- case TEP_PRINT_FIELD:
- if (!arg->field.field) {
- arg->field.field = tep_find_any_field(event, arg->field.name);
- if (!arg->field.field)
- goto out_warning_field;
-
- }
- /* must be a number */
- val = tep_read_number(tep, data + arg->field.field->offset,
- arg->field.field->size);
- break;
- case TEP_PRINT_FLAGS:
- case TEP_PRINT_SYMBOL:
- case TEP_PRINT_INT_ARRAY:
- case TEP_PRINT_HEX:
- case TEP_PRINT_HEX_STR:
- break;
- case TEP_PRINT_TYPE:
- val = eval_num_arg(data, size, event, arg->typecast.item);
- return eval_type(val, arg, 0);
- case TEP_PRINT_STRING:
- case TEP_PRINT_BSTRING:
- case TEP_PRINT_BITMASK:
- return 0;
- case TEP_PRINT_FUNC: {
- struct trace_seq s;
- trace_seq_init(&s);
- val = process_defined_func(&s, data, size, event, arg);
- trace_seq_destroy(&s);
- return val;
- }
- case TEP_PRINT_OP:
- if (strcmp(arg->op.op, "[") == 0) {
- /*
- * Arrays are special, since we don't want
- * to read the arg as is.
- */
- right = eval_num_arg(data, size, event, arg->op.right);
-
- /* handle typecasts */
- larg = arg->op.left;
- while (larg->type == TEP_PRINT_TYPE) {
- if (!typearg)
- typearg = larg;
- larg = larg->typecast.item;
- }
-
- /* Default to long size */
- field_size = tep->long_size;
-
- switch (larg->type) {
- case TEP_PRINT_DYNAMIC_ARRAY:
- offset = tep_read_number(tep,
- data + larg->dynarray.field->offset,
- larg->dynarray.field->size);
- if (larg->dynarray.field->elementsize)
- field_size = larg->dynarray.field->elementsize;
- /*
- * The actual length of the dynamic array is stored
- * in the top half of the field, and the offset
- * is in the bottom half of the 32 bit field.
- */
- offset &= 0xffff;
- offset += right;
- break;
- case TEP_PRINT_FIELD:
- if (!larg->field.field) {
- larg->field.field =
- tep_find_any_field(event, larg->field.name);
- if (!larg->field.field) {
- arg = larg;
- goto out_warning_field;
- }
- }
- field_size = larg->field.field->elementsize;
- offset = larg->field.field->offset +
- right * larg->field.field->elementsize;
- break;
- default:
- goto default_op; /* oops, all bets off */
- }
- val = tep_read_number(tep,
- data + offset, field_size);
- if (typearg)
- val = eval_type(val, typearg, 1);
- break;
- } else if (strcmp(arg->op.op, "?") == 0) {
- left = eval_num_arg(data, size, event, arg->op.left);
- arg = arg->op.right;
- if (left)
- val = eval_num_arg(data, size, event, arg->op.left);
- else
- val = eval_num_arg(data, size, event, arg->op.right);
- break;
- }
- default_op:
- left = eval_num_arg(data, size, event, arg->op.left);
- right = eval_num_arg(data, size, event, arg->op.right);
- switch (arg->op.op[0]) {
- case '!':
- switch (arg->op.op[1]) {
- case 0:
- val = !right;
- break;
- case '=':
- val = left != right;
- break;
- default:
- goto out_warning_op;
- }
- break;
- case '~':
- val = ~right;
- break;
- case '|':
- if (arg->op.op[1])
- val = left || right;
- else
- val = left | right;
- break;
- case '&':
- if (arg->op.op[1])
- val = left && right;
- else
- val = left & right;
- break;
- case '<':
- switch (arg->op.op[1]) {
- case 0:
- val = left < right;
- break;
- case '<':
- val = left << right;
- break;
- case '=':
- val = left <= right;
- break;
- default:
- goto out_warning_op;
- }
- break;
- case '>':
- switch (arg->op.op[1]) {
- case 0:
- val = left > right;
- break;
- case '>':
- val = left >> right;
- break;
- case '=':
- val = left >= right;
- break;
- default:
- goto out_warning_op;
- }
- break;
- case '=':
- if (arg->op.op[1] != '=')
- goto out_warning_op;
-
- val = left == right;
- break;
- case '-':
- val = left - right;
- break;
- case '+':
- val = left + right;
- break;
- case '/':
- val = left / right;
- break;
- case '%':
- val = left % right;
- break;
- case '*':
- val = left * right;
- break;
- default:
- goto out_warning_op;
- }
- break;
- case TEP_PRINT_DYNAMIC_ARRAY_LEN:
- offset = tep_read_number(tep,
- data + arg->dynarray.field->offset,
- arg->dynarray.field->size);
- /*
- * The total allocated length of the dynamic array is
- * stored in the top half of the field, and the offset
- * is in the bottom half of the 32 bit field.
- */
- val = (unsigned long long)(offset >> 16);
- break;
- case TEP_PRINT_DYNAMIC_ARRAY:
- /* Without [], we pass the address to the dynamic data */
- offset = tep_read_number(tep,
- data + arg->dynarray.field->offset,
- arg->dynarray.field->size);
- /*
- * The total allocated length of the dynamic array is
- * stored in the top half of the field, and the offset
- * is in the bottom half of the 32 bit field.
- */
- offset &= 0xffff;
- val = (unsigned long long)((unsigned long)data + offset);
- break;
- default: /* not sure what to do there */
- return 0;
- }
- return val;
-
-out_warning_op:
- do_warning_event(event, "%s: unknown op '%s'", __func__, arg->op.op);
- return 0;
-
-out_warning_field:
- do_warning_event(event, "%s: field %s not found",
- __func__, arg->field.name);
- return 0;
-}
-
-struct flag {
- const char *name;
- unsigned long long value;
-};
-
-static const struct flag flags[] = {
- { "HI_SOFTIRQ", 0 },
- { "TIMER_SOFTIRQ", 1 },
- { "NET_TX_SOFTIRQ", 2 },
- { "NET_RX_SOFTIRQ", 3 },
- { "BLOCK_SOFTIRQ", 4 },
- { "IRQ_POLL_SOFTIRQ", 5 },
- { "TASKLET_SOFTIRQ", 6 },
- { "SCHED_SOFTIRQ", 7 },
- { "HRTIMER_SOFTIRQ", 8 },
- { "RCU_SOFTIRQ", 9 },
-
- { "HRTIMER_NORESTART", 0 },
- { "HRTIMER_RESTART", 1 },
-};
-
-static long long eval_flag(const char *flag)
-{
- int i;
-
- /*
- * Some flags in the format files do not get converted.
- * If the flag is not numeric, see if it is something that
- * we already know about.
- */
- if (isdigit(flag[0]))
- return strtoull(flag, NULL, 0);
-
- for (i = 0; i < (int)(sizeof(flags)/sizeof(flags[0])); i++)
- if (strcmp(flags[i].name, flag) == 0)
- return flags[i].value;
-
- return -1LL;
-}
-
-static void print_str_to_seq(struct trace_seq *s, const char *format,
- int len_arg, const char *str)
-{
- if (len_arg >= 0)
- trace_seq_printf(s, format, len_arg, str);
- else
- trace_seq_printf(s, format, str);
-}
-
-static void print_bitmask_to_seq(struct tep_handle *tep,
- struct trace_seq *s, const char *format,
- int len_arg, const void *data, int size)
-{
- int nr_bits = size * 8;
- int str_size = (nr_bits + 3) / 4;
- int len = 0;
- char buf[3];
- char *str;
- int index;
- int i;
-
- /*
- * The kernel likes to put in commas every 32 bits, we
- * can do the same.
- */
- str_size += (nr_bits - 1) / 32;
-
- str = malloc(str_size + 1);
- if (!str) {
- do_warning("%s: not enough memory!", __func__);
- return;
- }
- str[str_size] = 0;
-
- /* Start out with -2 for the two chars per byte */
- for (i = str_size - 2; i >= 0; i -= 2) {
- /*
- * data points to a bit mask of size bytes.
- * In the kernel, this is an array of long words, thus
- * endianness is very important.
- */
- if (tep->file_bigendian)
- index = size - (len + 1);
- else
- index = len;
-
- snprintf(buf, 3, "%02x", *((unsigned char *)data + index));
- memcpy(str + i, buf, 2);
- len++;
- if (!(len & 3) && i > 0) {
- i--;
- str[i] = ',';
- }
- }
-
- if (len_arg >= 0)
- trace_seq_printf(s, format, len_arg, str);
- else
- trace_seq_printf(s, format, str);
-
- free(str);
-}
-
-static void print_str_arg(struct trace_seq *s, void *data, int size,
- struct tep_event *event, const char *format,
- int len_arg, struct tep_print_arg *arg)
-{
- struct tep_handle *tep = event->tep;
- struct tep_print_flag_sym *flag;
- struct tep_format_field *field;
- struct printk_map *printk;
- long long val, fval;
- unsigned long long addr;
- char *str;
- unsigned char *hex;
- int print;
- int i, len;
-
- switch (arg->type) {
- case TEP_PRINT_NULL:
- /* ?? */
- return;
- case TEP_PRINT_ATOM:
- print_str_to_seq(s, format, len_arg, arg->atom.atom);
- return;
- case TEP_PRINT_FIELD:
- field = arg->field.field;
- if (!field) {
- field = tep_find_any_field(event, arg->field.name);
- if (!field) {
- str = arg->field.name;
- goto out_warning_field;
- }
- arg->field.field = field;
- }
- /* Zero sized fields, mean the rest of the data */
- len = field->size ? : size - field->offset;
-
- /*
- * Some events pass in pointers. If this is not an array
- * and the size is the same as long_size, assume that it
- * is a pointer.
- */
- if (!(field->flags & TEP_FIELD_IS_ARRAY) &&
- field->size == tep->long_size) {
-
- /* Handle heterogeneous recording and processing
- * architectures
- *
- * CASE I:
- * Traces recorded on 32-bit devices (32-bit
- * addressing) and processed on 64-bit devices:
- * In this case, only 32 bits should be read.
- *
- * CASE II:
- * Traces recorded on 64 bit devices and processed
- * on 32-bit devices:
- * In this case, 64 bits must be read.
- */
- addr = (tep->long_size == 8) ?
- *(unsigned long long *)(data + field->offset) :
- (unsigned long long)*(unsigned int *)(data + field->offset);
-
- /* Check if it matches a print format */
- printk = find_printk(tep, addr);
- if (printk)
- trace_seq_puts(s, printk->printk);
- else
- trace_seq_printf(s, "%llx", addr);
- break;
- }
- str = malloc(len + 1);
- if (!str) {
- do_warning_event(event, "%s: not enough memory!",
- __func__);
- return;
- }
- memcpy(str, data + field->offset, len);
- str[len] = 0;
- print_str_to_seq(s, format, len_arg, str);
- free(str);
- break;
- case TEP_PRINT_FLAGS:
- val = eval_num_arg(data, size, event, arg->flags.field);
- print = 0;
- for (flag = arg->flags.flags; flag; flag = flag->next) {
- fval = eval_flag(flag->value);
- if (!val && fval < 0) {
- print_str_to_seq(s, format, len_arg, flag->str);
- break;
- }
- if (fval > 0 && (val & fval) == fval) {
- if (print && arg->flags.delim)
- trace_seq_puts(s, arg->flags.delim);
- print_str_to_seq(s, format, len_arg, flag->str);
- print = 1;
- val &= ~fval;
- }
- }
- if (val) {
- if (print && arg->flags.delim)
- trace_seq_puts(s, arg->flags.delim);
- trace_seq_printf(s, "0x%llx", val);
- }
- break;
- case TEP_PRINT_SYMBOL:
- val = eval_num_arg(data, size, event, arg->symbol.field);
- for (flag = arg->symbol.symbols; flag; flag = flag->next) {
- fval = eval_flag(flag->value);
- if (val == fval) {
- print_str_to_seq(s, format, len_arg, flag->str);
- break;
- }
- }
- if (!flag)
- trace_seq_printf(s, "0x%llx", val);
- break;
- case TEP_PRINT_HEX:
- case TEP_PRINT_HEX_STR:
- if (arg->hex.field->type == TEP_PRINT_DYNAMIC_ARRAY) {
- unsigned long offset;
- offset = tep_read_number(tep,
- data + arg->hex.field->dynarray.field->offset,
- arg->hex.field->dynarray.field->size);
- hex = data + (offset & 0xffff);
- } else {
- field = arg->hex.field->field.field;
- if (!field) {
- str = arg->hex.field->field.name;
- field = tep_find_any_field(event, str);
- if (!field)
- goto out_warning_field;
- arg->hex.field->field.field = field;
- }
- hex = data + field->offset;
- }
- len = eval_num_arg(data, size, event, arg->hex.size);
- for (i = 0; i < len; i++) {
- if (i && arg->type == TEP_PRINT_HEX)
- trace_seq_putc(s, ' ');
- trace_seq_printf(s, "%02x", hex[i]);
- }
- break;
-
- case TEP_PRINT_INT_ARRAY: {
- void *num;
- int el_size;
-
- if (arg->int_array.field->type == TEP_PRINT_DYNAMIC_ARRAY) {
- unsigned long offset;
- struct tep_format_field *field =
- arg->int_array.field->dynarray.field;
- offset = tep_read_number(tep,
- data + field->offset,
- field->size);
- num = data + (offset & 0xffff);
- } else {
- field = arg->int_array.field->field.field;
- if (!field) {
- str = arg->int_array.field->field.name;
- field = tep_find_any_field(event, str);
- if (!field)
- goto out_warning_field;
- arg->int_array.field->field.field = field;
- }
- num = data + field->offset;
- }
- len = eval_num_arg(data, size, event, arg->int_array.count);
- el_size = eval_num_arg(data, size, event,
- arg->int_array.el_size);
- for (i = 0; i < len; i++) {
- if (i)
- trace_seq_putc(s, ' ');
-
- if (el_size == 1) {
- trace_seq_printf(s, "%u", *(uint8_t *)num);
- } else if (el_size == 2) {
- trace_seq_printf(s, "%u", *(uint16_t *)num);
- } else if (el_size == 4) {
- trace_seq_printf(s, "%u", *(uint32_t *)num);
- } else if (el_size == 8) {
- trace_seq_printf(s, "%"PRIu64, *(uint64_t *)num);
- } else {
- trace_seq_printf(s, "BAD SIZE:%d 0x%x",
- el_size, *(uint8_t *)num);
- el_size = 1;
- }
-
- num += el_size;
- }
- break;
- }
- case TEP_PRINT_TYPE:
- break;
- case TEP_PRINT_STRING: {
- int str_offset;
-
- if (!arg->string.field)
- arg->string.field = tep_find_any_field(event, arg->string.string);
- if (!arg->string.field)
- break;
-
- str_offset = data2host4(tep,
- *(unsigned int *)(data + arg->string.field->offset));
- str_offset &= 0xffff;
- if (arg->string.field->flags & TEP_FIELD_IS_RELATIVE)
- str_offset += arg->string.field->offset + arg->string.field->size;
- print_str_to_seq(s, format, len_arg, ((char *)data) + str_offset);
- break;
- }
- case TEP_PRINT_BSTRING:
- print_str_to_seq(s, format, len_arg, arg->string.string);
- break;
- case TEP_PRINT_BITMASK: {
- int bitmask_offset;
- int bitmask_size;
-
- if (!arg->bitmask.field)
- arg->bitmask.field = tep_find_any_field(event, arg->bitmask.bitmask);
- if (!arg->bitmask.field)
- break;
- bitmask_offset = data2host4(tep,
- *(unsigned int *)(data + arg->bitmask.field->offset));
- bitmask_size = bitmask_offset >> 16;
- bitmask_offset &= 0xffff;
- if (arg->bitmask.field->flags & TEP_FIELD_IS_RELATIVE)
- bitmask_offset += arg->bitmask.field->offset + arg->bitmask.field->size;
- print_bitmask_to_seq(tep, s, format, len_arg,
- data + bitmask_offset, bitmask_size);
- break;
- }
- case TEP_PRINT_OP:
- /*
- * The only op for string should be ? :
- */
- if (arg->op.op[0] != '?')
- return;
- val = eval_num_arg(data, size, event, arg->op.left);
- if (val)
- print_str_arg(s, data, size, event,
- format, len_arg, arg->op.right->op.left);
- else
- print_str_arg(s, data, size, event,
- format, len_arg, arg->op.right->op.right);
- break;
- case TEP_PRINT_FUNC:
- process_defined_func(s, data, size, event, arg);
- break;
- default:
- /* well... */
- break;
- }
-
- return;
-
-out_warning_field:
- do_warning_event(event, "%s: field %s not found",
- __func__, arg->field.name);
-}
-
-static unsigned long long
-process_defined_func(struct trace_seq *s, void *data, int size,
- struct tep_event *event, struct tep_print_arg *arg)
-{
- struct tep_function_handler *func_handle = arg->func.func;
- struct func_params *param;
- unsigned long long *args;
- unsigned long long ret;
- struct tep_print_arg *farg;
- struct trace_seq str;
- struct save_str {
- struct save_str *next;
- char *str;
- } *strings = NULL, *string;
- int i;
-
- if (!func_handle->nr_args) {
- ret = (*func_handle->func)(s, NULL);
- goto out;
- }
-
- farg = arg->func.args;
- param = func_handle->params;
-
- ret = ULLONG_MAX;
- args = malloc(sizeof(*args) * func_handle->nr_args);
- if (!args)
- goto out;
-
- for (i = 0; i < func_handle->nr_args; i++) {
- switch (param->type) {
- case TEP_FUNC_ARG_INT:
- case TEP_FUNC_ARG_LONG:
- case TEP_FUNC_ARG_PTR:
- args[i] = eval_num_arg(data, size, event, farg);
- break;
- case TEP_FUNC_ARG_STRING:
- trace_seq_init(&str);
- print_str_arg(&str, data, size, event, "%s", -1, farg);
- trace_seq_terminate(&str);
- string = malloc(sizeof(*string));
- if (!string) {
- do_warning_event(event, "%s(%d): malloc str",
- __func__, __LINE__);
- goto out_free;
- }
- string->next = strings;
- string->str = strdup(str.buffer);
- if (!string->str) {
- free(string);
- do_warning_event(event, "%s(%d): malloc str",
- __func__, __LINE__);
- goto out_free;
- }
- args[i] = (uintptr_t)string->str;
- strings = string;
- trace_seq_destroy(&str);
- break;
- default:
- /*
- * Something went totally wrong, this is not
- * an input error, something in this code broke.
- */
- do_warning_event(event, "Unexpected end of arguments\n");
- goto out_free;
- }
- farg = farg->next;
- param = param->next;
- }
-
- ret = (*func_handle->func)(s, args);
-out_free:
- free(args);
- while (strings) {
- string = strings;
- strings = string->next;
- free(string->str);
- free(string);
- }
-
- out:
- /* TBD : handle return type here */
- return ret;
-}
-
-static void free_args(struct tep_print_arg *args)
-{
- struct tep_print_arg *next;
-
- while (args) {
- next = args->next;
-
- free_arg(args);
- args = next;
- }
-}
-
-static struct tep_print_arg *make_bprint_args(char *fmt, void *data, int size, struct tep_event *event)
-{
- struct tep_handle *tep = event->tep;
- struct tep_format_field *field, *ip_field;
- struct tep_print_arg *args, *arg, **next;
- unsigned long long ip, val;
- char *ptr;
- void *bptr;
- int vsize = 0;
-
- field = tep->bprint_buf_field;
- ip_field = tep->bprint_ip_field;
-
- if (!field) {
- field = tep_find_field(event, "buf");
- if (!field) {
- do_warning_event(event, "can't find buffer field for binary printk");
- return NULL;
- }
- ip_field = tep_find_field(event, "ip");
- if (!ip_field) {
- do_warning_event(event, "can't find ip field for binary printk");
- return NULL;
- }
- tep->bprint_buf_field = field;
- tep->bprint_ip_field = ip_field;
- }
-
- ip = tep_read_number(tep, data + ip_field->offset, ip_field->size);
-
- /*
- * The first arg is the IP pointer.
- */
- args = alloc_arg();
- if (!args) {
- do_warning_event(event, "%s(%d): not enough memory!",
- __func__, __LINE__);
- return NULL;
- }
- arg = args;
- arg->next = NULL;
- next = &arg->next;
-
- arg->type = TEP_PRINT_ATOM;
-
- if (asprintf(&arg->atom.atom, "%lld", ip) < 0)
- goto out_free;
-
- /* skip the first "%ps: " */
- for (ptr = fmt + 5, bptr = data + field->offset;
- bptr < data + size && *ptr; ptr++) {
- int ls = 0;
-
- if (*ptr == '%') {
- process_again:
- ptr++;
- switch (*ptr) {
- case '%':
- break;
- case 'l':
- ls++;
- goto process_again;
- case 'L':
- ls = 2;
- goto process_again;
- case '0' ... '9':
- goto process_again;
- case '.':
- goto process_again;
- case 'z':
- case 'Z':
- ls = 1;
- goto process_again;
- case 'p':
- ls = 1;
- if (isalnum(ptr[1])) {
- ptr++;
- /* Check for special pointers */
- switch (*ptr) {
- case 's':
- case 'S':
- case 'x':
- break;
- case 'f':
- case 'F':
- /*
- * Pre-5.5 kernels use %pf and
- * %pF for printing symbols
- * while kernels since 5.5 use
- * %pfw for fwnodes. So check
- * %p[fF] isn't followed by 'w'.
- */
- if (ptr[1] != 'w')
- break;
- /* fall through */
- default:
- /*
- * Older kernels do not process
- * dereferenced pointers.
- * Only process if the pointer
- * value is a printable.
- */
- if (isprint(*(char *)bptr))
- goto process_string;
- }
- }
- /* fall through */
- case 'd':
- case 'u':
- case 'i':
- case 'x':
- case 'X':
- case 'o':
- switch (ls) {
- case 0:
- vsize = 4;
- break;
- case 1:
- vsize = tep->long_size;
- break;
- case 2:
- vsize = 8;
- break;
- default:
- vsize = ls; /* ? */
- break;
- }
- /* fall through */
- case '*':
- if (*ptr == '*')
- vsize = 4;
-
- /* the pointers are always 4 bytes aligned */
- bptr = (void *)(((unsigned long)bptr + 3) &
- ~3);
- val = tep_read_number(tep, bptr, vsize);
- bptr += vsize;
- arg = alloc_arg();
- if (!arg) {
- do_warning_event(event, "%s(%d): not enough memory!",
- __func__, __LINE__);
- goto out_free;
- }
- arg->next = NULL;
- arg->type = TEP_PRINT_ATOM;
- if (asprintf(&arg->atom.atom, "%lld", val) < 0) {
- free(arg);
- goto out_free;
- }
- *next = arg;
- next = &arg->next;
- /*
- * The '*' case means that an arg is used as the length.
- * We need to continue to figure out for what.
- */
- if (*ptr == '*')
- goto process_again;
-
- break;
- case 's':
- process_string:
- arg = alloc_arg();
- if (!arg) {
- do_warning_event(event, "%s(%d): not enough memory!",
- __func__, __LINE__);
- goto out_free;
- }
- arg->next = NULL;
- arg->type = TEP_PRINT_BSTRING;
- arg->string.string = strdup(bptr);
- if (!arg->string.string)
- goto out_free;
- bptr += strlen(bptr) + 1;
- *next = arg;
- next = &arg->next;
- default:
- break;
- }
- }
- }
-
- return args;
-
-out_free:
- free_args(args);
- return NULL;
-}
-
-static char *
-get_bprint_format(void *data, int size __maybe_unused,
- struct tep_event *event)
-{
- struct tep_handle *tep = event->tep;
- unsigned long long addr;
- struct tep_format_field *field;
- struct printk_map *printk;
- char *format;
-
- field = tep->bprint_fmt_field;
-
- if (!field) {
- field = tep_find_field(event, "fmt");
- if (!field) {
- do_warning_event(event, "can't find format field for binary printk");
- return NULL;
- }
- tep->bprint_fmt_field = field;
- }
-
- addr = tep_read_number(tep, data + field->offset, field->size);
-
- printk = find_printk(tep, addr);
- if (!printk) {
- if (asprintf(&format, "%%ps: (NO FORMAT FOUND at %llx)\n", addr) < 0)
- return NULL;
- return format;
- }
-
- if (asprintf(&format, "%s: %s", "%ps", printk->printk) < 0)
- return NULL;
-
- return format;
-}
-
-static int print_mac_arg(struct trace_seq *s, const char *format,
- void *data, int size, struct tep_event *event,
- struct tep_print_arg *arg)
-{
- const char *fmt = "%.2x:%.2x:%.2x:%.2x:%.2x:%.2x";
- bool reverse = false;
- unsigned char *buf;
- int ret = 0;
-
- if (arg->type == TEP_PRINT_FUNC) {
- process_defined_func(s, data, size, event, arg);
- return 0;
- }
-
- if (arg->type != TEP_PRINT_FIELD) {
- trace_seq_printf(s, "ARG TYPE NOT FIELD BUT %d",
- arg->type);
- return 0;
- }
-
- if (format[0] == 'm') {
- fmt = "%.2x%.2x%.2x%.2x%.2x%.2x";
- } else if (format[0] == 'M' && format[1] == 'F') {
- fmt = "%.2x-%.2x-%.2x-%.2x-%.2x-%.2x";
- ret++;
- }
- if (format[1] == 'R') {
- reverse = true;
- ret++;
- }
-
- if (!arg->field.field) {
- arg->field.field =
- tep_find_any_field(event, arg->field.name);
- if (!arg->field.field) {
- do_warning_event(event, "%s: field %s not found",
- __func__, arg->field.name);
- return ret;
- }
- }
- if (arg->field.field->size != 6) {
- trace_seq_printf(s, "INVALIDMAC");
- return ret;
- }
-
- buf = data + arg->field.field->offset;
- if (reverse)
- trace_seq_printf(s, fmt, buf[5], buf[4], buf[3], buf[2], buf[1], buf[0]);
- else
- trace_seq_printf(s, fmt, buf[0], buf[1], buf[2], buf[3], buf[4], buf[5]);
-
- return ret;
-}
-
-static int parse_ip4_print_args(struct tep_handle *tep,
- const char *ptr, bool *reverse)
-{
- int ret = 0;
-
- *reverse = false;
-
- /* hnbl */
- switch (*ptr) {
- case 'h':
- if (tep->file_bigendian)
- *reverse = false;
- else
- *reverse = true;
- ret++;
- break;
- case 'l':
- *reverse = true;
- ret++;
- break;
- case 'n':
- case 'b':
- ret++;
- /* fall through */
- default:
- *reverse = false;
- break;
- }
-
- return ret;
-}
-
-static void print_ip4_addr(struct trace_seq *s, char i, bool reverse, unsigned char *buf)
-{
- const char *fmt;
-
- if (i == 'i')
- fmt = "%03d.%03d.%03d.%03d";
- else
- fmt = "%d.%d.%d.%d";
-
- if (reverse)
- trace_seq_printf(s, fmt, buf[3], buf[2], buf[1], buf[0]);
- else
- trace_seq_printf(s, fmt, buf[0], buf[1], buf[2], buf[3]);
-
-}
-
-static inline bool ipv6_addr_v4mapped(const struct in6_addr *a)
-{
- return ((unsigned long)(a->s6_addr32[0] | a->s6_addr32[1]) |
- (unsigned long)(a->s6_addr32[2] ^ htonl(0x0000ffff))) == 0UL;
-}
-
-static inline bool ipv6_addr_is_isatap(const struct in6_addr *addr)
-{
- return (addr->s6_addr32[2] | htonl(0x02000000)) == htonl(0x02005EFE);
-}
-
-static void print_ip6c_addr(struct trace_seq *s, unsigned char *addr)
-{
- int i, j, range;
- unsigned char zerolength[8];
- int longest = 1;
- int colonpos = -1;
- uint16_t word;
- uint8_t hi, lo;
- bool needcolon = false;
- bool useIPv4;
- struct in6_addr in6;
-
- memcpy(&in6, addr, sizeof(struct in6_addr));
-
- useIPv4 = ipv6_addr_v4mapped(&in6) || ipv6_addr_is_isatap(&in6);
-
- memset(zerolength, 0, sizeof(zerolength));
-
- if (useIPv4)
- range = 6;
- else
- range = 8;
-
- /* find position of longest 0 run */
- for (i = 0; i < range; i++) {
- for (j = i; j < range; j++) {
- if (in6.s6_addr16[j] != 0)
- break;
- zerolength[i]++;
- }
- }
- for (i = 0; i < range; i++) {
- if (zerolength[i] > longest) {
- longest = zerolength[i];
- colonpos = i;
- }
- }
- if (longest == 1) /* don't compress a single 0 */
- colonpos = -1;
-
- /* emit address */
- for (i = 0; i < range; i++) {
- if (i == colonpos) {
- if (needcolon || i == 0)
- trace_seq_printf(s, ":");
- trace_seq_printf(s, ":");
- needcolon = false;
- i += longest - 1;
- continue;
- }
- if (needcolon) {
- trace_seq_printf(s, ":");
- needcolon = false;
- }
- /* hex u16 without leading 0s */
- word = ntohs(in6.s6_addr16[i]);
- hi = word >> 8;
- lo = word & 0xff;
- if (hi)
- trace_seq_printf(s, "%x%02x", hi, lo);
- else
- trace_seq_printf(s, "%x", lo);
-
- needcolon = true;
- }
-
- if (useIPv4) {
- if (needcolon)
- trace_seq_printf(s, ":");
- print_ip4_addr(s, 'I', false, &in6.s6_addr[12]);
- }
-
- return;
-}
-
-static void print_ip6_addr(struct trace_seq *s, char i, unsigned char *buf)
-{
- int j;
-
- for (j = 0; j < 16; j += 2) {
- trace_seq_printf(s, "%02x%02x", buf[j], buf[j+1]);
- if (i == 'I' && j < 14)
- trace_seq_printf(s, ":");
- }
-}
-
-/*
- * %pi4 print an IPv4 address with leading zeros
- * %pI4 print an IPv4 address without leading zeros
- * %pi6 print an IPv6 address without colons
- * %pI6 print an IPv6 address with colons
- * %pI6c print an IPv6 address in compressed form with colons
- * %pISpc print an IP address based on sockaddr; p adds port.
- */
-static int print_ipv4_arg(struct trace_seq *s, const char *ptr, char i,
- void *data, int size, struct tep_event *event,
- struct tep_print_arg *arg)
-{
- bool reverse = false;
- unsigned char *buf;
- int ret;
-
- ret = parse_ip4_print_args(event->tep, ptr, &reverse);
-
- if (arg->type == TEP_PRINT_FUNC) {
- process_defined_func(s, data, size, event, arg);
- return ret;
- }
-
- if (arg->type != TEP_PRINT_FIELD) {
- trace_seq_printf(s, "ARG TYPE NOT FIELD BUT %d", arg->type);
- return ret;
- }
-
- if (!arg->field.field) {
- arg->field.field =
- tep_find_any_field(event, arg->field.name);
- if (!arg->field.field) {
- do_warning("%s: field %s not found",
- __func__, arg->field.name);
- return ret;
- }
- }
-
- buf = data + arg->field.field->offset;
-
- if (arg->field.field->size != 4) {
- trace_seq_printf(s, "INVALIDIPv4");
- return ret;
- }
-
- print_ip4_addr(s, i, reverse, buf);
- return ret;
-
-}
-
-static int print_ipv6_arg(struct trace_seq *s, const char *ptr, char i,
- void *data, int size, struct tep_event *event,
- struct tep_print_arg *arg)
-{
- char have_c = 0;
- unsigned char *buf;
- int rc = 0;
-
- /* pI6c */
- if (i == 'I' && *ptr == 'c') {
- have_c = 1;
- ptr++;
- rc++;
- }
-
- if (arg->type == TEP_PRINT_FUNC) {
- process_defined_func(s, data, size, event, arg);
- return rc;
- }
-
- if (arg->type != TEP_PRINT_FIELD) {
- trace_seq_printf(s, "ARG TYPE NOT FIELD BUT %d", arg->type);
- return rc;
- }
-
- if (!arg->field.field) {
- arg->field.field =
- tep_find_any_field(event, arg->field.name);
- if (!arg->field.field) {
- do_warning("%s: field %s not found",
- __func__, arg->field.name);
- return rc;
- }
- }
-
- buf = data + arg->field.field->offset;
-
- if (arg->field.field->size != 16) {
- trace_seq_printf(s, "INVALIDIPv6");
- return rc;
- }
-
- if (have_c)
- print_ip6c_addr(s, buf);
- else
- print_ip6_addr(s, i, buf);
-
- return rc;
-}
-
-static int print_ipsa_arg(struct trace_seq *s, const char *ptr, char i,
- void *data, int size, struct tep_event *event,
- struct tep_print_arg *arg)
-{
- char have_c = 0, have_p = 0;
- unsigned char *buf;
- struct sockaddr_storage *sa;
- bool reverse = false;
- int rc = 0;
- int ret;
-
- /* pISpc */
- if (i == 'I') {
- if (*ptr == 'p') {
- have_p = 1;
- ptr++;
- rc++;
- }
- if (*ptr == 'c') {
- have_c = 1;
- ptr++;
- rc++;
- }
- }
- ret = parse_ip4_print_args(event->tep, ptr, &reverse);
- ptr += ret;
- rc += ret;
-
- if (arg->type == TEP_PRINT_FUNC) {
- process_defined_func(s, data, size, event, arg);
- return rc;
- }
-
- if (arg->type != TEP_PRINT_FIELD) {
- trace_seq_printf(s, "ARG TYPE NOT FIELD BUT %d", arg->type);
- return rc;
- }
-
- if (!arg->field.field) {
- arg->field.field =
- tep_find_any_field(event, arg->field.name);
- if (!arg->field.field) {
- do_warning("%s: field %s not found",
- __func__, arg->field.name);
- return rc;
- }
- }
-
- sa = (struct sockaddr_storage *) (data + arg->field.field->offset);
-
- if (sa->ss_family == AF_INET) {
- struct sockaddr_in *sa4 = (struct sockaddr_in *) sa;
-
- if (arg->field.field->size < sizeof(struct sockaddr_in)) {
- trace_seq_printf(s, "INVALIDIPv4");
- return rc;
- }
-
- print_ip4_addr(s, i, reverse, (unsigned char *) &sa4->sin_addr);
- if (have_p)
- trace_seq_printf(s, ":%d", ntohs(sa4->sin_port));
-
-
- } else if (sa->ss_family == AF_INET6) {
- struct sockaddr_in6 *sa6 = (struct sockaddr_in6 *) sa;
-
- if (arg->field.field->size < sizeof(struct sockaddr_in6)) {
- trace_seq_printf(s, "INVALIDIPv6");
- return rc;
- }
-
- if (have_p)
- trace_seq_printf(s, "[");
-
- buf = (unsigned char *) &sa6->sin6_addr;
- if (have_c)
- print_ip6c_addr(s, buf);
- else
- print_ip6_addr(s, i, buf);
-
- if (have_p)
- trace_seq_printf(s, "]:%d", ntohs(sa6->sin6_port));
- }
-
- return rc;
-}
-
-static int print_ip_arg(struct trace_seq *s, const char *ptr,
- void *data, int size, struct tep_event *event,
- struct tep_print_arg *arg)
-{
- char i = *ptr; /* 'i' or 'I' */
- int rc = 1;
-
- /* IP version */
- ptr++;
-
- switch (*ptr) {
- case '4':
- rc += print_ipv4_arg(s, ptr + 1, i, data, size, event, arg);
- break;
- case '6':
- rc += print_ipv6_arg(s, ptr + 1, i, data, size, event, arg);
- break;
- case 'S':
- rc += print_ipsa_arg(s, ptr + 1, i, data, size, event, arg);
- break;
- default:
- return 0;
- }
-
- return rc;
-}
-
-static const int guid_index[16] = {3, 2, 1, 0, 5, 4, 7, 6, 8, 9, 10, 11, 12, 13, 14, 15};
-static const int uuid_index[16] = {0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15};
-
-static int print_uuid_arg(struct trace_seq *s, const char *ptr,
- void *data, int size, struct tep_event *event,
- struct tep_print_arg *arg)
-{
- const int *index = uuid_index;
- char *format = "%02x";
- int ret = 0;
- char *buf;
- int i;
-
- switch (*(ptr + 1)) {
- case 'L':
- format = "%02X";
- /* fall through */
- case 'l':
- index = guid_index;
- ret++;
- break;
- case 'B':
- format = "%02X";
- /* fall through */
- case 'b':
- ret++;
- break;
- }
-
- if (arg->type == TEP_PRINT_FUNC) {
- process_defined_func(s, data, size, event, arg);
- return ret;
- }
-
- if (arg->type != TEP_PRINT_FIELD) {
- trace_seq_printf(s, "ARG TYPE NOT FIELD BUT %d", arg->type);
- return ret;
- }
-
- if (!arg->field.field) {
- arg->field.field =
- tep_find_any_field(event, arg->field.name);
- if (!arg->field.field) {
- do_warning("%s: field %s not found",
- __func__, arg->field.name);
- return ret;
- }
- }
-
- if (arg->field.field->size != 16) {
- trace_seq_printf(s, "INVALIDUUID");
- return ret;
- }
-
- buf = data + arg->field.field->offset;
-
- for (i = 0; i < 16; i++) {
- trace_seq_printf(s, format, buf[index[i]] & 0xff);
- switch (i) {
- case 3:
- case 5:
- case 7:
- case 9:
- trace_seq_printf(s, "-");
- break;
- }
- }
-
- return ret;
-}
-
-static int print_raw_buff_arg(struct trace_seq *s, const char *ptr,
- void *data, int size, struct tep_event *event,
- struct tep_print_arg *arg, int print_len)
-{
- int plen = print_len;
- char *delim = " ";
- int ret = 0;
- char *buf;
- int i;
- unsigned long offset;
- int arr_len;
-
- switch (*(ptr + 1)) {
- case 'C':
- delim = ":";
- ret++;
- break;
- case 'D':
- delim = "-";
- ret++;
- break;
- case 'N':
- delim = "";
- ret++;
- break;
- }
-
- if (arg->type == TEP_PRINT_FUNC) {
- process_defined_func(s, data, size, event, arg);
- return ret;
- }
-
- if (arg->type != TEP_PRINT_DYNAMIC_ARRAY) {
- trace_seq_printf(s, "ARG TYPE NOT FIELD BUT %d", arg->type);
- return ret;
- }
-
- offset = tep_read_number(event->tep,
- data + arg->dynarray.field->offset,
- arg->dynarray.field->size);
- arr_len = (unsigned long long)(offset >> 16);
- buf = data + (offset & 0xffff);
-
- if (arr_len < plen)
- plen = arr_len;
-
- if (plen < 1)
- return ret;
-
- trace_seq_printf(s, "%02x", buf[0] & 0xff);
- for (i = 1; i < plen; i++)
- trace_seq_printf(s, "%s%02x", delim, buf[i] & 0xff);
-
- return ret;
-}
-
-static int is_printable_array(char *p, unsigned int len)
-{
- unsigned int i;
-
- for (i = 0; i < len && p[i]; i++)
- if (!isprint(p[i]) && !isspace(p[i]))
- return 0;
- return 1;
-}
-
-void tep_print_field(struct trace_seq *s, void *data,
- struct tep_format_field *field)
-{
- unsigned long long val;
- unsigned int offset, len, i;
- struct tep_handle *tep = field->event->tep;
-
- if (field->flags & TEP_FIELD_IS_ARRAY) {
- offset = field->offset;
- len = field->size;
- if (field->flags & TEP_FIELD_IS_DYNAMIC) {
- val = tep_read_number(tep, data + offset, len);
- offset = val;
- len = offset >> 16;
- offset &= 0xffff;
- if (field->flags & TEP_FIELD_IS_RELATIVE)
- offset += field->offset + field->size;
- }
- if (field->flags & TEP_FIELD_IS_STRING &&
- is_printable_array(data + offset, len)) {
- trace_seq_printf(s, "%s", (char *)data + offset);
- } else {
- trace_seq_puts(s, "ARRAY[");
- for (i = 0; i < len; i++) {
- if (i)
- trace_seq_puts(s, ", ");
- trace_seq_printf(s, "%02x",
- *((unsigned char *)data + offset + i));
- }
- trace_seq_putc(s, ']');
- field->flags &= ~TEP_FIELD_IS_STRING;
- }
- } else {
- val = tep_read_number(tep, data + field->offset,
- field->size);
- if (field->flags & TEP_FIELD_IS_POINTER) {
- trace_seq_printf(s, "0x%llx", val);
- } else if (field->flags & TEP_FIELD_IS_SIGNED) {
- switch (field->size) {
- case 4:
- /*
- * If field is long then print it in hex.
- * A long usually stores pointers.
- */
- if (field->flags & TEP_FIELD_IS_LONG)
- trace_seq_printf(s, "0x%x", (int)val);
- else
- trace_seq_printf(s, "%d", (int)val);
- break;
- case 2:
- trace_seq_printf(s, "%2d", (short)val);
- break;
- case 1:
- trace_seq_printf(s, "%1d", (char)val);
- break;
- default:
- trace_seq_printf(s, "%lld", val);
- }
- } else {
- if (field->flags & TEP_FIELD_IS_LONG)
- trace_seq_printf(s, "0x%llx", val);
- else
- trace_seq_printf(s, "%llu", val);
- }
- }
-}
-
-void tep_print_fields(struct trace_seq *s, void *data,
- int size __maybe_unused, struct tep_event *event)
-{
- struct tep_format_field *field;
-
- field = event->format.fields;
- while (field) {
- trace_seq_printf(s, " %s=", field->name);
- tep_print_field(s, data, field);
- field = field->next;
- }
-}
-
-static int print_function(struct trace_seq *s, const char *format,
- void *data, int size, struct tep_event *event,
- struct tep_print_arg *arg)
-{
- struct func_map *func;
- unsigned long long val;
-
- val = eval_num_arg(data, size, event, arg);
- func = find_func(event->tep, val);
- if (func) {
- trace_seq_puts(s, func->func);
- if (*format == 'F' || *format == 'S')
- trace_seq_printf(s, "+0x%llx", val - func->addr);
- } else {
- if (event->tep->long_size == 4)
- trace_seq_printf(s, "0x%lx", (long)val);
- else
- trace_seq_printf(s, "0x%llx", (long long)val);
- }
-
- return 0;
-}
-
-static int print_arg_pointer(struct trace_seq *s, const char *format, int plen,
- void *data, int size,
- struct tep_event *event, struct tep_print_arg *arg)
-{
- unsigned long long val;
- int ret = 1;
-
- if (arg->type == TEP_PRINT_BSTRING) {
- trace_seq_puts(s, arg->string.string);
- return 0;
- }
- while (*format) {
- if (*format == 'p') {
- format++;
- break;
- }
- format++;
- }
-
- switch (*format) {
- case 'F':
- case 'f':
- case 'S':
- case 's':
- ret += print_function(s, format, data, size, event, arg);
- break;
- case 'M':
- case 'm':
- ret += print_mac_arg(s, format, data, size, event, arg);
- break;
- case 'I':
- case 'i':
- ret += print_ip_arg(s, format, data, size, event, arg);
- break;
- case 'U':
- ret += print_uuid_arg(s, format, data, size, event, arg);
- break;
- case 'h':
- ret += print_raw_buff_arg(s, format, data, size, event, arg, plen);
- break;
- default:
- ret = 0;
- val = eval_num_arg(data, size, event, arg);
- trace_seq_printf(s, "%p", (void *)(intptr_t)val);
- break;
- }
-
- return ret;
-
-}
-
-static int print_arg_number(struct trace_seq *s, const char *format, int plen,
- void *data, int size, int ls,
- struct tep_event *event, struct tep_print_arg *arg)
-{
- unsigned long long val;
-
- val = eval_num_arg(data, size, event, arg);
-
- switch (ls) {
- case -2:
- if (plen >= 0)
- trace_seq_printf(s, format, plen, (char)val);
- else
- trace_seq_printf(s, format, (char)val);
- break;
- case -1:
- if (plen >= 0)
- trace_seq_printf(s, format, plen, (short)val);
- else
- trace_seq_printf(s, format, (short)val);
- break;
- case 0:
- if (plen >= 0)
- trace_seq_printf(s, format, plen, (int)val);
- else
- trace_seq_printf(s, format, (int)val);
- break;
- case 1:
- if (plen >= 0)
- trace_seq_printf(s, format, plen, (long)val);
- else
- trace_seq_printf(s, format, (long)val);
- break;
- case 2:
- if (plen >= 0)
- trace_seq_printf(s, format, plen, (long long)val);
- else
- trace_seq_printf(s, format, (long long)val);
- break;
- default:
- do_warning_event(event, "bad count (%d)", ls);
- event->flags |= TEP_EVENT_FL_FAILED;
- }
- return 0;
-}
-
-
-static void print_arg_string(struct trace_seq *s, const char *format, int plen,
- void *data, int size,
- struct tep_event *event, struct tep_print_arg *arg)
-{
- struct trace_seq p;
-
- /* Use helper trace_seq */
- trace_seq_init(&p);
- print_str_arg(&p, data, size, event,
- format, plen, arg);
- trace_seq_terminate(&p);
- trace_seq_puts(s, p.buffer);
- trace_seq_destroy(&p);
-}
-
-static int parse_arg_format_pointer(const char *format)
-{
- int ret = 0;
- int index;
- int loop;
-
- switch (*format) {
- case 'F':
- case 'S':
- case 'f':
- case 's':
- ret++;
- break;
- case 'M':
- case 'm':
- /* [mM]R , [mM]F */
- switch (format[1]) {
- case 'R':
- case 'F':
- ret++;
- break;
- }
- ret++;
- break;
- case 'I':
- case 'i':
- index = 2;
- loop = 1;
- switch (format[1]) {
- case 'S':
- /*[S][pfs]*/
- while (loop) {
- switch (format[index]) {
- case 'p':
- case 'f':
- case 's':
- ret++;
- index++;
- break;
- default:
- loop = 0;
- break;
- }
- }
- /* fall through */
- case '4':
- /* [4S][hnbl] */
- switch (format[index]) {
- case 'h':
- case 'n':
- case 'l':
- case 'b':
- ret++;
- index++;
- break;
- }
- if (format[1] == '4') {
- ret++;
- break;
- }
- /* fall through */
- case '6':
- /* [6S]c */
- if (format[index] == 'c')
- ret++;
- ret++;
- break;
- }
- ret++;
- break;
- case 'U':
- switch (format[1]) {
- case 'L':
- case 'l':
- case 'B':
- case 'b':
- ret++;
- break;
- }
- ret++;
- break;
- case 'h':
- switch (format[1]) {
- case 'C':
- case 'D':
- case 'N':
- ret++;
- break;
- }
- ret++;
- break;
- default:
- break;
- }
-
- return ret;
-}
-
-static void free_parse_args(struct tep_print_parse *arg)
-{
- struct tep_print_parse *del;
-
- while (arg) {
- del = arg;
- arg = del->next;
- free(del->format);
- free(del);
- }
-}
-
-static int parse_arg_add(struct tep_print_parse **parse, char *format,
- enum tep_print_parse_type type,
- struct tep_print_arg *arg,
- struct tep_print_arg *len_as_arg,
- int ls)
-{
- struct tep_print_parse *parg = NULL;
-
- parg = calloc(1, sizeof(*parg));
- if (!parg)
- goto error;
- parg->format = strdup(format);
- if (!parg->format)
- goto error;
- parg->type = type;
- parg->arg = arg;
- parg->len_as_arg = len_as_arg;
- parg->ls = ls;
- *parse = parg;
- return 0;
-error:
- if (parg) {
- free(parg->format);
- free(parg);
- }
- return -1;
-}
-
-static int parse_arg_format(struct tep_print_parse **parse,
- struct tep_event *event,
- const char *format, struct tep_print_arg **arg)
-{
- struct tep_print_arg *len_arg = NULL;
- char print_format[32];
- const char *start = format;
- int ret = 0;
- int ls = 0;
- int res;
- int len;
-
- format++;
- ret++;
- for (; *format; format++) {
- switch (*format) {
- case '#':
- /* FIXME: need to handle properly */
- break;
- case 'h':
- ls--;
- break;
- case 'l':
- ls++;
- break;
- case 'L':
- ls = 2;
- break;
- case '.':
- case 'z':
- case 'Z':
- case '0' ... '9':
- case '-':
- break;
- case '*':
- /* The argument is the length. */
- if (!*arg) {
- do_warning_event(event, "no argument match");
- event->flags |= TEP_EVENT_FL_FAILED;
- goto out_failed;
- }
- if (len_arg) {
- do_warning_event(event, "argument already matched");
- event->flags |= TEP_EVENT_FL_FAILED;
- goto out_failed;
- }
- len_arg = *arg;
- *arg = (*arg)->next;
- break;
- case 'p':
- if (!*arg) {
- do_warning_event(event, "no argument match");
- event->flags |= TEP_EVENT_FL_FAILED;
- goto out_failed;
- }
- res = parse_arg_format_pointer(format + 1);
- if (res > 0) {
- format += res;
- ret += res;
- }
- len = ((unsigned long)format + 1) -
- (unsigned long)start;
- /* should never happen */
- if (len > 31) {
- do_warning_event(event, "bad format!");
- event->flags |= TEP_EVENT_FL_FAILED;
- len = 31;
- }
- memcpy(print_format, start, len);
- print_format[len] = 0;
-
- parse_arg_add(parse, print_format,
- PRINT_FMT_ARG_POINTER, *arg, len_arg, ls);
- *arg = (*arg)->next;
- ret++;
- return ret;
- case 'd':
- case 'u':
- case 'i':
- case 'x':
- case 'X':
- case 'o':
- if (!*arg) {
- do_warning_event(event, "no argument match");
- event->flags |= TEP_EVENT_FL_FAILED;
- goto out_failed;
- }
-
- len = ((unsigned long)format + 1) -
- (unsigned long)start;
-
- /* should never happen */
- if (len > 30) {
- do_warning_event(event, "bad format!");
- event->flags |= TEP_EVENT_FL_FAILED;
- len = 31;
- }
- memcpy(print_format, start, len);
- print_format[len] = 0;
-
- if (event->tep->long_size == 8 && ls == 1 &&
- sizeof(long) != 8) {
- char *p;
-
- /* make %l into %ll */
- if (ls == 1 && (p = strchr(print_format, 'l')))
- memmove(p+1, p, strlen(p)+1);
- ls = 2;
- }
- if (ls < -2 || ls > 2) {
- do_warning_event(event, "bad count (%d)", ls);
- event->flags |= TEP_EVENT_FL_FAILED;
- }
- parse_arg_add(parse, print_format,
- PRINT_FMT_ARG_DIGIT, *arg, len_arg, ls);
- *arg = (*arg)->next;
- ret++;
- return ret;
- case 's':
- if (!*arg) {
- do_warning_event(event, "no matching argument");
- event->flags |= TEP_EVENT_FL_FAILED;
- goto out_failed;
- }
-
- len = ((unsigned long)format + 1) -
- (unsigned long)start;
-
- /* should never happen */
- if (len > 31) {
- do_warning_event(event, "bad format!");
- event->flags |= TEP_EVENT_FL_FAILED;
- len = 31;
- }
-
- memcpy(print_format, start, len);
- print_format[len] = 0;
-
- parse_arg_add(parse, print_format,
- PRINT_FMT_ARG_STRING, *arg, len_arg, 0);
- *arg = (*arg)->next;
- ret++;
- return ret;
- default:
- snprintf(print_format, 32, ">%c<", *format);
- parse_arg_add(parse, print_format,
- PRINT_FMT_STRING, NULL, NULL, 0);
- ret++;
- return ret;
- }
- ret++;
- }
-
-out_failed:
- return ret;
-
-}
-
-static int parse_arg_string(struct tep_print_parse **parse, const char *format)
-{
- struct trace_seq s;
- int ret = 0;
-
- trace_seq_init(&s);
- for (; *format; format++) {
- if (*format == '\\') {
- format++;
- ret++;
- switch (*format) {
- case 'n':
- trace_seq_putc(&s, '\n');
- break;
- case 't':
- trace_seq_putc(&s, '\t');
- break;
- case 'r':
- trace_seq_putc(&s, '\r');
- break;
- case '\\':
- trace_seq_putc(&s, '\\');
- break;
- default:
- trace_seq_putc(&s, *format);
- break;
- }
- } else if (*format == '%') {
- if (*(format + 1) == '%') {
- trace_seq_putc(&s, '%');
- format++;
- ret++;
- } else
- break;
- } else
- trace_seq_putc(&s, *format);
-
- ret++;
- }
- trace_seq_terminate(&s);
- parse_arg_add(parse, s.buffer, PRINT_FMT_STRING, NULL, NULL, 0);
- trace_seq_destroy(&s);
-
- return ret;
-}
-
-static struct tep_print_parse *
-parse_args(struct tep_event *event, const char *format, struct tep_print_arg *arg)
-{
- struct tep_print_parse *parse_ret = NULL;
- struct tep_print_parse **parse = NULL;
- int ret;
- int len;
-
- len = strlen(format);
- while (*format) {
- if (!parse_ret)
- parse = &parse_ret;
- if (*format == '%' && *(format + 1) != '%')
- ret = parse_arg_format(parse, event, format, &arg);
- else
- ret = parse_arg_string(parse, format);
- if (*parse)
- parse = &((*parse)->next);
-
- len -= ret;
- if (len > 0)
- format += ret;
- else
- break;
- }
- return parse_ret;
-}
-
-static void print_event_cache(struct tep_print_parse *parse, struct trace_seq *s,
- void *data, int size, struct tep_event *event)
-{
- int len_arg;
-
- while (parse) {
- if (parse->len_as_arg)
- len_arg = eval_num_arg(data, size, event, parse->len_as_arg);
- switch (parse->type) {
- case PRINT_FMT_ARG_DIGIT:
- print_arg_number(s, parse->format,
- parse->len_as_arg ? len_arg : -1, data,
- size, parse->ls, event, parse->arg);
- break;
- case PRINT_FMT_ARG_POINTER:
- print_arg_pointer(s, parse->format,
- parse->len_as_arg ? len_arg : 1,
- data, size, event, parse->arg);
- break;
- case PRINT_FMT_ARG_STRING:
- print_arg_string(s, parse->format,
- parse->len_as_arg ? len_arg : -1,
- data, size, event, parse->arg);
- break;
- case PRINT_FMT_STRING:
- default:
- trace_seq_printf(s, "%s", parse->format);
- break;
- }
- parse = parse->next;
- }
-}
-
-static void pretty_print(struct trace_seq *s, void *data, int size, struct tep_event *event)
-{
- struct tep_print_parse *parse = event->print_fmt.print_cache;
- struct tep_print_arg *args = NULL;
- char *bprint_fmt = NULL;
-
- if (event->flags & TEP_EVENT_FL_FAILED) {
- trace_seq_printf(s, "[FAILED TO PARSE]");
- tep_print_fields(s, data, size, event);
- return;
- }
-
- if (event->flags & TEP_EVENT_FL_ISBPRINT) {
- bprint_fmt = get_bprint_format(data, size, event);
- args = make_bprint_args(bprint_fmt, data, size, event);
- parse = parse_args(event, bprint_fmt, args);
- }
-
- print_event_cache(parse, s, data, size, event);
-
- if (event->flags & TEP_EVENT_FL_ISBPRINT) {
- free_parse_args(parse);
- free_args(args);
- free(bprint_fmt);
- }
-}
-
-/*
- * This parses out the Latency format (interrupts disabled,
- * need rescheduling, in hard/soft interrupt, preempt count
- * and lock depth) and places it into the trace_seq.
- */
-static void data_latency_format(struct tep_handle *tep, struct trace_seq *s,
- char *format, struct tep_record *record)
-{
- static int check_lock_depth = 1;
- static int check_migrate_disable = 1;
- static int lock_depth_exists;
- static int migrate_disable_exists;
- unsigned int lat_flags;
- struct trace_seq sq;
- unsigned int pc;
- int lock_depth = 0;
- int migrate_disable = 0;
- int hardirq;
- int softirq;
- void *data = record->data;
-
- trace_seq_init(&sq);
- lat_flags = parse_common_flags(tep, data);
- pc = parse_common_pc(tep, data);
- /* lock_depth may not always exist */
- if (lock_depth_exists)
- lock_depth = parse_common_lock_depth(tep, data);
- else if (check_lock_depth) {
- lock_depth = parse_common_lock_depth(tep, data);
- if (lock_depth < 0)
- check_lock_depth = 0;
- else
- lock_depth_exists = 1;
- }
-
- /* migrate_disable may not always exist */
- if (migrate_disable_exists)
- migrate_disable = parse_common_migrate_disable(tep, data);
- else if (check_migrate_disable) {
- migrate_disable = parse_common_migrate_disable(tep, data);
- if (migrate_disable < 0)
- check_migrate_disable = 0;
- else
- migrate_disable_exists = 1;
- }
-
- hardirq = lat_flags & TRACE_FLAG_HARDIRQ;
- softirq = lat_flags & TRACE_FLAG_SOFTIRQ;
-
- trace_seq_printf(&sq, "%c%c%c",
- (lat_flags & TRACE_FLAG_IRQS_OFF) ? 'd' :
- (lat_flags & TRACE_FLAG_IRQS_NOSUPPORT) ?
- 'X' : '.',
- (lat_flags & TRACE_FLAG_NEED_RESCHED) ?
- 'N' : '.',
- (hardirq && softirq) ? 'H' :
- hardirq ? 'h' : softirq ? 's' : '.');
-
- if (pc)
- trace_seq_printf(&sq, "%x", pc);
- else
- trace_seq_printf(&sq, ".");
-
- if (migrate_disable_exists) {
- if (migrate_disable < 0)
- trace_seq_printf(&sq, ".");
- else
- trace_seq_printf(&sq, "%d", migrate_disable);
- }
-
- if (lock_depth_exists) {
- if (lock_depth < 0)
- trace_seq_printf(&sq, ".");
- else
- trace_seq_printf(&sq, "%d", lock_depth);
- }
-
- if (sq.state == TRACE_SEQ__MEM_ALLOC_FAILED) {
- s->state = TRACE_SEQ__MEM_ALLOC_FAILED;
- return;
- }
-
- trace_seq_terminate(&sq);
- trace_seq_puts(s, sq.buffer);
- trace_seq_destroy(&sq);
- trace_seq_terminate(s);
-}
-
-/**
- * tep_data_type - parse out the given event type
- * @tep: a handle to the trace event parser context
- * @rec: the record to read from
- *
- * This returns the event id from the @rec.
- */
-int tep_data_type(struct tep_handle *tep, struct tep_record *rec)
-{
- return trace_parse_common_type(tep, rec->data);
-}
-
-/**
- * tep_data_pid - parse the PID from record
- * @tep: a handle to the trace event parser context
- * @rec: the record to parse
- *
- * This returns the PID from a record.
- */
-int tep_data_pid(struct tep_handle *tep, struct tep_record *rec)
-{
- return parse_common_pid(tep, rec->data);
-}
-
-/**
- * tep_data_preempt_count - parse the preempt count from the record
- * @tep: a handle to the trace event parser context
- * @rec: the record to parse
- *
- * This returns the preempt count from a record.
- */
-int tep_data_preempt_count(struct tep_handle *tep, struct tep_record *rec)
-{
- return parse_common_pc(tep, rec->data);
-}
-
-/**
- * tep_data_flags - parse the latency flags from the record
- * @tep: a handle to the trace event parser context
- * @rec: the record to parse
- *
- * This returns the latency flags from a record.
- *
- * Use trace_flag_type enum for the flags (see event-parse.h).
- */
-int tep_data_flags(struct tep_handle *tep, struct tep_record *rec)
-{
- return parse_common_flags(tep, rec->data);
-}
-
-/**
- * tep_data_comm_from_pid - return the command line from PID
- * @tep: a handle to the trace event parser context
- * @pid: the PID of the task to search for
- *
- * This returns a pointer to the command line that has the given
- * @pid.
- */
-const char *tep_data_comm_from_pid(struct tep_handle *tep, int pid)
-{
- const char *comm;
-
- comm = find_cmdline(tep, pid);
- return comm;
-}
-
-static struct tep_cmdline *
-pid_from_cmdlist(struct tep_handle *tep, const char *comm, struct tep_cmdline *next)
-{
- struct cmdline_list *cmdlist = (struct cmdline_list *)next;
-
- if (cmdlist)
- cmdlist = cmdlist->next;
- else
- cmdlist = tep->cmdlist;
-
- while (cmdlist && strcmp(cmdlist->comm, comm) != 0)
- cmdlist = cmdlist->next;
-
- return (struct tep_cmdline *)cmdlist;
-}
-
-/**
- * tep_data_pid_from_comm - return the pid from a given comm
- * @tep: a handle to the trace event parser context
- * @comm: the cmdline to find the pid from
- * @next: the cmdline structure to find the next comm
- *
- * This returns the cmdline structure that holds a pid for a given
- * comm, or NULL if none found. As there may be more than one pid for
- * a given comm, the result of this call can be passed back into
- * a recurring call in the @next parameter, and then it will find the
- * next pid.
- * Also, it does a linear search, so it may be slow.
- */
-struct tep_cmdline *tep_data_pid_from_comm(struct tep_handle *tep, const char *comm,
- struct tep_cmdline *next)
-{
- struct tep_cmdline *cmdline;
-
- /*
- * If the cmdlines have not been converted yet, then use
- * the list.
- */
- if (!tep->cmdlines)
- return pid_from_cmdlist(tep, comm, next);
-
- if (next) {
- /*
- * The next pointer could have been still from
- * a previous call before cmdlines were created
- */
- if (next < tep->cmdlines ||
- next >= tep->cmdlines + tep->cmdline_count)
- next = NULL;
- else
- cmdline = next++;
- }
-
- if (!next)
- cmdline = tep->cmdlines;
-
- while (cmdline < tep->cmdlines + tep->cmdline_count) {
- if (strcmp(cmdline->comm, comm) == 0)
- return cmdline;
- cmdline++;
- }
- return NULL;
-}
-
-/**
- * tep_cmdline_pid - return the pid associated to a given cmdline
- * @tep: a handle to the trace event parser context
- * @cmdline: The cmdline structure to get the pid from
- *
- * Returns the pid for a give cmdline. If @cmdline is NULL, then
- * -1 is returned.
- */
-int tep_cmdline_pid(struct tep_handle *tep, struct tep_cmdline *cmdline)
-{
- struct cmdline_list *cmdlist = (struct cmdline_list *)cmdline;
-
- if (!cmdline)
- return -1;
-
- /*
- * If cmdlines have not been created yet, or cmdline is
- * not part of the array, then treat it as a cmdlist instead.
- */
- if (!tep->cmdlines ||
- cmdline < tep->cmdlines ||
- cmdline >= tep->cmdlines + tep->cmdline_count)
- return cmdlist->pid;
-
- return cmdline->pid;
-}
-
-/*
- * This parses the raw @data using the given @event information and
- * writes the print format into the trace_seq.
- */
-static void print_event_info(struct trace_seq *s, char *format, bool raw,
- struct tep_event *event, struct tep_record *record)
-{
- int print_pretty = 1;
-
- if (raw || (event->flags & TEP_EVENT_FL_PRINTRAW))
- tep_print_fields(s, record->data, record->size, event);
- else {
-
- if (event->handler && !(event->flags & TEP_EVENT_FL_NOHANDLE))
- print_pretty = event->handler(s, record, event,
- event->context);
-
- if (print_pretty)
- pretty_print(s, record->data, record->size, event);
- }
-
- trace_seq_terminate(s);
-}
-
-/**
- * tep_find_event_by_record - return the event from a given record
- * @tep: a handle to the trace event parser context
- * @record: The record to get the event from
- *
- * Returns the associated event for a given record, or NULL if non is
- * is found.
- */
-struct tep_event *
-tep_find_event_by_record(struct tep_handle *tep, struct tep_record *record)
-{
- int type;
-
- if (record->size < 0) {
- do_warning("ug! negative record size %d", record->size);
- return NULL;
- }
-
- type = trace_parse_common_type(tep, record->data);
-
- return tep_find_event(tep, type);
-}
-
-/*
- * Writes the timestamp of the record into @s. Time divisor and precision can be
- * specified as part of printf @format string. Example:
- * "%3.1000d" - divide the time by 1000 and print the first 3 digits
- * before the dot. Thus, the timestamp "123456000" will be printed as
- * "123.456"
- */
-static void print_event_time(struct tep_handle *tep, struct trace_seq *s,
- char *format, struct tep_event *event,
- struct tep_record *record)
-{
- unsigned long long time;
- char *divstr;
- int prec = 0, pr;
- int div = 0;
- int p10 = 1;
-
- if (isdigit(*(format + 1)))
- prec = atoi(format + 1);
- divstr = strchr(format, '.');
- if (divstr && isdigit(*(divstr + 1)))
- div = atoi(divstr + 1);
- time = record->ts;
- if (div) {
- time += div / 2;
- time /= div;
- }
- pr = prec;
- while (pr--)
- p10 *= 10;
-
- if (p10 > 1 && p10 < time)
- trace_seq_printf(s, "%5llu.%0*llu", time / p10, prec, time % p10);
- else
- trace_seq_printf(s, "%12llu", time);
-}
-
-struct print_event_type {
- enum {
- EVENT_TYPE_INT = 1,
- EVENT_TYPE_STRING,
- EVENT_TYPE_UNKNOWN,
- } type;
- char format[32];
-};
-
-static void print_string(struct tep_handle *tep, struct trace_seq *s,
- struct tep_record *record, struct tep_event *event,
- const char *arg, struct print_event_type *type)
-{
- const char *comm;
- int pid;
-
- if (strncmp(arg, TEP_PRINT_LATENCY, strlen(TEP_PRINT_LATENCY)) == 0) {
- data_latency_format(tep, s, type->format, record);
- } else if (strncmp(arg, TEP_PRINT_COMM, strlen(TEP_PRINT_COMM)) == 0) {
- pid = parse_common_pid(tep, record->data);
- comm = find_cmdline(tep, pid);
- trace_seq_printf(s, type->format, comm);
- } else if (strncmp(arg, TEP_PRINT_INFO_RAW, strlen(TEP_PRINT_INFO_RAW)) == 0) {
- print_event_info(s, type->format, true, event, record);
- } else if (strncmp(arg, TEP_PRINT_INFO, strlen(TEP_PRINT_INFO)) == 0) {
- print_event_info(s, type->format, false, event, record);
- } else if (strncmp(arg, TEP_PRINT_NAME, strlen(TEP_PRINT_NAME)) == 0) {
- trace_seq_printf(s, type->format, event->name);
- } else {
- trace_seq_printf(s, "[UNKNOWN TEP TYPE %s]", arg);
- }
-
-}
-
-static void print_int(struct tep_handle *tep, struct trace_seq *s,
- struct tep_record *record, struct tep_event *event,
- int arg, struct print_event_type *type)
-{
- int param;
-
- switch (arg) {
- case TEP_PRINT_CPU:
- param = record->cpu;
- break;
- case TEP_PRINT_PID:
- param = parse_common_pid(tep, record->data);
- break;
- case TEP_PRINT_TIME:
- return print_event_time(tep, s, type->format, event, record);
- default:
- return;
- }
- trace_seq_printf(s, type->format, param);
-}
-
-static int tep_print_event_param_type(char *format,
- struct print_event_type *type)
-{
- char *str = format + 1;
- int i = 1;
-
- type->type = EVENT_TYPE_UNKNOWN;
- while (*str) {
- switch (*str) {
- case 'd':
- case 'u':
- case 'i':
- case 'x':
- case 'X':
- case 'o':
- type->type = EVENT_TYPE_INT;
- break;
- case 's':
- type->type = EVENT_TYPE_STRING;
- break;
- }
- str++;
- i++;
- if (type->type != EVENT_TYPE_UNKNOWN)
- break;
- }
- memset(type->format, 0, 32);
- memcpy(type->format, format, i < 32 ? i : 31);
- return i;
-}
-
-/**
- * tep_print_event - Write various event information
- * @tep: a handle to the trace event parser context
- * @s: the trace_seq to write to
- * @record: The record to get the event from
- * @format: a printf format string. Supported event fileds:
- * TEP_PRINT_PID, "%d" - event PID
- * TEP_PRINT_CPU, "%d" - event CPU
- * TEP_PRINT_COMM, "%s" - event command string
- * TEP_PRINT_NAME, "%s" - event name
- * TEP_PRINT_LATENCY, "%s" - event latency
- * TEP_PRINT_TIME, %d - event time stamp. A divisor and precision
- * can be specified as part of this format string:
- * "%precision.divisord". Example:
- * "%3.1000d" - divide the time by 1000 and print the first
- * 3 digits before the dot. Thus, the time stamp
- * "123456000" will be printed as "123.456"
- * TEP_PRINT_INFO, "%s" - event information. If any width is specified in
- * the format string, the event information will be printed
- * in raw format.
- * Writes the specified event information into @s.
- */
-void tep_print_event(struct tep_handle *tep, struct trace_seq *s,
- struct tep_record *record, const char *fmt, ...)
-{
- struct print_event_type type;
- char *format = strdup(fmt);
- char *current = format;
- char *str = format;
- int offset;
- va_list args;
- struct tep_event *event;
-
- if (!format)
- return;
-
- event = tep_find_event_by_record(tep, record);
- va_start(args, fmt);
- while (*current) {
- current = strchr(str, '%');
- if (!current) {
- trace_seq_puts(s, str);
- break;
- }
- memset(&type, 0, sizeof(type));
- offset = tep_print_event_param_type(current, &type);
- *current = '\0';
- trace_seq_puts(s, str);
- current += offset;
- switch (type.type) {
- case EVENT_TYPE_STRING:
- print_string(tep, s, record, event,
- va_arg(args, char*), &type);
- break;
- case EVENT_TYPE_INT:
- print_int(tep, s, record, event,
- va_arg(args, int), &type);
- break;
- case EVENT_TYPE_UNKNOWN:
- default:
- trace_seq_printf(s, "[UNKNOWN TYPE]");
- break;
- }
- str = current;
-
- }
- va_end(args);
- free(format);
-}
-
-static int events_id_cmp(const void *a, const void *b)
-{
- struct tep_event * const * ea = a;
- struct tep_event * const * eb = b;
-
- if ((*ea)->id < (*eb)->id)
- return -1;
-
- if ((*ea)->id > (*eb)->id)
- return 1;
-
- return 0;
-}
-
-static int events_name_cmp(const void *a, const void *b)
-{
- struct tep_event * const * ea = a;
- struct tep_event * const * eb = b;
- int res;
-
- res = strcmp((*ea)->name, (*eb)->name);
- if (res)
- return res;
-
- res = strcmp((*ea)->system, (*eb)->system);
- if (res)
- return res;
-
- return events_id_cmp(a, b);
-}
-
-static int events_system_cmp(const void *a, const void *b)
-{
- struct tep_event * const * ea = a;
- struct tep_event * const * eb = b;
- int res;
-
- res = strcmp((*ea)->system, (*eb)->system);
- if (res)
- return res;
-
- res = strcmp((*ea)->name, (*eb)->name);
- if (res)
- return res;
-
- return events_id_cmp(a, b);
-}
-
-static struct tep_event **list_events_copy(struct tep_handle *tep)
-{
- struct tep_event **events;
-
- if (!tep)
- return NULL;
-
- events = malloc(sizeof(*events) * (tep->nr_events + 1));
- if (!events)
- return NULL;
-
- memcpy(events, tep->events, sizeof(*events) * tep->nr_events);
- events[tep->nr_events] = NULL;
- return events;
-}
-
-static void list_events_sort(struct tep_event **events, int nr_events,
- enum tep_event_sort_type sort_type)
-{
- int (*sort)(const void *a, const void *b);
-
- switch (sort_type) {
- case TEP_EVENT_SORT_ID:
- sort = events_id_cmp;
- break;
- case TEP_EVENT_SORT_NAME:
- sort = events_name_cmp;
- break;
- case TEP_EVENT_SORT_SYSTEM:
- sort = events_system_cmp;
- break;
- default:
- sort = NULL;
- }
-
- if (sort)
- qsort(events, nr_events, sizeof(*events), sort);
-}
-
-/**
- * tep_list_events - Get events, sorted by given criteria.
- * @tep: a handle to the tep context
- * @sort_type: desired sort order of the events in the array
- *
- * Returns an array of pointers to all events, sorted by the given
- * @sort_type criteria. The last element of the array is NULL. The returned
- * memory must not be freed, it is managed by the library.
- * The function is not thread safe.
- */
-struct tep_event **tep_list_events(struct tep_handle *tep,
- enum tep_event_sort_type sort_type)
-{
- struct tep_event **events;
-
- if (!tep)
- return NULL;
-
- events = tep->sort_events;
- if (events && tep->last_type == sort_type)
- return events;
-
- if (!events) {
- events = list_events_copy(tep);
- if (!events)
- return NULL;
-
- tep->sort_events = events;
-
- /* the internal events are sorted by id */
- if (sort_type == TEP_EVENT_SORT_ID) {
- tep->last_type = sort_type;
- return events;
- }
- }
-
- list_events_sort(events, tep->nr_events, sort_type);
- tep->last_type = sort_type;
-
- return events;
-}
-
-
-/**
- * tep_list_events_copy - Thread safe version of tep_list_events()
- * @tep: a handle to the tep context
- * @sort_type: desired sort order of the events in the array
- *
- * Returns an array of pointers to all events, sorted by the given
- * @sort_type criteria. The last element of the array is NULL. The returned
- * array is newly allocated inside the function and must be freed by the caller
- */
-struct tep_event **tep_list_events_copy(struct tep_handle *tep,
- enum tep_event_sort_type sort_type)
-{
- struct tep_event **events;
-
- if (!tep)
- return NULL;
-
- events = list_events_copy(tep);
- if (!events)
- return NULL;
-
- /* the internal events are sorted by id */
- if (sort_type == TEP_EVENT_SORT_ID)
- return events;
-
- list_events_sort(events, tep->nr_events, sort_type);
-
- return events;
-}
-
-static struct tep_format_field **
-get_event_fields(const char *type, const char *name,
- int count, struct tep_format_field *list)
-{
- struct tep_format_field **fields;
- struct tep_format_field *field;
- int i = 0;
-
- fields = malloc(sizeof(*fields) * (count + 1));
- if (!fields)
- return NULL;
-
- for (field = list; field; field = field->next) {
- fields[i++] = field;
- if (i == count + 1) {
- do_warning("event %s has more %s fields than specified",
- name, type);
- i--;
- break;
- }
- }
-
- if (i != count)
- do_warning("event %s has less %s fields than specified",
- name, type);
-
- fields[i] = NULL;
-
- return fields;
-}
-
-/**
- * tep_event_common_fields - return a list of common fields for an event
- * @event: the event to return the common fields of.
- *
- * Returns an allocated array of fields. The last item in the array is NULL.
- * The array must be freed with free().
- */
-struct tep_format_field **tep_event_common_fields(struct tep_event *event)
-{
- return get_event_fields("common", event->name,
- event->format.nr_common,
- event->format.common_fields);
-}
-
-/**
- * tep_event_fields - return a list of event specific fields for an event
- * @event: the event to return the fields of.
- *
- * Returns an allocated array of fields. The last item in the array is NULL.
- * The array must be freed with free().
- */
-struct tep_format_field **tep_event_fields(struct tep_event *event)
-{
- return get_event_fields("event", event->name,
- event->format.nr_fields,
- event->format.fields);
-}
-
-static void print_fields(struct trace_seq *s, struct tep_print_flag_sym *field)
-{
- trace_seq_printf(s, "{ %s, %s }", field->value, field->str);
- if (field->next) {
- trace_seq_puts(s, ", ");
- print_fields(s, field->next);
- }
-}
-
-/* for debugging */
-static void print_args(struct tep_print_arg *args)
-{
- int print_paren = 1;
- struct trace_seq s;
-
- switch (args->type) {
- case TEP_PRINT_NULL:
- printf("null");
- break;
- case TEP_PRINT_ATOM:
- printf("%s", args->atom.atom);
- break;
- case TEP_PRINT_FIELD:
- printf("REC->%s", args->field.name);
- break;
- case TEP_PRINT_FLAGS:
- printf("__print_flags(");
- print_args(args->flags.field);
- printf(", %s, ", args->flags.delim);
- trace_seq_init(&s);
- print_fields(&s, args->flags.flags);
- trace_seq_do_printf(&s);
- trace_seq_destroy(&s);
- printf(")");
- break;
- case TEP_PRINT_SYMBOL:
- printf("__print_symbolic(");
- print_args(args->symbol.field);
- printf(", ");
- trace_seq_init(&s);
- print_fields(&s, args->symbol.symbols);
- trace_seq_do_printf(&s);
- trace_seq_destroy(&s);
- printf(")");
- break;
- case TEP_PRINT_HEX:
- printf("__print_hex(");
- print_args(args->hex.field);
- printf(", ");
- print_args(args->hex.size);
- printf(")");
- break;
- case TEP_PRINT_HEX_STR:
- printf("__print_hex_str(");
- print_args(args->hex.field);
- printf(", ");
- print_args(args->hex.size);
- printf(")");
- break;
- case TEP_PRINT_INT_ARRAY:
- printf("__print_array(");
- print_args(args->int_array.field);
- printf(", ");
- print_args(args->int_array.count);
- printf(", ");
- print_args(args->int_array.el_size);
- printf(")");
- break;
- case TEP_PRINT_STRING:
- case TEP_PRINT_BSTRING:
- printf("__get_str(%s)", args->string.string);
- break;
- case TEP_PRINT_BITMASK:
- printf("__get_bitmask(%s)", args->bitmask.bitmask);
- break;
- case TEP_PRINT_TYPE:
- printf("(%s)", args->typecast.type);
- print_args(args->typecast.item);
- break;
- case TEP_PRINT_OP:
- if (strcmp(args->op.op, ":") == 0)
- print_paren = 0;
- if (print_paren)
- printf("(");
- print_args(args->op.left);
- printf(" %s ", args->op.op);
- print_args(args->op.right);
- if (print_paren)
- printf(")");
- break;
- default:
- /* we should warn... */
- return;
- }
- if (args->next) {
- printf("\n");
- print_args(args->next);
- }
-}
-
-static void parse_header_field(const char *field,
- int *offset, int *size, int mandatory)
-{
- unsigned long long save_input_buf_ptr;
- unsigned long long save_input_buf_siz;
- char *token;
- int type;
-
- save_input_buf_ptr = input_buf_ptr;
- save_input_buf_siz = input_buf_siz;
-
- if (read_expected(TEP_EVENT_ITEM, "field") < 0)
- return;
- if (read_expected(TEP_EVENT_OP, ":") < 0)
- return;
-
- /* type */
- if (read_expect_type(TEP_EVENT_ITEM, &token) < 0)
- goto fail;
- free_token(token);
-
- /*
- * If this is not a mandatory field, then test it first.
- */
- if (mandatory) {
- if (read_expected(TEP_EVENT_ITEM, field) < 0)
- return;
- } else {
- if (read_expect_type(TEP_EVENT_ITEM, &token) < 0)
- goto fail;
- if (strcmp(token, field) != 0)
- goto discard;
- free_token(token);
- }
-
- if (read_expected(TEP_EVENT_OP, ";") < 0)
- return;
- if (read_expected(TEP_EVENT_ITEM, "offset") < 0)
- return;
- if (read_expected(TEP_EVENT_OP, ":") < 0)
- return;
- if (read_expect_type(TEP_EVENT_ITEM, &token) < 0)
- goto fail;
- *offset = atoi(token);
- free_token(token);
- if (read_expected(TEP_EVENT_OP, ";") < 0)
- return;
- if (read_expected(TEP_EVENT_ITEM, "size") < 0)
- return;
- if (read_expected(TEP_EVENT_OP, ":") < 0)
- return;
- if (read_expect_type(TEP_EVENT_ITEM, &token) < 0)
- goto fail;
- *size = atoi(token);
- free_token(token);
- if (read_expected(TEP_EVENT_OP, ";") < 0)
- return;
- type = read_token(&token);
- if (type != TEP_EVENT_NEWLINE) {
- /* newer versions of the kernel have a "signed" type */
- if (type != TEP_EVENT_ITEM)
- goto fail;
-
- if (strcmp(token, "signed") != 0)
- goto fail;
-
- free_token(token);
-
- if (read_expected(TEP_EVENT_OP, ":") < 0)
- return;
-
- if (read_expect_type(TEP_EVENT_ITEM, &token))
- goto fail;
-
- free_token(token);
- if (read_expected(TEP_EVENT_OP, ";") < 0)
- return;
-
- if (read_expect_type(TEP_EVENT_NEWLINE, &token))
- goto fail;
- }
- fail:
- free_token(token);
- return;
-
- discard:
- input_buf_ptr = save_input_buf_ptr;
- input_buf_siz = save_input_buf_siz;
- *offset = 0;
- *size = 0;
- free_token(token);
-}
-
-/**
- * tep_parse_header_page - parse the data stored in the header page
- * @tep: a handle to the trace event parser context
- * @buf: the buffer storing the header page format string
- * @size: the size of @buf
- * @long_size: the long size to use if there is no header
- *
- * This parses the header page format for information on the
- * ring buffer used. The @buf should be copied from
- *
- * /sys/kernel/debug/tracing/events/header_page
- */
-int tep_parse_header_page(struct tep_handle *tep, char *buf, unsigned long size,
- int long_size)
-{
- int ignore;
-
- if (!size) {
- /*
- * Old kernels did not have header page info.
- * Sorry but we just use what we find here in user space.
- */
- tep->header_page_ts_size = sizeof(long long);
- tep->header_page_size_size = long_size;
- tep->header_page_data_offset = sizeof(long long) + long_size;
- tep->old_format = 1;
- return -1;
- }
- init_input_buf(buf, size);
-
- parse_header_field("timestamp", &tep->header_page_ts_offset,
- &tep->header_page_ts_size, 1);
- parse_header_field("commit", &tep->header_page_size_offset,
- &tep->header_page_size_size, 1);
- parse_header_field("overwrite", &tep->header_page_overwrite,
- &ignore, 0);
- parse_header_field("data", &tep->header_page_data_offset,
- &tep->header_page_data_size, 1);
-
- return 0;
-}
-
-static int event_matches(struct tep_event *event,
- int id, const char *sys_name,
- const char *event_name)
-{
- if (id >= 0 && id != event->id)
- return 0;
-
- if (event_name && (strcmp(event_name, event->name) != 0))
- return 0;
-
- if (sys_name && (strcmp(sys_name, event->system) != 0))
- return 0;
-
- return 1;
-}
-
-static void free_handler(struct event_handler *handle)
-{
- free((void *)handle->sys_name);
- free((void *)handle->event_name);
- free(handle);
-}
-
-static int find_event_handle(struct tep_handle *tep, struct tep_event *event)
-{
- struct event_handler *handle, **next;
-
- for (next = &tep->handlers; *next;
- next = &(*next)->next) {
- handle = *next;
- if (event_matches(event, handle->id,
- handle->sys_name,
- handle->event_name))
- break;
- }
-
- if (!(*next))
- return 0;
-
- pr_stat("overriding event (%d) %s:%s with new print handler",
- event->id, event->system, event->name);
-
- event->handler = handle->func;
- event->context = handle->context;
-
- *next = handle->next;
- free_handler(handle);
-
- return 1;
-}
-
-/**
- * parse_format - parse the event format
- * @buf: the buffer storing the event format string
- * @size: the size of @buf
- * @sys: the system the event belongs to
- *
- * This parses the event format and creates an event structure
- * to quickly parse raw data for a given event.
- *
- * These files currently come from:
- *
- * /sys/kernel/debug/tracing/events/.../.../format
- */
-static enum tep_errno parse_format(struct tep_event **eventp,
- struct tep_handle *tep, const char *buf,
- unsigned long size, const char *sys)
-{
- struct tep_event *event;
- int ret;
-
- init_input_buf(buf, size);
-
- *eventp = event = alloc_event();
- if (!event)
- return TEP_ERRNO__MEM_ALLOC_FAILED;
-
- event->name = event_read_name();
- if (!event->name) {
- /* Bad event? */
- ret = TEP_ERRNO__MEM_ALLOC_FAILED;
- goto event_alloc_failed;
- }
-
- if (strcmp(sys, "ftrace") == 0) {
- event->flags |= TEP_EVENT_FL_ISFTRACE;
-
- if (strcmp(event->name, "bprint") == 0)
- event->flags |= TEP_EVENT_FL_ISBPRINT;
- }
-
- event->id = event_read_id();
- if (event->id < 0) {
- ret = TEP_ERRNO__READ_ID_FAILED;
- /*
- * This isn't an allocation error actually.
- * But as the ID is critical, just bail out.
- */
- goto event_alloc_failed;
- }
-
- event->system = strdup(sys);
- if (!event->system) {
- ret = TEP_ERRNO__MEM_ALLOC_FAILED;
- goto event_alloc_failed;
- }
-
- /* Add tep to event so that it can be referenced */
- event->tep = tep;
-
- ret = event_read_format(event);
- if (ret < 0) {
- ret = TEP_ERRNO__READ_FORMAT_FAILED;
- goto event_parse_failed;
- }
-
- /*
- * If the event has an override, don't print warnings if the event
- * print format fails to parse.
- */
- if (tep && find_event_handle(tep, event))
- show_warning = 0;
-
- ret = event_read_print(event);
- show_warning = 1;
-
- if (ret < 0) {
- ret = TEP_ERRNO__READ_PRINT_FAILED;
- goto event_parse_failed;
- }
-
- if (!ret && (event->flags & TEP_EVENT_FL_ISFTRACE)) {
- struct tep_format_field *field;
- struct tep_print_arg *arg, **list;
-
- /* old ftrace had no args */
- list = &event->print_fmt.args;
- for (field = event->format.fields; field; field = field->next) {
- arg = alloc_arg();
- if (!arg) {
- event->flags |= TEP_EVENT_FL_FAILED;
- return TEP_ERRNO__OLD_FTRACE_ARG_FAILED;
- }
- arg->type = TEP_PRINT_FIELD;
- arg->field.name = strdup(field->name);
- if (!arg->field.name) {
- event->flags |= TEP_EVENT_FL_FAILED;
- free_arg(arg);
- return TEP_ERRNO__OLD_FTRACE_ARG_FAILED;
- }
- arg->field.field = field;
- *list = arg;
- list = &arg->next;
- }
- }
-
- if (!(event->flags & TEP_EVENT_FL_ISBPRINT))
- event->print_fmt.print_cache = parse_args(event,
- event->print_fmt.format,
- event->print_fmt.args);
-
- return 0;
-
- event_parse_failed:
- event->flags |= TEP_EVENT_FL_FAILED;
- return ret;
-
- event_alloc_failed:
- free(event->system);
- free(event->name);
- free(event);
- *eventp = NULL;
- return ret;
-}
-
-static enum tep_errno
-__parse_event(struct tep_handle *tep,
- struct tep_event **eventp,
- const char *buf, unsigned long size,
- const char *sys)
-{
- int ret = parse_format(eventp, tep, buf, size, sys);
- struct tep_event *event = *eventp;
-
- if (event == NULL)
- return ret;
-
- if (tep && add_event(tep, event)) {
- ret = TEP_ERRNO__MEM_ALLOC_FAILED;
- goto event_add_failed;
- }
-
-#define PRINT_ARGS 0
- if (PRINT_ARGS && event->print_fmt.args)
- print_args(event->print_fmt.args);
-
- return 0;
-
-event_add_failed:
- free_tep_event(event);
- return ret;
-}
-
-/**
- * tep_parse_format - parse the event format
- * @tep: a handle to the trace event parser context
- * @eventp: returned format
- * @buf: the buffer storing the event format string
- * @size: the size of @buf
- * @sys: the system the event belongs to
- *
- * This parses the event format and creates an event structure
- * to quickly parse raw data for a given event.
- *
- * These files currently come from:
- *
- * /sys/kernel/debug/tracing/events/.../.../format
- */
-enum tep_errno tep_parse_format(struct tep_handle *tep,
- struct tep_event **eventp,
- const char *buf,
- unsigned long size, const char *sys)
-{
- return __parse_event(tep, eventp, buf, size, sys);
-}
-
-/**
- * tep_parse_event - parse the event format
- * @tep: a handle to the trace event parser context
- * @buf: the buffer storing the event format string
- * @size: the size of @buf
- * @sys: the system the event belongs to
- *
- * This parses the event format and creates an event structure
- * to quickly parse raw data for a given event.
- *
- * These files currently come from:
- *
- * /sys/kernel/debug/tracing/events/.../.../format
- */
-enum tep_errno tep_parse_event(struct tep_handle *tep, const char *buf,
- unsigned long size, const char *sys)
-{
- struct tep_event *event = NULL;
- return __parse_event(tep, &event, buf, size, sys);
-}
-
-int get_field_val(struct trace_seq *s, struct tep_format_field *field,
- const char *name, struct tep_record *record,
- unsigned long long *val, int err)
-{
- if (!field) {
- if (err)
- trace_seq_printf(s, "<CANT FIND FIELD %s>", name);
- return -1;
- }
-
- if (tep_read_number_field(field, record->data, val)) {
- if (err)
- trace_seq_printf(s, " %s=INVALID", name);
- return -1;
- }
-
- return 0;
-}
-
-/**
- * tep_get_field_raw - return the raw pointer into the data field
- * @s: The seq to print to on error
- * @event: the event that the field is for
- * @name: The name of the field
- * @record: The record with the field name.
- * @len: place to store the field length.
- * @err: print default error if failed.
- *
- * Returns a pointer into record->data of the field and places
- * the length of the field in @len.
- *
- * On failure, it returns NULL.
- */
-void *tep_get_field_raw(struct trace_seq *s, struct tep_event *event,
- const char *name, struct tep_record *record,
- int *len, int err)
-{
- struct tep_format_field *field;
- void *data = record->data;
- unsigned offset;
- int dummy;
-
- if (!event)
- return NULL;
-
- field = tep_find_field(event, name);
-
- if (!field) {
- if (err)
- trace_seq_printf(s, "<CANT FIND FIELD %s>", name);
- return NULL;
- }
-
- /* Allow @len to be NULL */
- if (!len)
- len = &dummy;
-
- offset = field->offset;
- if (field->flags & TEP_FIELD_IS_DYNAMIC) {
- offset = tep_read_number(event->tep,
- data + offset, field->size);
- *len = offset >> 16;
- offset &= 0xffff;
- if (field->flags & TEP_FIELD_IS_RELATIVE)
- offset += field->offset + field->size;
- } else
- *len = field->size;
-
- return data + offset;
-}
-
-/**
- * tep_get_field_val - find a field and return its value
- * @s: The seq to print to on error
- * @event: the event that the field is for
- * @name: The name of the field
- * @record: The record with the field name.
- * @val: place to store the value of the field.
- * @err: print default error if failed.
- *
- * Returns 0 on success -1 on field not found.
- */
-int tep_get_field_val(struct trace_seq *s, struct tep_event *event,
- const char *name, struct tep_record *record,
- unsigned long long *val, int err)
-{
- struct tep_format_field *field;
-
- if (!event)
- return -1;
-
- field = tep_find_field(event, name);
-
- return get_field_val(s, field, name, record, val, err);
-}
-
-/**
- * tep_get_common_field_val - find a common field and return its value
- * @s: The seq to print to on error
- * @event: the event that the field is for
- * @name: The name of the field
- * @record: The record with the field name.
- * @val: place to store the value of the field.
- * @err: print default error if failed.
- *
- * Returns 0 on success -1 on field not found.
- */
-int tep_get_common_field_val(struct trace_seq *s, struct tep_event *event,
- const char *name, struct tep_record *record,
- unsigned long long *val, int err)
-{
- struct tep_format_field *field;
-
- if (!event)
- return -1;
-
- field = tep_find_common_field(event, name);
-
- return get_field_val(s, field, name, record, val, err);
-}
-
-/**
- * tep_get_any_field_val - find a any field and return its value
- * @s: The seq to print to on error
- * @event: the event that the field is for
- * @name: The name of the field
- * @record: The record with the field name.
- * @val: place to store the value of the field.
- * @err: print default error if failed.
- *
- * Returns 0 on success -1 on field not found.
- */
-int tep_get_any_field_val(struct trace_seq *s, struct tep_event *event,
- const char *name, struct tep_record *record,
- unsigned long long *val, int err)
-{
- struct tep_format_field *field;
-
- if (!event)
- return -1;
-
- field = tep_find_any_field(event, name);
-
- return get_field_val(s, field, name, record, val, err);
-}
-
-/**
- * tep_print_num_field - print a field and a format
- * @s: The seq to print to
- * @fmt: The printf format to print the field with.
- * @event: the event that the field is for
- * @name: The name of the field
- * @record: The record with the field name.
- * @err: print default error if failed.
- *
- * Returns positive value on success, negative in case of an error,
- * or 0 if buffer is full.
- */
-int tep_print_num_field(struct trace_seq *s, const char *fmt,
- struct tep_event *event, const char *name,
- struct tep_record *record, int err)
-{
- struct tep_format_field *field = tep_find_field(event, name);
- unsigned long long val;
-
- if (!field)
- goto failed;
-
- if (tep_read_number_field(field, record->data, &val))
- goto failed;
-
- return trace_seq_printf(s, fmt, val);
-
- failed:
- if (err)
- trace_seq_printf(s, "CAN'T FIND FIELD \"%s\"", name);
- return -1;
-}
-
-/**
- * tep_print_func_field - print a field and a format for function pointers
- * @s: The seq to print to
- * @fmt: The printf format to print the field with.
- * @event: the event that the field is for
- * @name: The name of the field
- * @record: The record with the field name.
- * @err: print default error if failed.
- *
- * Returns positive value on success, negative in case of an error,
- * or 0 if buffer is full.
- */
-int tep_print_func_field(struct trace_seq *s, const char *fmt,
- struct tep_event *event, const char *name,
- struct tep_record *record, int err)
-{
- struct tep_format_field *field = tep_find_field(event, name);
- struct tep_handle *tep = event->tep;
- unsigned long long val;
- struct func_map *func;
- char tmp[128];
-
- if (!field)
- goto failed;
-
- if (tep_read_number_field(field, record->data, &val))
- goto failed;
-
- func = find_func(tep, val);
-
- if (func)
- snprintf(tmp, 128, "%s/0x%llx", func->func, func->addr - val);
- else
- sprintf(tmp, "0x%08llx", val);
-
- return trace_seq_printf(s, fmt, tmp);
-
- failed:
- if (err)
- trace_seq_printf(s, "CAN'T FIND FIELD \"%s\"", name);
- return -1;
-}
-
-static void free_func_handle(struct tep_function_handler *func)
-{
- struct func_params *params;
-
- free(func->name);
-
- while (func->params) {
- params = func->params;
- func->params = params->next;
- free(params);
- }
-
- free(func);
-}
-
-/**
- * tep_register_print_function - register a helper function
- * @tep: a handle to the trace event parser context
- * @func: the function to process the helper function
- * @ret_type: the return type of the helper function
- * @name: the name of the helper function
- * @parameters: A list of enum tep_func_arg_type
- *
- * Some events may have helper functions in the print format arguments.
- * This allows a plugin to dynamically create a way to process one
- * of these functions.
- *
- * The @parameters is a variable list of tep_func_arg_type enums that
- * must end with TEP_FUNC_ARG_VOID.
- */
-int tep_register_print_function(struct tep_handle *tep,
- tep_func_handler func,
- enum tep_func_arg_type ret_type,
- char *name, ...)
-{
- struct tep_function_handler *func_handle;
- struct func_params **next_param;
- struct func_params *param;
- enum tep_func_arg_type type;
- va_list ap;
- int ret;
-
- func_handle = find_func_handler(tep, name);
- if (func_handle) {
- /*
- * This is most like caused by the users own
- * plugins updating the function. This overrides the
- * system defaults.
- */
- pr_stat("override of function helper '%s'", name);
- remove_func_handler(tep, name);
- }
-
- func_handle = calloc(1, sizeof(*func_handle));
- if (!func_handle) {
- do_warning("Failed to allocate function handler");
- return TEP_ERRNO__MEM_ALLOC_FAILED;
- }
-
- func_handle->ret_type = ret_type;
- func_handle->name = strdup(name);
- func_handle->func = func;
- if (!func_handle->name) {
- do_warning("Failed to allocate function name");
- free(func_handle);
- return TEP_ERRNO__MEM_ALLOC_FAILED;
- }
-
- next_param = &(func_handle->params);
- va_start(ap, name);
- for (;;) {
- type = va_arg(ap, enum tep_func_arg_type);
- if (type == TEP_FUNC_ARG_VOID)
- break;
-
- if (type >= TEP_FUNC_ARG_MAX_TYPES) {
- do_warning("Invalid argument type %d", type);
- ret = TEP_ERRNO__INVALID_ARG_TYPE;
- goto out_free;
- }
-
- param = malloc(sizeof(*param));
- if (!param) {
- do_warning("Failed to allocate function param");
- ret = TEP_ERRNO__MEM_ALLOC_FAILED;
- goto out_free;
- }
- param->type = type;
- param->next = NULL;
-
- *next_param = param;
- next_param = &(param->next);
-
- func_handle->nr_args++;
- }
- va_end(ap);
-
- func_handle->next = tep->func_handlers;
- tep->func_handlers = func_handle;
-
- return 0;
- out_free:
- va_end(ap);
- free_func_handle(func_handle);
- return ret;
-}
-
-/**
- * tep_unregister_print_function - unregister a helper function
- * @tep: a handle to the trace event parser context
- * @func: the function to process the helper function
- * @name: the name of the helper function
- *
- * This function removes existing print handler for function @name.
- *
- * Returns 0 if the handler was removed successully, -1 otherwise.
- */
-int tep_unregister_print_function(struct tep_handle *tep,
- tep_func_handler func, char *name)
-{
- struct tep_function_handler *func_handle;
-
- func_handle = find_func_handler(tep, name);
- if (func_handle && func_handle->func == func) {
- remove_func_handler(tep, name);
- return 0;
- }
- return -1;
-}
-
-static struct tep_event *search_event(struct tep_handle *tep, int id,
- const char *sys_name,
- const char *event_name)
-{
- struct tep_event *event;
-
- if (id >= 0) {
- /* search by id */
- event = tep_find_event(tep, id);
- if (!event)
- return NULL;
- if (event_name && (strcmp(event_name, event->name) != 0))
- return NULL;
- if (sys_name && (strcmp(sys_name, event->system) != 0))
- return NULL;
- } else {
- event = tep_find_event_by_name(tep, sys_name, event_name);
- if (!event)
- return NULL;
- }
- return event;
-}
-
-/**
- * tep_register_event_handler - register a way to parse an event
- * @tep: a handle to the trace event parser context
- * @id: the id of the event to register
- * @sys_name: the system name the event belongs to
- * @event_name: the name of the event
- * @func: the function to call to parse the event information
- * @context: the data to be passed to @func
- *
- * This function allows a developer to override the parsing of
- * a given event. If for some reason the default print format
- * is not sufficient, this function will register a function
- * for an event to be used to parse the data instead.
- *
- * If @id is >= 0, then it is used to find the event.
- * else @sys_name and @event_name are used.
- *
- * Returns:
- * TEP_REGISTER_SUCCESS_OVERWRITE if an existing handler is overwritten
- * TEP_REGISTER_SUCCESS if a new handler is registered successfully
- * negative TEP_ERRNO_... in case of an error
- *
- */
-int tep_register_event_handler(struct tep_handle *tep, int id,
- const char *sys_name, const char *event_name,
- tep_event_handler_func func, void *context)
-{
- struct tep_event *event;
- struct event_handler *handle;
-
- event = search_event(tep, id, sys_name, event_name);
- if (event == NULL)
- goto not_found;
-
- pr_stat("overriding event (%d) %s:%s with new print handler",
- event->id, event->system, event->name);
-
- event->handler = func;
- event->context = context;
- return TEP_REGISTER_SUCCESS_OVERWRITE;
-
- not_found:
- /* Save for later use. */
- handle = calloc(1, sizeof(*handle));
- if (!handle) {
- do_warning("Failed to allocate event handler");
- return TEP_ERRNO__MEM_ALLOC_FAILED;
- }
-
- handle->id = id;
- if (event_name)
- handle->event_name = strdup(event_name);
- if (sys_name)
- handle->sys_name = strdup(sys_name);
-
- if ((event_name && !handle->event_name) ||
- (sys_name && !handle->sys_name)) {
- do_warning("Failed to allocate event/sys name");
- free((void *)handle->event_name);
- free((void *)handle->sys_name);
- free(handle);
- return TEP_ERRNO__MEM_ALLOC_FAILED;
- }
-
- handle->func = func;
- handle->next = tep->handlers;
- tep->handlers = handle;
- handle->context = context;
-
- return TEP_REGISTER_SUCCESS;
-}
-
-static int handle_matches(struct event_handler *handler, int id,
- const char *sys_name, const char *event_name,
- tep_event_handler_func func, void *context)
-{
- if (id >= 0 && id != handler->id)
- return 0;
-
- if (event_name && (strcmp(event_name, handler->event_name) != 0))
- return 0;
-
- if (sys_name && (strcmp(sys_name, handler->sys_name) != 0))
- return 0;
-
- if (func != handler->func || context != handler->context)
- return 0;
-
- return 1;
-}
-
-/**
- * tep_unregister_event_handler - unregister an existing event handler
- * @tep: a handle to the trace event parser context
- * @id: the id of the event to unregister
- * @sys_name: the system name the handler belongs to
- * @event_name: the name of the event handler
- * @func: the function to call to parse the event information
- * @context: the data to be passed to @func
- *
- * This function removes existing event handler (parser).
- *
- * If @id is >= 0, then it is used to find the event.
- * else @sys_name and @event_name are used.
- *
- * Returns 0 if handler was removed successfully, -1 if event was not found.
- */
-int tep_unregister_event_handler(struct tep_handle *tep, int id,
- const char *sys_name, const char *event_name,
- tep_event_handler_func func, void *context)
-{
- struct tep_event *event;
- struct event_handler *handle;
- struct event_handler **next;
-
- event = search_event(tep, id, sys_name, event_name);
- if (event == NULL)
- goto not_found;
-
- if (event->handler == func && event->context == context) {
- pr_stat("removing override handler for event (%d) %s:%s. Going back to default handler.",
- event->id, event->system, event->name);
-
- event->handler = NULL;
- event->context = NULL;
- return 0;
- }
-
-not_found:
- for (next = &tep->handlers; *next; next = &(*next)->next) {
- handle = *next;
- if (handle_matches(handle, id, sys_name, event_name,
- func, context))
- break;
- }
-
- if (!(*next))
- return -1;
-
- *next = handle->next;
- free_handler(handle);
-
- return 0;
-}
-
-/**
- * tep_alloc - create a tep handle
- */
-struct tep_handle *tep_alloc(void)
-{
- struct tep_handle *tep = calloc(1, sizeof(*tep));
-
- if (tep) {
- tep->ref_count = 1;
- tep->host_bigendian = tep_is_bigendian();
- }
-
- return tep;
-}
-
-void tep_ref(struct tep_handle *tep)
-{
- tep->ref_count++;
-}
-
-int tep_get_ref(struct tep_handle *tep)
-{
- if (tep)
- return tep->ref_count;
- return 0;
-}
-
-__hidden void free_tep_format_field(struct tep_format_field *field)
-{
- free(field->type);
- if (field->alias != field->name)
- free(field->alias);
- free(field->name);
- free(field);
-}
-
-static void free_format_fields(struct tep_format_field *field)
-{
- struct tep_format_field *next;
-
- while (field) {
- next = field->next;
- free_tep_format_field(field);
- field = next;
- }
-}
-
-static void free_formats(struct tep_format *format)
-{
- free_format_fields(format->common_fields);
- free_format_fields(format->fields);
-}
-
-__hidden void free_tep_event(struct tep_event *event)
-{
- free(event->name);
- free(event->system);
-
- free_formats(&event->format);
-
- free(event->print_fmt.format);
- free_args(event->print_fmt.args);
- free_parse_args(event->print_fmt.print_cache);
- free(event);
-}
-
-/**
- * tep_free - free a tep handle
- * @tep: the tep handle to free
- */
-void tep_free(struct tep_handle *tep)
-{
- struct cmdline_list *cmdlist, *cmdnext;
- struct func_list *funclist, *funcnext;
- struct printk_list *printklist, *printknext;
- struct tep_function_handler *func_handler;
- struct event_handler *handle;
- int i;
-
- if (!tep)
- return;
-
- cmdlist = tep->cmdlist;
- funclist = tep->funclist;
- printklist = tep->printklist;
-
- tep->ref_count--;
- if (tep->ref_count)
- return;
-
- if (tep->cmdlines) {
- for (i = 0; i < tep->cmdline_count; i++)
- free(tep->cmdlines[i].comm);
- free(tep->cmdlines);
- }
-
- while (cmdlist) {
- cmdnext = cmdlist->next;
- free(cmdlist->comm);
- free(cmdlist);
- cmdlist = cmdnext;
- }
-
- if (tep->func_map) {
- for (i = 0; i < (int)tep->func_count; i++) {
- free(tep->func_map[i].func);
- free(tep->func_map[i].mod);
- }
- free(tep->func_map);
- }
-
- while (funclist) {
- funcnext = funclist->next;
- free(funclist->func);
- free(funclist->mod);
- free(funclist);
- funclist = funcnext;
- }
-
- while (tep->func_handlers) {
- func_handler = tep->func_handlers;
- tep->func_handlers = func_handler->next;
- free_func_handle(func_handler);
- }
-
- if (tep->printk_map) {
- for (i = 0; i < (int)tep->printk_count; i++)
- free(tep->printk_map[i].printk);
- free(tep->printk_map);
- }
-
- while (printklist) {
- printknext = printklist->next;
- free(printklist->printk);
- free(printklist);
- printklist = printknext;
- }
-
- for (i = 0; i < tep->nr_events; i++)
- free_tep_event(tep->events[i]);
-
- while (tep->handlers) {
- handle = tep->handlers;
- tep->handlers = handle->next;
- free_handler(handle);
- }
-
- free(tep->events);
- free(tep->sort_events);
- free(tep->func_resolver);
- free_tep_plugin_paths(tep);
-
- free(tep);
-}
-
-void tep_unref(struct tep_handle *tep)
-{
- tep_free(tep);
-}
diff --git a/tools/lib/traceevent/event-parse.h b/tools/lib/traceevent/event-parse.h
deleted file mode 100644
index 41d4f9f..0000000
--- a/tools/lib/traceevent/event-parse.h
+++ /dev/null
@@ -1,750 +0,0 @@
-/* SPDX-License-Identifier: LGPL-2.1 */
-/*
- * Copyright (C) 2009, 2010 Red Hat Inc, Steven Rostedt <srostedt@redhat.com>
- *
- */
-#ifndef _PARSE_EVENTS_H
-#define _PARSE_EVENTS_H
-
-#include <stdbool.h>
-#include <stdarg.h>
-#include <stdio.h>
-#include <regex.h>
-#include <string.h>
-
-#include "trace-seq.h"
-
-#ifndef __maybe_unused
-#define __maybe_unused __attribute__((unused))
-#endif
-
-#ifndef DEBUG_RECORD
-#define DEBUG_RECORD 0
-#endif
-
-struct tep_record {
- unsigned long long ts;
- unsigned long long offset;
- long long missed_events; /* buffer dropped events before */
- int record_size; /* size of binary record */
- int size; /* size of data */
- void *data;
- int cpu;
- int ref_count;
- int locked; /* Do not free, even if ref_count is zero */
- void *priv;
-#if DEBUG_RECORD
- struct tep_record *prev;
- struct tep_record *next;
- long alloc_addr;
-#endif
-};
-
-/* ----------------------- tep ----------------------- */
-
-struct tep_handle;
-struct tep_event;
-
-typedef int (*tep_event_handler_func)(struct trace_seq *s,
- struct tep_record *record,
- struct tep_event *event,
- void *context);
-
-typedef int (*tep_plugin_load_func)(struct tep_handle *tep);
-typedef int (*tep_plugin_unload_func)(struct tep_handle *tep);
-
-struct tep_plugin_option {
- struct tep_plugin_option *next;
- void *handle;
- char *file;
- char *name;
- char *plugin_alias;
- char *description;
- const char *value;
- void *priv;
- int set;
-};
-
-/*
- * Plugin hooks that can be called:
- *
- * TEP_PLUGIN_LOADER: (required)
- * The function name to initialized the plugin.
- *
- * int TEP_PLUGIN_LOADER(struct tep_handle *tep)
- *
- * TEP_PLUGIN_UNLOADER: (optional)
- * The function called just before unloading
- *
- * int TEP_PLUGIN_UNLOADER(struct tep_handle *tep)
- *
- * TEP_PLUGIN_OPTIONS: (optional)
- * Plugin options that can be set before loading
- *
- * struct tep_plugin_option TEP_PLUGIN_OPTIONS[] = {
- * {
- * .name = "option-name",
- * .plugin_alias = "override-file-name", (optional)
- * .description = "description of option to show users",
- * },
- * {
- * .name = NULL,
- * },
- * };
- *
- * Array must end with .name = NULL;
- *
- *
- * .plugin_alias is used to give a shorter name to access
- * the vairable. Useful if a plugin handles more than one event.
- *
- * If .value is not set, then it is considered a boolean and only
- * .set will be processed. If .value is defined, then it is considered
- * a string option and .set will be ignored.
- *
- * TEP_PLUGIN_ALIAS: (optional)
- * The name to use for finding options (uses filename if not defined)
- */
-#define TEP_PLUGIN_LOADER tep_plugin_loader
-#define TEP_PLUGIN_UNLOADER tep_plugin_unloader
-#define TEP_PLUGIN_OPTIONS tep_plugin_options
-#define TEP_PLUGIN_ALIAS tep_plugin_alias
-#define _MAKE_STR(x) #x
-#define MAKE_STR(x) _MAKE_STR(x)
-#define TEP_PLUGIN_LOADER_NAME MAKE_STR(TEP_PLUGIN_LOADER)
-#define TEP_PLUGIN_UNLOADER_NAME MAKE_STR(TEP_PLUGIN_UNLOADER)
-#define TEP_PLUGIN_OPTIONS_NAME MAKE_STR(TEP_PLUGIN_OPTIONS)
-#define TEP_PLUGIN_ALIAS_NAME MAKE_STR(TEP_PLUGIN_ALIAS)
-
-enum tep_format_flags {
- TEP_FIELD_IS_ARRAY = 1,
- TEP_FIELD_IS_POINTER = 2,
- TEP_FIELD_IS_SIGNED = 4,
- TEP_FIELD_IS_STRING = 8,
- TEP_FIELD_IS_DYNAMIC = 16,
- TEP_FIELD_IS_LONG = 32,
- TEP_FIELD_IS_FLAG = 64,
- TEP_FIELD_IS_SYMBOLIC = 128,
- TEP_FIELD_IS_RELATIVE = 256,
-};
-
-struct tep_format_field {
- struct tep_format_field *next;
- struct tep_event *event;
- char *type;
- char *name;
- char *alias;
- int offset;
- int size;
- unsigned int arraylen;
- unsigned int elementsize;
- unsigned long flags;
-};
-
-struct tep_format {
- int nr_common;
- int nr_fields;
- struct tep_format_field *common_fields;
- struct tep_format_field *fields;
-};
-
-struct tep_print_arg_atom {
- char *atom;
-};
-
-struct tep_print_arg_string {
- char *string;
- struct tep_format_field *field;
-};
-
-struct tep_print_arg_bitmask {
- char *bitmask;
- struct tep_format_field *field;
-};
-
-struct tep_print_arg_field {
- char *name;
- struct tep_format_field *field;
-};
-
-struct tep_print_flag_sym {
- struct tep_print_flag_sym *next;
- char *value;
- char *str;
-};
-
-struct tep_print_arg_typecast {
- char *type;
- struct tep_print_arg *item;
-};
-
-struct tep_print_arg_flags {
- struct tep_print_arg *field;
- char *delim;
- struct tep_print_flag_sym *flags;
-};
-
-struct tep_print_arg_symbol {
- struct tep_print_arg *field;
- struct tep_print_flag_sym *symbols;
-};
-
-struct tep_print_arg_hex {
- struct tep_print_arg *field;
- struct tep_print_arg *size;
-};
-
-struct tep_print_arg_int_array {
- struct tep_print_arg *field;
- struct tep_print_arg *count;
- struct tep_print_arg *el_size;
-};
-
-struct tep_print_arg_dynarray {
- struct tep_format_field *field;
- struct tep_print_arg *index;
-};
-
-struct tep_print_arg;
-
-struct tep_print_arg_op {
- char *op;
- int prio;
- struct tep_print_arg *left;
- struct tep_print_arg *right;
-};
-
-struct tep_function_handler;
-
-struct tep_print_arg_func {
- struct tep_function_handler *func;
- struct tep_print_arg *args;
-};
-
-enum tep_print_arg_type {
- TEP_PRINT_NULL,
- TEP_PRINT_ATOM,
- TEP_PRINT_FIELD,
- TEP_PRINT_FLAGS,
- TEP_PRINT_SYMBOL,
- TEP_PRINT_HEX,
- TEP_PRINT_INT_ARRAY,
- TEP_PRINT_TYPE,
- TEP_PRINT_STRING,
- TEP_PRINT_BSTRING,
- TEP_PRINT_DYNAMIC_ARRAY,
- TEP_PRINT_OP,
- TEP_PRINT_FUNC,
- TEP_PRINT_BITMASK,
- TEP_PRINT_DYNAMIC_ARRAY_LEN,
- TEP_PRINT_HEX_STR,
-};
-
-struct tep_print_arg {
- struct tep_print_arg *next;
- enum tep_print_arg_type type;
- union {
- struct tep_print_arg_atom atom;
- struct tep_print_arg_field field;
- struct tep_print_arg_typecast typecast;
- struct tep_print_arg_flags flags;
- struct tep_print_arg_symbol symbol;
- struct tep_print_arg_hex hex;
- struct tep_print_arg_int_array int_array;
- struct tep_print_arg_func func;
- struct tep_print_arg_string string;
- struct tep_print_arg_bitmask bitmask;
- struct tep_print_arg_op op;
- struct tep_print_arg_dynarray dynarray;
- };
-};
-
-struct tep_print_parse;
-
-struct tep_print_fmt {
- char *format;
- struct tep_print_arg *args;
- struct tep_print_parse *print_cache;
-};
-
-struct tep_event {
- struct tep_handle *tep;
- char *name;
- int id;
- int flags;
- struct tep_format format;
- struct tep_print_fmt print_fmt;
- char *system;
- tep_event_handler_func handler;
- void *context;
-};
-
-enum {
- TEP_EVENT_FL_ISFTRACE = 0x01,
- TEP_EVENT_FL_ISPRINT = 0x02,
- TEP_EVENT_FL_ISBPRINT = 0x04,
- TEP_EVENT_FL_ISFUNCENT = 0x10,
- TEP_EVENT_FL_ISFUNCRET = 0x20,
- TEP_EVENT_FL_NOHANDLE = 0x40,
- TEP_EVENT_FL_PRINTRAW = 0x80,
-
- TEP_EVENT_FL_FAILED = 0x80000000
-};
-
-enum tep_event_sort_type {
- TEP_EVENT_SORT_ID,
- TEP_EVENT_SORT_NAME,
- TEP_EVENT_SORT_SYSTEM,
-};
-
-enum tep_event_type {
- TEP_EVENT_ERROR,
- TEP_EVENT_NONE,
- TEP_EVENT_SPACE,
- TEP_EVENT_NEWLINE,
- TEP_EVENT_OP,
- TEP_EVENT_DELIM,
- TEP_EVENT_ITEM,
- TEP_EVENT_DQUOTE,
- TEP_EVENT_SQUOTE,
-};
-
-typedef unsigned long long (*tep_func_handler)(struct trace_seq *s,
- unsigned long long *args);
-
-enum tep_func_arg_type {
- TEP_FUNC_ARG_VOID,
- TEP_FUNC_ARG_INT,
- TEP_FUNC_ARG_LONG,
- TEP_FUNC_ARG_STRING,
- TEP_FUNC_ARG_PTR,
- TEP_FUNC_ARG_MAX_TYPES
-};
-
-enum tep_flag {
- TEP_NSEC_OUTPUT = 1, /* output in NSECS */
- TEP_DISABLE_SYS_PLUGINS = 1 << 1,
- TEP_DISABLE_PLUGINS = 1 << 2,
-};
-
-#define TEP_ERRORS \
- _PE(MEM_ALLOC_FAILED, "failed to allocate memory"), \
- _PE(PARSE_EVENT_FAILED, "failed to parse event"), \
- _PE(READ_ID_FAILED, "failed to read event id"), \
- _PE(READ_FORMAT_FAILED, "failed to read event format"), \
- _PE(READ_PRINT_FAILED, "failed to read event print fmt"), \
- _PE(OLD_FTRACE_ARG_FAILED,"failed to allocate field name for ftrace"),\
- _PE(INVALID_ARG_TYPE, "invalid argument type"), \
- _PE(INVALID_EXP_TYPE, "invalid expression type"), \
- _PE(INVALID_OP_TYPE, "invalid operator type"), \
- _PE(INVALID_EVENT_NAME, "invalid event name"), \
- _PE(EVENT_NOT_FOUND, "no event found"), \
- _PE(SYNTAX_ERROR, "syntax error"), \
- _PE(ILLEGAL_RVALUE, "illegal rvalue"), \
- _PE(ILLEGAL_LVALUE, "illegal lvalue for string comparison"), \
- _PE(INVALID_REGEX, "regex did not compute"), \
- _PE(ILLEGAL_STRING_CMP, "illegal comparison for string"), \
- _PE(ILLEGAL_INTEGER_CMP,"illegal comparison for integer"), \
- _PE(REPARENT_NOT_OP, "cannot reparent other than OP"), \
- _PE(REPARENT_FAILED, "failed to reparent filter OP"), \
- _PE(BAD_FILTER_ARG, "bad arg in filter tree"), \
- _PE(UNEXPECTED_TYPE, "unexpected type (not a value)"), \
- _PE(ILLEGAL_TOKEN, "illegal token"), \
- _PE(INVALID_PAREN, "open parenthesis cannot come here"), \
- _PE(UNBALANCED_PAREN, "unbalanced number of parenthesis"), \
- _PE(UNKNOWN_TOKEN, "unknown token"), \
- _PE(FILTER_NOT_FOUND, "no filter found"), \
- _PE(NOT_A_NUMBER, "must have number field"), \
- _PE(NO_FILTER, "no filters exists"), \
- _PE(FILTER_MISS, "record does not match to filter")
-
-#undef _PE
-#define _PE(__code, __str) TEP_ERRNO__ ## __code
-enum tep_errno {
- TEP_ERRNO__SUCCESS = 0,
- TEP_ERRNO__FILTER_MATCH = TEP_ERRNO__SUCCESS,
-
- /*
- * Choose an arbitrary negative big number not to clash with standard
- * errno since SUS requires the errno has distinct positive values.
- * See 'Issue 6' in the link below.
- *
- * https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/errno.h.html
- */
- __TEP_ERRNO__START = -100000,
-
- TEP_ERRORS,
-
- __TEP_ERRNO__END,
-};
-#undef _PE
-
-struct tep_plugin_list;
-
-#define INVALID_PLUGIN_LIST_OPTION ((char **)((unsigned long)-1))
-
-enum tep_plugin_load_priority {
- TEP_PLUGIN_FIRST,
- TEP_PLUGIN_LAST,
-};
-
-int tep_add_plugin_path(struct tep_handle *tep, char *path,
- enum tep_plugin_load_priority prio);
-struct tep_plugin_list *tep_load_plugins(struct tep_handle *tep);
-void tep_unload_plugins(struct tep_plugin_list *plugin_list,
- struct tep_handle *tep);
-void tep_load_plugins_hook(struct tep_handle *tep, const char *suffix,
- void (*load_plugin)(struct tep_handle *tep,
- const char *path,
- const char *name,
- void *data),
- void *data);
-char **tep_plugin_list_options(void);
-void tep_plugin_free_options_list(char **list);
-int tep_plugin_add_options(const char *name,
- struct tep_plugin_option *options);
-int tep_plugin_add_option(const char *name, const char *val);
-void tep_plugin_remove_options(struct tep_plugin_option *options);
-void tep_plugin_print_options(struct trace_seq *s);
-void tep_print_plugins(struct trace_seq *s,
- const char *prefix, const char *suffix,
- const struct tep_plugin_list *list);
-
-/* tep_handle */
-typedef char *(tep_func_resolver_t)(void *priv,
- unsigned long long *addrp, char **modp);
-void tep_set_flag(struct tep_handle *tep, int flag);
-void tep_clear_flag(struct tep_handle *tep, enum tep_flag flag);
-bool tep_test_flag(struct tep_handle *tep, enum tep_flag flags);
-
-static inline int tep_is_bigendian(void)
-{
- unsigned char str[] = { 0x1, 0x2, 0x3, 0x4 };
- unsigned int val;
-
- memcpy(&val, str, 4);
- return val == 0x01020304;
-}
-
-/* taken from kernel/trace/trace.h */
-enum trace_flag_type {
- TRACE_FLAG_IRQS_OFF = 0x01,
- TRACE_FLAG_IRQS_NOSUPPORT = 0x02,
- TRACE_FLAG_NEED_RESCHED = 0x04,
- TRACE_FLAG_HARDIRQ = 0x08,
- TRACE_FLAG_SOFTIRQ = 0x10,
-};
-
-int tep_set_function_resolver(struct tep_handle *tep,
- tep_func_resolver_t *func, void *priv);
-void tep_reset_function_resolver(struct tep_handle *tep);
-int tep_register_comm(struct tep_handle *tep, const char *comm, int pid);
-int tep_override_comm(struct tep_handle *tep, const char *comm, int pid);
-int tep_register_function(struct tep_handle *tep, char *name,
- unsigned long long addr, char *mod);
-int tep_register_print_string(struct tep_handle *tep, const char *fmt,
- unsigned long long addr);
-bool tep_is_pid_registered(struct tep_handle *tep, int pid);
-
-struct tep_event *tep_get_event(struct tep_handle *tep, int index);
-
-#define TEP_PRINT_INFO "INFO"
-#define TEP_PRINT_INFO_RAW "INFO_RAW"
-#define TEP_PRINT_COMM "COMM"
-#define TEP_PRINT_LATENCY "LATENCY"
-#define TEP_PRINT_NAME "NAME"
-#define TEP_PRINT_PID 1U
-#define TEP_PRINT_TIME 2U
-#define TEP_PRINT_CPU 3U
-
-void tep_print_event(struct tep_handle *tep, struct trace_seq *s,
- struct tep_record *record, const char *fmt, ...)
- __attribute__ ((format (printf, 4, 5)));
-
-int tep_parse_header_page(struct tep_handle *tep, char *buf, unsigned long size,
- int long_size);
-
-enum tep_errno tep_parse_event(struct tep_handle *tep, const char *buf,
- unsigned long size, const char *sys);
-enum tep_errno tep_parse_format(struct tep_handle *tep,
- struct tep_event **eventp,
- const char *buf,
- unsigned long size, const char *sys);
-
-void *tep_get_field_raw(struct trace_seq *s, struct tep_event *event,
- const char *name, struct tep_record *record,
- int *len, int err);
-
-int tep_get_field_val(struct trace_seq *s, struct tep_event *event,
- const char *name, struct tep_record *record,
- unsigned long long *val, int err);
-int tep_get_common_field_val(struct trace_seq *s, struct tep_event *event,
- const char *name, struct tep_record *record,
- unsigned long long *val, int err);
-int tep_get_any_field_val(struct trace_seq *s, struct tep_event *event,
- const char *name, struct tep_record *record,
- unsigned long long *val, int err);
-
-int tep_print_num_field(struct trace_seq *s, const char *fmt,
- struct tep_event *event, const char *name,
- struct tep_record *record, int err);
-
-int tep_print_func_field(struct trace_seq *s, const char *fmt,
- struct tep_event *event, const char *name,
- struct tep_record *record, int err);
-
-enum tep_reg_handler {
- TEP_REGISTER_SUCCESS = 0,
- TEP_REGISTER_SUCCESS_OVERWRITE,
-};
-
-int tep_register_event_handler(struct tep_handle *tep, int id,
- const char *sys_name, const char *event_name,
- tep_event_handler_func func, void *context);
-int tep_unregister_event_handler(struct tep_handle *tep, int id,
- const char *sys_name, const char *event_name,
- tep_event_handler_func func, void *context);
-int tep_register_print_function(struct tep_handle *tep,
- tep_func_handler func,
- enum tep_func_arg_type ret_type,
- char *name, ...);
-int tep_unregister_print_function(struct tep_handle *tep,
- tep_func_handler func, char *name);
-
-struct tep_format_field *tep_find_common_field(struct tep_event *event, const char *name);
-struct tep_format_field *tep_find_field(struct tep_event *event, const char *name);
-struct tep_format_field *tep_find_any_field(struct tep_event *event, const char *name);
-
-const char *tep_find_function(struct tep_handle *tep, unsigned long long addr);
-unsigned long long
-tep_find_function_address(struct tep_handle *tep, unsigned long long addr);
-unsigned long long tep_read_number(struct tep_handle *tep, const void *ptr, int size);
-int tep_read_number_field(struct tep_format_field *field, const void *data,
- unsigned long long *value);
-
-struct tep_event *tep_get_first_event(struct tep_handle *tep);
-int tep_get_events_count(struct tep_handle *tep);
-struct tep_event *tep_find_event(struct tep_handle *tep, int id);
-
-struct tep_event *
-tep_find_event_by_name(struct tep_handle *tep, const char *sys, const char *name);
-struct tep_event *
-tep_find_event_by_record(struct tep_handle *tep, struct tep_record *record);
-
-int tep_data_type(struct tep_handle *tep, struct tep_record *rec);
-int tep_data_pid(struct tep_handle *tep, struct tep_record *rec);
-int tep_data_preempt_count(struct tep_handle *tep, struct tep_record *rec);
-int tep_data_flags(struct tep_handle *tep, struct tep_record *rec);
-const char *tep_data_comm_from_pid(struct tep_handle *tep, int pid);
-struct tep_cmdline;
-struct tep_cmdline *tep_data_pid_from_comm(struct tep_handle *tep, const char *comm,
- struct tep_cmdline *next);
-int tep_cmdline_pid(struct tep_handle *tep, struct tep_cmdline *cmdline);
-
-void tep_print_field(struct trace_seq *s, void *data,
- struct tep_format_field *field);
-void tep_print_fields(struct trace_seq *s, void *data,
- int size __maybe_unused, struct tep_event *event);
-int tep_strerror(struct tep_handle *tep, enum tep_errno errnum,
- char *buf, size_t buflen);
-
-struct tep_event **tep_list_events(struct tep_handle *tep, enum tep_event_sort_type);
-struct tep_event **tep_list_events_copy(struct tep_handle *tep,
- enum tep_event_sort_type);
-struct tep_format_field **tep_event_common_fields(struct tep_event *event);
-struct tep_format_field **tep_event_fields(struct tep_event *event);
-
-enum tep_endian {
- TEP_LITTLE_ENDIAN = 0,
- TEP_BIG_ENDIAN
-};
-int tep_get_cpus(struct tep_handle *tep);
-void tep_set_cpus(struct tep_handle *tep, int cpus);
-int tep_get_long_size(struct tep_handle *tep);
-void tep_set_long_size(struct tep_handle *tep, int long_size);
-int tep_get_page_size(struct tep_handle *tep);
-void tep_set_page_size(struct tep_handle *tep, int _page_size);
-bool tep_is_file_bigendian(struct tep_handle *tep);
-void tep_set_file_bigendian(struct tep_handle *tep, enum tep_endian endian);
-bool tep_is_local_bigendian(struct tep_handle *tep);
-void tep_set_local_bigendian(struct tep_handle *tep, enum tep_endian endian);
-int tep_get_header_page_size(struct tep_handle *tep);
-int tep_get_header_timestamp_size(struct tep_handle *tep);
-bool tep_is_old_format(struct tep_handle *tep);
-void tep_set_test_filters(struct tep_handle *tep, int test_filters);
-
-struct tep_handle *tep_alloc(void);
-void tep_free(struct tep_handle *tep);
-void tep_ref(struct tep_handle *tep);
-void tep_unref(struct tep_handle *tep);
-int tep_get_ref(struct tep_handle *tep);
-
-/* for debugging */
-void tep_print_funcs(struct tep_handle *tep);
-void tep_print_printk(struct tep_handle *tep);
-
-/* ----------------------- filtering ----------------------- */
-
-enum tep_filter_boolean_type {
- TEP_FILTER_FALSE,
- TEP_FILTER_TRUE,
-};
-
-enum tep_filter_op_type {
- TEP_FILTER_OP_AND = 1,
- TEP_FILTER_OP_OR,
- TEP_FILTER_OP_NOT,
-};
-
-enum tep_filter_cmp_type {
- TEP_FILTER_CMP_NONE,
- TEP_FILTER_CMP_EQ,
- TEP_FILTER_CMP_NE,
- TEP_FILTER_CMP_GT,
- TEP_FILTER_CMP_LT,
- TEP_FILTER_CMP_GE,
- TEP_FILTER_CMP_LE,
- TEP_FILTER_CMP_MATCH,
- TEP_FILTER_CMP_NOT_MATCH,
- TEP_FILTER_CMP_REGEX,
- TEP_FILTER_CMP_NOT_REGEX,
-};
-
-enum tep_filter_exp_type {
- TEP_FILTER_EXP_NONE,
- TEP_FILTER_EXP_ADD,
- TEP_FILTER_EXP_SUB,
- TEP_FILTER_EXP_MUL,
- TEP_FILTER_EXP_DIV,
- TEP_FILTER_EXP_MOD,
- TEP_FILTER_EXP_RSHIFT,
- TEP_FILTER_EXP_LSHIFT,
- TEP_FILTER_EXP_AND,
- TEP_FILTER_EXP_OR,
- TEP_FILTER_EXP_XOR,
- TEP_FILTER_EXP_NOT,
-};
-
-enum tep_filter_arg_type {
- TEP_FILTER_ARG_NONE,
- TEP_FILTER_ARG_BOOLEAN,
- TEP_FILTER_ARG_VALUE,
- TEP_FILTER_ARG_FIELD,
- TEP_FILTER_ARG_EXP,
- TEP_FILTER_ARG_OP,
- TEP_FILTER_ARG_NUM,
- TEP_FILTER_ARG_STR,
-};
-
-enum tep_filter_value_type {
- TEP_FILTER_NUMBER,
- TEP_FILTER_STRING,
- TEP_FILTER_CHAR
-};
-
-struct tep_filter_arg;
-
-struct tep_filter_arg_boolean {
- enum tep_filter_boolean_type value;
-};
-
-struct tep_filter_arg_field {
- struct tep_format_field *field;
-};
-
-struct tep_filter_arg_value {
- enum tep_filter_value_type type;
- union {
- char *str;
- unsigned long long val;
- };
-};
-
-struct tep_filter_arg_op {
- enum tep_filter_op_type type;
- struct tep_filter_arg *left;
- struct tep_filter_arg *right;
-};
-
-struct tep_filter_arg_exp {
- enum tep_filter_exp_type type;
- struct tep_filter_arg *left;
- struct tep_filter_arg *right;
-};
-
-struct tep_filter_arg_num {
- enum tep_filter_cmp_type type;
- struct tep_filter_arg *left;
- struct tep_filter_arg *right;
-};
-
-struct tep_filter_arg_str {
- enum tep_filter_cmp_type type;
- struct tep_format_field *field;
- char *val;
- char *buffer;
- regex_t reg;
-};
-
-struct tep_filter_arg {
- enum tep_filter_arg_type type;
- union {
- struct tep_filter_arg_boolean boolean;
- struct tep_filter_arg_field field;
- struct tep_filter_arg_value value;
- struct tep_filter_arg_op op;
- struct tep_filter_arg_exp exp;
- struct tep_filter_arg_num num;
- struct tep_filter_arg_str str;
- };
-};
-
-struct tep_filter_type {
- int event_id;
- struct tep_event *event;
- struct tep_filter_arg *filter;
-};
-
-#define TEP_FILTER_ERROR_BUFSZ 1024
-
-struct tep_event_filter {
- struct tep_handle *tep;
- int filters;
- struct tep_filter_type *event_filters;
- char error_buffer[TEP_FILTER_ERROR_BUFSZ];
-};
-
-struct tep_event_filter *tep_filter_alloc(struct tep_handle *tep);
-
-/* for backward compatibility */
-#define FILTER_NONE TEP_ERRNO__NO_FILTER
-#define FILTER_NOEXIST TEP_ERRNO__FILTER_NOT_FOUND
-#define FILTER_MISS TEP_ERRNO__FILTER_MISS
-#define FILTER_MATCH TEP_ERRNO__FILTER_MATCH
-
-enum tep_errno tep_filter_add_filter_str(struct tep_event_filter *filter,
- const char *filter_str);
-
-enum tep_errno tep_filter_match(struct tep_event_filter *filter,
- struct tep_record *record);
-
-int tep_filter_strerror(struct tep_event_filter *filter, enum tep_errno err,
- char *buf, size_t buflen);
-
-int tep_event_filtered(struct tep_event_filter *filter,
- int event_id);
-
-void tep_filter_reset(struct tep_event_filter *filter);
-
-void tep_filter_free(struct tep_event_filter *filter);
-
-char *tep_filter_make_string(struct tep_event_filter *filter, int event_id);
-
-int tep_filter_remove_event(struct tep_event_filter *filter,
- int event_id);
-
-int tep_filter_copy(struct tep_event_filter *dest, struct tep_event_filter *source);
-
-int tep_filter_compare(struct tep_event_filter *filter1, struct tep_event_filter *filter2);
-
-#endif /* _PARSE_EVENTS_H */
diff --git a/tools/lib/traceevent/event-plugin.c b/tools/lib/traceevent/event-plugin.c
deleted file mode 100644
index e7f93d5..0000000
--- a/tools/lib/traceevent/event-plugin.c
+++ /dev/null
@@ -1,711 +0,0 @@
-// SPDX-License-Identifier: LGPL-2.1
-/*
- * Copyright (C) 2009, 2010 Red Hat Inc, Steven Rostedt <srostedt@redhat.com>
- *
- */
-
-#include <ctype.h>
-#include <stdio.h>
-#include <string.h>
-#include <dlfcn.h>
-#include <stdlib.h>
-#include <sys/types.h>
-#include <sys/stat.h>
-#include <unistd.h>
-#include <dirent.h>
-#include <errno.h>
-#include "event-parse.h"
-#include "event-parse-local.h"
-#include "event-utils.h"
-#include "trace-seq.h"
-
-#define LOCAL_PLUGIN_DIR ".local/lib/traceevent/plugins/"
-
-static struct registered_plugin_options {
- struct registered_plugin_options *next;
- struct tep_plugin_option *options;
-} *registered_options;
-
-static struct trace_plugin_options {
- struct trace_plugin_options *next;
- char *plugin;
- char *option;
- char *value;
-} *trace_plugin_options;
-
-struct tep_plugin_list {
- struct tep_plugin_list *next;
- char *name;
- void *handle;
-};
-
-struct tep_plugins_dir {
- struct tep_plugins_dir *next;
- char *path;
- enum tep_plugin_load_priority prio;
-};
-
-static void lower_case(char *str)
-{
- if (!str)
- return;
- for (; *str; str++)
- *str = tolower(*str);
-}
-
-static int update_option_value(struct tep_plugin_option *op, const char *val)
-{
- char *op_val;
-
- if (!val) {
- /* toggle, only if option is boolean */
- if (op->value)
- /* Warn? */
- return 0;
- op->set ^= 1;
- return 0;
- }
-
- /*
- * If the option has a value then it takes a string
- * otherwise the option is a boolean.
- */
- if (op->value) {
- op->value = val;
- return 0;
- }
-
- /* Option is boolean, must be either "1", "0", "true" or "false" */
-
- op_val = strdup(val);
- if (!op_val)
- return -1;
- lower_case(op_val);
-
- if (strcmp(val, "1") == 0 || strcmp(val, "true") == 0)
- op->set = 1;
- else if (strcmp(val, "0") == 0 || strcmp(val, "false") == 0)
- op->set = 0;
- free(op_val);
-
- return 0;
-}
-
-/**
- * tep_plugin_list_options - get list of plugin options
- *
- * Returns an array of char strings that list the currently registered
- * plugin options in the format of <plugin>:<option>. This list can be
- * used by toggling the option.
- *
- * Returns NULL if there's no options registered. On error it returns
- * INVALID_PLUGIN_LIST_OPTION
- *
- * Must be freed with tep_plugin_free_options_list().
- */
-char **tep_plugin_list_options(void)
-{
- struct registered_plugin_options *reg;
- struct tep_plugin_option *op;
- char **list = NULL;
- char *name;
- int count = 0;
-
- for (reg = registered_options; reg; reg = reg->next) {
- for (op = reg->options; op->name; op++) {
- char *alias = op->plugin_alias ? op->plugin_alias : op->file;
- char **temp = list;
- int ret;
-
- ret = asprintf(&name, "%s:%s", alias, op->name);
- if (ret < 0)
- goto err;
-
- list = realloc(list, count + 2);
- if (!list) {
- list = temp;
- free(name);
- goto err;
- }
- list[count++] = name;
- list[count] = NULL;
- }
- }
- return list;
-
- err:
- while (--count >= 0)
- free(list[count]);
- free(list);
-
- return INVALID_PLUGIN_LIST_OPTION;
-}
-
-void tep_plugin_free_options_list(char **list)
-{
- int i;
-
- if (!list)
- return;
-
- if (list == INVALID_PLUGIN_LIST_OPTION)
- return;
-
- for (i = 0; list[i]; i++)
- free(list[i]);
-
- free(list);
-}
-
-static int
-update_option(const char *file, struct tep_plugin_option *option)
-{
- struct trace_plugin_options *op;
- char *plugin;
- int ret = 0;
-
- if (option->plugin_alias) {
- plugin = strdup(option->plugin_alias);
- if (!plugin)
- return -1;
- } else {
- char *p;
- plugin = strdup(file);
- if (!plugin)
- return -1;
- p = strstr(plugin, ".");
- if (p)
- *p = '\0';
- }
-
- /* first look for named options */
- for (op = trace_plugin_options; op; op = op->next) {
- if (!op->plugin)
- continue;
- if (strcmp(op->plugin, plugin) != 0)
- continue;
- if (strcmp(op->option, option->name) != 0)
- continue;
-
- ret = update_option_value(option, op->value);
- if (ret)
- goto out;
- break;
- }
-
- /* first look for unnamed options */
- for (op = trace_plugin_options; op; op = op->next) {
- if (op->plugin)
- continue;
- if (strcmp(op->option, option->name) != 0)
- continue;
-
- ret = update_option_value(option, op->value);
- break;
- }
-
- out:
- free(plugin);
- return ret;
-}
-
-/**
- * tep_plugin_add_options - Add a set of options by a plugin
- * @name: The name of the plugin adding the options
- * @options: The set of options being loaded
- *
- * Sets the options with the values that have been added by user.
- */
-int tep_plugin_add_options(const char *name,
- struct tep_plugin_option *options)
-{
- struct registered_plugin_options *reg;
-
- reg = malloc(sizeof(*reg));
- if (!reg)
- return -1;
- reg->next = registered_options;
- reg->options = options;
- registered_options = reg;
-
- while (options->name) {
- update_option(name, options);
- options++;
- }
- return 0;
-}
-
-/**
- * tep_plugin_remove_options - remove plugin options that were registered
- * @options: Options to removed that were registered with tep_plugin_add_options
- */
-void tep_plugin_remove_options(struct tep_plugin_option *options)
-{
- struct registered_plugin_options **last;
- struct registered_plugin_options *reg;
-
- for (last = ®istered_options; *last; last = &(*last)->next) {
- if ((*last)->options == options) {
- reg = *last;
- *last = reg->next;
- free(reg);
- return;
- }
- }
-}
-
-static int parse_option_name(char **option, char **plugin)
-{
- char *p;
-
- *plugin = NULL;
-
- if ((p = strstr(*option, ":"))) {
- *plugin = *option;
- *p = '\0';
- *option = strdup(p + 1);
- if (!*option)
- return -1;
- }
- return 0;
-}
-
-static struct tep_plugin_option *
-find_registered_option(const char *plugin, const char *option)
-{
- struct registered_plugin_options *reg;
- struct tep_plugin_option *op;
- const char *op_plugin;
-
- for (reg = registered_options; reg; reg = reg->next) {
- for (op = reg->options; op->name; op++) {
- if (op->plugin_alias)
- op_plugin = op->plugin_alias;
- else
- op_plugin = op->file;
-
- if (plugin && strcmp(plugin, op_plugin) != 0)
- continue;
- if (strcmp(option, op->name) != 0)
- continue;
-
- return op;
- }
- }
-
- return NULL;
-}
-
-static int process_option(const char *plugin, const char *option, const char *val)
-{
- struct tep_plugin_option *op;
-
- op = find_registered_option(plugin, option);
- if (!op)
- return 0;
-
- return update_option_value(op, val);
-}
-
-/**
- * tep_plugin_add_option - add an option/val pair to set plugin options
- * @name: The name of the option (format: <plugin>:<option> or just <option>)
- * @val: (optional) the value for the option
- *
- * Modify a plugin option. If @val is given than the value of the option
- * is set (note, some options just take a boolean, so @val must be either
- * "1" or "0" or "true" or "false").
- */
-int tep_plugin_add_option(const char *name, const char *val)
-{
- struct trace_plugin_options *op;
- char *option_str;
- char *plugin;
-
- option_str = strdup(name);
- if (!option_str)
- return -ENOMEM;
-
- if (parse_option_name(&option_str, &plugin) < 0)
- return -ENOMEM;
-
- /* If the option exists, update the val */
- for (op = trace_plugin_options; op; op = op->next) {
- /* Both must be NULL or not NULL */
- if ((!plugin || !op->plugin) && plugin != op->plugin)
- continue;
- if (plugin && strcmp(plugin, op->plugin) != 0)
- continue;
- if (strcmp(op->option, option_str) != 0)
- continue;
-
- /* update option */
- free(op->value);
- if (val) {
- op->value = strdup(val);
- if (!op->value)
- goto out_free;
- } else
- op->value = NULL;
-
- /* plugin and option_str don't get freed at the end */
- free(plugin);
- free(option_str);
-
- plugin = op->plugin;
- option_str = op->option;
- break;
- }
-
- /* If not found, create */
- if (!op) {
- op = malloc(sizeof(*op));
- if (!op)
- goto out_free;
- memset(op, 0, sizeof(*op));
- op->plugin = plugin;
- op->option = option_str;
- if (val) {
- op->value = strdup(val);
- if (!op->value) {
- free(op);
- goto out_free;
- }
- }
- op->next = trace_plugin_options;
- trace_plugin_options = op;
- }
-
- return process_option(plugin, option_str, val);
-
-out_free:
- free(plugin);
- free(option_str);
- return -ENOMEM;
-}
-
-static void print_op_data(struct trace_seq *s, const char *name,
- const char *op)
-{
- if (op)
- trace_seq_printf(s, "%8s:\t%s\n", name, op);
-}
-
-/**
- * tep_plugin_print_options - print out the registered plugin options
- * @s: The trace_seq descriptor to write the plugin options into
- *
- * Writes a list of options into trace_seq @s.
- */
-void tep_plugin_print_options(struct trace_seq *s)
-{
- struct registered_plugin_options *reg;
- struct tep_plugin_option *op;
-
- for (reg = registered_options; reg; reg = reg->next) {
- if (reg != registered_options)
- trace_seq_printf(s, "============\n");
- for (op = reg->options; op->name; op++) {
- if (op != reg->options)
- trace_seq_printf(s, "------------\n");
- print_op_data(s, "file", op->file);
- print_op_data(s, "plugin", op->plugin_alias);
- print_op_data(s, "option", op->name);
- print_op_data(s, "desc", op->description);
- print_op_data(s, "value", op->value);
- trace_seq_printf(s, "%8s:\t%d\n", "set", op->set);
- }
- }
-}
-
-/**
- * tep_print_plugins - print out the list of plugins loaded
- * @s: the trace_seq descripter to write to
- * @prefix: The prefix string to add before listing the option name
- * @suffix: The suffix string ot append after the option name
- * @list: The list of plugins (usually returned by tep_load_plugins()
- *
- * Writes to the trace_seq @s the list of plugins (files) that is
- * returned by tep_load_plugins(). Use @prefix and @suffix for formating:
- * @prefix = " ", @suffix = "\n".
- */
-void tep_print_plugins(struct trace_seq *s,
- const char *prefix, const char *suffix,
- const struct tep_plugin_list *list)
-{
- while (list) {
- trace_seq_printf(s, "%s%s%s", prefix, list->name, suffix);
- list = list->next;
- }
-}
-
-static void
-load_plugin(struct tep_handle *tep, const char *path,
- const char *file, void *data)
-{
- struct tep_plugin_list **plugin_list = data;
- struct tep_plugin_option *options;
- tep_plugin_load_func func;
- struct tep_plugin_list *list;
- const char *alias;
- char *plugin;
- void *handle;
- int ret;
-
- ret = asprintf(&plugin, "%s/%s", path, file);
- if (ret < 0) {
- warning("could not allocate plugin memory\n");
- return;
- }
-
- handle = dlopen(plugin, RTLD_NOW | RTLD_GLOBAL);
- if (!handle) {
- warning("could not load plugin '%s'\n%s\n",
- plugin, dlerror());
- goto out_free;
- }
-
- alias = dlsym(handle, TEP_PLUGIN_ALIAS_NAME);
- if (!alias)
- alias = file;
-
- options = dlsym(handle, TEP_PLUGIN_OPTIONS_NAME);
- if (options) {
- while (options->name) {
- ret = update_option(alias, options);
- if (ret < 0)
- goto out_free;
- options++;
- }
- }
-
- func = dlsym(handle, TEP_PLUGIN_LOADER_NAME);
- if (!func) {
- warning("could not find func '%s' in plugin '%s'\n%s\n",
- TEP_PLUGIN_LOADER_NAME, plugin, dlerror());
- goto out_free;
- }
-
- list = malloc(sizeof(*list));
- if (!list) {
- warning("could not allocate plugin memory\n");
- goto out_free;
- }
-
- list->next = *plugin_list;
- list->handle = handle;
- list->name = plugin;
- *plugin_list = list;
-
- pr_stat("registering plugin: %s", plugin);
- func(tep);
- return;
-
- out_free:
- free(plugin);
-}
-
-static void
-load_plugins_dir(struct tep_handle *tep, const char *suffix,
- const char *path,
- void (*load_plugin)(struct tep_handle *tep,
- const char *path,
- const char *name,
- void *data),
- void *data)
-{
- struct dirent *dent;
- struct stat st;
- DIR *dir;
- int ret;
-
- ret = stat(path, &st);
- if (ret < 0)
- return;
-
- if (!S_ISDIR(st.st_mode))
- return;
-
- dir = opendir(path);
- if (!dir)
- return;
-
- while ((dent = readdir(dir))) {
- const char *name = dent->d_name;
-
- if (strcmp(name, ".") == 0 ||
- strcmp(name, "..") == 0)
- continue;
-
- /* Only load plugins that end in suffix */
- if (strcmp(name + (strlen(name) - strlen(suffix)), suffix) != 0)
- continue;
-
- load_plugin(tep, path, name, data);
- }
-
- closedir(dir);
-}
-
-/**
- * tep_load_plugins_hook - call a user specified callback to load a plugin
- * @tep: handler to traceevent context
- * @suffix: filter only plugin files with given suffix
- * @load_plugin: user specified callback, called for each plugin file
- * @data: custom context, passed to @load_plugin
- *
- * Searches for traceevent plugin files and calls @load_plugin for each
- * The order of plugins search is:
- * - Directories, specified in @tep->plugins_dir and priority TEP_PLUGIN_FIRST
- * - Directory, specified at compile time with PLUGIN_TRACEEVENT_DIR
- * - Directory, specified by environment variable TRACEEVENT_PLUGIN_DIR
- * - In user's home: ~/.local/lib/traceevent/plugins/
- * - Directories, specified in @tep->plugins_dir and priority TEP_PLUGIN_LAST
- *
- */
-void tep_load_plugins_hook(struct tep_handle *tep, const char *suffix,
- void (*load_plugin)(struct tep_handle *tep,
- const char *path,
- const char *name,
- void *data),
- void *data)
-{
- struct tep_plugins_dir *dir = NULL;
- char *home;
- char *path;
- char *envdir;
- int ret;
-
- if (tep && tep->flags & TEP_DISABLE_PLUGINS)
- return;
-
- if (tep)
- dir = tep->plugins_dir;
- while (dir) {
- if (dir->prio == TEP_PLUGIN_FIRST)
- load_plugins_dir(tep, suffix, dir->path,
- load_plugin, data);
- dir = dir->next;
- }
-
- /*
- * If a system plugin directory was defined,
- * check that first.
- */
-#ifdef PLUGIN_DIR
- if (!tep || !(tep->flags & TEP_DISABLE_SYS_PLUGINS))
- load_plugins_dir(tep, suffix, PLUGIN_DIR,
- load_plugin, data);
-#endif
-
- /*
- * Next let the environment-set plugin directory
- * override the system defaults.
- */
- envdir = getenv("TRACEEVENT_PLUGIN_DIR");
- if (envdir)
- load_plugins_dir(tep, suffix, envdir, load_plugin, data);
-
- /*
- * Now let the home directory override the environment
- * or system defaults.
- */
- home = getenv("HOME");
- if (!home)
- return;
-
- ret = asprintf(&path, "%s/%s", home, LOCAL_PLUGIN_DIR);
- if (ret < 0) {
- warning("could not allocate plugin memory\n");
- return;
- }
-
- load_plugins_dir(tep, suffix, path, load_plugin, data);
-
- if (tep)
- dir = tep->plugins_dir;
- while (dir) {
- if (dir->prio == TEP_PLUGIN_LAST)
- load_plugins_dir(tep, suffix, dir->path,
- load_plugin, data);
- dir = dir->next;
- }
-
- free(path);
-}
-
-struct tep_plugin_list*
-tep_load_plugins(struct tep_handle *tep)
-{
- struct tep_plugin_list *list = NULL;
-
- tep_load_plugins_hook(tep, ".so", load_plugin, &list);
- return list;
-}
-
-/**
- * tep_add_plugin_path - Add a new plugin directory.
- * @tep: Trace event handler.
- * @path: Path to a directory. All plugin files in that
- * directory will be loaded.
- *@prio: Load priority of the plugins in that directory.
- *
- * Returns -1 in case of an error, 0 otherwise.
- */
-int tep_add_plugin_path(struct tep_handle *tep, char *path,
- enum tep_plugin_load_priority prio)
-{
- struct tep_plugins_dir *dir;
-
- if (!tep || !path)
- return -1;
-
- dir = calloc(1, sizeof(*dir));
- if (!dir)
- return -1;
-
- dir->path = strdup(path);
- if (!dir->path) {
- free(dir);
- return -1;
- }
- dir->prio = prio;
- dir->next = tep->plugins_dir;
- tep->plugins_dir = dir;
-
- return 0;
-}
-
-__hidden void free_tep_plugin_paths(struct tep_handle *tep)
-{
- struct tep_plugins_dir *dir;
-
- if (!tep)
- return;
-
- dir = tep->plugins_dir;
- while (dir) {
- tep->plugins_dir = tep->plugins_dir->next;
- free(dir->path);
- free(dir);
- dir = tep->plugins_dir;
- }
-}
-
-void
-tep_unload_plugins(struct tep_plugin_list *plugin_list, struct tep_handle *tep)
-{
- tep_plugin_unload_func func;
- struct tep_plugin_list *list;
-
- while (plugin_list) {
- list = plugin_list;
- plugin_list = list->next;
- func = dlsym(list->handle, TEP_PLUGIN_UNLOADER_NAME);
- if (func)
- func(tep);
- dlclose(list->handle);
- free(list->name);
- free(list);
- }
-}
diff --git a/tools/lib/traceevent/event-utils.h b/tools/lib/traceevent/event-utils.h
deleted file mode 100644
index 0560b96..0000000
--- a/tools/lib/traceevent/event-utils.h
+++ /dev/null
@@ -1,67 +0,0 @@
-/* SPDX-License-Identifier: LGPL-2.1 */
-/*
- * Copyright (C) 2010 Red Hat Inc, Steven Rostedt <srostedt@redhat.com>
- *
- */
-#ifndef __UTIL_H
-#define __UTIL_H
-
-#include <ctype.h>
-
-/* Can be overridden */
-void warning(const char *fmt, ...);
-void pr_stat(const char *fmt, ...);
-void vpr_stat(const char *fmt, va_list ap);
-
-/* Always available */
-void __warning(const char *fmt, ...);
-void __pr_stat(const char *fmt, ...);
-
-void __vwarning(const char *fmt, ...);
-void __vpr_stat(const char *fmt, ...);
-
-#define min(x, y) ({ \
- typeof(x) _min1 = (x); \
- typeof(y) _min2 = (y); \
- (void) (&_min1 == &_min2); \
- _min1 < _min2 ? _min1 : _min2; })
-
-static inline char *strim(char *string)
-{
- char *ret;
-
- if (!string)
- return NULL;
- while (*string) {
- if (!isspace(*string))
- break;
- string++;
- }
- ret = string;
-
- string = ret + strlen(ret) - 1;
- while (string > ret) {
- if (!isspace(*string))
- break;
- string--;
- }
- string[1] = 0;
-
- return ret;
-}
-
-static inline int has_text(const char *text)
-{
- if (!text)
- return 0;
-
- while (*text) {
- if (!isspace(*text))
- return 1;
- text++;
- }
-
- return 0;
-}
-
-#endif
diff --git a/tools/lib/traceevent/kbuffer-parse.c b/tools/lib/traceevent/kbuffer-parse.c
deleted file mode 100644
index f1640d6..0000000
--- a/tools/lib/traceevent/kbuffer-parse.c
+++ /dev/null
@@ -1,809 +0,0 @@
-// SPDX-License-Identifier: LGPL-2.1
-/*
- * Copyright (C) 2009, 2010 Red Hat Inc, Steven Rostedt <srostedt@redhat.com>
- *
- */
-#include <stdio.h>
-#include <stdlib.h>
-#include <string.h>
-
-#include "kbuffer.h"
-
-#define MISSING_EVENTS (1UL << 31)
-#define MISSING_STORED (1UL << 30)
-
-#define COMMIT_MASK ((1 << 27) - 1)
-
-enum {
- KBUFFER_FL_HOST_BIG_ENDIAN = (1<<0),
- KBUFFER_FL_BIG_ENDIAN = (1<<1),
- KBUFFER_FL_LONG_8 = (1<<2),
- KBUFFER_FL_OLD_FORMAT = (1<<3),
-};
-
-#define ENDIAN_MASK (KBUFFER_FL_HOST_BIG_ENDIAN | KBUFFER_FL_BIG_ENDIAN)
-
-/** kbuffer
- * @timestamp - timestamp of current event
- * @lost_events - # of lost events between this subbuffer and previous
- * @flags - special flags of the kbuffer
- * @subbuffer - pointer to the sub-buffer page
- * @data - pointer to the start of data on the sub-buffer page
- * @index - index from @data to the @curr event data
- * @curr - offset from @data to the start of current event
- * (includes metadata)
- * @next - offset from @data to the start of next event
- * @size - The size of data on @data
- * @start - The offset from @subbuffer where @data lives
- *
- * @read_4 - Function to read 4 raw bytes (may swap)
- * @read_8 - Function to read 8 raw bytes (may swap)
- * @read_long - Function to read a long word (4 or 8 bytes with needed swap)
- */
-struct kbuffer {
- unsigned long long timestamp;
- long long lost_events;
- unsigned long flags;
- void *subbuffer;
- void *data;
- unsigned int index;
- unsigned int curr;
- unsigned int next;
- unsigned int size;
- unsigned int start;
-
- unsigned int (*read_4)(void *ptr);
- unsigned long long (*read_8)(void *ptr);
- unsigned long long (*read_long)(struct kbuffer *kbuf, void *ptr);
- int (*next_event)(struct kbuffer *kbuf);
-};
-
-static void *zmalloc(size_t size)
-{
- return calloc(1, size);
-}
-
-static int host_is_bigendian(void)
-{
- unsigned char str[] = { 0x1, 0x2, 0x3, 0x4 };
- unsigned int *ptr;
-
- ptr = (unsigned int *)str;
- return *ptr == 0x01020304;
-}
-
-static int do_swap(struct kbuffer *kbuf)
-{
- return ((kbuf->flags & KBUFFER_FL_HOST_BIG_ENDIAN) + kbuf->flags) &
- ENDIAN_MASK;
-}
-
-static unsigned long long __read_8(void *ptr)
-{
- unsigned long long data = *(unsigned long long *)ptr;
-
- return data;
-}
-
-static unsigned long long __read_8_sw(void *ptr)
-{
- unsigned long long data = *(unsigned long long *)ptr;
- unsigned long long swap;
-
- swap = ((data & 0xffULL) << 56) |
- ((data & (0xffULL << 8)) << 40) |
- ((data & (0xffULL << 16)) << 24) |
- ((data & (0xffULL << 24)) << 8) |
- ((data & (0xffULL << 32)) >> 8) |
- ((data & (0xffULL << 40)) >> 24) |
- ((data & (0xffULL << 48)) >> 40) |
- ((data & (0xffULL << 56)) >> 56);
-
- return swap;
-}
-
-static unsigned int __read_4(void *ptr)
-{
- unsigned int data = *(unsigned int *)ptr;
-
- return data;
-}
-
-static unsigned int __read_4_sw(void *ptr)
-{
- unsigned int data = *(unsigned int *)ptr;
- unsigned int swap;
-
- swap = ((data & 0xffULL) << 24) |
- ((data & (0xffULL << 8)) << 8) |
- ((data & (0xffULL << 16)) >> 8) |
- ((data & (0xffULL << 24)) >> 24);
-
- return swap;
-}
-
-static unsigned long long read_8(struct kbuffer *kbuf, void *ptr)
-{
- return kbuf->read_8(ptr);
-}
-
-static unsigned int read_4(struct kbuffer *kbuf, void *ptr)
-{
- return kbuf->read_4(ptr);
-}
-
-static unsigned long long __read_long_8(struct kbuffer *kbuf, void *ptr)
-{
- return kbuf->read_8(ptr);
-}
-
-static unsigned long long __read_long_4(struct kbuffer *kbuf, void *ptr)
-{
- return kbuf->read_4(ptr);
-}
-
-static unsigned long long read_long(struct kbuffer *kbuf, void *ptr)
-{
- return kbuf->read_long(kbuf, ptr);
-}
-
-static int calc_index(struct kbuffer *kbuf, void *ptr)
-{
- return (unsigned long)ptr - (unsigned long)kbuf->data;
-}
-
-static int __next_event(struct kbuffer *kbuf);
-
-/**
- * kbuffer_alloc - allocat a new kbuffer
- * @size; enum to denote size of word
- * @endian: enum to denote endianness
- *
- * Allocates and returns a new kbuffer.
- */
-struct kbuffer *
-kbuffer_alloc(enum kbuffer_long_size size, enum kbuffer_endian endian)
-{
- struct kbuffer *kbuf;
- int flags = 0;
-
- switch (size) {
- case KBUFFER_LSIZE_4:
- break;
- case KBUFFER_LSIZE_8:
- flags |= KBUFFER_FL_LONG_8;
- break;
- default:
- return NULL;
- }
-
- switch (endian) {
- case KBUFFER_ENDIAN_LITTLE:
- break;
- case KBUFFER_ENDIAN_BIG:
- flags |= KBUFFER_FL_BIG_ENDIAN;
- break;
- default:
- return NULL;
- }
-
- kbuf = zmalloc(sizeof(*kbuf));
- if (!kbuf)
- return NULL;
-
- kbuf->flags = flags;
-
- if (host_is_bigendian())
- kbuf->flags |= KBUFFER_FL_HOST_BIG_ENDIAN;
-
- if (do_swap(kbuf)) {
- kbuf->read_8 = __read_8_sw;
- kbuf->read_4 = __read_4_sw;
- } else {
- kbuf->read_8 = __read_8;
- kbuf->read_4 = __read_4;
- }
-
- if (kbuf->flags & KBUFFER_FL_LONG_8)
- kbuf->read_long = __read_long_8;
- else
- kbuf->read_long = __read_long_4;
-
- /* May be changed by kbuffer_set_old_format() */
- kbuf->next_event = __next_event;
-
- return kbuf;
-}
-
-/** kbuffer_free - free an allocated kbuffer
- * @kbuf: The kbuffer to free
- *
- * Can take NULL as a parameter.
- */
-void kbuffer_free(struct kbuffer *kbuf)
-{
- free(kbuf);
-}
-
-static unsigned int type4host(struct kbuffer *kbuf,
- unsigned int type_len_ts)
-{
- if (kbuf->flags & KBUFFER_FL_BIG_ENDIAN)
- return (type_len_ts >> 29) & 3;
- else
- return type_len_ts & 3;
-}
-
-static unsigned int len4host(struct kbuffer *kbuf,
- unsigned int type_len_ts)
-{
- if (kbuf->flags & KBUFFER_FL_BIG_ENDIAN)
- return (type_len_ts >> 27) & 7;
- else
- return (type_len_ts >> 2) & 7;
-}
-
-static unsigned int type_len4host(struct kbuffer *kbuf,
- unsigned int type_len_ts)
-{
- if (kbuf->flags & KBUFFER_FL_BIG_ENDIAN)
- return (type_len_ts >> 27) & ((1 << 5) - 1);
- else
- return type_len_ts & ((1 << 5) - 1);
-}
-
-static unsigned int ts4host(struct kbuffer *kbuf,
- unsigned int type_len_ts)
-{
- if (kbuf->flags & KBUFFER_FL_BIG_ENDIAN)
- return type_len_ts & ((1 << 27) - 1);
- else
- return type_len_ts >> 5;
-}
-
-/*
- * Linux 2.6.30 and earlier (not much ealier) had a different
- * ring buffer format. It should be obsolete, but we handle it anyway.
- */
-enum old_ring_buffer_type {
- OLD_RINGBUF_TYPE_PADDING,
- OLD_RINGBUF_TYPE_TIME_EXTEND,
- OLD_RINGBUF_TYPE_TIME_STAMP,
- OLD_RINGBUF_TYPE_DATA,
-};
-
-static unsigned int old_update_pointers(struct kbuffer *kbuf)
-{
- unsigned long long extend;
- unsigned int type_len_ts;
- unsigned int type;
- unsigned int len;
- unsigned int delta;
- unsigned int length;
- void *ptr = kbuf->data + kbuf->curr;
-
- type_len_ts = read_4(kbuf, ptr);
- ptr += 4;
-
- type = type4host(kbuf, type_len_ts);
- len = len4host(kbuf, type_len_ts);
- delta = ts4host(kbuf, type_len_ts);
-
- switch (type) {
- case OLD_RINGBUF_TYPE_PADDING:
- kbuf->next = kbuf->size;
- return 0;
-
- case OLD_RINGBUF_TYPE_TIME_EXTEND:
- extend = read_4(kbuf, ptr);
- extend <<= TS_SHIFT;
- extend += delta;
- delta = extend;
- ptr += 4;
- length = 0;
- break;
-
- case OLD_RINGBUF_TYPE_TIME_STAMP:
- /* should never happen! */
- kbuf->curr = kbuf->size;
- kbuf->next = kbuf->size;
- kbuf->index = kbuf->size;
- return -1;
- default:
- if (len)
- length = len * 4;
- else {
- length = read_4(kbuf, ptr);
- length -= 4;
- ptr += 4;
- }
- break;
- }
-
- kbuf->timestamp += delta;
- kbuf->index = calc_index(kbuf, ptr);
- kbuf->next = kbuf->index + length;
-
- return type;
-}
-
-static int __old_next_event(struct kbuffer *kbuf)
-{
- int type;
-
- do {
- kbuf->curr = kbuf->next;
- if (kbuf->next >= kbuf->size)
- return -1;
- type = old_update_pointers(kbuf);
- } while (type == OLD_RINGBUF_TYPE_TIME_EXTEND || type == OLD_RINGBUF_TYPE_PADDING);
-
- return 0;
-}
-
-static unsigned int
-translate_data(struct kbuffer *kbuf, void *data, void **rptr,
- unsigned long long *delta, int *length)
-{
- unsigned long long extend;
- unsigned int type_len_ts;
- unsigned int type_len;
-
- type_len_ts = read_4(kbuf, data);
- data += 4;
-
- type_len = type_len4host(kbuf, type_len_ts);
- *delta = ts4host(kbuf, type_len_ts);
-
- switch (type_len) {
- case KBUFFER_TYPE_PADDING:
- *length = read_4(kbuf, data);
- break;
-
- case KBUFFER_TYPE_TIME_EXTEND:
- case KBUFFER_TYPE_TIME_STAMP:
- extend = read_4(kbuf, data);
- data += 4;
- extend <<= TS_SHIFT;
- extend += *delta;
- *delta = extend;
- *length = 0;
- break;
-
- case 0:
- *length = read_4(kbuf, data) - 4;
- *length = (*length + 3) & ~3;
- data += 4;
- break;
- default:
- *length = type_len * 4;
- break;
- }
-
- *rptr = data;
-
- return type_len;
-}
-
-static unsigned int update_pointers(struct kbuffer *kbuf)
-{
- unsigned long long delta;
- unsigned int type_len;
- int length;
- void *ptr = kbuf->data + kbuf->curr;
-
- type_len = translate_data(kbuf, ptr, &ptr, &delta, &length);
-
- if (type_len == KBUFFER_TYPE_TIME_STAMP)
- kbuf->timestamp = delta;
- else
- kbuf->timestamp += delta;
-
- kbuf->index = calc_index(kbuf, ptr);
- kbuf->next = kbuf->index + length;
-
- return type_len;
-}
-
-/**
- * kbuffer_translate_data - read raw data to get a record
- * @swap: Set to 1 if bytes in words need to be swapped when read
- * @data: The raw data to read
- * @size: Address to store the size of the event data.
- *
- * Returns a pointer to the event data. To determine the entire
- * record size (record metadata + data) just add the difference between
- * @data and the returned value to @size.
- */
-void *kbuffer_translate_data(int swap, void *data, unsigned int *size)
-{
- unsigned long long delta;
- struct kbuffer kbuf;
- int type_len;
- int length;
- void *ptr;
-
- if (swap) {
- kbuf.read_8 = __read_8_sw;
- kbuf.read_4 = __read_4_sw;
- kbuf.flags = host_is_bigendian() ? 0 : KBUFFER_FL_BIG_ENDIAN;
- } else {
- kbuf.read_8 = __read_8;
- kbuf.read_4 = __read_4;
- kbuf.flags = host_is_bigendian() ? KBUFFER_FL_BIG_ENDIAN: 0;
- }
-
- type_len = translate_data(&kbuf, data, &ptr, &delta, &length);
- switch (type_len) {
- case KBUFFER_TYPE_PADDING:
- case KBUFFER_TYPE_TIME_EXTEND:
- case KBUFFER_TYPE_TIME_STAMP:
- return NULL;
- }
-
- *size = length;
-
- return ptr;
-}
-
-static int __next_event(struct kbuffer *kbuf)
-{
- int type;
-
- do {
- kbuf->curr = kbuf->next;
- if (kbuf->next >= kbuf->size)
- return -1;
- type = update_pointers(kbuf);
- } while (type == KBUFFER_TYPE_TIME_EXTEND ||
- type == KBUFFER_TYPE_TIME_STAMP ||
- type == KBUFFER_TYPE_PADDING);
-
- return 0;
-}
-
-static int next_event(struct kbuffer *kbuf)
-{
- return kbuf->next_event(kbuf);
-}
-
-/**
- * kbuffer_next_event - increment the current pointer
- * @kbuf: The kbuffer to read
- * @ts: Address to store the next record's timestamp (may be NULL to ignore)
- *
- * Increments the pointers into the subbuffer of the kbuffer to point to the
- * next event so that the next kbuffer_read_event() will return a
- * new event.
- *
- * Returns the data of the next event if a new event exists on the subbuffer,
- * NULL otherwise.
- */
-void *kbuffer_next_event(struct kbuffer *kbuf, unsigned long long *ts)
-{
- int ret;
-
- if (!kbuf || !kbuf->subbuffer)
- return NULL;
-
- ret = next_event(kbuf);
- if (ret < 0)
- return NULL;
-
- if (ts)
- *ts = kbuf->timestamp;
-
- return kbuf->data + kbuf->index;
-}
-
-/**
- * kbuffer_load_subbuffer - load a new subbuffer into the kbuffer
- * @kbuf: The kbuffer to load
- * @subbuffer: The subbuffer to load into @kbuf.
- *
- * Load a new subbuffer (page) into @kbuf. This will reset all
- * the pointers and update the @kbuf timestamp. The next read will
- * return the first event on @subbuffer.
- *
- * Returns 0 on succes, -1 otherwise.
- */
-int kbuffer_load_subbuffer(struct kbuffer *kbuf, void *subbuffer)
-{
- unsigned long long flags;
- void *ptr = subbuffer;
-
- if (!kbuf || !subbuffer)
- return -1;
-
- kbuf->subbuffer = subbuffer;
-
- kbuf->timestamp = read_8(kbuf, ptr);
- ptr += 8;
-
- kbuf->curr = 0;
-
- if (kbuf->flags & KBUFFER_FL_LONG_8)
- kbuf->start = 16;
- else
- kbuf->start = 12;
-
- kbuf->data = subbuffer + kbuf->start;
-
- flags = read_long(kbuf, ptr);
- kbuf->size = (unsigned int)flags & COMMIT_MASK;
-
- if (flags & MISSING_EVENTS) {
- if (flags & MISSING_STORED) {
- ptr = kbuf->data + kbuf->size;
- kbuf->lost_events = read_long(kbuf, ptr);
- } else
- kbuf->lost_events = -1;
- } else
- kbuf->lost_events = 0;
-
- kbuf->index = 0;
- kbuf->next = 0;
-
- next_event(kbuf);
-
- return 0;
-}
-
-/**
- * kbuffer_subbuf_timestamp - read the timestamp from a sub buffer
- * @kbuf: The kbuffer to load
- * @subbuf: The subbuffer to read from.
- *
- * Return the timestamp from a subbuffer.
- */
-unsigned long long kbuffer_subbuf_timestamp(struct kbuffer *kbuf, void *subbuf)
-{
- return kbuf->read_8(subbuf);
-}
-
-/**
- * kbuffer_ptr_delta - read the delta field from a record
- * @kbuf: The kbuffer to load
- * @ptr: The record in the buffe.
- *
- * Return the timestamp delta from a record
- */
-unsigned int kbuffer_ptr_delta(struct kbuffer *kbuf, void *ptr)
-{
- unsigned int type_len_ts;
-
- type_len_ts = read_4(kbuf, ptr);
- return ts4host(kbuf, type_len_ts);
-}
-
-
-/**
- * kbuffer_read_event - read the next event in the kbuffer subbuffer
- * @kbuf: The kbuffer to read from
- * @ts: The address to store the timestamp of the event (may be NULL to ignore)
- *
- * Returns a pointer to the data part of the current event.
- * NULL if no event is left on the subbuffer.
- */
-void *kbuffer_read_event(struct kbuffer *kbuf, unsigned long long *ts)
-{
- if (!kbuf || !kbuf->subbuffer)
- return NULL;
-
- if (kbuf->curr >= kbuf->size)
- return NULL;
-
- if (ts)
- *ts = kbuf->timestamp;
- return kbuf->data + kbuf->index;
-}
-
-/**
- * kbuffer_timestamp - Return the timestamp of the current event
- * @kbuf: The kbuffer to read from
- *
- * Returns the timestamp of the current (next) event.
- */
-unsigned long long kbuffer_timestamp(struct kbuffer *kbuf)
-{
- return kbuf->timestamp;
-}
-
-/**
- * kbuffer_read_at_offset - read the event that is at offset
- * @kbuf: The kbuffer to read from
- * @offset: The offset into the subbuffer
- * @ts: The address to store the timestamp of the event (may be NULL to ignore)
- *
- * The @offset must be an index from the @kbuf subbuffer beginning.
- * If @offset is bigger than the stored subbuffer, NULL will be returned.
- *
- * Returns the data of the record that is at @offset. Note, @offset does
- * not need to be the start of the record, the offset just needs to be
- * in the record (or beginning of it).
- *
- * Note, the kbuf timestamp and pointers are updated to the
- * returned record. That is, kbuffer_read_event() will return the same
- * data and timestamp, and kbuffer_next_event() will increment from
- * this record.
- */
-void *kbuffer_read_at_offset(struct kbuffer *kbuf, int offset,
- unsigned long long *ts)
-{
- void *data;
-
- if (offset < kbuf->start)
- offset = 0;
- else
- offset -= kbuf->start;
-
- /* Reset the buffer */
- kbuffer_load_subbuffer(kbuf, kbuf->subbuffer);
- data = kbuffer_read_event(kbuf, ts);
-
- while (kbuf->curr < offset) {
- data = kbuffer_next_event(kbuf, ts);
- if (!data)
- break;
- }
-
- return data;
-}
-
-/**
- * kbuffer_subbuffer_size - the size of the loaded subbuffer
- * @kbuf: The kbuffer to read from
- *
- * Returns the size of the subbuffer. Note, this size is
- * where the last event resides. The stored subbuffer may actually be
- * bigger due to padding and such.
- */
-int kbuffer_subbuffer_size(struct kbuffer *kbuf)
-{
- return kbuf->size;
-}
-
-/**
- * kbuffer_curr_index - Return the index of the record
- * @kbuf: The kbuffer to read from
- *
- * Returns the index from the start of the data part of
- * the subbuffer to the current location. Note this is not
- * from the start of the subbuffer. An index of zero will
- * point to the first record. Use kbuffer_curr_offset() for
- * the actually offset (that can be used by kbuffer_read_at_offset())
- */
-int kbuffer_curr_index(struct kbuffer *kbuf)
-{
- return kbuf->curr;
-}
-
-/**
- * kbuffer_curr_offset - Return the offset of the record
- * @kbuf: The kbuffer to read from
- *
- * Returns the offset from the start of the subbuffer to the
- * current location.
- */
-int kbuffer_curr_offset(struct kbuffer *kbuf)
-{
- return kbuf->curr + kbuf->start;
-}
-
-/**
- * kbuffer_event_size - return the size of the event data
- * @kbuf: The kbuffer to read
- *
- * Returns the size of the event data (the payload not counting
- * the meta data of the record) of the current event.
- */
-int kbuffer_event_size(struct kbuffer *kbuf)
-{
- return kbuf->next - kbuf->index;
-}
-
-/**
- * kbuffer_curr_size - return the size of the entire record
- * @kbuf: The kbuffer to read
- *
- * Returns the size of the entire record (meta data and payload)
- * of the current event.
- */
-int kbuffer_curr_size(struct kbuffer *kbuf)
-{
- return kbuf->next - kbuf->curr;
-}
-
-/**
- * kbuffer_missed_events - return the # of missed events from last event.
- * @kbuf: The kbuffer to read from
- *
- * Returns the # of missed events (if recorded) before the current
- * event. Note, only events on the beginning of a subbuffer can
- * have missed events, all other events within the buffer will be
- * zero.
- */
-int kbuffer_missed_events(struct kbuffer *kbuf)
-{
- /* Only the first event can have missed events */
- if (kbuf->curr)
- return 0;
-
- return kbuf->lost_events;
-}
-
-/**
- * kbuffer_set_old_forma - set the kbuffer to use the old format parsing
- * @kbuf: The kbuffer to set
- *
- * This is obsolete (or should be). The first kernels to use the
- * new ring buffer had a slightly different ring buffer format
- * (2.6.30 and earlier). It is still somewhat supported by kbuffer,
- * but should not be counted on in the future.
- */
-void kbuffer_set_old_format(struct kbuffer *kbuf)
-{
- kbuf->flags |= KBUFFER_FL_OLD_FORMAT;
-
- kbuf->next_event = __old_next_event;
-}
-
-/**
- * kbuffer_start_of_data - return offset of where data starts on subbuffer
- * @kbuf: The kbuffer
- *
- * Returns the location on the subbuffer where the data starts.
- */
-int kbuffer_start_of_data(struct kbuffer *kbuf)
-{
- return kbuf->start;
-}
-
-/**
- * kbuffer_raw_get - get raw buffer info
- * @kbuf: The kbuffer
- * @subbuf: Start of mapped subbuffer
- * @info: Info descriptor to fill in
- *
- * For debugging. This can return internals of the ring buffer.
- * Expects to have info->next set to what it will read.
- * The type, length and timestamp delta will be filled in, and
- * @info->next will be updated to the next element.
- * The @subbuf is used to know if the info is passed the end of
- * data and NULL will be returned if it is.
- */
-struct kbuffer_raw_info *
-kbuffer_raw_get(struct kbuffer *kbuf, void *subbuf, struct kbuffer_raw_info *info)
-{
- unsigned long long flags;
- unsigned long long delta;
- unsigned int type_len;
- unsigned int size;
- int start;
- int length;
- void *ptr = info->next;
-
- if (!kbuf || !subbuf)
- return NULL;
-
- if (kbuf->flags & KBUFFER_FL_LONG_8)
- start = 16;
- else
- start = 12;
-
- flags = read_long(kbuf, subbuf + 8);
- size = (unsigned int)flags & COMMIT_MASK;
-
- if (ptr < subbuf || ptr >= subbuf + start + size)
- return NULL;
-
- type_len = translate_data(kbuf, ptr, &ptr, &delta, &length);
-
- info->next = ptr + length;
-
- info->type = type_len;
- info->delta = delta;
- info->length = length;
-
- return info;
-}
diff --git a/tools/lib/traceevent/kbuffer.h b/tools/lib/traceevent/kbuffer.h
deleted file mode 100644
index a2b5220..0000000
--- a/tools/lib/traceevent/kbuffer.h
+++ /dev/null
@@ -1,68 +0,0 @@
-/* SPDX-License-Identifier: LGPL-2.1 */
-/*
- * Copyright (C) 2012 Red Hat Inc, Steven Rostedt <srostedt@redhat.com>
- *
- */
-#ifndef _KBUFFER_H
-#define _KBUFFER_H
-
-#ifndef TS_SHIFT
-#define TS_SHIFT 27
-#endif
-
-enum kbuffer_endian {
- KBUFFER_ENDIAN_BIG,
- KBUFFER_ENDIAN_LITTLE,
-};
-
-enum kbuffer_long_size {
- KBUFFER_LSIZE_4,
- KBUFFER_LSIZE_8,
-};
-
-enum {
- KBUFFER_TYPE_PADDING = 29,
- KBUFFER_TYPE_TIME_EXTEND = 30,
- KBUFFER_TYPE_TIME_STAMP = 31,
-};
-
-struct kbuffer;
-
-struct kbuffer *kbuffer_alloc(enum kbuffer_long_size size, enum kbuffer_endian endian);
-void kbuffer_free(struct kbuffer *kbuf);
-int kbuffer_load_subbuffer(struct kbuffer *kbuf, void *subbuffer);
-void *kbuffer_read_event(struct kbuffer *kbuf, unsigned long long *ts);
-void *kbuffer_next_event(struct kbuffer *kbuf, unsigned long long *ts);
-unsigned long long kbuffer_timestamp(struct kbuffer *kbuf);
-unsigned long long kbuffer_subbuf_timestamp(struct kbuffer *kbuf, void *subbuf);
-unsigned int kbuffer_ptr_delta(struct kbuffer *kbuf, void *ptr);
-
-void *kbuffer_translate_data(int swap, void *data, unsigned int *size);
-
-void *kbuffer_read_at_offset(struct kbuffer *kbuf, int offset, unsigned long long *ts);
-
-int kbuffer_curr_index(struct kbuffer *kbuf);
-
-int kbuffer_curr_offset(struct kbuffer *kbuf);
-int kbuffer_curr_size(struct kbuffer *kbuf);
-int kbuffer_event_size(struct kbuffer *kbuf);
-int kbuffer_missed_events(struct kbuffer *kbuf);
-int kbuffer_subbuffer_size(struct kbuffer *kbuf);
-
-void kbuffer_set_old_format(struct kbuffer *kbuf);
-int kbuffer_start_of_data(struct kbuffer *kbuf);
-
-/* Debugging */
-
-struct kbuffer_raw_info {
- int type;
- int length;
- unsigned long long delta;
- void *next;
-};
-
-/* Read raw data */
-struct kbuffer_raw_info *kbuffer_raw_get(struct kbuffer *kbuf, void *subbuf,
- struct kbuffer_raw_info *info);
-
-#endif /* _K_BUFFER_H */
diff --git a/tools/lib/traceevent/libtraceevent.pc.template b/tools/lib/traceevent/libtraceevent.pc.template
deleted file mode 100644
index 86384fc..0000000
--- a/tools/lib/traceevent/libtraceevent.pc.template
+++ /dev/null
@@ -1,10 +0,0 @@
-prefix=INSTALL_PREFIX
-libdir=LIB_DIR
-includedir=HEADER_DIR
-
-Name: libtraceevent
-URL: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
-Description: Linux kernel trace event library
-Version: LIB_VERSION
-Cflags: -I${includedir}
-Libs: -L${libdir} -ltraceevent
diff --git a/tools/lib/traceevent/parse-filter.c b/tools/lib/traceevent/parse-filter.c
deleted file mode 100644
index 5df1770..0000000
--- a/tools/lib/traceevent/parse-filter.c
+++ /dev/null
@@ -1,2281 +0,0 @@
-// SPDX-License-Identifier: LGPL-2.1
-/*
- * Copyright (C) 2010 Red Hat Inc, Steven Rostedt <srostedt@redhat.com>
- *
- */
-#include <stdio.h>
-#include <stdlib.h>
-#include <string.h>
-#include <stdarg.h>
-#include <errno.h>
-#include <sys/types.h>
-
-#include "event-parse.h"
-#include "event-parse-local.h"
-#include "event-utils.h"
-
-#define COMM "COMM"
-#define CPU "CPU"
-
-static struct tep_format_field comm = {
- .name = "COMM",
-};
-
-static struct tep_format_field cpu = {
- .name = "CPU",
-};
-
-struct event_list {
- struct event_list *next;
- struct tep_event *event;
-};
-
-static void show_error(char *error_buf, const char *fmt, ...)
-{
- unsigned long long index;
- const char *input;
- va_list ap;
- int len;
- int i;
-
- input = get_input_buf();
- index = get_input_buf_ptr();
- len = input ? strlen(input) : 0;
-
- if (len) {
- strcpy(error_buf, input);
- error_buf[len] = '\n';
- for (i = 1; i < len && i < index; i++)
- error_buf[len+i] = ' ';
- error_buf[len + i] = '^';
- error_buf[len + i + 1] = '\n';
- len += i+2;
- }
-
- va_start(ap, fmt);
- vsnprintf(error_buf + len, TEP_FILTER_ERROR_BUFSZ - len, fmt, ap);
- va_end(ap);
-}
-
-static enum tep_event_type filter_read_token(char **tok)
-{
- enum tep_event_type type;
- char *token = NULL;
-
- do {
- free_token(token);
- type = read_token(&token);
- } while (type == TEP_EVENT_NEWLINE || type == TEP_EVENT_SPACE);
-
- /* If token is = or ! check to see if the next char is ~ */
- if (token &&
- (strcmp(token, "=") == 0 || strcmp(token, "!") == 0) &&
- peek_char() == '~') {
- /* append it */
- *tok = malloc(3);
- if (*tok == NULL) {
- free_token(token);
- return TEP_EVENT_ERROR;
- }
- sprintf(*tok, "%c%c", *token, '~');
- free_token(token);
- /* Now remove the '~' from the buffer */
- read_token(&token);
- free_token(token);
- } else
- *tok = token;
-
- return type;
-}
-
-static int filter_cmp(const void *a, const void *b)
-{
- const struct tep_filter_type *ea = a;
- const struct tep_filter_type *eb = b;
-
- if (ea->event_id < eb->event_id)
- return -1;
-
- if (ea->event_id > eb->event_id)
- return 1;
-
- return 0;
-}
-
-static struct tep_filter_type *
-find_filter_type(struct tep_event_filter *filter, int id)
-{
- struct tep_filter_type *filter_type;
- struct tep_filter_type key;
-
- key.event_id = id;
-
- filter_type = bsearch(&key, filter->event_filters,
- filter->filters,
- sizeof(*filter->event_filters),
- filter_cmp);
-
- return filter_type;
-}
-
-static struct tep_filter_type *
-add_filter_type(struct tep_event_filter *filter, int id)
-{
- struct tep_filter_type *filter_type;
- int i;
-
- filter_type = find_filter_type(filter, id);
- if (filter_type)
- return filter_type;
-
- filter_type = realloc(filter->event_filters,
- sizeof(*filter->event_filters) *
- (filter->filters + 1));
- if (!filter_type)
- return NULL;
-
- filter->event_filters = filter_type;
-
- for (i = 0; i < filter->filters; i++) {
- if (filter->event_filters[i].event_id > id)
- break;
- }
-
- if (i < filter->filters)
- memmove(&filter->event_filters[i+1],
- &filter->event_filters[i],
- sizeof(*filter->event_filters) *
- (filter->filters - i));
-
- filter_type = &filter->event_filters[i];
- filter_type->event_id = id;
- filter_type->event = tep_find_event(filter->tep, id);
- filter_type->filter = NULL;
-
- filter->filters++;
-
- return filter_type;
-}
-
-/**
- * tep_filter_alloc - create a new event filter
- * @tep: The tep that this filter is associated with
- */
-struct tep_event_filter *tep_filter_alloc(struct tep_handle *tep)
-{
- struct tep_event_filter *filter;
-
- filter = malloc(sizeof(*filter));
- if (filter == NULL)
- return NULL;
-
- memset(filter, 0, sizeof(*filter));
- filter->tep = tep;
- tep_ref(tep);
-
- return filter;
-}
-
-static struct tep_filter_arg *allocate_arg(void)
-{
- return calloc(1, sizeof(struct tep_filter_arg));
-}
-
-static void free_arg(struct tep_filter_arg *arg)
-{
- if (!arg)
- return;
-
- switch (arg->type) {
- case TEP_FILTER_ARG_NONE:
- case TEP_FILTER_ARG_BOOLEAN:
- break;
-
- case TEP_FILTER_ARG_NUM:
- free_arg(arg->num.left);
- free_arg(arg->num.right);
- break;
-
- case TEP_FILTER_ARG_EXP:
- free_arg(arg->exp.left);
- free_arg(arg->exp.right);
- break;
-
- case TEP_FILTER_ARG_STR:
- free(arg->str.val);
- regfree(&arg->str.reg);
- free(arg->str.buffer);
- break;
-
- case TEP_FILTER_ARG_VALUE:
- if (arg->value.type == TEP_FILTER_STRING ||
- arg->value.type == TEP_FILTER_CHAR)
- free(arg->value.str);
- break;
-
- case TEP_FILTER_ARG_OP:
- free_arg(arg->op.left);
- free_arg(arg->op.right);
- default:
- break;
- }
-
- free(arg);
-}
-
-static int add_event(struct event_list **events,
- struct tep_event *event)
-{
- struct event_list *list;
-
- list = malloc(sizeof(*list));
- if (list == NULL)
- return -1;
-
- list->next = *events;
- *events = list;
- list->event = event;
- return 0;
-}
-
-static int event_match(struct tep_event *event,
- regex_t *sreg, regex_t *ereg)
-{
- if (sreg) {
- return !regexec(sreg, event->system, 0, NULL, 0) &&
- !regexec(ereg, event->name, 0, NULL, 0);
- }
-
- return !regexec(ereg, event->system, 0, NULL, 0) ||
- !regexec(ereg, event->name, 0, NULL, 0);
-}
-
-static enum tep_errno
-find_event(struct tep_handle *tep, struct event_list **events,
- char *sys_name, char *event_name)
-{
- struct tep_event *event;
- regex_t ereg;
- regex_t sreg;
- int match = 0;
- int fail = 0;
- char *reg;
- int ret;
- int i;
-
- if (!event_name) {
- /* if no name is given, then swap sys and name */
- event_name = sys_name;
- sys_name = NULL;
- }
-
- ret = asprintf(®, "^%s$", event_name);
- if (ret < 0)
- return TEP_ERRNO__MEM_ALLOC_FAILED;
-
- ret = regcomp(&ereg, reg, REG_ICASE|REG_NOSUB);
- free(reg);
-
- if (ret)
- return TEP_ERRNO__INVALID_EVENT_NAME;
-
- if (sys_name) {
- ret = asprintf(®, "^%s$", sys_name);
- if (ret < 0) {
- regfree(&ereg);
- return TEP_ERRNO__MEM_ALLOC_FAILED;
- }
-
- ret = regcomp(&sreg, reg, REG_ICASE|REG_NOSUB);
- free(reg);
- if (ret) {
- regfree(&ereg);
- return TEP_ERRNO__INVALID_EVENT_NAME;
- }
- }
-
- for (i = 0; i < tep->nr_events; i++) {
- event = tep->events[i];
- if (event_match(event, sys_name ? &sreg : NULL, &ereg)) {
- match = 1;
- if (add_event(events, event) < 0) {
- fail = 1;
- break;
- }
- }
- }
-
- regfree(&ereg);
- if (sys_name)
- regfree(&sreg);
-
- if (!match)
- return TEP_ERRNO__EVENT_NOT_FOUND;
- if (fail)
- return TEP_ERRNO__MEM_ALLOC_FAILED;
-
- return 0;
-}
-
-static void free_events(struct event_list *events)
-{
- struct event_list *event;
-
- while (events) {
- event = events;
- events = events->next;
- free(event);
- }
-}
-
-static enum tep_errno
-create_arg_item(struct tep_event *event, const char *token,
- enum tep_event_type type, struct tep_filter_arg **parg, char *error_str)
-{
- struct tep_format_field *field;
- struct tep_filter_arg *arg;
-
- arg = allocate_arg();
- if (arg == NULL) {
- show_error(error_str, "failed to allocate filter arg");
- return TEP_ERRNO__MEM_ALLOC_FAILED;
- }
-
- switch (type) {
-
- case TEP_EVENT_SQUOTE:
- case TEP_EVENT_DQUOTE:
- arg->type = TEP_FILTER_ARG_VALUE;
- arg->value.type =
- type == TEP_EVENT_DQUOTE ? TEP_FILTER_STRING : TEP_FILTER_CHAR;
- arg->value.str = strdup(token);
- if (!arg->value.str) {
- free_arg(arg);
- show_error(error_str, "failed to allocate string filter arg");
- return TEP_ERRNO__MEM_ALLOC_FAILED;
- }
- break;
- case TEP_EVENT_ITEM:
- /* if it is a number, then convert it */
- if (isdigit(token[0])) {
- arg->type = TEP_FILTER_ARG_VALUE;
- arg->value.type = TEP_FILTER_NUMBER;
- arg->value.val = strtoull(token, NULL, 0);
- break;
- }
- /* Consider this a field */
- field = tep_find_any_field(event, token);
- if (!field) {
- /* If token is 'COMM' or 'CPU' then it is special */
- if (strcmp(token, COMM) == 0) {
- field = &comm;
- } else if (strcmp(token, CPU) == 0) {
- field = &cpu;
- } else {
- /* not a field, Make it false */
- arg->type = TEP_FILTER_ARG_BOOLEAN;
- arg->boolean.value = TEP_FILTER_FALSE;
- break;
- }
- }
- arg->type = TEP_FILTER_ARG_FIELD;
- arg->field.field = field;
- break;
- default:
- free_arg(arg);
- show_error(error_str, "expected a value but found %s", token);
- return TEP_ERRNO__UNEXPECTED_TYPE;
- }
- *parg = arg;
- return 0;
-}
-
-static struct tep_filter_arg *
-create_arg_op(enum tep_filter_op_type btype)
-{
- struct tep_filter_arg *arg;
-
- arg = allocate_arg();
- if (!arg)
- return NULL;
-
- arg->type = TEP_FILTER_ARG_OP;
- arg->op.type = btype;
-
- return arg;
-}
-
-static struct tep_filter_arg *
-create_arg_exp(enum tep_filter_exp_type etype)
-{
- struct tep_filter_arg *arg;
-
- arg = allocate_arg();
- if (!arg)
- return NULL;
-
- arg->type = TEP_FILTER_ARG_EXP;
- arg->exp.type = etype;
-
- return arg;
-}
-
-static struct tep_filter_arg *
-create_arg_cmp(enum tep_filter_cmp_type ctype)
-{
- struct tep_filter_arg *arg;
-
- arg = allocate_arg();
- if (!arg)
- return NULL;
-
- /* Use NUM and change if necessary */
- arg->type = TEP_FILTER_ARG_NUM;
- arg->num.type = ctype;
-
- return arg;
-}
-
-static enum tep_errno
-add_right(struct tep_filter_arg *op, struct tep_filter_arg *arg, char *error_str)
-{
- struct tep_filter_arg *left;
- char *str;
- int op_type;
- int ret;
-
- switch (op->type) {
- case TEP_FILTER_ARG_EXP:
- if (op->exp.right)
- goto out_fail;
- op->exp.right = arg;
- break;
-
- case TEP_FILTER_ARG_OP:
- if (op->op.right)
- goto out_fail;
- op->op.right = arg;
- break;
-
- case TEP_FILTER_ARG_NUM:
- if (op->op.right)
- goto out_fail;
- /*
- * The arg must be num, str, or field
- */
- switch (arg->type) {
- case TEP_FILTER_ARG_VALUE:
- case TEP_FILTER_ARG_FIELD:
- break;
- default:
- show_error(error_str, "Illegal rvalue");
- return TEP_ERRNO__ILLEGAL_RVALUE;
- }
-
- /*
- * Depending on the type, we may need to
- * convert this to a string or regex.
- */
- switch (arg->value.type) {
- case TEP_FILTER_CHAR:
- /*
- * A char should be converted to number if
- * the string is 1 byte, and the compare
- * is not a REGEX.
- */
- if (strlen(arg->value.str) == 1 &&
- op->num.type != TEP_FILTER_CMP_REGEX &&
- op->num.type != TEP_FILTER_CMP_NOT_REGEX) {
- arg->value.type = TEP_FILTER_NUMBER;
- goto do_int;
- }
- /* fall through */
- case TEP_FILTER_STRING:
-
- /* convert op to a string arg */
- op_type = op->num.type;
- left = op->num.left;
- str = arg->value.str;
-
- /* reset the op for the new field */
- memset(op, 0, sizeof(*op));
-
- /*
- * If left arg was a field not found then
- * NULL the entire op.
- */
- if (left->type == TEP_FILTER_ARG_BOOLEAN) {
- free_arg(left);
- free_arg(arg);
- op->type = TEP_FILTER_ARG_BOOLEAN;
- op->boolean.value = TEP_FILTER_FALSE;
- break;
- }
-
- /* Left arg must be a field */
- if (left->type != TEP_FILTER_ARG_FIELD) {
- show_error(error_str,
- "Illegal lvalue for string comparison");
- return TEP_ERRNO__ILLEGAL_LVALUE;
- }
-
- /* Make sure this is a valid string compare */
- switch (op_type) {
- case TEP_FILTER_CMP_EQ:
- op_type = TEP_FILTER_CMP_MATCH;
- break;
- case TEP_FILTER_CMP_NE:
- op_type = TEP_FILTER_CMP_NOT_MATCH;
- break;
-
- case TEP_FILTER_CMP_REGEX:
- case TEP_FILTER_CMP_NOT_REGEX:
- ret = regcomp(&op->str.reg, str, REG_ICASE|REG_NOSUB);
- if (ret) {
- show_error(error_str,
- "RegEx '%s' did not compute",
- str);
- return TEP_ERRNO__INVALID_REGEX;
- }
- break;
- default:
- show_error(error_str,
- "Illegal comparison for string");
- return TEP_ERRNO__ILLEGAL_STRING_CMP;
- }
-
- op->type = TEP_FILTER_ARG_STR;
- op->str.type = op_type;
- op->str.field = left->field.field;
- op->str.val = strdup(str);
- if (!op->str.val) {
- show_error(error_str, "Failed to allocate string filter");
- return TEP_ERRNO__MEM_ALLOC_FAILED;
- }
- /*
- * Need a buffer to copy data for tests
- */
- op->str.buffer = malloc(op->str.field->size + 1);
- if (!op->str.buffer) {
- show_error(error_str, "Failed to allocate string filter");
- return TEP_ERRNO__MEM_ALLOC_FAILED;
- }
- /* Null terminate this buffer */
- op->str.buffer[op->str.field->size] = 0;
-
- /* We no longer have left or right args */
- free_arg(arg);
- free_arg(left);
-
- break;
-
- case TEP_FILTER_NUMBER:
-
- do_int:
- switch (op->num.type) {
- case TEP_FILTER_CMP_REGEX:
- case TEP_FILTER_CMP_NOT_REGEX:
- show_error(error_str,
- "Op not allowed with integers");
- return TEP_ERRNO__ILLEGAL_INTEGER_CMP;
-
- default:
- break;
- }
-
- /* numeric compare */
- op->num.right = arg;
- break;
- default:
- goto out_fail;
- }
- break;
- default:
- goto out_fail;
- }
-
- return 0;
-
- out_fail:
- show_error(error_str, "Syntax error");
- return TEP_ERRNO__SYNTAX_ERROR;
-}
-
-static struct tep_filter_arg *
-rotate_op_right(struct tep_filter_arg *a, struct tep_filter_arg *b)
-{
- struct tep_filter_arg *arg;
-
- arg = a->op.right;
- a->op.right = b;
- return arg;
-}
-
-static enum tep_errno add_left(struct tep_filter_arg *op, struct tep_filter_arg *arg)
-{
- switch (op->type) {
- case TEP_FILTER_ARG_EXP:
- if (arg->type == TEP_FILTER_ARG_OP)
- arg = rotate_op_right(arg, op);
- op->exp.left = arg;
- break;
-
- case TEP_FILTER_ARG_OP:
- op->op.left = arg;
- break;
- case TEP_FILTER_ARG_NUM:
- if (arg->type == TEP_FILTER_ARG_OP)
- arg = rotate_op_right(arg, op);
-
- /* left arg of compares must be a field */
- if (arg->type != TEP_FILTER_ARG_FIELD &&
- arg->type != TEP_FILTER_ARG_BOOLEAN)
- return TEP_ERRNO__INVALID_ARG_TYPE;
- op->num.left = arg;
- break;
- default:
- return TEP_ERRNO__INVALID_ARG_TYPE;
- }
- return 0;
-}
-
-enum op_type {
- OP_NONE,
- OP_BOOL,
- OP_NOT,
- OP_EXP,
- OP_CMP,
-};
-
-static enum op_type process_op(const char *token,
- enum tep_filter_op_type *btype,
- enum tep_filter_cmp_type *ctype,
- enum tep_filter_exp_type *etype)
-{
- *btype = TEP_FILTER_OP_NOT;
- *etype = TEP_FILTER_EXP_NONE;
- *ctype = TEP_FILTER_CMP_NONE;
-
- if (strcmp(token, "&&") == 0)
- *btype = TEP_FILTER_OP_AND;
- else if (strcmp(token, "||") == 0)
- *btype = TEP_FILTER_OP_OR;
- else if (strcmp(token, "!") == 0)
- return OP_NOT;
-
- if (*btype != TEP_FILTER_OP_NOT)
- return OP_BOOL;
-
- /* Check for value expressions */
- if (strcmp(token, "+") == 0) {
- *etype = TEP_FILTER_EXP_ADD;
- } else if (strcmp(token, "-") == 0) {
- *etype = TEP_FILTER_EXP_SUB;
- } else if (strcmp(token, "*") == 0) {
- *etype = TEP_FILTER_EXP_MUL;
- } else if (strcmp(token, "/") == 0) {
- *etype = TEP_FILTER_EXP_DIV;
- } else if (strcmp(token, "%") == 0) {
- *etype = TEP_FILTER_EXP_MOD;
- } else if (strcmp(token, ">>") == 0) {
- *etype = TEP_FILTER_EXP_RSHIFT;
- } else if (strcmp(token, "<<") == 0) {
- *etype = TEP_FILTER_EXP_LSHIFT;
- } else if (strcmp(token, "&") == 0) {
- *etype = TEP_FILTER_EXP_AND;
- } else if (strcmp(token, "|") == 0) {
- *etype = TEP_FILTER_EXP_OR;
- } else if (strcmp(token, "^") == 0) {
- *etype = TEP_FILTER_EXP_XOR;
- } else if (strcmp(token, "~") == 0)
- *etype = TEP_FILTER_EXP_NOT;
-
- if (*etype != TEP_FILTER_EXP_NONE)
- return OP_EXP;
-
- /* Check for compares */
- if (strcmp(token, "==") == 0)
- *ctype = TEP_FILTER_CMP_EQ;
- else if (strcmp(token, "!=") == 0)
- *ctype = TEP_FILTER_CMP_NE;
- else if (strcmp(token, "<") == 0)
- *ctype = TEP_FILTER_CMP_LT;
- else if (strcmp(token, ">") == 0)
- *ctype = TEP_FILTER_CMP_GT;
- else if (strcmp(token, "<=") == 0)
- *ctype = TEP_FILTER_CMP_LE;
- else if (strcmp(token, ">=") == 0)
- *ctype = TEP_FILTER_CMP_GE;
- else if (strcmp(token, "=~") == 0)
- *ctype = TEP_FILTER_CMP_REGEX;
- else if (strcmp(token, "!~") == 0)
- *ctype = TEP_FILTER_CMP_NOT_REGEX;
- else
- return OP_NONE;
-
- return OP_CMP;
-}
-
-static int check_op_done(struct tep_filter_arg *arg)
-{
- switch (arg->type) {
- case TEP_FILTER_ARG_EXP:
- return arg->exp.right != NULL;
-
- case TEP_FILTER_ARG_OP:
- return arg->op.right != NULL;
-
- case TEP_FILTER_ARG_NUM:
- return arg->num.right != NULL;
-
- case TEP_FILTER_ARG_STR:
- /* A string conversion is always done */
- return 1;
-
- case TEP_FILTER_ARG_BOOLEAN:
- /* field not found, is ok */
- return 1;
-
- default:
- return 0;
- }
-}
-
-enum filter_vals {
- FILTER_VAL_NORM,
- FILTER_VAL_FALSE,
- FILTER_VAL_TRUE,
-};
-
-static enum tep_errno
-reparent_op_arg(struct tep_filter_arg *parent, struct tep_filter_arg *old_child,
- struct tep_filter_arg *arg, char *error_str)
-{
- struct tep_filter_arg *other_child;
- struct tep_filter_arg **ptr;
-
- if (parent->type != TEP_FILTER_ARG_OP &&
- arg->type != TEP_FILTER_ARG_OP) {
- show_error(error_str, "can not reparent other than OP");
- return TEP_ERRNO__REPARENT_NOT_OP;
- }
-
- /* Get the sibling */
- if (old_child->op.right == arg) {
- ptr = &old_child->op.right;
- other_child = old_child->op.left;
- } else if (old_child->op.left == arg) {
- ptr = &old_child->op.left;
- other_child = old_child->op.right;
- } else {
- show_error(error_str, "Error in reparent op, find other child");
- return TEP_ERRNO__REPARENT_FAILED;
- }
-
- /* Detach arg from old_child */
- *ptr = NULL;
-
- /* Check for root */
- if (parent == old_child) {
- free_arg(other_child);
- *parent = *arg;
- /* Free arg without recussion */
- free(arg);
- return 0;
- }
-
- if (parent->op.right == old_child)
- ptr = &parent->op.right;
- else if (parent->op.left == old_child)
- ptr = &parent->op.left;
- else {
- show_error(error_str, "Error in reparent op");
- return TEP_ERRNO__REPARENT_FAILED;
- }
-
- *ptr = arg;
-
- free_arg(old_child);
- return 0;
-}
-
-/* Returns either filter_vals (success) or tep_errno (failfure) */
-static int test_arg(struct tep_filter_arg *parent, struct tep_filter_arg *arg,
- char *error_str)
-{
- int lval, rval;
-
- switch (arg->type) {
-
- /* bad case */
- case TEP_FILTER_ARG_BOOLEAN:
- return FILTER_VAL_FALSE + arg->boolean.value;
-
- /* good cases: */
- case TEP_FILTER_ARG_STR:
- case TEP_FILTER_ARG_VALUE:
- case TEP_FILTER_ARG_FIELD:
- return FILTER_VAL_NORM;
-
- case TEP_FILTER_ARG_EXP:
- lval = test_arg(arg, arg->exp.left, error_str);
- if (lval != FILTER_VAL_NORM)
- return lval;
- rval = test_arg(arg, arg->exp.right, error_str);
- if (rval != FILTER_VAL_NORM)
- return rval;
- return FILTER_VAL_NORM;
-
- case TEP_FILTER_ARG_NUM:
- lval = test_arg(arg, arg->num.left, error_str);
- if (lval != FILTER_VAL_NORM)
- return lval;
- rval = test_arg(arg, arg->num.right, error_str);
- if (rval != FILTER_VAL_NORM)
- return rval;
- return FILTER_VAL_NORM;
-
- case TEP_FILTER_ARG_OP:
- if (arg->op.type != TEP_FILTER_OP_NOT) {
- lval = test_arg(arg, arg->op.left, error_str);
- switch (lval) {
- case FILTER_VAL_NORM:
- break;
- case FILTER_VAL_TRUE:
- if (arg->op.type == TEP_FILTER_OP_OR)
- return FILTER_VAL_TRUE;
- rval = test_arg(arg, arg->op.right, error_str);
- if (rval != FILTER_VAL_NORM)
- return rval;
-
- return reparent_op_arg(parent, arg, arg->op.right,
- error_str);
-
- case FILTER_VAL_FALSE:
- if (arg->op.type == TEP_FILTER_OP_AND)
- return FILTER_VAL_FALSE;
- rval = test_arg(arg, arg->op.right, error_str);
- if (rval != FILTER_VAL_NORM)
- return rval;
-
- return reparent_op_arg(parent, arg, arg->op.right,
- error_str);
-
- default:
- return lval;
- }
- }
-
- rval = test_arg(arg, arg->op.right, error_str);
- switch (rval) {
- case FILTER_VAL_NORM:
- default:
- break;
-
- case FILTER_VAL_TRUE:
- if (arg->op.type == TEP_FILTER_OP_OR)
- return FILTER_VAL_TRUE;
- if (arg->op.type == TEP_FILTER_OP_NOT)
- return FILTER_VAL_FALSE;
-
- return reparent_op_arg(parent, arg, arg->op.left,
- error_str);
-
- case FILTER_VAL_FALSE:
- if (arg->op.type == TEP_FILTER_OP_AND)
- return FILTER_VAL_FALSE;
- if (arg->op.type == TEP_FILTER_OP_NOT)
- return FILTER_VAL_TRUE;
-
- return reparent_op_arg(parent, arg, arg->op.left,
- error_str);
- }
-
- return rval;
- default:
- show_error(error_str, "bad arg in filter tree");
- return TEP_ERRNO__BAD_FILTER_ARG;
- }
- return FILTER_VAL_NORM;
-}
-
-/* Remove any unknown event fields */
-static int collapse_tree(struct tep_filter_arg *arg,
- struct tep_filter_arg **arg_collapsed, char *error_str)
-{
- int ret;
-
- ret = test_arg(arg, arg, error_str);
- switch (ret) {
- case FILTER_VAL_NORM:
- break;
-
- case FILTER_VAL_TRUE:
- case FILTER_VAL_FALSE:
- free_arg(arg);
- arg = allocate_arg();
- if (arg) {
- arg->type = TEP_FILTER_ARG_BOOLEAN;
- arg->boolean.value = ret == FILTER_VAL_TRUE;
- } else {
- show_error(error_str, "Failed to allocate filter arg");
- ret = TEP_ERRNO__MEM_ALLOC_FAILED;
- }
- break;
-
- default:
- /* test_arg() already set the error_str */
- free_arg(arg);
- arg = NULL;
- break;
- }
-
- *arg_collapsed = arg;
- return ret;
-}
-
-static enum tep_errno
-process_filter(struct tep_event *event, struct tep_filter_arg **parg,
- char *error_str, int not)
-{
- enum tep_event_type type;
- char *token = NULL;
- struct tep_filter_arg *current_op = NULL;
- struct tep_filter_arg *current_exp = NULL;
- struct tep_filter_arg *left_item = NULL;
- struct tep_filter_arg *arg = NULL;
- enum op_type op_type;
- enum tep_filter_op_type btype;
- enum tep_filter_exp_type etype;
- enum tep_filter_cmp_type ctype;
- enum tep_errno ret;
-
- *parg = NULL;
-
- do {
- free(token);
- type = filter_read_token(&token);
- switch (type) {
- case TEP_EVENT_SQUOTE:
- case TEP_EVENT_DQUOTE:
- case TEP_EVENT_ITEM:
- ret = create_arg_item(event, token, type, &arg, error_str);
- if (ret < 0)
- goto fail;
- if (!left_item)
- left_item = arg;
- else if (current_exp) {
- ret = add_right(current_exp, arg, error_str);
- if (ret < 0)
- goto fail;
- left_item = NULL;
- /* Not's only one one expression */
- if (not) {
- arg = NULL;
- if (current_op)
- goto fail_syntax;
- free(token);
- *parg = current_exp;
- return 0;
- }
- } else
- goto fail_syntax;
- arg = NULL;
- break;
-
- case TEP_EVENT_DELIM:
- if (*token == ',') {
- show_error(error_str, "Illegal token ','");
- ret = TEP_ERRNO__ILLEGAL_TOKEN;
- goto fail;
- }
-
- if (*token == '(') {
- if (left_item) {
- show_error(error_str,
- "Open paren can not come after item");
- ret = TEP_ERRNO__INVALID_PAREN;
- goto fail;
- }
- if (current_exp) {
- show_error(error_str,
- "Open paren can not come after expression");
- ret = TEP_ERRNO__INVALID_PAREN;
- goto fail;
- }
-
- ret = process_filter(event, &arg, error_str, 0);
- if (ret != TEP_ERRNO__UNBALANCED_PAREN) {
- if (ret == 0) {
- show_error(error_str,
- "Unbalanced number of '('");
- ret = TEP_ERRNO__UNBALANCED_PAREN;
- }
- goto fail;
- }
- ret = 0;
-
- /* A not wants just one expression */
- if (not) {
- if (current_op)
- goto fail_syntax;
- *parg = arg;
- return 0;
- }
-
- if (current_op)
- ret = add_right(current_op, arg, error_str);
- else
- current_exp = arg;
-
- if (ret < 0)
- goto fail;
-
- } else { /* ')' */
- if (!current_op && !current_exp)
- goto fail_syntax;
-
- /* Make sure everything is finished at this level */
- if (current_exp && !check_op_done(current_exp))
- goto fail_syntax;
- if (current_op && !check_op_done(current_op))
- goto fail_syntax;
-
- if (current_op)
- *parg = current_op;
- else
- *parg = current_exp;
- free(token);
- return TEP_ERRNO__UNBALANCED_PAREN;
- }
- break;
-
- case TEP_EVENT_OP:
- op_type = process_op(token, &btype, &ctype, &etype);
-
- /* All expect a left arg except for NOT */
- switch (op_type) {
- case OP_BOOL:
- /* Logic ops need a left expression */
- if (!current_exp && !current_op)
- goto fail_syntax;
- /* fall through */
- case OP_NOT:
- /* logic only processes ops and exp */
- if (left_item)
- goto fail_syntax;
- break;
- case OP_EXP:
- case OP_CMP:
- if (!left_item)
- goto fail_syntax;
- break;
- case OP_NONE:
- show_error(error_str,
- "Unknown op token %s", token);
- ret = TEP_ERRNO__UNKNOWN_TOKEN;
- goto fail;
- }
-
- ret = 0;
- switch (op_type) {
- case OP_BOOL:
- arg = create_arg_op(btype);
- if (arg == NULL)
- goto fail_alloc;
- if (current_op)
- ret = add_left(arg, current_op);
- else
- ret = add_left(arg, current_exp);
- current_op = arg;
- current_exp = NULL;
- break;
-
- case OP_NOT:
- arg = create_arg_op(btype);
- if (arg == NULL)
- goto fail_alloc;
- if (current_op)
- ret = add_right(current_op, arg, error_str);
- if (ret < 0)
- goto fail;
- current_exp = arg;
- ret = process_filter(event, &arg, error_str, 1);
- if (ret < 0)
- goto fail;
- ret = add_right(current_exp, arg, error_str);
- if (ret < 0)
- goto fail;
- break;
-
- case OP_EXP:
- case OP_CMP:
- if (op_type == OP_EXP)
- arg = create_arg_exp(etype);
- else
- arg = create_arg_cmp(ctype);
- if (arg == NULL)
- goto fail_alloc;
-
- if (current_op)
- ret = add_right(current_op, arg, error_str);
- if (ret < 0)
- goto fail;
- ret = add_left(arg, left_item);
- if (ret < 0) {
- arg = NULL;
- goto fail_syntax;
- }
- current_exp = arg;
- break;
- default:
- break;
- }
- arg = NULL;
- if (ret < 0)
- goto fail_syntax;
- break;
- case TEP_EVENT_NONE:
- break;
- case TEP_EVENT_ERROR:
- goto fail_alloc;
- default:
- goto fail_syntax;
- }
- } while (type != TEP_EVENT_NONE);
-
- if (!current_op && !current_exp)
- goto fail_syntax;
-
- if (!current_op)
- current_op = current_exp;
-
- ret = collapse_tree(current_op, parg, error_str);
- /* collapse_tree() may free current_op, and updates parg accordingly */
- current_op = NULL;
- if (ret < 0)
- goto fail;
-
- free(token);
- return 0;
-
- fail_alloc:
- show_error(error_str, "failed to allocate filter arg");
- ret = TEP_ERRNO__MEM_ALLOC_FAILED;
- goto fail;
- fail_syntax:
- show_error(error_str, "Syntax error");
- ret = TEP_ERRNO__SYNTAX_ERROR;
- fail:
- free_arg(current_op);
- free_arg(current_exp);
- free_arg(arg);
- free(token);
- return ret;
-}
-
-static enum tep_errno
-process_event(struct tep_event *event, const char *filter_str,
- struct tep_filter_arg **parg, char *error_str)
-{
- int ret;
-
- init_input_buf(filter_str, strlen(filter_str));
-
- ret = process_filter(event, parg, error_str, 0);
- if (ret < 0)
- return ret;
-
- /* If parg is NULL, then make it into FALSE */
- if (!*parg) {
- *parg = allocate_arg();
- if (*parg == NULL)
- return TEP_ERRNO__MEM_ALLOC_FAILED;
-
- (*parg)->type = TEP_FILTER_ARG_BOOLEAN;
- (*parg)->boolean.value = TEP_FILTER_FALSE;
- }
-
- return 0;
-}
-
-static enum tep_errno
-filter_event(struct tep_event_filter *filter, struct tep_event *event,
- const char *filter_str, char *error_str)
-{
- struct tep_filter_type *filter_type;
- struct tep_filter_arg *arg;
- enum tep_errno ret;
-
- if (filter_str) {
- ret = process_event(event, filter_str, &arg, error_str);
- if (ret < 0)
- return ret;
-
- } else {
- /* just add a TRUE arg */
- arg = allocate_arg();
- if (arg == NULL)
- return TEP_ERRNO__MEM_ALLOC_FAILED;
-
- arg->type = TEP_FILTER_ARG_BOOLEAN;
- arg->boolean.value = TEP_FILTER_TRUE;
- }
-
- filter_type = add_filter_type(filter, event->id);
- if (filter_type == NULL) {
- free_arg(arg);
- return TEP_ERRNO__MEM_ALLOC_FAILED;
- }
-
- if (filter_type->filter)
- free_arg(filter_type->filter);
- filter_type->filter = arg;
-
- return 0;
-}
-
-static void filter_init_error_buf(struct tep_event_filter *filter)
-{
- /* clear buffer to reset show error */
- init_input_buf("", 0);
- filter->error_buffer[0] = '\0';
-}
-
-/**
- * tep_filter_add_filter_str - add a new filter
- * @filter: the event filter to add to
- * @filter_str: the filter string that contains the filter
- *
- * Returns 0 if the filter was successfully added or a
- * negative error code. Use tep_filter_strerror() to see
- * actual error message in case of error.
- */
-enum tep_errno tep_filter_add_filter_str(struct tep_event_filter *filter,
- const char *filter_str)
-{
- struct tep_handle *tep = filter->tep;
- struct event_list *event;
- struct event_list *events = NULL;
- const char *filter_start;
- const char *next_event;
- char *this_event;
- char *event_name = NULL;
- char *sys_name = NULL;
- char *sp;
- enum tep_errno rtn = 0; /* TEP_ERRNO__SUCCESS */
- int len;
- int ret;
-
- filter_init_error_buf(filter);
-
- filter_start = strchr(filter_str, ':');
- if (filter_start)
- len = filter_start - filter_str;
- else
- len = strlen(filter_str);
-
- do {
- next_event = strchr(filter_str, ',');
- if (next_event &&
- (!filter_start || next_event < filter_start))
- len = next_event - filter_str;
- else if (filter_start)
- len = filter_start - filter_str;
- else
- len = strlen(filter_str);
-
- this_event = malloc(len + 1);
- if (this_event == NULL) {
- /* This can only happen when events is NULL, but still */
- free_events(events);
- return TEP_ERRNO__MEM_ALLOC_FAILED;
- }
- memcpy(this_event, filter_str, len);
- this_event[len] = 0;
-
- if (next_event)
- next_event++;
-
- filter_str = next_event;
-
- sys_name = strtok_r(this_event, "/", &sp);
- event_name = strtok_r(NULL, "/", &sp);
-
- if (!sys_name) {
- /* This can only happen when events is NULL, but still */
- free_events(events);
- free(this_event);
- return TEP_ERRNO__FILTER_NOT_FOUND;
- }
-
- /* Find this event */
- ret = find_event(tep, &events, strim(sys_name), strim(event_name));
- if (ret < 0) {
- free_events(events);
- free(this_event);
- return ret;
- }
- free(this_event);
- } while (filter_str);
-
- /* Skip the ':' */
- if (filter_start)
- filter_start++;
-
- /* filter starts here */
- for (event = events; event; event = event->next) {
- ret = filter_event(filter, event->event, filter_start,
- filter->error_buffer);
- /* Failures are returned if a parse error happened */
- if (ret < 0)
- rtn = ret;
-
- if (ret >= 0 && tep->test_filters) {
- char *test;
- test = tep_filter_make_string(filter, event->event->id);
- if (test) {
- printf(" '%s: %s'\n", event->event->name, test);
- free(test);
- }
- }
- }
-
- free_events(events);
-
- return rtn;
-}
-
-static void free_filter_type(struct tep_filter_type *filter_type)
-{
- free_arg(filter_type->filter);
-}
-
-/**
- * tep_filter_strerror - fill error message in a buffer
- * @filter: the event filter contains error
- * @err: the error code
- * @buf: the buffer to be filled in
- * @buflen: the size of the buffer
- *
- * Returns 0 if message was filled successfully, -1 if error
- */
-int tep_filter_strerror(struct tep_event_filter *filter, enum tep_errno err,
- char *buf, size_t buflen)
-{
- if (err <= __TEP_ERRNO__START || err >= __TEP_ERRNO__END)
- return -1;
-
- if (strlen(filter->error_buffer) > 0) {
- size_t len = snprintf(buf, buflen, "%s", filter->error_buffer);
-
- if (len > buflen)
- return -1;
- return 0;
- }
-
- return tep_strerror(filter->tep, err, buf, buflen);
-}
-
-/**
- * tep_filter_remove_event - remove a filter for an event
- * @filter: the event filter to remove from
- * @event_id: the event to remove a filter for
- *
- * Removes the filter saved for an event defined by @event_id
- * from the @filter.
- *
- * Returns 1: if an event was removed
- * 0: if the event was not found
- */
-int tep_filter_remove_event(struct tep_event_filter *filter,
- int event_id)
-{
- struct tep_filter_type *filter_type;
- unsigned long len;
-
- if (!filter->filters)
- return 0;
-
- filter_type = find_filter_type(filter, event_id);
-
- if (!filter_type)
- return 0;
-
- free_filter_type(filter_type);
-
- /* The filter_type points into the event_filters array */
- len = (unsigned long)(filter->event_filters + filter->filters) -
- (unsigned long)(filter_type + 1);
-
- memmove(filter_type, filter_type + 1, len);
- filter->filters--;
-
- memset(&filter->event_filters[filter->filters], 0,
- sizeof(*filter_type));
-
- return 1;
-}
-
-/**
- * tep_filter_reset - clear all filters in a filter
- * @filter: the event filter to reset
- *
- * Removes all filters from a filter and resets it.
- */
-void tep_filter_reset(struct tep_event_filter *filter)
-{
- int i;
-
- for (i = 0; i < filter->filters; i++)
- free_filter_type(&filter->event_filters[i]);
-
- free(filter->event_filters);
- filter->filters = 0;
- filter->event_filters = NULL;
-}
-
-void tep_filter_free(struct tep_event_filter *filter)
-{
- tep_unref(filter->tep);
-
- tep_filter_reset(filter);
-
- free(filter);
-}
-
-static char *arg_to_str(struct tep_event_filter *filter, struct tep_filter_arg *arg);
-
-static int copy_filter_type(struct tep_event_filter *filter,
- struct tep_event_filter *source,
- struct tep_filter_type *filter_type)
-{
- struct tep_filter_arg *arg;
- struct tep_event *event;
- const char *sys;
- const char *name;
- char *str;
-
- /* Can't assume that the tep's are the same */
- sys = filter_type->event->system;
- name = filter_type->event->name;
- event = tep_find_event_by_name(filter->tep, sys, name);
- if (!event)
- return -1;
-
- str = arg_to_str(source, filter_type->filter);
- if (!str)
- return -1;
-
- if (strcmp(str, "TRUE") == 0 || strcmp(str, "FALSE") == 0) {
- /* Add trivial event */
- arg = allocate_arg();
- if (arg == NULL) {
- free(str);
- return -1;
- }
-
- arg->type = TEP_FILTER_ARG_BOOLEAN;
- if (strcmp(str, "TRUE") == 0)
- arg->boolean.value = 1;
- else
- arg->boolean.value = 0;
-
- filter_type = add_filter_type(filter, event->id);
- if (filter_type == NULL) {
- free(str);
- free_arg(arg);
- return -1;
- }
-
- filter_type->filter = arg;
-
- free(str);
- return 0;
- }
-
- filter_event(filter, event, str, NULL);
- free(str);
-
- return 0;
-}
-
-/**
- * tep_filter_copy - copy a filter using another filter
- * @dest - the filter to copy to
- * @source - the filter to copy from
- *
- * Returns 0 on success and -1 if not all filters were copied
- */
-int tep_filter_copy(struct tep_event_filter *dest, struct tep_event_filter *source)
-{
- int ret = 0;
- int i;
-
- tep_filter_reset(dest);
-
- for (i = 0; i < source->filters; i++) {
- if (copy_filter_type(dest, source, &source->event_filters[i]))
- ret = -1;
- }
- return ret;
-}
-
-static int test_filter(struct tep_event *event, struct tep_filter_arg *arg,
- struct tep_record *record, enum tep_errno *err);
-
-static const char *
-get_comm(struct tep_event *event, struct tep_record *record)
-{
- const char *comm;
- int pid;
-
- pid = tep_data_pid(event->tep, record);
- comm = tep_data_comm_from_pid(event->tep, pid);
- return comm;
-}
-
-static unsigned long long
-get_value(struct tep_event *event,
- struct tep_format_field *field, struct tep_record *record)
-{
- unsigned long long val;
-
- /* Handle our dummy "comm" field */
- if (field == &comm) {
- const char *name;
-
- name = get_comm(event, record);
- return (unsigned long)name;
- }
-
- /* Handle our dummy "cpu" field */
- if (field == &cpu)
- return record->cpu;
-
- tep_read_number_field(field, record->data, &val);
-
- if (!(field->flags & TEP_FIELD_IS_SIGNED))
- return val;
-
- switch (field->size) {
- case 1:
- return (char)val;
- case 2:
- return (short)val;
- case 4:
- return (int)val;
- case 8:
- return (long long)val;
- }
- return val;
-}
-
-static unsigned long long
-get_arg_value(struct tep_event *event, struct tep_filter_arg *arg,
- struct tep_record *record, enum tep_errno *err);
-
-static unsigned long long
-get_exp_value(struct tep_event *event, struct tep_filter_arg *arg,
- struct tep_record *record, enum tep_errno *err)
-{
- unsigned long long lval, rval;
-
- lval = get_arg_value(event, arg->exp.left, record, err);
- rval = get_arg_value(event, arg->exp.right, record, err);
-
- if (*err) {
- /*
- * There was an error, no need to process anymore.
- */
- return 0;
- }
-
- switch (arg->exp.type) {
- case TEP_FILTER_EXP_ADD:
- return lval + rval;
-
- case TEP_FILTER_EXP_SUB:
- return lval - rval;
-
- case TEP_FILTER_EXP_MUL:
- return lval * rval;
-
- case TEP_FILTER_EXP_DIV:
- return lval / rval;
-
- case TEP_FILTER_EXP_MOD:
- return lval % rval;
-
- case TEP_FILTER_EXP_RSHIFT:
- return lval >> rval;
-
- case TEP_FILTER_EXP_LSHIFT:
- return lval << rval;
-
- case TEP_FILTER_EXP_AND:
- return lval & rval;
-
- case TEP_FILTER_EXP_OR:
- return lval | rval;
-
- case TEP_FILTER_EXP_XOR:
- return lval ^ rval;
-
- case TEP_FILTER_EXP_NOT:
- default:
- if (!*err)
- *err = TEP_ERRNO__INVALID_EXP_TYPE;
- }
- return 0;
-}
-
-static unsigned long long
-get_arg_value(struct tep_event *event, struct tep_filter_arg *arg,
- struct tep_record *record, enum tep_errno *err)
-{
- switch (arg->type) {
- case TEP_FILTER_ARG_FIELD:
- return get_value(event, arg->field.field, record);
-
- case TEP_FILTER_ARG_VALUE:
- if (arg->value.type != TEP_FILTER_NUMBER) {
- if (!*err)
- *err = TEP_ERRNO__NOT_A_NUMBER;
- }
- return arg->value.val;
-
- case TEP_FILTER_ARG_EXP:
- return get_exp_value(event, arg, record, err);
-
- default:
- if (!*err)
- *err = TEP_ERRNO__INVALID_ARG_TYPE;
- }
- return 0;
-}
-
-static int test_num(struct tep_event *event, struct tep_filter_arg *arg,
- struct tep_record *record, enum tep_errno *err)
-{
- unsigned long long lval, rval;
-
- lval = get_arg_value(event, arg->num.left, record, err);
- rval = get_arg_value(event, arg->num.right, record, err);
-
- if (*err) {
- /*
- * There was an error, no need to process anymore.
- */
- return 0;
- }
-
- switch (arg->num.type) {
- case TEP_FILTER_CMP_EQ:
- return lval == rval;
-
- case TEP_FILTER_CMP_NE:
- return lval != rval;
-
- case TEP_FILTER_CMP_GT:
- return lval > rval;
-
- case TEP_FILTER_CMP_LT:
- return lval < rval;
-
- case TEP_FILTER_CMP_GE:
- return lval >= rval;
-
- case TEP_FILTER_CMP_LE:
- return lval <= rval;
-
- default:
- if (!*err)
- *err = TEP_ERRNO__ILLEGAL_INTEGER_CMP;
- return 0;
- }
-}
-
-static const char *get_field_str(struct tep_filter_arg *arg, struct tep_record *record)
-{
- struct tep_event *event;
- struct tep_handle *tep;
- unsigned long long addr;
- const char *val = NULL;
- unsigned int size;
- char hex[64];
-
- /* If the field is not a string convert it */
- if (arg->str.field->flags & TEP_FIELD_IS_STRING) {
- val = record->data + arg->str.field->offset;
- size = arg->str.field->size;
-
- if (arg->str.field->flags & TEP_FIELD_IS_DYNAMIC) {
- addr = *(unsigned int *)val;
- size = addr >> 16;
- addr &= 0xffff;
- if (arg->str.field->flags & TEP_FIELD_IS_RELATIVE)
- addr += arg->str.field->offset + arg->str.field->size;
- val = record->data + addr;
- }
-
- /*
- * We need to copy the data since we can't be sure the field
- * is null terminated.
- */
- if (*(val + size - 1)) {
- /* copy it */
- memcpy(arg->str.buffer, val, arg->str.field->size);
- /* the buffer is already NULL terminated */
- val = arg->str.buffer;
- }
-
- } else {
- event = arg->str.field->event;
- tep = event->tep;
- addr = get_value(event, arg->str.field, record);
-
- if (arg->str.field->flags & (TEP_FIELD_IS_POINTER | TEP_FIELD_IS_LONG))
- /* convert to a kernel symbol */
- val = tep_find_function(tep, addr);
-
- if (val == NULL) {
- /* just use the hex of the string name */
- snprintf(hex, 64, "0x%llx", addr);
- val = hex;
- }
- }
-
- return val;
-}
-
-static int test_str(struct tep_event *event, struct tep_filter_arg *arg,
- struct tep_record *record, enum tep_errno *err)
-{
- const char *val;
-
- if (arg->str.field == &comm)
- val = get_comm(event, record);
- else
- val = get_field_str(arg, record);
-
- switch (arg->str.type) {
- case TEP_FILTER_CMP_MATCH:
- return strcmp(val, arg->str.val) == 0;
-
- case TEP_FILTER_CMP_NOT_MATCH:
- return strcmp(val, arg->str.val) != 0;
-
- case TEP_FILTER_CMP_REGEX:
- /* Returns zero on match */
- return !regexec(&arg->str.reg, val, 0, NULL, 0);
-
- case TEP_FILTER_CMP_NOT_REGEX:
- return regexec(&arg->str.reg, val, 0, NULL, 0);
-
- default:
- if (!*err)
- *err = TEP_ERRNO__ILLEGAL_STRING_CMP;
- return 0;
- }
-}
-
-static int test_op(struct tep_event *event, struct tep_filter_arg *arg,
- struct tep_record *record, enum tep_errno *err)
-{
- switch (arg->op.type) {
- case TEP_FILTER_OP_AND:
- return test_filter(event, arg->op.left, record, err) &&
- test_filter(event, arg->op.right, record, err);
-
- case TEP_FILTER_OP_OR:
- return test_filter(event, arg->op.left, record, err) ||
- test_filter(event, arg->op.right, record, err);
-
- case TEP_FILTER_OP_NOT:
- return !test_filter(event, arg->op.right, record, err);
-
- default:
- if (!*err)
- *err = TEP_ERRNO__INVALID_OP_TYPE;
- return 0;
- }
-}
-
-static int test_filter(struct tep_event *event, struct tep_filter_arg *arg,
- struct tep_record *record, enum tep_errno *err)
-{
- if (*err) {
- /*
- * There was an error, no need to process anymore.
- */
- return 0;
- }
-
- switch (arg->type) {
- case TEP_FILTER_ARG_BOOLEAN:
- /* easy case */
- return arg->boolean.value;
-
- case TEP_FILTER_ARG_OP:
- return test_op(event, arg, record, err);
-
- case TEP_FILTER_ARG_NUM:
- return test_num(event, arg, record, err);
-
- case TEP_FILTER_ARG_STR:
- return test_str(event, arg, record, err);
-
- case TEP_FILTER_ARG_EXP:
- case TEP_FILTER_ARG_VALUE:
- case TEP_FILTER_ARG_FIELD:
- /*
- * Expressions, fields and values evaluate
- * to true if they return non zero
- */
- return !!get_arg_value(event, arg, record, err);
-
- default:
- if (!*err)
- *err = TEP_ERRNO__INVALID_ARG_TYPE;
- return 0;
- }
-}
-
-/**
- * tep_event_filtered - return true if event has filter
- * @filter: filter struct with filter information
- * @event_id: event id to test if filter exists
- *
- * Returns 1 if filter found for @event_id
- * otherwise 0;
- */
-int tep_event_filtered(struct tep_event_filter *filter, int event_id)
-{
- struct tep_filter_type *filter_type;
-
- if (!filter->filters)
- return 0;
-
- filter_type = find_filter_type(filter, event_id);
-
- return filter_type ? 1 : 0;
-}
-
-/**
- * tep_filter_match - test if a record matches a filter
- * @filter: filter struct with filter information
- * @record: the record to test against the filter
- *
- * Returns: match result or error code (prefixed with TEP_ERRNO__)
- * FILTER_MATCH - filter found for event and @record matches
- * FILTER_MISS - filter found for event and @record does not match
- * FILTER_NOT_FOUND - no filter found for @record's event
- * NO_FILTER - if no filters exist
- * otherwise - error occurred during test
- */
-enum tep_errno tep_filter_match(struct tep_event_filter *filter,
- struct tep_record *record)
-{
- struct tep_handle *tep = filter->tep;
- struct tep_filter_type *filter_type;
- int event_id;
- int ret;
- enum tep_errno err = 0;
-
- filter_init_error_buf(filter);
-
- if (!filter->filters)
- return TEP_ERRNO__NO_FILTER;
-
- event_id = tep_data_type(tep, record);
-
- filter_type = find_filter_type(filter, event_id);
- if (!filter_type)
- return TEP_ERRNO__FILTER_NOT_FOUND;
-
- ret = test_filter(filter_type->event, filter_type->filter, record, &err);
- if (err)
- return err;
-
- return ret ? TEP_ERRNO__FILTER_MATCH : TEP_ERRNO__FILTER_MISS;
-}
-
-static char *op_to_str(struct tep_event_filter *filter, struct tep_filter_arg *arg)
-{
- char *str = NULL;
- char *left = NULL;
- char *right = NULL;
- char *op = NULL;
- int left_val = -1;
- int right_val = -1;
- int val;
-
- switch (arg->op.type) {
- case TEP_FILTER_OP_AND:
- op = "&&";
- /* fall through */
- case TEP_FILTER_OP_OR:
- if (!op)
- op = "||";
-
- left = arg_to_str(filter, arg->op.left);
- right = arg_to_str(filter, arg->op.right);
- if (!left || !right)
- break;
-
- /* Try to consolidate boolean values */
- if (strcmp(left, "TRUE") == 0)
- left_val = 1;
- else if (strcmp(left, "FALSE") == 0)
- left_val = 0;
-
- if (strcmp(right, "TRUE") == 0)
- right_val = 1;
- else if (strcmp(right, "FALSE") == 0)
- right_val = 0;
-
- if (left_val >= 0) {
- if ((arg->op.type == TEP_FILTER_OP_AND && !left_val) ||
- (arg->op.type == TEP_FILTER_OP_OR && left_val)) {
- /* Just return left value */
- str = left;
- left = NULL;
- break;
- }
- if (right_val >= 0) {
- /* just evaluate this. */
- val = 0;
- switch (arg->op.type) {
- case TEP_FILTER_OP_AND:
- val = left_val && right_val;
- break;
- case TEP_FILTER_OP_OR:
- val = left_val || right_val;
- break;
- default:
- break;
- }
- if (asprintf(&str, val ? "TRUE" : "FALSE") < 0)
- str = NULL;
- break;
- }
- }
- if (right_val >= 0) {
- if ((arg->op.type == TEP_FILTER_OP_AND && !right_val) ||
- (arg->op.type == TEP_FILTER_OP_OR && right_val)) {
- /* Just return right value */
- str = right;
- right = NULL;
- break;
- }
- /* The right value is meaningless */
- str = left;
- left = NULL;
- break;
- }
-
- if (asprintf(&str, "(%s) %s (%s)", left, op, right) < 0)
- str = NULL;
- break;
-
- case TEP_FILTER_OP_NOT:
- op = "!";
- right = arg_to_str(filter, arg->op.right);
- if (!right)
- break;
-
- /* See if we can consolidate */
- if (strcmp(right, "TRUE") == 0)
- right_val = 1;
- else if (strcmp(right, "FALSE") == 0)
- right_val = 0;
- if (right_val >= 0) {
- /* just return the opposite */
- if (asprintf(&str, right_val ? "FALSE" : "TRUE") < 0)
- str = NULL;
- break;
- }
- if (asprintf(&str, "%s(%s)", op, right) < 0)
- str = NULL;
- break;
-
- default:
- /* ?? */
- break;
- }
- free(left);
- free(right);
- return str;
-}
-
-static char *val_to_str(struct tep_event_filter *filter, struct tep_filter_arg *arg)
-{
- char *str = NULL;
-
- if (asprintf(&str, "%lld", arg->value.val) < 0)
- str = NULL;
-
- return str;
-}
-
-static char *field_to_str(struct tep_event_filter *filter, struct tep_filter_arg *arg)
-{
- return strdup(arg->field.field->name);
-}
-
-static char *exp_to_str(struct tep_event_filter *filter, struct tep_filter_arg *arg)
-{
- char *lstr;
- char *rstr;
- char *op;
- char *str = NULL;
-
- lstr = arg_to_str(filter, arg->exp.left);
- rstr = arg_to_str(filter, arg->exp.right);
- if (!lstr || !rstr)
- goto out;
-
- switch (arg->exp.type) {
- case TEP_FILTER_EXP_ADD:
- op = "+";
- break;
- case TEP_FILTER_EXP_SUB:
- op = "-";
- break;
- case TEP_FILTER_EXP_MUL:
- op = "*";
- break;
- case TEP_FILTER_EXP_DIV:
- op = "/";
- break;
- case TEP_FILTER_EXP_MOD:
- op = "%";
- break;
- case TEP_FILTER_EXP_RSHIFT:
- op = ">>";
- break;
- case TEP_FILTER_EXP_LSHIFT:
- op = "<<";
- break;
- case TEP_FILTER_EXP_AND:
- op = "&";
- break;
- case TEP_FILTER_EXP_OR:
- op = "|";
- break;
- case TEP_FILTER_EXP_XOR:
- op = "^";
- break;
- default:
- op = "[ERROR IN EXPRESSION TYPE]";
- break;
- }
-
- if (asprintf(&str, "%s %s %s", lstr, op, rstr) < 0)
- str = NULL;
-out:
- free(lstr);
- free(rstr);
-
- return str;
-}
-
-static char *num_to_str(struct tep_event_filter *filter, struct tep_filter_arg *arg)
-{
- char *lstr;
- char *rstr;
- char *str = NULL;
- char *op = NULL;
-
- lstr = arg_to_str(filter, arg->num.left);
- rstr = arg_to_str(filter, arg->num.right);
- if (!lstr || !rstr)
- goto out;
-
- switch (arg->num.type) {
- case TEP_FILTER_CMP_EQ:
- op = "==";
- /* fall through */
- case TEP_FILTER_CMP_NE:
- if (!op)
- op = "!=";
- /* fall through */
- case TEP_FILTER_CMP_GT:
- if (!op)
- op = ">";
- /* fall through */
- case TEP_FILTER_CMP_LT:
- if (!op)
- op = "<";
- /* fall through */
- case TEP_FILTER_CMP_GE:
- if (!op)
- op = ">=";
- /* fall through */
- case TEP_FILTER_CMP_LE:
- if (!op)
- op = "<=";
-
- if (asprintf(&str, "%s %s %s", lstr, op, rstr) < 0)
- str = NULL;
- break;
-
- default:
- /* ?? */
- break;
- }
-
-out:
- free(lstr);
- free(rstr);
- return str;
-}
-
-static char *str_to_str(struct tep_event_filter *filter, struct tep_filter_arg *arg)
-{
- char *str = NULL;
- char *op = NULL;
-
- switch (arg->str.type) {
- case TEP_FILTER_CMP_MATCH:
- op = "==";
- /* fall through */
- case TEP_FILTER_CMP_NOT_MATCH:
- if (!op)
- op = "!=";
- /* fall through */
- case TEP_FILTER_CMP_REGEX:
- if (!op)
- op = "=~";
- /* fall through */
- case TEP_FILTER_CMP_NOT_REGEX:
- if (!op)
- op = "!~";
-
- if (asprintf(&str, "%s %s \"%s\"",
- arg->str.field->name, op, arg->str.val) < 0)
- str = NULL;
- break;
-
- default:
- /* ?? */
- break;
- }
- return str;
-}
-
-static char *arg_to_str(struct tep_event_filter *filter, struct tep_filter_arg *arg)
-{
- char *str = NULL;
-
- switch (arg->type) {
- case TEP_FILTER_ARG_BOOLEAN:
- if (asprintf(&str, arg->boolean.value ? "TRUE" : "FALSE") < 0)
- str = NULL;
- return str;
-
- case TEP_FILTER_ARG_OP:
- return op_to_str(filter, arg);
-
- case TEP_FILTER_ARG_NUM:
- return num_to_str(filter, arg);
-
- case TEP_FILTER_ARG_STR:
- return str_to_str(filter, arg);
-
- case TEP_FILTER_ARG_VALUE:
- return val_to_str(filter, arg);
-
- case TEP_FILTER_ARG_FIELD:
- return field_to_str(filter, arg);
-
- case TEP_FILTER_ARG_EXP:
- return exp_to_str(filter, arg);
-
- default:
- /* ?? */
- return NULL;
- }
-
-}
-
-/**
- * tep_filter_make_string - return a string showing the filter
- * @filter: filter struct with filter information
- * @event_id: the event id to return the filter string with
- *
- * Returns a string that displays the filter contents.
- * This string must be freed with free(str).
- * NULL is returned if no filter is found or allocation failed.
- */
-char *
-tep_filter_make_string(struct tep_event_filter *filter, int event_id)
-{
- struct tep_filter_type *filter_type;
-
- if (!filter->filters)
- return NULL;
-
- filter_type = find_filter_type(filter, event_id);
-
- if (!filter_type)
- return NULL;
-
- return arg_to_str(filter, filter_type->filter);
-}
-
-/**
- * tep_filter_compare - compare two filters and return if they are the same
- * @filter1: Filter to compare with @filter2
- * @filter2: Filter to compare with @filter1
- *
- * Returns:
- * 1 if the two filters hold the same content.
- * 0 if they do not.
- */
-int tep_filter_compare(struct tep_event_filter *filter1, struct tep_event_filter *filter2)
-{
- struct tep_filter_type *filter_type1;
- struct tep_filter_type *filter_type2;
- char *str1, *str2;
- int result;
- int i;
-
- /* Do the easy checks first */
- if (filter1->filters != filter2->filters)
- return 0;
- if (!filter1->filters && !filter2->filters)
- return 1;
-
- /*
- * Now take a look at each of the events to see if they have the same
- * filters to them.
- */
- for (i = 0; i < filter1->filters; i++) {
- filter_type1 = &filter1->event_filters[i];
- filter_type2 = find_filter_type(filter2, filter_type1->event_id);
- if (!filter_type2)
- break;
- if (filter_type1->filter->type != filter_type2->filter->type)
- break;
- /* The best way to compare complex filters is with strings */
- str1 = arg_to_str(filter1, filter_type1->filter);
- str2 = arg_to_str(filter2, filter_type2->filter);
- if (str1 && str2)
- result = strcmp(str1, str2) != 0;
- else
- /* bail out if allocation fails */
- result = 1;
-
- free(str1);
- free(str2);
- if (result)
- break;
- }
-
- if (i < filter1->filters)
- return 0;
- return 1;
-}
-
diff --git a/tools/lib/traceevent/parse-utils.c b/tools/lib/traceevent/parse-utils.c
deleted file mode 100644
index e998671..0000000
--- a/tools/lib/traceevent/parse-utils.c
+++ /dev/null
@@ -1,71 +0,0 @@
-// SPDX-License-Identifier: LGPL-2.1
-/*
- * Copyright (C) 2010 Red Hat Inc, Steven Rostedt <srostedt@redhat.com>
- *
- */
-#include <stdio.h>
-#include <stdlib.h>
-#include <string.h>
-#include <stdarg.h>
-#include <errno.h>
-
-#define __weak __attribute__((weak))
-
-void __vwarning(const char *fmt, va_list ap)
-{
- if (errno)
- perror("libtraceevent");
- errno = 0;
-
- fprintf(stderr, " ");
- vfprintf(stderr, fmt, ap);
-
- fprintf(stderr, "\n");
-}
-
-void __warning(const char *fmt, ...)
-{
- va_list ap;
-
- va_start(ap, fmt);
- __vwarning(fmt, ap);
- va_end(ap);
-}
-
-void __weak warning(const char *fmt, ...)
-{
- va_list ap;
-
- va_start(ap, fmt);
- __vwarning(fmt, ap);
- va_end(ap);
-}
-
-void __vpr_stat(const char *fmt, va_list ap)
-{
- vprintf(fmt, ap);
- printf("\n");
-}
-
-void __pr_stat(const char *fmt, ...)
-{
- va_list ap;
-
- va_start(ap, fmt);
- __vpr_stat(fmt, ap);
- va_end(ap);
-}
-
-void __weak vpr_stat(const char *fmt, va_list ap)
-{
- __vpr_stat(fmt, ap);
-}
-
-void __weak pr_stat(const char *fmt, ...)
-{
- va_list ap;
-
- va_start(ap, fmt);
- __vpr_stat(fmt, ap);
- va_end(ap);
-}
diff --git a/tools/lib/traceevent/plugins/Build b/tools/lib/traceevent/plugins/Build
deleted file mode 100644
index dd4da82..0000000
--- a/tools/lib/traceevent/plugins/Build
+++ /dev/null
@@ -1,12 +0,0 @@
-plugin_jbd2-y += plugin_jbd2.o
-plugin_hrtimer-y += plugin_hrtimer.o
-plugin_kmem-y += plugin_kmem.o
-plugin_kvm-y += plugin_kvm.o
-plugin_mac80211-y += plugin_mac80211.o
-plugin_sched_switch-y += plugin_sched_switch.o
-plugin_function-y += plugin_function.o
-plugin_futex-y += plugin_futex.o
-plugin_xen-y += plugin_xen.o
-plugin_scsi-y += plugin_scsi.o
-plugin_cfg80211-y += plugin_cfg80211.o
-plugin_tlb-y += plugin_tlb.o
\ No newline at end of file
diff --git a/tools/lib/traceevent/plugins/Makefile b/tools/lib/traceevent/plugins/Makefile
deleted file mode 100644
index 47e8025..0000000
--- a/tools/lib/traceevent/plugins/Makefile
+++ /dev/null
@@ -1,225 +0,0 @@
-# SPDX-License-Identifier: GPL-2.0
-
-#MAKEFLAGS += --no-print-directory
-
-
-# Makefiles suck: This macro sets a default value of $(2) for the
-# variable named by $(1), unless the variable has been set by
-# environment or command line. This is necessary for CC and AR
-# because make sets default values, so the simpler ?= approach
-# won't work as expected.
-define allow-override
- $(if $(or $(findstring environment,$(origin $(1))),\
- $(findstring command line,$(origin $(1)))),,\
- $(eval $(1) = $(2)))
-endef
-
-# Allow setting CC and AR, or setting CROSS_COMPILE as a prefix.
-$(call allow-override,CC,$(CROSS_COMPILE)gcc)
-$(call allow-override,AR,$(CROSS_COMPILE)ar)
-$(call allow-override,NM,$(CROSS_COMPILE)nm)
-$(call allow-override,PKG_CONFIG,pkg-config)
-
-EXT = -std=gnu99
-INSTALL = install
-
-# Use DESTDIR for installing into a different root directory.
-# This is useful for building a package. The program will be
-# installed in this directory as if it was the root directory.
-# Then the build tool can move it later.
-DESTDIR ?=
-DESTDIR_SQ = '$(subst ','\'',$(DESTDIR))'
-
-LP64 := $(shell echo __LP64__ | ${CC} ${CFLAGS} -E -x c - | tail -n 1)
-ifeq ($(LP64), 1)
- libdir_relative_tmp = lib64
-else
- libdir_relative_tmp = lib
-endif
-
-libdir_relative ?= $(libdir_relative_tmp)
-prefix ?= /usr/local
-libdir = $(prefix)/$(libdir_relative)
-
-set_plugin_dir := 1
-
-# Set plugin_dir to preffered global plugin location
-# If we install under $HOME directory we go under
-# $(HOME)/.local/lib/traceevent/plugins
-#
-# We dont set PLUGIN_DIR in case we install under $HOME
-# directory, because by default the code looks under:
-# $(HOME)/.local/lib/traceevent/plugins by default.
-#
-ifeq ($(plugin_dir),)
-ifeq ($(prefix),$(HOME))
-override plugin_dir = $(HOME)/.local/lib/traceevent/plugins
-set_plugin_dir := 0
-else
-override plugin_dir = $(libdir)/traceevent/plugins
-endif
-endif
-
-ifeq ($(set_plugin_dir),1)
-PLUGIN_DIR = -DPLUGIN_DIR="$(plugin_dir)"
-PLUGIN_DIR_SQ = '$(subst ','\'',$(PLUGIN_DIR))'
-endif
-
-include ../../../scripts/Makefile.include
-
-# copy a bit from Linux kbuild
-
-ifeq ("$(origin V)", "command line")
- VERBOSE = $(V)
-endif
-ifndef VERBOSE
- VERBOSE = 0
-endif
-
-ifeq ($(srctree),)
-srctree := $(patsubst %/,%,$(dir $(CURDIR)))
-srctree := $(patsubst %/,%,$(dir $(srctree)))
-srctree := $(patsubst %/,%,$(dir $(srctree)))
-srctree := $(patsubst %/,%,$(dir $(srctree)))
-#$(info Determined 'srctree' to be $(srctree))
-endif
-
-export prefix libdir src obj
-
-# Shell quotes
-plugin_dir_SQ = $(subst ','\'',$(plugin_dir))
-
-CONFIG_INCLUDES =
-CONFIG_LIBS =
-CONFIG_FLAGS =
-
-OBJ = $@
-N =
-
-INCLUDES = -I. -I.. -I $(srctree)/tools/include $(CONFIG_INCLUDES)
-
-# Set compile option CFLAGS
-ifdef EXTRA_CFLAGS
- CFLAGS := $(EXTRA_CFLAGS)
-else
- CFLAGS := -g -Wall
-endif
-
-# Append required CFLAGS
-override CFLAGS += -fPIC
-override CFLAGS += $(CONFIG_FLAGS) $(INCLUDES) $(PLUGIN_DIR_SQ)
-override CFLAGS += $(udis86-flags) -D_GNU_SOURCE
-
-ifeq ($(VERBOSE),1)
- Q =
-else
- Q = @
-endif
-
-# Disable command line variables (CFLAGS) override from top
-# level Makefile (perf), otherwise build Makefile will get
-# the same command line setup.
-MAKEOVERRIDES=
-
-export srctree OUTPUT CC LD CFLAGS V
-
-build := -f $(srctree)/tools/build/Makefile.build dir=. obj
-
-DYNAMIC_LIST_FILE := $(OUTPUT)libtraceevent-dynamic-list
-
-PLUGINS = plugin_jbd2.so
-PLUGINS += plugin_hrtimer.so
-PLUGINS += plugin_kmem.so
-PLUGINS += plugin_kvm.so
-PLUGINS += plugin_mac80211.so
-PLUGINS += plugin_sched_switch.so
-PLUGINS += plugin_function.so
-PLUGINS += plugin_futex.so
-PLUGINS += plugin_xen.so
-PLUGINS += plugin_scsi.so
-PLUGINS += plugin_cfg80211.so
-PLUGINS += plugin_tlb.so
-
-PLUGINS := $(addprefix $(OUTPUT),$(PLUGINS))
-PLUGINS_IN := $(PLUGINS:.so=-in.o)
-
-plugins: $(PLUGINS) $(DYNAMIC_LIST_FILE)
-
-__plugin_obj = $(notdir $@)
- plugin_obj = $(__plugin_obj:-in.o=)
-
-$(PLUGINS_IN): force
- $(Q)$(MAKE) $(build)=$(plugin_obj)
-
-$(OUTPUT)libtraceevent-dynamic-list: $(PLUGINS)
- $(QUIET_GEN)$(call do_generate_dynamic_list_file, $(PLUGINS), $@)
-
-$(OUTPUT)%.so: $(OUTPUT)%-in.o
- $(QUIET_LINK)$(CC) $(CFLAGS) -shared $(LDFLAGS) -nostartfiles -o $@ $^
-
-define update_dir
- (echo $1 > $@.tmp; \
- if [ -r $@ ] && cmp -s $@ $@.tmp; then \
- rm -f $@.tmp; \
- else \
- echo ' UPDATE $@'; \
- mv -f $@.tmp $@; \
- fi);
-endef
-
-tags: force
- $(RM) tags
- find . -name '*.[ch]' | xargs ctags --extra=+f --c-kinds=+px \
- --regex-c++='/_PE\(([^,)]*).*/TEP_ERRNO__\1/'
-
-TAGS: force
- $(RM) TAGS
- find . -name '*.[ch]' | xargs etags \
- --regex='/_PE(\([^,)]*\).*/TEP_ERRNO__\1/'
-
-define do_install_mkdir
- if [ ! -d '$(DESTDIR_SQ)$1' ]; then \
- $(INSTALL) -d -m 755 '$(DESTDIR_SQ)$1'; \
- fi
-endef
-
-define do_install
- $(call do_install_mkdir,$2); \
- $(INSTALL) $(if $3,-m $3,) $1 '$(DESTDIR_SQ)$2'
-endef
-
-define do_install_plugins
- for plugin in $1; do \
- $(call do_install,$$plugin,$(plugin_dir_SQ)); \
- done
-endef
-
-define do_generate_dynamic_list_file
- symbol_type=`$(NM) -u -D $1 | awk 'NF>1 {print $$1}' | \
- xargs echo "U w W" | tr 'w ' 'W\n' | sort -u | xargs echo`;\
- if [ "$$symbol_type" = "U W" ];then \
- (echo '{'; \
- $(NM) -u -D $1 | awk 'NF>1 {sub("@.*", "", $$2); print "\t"$$2";"}' | sort -u;\
- echo '};'; \
- ) > $2; \
- else \
- (echo Either missing one of [$1] or bad version of $(NM)) 1>&2;\
- fi
-endef
-
-install: $(PLUGINS)
- $(call QUIET_INSTALL, trace_plugins) \
- $(call do_install_plugins, $(PLUGINS))
-
-clean:
- $(call QUIET_CLEAN, trace_plugins) \
- $(RM) *.o *~ $(TARGETS) *.a *.so $(VERSION_FILES) .*.d .*.cmd; \
- $(RM) $(OUTPUT)libtraceevent-dynamic-list \
- $(RM) TRACEEVENT-CFLAGS tags TAGS;
-
-PHONY += force plugins
-force:
-
-# Declare the contents of the .PHONY variable as phony. We keep that
-# information in a variable so we can use it in if_changed and friends.
-.PHONY: $(PHONY)
diff --git a/tools/lib/traceevent/plugins/plugin_cfg80211.c b/tools/lib/traceevent/plugins/plugin_cfg80211.c
deleted file mode 100644
index 3d43b56..0000000
--- a/tools/lib/traceevent/plugins/plugin_cfg80211.c
+++ /dev/null
@@ -1,43 +0,0 @@
-// SPDX-License-Identifier: GPL-2.0
-#include <stdio.h>
-#include <string.h>
-#include <inttypes.h>
-#include <endian.h>
-#include "event-parse.h"
-
-/*
- * From glibc endian.h, for older systems where it is not present, e.g.: RHEL5,
- * Fedora6.
- */
-#ifndef le16toh
-# if __BYTE_ORDER == __LITTLE_ENDIAN
-# define le16toh(x) (x)
-# else
-# define le16toh(x) __bswap_16 (x)
-# endif
-#endif
-
-
-static unsigned long long
-process___le16_to_cpup(struct trace_seq *s, unsigned long long *args)
-{
- uint16_t *val = (uint16_t *) (unsigned long) args[0];
- return val ? (long long) le16toh(*val) : 0;
-}
-
-int TEP_PLUGIN_LOADER(struct tep_handle *tep)
-{
- tep_register_print_function(tep,
- process___le16_to_cpup,
- TEP_FUNC_ARG_INT,
- "__le16_to_cpup",
- TEP_FUNC_ARG_PTR,
- TEP_FUNC_ARG_VOID);
- return 0;
-}
-
-void TEP_PLUGIN_UNLOADER(struct tep_handle *tep)
-{
- tep_unregister_print_function(tep, process___le16_to_cpup,
- "__le16_to_cpup");
-}
diff --git a/tools/lib/traceevent/plugins/plugin_function.c b/tools/lib/traceevent/plugins/plugin_function.c
deleted file mode 100644
index 807b16e..0000000
--- a/tools/lib/traceevent/plugins/plugin_function.c
+++ /dev/null
@@ -1,282 +0,0 @@
-// SPDX-License-Identifier: LGPL-2.1
-/*
- * Copyright (C) 2009, 2010 Red Hat Inc, Steven Rostedt <srostedt@redhat.com>
- */
-#include <stdio.h>
-#include <stdlib.h>
-#include <string.h>
-
-#include "event-parse.h"
-#include "event-utils.h"
-#include "trace-seq.h"
-
-static struct func_stack {
- int size;
- char **stack;
-} *fstack;
-
-static int cpus = -1;
-
-#define STK_BLK 10
-
-struct tep_plugin_option plugin_options[] =
-{
- {
- .name = "parent",
- .plugin_alias = "ftrace",
- .description =
- "Print parent of functions for function events",
- },
- {
- .name = "indent",
- .plugin_alias = "ftrace",
- .description =
- "Try to show function call indents, based on parents",
- .set = 1,
- },
- {
- .name = "offset",
- .plugin_alias = "ftrace",
- .description =
- "Show function names as well as their offsets",
- .set = 0,
- },
- {
- .name = NULL,
- }
-};
-
-static struct tep_plugin_option *ftrace_parent = &plugin_options[0];
-static struct tep_plugin_option *ftrace_indent = &plugin_options[1];
-static struct tep_plugin_option *ftrace_offset = &plugin_options[2];
-
-static void add_child(struct func_stack *stack, const char *child, int pos)
-{
- int i;
-
- if (!child)
- return;
-
- if (pos < stack->size)
- free(stack->stack[pos]);
- else {
- char **ptr;
-
- ptr = realloc(stack->stack, sizeof(char *) *
- (stack->size + STK_BLK));
- if (!ptr) {
- warning("could not allocate plugin memory\n");
- return;
- }
-
- stack->stack = ptr;
-
- for (i = stack->size; i < stack->size + STK_BLK; i++)
- stack->stack[i] = NULL;
- stack->size += STK_BLK;
- }
-
- stack->stack[pos] = strdup(child);
-}
-
-static int add_and_get_index(const char *parent, const char *child, int cpu)
-{
- int i;
-
- if (cpu < 0)
- return 0;
-
- if (cpu > cpus) {
- struct func_stack *ptr;
-
- ptr = realloc(fstack, sizeof(*fstack) * (cpu + 1));
- if (!ptr) {
- warning("could not allocate plugin memory\n");
- return 0;
- }
-
- fstack = ptr;
-
- /* Account for holes in the cpu count */
- for (i = cpus + 1; i <= cpu; i++)
- memset(&fstack[i], 0, sizeof(fstack[i]));
- cpus = cpu;
- }
-
- for (i = 0; i < fstack[cpu].size && fstack[cpu].stack[i]; i++) {
- if (strcmp(parent, fstack[cpu].stack[i]) == 0) {
- add_child(&fstack[cpu], child, i+1);
- return i;
- }
- }
-
- /* Not found */
- add_child(&fstack[cpu], parent, 0);
- add_child(&fstack[cpu], child, 1);
- return 0;
-}
-
-static void show_function(struct trace_seq *s, struct tep_handle *tep,
- const char *func, unsigned long long function)
-{
- unsigned long long offset;
-
- trace_seq_printf(s, "%s", func);
- if (ftrace_offset->set) {
- offset = tep_find_function_address(tep, function);
- trace_seq_printf(s, "+0x%x ", (int)(function - offset));
- }
-}
-
-static int function_handler(struct trace_seq *s, struct tep_record *record,
- struct tep_event *event, void *context)
-{
- struct tep_handle *tep = event->tep;
- unsigned long long function;
- unsigned long long pfunction;
- const char *func;
- const char *parent;
- int index = 0;
-
- if (tep_get_field_val(s, event, "ip", record, &function, 1))
- return trace_seq_putc(s, '!');
-
- func = tep_find_function(tep, function);
-
- if (tep_get_field_val(s, event, "parent_ip", record, &pfunction, 1))
- return trace_seq_putc(s, '!');
-
- parent = tep_find_function(tep, pfunction);
-
- if (parent && ftrace_indent->set)
- index = add_and_get_index(parent, func, record->cpu);
-
- trace_seq_printf(s, "%*s", index*3, "");
-
- if (func)
- show_function(s, tep, func, function);
- else
- trace_seq_printf(s, "0x%llx", function);
-
- if (ftrace_parent->set) {
- trace_seq_printf(s, " <-- ");
- if (parent)
- show_function(s, tep, parent, pfunction);
- else
- trace_seq_printf(s, "0x%llx", pfunction);
- }
-
- return 0;
-}
-
-static int
-trace_stack_handler(struct trace_seq *s, struct tep_record *record,
- struct tep_event *event, void *context)
-{
- struct tep_format_field *field;
- unsigned long long addr;
- const char *func;
- int long_size;
- void *data = record->data;
-
- field = tep_find_any_field(event, "caller");
- if (!field) {
- trace_seq_printf(s, "<CANT FIND FIELD %s>", "caller");
- return 0;
- }
-
- trace_seq_puts(s, "<stack trace >\n");
-
- long_size = tep_get_long_size(event->tep);
-
- for (data += field->offset; data < record->data + record->size;
- data += long_size) {
- addr = tep_read_number(event->tep, data, long_size);
-
- if ((long_size == 8 && addr == (unsigned long long)-1) ||
- ((int)addr == -1))
- break;
-
- func = tep_find_function(event->tep, addr);
- if (func)
- trace_seq_printf(s, "=> %s (%llx)\n", func, addr);
- else
- trace_seq_printf(s, "=> %llx\n", addr);
- }
-
- return 0;
-}
-
-static int
-trace_raw_data_handler(struct trace_seq *s, struct tep_record *record,
- struct tep_event *event, void *context)
-{
- struct tep_format_field *field;
- unsigned long long id;
- int long_size;
- void *data = record->data;
-
- if (tep_get_field_val(s, event, "id", record, &id, 1))
- return trace_seq_putc(s, '!');
-
- trace_seq_printf(s, "# %llx", id);
-
- field = tep_find_any_field(event, "buf");
- if (!field) {
- trace_seq_printf(s, "<CANT FIND FIELD %s>", "buf");
- return 0;
- }
-
- long_size = tep_get_long_size(event->tep);
-
- for (data += field->offset; data < record->data + record->size;
- data += long_size) {
- int size = sizeof(long);
- int left = (record->data + record->size) - data;
- int i;
-
- if (size > left)
- size = left;
-
- for (i = 0; i < size; i++)
- trace_seq_printf(s, " %02x", *(unsigned char *)(data + i));
- }
-
- return 0;
-}
-
-int TEP_PLUGIN_LOADER(struct tep_handle *tep)
-{
- tep_register_event_handler(tep, -1, "ftrace", "function",
- function_handler, NULL);
-
- tep_register_event_handler(tep, -1, "ftrace", "kernel_stack",
- trace_stack_handler, NULL);
-
- tep_register_event_handler(tep, -1, "ftrace", "raw_data",
- trace_raw_data_handler, NULL);
-
- tep_plugin_add_options("ftrace", plugin_options);
-
- return 0;
-}
-
-void TEP_PLUGIN_UNLOADER(struct tep_handle *tep)
-{
- int i, x;
-
- tep_unregister_event_handler(tep, -1, "ftrace", "function",
- function_handler, NULL);
-
- for (i = 0; i <= cpus; i++) {
- for (x = 0; x < fstack[i].size && fstack[i].stack[x]; x++)
- free(fstack[i].stack[x]);
- free(fstack[i].stack);
- }
-
- tep_plugin_remove_options(plugin_options);
-
- free(fstack);
- fstack = NULL;
- cpus = -1;
-}
diff --git a/tools/lib/traceevent/plugins/plugin_futex.c b/tools/lib/traceevent/plugins/plugin_futex.c
deleted file mode 100644
index eb7c9f8..0000000
--- a/tools/lib/traceevent/plugins/plugin_futex.c
+++ /dev/null
@@ -1,123 +0,0 @@
-// SPDX-License-Identifier: LGPL-2.1
-/*
- * Copyright (C) 2017 National Instruments Corp.
- *
- * Author: Julia Cartwright <julia@ni.com>
- *
- */
-#include <stdio.h>
-#include <stdlib.h>
-#include <string.h>
-#include <linux/futex.h>
-
-#include "event-parse.h"
-
-#define ARRAY_SIZE(_a) (sizeof(_a) / sizeof((_a)[0]))
-
-struct futex_args {
- unsigned long long uaddr;
- unsigned long long op;
- unsigned long long val;
- unsigned long long utime; /* or val2 */
- unsigned long long uaddr2;
- unsigned long long val3;
-};
-
-struct futex_op {
- const char *name;
- const char *fmt_val;
- const char *fmt_utime;
- const char *fmt_uaddr2;
- const char *fmt_val3;
-};
-
-static const struct futex_op futex_op_tbl[] = {
- { "FUTEX_WAIT", " val=0x%08llx", " utime=0x%08llx", NULL, NULL },
- { "FUTEX_WAKE", " val=%llu", NULL, NULL, NULL },
- { "FUTEX_FD", " val=%llu", NULL, NULL, NULL },
- { "FUTEX_REQUEUE", " val=%llu", " val2=%llu", " uaddr2=0x%08llx", NULL },
- { "FUTEX_CMP_REQUEUE", " val=%llu", " val2=%llu", " uaddr2=0x%08llx", " val3=0x%08llx" },
- { "FUTEX_WAKE_OP", " val=%llu", " val2=%llu", " uaddr2=0x%08llx", " val3=0x%08llx" },
- { "FUTEX_LOCK_PI", NULL, " utime=0x%08llx", NULL, NULL },
- { "FUTEX_UNLOCK_PI", NULL, NULL, NULL, NULL },
- { "FUTEX_TRYLOCK_PI", NULL, NULL, NULL, NULL },
- { "FUTEX_WAIT_BITSET", " val=0x%08llx", " utime=0x%08llx", NULL, " val3=0x%08llx" },
- { "FUTEX_WAKE_BITSET", " val=%llu", NULL, NULL, " val3=0x%08llx" },
- { "FUTEX_WAIT_REQUEUE_PI", " val=0x%08llx", " utime=0x%08llx", " uaddr2=0x%08llx", " val3=0x%08llx" },
- { "FUTEX_CMP_REQUEUE_PI", " val=%llu", " val2=%llu", " uaddr2=0x%08llx", " val3=0x%08llx" },
-};
-
-
-static void futex_print(struct trace_seq *s, const struct futex_args *args,
- const struct futex_op *fop)
-{
- trace_seq_printf(s, " uaddr=0x%08llx", args->uaddr);
-
- if (fop->fmt_val)
- trace_seq_printf(s, fop->fmt_val, args->val);
-
- if (fop->fmt_utime)
- trace_seq_printf(s,fop->fmt_utime, args->utime);
-
- if (fop->fmt_uaddr2)
- trace_seq_printf(s, fop->fmt_uaddr2, args->uaddr2);
-
- if (fop->fmt_val3)
- trace_seq_printf(s, fop->fmt_val3, args->val3);
-}
-
-static int futex_handler(struct trace_seq *s, struct tep_record *record,
- struct tep_event *event, void *context)
-{
- const struct futex_op *fop;
- struct futex_args args;
- unsigned long long cmd;
-
- if (tep_get_field_val(s, event, "uaddr", record, &args.uaddr, 1))
- return 1;
-
- if (tep_get_field_val(s, event, "op", record, &args.op, 1))
- return 1;
-
- if (tep_get_field_val(s, event, "val", record, &args.val, 1))
- return 1;
-
- if (tep_get_field_val(s, event, "utime", record, &args.utime, 1))
- return 1;
-
- if (tep_get_field_val(s, event, "uaddr2", record, &args.uaddr2, 1))
- return 1;
-
- if (tep_get_field_val(s, event, "val3", record, &args.val3, 1))
- return 1;
-
- cmd = args.op & FUTEX_CMD_MASK;
- if (cmd >= ARRAY_SIZE(futex_op_tbl))
- return 1;
-
- fop = &futex_op_tbl[cmd];
-
- trace_seq_printf(s, "op=%s", fop->name);
-
- if (args.op & FUTEX_PRIVATE_FLAG)
- trace_seq_puts(s, "|FUTEX_PRIVATE_FLAG");
-
- if (args.op & FUTEX_CLOCK_REALTIME)
- trace_seq_puts(s, "|FUTEX_CLOCK_REALTIME");
-
- futex_print(s, &args, fop);
- return 0;
-}
-
-int TEP_PLUGIN_LOADER(struct tep_handle *tep)
-{
- tep_register_event_handler(tep, -1, "syscalls", "sys_enter_futex",
- futex_handler, NULL);
- return 0;
-}
-
-void TEP_PLUGIN_UNLOADER(struct tep_handle *tep)
-{
- tep_unregister_event_handler(tep, -1, "syscalls", "sys_enter_futex",
- futex_handler, NULL);
-}
diff --git a/tools/lib/traceevent/plugins/plugin_hrtimer.c b/tools/lib/traceevent/plugins/plugin_hrtimer.c
deleted file mode 100644
index d984667..0000000
--- a/tools/lib/traceevent/plugins/plugin_hrtimer.c
+++ /dev/null
@@ -1,74 +0,0 @@
-// SPDX-License-Identifier: LGPL-2.1
-/*
- * Copyright (C) 2009 Red Hat Inc, Steven Rostedt <srostedt@redhat.com>
- * Copyright (C) 2009 Johannes Berg <johannes@sipsolutions.net>
- */
-#include <stdio.h>
-#include <stdlib.h>
-#include <string.h>
-
-#include "event-parse.h"
-#include "trace-seq.h"
-
-static int timer_expire_handler(struct trace_seq *s,
- struct tep_record *record,
- struct tep_event *event, void *context)
-{
- trace_seq_printf(s, "hrtimer=");
-
- if (tep_print_num_field(s, "0x%llx", event, "timer",
- record, 0) == -1)
- tep_print_num_field(s, "0x%llx", event, "hrtimer",
- record, 1);
-
- trace_seq_printf(s, " now=");
-
- tep_print_num_field(s, "%llu", event, "now", record, 1);
-
- tep_print_func_field(s, " function=%s", event, "function",
- record, 0);
- return 0;
-}
-
-static int timer_start_handler(struct trace_seq *s,
- struct tep_record *record,
- struct tep_event *event, void *context)
-{
- trace_seq_printf(s, "hrtimer=");
-
- if (tep_print_num_field(s, "0x%llx", event, "timer",
- record, 0) == -1)
- tep_print_num_field(s, "0x%llx", event, "hrtimer",
- record, 1);
-
- tep_print_func_field(s, " function=%s", event, "function",
- record, 0);
-
- trace_seq_printf(s, " expires=");
- tep_print_num_field(s, "%llu", event, "expires", record, 1);
-
- trace_seq_printf(s, " softexpires=");
- tep_print_num_field(s, "%llu", event, "softexpires", record, 1);
- return 0;
-}
-
-int TEP_PLUGIN_LOADER(struct tep_handle *tep)
-{
- tep_register_event_handler(tep, -1,
- "timer", "hrtimer_expire_entry",
- timer_expire_handler, NULL);
-
- tep_register_event_handler(tep, -1, "timer", "hrtimer_start",
- timer_start_handler, NULL);
- return 0;
-}
-
-void TEP_PLUGIN_UNLOADER(struct tep_handle *tep)
-{
- tep_unregister_event_handler(tep, -1,
- "timer", "hrtimer_expire_entry",
- timer_expire_handler, NULL);
-
- tep_unregister_event_handler(tep, -1, "timer", "hrtimer_start",
- timer_start_handler, NULL);
-}
diff --git a/tools/lib/traceevent/plugins/plugin_jbd2.c b/tools/lib/traceevent/plugins/plugin_jbd2.c
deleted file mode 100644
index 69111a6..0000000
--- a/tools/lib/traceevent/plugins/plugin_jbd2.c
+++ /dev/null
@@ -1,61 +0,0 @@
-// SPDX-License-Identifier: LGPL-2.1
-/*
- * Copyright (C) 2010 Red Hat Inc, Steven Rostedt <srostedt@redhat.com>
- */
-#include <stdio.h>
-#include <stdlib.h>
-#include <string.h>
-
-#include "event-parse.h"
-#include "trace-seq.h"
-
-#define MINORBITS 20
-#define MINORMASK ((1U << MINORBITS) - 1)
-
-#define MAJOR(dev) ((unsigned int) ((dev) >> MINORBITS))
-#define MINOR(dev) ((unsigned int) ((dev) & MINORMASK))
-
-static unsigned long long
-process_jbd2_dev_to_name(struct trace_seq *s, unsigned long long *args)
-{
- unsigned int dev = args[0];
-
- trace_seq_printf(s, "%d:%d", MAJOR(dev), MINOR(dev));
- return 0;
-}
-
-static unsigned long long
-process_jiffies_to_msecs(struct trace_seq *s, unsigned long long *args)
-{
- unsigned long long jiffies = args[0];
-
- trace_seq_printf(s, "%lld", jiffies);
- return jiffies;
-}
-
-int TEP_PLUGIN_LOADER(struct tep_handle *tep)
-{
- tep_register_print_function(tep,
- process_jbd2_dev_to_name,
- TEP_FUNC_ARG_STRING,
- "jbd2_dev_to_name",
- TEP_FUNC_ARG_INT,
- TEP_FUNC_ARG_VOID);
-
- tep_register_print_function(tep,
- process_jiffies_to_msecs,
- TEP_FUNC_ARG_LONG,
- "jiffies_to_msecs",
- TEP_FUNC_ARG_LONG,
- TEP_FUNC_ARG_VOID);
- return 0;
-}
-
-void TEP_PLUGIN_UNLOADER(struct tep_handle *tep)
-{
- tep_unregister_print_function(tep, process_jbd2_dev_to_name,
- "jbd2_dev_to_name");
-
- tep_unregister_print_function(tep, process_jiffies_to_msecs,
- "jiffies_to_msecs");
-}
diff --git a/tools/lib/traceevent/plugins/plugin_kmem.c b/tools/lib/traceevent/plugins/plugin_kmem.c
deleted file mode 100644
index 4b4f7f9..0000000
--- a/tools/lib/traceevent/plugins/plugin_kmem.c
+++ /dev/null
@@ -1,80 +0,0 @@
-// SPDX-License-Identifier: LGPL-2.1
-/*
- * Copyright (C) 2009 Red Hat Inc, Steven Rostedt <srostedt@redhat.com>
- */
-#include <stdio.h>
-#include <stdlib.h>
-#include <string.h>
-
-#include "event-parse.h"
-#include "trace-seq.h"
-
-static int call_site_handler(struct trace_seq *s, struct tep_record *record,
- struct tep_event *event, void *context)
-{
- struct tep_format_field *field;
- unsigned long long val, addr;
- void *data = record->data;
- const char *func;
-
- field = tep_find_field(event, "call_site");
- if (!field)
- return 1;
-
- if (tep_read_number_field(field, data, &val))
- return 1;
-
- func = tep_find_function(event->tep, val);
- if (!func)
- return 1;
-
- addr = tep_find_function_address(event->tep, val);
-
- trace_seq_printf(s, "(%s+0x%x) ", func, (int)(val - addr));
- return 1;
-}
-
-int TEP_PLUGIN_LOADER(struct tep_handle *tep)
-{
- tep_register_event_handler(tep, -1, "kmem", "kfree",
- call_site_handler, NULL);
-
- tep_register_event_handler(tep, -1, "kmem", "kmalloc",
- call_site_handler, NULL);
-
- tep_register_event_handler(tep, -1, "kmem", "kmalloc_node",
- call_site_handler, NULL);
-
- tep_register_event_handler(tep, -1, "kmem", "kmem_cache_alloc",
- call_site_handler, NULL);
-
- tep_register_event_handler(tep, -1, "kmem",
- "kmem_cache_alloc_node",
- call_site_handler, NULL);
-
- tep_register_event_handler(tep, -1, "kmem", "kmem_cache_free",
- call_site_handler, NULL);
- return 0;
-}
-
-void TEP_PLUGIN_UNLOADER(struct tep_handle *tep)
-{
- tep_unregister_event_handler(tep, -1, "kmem", "kfree",
- call_site_handler, NULL);
-
- tep_unregister_event_handler(tep, -1, "kmem", "kmalloc",
- call_site_handler, NULL);
-
- tep_unregister_event_handler(tep, -1, "kmem", "kmalloc_node",
- call_site_handler, NULL);
-
- tep_unregister_event_handler(tep, -1, "kmem", "kmem_cache_alloc",
- call_site_handler, NULL);
-
- tep_unregister_event_handler(tep, -1, "kmem",
- "kmem_cache_alloc_node",
- call_site_handler, NULL);
-
- tep_unregister_event_handler(tep, -1, "kmem", "kmem_cache_free",
- call_site_handler, NULL);
-}
diff --git a/tools/lib/traceevent/plugins/plugin_kvm.c b/tools/lib/traceevent/plugins/plugin_kvm.c
deleted file mode 100644
index 9ce7b4b..0000000
--- a/tools/lib/traceevent/plugins/plugin_kvm.c
+++ /dev/null
@@ -1,527 +0,0 @@
-// SPDX-License-Identifier: LGPL-2.1
-/*
- * Copyright (C) 2009 Red Hat Inc, Steven Rostedt <srostedt@redhat.com>
- */
-#include <stdio.h>
-#include <stdlib.h>
-#include <string.h>
-#include <stdint.h>
-
-#include "event-parse.h"
-#include "trace-seq.h"
-
-#ifdef HAVE_UDIS86
-
-#include <udis86.h>
-
-static ud_t ud;
-
-static void init_disassembler(void)
-{
- ud_init(&ud);
- ud_set_syntax(&ud, UD_SYN_ATT);
-}
-
-static const char *disassemble(unsigned char *insn, int len, uint64_t rip,
- int cr0_pe, int eflags_vm,
- int cs_d, int cs_l)
-{
- int mode;
-
- if (!cr0_pe)
- mode = 16;
- else if (eflags_vm)
- mode = 16;
- else if (cs_l)
- mode = 64;
- else if (cs_d)
- mode = 32;
- else
- mode = 16;
-
- ud_set_pc(&ud, rip);
- ud_set_mode(&ud, mode);
- ud_set_input_buffer(&ud, insn, len);
- ud_disassemble(&ud);
- return ud_insn_asm(&ud);
-}
-
-#else
-
-static void init_disassembler(void)
-{
-}
-
-static const char *disassemble(unsigned char *insn, int len, uint64_t rip,
- int cr0_pe, int eflags_vm,
- int cs_d, int cs_l)
-{
- static char out[15*3+1];
- int i;
-
- for (i = 0; i < len; ++i)
- sprintf(out + i * 3, "%02x ", insn[i]);
- out[len*3-1] = '\0';
- return out;
-}
-
-#endif
-
-
-#define VMX_EXIT_REASONS \
- _ER(EXCEPTION_NMI, 0) \
- _ER(EXTERNAL_INTERRUPT, 1) \
- _ER(TRIPLE_FAULT, 2) \
- _ER(PENDING_INTERRUPT, 7) \
- _ER(NMI_WINDOW, 8) \
- _ER(TASK_SWITCH, 9) \
- _ER(CPUID, 10) \
- _ER(HLT, 12) \
- _ER(INVD, 13) \
- _ER(INVLPG, 14) \
- _ER(RDPMC, 15) \
- _ER(RDTSC, 16) \
- _ER(VMCALL, 18) \
- _ER(VMCLEAR, 19) \
- _ER(VMLAUNCH, 20) \
- _ER(VMPTRLD, 21) \
- _ER(VMPTRST, 22) \
- _ER(VMREAD, 23) \
- _ER(VMRESUME, 24) \
- _ER(VMWRITE, 25) \
- _ER(VMOFF, 26) \
- _ER(VMON, 27) \
- _ER(CR_ACCESS, 28) \
- _ER(DR_ACCESS, 29) \
- _ER(IO_INSTRUCTION, 30) \
- _ER(MSR_READ, 31) \
- _ER(MSR_WRITE, 32) \
- _ER(MWAIT_INSTRUCTION, 36) \
- _ER(MONITOR_INSTRUCTION, 39) \
- _ER(PAUSE_INSTRUCTION, 40) \
- _ER(MCE_DURING_VMENTRY, 41) \
- _ER(TPR_BELOW_THRESHOLD, 43) \
- _ER(APIC_ACCESS, 44) \
- _ER(EOI_INDUCED, 45) \
- _ER(EPT_VIOLATION, 48) \
- _ER(EPT_MISCONFIG, 49) \
- _ER(INVEPT, 50) \
- _ER(PREEMPTION_TIMER, 52) \
- _ER(WBINVD, 54) \
- _ER(XSETBV, 55) \
- _ER(APIC_WRITE, 56) \
- _ER(INVPCID, 58) \
- _ER(PML_FULL, 62) \
- _ER(XSAVES, 63) \
- _ER(XRSTORS, 64)
-
-#define SVM_EXIT_REASONS \
- _ER(EXIT_READ_CR0, 0x000) \
- _ER(EXIT_READ_CR3, 0x003) \
- _ER(EXIT_READ_CR4, 0x004) \
- _ER(EXIT_READ_CR8, 0x008) \
- _ER(EXIT_WRITE_CR0, 0x010) \
- _ER(EXIT_WRITE_CR3, 0x013) \
- _ER(EXIT_WRITE_CR4, 0x014) \
- _ER(EXIT_WRITE_CR8, 0x018) \
- _ER(EXIT_READ_DR0, 0x020) \
- _ER(EXIT_READ_DR1, 0x021) \
- _ER(EXIT_READ_DR2, 0x022) \
- _ER(EXIT_READ_DR3, 0x023) \
- _ER(EXIT_READ_DR4, 0x024) \
- _ER(EXIT_READ_DR5, 0x025) \
- _ER(EXIT_READ_DR6, 0x026) \
- _ER(EXIT_READ_DR7, 0x027) \
- _ER(EXIT_WRITE_DR0, 0x030) \
- _ER(EXIT_WRITE_DR1, 0x031) \
- _ER(EXIT_WRITE_DR2, 0x032) \
- _ER(EXIT_WRITE_DR3, 0x033) \
- _ER(EXIT_WRITE_DR4, 0x034) \
- _ER(EXIT_WRITE_DR5, 0x035) \
- _ER(EXIT_WRITE_DR6, 0x036) \
- _ER(EXIT_WRITE_DR7, 0x037) \
- _ER(EXIT_EXCP_DE, 0x040) \
- _ER(EXIT_EXCP_DB, 0x041) \
- _ER(EXIT_EXCP_BP, 0x043) \
- _ER(EXIT_EXCP_OF, 0x044) \
- _ER(EXIT_EXCP_BR, 0x045) \
- _ER(EXIT_EXCP_UD, 0x046) \
- _ER(EXIT_EXCP_NM, 0x047) \
- _ER(EXIT_EXCP_DF, 0x048) \
- _ER(EXIT_EXCP_TS, 0x04a) \
- _ER(EXIT_EXCP_NP, 0x04b) \
- _ER(EXIT_EXCP_SS, 0x04c) \
- _ER(EXIT_EXCP_GP, 0x04d) \
- _ER(EXIT_EXCP_PF, 0x04e) \
- _ER(EXIT_EXCP_MF, 0x050) \
- _ER(EXIT_EXCP_AC, 0x051) \
- _ER(EXIT_EXCP_MC, 0x052) \
- _ER(EXIT_EXCP_XF, 0x053) \
- _ER(EXIT_INTR, 0x060) \
- _ER(EXIT_NMI, 0x061) \
- _ER(EXIT_SMI, 0x062) \
- _ER(EXIT_INIT, 0x063) \
- _ER(EXIT_VINTR, 0x064) \
- _ER(EXIT_CR0_SEL_WRITE, 0x065) \
- _ER(EXIT_IDTR_READ, 0x066) \
- _ER(EXIT_GDTR_READ, 0x067) \
- _ER(EXIT_LDTR_READ, 0x068) \
- _ER(EXIT_TR_READ, 0x069) \
- _ER(EXIT_IDTR_WRITE, 0x06a) \
- _ER(EXIT_GDTR_WRITE, 0x06b) \
- _ER(EXIT_LDTR_WRITE, 0x06c) \
- _ER(EXIT_TR_WRITE, 0x06d) \
- _ER(EXIT_RDTSC, 0x06e) \
- _ER(EXIT_RDPMC, 0x06f) \
- _ER(EXIT_PUSHF, 0x070) \
- _ER(EXIT_POPF, 0x071) \
- _ER(EXIT_CPUID, 0x072) \
- _ER(EXIT_RSM, 0x073) \
- _ER(EXIT_IRET, 0x074) \
- _ER(EXIT_SWINT, 0x075) \
- _ER(EXIT_INVD, 0x076) \
- _ER(EXIT_PAUSE, 0x077) \
- _ER(EXIT_HLT, 0x078) \
- _ER(EXIT_INVLPG, 0x079) \
- _ER(EXIT_INVLPGA, 0x07a) \
- _ER(EXIT_IOIO, 0x07b) \
- _ER(EXIT_MSR, 0x07c) \
- _ER(EXIT_TASK_SWITCH, 0x07d) \
- _ER(EXIT_FERR_FREEZE, 0x07e) \
- _ER(EXIT_SHUTDOWN, 0x07f) \
- _ER(EXIT_VMRUN, 0x080) \
- _ER(EXIT_VMMCALL, 0x081) \
- _ER(EXIT_VMLOAD, 0x082) \
- _ER(EXIT_VMSAVE, 0x083) \
- _ER(EXIT_STGI, 0x084) \
- _ER(EXIT_CLGI, 0x085) \
- _ER(EXIT_SKINIT, 0x086) \
- _ER(EXIT_RDTSCP, 0x087) \
- _ER(EXIT_ICEBP, 0x088) \
- _ER(EXIT_WBINVD, 0x089) \
- _ER(EXIT_MONITOR, 0x08a) \
- _ER(EXIT_MWAIT, 0x08b) \
- _ER(EXIT_MWAIT_COND, 0x08c) \
- _ER(EXIT_XSETBV, 0x08d) \
- _ER(EXIT_NPF, 0x400) \
- _ER(EXIT_AVIC_INCOMPLETE_IPI, 0x401) \
- _ER(EXIT_AVIC_UNACCELERATED_ACCESS, 0x402) \
- _ER(EXIT_ERR, -1)
-
-#define _ER(reason, val) { #reason, val },
-struct str_values {
- const char *str;
- int val;
-};
-
-static struct str_values vmx_exit_reasons[] = {
- VMX_EXIT_REASONS
- { NULL, -1}
-};
-
-static struct str_values svm_exit_reasons[] = {
- SVM_EXIT_REASONS
- { NULL, -1}
-};
-
-static struct isa_exit_reasons {
- unsigned isa;
- struct str_values *strings;
-} isa_exit_reasons[] = {
- { .isa = 1, .strings = vmx_exit_reasons },
- { .isa = 2, .strings = svm_exit_reasons },
- { }
-};
-
-static const char *find_exit_reason(unsigned isa, int val)
-{
- struct str_values *strings = NULL;
- int i;
-
- for (i = 0; isa_exit_reasons[i].strings; ++i)
- if (isa_exit_reasons[i].isa == isa) {
- strings = isa_exit_reasons[i].strings;
- break;
- }
- if (!strings)
- return "UNKNOWN-ISA";
- for (i = 0; strings[i].str; i++)
- if (strings[i].val == val)
- break;
-
- return strings[i].str;
-}
-
-static int print_exit_reason(struct trace_seq *s, struct tep_record *record,
- struct tep_event *event, const char *field)
-{
- unsigned long long isa;
- unsigned long long val;
- const char *reason;
-
- if (tep_get_field_val(s, event, field, record, &val, 1) < 0)
- return -1;
-
- if (tep_get_field_val(s, event, "isa", record, &isa, 0) < 0)
- isa = 1;
-
- reason = find_exit_reason(isa, val);
- if (reason)
- trace_seq_printf(s, "reason %s", reason);
- else
- trace_seq_printf(s, "reason UNKNOWN (%llu)", val);
- return 0;
-}
-
-static int kvm_exit_handler(struct trace_seq *s, struct tep_record *record,
- struct tep_event *event, void *context)
-{
- unsigned long long info1 = 0, info2 = 0;
-
- if (print_exit_reason(s, record, event, "exit_reason") < 0)
- return -1;
-
- tep_print_num_field(s, " rip 0x%lx", event, "guest_rip", record, 1);
-
- if (tep_get_field_val(s, event, "info1", record, &info1, 0) >= 0
- && tep_get_field_val(s, event, "info2", record, &info2, 0) >= 0)
- trace_seq_printf(s, " info %llx %llx", info1, info2);
-
- return 0;
-}
-
-#define KVM_EMUL_INSN_F_CR0_PE (1 << 0)
-#define KVM_EMUL_INSN_F_EFL_VM (1 << 1)
-#define KVM_EMUL_INSN_F_CS_D (1 << 2)
-#define KVM_EMUL_INSN_F_CS_L (1 << 3)
-
-static int kvm_emulate_insn_handler(struct trace_seq *s,
- struct tep_record *record,
- struct tep_event *event, void *context)
-{
- unsigned long long rip, csbase, len, flags, failed;
- int llen;
- uint8_t *insn;
- const char *disasm;
-
- if (tep_get_field_val(s, event, "rip", record, &rip, 1) < 0)
- return -1;
-
- if (tep_get_field_val(s, event, "csbase", record, &csbase, 1) < 0)
- return -1;
-
- if (tep_get_field_val(s, event, "len", record, &len, 1) < 0)
- return -1;
-
- if (tep_get_field_val(s, event, "flags", record, &flags, 1) < 0)
- return -1;
-
- if (tep_get_field_val(s, event, "failed", record, &failed, 1) < 0)
- return -1;
-
- insn = tep_get_field_raw(s, event, "insn", record, &llen, 1);
- if (!insn)
- return -1;
-
- disasm = disassemble(insn, len, rip,
- flags & KVM_EMUL_INSN_F_CR0_PE,
- flags & KVM_EMUL_INSN_F_EFL_VM,
- flags & KVM_EMUL_INSN_F_CS_D,
- flags & KVM_EMUL_INSN_F_CS_L);
-
- trace_seq_printf(s, "%llx:%llx: %s%s", csbase, rip, disasm,
- failed ? " FAIL" : "");
- return 0;
-}
-
-
-static int kvm_nested_vmexit_inject_handler(struct trace_seq *s, struct tep_record *record,
- struct tep_event *event, void *context)
-{
- if (print_exit_reason(s, record, event, "exit_code") < 0)
- return -1;
-
- tep_print_num_field(s, " info1 %llx", event, "exit_info1", record, 1);
- tep_print_num_field(s, " info2 %llx", event, "exit_info2", record, 1);
- tep_print_num_field(s, " int_info %llx", event, "exit_int_info", record, 1);
- tep_print_num_field(s, " int_info_err %llx", event, "exit_int_info_err", record, 1);
-
- return 0;
-}
-
-static int kvm_nested_vmexit_handler(struct trace_seq *s, struct tep_record *record,
- struct tep_event *event, void *context)
-{
- tep_print_num_field(s, "rip %llx ", event, "rip", record, 1);
-
- return kvm_nested_vmexit_inject_handler(s, record, event, context);
-}
-
-union kvm_mmu_page_role {
- unsigned word;
- struct {
- unsigned level:4;
- unsigned cr4_pae:1;
- unsigned quadrant:2;
- unsigned direct:1;
- unsigned access:3;
- unsigned invalid:1;
- unsigned efer_nx:1;
- unsigned cr0_wp:1;
- unsigned smep_and_not_wp:1;
- unsigned smap_and_not_wp:1;
- unsigned pad_for_nice_hex_output:8;
- unsigned smm:8;
- };
-};
-
-static int kvm_mmu_print_role(struct trace_seq *s, struct tep_record *record,
- struct tep_event *event, void *context)
-{
- unsigned long long val;
- static const char *access_str[] = {
- "---", "--x", "w--", "w-x", "-u-", "-ux", "wu-", "wux"
- };
- union kvm_mmu_page_role role;
-
- if (tep_get_field_val(s, event, "role", record, &val, 1) < 0)
- return -1;
-
- role.word = (int)val;
-
- /*
- * We can only use the structure if file is of the same
- * endianness.
- */
- if (tep_is_file_bigendian(event->tep) ==
- tep_is_local_bigendian(event->tep)) {
-
- trace_seq_printf(s, "%u q%u%s %s%s %spae %snxe %swp%s%s%s",
- role.level,
- role.quadrant,
- role.direct ? " direct" : "",
- access_str[role.access],
- role.invalid ? " invalid" : "",
- role.cr4_pae ? "" : "!",
- role.efer_nx ? "" : "!",
- role.cr0_wp ? "" : "!",
- role.smep_and_not_wp ? " smep" : "",
- role.smap_and_not_wp ? " smap" : "",
- role.smm ? " smm" : "");
- } else
- trace_seq_printf(s, "WORD: %08x", role.word);
-
- tep_print_num_field(s, " root %u ", event,
- "root_count", record, 1);
-
- if (tep_get_field_val(s, event, "unsync", record, &val, 1) < 0)
- return -1;
-
- trace_seq_printf(s, "%s%c", val ? "unsync" : "sync", 0);
- return 0;
-}
-
-static int kvm_mmu_get_page_handler(struct trace_seq *s,
- struct tep_record *record,
- struct tep_event *event, void *context)
-{
- unsigned long long val;
-
- if (tep_get_field_val(s, event, "created", record, &val, 1) < 0)
- return -1;
-
- trace_seq_printf(s, "%s ", val ? "new" : "existing");
-
- if (tep_get_field_val(s, event, "gfn", record, &val, 1) < 0)
- return -1;
-
- trace_seq_printf(s, "sp gfn %llx ", val);
- return kvm_mmu_print_role(s, record, event, context);
-}
-
-#define PT_WRITABLE_SHIFT 1
-#define PT_WRITABLE_MASK (1ULL << PT_WRITABLE_SHIFT)
-
-static unsigned long long
-process_is_writable_pte(struct trace_seq *s, unsigned long long *args)
-{
- unsigned long pte = args[0];
- return pte & PT_WRITABLE_MASK;
-}
-
-int TEP_PLUGIN_LOADER(struct tep_handle *tep)
-{
- init_disassembler();
-
- tep_register_event_handler(tep, -1, "kvm", "kvm_exit",
- kvm_exit_handler, NULL);
-
- tep_register_event_handler(tep, -1, "kvm", "kvm_emulate_insn",
- kvm_emulate_insn_handler, NULL);
-
- tep_register_event_handler(tep, -1, "kvm", "kvm_nested_vmexit",
- kvm_nested_vmexit_handler, NULL);
-
- tep_register_event_handler(tep, -1, "kvm", "kvm_nested_vmexit_inject",
- kvm_nested_vmexit_inject_handler, NULL);
-
- tep_register_event_handler(tep, -1, "kvmmmu", "kvm_mmu_get_page",
- kvm_mmu_get_page_handler, NULL);
-
- tep_register_event_handler(tep, -1, "kvmmmu", "kvm_mmu_sync_page",
- kvm_mmu_print_role, NULL);
-
- tep_register_event_handler(tep, -1,
- "kvmmmu", "kvm_mmu_unsync_page",
- kvm_mmu_print_role, NULL);
-
- tep_register_event_handler(tep, -1, "kvmmmu", "kvm_mmu_zap_page",
- kvm_mmu_print_role, NULL);
-
- tep_register_event_handler(tep, -1, "kvmmmu",
- "kvm_mmu_prepare_zap_page", kvm_mmu_print_role,
- NULL);
-
- tep_register_print_function(tep,
- process_is_writable_pte,
- TEP_FUNC_ARG_INT,
- "is_writable_pte",
- TEP_FUNC_ARG_LONG,
- TEP_FUNC_ARG_VOID);
- return 0;
-}
-
-void TEP_PLUGIN_UNLOADER(struct tep_handle *tep)
-{
- tep_unregister_event_handler(tep, -1, "kvm", "kvm_exit",
- kvm_exit_handler, NULL);
-
- tep_unregister_event_handler(tep, -1, "kvm", "kvm_emulate_insn",
- kvm_emulate_insn_handler, NULL);
-
- tep_unregister_event_handler(tep, -1, "kvm", "kvm_nested_vmexit",
- kvm_nested_vmexit_handler, NULL);
-
- tep_unregister_event_handler(tep, -1, "kvm", "kvm_nested_vmexit_inject",
- kvm_nested_vmexit_inject_handler, NULL);
-
- tep_unregister_event_handler(tep, -1, "kvmmmu", "kvm_mmu_get_page",
- kvm_mmu_get_page_handler, NULL);
-
- tep_unregister_event_handler(tep, -1, "kvmmmu", "kvm_mmu_sync_page",
- kvm_mmu_print_role, NULL);
-
- tep_unregister_event_handler(tep, -1,
- "kvmmmu", "kvm_mmu_unsync_page",
- kvm_mmu_print_role, NULL);
-
- tep_unregister_event_handler(tep, -1, "kvmmmu", "kvm_mmu_zap_page",
- kvm_mmu_print_role, NULL);
-
- tep_unregister_event_handler(tep, -1, "kvmmmu",
- "kvm_mmu_prepare_zap_page", kvm_mmu_print_role,
- NULL);
-
- tep_unregister_print_function(tep, process_is_writable_pte,
- "is_writable_pte");
-}
diff --git a/tools/lib/traceevent/plugins/plugin_mac80211.c b/tools/lib/traceevent/plugins/plugin_mac80211.c
deleted file mode 100644
index f48071e..0000000
--- a/tools/lib/traceevent/plugins/plugin_mac80211.c
+++ /dev/null
@@ -1,88 +0,0 @@
-// SPDX-License-Identifier: LGPL-2.1
-/*
- * Copyright (C) 2009 Johannes Berg <johannes@sipsolutions.net>
- */
-#include <stdio.h>
-#include <stdlib.h>
-#include <string.h>
-
-#include "event-parse.h"
-#include "trace-seq.h"
-
-#define INDENT 65
-
-static void print_string(struct trace_seq *s, struct tep_event *event,
- const char *name, const void *data)
-{
- struct tep_format_field *f = tep_find_field(event, name);
- int offset;
- int length;
-
- if (!f) {
- trace_seq_printf(s, "NOTFOUND:%s", name);
- return;
- }
-
- offset = f->offset;
- length = f->size;
-
- if (!strncmp(f->type, "__data_loc", 10)) {
- unsigned long long v;
- if (tep_read_number_field(f, data, &v)) {
- trace_seq_printf(s, "invalid_data_loc");
- return;
- }
- offset = v & 0xffff;
- length = v >> 16;
- }
-
- trace_seq_printf(s, "%.*s", length, (char *)data + offset);
-}
-
-#define SF(fn) tep_print_num_field(s, fn ":%d", event, fn, record, 0)
-#define SFX(fn) tep_print_num_field(s, fn ":%#x", event, fn, record, 0)
-#define SP() trace_seq_putc(s, ' ')
-
-static int drv_bss_info_changed(struct trace_seq *s,
- struct tep_record *record,
- struct tep_event *event, void *context)
-{
- void *data = record->data;
-
- print_string(s, event, "wiphy_name", data);
- trace_seq_printf(s, " vif:");
- print_string(s, event, "vif_name", data);
- tep_print_num_field(s, "(%d)", event, "vif_type", record, 1);
-
- trace_seq_printf(s, "\n%*s", INDENT, "");
- SF("assoc"); SP();
- SF("aid"); SP();
- SF("cts"); SP();
- SF("shortpre"); SP();
- SF("shortslot"); SP();
- SF("dtimper"); SP();
- trace_seq_printf(s, "\n%*s", INDENT, "");
- SF("bcnint"); SP();
- SFX("assoc_cap"); SP();
- SFX("basic_rates"); SP();
- SF("enable_beacon");
- trace_seq_printf(s, "\n%*s", INDENT, "");
- SF("ht_operation_mode");
-
- return 0;
-}
-
-int TEP_PLUGIN_LOADER(struct tep_handle *tep)
-{
- tep_register_event_handler(tep, -1, "mac80211",
- "drv_bss_info_changed",
- drv_bss_info_changed, NULL);
- return 0;
-}
-
-void TEP_PLUGIN_UNLOADER(struct tep_handle *tep)
-{
- tep_unregister_event_handler(tep, -1, "mac80211",
- "drv_bss_info_changed",
- drv_bss_info_changed, NULL);
-}
diff --git a/tools/lib/traceevent/plugins/plugin_sched_switch.c b/tools/lib/traceevent/plugins/plugin_sched_switch.c
deleted file mode 100644
index e12fa10..0000000
--- a/tools/lib/traceevent/plugins/plugin_sched_switch.c
+++ /dev/null
@@ -1,146 +0,0 @@
-// SPDX-License-Identifier: LGPL-2.1
-/*
- * Copyright (C) 2009, 2010 Red Hat Inc, Steven Rostedt <srostedt@redhat.com>
- */
-#include <stdio.h>
-#include <stdlib.h>
-#include <string.h>
-
-#include "event-parse.h"
-#include "trace-seq.h"
-
-static void write_state(struct trace_seq *s, int val)
-{
- const char states[] = "SDTtZXxW";
- int found = 0;
- int i;
-
- for (i = 0; i < (sizeof(states) - 1); i++) {
- if (!(val & (1 << i)))
- continue;
-
- if (found)
- trace_seq_putc(s, '|');
-
- found = 1;
- trace_seq_putc(s, states[i]);
- }
-
- if (!found)
- trace_seq_putc(s, 'R');
-}
-
-static void write_and_save_comm(struct tep_format_field *field,
- struct tep_record *record,
- struct trace_seq *s, int pid)
-{
- const char *comm;
- int len;
-
- comm = (char *)(record->data + field->offset);
- len = s->len;
- trace_seq_printf(s, "%.*s",
- field->size, comm);
-
- /* make sure the comm has a \0 at the end. */
- trace_seq_terminate(s);
- comm = &s->buffer[len];
-
- /* Help out the comm to ids. This will handle dups */
- tep_register_comm(field->event->tep, comm, pid);
-}
-
-static int sched_wakeup_handler(struct trace_seq *s,
- struct tep_record *record,
- struct tep_event *event, void *context)
-{
- struct tep_format_field *field;
- unsigned long long val;
-
- if (tep_get_field_val(s, event, "pid", record, &val, 1))
- return trace_seq_putc(s, '!');
-
- field = tep_find_any_field(event, "comm");
- if (field) {
- write_and_save_comm(field, record, s, val);
- trace_seq_putc(s, ':');
- }
- trace_seq_printf(s, "%lld", val);
-
- if (tep_get_field_val(s, event, "prio", record, &val, 0) == 0)
- trace_seq_printf(s, " [%lld]", val);
-
- if (tep_get_field_val(s, event, "success", record, &val, 1) == 0)
- trace_seq_printf(s, " success=%lld", val);
-
- if (tep_get_field_val(s, event, "target_cpu", record, &val, 0) == 0)
- trace_seq_printf(s, " CPU:%03llu", val);
-
- return 0;
-}
-
-static int sched_switch_handler(struct trace_seq *s,
- struct tep_record *record,
- struct tep_event *event, void *context)
-{
- struct tep_format_field *field;
- unsigned long long val;
-
- if (tep_get_field_val(s, event, "prev_pid", record, &val, 1))
- return trace_seq_putc(s, '!');
-
- field = tep_find_any_field(event, "prev_comm");
- if (field) {
- write_and_save_comm(field, record, s, val);
- trace_seq_putc(s, ':');
- }
- trace_seq_printf(s, "%lld ", val);
-
- if (tep_get_field_val(s, event, "prev_prio", record, &val, 0) == 0)
- trace_seq_printf(s, "[%d] ", (int) val);
-
- if (tep_get_field_val(s, event, "prev_state", record, &val, 0) == 0)
- write_state(s, val);
-
- trace_seq_puts(s, " ==> ");
-
- if (tep_get_field_val(s, event, "next_pid", record, &val, 1))
- return trace_seq_putc(s, '!');
-
- field = tep_find_any_field(event, "next_comm");
- if (field) {
- write_and_save_comm(field, record, s, val);
- trace_seq_putc(s, ':');
- }
- trace_seq_printf(s, "%lld", val);
-
- if (tep_get_field_val(s, event, "next_prio", record, &val, 0) == 0)
- trace_seq_printf(s, " [%d]", (int) val);
-
- return 0;
-}
-
-int TEP_PLUGIN_LOADER(struct tep_handle *tep)
-{
- tep_register_event_handler(tep, -1, "sched", "sched_switch",
- sched_switch_handler, NULL);
-
- tep_register_event_handler(tep, -1, "sched", "sched_wakeup",
- sched_wakeup_handler, NULL);
-
- tep_register_event_handler(tep, -1, "sched", "sched_wakeup_new",
- sched_wakeup_handler, NULL);
- return 0;
-}
-
-void TEP_PLUGIN_UNLOADER(struct tep_handle *tep)
-{
- tep_unregister_event_handler(tep, -1, "sched", "sched_switch",
- sched_switch_handler, NULL);
-
- tep_unregister_event_handler(tep, -1, "sched", "sched_wakeup",
- sched_wakeup_handler, NULL);
-
- tep_unregister_event_handler(tep, -1, "sched", "sched_wakeup_new",
- sched_wakeup_handler, NULL);
-}
diff --git a/tools/lib/traceevent/plugins/plugin_scsi.c b/tools/lib/traceevent/plugins/plugin_scsi.c
deleted file mode 100644
index 5d0387a..0000000
--- a/tools/lib/traceevent/plugins/plugin_scsi.c
+++ /dev/null
@@ -1,434 +0,0 @@
-// SPDX-License-Identifier: GPL-2.0
-#include <stdio.h>
-#include <string.h>
-#include <inttypes.h>
-#include "event-parse.h"
-#include "trace-seq.h"
-
-typedef unsigned long sector_t;
-typedef uint64_t u64;
-typedef unsigned int u32;
-
-/*
- * SCSI opcodes
- */
-#define TEST_UNIT_READY 0x00
-#define REZERO_UNIT 0x01
-#define REQUEST_SENSE 0x03
-#define FORMAT_UNIT 0x04
-#define READ_BLOCK_LIMITS 0x05
-#define REASSIGN_BLOCKS 0x07
-#define INITIALIZE_ELEMENT_STATUS 0x07
-#define READ_6 0x08
-#define WRITE_6 0x0a
-#define SEEK_6 0x0b
-#define READ_REVERSE 0x0f
-#define WRITE_FILEMARKS 0x10
-#define SPACE 0x11
-#define INQUIRY 0x12
-#define RECOVER_BUFFERED_DATA 0x14
-#define MODE_SELECT 0x15
-#define RESERVE 0x16
-#define RELEASE 0x17
-#define COPY 0x18
-#define ERASE 0x19
-#define MODE_SENSE 0x1a
-#define START_STOP 0x1b
-#define RECEIVE_DIAGNOSTIC 0x1c
-#define SEND_DIAGNOSTIC 0x1d
-#define ALLOW_MEDIUM_REMOVAL 0x1e
-
-#define READ_FORMAT_CAPACITIES 0x23
-#define SET_WINDOW 0x24
-#define READ_CAPACITY 0x25
-#define READ_10 0x28
-#define WRITE_10 0x2a
-#define SEEK_10 0x2b
-#define POSITION_TO_ELEMENT 0x2b
-#define WRITE_VERIFY 0x2e
-#define VERIFY 0x2f
-#define SEARCH_HIGH 0x30
-#define SEARCH_EQUAL 0x31
-#define SEARCH_LOW 0x32
-#define SET_LIMITS 0x33
-#define PRE_FETCH 0x34
-#define READ_POSITION 0x34
-#define SYNCHRONIZE_CACHE 0x35
-#define LOCK_UNLOCK_CACHE 0x36
-#define READ_DEFECT_DATA 0x37
-#define MEDIUM_SCAN 0x38
-#define COMPARE 0x39
-#define COPY_VERIFY 0x3a
-#define WRITE_BUFFER 0x3b
-#define READ_BUFFER 0x3c
-#define UPDATE_BLOCK 0x3d
-#define READ_LONG 0x3e
-#define WRITE_LONG 0x3f
-#define CHANGE_DEFINITION 0x40
-#define WRITE_SAME 0x41
-#define UNMAP 0x42
-#define READ_TOC 0x43
-#define READ_HEADER 0x44
-#define GET_EVENT_STATUS_NOTIFICATION 0x4a
-#define LOG_SELECT 0x4c
-#define LOG_SENSE 0x4d
-#define XDWRITEREAD_10 0x53
-#define MODE_SELECT_10 0x55
-#define RESERVE_10 0x56
-#define RELEASE_10 0x57
-#define MODE_SENSE_10 0x5a
-#define PERSISTENT_RESERVE_IN 0x5e
-#define PERSISTENT_RESERVE_OUT 0x5f
-#define VARIABLE_LENGTH_CMD 0x7f
-#define REPORT_LUNS 0xa0
-#define SECURITY_PROTOCOL_IN 0xa2
-#define MAINTENANCE_IN 0xa3
-#define MAINTENANCE_OUT 0xa4
-#define MOVE_MEDIUM 0xa5
-#define EXCHANGE_MEDIUM 0xa6
-#define READ_12 0xa8
-#define SERVICE_ACTION_OUT_12 0xa9
-#define WRITE_12 0xaa
-#define SERVICE_ACTION_IN_12 0xab
-#define WRITE_VERIFY_12 0xae
-#define VERIFY_12 0xaf
-#define SEARCH_HIGH_12 0xb0
-#define SEARCH_EQUAL_12 0xb1
-#define SEARCH_LOW_12 0xb2
-#define SECURITY_PROTOCOL_OUT 0xb5
-#define READ_ELEMENT_STATUS 0xb8
-#define SEND_VOLUME_TAG 0xb6
-#define WRITE_LONG_2 0xea
-#define EXTENDED_COPY 0x83
-#define RECEIVE_COPY_RESULTS 0x84
-#define ACCESS_CONTROL_IN 0x86
-#define ACCESS_CONTROL_OUT 0x87
-#define READ_16 0x88
-#define WRITE_16 0x8a
-#define READ_ATTRIBUTE 0x8c
-#define WRITE_ATTRIBUTE 0x8d
-#define VERIFY_16 0x8f
-#define SYNCHRONIZE_CACHE_16 0x91
-#define WRITE_SAME_16 0x93
-#define SERVICE_ACTION_BIDIRECTIONAL 0x9d
-#define SERVICE_ACTION_IN_16 0x9e
-#define SERVICE_ACTION_OUT_16 0x9f
-/* values for service action in */
-#define SAI_READ_CAPACITY_16 0x10
-#define SAI_GET_LBA_STATUS 0x12
-/* values for VARIABLE_LENGTH_CMD service action codes
- * see spc4r17 Section D.3.5, table D.7 and D.8 */
-#define VLC_SA_RECEIVE_CREDENTIAL 0x1800
-/* values for maintenance in */
-#define MI_REPORT_IDENTIFYING_INFORMATION 0x05
-#define MI_REPORT_TARGET_PGS 0x0a
-#define MI_REPORT_ALIASES 0x0b
-#define MI_REPORT_SUPPORTED_OPERATION_CODES 0x0c
-#define MI_REPORT_SUPPORTED_TASK_MANAGEMENT_FUNCTIONS 0x0d
-#define MI_REPORT_PRIORITY 0x0e
-#define MI_REPORT_TIMESTAMP 0x0f
-#define MI_MANAGEMENT_PROTOCOL_IN 0x10
-/* value for MI_REPORT_TARGET_PGS ext header */
-#define MI_EXT_HDR_PARAM_FMT 0x20
-/* values for maintenance out */
-#define MO_SET_IDENTIFYING_INFORMATION 0x06
-#define MO_SET_TARGET_PGS 0x0a
-#define MO_CHANGE_ALIASES 0x0b
-#define MO_SET_PRIORITY 0x0e
-#define MO_SET_TIMESTAMP 0x0f
-#define MO_MANAGEMENT_PROTOCOL_OUT 0x10
-/* values for variable length command */
-#define XDREAD_32 0x03
-#define XDWRITE_32 0x04
-#define XPWRITE_32 0x06
-#define XDWRITEREAD_32 0x07
-#define READ_32 0x09
-#define VERIFY_32 0x0a
-#define WRITE_32 0x0b
-#define WRITE_SAME_32 0x0d
-
-#define SERVICE_ACTION16(cdb) (cdb[1] & 0x1f)
-#define SERVICE_ACTION32(cdb) ((cdb[8] << 8) | cdb[9])
-
-static const char *
-scsi_trace_misc(struct trace_seq *, unsigned char *, int);
-
-static const char *
-scsi_trace_rw6(struct trace_seq *p, unsigned char *cdb, int len)
-{
- const char *ret = p->buffer + p->len;
- sector_t lba = 0, txlen = 0;
-
- lba |= ((cdb[1] & 0x1F) << 16);
- lba |= (cdb[2] << 8);
- lba |= cdb[3];
- txlen = cdb[4];
-
- trace_seq_printf(p, "lba=%llu txlen=%llu",
- (unsigned long long)lba, (unsigned long long)txlen);
- trace_seq_putc(p, 0);
- return ret;
-}
-
-static const char *
-scsi_trace_rw10(struct trace_seq *p, unsigned char *cdb, int len)
-{
- const char *ret = p->buffer + p->len;
- sector_t lba = 0, txlen = 0;
-
- lba |= (cdb[2] << 24);
- lba |= (cdb[3] << 16);
- lba |= (cdb[4] << 8);
- lba |= cdb[5];
- txlen |= (cdb[7] << 8);
- txlen |= cdb[8];
-
- trace_seq_printf(p, "lba=%llu txlen=%llu protect=%u",
- (unsigned long long)lba, (unsigned long long)txlen,
- cdb[1] >> 5);
-
- if (cdb[0] == WRITE_SAME)
- trace_seq_printf(p, " unmap=%u", cdb[1] >> 3 & 1);
-
- trace_seq_putc(p, 0);
- return ret;
-}
-
-static const char *
-scsi_trace_rw12(struct trace_seq *p, unsigned char *cdb, int len)
-{
- const char *ret = p->buffer + p->len;
- sector_t lba = 0, txlen = 0;
-
- lba |= (cdb[2] << 24);
- lba |= (cdb[3] << 16);
- lba |= (cdb[4] << 8);
- lba |= cdb[5];
- txlen |= (cdb[6] << 24);
- txlen |= (cdb[7] << 16);
- txlen |= (cdb[8] << 8);
- txlen |= cdb[9];
-
- trace_seq_printf(p, "lba=%llu txlen=%llu protect=%u",
- (unsigned long long)lba, (unsigned long long)txlen,
- cdb[1] >> 5);
- trace_seq_putc(p, 0);
- return ret;
-}
-
-static const char *
-scsi_trace_rw16(struct trace_seq *p, unsigned char *cdb, int len)
-{
- const char *ret = p->buffer + p->len;
- sector_t lba = 0, txlen = 0;
-
- lba |= ((u64)cdb[2] << 56);
- lba |= ((u64)cdb[3] << 48);
- lba |= ((u64)cdb[4] << 40);
- lba |= ((u64)cdb[5] << 32);
- lba |= (cdb[6] << 24);
- lba |= (cdb[7] << 16);
- lba |= (cdb[8] << 8);
- lba |= cdb[9];
- txlen |= (cdb[10] << 24);
- txlen |= (cdb[11] << 16);
- txlen |= (cdb[12] << 8);
- txlen |= cdb[13];
-
- trace_seq_printf(p, "lba=%llu txlen=%llu protect=%u",
- (unsigned long long)lba, (unsigned long long)txlen,
- cdb[1] >> 5);
-
- if (cdb[0] == WRITE_SAME_16)
- trace_seq_printf(p, " unmap=%u", cdb[1] >> 3 & 1);
-
- trace_seq_putc(p, 0);
- return ret;
-}
-
-static const char *
-scsi_trace_rw32(struct trace_seq *p, unsigned char *cdb, int len)
-{
- const char *ret = p->buffer + p->len, *cmd;
- sector_t lba = 0, txlen = 0;
- u32 ei_lbrt = 0;
-
- switch (SERVICE_ACTION32(cdb)) {
- case READ_32:
- cmd = "READ";
- break;
- case VERIFY_32:
- cmd = "VERIFY";
- break;
- case WRITE_32:
- cmd = "WRITE";
- break;
- case WRITE_SAME_32:
- cmd = "WRITE_SAME";
- break;
- default:
- trace_seq_printf(p, "UNKNOWN");
- goto out;
- }
-
- lba |= ((u64)cdb[12] << 56);
- lba |= ((u64)cdb[13] << 48);
- lba |= ((u64)cdb[14] << 40);
- lba |= ((u64)cdb[15] << 32);
- lba |= (cdb[16] << 24);
- lba |= (cdb[17] << 16);
- lba |= (cdb[18] << 8);
- lba |= cdb[19];
- ei_lbrt |= (cdb[20] << 24);
- ei_lbrt |= (cdb[21] << 16);
- ei_lbrt |= (cdb[22] << 8);
- ei_lbrt |= cdb[23];
- txlen |= (cdb[28] << 24);
- txlen |= (cdb[29] << 16);
- txlen |= (cdb[30] << 8);
- txlen |= cdb[31];
-
- trace_seq_printf(p, "%s_32 lba=%llu txlen=%llu protect=%u ei_lbrt=%u",
- cmd, (unsigned long long)lba,
- (unsigned long long)txlen, cdb[10] >> 5, ei_lbrt);
-
- if (SERVICE_ACTION32(cdb) == WRITE_SAME_32)
- trace_seq_printf(p, " unmap=%u", cdb[10] >> 3 & 1);
-
-out:
- trace_seq_putc(p, 0);
- return ret;
-}
-
-static const char *
-scsi_trace_unmap(struct trace_seq *p, unsigned char *cdb, int len)
-{
- const char *ret = p->buffer + p->len;
- unsigned int regions = cdb[7] << 8 | cdb[8];
-
- trace_seq_printf(p, "regions=%u", (regions - 8) / 16);
- trace_seq_putc(p, 0);
- return ret;
-}
-
-static const char *
-scsi_trace_service_action_in(struct trace_seq *p, unsigned char *cdb, int len)
-{
- const char *ret = p->buffer + p->len, *cmd;
- sector_t lba = 0;
- u32 alloc_len = 0;
-
- switch (SERVICE_ACTION16(cdb)) {
- case SAI_READ_CAPACITY_16:
- cmd = "READ_CAPACITY_16";
- break;
- case SAI_GET_LBA_STATUS:
- cmd = "GET_LBA_STATUS";
- break;
- default:
- trace_seq_printf(p, "UNKNOWN");
- goto out;
- }
-
- lba |= ((u64)cdb[2] << 56);
- lba |= ((u64)cdb[3] << 48);
- lba |= ((u64)cdb[4] << 40);
- lba |= ((u64)cdb[5] << 32);
- lba |= (cdb[6] << 24);
- lba |= (cdb[7] << 16);
- lba |= (cdb[8] << 8);
- lba |= cdb[9];
- alloc_len |= (cdb[10] << 24);
- alloc_len |= (cdb[11] << 16);
- alloc_len |= (cdb[12] << 8);
- alloc_len |= cdb[13];
-
- trace_seq_printf(p, "%s lba=%llu alloc_len=%u", cmd,
- (unsigned long long)lba, alloc_len);
-
-out:
- trace_seq_putc(p, 0);
- return ret;
-}
-
-static const char *
-scsi_trace_varlen(struct trace_seq *p, unsigned char *cdb, int len)
-{
- switch (SERVICE_ACTION32(cdb)) {
- case READ_32:
- case VERIFY_32:
- case WRITE_32:
- case WRITE_SAME_32:
- return scsi_trace_rw32(p, cdb, len);
- default:
- return scsi_trace_misc(p, cdb, len);
- }
-}
-
-static const char *
-scsi_trace_misc(struct trace_seq *p, unsigned char *cdb, int len)
-{
- const char *ret = p->buffer + p->len;
-
- trace_seq_printf(p, "-");
- trace_seq_putc(p, 0);
- return ret;
-}
-
-const char *
-scsi_trace_parse_cdb(struct trace_seq *p, unsigned char *cdb, int len)
-{
- switch (cdb[0]) {
- case READ_6:
- case WRITE_6:
- return scsi_trace_rw6(p, cdb, len);
- case READ_10:
- case VERIFY:
- case WRITE_10:
- case WRITE_SAME:
- return scsi_trace_rw10(p, cdb, len);
- case READ_12:
- case VERIFY_12:
- case WRITE_12:
- return scsi_trace_rw12(p, cdb, len);
- case READ_16:
- case VERIFY_16:
- case WRITE_16:
- case WRITE_SAME_16:
- return scsi_trace_rw16(p, cdb, len);
- case UNMAP:
- return scsi_trace_unmap(p, cdb, len);
- case SERVICE_ACTION_IN_16:
- return scsi_trace_service_action_in(p, cdb, len);
- case VARIABLE_LENGTH_CMD:
- return scsi_trace_varlen(p, cdb, len);
- default:
- return scsi_trace_misc(p, cdb, len);
- }
-}
-
-unsigned long long process_scsi_trace_parse_cdb(struct trace_seq *s,
- unsigned long long *args)
-{
- scsi_trace_parse_cdb(s, (unsigned char *) (unsigned long) args[1], args[2]);
- return 0;
-}
-
-int TEP_PLUGIN_LOADER(struct tep_handle *tep)
-{
- tep_register_print_function(tep,
- process_scsi_trace_parse_cdb,
- TEP_FUNC_ARG_STRING,
- "scsi_trace_parse_cdb",
- TEP_FUNC_ARG_PTR,
- TEP_FUNC_ARG_PTR,
- TEP_FUNC_ARG_INT,
- TEP_FUNC_ARG_VOID);
- return 0;
-}
-
-void TEP_PLUGIN_UNLOADER(struct tep_handle *tep)
-{
- tep_unregister_print_function(tep, process_scsi_trace_parse_cdb,
- "scsi_trace_parse_cdb");
-}
diff --git a/tools/lib/traceevent/plugins/plugin_tlb.c b/tools/lib/traceevent/plugins/plugin_tlb.c
deleted file mode 100644
index 43657fb..0000000
--- a/tools/lib/traceevent/plugins/plugin_tlb.c
+++ /dev/null
@@ -1,66 +0,0 @@
-// SPDX-License-Identifier: LGPL-2.1
-/*
- * Copyright (C) 2015 Red Hat Inc, Steven Rostedt <srostedt@redhat.com>
- */
-#include <stdio.h>
-#include <stdlib.h>
-#include <string.h>
-
-#include "event-parse.h"
-
-enum tlb_flush_reason {
- TLB_FLUSH_ON_TASK_SWITCH,
- TLB_REMOTE_SHOOTDOWN,
- TLB_LOCAL_SHOOTDOWN,
- TLB_LOCAL_MM_SHOOTDOWN,
- NR_TLB_FLUSH_REASONS,
-};
-
-static int tlb_flush_handler(struct trace_seq *s, struct tep_record *record,
- struct tep_event *event, void *context)
-{
- unsigned long long val;
-
- trace_seq_printf(s, "pages=");
-
- tep_print_num_field(s, "%ld", event, "pages", record, 1);
-
- if (tep_get_field_val(s, event, "reason", record, &val, 1) < 0)
- return -1;
-
- trace_seq_puts(s, " reason=");
-
- switch (val) {
- case TLB_FLUSH_ON_TASK_SWITCH:
- trace_seq_puts(s, "flush on task switch");
- break;
- case TLB_REMOTE_SHOOTDOWN:
- trace_seq_puts(s, "remote shootdown");
- break;
- case TLB_LOCAL_SHOOTDOWN:
- trace_seq_puts(s, "local shootdown");
- break;
- case TLB_LOCAL_MM_SHOOTDOWN:
- trace_seq_puts(s, "local mm shootdown");
- break;
- }
-
- trace_seq_printf(s, " (%lld)", val);
-
- return 0;
-}
-
-int TEP_PLUGIN_LOADER(struct tep_handle *tep)
-{
- tep_register_event_handler(tep, -1, "tlb", "tlb_flush",
- tlb_flush_handler, NULL);
-
- return 0;
-}
-
-void TEP_PLUGIN_UNLOADER(struct tep_handle *tep)
-{
- tep_unregister_event_handler(tep, -1,
- "tlb", "tlb_flush",
- tlb_flush_handler, NULL);
-}
diff --git a/tools/lib/traceevent/plugins/plugin_xen.c b/tools/lib/traceevent/plugins/plugin_xen.c
deleted file mode 100644
index 993b208..0000000
--- a/tools/lib/traceevent/plugins/plugin_xen.c
+++ /dev/null
@@ -1,138 +0,0 @@
-// SPDX-License-Identifier: GPL-2.0
-#include <stdio.h>
-#include <stdlib.h>
-#include <string.h>
-#include "event-parse.h"
-#include "trace-seq.h"
-
-#define __HYPERVISOR_set_trap_table 0
-#define __HYPERVISOR_mmu_update 1
-#define __HYPERVISOR_set_gdt 2
-#define __HYPERVISOR_stack_switch 3
-#define __HYPERVISOR_set_callbacks 4
-#define __HYPERVISOR_fpu_taskswitch 5
-#define __HYPERVISOR_sched_op_compat 6
-#define __HYPERVISOR_dom0_op 7
-#define __HYPERVISOR_set_debugreg 8
-#define __HYPERVISOR_get_debugreg 9
-#define __HYPERVISOR_update_descriptor 10
-#define __HYPERVISOR_memory_op 12
-#define __HYPERVISOR_multicall 13
-#define __HYPERVISOR_update_va_mapping 14
-#define __HYPERVISOR_set_timer_op 15
-#define __HYPERVISOR_event_channel_op_compat 16
-#define __HYPERVISOR_xen_version 17
-#define __HYPERVISOR_console_io 18
-#define __HYPERVISOR_physdev_op_compat 19
-#define __HYPERVISOR_grant_table_op 20
-#define __HYPERVISOR_vm_assist 21
-#define __HYPERVISOR_update_va_mapping_otherdomain 22
-#define __HYPERVISOR_iret 23 /* x86 only */
-#define __HYPERVISOR_vcpu_op 24
-#define __HYPERVISOR_set_segment_base 25 /* x86/64 only */
-#define __HYPERVISOR_mmuext_op 26
-#define __HYPERVISOR_acm_op 27
-#define __HYPERVISOR_nmi_op 28
-#define __HYPERVISOR_sched_op 29
-#define __HYPERVISOR_callback_op 30
-#define __HYPERVISOR_xenoprof_op 31
-#define __HYPERVISOR_event_channel_op 32
-#define __HYPERVISOR_physdev_op 33
-#define __HYPERVISOR_hvm_op 34
-#define __HYPERVISOR_tmem_op 38
-
-/* Architecture-specific hypercall definitions. */
-#define __HYPERVISOR_arch_0 48
-#define __HYPERVISOR_arch_1 49
-#define __HYPERVISOR_arch_2 50
-#define __HYPERVISOR_arch_3 51
-#define __HYPERVISOR_arch_4 52
-#define __HYPERVISOR_arch_5 53
-#define __HYPERVISOR_arch_6 54
-#define __HYPERVISOR_arch_7 55
-
-#define N(x) [__HYPERVISOR_##x] = "("#x")"
-static const char *xen_hypercall_names[] = {
- N(set_trap_table),
- N(mmu_update),
- N(set_gdt),
- N(stack_switch),
- N(set_callbacks),
- N(fpu_taskswitch),
- N(sched_op_compat),
- N(dom0_op),
- N(set_debugreg),
- N(get_debugreg),
- N(update_descriptor),
- N(memory_op),
- N(multicall),
- N(update_va_mapping),
- N(set_timer_op),
- N(event_channel_op_compat),
- N(xen_version),
- N(console_io),
- N(physdev_op_compat),
- N(grant_table_op),
- N(vm_assist),
- N(update_va_mapping_otherdomain),
- N(iret),
- N(vcpu_op),
- N(set_segment_base),
- N(mmuext_op),
- N(acm_op),
- N(nmi_op),
- N(sched_op),
- N(callback_op),
- N(xenoprof_op),
- N(event_channel_op),
- N(physdev_op),
- N(hvm_op),
-
-/* Architecture-specific hypercall definitions. */
- N(arch_0),
- N(arch_1),
- N(arch_2),
- N(arch_3),
- N(arch_4),
- N(arch_5),
- N(arch_6),
- N(arch_7),
-};
-#undef N
-
-#define ARRAY_SIZE(arr) (sizeof(arr) / sizeof((arr)[0]))
-
-static const char *xen_hypercall_name(unsigned op)
-{
- if (op < ARRAY_SIZE(xen_hypercall_names) &&
- xen_hypercall_names[op] != NULL)
- return xen_hypercall_names[op];
-
- return "";
-}
-
-unsigned long long process_xen_hypercall_name(struct trace_seq *s,
- unsigned long long *args)
-{
- unsigned int op = args[0];
-
- trace_seq_printf(s, "%s", xen_hypercall_name(op));
- return 0;
-}
-
-int TEP_PLUGIN_LOADER(struct tep_handle *tep)
-{
- tep_register_print_function(tep,
- process_xen_hypercall_name,
- TEP_FUNC_ARG_STRING,
- "xen_hypercall_name",
- TEP_FUNC_ARG_INT,
- TEP_FUNC_ARG_VOID);
- return 0;
-}
-
-void TEP_PLUGIN_UNLOADER(struct tep_handle *tep)
-{
- tep_unregister_print_function(tep, process_xen_hypercall_name,
- "xen_hypercall_name");
-}
diff --git a/tools/lib/traceevent/tep_strerror.c b/tools/lib/traceevent/tep_strerror.c
deleted file mode 100644
index 4ac2644..0000000
--- a/tools/lib/traceevent/tep_strerror.c
+++ /dev/null
@@ -1,53 +0,0 @@
-// SPDX-License-Identifier: LGPL-2.1
-#undef _GNU_SOURCE
-#include <string.h>
-#include <stdio.h>
-
-#include "event-parse.h"
-
-#undef _PE
-#define _PE(code, str) str
-static const char * const tep_error_str[] = {
- TEP_ERRORS
-};
-#undef _PE
-
-/*
- * The tools so far have been using the strerror_r() GNU variant, that returns
- * a string, be it the buffer passed or something else.
- *
- * But that, besides being tricky in cases where we expect that the function
- * using strerror_r() returns the error formatted in a provided buffer (we have
- * to check if it returned something else and copy that instead), breaks the
- * build on systems not using glibc, like Alpine Linux, where musl libc is
- * used.
- *
- * So, introduce yet another wrapper, str_error_r(), that has the GNU
- * interface, but uses the portable XSI variant of strerror_r(), so that users
- * rest asured that the provided buffer is used and it is what is returned.
- */
-int tep_strerror(struct tep_handle *tep __maybe_unused,
- enum tep_errno errnum, char *buf, size_t buflen)
-{
- const char *msg;
- int idx;
-
- if (!buflen)
- return 0;
-
- if (errnum >= 0) {
- int err = strerror_r(errnum, buf, buflen);
- buf[buflen - 1] = 0;
- return err;
- }
-
- if (errnum <= __TEP_ERRNO__START ||
- errnum >= __TEP_ERRNO__END)
- return -1;
-
- idx = errnum - __TEP_ERRNO__START - 1;
- msg = tep_error_str[idx];
- snprintf(buf, buflen, "%s", msg);
-
- return 0;
-}
diff --git a/tools/lib/traceevent/trace-seq.c b/tools/lib/traceevent/trace-seq.c
deleted file mode 100644
index 8d5ecd2..0000000
--- a/tools/lib/traceevent/trace-seq.c
+++ /dev/null
@@ -1,249 +0,0 @@
-// SPDX-License-Identifier: LGPL-2.1
-/*
- * Copyright (C) 2009 Red Hat Inc, Steven Rostedt <srostedt@redhat.com>
- *
- */
-#include "trace-seq.h"
-
-#include <stdio.h>
-#include <stdlib.h>
-#include <string.h>
-#include <stdarg.h>
-
-#include <asm/bug.h>
-#include "event-parse.h"
-#include "event-utils.h"
-
-/*
- * The TRACE_SEQ_POISON is to catch the use of using
- * a trace_seq structure after it was destroyed.
- */
-#define TRACE_SEQ_POISON ((void *)0xdeadbeef)
-#define TRACE_SEQ_CHECK(s) \
-do { \
- if (WARN_ONCE((s)->buffer == TRACE_SEQ_POISON, \
- "Usage of trace_seq after it was destroyed")) \
- (s)->state = TRACE_SEQ__BUFFER_POISONED; \
-} while (0)
-
-#define TRACE_SEQ_CHECK_RET_N(s, n) \
-do { \
- TRACE_SEQ_CHECK(s); \
- if ((s)->state != TRACE_SEQ__GOOD) \
- return n; \
-} while (0)
-
-#define TRACE_SEQ_CHECK_RET(s) TRACE_SEQ_CHECK_RET_N(s, )
-#define TRACE_SEQ_CHECK_RET0(s) TRACE_SEQ_CHECK_RET_N(s, 0)
-
-/**
- * trace_seq_init - initialize the trace_seq structure
- * @s: a pointer to the trace_seq structure to initialize
- */
-void trace_seq_init(struct trace_seq *s)
-{
- s->len = 0;
- s->readpos = 0;
- s->buffer_size = TRACE_SEQ_BUF_SIZE;
- s->buffer = malloc(s->buffer_size);
- if (s->buffer != NULL)
- s->state = TRACE_SEQ__GOOD;
- else
- s->state = TRACE_SEQ__MEM_ALLOC_FAILED;
-}
-
-/**
- * trace_seq_reset - re-initialize the trace_seq structure
- * @s: a pointer to the trace_seq structure to reset
- */
-void trace_seq_reset(struct trace_seq *s)
-{
- if (!s)
- return;
- TRACE_SEQ_CHECK(s);
- s->len = 0;
- s->readpos = 0;
-}
-
-/**
- * trace_seq_destroy - free up memory of a trace_seq
- * @s: a pointer to the trace_seq to free the buffer
- *
- * Only frees the buffer, not the trace_seq struct itself.
- */
-void trace_seq_destroy(struct trace_seq *s)
-{
- if (!s)
- return;
- TRACE_SEQ_CHECK_RET(s);
- free(s->buffer);
- s->buffer = TRACE_SEQ_POISON;
-}
-
-static void expand_buffer(struct trace_seq *s)
-{
- char *buf;
-
- buf = realloc(s->buffer, s->buffer_size + TRACE_SEQ_BUF_SIZE);
- if (WARN_ONCE(!buf, "Can't allocate trace_seq buffer memory")) {
- s->state = TRACE_SEQ__MEM_ALLOC_FAILED;
- return;
- }
-
- s->buffer = buf;
- s->buffer_size += TRACE_SEQ_BUF_SIZE;
-}
-
-/**
- * trace_seq_printf - sequence printing of trace information
- * @s: trace sequence descriptor
- * @fmt: printf format string
- *
- * It returns 0 if the trace oversizes the buffer's free
- * space, the number of characters printed, or a negative
- * value in case of an error.
- *
- * The tracer may use either sequence operations or its own
- * copy to user routines. To simplify formating of a trace
- * trace_seq_printf is used to store strings into a special
- * buffer (@s). Then the output may be either used by
- * the sequencer or pulled into another buffer.
- */
-int
-trace_seq_printf(struct trace_seq *s, const char *fmt, ...)
-{
- va_list ap;
- int len;
- int ret;
-
- try_again:
- TRACE_SEQ_CHECK_RET0(s);
-
- len = (s->buffer_size - 1) - s->len;
-
- va_start(ap, fmt);
- ret = vsnprintf(s->buffer + s->len, len, fmt, ap);
- va_end(ap);
-
- if (ret >= len) {
- expand_buffer(s);
- goto try_again;
- }
-
- if (ret > 0)
- s->len += ret;
-
- return ret;
-}
-
-/**
- * trace_seq_vprintf - sequence printing of trace information
- * @s: trace sequence descriptor
- * @fmt: printf format string
- *
- * It returns 0 if the trace oversizes the buffer's free
- * space, the number of characters printed, or a negative
- * value in case of an error.
- * *
- * The tracer may use either sequence operations or its own
- * copy to user routines. To simplify formating of a trace
- * trace_seq_printf is used to store strings into a special
- * buffer (@s). Then the output may be either used by
- * the sequencer or pulled into another buffer.
- */
-int
-trace_seq_vprintf(struct trace_seq *s, const char *fmt, va_list args)
-{
- int len;
- int ret;
-
- try_again:
- TRACE_SEQ_CHECK_RET0(s);
-
- len = (s->buffer_size - 1) - s->len;
-
- ret = vsnprintf(s->buffer + s->len, len, fmt, args);
-
- if (ret >= len) {
- expand_buffer(s);
- goto try_again;
- }
-
- if (ret > 0)
- s->len += ret;
-
- return ret;
-}
-
-/**
- * trace_seq_puts - trace sequence printing of simple string
- * @s: trace sequence descriptor
- * @str: simple string to record
- *
- * The tracer may use either the sequence operations or its own
- * copy to user routines. This function records a simple string
- * into a special buffer (@s) for later retrieval by a sequencer
- * or other mechanism.
- */
-int trace_seq_puts(struct trace_seq *s, const char *str)
-{
- int len;
-
- TRACE_SEQ_CHECK_RET0(s);
-
- len = strlen(str);
-
- while (len > ((s->buffer_size - 1) - s->len))
- expand_buffer(s);
-
- TRACE_SEQ_CHECK_RET0(s);
-
- memcpy(s->buffer + s->len, str, len);
- s->len += len;
-
- return len;
-}
-
-int trace_seq_putc(struct trace_seq *s, unsigned char c)
-{
- TRACE_SEQ_CHECK_RET0(s);
-
- while (s->len >= (s->buffer_size - 1))
- expand_buffer(s);
-
- TRACE_SEQ_CHECK_RET0(s);
-
- s->buffer[s->len++] = c;
-
- return 1;
-}
-
-void trace_seq_terminate(struct trace_seq *s)
-{
- TRACE_SEQ_CHECK_RET(s);
-
- /* There's always one character left on the buffer */
- s->buffer[s->len] = 0;
-}
-
-int trace_seq_do_fprintf(struct trace_seq *s, FILE *fp)
-{
- TRACE_SEQ_CHECK(s);
-
- switch (s->state) {
- case TRACE_SEQ__GOOD:
- return fprintf(fp, "%.*s", s->len, s->buffer);
- case TRACE_SEQ__BUFFER_POISONED:
- fprintf(fp, "%s\n", "Usage of trace_seq after it was destroyed");
- break;
- case TRACE_SEQ__MEM_ALLOC_FAILED:
- fprintf(fp, "%s\n", "Can't allocate trace_seq buffer memory");
- break;
- }
- return -1;
-}
-
-int trace_seq_do_printf(struct trace_seq *s)
-{
- return trace_seq_do_fprintf(s, stdout);
-}
diff --git a/tools/lib/traceevent/trace-seq.h b/tools/lib/traceevent/trace-seq.h
deleted file mode 100644
index d68ec69..0000000
--- a/tools/lib/traceevent/trace-seq.h
+++ /dev/null
@@ -1,55 +0,0 @@
-// SPDX-License-Identifier: LGPL-2.1
-/*
- * Copyright (C) 2009, 2010 Red Hat Inc, Steven Rostedt <srostedt@redhat.com>
- *
- */
-
-#ifndef _TRACE_SEQ_H
-#define _TRACE_SEQ_H
-
-#include <stdarg.h>
-#include <stdio.h>
-
-/* ----------------------- trace_seq ----------------------- */
-
-#ifndef TRACE_SEQ_BUF_SIZE
-#define TRACE_SEQ_BUF_SIZE 4096
-#endif
-
-enum trace_seq_fail {
- TRACE_SEQ__GOOD,
- TRACE_SEQ__BUFFER_POISONED,
- TRACE_SEQ__MEM_ALLOC_FAILED,
-};
-
-/*
- * Trace sequences are used to allow a function to call several other functions
- * to create a string of data to use (up to a max of PAGE_SIZE).
- */
-
-struct trace_seq {
- char *buffer;
- unsigned int buffer_size;
- unsigned int len;
- unsigned int readpos;
- enum trace_seq_fail state;
-};
-
-void trace_seq_init(struct trace_seq *s);
-void trace_seq_reset(struct trace_seq *s);
-void trace_seq_destroy(struct trace_seq *s);
-
-extern int trace_seq_printf(struct trace_seq *s, const char *fmt, ...)
- __attribute__ ((format (printf, 2, 3)));
-extern int trace_seq_vprintf(struct trace_seq *s, const char *fmt, va_list args)
- __attribute__ ((format (printf, 2, 0)));
-
-extern int trace_seq_puts(struct trace_seq *s, const char *str);
-extern int trace_seq_putc(struct trace_seq *s, unsigned char c);
-
-extern void trace_seq_terminate(struct trace_seq *s);
-
-extern int trace_seq_do_fprintf(struct trace_seq *s, FILE *fp);
-extern int trace_seq_do_printf(struct trace_seq *s);
-
-#endif /* _TRACE_SEQ_H */
diff --git a/tools/objtool/arch/powerpc/Build b/tools/objtool/arch/powerpc/Build
new file mode 100644
index 0000000..d24d563
--- /dev/null
+++ b/tools/objtool/arch/powerpc/Build
@@ -0,0 +1,2 @@
+objtool-y += decode.o
+objtool-y += special.o
diff --git a/tools/objtool/arch/powerpc/decode.c b/tools/objtool/arch/powerpc/decode.c
new file mode 100644
index 0000000..9c65380
--- /dev/null
+++ b/tools/objtool/arch/powerpc/decode.c
@@ -0,0 +1,110 @@
+// SPDX-License-Identifier: GPL-2.0-or-later
+
+#include <stdio.h>
+#include <stdlib.h>
+#include <objtool/check.h>
+#include <objtool/elf.h>
+#include <objtool/arch.h>
+#include <objtool/warn.h>
+#include <objtool/builtin.h>
+#include <objtool/endianness.h>
+
+int arch_ftrace_match(char *name)
+{
+ return !strcmp(name, "_mcount");
+}
+
+unsigned long arch_dest_reloc_offset(int addend)
+{
+ return addend;
+}
+
+bool arch_callee_saved_reg(unsigned char reg)
+{
+ return false;
+}
+
+int arch_decode_hint_reg(u8 sp_reg, int *base)
+{
+ exit(-1);
+}
+
+const char *arch_nop_insn(int len)
+{
+ exit(-1);
+}
+
+const char *arch_ret_insn(int len)
+{
+ exit(-1);
+}
+
+int arch_decode_instruction(struct objtool_file *file, const struct section *sec,
+ unsigned long offset, unsigned int maxlen,
+ unsigned int *len, enum insn_type *type,
+ unsigned long *immediate,
+ struct list_head *ops_list)
+{
+ unsigned int opcode;
+ enum insn_type typ;
+ unsigned long imm;
+ u32 insn;
+
+ insn = bswap_if_needed(file->elf, *(u32 *)(sec->data->d_buf + offset));
+ opcode = insn >> 26;
+ typ = INSN_OTHER;
+ imm = 0;
+
+ switch (opcode) {
+ case 18: /* b[l][a] */
+ if ((insn & 3) == 1) /* bl */
+ typ = INSN_CALL;
+
+ imm = insn & 0x3fffffc;
+ if (imm & 0x2000000)
+ imm -= 0x4000000;
+ break;
+ }
+
+ if (opcode == 1)
+ *len = 8;
+ else
+ *len = 4;
+
+ *type = typ;
+ *immediate = imm;
+
+ return 0;
+}
+
+unsigned long arch_jump_destination(struct instruction *insn)
+{
+ return insn->offset + insn->immediate;
+}
+
+bool arch_pc_relative_reloc(struct reloc *reloc)
+{
+ /*
+ * The powerpc build only allows certain relocation types, see
+ * relocs_check.sh, and none of those accepted are PC relative.
+ */
+ return false;
+}
+
+void arch_initial_func_cfi_state(struct cfi_init_state *state)
+{
+ int i;
+
+ for (i = 0; i < CFI_NUM_REGS; i++) {
+ state->regs[i].base = CFI_UNDEFINED;
+ state->regs[i].offset = 0;
+ }
+
+ /* initial CFA (call frame address) */
+ state->cfa.base = CFI_SP;
+ state->cfa.offset = 0;
+
+ /* initial LR (return address) */
+ state->regs[CFI_RA].base = CFI_CFA;
+ state->regs[CFI_RA].offset = 0;
+}
diff --git a/tools/objtool/arch/powerpc/include/arch/cfi_regs.h b/tools/objtool/arch/powerpc/include/arch/cfi_regs.h
new file mode 100644
index 0000000..59638eb
--- /dev/null
+++ b/tools/objtool/arch/powerpc/include/arch/cfi_regs.h
@@ -0,0 +1,11 @@
+/* SPDX-License-Identifier: GPL-2.0-or-later */
+
+#ifndef _OBJTOOL_CFI_REGS_H
+#define _OBJTOOL_CFI_REGS_H
+
+#define CFI_BP 1
+#define CFI_SP CFI_BP
+#define CFI_RA 32
+#define CFI_NUM_REGS 33
+
+#endif
diff --git a/tools/objtool/arch/powerpc/include/arch/elf.h b/tools/objtool/arch/powerpc/include/arch/elf.h
new file mode 100644
index 0000000..73f9ae1
--- /dev/null
+++ b/tools/objtool/arch/powerpc/include/arch/elf.h
@@ -0,0 +1,10 @@
+/* SPDX-License-Identifier: GPL-2.0-or-later */
+
+#ifndef _OBJTOOL_ARCH_ELF
+#define _OBJTOOL_ARCH_ELF
+
+#define R_NONE R_PPC_NONE
+#define R_ABS64 R_PPC64_ADDR64
+#define R_ABS32 R_PPC_ADDR32
+
+#endif /* _OBJTOOL_ARCH_ELF */
diff --git a/tools/objtool/arch/powerpc/include/arch/special.h b/tools/objtool/arch/powerpc/include/arch/special.h
new file mode 100644
index 0000000..ffef9ad
--- /dev/null
+++ b/tools/objtool/arch/powerpc/include/arch/special.h
@@ -0,0 +1,21 @@
+/* SPDX-License-Identifier: GPL-2.0-or-later */
+#ifndef _PPC_ARCH_SPECIAL_H
+#define _PPC_ARCH_SPECIAL_H
+
+#define EX_ENTRY_SIZE 8
+#define EX_ORIG_OFFSET 0
+#define EX_NEW_OFFSET 4
+
+#define JUMP_ENTRY_SIZE 16
+#define JUMP_ORIG_OFFSET 0
+#define JUMP_NEW_OFFSET 4
+#define JUMP_KEY_OFFSET 8
+
+#define ALT_ENTRY_SIZE 12
+#define ALT_ORIG_OFFSET 0
+#define ALT_NEW_OFFSET 4
+#define ALT_FEATURE_OFFSET 8
+#define ALT_ORIG_LEN_OFFSET 10
+#define ALT_NEW_LEN_OFFSET 11
+
+#endif /* _PPC_ARCH_SPECIAL_H */
diff --git a/tools/objtool/arch/powerpc/special.c b/tools/objtool/arch/powerpc/special.c
new file mode 100644
index 0000000..d338681
--- /dev/null
+++ b/tools/objtool/arch/powerpc/special.c
@@ -0,0 +1,19 @@
+// SPDX-License-Identifier: GPL-2.0-or-later
+#include <string.h>
+#include <stdlib.h>
+#include <objtool/special.h>
+#include <objtool/builtin.h>
+
+
+bool arch_support_alt_relocation(struct special_alt *special_alt,
+ struct instruction *insn,
+ struct reloc *reloc)
+{
+ exit(-1);
+}
+
+struct reloc *arch_find_switch_table(struct objtool_file *file,
+ struct instruction *insn)
+{
+ exit(-1);
+}
diff --git a/tools/objtool/arch/x86/decode.c b/tools/objtool/arch/x86/decode.c
index f094383..e7b030f 100644
--- a/tools/objtool/arch/x86/decode.c
+++ b/tools/objtool/arch/x86/decode.c
@@ -23,6 +23,11 @@
#include <objtool/builtin.h>
#include <arch/elf.h>
+int arch_ftrace_match(char *name)
+{
+ return !strcmp(name, "__fentry__");
+}
+
static int is_x86_64(const struct elf *elf)
{
switch (elf->ehdr.e_machine) {
diff --git a/tools/objtool/arch/x86/include/arch/elf.h b/tools/objtool/arch/x86/include/arch/elf.h
index 69cc426..ac14987 100644
--- a/tools/objtool/arch/x86/include/arch/elf.h
+++ b/tools/objtool/arch/x86/include/arch/elf.h
@@ -2,5 +2,7 @@
#define _OBJTOOL_ARCH_ELF
#define R_NONE R_X86_64_NONE
+#define R_ABS64 R_X86_64_64
+#define R_ABS32 R_X86_64_32
#endif /* _OBJTOOL_ARCH_ELF */
diff --git a/tools/objtool/arch/x86/include/arch/endianness.h b/tools/objtool/arch/x86/include/arch/endianness.h
deleted file mode 100644
index 7c36252..0000000
--- a/tools/objtool/arch/x86/include/arch/endianness.h
+++ /dev/null
@@ -1,9 +0,0 @@
-/* SPDX-License-Identifier: GPL-2.0-or-later */
-#ifndef _ARCH_ENDIANNESS_H
-#define _ARCH_ENDIANNESS_H
-
-#include <endian.h>
-
-#define __TARGET_BYTE_ORDER __LITTLE_ENDIAN
-
-#endif /* _ARCH_ENDIANNESS_H */
diff --git a/tools/objtool/builtin-check.c b/tools/objtool/builtin-check.c
index 868e3e3..a4f3940 100644
--- a/tools/objtool/builtin-check.c
+++ b/tools/objtool/builtin-check.c
@@ -89,6 +89,7 @@ const struct option check_options[] = {
OPT_BOOLEAN(0, "dry-run", &opts.dryrun, "don't write modifications"),
OPT_BOOLEAN(0, "link", &opts.link, "object is a linked object"),
OPT_BOOLEAN(0, "module", &opts.module, "object is part of a kernel module"),
+ OPT_BOOLEAN(0, "mnop", &opts.mnop, "nop out mcount call sites"),
OPT_BOOLEAN(0, "no-unreachable", &opts.no_unreachable, "skip 'unreachable instruction' warnings"),
OPT_BOOLEAN(0, "sec-address", &opts.sec_address, "print section addresses in warnings"),
OPT_BOOLEAN(0, "stats", &opts.stats, "print statistics"),
@@ -157,6 +158,16 @@ static bool opts_valid(void)
return false;
}
+static bool mnop_opts_valid(void)
+{
+ if (opts.mnop && !opts.mcount) {
+ ERROR("--mnop requires --mcount");
+ return false;
+ }
+
+ return true;
+}
+
static bool link_opts_valid(struct objtool_file *file)
{
if (opts.link)
@@ -205,6 +216,9 @@ int objtool_run(int argc, const char **argv)
if (!file)
return 1;
+ if (!mnop_opts_valid())
+ return 1;
+
if (!link_opts_valid(file))
return 1;
diff --git a/tools/objtool/check.c b/tools/objtool/check.c
index 14130ab8..4350be7 100644
--- a/tools/objtool/check.c
+++ b/tools/objtool/check.c
@@ -204,7 +204,7 @@ static bool __dead_end_function(struct objtool_file *file, struct symbol *func,
return false;
insn = find_insn(file, func->sec, func->offset);
- if (!insn_func(insn))
+ if (!insn || !insn_func(insn))
return false;
func_for_each_insn(file, func, insn) {
@@ -925,9 +925,9 @@ static int create_cfi_sections(struct objtool_file *file)
static int create_mcount_loc_sections(struct objtool_file *file)
{
- struct section *sec;
- unsigned long *loc;
+ int addrsize = elf_class_addrsize(file->elf);
struct instruction *insn;
+ struct section *sec;
int idx;
sec = find_section_by_name(file->elf, "__mcount_loc");
@@ -944,23 +944,25 @@ static int create_mcount_loc_sections(struct objtool_file *file)
list_for_each_entry(insn, &file->mcount_loc_list, call_node)
idx++;
- sec = elf_create_section(file->elf, "__mcount_loc", 0, sizeof(unsigned long), idx);
+ sec = elf_create_section(file->elf, "__mcount_loc", 0, addrsize, idx);
if (!sec)
return -1;
+ sec->sh.sh_addralign = addrsize;
+
idx = 0;
list_for_each_entry(insn, &file->mcount_loc_list, call_node) {
+ void *loc;
- loc = (unsigned long *)sec->data->d_buf + idx;
- memset(loc, 0, sizeof(unsigned long));
+ loc = sec->data->d_buf + idx;
+ memset(loc, 0, addrsize);
- if (elf_add_reloc_to_insn(file->elf, sec,
- idx * sizeof(unsigned long),
- R_X86_64_64,
+ if (elf_add_reloc_to_insn(file->elf, sec, idx,
+ addrsize == sizeof(u64) ? R_ABS64 : R_ABS32,
insn->sec, insn->offset))
return -1;
- idx++;
+ idx += addrsize;
}
return 0;
@@ -1380,18 +1382,19 @@ static void annotate_call_site(struct objtool_file *file,
if (opts.mcount && sym->fentry) {
if (sibling)
WARN_FUNC("Tail call to __fentry__ !?!?", insn->sec, insn->offset);
+ if (opts.mnop) {
+ if (reloc) {
+ reloc->type = R_NONE;
+ elf_write_reloc(file->elf, reloc);
+ }
- if (reloc) {
- reloc->type = R_NONE;
- elf_write_reloc(file->elf, reloc);
+ elf_write_insn(file->elf, insn->sec,
+ insn->offset, insn->len,
+ arch_nop_insn(insn->len));
+
+ insn->type = INSN_NOP;
}
- elf_write_insn(file->elf, insn->sec,
- insn->offset, insn->len,
- arch_nop_insn(insn->len));
-
- insn->type = INSN_NOP;
-
list_add_tail(&insn->call_node, &file->mcount_loc_list);
return;
}
@@ -2252,7 +2255,7 @@ static int read_unwind_hints(struct objtool_file *file)
return -1;
}
- cfi.cfa.offset = bswap_if_needed(hint->sp_offset);
+ cfi.cfa.offset = bswap_if_needed(file->elf, hint->sp_offset);
cfi.type = hint->type;
cfi.end = hint->end;
@@ -2465,7 +2468,7 @@ static int classify_symbols(struct objtool_file *file)
if (arch_is_rethunk(func))
func->return_thunk = true;
- if (!strcmp(func->name, "__fentry__"))
+ if (arch_ftrace_match(func->name))
func->fentry = true;
if (is_profiling_func(func->name))
@@ -2541,9 +2544,11 @@ static int decode_sections(struct objtool_file *file)
* Must be before add_jump_destinations(), which depends on 'func'
* being set for alternatives, to enable proper sibling call detection.
*/
- ret = add_special_section_alts(file);
- if (ret)
- return ret;
+ if (opts.stackval || opts.orc || opts.uaccess || opts.noinstr) {
+ ret = add_special_section_alts(file);
+ if (ret)
+ return ret;
+ }
ret = add_jump_destinations(file);
if (ret)
diff --git a/tools/objtool/elf.c b/tools/objtool/elf.c
index 8cd7f01..64443a7 100644
--- a/tools/objtool/elf.c
+++ b/tools/objtool/elf.c
@@ -1174,6 +1174,7 @@ static struct section *elf_create_rela_reloc_section(struct elf *elf, struct sec
{
char *relocname;
struct section *sec;
+ int addrsize = elf_class_addrsize(elf);
relocname = malloc(strlen(base->name) + strlen(".rela") + 1);
if (!relocname) {
@@ -1183,7 +1184,10 @@ static struct section *elf_create_rela_reloc_section(struct elf *elf, struct sec
strcpy(relocname, ".rela");
strcat(relocname, base->name);
- sec = elf_create_section(elf, relocname, 0, sizeof(GElf_Rela), 0);
+ if (addrsize == sizeof(u32))
+ sec = elf_create_section(elf, relocname, 0, sizeof(Elf32_Rela), 0);
+ else
+ sec = elf_create_section(elf, relocname, 0, sizeof(GElf_Rela), 0);
free(relocname);
if (!sec)
return NULL;
@@ -1192,7 +1196,7 @@ static struct section *elf_create_rela_reloc_section(struct elf *elf, struct sec
sec->base = base;
sec->sh.sh_type = SHT_RELA;
- sec->sh.sh_addralign = 8;
+ sec->sh.sh_addralign = addrsize;
sec->sh.sh_link = find_section_by_name(elf, ".symtab")->idx;
sec->sh.sh_info = base->idx;
sec->sh.sh_flags = SHF_INFO_LINK;
diff --git a/tools/objtool/include/objtool/arch.h b/tools/objtool/include/objtool/arch.h
index fe2ea4b..4ecb480 100644
--- a/tools/objtool/include/objtool/arch.h
+++ b/tools/objtool/include/objtool/arch.h
@@ -69,6 +69,8 @@ struct stack_op {
struct instruction;
+int arch_ftrace_match(char *name);
+
void arch_initial_func_cfi_state(struct cfi_init_state *state);
int arch_decode_instruction(struct objtool_file *file, const struct section *sec,
diff --git a/tools/objtool/include/objtool/builtin.h b/tools/objtool/include/objtool/builtin.h
index c44ff39..fa45044 100644
--- a/tools/objtool/include/objtool/builtin.h
+++ b/tools/objtool/include/objtool/builtin.h
@@ -34,6 +34,7 @@ struct opts {
bool backup;
bool dryrun;
bool link;
+ bool mnop;
bool module;
bool no_unreachable;
bool sec_address;
diff --git a/tools/objtool/include/objtool/elf.h b/tools/objtool/include/objtool/elf.h
index bca719b..bb60fd4 100644
--- a/tools/objtool/include/objtool/elf.h
+++ b/tools/objtool/include/objtool/elf.h
@@ -145,6 +145,14 @@ static inline bool has_multiple_files(struct elf *elf)
return elf->num_files > 1;
}
+static inline int elf_class_addrsize(struct elf *elf)
+{
+ if (elf->ehdr.e_ident[EI_CLASS] == ELFCLASS32)
+ return sizeof(u32);
+ else
+ return sizeof(u64);
+}
+
struct elf *elf_open_read(const char *name, int flags);
struct section *elf_create_section(struct elf *elf, const char *name, unsigned int sh_flags, size_t entsize, int nr);
diff --git a/tools/objtool/include/objtool/endianness.h b/tools/objtool/include/objtool/endianness.h
index 1024134..4d2aa9b 100644
--- a/tools/objtool/include/objtool/endianness.h
+++ b/tools/objtool/include/objtool/endianness.h
@@ -2,33 +2,33 @@
#ifndef _OBJTOOL_ENDIANNESS_H
#define _OBJTOOL_ENDIANNESS_H
-#include <arch/endianness.h>
#include <linux/kernel.h>
#include <endian.h>
-
-#ifndef __TARGET_BYTE_ORDER
-#error undefined arch __TARGET_BYTE_ORDER
-#endif
-
-#if __BYTE_ORDER != __TARGET_BYTE_ORDER
-#define __NEED_BSWAP 1
-#else
-#define __NEED_BSWAP 0
-#endif
+#include <objtool/elf.h>
/*
- * Does a byte swap if target endianness doesn't match the host, i.e. cross
+ * Does a byte swap if target file endianness doesn't match the host, i.e. cross
* compilation for little endian on big endian and vice versa.
* To be used for multi-byte values conversion, which are read from / about
* to be written to a target native endianness ELF file.
*/
-#define bswap_if_needed(val) \
+static inline bool need_bswap(struct elf *elf)
+{
+ return (__BYTE_ORDER == __LITTLE_ENDIAN) ^
+ (elf->ehdr.e_ident[EI_DATA] == ELFDATA2LSB);
+}
+
+#define bswap_if_needed(elf, val) \
({ \
__typeof__(val) __ret; \
+ bool __need_bswap = need_bswap(elf); \
switch (sizeof(val)) { \
- case 8: __ret = __NEED_BSWAP ? bswap_64(val) : (val); break; \
- case 4: __ret = __NEED_BSWAP ? bswap_32(val) : (val); break; \
- case 2: __ret = __NEED_BSWAP ? bswap_16(val) : (val); break; \
+ case 8: \
+ __ret = __need_bswap ? bswap_64(val) : (val); break; \
+ case 4: \
+ __ret = __need_bswap ? bswap_32(val) : (val); break; \
+ case 2: \
+ __ret = __need_bswap ? bswap_16(val) : (val); break; \
default: \
BUILD_BUG(); break; \
} \
diff --git a/tools/objtool/orc_dump.c b/tools/objtool/orc_dump.c
index f5a8508c..4f1211f 100644
--- a/tools/objtool/orc_dump.c
+++ b/tools/objtool/orc_dump.c
@@ -76,6 +76,7 @@ int orc_dump(const char *_objname)
GElf_Rela rela;
GElf_Sym sym;
Elf_Data *data, *symtab = NULL, *rela_orc_ip = NULL;
+ struct elf dummy_elf = {};
objname = _objname;
@@ -94,6 +95,12 @@ int orc_dump(const char *_objname)
return -1;
}
+ if (!elf64_getehdr(elf)) {
+ WARN_ELF("elf64_getehdr");
+ return -1;
+ }
+ memcpy(&dummy_elf.ehdr, elf64_getehdr(elf), sizeof(dummy_elf.ehdr));
+
if (elf_getshdrnum(elf, &nr_sections)) {
WARN_ELF("elf_getshdrnum");
return -1;
@@ -198,11 +205,11 @@ int orc_dump(const char *_objname)
printf(" sp:");
- print_reg(orc[i].sp_reg, bswap_if_needed(orc[i].sp_offset));
+ print_reg(orc[i].sp_reg, bswap_if_needed(&dummy_elf, orc[i].sp_offset));
printf(" bp:");
- print_reg(orc[i].bp_reg, bswap_if_needed(orc[i].bp_offset));
+ print_reg(orc[i].bp_reg, bswap_if_needed(&dummy_elf, orc[i].bp_offset));
printf(" type:%s end:%d\n",
orc_type_name(orc[i].type), orc[i].end);
diff --git a/tools/objtool/orc_gen.c b/tools/objtool/orc_gen.c
index dd3c64a..1f22b7e 100644
--- a/tools/objtool/orc_gen.c
+++ b/tools/objtool/orc_gen.c
@@ -97,8 +97,8 @@ static int write_orc_entry(struct elf *elf, struct section *orc_sec,
/* populate ORC data */
orc = (struct orc_entry *)orc_sec->data->d_buf + idx;
memcpy(orc, o, sizeof(*orc));
- orc->sp_offset = bswap_if_needed(orc->sp_offset);
- orc->bp_offset = bswap_if_needed(orc->bp_offset);
+ orc->sp_offset = bswap_if_needed(elf, orc->sp_offset);
+ orc->bp_offset = bswap_if_needed(elf, orc->bp_offset);
/* populate reloc for ip */
if (elf_add_reloc_to_insn(elf, ip_sec, idx * sizeof(int), R_X86_64_PC32,
diff --git a/tools/objtool/special.c b/tools/objtool/special.c
index e2223dd..9c8d827 100644
--- a/tools/objtool/special.c
+++ b/tools/objtool/special.c
@@ -87,7 +87,8 @@ static int get_alt_entry(struct elf *elf, struct special_entry *entry,
if (entry->feature) {
unsigned short feature;
- feature = bswap_if_needed(*(unsigned short *)(sec->data->d_buf +
+ feature = bswap_if_needed(elf,
+ *(unsigned short *)(sec->data->d_buf +
offset +
entry->feature));
arch_handle_alternative(feature, alt);
diff --git a/tools/perf/.gitignore b/tools/perf/.gitignore
index fd7a6ff..05806ec 100644
--- a/tools/perf/.gitignore
+++ b/tools/perf/.gitignore
@@ -39,7 +39,12 @@
pmu-events/pmu-events.c
pmu-events/jevents
feature/
+libapi/
libbpf/
+libperf/
+libsubcmd/
+libsymbol/
+libtraceevent/
+libtraceevent_plugins/
fixdep
-libtraceevent-dynamic-list
Documentation/doc.dep
diff --git a/tools/perf/Build b/tools/perf/Build
index 496b096..6dd67e5 100644
--- a/tools/perf/Build
+++ b/tools/perf/Build
@@ -5,7 +5,6 @@
perf-y += builtin-evlist.o
perf-y += builtin-ftrace.o
perf-y += builtin-help.o
-perf-y += builtin-sched.o
perf-y += builtin-buildid-list.o
perf-y += builtin-buildid-cache.o
perf-y += builtin-kallsyms.o
@@ -13,11 +12,8 @@
perf-y += builtin-record.o
perf-y += builtin-report.o
perf-y += builtin-stat.o
-perf-y += builtin-timechart.o
perf-y += builtin-top.o
perf-y += builtin-script.o
-perf-y += builtin-kmem.o
-perf-y += builtin-lock.o
perf-y += builtin-kvm.o
perf-y += builtin-inject.o
perf-y += builtin-mem.o
@@ -25,9 +21,18 @@
perf-y += builtin-version.o
perf-y += builtin-c2c.o
perf-y += builtin-daemon.o
-perf-y += builtin-kwork.o
-perf-$(CONFIG_TRACE) += builtin-trace.o
+perf-$(CONFIG_LIBTRACEEVENT) += builtin-kmem.o
+perf-$(CONFIG_LIBTRACEEVENT) += builtin-kwork.o
+perf-$(CONFIG_LIBTRACEEVENT) += builtin-lock.o
+perf-$(CONFIG_LIBTRACEEVENT) += builtin-sched.o
+perf-$(CONFIG_LIBTRACEEVENT) += builtin-timechart.o
+
+ifeq ($(CONFIG_LIBTRACEEVENT),y)
+ perf-$(CONFIG_TRACE) += builtin-trace.o
+ perf-$(CONFIG_TRACE) += trace/beauty/
+endif
+
perf-$(CONFIG_LIBELF) += builtin-probe.o
perf-y += bench/
@@ -51,7 +56,6 @@
perf-y += util/
perf-y += arch/
perf-y += ui/
-perf-y += scripts/
-perf-$(CONFIG_TRACE) += trace/beauty/
+perf-$(CONFIG_LIBTRACEEVENT) += scripts/
gtk-y += ui/gtk/
diff --git a/tools/perf/Documentation/perf-annotate.txt b/tools/perf/Documentation/perf-annotate.txt
index 18fcc52..980fe2c 100644
--- a/tools/perf/Documentation/perf-annotate.txt
+++ b/tools/perf/Documentation/perf-annotate.txt
@@ -41,7 +41,7 @@
-q::
--quiet::
- Do not show any message. (Suppress -v)
+ Do not show any warnings or messages. (Suppress -v)
-n::
--show-nr-samples::
diff --git a/tools/perf/Documentation/perf-diff.txt b/tools/perf/Documentation/perf-diff.txt
index be65bd5..f3067a4 100644
--- a/tools/perf/Documentation/perf-diff.txt
+++ b/tools/perf/Documentation/perf-diff.txt
@@ -75,7 +75,7 @@
-q::
--quiet::
- Do not show any message. (Suppress -v)
+ Do not show any warnings or messages. (Suppress -v)
-f::
--force::
diff --git a/tools/perf/Documentation/perf-intel-pt.txt b/tools/perf/Documentation/perf-intel-pt.txt
index 92464a5..7b6ccd2 100644
--- a/tools/perf/Documentation/perf-intel-pt.txt
+++ b/tools/perf/Documentation/perf-intel-pt.txt
@@ -189,8 +189,16 @@
unpack the raw data for power events and PTWRITE. The script also displays
branches, and supports 2 additional modes selected by option:
- --insn-trace - instruction trace
- --src-trace - source trace
+ - --insn-trace - instruction trace
+ - --src-trace - source trace
+
+The intel-pt-events.py script also has options:
+
+ - --all-switch-events - display all switch events, not only the last consecutive.
+ - --interleave [<n>] - interleave sample output for the same timestamp so that
+ no more than n samples for a CPU are displayed in a row. 'n' defaults to 4.
+ Note this only affects the order of output, and only when the timestamp is the
+ same.
As mentioned above, it is easy to capture too much data. One way to limit the
data captured is to use 'snapshot' mode which is explained further below.
diff --git a/tools/perf/Documentation/perf-list.txt b/tools/perf/Documentation/perf-list.txt
index 57384a9..c5a3cb0 100644
--- a/tools/perf/Documentation/perf-list.txt
+++ b/tools/perf/Documentation/perf-list.txt
@@ -39,9 +39,13 @@
--deprecated::
Print deprecated events. By default the deprecated events are hidden.
---cputype::
-Print events applying cpu with this type for hybrid platform
-(e.g. --cputype core or --cputype atom)
+--unit::
+Print PMU events and metrics limited to the specific PMU name.
+(e.g. --unit cpu, --unit msr, --unit cpu_core, --unit cpu_atom)
+
+-j::
+--json::
+Output in JSON format.
[[EVENT_MODIFIERS]]
EVENT MODIFIERS
diff --git a/tools/perf/Documentation/perf-lock.txt b/tools/perf/Documentation/perf-lock.txt
index 3b1e165..38e79d45 100644
--- a/tools/perf/Documentation/perf-lock.txt
+++ b/tools/perf/Documentation/perf-lock.txt
@@ -42,7 +42,7 @@
-q::
--quiet::
- Do not show any message. (Suppress -v)
+ Do not show any warnings or messages. (Suppress -v)
-D::
--dump-raw-trace::
@@ -168,6 +168,10 @@
--entries=<value>::
Display this many entries.
+-l::
+--lock-addr::
+ Show lock contention stat by address
+
SEE ALSO
--------
diff --git a/tools/perf/Documentation/perf-probe.txt b/tools/perf/Documentation/perf-probe.txt
index 080981d..7f8e8ba 100644
--- a/tools/perf/Documentation/perf-probe.txt
+++ b/tools/perf/Documentation/perf-probe.txt
@@ -57,7 +57,7 @@
-q::
--quiet::
- Be quiet (do not show any messages including errors).
+ Do not show any warnings or messages.
Can not use with -v.
-a::
diff --git a/tools/perf/Documentation/perf-record.txt b/tools/perf/Documentation/perf-record.txt
index e41ae95..ff815c2 100644
--- a/tools/perf/Documentation/perf-record.txt
+++ b/tools/perf/Documentation/perf-record.txt
@@ -238,10 +238,6 @@
Also, by adding a comma, the number of mmap pages for AUX
area tracing can be specified.
---group::
- Put all events in a single event group. This precedes the --event
- option and remains only for backward compatibility. See --event.
-
-g::
Enables call-graph (stack chain/backtrace) recording for both
kernel space and user space.
@@ -282,7 +278,7 @@
-q::
--quiet::
- Don't print any message, useful for scripting.
+ Don't print any warnings or messages, useful for scripting.
-v::
--verbose::
@@ -388,6 +384,7 @@
- any_call: any function call or system call
- any_ret: any function return or system call return
- ind_call: any indirect branch
+ - ind_jmp: any indirect jump
- call: direct calls, including far (to/from kernel) calls
- u: only when the branch target is at the user level
- k: only when the branch target is in the kernel
@@ -396,6 +393,10 @@
- no_tx: only when the target is not in a hardware transaction
- abort_tx: only when the target is a hardware transaction abort
- cond: conditional branches
+ - call_stack: save call stack
+ - no_flags: don't save branch flags e.g prediction, misprediction etc
+ - no_cycles: don't save branch cycles
+ - hw_index: save branch hardware index
- save_type: save branch type during sampling in case binary is not available later
For the platforms with Intel Arch LBR support (12th-Gen+ client or
4th-Gen Xeon+ server), the save branch type is unconditionally enabled
diff --git a/tools/perf/Documentation/perf-report.txt b/tools/perf/Documentation/perf-report.txt
index 4533db2..4fa509b 100644
--- a/tools/perf/Documentation/perf-report.txt
+++ b/tools/perf/Documentation/perf-report.txt
@@ -27,7 +27,7 @@
-q::
--quiet::
- Do not show any message. (Suppress -v)
+ Do not show any warnings or messages. (Suppress -v)
-n::
--show-nr-samples::
diff --git a/tools/perf/Documentation/perf-stat.txt b/tools/perf/Documentation/perf-stat.txt
index d7ff186..18abdc1 100644
--- a/tools/perf/Documentation/perf-stat.txt
+++ b/tools/perf/Documentation/perf-stat.txt
@@ -354,8 +354,8 @@
may be used to increase accuracy in this case.
--quiet::
-Don't print output. This is useful with perf stat record below to only
-write data to the perf.data file.
+Don't print output, warnings or messages. This is useful with perf stat
+record below to only write data to the perf.data file.
STAT RECORD
-----------
diff --git a/tools/perf/Documentation/perf-top.txt b/tools/perf/Documentation/perf-top.txt
index c1fdba2..e534d70 100644
--- a/tools/perf/Documentation/perf-top.txt
+++ b/tools/perf/Documentation/perf-top.txt
@@ -51,9 +51,6 @@
--count-filter=<count>::
Only display functions with more events than this.
---group::
- Put the counters into a counter group.
-
--group-sort-idx::
Sort the output by the event at the index n in group. If n is invalid,
sort by the first event. It can support multiple groups with different
@@ -313,10 +310,10 @@
perf top -e cycles,probe:icmp_rcv --switch-on=probe:icmp_rcv
- Alternatively one can ask for --group and then two overhead columns
+ Alternatively one can ask for a group and then two overhead columns
will appear, the first for cycles and the second for the switch-on event.
- perf top --group -e cycles,probe:icmp_rcv --switch-on=probe:icmp_rcv
+ perf top -e '{cycles,probe:icmp_rcv}' --switch-on=probe:icmp_rcv
This may be interesting to measure a workload only after some initialization
phase is over, i.e. insert a perf probe at that point and use the above
diff --git a/tools/perf/MANIFEST b/tools/perf/MANIFEST
index f5d72f9..1da7f4b 100644
--- a/tools/perf/MANIFEST
+++ b/tools/perf/MANIFEST
@@ -3,7 +3,6 @@
tools/scripts
tools/build
tools/include
-tools/lib/traceevent
tools/lib/api
tools/lib/bpf
tools/lib/subcmd
@@ -13,8 +12,7 @@
tools/lib/hweight.c
tools/lib/rbtree.c
tools/lib/string.c
-tools/lib/symbol/kallsyms.c
-tools/lib/symbol/kallsyms.h
+tools/lib/symbol
tools/lib/find_bit.c
tools/lib/bitmap.c
tools/lib/list_sort.c
diff --git a/tools/perf/Makefile b/tools/perf/Makefile
index f3fe360..75f3f6e 100644
--- a/tools/perf/Makefile
+++ b/tools/perf/Makefile
@@ -25,7 +25,7 @@
# (To override it, run 'make JOBS=1' and similar.)
#
ifeq ($(JOBS),)
- JOBS := $(shell (getconf _NPROCESSORS_ONLN || egrep -c '^processor|^CPU[0-9]' /proc/cpuinfo) 2>/dev/null)
+ JOBS := $(shell (getconf _NPROCESSORS_ONLN || grep -E -c '^processor|^CPU[0-9]' /proc/cpuinfo) 2>/dev/null)
ifeq ($(JOBS),0)
JOBS := 1
endif
diff --git a/tools/perf/Makefile.config b/tools/perf/Makefile.config
index 898226e..83ed969 100644
--- a/tools/perf/Makefile.config
+++ b/tools/perf/Makefile.config
@@ -307,7 +307,7 @@
CORE_CFLAGS += -funwind-tables
CORE_CFLAGS += -Wall
CORE_CFLAGS += -Wextra
-CORE_CFLAGS += -std=gnu99
+CORE_CFLAGS += -std=gnu11
CXXFLAGS += -std=gnu++14 -fno-exceptions -fno-rtti
CXXFLAGS += -Wall
@@ -349,7 +349,6 @@
endif
endif
-INC_FLAGS += -I$(srctree)/tools/lib/perf/include
INC_FLAGS += -I$(src-perf)/util/include
INC_FLAGS += -I$(src-perf)/arch/$(SRCARCH)/include
INC_FLAGS += -I$(srctree)/tools/include/
@@ -367,7 +366,6 @@
INC_FLAGS += -I$(src-perf)/util
INC_FLAGS += -I$(src-perf)
-INC_FLAGS += -I$(srctree)/tools/lib/
CORE_CFLAGS += -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64 -D_GNU_SOURCE
@@ -765,18 +763,20 @@
EXTLIBS += $(EXTLIBS_LIBUNWIND)
endif
-ifeq ($(NO_SYSCALL_TABLE),0)
- $(call detected,CONFIG_TRACE)
-else
- ifndef NO_LIBAUDIT
- $(call feature_check,libaudit)
- ifneq ($(feature-libaudit), 1)
- msg := $(warning No libaudit.h found, disables 'trace' tool, please install audit-libs-devel or libaudit-dev);
- NO_LIBAUDIT := 1
- else
- CFLAGS += -DHAVE_LIBAUDIT_SUPPORT
- EXTLIBS += -laudit
- $(call detected,CONFIG_TRACE)
+ifneq ($(NO_LIBTRACEEVENT),1)
+ ifeq ($(NO_SYSCALL_TABLE),0)
+ $(call detected,CONFIG_TRACE)
+ else
+ ifndef NO_LIBAUDIT
+ $(call feature_check,libaudit)
+ ifneq ($(feature-libaudit), 1)
+ msg := $(warning No libaudit.h found, disables 'trace' tool, please install audit-libs-devel or libaudit-dev);
+ NO_LIBAUDIT := 1
+ else
+ CFLAGS += -DHAVE_LIBAUDIT_SUPPORT
+ EXTLIBS += -laudit
+ $(call detected,CONFIG_TRACE)
+ endif
endif
endif
endif
@@ -871,6 +871,7 @@
NO_LIBPYTHON := 1
endef
+PYTHON_EXTENSION_SUFFIX := '.so'
ifdef NO_LIBPYTHON
$(call disable-python,Python support disabled by user)
else
@@ -889,7 +890,8 @@
else
LDFLAGS += $(PYTHON_EMBED_LDFLAGS)
EXTLIBS += $(PYTHON_EMBED_LIBADD)
- LANG_BINDINGS += $(obj-perf)python/perf.so
+ PYTHON_EXTENSION_SUFFIX := $(shell $(PYTHON) -c 'from importlib import machinery; print(machinery.EXTENSION_SUFFIXES[0])')
+ LANG_BINDINGS += $(obj-perf)python/perf$(PYTHON_EXTENSION_SUFFIX)
CFLAGS += -DHAVE_LIBPYTHON_SUPPORT
$(call detected,CONFIG_LIBPYTHON)
endif
@@ -1184,9 +1186,11 @@
endif
endif
-ifdef LIBTRACEEVENT_DYNAMIC
+# libtraceevent is a recommended dependency picked up from the system.
+ifneq ($(NO_LIBTRACEEVENT),1)
$(call feature_check,libtraceevent)
ifeq ($(feature-libtraceevent), 1)
+ CFLAGS += -DHAVE_LIBTRACEEVENT
EXTLIBS += -ltraceevent
LIBTRACEEVENT_VERSION := $(shell $(PKG_CONFIG) --modversion libtraceevent)
LIBTRACEEVENT_VERSION_1 := $(word 1, $(subst ., ,$(LIBTRACEEVENT_VERSION)))
@@ -1194,12 +1198,15 @@
LIBTRACEEVENT_VERSION_3 := $(word 3, $(subst ., ,$(LIBTRACEEVENT_VERSION)))
LIBTRACEEVENT_VERSION_CPP := $(shell expr $(LIBTRACEEVENT_VERSION_1) \* 255 \* 255 + $(LIBTRACEEVENT_VERSION_2) \* 255 + $(LIBTRACEEVENT_VERSION_3))
CFLAGS += -DLIBTRACEEVENT_VERSION=$(LIBTRACEEVENT_VERSION_CPP)
+ $(call detected,CONFIG_LIBTRACEEVENT)
+ LIBTRACEEVENT_VERSION_WITH_TEP_FIELD_IS_RELATIVE := $(shell expr 1 \* 255 \* 255 + 5 \* 255 + 0) # 1.5.0
+ ifeq ($(shell test $(LIBTRACEEVENT_VERSION_CPP) -gt $(LIBTRACEEVENT_VERSION_WITH_TEP_FIELD_IS_RELATIVE); echo $$?),0)
+ CFLAGS += -DHAVE_LIBTRACEEVENT_TEP_FIELD_IS_RELATIVE
+ endif
else
- dummy := $(error Error: No libtraceevent devel library found, please install libtraceevent-devel);
+ dummy := $(warning Warning: libtraceevent is missing limiting functionality, please install libtraceevent-dev/libtraceevent-devel)
endif
-endif
-ifdef LIBTRACEFS_DYNAMIC
$(call feature_check,libtracefs)
ifeq ($(feature-libtracefs), 1)
EXTLIBS += -ltracefs
@@ -1209,14 +1216,12 @@
LIBTRACEFS_VERSION_3 := $(word 3, $(subst ., ,$(LIBTRACEFS_VERSION)))
LIBTRACEFS_VERSION_CPP := $(shell expr $(LIBTRACEFS_VERSION_1) \* 255 \* 255 + $(LIBTRACEFS_VERSION_2) \* 255 + $(LIBTRACEFS_VERSION_3))
CFLAGS += -DLIBTRACEFS_VERSION=$(LIBTRACEFS_VERSION_CPP)
- else
- dummy := $(error Error: No libtracefs devel library found, please install libtracefs-dev);
endif
endif
# Among the variables below, these:
# perfexecdir
-# perf_include_dir
+# libbpf_include_dir
# perf_examples_dir
# template_dir
# mandir
@@ -1239,7 +1244,8 @@
mandir = share/man
infodir = share/info
perfexecdir = libexec/perf-core
-perf_include_dir = lib/perf/include
+# FIXME: system's libbpf header directory, where we expect to find bpf/bpf_helpers.h, for instance
+libbpf_include_dir = /usr/include
perf_examples_dir = lib/perf/examples
sharedir = $(prefix)/share
template_dir = share/perf-core/templates
@@ -1272,7 +1278,7 @@
mandir_SQ = $(subst ','\'',$(mandir))
infodir_SQ = $(subst ','\'',$(infodir))
perfexecdir_SQ = $(subst ','\'',$(perfexecdir))
-perf_include_dir_SQ = $(subst ','\'',$(perf_include_dir))
+libbpf_include_dir_SQ = $(subst ','\'',$(libbpf_include_dir))
perf_examples_dir_SQ = $(subst ','\'',$(perf_examples_dir))
template_dir_SQ = $(subst ','\'',$(template_dir))
htmldir_SQ = $(subst ','\'',$(htmldir))
@@ -1284,13 +1290,13 @@
ifneq ($(filter /%,$(firstword $(perfexecdir))),)
perfexec_instdir = $(perfexecdir)
-perf_include_instdir = $(perf_include_dir)
+perf_include_instdir = $(libbpf_include_dir)
perf_examples_instdir = $(perf_examples_dir)
STRACE_GROUPS_INSTDIR = $(STRACE_GROUPS_DIR)
tip_instdir = $(tipdir)
else
perfexec_instdir = $(prefix)/$(perfexecdir)
-perf_include_instdir = $(prefix)/$(perf_include_dir)
+perf_include_instdir = $(prefix)/$(libbpf_include_dir)
perf_examples_instdir = $(prefix)/$(perf_examples_dir)
STRACE_GROUPS_INSTDIR = $(prefix)/$(STRACE_GROUPS_DIR)
tip_instdir = $(prefix)/$(tipdir)
@@ -1352,7 +1358,7 @@
$(call detected_var,STRACE_GROUPS_DIR_SQ)
$(call detected_var,prefix_SQ)
$(call detected_var,perfexecdir_SQ)
-$(call detected_var,perf_include_dir_SQ)
+$(call detected_var,libbpf_include_dir_SQ)
$(call detected_var,perf_examples_dir_SQ)
$(call detected_var,tipdir_SQ)
$(call detected_var,srcdir_SQ)
diff --git a/tools/perf/Makefile.perf b/tools/perf/Makefile.perf
index a432e59..9b7886c 100644
--- a/tools/perf/Makefile.perf
+++ b/tools/perf/Makefile.perf
@@ -128,10 +128,6 @@
#
# Define BUILD_BPF_SKEL to enable BPF skeletons
#
-# Define LIBTRACEEVENT_DYNAMIC to enable libtraceevent dynamic linking
-#
-# Define LIBTRACEFS_DYNAMIC to enable libtracefs dynamic linking
-#
# As per kernel Makefile, avoid funny character set dependencies
unexport LC_ALL
@@ -241,10 +237,10 @@
else # force_fixdep
-LIB_DIR = $(srctree)/tools/lib/api/
-TRACE_EVENT_DIR = $(srctree)/tools/lib/traceevent/
+LIBAPI_DIR = $(srctree)/tools/lib/api/
LIBBPF_DIR = $(srctree)/tools/lib/bpf/
-SUBCMD_DIR = $(srctree)/tools/lib/subcmd/
+LIBSUBCMD_DIR = $(srctree)/tools/lib/subcmd/
+LIBSYMBOL_DIR = $(srctree)/tools/lib/symbol/
LIBPERF_DIR = $(srctree)/tools/lib/perf/
DOC_DIR = $(srctree)/tools/perf/Documentation/
@@ -292,36 +288,15 @@
strip-libs = $(filter-out -l%,$(1))
ifneq ($(OUTPUT),)
- TE_PATH=$(OUTPUT)
- PLUGINS_PATH=$(OUTPUT)
- SUBCMD_PATH=$(OUTPUT)
- LIBPERF_PATH=$(OUTPUT)
-ifneq ($(subdir),)
- API_PATH=$(OUTPUT)/../lib/api/
+ LIBAPI_OUTPUT = $(abspath $(OUTPUT))/libapi
else
- API_PATH=$(OUTPUT)
+ LIBAPI_OUTPUT = $(CURDIR)/libapi
endif
-else
- TE_PATH=$(TRACE_EVENT_DIR)
- PLUGINS_PATH=$(TRACE_EVENT_DIR)plugins/
- API_PATH=$(LIB_DIR)
- SUBCMD_PATH=$(SUBCMD_DIR)
- LIBPERF_PATH=$(LIBPERF_DIR)
-endif
-
-LIBTRACEEVENT = $(TE_PATH)libtraceevent.a
-export LIBTRACEEVENT
-LIBTRACEEVENT_DYNAMIC_LIST = $(PLUGINS_PATH)libtraceevent-dynamic-list
-
-#
-# The static build has no dynsym table, so this does not work for
-# static build. Looks like linker starts to scream about that now
-# (in Fedora 26) so we need to switch it off for static build.
-DYNAMIC_LIST_LDFLAGS = -Xlinker --dynamic-list=$(LIBTRACEEVENT_DYNAMIC_LIST)
-LIBTRACEEVENT_DYNAMIC_LIST_LDFLAGS = $(if $(findstring -static,$(LDFLAGS)),,$(DYNAMIC_LIST_LDFLAGS))
-
-LIBAPI = $(API_PATH)libapi.a
+LIBAPI_DESTDIR = $(LIBAPI_OUTPUT)
+LIBAPI_INCLUDE = $(LIBAPI_DESTDIR)/include
+LIBAPI = $(LIBAPI_OUTPUT)/libapi.a
export LIBAPI
+CFLAGS += -I$(LIBAPI_OUTPUT)/include
ifneq ($(OUTPUT),)
LIBBPF_OUTPUT = $(abspath $(OUTPUT))/libbpf
@@ -331,11 +306,38 @@
LIBBPF_DESTDIR = $(LIBBPF_OUTPUT)
LIBBPF_INCLUDE = $(LIBBPF_DESTDIR)/include
LIBBPF = $(LIBBPF_OUTPUT)/libbpf.a
+CFLAGS += -I$(LIBBPF_OUTPUT)/include
-LIBSUBCMD = $(SUBCMD_PATH)libsubcmd.a
+ifneq ($(OUTPUT),)
+ LIBSUBCMD_OUTPUT = $(abspath $(OUTPUT))/libsubcmd
+else
+ LIBSUBCMD_OUTPUT = $(CURDIR)/libsubcmd
+endif
+LIBSUBCMD_DESTDIR = $(LIBSUBCMD_OUTPUT)
+LIBSUBCMD_INCLUDE = $(LIBSUBCMD_DESTDIR)/include
+LIBSUBCMD = $(LIBSUBCMD_OUTPUT)/libsubcmd.a
+CFLAGS += -I$(LIBSUBCMD_OUTPUT)/include
-LIBPERF = $(LIBPERF_PATH)libperf.a
+ifneq ($(OUTPUT),)
+ LIBSYMBOL_OUTPUT = $(abspath $(OUTPUT))/libsymbol
+else
+ LIBSYMBOL_OUTPUT = $(CURDIR)/libsymbol
+endif
+LIBSYMBOL_DESTDIR = $(LIBSYMBOL_OUTPUT)
+LIBSYMBOL_INCLUDE = $(LIBSYMBOL_DESTDIR)/include
+LIBSYMBOL = $(LIBSYMBOL_OUTPUT)/libsymbol.a
+CFLAGS += -I$(LIBSYMBOL_OUTPUT)/include
+
+ifneq ($(OUTPUT),)
+ LIBPERF_OUTPUT = $(abspath $(OUTPUT))/libperf
+else
+ LIBPERF_OUTPUT = $(CURDIR)/libperf
+endif
+LIBPERF_DESTDIR = $(LIBPERF_OUTPUT)
+LIBPERF_INCLUDE = $(LIBPERF_DESTDIR)/include
+LIBPERF = $(LIBPERF_OUTPUT)/libperf.a
export LIBPERF
+CFLAGS += -I$(LIBPERF_OUTPUT)/include
# python extension build directories
PYTHON_EXTBUILD := $(OUTPUT)python_ext_build/
@@ -345,8 +347,13 @@
python-clean := $(call QUIET_CLEAN, python) $(RM) -r $(PYTHON_EXTBUILD) $(OUTPUT)python/perf*.so
-PYTHON_EXT_SRCS := $(shell grep -v ^\# util/python-ext-sources)
-PYTHON_EXT_DEPS := util/python-ext-sources util/setup.py $(LIBTRACEEVENT) $(LIBAPI)
+ifeq ($(CONFIG_LIBTRACEEVENT),y)
+ PYTHON_EXT_SRCS := $(shell grep -v ^\# util/python-ext-sources)
+else
+ PYTHON_EXT_SRCS := $(shell grep -v '^\#\|util/trace-event.c' util/python-ext-sources)
+endif
+
+PYTHON_EXT_DEPS := util/python-ext-sources util/setup.py $(LIBAPI)
SCRIPTS = $(patsubst %.sh,%,$(SCRIPT_SH))
@@ -385,15 +392,12 @@
export PERL_PATH
-PERFLIBS = $(LIBAPI) $(LIBSUBCMD) $(LIBPERF)
+PERFLIBS = $(LIBAPI) $(LIBPERF) $(LIBSUBCMD) $(LIBSYMBOL)
ifndef NO_LIBBPF
ifndef LIBBPF_DYNAMIC
PERFLIBS += $(LIBBPF)
endif
endif
-ifndef LIBTRACEEVENT_DYNAMIC
- PERFLIBS += $(LIBTRACEEVENT)
-endif
# We choose to avoid "if .. else if .. else .. endif endif"
# because maintaining the nesting to match is a pain. If
@@ -643,9 +647,9 @@
# Create python binding output directory if not already present
_dummy := $(shell [ -d '$(OUTPUT)python' ] || mkdir -p '$(OUTPUT)python')
-$(OUTPUT)python/perf.so: $(PYTHON_EXT_SRCS) $(PYTHON_EXT_DEPS) $(LIBTRACEEVENT_DYNAMIC_LIST) $(LIBPERF)
+$(OUTPUT)python/perf$(PYTHON_EXTENSION_SUFFIX): $(PYTHON_EXT_SRCS) $(PYTHON_EXT_DEPS) $(LIBPERF)
$(QUIET_GEN)LDSHARED="$(CC) -pthread -shared" \
- CFLAGS='$(CFLAGS)' LDFLAGS='$(LDFLAGS) $(LIBTRACEEVENT_DYNAMIC_LIST_LDFLAGS)' \
+ CFLAGS='$(CFLAGS)' LDFLAGS='$(LDFLAGS)' \
$(PYTHON_WORD) util/setup.py \
--quiet build_ext; \
cp $(PYTHON_EXTBUILD_LIB)perf*.so $(OUTPUT)python/
@@ -668,14 +672,14 @@
$(PERF_IN): prepare FORCE
$(Q)$(MAKE) $(build)=perf
-$(PMU_EVENTS_IN): FORCE
+$(PMU_EVENTS_IN): FORCE prepare
$(Q)$(MAKE) -f $(srctree)/tools/build/Makefile.build dir=pmu-events obj=pmu-events
-$(OUTPUT)perf: $(PERFLIBS) $(PERF_IN) $(PMU_EVENTS_IN) $(LIBTRACEEVENT_DYNAMIC_LIST)
- $(QUIET_LINK)$(CC) $(CFLAGS) $(LDFLAGS) $(LIBTRACEEVENT_DYNAMIC_LIST_LDFLAGS) \
+$(OUTPUT)perf: $(PERFLIBS) $(PERF_IN) $(PMU_EVENTS_IN)
+ $(QUIET_LINK)$(CC) $(CFLAGS) $(LDFLAGS) \
$(PERF_IN) $(PMU_EVENTS_IN) $(LIBS) -o $@
-$(GTK_IN): FORCE
+$(GTK_IN): FORCE prepare
$(Q)$(MAKE) $(build)=gtk
$(OUTPUT)libperf-gtk.so: $(GTK_IN) $(PERFLIBS)
@@ -751,6 +755,11 @@
$(rename_flags_array) \
$(arch_errno_name_array) \
$(sync_file_range_arrays) \
+ $(LIBAPI) \
+ $(LIBBPF) \
+ $(LIBPERF) \
+ $(LIBSUBCMD) \
+ $(LIBSYMBOL) \
bpf-skel
$(OUTPUT)%.o: %.c prepare FORCE
@@ -808,30 +817,14 @@
$(patsubst perf-%,%.o,$(PROGRAMS)): $(wildcard */*.h)
-LIBTRACEEVENT_FLAGS += plugin_dir=$(plugindir_SQ) 'EXTRA_CFLAGS=$(EXTRA_CFLAGS)' 'LDFLAGS=$(filter-out -static,$(LDFLAGS))'
-
-$(LIBTRACEEVENT): FORCE
- $(Q)$(MAKE) -C $(TRACE_EVENT_DIR) $(LIBTRACEEVENT_FLAGS) O=$(OUTPUT) $(OUTPUT)libtraceevent.a
-
-libtraceevent_plugins: FORCE
- $(Q)$(MAKE) -C $(TRACE_EVENT_DIR)plugins $(LIBTRACEEVENT_FLAGS) O=$(OUTPUT) plugins
-
-$(LIBTRACEEVENT_DYNAMIC_LIST): libtraceevent_plugins
- $(Q)$(MAKE) -C $(TRACE_EVENT_DIR)plugins $(LIBTRACEEVENT_FLAGS) O=$(OUTPUT) $(OUTPUT)libtraceevent-dynamic-list
-
-$(LIBTRACEEVENT)-clean:
- $(call QUIET_CLEAN, libtraceevent)
- $(Q)$(MAKE) -C $(TRACE_EVENT_DIR) O=$(OUTPUT) clean >/dev/null
-
-install-traceevent-plugins: libtraceevent_plugins
- $(Q)$(MAKE) -C $(TRACE_EVENT_DIR) $(LIBTRACEEVENT_FLAGS) O=$(OUTPUT) install_plugins
-
-$(LIBAPI): FORCE
- $(Q)$(MAKE) -C $(LIB_DIR) O=$(OUTPUT) $(OUTPUT)libapi.a
+$(LIBAPI): FORCE | $(LIBAPI_OUTPUT)
+ $(Q)$(MAKE) -C $(LIBAPI_DIR) O=$(LIBAPI_OUTPUT) \
+ DESTDIR=$(LIBAPI_DESTDIR) prefix= \
+ $@ install_headers
$(LIBAPI)-clean:
$(call QUIET_CLEAN, libapi)
- $(Q)$(MAKE) -C $(LIB_DIR) O=$(OUTPUT) clean >/dev/null
+ $(Q)$(RM) -r -- $(LIBAPI_OUTPUT)
$(LIBBPF): FORCE | $(LIBBPF_OUTPUT)
$(Q)$(MAKE) -C $(LIBBPF_DIR) FEATURES_DUMP=$(FEATURE_DUMP_EXPORT) \
@@ -842,18 +835,32 @@
$(call QUIET_CLEAN, libbpf)
$(Q)$(RM) -r -- $(LIBBPF_OUTPUT)
-$(LIBPERF): FORCE
- $(Q)$(MAKE) -C $(LIBPERF_DIR) EXTRA_CFLAGS="$(LIBPERF_CFLAGS)" O=$(OUTPUT) $(OUTPUT)libperf.a
+$(LIBPERF): FORCE | $(LIBPERF_OUTPUT)
+ $(Q)$(MAKE) -C $(LIBPERF_DIR) O=$(LIBPERF_OUTPUT) \
+ DESTDIR=$(LIBPERF_DESTDIR) prefix= \
+ $@ install_headers
$(LIBPERF)-clean:
$(call QUIET_CLEAN, libperf)
- $(Q)$(MAKE) -C $(LIBPERF_DIR) O=$(OUTPUT) clean >/dev/null
+ $(Q)$(RM) -r -- $(LIBPERF_OUTPUT)
-$(LIBSUBCMD): FORCE
- $(Q)$(MAKE) -C $(SUBCMD_DIR) O=$(OUTPUT) $(OUTPUT)libsubcmd.a
+$(LIBSUBCMD): FORCE | $(LIBSUBCMD_OUTPUT)
+ $(Q)$(MAKE) -C $(LIBSUBCMD_DIR) O=$(LIBSUBCMD_OUTPUT) \
+ DESTDIR=$(LIBSUBCMD_DESTDIR) prefix= \
+ $@ install_headers
$(LIBSUBCMD)-clean:
- $(Q)$(MAKE) -C $(SUBCMD_DIR) O=$(OUTPUT) clean
+ $(call QUIET_CLEAN, libsubcmd)
+ $(Q)$(RM) -r -- $(LIBSUBCMD_OUTPUT)
+
+$(LIBSYMBOL): FORCE | $(LIBSYMBOL_OUTPUT)
+ $(Q)$(MAKE) -C $(LIBSYMBOL_DIR) O=$(LIBSYMBOL_OUTPUT) \
+ DESTDIR=$(LIBSYMBOL_DESTDIR) prefix= \
+ $@ install_headers
+
+$(LIBSYMBOL)-clean:
+ $(call QUIET_CLEAN, libsymbol)
+ $(Q)$(RM) -r -- $(LIBSYMBOL_OUTPUT)
help:
@echo 'Perf make targets:'
@@ -960,11 +967,6 @@
$(call QUIET_INSTALL, libexec) \
$(INSTALL) -d -m 755 '$(DESTDIR_SQ)$(perfexec_instdir_SQ)'
ifndef NO_LIBBPF
- $(call QUIET_INSTALL, bpf-headers) \
- $(INSTALL) -d -m 755 '$(DESTDIR_SQ)$(perf_include_instdir_SQ)/bpf'; \
- $(INSTALL) -d -m 755 '$(DESTDIR_SQ)$(perf_include_instdir_SQ)/bpf/linux'; \
- $(INSTALL) include/bpf/*.h -m 644 -t '$(DESTDIR_SQ)$(perf_include_instdir_SQ)/bpf'; \
- $(INSTALL) include/bpf/linux/*.h -m 644 -t '$(DESTDIR_SQ)$(perf_include_instdir_SQ)/bpf/linux'
$(call QUIET_INSTALL, bpf-examples) \
$(INSTALL) -d -m 755 '$(DESTDIR_SQ)$(perf_examples_instdir_SQ)/bpf'; \
$(INSTALL) examples/bpf/*.c -m 644 -t '$(DESTDIR_SQ)$(perf_examples_instdir_SQ)/bpf'
@@ -1020,7 +1022,7 @@
$(INSTALL) tests/shell/coresight/*.sh '$(DESTDIR_SQ)$(perfexec_instdir_SQ)/tests/shell/coresight'
$(Q)$(MAKE) -C tests/shell/coresight install-tests
-install-bin: install-tools install-tests install-traceevent-plugins
+install-bin: install-tools install-tests
install: install-bin try-install-man
@@ -1044,7 +1046,7 @@
SKELETONS += $(SKEL_OUT)/off_cpu.skel.h $(SKEL_OUT)/lock_contention.skel.h
SKELETONS += $(SKEL_OUT)/kwork_trace.skel.h
-$(SKEL_TMP_OUT) $(LIBBPF_OUTPUT):
+$(SKEL_TMP_OUT) $(LIBAPI_OUTPUT) $(LIBBPF_OUTPUT) $(LIBPERF_OUTPUT) $(LIBSUBCMD_OUTPUT) $(LIBSYMBOL_OUTPUT):
$(Q)$(MKDIR) -p $@
ifdef BUILD_BPF_SKEL
@@ -1089,7 +1091,7 @@
bpf-skel-clean:
$(call QUIET_CLEAN, bpf-skel) $(RM) -r $(SKEL_TMP_OUT) $(SKELETONS)
-clean:: $(LIBTRACEEVENT)-clean $(LIBAPI)-clean $(LIBBPF)-clean $(LIBSUBCMD)-clean $(LIBPERF)-clean fixdep-clean python-clean bpf-skel-clean tests-coresight-targets-clean
+clean:: $(LIBAPI)-clean $(LIBBPF)-clean $(LIBSUBCMD)-clean $(LIBSYMBOL)-clean $(LIBPERF)-clean fixdep-clean python-clean bpf-skel-clean tests-coresight-targets-clean
$(call QUIET_CLEAN, core-objs) $(RM) $(LIBPERF_A) $(OUTPUT)perf-archive $(OUTPUT)perf-iostat $(LANG_BINDINGS)
$(Q)find $(or $(OUTPUT),.) -name '*.o' -delete -o -name '\.*.cmd' -delete -o -name '\.*.d' -delete
$(Q)$(RM) $(OUTPUT).config-detected
@@ -1146,6 +1148,6 @@
.PHONY: all install clean config-clean strip install-gtk
.PHONY: shell_compatibility_test please_set_SHELL_PATH_to_a_more_modern_shell
.PHONY: .FORCE-PERF-VERSION-FILE TAGS tags cscope FORCE prepare
-.PHONY: libtraceevent_plugins archheaders
+.PHONY: archheaders
endif # force_fixdep
diff --git a/tools/perf/arch/arm/util/unwind-libdw.c b/tools/perf/arch/arm/util/unwind-libdw.c
index b7692cb..1834a0c 100644
--- a/tools/perf/arch/arm/util/unwind-libdw.c
+++ b/tools/perf/arch/arm/util/unwind-libdw.c
@@ -2,7 +2,7 @@
#include <elfutils/libdwfl.h>
#include "../../../util/unwind-libdw.h"
#include "../../../util/perf_regs.h"
-#include "../../../util/event.h"
+#include "../../../util/sample.h"
bool libdw__arch_set_initial_registers(Dwfl_Thread *thread, void *arg)
{
diff --git a/tools/perf/arch/arm64/entry/syscalls/mksyscalltbl b/tools/perf/arch/arm64/entry/syscalls/mksyscalltbl
index 459469b..a7ca48d 100755
--- a/tools/perf/arch/arm64/entry/syscalls/mksyscalltbl
+++ b/tools/perf/arch/arm64/entry/syscalls/mksyscalltbl
@@ -58,5 +58,5 @@
$gcc -E -dM -x c -I $incpath/include/uapi $input \
|sed -ne 's/^#define __NR_//p' \
- |sort -t' ' -k2 -nu \
+ |sort -t' ' -k2 -n \
|create_table
diff --git a/tools/perf/arch/arm64/util/Build b/tools/perf/arch/arm64/util/Build
index 337aa9b..78ef711 100644
--- a/tools/perf/arch/arm64/util/Build
+++ b/tools/perf/arch/arm64/util/Build
@@ -3,7 +3,7 @@
perf-y += perf_regs.o
perf-y += tsc.o
perf-y += pmu.o
-perf-y += kvm-stat.o
+perf-$(CONFIG_LIBTRACEEVENT) += kvm-stat.o
perf-$(CONFIG_DWARF) += dwarf-regs.o
perf-$(CONFIG_LOCAL_LIBUNWIND) += unwind-libunwind.o
perf-$(CONFIG_LIBDW_DWARF_UNWIND) += unwind-libdw.o
diff --git a/tools/perf/arch/arm64/util/machine.c b/tools/perf/arch/arm64/util/machine.c
index 41c1596..235a0a1 100644
--- a/tools/perf/arch/arm64/util/machine.c
+++ b/tools/perf/arch/arm64/util/machine.c
@@ -7,6 +7,7 @@
#include "symbol.h"
#include "callchain.h"
#include "record.h"
+#include "util/perf_regs.h"
void arch__add_leaf_frame_record_opts(struct record_opts *opts)
{
diff --git a/tools/perf/arch/arm64/util/pmu.c b/tools/perf/arch/arm64/util/pmu.c
index f849b1e..477e513 100644
--- a/tools/perf/arch/arm64/util/pmu.c
+++ b/tools/perf/arch/arm64/util/pmu.c
@@ -1,5 +1,6 @@
// SPDX-License-Identifier: GPL-2.0
+#include <internal/cpumap.h>
#include "../../../util/cpumap.h"
#include "../../../util/pmu.h"
diff --git a/tools/perf/arch/arm64/util/unwind-libdw.c b/tools/perf/arch/arm64/util/unwind-libdw.c
index a509416..0938508 100644
--- a/tools/perf/arch/arm64/util/unwind-libdw.c
+++ b/tools/perf/arch/arm64/util/unwind-libdw.c
@@ -2,7 +2,7 @@
#include <elfutils/libdwfl.h>
#include "../../../util/unwind-libdw.h"
#include "../../../util/perf_regs.h"
-#include "../../../util/event.h"
+#include "../../../util/sample.h"
bool libdw__arch_set_initial_registers(Dwfl_Thread *thread, void *arg)
{
diff --git a/tools/perf/arch/powerpc/util/Build b/tools/perf/arch/powerpc/util/Build
index 0115f31..9889245 100644
--- a/tools/perf/arch/powerpc/util/Build
+++ b/tools/perf/arch/powerpc/util/Build
@@ -1,5 +1,5 @@
perf-y += header.o
-perf-y += kvm-stat.o
+perf-$(CONFIG_LIBTRACEEVENT) += kvm-stat.o
perf-y += perf_regs.o
perf-y += mem-events.o
perf-y += sym-handling.o
diff --git a/tools/perf/arch/powerpc/util/event.c b/tools/perf/arch/powerpc/util/event.c
index cf430a4..77d8cc2 100644
--- a/tools/perf/arch/powerpc/util/event.c
+++ b/tools/perf/arch/powerpc/util/event.c
@@ -9,6 +9,7 @@
#include "../../../util/tool.h"
#include "../../../util/map.h"
#include "../../../util/debug.h"
+#include "../../../util/sample.h"
void arch_perf_parse_sample_weight(struct perf_sample *data,
const __u64 *array, u64 type)
diff --git a/tools/perf/arch/powerpc/util/unwind-libdw.c b/tools/perf/arch/powerpc/util/unwind-libdw.c
index 7b2d96e..e616642 100644
--- a/tools/perf/arch/powerpc/util/unwind-libdw.c
+++ b/tools/perf/arch/powerpc/util/unwind-libdw.c
@@ -3,7 +3,7 @@
#include <linux/kernel.h>
#include "../../../util/unwind-libdw.h"
#include "../../../util/perf_regs.h"
-#include "../../../util/event.h"
+#include "../../../util/sample.h"
/* See backends/ppc_initreg.c and backends/ppc_regs.c in elfutils. */
static const int special_regs[3][2] = {
diff --git a/tools/perf/arch/riscv/util/Build b/tools/perf/arch/riscv/util/Build
index 7d30501..603dbb5 100644
--- a/tools/perf/arch/riscv/util/Build
+++ b/tools/perf/arch/riscv/util/Build
@@ -1,4 +1,5 @@
perf-y += perf_regs.o
+perf-y += header.o
perf-$(CONFIG_DWARF) += dwarf-regs.o
perf-$(CONFIG_LIBDW_DWARF_UNWIND) += unwind-libdw.o
diff --git a/tools/perf/arch/riscv/util/header.c b/tools/perf/arch/riscv/util/header.c
new file mode 100644
index 0000000..4a41856
--- /dev/null
+++ b/tools/perf/arch/riscv/util/header.c
@@ -0,0 +1,104 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Implementation of get_cpuid().
+ *
+ * Author: Nikita Shubin <n.shubin@yadro.com>
+ */
+
+#include <stdio.h>
+#include <stdlib.h>
+#include <api/fs/fs.h>
+#include <errno.h>
+#include "../../util/debug.h"
+#include "../../util/header.h"
+
+#define CPUINFO_MVEN "mvendorid"
+#define CPUINFO_MARCH "marchid"
+#define CPUINFO_MIMP "mimpid"
+#define CPUINFO "/proc/cpuinfo"
+
+static char *_get_field(const char *line)
+{
+ char *line2, *nl;
+
+ line2 = strrchr(line, ' ');
+ if (!line2)
+ return NULL;
+
+ line2++;
+ nl = strrchr(line, '\n');
+ if (!nl)
+ return NULL;
+
+ return strndup(line2, nl - line2);
+}
+
+static char *_get_cpuid(void)
+{
+ char *line = NULL;
+ char *mvendorid = NULL;
+ char *marchid = NULL;
+ char *mimpid = NULL;
+ char *cpuid = NULL;
+ int read;
+ unsigned long line_sz;
+ FILE *cpuinfo;
+
+ cpuinfo = fopen(CPUINFO, "r");
+ if (cpuinfo == NULL)
+ return cpuid;
+
+ while ((read = getline(&line, &line_sz, cpuinfo)) != -1) {
+ if (!strncmp(line, CPUINFO_MVEN, strlen(CPUINFO_MVEN))) {
+ mvendorid = _get_field(line);
+ if (!mvendorid)
+ goto free;
+ } else if (!strncmp(line, CPUINFO_MARCH, strlen(CPUINFO_MARCH))) {
+ marchid = _get_field(line);
+ if (!marchid)
+ goto free;
+ } else if (!strncmp(line, CPUINFO_MIMP, strlen(CPUINFO_MIMP))) {
+ mimpid = _get_field(line);
+ if (!mimpid)
+ goto free;
+
+ break;
+ }
+ }
+
+ if (!mvendorid || !marchid || !mimpid)
+ goto free;
+
+ if (asprintf(&cpuid, "%s-%s-%s", mvendorid, marchid, mimpid) < 0)
+ cpuid = NULL;
+
+free:
+ fclose(cpuinfo);
+ free(mvendorid);
+ free(marchid);
+ free(mimpid);
+
+ return cpuid;
+}
+
+int get_cpuid(char *buffer, size_t sz)
+{
+ char *cpuid = _get_cpuid();
+ int ret = 0;
+
+ if (sz < strlen(cpuid)) {
+ ret = -EINVAL;
+ goto free;
+ }
+
+ scnprintf(buffer, sz, "%s", cpuid);
+free:
+ free(cpuid);
+ return ret;
+}
+
+char *
+get_cpuid_str(struct perf_pmu *pmu __maybe_unused)
+{
+ return _get_cpuid();
+}
diff --git a/tools/perf/arch/s390/util/Build b/tools/perf/arch/s390/util/Build
index 3d9d0f4..db68840 100644
--- a/tools/perf/arch/s390/util/Build
+++ b/tools/perf/arch/s390/util/Build
@@ -1,5 +1,5 @@
perf-y += header.o
-perf-y += kvm-stat.o
+perf-$(CONFIG_LIBTRACEEVENT) += kvm-stat.o
perf-y += perf_regs.o
perf-$(CONFIG_DWARF) += dwarf-regs.o
diff --git a/tools/perf/arch/s390/util/unwind-libdw.c b/tools/perf/arch/s390/util/unwind-libdw.c
index 387c698..7d92452 100644
--- a/tools/perf/arch/s390/util/unwind-libdw.c
+++ b/tools/perf/arch/s390/util/unwind-libdw.c
@@ -3,6 +3,7 @@
#include "../../util/unwind-libdw.h"
#include "../../util/perf_regs.h"
#include "../../util/event.h"
+#include "../../util/sample.h"
#include "dwarf-regs-table.h"
diff --git a/tools/perf/arch/x86/include/arch-tests.h b/tools/perf/arch/x86/include/arch-tests.h
index 6a1a1b3..902e9ea 100644
--- a/tools/perf/arch/x86/include/arch-tests.h
+++ b/tools/perf/arch/x86/include/arch-tests.h
@@ -8,6 +8,7 @@ struct test_suite;
int test__rdpmc(struct test_suite *test, int subtest);
int test__insn_x86(struct test_suite *test, int subtest);
int test__intel_pt_pkt_decoder(struct test_suite *test, int subtest);
+int test__intel_pt_hybrid_compat(struct test_suite *test, int subtest);
int test__bp_modify(struct test_suite *test, int subtest);
int test__x86_sample_parsing(struct test_suite *test, int subtest);
diff --git a/tools/perf/arch/x86/tests/Build b/tools/perf/arch/x86/tests/Build
index 70b5bcb..6f4e863 100644
--- a/tools/perf/arch/x86/tests/Build
+++ b/tools/perf/arch/x86/tests/Build
@@ -3,5 +3,5 @@
perf-y += arch-tests.o
perf-y += sample-parsing.o
-perf-$(CONFIG_AUXTRACE) += insn-x86.o intel-pt-pkt-decoder-test.o
+perf-$(CONFIG_AUXTRACE) += insn-x86.o intel-pt-test.o
perf-$(CONFIG_X86_64) += bp-modify.o
diff --git a/tools/perf/arch/x86/tests/arch-tests.c b/tools/perf/arch/x86/tests/arch-tests.c
index 04018b8..aae6ea0 100644
--- a/tools/perf/arch/x86/tests/arch-tests.c
+++ b/tools/perf/arch/x86/tests/arch-tests.c
@@ -5,7 +5,18 @@
#ifdef HAVE_AUXTRACE_SUPPORT
DEFINE_SUITE("x86 instruction decoder - new instructions", insn_x86);
-DEFINE_SUITE("Intel PT packet decoder", intel_pt_pkt_decoder);
+
+static struct test_case intel_pt_tests[] = {
+ TEST_CASE("Intel PT packet decoder", intel_pt_pkt_decoder),
+ TEST_CASE("Intel PT hybrid CPU compatibility", intel_pt_hybrid_compat),
+ { .name = NULL, }
+};
+
+struct test_suite suite__intel_pt = {
+ .desc = "Intel PT",
+ .test_cases = intel_pt_tests,
+};
+
#endif
#if defined(__x86_64__)
DEFINE_SUITE("x86 bp modify", bp_modify);
@@ -18,7 +29,7 @@ struct test_suite *arch_tests[] = {
#endif
#ifdef HAVE_AUXTRACE_SUPPORT
&suite__insn_x86,
- &suite__intel_pt_pkt_decoder,
+ &suite__intel_pt,
#endif
#if defined(__x86_64__)
&suite__bp_modify,
diff --git a/tools/perf/arch/x86/tests/intel-pt-pkt-decoder-test.c b/tools/perf/arch/x86/tests/intel-pt-test.c
similarity index 80%
rename from tools/perf/arch/x86/tests/intel-pt-pkt-decoder-test.c
rename to tools/perf/arch/x86/tests/intel-pt-test.c
index 4223765..70b7f79 100644
--- a/tools/perf/arch/x86/tests/intel-pt-pkt-decoder-test.c
+++ b/tools/perf/arch/x86/tests/intel-pt-test.c
@@ -1,12 +1,17 @@
// SPDX-License-Identifier: GPL-2.0
+#include <linux/compiler.h>
+#include <linux/bits.h>
#include <string.h>
+#include <cpuid.h>
+#include <sched.h>
#include "intel-pt-decoder/intel-pt-pkt-decoder.h"
#include "debug.h"
#include "tests/tests.h"
#include "arch-tests.h"
+#include "cpumap.h"
/**
* struct test_data - Test data.
@@ -313,3 +318,152 @@ int test__intel_pt_pkt_decoder(struct test_suite *test __maybe_unused, int subte
return TEST_OK;
}
+
+static int setaffinity(int cpu)
+{
+ cpu_set_t cpu_set;
+
+ CPU_ZERO(&cpu_set);
+ CPU_SET(cpu, &cpu_set);
+ if (sched_setaffinity(0, sizeof(cpu_set), &cpu_set)) {
+ pr_debug("sched_setaffinity() failed for CPU %d\n", cpu);
+ return -1;
+ }
+ return 0;
+}
+
+#define INTEL_PT_ADDR_FILT_CNT_MASK GENMASK(2, 0)
+#define INTEL_PT_SUBLEAF_CNT 2
+#define CPUID_REG_CNT 4
+
+struct cpuid_result {
+ union {
+ struct {
+ unsigned int eax;
+ unsigned int ebx;
+ unsigned int ecx;
+ unsigned int edx;
+ };
+ unsigned int reg[CPUID_REG_CNT];
+ };
+};
+
+struct pt_caps {
+ struct cpuid_result subleaf[INTEL_PT_SUBLEAF_CNT];
+};
+
+static int get_pt_caps(int cpu, struct pt_caps *caps)
+{
+ struct cpuid_result r;
+ int i;
+
+ if (setaffinity(cpu))
+ return -1;
+
+ memset(caps, 0, sizeof(*caps));
+
+ for (i = 0; i < INTEL_PT_SUBLEAF_CNT; i++) {
+ __get_cpuid_count(20, i, &r.eax, &r.ebx, &r.ecx, &r.edx);
+ pr_debug("CPU %d CPUID leaf 20 subleaf %d\n", cpu, i);
+ pr_debug("eax = 0x%08x\n", r.eax);
+ pr_debug("ebx = 0x%08x\n", r.ebx);
+ pr_debug("ecx = 0x%08x\n", r.ecx);
+ pr_debug("edx = 0x%08x\n", r.edx);
+ caps->subleaf[i] = r;
+ }
+
+ return 0;
+}
+
+static bool is_hydrid(void)
+{
+ unsigned int eax, ebx, ecx, edx = 0;
+ bool result;
+
+ __get_cpuid_count(7, 0, &eax, &ebx, &ecx, &edx);
+ result = edx & BIT(15);
+ pr_debug("Is %shybrid : CPUID leaf 7 subleaf 0 edx %#x (bit-15 indicates hybrid)\n",
+ result ? "" : "not ", edx);
+ return result;
+}
+
+static int compare_caps(int cpu, struct pt_caps *caps, struct pt_caps *caps0)
+{
+ struct pt_caps mask = { /* Mask of bits to check*/
+ .subleaf = {
+ [0] = {
+ .ebx = GENMASK(8, 0),
+ .ecx = GENMASK(3, 0),
+ },
+ [1] = {
+ .eax = GENMASK(31, 16),
+ .ebx = GENMASK(31, 0),
+ }
+ }
+ };
+ unsigned int m, reg, reg0;
+ int ret = 0;
+ int i, j;
+
+ for (i = 0; i < INTEL_PT_SUBLEAF_CNT; i++) {
+ for (j = 0; j < CPUID_REG_CNT; j++) {
+ m = mask.subleaf[i].reg[j];
+ reg = m & caps->subleaf[i].reg[j];
+ reg0 = m & caps0->subleaf[i].reg[j];
+ if ((reg & reg0) != reg0) {
+ pr_debug("CPU %d subleaf %d reg %d FAIL %#x vs %#x\n",
+ cpu, i, j, reg, reg0);
+ ret = -1;
+ }
+ }
+ }
+
+ m = INTEL_PT_ADDR_FILT_CNT_MASK;
+ reg = m & caps->subleaf[1].eax;
+ reg0 = m & caps0->subleaf[1].eax;
+ if (reg < reg0) {
+ pr_debug("CPU %d subleaf 1 reg 0 FAIL address filter count %#x vs %#x\n",
+ cpu, reg, reg0);
+ ret = -1;
+ }
+
+ if (!ret)
+ pr_debug("CPU %d OK\n", cpu);
+
+ return ret;
+}
+
+int test__intel_pt_hybrid_compat(struct test_suite *test, int subtest)
+{
+ int max_cpu = cpu__max_cpu().cpu;
+ struct pt_caps last_caps;
+ struct pt_caps caps0;
+ int ret = TEST_OK;
+ int cpu;
+
+ if (!is_hydrid()) {
+ test->test_cases[subtest].skip_reason = "not hybrid";
+ return TEST_SKIP;
+ }
+
+ if (get_pt_caps(0, &caps0))
+ return TEST_FAIL;
+
+ for (cpu = 1, last_caps = caps0; cpu < max_cpu; cpu++) {
+ struct pt_caps caps;
+
+ if (get_pt_caps(cpu, &caps)) {
+ pr_debug("CPU %d not found\n", cpu);
+ continue;
+ }
+ if (!memcmp(&caps, &last_caps, sizeof(caps))) {
+ pr_debug("CPU %d same caps as previous CPU\n", cpu);
+ continue;
+ }
+ if (compare_caps(cpu, &caps, &caps0))
+ ret = TEST_FAIL;
+ last_caps = caps;
+ }
+
+ return ret;
+}
diff --git a/tools/perf/arch/x86/tests/sample-parsing.c b/tools/perf/arch/x86/tests/sample-parsing.c
index bfbd366..690c7c07 100644
--- a/tools/perf/arch/x86/tests/sample-parsing.c
+++ b/tools/perf/arch/x86/tests/sample-parsing.c
@@ -10,6 +10,7 @@
#include "event.h"
#include "evsel.h"
#include "debug.h"
+#include "util/sample.h"
#include "util/synthetic-events.h"
#include "tests/tests.h"
diff --git a/tools/perf/arch/x86/util/Build b/tools/perf/arch/x86/util/Build
index dbeb04cb..195ccfd 100644
--- a/tools/perf/arch/x86/util/Build
+++ b/tools/perf/arch/x86/util/Build
@@ -1,7 +1,7 @@
perf-y += header.o
perf-y += tsc.o
perf-y += pmu.o
-perf-y += kvm-stat.o
+perf-$(CONFIG_LIBTRACEEVENT) += kvm-stat.o
perf-y += perf_regs.o
perf-y += topdown.o
perf-y += machine.o
diff --git a/tools/perf/arch/x86/util/event.c b/tools/perf/arch/x86/util/event.c
index e670f35..a3acefe 100644
--- a/tools/perf/arch/x86/util/event.c
+++ b/tools/perf/arch/x86/util/event.c
@@ -2,6 +2,7 @@
#include <linux/types.h>
#include <linux/string.h>
#include <linux/zalloc.h>
+#include <stdlib.h>
#include "../../../util/event.h"
#include "../../../util/synthetic-events.h"
@@ -9,6 +10,7 @@
#include "../../../util/tool.h"
#include "../../../util/map.h"
#include "../../../util/debug.h"
+#include "util/sample.h"
#if defined(__x86_64__)
diff --git a/tools/perf/arch/x86/util/intel-pt.c b/tools/perf/arch/x86/util/intel-pt.c
index af102f4..1e39a03 100644
--- a/tools/perf/arch/x86/util/intel-pt.c
+++ b/tools/perf/arch/x86/util/intel-pt.c
@@ -418,6 +418,7 @@ static int intel_pt_info_fill(struct auxtrace_record *itr,
return 0;
}
+#ifdef HAVE_LIBTRACEEVENT
static int intel_pt_track_switches(struct evlist *evlist)
{
const char *sched_switch = "sched:sched_switch";
@@ -439,6 +440,7 @@ static int intel_pt_track_switches(struct evlist *evlist)
return 0;
}
+#endif
static void intel_pt_valid_str(char *str, size_t len, u64 valid)
{
@@ -829,6 +831,7 @@ static int intel_pt_recording_options(struct auxtrace_record *itr,
ptr->have_sched_switch = 2;
}
} else {
+#ifdef HAVE_LIBTRACEEVENT
err = intel_pt_track_switches(evlist);
if (err == -EPERM)
pr_debug2("Unable to select sched:sched_switch\n");
@@ -836,6 +839,7 @@ static int intel_pt_recording_options(struct auxtrace_record *itr,
return err;
else
ptr->have_sched_switch = 1;
+#endif
}
}
diff --git a/tools/perf/arch/x86/util/iostat.c b/tools/perf/arch/x86/util/iostat.c
index 404de79..7eb0a7b 100644
--- a/tools/perf/arch/x86/util/iostat.c
+++ b/tools/perf/arch/x86/util/iostat.c
@@ -449,7 +449,7 @@ void iostat_print_metric(struct perf_stat_config *config, struct evsel *evsel,
void iostat_print_counters(struct evlist *evlist,
struct perf_stat_config *config, struct timespec *ts,
- char *prefix, iostat_print_counter_t print_cnt_cb)
+ char *prefix, iostat_print_counter_t print_cnt_cb, void *arg)
{
void *perf_device = NULL;
struct evsel *counter = evlist__first(evlist);
@@ -464,7 +464,7 @@ void iostat_print_counters(struct evlist *evlist,
iostat_prefix(evlist, config, prefix, ts);
fprintf(config->output, "\n%s", prefix);
}
- print_cnt_cb(config, counter, prefix);
+ print_cnt_cb(config, counter, arg);
}
fputc('\n', config->output);
}
diff --git a/tools/perf/arch/x86/util/tsc.c b/tools/perf/arch/x86/util/tsc.c
index eb2b519..9b99f48 100644
--- a/tools/perf/arch/x86/util/tsc.c
+++ b/tools/perf/arch/x86/util/tsc.c
@@ -2,6 +2,7 @@
#include <linux/types.h>
#include <math.h>
#include <string.h>
+#include <stdlib.h>
#include "../../../util/debug.h"
#include "../../../util/tsc.h"
diff --git a/tools/perf/arch/x86/util/unwind-libdw.c b/tools/perf/arch/x86/util/unwind-libdw.c
index eea2bf8..ef71e8b 100644
--- a/tools/perf/arch/x86/util/unwind-libdw.c
+++ b/tools/perf/arch/x86/util/unwind-libdw.c
@@ -2,7 +2,7 @@
#include <elfutils/libdwfl.h>
#include "../../../util/unwind-libdw.h"
#include "../../../util/perf_regs.h"
-#include "../../../util/event.h"
+#include "util/sample.h"
bool libdw__arch_set_initial_registers(Dwfl_Thread *thread, void *arg)
{
diff --git a/tools/perf/bench/bench.h b/tools/perf/bench/bench.h
index 6cefb43..a5d49b3 100644
--- a/tools/perf/bench/bench.h
+++ b/tools/perf/bench/bench.h
@@ -10,25 +10,13 @@ extern struct timeval bench__start, bench__end, bench__runtime;
* The madvise transparent hugepage constants were added in glibc
* 2.13. For compatibility with older versions of glibc, define these
* tokens if they are not already defined.
- *
- * PA-RISC uses different madvise values from other architectures and
- * needs to be special-cased.
*/
-#ifdef __hppa__
-# ifndef MADV_HUGEPAGE
-# define MADV_HUGEPAGE 67
-# endif
-# ifndef MADV_NOHUGEPAGE
-# define MADV_NOHUGEPAGE 68
-# endif
-#else
# ifndef MADV_HUGEPAGE
# define MADV_HUGEPAGE 14
# endif
# ifndef MADV_NOHUGEPAGE
# define MADV_NOHUGEPAGE 15
# endif
-#endif
int bench_numa(int argc, const char **argv);
int bench_sched_messaging(int argc, const char **argv);
diff --git a/tools/perf/bench/inject-buildid.c b/tools/perf/bench/inject-buildid.c
index 1767279..4561bda 100644
--- a/tools/perf/bench/inject-buildid.c
+++ b/tools/perf/bench/inject-buildid.c
@@ -19,10 +19,10 @@
#include "util/data.h"
#include "util/stat.h"
#include "util/debug.h"
-#include "util/event.h"
#include "util/symbol.h"
#include "util/session.h"
#include "util/build-id.h"
+#include "util/sample.h"
#include "util/synthetic-events.h"
#define MMAP_DEV_MAJOR 8
diff --git a/tools/perf/bench/numa.c b/tools/perf/bench/numa.c
index e78dedf..9717c6c 100644
--- a/tools/perf/bench/numa.c
+++ b/tools/perf/bench/numa.c
@@ -16,6 +16,7 @@
#include <sched.h>
#include <stdio.h>
#include <assert.h>
+#include <debug.h>
#include <malloc.h>
#include <signal.h>
#include <stdlib.h>
@@ -116,7 +117,6 @@ struct params {
long bytes_thread;
int nr_tasks;
- bool show_quiet;
bool show_convergence;
bool measure_convergence;
@@ -197,7 +197,8 @@ static const struct option options[] = {
OPT_BOOLEAN('c', "show_convergence", &p0.show_convergence, "show convergence details, "
"convergence is reached when each process (all its threads) is running on a single NUMA node."),
OPT_BOOLEAN('m', "measure_convergence", &p0.measure_convergence, "measure convergence latency"),
- OPT_BOOLEAN('q', "quiet" , &p0.show_quiet, "quiet mode"),
+ OPT_BOOLEAN('q', "quiet" , &quiet,
+ "quiet mode (do not show any warnings or messages)"),
OPT_BOOLEAN('S', "serialize-startup", &p0.serialize_startup,"serialize thread startup"),
/* Special option string parsing callbacks: */
@@ -1474,7 +1475,7 @@ static int init(void)
/* char array in count_process_nodes(): */
BUG_ON(g->p.nr_nodes < 0);
- if (g->p.show_quiet && !g->p.show_details)
+ if (quiet && !g->p.show_details)
g->p.show_details = -1;
/* Some memory should be specified: */
@@ -1553,7 +1554,7 @@ static void print_res(const char *name, double val,
if (!name)
name = "main,";
- if (!g->p.show_quiet)
+ if (!quiet)
printf(" %-30s %15.3f, %-15s %s\n", name, val, txt_unit, txt_short);
else
printf(" %14.3f %s\n", val, txt_long);
diff --git a/tools/perf/builtin-annotate.c b/tools/perf/builtin-annotate.c
index f839e69..90458ca 100644
--- a/tools/perf/builtin-annotate.c
+++ b/tools/perf/builtin-annotate.c
@@ -499,7 +499,9 @@ int cmd_annotate(int argc, const char **argv)
.namespaces = perf_event__process_namespaces,
.attr = perf_event__process_attr,
.build_id = perf_event__process_build_id,
+#ifdef HAVE_LIBTRACEEVENT
.tracing_data = perf_event__process_tracing_data,
+#endif
.id_index = perf_event__process_id_index,
.auxtrace_info = perf_event__process_auxtrace_info,
.auxtrace = perf_event__process_auxtrace,
@@ -525,7 +527,7 @@ int cmd_annotate(int argc, const char **argv)
OPT_BOOLEAN('f', "force", &data.force, "don't complain, do it"),
OPT_INCR('v', "verbose", &verbose,
"be more verbose (show symbol address, etc)"),
- OPT_BOOLEAN('q', "quiet", &quiet, "do now show any message"),
+ OPT_BOOLEAN('q', "quiet", &quiet, "do now show any warnings or messages"),
OPT_BOOLEAN('D', "dump-raw-trace", &dump_trace,
"dump raw trace in ASCII"),
#ifdef HAVE_GTK2_SUPPORT
diff --git a/tools/perf/builtin-daemon.c b/tools/perf/builtin-daemon.c
index 6cb3f6c..7036ec9 100644
--- a/tools/perf/builtin-daemon.c
+++ b/tools/perf/builtin-daemon.c
@@ -100,12 +100,12 @@ static struct daemon __daemon = {
};
static const char * const daemon_usage[] = {
- "perf daemon start [<options>]",
+ "perf daemon {start|signal|stop|ping} [<options>]",
"perf daemon [<options>]",
NULL
};
-static bool done;
+static volatile sig_atomic_t done;
static void sig_handler(int sig __maybe_unused)
{
diff --git a/tools/perf/builtin-data.c b/tools/perf/builtin-data.c
index c22d82d..b2a9a3b7 100644
--- a/tools/perf/builtin-data.c
+++ b/tools/perf/builtin-data.c
@@ -78,12 +78,13 @@ static int cmd_data_convert(int argc, const char **argv)
return bt_convert__perf2json(input_name, to_json, &opts);
if (to_ctf) {
-#ifdef HAVE_LIBBABELTRACE_SUPPORT
+#if defined(HAVE_LIBBABELTRACE_SUPPORT) && defined(HAVE_LIBTRACEEVENT)
return bt_convert__perf2ctf(input_name, to_ctf, &opts);
#else
pr_err("The libbabeltrace support is not compiled in. perf should be "
"compiled with environment variables LIBBABELTRACE=1 and "
- "LIBBABELTRACE_DIR=/path/to/libbabeltrace/\n");
+ "LIBBABELTRACE_DIR=/path/to/libbabeltrace/.\n"
+ "Check also if libbtraceevent devel files are available.\n");
return -1;
#endif
}
diff --git a/tools/perf/builtin-diff.c b/tools/perf/builtin-diff.c
index d925096..ed07cc6 100644
--- a/tools/perf/builtin-diff.c
+++ b/tools/perf/builtin-diff.c
@@ -1260,7 +1260,7 @@ static const char * const diff_usage[] = {
static const struct option options[] = {
OPT_INCR('v', "verbose", &verbose,
"be more verbose (show symbol address, etc)"),
- OPT_BOOLEAN('q', "quiet", &quiet, "Do not show any message"),
+ OPT_BOOLEAN('q', "quiet", &quiet, "Do not show any warnings or messages"),
OPT_BOOLEAN('b', "baseline-only", &show_baseline_only,
"Show only items with match in baseline"),
OPT_CALLBACK('c', "compute", &compute,
diff --git a/tools/perf/builtin-ftrace.c b/tools/perf/builtin-ftrace.c
index 7de07bb..d7fe00f 100644
--- a/tools/perf/builtin-ftrace.c
+++ b/tools/perf/builtin-ftrace.c
@@ -36,8 +36,8 @@
#define DEFAULT_TRACER "function_graph"
-static volatile int workload_exec_errno;
-static bool done;
+static volatile sig_atomic_t workload_exec_errno;
+static volatile sig_atomic_t done;
static void sig_handler(int sig __maybe_unused)
{
diff --git a/tools/perf/builtin-inject.c b/tools/perf/builtin-inject.c
index e254f18..3f4e4dd 100644
--- a/tools/perf/builtin-inject.c
+++ b/tools/perf/builtin-inject.c
@@ -607,6 +607,7 @@ static int perf_event__repipe_exit(struct perf_tool *tool,
return err;
}
+#ifdef HAVE_LIBTRACEEVENT
static int perf_event__repipe_tracing_data(struct perf_session *session,
union perf_event *event)
{
@@ -614,6 +615,7 @@ static int perf_event__repipe_tracing_data(struct perf_session *session,
return perf_event__process_tracing_data(session, event);
}
+#endif
static int dso__read_build_id(struct dso *dso)
{
@@ -807,6 +809,7 @@ static int perf_inject__sched_switch(struct perf_tool *tool,
return 0;
}
+#ifdef HAVE_LIBTRACEEVENT
static int perf_inject__sched_stat(struct perf_tool *tool,
union perf_event *event __maybe_unused,
struct perf_sample *sample,
@@ -836,6 +839,7 @@ static int perf_inject__sched_stat(struct perf_tool *tool,
build_id__mark_dso_hit(tool, event_sw, &sample_sw, evsel, machine);
return perf_event__repipe(tool, event_sw, &sample_sw, machine);
}
+#endif
static struct guest_vcpu *guest_session__vcpu(struct guest_session *gs, u32 vcpu)
{
@@ -1961,7 +1965,9 @@ static int __cmd_inject(struct perf_inject *inject)
inject->tool.mmap = perf_event__repipe_mmap;
inject->tool.mmap2 = perf_event__repipe_mmap2;
inject->tool.fork = perf_event__repipe_fork;
+#ifdef HAVE_LIBTRACEEVENT
inject->tool.tracing_data = perf_event__repipe_tracing_data;
+#endif
}
output_data_offset = perf_session__data_offset(session->evlist);
@@ -1984,8 +1990,10 @@ static int __cmd_inject(struct perf_inject *inject)
evsel->handler = perf_inject__sched_switch;
} else if (!strcmp(name, "sched:sched_process_exit"))
evsel->handler = perf_inject__sched_process_exit;
+#ifdef HAVE_LIBTRACEEVENT
else if (!strncmp(name, "sched:sched_stat_", 17))
evsel->handler = perf_inject__sched_stat;
+#endif
}
} else if (inject->itrace_synth_opts.vm_time_correlation) {
session->itrace_synth_opts = &inject->itrace_synth_opts;
diff --git a/tools/perf/builtin-kmem.c b/tools/perf/builtin-kmem.c
index ebfab2c..e20656c 100644
--- a/tools/perf/builtin-kmem.c
+++ b/tools/perf/builtin-kmem.c
@@ -35,6 +35,7 @@
#include <regex.h>
#include <linux/ctype.h>
+#include <traceevent/event-parse.h>
static int kmem_slab;
static int kmem_page;
diff --git a/tools/perf/builtin-kvm.c b/tools/perf/builtin-kvm.c
index 7d9ec1b..641e739 100644
--- a/tools/perf/builtin-kvm.c
+++ b/tools/perf/builtin-kvm.c
@@ -63,7 +63,7 @@ static const char *get_filename_for_perf_kvm(void)
return filename;
}
-#ifdef HAVE_KVM_STAT_SUPPORT
+#if defined(HAVE_KVM_STAT_SUPPORT) && defined(HAVE_LIBTRACEEVENT)
void exit_event_get_key(struct evsel *evsel,
struct perf_sample *sample,
@@ -654,7 +654,7 @@ static void print_result(struct perf_kvm_stat *kvm)
pr_info("\nLost events: %" PRIu64 "\n\n", kvm->lost_events);
}
-#ifdef HAVE_TIMERFD_SUPPORT
+#if defined(HAVE_TIMERFD_SUPPORT) && defined(HAVE_LIBTRACEEVENT)
static int process_lost_event(struct perf_tool *tool,
union perf_event *event __maybe_unused,
struct perf_sample *sample __maybe_unused,
@@ -742,7 +742,7 @@ static bool verify_vcpu(int vcpu)
return true;
}
-#ifdef HAVE_TIMERFD_SUPPORT
+#if defined(HAVE_TIMERFD_SUPPORT) && defined(HAVE_LIBTRACEEVENT)
/* keeping the max events to a modest level to keep
* the processing of samples per mmap smooth.
*/
@@ -1290,7 +1290,7 @@ kvm_events_report(struct perf_kvm_stat *kvm, int argc, const char **argv)
return kvm_events_report_vcpu(kvm);
}
-#ifdef HAVE_TIMERFD_SUPPORT
+#if defined(HAVE_TIMERFD_SUPPORT) && defined(HAVE_LIBTRACEEVENT)
static struct evlist *kvm_live_event_list(void)
{
struct evlist *evlist;
@@ -1507,7 +1507,7 @@ static int kvm_cmd_stat(const char *file_name, int argc, const char **argv)
if (strlen(argv[1]) > 2 && strstarts("report", argv[1]))
return kvm_events_report(&kvm, argc - 1 , argv + 1);
-#ifdef HAVE_TIMERFD_SUPPORT
+#if defined(HAVE_TIMERFD_SUPPORT) && defined(HAVE_LIBTRACEEVENT)
if (!strncmp(argv[1], "live", 4))
return kvm_events_live(&kvm, argc - 1 , argv + 1);
#endif
@@ -1644,7 +1644,7 @@ int cmd_kvm(int argc, const char **argv)
return cmd_top(argc, argv);
else if (strlen(argv[0]) > 2 && strstarts("buildid-list", argv[0]))
return __cmd_buildid_list(file_name, argc, argv);
-#ifdef HAVE_KVM_STAT_SUPPORT
+#if defined(HAVE_KVM_STAT_SUPPORT) && defined(HAVE_LIBTRACEEVENT)
else if (strlen(argv[0]) > 2 && strstarts("stat", argv[0]))
return kvm_cmd_stat(file_name, argc, argv);
#endif
diff --git a/tools/perf/builtin-kwork.c b/tools/perf/builtin-kwork.c
index 1f63e24..dc59d75 100644
--- a/tools/perf/builtin-kwork.c
+++ b/tools/perf/builtin-kwork.c
@@ -6,10 +6,15 @@
*/
#include "builtin.h"
+#include "perf.h"
#include "util/data.h"
+#include "util/evlist.h"
+#include "util/evsel.h"
+#include "util/header.h"
#include "util/kwork.h"
#include "util/debug.h"
+#include "util/session.h"
#include "util/symbol.h"
#include "util/thread.h"
#include "util/string2.h"
@@ -18,9 +23,11 @@
#include <subcmd/pager.h>
#include <subcmd/parse-options.h>
+#include <traceevent/event-parse.h>
#include <errno.h>
#include <inttypes.h>
+#include <signal.h>
#include <linux/err.h>
#include <linux/time64.h>
#include <linux/zalloc.h>
diff --git a/tools/perf/builtin-list.c b/tools/perf/builtin-list.c
index 58e1ec1..137d73e 100644
--- a/tools/perf/builtin-list.c
+++ b/tools/perf/builtin-list.c
@@ -15,34 +15,443 @@
#include "util/pmu-hybrid.h"
#include "util/debug.h"
#include "util/metricgroup.h"
+#include "util/string2.h"
+#include "util/strlist.h"
+#include "util/strbuf.h"
#include <subcmd/pager.h>
#include <subcmd/parse-options.h>
+#include <linux/zalloc.h>
+#include <stdarg.h>
#include <stdio.h>
-static bool desc_flag = true;
-static bool details_flag;
-static const char *hybrid_type;
+/**
+ * struct print_state - State and configuration passed to the default_print
+ * functions.
+ */
+struct print_state {
+ /**
+ * @pmu_glob: Optionally restrict PMU and metric matching to PMU or
+ * debugfs subsystem name.
+ */
+ char *pmu_glob;
+ /** @event_glob: Optional pattern matching glob. */
+ char *event_glob;
+ /** @name_only: Print event or metric names only. */
+ bool name_only;
+ /** @desc: Print the event or metric description. */
+ bool desc;
+ /** @long_desc: Print longer event or metric description. */
+ bool long_desc;
+ /** @deprecated: Print deprecated events or metrics. */
+ bool deprecated;
+ /**
+ * @detailed: Print extra information on the perf event such as names
+ * and expressions used internally by events.
+ */
+ bool detailed;
+ /** @metrics: Controls printing of metric and metric groups. */
+ bool metrics;
+ /** @metricgroups: Controls printing of metric and metric groups. */
+ bool metricgroups;
+ /** @last_topic: The last printed event topic. */
+ char *last_topic;
+ /** @last_metricgroups: The last printed metric group. */
+ char *last_metricgroups;
+ /** @visited_metrics: Metrics that are printed to avoid duplicates. */
+ struct strlist *visited_metrics;
+};
+
+static void default_print_start(void *ps)
+{
+ struct print_state *print_state = ps;
+
+ if (!print_state->name_only && pager_in_use())
+ printf("\nList of pre-defined events (to be used in -e or -M):\n\n");
+}
+
+static void default_print_end(void *print_state __maybe_unused) {}
+
+static void wordwrap(const char *s, int start, int max, int corr)
+{
+ int column = start;
+ int n;
+ bool saw_newline = false;
+
+ while (*s) {
+ int wlen = strcspn(s, " \t\n");
+
+ if ((column + wlen >= max && column > start) || saw_newline) {
+ printf("\n%*s", start, "");
+ column = start + corr;
+ }
+ n = printf("%s%.*s", column > start ? " " : "", wlen, s);
+ if (n <= 0)
+ break;
+ saw_newline = s[wlen] == '\n';
+ s += wlen;
+ column += n;
+ s = skip_spaces(s);
+ }
+}
+
+static void default_print_event(void *ps, const char *pmu_name, const char *topic,
+ const char *event_name, const char *event_alias,
+ const char *scale_unit __maybe_unused,
+ bool deprecated, const char *event_type_desc,
+ const char *desc, const char *long_desc,
+ const char *encoding_desc,
+ const char *metric_name, const char *metric_expr)
+{
+ struct print_state *print_state = ps;
+ int pos;
+
+ if (deprecated && !print_state->deprecated)
+ return;
+
+ if (print_state->pmu_glob && pmu_name && !strglobmatch(pmu_name, print_state->pmu_glob))
+ return;
+
+ if (print_state->event_glob &&
+ (!event_name || !strglobmatch(event_name, print_state->event_glob)) &&
+ (!event_alias || !strglobmatch(event_alias, print_state->event_glob)) &&
+ (!topic || !strglobmatch_nocase(topic, print_state->event_glob)))
+ return;
+
+ if (print_state->name_only) {
+ if (event_alias && strlen(event_alias))
+ printf("%s ", event_alias);
+ else
+ printf("%s ", event_name);
+ return;
+ }
+
+ if (strcmp(print_state->last_topic, topic ?: "")) {
+ if (topic)
+ printf("\n%s:\n", topic);
+ free(print_state->last_topic);
+ print_state->last_topic = strdup(topic ?: "");
+ }
+
+ if (event_alias && strlen(event_alias))
+ pos = printf(" %s OR %s", event_name, event_alias);
+ else
+ pos = printf(" %s", event_name);
+
+ if (!topic && event_type_desc) {
+ for (; pos < 53; pos++)
+ putchar(' ');
+ printf("[%s]\n", event_type_desc);
+ } else
+ putchar('\n');
+
+ if (desc && print_state->desc) {
+ printf("%*s", 8, "[");
+ wordwrap(desc, 8, pager_get_columns(), 0);
+ printf("]\n");
+ }
+ long_desc = long_desc ?: desc;
+ if (long_desc && print_state->long_desc) {
+ printf("%*s", 8, "[");
+ wordwrap(long_desc, 8, pager_get_columns(), 0);
+ printf("]\n");
+ }
+
+ if (print_state->detailed && encoding_desc) {
+ printf("%*s", 8, "");
+ wordwrap(encoding_desc, 8, pager_get_columns(), 0);
+ if (metric_name)
+ printf(" MetricName: %s", metric_name);
+ if (metric_expr)
+ printf(" MetricExpr: %s", metric_expr);
+ putchar('\n');
+ }
+}
+
+static void default_print_metric(void *ps,
+ const char *group,
+ const char *name,
+ const char *desc,
+ const char *long_desc,
+ const char *expr,
+ const char *unit __maybe_unused)
+{
+ struct print_state *print_state = ps;
+
+ if (print_state->event_glob &&
+ (!print_state->metrics || !name || !strglobmatch(name, print_state->event_glob)) &&
+ (!print_state->metricgroups || !group || !strglobmatch(group, print_state->event_glob)))
+ return;
+
+ if (!print_state->name_only && !print_state->last_metricgroups) {
+ if (print_state->metricgroups) {
+ printf("\nMetric Groups:\n");
+ if (!print_state->metrics)
+ putchar('\n');
+ } else {
+ printf("\nMetrics:\n\n");
+ }
+ }
+ if (!print_state->last_metricgroups ||
+ strcmp(print_state->last_metricgroups, group ?: "")) {
+ if (group && print_state->metricgroups) {
+ if (print_state->name_only)
+ printf("%s ", group);
+ else if (print_state->metrics)
+ printf("\n%s:\n", group);
+ else
+ printf("%s\n", group);
+ }
+ free(print_state->last_metricgroups);
+ print_state->last_metricgroups = strdup(group ?: "");
+ }
+ if (!print_state->metrics)
+ return;
+
+ if (print_state->name_only) {
+ if (print_state->metrics &&
+ !strlist__has_entry(print_state->visited_metrics, name)) {
+ printf("%s ", name);
+ strlist__add(print_state->visited_metrics, name);
+ }
+ return;
+ }
+ printf(" %s\n", name);
+
+ if (desc && print_state->desc) {
+ printf("%*s", 8, "[");
+ wordwrap(desc, 8, pager_get_columns(), 0);
+ printf("]\n");
+ }
+ if (long_desc && print_state->long_desc) {
+ printf("%*s", 8, "[");
+ wordwrap(long_desc, 8, pager_get_columns(), 0);
+ printf("]\n");
+ }
+ if (expr && print_state->detailed) {
+ printf("%*s", 8, "[");
+ wordwrap(expr, 8, pager_get_columns(), 0);
+ printf("]\n");
+ }
+}
+
+struct json_print_state {
+ /** Should a separator be printed prior to the next item? */
+ bool need_sep;
+};
+
+static void json_print_start(void *print_state __maybe_unused)
+{
+ printf("[\n");
+}
+
+static void json_print_end(void *ps)
+{
+ struct json_print_state *print_state = ps;
+
+ printf("%s]\n", print_state->need_sep ? "\n" : "");
+}
+
+static void fix_escape_printf(struct strbuf *buf, const char *fmt, ...)
+{
+ va_list args;
+
+ va_start(args, fmt);
+ strbuf_setlen(buf, 0);
+ for (size_t fmt_pos = 0; fmt_pos < strlen(fmt); fmt_pos++) {
+ switch (fmt[fmt_pos]) {
+ case '%':
+ fmt_pos++;
+ switch (fmt[fmt_pos]) {
+ case 's': {
+ const char *s = va_arg(args, const char*);
+
+ strbuf_addstr(buf, s);
+ break;
+ }
+ case 'S': {
+ const char *s = va_arg(args, const char*);
+
+ for (size_t s_pos = 0; s_pos < strlen(s); s_pos++) {
+ switch (s[s_pos]) {
+ case '\n':
+ strbuf_addstr(buf, "\\n");
+ break;
+ case '\\':
+ __fallthrough;
+ case '\"':
+ strbuf_addch(buf, '\\');
+ __fallthrough;
+ default:
+ strbuf_addch(buf, s[s_pos]);
+ break;
+ }
+ }
+ break;
+ }
+ default:
+ pr_err("Unexpected format character '%c'\n", fmt[fmt_pos]);
+ strbuf_addch(buf, '%');
+ strbuf_addch(buf, fmt[fmt_pos]);
+ }
+ break;
+ default:
+ strbuf_addch(buf, fmt[fmt_pos]);
+ break;
+ }
+ }
+ va_end(args);
+ fputs(buf->buf, stdout);
+}
+
+static void json_print_event(void *ps, const char *pmu_name, const char *topic,
+ const char *event_name, const char *event_alias,
+ const char *scale_unit,
+ bool deprecated, const char *event_type_desc,
+ const char *desc, const char *long_desc,
+ const char *encoding_desc,
+ const char *metric_name, const char *metric_expr)
+{
+ struct json_print_state *print_state = ps;
+ bool need_sep = false;
+ struct strbuf buf;
+
+ strbuf_init(&buf, 0);
+ printf("%s{\n", print_state->need_sep ? ",\n" : "");
+ print_state->need_sep = true;
+ if (pmu_name) {
+ fix_escape_printf(&buf, "\t\"Unit\": \"%S\"", pmu_name);
+ need_sep = true;
+ }
+ if (topic) {
+ fix_escape_printf(&buf, "%s\t\"Topic\": \"%S\"", need_sep ? ",\n" : "", topic);
+ need_sep = true;
+ }
+ if (event_name) {
+ fix_escape_printf(&buf, "%s\t\"EventName\": \"%S\"", need_sep ? ",\n" : "",
+ event_name);
+ need_sep = true;
+ }
+ if (event_alias && strlen(event_alias)) {
+ fix_escape_printf(&buf, "%s\t\"EventAlias\": \"%S\"", need_sep ? ",\n" : "",
+ event_alias);
+ need_sep = true;
+ }
+ if (scale_unit && strlen(scale_unit)) {
+ fix_escape_printf(&buf, "%s\t\"ScaleUnit\": \"%S\"", need_sep ? ",\n" : "",
+ scale_unit);
+ need_sep = true;
+ }
+ if (event_type_desc) {
+ fix_escape_printf(&buf, "%s\t\"EventType\": \"%S\"", need_sep ? ",\n" : "",
+ event_type_desc);
+ need_sep = true;
+ }
+ if (deprecated) {
+ fix_escape_printf(&buf, "%s\t\"Deprecated\": \"%S\"", need_sep ? ",\n" : "",
+ deprecated ? "1" : "0");
+ need_sep = true;
+ }
+ if (desc) {
+ fix_escape_printf(&buf, "%s\t\"BriefDescription\": \"%S\"", need_sep ? ",\n" : "",
+ desc);
+ need_sep = true;
+ }
+ if (long_desc) {
+ fix_escape_printf(&buf, "%s\t\"PublicDescription\": \"%S\"", need_sep ? ",\n" : "",
+ long_desc);
+ need_sep = true;
+ }
+ if (encoding_desc) {
+ fix_escape_printf(&buf, "%s\t\"Encoding\": \"%S\"", need_sep ? ",\n" : "",
+ encoding_desc);
+ need_sep = true;
+ }
+ if (metric_name) {
+ fix_escape_printf(&buf, "%s\t\"MetricName\": \"%S\"", need_sep ? ",\n" : "",
+ metric_name);
+ need_sep = true;
+ }
+ if (metric_expr) {
+ fix_escape_printf(&buf, "%s\t\"MetricExpr\": \"%S\"", need_sep ? ",\n" : "",
+ metric_expr);
+ need_sep = true;
+ }
+ printf("%s}", need_sep ? "\n" : "");
+ strbuf_release(&buf);
+}
+
+static void json_print_metric(void *ps __maybe_unused, const char *group,
+ const char *name, const char *desc,
+ const char *long_desc, const char *expr,
+ const char *unit)
+{
+ struct json_print_state *print_state = ps;
+ bool need_sep = false;
+ struct strbuf buf;
+
+ strbuf_init(&buf, 0);
+ printf("%s{\n", print_state->need_sep ? ",\n" : "");
+ print_state->need_sep = true;
+ if (group) {
+ fix_escape_printf(&buf, "\t\"MetricGroup\": \"%S\"", group);
+ need_sep = true;
+ }
+ if (name) {
+ fix_escape_printf(&buf, "%s\t\"MetricName\": \"%S\"", need_sep ? ",\n" : "", name);
+ need_sep = true;
+ }
+ if (expr) {
+ fix_escape_printf(&buf, "%s\t\"MetricExpr\": \"%S\"", need_sep ? ",\n" : "", expr);
+ need_sep = true;
+ }
+ if (unit) {
+ fix_escape_printf(&buf, "%s\t\"ScaleUnit\": \"%S\"", need_sep ? ",\n" : "", unit);
+ need_sep = true;
+ }
+ if (desc) {
+ fix_escape_printf(&buf, "%s\t\"BriefDescription\": \"%S\"", need_sep ? ",\n" : "",
+ desc);
+ need_sep = true;
+ }
+ if (long_desc) {
+ fix_escape_printf(&buf, "%s\t\"PublicDescription\": \"%S\"", need_sep ? ",\n" : "",
+ long_desc);
+ need_sep = true;
+ }
+ printf("%s}", need_sep ? "\n" : "");
+ strbuf_release(&buf);
+}
int cmd_list(int argc, const char **argv)
{
int i, ret = 0;
- bool raw_dump = false;
- bool long_desc_flag = false;
- bool deprecated = false;
- char *pmu_name = NULL;
+ struct print_state default_ps = {};
+ struct print_state json_ps = {};
+ void *ps = &default_ps;
+ struct print_callbacks print_cb = {
+ .print_start = default_print_start,
+ .print_end = default_print_end,
+ .print_event = default_print_event,
+ .print_metric = default_print_metric,
+ };
+ const char *hybrid_name = NULL;
+ const char *unit_name = NULL;
+ bool json = false;
struct option list_options[] = {
- OPT_BOOLEAN(0, "raw-dump", &raw_dump, "Dump raw events"),
- OPT_BOOLEAN('d', "desc", &desc_flag,
+ OPT_BOOLEAN(0, "raw-dump", &default_ps.name_only, "Dump raw events"),
+ OPT_BOOLEAN('j', "json", &json, "JSON encode events and metrics"),
+ OPT_BOOLEAN('d', "desc", &default_ps.desc,
"Print extra event descriptions. --no-desc to not print."),
- OPT_BOOLEAN('v', "long-desc", &long_desc_flag,
+ OPT_BOOLEAN('v', "long-desc", &default_ps.long_desc,
"Print longer event descriptions."),
- OPT_BOOLEAN(0, "details", &details_flag,
+ OPT_BOOLEAN(0, "details", &default_ps.detailed,
"Print information on the perf event names and expressions used internally by events."),
- OPT_BOOLEAN(0, "deprecated", &deprecated,
+ OPT_BOOLEAN(0, "deprecated", &default_ps.deprecated,
"Print deprecated events."),
- OPT_STRING(0, "cputype", &hybrid_type, "hybrid cpu type",
- "Print events applying cpu with this type for hybrid platform "
- "(e.g. core or atom)"),
+ OPT_STRING(0, "cputype", &hybrid_name, "hybrid cpu type",
+ "Limit PMU or metric printing to the given hybrid PMU (e.g. core or atom)."),
+ OPT_STRING(0, "unit", &unit_name, "PMU name",
+ "Limit PMU or metric printing to the specified PMU."),
OPT_INCR(0, "debug", &verbose,
"Enable debugging output"),
OPT_END()
@@ -53,24 +462,45 @@ int cmd_list(int argc, const char **argv)
};
set_option_flag(list_options, 0, "raw-dump", PARSE_OPT_HIDDEN);
+ /* Hide hybrid flag for the more generic 'unit' flag. */
+ set_option_flag(list_options, 0, "cputype", PARSE_OPT_HIDDEN);
argc = parse_options(argc, argv, list_options, list_usage,
PARSE_OPT_STOP_AT_NON_OPTION);
setup_pager();
- if (!raw_dump && pager_in_use())
- printf("\nList of pre-defined events (to be used in -e or -M):\n\n");
+ if (!default_ps.name_only)
+ setup_pager();
- if (hybrid_type) {
- pmu_name = perf_pmu__hybrid_type_to_pmu(hybrid_type);
- if (!pmu_name)
- pr_warning("WARNING: hybrid cputype is not supported!\n");
+ if (json) {
+ print_cb = (struct print_callbacks){
+ .print_start = json_print_start,
+ .print_end = json_print_end,
+ .print_event = json_print_event,
+ .print_metric = json_print_metric,
+ };
+ ps = &json_ps;
+ } else {
+ default_ps.desc = !default_ps.long_desc;
+ default_ps.last_topic = strdup("");
+ assert(default_ps.last_topic);
+ default_ps.visited_metrics = strlist__new(NULL, NULL);
+ assert(default_ps.visited_metrics);
+ if (unit_name)
+ default_ps.pmu_glob = strdup(unit_name);
+ else if (hybrid_name) {
+ default_ps.pmu_glob = perf_pmu__hybrid_type_to_pmu(hybrid_name);
+ if (!default_ps.pmu_glob)
+ pr_warning("WARNING: hybrid cputype is not supported!\n");
+ }
}
+ print_cb.print_start(ps);
if (argc == 0) {
- print_events(NULL, raw_dump, !desc_flag, long_desc_flag,
- details_flag, deprecated, pmu_name);
+ default_ps.metrics = true;
+ default_ps.metricgroups = true;
+ print_events(&print_cb, ps);
goto out;
}
@@ -78,68 +508,75 @@ int cmd_list(int argc, const char **argv)
char *sep, *s;
if (strcmp(argv[i], "tracepoint") == 0)
- print_tracepoint_events(NULL, NULL, raw_dump);
+ print_tracepoint_events(&print_cb, ps);
else if (strcmp(argv[i], "hw") == 0 ||
strcmp(argv[i], "hardware") == 0)
- print_symbol_events(NULL, PERF_TYPE_HARDWARE,
- event_symbols_hw, PERF_COUNT_HW_MAX, raw_dump);
+ print_symbol_events(&print_cb, ps, PERF_TYPE_HARDWARE,
+ event_symbols_hw, PERF_COUNT_HW_MAX);
else if (strcmp(argv[i], "sw") == 0 ||
strcmp(argv[i], "software") == 0) {
- print_symbol_events(NULL, PERF_TYPE_SOFTWARE,
- event_symbols_sw, PERF_COUNT_SW_MAX, raw_dump);
- print_tool_events(NULL, raw_dump);
+ print_symbol_events(&print_cb, ps, PERF_TYPE_SOFTWARE,
+ event_symbols_sw, PERF_COUNT_SW_MAX);
+ print_tool_events(&print_cb, ps);
} else if (strcmp(argv[i], "cache") == 0 ||
strcmp(argv[i], "hwcache") == 0)
- print_hwcache_events(NULL, raw_dump);
+ print_hwcache_events(&print_cb, ps);
else if (strcmp(argv[i], "pmu") == 0)
- print_pmu_events(NULL, raw_dump, !desc_flag,
- long_desc_flag, details_flag,
- deprecated, pmu_name);
+ print_pmu_events(&print_cb, ps);
else if (strcmp(argv[i], "sdt") == 0)
- print_sdt_events(NULL, NULL, raw_dump);
- else if (strcmp(argv[i], "metric") == 0 || strcmp(argv[i], "metrics") == 0)
- metricgroup__print(true, false, NULL, raw_dump, details_flag, pmu_name);
- else if (strcmp(argv[i], "metricgroup") == 0 || strcmp(argv[i], "metricgroups") == 0)
- metricgroup__print(false, true, NULL, raw_dump, details_flag, pmu_name);
- else if ((sep = strchr(argv[i], ':')) != NULL) {
- int sep_idx;
+ print_sdt_events(&print_cb, ps);
+ else if (strcmp(argv[i], "metric") == 0 || strcmp(argv[i], "metrics") == 0) {
+ default_ps.metricgroups = false;
+ default_ps.metrics = true;
+ metricgroup__print(&print_cb, ps);
+ } else if (strcmp(argv[i], "metricgroup") == 0 ||
+ strcmp(argv[i], "metricgroups") == 0) {
+ default_ps.metricgroups = true;
+ default_ps.metrics = false;
+ metricgroup__print(&print_cb, ps);
+ } else if ((sep = strchr(argv[i], ':')) != NULL) {
+ char *old_pmu_glob = default_ps.pmu_glob;
- sep_idx = sep - argv[i];
- s = strdup(argv[i]);
- if (s == NULL) {
+ default_ps.event_glob = strdup(argv[i]);
+ if (!default_ps.event_glob) {
ret = -1;
goto out;
}
- s[sep_idx] = '\0';
- print_tracepoint_events(s, s + sep_idx + 1, raw_dump);
- print_sdt_events(s, s + sep_idx + 1, raw_dump);
- metricgroup__print(true, true, s, raw_dump, details_flag, pmu_name);
- free(s);
+ print_tracepoint_events(&print_cb, ps);
+ print_sdt_events(&print_cb, ps);
+ default_ps.metrics = true;
+ default_ps.metricgroups = true;
+ metricgroup__print(&print_cb, ps);
+ zfree(&default_ps.event_glob);
+ default_ps.pmu_glob = old_pmu_glob;
} else {
if (asprintf(&s, "*%s*", argv[i]) < 0) {
printf("Critical: Not enough memory! Trying to continue...\n");
continue;
}
- print_symbol_events(s, PERF_TYPE_HARDWARE,
- event_symbols_hw, PERF_COUNT_HW_MAX, raw_dump);
- print_symbol_events(s, PERF_TYPE_SOFTWARE,
- event_symbols_sw, PERF_COUNT_SW_MAX, raw_dump);
- print_tool_events(s, raw_dump);
- print_hwcache_events(s, raw_dump);
- print_pmu_events(s, raw_dump, !desc_flag,
- long_desc_flag,
- details_flag,
- deprecated,
- pmu_name);
- print_tracepoint_events(NULL, s, raw_dump);
- print_sdt_events(NULL, s, raw_dump);
- metricgroup__print(true, true, s, raw_dump, details_flag, pmu_name);
+ default_ps.event_glob = s;
+ print_symbol_events(&print_cb, ps, PERF_TYPE_HARDWARE,
+ event_symbols_hw, PERF_COUNT_HW_MAX);
+ print_symbol_events(&print_cb, ps, PERF_TYPE_SOFTWARE,
+ event_symbols_sw, PERF_COUNT_SW_MAX);
+ print_tool_events(&print_cb, ps);
+ print_hwcache_events(&print_cb, ps);
+ print_pmu_events(&print_cb, ps);
+ print_tracepoint_events(&print_cb, ps);
+ print_sdt_events(&print_cb, ps);
+ default_ps.metrics = true;
+ default_ps.metricgroups = true;
+ metricgroup__print(&print_cb, ps);
free(s);
}
}
out:
- free(pmu_name);
+ print_cb.print_end(ps);
+ free(default_ps.pmu_glob);
+ free(default_ps.last_topic);
+ free(default_ps.last_metricgroups);
+ strlist__delete(default_ps.visited_metrics);
return ret;
}
diff --git a/tools/perf/builtin-lock.c b/tools/perf/builtin-lock.c
index 9722d4a..25c0a5e 100644
--- a/tools/perf/builtin-lock.c
+++ b/tools/perf/builtin-lock.c
@@ -12,6 +12,7 @@
#include "util/target.h"
#include "util/callchain.h"
#include "util/lock-contention.h"
+#include "util/bpf_skel/lock_data.h"
#include <subcmd/pager.h>
#include <subcmd/parse-options.h>
@@ -24,6 +25,7 @@
#include "util/data.h"
#include "util/string2.h"
#include "util/map.h"
+#include "util/util.h"
#include <sys/types.h>
#include <sys/prctl.h>
@@ -54,22 +56,14 @@ static struct rb_root thread_stats;
static bool combine_locks;
static bool show_thread_stats;
+static bool show_lock_addrs;
static bool use_bpf;
static unsigned long bpf_map_entries = 10240;
static int max_stack_depth = CONTENTION_STACK_DEPTH;
static int stack_skip = CONTENTION_STACK_SKIP;
static int print_nr_entries = INT_MAX / 2;
-static enum {
- LOCK_AGGR_ADDR,
- LOCK_AGGR_TASK,
- LOCK_AGGR_CALLER,
-} aggr_mode = LOCK_AGGR_ADDR;
-
-static u64 sched_text_start;
-static u64 sched_text_end;
-static u64 lock_text_start;
-static u64 lock_text_end;
+static enum lock_aggr_mode aggr_mode = LOCK_AGGR_ADDR;
static struct thread_stat *thread_stat_find(u32 tid)
{
@@ -853,55 +847,6 @@ static int report_lock_release_event(struct evsel *evsel,
return 0;
}
-bool is_lock_function(struct machine *machine, u64 addr)
-{
- if (!sched_text_start) {
- struct map *kmap;
- struct symbol *sym;
-
- sym = machine__find_kernel_symbol_by_name(machine,
- "__sched_text_start",
- &kmap);
- if (!sym) {
- /* to avoid retry */
- sched_text_start = 1;
- return false;
- }
-
- sched_text_start = kmap->unmap_ip(kmap, sym->start);
-
- /* should not fail from here */
- sym = machine__find_kernel_symbol_by_name(machine,
- "__sched_text_end",
- &kmap);
- sched_text_end = kmap->unmap_ip(kmap, sym->start);
-
- sym = machine__find_kernel_symbol_by_name(machine,
- "__lock_text_start",
- &kmap);
- lock_text_start = kmap->unmap_ip(kmap, sym->start);
-
- sym = machine__find_kernel_symbol_by_name(machine,
- "__lock_text_end",
- &kmap);
- lock_text_end = kmap->unmap_ip(kmap, sym->start);
- }
-
- /* failed to get kernel symbols */
- if (sched_text_start == 1)
- return false;
-
- /* mutex and rwsem functions are in sched text section */
- if (sched_text_start <= addr && addr < sched_text_end)
- return true;
-
- /* spinlock functions are in lock text section */
- if (lock_text_start <= addr && addr < lock_text_end)
- return true;
-
- return false;
-}
-
static int get_symbol_name_offset(struct map *map, struct symbol *sym, u64 ip,
char *buf, int size)
{
@@ -960,7 +905,7 @@ static int lock_contention_caller(struct evsel *evsel, struct perf_sample *sampl
goto next;
sym = node->ms.sym;
- if (sym && !is_lock_function(machine, node->ip)) {
+ if (sym && !machine__is_lock_function(machine, node->ip)) {
get_symbol_name_offset(node->ms.map, sym, node->ip,
buf, size);
return 0;
@@ -1006,7 +951,7 @@ static u64 callchain_id(struct evsel *evsel, struct perf_sample *sample)
if (++skip <= stack_skip)
goto next;
- if (node->ms.sym && is_lock_function(machine, node->ip))
+ if (node->ms.sym && machine__is_lock_function(machine, node->ip))
goto next;
hash ^= hash_long((unsigned long)node->ip, 64);
@@ -1055,13 +1000,32 @@ static int report_lock_contention_begin_event(struct evsel *evsel,
ls = lock_stat_find(key);
if (!ls) {
char buf[128];
- const char *caller = buf;
+ const char *name = "";
unsigned int flags = evsel__intval(evsel, sample, "flags");
+ struct machine *machine = &session->machines.host;
+ struct map *kmap;
+ struct symbol *sym;
- if (lock_contention_caller(evsel, sample, buf, sizeof(buf)) < 0)
- caller = "Unknown";
+ switch (aggr_mode) {
+ case LOCK_AGGR_ADDR:
+ /* make sure it loads the kernel map to find lock symbols */
+ map__load(machine__kernel_map(machine));
- ls = lock_stat_findnew(key, caller, flags);
+ sym = machine__find_kernel_symbol(machine, key, &kmap);
+ if (sym)
+ name = sym->name;
+ break;
+ case LOCK_AGGR_CALLER:
+ name = buf;
+ if (lock_contention_caller(evsel, sample, buf, sizeof(buf)) < 0)
+ name = "Unknown";
+ break;
+ case LOCK_AGGR_TASK:
+ default:
+ break;
+ }
+
+ ls = lock_stat_findnew(key, name, flags);
if (!ls)
return -ENOMEM;
@@ -1389,6 +1353,34 @@ static int dump_info(void)
return rc;
}
+static const struct evsel_str_handler lock_tracepoints[] = {
+ { "lock:lock_acquire", evsel__process_lock_acquire, }, /* CONFIG_LOCKDEP */
+ { "lock:lock_acquired", evsel__process_lock_acquired, }, /* CONFIG_LOCKDEP, CONFIG_LOCK_STAT */
+ { "lock:lock_contended", evsel__process_lock_contended, }, /* CONFIG_LOCKDEP, CONFIG_LOCK_STAT */
+ { "lock:lock_release", evsel__process_lock_release, }, /* CONFIG_LOCKDEP */
+};
+
+static const struct evsel_str_handler contention_tracepoints[] = {
+ { "lock:contention_begin", evsel__process_contention_begin, },
+ { "lock:contention_end", evsel__process_contention_end, },
+};
+
+static int process_event_update(struct perf_tool *tool,
+ union perf_event *event,
+ struct evlist **pevlist)
+{
+ int ret;
+
+ ret = perf_event__process_event_update(tool, event, pevlist);
+ if (ret < 0)
+ return ret;
+
+ /* this can return -EEXIST since we call it for each evsel */
+ perf_session__set_tracepoints_handlers(session, lock_tracepoints);
+ perf_session__set_tracepoints_handlers(session, contention_tracepoints);
+ return 0;
+}
+
typedef int (*tracepoint_handler)(struct evsel *evsel,
struct perf_sample *sample);
@@ -1488,10 +1480,19 @@ static void print_contention_result(struct lock_contention *con)
list_for_each_entry(key, &lock_keys, list)
pr_info("%*s ", key->len, key->header);
- if (show_thread_stats)
+ switch (aggr_mode) {
+ case LOCK_AGGR_TASK:
pr_info(" %10s %s\n\n", "pid", "comm");
- else
+ break;
+ case LOCK_AGGR_CALLER:
pr_info(" %10s %s\n\n", "type", "caller");
+ break;
+ case LOCK_AGGR_ADDR:
+ pr_info(" %16s %s\n\n", "address", "symbol");
+ break;
+ default:
+ break;
+ }
}
bad = total = printed = 0;
@@ -1499,6 +1500,9 @@ static void print_contention_result(struct lock_contention *con)
bad = bad_hist[BROKEN_CONTENDED];
while ((st = pop_from_result())) {
+ struct thread *t;
+ int pid;
+
total += use_bpf ? st->nr_contended : 1;
if (st->broken)
bad++;
@@ -1508,18 +1512,24 @@ static void print_contention_result(struct lock_contention *con)
pr_info(" ");
}
- if (show_thread_stats) {
- struct thread *t;
- int pid = st->addr;
-
- /* st->addr contains tid of thread */
+ switch (aggr_mode) {
+ case LOCK_AGGR_CALLER:
+ pr_info(" %10s %s\n", get_type_str(st), st->name);
+ break;
+ case LOCK_AGGR_TASK:
+ pid = st->addr;
t = perf_session__findnew(session, pid);
pr_info(" %10d %s\n", pid, thread__comm_str(t));
- goto next;
+ break;
+ case LOCK_AGGR_ADDR:
+ pr_info(" %016llx %s\n", (unsigned long long)st->addr,
+ st->name ? : "");
+ break;
+ default:
+ break;
}
- pr_info(" %10s %s\n", get_type_str(st), st->name);
- if (verbose) {
+ if (aggr_mode == LOCK_AGGR_CALLER && verbose) {
struct map *kmap;
struct symbol *sym;
char buf[128];
@@ -1536,7 +1546,6 @@ static void print_contention_result(struct lock_contention *con)
}
}
-next:
if (++printed >= print_nr_entries)
break;
}
@@ -1544,28 +1553,19 @@ static void print_contention_result(struct lock_contention *con)
print_bad_events(bad, total);
}
-static const struct evsel_str_handler lock_tracepoints[] = {
- { "lock:lock_acquire", evsel__process_lock_acquire, }, /* CONFIG_LOCKDEP */
- { "lock:lock_acquired", evsel__process_lock_acquired, }, /* CONFIG_LOCKDEP, CONFIG_LOCK_STAT */
- { "lock:lock_contended", evsel__process_lock_contended, }, /* CONFIG_LOCKDEP, CONFIG_LOCK_STAT */
- { "lock:lock_release", evsel__process_lock_release, }, /* CONFIG_LOCKDEP */
-};
-
-static const struct evsel_str_handler contention_tracepoints[] = {
- { "lock:contention_begin", evsel__process_contention_begin, },
- { "lock:contention_end", evsel__process_contention_end, },
-};
-
static bool force;
static int __cmd_report(bool display_info)
{
int err = -EINVAL;
struct perf_tool eops = {
+ .attr = perf_event__process_attr,
+ .event_update = process_event_update,
.sample = process_sample_event,
.comm = perf_event__process_comm,
.mmap = perf_event__process_mmap,
.namespaces = perf_event__process_namespaces,
+ .tracing_data = perf_event__process_tracing_data,
.ordered_events = true,
};
struct perf_data data = {
@@ -1584,17 +1584,19 @@ static int __cmd_report(bool display_info)
symbol_conf.sort_by_name = true;
symbol__init(&session->header.env);
- if (!perf_session__has_traces(session, "lock record"))
- goto out_delete;
+ if (!data.is_pipe) {
+ if (!perf_session__has_traces(session, "lock record"))
+ goto out_delete;
- if (perf_session__set_tracepoints_handlers(session, lock_tracepoints)) {
- pr_err("Initializing perf session tracepoint handlers failed\n");
- goto out_delete;
- }
+ if (perf_session__set_tracepoints_handlers(session, lock_tracepoints)) {
+ pr_err("Initializing perf session tracepoint handlers failed\n");
+ goto out_delete;
+ }
- if (perf_session__set_tracepoints_handlers(session, contention_tracepoints)) {
- pr_err("Initializing perf session tracepoint handlers failed\n");
- goto out_delete;
+ if (perf_session__set_tracepoints_handlers(session, contention_tracepoints)) {
+ pr_err("Initializing perf session tracepoint handlers failed\n");
+ goto out_delete;
+ }
}
if (setup_output_field(false, output_fields))
@@ -1632,9 +1634,12 @@ static int __cmd_contention(int argc, const char **argv)
{
int err = -EINVAL;
struct perf_tool eops = {
+ .attr = perf_event__process_attr,
+ .event_update = process_event_update,
.sample = process_sample_event,
.comm = perf_event__process_comm,
.mmap = perf_event__process_mmap,
+ .tracing_data = perf_event__process_tracing_data,
.ordered_events = true,
};
struct perf_data data = {
@@ -1658,6 +1663,9 @@ static int __cmd_contention(int argc, const char **argv)
con.machine = &session->machines.host;
+ con.aggr_mode = aggr_mode = show_thread_stats ? LOCK_AGGR_TASK :
+ show_lock_addrs ? LOCK_AGGR_ADDR : LOCK_AGGR_CALLER;
+
/* for lock function check */
symbol_conf.sort_by_name = true;
symbol__init(&session->header.env);
@@ -1697,7 +1705,7 @@ static int __cmd_contention(int argc, const char **argv)
pr_err("lock contention BPF setup failed\n");
goto out_delete;
}
- } else {
+ } else if (!data.is_pipe) {
if (!perf_session__has_traces(session, "lock record"))
goto out_delete;
@@ -1720,11 +1728,6 @@ static int __cmd_contention(int argc, const char **argv)
if (select_key(true))
goto out_delete;
- if (show_thread_stats)
- aggr_mode = LOCK_AGGR_TASK;
- else
- aggr_mode = LOCK_AGGR_CALLER;
-
if (use_bpf) {
lock_contention_start();
if (argc)
@@ -1858,6 +1861,29 @@ static int parse_map_entry(const struct option *opt, const char *str,
return 0;
}
+static int parse_max_stack(const struct option *opt, const char *str,
+ int unset __maybe_unused)
+{
+ unsigned long *len = (unsigned long *)opt->value;
+ long val;
+ char *endptr;
+
+ errno = 0;
+ val = strtol(str, &endptr, 0);
+ if (*endptr != '\0' || errno != 0) {
+ pr_err("invalid max stack depth: %s\n", str);
+ return -1;
+ }
+
+ if (val < 0 || val > sysctl__max_stack()) {
+ pr_err("invalid max stack depth: %ld\n", val);
+ return -1;
+ }
+
+ *len = val;
+ return 0;
+}
+
int cmd_lock(int argc, const char **argv)
{
const struct option lock_options[] = {
@@ -1869,7 +1895,7 @@ int cmd_lock(int argc, const char **argv)
"file", "vmlinux pathname"),
OPT_STRING(0, "kallsyms", &symbol_conf.kallsyms_name,
"file", "kallsyms pathname"),
- OPT_BOOLEAN('q', "quiet", &quiet, "Do not show any message"),
+ OPT_BOOLEAN('q', "quiet", &quiet, "Do not show any warnings or messages"),
OPT_END()
};
@@ -1913,13 +1939,14 @@ int cmd_lock(int argc, const char **argv)
"Trace on existing thread id (exclusive to --pid)"),
OPT_CALLBACK(0, "map-nr-entries", &bpf_map_entries, "num",
"Max number of BPF map entries", parse_map_entry),
- OPT_INTEGER(0, "max-stack", &max_stack_depth,
- "Set the maximum stack depth when collecting lock contention, "
- "Default: " __stringify(CONTENTION_STACK_DEPTH)),
+ OPT_CALLBACK(0, "max-stack", &max_stack_depth, "num",
+ "Set the maximum stack depth when collecting lopck contention, "
+ "Default: " __stringify(CONTENTION_STACK_DEPTH), parse_max_stack),
OPT_INTEGER(0, "stack-skip", &stack_skip,
"Set the number of stack depth to skip when finding a lock caller, "
"Default: " __stringify(CONTENTION_STACK_SKIP)),
OPT_INTEGER('E', "entries", &print_nr_entries, "display this many functions"),
+ OPT_BOOLEAN('l', "lock-addr", &show_lock_addrs, "show lock stats by address"),
OPT_PARENT(lock_options)
};
@@ -1989,6 +2016,16 @@ int cmd_lock(int argc, const char **argv)
argc = parse_options(argc, argv, contention_options,
contention_usage, 0);
}
+
+ if (show_thread_stats && show_lock_addrs) {
+ pr_err("Cannot use thread and addr mode together\n");
+ parse_options_usage(contention_usage, contention_options,
+ "threads", 0);
+ parse_options_usage(NULL, contention_options,
+ "lock-addr", 0);
+ return -1;
+ }
+
rc = __cmd_contention(argc, argv);
} else {
usage_with_options(lock_usage, lock_options);
diff --git a/tools/perf/builtin-mem.c b/tools/perf/builtin-mem.c
index 923fb83..dedd612 100644
--- a/tools/perf/builtin-mem.c
+++ b/tools/perf/builtin-mem.c
@@ -20,6 +20,7 @@
#include "util/symbol.h"
#include "util/pmu.h"
#include "util/pmu-hybrid.h"
+#include "util/sample.h"
#include "util/string2.h"
#include <linux/err.h>
diff --git a/tools/perf/builtin-probe.c b/tools/perf/builtin-probe.c
index f62298f..2ae50fc 100644
--- a/tools/perf/builtin-probe.c
+++ b/tools/perf/builtin-probe.c
@@ -40,7 +40,6 @@ static struct {
int command; /* Command short_name */
bool list_events;
bool uprobes;
- bool quiet;
bool target_used;
int nevents;
struct perf_probe_event events[MAX_PROBES];
@@ -514,8 +513,8 @@ __cmd_probe(int argc, const char **argv)
struct option options[] = {
OPT_INCR('v', "verbose", &verbose,
"be more verbose (show parsed arguments, etc)"),
- OPT_BOOLEAN('q', "quiet", ¶ms.quiet,
- "be quiet (do not show any messages)"),
+ OPT_BOOLEAN('q', "quiet", &quiet,
+ "be quiet (do not show any warnings or messages)"),
OPT_CALLBACK_DEFAULT('l', "list", NULL, "[GROUP:]EVENT",
"list up probe events",
opt_set_filter_with_command, DEFAULT_LIST_FILTER),
@@ -634,7 +633,7 @@ __cmd_probe(int argc, const char **argv)
if (ret)
return ret;
- if (params.quiet) {
+ if (quiet) {
if (verbose != 0) {
pr_err(" Error: -v and -q are exclusive.\n");
return -EINVAL;
diff --git a/tools/perf/builtin-record.c b/tools/perf/builtin-record.c
index 2711c14..8ecffa6 100644
--- a/tools/perf/builtin-record.c
+++ b/tools/perf/builtin-record.c
@@ -646,10 +646,10 @@ static int record__pushfn(struct mmap *map, void *to, void *bf, size_t size)
return record__write(rec, map, bf, size);
}
-static volatile int signr = -1;
-static volatile int child_finished;
+static volatile sig_atomic_t signr = -1;
+static volatile sig_atomic_t child_finished;
#ifdef HAVE_EVENTFD_SUPPORT
-static volatile int done_fd = -1;
+static volatile sig_atomic_t done_fd = -1;
#endif
static void sig_handler(int sig)
@@ -1701,8 +1701,10 @@ static void record__init_features(struct record *rec)
if (rec->no_buildid)
perf_header__clear_feat(&session->header, HEADER_BUILD_ID);
+#ifdef HAVE_LIBTRACEEVENT
if (!have_tracepoints(&rec->evlist->core.entries))
perf_header__clear_feat(&session->header, HEADER_TRACING_DATA);
+#endif
if (!rec->opts.branch_stack)
perf_header__clear_feat(&session->header, HEADER_BRANCH_STACK);
@@ -1926,7 +1928,7 @@ static void record__read_lost_samples(struct record *rec)
}
-static volatile int workload_exec_errno;
+static volatile sig_atomic_t workload_exec_errno;
/*
* evlist__prepare_workload will send a SIGUSR1
@@ -3378,8 +3380,6 @@ static struct option __record_options[] = {
OPT_CALLBACK(0, "mmap-flush", &record.opts, "number",
"Minimal number of bytes that is extracted from mmap data pages (default: 1)",
record__mmap_flush_parse),
- OPT_BOOLEAN(0, "group", &record.opts.group,
- "put the counters into a counter group"),
OPT_CALLBACK_NOOPT('g', NULL, &callchain_param,
NULL, "enables call-graph recording" ,
&record_callchain_opt),
@@ -3388,7 +3388,7 @@ static struct option __record_options[] = {
&record_parse_callchain_opt),
OPT_INCR('v', "verbose", &verbose,
"be more verbose (show counter open errors, etc)"),
- OPT_BOOLEAN('q', "quiet", &quiet, "don't print any message"),
+ OPT_BOOLEAN('q', "quiet", &quiet, "don't print any warnings or messages"),
OPT_BOOLEAN('s', "stat", &record.opts.inherit_stat,
"per thread counts"),
OPT_BOOLEAN('d', "data", &record.opts.sample_address, "Record the sample addresses"),
diff --git a/tools/perf/builtin-report.c b/tools/perf/builtin-report.c
index 8361890..2ee2ecc 100644
--- a/tools/perf/builtin-report.c
+++ b/tools/perf/builtin-report.c
@@ -67,6 +67,10 @@
#include <unistd.h>
#include <linux/mman.h>
+#ifdef HAVE_LIBTRACEEVENT
+#include <traceevent/event-parse.h>
+#endif
+
struct report {
struct perf_tool tool;
struct perf_session *session;
@@ -1199,7 +1203,9 @@ int cmd_report(int argc, const char **argv)
.lost = perf_event__process_lost,
.read = process_read_event,
.attr = process_attr,
+#ifdef HAVE_LIBTRACEEVENT
.tracing_data = perf_event__process_tracing_data,
+#endif
.build_id = perf_event__process_build_id,
.id_index = perf_event__process_id_index,
.auxtrace_info = perf_event__process_auxtrace_info,
@@ -1222,7 +1228,7 @@ int cmd_report(int argc, const char **argv)
"input file name"),
OPT_INCR('v', "verbose", &verbose,
"be more verbose (show symbol address, etc)"),
- OPT_BOOLEAN('q', "quiet", &quiet, "Do not show any message"),
+ OPT_BOOLEAN('q', "quiet", &quiet, "Do not show any warnings or messages"),
OPT_BOOLEAN('D', "dump-raw-trace", &dump_trace,
"dump raw trace in ASCII"),
OPT_BOOLEAN(0, "stats", &report.stats_mode, "Display event stats"),
@@ -1660,6 +1666,7 @@ int cmd_report(int argc, const char **argv)
report.range_num);
}
+#ifdef HAVE_LIBTRACEEVENT
if (session->tevent.pevent &&
tep_set_function_resolver(session->tevent.pevent,
machine__resolve_kernel_addr,
@@ -1668,7 +1675,7 @@ int cmd_report(int argc, const char **argv)
__func__);
return -1;
}
-
+#endif
sort__setup_elide(stdout);
ret = __cmd_report(&report);
diff --git a/tools/perf/builtin-script.c b/tools/perf/builtin-script.c
index 7ca2382..88888fb 100644
--- a/tools/perf/builtin-script.c
+++ b/tools/perf/builtin-script.c
@@ -62,6 +62,9 @@
#include "perf.h"
#include <linux/ctype.h>
+#ifdef HAVE_LIBTRACEEVENT
+#include <traceevent/event-parse.h>
+#endif
static char const *script_name;
static char const *generate_script_lang;
@@ -2049,7 +2052,7 @@ static void perf_sample__fprint_metric(struct perf_script *script,
u64 val;
if (!evsel->stats)
- evlist__alloc_stats(script->session->evlist, false);
+ evlist__alloc_stats(&stat_config, script->session->evlist, /*alloc_raw=*/false);
if (evsel_script(leader)->gnum++ == 0)
perf_stat__reset_shadow_stats();
val = sample->period * evsel->scale;
@@ -2154,12 +2157,12 @@ static void process_event(struct perf_script *script,
perf_sample__fprintf_bts(sample, evsel, thread, al, addr_al, machine, fp);
return;
}
-
+#ifdef HAVE_LIBTRACEEVENT
if (PRINT_FIELD(TRACE) && sample->raw_data) {
event_format__fprintf(evsel->tp_format, sample->cpu,
sample->raw_data, sample->raw_size, fp);
}
-
+#endif
if (attr->type == PERF_TYPE_SYNTH && PRINT_FIELD(SYNTH))
perf_sample__fprintf_synth(sample, evsel, fp);
@@ -2283,8 +2286,10 @@ static void process_stat_interval(u64 tstamp)
static void setup_scripting(void)
{
+#ifdef HAVE_LIBTRACEEVENT
setup_perl_scripting();
setup_python_scripting();
+#endif
}
static int flush_scripting(void)
@@ -3632,7 +3637,7 @@ static int set_maps(struct perf_script *script)
perf_evlist__set_maps(&evlist->core, script->cpus, script->threads);
- if (evlist__alloc_stats(evlist, true))
+ if (evlist__alloc_stats(&stat_config, evlist, /*alloc_raw=*/true))
return -ENOMEM;
script->allocated = true;
@@ -3784,7 +3789,9 @@ int cmd_script(int argc, const char **argv)
.fork = perf_event__process_fork,
.attr = process_attr,
.event_update = perf_event__process_event_update,
+#ifdef HAVE_LIBTRACEEVENT
.tracing_data = perf_event__process_tracing_data,
+#endif
.feature = process_feature_event,
.build_id = perf_event__process_build_id,
.id_index = perf_event__process_id_index,
@@ -4215,6 +4222,7 @@ int cmd_script(int argc, const char **argv)
else
symbol_conf.use_callchain = false;
+#ifdef HAVE_LIBTRACEEVENT
if (session->tevent.pevent &&
tep_set_function_resolver(session->tevent.pevent,
machine__resolve_kernel_addr,
@@ -4223,7 +4231,7 @@ int cmd_script(int argc, const char **argv)
err = -1;
goto out_delete;
}
-
+#endif
if (generate_script_lang) {
struct stat perf_stat;
int input;
@@ -4259,9 +4267,12 @@ int cmd_script(int argc, const char **argv)
err = -ENOENT;
goto out_delete;
}
-
+#ifdef HAVE_LIBTRACEEVENT
err = scripting_ops->generate_script(session->tevent.pevent,
"perf-script");
+#else
+ err = scripting_ops->generate_script(NULL, "perf-script");
+#endif
goto out_delete;
}
diff --git a/tools/perf/builtin-stat.c b/tools/perf/builtin-stat.c
index 265b051..bf640ab 100644
--- a/tools/perf/builtin-stat.c
+++ b/tools/perf/builtin-stat.c
@@ -93,6 +93,7 @@
#include <linux/ctype.h>
#include <perf/evlist.h>
+#include <internal/threadmap.h>
#define DEFAULT_SEPARATOR " "
#define FREEZE_ON_SMI_PATH "devices/cpu/freeze_on_smi"
@@ -173,14 +174,13 @@ static struct target target = {
#define METRIC_ONLY_LEN 20
-static volatile pid_t child_pid = -1;
+static volatile sig_atomic_t child_pid = -1;
static int detailed_run = 0;
static bool transaction_run;
static bool topdown_run = false;
static bool smi_cost = false;
static bool smi_reset = false;
static int big_num_opt = -1;
-static bool group = false;
static const char *pre_cmd = NULL;
static const char *post_cmd = NULL;
static bool sync_run = false;
@@ -208,7 +208,7 @@ struct perf_stat {
static struct perf_stat perf_stat;
#define STAT_RECORD perf_stat.record
-static volatile int done = 0;
+static volatile sig_atomic_t done = 0;
static struct perf_stat_config stat_config = {
.aggr_mode = AGGR_GLOBAL,
@@ -465,15 +465,19 @@ static int read_bpf_map_counters(void)
return 0;
}
-static void read_counters(struct timespec *rs)
+static int read_counters(struct timespec *rs)
{
- struct evsel *counter;
-
if (!stat_config.stop_read_counter) {
if (read_bpf_map_counters() ||
read_affinity_counters(rs))
- return;
+ return -1;
}
+ return 0;
+}
+
+static void process_counters(void)
+{
+ struct evsel *counter;
evlist__for_each_entry(evsel_list, counter) {
if (counter->err)
@@ -482,6 +486,10 @@ static void read_counters(struct timespec *rs)
pr_warning("failed to process counter %s\n", counter->name);
counter->err = 0;
}
+
+ perf_stat_merge_counters(&stat_config, evsel_list);
+ perf_stat_process_percore(&stat_config, evsel_list);
+ perf_stat_process_shadow_stats(&stat_config, evsel_list);
}
static void process_interval(void)
@@ -492,7 +500,10 @@ static void process_interval(void)
diff_timespec(&rs, &ts, &ref_time);
perf_stat__reset_shadow_per_stat(&rt_stat);
- read_counters(&rs);
+ evlist__reset_aggr_stats(evsel_list);
+
+ if (read_counters(&rs) == 0)
+ process_counters();
if (STAT_RECORD) {
if (WRITE_STAT_ROUND_EVENT(rs.tv_sec * NSEC_PER_SEC + rs.tv_nsec, INTERVAL))
@@ -528,26 +539,14 @@ static int enable_counters(void)
return err;
}
- if (stat_config.initial_delay < 0) {
- pr_info(EVLIST_DISABLED_MSG);
- return 0;
- }
-
- if (stat_config.initial_delay > 0) {
- pr_info(EVLIST_DISABLED_MSG);
- usleep(stat_config.initial_delay * USEC_PER_MSEC);
- }
-
/*
* We need to enable counters only if:
* - we don't have tracee (attaching to task or cpu)
* - we have initial delay configured
*/
- if (!target__none(&target) || stat_config.initial_delay) {
+ if (!target__none(&target)) {
if (!all_counters_use_bpf)
evlist__enable(evsel_list);
- if (stat_config.initial_delay > 0)
- pr_info(EVLIST_ENABLED_MSG);
}
return 0;
}
@@ -569,7 +568,7 @@ static void disable_counters(void)
}
}
-static volatile int workload_exec_errno;
+static volatile sig_atomic_t workload_exec_errno;
/*
* evlist__prepare_workload will send a SIGUSR1
@@ -769,9 +768,6 @@ static int __run_perf_stat(int argc, const char **argv, int run_idx)
child_pid = evsel_list->workload.pid;
}
- if (group)
- evlist__set_leader(evsel_list);
-
if (!cpu_map__is_dummy(evsel_list->core.user_requested_cpus)) {
if (affinity__setup(&saved_affinity) < 0)
return -1;
@@ -918,14 +914,27 @@ static int __run_perf_stat(int argc, const char **argv, int run_idx)
return err;
}
- err = enable_counters();
- if (err)
- return -1;
+ if (stat_config.initial_delay) {
+ pr_info(EVLIST_DISABLED_MSG);
+ } else {
+ err = enable_counters();
+ if (err)
+ return -1;
+ }
/* Exec the command, if any */
if (forks)
evlist__start_workload(evsel_list);
+ if (stat_config.initial_delay > 0) {
+ usleep(stat_config.initial_delay * USEC_PER_MSEC);
+ err = enable_counters();
+ if (err)
+ return -1;
+
+ pr_info(EVLIST_ENABLED_MSG);
+ }
+
t0 = rdclock();
clock_gettime(CLOCK_MONOTONIC, &ref_time);
@@ -963,11 +972,9 @@ static int __run_perf_stat(int argc, const char **argv, int run_idx)
init_stats(&walltime_nsecs_stats);
update_stats(&walltime_nsecs_stats, t1 - t0);
- if (stat_config.aggr_mode == AGGR_GLOBAL)
- evlist__save_aggr_prev_raw_counts(evsel_list);
-
evlist__copy_prev_raw_counts(evsel_list);
evlist__reset_prev_raw_counts(evsel_list);
+ evlist__reset_aggr_stats(evsel_list);
perf_stat__reset_shadow_per_stat(&rt_stat);
} else {
update_stats(&walltime_nsecs_stats, t1 - t0);
@@ -980,7 +987,8 @@ static int __run_perf_stat(int argc, const char **argv, int run_idx)
* avoid arbitrary skew, we must read all counters before closing any
* group leaders.
*/
- read_counters(&(struct timespec) { .tv_nsec = t1-t0 });
+ if (read_counters(&(struct timespec) { .tv_nsec = t1-t0 }) == 0)
+ process_counters();
/*
* We need to keep evsel_list alive, because it's processed
@@ -1023,13 +1031,13 @@ static void print_counters(struct timespec *ts, int argc, const char **argv)
/* Do not print anything if we record to the pipe. */
if (STAT_RECORD && perf_stat.data.is_pipe)
return;
- if (stat_config.quiet)
+ if (quiet)
return;
evlist__print_counters(evsel_list, &stat_config, &target, ts, argc, argv);
}
-static volatile int signr = -1;
+static volatile sig_atomic_t signr = -1;
static void skip_signal(int signo)
{
@@ -1181,8 +1189,6 @@ static struct option stat_options[] = {
#endif
OPT_BOOLEAN('a', "all-cpus", &target.system_wide,
"system-wide collection from all CPUs"),
- OPT_BOOLEAN('g', "group", &group,
- "put the counters into a counter group"),
OPT_BOOLEAN(0, "scale", &stat_config.scale,
"Use --no-scale to disable counter scaling for multiplexing"),
OPT_INCR('v', "verbose", &verbose,
@@ -1273,8 +1279,8 @@ static struct option stat_options[] = {
"print summary for interval mode"),
OPT_BOOLEAN(0, "no-csv-summary", &stat_config.no_csv_summary,
"don't print 'summary' for CSV summary output"),
- OPT_BOOLEAN(0, "quiet", &stat_config.quiet,
- "don't print output (useful with record)"),
+ OPT_BOOLEAN(0, "quiet", &quiet,
+ "don't print any output, messages or warnings (useful with record)"),
OPT_CALLBACK(0, "cputype", &evsel_list, "hybrid cpu type",
"Only enable events on applying cpu with this type "
"for hybrid platform (e.g. core or atom)",
@@ -1330,10 +1336,26 @@ static struct aggr_cpu_id perf_stat__get_node(struct perf_stat_config *config __
return aggr_cpu_id__node(cpu, /*data=*/NULL);
}
+static struct aggr_cpu_id perf_stat__get_global(struct perf_stat_config *config __maybe_unused,
+ struct perf_cpu cpu)
+{
+ return aggr_cpu_id__global(cpu, /*data=*/NULL);
+}
+
+static struct aggr_cpu_id perf_stat__get_cpu(struct perf_stat_config *config __maybe_unused,
+ struct perf_cpu cpu)
+{
+ return aggr_cpu_id__cpu(cpu, /*data=*/NULL);
+}
+
static struct aggr_cpu_id perf_stat__get_aggr(struct perf_stat_config *config,
aggr_get_id_t get_id, struct perf_cpu cpu)
{
- struct aggr_cpu_id id = aggr_cpu_id__empty();
+ struct aggr_cpu_id id;
+
+ /* per-process mode - should use global aggr mode */
+ if (cpu.cpu == -1)
+ return get_id(config, cpu);
if (aggr_cpu_id__is_empty(&config->cpus_aggr_map->map[cpu.cpu]))
config->cpus_aggr_map->map[cpu.cpu] = get_id(config, cpu);
@@ -1366,16 +1388,16 @@ static struct aggr_cpu_id perf_stat__get_node_cached(struct perf_stat_config *co
return perf_stat__get_aggr(config, perf_stat__get_node, cpu);
}
-static bool term_percore_set(void)
+static struct aggr_cpu_id perf_stat__get_global_cached(struct perf_stat_config *config,
+ struct perf_cpu cpu)
{
- struct evsel *counter;
+ return perf_stat__get_aggr(config, perf_stat__get_global, cpu);
+}
- evlist__for_each_entry(evsel_list, counter) {
- if (counter->percore)
- return true;
- }
-
- return false;
+static struct aggr_cpu_id perf_stat__get_cpu_cached(struct perf_stat_config *config,
+ struct perf_cpu cpu)
+{
+ return perf_stat__get_aggr(config, perf_stat__get_cpu, cpu);
}
static aggr_cpu_id_get_t aggr_mode__get_aggr(enum aggr_mode aggr_mode)
@@ -1390,11 +1412,9 @@ static aggr_cpu_id_get_t aggr_mode__get_aggr(enum aggr_mode aggr_mode)
case AGGR_NODE:
return aggr_cpu_id__node;
case AGGR_NONE:
- if (term_percore_set())
- return aggr_cpu_id__core;
-
- return NULL;
+ return aggr_cpu_id__cpu;
case AGGR_GLOBAL:
+ return aggr_cpu_id__global;
case AGGR_THREAD:
case AGGR_UNSET:
case AGGR_MAX:
@@ -1415,11 +1435,9 @@ static aggr_get_id_t aggr_mode__get_id(enum aggr_mode aggr_mode)
case AGGR_NODE:
return perf_stat__get_node_cached;
case AGGR_NONE:
- if (term_percore_set()) {
- return perf_stat__get_core_cached;
- }
- return NULL;
+ return perf_stat__get_cpu_cached;
case AGGR_GLOBAL:
+ return perf_stat__get_global_cached;
case AGGR_THREAD:
case AGGR_UNSET:
case AGGR_MAX:
@@ -1434,8 +1452,9 @@ static int perf_stat_init_aggr_mode(void)
aggr_cpu_id_get_t get_id = aggr_mode__get_aggr(stat_config.aggr_mode);
if (get_id) {
+ bool needs_sort = stat_config.aggr_mode != AGGR_NONE;
stat_config.aggr_map = cpu_aggr_map__new(evsel_list->core.user_requested_cpus,
- get_id, /*data=*/NULL);
+ get_id, /*data=*/NULL, needs_sort);
if (!stat_config.aggr_map) {
pr_err("cannot build %s map", aggr_mode__string[stat_config.aggr_mode]);
return -1;
@@ -1443,6 +1462,21 @@ static int perf_stat_init_aggr_mode(void)
stat_config.aggr_get_id = aggr_mode__get_id(stat_config.aggr_mode);
}
+ if (stat_config.aggr_mode == AGGR_THREAD) {
+ nr = perf_thread_map__nr(evsel_list->core.threads);
+ stat_config.aggr_map = cpu_aggr_map__empty_new(nr);
+ if (stat_config.aggr_map == NULL)
+ return -ENOMEM;
+
+ for (int s = 0; s < nr; s++) {
+ struct aggr_cpu_id id = aggr_cpu_id__empty();
+
+ id.thread_idx = s;
+ stat_config.aggr_map->map[s] = id;
+ }
+ return 0;
+ }
+
/*
* The evsel_list->cpus is the base we operate on,
* taking the highest cpu number to be the size of
@@ -1527,6 +1561,26 @@ static struct aggr_cpu_id perf_env__get_core_aggr_by_cpu(struct perf_cpu cpu, vo
return id;
}
+static struct aggr_cpu_id perf_env__get_cpu_aggr_by_cpu(struct perf_cpu cpu, void *data)
+{
+ struct perf_env *env = data;
+ struct aggr_cpu_id id = aggr_cpu_id__empty();
+
+ if (cpu.cpu != -1) {
+ /*
+ * core_id is relative to socket and die,
+ * we need a global id. So we set
+ * socket, die id and core id
+ */
+ id.socket = env->cpu[cpu.cpu].socket_id;
+ id.die = env->cpu[cpu.cpu].die_id;
+ id.core = env->cpu[cpu.cpu].core_id;
+ id.cpu = cpu;
+ }
+
+ return id;
+}
+
static struct aggr_cpu_id perf_env__get_node_aggr_by_cpu(struct perf_cpu cpu, void *data)
{
struct aggr_cpu_id id = aggr_cpu_id__empty();
@@ -1535,6 +1589,16 @@ static struct aggr_cpu_id perf_env__get_node_aggr_by_cpu(struct perf_cpu cpu, vo
return id;
}
+static struct aggr_cpu_id perf_env__get_global_aggr_by_cpu(struct perf_cpu cpu __maybe_unused,
+ void *data __maybe_unused)
+{
+ struct aggr_cpu_id id = aggr_cpu_id__empty();
+
+ /* it always aggregates to the cpu 0 */
+ id.cpu = (struct perf_cpu){ .cpu = 0 };
+ return id;
+}
+
static struct aggr_cpu_id perf_stat__get_socket_file(struct perf_stat_config *config __maybe_unused,
struct perf_cpu cpu)
{
@@ -1552,12 +1616,24 @@ static struct aggr_cpu_id perf_stat__get_core_file(struct perf_stat_config *conf
return perf_env__get_core_aggr_by_cpu(cpu, &perf_stat.session->header.env);
}
+static struct aggr_cpu_id perf_stat__get_cpu_file(struct perf_stat_config *config __maybe_unused,
+ struct perf_cpu cpu)
+{
+ return perf_env__get_cpu_aggr_by_cpu(cpu, &perf_stat.session->header.env);
+}
+
static struct aggr_cpu_id perf_stat__get_node_file(struct perf_stat_config *config __maybe_unused,
struct perf_cpu cpu)
{
return perf_env__get_node_aggr_by_cpu(cpu, &perf_stat.session->header.env);
}
+static struct aggr_cpu_id perf_stat__get_global_file(struct perf_stat_config *config __maybe_unused,
+ struct perf_cpu cpu)
+{
+ return perf_env__get_global_aggr_by_cpu(cpu, &perf_stat.session->header.env);
+}
+
static aggr_cpu_id_get_t aggr_mode__get_aggr_file(enum aggr_mode aggr_mode)
{
switch (aggr_mode) {
@@ -1569,8 +1645,10 @@ static aggr_cpu_id_get_t aggr_mode__get_aggr_file(enum aggr_mode aggr_mode)
return perf_env__get_core_aggr_by_cpu;
case AGGR_NODE:
return perf_env__get_node_aggr_by_cpu;
- case AGGR_NONE:
case AGGR_GLOBAL:
+ return perf_env__get_global_aggr_by_cpu;
+ case AGGR_NONE:
+ return perf_env__get_cpu_aggr_by_cpu;
case AGGR_THREAD:
case AGGR_UNSET:
case AGGR_MAX:
@@ -1590,8 +1668,10 @@ static aggr_get_id_t aggr_mode__get_id_file(enum aggr_mode aggr_mode)
return perf_stat__get_core_file;
case AGGR_NODE:
return perf_stat__get_node_file;
- case AGGR_NONE:
case AGGR_GLOBAL:
+ return perf_stat__get_global_file;
+ case AGGR_NONE:
+ return perf_stat__get_cpu_file;
case AGGR_THREAD:
case AGGR_UNSET:
case AGGR_MAX:
@@ -1604,11 +1684,29 @@ static int perf_stat_init_aggr_mode_file(struct perf_stat *st)
{
struct perf_env *env = &st->session->header.env;
aggr_cpu_id_get_t get_id = aggr_mode__get_aggr_file(stat_config.aggr_mode);
+ bool needs_sort = stat_config.aggr_mode != AGGR_NONE;
+
+ if (stat_config.aggr_mode == AGGR_THREAD) {
+ int nr = perf_thread_map__nr(evsel_list->core.threads);
+
+ stat_config.aggr_map = cpu_aggr_map__empty_new(nr);
+ if (stat_config.aggr_map == NULL)
+ return -ENOMEM;
+
+ for (int s = 0; s < nr; s++) {
+ struct aggr_cpu_id id = aggr_cpu_id__empty();
+
+ id.thread_idx = s;
+ stat_config.aggr_map->map[s] = id;
+ }
+ return 0;
+ }
if (!get_id)
return 0;
- stat_config.aggr_map = cpu_aggr_map__new(evsel_list->core.user_requested_cpus, get_id, env);
+ stat_config.aggr_map = cpu_aggr_map__new(evsel_list->core.user_requested_cpus,
+ get_id, env, needs_sort);
if (!stat_config.aggr_map) {
pr_err("cannot build %s map", aggr_mode__string[stat_config.aggr_mode]);
return -1;
@@ -1991,13 +2089,11 @@ static int process_stat_round_event(struct perf_session *session,
union perf_event *event)
{
struct perf_record_stat_round *stat_round = &event->stat_round;
- struct evsel *counter;
struct timespec tsh, *ts = NULL;
const char **argv = session->header.env.cmdline_argv;
int argc = session->header.env.nr_cmdline;
- evlist__for_each_entry(evsel_list, counter)
- perf_stat_process_counter(&stat_config, counter);
+ process_counters();
if (stat_round->type == PERF_STAT_ROUND_TYPE__FINAL)
update_stats(&walltime_nsecs_stats, stat_round->time);
@@ -2024,17 +2120,23 @@ int process_stat_config_event(struct perf_session *session,
if (perf_cpu_map__empty(st->cpus)) {
if (st->aggr_mode != AGGR_UNSET)
pr_warning("warning: processing task data, aggregation mode not set\n");
- return 0;
- }
-
- if (st->aggr_mode != AGGR_UNSET)
+ } else if (st->aggr_mode != AGGR_UNSET) {
stat_config.aggr_mode = st->aggr_mode;
+ }
if (perf_stat.data.is_pipe)
perf_stat_init_aggr_mode();
else
perf_stat_init_aggr_mode_file(st);
+ if (stat_config.aggr_map) {
+ int nr_aggr = stat_config.aggr_map->nr;
+
+ if (evlist__alloc_aggr_stats(session->evlist, nr_aggr) < 0) {
+ pr_err("cannot allocate aggr counts\n");
+ return -1;
+ }
+ }
return 0;
}
@@ -2048,7 +2150,7 @@ static int set_maps(struct perf_stat *st)
perf_evlist__set_maps(&evsel_list->core, st->cpus, st->threads);
- if (evlist__alloc_stats(evsel_list, true))
+ if (evlist__alloc_stats(&stat_config, evsel_list, /*alloc_raw=*/true))
return -ENOMEM;
st->maps_allocated = true;
@@ -2277,7 +2379,7 @@ int cmd_stat(int argc, const char **argv)
goto out;
}
- if (!output && !stat_config.quiet) {
+ if (!output && !quiet) {
struct timespec tm;
mode = append_file ? "a" : "w";
@@ -2297,6 +2399,14 @@ int cmd_stat(int argc, const char **argv)
}
}
+ if (stat_config.interval_clear && !isatty(fileno(output))) {
+ fprintf(stderr, "--interval-clear does not work with output\n");
+ parse_options_usage(stat_usage, stat_options, "o", 1);
+ parse_options_usage(NULL, stat_options, "log-fd", 0);
+ parse_options_usage(NULL, stat_options, "interval-clear", 0);
+ return -1;
+ }
+
stat_config.output = output;
/*
@@ -2495,10 +2605,10 @@ int cmd_stat(int argc, const char **argv)
goto out;
}
- if (evlist__alloc_stats(evsel_list, interval))
+ if (perf_stat_init_aggr_mode())
goto out;
- if (perf_stat_init_aggr_mode())
+ if (evlist__alloc_stats(&stat_config, evsel_list, interval))
goto out;
/*
diff --git a/tools/perf/builtin-timechart.c b/tools/perf/builtin-timechart.c
index c36296b..6c629e7 100644
--- a/tools/perf/builtin-timechart.c
+++ b/tools/perf/builtin-timechart.c
@@ -38,6 +38,7 @@
#include "util/string2.h"
#include "util/tracepoint.h"
#include <linux/err.h>
+#include <traceevent/event-parse.h>
#ifdef LACKS_OPEN_MEMSTREAM_PROTOTYPE
FILE *open_memstream(char **ptr, size_t *sizeloc);
diff --git a/tools/perf/builtin-top.c b/tools/perf/builtin-top.c
index 4b3ff76..d4b5b02 100644
--- a/tools/perf/builtin-top.c
+++ b/tools/perf/builtin-top.c
@@ -87,8 +87,8 @@
#include <linux/ctype.h>
#include <perf/mmap.h>
-static volatile int done;
-static volatile int resize;
+static volatile sig_atomic_t done;
+static volatile sig_atomic_t resize;
#define HEADER_LINE_NR 5
@@ -1471,8 +1471,6 @@ int cmd_top(int argc, const char **argv)
"dump the symbol table used for profiling"),
OPT_INTEGER('f', "count-filter", &top.count_filter,
"only display functions with more events than this"),
- OPT_BOOLEAN(0, "group", &opts->group,
- "put the counters into a counter group"),
OPT_BOOLEAN('i', "no-inherit", &opts->no_inherit,
"child tasks do not inherit counters"),
OPT_STRING(0, "sym-annotate", &top.sym_filter, "symbol name",
diff --git a/tools/perf/builtin-trace.c b/tools/perf/builtin-trace.c
index d3c7577..86e06f1 100644
--- a/tools/perf/builtin-trace.c
+++ b/tools/perf/builtin-trace.c
@@ -15,7 +15,6 @@
*/
#include "util/record.h"
-#include <traceevent/event-parse.h>
#include <api/fs/tracing_path.h>
#include <bpf/bpf.h>
#include "util/bpf_map.h"
@@ -80,6 +79,10 @@
#include <linux/ctype.h>
#include <perf/mmap.h>
+#ifdef HAVE_LIBTRACEEVENT
+#include <traceevent/event-parse.h>
+#endif
+
#ifndef O_CLOEXEC
# define O_CLOEXEC 02000000
#endif
@@ -88,6 +91,8 @@
# define F_LINUX_SPECIFIC_BASE 1024
#endif
+#define RAW_SYSCALL_ARGS_NUM 6
+
/*
* strtoul: Go from a string to a value, i.e. for msr: MSR_FS_BASE to 0xc0000100
*/
@@ -108,7 +113,7 @@ struct syscall_fmt {
const char *sys_enter,
*sys_exit;
} bpf_prog_name;
- struct syscall_arg_fmt arg[6];
+ struct syscall_arg_fmt arg[RAW_SYSCALL_ARGS_NUM];
u8 nr_args;
bool errpid;
bool timeout;
@@ -120,7 +125,6 @@ struct trace {
struct syscalltbl *sctbl;
struct {
struct syscall *table;
- struct bpf_map *map;
struct { // per syscall BPF_MAP_TYPE_PROG_ARRAY
struct bpf_map *sys_enter,
*sys_exit;
@@ -924,6 +928,8 @@ static struct syscall_fmt syscall_fmts[] = {
.arg = { [0] = { .scnprintf = SCA_PTR, /* brk */ }, }, },
{ .name = "clock_gettime",
.arg = { [0] = STRARRAY(clk_id, clockid), }, },
+ { .name = "clock_nanosleep",
+ .arg = { [2] = { .scnprintf = SCA_TIMESPEC, /* rqtp */ }, }, },
{ .name = "clone", .errpid = true, .nr_args = 5,
.arg = { [0] = { .name = "flags", .scnprintf = SCA_CLONE_FLAGS, },
[1] = { .name = "child_stack", .scnprintf = SCA_HEX, },
@@ -1053,7 +1059,8 @@ static struct syscall_fmt syscall_fmts[] = {
.arg = { [0] = { .scnprintf = SCA_FDAT, /* dfd */ },
[2] = { .scnprintf = SCA_OPEN_FLAGS, /* flags */ }, }, },
{ .name = "perf_event_open",
- .arg = { [2] = { .scnprintf = SCA_INT, /* cpu */ },
+ .arg = { [0] = { .scnprintf = SCA_PERF_ATTR, /* attr */ },
+ [2] = { .scnprintf = SCA_INT, /* cpu */ },
[3] = { .scnprintf = SCA_FD, /* group_fd */ },
[4] = { .scnprintf = SCA_PERF_FLAGS, /* flags */ }, }, },
{ .name = "pipe2",
@@ -1220,16 +1227,6 @@ struct syscall {
};
/*
- * Must match what is in the BPF program:
- *
- * tools/perf/examples/bpf/augmented_raw_syscalls.c
- */
-struct bpf_map_syscall_entry {
- bool enabled;
- u16 string_args_len[6];
-};
-
-/*
* We need to have this 'calculated' boolean because in some cases we really
* don't know what is the duration of a syscall, for instance, when we start
* a session and some threads are waiting for a syscall to finish, say 'poll',
@@ -1535,8 +1532,8 @@ static size_t trace__fprintf_tstamp(struct trace *trace, u64 tstamp, FILE *fp)
}
static pid_t workload_pid = -1;
-static bool done = false;
-static bool interrupted = false;
+static volatile sig_atomic_t done = false;
+static volatile sig_atomic_t interrupted = false;
static void sighandler_interrupt(int sig __maybe_unused)
{
@@ -1658,7 +1655,7 @@ static int syscall__alloc_arg_fmts(struct syscall *sc, int nr_args)
{
int idx;
- if (nr_args == 6 && sc->fmt && sc->fmt->nr_args != 0)
+ if (nr_args == RAW_SYSCALL_ARGS_NUM && sc->fmt && sc->fmt->nr_args != 0)
nr_args = sc->fmt->nr_args;
sc->arg_fmt = calloc(nr_args, sizeof(*sc->arg_fmt));
@@ -1730,7 +1727,7 @@ syscall_arg_fmt__init_array(struct syscall_arg_fmt *arg, struct tep_format_field
len >= 2 && strcmp(field->name + len - 2, "fd") == 0) {
/*
* /sys/kernel/tracing/events/syscalls/sys_enter*
- * egrep 'field:.*fd;' .../format|sed -r 's/.*field:([a-z ]+) [a-z_]*fd.+/\1/g'|sort|uniq -c
+ * grep -E 'field:.*fd;' .../format|sed -r 's/.*field:([a-z ]+) [a-z_]*fd.+/\1/g'|sort|uniq -c
* 65 int
* 23 unsigned int
* 7 unsigned long
@@ -1791,11 +1788,11 @@ static int trace__read_syscall_info(struct trace *trace, int id)
#endif
sc = trace->syscalls.table + id;
if (sc->nonexistent)
- return 0;
+ return -EEXIST;
if (name == NULL) {
sc->nonexistent = true;
- return 0;
+ return -EEXIST;
}
sc->name = name;
@@ -1809,11 +1806,18 @@ static int trace__read_syscall_info(struct trace *trace, int id)
sc->tp_format = trace_event__tp_format("syscalls", tp_name);
}
- if (syscall__alloc_arg_fmts(sc, IS_ERR(sc->tp_format) ? 6 : sc->tp_format->format.nr_fields))
- return -ENOMEM;
-
- if (IS_ERR(sc->tp_format))
+ /*
+ * Fails to read trace point format via sysfs node, so the trace point
+ * doesn't exist. Set the 'nonexistent' flag as true.
+ */
+ if (IS_ERR(sc->tp_format)) {
+ sc->nonexistent = true;
return PTR_ERR(sc->tp_format);
+ }
+
+ if (syscall__alloc_arg_fmts(sc, IS_ERR(sc->tp_format) ?
+ RAW_SYSCALL_ARGS_NUM : sc->tp_format->format.nr_fields))
+ return -ENOMEM;
sc->args = sc->tp_format->format.fields;
/*
@@ -2131,11 +2135,8 @@ static struct syscall *trace__syscall_info(struct trace *trace,
(err = trace__read_syscall_info(trace, id)) != 0)
goto out_cant_read;
- if (trace->syscalls.table[id].name == NULL) {
- if (trace->syscalls.table[id].nonexistent)
- return NULL;
+ if (trace->syscalls.table && trace->syscalls.table[id].nonexistent)
goto out_cant_read;
- }
return &trace->syscalls.table[id];
@@ -2728,8 +2729,10 @@ static size_t trace__fprintf_tp_fields(struct trace *trace, struct evsel *evsel,
offset = format_field__intval(field, sample, evsel->needs_swap);
syscall_arg.len = offset >> 16;
offset &= 0xffff;
+#ifdef HAVE_LIBTRACEEVENT_TEP_FIELD_IS_RELATIVE
if (field->flags & TEP_FIELD_IS_RELATIVE)
offset += field->offset + field->size;
+#endif
}
val = (uintptr_t)(sample->raw_data + offset);
@@ -3250,7 +3253,6 @@ static void trace__set_bpf_map_filtered_pids(struct trace *trace)
static void trace__set_bpf_map_syscalls(struct trace *trace)
{
- trace->syscalls.map = trace__find_bpf_map_by_name(trace, "syscalls");
trace->syscalls.prog_array.sys_enter = trace__find_bpf_map_by_name(trace, "syscalls_sys_enter");
trace->syscalls.prog_array.sys_exit = trace__find_bpf_map_by_name(trace, "syscalls_sys_exit");
}
@@ -3330,80 +3332,6 @@ static int trace__bpf_prog_sys_exit_fd(struct trace *trace, int id)
return sc ? bpf_program__fd(sc->bpf_prog.sys_exit) : bpf_program__fd(trace->syscalls.unaugmented_prog);
}
-static void trace__init_bpf_map_syscall_args(struct trace *trace, int id, struct bpf_map_syscall_entry *entry)
-{
- struct syscall *sc = trace__syscall_info(trace, NULL, id);
- int arg = 0;
-
- if (sc == NULL)
- goto out;
-
- for (; arg < sc->nr_args; ++arg) {
- entry->string_args_len[arg] = 0;
- if (sc->arg_fmt[arg].scnprintf == SCA_FILENAME) {
- /* Should be set like strace -s strsize */
- entry->string_args_len[arg] = PATH_MAX;
- }
- }
-out:
- for (; arg < 6; ++arg)
- entry->string_args_len[arg] = 0;
-}
-static int trace__set_ev_qualifier_bpf_filter(struct trace *trace)
-{
- int fd = bpf_map__fd(trace->syscalls.map);
- struct bpf_map_syscall_entry value = {
- .enabled = !trace->not_ev_qualifier,
- };
- int err = 0;
- size_t i;
-
- for (i = 0; i < trace->ev_qualifier_ids.nr; ++i) {
- int key = trace->ev_qualifier_ids.entries[i];
-
- if (value.enabled) {
- trace__init_bpf_map_syscall_args(trace, key, &value);
- trace__init_syscall_bpf_progs(trace, key);
- }
-
- err = bpf_map_update_elem(fd, &key, &value, BPF_EXIST);
- if (err)
- break;
- }
-
- return err;
-}
-
-static int __trace__init_syscalls_bpf_map(struct trace *trace, bool enabled)
-{
- int fd = bpf_map__fd(trace->syscalls.map);
- struct bpf_map_syscall_entry value = {
- .enabled = enabled,
- };
- int err = 0, key;
-
- for (key = 0; key < trace->sctbl->syscalls.nr_entries; ++key) {
- if (enabled)
- trace__init_bpf_map_syscall_args(trace, key, &value);
-
- err = bpf_map_update_elem(fd, &key, &value, BPF_ANY);
- if (err)
- break;
- }
-
- return err;
-}
-
-static int trace__init_syscalls_bpf_map(struct trace *trace)
-{
- bool enabled = true;
-
- if (trace->ev_qualifier_ids.nr)
- enabled = trace->not_ev_qualifier;
-
- return __trace__init_syscalls_bpf_map(trace, enabled);
-}
-
static struct bpf_program *trace__find_usable_bpf_prog_entry(struct trace *trace, struct syscall *sc)
{
struct tep_format_field *field, *candidate_field;
@@ -3618,16 +3546,6 @@ static void trace__set_bpf_map_syscalls(struct trace *trace __maybe_unused)
{
}
-static int trace__set_ev_qualifier_bpf_filter(struct trace *trace __maybe_unused)
-{
- return 0;
-}
-
-static int trace__init_syscalls_bpf_map(struct trace *trace __maybe_unused)
-{
- return 0;
-}
-
static struct bpf_program *trace__find_bpf_program_by_title(struct trace *trace __maybe_unused,
const char *name __maybe_unused)
{
@@ -3661,8 +3579,6 @@ static bool trace__only_augmented_syscalls_evsels(struct trace *trace)
static int trace__set_ev_qualifier_filter(struct trace *trace)
{
- if (trace->syscalls.map)
- return trace__set_ev_qualifier_bpf_filter(trace);
if (trace->syscalls.events.sys_enter)
return trace__set_ev_qualifier_tp_filter(trace);
return 0;
@@ -4036,9 +3952,6 @@ static int trace__run(struct trace *trace, int argc, const char **argv)
if (err < 0)
goto out_error_mem;
- if (trace->syscalls.map)
- trace__init_syscalls_bpf_map(trace);
-
if (trace->syscalls.prog_array.sys_enter)
trace__init_syscalls_bpf_prog_array_maps(trace);
@@ -4092,8 +4005,8 @@ static int trace__run(struct trace *trace, int argc, const char **argv)
}
trace->multiple_threads = perf_thread_map__pid(evlist->core.threads, 0) == -1 ||
- evlist->core.threads->nr > 1 ||
- evlist__first(evlist)->core.attr.inherit;
+ perf_thread_map__nr(evlist->core.threads) > 1 ||
+ evlist__first(evlist)->core.attr.inherit;
/*
* Now that we already used evsel->core.attr to ask the kernel to setup the
diff --git a/tools/perf/builtin-version.c b/tools/perf/builtin-version.c
index a71f491..a886929 100644
--- a/tools/perf/builtin-version.c
+++ b/tools/perf/builtin-version.c
@@ -82,6 +82,7 @@ static void library_status(void)
STATUS(HAVE_AIO_SUPPORT, aio);
STATUS(HAVE_ZSTD_SUPPORT, zstd);
STATUS(HAVE_LIBPFM, libpfm4);
+ STATUS(HAVE_LIBTRACEEVENT, libtraceevent);
}
int cmd_version(int argc, const char **argv)
diff --git a/tools/perf/examples/bpf/5sec.c b/tools/perf/examples/bpf/5sec.c
index e6b6181..3bd7fc17 100644
--- a/tools/perf/examples/bpf/5sec.c
+++ b/tools/perf/examples/bpf/5sec.c
@@ -39,13 +39,15 @@
Copyright (C) 2018 Red Hat, Inc., Arnaldo Carvalho de Melo <acme@redhat.com>
*/
-#include <bpf.h>
+#include <linux/bpf.h>
+#include <bpf/bpf_helpers.h>
#define NSEC_PER_SEC 1000000000L
-int probe(hrtimer_nanosleep, rqtp)(void *ctx, int err, long long sec)
+SEC("hrtimer_nanosleep=hrtimer_nanosleep rqtp")
+int hrtimer_nanosleep(void *ctx, int err, long long sec)
{
return sec / NSEC_PER_SEC == 5ULL;
}
-license(GPL);
+char _license[] SEC("license") = "GPL";
diff --git a/tools/perf/examples/bpf/augmented_raw_syscalls.c b/tools/perf/examples/bpf/augmented_raw_syscalls.c
index a262dcd..9a03189 100644
--- a/tools/perf/examples/bpf/augmented_raw_syscalls.c
+++ b/tools/perf/examples/bpf/augmented_raw_syscalls.c
@@ -14,39 +14,52 @@
* code that will combine entry/exit in a strace like way.
*/
-#include <unistd.h>
+#include <linux/bpf.h>
+#include <bpf/bpf_helpers.h>
#include <linux/limits.h>
-#include <linux/socket.h>
-#include <pid_filter.h>
-/* bpf-output associated map */
-bpf_map(__augmented_syscalls__, PERF_EVENT_ARRAY, int, u32, __NR_CPUS__);
+// FIXME: These should come from system headers
+typedef char bool;
+typedef int pid_t;
+typedef long long int __s64;
+typedef __s64 time64_t;
-/*
- * string_args_len: one per syscall arg, 0 means not a string or don't copy it,
- * PATH_MAX for copying everything, any other value to limit
- * it a la 'strace -s strsize'.
- */
-struct syscall {
- bool enabled;
- u16 string_args_len[6];
+struct timespec64 {
+ time64_t tv_sec;
+ long int tv_nsec;
};
-bpf_map(syscalls, ARRAY, int, struct syscall, 512);
+/* bpf-output associated map */
+struct __augmented_syscalls__ {
+ __uint(type, BPF_MAP_TYPE_PERF_EVENT_ARRAY);
+ __type(key, int);
+ __type(value, __u32);
+ __uint(max_entries, __NR_CPUS__);
+} __augmented_syscalls__ SEC(".maps");
/*
* What to augment at entry?
*
* Pointer arg payloads (filenames, etc) passed from userspace to the kernel
*/
-bpf_map(syscalls_sys_enter, PROG_ARRAY, u32, u32, 512);
+struct syscalls_sys_enter {
+ __uint(type, BPF_MAP_TYPE_PROG_ARRAY);
+ __type(key, __u32);
+ __type(value, __u32);
+ __uint(max_entries, 512);
+} syscalls_sys_enter SEC(".maps");
/*
* What to augment at exit?
*
* Pointer arg payloads returned from the kernel (struct stat, etc) to userspace.
*/
-bpf_map(syscalls_sys_exit, PROG_ARRAY, u32, u32, 512);
+struct syscalls_sys_exit {
+ __uint(type, BPF_MAP_TYPE_PROG_ARRAY);
+ __type(key, __u32);
+ __type(value, __u32);
+ __uint(max_entries, 512);
+} syscalls_sys_exit SEC(".maps");
struct syscall_enter_args {
unsigned long long common_tp_fields;
@@ -66,7 +79,38 @@ struct augmented_arg {
char value[PATH_MAX];
};
-pid_filter(pids_filtered);
+struct pids_filtered {
+ __uint(type, BPF_MAP_TYPE_HASH);
+ __type(key, pid_t);
+ __type(value, bool);
+ __uint(max_entries, 64);
+} pids_filtered SEC(".maps");
+
+/*
+ * Desired design of maximum size and alignment (see RFC2553)
+ */
+#define SS_MAXSIZE 128 /* Implementation specific max size */
+
+typedef unsigned short sa_family_t;
+
+/*
+ * FIXME: Should come from system headers
+ *
+ * The definition uses anonymous union and struct in order to control the
+ * default alignment.
+ */
+struct sockaddr_storage {
+ union {
+ struct {
+ sa_family_t ss_family; /* address family */
+ /* Following field(s) are implementation specific */
+ char __data[SS_MAXSIZE - sizeof(unsigned short)];
+ /* space to achieve desired size, */
+ /* _SS_MAXSIZE value minus size of ss_family */
+ };
+ void *__align; /* implementation specific desired alignment */
+ };
+};
struct augmented_args_payload {
struct syscall_enter_args args;
@@ -75,11 +119,17 @@ struct augmented_args_payload {
struct augmented_arg arg, arg2;
};
struct sockaddr_storage saddr;
+ char __data[sizeof(struct augmented_arg)];
};
};
// We need more tmp space than the BPF stack can give us
-bpf_map(augmented_args_tmp, PERCPU_ARRAY, int, struct augmented_args_payload, 1);
+struct augmented_args_tmp {
+ __uint(type, BPF_MAP_TYPE_PERCPU_ARRAY);
+ __type(key, int);
+ __type(value, struct augmented_args_payload);
+ __uint(max_entries, 1);
+} augmented_args_tmp SEC(".maps");
static inline struct augmented_args_payload *augmented_args_payload(void)
{
@@ -90,14 +140,14 @@ static inline struct augmented_args_payload *augmented_args_payload(void)
static inline int augmented__output(void *ctx, struct augmented_args_payload *args, int len)
{
/* If perf_event_output fails, return non-zero so that it gets recorded unaugmented */
- return perf_event_output(ctx, &__augmented_syscalls__, BPF_F_CURRENT_CPU, args, len);
+ return bpf_perf_event_output(ctx, &__augmented_syscalls__, BPF_F_CURRENT_CPU, args, len);
}
static inline
unsigned int augmented_arg__read_str(struct augmented_arg *augmented_arg, const void *arg, unsigned int arg_len)
{
unsigned int augmented_len = sizeof(*augmented_arg);
- int string_len = probe_read_str(&augmented_arg->value, arg_len, arg);
+ int string_len = bpf_probe_read_str(&augmented_arg->value, arg_len, arg);
augmented_arg->size = augmented_arg->err = 0;
/*
@@ -146,7 +196,7 @@ int sys_enter_connect(struct syscall_enter_args *args)
if (socklen > sizeof(augmented_args->saddr))
socklen = sizeof(augmented_args->saddr);
- probe_read(&augmented_args->saddr, socklen, sockaddr_arg);
+ bpf_probe_read(&augmented_args->saddr, socklen, sockaddr_arg);
return augmented__output(args, augmented_args, len + socklen);
}
@@ -165,7 +215,7 @@ int sys_enter_sendto(struct syscall_enter_args *args)
if (socklen > sizeof(augmented_args->saddr))
socklen = sizeof(augmented_args->saddr);
- probe_read(&augmented_args->saddr, socklen, sockaddr_arg);
+ bpf_probe_read(&augmented_args->saddr, socklen, sockaddr_arg);
return augmented__output(args, augmented_args, len + socklen);
}
@@ -234,6 +284,80 @@ int sys_enter_renameat(struct syscall_enter_args *args)
return augmented__output(args, augmented_args, len);
}
+#define PERF_ATTR_SIZE_VER0 64 /* sizeof first published struct */
+
+// we need just the start, get the size to then copy it
+struct perf_event_attr_size {
+ __u32 type;
+ /*
+ * Size of the attr structure, for fwd/bwd compat.
+ */
+ __u32 size;
+};
+
+SEC("!syscalls:sys_enter_perf_event_open")
+int sys_enter_perf_event_open(struct syscall_enter_args *args)
+{
+ struct augmented_args_payload *augmented_args = augmented_args_payload();
+ const struct perf_event_attr_size *attr = (const struct perf_event_attr_size *)args->args[0], *attr_read;
+ unsigned int len = sizeof(augmented_args->args);
+
+ if (augmented_args == NULL)
+ goto failure;
+
+ if (bpf_probe_read(&augmented_args->__data, sizeof(*attr), attr) < 0)
+ goto failure;
+
+ attr_read = (const struct perf_event_attr_size *)augmented_args->__data;
+
+ __u32 size = attr_read->size;
+
+ if (!size)
+ size = PERF_ATTR_SIZE_VER0;
+
+ if (size > sizeof(augmented_args->__data))
+ goto failure;
+
+ // Now that we read attr->size and tested it against the size limits, read it completely
+ if (bpf_probe_read(&augmented_args->__data, size, attr) < 0)
+ goto failure;
+
+ return augmented__output(args, augmented_args, len + size);
+failure:
+ return 1; /* Failure: don't filter */
+}
+
+SEC("!syscalls:sys_enter_clock_nanosleep")
+int sys_enter_clock_nanosleep(struct syscall_enter_args *args)
+{
+ struct augmented_args_payload *augmented_args = augmented_args_payload();
+ const void *rqtp_arg = (const void *)args->args[2];
+ unsigned int len = sizeof(augmented_args->args);
+ __u32 size = sizeof(struct timespec64);
+
+ if (augmented_args == NULL)
+ goto failure;
+
+ if (size > sizeof(augmented_args->__data))
+ goto failure;
+
+ bpf_probe_read(&augmented_args->__data, size, rqtp_arg);
+
+ return augmented__output(args, augmented_args, len + size);
+failure:
+ return 1; /* Failure: don't filter */
+}
+
+static pid_t getpid(void)
+{
+ return bpf_get_current_pid_tgid();
+}
+
+static bool pid_filter__has(struct pids_filtered *pids, pid_t pid)
+{
+ return bpf_map_lookup_elem(pids, &pid) != NULL;
+}
+
SEC("raw_syscalls:sys_enter")
int sys_enter(struct syscall_enter_args *args)
{
@@ -248,7 +372,6 @@ int sys_enter(struct syscall_enter_args *args)
* initial, non-augmented raw_syscalls:sys_enter payload.
*/
unsigned int len = sizeof(augmented_args->args);
- struct syscall *syscall;
if (pid_filter__has(&pids_filtered, getpid()))
return 0;
@@ -257,7 +380,7 @@ int sys_enter(struct syscall_enter_args *args)
if (augmented_args == NULL)
return 1;
- probe_read(&augmented_args->args, sizeof(augmented_args->args), args);
+ bpf_probe_read(&augmented_args->args, sizeof(augmented_args->args), args);
/*
* Jump to syscall specific augmenter, even if the default one,
@@ -278,7 +401,7 @@ int sys_exit(struct syscall_exit_args *args)
if (pid_filter__has(&pids_filtered, getpid()))
return 0;
- probe_read(&exit_args, sizeof(exit_args), args);
+ bpf_probe_read(&exit_args, sizeof(exit_args), args);
/*
* Jump to syscall specific return augmenter, even if the default one,
* "!raw_syscalls:unaugmented" that will just return 1 to return the
@@ -291,4 +414,4 @@ int sys_exit(struct syscall_exit_args *args)
return 0;
}
-license(GPL);
+char _license[] SEC("license") = "GPL";
diff --git a/tools/perf/examples/bpf/augmented_syscalls.c b/tools/perf/examples/bpf/augmented_syscalls.c
deleted file mode 100644
index 524fdb8..0000000
--- a/tools/perf/examples/bpf/augmented_syscalls.c
+++ /dev/null
@@ -1,169 +0,0 @@
-// SPDX-License-Identifier: GPL-2.0
-/*
- * Augment syscalls with the contents of the pointer arguments.
- *
- * Test it with:
- *
- * perf trace -e tools/perf/examples/bpf/augmented_syscalls.c cat /etc/passwd > /dev/null
- *
- * It'll catch some openat syscalls related to the dynamic linked and
- * the last one should be the one for '/etc/passwd'.
- *
- * This matches what is marshalled into the raw_syscall:sys_enter payload
- * expected by the 'perf trace' beautifiers, and can be used by them, that will
- * check if perf_sample->raw_data is more than what is expected for each
- * syscalls:sys_{enter,exit}_SYSCALL tracepoint, uing the extra data as the
- * contents of pointer arguments.
- */
-
-#include <stdio.h>
-#include <linux/socket.h>
-
-/* bpf-output associated map */
-bpf_map(__augmented_syscalls__, PERF_EVENT_ARRAY, int, u32, __NR_CPUS__);
-
-struct syscall_exit_args {
- unsigned long long common_tp_fields;
- long syscall_nr;
- long ret;
-};
-
-struct augmented_filename {
- unsigned int size;
- int reserved;
- char value[256];
-};
-
-#define augmented_filename_syscall(syscall) \
-struct augmented_enter_##syscall##_args { \
- struct syscall_enter_##syscall##_args args; \
- struct augmented_filename filename; \
-}; \
-int syscall_enter(syscall)(struct syscall_enter_##syscall##_args *args) \
-{ \
- struct augmented_enter_##syscall##_args augmented_args = { .filename.reserved = 0, }; \
- unsigned int len = sizeof(augmented_args); \
- probe_read(&augmented_args.args, sizeof(augmented_args.args), args); \
- augmented_args.filename.size = probe_read_str(&augmented_args.filename.value, \
- sizeof(augmented_args.filename.value), \
- args->filename_ptr); \
- if (augmented_args.filename.size < sizeof(augmented_args.filename.value)) { \
- len -= sizeof(augmented_args.filename.value) - augmented_args.filename.size; \
- len &= sizeof(augmented_args.filename.value) - 1; \
- } \
- /* If perf_event_output fails, return non-zero so that it gets recorded unaugmented */ \
- return perf_event_output(args, &__augmented_syscalls__, BPF_F_CURRENT_CPU, \
- &augmented_args, len); \
-} \
-int syscall_exit(syscall)(struct syscall_exit_args *args) \
-{ \
- return 1; /* 0 as soon as we start copying data returned by the kernel, e.g. 'read' */ \
-}
-
-struct syscall_enter_openat_args {
- unsigned long long common_tp_fields;
- long syscall_nr;
- long dfd;
- char *filename_ptr;
- long flags;
- long mode;
-};
-
-augmented_filename_syscall(openat);
-
-struct syscall_enter_open_args {
- unsigned long long common_tp_fields;
- long syscall_nr;
- char *filename_ptr;
- long flags;
- long mode;
-};
-
-augmented_filename_syscall(open);
-
-struct syscall_enter_inotify_add_watch_args {
- unsigned long long common_tp_fields;
- long syscall_nr;
- long fd;
- char *filename_ptr;
- long mask;
-};
-
-augmented_filename_syscall(inotify_add_watch);
-
-struct statbuf;
-
-struct syscall_enter_newstat_args {
- unsigned long long common_tp_fields;
- long syscall_nr;
- char *filename_ptr;
- struct stat *statbuf;
-};
-
-augmented_filename_syscall(newstat);
-
-#ifndef _K_SS_MAXSIZE
-#define _K_SS_MAXSIZE 128
-#endif
-
-#define augmented_sockaddr_syscall(syscall) \
-struct augmented_enter_##syscall##_args { \
- struct syscall_enter_##syscall##_args args; \
- struct sockaddr_storage addr; \
-}; \
-int syscall_enter(syscall)(struct syscall_enter_##syscall##_args *args) \
-{ \
- struct augmented_enter_##syscall##_args augmented_args; \
- unsigned long addrlen = sizeof(augmented_args.addr); \
- probe_read(&augmented_args.args, sizeof(augmented_args.args), args); \
-/* FIXME_CLANG_OPTIMIZATION_THAT_ACCESSES_USER_CONTROLLED_ADDRLEN_DESPITE_THIS_CHECK */ \
-/* if (addrlen > augmented_args.args.addrlen) */ \
-/* addrlen = augmented_args.args.addrlen; */ \
-/* */ \
- probe_read(&augmented_args.addr, addrlen, args->addr_ptr); \
- /* If perf_event_output fails, return non-zero so that it gets recorded unaugmented */ \
- return perf_event_output(args, &__augmented_syscalls__, BPF_F_CURRENT_CPU, \
- &augmented_args, \
- sizeof(augmented_args) - sizeof(augmented_args.addr) + addrlen);\
-} \
-int syscall_exit(syscall)(struct syscall_exit_args *args) \
-{ \
- return 1; /* 0 as soon as we start copying data returned by the kernel, e.g. 'read' */ \
-}
-
-struct sockaddr;
-
-struct syscall_enter_bind_args {
- unsigned long long common_tp_fields;
- long syscall_nr;
- long fd;
- struct sockaddr *addr_ptr;
- unsigned long addrlen;
-};
-
-augmented_sockaddr_syscall(bind);
-
-struct syscall_enter_connect_args {
- unsigned long long common_tp_fields;
- long syscall_nr;
- long fd;
- struct sockaddr *addr_ptr;
- unsigned long addrlen;
-};
-
-augmented_sockaddr_syscall(connect);
-
-struct syscall_enter_sendto_args {
- unsigned long long common_tp_fields;
- long syscall_nr;
- long fd;
- void *buff;
- long len;
- unsigned long flags;
- struct sockaddr *addr_ptr;
- long addr_len;
-};
-
-augmented_sockaddr_syscall(sendto);
-
-license(GPL);
diff --git a/tools/perf/examples/bpf/empty.c b/tools/perf/examples/bpf/empty.c
index 7d7fb0c..3e296c0 100644
--- a/tools/perf/examples/bpf/empty.c
+++ b/tools/perf/examples/bpf/empty.c
@@ -1,3 +1,12 @@
-#include <bpf/bpf.h>
+// SPDX-License-Identifier: GPL-2.0
+#include <linux/bpf.h>
+#include <bpf/bpf_helpers.h>
-license(GPL);
+struct syscall_enter_args;
+
+SEC("raw_syscalls:sys_enter")
+int sys_enter(struct syscall_enter_args *args)
+{
+ return 0;
+}
+char _license[] SEC("license") = "GPL";
diff --git a/tools/perf/examples/bpf/etcsnoop.c b/tools/perf/examples/bpf/etcsnoop.c
deleted file mode 100644
index e81b535..0000000
--- a/tools/perf/examples/bpf/etcsnoop.c
+++ /dev/null
@@ -1,76 +0,0 @@
-// SPDX-License-Identifier: GPL-2.0
-/*
- * Augment the filename syscalls with the contents of the filename pointer argument
- * filtering only those that do not start with /etc/.
- *
- * Test it with:
- *
- * perf trace -e tools/perf/examples/bpf/augmented_syscalls.c cat /etc/passwd > /dev/null
- *
- * It'll catch some openat syscalls related to the dynamic linked and
- * the last one should be the one for '/etc/passwd'.
- *
- * This matches what is marshalled into the raw_syscall:sys_enter payload
- * expected by the 'perf trace' beautifiers, and can be used by them unmodified,
- * which will be done as that feature is implemented in the next csets, for now
- * it will appear in a dump done by the default tracepoint handler in 'perf trace',
- * that uses bpf_output__fprintf() to just dump those contents, as done with
- * the bpf-output event associated with the __bpf_output__ map declared in
- * tools/perf/include/bpf/stdio.h.
- */
-
-#include <stdio.h>
-
-/* bpf-output associated map */
-bpf_map(__augmented_syscalls__, PERF_EVENT_ARRAY, int, u32, __NR_CPUS__);
-
-struct augmented_filename {
- int size;
- int reserved;
- char value[64];
-};
-
-#define augmented_filename_syscall_enter(syscall) \
-struct augmented_enter_##syscall##_args { \
- struct syscall_enter_##syscall##_args args; \
- struct augmented_filename filename; \
-}; \
-int syscall_enter(syscall)(struct syscall_enter_##syscall##_args *args) \
-{ \
- char etc[6] = "/etc/"; \
- struct augmented_enter_##syscall##_args augmented_args = { .filename.reserved = 0, }; \
- probe_read(&augmented_args.args, sizeof(augmented_args.args), args); \
- augmented_args.filename.size = probe_read_str(&augmented_args.filename.value, \
- sizeof(augmented_args.filename.value), \
- args->filename_ptr); \
- if (__builtin_memcmp(augmented_args.filename.value, etc, 4) != 0) \
- return 0; \
- /* If perf_event_output fails, return non-zero so that it gets recorded unaugmented */ \
- return perf_event_output(args, &__augmented_syscalls__, BPF_F_CURRENT_CPU, \
- &augmented_args, \
- (sizeof(augmented_args) - sizeof(augmented_args.filename.value) + \
- augmented_args.filename.size)); \
-}
-
-struct syscall_enter_openat_args {
- unsigned long long common_tp_fields;
- long syscall_nr;
- long dfd;
- char *filename_ptr;
- long flags;
- long mode;
-};
-
-augmented_filename_syscall_enter(openat);
-
-struct syscall_enter_open_args {
- unsigned long long common_tp_fields;
- long syscall_nr;
- char *filename_ptr;
- long flags;
- long mode;
-};
-
-augmented_filename_syscall_enter(open);
-
-license(GPL);
diff --git a/tools/perf/examples/bpf/hello.c b/tools/perf/examples/bpf/hello.c
index cf3c2fd..e9080b0 100644
--- a/tools/perf/examples/bpf/hello.c
+++ b/tools/perf/examples/bpf/hello.c
@@ -1,9 +1,27 @@
-#include <stdio.h>
+// SPDX-License-Identifier: GPL-2.0
+#include <linux/bpf.h>
+#include <bpf/bpf_helpers.h>
-int syscall_enter(openat)(void *args)
+struct __bpf_stdout__ {
+ __uint(type, BPF_MAP_TYPE_PERF_EVENT_ARRAY);
+ __type(key, int);
+ __type(value, __u32);
+ __uint(max_entries, __NR_CPUS__);
+} __bpf_stdout__ SEC(".maps");
+
+#define puts(from) \
+ ({ const int __len = sizeof(from); \
+ char __from[sizeof(from)] = from; \
+ bpf_perf_event_output(args, &__bpf_stdout__, BPF_F_CURRENT_CPU, \
+ &__from, __len & (sizeof(from) - 1)); })
+
+struct syscall_enter_args;
+
+SEC("raw_syscalls:sys_enter")
+int sys_enter(struct syscall_enter_args *args)
{
puts("Hello, world\n");
return 0;
}
-license(GPL);
+char _license[] SEC("license") = "GPL";
diff --git a/tools/perf/include/bpf/bpf.h b/tools/perf/include/bpf/bpf.h
deleted file mode 100644
index b422aee..0000000
--- a/tools/perf/include/bpf/bpf.h
+++ /dev/null
@@ -1,70 +0,0 @@
-// SPDX-License-Identifier: GPL-2.0
-#ifndef _PERF_BPF_H
-#define _PERF_BPF_H
-
-#include <uapi/linux/bpf.h>
-
-/*
- * A helper structure used by eBPF C program to describe map attributes to
- * elf_bpf loader, taken from tools/testing/selftests/bpf/bpf_helpers.h:
- */
-struct bpf_map {
- unsigned int type;
- unsigned int key_size;
- unsigned int value_size;
- unsigned int max_entries;
- unsigned int map_flags;
- unsigned int inner_map_idx;
- unsigned int numa_node;
-};
-
-#define bpf_map(name, _type, type_key, type_val, _max_entries) \
-struct bpf_map SEC("maps") name = { \
- .type = BPF_MAP_TYPE_##_type, \
- .key_size = sizeof(type_key), \
- .value_size = sizeof(type_val), \
- .max_entries = _max_entries, \
-}; \
-struct ____btf_map_##name { \
- type_key key; \
- type_val value; \
-}; \
-struct ____btf_map_##name __attribute__((section(".maps." #name), used)) \
- ____btf_map_##name = { }
-
-/*
- * FIXME: this should receive .max_entries as a parameter, as careful
- * tuning of these limits is needed to avoid hitting limits that
- * prevents other BPF constructs, such as tracepoint handlers,
- * to get installed, with cryptic messages from libbpf, etc.
- * For the current need, 'perf trace --filter-pids', 64 should
- * be good enough, but this surely needs to be revisited.
- */
-#define pid_map(name, value_type) bpf_map(name, HASH, pid_t, value_type, 64)
-
-static int (*bpf_map_update_elem)(struct bpf_map *map, void *key, void *value, u64 flags) = (void *)BPF_FUNC_map_update_elem;
-static void *(*bpf_map_lookup_elem)(struct bpf_map *map, void *key) = (void *)BPF_FUNC_map_lookup_elem;
-
-static void (*bpf_tail_call)(void *ctx, void *map, int index) = (void *)BPF_FUNC_tail_call;
-
-#define SEC(NAME) __attribute__((section(NAME), used))
-
-#define probe(function, vars) \
- SEC(#function "=" #function " " #vars) function
-
-#define syscall_enter(name) \
- SEC("syscalls:sys_enter_" #name) syscall_enter_ ## name
-
-#define syscall_exit(name) \
- SEC("syscalls:sys_exit_" #name) syscall_exit_ ## name
-
-#define license(name) \
-char _license[] SEC("license") = #name; \
-int _version SEC("version") = LINUX_VERSION_CODE;
-
-static int (*probe_read)(void *dst, int size, const void *unsafe_addr) = (void *)BPF_FUNC_probe_read;
-static int (*probe_read_str)(void *dst, int size, const void *unsafe_addr) = (void *)BPF_FUNC_probe_read_str;
-
-static int (*perf_event_output)(void *, struct bpf_map *, int, void *, unsigned long) = (void *)BPF_FUNC_perf_event_output;
-
-#endif /* _PERF_BPF_H */
diff --git a/tools/perf/include/bpf/linux/socket.h b/tools/perf/include/bpf/linux/socket.h
deleted file mode 100644
index 7f844568..0000000
--- a/tools/perf/include/bpf/linux/socket.h
+++ /dev/null
@@ -1,24 +0,0 @@
-/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */
-#ifndef _UAPI_LINUX_SOCKET_H
-#define _UAPI_LINUX_SOCKET_H
-
-/*
- * Desired design of maximum size and alignment (see RFC2553)
- */
-#define _K_SS_MAXSIZE 128 /* Implementation specific max size */
-#define _K_SS_ALIGNSIZE (__alignof__ (struct sockaddr *))
- /* Implementation specific desired alignment */
-
-typedef unsigned short __kernel_sa_family_t;
-
-struct __kernel_sockaddr_storage {
- __kernel_sa_family_t ss_family; /* address family */
- /* Following field(s) are implementation specific */
- char __data[_K_SS_MAXSIZE - sizeof(unsigned short)];
- /* space to achieve desired size, */
- /* _SS_MAXSIZE value minus size of ss_family */
-} __attribute__ ((aligned(_K_SS_ALIGNSIZE))); /* force desired alignment */
-
-#define sockaddr_storage __kernel_sockaddr_storage
-
-#endif /* _UAPI_LINUX_SOCKET_H */
diff --git a/tools/perf/include/bpf/pid_filter.h b/tools/perf/include/bpf/pid_filter.h
deleted file mode 100644
index 6e61c4b..0000000
--- a/tools/perf/include/bpf/pid_filter.h
+++ /dev/null
@@ -1,21 +0,0 @@
-// SPDX-License-Identifier: LGPL-2.1
-
-#ifndef _PERF_BPF_PID_FILTER_
-#define _PERF_BPF_PID_FILTER_
-
-#include <bpf.h>
-
-#define pid_filter(name) pid_map(name, bool)
-
-static int pid_filter__add(struct bpf_map *pids, pid_t pid)
-{
- bool value = true;
- return bpf_map_update_elem(pids, &pid, &value, BPF_NOEXIST);
-}
-
-static bool pid_filter__has(struct bpf_map *pids, pid_t pid)
-{
- return bpf_map_lookup_elem(pids, &pid) != NULL;
-}
-
-#endif // _PERF_BPF_PID_FILTER_
diff --git a/tools/perf/include/bpf/stdio.h b/tools/perf/include/bpf/stdio.h
deleted file mode 100644
index 316af5b..0000000
--- a/tools/perf/include/bpf/stdio.h
+++ /dev/null
@@ -1,16 +0,0 @@
-// SPDX-License-Identifier: GPL-2.0
-
-#include <bpf.h>
-
-struct bpf_map SEC("maps") __bpf_stdout__ = {
- .type = BPF_MAP_TYPE_PERF_EVENT_ARRAY,
- .key_size = sizeof(int),
- .value_size = sizeof(u32),
- .max_entries = __NR_CPUS__,
-};
-
-#define puts(from) \
- ({ const int __len = sizeof(from); \
- char __from[__len] = from; \
- perf_event_output(args, &__bpf_stdout__, BPF_F_CURRENT_CPU, \
- &__from, __len & (sizeof(from) - 1)); })
diff --git a/tools/perf/include/bpf/unistd.h b/tools/perf/include/bpf/unistd.h
deleted file mode 100644
index ca7877f..0000000
--- a/tools/perf/include/bpf/unistd.h
+++ /dev/null
@@ -1,10 +0,0 @@
-// SPDX-License-Identifier: LGPL-2.1
-
-#include <bpf.h>
-
-static int (*bpf_get_current_pid_tgid)(void) = (void *)BPF_FUNC_get_current_pid_tgid;
-
-static pid_t getpid(void)
-{
- return bpf_get_current_pid_tgid();
-}
diff --git a/tools/perf/perf.c b/tools/perf/perf.c
index 7af135d..82bbe0c 100644
--- a/tools/perf/perf.c
+++ b/tools/perf/perf.c
@@ -70,20 +70,26 @@ static struct cmd_struct commands[] = {
{ "report", cmd_report, 0 },
{ "bench", cmd_bench, 0 },
{ "stat", cmd_stat, 0 },
+#ifdef HAVE_LIBTRACEEVENT
{ "timechart", cmd_timechart, 0 },
+#endif
{ "top", cmd_top, 0 },
{ "annotate", cmd_annotate, 0 },
{ "version", cmd_version, 0 },
{ "script", cmd_script, 0 },
+#ifdef HAVE_LIBTRACEEVENT
{ "sched", cmd_sched, 0 },
+#endif
#ifdef HAVE_LIBELF_SUPPORT
{ "probe", cmd_probe, 0 },
#endif
+#ifdef HAVE_LIBTRACEEVENT
{ "kmem", cmd_kmem, 0 },
{ "lock", cmd_lock, 0 },
+#endif
{ "kvm", cmd_kvm, 0 },
{ "test", cmd_test, 0 },
-#if defined(HAVE_LIBAUDIT_SUPPORT) || defined(HAVE_SYSCALL_TABLE_SUPPORT)
+#if defined(HAVE_LIBTRACEEVENT) && (defined(HAVE_LIBAUDIT_SUPPORT) || defined(HAVE_SYSCALL_TABLE_SUPPORT))
{ "trace", cmd_trace, 0 },
#endif
{ "inject", cmd_inject, 0 },
@@ -91,7 +97,9 @@ static struct cmd_struct commands[] = {
{ "data", cmd_data, 0 },
{ "ftrace", cmd_ftrace, 0 },
{ "daemon", cmd_daemon, 0 },
+#ifdef HAVE_LIBTRACEEVENT
{ "kwork", cmd_kwork, 0 },
+#endif
};
struct pager_config {
@@ -500,14 +508,18 @@ int main(int argc, const char **argv)
argv[0] = cmd;
}
if (strstarts(cmd, "trace")) {
-#if defined(HAVE_LIBAUDIT_SUPPORT) || defined(HAVE_SYSCALL_TABLE_SUPPORT)
- setup_path();
- argv[0] = "trace";
- return cmd_trace(argc, argv);
-#else
+#ifndef HAVE_LIBTRACEEVENT
+ fprintf(stderr,
+ "trace command not available: missing libtraceevent devel package at build time.\n");
+ goto out;
+#elif !defined(HAVE_LIBAUDIT_SUPPORT) && !defined(HAVE_SYSCALL_TABLE_SUPPORT)
fprintf(stderr,
"trace command not available: missing audit-libs devel package at build time.\n");
goto out;
+#else
+ setup_path();
+ argv[0] = "trace";
+ return cmd_trace(argc, argv);
#endif
}
/* Look for flags.. */
diff --git a/tools/perf/pmu-events/Build b/tools/perf/pmu-events/Build
index 04ef951..15b9e8f 100644
--- a/tools/perf/pmu-events/Build
+++ b/tools/perf/pmu-events/Build
@@ -21,7 +21,7 @@
$(call rule_mkdir)
$(Q)$(call echo-cmd,gen)cp $< $@
else
-$(OUTPUT)pmu-events/pmu-events.c: $(JSON) $(JSON_TEST) $(JEVENTS_PY)
+$(OUTPUT)pmu-events/pmu-events.c: $(JSON) $(JSON_TEST) $(JEVENTS_PY) pmu-events/metric.py
$(call rule_mkdir)
$(Q)$(call echo-cmd,gen)$(PYTHON) $(JEVENTS_PY) $(JEVENTS_ARCH) pmu-events/arch $@
endif
diff --git a/tools/perf/pmu-events/arch/arm64/arm/neoverse-n2/branch.json b/tools/perf/pmu-events/arch/arm64/arm/neoverse-n2-v2/branch.json
similarity index 100%
rename from tools/perf/pmu-events/arch/arm64/arm/neoverse-n2/branch.json
rename to tools/perf/pmu-events/arch/arm64/arm/neoverse-n2-v2/branch.json
diff --git a/tools/perf/pmu-events/arch/arm64/arm/neoverse-n2/bus.json b/tools/perf/pmu-events/arch/arm64/arm/neoverse-n2-v2/bus.json
similarity index 100%
rename from tools/perf/pmu-events/arch/arm64/arm/neoverse-n2/bus.json
rename to tools/perf/pmu-events/arch/arm64/arm/neoverse-n2-v2/bus.json
diff --git a/tools/perf/pmu-events/arch/arm64/arm/neoverse-n2/cache.json b/tools/perf/pmu-events/arch/arm64/arm/neoverse-n2-v2/cache.json
similarity index 100%
rename from tools/perf/pmu-events/arch/arm64/arm/neoverse-n2/cache.json
rename to tools/perf/pmu-events/arch/arm64/arm/neoverse-n2-v2/cache.json
diff --git a/tools/perf/pmu-events/arch/arm64/arm/neoverse-n2/exception.json b/tools/perf/pmu-events/arch/arm64/arm/neoverse-n2-v2/exception.json
similarity index 100%
rename from tools/perf/pmu-events/arch/arm64/arm/neoverse-n2/exception.json
rename to tools/perf/pmu-events/arch/arm64/arm/neoverse-n2-v2/exception.json
diff --git a/tools/perf/pmu-events/arch/arm64/arm/neoverse-n2/instruction.json b/tools/perf/pmu-events/arch/arm64/arm/neoverse-n2-v2/instruction.json
similarity index 100%
rename from tools/perf/pmu-events/arch/arm64/arm/neoverse-n2/instruction.json
rename to tools/perf/pmu-events/arch/arm64/arm/neoverse-n2-v2/instruction.json
diff --git a/tools/perf/pmu-events/arch/arm64/arm/neoverse-n2/memory.json b/tools/perf/pmu-events/arch/arm64/arm/neoverse-n2-v2/memory.json
similarity index 100%
rename from tools/perf/pmu-events/arch/arm64/arm/neoverse-n2/memory.json
rename to tools/perf/pmu-events/arch/arm64/arm/neoverse-n2-v2/memory.json
diff --git a/tools/perf/pmu-events/arch/arm64/arm/neoverse-n2/pipeline.json b/tools/perf/pmu-events/arch/arm64/arm/neoverse-n2-v2/pipeline.json
similarity index 100%
rename from tools/perf/pmu-events/arch/arm64/arm/neoverse-n2/pipeline.json
rename to tools/perf/pmu-events/arch/arm64/arm/neoverse-n2-v2/pipeline.json
diff --git a/tools/perf/pmu-events/arch/arm64/arm/neoverse-n2/spe.json b/tools/perf/pmu-events/arch/arm64/arm/neoverse-n2-v2/spe.json
similarity index 100%
rename from tools/perf/pmu-events/arch/arm64/arm/neoverse-n2/spe.json
rename to tools/perf/pmu-events/arch/arm64/arm/neoverse-n2-v2/spe.json
diff --git a/tools/perf/pmu-events/arch/arm64/arm/neoverse-n2/trace.json b/tools/perf/pmu-events/arch/arm64/arm/neoverse-n2-v2/trace.json
similarity index 100%
rename from tools/perf/pmu-events/arch/arm64/arm/neoverse-n2/trace.json
rename to tools/perf/pmu-events/arch/arm64/arm/neoverse-n2-v2/trace.json
diff --git a/tools/perf/pmu-events/arch/arm64/mapfile.csv b/tools/perf/pmu-events/arch/arm64/mapfile.csv
index ad502d0..f134e83 100644
--- a/tools/perf/pmu-events/arch/arm64/mapfile.csv
+++ b/tools/perf/pmu-events/arch/arm64/mapfile.csv
@@ -34,7 +34,8 @@
0x00000000410fd460,v1,arm/cortex-a510,core
0x00000000410fd470,v1,arm/cortex-a710,core
0x00000000410fd480,v1,arm/cortex-x2,core
-0x00000000410fd490,v1,arm/neoverse-n2,core
+0x00000000410fd490,v1,arm/neoverse-n2-v2,core
+0x00000000410fd4f0,v1,arm/neoverse-n2-v2,core
0x00000000420f5160,v1,cavium/thunderx2,core
0x00000000430f0af0,v1,cavium/thunderx2,core
0x00000000460f0010,v1,fujitsu/a64fx,core
diff --git a/tools/perf/pmu-events/arch/riscv/mapfile.csv b/tools/perf/pmu-events/arch/riscv/mapfile.csv
new file mode 100644
index 0000000..c61b3d6
--- /dev/null
+++ b/tools/perf/pmu-events/arch/riscv/mapfile.csv
@@ -0,0 +1,17 @@
+# Format:
+# MVENDORID-MARCHID-MIMPID,Version,JSON/file/pathname,Type
+#
+# where
+# MVENDORID JEDEC code of the core provider
+# MARCHID base microarchitecture of the hart
+# MIMPID unique encoding of the version
+# of the processor implementation
+# Version could be used to track version of JSON file
+# but currently unused.
+# JSON/file/pathname is the path to JSON file, relative
+# to tools/perf/pmu-events/arch/riscv/.
+# Type is core, uncore etc
+#
+#
+#MVENDORID-MARCHID-MIMPID,Version,Filename,EventType
+0x489-0x8000000000000007-0x[[:xdigit:]]+,v1,sifive/u74,core
diff --git a/tools/perf/pmu-events/arch/riscv/riscv-sbi-firmware.json b/tools/perf/pmu-events/arch/riscv/riscv-sbi-firmware.json
new file mode 100644
index 0000000..a993982
--- /dev/null
+++ b/tools/perf/pmu-events/arch/riscv/riscv-sbi-firmware.json
@@ -0,0 +1,134 @@
+[
+ {
+ "PublicDescription": "Misaligned load trap",
+ "ConfigCode": "0x8000000000000000",
+ "EventName": "FW_MISALIGNED_LOAD",
+ "BriefDescription": "Misaligned load trap event"
+ },
+ {
+ "PublicDescription": "Misaligned store trap",
+ "ConfigCode": "0x8000000000000001",
+ "EventName": "FW_MISALIGNED_STORE",
+ "BriefDescription": "Misaligned store trap event"
+ },
+ {
+ "PublicDescription": "Load access trap",
+ "ConfigCode": "0x8000000000000002",
+ "EventName": "FW_ACCESS_LOAD",
+ "BriefDescription": "Load access trap event"
+ },
+ {
+ "PublicDescription": "Store access trap",
+ "ConfigCode": "0x8000000000000003",
+ "EventName": "FW_ACCESS_STORE",
+ "BriefDescription": "Store access trap event"
+ },
+ {
+ "PublicDescription": "Illegal instruction trap",
+ "ConfigCode": "0x8000000000000004",
+ "EventName": "FW_ILLEGAL_INSN",
+ "BriefDescription": "Illegal instruction trap event"
+ },
+ {
+ "PublicDescription": "Set timer event",
+ "ConfigCode": "0x8000000000000005",
+ "EventName": "FW_SET_TIMER",
+ "BriefDescription": "Set timer event"
+ },
+ {
+ "PublicDescription": "Sent IPI to other HART event",
+ "ConfigCode": "0x8000000000000006",
+ "EventName": "FW_IPI_SENT",
+ "BriefDescription": "Sent IPI to other HART event"
+ },
+ {
+ "PublicDescription": "Received IPI from other HART event",
+ "ConfigCode": "0x8000000000000007",
+ "EventName": "FW_IPI_RECEIVED",
+ "BriefDescription": "Received IPI from other HART event"
+ },
+ {
+ "PublicDescription": "Sent FENCE.I request to other HART event",
+ "ConfigCode": "0x8000000000000008",
+ "EventName": "FW_FENCE_I_SENT",
+ "BriefDescription": "Sent FENCE.I request to other HART event"
+ },
+ {
+ "PublicDescription": "Received FENCE.I request from other HART event",
+ "ConfigCode": "0x8000000000000009",
+ "EventName": "FW_FENCE_I_RECEIVED",
+ "BriefDescription": "Received FENCE.I request from other HART event"
+ },
+ {
+ "PublicDescription": "Sent SFENCE.VMA request to other HART event",
+ "ConfigCode": "0x800000000000000a",
+ "EventName": "FW_SFENCE_VMA_SENT",
+ "BriefDescription": "Sent SFENCE.VMA request to other HART event"
+ },
+ {
+ "PublicDescription": "Received SFENCE.VMA request from other HART event",
+ "ConfigCode": "0x800000000000000b",
+ "EventName": "FW_SFENCE_VMA_RECEIVED",
+ "BriefDescription": "Received SFENCE.VMA request from other HART event"
+ },
+ {
+ "PublicDescription": "Sent SFENCE.VMA with ASID request to other HART event",
+ "ConfigCode": "0x800000000000000c",
+ "EventName": "FW_SFENCE_VMA_RECEIVED",
+ "BriefDescription": "Sent SFENCE.VMA with ASID request to other HART event"
+ },
+ {
+ "PublicDescription": "Received SFENCE.VMA with ASID request from other HART event",
+ "ConfigCode": "0x800000000000000d",
+ "EventName": "FW_SFENCE_VMA_ASID_RECEIVED",
+ "BriefDescription": "Received SFENCE.VMA with ASID request from other HART event"
+ },
+ {
+ "PublicDescription": "Sent HFENCE.GVMA request to other HART event",
+ "ConfigCode": "0x800000000000000e",
+ "EventName": "FW_HFENCE_GVMA_SENT",
+ "BriefDescription": "Sent HFENCE.GVMA request to other HART event"
+ },
+ {
+ "PublicDescription": "Received HFENCE.GVMA request from other HART event",
+ "ConfigCode": "0x800000000000000f",
+ "EventName": "FW_HFENCE_GVMA_RECEIVED",
+ "BriefDescription": "Received HFENCE.GVMA request from other HART event"
+ },
+ {
+ "PublicDescription": "Sent HFENCE.GVMA with VMID request to other HART event",
+ "ConfigCode": "0x8000000000000010",
+ "EventName": "FW_HFENCE_GVMA_VMID_SENT",
+ "BriefDescription": "Sent HFENCE.GVMA with VMID request to other HART event"
+ },
+ {
+ "PublicDescription": "Received HFENCE.GVMA with VMID request from other HART event",
+ "ConfigCode": "0x8000000000000011",
+ "EventName": "FW_HFENCE_GVMA_VMID_RECEIVED",
+ "BriefDescription": "Received HFENCE.GVMA with VMID request from other HART event"
+ },
+ {
+ "PublicDescription": "Sent HFENCE.VVMA request to other HART event",
+ "ConfigCode": "0x8000000000000012",
+ "EventName": "FW_HFENCE_VVMA_SENT",
+ "BriefDescription": "Sent HFENCE.VVMA request to other HART event"
+ },
+ {
+ "PublicDescription": "Received HFENCE.VVMA request from other HART event",
+ "ConfigCode": "0x8000000000000013",
+ "EventName": "FW_HFENCE_VVMA_RECEIVED",
+ "BriefDescription": "Received HFENCE.VVMA request from other HART event"
+ },
+ {
+ "PublicDescription": "Sent HFENCE.VVMA with ASID request to other HART event",
+ "ConfigCode": "0x8000000000000014",
+ "EventName": "FW_HFENCE_VVMA_ASID_SENT",
+ "BriefDescription": "Sent HFENCE.VVMA with ASID request to other HART event"
+ },
+ {
+ "PublicDescription": "Received HFENCE.VVMA with ASID request from other HART event",
+ "ConfigCode": "0x8000000000000015",
+ "EventName": "FW_HFENCE_VVMA_ASID_RECEIVED",
+ "BriefDescription": "Received HFENCE.VVMA with ASID request from other HART event"
+ }
+]
diff --git a/tools/perf/pmu-events/arch/riscv/sifive/u74/firmware.json b/tools/perf/pmu-events/arch/riscv/sifive/u74/firmware.json
new file mode 100644
index 0000000..9b4a032
--- /dev/null
+++ b/tools/perf/pmu-events/arch/riscv/sifive/u74/firmware.json
@@ -0,0 +1,68 @@
+[
+ {
+ "ArchStdEvent": "FW_MISALIGNED_LOAD"
+ },
+ {
+ "ArchStdEvent": "FW_MISALIGNED_STORE"
+ },
+ {
+ "ArchStdEvent": "FW_ACCESS_LOAD"
+ },
+ {
+ "ArchStdEvent": "FW_ACCESS_STORE"
+ },
+ {
+ "ArchStdEvent": "FW_ILLEGAL_INSN"
+ },
+ {
+ "ArchStdEvent": "FW_SET_TIMER"
+ },
+ {
+ "ArchStdEvent": "FW_IPI_SENT"
+ },
+ {
+ "ArchStdEvent": "FW_IPI_RECEIVED"
+ },
+ {
+ "ArchStdEvent": "FW_FENCE_I_SENT"
+ },
+ {
+ "ArchStdEvent": "FW_FENCE_I_RECEIVED"
+ },
+ {
+ "ArchStdEvent": "FW_SFENCE_VMA_SENT"
+ },
+ {
+ "ArchStdEvent": "FW_SFENCE_VMA_RECEIVED"
+ },
+ {
+ "ArchStdEvent": "FW_SFENCE_VMA_RECEIVED"
+ },
+ {
+ "ArchStdEvent": "FW_SFENCE_VMA_ASID_RECEIVED"
+ },
+ {
+ "ArchStdEvent": "FW_HFENCE_GVMA_SENT"
+ },
+ {
+ "ArchStdEvent": "FW_HFENCE_GVMA_RECEIVED"
+ },
+ {
+ "ArchStdEvent": "FW_HFENCE_GVMA_VMID_SENT"
+ },
+ {
+ "ArchStdEvent": "FW_HFENCE_GVMA_VMID_RECEIVED"
+ },
+ {
+ "ArchStdEvent": "FW_HFENCE_VVMA_SENT"
+ },
+ {
+ "ArchStdEvent": "FW_HFENCE_VVMA_RECEIVED"
+ },
+ {
+ "ArchStdEvent": "FW_HFENCE_VVMA_ASID_SENT"
+ },
+ {
+ "ArchStdEvent": "FW_HFENCE_VVMA_ASID_RECEIVED"
+ }
+]
diff --git a/tools/perf/pmu-events/arch/riscv/sifive/u74/instructions.json b/tools/perf/pmu-events/arch/riscv/sifive/u74/instructions.json
new file mode 100644
index 0000000..5eab718
--- /dev/null
+++ b/tools/perf/pmu-events/arch/riscv/sifive/u74/instructions.json
@@ -0,0 +1,92 @@
+[
+ {
+ "EventName": "EXCEPTION_TAKEN",
+ "EventCode": "0x0000100",
+ "BriefDescription": "Exception taken"
+ },
+ {
+ "EventName": "INTEGER_LOAD_RETIRED",
+ "EventCode": "0x0000200",
+ "BriefDescription": "Integer load instruction retired"
+ },
+ {
+ "EventName": "INTEGER_STORE_RETIRED",
+ "EventCode": "0x0000400",
+ "BriefDescription": "Integer store instruction retired"
+ },
+ {
+ "EventName": "ATOMIC_MEMORY_RETIRED",
+ "EventCode": "0x0000800",
+ "BriefDescription": "Atomic memory operation retired"
+ },
+ {
+ "EventName": "SYSTEM_INSTRUCTION_RETIRED",
+ "EventCode": "0x0001000",
+ "BriefDescription": "System instruction retired"
+ },
+ {
+ "EventName": "INTEGER_ARITHMETIC_RETIRED",
+ "EventCode": "0x0002000",
+ "BriefDescription": "Integer arithmetic instruction retired"
+ },
+ {
+ "EventName": "CONDITIONAL_BRANCH_RETIRED",
+ "EventCode": "0x0004000",
+ "BriefDescription": "Conditional branch retired"
+ },
+ {
+ "EventName": "JAL_INSTRUCTION_RETIRED",
+ "EventCode": "0x0008000",
+ "BriefDescription": "JAL instruction retired"
+ },
+ {
+ "EventName": "JALR_INSTRUCTION_RETIRED",
+ "EventCode": "0x0010000",
+ "BriefDescription": "JALR instruction retired"
+ },
+ {
+ "EventName": "INTEGER_MULTIPLICATION_RETIRED",
+ "EventCode": "0x0020000",
+ "BriefDescription": "Integer multiplication instruction retired"
+ },
+ {
+ "EventName": "INTEGER_DIVISION_RETIRED",
+ "EventCode": "0x0040000",
+ "BriefDescription": "Integer division instruction retired"
+ },
+ {
+ "EventName": "FP_LOAD_RETIRED",
+ "EventCode": "0x0080000",
+ "BriefDescription": "Floating-point load instruction retired"
+ },
+ {
+ "EventName": "FP_STORE_RETIRED",
+ "EventCode": "0x0100000",
+ "BriefDescription": "Floating-point store instruction retired"
+ },
+ {
+ "EventName": "FP_ADDITION_RETIRED",
+ "EventCode": "0x0200000",
+ "BriefDescription": "Floating-point addition retired"
+ },
+ {
+ "EventName": "FP_MULTIPLICATION_RETIRED",
+ "EventCode": "0x0400000",
+ "BriefDescription": "Floating-point multiplication retired"
+ },
+ {
+ "EventName": "FP_FUSEDMADD_RETIRED",
+ "EventCode": "0x0800000",
+ "BriefDescription": "Floating-point fused multiply-add retired"
+ },
+ {
+ "EventName": "FP_DIV_SQRT_RETIRED",
+ "EventCode": "0x1000000",
+ "BriefDescription": "Floating-point division or square-root retired"
+ },
+ {
+ "EventName": "OTHER_FP_RETIRED",
+ "EventCode": "0x2000000",
+ "BriefDescription": "Other floating-point instruction retired"
+ }
+]
\ No newline at end of file
diff --git a/tools/perf/pmu-events/arch/riscv/sifive/u74/memory.json b/tools/perf/pmu-events/arch/riscv/sifive/u74/memory.json
new file mode 100644
index 0000000..be1a463
--- /dev/null
+++ b/tools/perf/pmu-events/arch/riscv/sifive/u74/memory.json
@@ -0,0 +1,32 @@
+[
+ {
+ "EventName": "ICACHE_RETIRED",
+ "EventCode": "0x0000102",
+ "BriefDescription": "Instruction cache miss"
+ },
+ {
+ "EventName": "DCACHE_MISS_MMIO_ACCESSES",
+ "EventCode": "0x0000202",
+ "BriefDescription": "Data cache miss or memory-mapped I/O access"
+ },
+ {
+ "EventName": "DCACHE_WRITEBACK",
+ "EventCode": "0x0000402",
+ "BriefDescription": "Data cache write-back"
+ },
+ {
+ "EventName": "INST_TLB_MISS",
+ "EventCode": "0x0000802",
+ "BriefDescription": "Instruction TLB miss"
+ },
+ {
+ "EventName": "DATA_TLB_MISS",
+ "EventCode": "0x0001002",
+ "BriefDescription": "Data TLB miss"
+ },
+ {
+ "EventName": "UTLB_MISS",
+ "EventCode": "0x0002002",
+ "BriefDescription": "UTLB miss"
+ }
+]
\ No newline at end of file
diff --git a/tools/perf/pmu-events/arch/riscv/sifive/u74/microarch.json b/tools/perf/pmu-events/arch/riscv/sifive/u74/microarch.json
new file mode 100644
index 0000000..50ffa55
--- /dev/null
+++ b/tools/perf/pmu-events/arch/riscv/sifive/u74/microarch.json
@@ -0,0 +1,57 @@
+[
+ {
+ "EventName": "ADDRESSGEN_INTERLOCK",
+ "EventCode": "0x0000101",
+ "BriefDescription": "Address-generation interlock"
+ },
+ {
+ "EventName": "LONGLAT_INTERLOCK",
+ "EventCode": "0x0000201",
+ "BriefDescription": "Long-latency interlock"
+ },
+ {
+ "EventName": "CSR_READ_INTERLOCK",
+ "EventCode": "0x0000401",
+ "BriefDescription": "CSR read interlock"
+ },
+ {
+ "EventName": "ICACHE_ITIM_BUSY",
+ "EventCode": "0x0000801",
+ "BriefDescription": "Instruction cache/ITIM busy"
+ },
+ {
+ "EventName": "DCACHE_DTIM_BUSY",
+ "EventCode": "0x0001001",
+ "BriefDescription": "Data cache/DTIM busy"
+ },
+ {
+ "EventName": "BRANCH_DIRECTION_MISPREDICTION",
+ "EventCode": "0x0002001",
+ "BriefDescription": "Branch direction misprediction"
+ },
+ {
+ "EventName": "BRANCH_TARGET_MISPREDICTION",
+ "EventCode": "0x0004001",
+ "BriefDescription": "Branch/jump target misprediction"
+ },
+ {
+ "EventName": "PIPE_FLUSH_CSR_WRITE",
+ "EventCode": "0x0008001",
+ "BriefDescription": "Pipeline flush from CSR write"
+ },
+ {
+ "EventName": "PIPE_FLUSH_OTHER_EVENT",
+ "EventCode": "0x0010001",
+ "BriefDescription": "Pipeline flush from other event"
+ },
+ {
+ "EventName": "INTEGER_MULTIPLICATION_INTERLOCK",
+ "EventCode": "0x0020001",
+ "BriefDescription": "Integer multiplication interlock"
+ },
+ {
+ "EventName": "FP_INTERLOCK",
+ "EventCode": "0x0040001",
+ "BriefDescription": "Floating-point interlock"
+ }
+]
\ No newline at end of file
diff --git a/tools/perf/pmu-events/arch/x86/alderlake/adl-metrics.json b/tools/perf/pmu-events/arch/x86/alderlake/adl-metrics.json
index e06d26a..edf440e 100644
--- a/tools/perf/pmu-events/arch/x86/alderlake/adl-metrics.json
+++ b/tools/perf/pmu-events/arch/x86/alderlake/adl-metrics.json
@@ -1287,14 +1287,14 @@
},
{
"BriefDescription": "Average CPU Utilization",
- "MetricExpr": "CPU_CLK_UNHALTED.REF_TSC / msr@tsc@",
+ "MetricExpr": "CPU_CLK_UNHALTED.REF_TSC / TSC",
"MetricGroup": "HPC;Summary",
"MetricName": "CPU_Utilization",
"Unit": "cpu_core"
},
{
"BriefDescription": "Measured Average Frequency for unhalted processors [GHz]",
- "MetricExpr": "Turbo_Utilization * msr@tsc@ / 1000000000 / duration_time",
+ "MetricExpr": "Turbo_Utilization * TSC / 1000000000 / duration_time",
"MetricGroup": "Power;Summary",
"MetricName": "Average_Frequency",
"Unit": "cpu_core"
@@ -1337,19 +1337,26 @@
},
{
"BriefDescription": "Average external Memory Bandwidth Use for reads and writes [GB / sec]",
- "MetricExpr": "64 * (arb@event\\=0x81\\,umask\\=0x1@ + arb@event\\=0x84\\,umask\\=0x1@) / 1000000 / duration_time / 1000",
+ "MetricExpr": "64 * (UNC_ARB_TRK_REQUESTS.ALL + UNC_ARB_COH_TRK_REQUESTS.ALL) / 1000000 / duration_time / 1000",
"MetricGroup": "HPC;Mem;MemoryBW;SoC",
"MetricName": "DRAM_BW_Use",
"Unit": "cpu_core"
},
{
"BriefDescription": "Average number of parallel requests to external memory. Accounts for all requests",
- "MetricExpr": "UNC_ARB_TRK_OCCUPANCY.ALL / arb@event\\=0x81\\,umask\\=0x1@",
+ "MetricExpr": "UNC_ARB_TRK_OCCUPANCY.ALL / UNC_ARB_TRK_REQUESTS.ALL",
"MetricGroup": "Mem;SoC",
"MetricName": "MEM_Parallel_Requests",
"Unit": "cpu_core"
},
{
+ "BriefDescription": "Socket actual clocks when any core is active on that socket",
+ "MetricExpr": "UNC_CLOCK.SOCKET",
+ "MetricGroup": "SoC",
+ "MetricName": "Socket_CLKS",
+ "Unit": "cpu_core"
+ },
+ {
"BriefDescription": "Instructions per Far Branch ( Far Branches apply upon transition from application to operating system, handling interrupts, exceptions) [lower number means higher occurrence rate]",
"MetricExpr": "INST_RETIRED.ANY / BR_INST_RETIRED.FAR_BRANCH:u",
"MetricGroup": "Branches;OS",
@@ -1357,6 +1364,12 @@
"Unit": "cpu_core"
},
{
+ "BriefDescription": "Uncore frequency per die [GHZ]",
+ "MetricExpr": "Socket_CLKS / #num_dies / duration_time / 1000000000",
+ "MetricGroup": "SoC",
+ "MetricName": "UNCORE_FREQ"
+ },
+ {
"BriefDescription": "Counts the number of issue slots that were not consumed by the backend due to frontend stalls.",
"MetricExpr": "TOPDOWN_FE_BOUND.ALL / SLOTS",
"MetricGroup": "TopdownL1",
@@ -1902,7 +1915,7 @@
},
{
"BriefDescription": "Average CPU Utilization",
- "MetricExpr": "CPU_CLK_UNHALTED.REF_TSC / msr@tsc@",
+ "MetricExpr": "CPU_CLK_UNHALTED.REF_TSC / TSC",
"MetricName": "CPU_Utilization",
"Unit": "cpu_atom"
},
@@ -1950,62 +1963,72 @@
},
{
"BriefDescription": "C1 residency percent per core",
- "MetricExpr": "(cstate_core@c1\\-residency@ / msr@tsc@) * 100",
+ "MetricExpr": "cstate_core@c1\\-residency@ / TSC",
"MetricGroup": "Power",
- "MetricName": "C1_Core_Residency"
+ "MetricName": "C1_Core_Residency",
+ "ScaleUnit": "100%"
},
{
"BriefDescription": "C6 residency percent per core",
- "MetricExpr": "(cstate_core@c6\\-residency@ / msr@tsc@) * 100",
+ "MetricExpr": "cstate_core@c6\\-residency@ / TSC",
"MetricGroup": "Power",
- "MetricName": "C6_Core_Residency"
+ "MetricName": "C6_Core_Residency",
+ "ScaleUnit": "100%"
},
{
"BriefDescription": "C7 residency percent per core",
- "MetricExpr": "(cstate_core@c7\\-residency@ / msr@tsc@) * 100",
+ "MetricExpr": "cstate_core@c7\\-residency@ / TSC",
"MetricGroup": "Power",
- "MetricName": "C7_Core_Residency"
+ "MetricName": "C7_Core_Residency",
+ "ScaleUnit": "100%"
},
{
"BriefDescription": "C2 residency percent per package",
- "MetricExpr": "(cstate_pkg@c2\\-residency@ / msr@tsc@) * 100",
+ "MetricExpr": "cstate_pkg@c2\\-residency@ / TSC",
"MetricGroup": "Power",
- "MetricName": "C2_Pkg_Residency"
+ "MetricName": "C2_Pkg_Residency",
+ "ScaleUnit": "100%"
},
{
"BriefDescription": "C3 residency percent per package",
- "MetricExpr": "(cstate_pkg@c3\\-residency@ / msr@tsc@) * 100",
+ "MetricExpr": "cstate_pkg@c3\\-residency@ / TSC",
"MetricGroup": "Power",
- "MetricName": "C3_Pkg_Residency"
+ "MetricName": "C3_Pkg_Residency",
+ "ScaleUnit": "100%"
},
{
"BriefDescription": "C6 residency percent per package",
- "MetricExpr": "(cstate_pkg@c6\\-residency@ / msr@tsc@) * 100",
+ "MetricExpr": "cstate_pkg@c6\\-residency@ / TSC",
"MetricGroup": "Power",
- "MetricName": "C6_Pkg_Residency"
+ "MetricName": "C6_Pkg_Residency",
+ "ScaleUnit": "100%"
},
{
"BriefDescription": "C7 residency percent per package",
- "MetricExpr": "(cstate_pkg@c7\\-residency@ / msr@tsc@) * 100",
+ "MetricExpr": "cstate_pkg@c7\\-residency@ / TSC",
"MetricGroup": "Power",
- "MetricName": "C7_Pkg_Residency"
+ "MetricName": "C7_Pkg_Residency",
+ "ScaleUnit": "100%"
},
{
"BriefDescription": "C8 residency percent per package",
- "MetricExpr": "(cstate_pkg@c8\\-residency@ / msr@tsc@) * 100",
+ "MetricExpr": "cstate_pkg@c8\\-residency@ / TSC",
"MetricGroup": "Power",
- "MetricName": "C8_Pkg_Residency"
+ "MetricName": "C8_Pkg_Residency",
+ "ScaleUnit": "100%"
},
{
"BriefDescription": "C9 residency percent per package",
- "MetricExpr": "(cstate_pkg@c9\\-residency@ / msr@tsc@) * 100",
+ "MetricExpr": "cstate_pkg@c9\\-residency@ / TSC",
"MetricGroup": "Power",
- "MetricName": "C9_Pkg_Residency"
+ "MetricName": "C9_Pkg_Residency",
+ "ScaleUnit": "100%"
},
{
"BriefDescription": "C10 residency percent per package",
- "MetricExpr": "(cstate_pkg@c10\\-residency@ / msr@tsc@) * 100",
+ "MetricExpr": "cstate_pkg@c10\\-residency@ / TSC",
"MetricGroup": "Power",
- "MetricName": "C10_Pkg_Residency"
+ "MetricName": "C10_Pkg_Residency",
+ "ScaleUnit": "100%"
}
]
diff --git a/tools/perf/pmu-events/arch/x86/alderlake/cache.json b/tools/perf/pmu-events/arch/x86/alderlake/cache.json
index 2cc62d2..adc9887 100644
--- a/tools/perf/pmu-events/arch/x86/alderlake/cache.json
+++ b/tools/perf/pmu-events/arch/x86/alderlake/cache.json
@@ -1,403 +1,851 @@
[
{
+ "BriefDescription": "L1D.HWPF_MISS",
+ "EventCode": "0x51",
+ "EventName": "L1D.HWPF_MISS",
+ "SampleAfterValue": "1000003",
+ "UMask": "0x20",
+ "Unit": "cpu_core"
+ },
+ {
+ "BriefDescription": "Counts the number of cache lines replaced in L1 data cache.",
+ "EventCode": "0x51",
+ "EventName": "L1D.REPLACEMENT",
+ "PublicDescription": "Counts L1D data line replacements including opportunistic replacements, and replacements that require stall-for-replace or block-for-replace.",
+ "SampleAfterValue": "100003",
+ "UMask": "0x1",
+ "Unit": "cpu_core"
+ },
+ {
+ "BriefDescription": "Number of cycles a demand request has waited due to L1D Fill Buffer (FB) unavailability.",
+ "EventCode": "0x48",
+ "EventName": "L1D_PEND_MISS.FB_FULL",
+ "PublicDescription": "Counts number of cycles a demand request has waited due to L1D Fill Buffer (FB) unavailability. Demand requests include cacheable/uncacheable demand load, store, lock or SW prefetch accesses.",
+ "SampleAfterValue": "1000003",
+ "UMask": "0x2",
+ "Unit": "cpu_core"
+ },
+ {
+ "BriefDescription": "Number of phases a demand request has waited due to L1D Fill Buffer (FB) unavailability.",
+ "CounterMask": "1",
+ "EdgeDetect": "1",
+ "EventCode": "0x48",
+ "EventName": "L1D_PEND_MISS.FB_FULL_PERIODS",
+ "PublicDescription": "Counts number of phases a demand request has waited due to L1D Fill Buffer (FB) unavailability. Demand requests include cacheable/uncacheable demand load, store, lock or SW prefetch accesses.",
+ "SampleAfterValue": "1000003",
+ "UMask": "0x2",
+ "Unit": "cpu_core"
+ },
+ {
+ "BriefDescription": "This event is deprecated. Refer to new event L1D_PEND_MISS.L2_STALLS",
+ "Deprecated": "1",
+ "EventCode": "0x48",
+ "EventName": "L1D_PEND_MISS.L2_STALL",
+ "SampleAfterValue": "1000003",
+ "UMask": "0x4",
+ "Unit": "cpu_core"
+ },
+ {
+ "BriefDescription": "Number of cycles a demand request has waited due to L1D due to lack of L2 resources.",
+ "EventCode": "0x48",
+ "EventName": "L1D_PEND_MISS.L2_STALLS",
+ "PublicDescription": "Counts number of cycles a demand request has waited due to L1D due to lack of L2 resources. Demand requests include cacheable/uncacheable demand load, store, lock or SW prefetch accesses.",
+ "SampleAfterValue": "1000003",
+ "UMask": "0x4",
+ "Unit": "cpu_core"
+ },
+ {
+ "BriefDescription": "Number of L1D misses that are outstanding",
+ "EventCode": "0x48",
+ "EventName": "L1D_PEND_MISS.PENDING",
+ "PublicDescription": "Counts number of L1D misses that are outstanding in each cycle, that is each cycle the number of Fill Buffers (FB) outstanding required by Demand Reads. FB either is held by demand loads, or it is held by non-demand loads and gets hit at least once by demand. The valid outstanding interval is defined until the FB deallocation by one of the following ways: from FB allocation, if FB is allocated by demand from the demand Hit FB, if it is allocated by hardware or software prefetch. Note: In the L1D, a Demand Read contains cacheable or noncacheable demand loads, including ones causing cache-line splits and reads due to page walks resulted from any request type.",
+ "SampleAfterValue": "1000003",
+ "UMask": "0x1",
+ "Unit": "cpu_core"
+ },
+ {
+ "BriefDescription": "Cycles with L1D load Misses outstanding.",
+ "CounterMask": "1",
+ "EventCode": "0x48",
+ "EventName": "L1D_PEND_MISS.PENDING_CYCLES",
+ "PublicDescription": "Counts duration of L1D miss outstanding in cycles.",
+ "SampleAfterValue": "1000003",
+ "UMask": "0x1",
+ "Unit": "cpu_core"
+ },
+ {
+ "BriefDescription": "L2 cache lines filling L2",
+ "EventCode": "0x25",
+ "EventName": "L2_LINES_IN.ALL",
+ "PublicDescription": "Counts the number of L2 cache lines filling the L2. Counting does not cover rejects.",
+ "SampleAfterValue": "100003",
+ "UMask": "0x1f",
+ "Unit": "cpu_core"
+ },
+ {
+ "BriefDescription": "Cache lines that have been L2 hardware prefetched but not used by demand accesses",
+ "EventCode": "0x26",
+ "EventName": "L2_LINES_OUT.USELESS_HWPF",
+ "PublicDescription": "Counts the number of cache lines that have been prefetched by the L2 hardware prefetcher but not used by demand access when evicted from the L2 cache",
+ "SampleAfterValue": "200003",
+ "UMask": "0x4",
+ "Unit": "cpu_core"
+ },
+ {
+ "BriefDescription": "All accesses to L2 cache[This event is alias to L2_RQSTS.REFERENCES]",
+ "EventCode": "0x24",
+ "EventName": "L2_REQUEST.ALL",
+ "PublicDescription": "Counts all requests that were hit or true misses in L2 cache. True-miss excludes misses that were merged with ongoing L2 misses.[This event is alias to L2_RQSTS.REFERENCES]",
+ "SampleAfterValue": "200003",
+ "UMask": "0xff",
+ "Unit": "cpu_core"
+ },
+ {
+ "BriefDescription": "Read requests with true-miss in L2 cache.[This event is alias to L2_RQSTS.MISS]",
+ "EventCode": "0x24",
+ "EventName": "L2_REQUEST.MISS",
+ "PublicDescription": "Counts read requests of any type with true-miss in the L2 cache. True-miss excludes L2 misses that were merged with ongoing L2 misses.[This event is alias to L2_RQSTS.MISS]",
+ "SampleAfterValue": "200003",
+ "UMask": "0x3f",
+ "Unit": "cpu_core"
+ },
+ {
+ "BriefDescription": "L2 code requests",
+ "EventCode": "0x24",
+ "EventName": "L2_RQSTS.ALL_CODE_RD",
+ "PublicDescription": "Counts the total number of L2 code requests.",
+ "SampleAfterValue": "200003",
+ "UMask": "0xe4",
+ "Unit": "cpu_core"
+ },
+ {
+ "BriefDescription": "Demand Data Read access L2 cache",
+ "EventCode": "0x24",
+ "EventName": "L2_RQSTS.ALL_DEMAND_DATA_RD",
+ "PublicDescription": "Counts Demand Data Read requests accessing the L2 cache. These requests may hit or miss L2 cache. True-miss exclude misses that were merged with ongoing L2 misses. An access is counted once.",
+ "SampleAfterValue": "200003",
+ "UMask": "0xe1",
+ "Unit": "cpu_core"
+ },
+ {
+ "BriefDescription": "Demand requests that miss L2 cache",
+ "EventCode": "0x24",
+ "EventName": "L2_RQSTS.ALL_DEMAND_MISS",
+ "PublicDescription": "Counts demand requests that miss L2 cache.",
+ "SampleAfterValue": "200003",
+ "UMask": "0x27",
+ "Unit": "cpu_core"
+ },
+ {
+ "BriefDescription": "L2_RQSTS.ALL_HWPF",
+ "EventCode": "0x24",
+ "EventName": "L2_RQSTS.ALL_HWPF",
+ "SampleAfterValue": "200003",
+ "UMask": "0xf0",
+ "Unit": "cpu_core"
+ },
+ {
+ "BriefDescription": "RFO requests to L2 cache.",
+ "EventCode": "0x24",
+ "EventName": "L2_RQSTS.ALL_RFO",
+ "PublicDescription": "Counts the total number of RFO (read for ownership) requests to L2 cache. L2 RFO requests include both L1D demand RFO misses as well as L1D RFO prefetches.",
+ "SampleAfterValue": "200003",
+ "UMask": "0xe2",
+ "Unit": "cpu_core"
+ },
+ {
+ "BriefDescription": "L2 cache hits when fetching instructions, code reads.",
+ "EventCode": "0x24",
+ "EventName": "L2_RQSTS.CODE_RD_HIT",
+ "PublicDescription": "Counts L2 cache hits when fetching instructions, code reads.",
+ "SampleAfterValue": "200003",
+ "UMask": "0xc4",
+ "Unit": "cpu_core"
+ },
+ {
+ "BriefDescription": "L2 cache misses when fetching instructions",
+ "EventCode": "0x24",
+ "EventName": "L2_RQSTS.CODE_RD_MISS",
+ "PublicDescription": "Counts L2 cache misses when fetching instructions.",
+ "SampleAfterValue": "200003",
+ "UMask": "0x24",
+ "Unit": "cpu_core"
+ },
+ {
+ "BriefDescription": "Demand Data Read requests that hit L2 cache",
+ "EventCode": "0x24",
+ "EventName": "L2_RQSTS.DEMAND_DATA_RD_HIT",
+ "PublicDescription": "Counts the number of demand Data Read requests initiated by load instructions that hit L2 cache.",
+ "SampleAfterValue": "200003",
+ "UMask": "0xc1",
+ "Unit": "cpu_core"
+ },
+ {
+ "BriefDescription": "Demand Data Read miss L2 cache",
+ "EventCode": "0x24",
+ "EventName": "L2_RQSTS.DEMAND_DATA_RD_MISS",
+ "PublicDescription": "Counts demand Data Read requests with true-miss in the L2 cache. True-miss excludes misses that were merged with ongoing L2 misses. An access is counted once.",
+ "SampleAfterValue": "200003",
+ "UMask": "0x21",
+ "Unit": "cpu_core"
+ },
+ {
+ "BriefDescription": "L2_RQSTS.HWPF_MISS",
+ "EventCode": "0x24",
+ "EventName": "L2_RQSTS.HWPF_MISS",
+ "SampleAfterValue": "200003",
+ "UMask": "0x30",
+ "Unit": "cpu_core"
+ },
+ {
+ "BriefDescription": "Read requests with true-miss in L2 cache.[This event is alias to L2_REQUEST.MISS]",
+ "EventCode": "0x24",
+ "EventName": "L2_RQSTS.MISS",
+ "PublicDescription": "Counts read requests of any type with true-miss in the L2 cache. True-miss excludes L2 misses that were merged with ongoing L2 misses.[This event is alias to L2_REQUEST.MISS]",
+ "SampleAfterValue": "200003",
+ "UMask": "0x3f",
+ "Unit": "cpu_core"
+ },
+ {
+ "BriefDescription": "All accesses to L2 cache[This event is alias to L2_REQUEST.ALL]",
+ "EventCode": "0x24",
+ "EventName": "L2_RQSTS.REFERENCES",
+ "PublicDescription": "Counts all requests that were hit or true misses in L2 cache. True-miss excludes misses that were merged with ongoing L2 misses.[This event is alias to L2_REQUEST.ALL]",
+ "SampleAfterValue": "200003",
+ "UMask": "0xff",
+ "Unit": "cpu_core"
+ },
+ {
+ "BriefDescription": "RFO requests that hit L2 cache.",
+ "EventCode": "0x24",
+ "EventName": "L2_RQSTS.RFO_HIT",
+ "PublicDescription": "Counts the RFO (Read-for-Ownership) requests that hit L2 cache.",
+ "SampleAfterValue": "200003",
+ "UMask": "0xc2",
+ "Unit": "cpu_core"
+ },
+ {
+ "BriefDescription": "RFO requests that miss L2 cache",
+ "EventCode": "0x24",
+ "EventName": "L2_RQSTS.RFO_MISS",
+ "PublicDescription": "Counts the RFO (Read-for-Ownership) requests that miss L2 cache.",
+ "SampleAfterValue": "200003",
+ "UMask": "0x22",
+ "Unit": "cpu_core"
+ },
+ {
+ "BriefDescription": "SW prefetch requests that hit L2 cache.",
+ "EventCode": "0x24",
+ "EventName": "L2_RQSTS.SWPF_HIT",
+ "PublicDescription": "Counts Software prefetch requests that hit the L2 cache. Accounts for PREFETCHNTA and PREFETCHT0/1/2 instructions when FB is not full.",
+ "SampleAfterValue": "200003",
+ "UMask": "0xc8",
+ "Unit": "cpu_core"
+ },
+ {
+ "BriefDescription": "SW prefetch requests that miss L2 cache.",
+ "EventCode": "0x24",
+ "EventName": "L2_RQSTS.SWPF_MISS",
+ "PublicDescription": "Counts Software prefetch requests that miss the L2 cache. Accounts for PREFETCHNTA and PREFETCHT0/1/2 instructions when FB is not full.",
+ "SampleAfterValue": "200003",
+ "UMask": "0x28",
+ "Unit": "cpu_core"
+ },
+ {
"BriefDescription": "Counts the number of cacheable memory requests that miss in the LLC. Counts on a per core basis.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5",
"EventCode": "0x2e",
"EventName": "LONGEST_LAT_CACHE.MISS",
- "PEBScounters": "0,1,2,3,4,5",
+ "PublicDescription": "Counts the number of cacheable memory requests that miss in the Last Level Cache (LLC). Requests include demand loads, reads for ownership (RFO), instruction fetches and L1 HW prefetches. If the platform has an L3 cache, the LLC is the L3 cache, otherwise it is the L2 cache. Counts on a per core basis.",
"SampleAfterValue": "200003",
- "Speculative": "1",
"UMask": "0x41",
"Unit": "cpu_atom"
},
{
+ "BriefDescription": "Core-originated cacheable requests that missed L3 (Except hardware prefetches to the L3)",
+ "EventCode": "0x2e",
+ "EventName": "LONGEST_LAT_CACHE.MISS",
+ "PublicDescription": "Counts core-originated cacheable requests that miss the L3 cache (Longest Latency cache). Requests include data and code reads, Reads-for-Ownership (RFOs), speculative accesses and hardware prefetches to the L1 and L2. It does not include hardware prefetches to the L3, and may not count other types of requests to the L3.",
+ "SampleAfterValue": "100003",
+ "UMask": "0x41",
+ "Unit": "cpu_core"
+ },
+ {
"BriefDescription": "Counts the number of cacheable memory requests that access the LLC. Counts on a per core basis.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5",
"EventCode": "0x2e",
"EventName": "LONGEST_LAT_CACHE.REFERENCE",
- "PEBScounters": "0,1,2,3,4,5",
+ "PublicDescription": "Counts the number of cacheable memory requests that access the Last Level Cache (LLC). Requests include demand loads, reads for ownership (RFO), instruction fetches and L1 HW prefetches. If the platform has an L3 cache, the LLC is the L3 cache, otherwise it is the L2 cache. Counts on a per core basis.",
"SampleAfterValue": "200003",
- "Speculative": "1",
"UMask": "0x4f",
"Unit": "cpu_atom"
},
{
+ "BriefDescription": "Core-originated cacheable requests that refer to L3 (Except hardware prefetches to the L3)",
+ "EventCode": "0x2e",
+ "EventName": "LONGEST_LAT_CACHE.REFERENCE",
+ "PublicDescription": "Counts core-originated cacheable requests to the L3 cache (Longest Latency cache). Requests include data and code reads, Reads-for-Ownership (RFOs), speculative accesses and hardware prefetches to the L1 and L2. It does not include hardware prefetches to the L3, and may not count other types of requests to the L3.",
+ "SampleAfterValue": "100003",
+ "UMask": "0x4f",
+ "Unit": "cpu_core"
+ },
+ {
"BriefDescription": "Counts the number of cycles the core is stalled due to an instruction cache or TLB miss which hit in the L2, LLC, DRAM or MMIO (Non-DRAM).",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5",
"EventCode": "0x34",
"EventName": "MEM_BOUND_STALLS.IFETCH",
- "PEBScounters": "0,1,2,3,4,5",
+ "PublicDescription": "Counts the number of cycles the core is stalled due to an instruction cache or translation lookaside buffer (TLB) miss which hit in the L2, LLC, DRAM or MMIO (Non-DRAM).",
"SampleAfterValue": "200003",
- "Speculative": "1",
"UMask": "0x38",
"Unit": "cpu_atom"
},
{
"BriefDescription": "Counts the number of cycles the core is stalled due to an instruction cache or TLB miss which hit in DRAM or MMIO (Non-DRAM).",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5",
"EventCode": "0x34",
"EventName": "MEM_BOUND_STALLS.IFETCH_DRAM_HIT",
- "PEBScounters": "0,1,2,3,4,5",
+ "PublicDescription": "Counts the number of cycles the core is stalled due to an instruction cache or translation lookaside buffer (TLB) miss which hit in DRAM or MMIO (non-DRAM).",
"SampleAfterValue": "200003",
- "Speculative": "1",
"UMask": "0x20",
"Unit": "cpu_atom"
},
{
"BriefDescription": "Counts the number of cycles the core is stalled due to an instruction cache or TLB miss which hit in the L2 cache.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5",
"EventCode": "0x34",
"EventName": "MEM_BOUND_STALLS.IFETCH_L2_HIT",
- "PEBScounters": "0,1,2,3,4,5",
+ "PublicDescription": "Counts the number of cycles the core is stalled due to an instruction cache or Translation Lookaside Buffer (TLB) miss which hit in the L2 cache.",
"SampleAfterValue": "200003",
- "Speculative": "1",
"UMask": "0x8",
"Unit": "cpu_atom"
},
{
"BriefDescription": "Counts the number of cycles the core is stalled due to an instruction cache or TLB miss which hit in the LLC or other core with HITE/F/M.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5",
"EventCode": "0x34",
"EventName": "MEM_BOUND_STALLS.IFETCH_LLC_HIT",
- "PEBScounters": "0,1,2,3,4,5",
+ "PublicDescription": "Counts the number of cycles the core is stalled due to an instruction cache or Translation Lookaside Buffer (TLB) miss which hit in the Last Level Cache (LLC) or other core with HITE/F/M.",
"SampleAfterValue": "200003",
- "Speculative": "1",
"UMask": "0x10",
"Unit": "cpu_atom"
},
{
"BriefDescription": "Counts the number of cycles the core is stalled due to a demand load miss which hit in the L2, LLC, DRAM or MMIO (Non-DRAM).",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5",
"EventCode": "0x34",
"EventName": "MEM_BOUND_STALLS.LOAD",
- "PEBScounters": "0,1,2,3,4,5",
"SampleAfterValue": "200003",
- "Speculative": "1",
"UMask": "0x7",
"Unit": "cpu_atom"
},
{
"BriefDescription": "Counts the number of cycles the core is stalled due to a demand load miss which hit in DRAM or MMIO (Non-DRAM).",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5",
"EventCode": "0x34",
"EventName": "MEM_BOUND_STALLS.LOAD_DRAM_HIT",
- "PEBScounters": "0,1,2,3,4,5",
"SampleAfterValue": "200003",
- "Speculative": "1",
"UMask": "0x4",
"Unit": "cpu_atom"
},
{
"BriefDescription": "Counts the number of cycles the core is stalled due to a demand load which hit in the L2 cache.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5",
"EventCode": "0x34",
"EventName": "MEM_BOUND_STALLS.LOAD_L2_HIT",
- "PEBScounters": "0,1,2,3,4,5",
"SampleAfterValue": "200003",
- "Speculative": "1",
"UMask": "0x1",
"Unit": "cpu_atom"
},
{
"BriefDescription": "Counts the number of cycles the core is stalled due to a demand load which hit in the LLC or other core with HITE/F/M.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5",
"EventCode": "0x34",
"EventName": "MEM_BOUND_STALLS.LOAD_LLC_HIT",
- "PEBScounters": "0,1,2,3,4,5",
+ "PublicDescription": "Counts the number of cycles the core is stalled due to a demand load which hit in the Last Level Cache (LLC) or other core with HITE/F/M.",
"SampleAfterValue": "200003",
- "Speculative": "1",
"UMask": "0x2",
"Unit": "cpu_atom"
},
{
+ "BriefDescription": "Retired load instructions.",
+ "Data_LA": "1",
+ "EventCode": "0xd0",
+ "EventName": "MEM_INST_RETIRED.ALL_LOADS",
+ "PEBS": "1",
+ "PublicDescription": "Counts all retired load instructions. This event accounts for SW prefetch instructions of PREFETCHNTA or PREFETCHT0/1/2 or PREFETCHW.",
+ "SampleAfterValue": "1000003",
+ "UMask": "0x81",
+ "Unit": "cpu_core"
+ },
+ {
+ "BriefDescription": "Retired store instructions.",
+ "Data_LA": "1",
+ "EventCode": "0xd0",
+ "EventName": "MEM_INST_RETIRED.ALL_STORES",
+ "PEBS": "1",
+ "PublicDescription": "Counts all retired store instructions.",
+ "SampleAfterValue": "1000003",
+ "UMask": "0x82",
+ "Unit": "cpu_core"
+ },
+ {
+ "BriefDescription": "All retired memory instructions.",
+ "Data_LA": "1",
+ "EventCode": "0xd0",
+ "EventName": "MEM_INST_RETIRED.ANY",
+ "PEBS": "1",
+ "PublicDescription": "Counts all retired memory instructions - loads and stores.",
+ "SampleAfterValue": "1000003",
+ "UMask": "0x83",
+ "Unit": "cpu_core"
+ },
+ {
+ "BriefDescription": "Retired load instructions with locked access.",
+ "Data_LA": "1",
+ "EventCode": "0xd0",
+ "EventName": "MEM_INST_RETIRED.LOCK_LOADS",
+ "PEBS": "1",
+ "PublicDescription": "Counts retired load instructions with locked access.",
+ "SampleAfterValue": "100007",
+ "UMask": "0x21",
+ "Unit": "cpu_core"
+ },
+ {
+ "BriefDescription": "Retired load instructions that split across a cacheline boundary.",
+ "Data_LA": "1",
+ "EventCode": "0xd0",
+ "EventName": "MEM_INST_RETIRED.SPLIT_LOADS",
+ "PEBS": "1",
+ "PublicDescription": "Counts retired load instructions that split across a cacheline boundary.",
+ "SampleAfterValue": "100003",
+ "UMask": "0x41",
+ "Unit": "cpu_core"
+ },
+ {
+ "BriefDescription": "Retired store instructions that split across a cacheline boundary.",
+ "Data_LA": "1",
+ "EventCode": "0xd0",
+ "EventName": "MEM_INST_RETIRED.SPLIT_STORES",
+ "PEBS": "1",
+ "PublicDescription": "Counts retired store instructions that split across a cacheline boundary.",
+ "SampleAfterValue": "100003",
+ "UMask": "0x42",
+ "Unit": "cpu_core"
+ },
+ {
+ "BriefDescription": "Retired load instructions that miss the STLB.",
+ "Data_LA": "1",
+ "EventCode": "0xd0",
+ "EventName": "MEM_INST_RETIRED.STLB_MISS_LOADS",
+ "PEBS": "1",
+ "PublicDescription": "Number of retired load instructions that (start a) miss in the 2nd-level TLB (STLB).",
+ "SampleAfterValue": "100003",
+ "UMask": "0x11",
+ "Unit": "cpu_core"
+ },
+ {
+ "BriefDescription": "Retired store instructions that miss the STLB.",
+ "Data_LA": "1",
+ "EventCode": "0xd0",
+ "EventName": "MEM_INST_RETIRED.STLB_MISS_STORES",
+ "PEBS": "1",
+ "PublicDescription": "Number of retired store instructions that (start a) miss in the 2nd-level TLB (STLB).",
+ "SampleAfterValue": "100003",
+ "UMask": "0x12",
+ "Unit": "cpu_core"
+ },
+ {
+ "BriefDescription": "Completed demand load uops that miss the L1 d-cache.",
+ "EventCode": "0x43",
+ "EventName": "MEM_LOAD_COMPLETED.L1_MISS_ANY",
+ "PublicDescription": "Number of completed demand load requests that missed the L1 data cache including shadow misses (FB hits, merge to an ongoing L1D miss)",
+ "SampleAfterValue": "1000003",
+ "UMask": "0xfd",
+ "Unit": "cpu_core"
+ },
+ {
+ "BriefDescription": "Retired load instructions whose data sources were HitM responses from shared L3",
+ "Data_LA": "1",
+ "EventCode": "0xd2",
+ "EventName": "MEM_LOAD_L3_HIT_RETIRED.XSNP_FWD",
+ "PEBS": "1",
+ "PublicDescription": "Counts retired load instructions whose data sources were HitM responses from shared L3.",
+ "SampleAfterValue": "20011",
+ "UMask": "0x4",
+ "Unit": "cpu_core"
+ },
+ {
+ "BriefDescription": "Retired load instructions whose data sources were L3 and cross-core snoop hits in on-pkg core cache",
+ "Data_LA": "1",
+ "EventCode": "0xd2",
+ "EventName": "MEM_LOAD_L3_HIT_RETIRED.XSNP_HIT",
+ "PEBS": "1",
+ "PublicDescription": "Counts retired load instructions whose data sources were L3 and cross-core snoop hits in on-pkg core cache.",
+ "SampleAfterValue": "20011",
+ "UMask": "0x2",
+ "Unit": "cpu_core"
+ },
+ {
+ "BriefDescription": "Retired load instructions whose data sources were HitM responses from shared L3",
+ "Data_LA": "1",
+ "EventCode": "0xd2",
+ "EventName": "MEM_LOAD_L3_HIT_RETIRED.XSNP_HITM",
+ "PEBS": "1",
+ "PublicDescription": "Counts retired load instructions whose data sources were HitM responses from shared L3.",
+ "SampleAfterValue": "20011",
+ "UMask": "0x4",
+ "Unit": "cpu_core"
+ },
+ {
+ "BriefDescription": "Retired load instructions whose data sources were L3 hit and cross-core snoop missed in on-pkg core cache.",
+ "Data_LA": "1",
+ "EventCode": "0xd2",
+ "EventName": "MEM_LOAD_L3_HIT_RETIRED.XSNP_MISS",
+ "PEBS": "1",
+ "PublicDescription": "Counts the retired load instructions whose data sources were L3 hit and cross-core snoop missed in on-pkg core cache.",
+ "SampleAfterValue": "20011",
+ "UMask": "0x1",
+ "Unit": "cpu_core"
+ },
+ {
+ "BriefDescription": "Retired load instructions whose data sources were hits in L3 without snoops required",
+ "Data_LA": "1",
+ "EventCode": "0xd2",
+ "EventName": "MEM_LOAD_L3_HIT_RETIRED.XSNP_NONE",
+ "PEBS": "1",
+ "PublicDescription": "Counts retired load instructions whose data sources were hits in L3 without snoops required.",
+ "SampleAfterValue": "100003",
+ "UMask": "0x8",
+ "Unit": "cpu_core"
+ },
+ {
+ "BriefDescription": "Retired load instructions whose data sources were L3 and cross-core snoop hits in on-pkg core cache",
+ "Data_LA": "1",
+ "EventCode": "0xd2",
+ "EventName": "MEM_LOAD_L3_HIT_RETIRED.XSNP_NO_FWD",
+ "PEBS": "1",
+ "PublicDescription": "Counts retired load instructions whose data sources were L3 and cross-core snoop hits in on-pkg core cache.",
+ "SampleAfterValue": "20011",
+ "UMask": "0x2",
+ "Unit": "cpu_core"
+ },
+ {
+ "BriefDescription": "Retired load instructions which data sources missed L3 but serviced from local dram",
+ "Data_LA": "1",
+ "EventCode": "0xd3",
+ "EventName": "MEM_LOAD_L3_MISS_RETIRED.LOCAL_DRAM",
+ "PEBS": "1",
+ "PublicDescription": "Retired load instructions which data sources missed L3 but serviced from local DRAM.",
+ "SampleAfterValue": "100007",
+ "UMask": "0x1",
+ "Unit": "cpu_core"
+ },
+ {
+ "BriefDescription": "Retired instructions with at least 1 uncacheable load or lock.",
+ "Data_LA": "1",
+ "EventCode": "0xd4",
+ "EventName": "MEM_LOAD_MISC_RETIRED.UC",
+ "PEBS": "1",
+ "PublicDescription": "Retired instructions with at least one load to uncacheable memory-type, or at least one cache-line split locked access (Bus Lock).",
+ "SampleAfterValue": "100007",
+ "UMask": "0x4",
+ "Unit": "cpu_core"
+ },
+ {
+ "BriefDescription": "Number of completed demand load requests that missed the L1, but hit the FB(fill buffer), because a preceding miss to the same cacheline initiated the line to be brought into L1, but data is not yet ready in L1.",
+ "Data_LA": "1",
+ "EventCode": "0xd1",
+ "EventName": "MEM_LOAD_RETIRED.FB_HIT",
+ "PEBS": "1",
+ "PublicDescription": "Counts retired load instructions with at least one uop was load missed in L1 but hit FB (Fill Buffers) due to preceding miss to the same cache line with data not ready.",
+ "SampleAfterValue": "100007",
+ "UMask": "0x40",
+ "Unit": "cpu_core"
+ },
+ {
+ "BriefDescription": "Retired load instructions with L1 cache hits as data sources",
+ "Data_LA": "1",
+ "EventCode": "0xd1",
+ "EventName": "MEM_LOAD_RETIRED.L1_HIT",
+ "PEBS": "1",
+ "PublicDescription": "Counts retired load instructions with at least one uop that hit in the L1 data cache. This event includes all SW prefetches and lock instructions regardless of the data source.",
+ "SampleAfterValue": "1000003",
+ "UMask": "0x1",
+ "Unit": "cpu_core"
+ },
+ {
+ "BriefDescription": "Retired load instructions missed L1 cache as data sources",
+ "Data_LA": "1",
+ "EventCode": "0xd1",
+ "EventName": "MEM_LOAD_RETIRED.L1_MISS",
+ "PEBS": "1",
+ "PublicDescription": "Counts retired load instructions with at least one uop that missed in the L1 cache.",
+ "SampleAfterValue": "200003",
+ "UMask": "0x8",
+ "Unit": "cpu_core"
+ },
+ {
+ "BriefDescription": "Retired load instructions with L2 cache hits as data sources",
+ "Data_LA": "1",
+ "EventCode": "0xd1",
+ "EventName": "MEM_LOAD_RETIRED.L2_HIT",
+ "PEBS": "1",
+ "PublicDescription": "Counts retired load instructions with L2 cache hits as data sources.",
+ "SampleAfterValue": "200003",
+ "UMask": "0x2",
+ "Unit": "cpu_core"
+ },
+ {
+ "BriefDescription": "Retired load instructions missed L2 cache as data sources",
+ "Data_LA": "1",
+ "EventCode": "0xd1",
+ "EventName": "MEM_LOAD_RETIRED.L2_MISS",
+ "PEBS": "1",
+ "PublicDescription": "Counts retired load instructions missed L2 cache as data sources.",
+ "SampleAfterValue": "100021",
+ "UMask": "0x10",
+ "Unit": "cpu_core"
+ },
+ {
+ "BriefDescription": "Retired load instructions with L3 cache hits as data sources",
+ "Data_LA": "1",
+ "EventCode": "0xd1",
+ "EventName": "MEM_LOAD_RETIRED.L3_HIT",
+ "PEBS": "1",
+ "PublicDescription": "Counts retired load instructions with at least one uop that hit in the L3 cache.",
+ "SampleAfterValue": "100021",
+ "UMask": "0x4",
+ "Unit": "cpu_core"
+ },
+ {
+ "BriefDescription": "Retired load instructions missed L3 cache as data sources",
+ "Data_LA": "1",
+ "EventCode": "0xd1",
+ "EventName": "MEM_LOAD_RETIRED.L3_MISS",
+ "PEBS": "1",
+ "PublicDescription": "Counts retired load instructions with at least one uop that missed in the L3 cache.",
+ "SampleAfterValue": "50021",
+ "UMask": "0x20",
+ "Unit": "cpu_core"
+ },
+ {
"BriefDescription": "Counts the number of load uops retired that hit in DRAM.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5",
"Data_LA": "1",
"EventCode": "0xd1",
"EventName": "MEM_LOAD_UOPS_RETIRED.DRAM_HIT",
"PEBS": "1",
- "PEBScounters": "0,1,2,3,4,5",
"SampleAfterValue": "200003",
"UMask": "0x80",
"Unit": "cpu_atom"
},
{
"BriefDescription": "Counts the number of load uops retired that hit in the L2 cache.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5",
"Data_LA": "1",
"EventCode": "0xd1",
"EventName": "MEM_LOAD_UOPS_RETIRED.L2_HIT",
"PEBS": "1",
- "PEBScounters": "0,1,2,3,4,5",
"SampleAfterValue": "200003",
"UMask": "0x2",
"Unit": "cpu_atom"
},
{
"BriefDescription": "Counts the number of load uops retired that hit in the L3 cache.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5",
"Data_LA": "1",
"EventCode": "0xd1",
"EventName": "MEM_LOAD_UOPS_RETIRED.L3_HIT",
"PEBS": "1",
- "PEBScounters": "0,1,2,3,4,5",
"SampleAfterValue": "200003",
"UMask": "0x4",
"Unit": "cpu_atom"
},
{
"BriefDescription": "Counts the number of cycles that uops are blocked for any of the following reasons: load buffer, store buffer or RSV full.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5",
"EventCode": "0x04",
"EventName": "MEM_SCHEDULER_BLOCK.ALL",
- "PEBScounters": "0,1,2,3,4,5",
"SampleAfterValue": "20003",
- "Speculative": "1",
"UMask": "0x7",
"Unit": "cpu_atom"
},
{
"BriefDescription": "Counts the number of cycles that uops are blocked due to a load buffer full condition.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5",
"EventCode": "0x04",
"EventName": "MEM_SCHEDULER_BLOCK.LD_BUF",
- "PEBScounters": "0,1,2,3,4,5",
"SampleAfterValue": "20003",
- "Speculative": "1",
"UMask": "0x2",
"Unit": "cpu_atom"
},
{
"BriefDescription": "Counts the number of cycles that uops are blocked due to an RSV full condition.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5",
"EventCode": "0x04",
"EventName": "MEM_SCHEDULER_BLOCK.RSV",
- "PEBScounters": "0,1,2,3,4,5",
"SampleAfterValue": "20003",
- "Speculative": "1",
"UMask": "0x4",
"Unit": "cpu_atom"
},
{
"BriefDescription": "Counts the number of cycles that uops are blocked due to a store buffer full condition.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5",
"EventCode": "0x04",
"EventName": "MEM_SCHEDULER_BLOCK.ST_BUF",
- "PEBScounters": "0,1,2,3,4,5",
"SampleAfterValue": "20003",
- "Speculative": "1",
"UMask": "0x1",
"Unit": "cpu_atom"
},
{
+ "BriefDescription": "MEM_STORE_RETIRED.L2_HIT",
+ "EventCode": "0x44",
+ "EventName": "MEM_STORE_RETIRED.L2_HIT",
+ "SampleAfterValue": "200003",
+ "UMask": "0x1",
+ "Unit": "cpu_core"
+ },
+ {
"BriefDescription": "Counts the number of load uops retired.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5",
"Data_LA": "1",
"EventCode": "0xd0",
"EventName": "MEM_UOPS_RETIRED.ALL_LOADS",
"PEBS": "1",
- "PEBScounters": "0,1,2,3,4,5",
+ "PublicDescription": "Counts the total number of load uops retired.",
"SampleAfterValue": "200003",
"UMask": "0x81",
"Unit": "cpu_atom"
},
{
"BriefDescription": "Counts the number of store uops retired.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5",
"Data_LA": "1",
"EventCode": "0xd0",
"EventName": "MEM_UOPS_RETIRED.ALL_STORES",
"PEBS": "1",
- "PEBScounters": "0,1,2,3,4,5",
+ "PublicDescription": "Counts the total number of store uops retired.",
"SampleAfterValue": "200003",
"UMask": "0x82",
"Unit": "cpu_atom"
},
{
"BriefDescription": "Counts the number of tagged loads with an instruction latency that exceeds or equals the threshold of 128 cycles as defined in MEC_CR_PEBS_LD_LAT_THRESHOLD (3F6H). Only counts with PEBS enabled.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1",
"Data_LA": "1",
"EventCode": "0xd0",
"EventName": "MEM_UOPS_RETIRED.LOAD_LATENCY_GT_128",
- "L1_Hit_Indication": "1",
"MSRIndex": "0x3F6",
"MSRValue": "0x80",
"PEBS": "2",
- "PEBScounters": "0,1",
+ "PublicDescription": "Counts the number of tagged loads with an instruction latency that exceeds or equals the threshold of 128 cycles as defined in MEC_CR_PEBS_LD_LAT_THRESHOLD (3F6H). Only counts with PEBS enabled. If a PEBS record is generated, will populate the PEBS Latency and PEBS Data Source fields accordingly.",
"SampleAfterValue": "1000003",
- "TakenAlone": "1",
"UMask": "0x5",
"Unit": "cpu_atom"
},
{
"BriefDescription": "Counts the number of tagged loads with an instruction latency that exceeds or equals the threshold of 16 cycles as defined in MEC_CR_PEBS_LD_LAT_THRESHOLD (3F6H). Only counts with PEBS enabled.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1",
"Data_LA": "1",
"EventCode": "0xd0",
"EventName": "MEM_UOPS_RETIRED.LOAD_LATENCY_GT_16",
- "L1_Hit_Indication": "1",
"MSRIndex": "0x3F6",
"MSRValue": "0x10",
"PEBS": "2",
- "PEBScounters": "0,1",
+ "PublicDescription": "Counts the number of tagged loads with an instruction latency that exceeds or equals the threshold of 16 cycles as defined in MEC_CR_PEBS_LD_LAT_THRESHOLD (3F6H). Only counts with PEBS enabled. If a PEBS record is generated, will populate the PEBS Latency and PEBS Data Source fields accordingly.",
"SampleAfterValue": "1000003",
- "TakenAlone": "1",
"UMask": "0x5",
"Unit": "cpu_atom"
},
{
"BriefDescription": "Counts the number of tagged loads with an instruction latency that exceeds or equals the threshold of 256 cycles as defined in MEC_CR_PEBS_LD_LAT_THRESHOLD (3F6H). Only counts with PEBS enabled.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1",
"Data_LA": "1",
"EventCode": "0xd0",
"EventName": "MEM_UOPS_RETIRED.LOAD_LATENCY_GT_256",
- "L1_Hit_Indication": "1",
"MSRIndex": "0x3F6",
"MSRValue": "0x100",
"PEBS": "2",
- "PEBScounters": "0,1",
+ "PublicDescription": "Counts the number of tagged loads with an instruction latency that exceeds or equals the threshold of 256 cycles as defined in MEC_CR_PEBS_LD_LAT_THRESHOLD (3F6H). Only counts with PEBS enabled. If a PEBS record is generated, will populate the PEBS Latency and PEBS Data Source fields accordingly.",
"SampleAfterValue": "1000003",
- "TakenAlone": "1",
"UMask": "0x5",
"Unit": "cpu_atom"
},
{
"BriefDescription": "Counts the number of tagged loads with an instruction latency that exceeds or equals the threshold of 32 cycles as defined in MEC_CR_PEBS_LD_LAT_THRESHOLD (3F6H). Only counts with PEBS enabled.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1",
"Data_LA": "1",
"EventCode": "0xd0",
"EventName": "MEM_UOPS_RETIRED.LOAD_LATENCY_GT_32",
- "L1_Hit_Indication": "1",
"MSRIndex": "0x3F6",
"MSRValue": "0x20",
"PEBS": "2",
- "PEBScounters": "0,1",
+ "PublicDescription": "Counts the number of tagged loads with an instruction latency that exceeds or equals the threshold of 32 cycles as defined in MEC_CR_PEBS_LD_LAT_THRESHOLD (3F6H). Only counts with PEBS enabled. If a PEBS record is generated, will populate the PEBS Latency and PEBS Data Source fields accordingly.",
"SampleAfterValue": "1000003",
- "TakenAlone": "1",
"UMask": "0x5",
"Unit": "cpu_atom"
},
{
"BriefDescription": "Counts the number of tagged loads with an instruction latency that exceeds or equals the threshold of 4 cycles as defined in MEC_CR_PEBS_LD_LAT_THRESHOLD (3F6H). Only counts with PEBS enabled.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1",
"Data_LA": "1",
"EventCode": "0xd0",
"EventName": "MEM_UOPS_RETIRED.LOAD_LATENCY_GT_4",
- "L1_Hit_Indication": "1",
"MSRIndex": "0x3F6",
"MSRValue": "0x4",
"PEBS": "2",
- "PEBScounters": "0,1",
+ "PublicDescription": "Counts the number of tagged loads with an instruction latency that exceeds or equals the threshold of 4 cycles as defined in MEC_CR_PEBS_LD_LAT_THRESHOLD (3F6H). Only counts with PEBS enabled. If a PEBS record is generated, will populate the PEBS Latency and PEBS Data Source fields accordingly.",
"SampleAfterValue": "1000003",
- "TakenAlone": "1",
"UMask": "0x5",
"Unit": "cpu_atom"
},
{
"BriefDescription": "Counts the number of tagged loads with an instruction latency that exceeds or equals the threshold of 512 cycles as defined in MEC_CR_PEBS_LD_LAT_THRESHOLD (3F6H). Only counts with PEBS enabled.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1",
"Data_LA": "1",
"EventCode": "0xd0",
"EventName": "MEM_UOPS_RETIRED.LOAD_LATENCY_GT_512",
- "L1_Hit_Indication": "1",
"MSRIndex": "0x3F6",
"MSRValue": "0x200",
"PEBS": "2",
- "PEBScounters": "0,1",
+ "PublicDescription": "Counts the number of tagged loads with an instruction latency that exceeds or equals the threshold of 512 cycles as defined in MEC_CR_PEBS_LD_LAT_THRESHOLD (3F6H). Only counts with PEBS enabled. If a PEBS record is generated, will populate the PEBS Latency and PEBS Data Source fields accordingly.",
"SampleAfterValue": "1000003",
- "TakenAlone": "1",
"UMask": "0x5",
"Unit": "cpu_atom"
},
{
"BriefDescription": "Counts the number of tagged loads with an instruction latency that exceeds or equals the threshold of 64 cycles as defined in MEC_CR_PEBS_LD_LAT_THRESHOLD (3F6H). Only counts with PEBS enabled.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1",
"Data_LA": "1",
"EventCode": "0xd0",
"EventName": "MEM_UOPS_RETIRED.LOAD_LATENCY_GT_64",
- "L1_Hit_Indication": "1",
"MSRIndex": "0x3F6",
"MSRValue": "0x40",
"PEBS": "2",
- "PEBScounters": "0,1",
+ "PublicDescription": "Counts the number of tagged loads with an instruction latency that exceeds or equals the threshold of 64 cycles as defined in MEC_CR_PEBS_LD_LAT_THRESHOLD (3F6H). Only counts with PEBS enabled. If a PEBS record is generated, will populate the PEBS Latency and PEBS Data Source fields accordingly.",
"SampleAfterValue": "1000003",
- "TakenAlone": "1",
"UMask": "0x5",
"Unit": "cpu_atom"
},
{
"BriefDescription": "Counts the number of tagged loads with an instruction latency that exceeds or equals the threshold of 8 cycles as defined in MEC_CR_PEBS_LD_LAT_THRESHOLD (3F6H). Only counts with PEBS enabled.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1",
"Data_LA": "1",
"EventCode": "0xd0",
"EventName": "MEM_UOPS_RETIRED.LOAD_LATENCY_GT_8",
- "L1_Hit_Indication": "1",
"MSRIndex": "0x3F6",
"MSRValue": "0x8",
"PEBS": "2",
- "PEBScounters": "0,1",
+ "PublicDescription": "Counts the number of tagged loads with an instruction latency that exceeds or equals the threshold of 8 cycles as defined in MEC_CR_PEBS_LD_LAT_THRESHOLD (3F6H). Only counts with PEBS enabled. If a PEBS record is generated, will populate the PEBS Latency and PEBS Data Source fields accordingly.",
"SampleAfterValue": "1000003",
- "TakenAlone": "1",
"UMask": "0x5",
"Unit": "cpu_atom"
},
{
"BriefDescription": "Counts the number of retired split load uops.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5",
"Data_LA": "1",
"EventCode": "0xd0",
"EventName": "MEM_UOPS_RETIRED.SPLIT_LOADS",
"PEBS": "1",
- "PEBScounters": "0,1,2,3,4,5",
"SampleAfterValue": "200003",
"UMask": "0x41",
"Unit": "cpu_atom"
},
{
"BriefDescription": "Counts the number of stores uops retired. Counts with or without PEBS enabled.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5",
"Data_LA": "1",
"EventCode": "0xd0",
"EventName": "MEM_UOPS_RETIRED.STORE_LATENCY",
- "L1_Hit_Indication": "1",
"PEBS": "2",
- "PEBScounters": "0,1,2,3,4,5",
+ "PublicDescription": "Counts the number of stores uops retired. Counts with or without PEBS enabled. If PEBS is enabled and a PEBS record is generated, will populate PEBS Latency and PEBS Data Source fields accordingly.",
"SampleAfterValue": "1000003",
"UMask": "0x6",
"Unit": "cpu_atom"
},
{
+ "BriefDescription": "Retired memory uops for any access",
+ "EventCode": "0xe5",
+ "EventName": "MEM_UOP_RETIRED.ANY",
+ "PublicDescription": "Number of retired micro-operations (uops) for load or store memory accesses",
+ "SampleAfterValue": "1000003",
+ "UMask": "0x3",
+ "Unit": "cpu_core"
+ },
+ {
"BriefDescription": "Counts demand data reads that were supplied by the L3 cache.",
- "Counter": "0,1,2,3,4,5",
"EventCode": "0xB7",
"EventName": "OCR.DEMAND_DATA_RD.L3_HIT",
"MSRIndex": "0x1a6,0x1a7",
@@ -408,7 +856,6 @@
},
{
"BriefDescription": "Counts demand data reads that were supplied by the L3 cache where a snoop was sent, the snoop hit, and modified data was forwarded.",
- "Counter": "0,1,2,3,4,5",
"EventCode": "0xB7",
"EventName": "OCR.DEMAND_DATA_RD.L3_HIT.SNOOP_HITM",
"MSRIndex": "0x1a6,0x1a7",
@@ -418,8 +865,17 @@
"Unit": "cpu_atom"
},
{
+ "BriefDescription": "Counts demand data reads that resulted in a snoop hit in another cores caches, data forwarding is required as the data is modified.",
+ "EventCode": "0x2A,0x2B",
+ "EventName": "OCR.DEMAND_DATA_RD.L3_HIT.SNOOP_HITM",
+ "MSRIndex": "0x1a6,0x1a7",
+ "MSRValue": "0x10003C0001",
+ "SampleAfterValue": "100003",
+ "UMask": "0x1",
+ "Unit": "cpu_core"
+ },
+ {
"BriefDescription": "Counts demand data reads that were supplied by the L3 cache where a snoop was sent, the snoop hit, but no data was forwarded.",
- "Counter": "0,1,2,3,4,5",
"EventCode": "0xB7",
"EventName": "OCR.DEMAND_DATA_RD.L3_HIT.SNOOP_HIT_NO_FWD",
"MSRIndex": "0x1a6,0x1a7",
@@ -430,7 +886,6 @@
},
{
"BriefDescription": "Counts demand data reads that were supplied by the L3 cache where a snoop was sent, the snoop hit, and non-modified data was forwarded.",
- "Counter": "0,1,2,3,4,5",
"EventCode": "0xB7",
"EventName": "OCR.DEMAND_DATA_RD.L3_HIT.SNOOP_HIT_WITH_FWD",
"MSRIndex": "0x1a6,0x1a7",
@@ -440,8 +895,17 @@
"Unit": "cpu_atom"
},
{
+ "BriefDescription": "OCR.DEMAND_DATA_RD.L3_HIT.SNOOP_HIT_WITH_FWD",
+ "EventCode": "0x2A,0x2B",
+ "EventName": "OCR.DEMAND_DATA_RD.L3_HIT.SNOOP_HIT_WITH_FWD",
+ "MSRIndex": "0x1a6,0x1a7",
+ "MSRValue": "0x8003C0001",
+ "SampleAfterValue": "100003",
+ "UMask": "0x1",
+ "Unit": "cpu_core"
+ },
+ {
"BriefDescription": "Counts demand reads for ownership (RFO) and software prefetches for exclusive ownership (PREFETCHW) that were supplied by the L3 cache.",
- "Counter": "0,1,2,3,4,5",
"EventCode": "0xB7",
"EventName": "OCR.DEMAND_RFO.L3_HIT",
"MSRIndex": "0x1a6,0x1a7",
@@ -452,7 +916,6 @@
},
{
"BriefDescription": "Counts demand reads for ownership (RFO) and software prefetches for exclusive ownership (PREFETCHW) that were supplied by the L3 cache where a snoop was sent, the snoop hit, and modified data was forwarded.",
- "Counter": "0,1,2,3,4,5",
"EventCode": "0xB7",
"EventName": "OCR.DEMAND_RFO.L3_HIT.SNOOP_HITM",
"MSRIndex": "0x1a6,0x1a7",
@@ -462,739 +925,7 @@
"Unit": "cpu_atom"
},
{
- "BriefDescription": "Counts the number of issue slots every cycle that were not delivered by the frontend due to instruction cache misses.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5",
- "EventCode": "0x71",
- "EventName": "TOPDOWN_FE_BOUND.ICACHE",
- "PEBScounters": "0,1,2,3,4,5",
- "SampleAfterValue": "1000003",
- "Speculative": "1",
- "UMask": "0x20",
- "Unit": "cpu_atom"
- },
- {
- "BriefDescription": "L1D.HWPF_MISS",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3",
- "EventCode": "0x51",
- "EventName": "L1D.HWPF_MISS",
- "PEBScounters": "0,1,2,3",
- "SampleAfterValue": "1000003",
- "Speculative": "1",
- "UMask": "0x20",
- "Unit": "cpu_core"
- },
- {
- "BriefDescription": "Counts the number of cache lines replaced in L1 data cache.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3",
- "EventCode": "0x51",
- "EventName": "L1D.REPLACEMENT",
- "PEBScounters": "0,1,2,3",
- "SampleAfterValue": "100003",
- "Speculative": "1",
- "UMask": "0x1",
- "Unit": "cpu_core"
- },
- {
- "BriefDescription": "Number of cycles a demand request has waited due to L1D Fill Buffer (FB) unavailability.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3",
- "EventCode": "0x48",
- "EventName": "L1D_PEND_MISS.FB_FULL",
- "PEBScounters": "0,1,2,3",
- "SampleAfterValue": "1000003",
- "Speculative": "1",
- "UMask": "0x2",
- "Unit": "cpu_core"
- },
- {
- "BriefDescription": "Number of phases a demand request has waited due to L1D Fill Buffer (FB) unavailablability.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3",
- "CounterMask": "1",
- "EdgeDetect": "1",
- "EventCode": "0x48",
- "EventName": "L1D_PEND_MISS.FB_FULL_PERIODS",
- "PEBScounters": "0,1,2,3",
- "SampleAfterValue": "1000003",
- "Speculative": "1",
- "UMask": "0x2",
- "Unit": "cpu_core"
- },
- {
- "BriefDescription": "This event is deprecated. Refer to new event L1D_PEND_MISS.L2_STALLS",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3",
- "EventCode": "0x48",
- "EventName": "L1D_PEND_MISS.L2_STALL",
- "PEBScounters": "0,1,2,3",
- "SampleAfterValue": "1000003",
- "Speculative": "1",
- "UMask": "0x4",
- "Unit": "cpu_core"
- },
- {
- "BriefDescription": "Number of cycles a demand request has waited due to L1D due to lack of L2 resources.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3",
- "EventCode": "0x48",
- "EventName": "L1D_PEND_MISS.L2_STALLS",
- "PEBScounters": "0,1,2,3",
- "SampleAfterValue": "1000003",
- "Speculative": "1",
- "UMask": "0x4",
- "Unit": "cpu_core"
- },
- {
- "BriefDescription": "Number of L1D misses that are outstanding",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3",
- "EventCode": "0x48",
- "EventName": "L1D_PEND_MISS.PENDING",
- "PEBScounters": "0,1,2,3",
- "SampleAfterValue": "1000003",
- "Speculative": "1",
- "UMask": "0x1",
- "Unit": "cpu_core"
- },
- {
- "BriefDescription": "Cycles with L1D load Misses outstanding.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3",
- "CounterMask": "1",
- "EventCode": "0x48",
- "EventName": "L1D_PEND_MISS.PENDING_CYCLES",
- "PEBScounters": "0,1,2,3",
- "SampleAfterValue": "1000003",
- "Speculative": "1",
- "UMask": "0x1",
- "Unit": "cpu_core"
- },
- {
- "BriefDescription": "L2 cache lines filling L2",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3",
- "EventCode": "0x25",
- "EventName": "L2_LINES_IN.ALL",
- "PEBScounters": "0,1,2,3",
- "SampleAfterValue": "100003",
- "Speculative": "1",
- "UMask": "0x1f",
- "Unit": "cpu_core"
- },
- {
- "BriefDescription": "Cache lines that have been L2 hardware prefetched but not used by demand accesses",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3",
- "EventCode": "0x26",
- "EventName": "L2_LINES_OUT.USELESS_HWPF",
- "PEBScounters": "0,1,2,3",
- "SampleAfterValue": "200003",
- "Speculative": "1",
- "UMask": "0x4",
- "Unit": "cpu_core"
- },
- {
- "BriefDescription": "All accesses to L2 cache[This event is alias to L2_RQSTS.REFERENCES]",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3",
- "EventCode": "0x24",
- "EventName": "L2_REQUEST.ALL",
- "PEBScounters": "0,1,2,3",
- "SampleAfterValue": "200003",
- "Speculative": "1",
- "UMask": "0xff",
- "Unit": "cpu_core"
- },
- {
- "BriefDescription": "Read requests with true-miss in L2 cache.[This event is alias to L2_RQSTS.MISS]",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3",
- "EventCode": "0x24",
- "EventName": "L2_REQUEST.MISS",
- "PEBScounters": "0,1,2,3",
- "SampleAfterValue": "200003",
- "Speculative": "1",
- "UMask": "0x3f",
- "Unit": "cpu_core"
- },
- {
- "BriefDescription": "L2 code requests",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3",
- "EventCode": "0x24",
- "EventName": "L2_RQSTS.ALL_CODE_RD",
- "PEBScounters": "0,1,2,3",
- "SampleAfterValue": "200003",
- "Speculative": "1",
- "UMask": "0xe4",
- "Unit": "cpu_core"
- },
- {
- "BriefDescription": "Demand Data Read access L2 cache",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3",
- "EventCode": "0x24",
- "EventName": "L2_RQSTS.ALL_DEMAND_DATA_RD",
- "PEBScounters": "0,1,2,3",
- "SampleAfterValue": "200003",
- "Speculative": "1",
- "UMask": "0xe1",
- "Unit": "cpu_core"
- },
- {
- "BriefDescription": "Demand requests that miss L2 cache",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3",
- "EventCode": "0x24",
- "EventName": "L2_RQSTS.ALL_DEMAND_MISS",
- "PEBScounters": "0,1,2,3",
- "SampleAfterValue": "200003",
- "Speculative": "1",
- "UMask": "0x27",
- "Unit": "cpu_core"
- },
- {
- "BriefDescription": "L2_RQSTS.ALL_HWPF",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3",
- "EventCode": "0x24",
- "EventName": "L2_RQSTS.ALL_HWPF",
- "PEBScounters": "0,1,2,3",
- "SampleAfterValue": "200003",
- "Speculative": "1",
- "UMask": "0xf0",
- "Unit": "cpu_core"
- },
- {
- "BriefDescription": "RFO requests to L2 cache.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3",
- "EventCode": "0x24",
- "EventName": "L2_RQSTS.ALL_RFO",
- "PEBScounters": "0,1,2,3",
- "SampleAfterValue": "200003",
- "Speculative": "1",
- "UMask": "0xe2",
- "Unit": "cpu_core"
- },
- {
- "BriefDescription": "L2 cache hits when fetching instructions, code reads.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3",
- "EventCode": "0x24",
- "EventName": "L2_RQSTS.CODE_RD_HIT",
- "PEBScounters": "0,1,2,3",
- "SampleAfterValue": "200003",
- "Speculative": "1",
- "UMask": "0xc4",
- "Unit": "cpu_core"
- },
- {
- "BriefDescription": "L2 cache misses when fetching instructions",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3",
- "EventCode": "0x24",
- "EventName": "L2_RQSTS.CODE_RD_MISS",
- "PEBScounters": "0,1,2,3",
- "SampleAfterValue": "200003",
- "Speculative": "1",
- "UMask": "0x24",
- "Unit": "cpu_core"
- },
- {
- "BriefDescription": "Demand Data Read requests that hit L2 cache",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3",
- "EventCode": "0x24",
- "EventName": "L2_RQSTS.DEMAND_DATA_RD_HIT",
- "PEBScounters": "0,1,2,3",
- "SampleAfterValue": "200003",
- "Speculative": "1",
- "UMask": "0xc1",
- "Unit": "cpu_core"
- },
- {
- "BriefDescription": "Demand Data Read miss L2 cache",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3",
- "EventCode": "0x24",
- "EventName": "L2_RQSTS.DEMAND_DATA_RD_MISS",
- "PEBScounters": "0,1,2,3",
- "SampleAfterValue": "200003",
- "Speculative": "1",
- "UMask": "0x21",
- "Unit": "cpu_core"
- },
- {
- "BriefDescription": "L2_RQSTS.HWPF_MISS",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3",
- "EventCode": "0x24",
- "EventName": "L2_RQSTS.HWPF_MISS",
- "PEBScounters": "0,1,2,3",
- "SampleAfterValue": "200003",
- "Speculative": "1",
- "UMask": "0x30",
- "Unit": "cpu_core"
- },
- {
- "BriefDescription": "Read requests with true-miss in L2 cache.[This event is alias to L2_REQUEST.MISS]",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3",
- "EventCode": "0x24",
- "EventName": "L2_RQSTS.MISS",
- "PEBScounters": "0,1,2,3",
- "SampleAfterValue": "200003",
- "Speculative": "1",
- "UMask": "0x3f",
- "Unit": "cpu_core"
- },
- {
- "BriefDescription": "All accesses to L2 cache[This event is alias to L2_REQUEST.ALL]",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3",
- "EventCode": "0x24",
- "EventName": "L2_RQSTS.REFERENCES",
- "PEBScounters": "0,1,2,3",
- "SampleAfterValue": "200003",
- "Speculative": "1",
- "UMask": "0xff",
- "Unit": "cpu_core"
- },
- {
- "BriefDescription": "RFO requests that hit L2 cache.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3",
- "EventCode": "0x24",
- "EventName": "L2_RQSTS.RFO_HIT",
- "PEBScounters": "0,1,2,3",
- "SampleAfterValue": "200003",
- "Speculative": "1",
- "UMask": "0xc2",
- "Unit": "cpu_core"
- },
- {
- "BriefDescription": "RFO requests that miss L2 cache",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3",
- "EventCode": "0x24",
- "EventName": "L2_RQSTS.RFO_MISS",
- "PEBScounters": "0,1,2,3",
- "SampleAfterValue": "200003",
- "Speculative": "1",
- "UMask": "0x22",
- "Unit": "cpu_core"
- },
- {
- "BriefDescription": "SW prefetch requests that hit L2 cache.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3",
- "EventCode": "0x24",
- "EventName": "L2_RQSTS.SWPF_HIT",
- "PEBScounters": "0,1,2,3",
- "SampleAfterValue": "200003",
- "Speculative": "1",
- "UMask": "0xc8",
- "Unit": "cpu_core"
- },
- {
- "BriefDescription": "SW prefetch requests that miss L2 cache.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3",
- "EventCode": "0x24",
- "EventName": "L2_RQSTS.SWPF_MISS",
- "PEBScounters": "0,1,2,3",
- "SampleAfterValue": "200003",
- "Speculative": "1",
- "UMask": "0x28",
- "Unit": "cpu_core"
- },
- {
- "BriefDescription": "Core-originated cacheable requests that missed L3 (Except hardware prefetches to the L3)",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5,6,7",
- "EventCode": "0x2e",
- "EventName": "LONGEST_LAT_CACHE.MISS",
- "PEBScounters": "0,1,2,3,4,5,6,7",
- "SampleAfterValue": "100003",
- "Speculative": "1",
- "UMask": "0x41",
- "Unit": "cpu_core"
- },
- {
- "BriefDescription": "Core-originated cacheable requests that refer to L3 (Except hardware prefetches to the L3)",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5,6,7",
- "EventCode": "0x2e",
- "EventName": "LONGEST_LAT_CACHE.REFERENCE",
- "PEBScounters": "0,1,2,3,4,5,6,7",
- "SampleAfterValue": "100003",
- "Speculative": "1",
- "UMask": "0x4f",
- "Unit": "cpu_core"
- },
- {
- "BriefDescription": "Retired load instructions.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3",
- "Data_LA": "1",
- "EventCode": "0xd0",
- "EventName": "MEM_INST_RETIRED.ALL_LOADS",
- "PEBS": "1",
- "PEBScounters": "0,1,2,3",
- "SampleAfterValue": "1000003",
- "UMask": "0x81",
- "Unit": "cpu_core"
- },
- {
- "BriefDescription": "Retired store instructions.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3",
- "Data_LA": "1",
- "EventCode": "0xd0",
- "EventName": "MEM_INST_RETIRED.ALL_STORES",
- "L1_Hit_Indication": "1",
- "PEBS": "1",
- "PEBScounters": "0,1,2,3",
- "SampleAfterValue": "1000003",
- "UMask": "0x82",
- "Unit": "cpu_core"
- },
- {
- "BriefDescription": "All retired memory instructions.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3",
- "Data_LA": "1",
- "EventCode": "0xd0",
- "EventName": "MEM_INST_RETIRED.ANY",
- "L1_Hit_Indication": "1",
- "PEBS": "1",
- "PEBScounters": "0,1,2,3",
- "SampleAfterValue": "1000003",
- "UMask": "0x83",
- "Unit": "cpu_core"
- },
- {
- "BriefDescription": "Retired load instructions with locked access.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3",
- "Data_LA": "1",
- "EventCode": "0xd0",
- "EventName": "MEM_INST_RETIRED.LOCK_LOADS",
- "PEBS": "1",
- "PEBScounters": "0,1,2,3",
- "SampleAfterValue": "100007",
- "UMask": "0x21",
- "Unit": "cpu_core"
- },
- {
- "BriefDescription": "Retired load instructions that split across a cacheline boundary.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3",
- "Data_LA": "1",
- "EventCode": "0xd0",
- "EventName": "MEM_INST_RETIRED.SPLIT_LOADS",
- "PEBS": "1",
- "PEBScounters": "0,1,2,3",
- "SampleAfterValue": "100003",
- "UMask": "0x41",
- "Unit": "cpu_core"
- },
- {
- "BriefDescription": "Retired store instructions that split across a cacheline boundary.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3",
- "Data_LA": "1",
- "EventCode": "0xd0",
- "EventName": "MEM_INST_RETIRED.SPLIT_STORES",
- "L1_Hit_Indication": "1",
- "PEBS": "1",
- "PEBScounters": "0,1,2,3",
- "SampleAfterValue": "100003",
- "UMask": "0x42",
- "Unit": "cpu_core"
- },
- {
- "BriefDescription": "Retired load instructions that miss the STLB.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3",
- "Data_LA": "1",
- "EventCode": "0xd0",
- "EventName": "MEM_INST_RETIRED.STLB_MISS_LOADS",
- "PEBS": "1",
- "PEBScounters": "0,1,2,3",
- "SampleAfterValue": "100003",
- "UMask": "0x11",
- "Unit": "cpu_core"
- },
- {
- "BriefDescription": "Retired store instructions that miss the STLB.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3",
- "Data_LA": "1",
- "EventCode": "0xd0",
- "EventName": "MEM_INST_RETIRED.STLB_MISS_STORES",
- "L1_Hit_Indication": "1",
- "PEBS": "1",
- "PEBScounters": "0,1,2,3",
- "SampleAfterValue": "100003",
- "UMask": "0x12",
- "Unit": "cpu_core"
- },
- {
- "BriefDescription": "Completed demand load uops that miss the L1 d-cache.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3",
- "EventCode": "0x43",
- "EventName": "MEM_LOAD_COMPLETED.L1_MISS_ANY",
- "PEBScounters": "0,1,2,3",
- "SampleAfterValue": "1000003",
- "Speculative": "1",
- "UMask": "0xfd",
- "Unit": "cpu_core"
- },
- {
- "BriefDescription": "Retired load instructions whose data sources were HitM responses from shared L3",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3",
- "Data_LA": "1",
- "EventCode": "0xd2",
- "EventName": "MEM_LOAD_L3_HIT_RETIRED.XSNP_FWD",
- "PEBS": "1",
- "PEBScounters": "0,1,2,3",
- "SampleAfterValue": "20011",
- "UMask": "0x4",
- "Unit": "cpu_core"
- },
- {
- "BriefDescription": "Retired load instructions whose data sources were L3 and cross-core snoop hits in on-pkg core cache",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3",
- "Data_LA": "1",
- "EventCode": "0xd2",
- "EventName": "MEM_LOAD_L3_HIT_RETIRED.XSNP_HIT",
- "PEBS": "1",
- "PEBScounters": "0,1,2,3",
- "SampleAfterValue": "20011",
- "UMask": "0x2",
- "Unit": "cpu_core"
- },
- {
- "BriefDescription": "Retired load instructions whose data sources were HitM responses from shared L3",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3",
- "Data_LA": "1",
- "EventCode": "0xd2",
- "EventName": "MEM_LOAD_L3_HIT_RETIRED.XSNP_HITM",
- "PEBS": "1",
- "PEBScounters": "0,1,2,3",
- "SampleAfterValue": "20011",
- "UMask": "0x4",
- "Unit": "cpu_core"
- },
- {
- "BriefDescription": "Retired load instructions whose data sources were L3 hit and cross-core snoop missed in on-pkg core cache.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3",
- "Data_LA": "1",
- "EventCode": "0xd2",
- "EventName": "MEM_LOAD_L3_HIT_RETIRED.XSNP_MISS",
- "PEBS": "1",
- "PEBScounters": "0,1,2,3",
- "SampleAfterValue": "20011",
- "UMask": "0x1",
- "Unit": "cpu_core"
- },
- {
- "BriefDescription": "Retired load instructions whose data sources were hits in L3 without snoops required",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3",
- "Data_LA": "1",
- "EventCode": "0xd2",
- "EventName": "MEM_LOAD_L3_HIT_RETIRED.XSNP_NONE",
- "PEBS": "1",
- "PEBScounters": "0,1,2,3",
- "SampleAfterValue": "100003",
- "UMask": "0x8",
- "Unit": "cpu_core"
- },
- {
- "BriefDescription": "Retired load instructions whose data sources were L3 and cross-core snoop hits in on-pkg core cache",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3",
- "Data_LA": "1",
- "EventCode": "0xd2",
- "EventName": "MEM_LOAD_L3_HIT_RETIRED.XSNP_NO_FWD",
- "PEBS": "1",
- "PEBScounters": "0,1,2,3",
- "SampleAfterValue": "20011",
- "UMask": "0x2",
- "Unit": "cpu_core"
- },
- {
- "BriefDescription": "Retired load instructions which data sources missed L3 but serviced from local dram",
- "Counter": "0,1,2,3",
- "Data_LA": "1",
- "EventCode": "0xd3",
- "EventName": "MEM_LOAD_L3_MISS_RETIRED.LOCAL_DRAM",
- "PEBScounters": "0,1,2,3",
- "SampleAfterValue": "100007",
- "UMask": "0x1",
- "Unit": "cpu_core"
- },
- {
- "BriefDescription": "Retired instructions with at least 1 uncacheable load or lock.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3",
- "Data_LA": "1",
- "EventCode": "0xd4",
- "EventName": "MEM_LOAD_MISC_RETIRED.UC",
- "PEBS": "1",
- "PEBScounters": "0,1,2,3",
- "SampleAfterValue": "100007",
- "UMask": "0x4",
- "Unit": "cpu_core"
- },
- {
- "BriefDescription": "Number of completed demand load requests that missed the L1, but hit the FB(fill buffer), because a preceding miss to the same cacheline initiated the line to be brought into L1, but data is not yet ready in L1.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3",
- "Data_LA": "1",
- "EventCode": "0xd1",
- "EventName": "MEM_LOAD_RETIRED.FB_HIT",
- "PEBS": "1",
- "PEBScounters": "0,1,2,3",
- "SampleAfterValue": "100007",
- "UMask": "0x40",
- "Unit": "cpu_core"
- },
- {
- "BriefDescription": "Retired load instructions with L1 cache hits as data sources",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3",
- "Data_LA": "1",
- "EventCode": "0xd1",
- "EventName": "MEM_LOAD_RETIRED.L1_HIT",
- "PEBS": "1",
- "PEBScounters": "0,1,2,3",
- "SampleAfterValue": "1000003",
- "UMask": "0x1",
- "Unit": "cpu_core"
- },
- {
- "BriefDescription": "Retired load instructions missed L1 cache as data sources",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3",
- "Data_LA": "1",
- "EventCode": "0xd1",
- "EventName": "MEM_LOAD_RETIRED.L1_MISS",
- "PEBS": "1",
- "PEBScounters": "0,1,2,3",
- "SampleAfterValue": "200003",
- "UMask": "0x8",
- "Unit": "cpu_core"
- },
- {
- "BriefDescription": "Retired load instructions with L2 cache hits as data sources",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3",
- "Data_LA": "1",
- "EventCode": "0xd1",
- "EventName": "MEM_LOAD_RETIRED.L2_HIT",
- "PEBS": "1",
- "PEBScounters": "0,1,2,3",
- "SampleAfterValue": "200003",
- "UMask": "0x2",
- "Unit": "cpu_core"
- },
- {
- "BriefDescription": "Retired load instructions missed L2 cache as data sources",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3",
- "Data_LA": "1",
- "EventCode": "0xd1",
- "EventName": "MEM_LOAD_RETIRED.L2_MISS",
- "PEBS": "1",
- "PEBScounters": "0,1,2,3",
- "SampleAfterValue": "100021",
- "UMask": "0x10",
- "Unit": "cpu_core"
- },
- {
- "BriefDescription": "Retired load instructions with L3 cache hits as data sources",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3",
- "Data_LA": "1",
- "EventCode": "0xd1",
- "EventName": "MEM_LOAD_RETIRED.L3_HIT",
- "PEBS": "1",
- "PEBScounters": "0,1,2,3",
- "SampleAfterValue": "100021",
- "UMask": "0x4",
- "Unit": "cpu_core"
- },
- {
- "BriefDescription": "Retired load instructions missed L3 cache as data sources",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3",
- "Data_LA": "1",
- "EventCode": "0xd1",
- "EventName": "MEM_LOAD_RETIRED.L3_MISS",
- "PEBS": "1",
- "PEBScounters": "0,1,2,3",
- "SampleAfterValue": "50021",
- "UMask": "0x20",
- "Unit": "cpu_core"
- },
- {
- "BriefDescription": "MEM_STORE_RETIRED.L2_HIT",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3",
- "EventCode": "0x44",
- "EventName": "MEM_STORE_RETIRED.L2_HIT",
- "PEBScounters": "0,1,2,3",
- "SampleAfterValue": "200003",
- "UMask": "0x1",
- "Unit": "cpu_core"
- },
- {
- "BriefDescription": "Retired memory uops for any access",
- "Counter": "0,1,2,3,4,5,6,7",
- "EventCode": "0xe5",
- "EventName": "MEM_UOP_RETIRED.ANY",
- "PEBScounters": "0,1,2,3,4,5,6,7",
- "SampleAfterValue": "1000003",
- "UMask": "0x3",
- "Unit": "cpu_core"
- },
- {
- "BriefDescription": "Counts demand data reads that resulted in a snoop hit in another cores caches, data forwarding is required as the data is modified.",
- "Counter": "0,1,2,3,4,5,6,7",
- "EventCode": "0x2A,0x2B",
- "EventName": "OCR.DEMAND_DATA_RD.L3_HIT.SNOOP_HITM",
- "MSRIndex": "0x1a6,0x1a7",
- "MSRValue": "0x10003C0001",
- "SampleAfterValue": "100003",
- "UMask": "0x1",
- "Unit": "cpu_core"
- },
- {
- "BriefDescription": "OCR.DEMAND_DATA_RD.L3_HIT.SNOOP_HIT_WITH_FWD",
- "Counter": "0,1,2,3,4,5,6,7",
- "EventCode": "0x2A,0x2B",
- "EventName": "OCR.DEMAND_DATA_RD.L3_HIT.SNOOP_HIT_WITH_FWD",
- "MSRIndex": "0x1a6,0x1a7",
- "MSRValue": "0x8003C0001",
- "SampleAfterValue": "100003",
- "UMask": "0x1",
- "Unit": "cpu_core"
- },
- {
"BriefDescription": "Counts demand read for ownership (RFO) requests and software prefetches for exclusive ownership (PREFETCHW) that resulted in a snoop hit in another cores caches, data forwarding is required as the data is modified.",
- "Counter": "0,1,2,3,4,5,6,7",
"EventCode": "0x2A,0x2B",
"EventName": "OCR.DEMAND_RFO.L3_HIT.SNOOP_HITM",
"MSRIndex": "0x1a6,0x1a7",
@@ -1205,139 +936,111 @@
},
{
"BriefDescription": "OFFCORE_REQUESTS.ALL_REQUESTS",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3",
"EventCode": "0x21",
"EventName": "OFFCORE_REQUESTS.ALL_REQUESTS",
- "PEBScounters": "0,1,2,3",
"SampleAfterValue": "100003",
- "Speculative": "1",
"UMask": "0x80",
"Unit": "cpu_core"
},
{
"BriefDescription": "Demand and prefetch data reads",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3",
"EventCode": "0x21",
"EventName": "OFFCORE_REQUESTS.DATA_RD",
- "PEBScounters": "0,1,2,3",
+ "PublicDescription": "Counts the demand and prefetch data reads. All Core Data Reads include cacheable 'Demands' and L2 prefetchers (not L3 prefetchers). Counting also covers reads due to page walks resulted from any request type.",
"SampleAfterValue": "100003",
- "Speculative": "1",
"UMask": "0x8",
"Unit": "cpu_core"
},
{
"BriefDescription": "Demand Data Read requests sent to uncore",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3",
"EventCode": "0x21",
"EventName": "OFFCORE_REQUESTS.DEMAND_DATA_RD",
- "PEBScounters": "0,1,2,3",
+ "PublicDescription": "Counts the Demand Data Read requests sent to uncore. Use it in conjunction with OFFCORE_REQUESTS_OUTSTANDING to determine average latency in the uncore.",
"SampleAfterValue": "100003",
- "Speculative": "1",
"UMask": "0x1",
"Unit": "cpu_core"
},
{
"BriefDescription": "This event is deprecated. Refer to new event OFFCORE_REQUESTS_OUTSTANDING.DATA_RD",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3",
+ "Deprecated": "1",
"Errata": "ADL038",
"EventCode": "0x20",
"EventName": "OFFCORE_REQUESTS_OUTSTANDING.ALL_DATA_RD",
- "PEBScounters": "0,1,2,3",
"SampleAfterValue": "1000003",
- "Speculative": "1",
"UMask": "0x8",
"Unit": "cpu_core"
},
{
"BriefDescription": "OFFCORE_REQUESTS_OUTSTANDING.CYCLES_WITH_DATA_RD",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3",
"CounterMask": "1",
"Errata": "ADL038",
"EventCode": "0x20",
"EventName": "OFFCORE_REQUESTS_OUTSTANDING.CYCLES_WITH_DATA_RD",
- "PEBScounters": "0,1,2,3",
"SampleAfterValue": "1000003",
- "Speculative": "1",
"UMask": "0x8",
"Unit": "cpu_core"
},
{
"BriefDescription": "For every cycle where the core is waiting on at least 1 outstanding Demand RFO request, increments by 1.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3",
"CounterMask": "1",
"EventCode": "0x20",
"EventName": "OFFCORE_REQUESTS_OUTSTANDING.CYCLES_WITH_DEMAND_RFO",
- "PEBScounters": "0,1,2,3",
+ "PublicDescription": "OFFCORE_REQUESTS_OUTSTANDING.CYCLES_WITH_DEMAND_RFO",
"SampleAfterValue": "1000003",
- "Speculative": "1",
"UMask": "0x4",
"Unit": "cpu_core"
},
{
"BriefDescription": "OFFCORE_REQUESTS_OUTSTANDING.DATA_RD",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3",
"Errata": "ADL038",
"EventCode": "0x20",
"EventName": "OFFCORE_REQUESTS_OUTSTANDING.DATA_RD",
- "PEBScounters": "0,1,2,3",
"SampleAfterValue": "1000003",
- "Speculative": "1",
"UMask": "0x8",
"Unit": "cpu_core"
},
{
"BriefDescription": "Number of PREFETCHNTA instructions executed.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3",
"EventCode": "0x40",
"EventName": "SW_PREFETCH_ACCESS.NTA",
- "PEBScounters": "0,1,2,3",
+ "PublicDescription": "Counts the number of PREFETCHNTA instructions executed.",
"SampleAfterValue": "100003",
- "Speculative": "1",
"UMask": "0x1",
"Unit": "cpu_core"
},
{
"BriefDescription": "Number of PREFETCHW instructions executed.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3",
"EventCode": "0x40",
"EventName": "SW_PREFETCH_ACCESS.PREFETCHW",
- "PEBScounters": "0,1,2,3",
+ "PublicDescription": "Counts the number of PREFETCHW instructions executed.",
"SampleAfterValue": "100003",
- "Speculative": "1",
"UMask": "0x8",
"Unit": "cpu_core"
},
{
"BriefDescription": "Number of PREFETCHT0 instructions executed.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3",
"EventCode": "0x40",
"EventName": "SW_PREFETCH_ACCESS.T0",
- "PEBScounters": "0,1,2,3",
+ "PublicDescription": "Counts the number of PREFETCHT0 instructions executed.",
"SampleAfterValue": "100003",
- "Speculative": "1",
"UMask": "0x2",
"Unit": "cpu_core"
},
{
"BriefDescription": "Number of PREFETCHT1 or PREFETCHT2 instructions executed.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3",
"EventCode": "0x40",
"EventName": "SW_PREFETCH_ACCESS.T1_T2",
- "PEBScounters": "0,1,2,3",
+ "PublicDescription": "Counts the number of PREFETCHT1 or PREFETCHT2 instructions executed.",
"SampleAfterValue": "100003",
- "Speculative": "1",
"UMask": "0x4",
"Unit": "cpu_core"
+ },
+ {
+ "BriefDescription": "Counts the number of issue slots every cycle that were not delivered by the frontend due to instruction cache misses.",
+ "EventCode": "0x71",
+ "EventName": "TOPDOWN_FE_BOUND.ICACHE",
+ "SampleAfterValue": "1000003",
+ "UMask": "0x20",
+ "Unit": "cpu_atom"
}
]
diff --git a/tools/perf/pmu-events/arch/x86/alderlake/floating-point.json b/tools/perf/pmu-events/arch/x86/alderlake/floating-point.json
index 48a4605..3eb7cab 100644
--- a/tools/perf/pmu-events/arch/x86/alderlake/floating-point.json
+++ b/tools/perf/pmu-events/arch/x86/alderlake/floating-point.json
@@ -1,165 +1,124 @@
[
{
- "BriefDescription": "Counts the number of floating point operations retired that required microcode assist.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5",
- "EventCode": "0xc3",
- "EventName": "MACHINE_CLEARS.FP_ASSIST",
- "PEBScounters": "0,1,2,3,4,5",
- "SampleAfterValue": "20003",
- "Speculative": "1",
- "UMask": "0x4",
- "Unit": "cpu_atom"
- },
- {
- "BriefDescription": "Counts the number of floating point divide uops retired (x87 and SSE, including x87 sqrt).",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5",
- "EventCode": "0xc2",
- "EventName": "UOPS_RETIRED.FPDIV",
- "PEBS": "1",
- "PEBScounters": "0,1,2,3,4,5",
- "SampleAfterValue": "2000003",
- "UMask": "0x8",
- "Unit": "cpu_atom"
- },
- {
"BriefDescription": "ARITH.FPDIV_ACTIVE",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5,6,7",
"CounterMask": "1",
"EventCode": "0xb0",
"EventName": "ARITH.FPDIV_ACTIVE",
- "PEBScounters": "0,1,2,3,4,5,6,7",
"SampleAfterValue": "1000003",
- "Speculative": "1",
"UMask": "0x1",
"Unit": "cpu_core"
},
{
"BriefDescription": "Counts all microcode FP assists.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5,6,7",
"EventCode": "0xc1",
"EventName": "ASSISTS.FP",
- "PEBScounters": "0,1,2,3,4,5,6,7",
+ "PublicDescription": "Counts all microcode Floating Point assists.",
"SampleAfterValue": "100003",
- "Speculative": "1",
"UMask": "0x2",
"Unit": "cpu_core"
},
{
"BriefDescription": "ASSISTS.SSE_AVX_MIX",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5,6,7",
"EventCode": "0xc1",
"EventName": "ASSISTS.SSE_AVX_MIX",
- "PEBScounters": "0,1,2,3,4,5,6,7",
"SampleAfterValue": "1000003",
- "Speculative": "1",
"UMask": "0x10",
"Unit": "cpu_core"
},
{
"BriefDescription": "FP_ARITH_DISPATCHED.PORT_0",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5,6,7",
"EventCode": "0xb3",
"EventName": "FP_ARITH_DISPATCHED.PORT_0",
- "PEBScounters": "0,1,2,3,4,5,6,7",
"SampleAfterValue": "2000003",
- "Speculative": "1",
"UMask": "0x1",
"Unit": "cpu_core"
},
{
"BriefDescription": "FP_ARITH_DISPATCHED.PORT_1",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5,6,7",
"EventCode": "0xb3",
"EventName": "FP_ARITH_DISPATCHED.PORT_1",
- "PEBScounters": "0,1,2,3,4,5,6,7",
"SampleAfterValue": "2000003",
- "Speculative": "1",
"UMask": "0x2",
"Unit": "cpu_core"
},
{
"BriefDescription": "FP_ARITH_DISPATCHED.PORT_5",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5,6,7",
"EventCode": "0xb3",
"EventName": "FP_ARITH_DISPATCHED.PORT_5",
- "PEBScounters": "0,1,2,3,4,5,6,7",
"SampleAfterValue": "2000003",
- "Speculative": "1",
"UMask": "0x4",
"Unit": "cpu_core"
},
{
"BriefDescription": "Counts number of SSE/AVX computational 128-bit packed double precision floating-point instructions retired; some instructions will count twice as noted below. Each count represents 2 computation operations, one for each element. Applies to SSE* and AVX* packed double precision floating-point instructions: ADD SUB HADD HSUB SUBADD MUL DIV MIN MAX SQRT DPP FM(N)ADD/SUB. DPP and FM(N)ADD/SUB instructions count twice as they perform 2 calculations per element.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5,6,7",
"EventCode": "0xc7",
"EventName": "FP_ARITH_INST_RETIRED.128B_PACKED_DOUBLE",
- "PEBScounters": "0,1,2,3,4,5,6,7",
+ "PublicDescription": "Number of SSE/AVX computational 128-bit packed double precision floating-point instructions retired; some instructions will count twice as noted below. Each count represents 2 computation operations, one for each element. Applies to SSE* and AVX* packed double precision floating-point instructions: ADD SUB HADD HSUB SUBADD MUL DIV MIN MAX SQRT DPP FM(N)ADD/SUB. DPP and FM(N)ADD/SUB instructions count twice as they perform 2 calculations per element. The DAZ and FTZ flags in the MXCSR register need to be set when using these events.",
"SampleAfterValue": "100003",
"UMask": "0x4",
"Unit": "cpu_core"
},
{
"BriefDescription": "Number of SSE/AVX computational 128-bit packed single precision floating-point instructions retired; some instructions will count twice as noted below. Each count represents 4 computation operations, one for each element. Applies to SSE* and AVX* packed single precision floating-point instructions: ADD SUB MUL DIV MIN MAX RCP14 RSQRT14 SQRT DPP FM(N)ADD/SUB. DPP and FM(N)ADD/SUB instructions count twice as they perform 2 calculations per element.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5,6,7",
"EventCode": "0xc7",
"EventName": "FP_ARITH_INST_RETIRED.128B_PACKED_SINGLE",
- "PEBScounters": "0,1,2,3,4,5,6,7",
+ "PublicDescription": "Number of SSE/AVX computational 128-bit packed single precision floating-point instructions retired; some instructions will count twice as noted below. Each count represents 4 computation operations, one for each element. Applies to SSE* and AVX* packed single precision floating-point instructions: ADD SUB HADD HSUB SUBADD MUL DIV MIN MAX SQRT RSQRT RCP DPP FM(N)ADD/SUB. DPP and FM(N)ADD/SUB instructions count twice as they perform 2 calculations per element. The DAZ and FTZ flags in the MXCSR register need to be set when using these events.",
"SampleAfterValue": "100003",
"UMask": "0x8",
"Unit": "cpu_core"
},
{
"BriefDescription": "Counts number of SSE/AVX computational 256-bit packed double precision floating-point instructions retired; some instructions will count twice as noted below. Each count represents 4 computation operations, one for each element. Applies to SSE* and AVX* packed double precision floating-point instructions: ADD SUB HADD HSUB SUBADD MUL DIV MIN MAX SQRT FM(N)ADD/SUB. FM(N)ADD/SUB instructions count twice as they perform 2 calculations per element.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5,6,7",
"EventCode": "0xc7",
"EventName": "FP_ARITH_INST_RETIRED.256B_PACKED_DOUBLE",
- "PEBScounters": "0,1,2,3,4,5,6,7",
+ "PublicDescription": "Number of SSE/AVX computational 256-bit packed double precision floating-point instructions retired; some instructions will count twice as noted below. Each count represents 4 computation operations, one for each element. Applies to SSE* and AVX* packed double precision floating-point instructions: ADD SUB HADD HSUB SUBADD MUL DIV MIN MAX SQRT FM(N)ADD/SUB. FM(N)ADD/SUB instructions count twice as they perform 2 calculations per element. The DAZ and FTZ flags in the MXCSR register need to be set when using these events.",
"SampleAfterValue": "100003",
"UMask": "0x10",
"Unit": "cpu_core"
},
{
"BriefDescription": "Counts number of SSE/AVX computational 256-bit packed single precision floating-point instructions retired; some instructions will count twice as noted below. Each count represents 8 computation operations, one for each element. Applies to SSE* and AVX* packed single precision floating-point instructions: ADD SUB HADD HSUB SUBADD MUL DIV MIN MAX SQRT RSQRT RCP DPP FM(N)ADD/SUB. DPP and FM(N)ADD/SUB instructions count twice as they perform 2 calculations per element.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5,6,7",
"EventCode": "0xc7",
"EventName": "FP_ARITH_INST_RETIRED.256B_PACKED_SINGLE",
- "PEBScounters": "0,1,2,3,4,5,6,7",
+ "PublicDescription": "Number of SSE/AVX computational 256-bit packed single precision floating-point instructions retired; some instructions will count twice as noted below. Each count represents 8 computation operations, one for each element. Applies to SSE* and AVX* packed single precision floating-point instructions: ADD SUB HADD HSUB SUBADD MUL DIV MIN MAX SQRT RSQRT RCP DPP FM(N)ADD/SUB. DPP and FM(N)ADD/SUB instructions count twice as they perform 2 calculations per element. The DAZ and FTZ flags in the MXCSR register need to be set when using these events.",
"SampleAfterValue": "100003",
"UMask": "0x20",
"Unit": "cpu_core"
},
{
"BriefDescription": "Counts number of SSE/AVX computational scalar double precision floating-point instructions retired; some instructions will count twice as noted below. Each count represents 1 computational operation. Applies to SSE* and AVX* scalar double precision floating-point instructions: ADD SUB MUL DIV MIN MAX SQRT FM(N)ADD/SUB. FM(N)ADD/SUB instructions count twice as they perform 2 calculations per element.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5,6,7",
"EventCode": "0xc7",
"EventName": "FP_ARITH_INST_RETIRED.SCALAR_DOUBLE",
- "PEBScounters": "0,1,2,3,4,5,6,7",
+ "PublicDescription": "Number of SSE/AVX computational scalar double precision floating-point instructions retired; some instructions will count twice as noted below. Each count represents 1 computational operation. Applies to SSE* and AVX* scalar double precision floating-point instructions: ADD SUB MUL DIV MIN MAX SQRT FM(N)ADD/SUB. FM(N)ADD/SUB instructions count twice as they perform 2 calculations per element. The DAZ and FTZ flags in the MXCSR register need to be set when using these events.",
"SampleAfterValue": "100003",
"UMask": "0x1",
"Unit": "cpu_core"
},
{
"BriefDescription": "Counts number of SSE/AVX computational scalar single precision floating-point instructions retired; some instructions will count twice as noted below. Each count represents 1 computational operation. Applies to SSE* and AVX* scalar single precision floating-point instructions: ADD SUB MUL DIV MIN MAX SQRT RSQRT RCP FM(N)ADD/SUB. FM(N)ADD/SUB instructions count twice as they perform 2 calculations per element.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5,6,7",
"EventCode": "0xc7",
"EventName": "FP_ARITH_INST_RETIRED.SCALAR_SINGLE",
- "PEBScounters": "0,1,2,3,4,5,6,7",
+ "PublicDescription": "Number of SSE/AVX computational scalar single precision floating-point instructions retired; some instructions will count twice as noted below. Each count represents 1 computational operation. Applies to SSE* and AVX* scalar single precision floating-point instructions: ADD SUB MUL DIV MIN MAX SQRT RSQRT RCP FM(N)ADD/SUB. FM(N)ADD/SUB instructions count twice as they perform 2 calculations per element. The DAZ and FTZ flags in the MXCSR register need to be set when using these events.",
"SampleAfterValue": "100003",
"UMask": "0x2",
"Unit": "cpu_core"
+ },
+ {
+ "BriefDescription": "Counts the number of floating point operations retired that required microcode assist.",
+ "EventCode": "0xc3",
+ "EventName": "MACHINE_CLEARS.FP_ASSIST",
+ "PublicDescription": "Counts the number of floating point operations retired that required microcode assist, which is not a reflection of the number of FP operations, instructions or uops.",
+ "SampleAfterValue": "20003",
+ "UMask": "0x4",
+ "Unit": "cpu_atom"
+ },
+ {
+ "BriefDescription": "Counts the number of floating point divide uops retired (x87 and SSE, including x87 sqrt).",
+ "EventCode": "0xc2",
+ "EventName": "UOPS_RETIRED.FPDIV",
+ "PEBS": "1",
+ "SampleAfterValue": "2000003",
+ "UMask": "0x8",
+ "Unit": "cpu_atom"
}
]
diff --git a/tools/perf/pmu-events/arch/x86/alderlake/frontend.json b/tools/perf/pmu-events/arch/x86/alderlake/frontend.json
index da1a7ba..250cd12 100644
--- a/tools/perf/pmu-events/arch/x86/alderlake/frontend.json
+++ b/tools/perf/pmu-events/arch/x86/alderlake/frontend.json
@@ -1,536 +1,416 @@
[
{
"BriefDescription": "Counts the total number of BACLEARS due to all branch types including conditional and unconditional jumps, returns, and indirect branches.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5",
"EventCode": "0xe6",
"EventName": "BACLEARS.ANY",
- "PEBScounters": "0,1,2,3,4,5",
+ "PublicDescription": "Counts the total number of BACLEARS, which occur when the Branch Target Buffer (BTB) prediction or lack thereof, was corrected by a later branch predictor in the frontend. Includes BACLEARS due to all branch types including conditional and unconditional jumps, returns, and indirect branches.",
"SampleAfterValue": "100003",
- "Speculative": "1",
"UMask": "0x1",
"Unit": "cpu_atom"
},
{
- "BriefDescription": "Counts the number of requests to the instruction cache for one or more bytes of a cache line.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5",
- "EventCode": "0x80",
- "EventName": "ICACHE.ACCESSES",
- "PEBScounters": "0,1,2,3,4,5",
- "SampleAfterValue": "200003",
- "Speculative": "1",
- "UMask": "0x3",
- "Unit": "cpu_atom"
- },
- {
- "BriefDescription": "Counts the number of instruction cache misses.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5",
- "EventCode": "0x80",
- "EventName": "ICACHE.MISSES",
- "PEBScounters": "0,1,2,3,4,5",
- "SampleAfterValue": "200003",
- "Speculative": "1",
- "UMask": "0x2",
- "Unit": "cpu_atom"
- },
- {
"BriefDescription": "Stalls caused by changing prefix length of the instruction.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3",
"EventCode": "0x87",
"EventName": "DECODE.LCP",
- "PEBScounters": "0,1,2,3",
+ "PublicDescription": "Counts cycles that the Instruction Length decoder (ILD) stalls occurred due to dynamically changing prefix length of the decoded instruction (by operand size prefix instruction 0x66, address size prefix instruction 0x67 or REX.W for Intel64). Count is proportional to the number of prefixes in a 16B-line. This may result in a three-cycle penalty for each LCP (Length changing prefix) in a 16-byte chunk.",
"SampleAfterValue": "500009",
- "Speculative": "1",
"UMask": "0x1",
"Unit": "cpu_core"
},
{
"BriefDescription": "Cycles the Microcode Sequencer is busy.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3",
"EventCode": "0x87",
"EventName": "DECODE.MS_BUSY",
- "PEBScounters": "0,1,2,3",
"SampleAfterValue": "500009",
- "Speculative": "1",
"UMask": "0x2",
"Unit": "cpu_core"
},
{
"BriefDescription": "DSB-to-MITE switch true penalty cycles.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3",
"EventCode": "0x61",
"EventName": "DSB2MITE_SWITCHES.PENALTY_CYCLES",
- "PEBScounters": "0,1,2,3",
+ "PublicDescription": "Decode Stream Buffer (DSB) is a Uop-cache that holds translations of previously fetched instructions that were decoded by the legacy x86 decode pipeline (MITE). This event counts fetch penalty cycles when a transition occurs from DSB to MITE.",
"SampleAfterValue": "100003",
- "Speculative": "1",
"UMask": "0x2",
"Unit": "cpu_core"
},
{
"BriefDescription": "Retired Instructions who experienced DSB miss.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5,6,7",
"EventCode": "0xc6",
"EventName": "FRONTEND_RETIRED.ANY_DSB_MISS",
"MSRIndex": "0x3F7",
"MSRValue": "0x1",
"PEBS": "1",
- "PEBScounters": "0,1,2,3,4,5,6,7",
+ "PublicDescription": "Counts retired Instructions that experienced DSB (Decode stream buffer i.e. the decoded instruction-cache) miss.",
"SampleAfterValue": "100007",
- "TakenAlone": "1",
"UMask": "0x1",
"Unit": "cpu_core"
},
{
"BriefDescription": "Retired Instructions who experienced a critical DSB miss.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5,6,7",
"EventCode": "0xc6",
"EventName": "FRONTEND_RETIRED.DSB_MISS",
"MSRIndex": "0x3F7",
"MSRValue": "0x11",
"PEBS": "1",
- "PEBScounters": "0,1,2,3,4,5,6,7",
+ "PublicDescription": "Number of retired Instructions that experienced a critical DSB (Decode stream buffer i.e. the decoded instruction-cache) miss. Critical means stalls were exposed to the back-end as a result of the DSB miss.",
"SampleAfterValue": "100007",
- "TakenAlone": "1",
"UMask": "0x1",
"Unit": "cpu_core"
},
{
"BriefDescription": "Retired Instructions who experienced iTLB true miss.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5,6,7",
"EventCode": "0xc6",
"EventName": "FRONTEND_RETIRED.ITLB_MISS",
"MSRIndex": "0x3F7",
"MSRValue": "0x14",
"PEBS": "1",
- "PEBScounters": "0,1,2,3,4,5,6,7",
+ "PublicDescription": "Counts retired Instructions that experienced iTLB (Instruction TLB) true miss.",
"SampleAfterValue": "100007",
- "TakenAlone": "1",
"UMask": "0x1",
"Unit": "cpu_core"
},
{
"BriefDescription": "Retired Instructions who experienced Instruction L1 Cache true miss.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5,6,7",
"EventCode": "0xc6",
"EventName": "FRONTEND_RETIRED.L1I_MISS",
"MSRIndex": "0x3F7",
"MSRValue": "0x12",
"PEBS": "1",
- "PEBScounters": "0,1,2,3,4,5,6,7",
+ "PublicDescription": "Counts retired Instructions who experienced Instruction L1 Cache true miss.",
"SampleAfterValue": "100007",
- "TakenAlone": "1",
"UMask": "0x1",
"Unit": "cpu_core"
},
{
"BriefDescription": "Retired Instructions who experienced Instruction L2 Cache true miss.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5,6,7",
"EventCode": "0xc6",
"EventName": "FRONTEND_RETIRED.L2_MISS",
"MSRIndex": "0x3F7",
"MSRValue": "0x13",
"PEBS": "1",
- "PEBScounters": "0,1,2,3,4,5,6,7",
+ "PublicDescription": "Counts retired Instructions who experienced Instruction L2 Cache true miss.",
"SampleAfterValue": "100007",
- "TakenAlone": "1",
"UMask": "0x1",
"Unit": "cpu_core"
},
{
"BriefDescription": "Retired instructions after front-end starvation of at least 1 cycle",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5,6,7",
"EventCode": "0xc6",
"EventName": "FRONTEND_RETIRED.LATENCY_GE_1",
"MSRIndex": "0x3F7",
"MSRValue": "0x600106",
"PEBS": "1",
- "PEBScounters": "0,1,2,3,4,5,6,7",
+ "PublicDescription": "Retired instructions that are fetched after an interval where the front-end delivered no uops for a period of at least 1 cycle which was not interrupted by a back-end stall.",
"SampleAfterValue": "100007",
- "TakenAlone": "1",
"UMask": "0x1",
"Unit": "cpu_core"
},
{
"BriefDescription": "Retired instructions that are fetched after an interval where the front-end delivered no uops for a period of 128 cycles which was not interrupted by a back-end stall.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5,6,7",
"EventCode": "0xc6",
"EventName": "FRONTEND_RETIRED.LATENCY_GE_128",
"MSRIndex": "0x3F7",
"MSRValue": "0x608006",
"PEBS": "1",
- "PEBScounters": "0,1,2,3,4,5,6,7",
+ "PublicDescription": "Counts retired instructions that are fetched after an interval where the front-end delivered no uops for a period of 128 cycles which was not interrupted by a back-end stall.",
"SampleAfterValue": "100007",
- "TakenAlone": "1",
"UMask": "0x1",
"Unit": "cpu_core"
},
{
"BriefDescription": "Retired instructions that are fetched after an interval where the front-end delivered no uops for a period of 16 cycles which was not interrupted by a back-end stall.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5,6,7",
"EventCode": "0xc6",
"EventName": "FRONTEND_RETIRED.LATENCY_GE_16",
"MSRIndex": "0x3F7",
"MSRValue": "0x601006",
"PEBS": "1",
- "PEBScounters": "0,1,2,3,4,5,6,7",
+ "PublicDescription": "Counts retired instructions that are delivered to the back-end after a front-end stall of at least 16 cycles. During this period the front-end delivered no uops.",
"SampleAfterValue": "100007",
- "TakenAlone": "1",
"UMask": "0x1",
"Unit": "cpu_core"
},
{
"BriefDescription": "Retired instructions after front-end starvation of at least 2 cycles",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5,6,7",
"EventCode": "0xc6",
"EventName": "FRONTEND_RETIRED.LATENCY_GE_2",
"MSRIndex": "0x3F7",
"MSRValue": "0x600206",
"PEBS": "1",
- "PEBScounters": "0,1,2,3,4,5,6,7",
+ "PublicDescription": "Retired instructions that are fetched after an interval where the front-end delivered no uops for a period of at least 2 cycles which was not interrupted by a back-end stall.",
"SampleAfterValue": "100007",
- "TakenAlone": "1",
"UMask": "0x1",
"Unit": "cpu_core"
},
{
"BriefDescription": "Retired instructions that are fetched after an interval where the front-end delivered no uops for a period of 256 cycles which was not interrupted by a back-end stall.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5,6,7",
"EventCode": "0xc6",
"EventName": "FRONTEND_RETIRED.LATENCY_GE_256",
"MSRIndex": "0x3F7",
"MSRValue": "0x610006",
"PEBS": "1",
- "PEBScounters": "0,1,2,3,4,5,6,7",
+ "PublicDescription": "Counts retired instructions that are fetched after an interval where the front-end delivered no uops for a period of 256 cycles which was not interrupted by a back-end stall.",
"SampleAfterValue": "100007",
- "TakenAlone": "1",
"UMask": "0x1",
"Unit": "cpu_core"
},
{
"BriefDescription": "Retired instructions that are fetched after an interval where the front-end had at least 1 bubble-slot for a period of 2 cycles which was not interrupted by a back-end stall.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5,6,7",
"EventCode": "0xc6",
"EventName": "FRONTEND_RETIRED.LATENCY_GE_2_BUBBLES_GE_1",
"MSRIndex": "0x3F7",
"MSRValue": "0x100206",
"PEBS": "1",
- "PEBScounters": "0,1,2,3,4,5,6,7",
+ "PublicDescription": "Counts retired instructions that are delivered to the back-end after the front-end had at least 1 bubble-slot for a period of 2 cycles. A bubble-slot is an empty issue-pipeline slot while there was no RAT stall.",
"SampleAfterValue": "100007",
- "TakenAlone": "1",
"UMask": "0x1",
"Unit": "cpu_core"
},
{
"BriefDescription": "Retired instructions that are fetched after an interval where the front-end delivered no uops for a period of 32 cycles which was not interrupted by a back-end stall.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5,6,7",
"EventCode": "0xc6",
"EventName": "FRONTEND_RETIRED.LATENCY_GE_32",
"MSRIndex": "0x3F7",
"MSRValue": "0x602006",
"PEBS": "1",
- "PEBScounters": "0,1,2,3,4,5,6,7",
+ "PublicDescription": "Counts retired instructions that are delivered to the back-end after a front-end stall of at least 32 cycles. During this period the front-end delivered no uops.",
"SampleAfterValue": "100007",
- "TakenAlone": "1",
"UMask": "0x1",
"Unit": "cpu_core"
},
{
"BriefDescription": "Retired instructions that are fetched after an interval where the front-end delivered no uops for a period of 4 cycles which was not interrupted by a back-end stall.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5,6,7",
"EventCode": "0xc6",
"EventName": "FRONTEND_RETIRED.LATENCY_GE_4",
"MSRIndex": "0x3F7",
"MSRValue": "0x600406",
"PEBS": "1",
- "PEBScounters": "0,1,2,3,4,5,6,7",
+ "PublicDescription": "Counts retired instructions that are fetched after an interval where the front-end delivered no uops for a period of 4 cycles which was not interrupted by a back-end stall.",
"SampleAfterValue": "100007",
- "TakenAlone": "1",
"UMask": "0x1",
"Unit": "cpu_core"
},
{
"BriefDescription": "Retired instructions that are fetched after an interval where the front-end delivered no uops for a period of 512 cycles which was not interrupted by a back-end stall.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5,6,7",
"EventCode": "0xc6",
"EventName": "FRONTEND_RETIRED.LATENCY_GE_512",
"MSRIndex": "0x3F7",
"MSRValue": "0x620006",
"PEBS": "1",
- "PEBScounters": "0,1,2,3,4,5,6,7",
+ "PublicDescription": "Counts retired instructions that are fetched after an interval where the front-end delivered no uops for a period of 512 cycles which was not interrupted by a back-end stall.",
"SampleAfterValue": "100007",
- "TakenAlone": "1",
"UMask": "0x1",
"Unit": "cpu_core"
},
{
"BriefDescription": "Retired instructions that are fetched after an interval where the front-end delivered no uops for a period of 64 cycles which was not interrupted by a back-end stall.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5,6,7",
"EventCode": "0xc6",
"EventName": "FRONTEND_RETIRED.LATENCY_GE_64",
"MSRIndex": "0x3F7",
"MSRValue": "0x604006",
"PEBS": "1",
- "PEBScounters": "0,1,2,3,4,5,6,7",
+ "PublicDescription": "Counts retired instructions that are fetched after an interval where the front-end delivered no uops for a period of 64 cycles which was not interrupted by a back-end stall.",
"SampleAfterValue": "100007",
- "TakenAlone": "1",
"UMask": "0x1",
"Unit": "cpu_core"
},
{
"BriefDescription": "Retired instructions that are fetched after an interval where the front-end delivered no uops for a period of 8 cycles which was not interrupted by a back-end stall.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5,6,7",
"EventCode": "0xc6",
"EventName": "FRONTEND_RETIRED.LATENCY_GE_8",
"MSRIndex": "0x3F7",
"MSRValue": "0x600806",
"PEBS": "1",
- "PEBScounters": "0,1,2,3,4,5,6,7",
+ "PublicDescription": "Counts retired instructions that are delivered to the back-end after a front-end stall of at least 8 cycles. During this period the front-end delivered no uops.",
"SampleAfterValue": "100007",
- "TakenAlone": "1",
"UMask": "0x1",
"Unit": "cpu_core"
},
{
"BriefDescription": "FRONTEND_RETIRED.MS_FLOWS",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5,6,7",
"EventCode": "0xc6",
"EventName": "FRONTEND_RETIRED.MS_FLOWS",
"MSRIndex": "0x3F7",
"MSRValue": "0x8",
"PEBS": "1",
- "PEBScounters": "0,1,2,3,4,5,6,7",
"SampleAfterValue": "100007",
- "TakenAlone": "1",
"UMask": "0x1",
"Unit": "cpu_core"
},
{
"BriefDescription": "Retired Instructions who experienced STLB (2nd level TLB) true miss.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5,6,7",
"EventCode": "0xc6",
"EventName": "FRONTEND_RETIRED.STLB_MISS",
"MSRIndex": "0x3F7",
"MSRValue": "0x15",
"PEBS": "1",
- "PEBScounters": "0,1,2,3,4,5,6,7",
+ "PublicDescription": "Counts retired Instructions that experienced STLB (2nd level TLB) true miss.",
"SampleAfterValue": "100007",
- "TakenAlone": "1",
"UMask": "0x1",
"Unit": "cpu_core"
},
{
"BriefDescription": "FRONTEND_RETIRED.UNKNOWN_BRANCH",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5,6,7",
"EventCode": "0xc6",
"EventName": "FRONTEND_RETIRED.UNKNOWN_BRANCH",
"MSRIndex": "0x3F7",
"MSRValue": "0x17",
"PEBS": "1",
- "PEBScounters": "0,1,2,3,4,5,6,7",
"SampleAfterValue": "100007",
- "TakenAlone": "1",
"UMask": "0x1",
"Unit": "cpu_core"
},
{
+ "BriefDescription": "Counts the number of requests to the instruction cache for one or more bytes of a cache line.",
+ "EventCode": "0x80",
+ "EventName": "ICACHE.ACCESSES",
+ "PublicDescription": "Counts the total number of requests to the instruction cache. The event only counts new cache line accesses, so that multiple back to back fetches to the exact same cache line or byte chunk count as one. Specifically, the event counts when accesses from sequential code crosses the cache line boundary, or when a branch target is moved to a new line or to a non-sequential byte chunk of the same line.",
+ "SampleAfterValue": "200003",
+ "UMask": "0x3",
+ "Unit": "cpu_atom"
+ },
+ {
+ "BriefDescription": "Counts the number of instruction cache misses.",
+ "EventCode": "0x80",
+ "EventName": "ICACHE.MISSES",
+ "PublicDescription": "Counts the number of missed requests to the instruction cache. The event only counts new cache line accesses, so that multiple back to back fetches to the exact same cache line and byte chunk count as one. Specifically, the event counts when accesses from sequential code crosses the cache line boundary, or when a branch target is moved to a new line or to a non-sequential byte chunk of the same line.",
+ "SampleAfterValue": "200003",
+ "UMask": "0x2",
+ "Unit": "cpu_atom"
+ },
+ {
"BriefDescription": "Cycles where a code fetch is stalled due to L1 instruction cache miss.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3",
"EventCode": "0x80",
"EventName": "ICACHE_DATA.STALLS",
- "PEBScounters": "0,1,2,3",
+ "PublicDescription": "Counts cycles where a code line fetch is stalled due to an L1 instruction cache miss. The decode pipeline works at a 32 Byte granularity.",
"SampleAfterValue": "500009",
- "Speculative": "1",
"UMask": "0x4",
"Unit": "cpu_core"
},
{
"BriefDescription": "Cycles where a code fetch is stalled due to L1 instruction cache tag miss.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3",
"EventCode": "0x83",
"EventName": "ICACHE_TAG.STALLS",
- "PEBScounters": "0,1,2,3",
+ "PublicDescription": "Counts cycles where a code fetch is stalled due to L1 instruction cache tag miss.",
"SampleAfterValue": "200003",
- "Speculative": "1",
"UMask": "0x4",
"Unit": "cpu_core"
},
{
"BriefDescription": "Cycles Decode Stream Buffer (DSB) is delivering any Uop",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3",
"CounterMask": "1",
"EventCode": "0x79",
"EventName": "IDQ.DSB_CYCLES_ANY",
- "PEBScounters": "0,1,2,3",
+ "PublicDescription": "Counts the number of cycles uops were delivered to Instruction Decode Queue (IDQ) from the Decode Stream Buffer (DSB) path.",
"SampleAfterValue": "2000003",
- "Speculative": "1",
"UMask": "0x8",
"Unit": "cpu_core"
},
{
"BriefDescription": "Cycles DSB is delivering optimal number of Uops",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3",
"CounterMask": "6",
"EventCode": "0x79",
"EventName": "IDQ.DSB_CYCLES_OK",
- "PEBScounters": "0,1,2,3",
+ "PublicDescription": "Counts the number of cycles where optimal number of uops was delivered to the Instruction Decode Queue (IDQ) from the MITE (legacy decode pipeline) path. During these cycles uops are not being delivered from the Decode Stream Buffer (DSB).",
"SampleAfterValue": "2000003",
- "Speculative": "1",
"UMask": "0x8",
"Unit": "cpu_core"
},
{
"BriefDescription": "Uops delivered to Instruction Decode Queue (IDQ) from the Decode Stream Buffer (DSB) path",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3",
"EventCode": "0x79",
"EventName": "IDQ.DSB_UOPS",
- "PEBScounters": "0,1,2,3",
+ "PublicDescription": "Counts the number of uops delivered to Instruction Decode Queue (IDQ) from the Decode Stream Buffer (DSB) path.",
"SampleAfterValue": "2000003",
- "Speculative": "1",
"UMask": "0x8",
"Unit": "cpu_core"
},
{
"BriefDescription": "Cycles MITE is delivering any Uop",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3",
"CounterMask": "1",
"EventCode": "0x79",
"EventName": "IDQ.MITE_CYCLES_ANY",
- "PEBScounters": "0,1,2,3",
+ "PublicDescription": "Counts the number of cycles uops were delivered to the Instruction Decode Queue (IDQ) from the MITE (legacy decode pipeline) path. During these cycles uops are not being delivered from the Decode Stream Buffer (DSB).",
"SampleAfterValue": "2000003",
- "Speculative": "1",
"UMask": "0x4",
"Unit": "cpu_core"
},
{
"BriefDescription": "Cycles MITE is delivering optimal number of Uops",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3",
"CounterMask": "6",
"EventCode": "0x79",
"EventName": "IDQ.MITE_CYCLES_OK",
- "PEBScounters": "0,1,2,3",
+ "PublicDescription": "Counts the number of cycles where optimal number of uops was delivered to the Instruction Decode Queue (IDQ) from the MITE (legacy decode pipeline) path. During these cycles uops are not being delivered from the Decode Stream Buffer (DSB).",
"SampleAfterValue": "2000003",
- "Speculative": "1",
"UMask": "0x4",
"Unit": "cpu_core"
},
{
"BriefDescription": "Uops delivered to Instruction Decode Queue (IDQ) from MITE path",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3",
"EventCode": "0x79",
"EventName": "IDQ.MITE_UOPS",
- "PEBScounters": "0,1,2,3",
+ "PublicDescription": "Counts the number of uops delivered to Instruction Decode Queue (IDQ) from the MITE path. This also means that uops are not being delivered from the Decode Stream Buffer (DSB).",
"SampleAfterValue": "2000003",
- "Speculative": "1",
"UMask": "0x4",
"Unit": "cpu_core"
},
{
"BriefDescription": "Cycles when uops are being delivered to IDQ while MS is busy",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3",
"CounterMask": "1",
"EventCode": "0x79",
"EventName": "IDQ.MS_CYCLES_ANY",
- "PEBScounters": "0,1,2,3",
+ "PublicDescription": "Counts cycles during which uops are being delivered to Instruction Decode Queue (IDQ) while the Microcode Sequencer (MS) is busy. Uops maybe initiated by Decode Stream Buffer (DSB) or MITE.",
"SampleAfterValue": "2000003",
- "Speculative": "1",
"UMask": "0x20",
"Unit": "cpu_core"
},
{
"BriefDescription": "Number of switches from DSB or MITE to the MS",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3",
"CounterMask": "1",
"EdgeDetect": "1",
"EventCode": "0x79",
"EventName": "IDQ.MS_SWITCHES",
- "PEBScounters": "0,1,2,3",
+ "PublicDescription": "Number of switches from DSB (Decode Stream Buffer) or MITE (legacy decode pipeline) to the Microcode Sequencer.",
"SampleAfterValue": "100003",
- "Speculative": "1",
"UMask": "0x20",
"Unit": "cpu_core"
},
{
"BriefDescription": "Uops delivered to IDQ while MS is busy",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3",
"EventCode": "0x79",
"EventName": "IDQ.MS_UOPS",
- "PEBScounters": "0,1,2,3",
+ "PublicDescription": "Counts the total number of uops delivered by the Microcode Sequencer (MS).",
"SampleAfterValue": "1000003",
- "Speculative": "1",
"UMask": "0x20",
"Unit": "cpu_core"
},
{
"BriefDescription": "Uops not delivered by IDQ when backend of the machine is not stalled",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5,6,7",
"EventCode": "0x9c",
"EventName": "IDQ_UOPS_NOT_DELIVERED.CORE",
- "PEBScounters": "0,1,2,3,4,5,6,7",
+ "PublicDescription": "Counts the number of uops not delivered to by the Instruction Decode Queue (IDQ) to the back-end of the pipeline when there was no back-end stalls. This event counts for one SMT thread in a given cycle.",
"SampleAfterValue": "1000003",
- "Speculative": "1",
"UMask": "0x1",
"Unit": "cpu_core"
},
{
"BriefDescription": "Cycles when no uops are not delivered by the IDQ when backend of the machine is not stalled",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5,6,7",
"CounterMask": "6",
"EventCode": "0x9c",
"EventName": "IDQ_UOPS_NOT_DELIVERED.CYCLES_0_UOPS_DELIV.CORE",
- "PEBScounters": "0,1,2,3,4,5,6,7",
+ "PublicDescription": "Counts the number of cycles when no uops were delivered by the Instruction Decode Queue (IDQ) to the back-end of the pipeline when there was no back-end stalls. This event counts for one SMT thread in a given cycle.",
"SampleAfterValue": "1000003",
- "Speculative": "1",
"UMask": "0x1",
"Unit": "cpu_core"
},
{
"BriefDescription": "Cycles when optimal number of uops was delivered to the back-end when the back-end is not stalled",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5,6,7",
"CounterMask": "1",
"EventCode": "0x9c",
"EventName": "IDQ_UOPS_NOT_DELIVERED.CYCLES_FE_WAS_OK",
"Invert": "1",
- "PEBScounters": "0,1,2,3,4,5,6,7",
+ "PublicDescription": "Counts the number of cycles when the optimal number of uops were delivered by the Instruction Decode Queue (IDQ) to the back-end of the pipeline when there was no back-end stalls. This event counts for one SMT thread in a given cycle.",
"SampleAfterValue": "1000003",
- "Speculative": "1",
"UMask": "0x1",
"Unit": "cpu_core"
}
diff --git a/tools/perf/pmu-events/arch/x86/alderlake/memory.json b/tools/perf/pmu-events/arch/x86/alderlake/memory.json
index f894e4a..7595eb4 100644
--- a/tools/perf/pmu-events/arch/x86/alderlake/memory.json
+++ b/tools/perf/pmu-events/arch/x86/alderlake/memory.json
@@ -1,79 +1,224 @@
[
{
+ "BriefDescription": "Execution stalls while L3 cache miss demand load is outstanding.",
+ "CounterMask": "6",
+ "EventCode": "0xa3",
+ "EventName": "CYCLE_ACTIVITY.STALLS_L3_MISS",
+ "SampleAfterValue": "1000003",
+ "UMask": "0x6",
+ "Unit": "cpu_core"
+ },
+ {
"BriefDescription": "Counts the number of cycles that the head (oldest load) of the load buffer is stalled due to any number of reasons, including an L1 miss, WCB full, pagewalk, store address block or store data block, on a load that retires.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5",
"EventCode": "0x05",
"EventName": "LD_HEAD.ANY_AT_RET",
- "PEBScounters": "0,1,2,3,4,5",
"SampleAfterValue": "1000003",
- "Speculative": "1",
"UMask": "0xff",
"Unit": "cpu_atom"
},
{
"BriefDescription": "Counts the number of cycles that the head (oldest load) of the load buffer is stalled due to a core bound stall including a store address match, a DTLB miss or a page walk that detains the load from retiring.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5",
"EventCode": "0x05",
"EventName": "LD_HEAD.L1_BOUND_AT_RET",
- "PEBScounters": "0,1,2,3,4,5",
"SampleAfterValue": "1000003",
- "Speculative": "1",
"UMask": "0xf4",
"Unit": "cpu_atom"
},
{
"BriefDescription": "Counts the number of cycles that the head (oldest load) of the load buffer and retirement are both stalled due to other block cases.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5",
"EventCode": "0x05",
"EventName": "LD_HEAD.OTHER_AT_RET",
- "PEBScounters": "0,1,2,3,4,5",
+ "PublicDescription": "Counts the number of cycles that the head (oldest load) of the load buffer and retirement are both stalled due to other block cases such as pipeline conflicts, fences, etc.",
"SampleAfterValue": "1000003",
- "Speculative": "1",
"UMask": "0xc0",
"Unit": "cpu_atom"
},
{
"BriefDescription": "Counts the number of cycles that the head (oldest load) of the load buffer and retirement are both stalled due to a pagewalk.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5",
"EventCode": "0x05",
"EventName": "LD_HEAD.PGWALK_AT_RET",
- "PEBScounters": "0,1,2,3,4,5",
"SampleAfterValue": "1000003",
- "Speculative": "1",
"UMask": "0xa0",
"Unit": "cpu_atom"
},
{
"BriefDescription": "Counts the number of cycles that the head (oldest load) of the load buffer and retirement are both stalled due to a store address match.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5",
"EventCode": "0x05",
"EventName": "LD_HEAD.ST_ADDR_AT_RET",
- "PEBScounters": "0,1,2,3,4,5",
"SampleAfterValue": "1000003",
- "Speculative": "1",
"UMask": "0x84",
"Unit": "cpu_atom"
},
{
"BriefDescription": "Counts the number of machine clears due to memory ordering caused by a snoop from an external agent. Does not count internally generated machine clears such as those due to memory disambiguation.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5",
"EventCode": "0xc3",
"EventName": "MACHINE_CLEARS.MEMORY_ORDERING",
- "PEBScounters": "0,1,2,3,4,5",
"SampleAfterValue": "20003",
- "Speculative": "1",
"UMask": "0x2",
"Unit": "cpu_atom"
},
{
+ "BriefDescription": "Number of machine clears due to memory ordering conflicts.",
+ "EventCode": "0xc3",
+ "EventName": "MACHINE_CLEARS.MEMORY_ORDERING",
+ "PublicDescription": "Counts the number of Machine Clears detected dye to memory ordering. Memory Ordering Machine Clears may apply when a memory read may not conform to the memory ordering rules of the x86 architecture",
+ "SampleAfterValue": "100003",
+ "UMask": "0x2",
+ "Unit": "cpu_core"
+ },
+ {
+ "BriefDescription": "Cycles while L1 cache miss demand load is outstanding.",
+ "CounterMask": "2",
+ "EventCode": "0x47",
+ "EventName": "MEMORY_ACTIVITY.CYCLES_L1D_MISS",
+ "SampleAfterValue": "1000003",
+ "UMask": "0x2",
+ "Unit": "cpu_core"
+ },
+ {
+ "BriefDescription": "Execution stalls while L1 cache miss demand load is outstanding.",
+ "CounterMask": "3",
+ "EventCode": "0x47",
+ "EventName": "MEMORY_ACTIVITY.STALLS_L1D_MISS",
+ "SampleAfterValue": "1000003",
+ "UMask": "0x3",
+ "Unit": "cpu_core"
+ },
+ {
+ "BriefDescription": "MEMORY_ACTIVITY.STALLS_L2_MISS",
+ "CounterMask": "5",
+ "EventCode": "0x47",
+ "EventName": "MEMORY_ACTIVITY.STALLS_L2_MISS",
+ "SampleAfterValue": "1000003",
+ "UMask": "0x5",
+ "Unit": "cpu_core"
+ },
+ {
+ "BriefDescription": "MEMORY_ACTIVITY.STALLS_L3_MISS",
+ "CounterMask": "9",
+ "EventCode": "0x47",
+ "EventName": "MEMORY_ACTIVITY.STALLS_L3_MISS",
+ "SampleAfterValue": "1000003",
+ "UMask": "0x9",
+ "Unit": "cpu_core"
+ },
+ {
+ "BriefDescription": "Counts randomly selected loads when the latency from first dispatch to completion is greater than 128 cycles.",
+ "Data_LA": "1",
+ "EventCode": "0xcd",
+ "EventName": "MEM_TRANS_RETIRED.LOAD_LATENCY_GT_128",
+ "MSRIndex": "0x3F6",
+ "MSRValue": "0x80",
+ "PEBS": "2",
+ "PublicDescription": "Counts randomly selected loads when the latency from first dispatch to completion is greater than 128 cycles. Reported latency may be longer than just the memory latency.",
+ "SampleAfterValue": "1009",
+ "UMask": "0x1",
+ "Unit": "cpu_core"
+ },
+ {
+ "BriefDescription": "Counts randomly selected loads when the latency from first dispatch to completion is greater than 16 cycles.",
+ "Data_LA": "1",
+ "EventCode": "0xcd",
+ "EventName": "MEM_TRANS_RETIRED.LOAD_LATENCY_GT_16",
+ "MSRIndex": "0x3F6",
+ "MSRValue": "0x10",
+ "PEBS": "2",
+ "PublicDescription": "Counts randomly selected loads when the latency from first dispatch to completion is greater than 16 cycles. Reported latency may be longer than just the memory latency.",
+ "SampleAfterValue": "20011",
+ "UMask": "0x1",
+ "Unit": "cpu_core"
+ },
+ {
+ "BriefDescription": "Counts randomly selected loads when the latency from first dispatch to completion is greater than 256 cycles.",
+ "Data_LA": "1",
+ "EventCode": "0xcd",
+ "EventName": "MEM_TRANS_RETIRED.LOAD_LATENCY_GT_256",
+ "MSRIndex": "0x3F6",
+ "MSRValue": "0x100",
+ "PEBS": "2",
+ "PublicDescription": "Counts randomly selected loads when the latency from first dispatch to completion is greater than 256 cycles. Reported latency may be longer than just the memory latency.",
+ "SampleAfterValue": "503",
+ "UMask": "0x1",
+ "Unit": "cpu_core"
+ },
+ {
+ "BriefDescription": "Counts randomly selected loads when the latency from first dispatch to completion is greater than 32 cycles.",
+ "Data_LA": "1",
+ "EventCode": "0xcd",
+ "EventName": "MEM_TRANS_RETIRED.LOAD_LATENCY_GT_32",
+ "MSRIndex": "0x3F6",
+ "MSRValue": "0x20",
+ "PEBS": "2",
+ "PublicDescription": "Counts randomly selected loads when the latency from first dispatch to completion is greater than 32 cycles. Reported latency may be longer than just the memory latency.",
+ "SampleAfterValue": "100007",
+ "UMask": "0x1",
+ "Unit": "cpu_core"
+ },
+ {
+ "BriefDescription": "Counts randomly selected loads when the latency from first dispatch to completion is greater than 4 cycles.",
+ "Data_LA": "1",
+ "EventCode": "0xcd",
+ "EventName": "MEM_TRANS_RETIRED.LOAD_LATENCY_GT_4",
+ "MSRIndex": "0x3F6",
+ "MSRValue": "0x4",
+ "PEBS": "2",
+ "PublicDescription": "Counts randomly selected loads when the latency from first dispatch to completion is greater than 4 cycles. Reported latency may be longer than just the memory latency.",
+ "SampleAfterValue": "100003",
+ "UMask": "0x1",
+ "Unit": "cpu_core"
+ },
+ {
+ "BriefDescription": "Counts randomly selected loads when the latency from first dispatch to completion is greater than 512 cycles.",
+ "Data_LA": "1",
+ "EventCode": "0xcd",
+ "EventName": "MEM_TRANS_RETIRED.LOAD_LATENCY_GT_512",
+ "MSRIndex": "0x3F6",
+ "MSRValue": "0x200",
+ "PEBS": "2",
+ "PublicDescription": "Counts randomly selected loads when the latency from first dispatch to completion is greater than 512 cycles. Reported latency may be longer than just the memory latency.",
+ "SampleAfterValue": "101",
+ "UMask": "0x1",
+ "Unit": "cpu_core"
+ },
+ {
+ "BriefDescription": "Counts randomly selected loads when the latency from first dispatch to completion is greater than 64 cycles.",
+ "Data_LA": "1",
+ "EventCode": "0xcd",
+ "EventName": "MEM_TRANS_RETIRED.LOAD_LATENCY_GT_64",
+ "MSRIndex": "0x3F6",
+ "MSRValue": "0x40",
+ "PEBS": "2",
+ "PublicDescription": "Counts randomly selected loads when the latency from first dispatch to completion is greater than 64 cycles. Reported latency may be longer than just the memory latency.",
+ "SampleAfterValue": "2003",
+ "UMask": "0x1",
+ "Unit": "cpu_core"
+ },
+ {
+ "BriefDescription": "Counts randomly selected loads when the latency from first dispatch to completion is greater than 8 cycles.",
+ "Data_LA": "1",
+ "EventCode": "0xcd",
+ "EventName": "MEM_TRANS_RETIRED.LOAD_LATENCY_GT_8",
+ "MSRIndex": "0x3F6",
+ "MSRValue": "0x8",
+ "PEBS": "2",
+ "PublicDescription": "Counts randomly selected loads when the latency from first dispatch to completion is greater than 8 cycles. Reported latency may be longer than just the memory latency.",
+ "SampleAfterValue": "50021",
+ "UMask": "0x1",
+ "Unit": "cpu_core"
+ },
+ {
+ "BriefDescription": "Retired memory store access operations. A PDist event for PEBS Store Latency Facility.",
+ "Data_LA": "1",
+ "EventCode": "0xcd",
+ "EventName": "MEM_TRANS_RETIRED.STORE_SAMPLE",
+ "PEBS": "2",
+ "PublicDescription": "Counts Retired memory accesses with at least 1 store operation. This PEBS event is the precisely-distributed (PDist) trigger covering all stores uops for sampling by the PEBS Store Latency Facility. The facility is described in Intel SDM Volume 3 section 19.9.8",
+ "SampleAfterValue": "1000003",
+ "UMask": "0x2",
+ "Unit": "cpu_core"
+ },
+ {
"BriefDescription": "Counts demand data reads that were not supplied by the L3 cache.",
- "Counter": "0,1,2,3,4,5",
"EventCode": "0xB7",
"EventName": "OCR.DEMAND_DATA_RD.L3_MISS",
"MSRIndex": "0x1a6,0x1a7",
@@ -84,7 +229,16 @@
},
{
"BriefDescription": "Counts demand data reads that were not supplied by the L3 cache.",
- "Counter": "0,1,2,3,4,5",
+ "EventCode": "0x2A,0x2B",
+ "EventName": "OCR.DEMAND_DATA_RD.L3_MISS",
+ "MSRIndex": "0x1a6,0x1a7",
+ "MSRValue": "0x3FBFC00001",
+ "SampleAfterValue": "100003",
+ "UMask": "0x1",
+ "Unit": "cpu_core"
+ },
+ {
+ "BriefDescription": "Counts demand data reads that were not supplied by the L3 cache.",
"EventCode": "0xB7",
"EventName": "OCR.DEMAND_DATA_RD.L3_MISS_LOCAL",
"MSRIndex": "0x1a6,0x1a7",
@@ -95,7 +249,6 @@
},
{
"BriefDescription": "Counts demand reads for ownership (RFO) and software prefetches for exclusive ownership (PREFETCHW) that were not supplied by the L3 cache.",
- "Counter": "0,1,2,3,4,5",
"EventCode": "0xB7",
"EventName": "OCR.DEMAND_RFO.L3_MISS",
"MSRIndex": "0x1a6,0x1a7",
@@ -105,8 +258,17 @@
"Unit": "cpu_atom"
},
{
+ "BriefDescription": "Counts demand read for ownership (RFO) requests and software prefetches for exclusive ownership (PREFETCHW) that were not supplied by the L3 cache.",
+ "EventCode": "0x2A,0x2B",
+ "EventName": "OCR.DEMAND_RFO.L3_MISS",
+ "MSRIndex": "0x1a6,0x1a7",
+ "MSRValue": "0x3FBFC00002",
+ "SampleAfterValue": "100003",
+ "UMask": "0x1",
+ "Unit": "cpu_core"
+ },
+ {
"BriefDescription": "Counts demand reads for ownership (RFO) and software prefetches for exclusive ownership (PREFETCHW) that were not supplied by the L3 cache.",
- "Counter": "0,1,2,3,4,5",
"EventCode": "0xB7",
"EventName": "OCR.DEMAND_RFO.L3_MISS_LOCAL",
"MSRIndex": "0x1a6,0x1a7",
@@ -116,241 +278,21 @@
"Unit": "cpu_atom"
},
{
- "BriefDescription": "Execution stalls while L3 cache miss demand load is outstanding.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3",
- "CounterMask": "6",
- "EventCode": "0xa3",
- "EventName": "CYCLE_ACTIVITY.STALLS_L3_MISS",
- "PEBScounters": "0,1,2,3",
- "SampleAfterValue": "1000003",
- "Speculative": "1",
- "UMask": "0x6",
- "Unit": "cpu_core"
- },
- {
- "BriefDescription": "Number of machine clears due to memory ordering conflicts.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5,6,7",
- "EventCode": "0xc3",
- "EventName": "MACHINE_CLEARS.MEMORY_ORDERING",
- "PEBScounters": "0,1,2,3,4,5,6,7",
+ "BriefDescription": "Demand Data Read requests who miss L3 cache",
+ "EventCode": "0x21",
+ "EventName": "OFFCORE_REQUESTS.L3_MISS_DEMAND_DATA_RD",
+ "PublicDescription": "Demand Data Read requests who miss L3 cache.",
"SampleAfterValue": "100003",
- "Speculative": "1",
- "UMask": "0x2",
+ "UMask": "0x10",
"Unit": "cpu_core"
},
{
- "BriefDescription": "Cycles while L1 cache miss demand load is outstanding.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3",
- "CounterMask": "2",
- "EventCode": "0x47",
- "EventName": "MEMORY_ACTIVITY.CYCLES_L1D_MISS",
- "PEBScounters": "0,1,2,3",
- "SampleAfterValue": "1000003",
- "Speculative": "1",
- "UMask": "0x2",
- "Unit": "cpu_core"
- },
- {
- "BriefDescription": "Execution stalls while L1 cache miss demand load is outstanding.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3",
- "CounterMask": "3",
- "EventCode": "0x47",
- "EventName": "MEMORY_ACTIVITY.STALLS_L1D_MISS",
- "PEBScounters": "0,1,2,3",
- "SampleAfterValue": "1000003",
- "Speculative": "1",
- "UMask": "0x3",
- "Unit": "cpu_core"
- },
- {
- "BriefDescription": "MEMORY_ACTIVITY.STALLS_L2_MISS",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3",
- "CounterMask": "5",
- "EventCode": "0x47",
- "EventName": "MEMORY_ACTIVITY.STALLS_L2_MISS",
- "PEBScounters": "0,1,2,3",
- "SampleAfterValue": "1000003",
- "Speculative": "1",
- "UMask": "0x5",
- "Unit": "cpu_core"
- },
- {
- "BriefDescription": "MEMORY_ACTIVITY.STALLS_L3_MISS",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3",
- "CounterMask": "9",
- "EventCode": "0x47",
- "EventName": "MEMORY_ACTIVITY.STALLS_L3_MISS",
- "PEBScounters": "0,1,2,3",
- "SampleAfterValue": "1000003",
- "Speculative": "1",
- "UMask": "0x9",
- "Unit": "cpu_core"
- },
- {
- "BriefDescription": "Counts randomly selected loads when the latency from first dispatch to completion is greater than 128 cycles.",
- "CollectPEBSRecord": "2",
- "Counter": "1,2,3,4,5,6,7",
- "Data_LA": "1",
- "EventCode": "0xcd",
- "EventName": "MEM_TRANS_RETIRED.LOAD_LATENCY_GT_128",
- "MSRIndex": "0x3F6",
- "MSRValue": "0x80",
- "PEBS": "2",
- "PEBScounters": "1,2,3,4,5,6,7",
- "SampleAfterValue": "1009",
- "TakenAlone": "1",
- "UMask": "0x1",
- "Unit": "cpu_core"
- },
- {
- "BriefDescription": "Counts randomly selected loads when the latency from first dispatch to completion is greater than 16 cycles.",
- "CollectPEBSRecord": "2",
- "Counter": "1,2,3,4,5,6,7",
- "Data_LA": "1",
- "EventCode": "0xcd",
- "EventName": "MEM_TRANS_RETIRED.LOAD_LATENCY_GT_16",
- "MSRIndex": "0x3F6",
- "MSRValue": "0x10",
- "PEBS": "2",
- "PEBScounters": "1,2,3,4,5,6,7",
- "SampleAfterValue": "20011",
- "TakenAlone": "1",
- "UMask": "0x1",
- "Unit": "cpu_core"
- },
- {
- "BriefDescription": "Counts randomly selected loads when the latency from first dispatch to completion is greater than 256 cycles.",
- "CollectPEBSRecord": "2",
- "Counter": "1,2,3,4,5,6,7",
- "Data_LA": "1",
- "EventCode": "0xcd",
- "EventName": "MEM_TRANS_RETIRED.LOAD_LATENCY_GT_256",
- "MSRIndex": "0x3F6",
- "MSRValue": "0x100",
- "PEBS": "2",
- "PEBScounters": "1,2,3,4,5,6,7",
- "SampleAfterValue": "503",
- "TakenAlone": "1",
- "UMask": "0x1",
- "Unit": "cpu_core"
- },
- {
- "BriefDescription": "Counts randomly selected loads when the latency from first dispatch to completion is greater than 32 cycles.",
- "CollectPEBSRecord": "2",
- "Counter": "1,2,3,4,5,6,7",
- "Data_LA": "1",
- "EventCode": "0xcd",
- "EventName": "MEM_TRANS_RETIRED.LOAD_LATENCY_GT_32",
- "MSRIndex": "0x3F6",
- "MSRValue": "0x20",
- "PEBS": "2",
- "PEBScounters": "1,2,3,4,5,6,7",
- "SampleAfterValue": "100007",
- "TakenAlone": "1",
- "UMask": "0x1",
- "Unit": "cpu_core"
- },
- {
- "BriefDescription": "Counts randomly selected loads when the latency from first dispatch to completion is greater than 4 cycles.",
- "CollectPEBSRecord": "2",
- "Counter": "1,2,3,4,5,6,7",
- "Data_LA": "1",
- "EventCode": "0xcd",
- "EventName": "MEM_TRANS_RETIRED.LOAD_LATENCY_GT_4",
- "MSRIndex": "0x3F6",
- "MSRValue": "0x4",
- "PEBS": "2",
- "PEBScounters": "1,2,3,4,5,6,7",
- "SampleAfterValue": "100003",
- "TakenAlone": "1",
- "UMask": "0x1",
- "Unit": "cpu_core"
- },
- {
- "BriefDescription": "Counts randomly selected loads when the latency from first dispatch to completion is greater than 512 cycles.",
- "CollectPEBSRecord": "2",
- "Counter": "1,2,3,4,5,6,7",
- "Data_LA": "1",
- "EventCode": "0xcd",
- "EventName": "MEM_TRANS_RETIRED.LOAD_LATENCY_GT_512",
- "MSRIndex": "0x3F6",
- "MSRValue": "0x200",
- "PEBS": "2",
- "PEBScounters": "1,2,3,4,5,6,7",
- "SampleAfterValue": "101",
- "TakenAlone": "1",
- "UMask": "0x1",
- "Unit": "cpu_core"
- },
- {
- "BriefDescription": "Counts randomly selected loads when the latency from first dispatch to completion is greater than 64 cycles.",
- "CollectPEBSRecord": "2",
- "Counter": "1,2,3,4,5,6,7",
- "Data_LA": "1",
- "EventCode": "0xcd",
- "EventName": "MEM_TRANS_RETIRED.LOAD_LATENCY_GT_64",
- "MSRIndex": "0x3F6",
- "MSRValue": "0x40",
- "PEBS": "2",
- "PEBScounters": "1,2,3,4,5,6,7",
- "SampleAfterValue": "2003",
- "TakenAlone": "1",
- "UMask": "0x1",
- "Unit": "cpu_core"
- },
- {
- "BriefDescription": "Counts randomly selected loads when the latency from first dispatch to completion is greater than 8 cycles.",
- "CollectPEBSRecord": "2",
- "Counter": "1,2,3,4,5,6,7",
- "Data_LA": "1",
- "EventCode": "0xcd",
- "EventName": "MEM_TRANS_RETIRED.LOAD_LATENCY_GT_8",
- "MSRIndex": "0x3F6",
- "MSRValue": "0x8",
- "PEBS": "2",
- "PEBScounters": "1,2,3,4,5,6,7",
- "SampleAfterValue": "50021",
- "TakenAlone": "1",
- "UMask": "0x1",
- "Unit": "cpu_core"
- },
- {
- "BriefDescription": "Retired memory store access operations. A PDist event for PEBS Store Latency Facility.",
- "CollectPEBSRecord": "2",
- "Data_LA": "1",
- "EventCode": "0xcd",
- "EventName": "MEM_TRANS_RETIRED.STORE_SAMPLE",
- "PEBS": "2",
- "SampleAfterValue": "1000003",
- "UMask": "0x2",
- "Unit": "cpu_core"
- },
- {
- "BriefDescription": "Counts demand data reads that were not supplied by the L3 cache.",
- "Counter": "0,1,2,3,4,5,6,7",
- "EventCode": "0x2A,0x2B",
- "EventName": "OCR.DEMAND_DATA_RD.L3_MISS",
- "MSRIndex": "0x1a6,0x1a7",
- "MSRValue": "0x3FBFC00001",
- "SampleAfterValue": "100003",
- "UMask": "0x1",
- "Unit": "cpu_core"
- },
- {
- "BriefDescription": "Counts demand read for ownership (RFO) requests and software prefetches for exclusive ownership (PREFETCHW) that were not supplied by the L3 cache.",
- "Counter": "0,1,2,3,4,5,6,7",
- "EventCode": "0x2A,0x2B",
- "EventName": "OCR.DEMAND_RFO.L3_MISS",
- "MSRIndex": "0x1a6,0x1a7",
- "MSRValue": "0x3FBFC00002",
- "SampleAfterValue": "100003",
- "UMask": "0x1",
+ "BriefDescription": "For every cycle, increments by the number of demand data read requests pending that are known to have missed the L3 cache.",
+ "EventCode": "0x20",
+ "EventName": "OFFCORE_REQUESTS_OUTSTANDING.L3_MISS_DEMAND_DATA_RD",
+ "PublicDescription": "For every cycle, increments by the number of demand data read requests pending that are known to have missed the L3 cache. Note that this does not capture all elapsed cycles while requests are outstanding - only cycles from when the requests were known by the requesting core to have missed the L3 cache.",
+ "SampleAfterValue": "2000003",
+ "UMask": "0x10",
"Unit": "cpu_core"
}
]
diff --git a/tools/perf/pmu-events/arch/x86/alderlake/other.json b/tools/perf/pmu-events/arch/x86/alderlake/other.json
index c49d8ce..329c611 100644
--- a/tools/perf/pmu-events/arch/x86/alderlake/other.json
+++ b/tools/perf/pmu-events/arch/x86/alderlake/other.json
@@ -1,7 +1,46 @@
[
{
+ "BriefDescription": "ASSISTS.HARDWARE",
+ "EventCode": "0xc1",
+ "EventName": "ASSISTS.HARDWARE",
+ "SampleAfterValue": "100003",
+ "UMask": "0x4",
+ "Unit": "cpu_core"
+ },
+ {
+ "BriefDescription": "ASSISTS.PAGE_FAULT",
+ "EventCode": "0xc1",
+ "EventName": "ASSISTS.PAGE_FAULT",
+ "SampleAfterValue": "1000003",
+ "UMask": "0x8",
+ "Unit": "cpu_core"
+ },
+ {
+ "BriefDescription": "CORE_POWER.LICENSE_1",
+ "EventCode": "0x28",
+ "EventName": "CORE_POWER.LICENSE_1",
+ "SampleAfterValue": "200003",
+ "UMask": "0x2",
+ "Unit": "cpu_core"
+ },
+ {
+ "BriefDescription": "CORE_POWER.LICENSE_2",
+ "EventCode": "0x28",
+ "EventName": "CORE_POWER.LICENSE_2",
+ "SampleAfterValue": "200003",
+ "UMask": "0x4",
+ "Unit": "cpu_core"
+ },
+ {
+ "BriefDescription": "CORE_POWER.LICENSE_3",
+ "EventCode": "0x28",
+ "EventName": "CORE_POWER.LICENSE_3",
+ "SampleAfterValue": "200003",
+ "UMask": "0x8",
+ "Unit": "cpu_core"
+ },
+ {
"BriefDescription": "Counts modified writebacks from L1 cache and L2 cache that have any type of response.",
- "Counter": "0,1,2,3,4,5",
"EventCode": "0xB7",
"EventName": "OCR.COREWB_M.ANY_RESPONSE",
"MSRIndex": "0x1a6,0x1a7",
@@ -12,7 +51,6 @@
},
{
"BriefDescription": "Counts demand data reads that have any type of response.",
- "Counter": "0,1,2,3,4,5",
"EventCode": "0xB7",
"EventName": "OCR.DEMAND_DATA_RD.ANY_RESPONSE",
"MSRIndex": "0x1a6,0x1a7",
@@ -22,90 +60,7 @@
"Unit": "cpu_atom"
},
{
- "BriefDescription": "Counts demand reads for ownership (RFO) and software prefetches for exclusive ownership (PREFETCHW) that have any type of response.",
- "Counter": "0,1,2,3,4,5",
- "EventCode": "0xB7",
- "EventName": "OCR.DEMAND_RFO.ANY_RESPONSE",
- "MSRIndex": "0x1a6,0x1a7",
- "MSRValue": "0x10002",
- "SampleAfterValue": "100003",
- "UMask": "0x1",
- "Unit": "cpu_atom"
- },
- {
- "BriefDescription": "Counts streaming stores that have any type of response.",
- "Counter": "0,1,2,3,4,5",
- "EventCode": "0xB7",
- "EventName": "OCR.STREAMING_WR.ANY_RESPONSE",
- "MSRIndex": "0x1a6,0x1a7",
- "MSRValue": "0x10800",
- "SampleAfterValue": "100003",
- "UMask": "0x1",
- "Unit": "cpu_atom"
- },
- {
- "BriefDescription": "ASSISTS.HARDWARE",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5,6,7",
- "EventCode": "0xc1",
- "EventName": "ASSISTS.HARDWARE",
- "PEBScounters": "0,1,2,3,4,5,6,7",
- "SampleAfterValue": "100003",
- "Speculative": "1",
- "UMask": "0x4",
- "Unit": "cpu_core"
- },
- {
- "BriefDescription": "ASSISTS.PAGE_FAULT",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5,6,7",
- "EventCode": "0xc1",
- "EventName": "ASSISTS.PAGE_FAULT",
- "PEBScounters": "0,1,2,3,4,5,6,7",
- "SampleAfterValue": "1000003",
- "Speculative": "1",
- "UMask": "0x8",
- "Unit": "cpu_core"
- },
- {
- "BriefDescription": "CORE_POWER.LICENSE_1",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3",
- "EventCode": "0x28",
- "EventName": "CORE_POWER.LICENSE_1",
- "PEBScounters": "0,1,2,3",
- "SampleAfterValue": "200003",
- "Speculative": "1",
- "UMask": "0x2",
- "Unit": "cpu_core"
- },
- {
- "BriefDescription": "CORE_POWER.LICENSE_2",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3",
- "EventCode": "0x28",
- "EventName": "CORE_POWER.LICENSE_2",
- "PEBScounters": "0,1,2,3",
- "SampleAfterValue": "200003",
- "Speculative": "1",
- "UMask": "0x4",
- "Unit": "cpu_core"
- },
- {
- "BriefDescription": "CORE_POWER.LICENSE_3",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3",
- "EventCode": "0x28",
- "EventName": "CORE_POWER.LICENSE_3",
- "PEBScounters": "0,1,2,3",
- "SampleAfterValue": "200003",
- "Speculative": "1",
- "UMask": "0x8",
- "Unit": "cpu_core"
- },
- {
"BriefDescription": "Counts demand data reads that have any type of response.",
- "Counter": "0,1,2,3,4,5,6,7",
"EventCode": "0x2A,0x2B",
"EventName": "OCR.DEMAND_DATA_RD.ANY_RESPONSE",
"MSRIndex": "0x1a6,0x1a7",
@@ -116,7 +71,6 @@
},
{
"BriefDescription": "Counts demand data reads that were supplied by DRAM.",
- "Counter": "0,1,2,3,4,5,6,7",
"EventCode": "0x2A,0x2B",
"EventName": "OCR.DEMAND_DATA_RD.DRAM",
"MSRIndex": "0x1a6,0x1a7",
@@ -126,8 +80,17 @@
"Unit": "cpu_core"
},
{
+ "BriefDescription": "Counts demand reads for ownership (RFO) and software prefetches for exclusive ownership (PREFETCHW) that have any type of response.",
+ "EventCode": "0xB7",
+ "EventName": "OCR.DEMAND_RFO.ANY_RESPONSE",
+ "MSRIndex": "0x1a6,0x1a7",
+ "MSRValue": "0x10002",
+ "SampleAfterValue": "100003",
+ "UMask": "0x1",
+ "Unit": "cpu_atom"
+ },
+ {
"BriefDescription": "Counts demand read for ownership (RFO) requests and software prefetches for exclusive ownership (PREFETCHW) that have any type of response.",
- "Counter": "0,1,2,3,4,5,6,7",
"EventCode": "0x2A,0x2B",
"EventName": "OCR.DEMAND_RFO.ANY_RESPONSE",
"MSRIndex": "0x1a6,0x1a7",
@@ -138,7 +101,16 @@
},
{
"BriefDescription": "Counts streaming stores that have any type of response.",
- "Counter": "0,1,2,3,4,5,6,7",
+ "EventCode": "0xB7",
+ "EventName": "OCR.STREAMING_WR.ANY_RESPONSE",
+ "MSRIndex": "0x1a6,0x1a7",
+ "MSRValue": "0x10800",
+ "SampleAfterValue": "100003",
+ "UMask": "0x1",
+ "Unit": "cpu_atom"
+ },
+ {
+ "BriefDescription": "Counts streaming stores that have any type of response.",
"EventCode": "0x2A,0x2B",
"EventName": "OCR.STREAMING_WR.ANY_RESPONSE",
"MSRIndex": "0x1a6,0x1a7",
@@ -149,68 +121,52 @@
},
{
"BriefDescription": "Cycles when Reservation Station (RS) is empty for the thread.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5,6,7",
"EventCode": "0xa5",
"EventName": "RS.EMPTY",
- "PEBScounters": "0,1,2,3,4,5,6,7",
+ "PublicDescription": "Counts cycles during which the reservation station (RS) is empty for this logical processor. This is usually caused when the front-end pipeline runs into starvation periods (e.g. branch mispredictions or i-cache misses)",
"SampleAfterValue": "1000003",
- "Speculative": "1",
"UMask": "0x7",
"Unit": "cpu_core"
},
{
"BriefDescription": "Counts end of periods where the Reservation Station (RS) was empty.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5,6,7",
"CounterMask": "1",
"EdgeDetect": "1",
"EventCode": "0xa5",
"EventName": "RS.EMPTY_COUNT",
"Invert": "1",
- "PEBScounters": "0,1,2,3,4,5,6,7",
+ "PublicDescription": "Counts end of periods where the Reservation Station (RS) was empty. Could be useful to closely sample on front-end latency issues (see the FRONTEND_RETIRED event of designated precise events)",
"SampleAfterValue": "100003",
- "Speculative": "1",
"UMask": "0x7",
"Unit": "cpu_core"
},
{
"BriefDescription": "This event is deprecated. Refer to new event RS.EMPTY_COUNT",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5,6,7",
"CounterMask": "1",
+ "Deprecated": "1",
"EdgeDetect": "1",
"EventCode": "0xa5",
"EventName": "RS_EMPTY.COUNT",
"Invert": "1",
- "PEBScounters": "0,1,2,3,4,5,6,7",
"SampleAfterValue": "100003",
- "Speculative": "1",
"UMask": "0x7",
"Unit": "cpu_core"
},
{
"BriefDescription": "This event is deprecated. Refer to new event RS.EMPTY",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5,6,7",
+ "Deprecated": "1",
"EventCode": "0xa5",
"EventName": "RS_EMPTY.CYCLES",
- "PEBScounters": "0,1,2,3,4,5,6,7",
"SampleAfterValue": "1000003",
- "Speculative": "1",
"UMask": "0x7",
"Unit": "cpu_core"
},
{
"BriefDescription": "XQ.FULL_CYCLES",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3",
"CounterMask": "1",
"EventCode": "0x2d",
"EventName": "XQ.FULL_CYCLES",
- "PEBScounters": "0,1,2,3",
"SampleAfterValue": "1000003",
- "Speculative": "1",
"UMask": "0x1",
"Unit": "cpu_core"
}
diff --git a/tools/perf/pmu-events/arch/x86/alderlake/pipeline.json b/tools/perf/pmu-events/arch/x86/alderlake/pipeline.json
index 1a137f7..f46fa7b 100644
--- a/tools/perf/pmu-events/arch/x86/alderlake/pipeline.json
+++ b/tools/perf/pmu-events/arch/x86/alderlake/pipeline.json
@@ -1,2168 +1,1634 @@
[
{
+ "BriefDescription": "This event is deprecated. Refer to new event ARITH.DIV_ACTIVE",
+ "CounterMask": "1",
+ "Deprecated": "1",
+ "EventCode": "0xb0",
+ "EventName": "ARITH.DIVIDER_ACTIVE",
+ "SampleAfterValue": "1000003",
+ "UMask": "0x9",
+ "Unit": "cpu_core"
+ },
+ {
+ "BriefDescription": "Cycles when divide unit is busy executing divide or square root operations.",
+ "CounterMask": "1",
+ "EventCode": "0xb0",
+ "EventName": "ARITH.DIV_ACTIVE",
+ "PublicDescription": "Counts cycles when divide unit is busy executing divide or square root operations. Accounts for integer and floating-point operations.",
+ "SampleAfterValue": "1000003",
+ "UMask": "0x9",
+ "Unit": "cpu_core"
+ },
+ {
+ "BriefDescription": "This event is deprecated. Refer to new event ARITH.FPDIV_ACTIVE",
+ "CounterMask": "1",
+ "Deprecated": "1",
+ "EventCode": "0xb0",
+ "EventName": "ARITH.FP_DIVIDER_ACTIVE",
+ "SampleAfterValue": "1000003",
+ "UMask": "0x1",
+ "Unit": "cpu_core"
+ },
+ {
+ "BriefDescription": "This event counts the cycles the integer divider is busy.",
+ "EventCode": "0xb0",
+ "EventName": "ARITH.IDIV_ACTIVE",
+ "SampleAfterValue": "1000003",
+ "UMask": "0x8",
+ "Unit": "cpu_core"
+ },
+ {
+ "BriefDescription": "This event is deprecated. Refer to new event ARITH.IDIV_ACTIVE",
+ "CounterMask": "1",
+ "Deprecated": "1",
+ "EventCode": "0xb0",
+ "EventName": "ARITH.INT_DIVIDER_ACTIVE",
+ "SampleAfterValue": "1000003",
+ "UMask": "0x8",
+ "Unit": "cpu_core"
+ },
+ {
+ "BriefDescription": "Number of occurrences where a microcode assist is invoked by hardware.",
+ "EventCode": "0xc1",
+ "EventName": "ASSISTS.ANY",
+ "PublicDescription": "Counts the number of occurrences where a microcode assist is invoked by hardware. Examples include AD (page Access Dirty), FP and AVX related assists.",
+ "SampleAfterValue": "100003",
+ "UMask": "0x1b",
+ "Unit": "cpu_core"
+ },
+ {
"BriefDescription": "Counts the total number of branch instructions retired for all branch types.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5",
"EventCode": "0xc4",
"EventName": "BR_INST_RETIRED.ALL_BRANCHES",
"PEBS": "1",
- "PEBScounters": "0,1,2,3,4,5",
+ "PublicDescription": "Counts the total number of instructions in which the instruction pointer (IP) of the processor is resteered due to a branch instruction and the branch instruction successfully retires. All branch type instructions are accounted for.",
"SampleAfterValue": "200003",
"Unit": "cpu_atom"
},
{
+ "BriefDescription": "All branch instructions retired.",
+ "EventCode": "0xc4",
+ "EventName": "BR_INST_RETIRED.ALL_BRANCHES",
+ "PEBS": "1",
+ "PublicDescription": "Counts all branch instructions retired.",
+ "SampleAfterValue": "400009",
+ "Unit": "cpu_core"
+ },
+ {
"BriefDescription": "This event is deprecated. Refer to new event BR_INST_RETIRED.NEAR_CALL",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5",
+ "Deprecated": "1",
"EventCode": "0xc4",
"EventName": "BR_INST_RETIRED.CALL",
"PEBS": "1",
- "PEBScounters": "0,1,2,3,4,5",
"SampleAfterValue": "200003",
"UMask": "0xf9",
"Unit": "cpu_atom"
},
{
"BriefDescription": "Counts the number of retired JCC (Jump on Conditional Code) branch instructions retired, includes both taken and not taken branches.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5",
"EventCode": "0xc4",
"EventName": "BR_INST_RETIRED.COND",
"PEBS": "1",
- "PEBScounters": "0,1,2,3,4,5",
"SampleAfterValue": "200003",
"UMask": "0x7e",
"Unit": "cpu_atom"
},
{
- "BriefDescription": "Counts the number of taken JCC (Jump on Conditional Code) branch instructions retired.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5",
- "EventCode": "0xc4",
- "EventName": "BR_INST_RETIRED.COND_TAKEN",
- "PEBS": "1",
- "PEBScounters": "0,1,2,3,4,5",
- "SampleAfterValue": "200003",
- "UMask": "0xfe",
- "Unit": "cpu_atom"
- },
- {
- "BriefDescription": "Counts the number of far branch instructions retired, includes far jump, far call and return, and interrupt call and return.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5",
- "EventCode": "0xc4",
- "EventName": "BR_INST_RETIRED.FAR_BRANCH",
- "PEBS": "1",
- "PEBScounters": "0,1,2,3,4,5",
- "SampleAfterValue": "200003",
- "UMask": "0xbf",
- "Unit": "cpu_atom"
- },
- {
- "BriefDescription": "Counts the number of near indirect JMP and near indirect CALL branch instructions retired.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5",
- "EventCode": "0xc4",
- "EventName": "BR_INST_RETIRED.INDIRECT",
- "PEBS": "1",
- "PEBScounters": "0,1,2,3,4,5",
- "SampleAfterValue": "200003",
- "UMask": "0xeb",
- "Unit": "cpu_atom"
- },
- {
- "BriefDescription": "Counts the number of near indirect CALL branch instructions retired.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5",
- "EventCode": "0xc4",
- "EventName": "BR_INST_RETIRED.INDIRECT_CALL",
- "PEBS": "1",
- "PEBScounters": "0,1,2,3,4,5",
- "SampleAfterValue": "200003",
- "UMask": "0xfb",
- "Unit": "cpu_atom"
- },
- {
- "BriefDescription": "This event is deprecated. Refer to new event BR_INST_RETIRED.INDIRECT_CALL",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5",
- "EventCode": "0xc4",
- "EventName": "BR_INST_RETIRED.IND_CALL",
- "PEBS": "1",
- "PEBScounters": "0,1,2,3,4,5",
- "SampleAfterValue": "200003",
- "UMask": "0xfb",
- "Unit": "cpu_atom"
- },
- {
- "BriefDescription": "This event is deprecated. Refer to new event BR_INST_RETIRED.COND",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5",
- "EventCode": "0xc4",
- "EventName": "BR_INST_RETIRED.JCC",
- "PEBS": "1",
- "PEBScounters": "0,1,2,3,4,5",
- "SampleAfterValue": "200003",
- "UMask": "0x7e",
- "Unit": "cpu_atom"
- },
- {
- "BriefDescription": "Counts the number of near CALL branch instructions retired.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5",
- "EventCode": "0xc4",
- "EventName": "BR_INST_RETIRED.NEAR_CALL",
- "PEBS": "1",
- "PEBScounters": "0,1,2,3,4,5",
- "SampleAfterValue": "200003",
- "UMask": "0xf9",
- "Unit": "cpu_atom"
- },
- {
- "BriefDescription": "Counts the number of near RET branch instructions retired.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5",
- "EventCode": "0xc4",
- "EventName": "BR_INST_RETIRED.NEAR_RETURN",
- "PEBS": "1",
- "PEBScounters": "0,1,2,3,4,5",
- "SampleAfterValue": "200003",
- "UMask": "0xf7",
- "Unit": "cpu_atom"
- },
- {
- "BriefDescription": "This event is deprecated. Refer to new event BR_INST_RETIRED.INDIRECT",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5",
- "EventCode": "0xc4",
- "EventName": "BR_INST_RETIRED.NON_RETURN_IND",
- "PEBS": "1",
- "PEBScounters": "0,1,2,3,4,5",
- "SampleAfterValue": "200003",
- "UMask": "0xeb",
- "Unit": "cpu_atom"
- },
- {
- "BriefDescription": "Counts the number of near relative CALL branch instructions retired.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5",
- "EventCode": "0xc4",
- "EventName": "BR_INST_RETIRED.REL_CALL",
- "PEBS": "1",
- "PEBScounters": "0,1,2,3,4,5",
- "SampleAfterValue": "200003",
- "UMask": "0xfd",
- "Unit": "cpu_atom"
- },
- {
- "BriefDescription": "This event is deprecated. Refer to new event BR_INST_RETIRED.NEAR_RETURN",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5",
- "EventCode": "0xc4",
- "EventName": "BR_INST_RETIRED.RETURN",
- "PEBS": "1",
- "PEBScounters": "0,1,2,3,4,5",
- "SampleAfterValue": "200003",
- "UMask": "0xf7",
- "Unit": "cpu_atom"
- },
- {
- "BriefDescription": "This event is deprecated. Refer to new event BR_INST_RETIRED.COND_TAKEN",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5",
- "EventCode": "0xc4",
- "EventName": "BR_INST_RETIRED.TAKEN_JCC",
- "PEBS": "1",
- "PEBScounters": "0,1,2,3,4,5",
- "SampleAfterValue": "200003",
- "UMask": "0xfe",
- "Unit": "cpu_atom"
- },
- {
- "BriefDescription": "Counts the total number of mispredicted branch instructions retired for all branch types.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5",
- "EventCode": "0xc5",
- "EventName": "BR_MISP_RETIRED.ALL_BRANCHES",
- "PEBS": "1",
- "PEBScounters": "0,1,2,3,4,5",
- "SampleAfterValue": "200003",
- "Unit": "cpu_atom"
- },
- {
- "BriefDescription": "Counts the number of mispredicted JCC (Jump on Conditional Code) branch instructions retired.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5",
- "EventCode": "0xc5",
- "EventName": "BR_MISP_RETIRED.COND",
- "PEBS": "1",
- "PEBScounters": "0,1,2,3,4,5",
- "SampleAfterValue": "200003",
- "UMask": "0x7e",
- "Unit": "cpu_atom"
- },
- {
- "BriefDescription": "Counts the number of mispredicted taken JCC (Jump on Conditional Code) branch instructions retired.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5",
- "EventCode": "0xc5",
- "EventName": "BR_MISP_RETIRED.COND_TAKEN",
- "PEBS": "1",
- "PEBScounters": "0,1,2,3,4,5",
- "SampleAfterValue": "200003",
- "UMask": "0xfe",
- "Unit": "cpu_atom"
- },
- {
- "BriefDescription": "Counts the number of mispredicted near indirect JMP and near indirect CALL branch instructions retired.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5",
- "EventCode": "0xc5",
- "EventName": "BR_MISP_RETIRED.INDIRECT",
- "PEBS": "1",
- "PEBScounters": "0,1,2,3,4,5",
- "SampleAfterValue": "200003",
- "UMask": "0xeb",
- "Unit": "cpu_atom"
- },
- {
- "BriefDescription": "Counts the number of mispredicted near indirect CALL branch instructions retired.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5",
- "EventCode": "0xc5",
- "EventName": "BR_MISP_RETIRED.INDIRECT_CALL",
- "PEBS": "1",
- "PEBScounters": "0,1,2,3,4,5",
- "SampleAfterValue": "200003",
- "UMask": "0xfb",
- "Unit": "cpu_atom"
- },
- {
- "BriefDescription": "This event is deprecated. Refer to new event BR_MISP_RETIRED.INDIRECT_CALL",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5",
- "EventCode": "0xc5",
- "EventName": "BR_MISP_RETIRED.IND_CALL",
- "PEBS": "1",
- "PEBScounters": "0,1,2,3,4,5",
- "SampleAfterValue": "200003",
- "UMask": "0xfb",
- "Unit": "cpu_atom"
- },
- {
- "BriefDescription": "This event is deprecated. Refer to new event BR_MISP_RETIRED.COND",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5",
- "EventCode": "0xc5",
- "EventName": "BR_MISP_RETIRED.JCC",
- "PEBS": "1",
- "PEBScounters": "0,1,2,3,4,5",
- "SampleAfterValue": "200003",
- "UMask": "0x7e",
- "Unit": "cpu_atom"
- },
- {
- "BriefDescription": "This event is deprecated. Refer to new event BR_MISP_RETIRED.INDIRECT",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5",
- "EventCode": "0xc5",
- "EventName": "BR_MISP_RETIRED.NON_RETURN_IND",
- "PEBS": "1",
- "PEBScounters": "0,1,2,3,4,5",
- "SampleAfterValue": "200003",
- "UMask": "0xeb",
- "Unit": "cpu_atom"
- },
- {
- "BriefDescription": "Counts the number of mispredicted near RET branch instructions retired.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5",
- "EventCode": "0xc5",
- "EventName": "BR_MISP_RETIRED.RETURN",
- "PEBS": "1",
- "PEBScounters": "0,1,2,3,4,5",
- "SampleAfterValue": "200003",
- "UMask": "0xf7",
- "Unit": "cpu_atom"
- },
- {
- "BriefDescription": "This event is deprecated. Refer to new event BR_MISP_RETIRED.COND_TAKEN",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5",
- "EventCode": "0xc5",
- "EventName": "BR_MISP_RETIRED.TAKEN_JCC",
- "PEBS": "1",
- "PEBScounters": "0,1,2,3,4,5",
- "SampleAfterValue": "200003",
- "UMask": "0xfe",
- "Unit": "cpu_atom"
- },
- {
- "BriefDescription": "Counts the number of unhalted core clock cycles. (Fixed event)",
- "CollectPEBSRecord": "2",
- "Counter": "Fixed counter 1",
- "EventName": "CPU_CLK_UNHALTED.CORE",
- "PEBScounters": "33",
- "SampleAfterValue": "2000003",
- "Speculative": "1",
- "UMask": "0x2",
- "Unit": "cpu_atom"
- },
- {
- "BriefDescription": "Counts the number of unhalted core clock cycles.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5",
- "EventCode": "0x3c",
- "EventName": "CPU_CLK_UNHALTED.CORE_P",
- "PEBScounters": "0,1,2,3,4,5",
- "SampleAfterValue": "2000003",
- "Speculative": "1",
- "Unit": "cpu_atom"
- },
- {
- "BriefDescription": "Counts the number of unhalted reference clock cycles at TSC frequency. (Fixed event)",
- "CollectPEBSRecord": "2",
- "Counter": "Fixed counter 2",
- "EventName": "CPU_CLK_UNHALTED.REF_TSC",
- "PEBScounters": "34",
- "SampleAfterValue": "2000003",
- "Speculative": "1",
- "UMask": "0x3",
- "Unit": "cpu_atom"
- },
- {
- "BriefDescription": "Counts the number of unhalted reference clock cycles at TSC frequency.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5",
- "EventCode": "0x3c",
- "EventName": "CPU_CLK_UNHALTED.REF_TSC_P",
- "PEBScounters": "0,1,2,3,4,5",
- "SampleAfterValue": "2000003",
- "Speculative": "1",
- "UMask": "0x1",
- "Unit": "cpu_atom"
- },
- {
- "BriefDescription": "Counts the number of unhalted core clock cycles. (Fixed event)",
- "CollectPEBSRecord": "2",
- "Counter": "Fixed counter 1",
- "EventName": "CPU_CLK_UNHALTED.THREAD",
- "PEBScounters": "33",
- "SampleAfterValue": "2000003",
- "Speculative": "1",
- "UMask": "0x2",
- "Unit": "cpu_atom"
- },
- {
- "BriefDescription": "Counts the number of unhalted core clock cycles.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5",
- "EventCode": "0x3c",
- "EventName": "CPU_CLK_UNHALTED.THREAD_P",
- "PEBScounters": "0,1,2,3,4,5",
- "SampleAfterValue": "2000003",
- "Speculative": "1",
- "Unit": "cpu_atom"
- },
- {
- "BriefDescription": "Counts the total number of instructions retired. (Fixed event)",
- "CollectPEBSRecord": "2",
- "Counter": "Fixed counter 0",
- "EventName": "INST_RETIRED.ANY",
- "PEBS": "1",
- "PEBScounters": "32",
- "SampleAfterValue": "2000003",
- "UMask": "0x1",
- "Unit": "cpu_atom"
- },
- {
- "BriefDescription": "Counts the total number of instructions retired.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5",
- "EventCode": "0xc0",
- "EventName": "INST_RETIRED.ANY_P",
- "PEBS": "1",
- "PEBScounters": "0,1,2,3,4,5",
- "SampleAfterValue": "2000003",
- "Unit": "cpu_atom"
- },
- {
- "BriefDescription": "This event is deprecated. Refer to new event LD_BLOCKS.ADDRESS_ALIAS",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5",
- "EventCode": "0x03",
- "EventName": "LD_BLOCKS.4K_ALIAS",
- "PEBS": "1",
- "PEBScounters": "0,1,2,3,4,5",
- "SampleAfterValue": "1000003",
- "UMask": "0x4",
- "Unit": "cpu_atom"
- },
- {
- "BriefDescription": "Counts the number of retired loads that are blocked because it initially appears to be store forward blocked, but subsequently is shown not to be blocked based on 4K alias check.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5",
- "EventCode": "0x03",
- "EventName": "LD_BLOCKS.ADDRESS_ALIAS",
- "PEBS": "1",
- "PEBScounters": "0,1,2,3,4,5",
- "SampleAfterValue": "1000003",
- "UMask": "0x4",
- "Unit": "cpu_atom"
- },
- {
- "BriefDescription": "Counts the number of retired loads that are blocked because its address exactly matches an older store whose data is not ready.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5",
- "EventCode": "0x03",
- "EventName": "LD_BLOCKS.DATA_UNKNOWN",
- "PEBS": "1",
- "PEBScounters": "0,1,2,3,4,5",
- "SampleAfterValue": "1000003",
- "UMask": "0x1",
- "Unit": "cpu_atom"
- },
- {
- "BriefDescription": "Counts the number of machine clears due to memory ordering in which an internal load passes an older store within the same CPU.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5",
- "EventCode": "0xc3",
- "EventName": "MACHINE_CLEARS.DISAMBIGUATION",
- "PEBScounters": "0,1,2,3,4,5",
- "SampleAfterValue": "20003",
- "Speculative": "1",
- "UMask": "0x8",
- "Unit": "cpu_atom"
- },
- {
- "BriefDescription": "Counts the number of machines clears due to memory renaming.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5",
- "EventCode": "0xc3",
- "EventName": "MACHINE_CLEARS.MRN_NUKE",
- "PEBScounters": "0,1,2,3,4,5",
- "SampleAfterValue": "1000003",
- "Speculative": "1",
- "UMask": "0x80",
- "Unit": "cpu_atom"
- },
- {
- "BriefDescription": "Counts the number of machine clears due to a page fault. Counts both I-Side and D-Side (Loads/Stores) page faults. A page fault occurs when either the page is not present, or an access violation occurs.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5",
- "EventCode": "0xc3",
- "EventName": "MACHINE_CLEARS.PAGE_FAULT",
- "PEBScounters": "0,1,2,3,4,5",
- "SampleAfterValue": "20003",
- "Speculative": "1",
- "UMask": "0x20",
- "Unit": "cpu_atom"
- },
- {
- "BriefDescription": "Counts the number of machine clears that flush the pipeline and restart the machine with the use of microcode due to SMC, MEMORY_ORDERING, FP_ASSISTS, PAGE_FAULT, DISAMBIGUATION, and FPC_VIRTUAL_TRAP.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5",
- "EventCode": "0xc3",
- "EventName": "MACHINE_CLEARS.SLOW",
- "PEBScounters": "0,1,2,3,4,5",
- "SampleAfterValue": "20003",
- "Speculative": "1",
- "UMask": "0x6f",
- "Unit": "cpu_atom"
- },
- {
- "BriefDescription": "Counts the number of machine clears due to program modifying data (self modifying code) within 1K of a recently fetched code page.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5",
- "EventCode": "0xc3",
- "EventName": "MACHINE_CLEARS.SMC",
- "PEBScounters": "0,1,2,3,4,5",
- "SampleAfterValue": "20003",
- "Speculative": "1",
- "UMask": "0x1",
- "Unit": "cpu_atom"
- },
- {
- "BriefDescription": "Counts the number of issue slots not consumed by the backend due to a micro-sequencer (MS) scoreboard, which stalls the front-end from issuing from the UROM until a specified older uop retires.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5",
- "EventCode": "0x75",
- "EventName": "SERIALIZATION.NON_C01_MS_SCB",
- "PEBScounters": "0,1,2,3,4,5",
- "SampleAfterValue": "200003",
- "Speculative": "1",
- "UMask": "0x2",
- "Unit": "cpu_atom"
- },
- {
- "BriefDescription": "Counts the total number of issue slots that were not consumed by the backend because allocation is stalled due to a mispredicted jump or a machine clear.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5",
- "EventCode": "0x73",
- "EventName": "TOPDOWN_BAD_SPECULATION.ALL",
- "PEBScounters": "0,1,2,3,4,5",
- "SampleAfterValue": "1000003",
- "Speculative": "1",
- "Unit": "cpu_atom"
- },
- {
- "BriefDescription": "Counts the number of issue slots every cycle that were not consumed by the backend due to fast nukes such as memory ordering and memory disambiguation machine clears.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5",
- "EventCode": "0x73",
- "EventName": "TOPDOWN_BAD_SPECULATION.FASTNUKE",
- "PEBScounters": "0,1,2,3,4,5",
- "SampleAfterValue": "1000003",
- "Speculative": "1",
- "UMask": "0x2",
- "Unit": "cpu_atom"
- },
- {
- "BriefDescription": "Counts the total number of issue slots that were not consumed by the backend because allocation is stalled due to a machine clear (nuke) of any kind including memory ordering and memory disambiguation.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5",
- "EventCode": "0x73",
- "EventName": "TOPDOWN_BAD_SPECULATION.MACHINE_CLEARS",
- "PEBScounters": "0,1,2,3,4,5",
- "SampleAfterValue": "1000003",
- "Speculative": "1",
- "UMask": "0x3",
- "Unit": "cpu_atom"
- },
- {
- "BriefDescription": "Counts the number of issue slots every cycle that were not consumed by the backend due to branch mispredicts.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5",
- "EventCode": "0x73",
- "EventName": "TOPDOWN_BAD_SPECULATION.MISPREDICT",
- "PEBScounters": "0,1,2,3,4,5",
- "SampleAfterValue": "1000003",
- "Speculative": "1",
- "UMask": "0x4",
- "Unit": "cpu_atom"
- },
- {
- "BriefDescription": "Counts the number of issue slots every cycle that were not consumed by the backend due to a machine clear (nuke).",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5",
- "EventCode": "0x73",
- "EventName": "TOPDOWN_BAD_SPECULATION.NUKE",
- "PEBScounters": "0,1,2,3,4,5",
- "SampleAfterValue": "1000003",
- "Speculative": "1",
- "UMask": "0x1",
- "Unit": "cpu_atom"
- },
- {
- "BriefDescription": "Counts the total number of issue slots every cycle that were not consumed by the backend due to backend stalls.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5",
- "EventCode": "0x74",
- "EventName": "TOPDOWN_BE_BOUND.ALL",
- "PEBScounters": "0,1,2,3,4,5",
- "SampleAfterValue": "1000003",
- "Speculative": "1",
- "Unit": "cpu_atom"
- },
- {
- "BriefDescription": "Counts the number of issue slots every cycle that were not consumed by the backend due to certain allocation restrictions.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5",
- "EventCode": "0x74",
- "EventName": "TOPDOWN_BE_BOUND.ALLOC_RESTRICTIONS",
- "PEBScounters": "0,1,2,3,4,5",
- "SampleAfterValue": "1000003",
- "Speculative": "1",
- "UMask": "0x1",
- "Unit": "cpu_atom"
- },
- {
- "BriefDescription": "Counts the number of issue slots every cycle that were not consumed by the backend due to memory reservation stalls in which a scheduler is not able to accept uops.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5",
- "EventCode": "0x74",
- "EventName": "TOPDOWN_BE_BOUND.MEM_SCHEDULER",
- "PEBScounters": "0,1,2,3,4,5",
- "SampleAfterValue": "1000003",
- "Speculative": "1",
- "UMask": "0x2",
- "Unit": "cpu_atom"
- },
- {
- "BriefDescription": "Counts the number of issue slots every cycle that were not consumed by the backend due to IEC or FPC RAT stalls, which can be due to FIQ or IEC reservation stalls in which the integer, floating point or SIMD scheduler is not able to accept uops.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5",
- "EventCode": "0x74",
- "EventName": "TOPDOWN_BE_BOUND.NON_MEM_SCHEDULER",
- "PEBScounters": "0,1,2,3,4,5",
- "SampleAfterValue": "1000003",
- "Speculative": "1",
- "UMask": "0x8",
- "Unit": "cpu_atom"
- },
- {
- "BriefDescription": "Counts the number of issue slots every cycle that were not consumed by the backend due to the physical register file unable to accept an entry (marble stalls).",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5",
- "EventCode": "0x74",
- "EventName": "TOPDOWN_BE_BOUND.REGISTER",
- "PEBScounters": "0,1,2,3,4,5",
- "SampleAfterValue": "1000003",
- "Speculative": "1",
- "UMask": "0x20",
- "Unit": "cpu_atom"
- },
- {
- "BriefDescription": "Counts the number of issue slots every cycle that were not consumed by the backend due to the reorder buffer being full (ROB stalls).",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5",
- "EventCode": "0x74",
- "EventName": "TOPDOWN_BE_BOUND.REORDER_BUFFER",
- "PEBScounters": "0,1,2,3,4,5",
- "SampleAfterValue": "1000003",
- "Speculative": "1",
- "UMask": "0x40",
- "Unit": "cpu_atom"
- },
- {
- "BriefDescription": "Counts the number of issue slots every cycle that were not consumed by the backend due to scoreboards from the instruction queue (IQ), jump execution unit (JEU), or microcode sequencer (MS).",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5",
- "EventCode": "0x74",
- "EventName": "TOPDOWN_BE_BOUND.SERIALIZATION",
- "PEBScounters": "0,1,2,3,4,5",
- "SampleAfterValue": "1000003",
- "Speculative": "1",
- "UMask": "0x10",
- "Unit": "cpu_atom"
- },
- {
- "BriefDescription": "Counts the total number of issue slots every cycle that were not consumed by the backend due to frontend stalls.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5",
- "EventCode": "0x71",
- "EventName": "TOPDOWN_FE_BOUND.ALL",
- "PEBScounters": "0,1,2,3,4,5",
- "SampleAfterValue": "1000003",
- "Speculative": "1",
- "Unit": "cpu_atom"
- },
- {
- "BriefDescription": "Counts the number of issue slots every cycle that were not delivered by the frontend due to BACLEARS.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5",
- "EventCode": "0x71",
- "EventName": "TOPDOWN_FE_BOUND.BRANCH_DETECT",
- "PEBScounters": "0,1,2,3,4,5",
- "SampleAfterValue": "1000003",
- "Speculative": "1",
- "UMask": "0x2",
- "Unit": "cpu_atom"
- },
- {
- "BriefDescription": "Counts the number of issue slots every cycle that were not delivered by the frontend due to BTCLEARS.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5",
- "EventCode": "0x71",
- "EventName": "TOPDOWN_FE_BOUND.BRANCH_RESTEER",
- "PEBScounters": "0,1,2,3,4,5",
- "SampleAfterValue": "1000003",
- "Speculative": "1",
- "UMask": "0x40",
- "Unit": "cpu_atom"
- },
- {
- "BriefDescription": "Counts the number of issue slots every cycle that were not delivered by the frontend due to the microcode sequencer (MS).",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5",
- "EventCode": "0x71",
- "EventName": "TOPDOWN_FE_BOUND.CISC",
- "PEBScounters": "0,1,2,3,4,5",
- "SampleAfterValue": "1000003",
- "Speculative": "1",
- "UMask": "0x1",
- "Unit": "cpu_atom"
- },
- {
- "BriefDescription": "Counts the number of issue slots every cycle that were not delivered by the frontend due to decode stalls.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5",
- "EventCode": "0x71",
- "EventName": "TOPDOWN_FE_BOUND.DECODE",
- "PEBScounters": "0,1,2,3,4,5",
- "SampleAfterValue": "1000003",
- "Speculative": "1",
- "UMask": "0x8",
- "Unit": "cpu_atom"
- },
- {
- "BriefDescription": "Counts the number of issue slots every cycle that were not delivered by the frontend due to frontend bandwidth restrictions due to decode, predecode, cisc, and other limitations.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5",
- "EventCode": "0x71",
- "EventName": "TOPDOWN_FE_BOUND.FRONTEND_BANDWIDTH",
- "PEBScounters": "0,1,2,3,4,5",
- "SampleAfterValue": "1000003",
- "Speculative": "1",
- "UMask": "0x8d",
- "Unit": "cpu_atom"
- },
- {
- "BriefDescription": "Counts the number of issue slots every cycle that were not delivered by the frontend due to a latency related stalls including BACLEARs, BTCLEARs, ITLB misses, and ICache misses.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5",
- "EventCode": "0x71",
- "EventName": "TOPDOWN_FE_BOUND.FRONTEND_LATENCY",
- "PEBScounters": "0,1,2,3,4,5",
- "SampleAfterValue": "1000003",
- "Speculative": "1",
- "UMask": "0x72",
- "Unit": "cpu_atom"
- },
- {
- "BriefDescription": "Counts the number of issue slots every cycle that were not delivered by the frontend due to ITLB misses.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5",
- "EventCode": "0x71",
- "EventName": "TOPDOWN_FE_BOUND.ITLB",
- "PEBScounters": "0,1,2,3,4,5",
- "SampleAfterValue": "1000003",
- "Speculative": "1",
- "UMask": "0x10",
- "Unit": "cpu_atom"
- },
- {
- "BriefDescription": "Counts the number of issue slots every cycle that were not delivered by the frontend due to other common frontend stalls not categorized.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5",
- "EventCode": "0x71",
- "EventName": "TOPDOWN_FE_BOUND.OTHER",
- "PEBScounters": "0,1,2,3,4,5",
- "SampleAfterValue": "1000003",
- "Speculative": "1",
- "UMask": "0x80",
- "Unit": "cpu_atom"
- },
- {
- "BriefDescription": "Counts the number of issue slots every cycle that were not delivered by the frontend due to wrong predecodes.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5",
- "EventCode": "0x71",
- "EventName": "TOPDOWN_FE_BOUND.PREDECODE",
- "PEBScounters": "0,1,2,3,4,5",
- "SampleAfterValue": "1000003",
- "Speculative": "1",
- "UMask": "0x4",
- "Unit": "cpu_atom"
- },
- {
- "BriefDescription": "Counts the total number of consumed retirement slots.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5",
- "EventCode": "0xc2",
- "EventName": "TOPDOWN_RETIRING.ALL",
- "PEBS": "1",
- "PEBScounters": "0,1,2,3,4,5",
- "SampleAfterValue": "1000003",
- "Unit": "cpu_atom"
- },
- {
- "BriefDescription": "Counts the total number of uops retired.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5",
- "EventCode": "0xc2",
- "EventName": "UOPS_RETIRED.ALL",
- "PEBS": "1",
- "PEBScounters": "0,1,2,3,4,5",
- "SampleAfterValue": "2000003",
- "Unit": "cpu_atom"
- },
- {
- "BriefDescription": "Counts the number of integer divide uops retired.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5",
- "EventCode": "0xc2",
- "EventName": "UOPS_RETIRED.IDIV",
- "PEBS": "1",
- "PEBScounters": "0,1,2,3,4,5",
- "SampleAfterValue": "2000003",
- "UMask": "0x10",
- "Unit": "cpu_atom"
- },
- {
- "BriefDescription": "Counts the number of uops that are from complex flows issued by the micro-sequencer (MS).",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5",
- "EventCode": "0xc2",
- "EventName": "UOPS_RETIRED.MS",
- "PEBS": "1",
- "PEBScounters": "0,1,2,3,4,5",
- "SampleAfterValue": "2000003",
- "UMask": "0x1",
- "Unit": "cpu_atom"
- },
- {
- "BriefDescription": "Counts the number of x87 uops retired, includes those in MS flows.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5",
- "EventCode": "0xc2",
- "EventName": "UOPS_RETIRED.X87",
- "PEBS": "1",
- "PEBScounters": "0,1,2,3,4,5",
- "SampleAfterValue": "2000003",
- "UMask": "0x2",
- "Unit": "cpu_atom"
- },
- {
- "BriefDescription": "This event is deprecated. Refer to new event ARITH.DIV_ACTIVE",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5,6,7",
- "CounterMask": "1",
- "EventCode": "0xb0",
- "EventName": "ARITH.DIVIDER_ACTIVE",
- "PEBScounters": "0,1,2,3,4,5,6,7",
- "SampleAfterValue": "1000003",
- "Speculative": "1",
- "UMask": "0x9",
- "Unit": "cpu_core"
- },
- {
- "BriefDescription": "Cycles when divide unit is busy executing divide or square root operations.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5,6,7",
- "CounterMask": "1",
- "EventCode": "0xb0",
- "EventName": "ARITH.DIV_ACTIVE",
- "PEBScounters": "0,1,2,3,4,5,6,7",
- "SampleAfterValue": "1000003",
- "Speculative": "1",
- "UMask": "0x9",
- "Unit": "cpu_core"
- },
- {
- "BriefDescription": "This event is deprecated. Refer to new event ARITH.FPDIV_ACTIVE",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5,6,7",
- "CounterMask": "1",
- "EventCode": "0xb0",
- "EventName": "ARITH.FP_DIVIDER_ACTIVE",
- "PEBScounters": "0,1,2,3,4,5,6,7",
- "SampleAfterValue": "1000003",
- "Speculative": "1",
- "UMask": "0x1",
- "Unit": "cpu_core"
- },
- {
- "BriefDescription": "This event counts the cycles the integer divider is busy.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5,6,7",
- "EventCode": "0xb0",
- "EventName": "ARITH.IDIV_ACTIVE",
- "PEBScounters": "0,1,2,3,4,5,6,7",
- "SampleAfterValue": "1000003",
- "Speculative": "1",
- "UMask": "0x8",
- "Unit": "cpu_core"
- },
- {
- "BriefDescription": "This event is deprecated. Refer to new event ARITH.IDIV_ACTIVE",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5,6,7",
- "CounterMask": "1",
- "EventCode": "0xb0",
- "EventName": "ARITH.INT_DIVIDER_ACTIVE",
- "PEBScounters": "0,1,2,3,4,5,6,7",
- "SampleAfterValue": "1000003",
- "Speculative": "1",
- "UMask": "0x8",
- "Unit": "cpu_core"
- },
- {
- "BriefDescription": "Number of occurrences where a microcode assist is invoked by hardware.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5,6,7",
- "EventCode": "0xc1",
- "EventName": "ASSISTS.ANY",
- "PEBScounters": "0,1,2,3,4,5,6,7",
- "SampleAfterValue": "100003",
- "Speculative": "1",
- "UMask": "0x1b",
- "Unit": "cpu_core"
- },
- {
- "BriefDescription": "All branch instructions retired.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5,6,7",
- "EventCode": "0xc4",
- "EventName": "BR_INST_RETIRED.ALL_BRANCHES",
- "PEBS": "1",
- "PEBScounters": "0,1,2,3,4,5,6,7",
- "SampleAfterValue": "400009",
- "Unit": "cpu_core"
- },
- {
"BriefDescription": "Conditional branch instructions retired.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5,6,7",
"EventCode": "0xc4",
"EventName": "BR_INST_RETIRED.COND",
"PEBS": "1",
- "PEBScounters": "0,1,2,3,4,5,6,7",
+ "PublicDescription": "Counts conditional branch instructions retired.",
"SampleAfterValue": "400009",
"UMask": "0x11",
"Unit": "cpu_core"
},
{
"BriefDescription": "Not taken branch instructions retired.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5,6,7",
"EventCode": "0xc4",
"EventName": "BR_INST_RETIRED.COND_NTAKEN",
"PEBS": "1",
- "PEBScounters": "0,1,2,3,4,5,6,7",
+ "PublicDescription": "Counts not taken branch instructions retired.",
"SampleAfterValue": "400009",
"UMask": "0x10",
"Unit": "cpu_core"
},
{
- "BriefDescription": "Taken conditional branch instructions retired.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5,6,7",
+ "BriefDescription": "Counts the number of taken JCC (Jump on Conditional Code) branch instructions retired.",
"EventCode": "0xc4",
"EventName": "BR_INST_RETIRED.COND_TAKEN",
"PEBS": "1",
- "PEBScounters": "0,1,2,3,4,5,6,7",
+ "SampleAfterValue": "200003",
+ "UMask": "0xfe",
+ "Unit": "cpu_atom"
+ },
+ {
+ "BriefDescription": "Taken conditional branch instructions retired.",
+ "EventCode": "0xc4",
+ "EventName": "BR_INST_RETIRED.COND_TAKEN",
+ "PEBS": "1",
+ "PublicDescription": "Counts taken conditional branch instructions retired.",
"SampleAfterValue": "400009",
"UMask": "0x1",
"Unit": "cpu_core"
},
{
- "BriefDescription": "Far branch instructions retired.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5,6,7",
+ "BriefDescription": "Counts the number of far branch instructions retired, includes far jump, far call and return, and interrupt call and return.",
"EventCode": "0xc4",
"EventName": "BR_INST_RETIRED.FAR_BRANCH",
"PEBS": "1",
- "PEBScounters": "0,1,2,3,4,5,6,7",
+ "SampleAfterValue": "200003",
+ "UMask": "0xbf",
+ "Unit": "cpu_atom"
+ },
+ {
+ "BriefDescription": "Far branch instructions retired.",
+ "EventCode": "0xc4",
+ "EventName": "BR_INST_RETIRED.FAR_BRANCH",
+ "PEBS": "1",
+ "PublicDescription": "Counts far branch instructions retired.",
"SampleAfterValue": "100007",
"UMask": "0x40",
"Unit": "cpu_core"
},
{
- "BriefDescription": "Indirect near branch instructions retired (excluding returns)",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5,6,7",
+ "BriefDescription": "Counts the number of near indirect JMP and near indirect CALL branch instructions retired.",
"EventCode": "0xc4",
"EventName": "BR_INST_RETIRED.INDIRECT",
"PEBS": "1",
- "PEBScounters": "0,1,2,3,4,5,6,7",
+ "SampleAfterValue": "200003",
+ "UMask": "0xeb",
+ "Unit": "cpu_atom"
+ },
+ {
+ "BriefDescription": "Indirect near branch instructions retired (excluding returns)",
+ "EventCode": "0xc4",
+ "EventName": "BR_INST_RETIRED.INDIRECT",
+ "PEBS": "1",
+ "PublicDescription": "Counts near indirect branch instructions retired excluding returns. TSX abort is an indirect branch.",
"SampleAfterValue": "100003",
"UMask": "0x80",
"Unit": "cpu_core"
},
{
- "BriefDescription": "Direct and indirect near call instructions retired.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5,6,7",
+ "BriefDescription": "Counts the number of near indirect CALL branch instructions retired.",
+ "EventCode": "0xc4",
+ "EventName": "BR_INST_RETIRED.INDIRECT_CALL",
+ "PEBS": "1",
+ "SampleAfterValue": "200003",
+ "UMask": "0xfb",
+ "Unit": "cpu_atom"
+ },
+ {
+ "BriefDescription": "This event is deprecated. Refer to new event BR_INST_RETIRED.INDIRECT_CALL",
+ "Deprecated": "1",
+ "EventCode": "0xc4",
+ "EventName": "BR_INST_RETIRED.IND_CALL",
+ "PEBS": "1",
+ "SampleAfterValue": "200003",
+ "UMask": "0xfb",
+ "Unit": "cpu_atom"
+ },
+ {
+ "BriefDescription": "This event is deprecated. Refer to new event BR_INST_RETIRED.COND",
+ "Deprecated": "1",
+ "EventCode": "0xc4",
+ "EventName": "BR_INST_RETIRED.JCC",
+ "PEBS": "1",
+ "SampleAfterValue": "200003",
+ "UMask": "0x7e",
+ "Unit": "cpu_atom"
+ },
+ {
+ "BriefDescription": "Counts the number of near CALL branch instructions retired.",
"EventCode": "0xc4",
"EventName": "BR_INST_RETIRED.NEAR_CALL",
"PEBS": "1",
- "PEBScounters": "0,1,2,3,4,5,6,7",
+ "SampleAfterValue": "200003",
+ "UMask": "0xf9",
+ "Unit": "cpu_atom"
+ },
+ {
+ "BriefDescription": "Direct and indirect near call instructions retired.",
+ "EventCode": "0xc4",
+ "EventName": "BR_INST_RETIRED.NEAR_CALL",
+ "PEBS": "1",
+ "PublicDescription": "Counts both direct and indirect near call instructions retired.",
"SampleAfterValue": "100007",
"UMask": "0x2",
"Unit": "cpu_core"
},
{
- "BriefDescription": "Return instructions retired.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5,6,7",
+ "BriefDescription": "Counts the number of near RET branch instructions retired.",
"EventCode": "0xc4",
"EventName": "BR_INST_RETIRED.NEAR_RETURN",
"PEBS": "1",
- "PEBScounters": "0,1,2,3,4,5,6,7",
+ "SampleAfterValue": "200003",
+ "UMask": "0xf7",
+ "Unit": "cpu_atom"
+ },
+ {
+ "BriefDescription": "Return instructions retired.",
+ "EventCode": "0xc4",
+ "EventName": "BR_INST_RETIRED.NEAR_RETURN",
+ "PEBS": "1",
+ "PublicDescription": "Counts return instructions retired.",
"SampleAfterValue": "100007",
"UMask": "0x8",
"Unit": "cpu_core"
},
{
"BriefDescription": "Taken branch instructions retired.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5,6,7",
"EventCode": "0xc4",
"EventName": "BR_INST_RETIRED.NEAR_TAKEN",
"PEBS": "1",
- "PEBScounters": "0,1,2,3,4,5,6,7",
+ "PublicDescription": "Counts taken branch instructions retired.",
"SampleAfterValue": "400009",
"UMask": "0x20",
"Unit": "cpu_core"
},
{
- "BriefDescription": "All mispredicted branch instructions retired.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5,6,7",
+ "BriefDescription": "This event is deprecated. Refer to new event BR_INST_RETIRED.INDIRECT",
+ "Deprecated": "1",
+ "EventCode": "0xc4",
+ "EventName": "BR_INST_RETIRED.NON_RETURN_IND",
+ "PEBS": "1",
+ "SampleAfterValue": "200003",
+ "UMask": "0xeb",
+ "Unit": "cpu_atom"
+ },
+ {
+ "BriefDescription": "Counts the number of near relative CALL branch instructions retired.",
+ "EventCode": "0xc4",
+ "EventName": "BR_INST_RETIRED.REL_CALL",
+ "PEBS": "1",
+ "SampleAfterValue": "200003",
+ "UMask": "0xfd",
+ "Unit": "cpu_atom"
+ },
+ {
+ "BriefDescription": "This event is deprecated. Refer to new event BR_INST_RETIRED.NEAR_RETURN",
+ "Deprecated": "1",
+ "EventCode": "0xc4",
+ "EventName": "BR_INST_RETIRED.RETURN",
+ "PEBS": "1",
+ "SampleAfterValue": "200003",
+ "UMask": "0xf7",
+ "Unit": "cpu_atom"
+ },
+ {
+ "BriefDescription": "This event is deprecated. Refer to new event BR_INST_RETIRED.COND_TAKEN",
+ "Deprecated": "1",
+ "EventCode": "0xc4",
+ "EventName": "BR_INST_RETIRED.TAKEN_JCC",
+ "PEBS": "1",
+ "SampleAfterValue": "200003",
+ "UMask": "0xfe",
+ "Unit": "cpu_atom"
+ },
+ {
+ "BriefDescription": "Counts the total number of mispredicted branch instructions retired for all branch types.",
"EventCode": "0xc5",
"EventName": "BR_MISP_RETIRED.ALL_BRANCHES",
"PEBS": "1",
- "PEBScounters": "0,1,2,3,4,5,6,7",
+ "PublicDescription": "Counts the total number of mispredicted branch instructions retired. All branch type instructions are accounted for. Prediction of the branch target address enables the processor to begin executing instructions before the non-speculative execution path is known. The branch prediction unit (BPU) predicts the target address based on the instruction pointer (IP) of the branch and on the execution path through which execution reached this IP. A branch misprediction occurs when the prediction is wrong, and results in discarding all instructions executed in the speculative path and re-fetching from the correct path.",
+ "SampleAfterValue": "200003",
+ "Unit": "cpu_atom"
+ },
+ {
+ "BriefDescription": "All mispredicted branch instructions retired.",
+ "EventCode": "0xc5",
+ "EventName": "BR_MISP_RETIRED.ALL_BRANCHES",
+ "PEBS": "1",
+ "PublicDescription": "Counts all the retired branch instructions that were mispredicted by the processor. A branch misprediction occurs when the processor incorrectly predicts the destination of the branch. When the misprediction is discovered at execution, all the instructions executed in the wrong (speculative) path must be discarded, and the processor must start fetching from the correct path.",
"SampleAfterValue": "400009",
"Unit": "cpu_core"
},
{
- "BriefDescription": "Mispredicted conditional branch instructions retired.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5,6,7",
+ "BriefDescription": "Counts the number of mispredicted JCC (Jump on Conditional Code) branch instructions retired.",
"EventCode": "0xc5",
"EventName": "BR_MISP_RETIRED.COND",
"PEBS": "1",
- "PEBScounters": "0,1,2,3,4,5,6,7",
+ "SampleAfterValue": "200003",
+ "UMask": "0x7e",
+ "Unit": "cpu_atom"
+ },
+ {
+ "BriefDescription": "Mispredicted conditional branch instructions retired.",
+ "EventCode": "0xc5",
+ "EventName": "BR_MISP_RETIRED.COND",
+ "PEBS": "1",
+ "PublicDescription": "Counts mispredicted conditional branch instructions retired.",
"SampleAfterValue": "400009",
"UMask": "0x11",
"Unit": "cpu_core"
},
{
"BriefDescription": "Mispredicted non-taken conditional branch instructions retired.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5,6,7",
"EventCode": "0xc5",
"EventName": "BR_MISP_RETIRED.COND_NTAKEN",
"PEBS": "1",
- "PEBScounters": "0,1,2,3,4,5,6,7",
+ "PublicDescription": "Counts the number of conditional branch instructions retired that were mispredicted and the branch direction was not taken.",
"SampleAfterValue": "400009",
"UMask": "0x10",
"Unit": "cpu_core"
},
{
- "BriefDescription": "number of branch instructions retired that were mispredicted and taken.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5,6,7",
+ "BriefDescription": "Counts the number of mispredicted taken JCC (Jump on Conditional Code) branch instructions retired.",
"EventCode": "0xc5",
"EventName": "BR_MISP_RETIRED.COND_TAKEN",
"PEBS": "1",
- "PEBScounters": "0,1,2,3,4,5,6,7",
+ "SampleAfterValue": "200003",
+ "UMask": "0xfe",
+ "Unit": "cpu_atom"
+ },
+ {
+ "BriefDescription": "number of branch instructions retired that were mispredicted and taken.",
+ "EventCode": "0xc5",
+ "EventName": "BR_MISP_RETIRED.COND_TAKEN",
+ "PEBS": "1",
+ "PublicDescription": "Counts taken conditional mispredicted branch instructions retired.",
"SampleAfterValue": "400009",
"UMask": "0x1",
"Unit": "cpu_core"
},
{
- "BriefDescription": "Mispredicted indirect CALL retired.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5,6,7",
+ "BriefDescription": "Counts the number of mispredicted near indirect JMP and near indirect CALL branch instructions retired.",
+ "EventCode": "0xc5",
+ "EventName": "BR_MISP_RETIRED.INDIRECT",
+ "PEBS": "1",
+ "SampleAfterValue": "200003",
+ "UMask": "0xeb",
+ "Unit": "cpu_atom"
+ },
+ {
+ "BriefDescription": "Counts the number of mispredicted near indirect CALL branch instructions retired.",
"EventCode": "0xc5",
"EventName": "BR_MISP_RETIRED.INDIRECT_CALL",
"PEBS": "1",
- "PEBScounters": "0,1,2,3,4,5,6,7",
+ "SampleAfterValue": "200003",
+ "UMask": "0xfb",
+ "Unit": "cpu_atom"
+ },
+ {
+ "BriefDescription": "Mispredicted indirect CALL retired.",
+ "EventCode": "0xc5",
+ "EventName": "BR_MISP_RETIRED.INDIRECT_CALL",
+ "PEBS": "1",
+ "PublicDescription": "Counts retired mispredicted indirect (near taken) CALL instructions, including both register and memory indirect.",
"SampleAfterValue": "400009",
"UMask": "0x2",
"Unit": "cpu_core"
},
{
+ "BriefDescription": "This event is deprecated. Refer to new event BR_MISP_RETIRED.INDIRECT_CALL",
+ "Deprecated": "1",
+ "EventCode": "0xc5",
+ "EventName": "BR_MISP_RETIRED.IND_CALL",
+ "PEBS": "1",
+ "SampleAfterValue": "200003",
+ "UMask": "0xfb",
+ "Unit": "cpu_atom"
+ },
+ {
+ "BriefDescription": "This event is deprecated. Refer to new event BR_MISP_RETIRED.COND",
+ "Deprecated": "1",
+ "EventCode": "0xc5",
+ "EventName": "BR_MISP_RETIRED.JCC",
+ "PEBS": "1",
+ "SampleAfterValue": "200003",
+ "UMask": "0x7e",
+ "Unit": "cpu_atom"
+ },
+ {
"BriefDescription": "Number of near branch instructions retired that were mispredicted and taken.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5,6,7",
"EventCode": "0xc5",
"EventName": "BR_MISP_RETIRED.NEAR_TAKEN",
"PEBS": "1",
- "PEBScounters": "0,1,2,3,4,5,6,7",
+ "PublicDescription": "Counts number of near branch instructions retired that were mispredicted and taken.",
"SampleAfterValue": "400009",
"UMask": "0x20",
"Unit": "cpu_core"
},
{
+ "BriefDescription": "This event is deprecated. Refer to new event BR_MISP_RETIRED.INDIRECT",
+ "Deprecated": "1",
+ "EventCode": "0xc5",
+ "EventName": "BR_MISP_RETIRED.NON_RETURN_IND",
+ "PEBS": "1",
+ "SampleAfterValue": "200003",
+ "UMask": "0xeb",
+ "Unit": "cpu_atom"
+ },
+ {
"BriefDescription": "This event counts the number of mispredicted ret instructions retired. Non PEBS",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5,6,7",
"EventCode": "0xc5",
"EventName": "BR_MISP_RETIRED.RET",
"PEBS": "1",
- "PEBScounters": "0,1,2,3,4,5,6,7",
+ "PublicDescription": "This is a non-precise version (that is, does not use PEBS) of the event that counts mispredicted return instructions retired.",
"SampleAfterValue": "100007",
"UMask": "0x8",
"Unit": "cpu_core"
},
{
+ "BriefDescription": "Counts the number of mispredicted near RET branch instructions retired.",
+ "EventCode": "0xc5",
+ "EventName": "BR_MISP_RETIRED.RETURN",
+ "PEBS": "1",
+ "SampleAfterValue": "200003",
+ "UMask": "0xf7",
+ "Unit": "cpu_atom"
+ },
+ {
+ "BriefDescription": "This event is deprecated. Refer to new event BR_MISP_RETIRED.COND_TAKEN",
+ "Deprecated": "1",
+ "EventCode": "0xc5",
+ "EventName": "BR_MISP_RETIRED.TAKEN_JCC",
+ "PEBS": "1",
+ "SampleAfterValue": "200003",
+ "UMask": "0xfe",
+ "Unit": "cpu_atom"
+ },
+ {
"BriefDescription": "Core clocks when the thread is in the C0.1 light-weight slower wakeup time but more power saving optimized state.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5,6,7",
"EventCode": "0xec",
"EventName": "CPU_CLK_UNHALTED.C01",
- "PEBScounters": "0,1,2,3,4,5,6,7",
+ "PublicDescription": "Counts core clocks when the thread is in the C0.1 light-weight slower wakeup time but more power saving optimized state. This state can be entered via the TPAUSE or UMWAIT instructions.",
"SampleAfterValue": "2000003",
- "Speculative": "1",
"UMask": "0x10",
"Unit": "cpu_core"
},
{
"BriefDescription": "Core clocks when the thread is in the C0.2 light-weight faster wakeup time but less power saving optimized state.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5,6,7",
"EventCode": "0xec",
"EventName": "CPU_CLK_UNHALTED.C02",
- "PEBScounters": "0,1,2,3,4,5,6,7",
+ "PublicDescription": "Counts core clocks when the thread is in the C0.2 light-weight faster wakeup time but less power saving optimized state. This state can be entered via the TPAUSE or UMWAIT instructions.",
"SampleAfterValue": "2000003",
- "Speculative": "1",
"UMask": "0x20",
"Unit": "cpu_core"
},
{
"BriefDescription": "Core clocks when the thread is in the C0.1 or C0.2 or running a PAUSE in C0 ACPI state.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5,6,7",
"EventCode": "0xec",
"EventName": "CPU_CLK_UNHALTED.C0_WAIT",
- "PEBScounters": "0,1,2,3,4,5,6,7",
+ "PublicDescription": "Counts core clocks when the thread is in the C0.1 or C0.2 power saving optimized states (TPAUSE or UMWAIT instructions) or running the PAUSE instruction.",
"SampleAfterValue": "2000003",
- "Speculative": "1",
"UMask": "0x70",
"Unit": "cpu_core"
},
{
+ "BriefDescription": "Counts the number of unhalted core clock cycles. (Fixed event)",
+ "EventName": "CPU_CLK_UNHALTED.CORE",
+ "PublicDescription": "Counts the number of core cycles while the core is not in a halt state. The core enters the halt state when it is running the HLT instruction. The core frequency may change from time to time. For this reason this event may have a changing ratio with regards to time. This event uses fixed counter 1.",
+ "SampleAfterValue": "2000003",
+ "UMask": "0x2",
+ "Unit": "cpu_atom"
+ },
+ {
+ "BriefDescription": "Counts the number of unhalted core clock cycles.",
+ "EventCode": "0x3c",
+ "EventName": "CPU_CLK_UNHALTED.CORE_P",
+ "PublicDescription": "Counts the number of core cycles while the core is not in a halt state. The core enters the halt state when it is running the HLT instruction. The core frequency may change from time to time. For this reason this event may have a changing ratio with regards to time. This event uses a programmable general purpose performance counter.",
+ "SampleAfterValue": "2000003",
+ "Unit": "cpu_atom"
+ },
+ {
"BriefDescription": "Cycle counts are evenly distributed between active threads in the Core.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5,6,7",
"EventCode": "0xec",
"EventName": "CPU_CLK_UNHALTED.DISTRIBUTED",
- "PEBScounters": "0,1,2,3,4,5,6,7",
+ "PublicDescription": "This event distributes cycle counts between active hyperthreads, i.e., those in C0. A hyperthread becomes inactive when it executes the HLT or MWAIT instructions. If all other hyperthreads are inactive (or disabled or do not exist), all counts are attributed to this hyperthread. To obtain the full count when the Core is active, sum the counts from each hyperthread.",
"SampleAfterValue": "2000003",
- "Speculative": "1",
"UMask": "0x2",
"Unit": "cpu_core"
},
{
"BriefDescription": "Core crystal clock cycles when this thread is unhalted and the other thread is halted.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5,6,7",
"EventCode": "0x3c",
"EventName": "CPU_CLK_UNHALTED.ONE_THREAD_ACTIVE",
- "PEBScounters": "0,1,2,3,4,5,6,7",
+ "PublicDescription": "Counts Core crystal clock cycles when current thread is unhalted and the other thread is halted.",
"SampleAfterValue": "25003",
- "Speculative": "1",
"UMask": "0x2",
"Unit": "cpu_core"
},
{
"BriefDescription": "CPU_CLK_UNHALTED.PAUSE",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5,6,7",
"EventCode": "0xec",
"EventName": "CPU_CLK_UNHALTED.PAUSE",
- "PEBScounters": "0,1,2,3,4,5,6,7",
"SampleAfterValue": "2000003",
- "Speculative": "1",
"UMask": "0x40",
"Unit": "cpu_core"
},
{
"BriefDescription": "CPU_CLK_UNHALTED.PAUSE_INST",
- "Counter": "0,1,2,3,4,5,6,7",
"CounterMask": "1",
"EdgeDetect": "1",
"EventCode": "0xec",
"EventName": "CPU_CLK_UNHALTED.PAUSE_INST",
- "PEBScounters": "0,1,2,3,4,5,6,7",
"SampleAfterValue": "2000003",
- "Speculative": "1",
"UMask": "0x40",
"Unit": "cpu_core"
},
{
"BriefDescription": "Core crystal clock cycles. Cycle counts are evenly distributed between active threads in the Core.",
- "Counter": "0,1,2,3,4,5,6,7",
"EventCode": "0x3c",
"EventName": "CPU_CLK_UNHALTED.REF_DISTRIBUTED",
- "PEBScounters": "0,1,2,3,4,5,6,7",
+ "PublicDescription": "This event distributes Core crystal clock cycle counts between active hyperthreads, i.e., those in C0 sleep-state. A hyperthread becomes inactive when it executes the HLT or MWAIT instructions. If one thread is active in a core, all counts are attributed to this hyperthread. To obtain the full count when the Core is active, sum the counts from each hyperthread.",
"SampleAfterValue": "2000003",
- "Speculative": "1",
"UMask": "0x8",
"Unit": "cpu_core"
},
{
- "BriefDescription": "Reference cycles when the core is not in halt state.",
- "CollectPEBSRecord": "2",
- "Counter": "Fixed counter 2",
+ "BriefDescription": "Counts the number of unhalted reference clock cycles at TSC frequency. (Fixed event)",
"EventName": "CPU_CLK_UNHALTED.REF_TSC",
- "PEBScounters": "34",
+ "PublicDescription": "Counts the number of reference cycles that the core is not in a halt state. The core enters the halt state when it is running the HLT instruction. This event is not affected by core frequency changes and increments at a fixed frequency that is also used for the Time Stamp Counter (TSC). This event uses fixed counter 2.",
"SampleAfterValue": "2000003",
- "Speculative": "1",
+ "UMask": "0x3",
+ "Unit": "cpu_atom"
+ },
+ {
+ "BriefDescription": "Reference cycles when the core is not in halt state.",
+ "EventName": "CPU_CLK_UNHALTED.REF_TSC",
+ "PublicDescription": "Counts the number of reference cycles when the core is not in a halt state. The core enters the halt state when it is running the HLT instruction or the MWAIT instruction. This event is not affected by core frequency changes (for example, P states, TM2 transitions) but has the same incrementing frequency as the time stamp counter. This event can approximate elapsed time while the core was not in a halt state. It is counted on a dedicated fixed counter, leaving the eight programmable counters available for other events. Note: On all current platforms this event stops counting during 'throttling (TM)' states duty off periods the processor is 'halted'. The counter update is done at a lower clock rate then the core clock the overflow status bit for this counter may appear 'sticky'. After the counter has overflowed and software clears the overflow status bit and resets the counter to less than MAX. The reset value to the counter is not clocked immediately so the overflow status bit will flip 'high (1)' and generate another PMI (if enabled) after which the reset value gets clocked into the counter. Therefore, software will get the interrupt, read the overflow status bit '1 for bit 34 while the counter value is less than MAX. Software should ignore this case.",
+ "SampleAfterValue": "2000003",
"UMask": "0x3",
"Unit": "cpu_core"
},
{
- "BriefDescription": "Reference cycles when the core is not in halt state.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5,6,7",
+ "BriefDescription": "Counts the number of unhalted reference clock cycles at TSC frequency.",
"EventCode": "0x3c",
"EventName": "CPU_CLK_UNHALTED.REF_TSC_P",
- "PEBScounters": "0,1,2,3,4,5,6,7",
+ "PublicDescription": "Counts the number of reference cycles that the core is not in a halt state. The core enters the halt state when it is running the HLT instruction. This event is not affected by core frequency changes and increments at a fixed frequency that is also used for the Time Stamp Counter (TSC). This event uses a programmable general purpose performance counter.",
"SampleAfterValue": "2000003",
- "Speculative": "1",
+ "UMask": "0x1",
+ "Unit": "cpu_atom"
+ },
+ {
+ "BriefDescription": "Reference cycles when the core is not in halt state.",
+ "EventCode": "0x3c",
+ "EventName": "CPU_CLK_UNHALTED.REF_TSC_P",
+ "PublicDescription": "Counts the number of reference cycles when the core is not in a halt state. The core enters the halt state when it is running the HLT instruction or the MWAIT instruction. This event is not affected by core frequency changes (for example, P states, TM2 transitions) but has the same incrementing frequency as the time stamp counter. This event can approximate elapsed time while the core was not in a halt state. It is counted on a dedicated fixed counter, leaving the four (eight when Hyperthreading is disabled) programmable counters available for other events. Note: On all current platforms this event stops counting during 'throttling (TM)' states duty off periods the processor is 'halted'. The counter update is done at a lower clock rate then the core clock the overflow status bit for this counter may appear 'sticky'. After the counter has overflowed and software clears the overflow status bit and resets the counter to less than MAX. The reset value to the counter is not clocked immediately so the overflow status bit will flip 'high (1)' and generate another PMI (if enabled) after which the reset value gets clocked into the counter. Therefore, software will get the interrupt, read the overflow status bit '1 for bit 34 while the counter value is less than MAX. Software should ignore this case.",
+ "SampleAfterValue": "2000003",
"UMask": "0x1",
"Unit": "cpu_core"
},
{
- "BriefDescription": "Core cycles when the thread is not in halt state",
- "CollectPEBSRecord": "2",
- "Counter": "Fixed counter 1",
+ "BriefDescription": "Counts the number of unhalted core clock cycles. (Fixed event)",
"EventName": "CPU_CLK_UNHALTED.THREAD",
- "PEBScounters": "33",
+ "PublicDescription": "Counts the number of core cycles while the core is not in a halt state. The core enters the halt state when it is running the HLT instruction. The core frequency may change from time to time. For this reason this event may have a changing ratio with regards to time. This event uses fixed counter 1.",
"SampleAfterValue": "2000003",
- "Speculative": "1",
+ "UMask": "0x2",
+ "Unit": "cpu_atom"
+ },
+ {
+ "BriefDescription": "Core cycles when the thread is not in halt state",
+ "EventName": "CPU_CLK_UNHALTED.THREAD",
+ "PublicDescription": "Counts the number of core cycles while the thread is not in a halt state. The thread enters the halt state when it is running the HLT instruction. This event is a component in many key event ratios. The core frequency may change from time to time due to transitions associated with Enhanced Intel SpeedStep Technology or TM2. For this reason this event may have a changing ratio with regards to time. When the core frequency is constant, this event can approximate elapsed time while the core was not in the halt state. It is counted on a dedicated fixed counter, leaving the eight programmable counters available for other events.",
+ "SampleAfterValue": "2000003",
"UMask": "0x2",
"Unit": "cpu_core"
},
{
- "BriefDescription": "Thread cycles when thread is not in halt state",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5,6,7",
+ "BriefDescription": "Counts the number of unhalted core clock cycles.",
"EventCode": "0x3c",
"EventName": "CPU_CLK_UNHALTED.THREAD_P",
- "PEBScounters": "0,1,2,3,4,5,6,7",
+ "PublicDescription": "Counts the number of core cycles while the core is not in a halt state. The core enters the halt state when it is running the HLT instruction. The core frequency may change from time to time. For this reason this event may have a changing ratio with regards to time. This event uses a programmable general purpose performance counter.",
"SampleAfterValue": "2000003",
- "Speculative": "1",
+ "Unit": "cpu_atom"
+ },
+ {
+ "BriefDescription": "Thread cycles when thread is not in halt state",
+ "EventCode": "0x3c",
+ "EventName": "CPU_CLK_UNHALTED.THREAD_P",
+ "PublicDescription": "This is an architectural event that counts the number of thread cycles while the thread is not in a halt state. The thread enters the halt state when it is running the HLT instruction. The core frequency may change from time to time due to power or thermal throttling. For this reason, this event may have a changing ratio with regards to wall clock time.",
+ "SampleAfterValue": "2000003",
"Unit": "cpu_core"
},
{
"BriefDescription": "Cycles while L1 cache miss demand load is outstanding.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3",
"CounterMask": "8",
"EventCode": "0xa3",
"EventName": "CYCLE_ACTIVITY.CYCLES_L1D_MISS",
- "PEBScounters": "0,1,2,3",
"SampleAfterValue": "1000003",
- "Speculative": "1",
"UMask": "0x8",
"Unit": "cpu_core"
},
{
"BriefDescription": "Cycles while L2 cache miss demand load is outstanding.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3",
"CounterMask": "1",
"EventCode": "0xa3",
"EventName": "CYCLE_ACTIVITY.CYCLES_L2_MISS",
- "PEBScounters": "0,1,2,3",
"SampleAfterValue": "1000003",
- "Speculative": "1",
"UMask": "0x1",
"Unit": "cpu_core"
},
{
"BriefDescription": "Cycles while memory subsystem has an outstanding load.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5,6,7",
"CounterMask": "16",
"EventCode": "0xa3",
"EventName": "CYCLE_ACTIVITY.CYCLES_MEM_ANY",
- "PEBScounters": "0,1,2,3,4,5,6,7",
"SampleAfterValue": "1000003",
- "Speculative": "1",
"UMask": "0x10",
"Unit": "cpu_core"
},
{
"BriefDescription": "Execution stalls while L1 cache miss demand load is outstanding.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3",
"CounterMask": "12",
"EventCode": "0xa3",
"EventName": "CYCLE_ACTIVITY.STALLS_L1D_MISS",
- "PEBScounters": "0,1,2,3",
"SampleAfterValue": "1000003",
- "Speculative": "1",
"UMask": "0xc",
"Unit": "cpu_core"
},
{
"BriefDescription": "Execution stalls while L2 cache miss demand load is outstanding.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3",
"CounterMask": "5",
"EventCode": "0xa3",
"EventName": "CYCLE_ACTIVITY.STALLS_L2_MISS",
- "PEBScounters": "0,1,2,3",
"SampleAfterValue": "1000003",
- "Speculative": "1",
"UMask": "0x5",
"Unit": "cpu_core"
},
{
"BriefDescription": "Total execution stalls.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5,6,7",
"CounterMask": "4",
"EventCode": "0xa3",
"EventName": "CYCLE_ACTIVITY.STALLS_TOTAL",
- "PEBScounters": "0,1,2,3,4,5,6,7",
"SampleAfterValue": "1000003",
- "Speculative": "1",
"UMask": "0x4",
"Unit": "cpu_core"
},
{
"BriefDescription": "Cycles total of 1 uop is executed on all ports and Reservation Station was not empty.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5,6,7",
"EventCode": "0xa6",
"EventName": "EXE_ACTIVITY.1_PORTS_UTIL",
- "PEBScounters": "0,1,2,3,4,5,6,7",
+ "PublicDescription": "Counts cycles during which a total of 1 uop was executed on all ports and Reservation Station (RS) was not empty.",
"SampleAfterValue": "2000003",
- "Speculative": "1",
"UMask": "0x2",
"Unit": "cpu_core"
},
{
"BriefDescription": "Cycles total of 2 uops are executed on all ports and Reservation Station was not empty.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5,6,7",
"EventCode": "0xa6",
"EventName": "EXE_ACTIVITY.2_PORTS_UTIL",
- "PEBScounters": "0,1,2,3,4,5,6,7",
+ "PublicDescription": "Counts cycles during which a total of 2 uops were executed on all ports and Reservation Station (RS) was not empty.",
"SampleAfterValue": "2000003",
- "Speculative": "1",
"UMask": "0x4",
"Unit": "cpu_core"
},
{
"BriefDescription": "Cycles total of 3 uops are executed on all ports and Reservation Station was not empty.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5,6,7",
"EventCode": "0xa6",
"EventName": "EXE_ACTIVITY.3_PORTS_UTIL",
- "PEBScounters": "0,1,2,3,4,5,6,7",
+ "PublicDescription": "Cycles total of 3 uops are executed on all ports and Reservation Station (RS) was not empty.",
"SampleAfterValue": "2000003",
- "Speculative": "1",
"UMask": "0x8",
"Unit": "cpu_core"
},
{
"BriefDescription": "Cycles total of 4 uops are executed on all ports and Reservation Station was not empty.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5,6,7",
"EventCode": "0xa6",
"EventName": "EXE_ACTIVITY.4_PORTS_UTIL",
- "PEBScounters": "0,1,2,3,4,5,6,7",
+ "PublicDescription": "Cycles total of 4 uops are executed on all ports and Reservation Station (RS) was not empty.",
"SampleAfterValue": "2000003",
- "Speculative": "1",
"UMask": "0x10",
"Unit": "cpu_core"
},
{
"BriefDescription": "Execution stalls while memory subsystem has an outstanding load.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5,6,7",
"CounterMask": "5",
"EventCode": "0xa6",
"EventName": "EXE_ACTIVITY.BOUND_ON_LOADS",
- "PEBScounters": "0,1,2,3,4,5,6,7",
"SampleAfterValue": "2000003",
- "Speculative": "1",
"UMask": "0x21",
"Unit": "cpu_core"
},
{
"BriefDescription": "Cycles where the Store Buffer was full and no loads caused an execution stall.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5,6,7",
"CounterMask": "2",
"EventCode": "0xa6",
"EventName": "EXE_ACTIVITY.BOUND_ON_STORES",
- "PEBScounters": "0,1,2,3,4,5,6,7",
+ "PublicDescription": "Counts cycles where the Store Buffer was full and no loads caused an execution stall.",
"SampleAfterValue": "1000003",
- "Speculative": "1",
"UMask": "0x40",
"Unit": "cpu_core"
},
{
"BriefDescription": "Cycles no uop executed while RS was not empty, the SB was not full and there was no outstanding load.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5,6,7",
"EventCode": "0xa6",
"EventName": "EXE_ACTIVITY.EXE_BOUND_0_PORTS",
- "PEBScounters": "0,1,2,3,4,5,6,7",
+ "PublicDescription": "Number of cycles total of 0 uops executed on all ports, Reservation Station (RS) was not empty, the Store Buffer (SB) was not full and there was no outstanding load.",
"SampleAfterValue": "1000003",
- "Speculative": "1",
"UMask": "0x80",
"Unit": "cpu_core"
},
{
"BriefDescription": "Instruction decoders utilized in a cycle",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3",
"EventCode": "0x75",
"EventName": "INST_DECODED.DECODERS",
- "PEBScounters": "0,1,2,3",
+ "PublicDescription": "Number of decoders utilized in a cycle when the MITE (legacy decode pipeline) fetches instructions.",
"SampleAfterValue": "2000003",
- "Speculative": "1",
"UMask": "0x1",
"Unit": "cpu_core"
},
{
+ "BriefDescription": "Counts the total number of instructions retired. (Fixed event)",
+ "EventName": "INST_RETIRED.ANY",
+ "PEBS": "1",
+ "PublicDescription": "Counts the total number of instructions that retired. For instructions that consist of multiple uops, this event counts the retirement of the last uop of the instruction. This event continues counting during hardware interrupts, traps, and inside interrupt handlers. This event uses fixed counter 0.",
+ "SampleAfterValue": "2000003",
+ "UMask": "0x1",
+ "Unit": "cpu_atom"
+ },
+ {
"BriefDescription": "Number of instructions retired. Fixed Counter - architectural event",
- "CollectPEBSRecord": "2",
- "Counter": "Fixed counter 0",
"EventName": "INST_RETIRED.ANY",
"PEBS": "1",
- "PEBScounters": "32",
+ "PublicDescription": "Counts the number of X86 instructions retired - an Architectural PerfMon event. Counting continues during hardware interrupts, traps, and inside interrupt handlers. Notes: INST_RETIRED.ANY is counted by a designated fixed counter freeing up programmable counters to count other events. INST_RETIRED.ANY_P is counted by a programmable counter.",
"SampleAfterValue": "2000003",
"UMask": "0x1",
"Unit": "cpu_core"
},
{
- "BriefDescription": "Number of instructions retired. General Counter - architectural event",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5,6,7",
+ "BriefDescription": "Counts the total number of instructions retired.",
"EventCode": "0xc0",
"EventName": "INST_RETIRED.ANY_P",
"PEBS": "1",
- "PEBScounters": "1,2,3,4,5,6,7",
+ "PublicDescription": "Counts the total number of instructions that retired. For instructions that consist of multiple uops, this event counts the retirement of the last uop of the instruction. This event continues counting during hardware interrupts, traps, and inside interrupt handlers. This event uses a programmable general purpose performance counter.",
+ "SampleAfterValue": "2000003",
+ "Unit": "cpu_atom"
+ },
+ {
+ "BriefDescription": "Number of instructions retired. General Counter - architectural event",
+ "EventCode": "0xc0",
+ "EventName": "INST_RETIRED.ANY_P",
+ "PEBS": "1",
+ "PublicDescription": "Counts the number of X86 instructions retired - an Architectural PerfMon event. Counting continues during hardware interrupts, traps, and inside interrupt handlers. Notes: INST_RETIRED.ANY is counted by a designated fixed counter freeing up programmable counters to count other events. INST_RETIRED.ANY_P is counted by a programmable counter.",
"SampleAfterValue": "2000003",
"Unit": "cpu_core"
},
{
"BriefDescription": "INST_RETIRED.MACRO_FUSED",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5,6,7",
"EventCode": "0xc0",
"EventName": "INST_RETIRED.MACRO_FUSED",
- "PEBScounters": "1,2,3,4,5,6,7",
"SampleAfterValue": "2000003",
"UMask": "0x10",
"Unit": "cpu_core"
},
{
"BriefDescription": "Retired NOP instructions.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5,6,7",
"EventCode": "0xc0",
"EventName": "INST_RETIRED.NOP",
- "PEBScounters": "1,2,3,4,5,6,7",
+ "PublicDescription": "Counts all retired NOP or ENDBR32/64 instructions",
"SampleAfterValue": "2000003",
"UMask": "0x2",
"Unit": "cpu_core"
},
{
"BriefDescription": "Precise instruction retired with PEBS precise-distribution",
- "CollectPEBSRecord": "2",
- "Counter": "Fixed counter 0",
"EventName": "INST_RETIRED.PREC_DIST",
"PEBS": "1",
- "PEBScounters": "32",
+ "PublicDescription": "A version of INST_RETIRED that allows for a precise distribution of samples across instructions retired. It utilizes the Precise Distribution of Instructions Retired (PDIR++) feature to fix bias in how retired instructions get sampled. Use on Fixed Counter 0.",
"SampleAfterValue": "2000003",
"UMask": "0x1",
"Unit": "cpu_core"
},
{
"BriefDescription": "INST_RETIRED.REP_ITERATION",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5,6,7",
"EventCode": "0xc0",
"EventName": "INST_RETIRED.REP_ITERATION",
- "PEBScounters": "1,2,3,4,5,6,7",
"SampleAfterValue": "2000003",
"UMask": "0x8",
"Unit": "cpu_core"
},
{
"BriefDescription": "Counts cycles after recovery from a branch misprediction or machine clear till the first uop is issued from the resteered path.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5,6,7",
"EventCode": "0xad",
"EventName": "INT_MISC.CLEAR_RESTEER_CYCLES",
- "PEBScounters": "0,1,2,3,4,5,6,7",
+ "PublicDescription": "Cycles after recovery from a branch misprediction or machine clear till the first uop is issued from the resteered path.",
"SampleAfterValue": "500009",
- "Speculative": "1",
"UMask": "0x80",
"Unit": "cpu_core"
},
{
"BriefDescription": "Core cycles the allocator was stalled due to recovery from earlier clear event for this thread",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5,6,7",
"EventCode": "0xad",
"EventName": "INT_MISC.RECOVERY_CYCLES",
- "PEBScounters": "0,1,2,3,4,5,6,7",
+ "PublicDescription": "Counts core cycles when the Resource allocator was stalled due to recovery from an earlier branch misprediction or machine clear event.",
"SampleAfterValue": "500009",
- "Speculative": "1",
"UMask": "0x1",
"Unit": "cpu_core"
},
{
"BriefDescription": "INT_MISC.UNKNOWN_BRANCH_CYCLES",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5,6,7",
"EventCode": "0xad",
"EventName": "INT_MISC.UNKNOWN_BRANCH_CYCLES",
"MSRIndex": "0x3F7",
"MSRValue": "0x7",
- "PEBScounters": "0,1,2,3,4,5,6,7",
"SampleAfterValue": "1000003",
- "Speculative": "1",
- "TakenAlone": "1",
"UMask": "0x40",
"Unit": "cpu_core"
},
{
"BriefDescription": "TMA slots where uops got dropped",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5,6,7",
"EventCode": "0xad",
"EventName": "INT_MISC.UOP_DROPPING",
- "PEBScounters": "0,1,2,3,4,5,6,7",
+ "PublicDescription": "Estimated number of Top-down Microarchitecture Analysis slots that got dropped due to non front-end reasons",
"SampleAfterValue": "1000003",
- "Speculative": "1",
"UMask": "0x10",
"Unit": "cpu_core"
},
{
"BriefDescription": "INT_VEC_RETIRED.128BIT",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5,6,7",
"EventCode": "0xe7",
"EventName": "INT_VEC_RETIRED.128BIT",
- "PEBScounters": "0,1,2,3,4,5,6,7",
"SampleAfterValue": "1000003",
"UMask": "0x13",
"Unit": "cpu_core"
},
{
"BriefDescription": "INT_VEC_RETIRED.256BIT",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5,6,7",
"EventCode": "0xe7",
"EventName": "INT_VEC_RETIRED.256BIT",
- "PEBScounters": "0,1,2,3,4,5,6,7",
"SampleAfterValue": "1000003",
"UMask": "0xac",
"Unit": "cpu_core"
},
{
"BriefDescription": "integer ADD, SUB, SAD 128-bit vector instructions.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5,6,7",
"EventCode": "0xe7",
"EventName": "INT_VEC_RETIRED.ADD_128",
- "PEBScounters": "0,1,2,3,4,5,6,7",
+ "PublicDescription": "Number of retired integer ADD/SUB (regular or horizontal), SAD 128-bit vector instructions.",
"SampleAfterValue": "1000003",
"UMask": "0x3",
"Unit": "cpu_core"
},
{
"BriefDescription": "integer ADD, SUB, SAD 256-bit vector instructions.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5,6,7",
"EventCode": "0xe7",
"EventName": "INT_VEC_RETIRED.ADD_256",
- "PEBScounters": "0,1,2,3,4,5,6,7",
+ "PublicDescription": "Number of retired integer ADD/SUB (regular or horizontal), SAD 256-bit vector instructions.",
"SampleAfterValue": "1000003",
"UMask": "0xc",
"Unit": "cpu_core"
},
{
"BriefDescription": "INT_VEC_RETIRED.MUL_256",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5,6,7",
"EventCode": "0xe7",
"EventName": "INT_VEC_RETIRED.MUL_256",
- "PEBScounters": "0,1,2,3,4,5,6,7",
"SampleAfterValue": "1000003",
"UMask": "0x80",
"Unit": "cpu_core"
},
{
"BriefDescription": "INT_VEC_RETIRED.SHUFFLES",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5,6,7",
"EventCode": "0xe7",
"EventName": "INT_VEC_RETIRED.SHUFFLES",
- "PEBScounters": "0,1,2,3,4,5,6,7",
"SampleAfterValue": "1000003",
"UMask": "0x40",
"Unit": "cpu_core"
},
{
"BriefDescription": "INT_VEC_RETIRED.VNNI_128",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5,6,7",
"EventCode": "0xe7",
"EventName": "INT_VEC_RETIRED.VNNI_128",
- "PEBScounters": "0,1,2,3,4,5,6,7",
"SampleAfterValue": "1000003",
"UMask": "0x10",
"Unit": "cpu_core"
},
{
"BriefDescription": "INT_VEC_RETIRED.VNNI_256",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5,6,7",
"EventCode": "0xe7",
"EventName": "INT_VEC_RETIRED.VNNI_256",
- "PEBScounters": "0,1,2,3,4,5,6,7",
"SampleAfterValue": "1000003",
"UMask": "0x20",
"Unit": "cpu_core"
},
{
- "BriefDescription": "False dependencies in MOB due to partial compare on address.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3",
+ "BriefDescription": "This event is deprecated. Refer to new event LD_BLOCKS.ADDRESS_ALIAS",
+ "Deprecated": "1",
+ "EventCode": "0x03",
+ "EventName": "LD_BLOCKS.4K_ALIAS",
+ "PEBS": "1",
+ "SampleAfterValue": "1000003",
+ "UMask": "0x4",
+ "Unit": "cpu_atom"
+ },
+ {
+ "BriefDescription": "Counts the number of retired loads that are blocked because it initially appears to be store forward blocked, but subsequently is shown not to be blocked based on 4K alias check.",
"EventCode": "0x03",
"EventName": "LD_BLOCKS.ADDRESS_ALIAS",
- "PEBScounters": "0,1,2,3",
+ "PEBS": "1",
+ "SampleAfterValue": "1000003",
+ "UMask": "0x4",
+ "Unit": "cpu_atom"
+ },
+ {
+ "BriefDescription": "False dependencies in MOB due to partial compare on address.",
+ "EventCode": "0x03",
+ "EventName": "LD_BLOCKS.ADDRESS_ALIAS",
+ "PublicDescription": "Counts the number of times a load got blocked due to false dependencies in MOB due to partial compare on address.",
"SampleAfterValue": "100003",
- "Speculative": "1",
"UMask": "0x4",
"Unit": "cpu_core"
},
{
+ "BriefDescription": "Counts the number of retired loads that are blocked because its address exactly matches an older store whose data is not ready.",
+ "EventCode": "0x03",
+ "EventName": "LD_BLOCKS.DATA_UNKNOWN",
+ "PEBS": "1",
+ "SampleAfterValue": "1000003",
+ "UMask": "0x1",
+ "Unit": "cpu_atom"
+ },
+ {
"BriefDescription": "The number of times that split load operations are temporarily blocked because all resources for handling the split accesses are in use.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3",
"EventCode": "0x03",
"EventName": "LD_BLOCKS.NO_SR",
- "PEBScounters": "0,1,2,3",
+ "PublicDescription": "Counts the number of times that split load operations are temporarily blocked because all resources for handling the split accesses are in use.",
"SampleAfterValue": "100003",
- "Speculative": "1",
"UMask": "0x88",
"Unit": "cpu_core"
},
{
"BriefDescription": "Loads blocked due to overlapping with a preceding store that cannot be forwarded.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3",
"EventCode": "0x03",
"EventName": "LD_BLOCKS.STORE_FORWARD",
- "PEBScounters": "0,1,2,3",
+ "PublicDescription": "Counts the number of times where store forwarding was prevented for a load operation. The most common case is a load blocked due to the address of memory access (partially) overlapping with a preceding uncompleted store. Note: See the table of not supported store forwards in the Optimization Guide.",
"SampleAfterValue": "100003",
- "Speculative": "1",
"UMask": "0x82",
"Unit": "cpu_core"
},
{
"BriefDescription": "Counts the number of demand load dispatches that hit L1D fill buffer (FB) allocated for software prefetch.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3",
"EventCode": "0x4c",
"EventName": "LOAD_HIT_PREFETCH.SWPF",
- "PEBScounters": "0,1,2,3",
+ "PublicDescription": "Counts all not software-prefetch load dispatches that hit the fill buffer (FB) allocated for the software prefetch. It can also be incremented by some lock instructions. So it should only be used with profiling so that the locks can be excluded by ASM (Assembly File) inspection of the nearby instructions.",
"SampleAfterValue": "100003",
- "Speculative": "1",
"UMask": "0x1",
"Unit": "cpu_core"
},
{
"BriefDescription": "Cycles Uops delivered by the LSD, but didn't come from the decoder.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5,6,7",
"CounterMask": "1",
"EventCode": "0xa8",
"EventName": "LSD.CYCLES_ACTIVE",
- "PEBScounters": "0,1,2,3,4,5,6,7",
+ "PublicDescription": "Counts the cycles when at least one uop is delivered by the LSD (Loop-stream detector).",
"SampleAfterValue": "2000003",
- "Speculative": "1",
"UMask": "0x1",
"Unit": "cpu_core"
},
{
"BriefDescription": "Cycles optimal number of Uops delivered by the LSD, but did not come from the decoder.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5,6,7",
"CounterMask": "6",
"EventCode": "0xa8",
"EventName": "LSD.CYCLES_OK",
- "PEBScounters": "0,1,2,3,4,5,6,7",
+ "PublicDescription": "Counts the cycles when optimal number of uops is delivered by the LSD (Loop-stream detector).",
"SampleAfterValue": "2000003",
- "Speculative": "1",
"UMask": "0x1",
"Unit": "cpu_core"
},
{
"BriefDescription": "Number of Uops delivered by the LSD.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5,6,7",
"EventCode": "0xa8",
"EventName": "LSD.UOPS",
- "PEBScounters": "0,1,2,3,4,5,6,7",
+ "PublicDescription": "Counts the number of uops delivered to the back-end by the LSD(Loop Stream Detector).",
"SampleAfterValue": "2000003",
- "Speculative": "1",
"UMask": "0x1",
"Unit": "cpu_core"
},
{
"BriefDescription": "Number of machine clears (nukes) of any type.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5,6,7",
"CounterMask": "1",
"EdgeDetect": "1",
"EventCode": "0xc3",
"EventName": "MACHINE_CLEARS.COUNT",
- "PEBScounters": "0,1,2,3,4,5,6,7",
+ "PublicDescription": "Counts the number of machine clears (nukes) of any type.",
"SampleAfterValue": "100003",
- "Speculative": "1",
"UMask": "0x1",
"Unit": "cpu_core"
},
{
- "BriefDescription": "Self-modifying code (SMC) detected.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5,6,7",
+ "BriefDescription": "Counts the number of machine clears due to memory ordering in which an internal load passes an older store within the same CPU.",
+ "EventCode": "0xc3",
+ "EventName": "MACHINE_CLEARS.DISAMBIGUATION",
+ "SampleAfterValue": "20003",
+ "UMask": "0x8",
+ "Unit": "cpu_atom"
+ },
+ {
+ "BriefDescription": "Counts the number of machines clears due to memory renaming.",
+ "EventCode": "0xc3",
+ "EventName": "MACHINE_CLEARS.MRN_NUKE",
+ "SampleAfterValue": "1000003",
+ "UMask": "0x80",
+ "Unit": "cpu_atom"
+ },
+ {
+ "BriefDescription": "Counts the number of machine clears due to a page fault. Counts both I-Side and D-Side (Loads/Stores) page faults. A page fault occurs when either the page is not present, or an access violation occurs.",
+ "EventCode": "0xc3",
+ "EventName": "MACHINE_CLEARS.PAGE_FAULT",
+ "SampleAfterValue": "20003",
+ "UMask": "0x20",
+ "Unit": "cpu_atom"
+ },
+ {
+ "BriefDescription": "Counts the number of machine clears that flush the pipeline and restart the machine with the use of microcode due to SMC, MEMORY_ORDERING, FP_ASSISTS, PAGE_FAULT, DISAMBIGUATION, and FPC_VIRTUAL_TRAP.",
+ "EventCode": "0xc3",
+ "EventName": "MACHINE_CLEARS.SLOW",
+ "SampleAfterValue": "20003",
+ "UMask": "0x6f",
+ "Unit": "cpu_atom"
+ },
+ {
+ "BriefDescription": "Counts the number of machine clears due to program modifying data (self modifying code) within 1K of a recently fetched code page.",
"EventCode": "0xc3",
"EventName": "MACHINE_CLEARS.SMC",
- "PEBScounters": "0,1,2,3,4,5,6,7",
+ "SampleAfterValue": "20003",
+ "UMask": "0x1",
+ "Unit": "cpu_atom"
+ },
+ {
+ "BriefDescription": "Self-modifying code (SMC) detected.",
+ "EventCode": "0xc3",
+ "EventName": "MACHINE_CLEARS.SMC",
+ "PublicDescription": "Counts self-modifying code (SMC) detected, which causes a machine clear.",
"SampleAfterValue": "100003",
- "Speculative": "1",
"UMask": "0x4",
"Unit": "cpu_core"
},
{
"BriefDescription": "MISC2_RETIRED.LFENCE",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5,6,7",
"EventCode": "0xe0",
"EventName": "MISC2_RETIRED.LFENCE",
- "PEBScounters": "0,1,2,3,4,5,6,7",
"SampleAfterValue": "400009",
- "Speculative": "1",
"UMask": "0x20",
"Unit": "cpu_core"
},
{
"BriefDescription": "Increments whenever there is an update to the LBR array.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5,6,7",
"EventCode": "0xcc",
"EventName": "MISC_RETIRED.LBR_INSERTS",
- "PEBScounters": "0,1,2,3,4,5,6,7",
+ "PublicDescription": "Increments when an entry is added to the Last Branch Record (LBR) array (or removed from the array in case of RETURNs in call stack mode). The event requires LBR enable via IA32_DEBUGCTL MSR and branch type selection via MSR_LBR_SELECT.",
"SampleAfterValue": "100003",
"UMask": "0x20",
"Unit": "cpu_core"
},
{
"BriefDescription": "Cycles stalled due to no store buffers available. (not including draining form sync).",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5,6,7",
"EventCode": "0xa2",
"EventName": "RESOURCE_STALLS.SB",
- "PEBScounters": "0,1,2,3,4,5,6,7",
+ "PublicDescription": "Counts allocation stall cycles caused by the store buffer (SB) being full. This counts cycles that the pipeline back-end blocked uop delivery from the front-end.",
"SampleAfterValue": "100003",
- "Speculative": "1",
"UMask": "0x8",
"Unit": "cpu_core"
},
{
"BriefDescription": "Counts cycles where the pipeline is stalled due to serializing operations.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5,6,7",
"EventCode": "0xa2",
"EventName": "RESOURCE_STALLS.SCOREBOARD",
- "PEBScounters": "0,1,2,3,4,5,6,7",
"SampleAfterValue": "100003",
- "Speculative": "1",
"UMask": "0x2",
"Unit": "cpu_core"
},
{
+ "BriefDescription": "Counts the number of issue slots not consumed by the backend due to a micro-sequencer (MS) scoreboard, which stalls the front-end from issuing from the UROM until a specified older uop retires.",
+ "EventCode": "0x75",
+ "EventName": "SERIALIZATION.NON_C01_MS_SCB",
+ "PublicDescription": "Counts the number of issue slots not consumed by the backend due to a micro-sequencer (MS) scoreboard, which stalls the front-end from issuing from the UROM until a specified older uop retires. The most commonly executed instruction with an MS scoreboard is PAUSE.",
+ "SampleAfterValue": "200003",
+ "UMask": "0x2",
+ "Unit": "cpu_atom"
+ },
+ {
"BriefDescription": "TMA slots where no uops were being issued due to lack of back-end resources.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5,6,7",
"EventCode": "0xa4",
"EventName": "TOPDOWN.BACKEND_BOUND_SLOTS",
- "PEBScounters": "0,1,2,3,4,5,6,7",
+ "PublicDescription": "Number of slots in TMA method where no micro-operations were being issued from front-end to back-end of the machine due to lack of back-end resources.",
"SampleAfterValue": "10000003",
- "Speculative": "1",
"UMask": "0x2",
"Unit": "cpu_core"
},
{
"BriefDescription": "TMA slots wasted due to incorrect speculations.",
- "CollectPEBSRecord": "2",
"EventCode": "0xa4",
"EventName": "TOPDOWN.BAD_SPEC_SLOTS",
+ "PublicDescription": "Number of slots of TMA method that were wasted due to incorrect speculation. It covers all types of control-flow or data-related mis-speculations.",
"SampleAfterValue": "10000003",
- "Speculative": "1",
"UMask": "0x4",
"Unit": "cpu_core"
},
{
"BriefDescription": "TMA slots wasted due to incorrect speculation by branch mispredictions",
- "CollectPEBSRecord": "2",
"EventCode": "0xa4",
"EventName": "TOPDOWN.BR_MISPREDICT_SLOTS",
+ "PublicDescription": "Number of TMA slots that were wasted due to incorrect speculation by (any type of) branch mispredictions. This event estimates number of specualtive operations that were issued but not retired as well as the out-of-order engine recovery past a branch misprediction.",
"SampleAfterValue": "10000003",
- "Speculative": "1",
"UMask": "0x8",
"Unit": "cpu_core"
},
{
"BriefDescription": "TOPDOWN.MEMORY_BOUND_SLOTS",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5,6,7",
"EventCode": "0xa4",
"EventName": "TOPDOWN.MEMORY_BOUND_SLOTS",
- "PEBScounters": "0,1,2,3,4,5,6,7",
"SampleAfterValue": "10000003",
- "Speculative": "1",
"UMask": "0x10",
"Unit": "cpu_core"
},
{
"BriefDescription": "TMA slots available for an unhalted logical processor. Fixed counter - architectural event",
- "CollectPEBSRecord": "2",
- "Counter": "Fixed counter 3",
"EventName": "TOPDOWN.SLOTS",
- "PEBScounters": "35",
+ "PublicDescription": "Number of available slots for an unhalted logical processor. The event increments by machine-width of the narrowest pipeline as employed by the Top-down Microarchitecture Analysis method (TMA). The count is distributed among unhalted logical processors (hyper-threads) who share the same physical core. Software can use this event as the denominator for the top-level metrics of the TMA method. This architectural event is counted on a designated fixed counter (Fixed Counter 3).",
"SampleAfterValue": "10000003",
- "Speculative": "1",
"UMask": "0x4",
"Unit": "cpu_core"
},
{
"BriefDescription": "TMA slots available for an unhalted logical processor. General counter - architectural event",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5,6,7",
"EventCode": "0xa4",
"EventName": "TOPDOWN.SLOTS_P",
- "PEBScounters": "0,1,2,3,4,5,6,7",
+ "PublicDescription": "Counts the number of available slots for an unhalted logical processor. The event increments by machine-width of the narrowest pipeline as employed by the Top-down Microarchitecture Analysis method. The count is distributed among unhalted logical processors (hyper-threads) who share the same physical core.",
"SampleAfterValue": "10000003",
- "Speculative": "1",
"UMask": "0x1",
"Unit": "cpu_core"
},
{
+ "BriefDescription": "Counts the total number of issue slots that were not consumed by the backend because allocation is stalled due to a mispredicted jump or a machine clear.",
+ "EventCode": "0x73",
+ "EventName": "TOPDOWN_BAD_SPECULATION.ALL",
+ "PublicDescription": "Counts the total number of issue slots that were not consumed by the backend because allocation is stalled due to a mispredicted jump or a machine clear. Only issue slots wasted due to fast nukes such as memory ordering nukes are counted. Other nukes are not accounted for. Counts all issue slots blocked during this recovery window including relevant microcode flows and while uops are not yet available in the instruction queue (IQ) even if an FE_bound event occurs during this period. Also includes the issue slots that were consumed by the backend but were thrown away because they were younger than the mispredict or machine clear.",
+ "SampleAfterValue": "1000003",
+ "Unit": "cpu_atom"
+ },
+ {
+ "BriefDescription": "Counts the number of issue slots every cycle that were not consumed by the backend due to fast nukes such as memory ordering and memory disambiguation machine clears.",
+ "EventCode": "0x73",
+ "EventName": "TOPDOWN_BAD_SPECULATION.FASTNUKE",
+ "SampleAfterValue": "1000003",
+ "UMask": "0x2",
+ "Unit": "cpu_atom"
+ },
+ {
+ "BriefDescription": "Counts the total number of issue slots that were not consumed by the backend because allocation is stalled due to a machine clear (nuke) of any kind including memory ordering and memory disambiguation.",
+ "EventCode": "0x73",
+ "EventName": "TOPDOWN_BAD_SPECULATION.MACHINE_CLEARS",
+ "SampleAfterValue": "1000003",
+ "UMask": "0x3",
+ "Unit": "cpu_atom"
+ },
+ {
+ "BriefDescription": "Counts the number of issue slots every cycle that were not consumed by the backend due to branch mispredicts.",
+ "EventCode": "0x73",
+ "EventName": "TOPDOWN_BAD_SPECULATION.MISPREDICT",
+ "SampleAfterValue": "1000003",
+ "UMask": "0x4",
+ "Unit": "cpu_atom"
+ },
+ {
+ "BriefDescription": "Counts the number of issue slots every cycle that were not consumed by the backend due to a machine clear (nuke).",
+ "EventCode": "0x73",
+ "EventName": "TOPDOWN_BAD_SPECULATION.NUKE",
+ "SampleAfterValue": "1000003",
+ "UMask": "0x1",
+ "Unit": "cpu_atom"
+ },
+ {
+ "BriefDescription": "Counts the total number of issue slots every cycle that were not consumed by the backend due to backend stalls.",
+ "EventCode": "0x74",
+ "EventName": "TOPDOWN_BE_BOUND.ALL",
+ "SampleAfterValue": "1000003",
+ "Unit": "cpu_atom"
+ },
+ {
+ "BriefDescription": "Counts the number of issue slots every cycle that were not consumed by the backend due to certain allocation restrictions.",
+ "EventCode": "0x74",
+ "EventName": "TOPDOWN_BE_BOUND.ALLOC_RESTRICTIONS",
+ "SampleAfterValue": "1000003",
+ "UMask": "0x1",
+ "Unit": "cpu_atom"
+ },
+ {
+ "BriefDescription": "Counts the number of issue slots every cycle that were not consumed by the backend due to memory reservation stalls in which a scheduler is not able to accept uops.",
+ "EventCode": "0x74",
+ "EventName": "TOPDOWN_BE_BOUND.MEM_SCHEDULER",
+ "SampleAfterValue": "1000003",
+ "UMask": "0x2",
+ "Unit": "cpu_atom"
+ },
+ {
+ "BriefDescription": "Counts the number of issue slots every cycle that were not consumed by the backend due to IEC or FPC RAT stalls, which can be due to FIQ or IEC reservation stalls in which the integer, floating point or SIMD scheduler is not able to accept uops.",
+ "EventCode": "0x74",
+ "EventName": "TOPDOWN_BE_BOUND.NON_MEM_SCHEDULER",
+ "SampleAfterValue": "1000003",
+ "UMask": "0x8",
+ "Unit": "cpu_atom"
+ },
+ {
+ "BriefDescription": "Counts the number of issue slots every cycle that were not consumed by the backend due to the physical register file unable to accept an entry (marble stalls).",
+ "EventCode": "0x74",
+ "EventName": "TOPDOWN_BE_BOUND.REGISTER",
+ "SampleAfterValue": "1000003",
+ "UMask": "0x20",
+ "Unit": "cpu_atom"
+ },
+ {
+ "BriefDescription": "Counts the number of issue slots every cycle that were not consumed by the backend due to the reorder buffer being full (ROB stalls).",
+ "EventCode": "0x74",
+ "EventName": "TOPDOWN_BE_BOUND.REORDER_BUFFER",
+ "SampleAfterValue": "1000003",
+ "UMask": "0x40",
+ "Unit": "cpu_atom"
+ },
+ {
+ "BriefDescription": "Counts the number of issue slots every cycle that were not consumed by the backend due to scoreboards from the instruction queue (IQ), jump execution unit (JEU), or microcode sequencer (MS).",
+ "EventCode": "0x74",
+ "EventName": "TOPDOWN_BE_BOUND.SERIALIZATION",
+ "SampleAfterValue": "1000003",
+ "UMask": "0x10",
+ "Unit": "cpu_atom"
+ },
+ {
+ "BriefDescription": "Counts the total number of issue slots every cycle that were not consumed by the backend due to frontend stalls.",
+ "EventCode": "0x71",
+ "EventName": "TOPDOWN_FE_BOUND.ALL",
+ "SampleAfterValue": "1000003",
+ "Unit": "cpu_atom"
+ },
+ {
+ "BriefDescription": "Counts the number of issue slots every cycle that were not delivered by the frontend due to BACLEARS.",
+ "EventCode": "0x71",
+ "EventName": "TOPDOWN_FE_BOUND.BRANCH_DETECT",
+ "PublicDescription": "Counts the number of issue slots every cycle that were not delivered by the frontend due to BACLEARS, which occurs when the Branch Target Buffer (BTB) prediction or lack thereof, was corrected by a later branch predictor in the frontend. Includes BACLEARS due to all branch types including conditional and unconditional jumps, returns, and indirect branches.",
+ "SampleAfterValue": "1000003",
+ "UMask": "0x2",
+ "Unit": "cpu_atom"
+ },
+ {
+ "BriefDescription": "Counts the number of issue slots every cycle that were not delivered by the frontend due to BTCLEARS.",
+ "EventCode": "0x71",
+ "EventName": "TOPDOWN_FE_BOUND.BRANCH_RESTEER",
+ "PublicDescription": "Counts the number of issue slots every cycle that were not delivered by the frontend due to BTCLEARS, which occurs when the Branch Target Buffer (BTB) predicts a taken branch.",
+ "SampleAfterValue": "1000003",
+ "UMask": "0x40",
+ "Unit": "cpu_atom"
+ },
+ {
+ "BriefDescription": "Counts the number of issue slots every cycle that were not delivered by the frontend due to the microcode sequencer (MS).",
+ "EventCode": "0x71",
+ "EventName": "TOPDOWN_FE_BOUND.CISC",
+ "SampleAfterValue": "1000003",
+ "UMask": "0x1",
+ "Unit": "cpu_atom"
+ },
+ {
+ "BriefDescription": "Counts the number of issue slots every cycle that were not delivered by the frontend due to decode stalls.",
+ "EventCode": "0x71",
+ "EventName": "TOPDOWN_FE_BOUND.DECODE",
+ "SampleAfterValue": "1000003",
+ "UMask": "0x8",
+ "Unit": "cpu_atom"
+ },
+ {
+ "BriefDescription": "Counts the number of issue slots every cycle that were not delivered by the frontend due to frontend bandwidth restrictions due to decode, predecode, cisc, and other limitations.",
+ "EventCode": "0x71",
+ "EventName": "TOPDOWN_FE_BOUND.FRONTEND_BANDWIDTH",
+ "SampleAfterValue": "1000003",
+ "UMask": "0x8d",
+ "Unit": "cpu_atom"
+ },
+ {
+ "BriefDescription": "Counts the number of issue slots every cycle that were not delivered by the frontend due to a latency related stalls including BACLEARs, BTCLEARs, ITLB misses, and ICache misses.",
+ "EventCode": "0x71",
+ "EventName": "TOPDOWN_FE_BOUND.FRONTEND_LATENCY",
+ "SampleAfterValue": "1000003",
+ "UMask": "0x72",
+ "Unit": "cpu_atom"
+ },
+ {
+ "BriefDescription": "Counts the number of issue slots every cycle that were not delivered by the frontend due to ITLB misses.",
+ "EventCode": "0x71",
+ "EventName": "TOPDOWN_FE_BOUND.ITLB",
+ "PublicDescription": "Counts the number of issue slots every cycle that were not delivered by the frontend due to Instruction Table Lookaside Buffer (ITLB) misses.",
+ "SampleAfterValue": "1000003",
+ "UMask": "0x10",
+ "Unit": "cpu_atom"
+ },
+ {
+ "BriefDescription": "Counts the number of issue slots every cycle that were not delivered by the frontend due to other common frontend stalls not categorized.",
+ "EventCode": "0x71",
+ "EventName": "TOPDOWN_FE_BOUND.OTHER",
+ "SampleAfterValue": "1000003",
+ "UMask": "0x80",
+ "Unit": "cpu_atom"
+ },
+ {
+ "BriefDescription": "Counts the number of issue slots every cycle that were not delivered by the frontend due to wrong predecodes.",
+ "EventCode": "0x71",
+ "EventName": "TOPDOWN_FE_BOUND.PREDECODE",
+ "SampleAfterValue": "1000003",
+ "UMask": "0x4",
+ "Unit": "cpu_atom"
+ },
+ {
+ "BriefDescription": "Counts the total number of consumed retirement slots.",
+ "EventCode": "0xc2",
+ "EventName": "TOPDOWN_RETIRING.ALL",
+ "PEBS": "1",
+ "SampleAfterValue": "1000003",
+ "Unit": "cpu_atom"
+ },
+ {
"BriefDescription": "UOPS_DECODED.DEC0_UOPS",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3",
"EventCode": "0x76",
"EventName": "UOPS_DECODED.DEC0_UOPS",
- "PEBScounters": "0,1,2,3",
"SampleAfterValue": "1000003",
- "Speculative": "1",
"UMask": "0x1",
"Unit": "cpu_core"
},
{
"BriefDescription": "Uops executed on port 0",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5,6,7",
"EventCode": "0xb2",
"EventName": "UOPS_DISPATCHED.PORT_0",
- "PEBScounters": "0,1,2,3,4,5,6,7",
+ "PublicDescription": "Number of uops dispatch to execution port 0.",
"SampleAfterValue": "2000003",
- "Speculative": "1",
"UMask": "0x1",
"Unit": "cpu_core"
},
{
"BriefDescription": "Uops executed on port 1",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5,6,7",
"EventCode": "0xb2",
"EventName": "UOPS_DISPATCHED.PORT_1",
- "PEBScounters": "0,1,2,3,4,5,6,7",
+ "PublicDescription": "Number of uops dispatch to execution port 1.",
"SampleAfterValue": "2000003",
- "Speculative": "1",
"UMask": "0x2",
"Unit": "cpu_core"
},
{
"BriefDescription": "Uops executed on ports 2, 3 and 10",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5,6,7",
"EventCode": "0xb2",
"EventName": "UOPS_DISPATCHED.PORT_2_3_10",
- "PEBScounters": "0,1,2,3,4,5,6,7",
+ "PublicDescription": "Number of uops dispatch to execution ports 2, 3 and 10",
"SampleAfterValue": "2000003",
- "Speculative": "1",
"UMask": "0x4",
"Unit": "cpu_core"
},
{
"BriefDescription": "Uops executed on ports 4 and 9",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5,6,7",
"EventCode": "0xb2",
"EventName": "UOPS_DISPATCHED.PORT_4_9",
- "PEBScounters": "0,1,2,3,4,5,6,7",
+ "PublicDescription": "Number of uops dispatch to execution ports 4 and 9",
"SampleAfterValue": "2000003",
- "Speculative": "1",
"UMask": "0x10",
"Unit": "cpu_core"
},
{
"BriefDescription": "Uops executed on ports 5 and 11",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5,6,7",
"EventCode": "0xb2",
"EventName": "UOPS_DISPATCHED.PORT_5_11",
- "PEBScounters": "0,1,2,3,4,5,6,7",
+ "PublicDescription": "Number of uops dispatch to execution ports 5 and 11",
"SampleAfterValue": "2000003",
- "Speculative": "1",
"UMask": "0x20",
"Unit": "cpu_core"
},
{
"BriefDescription": "Uops executed on port 6",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5,6,7",
"EventCode": "0xb2",
"EventName": "UOPS_DISPATCHED.PORT_6",
- "PEBScounters": "0,1,2,3,4,5,6,7",
+ "PublicDescription": "Number of uops dispatch to execution port 6.",
"SampleAfterValue": "2000003",
- "Speculative": "1",
"UMask": "0x40",
"Unit": "cpu_core"
},
{
"BriefDescription": "Uops executed on ports 7 and 8",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5,6,7",
"EventCode": "0xb2",
"EventName": "UOPS_DISPATCHED.PORT_7_8",
- "PEBScounters": "0,1,2,3,4,5,6,7",
+ "PublicDescription": "Number of uops dispatch to execution ports 7 and 8.",
"SampleAfterValue": "2000003",
- "Speculative": "1",
"UMask": "0x80",
"Unit": "cpu_core"
},
{
"BriefDescription": "Cycles at least 1 micro-op is executed from any thread on physical core.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5,6,7",
"CounterMask": "1",
"EventCode": "0xb1",
"EventName": "UOPS_EXECUTED.CORE_CYCLES_GE_1",
- "PEBScounters": "0,1,2,3,4,5,6,7",
+ "PublicDescription": "Counts cycles when at least 1 micro-op is executed from any thread on physical core.",
"SampleAfterValue": "2000003",
- "Speculative": "1",
"UMask": "0x2",
"Unit": "cpu_core"
},
{
"BriefDescription": "Cycles at least 2 micro-op is executed from any thread on physical core.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5,6,7",
"CounterMask": "2",
"EventCode": "0xb1",
"EventName": "UOPS_EXECUTED.CORE_CYCLES_GE_2",
- "PEBScounters": "0,1,2,3,4,5,6,7",
+ "PublicDescription": "Counts cycles when at least 2 micro-ops are executed from any thread on physical core.",
"SampleAfterValue": "2000003",
- "Speculative": "1",
"UMask": "0x2",
"Unit": "cpu_core"
},
{
"BriefDescription": "Cycles at least 3 micro-op is executed from any thread on physical core.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5,6,7",
"CounterMask": "3",
"EventCode": "0xb1",
"EventName": "UOPS_EXECUTED.CORE_CYCLES_GE_3",
- "PEBScounters": "0,1,2,3,4,5,6,7",
+ "PublicDescription": "Counts cycles when at least 3 micro-ops are executed from any thread on physical core.",
"SampleAfterValue": "2000003",
- "Speculative": "1",
"UMask": "0x2",
"Unit": "cpu_core"
},
{
"BriefDescription": "Cycles at least 4 micro-op is executed from any thread on physical core.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5,6,7",
"CounterMask": "4",
"EventCode": "0xb1",
"EventName": "UOPS_EXECUTED.CORE_CYCLES_GE_4",
- "PEBScounters": "0,1,2,3,4,5,6,7",
+ "PublicDescription": "Counts cycles when at least 4 micro-ops are executed from any thread on physical core.",
"SampleAfterValue": "2000003",
- "Speculative": "1",
"UMask": "0x2",
"Unit": "cpu_core"
},
{
"BriefDescription": "Cycles where at least 1 uop was executed per-thread",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5,6,7",
"CounterMask": "1",
"EventCode": "0xb1",
"EventName": "UOPS_EXECUTED.CYCLES_GE_1",
- "PEBScounters": "0,1,2,3,4,5,6,7",
+ "PublicDescription": "Cycles where at least 1 uop was executed per-thread.",
"SampleAfterValue": "2000003",
- "Speculative": "1",
"UMask": "0x1",
"Unit": "cpu_core"
},
{
"BriefDescription": "Cycles where at least 2 uops were executed per-thread",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5,6,7",
"CounterMask": "2",
"EventCode": "0xb1",
"EventName": "UOPS_EXECUTED.CYCLES_GE_2",
- "PEBScounters": "0,1,2,3,4,5,6,7",
+ "PublicDescription": "Cycles where at least 2 uops were executed per-thread.",
"SampleAfterValue": "2000003",
- "Speculative": "1",
"UMask": "0x1",
"Unit": "cpu_core"
},
{
"BriefDescription": "Cycles where at least 3 uops were executed per-thread",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5,6,7",
"CounterMask": "3",
"EventCode": "0xb1",
"EventName": "UOPS_EXECUTED.CYCLES_GE_3",
- "PEBScounters": "0,1,2,3,4,5,6,7",
+ "PublicDescription": "Cycles where at least 3 uops were executed per-thread.",
"SampleAfterValue": "2000003",
- "Speculative": "1",
"UMask": "0x1",
"Unit": "cpu_core"
},
{
"BriefDescription": "Cycles where at least 4 uops were executed per-thread",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5,6,7",
"CounterMask": "4",
"EventCode": "0xb1",
"EventName": "UOPS_EXECUTED.CYCLES_GE_4",
- "PEBScounters": "0,1,2,3,4,5,6,7",
+ "PublicDescription": "Cycles where at least 4 uops were executed per-thread.",
"SampleAfterValue": "2000003",
- "Speculative": "1",
"UMask": "0x1",
"Unit": "cpu_core"
},
{
"BriefDescription": "Counts number of cycles no uops were dispatched to be executed on this thread.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5,6,7",
"CounterMask": "1",
"EventCode": "0xb1",
"EventName": "UOPS_EXECUTED.STALLS",
"Invert": "1",
- "PEBScounters": "0,1,2,3,4,5,6,7",
+ "PublicDescription": "Counts cycles during which no uops were dispatched from the Reservation Station (RS) per thread.",
"SampleAfterValue": "2000003",
- "Speculative": "1",
"UMask": "0x1",
"Unit": "cpu_core"
},
{
"BriefDescription": "This event is deprecated. Refer to new event UOPS_EXECUTED.STALLS",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5,6,7",
"CounterMask": "1",
+ "Deprecated": "1",
"EventCode": "0xb1",
"EventName": "UOPS_EXECUTED.STALL_CYCLES",
"Invert": "1",
- "PEBScounters": "0,1,2,3,4,5,6,7",
"SampleAfterValue": "2000003",
- "Speculative": "1",
"UMask": "0x1",
"Unit": "cpu_core"
},
{
"BriefDescription": "Counts the number of uops to be executed per-thread each cycle.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5,6,7",
"EventCode": "0xb1",
"EventName": "UOPS_EXECUTED.THREAD",
- "PEBScounters": "0,1,2,3,4,5,6,7",
"SampleAfterValue": "2000003",
- "Speculative": "1",
"UMask": "0x1",
"Unit": "cpu_core"
},
{
"BriefDescription": "Counts the number of x87 uops dispatched.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5,6,7",
"EventCode": "0xb1",
"EventName": "UOPS_EXECUTED.X87",
- "PEBScounters": "0,1,2,3,4,5,6,7",
+ "PublicDescription": "Counts the number of x87 uops executed.",
"SampleAfterValue": "2000003",
- "Speculative": "1",
"UMask": "0x10",
"Unit": "cpu_core"
},
{
"BriefDescription": "Uops that RAT issues to RS",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5,6,7",
"EventCode": "0xae",
"EventName": "UOPS_ISSUED.ANY",
- "PEBScounters": "0,1,2,3,4,5,6,7",
+ "PublicDescription": "Counts the number of uops that the Resource Allocation Table (RAT) issues to the Reservation Station (RS).",
"SampleAfterValue": "2000003",
- "Speculative": "1",
"UMask": "0x1",
"Unit": "cpu_core"
},
{
+ "BriefDescription": "Counts the total number of uops retired.",
+ "EventCode": "0xc2",
+ "EventName": "UOPS_RETIRED.ALL",
+ "PEBS": "1",
+ "SampleAfterValue": "2000003",
+ "Unit": "cpu_atom"
+ },
+ {
"BriefDescription": "Cycles with retired uop(s).",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5,6,7",
"CounterMask": "1",
"EventCode": "0xc2",
"EventName": "UOPS_RETIRED.CYCLES",
- "PEBScounters": "0,1,2,3,4,5,6,7",
+ "PublicDescription": "Counts cycles where at least one uop has retired.",
"SampleAfterValue": "1000003",
"UMask": "0x2",
"Unit": "cpu_core"
},
{
"BriefDescription": "Retired uops except the last uop of each instruction.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5,6,7",
"EventCode": "0xc2",
"EventName": "UOPS_RETIRED.HEAVY",
- "PEBScounters": "0,1,2,3,4,5,6,7",
+ "PublicDescription": "Counts the number of retired micro-operations (uops) except the last uop of each instruction. An instruction that is decoded into less than two uops does not contribute to the count.",
"SampleAfterValue": "2000003",
"UMask": "0x1",
"Unit": "cpu_core"
},
{
+ "BriefDescription": "Counts the number of integer divide uops retired.",
+ "EventCode": "0xc2",
+ "EventName": "UOPS_RETIRED.IDIV",
+ "PEBS": "1",
+ "SampleAfterValue": "2000003",
+ "UMask": "0x10",
+ "Unit": "cpu_atom"
+ },
+ {
+ "BriefDescription": "Counts the number of uops that are from complex flows issued by the micro-sequencer (MS).",
+ "EventCode": "0xc2",
+ "EventName": "UOPS_RETIRED.MS",
+ "PEBS": "1",
+ "PublicDescription": "Counts the number of uops that are from complex flows issued by the Microcode Sequencer (MS). This includes uops from flows due to complex instructions, faults, assists, and inserted flows.",
+ "SampleAfterValue": "2000003",
+ "UMask": "0x1",
+ "Unit": "cpu_atom"
+ },
+ {
"BriefDescription": "UOPS_RETIRED.MS",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5,6,7",
"EventCode": "0xc2",
"EventName": "UOPS_RETIRED.MS",
"MSRIndex": "0x3F7",
"MSRValue": "0x8",
- "PEBScounters": "0,1,2,3,4,5,6,7",
"SampleAfterValue": "2000003",
- "TakenAlone": "1",
"UMask": "0x4",
"Unit": "cpu_core"
},
{
"BriefDescription": "Retirement slots used.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5,6,7",
"EventCode": "0xc2",
"EventName": "UOPS_RETIRED.SLOTS",
- "PEBScounters": "0,1,2,3,4,5,6,7",
+ "PublicDescription": "Counts the retirement slots used each cycle.",
"SampleAfterValue": "2000003",
"UMask": "0x2",
"Unit": "cpu_core"
},
{
"BriefDescription": "Cycles without actually retired uops.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5,6,7",
"CounterMask": "1",
"EventCode": "0xc2",
"EventName": "UOPS_RETIRED.STALLS",
"Invert": "1",
- "PEBScounters": "0,1,2,3,4,5,6,7",
+ "PublicDescription": "This event counts cycles without actually retired uops.",
"SampleAfterValue": "1000003",
"UMask": "0x2",
"Unit": "cpu_core"
},
{
"BriefDescription": "This event is deprecated. Refer to new event UOPS_RETIRED.STALLS",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5,6,7",
"CounterMask": "1",
+ "Deprecated": "1",
"EventCode": "0xc2",
"EventName": "UOPS_RETIRED.STALL_CYCLES",
"Invert": "1",
- "PEBScounters": "0,1,2,3,4,5,6,7",
"SampleAfterValue": "1000003",
"UMask": "0x2",
"Unit": "cpu_core"
+ },
+ {
+ "BriefDescription": "Counts the number of x87 uops retired, includes those in MS flows.",
+ "EventCode": "0xc2",
+ "EventName": "UOPS_RETIRED.X87",
+ "PEBS": "1",
+ "SampleAfterValue": "2000003",
+ "UMask": "0x2",
+ "Unit": "cpu_atom"
}
]
diff --git a/tools/perf/pmu-events/arch/x86/alderlake/uncore-memory.json b/tools/perf/pmu-events/arch/x86/alderlake/uncore-memory.json
index d82d6f6..2ccd9cf 100644
--- a/tools/perf/pmu-events/arch/x86/alderlake/uncore-memory.json
+++ b/tools/perf/pmu-events/arch/x86/alderlake/uncore-memory.json
@@ -1,134 +1,53 @@
[
{
- "BriefDescription": "Number of clocks",
- "Counter": "0,1,2,3,4",
- "CounterType": "PGMABLE",
- "EventCode": "0x01",
- "EventName": "UNC_M_CLOCKTICKS",
+ "BriefDescription": "Counts every 64B read request entering the Memory Controller 0 to DRAM (sum of all channels).",
+ "EventName": "UNC_MC0_RDCAS_COUNT_FREERUN",
+ "PerPkg": "1",
+ "PublicDescription": "Counts every 64B read request entering the Memory Controller 0 to DRAM (sum of all channels).",
+ "Unit": "iMC"
+ },
+ {
+ "BriefDescription": "Counts every 64B write request entering the Memory Controller 0 to DRAM (sum of all channels). Each write request counts as a new request incrementing this counter. However, same cache line write requests (both full and partial) are combined to a single 64 byte data transfer to DRAM.",
+ "EventName": "UNC_MC0_WRCAS_COUNT_FREERUN",
"PerPkg": "1",
"Unit": "iMC"
},
{
- "BriefDescription": "Incoming VC0 read request",
- "Counter": "0,1,2,3,4",
- "CounterType": "PGMABLE",
- "EventCode": "0x02",
- "EventName": "UNC_M_VC0_REQUESTS_RD",
+ "BriefDescription": "Counts every 64B read request entering the Memory Controller 1 to DRAM (sum of all channels).",
+ "EventName": "UNC_MC1_RDCAS_COUNT_FREERUN",
+ "PerPkg": "1",
+ "PublicDescription": "Counts every 64B read entering the Memory Controller 1 to DRAM (sum of all channels).",
+ "Unit": "iMC"
+ },
+ {
+ "BriefDescription": "Counts every 64B write request entering the Memory Controller 1 to DRAM (sum of all channels). Each write request counts as a new request incrementing this counter. However, same cache line write requests (both full and partial) are combined to a single 64 byte data transfer to DRAM.",
+ "EventName": "UNC_MC1_WRCAS_COUNT_FREERUN",
"PerPkg": "1",
"Unit": "iMC"
},
{
- "BriefDescription": "Incoming VC0 write request",
- "Counter": "0,1,2,3,4",
- "CounterType": "PGMABLE",
- "EventCode": "0x03",
- "EventName": "UNC_M_VC0_REQUESTS_WR",
+ "BriefDescription": "ACT command for a read request sent to DRAM",
+ "EventCode": "0x24",
+ "EventName": "UNC_M_ACT_COUNT_RD",
"PerPkg": "1",
"Unit": "iMC"
},
{
- "BriefDescription": "Incoming VC1 read request",
- "Counter": "0,1,2,3,4",
- "CounterType": "PGMABLE",
- "EventCode": "0x04",
- "EventName": "UNC_M_VC1_REQUESTS_RD",
+ "BriefDescription": "ACT command sent to DRAM",
+ "EventCode": "0x26",
+ "EventName": "UNC_M_ACT_COUNT_TOTAL",
"PerPkg": "1",
"Unit": "iMC"
},
{
- "BriefDescription": "Incoming VC1 write request",
- "Counter": "0,1,2,3,4",
- "CounterType": "PGMABLE",
- "EventCode": "0x05",
- "EventName": "UNC_M_VC1_REQUESTS_WR",
- "PerPkg": "1",
- "Unit": "iMC"
- },
- {
- "BriefDescription": "Incoming read prefetch request from IA",
- "Counter": "0,1,2,3,4",
- "CounterType": "PGMABLE",
- "EventCode": "0x0A",
- "EventName": "UNC_M_PREFETCH_RD",
- "PerPkg": "1",
- "Unit": "iMC"
- },
- {
- "BriefDescription": "Any Rank at Hot state",
- "Counter": "0,1,2,3,4",
- "CounterType": "PGMABLE",
- "EventCode": "0x19",
- "EventName": "UNC_M_DRAM_THERMAL_HOT",
- "PerPkg": "1",
- "Unit": "iMC"
- },
- {
- "BriefDescription": "Any Rank at Warm state",
- "Counter": "0,1,2,3,4",
- "CounterType": "PGMABLE",
- "EventCode": "0x1A",
- "EventName": "UNC_M_DRAM_THERMAL_WARM",
- "PerPkg": "1",
- "Unit": "iMC"
- },
- {
- "BriefDescription": "incoming read request page status is Page Hit",
- "Counter": "0,1,2,3,4",
- "CounterType": "PGMABLE",
- "EventCode": "0x1C",
- "EventName": "UNC_M_DRAM_PAGE_HIT_RD",
- "PerPkg": "1",
- "Unit": "iMC"
- },
- {
- "BriefDescription": "incoming read request page status is Page Empty",
- "Counter": "0,1,2,3,4",
- "CounterType": "PGMABLE",
- "EventCode": "0x1D",
- "EventName": "UNC_M_DRAM_PAGE_EMPTY_RD",
- "PerPkg": "1",
- "Unit": "iMC"
- },
- {
- "BriefDescription": "incoming read request page status is Page Miss",
- "Counter": "0,1,2,3,4",
- "CounterType": "PGMABLE",
- "EventCode": "0x1E",
- "EventName": "UNC_M_DRAM_PAGE_MISS_RD",
- "PerPkg": "1",
- "Unit": "iMC"
- },
- {
- "BriefDescription": "incoming write request page status is Page Hit",
- "Counter": "0,1,2,3,4",
- "CounterType": "PGMABLE",
- "EventCode": "0x1F",
- "EventName": "UNC_M_DRAM_PAGE_HIT_WR",
- "PerPkg": "1",
- "Unit": "iMC"
- },
- {
- "BriefDescription": "incoming write request page status is Page Empty",
- "Counter": "0,1,2,3,4",
- "CounterType": "PGMABLE",
- "EventCode": "0x20",
- "EventName": "UNC_M_DRAM_PAGE_EMPTY_WR",
- "PerPkg": "1",
- "Unit": "iMC"
- },
- {
- "BriefDescription": "incoming write request page status is Page Miss",
- "Counter": "0,1,2,3,4",
- "CounterType": "PGMABLE",
- "EventCode": "0x21",
- "EventName": "UNC_M_DRAM_PAGE_MISS_WR",
+ "BriefDescription": "ACT command for a write request sent to DRAM",
+ "EventCode": "0x25",
+ "EventName": "UNC_M_ACT_COUNT_WR",
"PerPkg": "1",
"Unit": "iMC"
},
{
"BriefDescription": "Read CAS command sent to DRAM",
- "Counter": "0,1,2,3,4",
- "CounterType": "PGMABLE",
"EventCode": "0x22",
"EventName": "UNC_M_CAS_COUNT_RD",
"PerPkg": "1",
@@ -136,86 +55,120 @@
},
{
"BriefDescription": "Write CAS command sent to DRAM",
- "Counter": "0,1,2,3,4",
- "CounterType": "PGMABLE",
"EventCode": "0x23",
"EventName": "UNC_M_CAS_COUNT_WR",
"PerPkg": "1",
"Unit": "iMC"
},
{
- "BriefDescription": "ACT command for a read request sent to DRAM",
- "Counter": "0,1,2,3,4",
- "CounterType": "PGMABLE",
- "EventCode": "0x24",
- "EventName": "UNC_M_ACT_COUNT_RD",
+ "BriefDescription": "Number of clocks",
+ "EventCode": "0x01",
+ "EventName": "UNC_M_CLOCKTICKS",
"PerPkg": "1",
"Unit": "iMC"
},
{
- "BriefDescription": "ACT command for a write request sent to DRAM",
- "Counter": "0,1,2,3,4",
- "CounterType": "PGMABLE",
- "EventCode": "0x25",
- "EventName": "UNC_M_ACT_COUNT_WR",
+ "BriefDescription": "incoming read request page status is Page Empty",
+ "EventCode": "0x1D",
+ "EventName": "UNC_M_DRAM_PAGE_EMPTY_RD",
"PerPkg": "1",
"Unit": "iMC"
},
{
- "BriefDescription": "ACT command sent to DRAM",
- "Counter": "0,1,2,3,4",
- "CounterType": "PGMABLE",
- "EventCode": "0x26",
- "EventName": "UNC_M_ACT_COUNT_TOTAL",
+ "BriefDescription": "incoming write request page status is Page Empty",
+ "EventCode": "0x20",
+ "EventName": "UNC_M_DRAM_PAGE_EMPTY_WR",
"PerPkg": "1",
"Unit": "iMC"
},
{
- "BriefDescription": "PRE command sent to DRAM for a read/write request",
- "Counter": "0,1,2,3,4",
- "CounterType": "PGMABLE",
- "EventCode": "0x27",
- "EventName": "UNC_M_PRE_COUNT_PAGE_MISS",
+ "BriefDescription": "incoming read request page status is Page Hit",
+ "EventCode": "0x1C",
+ "EventName": "UNC_M_DRAM_PAGE_HIT_RD",
+ "PerPkg": "1",
+ "Unit": "iMC"
+ },
+ {
+ "BriefDescription": "incoming write request page status is Page Hit",
+ "EventCode": "0x1F",
+ "EventName": "UNC_M_DRAM_PAGE_HIT_WR",
+ "PerPkg": "1",
+ "Unit": "iMC"
+ },
+ {
+ "BriefDescription": "incoming read request page status is Page Miss",
+ "EventCode": "0x1E",
+ "EventName": "UNC_M_DRAM_PAGE_MISS_RD",
+ "PerPkg": "1",
+ "Unit": "iMC"
+ },
+ {
+ "BriefDescription": "incoming write request page status is Page Miss",
+ "EventCode": "0x21",
+ "EventName": "UNC_M_DRAM_PAGE_MISS_WR",
+ "PerPkg": "1",
+ "Unit": "iMC"
+ },
+ {
+ "BriefDescription": "Any Rank at Hot state",
+ "EventCode": "0x19",
+ "EventName": "UNC_M_DRAM_THERMAL_HOT",
+ "PerPkg": "1",
+ "Unit": "iMC"
+ },
+ {
+ "BriefDescription": "Any Rank at Warm state",
+ "EventCode": "0x1A",
+ "EventName": "UNC_M_DRAM_THERMAL_WARM",
+ "PerPkg": "1",
+ "Unit": "iMC"
+ },
+ {
+ "BriefDescription": "Incoming read prefetch request from IA.",
+ "EventCode": "0x0A",
+ "EventName": "UNC_M_PREFETCH_RD",
"PerPkg": "1",
"Unit": "iMC"
},
{
"BriefDescription": "PRE command sent to DRAM due to page table idle timer expiration",
- "Counter": "0,1,2,3,4",
- "CounterType": "PGMABLE",
"EventCode": "0x28",
"EventName": "UNC_M_PRE_COUNT_IDLE",
"PerPkg": "1",
"Unit": "iMC"
},
{
- "BriefDescription": "Counts every 64B read request entering the Memory Controller 0 to DRAM (sum of all channels)",
- "CounterType": "FREERUN",
- "EventName": "UNC_MC0_RDCAS_COUNT_FREERUN",
+ "BriefDescription": "PRE command sent to DRAM for a read/write request",
+ "EventCode": "0x27",
+ "EventName": "UNC_M_PRE_COUNT_PAGE_MISS",
"PerPkg": "1",
"Unit": "iMC"
},
{
- "BriefDescription": "Counts every 64B read request entering the Memory Controller 1 to DRAM (sum of all channels)",
- "Counter": "3",
- "CounterType": "FREERUN",
- "EventName": "UNC_MC1_RDCAS_COUNT_FREERUN",
+ "BriefDescription": "Incoming VC0 read request",
+ "EventCode": "0x02",
+ "EventName": "UNC_M_VC0_REQUESTS_RD",
"PerPkg": "1",
"Unit": "iMC"
},
{
- "BriefDescription": "Counts every 64B write request entering the Memory Controller 0 to DRAM (sum of all channels). Each write request counts as a new request incrementing this counter. However, same cache line write requests (both full and partial) are combined to a single 64 byte data transfer to DRAM",
- "Counter": "1",
- "CounterType": "FREERUN",
- "EventName": "UNC_MC0_WRCAS_COUNT_FREERUN",
+ "BriefDescription": "Incoming VC0 write request",
+ "EventCode": "0x03",
+ "EventName": "UNC_M_VC0_REQUESTS_WR",
"PerPkg": "1",
"Unit": "iMC"
},
{
- "BriefDescription": "Counts every 64B write request entering the Memory Controller 1 to DRAM (sum of all channels). Each write request counts as a new request incrementing this counter. However, same cache line write requests (both full and partial) are combined to a single 64 byte data transfer to DRAM",
- "Counter": "4",
- "CounterType": "FREERUN",
- "EventName": "UNC_MC1_WRCAS_COUNT_FREERUN",
+ "BriefDescription": "Incoming VC1 read request",
+ "EventCode": "0x04",
+ "EventName": "UNC_M_VC1_REQUESTS_RD",
+ "PerPkg": "1",
+ "Unit": "iMC"
+ },
+ {
+ "BriefDescription": "Incoming VC1 write request",
+ "EventCode": "0x05",
+ "EventName": "UNC_M_VC1_REQUESTS_WR",
"PerPkg": "1",
"Unit": "iMC"
}
diff --git a/tools/perf/pmu-events/arch/x86/alderlake/uncore-other.json b/tools/perf/pmu-events/arch/x86/alderlake/uncore-other.json
index b1ae349..bc5fb6b 100644
--- a/tools/perf/pmu-events/arch/x86/alderlake/uncore-other.json
+++ b/tools/perf/pmu-events/arch/x86/alderlake/uncore-other.json
@@ -1,40 +1,73 @@
[
{
- "BriefDescription": "This 48-bit fixed counter counts the UCLK cycles",
- "Counter": "Fixed",
- "CounterType": "PGMABLE",
+ "BriefDescription": "Number of requests allocated in Coherency Tracker.",
+ "EventCode": "0x84",
+ "EventName": "UNC_ARB_COH_TRK_REQUESTS.ALL",
+ "PerPkg": "1",
+ "UMask": "0x1",
+ "Unit": "ARB"
+ },
+ {
+ "BriefDescription": "Each cycle counts number of any coherent request at memory controller that were issued by any core.",
+ "EventCode": "0x85",
+ "EventName": "UNC_ARB_DAT_OCCUPANCY.ALL",
+ "PerPkg": "1",
+ "UMask": "0x1",
+ "Unit": "ARB"
+ },
+ {
+ "BriefDescription": "Each cycle counts number of coherent reads pending on data return from memory controller that were issued by any core.",
+ "EventCode": "0x85",
+ "EventName": "UNC_ARB_DAT_OCCUPANCY.RD",
+ "PerPkg": "1",
+ "UMask": "0x2",
+ "Unit": "ARB"
+ },
+ {
+ "BriefDescription": "Number of coherent read requests sent to memory controller that were issued by any core.",
+ "EventCode": "0x81",
+ "EventName": "UNC_ARB_DAT_REQUESTS.RD",
+ "PerPkg": "1",
+ "UMask": "0x2",
+ "Unit": "ARB"
+ },
+ {
+ "BriefDescription": "This event is deprecated. Refer to new event UNC_ARB_DAT_OCCUPANCY.ALL",
+ "EventCode": "0x85",
+ "EventName": "UNC_ARB_IFA_OCCUPANCY.ALL",
+ "PerPkg": "1",
+ "UMask": "0x1",
+ "Unit": "ARB"
+ },
+ {
+ "BriefDescription": "This event is deprecated. Refer to new event UNC_ARB_DAT_REQUESTS.RD",
+ "EventCode": "0x81",
+ "EventName": "UNC_ARB_REQ_TRK_REQUEST.DRD",
+ "PerPkg": "1",
+ "UMask": "0x2",
+ "Unit": "ARB"
+ },
+ {
+ "BriefDescription": "Each cycle counts number of all outgoing valid entries in ReqTrk. Such entry is defined as valid from its allocation in ReqTrk till deallocation. Accounts for Coherent and non-coherent traffic.",
+ "EventCode": "0x80",
+ "EventName": "UNC_ARB_TRK_OCCUPANCY.ALL",
+ "PerPkg": "1",
+ "UMask": "0x1",
+ "Unit": "ARB"
+ },
+ {
+ "BriefDescription": "Counts the number of coherent and in-coherent requests initiated by IA cores, processor graphic units, or LLC.",
+ "EventCode": "0x81",
+ "EventName": "UNC_ARB_TRK_REQUESTS.ALL",
+ "PerPkg": "1",
+ "UMask": "0x1",
+ "Unit": "ARB"
+ },
+ {
+ "BriefDescription": "This 48-bit fixed counter counts the UCLK cycles.",
"EventCode": "0xff",
"EventName": "UNC_CLOCK.SOCKET",
"PerPkg": "1",
"Unit": "CLOCK"
- },
- {
- "BriefDescription": "Counts the number of coherent and in-coherent requests initiated by IA cores, processor graphic units, or LLC",
- "Counter": "0,1",
- "CounterType": "PGMABLE",
- "EventCode": "0x81",
- "EventName": "UNC_ARB_TRK_REQUESTS.ALL",
- "PerPkg": "1",
- "UMask": "0x01",
- "Unit": "ARB"
- },
- {
- "BriefDescription": "Number of requests allocated in Coherency Tracker",
- "Counter": "0,1",
- "CounterType": "PGMABLE",
- "EventCode": "0x84",
- "EventName": "UNC_ARB_COH_TRK_REQUESTS.ALL",
- "PerPkg": "1",
- "UMask": "0x01",
- "Unit": "ARB"
- },
- {
- "BriefDescription": "Each cycle counts number of all outgoing valid entries in ReqTrk. Such entry is defined as valid from its allocation in ReqTrk till deallocation. Accounts for Coherent and non-coherent traffic",
- "CounterType": "PGMABLE",
- "EventCode": "0x80",
- "EventName": "UNC_ARB_TRK_OCCUPANCY.ALL",
- "PerPkg": "1",
- "UMask": "0x01",
- "Unit": "ARB"
}
]
diff --git a/tools/perf/pmu-events/arch/x86/alderlake/virtual-memory.json b/tools/perf/pmu-events/arch/x86/alderlake/virtual-memory.json
index 12baf76..3827d29 100644
--- a/tools/perf/pmu-events/arch/x86/alderlake/virtual-memory.json
+++ b/tools/perf/pmu-events/arch/x86/alderlake/virtual-memory.json
@@ -1,317 +1,236 @@
[
{
- "BriefDescription": "Counts the number of page walks completed due to load DTLB misses to any page size.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5",
- "EventCode": "0x08",
- "EventName": "DTLB_LOAD_MISSES.WALK_COMPLETED",
- "PEBScounters": "0,1,2,3,4,5",
- "SampleAfterValue": "200003",
- "Speculative": "1",
- "UMask": "0xe",
- "Unit": "cpu_atom"
- },
- {
- "BriefDescription": "Counts the number of page walks completed due to store DTLB misses to any page size.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5",
- "EventCode": "0x49",
- "EventName": "DTLB_STORE_MISSES.WALK_COMPLETED",
- "PEBScounters": "0,1,2,3,4,5",
- "SampleAfterValue": "2000003",
- "Speculative": "1",
- "UMask": "0xe",
- "Unit": "cpu_atom"
- },
- {
- "BriefDescription": "Counts the number of page walks initiated by a instruction fetch that missed the first and second level TLBs.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5",
- "EventCode": "0x85",
- "EventName": "ITLB_MISSES.MISS_CAUSED_WALK",
- "PEBScounters": "0,1,2,3,4,5",
- "SampleAfterValue": "1000003",
- "Speculative": "1",
- "UMask": "0x1",
- "Unit": "cpu_atom"
- },
- {
- "BriefDescription": "Counts the number of page walks due to an instruction fetch that miss the PDE (Page Directory Entry) cache.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5",
- "EventCode": "0x85",
- "EventName": "ITLB_MISSES.PDE_CACHE_MISS",
- "PEBScounters": "0,1,2,3,4,5",
- "SampleAfterValue": "2000003",
- "Speculative": "1",
- "UMask": "0x80",
- "Unit": "cpu_atom"
- },
- {
- "BriefDescription": "Counts the number of page walks completed due to instruction fetch misses to any page size.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5",
- "EventCode": "0x85",
- "EventName": "ITLB_MISSES.WALK_COMPLETED",
- "PEBScounters": "0,1,2,3,4,5",
- "SampleAfterValue": "200003",
- "Speculative": "1",
- "UMask": "0xe",
- "Unit": "cpu_atom"
- },
- {
- "BriefDescription": "Counts the number of cycles that the head (oldest load) of the load buffer and retirement are both stalled due to a DTLB miss.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3,4,5",
- "EventCode": "0x05",
- "EventName": "LD_HEAD.DTLB_MISS_AT_RET",
- "PEBScounters": "0,1,2,3,4,5",
- "SampleAfterValue": "1000003",
- "Speculative": "1",
- "UMask": "0x90",
- "Unit": "cpu_atom"
- },
- {
"BriefDescription": "Loads that miss the DTLB and hit the STLB.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3",
"EventCode": "0x12",
"EventName": "DTLB_LOAD_MISSES.STLB_HIT",
- "PEBScounters": "0,1,2,3",
+ "PublicDescription": "Counts loads that miss the DTLB (Data TLB) and hit the STLB (Second level TLB).",
"SampleAfterValue": "100003",
- "Speculative": "1",
"UMask": "0x20",
"Unit": "cpu_core"
},
{
"BriefDescription": "Cycles when at least one PMH is busy with a page walk for a demand load.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3",
"CounterMask": "1",
"EventCode": "0x12",
"EventName": "DTLB_LOAD_MISSES.WALK_ACTIVE",
- "PEBScounters": "0,1,2,3",
+ "PublicDescription": "Counts cycles when at least one PMH (Page Miss Handler) is busy with a page walk for a demand load.",
"SampleAfterValue": "100003",
- "Speculative": "1",
"UMask": "0x10",
"Unit": "cpu_core"
},
{
+ "BriefDescription": "Counts the number of page walks completed due to load DTLB misses to any page size.",
+ "EventCode": "0x08",
+ "EventName": "DTLB_LOAD_MISSES.WALK_COMPLETED",
+ "PublicDescription": "Counts the number of page walks completed due to loads (including SW prefetches) whose address translations missed in all Translation Lookaside Buffer (TLB) levels and were mapped to any page size. Includes page walks that page fault.",
+ "SampleAfterValue": "200003",
+ "UMask": "0xe",
+ "Unit": "cpu_atom"
+ },
+ {
"BriefDescription": "Load miss in all TLB levels causes a page walk that completes. (All page sizes)",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3",
"EventCode": "0x12",
"EventName": "DTLB_LOAD_MISSES.WALK_COMPLETED",
- "PEBScounters": "0,1,2,3",
+ "PublicDescription": "Counts completed page walks (all page sizes) caused by demand data loads. This implies it missed in the DTLB and further levels of TLB. The page walk can end with or without a fault.",
"SampleAfterValue": "100003",
- "Speculative": "1",
"UMask": "0xe",
"Unit": "cpu_core"
},
{
"BriefDescription": "Page walks completed due to a demand data load to a 1G page.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3",
"EventCode": "0x12",
"EventName": "DTLB_LOAD_MISSES.WALK_COMPLETED_1G",
- "PEBScounters": "0,1,2,3",
+ "PublicDescription": "Counts completed page walks (1G sizes) caused by demand data loads. This implies address translations missed in the DTLB and further levels of TLB. The page walk can end with or without a fault.",
"SampleAfterValue": "100003",
- "Speculative": "1",
"UMask": "0x8",
"Unit": "cpu_core"
},
{
"BriefDescription": "Page walks completed due to a demand data load to a 2M/4M page.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3",
"EventCode": "0x12",
"EventName": "DTLB_LOAD_MISSES.WALK_COMPLETED_2M_4M",
- "PEBScounters": "0,1,2,3",
+ "PublicDescription": "Counts completed page walks (2M/4M sizes) caused by demand data loads. This implies address translations missed in the DTLB and further levels of TLB. The page walk can end with or without a fault.",
"SampleAfterValue": "100003",
- "Speculative": "1",
"UMask": "0x4",
"Unit": "cpu_core"
},
{
"BriefDescription": "Page walks completed due to a demand data load to a 4K page.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3",
"EventCode": "0x12",
"EventName": "DTLB_LOAD_MISSES.WALK_COMPLETED_4K",
- "PEBScounters": "0,1,2,3",
+ "PublicDescription": "Counts completed page walks (4K sizes) caused by demand data loads. This implies address translations missed in the DTLB and further levels of TLB. The page walk can end with or without a fault.",
"SampleAfterValue": "100003",
- "Speculative": "1",
"UMask": "0x2",
"Unit": "cpu_core"
},
{
"BriefDescription": "Number of page walks outstanding for a demand load in the PMH each cycle.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3",
"EventCode": "0x12",
"EventName": "DTLB_LOAD_MISSES.WALK_PENDING",
- "PEBScounters": "0,1,2,3",
+ "PublicDescription": "Counts the number of page walks outstanding for a demand load in the PMH (Page Miss Handler) each cycle.",
"SampleAfterValue": "100003",
- "Speculative": "1",
"UMask": "0x10",
"Unit": "cpu_core"
},
{
"BriefDescription": "Stores that miss the DTLB and hit the STLB.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3",
"EventCode": "0x13",
"EventName": "DTLB_STORE_MISSES.STLB_HIT",
- "PEBScounters": "0,1,2,3",
+ "PublicDescription": "Counts stores that miss the DTLB (Data TLB) and hit the STLB (2nd Level TLB).",
"SampleAfterValue": "100003",
- "Speculative": "1",
"UMask": "0x20",
"Unit": "cpu_core"
},
{
"BriefDescription": "Cycles when at least one PMH is busy with a page walk for a store.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3",
"CounterMask": "1",
"EventCode": "0x13",
"EventName": "DTLB_STORE_MISSES.WALK_ACTIVE",
- "PEBScounters": "0,1,2,3",
+ "PublicDescription": "Counts cycles when at least one PMH (Page Miss Handler) is busy with a page walk for a store.",
"SampleAfterValue": "100003",
- "Speculative": "1",
"UMask": "0x10",
"Unit": "cpu_core"
},
{
+ "BriefDescription": "Counts the number of page walks completed due to store DTLB misses to any page size.",
+ "EventCode": "0x49",
+ "EventName": "DTLB_STORE_MISSES.WALK_COMPLETED",
+ "PublicDescription": "Counts the number of page walks completed due to stores whose address translations missed in all Translation Lookaside Buffer (TLB) levels and were mapped to any page size. Includes page walks that page fault.",
+ "SampleAfterValue": "2000003",
+ "UMask": "0xe",
+ "Unit": "cpu_atom"
+ },
+ {
"BriefDescription": "Store misses in all TLB levels causes a page walk that completes. (All page sizes)",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3",
"EventCode": "0x13",
"EventName": "DTLB_STORE_MISSES.WALK_COMPLETED",
- "PEBScounters": "0,1,2,3",
+ "PublicDescription": "Counts completed page walks (all page sizes) caused by demand data stores. This implies it missed in the DTLB and further levels of TLB. The page walk can end with or without a fault.",
"SampleAfterValue": "100003",
- "Speculative": "1",
"UMask": "0xe",
"Unit": "cpu_core"
},
{
"BriefDescription": "Page walks completed due to a demand data store to a 1G page.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3",
"EventCode": "0x13",
"EventName": "DTLB_STORE_MISSES.WALK_COMPLETED_1G",
- "PEBScounters": "0,1,2,3",
+ "PublicDescription": "Counts completed page walks (1G sizes) caused by demand data stores. This implies address translations missed in the DTLB and further levels of TLB. The page walk can end with or without a fault.",
"SampleAfterValue": "100003",
- "Speculative": "1",
"UMask": "0x8",
"Unit": "cpu_core"
},
{
"BriefDescription": "Page walks completed due to a demand data store to a 2M/4M page.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3",
"EventCode": "0x13",
"EventName": "DTLB_STORE_MISSES.WALK_COMPLETED_2M_4M",
- "PEBScounters": "0,1,2,3",
+ "PublicDescription": "Counts completed page walks (2M/4M sizes) caused by demand data stores. This implies address translations missed in the DTLB and further levels of TLB. The page walk can end with or without a fault.",
"SampleAfterValue": "100003",
- "Speculative": "1",
"UMask": "0x4",
"Unit": "cpu_core"
},
{
"BriefDescription": "Page walks completed due to a demand data store to a 4K page.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3",
"EventCode": "0x13",
"EventName": "DTLB_STORE_MISSES.WALK_COMPLETED_4K",
- "PEBScounters": "0,1,2,3",
+ "PublicDescription": "Counts completed page walks (4K sizes) caused by demand data stores. This implies address translations missed in the DTLB and further levels of TLB. The page walk can end with or without a fault.",
"SampleAfterValue": "100003",
- "Speculative": "1",
"UMask": "0x2",
"Unit": "cpu_core"
},
{
"BriefDescription": "Number of page walks outstanding for a store in the PMH each cycle.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3",
"EventCode": "0x13",
"EventName": "DTLB_STORE_MISSES.WALK_PENDING",
- "PEBScounters": "0,1,2,3",
+ "PublicDescription": "Counts the number of page walks outstanding for a store in the PMH (Page Miss Handler) each cycle.",
"SampleAfterValue": "100003",
- "Speculative": "1",
"UMask": "0x10",
"Unit": "cpu_core"
},
{
+ "BriefDescription": "Counts the number of page walks initiated by a instruction fetch that missed the first and second level TLBs.",
+ "EventCode": "0x85",
+ "EventName": "ITLB_MISSES.MISS_CAUSED_WALK",
+ "SampleAfterValue": "1000003",
+ "UMask": "0x1",
+ "Unit": "cpu_atom"
+ },
+ {
+ "BriefDescription": "Counts the number of page walks due to an instruction fetch that miss the PDE (Page Directory Entry) cache.",
+ "EventCode": "0x85",
+ "EventName": "ITLB_MISSES.PDE_CACHE_MISS",
+ "SampleAfterValue": "2000003",
+ "UMask": "0x80",
+ "Unit": "cpu_atom"
+ },
+ {
"BriefDescription": "Instruction fetch requests that miss the ITLB and hit the STLB.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3",
"EventCode": "0x11",
"EventName": "ITLB_MISSES.STLB_HIT",
- "PEBScounters": "0,1,2,3",
+ "PublicDescription": "Counts instruction fetch requests that miss the ITLB (Instruction TLB) and hit the STLB (Second-level TLB).",
"SampleAfterValue": "100003",
- "Speculative": "1",
"UMask": "0x20",
"Unit": "cpu_core"
},
{
"BriefDescription": "Cycles when at least one PMH is busy with a page walk for code (instruction fetch) request.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3",
"CounterMask": "1",
"EventCode": "0x11",
"EventName": "ITLB_MISSES.WALK_ACTIVE",
- "PEBScounters": "0,1,2,3",
+ "PublicDescription": "Counts cycles when at least one PMH (Page Miss Handler) is busy with a page walk for a code (instruction fetch) request.",
"SampleAfterValue": "100003",
- "Speculative": "1",
"UMask": "0x10",
"Unit": "cpu_core"
},
{
+ "BriefDescription": "Counts the number of page walks completed due to instruction fetch misses to any page size.",
+ "EventCode": "0x85",
+ "EventName": "ITLB_MISSES.WALK_COMPLETED",
+ "PublicDescription": "Counts the number of page walks completed due to instruction fetches whose address translations missed in all Translation Lookaside Buffer (TLB) levels and were mapped to any page size. Includes page walks that page fault.",
+ "SampleAfterValue": "200003",
+ "UMask": "0xe",
+ "Unit": "cpu_atom"
+ },
+ {
"BriefDescription": "Code miss in all TLB levels causes a page walk that completes. (All page sizes)",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3",
"EventCode": "0x11",
"EventName": "ITLB_MISSES.WALK_COMPLETED",
- "PEBScounters": "0,1,2,3",
+ "PublicDescription": "Counts completed page walks (all page sizes) caused by a code fetch. This implies it missed in the ITLB (Instruction TLB) and further levels of TLB. The page walk can end with or without a fault.",
"SampleAfterValue": "100003",
- "Speculative": "1",
"UMask": "0xe",
"Unit": "cpu_core"
},
{
"BriefDescription": "Code miss in all TLB levels causes a page walk that completes. (2M/4M)",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3",
"EventCode": "0x11",
"EventName": "ITLB_MISSES.WALK_COMPLETED_2M_4M",
- "PEBScounters": "0,1,2,3",
+ "PublicDescription": "Counts completed page walks (2M/4M page sizes) caused by a code fetch. This implies it missed in the ITLB (Instruction TLB) and further levels of TLB. The page walk can end with or without a fault.",
"SampleAfterValue": "100003",
- "Speculative": "1",
"UMask": "0x4",
"Unit": "cpu_core"
},
{
"BriefDescription": "Code miss in all TLB levels causes a page walk that completes. (4K)",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3",
"EventCode": "0x11",
"EventName": "ITLB_MISSES.WALK_COMPLETED_4K",
- "PEBScounters": "0,1,2,3",
+ "PublicDescription": "Counts completed page walks (4K page sizes) caused by a code fetch. This implies it missed in the ITLB (Instruction TLB) and further levels of TLB. The page walk can end with or without a fault.",
"SampleAfterValue": "100003",
- "Speculative": "1",
"UMask": "0x2",
"Unit": "cpu_core"
},
{
"BriefDescription": "Number of page walks outstanding for an outstanding code request in the PMH each cycle.",
- "CollectPEBSRecord": "2",
- "Counter": "0,1,2,3",
"EventCode": "0x11",
"EventName": "ITLB_MISSES.WALK_PENDING",
- "PEBScounters": "0,1,2,3",
+ "PublicDescription": "Counts the number of page walks outstanding for an outstanding code (instruction fetch) request in the PMH (Page Miss Handler) each cycle.",
"SampleAfterValue": "100003",
- "Speculative": "1",
"UMask": "0x10",
"Unit": "cpu_core"
+ },
+ {
+ "BriefDescription": "Counts the number of cycles that the head (oldest load) of the load buffer and retirement are both stalled due to a DTLB miss.",
+ "EventCode": "0x05",
+ "EventName": "LD_HEAD.DTLB_MISS_AT_RET",
+ "SampleAfterValue": "1000003",
+ "UMask": "0x90",
+ "Unit": "cpu_atom"
}
]
diff --git a/tools/perf/pmu-events/arch/x86/alderlaken/adln-metrics.json b/tools/perf/pmu-events/arch/x86/alderlaken/adln-metrics.json
new file mode 100644
index 0000000..c57e9f3
--- /dev/null
+++ b/tools/perf/pmu-events/arch/x86/alderlaken/adln-metrics.json
@@ -0,0 +1,583 @@
+[
+ {
+ "BriefDescription": "Counts the number of issue slots that were not consumed by the backend due to frontend stalls.",
+ "MetricExpr": "TOPDOWN_FE_BOUND.ALL / SLOTS",
+ "MetricGroup": "TopdownL1",
+ "MetricName": "tma_frontend_bound",
+ "ScaleUnit": "100%"
+ },
+ {
+ "BriefDescription": "Counts the number of issue slots that were not delivered by the frontend due to frontend bandwidth restrictions due to decode, predecode, cisc, and other limitations.",
+ "MetricExpr": "TOPDOWN_FE_BOUND.FRONTEND_LATENCY / SLOTS",
+ "MetricGroup": "TopdownL2;tma_frontend_bound_group",
+ "MetricName": "tma_frontend_latency",
+ "ScaleUnit": "100%"
+ },
+ {
+ "BriefDescription": "Counts the number of issue slots that were not delivered by the frontend due to instruction cache misses.",
+ "MetricExpr": "TOPDOWN_FE_BOUND.ICACHE / SLOTS",
+ "MetricGroup": "TopdownL3;tma_frontend_latency_group",
+ "MetricName": "tma_icache",
+ "ScaleUnit": "100%"
+ },
+ {
+ "BriefDescription": "Counts the number of issue slots that were not delivered by the frontend due to Instruction Table Lookaside Buffer (ITLB) misses.",
+ "MetricExpr": "TOPDOWN_FE_BOUND.ITLB / SLOTS",
+ "MetricGroup": "TopdownL3;tma_frontend_latency_group",
+ "MetricName": "tma_itlb",
+ "ScaleUnit": "100%"
+ },
+ {
+ "BriefDescription": "Counts the number of issue slots that were not delivered by the frontend due to BACLEARS, which occurs when the Branch Target Buffer (BTB) prediction or lack thereof, was corrected by a later branch predictor in the frontend",
+ "MetricExpr": "TOPDOWN_FE_BOUND.BRANCH_DETECT / SLOTS",
+ "MetricGroup": "TopdownL3;tma_frontend_latency_group",
+ "MetricName": "tma_branch_detect",
+ "PublicDescription": "Counts the number of issue slots that were not delivered by the frontend due to BACLEARS, which occurs when the Branch Target Buffer (BTB) prediction or lack thereof, was corrected by a later branch predictor in the frontend. Includes BACLEARS due to all branch types including conditional and unconditional jumps, returns, and indirect branches.",
+ "ScaleUnit": "100%"
+ },
+ {
+ "BriefDescription": "Counts the number of issue slots that were not delivered by the frontend due to BTCLEARS, which occurs when the Branch Target Buffer (BTB) predicts a taken branch.",
+ "MetricExpr": "TOPDOWN_FE_BOUND.BRANCH_RESTEER / SLOTS",
+ "MetricGroup": "TopdownL3;tma_frontend_latency_group",
+ "MetricName": "tma_branch_resteer",
+ "ScaleUnit": "100%"
+ },
+ {
+ "BriefDescription": "Counts the number of issue slots that were not delivered by the frontend due to frontend bandwidth restrictions due to decode, predecode, cisc, and other limitations.",
+ "MetricExpr": "TOPDOWN_FE_BOUND.FRONTEND_BANDWIDTH / SLOTS",
+ "MetricGroup": "TopdownL2;tma_frontend_bound_group",
+ "MetricName": "tma_frontend_bandwidth",
+ "ScaleUnit": "100%"
+ },
+ {
+ "BriefDescription": "Counts the number of issue slots that were not delivered by the frontend due to the microcode sequencer (MS).",
+ "MetricExpr": "TOPDOWN_FE_BOUND.CISC / SLOTS",
+ "MetricGroup": "TopdownL3;tma_frontend_bandwidth_group",
+ "MetricName": "tma_cisc",
+ "ScaleUnit": "100%"
+ },
+ {
+ "BriefDescription": "Counts the number of issue slots that were not delivered by the frontend due to decode stalls.",
+ "MetricExpr": "TOPDOWN_FE_BOUND.DECODE / SLOTS",
+ "MetricGroup": "TopdownL3;tma_frontend_bandwidth_group",
+ "MetricName": "tma_decode",
+ "ScaleUnit": "100%"
+ },
+ {
+ "BriefDescription": "Counts the number of issue slots that were not delivered by the frontend due to wrong predecodes.",
+ "MetricExpr": "TOPDOWN_FE_BOUND.PREDECODE / SLOTS",
+ "MetricGroup": "TopdownL3;tma_frontend_bandwidth_group",
+ "MetricName": "tma_predecode",
+ "ScaleUnit": "100%"
+ },
+ {
+ "BriefDescription": "Counts the number of issue slots that were not delivered by the frontend due to other common frontend stalls not categorized.",
+ "MetricExpr": "TOPDOWN_FE_BOUND.OTHER / SLOTS",
+ "MetricGroup": "TopdownL3;tma_frontend_bandwidth_group",
+ "MetricName": "tma_other_fb",
+ "ScaleUnit": "100%"
+ },
+ {
+ "BriefDescription": "Counts the total number of issue slots that were not consumed by the backend because allocation is stalled due to a mispredicted jump or a machine clear",
+ "MetricExpr": "(SLOTS - (TOPDOWN_FE_BOUND.ALL + TOPDOWN_BE_BOUND.ALL + TOPDOWN_RETIRING.ALL)) / SLOTS",
+ "MetricGroup": "TopdownL1",
+ "MetricName": "tma_bad_speculation",
+ "PublicDescription": "Counts the total number of issue slots that were not consumed by the backend because allocation is stalled due to a mispredicted jump or a machine clear. Only issue slots wasted due to fast nukes such as memory ordering nukes are counted. Other nukes are not accounted for. Counts all issue slots blocked during this recovery window including relevant microcode flows and while uops are not yet available in the instruction queue (IQ). Also includes the issue slots that were consumed by the backend but were thrown away because they were younger than the mispredict or machine clear.",
+ "ScaleUnit": "100%"
+ },
+ {
+ "BriefDescription": "Counts the number of issue slots that were not consumed by the backend due to branch mispredicts.",
+ "MetricExpr": "TOPDOWN_BAD_SPECULATION.MISPREDICT / SLOTS",
+ "MetricGroup": "TopdownL2;tma_bad_speculation_group",
+ "MetricName": "tma_branch_mispredicts",
+ "ScaleUnit": "100%"
+ },
+ {
+ "BriefDescription": "Counts the total number of issue slots that were not consumed by the backend because allocation is stalled due to a machine clear (nuke) of any kind including memory ordering and memory disambiguation.",
+ "MetricExpr": "TOPDOWN_BAD_SPECULATION.MACHINE_CLEARS / SLOTS",
+ "MetricGroup": "TopdownL2;tma_bad_speculation_group",
+ "MetricName": "tma_machine_clears",
+ "ScaleUnit": "100%"
+ },
+ {
+ "BriefDescription": "Counts the number of issue slots that were not consumed by the backend due to a machine clear (slow nuke).",
+ "MetricExpr": "TOPDOWN_BAD_SPECULATION.NUKE / SLOTS",
+ "MetricGroup": "TopdownL3;tma_machine_clears_group",
+ "MetricName": "tma_nuke",
+ "ScaleUnit": "100%"
+ },
+ {
+ "BriefDescription": "Counts the number of machine clears relative to the number of nuke slots due to SMC. ",
+ "MetricExpr": "tma_nuke * (MACHINE_CLEARS.SMC / MACHINE_CLEARS.SLOW)",
+ "MetricGroup": "TopdownL4;tma_nuke_group",
+ "MetricName": "tma_smc",
+ "ScaleUnit": "100%"
+ },
+ {
+ "BriefDescription": "Counts the number of machine clears relative to the number of nuke slots due to memory ordering. ",
+ "MetricExpr": "tma_nuke * (MACHINE_CLEARS.MEMORY_ORDERING / MACHINE_CLEARS.SLOW)",
+ "MetricGroup": "TopdownL4;tma_nuke_group",
+ "MetricName": "tma_memory_ordering",
+ "ScaleUnit": "100%"
+ },
+ {
+ "BriefDescription": "Counts the number of machine clears relative to the number of nuke slots due to FP assists. ",
+ "MetricExpr": "tma_nuke * (MACHINE_CLEARS.FP_ASSIST / MACHINE_CLEARS.SLOW)",
+ "MetricGroup": "TopdownL4;tma_nuke_group",
+ "MetricName": "tma_fp_assist",
+ "ScaleUnit": "100%"
+ },
+ {
+ "BriefDescription": "Counts the number of machine clears relative to the number of nuke slots due to memory disambiguation. ",
+ "MetricExpr": "tma_nuke * (MACHINE_CLEARS.DISAMBIGUATION / MACHINE_CLEARS.SLOW)",
+ "MetricGroup": "TopdownL4;tma_nuke_group",
+ "MetricName": "tma_disambiguation",
+ "ScaleUnit": "100%"
+ },
+ {
+ "BriefDescription": "Counts the number of machine clears relative to the number of nuke slots due to page faults. ",
+ "MetricExpr": "tma_nuke * (MACHINE_CLEARS.PAGE_FAULT / MACHINE_CLEARS.SLOW)",
+ "MetricGroup": "TopdownL4;tma_nuke_group",
+ "MetricName": "tma_page_fault",
+ "ScaleUnit": "100%"
+ },
+ {
+ "BriefDescription": "Counts the number of issue slots that were not consumed by the backend due to a machine clear classified as a fast nuke due to memory ordering, memory disambiguation and memory renaming.",
+ "MetricExpr": "TOPDOWN_BAD_SPECULATION.FASTNUKE / SLOTS",
+ "MetricGroup": "TopdownL3;tma_machine_clears_group",
+ "MetricName": "tma_fast_nuke",
+ "ScaleUnit": "100%"
+ },
+ {
+ "BriefDescription": "Counts the total number of issue slots that were not consumed by the backend due to backend stalls",
+ "MetricExpr": "TOPDOWN_BE_BOUND.ALL / SLOTS",
+ "MetricGroup": "TopdownL1",
+ "MetricName": "tma_backend_bound",
+ "PublicDescription": "Counts the total number of issue slots that were not consumed by the backend due to backend stalls. Note that uops must be available for consumption in order for this event to count. If a uop is not available (IQ is empty), this event will not count. The rest of these subevents count backend stalls, in cycles, due to an outstanding request which is memory bound vs core bound. The subevents are not slot based events and therefore can not be precisely added or subtracted from the Backend_Bound_Aux subevents which are slot based.",
+ "ScaleUnit": "100%"
+ },
+ {
+ "BriefDescription": "Counts the number of cycles due to backend bound stalls that are core execution bound and not attributed to outstanding demand load or store stalls. ",
+ "MetricExpr": "max(0, tma_backend_bound - tma_load_store_bound)",
+ "MetricGroup": "TopdownL2;tma_backend_bound_group",
+ "MetricName": "tma_core_bound",
+ "ScaleUnit": "100%"
+ },
+ {
+ "BriefDescription": "Counts the number of cycles the core is stalled due to stores or loads. ",
+ "MetricExpr": "min((TOPDOWN_BE_BOUND.ALL / SLOTS), (LD_HEAD.ANY_AT_RET / CLKS) + tma_store_bound)",
+ "MetricGroup": "TopdownL2;tma_backend_bound_group",
+ "MetricName": "tma_load_store_bound",
+ "ScaleUnit": "100%"
+ },
+ {
+ "BriefDescription": "Counts the number of cycles the core is stalled due to store buffer full.",
+ "MetricExpr": "tma_mem_scheduler * (MEM_SCHEDULER_BLOCK.ST_BUF / MEM_SCHEDULER_BLOCK.ALL)",
+ "MetricGroup": "TopdownL3;tma_load_store_bound_group",
+ "MetricName": "tma_store_bound",
+ "ScaleUnit": "100%"
+ },
+ {
+ "BriefDescription": "Counts the number of cycles that the oldest load of the load buffer is stalled at retirement due to a load block.",
+ "MetricExpr": "LD_HEAD.L1_BOUND_AT_RET / CLKS",
+ "MetricGroup": "TopdownL3;tma_load_store_bound_group",
+ "MetricName": "tma_l1_bound",
+ "ScaleUnit": "100%"
+ },
+ {
+ "BriefDescription": "Counts the number of cycles that the oldest load of the load buffer is stalled at retirement due to a store forward block.",
+ "MetricExpr": "LD_HEAD.ST_ADDR_AT_RET / CLKS",
+ "MetricGroup": "TopdownL4;tma_l1_bound_group",
+ "MetricName": "tma_store_fwd",
+ "ScaleUnit": "100%"
+ },
+ {
+ "BriefDescription": "Counts the number of cycles that the oldest load of the load buffer is stalled at retirement due to a first level TLB miss.",
+ "MetricExpr": "LD_HEAD.DTLB_MISS_AT_RET / CLKS",
+ "MetricGroup": "TopdownL4;tma_l1_bound_group",
+ "MetricName": "tma_stlb_hit",
+ "ScaleUnit": "100%"
+ },
+ {
+ "BriefDescription": "Counts the number of cycles that the oldest load of the load buffer is stalled at retirement due to a second level TLB miss requiring a page walk.",
+ "MetricExpr": "LD_HEAD.PGWALK_AT_RET / CLKS",
+ "MetricGroup": "TopdownL4;tma_l1_bound_group",
+ "MetricName": "tma_stlb_miss",
+ "ScaleUnit": "100%"
+ },
+ {
+ "BriefDescription": "Counts the number of cycles that the oldest load of the load buffer is stalled at retirement due to a number of other load blocks.",
+ "MetricExpr": "LD_HEAD.OTHER_AT_RET / CLKS",
+ "MetricGroup": "TopdownL4;tma_l1_bound_group",
+ "MetricName": "tma_other_l1",
+ "ScaleUnit": "100%"
+ },
+ {
+ "BriefDescription": "Counts the number of cycles a core is stalled due to a demand load which hit in the L2 Cache.",
+ "MetricExpr": "(MEM_BOUND_STALLS.LOAD_L2_HIT / CLKS) - (MEM_BOUND_STALLS_AT_RET_CORRECTION * MEM_BOUND_STALLS.LOAD_L2_HIT / MEM_BOUND_STALLS.LOAD)",
+ "MetricGroup": "TopdownL3;tma_load_store_bound_group",
+ "MetricName": "tma_l2_bound",
+ "ScaleUnit": "100%"
+ },
+ {
+ "BriefDescription": "Counts the number of cycles a core is stalled due to a demand load which hit in the Last Level Cache (LLC) or other core with HITE/F/M.",
+ "MetricExpr": "(MEM_BOUND_STALLS.LOAD_LLC_HIT / CLKS) - (MEM_BOUND_STALLS_AT_RET_CORRECTION * MEM_BOUND_STALLS.LOAD_LLC_HIT / MEM_BOUND_STALLS.LOAD)",
+ "MetricGroup": "TopdownL3;tma_load_store_bound_group",
+ "MetricName": "tma_l3_bound",
+ "ScaleUnit": "100%"
+ },
+ {
+ "BriefDescription": "Counts the number of cycles the core is stalled due to a demand load miss which hit in DRAM or MMIO (Non-DRAM).",
+ "MetricExpr": "(MEM_BOUND_STALLS.LOAD_DRAM_HIT / CLKS) - (MEM_BOUND_STALLS_AT_RET_CORRECTION * MEM_BOUND_STALLS.LOAD_DRAM_HIT / MEM_BOUND_STALLS.LOAD)",
+ "MetricGroup": "TopdownL3;tma_load_store_bound_group",
+ "MetricName": "tma_dram_bound",
+ "ScaleUnit": "100%"
+ },
+ {
+ "BriefDescription": "Counts the number of cycles the core is stalled due to a demand load miss which hits in the L2, LLC, DRAM or MMIO (Non-DRAM) but could not be correctly attributed or cycles in which the load miss is waiting on a request buffer.",
+ "MetricExpr": "max(0, tma_load_store_bound - (tma_store_bound + tma_l1_bound + tma_l2_bound + tma_l3_bound + tma_dram_bound))",
+ "MetricGroup": "TopdownL3;tma_load_store_bound_group",
+ "MetricName": "tma_other_load_store",
+ "ScaleUnit": "100%"
+ },
+ {
+ "BriefDescription": "Counts the total number of issue slots that were not consumed by the backend due to backend stalls",
+ "MetricExpr": "tma_backend_bound",
+ "MetricGroup": "TopdownL1",
+ "MetricName": "tma_backend_bound_aux",
+ "PublicDescription": "Counts the total number of issue slots that were not consumed by the backend due to backend stalls. Note that UOPS must be available for consumption in order for this event to count. If a uop is not available (IQ is empty), this event will not count. All of these subevents count backend stalls, in slots, due to a resource limitation. These are not cycle based events and therefore can not be precisely added or subtracted from the Backend_Bound subevents which are cycle based. These subevents are supplementary to Backend_Bound and can be used to analyze results from a resource perspective at allocation. ",
+ "ScaleUnit": "100%"
+ },
+ {
+ "BriefDescription": "Counts the total number of issue slots that were not consumed by the backend due to backend stalls",
+ "MetricExpr": "tma_backend_bound",
+ "MetricGroup": "TopdownL2;tma_backend_bound_aux_group",
+ "MetricName": "tma_resource_bound",
+ "PublicDescription": "Counts the total number of issue slots that were not consumed by the backend due to backend stalls. Note that uops must be available for consumption in order for this event to count. If a uop is not available (IQ is empty), this event will not count. ",
+ "ScaleUnit": "100%"
+ },
+ {
+ "BriefDescription": "Counts the number of issue slots that were not consumed by the backend due to memory reservation stalls in which a scheduler is not able to accept uops.",
+ "MetricExpr": "TOPDOWN_BE_BOUND.MEM_SCHEDULER / SLOTS",
+ "MetricGroup": "TopdownL3;tma_resource_bound_group",
+ "MetricName": "tma_mem_scheduler",
+ "ScaleUnit": "100%"
+ },
+ {
+ "BriefDescription": "Counts the number of cycles, relative to the number of mem_scheduler slots, in which uops are blocked due to store buffer full",
+ "MetricExpr": "tma_mem_scheduler * (MEM_SCHEDULER_BLOCK.ST_BUF / MEM_SCHEDULER_BLOCK.ALL)",
+ "MetricGroup": "TopdownL4;tma_mem_scheduler_group",
+ "MetricName": "tma_st_buffer",
+ "ScaleUnit": "100%"
+ },
+ {
+ "BriefDescription": "Counts the number of cycles, relative to the number of mem_scheduler slots, in which uops are blocked due to load buffer full",
+ "MetricExpr": "tma_mem_scheduler * MEM_SCHEDULER_BLOCK.LD_BUF / MEM_SCHEDULER_BLOCK.ALL",
+ "MetricGroup": "TopdownL4;tma_mem_scheduler_group",
+ "MetricName": "tma_ld_buffer",
+ "ScaleUnit": "100%"
+ },
+ {
+ "BriefDescription": "Counts the number of cycles, relative to the number of mem_scheduler slots, in which uops are blocked due to RSV full relative ",
+ "MetricExpr": "tma_mem_scheduler * MEM_SCHEDULER_BLOCK.RSV / MEM_SCHEDULER_BLOCK.ALL",
+ "MetricGroup": "TopdownL4;tma_mem_scheduler_group",
+ "MetricName": "tma_rsv",
+ "ScaleUnit": "100%"
+ },
+ {
+ "BriefDescription": "Counts the number of issue slots that were not consumed by the backend due to IEC or FPC RAT stalls, which can be due to FIQ or IEC reservation stalls in which the integer, floating point or SIMD scheduler is not able to accept uops.",
+ "MetricExpr": "TOPDOWN_BE_BOUND.NON_MEM_SCHEDULER / SLOTS",
+ "MetricGroup": "TopdownL3;tma_resource_bound_group",
+ "MetricName": "tma_non_mem_scheduler",
+ "ScaleUnit": "100%"
+ },
+ {
+ "BriefDescription": "Counts the number of issue slots that were not consumed by the backend due to the physical register file unable to accept an entry (marble stalls).",
+ "MetricExpr": "TOPDOWN_BE_BOUND.REGISTER / SLOTS",
+ "MetricGroup": "TopdownL3;tma_resource_bound_group",
+ "MetricName": "tma_register",
+ "ScaleUnit": "100%"
+ },
+ {
+ "BriefDescription": "Counts the number of issue slots that were not consumed by the backend due to the reorder buffer being full (ROB stalls).",
+ "MetricExpr": "TOPDOWN_BE_BOUND.REORDER_BUFFER / SLOTS",
+ "MetricGroup": "TopdownL3;tma_resource_bound_group",
+ "MetricName": "tma_reorder_buffer",
+ "ScaleUnit": "100%"
+ },
+ {
+ "BriefDescription": "Counts the number of issue slots that were not consumed by the backend due to certain allocation restrictions.",
+ "MetricExpr": "TOPDOWN_BE_BOUND.ALLOC_RESTRICTIONS / SLOTS",
+ "MetricGroup": "TopdownL3;tma_resource_bound_group",
+ "MetricName": "tma_alloc_restriction",
+ "ScaleUnit": "100%"
+ },
+ {
+ "BriefDescription": "Counts the number of issue slots that were not consumed by the backend due to scoreboards from the instruction queue (IQ), jump execution unit (JEU), or microcode sequencer (MS).",
+ "MetricExpr": "TOPDOWN_BE_BOUND.SERIALIZATION / SLOTS",
+ "MetricGroup": "TopdownL3;tma_resource_bound_group",
+ "MetricName": "tma_serialization",
+ "ScaleUnit": "100%"
+ },
+ {
+ "BriefDescription": "Counts the numer of issue slots that result in retirement slots. ",
+ "MetricExpr": "TOPDOWN_RETIRING.ALL / SLOTS",
+ "MetricGroup": "TopdownL1",
+ "MetricName": "tma_retiring",
+ "ScaleUnit": "100%"
+ },
+ {
+ "BriefDescription": "Counts the number of uops that are not from the microsequencer. ",
+ "MetricExpr": "(TOPDOWN_RETIRING.ALL - UOPS_RETIRED.MS) / SLOTS",
+ "MetricGroup": "TopdownL2;tma_retiring_group",
+ "MetricName": "tma_base",
+ "ScaleUnit": "100%"
+ },
+ {
+ "BriefDescription": "Counts the number of floating point operations per uop with all default weighting.",
+ "MetricExpr": "UOPS_RETIRED.FPDIV / SLOTS",
+ "MetricGroup": "TopdownL3;tma_base_group",
+ "MetricName": "tma_fp_uops",
+ "ScaleUnit": "100%"
+ },
+ {
+ "BriefDescription": "Counts the number of uops retired excluding ms and fp div uops.",
+ "MetricExpr": "(TOPDOWN_RETIRING.ALL - UOPS_RETIRED.MS - UOPS_RETIRED.FPDIV) / SLOTS",
+ "MetricGroup": "TopdownL3;tma_base_group",
+ "MetricName": "tma_other_ret",
+ "ScaleUnit": "100%"
+ },
+ {
+ "BriefDescription": "Counts the number of uops that are from the complex flows issued by the micro-sequencer (MS)",
+ "MetricExpr": "UOPS_RETIRED.MS / SLOTS",
+ "MetricGroup": "TopdownL2;tma_retiring_group",
+ "MetricName": "tma_ms_uops",
+ "PublicDescription": "Counts the number of uops that are from the complex flows issued by the micro-sequencer (MS). This includes uops from flows due to complex instructions, faults, assists, and inserted flows.",
+ "ScaleUnit": "100%"
+ },
+ {
+ "BriefDescription": "",
+ "MetricExpr": "CPU_CLK_UNHALTED.CORE",
+ "MetricName": "CLKS"
+ },
+ {
+ "BriefDescription": "",
+ "MetricExpr": "CPU_CLK_UNHALTED.CORE_P",
+ "MetricName": "CLKS_P"
+ },
+ {
+ "BriefDescription": "",
+ "MetricExpr": "5 * CLKS",
+ "MetricName": "SLOTS"
+ },
+ {
+ "BriefDescription": "Instructions Per Cycle",
+ "MetricExpr": "INST_RETIRED.ANY / CLKS",
+ "MetricName": "IPC"
+ },
+ {
+ "BriefDescription": "Cycles Per Instruction",
+ "MetricExpr": "CLKS / INST_RETIRED.ANY",
+ "MetricName": "CPI"
+ },
+ {
+ "BriefDescription": "Uops Per Instruction",
+ "MetricExpr": "UOPS_RETIRED.ALL / INST_RETIRED.ANY",
+ "MetricName": "UPI"
+ },
+ {
+ "BriefDescription": "Percentage of total non-speculative loads with a store forward or unknown store address block",
+ "MetricExpr": "100 * LD_BLOCKS.DATA_UNKNOWN / MEM_UOPS_RETIRED.ALL_LOADS",
+ "MetricName": "Store_Fwd_Blocks"
+ },
+ {
+ "BriefDescription": "Percentage of total non-speculative loads with a address aliasing block",
+ "MetricExpr": "100 * LD_BLOCKS.4K_ALIAS / MEM_UOPS_RETIRED.ALL_LOADS",
+ "MetricName": "Address_Alias_Blocks"
+ },
+ {
+ "BriefDescription": "Percentage of total non-speculative loads that are splits",
+ "MetricExpr": "100 * MEM_UOPS_RETIRED.SPLIT_LOADS / MEM_UOPS_RETIRED.ALL_LOADS",
+ "MetricName": "Load_Splits"
+ },
+ {
+ "BriefDescription": "Instructions per Branch (lower number means higher occurrence rate)",
+ "MetricExpr": "INST_RETIRED.ANY / BR_INST_RETIRED.ALL_BRANCHES",
+ "MetricName": "IpBranch"
+ },
+ {
+ "BriefDescription": "Instruction per (near) call (lower number means higher occurrence rate)",
+ "MetricExpr": "INST_RETIRED.ANY / BR_INST_RETIRED.CALL",
+ "MetricName": "IpCall"
+ },
+ {
+ "BriefDescription": "Instructions per Load",
+ "MetricExpr": "INST_RETIRED.ANY / MEM_UOPS_RETIRED.ALL_LOADS",
+ "MetricName": "IpLoad"
+ },
+ {
+ "BriefDescription": "Instructions per Store",
+ "MetricExpr": "INST_RETIRED.ANY / MEM_UOPS_RETIRED.ALL_STORES",
+ "MetricName": "IpStore"
+ },
+ {
+ "BriefDescription": "Number of Instructions per non-speculative Branch Misprediction",
+ "MetricExpr": "INST_RETIRED.ANY / BR_MISP_RETIRED.ALL_BRANCHES",
+ "MetricName": "IpMispredict"
+ },
+ {
+ "BriefDescription": "Instructions per Far Branch",
+ "MetricExpr": "INST_RETIRED.ANY / (BR_INST_RETIRED.FAR_BRANCH / 2)",
+ "MetricName": "IpFarBranch"
+ },
+ {
+ "BriefDescription": "Ratio of all branches which mispredict",
+ "MetricExpr": "BR_MISP_RETIRED.ALL_BRANCHES / BR_INST_RETIRED.ALL_BRANCHES",
+ "MetricName": "Branch_Mispredict_Ratio"
+ },
+ {
+ "BriefDescription": "Ratio between Mispredicted branches and unknown branches",
+ "MetricExpr": "BR_MISP_RETIRED.ALL_BRANCHES / BACLEARS.ANY",
+ "MetricName": "Branch_Mispredict_to_Unknown_Branch_Ratio"
+ },
+ {
+ "BriefDescription": "Percentage of all uops which are ucode ops",
+ "MetricExpr": "100 * UOPS_RETIRED.MS / UOPS_RETIRED.ALL",
+ "MetricName": "Microcode_Uop_Ratio"
+ },
+ {
+ "BriefDescription": "Percentage of all uops which are FPDiv uops",
+ "MetricExpr": "100 * UOPS_RETIRED.FPDIV / UOPS_RETIRED.ALL",
+ "MetricName": "FPDiv_Uop_Ratio"
+ },
+ {
+ "BriefDescription": "Percentage of all uops which are IDiv uops",
+ "MetricExpr": "100 * UOPS_RETIRED.IDIV / UOPS_RETIRED.ALL",
+ "MetricName": "IDiv_Uop_Ratio"
+ },
+ {
+ "BriefDescription": "Percentage of all uops which are x87 uops",
+ "MetricExpr": "100 * UOPS_RETIRED.X87 / UOPS_RETIRED.ALL",
+ "MetricName": "X87_Uop_Ratio"
+ },
+ {
+ "BriefDescription": "Average Frequency Utilization relative nominal frequency",
+ "MetricExpr": "CLKS / CPU_CLK_UNHALTED.REF_TSC",
+ "MetricName": "Turbo_Utilization"
+ },
+ {
+ "BriefDescription": "Fraction of cycles spent in Kernel mode",
+ "MetricExpr": "cpu@CPU_CLK_UNHALTED.CORE@k / CPU_CLK_UNHALTED.CORE",
+ "MetricName": "Kernel_Utilization"
+ },
+ {
+ "BriefDescription": "Average CPU Utilization",
+ "MetricExpr": "CPU_CLK_UNHALTED.REF_TSC / TSC",
+ "MetricName": "CPU_Utilization"
+ },
+ {
+ "BriefDescription": "Cycle cost per L2 hit",
+ "MetricExpr": "MEM_BOUND_STALLS.LOAD_L2_HIT / MEM_LOAD_UOPS_RETIRED.L2_HIT",
+ "MetricName": "Cycles_per_Demand_Load_L2_Hit"
+ },
+ {
+ "BriefDescription": "Cycle cost per LLC hit",
+ "MetricExpr": "MEM_BOUND_STALLS.LOAD_LLC_HIT / MEM_LOAD_UOPS_RETIRED.L3_HIT",
+ "MetricName": "Cycles_per_Demand_Load_L3_Hit"
+ },
+ {
+ "BriefDescription": "Cycle cost per DRAM hit",
+ "MetricExpr": "MEM_BOUND_STALLS.LOAD_DRAM_HIT / MEM_LOAD_UOPS_RETIRED.DRAM_HIT",
+ "MetricName": "Cycles_per_Demand_Load_DRAM_Hit"
+ },
+ {
+ "BriefDescription": "Percent of instruction miss cost that hit in the L2",
+ "MetricExpr": "100 * MEM_BOUND_STALLS.IFETCH_L2_HIT / (MEM_BOUND_STALLS.IFETCH)",
+ "MetricName": "Inst_Miss_Cost_L2Hit_Percent"
+ },
+ {
+ "BriefDescription": "Percent of instruction miss cost that hit in the L3",
+ "MetricExpr": "100 * MEM_BOUND_STALLS.IFETCH_LLC_HIT / (MEM_BOUND_STALLS.IFETCH)",
+ "MetricName": "Inst_Miss_Cost_L3Hit_Percent"
+ },
+ {
+ "BriefDescription": "Percent of instruction miss cost that hit in DRAM",
+ "MetricExpr": "100 * MEM_BOUND_STALLS.IFETCH_DRAM_HIT / (MEM_BOUND_STALLS.IFETCH)",
+ "MetricName": "Inst_Miss_Cost_DRAMHit_Percent"
+ },
+ {
+ "BriefDescription": "load ops retired per 1000 instruction",
+ "MetricExpr": "1000 * MEM_UOPS_RETIRED.ALL_LOADS / INST_RETIRED.ANY",
+ "MetricName": "MemLoadPKI"
+ },
+ {
+ "BriefDescription": "C1 residency percent per core",
+ "MetricExpr": "cstate_core@c1\\-residency@ / TSC",
+ "MetricGroup": "Power",
+ "MetricName": "C1_Core_Residency",
+ "ScaleUnit": "100%"
+ },
+ {
+ "BriefDescription": "C6 residency percent per core",
+ "MetricExpr": "cstate_core@c6\\-residency@ / TSC",
+ "MetricGroup": "Power",
+ "MetricName": "C6_Core_Residency",
+ "ScaleUnit": "100%"
+ },
+ {
+ "BriefDescription": "C7 residency percent per core",
+ "MetricExpr": "cstate_core@c7\\-residency@ / TSC",
+ "MetricGroup": "Power",
+ "MetricName": "C7_Core_Residency",
+ "ScaleUnit": "100%"
+ },
+ {
+ "BriefDescription": "C2 residency percent per package",
+ "MetricExpr": "cstate_pkg@c2\\-residency@ / TSC",
+ "MetricGroup": "Power",
+ "MetricName": "C2_Pkg_Residency",
+ "ScaleUnit": "100%"
+ },
+ {
+ "BriefDescription": "C3 residency percent per package",
+ "MetricExpr": "cstate_pkg@c3\\-residency@ / TSC",
+ "MetricGroup": "Power",
+ "MetricName": "C3_Pkg_Residency",
+ "ScaleUnit": "100%"
+ },
+ {
+ "BriefDescription": "C6 residency percent per package",
+ "MetricExpr": "cstate_pkg@c6\\-residency@ / TSC",
+ "MetricGroup": "Power",
+ "MetricName": "C6_Pkg_Residency",
+ "ScaleUnit": "100%"
+ },
+ {
+ "BriefDescription": "C7 residency percent per package",
+ "MetricExpr": "cstate_pkg@c7\\-residency@ / TSC",
+ "MetricGroup": "Power",
+ "MetricName": "C7_Pkg_Residency",
+ "ScaleUnit": "100%"
+ },
+ {
+ "BriefDescription": "C8 residency percent per package",
+ "MetricExpr": "cstate_pkg@c8\\-residency@ / TSC",
+ "MetricGroup": "Power",
+ "MetricName": "C8_Pkg_Residency",
+ "ScaleUnit": "100%"
+ },
+ {
+ "BriefDescription": "C9 residency percent per package",
+ "MetricExpr": "cstate_pkg@c9\\-residency@ / TSC",
+ "MetricGroup": "Power",
+ "MetricName": "C9_Pkg_Residency",
+ "ScaleUnit": "100%"
+ },
+ {
+ "BriefDescription": "C10 residency percent per package",
+ "MetricExpr": "cstate_pkg@c10\\-residency@ / TSC",
+ "MetricGroup": "Power",
+ "MetricName": "C10_Pkg_Residency",
+ "ScaleUnit": "100%"
+ }
+]
diff --git a/tools/perf/pmu-events/arch/x86/alderlaken/cache.json b/tools/perf/pmu-events/arch/x86/alderlaken/cache.json
new file mode 100644
index 0000000..043445a
--- /dev/null
+++ b/tools/perf/pmu-events/arch/x86/alderlaken/cache.json
@@ -0,0 +1,330 @@
+[
+ {
+ "BriefDescription": "Counts the number of cacheable memory requests that miss in the LLC. Counts on a per core basis.",
+ "EventCode": "0x2e",
+ "EventName": "LONGEST_LAT_CACHE.MISS",
+ "PublicDescription": "Counts the number of cacheable memory requests that miss in the Last Level Cache (LLC). Requests include demand loads, reads for ownership (RFO), instruction fetches and L1 HW prefetches. If the platform has an L3 cache, the LLC is the L3 cache, otherwise it is the L2 cache. Counts on a per core basis.",
+ "SampleAfterValue": "200003",
+ "UMask": "0x41"
+ },
+ {
+ "BriefDescription": "Counts the number of cacheable memory requests that access the LLC. Counts on a per core basis.",
+ "EventCode": "0x2e",
+ "EventName": "LONGEST_LAT_CACHE.REFERENCE",
+ "PublicDescription": "Counts the number of cacheable memory requests that access the Last Level Cache (LLC). Requests include demand loads, reads for ownership (RFO), instruction fetches and L1 HW prefetches. If the platform has an L3 cache, the LLC is the L3 cache, otherwise it is the L2 cache. Counts on a per core basis.",
+ "SampleAfterValue": "200003",
+ "UMask": "0x4f"
+ },
+ {
+ "BriefDescription": "Counts the number of cycles the core is stalled due to an instruction cache or TLB miss which hit in the L2, LLC, DRAM or MMIO (Non-DRAM).",
+ "EventCode": "0x34",
+ "EventName": "MEM_BOUND_STALLS.IFETCH",
+ "PublicDescription": "Counts the number of cycles the core is stalled due to an instruction cache or translation lookaside buffer (TLB) miss which hit in the L2, LLC, DRAM or MMIO (Non-DRAM).",
+ "SampleAfterValue": "200003",
+ "UMask": "0x38"
+ },
+ {
+ "BriefDescription": "Counts the number of cycles the core is stalled due to an instruction cache or TLB miss which hit in DRAM or MMIO (Non-DRAM).",
+ "EventCode": "0x34",
+ "EventName": "MEM_BOUND_STALLS.IFETCH_DRAM_HIT",
+ "PublicDescription": "Counts the number of cycles the core is stalled due to an instruction cache or translation lookaside buffer (TLB) miss which hit in DRAM or MMIO (non-DRAM).",
+ "SampleAfterValue": "200003",
+ "UMask": "0x20"
+ },
+ {
+ "BriefDescription": "Counts the number of cycles the core is stalled due to an instruction cache or TLB miss which hit in the L2 cache.",
+ "EventCode": "0x34",
+ "EventName": "MEM_BOUND_STALLS.IFETCH_L2_HIT",
+ "PublicDescription": "Counts the number of cycles the core is stalled due to an instruction cache or Translation Lookaside Buffer (TLB) miss which hit in the L2 cache.",
+ "SampleAfterValue": "200003",
+ "UMask": "0x8"
+ },
+ {
+ "BriefDescription": "Counts the number of cycles the core is stalled due to an instruction cache or TLB miss which hit in the LLC or other core with HITE/F/M.",
+ "EventCode": "0x34",
+ "EventName": "MEM_BOUND_STALLS.IFETCH_LLC_HIT",
+ "PublicDescription": "Counts the number of cycles the core is stalled due to an instruction cache or Translation Lookaside Buffer (TLB) miss which hit in the Last Level Cache (LLC) or other core with HITE/F/M.",
+ "SampleAfterValue": "200003",
+ "UMask": "0x10"
+ },
+ {
+ "BriefDescription": "Counts the number of cycles the core is stalled due to a demand load miss which hit in the L2, LLC, DRAM or MMIO (Non-DRAM).",
+ "EventCode": "0x34",
+ "EventName": "MEM_BOUND_STALLS.LOAD",
+ "SampleAfterValue": "200003",
+ "UMask": "0x7"
+ },
+ {
+ "BriefDescription": "Counts the number of cycles the core is stalled due to a demand load miss which hit in DRAM or MMIO (Non-DRAM).",
+ "EventCode": "0x34",
+ "EventName": "MEM_BOUND_STALLS.LOAD_DRAM_HIT",
+ "SampleAfterValue": "200003",
+ "UMask": "0x4"
+ },
+ {
+ "BriefDescription": "Counts the number of cycles the core is stalled due to a demand load which hit in the L2 cache.",
+ "EventCode": "0x34",
+ "EventName": "MEM_BOUND_STALLS.LOAD_L2_HIT",
+ "SampleAfterValue": "200003",
+ "UMask": "0x1"
+ },
+ {
+ "BriefDescription": "Counts the number of cycles the core is stalled due to a demand load which hit in the LLC or other core with HITE/F/M.",
+ "EventCode": "0x34",
+ "EventName": "MEM_BOUND_STALLS.LOAD_LLC_HIT",
+ "PublicDescription": "Counts the number of cycles the core is stalled due to a demand load which hit in the Last Level Cache (LLC) or other core with HITE/F/M.",
+ "SampleAfterValue": "200003",
+ "UMask": "0x2"
+ },
+ {
+ "BriefDescription": "Counts the number of load uops retired that hit in DRAM.",
+ "Data_LA": "1",
+ "EventCode": "0xd1",
+ "EventName": "MEM_LOAD_UOPS_RETIRED.DRAM_HIT",
+ "PEBS": "1",
+ "SampleAfterValue": "200003",
+ "UMask": "0x80"
+ },
+ {
+ "BriefDescription": "Counts the number of load uops retired that hit in the L2 cache.",
+ "Data_LA": "1",
+ "EventCode": "0xd1",
+ "EventName": "MEM_LOAD_UOPS_RETIRED.L2_HIT",
+ "PEBS": "1",
+ "SampleAfterValue": "200003",
+ "UMask": "0x2"
+ },
+ {
+ "BriefDescription": "Counts the number of load uops retired that hit in the L3 cache.",
+ "Data_LA": "1",
+ "EventCode": "0xd1",
+ "EventName": "MEM_LOAD_UOPS_RETIRED.L3_HIT",
+ "PEBS": "1",
+ "SampleAfterValue": "200003",
+ "UMask": "0x4"
+ },
+ {
+ "BriefDescription": "Counts the number of cycles that uops are blocked for any of the following reasons: load buffer, store buffer or RSV full.",
+ "EventCode": "0x04",
+ "EventName": "MEM_SCHEDULER_BLOCK.ALL",
+ "SampleAfterValue": "20003",
+ "UMask": "0x7"
+ },
+ {
+ "BriefDescription": "Counts the number of cycles that uops are blocked due to a load buffer full condition.",
+ "EventCode": "0x04",
+ "EventName": "MEM_SCHEDULER_BLOCK.LD_BUF",
+ "SampleAfterValue": "20003",
+ "UMask": "0x2"
+ },
+ {
+ "BriefDescription": "Counts the number of cycles that uops are blocked due to an RSV full condition.",
+ "EventCode": "0x04",
+ "EventName": "MEM_SCHEDULER_BLOCK.RSV",
+ "SampleAfterValue": "20003",
+ "UMask": "0x4"
+ },
+ {
+ "BriefDescription": "Counts the number of cycles that uops are blocked due to a store buffer full condition.",
+ "EventCode": "0x04",
+ "EventName": "MEM_SCHEDULER_BLOCK.ST_BUF",
+ "SampleAfterValue": "20003",
+ "UMask": "0x1"
+ },
+ {
+ "BriefDescription": "Counts the number of load uops retired.",
+ "Data_LA": "1",
+ "EventCode": "0xd0",
+ "EventName": "MEM_UOPS_RETIRED.ALL_LOADS",
+ "PEBS": "1",
+ "PublicDescription": "Counts the total number of load uops retired.",
+ "SampleAfterValue": "200003",
+ "UMask": "0x81"
+ },
+ {
+ "BriefDescription": "Counts the number of store uops retired.",
+ "Data_LA": "1",
+ "EventCode": "0xd0",
+ "EventName": "MEM_UOPS_RETIRED.ALL_STORES",
+ "PEBS": "1",
+ "PublicDescription": "Counts the total number of store uops retired.",
+ "SampleAfterValue": "200003",
+ "UMask": "0x82"
+ },
+ {
+ "BriefDescription": "Counts the number of tagged loads with an instruction latency that exceeds or equals the threshold of 128 cycles as defined in MEC_CR_PEBS_LD_LAT_THRESHOLD (3F6H). Only counts with PEBS enabled.",
+ "Data_LA": "1",
+ "EventCode": "0xd0",
+ "EventName": "MEM_UOPS_RETIRED.LOAD_LATENCY_GT_128",
+ "MSRIndex": "0x3F6",
+ "MSRValue": "0x80",
+ "PEBS": "2",
+ "PublicDescription": "Counts the number of tagged loads with an instruction latency that exceeds or equals the threshold of 128 cycles as defined in MEC_CR_PEBS_LD_LAT_THRESHOLD (3F6H). Only counts with PEBS enabled. If a PEBS record is generated, will populate the PEBS Latency and PEBS Data Source fields accordingly.",
+ "SampleAfterValue": "1000003",
+ "UMask": "0x5"
+ },
+ {
+ "BriefDescription": "Counts the number of tagged loads with an instruction latency that exceeds or equals the threshold of 16 cycles as defined in MEC_CR_PEBS_LD_LAT_THRESHOLD (3F6H). Only counts with PEBS enabled.",
+ "Data_LA": "1",
+ "EventCode": "0xd0",
+ "EventName": "MEM_UOPS_RETIRED.LOAD_LATENCY_GT_16",
+ "MSRIndex": "0x3F6",
+ "MSRValue": "0x10",
+ "PEBS": "2",
+ "PublicDescription": "Counts the number of tagged loads with an instruction latency that exceeds or equals the threshold of 16 cycles as defined in MEC_CR_PEBS_LD_LAT_THRESHOLD (3F6H). Only counts with PEBS enabled. If a PEBS record is generated, will populate the PEBS Latency and PEBS Data Source fields accordingly.",
+ "SampleAfterValue": "1000003",
+ "UMask": "0x5"
+ },
+ {
+ "BriefDescription": "Counts the number of tagged loads with an instruction latency that exceeds or equals the threshold of 256 cycles as defined in MEC_CR_PEBS_LD_LAT_THRESHOLD (3F6H). Only counts with PEBS enabled.",
+ "Data_LA": "1",
+ "EventCode": "0xd0",
+ "EventName": "MEM_UOPS_RETIRED.LOAD_LATENCY_GT_256",
+ "MSRIndex": "0x3F6",
+ "MSRValue": "0x100",
+ "PEBS": "2",
+ "PublicDescription": "Counts the number of tagged loads with an instruction latency that exceeds or equals the threshold of 256 cycles as defined in MEC_CR_PEBS_LD_LAT_THRESHOLD (3F6H). Only counts with PEBS enabled. If a PEBS record is generated, will populate the PEBS Latency and PEBS Data Source fields accordingly.",
+ "SampleAfterValue": "1000003",
+ "UMask": "0x5"
+ },
+ {
+ "BriefDescription": "Counts the number of tagged loads with an instruction latency that exceeds or equals the threshold of 32 cycles as defined in MEC_CR_PEBS_LD_LAT_THRESHOLD (3F6H). Only counts with PEBS enabled.",
+ "Data_LA": "1",
+ "EventCode": "0xd0",
+ "EventName": "MEM_UOPS_RETIRED.LOAD_LATENCY_GT_32",
+ "MSRIndex": "0x3F6",
+ "MSRValue": "0x20",
+ "PEBS": "2",
+ "PublicDescription": "Counts the number of tagged loads with an instruction latency that exceeds or equals the threshold of 32 cycles as defined in MEC_CR_PEBS_LD_LAT_THRESHOLD (3F6H). Only counts with PEBS enabled. If a PEBS record is generated, will populate the PEBS Latency and PEBS Data Source fields accordingly.",
+ "SampleAfterValue": "1000003",
+ "UMask": "0x5"
+ },
+ {
+ "BriefDescription": "Counts the number of tagged loads with an instruction latency that exceeds or equals the threshold of 4 cycles as defined in MEC_CR_PEBS_LD_LAT_THRESHOLD (3F6H). Only counts with PEBS enabled.",
+ "Data_LA": "1",
+ "EventCode": "0xd0",
+ "EventName": "MEM_UOPS_RETIRED.LOAD_LATENCY_GT_4",
+ "MSRIndex": "0x3F6",
+ "MSRValue": "0x4",
+ "PEBS": "2",
+ "PublicDescription": "Counts the number of tagged loads with an instruction latency that exceeds or equals the threshold of 4 cycles as defined in MEC_CR_PEBS_LD_LAT_THRESHOLD (3F6H). Only counts with PEBS enabled. If a PEBS record is generated, will populate the PEBS Latency and PEBS Data Source fields accordingly.",
+ "SampleAfterValue": "1000003",
+ "UMask": "0x5"
+ },
+ {
+ "BriefDescription": "Counts the number of tagged loads with an instruction latency that exceeds or equals the threshold of 512 cycles as defined in MEC_CR_PEBS_LD_LAT_THRESHOLD (3F6H). Only counts with PEBS enabled.",
+ "Data_LA": "1",
+ "EventCode": "0xd0",
+ "EventName": "MEM_UOPS_RETIRED.LOAD_LATENCY_GT_512",
+ "MSRIndex": "0x3F6",
+ "MSRValue": "0x200",
+ "PEBS": "2",
+ "PublicDescription": "Counts the number of tagged loads with an instruction latency that exceeds or equals the threshold of 512 cycles as defined in MEC_CR_PEBS_LD_LAT_THRESHOLD (3F6H). Only counts with PEBS enabled. If a PEBS record is generated, will populate the PEBS Latency and PEBS Data Source fields accordingly.",
+ "SampleAfterValue": "1000003",
+ "UMask": "0x5"
+ },
+ {
+ "BriefDescription": "Counts the number of tagged loads with an instruction latency that exceeds or equals the threshold of 64 cycles as defined in MEC_CR_PEBS_LD_LAT_THRESHOLD (3F6H). Only counts with PEBS enabled.",
+ "Data_LA": "1",
+ "EventCode": "0xd0",
+ "EventName": "MEM_UOPS_RETIRED.LOAD_LATENCY_GT_64",
+ "MSRIndex": "0x3F6",
+ "MSRValue": "0x40",
+ "PEBS": "2",
+ "PublicDescription": "Counts the number of tagged loads with an instruction latency that exceeds or equals the threshold of 64 cycles as defined in MEC_CR_PEBS_LD_LAT_THRESHOLD (3F6H). Only counts with PEBS enabled. If a PEBS record is generated, will populate the PEBS Latency and PEBS Data Source fields accordingly.",
+ "SampleAfterValue": "1000003",
+ "UMask": "0x5"
+ },
+ {
+ "BriefDescription": "Counts the number of tagged loads with an instruction latency that exceeds or equals the threshold of 8 cycles as defined in MEC_CR_PEBS_LD_LAT_THRESHOLD (3F6H). Only counts with PEBS enabled.",
+ "Data_LA": "1",
+ "EventCode": "0xd0",
+ "EventName": "MEM_UOPS_RETIRED.LOAD_LATENCY_GT_8",
+ "MSRIndex": "0x3F6",
+ "MSRValue": "0x8",
+ "PEBS": "2",
+ "PublicDescription": "Counts the number of tagged loads with an instruction latency that exceeds or equals the threshold of 8 cycles as defined in MEC_CR_PEBS_LD_LAT_THRESHOLD (3F6H). Only counts with PEBS enabled. If a PEBS record is generated, will populate the PEBS Latency and PEBS Data Source fields accordingly.",
+ "SampleAfterValue": "1000003",
+ "UMask": "0x5"
+ },
+ {
+ "BriefDescription": "Counts the number of retired split load uops.",
+ "Data_LA": "1",
+ "EventCode": "0xd0",
+ "EventName": "MEM_UOPS_RETIRED.SPLIT_LOADS",
+ "PEBS": "1",
+ "SampleAfterValue": "200003",
+ "UMask": "0x41"
+ },
+ {
+ "BriefDescription": "Counts the number of stores uops retired. Counts with or without PEBS enabled.",
+ "Data_LA": "1",
+ "EventCode": "0xd0",
+ "EventName": "MEM_UOPS_RETIRED.STORE_LATENCY",
+ "PEBS": "2",
+ "PublicDescription": "Counts the number of stores uops retired. Counts with or without PEBS enabled. If PEBS is enabled and a PEBS record is generated, will populate PEBS Latency and PEBS Data Source fields accordingly.",
+ "SampleAfterValue": "1000003",
+ "UMask": "0x6"
+ },
+ {
+ "BriefDescription": "Counts demand data reads that were supplied by the L3 cache.",
+ "EventCode": "0xB7",
+ "EventName": "OCR.DEMAND_DATA_RD.L3_HIT",
+ "MSRIndex": "0x1a6,0x1a7",
+ "MSRValue": "0x3F803C0001",
+ "SampleAfterValue": "100003",
+ "UMask": "0x1"
+ },
+ {
+ "BriefDescription": "Counts demand data reads that were supplied by the L3 cache where a snoop was sent, the snoop hit, and modified data was forwarded.",
+ "EventCode": "0xB7",
+ "EventName": "OCR.DEMAND_DATA_RD.L3_HIT.SNOOP_HITM",
+ "MSRIndex": "0x1a6,0x1a7",
+ "MSRValue": "0x10003C0001",
+ "SampleAfterValue": "100003",
+ "UMask": "0x1"
+ },
+ {
+ "BriefDescription": "Counts demand data reads that were supplied by the L3 cache where a snoop was sent, the snoop hit, but no data was forwarded.",
+ "EventCode": "0xB7",
+ "EventName": "OCR.DEMAND_DATA_RD.L3_HIT.SNOOP_HIT_NO_FWD",
+ "MSRIndex": "0x1a6,0x1a7",
+ "MSRValue": "0x4003C0001",
+ "SampleAfterValue": "100003",
+ "UMask": "0x1"
+ },
+ {
+ "BriefDescription": "Counts demand data reads that were supplied by the L3 cache where a snoop was sent, the snoop hit, and non-modified data was forwarded.",
+ "EventCode": "0xB7",
+ "EventName": "OCR.DEMAND_DATA_RD.L3_HIT.SNOOP_HIT_WITH_FWD",
+ "MSRIndex": "0x1a6,0x1a7",
+ "MSRValue": "0x8003C0001",
+ "SampleAfterValue": "100003",
+ "UMask": "0x1"
+ },
+ {
+ "BriefDescription": "Counts demand reads for ownership (RFO) and software prefetches for exclusive ownership (PREFETCHW) that were supplied by the L3 cache.",
+ "EventCode": "0xB7",
+ "EventName": "OCR.DEMAND_RFO.L3_HIT",
+ "MSRIndex": "0x1a6,0x1a7",
+ "MSRValue": "0x3F803C0002",
+ "SampleAfterValue": "100003",
+ "UMask": "0x1"
+ },
+ {
+ "BriefDescription": "Counts demand reads for ownership (RFO) and software prefetches for exclusive ownership (PREFETCHW) that were supplied by the L3 cache where a snoop was sent, the snoop hit, and modified data was forwarded.",
+ "EventCode": "0xB7",
+ "EventName": "OCR.DEMAND_RFO.L3_HIT.SNOOP_HITM",
+ "MSRIndex": "0x1a6,0x1a7",
+ "MSRValue": "0x10003C0002",
+ "SampleAfterValue": "100003",
+ "UMask": "0x1"
+ },
+ {
+ "BriefDescription": "Counts the number of issue slots every cycle that were not delivered by the frontend due to instruction cache misses.",
+ "EventCode": "0x71",
+ "EventName": "TOPDOWN_FE_BOUND.ICACHE",
+ "SampleAfterValue": "1000003",
+ "UMask": "0x20"
+ }
+]
diff --git a/tools/perf/pmu-events/arch/x86/alderlaken/floating-point.json b/tools/perf/pmu-events/arch/x86/alderlaken/floating-point.json
new file mode 100644
index 0000000..30e8ca3
--- /dev/null
+++ b/tools/perf/pmu-events/arch/x86/alderlaken/floating-point.json
@@ -0,0 +1,18 @@
+[
+ {
+ "BriefDescription": "Counts the number of floating point operations retired that required microcode assist.",
+ "EventCode": "0xc3",
+ "EventName": "MACHINE_CLEARS.FP_ASSIST",
+ "PublicDescription": "Counts the number of floating point operations retired that required microcode assist, which is not a reflection of the number of FP operations, instructions or uops.",
+ "SampleAfterValue": "20003",
+ "UMask": "0x4"
+ },
+ {
+ "BriefDescription": "Counts the number of floating point divide uops retired (x87 and SSE, including x87 sqrt).",
+ "EventCode": "0xc2",
+ "EventName": "UOPS_RETIRED.FPDIV",
+ "PEBS": "1",
+ "SampleAfterValue": "2000003",
+ "UMask": "0x8"
+ }
+]
diff --git a/tools/perf/pmu-events/arch/x86/alderlaken/frontend.json b/tools/perf/pmu-events/arch/x86/alderlaken/frontend.json
new file mode 100644
index 0000000..36898ba
--- /dev/null
+++ b/tools/perf/pmu-events/arch/x86/alderlaken/frontend.json
@@ -0,0 +1,26 @@
+[
+ {
+ "BriefDescription": "Counts the total number of BACLEARS due to all branch types including conditional and unconditional jumps, returns, and indirect branches.",
+ "EventCode": "0xe6",
+ "EventName": "BACLEARS.ANY",
+ "PublicDescription": "Counts the total number of BACLEARS, which occur when the Branch Target Buffer (BTB) prediction or lack thereof, was corrected by a later branch predictor in the frontend. Includes BACLEARS due to all branch types including conditional and unconditional jumps, returns, and indirect branches.",
+ "SampleAfterValue": "100003",
+ "UMask": "0x1"
+ },
+ {
+ "BriefDescription": "Counts the number of requests to the instruction cache for one or more bytes of a cache line.",
+ "EventCode": "0x80",
+ "EventName": "ICACHE.ACCESSES",
+ "PublicDescription": "Counts the total number of requests to the instruction cache. The event only counts new cache line accesses, so that multiple back to back fetches to the exact same cache line or byte chunk count as one. Specifically, the event counts when accesses from sequential code crosses the cache line boundary, or when a branch target is moved to a new line or to a non-sequential byte chunk of the same line.",
+ "SampleAfterValue": "200003",
+ "UMask": "0x3"
+ },
+ {
+ "BriefDescription": "Counts the number of instruction cache misses.",
+ "EventCode": "0x80",
+ "EventName": "ICACHE.MISSES",
+ "PublicDescription": "Counts the number of missed requests to the instruction cache. The event only counts new cache line accesses, so that multiple back to back fetches to the exact same cache line and byte chunk count as one. Specifically, the event counts when accesses from sequential code crosses the cache line boundary, or when a branch target is moved to a new line or to a non-sequential byte chunk of the same line.",
+ "SampleAfterValue": "200003",
+ "UMask": "0x2"
+ }
+]
diff --git a/tools/perf/pmu-events/arch/x86/alderlaken/memory.json b/tools/perf/pmu-events/arch/x86/alderlaken/memory.json
new file mode 100644
index 0000000..f84bf8c
--- /dev/null
+++ b/tools/perf/pmu-events/arch/x86/alderlaken/memory.json
@@ -0,0 +1,81 @@
+[
+ {
+ "BriefDescription": "Counts the number of cycles that the head (oldest load) of the load buffer is stalled due to any number of reasons, including an L1 miss, WCB full, pagewalk, store address block or store data block, on a load that retires.",
+ "EventCode": "0x05",
+ "EventName": "LD_HEAD.ANY_AT_RET",
+ "SampleAfterValue": "1000003",
+ "UMask": "0xff"
+ },
+ {
+ "BriefDescription": "Counts the number of cycles that the head (oldest load) of the load buffer is stalled due to a core bound stall including a store address match, a DTLB miss or a page walk that detains the load from retiring.",
+ "EventCode": "0x05",
+ "EventName": "LD_HEAD.L1_BOUND_AT_RET",
+ "SampleAfterValue": "1000003",
+ "UMask": "0xf4"
+ },
+ {
+ "BriefDescription": "Counts the number of cycles that the head (oldest load) of the load buffer and retirement are both stalled due to other block cases.",
+ "EventCode": "0x05",
+ "EventName": "LD_HEAD.OTHER_AT_RET",
+ "PublicDescription": "Counts the number of cycles that the head (oldest load) of the load buffer and retirement are both stalled due to other block cases such as pipeline conflicts, fences, etc.",
+ "SampleAfterValue": "1000003",
+ "UMask": "0xc0"
+ },
+ {
+ "BriefDescription": "Counts the number of cycles that the head (oldest load) of the load buffer and retirement are both stalled due to a pagewalk.",
+ "EventCode": "0x05",
+ "EventName": "LD_HEAD.PGWALK_AT_RET",
+ "SampleAfterValue": "1000003",
+ "UMask": "0xa0"
+ },
+ {
+ "BriefDescription": "Counts the number of cycles that the head (oldest load) of the load buffer and retirement are both stalled due to a store address match.",
+ "EventCode": "0x05",
+ "EventName": "LD_HEAD.ST_ADDR_AT_RET",
+ "SampleAfterValue": "1000003",
+ "UMask": "0x84"
+ },
+ {
+ "BriefDescription": "Counts the number of machine clears due to memory ordering caused by a snoop from an external agent. Does not count internally generated machine clears such as those due to memory disambiguation.",
+ "EventCode": "0xc3",
+ "EventName": "MACHINE_CLEARS.MEMORY_ORDERING",
+ "SampleAfterValue": "20003",
+ "UMask": "0x2"
+ },
+ {
+ "BriefDescription": "Counts demand data reads that were not supplied by the L3 cache.",
+ "EventCode": "0xB7",
+ "EventName": "OCR.DEMAND_DATA_RD.L3_MISS",
+ "MSRIndex": "0x1a6,0x1a7",
+ "MSRValue": "0x3F84400001",
+ "SampleAfterValue": "100003",
+ "UMask": "0x1"
+ },
+ {
+ "BriefDescription": "Counts demand data reads that were not supplied by the L3 cache.",
+ "EventCode": "0xB7",
+ "EventName": "OCR.DEMAND_DATA_RD.L3_MISS_LOCAL",
+ "MSRIndex": "0x1a6,0x1a7",
+ "MSRValue": "0x3F84400001",
+ "SampleAfterValue": "100003",
+ "UMask": "0x1"
+ },
+ {
+ "BriefDescription": "Counts demand reads for ownership (RFO) and software prefetches for exclusive ownership (PREFETCHW) that were not supplied by the L3 cache.",
+ "EventCode": "0xB7",
+ "EventName": "OCR.DEMAND_RFO.L3_MISS",
+ "MSRIndex": "0x1a6,0x1a7",
+ "MSRValue": "0x3F84400002",
+ "SampleAfterValue": "100003",
+ "UMask": "0x1"
+ },
+ {
+ "BriefDescription": "Counts demand reads for ownership (RFO) and software prefetches for exclusive ownership (PREFETCHW) that were not supplied by the L3 cache.",
+ "EventCode": "0xB7",
+ "EventName": "OCR.DEMAND_RFO.L3_MISS_LOCAL",
+ "MSRIndex": "0x1a6,0x1a7",
+ "MSRValue": "0x3F84400002",
+ "SampleAfterValue": "100003",
+ "UMask": "0x1"
+ }
+]
diff --git a/tools/perf/pmu-events/arch/x86/alderlaken/other.json b/tools/perf/pmu-events/arch/x86/alderlaken/other.json
new file mode 100644
index 0000000..6336de6
--- /dev/null
+++ b/tools/perf/pmu-events/arch/x86/alderlaken/other.json
@@ -0,0 +1,38 @@
+[
+ {
+ "BriefDescription": "Counts modified writebacks from L1 cache and L2 cache that have any type of response.",
+ "EventCode": "0xB7",
+ "EventName": "OCR.COREWB_M.ANY_RESPONSE",
+ "MSRIndex": "0x1a6,0x1a7",
+ "MSRValue": "0x10008",
+ "SampleAfterValue": "100003",
+ "UMask": "0x1"
+ },
+ {
+ "BriefDescription": "Counts demand data reads that have any type of response.",
+ "EventCode": "0xB7",
+ "EventName": "OCR.DEMAND_DATA_RD.ANY_RESPONSE",
+ "MSRIndex": "0x1a6,0x1a7",
+ "MSRValue": "0x10001",
+ "SampleAfterValue": "100003",
+ "UMask": "0x1"
+ },
+ {
+ "BriefDescription": "Counts demand reads for ownership (RFO) and software prefetches for exclusive ownership (PREFETCHW) that have any type of response.",
+ "EventCode": "0xB7",
+ "EventName": "OCR.DEMAND_RFO.ANY_RESPONSE",
+ "MSRIndex": "0x1a6,0x1a7",
+ "MSRValue": "0x10002",
+ "SampleAfterValue": "100003",
+ "UMask": "0x1"
+ },
+ {
+ "BriefDescription": "Counts streaming stores that have any type of response.",
+ "EventCode": "0xB7",
+ "EventName": "OCR.STREAMING_WR.ANY_RESPONSE",
+ "MSRIndex": "0x1a6,0x1a7",
+ "MSRValue": "0x10800",
+ "SampleAfterValue": "100003",
+ "UMask": "0x1"
+ }
+]
diff --git a/tools/perf/pmu-events/arch/x86/alderlaken/pipeline.json b/tools/perf/pmu-events/arch/x86/alderlaken/pipeline.json
new file mode 100644
index 0000000..fa53ff1
--- /dev/null
+++ b/tools/perf/pmu-events/arch/x86/alderlaken/pipeline.json
@@ -0,0 +1,533 @@
+[
+ {
+ "BriefDescription": "Counts the total number of branch instructions retired for all branch types.",
+ "EventCode": "0xc4",
+ "EventName": "BR_INST_RETIRED.ALL_BRANCHES",
+ "PEBS": "1",
+ "PublicDescription": "Counts the total number of instructions in which the instruction pointer (IP) of the processor is resteered due to a branch instruction and the branch instruction successfully retires. All branch type instructions are accounted for.",
+ "SampleAfterValue": "200003"
+ },
+ {
+ "BriefDescription": "This event is deprecated. Refer to new event BR_INST_RETIRED.NEAR_CALL",
+ "Deprecated": "1",
+ "EventCode": "0xc4",
+ "EventName": "BR_INST_RETIRED.CALL",
+ "PEBS": "1",
+ "SampleAfterValue": "200003",
+ "UMask": "0xf9"
+ },
+ {
+ "BriefDescription": "Counts the number of retired JCC (Jump on Conditional Code) branch instructions retired, includes both taken and not taken branches.",
+ "EventCode": "0xc4",
+ "EventName": "BR_INST_RETIRED.COND",
+ "PEBS": "1",
+ "SampleAfterValue": "200003",
+ "UMask": "0x7e"
+ },
+ {
+ "BriefDescription": "Counts the number of taken JCC (Jump on Conditional Code) branch instructions retired.",
+ "EventCode": "0xc4",
+ "EventName": "BR_INST_RETIRED.COND_TAKEN",
+ "PEBS": "1",
+ "SampleAfterValue": "200003",
+ "UMask": "0xfe"
+ },
+ {
+ "BriefDescription": "Counts the number of far branch instructions retired, includes far jump, far call and return, and interrupt call and return.",
+ "EventCode": "0xc4",
+ "EventName": "BR_INST_RETIRED.FAR_BRANCH",
+ "PEBS": "1",
+ "SampleAfterValue": "200003",
+ "UMask": "0xbf"
+ },
+ {
+ "BriefDescription": "Counts the number of near indirect JMP and near indirect CALL branch instructions retired.",
+ "EventCode": "0xc4",
+ "EventName": "BR_INST_RETIRED.INDIRECT",
+ "PEBS": "1",
+ "SampleAfterValue": "200003",
+ "UMask": "0xeb"
+ },
+ {
+ "BriefDescription": "Counts the number of near indirect CALL branch instructions retired.",
+ "EventCode": "0xc4",
+ "EventName": "BR_INST_RETIRED.INDIRECT_CALL",
+ "PEBS": "1",
+ "SampleAfterValue": "200003",
+ "UMask": "0xfb"
+ },
+ {
+ "BriefDescription": "This event is deprecated. Refer to new event BR_INST_RETIRED.INDIRECT_CALL",
+ "Deprecated": "1",
+ "EventCode": "0xc4",
+ "EventName": "BR_INST_RETIRED.IND_CALL",
+ "PEBS": "1",
+ "SampleAfterValue": "200003",
+ "UMask": "0xfb"
+ },
+ {
+ "BriefDescription": "This event is deprecated. Refer to new event BR_INST_RETIRED.COND",
+ "Deprecated": "1",
+ "EventCode": "0xc4",
+ "EventName": "BR_INST_RETIRED.JCC",
+ "PEBS": "1",
+ "SampleAfterValue": "200003",
+ "UMask": "0x7e"
+ },
+ {
+ "BriefDescription": "Counts the number of near CALL branch instructions retired.",
+ "EventCode": "0xc4",
+ "EventName": "BR_INST_RETIRED.NEAR_CALL",
+ "PEBS": "1",
+ "SampleAfterValue": "200003",
+ "UMask": "0xf9"
+ },
+ {
+ "BriefDescription": "Counts the number of near RET branch instructions retired.",
+ "EventCode": "0xc4",
+ "EventName": "BR_INST_RETIRED.NEAR_RETURN",
+ "PEBS": "1",
+ "SampleAfterValue": "200003",
+ "UMask": "0xf7"
+ },
+ {
+ "BriefDescription": "This event is deprecated. Refer to new event BR_INST_RETIRED.INDIRECT",
+ "Deprecated": "1",
+ "EventCode": "0xc4",
+ "EventName": "BR_INST_RETIRED.NON_RETURN_IND",
+ "PEBS": "1",
+ "SampleAfterValue": "200003",
+ "UMask": "0xeb"
+ },
+ {
+ "BriefDescription": "Counts the number of near relative CALL branch instructions retired.",
+ "EventCode": "0xc4",
+ "EventName": "BR_INST_RETIRED.REL_CALL",
+ "PEBS": "1",
+ "SampleAfterValue": "200003",
+ "UMask": "0xfd"
+ },
+ {
+ "BriefDescription": "This event is deprecated. Refer to new event BR_INST_RETIRED.NEAR_RETURN",
+ "Deprecated": "1",
+ "EventCode": "0xc4",
+ "EventName": "BR_INST_RETIRED.RETURN",
+ "PEBS": "1",
+ "SampleAfterValue": "200003",
+ "UMask": "0xf7"
+ },
+ {
+ "BriefDescription": "This event is deprecated. Refer to new event BR_INST_RETIRED.COND_TAKEN",
+ "Deprecated": "1",
+ "EventCode": "0xc4",
+ "EventName": "BR_INST_RETIRED.TAKEN_JCC",
+ "PEBS": "1",
+ "SampleAfterValue": "200003",
+ "UMask": "0xfe"
+ },
+ {
+ "BriefDescription": "Counts the total number of mispredicted branch instructions retired for all branch types.",
+ "EventCode": "0xc5",
+ "EventName": "BR_MISP_RETIRED.ALL_BRANCHES",
+ "PEBS": "1",
+ "PublicDescription": "Counts the total number of mispredicted branch instructions retired. All branch type instructions are accounted for. Prediction of the branch target address enables the processor to begin executing instructions before the non-speculative execution path is known. The branch prediction unit (BPU) predicts the target address based on the instruction pointer (IP) of the branch and on the execution path through which execution reached this IP. A branch misprediction occurs when the prediction is wrong, and results in discarding all instructions executed in the speculative path and re-fetching from the correct path.",
+ "SampleAfterValue": "200003"
+ },
+ {
+ "BriefDescription": "Counts the number of mispredicted JCC (Jump on Conditional Code) branch instructions retired.",
+ "EventCode": "0xc5",
+ "EventName": "BR_MISP_RETIRED.COND",
+ "PEBS": "1",
+ "SampleAfterValue": "200003",
+ "UMask": "0x7e"
+ },
+ {
+ "BriefDescription": "Counts the number of mispredicted taken JCC (Jump on Conditional Code) branch instructions retired.",
+ "EventCode": "0xc5",
+ "EventName": "BR_MISP_RETIRED.COND_TAKEN",
+ "PEBS": "1",
+ "SampleAfterValue": "200003",
+ "UMask": "0xfe"
+ },
+ {
+ "BriefDescription": "Counts the number of mispredicted near indirect JMP and near indirect CALL branch instructions retired.",
+ "EventCode": "0xc5",
+ "EventName": "BR_MISP_RETIRED.INDIRECT",
+ "PEBS": "1",
+ "SampleAfterValue": "200003",
+ "UMask": "0xeb"
+ },
+ {
+ "BriefDescription": "Counts the number of mispredicted near indirect CALL branch instructions retired.",
+ "EventCode": "0xc5",
+ "EventName": "BR_MISP_RETIRED.INDIRECT_CALL",
+ "PEBS": "1",
+ "SampleAfterValue": "200003",
+ "UMask": "0xfb"
+ },
+ {
+ "BriefDescription": "This event is deprecated. Refer to new event BR_MISP_RETIRED.INDIRECT_CALL",
+ "Deprecated": "1",
+ "EventCode": "0xc5",
+ "EventName": "BR_MISP_RETIRED.IND_CALL",
+ "PEBS": "1",
+ "SampleAfterValue": "200003",
+ "UMask": "0xfb"
+ },
+ {
+ "BriefDescription": "This event is deprecated. Refer to new event BR_MISP_RETIRED.COND",
+ "Deprecated": "1",
+ "EventCode": "0xc5",
+ "EventName": "BR_MISP_RETIRED.JCC",
+ "PEBS": "1",
+ "SampleAfterValue": "200003",
+ "UMask": "0x7e"
+ },
+ {
+ "BriefDescription": "This event is deprecated. Refer to new event BR_MISP_RETIRED.INDIRECT",
+ "Deprecated": "1",
+ "EventCode": "0xc5",
+ "EventName": "BR_MISP_RETIRED.NON_RETURN_IND",
+ "PEBS": "1",
+ "SampleAfterValue": "200003",
+ "UMask": "0xeb"
+ },
+ {
+ "BriefDescription": "Counts the number of mispredicted near RET branch instructions retired.",
+ "EventCode": "0xc5",
+ "EventName": "BR_MISP_RETIRED.RETURN",
+ "PEBS": "1",
+ "SampleAfterValue": "200003",
+ "UMask": "0xf7"
+ },
+ {
+ "BriefDescription": "This event is deprecated. Refer to new event BR_MISP_RETIRED.COND_TAKEN",
+ "Deprecated": "1",
+ "EventCode": "0xc5",
+ "EventName": "BR_MISP_RETIRED.TAKEN_JCC",
+ "PEBS": "1",
+ "SampleAfterValue": "200003",
+ "UMask": "0xfe"
+ },
+ {
+ "BriefDescription": "Counts the number of unhalted core clock cycles. (Fixed event)",
+ "EventName": "CPU_CLK_UNHALTED.CORE",
+ "PublicDescription": "Counts the number of core cycles while the core is not in a halt state. The core enters the halt state when it is running the HLT instruction. The core frequency may change from time to time. For this reason this event may have a changing ratio with regards to time. This event uses fixed counter 1.",
+ "SampleAfterValue": "2000003",
+ "UMask": "0x2"
+ },
+ {
+ "BriefDescription": "Counts the number of unhalted core clock cycles.",
+ "EventCode": "0x3c",
+ "EventName": "CPU_CLK_UNHALTED.CORE_P",
+ "PublicDescription": "Counts the number of core cycles while the core is not in a halt state. The core enters the halt state when it is running the HLT instruction. The core frequency may change from time to time. For this reason this event may have a changing ratio with regards to time. This event uses a programmable general purpose performance counter.",
+ "SampleAfterValue": "2000003"
+ },
+ {
+ "BriefDescription": "Counts the number of unhalted reference clock cycles at TSC frequency. (Fixed event)",
+ "EventName": "CPU_CLK_UNHALTED.REF_TSC",
+ "PublicDescription": "Counts the number of reference cycles that the core is not in a halt state. The core enters the halt state when it is running the HLT instruction. This event is not affected by core frequency changes and increments at a fixed frequency that is also used for the Time Stamp Counter (TSC). This event uses fixed counter 2.",
+ "SampleAfterValue": "2000003",
+ "UMask": "0x3"
+ },
+ {
+ "BriefDescription": "Counts the number of unhalted reference clock cycles at TSC frequency.",
+ "EventCode": "0x3c",
+ "EventName": "CPU_CLK_UNHALTED.REF_TSC_P",
+ "PublicDescription": "Counts the number of reference cycles that the core is not in a halt state. The core enters the halt state when it is running the HLT instruction. This event is not affected by core frequency changes and increments at a fixed frequency that is also used for the Time Stamp Counter (TSC). This event uses a programmable general purpose performance counter.",
+ "SampleAfterValue": "2000003",
+ "UMask": "0x1"
+ },
+ {
+ "BriefDescription": "Counts the number of unhalted core clock cycles. (Fixed event)",
+ "EventName": "CPU_CLK_UNHALTED.THREAD",
+ "PublicDescription": "Counts the number of core cycles while the core is not in a halt state. The core enters the halt state when it is running the HLT instruction. The core frequency may change from time to time. For this reason this event may have a changing ratio with regards to time. This event uses fixed counter 1.",
+ "SampleAfterValue": "2000003",
+ "UMask": "0x2"
+ },
+ {
+ "BriefDescription": "Counts the number of unhalted core clock cycles.",
+ "EventCode": "0x3c",
+ "EventName": "CPU_CLK_UNHALTED.THREAD_P",
+ "PublicDescription": "Counts the number of core cycles while the core is not in a halt state. The core enters the halt state when it is running the HLT instruction. The core frequency may change from time to time. For this reason this event may have a changing ratio with regards to time. This event uses a programmable general purpose performance counter.",
+ "SampleAfterValue": "2000003"
+ },
+ {
+ "BriefDescription": "Counts the total number of instructions retired. (Fixed event)",
+ "EventName": "INST_RETIRED.ANY",
+ "PEBS": "1",
+ "PublicDescription": "Counts the total number of instructions that retired. For instructions that consist of multiple uops, this event counts the retirement of the last uop of the instruction. This event continues counting during hardware interrupts, traps, and inside interrupt handlers. This event uses fixed counter 0.",
+ "SampleAfterValue": "2000003",
+ "UMask": "0x1"
+ },
+ {
+ "BriefDescription": "Counts the total number of instructions retired.",
+ "EventCode": "0xc0",
+ "EventName": "INST_RETIRED.ANY_P",
+ "PEBS": "1",
+ "PublicDescription": "Counts the total number of instructions that retired. For instructions that consist of multiple uops, this event counts the retirement of the last uop of the instruction. This event continues counting during hardware interrupts, traps, and inside interrupt handlers. This event uses a programmable general purpose performance counter.",
+ "SampleAfterValue": "2000003"
+ },
+ {
+ "BriefDescription": "This event is deprecated. Refer to new event LD_BLOCKS.ADDRESS_ALIAS",
+ "Deprecated": "1",
+ "EventCode": "0x03",
+ "EventName": "LD_BLOCKS.4K_ALIAS",
+ "PEBS": "1",
+ "SampleAfterValue": "1000003",
+ "UMask": "0x4"
+ },
+ {
+ "BriefDescription": "Counts the number of retired loads that are blocked because it initially appears to be store forward blocked, but subsequently is shown not to be blocked based on 4K alias check.",
+ "EventCode": "0x03",
+ "EventName": "LD_BLOCKS.ADDRESS_ALIAS",
+ "PEBS": "1",
+ "SampleAfterValue": "1000003",
+ "UMask": "0x4"
+ },
+ {
+ "BriefDescription": "Counts the number of retired loads that are blocked because its address exactly matches an older store whose data is not ready.",
+ "EventCode": "0x03",
+ "EventName": "LD_BLOCKS.DATA_UNKNOWN",
+ "PEBS": "1",
+ "SampleAfterValue": "1000003",
+ "UMask": "0x1"
+ },
+ {
+ "BriefDescription": "Counts the number of machine clears due to memory ordering in which an internal load passes an older store within the same CPU.",
+ "EventCode": "0xc3",
+ "EventName": "MACHINE_CLEARS.DISAMBIGUATION",
+ "SampleAfterValue": "20003",
+ "UMask": "0x8"
+ },
+ {
+ "BriefDescription": "Counts the number of machines clears due to memory renaming.",
+ "EventCode": "0xc3",
+ "EventName": "MACHINE_CLEARS.MRN_NUKE",
+ "SampleAfterValue": "1000003",
+ "UMask": "0x80"
+ },
+ {
+ "BriefDescription": "Counts the number of machine clears due to a page fault. Counts both I-Side and D-Side (Loads/Stores) page faults. A page fault occurs when either the page is not present, or an access violation occurs.",
+ "EventCode": "0xc3",
+ "EventName": "MACHINE_CLEARS.PAGE_FAULT",
+ "SampleAfterValue": "20003",
+ "UMask": "0x20"
+ },
+ {
+ "BriefDescription": "Counts the number of machine clears that flush the pipeline and restart the machine with the use of microcode due to SMC, MEMORY_ORDERING, FP_ASSISTS, PAGE_FAULT, DISAMBIGUATION, and FPC_VIRTUAL_TRAP.",
+ "EventCode": "0xc3",
+ "EventName": "MACHINE_CLEARS.SLOW",
+ "SampleAfterValue": "20003",
+ "UMask": "0x6f"
+ },
+ {
+ "BriefDescription": "Counts the number of machine clears due to program modifying data (self modifying code) within 1K of a recently fetched code page.",
+ "EventCode": "0xc3",
+ "EventName": "MACHINE_CLEARS.SMC",
+ "SampleAfterValue": "20003",
+ "UMask": "0x1"
+ },
+ {
+ "BriefDescription": "Counts the number of issue slots not consumed by the backend due to a micro-sequencer (MS) scoreboard, which stalls the front-end from issuing from the UROM until a specified older uop retires.",
+ "EventCode": "0x75",
+ "EventName": "SERIALIZATION.NON_C01_MS_SCB",
+ "PublicDescription": "Counts the number of issue slots not consumed by the backend due to a micro-sequencer (MS) scoreboard, which stalls the front-end from issuing from the UROM until a specified older uop retires. The most commonly executed instruction with an MS scoreboard is PAUSE.",
+ "SampleAfterValue": "200003",
+ "UMask": "0x2"
+ },
+ {
+ "BriefDescription": "Counts the total number of issue slots that were not consumed by the backend because allocation is stalled due to a mispredicted jump or a machine clear.",
+ "EventCode": "0x73",
+ "EventName": "TOPDOWN_BAD_SPECULATION.ALL",
+ "PublicDescription": "Counts the total number of issue slots that were not consumed by the backend because allocation is stalled due to a mispredicted jump or a machine clear. Only issue slots wasted due to fast nukes such as memory ordering nukes are counted. Other nukes are not accounted for. Counts all issue slots blocked during this recovery window including relevant microcode flows and while uops are not yet available in the instruction queue (IQ) even if an FE_bound event occurs during this period. Also includes the issue slots that were consumed by the backend but were thrown away because they were younger than the mispredict or machine clear.",
+ "SampleAfterValue": "1000003"
+ },
+ {
+ "BriefDescription": "Counts the number of issue slots every cycle that were not consumed by the backend due to fast nukes such as memory ordering and memory disambiguation machine clears.",
+ "EventCode": "0x73",
+ "EventName": "TOPDOWN_BAD_SPECULATION.FASTNUKE",
+ "SampleAfterValue": "1000003",
+ "UMask": "0x2"
+ },
+ {
+ "BriefDescription": "Counts the total number of issue slots that were not consumed by the backend because allocation is stalled due to a machine clear (nuke) of any kind including memory ordering and memory disambiguation.",
+ "EventCode": "0x73",
+ "EventName": "TOPDOWN_BAD_SPECULATION.MACHINE_CLEARS",
+ "SampleAfterValue": "1000003",
+ "UMask": "0x3"
+ },
+ {
+ "BriefDescription": "Counts the number of issue slots every cycle that were not consumed by the backend due to branch mispredicts.",
+ "EventCode": "0x73",
+ "EventName": "TOPDOWN_BAD_SPECULATION.MISPREDICT",
+ "SampleAfterValue": "1000003",
+ "UMask": "0x4"
+ },
+ {
+ "BriefDescription": "Counts the number of issue slots every cycle that were not consumed by the backend due to a machine clear (nuke).",
+ "EventCode": "0x73",
+ "EventName": "TOPDOWN_BAD_SPECULATION.NUKE",
+ "SampleAfterValue": "1000003",
+ "UMask": "0x1"
+ },
+ {
+ "BriefDescription": "Counts the total number of issue slots every cycle that were not consumed by the backend due to backend stalls.",
+ "EventCode": "0x74",
+ "EventName": "TOPDOWN_BE_BOUND.ALL",
+ "SampleAfterValue": "1000003"
+ },
+ {
+ "BriefDescription": "Counts the number of issue slots every cycle that were not consumed by the backend due to certain allocation restrictions.",
+ "EventCode": "0x74",
+ "EventName": "TOPDOWN_BE_BOUND.ALLOC_RESTRICTIONS",
+ "SampleAfterValue": "1000003",
+ "UMask": "0x1"
+ },
+ {
+ "BriefDescription": "Counts the number of issue slots every cycle that were not consumed by the backend due to memory reservation stalls in which a scheduler is not able to accept uops.",
+ "EventCode": "0x74",
+ "EventName": "TOPDOWN_BE_BOUND.MEM_SCHEDULER",
+ "SampleAfterValue": "1000003",
+ "UMask": "0x2"
+ },
+ {
+ "BriefDescription": "Counts the number of issue slots every cycle that were not consumed by the backend due to IEC or FPC RAT stalls, which can be due to FIQ or IEC reservation stalls in which the integer, floating point or SIMD scheduler is not able to accept uops.",
+ "EventCode": "0x74",
+ "EventName": "TOPDOWN_BE_BOUND.NON_MEM_SCHEDULER",
+ "SampleAfterValue": "1000003",
+ "UMask": "0x8"
+ },
+ {
+ "BriefDescription": "Counts the number of issue slots every cycle that were not consumed by the backend due to the physical register file unable to accept an entry (marble stalls).",
+ "EventCode": "0x74",
+ "EventName": "TOPDOWN_BE_BOUND.REGISTER",
+ "SampleAfterValue": "1000003",
+ "UMask": "0x20"
+ },
+ {
+ "BriefDescription": "Counts the number of issue slots every cycle that were not consumed by the backend due to the reorder buffer being full (ROB stalls).",
+ "EventCode": "0x74",
+ "EventName": "TOPDOWN_BE_BOUND.REORDER_BUFFER",
+ "SampleAfterValue": "1000003",
+ "UMask": "0x40"
+ },
+ {
+ "BriefDescription": "Counts the number of issue slots every cycle that were not consumed by the backend due to scoreboards from the instruction queue (IQ), jump execution unit (JEU), or microcode sequencer (MS).",
+ "EventCode": "0x74",
+ "EventName": "TOPDOWN_BE_BOUND.SERIALIZATION",
+ "SampleAfterValue": "1000003",
+ "UMask": "0x10"
+ },
+ {
+ "BriefDescription": "Counts the total number of issue slots every cycle that were not consumed by the backend due to frontend stalls.",
+ "EventCode": "0x71",
+ "EventName": "TOPDOWN_FE_BOUND.ALL",
+ "SampleAfterValue": "1000003"
+ },
+ {
+ "BriefDescription": "Counts the number of issue slots every cycle that were not delivered by the frontend due to BACLEARS.",
+ "EventCode": "0x71",
+ "EventName": "TOPDOWN_FE_BOUND.BRANCH_DETECT",
+ "PublicDescription": "Counts the number of issue slots every cycle that were not delivered by the frontend due to BACLEARS, which occurs when the Branch Target Buffer (BTB) prediction or lack thereof, was corrected by a later branch predictor in the frontend. Includes BACLEARS due to all branch types including conditional and unconditional jumps, returns, and indirect branches.",
+ "SampleAfterValue": "1000003",
+ "UMask": "0x2"
+ },
+ {
+ "BriefDescription": "Counts the number of issue slots every cycle that were not delivered by the frontend due to BTCLEARS.",
+ "EventCode": "0x71",
+ "EventName": "TOPDOWN_FE_BOUND.BRANCH_RESTEER",
+ "PublicDescription": "Counts the number of issue slots every cycle that were not delivered by the frontend due to BTCLEARS, which occurs when the Branch Target Buffer (BTB) predicts a taken branch.",
+ "SampleAfterValue": "1000003",
+ "UMask": "0x40"
+ },
+ {
+ "BriefDescription": "Counts the number of issue slots every cycle that were not delivered by the frontend due to the microcode sequencer (MS).",
+ "EventCode": "0x71",
+ "EventName": "TOPDOWN_FE_BOUND.CISC",
+ "SampleAfterValue": "1000003",
+ "UMask": "0x1"
+ },
+ {
+ "BriefDescription": "Counts the number of issue slots every cycle that were not delivered by the frontend due to decode stalls.",
+ "EventCode": "0x71",
+ "EventName": "TOPDOWN_FE_BOUND.DECODE",
+ "SampleAfterValue": "1000003",
+ "UMask": "0x8"
+ },
+ {
+ "BriefDescription": "Counts the number of issue slots every cycle that were not delivered by the frontend due to frontend bandwidth restrictions due to decode, predecode, cisc, and other limitations.",
+ "EventCode": "0x71",
+ "EventName": "TOPDOWN_FE_BOUND.FRONTEND_BANDWIDTH",
+ "SampleAfterValue": "1000003",
+ "UMask": "0x8d"
+ },
+ {
+ "BriefDescription": "Counts the number of issue slots every cycle that were not delivered by the frontend due to a latency related stalls including BACLEARs, BTCLEARs, ITLB misses, and ICache misses.",
+ "EventCode": "0x71",
+ "EventName": "TOPDOWN_FE_BOUND.FRONTEND_LATENCY",
+ "SampleAfterValue": "1000003",
+ "UMask": "0x72"
+ },
+ {
+ "BriefDescription": "Counts the number of issue slots every cycle that were not delivered by the frontend due to ITLB misses.",
+ "EventCode": "0x71",
+ "EventName": "TOPDOWN_FE_BOUND.ITLB",
+ "PublicDescription": "Counts the number of issue slots every cycle that were not delivered by the frontend due to Instruction Table Lookaside Buffer (ITLB) misses.",
+ "SampleAfterValue": "1000003",
+ "UMask": "0x10"
+ },
+ {
+ "BriefDescription": "Counts the number of issue slots every cycle that were not delivered by the frontend due to other common frontend stalls not categorized.",
+ "EventCode": "0x71",
+ "EventName": "TOPDOWN_FE_BOUND.OTHER",
+ "SampleAfterValue": "1000003",
+ "UMask": "0x80"
+ },
+ {
+ "BriefDescription": "Counts the number of issue slots every cycle that were not delivered by the frontend due to wrong predecodes.",
+ "EventCode": "0x71",
+ "EventName": "TOPDOWN_FE_BOUND.PREDECODE",
+ "SampleAfterValue": "1000003",
+ "UMask": "0x4"
+ },
+ {
+ "BriefDescription": "Counts the total number of consumed retirement slots.",
+ "EventCode": "0xc2",
+ "EventName": "TOPDOWN_RETIRING.ALL",
+ "PEBS": "1",
+ "SampleAfterValue": "1000003"
+ },
+ {
+ "BriefDescription": "Counts the total number of uops retired.",
+ "EventCode": "0xc2",
+ "EventName": "UOPS_RETIRED.ALL",
+ "PEBS": "1",
+ "SampleAfterValue": "2000003"
+ },
+ {
+ "BriefDescription": "Counts the number of integer divide uops retired.",
+ "EventCode": "0xc2",
+ "EventName": "UOPS_RETIRED.IDIV",
+ "PEBS": "1",
+ "SampleAfterValue": "2000003",
+ "UMask": "0x10"
+ },
+ {
+ "BriefDescription": "Counts the number of uops that are from complex flows issued by the micro-sequencer (MS).",
+ "EventCode": "0xc2",
+ "EventName": "UOPS_RETIRED.MS",
+ "PEBS": "1",
+ "PublicDescription": "Counts the number of uops that are from complex flows issued by the Microcode Sequencer (MS). This includes uops from flows due to complex instructions, faults, assists, and inserted flows.",
+ "SampleAfterValue": "2000003",
+ "UMask": "0x1"
+ },
+ {
+ "BriefDescription": "Counts the number of x87 uops retired, includes those in MS flows.",
+ "EventCode": "0xc2",
+ "EventName": "UOPS_RETIRED.X87",
+ "PEBS": "1",
+ "SampleAfterValue": "2000003",
+ "UMask": "0x2"
+ }
+]
diff --git a/tools/perf/pmu-events/arch/x86/alderlaken/uncore-memory.json b/tools/perf/pmu-events/arch/x86/alderlaken/uncore-memory.json
new file mode 100644
index 0000000..2ccd9cf
--- /dev/null
+++ b/tools/perf/pmu-events/arch/x86/alderlaken/uncore-memory.json
@@ -0,0 +1,175 @@
+[
+ {
+ "BriefDescription": "Counts every 64B read request entering the Memory Controller 0 to DRAM (sum of all channels).",
+ "EventName": "UNC_MC0_RDCAS_COUNT_FREERUN",
+ "PerPkg": "1",
+ "PublicDescription": "Counts every 64B read request entering the Memory Controller 0 to DRAM (sum of all channels).",
+ "Unit": "iMC"
+ },
+ {
+ "BriefDescription": "Counts every 64B write request entering the Memory Controller 0 to DRAM (sum of all channels). Each write request counts as a new request incrementing this counter. However, same cache line write requests (both full and partial) are combined to a single 64 byte data transfer to DRAM.",
+ "EventName": "UNC_MC0_WRCAS_COUNT_FREERUN",
+ "PerPkg": "1",
+ "Unit": "iMC"
+ },
+ {
+ "BriefDescription": "Counts every 64B read request entering the Memory Controller 1 to DRAM (sum of all channels).",
+ "EventName": "UNC_MC1_RDCAS_COUNT_FREERUN",
+ "PerPkg": "1",
+ "PublicDescription": "Counts every 64B read entering the Memory Controller 1 to DRAM (sum of all channels).",
+ "Unit": "iMC"
+ },
+ {
+ "BriefDescription": "Counts every 64B write request entering the Memory Controller 1 to DRAM (sum of all channels). Each write request counts as a new request incrementing this counter. However, same cache line write requests (both full and partial) are combined to a single 64 byte data transfer to DRAM.",
+ "EventName": "UNC_MC1_WRCAS_COUNT_FREERUN",
+ "PerPkg": "1",
+ "Unit": "iMC"
+ },
+ {
+ "BriefDescription": "ACT command for a read request sent to DRAM",
+ "EventCode": "0x24",
+ "EventName": "UNC_M_ACT_COUNT_RD",
+ "PerPkg": "1",
+ "Unit": "iMC"
+ },
+ {
+ "BriefDescription": "ACT command sent to DRAM",
+ "EventCode": "0x26",
+ "EventName": "UNC_M_ACT_COUNT_TOTAL",
+ "PerPkg": "1",
+ "Unit": "iMC"
+ },
+ {
+ "BriefDescription": "ACT command for a write request sent to DRAM",
+ "EventCode": "0x25",
+ "EventName": "UNC_M_ACT_COUNT_WR",
+ "PerPkg": "1",
+ "Unit": "iMC"
+ },
+ {
+ "BriefDescription": "Read CAS command sent to DRAM",
+ "EventCode": "0x22",
+ "EventName": "UNC_M_CAS_COUNT_RD",
+ "PerPkg": "1",
+ "Unit": "iMC"
+ },
+ {
+ "BriefDescription": "Write CAS command sent to DRAM",
+ "EventCode": "0x23",
+ "EventName": "UNC_M_CAS_COUNT_WR",
+ "PerPkg": "1",
+ "Unit": "iMC"
+ },
+ {
+ "BriefDescription": "Number of clocks",
+ "EventCode": "0x01",
+ "EventName": "UNC_M_CLOCKTICKS",
+ "PerPkg": "1",
+ "Unit": "iMC"
+ },
+ {
+ "BriefDescription": "incoming read request page status is Page Empty",
+ "EventCode": "0x1D",
+ "EventName": "UNC_M_DRAM_PAGE_EMPTY_RD",
+ "PerPkg": "1",
+ "Unit": "iMC"
+ },
+ {
+ "BriefDescription": "incoming write request page status is Page Empty",
+ "EventCode": "0x20",
+ "EventName": "UNC_M_DRAM_PAGE_EMPTY_WR",
+ "PerPkg": "1",
+ "Unit": "iMC"
+ },
+ {
+ "BriefDescription": "incoming read request page status is Page Hit",
+ "EventCode": "0x1C",
+ "EventName": "UNC_M_DRAM_PAGE_HIT_RD",
+ "PerPkg": "1",
+ "Unit": "iMC"
+ },
+ {
+ "BriefDescription": "incoming write request page status is Page Hit",
+ "EventCode": "0x1F",
+ "EventName": "UNC_M_DRAM_PAGE_HIT_WR",
+ "PerPkg": "1",
+ "Unit": "iMC"
+ },
+ {
+ "BriefDescription": "incoming read request page status is Page Miss",
+ "EventCode": "0x1E",
+ "EventName": "UNC_M_DRAM_PAGE_MISS_RD",
+ "PerPkg": "1",
+ "Unit": "iMC"
+ },
+ {
+ "BriefDescription": "incoming write request page status is Page Miss",
+ "EventCode": "0x21",
+ "EventName": "UNC_M_DRAM_PAGE_MISS_WR",
+ "PerPkg": "1",
+ "Unit": "iMC"
+ },
+ {
+ "BriefDescription": "Any Rank at Hot state",
+ "EventCode": "0x19",
+ "EventName": "UNC_M_DRAM_THERMAL_HOT",
+ "PerPkg": "1",
+ "Unit": "iMC"
+ },
+ {
+ "BriefDescription": "Any Rank at Warm state",
+ "EventCode": "0x1A",
+ "EventName": "UNC_M_DRAM_THERMAL_WARM",
+ "PerPkg": "1",
+ "Unit": "iMC"
+ },
+ {
+ "BriefDescription": "Incoming read prefetch request from IA.",
+ "EventCode": "0x0A",
+ "EventName": "UNC_M_PREFETCH_RD",
+ "PerPkg": "1",
+ "Unit": "iMC"
+ },
+ {
+ "BriefDescription": "PRE command sent to DRAM due to page table idle timer expiration",
+ "EventCode": "0x28",
+ "EventName": "UNC_M_PRE_COUNT_IDLE",
+ "PerPkg": "1",
+ "Unit": "iMC"
+ },
+ {
+ "BriefDescription": "PRE command sent to DRAM for a read/write request",
+ "EventCode": "0x27",
+ "EventName": "UNC_M_PRE_COUNT_PAGE_MISS",
+ "PerPkg": "1",
+ "Unit": "iMC"
+ },
+ {
+ "BriefDescription": "Incoming VC0 read request",
+ "EventCode": "0x02",
+ "EventName": "UNC_M_VC0_REQUESTS_RD",
+ "PerPkg": "1",
+ "Unit": "iMC"
+ },
+ {
+ "BriefDescription": "Incoming VC0 write request",
+ "EventCode": "0x03",
+ "EventName": "UNC_M_VC0_REQUESTS_WR",
+ "PerPkg": "1",
+ "Unit": "iMC"
+ },
+ {
+ "BriefDescription": "Incoming VC1 read request",
+ "EventCode": "0x04",
+ "EventName": "UNC_M_VC1_REQUESTS_RD",
+ "PerPkg": "1",
+ "Unit": "iMC"
+ },
+ {
+ "BriefDescription": "Incoming VC1 write request",
+ "EventCode": "0x05",
+ "EventName": "UNC_M_VC1_REQUESTS_WR",
+ "PerPkg": "1",
+ "Unit": "iMC"
+ }
+]
diff --git a/tools/perf/pmu-events/arch/x86/alderlaken/uncore-other.json b/tools/perf/pmu-events/arch/x86/alderlaken/uncore-other.json
new file mode 100644
index 0000000..f9e7777
--- /dev/null
+++ b/tools/perf/pmu-events/arch/x86/alderlaken/uncore-other.json
@@ -0,0 +1,33 @@
+[
+ {
+ "BriefDescription": "Number of requests allocated in Coherency Tracker.",
+ "EventCode": "0x84",
+ "EventName": "UNC_ARB_COH_TRK_REQUESTS.ALL",
+ "PerPkg": "1",
+ "UMask": "0x1",
+ "Unit": "ARB"
+ },
+ {
+ "BriefDescription": "Each cycle counts number of all outgoing valid entries in ReqTrk. Such entry is defined as valid from its allocation in ReqTrk till deallocation. Accounts for Coherent and non-coherent traffic.",
+ "EventCode": "0x80",
+ "EventName": "UNC_ARB_TRK_OCCUPANCY.ALL",
+ "PerPkg": "1",
+ "UMask": "0x1",
+ "Unit": "ARB"
+ },
+ {
+ "BriefDescription": "Counts the number of coherent and in-coherent requests initiated by IA cores, processor graphic units, or LLC.",
+ "EventCode": "0x81",
+ "EventName": "UNC_ARB_TRK_REQUESTS.ALL",
+ "PerPkg": "1",
+ "UMask": "0x1",
+ "Unit": "ARB"
+ },
+ {
+ "BriefDescription": "This 48-bit fixed counter counts the UCLK cycles.",
+ "EventCode": "0xff",
+ "EventName": "UNC_CLOCK.SOCKET",
+ "PerPkg": "1",
+ "Unit": "CLOCK"
+ }
+]
diff --git a/tools/perf/pmu-events/arch/x86/alderlaken/virtual-memory.json b/tools/perf/pmu-events/arch/x86/alderlaken/virtual-memory.json
new file mode 100644
index 0000000..67fd640
--- /dev/null
+++ b/tools/perf/pmu-events/arch/x86/alderlaken/virtual-memory.json
@@ -0,0 +1,47 @@
+[
+ {
+ "BriefDescription": "Counts the number of page walks completed due to load DTLB misses to any page size.",
+ "EventCode": "0x08",
+ "EventName": "DTLB_LOAD_MISSES.WALK_COMPLETED",
+ "PublicDescription": "Counts the number of page walks completed due to loads (including SW prefetches) whose address translations missed in all Translation Lookaside Buffer (TLB) levels and were mapped to any page size. Includes page walks that page fault.",
+ "SampleAfterValue": "200003",
+ "UMask": "0xe"
+ },
+ {
+ "BriefDescription": "Counts the number of page walks completed due to store DTLB misses to any page size.",
+ "EventCode": "0x49",
+ "EventName": "DTLB_STORE_MISSES.WALK_COMPLETED",
+ "PublicDescription": "Counts the number of page walks completed due to stores whose address translations missed in all Translation Lookaside Buffer (TLB) levels and were mapped to any page size. Includes page walks that page fault.",
+ "SampleAfterValue": "2000003",
+ "UMask": "0xe"
+ },
+ {
+ "BriefDescription": "Counts the number of page walks initiated by a instruction fetch that missed the first and second level TLBs.",
+ "EventCode": "0x85",
+ "EventName": "ITLB_MISSES.MISS_CAUSED_WALK",
+ "SampleAfterValue": "1000003",
+ "UMask": "0x1"
+ },
+ {
+ "BriefDescription": "Counts the number of page walks due to an instruction fetch that miss the PDE (Page Directory Entry) cache.",
+ "EventCode": "0x85",
+ "EventName": "ITLB_MISSES.PDE_CACHE_MISS",
+ "SampleAfterValue": "2000003",
+ "UMask": "0x80"
+ },
+ {
+ "BriefDescription": "Counts the number of page walks completed due to instruction fetch misses to any page size.",
+ "EventCode": "0x85",
+ "EventName": "ITLB_MISSES.WALK_COMPLETED",
+ "PublicDescription": "Counts the number of page walks completed due to instruction fetches whose address translations missed in all Translation Lookaside Buffer (TLB) levels and were mapped to any page size. Includes page walks that page fault.",
+ "SampleAfterValue": "200003",
+ "UMask": "0xe"
+ },
+ {
+ "BriefDescription": "Counts the number of cycles that the head (oldest load) of the load buffer and retirement are both stalled due to a DTLB miss.",
+ "EventCode": "0x05",
+ "EventName": "LD_HEAD.DTLB_MISS_AT_RET",
+ "SampleAfterValue": "1000003",
+ "UMask": "0x90"
+ }
+]
diff --git a/tools/perf/pmu-events/arch/x86/mapfile.csv b/tools/perf/pmu-events/arch/x86/mapfile.csv
index 5e609b8..df47462 100644
--- a/tools/perf/pmu-events/arch/x86/mapfile.csv
+++ b/tools/perf/pmu-events/arch/x86/mapfile.csv
@@ -1,5 +1,6 @@
Family-model,Version,Filename,EventType
-GenuineIntel-6-(97|9A|B7|BA|BE|BF),v1.15,alderlake,core
+GenuineIntel-6-(97|9A|B7|BA|BF),v1.16,alderlake,core
+GenuineIntel-6-BE,v1.16,alderlaken,core
GenuineIntel-6-(1C|26|27|35|36),v4,bonnell,core
GenuineIntel-6-(3D|47),v26,broadwell,core
GenuineIntel-6-56,v23,broadwellde,core
diff --git a/tools/perf/pmu-events/jevents.py b/tools/perf/pmu-events/jevents.py
index 0daa3e0..4c398e0 100755
--- a/tools/perf/pmu-events/jevents.py
+++ b/tools/perf/pmu-events/jevents.py
@@ -4,6 +4,7 @@
import argparse
import csv
import json
+import metric
import os
import sys
from typing import (Callable, Dict, Optional, Sequence, Set, Tuple)
@@ -268,9 +269,10 @@
self.metric_name = jd.get('MetricName')
self.metric_group = jd.get('MetricGroup')
self.metric_constraint = jd.get('MetricConstraint')
- self.metric_expr = jd.get('MetricExpr')
- if self.metric_expr:
- self.metric_expr = self.metric_expr.replace('\\', '\\\\')
+ self.metric_expr = None
+ if 'MetricExpr' in jd:
+ self.metric_expr = metric.ParsePerfJson(jd['MetricExpr']).Simplify()
+
arch_std = jd.get('ArchStdEvent')
if precise and self.desc and '(Precise Event)' not in self.desc:
extra_desc += ' (Must be precise)' if precise == '2' else (' (Precise '
@@ -322,6 +324,10 @@
s = ''
for attr in _json_event_attributes:
x = getattr(self, attr)
+ if x and attr == 'metric_expr':
+ # Convert parsed metric expressions into a string. Slashes
+ # must be doubled in the file.
+ x = x.ToPerfJson().replace('\\', '\\\\')
s += f'{x}\\000' if x else '\\000'
return s
diff --git a/tools/perf/pmu-events/metric.py b/tools/perf/pmu-events/metric.py
new file mode 100644
index 0000000..4797ed4
--- /dev/null
+++ b/tools/perf/pmu-events/metric.py
@@ -0,0 +1,502 @@
+# SPDX-License-Identifier: (LGPL-2.1 OR BSD-2-Clause)
+"""Parse or generate representations of perf metrics."""
+import ast
+import decimal
+import json
+import re
+from typing import Dict, List, Optional, Set, Union
+
+
+class Expression:
+ """Abstract base class of elements in a metric expression."""
+
+ def ToPerfJson(self) -> str:
+ """Returns a perf json file encoded representation."""
+ raise NotImplementedError()
+
+ def ToPython(self) -> str:
+ """Returns a python expr parseable representation."""
+ raise NotImplementedError()
+
+ def Simplify(self):
+ """Returns a simplified version of self."""
+ raise NotImplementedError()
+
+ def Equals(self, other) -> bool:
+ """Returns true when two expressions are the same."""
+ raise NotImplementedError()
+
+ def __str__(self) -> str:
+ return self.ToPerfJson()
+
+ def __or__(self, other: Union[int, float, 'Expression']) -> 'Operator':
+ return Operator('|', self, other)
+
+ def __ror__(self, other: Union[int, float, 'Expression']) -> 'Operator':
+ return Operator('|', other, self)
+
+ def __xor__(self, other: Union[int, float, 'Expression']) -> 'Operator':
+ return Operator('^', self, other)
+
+ def __and__(self, other: Union[int, float, 'Expression']) -> 'Operator':
+ return Operator('&', self, other)
+
+ def __lt__(self, other: Union[int, float, 'Expression']) -> 'Operator':
+ return Operator('<', self, other)
+
+ def __gt__(self, other: Union[int, float, 'Expression']) -> 'Operator':
+ return Operator('>', self, other)
+
+ def __add__(self, other: Union[int, float, 'Expression']) -> 'Operator':
+ return Operator('+', self, other)
+
+ def __radd__(self, other: Union[int, float, 'Expression']) -> 'Operator':
+ return Operator('+', other, self)
+
+ def __sub__(self, other: Union[int, float, 'Expression']) -> 'Operator':
+ return Operator('-', self, other)
+
+ def __rsub__(self, other: Union[int, float, 'Expression']) -> 'Operator':
+ return Operator('-', other, self)
+
+ def __mul__(self, other: Union[int, float, 'Expression']) -> 'Operator':
+ return Operator('*', self, other)
+
+ def __rmul__(self, other: Union[int, float, 'Expression']) -> 'Operator':
+ return Operator('*', other, self)
+
+ def __truediv__(self, other: Union[int, float, 'Expression']) -> 'Operator':
+ return Operator('/', self, other)
+
+ def __rtruediv__(self, other: Union[int, float, 'Expression']) -> 'Operator':
+ return Operator('/', other, self)
+
+ def __mod__(self, other: Union[int, float, 'Expression']) -> 'Operator':
+ return Operator('%', self, other)
+
+
+def _Constify(val: Union[bool, int, float, Expression]) -> Expression:
+ """Used to ensure that the nodes in the expression tree are all Expression."""
+ if isinstance(val, bool):
+ return Constant(1 if val else 0)
+ if isinstance(val, (int, float)):
+ return Constant(val)
+ return val
+
+
+# Simple lookup for operator precedence, used to avoid unnecessary
+# brackets. Precedence matches that of python and the simple expression parser.
+_PRECEDENCE = {
+ '|': 0,
+ '^': 1,
+ '&': 2,
+ '<': 3,
+ '>': 3,
+ '+': 4,
+ '-': 4,
+ '*': 5,
+ '/': 5,
+ '%': 5,
+}
+
+
+class Operator(Expression):
+ """Represents a binary operator in the parse tree."""
+
+ def __init__(self, operator: str, lhs: Union[int, float, Expression],
+ rhs: Union[int, float, Expression]):
+ self.operator = operator
+ self.lhs = _Constify(lhs)
+ self.rhs = _Constify(rhs)
+
+ def Bracket(self,
+ other: Expression,
+ other_str: str,
+ rhs: bool = False) -> str:
+ """If necessary brackets the given other value.
+
+ If ``other`` is an operator then a bracket is necessary when
+ this/self operator has higher precedence. Consider: '(a + b) * c',
+ ``other_str`` will be 'a + b'. A bracket is necessary as without
+ the bracket 'a + b * c' will evaluate 'b * c' first. However, '(a
+ * b) + c' doesn't need a bracket as 'a * b' will always be
+ evaluated first. For 'a / (b * c)' (ie the same precedence level
+ operations) then we add the bracket to best match the original
+ input, but not for '(a / b) * c' where the bracket is unnecessary.
+
+ Args:
+ other (Expression): is a lhs or rhs operator
+ other_str (str): ``other`` in the appropriate string form
+ rhs (bool): is ``other`` on the RHS
+
+ Returns:
+ str: possibly bracketed other_str
+ """
+ if isinstance(other, Operator):
+ if _PRECEDENCE.get(self.operator, -1) > _PRECEDENCE.get(
+ other.operator, -1):
+ return f'({other_str})'
+ if rhs and _PRECEDENCE.get(self.operator, -1) == _PRECEDENCE.get(
+ other.operator, -1):
+ return f'({other_str})'
+ return other_str
+
+ def ToPerfJson(self):
+ return (f'{self.Bracket(self.lhs, self.lhs.ToPerfJson())} {self.operator} '
+ f'{self.Bracket(self.rhs, self.rhs.ToPerfJson(), True)}')
+
+ def ToPython(self):
+ return (f'{self.Bracket(self.lhs, self.lhs.ToPython())} {self.operator} '
+ f'{self.Bracket(self.rhs, self.rhs.ToPython(), True)}')
+
+ def Simplify(self) -> Expression:
+ lhs = self.lhs.Simplify()
+ rhs = self.rhs.Simplify()
+ if isinstance(lhs, Constant) and isinstance(rhs, Constant):
+ return Constant(ast.literal_eval(lhs + self.operator + rhs))
+
+ if isinstance(self.lhs, Constant):
+ if self.operator in ('+', '|') and lhs.value == '0':
+ return rhs
+
+ # Simplify multiplication by 0 except for the slot event which
+ # is deliberately introduced using this pattern.
+ if self.operator == '*' and lhs.value == '0' and (
+ not isinstance(rhs, Event) or 'slots' not in rhs.name.lower()):
+ return Constant(0)
+
+ if self.operator == '*' and lhs.value == '1':
+ return rhs
+
+ if isinstance(rhs, Constant):
+ if self.operator in ('+', '|') and rhs.value == '0':
+ return lhs
+
+ if self.operator == '*' and rhs.value == '0':
+ return Constant(0)
+
+ if self.operator == '*' and self.rhs.value == '1':
+ return lhs
+
+ return Operator(self.operator, lhs, rhs)
+
+ def Equals(self, other: Expression) -> bool:
+ if isinstance(other, Operator):
+ return self.operator == other.operator and self.lhs.Equals(
+ other.lhs) and self.rhs.Equals(other.rhs)
+ return False
+
+
+class Select(Expression):
+ """Represents a select ternary in the parse tree."""
+
+ def __init__(self, true_val: Union[int, float, Expression],
+ cond: Union[int, float, Expression],
+ false_val: Union[int, float, Expression]):
+ self.true_val = _Constify(true_val)
+ self.cond = _Constify(cond)
+ self.false_val = _Constify(false_val)
+
+ def ToPerfJson(self):
+ true_str = self.true_val.ToPerfJson()
+ cond_str = self.cond.ToPerfJson()
+ false_str = self.false_val.ToPerfJson()
+ return f'({true_str} if {cond_str} else {false_str})'
+
+ def ToPython(self):
+ return (f'Select({self.true_val.ToPython()}, {self.cond.ToPython()}, '
+ f'{self.false_val.ToPython()})')
+
+ def Simplify(self) -> Expression:
+ cond = self.cond.Simplify()
+ true_val = self.true_val.Simplify()
+ false_val = self.false_val.Simplify()
+ if isinstance(cond, Constant):
+ return false_val if cond.value == '0' else true_val
+
+ if true_val.Equals(false_val):
+ return true_val
+
+ return Select(true_val, cond, false_val)
+
+ def Equals(self, other: Expression) -> bool:
+ if isinstance(other, Select):
+ return self.cond.Equals(other.cond) and self.false_val.Equals(
+ other.false_val) and self.true_val.Equals(other.true_val)
+ return False
+
+
+class Function(Expression):
+ """A function in an expression like min, max, d_ratio."""
+
+ def __init__(self,
+ fn: str,
+ lhs: Union[int, float, Expression],
+ rhs: Optional[Union[int, float, Expression]] = None):
+ self.fn = fn
+ self.lhs = _Constify(lhs)
+ self.rhs = _Constify(rhs)
+
+ def ToPerfJson(self):
+ if self.rhs:
+ return f'{self.fn}({self.lhs.ToPerfJson()}, {self.rhs.ToPerfJson()})'
+ return f'{self.fn}({self.lhs.ToPerfJson()})'
+
+ def ToPython(self):
+ if self.rhs:
+ return f'{self.fn}({self.lhs.ToPython()}, {self.rhs.ToPython()})'
+ return f'{self.fn}({self.lhs.ToPython()})'
+
+ def Simplify(self) -> Expression:
+ lhs = self.lhs.Simplify()
+ rhs = self.rhs.Simplify() if self.rhs else None
+ if isinstance(lhs, Constant) and isinstance(rhs, Constant):
+ if self.fn == 'd_ratio':
+ if rhs.value == '0':
+ return Constant(0)
+ Constant(ast.literal_eval(f'{lhs} / {rhs}'))
+ return Constant(ast.literal_eval(f'{self.fn}({lhs}, {rhs})'))
+
+ return Function(self.fn, lhs, rhs)
+
+ def Equals(self, other: Expression) -> bool:
+ if isinstance(other, Function):
+ return self.fn == other.fn and self.lhs.Equals(
+ other.lhs) and self.rhs.Equals(other.rhs)
+ return False
+
+
+def _FixEscapes(s: str) -> str:
+ s = re.sub(r'([^\\]),', r'\1\\,', s)
+ return re.sub(r'([^\\])=', r'\1\\=', s)
+
+
+class Event(Expression):
+ """An event in an expression."""
+
+ def __init__(self, name: str, legacy_name: str = ''):
+ self.name = _FixEscapes(name)
+ self.legacy_name = _FixEscapes(legacy_name)
+
+ def ToPerfJson(self):
+ result = re.sub('/', '@', self.name)
+ return result
+
+ def ToPython(self):
+ return f'Event(r"{self.name}")'
+
+ def Simplify(self) -> Expression:
+ return self
+
+ def Equals(self, other: Expression) -> bool:
+ return isinstance(other, Event) and self.name == other.name
+
+
+class Constant(Expression):
+ """A constant within the expression tree."""
+
+ def __init__(self, value: Union[float, str]):
+ ctx = decimal.Context()
+ ctx.prec = 20
+ dec = ctx.create_decimal(repr(value) if isinstance(value, float) else value)
+ self.value = dec.normalize().to_eng_string()
+ self.value = self.value.replace('+', '')
+ self.value = self.value.replace('E', 'e')
+
+ def ToPerfJson(self):
+ return self.value
+
+ def ToPython(self):
+ return f'Constant({self.value})'
+
+ def Simplify(self) -> Expression:
+ return self
+
+ def Equals(self, other: Expression) -> bool:
+ return isinstance(other, Constant) and self.value == other.value
+
+
+class Literal(Expression):
+ """A runtime literal within the expression tree."""
+
+ def __init__(self, value: str):
+ self.value = value
+
+ def ToPerfJson(self):
+ return self.value
+
+ def ToPython(self):
+ return f'Literal({self.value})'
+
+ def Simplify(self) -> Expression:
+ return self
+
+ def Equals(self, other: Expression) -> bool:
+ return isinstance(other, Literal) and self.value == other.value
+
+
+def min(lhs: Union[int, float, Expression], rhs: Union[int, float,
+ Expression]) -> Function:
+ # pylint: disable=redefined-builtin
+ # pylint: disable=invalid-name
+ return Function('min', lhs, rhs)
+
+
+def max(lhs: Union[int, float, Expression], rhs: Union[int, float,
+ Expression]) -> Function:
+ # pylint: disable=redefined-builtin
+ # pylint: disable=invalid-name
+ return Function('max', lhs, rhs)
+
+
+def d_ratio(lhs: Union[int, float, Expression],
+ rhs: Union[int, float, Expression]) -> Function:
+ # pylint: disable=redefined-builtin
+ # pylint: disable=invalid-name
+ return Function('d_ratio', lhs, rhs)
+
+
+def source_count(event: Event) -> Function:
+ # pylint: disable=redefined-builtin
+ # pylint: disable=invalid-name
+ return Function('source_count', event)
+
+
+class Metric:
+ """An individual metric that will specifiable on the perf command line."""
+ groups: Set[str]
+ expr: Expression
+ scale_unit: str
+ constraint: bool
+
+ def __init__(self,
+ name: str,
+ description: str,
+ expr: Expression,
+ scale_unit: str,
+ constraint: bool = False):
+ self.name = name
+ self.description = description
+ self.expr = expr.Simplify()
+ # Workraound valid_only_metric hiding certain metrics based on unit.
+ scale_unit = scale_unit.replace('/sec', ' per sec')
+ if scale_unit[0].isdigit():
+ self.scale_unit = scale_unit
+ else:
+ self.scale_unit = f'1{scale_unit}'
+ self.constraint = constraint
+ self.groups = set()
+
+ def __lt__(self, other):
+ """Sort order."""
+ return self.name < other.name
+
+ def AddToMetricGroup(self, group):
+ """Callback used when being added to a MetricGroup."""
+ self.groups.add(group.name)
+
+ def Flatten(self) -> Set['Metric']:
+ """Return a leaf metric."""
+ return set([self])
+
+ def ToPerfJson(self) -> Dict[str, str]:
+ """Return as dictionary for Json generation."""
+ result = {
+ 'MetricName': self.name,
+ 'MetricGroup': ';'.join(sorted(self.groups)),
+ 'BriefDescription': self.description,
+ 'MetricExpr': self.expr.ToPerfJson(),
+ 'ScaleUnit': self.scale_unit
+ }
+ if self.constraint:
+ result['MetricConstraint'] = 'NO_NMI_WATCHDOG'
+
+ return result
+
+
+class _MetricJsonEncoder(json.JSONEncoder):
+ """Special handling for Metric objects."""
+
+ def default(self, o):
+ if isinstance(o, Metric):
+ return o.ToPerfJson()
+ return json.JSONEncoder.default(self, o)
+
+
+class MetricGroup:
+ """A group of metrics.
+
+ Metric groups may be specificd on the perf command line, but within
+ the json they aren't encoded. Metrics may be in multiple groups
+ which can facilitate arrangements similar to trees.
+ """
+
+ def __init__(self, name: str, metric_list: List[Union[Metric,
+ 'MetricGroup']]):
+ self.name = name
+ self.metric_list = metric_list
+ for metric in metric_list:
+ metric.AddToMetricGroup(self)
+
+ def AddToMetricGroup(self, group):
+ """Callback used when a MetricGroup is added into another."""
+ for metric in self.metric_list:
+ metric.AddToMetricGroup(group)
+
+ def Flatten(self) -> Set[Metric]:
+ """Returns a set of all leaf metrics."""
+ result = set()
+ for x in self.metric_list:
+ result = result.union(x.Flatten())
+
+ return result
+
+ def ToPerfJson(self) -> str:
+ return json.dumps(sorted(self.Flatten()), indent=2, cls=_MetricJsonEncoder)
+
+ def __str__(self) -> str:
+ return self.ToPerfJson()
+
+
+class _RewriteIfExpToSelect(ast.NodeTransformer):
+
+ def visit_IfExp(self, node):
+ # pylint: disable=invalid-name
+ self.generic_visit(node)
+ call = ast.Call(
+ func=ast.Name(id='Select', ctx=ast.Load()),
+ args=[node.body, node.test, node.orelse],
+ keywords=[])
+ ast.copy_location(call, node.test)
+ return call
+
+
+def ParsePerfJson(orig: str) -> Expression:
+ """A simple json metric expression decoder.
+
+ Converts a json encoded metric expression by way of python's ast and
+ eval routine. First tokens are mapped to Event calls, then
+ accidentally converted keywords or literals are mapped to their
+ appropriate calls. Python's ast is used to match if-else that can't
+ be handled via operator overloading. Finally the ast is evaluated.
+
+ Args:
+ orig (str): String to parse.
+
+ Returns:
+ Expression: The parsed string.
+ """
+ # pylint: disable=eval-used
+ py = orig.strip()
+ py = re.sub(r'([a-zA-Z][^-+/\* \\\(\),]*(?:\\.[^-+/\* \\\(\),]*)*)',
+ r'Event(r"\1")', py)
+ py = re.sub(r'#Event\(r"([^"]*)"\)', r'Literal("#\1")', py)
+ py = re.sub(r'([0-9]+)Event\(r"(e[0-9]+)"\)', r'\1\2', py)
+ keywords = ['if', 'else', 'min', 'max', 'd_ratio', 'source_count']
+ for kw in keywords:
+ py = re.sub(rf'Event\(r"{kw}"\)', kw, py)
+
+ parsed = ast.parse(py, mode='eval')
+ _RewriteIfExpToSelect().visit(parsed)
+ parsed = ast.fix_missing_locations(parsed)
+ return _Constify(eval(compile(parsed, orig, 'eval')))
diff --git a/tools/perf/pmu-events/metric_test.py b/tools/perf/pmu-events/metric_test.py
new file mode 100644
index 0000000..15315d0f
--- /dev/null
+++ b/tools/perf/pmu-events/metric_test.py
@@ -0,0 +1,157 @@
+# SPDX-License-Identifier: (LGPL-2.1 OR BSD-2-Clause)
+import unittest
+from metric import Constant
+from metric import Event
+from metric import ParsePerfJson
+
+
+class TestMetricExpressions(unittest.TestCase):
+
+ def test_Operators(self):
+ a = Event('a')
+ b = Event('b')
+ self.assertEqual((a | b).ToPerfJson(), 'a | b')
+ self.assertEqual((a ^ b).ToPerfJson(), 'a ^ b')
+ self.assertEqual((a & b).ToPerfJson(), 'a & b')
+ self.assertEqual((a < b).ToPerfJson(), 'a < b')
+ self.assertEqual((a > b).ToPerfJson(), 'a > b')
+ self.assertEqual((a + b).ToPerfJson(), 'a + b')
+ self.assertEqual((a - b).ToPerfJson(), 'a - b')
+ self.assertEqual((a * b).ToPerfJson(), 'a * b')
+ self.assertEqual((a / b).ToPerfJson(), 'a / b')
+ self.assertEqual((a % b).ToPerfJson(), 'a % b')
+ one = Constant(1)
+ self.assertEqual((a + one).ToPerfJson(), 'a + 1')
+
+ def test_Brackets(self):
+ a = Event('a')
+ b = Event('b')
+ c = Event('c')
+ self.assertEqual((a * b + c).ToPerfJson(), 'a * b + c')
+ self.assertEqual((a + b * c).ToPerfJson(), 'a + b * c')
+ self.assertEqual(((a + a) + a).ToPerfJson(), 'a + a + a')
+ self.assertEqual(((a + b) * c).ToPerfJson(), '(a + b) * c')
+ self.assertEqual((a + (b * c)).ToPerfJson(), 'a + b * c')
+ self.assertEqual(((a / b) * c).ToPerfJson(), 'a / b * c')
+ self.assertEqual((a / (b * c)).ToPerfJson(), 'a / (b * c)')
+
+ def test_ParsePerfJson(self):
+ # Based on an example of a real metric.
+ before = '(a + b + c + d) / (2 * e)'
+ after = before
+ self.assertEqual(ParsePerfJson(before).ToPerfJson(), after)
+
+ # Parsing should handle events with '-' in their name. Note, in
+ # the json file the '\' are doubled to '\\'.
+ before = r'topdown\-fe\-bound / topdown\-slots - 1'
+ after = before
+ self.assertEqual(ParsePerfJson(before).ToPerfJson(), after)
+
+ # Parsing should handle escaped modifiers. Note, in the json file
+ # the '\' are doubled to '\\'.
+ before = r'arb@event\=0x81\,umask\=0x1@ + arb@event\=0x84\,umask\=0x1@'
+ after = before
+ self.assertEqual(ParsePerfJson(before).ToPerfJson(), after)
+
+ # Parsing should handle exponents in numbers.
+ before = r'a + 1e12 + b'
+ after = before
+ self.assertEqual(ParsePerfJson(before).ToPerfJson(), after)
+
+ def test_IfElseTests(self):
+ # if-else needs rewriting to Select and back.
+ before = r'Event1 if #smt_on else Event2'
+ after = f'({before})'
+ self.assertEqual(ParsePerfJson(before).ToPerfJson(), after)
+
+ before = r'Event1 if 0 else Event2'
+ after = f'({before})'
+ self.assertEqual(ParsePerfJson(before).ToPerfJson(), after)
+
+ before = r'Event1 if 1 else Event2'
+ after = f'({before})'
+ self.assertEqual(ParsePerfJson(before).ToPerfJson(), after)
+
+ # Ensure the select is evaluate last.
+ before = r'Event1 + 1 if Event2 < 2 else Event3 + 3'
+ after = (r'Select(Event(r"Event1") + Constant(1), Event(r"Event2") < '
+ r'Constant(2), Event(r"Event3") + Constant(3))')
+ self.assertEqual(ParsePerfJson(before).ToPython(), after)
+
+ before = r'Event1 > 1 if Event2 < 2 else Event3 > 3'
+ after = (r'Select(Event(r"Event1") > Constant(1), Event(r"Event2") < '
+ r'Constant(2), Event(r"Event3") > Constant(3))')
+ self.assertEqual(ParsePerfJson(before).ToPython(), after)
+
+ before = r'min(a + b if c > 1 else c + d, e + f)'
+ after = r'min((a + b if c > 1 else c + d), e + f)'
+ self.assertEqual(ParsePerfJson(before).ToPerfJson(), after)
+
+ before =3D r'a if b else c if d else e'
+ after =3D r'(a if b else (c if d else e))'
+ self.assertEqual(ParsePerfJson(before).ToPerfJson(), after)
+
+ def test_ToPython(self):
+ # pylint: disable=eval-used
+ # Based on an example of a real metric.
+ before = '(a + b + c + d) / (2 * e)'
+ py = ParsePerfJson(before).ToPython()
+ after = eval(py).ToPerfJson()
+ self.assertEqual(before, after)
+
+ def test_Simplify(self):
+ before = '1 + 2 + 3'
+ after = '6'
+ self.assertEqual(ParsePerfJson(before).Simplify().ToPerfJson(), after)
+
+ before = 'a + 0'
+ after = 'a'
+ self.assertEqual(ParsePerfJson(before).Simplify().ToPerfJson(), after)
+
+ before = '0 + a'
+ after = 'a'
+ self.assertEqual(ParsePerfJson(before).Simplify().ToPerfJson(), after)
+
+ before = 'a | 0'
+ after = 'a'
+ self.assertEqual(ParsePerfJson(before).Simplify().ToPerfJson(), after)
+
+ before = '0 | a'
+ after = 'a'
+ self.assertEqual(ParsePerfJson(before).Simplify().ToPerfJson(), after)
+
+ before = 'a * 0'
+ after = '0'
+ self.assertEqual(ParsePerfJson(before).Simplify().ToPerfJson(), after)
+
+ before = '0 * a'
+ after = '0'
+ self.assertEqual(ParsePerfJson(before).Simplify().ToPerfJson(), after)
+
+ before = 'a * 1'
+ after = 'a'
+ self.assertEqual(ParsePerfJson(before).Simplify().ToPerfJson(), after)
+
+ before = '1 * a'
+ after = 'a'
+ self.assertEqual(ParsePerfJson(before).Simplify().ToPerfJson(), after)
+
+ before = 'a if 0 else b'
+ after = 'b'
+ self.assertEqual(ParsePerfJson(before).Simplify().ToPerfJson(), after)
+
+ before = 'a if 1 else b'
+ after = 'a'
+ self.assertEqual(ParsePerfJson(before).Simplify().ToPerfJson(), after)
+
+ before = 'a if b else a'
+ after = 'a'
+ self.assertEqual(ParsePerfJson(before).Simplify().ToPerfJson(), after)
+
+ # Pattern used to add a slots event to metrics that require it.
+ before = '0 * SLOTS'
+ after = '0 * SLOTS'
+ self.assertEqual(ParsePerfJson(before).Simplify().ToPerfJson(), after)
+
+if __name__ == '__main__':
+ unittest.main()
diff --git a/tools/perf/scripts/python/Perf-Trace-Util/Build b/tools/perf/scripts/python/Perf-Trace-Util/Build
index 7d0e33c..d5fed4e 100644
--- a/tools/perf/scripts/python/Perf-Trace-Util/Build
+++ b/tools/perf/scripts/python/Perf-Trace-Util/Build
@@ -1,3 +1,3 @@
-perf-y += Context.o
+perf-$(CONFIG_LIBTRACEEVENT) += Context.o
CFLAGS_Context.o += $(PYTHON_EMBED_CCOPTS) -Wno-redundant-decls -Wno-strict-prototypes -Wno-unused-parameter -Wno-nested-externs
diff --git a/tools/perf/scripts/python/bin/task-analyzer-record b/tools/perf/scripts/python/bin/task-analyzer-record
new file mode 100755
index 0000000..0f6b51b
--- /dev/null
+++ b/tools/perf/scripts/python/bin/task-analyzer-record
@@ -0,0 +1,2 @@
+#!/bin/bash
+perf record -e sched:sched_switch -e sched:sched_migrate_task "$@"
diff --git a/tools/perf/scripts/python/bin/task-analyzer-report b/tools/perf/scripts/python/bin/task-analyzer-report
new file mode 100755
index 0000000..4b16a8c
--- /dev/null
+++ b/tools/perf/scripts/python/bin/task-analyzer-report
@@ -0,0 +1,3 @@
+#!/bin/bash
+# description: analyze timings of tasks
+perf script -s "$PERF_EXEC_PATH"/scripts/python/task-analyzer.py -- "$@"
diff --git a/tools/perf/scripts/python/intel-pt-events.py b/tools/perf/scripts/python/intel-pt-events.py
index 6be7fd8..08862a2 100644
--- a/tools/perf/scripts/python/intel-pt-events.py
+++ b/tools/perf/scripts/python/intel-pt-events.py
@@ -13,10 +13,12 @@
from __future__ import print_function
+import io
import os
import sys
import struct
import argparse
+import contextlib
from libxed import LibXED
from ctypes import create_string_buffer, addressof
@@ -39,6 +41,11 @@
glb_source_file_name = None
glb_line_number = None
glb_dso = None
+glb_stash_dict = {}
+glb_output = None
+glb_output_pos = 0
+glb_cpu = -1
+glb_time = 0
def get_optional_null(perf_dict, field):
if field in perf_dict:
@@ -70,6 +77,7 @@
ap.add_argument("--insn-trace", action='store_true')
ap.add_argument("--src-trace", action='store_true')
ap.add_argument("--all-switch-events", action='store_true')
+ ap.add_argument("--interleave", type=int, nargs='?', const=4, default=0)
global glb_args
global glb_insn
global glb_src
@@ -94,11 +102,39 @@
perf_set_itrace_options(perf_script_context, itrace)
def trace_end():
+ if glb_args.interleave:
+ flush_stashed_output()
print("End")
def trace_unhandled(event_name, context, event_fields_dict):
print(' '.join(['%s=%s'%(k,str(v))for k,v in sorted(event_fields_dict.items())]))
+def stash_output():
+ global glb_stash_dict
+ global glb_output_pos
+ output_str = glb_output.getvalue()[glb_output_pos:]
+ n = len(output_str)
+ if n:
+ glb_output_pos += n
+ if glb_cpu not in glb_stash_dict:
+ glb_stash_dict[glb_cpu] = []
+ glb_stash_dict[glb_cpu].append(output_str)
+
+def flush_stashed_output():
+ global glb_stash_dict
+ while glb_stash_dict:
+ cpus = list(glb_stash_dict.keys())
+ # Output at most glb_args.interleave output strings per cpu
+ for cpu in cpus:
+ items = glb_stash_dict[cpu]
+ countdown = glb_args.interleave
+ while len(items) and countdown:
+ sys.stdout.write(items[0])
+ del items[0]
+ countdown -= 1
+ if not items:
+ del glb_stash_dict[cpu]
+
def print_ptwrite(raw_buf):
data = struct.unpack_from("<IQ", raw_buf)
flags = data[0]
@@ -375,15 +411,40 @@
print_common_start(comm, sample, name)
print_common_ip(param_dict, sample, symbol, dso)
+def interleave_events(param_dict):
+ global glb_cpu
+ global glb_time
+ global glb_output
+ global glb_output_pos
+
+ sample = param_dict["sample"]
+ glb_cpu = sample["cpu"]
+ ts = sample["time"]
+
+ if glb_time != ts:
+ glb_time = ts
+ flush_stashed_output()
+
+ glb_output_pos = 0
+ with contextlib.redirect_stdout(io.StringIO()) as glb_output:
+ do_process_event(param_dict)
+
+ stash_output()
+
def process_event(param_dict):
try:
- do_process_event(param_dict)
+ if glb_args.interleave:
+ interleave_events(param_dict)
+ else:
+ do_process_event(param_dict)
except broken_pipe_exception:
# Stop python printing broken pipe errors and traceback
sys.stdout = open(os.devnull, 'w')
sys.exit(1)
def auxtrace_error(typ, code, cpu, pid, tid, ip, ts, msg, cpumode, *x):
+ if glb_args.interleave:
+ flush_stashed_output()
if len(x) >= 2 and x[0]:
machine_pid = x[0]
vcpu = x[1]
@@ -403,6 +464,8 @@
sys.exit(1)
def context_switch(ts, cpu, pid, tid, np_pid, np_tid, machine_pid, out, out_preempt, *x):
+ if glb_args.interleave:
+ flush_stashed_output()
if out:
out_str = "Switch out "
else:
diff --git a/tools/perf/scripts/python/task-analyzer.py b/tools/perf/scripts/python/task-analyzer.py
new file mode 100755
index 0000000..52e8dae
--- /dev/null
+++ b/tools/perf/scripts/python/task-analyzer.py
@@ -0,0 +1,934 @@
+# task-analyzer.py - comprehensive perf tasks analysis
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (c) 2022, Hagen Paul Pfeifer <hagen@jauu.net>
+# Licensed under the terms of the GNU GPL License version 2
+#
+# Usage:
+#
+# perf record -e sched:sched_switch -a -- sleep 10
+# perf script report task-analyzer
+#
+
+from __future__ import print_function
+import sys
+import os
+import string
+import argparse
+import decimal
+
+
+sys.path.append(
+ os.environ["PERF_EXEC_PATH"] + "/scripts/python/Perf-Trace-Util/lib/Perf/Trace"
+)
+from perf_trace_context import *
+from Core import *
+
+# Definition of possible ASCII color codes
+_COLORS = {
+ "grey": "\033[90m",
+ "red": "\033[91m",
+ "green": "\033[92m",
+ "yellow": "\033[93m",
+ "blue": "\033[94m",
+ "violet": "\033[95m",
+ "reset": "\033[0m",
+}
+
+# Columns will have a static size to align everything properly
+# Support of 116 days of active update with nano precision
+LEN_SWITCHED_IN = len("9999999.999999999") # 17
+LEN_SWITCHED_OUT = len("9999999.999999999") # 17
+LEN_CPU = len("000")
+LEN_PID = len("maxvalue") # 8
+LEN_TID = len("maxvalue") # 8
+LEN_COMM = len("max-comms-length") # 16
+LEN_RUNTIME = len("999999.999") # 10
+# Support of 3.45 hours of timespans
+LEN_OUT_IN = len("99999999999.999") # 15
+LEN_OUT_OUT = len("99999999999.999") # 15
+LEN_IN_IN = len("99999999999.999") # 15
+LEN_IN_OUT = len("99999999999.999") # 15
+
+
+# py2/py3 compatibility layer, see PEP469
+try:
+ dict.iteritems
+except AttributeError:
+ # py3
+ def itervalues(d):
+ return iter(d.values())
+
+ def iteritems(d):
+ return iter(d.items())
+
+else:
+ # py2
+ def itervalues(d):
+ return d.itervalues()
+
+ def iteritems(d):
+ return d.iteritems()
+
+
+def _check_color():
+ global _COLORS
+ """user enforced no-color or if stdout is no tty we disable colors"""
+ if sys.stdout.isatty() and args.stdio_color != "never":
+ return
+ _COLORS = {
+ "grey": "",
+ "red": "",
+ "green": "",
+ "yellow": "",
+ "blue": "",
+ "violet": "",
+ "reset": "",
+ }
+
+
+def _parse_args():
+ global args
+ parser = argparse.ArgumentParser(description="Analyze tasks behavior")
+ parser.add_argument(
+ "--time-limit",
+ default=[],
+ help=
+ "print tasks only in time[s] window e.g"
+ " --time-limit 123.111:789.222(print all between 123.111 and 789.222)"
+ " --time-limit 123: (print all from 123)"
+ " --time-limit :456 (print all until incl. 456)",
+ )
+ parser.add_argument(
+ "--summary", action="store_true", help="print addtional runtime information"
+ )
+ parser.add_argument(
+ "--summary-only", action="store_true", help="print only summary without traces"
+ )
+ parser.add_argument(
+ "--summary-extended",
+ action="store_true",
+ help="print the summary with additional information of max inter task times"
+ " relative to the prev task",
+ )
+ parser.add_argument(
+ "--ns", action="store_true", help="show timestamps in nanoseconds"
+ )
+ parser.add_argument(
+ "--ms", action="store_true", help="show timestamps in miliseconds"
+ )
+ parser.add_argument(
+ "--extended-times",
+ action="store_true",
+ help="Show the elapsed times between schedule in/schedule out"
+ " of this task and the schedule in/schedule out of previous occurrence"
+ " of the same task",
+ )
+ parser.add_argument(
+ "--filter-tasks",
+ default=[],
+ help="filter out unneeded tasks by tid, pid or processname."
+ " E.g --filter-task 1337,/sbin/init ",
+ )
+ parser.add_argument(
+ "--limit-to-tasks",
+ default=[],
+ help="limit output to selected task by tid, pid, processname."
+ " E.g --limit-to-tasks 1337,/sbin/init",
+ )
+ parser.add_argument(
+ "--highlight-tasks",
+ default="",
+ help="colorize special tasks by their pid/tid/comm."
+ " E.g. --highlight-tasks 1:red,mutt:yellow"
+ " Colors available: red,grey,yellow,blue,violet,green",
+ )
+ parser.add_argument(
+ "--rename-comms-by-tids",
+ default="",
+ help="rename task names by using tid (<tid>:<newname>,<tid>:<newname>)"
+ " This option is handy for inexpressive processnames like python interpreted"
+ " process. E.g --rename 1337:my-python-app",
+ )
+ parser.add_argument(
+ "--stdio-color",
+ default="auto",
+ choices=["always", "never", "auto"],
+ help="always, never or auto, allowing configuring color output"
+ " via the command line",
+ )
+ parser.add_argument(
+ "--csv",
+ default="",
+ help="Write trace to file selected by user. Options, like --ns or --extended"
+ "-times are used.",
+ )
+ parser.add_argument(
+ "--csv-summary",
+ default="",
+ help="Write summary to file selected by user. Options, like --ns or"
+ " --summary-extended are used.",
+ )
+ args = parser.parse_args()
+ args.tid_renames = dict()
+
+ _argument_filter_sanity_check()
+ _argument_prepare_check()
+
+
+def time_uniter(unit):
+ picker = {
+ "s": 1,
+ "ms": 1e3,
+ "us": 1e6,
+ "ns": 1e9,
+ }
+ return picker[unit]
+
+
+def _init_db():
+ global db
+ db = dict()
+ db["running"] = dict()
+ db["cpu"] = dict()
+ db["tid"] = dict()
+ db["global"] = []
+ if args.summary or args.summary_extended or args.summary_only:
+ db["task_info"] = dict()
+ db["runtime_info"] = dict()
+ # min values for summary depending on the header
+ db["task_info"]["pid"] = len("PID")
+ db["task_info"]["tid"] = len("TID")
+ db["task_info"]["comm"] = len("Comm")
+ db["runtime_info"]["runs"] = len("Runs")
+ db["runtime_info"]["acc"] = len("Accumulated")
+ db["runtime_info"]["max"] = len("Max")
+ db["runtime_info"]["max_at"] = len("Max At")
+ db["runtime_info"]["min"] = len("Min")
+ db["runtime_info"]["mean"] = len("Mean")
+ db["runtime_info"]["median"] = len("Median")
+ if args.summary_extended:
+ db["inter_times"] = dict()
+ db["inter_times"]["out_in"] = len("Out-In")
+ db["inter_times"]["inter_at"] = len("At")
+ db["inter_times"]["out_out"] = len("Out-Out")
+ db["inter_times"]["in_in"] = len("In-In")
+ db["inter_times"]["in_out"] = len("In-Out")
+
+
+def _median(numbers):
+ """phython3 hat statistics module - we have nothing"""
+ n = len(numbers)
+ index = n // 2
+ if n % 2:
+ return sorted(numbers)[index]
+ return sum(sorted(numbers)[index - 1 : index + 1]) / 2
+
+
+def _mean(numbers):
+ return sum(numbers) / len(numbers)
+
+
+class Timespans(object):
+ """
+ The elapsed time between two occurrences of the same task is being tracked with the
+ help of this class. There are 4 of those Timespans Out-Out, In-Out, Out-In and
+ In-In.
+ The first half of the name signals the first time point of the
+ first task. The second half of the name represents the second
+ timepoint of the second task.
+ """
+
+ def __init__(self):
+ self._last_start = None
+ self._last_finish = None
+ self.out_out = -1
+ self.in_out = -1
+ self.out_in = -1
+ self.in_in = -1
+ if args.summary_extended:
+ self._time_in = -1
+ self.max_out_in = -1
+ self.max_at = -1
+ self.max_in_out = -1
+ self.max_in_in = -1
+ self.max_out_out = -1
+
+ def feed(self, task):
+ """
+ Called for every recorded trace event to find process pair and calculate the
+ task timespans. Chronological ordering, feed does not do reordering
+ """
+ if not self._last_finish:
+ self._last_start = task.time_in(time_unit)
+ self._last_finish = task.time_out(time_unit)
+ return
+ self._time_in = task.time_in()
+ time_in = task.time_in(time_unit)
+ time_out = task.time_out(time_unit)
+ self.in_in = time_in - self._last_start
+ self.out_in = time_in - self._last_finish
+ self.in_out = time_out - self._last_start
+ self.out_out = time_out - self._last_finish
+ if args.summary_extended:
+ self._update_max_entries()
+ self._last_finish = task.time_out(time_unit)
+ self._last_start = task.time_in(time_unit)
+
+ def _update_max_entries(self):
+ if self.in_in > self.max_in_in:
+ self.max_in_in = self.in_in
+ if self.out_out > self.max_out_out:
+ self.max_out_out = self.out_out
+ if self.in_out > self.max_in_out:
+ self.max_in_out = self.in_out
+ if self.out_in > self.max_out_in:
+ self.max_out_in = self.out_in
+ self.max_at = self._time_in
+
+
+
+class Summary(object):
+ """
+ Primary instance for calculating the summary output. Processes the whole trace to
+ find and memorize relevant data such as mean, max et cetera. This instance handles
+ dynamic alignment aspects for summary output.
+ """
+
+ def __init__(self):
+ self._body = []
+
+ class AlignmentHelper:
+ """
+ Used to calculated the alignment for the output of the summary.
+ """
+ def __init__(self, pid, tid, comm, runs, acc, mean,
+ median, min, max, max_at):
+ self.pid = pid
+ self.tid = tid
+ self.comm = comm
+ self.runs = runs
+ self.acc = acc
+ self.mean = mean
+ self.median = median
+ self.min = min
+ self.max = max
+ self.max_at = max_at
+ if args.summary_extended:
+ self.out_in = None
+ self.inter_at = None
+ self.out_out = None
+ self.in_in = None
+ self.in_out = None
+
+ def _print_header(self):
+ '''
+ Output is trimmed in _format_stats thus additional adjustment in the header
+ is needed, depending on the choice of timeunit. The adjustment corresponds
+ to the amount of column titles being adjusted in _column_titles.
+ '''
+ decimal_precision = 6 if not args.ns else 9
+ fmt = " {{:^{}}}".format(sum(db["task_info"].values()))
+ fmt += " {{:^{}}}".format(
+ sum(db["runtime_info"].values()) - 2 * decimal_precision
+ )
+ _header = ("Task Information", "Runtime Information")
+
+ if args.summary_extended:
+ fmt += " {{:^{}}}".format(
+ sum(db["inter_times"].values()) - 4 * decimal_precision
+ )
+ _header += ("Max Inter Task Times",)
+ fd_sum.write(fmt.format(*_header) + "\n")
+
+ def _column_titles(self):
+ """
+ Cells are being processed and displayed in different way so an alignment adjust
+ is implemented depeding on the choice of the timeunit. The positions of the max
+ values are being displayed in grey. Thus in their format two additional {},
+ are placed for color set and reset.
+ """
+ separator, fix_csv_align = _prepare_fmt_sep()
+ decimal_precision, time_precision = _prepare_fmt_precision()
+ fmt = "{{:>{}}}".format(db["task_info"]["pid"] * fix_csv_align)
+ fmt += "{}{{:>{}}}".format(separator, db["task_info"]["tid"] * fix_csv_align)
+ fmt += "{}{{:>{}}}".format(separator, db["task_info"]["comm"] * fix_csv_align)
+ fmt += "{}{{:>{}}}".format(separator, db["runtime_info"]["runs"] * fix_csv_align)
+ fmt += "{}{{:>{}}}".format(separator, db["runtime_info"]["acc"] * fix_csv_align)
+ fmt += "{}{{:>{}}}".format(separator, db["runtime_info"]["mean"] * fix_csv_align)
+ fmt += "{}{{:>{}}}".format(
+ separator, db["runtime_info"]["median"] * fix_csv_align
+ )
+ fmt += "{}{{:>{}}}".format(
+ separator, (db["runtime_info"]["min"] - decimal_precision) * fix_csv_align
+ )
+ fmt += "{}{{:>{}}}".format(
+ separator, (db["runtime_info"]["max"] - decimal_precision) * fix_csv_align
+ )
+ fmt += "{}{{}}{{:>{}}}{{}}".format(
+ separator, (db["runtime_info"]["max_at"] - time_precision) * fix_csv_align
+ )
+
+ column_titles = ("PID", "TID", "Comm")
+ column_titles += ("Runs", "Accumulated", "Mean", "Median", "Min", "Max")
+ column_titles += (_COLORS["grey"], "Max At", _COLORS["reset"])
+
+ if args.summary_extended:
+ fmt += "{}{{:>{}}}".format(
+ separator,
+ (db["inter_times"]["out_in"] - decimal_precision) * fix_csv_align
+ )
+ fmt += "{}{{}}{{:>{}}}{{}}".format(
+ separator,
+ (db["inter_times"]["inter_at"] - time_precision) * fix_csv_align
+ )
+ fmt += "{}{{:>{}}}".format(
+ separator,
+ (db["inter_times"]["out_out"] - decimal_precision) * fix_csv_align
+ )
+ fmt += "{}{{:>{}}}".format(
+ separator,
+ (db["inter_times"]["in_in"] - decimal_precision) * fix_csv_align
+ )
+ fmt += "{}{{:>{}}}".format(
+ separator,
+ (db["inter_times"]["in_out"] - decimal_precision) * fix_csv_align
+ )
+
+ column_titles += ("Out-In", _COLORS["grey"], "Max At", _COLORS["reset"],
+ "Out-Out", "In-In", "In-Out")
+
+ fd_sum.write(fmt.format(*column_titles) + "\n")
+
+
+ def _task_stats(self):
+ """calculates the stats of every task and constructs the printable summary"""
+ for tid in sorted(db["tid"]):
+ color_one_sample = _COLORS["grey"]
+ color_reset = _COLORS["reset"]
+ no_executed = 0
+ runtimes = []
+ time_in = []
+ timespans = Timespans()
+ for task in db["tid"][tid]:
+ pid = task.pid
+ comm = task.comm
+ no_executed += 1
+ runtimes.append(task.runtime(time_unit))
+ time_in.append(task.time_in())
+ timespans.feed(task)
+ if len(runtimes) > 1:
+ color_one_sample = ""
+ color_reset = ""
+ time_max = max(runtimes)
+ time_min = min(runtimes)
+ max_at = time_in[runtimes.index(max(runtimes))]
+
+ # The size of the decimal after sum,mean and median varies, thus we cut
+ # the decimal number, by rounding it. It has no impact on the output,
+ # because we have a precision of the decimal points at the output.
+ time_sum = round(sum(runtimes), 3)
+ time_mean = round(_mean(runtimes), 3)
+ time_median = round(_median(runtimes), 3)
+
+ align_helper = self.AlignmentHelper(pid, tid, comm, no_executed, time_sum,
+ time_mean, time_median, time_min, time_max, max_at)
+ self._body.append([pid, tid, comm, no_executed, time_sum, color_one_sample,
+ time_mean, time_median, time_min, time_max,
+ _COLORS["grey"], max_at, _COLORS["reset"], color_reset])
+ if args.summary_extended:
+ self._body[-1].extend([timespans.max_out_in,
+ _COLORS["grey"], timespans.max_at,
+ _COLORS["reset"], timespans.max_out_out,
+ timespans.max_in_in,
+ timespans.max_in_out])
+ align_helper.out_in = timespans.max_out_in
+ align_helper.inter_at = timespans.max_at
+ align_helper.out_out = timespans.max_out_out
+ align_helper.in_in = timespans.max_in_in
+ align_helper.in_out = timespans.max_in_out
+ self._calc_alignments_summary(align_helper)
+
+ def _format_stats(self):
+ separator, fix_csv_align = _prepare_fmt_sep()
+ decimal_precision, time_precision = _prepare_fmt_precision()
+ len_pid = db["task_info"]["pid"] * fix_csv_align
+ len_tid = db["task_info"]["tid"] * fix_csv_align
+ len_comm = db["task_info"]["comm"] * fix_csv_align
+ len_runs = db["runtime_info"]["runs"] * fix_csv_align
+ len_acc = db["runtime_info"]["acc"] * fix_csv_align
+ len_mean = db["runtime_info"]["mean"] * fix_csv_align
+ len_median = db["runtime_info"]["median"] * fix_csv_align
+ len_min = (db["runtime_info"]["min"] - decimal_precision) * fix_csv_align
+ len_max = (db["runtime_info"]["max"] - decimal_precision) * fix_csv_align
+ len_max_at = (db["runtime_info"]["max_at"] - time_precision) * fix_csv_align
+ if args.summary_extended:
+ len_out_in = (
+ db["inter_times"]["out_in"] - decimal_precision
+ ) * fix_csv_align
+ len_inter_at = (
+ db["inter_times"]["inter_at"] - time_precision
+ ) * fix_csv_align
+ len_out_out = (
+ db["inter_times"]["out_out"] - decimal_precision
+ ) * fix_csv_align
+ len_in_in = (db["inter_times"]["in_in"] - decimal_precision) * fix_csv_align
+ len_in_out = (
+ db["inter_times"]["in_out"] - decimal_precision
+ ) * fix_csv_align
+
+ fmt = "{{:{}d}}".format(len_pid)
+ fmt += "{}{{:{}d}}".format(separator, len_tid)
+ fmt += "{}{{:>{}}}".format(separator, len_comm)
+ fmt += "{}{{:{}d}}".format(separator, len_runs)
+ fmt += "{}{{:{}.{}f}}".format(separator, len_acc, time_precision)
+ fmt += "{}{{}}{{:{}.{}f}}".format(separator, len_mean, time_precision)
+ fmt += "{}{{:{}.{}f}}".format(separator, len_median, time_precision)
+ fmt += "{}{{:{}.{}f}}".format(separator, len_min, time_precision)
+ fmt += "{}{{:{}.{}f}}".format(separator, len_max, time_precision)
+ fmt += "{}{{}}{{:{}.{}f}}{{}}{{}}".format(
+ separator, len_max_at, decimal_precision
+ )
+ if args.summary_extended:
+ fmt += "{}{{:{}.{}f}}".format(separator, len_out_in, time_precision)
+ fmt += "{}{{}}{{:{}.{}f}}{{}}".format(
+ separator, len_inter_at, decimal_precision
+ )
+ fmt += "{}{{:{}.{}f}}".format(separator, len_out_out, time_precision)
+ fmt += "{}{{:{}.{}f}}".format(separator, len_in_in, time_precision)
+ fmt += "{}{{:{}.{}f}}".format(separator, len_in_out, time_precision)
+ return fmt
+
+
+ def _calc_alignments_summary(self, align_helper):
+ # Length is being cut in 3 groups so that further addition is easier to handle.
+ # The length of every argument from the alignment helper is being checked if it
+ # is longer than the longest until now. In that case the length is being saved.
+ for key in db["task_info"]:
+ if len(str(getattr(align_helper, key))) > db["task_info"][key]:
+ db["task_info"][key] = len(str(getattr(align_helper, key)))
+ for key in db["runtime_info"]:
+ if len(str(getattr(align_helper, key))) > db["runtime_info"][key]:
+ db["runtime_info"][key] = len(str(getattr(align_helper, key)))
+ if args.summary_extended:
+ for key in db["inter_times"]:
+ if len(str(getattr(align_helper, key))) > db["inter_times"][key]:
+ db["inter_times"][key] = len(str(getattr(align_helper, key)))
+
+
+ def print(self):
+ self._task_stats()
+ fmt = self._format_stats()
+
+ if not args.csv_summary:
+ print("\nSummary")
+ self._print_header()
+ self._column_titles()
+ for i in range(len(self._body)):
+ fd_sum.write(fmt.format(*tuple(self._body[i])) + "\n")
+
+
+
+class Task(object):
+ """ The class is used to handle the information of a given task."""
+
+ def __init__(self, id, tid, cpu, comm):
+ self.id = id
+ self.tid = tid
+ self.cpu = cpu
+ self.comm = comm
+ self.pid = None
+ self._time_in = None
+ self._time_out = None
+
+ def schedule_in_at(self, time):
+ """set the time where the task was scheduled in"""
+ self._time_in = time
+
+ def schedule_out_at(self, time):
+ """set the time where the task was scheduled out"""
+ self._time_out = time
+
+ def time_out(self, unit="s"):
+ """return time where a given task was scheduled out"""
+ factor = time_uniter(unit)
+ return self._time_out * decimal.Decimal(factor)
+
+ def time_in(self, unit="s"):
+ """return time where a given task was scheduled in"""
+ factor = time_uniter(unit)
+ return self._time_in * decimal.Decimal(factor)
+
+ def runtime(self, unit="us"):
+ factor = time_uniter(unit)
+ return (self._time_out - self._time_in) * decimal.Decimal(factor)
+
+ def update_pid(self, pid):
+ self.pid = pid
+
+
+def _task_id(pid, cpu):
+ """returns a "unique-enough" identifier, please do not change"""
+ return "{}-{}".format(pid, cpu)
+
+
+def _filter_non_printable(unfiltered):
+ """comm names may contain loony chars like '\x00000'"""
+ filtered = ""
+ for char in unfiltered:
+ if char not in string.printable:
+ continue
+ filtered += char
+ return filtered
+
+
+def _fmt_header():
+ separator, fix_csv_align = _prepare_fmt_sep()
+ fmt = "{{:>{}}}".format(LEN_SWITCHED_IN*fix_csv_align)
+ fmt += "{}{{:>{}}}".format(separator, LEN_SWITCHED_OUT*fix_csv_align)
+ fmt += "{}{{:>{}}}".format(separator, LEN_CPU*fix_csv_align)
+ fmt += "{}{{:>{}}}".format(separator, LEN_PID*fix_csv_align)
+ fmt += "{}{{:>{}}}".format(separator, LEN_TID*fix_csv_align)
+ fmt += "{}{{:>{}}}".format(separator, LEN_COMM*fix_csv_align)
+ fmt += "{}{{:>{}}}".format(separator, LEN_RUNTIME*fix_csv_align)
+ fmt += "{}{{:>{}}}".format(separator, LEN_OUT_IN*fix_csv_align)
+ if args.extended_times:
+ fmt += "{}{{:>{}}}".format(separator, LEN_OUT_OUT*fix_csv_align)
+ fmt += "{}{{:>{}}}".format(separator, LEN_IN_IN*fix_csv_align)
+ fmt += "{}{{:>{}}}".format(separator, LEN_IN_OUT*fix_csv_align)
+ return fmt
+
+
+def _fmt_body():
+ separator, fix_csv_align = _prepare_fmt_sep()
+ decimal_precision, time_precision = _prepare_fmt_precision()
+ fmt = "{{}}{{:{}.{}f}}".format(LEN_SWITCHED_IN*fix_csv_align, decimal_precision)
+ fmt += "{}{{:{}.{}f}}".format(
+ separator, LEN_SWITCHED_OUT*fix_csv_align, decimal_precision
+ )
+ fmt += "{}{{:{}d}}".format(separator, LEN_CPU*fix_csv_align)
+ fmt += "{}{{:{}d}}".format(separator, LEN_PID*fix_csv_align)
+ fmt += "{}{{}}{{:{}d}}{{}}".format(separator, LEN_TID*fix_csv_align)
+ fmt += "{}{{}}{{:>{}}}".format(separator, LEN_COMM*fix_csv_align)
+ fmt += "{}{{:{}.{}f}}".format(separator, LEN_RUNTIME*fix_csv_align, time_precision)
+ if args.extended_times:
+ fmt += "{}{{:{}.{}f}}".format(separator, LEN_OUT_IN*fix_csv_align, time_precision)
+ fmt += "{}{{:{}.{}f}}".format(separator, LEN_OUT_OUT*fix_csv_align, time_precision)
+ fmt += "{}{{:{}.{}f}}".format(separator, LEN_IN_IN*fix_csv_align, time_precision)
+ fmt += "{}{{:{}.{}f}}{{}}".format(
+ separator, LEN_IN_OUT*fix_csv_align, time_precision
+ )
+ else:
+ fmt += "{}{{:{}.{}f}}{{}}".format(
+ separator, LEN_OUT_IN*fix_csv_align, time_precision
+ )
+ return fmt
+
+
+def _print_header():
+ fmt = _fmt_header()
+ header = ("Switched-In", "Switched-Out", "CPU", "PID", "TID", "Comm", "Runtime",
+ "Time Out-In")
+ if args.extended_times:
+ header += ("Time Out-Out", "Time In-In", "Time In-Out")
+ fd_task.write(fmt.format(*header) + "\n")
+
+
+
+def _print_task_finish(task):
+ """calculating every entry of a row and printing it immediately"""
+ c_row_set = ""
+ c_row_reset = ""
+ out_in = -1
+ out_out = -1
+ in_in = -1
+ in_out = -1
+ fmt = _fmt_body()
+ # depending on user provided highlight option we change the color
+ # for particular tasks
+ if str(task.tid) in args.highlight_tasks_map:
+ c_row_set = _COLORS[args.highlight_tasks_map[str(task.tid)]]
+ c_row_reset = _COLORS["reset"]
+ if task.comm in args.highlight_tasks_map:
+ c_row_set = _COLORS[args.highlight_tasks_map[task.comm]]
+ c_row_reset = _COLORS["reset"]
+ # grey-out entries if PID == TID, they
+ # are identical, no threaded model so the
+ # thread id (tid) do not matter
+ c_tid_set = ""
+ c_tid_reset = ""
+ if task.pid == task.tid:
+ c_tid_set = _COLORS["grey"]
+ c_tid_reset = _COLORS["reset"]
+ if task.tid in db["tid"]:
+ # get last task of tid
+ last_tid_task = db["tid"][task.tid][-1]
+ # feed the timespan calculate, last in tid db
+ # and second the current one
+ timespan_gap_tid = Timespans()
+ timespan_gap_tid.feed(last_tid_task)
+ timespan_gap_tid.feed(task)
+ out_in = timespan_gap_tid.out_in
+ out_out = timespan_gap_tid.out_out
+ in_in = timespan_gap_tid.in_in
+ in_out = timespan_gap_tid.in_out
+
+
+ if args.extended_times:
+ line_out = fmt.format(c_row_set, task.time_in(), task.time_out(), task.cpu,
+ task.pid, c_tid_set, task.tid, c_tid_reset, c_row_set, task.comm,
+ task.runtime(time_unit), out_in, out_out, in_in, in_out,
+ c_row_reset) + "\n"
+ else:
+ line_out = fmt.format(c_row_set, task.time_in(), task.time_out(), task.cpu,
+ task.pid, c_tid_set, task.tid, c_tid_reset, c_row_set, task.comm,
+ task.runtime(time_unit), out_in, c_row_reset) + "\n"
+ try:
+ fd_task.write(line_out)
+ except(IOError):
+ # don't mangle the output if user SIGINT this script
+ sys.exit()
+
+def _record_cleanup(_list):
+ """
+ no need to store more then one element if --summarize
+ is not enabled
+ """
+ if not args.summary and len(_list) > 1:
+ _list = _list[len(_list) - 1 :]
+
+
+def _record_by_tid(task):
+ tid = task.tid
+ if tid not in db["tid"]:
+ db["tid"][tid] = []
+ db["tid"][tid].append(task)
+ _record_cleanup(db["tid"][tid])
+
+
+def _record_by_cpu(task):
+ cpu = task.cpu
+ if cpu not in db["cpu"]:
+ db["cpu"][cpu] = []
+ db["cpu"][cpu].append(task)
+ _record_cleanup(db["cpu"][cpu])
+
+
+def _record_global(task):
+ """record all executed task, ordered by finish chronological"""
+ db["global"].append(task)
+ _record_cleanup(db["global"])
+
+
+def _handle_task_finish(tid, cpu, time, perf_sample_dict):
+ if tid == 0:
+ return
+ _id = _task_id(tid, cpu)
+ if _id not in db["running"]:
+ # may happen, if we missed the switch to
+ # event. Seen in combination with --exclude-perf
+ # where the start is filtered out, but not the
+ # switched in. Probably a bug in exclude-perf
+ # option.
+ return
+ task = db["running"][_id]
+ task.schedule_out_at(time)
+
+ # record tid, during schedule in the tid
+ # is not available, update now
+ pid = int(perf_sample_dict["sample"]["pid"])
+
+ task.update_pid(pid)
+ del db["running"][_id]
+
+ # print only tasks which are not being filtered and no print of trace
+ # for summary only, but record every task.
+ if not _limit_filtered(tid, pid, task.comm) and not args.summary_only:
+ _print_task_finish(task)
+ _record_by_tid(task)
+ _record_by_cpu(task)
+ _record_global(task)
+
+
+def _handle_task_start(tid, cpu, comm, time):
+ if tid == 0:
+ return
+ if tid in args.tid_renames:
+ comm = args.tid_renames[tid]
+ _id = _task_id(tid, cpu)
+ if _id in db["running"]:
+ # handle corner cases where already running tasks
+ # are switched-to again - saw this via --exclude-perf
+ # recorded traces. We simple ignore this "second start"
+ # event.
+ return
+ assert _id not in db["running"]
+ task = Task(_id, tid, cpu, comm)
+ task.schedule_in_at(time)
+ db["running"][_id] = task
+
+
+def _time_to_internal(time_ns):
+ """
+ To prevent float rounding errors we use Decimal internally
+ """
+ return decimal.Decimal(time_ns) / decimal.Decimal(1e9)
+
+
+def _limit_filtered(tid, pid, comm):
+ if args.filter_tasks:
+ if str(tid) in args.filter_tasks or comm in args.filter_tasks:
+ return True
+ else:
+ return False
+ if args.limit_to_tasks:
+ if str(tid) in args.limit_to_tasks or comm in args.limit_to_tasks:
+ return False
+ else:
+ return True
+
+
+def _argument_filter_sanity_check():
+ if args.limit_to_tasks and args.filter_tasks:
+ sys.exit("Error: Filter and Limit at the same time active.")
+ if args.extended_times and args.summary_only:
+ sys.exit("Error: Summary only and extended times active.")
+ if args.time_limit and ":" not in args.time_limit:
+ sys.exit(
+ "Error: No bound set for time limit. Please set bound by ':' e.g :123."
+ )
+ if args.time_limit and (args.summary or args.summary_only or args.summary_extended):
+ sys.exit("Error: Cannot set time limit and print summary")
+ if args.csv_summary:
+ args.summary = True
+ if args.csv == args.csv_summary:
+ sys.exit("Error: Chosen files for csv and csv summary are the same")
+ if args.csv and (args.summary_extended or args.summary) and not args.csv_summary:
+ sys.exit("Error: No file chosen to write summary to. Choose with --csv-summary "
+ "<file>")
+ if args.csv and args.summary_only:
+ sys.exit("Error: --csv chosen and --summary-only. Standard task would not be"
+ "written to csv file.")
+
+def _argument_prepare_check():
+ global time_unit, fd_task, fd_sum
+ if args.filter_tasks:
+ args.filter_tasks = args.filter_tasks.split(",")
+ if args.limit_to_tasks:
+ args.limit_to_tasks = args.limit_to_tasks.split(",")
+ if args.time_limit:
+ args.time_limit = args.time_limit.split(":")
+ for rename_tuple in args.rename_comms_by_tids.split(","):
+ tid_name = rename_tuple.split(":")
+ if len(tid_name) != 2:
+ continue
+ args.tid_renames[int(tid_name[0])] = tid_name[1]
+ args.highlight_tasks_map = dict()
+ for highlight_tasks_tuple in args.highlight_tasks.split(","):
+ tasks_color_map = highlight_tasks_tuple.split(":")
+ # default highlight color to red if no color set by user
+ if len(tasks_color_map) == 1:
+ tasks_color_map.append("red")
+ if args.highlight_tasks and tasks_color_map[1].lower() not in _COLORS:
+ sys.exit(
+ "Error: Color not defined, please choose from grey,red,green,yellow,blue,"
+ "violet"
+ )
+ if len(tasks_color_map) != 2:
+ continue
+ args.highlight_tasks_map[tasks_color_map[0]] = tasks_color_map[1]
+ time_unit = "us"
+ if args.ns:
+ time_unit = "ns"
+ elif args.ms:
+ time_unit = "ms"
+
+
+ fd_task = sys.stdout
+ if args.csv:
+ args.stdio_color = "never"
+ fd_task = open(args.csv, "w")
+ print("generating csv at",args.csv,)
+
+ fd_sum = sys.stdout
+ if args.csv_summary:
+ args.stdio_color = "never"
+ fd_sum = open(args.csv_summary, "w")
+ print("generating csv summary at",args.csv_summary)
+ if not args.csv:
+ args.summary_only = True
+
+
+def _is_within_timelimit(time):
+ """
+ Check if a time limit was given by parameter, if so ignore the rest. If not,
+ process the recorded trace in its entirety.
+ """
+ if not args.time_limit:
+ return True
+ lower_time_limit = args.time_limit[0]
+ upper_time_limit = args.time_limit[1]
+ # check for upper limit
+ if upper_time_limit == "":
+ if time >= decimal.Decimal(lower_time_limit):
+ return True
+ # check for lower limit
+ if lower_time_limit == "":
+ if time <= decimal.Decimal(upper_time_limit):
+ return True
+ # quit if time exceeds upper limit. Good for big datasets
+ else:
+ quit()
+ if lower_time_limit != "" and upper_time_limit != "":
+ if (time >= decimal.Decimal(lower_time_limit) and
+ time <= decimal.Decimal(upper_time_limit)):
+ return True
+ # quit if time exceeds upper limit. Good for big datasets
+ elif time > decimal.Decimal(upper_time_limit):
+ quit()
+
+def _prepare_fmt_precision():
+ decimal_precision = 6
+ time_precision = 3
+ if args.ns:
+ decimal_precision = 9
+ time_precision = 0
+ return decimal_precision, time_precision
+
+def _prepare_fmt_sep():
+ separator = " "
+ fix_csv_align = 1
+ if args.csv or args.csv_summary:
+ separator = ";"
+ fix_csv_align = 0
+ return separator, fix_csv_align
+
+def trace_unhandled(event_name, context, event_fields_dict, perf_sample_dict):
+ pass
+
+
+def trace_begin():
+ _parse_args()
+ _check_color()
+ _init_db()
+ if not args.summary_only:
+ _print_header()
+
+def trace_end():
+ if args.summary or args.summary_extended or args.summary_only:
+ Summary().print()
+
+def sched__sched_switch(event_name, context, common_cpu, common_secs, common_nsecs,
+ common_pid, common_comm, common_callchain, prev_comm,
+ prev_pid, prev_prio, prev_state, next_comm, next_pid,
+ next_prio, perf_sample_dict):
+ # ignore common_secs & common_nsecs cause we need
+ # high res timestamp anyway, using the raw value is
+ # faster
+ time = _time_to_internal(perf_sample_dict["sample"]["time"])
+ if not _is_within_timelimit(time):
+ # user specific --time-limit a:b set
+ return
+
+ next_comm = _filter_non_printable(next_comm)
+ _handle_task_finish(prev_pid, common_cpu, time, perf_sample_dict)
+ _handle_task_start(next_pid, common_cpu, next_comm, time)
diff --git a/tools/perf/tests/Build b/tools/perf/tests/Build
index 2064a64..90fd1eb 100644
--- a/tools/perf/tests/Build
+++ b/tools/perf/tests/Build
@@ -6,13 +6,13 @@
perf-y += dso-data.o
perf-y += attr.o
perf-y += vmlinux-kallsyms.o
-perf-y += openat-syscall.o
-perf-y += openat-syscall-all-cpus.o
-perf-y += openat-syscall-tp-fields.o
-perf-y += mmap-basic.o
+perf-$(CONFIG_LIBTRACEEVENT) += openat-syscall.o
+perf-$(CONFIG_LIBTRACEEVENT) += openat-syscall-all-cpus.o
+perf-$(CONFIG_LIBTRACEEVENT) += openat-syscall-tp-fields.o
+perf-$(CONFIG_LIBTRACEEVENT) += mmap-basic.o
perf-y += perf-record.o
perf-y += evsel-roundtrip-name.o
-perf-y += evsel-tp-sched.o
+perf-$(CONFIG_LIBTRACEEVENT) += evsel-tp-sched.o
perf-y += fdarray.o
perf-y += pmu.o
perf-y += pmu-events.o
@@ -30,7 +30,7 @@
perf-y += sw-clock.o
perf-y += mmap-thread-lookup.o
perf-y += thread-maps-share.o
-perf-y += switch-tracking.o
+perf-$(CONFIG_LIBTRACEEVENT) += switch-tracking.o
perf-y += keep-tracking.o
perf-y += code-reading.o
perf-y += sample-parsing.o
@@ -67,6 +67,7 @@
perf-y += perf-time-to-tsc.o
perf-y += dlfilter-test.o
perf-y += sigtrap.o
+perf-y += event_groups.o
$(OUTPUT)tests/llvm-src-base.c: tests/bpf-script-example.c tests/Build
$(call rule_mkdir)
@@ -103,3 +104,5 @@
CFLAGS_attr.o += -DBINDIR="BUILD_STR($(bindir_SQ))" -DPYTHON="BUILD_STR($(PYTHON_WORD))"
CFLAGS_python-use.o += -DPYTHONPATH="BUILD_STR($(OUTPUT)python)" -DPYTHON="BUILD_STR($(PYTHON_WORD))"
CFLAGS_dwarf-unwind.o += -fno-optimize-sibling-calls
+
+perf-y += workloads/
diff --git a/tools/perf/tests/attr.py b/tools/perf/tests/attr.py
index cb39ac4..ccfef86 100644
--- a/tools/perf/tests/attr.py
+++ b/tools/perf/tests/attr.py
@@ -6,9 +6,12 @@
import sys
import glob
import optparse
+import platform
import tempfile
import logging
+import re
import shutil
+import subprocess
try:
import configparser
@@ -123,17 +126,27 @@
if not data_equal(self[t], other[t]):
log.warning("expected %s=%s, got %s" % (t, self[t], other[t]))
+def parse_version(version):
+ if not version:
+ return None
+ return [int(v) for v in version.split(".")[0:2]]
+
# Test file description needs to have following sections:
# [config]
# - just single instance in file
# - needs to specify:
# 'command' - perf command name
# 'args' - special command arguments
-# 'ret' - expected command return value (0 by default)
+# 'ret' - Skip test if Perf doesn't exit with this value (0 by default)
+# 'test_ret'- If set to 'true', fail test instead of skipping for 'ret' argument
# 'arch' - architecture specific test (optional)
# comma separated list, ! at the beginning
# negates it.
-#
+# 'auxv' - Truthy statement that is evaled in the scope of the auxv map. When false,
+# the test is skipped. For example 'auxv["AT_HWCAP"] == 10'. (optional)
+# 'kernel_since' - Inclusive kernel version from which the test will start running. Only the
+# first two values are supported, for example "6.1" (optional)
+# 'kernel_until' - Exclusive kernel version from which the test will stop running. (optional)
# [eventX:base]
# - one or multiple instances in file
# - expected values assignments
@@ -155,12 +168,17 @@
except:
self.ret = 0
+ self.test_ret = parser.getboolean('config', 'test_ret', fallback=False)
+
try:
self.arch = parser.get('config', 'arch')
log.warning("test limitation '%s'" % self.arch)
except:
self.arch = ''
+ self.auxv = parser.get('config', 'auxv', fallback=None)
+ self.kernel_since = parse_version(parser.get('config', 'kernel_since', fallback=None))
+ self.kernel_until = parse_version(parser.get('config', 'kernel_until', fallback=None))
self.expect = {}
self.result = {}
log.debug(" loading expected events");
@@ -172,7 +190,38 @@
else:
return True
- def skip_test(self, myarch):
+ def skip_test_kernel_since(self):
+ if not self.kernel_since:
+ return False
+ return not self.kernel_since <= parse_version(platform.release())
+
+ def skip_test_kernel_until(self):
+ if not self.kernel_until:
+ return False
+ return not parse_version(platform.release()) < self.kernel_until
+
+ def skip_test_auxv(self):
+ def new_auxv(a, pattern):
+ items = list(filter(None, pattern.split(a)))
+ # AT_HWCAP is hex but doesn't have a prefix, so special case it
+ if items[0] == "AT_HWCAP":
+ value = int(items[-1], 16)
+ else:
+ try:
+ value = int(items[-1], 0)
+ except:
+ value = items[-1]
+ return (items[0], value)
+
+ if not self.auxv:
+ return False
+ auxv = subprocess.check_output("LD_SHOW_AUXV=1 sleep 0", shell=True) \
+ .decode(sys.stdout.encoding)
+ pattern = re.compile(r"[: ]+")
+ auxv = dict([new_auxv(a, pattern) for a in auxv.splitlines()])
+ return not eval(self.auxv)
+
+ def skip_test_arch(self, myarch):
# If architecture not set always run test
if self.arch == '':
# log.warning("test for arch %s is ok" % myarch)
@@ -222,9 +271,18 @@
def run_cmd(self, tempdir):
junk1, junk2, junk3, junk4, myarch = (os.uname())
- if self.skip_test(myarch):
+ if self.skip_test_arch(myarch):
raise Notest(self, myarch)
+ if self.skip_test_auxv():
+ raise Notest(self, "auxv skip")
+
+ if self.skip_test_kernel_since():
+ raise Notest(self, "old kernel skip")
+
+ if self.skip_test_kernel_until():
+ raise Notest(self, "new kernel skip")
+
cmd = "PERF_TEST_ATTR=%s %s %s -o %s/perf.data %s" % (tempdir,
self.perf, self.command, tempdir, self.args)
ret = os.WEXITSTATUS(os.system(cmd))
@@ -232,7 +290,10 @@
log.info(" '%s' ret '%s', expected '%s'" % (cmd, str(ret), str(self.ret)))
if not data_equal(str(ret), str(self.ret)):
- raise Unsup(self)
+ if self.test_ret:
+ raise Fail(self, "Perf exit code failure")
+ else:
+ raise Unsup(self)
def compare(self, expect, result):
match = {}
diff --git a/tools/perf/tests/attr/README b/tools/perf/tests/attr/README
index eb3f7d4..4066fec 100644
--- a/tools/perf/tests/attr/README
+++ b/tools/perf/tests/attr/README
@@ -49,7 +49,6 @@
perf record --call-graph dwarf kill (test-record-graph-dwarf)
perf record --call-graph fp kill (test-record-graph-fp)
perf record --call-graph fp kill (test-record-graph-fp-aarch64)
- perf record --group -e cycles,instructions kill (test-record-group)
perf record -e '{cycles,instructions}' kill (test-record-group1)
perf record -e '{cycles/period=1/,instructions/period=2/}:S' kill (test-record-group2)
perf record -D kill (test-record-no-delay)
@@ -66,6 +65,5 @@
perf stat -d kill (test-stat-detailed-1)
perf stat -dd kill (test-stat-detailed-2)
perf stat -ddd kill (test-stat-detailed-3)
- perf stat --group -e cycles,instructions kill (test-stat-group)
perf stat -e '{cycles,instructions}' kill (test-stat-group1)
perf stat -i -e cycles kill (test-stat-no-inherit)
diff --git a/tools/perf/tests/attr/test-record-group b/tools/perf/tests/attr/test-record-group
deleted file mode 100644
index 6c1cff8..0000000
--- a/tools/perf/tests/attr/test-record-group
+++ /dev/null
@@ -1,22 +0,0 @@
-[config]
-command = record
-args = --no-bpf-event --group -e cycles,instructions kill >/dev/null 2>&1
-ret = 1
-
-[event-1:base-record]
-fd=1
-group_fd=-1
-sample_type=327
-read_format=4|20
-
-[event-2:base-record]
-fd=2
-group_fd=1
-config=1
-sample_type=327
-read_format=4|20
-mmap=0
-comm=0
-task=0
-enable_on_exec=0
-disabled=0
diff --git a/tools/perf/tests/attr/test-record-user-regs-no-sve-aarch64 b/tools/perf/tests/attr/test-record-user-regs-no-sve-aarch64
new file mode 100644
index 0000000..fbb0658
--- /dev/null
+++ b/tools/perf/tests/attr/test-record-user-regs-no-sve-aarch64
@@ -0,0 +1,9 @@
+# Test that asking for VG fails if the system doesn't support SVE. This
+# applies both before and after the feature was added in 6.1
+[config]
+command = record
+args = --no-bpf-event --user-regs=vg kill >/dev/null 2>&1
+ret = 129
+test_ret = true
+arch = aarch64
+auxv = auxv["AT_HWCAP"] & 0x200000 == 0
diff --git a/tools/perf/tests/attr/test-record-user-regs-old-sve-aarch64 b/tools/perf/tests/attr/test-record-user-regs-old-sve-aarch64
new file mode 100644
index 0000000..15ebfc3
--- /dev/null
+++ b/tools/perf/tests/attr/test-record-user-regs-old-sve-aarch64
@@ -0,0 +1,10 @@
+# Test that asking for VG always fails on old kernels because it was
+# added in 6.1. This applies to systems that either support or don't
+# support SVE.
+[config]
+command = record
+args = --no-bpf-event --user-regs=vg kill >/dev/null 2>&1
+ret = 129
+test_ret = true
+arch = aarch64
+kernel_until = 6.1
diff --git a/tools/perf/tests/attr/test-record-user-regs-sve-aarch64 b/tools/perf/tests/attr/test-record-user-regs-sve-aarch64
new file mode 100644
index 0000000..c598c80
--- /dev/null
+++ b/tools/perf/tests/attr/test-record-user-regs-sve-aarch64
@@ -0,0 +1,14 @@
+# Test that asking for VG works if the system has SVE and after the
+# feature was added in 6.1
+[config]
+command = record
+args = --no-bpf-event --user-regs=vg kill >/dev/null 2>&1
+ret = 1
+test_ret = true
+arch = aarch64
+auxv = auxv["AT_HWCAP"] & 0x200000 == 0x200000
+kernel_since = 6.1
+
+[event:base-record]
+sample_type=4359
+sample_regs_user=70368744177664
diff --git a/tools/perf/tests/attr/test-stat-group b/tools/perf/tests/attr/test-stat-group
deleted file mode 100644
index e15d694..0000000
--- a/tools/perf/tests/attr/test-stat-group
+++ /dev/null
@@ -1,17 +0,0 @@
-[config]
-command = stat
-args = --group -e cycles,instructions kill >/dev/null 2>&1
-ret = 1
-
-[event-1:base-stat]
-fd=1
-group_fd=-1
-read_format=3|15
-
-[event-2:base-stat]
-fd=2
-group_fd=1
-config=1
-disabled=0
-enable_on_exec=0
-read_format=3|15
diff --git a/tools/perf/tests/builtin-test.c b/tools/perf/tests/builtin-test.c
index 7122eae..f6c16ad 100644
--- a/tools/perf/tests/builtin-test.c
+++ b/tools/perf/tests/builtin-test.c
@@ -38,9 +38,11 @@ struct test_suite *__weak arch_tests[] = {
static struct test_suite *generic_tests[] = {
&suite__vmlinux_matches_kallsyms,
+#ifdef HAVE_LIBTRACEEVENT
&suite__openat_syscall_event,
&suite__openat_syscall_event_on_all_cpus,
&suite__basic_mmap,
+#endif
&suite__mem,
&suite__parse_events,
&suite__expr,
@@ -51,8 +53,10 @@ static struct test_suite *generic_tests[] = {
&suite__dso_data_cache,
&suite__dso_data_reopen,
&suite__perf_evsel__roundtrip_name_test,
+#ifdef HAVE_LIBTRACEEVENT
&suite__perf_evsel__tp_sched_test,
&suite__syscall_openat_tp_fields,
+#endif
&suite__attr,
&suite__hists_link,
&suite__python_use,
@@ -71,7 +75,9 @@ static struct test_suite *generic_tests[] = {
&suite__thread_maps_share,
&suite__hists_output,
&suite__hists_cumulate,
+#ifdef HAVE_LIBTRACEEVENT
&suite__switch_tracking,
+#endif
&suite__fdarray__filter,
&suite__fdarray__add,
&suite__kmod_path__parse,
@@ -110,6 +116,7 @@ static struct test_suite *generic_tests[] = {
&suite__perf_time_to_tsc,
&suite__dlfilter,
&suite__sigtrap,
+ &suite__event_groups,
NULL,
};
@@ -118,6 +125,15 @@ static struct test_suite **tests[] = {
arch_tests,
};
+static struct test_workload *workloads[] = {
+ &workload__noploop,
+ &workload__thloop,
+ &workload__leafloop,
+ &workload__sqrtloop,
+ &workload__brstack,
+ &workload__datasym,
+};
+
static int num_subtests(const struct test_suite *t)
{
int num;
@@ -475,6 +491,21 @@ static int perf_test__list(int argc, const char **argv)
return 0;
}
+static int run_workload(const char *work, int argc, const char **argv)
+{
+ unsigned int i = 0;
+ struct test_workload *twl;
+
+ for (i = 0; i < ARRAY_SIZE(workloads); i++) {
+ twl = workloads[i];
+ if (!strcmp(twl->name, work))
+ return twl->func(argc, argv);
+ }
+
+ pr_info("No workload found: %s\n", work);
+ return -1;
+}
+
int cmd_test(int argc, const char **argv)
{
const char *test_usage[] = {
@@ -482,12 +513,14 @@ int cmd_test(int argc, const char **argv)
NULL,
};
const char *skip = NULL;
+ const char *workload = NULL;
const struct option test_options[] = {
OPT_STRING('s', "skip", &skip, "tests", "tests to skip"),
OPT_INCR('v', "verbose", &verbose,
"be more verbose (show symbol address, etc)"),
OPT_BOOLEAN('F', "dont-fork", &dont_fork,
"Do not fork for testcase"),
+ OPT_STRING('w', "workload", &workload, "work", "workload to run for testing"),
OPT_END()
};
const char * const test_subcommands[] = { "list", NULL };
@@ -504,6 +537,9 @@ int cmd_test(int argc, const char **argv)
if (argc >= 1 && !strcmp(argv[0], "list"))
return perf_test__list(argc - 1, argv + 1);
+ if (workload)
+ return run_workload(workload, argc, argv);
+
symbol_conf.priv_size = sizeof(int);
symbol_conf.sort_by_name = true;
symbol_conf.try_vmlinux_path = true;
diff --git a/tools/perf/tests/code-reading.c b/tools/perf/tests/code-reading.c
index 95feb6ef..cb8cd09 100644
--- a/tools/perf/tests/code-reading.c
+++ b/tools/perf/tests/code-reading.c
@@ -16,7 +16,6 @@
#include "dso.h"
#include "env.h"
#include "parse-events.h"
-#include "trace-event.h"
#include "evlist.h"
#include "evsel.h"
#include "thread_map.h"
@@ -28,6 +27,7 @@
#include "util/mmap.h"
#include "util/string2.h"
#include "util/synthetic-events.h"
+#include "util/util.h"
#include "thread.h"
#include "tests.h"
@@ -79,7 +79,7 @@ static size_t read_objdump_chunk(const char **line, unsigned char **buf,
* see disassemble_bytes() at binutils/objdump.c for details
* how objdump chooses display endian)
*/
- if (bytes_read > 1 && !bigendian()) {
+ if (bytes_read > 1 && !host_is_bigendian()) {
unsigned char *chunk_end = chunk_start + bytes_read - 1;
unsigned char tmp;
diff --git a/tools/perf/tests/cpumap.c b/tools/perf/tests/cpumap.c
index 7c873c6..3150fc1 100644
--- a/tools/perf/tests/cpumap.c
+++ b/tools/perf/tests/cpumap.c
@@ -6,7 +6,7 @@
#include "util/synthetic-events.h"
#include <string.h>
#include <linux/bitops.h>
-#include <perf/cpumap.h>
+#include <internal/cpumap.h>
#include "debug.h"
struct machine;
diff --git a/tools/perf/tests/dlfilter-test.c b/tools/perf/tests/dlfilter-test.c
index 84352d5..99aa72e4 100644
--- a/tools/perf/tests/dlfilter-test.c
+++ b/tools/perf/tests/dlfilter-test.c
@@ -33,6 +33,7 @@
#include "archinsn.h"
#include "dlfilter.h"
#include "tests.h"
+#include "util/sample.h"
#define MAP_START 0x400000
diff --git a/tools/perf/tests/event_groups.c b/tools/perf/tests/event_groups.c
new file mode 100644
index 0000000..029442b
--- /dev/null
+++ b/tools/perf/tests/event_groups.c
@@ -0,0 +1,139 @@
+// SPDX-License-Identifier: GPL-2.0
+#include <string.h>
+#include <unistd.h>
+#include <stdio.h>
+#include "linux/perf_event.h"
+#include "tests.h"
+#include "debug.h"
+#include "pmu.h"
+#include "pmus.h"
+#include "header.h"
+#include "../perf-sys.h"
+
+/* hw: cycles, sw: context-switch, uncore: [arch dependent] */
+static int types[] = {0, 1, -1};
+static unsigned long configs[] = {0, 3, 0};
+
+#define NR_UNCORE_PMUS 5
+
+/* Uncore pmus that support more than 3 counters */
+static struct uncore_pmus {
+ const char *name;
+ __u64 config;
+} uncore_pmus[NR_UNCORE_PMUS] = {
+ { "amd_l3", 0x0 },
+ { "amd_df", 0x0 },
+ { "uncore_imc_0", 0x1 }, /* Intel */
+ { "core_imc", 0x318 }, /* PowerPC: core_imc/CPM_STCX_FIN/ */
+ { "hv_24x7", 0x22000000003 }, /* PowerPC: hv_24x7/CPM_STCX_FIN/ */
+};
+
+static int event_open(int type, unsigned long config, int group_fd)
+{
+ struct perf_event_attr attr;
+
+ memset(&attr, 0, sizeof(struct perf_event_attr));
+ attr.type = type;
+ attr.size = sizeof(struct perf_event_attr);
+ attr.config = config;
+ /*
+ * When creating an event group, typically the group leader is
+ * initialized with disabled set to 1 and any child events are
+ * initialized with disabled set to 0. Despite disabled being 0,
+ * the child events will not start until the group leader is
+ * enabled.
+ */
+ attr.disabled = group_fd == -1 ? 1 : 0;
+
+ return sys_perf_event_open(&attr, -1, 0, group_fd, 0);
+}
+
+static int setup_uncore_event(void)
+{
+ struct perf_pmu *pmu;
+ int i, fd;
+
+ if (list_empty(&pmus))
+ perf_pmu__scan(NULL);
+
+ perf_pmus__for_each_pmu(pmu) {
+ for (i = 0; i < NR_UNCORE_PMUS; i++) {
+ if (!strcmp(uncore_pmus[i].name, pmu->name)) {
+ pr_debug("Using %s for uncore pmu event\n", pmu->name);
+ types[2] = pmu->type;
+ configs[2] = uncore_pmus[i].config;
+ /*
+ * Check if the chosen uncore pmu event can be
+ * used in the test. For example, incase of accessing
+ * hv_24x7 pmu counters, partition should have
+ * additional permissions. If not, event open will
+ * fail. So check if the event open succeeds
+ * before proceeding.
+ */
+ fd = event_open(types[2], configs[2], -1);
+ if (fd < 0)
+ return -1;
+ close(fd);
+ return 0;
+ }
+ }
+ }
+ return -1;
+}
+
+static int run_test(int i, int j, int k)
+{
+ int erroneous = ((((1 << i) | (1 << j) | (1 << k)) & 5) == 5);
+ int group_fd, sibling_fd1, sibling_fd2;
+
+ group_fd = event_open(types[i], configs[i], -1);
+ if (group_fd == -1)
+ return -1;
+
+ sibling_fd1 = event_open(types[j], configs[j], group_fd);
+ if (sibling_fd1 == -1) {
+ close(group_fd);
+ return erroneous ? 0 : -1;
+ }
+
+ sibling_fd2 = event_open(types[k], configs[k], group_fd);
+ if (sibling_fd2 == -1) {
+ close(sibling_fd1);
+ close(group_fd);
+ return erroneous ? 0 : -1;
+ }
+
+ close(sibling_fd2);
+ close(sibling_fd1);
+ close(group_fd);
+ return erroneous ? -1 : 0;
+}
+
+static int test__event_groups(struct test_suite *text __maybe_unused, int subtest __maybe_unused)
+{
+ int i, j, k;
+ int ret;
+ int r;
+
+ ret = setup_uncore_event();
+ if (ret || types[2] == -1)
+ return TEST_SKIP;
+
+ ret = TEST_OK;
+ for (i = 0; i < 3; i++) {
+ for (j = 0; j < 3; j++) {
+ for (k = 0; k < 3; k++) {
+ r = run_test(i, j, k);
+ if (r)
+ ret = TEST_FAIL;
+
+ pr_debug("0x%x 0x%lx, 0x%x 0x%lx, 0x%x 0x%lx: %s\n",
+ types[i], configs[i], types[j], configs[j],
+ types[k], configs[k], r ? "Fail" : "Pass");
+ }
+ }
+ }
+ return ret;
+}
+
+DEFINE_SUITE("Event groups", event_groups);
diff --git a/tools/perf/tests/expr.c b/tools/perf/tests/expr.c
index c598f95a..a9eb1ed 100644
--- a/tools/perf/tests/expr.c
+++ b/tools/perf/tests/expr.c
@@ -2,6 +2,7 @@
#include "util/cputopo.h"
#include "util/debug.h"
#include "util/expr.h"
+#include "util/hashmap.h"
#include "util/header.h"
#include "util/smt.h"
#include "tests.h"
diff --git a/tools/perf/tests/make b/tools/perf/tests/make
index da013e9..05e818a 100644
--- a/tools/perf/tests/make
+++ b/tools/perf/tests/make
@@ -29,7 +29,7 @@
PARALLEL_OPT=
ifeq ($(SET_PARALLEL),1)
ifeq ($(JOBS),)
- cores := $(shell (getconf _NPROCESSORS_ONLN || egrep -c '^processor|^CPU[0-9]' /proc/cpuinfo) 2>/dev/null)
+ cores := $(shell (getconf _NPROCESSORS_ONLN || grep -E -c '^processor|^CPU[0-9]' /proc/cpuinfo) 2>/dev/null)
ifeq ($(cores),0)
cores := 1
endif
diff --git a/tools/perf/tests/mmap-basic.c b/tools/perf/tests/mmap-basic.c
index 8322fc2..e68ca62 100644
--- a/tools/perf/tests/mmap-basic.c
+++ b/tools/perf/tests/mmap-basic.c
@@ -5,11 +5,13 @@
#include <perf/cpumap.h>
#include "debug.h"
+#include "event.h"
#include "evlist.h"
#include "evsel.h"
#include "thread_map.h"
#include "tests.h"
#include "util/mmap.h"
+#include "util/sample.h"
#include <linux/err.h>
#include <linux/kernel.h>
#include <linux/string.h>
diff --git a/tools/perf/tests/openat-syscall-tp-fields.c b/tools/perf/tests/openat-syscall-tp-fields.c
index a7b2800..888df8e 100644
--- a/tools/perf/tests/openat-syscall-tp-fields.c
+++ b/tools/perf/tests/openat-syscall-tp-fields.c
@@ -14,6 +14,7 @@
#include "util/mmap.h"
#include <errno.h>
#include <perf/mmap.h>
+#include "util/sample.h"
#ifndef O_DIRECTORY
#define O_DIRECTORY 00200000
diff --git a/tools/perf/tests/openat-syscall.c b/tools/perf/tests/openat-syscall.c
index 7e05b8b..131b622 100644
--- a/tools/perf/tests/openat-syscall.c
+++ b/tools/perf/tests/openat-syscall.c
@@ -7,6 +7,7 @@
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
+#include <unistd.h>
#include "thread_map.h"
#include "evsel.h"
#include "debug.h"
diff --git a/tools/perf/tests/parse-events.c b/tools/perf/tests/parse-events.c
index 459afdb..71a5cb34 100644
--- a/tools/perf/tests/parse-events.c
+++ b/tools/perf/tests/parse-events.c
@@ -20,6 +20,8 @@
#define PERF_TP_SAMPLE_TYPE (PERF_SAMPLE_RAW | PERF_SAMPLE_TIME | \
PERF_SAMPLE_CPU | PERF_SAMPLE_PERIOD)
+#ifdef HAVE_LIBTRACEEVENT
+
#if defined(__s390x__)
/* Return true if kvm module is available and loaded. Test this
* and return success when trace point kvm_s390_create_vm
@@ -76,6 +78,7 @@ static int test__checkevent_tracepoint_multi(struct evlist *evlist)
}
return TEST_OK;
}
+#endif /* HAVE_LIBTRACEEVENT */
static int test__checkevent_raw(struct evlist *evlist)
{
@@ -222,6 +225,7 @@ static int test__checkevent_breakpoint_rw(struct evlist *evlist)
return TEST_OK;
}
+#ifdef HAVE_LIBTRACEEVENT
static int test__checkevent_tracepoint_modifier(struct evlist *evlist)
{
struct evsel *evsel = evlist__first(evlist);
@@ -252,6 +256,7 @@ test__checkevent_tracepoint_multi_modifier(struct evlist *evlist)
return test__checkevent_tracepoint_multi(evlist);
}
+#endif /* HAVE_LIBTRACEEVENT */
static int test__checkevent_raw_modifier(struct evlist *evlist)
{
@@ -453,6 +458,7 @@ static int test__checkevent_pmu(struct evlist *evlist)
return TEST_OK;
}
+#ifdef HAVE_LIBTRACEEVENT
static int test__checkevent_list(struct evlist *evlist)
{
struct evsel *evsel = evlist__first(evlist);
@@ -491,6 +497,7 @@ static int test__checkevent_list(struct evlist *evlist)
return TEST_OK;
}
+#endif
static int test__checkevent_pmu_name(struct evlist *evlist)
{
@@ -762,6 +769,7 @@ static int test__group2(struct evlist *evlist)
return TEST_OK;
}
+#ifdef HAVE_LIBTRACEEVENT
static int test__group3(struct evlist *evlist __maybe_unused)
{
struct evsel *evsel, *leader;
@@ -853,6 +861,7 @@ static int test__group3(struct evlist *evlist __maybe_unused)
return TEST_OK;
}
+#endif
static int test__group4(struct evlist *evlist __maybe_unused)
{
@@ -1460,6 +1469,7 @@ static int test__sym_event_dc(struct evlist *evlist)
return TEST_OK;
}
+#ifdef HAVE_LIBTRACEEVENT
static int count_tracepoints(void)
{
struct dirent *events_ent;
@@ -1513,6 +1523,7 @@ static int test__all_tracepoints(struct evlist *evlist)
return test__checkevent_tracepoint_multi(evlist);
}
+#endif /* HAVE_LIBTRACEVENT */
static int test__hybrid_hw_event_with_pmu(struct evlist *evlist)
{
@@ -1642,6 +1653,7 @@ struct evlist_test {
};
static const struct evlist_test test__events[] = {
+#ifdef HAVE_LIBTRACEEVENT
{
.name = "syscalls:sys_enter_openat",
.check = test__checkevent_tracepoint,
@@ -1652,6 +1664,7 @@ static const struct evlist_test test__events[] = {
.check = test__checkevent_tracepoint_multi,
/* 1 */
},
+#endif
{
.name = "r1a",
.check = test__checkevent_raw,
@@ -1702,6 +1715,7 @@ static const struct evlist_test test__events[] = {
.check = test__checkevent_breakpoint_w,
/* 1 */
},
+#ifdef HAVE_LIBTRACEEVENT
{
.name = "syscalls:sys_enter_openat:k",
.check = test__checkevent_tracepoint_modifier,
@@ -1712,6 +1726,7 @@ static const struct evlist_test test__events[] = {
.check = test__checkevent_tracepoint_multi_modifier,
/* 3 */
},
+#endif
{
.name = "r1a:kp",
.check = test__checkevent_raw_modifier,
@@ -1757,11 +1772,13 @@ static const struct evlist_test test__events[] = {
.check = test__checkevent_breakpoint_w_modifier,
/* 2 */
},
+#ifdef HAVE_LIBTRACEEVENT
{
.name = "r1,syscalls:sys_enter_openat:k,1:1:hp",
.check = test__checkevent_list,
/* 3 */
},
+#endif
{
.name = "instructions:G",
.check = test__checkevent_exclude_host_modifier,
@@ -1792,11 +1809,13 @@ static const struct evlist_test test__events[] = {
.check = test__group2,
/* 9 */
},
+#ifdef HAVE_LIBTRACEEVENT
{
.name = "group1{syscalls:sys_enter_openat:H,cycles:kppp},group2{cycles,1:3}:G,instructions:u",
.check = test__group3,
/* 0 */
},
+#endif
{
.name = "{cycles:u,instructions:kp}:p",
.check = test__group4,
@@ -1807,11 +1826,13 @@ static const struct evlist_test test__events[] = {
.check = test__group5,
/* 2 */
},
+#ifdef HAVE_LIBTRACEEVENT
{
.name = "*:*",
.check = test__all_tracepoints,
/* 3 */
},
+#endif
{
.name = "{cycles,cache-misses:G}:H",
.check = test__group_gh1,
@@ -1867,7 +1888,7 @@ static const struct evlist_test test__events[] = {
.check = test__checkevent_breakpoint_len_rw_modifier,
/* 4 */
},
-#if defined(__s390x__)
+#if defined(__s390x__) && defined(HAVE_LIBTRACEEVENT)
{
.name = "kvm-s390:kvm_s390_create_vm",
.check = test__checkevent_tracepoint,
@@ -2237,6 +2258,19 @@ static int test__pmu_events(struct test_suite *test __maybe_unused, int subtest
pr_debug("Test PMU event failed for '%s'", name);
ret = combine_test_results(ret, test_ret);
}
+ /*
+ * Names containing '-' are recognized as prefixes and suffixes
+ * due to '-' being a legacy PMU separator. This fails when the
+ * prefix or suffix collides with an existing legacy token. For
+ * example, branch-brs has a prefix (branch) that collides with
+ * a PE_NAME_CACHE_TYPE token causing a parse error as a suffix
+ * isn't expected after this. As event names in the config
+ * slashes are allowed a '-' in the name we check this works
+ * above.
+ */
+ if (strchr(ent->d_name, '-'))
+ continue;
+
snprintf(name, sizeof(name), "%s:u,cpu/event=%s/u", ent->d_name, ent->d_name);
e.name = name;
e.check = test__checkevent_pmu_events_mix;
diff --git a/tools/perf/tests/parse-metric.c b/tools/perf/tests/parse-metric.c
index 68f5a2a..21b7ac0 100644
--- a/tools/perf/tests/parse-metric.c
+++ b/tools/perf/tests/parse-metric.c
@@ -103,7 +103,7 @@ static int __compute_metric(const char *name, struct value *vals,
if (err)
goto out;
- err = evlist__alloc_stats(evlist, false);
+ err = evlist__alloc_stats(/*config=*/NULL, evlist, /*alloc_raw=*/false);
if (err)
goto out;
diff --git a/tools/perf/tests/parse-no-sample-id-all.c b/tools/perf/tests/parse-no-sample-id-all.c
index d62e315..202f0a9 100644
--- a/tools/perf/tests/parse-no-sample-id-all.c
+++ b/tools/perf/tests/parse-no-sample-id-all.c
@@ -8,6 +8,7 @@
#include "evlist.h"
#include "header.h"
#include "debug.h"
+#include "util/sample.h"
static int process_event(struct evlist **pevlist, union perf_event *event)
{
diff --git a/tools/perf/tests/perf-record.c b/tools/perf/tests/perf-record.c
index 7aa946a..1c4feec 100644
--- a/tools/perf/tests/perf-record.c
+++ b/tools/perf/tests/perf-record.c
@@ -5,12 +5,14 @@
#include <sched.h>
#include <perf/mmap.h>
+#include "event.h"
#include "evlist.h"
#include "evsel.h"
#include "debug.h"
#include "record.h"
#include "tests.h"
#include "util/mmap.h"
+#include "util/sample.h"
static int sched__get_first_possible_cpu(pid_t pid, cpu_set_t *maskp)
{
diff --git a/tools/perf/tests/perf-time-to-tsc.c b/tools/perf/tests/perf-time-to-tsc.c
index c3aaa1d..efcd71c 100644
--- a/tools/perf/tests/perf-time-to-tsc.c
+++ b/tools/perf/tests/perf-time-to-tsc.c
@@ -20,6 +20,7 @@
#include "tsc.h"
#include "mmap.h"
#include "tests.h"
+#include "util/sample.h"
/*
* Except x86_64/i386 and Arm64, other archs don't support TSC in perf. Just
diff --git a/tools/perf/tests/pmu-events.c b/tools/perf/tests/pmu-events.c
index 3c2ee55..e36d8b1 100644
--- a/tools/perf/tests/pmu-events.c
+++ b/tools/perf/tests/pmu-events.c
@@ -12,6 +12,7 @@
#include <perf/evlist.h>
#include "util/evlist.h"
#include "util/expr.h"
+#include "util/hashmap.h"
#include "util/parse-events.h"
#include "metricgroup.h"
#include "stat.h"
@@ -889,7 +890,7 @@ static int test__parsing_callback(const struct pmu_event *pe, const struct pmu_e
goto out_err;
}
- err = evlist__alloc_stats(evlist, false);
+ err = evlist__alloc_stats(/*config=*/NULL, evlist, /*alloc_raw=*/false);
if (err)
goto out_err;
/*
diff --git a/tools/perf/tests/sample-parsing.c b/tools/perf/tests/sample-parsing.c
index 20930dd..927c7f0 100644
--- a/tools/perf/tests/sample-parsing.c
+++ b/tools/perf/tests/sample-parsing.c
@@ -13,7 +13,7 @@
#include "evsel.h"
#include "debug.h"
#include "util/synthetic-events.h"
-#include "util/trace-event.h"
+#include "util/util.h"
#include "tests.h"
@@ -117,7 +117,7 @@ static bool samples_same(const struct perf_sample *s1,
COMP(branch_stack->hw_idx);
for (i = 0; i < s1->branch_stack->nr; i++) {
if (needs_swap)
- return ((tep_is_bigendian()) ?
+ return ((host_is_bigendian()) ?
(FLAG(s2).value == BS_EXPECTED_BE) :
(FLAG(s2).value == BS_EXPECTED_LE));
else
diff --git a/tools/perf/tests/shell/lib/probe_vfs_getname.sh b/tools/perf/tests/shell/lib/probe_vfs_getname.sh
index b616d42..ed0a397 100644
--- a/tools/perf/tests/shell/lib/probe_vfs_getname.sh
+++ b/tools/perf/tests/shell/lib/probe_vfs_getname.sh
@@ -12,13 +12,13 @@
add_probe_vfs_getname() {
local verbose=$1
if [ $had_vfs_getname -eq 1 ] ; then
- line=$(perf probe -L getname_flags 2>&1 | egrep 'result.*=.*filename;' | sed -r 's/[[:space:]]+([[:digit:]]+)[[:space:]]+result->uptr.*/\1/')
+ line=$(perf probe -L getname_flags 2>&1 | grep -E 'result.*=.*filename;' | sed -r 's/[[:space:]]+([[:digit:]]+)[[:space:]]+result->uptr.*/\1/')
perf probe -q "vfs_getname=getname_flags:${line} pathname=result->name:string" || \
perf probe $verbose "vfs_getname=getname_flags:${line} pathname=filename:ustring"
fi
}
skip_if_no_debuginfo() {
- add_probe_vfs_getname -v 2>&1 | egrep -q "^(Failed to find the path for the kernel|Debuginfo-analysis is not supported)|(file has no debug information)" && return 2
+ add_probe_vfs_getname -v 2>&1 | grep -E -q "^(Failed to find the path for the kernel|Debuginfo-analysis is not supported)|(file has no debug information)" && return 2
return 1
}
diff --git a/tools/perf/tests/shell/lock_contention.sh b/tools/perf/tests/shell/lock_contention.sh
index 04bf604..cc9ceb9 100755
--- a/tools/perf/tests/shell/lock_contention.sh
+++ b/tools/perf/tests/shell/lock_contention.sh
@@ -53,7 +53,7 @@
if ! perf lock con -b true > /dev/null 2>&1 ; then
echo "[Skip] No BPF support"
- exit
+ return
fi
# the perf lock contention output goes to the stderr
@@ -65,9 +65,70 @@
fi
}
+test_record_concurrent()
+{
+ echo "Testing perf lock record and perf lock contention at the same time"
+ perf lock record -o- -- perf bench sched messaging 2> /dev/null | \
+ perf lock contention -i- -E 1 -q 2> ${result}
+ if [ $(cat "${result}" | wc -l) != "1" ]; then
+ echo "[Fail] Recorded result count is not 1:" $(cat "${result}" | wc -l)
+ err=1
+ exit
+ fi
+}
+
+test_aggr_task()
+{
+ echo "Testing perf lock contention --threads"
+ perf lock contention -i ${perfdata} -t -E 1 -q 2> ${result}
+ if [ $(cat "${result}" | wc -l) != "1" ]; then
+ echo "[Fail] Recorded result count is not 1:" $(cat "${result}" | wc -l)
+ err=1
+ exit
+ fi
+
+ if ! perf lock con -b true > /dev/null 2>&1 ; then
+ return
+ fi
+
+ # the perf lock contention output goes to the stderr
+ perf lock con -a -b -t -E 1 -q -- perf bench sched messaging > /dev/null 2> ${result}
+ if [ $(cat "${result}" | wc -l) != "1" ]; then
+ echo "[Fail] BPF result count is not 1:" $(cat "${result}" | wc -l)
+ err=1
+ exit
+ fi
+}
+
+test_aggr_addr()
+{
+ echo "Testing perf lock contention --lock-addr"
+ perf lock contention -i ${perfdata} -l -E 1 -q 2> ${result}
+ if [ $(cat "${result}" | wc -l) != "1" ]; then
+ echo "[Fail] Recorded result count is not 1:" $(cat "${result}" | wc -l)
+ err=1
+ exit
+ fi
+
+ if ! perf lock con -b true > /dev/null 2>&1 ; then
+ return
+ fi
+
+ # the perf lock contention output goes to the stderr
+ perf lock con -a -b -t -E 1 -q -- perf bench sched messaging > /dev/null 2> ${result}
+ if [ $(cat "${result}" | wc -l) != "1" ]; then
+ echo "[Fail] BPF result count is not 1:" $(cat "${result}" | wc -l)
+ err=1
+ exit
+ fi
+}
+
check
test_record
test_bpf
+test_record_concurrent
+test_aggr_task
+test_aggr_addr
exit ${err}
diff --git a/tools/perf/tests/shell/pipe_test.sh b/tools/perf/tests/shell/pipe_test.sh
index 1b32b4f..8dd115d 100755
--- a/tools/perf/tests/shell/pipe_test.sh
+++ b/tools/perf/tests/shell/pipe_test.sh
@@ -2,68 +2,33 @@
# perf pipe recording and injection test
# SPDX-License-Identifier: GPL-2.0
-# skip if there's no compiler
-if ! [ -x "$(command -v cc)" ]; then
- echo "failed: no compiler, install gcc"
- exit 2
-fi
-
-file=$(mktemp /tmp/test.file.XXXXXX)
data=$(mktemp /tmp/perf.data.XXXXXX)
+prog="perf test -w noploop"
+task="perf"
+sym="noploop"
-cat <<EOF | cc -o ${file} -x c -
-#include <signal.h>
-#include <stdlib.h>
-#include <unistd.h>
-
-volatile int done;
-
-void sigalrm(int sig) {
- done = 1;
-}
-
-__attribute__((noinline)) void noploop(void) {
- while (!done)
- continue;
-}
-
-int main(int argc, char *argv[]) {
- int sec = 1;
-
- if (argc > 1)
- sec = atoi(argv[1]);
-
- signal(SIGALRM, sigalrm);
- alarm(sec);
-
- noploop();
- return 0;
-}
-EOF
-
-
-if ! perf record -e task-clock:u -o - ${file} | perf report -i - --task | grep test.file; then
+if ! perf record -e task-clock:u -o - ${prog} | perf report -i - --task | grep ${task}; then
echo "cannot find the test file in the perf report"
exit 1
fi
-if ! perf record -e task-clock:u -o - ${file} | perf inject -b | perf report -i - | grep noploop; then
+if ! perf record -e task-clock:u -o - ${prog} | perf inject -b | perf report -i - | grep ${sym}; then
echo "cannot find noploop function in pipe #1"
exit 1
fi
-perf record -e task-clock:u -o - ${file} | perf inject -b -o ${data}
-if ! perf report -i ${data} | grep noploop; then
+perf record -e task-clock:u -o - ${prog} | perf inject -b -o ${data}
+if ! perf report -i ${data} | grep ${sym}; then
echo "cannot find noploop function in pipe #2"
exit 1
fi
-perf record -e task-clock:u -o ${data} ${file}
-if ! perf inject -b -i ${data} | perf report -i - | grep noploop; then
+perf record -e task-clock:u -o ${data} ${prog}
+if ! perf inject -b -i ${data} | perf report -i - | grep ${sym}; then
echo "cannot find noploop function in pipe #3"
exit 1
fi
-rm -f ${file} ${data} ${data}.old
+rm -f ${data} ${data}.old
exit 0
diff --git a/tools/perf/tests/shell/record+probe_libc_inet_pton.sh b/tools/perf/tests/shell/record+probe_libc_inet_pton.sh
index f12a4e2..34c400c 100755
--- a/tools/perf/tests/shell/record+probe_libc_inet_pton.sh
+++ b/tools/perf/tests/shell/record+probe_libc_inet_pton.sh
@@ -64,7 +64,7 @@
while read line <&3 && read -r pattern <&4; do
[ -z "$pattern" ] && break
echo $line
- echo "$line" | egrep -q "$pattern"
+ echo "$line" | grep -E -q "$pattern"
if [ $? -ne 0 ] ; then
printf "FAIL: expected backtrace entry \"%s\" got \"%s\"\n" "$pattern" "$line"
return 1
diff --git a/tools/perf/tests/shell/record+script_probe_vfs_getname.sh b/tools/perf/tests/shell/record+script_probe_vfs_getname.sh
index 8d9c04e..7f83b27 100755
--- a/tools/perf/tests/shell/record+script_probe_vfs_getname.sh
+++ b/tools/perf/tests/shell/record+script_probe_vfs_getname.sh
@@ -23,7 +23,7 @@
perf_script_filenames() {
echo "Looking at perf.data file for vfs_getname records for the file we touched:"
perf script -i ${perfdata} | \
- egrep " +touch +[0-9]+ +\[[0-9]+\] +[0-9]+\.[0-9]+: +probe:vfs_getname[_0-9]*: +\([[:xdigit:]]+\) +pathname=\"${file}\""
+ grep -E " +touch +[0-9]+ +\[[0-9]+\] +[0-9]+\.[0-9]+: +probe:vfs_getname[_0-9]*: +\([[:xdigit:]]+\) +pathname=\"${file}\""
}
add_probe_vfs_getname || skip_if_no_debuginfo
diff --git a/tools/perf/tests/shell/record.sh b/tools/perf/tests/shell/record.sh
index 301f954..4fbc748 100755
--- a/tools/perf/tests/shell/record.sh
+++ b/tools/perf/tests/shell/record.sh
@@ -4,67 +4,89 @@
set -e
+shelldir=$(dirname "$0")
+. "${shelldir}"/lib/waiting.sh
+
err=0
perfdata=$(mktemp /tmp/__perf_test.perf.data.XXXXX)
+testprog="perf test -w thloop"
+testsym="test_loop"
cleanup() {
- rm -f ${perfdata}
- rm -f ${perfdata}.old
- trap - exit term int
+ rm -rf "${perfdata}"
+ rm -rf "${perfdata}".old
+
+ trap - EXIT TERM INT
}
trap_cleanup() {
cleanup
exit 1
}
-trap trap_cleanup exit term int
+trap trap_cleanup EXIT TERM INT
test_per_thread() {
echo "Basic --per-thread mode test"
- if ! perf record -e instructions:u -o ${perfdata} --quiet true 2> /dev/null
+ if ! perf record -o /dev/null --quiet ${testprog} 2> /dev/null
then
- echo "Per-thread record [Skipped instructions:u not supported]"
- if [ $err -ne 1 ]
- then
- err=2
- fi
+ echo "Per-thread record [Skipped event not supported]"
return
fi
- if ! perf record -e instructions:u --per-thread -o ${perfdata} true 2> /dev/null
+ if ! perf record --per-thread -o "${perfdata}" ${testprog} 2> /dev/null
then
- echo "Per-thread record of instructions:u [Failed]"
+ echo "Per-thread record [Failed record]"
err=1
return
fi
- if ! perf report -i ${perfdata} -q | egrep -q true
+ if ! perf report -i "${perfdata}" -q | grep -q "${testsym}"
then
echo "Per-thread record [Failed missing output]"
err=1
return
fi
+
+ # run the test program in background (for 30 seconds)
+ ${testprog} 30 &
+ TESTPID=$!
+
+ rm -f "${perfdata}"
+
+ wait_for_threads ${TESTPID} 2
+ perf record -p "${TESTPID}" --per-thread -o "${perfdata}" sleep 1 2> /dev/null
+ kill ${TESTPID}
+
+ if [ ! -e "${perfdata}" ]
+ then
+ echo "Per-thread record [Failed record -p]"
+ err=1
+ return
+ fi
+ if ! perf report -i "${perfdata}" -q | grep -q "${testsym}"
+ then
+ echo "Per-thread record [Failed -p missing output]"
+ err=1
+ return
+ fi
+
echo "Basic --per-thread mode test [Success]"
}
test_register_capture() {
echo "Register capture test"
- if ! perf list | egrep -q 'br_inst_retired.near_call'
+ if ! perf list | grep -q 'br_inst_retired.near_call'
then
- echo "Register capture test [Skipped missing instruction]"
- if [ $err -ne 1 ]
- then
- err=2
- fi
+ echo "Register capture test [Skipped missing event]"
return
fi
- if ! perf record --intr-regs=\? 2>&1 | egrep -q 'available registers: AX BX CX DX SI DI BP SP IP FLAGS CS SS R8 R9 R10 R11 R12 R13 R14 R15'
+ if ! perf record --intr-regs=\? 2>&1 | grep -q 'available registers: AX BX CX DX SI DI BP SP IP FLAGS CS SS R8 R9 R10 R11 R12 R13 R14 R15'
then
echo "Register capture test [Skipped missing registers]"
return
fi
- if ! perf record -o - --intr-regs=di,r8,dx,cx -e br_inst_retired.near_call:p \
- -c 1000 --per-thread true 2> /dev/null \
+ if ! perf record -o - --intr-regs=di,r8,dx,cx -e br_inst_retired.near_call \
+ -c 1000 --per-thread ${testprog} 2> /dev/null \
| perf script -F ip,sym,iregs -i - 2> /dev/null \
- | egrep -q "DI:"
+ | grep -q "DI:"
then
echo "Register capture test [Failed missing output]"
err=1
@@ -73,8 +95,69 @@
echo "Register capture test [Success]"
}
+test_system_wide() {
+ echo "Basic --system-wide mode test"
+ if ! perf record -aB --synth=no -o "${perfdata}" ${testprog} 2> /dev/null
+ then
+ echo "System-wide record [Skipped not supported]"
+ return
+ fi
+ if ! perf report -i "${perfdata}" -q | grep -q "${testsym}"
+ then
+ echo "System-wide record [Failed missing output]"
+ err=1
+ return
+ fi
+ if ! perf record -aB --synth=no -e cpu-clock,cs --threads=cpu \
+ -o "${perfdata}" ${testprog} 2> /dev/null
+ then
+ echo "System-wide record [Failed record --threads option]"
+ err=1
+ return
+ fi
+ if ! perf report -i "${perfdata}" -q | grep -q "${testsym}"
+ then
+ echo "System-wide record [Failed --threads missing output]"
+ err=1
+ return
+ fi
+ echo "Basic --system-wide mode test [Success]"
+}
+
+test_workload() {
+ echo "Basic target workload test"
+ if ! perf record -o "${perfdata}" ${testprog} 2> /dev/null
+ then
+ echo "Workload record [Failed record]"
+ err=1
+ return
+ fi
+ if ! perf report -i "${perfdata}" -q | grep -q "${testsym}"
+ then
+ echo "Workload record [Failed missing output]"
+ err=1
+ return
+ fi
+ if ! perf record -e cpu-clock,cs --threads=package \
+ -o "${perfdata}" ${testprog} 2> /dev/null
+ then
+ echo "Workload record [Failed record --threads option]"
+ err=1
+ return
+ fi
+ if ! perf report -i "${perfdata}" -q | grep -q "${testsym}"
+ then
+ echo "Workload record [Failed --threads missing output]"
+ err=1
+ return
+ fi
+ echo "Basic target workload test [Success]"
+}
+
test_per_thread
test_register_capture
+test_system_wide
+test_workload
cleanup
exit $err
diff --git a/tools/perf/tests/shell/record_offcpu.sh b/tools/perf/tests/shell/record_offcpu.sh
index d2eba58..e01973d 100755
--- a/tools/perf/tests/shell/record_offcpu.sh
+++ b/tools/perf/tests/shell/record_offcpu.sh
@@ -51,7 +51,7 @@
err=1
return
fi
- if ! perf report -i ${perfdata} -q --percent-limit=90 | egrep -q sleep
+ if ! perf report -i ${perfdata} -q --percent-limit=90 | grep -E -q sleep
then
echo "Basic off-cpu test [Failed missing output]"
err=1
diff --git a/tools/perf/tests/shell/stat.sh b/tools/perf/tests/shell/stat.sh
index 26a51b4..2c1d3f7 100755
--- a/tools/perf/tests/shell/stat.sh
+++ b/tools/perf/tests/shell/stat.sh
@@ -7,7 +7,7 @@
err=0
test_default_stat() {
echo "Basic stat command test"
- if ! perf stat true 2>&1 | egrep -q "Performance counter stats for 'true':"
+ if ! perf stat true 2>&1 | grep -E -q "Performance counter stats for 'true':"
then
echo "Basic stat command test [Failed]"
err=1
@@ -19,7 +19,7 @@
test_stat_record_report() {
echo "stat record and report test"
if ! perf stat record -o - true | perf stat report -i - 2>&1 | \
- egrep -q "Performance counter stats for 'pipe':"
+ grep -E -q "Performance counter stats for 'pipe':"
then
echo "stat record and report test [Failed]"
err=1
@@ -55,13 +55,13 @@
echo "Topdown event group test [Skipped event parsing failed]"
return
fi
- if perf stat -e '{slots,topdown-retiring}' true 2>&1 | egrep -q "<not supported>"
+ if perf stat -e '{slots,topdown-retiring}' true 2>&1 | grep -E -q "<not supported>"
then
echo "Topdown event group test [Failed events not supported]"
err=1
return
fi
- if perf stat -e '{topdown-retiring,slots}' true 2>&1 | egrep -q "<not supported>"
+ if perf stat -e '{topdown-retiring,slots}' true 2>&1 | grep -E -q "<not supported>"
then
echo "Topdown event group test [Failed slots not reordered first]"
err=1
@@ -82,7 +82,7 @@
return
fi
group_needs_break="{slots,topdown-bad-spec,topdown-be-bound,topdown-fe-bound,topdown-retiring,branch-instructions,branch-misses,bus-cycles,cache-misses,cache-references,cpu-cycles,instructions,mem-loads,mem-stores,ref-cycles,cache-misses,cache-references}:W"
- if perf stat --no-merge -e "$group_needs_break" true 2>&1 | egrep -q "<not supported>"
+ if perf stat --no-merge -e "$group_needs_break" true 2>&1 | grep -E -q "<not supported>"
then
echo "Topdown weak groups test [Failed events not supported]"
err=1
diff --git a/tools/perf/tests/shell/test_arm_callgraph_fp.sh b/tools/perf/tests/shell/test_arm_callgraph_fp.sh
index ec108d4..e61d8de 100755
--- a/tools/perf/tests/shell/test_arm_callgraph_fp.sh
+++ b/tools/perf/tests/shell/test_arm_callgraph_fp.sh
@@ -4,44 +4,16 @@
lscpu | grep -q "aarch64" || exit 2
-if ! [ -x "$(command -v cc)" ]; then
- echo "failed: no compiler, install gcc"
- exit 2
-fi
-
PERF_DATA=$(mktemp /tmp/__perf_test.perf.data.XXXXX)
-TEST_PROGRAM_SOURCE=$(mktemp /tmp/test_program.XXXXX.c)
-TEST_PROGRAM=$(mktemp /tmp/test_program.XXXXX)
+TEST_PROGRAM="perf test -w leafloop"
cleanup_files()
{
rm -f $PERF_DATA
- rm -f $TEST_PROGRAM_SOURCE
- rm -f $TEST_PROGRAM
}
trap cleanup_files exit term int
-cat << EOF > $TEST_PROGRAM_SOURCE
-int a = 0;
-void leaf(void) {
- for (;;)
- a += a;
-}
-void parent(void) {
- leaf();
-}
-int main(void) {
- parent();
- return 0;
-}
-EOF
-
-echo " + Compiling test program ($TEST_PROGRAM)..."
-
-CFLAGS="-g -O0 -fno-inline -fno-omit-frame-pointer"
-cc $CFLAGS $TEST_PROGRAM_SOURCE -o $TEST_PROGRAM || exit 1
-
# Add a 1 second delay to skip samples that are not in the leaf() function
perf record -o $PERF_DATA --call-graph fp -e cycles//u -D 1000 --user-callchains -- $TEST_PROGRAM 2> /dev/null &
PID=$!
@@ -58,11 +30,11 @@
# program
# 728 leaf
# 753 parent
-# 76c main
+# 76c leafloop
# ...
perf script -i $PERF_DATA -F comm,ip,sym | head -n4
perf script -i $PERF_DATA -F comm,ip,sym | head -n4 | \
awk '{ if ($2 != "") sym[i++] = $2 } END { if (sym[0] != "leaf" ||
sym[1] != "parent" ||
- sym[2] != "main") exit 1 }'
+ sym[2] != "leafloop") exit 1 }'
diff --git a/tools/perf/tests/shell/test_arm_coresight.sh b/tools/perf/tests/shell/test_arm_coresight.sh
index daad786..565ce52 100755
--- a/tools/perf/tests/shell/test_arm_coresight.sh
+++ b/tools/perf/tests/shell/test_arm_coresight.sh
@@ -49,7 +49,7 @@
# touch 6512 1 branches:u: ffffb22082e0 strcmp+0xa0 (/lib/aarch64-linux-gnu/ld-2.27.so)
# touch 6512 1 branches:u: ffffb2208320 strcmp+0xe0 (/lib/aarch64-linux-gnu/ld-2.27.so)
perf script -F,-time -i ${perfdata} 2>&1 | \
- egrep " +$1 +[0-9]+ .* +branches:(.*:)? +" > /dev/null 2>&1
+ grep -E " +$1 +[0-9]+ .* +branches:(.*:)? +" > /dev/null 2>&1
}
perf_report_branch_samples() {
@@ -60,7 +60,7 @@
# 7.71% 7.71% touch libc-2.27.so [.] getenv
# 2.59% 2.59% touch ld-2.27.so [.] strcmp
perf report --stdio -i ${perfdata} 2>&1 | \
- egrep " +[0-9]+\.[0-9]+% +[0-9]+\.[0-9]+% +$1 " > /dev/null 2>&1
+ grep -E " +[0-9]+\.[0-9]+% +[0-9]+\.[0-9]+% +$1 " > /dev/null 2>&1
}
perf_report_instruction_samples() {
@@ -71,7 +71,7 @@
# 5.80% touch libc-2.27.so [.] getenv
# 4.35% touch ld-2.27.so [.] _dl_fixup
perf report --itrace=i20i --stdio -i ${perfdata} 2>&1 | \
- egrep " +[0-9]+\.[0-9]+% +$1" > /dev/null 2>&1
+ grep -E " +[0-9]+\.[0-9]+% +$1" > /dev/null 2>&1
}
arm_cs_report() {
@@ -87,7 +87,7 @@
# If the node of "enable_sink" is existed under the device path, this
# means the device is a sink device. Need to exclude 'tpiu' since it
# cannot support perf PMU.
- echo "$1" | egrep -q -v "tpiu"
+ echo "$1" | grep -E -q -v "tpiu"
if [ $? -eq 0 -a -e "$1/enable_sink" ]; then
diff --git a/tools/perf/tests/shell/test_arm_spe.sh b/tools/perf/tests/shell/test_arm_spe.sh
index 0d47479..aa094d7 100755
--- a/tools/perf/tests/shell/test_arm_spe.sh
+++ b/tools/perf/tests/shell/test_arm_spe.sh
@@ -9,7 +9,7 @@
# German Gomez <german.gomez@arm.com>, 2021
skip_if_no_arm_spe_event() {
- perf list | egrep -q 'arm_spe_[0-9]+//' && return 0
+ perf list | grep -E -q 'arm_spe_[0-9]+//' && return 0
# arm_spe event doesn't exist
return 2
@@ -51,7 +51,7 @@
# dd 3048 [002] 1 tlb-access: ffffaa64999c __GI___libc_write+0x3c (/lib/aarch64-linux-gnu/libc-2.27.so)
# dd 3048 [002] 1 memory: ffffaa64999c __GI___libc_write+0x3c (/lib/aarch64-linux-gnu/libc-2.27.so)
perf script -F,-time -i ${perfdata} 2>&1 | \
- egrep " +$1 +[0-9]+ .* +${events}:(.*:)? +" > /dev/null 2>&1
+ grep -E " +$1 +[0-9]+ .* +${events}:(.*:)? +" > /dev/null 2>&1
}
perf_report_samples() {
@@ -62,7 +62,7 @@
# 7.71% 7.71% dd libc-2.27.so [.] getenv
# 2.59% 2.59% dd ld-2.27.so [.] strcmp
perf report --stdio -i ${perfdata} 2>&1 | \
- egrep " +[0-9]+\.[0-9]+% +[0-9]+\.[0-9]+% +$1 " > /dev/null 2>&1
+ grep -E " +[0-9]+\.[0-9]+% +[0-9]+\.[0-9]+% +$1 " > /dev/null 2>&1
}
arm_spe_snapshot_test() {
diff --git a/tools/perf/tests/shell/test_arm_spe_fork.sh b/tools/perf/tests/shell/test_arm_spe_fork.sh
index c920d35..fad3616 100755
--- a/tools/perf/tests/shell/test_arm_spe_fork.sh
+++ b/tools/perf/tests/shell/test_arm_spe_fork.sh
@@ -5,20 +5,13 @@
# German Gomez <german.gomez@arm.com>, 2022
skip_if_no_arm_spe_event() {
- perf list | egrep -q 'arm_spe_[0-9]+//' && return 0
+ perf list | grep -E -q 'arm_spe_[0-9]+//' && return 0
return 2
}
skip_if_no_arm_spe_event || exit 2
-# skip if there's no compiler
-if ! [ -x "$(command -v cc)" ]; then
- echo "failed: no compiler, install gcc"
- exit 2
-fi
-
-TEST_PROGRAM_SOURCE=$(mktemp /tmp/__perf_test.program.XXXXX.c)
-TEST_PROGRAM=$(mktemp /tmp/__perf_test.program.XXXXX)
+TEST_PROGRAM="perf test -w sqrtloop 10"
PERF_DATA=$(mktemp /tmp/__perf_test.perf.data.XXXXX)
PERF_RECORD_LOG=$(mktemp /tmp/__perf_test.log.XXXXX)
@@ -27,43 +20,10 @@
echo "Cleaning up files..."
rm -f ${PERF_RECORD_LOG}
rm -f ${PERF_DATA}
- rm -f ${TEST_PROGRAM_SOURCE}
- rm -f ${TEST_PROGRAM}
}
trap cleanup_files exit term int
-# compile test program
-cat << EOF > $TEST_PROGRAM_SOURCE
-#include <math.h>
-#include <stdio.h>
-#include <stdlib.h>
-#include <unistd.h>
-#include <sys/wait.h>
-
-int workload() {
- while (1)
- sqrt(rand());
- return 0;
-}
-
-int main() {
- switch (fork()) {
- case 0:
- return workload();
- case -1:
- return 1;
- default:
- wait(NULL);
- }
- return 0;
-}
-EOF
-
-echo "Compiling test program..."
-CFLAGS="-lm"
-cc $TEST_PROGRAM_SOURCE $CFLAGS -o $TEST_PROGRAM || exit 1
-
echo "Recording workload..."
perf record -o ${PERF_DATA} -e arm_spe/period=65536/ -vvv -- $TEST_PROGRAM > ${PERF_RECORD_LOG} 2>&1 &
PERFPID=$!
@@ -78,8 +38,6 @@
kill $PERFPID
wait $PERFPID
-# test program may leave an orphan process running the workload
-killall $(basename $TEST_PROGRAM)
if [ "$log0" = "$log1" ];
then
diff --git a/tools/perf/tests/shell/test_brstack.sh b/tools/perf/tests/shell/test_brstack.sh
index d7ff5c4..59195eb 100755
--- a/tools/perf/tests/shell/test_brstack.sh
+++ b/tools/perf/tests/shell/test_brstack.sh
@@ -4,13 +4,6 @@
# SPDX-License-Identifier: GPL-2.0
# German Gomez <german.gomez@arm.com>, 2022
-# we need a C compiler to build the test programs
-# so bail if none is found
-if ! [ -x "$(command -v cc)" ]; then
- echo "failed: no compiler, install gcc"
- exit 2
-fi
-
# skip the test if the hardware doesn't support branch stack sampling
# and if the architecture doesn't support filter types: any,save_type,u
if ! perf record -o- --no-buildid --branch-filter any,save_type,u -- true > /dev/null 2>&1 ; then
@@ -19,6 +12,7 @@
fi
TMPDIR=$(mktemp -d /tmp/__perf_test.program.XXXXX)
+TESTPROG="perf test -w brstack"
cleanup() {
rm -rf $TMPDIR
@@ -26,57 +20,24 @@
trap cleanup exit term int
-gen_test_program() {
- # generate test program
- cat << EOF > $1
-#define BENCH_RUNS 999999
-int cnt;
-void bar(void) {
-} /* return */
-void foo(void) {
- bar(); /* call */
-} /* return */
-void bench(void) {
- void (*foo_ind)(void) = foo;
- if ((cnt++) % 3) /* branch (cond) */
- foo(); /* call */
- bar(); /* call */
- foo_ind(); /* call (ind) */
-}
-int main(void)
-{
- int cnt = 0;
- while (1) {
- if ((cnt++) > BENCH_RUNS)
- break;
- bench(); /* call */
- } /* branch (uncond) */
- return 0;
-}
-EOF
-}
-
test_user_branches() {
echo "Testing user branch stack sampling"
- gen_test_program "$TEMPDIR/program.c"
- cc -fno-inline -g "$TEMPDIR/program.c" -o $TMPDIR/a.out
-
- perf record -o $TMPDIR/perf.data --branch-filter any,save_type,u -- $TMPDIR/a.out > /dev/null 2>&1
+ perf record -o $TMPDIR/perf.data --branch-filter any,save_type,u -- ${TESTPROG} > /dev/null 2>&1
perf script -i $TMPDIR/perf.data --fields brstacksym | xargs -n1 > $TMPDIR/perf.script
# example of branch entries:
- # foo+0x14/bar+0x40/P/-/-/0/CALL
+ # brstack_foo+0x14/brstack_bar+0x40/P/-/-/0/CALL
set -x
- egrep -m1 "^bench\+[^ ]*/foo\+[^ ]*/IND_CALL$" $TMPDIR/perf.script
- egrep -m1 "^foo\+[^ ]*/bar\+[^ ]*/CALL$" $TMPDIR/perf.script
- egrep -m1 "^bench\+[^ ]*/foo\+[^ ]*/CALL$" $TMPDIR/perf.script
- egrep -m1 "^bench\+[^ ]*/bar\+[^ ]*/CALL$" $TMPDIR/perf.script
- egrep -m1 "^bar\+[^ ]*/foo\+[^ ]*/RET$" $TMPDIR/perf.script
- egrep -m1 "^foo\+[^ ]*/bench\+[^ ]*/RET$" $TMPDIR/perf.script
- egrep -m1 "^bench\+[^ ]*/bench\+[^ ]*/COND$" $TMPDIR/perf.script
- egrep -m1 "^main\+[^ ]*/main\+[^ ]*/UNCOND$" $TMPDIR/perf.script
+ grep -E -m1 "^brstack_bench\+[^ ]*/brstack_foo\+[^ ]*/IND_CALL$" $TMPDIR/perf.script
+ grep -E -m1 "^brstack_foo\+[^ ]*/brstack_bar\+[^ ]*/CALL$" $TMPDIR/perf.script
+ grep -E -m1 "^brstack_bench\+[^ ]*/brstack_foo\+[^ ]*/CALL$" $TMPDIR/perf.script
+ grep -E -m1 "^brstack_bench\+[^ ]*/brstack_bar\+[^ ]*/CALL$" $TMPDIR/perf.script
+ grep -E -m1 "^brstack_bar\+[^ ]*/brstack_foo\+[^ ]*/RET$" $TMPDIR/perf.script
+ grep -E -m1 "^brstack_foo\+[^ ]*/brstack_bench\+[^ ]*/RET$" $TMPDIR/perf.script
+ grep -E -m1 "^brstack_bench\+[^ ]*/brstack_bench\+[^ ]*/COND$" $TMPDIR/perf.script
+ grep -E -m1 "^brstack\+[^ ]*/brstack\+[^ ]*/UNCOND$" $TMPDIR/perf.script
set +x
# some branch types are still not being tested:
@@ -91,15 +52,12 @@
echo "Testing branch stack filtering permutation ($filter,$expect)"
- gen_test_program "$TEMPDIR/program.c"
- cc -fno-inline -g "$TEMPDIR/program.c" -o $TMPDIR/a.out
-
- perf record -o $TMPDIR/perf.data --branch-filter $filter,save_type,u -- $TMPDIR/a.out > /dev/null 2>&1
+ perf record -o $TMPDIR/perf.data --branch-filter $filter,save_type,u -- ${TESTPROG} > /dev/null 2>&1
perf script -i $TMPDIR/perf.data --fields brstack | xargs -n1 > $TMPDIR/perf.script
# fail if we find any branch type that doesn't match any of the expected ones
# also consider UNKNOWN branch types (-)
- if egrep -vm1 "^[^ ]*/($expect|-|( *))$" $TMPDIR/perf.script; then
+ if grep -E -vm1 "^[^ ]*/($expect|-|( *))$" $TMPDIR/perf.script; then
return 1
fi
}
diff --git a/tools/perf/tests/shell/test_data_symbol.sh b/tools/perf/tests/shell/test_data_symbol.sh
index cd6eb54..69bb6fe 100755
--- a/tools/perf/tests/shell/test_data_symbol.sh
+++ b/tools/perf/tests/shell/test_data_symbol.sh
@@ -5,19 +5,13 @@
# Leo Yan <leo.yan@linaro.org>, 2022
skip_if_no_mem_event() {
- perf mem record -e list 2>&1 | egrep -q 'available' && return 0
+ perf mem record -e list 2>&1 | grep -E -q 'available' && return 0
return 2
}
skip_if_no_mem_event || exit 2
-# skip if there's no compiler
-if ! [ -x "$(command -v cc)" ]; then
- echo "skip: no compiler, install gcc"
- exit 2
-fi
-
-TEST_PROGRAM=$(mktemp /tmp/__perf_test.program.XXXXX)
+TEST_PROGRAM="perf test -w datasym"
PERF_DATA=$(mktemp /tmp/__perf_test.perf.data.XXXXX)
check_result() {
@@ -45,37 +39,16 @@
{
echo "Cleaning up files..."
rm -f ${PERF_DATA}
- rm -f ${TEST_PROGRAM}
}
trap cleanup_files exit term int
-# compile test program
-echo "Compiling test program..."
-cat << EOF | cc -o ${TEST_PROGRAM} -x c -
-typedef struct _buf {
- char data1;
- char reserved[55];
- char data2;
-} buf __attribute__((aligned(64)));
-
-static buf buf1;
-
-int main(void) {
- for (;;) {
- buf1.data1++;
- buf1.data2 += buf1.data1;
- }
- return 0;
-}
-EOF
-
echo "Recording workload..."
# perf mem/c2c internally uses IBS PMU on AMD CPU which doesn't support
# user/kernel filtering and per-process monitoring, spin program on
# specific CPU and test in per-CPU mode.
-is_amd=$(egrep -c 'vendor_id.*AuthenticAMD' /proc/cpuinfo)
+is_amd=$(grep -E -c 'vendor_id.*AuthenticAMD' /proc/cpuinfo)
if (($is_amd >= 1)); then
perf mem record -o ${PERF_DATA} -C 0 -- taskset -c 0 $TEST_PROGRAM &
else
diff --git a/tools/perf/tests/shell/test_java_symbol.sh b/tools/perf/tests/shell/test_java_symbol.sh
index f221225..90cea88 100755
--- a/tools/perf/tests/shell/test_java_symbol.sh
+++ b/tools/perf/tests/shell/test_java_symbol.sh
@@ -65,7 +65,7 @@
# 8.18% jshell jitted-50116-29.so [.] Interpreter
# 0.75% Thread-1 jitted-83602-1670.so [.] jdk.internal.jimage.BasicImageReader.getString(int)
perf report --stdio -i ${PERF_INJ_DATA} 2>&1 | \
- egrep " +[0-9]+\.[0-9]+% .* (Interpreter|jdk\.internal).*" > /dev/null 2>&1
+ grep -E " +[0-9]+\.[0-9]+% .* (Interpreter|jdk\.internal).*" > /dev/null 2>&1
if [ $? -ne 0 ]; then
echo "Fail to find java symbols"
diff --git a/tools/perf/tests/shell/test_task_analyzer.sh b/tools/perf/tests/shell/test_task_analyzer.sh
new file mode 100755
index 0000000..a98e4ab
--- /dev/null
+++ b/tools/perf/tests/shell/test_task_analyzer.sh
@@ -0,0 +1,151 @@
+#!/bin/bash
+# perf script task-analyzer tests
+# SPDX-License-Identifier: GPL-2.0
+
+tmpdir=$(mktemp -d /tmp/perf-script-task-analyzer-XXXXX)
+err=0
+
+cleanup() {
+ rm -f perf.data
+ rm -f perf.data.old
+ rm -f csv
+ rm -f csvsummary
+ rm -rf $tmpdir
+ trap - exit term int
+}
+
+trap_cleanup() {
+ cleanup
+ exit 1
+}
+trap trap_cleanup exit term int
+
+report() {
+ if [ $1 = 0 ]; then
+ echo "PASS: \"$2\""
+ else
+ echo "FAIL: \"$2\" Error message: \"$3\""
+ err=1
+ fi
+}
+
+check_exec_0() {
+ if [ $? != 0 ]; then
+ report 1 "invokation of ${$1} command failed"
+ fi
+}
+
+find_str_or_fail() {
+ grep -q "$1" $2
+ if [ $? != 0 ]; then
+ report 1 $3 "Failed to find required string:'${1}'."
+ else
+ report 0 $3
+ fi
+}
+
+prepare_perf_data() {
+ # 1s should be sufficient to catch at least some switches
+ perf record -e sched:sched_switch -a -- sleep 1 > /dev/null 2>&1
+}
+
+# check standard inkvokation with no arguments
+test_basic() {
+ out="$tmpdir/perf.out"
+ perf script report task-analyzer > $out
+ check_exec_0 "perf"
+ find_str_or_fail "Comm" $out ${FUNCNAME[0]}
+}
+
+test_ns_rename(){
+ out="$tmpdir/perf.out"
+ perf script report task-analyzer --ns --rename-comms-by-tids 0:random > $out
+ check_exec_0 "perf"
+ find_str_or_fail "Comm" $out ${FUNCNAME[0]}
+}
+
+test_ms_filtertasks_highlight(){
+ out="$tmpdir/perf.out"
+ perf script report task-analyzer --ms --filter-tasks perf --highlight-tasks perf \
+ > $out
+ check_exec_0 "perf"
+ find_str_or_fail "Comm" $out ${FUNCNAME[0]}
+}
+
+test_extended_times_timelimit_limittasks() {
+ out="$tmpdir/perf.out"
+ perf script report task-analyzer --extended-times --time-limit :99999 \
+ --limit-to-tasks perf > $out
+ check_exec_0 "perf"
+ find_str_or_fail "Out-Out" $out ${FUNCNAME[0]}
+}
+
+test_summary() {
+ out="$tmpdir/perf.out"
+ perf script report task-analyzer --summary > $out
+ check_exec_0 "perf"
+ find_str_or_fail "Summary" $out ${FUNCNAME[0]}
+}
+
+test_summaryextended() {
+ out="$tmpdir/perf.out"
+ perf script report task-analyzer --summary-extended > $out
+ check_exec_0 "perf"
+ find_str_or_fail "Inter Task Times" $out ${FUNCNAME[0]}
+}
+
+test_summaryonly() {
+ out="$tmpdir/perf.out"
+ perf script report task-analyzer --summary-only > $out
+ check_exec_0 "perf"
+ find_str_or_fail "Summary" $out ${FUNCNAME[0]}
+}
+
+test_extended_times_summary_ns() {
+ out="$tmpdir/perf.out"
+ perf script report task-analyzer --extended-times --summary --ns > $out
+ check_exec_0 "perf"
+ find_str_or_fail "Out-Out" $out ${FUNCNAME[0]}
+ find_str_or_fail "Summary" $out ${FUNCNAME[0]}
+}
+
+test_csv() {
+ perf script report task-analyzer --csv csv > /dev/null
+ check_exec_0 "perf"
+ find_str_or_fail "Comm;" csv ${FUNCNAME[0]}
+}
+
+test_csv_extended_times() {
+ perf script report task-analyzer --csv csv --extended-times > /dev/null
+ check_exec_0 "perf"
+ find_str_or_fail "Out-Out;" csv ${FUNCNAME[0]}
+}
+
+test_csvsummary() {
+ perf script report task-analyzer --csv-summary csvsummary > /dev/null
+ check_exec_0 "perf"
+ find_str_or_fail "Comm;" csvsummary ${FUNCNAME[0]}
+}
+
+test_csvsummary_extended() {
+ perf script report task-analyzer --csv-summary csvsummary --summary-extended \
+ >/dev/null
+ check_exec_0 "perf"
+ find_str_or_fail "Out-Out;" csvsummary ${FUNCNAME[0]}
+}
+
+prepare_perf_data
+test_basic
+test_ns_rename
+test_ms_filtertasks_highlight
+test_extended_times_timelimit_limittasks
+test_summary
+test_summaryextended
+test_summaryonly
+test_extended_times_summary_ns
+test_csv
+test_csvsummary
+test_csv_extended_times
+test_csvsummary_extended
+cleanup
+exit $err
diff --git a/tools/perf/tests/shell/trace+probe_vfs_getname.sh b/tools/perf/tests/shell/trace+probe_vfs_getname.sh
index 3d60e99..0a4bac3 100755
--- a/tools/perf/tests/shell/trace+probe_vfs_getname.sh
+++ b/tools/perf/tests/shell/trace+probe_vfs_getname.sh
@@ -18,9 +18,9 @@
. $(dirname $0)/lib/probe_vfs_getname.sh
trace_open_vfs_getname() {
- evts=$(echo $(perf list syscalls:sys_enter_open* 2>/dev/null | egrep 'open(at)? ' | sed -r 's/.*sys_enter_([a-z]+) +\[.*$/\1/') | sed 's/ /,/')
+ evts=$(echo $(perf list syscalls:sys_enter_open* 2>/dev/null | grep -E 'open(at)? ' | sed -r 's/.*sys_enter_([a-z]+) +\[.*$/\1/') | sed 's/ /,/')
perf trace -e $evts touch $file 2>&1 | \
- egrep " +[0-9]+\.[0-9]+ +\( +[0-9]+\.[0-9]+ ms\): +touch\/[0-9]+ open(at)?\((dfd: +CWD, +)?filename: +${file}, +flags: CREAT\|NOCTTY\|NONBLOCK\|WRONLY, +mode: +IRUGO\|IWUGO\) += +[0-9]+$"
+ grep -E " +[0-9]+\.[0-9]+ +\( +[0-9]+\.[0-9]+ ms\): +touch\/[0-9]+ open(at)?\((dfd: +CWD, +)?filename: +${file}, +flags: CREAT\|NOCTTY\|NONBLOCK\|WRONLY, +mode: +IRUGO\|IWUGO\) += +[0-9]+$"
}
diff --git a/tools/perf/tests/sw-clock.c b/tools/perf/tests/sw-clock.c
index 9cd6fec..4d7493f 100644
--- a/tools/perf/tests/sw-clock.c
+++ b/tools/perf/tests/sw-clock.c
@@ -13,6 +13,7 @@
#include "util/evlist.h"
#include "util/cpumap.h"
#include "util/mmap.h"
+#include "util/sample.h"
#include "util/thread_map.h"
#include <perf/evlist.h>
#include <perf/mmap.h>
diff --git a/tools/perf/tests/switch-tracking.c b/tools/perf/tests/switch-tracking.c
index 87f565c..b3bd14b 100644
--- a/tools/perf/tests/switch-tracking.c
+++ b/tools/perf/tests/switch-tracking.c
@@ -19,6 +19,7 @@
#include "record.h"
#include "tests.h"
#include "util/mmap.h"
+#include "util/sample.h"
#include "pmu.h"
static int spin_sleep(void)
diff --git a/tools/perf/tests/tests.h b/tools/perf/tests/tests.h
index 5bbb8f6..fb4b5ad 100644
--- a/tools/perf/tests/tests.h
+++ b/tools/perf/tests/tests.h
@@ -147,6 +147,7 @@ DECLARE_SUITE(expand_cgroup_events);
DECLARE_SUITE(perf_time_to_tsc);
DECLARE_SUITE(dlfilter);
DECLARE_SUITE(sigtrap);
+DECLARE_SUITE(event_groups);
/*
* PowerPC and S390 do not support creation of instruction breakpoints using the
@@ -180,4 +181,31 @@ int test__arch_unwind_sample(struct perf_sample *sample,
DECLARE_SUITE(vectors_page);
#endif
+/*
+ * Define test workloads to be used in test suites.
+ */
+typedef int (*workload_fnptr)(int argc, const char **argv);
+
+struct test_workload {
+ const char *name;
+ workload_fnptr func;
+};
+
+#define DECLARE_WORKLOAD(work) \
+ extern struct test_workload workload__##work
+
+#define DEFINE_WORKLOAD(work) \
+struct test_workload workload__##work = { \
+ .name = #work, \
+ .func = work, \
+}
+
+/* The list of test workloads */
+DECLARE_WORKLOAD(noploop);
+DECLARE_WORKLOAD(thloop);
+DECLARE_WORKLOAD(leafloop);
+DECLARE_WORKLOAD(sqrtloop);
+DECLARE_WORKLOAD(brstack);
+DECLARE_WORKLOAD(datasym);
+
#endif /* TESTS_H */
diff --git a/tools/perf/tests/thread-map.c b/tools/perf/tests/thread-map.c
index e413c13..74308c1 100644
--- a/tools/perf/tests/thread-map.c
+++ b/tools/perf/tests/thread-map.c
@@ -11,6 +11,7 @@
#include "util/synthetic-events.h"
#include <linux/zalloc.h>
#include <perf/event.h>
+#include <internal/threadmap.h>
struct perf_sample;
struct perf_tool;
diff --git a/tools/perf/tests/workloads/Build b/tools/perf/tests/workloads/Build
new file mode 100644
index 0000000..a1f34d5
--- /dev/null
+++ b/tools/perf/tests/workloads/Build
@@ -0,0 +1,13 @@
+# SPDX-License-Identifier: GPL-2.0
+
+perf-y += noploop.o
+perf-y += thloop.o
+perf-y += leafloop.o
+perf-y += sqrtloop.o
+perf-y += brstack.o
+perf-y += datasym.o
+
+CFLAGS_sqrtloop.o = -g -O0 -fno-inline -U_FORTIFY_SOURCE
+CFLAGS_leafloop.o = -g -O0 -fno-inline -fno-omit-frame-pointer -U_FORTIFY_SOURCE
+CFLAGS_brstack.o = -g -O0 -fno-inline -U_FORTIFY_SOURCE
+CFLAGS_datasym.o = -g -O0 -fno-inline -U_FORTIFY_SOURCE
diff --git a/tools/perf/tests/workloads/brstack.c b/tools/perf/tests/workloads/brstack.c
new file mode 100644
index 0000000..0b60bd3
--- /dev/null
+++ b/tools/perf/tests/workloads/brstack.c
@@ -0,0 +1,40 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#include <stdlib.h>
+#include "../tests.h"
+
+#define BENCH_RUNS 999999
+
+static volatile int cnt;
+
+static void brstack_bar(void) {
+} /* return */
+
+static void brstack_foo(void) {
+ brstack_bar(); /* call */
+} /* return */
+
+static void brstack_bench(void) {
+ void (*brstack_foo_ind)(void) = brstack_foo;
+
+ if ((cnt++) % 3) /* branch (cond) */
+ brstack_foo(); /* call */
+ brstack_bar(); /* call */
+ brstack_foo_ind(); /* call (ind) */
+}
+
+static int brstack(int argc, const char **argv)
+{
+ int num_loops = BENCH_RUNS;
+
+ if (argc > 0)
+ num_loops = atoi(argv[0]);
+
+ while (1) {
+ if ((cnt++) > num_loops)
+ break;
+ brstack_bench();/* call */
+ } /* branch (uncond) */
+ return 0;
+}
+
+DEFINE_WORKLOAD(brstack);
diff --git a/tools/perf/tests/workloads/datasym.c b/tools/perf/tests/workloads/datasym.c
new file mode 100644
index 0000000..ddd40bc
--- /dev/null
+++ b/tools/perf/tests/workloads/datasym.c
@@ -0,0 +1,24 @@
+#include <linux/compiler.h>
+#include "../tests.h"
+
+typedef struct _buf {
+ char data1;
+ char reserved[55];
+ char data2;
+} buf __attribute__((aligned(64)));
+
+static buf buf1 = {
+ /* to have this in the data section */
+ .reserved[0] = 1,
+};
+
+static int datasym(int argc __maybe_unused, const char **argv __maybe_unused)
+{
+ for (;;) {
+ buf1.data1++;
+ buf1.data2 += buf1.data1;
+ }
+ return 0;
+}
+
+DEFINE_WORKLOAD(datasym);
diff --git a/tools/perf/tests/workloads/leafloop.c b/tools/perf/tests/workloads/leafloop.c
new file mode 100644
index 0000000..1bf5cc9
--- /dev/null
+++ b/tools/perf/tests/workloads/leafloop.c
@@ -0,0 +1,34 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#include <stdlib.h>
+#include <linux/compiler.h>
+#include "../tests.h"
+
+/* We want to check these symbols in perf script */
+noinline void leaf(volatile int b);
+noinline void parent(volatile int b);
+
+static volatile int a;
+
+noinline void leaf(volatile int b)
+{
+ for (;;)
+ a += b;
+}
+
+noinline void parent(volatile int b)
+{
+ leaf(b);
+}
+
+static int leafloop(int argc, const char **argv)
+{
+ int c = 1;
+
+ if (argc > 0)
+ c = atoi(argv[0]);
+
+ parent(c);
+ return 0;
+}
+
+DEFINE_WORKLOAD(leafloop);
diff --git a/tools/perf/tests/workloads/noploop.c b/tools/perf/tests/workloads/noploop.c
new file mode 100644
index 0000000..940ea59
--- /dev/null
+++ b/tools/perf/tests/workloads/noploop.c
@@ -0,0 +1,32 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#include <stdlib.h>
+#include <signal.h>
+#include <unistd.h>
+#include <linux/compiler.h>
+#include "../tests.h"
+
+static volatile sig_atomic_t done;
+
+static void sighandler(int sig __maybe_unused)
+{
+ done = 1;
+}
+
+static int noploop(int argc, const char **argv)
+{
+ int sec = 1;
+
+ if (argc > 0)
+ sec = atoi(argv[0]);
+
+ signal(SIGINT, sighandler);
+ signal(SIGALRM, sighandler);
+ alarm(sec);
+
+ while (!done)
+ continue;
+
+ return 0;
+}
+
+DEFINE_WORKLOAD(noploop);
diff --git a/tools/perf/tests/workloads/sqrtloop.c b/tools/perf/tests/workloads/sqrtloop.c
new file mode 100644
index 0000000..ccc94c6
--- /dev/null
+++ b/tools/perf/tests/workloads/sqrtloop.c
@@ -0,0 +1,45 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#include <math.h>
+#include <signal.h>
+#include <stdlib.h>
+#include <unistd.h>
+#include <linux/compiler.h>
+#include <sys/wait.h>
+#include "../tests.h"
+
+static volatile sig_atomic_t done;
+
+static void sighandler(int sig __maybe_unused)
+{
+ done = 1;
+}
+
+static int __sqrtloop(int sec)
+{
+ signal(SIGALRM, sighandler);
+ alarm(sec);
+
+ while (!done)
+ (void)sqrt(rand());
+ return 0;
+}
+
+static int sqrtloop(int argc, const char **argv)
+{
+ int sec = 1;
+
+ if (argc > 0)
+ sec = atoi(argv[0]);
+
+ switch (fork()) {
+ case 0:
+ return __sqrtloop(sec);
+ case -1:
+ return -1;
+ default:
+ wait(NULL);
+ }
+ return 0;
+}
+
+DEFINE_WORKLOAD(sqrtloop);
diff --git a/tools/perf/tests/workloads/thloop.c b/tools/perf/tests/workloads/thloop.c
new file mode 100644
index 0000000..29193b7
--- /dev/null
+++ b/tools/perf/tests/workloads/thloop.c
@@ -0,0 +1,53 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#include <pthread.h>
+#include <stdlib.h>
+#include <signal.h>
+#include <unistd.h>
+#include <linux/compiler.h>
+#include "../tests.h"
+
+static volatile sig_atomic_t done;
+static volatile unsigned count;
+
+/* We want to check this symbol in perf report */
+noinline void test_loop(void);
+
+static void sighandler(int sig __maybe_unused)
+{
+ done = 1;
+}
+
+noinline void test_loop(void)
+{
+ while (!done)
+ count++;
+}
+
+static void *thfunc(void *arg)
+{
+ void (*loop_fn)(void) = arg;
+
+ loop_fn();
+ return NULL;
+}
+
+static int thloop(int argc, const char **argv)
+{
+ int sec = 1;
+ pthread_t th;
+
+ if (argc > 0)
+ sec = atoi(argv[0]);
+
+ signal(SIGINT, sighandler);
+ signal(SIGALRM, sighandler);
+ alarm(sec);
+
+ pthread_create(&th, NULL, thfunc, test_loop);
+ test_loop();
+ pthread_join(th, NULL);
+
+ return 0;
+}
+
+DEFINE_WORKLOAD(thloop);
diff --git a/tools/perf/tests/wp.c b/tools/perf/tests/wp.c
index 56455da..cc87196 100644
--- a/tools/perf/tests/wp.c
+++ b/tools/perf/tests/wp.c
@@ -59,8 +59,10 @@ static int __event(int wp_type, void *wp_addr, unsigned long wp_len)
get__perf_event_attr(&attr, wp_type, wp_addr, wp_len);
fd = sys_perf_event_open(&attr, 0, -1, -1,
perf_event_open_cloexec_flag());
- if (fd < 0)
+ if (fd < 0) {
+ fd = -errno;
pr_debug("failed opening event %x\n", attr.bp_type);
+ }
return fd;
}
@@ -77,7 +79,7 @@ static int test__wp_ro(struct test_suite *test __maybe_unused,
fd = __event(HW_BREAKPOINT_R, (void *)&data1, sizeof(data1));
if (fd < 0)
- return -1;
+ return fd == -ENODEV ? TEST_SKIP : -1;
tmp = data1;
WP_TEST_ASSERT_VAL(fd, "RO watchpoint", 1);
@@ -101,7 +103,7 @@ static int test__wp_wo(struct test_suite *test __maybe_unused,
fd = __event(HW_BREAKPOINT_W, (void *)&data1, sizeof(data1));
if (fd < 0)
- return -1;
+ return fd == -ENODEV ? TEST_SKIP : -1;
tmp = data1;
WP_TEST_ASSERT_VAL(fd, "WO watchpoint", 0);
@@ -126,7 +128,7 @@ static int test__wp_rw(struct test_suite *test __maybe_unused,
fd = __event(HW_BREAKPOINT_R | HW_BREAKPOINT_W, (void *)&data1,
sizeof(data1));
if (fd < 0)
- return -1;
+ return fd == -ENODEV ? TEST_SKIP : -1;
tmp = data1;
WP_TEST_ASSERT_VAL(fd, "RW watchpoint", 1);
@@ -150,7 +152,7 @@ static int test__wp_modify(struct test_suite *test __maybe_unused, int subtest _
fd = __event(HW_BREAKPOINT_W, (void *)&data1, sizeof(data1));
if (fd < 0)
- return -1;
+ return fd == -ENODEV ? TEST_SKIP : -1;
data1 = tmp;
WP_TEST_ASSERT_VAL(fd, "Modify watchpoint", 1);
diff --git a/tools/perf/trace/beauty/Build b/tools/perf/trace/beauty/Build
index 433dc39..d11ce25 100644
--- a/tools/perf/trace/beauty/Build
+++ b/tools/perf/trace/beauty/Build
@@ -17,4 +17,5 @@
perf-y += socket.o
perf-y += statx.o
perf-y += sync_file_range.o
+perf-y += timespec.o
perf-y += tracepoints/
diff --git a/tools/perf/trace/beauty/beauty.h b/tools/perf/trace/beauty/beauty.h
index f527a46..4c59edd 100644
--- a/tools/perf/trace/beauty/beauty.h
+++ b/tools/perf/trace/beauty/beauty.h
@@ -244,6 +244,9 @@ size_t syscall_arg__scnprintf_statx_mask(char *bf, size_t size, struct syscall_a
size_t syscall_arg__scnprintf_sync_file_range_flags(char *bf, size_t size, struct syscall_arg *arg);
#define SCA_SYNC_FILE_RANGE_FLAGS syscall_arg__scnprintf_sync_file_range_flags
+size_t syscall_arg__scnprintf_timespec(char *bf, size_t size, struct syscall_arg *arg);
+#define SCA_TIMESPEC syscall_arg__scnprintf_timespec
+
size_t open__scnprintf_flags(unsigned long flags, char *bf, size_t size, bool show_prefix);
void syscall_arg__set_ret_scnprintf(struct syscall_arg *arg,
diff --git a/tools/perf/trace/beauty/fadvise.sh b/tools/perf/trace/beauty/fadvise.sh
index b15ae38..4d3dd6e 100755
--- a/tools/perf/trace/beauty/fadvise.sh
+++ b/tools/perf/trace/beauty/fadvise.sh
@@ -6,7 +6,7 @@
printf "static const char *fadvise_advices[] = {\n"
regex='^[[:space:]]*#[[:space:]]*define[[:space:]]+POSIX_FADV_(\w+)[[:space:]]+([[:digit:]]+)[[:space:]]+.*'
-egrep $regex ${header_dir}/fadvise.h | \
+grep -E $regex ${header_dir}/fadvise.h | \
sed -r "s/$regex/\2 \1/g" | \
sort | xargs printf "\t[%s] = \"%s\",\n" | \
grep -v "[6].*DONTNEED" | grep -v "[7].*NOREUSE"
diff --git a/tools/perf/trace/beauty/fsmount.sh b/tools/perf/trace/beauty/fsmount.sh
index 615cc0f..cba8897 100755
--- a/tools/perf/trace/beauty/fsmount.sh
+++ b/tools/perf/trace/beauty/fsmount.sh
@@ -16,7 +16,7 @@
printf "static const char *fsmount_attr_flags[] = {\n"
regex='^[[:space:]]*#[[:space:]]*define[[:space:]]+MOUNT_ATTR_([[:alnum:]][[:alnum:]_]+)[[:space:]]+(0x[[:xdigit:]]+)[[:space:]]*.*'
-egrep $regex ${linux_mount} | grep -v MOUNT_ATTR_RELATIME | \
+grep -E $regex ${linux_mount} | grep -v MOUNT_ATTR_RELATIME | \
sed -r "s/$regex/\2 \1/g" | \
xargs printf "\t[ilog2(%s) + 1] = \"%s\",\n"
printf "};\n"
diff --git a/tools/perf/trace/beauty/fspick.sh b/tools/perf/trace/beauty/fspick.sh
index b220e07..1f08832 100755
--- a/tools/perf/trace/beauty/fspick.sh
+++ b/tools/perf/trace/beauty/fspick.sh
@@ -11,7 +11,7 @@
printf "static const char *fspick_flags[] = {\n"
regex='^[[:space:]]*#[[:space:]]*define[[:space:]]+FSPICK_([[:alnum:]_]+)[[:space:]]+(0x[[:xdigit:]]+)[[:space:]]*.*'
-egrep $regex ${linux_mount} | \
+grep -E $regex ${linux_mount} | \
sed -r "s/$regex/\2 \1/g" | \
xargs printf "\t[ilog2(%s) + 1] = \"%s\",\n"
printf "};\n"
diff --git a/tools/perf/trace/beauty/kcmp_type.sh b/tools/perf/trace/beauty/kcmp_type.sh
index df8b174..5df9dcb 100755
--- a/tools/perf/trace/beauty/kcmp_type.sh
+++ b/tools/perf/trace/beauty/kcmp_type.sh
@@ -5,7 +5,7 @@
printf "static const char *kcmp_types[] = {\n"
regex='^[[:space:]]+(KCMP_(\w+)),'
-egrep $regex ${header_dir}/kcmp.h | grep -v KCMP_TYPES, | \
+grep -E $regex ${header_dir}/kcmp.h | grep -v KCMP_TYPES, | \
sed -r "s/$regex/\1 \2/g" | \
xargs printf "\t[%s]\t= \"%s\",\n"
printf "};\n"
diff --git a/tools/perf/trace/beauty/kvm_ioctl.sh b/tools/perf/trace/beauty/kvm_ioctl.sh
index 4ce54f5..bd0efd4 100755
--- a/tools/perf/trace/beauty/kvm_ioctl.sh
+++ b/tools/perf/trace/beauty/kvm_ioctl.sh
@@ -5,8 +5,8 @@
printf "static const char *kvm_ioctl_cmds[] = {\n"
regex='^#[[:space:]]*define[[:space:]]+KVM_(\w+)[[:space:]]+_IO[RW]*\([[:space:]]*KVMIO[[:space:]]*,[[:space:]]*(0x[[:xdigit:]]+).*'
-egrep $regex ${header_dir}/kvm.h | \
+grep -E $regex ${header_dir}/kvm.h | \
sed -r "s/$regex/\2 \1/g" | \
- egrep -v " ((ARM|PPC|S390)_|[GS]ET_(DEBUGREGS|PIT2|XSAVE|TSC_KHZ)|CREATE_SPAPR_TCE_64)" | \
+ grep -E -v " ((ARM|PPC|S390)_|[GS]ET_(DEBUGREGS|PIT2|XSAVE|TSC_KHZ)|CREATE_SPAPR_TCE_64)" | \
sort | xargs printf "\t[%s] = \"%s\",\n"
printf "};\n"
diff --git a/tools/perf/trace/beauty/madvise_behavior.sh b/tools/perf/trace/beauty/madvise_behavior.sh
index 4527d29..c659c33 100755
--- a/tools/perf/trace/beauty/madvise_behavior.sh
+++ b/tools/perf/trace/beauty/madvise_behavior.sh
@@ -5,7 +5,7 @@
printf "static const char *madvise_advices[] = {\n"
regex='^[[:space:]]*#[[:space:]]*define[[:space:]]+MADV_([[:alnum:]_]+)[[:space:]]+([[:digit:]]+)[[:space:]]*.*'
-egrep $regex ${header_dir}/mman-common.h | \
+grep -E $regex ${header_dir}/mman-common.h | \
sed -r "s/$regex/\2 \1/g" | \
sort -n | xargs printf "\t[%s] = \"%s\",\n"
printf "};\n"
diff --git a/tools/perf/trace/beauty/mmap_flags.sh b/tools/perf/trace/beauty/mmap_flags.sh
index 7682571..3022597 100755
--- a/tools/perf/trace/beauty/mmap_flags.sh
+++ b/tools/perf/trace/beauty/mmap_flags.sh
@@ -15,26 +15,26 @@
linux_mman=${linux_header_dir}/mman.h
arch_mman=${arch_header_dir}/mman.h
-# those in egrep -vw are flags, we want just the bits
+# those in grep -E -vw are flags, we want just the bits
printf "static const char *mmap_flags[] = {\n"
regex='^[[:space:]]*#[[:space:]]*define[[:space:]]+MAP_([[:alnum:]_]+)[[:space:]]+(0x[[:xdigit:]]+)[[:space:]]*.*'
-egrep -q $regex ${arch_mman} && \
-(egrep $regex ${arch_mman} | \
+grep -E -q $regex ${arch_mman} && \
+(grep -E $regex ${arch_mman} | \
sed -r "s/$regex/\2 \1 \1 \1 \2/g" | \
xargs printf "\t[ilog2(%s) + 1] = \"%s\",\n#ifndef MAP_%s\n#define MAP_%s %s\n#endif\n")
-egrep -q $regex ${linux_mman} && \
-(egrep $regex ${linux_mman} | \
- egrep -vw 'MAP_(UNINITIALIZED|TYPE|SHARED_VALIDATE)' | \
+grep -E -q $regex ${linux_mman} && \
+(grep -E $regex ${linux_mman} | \
+ grep -E -vw 'MAP_(UNINITIALIZED|TYPE|SHARED_VALIDATE)' | \
sed -r "s/$regex/\2 \1 \1 \1 \2/g" | \
xargs printf "\t[ilog2(%s) + 1] = \"%s\",\n#ifndef MAP_%s\n#define MAP_%s %s\n#endif\n")
-([ ! -f ${arch_mman} ] || egrep -q '#[[:space:]]*include[[:space:]]+.*uapi/asm-generic/mman.*' ${arch_mman}) &&
-(egrep $regex ${header_dir}/mman-common.h | \
- egrep -vw 'MAP_(UNINITIALIZED|TYPE|SHARED_VALIDATE)' | \
+([ ! -f ${arch_mman} ] || grep -E -q '#[[:space:]]*include[[:space:]]+.*uapi/asm-generic/mman.*' ${arch_mman}) &&
+(grep -E $regex ${header_dir}/mman-common.h | \
+ grep -E -vw 'MAP_(UNINITIALIZED|TYPE|SHARED_VALIDATE)' | \
sed -r "s/$regex/\2 \1 \1 \1 \2/g" | \
xargs printf "\t[ilog2(%s) + 1] = \"%s\",\n#ifndef MAP_%s\n#define MAP_%s %s\n#endif\n")
-([ ! -f ${arch_mman} ] || egrep -q '#[[:space:]]*include[[:space:]]+.*uapi/asm-generic/mman.h>.*' ${arch_mman}) &&
-(egrep $regex ${header_dir}/mman.h | \
+([ ! -f ${arch_mman} ] || grep -E -q '#[[:space:]]*include[[:space:]]+.*uapi/asm-generic/mman.h>.*' ${arch_mman}) &&
+(grep -E $regex ${header_dir}/mman.h | \
sed -r "s/$regex/\2 \1 \1 \1 \2/g" | \
xargs printf "\t[ilog2(%s) + 1] = \"%s\",\n#ifndef MAP_%s\n#define MAP_%s %s\n#endif\n")
printf "};\n"
diff --git a/tools/perf/trace/beauty/mmap_prot.sh b/tools/perf/trace/beauty/mmap_prot.sh
index 664d8d5..49e8c86 100755
--- a/tools/perf/trace/beauty/mmap_prot.sh
+++ b/tools/perf/trace/beauty/mmap_prot.sh
@@ -17,14 +17,14 @@
printf "static const char *mmap_prot[] = {\n"
regex=`printf '^[[:space:]]*#[[:space:]]*define[[:space:]]+%s_([[:alnum:]_]+)[[:space:]]+(0x[[:xdigit:]]+)[[:space:]]*.*' ${prefix}`
-([ ! -f ${arch_mman} ] || egrep -q '#[[:space:]]*include[[:space:]]+.*uapi/asm-generic/mman.*' ${arch_mman}) &&
-(egrep $regex ${common_mman} | \
- egrep -vw PROT_NONE | \
+([ ! -f ${arch_mman} ] || grep -E -q '#[[:space:]]*include[[:space:]]+.*uapi/asm-generic/mman.*' ${arch_mman}) &&
+(grep -E $regex ${common_mman} | \
+ grep -E -vw PROT_NONE | \
sed -r "s/$regex/\2 \1 \1 \1 \2/g" | \
xargs printf "\t[ilog2(%s) + 1] = \"%s\",\n#ifndef ${prefix}_%s\n#define ${prefix}_%s %s\n#endif\n")
-[ -f ${arch_mman} ] && egrep -q $regex ${arch_mman} &&
-(egrep $regex ${arch_mman} | \
- egrep -vw PROT_NONE | \
+[ -f ${arch_mman} ] && grep -E -q $regex ${arch_mman} &&
+(grep -E $regex ${arch_mman} | \
+ grep -E -vw PROT_NONE | \
sed -r "s/$regex/\2 \1 \1 \1 \2/g" | \
xargs printf "\t[ilog2(%s) + 1] = \"%s\",\n#ifndef ${prefix}_%s\n#define ${prefix}_%s %s\n#endif\n")
printf "};\n"
diff --git a/tools/perf/trace/beauty/mount_flags.sh b/tools/perf/trace/beauty/mount_flags.sh
index 847850b..730099a 100755
--- a/tools/perf/trace/beauty/mount_flags.sh
+++ b/tools/perf/trace/beauty/mount_flags.sh
@@ -5,11 +5,11 @@
printf "static const char *mount_flags[] = {\n"
regex='^[[:space:]]*#[[:space:]]*define[[:space:]]+MS_([[:alnum:]_]+)[[:space:]]+([[:digit:]]+)[[:space:]]*.*'
-egrep $regex ${header_dir}/mount.h | egrep -v '(MSK|VERBOSE|MGC_VAL)\>' | \
+grep -E $regex ${header_dir}/mount.h | grep -E -v '(MSK|VERBOSE|MGC_VAL)\>' | \
sed -r "s/$regex/\2 \2 \1/g" | sort -n | \
xargs printf "\t[%s ? (ilog2(%s) + 1) : 0] = \"%s\",\n"
regex='^[[:space:]]*#[[:space:]]*define[[:space:]]+MS_([[:alnum:]_]+)[[:space:]]+\(1<<([[:digit:]]+)\)[[:space:]]*.*'
-egrep $regex ${header_dir}/mount.h | \
+grep -E $regex ${header_dir}/mount.h | \
sed -r "s/$regex/\2 \1/g" | \
xargs printf "\t[%s + 1] = \"%s\",\n"
printf "};\n"
diff --git a/tools/perf/trace/beauty/move_mount_flags.sh b/tools/perf/trace/beauty/move_mount_flags.sh
index 4b1d9ac..32e552f 100755
--- a/tools/perf/trace/beauty/move_mount_flags.sh
+++ b/tools/perf/trace/beauty/move_mount_flags.sh
@@ -11,7 +11,7 @@
printf "static const char *move_mount_flags[] = {\n"
regex='^[[:space:]]*#[[:space:]]*define[[:space:]]+MOVE_MOUNT_([^_]+_[[:alnum:]_]+)[[:space:]]+(0x[[:xdigit:]]+)[[:space:]]*.*'
-egrep $regex ${linux_mount} | \
+grep -E $regex ${linux_mount} | \
sed -r "s/$regex/\2 \1/g" | \
xargs printf "\t[ilog2(%s) + 1] = \"%s\",\n"
printf "};\n"
diff --git a/tools/perf/trace/beauty/mremap_flags.sh b/tools/perf/trace/beauty/mremap_flags.sh
index d581823..4d01835 100755
--- a/tools/perf/trace/beauty/mremap_flags.sh
+++ b/tools/perf/trace/beauty/mremap_flags.sh
@@ -11,8 +11,8 @@
printf "static const char *mremap_flags[] = {\n"
regex='^[[:space:]]*#[[:space:]]*define[[:space:]]+MREMAP_([[:alnum:]_]+)[[:space:]]+((0x)?[[:xdigit:]]+)[[:space:]]*.*'
-egrep -q $regex ${linux_mman} && \
-(egrep $regex ${linux_mman} | \
+grep -E -q $regex ${linux_mman} && \
+(grep -E $regex ${linux_mman} | \
sed -r "s/$regex/\2 \1 \1 \1 \2/g" | \
xargs printf "\t[ilog2(%s) + 1] = \"%s\",\n#ifndef MREMAP_%s\n#define MREMAP_%s %s\n#endif\n")
printf "};\n"
diff --git a/tools/perf/trace/beauty/perf_event_open.c b/tools/perf/trace/beauty/perf_event_open.c
index 11d47db..01ee15f 100644
--- a/tools/perf/trace/beauty/perf_event_open.c
+++ b/tools/perf/trace/beauty/perf_event_open.c
@@ -44,3 +44,47 @@ static size_t syscall_arg__scnprintf_perf_flags(char *bf, size_t size,
}
#define SCA_PERF_FLAGS syscall_arg__scnprintf_perf_flags
+
+struct attr_fprintf_args {
+ size_t size, printed;
+ char *bf;
+ bool first;
+};
+
+static int attr__fprintf(FILE *fp __maybe_unused, const char *name, const char *val, void *priv)
+{
+ struct attr_fprintf_args *args = priv;
+ size_t printed = scnprintf(args->bf + args->printed , args->size - args->printed, "%s%s: %s", args->first ? "" : ", ", name, val);
+
+ args->first = false;
+ args->printed += printed;
+ return printed;
+}
+
+static size_t perf_event_attr___scnprintf(struct perf_event_attr *attr, char *bf, size_t size, bool show_zeros __maybe_unused)
+{
+ struct attr_fprintf_args args = {
+ .printed = scnprintf(bf, size, "{ "),
+ .size = size,
+ .first = true,
+ .bf = bf,
+ };
+
+ perf_event_attr__fprintf(stdout, attr, attr__fprintf, &args);
+ return args.printed + scnprintf(bf + args.printed, size - args.printed, " }");
+}
+
+static size_t syscall_arg__scnprintf_augmented_perf_event_attr(struct syscall_arg *arg, char *bf, size_t size)
+{
+ return perf_event_attr___scnprintf((void *)arg->augmented.args, bf, size, arg->trace->show_zeros);
+}
+
+static size_t syscall_arg__scnprintf_perf_event_attr(char *bf, size_t size, struct syscall_arg *arg)
+{
+ if (arg->augmented.args)
+ return syscall_arg__scnprintf_augmented_perf_event_attr(arg, bf, size);
+
+ return scnprintf(bf, size, "%#lx", arg->val);
+}
+
+#define SCA_PERF_ATTR syscall_arg__scnprintf_perf_event_attr
diff --git a/tools/perf/trace/beauty/perf_ioctl.sh b/tools/perf/trace/beauty/perf_ioctl.sh
index 9aabd97..06c2774 100755
--- a/tools/perf/trace/beauty/perf_ioctl.sh
+++ b/tools/perf/trace/beauty/perf_ioctl.sh
@@ -5,7 +5,7 @@
printf "static const char *perf_ioctl_cmds[] = {\n"
regex='^#[[:space:]]*define[[:space:]]+PERF_EVENT_IOC_(\w+)[[:space:]]+_IO[RW]*[[:space:]]*\([[:space:]]*.\$.[[:space:]]*,[[:space:]]*([[:digit:]]+).*'
-egrep $regex ${header_dir}/perf_event.h | \
+grep -E $regex ${header_dir}/perf_event.h | \
sed -r "s/$regex/\2 \1/g" | \
sort | xargs printf "\t[%s] = \"%s\",\n"
printf "};\n"
diff --git a/tools/perf/trace/beauty/pkey_alloc_access_rights.sh b/tools/perf/trace/beauty/pkey_alloc_access_rights.sh
index f8f1b56..74da888 100755
--- a/tools/perf/trace/beauty/pkey_alloc_access_rights.sh
+++ b/tools/perf/trace/beauty/pkey_alloc_access_rights.sh
@@ -5,7 +5,7 @@
printf "static const char *pkey_alloc_access_rights[] = {\n"
regex='^[[:space:]]*#[[:space:]]*define[[:space:]]+PKEY_([[:alnum:]_]+)[[:space:]]+(0x[[:xdigit:]]+)[[:space:]]*'
-egrep $regex ${header_dir}/mman-common.h | \
+grep -E $regex ${header_dir}/mman-common.h | \
sed -r "s/$regex/\2 \2 \1/g" | \
sort | xargs printf "\t[%s ? (ilog2(%s) + 1) : 0] = \"%s\",\n"
printf "};\n"
diff --git a/tools/perf/trace/beauty/prctl_option.sh b/tools/perf/trace/beauty/prctl_option.sh
index 3d27878..8059342 100755
--- a/tools/perf/trace/beauty/prctl_option.sh
+++ b/tools/perf/trace/beauty/prctl_option.sh
@@ -5,14 +5,14 @@
printf "static const char *prctl_options[] = {\n"
regex='^#define[[:space:]]{1}PR_(\w+)[[:space:]]*([[:xdigit:]]+)([[:space:]]*\/.*)?$'
-egrep $regex ${header_dir}/prctl.h | grep -v PR_SET_PTRACER | \
+grep -E $regex ${header_dir}/prctl.h | grep -v PR_SET_PTRACER | \
sed -r "s/$regex/\2 \1/g" | \
sort -n | xargs printf "\t[%s] = \"%s\",\n"
printf "};\n"
printf "static const char *prctl_set_mm_options[] = {\n"
regex='^#[[:space:]]+define[[:space:]]+PR_SET_MM_(\w+)[[:space:]]*([[:digit:]]+).*'
-egrep $regex ${header_dir}/prctl.h | \
+grep -E $regex ${header_dir}/prctl.h | \
sed -r "s/$regex/\2 \1/g" | \
sort -n | xargs printf "\t[%s] = \"%s\",\n"
printf "};\n"
diff --git a/tools/perf/trace/beauty/rename_flags.sh b/tools/perf/trace/beauty/rename_flags.sh
index 54c87c7..94bf7f4 100755
--- a/tools/perf/trace/beauty/rename_flags.sh
+++ b/tools/perf/trace/beauty/rename_flags.sh
@@ -8,8 +8,8 @@
printf "static const char *rename_flags[] = {\n"
regex='^[[:space:]]*#[[:space:]]*define[[:space:]]+RENAME_([[:alnum:]_]+)[[:space:]]+\(1[[:space:]]*<<[[:space:]]*([[:xdigit:]]+)[[:space:]]*\)[[:space:]]*.*'
-egrep -q $regex ${fs_header} && \
-(egrep $regex ${fs_header} | \
+grep -E -q $regex ${fs_header} && \
+(grep -E $regex ${fs_header} | \
sed -r "s/$regex/\2 \1/g" | \
xargs printf "\t[%d + 1] = \"%s\",\n")
printf "};\n"
diff --git a/tools/perf/trace/beauty/sockaddr.sh b/tools/perf/trace/beauty/sockaddr.sh
index 3820e5c..a59827e 100755
--- a/tools/perf/trace/beauty/sockaddr.sh
+++ b/tools/perf/trace/beauty/sockaddr.sh
@@ -17,8 +17,8 @@
# #define AF_LOCAL 1 /* POSIX name for AF_UNIX */
regex='^#define[[:space:]]+AF_(\w+)[[:space:]]+([[:digit:]]+).*'
-egrep $regex ${header_dir}/socket.h | \
+grep -E $regex ${header_dir}/socket.h | \
sed -r "s/$regex/\2 \1/g" | \
xargs printf "\t[%s] = \"%s\",\n" | \
- egrep -v "\"(UNIX|MAX)\""
+ grep -E -v "\"(UNIX|MAX)\""
printf "};\n"
diff --git a/tools/perf/trace/beauty/socket.sh b/tools/perf/trace/beauty/socket.sh
index 76330ac..8bc7ba62 100755
--- a/tools/perf/trace/beauty/socket.sh
+++ b/tools/perf/trace/beauty/socket.sh
@@ -12,7 +12,7 @@
printf "static const char *socket_ipproto[] = {\n"
ipproto_regex='^[[:space:]]+IPPROTO_(\w+)[[:space:]]+=[[:space:]]+([[:digit:]]+),.*'
-egrep $ipproto_regex ${uapi_header_dir}/in.h | \
+grep -E $ipproto_regex ${uapi_header_dir}/in.h | \
sed -r "s/$ipproto_regex/\2 \1/g" | \
sort -n | xargs printf "\t[%s] = \"%s\",\n"
printf "};\n\n"
@@ -20,7 +20,7 @@
printf "static const char *socket_level[] = {\n"
socket_level_regex='^#define[[:space:]]+SOL_(\w+)[[:space:]]+([[:digit:]]+)([[:space:]]+\/.*)?'
-egrep $socket_level_regex ${beauty_header_dir}/socket.h | \
+grep -E $socket_level_regex ${beauty_header_dir}/socket.h | \
sed -r "s/$socket_level_regex/\2 \1/g" | \
sort -n | xargs printf "\t[%s] = \"%s\",\n"
printf "};\n\n"
diff --git a/tools/perf/trace/beauty/sync_file_range.sh b/tools/perf/trace/beauty/sync_file_range.sh
index 7a9282d..90bf633 100755
--- a/tools/perf/trace/beauty/sync_file_range.sh
+++ b/tools/perf/trace/beauty/sync_file_range.sh
@@ -11,7 +11,7 @@
printf "static const char *sync_file_range_flags[] = {\n"
regex='^[[:space:]]*#[[:space:]]*define[[:space:]]+SYNC_FILE_RANGE_([[:alnum:]_]+)[[:space:]]+([[:xdigit:]]+)[[:space:]]*.*'
-egrep $regex ${linux_fs} | \
+grep -E $regex ${linux_fs} | \
sed -r "s/$regex/\2 \1/g" | \
xargs printf "\t[ilog2(%s) + 1] = \"%s\",\n"
printf "};\n"
diff --git a/tools/perf/trace/beauty/timespec.c b/tools/perf/trace/beauty/timespec.c
new file mode 100644
index 0000000..e1a61f0
--- /dev/null
+++ b/tools/perf/trace/beauty/timespec.c
@@ -0,0 +1,21 @@
+// SPDX-License-Identifier: LGPL-2.1
+// Copyright (C) 2022, Red Hat Inc, Arnaldo Carvalho de Melo <acme@redhat.com>
+
+#include "trace/beauty/beauty.h"
+#include <inttypes.h>
+#include <time.h>
+
+static size_t syscall_arg__scnprintf_augmented_timespec(struct syscall_arg *arg, char *bf, size_t size)
+{
+ struct timespec *ts = (struct timespec *)arg->augmented.args;
+
+ return scnprintf(bf, size, "{ .tv_sec: %" PRIu64 ", .tv_nsec: %" PRIu64 " }", ts->tv_sec, ts->tv_nsec);
+}
+
+size_t syscall_arg__scnprintf_timespec(char *bf, size_t size, struct syscall_arg *arg)
+{
+ if (arg->augmented.args)
+ return syscall_arg__scnprintf_augmented_timespec(arg, bf, size);
+
+ return scnprintf(bf, size, "%#lx", arg->val);
+}
diff --git a/tools/perf/trace/beauty/tracepoints/x86_irq_vectors.sh b/tools/perf/trace/beauty/tracepoints/x86_irq_vectors.sh
index f920003..eed9ce0 100755
--- a/tools/perf/trace/beauty/tracepoints/x86_irq_vectors.sh
+++ b/tools/perf/trace/beauty/tracepoints/x86_irq_vectors.sh
@@ -15,12 +15,12 @@
# the time of writing of this script was: IRQ_MOVE_CLEANUP_VECTOR.
first_external_regex='^#define[[:space:]]+FIRST_EXTERNAL_VECTOR[[:space:]]+(0x[[:xdigit:]]+)$'
-first_external_vector=$(egrep ${first_external_regex} ${x86_irq_vectors} | sed -r "s/${first_external_regex}/\1/g")
+first_external_vector=$(grep -E ${first_external_regex} ${x86_irq_vectors} | sed -r "s/${first_external_regex}/\1/g")
printf "static const char *x86_irq_vectors[] = {\n"
regex='^#define[[:space:]]+([[:alnum:]_]+)_VECTOR[[:space:]]+(0x[[:xdigit:]]+)$'
sed -r "s/FIRST_EXTERNAL_VECTOR/${first_external_vector}/g" ${x86_irq_vectors} | \
-egrep ${regex} | \
+grep -E ${regex} | \
sed -r "s/${regex}/\2 \1/g" | sort -n | \
xargs printf "\t[%s] = \"%s\",\n"
printf "};\n\n"
diff --git a/tools/perf/trace/beauty/tracepoints/x86_msr.sh b/tools/perf/trace/beauty/tracepoints/x86_msr.sh
index 9b0614a..0078689 100755
--- a/tools/perf/trace/beauty/tracepoints/x86_msr.sh
+++ b/tools/perf/trace/beauty/tracepoints/x86_msr.sh
@@ -15,7 +15,7 @@
printf "static const char *x86_MSRs[] = {\n"
regex='^[[:space:]]*#[[:space:]]*define[[:space:]]+MSR_([[:alnum:]][[:alnum:]_]+)[[:space:]]+(0x00000[[:xdigit:]]+)[[:space:]]*.*'
-egrep $regex ${x86_msr_index} | egrep -v 'MSR_(ATOM|P[46]|IA32_(TSC_DEADLINE|UCODE_REV)|IDT_FCR4)' | \
+grep -E $regex ${x86_msr_index} | grep -E -v 'MSR_(ATOM|P[46]|IA32_(TSC_DEADLINE|UCODE_REV)|IDT_FCR4)' | \
sed -r "s/$regex/\2 \1/g" | sort -n | \
xargs printf "\t[%s] = \"%s\",\n"
printf "};\n\n"
@@ -23,18 +23,18 @@
# Remove MSR_K6_WHCR, clashes with MSR_LSTAR
regex='^[[:space:]]*#[[:space:]]*define[[:space:]]+MSR_([[:alnum:]][[:alnum:]_]+)[[:space:]]+(0xc0000[[:xdigit:]]+)[[:space:]]*.*'
printf "#define x86_64_specific_MSRs_offset "
-egrep $regex ${x86_msr_index} | sed -r "s/$regex/\2/g" | sort -n | head -1
+grep -E $regex ${x86_msr_index} | sed -r "s/$regex/\2/g" | sort -n | head -1
printf "static const char *x86_64_specific_MSRs[] = {\n"
-egrep $regex ${x86_msr_index} | \
- sed -r "s/$regex/\2 \1/g" | egrep -vw 'K6_WHCR' | sort -n | \
+grep -E $regex ${x86_msr_index} | \
+ sed -r "s/$regex/\2 \1/g" | grep -E -vw 'K6_WHCR' | sort -n | \
xargs printf "\t[%s - x86_64_specific_MSRs_offset] = \"%s\",\n"
printf "};\n\n"
regex='^[[:space:]]*#[[:space:]]*define[[:space:]]+MSR_([[:alnum:]][[:alnum:]_]+)[[:space:]]+(0xc0010[[:xdigit:]]+)[[:space:]]*.*'
printf "#define x86_AMD_V_KVM_MSRs_offset "
-egrep $regex ${x86_msr_index} | sed -r "s/$regex/\2/g" | sort -n | head -1
+grep -E $regex ${x86_msr_index} | sed -r "s/$regex/\2/g" | sort -n | head -1
printf "static const char *x86_AMD_V_KVM_MSRs[] = {\n"
-egrep $regex ${x86_msr_index} | \
+grep -E $regex ${x86_msr_index} | \
sed -r "s/$regex/\2 \1/g" | sort -n | \
xargs printf "\t[%s - x86_AMD_V_KVM_MSRs_offset] = \"%s\",\n"
printf "};\n"
diff --git a/tools/perf/trace/beauty/usbdevfs_ioctl.sh b/tools/perf/trace/beauty/usbdevfs_ioctl.sh
index aa597ae..b39cfb3 100755
--- a/tools/perf/trace/beauty/usbdevfs_ioctl.sh
+++ b/tools/perf/trace/beauty/usbdevfs_ioctl.sh
@@ -8,14 +8,14 @@
printf "static const char *usbdevfs_ioctl_cmds[] = {\n"
regex="^#[[:space:]]*define[[:space:]]+USBDEVFS_(\w+)(\(\w+\))?[[:space:]]+_IO[CWR]{0,2}\([[:space:]]*(_IOC_\w+,[[:space:]]*)?'U'[[:space:]]*,[[:space:]]*([[:digit:]]+).*"
-egrep "$regex" ${header_dir}/usbdevice_fs.h | egrep -v 'USBDEVFS_\w+32[[:space:]]' | \
+grep -E "$regex" ${header_dir}/usbdevice_fs.h | grep -E -v 'USBDEVFS_\w+32[[:space:]]' | \
sed -r "s/$regex/\4 \1/g" | \
sort | xargs printf "\t[%s] = \"%s\",\n"
printf "};\n\n"
printf "#if 0\n"
printf "static const char *usbdevfs_ioctl_32_cmds[] = {\n"
regex="^#[[:space:]]*define[[:space:]]+USBDEVFS_(\w+)[[:space:]]+_IO[WR]{0,2}\([[:space:]]*'U'[[:space:]]*,[[:space:]]*([[:digit:]]+).*"
-egrep $regex ${header_dir}/usbdevice_fs.h | egrep 'USBDEVFS_\w+32[[:space:]]' | \
+grep -E $regex ${header_dir}/usbdevice_fs.h | grep -E 'USBDEVFS_\w+32[[:space:]]' | \
sed -r "s/$regex/\2 \1/g" | \
sort | xargs printf "\t[%s] = \"%s\",\n"
printf "};\n"
diff --git a/tools/perf/trace/beauty/vhost_virtio_ioctl.sh b/tools/perf/trace/beauty/vhost_virtio_ioctl.sh
index 439773d..2dd0a3b 100755
--- a/tools/perf/trace/beauty/vhost_virtio_ioctl.sh
+++ b/tools/perf/trace/beauty/vhost_virtio_ioctl.sh
@@ -5,14 +5,14 @@
printf "static const char *vhost_virtio_ioctl_cmds[] = {\n"
regex='^#[[:space:]]*define[[:space:]]+VHOST_(\w+)[[:space:]]+_IOW?\([[:space:]]*VHOST_VIRTIO[[:space:]]*,[[:space:]]*(0x[[:xdigit:]]+).*'
-egrep $regex ${header_dir}/vhost.h | \
+grep -E $regex ${header_dir}/vhost.h | \
sed -r "s/$regex/\2 \1/g" | \
sort | xargs printf "\t[%s] = \"%s\",\n"
printf "};\n"
printf "static const char *vhost_virtio_ioctl_read_cmds[] = {\n"
regex='^#[[:space:]]*define[[:space:]]+VHOST_(\w+)[[:space:]]+_IOW?R\([[:space:]]*VHOST_VIRTIO[[:space:]]*,[[:space:]]*(0x[[:xdigit:]]+).*'
-egrep $regex ${header_dir}/vhost.h | \
+grep -E $regex ${header_dir}/vhost.h | \
sed -r "s/$regex/\2 \1/g" | \
sort | xargs printf "\t[%s] = \"%s\",\n"
printf "};\n"
diff --git a/tools/perf/trace/beauty/x86_arch_prctl.sh b/tools/perf/trace/beauty/x86_arch_prctl.sh
index 7372d3c..57fa6aa 100755
--- a/tools/perf/trace/beauty/x86_arch_prctl.sh
+++ b/tools/perf/trace/beauty/x86_arch_prctl.sh
@@ -15,8 +15,8 @@
printf "static const char *x86_arch_prctl_codes_%d[] = {\n" $idx
regex=`printf '^[[:space:]]*#[[:space:]]*define[[:space:]]+ARCH_([[:alnum:]_]+)[[:space:]]+(%s[[:xdigit:]]+).*' ${prefix}`
fmt="\t[%#x - ${first_entry}]= \"%s\",\n"
- egrep -q $regex ${prctl_arch_header} && \
- (egrep $regex ${prctl_arch_header} | \
+ grep -E -q $regex ${prctl_arch_header} && \
+ (grep -E $regex ${prctl_arch_header} | \
sed -r "s/$regex/\2 \1/g" | \
xargs printf "$fmt")
printf "};\n\n"
diff --git a/tools/perf/ui/util.c b/tools/perf/ui/util.c
index 689b27c..1d38ddf 100644
--- a/tools/perf/ui/util.c
+++ b/tools/perf/ui/util.c
@@ -15,6 +15,9 @@ static int perf_stdio__error(const char *format, va_list args)
static int perf_stdio__warning(const char *format, va_list args)
{
+ if (quiet)
+ return 0;
+
fprintf(stderr, "Warning:\n");
vfprintf(stderr, format, args);
return 0;
@@ -45,6 +48,8 @@ int ui__warning(const char *format, ...)
{
int ret;
va_list args;
+ if (quiet)
+ return 0;
va_start(args, format);
ret = perf_eops->warning(format, args);
diff --git a/tools/perf/util/Build b/tools/perf/util/Build
index e315eca..79b9498 100644
--- a/tools/perf/util/Build
+++ b/tools/perf/util/Build
@@ -19,7 +19,6 @@
perf-y += evswitch.o
perf-y += find_bit.o
perf-y += get_current_dir_name.o
-perf-y += kallsyms.o
perf-y += levenshtein.o
perf-y += llvm-utils.o
perf-y += mmap.o
@@ -70,18 +69,19 @@
perf-y += comm.o
perf-y += thread.o
perf-y += thread_map.o
-perf-y += trace-event-parse.o
perf-y += parse-events-flex.o
perf-y += parse-events-bison.o
perf-y += pmu.o
+perf-y += pmus.o
perf-y += pmu-flex.o
perf-y += pmu-bison.o
perf-y += pmu-hybrid.o
-perf-y += trace-event-read.o
-perf-y += trace-event-info.o
-perf-y += trace-event-scripting.o
-perf-y += trace-event.o
perf-y += svghelper.o
+perf-$(CONFIG_LIBTRACEEVENT) += trace-event-info.o
+perf-$(CONFIG_LIBTRACEEVENT) += trace-event-scripting.o
+perf-$(CONFIG_LIBTRACEEVENT) += trace-event.o
+perf-$(CONFIG_LIBTRACEEVENT) += trace-event-parse.o
+perf-$(CONFIG_LIBTRACEEVENT) += trace-event-read.o
perf-y += sort.o
perf-y += hist.o
perf-y += util.o
@@ -126,6 +126,7 @@
perf-$(CONFIG_AUXTRACE) += cs-etm.o
perf-$(CONFIG_AUXTRACE) += cs-etm-decoder/
endif
+perf-$(CONFIG_AUXTRACE) += cs-etm-base.o
perf-y += parse-branch-options.o
perf-y += dump-insn.o
@@ -153,8 +154,12 @@
perf-$(CONFIG_PERF_BPF_SKEL) += bpf_counter_cgroup.o
perf-$(CONFIG_PERF_BPF_SKEL) += bpf_ftrace.o
perf-$(CONFIG_PERF_BPF_SKEL) += bpf_off_cpu.o
-perf-$(CONFIG_PERF_BPF_SKEL) += bpf_kwork.o
perf-$(CONFIG_PERF_BPF_SKEL) += bpf_lock_contention.o
+
+ifeq ($(CONFIG_LIBTRACEEVENT),y)
+ perf-$(CONFIG_PERF_BPF_SKEL) += bpf_kwork.o
+endif
+
perf-$(CONFIG_BPF_PROLOGUE) += bpf-prologue.o
perf-$(CONFIG_LIBELF) += symbol-elf.o
perf-$(CONFIG_LIBELF) += probe-file.o
@@ -189,7 +194,10 @@
perf-$(CONFIG_LIBUNWIND_X86) += libunwind/x86_32.o
perf-$(CONFIG_LIBUNWIND_AARCH64) += libunwind/arm64.o
-perf-$(CONFIG_LIBBABELTRACE) += data-convert-bt.o
+ifeq ($(CONFIG_LIBTRACEEVENT),y)
+ perf-$(CONFIG_LIBBABELTRACE) += data-convert-bt.o
+endif
+
perf-y += data-convert-json.o
perf-y += scripting-engines/
@@ -220,7 +228,7 @@
perf-$(CONFIG_LIBPFM4) += pfm.o
CFLAGS_config.o += -DETC_PERFCONFIG="BUILD_STR($(ETC_PERFCONFIG_SQ))"
-CFLAGS_llvm-utils.o += -DPERF_INCLUDE_DIR="BUILD_STR($(perf_include_dir_SQ))"
+CFLAGS_llvm-utils.o += -DLIBBPF_INCLUDE_DIR="BUILD_STR($(libbpf_include_dir_SQ))"
# avoid compiler warnings in 32-bit mode
CFLAGS_genelf_debug.o += -Wno-packed
@@ -294,10 +302,6 @@
CFLAGS_header.o += -include $(OUTPUT)PERF-VERSION-FILE
CFLAGS_arm-spe.o += -I$(srctree)/tools/arch/arm64/include/
-$(OUTPUT)util/kallsyms.o: ../lib/symbol/kallsyms.c FORCE
- $(call rule_mkdir)
- $(call if_changed_dep,cc_o_c)
-
$(OUTPUT)util/argv_split.o: ../lib/argv_split.c FORCE
$(call rule_mkdir)
$(call if_changed_dep,cc_o_c)
diff --git a/tools/perf/util/amd-sample-raw.c b/tools/perf/util/amd-sample-raw.c
index 2383058..b0e70ce 100644
--- a/tools/perf/util/amd-sample-raw.c
+++ b/tools/perf/util/amd-sample-raw.c
@@ -16,6 +16,7 @@
#include "evlist.h"
#include "sample-raw.h"
#include "pmu-events/pmu-events.h"
+#include "util/sample.h"
static u32 cpu_family, cpu_model, ibs_fetch_type, ibs_op_type;
static bool zen4_ibs_extensions;
diff --git a/tools/perf/util/arm64-frame-pointer-unwind-support.h b/tools/perf/util/arm64-frame-pointer-unwind-support.h
index 32af9ce..42d3a45 100644
--- a/tools/perf/util/arm64-frame-pointer-unwind-support.h
+++ b/tools/perf/util/arm64-frame-pointer-unwind-support.h
@@ -2,8 +2,10 @@
#ifndef __PERF_ARM_FRAME_POINTER_UNWIND_SUPPORT_H
#define __PERF_ARM_FRAME_POINTER_UNWIND_SUPPORT_H
-#include "event.h"
-#include "thread.h"
+#include <linux/types.h>
+
+struct perf_sample;
+struct thread;
u64 get_leaf_frame_caller_aarch64(struct perf_sample *sample, struct thread *thread, int user_idx);
diff --git a/tools/perf/util/auxtrace.c b/tools/perf/util/auxtrace.c
index 46ada5e..265d20c 100644
--- a/tools/perf/util/auxtrace.c
+++ b/tools/perf/util/auxtrace.c
@@ -59,6 +59,7 @@
#include <linux/ctype.h>
#include "symbol/kallsyms.h"
#include <internal/lib.h>
+#include "util/sample.h"
/*
* Make a group from 'leader' to 'last', requiring that the events were not
diff --git a/tools/perf/util/auxtrace.h b/tools/perf/util/auxtrace.h
index 6a0f9b9..2cf63d3 100644
--- a/tools/perf/util/auxtrace.h
+++ b/tools/perf/util/auxtrace.h
@@ -15,7 +15,7 @@
#include <linux/list.h>
#include <linux/perf_event.h>
#include <linux/types.h>
-#include <internal/cpumap.h>
+#include <perf/cpumap.h>
#include <asm/bitsperlong.h>
#include <asm/barrier.h>
diff --git a/tools/perf/util/bpf-loader.c b/tools/perf/util/bpf-loader.c
index a5dbd71..6e9b06c 100644
--- a/tools/perf/util/bpf-loader.c
+++ b/tools/perf/util/bpf-loader.c
@@ -27,11 +27,7 @@
#include "util.h"
#include "llvm-utils.h"
#include "c++/clang-c.h"
-#ifdef HAVE_LIBBPF_SUPPORT
-#include <bpf/hashmap.h>
-#else
#include "util/hashmap.h"
-#endif
#include "asm/bug.h"
#include <internal/xyarray.h>
diff --git a/tools/perf/util/bpf-prologue.h b/tools/perf/util/bpf-prologue.h
index c50c735..66dcf75 100644
--- a/tools/perf/util/bpf-prologue.h
+++ b/tools/perf/util/bpf-prologue.h
@@ -6,9 +6,8 @@
#ifndef __BPF_PROLOGUE_H
#define __BPF_PROLOGUE_H
-#include <linux/compiler.h>
-#include <linux/filter.h>
-#include "probe-event.h"
+struct probe_trace_arg;
+struct bpf_insn;
#define BPF_PROLOGUE_MAX_ARGS 3
#define BPF_PROLOGUE_START_ARG_REG BPF_REG_3
@@ -19,6 +18,7 @@ int bpf__gen_prologue(struct probe_trace_arg *args, int nargs,
struct bpf_insn *new_prog, size_t *new_cnt,
size_t cnt_space);
#else
+#include <linux/compiler.h>
#include <errno.h>
static inline int
diff --git a/tools/perf/util/bpf_counter.c b/tools/perf/util/bpf_counter.c
index ef1c15e..eeee899 100644
--- a/tools/perf/util/bpf_counter.c
+++ b/tools/perf/util/bpf_counter.c
@@ -561,7 +561,7 @@ static int bperf__load(struct evsel *evsel, struct target *target)
if (filter_type == BPERF_FILTER_PID ||
filter_type == BPERF_FILTER_TGID)
- key = evsel->core.threads->map[i].pid;
+ key = perf_thread_map__pid(evsel->core.threads, i);
else if (filter_type == BPERF_FILTER_CPU)
key = evsel->core.cpus->map[i].cpu;
else
diff --git a/tools/perf/util/bpf_kwork.c b/tools/perf/util/bpf_kwork.c
index b629dd6..6eb2c78 100644
--- a/tools/perf/util/bpf_kwork.c
+++ b/tools/perf/util/bpf_kwork.c
@@ -7,15 +7,18 @@
#include <time.h>
#include <fcntl.h>
+#include <signal.h>
#include <stdio.h>
#include <unistd.h>
#include <linux/time64.h>
#include "util/debug.h"
+#include "util/evsel.h"
#include "util/kwork.h"
#include <bpf/bpf.h>
+#include <perf/cpumap.h>
#include "util/bpf_skel/kwork_trace.skel.h"
diff --git a/tools/perf/util/bpf_lock_contention.c b/tools/perf/util/bpf_lock_contention.c
index fc4d613..8e1b791 100644
--- a/tools/perf/util/bpf_lock_contention.c
+++ b/tools/perf/util/bpf_lock_contention.c
@@ -5,6 +5,7 @@
#include "util/map.h"
#include "util/symbol.h"
#include "util/target.h"
+#include "util/thread.h"
#include "util/thread_map.h"
#include "util/lock-contention.h"
#include <linux/zalloc.h>
@@ -12,17 +13,10 @@
#include <bpf/bpf.h>
#include "bpf_skel/lock_contention.skel.h"
+#include "bpf_skel/lock_data.h"
static struct lock_contention_bpf *skel;
-struct lock_contention_data {
- u64 total_time;
- u64 min_time;
- u64 max_time;
- u32 count;
- u32 flags;
-};
-
int lock_contention_prepare(struct lock_contention *con)
{
int i, fd;
@@ -37,8 +31,16 @@ int lock_contention_prepare(struct lock_contention *con)
}
bpf_map__set_value_size(skel->maps.stacks, con->max_stack * sizeof(u64));
- bpf_map__set_max_entries(skel->maps.stacks, con->map_nr_entries);
bpf_map__set_max_entries(skel->maps.lock_stat, con->map_nr_entries);
+ bpf_map__set_max_entries(skel->maps.tstamp, con->map_nr_entries);
+
+ if (con->aggr_mode == LOCK_AGGR_TASK) {
+ bpf_map__set_max_entries(skel->maps.task_data, con->map_nr_entries);
+ bpf_map__set_max_entries(skel->maps.stacks, 1);
+ } else {
+ bpf_map__set_max_entries(skel->maps.task_data, 1);
+ bpf_map__set_max_entries(skel->maps.stacks, con->map_nr_entries);
+ }
if (target__has_cpu(target))
ncpus = perf_cpu_map__nr(evlist->core.user_requested_cpus);
@@ -88,7 +90,9 @@ int lock_contention_prepare(struct lock_contention *con)
bpf_map_update_elem(fd, &pid, &val, BPF_ANY);
}
+ /* these don't work well if in the rodata section */
skel->bss->stack_skip = con->stack_skip;
+ skel->bss->aggr_mode = con->aggr_mode;
lock_contention_bpf__attach(skel);
return 0;
@@ -108,28 +112,48 @@ int lock_contention_stop(void)
int lock_contention_read(struct lock_contention *con)
{
- int fd, stack;
- s32 prev_key, key;
- struct lock_contention_data data;
- struct lock_stat *st;
+ int fd, stack, task_fd, err = 0;
+ struct contention_key *prev_key, key;
+ struct contention_data data = {};
+ struct lock_stat *st = NULL;
struct machine *machine = con->machine;
- u64 stack_trace[con->max_stack];
+ u64 *stack_trace;
+ size_t stack_size = con->max_stack * sizeof(*stack_trace);
fd = bpf_map__fd(skel->maps.lock_stat);
stack = bpf_map__fd(skel->maps.stacks);
+ task_fd = bpf_map__fd(skel->maps.task_data);
con->lost = skel->bss->lost;
- prev_key = 0;
- while (!bpf_map_get_next_key(fd, &prev_key, &key)) {
+ stack_trace = zalloc(stack_size);
+ if (stack_trace == NULL)
+ return -1;
+
+ if (con->aggr_mode == LOCK_AGGR_TASK) {
+ struct thread *idle = __machine__findnew_thread(machine,
+ /*pid=*/0,
+ /*tid=*/0);
+ thread__set_comm(idle, "swapper", /*timestamp=*/0);
+ }
+
+ /* make sure it loads the kernel map */
+ map__load(maps__first(machine->kmaps));
+
+ prev_key = NULL;
+ while (!bpf_map_get_next_key(fd, prev_key, &key)) {
struct map *kmap;
struct symbol *sym;
int idx = 0;
+ s32 stack_id;
+
+ /* to handle errors in the loop body */
+ err = -1;
bpf_map_lookup_elem(fd, &key, &data);
st = zalloc(sizeof(*st));
if (st == NULL)
- return -1;
+ break;
st->nr_contended = data.count;
st->wait_time_total = data.total_time;
@@ -140,11 +164,34 @@ int lock_contention_read(struct lock_contention *con)
st->avg_wait_time = data.total_time / data.count;
st->flags = data.flags;
+ st->addr = key.aggr_key;
- bpf_map_lookup_elem(stack, &key, stack_trace);
+ if (con->aggr_mode == LOCK_AGGR_TASK) {
+ struct contention_task_data task;
+ struct thread *t;
+ int pid = key.aggr_key;
+
+ /* do not update idle comm which contains CPU number */
+ if (st->addr) {
+ bpf_map_lookup_elem(task_fd, &pid, &task);
+ t = __machine__findnew_thread(machine, /*pid=*/-1, pid);
+ thread__set_comm(t, task.comm, /*timestamp=*/0);
+ }
+ goto next;
+ }
+
+ if (con->aggr_mode == LOCK_AGGR_ADDR) {
+ sym = machine__find_kernel_symbol(machine, st->addr, &kmap);
+ if (sym)
+ st->name = strdup(sym->name);
+ goto next;
+ }
+
+ stack_id = key.aggr_key;
+ bpf_map_lookup_elem(stack, &stack_id, stack_trace);
/* skip lock internal functions */
- while (is_lock_function(machine, stack_trace[idx]) &&
+ while (machine__is_lock_function(machine, stack_trace[idx]) &&
idx < con->max_stack - 1)
idx++;
@@ -163,25 +210,32 @@ int lock_contention_read(struct lock_contention *con)
st->name = strdup(sym->name);
if (ret < 0 || st->name == NULL)
- return -1;
+ break;
} else if (asprintf(&st->name, "%#lx", (unsigned long)st->addr) < 0) {
- free(st);
- return -1;
+ break;
}
if (verbose) {
- st->callstack = memdup(stack_trace, sizeof(stack_trace));
- if (st->callstack == NULL) {
- free(st);
- return -1;
- }
+ st->callstack = memdup(stack_trace, stack_size);
+ if (st->callstack == NULL)
+ break;
}
-
+next:
hlist_add_head(&st->hash_entry, con->result);
- prev_key = key;
+ prev_key = &key;
+
+ /* we're fine now, reset the values */
+ st = NULL;
+ err = 0;
}
- return 0;
+ free(stack_trace);
+ if (st) {
+ free(st->name);
+ free(st);
+ }
+
+ return err;
}
int lock_contention_finish(void)
diff --git a/tools/perf/util/bpf_map.h b/tools/perf/util/bpf_map.h
index d6abd5e..c2f7c13 100644
--- a/tools/perf/util/bpf_map.h
+++ b/tools/perf/util/bpf_map.h
@@ -3,7 +3,6 @@
#define __PERF_BPF_MAP_H 1
#include <stdio.h>
-#include <linux/compiler.h>
struct bpf_map;
#ifdef HAVE_LIBBPF_SUPPORT
@@ -12,6 +11,8 @@ int bpf_map__fprintf(struct bpf_map *map, FILE *fp);
#else
+#include <linux/compiler.h>
+
static inline int bpf_map__fprintf(struct bpf_map *map __maybe_unused, FILE *fp __maybe_unused)
{
return 0;
diff --git a/tools/perf/util/bpf_off_cpu.c b/tools/perf/util/bpf_off_cpu.c
index c257813..01f70b8 100644
--- a/tools/perf/util/bpf_off_cpu.c
+++ b/tools/perf/util/bpf_off_cpu.c
@@ -102,7 +102,7 @@ static void check_sched_switch_args(void)
const struct btf_type *t1, *t2, *t3;
u32 type_id;
- type_id = btf__find_by_name_kind(btf, "bpf_trace_sched_switch",
+ type_id = btf__find_by_name_kind(btf, "btf_trace_sched_switch",
BTF_KIND_TYPEDEF);
if ((s32)type_id < 0)
return;
diff --git a/tools/perf/util/bpf_skel/lock_contention.bpf.c b/tools/perf/util/bpf_skel/lock_contention.bpf.c
index 1bb8628..11b0fc7 100644
--- a/tools/perf/util/bpf_skel/lock_contention.bpf.c
+++ b/tools/perf/util/bpf_skel/lock_contention.bpf.c
@@ -5,24 +5,11 @@
#include <bpf/bpf_tracing.h>
#include <bpf/bpf_core_read.h>
-/* maximum stack trace depth */
-#define MAX_STACKS 8
+#include "lock_data.h"
/* default buffer size */
#define MAX_ENTRIES 10240
-struct contention_key {
- __s32 stack_id;
-};
-
-struct contention_data {
- __u64 total_time;
- __u64 min_time;
- __u64 max_time;
- __u32 count;
- __u32 flags;
-};
-
struct tstamp_data {
__u64 timestamp;
__u64 lock;
@@ -34,16 +21,16 @@ struct tstamp_data {
struct {
__uint(type, BPF_MAP_TYPE_STACK_TRACE);
__uint(key_size, sizeof(__u32));
- __uint(value_size, MAX_STACKS * sizeof(__u64));
+ __uint(value_size, sizeof(__u64));
__uint(max_entries, MAX_ENTRIES);
} stacks SEC(".maps");
/* maintain timestamp at the beginning of contention */
struct {
- __uint(type, BPF_MAP_TYPE_TASK_STORAGE);
- __uint(map_flags, BPF_F_NO_PREALLOC);
+ __uint(type, BPF_MAP_TYPE_HASH);
__type(key, int);
__type(value, struct tstamp_data);
+ __uint(max_entries, MAX_ENTRIES);
} tstamp SEC(".maps");
/* actual lock contention statistics */
@@ -57,6 +44,13 @@ struct {
struct {
__uint(type, BPF_MAP_TYPE_HASH);
__uint(key_size, sizeof(__u32));
+ __uint(value_size, sizeof(struct contention_task_data));
+ __uint(max_entries, MAX_ENTRIES);
+} task_data SEC(".maps");
+
+struct {
+ __uint(type, BPF_MAP_TYPE_HASH);
+ __uint(key_size, sizeof(__u32));
__uint(value_size, sizeof(__u8));
__uint(max_entries, 1);
} cpu_filter SEC(".maps");
@@ -74,6 +68,9 @@ int has_cpu;
int has_task;
int stack_skip;
+/* determine the key of lock stat */
+int aggr_mode;
+
/* error stat */
int lost;
@@ -100,35 +97,62 @@ static inline int can_record(void)
return 1;
}
+static inline void update_task_data(__u32 pid)
+{
+ struct contention_task_data *p;
+
+ p = bpf_map_lookup_elem(&task_data, &pid);
+ if (p == NULL) {
+ struct contention_task_data data;
+
+ bpf_get_current_comm(data.comm, sizeof(data.comm));
+ bpf_map_update_elem(&task_data, &pid, &data, BPF_NOEXIST);
+ }
+}
+
SEC("tp_btf/contention_begin")
int contention_begin(u64 *ctx)
{
- struct task_struct *curr;
+ __u32 pid;
struct tstamp_data *pelem;
if (!enabled || !can_record())
return 0;
- curr = bpf_get_current_task_btf();
- pelem = bpf_task_storage_get(&tstamp, curr, NULL,
- BPF_LOCAL_STORAGE_GET_F_CREATE);
- if (!pelem || pelem->lock)
+ pid = bpf_get_current_pid_tgid();
+ pelem = bpf_map_lookup_elem(&tstamp, &pid);
+ if (pelem && pelem->lock)
return 0;
+ if (pelem == NULL) {
+ struct tstamp_data zero = {};
+
+ bpf_map_update_elem(&tstamp, &pid, &zero, BPF_ANY);
+ pelem = bpf_map_lookup_elem(&tstamp, &pid);
+ if (pelem == NULL) {
+ lost++;
+ return 0;
+ }
+ }
+
pelem->timestamp = bpf_ktime_get_ns();
pelem->lock = (__u64)ctx[0];
pelem->flags = (__u32)ctx[1];
- pelem->stack_id = bpf_get_stackid(ctx, &stacks, BPF_F_FAST_STACK_CMP | stack_skip);
- if (pelem->stack_id < 0)
- lost++;
+ if (aggr_mode == LOCK_AGGR_CALLER) {
+ pelem->stack_id = bpf_get_stackid(ctx, &stacks,
+ BPF_F_FAST_STACK_CMP | stack_skip);
+ if (pelem->stack_id < 0)
+ lost++;
+ }
+
return 0;
}
SEC("tp_btf/contention_end")
int contention_end(u64 *ctx)
{
- struct task_struct *curr;
+ __u32 pid;
struct tstamp_data *pelem;
struct contention_key key;
struct contention_data *data;
@@ -137,14 +161,29 @@ int contention_end(u64 *ctx)
if (!enabled)
return 0;
- curr = bpf_get_current_task_btf();
- pelem = bpf_task_storage_get(&tstamp, curr, NULL, 0);
+ pid = bpf_get_current_pid_tgid();
+ pelem = bpf_map_lookup_elem(&tstamp, &pid);
if (!pelem || pelem->lock != ctx[0])
return 0;
duration = bpf_ktime_get_ns() - pelem->timestamp;
- key.stack_id = pelem->stack_id;
+ switch (aggr_mode) {
+ case LOCK_AGGR_CALLER:
+ key.aggr_key = pelem->stack_id;
+ break;
+ case LOCK_AGGR_TASK:
+ key.aggr_key = pid;
+ update_task_data(pid);
+ break;
+ case LOCK_AGGR_ADDR:
+ key.aggr_key = pelem->lock;
+ break;
+ default:
+ /* should not happen */
+ return 0;
+ }
+
data = bpf_map_lookup_elem(&lock_stat, &key);
if (!data) {
struct contention_data first = {
@@ -156,7 +195,7 @@ int contention_end(u64 *ctx)
};
bpf_map_update_elem(&lock_stat, &key, &first, BPF_NOEXIST);
- pelem->lock = 0;
+ bpf_map_delete_elem(&tstamp, &pid);
return 0;
}
@@ -169,7 +208,7 @@ int contention_end(u64 *ctx)
if (data->min_time > duration)
data->min_time = duration;
- pelem->lock = 0;
+ bpf_map_delete_elem(&tstamp, &pid);
return 0;
}
diff --git a/tools/perf/util/bpf_skel/lock_data.h b/tools/perf/util/bpf_skel/lock_data.h
new file mode 100644
index 0000000..ce71cf1
--- /dev/null
+++ b/tools/perf/util/bpf_skel/lock_data.h
@@ -0,0 +1,30 @@
+// SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause)
+/* Data structures shared between BPF and tools. */
+#ifndef UTIL_BPF_SKEL_LOCK_DATA_H
+#define UTIL_BPF_SKEL_LOCK_DATA_H
+
+struct contention_key {
+ u64 aggr_key; /* can be stack_id, pid or lock addr */
+};
+
+#define TASK_COMM_LEN 16
+
+struct contention_task_data {
+ char comm[TASK_COMM_LEN];
+};
+
+struct contention_data {
+ u64 total_time;
+ u64 min_time;
+ u64 max_time;
+ u32 count;
+ u32 flags;
+};
+
+enum lock_aggr_mode {
+ LOCK_AGGR_ADDR = 0,
+ LOCK_AGGR_TASK,
+ LOCK_AGGR_CALLER,
+};
+
+#endif /* UTIL_BPF_SKEL_LOCK_DATA_H */
diff --git a/tools/perf/util/branch.h b/tools/perf/util/branch.h
index f838b23..3ed792d 100644
--- a/tools/perf/util/branch.h
+++ b/tools/perf/util/branch.h
@@ -7,12 +7,10 @@
* detected in at least musl libc, used in Alpine Linux. -acme
*/
#include <stdio.h>
-#include <stdint.h>
-#include <linux/compiler.h>
-#include <linux/stddef.h>
#include <linux/perf_event.h>
#include <linux/types.h>
-#include "event.h"
+#include "util/map_symbol.h"
+#include "util/sample.h"
struct branch_flags {
union {
@@ -24,9 +22,10 @@ struct branch_flags {
u64 abort:1;
u64 cycles:16;
u64 type:4;
+ u64 spec:2;
u64 new_type:4;
u64 priv:3;
- u64 reserved:33;
+ u64 reserved:31;
};
};
};
diff --git a/tools/perf/util/config.c b/tools/perf/util/config.c
index 3f2ae19a..658170b 100644
--- a/tools/perf/util/config.c
+++ b/tools/perf/util/config.c
@@ -556,7 +556,7 @@ static char *home_perfconfig(void)
config = strdup(mkpath("%s/.perfconfig", home));
if (config == NULL) {
- pr_warning("Not enough memory to process %s/.perfconfig, ignoring it.", home);
+ pr_warning("Not enough memory to process %s/.perfconfig, ignoring it.\n", home);
return NULL;
}
@@ -564,7 +564,7 @@ static char *home_perfconfig(void)
goto out_free;
if (st.st_uid && (st.st_uid != geteuid())) {
- pr_warning("File %s not owned by current user or root, ignoring it.", config);
+ pr_warning("File %s not owned by current user or root, ignoring it.\n", config);
goto out_free;
}
diff --git a/tools/perf/util/counts.c b/tools/perf/util/counts.c
index 7a447d9..11cd85b 100644
--- a/tools/perf/util/counts.c
+++ b/tools/perf/util/counts.c
@@ -48,7 +48,6 @@ void perf_counts__reset(struct perf_counts *counts)
{
xyarray__reset(counts->loaded);
xyarray__reset(counts->values);
- memset(&counts->aggr, 0, sizeof(struct perf_counts_values));
}
void evsel__reset_counts(struct evsel *evsel)
diff --git a/tools/perf/util/counts.h b/tools/perf/util/counts.h
index 5de2751..4276024 100644
--- a/tools/perf/util/counts.h
+++ b/tools/perf/util/counts.h
@@ -11,7 +11,6 @@ struct evsel;
struct perf_counts {
s8 scaled;
- struct perf_counts_values aggr;
struct xyarray *values;
struct xyarray *loaded;
};
diff --git a/tools/perf/util/cpumap.c b/tools/perf/util/cpumap.c
index 8486ca3b..5e56497 100644
--- a/tools/perf/util/cpumap.c
+++ b/tools/perf/util/cpumap.c
@@ -12,6 +12,7 @@
#include <linux/ctype.h>
#include <linux/zalloc.h>
+#include <internal/cpumap.h>
static struct perf_cpu max_cpu_num;
static struct perf_cpu max_present_cpu_num;
@@ -234,7 +235,7 @@ static int aggr_cpu_id__cmp(const void *a_pointer, const void *b_pointer)
struct cpu_aggr_map *cpu_aggr_map__new(const struct perf_cpu_map *cpus,
aggr_cpu_id_get_t get_id,
- void *data)
+ void *data, bool needs_sort)
{
int idx;
struct perf_cpu cpu;
@@ -270,8 +271,10 @@ struct cpu_aggr_map *cpu_aggr_map__new(const struct perf_cpu_map *cpus,
if (trimmed_c)
c = trimmed_c;
}
+
/* ensure we process id in increasing order */
- qsort(c->map, c->nr, sizeof(struct aggr_cpu_id), aggr_cpu_id__cmp);
+ if (needs_sort)
+ qsort(c->map, c->nr, sizeof(struct aggr_cpu_id), aggr_cpu_id__cmp);
return c;
@@ -354,6 +357,16 @@ struct aggr_cpu_id aggr_cpu_id__node(struct perf_cpu cpu, void *data __maybe_unu
return id;
}
+struct aggr_cpu_id aggr_cpu_id__global(struct perf_cpu cpu, void *data __maybe_unused)
+{
+ struct aggr_cpu_id id = aggr_cpu_id__empty();
+
+ /* it always aggregates to the cpu 0 */
+ cpu.cpu = 0;
+ id.cpu = cpu;
+ return id;
+}
+
/* setup simple routines to easily access node numbers given a cpu number */
static int get_max_num(char *path, int *max)
{
diff --git a/tools/perf/util/cpumap.h b/tools/perf/util/cpumap.h
index 4a6d029..c2f5824 100644
--- a/tools/perf/util/cpumap.h
+++ b/tools/perf/util/cpumap.h
@@ -4,8 +4,8 @@
#include <stdbool.h>
#include <stdio.h>
-#include <internal/cpumap.h>
#include <perf/cpumap.h>
+#include <linux/refcount.h>
/** Identify where counts are aggregated, -1 implies not to aggregate. */
struct aggr_cpu_id {
@@ -97,7 +97,7 @@ typedef struct aggr_cpu_id (*aggr_cpu_id_get_t)(struct perf_cpu cpu, void *data)
*/
struct cpu_aggr_map *cpu_aggr_map__new(const struct perf_cpu_map *cpus,
aggr_cpu_id_get_t get_id,
- void *data);
+ void *data, bool needs_sort);
bool aggr_cpu_id__equal(const struct aggr_cpu_id *a, const struct aggr_cpu_id *b);
bool aggr_cpu_id__is_empty(const struct aggr_cpu_id *a);
@@ -133,5 +133,9 @@ struct aggr_cpu_id aggr_cpu_id__cpu(struct perf_cpu cpu, void *data);
* cpu. The function signature is compatible with aggr_cpu_id_get_t.
*/
struct aggr_cpu_id aggr_cpu_id__node(struct perf_cpu cpu, void *data);
-
+/**
+ * aggr_cpu_id__global - Create an aggr_cpu_id for global aggregation.
+ * The function signature is compatible with aggr_cpu_id_get_t.
+ */
+struct aggr_cpu_id aggr_cpu_id__global(struct perf_cpu cpu, void *data);
#endif /* __PERF_CPUMAP_H */
diff --git a/tools/perf/util/cs-etm-base.c b/tools/perf/util/cs-etm-base.c
new file mode 100644
index 0000000..5975424
--- /dev/null
+++ b/tools/perf/util/cs-etm-base.c
@@ -0,0 +1,174 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * File for any parts of the Coresight decoding that don't require
+ * OpenCSD.
+ */
+
+#include <errno.h>
+#include <inttypes.h>
+
+#include "cs-etm.h"
+
+static const char * const cs_etm_global_header_fmts[] = {
+ [CS_HEADER_VERSION] = " Header version %llx\n",
+ [CS_PMU_TYPE_CPUS] = " PMU type/num cpus %llx\n",
+ [CS_ETM_SNAPSHOT] = " Snapshot %llx\n",
+};
+
+static const char * const cs_etm_priv_fmts[] = {
+ [CS_ETM_MAGIC] = " Magic number %llx\n",
+ [CS_ETM_CPU] = " CPU %lld\n",
+ [CS_ETM_NR_TRC_PARAMS] = " NR_TRC_PARAMS %llx\n",
+ [CS_ETM_ETMCR] = " ETMCR %llx\n",
+ [CS_ETM_ETMTRACEIDR] = " ETMTRACEIDR %llx\n",
+ [CS_ETM_ETMCCER] = " ETMCCER %llx\n",
+ [CS_ETM_ETMIDR] = " ETMIDR %llx\n",
+};
+
+static const char * const cs_etmv4_priv_fmts[] = {
+ [CS_ETM_MAGIC] = " Magic number %llx\n",
+ [CS_ETM_CPU] = " CPU %lld\n",
+ [CS_ETM_NR_TRC_PARAMS] = " NR_TRC_PARAMS %llx\n",
+ [CS_ETMV4_TRCCONFIGR] = " TRCCONFIGR %llx\n",
+ [CS_ETMV4_TRCTRACEIDR] = " TRCTRACEIDR %llx\n",
+ [CS_ETMV4_TRCIDR0] = " TRCIDR0 %llx\n",
+ [CS_ETMV4_TRCIDR1] = " TRCIDR1 %llx\n",
+ [CS_ETMV4_TRCIDR2] = " TRCIDR2 %llx\n",
+ [CS_ETMV4_TRCIDR8] = " TRCIDR8 %llx\n",
+ [CS_ETMV4_TRCAUTHSTATUS] = " TRCAUTHSTATUS %llx\n",
+ [CS_ETE_TRCDEVARCH] = " TRCDEVARCH %llx\n"
+};
+
+static const char * const param_unk_fmt =
+ " Unknown parameter [%d] %"PRIx64"\n";
+static const char * const magic_unk_fmt =
+ " Magic number Unknown %"PRIx64"\n";
+
+static int cs_etm__print_cpu_metadata_v0(u64 *val, int *offset)
+{
+ int i = *offset, j, nr_params = 0, fmt_offset;
+ u64 magic;
+
+ /* check magic value */
+ magic = val[i + CS_ETM_MAGIC];
+ if ((magic != __perf_cs_etmv3_magic) &&
+ (magic != __perf_cs_etmv4_magic)) {
+ /* failure - note bad magic value */
+ fprintf(stdout, magic_unk_fmt, magic);
+ return -EINVAL;
+ }
+
+ /* print common header block */
+ fprintf(stdout, cs_etm_priv_fmts[CS_ETM_MAGIC], val[i++]);
+ fprintf(stdout, cs_etm_priv_fmts[CS_ETM_CPU], val[i++]);
+
+ if (magic == __perf_cs_etmv3_magic) {
+ nr_params = CS_ETM_NR_TRC_PARAMS_V0;
+ fmt_offset = CS_ETM_ETMCR;
+ /* after common block, offset format index past NR_PARAMS */
+ for (j = fmt_offset; j < nr_params + fmt_offset; j++, i++)
+ fprintf(stdout, cs_etm_priv_fmts[j], val[i]);
+ } else if (magic == __perf_cs_etmv4_magic) {
+ nr_params = CS_ETMV4_NR_TRC_PARAMS_V0;
+ fmt_offset = CS_ETMV4_TRCCONFIGR;
+ /* after common block, offset format index past NR_PARAMS */
+ for (j = fmt_offset; j < nr_params + fmt_offset; j++, i++)
+ fprintf(stdout, cs_etmv4_priv_fmts[j], val[i]);
+ }
+ *offset = i;
+ return 0;
+}
+
+static int cs_etm__print_cpu_metadata_v1(u64 *val, int *offset)
+{
+ int i = *offset, j, total_params = 0;
+ u64 magic;
+
+ magic = val[i + CS_ETM_MAGIC];
+ /* total params to print is NR_PARAMS + common block size for v1 */
+ total_params = val[i + CS_ETM_NR_TRC_PARAMS] + CS_ETM_COMMON_BLK_MAX_V1;
+
+ if (magic == __perf_cs_etmv3_magic) {
+ for (j = 0; j < total_params; j++, i++) {
+ /* if newer record - could be excess params */
+ if (j >= CS_ETM_PRIV_MAX)
+ fprintf(stdout, param_unk_fmt, j, val[i]);
+ else
+ fprintf(stdout, cs_etm_priv_fmts[j], val[i]);
+ }
+ } else if (magic == __perf_cs_etmv4_magic || magic == __perf_cs_ete_magic) {
+ /*
+ * ETE and ETMv4 can be printed in the same block because the number of parameters
+ * is saved and they share the list of parameter names. ETE is also only supported
+ * in V1 files.
+ */
+ for (j = 0; j < total_params; j++, i++) {
+ /* if newer record - could be excess params */
+ if (j >= CS_ETE_PRIV_MAX)
+ fprintf(stdout, param_unk_fmt, j, val[i]);
+ else
+ fprintf(stdout, cs_etmv4_priv_fmts[j], val[i]);
+ }
+ } else {
+ /* failure - note bad magic value and error out */
+ fprintf(stdout, magic_unk_fmt, magic);
+ return -EINVAL;
+ }
+ *offset = i;
+ return 0;
+}
+
+static void cs_etm__print_auxtrace_info(u64 *val, int num)
+{
+ int i, cpu = 0, version, err;
+
+ version = val[0];
+
+ for (i = 0; i < CS_HEADER_VERSION_MAX; i++)
+ fprintf(stdout, cs_etm_global_header_fmts[i], val[i]);
+
+ for (i = CS_HEADER_VERSION_MAX; cpu < num; cpu++) {
+ if (version == 0)
+ err = cs_etm__print_cpu_metadata_v0(val, &i);
+ else if (version == 1)
+ err = cs_etm__print_cpu_metadata_v1(val, &i);
+ if (err)
+ return;
+ }
+}
+
+/*
+ * Do some basic checks and print the auxtrace info header before calling
+ * into cs_etm__process_auxtrace_info_full() which requires OpenCSD to be
+ * linked in. This allows some basic debugging if OpenCSD is missing.
+ */
+int cs_etm__process_auxtrace_info(union perf_event *event,
+ struct perf_session *session)
+{
+ struct perf_record_auxtrace_info *auxtrace_info = &event->auxtrace_info;
+ int event_header_size = sizeof(struct perf_event_header);
+ int num_cpu;
+ u64 *ptr = NULL;
+ u64 hdr_version;
+
+ if (auxtrace_info->header.size < (event_header_size + INFO_HEADER_SIZE))
+ return -EINVAL;
+
+ /* First the global part */
+ ptr = (u64 *) auxtrace_info->priv;
+
+ /* Look for version of the header */
+ hdr_version = ptr[0];
+ if (hdr_version > CS_HEADER_CURRENT_VERSION) {
+ pr_err("\nCS ETM Trace: Unknown Header Version = %#" PRIx64, hdr_version);
+ pr_err(", version supported <= %x\n", CS_HEADER_CURRENT_VERSION);
+ return -EINVAL;
+ }
+
+ if (dump_trace) {
+ num_cpu = ptr[CS_PMU_TYPE_CPUS] & 0xffffffff;
+ cs_etm__print_auxtrace_info(ptr, num_cpu);
+ }
+
+ return cs_etm__process_auxtrace_info_full(event, session);
+}
diff --git a/tools/perf/util/cs-etm.c b/tools/perf/util/cs-etm.c
index 16db965..33303d0 100644
--- a/tools/perf/util/cs-etm.c
+++ b/tools/perf/util/cs-etm.c
@@ -2510,141 +2510,6 @@ static bool cs_etm__is_timeless_decoding(struct cs_etm_auxtrace *etm)
return timeless_decoding;
}
-static const char * const cs_etm_global_header_fmts[] = {
- [CS_HEADER_VERSION] = " Header version %llx\n",
- [CS_PMU_TYPE_CPUS] = " PMU type/num cpus %llx\n",
- [CS_ETM_SNAPSHOT] = " Snapshot %llx\n",
-};
-
-static const char * const cs_etm_priv_fmts[] = {
- [CS_ETM_MAGIC] = " Magic number %llx\n",
- [CS_ETM_CPU] = " CPU %lld\n",
- [CS_ETM_NR_TRC_PARAMS] = " NR_TRC_PARAMS %llx\n",
- [CS_ETM_ETMCR] = " ETMCR %llx\n",
- [CS_ETM_ETMTRACEIDR] = " ETMTRACEIDR %llx\n",
- [CS_ETM_ETMCCER] = " ETMCCER %llx\n",
- [CS_ETM_ETMIDR] = " ETMIDR %llx\n",
-};
-
-static const char * const cs_etmv4_priv_fmts[] = {
- [CS_ETM_MAGIC] = " Magic number %llx\n",
- [CS_ETM_CPU] = " CPU %lld\n",
- [CS_ETM_NR_TRC_PARAMS] = " NR_TRC_PARAMS %llx\n",
- [CS_ETMV4_TRCCONFIGR] = " TRCCONFIGR %llx\n",
- [CS_ETMV4_TRCTRACEIDR] = " TRCTRACEIDR %llx\n",
- [CS_ETMV4_TRCIDR0] = " TRCIDR0 %llx\n",
- [CS_ETMV4_TRCIDR1] = " TRCIDR1 %llx\n",
- [CS_ETMV4_TRCIDR2] = " TRCIDR2 %llx\n",
- [CS_ETMV4_TRCIDR8] = " TRCIDR8 %llx\n",
- [CS_ETMV4_TRCAUTHSTATUS] = " TRCAUTHSTATUS %llx\n",
- [CS_ETE_TRCDEVARCH] = " TRCDEVARCH %llx\n"
-};
-
-static const char * const param_unk_fmt =
- " Unknown parameter [%d] %llx\n";
-static const char * const magic_unk_fmt =
- " Magic number Unknown %llx\n";
-
-static int cs_etm__print_cpu_metadata_v0(__u64 *val, int *offset)
-{
- int i = *offset, j, nr_params = 0, fmt_offset;
- __u64 magic;
-
- /* check magic value */
- magic = val[i + CS_ETM_MAGIC];
- if ((magic != __perf_cs_etmv3_magic) &&
- (magic != __perf_cs_etmv4_magic)) {
- /* failure - note bad magic value */
- fprintf(stdout, magic_unk_fmt, magic);
- return -EINVAL;
- }
-
- /* print common header block */
- fprintf(stdout, cs_etm_priv_fmts[CS_ETM_MAGIC], val[i++]);
- fprintf(stdout, cs_etm_priv_fmts[CS_ETM_CPU], val[i++]);
-
- if (magic == __perf_cs_etmv3_magic) {
- nr_params = CS_ETM_NR_TRC_PARAMS_V0;
- fmt_offset = CS_ETM_ETMCR;
- /* after common block, offset format index past NR_PARAMS */
- for (j = fmt_offset; j < nr_params + fmt_offset; j++, i++)
- fprintf(stdout, cs_etm_priv_fmts[j], val[i]);
- } else if (magic == __perf_cs_etmv4_magic) {
- nr_params = CS_ETMV4_NR_TRC_PARAMS_V0;
- fmt_offset = CS_ETMV4_TRCCONFIGR;
- /* after common block, offset format index past NR_PARAMS */
- for (j = fmt_offset; j < nr_params + fmt_offset; j++, i++)
- fprintf(stdout, cs_etmv4_priv_fmts[j], val[i]);
- }
- *offset = i;
- return 0;
-}
-
-static int cs_etm__print_cpu_metadata_v1(__u64 *val, int *offset)
-{
- int i = *offset, j, total_params = 0;
- __u64 magic;
-
- magic = val[i + CS_ETM_MAGIC];
- /* total params to print is NR_PARAMS + common block size for v1 */
- total_params = val[i + CS_ETM_NR_TRC_PARAMS] + CS_ETM_COMMON_BLK_MAX_V1;
-
- if (magic == __perf_cs_etmv3_magic) {
- for (j = 0; j < total_params; j++, i++) {
- /* if newer record - could be excess params */
- if (j >= CS_ETM_PRIV_MAX)
- fprintf(stdout, param_unk_fmt, j, val[i]);
- else
- fprintf(stdout, cs_etm_priv_fmts[j], val[i]);
- }
- } else if (magic == __perf_cs_etmv4_magic || magic == __perf_cs_ete_magic) {
- /*
- * ETE and ETMv4 can be printed in the same block because the number of parameters
- * is saved and they share the list of parameter names. ETE is also only supported
- * in V1 files.
- */
- for (j = 0; j < total_params; j++, i++) {
- /* if newer record - could be excess params */
- if (j >= CS_ETE_PRIV_MAX)
- fprintf(stdout, param_unk_fmt, j, val[i]);
- else
- fprintf(stdout, cs_etmv4_priv_fmts[j], val[i]);
- }
- } else {
- /* failure - note bad magic value and error out */
- fprintf(stdout, magic_unk_fmt, magic);
- return -EINVAL;
- }
- *offset = i;
- return 0;
-}
-
-static void cs_etm__print_auxtrace_info(__u64 *val, int num)
-{
- int i, cpu = 0, version, err;
-
- /* bail out early on bad header version */
- version = val[0];
- if (version > CS_HEADER_CURRENT_VERSION) {
- /* failure.. return */
- fprintf(stdout, " Unknown Header Version = %x, ", version);
- fprintf(stdout, "Version supported <= %x\n", CS_HEADER_CURRENT_VERSION);
- return;
- }
-
- for (i = 0; i < CS_HEADER_VERSION_MAX; i++)
- fprintf(stdout, cs_etm_global_header_fmts[i], val[i]);
-
- for (i = CS_HEADER_VERSION_MAX; cpu < num; cpu++) {
- if (version == 0)
- err = cs_etm__print_cpu_metadata_v0(val, &i);
- else if (version == 1)
- err = cs_etm__print_cpu_metadata_v1(val, &i);
- if (err)
- return;
- }
-}
-
/*
* Read a single cpu parameter block from the auxtrace_info priv block.
*
@@ -2881,57 +2746,20 @@ static int cs_etm__queue_aux_records(struct perf_session *session)
return 0;
}
-int cs_etm__process_auxtrace_info(union perf_event *event,
- struct perf_session *session)
+int cs_etm__process_auxtrace_info_full(union perf_event *event,
+ struct perf_session *session)
{
struct perf_record_auxtrace_info *auxtrace_info = &event->auxtrace_info;
struct cs_etm_auxtrace *etm = NULL;
struct int_node *inode;
- unsigned int pmu_type;
int event_header_size = sizeof(struct perf_event_header);
- int info_header_size;
int total_size = auxtrace_info->header.size;
int priv_size = 0;
int num_cpu, trcidr_idx;
int err = 0;
int i, j;
- u64 *ptr, *hdr = NULL;
+ u64 *ptr = NULL;
u64 **metadata = NULL;
- u64 hdr_version;
-
- /*
- * sizeof(auxtrace_info_event::type) +
- * sizeof(auxtrace_info_event::reserved) == 8
- */
- info_header_size = 8;
-
- if (total_size < (event_header_size + info_header_size))
- return -EINVAL;
-
- priv_size = total_size - event_header_size - info_header_size;
-
- /* First the global part */
- ptr = (u64 *) auxtrace_info->priv;
-
- /* Look for version of the header */
- hdr_version = ptr[0];
- if (hdr_version > CS_HEADER_CURRENT_VERSION) {
- /* print routine will print an error on bad version */
- if (dump_trace)
- cs_etm__print_auxtrace_info(auxtrace_info->priv, 0);
- return -EINVAL;
- }
-
- hdr = zalloc(sizeof(*hdr) * CS_HEADER_VERSION_MAX);
- if (!hdr)
- return -ENOMEM;
-
- /* Extract header information - see cs-etm.h for format */
- for (i = 0; i < CS_HEADER_VERSION_MAX; i++)
- hdr[i] = ptr[i];
- num_cpu = hdr[CS_PMU_TYPE_CPUS] & 0xffffffff;
- pmu_type = (unsigned int) ((hdr[CS_PMU_TYPE_CPUS] >> 32) &
- 0xffffffff);
/*
* Create an RB tree for traceID-metadata tuple. Since the conversion
@@ -2939,17 +2767,21 @@ int cs_etm__process_auxtrace_info(union perf_event *event,
* in anything other than a sequential array is worth doing.
*/
traceid_list = intlist__new(NULL);
- if (!traceid_list) {
- err = -ENOMEM;
- goto err_free_hdr;
- }
+ if (!traceid_list)
+ return -ENOMEM;
+ /* First the global part */
+ ptr = (u64 *) auxtrace_info->priv;
+ num_cpu = ptr[CS_PMU_TYPE_CPUS] & 0xffffffff;
metadata = zalloc(sizeof(*metadata) * num_cpu);
if (!metadata) {
err = -ENOMEM;
goto err_free_traceid_list;
}
+ /* Start parsing after the common part of the header */
+ i = CS_HEADER_VERSION_MAX;
+
/*
* The metadata is stored in the auxtrace_info section and encodes
* the configuration of the ARM embedded trace macrocell which is
@@ -3019,6 +2851,7 @@ int cs_etm__process_auxtrace_info(union perf_event *event,
* The following tests if the correct number of double words was
* present in the auxtrace info section.
*/
+ priv_size = total_size - event_header_size - INFO_HEADER_SIZE;
if (i * 8 != priv_size) {
err = -EINVAL;
goto err_free_metadata;
@@ -3047,8 +2880,8 @@ int cs_etm__process_auxtrace_info(union perf_event *event,
etm->machine = &session->machines.host;
etm->num_cpu = num_cpu;
- etm->pmu_type = pmu_type;
- etm->snapshot_mode = (hdr[CS_ETM_SNAPSHOT] != 0);
+ etm->pmu_type = (unsigned int) ((ptr[CS_PMU_TYPE_CPUS] >> 32) & 0xffffffff);
+ etm->snapshot_mode = (ptr[CS_ETM_SNAPSHOT] != 0);
etm->metadata = metadata;
etm->auxtrace_type = auxtrace_info->type;
etm->timeless_decoding = cs_etm__is_timeless_decoding(etm);
@@ -3082,10 +2915,6 @@ int cs_etm__process_auxtrace_info(union perf_event *event,
goto err_delete_thread;
}
- if (dump_trace) {
- cs_etm__print_auxtrace_info(auxtrace_info->priv, num_cpu);
- }
-
err = cs_etm__synth_events(etm, session);
if (err)
goto err_delete_thread;
@@ -3119,14 +2948,5 @@ int cs_etm__process_auxtrace_info(union perf_event *event,
zfree(&metadata);
err_free_traceid_list:
intlist__delete(traceid_list);
-err_free_hdr:
- zfree(&hdr);
- /*
- * At this point, as a minimum we have valid header. Dump the rest of
- * the info section - the print routines will error out on structural
- * issues.
- */
- if (dump_trace)
- cs_etm__print_auxtrace_info(auxtrace_info->priv, num_cpu);
return err;
}
diff --git a/tools/perf/util/cs-etm.h b/tools/perf/util/cs-etm.h
index 90c83f9..5da50d5 100644
--- a/tools/perf/util/cs-etm.h
+++ b/tools/perf/util/cs-etm.h
@@ -7,6 +7,7 @@
#ifndef INCLUDE__UTIL_PERF_CS_ETM_H__
#define INCLUDE__UTIL_PERF_CS_ETM_H__
+#include "debug.h"
#include "util/event.h"
#include <linux/bits.h>
@@ -201,9 +202,13 @@ struct cs_etm_packet_queue {
#define CS_ETMV4_PRIV_SIZE (CS_ETMV4_PRIV_MAX * sizeof(u64))
#define CS_ETE_PRIV_SIZE (CS_ETE_PRIV_MAX * sizeof(u64))
-#ifdef HAVE_CSTRACE_SUPPORT
+#define INFO_HEADER_SIZE (sizeof(((struct perf_record_auxtrace_info *)0)->type) + \
+ sizeof(((struct perf_record_auxtrace_info *)0)->reserved__))
+
int cs_etm__process_auxtrace_info(union perf_event *event,
struct perf_session *session);
+
+#ifdef HAVE_CSTRACE_SUPPORT
int cs_etm__get_cpu(u8 trace_chan_id, int *cpu);
int cs_etm__get_pid_fmt(u8 trace_chan_id, u64 *pid_fmt);
int cs_etm__etmq_set_tid(struct cs_etm_queue *etmq,
@@ -213,45 +218,16 @@ void cs_etm__etmq_set_traceid_queue_timestamp(struct cs_etm_queue *etmq,
u8 trace_chan_id);
struct cs_etm_packet_queue
*cs_etm__etmq_get_packet_queue(struct cs_etm_queue *etmq, u8 trace_chan_id);
+int cs_etm__process_auxtrace_info_full(union perf_event *event __maybe_unused,
+ struct perf_session *session __maybe_unused);
#else
static inline int
-cs_etm__process_auxtrace_info(union perf_event *event __maybe_unused,
- struct perf_session *session __maybe_unused)
+cs_etm__process_auxtrace_info_full(union perf_event *event __maybe_unused,
+ struct perf_session *session __maybe_unused)
{
+ pr_err("\nCS ETM Trace: OpenCSD is not linked in, please recompile with CORESIGHT=1\n");
return -1;
}
-
-static inline int cs_etm__get_cpu(u8 trace_chan_id __maybe_unused,
- int *cpu __maybe_unused)
-{
- return -1;
-}
-
-static inline int cs_etm__etmq_set_tid(
- struct cs_etm_queue *etmq __maybe_unused,
- pid_t tid __maybe_unused,
- u8 trace_chan_id __maybe_unused)
-{
- return -1;
-}
-
-static inline bool cs_etm__etmq_is_timeless(
- struct cs_etm_queue *etmq __maybe_unused)
-{
- /* What else to return? */
- return true;
-}
-
-static inline void cs_etm__etmq_set_traceid_queue_timestamp(
- struct cs_etm_queue *etmq __maybe_unused,
- u8 trace_chan_id __maybe_unused) {}
-
-static inline struct cs_etm_packet_queue *cs_etm__etmq_get_packet_queue(
- struct cs_etm_queue *etmq __maybe_unused,
- u8 trace_chan_id __maybe_unused)
-{
- return NULL;
-}
#endif
#endif
diff --git a/tools/perf/util/data-convert-bt.c b/tools/perf/util/data-convert-bt.c
index 9e0aee2..b842273 100644
--- a/tools/perf/util/data-convert-bt.c
+++ b/tools/perf/util/data-convert-bt.c
@@ -19,7 +19,6 @@
#include <babeltrace/ctf-writer/event-fields.h>
#include <babeltrace/ctf-ir/utils.h>
#include <babeltrace/ctf/events.h>
-#include <traceevent/event-parse.h>
#include "asm/bug.h"
#include "data-convert.h"
#include "session.h"
@@ -34,6 +33,11 @@
#include <linux/time64.h>
#include "util.h"
#include "clockid.h"
+#include "util/sample.h"
+
+#ifdef HAVE_LIBTRACEEVENT
+#include <traceevent/event-parse.h>
+#endif
#define pr_N(n, fmt, ...) \
eprintf(n, debug_data_convert, fmt, ##__VA_ARGS__)
@@ -318,8 +322,10 @@ static int add_tracepoint_field_value(struct ctf_writer *cw,
offset = tmp_val;
len = offset >> 16;
offset &= 0xffff;
+#ifdef HAVE_LIBTRACEEVENT_TEP_FIELD_IS_RELATIVE
if (flags & TEP_FIELD_IS_RELATIVE)
offset += fmtf->offset + fmtf->size;
+#endif
}
if (flags & TEP_FIELD_IS_ARRAY) {
diff --git a/tools/perf/util/data-convert-json.c b/tools/perf/util/data-convert-json.c
index 613d6ae..ba9d93c 100644
--- a/tools/perf/util/data-convert-json.c
+++ b/tools/perf/util/data-convert-json.c
@@ -27,6 +27,10 @@
#include "util/thread.h"
#include "util/tool.h"
+#ifdef HAVE_LIBTRACEEVENT
+#include <traceevent/event-parse.h>
+#endif
+
struct convert_json {
struct perf_tool tool;
FILE *out;
@@ -217,6 +221,27 @@ static int process_sample_event(struct perf_tool *tool,
}
output_json_format(out, false, 3, "]");
+#ifdef HAVE_LIBTRACEEVENT
+ if (sample->raw_data) {
+ int i;
+ struct tep_format_field **fields;
+
+ fields = tep_event_fields(evsel->tp_format);
+ if (fields) {
+ i = 0;
+ while (fields[i]) {
+ struct trace_seq s;
+
+ trace_seq_init(&s);
+ tep_print_field(&s, sample->raw_data, fields[i]);
+ output_json_key_string(out, true, 3, fields[i]->name, s.buffer);
+
+ i++;
+ }
+ free(fields);
+ }
+ }
+#endif
output_json_format(out, false, 2, "}");
return 0;
}
@@ -293,7 +318,9 @@ int bt_convert__perf2json(const char *input_name, const char *output_name,
.exit = perf_event__process_exit,
.fork = perf_event__process_fork,
.lost = perf_event__process_lost,
+#ifdef HAVE_LIBTRACEEVENT
.tracing_data = perf_event__process_tracing_data,
+#endif
.build_id = perf_event__process_build_id,
.id_index = perf_event__process_id_index,
.auxtrace_info = perf_event__process_auxtrace_info,
diff --git a/tools/perf/util/dwarf-aux.c b/tools/perf/util/dwarf-aux.c
index 609ca167..b0741440 100644
--- a/tools/perf/util/dwarf-aux.c
+++ b/tools/perf/util/dwarf-aux.c
@@ -123,7 +123,7 @@ int cu_find_lineinfo(Dwarf_Die *cu_die, Dwarf_Addr addr,
if (die_find_realfunc(cu_die, addr, &die_mem)
&& die_entrypc(&die_mem, &faddr) == 0 &&
faddr == addr) {
- *fname = dwarf_decl_file(&die_mem);
+ *fname = die_get_decl_file(&die_mem);
dwarf_decl_line(&die_mem, lineno);
goto out;
}
@@ -137,7 +137,7 @@ int cu_find_lineinfo(Dwarf_Die *cu_die, Dwarf_Addr addr,
}
out:
- return *lineno ?: -ENOENT;
+ return (*lineno && *fname) ? *lineno : -ENOENT;
}
static int __die_find_inline_cb(Dwarf_Die *die_mem, void *data);
@@ -308,26 +308,13 @@ static int die_get_attr_udata(Dwarf_Die *tp_die, unsigned int attr_name,
{
Dwarf_Attribute attr;
- if (dwarf_attr(tp_die, attr_name, &attr) == NULL ||
+ if (dwarf_attr_integrate(tp_die, attr_name, &attr) == NULL ||
dwarf_formudata(&attr, result) != 0)
return -ENOENT;
return 0;
}
-/* Get attribute and translate it as a sdata */
-static int die_get_attr_sdata(Dwarf_Die *tp_die, unsigned int attr_name,
- Dwarf_Sword *result)
-{
- Dwarf_Attribute attr;
-
- if (dwarf_attr(tp_die, attr_name, &attr) == NULL ||
- dwarf_formsdata(&attr, result) != 0)
- return -ENOENT;
-
- return 0;
-}
-
/**
* die_is_signed_type - Check whether a type DIE is signed or not
* @tp_die: a DIE of a type
@@ -467,9 +454,9 @@ int die_get_data_member_location(Dwarf_Die *mb_die, Dwarf_Word *offs)
/* Get the call file index number in CU DIE */
static int die_get_call_fileno(Dwarf_Die *in_die)
{
- Dwarf_Sword idx;
+ Dwarf_Word idx;
- if (die_get_attr_sdata(in_die, DW_AT_call_file, &idx) == 0)
+ if (die_get_attr_udata(in_die, DW_AT_call_file, &idx) == 0)
return (int)idx;
else
return -ENOENT;
@@ -478,14 +465,27 @@ static int die_get_call_fileno(Dwarf_Die *in_die)
/* Get the declared file index number in CU DIE */
static int die_get_decl_fileno(Dwarf_Die *pdie)
{
- Dwarf_Sword idx;
+ Dwarf_Word idx;
- if (die_get_attr_sdata(pdie, DW_AT_decl_file, &idx) == 0)
+ if (die_get_attr_udata(pdie, DW_AT_decl_file, &idx) == 0)
return (int)idx;
else
return -ENOENT;
}
+/* Return the file name by index */
+static const char *die_get_file_name(Dwarf_Die *dw_die, int idx)
+{
+ Dwarf_Die cu_die;
+ Dwarf_Files *files;
+
+ if (idx < 0 || !dwarf_diecu(dw_die, &cu_die, NULL, NULL) ||
+ dwarf_getsrcfiles(&cu_die, &files, NULL) != 0)
+ return NULL;
+
+ return dwarf_filesrc(files, idx, NULL, NULL);
+}
+
/**
* die_get_call_file - Get callsite file name of inlined function instance
* @in_die: a DIE of an inlined function instance
@@ -495,18 +495,22 @@ static int die_get_decl_fileno(Dwarf_Die *pdie)
*/
const char *die_get_call_file(Dwarf_Die *in_die)
{
- Dwarf_Die cu_die;
- Dwarf_Files *files;
- int idx;
-
- idx = die_get_call_fileno(in_die);
- if (idx < 0 || !dwarf_diecu(in_die, &cu_die, NULL, NULL) ||
- dwarf_getsrcfiles(&cu_die, &files, NULL) != 0)
- return NULL;
-
- return dwarf_filesrc(files, idx, NULL, NULL);
+ return die_get_file_name(in_die, die_get_call_fileno(in_die));
}
+/**
+ * die_get_decl_file - Find the declared file name of this DIE
+ * @dw_die: a DIE for something declared.
+ *
+ * Get declared file name of @dw_die.
+ * NOTE: Since some version of clang DWARF5 implementation incorrectly uses
+ * file index 0 for DW_AT_decl_file, die_get_decl_file() will return NULL for
+ * such cases. Use this function instead.
+ */
+const char *die_get_decl_file(Dwarf_Die *dw_die)
+{
+ return die_get_file_name(dw_die, die_get_decl_fileno(dw_die));
+}
/**
* die_find_child - Generic DIE search function in DIE tree
@@ -790,7 +794,7 @@ static int __die_walk_funclines_cb(Dwarf_Die *in_die, void *data)
}
if (addr) {
- fname = dwarf_decl_file(in_die);
+ fname = die_get_decl_file(in_die);
if (fname && dwarf_decl_line(in_die, &lineno) == 0) {
lw->retval = lw->callback(fname, lineno, addr, lw->data);
if (lw->retval != 0)
@@ -818,7 +822,7 @@ static int __die_walk_funclines(Dwarf_Die *sp_die, bool recursive,
int lineno;
/* Handle function declaration line */
- fname = dwarf_decl_file(sp_die);
+ fname = die_get_decl_file(sp_die);
if (fname && dwarf_decl_line(sp_die, &lineno) == 0 &&
die_entrypc(sp_die, &addr) == 0) {
lw.retval = callback(fname, lineno, addr, data);
@@ -873,7 +877,12 @@ int die_walk_lines(Dwarf_Die *rt_die, line_walk_callback_t callback, void *data)
if (dwarf_tag(rt_die) != DW_TAG_compile_unit) {
cu_die = dwarf_diecu(rt_die, &die_mem, NULL, NULL);
dwarf_decl_line(rt_die, &decl);
- decf = dwarf_decl_file(rt_die);
+ decf = die_get_decl_file(rt_die);
+ if (!decf) {
+ pr_debug2("Failed to get the declared file name of %s\n",
+ dwarf_diename(rt_die));
+ return -EINVAL;
+ }
} else
cu_die = rt_die;
if (!cu_die) {
@@ -923,7 +932,7 @@ int die_walk_lines(Dwarf_Die *rt_die, line_walk_callback_t callback, void *data)
dwarf_decl_line(&die_mem, &inl);
if (inl != decl ||
- decf != dwarf_decl_file(&die_mem))
+ decf != die_get_decl_file(&die_mem))
continue;
}
}
diff --git a/tools/perf/util/dwarf-aux.h b/tools/perf/util/dwarf-aux.h
index 7ee0fa1..7ec8bc1 100644
--- a/tools/perf/util/dwarf-aux.h
+++ b/tools/perf/util/dwarf-aux.h
@@ -50,6 +50,9 @@ int die_get_call_lineno(Dwarf_Die *in_die);
/* Get callsite file name of inlined function instance */
const char *die_get_call_file(Dwarf_Die *in_die);
+/* Get declared file name of a DIE */
+const char *die_get_decl_file(Dwarf_Die *dw_die);
+
/* Get type die */
Dwarf_Die *die_get_type(Dwarf_Die *vr_die, Dwarf_Die *die_mem);
diff --git a/tools/perf/util/event.h b/tools/perf/util/event.h
index 12eae69..6663a67 100644
--- a/tools/perf/util/event.h
+++ b/tools/perf/util/event.h
@@ -12,11 +12,10 @@
#include <perf/event.h>
#include <linux/types.h>
-#include "perf_regs.h"
-
struct dso;
struct machine;
struct perf_event_attr;
+struct perf_sample;
#ifdef __LP64__
/*
@@ -44,61 +43,6 @@ struct perf_event_attr;
/* perf sample has 16 bits size limit */
#define PERF_SAMPLE_MAX_SIZE (1 << 16)
-/* number of register is bound by the number of bits in regs_dump::mask (64) */
-#define PERF_SAMPLE_REGS_CACHE_SIZE (8 * sizeof(u64))
-
-struct regs_dump {
- u64 abi;
- u64 mask;
- u64 *regs;
-
- /* Cached values/mask filled by first register access. */
- u64 cache_regs[PERF_SAMPLE_REGS_CACHE_SIZE];
- u64 cache_mask;
-};
-
-struct stack_dump {
- u16 offset;
- u64 size;
- char *data;
-};
-
-struct sample_read_value {
- u64 value;
- u64 id; /* only if PERF_FORMAT_ID */
- u64 lost; /* only if PERF_FORMAT_LOST */
-};
-
-struct sample_read {
- u64 time_enabled;
- u64 time_running;
- union {
- struct {
- u64 nr;
- struct sample_read_value *values;
- } group;
- struct sample_read_value one;
- };
-};
-
-static inline size_t sample_read_value_size(u64 read_format)
-{
- /* PERF_FORMAT_ID is forced for PERF_SAMPLE_READ */
- if (read_format & PERF_FORMAT_LOST)
- return sizeof(struct sample_read_value);
- else
- return offsetof(struct sample_read_value, lost);
-}
-
-static inline struct sample_read_value *
-next_sample_read_value(struct sample_read_value *v, u64 read_format)
-{
- return (void *)v + sample_read_value_size(read_format);
-}
-
-#define sample_read_group__for_each(v, nr, rf) \
- for (int __i = 0; __i < (int)nr; v = next_sample_read_value(v, rf), __i++)
-
struct ip_callchain {
u64 nr;
u64 ips[];
@@ -140,52 +84,6 @@ enum {
PERF_IP_FLAG_VMENTRY |\
PERF_IP_FLAG_VMEXIT)
-#define MAX_INSN 16
-
-struct aux_sample {
- u64 size;
- void *data;
-};
-
-struct perf_sample {
- u64 ip;
- u32 pid, tid;
- u64 time;
- u64 addr;
- u64 id;
- u64 stream_id;
- u64 period;
- u64 weight;
- u64 transaction;
- u64 insn_cnt;
- u64 cyc_cnt;
- u32 cpu;
- u32 raw_size;
- u64 data_src;
- u64 phys_addr;
- u64 data_page_size;
- u64 code_page_size;
- u64 cgroup;
- u32 flags;
- u32 machine_pid;
- u32 vcpu;
- u16 insn_len;
- u8 cpumode;
- u16 misc;
- u16 ins_lat;
- u16 p_stage_cyc;
- bool no_hw_idx; /* No hw_idx collected in branch_stack */
- char insn[MAX_INSN];
- void *raw_data;
- struct ip_callchain *callchain;
- struct branch_stack *branch_stack;
- struct regs_dump user_regs;
- struct regs_dump intr_regs;
- struct stack_dump user_stack;
- struct sample_read read;
- struct aux_sample aux_sample;
-};
-
#define PERF_MEM_DATA_SRC_NONE \
(PERF_MEM_S(OP, NA) |\
PERF_MEM_S(LVL, NA) |\
@@ -344,15 +242,6 @@ struct perf_synth_intel_iflag_chg {
u64 branch_ip; /* If via_branch */
};
-/*
- * raw_data is always 4 bytes from an 8-byte boundary, so subtract 4 to get
- * 8-byte alignment.
- */
-static inline void *perf_sample__synth_ptr(struct perf_sample *sample)
-{
- return sample->raw_data - 4;
-}
-
static inline void *perf_synth__raw_data(void *p)
{
return p + 4;
@@ -446,19 +335,8 @@ int perf_event__process(struct perf_tool *tool,
struct perf_sample *sample,
struct machine *machine);
-struct addr_location;
-
-int machine__resolve(struct machine *machine, struct addr_location *al,
- struct perf_sample *sample);
-
-void addr_location__put(struct addr_location *al);
-
-struct thread;
-
bool is_bts_event(struct perf_event_attr *attr);
bool sample_addr_correlates_sym(struct perf_event_attr *attr);
-void thread__resolve(struct thread *thread, struct addr_location *al,
- struct perf_sample *sample);
const char *perf_event__name(unsigned int id);
diff --git a/tools/perf/util/evlist.c b/tools/perf/util/evlist.c
index 6612b00..817df25 100644
--- a/tools/perf/util/evlist.c
+++ b/tools/perf/util/evlist.c
@@ -24,11 +24,13 @@
#include "../perf.h"
#include "asm/bug.h"
#include "bpf-event.h"
+#include "util/event.h"
#include "util/string2.h"
#include "util/perf_api_probe.h"
#include "util/evsel_fprintf.h"
#include "util/evlist-hybrid.h"
#include "util/pmu.h"
+#include "util/sample.h"
#include <signal.h>
#include <unistd.h>
#include <sched.h>
@@ -228,7 +230,7 @@ int __evlist__set_tracepoints_handlers(struct evlist *evlist,
return err;
}
-void evlist__set_leader(struct evlist *evlist)
+static void evlist__set_leader(struct evlist *evlist)
{
perf_evlist__set_leader(&evlist->core);
}
@@ -288,6 +290,7 @@ struct evsel *evlist__add_aux_dummy(struct evlist *evlist, bool system_wide)
return evsel;
}
+#ifdef HAVE_LIBTRACEEVENT
struct evsel *evlist__add_sched_switch(struct evlist *evlist, bool system_wide)
{
struct evsel *evsel = evsel__newtp_idx("sched", "sched_switch", 0);
@@ -303,7 +306,8 @@ struct evsel *evlist__add_sched_switch(struct evlist *evlist, bool system_wide)
evlist__add(evlist, evsel);
return evsel;
-};
+}
+#endif
int evlist__add_attrs(struct evlist *evlist, struct perf_event_attr *attrs, size_t nr_attrs)
{
@@ -374,6 +378,7 @@ struct evsel *evlist__find_tracepoint_by_name(struct evlist *evlist, const char
return NULL;
}
+#ifdef HAVE_LIBTRACEEVENT
int evlist__add_newtp(struct evlist *evlist, const char *sys, const char *name, void *handler)
{
struct evsel *evsel = evsel__newtp(sys, name);
@@ -385,6 +390,7 @@ int evlist__add_newtp(struct evlist *evlist, const char *sys, const char *name,
evlist__add(evlist, evsel);
return 0;
}
+#endif
struct evlist_cpu_iterator evlist__cpu_begin(struct evlist *evlist, struct affinity *affinity)
{
diff --git a/tools/perf/util/evlist.h b/tools/perf/util/evlist.h
index 16734c6..01fa9d5 100644
--- a/tools/perf/util/evlist.h
+++ b/tools/perf/util/evlist.h
@@ -127,7 +127,9 @@ static inline struct evsel *evlist__add_dummy_on_all_cpus(struct evlist *evlist)
{
return evlist__add_aux_dummy(evlist, true);
}
+#ifdef HAVE_LIBTRACEEVENT
struct evsel *evlist__add_sched_switch(struct evlist *evlist, bool system_wide);
+#endif
int evlist__add_sb_event(struct evlist *evlist, struct perf_event_attr *attr,
evsel__sb_cb_t cb, void *data);
@@ -135,7 +137,9 @@ void evlist__set_cb(struct evlist *evlist, evsel__sb_cb_t cb, void *data);
int evlist__start_sb_thread(struct evlist *evlist, struct target *target);
void evlist__stop_sb_thread(struct evlist *evlist);
+#ifdef HAVE_LIBTRACEEVENT
int evlist__add_newtp(struct evlist *evlist, const char *sys, const char *name, void *handler);
+#endif
int __evlist__set_tracepoints_handlers(struct evlist *evlist,
const struct evsel_str_handler *assocs,
@@ -217,8 +221,6 @@ void evlist__set_selected(struct evlist *evlist, struct evsel *evsel);
int evlist__create_maps(struct evlist *evlist, struct target *target);
int evlist__apply_filters(struct evlist *evlist, struct evsel **err_evsel);
-void evlist__set_leader(struct evlist *evlist);
-
u64 __evlist__combined_sample_type(struct evlist *evlist);
u64 evlist__combined_sample_type(struct evlist *evlist);
u64 evlist__combined_branch_type(struct evlist *evlist);
diff --git a/tools/perf/util/evsel.c b/tools/perf/util/evsel.c
index 9e8a129..999dd17 100644
--- a/tools/perf/util/evsel.c
+++ b/tools/perf/util/evsel.c
@@ -12,7 +12,6 @@
#include <linux/bitops.h>
#include <api/fs/fs.h>
#include <api/fs/tracing_path.h>
-#include <traceevent/event-parse.h>
#include <linux/hw_breakpoint.h>
#include <linux/perf_event.h>
#include <linux/compiler.h>
@@ -46,20 +45,21 @@
#include "string2.h"
#include "memswap.h"
#include "util.h"
-#ifdef HAVE_LIBBPF_SUPPORT
-#include <bpf/hashmap.h>
-#else
#include "util/hashmap.h"
-#endif
#include "pmu-hybrid.h"
#include "off_cpu.h"
#include "../perf-sys.h"
#include "util/parse-branch-options.h"
#include <internal/xyarray.h>
#include <internal/lib.h>
+#include <internal/threadmap.h>
#include <linux/ctype.h>
+#ifdef HAVE_LIBTRACEEVENT
+#include <traceevent/event-parse.h>
+#endif
+
struct perf_missing_features perf_missing_features;
static clockid_t clockid;
@@ -442,7 +442,9 @@ struct evsel *evsel__clone(struct evsel *orig)
goto out_err;
}
evsel->cgrp = cgroup__get(orig->cgrp);
+#ifdef HAVE_LIBTRACEEVENT
evsel->tp_format = orig->tp_format;
+#endif
evsel->handler = orig->handler;
evsel->core.leader = orig->core.leader;
@@ -467,6 +469,7 @@ struct evsel *evsel__clone(struct evsel *orig)
evsel->collect_stat = orig->collect_stat;
evsel->weak_group = orig->weak_group;
evsel->use_config_name = orig->use_config_name;
+ evsel->pmu = orig->pmu;
if (evsel__copy_config_terms(evsel, orig) < 0)
goto out_err;
@@ -481,6 +484,7 @@ struct evsel *evsel__clone(struct evsel *orig)
/*
* Returns pointer with encoded error via <linux/err.h> interface.
*/
+#ifdef HAVE_LIBTRACEEVENT
struct evsel *evsel__newtp_idx(const char *sys, const char *name, int idx)
{
struct evsel *evsel = zalloc(perf_evsel__object.size);
@@ -518,6 +522,7 @@ struct evsel *evsel__newtp_idx(const char *sys, const char *name, int idx)
out_err:
return ERR_PTR(err);
}
+#endif
const char *const evsel__hw_names[PERF_COUNT_HW_MAX] = {
"cycles",
@@ -1525,13 +1530,8 @@ void evsel__compute_deltas(struct evsel *evsel, int cpu_map_idx, int thread,
if (!evsel->prev_raw_counts)
return;
- if (cpu_map_idx == -1) {
- tmp = evsel->prev_raw_counts->aggr;
- evsel->prev_raw_counts->aggr = *count;
- } else {
- tmp = *perf_counts(evsel->prev_raw_counts, cpu_map_idx, thread);
- *perf_counts(evsel->prev_raw_counts, cpu_map_idx, thread) = *count;
- }
+ tmp = *perf_counts(evsel->prev_raw_counts, cpu_map_idx, thread);
+ *perf_counts(evsel->prev_raw_counts, cpu_map_idx, thread) = *count;
count->val = count->val - tmp.val;
count->ena = count->ena - tmp.ena;
@@ -1966,17 +1966,16 @@ bool evsel__detect_missing_features(struct evsel *evsel)
perf_missing_features.mmap2 = true;
pr_debug2_peo("switching off mmap2\n");
return true;
- } else if ((evsel->core.attr.exclude_guest || evsel->core.attr.exclude_host) &&
- (evsel->pmu == NULL || evsel->pmu->missing_features.exclude_guest)) {
- if (evsel->pmu == NULL) {
+ } else if (evsel->core.attr.exclude_guest || evsel->core.attr.exclude_host) {
+ if (evsel->pmu == NULL)
evsel->pmu = evsel__find_pmu(evsel);
- if (evsel->pmu)
- evsel->pmu->missing_features.exclude_guest = true;
- else {
- /* we cannot find PMU, disable attrs now */
- evsel->core.attr.exclude_host = false;
- evsel->core.attr.exclude_guest = false;
- }
+
+ if (evsel->pmu)
+ evsel->pmu->missing_features.exclude_guest = true;
+ else {
+ /* we cannot find PMU, disable attrs now */
+ evsel->core.attr.exclude_host = false;
+ evsel->core.attr.exclude_guest = false;
}
if (evsel->exclude_GH) {
@@ -2328,11 +2327,8 @@ u64 evsel__bitfield_swap_branch_flags(u64 value)
* as it has variable bit-field sizes. Instead the
* macro takes the bit-field position/size,
* swaps it based on the host endianness.
- *
- * tep_is_bigendian() is used here instead of
- * bigendian() to avoid python test fails.
*/
- if (tep_is_bigendian()) {
+ if (host_is_bigendian()) {
new_val = bitfield_swap(value, 0, 1);
new_val |= bitfield_swap(value, 1, 1);
new_val |= bitfield_swap(value, 2, 1);
@@ -2769,6 +2765,7 @@ u16 evsel__id_hdr_size(struct evsel *evsel)
return size;
}
+#ifdef HAVE_LIBTRACEEVENT
struct tep_format_field *evsel__field(struct evsel *evsel, const char *name)
{
return tep_find_field(evsel->tp_format, name);
@@ -2787,8 +2784,10 @@ void *evsel__rawptr(struct evsel *evsel, struct perf_sample *sample, const char
if (field->flags & TEP_FIELD_IS_DYNAMIC) {
offset = *(int *)(sample->raw_data + field->offset);
offset &= 0xffff;
+#ifdef HAVE_LIBTRACEEVENT_TEP_FIELD_IS_RELATIVE
if (field->flags & TEP_FIELD_IS_RELATIVE)
offset += field->offset + field->size;
+#endif
}
return sample->raw_data + offset;
@@ -2842,6 +2841,7 @@ u64 evsel__intval(struct evsel *evsel, struct perf_sample *sample, const char *n
return field ? format_field__intval(field, sample, evsel->needs_swap) : 0;
}
+#endif
bool evsel__fallback(struct evsel *evsel, int err, char *msg, size_t msgsize)
{
@@ -3129,7 +3129,7 @@ void evsel__zero_per_pkg(struct evsel *evsel)
}
}
-bool evsel__is_hybrid(struct evsel *evsel)
+bool evsel__is_hybrid(const struct evsel *evsel)
{
return evsel->pmu_name && perf_pmu__is_hybrid(evsel->pmu_name);
}
diff --git a/tools/perf/util/evsel.h b/tools/perf/util/evsel.h
index 989865e..d572be4 100644
--- a/tools/perf/util/evsel.h
+++ b/tools/perf/util/evsel.h
@@ -10,8 +10,6 @@
#include <internal/evsel.h>
#include <perf/evsel.h>
#include "symbol_conf.h"
-#include <internal/cpumap.h>
-#include <perf/cpumap.h>
struct bpf_object;
struct cgroup;
@@ -74,7 +72,9 @@ struct evsel {
char *name;
char *group_name;
const char *pmu_name;
+#ifdef HAVE_LIBTRACEEVENT
struct tep_event *tp_format;
+#endif
char *filter;
unsigned long max_events;
double scale;
@@ -225,11 +225,14 @@ static inline struct evsel *evsel__new(struct perf_event_attr *attr)
}
struct evsel *evsel__clone(struct evsel *orig);
-struct evsel *evsel__newtp_idx(const char *sys, const char *name, int idx);
int copy_config_terms(struct list_head *dst, struct list_head *src);
void free_config_terms(struct list_head *config_terms);
+
+#ifdef HAVE_LIBTRACEEVENT
+struct evsel *evsel__newtp_idx(const char *sys, const char *name, int idx);
+
/*
* Returns pointer with encoded error via <linux/err.h> interface.
*/
@@ -237,10 +240,13 @@ static inline struct evsel *evsel__newtp(const char *sys, const char *name)
{
return evsel__newtp_idx(sys, name, 0);
}
+#endif
struct evsel *evsel__new_cycles(bool precise, __u32 type, __u64 config);
+#ifdef HAVE_LIBTRACEEVENT
struct tep_event *event_format__new(const char *sys, const char *name);
+#endif
void evsel__init(struct evsel *evsel, struct perf_event_attr *attr, int idx);
void evsel__exit(struct evsel *evsel);
@@ -325,6 +331,7 @@ bool evsel__precise_ip_fallback(struct evsel *evsel);
struct perf_sample;
+#ifdef HAVE_LIBTRACEEVENT
void *evsel__rawptr(struct evsel *evsel, struct perf_sample *sample, const char *name);
u64 evsel__intval(struct evsel *evsel, struct perf_sample *sample, const char *name);
@@ -332,6 +339,7 @@ static inline char *evsel__strval(struct evsel *evsel, struct perf_sample *sampl
{
return evsel__rawptr(evsel, sample, name);
}
+#endif
struct tep_format_field;
@@ -498,7 +506,7 @@ struct perf_env *evsel__env(struct evsel *evsel);
int evsel__store_ids(struct evsel *evsel, struct evlist *evlist);
void evsel__zero_per_pkg(struct evsel *evsel);
-bool evsel__is_hybrid(struct evsel *evsel);
+bool evsel__is_hybrid(const struct evsel *evsel);
struct evsel *evsel__leader(struct evsel *evsel);
bool evsel__has_leader(struct evsel *evsel, struct evsel *leader);
bool evsel__is_leader(struct evsel *evsel);
diff --git a/tools/perf/util/evsel_fprintf.c b/tools/perf/util/evsel_fprintf.c
index 8c2ea80..bd22c493 100644
--- a/tools/perf/util/evsel_fprintf.c
+++ b/tools/perf/util/evsel_fprintf.c
@@ -2,7 +2,6 @@
#include <inttypes.h>
#include <stdio.h>
#include <stdbool.h>
-#include <traceevent/event-parse.h>
#include "evsel.h"
#include "util/evsel_fprintf.h"
#include "util/event.h"
@@ -13,6 +12,10 @@
#include "srcline.h"
#include "dso.h"
+#ifdef HAVE_LIBTRACEEVENT
+#include <traceevent/event-parse.h>
+#endif
+
static int comma_fprintf(FILE *fp, bool *first, const char *fmt, ...)
{
va_list args;
@@ -74,6 +77,7 @@ int evsel__fprintf(struct evsel *evsel, struct perf_attr_details *details, FILE
term, (u64)evsel->core.attr.sample_freq);
}
+#ifdef HAVE_LIBTRACEEVENT
if (details->trace_fields) {
struct tep_format_field *field;
@@ -96,6 +100,7 @@ int evsel__fprintf(struct evsel *evsel, struct perf_attr_details *details, FILE
field = field->next;
}
}
+#endif
out:
fputc('\n', fp);
return ++printed;
diff --git a/tools/perf/util/expr.c b/tools/perf/util/expr.c
index 2f05ecd..00dcde3 100644
--- a/tools/perf/util/expr.c
+++ b/tools/perf/util/expr.c
@@ -11,6 +11,7 @@
#include "expr.h"
#include "expr-bison.h"
#include "expr-flex.h"
+#include "util/hashmap.h"
#include "smt.h"
#include "tsc.h"
#include <linux/err.h>
diff --git a/tools/perf/util/expr.h b/tools/perf/util/expr.h
index d6c1668..0292715 100644
--- a/tools/perf/util/expr.h
+++ b/tools/perf/util/expr.h
@@ -2,12 +2,7 @@
#ifndef PARSE_CTX_H
#define PARSE_CTX_H 1
-#ifdef HAVE_LIBBPF_SUPPORT
-#include <bpf/hashmap.h>
-#else
-#include "util/hashmap.h"
-#endif
-
+struct hashmap;
struct metric_ref;
struct expr_scanner_ctx {
diff --git a/tools/perf/util/header.c b/tools/perf/util/header.c
index dc2ae39..404d816 100644
--- a/tools/perf/util/header.c
+++ b/tools/perf/util/header.c
@@ -6,6 +6,7 @@
#include <sys/types.h>
#include <byteswap.h>
#include <unistd.h>
+#include <regex.h>
#include <stdio.h>
#include <stdlib.h>
#include <linux/compiler.h>
@@ -55,6 +56,10 @@
#include <linux/ctype.h>
#include <internal/lib.h>
+#ifdef HAVE_LIBTRACEEVENT
+#include <traceevent/event-parse.h>
+#endif
+
/*
* magic2 = "PERFILE2"
* must be a numerical value to let the endianness
@@ -298,6 +303,7 @@ static int do_read_bitmap(struct feat_fd *ff, unsigned long **pset, u64 *psize)
return 0;
}
+#ifdef HAVE_LIBTRACEEVENT
static int write_tracing_data(struct feat_fd *ff,
struct evlist *evlist)
{
@@ -306,6 +312,7 @@ static int write_tracing_data(struct feat_fd *ff,
return read_tracing_data(ff->fd, &evlist->core.entries);
}
+#endif
static int write_build_id(struct feat_fd *ff,
struct evlist *evlist __maybe_unused)
@@ -2394,12 +2401,14 @@ FEAT_PROCESS_STR_FUN(arch, arch);
FEAT_PROCESS_STR_FUN(cpudesc, cpu_desc);
FEAT_PROCESS_STR_FUN(cpuid, cpuid);
+#ifdef HAVE_LIBTRACEEVENT
static int process_tracing_data(struct feat_fd *ff, void *data)
{
ssize_t ret = trace_report(ff->fd, data, false);
return ret < 0 ? -1 : 0;
}
+#endif
static int process_build_id(struct feat_fd *ff, void *data __maybe_unused)
{
@@ -3366,7 +3375,9 @@ static int process_pmu_caps(struct feat_fd *ff, void *data __maybe_unused)
const struct perf_header_feature_ops feat_ops[HEADER_LAST_FEATURE];
const struct perf_header_feature_ops feat_ops[HEADER_LAST_FEATURE] = {
+#ifdef HAVE_LIBTRACEEVENT
FEAT_OPN(TRACING_DATA, tracing_data, false),
+#endif
FEAT_OPN(BUILD_ID, build_id, false),
FEAT_OPR(HOSTNAME, hostname, false),
FEAT_OPR(OSRELEASE, osrelease, false),
@@ -4082,6 +4093,7 @@ static int read_attr(int fd, struct perf_header *ph,
return ret <= 0 ? -1 : 0;
}
+#ifdef HAVE_LIBTRACEEVENT
static int evsel__prepare_tracepoint_event(struct evsel *evsel, struct tep_handle *pevent)
{
struct tep_event *event;
@@ -4125,6 +4137,7 @@ static int evlist__prepare_tracepoint_events(struct evlist *evlist, struct tep_h
return 0;
}
+#endif
int perf_session__read_header(struct perf_session *session, int repipe_fd)
{
@@ -4230,11 +4243,15 @@ int perf_session__read_header(struct perf_session *session, int repipe_fd)
lseek(fd, tmp, SEEK_SET);
}
+#ifdef HAVE_LIBTRACEEVENT
perf_header__process_sections(header, fd, &session->tevent,
perf_file_section__process);
if (evlist__prepare_tracepoint_events(session->evlist, session->tevent.pevent))
goto out_delete_evlist;
+#else
+ perf_header__process_sections(header, fd, NULL, perf_file_section__process);
+#endif
return 0;
out_errno:
@@ -4412,6 +4429,7 @@ int perf_event__process_event_update(struct perf_tool *tool __maybe_unused,
return 0;
}
+#ifdef HAVE_LIBTRACEEVENT
int perf_event__process_tracing_data(struct perf_session *session,
union perf_event *event)
{
@@ -4459,6 +4477,7 @@ int perf_event__process_tracing_data(struct perf_session *session,
return size_read + padding;
}
+#endif
int perf_event__process_build_id(struct perf_session *session,
union perf_event *event)
diff --git a/tools/perf/util/header.h b/tools/perf/util/header.h
index 2d5e601..e3861ae 100644
--- a/tools/perf/util/header.h
+++ b/tools/perf/util/header.h
@@ -160,8 +160,10 @@ int perf_event__process_event_update(struct perf_tool *tool,
union perf_event *event,
struct evlist **pevlist);
size_t perf_event__fprintf_event_update(union perf_event *event, FILE *fp);
+#ifdef HAVE_LIBTRACEEVENT
int perf_event__process_tracing_data(struct perf_session *session,
union perf_event *event);
+#endif
int perf_event__process_build_id(struct perf_session *session,
union perf_event *event);
bool is_perf_magic(u64 magic);
diff --git a/tools/perf/util/intel-pt-decoder/intel-pt-insn-decoder.c b/tools/perf/util/intel-pt-decoder/intel-pt-insn-decoder.c
index 1376077..22308dd 100644
--- a/tools/perf/util/intel-pt-decoder/intel-pt-insn-decoder.c
+++ b/tools/perf/util/intel-pt-decoder/intel-pt-insn-decoder.c
@@ -18,6 +18,7 @@
#include "intel-pt-insn-decoder.h"
#include "dump-insn.h"
+#include "util/sample.h"
#if INTEL_PT_INSN_BUF_SZ < MAX_INSN_SIZE || INTEL_PT_INSN_BUF_SZ > MAX_INSN
#error Instruction buffer size too small
diff --git a/tools/perf/util/intel-pt.c b/tools/perf/util/intel-pt.c
index e3548dd..6d39216 100644
--- a/tools/perf/util/intel-pt.c
+++ b/tools/perf/util/intel-pt.c
@@ -3142,6 +3142,7 @@ static int intel_pt_sync_switch(struct intel_pt *pt, int cpu, pid_t tid,
return 1;
}
+#ifdef HAVE_LIBTRACEEVENT
static int intel_pt_process_switch(struct intel_pt *pt,
struct perf_sample *sample)
{
@@ -3165,6 +3166,7 @@ static int intel_pt_process_switch(struct intel_pt *pt,
return machine__set_current_tid(pt->machine, cpu, -1, tid);
}
+#endif /* HAVE_LIBTRACEEVENT */
static int intel_pt_context_switch_in(struct intel_pt *pt,
struct perf_sample *sample)
@@ -3433,9 +3435,12 @@ static int intel_pt_process_event(struct perf_session *session,
return err;
}
+#ifdef HAVE_LIBTRACEEVENT
if (pt->switch_evsel && event->header.type == PERF_RECORD_SAMPLE)
err = intel_pt_process_switch(pt, sample);
- else if (event->header.type == PERF_RECORD_ITRACE_START)
+ else
+#endif
+ if (event->header.type == PERF_RECORD_ITRACE_START)
err = intel_pt_process_itrace_start(pt, event, sample);
else if (event->header.type == PERF_RECORD_AUX_OUTPUT_HW_ID)
err = intel_pt_process_aux_output_hw_id(pt, event, sample);
diff --git a/tools/perf/util/iostat.c b/tools/perf/util/iostat.c
index 57dd49d..b770bd4 100644
--- a/tools/perf/util/iostat.c
+++ b/tools/perf/util/iostat.c
@@ -48,6 +48,7 @@ __weak void iostat_print_counters(struct evlist *evlist __maybe_unused,
struct perf_stat_config *config __maybe_unused,
struct timespec *ts __maybe_unused,
char *prefix __maybe_unused,
- iostat_print_counter_t print_cnt_cb __maybe_unused)
+ iostat_print_counter_t print_cnt_cb __maybe_unused,
+ void *arg __maybe_unused)
{
}
diff --git a/tools/perf/util/iostat.h b/tools/perf/util/iostat.h
index 23c1c46..a4e7299 100644
--- a/tools/perf/util/iostat.h
+++ b/tools/perf/util/iostat.h
@@ -28,7 +28,7 @@ enum iostat_mode_t {
extern enum iostat_mode_t iostat_mode;
-typedef void (*iostat_print_counter_t)(struct perf_stat_config *, struct evsel *, char *);
+typedef void (*iostat_print_counter_t)(struct perf_stat_config *, struct evsel *, void *);
int iostat_prepare(struct evlist *evlist, struct perf_stat_config *config);
int iostat_parse(const struct option *opt, const char *str,
@@ -42,6 +42,6 @@ void iostat_print_metric(struct perf_stat_config *config, struct evsel *evsel,
struct perf_stat_output_ctx *out);
void iostat_print_counters(struct evlist *evlist,
struct perf_stat_config *config, struct timespec *ts,
- char *prefix, iostat_print_counter_t print_cnt_cb);
+ char *prefix, iostat_print_counter_t print_cnt_cb, void *arg);
#endif /* _IOSTAT_H */
diff --git a/tools/perf/util/kwork.h b/tools/perf/util/kwork.h
index 320c0a6..53b7327 100644
--- a/tools/perf/util/kwork.h
+++ b/tools/perf/util/kwork.h
@@ -1,16 +1,16 @@
#ifndef PERF_UTIL_KWORK_H
#define PERF_UTIL_KWORK_H
-#include "perf.h"
-
#include "util/tool.h"
-#include "util/event.h"
-#include "util/evlist.h"
-#include "util/session.h"
#include "util/time-utils.h"
-#include <linux/list.h>
#include <linux/bitmap.h>
+#include <linux/list.h>
+#include <linux/rbtree.h>
+#include <linux/types.h>
+
+struct perf_sample;
+struct perf_session;
enum kwork_class_type {
KWORK_CLASS_IRQ,
diff --git a/tools/perf/util/llvm-utils.c b/tools/perf/util/llvm-utils.c
index 2dc7970..650ffe3 100644
--- a/tools/perf/util/llvm-utils.c
+++ b/tools/perf/util/llvm-utils.c
@@ -463,7 +463,7 @@ int llvm__compile_bpf(const char *path, void **p_obj_buf,
char *pipe_template = NULL;
const char *opts = llvm_param.opts;
char *command_echo = NULL, *command_out;
- char *perf_include_dir = system_path(PERF_INCLUDE_DIR);
+ char *libbpf_include_dir = system_path(LIBBPF_INCLUDE_DIR);
if (path[0] != '-' && realpath(path, abspath) == NULL) {
err = errno;
@@ -495,7 +495,7 @@ int llvm__compile_bpf(const char *path, void **p_obj_buf,
snprintf(linux_version_code_str, sizeof(linux_version_code_str),
"0x%x", kernel_version);
- if (asprintf(&perf_bpf_include_opts, "-I%s/bpf", perf_include_dir) < 0)
+ if (asprintf(&perf_bpf_include_opts, "-I%s/", libbpf_include_dir) < 0)
goto errout;
force_set_env("NR_CPUS", nr_cpus_avail_str);
force_set_env("LINUX_VERSION_CODE", linux_version_code_str);
@@ -556,7 +556,7 @@ int llvm__compile_bpf(const char *path, void **p_obj_buf,
free(kbuild_dir);
free(kbuild_include_opts);
free(perf_bpf_include_opts);
- free(perf_include_dir);
+ free(libbpf_include_dir);
if (!p_obj_buf)
free(obj_buf);
@@ -572,7 +572,7 @@ int llvm__compile_bpf(const char *path, void **p_obj_buf,
free(kbuild_include_opts);
free(obj_buf);
free(perf_bpf_include_opts);
- free(perf_include_dir);
+ free(libbpf_include_dir);
free(pipe_template);
if (p_obj_buf)
*p_obj_buf = NULL;
diff --git a/tools/perf/util/lock-contention.h b/tools/perf/util/lock-contention.h
index b8cb883..47fd47f 100644
--- a/tools/perf/util/lock-contention.h
+++ b/tools/perf/util/lock-contention.h
@@ -91,7 +91,7 @@ struct thread_stat {
* Number of stack trace entries to skip when finding callers.
* The first few entries belong to the locking implementation itself.
*/
-#define CONTENTION_STACK_SKIP 3
+#define CONTENTION_STACK_SKIP 4
/*
* flags for lock:contention_begin
@@ -117,6 +117,7 @@ struct lock_contention {
int lost;
int max_stack;
int stack_skip;
+ int aggr_mode;
};
#ifdef HAVE_BPF_SKEL
@@ -145,6 +146,4 @@ static inline int lock_contention_read(struct lock_contention *con __maybe_unuse
#endif /* HAVE_BPF_SKEL */
-bool is_lock_function(struct machine *machine, u64 addr);
-
#endif /* PERF_LOCK_CONTENTION_H */
diff --git a/tools/perf/util/machine.c b/tools/perf/util/machine.c
index 76316e4..803c9d1 100644
--- a/tools/perf/util/machine.c
+++ b/tools/perf/util/machine.c
@@ -3336,3 +3336,43 @@ int machine__for_each_kernel_map(struct machine *machine, machine__map_t fn, voi
}
return err;
}
+
+bool machine__is_lock_function(struct machine *machine, u64 addr)
+{
+ if (!machine->sched.text_start) {
+ struct map *kmap;
+ struct symbol *sym = machine__find_kernel_symbol_by_name(machine, "__sched_text_start", &kmap);
+
+ if (!sym) {
+ /* to avoid retry */
+ machine->sched.text_start = 1;
+ return false;
+ }
+
+ machine->sched.text_start = kmap->unmap_ip(kmap, sym->start);
+
+ /* should not fail from here */
+ sym = machine__find_kernel_symbol_by_name(machine, "__sched_text_end", &kmap);
+ machine->sched.text_end = kmap->unmap_ip(kmap, sym->start);
+
+ sym = machine__find_kernel_symbol_by_name(machine, "__lock_text_start", &kmap);
+ machine->lock.text_start = kmap->unmap_ip(kmap, sym->start);
+
+ sym = machine__find_kernel_symbol_by_name(machine, "__lock_text_end", &kmap);
+ machine->lock.text_end = kmap->unmap_ip(kmap, sym->start);
+ }
+
+ /* failed to get kernel symbols */
+ if (machine->sched.text_start == 1)
+ return false;
+
+ /* mutex and rwsem functions are in sched text section */
+ if (machine->sched.text_start <= addr && addr < machine->sched.text_end)
+ return true;
+
+ /* spinlock functions are in lock text section */
+ if (machine->lock.text_start <= addr && addr < machine->lock.text_end)
+ return true;
+
+ return false;
+}
diff --git a/tools/perf/util/machine.h b/tools/perf/util/machine.h
index 74935df..d034eca 100644
--- a/tools/perf/util/machine.h
+++ b/tools/perf/util/machine.h
@@ -56,6 +56,10 @@ struct machine {
struct maps *kmaps;
struct map *vmlinux_map;
u64 kernel_start;
+ struct {
+ u64 text_start;
+ u64 text_end;
+ } sched, lock;
pid_t *current_tid;
size_t current_tid_sz;
union { /* Tool specific area */
@@ -212,6 +216,7 @@ static inline bool machine__is_host(struct machine *machine)
return machine ? machine->pid == HOST_KERNEL_ID : false;
}
+bool machine__is_lock_function(struct machine *machine, u64 addr);
bool machine__is(struct machine *machine, const char *arch);
bool machine__normalized_is(struct machine *machine, const char *arch);
int machine__nr_cpus_avail(struct machine *machine);
@@ -305,4 +310,7 @@ int machine__create_extra_kernel_map(struct machine *machine,
int machine__map_x86_64_entry_trampolines(struct machine *machine,
struct dso *kernel);
+int machine__resolve(struct machine *machine, struct addr_location *al,
+ struct perf_sample *sample);
+
#endif /* __PERF_MACHINE_H */
diff --git a/tools/perf/util/metricgroup.c b/tools/perf/util/metricgroup.c
index 6b3505b..b9c273e 100644
--- a/tools/perf/util/metricgroup.c
+++ b/tools/perf/util/metricgroup.c
@@ -12,6 +12,7 @@
#include "strbuf.h"
#include "pmu.h"
#include "pmu-hybrid.h"
+#include "print-events.h"
#include "expr.h"
#include "rblist.h"
#include <string.h>
@@ -28,6 +29,7 @@
#include "util.h"
#include <asm/bug.h>
#include "cgroup.h"
+#include "util/hashmap.h"
struct metric_event *metricgroup__lookup(struct rblist *metric_events,
struct evsel *evsel,
@@ -352,56 +354,41 @@ static bool match_pe_metric(const struct pmu_event *pe, const char *metric)
match_metric(pe->metric_name, metric);
}
+/** struct mep - RB-tree node for building printing information. */
struct mep {
+ /** nd - RB-tree element. */
struct rb_node nd;
- const char *name;
- struct strlist *metrics;
+ /** @metric_group: Owned metric group name, separated others with ';'. */
+ char *metric_group;
+ const char *metric_name;
+ const char *metric_desc;
+ const char *metric_long_desc;
+ const char *metric_expr;
+ const char *metric_unit;
};
static int mep_cmp(struct rb_node *rb_node, const void *entry)
{
struct mep *a = container_of(rb_node, struct mep, nd);
struct mep *b = (struct mep *)entry;
+ int ret;
- return strcmp(a->name, b->name);
+ ret = strcmp(a->metric_group, b->metric_group);
+ if (ret)
+ return ret;
+
+ return strcmp(a->metric_name, b->metric_name);
}
-static struct rb_node *mep_new(struct rblist *rl __maybe_unused,
- const void *entry)
+static struct rb_node *mep_new(struct rblist *rl __maybe_unused, const void *entry)
{
struct mep *me = malloc(sizeof(struct mep));
if (!me)
return NULL;
- memcpy(me, entry, sizeof(struct mep));
- me->name = strdup(me->name);
- if (!me->name)
- goto out_me;
- me->metrics = strlist__new(NULL, NULL);
- if (!me->metrics)
- goto out_name;
- return &me->nd;
-out_name:
- zfree(&me->name);
-out_me:
- free(me);
- return NULL;
-}
-static struct mep *mep_lookup(struct rblist *groups, const char *name)
-{
- struct rb_node *nd;
- struct mep me = {
- .name = name
- };
- nd = rblist__find(groups, &me);
- if (nd)
- return container_of(nd, struct mep, nd);
- rblist__add_node(groups, &me);
- nd = rblist__find(groups, &me);
- if (nd)
- return container_of(nd, struct mep, nd);
- return NULL;
+ memcpy(me, entry, sizeof(struct mep));
+ return &me->nd;
}
static void mep_delete(struct rblist *rl __maybe_unused,
@@ -409,102 +396,61 @@ static void mep_delete(struct rblist *rl __maybe_unused,
{
struct mep *me = container_of(nd, struct mep, nd);
- strlist__delete(me->metrics);
- zfree(&me->name);
+ zfree(&me->metric_group);
free(me);
}
-static void metricgroup__print_strlist(struct strlist *metrics, bool raw)
+static struct mep *mep_lookup(struct rblist *groups, const char *metric_group,
+ const char *metric_name)
{
- struct str_node *sn;
- int n = 0;
-
- strlist__for_each_entry (sn, metrics) {
- if (raw)
- printf("%s%s", n > 0 ? " " : "", sn->s);
- else
- printf(" %s\n", sn->s);
- n++;
+ struct rb_node *nd;
+ struct mep me = {
+ .metric_group = strdup(metric_group),
+ .metric_name = metric_name,
+ };
+ nd = rblist__find(groups, &me);
+ if (nd) {
+ free(me.metric_group);
+ return container_of(nd, struct mep, nd);
}
- if (raw)
- putchar('\n');
+ rblist__add_node(groups, &me);
+ nd = rblist__find(groups, &me);
+ if (nd)
+ return container_of(nd, struct mep, nd);
+ return NULL;
}
-static int metricgroup__print_pmu_event(const struct pmu_event *pe,
- bool metricgroups, char *filter,
- bool raw, bool details,
- struct rblist *groups,
- struct strlist *metriclist)
+static int metricgroup__add_to_mep_groups(const struct pmu_event *pe,
+ struct rblist *groups)
{
const char *g;
char *omg, *mg;
- g = pe->metric_group;
- if (!g && pe->metric_name) {
- if (pe->name)
- return 0;
- g = "No_group";
- }
-
- if (!g)
- return 0;
-
- mg = strdup(g);
-
+ mg = strdup(pe->metric_group ?: "No_group");
if (!mg)
return -ENOMEM;
omg = mg;
while ((g = strsep(&mg, ";")) != NULL) {
struct mep *me;
- char *s;
g = skip_spaces(g);
- if (*g == 0)
- g = "No_group";
- if (filter && !strstr(g, filter))
- continue;
- if (raw)
- s = (char *)pe->metric_name;
- else {
- if (asprintf(&s, "%s\n%*s%s]",
- pe->metric_name, 8, "[", pe->desc) < 0)
- return -1;
- if (details) {
- if (asprintf(&s, "%s\n%*s%s]",
- s, 8, "[", pe->metric_expr) < 0)
- return -1;
- }
+ if (strlen(g))
+ me = mep_lookup(groups, g, pe->metric_name);
+ else
+ me = mep_lookup(groups, "No_group", pe->metric_name);
+
+ if (me) {
+ me->metric_desc = pe->desc;
+ me->metric_long_desc = pe->long_desc;
+ me->metric_expr = pe->metric_expr;
+ me->metric_unit = pe->unit;
}
-
- if (!s)
- continue;
-
- if (!metricgroups) {
- strlist__add(metriclist, s);
- } else {
- me = mep_lookup(groups, g);
- if (!me)
- continue;
- strlist__add(me->metrics, s);
- }
-
- if (!raw)
- free(s);
}
free(omg);
return 0;
}
-struct metricgroup_print_sys_idata {
- struct strlist *metriclist;
- char *filter;
- struct rblist *groups;
- bool metricgroups;
- bool raw;
- bool details;
-};
-
struct metricgroup_iter_data {
pmu_event_iter_fn fn;
void *data;
@@ -527,60 +473,26 @@ static int metricgroup__sys_event_iter(const struct pmu_event *pe,
return d->fn(pe, table, d->data);
}
-
return 0;
}
-static int metricgroup__print_sys_event_iter(const struct pmu_event *pe,
- const struct pmu_events_table *table __maybe_unused,
- void *data)
+static int metricgroup__add_to_mep_groups_callback(const struct pmu_event *pe,
+ const struct pmu_events_table *table __maybe_unused,
+ void *vdata)
{
- struct metricgroup_print_sys_idata *d = data;
+ struct rblist *groups = vdata;
- return metricgroup__print_pmu_event(pe, d->metricgroups, d->filter, d->raw,
- d->details, d->groups, d->metriclist);
-}
-
-struct metricgroup_print_data {
- const char *pmu_name;
- struct strlist *metriclist;
- char *filter;
- struct rblist *groups;
- bool metricgroups;
- bool raw;
- bool details;
-};
-
-static int metricgroup__print_callback(const struct pmu_event *pe,
- const struct pmu_events_table *table __maybe_unused,
- void *vdata)
-{
- struct metricgroup_print_data *data = vdata;
-
- if (!pe->metric_expr)
+ if (!pe->metric_name)
return 0;
- if (data->pmu_name && perf_pmu__is_hybrid(pe->pmu) && strcmp(data->pmu_name, pe->pmu))
- return 0;
-
- return metricgroup__print_pmu_event(pe, data->metricgroups, data->filter,
- data->raw, data->details, data->groups,
- data->metriclist);
+ return metricgroup__add_to_mep_groups(pe, groups);
}
-void metricgroup__print(bool metrics, bool metricgroups, char *filter,
- bool raw, bool details, const char *pmu_name)
+void metricgroup__print(const struct print_callbacks *print_cb, void *print_state)
{
struct rblist groups;
- struct rb_node *node, *next;
- struct strlist *metriclist = NULL;
const struct pmu_events_table *table;
-
- if (!metricgroups) {
- metriclist = strlist__new(NULL, NULL);
- if (!metriclist)
- return;
- }
+ struct rb_node *node, *next;
rblist__init(&groups);
groups.node_new = mep_new;
@@ -588,56 +500,31 @@ void metricgroup__print(bool metrics, bool metricgroups, char *filter,
groups.node_delete = mep_delete;
table = pmu_events_table__find();
if (table) {
- struct metricgroup_print_data data = {
- .pmu_name = pmu_name,
- .metriclist = metriclist,
- .metricgroups = metricgroups,
- .filter = filter,
- .raw = raw,
- .details = details,
- .groups = &groups,
- };
-
pmu_events_table_for_each_event(table,
- metricgroup__print_callback,
- &data);
+ metricgroup__add_to_mep_groups_callback,
+ &groups);
}
{
struct metricgroup_iter_data data = {
- .fn = metricgroup__print_sys_event_iter,
- .data = (void *) &(struct metricgroup_print_sys_idata){
- .metriclist = metriclist,
- .metricgroups = metricgroups,
- .filter = filter,
- .raw = raw,
- .details = details,
- .groups = &groups,
- },
+ .fn = metricgroup__add_to_mep_groups_callback,
+ .data = &groups,
};
-
pmu_for_each_sys_event(metricgroup__sys_event_iter, &data);
}
- if (!filter || !rblist__empty(&groups)) {
- if (metricgroups && !raw)
- printf("\nMetric Groups:\n\n");
- else if (metrics && !raw)
- printf("\nMetrics:\n\n");
- }
-
for (node = rb_first_cached(&groups.entries); node; node = next) {
struct mep *me = container_of(node, struct mep, nd);
- if (metricgroups)
- printf("%s%s%s", me->name, metrics && !raw ? ":" : "", raw ? " " : "\n");
- if (metrics)
- metricgroup__print_strlist(me->metrics, raw);
+ print_cb->print_metric(print_state,
+ me->metric_group,
+ me->metric_name,
+ me->metric_desc,
+ me->metric_long_desc,
+ me->metric_expr,
+ me->metric_unit);
next = rb_next(node);
rblist__remove_node(&groups, node);
}
- if (!metricgroups)
- metricgroup__print_strlist(metriclist, raw);
- strlist__delete(metriclist);
}
static const char *code_characters = ",-=@";
diff --git a/tools/perf/util/metricgroup.h b/tools/perf/util/metricgroup.h
index 732d3a0..0013cf5 100644
--- a/tools/perf/util/metricgroup.h
+++ b/tools/perf/util/metricgroup.h
@@ -10,6 +10,7 @@
struct evlist;
struct evsel;
struct option;
+struct print_callbacks;
struct rblist;
struct cgroup;
@@ -78,8 +79,7 @@ int metricgroup__parse_groups_test(struct evlist *evlist,
bool metric_no_merge,
struct rblist *metric_events);
-void metricgroup__print(bool metrics, bool groups, char *filter,
- bool raw, bool details, const char *pmu_name);
+void metricgroup__print(const struct print_callbacks *print_cb, void *print_state);
bool metricgroup__has_metric(const char *metric);
int arch_get_runtimeparam(const struct pmu_event *pe __maybe_unused);
void metricgroup__rblist_exit(struct rblist *metric_events);
diff --git a/tools/perf/util/mmap.h b/tools/perf/util/mmap.h
index cd4ccec..f944c3c 100644
--- a/tools/perf/util/mmap.h
+++ b/tools/perf/util/mmap.h
@@ -2,18 +2,13 @@
#define __PERF_MMAP_H 1
#include <internal/mmap.h>
-#include <linux/compiler.h>
-#include <linux/refcount.h>
#include <linux/types.h>
-#include <linux/ring_buffer.h>
#include <linux/bitops.h>
#include <perf/cpumap.h>
-#include <stdbool.h>
#ifdef HAVE_AIO_SUPPORT
#include <aio.h>
#endif
#include "auxtrace.h"
-#include "event.h"
#include "util/compress.h"
struct aiocb;
diff --git a/tools/perf/util/parse-branch-options.c b/tools/perf/util/parse-branch-options.c
index 31faf2b..fd67d20 100644
--- a/tools/perf/util/parse-branch-options.c
+++ b/tools/perf/util/parse-branch-options.c
@@ -30,8 +30,11 @@ static const struct branch_mode branch_modes[] = {
BRANCH_OPT("cond", PERF_SAMPLE_BRANCH_COND),
BRANCH_OPT("ind_jmp", PERF_SAMPLE_BRANCH_IND_JUMP),
BRANCH_OPT("call", PERF_SAMPLE_BRANCH_CALL),
+ BRANCH_OPT("no_flags", PERF_SAMPLE_BRANCH_NO_FLAGS),
+ BRANCH_OPT("no_cycles", PERF_SAMPLE_BRANCH_NO_CYCLES),
BRANCH_OPT("save_type", PERF_SAMPLE_BRANCH_TYPE_SAVE),
BRANCH_OPT("stack", PERF_SAMPLE_BRANCH_CALL_STACK),
+ BRANCH_OPT("hw_index", PERF_SAMPLE_BRANCH_HW_INDEX),
BRANCH_OPT("priv", PERF_SAMPLE_BRANCH_PRIV_SAVE),
BRANCH_END
};
diff --git a/tools/perf/util/parse-events.c b/tools/perf/util/parse-events.c
index 5973f46..21cce83 100644
--- a/tools/perf/util/parse-events.c
+++ b/tools/perf/util/parse-events.c
@@ -266,6 +266,7 @@ __add_event(struct list_head *list, int *idx,
evsel->core.own_cpus = perf_cpu_map__get(cpus);
evsel->core.requires_cpu = pmu ? pmu->is_uncore : false;
evsel->auto_merge_stats = auto_merge_stats;
+ evsel->pmu = pmu;
if (name)
evsel->name = strdup(name);
@@ -444,6 +445,7 @@ int parse_events_add_cache(struct list_head *list, int *idx,
return ret;
}
+#ifdef HAVE_LIBTRACEEVENT
static void tracepoint_error(struct parse_events_error *e, int err,
const char *sys, const char *name)
{
@@ -592,6 +594,7 @@ static int add_tracepoint_multi_sys(struct list_head *list, int *idx,
closedir(events_dir);
return ret;
}
+#endif /* HAVE_LIBTRACEEVENT */
#ifdef HAVE_LIBBPF_SUPPORT
struct __add_bpf_event_param {
@@ -1142,6 +1145,7 @@ static int config_term_pmu(struct perf_event_attr *attr,
return config_term_common(attr, term, err);
}
+#ifdef HAVE_LIBTRACEEVENT
static int config_term_tracepoint(struct perf_event_attr *attr,
struct parse_events_term *term,
struct parse_events_error *err)
@@ -1169,6 +1173,7 @@ static int config_term_tracepoint(struct perf_event_attr *attr,
return 0;
}
+#endif
static int config_attr(struct perf_event_attr *attr,
struct list_head *head,
@@ -1324,6 +1329,7 @@ int parse_events_add_tracepoint(struct list_head *list, int *idx,
struct parse_events_error *err,
struct list_head *head_config)
{
+#ifdef HAVE_LIBTRACEEVENT
if (head_config) {
struct perf_event_attr attr;
@@ -1338,6 +1344,16 @@ int parse_events_add_tracepoint(struct list_head *list, int *idx,
else
return add_tracepoint_event(list, idx, sys, event,
err, head_config);
+#else
+ (void)list;
+ (void)idx;
+ (void)sys;
+ (void)event;
+ (void)head_config;
+ parse_events_error__handle(err, 0, strdup("unsupported tracepoint"),
+ strdup("libtraceevent is necessary for tracepoint support"));
+ return -1;
+#endif
}
int parse_events_add_numeric(struct parse_events_state *parse_state,
diff --git a/tools/perf/util/parse-events.h b/tools/perf/util/parse-events.h
index 07df7bb..428e72e 100644
--- a/tools/perf/util/parse-events.h
+++ b/tools/perf/util/parse-events.h
@@ -18,7 +18,6 @@ struct parse_events_error;
struct option;
struct perf_pmu;
-bool have_tracepoints(struct list_head *evlist);
bool is_event_supported(u8 type, u64 config);
const char *event_type(int type);
diff --git a/tools/perf/util/perf_regs.c b/tools/perf/util/perf_regs.c
index 872dd3d..57a567e 100644
--- a/tools/perf/util/perf_regs.c
+++ b/tools/perf/util/perf_regs.c
@@ -2,7 +2,7 @@
#include <errno.h>
#include <string.h>
#include "perf_regs.h"
-#include "event.h"
+#include "util/sample.h"
int __weak arch_sdt_arg_parse_op(char *old_op __maybe_unused,
char **new_op __maybe_unused)
diff --git a/tools/perf/util/pfm.c b/tools/perf/util/pfm.c
index f0bcfca..ac3227b 100644
--- a/tools/perf/util/pfm.c
+++ b/tools/perf/util/pfm.c
@@ -12,6 +12,7 @@
#include "util/parse-events.h"
#include "util/pmu.h"
#include "util/pfm.h"
+#include "util/strbuf.h"
#include <string.h>
#include <linux/kernel.h>
@@ -130,118 +131,111 @@ static const char *srcs[PFM_ATTR_CTRL_MAX] = {
};
static void
-print_attr_flags(pfm_event_attr_info_t *info)
+print_attr_flags(struct strbuf *buf, const pfm_event_attr_info_t *info)
{
- int n = 0;
+ if (info->is_dfl)
+ strbuf_addf(buf, "[default] ");
- if (info->is_dfl) {
- printf("[default] ");
- n++;
- }
-
- if (info->is_precise) {
- printf("[precise] ");
- n++;
- }
-
- if (!n)
- printf("- ");
+ if (info->is_precise)
+ strbuf_addf(buf, "[precise] ");
}
static void
-print_libpfm_events_detailed(pfm_event_info_t *info, bool long_desc)
+print_libpfm_event(const struct print_callbacks *print_cb, void *print_state,
+ const pfm_pmu_info_t *pinfo, const pfm_event_info_t *info,
+ struct strbuf *buf)
{
- pfm_event_attr_info_t ainfo;
- const char *src;
int j, ret;
+ char topic[80], name[80];
- ainfo.size = sizeof(ainfo);
+ strbuf_setlen(buf, 0);
+ snprintf(topic, sizeof(topic), "pfm %s", pinfo->name);
- printf(" %s\n", info->name);
- printf(" [%s]\n", info->desc);
- if (long_desc) {
- if (info->equiv)
- printf(" Equiv: %s\n", info->equiv);
+ snprintf(name, sizeof(name), "%s::%s", pinfo->name, info->name);
+ strbuf_addf(buf, "Code: 0x%"PRIx64"\n", info->code);
- printf(" Code : 0x%"PRIx64"\n", info->code);
- }
pfm_for_each_event_attr(j, info) {
- ret = pfm_get_event_attr_info(info->idx, j,
- PFM_OS_PERF_EVENT_EXT, &ainfo);
+ pfm_event_attr_info_t ainfo;
+ const char *src;
+
+ ainfo.size = sizeof(ainfo);
+ ret = pfm_get_event_attr_info(info->idx, j, PFM_OS_PERF_EVENT_EXT, &ainfo);
if (ret != PFM_SUCCESS)
continue;
- if (ainfo.type == PFM_ATTR_UMASK) {
- printf(" %s:%s\n", info->name, ainfo.name);
- printf(" [%s]\n", ainfo.desc);
- }
-
- if (!long_desc)
- continue;
-
if (ainfo.ctrl >= PFM_ATTR_CTRL_MAX)
ainfo.ctrl = PFM_ATTR_CTRL_UNKNOWN;
src = srcs[ainfo.ctrl];
switch (ainfo.type) {
- case PFM_ATTR_UMASK:
- printf(" Umask : 0x%02"PRIx64" : %s: ",
- ainfo.code, src);
- print_attr_flags(&ainfo);
- putchar('\n');
+ case PFM_ATTR_UMASK: /* Ignore for now */
break;
case PFM_ATTR_MOD_BOOL:
- printf(" Modif : %s: [%s] : %s (boolean)\n", src,
- ainfo.name, ainfo.desc);
+ strbuf_addf(buf, " Modif: %s: [%s] : %s (boolean)\n", src,
+ ainfo.name, ainfo.desc);
break;
case PFM_ATTR_MOD_INTEGER:
- printf(" Modif : %s: [%s] : %s (integer)\n", src,
- ainfo.name, ainfo.desc);
+ strbuf_addf(buf, " Modif: %s: [%s] : %s (integer)\n", src,
+ ainfo.name, ainfo.desc);
break;
case PFM_ATTR_NONE:
case PFM_ATTR_RAW_UMASK:
case PFM_ATTR_MAX:
default:
- printf(" Attr : %s: [%s] : %s\n", src,
- ainfo.name, ainfo.desc);
+ strbuf_addf(buf, " Attr: %s: [%s] : %s\n", src,
+ ainfo.name, ainfo.desc);
+ }
+ }
+ print_cb->print_event(print_state,
+ pinfo->name,
+ topic,
+ name, info->equiv,
+ /*scale_unit=*/NULL,
+ /*deprecated=*/NULL, "PFM event",
+ info->desc, /*long_desc=*/NULL,
+ /*encoding_desc=*/buf->buf,
+ /*metric_name=*/NULL, /*metric_expr=*/NULL);
+
+ pfm_for_each_event_attr(j, info) {
+ pfm_event_attr_info_t ainfo;
+ const char *src;
+
+ strbuf_setlen(buf, 0);
+
+ ainfo.size = sizeof(ainfo);
+ ret = pfm_get_event_attr_info(info->idx, j, PFM_OS_PERF_EVENT_EXT, &ainfo);
+ if (ret != PFM_SUCCESS)
+ continue;
+
+ if (ainfo.ctrl >= PFM_ATTR_CTRL_MAX)
+ ainfo.ctrl = PFM_ATTR_CTRL_UNKNOWN;
+
+ src = srcs[ainfo.ctrl];
+ if (ainfo.type == PFM_ATTR_UMASK) {
+ strbuf_addf(buf, "Umask: 0x%02"PRIx64" : %s: ",
+ ainfo.code, src);
+ print_attr_flags(buf, &ainfo);
+ snprintf(name, sizeof(name), "%s::%s:%s",
+ pinfo->name, info->name, ainfo.name);
+ print_cb->print_event(print_state,
+ pinfo->name,
+ topic,
+ name, /*alias=*/NULL,
+ /*scale_unit=*/NULL,
+ /*deprecated=*/NULL, "PFM event",
+ ainfo.desc, /*long_desc=*/NULL,
+ /*encoding_desc=*/buf->buf,
+ /*metric_name=*/NULL, /*metric_expr=*/NULL);
}
}
}
-/*
- * list all pmu::event:umask, pmu::event
- * printed events may not be all valid combinations of umask for an event
- */
-static void
-print_libpfm_events_raw(pfm_pmu_info_t *pinfo, pfm_event_info_t *info)
-{
- pfm_event_attr_info_t ainfo;
- int j, ret;
- bool has_umask = false;
-
- ainfo.size = sizeof(ainfo);
-
- pfm_for_each_event_attr(j, info) {
- ret = pfm_get_event_attr_info(info->idx, j,
- PFM_OS_PERF_EVENT_EXT, &ainfo);
- if (ret != PFM_SUCCESS)
- continue;
-
- if (ainfo.type != PFM_ATTR_UMASK)
- continue;
-
- printf("%s::%s:%s\n", pinfo->name, info->name, ainfo.name);
- has_umask = true;
- }
- if (!has_umask)
- printf("%s::%s\n", pinfo->name, info->name);
-}
-
-void print_libpfm_events(bool name_only, bool long_desc)
+void print_libpfm_events(const struct print_callbacks *print_cb, void *print_state)
{
pfm_event_info_t info;
pfm_pmu_info_t pinfo;
- int i, p, ret;
+ int p, ret;
+ struct strbuf storage;
libpfm_initialize();
@@ -249,12 +243,9 @@ void print_libpfm_events(bool name_only, bool long_desc)
info.size = sizeof(info);
pinfo.size = sizeof(pinfo);
- if (!name_only)
- puts("\nList of pre-defined events (to be used in --pfm-events):\n");
+ strbuf_init(&storage, 2048);
pfm_for_all_pmus(p) {
- bool printed_pmu = false;
-
ret = pfm_get_pmu_info(p, &pinfo);
if (ret != PFM_SUCCESS)
continue;
@@ -267,25 +258,14 @@ void print_libpfm_events(bool name_only, bool long_desc)
if (pinfo.pmu == PFM_PMU_PERF_EVENT)
continue;
- for (i = pinfo.first_event; i != -1;
- i = pfm_get_event_next(i)) {
-
+ for (int i = pinfo.first_event; i != -1; i = pfm_get_event_next(i)) {
ret = pfm_get_event_info(i, PFM_OS_PERF_EVENT_EXT,
&info);
if (ret != PFM_SUCCESS)
continue;
- if (!name_only && !printed_pmu) {
- printf("%s:\n", pinfo.name);
- printed_pmu = true;
- }
-
- if (!name_only)
- print_libpfm_events_detailed(&info, long_desc);
- else
- print_libpfm_events_raw(&pinfo, &info);
+ print_libpfm_event(print_cb, print_state, &pinfo, &info, &storage);
}
- if (!name_only && printed_pmu)
- putchar('\n');
}
+ strbuf_release(&storage);
}
diff --git a/tools/perf/util/pfm.h b/tools/perf/util/pfm.h
index 7d70dda..fb25c27 100644
--- a/tools/perf/util/pfm.h
+++ b/tools/perf/util/pfm.h
@@ -7,13 +7,14 @@
#ifndef __PERF_PFM_H
#define __PERF_PFM_H
+#include "print-events.h"
#include <subcmd/parse-options.h>
#ifdef HAVE_LIBPFM
int parse_libpfm_events_option(const struct option *opt, const char *str,
int unset);
-void print_libpfm_events(bool name_only, bool long_desc);
+void print_libpfm_events(const struct print_callbacks *print_cb, void *print_state);
#else
#include <linux/compiler.h>
@@ -26,8 +27,8 @@ static inline int parse_libpfm_events_option(
return 0;
}
-static inline void print_libpfm_events(bool name_only __maybe_unused,
- bool long_desc __maybe_unused)
+static inline void print_libpfm_events(const struct print_callbacks *print_cb __maybe_unused,
+ void *print_state __maybe_unused)
{
}
diff --git a/tools/perf/util/pmu.c b/tools/perf/util/pmu.c
index 371d8f7..2bdeb89 100644
--- a/tools/perf/util/pmu.c
+++ b/tools/perf/util/pmu.c
@@ -22,7 +22,9 @@
#include "debug.h"
#include "evsel.h"
#include "pmu.h"
+#include "pmus.h"
#include "parse-events.h"
+#include "print-events.h"
#include "header.h"
#include "string2.h"
#include "strbuf.h"
@@ -31,17 +33,32 @@
struct perf_pmu perf_pmu__fake;
+/**
+ * struct perf_pmu_format - Values from a format file read from
+ * <sysfs>/devices/cpu/format/ held in struct perf_pmu.
+ *
+ * For example, the contents of <sysfs>/devices/cpu/format/event may be
+ * "config:0-7" and will be represented here as name="event",
+ * value=PERF_PMU_FORMAT_VALUE_CONFIG and bits 0 to 7 will be set.
+ */
struct perf_pmu_format {
+ /** @name: The modifier/file name. */
char *name;
+ /**
+ * @value : Which config value the format relates to. Supported values
+ * are from PERF_PMU_FORMAT_VALUE_CONFIG to
+ * PERF_PMU_FORMAT_VALUE_CONFIG_END.
+ */
int value;
+ /** @bits: Which config bits are set by this format value. */
DECLARE_BITMAP(bits, PERF_PMU_FORMAT_BITS);
+ /** @list: Element on list within struct perf_pmu. */
struct list_head list;
};
int perf_pmu_parse(struct list_head *list, char *name);
extern FILE *perf_pmu_in;
-static LIST_HEAD(pmus);
static bool hybrid_scanned;
/*
@@ -980,7 +997,6 @@ static struct perf_pmu *pmu_lookup(const char *lookup_name)
pmu->is_uncore = pmu_is_uncore(name);
if (pmu->is_uncore)
pmu->id = pmu_id(name);
- pmu->is_hybrid = is_hybrid;
pmu->max_precise = pmu_max_precise(name);
pmu_add_cpu_aliases(&aliases, pmu);
pmu_add_sys_aliases(&aliases, pmu);
@@ -992,7 +1008,7 @@ static struct perf_pmu *pmu_lookup(const char *lookup_name)
list_splice(&aliases, &pmu->aliases);
list_add_tail(&pmu->list, &pmus);
- if (pmu->is_hybrid)
+ if (is_hybrid)
list_add_tail(&pmu->hybrid_list, &perf_pmu__hybrid_pmus);
pmu->default_config = perf_pmu__get_default_config(pmu);
@@ -1065,11 +1081,15 @@ struct perf_pmu *evsel__find_pmu(struct evsel *evsel)
{
struct perf_pmu *pmu = NULL;
+ if (evsel->pmu)
+ return evsel->pmu;
+
while ((pmu = perf_pmu__scan(pmu)) != NULL) {
if (pmu->type == evsel->core.attr.type)
break;
}
+ evsel->pmu = pmu;
return pmu;
}
@@ -1534,8 +1554,8 @@ static int sub_non_neg(int a, int b)
return a - b;
}
-static char *format_alias(char *buf, int len, struct perf_pmu *pmu,
- struct perf_pmu_alias *alias)
+static char *format_alias(char *buf, int len, const struct perf_pmu *pmu,
+ const struct perf_pmu_alias *alias)
{
struct parse_events_term *term;
int used = snprintf(buf, len, "%s/%s", pmu->name, alias->name);
@@ -1560,72 +1580,60 @@ static char *format_alias(char *buf, int len, struct perf_pmu *pmu,
return buf;
}
-static char *format_alias_or(char *buf, int len, struct perf_pmu *pmu,
- struct perf_pmu_alias *alias)
-{
- snprintf(buf, len, "%s OR %s/%s/", alias->name, pmu->name, alias->name);
- return buf;
-}
-
+/** Struct for ordering events as output in perf list. */
struct sevent {
- char *name;
- char *desc;
- char *topic;
- char *str;
- char *pmu;
- char *metric_expr;
- char *metric_name;
- int is_cpu;
+ /** PMU for event. */
+ const struct perf_pmu *pmu;
+ /**
+ * Optional event for name, desc, etc. If not present then this is a
+ * selectable PMU and the event name is shown as "//".
+ */
+ const struct perf_pmu_alias *event;
+ /** Is the PMU for the CPU? */
+ bool is_cpu;
};
static int cmp_sevent(const void *a, const void *b)
{
const struct sevent *as = a;
const struct sevent *bs = b;
+ const char *a_pmu_name, *b_pmu_name;
+ const char *a_name = "//", *a_desc = NULL, *a_topic = "";
+ const char *b_name = "//", *b_desc = NULL, *b_topic = "";
int ret;
- /* Put extra events last */
- if (!!as->desc != !!bs->desc)
- return !!as->desc - !!bs->desc;
- if (as->topic && bs->topic) {
- int n = strcmp(as->topic, bs->topic);
-
- if (n)
- return n;
+ if (as->event) {
+ a_name = as->event->name;
+ a_desc = as->event->desc;
+ a_topic = as->event->topic ?: "";
}
+ if (bs->event) {
+ b_name = bs->event->name;
+ b_desc = bs->event->desc;
+ b_topic = bs->event->topic ?: "";
+ }
+ /* Put extra events last. */
+ if (!!a_desc != !!b_desc)
+ return !!a_desc - !!b_desc;
+
+ /* Order by topics. */
+ ret = strcmp(a_topic, b_topic);
+ if (ret)
+ return ret;
/* Order CPU core events to be first */
if (as->is_cpu != bs->is_cpu)
- return bs->is_cpu - as->is_cpu;
+ return as->is_cpu ? -1 : 1;
- ret = strcmp(as->name, bs->name);
- if (!ret) {
- if (as->pmu && bs->pmu)
- return strcmp(as->pmu, bs->pmu);
- }
+ /* Order by PMU name. */
+ a_pmu_name = as->pmu->name ?: "";
+ b_pmu_name = bs->pmu->name ?: "";
+ ret = strcmp(a_pmu_name, b_pmu_name);
+ if (ret)
+ return ret;
- return ret;
-}
-
-static void wordwrap(char *s, int start, int max, int corr)
-{
- int column = start;
- int n;
-
- while (*s) {
- int wlen = strcspn(s, " \t");
-
- if (column + wlen >= max && column > start) {
- printf("\n%*s", start, "");
- column = start + corr;
- }
- n = printf("%s%.*s", column > start ? " " : "", wlen, s);
- if (n <= 0)
- break;
- s += wlen;
- column += n;
- s = skip_spaces(s);
- }
+ /* Order by event name. */
+ return strcmp(a_name, b_name);
}
bool is_pmu_core(const char *name)
@@ -1636,147 +1644,127 @@ bool is_pmu_core(const char *name)
static bool pmu_alias_is_duplicate(struct sevent *alias_a,
struct sevent *alias_b)
{
+ const char *a_pmu_name, *b_pmu_name;
+ const char *a_name = alias_a->event ? alias_a->event->name : "//";
+ const char *b_name = alias_b->event ? alias_b->event->name : "//";
+
/* Different names -> never duplicates */
- if (strcmp(alias_a->name, alias_b->name))
+ if (strcmp(a_name, b_name))
return false;
- /* Don't remove duplicates for hybrid PMUs */
- if (perf_pmu__is_hybrid(alias_a->pmu) &&
- perf_pmu__is_hybrid(alias_b->pmu))
- return false;
-
- return true;
+ /* Don't remove duplicates for different PMUs */
+ a_pmu_name = alias_a->pmu->name ?: "";
+ b_pmu_name = alias_b->pmu->name ?: "";
+ return strcmp(a_pmu_name, b_pmu_name) == 0;
}
-void print_pmu_events(const char *event_glob, bool name_only, bool quiet_flag,
- bool long_desc, bool details_flag, bool deprecated,
- const char *pmu_name)
+void print_pmu_events(const struct print_callbacks *print_cb, void *print_state)
{
struct perf_pmu *pmu;
- struct perf_pmu_alias *alias;
+ struct perf_pmu_alias *event;
char buf[1024];
int printed = 0;
int len, j;
struct sevent *aliases;
- int numdesc = 0;
- int columns = pager_get_columns();
- char *topic = NULL;
pmu = NULL;
len = 0;
while ((pmu = perf_pmu__scan(pmu)) != NULL) {
- list_for_each_entry(alias, &pmu->aliases, list)
+ list_for_each_entry(event, &pmu->aliases, list)
len++;
if (pmu->selectable)
len++;
}
aliases = zalloc(sizeof(struct sevent) * len);
- if (!aliases)
- goto out_enomem;
+ if (!aliases) {
+ pr_err("FATAL: not enough memory to print PMU events\n");
+ return;
+ }
pmu = NULL;
j = 0;
while ((pmu = perf_pmu__scan(pmu)) != NULL) {
- if (pmu_name && perf_pmu__is_hybrid(pmu->name) &&
- strcmp(pmu_name, pmu->name)) {
- continue;
- }
+ bool is_cpu = is_pmu_core(pmu->name) || perf_pmu__is_hybrid(pmu->name);
- list_for_each_entry(alias, &pmu->aliases, list) {
- char *name = alias->desc ? alias->name :
- format_alias(buf, sizeof(buf), pmu, alias);
- bool is_cpu = is_pmu_core(pmu->name) ||
- perf_pmu__is_hybrid(pmu->name);
-
- if (alias->deprecated && !deprecated)
- continue;
-
- if (event_glob != NULL &&
- !(strglobmatch_nocase(name, event_glob) ||
- (!is_cpu && strglobmatch_nocase(alias->name,
- event_glob)) ||
- (alias->topic &&
- strglobmatch_nocase(alias->topic, event_glob))))
- continue;
-
- if (is_cpu && !name_only && !alias->desc)
- name = format_alias_or(buf, sizeof(buf), pmu, alias);
-
- aliases[j].name = name;
- if (is_cpu && !name_only && !alias->desc)
- aliases[j].name = format_alias_or(buf,
- sizeof(buf),
- pmu, alias);
- aliases[j].name = strdup(aliases[j].name);
- if (!aliases[j].name)
- goto out_enomem;
-
- aliases[j].desc = long_desc ? alias->long_desc :
- alias->desc;
- aliases[j].topic = alias->topic;
- aliases[j].str = alias->str;
- aliases[j].pmu = pmu->name;
- aliases[j].metric_expr = alias->metric_expr;
- aliases[j].metric_name = alias->metric_name;
+ list_for_each_entry(event, &pmu->aliases, list) {
+ aliases[j].event = event;
+ aliases[j].pmu = pmu;
aliases[j].is_cpu = is_cpu;
j++;
}
- if (pmu->selectable &&
- (event_glob == NULL || strglobmatch(pmu->name, event_glob))) {
- char *s;
- if (asprintf(&s, "%s//", pmu->name) < 0)
- goto out_enomem;
- aliases[j].name = s;
+ if (pmu->selectable) {
+ aliases[j].event = NULL;
+ aliases[j].pmu = pmu;
+ aliases[j].is_cpu = is_cpu;
j++;
}
}
len = j;
qsort(aliases, len, sizeof(struct sevent), cmp_sevent);
for (j = 0; j < len; j++) {
+ const char *name, *alias = NULL, *scale_unit = NULL,
+ *desc = NULL, *long_desc = NULL,
+ *encoding_desc = NULL, *topic = NULL,
+ *metric_name = NULL, *metric_expr = NULL;
+ bool deprecated = false;
+ size_t buf_used;
+
/* Skip duplicates */
if (j > 0 && pmu_alias_is_duplicate(&aliases[j], &aliases[j - 1]))
continue;
- if (name_only) {
- printf("%s ", aliases[j].name);
- continue;
+ if (!aliases[j].event) {
+ /* A selectable event. */
+ buf_used = snprintf(buf, sizeof(buf), "%s//", aliases[j].pmu->name) + 1;
+ name = buf;
+ } else {
+ if (aliases[j].event->desc) {
+ name = aliases[j].event->name;
+ buf_used = 0;
+ } else {
+ name = format_alias(buf, sizeof(buf), aliases[j].pmu,
+ aliases[j].event);
+ if (aliases[j].is_cpu) {
+ alias = name;
+ name = aliases[j].event->name;
+ }
+ buf_used = strlen(buf) + 1;
+ }
+ if (strlen(aliases[j].event->unit) || aliases[j].event->scale != 1.0) {
+ scale_unit = buf + buf_used;
+ buf_used += snprintf(buf + buf_used, sizeof(buf) - buf_used,
+ "%G%s", aliases[j].event->scale,
+ aliases[j].event->unit) + 1;
+ }
+ desc = aliases[j].event->desc;
+ long_desc = aliases[j].event->long_desc;
+ topic = aliases[j].event->topic;
+ encoding_desc = buf + buf_used;
+ buf_used += snprintf(buf + buf_used, sizeof(buf) - buf_used,
+ "%s/%s/", aliases[j].pmu->name,
+ aliases[j].event->str) + 1;
+ metric_name = aliases[j].event->metric_name;
+ metric_expr = aliases[j].event->metric_expr;
+ deprecated = aliases[j].event->deprecated;
}
- if (aliases[j].desc && !quiet_flag) {
- if (numdesc++ == 0)
- printf("\n");
- if (aliases[j].topic && (!topic ||
- strcmp(topic, aliases[j].topic))) {
- printf("%s%s:\n", topic ? "\n" : "",
- aliases[j].topic);
- topic = aliases[j].topic;
- }
- printf(" %-50s\n", aliases[j].name);
- printf("%*s", 8, "[");
- wordwrap(aliases[j].desc, 8, columns, 0);
- printf("]\n");
- if (details_flag) {
- printf("%*s%s/%s/ ", 8, "", aliases[j].pmu, aliases[j].str);
- if (aliases[j].metric_name)
- printf(" MetricName: %s", aliases[j].metric_name);
- if (aliases[j].metric_expr)
- printf(" MetricExpr: %s", aliases[j].metric_expr);
- putchar('\n');
- }
- } else
- printf(" %-50s [Kernel PMU event]\n", aliases[j].name);
- printed++;
+ print_cb->print_event(print_state,
+ aliases[j].pmu->name,
+ topic,
+ name,
+ alias,
+ scale_unit,
+ deprecated,
+ "Kernel PMU event",
+ desc,
+ long_desc,
+ encoding_desc,
+ metric_name,
+ metric_expr);
}
if (printed && pager_in_use())
printf("\n");
-out_free:
- for (j = 0; j < len; j++)
- zfree(&aliases[j].name);
+
zfree(&aliases);
return;
-
-out_enomem:
- printf("FATAL: not enough memory to print PMU events\n");
- if (aliases)
- goto out_free;
}
bool pmu_have_event(const char *pname, const char *name)
diff --git a/tools/perf/util/pmu.h b/tools/perf/util/pmu.h
index 68e15c38..69ca000 100644
--- a/tools/perf/util/pmu.h
+++ b/tools/perf/util/pmu.h
@@ -12,6 +12,7 @@
struct evsel_config_term;
struct perf_cpu_map;
+struct print_callbacks;
enum {
PERF_PMU_FORMAT_VALUE_CONFIG,
@@ -33,31 +34,101 @@ struct perf_pmu_caps {
struct list_head list;
};
+/**
+ * struct perf_pmu - hi
+ */
struct perf_pmu {
+ /** @name: The name of the PMU such as "cpu". */
char *name;
+ /**
+ * @alias_name: Optional alternate name for the PMU determined in
+ * architecture specific code.
+ */
char *alias_name;
+ /**
+ * @id: Optional PMU identifier read from
+ * <sysfs>/bus/event_source/devices/<name>/identifier.
+ */
char *id;
+ /**
+ * @type: Perf event attributed type value, read from
+ * <sysfs>/bus/event_source/devices/<name>/type.
+ */
__u32 type;
+ /**
+ * @selectable: Can the PMU name be selected as if it were an event?
+ */
bool selectable;
+ /**
+ * @is_uncore: Is the PMU not within the CPU core? Determined by the
+ * presence of <sysfs>/bus/event_source/devices/<name>/cpumask.
+ */
bool is_uncore;
- bool is_hybrid;
+ /**
+ * @auxtrace: Are events auxiliary events? Determined in architecture
+ * specific code.
+ */
bool auxtrace;
+ /**
+ * @max_precise: Number of levels of :ppp precision supported by the
+ * PMU, read from
+ * <sysfs>/bus/event_source/devices/<name>/caps/max_precise.
+ */
int max_precise;
+ /**
+ * @default_config: Optional default perf_event_attr determined in
+ * architecture specific code.
+ */
struct perf_event_attr *default_config;
+ /**
+ * @cpus: Empty or the contents of either of:
+ * <sysfs>/bus/event_source/devices/<name>/cpumask.
+ * <sysfs>/bus/event_source/devices/<cpu>/cpus.
+ */
struct perf_cpu_map *cpus;
- struct list_head format; /* HEAD struct perf_pmu_format -> list */
- struct list_head aliases; /* HEAD struct perf_pmu_alias -> list */
+ /**
+ * @format: Holds the contents of files read from
+ * <sysfs>/bus/event_source/devices/<name>/format/. The contents specify
+ * which event parameter changes what config, config1 or config2 bits.
+ */
+ struct list_head format;
+ /**
+ * @aliases: List of struct perf_pmu_alias. Each alias corresponds to an
+ * event read from <sysfs>/bus/event_source/devices/<name>/events/ or
+ * from json events in pmu-events.c.
+ */
+ struct list_head aliases;
+ /** @caps_initialized: Has the list caps been initialized? */
bool caps_initialized;
+ /** @nr_caps: The length of the list caps. */
u32 nr_caps;
- struct list_head caps; /* HEAD struct perf_pmu_caps -> list */
- struct list_head list; /* ELEM */
+ /**
+ * @caps: Holds the contents of files read from
+ * <sysfs>/bus/event_source/devices/<name>/caps/.
+ *
+ * The contents are pairs of the filename with the value of its
+ * contents, for example, max_precise (see above) may have a value of 3.
+ */
+ struct list_head caps;
+ /** @list: Element on pmus list in pmu.c. */
+ struct list_head list;
+ /** @hybrid_list: Element on perf_pmu__hybrid_pmus. */
struct list_head hybrid_list;
+ /**
+ * @missing_features: Features to inhibit when events on this PMU are
+ * opened.
+ */
struct {
+ /**
+ * @exclude_guest: Disables perf_event_attr exclude_guest and
+ * exclude_host.
+ */
bool exclude_guest;
} missing_features;
};
+/** @perf_pmu__fake: A special global PMU used for testing. */
extern struct perf_pmu perf_pmu__fake;
struct perf_pmu_info {
@@ -71,21 +142,60 @@ struct perf_pmu_info {
#define UNIT_MAX_LEN 31 /* max length for event unit name */
+/**
+ * struct perf_pmu_alias - An event either read from sysfs or builtin in
+ * pmu-events.c, created by parsing the pmu-events json files.
+ */
struct perf_pmu_alias {
+ /** @name: Name of the event like "mem-loads". */
char *name;
+ /** @desc: Optional short description of the event. */
char *desc;
+ /** @long_desc: Optional long description. */
char *long_desc;
+ /**
+ * @topic: Optional topic such as cache or pipeline, particularly for
+ * json events.
+ */
char *topic;
+ /**
+ * @str: Comma separated parameter list like
+ * "event=0xcd,umask=0x1,ldlat=0x3".
+ */
char *str;
- struct list_head terms; /* HEAD struct parse_events_term -> list */
- struct list_head list; /* ELEM */
+ /** @terms: Owned list of the original parsed parameters. */
+ struct list_head terms;
+ /** @list: List element of struct perf_pmu aliases. */
+ struct list_head list;
+ /** @unit: Units for the event, such as bytes or cache lines. */
char unit[UNIT_MAX_LEN+1];
+ /** @scale: Value to scale read counter values by. */
double scale;
+ /**
+ * @per_pkg: Does the file
+ * <sysfs>/bus/event_source/devices/<pmu_name>/events/<name>.per-pkg or
+ * equivalent json value exist and have the value 1.
+ */
bool per_pkg;
+ /**
+ * @snapshot: Does the file
+ * <sysfs>/bus/event_source/devices/<pmu_name>/events/<name>.snapshot
+ * exist and have the value 1.
+ */
bool snapshot;
+ /**
+ * @deprecated: Is the event hidden and so not shown in perf list by
+ * default.
+ */
bool deprecated;
+ /**
+ * @metric_expr: A metric expression associated with an event. Doing
+ * this makes little sense due to scale and unit applying to both.
+ */
char *metric_expr;
+ /** @metric_name: A name for the metric. unit applying to both. */
char *metric_name;
+ /** @pmu_name: The name copied from struct perf_pmu. */
char *pmu_name;
};
@@ -116,9 +226,7 @@ void perf_pmu__del_formats(struct list_head *formats);
struct perf_pmu *perf_pmu__scan(struct perf_pmu *pmu);
bool is_pmu_core(const char *name);
-void print_pmu_events(const char *event_glob, bool name_only, bool quiet,
- bool long_desc, bool details_flag,
- bool deprecated, const char *pmu_name);
+void print_pmu_events(const struct print_callbacks *print_cb, void *print_state);
bool pmu_have_event(const char *pname, const char *name);
int perf_pmu__scan_file(struct perf_pmu *pmu, const char *name, const char *fmt, ...) __scanf(3, 4);
diff --git a/tools/perf/util/pmus.c b/tools/perf/util/pmus.c
new file mode 100644
index 0000000..7f3b93c
--- /dev/null
+++ b/tools/perf/util/pmus.c
@@ -0,0 +1,5 @@
+// SPDX-License-Identifier: GPL-2.0
+#include <linux/list.h>
+#include <pmus.h>
+
+LIST_HEAD(pmus);
diff --git a/tools/perf/util/pmus.h b/tools/perf/util/pmus.h
new file mode 100644
index 0000000..5ec1200
--- /dev/null
+++ b/tools/perf/util/pmus.h
@@ -0,0 +1,9 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef __PMUS_H
+#define __PMUS_H
+
+extern struct list_head pmus;
+
+#define perf_pmus__for_each_pmu(pmu) list_for_each_entry(pmu, &pmus, list)
+
+#endif /* __PMUS_H */
diff --git a/tools/perf/util/print-events.c b/tools/perf/util/print-events.c
index c4d5d87..2646ae1 100644
--- a/tools/perf/util/print-events.c
+++ b/tools/perf/util/print-events.c
@@ -28,6 +28,7 @@
#define MAX_NAME_LEN 100
+/** Strings corresponding to enum perf_type_id. */
static const char * const event_type_descriptors[] = {
"Hardware event",
"Software event",
@@ -52,125 +53,77 @@ static const struct event_symbol event_symbols_tool[PERF_TOOL_MAX] = {
},
};
-static int cmp_string(const void *a, const void *b)
-{
- const char * const *as = a;
- const char * const *bs = b;
-
- return strcmp(*as, *bs);
-}
-
/*
* Print the events from <debugfs_mount_point>/tracing/events
*/
-void print_tracepoint_events(const char *subsys_glob,
- const char *event_glob, bool name_only)
+void print_tracepoint_events(const struct print_callbacks *print_cb, void *print_state)
{
- DIR *sys_dir, *evt_dir;
- struct dirent *sys_dirent, *evt_dirent;
- char evt_path[MAXPATHLEN];
- char *dir_path;
- char **evt_list = NULL;
- unsigned int evt_i = 0, evt_num = 0;
- bool evt_num_known = false;
+ struct dirent **sys_namelist = NULL;
+ int sys_items = tracing_events__scandir_alphasort(&sys_namelist);
-restart:
- sys_dir = tracing_events__opendir();
- if (!sys_dir)
- return;
+ for (int i = 0; i < sys_items; i++) {
+ struct dirent *sys_dirent = sys_namelist[i];
+ struct dirent **evt_namelist = NULL;
+ char *dir_path;
+ int evt_items;
- if (evt_num_known) {
- evt_list = zalloc(sizeof(char *) * evt_num);
- if (!evt_list)
- goto out_close_sys_dir;
- }
-
- for_each_subsystem(sys_dir, sys_dirent) {
- if (subsys_glob != NULL &&
- !strglobmatch(sys_dirent->d_name, subsys_glob))
+ if (sys_dirent->d_type != DT_DIR ||
+ !strcmp(sys_dirent->d_name, ".") ||
+ !strcmp(sys_dirent->d_name, ".."))
continue;
dir_path = get_events_file(sys_dirent->d_name);
if (!dir_path)
continue;
- evt_dir = opendir(dir_path);
- if (!evt_dir)
- goto next;
- for_each_event(dir_path, evt_dir, evt_dirent) {
- if (event_glob != NULL &&
- !strglobmatch(evt_dirent->d_name, event_glob))
+ evt_items = scandir(dir_path, &evt_namelist, NULL, alphasort);
+ for (int j = 0; j < evt_items; j++) {
+ struct dirent *evt_dirent = evt_namelist[j];
+ char evt_path[MAXPATHLEN];
+
+ if (evt_dirent->d_type != DT_DIR ||
+ !strcmp(evt_dirent->d_name, ".") ||
+ !strcmp(evt_dirent->d_name, ".."))
continue;
- if (!evt_num_known) {
- evt_num++;
+ if (tp_event_has_id(dir_path, evt_dirent) != 0)
continue;
- }
snprintf(evt_path, MAXPATHLEN, "%s:%s",
sys_dirent->d_name, evt_dirent->d_name);
-
- evt_list[evt_i] = strdup(evt_path);
- if (evt_list[evt_i] == NULL) {
- put_events_file(dir_path);
- goto out_close_evt_dir;
- }
- evt_i++;
+ print_cb->print_event(print_state,
+ /*topic=*/NULL,
+ /*pmu_name=*/NULL,
+ evt_path,
+ /*event_alias=*/NULL,
+ /*scale_unit=*/NULL,
+ /*deprecated=*/false,
+ "Tracepoint event",
+ /*desc=*/NULL,
+ /*long_desc=*/NULL,
+ /*encoding_desc=*/NULL,
+ /*metric_name=*/NULL,
+ /*metric_expr=*/NULL);
}
- closedir(evt_dir);
-next:
- put_events_file(dir_path);
+ free(dir_path);
+ free(evt_namelist);
}
- closedir(sys_dir);
-
- if (!evt_num_known) {
- evt_num_known = true;
- goto restart;
- }
- qsort(evt_list, evt_num, sizeof(char *), cmp_string);
- evt_i = 0;
- while (evt_i < evt_num) {
- if (name_only) {
- printf("%s ", evt_list[evt_i++]);
- continue;
- }
- printf(" %-50s [%s]\n", evt_list[evt_i++],
- event_type_descriptors[PERF_TYPE_TRACEPOINT]);
- }
- if (evt_num && pager_in_use())
- printf("\n");
-
-out_free:
- evt_num = evt_i;
- for (evt_i = 0; evt_i < evt_num; evt_i++)
- zfree(&evt_list[evt_i]);
- zfree(&evt_list);
- return;
-
-out_close_evt_dir:
- closedir(evt_dir);
-out_close_sys_dir:
- closedir(sys_dir);
-
- printf("FATAL: not enough memory to print %s\n",
- event_type_descriptors[PERF_TYPE_TRACEPOINT]);
- if (evt_list)
- goto out_free;
+ free(sys_namelist);
}
-void print_sdt_events(const char *subsys_glob, const char *event_glob,
- bool name_only)
+void print_sdt_events(const struct print_callbacks *print_cb, void *print_state)
{
- struct probe_cache *pcache;
- struct probe_cache_entry *ent;
struct strlist *bidlist, *sdtlist;
- struct strlist_config cfg = {.dont_dupstr = true};
- struct str_node *nd, *nd2;
- char *buf, *path, *ptr = NULL;
- bool show_detail = false;
- int ret;
+ struct str_node *bid_nd, *sdt_name, *next_sdt_name;
+ const char *last_sdt_name = NULL;
- sdtlist = strlist__new(NULL, &cfg);
+ /*
+ * The implicitly sorted sdtlist will hold the tracepoint name followed
+ * by @<buildid>. If the tracepoint name is unique (determined by
+ * looking at the adjacent nodes) the @<buildid> is dropped otherwise
+ * the executable path and buildid are added to the name.
+ */
+ sdtlist = strlist__new(NULL, NULL);
if (!sdtlist) {
pr_debug("Failed to allocate new strlist for SDT\n");
return;
@@ -180,354 +133,274 @@ void print_sdt_events(const char *subsys_glob, const char *event_glob,
pr_debug("Failed to get buildids: %d\n", errno);
return;
}
- strlist__for_each_entry(nd, bidlist) {
- pcache = probe_cache__new(nd->s, NULL);
+ strlist__for_each_entry(bid_nd, bidlist) {
+ struct probe_cache *pcache;
+ struct probe_cache_entry *ent;
+
+ pcache = probe_cache__new(bid_nd->s, NULL);
if (!pcache)
continue;
list_for_each_entry(ent, &pcache->entries, node) {
- if (!ent->sdt)
- continue;
- if (subsys_glob &&
- !strglobmatch(ent->pev.group, subsys_glob))
- continue;
- if (event_glob &&
- !strglobmatch(ent->pev.event, event_glob))
- continue;
- ret = asprintf(&buf, "%s:%s@%s", ent->pev.group,
- ent->pev.event, nd->s);
- if (ret > 0)
- strlist__add(sdtlist, buf);
+ char buf[1024];
+
+ snprintf(buf, sizeof(buf), "%s:%s@%s",
+ ent->pev.group, ent->pev.event, bid_nd->s);
+ strlist__add(sdtlist, buf);
}
probe_cache__delete(pcache);
}
strlist__delete(bidlist);
- strlist__for_each_entry(nd, sdtlist) {
- buf = strchr(nd->s, '@');
- if (buf)
- *(buf++) = '\0';
- if (name_only) {
- printf("%s ", nd->s);
- continue;
- }
- nd2 = strlist__next(nd);
- if (nd2) {
- ptr = strchr(nd2->s, '@');
- if (ptr)
- *ptr = '\0';
- if (strcmp(nd->s, nd2->s) == 0)
- show_detail = true;
- }
- if (show_detail) {
- path = build_id_cache__origname(buf);
- ret = asprintf(&buf, "%s@%s(%.12s)", nd->s, path, buf);
- if (ret > 0) {
- printf(" %-50s [%s]\n", buf, "SDT event");
- free(buf);
+ strlist__for_each_entry(sdt_name, sdtlist) {
+ bool show_detail = false;
+ char *bid = strchr(sdt_name->s, '@');
+ char *evt_name = NULL;
+
+ if (bid)
+ *(bid++) = '\0';
+
+ if (last_sdt_name && !strcmp(last_sdt_name, sdt_name->s)) {
+ show_detail = true;
+ } else {
+ next_sdt_name = strlist__next(sdt_name);
+ if (next_sdt_name) {
+ char *bid2 = strchr(next_sdt_name->s, '@');
+
+ if (bid2)
+ *bid2 = '\0';
+ if (strcmp(sdt_name->s, next_sdt_name->s) == 0)
+ show_detail = true;
+ if (bid2)
+ *bid2 = '@';
}
- free(path);
- } else
- printf(" %-50s [%s]\n", nd->s, "SDT event");
- if (nd2) {
- if (strcmp(nd->s, nd2->s) != 0)
- show_detail = false;
- if (ptr)
- *ptr = '@';
}
+ last_sdt_name = sdt_name->s;
+
+ if (show_detail) {
+ char *path = build_id_cache__origname(bid);
+
+ if (path) {
+ if (asprintf(&evt_name, "%s@%s(%.12s)", sdt_name->s, path, bid) < 0)
+ evt_name = NULL;
+ free(path);
+ }
+ }
+ print_cb->print_event(print_state,
+ /*topic=*/NULL,
+ /*pmu_name=*/NULL,
+ evt_name ?: sdt_name->s,
+ /*event_alias=*/NULL,
+ /*deprecated=*/false,
+ /*scale_unit=*/NULL,
+ "SDT event",
+ /*desc=*/NULL,
+ /*long_desc=*/NULL,
+ /*encoding_desc=*/NULL,
+ /*metric_name=*/NULL,
+ /*metric_expr=*/NULL);
+
+ free(evt_name);
}
strlist__delete(sdtlist);
}
-int print_hwcache_events(const char *event_glob, bool name_only)
+int print_hwcache_events(const struct print_callbacks *print_cb, void *print_state)
{
- unsigned int type, op, i, evt_i = 0, evt_num = 0, npmus = 0;
- char name[64], new_name[128];
- char **evt_list = NULL, **evt_pmus = NULL;
- bool evt_num_known = false;
- struct perf_pmu *pmu = NULL;
+ struct strlist *evt_name_list = strlist__new(NULL, NULL);
+ struct str_node *nd;
- if (perf_pmu__has_hybrid()) {
- npmus = perf_pmu__hybrid_pmu_num();
- evt_pmus = zalloc(sizeof(char *) * npmus);
- if (!evt_pmus)
- goto out_enomem;
+ if (!evt_name_list) {
+ pr_debug("Failed to allocate new strlist for hwcache events\n");
+ return -ENOMEM;
}
-
-restart:
- if (evt_num_known) {
- evt_list = zalloc(sizeof(char *) * evt_num);
- if (!evt_list)
- goto out_enomem;
- }
-
- for (type = 0; type < PERF_COUNT_HW_CACHE_MAX; type++) {
- for (op = 0; op < PERF_COUNT_HW_CACHE_OP_MAX; op++) {
+ for (int type = 0; type < PERF_COUNT_HW_CACHE_MAX; type++) {
+ for (int op = 0; op < PERF_COUNT_HW_CACHE_OP_MAX; op++) {
/* skip invalid cache type */
if (!evsel__is_cache_op_valid(type, op))
continue;
- for (i = 0; i < PERF_COUNT_HW_CACHE_RESULT_MAX; i++) {
- unsigned int hybrid_supported = 0, j;
- bool supported;
+ for (int i = 0; i < PERF_COUNT_HW_CACHE_RESULT_MAX; i++) {
+ struct perf_pmu *pmu = NULL;
+ char name[64];
__evsel__hw_cache_type_op_res_name(type, op, i, name, sizeof(name));
- if (event_glob != NULL && !strglobmatch(name, event_glob))
- continue;
-
if (!perf_pmu__has_hybrid()) {
- if (!is_event_supported(PERF_TYPE_HW_CACHE,
- type | (op << 8) | (i << 16))) {
- continue;
- }
- } else {
- perf_pmu__for_each_hybrid_pmu(pmu) {
- if (!evt_num_known) {
- evt_num++;
- continue;
- }
-
- supported = is_event_supported(
- PERF_TYPE_HW_CACHE,
- type | (op << 8) | (i << 16) |
- ((__u64)pmu->type << PERF_PMU_TYPE_SHIFT));
- if (supported) {
+ if (is_event_supported(PERF_TYPE_HW_CACHE,
+ type | (op << 8) | (i << 16)))
+ strlist__add(evt_name_list, name);
+ continue;
+ }
+ perf_pmu__for_each_hybrid_pmu(pmu) {
+ if (is_event_supported(PERF_TYPE_HW_CACHE,
+ type | (op << 8) | (i << 16) |
+ ((__u64)pmu->type << PERF_PMU_TYPE_SHIFT))) {
+ char new_name[128];
snprintf(new_name, sizeof(new_name),
"%s/%s/", pmu->name, name);
- evt_pmus[hybrid_supported] =
- strdup(new_name);
- hybrid_supported++;
- }
+ strlist__add(evt_name_list, new_name);
}
-
- if (hybrid_supported == 0)
- continue;
}
-
- if (!evt_num_known) {
- evt_num++;
- continue;
- }
-
- if ((hybrid_supported == 0) ||
- (hybrid_supported == npmus)) {
- evt_list[evt_i] = strdup(name);
- if (npmus > 0) {
- for (j = 0; j < npmus; j++)
- zfree(&evt_pmus[j]);
- }
- } else {
- for (j = 0; j < hybrid_supported; j++) {
- evt_list[evt_i++] = evt_pmus[j];
- evt_pmus[j] = NULL;
- }
- continue;
- }
-
- if (evt_list[evt_i] == NULL)
- goto out_enomem;
- evt_i++;
}
}
}
- if (!evt_num_known) {
- evt_num_known = true;
- goto restart;
+ strlist__for_each_entry(nd, evt_name_list) {
+ print_cb->print_event(print_state,
+ "cache",
+ /*pmu_name=*/NULL,
+ nd->s,
+ /*event_alias=*/NULL,
+ /*scale_unit=*/NULL,
+ /*deprecated=*/false,
+ event_type_descriptors[PERF_TYPE_HW_CACHE],
+ /*desc=*/NULL,
+ /*long_desc=*/NULL,
+ /*encoding_desc=*/NULL,
+ /*metric_name=*/NULL,
+ /*metric_expr=*/NULL);
}
-
- for (evt_i = 0; evt_i < evt_num; evt_i++) {
- if (!evt_list[evt_i])
- break;
- }
-
- evt_num = evt_i;
- qsort(evt_list, evt_num, sizeof(char *), cmp_string);
- evt_i = 0;
- while (evt_i < evt_num) {
- if (name_only) {
- printf("%s ", evt_list[evt_i++]);
- continue;
- }
- printf(" %-50s [%s]\n", evt_list[evt_i++],
- event_type_descriptors[PERF_TYPE_HW_CACHE]);
- }
- if (evt_num && pager_in_use())
- printf("\n");
-
-out_free:
- evt_num = evt_i;
- for (evt_i = 0; evt_i < evt_num; evt_i++)
- zfree(&evt_list[evt_i]);
- zfree(&evt_list);
-
- for (evt_i = 0; evt_i < npmus; evt_i++)
- zfree(&evt_pmus[evt_i]);
- zfree(&evt_pmus);
- return evt_num;
-
-out_enomem:
- printf("FATAL: not enough memory to print %s\n",
- event_type_descriptors[PERF_TYPE_HW_CACHE]);
- if (evt_list)
- goto out_free;
- return evt_num;
+ strlist__delete(evt_name_list);
+ return 0;
}
-static void print_tool_event(const struct event_symbol *syms, const char *event_glob,
- bool name_only)
-{
- if (syms->symbol == NULL)
- return;
-
- if (event_glob && !(strglobmatch(syms->symbol, event_glob) ||
- (syms->alias && strglobmatch(syms->alias, event_glob))))
- return;
-
- if (name_only)
- printf("%s ", syms->symbol);
- else {
- char name[MAX_NAME_LEN];
-
- if (syms->alias && strlen(syms->alias))
- snprintf(name, MAX_NAME_LEN, "%s OR %s", syms->symbol, syms->alias);
- else
- strlcpy(name, syms->symbol, MAX_NAME_LEN);
- printf(" %-50s [%s]\n", name, "Tool event");
- }
-}
-
-void print_tool_events(const char *event_glob, bool name_only)
+void print_tool_events(const struct print_callbacks *print_cb, void *print_state)
{
// Start at 1 because the first enum entry means no tool event.
- for (int i = 1; i < PERF_TOOL_MAX; ++i)
- print_tool_event(event_symbols_tool + i, event_glob, name_only);
-
- if (pager_in_use())
- printf("\n");
+ for (int i = 1; i < PERF_TOOL_MAX; ++i) {
+ print_cb->print_event(print_state,
+ "tool",
+ /*pmu_name=*/NULL,
+ event_symbols_tool[i].symbol,
+ event_symbols_tool[i].alias,
+ /*scale_unit=*/NULL,
+ /*deprecated=*/false,
+ "Tool event",
+ /*desc=*/NULL,
+ /*long_desc=*/NULL,
+ /*encoding_desc=*/NULL,
+ /*metric_name=*/NULL,
+ /*metric_expr=*/NULL);
+ }
}
-void print_symbol_events(const char *event_glob, unsigned int type,
- struct event_symbol *syms, unsigned int max,
- bool name_only)
+void print_symbol_events(const struct print_callbacks *print_cb, void *print_state,
+ unsigned int type, const struct event_symbol *syms,
+ unsigned int max)
{
- unsigned int i, evt_i = 0, evt_num = 0;
- char name[MAX_NAME_LEN];
- char **evt_list = NULL;
- bool evt_num_known = false;
+ struct strlist *evt_name_list = strlist__new(NULL, NULL);
+ struct str_node *nd;
-restart:
- if (evt_num_known) {
- evt_list = zalloc(sizeof(char *) * evt_num);
- if (!evt_list)
- goto out_enomem;
- syms -= max;
+ if (!evt_name_list) {
+ pr_debug("Failed to allocate new strlist for symbol events\n");
+ return;
}
-
- for (i = 0; i < max; i++, syms++) {
+ for (unsigned int i = 0; i < max; i++) {
/*
* New attr.config still not supported here, the latest
* example was PERF_COUNT_SW_CGROUP_SWITCHES
*/
- if (syms->symbol == NULL)
- continue;
-
- if (event_glob != NULL && !(strglobmatch(syms->symbol, event_glob) ||
- (syms->alias && strglobmatch(syms->alias, event_glob))))
+ if (syms[i].symbol == NULL)
continue;
if (!is_event_supported(type, i))
continue;
- if (!evt_num_known) {
- evt_num++;
- continue;
+ if (strlen(syms[i].alias)) {
+ char name[MAX_NAME_LEN];
+
+ snprintf(name, MAX_NAME_LEN, "%s OR %s", syms[i].symbol, syms[i].alias);
+ strlist__add(evt_name_list, name);
+ } else
+ strlist__add(evt_name_list, syms[i].symbol);
+ }
+
+ strlist__for_each_entry(nd, evt_name_list) {
+ char *alias = strstr(nd->s, " OR ");
+
+ if (alias) {
+ *alias = '\0';
+ alias += 4;
}
-
- if (!name_only && strlen(syms->alias))
- snprintf(name, MAX_NAME_LEN, "%s OR %s", syms->symbol, syms->alias);
- else
- strlcpy(name, syms->symbol, MAX_NAME_LEN);
-
- evt_list[evt_i] = strdup(name);
- if (evt_list[evt_i] == NULL)
- goto out_enomem;
- evt_i++;
+ print_cb->print_event(print_state,
+ /*topic=*/NULL,
+ /*pmu_name=*/NULL,
+ nd->s,
+ alias,
+ /*scale_unit=*/NULL,
+ /*deprecated=*/false,
+ event_type_descriptors[type],
+ /*desc=*/NULL,
+ /*long_desc=*/NULL,
+ /*encoding_desc=*/NULL,
+ /*metric_name=*/NULL,
+ /*metric_expr=*/NULL);
}
-
- if (!evt_num_known) {
- evt_num_known = true;
- goto restart;
- }
- qsort(evt_list, evt_num, sizeof(char *), cmp_string);
- evt_i = 0;
- while (evt_i < evt_num) {
- if (name_only) {
- printf("%s ", evt_list[evt_i++]);
- continue;
- }
- printf(" %-50s [%s]\n", evt_list[evt_i++], event_type_descriptors[type]);
- }
- if (evt_num && pager_in_use())
- printf("\n");
-
-out_free:
- evt_num = evt_i;
- for (evt_i = 0; evt_i < evt_num; evt_i++)
- zfree(&evt_list[evt_i]);
- zfree(&evt_list);
- return;
-
-out_enomem:
- printf("FATAL: not enough memory to print %s\n", event_type_descriptors[type]);
- if (evt_list)
- goto out_free;
+ strlist__delete(evt_name_list);
}
/*
* Print the help text for the event symbols:
*/
-void print_events(const char *event_glob, bool name_only, bool quiet_flag,
- bool long_desc, bool details_flag, bool deprecated,
- const char *pmu_name)
+void print_events(const struct print_callbacks *print_cb, void *print_state)
{
- print_symbol_events(event_glob, PERF_TYPE_HARDWARE,
- event_symbols_hw, PERF_COUNT_HW_MAX, name_only);
+ print_symbol_events(print_cb, print_state, PERF_TYPE_HARDWARE,
+ event_symbols_hw, PERF_COUNT_HW_MAX);
+ print_symbol_events(print_cb, print_state, PERF_TYPE_SOFTWARE,
+ event_symbols_sw, PERF_COUNT_SW_MAX);
- print_symbol_events(event_glob, PERF_TYPE_SOFTWARE,
- event_symbols_sw, PERF_COUNT_SW_MAX, name_only);
- print_tool_events(event_glob, name_only);
+ print_tool_events(print_cb, print_state);
- print_hwcache_events(event_glob, name_only);
+ print_hwcache_events(print_cb, print_state);
- print_pmu_events(event_glob, name_only, quiet_flag, long_desc,
- details_flag, deprecated, pmu_name);
+ print_pmu_events(print_cb, print_state);
- if (event_glob != NULL)
- return;
+ print_cb->print_event(print_state,
+ /*topic=*/NULL,
+ /*pmu_name=*/NULL,
+ "rNNN",
+ /*event_alias=*/NULL,
+ /*scale_unit=*/NULL,
+ /*deprecated=*/false,
+ event_type_descriptors[PERF_TYPE_RAW],
+ /*desc=*/NULL,
+ /*long_desc=*/NULL,
+ /*encoding_desc=*/NULL,
+ /*metric_name=*/NULL,
+ /*metric_expr=*/NULL);
- if (!name_only) {
- printf(" %-50s [%s]\n",
- "rNNN",
- event_type_descriptors[PERF_TYPE_RAW]);
- printf(" %-50s [%s]\n",
- "cpu/t1=v1[,t2=v2,t3 ...]/modifier",
- event_type_descriptors[PERF_TYPE_RAW]);
- if (pager_in_use())
- printf(" (see 'man perf-list' on how to encode it)\n\n");
+ print_cb->print_event(print_state,
+ /*topic=*/NULL,
+ /*pmu_name=*/NULL,
+ "cpu/t1=v1[,t2=v2,t3 ...]/modifier",
+ /*event_alias=*/NULL,
+ /*scale_unit=*/NULL,
+ /*deprecated=*/false,
+ event_type_descriptors[PERF_TYPE_RAW],
+ "(see 'man perf-list' on how to encode it)",
+ /*long_desc=*/NULL,
+ /*encoding_desc=*/NULL,
+ /*metric_name=*/NULL,
+ /*metric_expr=*/NULL);
- printf(" %-50s [%s]\n",
- "mem:<addr>[/len][:access]",
- event_type_descriptors[PERF_TYPE_BREAKPOINT]);
- if (pager_in_use())
- printf("\n");
- }
+ print_cb->print_event(print_state,
+ /*topic=*/NULL,
+ /*pmu_name=*/NULL,
+ "mem:<addr>[/len][:access]",
+ /*scale_unit=*/NULL,
+ /*event_alias=*/NULL,
+ /*deprecated=*/false,
+ event_type_descriptors[PERF_TYPE_BREAKPOINT],
+ /*desc=*/NULL,
+ /*long_desc=*/NULL,
+ /*encoding_desc=*/NULL,
+ /*metric_name=*/NULL,
+ /*metric_expr=*/NULL);
- print_tracepoint_events(NULL, NULL, name_only);
+ print_tracepoint_events(print_cb, print_state);
- print_sdt_events(NULL, NULL, name_only);
+ print_sdt_events(print_cb, print_state);
- metricgroup__print(true, true, NULL, name_only, details_flag,
- pmu_name);
+ metricgroup__print(print_cb, print_state);
- print_libpfm_events(name_only, long_desc);
+ print_libpfm_events(print_cb, print_state);
}
diff --git a/tools/perf/util/print-events.h b/tools/perf/util/print-events.h
index 1da9910..c237e53 100644
--- a/tools/perf/util/print-events.h
+++ b/tools/perf/util/print-events.h
@@ -2,21 +2,39 @@
#ifndef __PERF_PRINT_EVENTS_H
#define __PERF_PRINT_EVENTS_H
+#include <linux/perf_event.h>
#include <stdbool.h>
struct event_symbol;
-void print_events(const char *event_glob, bool name_only, bool quiet_flag,
- bool long_desc, bool details_flag, bool deprecated,
- const char *pmu_name);
-int print_hwcache_events(const char *event_glob, bool name_only);
-void print_sdt_events(const char *subsys_glob, const char *event_glob,
- bool name_only);
-void print_symbol_events(const char *event_glob, unsigned int type,
- struct event_symbol *syms, unsigned int max,
- bool name_only);
-void print_tool_events(const char *event_glob, bool name_only);
-void print_tracepoint_events(const char *subsys_glob, const char *event_glob,
- bool name_only);
+struct print_callbacks {
+ void (*print_start)(void *print_state);
+ void (*print_end)(void *print_state);
+ void (*print_event)(void *print_state, const char *topic,
+ const char *pmu_name,
+ const char *event_name, const char *event_alias,
+ const char *scale_unit,
+ bool deprecated, const char *event_type_desc,
+ const char *desc, const char *long_desc,
+ const char *encoding_desc,
+ const char *metric_name, const char *metric_expr);
+ void (*print_metric)(void *print_state,
+ const char *group,
+ const char *name,
+ const char *desc,
+ const char *long_desc,
+ const char *expr,
+ const char *unit);
+};
+
+/** Print all events, the default when no options are specified. */
+void print_events(const struct print_callbacks *print_cb, void *print_state);
+int print_hwcache_events(const struct print_callbacks *print_cb, void *print_state);
+void print_sdt_events(const struct print_callbacks *print_cb, void *print_state);
+void print_symbol_events(const struct print_callbacks *print_cb, void *print_state,
+ unsigned int type, const struct event_symbol *syms,
+ unsigned int max);
+void print_tool_events(const struct print_callbacks *print_cb, void *print_state);
+void print_tracepoint_events(const struct print_callbacks *print_cb, void *print_state);
#endif /* __PERF_PRINT_EVENTS_H */
diff --git a/tools/perf/util/probe-finder.c b/tools/perf/util/probe-finder.c
index 50d861a..54b49ce 100644
--- a/tools/perf/util/probe-finder.c
+++ b/tools/perf/util/probe-finder.c
@@ -763,7 +763,7 @@ static int find_best_scope_cb(Dwarf_Die *fn_die, void *data)
/* Skip if declared file name does not match */
if (fsp->file) {
- file = dwarf_decl_file(fn_die);
+ file = die_get_decl_file(fn_die);
if (!file || strcmp(fsp->file, file) != 0)
return 0;
}
@@ -1063,6 +1063,7 @@ static int probe_point_search_cb(Dwarf_Die *sp_die, void *data)
struct dwarf_callback_param *param = data;
struct probe_finder *pf = param->data;
struct perf_probe_point *pp = &pf->pev->point;
+ const char *fname;
/* Check tag and diename */
if (!die_is_func_def(sp_die) ||
@@ -1070,12 +1071,17 @@ static int probe_point_search_cb(Dwarf_Die *sp_die, void *data)
return DWARF_CB_OK;
/* Check declared file */
- if (pp->file && strtailcmp(pp->file, dwarf_decl_file(sp_die)))
+ fname = die_get_decl_file(sp_die);
+ if (!fname) {
+ pr_warning("A function DIE doesn't have decl_line. Maybe broken DWARF?\n");
+ return DWARF_CB_OK;
+ }
+ if (pp->file && fname && strtailcmp(pp->file, fname))
return DWARF_CB_OK;
pr_debug("Matched function: %s [%lx]\n", dwarf_diename(sp_die),
(unsigned long)dwarf_dieoffset(sp_die));
- pf->fname = dwarf_decl_file(sp_die);
+ pf->fname = fname;
if (pp->line) { /* Function relative line */
dwarf_decl_line(sp_die, &pf->lno);
pf->lno += pp->line;
@@ -1134,6 +1140,7 @@ struct pubname_callback_param {
static int pubname_search_cb(Dwarf *dbg, Dwarf_Global *gl, void *data)
{
struct pubname_callback_param *param = data;
+ const char *fname;
if (dwarf_offdie(dbg, gl->die_offset, param->sp_die)) {
if (dwarf_tag(param->sp_die) != DW_TAG_subprogram)
@@ -1143,9 +1150,11 @@ static int pubname_search_cb(Dwarf *dbg, Dwarf_Global *gl, void *data)
if (!dwarf_offdie(dbg, gl->cu_offset, param->cu_die))
return DWARF_CB_OK;
- if (param->file &&
- strtailcmp(param->file, dwarf_decl_file(param->sp_die)))
- return DWARF_CB_OK;
+ if (param->file) {
+ fname = die_get_decl_file(param->sp_die);
+ if (!fname || strtailcmp(param->file, fname))
+ return DWARF_CB_OK;
+ }
param->found = 1;
return DWARF_CB_ABORT;
@@ -1741,7 +1750,7 @@ int debuginfo__find_probe_point(struct debuginfo *dbg, u64 addr,
goto post;
}
- fname = dwarf_decl_file(&spdie);
+ fname = die_get_decl_file(&spdie);
if (addr == baseaddr) {
/* Function entry - Relative line number is 0 */
lineno = baseline;
@@ -1778,8 +1787,8 @@ int debuginfo__find_probe_point(struct debuginfo *dbg, u64 addr,
}
}
/* Verify the lineno and baseline are in a same file */
- tmp = dwarf_decl_file(&spdie);
- if (!tmp || strcmp(tmp, fname) != 0)
+ tmp = die_get_decl_file(&spdie);
+ if (!tmp || (fname && strcmp(tmp, fname) != 0))
lineno = 0;
}
@@ -1889,13 +1898,17 @@ static int line_range_search_cb(Dwarf_Die *sp_die, void *data)
struct dwarf_callback_param *param = data;
struct line_finder *lf = param->data;
struct line_range *lr = lf->lr;
+ const char *fname;
/* Check declared file */
- if (lr->file && strtailcmp(lr->file, dwarf_decl_file(sp_die)))
- return DWARF_CB_OK;
+ if (lr->file) {
+ fname = die_get_decl_file(sp_die);
+ if (!fname || strtailcmp(lr->file, fname))
+ return DWARF_CB_OK;
+ }
if (die_match_name(sp_die, lr->function) && die_is_func_def(sp_die)) {
- lf->fname = dwarf_decl_file(sp_die);
+ lf->fname = die_get_decl_file(sp_die);
dwarf_decl_line(sp_die, &lr->offset);
pr_debug("fname: %s, lineno:%d\n", lf->fname, lr->offset);
lf->lno_s = lr->offset + lr->start;
diff --git a/tools/perf/util/python.c b/tools/perf/util/python.c
index 5be5fa2..212031b 100644
--- a/tools/perf/util/python.c
+++ b/tools/perf/util/python.c
@@ -5,7 +5,9 @@
#include <poll.h>
#include <linux/err.h>
#include <perf/cpumap.h>
+#ifdef HAVE_LIBTRACEEVENT
#include <traceevent/event-parse.h>
+#endif
#include <perf/mmap.h>
#include "evlist.h"
#include "callchain.h"
@@ -417,6 +419,7 @@ static PyObject *pyrf_sample_event__repr(struct pyrf_event *pevent)
return ret;
}
+#ifdef HAVE_LIBTRACEEVENT
static bool is_tracepoint(struct pyrf_event *pevent)
{
return pevent->evsel->core.attr.type == PERF_TYPE_TRACEPOINT;
@@ -439,8 +442,10 @@ tracepoint_field(struct pyrf_event *pe, struct tep_format_field *field)
offset = val;
len = offset >> 16;
offset &= 0xffff;
+#ifdef HAVE_LIBTRACEEVENT_TEP_FIELD_IS_RELATIVE
if (field->flags & TEP_FIELD_IS_RELATIVE)
offset += field->offset + field->size;
+#endif
}
if (field->flags & TEP_FIELD_IS_STRING &&
is_printable_array(data + offset, len)) {
@@ -486,14 +491,17 @@ get_tracepoint_field(struct pyrf_event *pevent, PyObject *attr_name)
return tracepoint_field(pevent, field);
}
+#endif /* HAVE_LIBTRACEEVENT */
static PyObject*
pyrf_sample_event__getattro(struct pyrf_event *pevent, PyObject *attr_name)
{
PyObject *obj = NULL;
+#ifdef HAVE_LIBTRACEEVENT
if (is_tracepoint(pevent))
obj = get_tracepoint_field(pevent, attr_name);
+#endif
return obj ?: PyObject_GenericGetAttr((PyObject *) pevent, attr_name);
}
@@ -718,17 +726,17 @@ static Py_ssize_t pyrf_thread_map__length(PyObject *obj)
{
struct pyrf_thread_map *pthreads = (void *)obj;
- return pthreads->threads->nr;
+ return perf_thread_map__nr(pthreads->threads);
}
static PyObject *pyrf_thread_map__item(PyObject *obj, Py_ssize_t i)
{
struct pyrf_thread_map *pthreads = (void *)obj;
- if (i >= pthreads->threads->nr)
+ if (i >= perf_thread_map__nr(pthreads->threads))
return NULL;
- return Py_BuildValue("i", pthreads->threads->map[i]);
+ return Py_BuildValue("i", perf_thread_map__pid(pthreads->threads, i));
}
static PySequenceMethods pyrf_thread_map__sequence_methods = {
@@ -1134,14 +1142,6 @@ static PyObject *pyrf_evlist__open(struct pyrf_evlist *pevlist,
PyObject *args, PyObject *kwargs)
{
struct evlist *evlist = &pevlist->evlist;
- int group = 0;
- static char *kwlist[] = { "group", NULL };
-
- if (!PyArg_ParseTupleAndKeywords(args, kwargs, "|OOii", kwlist, &group))
- return NULL;
-
- if (group)
- evlist__set_leader(evlist);
if (evlist__open(evlist) < 0) {
PyErr_SetFromErrno(PyExc_OSError);
@@ -1326,6 +1326,9 @@ static struct {
static PyObject *pyrf__tracepoint(struct pyrf_evsel *pevsel,
PyObject *args, PyObject *kwargs)
{
+#ifndef HAVE_LIBTRACEEVENT
+ return NULL;
+#else
struct tep_event *tp_format;
static char *kwlist[] = { "sys", "name", NULL };
char *sys = NULL;
@@ -1340,6 +1343,7 @@ static PyObject *pyrf__tracepoint(struct pyrf_evsel *pevsel,
return _PyLong_FromLong(-1);
return _PyLong_FromLong(tp_format->id);
+#endif // HAVE_LIBTRACEEVENT
}
static PyMethodDef perf__methods[] = {
diff --git a/tools/perf/util/record.c b/tools/perf/util/record.c
index 7b58f6c..9eb5c6a 100644
--- a/tools/perf/util/record.c
+++ b/tools/perf/util/record.c
@@ -99,13 +99,6 @@ void evlist__config(struct evlist *evlist, struct record_opts *opts, struct call
bool use_comm_exec;
bool sample_id = opts->sample_id;
- /*
- * Set the evsel leader links before we configure attributes,
- * since some might depend on this info.
- */
- if (opts->group)
- evlist__set_leader(evlist);
-
if (perf_cpu_map__cpu(evlist->core.user_requested_cpus, 0).cpu < 0)
opts->no_inherit = true;
diff --git a/tools/perf/util/record.h b/tools/perf/util/record.h
index 4269e91..46212bf 100644
--- a/tools/perf/util/record.h
+++ b/tools/perf/util/record.h
@@ -13,7 +13,6 @@ struct option;
struct record_opts {
struct target target;
- bool group;
bool inherit_stat;
bool no_buffering;
bool no_inherit;
diff --git a/tools/perf/util/s390-cpumsf.c b/tools/perf/util/s390-cpumsf.c
index f3fdad2..6fe478b 100644
--- a/tools/perf/util/s390-cpumsf.c
+++ b/tools/perf/util/s390-cpumsf.c
@@ -163,6 +163,7 @@
#include "s390-cpumsf-kernel.h"
#include "s390-cpumcf-kernel.h"
#include "config.h"
+#include "util/sample.h"
struct s390_cpumsf {
struct auxtrace auxtrace;
diff --git a/tools/perf/util/s390-sample-raw.c b/tools/perf/util/s390-sample-raw.c
index 9a631d9..c10b891 100644
--- a/tools/perf/util/s390-sample-raw.c
+++ b/tools/perf/util/s390-sample-raw.c
@@ -28,6 +28,7 @@
#include "sample-raw.h"
#include "s390-cpumcf-kernel.h"
#include "pmu-events/pmu-events.h"
+#include "util/sample.h"
static size_t ctrset_size(struct cf_ctrset_entry *set)
{
diff --git a/tools/perf/util/sample.h b/tools/perf/util/sample.h
new file mode 100644
index 0000000..60ec79d
--- /dev/null
+++ b/tools/perf/util/sample.h
@@ -0,0 +1,117 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef __PERF_SAMPLE_H
+#define __PERF_SAMPLE_H
+
+#include <linux/perf_event.h>
+#include <linux/types.h>
+
+/* number of register is bound by the number of bits in regs_dump::mask (64) */
+#define PERF_SAMPLE_REGS_CACHE_SIZE (8 * sizeof(u64))
+
+struct regs_dump {
+ u64 abi;
+ u64 mask;
+ u64 *regs;
+
+ /* Cached values/mask filled by first register access. */
+ u64 cache_regs[PERF_SAMPLE_REGS_CACHE_SIZE];
+ u64 cache_mask;
+};
+
+struct stack_dump {
+ u16 offset;
+ u64 size;
+ char *data;
+};
+
+struct sample_read_value {
+ u64 value;
+ u64 id; /* only if PERF_FORMAT_ID */
+ u64 lost; /* only if PERF_FORMAT_LOST */
+};
+
+struct sample_read {
+ u64 time_enabled;
+ u64 time_running;
+ union {
+ struct {
+ u64 nr;
+ struct sample_read_value *values;
+ } group;
+ struct sample_read_value one;
+ };
+};
+
+static inline size_t sample_read_value_size(u64 read_format)
+{
+ /* PERF_FORMAT_ID is forced for PERF_SAMPLE_READ */
+ if (read_format & PERF_FORMAT_LOST)
+ return sizeof(struct sample_read_value);
+ else
+ return offsetof(struct sample_read_value, lost);
+}
+
+static inline struct sample_read_value *next_sample_read_value(struct sample_read_value *v, u64 read_format)
+{
+ return (void *)v + sample_read_value_size(read_format);
+}
+
+#define sample_read_group__for_each(v, nr, rf) \
+ for (int __i = 0; __i < (int)nr; v = next_sample_read_value(v, rf), __i++)
+
+#define MAX_INSN 16
+
+struct aux_sample {
+ u64 size;
+ void *data;
+};
+
+struct perf_sample {
+ u64 ip;
+ u32 pid, tid;
+ u64 time;
+ u64 addr;
+ u64 id;
+ u64 stream_id;
+ u64 period;
+ u64 weight;
+ u64 transaction;
+ u64 insn_cnt;
+ u64 cyc_cnt;
+ u32 cpu;
+ u32 raw_size;
+ u64 data_src;
+ u64 phys_addr;
+ u64 data_page_size;
+ u64 code_page_size;
+ u64 cgroup;
+ u32 flags;
+ u32 machine_pid;
+ u32 vcpu;
+ u16 insn_len;
+ u8 cpumode;
+ u16 misc;
+ u16 ins_lat;
+ u16 p_stage_cyc;
+ bool no_hw_idx; /* No hw_idx collected in branch_stack */
+ char insn[MAX_INSN];
+ void *raw_data;
+ struct ip_callchain *callchain;
+ struct branch_stack *branch_stack;
+ struct regs_dump user_regs;
+ struct regs_dump intr_regs;
+ struct stack_dump user_stack;
+ struct sample_read read;
+ struct aux_sample aux_sample;
+};
+
+/*
+ * raw_data is always 4 bytes from an 8-byte boundary, so subtract 4 to get
+ * 8-byte alignment.
+ */
+static inline void *perf_sample__synth_ptr(struct perf_sample *sample)
+{
+ return sample->raw_data - 4;
+}
+
+#endif /* __PERF_SAMPLE_H */
diff --git a/tools/perf/util/scripting-engines/Build b/tools/perf/util/scripting-engines/Build
index 0f5ba28..d47820c 100644
--- a/tools/perf/util/scripting-engines/Build
+++ b/tools/perf/util/scripting-engines/Build
@@ -1,5 +1,7 @@
-perf-$(CONFIG_LIBPERL) += trace-event-perl.o
-perf-$(CONFIG_LIBPYTHON) += trace-event-python.o
+ifeq ($(CONFIG_LIBTRACEEVENT),y)
+ perf-$(CONFIG_LIBPERL) += trace-event-perl.o
+ perf-$(CONFIG_LIBPYTHON) += trace-event-python.o
+endif
CFLAGS_trace-event-perl.o += $(PERL_EMBED_CCOPTS) -Wno-redundant-decls -Wno-strict-prototypes -Wno-unused-parameter -Wno-shadow -Wno-nested-externs -Wno-undef -Wno-switch-default -Wno-bad-function-cast -Wno-declaration-after-statement -Wno-switch-enum
diff --git a/tools/perf/util/scripting-engines/trace-event-perl.c b/tools/perf/util/scripting-engines/trace-event-perl.c
index 5b602b6..c097b79 100644
--- a/tools/perf/util/scripting-engines/trace-event-perl.c
+++ b/tools/perf/util/scripting-engines/trace-event-perl.c
@@ -27,6 +27,7 @@
#include <errno.h>
#include <linux/bitmap.h>
#include <linux/time64.h>
+#include <traceevent/event-parse.h>
#include <stdbool.h>
/* perl needs the following define, right after including stdbool.h */
@@ -392,8 +393,10 @@ static void perl_process_tracepoint(struct perf_sample *sample,
if (field->flags & TEP_FIELD_IS_DYNAMIC) {
offset = *(int *)(data + field->offset);
offset &= 0xffff;
+#ifdef HAVE_LIBTRACEEVENT_TEP_FIELD_IS_RELATIVE
if (field->flags & TEP_FIELD_IS_RELATIVE)
offset += field->offset + field->size;
+#endif
} else
offset = field->offset;
XPUSHs(sv_2mortal(newSVpv((char *)data + offset, 0)));
diff --git a/tools/perf/util/scripting-engines/trace-event-python.c b/tools/perf/util/scripting-engines/trace-event-python.c
index 0f229fa..e930f5f 100644
--- a/tools/perf/util/scripting-engines/trace-event-python.c
+++ b/tools/perf/util/scripting-engines/trace-event-python.c
@@ -30,6 +30,7 @@
#include <linux/bitmap.h>
#include <linux/compiler.h>
#include <linux/time64.h>
+#include <traceevent/event-parse.h>
#include "../build-id.h"
#include "../counts.h"
@@ -52,6 +53,7 @@
#include "print_binary.h"
#include "stat.h"
#include "mem-events.h"
+#include "util/perf_regs.h"
#if PY_MAJOR_VERSION < 3
#define _PyUnicode_FromString(arg) \
@@ -992,8 +994,10 @@ static void python_process_tracepoint(struct perf_sample *sample,
offset = val;
len = offset >> 16;
offset &= 0xffff;
+#ifdef HAVE_LIBTRACEEVENT_TEP_FIELD_IS_RELATIVE
if (field->flags & TEP_FIELD_IS_RELATIVE)
offset += field->offset + field->size;
+#endif
}
if (field->flags & TEP_FIELD_IS_STRING &&
is_printable_array(data + offset, len)) {
@@ -1653,13 +1657,7 @@ static void python_process_stat(struct perf_stat_config *config,
struct perf_cpu_map *cpus = counter->core.cpus;
int cpu, thread;
- if (config->aggr_mode == AGGR_GLOBAL) {
- process_stat(counter, (struct perf_cpu){ .cpu = -1 }, -1, tstamp,
- &counter->counts->aggr);
- return;
- }
-
- for (thread = 0; thread < threads->nr; thread++) {
+ for (thread = 0; thread < perf_thread_map__nr(threads); thread++) {
for (cpu = 0; cpu < perf_cpu_map__nr(cpus); cpu++) {
process_stat(counter, perf_cpu_map__cpu(cpus, cpu),
perf_thread_map__pid(threads, thread), tstamp,
diff --git a/tools/perf/util/session.c b/tools/perf/util/session.c
index 873fd51..7c021c6 100644
--- a/tools/perf/util/session.c
+++ b/tools/perf/util/session.c
@@ -1,5 +1,6 @@
// SPDX-License-Identifier: GPL-2.0
#include <errno.h>
+#include <signal.h>
#include <inttypes.h>
#include <linux/err.h>
#include <linux/kernel.h>
@@ -313,7 +314,9 @@ void perf_session__delete(struct perf_session *session)
evlist__delete(session->evlist);
perf_data__close(session->data);
}
+#ifdef HAVE_LIBTRACEEVENT
trace_event__cleanup(&session->tevent);
+#endif
free(session);
}
@@ -2022,7 +2025,7 @@ static int perf_session__flush_thread_stacks(struct perf_session *session)
NULL);
}
-volatile int session_done;
+volatile sig_atomic_t session_done;
static int __perf_session__process_decomp_events(struct perf_session *session);
diff --git a/tools/perf/util/session.h b/tools/perf/util/session.h
index be5871e..ee3715e 100644
--- a/tools/perf/util/session.h
+++ b/tools/perf/util/session.h
@@ -33,7 +33,9 @@ struct perf_session {
struct auxtrace *auxtrace;
struct itrace_synth_opts *itrace_synth_opts;
struct list_head auxtrace_index;
+#ifdef HAVE_LIBTRACEEVENT
struct trace_event tevent;
+#endif
struct perf_record_time_conv time_conv;
bool repipe;
bool one_mmap;
diff --git a/tools/perf/util/setup.py b/tools/perf/util/setup.py
index 5b1e646..4f265d0 100644
--- a/tools/perf/util/setup.py
+++ b/tools/perf/util/setup.py
@@ -7,7 +7,7 @@
src_feature_tests = getenv('srctree') + '/tools/build/feature'
def clang_has_option(option):
- cc_output = Popen([cc, option, path.join(src_feature_tests, "test-hello.c") ], stderr=PIPE).stderr.readlines()
+ cc_output = Popen([cc.split()[0], str(cc.split()[1:]) + option, path.join(src_feature_tests, "test-hello.c") ], stderr=PIPE).stderr.readlines()
return [o for o in cc_output if ((b"unknown argument" in o) or (b"is not supported" in o))] == [ ]
if cc_is_clang:
@@ -63,12 +63,18 @@
ext_sources = [f.strip() for f in open('util/python-ext-sources')
if len(f.strip()) > 0 and f[0] != '#']
+extra_libraries = []
+
+if '-DHAVE_LIBTRACEEVENT' in cflags:
+ extra_libraries += [ 'traceevent' ]
+else:
+ ext_sources.remove('util/trace-event.c')
+
# use full paths with source files
ext_sources = list(map(lambda x: '%s/%s' % (src_perf, x) , ext_sources))
-extra_libraries = []
if '-DHAVE_LIBNUMA_SUPPORT' in cflags:
- extra_libraries = [ 'numa' ]
+ extra_libraries += [ 'numa' ]
if '-DHAVE_LIBCAP_SUPPORT' in cflags:
extra_libraries += [ 'cap' ]
@@ -77,7 +83,8 @@
include_dirs = ['util/include'],
libraries = extra_libraries,
extra_compile_args = cflags,
- extra_objects = [libtraceevent, libapikfs, libperf],
+ extra_objects = [ x for x in [libtraceevent, libapikfs, libperf]
+ if x is not None],
)
setup(name='perf',
diff --git a/tools/perf/util/sort.c b/tools/perf/util/sort.c
index 2e73308..0ecc2cb 100644
--- a/tools/perf/util/sort.c
+++ b/tools/perf/util/sort.c
@@ -22,7 +22,6 @@
#include "srcline.h"
#include "strlist.h"
#include "strbuf.h"
-#include <traceevent/event-parse.h>
#include "mem-events.h"
#include "annotate.h"
#include "event.h"
@@ -32,6 +31,10 @@
#include <linux/kernel.h>
#include <linux/string.h>
+#ifdef HAVE_LIBTRACEEVENT
+#include <traceevent/event-parse.h>
+#endif
+
regex_t parent_regex;
const char default_parent_pattern[] = "^sys_|^do_page_fault";
const char *parent_pattern = default_parent_pattern;
@@ -743,6 +746,7 @@ struct sort_entry sort_time = {
/* --sort trace */
+#ifdef HAVE_LIBTRACEEVENT
static char *get_trace_output(struct hist_entry *he)
{
struct trace_seq seq;
@@ -806,6 +810,7 @@ struct sort_entry sort_trace = {
.se_snprintf = hist_entry__trace_snprintf,
.se_width_idx = HISTC_TRACE,
};
+#endif /* HAVE_LIBTRACEEVENT */
/* sort keys for branch stacks */
@@ -2022,7 +2027,9 @@ static struct sort_dimension common_sort_dimensions[] = {
DIM(SORT_LOCAL_WEIGHT, "local_weight", sort_local_weight),
DIM(SORT_GLOBAL_WEIGHT, "weight", sort_global_weight),
DIM(SORT_TRANSACTION, "transaction", sort_transaction),
+#ifdef HAVE_LIBTRACEEVENT
DIM(SORT_TRACE, "trace", sort_trace),
+#endif
DIM(SORT_SYM_SIZE, "symbol_size", sort_sym_size),
DIM(SORT_DSO_SIZE, "dso_size", sort_dso_size),
DIM(SORT_CGROUP, "cgroup", sort_cgroup),
@@ -2206,7 +2213,14 @@ bool perf_hpp__is_ ## key ## _entry(struct perf_hpp_fmt *fmt) \
return hse->se == &sort_ ## key ; \
}
+#ifdef HAVE_LIBTRACEEVENT
MK_SORT_ENTRY_CHK(trace)
+#else
+bool perf_hpp__is_trace_entry(struct perf_hpp_fmt *fmt __maybe_unused)
+{
+ return false;
+}
+#endif
MK_SORT_ENTRY_CHK(srcline)
MK_SORT_ENTRY_CHK(srcfile)
MK_SORT_ENTRY_CHK(thread)
@@ -2347,6 +2361,17 @@ static int __sort_dimension__add_hpp_output(struct sort_dimension *sd,
return 0;
}
+#ifndef HAVE_LIBTRACEEVENT
+bool perf_hpp__is_dynamic_entry(struct perf_hpp_fmt *fmt __maybe_unused)
+{
+ return false;
+}
+bool perf_hpp__defined_dynamic_entry(struct perf_hpp_fmt *fmt __maybe_unused,
+ struct hists *hists __maybe_unused)
+{
+ return false;
+}
+#else
struct hpp_dynamic_entry {
struct perf_hpp_fmt hpp;
struct evsel *evsel;
@@ -2543,9 +2568,10 @@ static int64_t __sort__hde_cmp(struct perf_hpp_fmt *fmt,
tep_read_number_field(field, a->raw_data, &dyn);
offset = dyn & 0xffff;
size = (dyn >> 16) & 0xffff;
+#ifdef HAVE_LIBTRACEEVENT_TEP_FIELD_IS_RELATIVE
if (field->flags & TEP_FIELD_IS_RELATIVE)
offset += field->offset + field->size;
-
+#endif
/* record max width for output */
if (size > hde->dynamic_len)
hde->dynamic_len = size;
@@ -2621,6 +2647,7 @@ __alloc_dynamic_entry(struct evsel *evsel, struct tep_format_field *field,
return hde;
}
+#endif /* HAVE_LIBTRACEEVENT */
struct perf_hpp_fmt *perf_hpp_fmt__dup(struct perf_hpp_fmt *fmt)
{
@@ -2633,6 +2660,7 @@ struct perf_hpp_fmt *perf_hpp_fmt__dup(struct perf_hpp_fmt *fmt)
new_hse = memdup(hse, sizeof(*hse));
if (new_hse)
new_fmt = &new_hse->hpp;
+#ifdef HAVE_LIBTRACEEVENT
} else if (perf_hpp__is_dynamic_entry(fmt)) {
struct hpp_dynamic_entry *hde, *new_hde;
@@ -2640,6 +2668,7 @@ struct perf_hpp_fmt *perf_hpp_fmt__dup(struct perf_hpp_fmt *fmt)
new_hde = memdup(hde, sizeof(*hde));
if (new_hde)
new_fmt = &new_hde->hpp;
+#endif
} else {
new_fmt = memdup(fmt, sizeof(*fmt));
}
@@ -2719,6 +2748,7 @@ static struct evsel *find_evsel(struct evlist *evlist, char *event_name)
return evsel;
}
+#ifdef HAVE_LIBTRACEEVENT
static int __dynamic_dimension__add(struct evsel *evsel,
struct tep_format_field *field,
bool raw_trace, int level)
@@ -2789,13 +2819,13 @@ static int add_all_matching_fields(struct evlist *evlist,
}
return ret;
}
+#endif /* HAVE_LIBTRACEEVENT */
static int add_dynamic_entry(struct evlist *evlist, const char *tok,
int level)
{
char *str, *event_name, *field_name, *opt_name;
struct evsel *evsel;
- struct tep_format_field *field;
bool raw_trace = symbol_conf.raw_trace;
int ret = 0;
@@ -2820,6 +2850,7 @@ static int add_dynamic_entry(struct evlist *evlist, const char *tok,
raw_trace = true;
}
+#ifdef HAVE_LIBTRACEEVENT
if (!strcmp(field_name, "trace_fields")) {
ret = add_all_dynamic_fields(evlist, raw_trace, level);
goto out;
@@ -2829,6 +2860,7 @@ static int add_dynamic_entry(struct evlist *evlist, const char *tok,
ret = add_all_matching_fields(evlist, field_name, raw_trace, level);
goto out;
}
+#endif
evsel = find_evsel(evlist, event_name);
if (evsel == NULL) {
@@ -2843,10 +2875,12 @@ static int add_dynamic_entry(struct evlist *evlist, const char *tok,
goto out;
}
+#ifdef HAVE_LIBTRACEEVENT
if (!strcmp(field_name, "*")) {
ret = add_evsel_fields(evsel, raw_trace, level);
} else {
- field = tep_find_any_field(evsel->tp_format, field_name);
+ struct tep_format_field *field = tep_find_any_field(evsel->tp_format, field_name);
+
if (field == NULL) {
pr_debug("Cannot find event field for %s.%s\n",
event_name, field_name);
@@ -2855,6 +2889,10 @@ static int add_dynamic_entry(struct evlist *evlist, const char *tok,
ret = __dynamic_dimension__add(evsel, field, raw_trace, level);
}
+#else
+ (void)level;
+ (void)raw_trace;
+#endif /* HAVE_LIBTRACEEVENT */
out:
free(str);
@@ -2955,11 +2993,11 @@ int sort_dimension__add(struct perf_hpp_list *list, const char *tok,
for (i = 0; i < ARRAY_SIZE(common_sort_dimensions); i++) {
struct sort_dimension *sd = &common_sort_dimensions[i];
- if (strncasecmp(tok, sd->name, strlen(tok)))
+ if (!sd->name || strncasecmp(tok, sd->name, strlen(tok)))
continue;
for (j = 0; j < ARRAY_SIZE(dynamic_headers); j++) {
- if (!strcmp(dynamic_headers[j], sd->name))
+ if (sd->name && !strcmp(dynamic_headers[j], sd->name))
sort_dimension_add_dynamic_header(sd);
}
@@ -3009,7 +3047,7 @@ int sort_dimension__add(struct perf_hpp_list *list, const char *tok,
for (i = 0; i < ARRAY_SIZE(bstack_sort_dimensions); i++) {
struct sort_dimension *sd = &bstack_sort_dimensions[i];
- if (strncasecmp(tok, sd->name, strlen(tok)))
+ if (!sd->name || strncasecmp(tok, sd->name, strlen(tok)))
continue;
if (sort__mode != SORT_MODE__BRANCH)
@@ -3025,7 +3063,7 @@ int sort_dimension__add(struct perf_hpp_list *list, const char *tok,
for (i = 0; i < ARRAY_SIZE(memory_sort_dimensions); i++) {
struct sort_dimension *sd = &memory_sort_dimensions[i];
- if (strncasecmp(tok, sd->name, strlen(tok)))
+ if (!sd->name || strncasecmp(tok, sd->name, strlen(tok)))
continue;
if (sort__mode != SORT_MODE__MEMORY)
@@ -3339,7 +3377,7 @@ int output_field_add(struct perf_hpp_list *list, char *tok)
for (i = 0; i < ARRAY_SIZE(common_sort_dimensions); i++) {
struct sort_dimension *sd = &common_sort_dimensions[i];
- if (strncasecmp(tok, sd->name, strlen(tok)))
+ if (!sd->name || strncasecmp(tok, sd->name, strlen(tok)))
continue;
return __sort_dimension__add_output(list, sd);
@@ -3357,7 +3395,7 @@ int output_field_add(struct perf_hpp_list *list, char *tok)
for (i = 0; i < ARRAY_SIZE(bstack_sort_dimensions); i++) {
struct sort_dimension *sd = &bstack_sort_dimensions[i];
- if (strncasecmp(tok, sd->name, strlen(tok)))
+ if (!sd->name || strncasecmp(tok, sd->name, strlen(tok)))
continue;
if (sort__mode != SORT_MODE__BRANCH)
@@ -3369,7 +3407,7 @@ int output_field_add(struct perf_hpp_list *list, char *tok)
for (i = 0; i < ARRAY_SIZE(memory_sort_dimensions); i++) {
struct sort_dimension *sd = &memory_sort_dimensions[i];
- if (strncasecmp(tok, sd->name, strlen(tok)))
+ if (!sd->name || strncasecmp(tok, sd->name, strlen(tok)))
continue;
if (sort__mode != SORT_MODE__MEMORY)
@@ -3508,6 +3546,9 @@ void reset_output_field(void)
static void add_key(struct strbuf *sb, const char *str, int *llen)
{
+ if (!str)
+ return;
+
if (*llen >= 75) {
strbuf_addstr(sb, "\n\t\t\t ");
*llen = INDENT;
diff --git a/tools/perf/util/stat-display.c b/tools/perf/util/stat-display.c
index ba66bb7..8bd8b01 100644
--- a/tools/perf/util/stat-display.c
+++ b/tools/perf/util/stat-display.c
@@ -25,41 +25,124 @@
#define CNTR_NOT_SUPPORTED "<not supported>"
#define CNTR_NOT_COUNTED "<not counted>"
-static void print_running(struct perf_stat_config *config,
- u64 run, u64 ena)
-{
+#define METRIC_LEN 38
+#define EVNAME_LEN 32
+#define COUNTS_LEN 18
+#define INTERVAL_LEN 16
+#define CGROUP_LEN 16
+#define COMM_LEN 16
+#define PID_LEN 7
+#define CPUS_LEN 4
+static int aggr_header_lens[] = {
+ [AGGR_CORE] = 18,
+ [AGGR_DIE] = 12,
+ [AGGR_SOCKET] = 6,
+ [AGGR_NODE] = 6,
+ [AGGR_NONE] = 6,
+ [AGGR_THREAD] = 16,
+ [AGGR_GLOBAL] = 0,
+};
+
+static const char *aggr_header_csv[] = {
+ [AGGR_CORE] = "core,cpus,",
+ [AGGR_DIE] = "die,cpus,",
+ [AGGR_SOCKET] = "socket,cpus,",
+ [AGGR_NONE] = "cpu,",
+ [AGGR_THREAD] = "comm-pid,",
+ [AGGR_NODE] = "node,",
+ [AGGR_GLOBAL] = ""
+};
+
+static const char *aggr_header_std[] = {
+ [AGGR_CORE] = "core",
+ [AGGR_DIE] = "die",
+ [AGGR_SOCKET] = "socket",
+ [AGGR_NONE] = "cpu",
+ [AGGR_THREAD] = "comm-pid",
+ [AGGR_NODE] = "node",
+ [AGGR_GLOBAL] = ""
+};
+
+static void print_running_std(struct perf_stat_config *config, u64 run, u64 ena)
+{
+ if (run != ena)
+ fprintf(config->output, " (%.2f%%)", 100.0 * run / ena);
+}
+
+static void print_running_csv(struct perf_stat_config *config, u64 run, u64 ena)
+{
double enabled_percent = 100;
if (run != ena)
enabled_percent = 100 * run / ena;
- if (config->json_output)
- fprintf(config->output,
- "\"event-runtime\" : %" PRIu64 ", \"pcnt-running\" : %.2f, ",
- run, enabled_percent);
- else if (config->csv_output)
- fprintf(config->output,
- "%s%" PRIu64 "%s%.2f", config->csv_sep,
- run, config->csv_sep, enabled_percent);
- else if (run != ena)
- fprintf(config->output, " (%.2f%%)", 100.0 * run / ena);
+ fprintf(config->output, "%s%" PRIu64 "%s%.2f",
+ config->csv_sep, run, config->csv_sep, enabled_percent);
}
-static void print_noise_pct(struct perf_stat_config *config,
- double total, double avg)
+static void print_running_json(struct perf_stat_config *config, u64 run, u64 ena)
{
- double pct = rel_stddev_stats(total, avg);
+ double enabled_percent = 100;
- if (config->json_output)
- fprintf(config->output, "\"variance\" : %.2f, ", pct);
- else if (config->csv_output)
- fprintf(config->output, "%s%.2f%%", config->csv_sep, pct);
- else if (pct)
+ if (run != ena)
+ enabled_percent = 100 * run / ena;
+ fprintf(config->output, "\"event-runtime\" : %" PRIu64 ", \"pcnt-running\" : %.2f, ",
+ run, enabled_percent);
+}
+
+static void print_running(struct perf_stat_config *config,
+ u64 run, u64 ena, bool before_metric)
+{
+ if (config->json_output) {
+ if (before_metric)
+ print_running_json(config, run, ena);
+ } else if (config->csv_output) {
+ if (before_metric)
+ print_running_csv(config, run, ena);
+ } else {
+ if (!before_metric)
+ print_running_std(config, run, ena);
+ }
+}
+
+static void print_noise_pct_std(struct perf_stat_config *config,
+ double pct)
+{
+ if (pct)
fprintf(config->output, " ( +-%6.2f%% )", pct);
}
+static void print_noise_pct_csv(struct perf_stat_config *config,
+ double pct)
+{
+ fprintf(config->output, "%s%.2f%%", config->csv_sep, pct);
+}
+
+static void print_noise_pct_json(struct perf_stat_config *config,
+ double pct)
+{
+ fprintf(config->output, "\"variance\" : %.2f, ", pct);
+}
+
+static void print_noise_pct(struct perf_stat_config *config,
+ double total, double avg, bool before_metric)
+{
+ double pct = rel_stddev_stats(total, avg);
+
+ if (config->json_output) {
+ if (before_metric)
+ print_noise_pct_json(config, pct);
+ } else if (config->csv_output) {
+ if (before_metric)
+ print_noise_pct_csv(config, pct);
+ } else {
+ if (!before_metric)
+ print_noise_pct_std(config, pct);
+ }
+}
+
static void print_noise(struct perf_stat_config *config,
- struct evsel *evsel, double avg)
+ struct evsel *evsel, double avg, bool before_metric)
{
struct perf_stat_evsel *ps;
@@ -67,138 +150,121 @@ static void print_noise(struct perf_stat_config *config,
return;
ps = evsel->stats;
- print_noise_pct(config, stddev_stats(&ps->res_stats), avg);
+ print_noise_pct(config, stddev_stats(&ps->res_stats), avg, before_metric);
}
-static void print_cgroup(struct perf_stat_config *config, struct evsel *evsel)
+static void print_cgroup_std(struct perf_stat_config *config, const char *cgrp_name)
{
- if (nr_cgroups) {
- const char *cgrp_name = evsel->cgrp ? evsel->cgrp->name : "";
+ fprintf(config->output, " %-*s", CGROUP_LEN, cgrp_name);
+}
+
+static void print_cgroup_csv(struct perf_stat_config *config, const char *cgrp_name)
+{
+ fprintf(config->output, "%s%s", config->csv_sep, cgrp_name);
+}
+
+static void print_cgroup_json(struct perf_stat_config *config, const char *cgrp_name)
+{
+ fprintf(config->output, "\"cgroup\" : \"%s\", ", cgrp_name);
+}
+
+static void print_cgroup(struct perf_stat_config *config, struct cgroup *cgrp)
+{
+ if (nr_cgroups || config->cgroup_list) {
+ const char *cgrp_name = cgrp ? cgrp->name : "";
if (config->json_output)
- fprintf(config->output, "\"cgroup\" : \"%s\", ", cgrp_name);
+ print_cgroup_json(config, cgrp_name);
+ else if (config->csv_output)
+ print_cgroup_csv(config, cgrp_name);
else
- fprintf(config->output, "%s%s", config->csv_sep, cgrp_name);
+ print_cgroup_std(config, cgrp_name);
}
}
-
-static void aggr_printout(struct perf_stat_config *config,
- struct evsel *evsel, struct aggr_cpu_id id, int nr)
+static void print_aggr_id_std(struct perf_stat_config *config,
+ struct evsel *evsel, struct aggr_cpu_id id, int nr)
{
-
-
- if (config->json_output && !config->interval)
- fprintf(config->output, "{");
+ FILE *output = config->output;
+ int idx = config->aggr_mode;
+ char buf[128];
switch (config->aggr_mode) {
case AGGR_CORE:
- if (config->json_output) {
- fprintf(config->output,
- "\"core\" : \"S%d-D%d-C%d\", \"aggregate-number\" : %d, ",
- id.socket,
- id.die,
- id.core,
- nr);
- } else {
- fprintf(config->output, "S%d-D%d-C%*d%s%*d%s",
- id.socket,
- id.die,
- config->csv_output ? 0 : -8,
- id.core,
- config->csv_sep,
- config->csv_output ? 0 : 4,
- nr,
- config->csv_sep);
- }
+ snprintf(buf, sizeof(buf), "S%d-D%d-C%d", id.socket, id.die, id.core);
break;
case AGGR_DIE:
- if (config->json_output) {
- fprintf(config->output,
- "\"die\" : \"S%d-D%d\", \"aggregate-number\" : %d, ",
- id.socket,
- id.die,
- nr);
- } else {
- fprintf(config->output, "S%d-D%*d%s%*d%s",
- id.socket,
- config->csv_output ? 0 : -8,
- id.die,
- config->csv_sep,
- config->csv_output ? 0 : 4,
- nr,
- config->csv_sep);
- }
+ snprintf(buf, sizeof(buf), "S%d-D%d", id.socket, id.die);
break;
case AGGR_SOCKET:
- if (config->json_output) {
- fprintf(config->output,
- "\"socket\" : \"S%d\", \"aggregate-number\" : %d, ",
- id.socket,
- nr);
- } else {
- fprintf(config->output, "S%*d%s%*d%s",
- config->csv_output ? 0 : -5,
- id.socket,
- config->csv_sep,
- config->csv_output ? 0 : 4,
- nr,
- config->csv_sep);
- }
+ snprintf(buf, sizeof(buf), "S%d", id.socket);
break;
case AGGR_NODE:
- if (config->json_output) {
- fprintf(config->output, "\"node\" : \"N%d\", \"aggregate-number\" : %d, ",
- id.node,
- nr);
- } else {
- fprintf(config->output, "N%*d%s%*d%s",
- config->csv_output ? 0 : -5,
- id.node,
- config->csv_sep,
- config->csv_output ? 0 : 4,
- nr,
- config->csv_sep);
- }
+ snprintf(buf, sizeof(buf), "N%d", id.node);
break;
case AGGR_NONE:
- if (config->json_output) {
- if (evsel->percore && !config->percore_show_thread) {
- fprintf(config->output, "\"core\" : \"S%d-D%d-C%d\"",
- id.socket,
- id.die,
- id.core);
- } else if (id.cpu.cpu > -1) {
- fprintf(config->output, "\"cpu\" : \"%d\", ",
- id.cpu.cpu);
- }
- } else {
- if (evsel->percore && !config->percore_show_thread) {
- fprintf(config->output, "S%d-D%d-C%*d%s",
- id.socket,
- id.die,
- config->csv_output ? 0 : -3,
- id.core, config->csv_sep);
- } else if (id.cpu.cpu > -1) {
- fprintf(config->output, "CPU%*d%s",
- config->csv_output ? 0 : -7,
- id.cpu.cpu, config->csv_sep);
- }
+ if (evsel->percore && !config->percore_show_thread) {
+ snprintf(buf, sizeof(buf), "S%d-D%d-C%d ",
+ id.socket, id.die, id.core);
+ fprintf(output, "%-*s ",
+ aggr_header_lens[AGGR_CORE], buf);
+ } else if (id.cpu.cpu > -1) {
+ fprintf(output, "CPU%-*d ",
+ aggr_header_lens[AGGR_NONE] - 3, id.cpu.cpu);
+ }
+ return;
+ case AGGR_THREAD:
+ fprintf(output, "%*s-%-*d ",
+ COMM_LEN, perf_thread_map__comm(evsel->core.threads, id.thread_idx),
+ PID_LEN, perf_thread_map__pid(evsel->core.threads, id.thread_idx));
+ return;
+ case AGGR_GLOBAL:
+ case AGGR_UNSET:
+ case AGGR_MAX:
+ default:
+ return;
+ }
+
+ fprintf(output, "%-*s %*d ", aggr_header_lens[idx], buf, 4, nr);
+}
+
+static void print_aggr_id_csv(struct perf_stat_config *config,
+ struct evsel *evsel, struct aggr_cpu_id id, int nr)
+{
+ FILE *output = config->output;
+ const char *sep = config->csv_sep;
+
+ switch (config->aggr_mode) {
+ case AGGR_CORE:
+ fprintf(output, "S%d-D%d-C%d%s%d%s",
+ id.socket, id.die, id.core, sep, nr, sep);
+ break;
+ case AGGR_DIE:
+ fprintf(output, "S%d-D%d%s%d%s",
+ id.socket, id.die, sep, nr, sep);
+ break;
+ case AGGR_SOCKET:
+ fprintf(output, "S%d%s%d%s",
+ id.socket, sep, nr, sep);
+ break;
+ case AGGR_NODE:
+ fprintf(output, "N%d%s%d%s",
+ id.node, sep, nr, sep);
+ break;
+ case AGGR_NONE:
+ if (evsel->percore && !config->percore_show_thread) {
+ fprintf(output, "S%d-D%d-C%d%s",
+ id.socket, id.die, id.core, sep);
+ } else if (id.cpu.cpu > -1) {
+ fprintf(output, "CPU%d%s",
+ id.cpu.cpu, sep);
}
break;
case AGGR_THREAD:
- if (config->json_output) {
- fprintf(config->output, "\"thread\" : \"%s-%d\", ",
- perf_thread_map__comm(evsel->core.threads, id.thread_idx),
- perf_thread_map__pid(evsel->core.threads, id.thread_idx));
- } else {
- fprintf(config->output, "%*s-%*d%s",
- config->csv_output ? 0 : 16,
- perf_thread_map__comm(evsel->core.threads, id.thread_idx),
- config->csv_output ? 0 : -8,
- perf_thread_map__pid(evsel->core.threads, id.thread_idx),
- config->csv_sep);
- }
+ fprintf(output, "%s-%d%s",
+ perf_thread_map__comm(evsel->core.threads, id.thread_idx),
+ perf_thread_map__pid(evsel->core.threads, id.thread_idx),
+ sep);
break;
case AGGR_GLOBAL:
case AGGR_UNSET:
@@ -208,18 +274,73 @@ static void aggr_printout(struct perf_stat_config *config,
}
}
+static void print_aggr_id_json(struct perf_stat_config *config,
+ struct evsel *evsel, struct aggr_cpu_id id, int nr)
+{
+ FILE *output = config->output;
+
+ switch (config->aggr_mode) {
+ case AGGR_CORE:
+ fprintf(output, "\"core\" : \"S%d-D%d-C%d\", \"aggregate-number\" : %d, ",
+ id.socket, id.die, id.core, nr);
+ break;
+ case AGGR_DIE:
+ fprintf(output, "\"die\" : \"S%d-D%d\", \"aggregate-number\" : %d, ",
+ id.socket, id.die, nr);
+ break;
+ case AGGR_SOCKET:
+ fprintf(output, "\"socket\" : \"S%d\", \"aggregate-number\" : %d, ",
+ id.socket, nr);
+ break;
+ case AGGR_NODE:
+ fprintf(output, "\"node\" : \"N%d\", \"aggregate-number\" : %d, ",
+ id.node, nr);
+ break;
+ case AGGR_NONE:
+ if (evsel->percore && !config->percore_show_thread) {
+ fprintf(output, "\"core\" : \"S%d-D%d-C%d\"",
+ id.socket, id.die, id.core);
+ } else if (id.cpu.cpu > -1) {
+ fprintf(output, "\"cpu\" : \"%d\", ",
+ id.cpu.cpu);
+ }
+ break;
+ case AGGR_THREAD:
+ fprintf(output, "\"thread\" : \"%s-%d\", ",
+ perf_thread_map__comm(evsel->core.threads, id.thread_idx),
+ perf_thread_map__pid(evsel->core.threads, id.thread_idx));
+ break;
+ case AGGR_GLOBAL:
+ case AGGR_UNSET:
+ case AGGR_MAX:
+ default:
+ break;
+ }
+}
+
+static void aggr_printout(struct perf_stat_config *config,
+ struct evsel *evsel, struct aggr_cpu_id id, int nr)
+{
+ if (config->json_output)
+ print_aggr_id_json(config, evsel, id, nr);
+ else if (config->csv_output)
+ print_aggr_id_csv(config, evsel, id, nr);
+ else
+ print_aggr_id_std(config, evsel, id, nr);
+}
+
struct outstate {
FILE *fh;
bool newline;
+ bool first;
const char *prefix;
int nfields;
int nr;
struct aggr_cpu_id id;
struct evsel *evsel;
+ struct cgroup *cgrp;
};
-#define METRIC_LEN 35
-
static void new_line_std(struct perf_stat_config *config __maybe_unused,
void *ctx)
{
@@ -232,7 +353,8 @@ static void do_new_line_std(struct perf_stat_config *config,
struct outstate *os)
{
fputc('\n', os->fh);
- fputs(os->prefix, os->fh);
+ if (os->prefix)
+ fputs(os->prefix, os->fh);
aggr_printout(config, os->evsel, os->id, os->nr);
if (config->aggr_mode == AGGR_NONE)
fprintf(os->fh, " ");
@@ -319,7 +441,7 @@ static void new_line_json(struct perf_stat_config *config, void *ctx)
{
struct outstate *os = ctx;
- fputc('\n', os->fh);
+ fputs("\n{", os->fh);
if (os->prefix)
fprintf(os->fh, "%s", os->prefix);
aggr_printout(config, os->evsel, os->id, os->nr);
@@ -368,6 +490,7 @@ static void print_metric_only(struct perf_stat_config *config,
color_snprintf(str, sizeof(str), color ?: "", fmt, val);
fprintf(out, "%*s ", mlen, str);
+ os->first = false;
}
static void print_metric_only_csv(struct perf_stat_config *config __maybe_unused,
@@ -389,6 +512,7 @@ static void print_metric_only_csv(struct perf_stat_config *config __maybe_unused
ends++;
*ends = 0;
fprintf(out, "%s%s", vals, config->csv_sep);
+ os->first = false;
}
static void print_metric_only_json(struct perf_stat_config *config __maybe_unused,
@@ -409,7 +533,10 @@ static void print_metric_only_json(struct perf_stat_config *config __maybe_unuse
while (isdigit(*ends) || *ends == '.')
ends++;
*ends = 0;
- fprintf(out, "{\"metric-value\" : \"%s\"}", vals);
+ if (!unit[0] || !vals[0])
+ return;
+ fprintf(out, "%s\"%s\" : \"%s\"", os->first ? "" : ", ", unit, vals);
+ os->first = false;
}
static void new_line_metric(struct perf_stat_config *config __maybe_unused,
@@ -430,84 +557,100 @@ static void print_metric_header(struct perf_stat_config *config,
os->evsel->priv != os->evsel->evlist->selected->priv)
return;
- if (!valid_only_metric(unit) && !config->json_output)
+ if (os->evsel->cgrp != os->cgrp)
+ return;
+
+ if (!valid_only_metric(unit))
return;
unit = fixunit(tbuf, os->evsel, unit);
if (config->json_output)
- fprintf(os->fh, "\"unit\" : \"%s\"", unit);
+ return;
else if (config->csv_output)
fprintf(os->fh, "%s%s", unit, config->csv_sep);
else
fprintf(os->fh, "%*s ", config->metric_only_len, unit);
}
-static int first_shadow_map_idx(struct perf_stat_config *config,
- struct evsel *evsel, const struct aggr_cpu_id *id)
-{
- struct perf_cpu_map *cpus = evsel__cpus(evsel);
- struct perf_cpu cpu;
- int idx;
-
- if (config->aggr_mode == AGGR_NONE)
- return perf_cpu_map__idx(cpus, id->cpu);
-
- if (config->aggr_mode == AGGR_THREAD)
- return id->thread_idx;
-
- if (!config->aggr_get_id)
- return 0;
-
- perf_cpu_map__for_each_cpu(cpu, idx, cpus) {
- struct aggr_cpu_id cpu_id = config->aggr_get_id(config, cpu);
-
- if (aggr_cpu_id__equal(&cpu_id, id))
- return idx;
- }
- return 0;
-}
-
-static void abs_printout(struct perf_stat_config *config,
- struct aggr_cpu_id id, int nr, struct evsel *evsel, double avg)
+static void print_counter_value_std(struct perf_stat_config *config,
+ struct evsel *evsel, double avg, bool ok)
{
FILE *output = config->output;
double sc = evsel->scale;
const char *fmt;
+ const char *bad_count = evsel->supported ? CNTR_NOT_COUNTED : CNTR_NOT_SUPPORTED;
- if (config->csv_output) {
- fmt = floor(sc) != sc ? "%.2f%s" : "%.0f%s";
- } else {
- if (config->big_num)
- fmt = floor(sc) != sc ? "%'18.2f%s" : "%'18.0f%s";
- else
- fmt = floor(sc) != sc ? "%18.2f%s" : "%18.0f%s";
- }
+ if (config->big_num)
+ fmt = floor(sc) != sc ? "%'*.2f " : "%'*.0f ";
+ else
+ fmt = floor(sc) != sc ? "%*.2f " : "%*.0f ";
- aggr_printout(config, evsel, id, nr);
+ if (ok)
+ fprintf(output, fmt, COUNTS_LEN, avg);
+ else
+ fprintf(output, "%*s ", COUNTS_LEN, bad_count);
- if (config->json_output)
+ if (evsel->unit)
+ fprintf(output, "%-*s ", config->unit_width, evsel->unit);
+
+ fprintf(output, "%-*s", EVNAME_LEN, evsel__name(evsel));
+}
+
+static void print_counter_value_csv(struct perf_stat_config *config,
+ struct evsel *evsel, double avg, bool ok)
+{
+ FILE *output = config->output;
+ double sc = evsel->scale;
+ const char *sep = config->csv_sep;
+ const char *fmt = floor(sc) != sc ? "%.2f%s" : "%.0f%s";
+ const char *bad_count = evsel->supported ? CNTR_NOT_COUNTED : CNTR_NOT_SUPPORTED;
+
+ if (ok)
+ fprintf(output, fmt, avg, sep);
+ else
+ fprintf(output, "%s%s", bad_count, sep);
+
+ if (evsel->unit)
+ fprintf(output, "%s%s", evsel->unit, sep);
+
+ fprintf(output, "%s", evsel__name(evsel));
+}
+
+static void print_counter_value_json(struct perf_stat_config *config,
+ struct evsel *evsel, double avg, bool ok)
+{
+ FILE *output = config->output;
+ const char *bad_count = evsel->supported ? CNTR_NOT_COUNTED : CNTR_NOT_SUPPORTED;
+
+ if (ok)
fprintf(output, "\"counter-value\" : \"%f\", ", avg);
else
- fprintf(output, fmt, avg, config->csv_sep);
+ fprintf(output, "\"counter-value\" : \"%s\", ", bad_count);
- if (config->json_output) {
- if (evsel->unit) {
- fprintf(output, "\"unit\" : \"%s\", ",
- evsel->unit);
- }
- } else {
- if (evsel->unit)
- fprintf(output, "%-*s%s",
- config->csv_output ? 0 : config->unit_width,
- evsel->unit, config->csv_sep);
- }
+ if (evsel->unit)
+ fprintf(output, "\"unit\" : \"%s\", ", evsel->unit);
+ fprintf(output, "\"event\" : \"%s\", ", evsel__name(evsel));
+}
+
+static void print_counter_value(struct perf_stat_config *config,
+ struct evsel *evsel, double avg, bool ok)
+{
if (config->json_output)
- fprintf(output, "\"event\" : \"%s\", ", evsel__name(evsel));
+ print_counter_value_json(config, evsel, avg, ok);
+ else if (config->csv_output)
+ print_counter_value_csv(config, evsel, avg, ok);
else
- fprintf(output, "%-*s", config->csv_output ? 0 : 32, evsel__name(evsel));
+ print_counter_value_std(config, evsel, avg, ok);
+}
- print_cgroup(config, evsel);
+static void abs_printout(struct perf_stat_config *config,
+ struct aggr_cpu_id id, int nr,
+ struct evsel *evsel, double avg, bool ok)
+{
+ aggr_printout(config, evsel, id, nr);
+ print_counter_value(config, evsel, avg, ok);
+ print_cgroup(config, evsel->cgrp);
}
static bool is_mixed_hw_group(struct evsel *counter)
@@ -534,37 +677,19 @@ static bool is_mixed_hw_group(struct evsel *counter)
return false;
}
-static void printout(struct perf_stat_config *config, struct aggr_cpu_id id, int nr,
- struct evsel *counter, double uval,
- char *prefix, u64 run, u64 ena, double noise,
- struct runtime_stat *st)
+static void printout(struct perf_stat_config *config, struct outstate *os,
+ double uval, u64 run, u64 ena, double noise, int map_idx)
{
struct perf_stat_output_ctx out;
- struct outstate os = {
- .fh = config->output,
- .prefix = prefix ? prefix : "",
- .id = id,
- .nr = nr,
- .evsel = counter,
- };
print_metric_t pm;
new_line_t nl;
+ bool ok = true;
+ struct evsel *counter = os->evsel;
if (config->csv_output) {
- static const int aggr_fields[AGGR_MAX] = {
- [AGGR_NONE] = 1,
- [AGGR_GLOBAL] = 0,
- [AGGR_SOCKET] = 2,
- [AGGR_DIE] = 2,
- [AGGR_CORE] = 2,
- [AGGR_THREAD] = 1,
- [AGGR_UNSET] = 0,
- [AGGR_NODE] = 1,
- };
-
pm = config->metric_only ? print_metric_only_csv : print_metric_csv;
nl = config->metric_only ? new_line_metric : new_line_csv;
- os.nfields = 3 + aggr_fields[config->aggr_mode] + (counter->cgrp ? 1 : 0);
+ os->nfields = 4 + (counter->cgrp ? 1 : 0);
} else if (config->json_output) {
pm = config->metric_only ? print_metric_only_json : print_metric_json;
nl = config->metric_only ? new_line_metric : new_line_json;
@@ -573,27 +698,13 @@ static void printout(struct perf_stat_config *config, struct aggr_cpu_id id, int
nl = config->metric_only ? new_line_metric : new_line_std;
}
- if (!config->no_csv_summary && config->csv_output &&
- config->summary && !config->interval) {
- fprintf(config->output, "%16s%s", "summary", config->csv_sep);
- }
-
if (run == 0 || ena == 0 || counter->counts->scaled == -1) {
if (config->metric_only) {
- pm(config, &os, NULL, "", "", 0);
+ pm(config, os, NULL, "", "", 0);
return;
}
- aggr_printout(config, counter, id, nr);
- if (config->json_output) {
- fprintf(config->output, "\"counter-value\" : \"%s\", ",
- counter->supported ? CNTR_NOT_COUNTED : CNTR_NOT_SUPPORTED);
- } else {
- fprintf(config->output, "%*s%s",
- config->csv_output ? 0 : 18,
- counter->supported ? CNTR_NOT_COUNTED : CNTR_NOT_SUPPORTED,
- config->csv_sep);
- }
+ ok = false;
if (counter->supported) {
if (!evlist__has_hybrid(counter->evlist)) {
@@ -602,86 +713,30 @@ static void printout(struct perf_stat_config *config, struct aggr_cpu_id id, int
config->print_mixed_hw_group_error = 1;
}
}
-
- if (config->json_output) {
- fprintf(config->output, "\"unit\" : \"%s\", ", counter->unit);
- } else {
- fprintf(config->output, "%-*s%s",
- config->csv_output ? 0 : config->unit_width,
- counter->unit, config->csv_sep);
- }
-
- if (config->json_output) {
- fprintf(config->output, "\"event\" : \"%s\", ",
- evsel__name(counter));
- } else {
- fprintf(config->output, "%*s",
- config->csv_output ? 0 : -25, evsel__name(counter));
- }
-
- print_cgroup(config, counter);
-
- if (!config->csv_output && !config->json_output)
- pm(config, &os, NULL, NULL, "", 0);
- print_noise(config, counter, noise);
- print_running(config, run, ena);
- if (config->csv_output)
- pm(config, &os, NULL, NULL, "", 0);
- else if (config->json_output)
- pm(config, &os, NULL, NULL, "", 0);
- return;
}
- if (!config->metric_only)
- abs_printout(config, id, nr, counter, uval);
-
out.print_metric = pm;
out.new_line = nl;
- out.ctx = &os;
+ out.ctx = os;
out.force_header = false;
- if (config->csv_output && !config->metric_only) {
- print_noise(config, counter, noise);
- print_running(config, run, ena);
- } else if (config->json_output && !config->metric_only) {
- print_noise(config, counter, noise);
- print_running(config, run, ena);
+ if (!config->metric_only) {
+ abs_printout(config, os->id, os->nr, counter, uval, ok);
+
+ print_noise(config, counter, noise, /*before_metric=*/true);
+ print_running(config, run, ena, /*before_metric=*/true);
}
- perf_stat__print_shadow_stats(config, counter, uval,
- first_shadow_map_idx(config, counter, &id),
- &out, &config->metric_events, st);
- if (!config->csv_output && !config->metric_only && !config->json_output) {
- print_noise(config, counter, noise);
- print_running(config, run, ena);
+ if (ok) {
+ perf_stat__print_shadow_stats(config, counter, uval, map_idx,
+ &out, &config->metric_events, &rt_stat);
+ } else {
+ pm(config, os, /*color=*/NULL, /*format=*/NULL, /*unit=*/"", /*val=*/0);
}
-}
-static void aggr_update_shadow(struct perf_stat_config *config,
- struct evlist *evlist)
-{
- int idx, s;
- struct perf_cpu cpu;
- struct aggr_cpu_id s2, id;
- u64 val;
- struct evsel *counter;
- struct perf_cpu_map *cpus;
-
- for (s = 0; s < config->aggr_map->nr; s++) {
- id = config->aggr_map->map[s];
- evlist__for_each_entry(evlist, counter) {
- cpus = evsel__cpus(counter);
- val = 0;
- perf_cpu_map__for_each_cpu(cpu, idx, cpus) {
- s2 = config->aggr_get_id(config, cpu);
- if (!aggr_cpu_id__equal(&s2, &id))
- continue;
- val += perf_counts(counter->counts, idx, 0)->val;
- }
- perf_stat__update_shadow_stats(counter, val,
- first_shadow_map_idx(config, counter, &id),
- &rt_stat);
- }
+ if (!config->metric_only) {
+ print_noise(config, counter, noise, /*before_metric=*/false);
+ print_running(config, run, ena, /*before_metric=*/false);
}
}
@@ -704,7 +759,7 @@ static void uniquify_event_name(struct evsel *counter)
counter->name = new_name;
}
} else {
- if (perf_pmu__has_hybrid()) {
+ if (evsel__is_hybrid(counter)) {
ret = asprintf(&new_name, "%s/%s/",
counter->pmu_name, counter->name);
} else {
@@ -721,366 +776,180 @@ static void uniquify_event_name(struct evsel *counter)
counter->uniquified_name = true;
}
-static void collect_all_aliases(struct perf_stat_config *config, struct evsel *counter,
- void (*cb)(struct perf_stat_config *config, struct evsel *counter, void *data,
- bool first),
- void *data)
+static bool hybrid_uniquify(struct evsel *evsel, struct perf_stat_config *config)
{
- struct evlist *evlist = counter->evlist;
- struct evsel *alias;
-
- alias = list_prepare_entry(counter, &(evlist->core.entries), core.node);
- list_for_each_entry_continue (alias, &evlist->core.entries, core.node) {
- /* Merge events with the same name, etc. but on different PMUs. */
- if (!strcmp(evsel__name(alias), evsel__name(counter)) &&
- alias->scale == counter->scale &&
- alias->cgrp == counter->cgrp &&
- !strcmp(alias->unit, counter->unit) &&
- evsel__is_clock(alias) == evsel__is_clock(counter) &&
- strcmp(alias->pmu_name, counter->pmu_name)) {
- alias->merged_stat = true;
- cb(config, alias, data, false);
- }
- }
+ return evsel__is_hybrid(evsel) && !config->hybrid_merge;
}
-static bool is_uncore(struct evsel *evsel)
+static void uniquify_counter(struct perf_stat_config *config, struct evsel *counter)
{
- struct perf_pmu *pmu = evsel__find_pmu(evsel);
-
- return pmu && pmu->is_uncore;
-}
-
-static bool hybrid_uniquify(struct evsel *evsel)
-{
- return perf_pmu__has_hybrid() && !is_uncore(evsel);
-}
-
-static bool hybrid_merge(struct evsel *counter, struct perf_stat_config *config,
- bool check)
-{
- if (hybrid_uniquify(counter)) {
- if (check)
- return config && config->hybrid_merge;
- else
- return config && !config->hybrid_merge;
- }
-
- return false;
-}
-
-static bool collect_data(struct perf_stat_config *config, struct evsel *counter,
- void (*cb)(struct perf_stat_config *config, struct evsel *counter, void *data,
- bool first),
- void *data)
-{
- if (counter->merged_stat)
- return false;
- cb(config, counter, data, true);
- if (config->no_merge || hybrid_merge(counter, config, false))
+ if (config->no_merge || hybrid_uniquify(counter, config))
uniquify_event_name(counter);
- else if (counter->auto_merge_stats || hybrid_merge(counter, config, true))
- collect_all_aliases(config, counter, cb, data);
- return true;
-}
-
-struct aggr_data {
- u64 ena, run, val;
- struct aggr_cpu_id id;
- int nr;
- int cpu_map_idx;
-};
-
-static void aggr_cb(struct perf_stat_config *config,
- struct evsel *counter, void *data, bool first)
-{
- struct aggr_data *ad = data;
- int idx;
- struct perf_cpu cpu;
- struct perf_cpu_map *cpus;
- struct aggr_cpu_id s2;
-
- cpus = evsel__cpus(counter);
- perf_cpu_map__for_each_cpu(cpu, idx, cpus) {
- struct perf_counts_values *counts;
-
- s2 = config->aggr_get_id(config, cpu);
- if (!aggr_cpu_id__equal(&s2, &ad->id))
- continue;
- if (first)
- ad->nr++;
- counts = perf_counts(counter->counts, idx, 0);
- /*
- * When any result is bad, make them all to give
- * consistent output in interval mode.
- */
- if (counts->ena == 0 || counts->run == 0 ||
- counter->counts->scaled == -1) {
- ad->ena = 0;
- ad->run = 0;
- break;
- }
- ad->val += counts->val;
- ad->ena += counts->ena;
- ad->run += counts->run;
- }
}
static void print_counter_aggrdata(struct perf_stat_config *config,
struct evsel *counter, int s,
- char *prefix, bool metric_only,
- bool *first, struct perf_cpu cpu)
+ struct outstate *os)
{
- struct aggr_data ad;
FILE *output = config->output;
u64 ena, run, val;
- int nr;
- struct aggr_cpu_id id;
double uval;
+ struct perf_stat_evsel *ps = counter->stats;
+ struct perf_stat_aggr *aggr = &ps->aggr[s];
+ struct aggr_cpu_id id = config->aggr_map->map[s];
+ double avg = aggr->counts.val;
+ bool metric_only = config->metric_only;
- ad.id = id = config->aggr_map->map[s];
- ad.val = ad.ena = ad.run = 0;
- ad.nr = 0;
- if (!collect_data(config, counter, aggr_cb, &ad))
+ os->id = id;
+ os->nr = aggr->nr;
+ os->evsel = counter;
+
+ /* Skip already merged uncore/hybrid events */
+ if (counter->merged_stat)
return;
- if (perf_pmu__has_hybrid() && ad.ena == 0)
+ uniquify_counter(config, counter);
+
+ val = aggr->counts.val;
+ ena = aggr->counts.ena;
+ run = aggr->counts.run;
+
+ /*
+ * Skip value 0 when enabling --per-thread globally, otherwise it will
+ * have too many 0 output.
+ */
+ if (val == 0 && config->aggr_mode == AGGR_THREAD && config->system_wide)
return;
- nr = ad.nr;
- ena = ad.ena;
- run = ad.run;
- val = ad.val;
- if (*first && metric_only) {
- *first = false;
- aggr_printout(config, counter, id, nr);
+ if (!metric_only) {
+ if (config->json_output)
+ fputc('{', output);
+ if (os->prefix)
+ fprintf(output, "%s", os->prefix);
+ else if (config->summary && config->csv_output &&
+ !config->no_csv_summary && !config->interval)
+ fprintf(output, "%s%s", "summary", config->csv_sep);
}
- if (prefix && !metric_only)
- fprintf(output, "%s", prefix);
uval = val * counter->scale;
- if (cpu.cpu != -1)
- id = aggr_cpu_id__cpu(cpu, /*data=*/NULL);
- printout(config, id, nr, counter, uval,
- prefix, run, ena, 1.0, &rt_stat);
+ printout(config, os, uval, run, ena, avg, s);
+
if (!metric_only)
fputc('\n', output);
}
+static void print_metric_begin(struct perf_stat_config *config,
+ struct evlist *evlist,
+ struct outstate *os, int aggr_idx)
+{
+ struct perf_stat_aggr *aggr;
+ struct aggr_cpu_id id;
+ struct evsel *evsel;
+
+ os->first = true;
+ if (!config->metric_only)
+ return;
+
+ if (config->json_output)
+ fputc('{', config->output);
+ if (os->prefix)
+ fprintf(config->output, "%s", os->prefix);
+
+ evsel = evlist__first(evlist);
+ id = config->aggr_map->map[aggr_idx];
+ aggr = &evsel->stats->aggr[aggr_idx];
+ aggr_printout(config, evsel, id, aggr->nr);
+
+ print_cgroup(config, os->cgrp ? : evsel->cgrp);
+}
+
+static void print_metric_end(struct perf_stat_config *config, struct outstate *os)
+{
+ FILE *output = config->output;
+
+ if (!config->metric_only)
+ return;
+
+ if (config->json_output) {
+ if (os->first)
+ fputs("\"metric-value\" : \"none\"", output);
+ fputc('}', output);
+ }
+ fputc('\n', output);
+}
+
static void print_aggr(struct perf_stat_config *config,
struct evlist *evlist,
- char *prefix)
+ struct outstate *os)
{
- bool metric_only = config->metric_only;
- FILE *output = config->output;
struct evsel *counter;
int s;
- bool first;
if (!config->aggr_map || !config->aggr_get_id)
return;
- aggr_update_shadow(config, evlist);
-
/*
* With metric_only everything is on a single line.
* Without each counter has its own line.
*/
for (s = 0; s < config->aggr_map->nr; s++) {
- if (prefix && metric_only)
- fprintf(output, "%s", prefix);
+ print_metric_begin(config, evlist, os, s);
- first = true;
evlist__for_each_entry(evlist, counter) {
- print_counter_aggrdata(config, counter, s,
- prefix, metric_only,
- &first, (struct perf_cpu){ .cpu = -1 });
+ print_counter_aggrdata(config, counter, s, os);
}
- if (metric_only)
- fputc('\n', output);
+ print_metric_end(config, os);
}
}
-static int cmp_val(const void *a, const void *b)
+static void print_aggr_cgroup(struct perf_stat_config *config,
+ struct evlist *evlist,
+ struct outstate *os)
{
- return ((struct perf_aggr_thread_value *)b)->val -
- ((struct perf_aggr_thread_value *)a)->val;
-}
+ struct evsel *counter, *evsel;
+ int s;
-static struct perf_aggr_thread_value *sort_aggr_thread(
- struct evsel *counter,
- int *ret,
- struct target *_target)
-{
- int nthreads = perf_thread_map__nr(counter->core.threads);
- int i = 0;
- double uval;
- struct perf_aggr_thread_value *buf;
+ if (!config->aggr_map || !config->aggr_get_id)
+ return;
- buf = calloc(nthreads, sizeof(struct perf_aggr_thread_value));
- if (!buf)
- return NULL;
-
- for (int thread = 0; thread < nthreads; thread++) {
- int idx;
- u64 ena = 0, run = 0, val = 0;
-
- perf_cpu_map__for_each_idx(idx, evsel__cpus(counter)) {
- struct perf_counts_values *counts =
- perf_counts(counter->counts, idx, thread);
-
- val += counts->val;
- ena += counts->ena;
- run += counts->run;
- }
-
- uval = val * counter->scale;
-
- /*
- * Skip value 0 when enabling --per-thread globally,
- * otherwise too many 0 output.
- */
- if (uval == 0.0 && target__has_per_thread(_target))
+ evlist__for_each_entry(evlist, evsel) {
+ if (os->cgrp == evsel->cgrp)
continue;
- buf[i].counter = counter;
- buf[i].id = aggr_cpu_id__empty();
- buf[i].id.thread_idx = thread;
- buf[i].uval = uval;
- buf[i].val = val;
- buf[i].run = run;
- buf[i].ena = ena;
- i++;
+ os->cgrp = evsel->cgrp;
+
+ for (s = 0; s < config->aggr_map->nr; s++) {
+ print_metric_begin(config, evlist, os, s);
+
+ evlist__for_each_entry(evlist, counter) {
+ if (counter->cgrp != os->cgrp)
+ continue;
+
+ print_counter_aggrdata(config, counter, s, os);
+ }
+ print_metric_end(config, os);
+ }
}
-
- qsort(buf, i, sizeof(struct perf_aggr_thread_value), cmp_val);
-
- if (ret)
- *ret = i;
-
- return buf;
}
-static void print_aggr_thread(struct perf_stat_config *config,
- struct target *_target,
- struct evsel *counter, char *prefix)
-{
- FILE *output = config->output;
- int thread, sorted_threads;
- struct aggr_cpu_id id;
- struct perf_aggr_thread_value *buf;
-
- buf = sort_aggr_thread(counter, &sorted_threads, _target);
- if (!buf) {
- perror("cannot sort aggr thread");
- return;
- }
-
- for (thread = 0; thread < sorted_threads; thread++) {
- if (prefix)
- fprintf(output, "%s", prefix);
-
- id = buf[thread].id;
- printout(config, id, 0, buf[thread].counter, buf[thread].uval,
- prefix, buf[thread].run, buf[thread].ena, 1.0,
- &rt_stat);
- fputc('\n', output);
- }
-
- free(buf);
-}
-
-struct caggr_data {
- double avg, avg_enabled, avg_running;
-};
-
-static void counter_aggr_cb(struct perf_stat_config *config __maybe_unused,
- struct evsel *counter, void *data,
- bool first __maybe_unused)
-{
- struct caggr_data *cd = data;
- struct perf_counts_values *aggr = &counter->counts->aggr;
-
- cd->avg += aggr->val;
- cd->avg_enabled += aggr->ena;
- cd->avg_running += aggr->run;
-}
-
-/*
- * Print out the results of a single counter:
- * aggregated counts in system-wide mode
- */
-static void print_counter_aggr(struct perf_stat_config *config,
- struct evsel *counter, char *prefix)
-{
- bool metric_only = config->metric_only;
- FILE *output = config->output;
- double uval;
- struct caggr_data cd = { .avg = 0.0 };
-
- if (!collect_data(config, counter, counter_aggr_cb, &cd))
- return;
-
- if (prefix && !metric_only)
- fprintf(output, "%s", prefix);
-
- uval = cd.avg * counter->scale;
- printout(config, aggr_cpu_id__empty(), 0, counter, uval, prefix, cd.avg_running,
- cd.avg_enabled, cd.avg, &rt_stat);
- if (!metric_only)
- fprintf(output, "\n");
-}
-
-static void counter_cb(struct perf_stat_config *config __maybe_unused,
- struct evsel *counter, void *data,
- bool first __maybe_unused)
-{
- struct aggr_data *ad = data;
-
- ad->val += perf_counts(counter->counts, ad->cpu_map_idx, 0)->val;
- ad->ena += perf_counts(counter->counts, ad->cpu_map_idx, 0)->ena;
- ad->run += perf_counts(counter->counts, ad->cpu_map_idx, 0)->run;
-}
-
-/*
- * Print out the results of a single counter:
- * does not use aggregated count in system-wide
- */
static void print_counter(struct perf_stat_config *config,
- struct evsel *counter, char *prefix)
+ struct evsel *counter, struct outstate *os)
{
- FILE *output = config->output;
- u64 ena, run, val;
- double uval;
- int idx;
- struct perf_cpu cpu;
- struct aggr_cpu_id id;
+ int s;
- perf_cpu_map__for_each_cpu(cpu, idx, evsel__cpus(counter)) {
- struct aggr_data ad = { .cpu_map_idx = idx };
+ /* AGGR_THREAD doesn't have config->aggr_get_id */
+ if (!config->aggr_map)
+ return;
- if (!collect_data(config, counter, counter_cb, &ad))
- return;
- val = ad.val;
- ena = ad.ena;
- run = ad.run;
-
- if (prefix)
- fprintf(output, "%s", prefix);
-
- uval = val * counter->scale;
- id = aggr_cpu_id__cpu(cpu, /*data=*/NULL);
- printout(config, id, 0, counter, uval, prefix,
- run, ena, 1.0, &rt_stat);
-
- fputc('\n', output);
+ for (s = 0; s < config->aggr_map->nr; s++) {
+ print_counter_aggrdata(config, counter, s, os);
}
}
static void print_no_aggr_metric(struct perf_stat_config *config,
struct evlist *evlist,
- char *prefix)
+ struct outstate *os)
{
int all_idx;
struct perf_cpu cpu;
@@ -1092,214 +961,241 @@ static void print_no_aggr_metric(struct perf_stat_config *config,
evlist__for_each_entry(evlist, counter) {
u64 ena, run, val;
double uval;
- struct aggr_cpu_id id;
+ struct perf_stat_evsel *ps = counter->stats;
int counter_idx = perf_cpu_map__idx(evsel__cpus(counter), cpu);
if (counter_idx < 0)
continue;
- id = aggr_cpu_id__cpu(cpu, /*data=*/NULL);
+ os->evsel = counter;
+ os->id = aggr_cpu_id__cpu(cpu, /*data=*/NULL);
if (first) {
- if (prefix)
- fputs(prefix, config->output);
- aggr_printout(config, counter, id, 0);
+ print_metric_begin(config, evlist, os, counter_idx);
first = false;
}
- val = perf_counts(counter->counts, counter_idx, 0)->val;
- ena = perf_counts(counter->counts, counter_idx, 0)->ena;
- run = perf_counts(counter->counts, counter_idx, 0)->run;
+ val = ps->aggr[counter_idx].counts.val;
+ ena = ps->aggr[counter_idx].counts.ena;
+ run = ps->aggr[counter_idx].counts.run;
uval = val * counter->scale;
- printout(config, id, 0, counter, uval, prefix,
- run, ena, 1.0, &rt_stat);
+ printout(config, os, uval, run, ena, 1.0, counter_idx);
}
if (!first)
- fputc('\n', config->output);
+ print_metric_end(config, os);
}
}
-static int aggr_header_lens[] = {
- [AGGR_CORE] = 24,
- [AGGR_DIE] = 18,
- [AGGR_SOCKET] = 12,
- [AGGR_NONE] = 6,
- [AGGR_THREAD] = 24,
- [AGGR_NODE] = 6,
- [AGGR_GLOBAL] = 0,
-};
+static void print_metric_headers_std(struct perf_stat_config *config,
+ bool no_indent)
+{
+ fputc(' ', config->output);
-static const char *aggr_header_csv[] = {
- [AGGR_CORE] = "core,cpus,",
- [AGGR_DIE] = "die,cpus",
- [AGGR_SOCKET] = "socket,cpus",
- [AGGR_NONE] = "cpu,",
- [AGGR_THREAD] = "comm-pid,",
- [AGGR_NODE] = "node,",
- [AGGR_GLOBAL] = ""
-};
+ if (!no_indent) {
+ int len = aggr_header_lens[config->aggr_mode];
+
+ if (nr_cgroups || config->cgroup_list)
+ len += CGROUP_LEN + 1;
+
+ fprintf(config->output, "%*s", len, "");
+ }
+}
+
+static void print_metric_headers_csv(struct perf_stat_config *config,
+ bool no_indent __maybe_unused)
+{
+ if (config->interval)
+ fputs("time,", config->output);
+ if (!config->iostat_run)
+ fputs(aggr_header_csv[config->aggr_mode], config->output);
+}
+
+static void print_metric_headers_json(struct perf_stat_config *config __maybe_unused,
+ bool no_indent __maybe_unused)
+{
+}
static void print_metric_headers(struct perf_stat_config *config,
- struct evlist *evlist,
- const char *prefix, bool no_indent)
+ struct evlist *evlist, bool no_indent)
{
- struct perf_stat_output_ctx out;
struct evsel *counter;
struct outstate os = {
.fh = config->output
};
- bool first = true;
+ struct perf_stat_output_ctx out = {
+ .ctx = &os,
+ .print_metric = print_metric_header,
+ .new_line = new_line_metric,
+ .force_header = true,
+ };
- if (config->json_output && !config->interval)
- fprintf(config->output, "{");
+ if (config->json_output)
+ print_metric_headers_json(config, no_indent);
+ else if (config->csv_output)
+ print_metric_headers_csv(config, no_indent);
+ else
+ print_metric_headers_std(config, no_indent);
- if (prefix && !config->json_output)
- fprintf(config->output, "%s", prefix);
-
- if (!config->csv_output && !no_indent)
- fprintf(config->output, "%*s",
- aggr_header_lens[config->aggr_mode], "");
- if (config->csv_output) {
- if (config->interval)
- fputs("time,", config->output);
- if (!config->iostat_run)
- fputs(aggr_header_csv[config->aggr_mode], config->output);
- }
if (config->iostat_run)
iostat_print_header_prefix(config);
+ if (config->cgroup_list)
+ os.cgrp = evlist__first(evlist)->cgrp;
+
/* Print metrics headers only */
evlist__for_each_entry(evlist, counter) {
os.evsel = counter;
- out.ctx = &os;
- out.print_metric = print_metric_header;
- if (!first && config->json_output)
- fprintf(config->output, ", ");
- first = false;
- out.new_line = new_line_metric;
- out.force_header = true;
+
perf_stat__print_shadow_stats(config, counter, 0,
0,
&out,
&config->metric_events,
&rt_stat);
}
- if (config->json_output)
- fprintf(config->output, "}");
- fputc('\n', config->output);
+
+ if (!config->json_output)
+ fputc('\n', config->output);
}
-static void print_interval(struct perf_stat_config *config,
- struct evlist *evlist,
- char *prefix, struct timespec *ts)
+static void prepare_interval(struct perf_stat_config *config,
+ char *prefix, size_t len, struct timespec *ts)
{
- bool metric_only = config->metric_only;
- unsigned int unit_width = config->unit_width;
- FILE *output = config->output;
- static int num_print_interval;
+ if (config->iostat_run)
+ return;
- if (config->interval_clear)
- puts(CONSOLE_CLEAR);
-
- if (!config->iostat_run && !config->json_output)
- sprintf(prefix, "%6lu.%09lu%s", (unsigned long) ts->tv_sec,
- ts->tv_nsec, config->csv_sep);
- if (!config->iostat_run && config->json_output && !config->metric_only)
- sprintf(prefix, "{\"interval\" : %lu.%09lu, ", (unsigned long)
- ts->tv_sec, ts->tv_nsec);
- if (!config->iostat_run && config->json_output && config->metric_only)
- sprintf(prefix, "{\"interval\" : %lu.%09lu}", (unsigned long)
- ts->tv_sec, ts->tv_nsec);
-
- if ((num_print_interval == 0 && !config->csv_output && !config->json_output)
- || config->interval_clear) {
- switch (config->aggr_mode) {
- case AGGR_NODE:
- fprintf(output, "# time node cpus");
- if (!metric_only)
- fprintf(output, " counts %*s events\n", unit_width, "unit");
- break;
- case AGGR_SOCKET:
- fprintf(output, "# time socket cpus");
- if (!metric_only)
- fprintf(output, " counts %*s events\n", unit_width, "unit");
- break;
- case AGGR_DIE:
- fprintf(output, "# time die cpus");
- if (!metric_only)
- fprintf(output, " counts %*s events\n", unit_width, "unit");
- break;
- case AGGR_CORE:
- fprintf(output, "# time core cpus");
- if (!metric_only)
- fprintf(output, " counts %*s events\n", unit_width, "unit");
- break;
- case AGGR_NONE:
- fprintf(output, "# time CPU ");
- if (!metric_only)
- fprintf(output, " counts %*s events\n", unit_width, "unit");
- break;
- case AGGR_THREAD:
- fprintf(output, "# time comm-pid");
- if (!metric_only)
- fprintf(output, " counts %*s events\n", unit_width, "unit");
- break;
- case AGGR_GLOBAL:
- default:
- if (!config->iostat_run) {
- fprintf(output, "# time");
- if (!metric_only)
- fprintf(output, " counts %*s events\n", unit_width, "unit");
- }
- case AGGR_UNSET:
- case AGGR_MAX:
- break;
- }
- }
-
- if ((num_print_interval == 0 || config->interval_clear)
- && metric_only && !config->json_output)
- print_metric_headers(config, evlist, " ", true);
- if ((num_print_interval == 0 || config->interval_clear)
- && metric_only && config->json_output) {
- fprintf(output, "{");
- print_metric_headers(config, evlist, " ", true);
- }
- if (++num_print_interval == 25)
- num_print_interval = 0;
+ if (config->json_output)
+ scnprintf(prefix, len, "\"interval\" : %lu.%09lu, ",
+ (unsigned long) ts->tv_sec, ts->tv_nsec);
+ else if (config->csv_output)
+ scnprintf(prefix, len, "%lu.%09lu%s",
+ (unsigned long) ts->tv_sec, ts->tv_nsec, config->csv_sep);
+ else
+ scnprintf(prefix, len, "%6lu.%09lu ",
+ (unsigned long) ts->tv_sec, ts->tv_nsec);
}
-static void print_header(struct perf_stat_config *config,
- struct target *_target,
- int argc, const char **argv)
+static void print_header_interval_std(struct perf_stat_config *config,
+ struct target *_target __maybe_unused,
+ struct evlist *evlist,
+ int argc __maybe_unused,
+ const char **argv __maybe_unused)
+{
+ FILE *output = config->output;
+
+ switch (config->aggr_mode) {
+ case AGGR_NODE:
+ case AGGR_SOCKET:
+ case AGGR_DIE:
+ case AGGR_CORE:
+ fprintf(output, "#%*s %-*s cpus",
+ INTERVAL_LEN - 1, "time",
+ aggr_header_lens[config->aggr_mode],
+ aggr_header_std[config->aggr_mode]);
+ break;
+ case AGGR_NONE:
+ fprintf(output, "#%*s %-*s",
+ INTERVAL_LEN - 1, "time",
+ aggr_header_lens[config->aggr_mode],
+ aggr_header_std[config->aggr_mode]);
+ break;
+ case AGGR_THREAD:
+ fprintf(output, "#%*s %*s-%-*s",
+ INTERVAL_LEN - 1, "time",
+ COMM_LEN, "comm", PID_LEN, "pid");
+ break;
+ case AGGR_GLOBAL:
+ default:
+ if (!config->iostat_run)
+ fprintf(output, "#%*s",
+ INTERVAL_LEN - 1, "time");
+ case AGGR_UNSET:
+ case AGGR_MAX:
+ break;
+ }
+
+ if (config->metric_only)
+ print_metric_headers(config, evlist, true);
+ else
+ fprintf(output, " %*s %*s events\n",
+ COUNTS_LEN, "counts", config->unit_width, "unit");
+}
+
+static void print_header_std(struct perf_stat_config *config,
+ struct target *_target, struct evlist *evlist,
+ int argc, const char **argv)
{
FILE *output = config->output;
int i;
+ fprintf(output, "\n");
+ fprintf(output, " Performance counter stats for ");
+ if (_target->bpf_str)
+ fprintf(output, "\'BPF program(s) %s", _target->bpf_str);
+ else if (_target->system_wide)
+ fprintf(output, "\'system wide");
+ else if (_target->cpu_list)
+ fprintf(output, "\'CPU(s) %s", _target->cpu_list);
+ else if (!target__has_task(_target)) {
+ fprintf(output, "\'%s", argv ? argv[0] : "pipe");
+ for (i = 1; argv && (i < argc); i++)
+ fprintf(output, " %s", argv[i]);
+ } else if (_target->pid)
+ fprintf(output, "process id \'%s", _target->pid);
+ else
+ fprintf(output, "thread id \'%s", _target->tid);
+
+ fprintf(output, "\'");
+ if (config->run_count > 1)
+ fprintf(output, " (%d runs)", config->run_count);
+ fprintf(output, ":\n\n");
+
+ if (config->metric_only)
+ print_metric_headers(config, evlist, false);
+}
+
+static void print_header_csv(struct perf_stat_config *config,
+ struct target *_target __maybe_unused,
+ struct evlist *evlist,
+ int argc __maybe_unused,
+ const char **argv __maybe_unused)
+{
+ if (config->metric_only)
+ print_metric_headers(config, evlist, true);
+}
+static void print_header_json(struct perf_stat_config *config,
+ struct target *_target __maybe_unused,
+ struct evlist *evlist,
+ int argc __maybe_unused,
+ const char **argv __maybe_unused)
+{
+ if (config->metric_only)
+ print_metric_headers(config, evlist, true);
+}
+
+static void print_header(struct perf_stat_config *config,
+ struct target *_target,
+ struct evlist *evlist,
+ int argc, const char **argv)
+{
+ static int num_print_iv;
+
fflush(stdout);
- if (!config->csv_output && !config->json_output) {
- fprintf(output, "\n");
- fprintf(output, " Performance counter stats for ");
- if (_target->bpf_str)
- fprintf(output, "\'BPF program(s) %s", _target->bpf_str);
- else if (_target->system_wide)
- fprintf(output, "\'system wide");
- else if (_target->cpu_list)
- fprintf(output, "\'CPU(s) %s", _target->cpu_list);
- else if (!target__has_task(_target)) {
- fprintf(output, "\'%s", argv ? argv[0] : "pipe");
- for (i = 1; argv && (i < argc); i++)
- fprintf(output, " %s", argv[i]);
- } else if (_target->pid)
- fprintf(output, "process id \'%s", _target->pid);
- else
- fprintf(output, "thread id \'%s", _target->tid);
+ if (config->interval_clear)
+ puts(CONSOLE_CLEAR);
- fprintf(output, "\'");
- if (config->run_count > 1)
- fprintf(output, " (%d runs)", config->run_count);
- fprintf(output, ":\n\n");
+ if (num_print_iv == 0 || config->interval_clear) {
+ if (config->json_output)
+ print_header_json(config, _target, evlist, argc, argv);
+ else if (config->csv_output)
+ print_header_csv(config, _target, evlist, argc, argv);
+ else if (config->interval)
+ print_header_interval_std(config, _target, evlist, argc, argv);
+ else
+ print_header_std(config, _target, evlist, argc, argv);
}
+
+ if (num_print_iv++ == 25)
+ num_print_iv = 0;
}
static int get_precision(double num)
@@ -1348,6 +1244,9 @@ static void print_footer(struct perf_stat_config *config)
double avg = avg_stats(config->walltime_nsecs_stats) / NSEC_PER_SEC;
FILE *output = config->output;
+ if (config->interval || config->csv_output || config->json_output)
+ return;
+
if (!config->null_run)
fprintf(output, "\n");
@@ -1376,7 +1275,7 @@ static void print_footer(struct perf_stat_config *config)
fprintf(output, " %17.*f +- %.*f seconds time elapsed",
precision, avg, precision, sd);
- print_noise_pct(config, sd, avg);
+ print_noise_pct(config, sd, avg, /*before_metric=*/false);
}
fprintf(output, "\n\n");
@@ -1393,121 +1292,127 @@ static void print_footer(struct perf_stat_config *config)
"the same PMU. Try reorganizing the group.\n");
}
-static void print_percore_thread(struct perf_stat_config *config,
- struct evsel *counter, char *prefix)
-{
- int s;
- struct aggr_cpu_id s2, id;
- struct perf_cpu_map *cpus;
- bool first = true;
- int idx;
- struct perf_cpu cpu;
-
- cpus = evsel__cpus(counter);
- perf_cpu_map__for_each_cpu(cpu, idx, cpus) {
- s2 = config->aggr_get_id(config, cpu);
- for (s = 0; s < config->aggr_map->nr; s++) {
- id = config->aggr_map->map[s];
- if (aggr_cpu_id__equal(&s2, &id))
- break;
- }
-
- print_counter_aggrdata(config, counter, s,
- prefix, false,
- &first, cpu);
- }
-}
-
static void print_percore(struct perf_stat_config *config,
- struct evsel *counter, char *prefix)
+ struct evsel *counter, struct outstate *os)
{
bool metric_only = config->metric_only;
FILE *output = config->output;
- int s;
- bool first = true;
+ struct cpu_aggr_map *core_map;
+ int s, c, i;
if (!config->aggr_map || !config->aggr_get_id)
return;
if (config->percore_show_thread)
- return print_percore_thread(config, counter, prefix);
+ return print_counter(config, counter, os);
- for (s = 0; s < config->aggr_map->nr; s++) {
- if (prefix && metric_only)
- fprintf(output, "%s", prefix);
-
- print_counter_aggrdata(config, counter, s,
- prefix, metric_only,
- &first, (struct perf_cpu){ .cpu = -1 });
+ core_map = cpu_aggr_map__empty_new(config->aggr_map->nr);
+ if (core_map == NULL) {
+ fprintf(output, "Cannot allocate per-core aggr map for display\n");
+ return;
}
+ for (s = 0, c = 0; s < config->aggr_map->nr; s++) {
+ struct perf_cpu curr_cpu = config->aggr_map->map[s].cpu;
+ struct aggr_cpu_id core_id = aggr_cpu_id__core(curr_cpu, NULL);
+ bool found = false;
+
+ for (i = 0; i < c; i++) {
+ if (aggr_cpu_id__equal(&core_map->map[i], &core_id)) {
+ found = true;
+ break;
+ }
+ }
+ if (found)
+ continue;
+
+ print_counter_aggrdata(config, counter, s, os);
+
+ core_map->map[c++] = core_id;
+ }
+ free(core_map);
+
if (metric_only)
fputc('\n', output);
}
+static void print_cgroup_counter(struct perf_stat_config *config, struct evlist *evlist,
+ struct outstate *os)
+{
+ struct evsel *counter;
+
+ evlist__for_each_entry(evlist, counter) {
+ if (os->cgrp != counter->cgrp) {
+ if (os->cgrp != NULL)
+ print_metric_end(config, os);
+
+ os->cgrp = counter->cgrp;
+ print_metric_begin(config, evlist, os, /*aggr_idx=*/0);
+ }
+
+ print_counter(config, counter, os);
+ }
+ if (os->cgrp)
+ print_metric_end(config, os);
+}
+
void evlist__print_counters(struct evlist *evlist, struct perf_stat_config *config,
- struct target *_target, struct timespec *ts, int argc, const char **argv)
+ struct target *_target, struct timespec *ts,
+ int argc, const char **argv)
{
bool metric_only = config->metric_only;
int interval = config->interval;
struct evsel *counter;
- char buf[64], *prefix = NULL;
+ char buf[64];
+ struct outstate os = {
+ .fh = config->output,
+ .first = true,
+ };
if (config->iostat_run)
evlist->selected = evlist__first(evlist);
- if (interval)
- print_interval(config, evlist, prefix = buf, ts);
- else
- print_header(config, _target, argc, argv);
-
- if (metric_only) {
- static int num_print_iv;
-
- if (num_print_iv == 0 && !interval)
- print_metric_headers(config, evlist, prefix, false);
- if (num_print_iv++ == 25)
- num_print_iv = 0;
- if (config->aggr_mode == AGGR_GLOBAL && prefix && !config->iostat_run)
- fprintf(config->output, "%s", prefix);
-
- if (config->json_output && !config->metric_only)
- fprintf(config->output, "}");
+ if (interval) {
+ os.prefix = buf;
+ prepare_interval(config, buf, sizeof(buf), ts);
}
+ print_header(config, _target, evlist, argc, argv);
+
switch (config->aggr_mode) {
case AGGR_CORE:
case AGGR_DIE:
case AGGR_SOCKET:
case AGGR_NODE:
- print_aggr(config, evlist, prefix);
+ if (config->cgroup_list)
+ print_aggr_cgroup(config, evlist, &os);
+ else
+ print_aggr(config, evlist, &os);
break;
case AGGR_THREAD:
- evlist__for_each_entry(evlist, counter) {
- print_aggr_thread(config, _target, counter, prefix);
- }
- break;
case AGGR_GLOBAL:
- if (config->iostat_run)
- iostat_print_counters(evlist, config, ts, prefix = buf,
- print_counter_aggr);
- else {
+ if (config->iostat_run) {
+ iostat_print_counters(evlist, config, ts, buf,
+ (iostat_print_counter_t)print_counter, &os);
+ } else if (config->cgroup_list) {
+ print_cgroup_counter(config, evlist, &os);
+ } else {
+ print_metric_begin(config, evlist, &os, /*aggr_idx=*/0);
evlist__for_each_entry(evlist, counter) {
- print_counter_aggr(config, counter, prefix);
+ print_counter(config, counter, &os);
}
- if (metric_only)
- fputc('\n', config->output);
+ print_metric_end(config, &os);
}
break;
case AGGR_NONE:
if (metric_only)
- print_no_aggr_metric(config, evlist, prefix);
+ print_no_aggr_metric(config, evlist, &os);
else {
evlist__for_each_entry(evlist, counter) {
if (counter->percore)
- print_percore(config, counter, prefix);
+ print_percore(config, counter, &os);
else
- print_counter(config, counter, prefix);
+ print_counter(config, counter, &os);
}
}
break;
@@ -1517,8 +1422,7 @@ void evlist__print_counters(struct evlist *evlist, struct perf_stat_config *conf
break;
}
- if (!interval && !config->csv_output && !config->json_output)
- print_footer(config);
+ print_footer(config);
fflush(config->output);
}
diff --git a/tools/perf/util/stat-shadow.c b/tools/perf/util/stat-shadow.c
index 0bf71b0..cadb2df 100644
--- a/tools/perf/util/stat-shadow.c
+++ b/tools/perf/util/stat-shadow.c
@@ -14,6 +14,7 @@
#include "units.h"
#include <linux/zalloc.h>
#include "iostat.h"
+#include "util/hashmap.h"
/*
* AGGR_GLOBAL: Use CPU 0
diff --git a/tools/perf/util/stat.c b/tools/perf/util/stat.c
index c0656f8..534d36d 100644
--- a/tools/perf/util/stat.c
+++ b/tools/perf/util/stat.c
@@ -14,11 +14,7 @@
#include "evlist.h"
#include "evsel.h"
#include "thread_map.h"
-#ifdef HAVE_LIBBPF_SUPPORT
-#include <bpf/hashmap.h>
-#else
#include "util/hashmap.h"
-#endif
#include <linux/zalloc.h>
void update_stats(struct stats *stats, u64 val)
@@ -130,18 +126,65 @@ static void perf_stat_evsel_id_init(struct evsel *evsel)
}
}
+static void evsel__reset_aggr_stats(struct evsel *evsel)
+{
+ struct perf_stat_evsel *ps = evsel->stats;
+ struct perf_stat_aggr *aggr = ps->aggr;
+
+ if (aggr)
+ memset(aggr, 0, sizeof(*aggr) * ps->nr_aggr);
+}
+
static void evsel__reset_stat_priv(struct evsel *evsel)
{
struct perf_stat_evsel *ps = evsel->stats;
init_stats(&ps->res_stats);
+ evsel__reset_aggr_stats(evsel);
}
-static int evsel__alloc_stat_priv(struct evsel *evsel)
+static int evsel__alloc_aggr_stats(struct evsel *evsel, int nr_aggr)
{
- evsel->stats = zalloc(sizeof(struct perf_stat_evsel));
- if (evsel->stats == NULL)
+ struct perf_stat_evsel *ps = evsel->stats;
+
+ if (ps == NULL)
+ return 0;
+
+ ps->nr_aggr = nr_aggr;
+ ps->aggr = calloc(nr_aggr, sizeof(*ps->aggr));
+ if (ps->aggr == NULL)
return -ENOMEM;
+
+ return 0;
+}
+
+int evlist__alloc_aggr_stats(struct evlist *evlist, int nr_aggr)
+{
+ struct evsel *evsel;
+
+ evlist__for_each_entry(evlist, evsel) {
+ if (evsel__alloc_aggr_stats(evsel, nr_aggr) < 0)
+ return -1;
+ }
+ return 0;
+}
+
+static int evsel__alloc_stat_priv(struct evsel *evsel, int nr_aggr)
+{
+ struct perf_stat_evsel *ps;
+
+ ps = zalloc(sizeof(*ps));
+ if (ps == NULL)
+ return -ENOMEM;
+
+ evsel->stats = ps;
+
+ if (nr_aggr && evsel__alloc_aggr_stats(evsel, nr_aggr) < 0) {
+ evsel->stats = NULL;
+ free(ps);
+ return -ENOMEM;
+ }
+
perf_stat_evsel_id_init(evsel);
evsel__reset_stat_priv(evsel);
return 0;
@@ -151,8 +194,10 @@ static void evsel__free_stat_priv(struct evsel *evsel)
{
struct perf_stat_evsel *ps = evsel->stats;
- if (ps)
+ if (ps) {
+ zfree(&ps->aggr);
zfree(&ps->group_data);
+ }
zfree(&evsel->stats);
}
@@ -181,9 +226,9 @@ static void evsel__reset_prev_raw_counts(struct evsel *evsel)
perf_counts__reset(evsel->prev_raw_counts);
}
-static int evsel__alloc_stats(struct evsel *evsel, bool alloc_raw)
+static int evsel__alloc_stats(struct evsel *evsel, int nr_aggr, bool alloc_raw)
{
- if (evsel__alloc_stat_priv(evsel) < 0 ||
+ if (evsel__alloc_stat_priv(evsel, nr_aggr) < 0 ||
evsel__alloc_counts(evsel) < 0 ||
(alloc_raw && evsel__alloc_prev_raw_counts(evsel) < 0))
return -ENOMEM;
@@ -191,12 +236,17 @@ static int evsel__alloc_stats(struct evsel *evsel, bool alloc_raw)
return 0;
}
-int evlist__alloc_stats(struct evlist *evlist, bool alloc_raw)
+int evlist__alloc_stats(struct perf_stat_config *config,
+ struct evlist *evlist, bool alloc_raw)
{
struct evsel *evsel;
+ int nr_aggr = 0;
+
+ if (config && config->aggr_map)
+ nr_aggr = config->aggr_map->nr;
evlist__for_each_entry(evlist, evsel) {
- if (evsel__alloc_stats(evsel, alloc_raw))
+ if (evsel__alloc_stats(evsel, nr_aggr, alloc_raw))
goto out_free;
}
@@ -228,6 +278,14 @@ void evlist__reset_stats(struct evlist *evlist)
}
}
+void evlist__reset_aggr_stats(struct evlist *evlist)
+{
+ struct evsel *evsel;
+
+ evlist__for_each_entry(evlist, evsel)
+ evsel__reset_aggr_stats(evsel);
+}
+
void evlist__reset_prev_raw_counts(struct evlist *evlist)
{
struct evsel *evsel;
@@ -246,8 +304,6 @@ static void evsel__copy_prev_raw_counts(struct evsel *evsel)
*perf_counts(evsel->prev_raw_counts, idx, thread);
}
}
-
- evsel->counts->aggr = evsel->prev_raw_counts->aggr;
}
void evlist__copy_prev_raw_counts(struct evlist *evlist)
@@ -258,26 +314,6 @@ void evlist__copy_prev_raw_counts(struct evlist *evlist)
evsel__copy_prev_raw_counts(evsel);
}
-void evlist__save_aggr_prev_raw_counts(struct evlist *evlist)
-{
- struct evsel *evsel;
-
- /*
- * To collect the overall statistics for interval mode,
- * we copy the counts from evsel->prev_raw_counts to
- * evsel->counts. The perf_stat_process_counter creates
- * aggr values from per cpu values, but the per cpu values
- * are 0 for AGGR_GLOBAL. So we use a trick that saves the
- * previous aggr value to the first member of perf_counts,
- * then aggr calculation in process_counter_values can work
- * correctly.
- */
- evlist__for_each_entry(evlist, evsel) {
- *perf_counts(evsel->prev_raw_counts, 0, 0) =
- evsel->prev_raw_counts->aggr;
- }
-}
-
static size_t pkg_id_hash(long __key, void *ctx __maybe_unused)
{
uint64_t *key = (uint64_t *) __key;
@@ -355,12 +391,31 @@ static int check_per_pkg(struct evsel *counter, struct perf_counts_values *vals,
return ret;
}
+static bool evsel__count_has_error(struct evsel *evsel,
+ struct perf_counts_values *count,
+ struct perf_stat_config *config)
+{
+ /* the evsel was failed already */
+ if (evsel->err || evsel->counts->scaled == -1)
+ return true;
+
+ /* this is meaningful for CPU aggregation modes only */
+ if (config->aggr_mode == AGGR_GLOBAL)
+ return false;
+
+ /* it's considered ok when it actually ran */
+ if (count->ena != 0 && count->run != 0)
+ return false;
+
+ return true;
+}
+
static int
process_counter_values(struct perf_stat_config *config, struct evsel *evsel,
int cpu_map_idx, int thread,
struct perf_counts_values *count)
{
- struct perf_counts_values *aggr = &evsel->counts->aggr;
+ struct perf_stat_evsel *ps = evsel->stats;
static struct perf_counts_values zero;
bool skip = false;
@@ -372,34 +427,60 @@ process_counter_values(struct perf_stat_config *config, struct evsel *evsel,
if (skip)
count = &zero;
- switch (config->aggr_mode) {
- case AGGR_THREAD:
- case AGGR_CORE:
- case AGGR_DIE:
- case AGGR_SOCKET:
- case AGGR_NODE:
- case AGGR_NONE:
- if (!evsel->snapshot)
- evsel__compute_deltas(evsel, cpu_map_idx, thread, count);
- perf_counts_values__scale(count, config->scale, NULL);
- if ((config->aggr_mode == AGGR_NONE) && (!evsel->percore)) {
- perf_stat__update_shadow_stats(evsel, count->val,
- cpu_map_idx, &rt_stat);
- }
+ if (!evsel->snapshot)
+ evsel__compute_deltas(evsel, cpu_map_idx, thread, count);
+ perf_counts_values__scale(count, config->scale, NULL);
- if (config->aggr_mode == AGGR_THREAD) {
- perf_stat__update_shadow_stats(evsel, count->val,
- thread, &rt_stat);
+ if (config->aggr_mode == AGGR_THREAD) {
+ struct perf_counts_values *aggr_counts = &ps->aggr[thread].counts;
+
+ /*
+ * Skip value 0 when enabling --per-thread globally,
+ * otherwise too many 0 output.
+ */
+ if (count->val == 0 && config->system_wide)
+ return 0;
+
+ ps->aggr[thread].nr++;
+
+ aggr_counts->val += count->val;
+ aggr_counts->ena += count->ena;
+ aggr_counts->run += count->run;
+ return 0;
+ }
+
+ if (ps->aggr) {
+ struct perf_cpu cpu = perf_cpu_map__cpu(evsel->core.cpus, cpu_map_idx);
+ struct aggr_cpu_id aggr_id = config->aggr_get_id(config, cpu);
+ struct perf_stat_aggr *ps_aggr;
+ int i;
+
+ for (i = 0; i < ps->nr_aggr; i++) {
+ if (!aggr_cpu_id__equal(&aggr_id, &config->aggr_map->map[i]))
+ continue;
+
+ ps_aggr = &ps->aggr[i];
+ ps_aggr->nr++;
+
+ /*
+ * When any result is bad, make them all to give consistent output
+ * in interval mode. But per-task counters can have 0 enabled time
+ * when some tasks are idle.
+ */
+ if (evsel__count_has_error(evsel, count, config) && !ps_aggr->failed) {
+ ps_aggr->counts.val = 0;
+ ps_aggr->counts.ena = 0;
+ ps_aggr->counts.run = 0;
+ ps_aggr->failed = true;
+ }
+
+ if (!ps_aggr->failed) {
+ ps_aggr->counts.val += count->val;
+ ps_aggr->counts.ena += count->ena;
+ ps_aggr->counts.run += count->run;
+ }
+ break;
}
- break;
- case AGGR_GLOBAL:
- aggr->val += count->val;
- aggr->ena += count->ena;
- aggr->run += count->run;
- case AGGR_UNSET:
- case AGGR_MAX:
- default:
- break;
}
return 0;
@@ -426,13 +507,10 @@ static int process_counter_maps(struct perf_stat_config *config,
int perf_stat_process_counter(struct perf_stat_config *config,
struct evsel *counter)
{
- struct perf_counts_values *aggr = &counter->counts->aggr;
struct perf_stat_evsel *ps = counter->stats;
- u64 *count = counter->counts->aggr.values;
+ u64 *count;
int ret;
- aggr->val = aggr->ena = aggr->run = 0;
-
if (counter->per_pkg)
evsel__zero_per_pkg(counter);
@@ -443,10 +521,11 @@ int perf_stat_process_counter(struct perf_stat_config *config,
if (config->aggr_mode != AGGR_GLOBAL)
return 0;
- if (!counter->snapshot)
- evsel__compute_deltas(counter, -1, -1, aggr);
- perf_counts_values__scale(aggr, config->scale, &counter->counts->scaled);
-
+ /*
+ * GLOBAL aggregation mode only has a single aggr counts,
+ * so we can use ps->aggr[0] as the actual output.
+ */
+ count = ps->aggr[0].counts.values;
update_stats(&ps->res_stats, *count);
if (verbose > 0) {
@@ -454,13 +533,194 @@ int perf_stat_process_counter(struct perf_stat_config *config,
evsel__name(counter), count[0], count[1], count[2]);
}
- /*
- * Save the full runtime - to allow normalization during printout:
- */
- perf_stat__update_shadow_stats(counter, *count, 0, &rt_stat);
+ return 0;
+}
+
+static int evsel__merge_aggr_counters(struct evsel *evsel, struct evsel *alias)
+{
+ struct perf_stat_evsel *ps_a = evsel->stats;
+ struct perf_stat_evsel *ps_b = alias->stats;
+ int i;
+
+ if (ps_a->aggr == NULL && ps_b->aggr == NULL)
+ return 0;
+
+ if (ps_a->nr_aggr != ps_b->nr_aggr) {
+ pr_err("Unmatched aggregation mode between aliases\n");
+ return -1;
+ }
+
+ for (i = 0; i < ps_a->nr_aggr; i++) {
+ struct perf_counts_values *aggr_counts_a = &ps_a->aggr[i].counts;
+ struct perf_counts_values *aggr_counts_b = &ps_b->aggr[i].counts;
+
+ /* NB: don't increase aggr.nr for aliases */
+
+ aggr_counts_a->val += aggr_counts_b->val;
+ aggr_counts_a->ena += aggr_counts_b->ena;
+ aggr_counts_a->run += aggr_counts_b->run;
+ }
return 0;
}
+/* events should have the same name, scale, unit, cgroup but on different PMUs */
+static bool evsel__is_alias(struct evsel *evsel_a, struct evsel *evsel_b)
+{
+ if (strcmp(evsel__name(evsel_a), evsel__name(evsel_b)))
+ return false;
+
+ if (evsel_a->scale != evsel_b->scale)
+ return false;
+
+ if (evsel_a->cgrp != evsel_b->cgrp)
+ return false;
+
+ if (strcmp(evsel_a->unit, evsel_b->unit))
+ return false;
+
+ if (evsel__is_clock(evsel_a) != evsel__is_clock(evsel_b))
+ return false;
+
+ return !!strcmp(evsel_a->pmu_name, evsel_b->pmu_name);
+}
+
+static void evsel__merge_aliases(struct evsel *evsel)
+{
+ struct evlist *evlist = evsel->evlist;
+ struct evsel *alias;
+
+ alias = list_prepare_entry(evsel, &(evlist->core.entries), core.node);
+ list_for_each_entry_continue(alias, &evlist->core.entries, core.node) {
+ /* Merge the same events on different PMUs. */
+ if (evsel__is_alias(evsel, alias)) {
+ evsel__merge_aggr_counters(evsel, alias);
+ alias->merged_stat = true;
+ }
+ }
+}
+
+static bool evsel__should_merge_hybrid(const struct evsel *evsel,
+ const struct perf_stat_config *config)
+{
+ return config->hybrid_merge && evsel__is_hybrid(evsel);
+}
+
+static void evsel__merge_stats(struct evsel *evsel, struct perf_stat_config *config)
+{
+ /* this evsel is already merged */
+ if (evsel->merged_stat)
+ return;
+
+ if (evsel->auto_merge_stats || evsel__should_merge_hybrid(evsel, config))
+ evsel__merge_aliases(evsel);
+}
+
+/* merge the same uncore and hybrid events if requested */
+void perf_stat_merge_counters(struct perf_stat_config *config, struct evlist *evlist)
+{
+ struct evsel *evsel;
+
+ if (config->no_merge)
+ return;
+
+ evlist__for_each_entry(evlist, evsel)
+ evsel__merge_stats(evsel, config);
+}
+
+static void evsel__update_percore_stats(struct evsel *evsel, struct aggr_cpu_id *core_id)
+{
+ struct perf_stat_evsel *ps = evsel->stats;
+ struct perf_counts_values counts = { 0, };
+ struct aggr_cpu_id id;
+ struct perf_cpu cpu;
+ int idx;
+
+ /* collect per-core counts */
+ perf_cpu_map__for_each_cpu(cpu, idx, evsel->core.cpus) {
+ struct perf_stat_aggr *aggr = &ps->aggr[idx];
+
+ id = aggr_cpu_id__core(cpu, NULL);
+ if (!aggr_cpu_id__equal(core_id, &id))
+ continue;
+
+ counts.val += aggr->counts.val;
+ counts.ena += aggr->counts.ena;
+ counts.run += aggr->counts.run;
+ }
+
+ /* update aggregated per-core counts for each CPU */
+ perf_cpu_map__for_each_cpu(cpu, idx, evsel->core.cpus) {
+ struct perf_stat_aggr *aggr = &ps->aggr[idx];
+
+ id = aggr_cpu_id__core(cpu, NULL);
+ if (!aggr_cpu_id__equal(core_id, &id))
+ continue;
+
+ aggr->counts.val = counts.val;
+ aggr->counts.ena = counts.ena;
+ aggr->counts.run = counts.run;
+
+ aggr->used = true;
+ }
+}
+
+/* we have an aggr_map for cpu, but want to aggregate the counters per-core */
+static void evsel__process_percore(struct evsel *evsel)
+{
+ struct perf_stat_evsel *ps = evsel->stats;
+ struct aggr_cpu_id core_id;
+ struct perf_cpu cpu;
+ int idx;
+
+ if (!evsel->percore)
+ return;
+
+ perf_cpu_map__for_each_cpu(cpu, idx, evsel->core.cpus) {
+ struct perf_stat_aggr *aggr = &ps->aggr[idx];
+
+ if (aggr->used)
+ continue;
+
+ core_id = aggr_cpu_id__core(cpu, NULL);
+ evsel__update_percore_stats(evsel, &core_id);
+ }
+}
+
+/* process cpu stats on per-core events */
+void perf_stat_process_percore(struct perf_stat_config *config, struct evlist *evlist)
+{
+ struct evsel *evsel;
+
+ if (config->aggr_mode != AGGR_NONE)
+ return;
+
+ evlist__for_each_entry(evlist, evsel)
+ evsel__process_percore(evsel);
+}
+
+static void evsel__update_shadow_stats(struct evsel *evsel)
+{
+ struct perf_stat_evsel *ps = evsel->stats;
+ int i;
+
+ if (ps->aggr == NULL)
+ return;
+
+ for (i = 0; i < ps->nr_aggr; i++) {
+ struct perf_counts_values *aggr_counts = &ps->aggr[i].counts;
+
+ perf_stat__update_shadow_stats(evsel, aggr_counts->val, i, &rt_stat);
+ }
+}
+
+void perf_stat_process_shadow_stats(struct perf_stat_config *config __maybe_unused,
+ struct evlist *evlist)
+{
+ struct evsel *evsel;
+
+ evlist__for_each_entry(evlist, evsel)
+ evsel__update_shadow_stats(evsel);
+}
int perf_event__process_stat_event(struct perf_session *session,
union perf_event *event)
diff --git a/tools/perf/util/stat.h b/tools/perf/util/stat.h
index b0899c6..499c3bf 100644
--- a/tools/perf/util/stat.h
+++ b/tools/perf/util/stat.h
@@ -8,6 +8,7 @@
#include <sys/resource.h>
#include "cpumap.h"
#include "rblist.h"
+#include "counts.h"
struct perf_cpu_map;
struct perf_stat_config;
@@ -42,9 +43,29 @@ enum perf_stat_evsel_id {
PERF_STAT_EVSEL_ID__MAX,
};
+/* hold aggregated event info */
+struct perf_stat_aggr {
+ /* aggregated values */
+ struct perf_counts_values counts;
+ /* number of entries (CPUs) aggregated */
+ int nr;
+ /* whether any entry has failed to read/process event */
+ bool failed;
+ /* to mark this data is processed already */
+ bool used;
+};
+
+/* per-evsel event stats */
struct perf_stat_evsel {
+ /* used for repeated runs */
struct stats res_stats;
+ /* evsel id for quick check */
enum perf_stat_evsel_id id;
+ /* number of allocated 'aggr' */
+ int nr_aggr;
+ /* aggregated event values */
+ struct perf_stat_aggr *aggr;
+ /* used for group read */
u64 *group_data;
};
@@ -139,7 +160,6 @@ struct perf_stat_config {
bool metric_no_group;
bool metric_no_merge;
bool stop_read_counter;
- bool quiet;
bool iostat_run;
char *user_requested_cpu_list;
bool system_wide;
@@ -203,15 +223,6 @@ static inline void update_rusage_stats(struct rusage_stats *ru_stats, struct rus
struct evsel;
struct evlist;
-struct perf_aggr_thread_value {
- struct evsel *counter;
- struct aggr_cpu_id id;
- double uval;
- u64 val;
- u64 run;
- u64 ena;
-};
-
bool __perf_stat_evsel__is(struct evsel *evsel, enum perf_stat_evsel_id id);
#define perf_stat_evsel__is(evsel, id) \
@@ -248,15 +259,23 @@ void perf_stat__print_shadow_stats(struct perf_stat_config *config,
struct runtime_stat *st);
void perf_stat__collect_metric_expr(struct evlist *);
-int evlist__alloc_stats(struct evlist *evlist, bool alloc_raw);
+int evlist__alloc_stats(struct perf_stat_config *config,
+ struct evlist *evlist, bool alloc_raw);
void evlist__free_stats(struct evlist *evlist);
void evlist__reset_stats(struct evlist *evlist);
void evlist__reset_prev_raw_counts(struct evlist *evlist);
void evlist__copy_prev_raw_counts(struct evlist *evlist);
void evlist__save_aggr_prev_raw_counts(struct evlist *evlist);
+int evlist__alloc_aggr_stats(struct evlist *evlist, int nr_aggr);
+void evlist__reset_aggr_stats(struct evlist *evlist);
+
int perf_stat_process_counter(struct perf_stat_config *config,
struct evsel *counter);
+void perf_stat_merge_counters(struct perf_stat_config *config, struct evlist *evlist);
+void perf_stat_process_percore(struct perf_stat_config *config, struct evlist *evlist);
+void perf_stat_process_shadow_stats(struct perf_stat_config *config, struct evlist *evlist);
+
struct perf_tool;
union perf_event;
struct perf_session;
diff --git a/tools/perf/util/symbol-elf.c b/tools/perf/util/symbol-elf.c
index 647b7df..8034569 100644
--- a/tools/perf/util/symbol-elf.c
+++ b/tools/perf/util/symbol-elf.c
@@ -1303,7 +1303,7 @@ dso__load_sym_internal(struct dso *dso, struct map *map, struct symsrc *syms_ss,
(!used_opd && syms_ss->adjust_symbols)) {
GElf_Phdr phdr;
- if (elf_read_program_header(syms_ss->elf,
+ if (elf_read_program_header(runtime_ss->elf,
(u64)sym.st_value, &phdr)) {
pr_debug4("%s: failed to find program header for "
"symbol: %s st_value: %#" PRIx64 "\n",
diff --git a/tools/perf/util/symbol.h b/tools/perf/util/symbol.h
index 0b893dc..e297de1 100644
--- a/tools/perf/util/symbol.h
+++ b/tools/perf/util/symbol.h
@@ -132,6 +132,8 @@ struct addr_location {
s32 socket;
};
+void addr_location__put(struct addr_location *al);
+
int dso__load(struct dso *dso, struct map *map);
int dso__load_vmlinux(struct dso *dso, struct map *map,
const char *vmlinux, bool vmlinux_allocated);
diff --git a/tools/perf/util/synthetic-events.c b/tools/perf/util/synthetic-events.c
index cccd293..3ab6a92 100644
--- a/tools/perf/util/synthetic-events.c
+++ b/tools/perf/util/synthetic-events.c
@@ -2157,6 +2157,7 @@ int perf_event__synthesize_attr(struct perf_tool *tool, struct perf_event_attr *
return err;
}
+#ifdef HAVE_LIBTRACEEVENT
int perf_event__synthesize_tracing_data(struct perf_tool *tool, int fd, struct evlist *evlist,
perf_event__handler_t process)
{
@@ -2203,6 +2204,7 @@ int perf_event__synthesize_tracing_data(struct perf_tool *tool, int fd, struct e
return aligned_size;
}
+#endif
int perf_event__synthesize_build_id(struct perf_tool *tool, struct dso *pos, u16 misc,
perf_event__handler_t process, struct machine *machine)
@@ -2218,8 +2220,9 @@ int perf_event__synthesize_build_id(struct perf_tool *tool, struct dso *pos, u16
len = pos->long_name_len + 1;
len = PERF_ALIGN(len, NAME_ALIGN);
memcpy(&ev.build_id.build_id, pos->bid.data, sizeof(pos->bid.data));
+ ev.build_id.size = pos->bid.size;
ev.build_id.header.type = PERF_RECORD_HEADER_BUILD_ID;
- ev.build_id.header.misc = misc;
+ ev.build_id.header.misc = misc | PERF_RECORD_MISC_BUILD_ID_SIZE;
ev.build_id.pid = machine->pid;
ev.build_id.header.size = sizeof(ev.build_id) + len;
memcpy(&ev.build_id.filename, pos->long_name, pos->long_name_len);
@@ -2354,6 +2357,7 @@ int perf_event__synthesize_for_pipe(struct perf_tool *tool,
}
ret += err;
+#ifdef HAVE_LIBTRACEEVENT
if (have_tracepoints(&evlist->core.entries)) {
int fd = perf_data__fd(data);
@@ -2373,6 +2377,9 @@ int perf_event__synthesize_for_pipe(struct perf_tool *tool,
}
ret += err;
}
+#else
+ (void)data;
+#endif
return ret;
}
diff --git a/tools/perf/util/thread.h b/tools/perf/util/thread.h
index 241f300..395c626 100644
--- a/tools/perf/util/thread.h
+++ b/tools/perf/util/thread.h
@@ -158,4 +158,7 @@ static inline bool thread__is_filtered(struct thread *thread)
void thread__free_stitch_list(struct thread *thread);
+void thread__resolve(struct thread *thread, struct addr_location *al,
+ struct perf_sample *sample);
+
#endif /* __PERF_THREAD_H */
diff --git a/tools/perf/util/thread_map.c b/tools/perf/util/thread_map.c
index c9bfe46..e848579 100644
--- a/tools/perf/util/thread_map.c
+++ b/tools/perf/util/thread_map.c
@@ -18,6 +18,7 @@
#include "thread_map.h"
#include "debug.h"
#include "event.h"
+#include <internal/threadmap.h>
/* Skip "." and ".." directories */
static int filter(const struct dirent *dir)
diff --git a/tools/perf/util/thread_map.h b/tools/perf/util/thread_map.h
index 3bb860a..00ec05f 100644
--- a/tools/perf/util/thread_map.h
+++ b/tools/perf/util/thread_map.h
@@ -4,8 +4,6 @@
#include <sys/types.h>
#include <stdio.h>
-#include <linux/refcount.h>
-#include <internal/threadmap.h>
#include <perf/threadmap.h>
struct perf_record_thread_map;
diff --git a/tools/perf/util/trace-event-info.c b/tools/perf/util/trace-event-info.c
index 892c323..c24b3a1 100644
--- a/tools/perf/util/trace-event-info.c
+++ b/tools/perf/util/trace-event-info.c
@@ -26,6 +26,7 @@
#include <api/fs/tracing_path.h>
#include "evsel.h"
#include "debug.h"
+#include "util.h"
#define VERSION "0.6"
#define MAX_EVENT_LENGTH 512
@@ -38,15 +39,6 @@ struct tracepoint_path {
struct tracepoint_path *next;
};
-int bigendian(void)
-{
- unsigned char str[] = { 0x1, 0x2, 0x3, 0x4, 0x0, 0x0, 0x0, 0x0};
- unsigned int *ptr;
-
- ptr = (unsigned int *)(void *)str;
- return *ptr == 0x01020304;
-}
-
/* unfortunately, you can not stat debugfs or proc files for size */
static int record_file(const char *file, ssize_t hdr_sz)
{
@@ -79,7 +71,7 @@ static int record_file(const char *file, ssize_t hdr_sz)
/* ugh, handle big-endian hdr_size == 4 */
sizep = (char*)&size;
- if (bigendian())
+ if (host_is_bigendian())
sizep += sizeof(u64) - hdr_sz;
if (hdr_sz && pwrite(output_fd, sizep, hdr_sz, hdr_pos) < 0) {
@@ -564,7 +556,7 @@ static int tracing_data_header(void)
return -1;
/* save endian */
- if (bigendian())
+ if (host_is_bigendian())
buf[0] = 1;
else
buf[0] = 0;
diff --git a/tools/perf/util/trace-event-parse.c b/tools/perf/util/trace-event-parse.c
index c9c83a4..2d3c257 100644
--- a/tools/perf/util/trace-event-parse.c
+++ b/tools/perf/util/trace-event-parse.c
@@ -11,6 +11,8 @@
#include "trace-event.h"
#include <linux/ctype.h>
+#include <linux/kernel.h>
+#include <traceevent/event-parse.h>
static int get_common_field(struct scripting_context *context,
int *offset, int *size, const char *type)
diff --git a/tools/perf/util/trace-event-read.c b/tools/perf/util/trace-event-read.c
index 8a01af7..1162c49 100644
--- a/tools/perf/util/trace-event-read.c
+++ b/tools/perf/util/trace-event-read.c
@@ -11,12 +11,14 @@
#include <sys/stat.h>
#include <sys/wait.h>
#include <sys/mman.h>
+#include <traceevent/event-parse.h>
#include <fcntl.h>
#include <unistd.h>
#include <errno.h>
#include "trace-event.h"
#include "debug.h"
+#include "util.h"
static int input_fd;
@@ -414,7 +416,7 @@ ssize_t trace_report(int fd, struct trace_event *tevent, bool __repipe)
return -1;
}
file_bigendian = buf[0];
- host_bigendian = bigendian();
+ host_bigendian = host_is_bigendian() ? 1 : 0;
if (trace_event__init(tevent)) {
pr_debug("trace_event__init failed");
diff --git a/tools/perf/util/trace-event-scripting.c b/tools/perf/util/trace-event-scripting.c
index 7172ca0..56175c5 100644
--- a/tools/perf/util/trace-event-scripting.c
+++ b/tools/perf/util/trace-event-scripting.c
@@ -9,12 +9,13 @@
#include <stdlib.h>
#include <string.h>
#include <errno.h>
+#include <traceevent/event-parse.h>
#include "debug.h"
#include "trace-event.h"
-#include "event.h"
#include "evsel.h"
#include <linux/zalloc.h>
+#include "util/sample.h"
struct scripting_context *scripting_context;
diff --git a/tools/perf/util/trace-event.c b/tools/perf/util/trace-event.c
index b3ee651..8ad75b3 100644
--- a/tools/perf/util/trace-event.c
+++ b/tools/perf/util/trace-event.c
@@ -1,5 +1,4 @@
// SPDX-License-Identifier: GPL-2.0
-
#include <stdio.h>
#include <unistd.h>
#include <stdlib.h>
diff --git a/tools/perf/util/trace-event.h b/tools/perf/util/trace-event.h
index 6409811..add6c5d 100644
--- a/tools/perf/util/trace-event.h
+++ b/tools/perf/util/trace-event.h
@@ -2,9 +2,11 @@
#ifndef _PERF_UTIL_TRACE_EVENT_H
#define _PERF_UTIL_TRACE_EVENT_H
-#include <traceevent/event-parse.h>
-#include "parse-events.h"
+#include <stdbool.h>
+#include <stdio.h>
+#include <linux/types.h>
+struct evlist;
struct machine;
struct perf_sample;
union perf_event;
@@ -18,6 +20,11 @@ struct trace_event {
struct tep_plugin_list *plugin_list;
};
+typedef char *(tep_func_resolver_t)(void *priv,
+ unsigned long long *addrp, char **modp);
+
+bool have_tracepoints(struct list_head *evlist);
+
int trace_event__init(struct trace_event *t);
void trace_event__cleanup(struct trace_event *t);
int trace_event__register_resolver(struct machine *machine,
@@ -27,8 +34,6 @@ trace_event__tp_format(const char *sys, const char *name);
struct tep_event *trace_event__tp_format_id(int id);
-int bigendian(void);
-
void event_format__fprintf(struct tep_event *event,
int cpu, void *data, int size, FILE *fp);
diff --git a/tools/perf/util/util.h b/tools/perf/util/util.h
index c1f2d42..1d3b300 100644
--- a/tools/perf/util/util.h
+++ b/tools/perf/util/util.h
@@ -1,6 +1,6 @@
/* SPDX-License-Identifier: GPL-2.0 */
-#ifndef GIT_COMPAT_UTIL_H
-#define GIT_COMPAT_UTIL_H
+#ifndef __PERF_UTIL_H
+#define __PERF_UTIL_H
#define _BSD_SOURCE 1
/* glibc 2.20 deprecates _BSD_SOURCE in favour of _DEFAULT_SOURCE */
@@ -94,4 +94,23 @@ int do_realloc_array_as_needed(void **arr, size_t *arr_sz, size_t x,
0; \
})
-#endif /* GIT_COMPAT_UTIL_H */
+static inline bool host_is_bigendian(void)
+{
+#ifdef __BYTE_ORDER__
+#if __BYTE_ORDER__ == __ORDER_LITTLE_ENDIAN__
+ return false;
+#elif __BYTE_ORDER__ == __ORDER_BIG_ENDIAN__
+ return true;
+#else
+#error "Unrecognized __BYTE_ORDER__"
+#endif
+#else /* !__BYTE_ORDER__ */
+ unsigned char str[] = { 0x1, 0x2, 0x3, 0x4, 0x0, 0x0, 0x0, 0x0};
+ unsigned int *ptr;
+
+ ptr = (unsigned int *)(void *)str;
+ return *ptr == 0x01020304;
+#endif
+}
+
+#endif /* __PERF_UTIL_H */
diff --git a/tools/testing/radix-tree/maple.c b/tools/testing/radix-tree/maple.c
index 2e91973..81fa7ec 100644
--- a/tools/testing/radix-tree/maple.c
+++ b/tools/testing/radix-tree/maple.c
@@ -1,7 +1,8 @@
// SPDX-License-Identifier: GPL-2.0+
/*
- * maple_tree.c: Userspace shim for maple tree test-suite
- * Copyright (c) 2018 Liam R. Howlett <Liam.Howlett@Oracle.com>
+ * maple_tree.c: Userspace testing for maple tree test-suite
+ * Copyright (c) 2018-2022 Oracle Corporation
+ * Author: Liam R. Howlett <Liam.Howlett@Oracle.com>
*
* Any tests that require internal knowledge of the tree or threads and other
* difficult to handle in kernel tests.
diff --git a/tools/testing/selftests/powerpc/dscr/dscr.h b/tools/testing/selftests/powerpc/dscr/dscr.h
index 13e9b9e..b703714 100644
--- a/tools/testing/selftests/powerpc/dscr/dscr.h
+++ b/tools/testing/selftests/powerpc/dscr/dscr.h
@@ -23,6 +23,7 @@
#include <sys/stat.h>
#include <sys/wait.h>
+#include "reg.h"
#include "utils.h"
#define THREADS 100 /* Max threads */
@@ -41,31 +42,23 @@
/* Prilvilege state DSCR access */
inline unsigned long get_dscr(void)
{
- unsigned long ret;
-
- asm volatile("mfspr %0,%1" : "=r" (ret) : "i" (SPRN_DSCR_PRIV));
-
- return ret;
+ return mfspr(SPRN_DSCR_PRIV);
}
inline void set_dscr(unsigned long val)
{
- asm volatile("mtspr %1,%0" : : "r" (val), "i" (SPRN_DSCR_PRIV));
+ mtspr(SPRN_DSCR_PRIV, val);
}
/* Problem state DSCR access */
inline unsigned long get_dscr_usr(void)
{
- unsigned long ret;
-
- asm volatile("mfspr %0,%1" : "=r" (ret) : "i" (SPRN_DSCR));
-
- return ret;
+ return mfspr(SPRN_DSCR);
}
inline void set_dscr_usr(unsigned long val)
{
- asm volatile("mtspr %1,%0" : : "r" (val), "i" (SPRN_DSCR));
+ mtspr(SPRN_DSCR, val);
}
/* Default DSCR access */
diff --git a/tools/testing/selftests/powerpc/dscr/dscr_sysfs_test.c b/tools/testing/selftests/powerpc/dscr/dscr_sysfs_test.c
index fbbdffd..f20d1c1 100644
--- a/tools/testing/selftests/powerpc/dscr/dscr_sysfs_test.c
+++ b/tools/testing/selftests/powerpc/dscr/dscr_sysfs_test.c
@@ -24,6 +24,7 @@ static int check_cpu_dscr_default(char *file, unsigned long val)
rc = read(fd, buf, sizeof(buf));
if (rc == -1) {
perror("read() failed");
+ close(fd);
return 1;
}
close(fd);
@@ -65,8 +66,10 @@ static int check_all_cpu_dscr_defaults(unsigned long val)
if (access(file, F_OK))
continue;
- if (check_cpu_dscr_default(file, val))
+ if (check_cpu_dscr_default(file, val)) {
+ closedir(sysfs);
return 1;
+ }
}
closedir(sysfs);
return 0;
diff --git a/tools/testing/selftests/powerpc/include/pkeys.h b/tools/testing/selftests/powerpc/include/pkeys.h
index 3312cb1..51729d9 100644
--- a/tools/testing/selftests/powerpc/include/pkeys.h
+++ b/tools/testing/selftests/powerpc/include/pkeys.h
@@ -24,7 +24,7 @@
#undef PKEY_DISABLE_EXECUTE
#define PKEY_DISABLE_EXECUTE 0x4
-/* Older versions of libc do not not define this */
+/* Older versions of libc do not define this */
#ifndef SEGV_PKUERR
#define SEGV_PKUERR 4
#endif
diff --git a/tools/testing/selftests/powerpc/ptrace/core-pkey.c b/tools/testing/selftests/powerpc/ptrace/core-pkey.c
index bbc05ff..4e8d0ce 100644
--- a/tools/testing/selftests/powerpc/ptrace/core-pkey.c
+++ b/tools/testing/selftests/powerpc/ptrace/core-pkey.c
@@ -329,7 +329,7 @@ static int parent(struct shared_info *info, pid_t pid)
core = mmap(NULL, core_size, PROT_READ, MAP_PRIVATE, fd, 0);
if (core == (void *) -1) {
- perror("Error mmaping core file");
+ perror("Error mmapping core file");
ret = TEST_FAIL;
goto out;
}
@@ -383,7 +383,7 @@ static int setup_core_pattern(char **core_pattern_, bool *changed_)
goto out;
}
- ret = fread(core_pattern, 1, PATH_MAX, f);
+ ret = fread(core_pattern, 1, PATH_MAX - 1, f);
fclose(f);
if (!ret) {
perror("Error reading core_pattern file");
@@ -391,6 +391,8 @@ static int setup_core_pattern(char **core_pattern_, bool *changed_)
goto out;
}
+ core_pattern[ret] = '\0';
+
/* Check whether we can predict the name of the core file. */
if (!strcmp(core_pattern, "core") || !strcmp(core_pattern, "core.%p"))
*changed_ = false;
diff --git a/tools/testing/selftests/powerpc/ptrace/perf-hwbreak.c b/tools/testing/selftests/powerpc/ptrace/perf-hwbreak.c
index ecde2c1..f75739b 100644
--- a/tools/testing/selftests/powerpc/ptrace/perf-hwbreak.c
+++ b/tools/testing/selftests/powerpc/ptrace/perf-hwbreak.c
@@ -17,8 +17,11 @@
* Copyright (C) 2018 Michael Neuling, IBM Corporation.
*/
+#define _GNU_SOURCE
+
#include <unistd.h>
#include <assert.h>
+#include <sched.h>
#include <stdio.h>
#include <stdlib.h>
#include <signal.h>
@@ -26,6 +29,7 @@
#include <sys/ioctl.h>
#include <sys/wait.h>
#include <sys/ptrace.h>
+#include <sys/resource.h>
#include <sys/sysinfo.h>
#include <asm/ptrace.h>
#include <elf.h>
@@ -140,17 +144,59 @@ static void disable_fds(int *fd, int n)
static int perf_systemwide_event_open(int *fd, __u32 type, __u64 addr, __u64 len)
{
- int i = 0;
+ int i, ncpus, cpu, ret = 0;
+ struct rlimit rlim;
+ cpu_set_t *mask;
+ size_t size;
- /* Assume online processors are 0 to nprocs for simplisity */
- for (i = 0; i < nprocs; i++) {
- fd[i] = perf_cpu_event_open(i, type, addr, len);
- if (fd[i] < 0) {
- close_fds(fd, i);
- return fd[i];
- }
+ if (getrlimit(RLIMIT_NOFILE, &rlim)) {
+ perror("getrlimit");
+ return -1;
}
- return 0;
+ rlim.rlim_cur = 65536;
+ if (setrlimit(RLIMIT_NOFILE, &rlim)) {
+ perror("setrlimit");
+ return -1;
+ }
+
+ ncpus = get_nprocs_conf();
+ size = CPU_ALLOC_SIZE(ncpus);
+ mask = CPU_ALLOC(ncpus);
+ if (!mask) {
+ perror("malloc");
+ return -1;
+ }
+
+ CPU_ZERO_S(size, mask);
+
+ if (sched_getaffinity(0, size, mask)) {
+ perror("sched_getaffinity");
+ ret = -1;
+ goto done;
+ }
+
+ for (i = 0, cpu = 0; i < nprocs && cpu < ncpus; cpu++) {
+ if (!CPU_ISSET_S(cpu, size, mask))
+ continue;
+ fd[i] = perf_cpu_event_open(cpu, type, addr, len);
+ if (fd[i] < 0) {
+ perror("perf_systemwide_event_open");
+ close_fds(fd, i);
+ ret = fd[i];
+ goto done;
+ }
+ i++;
+ }
+
+ if (i < nprocs) {
+ printf("Error: Number of online cpus reduced since start of test: %d < %d\n", i, nprocs);
+ close_fds(fd, i);
+ ret = -1;
+ }
+
+done:
+ CPU_FREE(mask);
+ return ret;
}
static inline bool breakpoint_test(int len)
@@ -543,15 +589,12 @@ static int test_syswide_multi_diff_addr(void)
int ret;
ret = perf_systemwide_event_open(fd1, HW_BREAKPOINT_RW, (__u64)&a, (__u64)sizeof(a));
- if (ret) {
- perror("perf_systemwide_event_open");
+ if (ret)
exit(EXIT_FAILURE);
- }
ret = perf_systemwide_event_open(fd2, HW_BREAKPOINT_RW, (__u64)&b, (__u64)sizeof(b));
if (ret) {
close_fds(fd1, nprocs);
- perror("perf_systemwide_event_open");
exit(EXIT_FAILURE);
}
@@ -590,15 +633,12 @@ static int test_syswide_multi_same_addr(void)
int ret;
ret = perf_systemwide_event_open(fd1, HW_BREAKPOINT_RW, (__u64)&a, (__u64)sizeof(a));
- if (ret) {
- perror("perf_systemwide_event_open");
+ if (ret)
exit(EXIT_FAILURE);
- }
ret = perf_systemwide_event_open(fd2, HW_BREAKPOINT_RW, (__u64)&a, (__u64)sizeof(a));
if (ret) {
close_fds(fd1, nprocs);
- perror("perf_systemwide_event_open");
exit(EXIT_FAILURE);
}
@@ -637,15 +677,12 @@ static int test_syswide_multi_diff_addr_ro_wo(void)
int ret;
ret = perf_systemwide_event_open(fd1, HW_BREAKPOINT_W, (__u64)&a, (__u64)sizeof(a));
- if (ret) {
- perror("perf_systemwide_event_open");
+ if (ret)
exit(EXIT_FAILURE);
- }
ret = perf_systemwide_event_open(fd2, HW_BREAKPOINT_R, (__u64)&b, (__u64)sizeof(b));
if (ret) {
close_fds(fd1, nprocs);
- perror("perf_systemwide_event_open");
exit(EXIT_FAILURE);
}
@@ -684,15 +721,12 @@ static int test_syswide_multi_same_addr_ro_wo(void)
int ret;
ret = perf_systemwide_event_open(fd1, HW_BREAKPOINT_W, (__u64)&a, (__u64)sizeof(a));
- if (ret) {
- perror("perf_systemwide_event_open");
+ if (ret)
exit(EXIT_FAILURE);
- }
ret = perf_systemwide_event_open(fd2, HW_BREAKPOINT_R, (__u64)&a, (__u64)sizeof(a));
if (ret) {
close_fds(fd1, nprocs);
- perror("perf_systemwide_event_open");
exit(EXIT_FAILURE);
}
diff --git a/tools/testing/selftests/powerpc/ptrace/ptrace-hwbreak.c b/tools/testing/selftests/powerpc/ptrace/ptrace-hwbreak.c
index a0635a3..1345e9b 100644
--- a/tools/testing/selftests/powerpc/ptrace/ptrace-hwbreak.c
+++ b/tools/testing/selftests/powerpc/ptrace/ptrace-hwbreak.c
@@ -23,6 +23,7 @@
#include <sys/syscall.h>
#include <linux/limits.h>
#include "ptrace.h"
+#include "reg.h"
#define SPRN_PVR 0x11F
#define PVR_8xx 0x00500000
@@ -620,10 +621,7 @@ static int ptrace_hwbreak(void)
int main(int argc, char **argv, char **envp)
{
- int pvr = 0;
- asm __volatile__ ("mfspr %0,%1" : "=r"(pvr) : "i"(SPRN_PVR));
- if (pvr == PVR_8xx)
- is_8xx = true;
+ is_8xx = mfspr(SPRN_PVR) == PVR_8xx;
return test_harness(ptrace_hwbreak, "ptrace-hwbreak");
}
diff --git a/tools/testing/selftests/powerpc/ptrace/ptrace.h b/tools/testing/selftests/powerpc/ptrace/ptrace.h
index 4e0233c..04788e5 100644
--- a/tools/testing/selftests/powerpc/ptrace/ptrace.h
+++ b/tools/testing/selftests/powerpc/ptrace/ptrace.h
@@ -745,10 +745,7 @@ int show_tm_spr(pid_t child, struct tm_spr_regs *out)
/* Analyse TEXASR after TM failure */
inline unsigned long get_tfiar(void)
{
- unsigned long ret;
-
- asm volatile("mfspr %0,%1" : "=r" (ret) : "i" (SPRN_TFIAR));
- return ret;
+ return mfspr(SPRN_TFIAR);
}
void analyse_texasr(unsigned long texasr)
diff --git a/tools/testing/selftests/powerpc/scripts/hmi.sh b/tools/testing/selftests/powerpc/scripts/hmi.sh
index dcdb392..bcc7b6b 100755
--- a/tools/testing/selftests/powerpc/scripts/hmi.sh
+++ b/tools/testing/selftests/powerpc/scripts/hmi.sh
@@ -36,7 +36,7 @@
# for each chip+core combination
# todo - less fragile parsing
-egrep -o 'OCC: Chip [0-9a-f]+ Core [0-9a-f]' < /sys/firmware/opal/msglog |
+grep -E -o 'OCC: Chip [0-9a-f]+ Core [0-9a-f]' < /sys/firmware/opal/msglog |
while read chipcore; do
chip=$(echo "$chipcore"|awk '{print $3}')
core=$(echo "$chipcore"|awk '{print $5}')
diff --git a/tools/testing/selftests/powerpc/security/flush_utils.c b/tools/testing/selftests/powerpc/security/flush_utils.c
index 4d95965..9c5c00e 100644
--- a/tools/testing/selftests/powerpc/security/flush_utils.c
+++ b/tools/testing/selftests/powerpc/security/flush_utils.c
@@ -14,6 +14,7 @@
#include <string.h>
#include <stdio.h>
#include <sys/utsname.h>
+#include "reg.h"
#include "utils.h"
#include "flush_utils.h"
@@ -79,5 +80,5 @@ void set_dscr(unsigned long val)
init = 1;
}
- asm volatile("mtspr %1,%0" : : "r" (val), "i" (SPRN_DSCR));
+ mtspr(SPRN_DSCR, val);
}
