Dual Port IB Device for RoCE
This feature allows RDMA resources (pd, mr, cq, qp, etc) to be used with
both physical ports of capable mlx5 devices. When enabled a single IB
device with two ports will be registered instead of two single port
There are still two PCI devices underlying the two port device, the
capabilities indicate which device is the "master" device and which is
When the add callback function is called for a slave device a list of IB
devices is searched for matching master device, indicated by the
capabilities and the system_image_guid. If a match is found the slave is
bound to the master device, otherwise it's placed on a list, in case
it's master becomes available in the future. When a master device is added it
searches the list of available slaves for a matching slave device. If a
match is found it binds the slave as its 2nd port. If no match as found
the device still appears as a dual port device, with the 2nd port down.
RDMA resources can still created that use the yet unavailable 2nd port.
Commands related to IB resources are all routed through the master
mlx5_core device. Port specific commands, like those for hardware
counters are routed to their respective port mlx5_core device. Since
devices can appear and disappear asynchronously a reference count on the
underlying mlx5_core device is maintained. Getting and putting this
reference is only necessary for commands destined to a specific port,
the master core device can be used freely, as it will exist while the IB
SR-IOV devices follow the same pattern as the physical ones. VFs of a
master port can bind VFs of slave ports, if available, and operate as
dual port devices.
Examples of devices passed to a VM:
(master) - One net device, one IB device that has two ports. The slave
port will always be down.
(slave) - One net device, no IB devices.
(slave, slave) - Two net devices and no IB devices.
(master, master) - Two net devices, two IB devices, each with two ports.
The slave port of each device will always be down.
(master, slave) - Two net devices, one IB device, with two ports. Both
ports can be used.
There are no changes to the existing design for net devices.
The feature is disabled by default and it is enabled in firmware
net/mlx5: Set num_vhca_ports capability
Set the current capability to the max capability. Doing so enables dual
port RoCE functionality if supported by the firmware.
Signed-off-by: Daniel Jurgens <firstname.lastname@example.org>
Reviewed-by: Parav Pandit <email@example.com>
Signed-off-by: Leon Romanovsky <firstname.lastname@example.org>
1 file changed