| From d6d78a02cb2b89a4a19c63ad86ebe247db5ae42e Mon Sep 17 00:00:00 2001 |
| From: Sasha Levin <sashal@kernel.org> |
| Date: Thu, 18 Feb 2021 20:50:31 +0000 |
| Subject: xsk: Respect device's headroom and tailroom on generic xmit path |
| |
| From: Alexander Lobakin <alobakin@pm.me> |
| |
| [ Upstream commit 3914d88f7608e6c2e80e344474fa289370c32451 ] |
| |
| xsk_generic_xmit() allocates a new skb and then queues it for |
| xmitting. The size of new skb's headroom is desc->len, so it comes |
| to the driver/device with no reserved headroom and/or tailroom. |
| Lots of drivers need some headroom (and sometimes tailroom) to |
| prepend (and/or append) some headers or data, e.g. CPU tags, |
| device-specific headers/descriptors (LSO, TLS etc.), and if case |
| of no available space skb_cow_head() will reallocate the skb. |
| Reallocations are unwanted on fast-path, especially when it comes |
| to XDP, so generic XSK xmit should reserve the spaces declared in |
| dev->needed_headroom and dev->needed tailroom to avoid them. |
| |
| Note on max(NET_SKB_PAD, L1_CACHE_ALIGN(dev->needed_headroom)): |
| |
| Usually, output functions reserve LL_RESERVED_SPACE(dev), which |
| consists of dev->hard_header_len + dev->needed_headroom, aligned |
| by 16. |
| |
| However, on XSK xmit hard header is already here in the chunk, so |
| hard_header_len is not needed. But it'd still be better to align |
| data up to cacheline, while reserving no less than driver requests |
| for headroom. NET_SKB_PAD here is to double-insure there will be |
| no reallocations even when the driver advertises no needed_headroom, |
| but in fact need it (not so rare case). |
| |
| Fixes: 35fcde7f8deb ("xsk: support for Tx") |
| Signed-off-by: Alexander Lobakin <alobakin@pm.me> |
| Signed-off-by: Daniel Borkmann <daniel@iogearbox.net> |
| Acked-by: Magnus Karlsson <magnus.karlsson@intel.com> |
| Acked-by: John Fastabend <john.fastabend@gmail.com> |
| Link: https://lore.kernel.org/bpf/20210218204908.5455-5-alobakin@pm.me |
| Signed-off-by: Sasha Levin <sashal@kernel.org> |
| --- |
| net/xdp/xsk.c | 8 +++++++- |
| 1 file changed, 7 insertions(+), 1 deletion(-) |
| |
| diff --git a/net/xdp/xsk.c b/net/xdp/xsk.c |
| index 4a83117507f5..9e865fe864b7 100644 |
| --- a/net/xdp/xsk.c |
| +++ b/net/xdp/xsk.c |
| @@ -439,12 +439,16 @@ static int xsk_generic_xmit(struct sock *sk) |
| struct sk_buff *skb; |
| unsigned long flags; |
| int err = 0; |
| + u32 hr, tr; |
| |
| mutex_lock(&xs->mutex); |
| |
| if (xs->queue_id >= xs->dev->real_num_tx_queues) |
| goto out; |
| |
| + hr = max(NET_SKB_PAD, L1_CACHE_ALIGN(xs->dev->needed_headroom)); |
| + tr = xs->dev->needed_tailroom; |
| + |
| while (xskq_cons_peek_desc(xs->tx, &desc, xs->pool)) { |
| char *buffer; |
| u64 addr; |
| @@ -456,11 +460,13 @@ static int xsk_generic_xmit(struct sock *sk) |
| } |
| |
| len = desc.len; |
| - skb = sock_alloc_send_skb(sk, len, 1, &err); |
| + skb = sock_alloc_send_skb(sk, hr + len + tr, 1, &err); |
| if (unlikely(!skb)) |
| goto out; |
| |
| + skb_reserve(skb, hr); |
| skb_put(skb, len); |
| + |
| addr = desc.addr; |
| buffer = xsk_buff_raw_get_data(xs->pool, addr); |
| err = skb_store_bits(skb, 0, buffer, len); |
| -- |
| 2.30.2 |
| |