From 13cb7504f410cbe6a545ac5665dec0a841b3fecc Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Linus=20L=C3=BCssing?= Date: Mon, 25 Jan 2021 01:43:34 +0100 Subject: [PATCH] kernel: bridge: Fix a deadlock when enabling multicast snooping [ Upstream commit 851d0a73c90e6c8c63fef106c6c1e73df7e05d9d ] From: Joseph Huang When enabling multicast snooping, bridge module deadlocks on multicast_lock if 1) IPv6 is enabled, and 2) there is an existing querier on the same L2 network. The deadlock was caused by the following sequence: While holding the lock, br_multicast_open calls br_multicast_join_snoopers, which eventually causes IP stack to (attempt to) send out a Listener Report (in igmp6_join_group). Since the destination Ethernet address is a multicast address, br_dev_xmit feeds the packet back to the bridge via br_multicast_rcv, which in turn calls br_multicast_add_group, which then deadlocks on multicast_lock. The fix is to move the call br_multicast_join_snoopers outside of the critical section. This works since br_multicast_join_snoopers only deals with IP and does not modify any multicast data structures of the bridge, so there's no need to hold the lock. Steps to reproduce: 1. sysctl net.ipv6.conf.all.force_mld_version=1 2. have another querier 3. ip link set dev bridge type bridge mcast_snooping 0 && \ ip link set dev bridge type bridge mcast_snooping 1 < deadlock > A typical call trace looks like the following: [ 936.251495] _raw_spin_lock+0x5c/0x68 [ 936.255221] br_multicast_add_group+0x40/0x170 [bridge] [ 936.260491] br_multicast_rcv+0x7ac/0xe30 [bridge] [ 936.265322] br_dev_xmit+0x140/0x368 [bridge] [ 936.269689] dev_hard_start_xmit+0x94/0x158 [ 936.273876] __dev_queue_xmit+0x5ac/0x7f8 [ 936.277890] dev_queue_xmit+0x10/0x18 [ 936.281563] neigh_resolve_output+0xec/0x198 [ 936.285845] ip6_finish_output2+0x240/0x710 [ 936.290039] __ip6_finish_output+0x130/0x170 [ 936.294318] ip6_output+0x6c/0x1c8 [ 936.297731] NF_HOOK.constprop.0+0xd8/0xe8 [ 936.301834] igmp6_send+0x358/0x558 [ 936.305326] igmp6_join_group.part.0+0x30/0xf0 [ 936.309774] igmp6_group_added+0xfc/0x110 [ 936.313787] __ipv6_dev_mc_inc+0x1a4/0x290 [ 936.317885] ipv6_dev_mc_inc+0x10/0x18 [ 936.321677] br_multicast_open+0xbc/0x110 [bridge] [ 936.326506] br_multicast_toggle+0xec/0x140 [bridge] Fixes: 4effd28c1245 ("bridge: join all-snoopers multicast address") Signed-off-by: Joseph Huang Acked-by: Nikolay Aleksandrov Link: https://lore.kernel.org/r/20201204235628.50653-1-Joseph.Huang@garmin.com Signed-off-by: Jakub Kicinski Signed-off-by: Greg Kroah-Hartman [linus.luessing@c0d3.blue: backported to 4.4] --- ...ock-when-enabling-multicast-snooping.patch | 275 ++++++++++++++++++ 1 file changed, 275 insertions(+) create mode 100644 patches/openwrt/0024-kernel-bridge-Fix-a-deadlock-when-enabling-multicast-snooping.patch diff --git a/patches/openwrt/0024-kernel-bridge-Fix-a-deadlock-when-enabling-multicast-snooping.patch b/patches/openwrt/0024-kernel-bridge-Fix-a-deadlock-when-enabling-multicast-snooping.patch new file mode 100644 index 00000000..ed867438 --- /dev/null +++ b/patches/openwrt/0024-kernel-bridge-Fix-a-deadlock-when-enabling-multicast-snooping.patch @@ -0,0 +1,275 @@ +From: Linus Lüssing +Date: Mon, 25 Jan 2021 01:38:19 +0100 +Subject: kernel: bridge: Fix a deadlock when enabling multicast snooping + +[ Upstream commit 851d0a73c90e6c8c63fef106c6c1e73df7e05d9d ] + +From: Joseph Huang + +When enabling multicast snooping, bridge module deadlocks on multicast_lock +if 1) IPv6 is enabled, and 2) there is an existing querier on the same L2 +network. + +The deadlock was caused by the following sequence: While holding the lock, +br_multicast_open calls br_multicast_join_snoopers, which eventually causes +IP stack to (attempt to) send out a Listener Report (in igmp6_join_group). +Since the destination Ethernet address is a multicast address, br_dev_xmit +feeds the packet back to the bridge via br_multicast_rcv, which in turn +calls br_multicast_add_group, which then deadlocks on multicast_lock. + +The fix is to move the call br_multicast_join_snoopers outside of the +critical section. This works since br_multicast_join_snoopers only deals +with IP and does not modify any multicast data structures of the bridge, +so there's no need to hold the lock. + +Steps to reproduce: +1. sysctl net.ipv6.conf.all.force_mld_version=1 +2. have another querier +3. ip link set dev bridge type bridge mcast_snooping 0 && \ + ip link set dev bridge type bridge mcast_snooping 1 < deadlock > + +A typical call trace looks like the following: + +[ 936.251495] _raw_spin_lock+0x5c/0x68 +[ 936.255221] br_multicast_add_group+0x40/0x170 [bridge] +[ 936.260491] br_multicast_rcv+0x7ac/0xe30 [bridge] +[ 936.265322] br_dev_xmit+0x140/0x368 [bridge] +[ 936.269689] dev_hard_start_xmit+0x94/0x158 +[ 936.273876] __dev_queue_xmit+0x5ac/0x7f8 +[ 936.277890] dev_queue_xmit+0x10/0x18 +[ 936.281563] neigh_resolve_output+0xec/0x198 +[ 936.285845] ip6_finish_output2+0x240/0x710 +[ 936.290039] __ip6_finish_output+0x130/0x170 +[ 936.294318] ip6_output+0x6c/0x1c8 +[ 936.297731] NF_HOOK.constprop.0+0xd8/0xe8 +[ 936.301834] igmp6_send+0x358/0x558 +[ 936.305326] igmp6_join_group.part.0+0x30/0xf0 +[ 936.309774] igmp6_group_added+0xfc/0x110 +[ 936.313787] __ipv6_dev_mc_inc+0x1a4/0x290 +[ 936.317885] ipv6_dev_mc_inc+0x10/0x18 +[ 936.321677] br_multicast_open+0xbc/0x110 [bridge] +[ 936.326506] br_multicast_toggle+0xec/0x140 [bridge] + +Fixes: 4effd28c1245 ("bridge: join all-snoopers multicast address") +Signed-off-by: Joseph Huang +Acked-by: Nikolay Aleksandrov +Link: https://lore.kernel.org/r/20201204235628.50653-1-Joseph.Huang@garmin.com +Signed-off-by: Jakub Kicinski +Signed-off-by: Greg Kroah-Hartman +[linus.luessing@c0d3.blue: backported to 4.4] + +diff --git a/target/linux/generic/backport-4.14/120-bridge-Fix-a-deadlock-when-enabling-multicast-snoopi.patch b/target/linux/generic/backport-4.14/120-bridge-Fix-a-deadlock-when-enabling-multicast-snoopi.patch +new file mode 100644 +index 0000000000000000000000000000000000000000..e89d9e3eb2063b1f478c0217aaa3c89d14880043 +--- /dev/null ++++ b/target/linux/generic/backport-4.14/120-bridge-Fix-a-deadlock-when-enabling-multicast-snoopi.patch +@@ -0,0 +1,209 @@ ++From 0e125bd346d3b9d6997e525b263834f6b650cf60 Mon Sep 17 00:00:00 2001 ++From: Joseph Huang ++Date: Fri, 4 Dec 2020 18:56:28 -0500 ++Subject: [PATCH] bridge: Fix a deadlock when enabling multicast snooping ++ ++[ Upstream commit 851d0a73c90e6c8c63fef106c6c1e73df7e05d9d ] ++ ++When enabling multicast snooping, bridge module deadlocks on multicast_lock ++if 1) IPv6 is enabled, and 2) there is an existing querier on the same L2 ++network. ++ ++The deadlock was caused by the following sequence: While holding the lock, ++br_multicast_open calls br_multicast_join_snoopers, which eventually causes ++IP stack to (attempt to) send out a Listener Report (in igmp6_join_group). ++Since the destination Ethernet address is a multicast address, br_dev_xmit ++feeds the packet back to the bridge via br_multicast_rcv, which in turn ++calls br_multicast_add_group, which then deadlocks on multicast_lock. ++ ++The fix is to move the call br_multicast_join_snoopers outside of the ++critical section. This works since br_multicast_join_snoopers only deals ++with IP and does not modify any multicast data structures of the bridge, ++so there's no need to hold the lock. ++ ++Steps to reproduce: ++1. sysctl net.ipv6.conf.all.force_mld_version=1 ++2. have another querier ++3. ip link set dev bridge type bridge mcast_snooping 0 && \ ++ ip link set dev bridge type bridge mcast_snooping 1 < deadlock > ++ ++A typical call trace looks like the following: ++ ++[ 936.251495] _raw_spin_lock+0x5c/0x68 ++[ 936.255221] br_multicast_add_group+0x40/0x170 [bridge] ++[ 936.260491] br_multicast_rcv+0x7ac/0xe30 [bridge] ++[ 936.265322] br_dev_xmit+0x140/0x368 [bridge] ++[ 936.269689] dev_hard_start_xmit+0x94/0x158 ++[ 936.273876] __dev_queue_xmit+0x5ac/0x7f8 ++[ 936.277890] dev_queue_xmit+0x10/0x18 ++[ 936.281563] neigh_resolve_output+0xec/0x198 ++[ 936.285845] ip6_finish_output2+0x240/0x710 ++[ 936.290039] __ip6_finish_output+0x130/0x170 ++[ 936.294318] ip6_output+0x6c/0x1c8 ++[ 936.297731] NF_HOOK.constprop.0+0xd8/0xe8 ++[ 936.301834] igmp6_send+0x358/0x558 ++[ 936.305326] igmp6_join_group.part.0+0x30/0xf0 ++[ 936.309774] igmp6_group_added+0xfc/0x110 ++[ 936.313787] __ipv6_dev_mc_inc+0x1a4/0x290 ++[ 936.317885] ipv6_dev_mc_inc+0x10/0x18 ++[ 936.321677] br_multicast_open+0xbc/0x110 [bridge] ++[ 936.326506] br_multicast_toggle+0xec/0x140 [bridge] ++ ++Fixes: 4effd28c1245 ("bridge: join all-snoopers multicast address") ++Signed-off-by: Joseph Huang ++Acked-by: Nikolay Aleksandrov ++Link: https://lore.kernel.org/r/20201204235628.50653-1-Joseph.Huang@garmin.com ++Signed-off-by: Jakub Kicinski ++Signed-off-by: Greg Kroah-Hartman ++[linus.luessing@c0d3.blue: backported to 4.4] ++--- ++ net/bridge/br_device.c | 6 ++++++ ++ net/bridge/br_multicast.c | 34 +++++++++++++++++++++++++--------- ++ net/bridge/br_private.h | 10 ++++++++++ ++ 3 files changed, 41 insertions(+), 9 deletions(-) ++ ++diff --git a/net/bridge/br_device.c b/net/bridge/br_device.c ++index b7cc322acdc8..4bd8a7ba002a 100644 ++--- a/net/bridge/br_device.c +++++ b/net/bridge/br_device.c ++@@ -140,6 +140,9 @@ static int br_dev_open(struct net_device *dev) ++ br_stp_enable_bridge(br); ++ br_multicast_open(br); ++ +++ if (!br->multicast_disabled) +++ br_multicast_join_snoopers(br); +++ ++ return 0; ++ } ++ ++@@ -160,6 +163,9 @@ static int br_dev_stop(struct net_device *dev) ++ br_stp_disable_bridge(br); ++ br_multicast_stop(br); ++ +++ if (!br->multicast_disabled) +++ br_multicast_leave_snoopers(br); +++ ++ netif_stop_queue(dev); ++ ++ return 0; ++diff --git a/net/bridge/br_multicast.c b/net/bridge/br_multicast.c ++index 6d8d1cf2ddc6..1b1d06549583 100644 ++--- a/net/bridge/br_multicast.c +++++ b/net/bridge/br_multicast.c ++@@ -2314,7 +2314,7 @@ static inline void br_ip6_multicast_join_snoopers(struct net_bridge *br) ++ } ++ #endif ++ ++-static void br_multicast_join_snoopers(struct net_bridge *br) +++void br_multicast_join_snoopers(struct net_bridge *br) ++ { ++ br_ip4_multicast_join_snoopers(br); ++ br_ip6_multicast_join_snoopers(br); ++@@ -2345,7 +2345,7 @@ static inline void br_ip6_multicast_leave_snoopers(struct net_bridge *br) ++ } ++ #endif ++ ++-static void br_multicast_leave_snoopers(struct net_bridge *br) +++void br_multicast_leave_snoopers(struct net_bridge *br) ++ { ++ br_ip4_multicast_leave_snoopers(br); ++ br_ip6_multicast_leave_snoopers(br); ++@@ -2364,9 +2364,6 @@ static void __br_multicast_open(struct net_bridge *br, ++ ++ void br_multicast_open(struct net_bridge *br) ++ { ++- if (!br->multicast_disabled) ++- br_multicast_join_snoopers(br); ++- ++ __br_multicast_open(br, &br->ip4_own_query); ++ #if IS_ENABLED(CONFIG_IPV6) ++ __br_multicast_open(br, &br->ip6_own_query); ++@@ -2382,9 +2379,6 @@ void br_multicast_stop(struct net_bridge *br) ++ del_timer_sync(&br->ip6_other_query.timer); ++ del_timer_sync(&br->ip6_own_query.timer); ++ #endif ++- ++- if (!br->multicast_disabled) ++- br_multicast_leave_snoopers(br); ++ } ++ ++ void br_multicast_dev_del(struct net_bridge *br) ++@@ -2540,6 +2534,7 @@ int br_multicast_toggle(struct net_bridge *br, unsigned long val) ++ { ++ struct net_bridge_mdb_htable *mdb; ++ struct net_bridge_port *port; +++ bool change_snoopers = false; ++ int err = 0; ++ ++ spin_lock_bh(&br->multicast_lock); ++@@ -2549,7 +2544,7 @@ int br_multicast_toggle(struct net_bridge *br, unsigned long val) ++ br_mc_disabled_update(br->dev, !val); ++ br->multicast_disabled = !val; ++ if (br->multicast_disabled) { ++- br_multicast_leave_snoopers(br); +++ change_snoopers = true; ++ goto unlock; ++ } ++ ++@@ -2575,9 +2570,30 @@ int br_multicast_toggle(struct net_bridge *br, unsigned long val) ++ list_for_each_entry(port, &br->port_list, list) ++ __br_multicast_enable_port(port); ++ +++ change_snoopers = true; +++ ++ unlock: ++ spin_unlock_bh(&br->multicast_lock); ++ +++ /* br_multicast_join_snoopers has the potential to cause +++ * an MLD Report/Leave to be delivered to br_multicast_rcv, +++ * which would in turn call br_multicast_add_group, which would +++ * attempt to acquire multicast_lock. This function should be +++ * called after the lock has been released to avoid deadlocks on +++ * multicast_lock. +++ * +++ * br_multicast_leave_snoopers does not have the problem since +++ * br_multicast_rcv first checks BROPT_MULTICAST_ENABLED, and +++ * returns without calling br_multicast_ipv4/6_rcv if it's not +++ * enabled. Moved both functions out just for symmetry. +++ */ +++ if (change_snoopers) { +++ if (!br->multicast_disabled) +++ br_multicast_join_snoopers(br); +++ else +++ br_multicast_leave_snoopers(br); +++ } +++ ++ return err; ++ } ++ ++diff --git a/net/bridge/br_private.h b/net/bridge/br_private.h ++index bbf7513c1d16..cb701c31199d 100644 ++--- a/net/bridge/br_private.h +++++ b/net/bridge/br_private.h ++@@ -611,6 +611,8 @@ void br_multicast_del_port(struct net_bridge_port *port); ++ void br_multicast_enable_port(struct net_bridge_port *port); ++ void br_multicast_disable_port(struct net_bridge_port *port); ++ void br_multicast_init(struct net_bridge *br); +++void br_multicast_join_snoopers(struct net_bridge *br); +++void br_multicast_leave_snoopers(struct net_bridge *br); ++ void br_multicast_open(struct net_bridge *br); ++ void br_multicast_stop(struct net_bridge *br); ++ void br_multicast_dev_del(struct net_bridge *br); ++@@ -736,6 +738,14 @@ static inline void br_multicast_init(struct net_bridge *br) ++ { ++ } ++ +++static inline void br_multicast_join_snoopers(struct net_bridge *br) +++{ +++} +++ +++static inline void br_multicast_leave_snoopers(struct net_bridge *br) +++{ +++} +++ ++ static inline void br_multicast_open(struct net_bridge *br) ++ { ++ } ++-- ++2.29.2 ++