Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-08-26
21:13:07 mriedem ^ is the other host
21:13:26 mriedem yup, 18:41:38
21:14:02 mriedem so the new host starts refreshing it's RT, the old host deletes the provider, then the new host blows up in the RT, but eventually corrects itself on the next run
21:14:22 mriedem dId SoMeOnE sAy EtCd?!?!
21:14:57 sean-k-mooney ... and follow cinders lead
21:15:02 mriedem i'm joking
21:15:24 sean-k-mooney i know
21:16:25 sean-k-mooney a distribute lock could prevent this by deadlocking the sytem so we dont get there
21:17:26 sean-k-mooney we proably need to cacht the error and if it happens remove the compute node form whatever data structure we were iterating over and move on
21:18:39 mriedem i think the bug is that we stash the node on the new host in RT.compute_nodes, then hit the failure, and the next pass in the RT periodic task we see the node is already in the RT.compute_nodes and don't attempt to go through the _update and flush to placement flow, so the new host doesn't recreate the provider until inventory changes
21:18:56 mriedem so find the thing on the new host and store it here https://github.com/openstack/nova/blob/71478c3eedd95e2eeb219f47460603221ee249b9/nova/compute/resource_tracker.py#L513
21:19:01 mriedem blow up here https://github.com/openstack/nova/blob/71478c3eedd95e2eeb219f47460603221ee249b9/nova/compute/resource_tracker.py#L516
21:19:12 mriedem and on the next pass find the node here https://github.com/openstack/nova/blob/71478c3eedd95e2eeb219f47460603221ee249b9/nova/compute/resource_tracker.py#L546
21:19:16 mriedem but don't call _update
21:19:18 mriedem until inventory changes
21:19:23 mriedem hence the RP not showing up again for awhile
21:19:25 mriedem TheJulia: ^
21:19:49 mriedem a la https://review.opendev.org/#/q/I9fa1d509a3de405d6246fb8670612c65c10cc93b
21:22:40 mriedem oh actually,
21:22:56 mriedem because we put the provider back into the cache and then failed, we don't attempt to create it later in the new host either
21:23:08 mriedem so the provider tree cache is hiding the fact the provider doesn't actually exist anymore
21:24:28 mriedem efried: fun stuff ^
21:25:36 mriedem guess i'll just report a bug for now
21:27:17 efried mriedem: Under what circumstances do we put a provider into the cache and then fail?
21:28:05 efried is this during update_from_provider_tree somehow?
21:28:43 mriedem yeah, it will probably be more clear when i post the bug report
21:39:39 mriedem efried: TheJulia: https://bugs.launchpad.net/nova/+bug/1841481
21:39:40 openstack Launchpad bug 1841481 in OpenStack Compute (nova) "Race during ironic re-balance corrupts local RT ProviderTree and compute_nodes cache" [Medium,Triaged]
21:42:18 TheJulia funnn..... and by fun, I think I mean anti-fun
21:44:10 openstackgerrit sean mooney proposed openstack/nova master: Libvirt: report storage bus traits https://review.opendev.org/666914
21:44:11 openstackgerrit sean mooney proposed openstack/nova master: Add transform_image_metadata request filter https://review.opendev.org/665775
21:44:11 openstackgerrit sean mooney proposed openstack/nova master: libvirt: use domain capabilities to get supported device models https://review.opendev.org/666915
21:45:06 fungi can anyone double-check for me that bug 1837252 was introduced in os-vif 1.12.0? sean-k-mooney said it "was introduced in stein" but there were multiple releases in master before it branched to stable/stein
21:45:07 openstack bug 1837252 in os-vif stein "IFLA_BR_AGEING_TIME of 0 causes flooding across bridges" [High,In progress] https://launchpad.net/bugs/1837252 - Assigned to sean mooney (sean-k-mooney)
21:45:48 fungi just want to be sure before i go sending this to mitre, so i don't have to update them with errata
21:45:51 sean-k-mooney i think the exact commit is in the bug somewhere but ya ill check
21:45:58 sean-k-mooney it was right at the end of the cycle
21:46:04 fungi oh, sorry if i overlooked it, checking
21:46:54 fungi 5027ce8
21:47:10 sean-k-mooney https://github.com/openstack/os-vif/commit/1f6fed6a69e9fd386e421f3cacae97c11cdd7c75
21:47:27 sean-k-mooney so 1.15.0
21:47:48 fungi aha, thanks
21:48:13 sean-k-mooney thats the commit where i swaped the linux bridge plugin to use the common code intoduced in the previous commit
21:48:44 fungi any chance you can skim the rest of the impact description in comment #17 there and see if it makes sense, aside from the affected versions obviously being an incorrect guess?
21:50:10 sean-k-mooney im not actully sure if it disable mac learning or if mac learning is still enabled but macs are never unlearnt
21:50:28 sean-k-mooney both would result in flooding in different situations
21:52:31 sean-k-mooney form the inital bug i woudl have suspected that former
21:52:54 sean-k-mooney sorry the latter
21:53:05 fungi hrm, yeah at first i thought it simply filled the mac table, but logan- suggested it disabled learning
21:53:09 sean-k-mooney the fdb does have macs
21:53:36 sean-k-mooney maybe ip route has some info
21:53:44 sean-k-mooney or ip tools
21:53:45 fungi hence the diff between the impact descriptions in comment #15 and #17
21:54:46 sean-k-mooney brctl setageing <brname> <time> sets the ethernet (MAC) address ageing time, in seconds. After <time> seconds of not having seen a frame coming from a certain address, the bridge
21:54:48 sean-k-mooney will time out (delete) that address from the Forwarding DataBase (fdb).
21:55:26 sean-k-mooney so i would think it never delete the macs
21:55:37 sean-k-mooney so it would only flood if the mac was learned on two ports
21:55:43 mriedem melwitt: dansmith: if either of you are around can we move this stein backport series through please? https://review.opendev.org/#/q/topic:bug/1839560+branch:stable/stein
21:55:53 sean-k-mooney that would only happen if the vm was migrated
21:56:02 fungi sean-k-mooney: or if the table filled i guess?
21:56:16 sean-k-mooney i dont think there is a limit on the table size
21:56:40 fungi there is bound to be some practical limit on table size, whether it is reachable or not is another matter ;)
21:56:46 sean-k-mooney maybe there is but there the docs on linux bridge are rather limited
21:57:59 dansmith mriedem: hit me tomorrow and I will, I gotta run
21:58:26 fungi not sure what sort of layer 2 filters are generally in place to prevent poisoning, but could an instance spoof frames from random macs to overrun the bridge table? or spoof specific macs for other victim instances so they show up on multiple ports?
21:58:27 dansmith nm, I'll just do it now
21:59:05 sean-k-mooney fungi: neutron adds iptbales rules to prevent mac spoofing and nova used to also
21:59:13 melwitt heh, dansmith wins
21:59:31 fungi sean-k-mooney: yeah, and i think that would be a more general concern beyond the scope of this particular bug anyway
22:00:17 mriedem dansmith: <3
22:00:34 fungi sean-k-mooney: so anyway, if the mac really does need to appear on multiple ports (e.g. because the instance was migrated) then that does significantly reduce the risk
22:01:57 fungi sean-k-mooney: i'll update the bug with a summary of the discussion here and see if anyone contests these assertions
22:02:04 sean-k-mooney reduce yes but not eliminate so its still a concern
22:02:22 sean-k-mooney cool
22:03:06 fungi absolutely
22:03:09 sean-k-mooney the primary concern is really the external netwrok / cross tenant shared networks
22:03:38 fungi but good to capture the exact risk scenario in the impact description and advsory
22:03:40 fungi thanks!
22:03:44 sean-k-mooney security groups are still applied so within a tenatn network its less concerning but still not good
22:03:52 sean-k-mooney no worries
22:10:17 melwitt sean-k-mooney: I just read through your discussion about the os-vif bug, does the bug only affect VMs that are being migrated or can it affect more? sorry I didn't quite get that part
22:12:28 sean-k-mooney melwitt: i originally thought i would only affect vms that were moved as my understand of setting ageing=0 was that all learned mac address would be permement and never age out
22:13:02 sean-k-mooney although in the bug the assertion was made that ageing=0 disable mac learning entirly and casues all packets to flood
22:13:17 sean-k-mooney i am not sure which is correct
22:13:44 sean-k-mooney the scope of the bug is obviosly change signifcantly based on that
22:14:33 melwitt hm, yeah. ok
22:15:55 fungi sean-k-mooney: looking more closely at the example in the bug description, it looks like the macs there are only duplicated between master and vlan1 entries for the same port?
22:16:31 sean-k-mooney that confused me a bit
22:16:38 fungi i don't see the same macs appearing on different port numbers
22:16:50 sean-k-mooney for linxu bridge we shoudl not be adding vlans to the tap devices
22:17:01 fungi so i think it's just vlan0/trunk and vlan1/tagged both being listed?
22:18:00 sean-k-mooney my understnading was the the linux bridge driver applied the vlan to the veth pari that connted the tenant bridge to the br-int
22:18:15 sean-k-mooney so the ports on the tenant network should be untagged
22:19:32 fungi also it's unclear to me whether those entries are marked permanent because they were added through some other mechanism than learning, or whether they're showing up as permanent because their ageing time is 0
22:19:36 sean-k-mooney i can deploy with linux bridge and try to see what happens
22:20:06 sean-k-mooney or perment because the port exist on the bridge
22:20:07 fungi if you get a chance that would be stellar, but if not i'll dig in the docs when i get a moment
22:21:22 sean-k-mooney it looks to me like the only macs in teh fdb are for the local ports
22:21:30 sean-k-mooney implying that the macs are never learned
22:21:38 sean-k-mooney for non local porst

Earlier   Later