Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-08-26
21:39:40 openstack Launchpad bug 1841481 in OpenStack Compute (nova) "Race during ironic re-balance corrupts local RT ProviderTree and compute_nodes cache" [Medium,Triaged]
21:42:18 TheJulia funnn..... and by fun, I think I mean anti-fun
21:44:10 openstackgerrit sean mooney proposed openstack/nova master: Libvirt: report storage bus traits https://review.opendev.org/666914
21:44:11 openstackgerrit sean mooney proposed openstack/nova master: Add transform_image_metadata request filter https://review.opendev.org/665775
21:44:11 openstackgerrit sean mooney proposed openstack/nova master: libvirt: use domain capabilities to get supported device models https://review.opendev.org/666915
21:45:06 fungi can anyone double-check for me that bug 1837252 was introduced in os-vif 1.12.0? sean-k-mooney said it "was introduced in stein" but there were multiple releases in master before it branched to stable/stein
21:45:07 openstack bug 1837252 in os-vif stein "IFLA_BR_AGEING_TIME of 0 causes flooding across bridges" [High,In progress] https://launchpad.net/bugs/1837252 - Assigned to sean mooney (sean-k-mooney)
21:45:48 fungi just want to be sure before i go sending this to mitre, so i don't have to update them with errata
21:45:51 sean-k-mooney i think the exact commit is in the bug somewhere but ya ill check
21:45:58 sean-k-mooney it was right at the end of the cycle
21:46:04 fungi oh, sorry if i overlooked it, checking
21:46:54 fungi 5027ce8
21:47:10 sean-k-mooney https://github.com/openstack/os-vif/commit/1f6fed6a69e9fd386e421f3cacae97c11cdd7c75
21:47:27 sean-k-mooney so 1.15.0
21:47:48 fungi aha, thanks
21:48:13 sean-k-mooney thats the commit where i swaped the linux bridge plugin to use the common code intoduced in the previous commit
21:48:44 fungi any chance you can skim the rest of the impact description in comment #17 there and see if it makes sense, aside from the affected versions obviously being an incorrect guess?
21:50:10 sean-k-mooney im not actully sure if it disable mac learning or if mac learning is still enabled but macs are never unlearnt
21:50:28 sean-k-mooney both would result in flooding in different situations
21:52:31 sean-k-mooney form the inital bug i woudl have suspected that former
21:52:54 sean-k-mooney sorry the latter
21:53:05 fungi hrm, yeah at first i thought it simply filled the mac table, but logan- suggested it disabled learning
21:53:09 sean-k-mooney the fdb does have macs
21:53:36 sean-k-mooney maybe ip route has some info
21:53:44 sean-k-mooney or ip tools
21:53:45 fungi hence the diff between the impact descriptions in comment #15 and #17
21:54:46 sean-k-mooney brctl setageing <brname> <time> sets the ethernet (MAC) address ageing time, in seconds. After <time> seconds of not having seen a frame coming from a certain address, the bridge
21:54:48 sean-k-mooney will time out (delete) that address from the Forwarding DataBase (fdb).
21:55:26 sean-k-mooney so i would think it never delete the macs
21:55:37 sean-k-mooney so it would only flood if the mac was learned on two ports
21:55:43 mriedem melwitt: dansmith: if either of you are around can we move this stein backport series through please? https://review.opendev.org/#/q/topic:bug/1839560+branch:stable/stein
21:55:53 sean-k-mooney that would only happen if the vm was migrated
21:56:02 fungi sean-k-mooney: or if the table filled i guess?
21:56:16 sean-k-mooney i dont think there is a limit on the table size
21:56:40 fungi there is bound to be some practical limit on table size, whether it is reachable or not is another matter ;)
21:56:46 sean-k-mooney maybe there is but there the docs on linux bridge are rather limited
21:57:59 dansmith mriedem: hit me tomorrow and I will, I gotta run
21:58:26 fungi not sure what sort of layer 2 filters are generally in place to prevent poisoning, but could an instance spoof frames from random macs to overrun the bridge table? or spoof specific macs for other victim instances so they show up on multiple ports?
21:58:27 dansmith nm, I'll just do it now
21:59:05 sean-k-mooney fungi: neutron adds iptbales rules to prevent mac spoofing and nova used to also
21:59:13 melwitt heh, dansmith wins
21:59:31 fungi sean-k-mooney: yeah, and i think that would be a more general concern beyond the scope of this particular bug anyway
22:00:17 mriedem dansmith: <3
22:00:34 fungi sean-k-mooney: so anyway, if the mac really does need to appear on multiple ports (e.g. because the instance was migrated) then that does significantly reduce the risk
22:01:57 fungi sean-k-mooney: i'll update the bug with a summary of the discussion here and see if anyone contests these assertions
22:02:04 sean-k-mooney reduce yes but not eliminate so its still a concern
22:02:22 sean-k-mooney cool
22:03:06 fungi absolutely
22:03:09 sean-k-mooney the primary concern is really the external netwrok / cross tenant shared networks
22:03:38 fungi but good to capture the exact risk scenario in the impact description and advsory
22:03:40 fungi thanks!
22:03:44 sean-k-mooney security groups are still applied so within a tenatn network its less concerning but still not good
22:03:52 sean-k-mooney no worries
22:10:17 melwitt sean-k-mooney: I just read through your discussion about the os-vif bug, does the bug only affect VMs that are being migrated or can it affect more? sorry I didn't quite get that part
22:12:28 sean-k-mooney melwitt: i originally thought i would only affect vms that were moved as my understand of setting ageing=0 was that all learned mac address would be permement and never age out
22:13:02 sean-k-mooney although in the bug the assertion was made that ageing=0 disable mac learning entirly and casues all packets to flood
22:13:17 sean-k-mooney i am not sure which is correct
22:13:44 sean-k-mooney the scope of the bug is obviosly change signifcantly based on that
22:14:33 melwitt hm, yeah. ok
22:15:55 fungi sean-k-mooney: looking more closely at the example in the bug description, it looks like the macs there are only duplicated between master and vlan1 entries for the same port?
22:16:31 sean-k-mooney that confused me a bit
22:16:38 fungi i don't see the same macs appearing on different port numbers
22:16:50 sean-k-mooney for linxu bridge we shoudl not be adding vlans to the tap devices
22:17:01 fungi so i think it's just vlan0/trunk and vlan1/tagged both being listed?
22:18:00 sean-k-mooney my understnading was the the linux bridge driver applied the vlan to the veth pari that connted the tenant bridge to the br-int
22:18:15 sean-k-mooney so the ports on the tenant network should be untagged
22:19:32 fungi also it's unclear to me whether those entries are marked permanent because they were added through some other mechanism than learning, or whether they're showing up as permanent because their ageing time is 0
22:19:36 sean-k-mooney i can deploy with linux bridge and try to see what happens
22:20:06 sean-k-mooney or perment because the port exist on the bridge
22:20:07 fungi if you get a chance that would be stellar, but if not i'll dig in the docs when i get a moment
22:21:22 sean-k-mooney it looks to me like the only macs in teh fdb are for the local ports
22:21:30 sean-k-mooney implying that the macs are never learned
22:21:38 sean-k-mooney for non local porst
22:22:17 sean-k-mooney nomaly we would expec the eno1.102 port to havee the macs of vms on other host listed
22:23:15 sean-k-mooney if that is the case the desciption in 17 is closer to being corerect as we would flood any packet to a non local port
22:23:53 sean-k-mooney which would mean i need a 2 node deployment to test
22:24:22 fungi aha, so it could be that the problem is traffic addressed to a non-local port is flooded to all ports including the local ports
22:24:27 sean-k-mooney which i guess i can do i jsut need to set it up but i likely wont get to it until tomorow
22:24:37 sean-k-mooney fungi: yep
22:24:43 sean-k-mooney that what im suspecting
22:25:14 fungi that would make more sense in light of the example bridge table and the other suggestions in the ensuing bug discussion
22:25:21 fungi thanks!
22:30:43 sean-k-mooney artom: do you mind if i rebase your series to fix the keystone_auth issues
22:30:54 sean-k-mooney artom: just running the test locally to confirm
22:32:35 sean-k-mooney ill quickly deploy and test it locally then sign off for the night
22:39:03 openstackgerrit sean mooney proposed openstack/nova master: Introduce live_migration_claim() https://review.opendev.org/635669
22:39:03 openstackgerrit sean mooney proposed openstack/nova master: New objects for NUMA live migration https://review.opendev.org/634827
22:39:04 openstackgerrit sean mooney proposed openstack/nova master: LM: add support for augmenting migrate_data with info from claims https://review.opendev.org/634828
22:39:04 openstackgerrit sean mooney proposed openstack/nova master: LM: add support for updating NUMA-related XML on the source https://review.opendev.org/635229
22:39:05 openstackgerrit sean mooney proposed openstack/nova master: Deprecate CONF.workarounds.enable_numa_live_migration https://review.opendev.org/640021
22:39:05 openstackgerrit sean mooney proposed openstack/nova master: NUMA live migration support https://review.opendev.org/634606
22:39:06 openstackgerrit sean mooney proposed openstack/nova master: Functional test for NUMA live migration https://review.opendev.org/672595
23:16:02 sean-k-mooney artom: it looks like your getting closer. did you modify the hugepage code.
23:16:24 sean-k-mooney i think migrtion with cpu pinning is now working but i think the hugepage migration is not
23:17:17 sean-k-mooney i will test it more corretly tomorrwo by force the cpus set to not over lap on each host and actully allocating hugepages rther then forcing small pages with hw:mem_page_size=small
23:36:36 openstackgerrit Merged openstack/nova master: Add test for create server with integer AZ https://review.opendev.org/678328
23:51:27 artom sean-k-mooney, ugh, I had local changes I wanted to push
23:51:48 artom You just rebased them?
23:52:00 artom I guess I'll keep my local branch, rebase, then push
#openstack-nova - 2019-08-27
00:16:06 sean-k-mooney it was an automatic rebase against master

Earlier   Later