| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-08-26 | |||
| 21:53:44 | sean-k-mooney | or ip tools | |
| 21:53:45 | fungi | hence the diff between the impact descriptions in comment #15 and #17 | |
| 21:54:46 | sean-k-mooney | brctl setageing <brname> <time> sets the ethernet (MAC) address ageing time, in seconds. After <time> seconds of not having seen a frame coming from a certain address, the bridge | |
| 21:54:48 | sean-k-mooney | will time out (delete) that address from the Forwarding DataBase (fdb). | |
| 21:55:26 | sean-k-mooney | so i would think it never delete the macs | |
| 21:55:37 | sean-k-mooney | so it would only flood if the mac was learned on two ports | |
| 21:55:43 | mriedem | melwitt: dansmith: if either of you are around can we move this stein backport series through please? https://review.opendev.org/#/q/topic:bug/1839560+branch:stable/stein | |
| 21:55:53 | sean-k-mooney | that would only happen if the vm was migrated | |
| 21:56:02 | fungi | sean-k-mooney: or if the table filled i guess? | |
| 21:56:16 | sean-k-mooney | i dont think there is a limit on the table size | |
| 21:56:40 | fungi | there is bound to be some practical limit on table size, whether it is reachable or not is another matter ;) | |
| 21:56:46 | sean-k-mooney | maybe there is but there the docs on linux bridge are rather limited | |
| 21:57:59 | dansmith | mriedem: hit me tomorrow and I will, I gotta run | |
| 21:58:26 | fungi | not sure what sort of layer 2 filters are generally in place to prevent poisoning, but could an instance spoof frames from random macs to overrun the bridge table? or spoof specific macs for other victim instances so they show up on multiple ports? | |
| 21:58:27 | dansmith | nm, I'll just do it now | |
| 21:59:05 | sean-k-mooney | fungi: neutron adds iptbales rules to prevent mac spoofing and nova used to also | |
| 21:59:13 | melwitt | heh, dansmith wins | |
| 21:59:31 | fungi | sean-k-mooney: yeah, and i think that would be a more general concern beyond the scope of this particular bug anyway | |
| 22:00:17 | mriedem | dansmith: <3 | |
| 22:00:34 | fungi | sean-k-mooney: so anyway, if the mac really does need to appear on multiple ports (e.g. because the instance was migrated) then that does significantly reduce the risk | |
| 22:01:57 | fungi | sean-k-mooney: i'll update the bug with a summary of the discussion here and see if anyone contests these assertions | |
| 22:02:04 | sean-k-mooney | reduce yes but not eliminate so its still a concern | |
| 22:02:22 | sean-k-mooney | cool | |
| 22:03:06 | fungi | absolutely | |
| 22:03:09 | sean-k-mooney | the primary concern is really the external netwrok / cross tenant shared networks | |
| 22:03:38 | fungi | but good to capture the exact risk scenario in the impact description and advsory | |
| 22:03:40 | fungi | thanks! | |
| 22:03:44 | sean-k-mooney | security groups are still applied so within a tenatn network its less concerning but still not good | |
| 22:03:52 | sean-k-mooney | no worries | |
| 22:10:17 | melwitt | sean-k-mooney: I just read through your discussion about the os-vif bug, does the bug only affect VMs that are being migrated or can it affect more? sorry I didn't quite get that part | |
| 22:12:28 | sean-k-mooney | melwitt: i originally thought i would only affect vms that were moved as my understand of setting ageing=0 was that all learned mac address would be permement and never age out | |
| 22:13:02 | sean-k-mooney | although in the bug the assertion was made that ageing=0 disable mac learning entirly and casues all packets to flood | |
| 22:13:17 | sean-k-mooney | i am not sure which is correct | |
| 22:13:44 | sean-k-mooney | the scope of the bug is obviosly change signifcantly based on that | |
| 22:14:33 | melwitt | hm, yeah. ok | |
| 22:15:55 | fungi | sean-k-mooney: looking more closely at the example in the bug description, it looks like the macs there are only duplicated between master and vlan1 entries for the same port? | |
| 22:16:31 | sean-k-mooney | that confused me a bit | |
| 22:16:38 | fungi | i don't see the same macs appearing on different port numbers | |
| 22:16:50 | sean-k-mooney | for linxu bridge we shoudl not be adding vlans to the tap devices | |
| 22:17:01 | fungi | so i think it's just vlan0/trunk and vlan1/tagged both being listed? | |
| 22:18:00 | sean-k-mooney | my understnading was the the linux bridge driver applied the vlan to the veth pari that connted the tenant bridge to the br-int | |
| 22:18:15 | sean-k-mooney | so the ports on the tenant network should be untagged | |
| 22:19:32 | fungi | also it's unclear to me whether those entries are marked permanent because they were added through some other mechanism than learning, or whether they're showing up as permanent because their ageing time is 0 | |
| 22:19:36 | sean-k-mooney | i can deploy with linux bridge and try to see what happens | |
| 22:20:06 | sean-k-mooney | or perment because the port exist on the bridge | |
| 22:20:07 | fungi | if you get a chance that would be stellar, but if not i'll dig in the docs when i get a moment | |
| 22:21:22 | sean-k-mooney | it looks to me like the only macs in teh fdb are for the local ports | |
| 22:21:30 | sean-k-mooney | implying that the macs are never learned | |
| 22:21:38 | sean-k-mooney | for non local porst | |
| 22:22:17 | sean-k-mooney | nomaly we would expec the eno1.102 port to havee the macs of vms on other host listed | |
| 22:23:15 | sean-k-mooney | if that is the case the desciption in 17 is closer to being corerect as we would flood any packet to a non local port | |
| 22:23:53 | sean-k-mooney | which would mean i need a 2 node deployment to test | |
| 22:24:22 | fungi | aha, so it could be that the problem is traffic addressed to a non-local port is flooded to all ports including the local ports | |
| 22:24:27 | sean-k-mooney | which i guess i can do i jsut need to set it up but i likely wont get to it until tomorow | |
| 22:24:37 | sean-k-mooney | fungi: yep | |
| 22:24:43 | sean-k-mooney | that what im suspecting | |
| 22:25:14 | fungi | that would make more sense in light of the example bridge table and the other suggestions in the ensuing bug discussion | |
| 22:25:21 | fungi | thanks! | |
| 22:30:43 | sean-k-mooney | artom: do you mind if i rebase your series to fix the keystone_auth issues | |
| 22:30:54 | sean-k-mooney | artom: just running the test locally to confirm | |
| 22:32:35 | sean-k-mooney | ill quickly deploy and test it locally then sign off for the night | |
| 22:39:03 | openstackgerrit | sean mooney proposed openstack/nova master: New objects for NUMA live migration https://review.opendev.org/634827 | |
| 22:39:03 | openstackgerrit | sean mooney proposed openstack/nova master: Introduce live_migration_claim() https://review.opendev.org/635669 | |
| 22:39:04 | openstackgerrit | sean mooney proposed openstack/nova master: LM: add support for updating NUMA-related XML on the source https://review.opendev.org/635229 | |
| 22:39:04 | openstackgerrit | sean mooney proposed openstack/nova master: LM: add support for augmenting migrate_data with info from claims https://review.opendev.org/634828 | |
| 22:39:05 | openstackgerrit | sean mooney proposed openstack/nova master: NUMA live migration support https://review.opendev.org/634606 | |
| 22:39:05 | openstackgerrit | sean mooney proposed openstack/nova master: Deprecate CONF.workarounds.enable_numa_live_migration https://review.opendev.org/640021 | |
| 22:39:06 | openstackgerrit | sean mooney proposed openstack/nova master: Functional test for NUMA live migration https://review.opendev.org/672595 | |
| 23:16:02 | sean-k-mooney | artom: it looks like your getting closer. did you modify the hugepage code. | |
| 23:16:24 | sean-k-mooney | i think migrtion with cpu pinning is now working but i think the hugepage migration is not | |
| 23:17:17 | sean-k-mooney | i will test it more corretly tomorrwo by force the cpus set to not over lap on each host and actully allocating hugepages rther then forcing small pages with hw:mem_page_size=small | |
| 23:36:36 | openstackgerrit | Merged openstack/nova master: Add test for create server with integer AZ https://review.opendev.org/678328 | |
| 23:51:27 | artom | sean-k-mooney, ugh, I had local changes I wanted to push | |
| 23:51:48 | artom | You just rebased them? | |
| 23:52:00 | artom | I guess I'll keep my local branch, rebase, then push | |
| #openstack-nova - 2019-08-27 | |||
| 00:16:06 | sean-k-mooney | it was an automatic rebase against master | |
| 00:16:09 | sean-k-mooney | no conflicts | |
| 00:16:24 | sean-k-mooney | so just commit and rebase against master and push | |
| 00:16:30 | sean-k-mooney | artom: ^ | |
| 00:16:40 | artom | sean-k-mooney, ack, thanks | |
| 00:18:41 | sean-k-mooney | also something is broken with hw:mem_page_size=small. i get an error about pagesize changeing when i try to migrate when i set that. | |
| 00:18:58 | sean-k-mooney | not sure if that is related to your changes or not | |
| 00:52:42 | artom | sean-k-mooney, do the 2 hosts have different page sizes? | |
| 00:52:58 | artom | (Or it could be a legit bug with that code, as it's new) | |
| 00:59:30 | artom | Oh snap, rollback works | |
| 00:59:35 | artom | *actually* works | |
| 01:00:01 | artom | If I don't drop the move claim on the dest (comment that bit out), the test fails with the expected pinning assertion | |
| 01:01:16 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: Introduce live_migration_claim() https://review.opendev.org/635669 | |
| 01:01:17 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: New objects for NUMA live migration https://review.opendev.org/634827 | |
| 01:01:17 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: LM: add support for augmenting migrate_data with info from claims https://review.opendev.org/634828 | |
| 01:01:18 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: LM: add support for updating NUMA-related XML on the source https://review.opendev.org/635229 | |
| 01:01:18 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: NUMA live migration support https://review.opendev.org/634606 | |
| 01:01:19 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: Functional tests for NUMA live migration https://review.opendev.org/672595 | |
| 01:01:19 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: Deprecate CONF.workarounds.enable_numa_live_migration https://review.opendev.org/640021 | |
| 01:35:00 | openstackgerrit | Merged openstack/nova master: [Trivial]Remove unused helper get_vif_devname_with_prefix https://review.opendev.org/678136 | |
| 02:30:39 | openstackgerrit | Merged openstack/nova stable/stein: rt: only map compute node if we created it https://review.opendev.org/676278 | |
| 02:30:46 | openstackgerrit | Merged openstack/nova stable/stein: Add functional regression recreate test for bug 1839560 https://review.opendev.org/676507 | |
| 02:30:47 | openstack | bug 1839560 in OpenStack Compute (nova) stein "ironic: moving node to maintenance makes it unusable afterwards" [High,In progress] https://launchpad.net/bugs/1839560 - Assigned to Matt Riedemann (mriedem) | |
| 02:30:53 | openstackgerrit | Merged openstack/nova stable/stein: Restore soft-deleted compute node with same uuid https://review.opendev.org/676509 | |
| 03:47:27 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: NUMA live migration support https://review.opendev.org/634606 | |