| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-02-28 | |||
| 14:59:50 | sean-k-mooney | mriedem: artom cfriesen found and fixed it | |
| 15:00:16 | sean-k-mooney | yep it will be QEMU not kvm | |
| 15:00:24 | mriedem | right so the blocker config doesn't actually do anything | |
| 15:00:30 | sean-k-mooney | right | |
| 15:00:32 | artom | I swear I see stuff like that and I can't help but think "we're all just morons" | |
| 15:00:41 | artom | Nothing against anyone, and I'm including myself in that group | |
| 15:01:08 | tacco | hi there.. i have a issue with a redeployed hypervisor.. after deployment i see a "ResourceProviderCreationFailed: Failed to create resource provider" error when trying to start nova or migrating a instance to this hypervisor https://pastebin.com/zMHUUYEW | |
| 15:01:10 | artom | At least we found and fixed it before release :) | |
| 15:01:32 | tacco | any idea? | |
| 15:01:35 | sean-k-mooney | its not merged yet. imgoing to recheck the patch. that said if we merger your code this is not needed | |
| 15:01:56 | artom | sean-k-mooney, still needed for anything but full Stein deployments | |
| 15:02:11 | artom | See https://review.openstack.org/#/c/640021/ :) | |
| 15:02:12 | sean-k-mooney | ya that is ture | |
| 15:02:16 | mriedem | artom: it's already been backported to stable/rocky | |
| 15:02:46 | artom | mriedem, the CONF workaround? *sigh* | |
| 15:03:59 | sean-k-mooney | ya so we will have to backport the fix too but sice its currently a noop it wont break anything | |
| 15:04:17 | sean-k-mooney | its annoying but thats all | |
| 15:05:06 | mriedem | artom: yes https://review.openstack.org/#/q/I217fba9138132b107e9d62895d699d238392e761 | |
| 15:05:29 | mriedem | me thinks we should probably start an rc-potential etherpad... | |
| 15:05:34 | sean-k-mooney | cfriesen: do you have time to repsin that by the way? if not ill add https://review.openstack.org/#/c/635350 to my list and ill file a bug | |
| 15:05:38 | mriedem | because there is just too much crap flying around for me to keep in my brain | |
| 15:06:51 | artom | mriedem, you mean rc blocker? | |
| 15:08:05 | mriedem | https://etherpad.openstack.org/p/nova-stein-rc-potential | |
| 15:08:08 | mriedem | melwitt: ^ | |
| 15:08:09 | sean-k-mooney | artom: not so much blocker but might warrent a second rc or shoudl be merged after ff as part of an rc | |
| 15:09:01 | artom | mriedem, so, sorry to pester, but the func tests blocker thing wasn't resolved | |
| 15:09:59 | mriedem | you want closure from me in other words? | |
| 15:10:26 | artom | mriedem, heh, yeah. A sense of priorities, o wise leader :) | |
| 15:12:44 | mriedem | well i personally think it's pretty risky to land something as complicated as live migration with numa without some functional tests, but at the same time, i guess if it turns out to have bugs then they just get fixed and we trust your integration testing downstream | |
| 15:12:56 | cfriesen | sean-k-mooney: will respin | |
| 15:13:12 | mriedem | in general i loathe landing anything related to pci/numa/sriov when we have no 3rd party integration testing for it | |
| 15:13:21 | cfriesen | we'll be doing integration testing for live migration too | |
| 15:13:35 | dansmith | mriedem: I too think it's fairly risky | |
| 15:13:41 | sean-k-mooney | cfriesen: ok it just need a bug added. | |
| 15:13:55 | sean-k-mooney | cfriesen: so it can be backported | |
| 15:14:02 | dansmith | I haven't really had time to go back through it from a holistic "how do these pieces fit together" sort of view | |
| 15:14:04 | artom | mriedem, I know :/ Which is why I want to replace the Intel CI with something that's actually maintained | |
| 15:14:17 | cfriesen | dansmith: mriedem: totally risky. but is it likely to break anything other than things that are already broken? | |
| 15:14:21 | dansmith | breaking live migration at all is a really bad thing to do | |
| 15:14:29 | dansmith | cfriesen: yes? | |
| 15:15:02 | artom | dansmith, basic live migration is tested in the gate though, right? | |
| 15:15:10 | mriedem | yes, | |
| 15:15:19 | dansmith | artom: during several partial upgrade situations? | |
| 15:15:31 | mriedem | so if worst case scenario you break live migration with numa instances, well, those were already broken anyway - and if you're using that config (once fixed) you can't even initiate it | |
| 15:15:34 | artom | No :( | |
| 15:15:40 | mriedem | dansmith: yes | |
| 15:15:44 | artom | Yes? | |
| 15:15:46 | mriedem | the live migration + grenade job | |
| 15:15:48 | dansmith | mriedem: we have one right? | |
| 15:15:50 | artom | \o/ | |
| 15:15:52 | mriedem | runs n-1 live migration back and forth | |
| 15:15:54 | mriedem | it's non-voting | |
| 15:16:13 | artom | We should *really* check its output on the NUMA LM patches then | |
| 15:16:15 | dansmith | mriedem: but not on systems that will report this stuff, even for non-numa flavors I think | |
| 15:16:24 | mriedem | https://review.openstack.org/#/c/637231/ | |
| 15:16:48 | mriedem | dansmith: correct we don't do any numa stuff in the gate | |
| 15:17:03 | sean-k-mooney | artom: i would start on the functional tests and we can add the whitebox tempest test after ff | |
| 15:17:04 | dansmith | it's not just numa-having instances, | |
| 15:17:33 | dansmith | it's that everyone's systems report numa stuff, and if any of this works on a numaless guest instance for CI but is weird on a real system... we won't have coverage of that | |
| 15:17:34 | mriedem | mrhillsman: would it be possible to use openlab for numa testing for nova? | |
| 15:17:37 | sean-k-mooney | artom: wether they are a blocker on not is seperate form if we should write them an i think we would all like to see functional tests for this eventually | |
| 15:18:41 | artom | sean-k-mooney, oh yeah, they'll be there | |
| 15:19:09 | dansmith | also, some amount of live migration of numa-having instances works today but just doesn't claim or do the right thing, right? | |
| 15:19:14 | artom | I'm wondering more "what gives others better confidence in the code to help it merge, automated integration tests that you have to run manually in your own env, or func tests?" | |
| 15:19:29 | dansmith | if we were to break that in a way that doesn't let people move their instances in an emergency, even if the numaness gets messed up, that's still a big problem I think | |
| 15:19:55 | mrhillsman | mriedem yes sir | |
| 15:20:28 | sean-k-mooney | dansmith: if the cpus are free on the destination then today everything works for pinned instance | |
| 15:20:37 | sean-k-mooney | similarly for hugepage instnace | |
| 15:20:39 | dansmith | right | |
| 15:21:04 | sean-k-mooney | the clamims get updated too i think | |
| 15:21:11 | dansmith | so if we break that in this because we didn't test it on real systems... that's not "the situation can't get worse" to me | |
| 15:21:25 | sean-k-mooney | they are either healed by the periodic task or done explicitly i cant remember | |
| 15:22:09 | mrhillsman | so long as we can provide what is needed. filling out https://github.com/theopenlab/openlab/issues/new?template=test-request.md with what is needed is required | |
| 15:22:47 | artom | sean-k-mooney, periodic task | |
| 15:24:26 | mriedem | mrhillsman: ah cool ok, i'll put this in our ptg agenda for discussion to see what minimum levels of testing we'd hope to get | |
| 15:24:27 | sean-k-mooney | mrhillsman: do numa testing we would need either nested virt on a host with at least 2 numa nodes or a dual numa beremetal host | |
| 15:24:32 | mriedem | mrhillsman: btw, who writes the tests? or is tempest used? | |
| 15:24:38 | cfriesen | sean-k-mooney: mriedem: hypervisor check bug is https://bugs.launchpad.net/nova/+bug/1818092 | |
| 15:24:39 | openstack | Launchpad bug 1818092 in OpenStack Compute (nova) "hypervisor check in _check_instance_has_no_numa() is broken" [Undecided,New] | |
| 15:25:31 | sean-k-mooney | cfriesen: cool can you add "Close-Bug: 1818092" to the commit in your patch | |
| 15:25:41 | cfriesen | yup, doing it now | |
| 15:25:46 | sean-k-mooney | awsome | |
| 15:25:52 | mrhillsman | mriedem you or some volunteers would | |
| 15:26:23 | artom | Unless it's like, in Perl or JS or something. | |
| 15:26:23 | sean-k-mooney | mriedem: we hope to move the redhat whitebox tempest plugin to openstack namespace | |
| 15:26:31 | mrhillsman | there is a lot of flexibility in how that works | |
| 15:27:36 | mrhillsman | i think adding to the ptg is good if it is not a priority right now | |
| 15:27:46 | mriedem | yup done L77 https://etherpad.openstack.org/p/nova-ptg-train | |
| 15:28:04 | mrhillsman | cool, i will be sure to be there | |
| 15:28:09 | mriedem | maybe chenrui's team can help a bit, i.e. Kevin_Zheng since he is familiar with nova | |
| 15:28:15 | mrhillsman | ++ | |
| 15:30:07 | openstackgerrit | Chris Friesen proposed openstack/nova master: fix up numa-topology live migration hypervisor check https://review.openstack.org/635350 | |
| 15:31:51 | cfriesen | sean-k-mooney: mriedem: there you go ^ | |
| 15:34:48 | mriedem | stephenfin: i saw your patch to deprecate config_drive_format for vfat | |
| 15:34:53 | mriedem | stephenfin: and it reminded me of this https://review.openstack.org/#/c/567860/ | |
| 15:35:12 | cfriesen | anyone know why requirements-check is failing for https://review.openstack.org/#/c/631363 ? It says "Package 'os-traits' is constrained to 0.4.0 which does not match the minimum version specifier 0.11.0 in requirements.txt" | |
| 15:35:17 | mriedem | stephenfin: i'm not 100% but i don't think we actually test live migration in the gate with a config drive on the instance | |
| 15:35:21 | bauzas | mriedem: yeah we agreed on following the deprecation path this morning | |
| 15:35:26 | mriedem | stephenfin: it would be good to have that wrinkle somewhere | |
| 15:35:33 | mriedem | bauzas: sure i understand, but that's not my point | |