| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-02-28 | |||
| 14:21:44 | mdbooth | bauzas: Yeah, it's a word. Pretty sure that's not how you spell it, but I wouldn't have called you on it as I'm not 100% sure myself ;) | |
| 14:23:03 | mriedem | what is the question? | |
| 14:23:12 | mriedem | why doesn't rebuild migrate? | |
| 14:23:21 | mriedem | why do we validate the image against the scheduler when it changes? | |
| 14:23:28 | mriedem | because build is on the same host | |
| 14:23:37 | mriedem | why? because that's the way it originated | |
| 14:23:41 | mdbooth | mriedem: Yeah. Specifically why does it go to length to prevent a change of host even when calling the scheduler? | |
| 14:23:52 | mdbooth | mriedem: Right, that was my first guess. | |
| 14:24:04 | mriedem | then over the years it turned out it was buggy and to maintain the api contract but resolve those bugs we had to build stuff into it | |
| 14:24:11 | mdbooth | So there's no fundamental reason that it shouldn't, it's just that the control flow doesn't support it. | |
| 14:24:25 | mriedem | i wasn't around when rebuild was added | |
| 14:24:26 | sean-k-mooney | mdbooth: it calls the schudler to validate the current host is still valide as changeing the image means the image metadata could invaldate the current host | |
| 14:24:42 | mriedem | fwiw, | |
| 14:25:01 | mriedem | it's also got a latent bug where the image can change numa topology which does not send new numa topology limits to the compute service to do a resource claim | |
| 14:25:03 | sean-k-mooney | mdbooth: you can rebuild instance with sriov | |
| 14:25:18 | mriedem | because rebuild doesn't do any resource claim, because it was assumed that the flavor never changed (because it doesn't in the api) and the scheduler wouldn't be used | |
| 14:25:23 | sean-k-mooney | actully it cold migration ignore that | |
| 14:25:30 | mriedem | so over time we added more features to nova (numa support) which broke existing flows | |
| 14:25:54 | mdbooth | mriedem: Going up a level, I was just considering the shelve/unshelve approach for attach/detach root volume. While that's obviously great if it allows the user to do something which is currently impossible, it still requires shelve/unshelve. I was considering if it could also be added to some other less nasty operation. | |
| 14:26:16 | mriedem | what is nasty about shelve/unshelve? | |
| 14:26:31 | mriedem | i don't see how that's any nastier than trying to make rebuild trigger a migration | |
| 14:26:54 | mriedem | lots of people actually want the stop operation to be a shelve | |
| 14:26:55 | mdbooth | I specifically don't like the way shelve/unshelve abuses the image cache. | |
| 14:27:48 | mdbooth | Probably fixing that would be simpler, I guess. | |
| 14:28:00 | sean-k-mooney | thats specifcally for non bvf instances right. bfv instnace should not mess with the image cache | |
| 14:28:40 | mdbooth | However, if all the user is trying to do is DR an instance (from the spec), going via glance isn't the most efficient solution. | |
| 14:29:22 | openstackgerrit | Jim Rollenhagen proposed openstack/nova master: Remove TypeError handling for get_info https://review.openstack.org/640043 | |
| 14:29:22 | openstackgerrit | Jim Rollenhagen proposed openstack/nova master: Fix typo in vmware get_info docstring https://review.openstack.org/640042 | |
| 14:29:52 | mdbooth | That's a point... in the DR case you'd presumably be evacuating anyway, so you still couldn't evacuate with a different root volume. | |
| 14:30:05 | sean-k-mooney | mdbooth: that depend some of the usescase in the spec impliend at least to me that the remote fail over sigt may not have direct conectivity at the hyperviosr level. e.g. a migrtion to the second sight may now actully work | |
| 14:31:15 | sean-k-mooney | *sigt -> site sight->site | |
| 14:32:03 | mdbooth | Shelve/unshelve is *extremely* space inefficient. You're basically storing a flattened snapshot at the point of shelve in the image cache forever. | |
| 14:32:29 | mdbooth | But not for BFV, I guess. | |
| 14:32:40 | sean-k-mooney | mdbooth: doesnt ath depend on your glance backend | |
| 14:33:35 | mdbooth | sean-k-mooney: No. It might depend on Nova's imagebackend, though. | |
| 14:34:23 | sean-k-mooney | ah right. i was thinking it might be effienet on ceph backed glance but may ceph imagebackend would help | |
| 14:34:40 | sean-k-mooney | e.g. for non bfv instnaces | |
| 14:35:07 | mdbooth | I don't think we're at all clever about shelve of ceph instance to ceph glance. | |
| 14:35:30 | mdbooth | And ceph glance to ceph instance is efficient but problematic. | |
| 14:38:04 | efried | gibi, bauzas: Sorry, I meant to mention it in the change set - feel free to squash. | |
| 14:41:37 | bauzas | efried: thanks | |
| 14:55:34 | artom | mriedem, hey, just to get a feel for what I should concentrate on next, is the absence of func tests for NUMA live migration a blocker? If so, I'll start on that now, otherwise, I'll advance integration testing - and can add func tests after FF? | |
| 14:56:04 | artom | (Assuming it merges, that is - but that's just my carebear optimism ;) | |
| 14:56:38 | mriedem | artom: did you see my comments about the compute service version check in the conductor task? | |
| 14:56:50 | artom | mriedem, did, and addressed in a new patch on top | |
| 14:57:00 | artom | (Figured it's legit to split them for easier reviewing) | |
| 14:57:25 | mriedem | artom: ok i didn't know how your integration testing could have even been working with that code the way it was, unless you were enabling that workaround | |
| 14:57:48 | artom | mriedem, that's... actually a good question | |
| 14:58:05 | artom | Because I sure as hell didn't set enable_numa_live_migration, and its default is False, right? | |
| 14:58:38 | mriedem | correct | |
| 14:58:50 | artom | And yet my integration tests *do* work, because they caught a logic flaw and failed when the pinning of 2 instances ended up overlapping | |
| 14:59:00 | mriedem | were you testing with that patch reverted? | |
| 14:59:03 | sean-k-mooney | there is a bug | |
| 14:59:14 | artom | mriedem, nope :/ | |
| 14:59:17 | sean-k-mooney | in the numa migration config code | |
| 14:59:35 | sean-k-mooney | https://review.openstack.org/#/c/635350 | |
| 14:59:49 | mriedem | you mean this if hypervisor_type != obj_fields.HVType.KVM: | |
| 14:59:50 | sean-k-mooney | mriedem: artom cfriesen found and fixed it | |
| 15:00:16 | sean-k-mooney | yep it will be QEMU not kvm | |
| 15:00:24 | mriedem | right so the blocker config doesn't actually do anything | |
| 15:00:30 | sean-k-mooney | right | |
| 15:00:32 | artom | I swear I see stuff like that and I can't help but think "we're all just morons" | |
| 15:00:41 | artom | Nothing against anyone, and I'm including myself in that group | |
| 15:01:08 | tacco | hi there.. i have a issue with a redeployed hypervisor.. after deployment i see a "ResourceProviderCreationFailed: Failed to create resource provider" error when trying to start nova or migrating a instance to this hypervisor https://pastebin.com/zMHUUYEW | |
| 15:01:10 | artom | At least we found and fixed it before release :) | |
| 15:01:32 | tacco | any idea? | |
| 15:01:35 | sean-k-mooney | its not merged yet. imgoing to recheck the patch. that said if we merger your code this is not needed | |
| 15:01:56 | artom | sean-k-mooney, still needed for anything but full Stein deployments | |
| 15:02:11 | artom | See https://review.openstack.org/#/c/640021/ :) | |
| 15:02:12 | sean-k-mooney | ya that is ture | |
| 15:02:16 | mriedem | artom: it's already been backported to stable/rocky | |
| 15:02:46 | artom | mriedem, the CONF workaround? *sigh* | |
| 15:03:59 | sean-k-mooney | ya so we will have to backport the fix too but sice its currently a noop it wont break anything | |
| 15:04:17 | sean-k-mooney | its annoying but thats all | |
| 15:05:06 | mriedem | artom: yes https://review.openstack.org/#/q/I217fba9138132b107e9d62895d699d238392e761 | |
| 15:05:29 | mriedem | me thinks we should probably start an rc-potential etherpad... | |
| 15:05:34 | sean-k-mooney | cfriesen: do you have time to repsin that by the way? if not ill add https://review.openstack.org/#/c/635350 to my list and ill file a bug | |
| 15:05:38 | mriedem | because there is just too much crap flying around for me to keep in my brain | |
| 15:06:51 | artom | mriedem, you mean rc blocker? | |
| 15:08:05 | mriedem | https://etherpad.openstack.org/p/nova-stein-rc-potential | |
| 15:08:08 | mriedem | melwitt: ^ | |
| 15:08:09 | sean-k-mooney | artom: not so much blocker but might warrent a second rc or shoudl be merged after ff as part of an rc | |
| 15:09:01 | artom | mriedem, so, sorry to pester, but the func tests blocker thing wasn't resolved | |
| 15:09:59 | mriedem | you want closure from me in other words? | |
| 15:10:26 | artom | mriedem, heh, yeah. A sense of priorities, o wise leader :) | |
| 15:12:44 | mriedem | well i personally think it's pretty risky to land something as complicated as live migration with numa without some functional tests, but at the same time, i guess if it turns out to have bugs then they just get fixed and we trust your integration testing downstream | |
| 15:12:56 | cfriesen | sean-k-mooney: will respin | |
| 15:13:12 | mriedem | in general i loathe landing anything related to pci/numa/sriov when we have no 3rd party integration testing for it | |
| 15:13:21 | cfriesen | we'll be doing integration testing for live migration too | |
| 15:13:35 | dansmith | mriedem: I too think it's fairly risky | |
| 15:13:41 | sean-k-mooney | cfriesen: ok it just need a bug added. | |
| 15:13:55 | sean-k-mooney | cfriesen: so it can be backported | |
| 15:14:02 | dansmith | I haven't really had time to go back through it from a holistic "how do these pieces fit together" sort of view | |
| 15:14:04 | artom | mriedem, I know :/ Which is why I want to replace the Intel CI with something that's actually maintained | |
| 15:14:17 | cfriesen | dansmith: mriedem: totally risky. but is it likely to break anything other than things that are already broken? | |
| 15:14:21 | dansmith | breaking live migration at all is a really bad thing to do | |
| 15:14:29 | dansmith | cfriesen: yes? | |
| 15:15:02 | artom | dansmith, basic live migration is tested in the gate though, right? | |
| 15:15:10 | mriedem | yes, | |
| 15:15:19 | dansmith | artom: during several partial upgrade situations? | |