| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-02-28 | |||
| 14:00:51 | mnaser | thanks a lot to the kata containers folks | |
| 14:00:57 | sean-k-mooney | there is a bug in the ubunut 18.04 kernel that breaks nested virt | |
| 14:01:18 | sean-k-mooney | i had to replace mine with the mainline 4.20 kernel to fix issues | |
| 14:01:19 | mnaser | yeah our ci is worked out pretty often both by magnum and kata and their nested virt has been super stable | |
| 14:02:46 | sean-k-mooney | ya i went with ubuntu 18.04 because kolla-ansibel source deploys work better on ubuntu then centos in my previos experience | |
| 14:03:02 | sean-k-mooney | mnaser: do ye use openstack ansible for your production installation | |
| 14:03:07 | mnaser | sean-k-mooney: yeah | |
| 14:03:56 | sean-k-mooney | i must get around to checking that out at some point but since its working im not going to touch it untill i need to upgrade :) | |
| 14:10:04 | mdbooth | bauzas: Do you recall why rebuild is forced to be on the same host? Is it just because the way rebuild is written doesn't allow for a migration, or is it fundamental to the intent of rebuild? Or some other reason? | |
| 14:12:10 | sean-k-mooney | mdbooth: why would a rebuild ever need to migrate if the image does not change | |
| 14:12:21 | sean-k-mooney | if the image does change we hit the schulder | |
| 14:12:25 | mdbooth | bauzas: I ask because I was looking at the attach/detach root volume patches. They look fine, but shelve/unshelve is just distasteful, so having it as the only available option seems undesirable. Reading the original spec, it sounds to me like they want to rebuild on a different host... | |
| 14:12:30 | mdbooth | Which is an evacuate# | |
| 14:12:42 | mdbooth | Evacuate with a different root volume. | |
| 14:12:43 | mdbooth | Hmm | |
| 14:13:06 | mnaser | sean-k-mooney: be happy to help you get setup on OSA | |
| 14:13:19 | mdbooth | sean-k-mooney: Yes, we hit the scheduler but constrain it to return the original host or return NoValidHost | |
| 14:14:01 | mdbooth | Rebuild on a different host with a different root volume... | |
| 14:14:02 | sean-k-mooney | mnaser: its more just i have not taken the time to try it out. i learn how kolla ansible worked a few years ago and it served well enough for my personal needs | |
| 14:14:20 | sean-k-mooney | mdbooth: ah ok | |
| 14:14:30 | bauzas | mdbooth: because it doesn't ask for other resources, just for an image | |
| 14:14:43 | bauzas | unless you evacuate :) | |
| 14:15:05 | mdbooth | bauzas: Given that evacuate exists, I'm guessing most of this is already wired up. | |
| 14:15:55 | bauzas | yeah, that's actually the change in between evacuate and rebuild | |
| 14:16:09 | mdbooth | In fact: cold migrate with a different root volume. | |
| 14:16:10 | bauzas | evacuate is just the same rebuild path but with a different API action :) | |
| 14:16:18 | bauzas | hum no | |
| 14:16:23 | bauzas | cold migrate != evacuate | |
| 14:16:32 | bauzas | evacuate == rebuild(different_host) | |
| 14:16:47 | mdbooth | bauzas: Hmm, no. It doesn't consider a change of image. | |
| 14:17:14 | mdbooth | But I was thinking that you might have a root volume and ephemeral disks, in which case you might want to migrate them. | |
| 14:19:38 | mdbooth | bauzas: So, going back to the original question, why do we go to lengths to ensure that rebuild doesn't change host? | |
| 14:20:04 | bauzas | mdbooth: I'm not the rebuild specialist | |
| 14:20:30 | bauzas | some others hitted so many times issues with rebuild that they're more knowledgable than me, like... mriedem ? :) | |
| 14:20:38 | mdbooth | bauzas: Ah, sorry. I pinged you because you were co-author in a patch which added this constraint. | |
| 14:20:47 | mdbooth | mriedem was the other author. | |
| 14:20:52 | bauzas | knowledgable * (not sure it's a word) | |
| 14:21:44 | mdbooth | bauzas: Yeah, it's a word. Pretty sure that's not how you spell it, but I wouldn't have called you on it as I'm not 100% sure myself ;) | |
| 14:23:03 | mriedem | what is the question? | |
| 14:23:12 | mriedem | why doesn't rebuild migrate? | |
| 14:23:21 | mriedem | why do we validate the image against the scheduler when it changes? | |
| 14:23:28 | mriedem | because build is on the same host | |
| 14:23:37 | mriedem | why? because that's the way it originated | |
| 14:23:41 | mdbooth | mriedem: Yeah. Specifically why does it go to length to prevent a change of host even when calling the scheduler? | |
| 14:23:52 | mdbooth | mriedem: Right, that was my first guess. | |
| 14:24:04 | mriedem | then over the years it turned out it was buggy and to maintain the api contract but resolve those bugs we had to build stuff into it | |
| 14:24:11 | mdbooth | So there's no fundamental reason that it shouldn't, it's just that the control flow doesn't support it. | |
| 14:24:25 | mriedem | i wasn't around when rebuild was added | |
| 14:24:26 | sean-k-mooney | mdbooth: it calls the schudler to validate the current host is still valide as changeing the image means the image metadata could invaldate the current host | |
| 14:24:42 | mriedem | fwiw, | |
| 14:25:01 | mriedem | it's also got a latent bug where the image can change numa topology which does not send new numa topology limits to the compute service to do a resource claim | |
| 14:25:03 | sean-k-mooney | mdbooth: you can rebuild instance with sriov | |
| 14:25:18 | mriedem | because rebuild doesn't do any resource claim, because it was assumed that the flavor never changed (because it doesn't in the api) and the scheduler wouldn't be used | |
| 14:25:23 | sean-k-mooney | actully it cold migration ignore that | |
| 14:25:30 | mriedem | so over time we added more features to nova (numa support) which broke existing flows | |
| 14:25:54 | mdbooth | mriedem: Going up a level, I was just considering the shelve/unshelve approach for attach/detach root volume. While that's obviously great if it allows the user to do something which is currently impossible, it still requires shelve/unshelve. I was considering if it could also be added to some other less nasty operation. | |
| 14:26:16 | mriedem | what is nasty about shelve/unshelve? | |
| 14:26:31 | mriedem | i don't see how that's any nastier than trying to make rebuild trigger a migration | |
| 14:26:54 | mriedem | lots of people actually want the stop operation to be a shelve | |
| 14:26:55 | mdbooth | I specifically don't like the way shelve/unshelve abuses the image cache. | |
| 14:27:48 | mdbooth | Probably fixing that would be simpler, I guess. | |
| 14:28:00 | sean-k-mooney | thats specifcally for non bvf instances right. bfv instnace should not mess with the image cache | |
| 14:28:40 | mdbooth | However, if all the user is trying to do is DR an instance (from the spec), going via glance isn't the most efficient solution. | |
| 14:29:22 | openstackgerrit | Jim Rollenhagen proposed openstack/nova master: Remove TypeError handling for get_info https://review.openstack.org/640043 | |
| 14:29:22 | openstackgerrit | Jim Rollenhagen proposed openstack/nova master: Fix typo in vmware get_info docstring https://review.openstack.org/640042 | |
| 14:29:52 | mdbooth | That's a point... in the DR case you'd presumably be evacuating anyway, so you still couldn't evacuate with a different root volume. | |
| 14:30:05 | sean-k-mooney | mdbooth: that depend some of the usescase in the spec impliend at least to me that the remote fail over sigt may not have direct conectivity at the hyperviosr level. e.g. a migrtion to the second sight may now actully work | |
| 14:31:15 | sean-k-mooney | *sigt -> site sight->site | |
| 14:32:03 | mdbooth | Shelve/unshelve is *extremely* space inefficient. You're basically storing a flattened snapshot at the point of shelve in the image cache forever. | |
| 14:32:29 | mdbooth | But not for BFV, I guess. | |
| 14:32:40 | sean-k-mooney | mdbooth: doesnt ath depend on your glance backend | |
| 14:33:35 | mdbooth | sean-k-mooney: No. It might depend on Nova's imagebackend, though. | |
| 14:34:23 | sean-k-mooney | ah right. i was thinking it might be effienet on ceph backed glance but may ceph imagebackend would help | |
| 14:34:40 | sean-k-mooney | e.g. for non bfv instnaces | |
| 14:35:07 | mdbooth | I don't think we're at all clever about shelve of ceph instance to ceph glance. | |
| 14:35:30 | mdbooth | And ceph glance to ceph instance is efficient but problematic. | |
| 14:38:04 | efried | gibi, bauzas: Sorry, I meant to mention it in the change set - feel free to squash. | |
| 14:41:37 | bauzas | efried: thanks | |
| 14:55:34 | artom | mriedem, hey, just to get a feel for what I should concentrate on next, is the absence of func tests for NUMA live migration a blocker? If so, I'll start on that now, otherwise, I'll advance integration testing - and can add func tests after FF? | |
| 14:56:04 | artom | (Assuming it merges, that is - but that's just my carebear optimism ;) | |
| 14:56:38 | mriedem | artom: did you see my comments about the compute service version check in the conductor task? | |
| 14:56:50 | artom | mriedem, did, and addressed in a new patch on top | |
| 14:57:00 | artom | (Figured it's legit to split them for easier reviewing) | |
| 14:57:25 | mriedem | artom: ok i didn't know how your integration testing could have even been working with that code the way it was, unless you were enabling that workaround | |
| 14:57:48 | artom | mriedem, that's... actually a good question | |
| 14:58:05 | artom | Because I sure as hell didn't set enable_numa_live_migration, and its default is False, right? | |
| 14:58:38 | mriedem | correct | |
| 14:58:50 | artom | And yet my integration tests *do* work, because they caught a logic flaw and failed when the pinning of 2 instances ended up overlapping | |
| 14:59:00 | mriedem | were you testing with that patch reverted? | |
| 14:59:03 | sean-k-mooney | there is a bug | |
| 14:59:14 | artom | mriedem, nope :/ | |
| 14:59:17 | sean-k-mooney | in the numa migration config code | |
| 14:59:35 | sean-k-mooney | https://review.openstack.org/#/c/635350 | |
| 14:59:49 | mriedem | you mean this if hypervisor_type != obj_fields.HVType.KVM: | |
| 14:59:50 | sean-k-mooney | mriedem: artom cfriesen found and fixed it | |
| 15:00:16 | sean-k-mooney | yep it will be QEMU not kvm | |
| 15:00:24 | mriedem | right so the blocker config doesn't actually do anything | |
| 15:00:30 | sean-k-mooney | right | |
| 15:00:32 | artom | I swear I see stuff like that and I can't help but think "we're all just morons" | |
| 15:00:41 | artom | Nothing against anyone, and I'm including myself in that group | |