| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-08-27 | |||
| 14:07:28 | dansmith | I'm saying the thing you're using does not make any sense | |
| 14:07:34 | mriedem | piling on | |
| 14:08:06 | artom | dansmith, and I agree :) It's a good point, I'll change it to a boolean sentinel, and the dest can pull numa_topology from the instance object it already has | |
| 14:08:07 | mriedem | can't the dest check the source compute service version like we did for file-backed memory live migration? https://github.com/openstack/nova/blob/18.0.0/nova/virt/libvirt/driver.py#L6612 | |
| 14:08:17 | mriedem | do we need the boolean sentinel at all? | |
| 14:08:25 | sean-k-mooney | artom: we do this https://github.com/openstack/nova/blob/master/nova/conductor/tasks/live_migrate.py#L35-L58 | |
| 14:08:33 | dansmith | mriedem: we have a boolean for file-backed in that same object | |
| 14:08:39 | artom | mriedem, we need *something* for both computes to agree that they can do NUMA-LM | |
| 14:09:22 | sean-k-mooney | i would emulate https://github.com/openstack/nova/blob/master/nova/conductor/tasks/live_migrate.py#L203-L237 | |
| 14:09:28 | mriedem | so we do https://github.com/openstack/nova/blob/18.0.0/nova/objects/migrate_data.py#L237 | |
| 14:10:41 | mriedem | which is used when generating the xml on the source to pass to the dest https://github.com/openstack/nova/blob/46a3bcd80b41e99ec4923c7cf3d0f8dd8505e97c/nova/virt/libvirt/migration.py#L269 | |
| 14:12:36 | bauzas | just catching the discussion, but why can't we assume checking the RPC versions ? | |
| 14:12:45 | artom | Thinking about it, I believe the service version check wasn't accepted because we still want to allow operators to shoot themselves in the foot, so to speak | |
| 14:12:53 | bauzas | like we do for a couple of RPC intercommunication ? | |
| 14:13:05 | dansmith | bauzas: you can't see the client version from the server side in rpc | |
| 14:13:06 | sean-k-mooney | artom: we can still do it and but not make it fail | |
| 14:13:11 | artom | So we don't just block the LM entirely if one of the hosts is too old | |
| 14:13:17 | dansmith | there's still no reason to even be talking about this | |
| 14:13:23 | dansmith | all we need is to pass a boolean and we're done | |
| 14:13:30 | artom | Yeah, I'm with dansmith | |
| 14:13:36 | artom | Seems way more elegant | |
| 14:13:41 | bauzas | cool, we did that with scheduler, IIRC | |
| 14:13:50 | dansmith | conductor will strip it out if you're talking to an old compute automatically | |
| 14:13:53 | bauzas | we had a sentinel | |
| 14:14:07 | sean-k-mooney | well we can sett the bool in the conductor | |
| 14:14:22 | dansmith | no | |
| 14:14:31 | dansmith | it's migrate_data, set it in the compute | |
| 14:15:00 | dansmith | again, the only reason we're even talking about this, is artom was sending a whole topology object instead of saying "yes". | |
| 14:15:07 | sean-k-mooney | you could. we need to tell the dest if the souce supprot the feature | |
| 14:15:15 | sean-k-mooney | does that line up wiht the calls | |
| 14:15:29 | dansmith | he's already sending it from the compute at the right time | |
| 14:15:35 | dansmith | he's just saying too much. much like this discussion. | |
| 14:15:40 | artom | sean-k-mooney, I'll be blunt and pull rank (though not my own): in case two things work the same dansmith's opinion is worth more than yours :P And I say that with massive admiration for the amount of information you keep in your head | |
| 14:15:40 | sean-k-mooney | then ok | |
| 14:16:35 | sean-k-mooney | artom: fair its anorrying not to have these checks in the same place in the conductor with the other migration checks | |
| 14:16:47 | sean-k-mooney | but i agree we should just have a bool flag | |
| 14:16:51 | dansmith | no, this isn't a rank thing, it's just a simplicity thing... | |
| 14:16:54 | sean-k-mooney | so just update your current code | |
| 14:17:07 | artom | dansmith, a not very convincing thing, apparently :) | |
| 14:17:08 | dansmith | migratedata is created after we're done talking to conductor for the last time, AFAIK | |
| 14:17:20 | dansmith | so I'm not sure how conductor would ever set it for us | |
| 14:17:36 | artom | Yeah, it's created by the source compute, IIRC | |
| 14:17:42 | artom | The source driver, in fact | |
| 14:17:51 | dansmith | hence why this flag should be set by the computes | |
| 14:18:15 | dansmith | it also depends on the config of each compute, | |
| 14:18:21 | dansmith | which you can't inspect from conductor | |
| 14:18:35 | dansmith | if the two computes involved are pinned to 5.0, then this can't happen even if they both support it | |
| 14:18:38 | artom | https://github.com/openstack/nova/blob/18.0.0/nova/compute/manager.py#L6056-L6089 | |
| 14:19:21 | stephenfin | gibi: calling 'self._delete_server' at the end of the test did the trick. I don't know why. Missing mocks or something like that. Good enough for me though | |
| 14:19:36 | artom | I need to get one kid to daycare, after that it's hax time | |
| 14:20:51 | artom | mriedem, you OK with squashing, as per dansmith's -1 here: https://review.opendev.org/#/c/635229/48? | |
| 14:21:04 | sean-k-mooney | ok i was suggeting passing in the booll here https://github.com/openstack/nova/blob/master/nova/conductor/tasks/live_migrate.py#L318-L322 to check_can_live_migrate_destination but im fine with what dan suggest too | |
| 14:23:50 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Add support for translating CPU policy extra specs, image meta https://review.opendev.org/671801 | |
| 14:23:51 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Add reshaper for PCPU https://review.opendev.org/674895 | |
| 14:23:51 | openstackgerrit | Stephen Finucane proposed openstack/nova master: libvirt: Fold in argument to '_update_provider_tree_for_vgpu' https://review.opendev.org/676729 | |
| 14:23:52 | openstackgerrit | Stephen Finucane proposed openstack/nova master: libvirt: Simplify 'fakelibvirt.HostInfo' object https://review.opendev.org/678861 | |
| 14:23:56 | mriedem | artom: my head is buried in something else atm so you're going to have to wait | |
| 14:24:07 | artom | mriedem, understood | |
| 14:25:08 | stephenfin | dansmith: If you're bored, I went and wrote that online migration to allow us to drop the Liberty-era NUMA topology blobs https://review.opendev.org/#/c/537414 I don't want to brag, but I think it's perfect | |
| 14:27:24 | dansmith | stephenfin: you know that if you say that, I double-fist red pens right? :D | |
| 14:27:52 | stephenfin | dansmith: I was counting on it ;) | |
| 14:27:57 | dansmith | heh | |
| 14:29:51 | gibi | stephenfin: I'm happy that you solved it. I have no idea what was the problem without digging | |
| 14:30:03 | sean-k-mooney | stephenfin: pet peve but i find the use of ~ for negation in filters really hard to spot | |
| 14:30:29 | sean-k-mooney | it took me 3 times to notice it was not containts give how far apart they are | |
| 14:31:20 | dansmith | artom: did you see my question about the multinode test? | |
| 14:31:24 | dansmith | artom: earlier in here | |
| 14:33:10 | artom | dansmith, yeah... I'd have to think about it some more, but I don't think it's possible to test it in the current gate | |
| 14:33:21 | artom | Or if it is, it'd be a very small part | |
| 14:33:22 | dansmith | stephenfin: you just added an online migration to the patch with the removal as before right? | |
| 14:33:44 | dansmith | artom: if I had a guest that just requested one numa node, wouldn't that boot in our current gate workers? | |
| 14:33:45 | artom | I *think* I'd rather concentrate on getting whitebox under openstack-qa and getting multi-NUMA node flavors from Fort Nebula | |
| 14:33:57 | stephenfin | dansmith: Yeah. That's what I should do, right? | |
| 14:34:22 | artom | dansmith, it might, but then what? | |
| 14:34:24 | dansmith | stephenfin: it doesn't change the fact that you can't remove that stuff in the same release | |
| 14:34:46 | dansmith | artom: live migrate such an instance between workers and that will exercise all of this code, even if it's a boring example right? | |
| 14:35:03 | stephenfin | Oh, because it would have to be a blocker migration to do that? | |
| 14:35:04 | dansmith | stephenfin: online migrations run after you've already rolled out new code | |
| 14:35:37 | dansmith | stephenfin: you can't have a blocker and a removal in the same release either, but I was saying a blocker migration is likely too expensive here because of what it has to check, so we should do it with time and warnings | |
| 14:36:07 | dansmith | stephenfin: i.e. land an online migration now, and a nova-status check, wait a release or two, then remove the compat code after everyone has had a chance to patch up all their data | |
| 14:36:21 | stephenfin | Damn. So how long does that have to hang around for once I've the migration in-place? | |
| 14:36:21 | dansmith | stephenfin: a blocker migration helps make sure they clean up before they move, but again, too expensive here I tink | |
| 14:36:41 | stephenfin | mkay | |
| 14:36:43 | dansmith | stephenfin: well, at least a cycle, but with no blocker, it'd be nice to leave it longer | |
| 14:37:04 | dansmith | maintaining compat sucks | |
| 14:37:17 | stephenfin | I'll probably pull that out of the series too, if I can | |
| 14:37:19 | dansmith | it's a good reason to try to get stuff right the first time | |
| 14:37:23 | dansmith | yeah, good idea | |
| 14:37:27 | stephenfin | true dat | |
| 14:38:44 | artom | dansmith, it'll exercise the code, but assuming the workers have identical NUMA topologies, we'll have no way of knowing | |
| 14:38:53 | artom | Because in those cases, current live migration "works" | |
| 14:39:08 | stephenfin | sean-k-mooney: fyi, six doesn't have a wrapper for the collections -> collections.abc thing | |
| 14:39:14 | stephenfin | there's an open bug but no patch for it | |
| 14:39:15 | artom | Both hosts being identical means we don't need to update the XML or claim any "new" resources | |
| 14:39:35 | sean-k-mooney | stephenfin: right but it does for is this python 3 or not | |
| 14:40:00 | sean-k-mooney | stephenfin: but ya i need to stil fix that actully so ill respin shortly | |
| 14:40:03 | stephenfin | Oh, right, gotcha | |
| 14:40:16 | stephenfin | Could you spin out 'flatten_iterable' into a separate patch too, in that case? | |
| 14:40:22 | stephenfin | If you have uses for it elsewhere | |
| 14:40:45 | dansmith | artom: no way of knowing what? if we see the debugs that show that you're doing the stuff, we'll know it's at least running your code, and not breaking something simple | |