| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-09-03 | |||
| 17:22:29 | mriedem | artom: the functional test patch is in merge conflict | |
| 17:23:36 | artom | mriedem, I know, working on it, it's https://review.opendev.org/678861 | |
| 17:47:21 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: Introduce live_migration_claim() https://review.opendev.org/635669 | |
| 17:47:22 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: LM: Use Claims to update numa-related XML on the source https://review.opendev.org/635229 | |
| 17:47:22 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: New objects for NUMA live migration https://review.opendev.org/634827 | |
| 17:47:23 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: Deprecate CONF.workarounds.enable_numa_live_migration https://review.opendev.org/640021 | |
| 17:47:23 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: NUMA live migration support https://review.opendev.org/634606 | |
| 17:47:24 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: Functional tests for NUMA live migration https://review.opendev.org/672595 | |
| 17:47:56 | artom | mriedem, ^^ | |
| 17:48:22 | efried | dansmith: Would you please have a look at the bottom two patches of https://review.opendev.org/#/c/678447/ when you get a chance? There's still a bit of work to be done up the series, but stephenfin and I are reviewing that. Want to be able to un-hold and send it once it's baked without having to bug you in emergency mode. | |
| 17:50:32 | openstackgerrit | Merged openstack/nova master: Use SDK for node.list https://review.opendev.org/656027 | |
| 17:50:56 | dansmith | artom: okay so it's passing everything or still failing legit on something? | |
| 17:51:15 | mriedem | landing that vpmems series in ~9 days makes me personally uncomfortable | |
| 17:51:20 | artom | dansmith, passing everything as far as I can tell | |
| 17:51:28 | dansmith | efried: yeah, just got off a call, so let me bio break and then I will | |
| 17:51:33 | dansmith | artom: oh okay, sweet | |
| 17:51:36 | efried | thank you dansmith | |
| 18:17:47 | openstackgerrit | Merged openstack/nova master: Use SDK for validating instance and node https://review.opendev.org/656028 | |
| 18:30:20 | openstackgerrit | Merged openstack/nova master: Use SDK for setting instance id https://review.opendev.org/659690 | |
| 18:37:37 | dansmith | is FF this thursday? | |
| 18:37:55 | mriedem | next | |
| 18:39:05 | dansmith | oh okay cool | |
| 18:41:27 | dansmith | artom: since you rebased it's a little hard to tell...can you summarize what you're now doing to indicate that both ends are new enough to do the numa LM? | |
| 18:42:46 | artom | dansmith, https://review.opendev.org/#/c/634606/70/nova/compute/manager.py@6486 | |
| 18:43:22 | artom | If the destination got a migration flag from the conductor, it sets the numa_live_migration sentinel to True | |
| 18:43:42 | artom | It then passes that sentinel (either True or False) to check_can_live_migrate_source | |
| 18:45:04 | dansmith | is that all? | |
| 18:45:24 | artom | The source compute manager calls the driver's check_can_live_migrate_source - if that driver sets src_supports_numa_live_migration but the numa_live_migration sentinel was False or None, src_supports_numa_live_migration gets removed from migrate_data: https://review.opendev.org/#/c/634606/70/nova/compute/manager.py@6561 | |
| 18:46:13 | artom | Which would then cause the check at https://review.opendev.org/#/c/634606/70/nova/compute/manager.py@6519 back on the destination to be False | |
| 18:46:41 | dansmith | okay, so we've passed a new thing in conductor->destination, proving the conductor's rpc allows it, | |
| 18:46:52 | artom | Exactly | |
| 18:46:59 | dansmith | and then a new thing from destination->source, proving that destination's rpc allows it | |
| 18:47:09 | artom | Well, that's always been the case (conductor -> destination), it was the migration object | |
| 18:47:17 | artom | The new thing is destination -> source | |
| 18:47:17 | dansmith | do we ever make a call from source->destination directly? | |
| 18:47:25 | dansmith | like on rollback? | |
| 18:48:04 | dansmith | if so, I don't think you know that source's config allows the newer version in order to be able to make the newer rollback call right? | |
| 18:48:56 | artom | https://review.opendev.org/#/c/634606/70/nova/compute/manager.py@7417 is the source -> destination call on rollback | |
| 18:49:23 | artom | No, not that | |
| 18:49:35 | dansmith | right, but the source will call to the destination during rollback to make the new drop claim call right? | |
| 18:50:36 | artom | https://review.opendev.org/#/c/634606/70/nova/compute/manager.py@7406 is the drop claim call | |
| 18:50:40 | artom | Which is entirely new in 5.3 | |
| 18:51:38 | dansmith | right, but I don't think you know whether or not the source can call the destination at 5.3 until you need to rollback and make that call right? | |
| 18:52:23 | artom | Isn't that what the check on https://review.opendev.org/#/c/634606/70/nova/compute/manager.py@7405 does? | |
| 18:52:36 | artom | 'dst_numa_info' will not be in migrate_data if conductor, source, or dest is pinned | |
| 18:52:43 | mriedem | if dst_numa_info is in migrate_data which is built up on the dest wouldn't the rollback be safe to assume it can call the drop claim method? | |
| 18:53:03 | dansmith | no, because the source's config is different | |
| 18:53:14 | dansmith | that's the point of what I was asking above, | |
| 18:53:31 | dansmith | you're passing a flag from conductor->destination, and if it makes it to dest, you know conductor is allowed to send the new version | |
| 18:53:45 | dansmith | and you pass a flag from dest->source during the check phase, which proves the dest can send the newer version, | |
| 18:54:12 | dansmith | but if the source hasn't had its config unpinned and restarted, it will be unable to make the drop_claim call during rollback, but you've already claimed on the dest | |
| 18:55:30 | artom | Ohh, I see | |
| 18:55:50 | artom | Before including src_supports_numa_live_migration in migate_data, the sources needs to be sure it can speak 5.3 | |
| 18:55:55 | artom | *source | |
| 18:56:05 | dansmith | right | |
| 18:58:27 | artom | I don't have ideas besides "try a new dummy 5.3 method in a try/except" | |
| 18:58:33 | artom | Which is ugly as hell | |
| 18:58:38 | artom | Is there precedent for this kind of thing? | |
| 18:59:03 | dansmith | no, don't do that :) | |
| 18:59:07 | dansmith | mriedem: see what I mean ^ ? | |
| 19:00:04 | mriedem | i feel like the most we've ever done is just checking the compute service versions, but this is more than that yeah? so not only are the source and dest upgraded, but also checking the compute rpcapi client can_send_version is True from source to dest | |
| 19:00:25 | mriedem | so you want check_can_live_migrate_source to check the compute rpcapi client.can_send_version right? | |
| 19:00:35 | artom | mriedem, basically yeah | |
| 19:00:43 | mriedem | with a todo to drop that in U | |
| 19:00:55 | artom | Whenever 6.0 happens | |
| 19:01:12 | dansmith | well, | |
| 19:01:20 | dansmith | I don't want to call can_send_version from outside rpcapi | |
| 19:01:22 | artom | I'm still fuzzy on the whole compatibility thing, but IIUC all of 5.x needs to be cross-compatible? | |
| 19:01:27 | dansmith | but something to that effect | |
| 19:02:17 | dansmith | service_version isn't really enough here, | |
| 19:02:31 | dansmith | because it indicates what that service can receive, but not if the config *allows* sending newer stuff | |
| 19:04:05 | artom | Hrmm, source calls pre_live_migration on the dest | |
| 19:04:27 | artom | No, that's after | |
| 19:04:29 | mriedem | long after this | |
| 19:04:42 | artom | Yeah, thinking out loud, and confused by the 'pre-' prefix | |
| 19:04:48 | mriedem | https://docs.openstack.org/nova/latest/reference/live-migration.html | |
| 19:05:08 | artom | Yeah, you'd thunk it'd be on my bedroom wall by now | |
| 19:05:28 | mriedem | but instead it's just posters of grimace | |
| 19:05:43 | artom | Nah brah, it's Ahnold | |
| 19:12:32 | artom | dansmith, straight up checking CONF.upgrade_level.compute? | |
| 19:12:39 | dansmith | no. | |
| 19:12:43 | artom | Haha | |
| 19:12:45 | dansmith | bad. | |
| 19:14:09 | mriedem | self.compute_rpcapi.router.client(context).can_send_version or something like that, but abstract it | |
| 19:15:17 | mriedem | def can_drop_live_migration_claim_at_dest | |
| 19:15:35 | artom | It's not even that | |
| 19:15:40 | artom | It's def can_tell_dest_to_drop_calim | |
| 19:17:16 | mriedem | yeah yeah | |
| 19:17:40 | mriedem | can_formally_please_to_be_requesting_you_rollback_a_live_migration_dest_claim_thank_you_very_much | |
| 19:17:47 | artom | version shouldn't leave rpcapi | |
| 19:18:01 | artom | So it would be something like can_send(method_name) | |
| 19:18:13 | artom | dansmith ^^ acceptable-eesh? | |
| 19:18:25 | mriedem | whatever, keep it simple since it gets dropped in U | |
| 19:18:29 | dansmith | well, that only works in this specific case, because most of the time it'll be | |
| 19:18:36 | dansmith | Can send? True...but... | |
| 19:18:51 | dansmith | but, I guess something like that is the quickets | |
| 19:20:19 | artom | I mean, unless we want to change the whole order, and have the conductor call the source, which then calls check_can_live_migrate_destination | |
| 19:20:27 | artom | Seems like overkill tho | |
| 19:22:17 | mriedem | you must be crazy | |
| 19:22:54 | sean-k-mooney | artom: why | |