| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-09-03 | |||
| 18:41:27 | dansmith | artom: since you rebased it's a little hard to tell...can you summarize what you're now doing to indicate that both ends are new enough to do the numa LM? | |
| 18:42:46 | artom | dansmith, https://review.opendev.org/#/c/634606/70/nova/compute/manager.py@6486 | |
| 18:43:22 | artom | If the destination got a migration flag from the conductor, it sets the numa_live_migration sentinel to True | |
| 18:43:42 | artom | It then passes that sentinel (either True or False) to check_can_live_migrate_source | |
| 18:45:04 | dansmith | is that all? | |
| 18:45:24 | artom | The source compute manager calls the driver's check_can_live_migrate_source - if that driver sets src_supports_numa_live_migration but the numa_live_migration sentinel was False or None, src_supports_numa_live_migration gets removed from migrate_data: https://review.opendev.org/#/c/634606/70/nova/compute/manager.py@6561 | |
| 18:46:13 | artom | Which would then cause the check at https://review.opendev.org/#/c/634606/70/nova/compute/manager.py@6519 back on the destination to be False | |
| 18:46:41 | dansmith | okay, so we've passed a new thing in conductor->destination, proving the conductor's rpc allows it, | |
| 18:46:52 | artom | Exactly | |
| 18:46:59 | dansmith | and then a new thing from destination->source, proving that destination's rpc allows it | |
| 18:47:09 | artom | Well, that's always been the case (conductor -> destination), it was the migration object | |
| 18:47:17 | dansmith | do we ever make a call from source->destination directly? | |
| 18:47:17 | artom | The new thing is destination -> source | |
| 18:47:25 | dansmith | like on rollback? | |
| 18:48:04 | dansmith | if so, I don't think you know that source's config allows the newer version in order to be able to make the newer rollback call right? | |
| 18:48:56 | artom | https://review.opendev.org/#/c/634606/70/nova/compute/manager.py@7417 is the source -> destination call on rollback | |
| 18:49:23 | artom | No, not that | |
| 18:49:35 | dansmith | right, but the source will call to the destination during rollback to make the new drop claim call right? | |
| 18:50:36 | artom | https://review.opendev.org/#/c/634606/70/nova/compute/manager.py@7406 is the drop claim call | |
| 18:50:40 | artom | Which is entirely new in 5.3 | |
| 18:51:38 | dansmith | right, but I don't think you know whether or not the source can call the destination at 5.3 until you need to rollback and make that call right? | |
| 18:52:23 | artom | Isn't that what the check on https://review.opendev.org/#/c/634606/70/nova/compute/manager.py@7405 does? | |
| 18:52:36 | artom | 'dst_numa_info' will not be in migrate_data if conductor, source, or dest is pinned | |
| 18:52:43 | mriedem | if dst_numa_info is in migrate_data which is built up on the dest wouldn't the rollback be safe to assume it can call the drop claim method? | |
| 18:53:03 | dansmith | no, because the source's config is different | |
| 18:53:14 | dansmith | that's the point of what I was asking above, | |
| 18:53:31 | dansmith | you're passing a flag from conductor->destination, and if it makes it to dest, you know conductor is allowed to send the new version | |
| 18:53:45 | dansmith | and you pass a flag from dest->source during the check phase, which proves the dest can send the newer version, | |
| 18:54:12 | dansmith | but if the source hasn't had its config unpinned and restarted, it will be unable to make the drop_claim call during rollback, but you've already claimed on the dest | |
| 18:55:30 | artom | Ohh, I see | |
| 18:55:50 | artom | Before including src_supports_numa_live_migration in migate_data, the sources needs to be sure it can speak 5.3 | |
| 18:55:55 | artom | *source | |
| 18:56:05 | dansmith | right | |
| 18:58:27 | artom | I don't have ideas besides "try a new dummy 5.3 method in a try/except" | |
| 18:58:33 | artom | Which is ugly as hell | |
| 18:58:38 | artom | Is there precedent for this kind of thing? | |
| 18:59:03 | dansmith | no, don't do that :) | |
| 18:59:07 | dansmith | mriedem: see what I mean ^ ? | |
| 19:00:04 | mriedem | i feel like the most we've ever done is just checking the compute service versions, but this is more than that yeah? so not only are the source and dest upgraded, but also checking the compute rpcapi client can_send_version is True from source to dest | |
| 19:00:25 | mriedem | so you want check_can_live_migrate_source to check the compute rpcapi client.can_send_version right? | |
| 19:00:35 | artom | mriedem, basically yeah | |
| 19:00:43 | mriedem | with a todo to drop that in U | |
| 19:00:55 | artom | Whenever 6.0 happens | |
| 19:01:12 | dansmith | well, | |
| 19:01:20 | dansmith | I don't want to call can_send_version from outside rpcapi | |
| 19:01:22 | artom | I'm still fuzzy on the whole compatibility thing, but IIUC all of 5.x needs to be cross-compatible? | |
| 19:01:27 | dansmith | but something to that effect | |
| 19:02:17 | dansmith | service_version isn't really enough here, | |
| 19:02:31 | dansmith | because it indicates what that service can receive, but not if the config *allows* sending newer stuff | |
| 19:04:05 | artom | Hrmm, source calls pre_live_migration on the dest | |
| 19:04:27 | artom | No, that's after | |
| 19:04:29 | mriedem | long after this | |
| 19:04:42 | artom | Yeah, thinking out loud, and confused by the 'pre-' prefix | |
| 19:04:48 | mriedem | https://docs.openstack.org/nova/latest/reference/live-migration.html | |
| 19:05:08 | artom | Yeah, you'd thunk it'd be on my bedroom wall by now | |
| 19:05:28 | mriedem | but instead it's just posters of grimace | |
| 19:05:43 | artom | Nah brah, it's Ahnold | |
| 19:12:32 | artom | dansmith, straight up checking CONF.upgrade_level.compute? | |
| 19:12:39 | dansmith | no. | |
| 19:12:43 | artom | Haha | |
| 19:12:45 | dansmith | bad. | |
| 19:14:09 | mriedem | self.compute_rpcapi.router.client(context).can_send_version or something like that, but abstract it | |
| 19:15:17 | mriedem | def can_drop_live_migration_claim_at_dest | |
| 19:15:35 | artom | It's not even that | |
| 19:15:40 | artom | It's def can_tell_dest_to_drop_calim | |
| 19:17:16 | mriedem | yeah yeah | |
| 19:17:40 | mriedem | can_formally_please_to_be_requesting_you_rollback_a_live_migration_dest_claim_thank_you_very_much | |
| 19:17:47 | artom | version shouldn't leave rpcapi | |
| 19:18:01 | artom | So it would be something like can_send(method_name) | |
| 19:18:13 | artom | dansmith ^^ acceptable-eesh? | |
| 19:18:25 | mriedem | whatever, keep it simple since it gets dropped in U | |
| 19:18:29 | dansmith | well, that only works in this specific case, because most of the time it'll be | |
| 19:18:36 | dansmith | Can send? True...but... | |
| 19:18:51 | dansmith | but, I guess something like that is the quickets | |
| 19:20:19 | artom | I mean, unless we want to change the whole order, and have the conductor call the source, which then calls check_can_live_migrate_destination | |
| 19:20:27 | artom | Seems like overkill tho | |
| 19:22:17 | mriedem | you must be crazy | |
| 19:22:54 | sean-k-mooney | artom: why | |
| 19:23:15 | sean-k-mooney | the conductor currenlty calls check_can_live_migrate_destination direclty | |
| 19:23:27 | artom | sean-k-mooney, source needs to be sure it can send RPC 5.3 before putting src_supports_numa_live_migration in migrate_data | |
| 19:23:35 | artom | Becaues it calls drop_move_claim on rollback | |
| 19:23:39 | artom | And that's new in 5.3 | |
| 19:23:53 | sean-k-mooney | ok but the souce is called from the dest | |
| 19:24:01 | sean-k-mooney | so the test can tell the souce right | |
| 19:24:14 | mriedem | sean-k-mooney: read back from this: | |
| 19:24:15 | mriedem | (1:41:27 PM) dansmith: artom: since you rebased it's a little hard to tell...can you summarize what you're now doing to indicate that both ends are new enough to do the numa LM? | |
| 19:24:17 | artom | Yes, but the source needs to know *it* can send 5.3 - ie that it's not pinned to 5.2 for example | |
| 19:24:48 | artom | Hence this discussion on how do we achieve that without exposing rpcapi internals, or having a dummy method for the source to attempt to call | |
| 19:25:32 | sean-k-mooney | mriedem: ok will do. but im concerned that if the mechanium we use for sriov livemigration and multipel port binidngs is not enough for numa migriton all 3 feature would be incorect | |
| 19:26:30 | mriedem | i have no idea how well sriov live migration handles rolling upgrades | |
| 19:26:49 | sean-k-mooney | it handesl exactly as well as multiple portbindings | |
| 19:26:55 | sean-k-mooney | as we did exeactly the same thing | |
| 19:27:22 | mriedem | checking the source and dest compute service versions yeah | |
| 19:27:51 | sean-k-mooney | we have service version checkes here https://github.com/openstack/nova/blob/master/nova/conductor/tasks/live_migrate.py#L35-L58 and we call them here https://github.com/openstack/nova/blob/master/nova/conductor/tasks/live_migrate.py#L317-L344 | |
| 19:28:03 | sean-k-mooney | mriedem: yes | |
| 19:28:10 | artom | I think SRIOV was different because it would outright forbid it everything wasn't updated | |
| 19:28:23 | artom | NUMA LM has to work across N/N-1 boundaries | |
| 19:28:25 | sean-k-mooney | so did the multiple port bindign yes | |
| 19:28:33 | dansmith | well, that doesn't necessarily mean it's good, | |
| 19:28:45 | dansmith | but if it didn't have the same amount of rpc involvement as this it would be different | |