Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-09-03
18:53:14 dansmith that's the point of what I was asking above,
18:53:31 dansmith you're passing a flag from conductor->destination, and if it makes it to dest, you know conductor is allowed to send the new version
18:53:45 dansmith and you pass a flag from dest->source during the check phase, which proves the dest can send the newer version,
18:54:12 dansmith but if the source hasn't had its config unpinned and restarted, it will be unable to make the drop_claim call during rollback, but you've already claimed on the dest
18:55:30 artom Ohh, I see
18:55:50 artom Before including src_supports_numa_live_migration in migate_data, the sources needs to be sure it can speak 5.3
18:55:55 artom *source
18:56:05 dansmith right
18:58:27 artom I don't have ideas besides "try a new dummy 5.3 method in a try/except"
18:58:33 artom Which is ugly as hell
18:58:38 artom Is there precedent for this kind of thing?
18:59:03 dansmith no, don't do that :)
18:59:07 dansmith mriedem: see what I mean ^ ?
19:00:04 mriedem i feel like the most we've ever done is just checking the compute service versions, but this is more than that yeah? so not only are the source and dest upgraded, but also checking the compute rpcapi client can_send_version is True from source to dest
19:00:25 mriedem so you want check_can_live_migrate_source to check the compute rpcapi client.can_send_version right?
19:00:35 artom mriedem, basically yeah
19:00:43 mriedem with a todo to drop that in U
19:00:55 artom Whenever 6.0 happens
19:01:12 dansmith well,
19:01:20 dansmith I don't want to call can_send_version from outside rpcapi
19:01:22 artom I'm still fuzzy on the whole compatibility thing, but IIUC all of 5.x needs to be cross-compatible?
19:01:27 dansmith but something to that effect
19:02:17 dansmith service_version isn't really enough here,
19:02:31 dansmith because it indicates what that service can receive, but not if the config *allows* sending newer stuff
19:04:05 artom Hrmm, source calls pre_live_migration on the dest
19:04:27 artom No, that's after
19:04:29 mriedem long after this
19:04:42 artom Yeah, thinking out loud, and confused by the 'pre-' prefix
19:04:48 mriedem https://docs.openstack.org/nova/latest/reference/live-migration.html
19:05:08 artom Yeah, you'd thunk it'd be on my bedroom wall by now
19:05:28 mriedem but instead it's just posters of grimace
19:05:43 artom Nah brah, it's Ahnold
19:12:32 artom dansmith, straight up checking CONF.upgrade_level.compute?
19:12:39 dansmith no.
19:12:43 artom Haha
19:12:45 dansmith bad.
19:14:09 mriedem self.compute_rpcapi.router.client(context).can_send_version or something like that, but abstract it
19:15:17 mriedem def can_drop_live_migration_claim_at_dest
19:15:35 artom It's not even that
19:15:40 artom It's def can_tell_dest_to_drop_calim
19:17:16 mriedem yeah yeah
19:17:40 mriedem can_formally_please_to_be_requesting_you_rollback_a_live_migration_dest_claim_thank_you_very_much
19:17:47 artom version shouldn't leave rpcapi
19:18:01 artom So it would be something like can_send(method_name)
19:18:13 artom dansmith ^^ acceptable-eesh?
19:18:25 mriedem whatever, keep it simple since it gets dropped in U
19:18:29 dansmith well, that only works in this specific case, because most of the time it'll be
19:18:36 dansmith Can send? True...but...
19:18:51 dansmith but, I guess something like that is the quickets
19:20:19 artom I mean, unless we want to change the whole order, and have the conductor call the source, which then calls check_can_live_migrate_destination
19:20:27 artom Seems like overkill tho
19:22:17 mriedem you must be crazy
19:22:54 sean-k-mooney artom: why
19:23:15 sean-k-mooney the conductor currenlty calls check_can_live_migrate_destination direclty
19:23:27 artom sean-k-mooney, source needs to be sure it can send RPC 5.3 before putting src_supports_numa_live_migration in migrate_data
19:23:35 artom Becaues it calls drop_move_claim on rollback
19:23:39 artom And that's new in 5.3
19:23:53 sean-k-mooney ok but the souce is called from the dest
19:24:01 sean-k-mooney so the test can tell the souce right
19:24:14 mriedem sean-k-mooney: read back from this:
19:24:15 mriedem (1:41:27 PM) dansmith: artom: since you rebased it's a little hard to tell...can you summarize what you're now doing to indicate that both ends are new enough to do the numa LM?
19:24:17 artom Yes, but the source needs to know *it* can send 5.3 - ie that it's not pinned to 5.2 for example
19:24:48 artom Hence this discussion on how do we achieve that without exposing rpcapi internals, or having a dummy method for the source to attempt to call
19:25:32 sean-k-mooney mriedem: ok will do. but im concerned that if the mechanium we use for sriov livemigration and multipel port binidngs is not enough for numa migriton all 3 feature would be incorect
19:26:30 mriedem i have no idea how well sriov live migration handles rolling upgrades
19:26:49 sean-k-mooney it handesl exactly as well as multiple portbindings
19:26:55 sean-k-mooney as we did exeactly the same thing
19:27:22 mriedem checking the source and dest compute service versions yeah
19:27:51 sean-k-mooney we have service version checkes here https://github.com/openstack/nova/blob/master/nova/conductor/tasks/live_migrate.py#L35-L58 and we call them here https://github.com/openstack/nova/blob/master/nova/conductor/tasks/live_migrate.py#L317-L344
19:28:03 sean-k-mooney mriedem: yes
19:28:10 artom I think SRIOV was different because it would outright forbid it everything wasn't updated
19:28:23 artom NUMA LM has to work across N/N-1 boundaries
19:28:25 sean-k-mooney so did the multiple port bindign yes
19:28:33 dansmith well, that doesn't necessarily mean it's good,
19:28:45 dansmith but if it didn't have the same amount of rpc involvement as this it would be different
19:29:13 artom Dunno about RPC, but actually yeah, they could all have the new service version, but still be pinned when sending
19:29:14 sean-k-mooney the numa migration should have the same rpc workflow
19:30:17 mriedem sriov live migration rolls back the claim on dest here https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L7496
19:30:21 dansmith it's not the workflow
19:30:22 mriedem which the source will call
19:30:28 mriedem so it's not really the same thing artom is dealing with,
19:30:32 mriedem since artom is calling a new method
19:30:39 dansmith probably better to just read the scrollback
19:30:41 sean-k-mooney mriedem: ah good point
19:30:56 sean-k-mooney we dont intoduce new calls jsut now object for sriov
19:31:07 sean-k-mooney and multiple port bindings
19:33:49 mriedem brinzhang: when it comes to your microversion API changes, you should probably focus on https://review.opendev.org/#/c/673133/ since i think it's closer and less complicated
19:38:05 sean-k-mooney i have read half the scrollback but dumb question. since the extra complexity comes form trying to support cases other that both host are upgraded to the new version. could we just prevent that. we could add a workaoruds config options to allow people to opt in and state if you set this you need to ensure the correct rpc versions are set.
19:38:30 sean-k-mooney it wont just magically work but it avoid this issue entirely.
19:40:15 artom sean-k-mooney, no, because in a rolling upgrade we want to allow old computes to be "live-evacuated" to new ones
19:40:34 sean-k-mooney artom: right im saying why dont we not support that
19:40:53 sean-k-mooney but different question
19:41:02 artom sean-k-mooney, because it's a pretty crucial thing, IMHO :)
19:41:05 sean-k-mooney if we dont call the cleanup function on rollback
19:41:07 artom Anyways, the complexity comes from thinking about it
19:41:22 artom Which you need to do regardless
19:41:29 sean-k-mooney will the RT on the dest clean it up when the migration is rolled back
19:41:31 artom The code itself is clean-ish
19:41:37 artom sean-k-mooney, yeah, next periodic run
19:42:00 sean-k-mooney artom: ok so if we did not call the cleanup funtion it would get cleaned up eventually

Earlier   Later