| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-09-03 | |||
| 19:04:29 | mriedem | long after this | |
| 19:04:42 | artom | Yeah, thinking out loud, and confused by the 'pre-' prefix | |
| 19:04:48 | mriedem | https://docs.openstack.org/nova/latest/reference/live-migration.html | |
| 19:05:08 | artom | Yeah, you'd thunk it'd be on my bedroom wall by now | |
| 19:05:28 | mriedem | but instead it's just posters of grimace | |
| 19:05:43 | artom | Nah brah, it's Ahnold | |
| 19:12:32 | artom | dansmith, straight up checking CONF.upgrade_level.compute? | |
| 19:12:39 | dansmith | no. | |
| 19:12:43 | artom | Haha | |
| 19:12:45 | dansmith | bad. | |
| 19:14:09 | mriedem | self.compute_rpcapi.router.client(context).can_send_version or something like that, but abstract it | |
| 19:15:17 | mriedem | def can_drop_live_migration_claim_at_dest | |
| 19:15:35 | artom | It's not even that | |
| 19:15:40 | artom | It's def can_tell_dest_to_drop_calim | |
| 19:17:16 | mriedem | yeah yeah | |
| 19:17:40 | mriedem | can_formally_please_to_be_requesting_you_rollback_a_live_migration_dest_claim_thank_you_very_much | |
| 19:17:47 | artom | version shouldn't leave rpcapi | |
| 19:18:01 | artom | So it would be something like can_send(method_name) | |
| 19:18:13 | artom | dansmith ^^ acceptable-eesh? | |
| 19:18:25 | mriedem | whatever, keep it simple since it gets dropped in U | |
| 19:18:29 | dansmith | well, that only works in this specific case, because most of the time it'll be | |
| 19:18:36 | dansmith | Can send? True...but... | |
| 19:18:51 | dansmith | but, I guess something like that is the quickets | |
| 19:20:19 | artom | I mean, unless we want to change the whole order, and have the conductor call the source, which then calls check_can_live_migrate_destination | |
| 19:20:27 | artom | Seems like overkill tho | |
| 19:22:17 | mriedem | you must be crazy | |
| 19:22:54 | sean-k-mooney | artom: why | |
| 19:23:15 | sean-k-mooney | the conductor currenlty calls check_can_live_migrate_destination direclty | |
| 19:23:27 | artom | sean-k-mooney, source needs to be sure it can send RPC 5.3 before putting src_supports_numa_live_migration in migrate_data | |
| 19:23:35 | artom | Becaues it calls drop_move_claim on rollback | |
| 19:23:39 | artom | And that's new in 5.3 | |
| 19:23:53 | sean-k-mooney | ok but the souce is called from the dest | |
| 19:24:01 | sean-k-mooney | so the test can tell the souce right | |
| 19:24:14 | mriedem | sean-k-mooney: read back from this: | |
| 19:24:15 | mriedem | (1:41:27 PM) dansmith: artom: since you rebased it's a little hard to tell...can you summarize what you're now doing to indicate that both ends are new enough to do the numa LM? | |
| 19:24:17 | artom | Yes, but the source needs to know *it* can send 5.3 - ie that it's not pinned to 5.2 for example | |
| 19:24:48 | artom | Hence this discussion on how do we achieve that without exposing rpcapi internals, or having a dummy method for the source to attempt to call | |
| 19:25:32 | sean-k-mooney | mriedem: ok will do. but im concerned that if the mechanium we use for sriov livemigration and multipel port binidngs is not enough for numa migriton all 3 feature would be incorect | |
| 19:26:30 | mriedem | i have no idea how well sriov live migration handles rolling upgrades | |
| 19:26:49 | sean-k-mooney | it handesl exactly as well as multiple portbindings | |
| 19:26:55 | sean-k-mooney | as we did exeactly the same thing | |
| 19:27:22 | mriedem | checking the source and dest compute service versions yeah | |
| 19:27:51 | sean-k-mooney | we have service version checkes here https://github.com/openstack/nova/blob/master/nova/conductor/tasks/live_migrate.py#L35-L58 and we call them here https://github.com/openstack/nova/blob/master/nova/conductor/tasks/live_migrate.py#L317-L344 | |
| 19:28:03 | sean-k-mooney | mriedem: yes | |
| 19:28:10 | artom | I think SRIOV was different because it would outright forbid it everything wasn't updated | |
| 19:28:23 | artom | NUMA LM has to work across N/N-1 boundaries | |
| 19:28:25 | sean-k-mooney | so did the multiple port bindign yes | |
| 19:28:33 | dansmith | well, that doesn't necessarily mean it's good, | |
| 19:28:45 | dansmith | but if it didn't have the same amount of rpc involvement as this it would be different | |
| 19:29:13 | artom | Dunno about RPC, but actually yeah, they could all have the new service version, but still be pinned when sending | |
| 19:29:14 | sean-k-mooney | the numa migration should have the same rpc workflow | |
| 19:30:17 | mriedem | sriov live migration rolls back the claim on dest here https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L7496 | |
| 19:30:21 | dansmith | it's not the workflow | |
| 19:30:22 | mriedem | which the source will call | |
| 19:30:28 | mriedem | so it's not really the same thing artom is dealing with, | |
| 19:30:32 | mriedem | since artom is calling a new method | |
| 19:30:39 | dansmith | probably better to just read the scrollback | |
| 19:30:41 | sean-k-mooney | mriedem: ah good point | |
| 19:30:56 | sean-k-mooney | we dont intoduce new calls jsut now object for sriov | |
| 19:31:07 | sean-k-mooney | and multiple port bindings | |
| 19:33:49 | mriedem | brinzhang: when it comes to your microversion API changes, you should probably focus on https://review.opendev.org/#/c/673133/ since i think it's closer and less complicated | |
| 19:38:05 | sean-k-mooney | i have read half the scrollback but dumb question. since the extra complexity comes form trying to support cases other that both host are upgraded to the new version. could we just prevent that. we could add a workaoruds config options to allow people to opt in and state if you set this you need to ensure the correct rpc versions are set. | |
| 19:38:30 | sean-k-mooney | it wont just magically work but it avoid this issue entirely. | |
| 19:40:15 | artom | sean-k-mooney, no, because in a rolling upgrade we want to allow old computes to be "live-evacuated" to new ones | |
| 19:40:34 | sean-k-mooney | artom: right im saying why dont we not support that | |
| 19:40:53 | sean-k-mooney | but different question | |
| 19:41:02 | artom | sean-k-mooney, because it's a pretty crucial thing, IMHO :) | |
| 19:41:05 | sean-k-mooney | if we dont call the cleanup function on rollback | |
| 19:41:07 | artom | Anyways, the complexity comes from thinking about it | |
| 19:41:22 | artom | Which you need to do regardless | |
| 19:41:29 | sean-k-mooney | will the RT on the dest clean it up when the migration is rolled back | |
| 19:41:31 | artom | The code itself is clean-ish | |
| 19:41:37 | artom | sean-k-mooney, yeah, next periodic run | |
| 19:42:00 | sean-k-mooney | artom: ok so if we did not call the cleanup funtion it would get cleaned up eventually | |
| 19:42:06 | artom | Eventually | |
| 19:42:19 | mriedem | can i get another core to approve this simple refactor https://review.opendev.org/#/c/679651/ | |
| 19:49:16 | mriedem | dansmith: btw on https://review.opendev.org/#/c/633293/46 i gave a paste for what it would look like to use the existing prep_resize | |
| 19:50:27 | mriedem | at this point we can defer caring about that series until U i guess | |
| 19:55:54 | artom | dansmith, btw, I have a func test for RPC pinning, and it did find 1 bug. Wonder why it didn't pick up drop_move_claim() raising an RPC version error | |
| 19:56:31 | dansmith | artom: I dunno, probably worth looking though | |
| 19:56:39 | artom | Yep. | |
| 19:56:57 | dansmith | mriedem: okay, sorry I didn't really mean to block everything on that, I just wanted to ponder a bit but I never got anywhere | |
| 19:57:32 | openstackgerrit | Dustin Cowles proposed openstack/nova master: Provider Config File: Update provider tree with new custom resources https://review.opendev.org/676522 | |
| 20:03:31 | sean-k-mooney | ok i think im following what ye are suggesting | |
| 20:03:51 | sean-k-mooney | i also like mriedem function name but pep8 would hate it | |
| 20:21:02 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: NUMA live migration support https://review.opendev.org/634606 | |
| 20:21:03 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: Deprecate CONF.workarounds.enable_numa_live_migration https://review.opendev.org/640021 | |
| 20:21:03 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: Functional tests for NUMA live migration https://review.opendev.org/672595 | |
| 20:21:38 | artom | dansmith, mriedem, ^^ and at this point I need to done for a bit, because it's dad taxi and supper time | |
| 20:22:24 | artom | But if there's anything I'll try to address it later tonight | |
| 20:26:09 | sean-k-mooney | aspiers: som of the sev functional test are asserting behavior im not sure is correct or rather is failing when in integrate my image-metadata-prefilter series | |
| 20:26:44 | sean-k-mooney | aspiers: you seam to be asserting the ablit for the sev support to be removed while the agent is running. | |
| 20:27:19 | sean-k-mooney | the commen stats that thsi coudl happen as a result of a reconfiguration or kernel downgrade which woul nomally inlvoe the agent being resarted | |
| 20:28:45 | sean-k-mooney | aspiers: it looks like you are adding the sev tratis to the existing cpu tratis function which i am now cacheing as a property | |
| 20:28:58 | sean-k-mooney | https://review.opendev.org/#/c/666915/15/nova/virt/libvirt/driver.py | |
| 20:29:34 | sean-k-mooney | i can fix your functional test by reseting the property when you simulate teh restart of the compute service | |
| 20:29:52 | sean-k-mooney | but i wan to confim that you dont intend this to change outside of an agent restart | |
| 20:36:37 | sean-k-mooney | aspiers: im going to proceed with the assumetion i have interpred the test coorectly based on the comments and you are simulating an agent restart and just reset the chached traits. if you get a chance to review and confim that woudl be good. | |
| 21:12:49 | efried | nova FYI https://review.opendev.org/#/c/679862/ | |
| 21:14:20 | dansmith | +1 | |