| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-09-03 | |||
| 16:26:07 | sean-k-mooney | donnyd: cool | |
| 16:26:31 | sean-k-mooney | ill try and swap the jobs over to use those during the week | |
| 16:28:25 | mriedem | gibi: i left comments on https://review.opendev.org/#/c/655113/23 but will withold voting for now | |
| 16:28:37 | mriedem | *withhold | |
| 16:30:18 | mriedem | https://review.opendev.org/#/c/679651/ and https://review.opendev.org/#/c/679653/ are pretty simple if another core can hit those - i'm going to have to rebase the cross-cell-resize series and i'd like to wait until after those are merged since they are going to merge conflict me as well | |
| 16:32:13 | sean-k-mooney | donnyd: are you ok with leaveing the current lable as it is till the end of the week and ill swap over to the new lable at the weekend | |
| 16:32:30 | donnyd | Makes no nevermind to me | |
| 16:32:54 | donnyd | I can leave it intact until asked to refactor the current jobs | |
| 16:56:14 | openstackgerrit | Merged openstack/nova master: objects: Rename 'fields' import to 'obj_fields' https://review.opendev.org/674103 | |
| 16:56:22 | openstackgerrit | Merged openstack/nova master: Nice to have test coverage for If1f465112b8e9b0304b8b5b864b985f72168d839 https://review.opendev.org/679629 | |
| 16:56:31 | openstackgerrit | Merged openstack/nova master: Remove dead code https://review.opendev.org/679668 | |
| 17:11:58 | sean-k-mooney | artom: https://46b5dc7879590352c3ef-b484a2b89d5d2c358f068133dfb2fa14.ssl.cf5.rackcdn.com/679805/2/experimental/nova-nfv-multi-numa-multinode/b040d28/testr_results.html.gz :) that is for https://review.opendev.org/#/c/679805/2 | |
| 17:12:28 | sean-k-mooney | artom: the other two are going to fail due to capasity issue with concurrent tests | |
| 17:12:35 | artom | sean-k-mooney, yeah, I just posted on https://review.opendev.org/#/c/679656/7 summarized the CI success | |
| 17:12:50 | sean-k-mooney | but they did pass some of the tests | |
| 17:13:10 | sean-k-mooney | artom: ok cool | |
| 17:13:14 | artom | dansmith, mriedem, if you guys want to take a look ^^ I'd say https://review.opendev.org/#/c/672595/ is ready | |
| 17:13:27 | artom | Err, https://review.opendev.org/#/c/635669/49 | |
| 17:13:30 | artom | The first patch | |
| 17:15:57 | artom | So there's a failure in https://storage.gra1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/logs_54/679754/2/experimental/nova-nfv-multi-numa-multinode/7956dad/testr_results.html.gz, but it's because it failed to even create the server | |
| 17:16:01 | artom | The other live migration tests passed | |
| 17:16:31 | sean-k-mooney | artom: yep that is due to tempest_concurrency 2 | |
| 17:22:29 | mriedem | artom: the functional test patch is in merge conflict | |
| 17:23:36 | artom | mriedem, I know, working on it, it's https://review.opendev.org/678861 | |
| 17:47:21 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: Introduce live_migration_claim() https://review.opendev.org/635669 | |
| 17:47:22 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: New objects for NUMA live migration https://review.opendev.org/634827 | |
| 17:47:22 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: LM: Use Claims to update numa-related XML on the source https://review.opendev.org/635229 | |
| 17:47:23 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: NUMA live migration support https://review.opendev.org/634606 | |
| 17:47:23 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: Deprecate CONF.workarounds.enable_numa_live_migration https://review.opendev.org/640021 | |
| 17:47:24 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: Functional tests for NUMA live migration https://review.opendev.org/672595 | |
| 17:47:56 | artom | mriedem, ^^ | |
| 17:48:22 | efried | dansmith: Would you please have a look at the bottom two patches of https://review.opendev.org/#/c/678447/ when you get a chance? There's still a bit of work to be done up the series, but stephenfin and I are reviewing that. Want to be able to un-hold and send it once it's baked without having to bug you in emergency mode. | |
| 17:50:32 | openstackgerrit | Merged openstack/nova master: Use SDK for node.list https://review.opendev.org/656027 | |
| 17:50:56 | dansmith | artom: okay so it's passing everything or still failing legit on something? | |
| 17:51:15 | mriedem | landing that vpmems series in ~9 days makes me personally uncomfortable | |
| 17:51:20 | artom | dansmith, passing everything as far as I can tell | |
| 17:51:28 | dansmith | efried: yeah, just got off a call, so let me bio break and then I will | |
| 17:51:33 | dansmith | artom: oh okay, sweet | |
| 17:51:36 | efried | thank you dansmith | |
| 18:17:47 | openstackgerrit | Merged openstack/nova master: Use SDK for validating instance and node https://review.opendev.org/656028 | |
| 18:30:20 | openstackgerrit | Merged openstack/nova master: Use SDK for setting instance id https://review.opendev.org/659690 | |
| 18:37:37 | dansmith | is FF this thursday? | |
| 18:37:55 | mriedem | next | |
| 18:39:05 | dansmith | oh okay cool | |
| 18:41:27 | dansmith | artom: since you rebased it's a little hard to tell...can you summarize what you're now doing to indicate that both ends are new enough to do the numa LM? | |
| 18:42:46 | artom | dansmith, https://review.opendev.org/#/c/634606/70/nova/compute/manager.py@6486 | |
| 18:43:22 | artom | If the destination got a migration flag from the conductor, it sets the numa_live_migration sentinel to True | |
| 18:43:42 | artom | It then passes that sentinel (either True or False) to check_can_live_migrate_source | |
| 18:45:04 | dansmith | is that all? | |
| 18:45:24 | artom | The source compute manager calls the driver's check_can_live_migrate_source - if that driver sets src_supports_numa_live_migration but the numa_live_migration sentinel was False or None, src_supports_numa_live_migration gets removed from migrate_data: https://review.opendev.org/#/c/634606/70/nova/compute/manager.py@6561 | |
| 18:46:13 | artom | Which would then cause the check at https://review.opendev.org/#/c/634606/70/nova/compute/manager.py@6519 back on the destination to be False | |
| 18:46:41 | dansmith | okay, so we've passed a new thing in conductor->destination, proving the conductor's rpc allows it, | |
| 18:46:52 | artom | Exactly | |
| 18:46:59 | dansmith | and then a new thing from destination->source, proving that destination's rpc allows it | |
| 18:47:09 | artom | Well, that's always been the case (conductor -> destination), it was the migration object | |
| 18:47:17 | dansmith | do we ever make a call from source->destination directly? | |
| 18:47:17 | artom | The new thing is destination -> source | |
| 18:47:25 | dansmith | like on rollback? | |
| 18:48:04 | dansmith | if so, I don't think you know that source's config allows the newer version in order to be able to make the newer rollback call right? | |
| 18:48:56 | artom | https://review.opendev.org/#/c/634606/70/nova/compute/manager.py@7417 is the source -> destination call on rollback | |
| 18:49:23 | artom | No, not that | |
| 18:49:35 | dansmith | right, but the source will call to the destination during rollback to make the new drop claim call right? | |
| 18:50:36 | artom | https://review.opendev.org/#/c/634606/70/nova/compute/manager.py@7406 is the drop claim call | |
| 18:50:40 | artom | Which is entirely new in 5.3 | |
| 18:51:38 | dansmith | right, but I don't think you know whether or not the source can call the destination at 5.3 until you need to rollback and make that call right? | |
| 18:52:23 | artom | Isn't that what the check on https://review.opendev.org/#/c/634606/70/nova/compute/manager.py@7405 does? | |
| 18:52:36 | artom | 'dst_numa_info' will not be in migrate_data if conductor, source, or dest is pinned | |
| 18:52:43 | mriedem | if dst_numa_info is in migrate_data which is built up on the dest wouldn't the rollback be safe to assume it can call the drop claim method? | |
| 18:53:03 | dansmith | no, because the source's config is different | |
| 18:53:14 | dansmith | that's the point of what I was asking above, | |
| 18:53:31 | dansmith | you're passing a flag from conductor->destination, and if it makes it to dest, you know conductor is allowed to send the new version | |
| 18:53:45 | dansmith | and you pass a flag from dest->source during the check phase, which proves the dest can send the newer version, | |
| 18:54:12 | dansmith | but if the source hasn't had its config unpinned and restarted, it will be unable to make the drop_claim call during rollback, but you've already claimed on the dest | |
| 18:55:30 | artom | Ohh, I see | |
| 18:55:50 | artom | Before including src_supports_numa_live_migration in migate_data, the sources needs to be sure it can speak 5.3 | |
| 18:55:55 | artom | *source | |
| 18:56:05 | dansmith | right | |
| 18:58:27 | artom | I don't have ideas besides "try a new dummy 5.3 method in a try/except" | |
| 18:58:33 | artom | Which is ugly as hell | |
| 18:58:38 | artom | Is there precedent for this kind of thing? | |
| 18:59:03 | dansmith | no, don't do that :) | |
| 18:59:07 | dansmith | mriedem: see what I mean ^ ? | |
| 19:00:04 | mriedem | i feel like the most we've ever done is just checking the compute service versions, but this is more than that yeah? so not only are the source and dest upgraded, but also checking the compute rpcapi client can_send_version is True from source to dest | |
| 19:00:25 | mriedem | so you want check_can_live_migrate_source to check the compute rpcapi client.can_send_version right? | |
| 19:00:35 | artom | mriedem, basically yeah | |
| 19:00:43 | mriedem | with a todo to drop that in U | |
| 19:00:55 | artom | Whenever 6.0 happens | |
| 19:01:12 | dansmith | well, | |
| 19:01:20 | dansmith | I don't want to call can_send_version from outside rpcapi | |
| 19:01:22 | artom | I'm still fuzzy on the whole compatibility thing, but IIUC all of 5.x needs to be cross-compatible? | |
| 19:01:27 | dansmith | but something to that effect | |
| 19:02:17 | dansmith | service_version isn't really enough here, | |
| 19:02:31 | dansmith | because it indicates what that service can receive, but not if the config *allows* sending newer stuff | |
| 19:04:05 | artom | Hrmm, source calls pre_live_migration on the dest | |
| 19:04:27 | artom | No, that's after | |
| 19:04:29 | mriedem | long after this | |
| 19:04:42 | artom | Yeah, thinking out loud, and confused by the 'pre-' prefix | |
| 19:04:48 | mriedem | https://docs.openstack.org/nova/latest/reference/live-migration.html | |
| 19:05:08 | artom | Yeah, you'd thunk it'd be on my bedroom wall by now | |
| 19:05:28 | mriedem | but instead it's just posters of grimace | |