| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-09-03 | |||
| 15:17:18 | sean-k-mooney | yes | |
| 15:17:28 | sean-k-mooney | that fix everything except this last edgecase | |
| 15:17:43 | artom | You mean "hugepages: node 1 not found" is till happening? | |
| 15:17:49 | sean-k-mooney | yes | |
| 15:19:18 | openstackgerrit | sean mooney proposed openstack/nova master: [DNM] test non overlaping vcpu pin sets. https://review.opendev.org/679805 | |
| 15:19:29 | sean-k-mooney | i for got to disable the other jobs in ^ so now its fixed | |
| 15:21:05 | openstackgerrit | Merged openstack/nova master: Add |
|
| 15:21:58 | artom | I'm trying to understand what that error means | |
| 15:22:48 | openstackgerrit | sean mooney proposed openstack/nova master: [DNM] test migration with pinning https://review.opendev.org/679754 | |
| 15:23:37 | sean-k-mooney | artom: by the way the non WIP/DNM patches have the job in the perodic and experimental pipline so you need to use check experimenttal or somthing like that not recheck | |
| 15:24:29 | sean-k-mooney | artom: ill repoduce it locally again and dump the migration xml | |
| 15:26:44 | artom | sean-k-mooney, I *think* I'm getting it | |
| 15:26:57 | artom | I'll poke danpb downstream to make sure | |
| 15:27:11 | sean-k-mooney | ok | |
| 15:46:24 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: NUMA live migration support https://review.opendev.org/634606 | |
| 15:46:24 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: LM: Use Claims to update numa-related XML on the source https://review.opendev.org/635229 | |
| 15:46:25 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: Functional tests for NUMA live migration https://review.opendev.org/672595 | |
| 15:46:25 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: Deprecate CONF.workarounds.enable_numa_live_migration https://review.opendev.org/640021 | |
| 15:49:34 | efried | dustinc: Three sdk patches merging; problems on 4 & 5. | |
| 16:01:51 | openstackgerrit | Merged openstack/nova master: libvirt: Simplify 'fakelibvirt.HostInfo' object https://review.opendev.org/678861 | |
| 16:02:01 | openstackgerrit | Merged openstack/nova master: libvirt: Start checking compute usage in functional tests https://review.opendev.org/678902 | |
| 16:24:03 | donnyd | sean-k-mooney: Ok FN should have the right stuff to run the NUMA jobs now | |
| 16:24:20 | donnyd | https://review.opendev.org/#/c/679789/ | |
| 16:26:07 | sean-k-mooney | donnyd: cool | |
| 16:26:31 | sean-k-mooney | ill try and swap the jobs over to use those during the week | |
| 16:28:25 | mriedem | gibi: i left comments on https://review.opendev.org/#/c/655113/23 but will withold voting for now | |
| 16:28:37 | mriedem | *withhold | |
| 16:30:18 | mriedem | https://review.opendev.org/#/c/679651/ and https://review.opendev.org/#/c/679653/ are pretty simple if another core can hit those - i'm going to have to rebase the cross-cell-resize series and i'd like to wait until after those are merged since they are going to merge conflict me as well | |
| 16:32:13 | sean-k-mooney | donnyd: are you ok with leaveing the current lable as it is till the end of the week and ill swap over to the new lable at the weekend | |
| 16:32:30 | donnyd | Makes no nevermind to me | |
| 16:32:54 | donnyd | I can leave it intact until asked to refactor the current jobs | |
| 16:56:14 | openstackgerrit | Merged openstack/nova master: objects: Rename 'fields' import to 'obj_fields' https://review.opendev.org/674103 | |
| 16:56:22 | openstackgerrit | Merged openstack/nova master: Nice to have test coverage for If1f465112b8e9b0304b8b5b864b985f72168d839 https://review.opendev.org/679629 | |
| 16:56:31 | openstackgerrit | Merged openstack/nova master: Remove dead code https://review.opendev.org/679668 | |
| 17:11:58 | sean-k-mooney | artom: https://46b5dc7879590352c3ef-b484a2b89d5d2c358f068133dfb2fa14.ssl.cf5.rackcdn.com/679805/2/experimental/nova-nfv-multi-numa-multinode/b040d28/testr_results.html.gz :) that is for https://review.opendev.org/#/c/679805/2 | |
| 17:12:28 | sean-k-mooney | artom: the other two are going to fail due to capasity issue with concurrent tests | |
| 17:12:35 | artom | sean-k-mooney, yeah, I just posted on https://review.opendev.org/#/c/679656/7 summarized the CI success | |
| 17:12:50 | sean-k-mooney | but they did pass some of the tests | |
| 17:13:10 | sean-k-mooney | artom: ok cool | |
| 17:13:14 | artom | dansmith, mriedem, if you guys want to take a look ^^ I'd say https://review.opendev.org/#/c/672595/ is ready | |
| 17:13:27 | artom | Err, https://review.opendev.org/#/c/635669/49 | |
| 17:13:30 | artom | The first patch | |
| 17:15:57 | artom | So there's a failure in https://storage.gra1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/logs_54/679754/2/experimental/nova-nfv-multi-numa-multinode/7956dad/testr_results.html.gz, but it's because it failed to even create the server | |
| 17:16:01 | artom | The other live migration tests passed | |
| 17:16:31 | sean-k-mooney | artom: yep that is due to tempest_concurrency 2 | |
| 17:22:29 | mriedem | artom: the functional test patch is in merge conflict | |
| 17:23:36 | artom | mriedem, I know, working on it, it's https://review.opendev.org/678861 | |
| 17:47:21 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: Introduce live_migration_claim() https://review.opendev.org/635669 | |
| 17:47:22 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: LM: Use Claims to update numa-related XML on the source https://review.opendev.org/635229 | |
| 17:47:22 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: New objects for NUMA live migration https://review.opendev.org/634827 | |
| 17:47:23 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: Deprecate CONF.workarounds.enable_numa_live_migration https://review.opendev.org/640021 | |
| 17:47:23 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: NUMA live migration support https://review.opendev.org/634606 | |
| 17:47:24 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: Functional tests for NUMA live migration https://review.opendev.org/672595 | |
| 17:47:56 | artom | mriedem, ^^ | |
| 17:48:22 | efried | dansmith: Would you please have a look at the bottom two patches of https://review.opendev.org/#/c/678447/ when you get a chance? There's still a bit of work to be done up the series, but stephenfin and I are reviewing that. Want to be able to un-hold and send it once it's baked without having to bug you in emergency mode. | |
| 17:50:32 | openstackgerrit | Merged openstack/nova master: Use SDK for node.list https://review.opendev.org/656027 | |
| 17:50:56 | dansmith | artom: okay so it's passing everything or still failing legit on something? | |
| 17:51:15 | mriedem | landing that vpmems series in ~9 days makes me personally uncomfortable | |
| 17:51:20 | artom | dansmith, passing everything as far as I can tell | |
| 17:51:28 | dansmith | efried: yeah, just got off a call, so let me bio break and then I will | |
| 17:51:33 | dansmith | artom: oh okay, sweet | |
| 17:51:36 | efried | thank you dansmith | |
| 18:17:47 | openstackgerrit | Merged openstack/nova master: Use SDK for validating instance and node https://review.opendev.org/656028 | |
| 18:30:20 | openstackgerrit | Merged openstack/nova master: Use SDK for setting instance id https://review.opendev.org/659690 | |
| 18:37:37 | dansmith | is FF this thursday? | |
| 18:37:55 | mriedem | next | |
| 18:39:05 | dansmith | oh okay cool | |
| 18:41:27 | dansmith | artom: since you rebased it's a little hard to tell...can you summarize what you're now doing to indicate that both ends are new enough to do the numa LM? | |
| 18:42:46 | artom | dansmith, https://review.opendev.org/#/c/634606/70/nova/compute/manager.py@6486 | |
| 18:43:22 | artom | If the destination got a migration flag from the conductor, it sets the numa_live_migration sentinel to True | |
| 18:43:42 | artom | It then passes that sentinel (either True or False) to check_can_live_migrate_source | |
| 18:45:04 | dansmith | is that all? | |
| 18:45:24 | artom | The source compute manager calls the driver's check_can_live_migrate_source - if that driver sets src_supports_numa_live_migration but the numa_live_migration sentinel was False or None, src_supports_numa_live_migration gets removed from migrate_data: https://review.opendev.org/#/c/634606/70/nova/compute/manager.py@6561 | |
| 18:46:13 | artom | Which would then cause the check at https://review.opendev.org/#/c/634606/70/nova/compute/manager.py@6519 back on the destination to be False | |
| 18:46:41 | dansmith | okay, so we've passed a new thing in conductor->destination, proving the conductor's rpc allows it, | |
| 18:46:52 | artom | Exactly | |
| 18:46:59 | dansmith | and then a new thing from destination->source, proving that destination's rpc allows it | |
| 18:47:09 | artom | Well, that's always been the case (conductor -> destination), it was the migration object | |
| 18:47:17 | artom | The new thing is destination -> source | |
| 18:47:17 | dansmith | do we ever make a call from source->destination directly? | |
| 18:47:25 | dansmith | like on rollback? | |
| 18:48:04 | dansmith | if so, I don't think you know that source's config allows the newer version in order to be able to make the newer rollback call right? | |
| 18:48:56 | artom | https://review.opendev.org/#/c/634606/70/nova/compute/manager.py@7417 is the source -> destination call on rollback | |
| 18:49:23 | artom | No, not that | |
| 18:49:35 | dansmith | right, but the source will call to the destination during rollback to make the new drop claim call right? | |
| 18:50:36 | artom | https://review.opendev.org/#/c/634606/70/nova/compute/manager.py@7406 is the drop claim call | |
| 18:50:40 | artom | Which is entirely new in 5.3 | |
| 18:51:38 | dansmith | right, but I don't think you know whether or not the source can call the destination at 5.3 until you need to rollback and make that call right? | |
| 18:52:23 | artom | Isn't that what the check on https://review.opendev.org/#/c/634606/70/nova/compute/manager.py@7405 does? | |
| 18:52:36 | artom | 'dst_numa_info' will not be in migrate_data if conductor, source, or dest is pinned | |
| 18:52:43 | mriedem | if dst_numa_info is in migrate_data which is built up on the dest wouldn't the rollback be safe to assume it can call the drop claim method? | |
| 18:53:03 | dansmith | no, because the source's config is different | |
| 18:53:14 | dansmith | that's the point of what I was asking above, | |
| 18:53:31 | dansmith | you're passing a flag from conductor->destination, and if it makes it to dest, you know conductor is allowed to send the new version | |
| 18:53:45 | dansmith | and you pass a flag from dest->source during the check phase, which proves the dest can send the newer version, | |
| 18:54:12 | dansmith | but if the source hasn't had its config unpinned and restarted, it will be unable to make the drop_claim call during rollback, but you've already claimed on the dest | |
| 18:55:30 | artom | Ohh, I see | |
| 18:55:50 | artom | Before including src_supports_numa_live_migration in migate_data, the sources needs to be sure it can speak 5.3 | |
| 18:55:55 | artom | *source | |
| 18:56:05 | dansmith | right | |