| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-09-04 | |||
| 15:35:58 | kashyap | aspiers: Right; so I can probably put my _guest_machine_type_for_secure_boot() still in utils.py. | |
| 15:36:06 | dansmith | sean-k-mooney: just want to make sure it's not doing numa migration I guess.. there are a couple logs that indicate it's doing the new stuff | |
| 15:36:18 | aspiers | kashyap: now I realise why you were asking that | |
| 15:36:25 | kashyap | aspiers: :-) | |
| 15:36:26 | dansmith | sean-k-mooney: you don't really have to pastebin, I trust you, just check to make sure you don't see those on either side | |
| 15:36:35 | artom | sean-k-mooney, dansmith, presence of LibvirtLiveMigrateNUMAInfo is a big sign | |
| 15:36:46 | dansmith | artom: like this one: https://review.opendev.org/#/c/634606/73/nova/virt/libvirt/driver.py@7465 | |
| 15:36:51 | dansmith | artom: yep | |
| 15:36:51 | aspiers | kashyap: yeah I guess yours could still go in utils.py | |
| 15:37:03 | sean-k-mooney | artom: are you expecting it to regenerat the xml when the RPC version is pinned? | |
| 15:37:12 | artom | We log the migrate_data all over the place, so if that's not in it, we haven't NUMA live-migrated | |
| 15:37:16 | artom | sean-k-mooney, no | |
| 15:37:22 | sean-k-mooney | ok ill check that so | |
| 15:37:30 | mriedem | stephenfin: i see the libvirt driver init_host raises Invalid, but we already have a InvalidConfiguration so that seems more appropriate | |
| 15:37:32 | artom | Any RPC pin on either source, dest or cond should cause the current half-broken behaviour | |
| 15:37:34 | mriedem | it doesn't really matter though | |
| 15:37:35 | kashyap | aspiers: It's a similar method. But in your case, I see you're clubbing it all together with the relevant mem encryption- / SEV-related checks | |
| 15:37:38 | stephenfin | ack | |
| 15:37:48 | sean-k-mooney | the config i have would require the xml to be update to be correct so i shoudl see the xml updated in only 1 direction | |
| 15:38:01 | sean-k-mooney | or not at all | |
| 15:38:25 | aspiers | kashyap: right | |
| 15:39:05 | kashyap | aspiers: I'll just Depends-On yours for the exception class | |
| 15:39:23 | aspiers | kashyap: or just make it the child commit | |
| 15:39:31 | aspiers | kashyap: isn't Depends-On more for cross-repo stuff? | |
| 15:39:55 | sean-k-mooney | artom: dansmith: pinned source => unpinned dest no xml updates on migration | |
| 15:40:26 | kashyap | aspiers: True, Depends-On is indeed normally used for cross-repo things | |
| 15:41:28 | sean-k-mooney | artom: and i get erros in the compute log because the vm xml is invalid for the new host vcpu_pin_set so that is working as we expect | |
| 15:41:53 | artom | "and i get erros in the comput" *panic* "so that is working as we expect" *relief | |
| 15:42:04 | artom | u kill me man | |
| 15:42:35 | sean-k-mooney | its compliing the vm cpu pinning is not valid for the host in the periodici task that caulates resouce usage | |
| 15:42:51 | artom | Which is a known bug I think | |
| 15:42:56 | sean-k-mooney | its compliing the vm cpu pinning is not valid for the host in the periodici task that caulates resouce usage | |
| 15:43:05 | sean-k-mooney | wrong terminal | |
| 15:43:34 | sean-k-mooney | well its not really a bug so much as use say hay you vm pinning is invalid | |
| 15:45:24 | artom | I mean there's a launchpad bug for it somewhere | |
| 15:46:25 | sean-k-mooney | ya there is a patch that sugges we skip updating the resouse if its invalid | |
| 15:46:35 | sean-k-mooney | which i think we both nack'd | |
| 15:47:06 | sean-k-mooney | artom: dansmith same behavior for new source to old dest. no xml update | |
| 15:47:17 | artom | sean-k-mooney, yep | |
| 15:47:20 | dansmith | cool | |
| 15:47:47 | artom | dansmith, well, figured out why my func tests weren't doing the expected thing | |
| 15:47:57 | artom | I was stubbing stuff on the ClientRouter, which is a global singleton | |
| 15:48:06 | artom | So *all* 3 services got capped | |
| 15:48:07 | dansmith | ah yeah | |
| 15:49:18 | artom | I shouldn't hit the same problem if I replace the actual compute_rpcapi objects, though, right? | |
| 15:49:40 | dansmith | you can get access to the managers in the services you start right? | |
| 15:49:48 | dansmith | so manager.rpcapi = $newthing | |
| 15:49:48 | artom | I couldn't figure out how to either cap the objects directly, or create new ones and replace. Looks like I have no choice to do that now. | |
| 15:49:48 | dansmith | I think | |
| 15:50:09 | artom | Yeah, it's "$newthing" that needs thinking about | |
| 15:50:49 | artom | There's no __init__, so how do I cap it? | |
| 15:50:52 | mriedem | compute = self.start_service('compute', host='host1') | |
| 15:50:58 | mriedem | compute.manager.compute_rpcapi = my_stub | |
| 15:51:25 | mriedem | or something like that | |
| 15:51:37 | mriedem | stub the compute_rcpapi on one of the services in the func test | |
| 15:51:43 | dansmith | that ^ | |
| 15:52:23 | dansmith | I think you could also not stub the full rpcapi, and just stub out can_send_version on the one it sets up for you | |
| 15:53:17 | openstackgerrit | Adam Spiers proposed openstack/nova master: Enable booting of libvirt guests with AMD SEV memory encryption https://review.opendev.org/666616 | |
| 15:53:19 | artom | can_send_version is on the client though | |
| 15:53:20 | sean-k-mooney | so ... there is one issue. if we live migrate the server to a host where the pinning is invalide. | |
| 15:53:26 | sean-k-mooney | then we cold migrate it | |
| 15:53:35 | sean-k-mooney | the cold migrtion fails in resize confim | |
| 15:53:58 | artom | sean-k-mooney, is pinning involved, or master to master? | |
| 15:54:08 | sean-k-mooney | yes | |
| 15:54:15 | artom | Yes to...? | |
| 15:54:17 | sean-k-mooney | nova.exception.CP│ | |
| 15:54:19 | sean-k-mooney | UUnpinningUnknown: CPU set to unpin [12, 15, 4, 7] must be a subset of known CPU set [] | |
| 15:54:40 | artom | Wouldn't that be a latent thing? | |
| 15:54:54 | sean-k-mooney | im going to triple check but i think cold migration of a pinned instnace is fine | |
| 15:55:19 | sean-k-mooney | jsut not one that has been live migrtation to a host with out regenrting the xml if the xml is invalid for that host | |
| 15:55:48 | artom | That's still a different bug though, and why we moved to block numa live migration by default? | |
| 15:55:51 | sean-k-mooney | i can check but i think this would happen on master too | |
| 15:57:00 | sean-k-mooney | by the way i was kind of expecting that might fail im just reporing what im finding as i try to break it | |
| 15:57:26 | artom | Totally, this sort of testing is great | |
| 15:57:51 | openstackgerrit | Adam Spiers proposed openstack/nova master: Enable booting of libvirt guests with AMD SEV memory encryption https://review.opendev.org/666616 | |
| 15:58:07 | artom | Wait, would live migrating to an invalid pinning situation go through? | |
| 15:58:07 | openstackgerrit | Merged openstack/nova master: Refactor MigrationTask._execute https://review.opendev.org/679651 | |
| 15:58:17 | openstackgerrit | Merged openstack/nova master: unit test: do not fill rp mapping for failed re-schedule https://review.opendev.org/679653 | |
| 15:58:34 | sean-k-mooney | ok cold migration of a new instance works fine | |
| 15:58:47 | artom | I guess as long as the host CPUs are online, we don't check whether they're part of vcpu_pin_set during the LM | |
| 15:58:59 | sean-k-mooney | ill check reverting to master on the compute node then master only | |
| 15:59:01 | artom | So it'd make sense for that to fail in a subsequent operation | |
| 15:59:46 | sean-k-mooney | artom: right libvirt checks they are are a thing but libvirt is not aware of the vcpu_pin_set | |
| 16:00:10 | sean-k-mooney | and i have intentionally configred theyse hsot so the vcpu_pin_sets on each host does not overlap | |
| 16:00:44 | artom | Right, to make sure the old broken behaviour is happening | |
| 16:01:13 | sean-k-mooney | yes | |
| 16:01:25 | sean-k-mooney | and to give a really hard edgecase | |
| 16:02:05 | aspiers | kashyap: I've just built an image with an experimental kernel and grub secure boot fails due to "invalid signature" | |
| 16:02:16 | aspiers | kashyap: any idea how to tell it the signature is trustworthy? | |
| 16:02:17 | sean-k-mooney | one server is set up to only allow resouce on numa node 0 the other for node 1 only and it have the cpu_shared_set configure to use the opistie numa node to the vcpu_pin_set | |
| 16:03:07 | kashyap | aspiers: Yeah, "experimental kernal" is the keyword :-) | |
| 16:03:55 | sean-k-mooney | aspiers: it has not been sgined with a key that is in the secure boot keystore would be my guess | |
| 16:04:10 | kashyap | So, there is a manual way to make it "trustworthy" | |
| 16:04:27 | kashyap | aspiers: But I don't see a document for it. Last I checked with an admin who does it, said it's quite tedious | |
| 16:05:27 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add delete_on_termination to volume-attach API https://review.opendev.org/673133 | |
| 16:05:35 | sean-k-mooney | dansmith: artom os i have a proablem with 1 node on master and the other node on artoms code | |
| 16:05:56 | sean-k-mooney | on startup the agent on master code is raising "ServiceTooOld: This service is older (v39) than the minimum (v40) version of the rest of the deployment." | |
| 16:06:05 | donnyd | Is there a way to ask nova for a specific trait on an instance without configuring a flavor first? | |
| 16:06:17 | openstackgerrit | Adam Spiers proposed openstack/nova master: Enable booting of libvirt guests with AMD SEV memory encryption https://review.opendev.org/666616 | |
| 16:06:28 | artom | sean-k-mooney, update the database I think | |