| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-07-15 | |||
| 17:20:04 | sean-k-mooney | which is invalid if you have sparc or many other arch specific qemu-* packages installed | |
| 17:20:28 | sean-k-mooney | so libvirt would raise an exception saying q35 is not supported by emulator X | |
| 17:21:25 | bbobrov | sean-k-mooney: understood, thanks. Don't we have a ci job to catch this kind of stuff? | |
| 17:21:59 | sean-k-mooney | this was not caught as this is currently not used outside of the tests and the test mock out all calls to libvirt | |
| 17:22:08 | sean-k-mooney | so it was passing because of invalid test data | |
| 17:22:43 | sean-k-mooney | if you tried to call that finciton in the agent then you got tracebacks | |
| 17:23:13 | sean-k-mooney | on ubunutu 18.04 they package all the emultor by defualt and install them when you install qemu and qemu-kvm | |
| 17:23:30 | bbobrov | ok, thanks. I'll review 670189 then | |
| 17:23:31 | sean-k-mooney | so i would have expect this to break but only when we added code to call this | |
| 17:25:02 | sean-k-mooney | i think the signiture of get_domain_capabilities is proably not what we want but i have fixed up the function without changing it for now to not break the sev code an allow me to contue with my own work | |
| 17:26:24 | sean-k-mooney | the api signature of _get_domain_capabilities is actully more useful IMO. | |
| 17:26:41 | bbobrov | sean-k-mooney: how should it look then? Maybe i could quickly fix the sev code | |
| 17:26:57 | bbobrov | sean-k-mooney: and rebase on top of the fix | |
| 17:27:33 | sean-k-mooney | i think we should be able to optionally be able to pass the arch and machine type to it and have it query them if its not cached | |
| 17:27:36 | sean-k-mooney | https://review.opendev.org/#/c/670189/3/nova/virt/libvirt/host.py@674 | |
| 17:27:58 | sean-k-mooney | have it default to arch=None mtype=None and retrun all of them | |
| 17:28:27 | sean-k-mooney | but it depend on how we will be using it | |
| 17:28:37 | sean-k-mooney | if we look up data by the arch only then its fine | |
| 17:29:01 | sean-k-mooney | if we look it up by arch and mtype there is no gurantee the combination will be in the dict | |
| 17:30:02 | sean-k-mooney | i can work with the api as it is for now as i just need it to create traits | |
| 17:30:19 | sean-k-mooney | but if i was using this in the driver the current interface would be limiting | |
| 17:31:36 | sean-k-mooney | ideally the libvirt driver would never need to call _get_domain_capabilities directly but since the mtype can be set in the image its possible that you will not find in in the cached copy and would need too | |
| 18:40:34 | efried | bbobrov: Hi, since you're here, are you working on rebasing the SEV series? | |
| 19:03:00 | openstackgerrit | Boris Bobrov proposed openstack/nova master: Provide HW_CPU_X86_AMD_SEV trait when SEV is supported https://review.opendev.org/638680 | |
| 19:03:01 | openstackgerrit | Boris Bobrov proposed openstack/nova master: Add extra spec parameter and image property for memory encryption https://review.opendev.org/664420 | |
| 19:03:01 | openstackgerrit | Boris Bobrov proposed openstack/nova master: Extract SEV-specific bits on host detection https://review.opendev.org/636334 | |
| 19:03:02 | openstackgerrit | Boris Bobrov proposed openstack/nova master: Add |
|
| 19:03:02 | openstackgerrit | Boris Bobrov proposed openstack/nova master: Apply SEV-specific guest config when SEV is required https://review.opendev.org/644565 | |
| 19:03:03 | openstackgerrit | Boris Bobrov proposed openstack/nova master: Enable booting of libvirt guests with AMD SEV memory encryption https://review.opendev.org/666616 | |
| 19:03:53 | bbobrov | efried: here is the answer :) | |
| 19:04:00 | efried | blam! | |
| 19:04:03 | efried | thanks bbobrov | |
| 19:04:19 | bbobrov | i will reply to the comments now | |
| 19:04:56 | artom | What happened to aspiers? | |
| 19:21:41 | efried | was wondering same | |
| 19:35:04 | openstackgerrit | Eric Fried proposed openstack/nova master: Use Adapter global_request_id kwarg https://review.opendev.org/670907 | |
| 19:35:15 | cdent | efried: the semaphore thing is indirectly related to https://bugs.launchpad.net/nova/+bug/1835958 (power states not under the same lock, but in the realm of nova-compute performance) | |
| 19:35:15 | openstack | Launchpad bug 1835958 in OpenStack Compute (nova) "Nova sync power state on large clusters causes poor performance" [Undecided,New] | |
| 19:37:04 | efried | cdent: Only slightly related, I've been talking to alex_xu about moving more stuff under that semaphore. | |
| 19:37:16 | efried | specifically driver-specific claim stuff | |
| 19:37:21 | efried | Not sure if you were around for those conversations. | |
| 19:37:33 | cdent | not that I'm aware of | |
| 19:38:33 | cdent | it's a problem for high throughput nova-computes (like in vmware where the nova-compute is a chokepoint, rather than a member of a nice bit of parallelism like in a big kvm cloud) | |
| 19:40:46 | efried | cdent: well, the plan is to start delegating virt-specific claimage to the virt driver itself. I could see where vmware could take its own semaphore on a specific internal nodeything and background the real claim job, so the next one could come in and get started (but on a different nodeything). | |
| 19:40:53 | efried | and it would be none of RT's business. | |
| 19:41:52 | efried | the more we move claimables into placement, the more we can delegate the claim logic for same to the virt driver. | |
| 19:42:05 | efried | so e.g. PCI devices would eventually become virt driver business. | |
| 19:42:23 | efried | and - as we were discussing this morning - not even stored in the db anymore. | |
| 19:44:05 | cdent | well that would be lovely | |
| 19:57:18 | slaweq | hi, is there any n-meta-api service expert here? Can You take a look at https://bugs.launchpad.net/neutron/+bug/1836642 from Nova PoV? Thx in advance :) | |
| 19:57:18 | openstack | Launchpad bug 1836642 in OpenStack Compute (nova) "Metadata responses are very slow sometimes" [Undecided,New] | |
| 20:01:06 | efried | artom: catching up, are you filing an elastic-recheck profile for bug 1836595 ? | |
| 20:01:06 | openstack | bug 1836595 in neutron "test_server_connectivity_cold_migration_revert failing" [Undecided,New] https://launchpad.net/bugs/1836595 | |
| 20:02:10 | artom | efried, I could - should I be? :) | |
| 20:02:48 | artom | efried, I'm kinda hoping https://review.opendev.org/#/c/670848/ would merge though | |
| 20:03:42 | artom | Speaking of which, gmann ^^ :) | |
| 20:03:51 | artom | (If you're around, not sure about your TZ) | |
| 20:05:11 | efried | artom: This is interesting, the failure on that skip patch http://logs.openstack.org/48/670848/1/check/neutron-tempest-dvr/ed2b81c/testr_results.html.gz | |
| 20:06:19 | artom | efried, looks like feral packets to me | |
| 20:06:26 | efried | conflict deleting allocations, looks like it's tempest itself that's being a placement client. Without having looked at the code, one wonders whether it should be retrying | |
| 20:06:52 | efried | feral packets? how so? | |
| 20:06:53 | artom | Yeah, tempest makes a point not to use any Python clients | |
| 20:07:09 | artom | efried, heh, a lazy way of saying "something unrelated I can't be arsed to debug" :P | |
| 20:07:22 | artom | So it'll do the GET requests itself | |
| 20:07:27 | efried | there is no placement client, my point is that this isn't tempest calling something in nova that's talking to placement and getting a conflict. | |
| 20:07:39 | efried | this is tempest talking directly to placement | |
| 20:07:49 | artom | And that's bad? | |
| 20:08:14 | efried | it means there's likely an opportunity to harden the tempest code so this failure doesn't happen anymore. | |
| 20:08:59 | artom | *facepalm*, oh just got what you're saying | |
| 20:09:11 | artom | Tempest shouldn't be deleting allocations itself, Nova should be doing it | |
| 20:09:47 | artom | I think that response is from Nova though | |
| 20:11:29 | efried | mmm, yes, it looks like you're right. | |
| 20:11:54 | artom | That test is really confusing | |
| 20:13:16 | efried | if the error is coming from nova, it means we have a bug in nova. | |
| 20:16:08 | artom | Well, looks like tempest is creating a server, not waiting for it to become ACTIVE, then immediately deleting it | |
| 20:16:33 | artom | So I suspect that makes Nova hit a race between the build process and the delete request | |
| 20:16:49 | cdent | artom: that sounds right | |
| 20:17:00 | artom | Short term that can be "fixed" by making Tempest wait | |
| 20:17:22 | artom | Longer term I guess we'll need to stick a lock in Nova somewhere | |
| 20:18:49 | cdent | placement log entries are near http://logs.openstack.org/48/670848/1/check/neutron-tempest-dvr/ed2b81c/controller/logs/screen-placement-api.txt.gz#_Jul_15_17_27_35_283720 | |
| 20:20:03 | cdent | handling for the req id starts here http://logs.openstack.org/48/670848/1/check/neutron-tempest-dvr/ed2b81c/controller/logs/screen-n-cpu.txt.gz#_Jul_15_17_27_33_968264 | |
| 20:20:18 | artom | Actually from the little I know about placement it should handle those kinds of races with the generation thing, right? | |
| 20:20:27 | artom | So maybe Nova just needs to handle Placement error better | |
| 20:20:28 | efried | the generation thing is exactly what's happening here | |
| 20:20:55 | efried | alloc deletion was specifically set up so that you couldn't race deletion with some other consumer op | |
| 20:21:24 | efried | so this guard is doing exactly what it's supposed to, and it's the overarching operation that's broken. As you say, trying to delete while creating. | |
| 20:21:30 | cdent | "instance disappeared during build" http://logs.openstack.org/48/670848/1/check/neutron-tempest-dvr/ed2b81c/controller/logs/screen-n-cpu.txt.gz#_Jul_15_17_27_34_806925 | |
| 20:21:49 | efried | only question is where to fix it. | |
| 20:22:24 | efried | it would be possible to redrive the alloc delete until it works. But that's still potentially racy if the other op is trying to create the alloc at the same time. | |
| 20:23:24 | efried | there's a couple races here actually. | |
| 20:23:29 | artom | @instance_state(ACTIVE) for delete :D | |
| 20:23:46 | efried | The other one is if the deletion goes through before the allocation is even created | |
| 20:25:12 | efried | it will not raise an exception (it returns False, but the caller doesn't check that) so we'll end up with a leaked allocation | |
| 20:26:15 | cdent | efried, artom if you end up creating a bug about this, please let me know what it is so I can follow along, tomorrow | |
| 20:27:29 | artom | efried, mind doing it? You have a better grasp of that stuff, and I'm off in 30 minutes anyways, daycare duty | |
| 20:27:46 | efried | ack | |
| 20:48:57 | artom | *snerk* | |
| 20:49:00 | artom | https://bugs.launchpad.net/nova/+bug/1836204 | |
| 20:49:00 | openstack | Launchpad bug 1836204 in OpenStack Compute (nova) "The allocation of VGPU has race problem" [High,Triaged] - Assigned to Alex Xu (xuhj) | |
| 20:49:08 | artom | I guess it hates blacks | |