| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-07-15 | |||
| 16:01:34 | efried | sean-k-mooney: okay, so back to that thing we were looking at earlier, it looks like setup_networks_on_host is caled in rebuild and resize flows as well as migrate | |
| 16:03:17 | efried | hm, unless teardown=True only happens on migrations... | |
| 16:03:36 | openstackgerrit | Merged openstack/os-resource-classes master: Propose FPGA and PGPU resource classes https://review.opendev.org/657464 | |
| 16:03:42 | efried | does _confirm_resize only happen on migration-y resizes? | |
| 16:04:53 | sean-k-mooney | efried: it happesn on resize or cold migration | |
| 16:05:22 | sean-k-mooney | in both cases we go into resize_verify and you chave to do reseize --confim | |
| 16:05:25 | efried | rats, nother meeting, will come back to this... | |
| 16:06:01 | sean-k-mooney | efried: stephenfin is adding an openstack server migrate --confirm as synatactic sugar but its the same call underneath | |
| 16:07:14 | stephenfin | that threw me too, fwiw | |
| 16:07:32 | stephenfin | (whether cold migrations needed to be confirmed or not) | |
| 16:07:51 | sean-k-mooney | stephenfin: well that depends on your config settings | |
| 16:07:58 | sean-k-mooney | but ya | |
| 16:08:40 | sean-k-mooney | i hit that oddity back in hevana so at this point i dont even think about them differently anymore. | |
| 16:08:48 | sean-k-mooney | (cold migraton vs resize) | |
| 17:17:36 | bbobrov | sean-k-mooney: hey | |
| 17:17:46 | bbobrov | sean-k-mooney: regarding your comment to https://review.opendev.org/#/c/638680/24/nova/virt/libvirt/driver.py | |
| 17:18:09 | bbobrov | sean-k-mooney: do you have a traceback how it blows up? | |
| 17:18:34 | sean-k-mooney | i have fixed it in https://review.opendev.org/#/c/670189/ | |
| 17:19:15 | sean-k-mooney | bbobrov: basicaly the old code would use q35 if there was not default for an archatecutre | |
| 17:20:04 | sean-k-mooney | which is invalid if you have sparc or many other arch specific qemu-* packages installed | |
| 17:20:28 | sean-k-mooney | so libvirt would raise an exception saying q35 is not supported by emulator X | |
| 17:21:25 | bbobrov | sean-k-mooney: understood, thanks. Don't we have a ci job to catch this kind of stuff? | |
| 17:21:59 | sean-k-mooney | this was not caught as this is currently not used outside of the tests and the test mock out all calls to libvirt | |
| 17:22:08 | sean-k-mooney | so it was passing because of invalid test data | |
| 17:22:43 | sean-k-mooney | if you tried to call that finciton in the agent then you got tracebacks | |
| 17:23:13 | sean-k-mooney | on ubunutu 18.04 they package all the emultor by defualt and install them when you install qemu and qemu-kvm | |
| 17:23:30 | bbobrov | ok, thanks. I'll review 670189 then | |
| 17:23:31 | sean-k-mooney | so i would have expect this to break but only when we added code to call this | |
| 17:25:02 | sean-k-mooney | i think the signiture of get_domain_capabilities is proably not what we want but i have fixed up the function without changing it for now to not break the sev code an allow me to contue with my own work | |
| 17:26:24 | sean-k-mooney | the api signature of _get_domain_capabilities is actully more useful IMO. | |
| 17:26:41 | bbobrov | sean-k-mooney: how should it look then? Maybe i could quickly fix the sev code | |
| 17:26:57 | bbobrov | sean-k-mooney: and rebase on top of the fix | |
| 17:27:33 | sean-k-mooney | i think we should be able to optionally be able to pass the arch and machine type to it and have it query them if its not cached | |
| 17:27:36 | sean-k-mooney | https://review.opendev.org/#/c/670189/3/nova/virt/libvirt/host.py@674 | |
| 17:27:58 | sean-k-mooney | have it default to arch=None mtype=None and retrun all of them | |
| 17:28:27 | sean-k-mooney | but it depend on how we will be using it | |
| 17:28:37 | sean-k-mooney | if we look up data by the arch only then its fine | |
| 17:29:01 | sean-k-mooney | if we look it up by arch and mtype there is no gurantee the combination will be in the dict | |
| 17:30:02 | sean-k-mooney | i can work with the api as it is for now as i just need it to create traits | |
| 17:30:19 | sean-k-mooney | but if i was using this in the driver the current interface would be limiting | |
| 17:31:36 | sean-k-mooney | ideally the libvirt driver would never need to call _get_domain_capabilities directly but since the mtype can be set in the image its possible that you will not find in in the cached copy and would need too | |
| 18:40:34 | efried | bbobrov: Hi, since you're here, are you working on rebasing the SEV series? | |
| 19:03:00 | openstackgerrit | Boris Bobrov proposed openstack/nova master: Provide HW_CPU_X86_AMD_SEV trait when SEV is supported https://review.opendev.org/638680 | |
| 19:03:01 | openstackgerrit | Boris Bobrov proposed openstack/nova master: Extract SEV-specific bits on host detection https://review.opendev.org/636334 | |
| 19:03:01 | openstackgerrit | Boris Bobrov proposed openstack/nova master: Add extra spec parameter and image property for memory encryption https://review.opendev.org/664420 | |
| 19:03:02 | openstackgerrit | Boris Bobrov proposed openstack/nova master: Apply SEV-specific guest config when SEV is required https://review.opendev.org/644565 | |
| 19:03:02 | openstackgerrit | Boris Bobrov proposed openstack/nova master: Add |
|
| 19:03:03 | openstackgerrit | Boris Bobrov proposed openstack/nova master: Enable booting of libvirt guests with AMD SEV memory encryption https://review.opendev.org/666616 | |
| 19:03:53 | bbobrov | efried: here is the answer :) | |
| 19:04:00 | efried | blam! | |
| 19:04:03 | efried | thanks bbobrov | |
| 19:04:19 | bbobrov | i will reply to the comments now | |
| 19:04:56 | artom | What happened to aspiers? | |
| 19:21:41 | efried | was wondering same | |
| 19:35:04 | openstackgerrit | Eric Fried proposed openstack/nova master: Use Adapter global_request_id kwarg https://review.opendev.org/670907 | |
| 19:35:15 | openstack | Launchpad bug 1835958 in OpenStack Compute (nova) "Nova sync power state on large clusters causes poor performance" [Undecided,New] | |
| 19:35:15 | cdent | efried: the semaphore thing is indirectly related to https://bugs.launchpad.net/nova/+bug/1835958 (power states not under the same lock, but in the realm of nova-compute performance) | |
| 19:37:04 | efried | cdent: Only slightly related, I've been talking to alex_xu about moving more stuff under that semaphore. | |
| 19:37:16 | efried | specifically driver-specific claim stuff | |
| 19:37:21 | efried | Not sure if you were around for those conversations. | |
| 19:37:33 | cdent | not that I'm aware of | |
| 19:38:33 | cdent | it's a problem for high throughput nova-computes (like in vmware where the nova-compute is a chokepoint, rather than a member of a nice bit of parallelism like in a big kvm cloud) | |
| 19:40:46 | efried | cdent: well, the plan is to start delegating virt-specific claimage to the virt driver itself. I could see where vmware could take its own semaphore on a specific internal nodeything and background the real claim job, so the next one could come in and get started (but on a different nodeything). | |
| 19:40:53 | efried | and it would be none of RT's business. | |
| 19:41:52 | efried | the more we move claimables into placement, the more we can delegate the claim logic for same to the virt driver. | |
| 19:42:05 | efried | so e.g. PCI devices would eventually become virt driver business. | |
| 19:42:23 | efried | and - as we were discussing this morning - not even stored in the db anymore. | |
| 19:44:05 | cdent | well that would be lovely | |
| 19:57:18 | openstack | Launchpad bug 1836642 in OpenStack Compute (nova) "Metadata responses are very slow sometimes" [Undecided,New] | |
| 19:57:18 | slaweq | hi, is there any n-meta-api service expert here? Can You take a look at https://bugs.launchpad.net/neutron/+bug/1836642 from Nova PoV? Thx in advance :) | |
| 20:01:06 | openstack | bug 1836595 in neutron "test_server_connectivity_cold_migration_revert failing" [Undecided,New] https://launchpad.net/bugs/1836595 | |
| 20:01:06 | efried | artom: catching up, are you filing an elastic-recheck profile for bug 1836595 ? | |
| 20:02:10 | artom | efried, I could - should I be? :) | |
| 20:02:48 | artom | efried, I'm kinda hoping https://review.opendev.org/#/c/670848/ would merge though | |
| 20:03:42 | artom | Speaking of which, gmann ^^ :) | |
| 20:03:51 | artom | (If you're around, not sure about your TZ) | |
| 20:05:11 | efried | artom: This is interesting, the failure on that skip patch http://logs.openstack.org/48/670848/1/check/neutron-tempest-dvr/ed2b81c/testr_results.html.gz | |
| 20:06:19 | artom | efried, looks like feral packets to me | |
| 20:06:26 | efried | conflict deleting allocations, looks like it's tempest itself that's being a placement client. Without having looked at the code, one wonders whether it should be retrying | |
| 20:06:52 | efried | feral packets? how so? | |
| 20:06:53 | artom | Yeah, tempest makes a point not to use any Python clients | |
| 20:07:09 | artom | efried, heh, a lazy way of saying "something unrelated I can't be arsed to debug" :P | |
| 20:07:22 | artom | So it'll do the GET requests itself | |
| 20:07:27 | efried | there is no placement client, my point is that this isn't tempest calling something in nova that's talking to placement and getting a conflict. | |
| 20:07:39 | efried | this is tempest talking directly to placement | |
| 20:07:49 | artom | And that's bad? | |
| 20:08:14 | efried | it means there's likely an opportunity to harden the tempest code so this failure doesn't happen anymore. | |
| 20:08:59 | artom | *facepalm*, oh just got what you're saying | |
| 20:09:11 | artom | Tempest shouldn't be deleting allocations itself, Nova should be doing it | |
| 20:09:47 | artom | I think that response is from Nova though | |
| 20:11:29 | efried | mmm, yes, it looks like you're right. | |
| 20:11:54 | artom | That test is really confusing | |
| 20:13:16 | efried | if the error is coming from nova, it means we have a bug in nova. | |
| 20:16:08 | artom | Well, looks like tempest is creating a server, not waiting for it to become ACTIVE, then immediately deleting it | |
| 20:16:33 | artom | So I suspect that makes Nova hit a race between the build process and the delete request | |
| 20:16:49 | cdent | artom: that sounds right | |
| 20:17:00 | artom | Short term that can be "fixed" by making Tempest wait | |
| 20:17:22 | artom | Longer term I guess we'll need to stick a lock in Nova somewhere | |
| 20:18:49 | cdent | placement log entries are near http://logs.openstack.org/48/670848/1/check/neutron-tempest-dvr/ed2b81c/controller/logs/screen-placement-api.txt.gz#_Jul_15_17_27_35_283720 | |
| 20:20:03 | cdent | handling for the req id starts here http://logs.openstack.org/48/670848/1/check/neutron-tempest-dvr/ed2b81c/controller/logs/screen-n-cpu.txt.gz#_Jul_15_17_27_33_968264 | |