| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-09-04 | |||
| 12:03:33 | lyarwood | sean-k-mooney: I'm going to wire up the libvirt events for this and see if they show the disk eventually being detached later on or something | |
| 12:06:35 | openstackgerrit | Lee Yarwood proposed openstack/nova master: WIP/DNM libvirt: Increase device detach wait times https://review.opendev.org/749827 | |
| 12:06:36 | openstackgerrit | Lee Yarwood proposed openstack/nova master: WIP/DNM libvirt: Start emitting DeviceRemovedEvent and DeviceRemovalFailedEvent events https://review.opendev.org/749929 | |
| 12:07:21 | sean-k-mooney | lyarwood: i wonder if this is realted to the pc machine type too | |
| 12:07:38 | sean-k-mooney | lyarwood: we handel pci hotplug differently in q35 | |
| 12:07:48 | sean-k-mooney | the virtio-block interfaces we use by default | |
| 12:08:07 | lyarwood | sean-k-mooney: the machine type should be the same as earlier releases no? | |
| 12:08:07 | sean-k-mooney | all are hotpulged and unpluged when we attach and detach them | |
| 12:08:18 | sean-k-mooney | it is | |
| 12:08:49 | sean-k-mooney | but for the pc machine type we are usin ahci interupt to do the hot unplug for the detach | |
| 12:09:03 | sean-k-mooney | but we use the native pcie support in q35 | |
| 12:09:12 | sean-k-mooney | the ahci appoch is slower and more error prone | |
| 12:09:31 | sean-k-mooney | lyarwood: so im jsut aggreeing really that this coudl be a slow guest | |
| 12:09:37 | lyarwood | ah right | |
| 12:09:38 | sean-k-mooney | or at least its taking a while | |
| 12:09:46 | sean-k-mooney | q35 might mitigate that | |
| 12:09:55 | sean-k-mooney | __might__ | |
| 12:10:00 | sean-k-mooney | its really just a guess | |
| 12:10:17 | lyarwood | it could be that the newer versions of QEMU are consuming more resources on the host leading to instances being slower than previous releases | |
| 12:10:31 | sean-k-mooney | maybe | |
| 12:10:33 | lyarwood | but agreed that q35 should make this better | |
| 12:10:43 | sean-k-mooney | i wonder if the issue would go away if we used virtio-scsi too | |
| 12:11:16 | sean-k-mooney | if nothing else if we confrimed that that could be a workaround for the guest | |
| 12:11:18 | sean-k-mooney | *ci | |
| 12:11:56 | sean-k-mooney | im guessing we need a retry loop or something else to handel it more robustly however | |
| 12:12:03 | sean-k-mooney | if i can help let me know | |
| 12:15:10 | lyarwood | there is a loop at the moment that I'd like to replace with the events listed above instead eventually | |
| 12:15:24 | lyarwood | that still has corner cases but it would be better than this imho | |
| 12:16:48 | sean-k-mooney | ya the events would be better | |
| 12:52:52 | openstackgerrit | Merged openstack/nova stable/ussuri: Add a lock to prevent race during detach/attach of interface https://review.opendev.org/749033 | |
| 13:12:51 | bauzas | dansmith: when you're up, remember me, we can't just change a method name in an o.vo object ? | |
| 13:13:04 | bauzas | because of the compatibility | |
| 13:13:53 | bauzas | dansmith: like, I'd like to rename https://github.com/openstack/nova/blob/master/nova/objects/request_spec.py#L538 | |
| 13:14:13 | bauzas | to be, say, ensure_network_information(ctxt, instance) | |
| 13:14:53 | bauzas | ah no | |
| 13:14:57 | bauzas | now I remember | |
| 13:15:11 | bauzas | unless you say it's a remotable method, you can do it | |
| 13:16:00 | bauzas | like https://github.com/openstack/nova/blob/master/nova/objects/compute_node.py#L256 | |
| 13:16:25 | bauzas | kk, all good | |
| 13:28:17 | dansmith | bauzas: sure you can change a remotable one all you want :) | |
| 13:28:22 | dansmith | er a NON-remotable on | |
| 13:28:30 | bauzas | yup, now I remember | |
| 13:28:53 | bauzas | only decorated methods need to have a major version to be removed or renamed | |
| 13:29:51 | bauzas | that's what happens when you stay too much in libvirt :) | |
| 13:35:41 | dansmith | heh | |
| 13:42:12 | sean-k-mooney | dansmith: added you to this https://review.opendev.org/#/c/747395/ | |
| 13:42:31 | sean-k-mooney | dansmith: its an oslo messigning change to move the heartbeat to a real thread | |
| 13:42:42 | dansmith | oh boy | |
| 13:42:48 | sean-k-mooney | i know you had concens about that and wsgi before | |
| 13:43:06 | sean-k-mooney | mainly breaking the wsgi server thread lifecycle | |
| 13:43:50 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Add type hints to 'nova.virt.libvirt.utils' https://review.opendev.org/742865 | |
| 13:43:51 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Add support for resize and cold migration of emulated TPM files https://review.opendev.org/639934 | |
| 13:47:13 | dansmith | sean-k-mooney: yeah, I just added native threading support to glance's API so that they could do the things they need to do in wsgi mode and it seems to be fine | |
| 13:47:26 | dansmith | dug into what uwsgi expects you to do for threading more so I have a better handle on it | |
| 13:53:50 | sean-k-mooney | ok just wanted to keep you in the loop incase that could cause issues there is a FFE request on the mailing list related to it | |
| 13:58:37 | dansmith | ugh, well, *rushing* that doesn't seem like a good plan | |
| 13:59:05 | dansmith | oh, I see, | |
| 13:59:19 | dansmith | it's already there, just not defaulted | |
| 13:59:37 | dansmith | have we tried turning that on for nova with wsgi api to see if it solves our rabbit heartbeating problem? | |
| 14:00:19 | dansmith | in glance, I avoided the eventlet setup if we were in pure wsgi mode, but I think nova is always doing it right now | |
| 14:05:28 | sean-k-mooney | it is because fo the multi cell scatter gatther | |
| 14:05:46 | sean-k-mooney | before that the api did not use eventlets at all | |
| 14:05:54 | sean-k-mooney | in wsig mode at least | |
| 14:06:27 | sean-k-mooney | it id with the console_srcipt entrypoint by being in the cmd module | |
| 14:06:34 | dansmith | eh? it has always used eventlet for greenthreading right? you mean we didn't call any of the eventlet primitives ourselves before that | |
| 14:06:39 | sean-k-mooney | we monkey patch in __init__.py | |
| 14:06:45 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Support SRIOV interface attach and detach https://review.opendev.org/740995 | |
| 14:06:54 | sean-k-mooney | dansmith: it did not use anyting directly ya | |
| 14:07:19 | dansmith | sure, but otherwise we'd be limited to a single request per worker which'd be pretty limiting | |
| 14:07:21 | sean-k-mooney | but when it was not run by the console_script and was run under uswigi id was not mokeypatched at all in that config | |
| 14:07:32 | dansmith | are you sure? | |
| 14:07:37 | sean-k-mooney | pretty sure | |
| 14:07:53 | sean-k-mooney | i didnt see the monkeypatch call in that code path before | |
| 14:08:08 | dansmith | so we relied on uwsgi spawning enough worker processes to handle any blocking calls we made? that seems unlikely to me | |
| 14:08:12 | sean-k-mooney | i could have missed it but it was not obvious if it was beeing monkey patched | |
| 14:08:40 | dansmith | like any long-running rpc call would block up that worker in the api, so two on a two-worker system wouldn't be able to handle anything else | |
| 14:09:03 | sean-k-mooney | in the api | |
| 14:09:28 | sean-k-mooney | perhaps i just recall we had all those issue that mdbooth tried to fix when we started monkeypatching | |
| 14:09:35 | sean-k-mooney | which seamed to indiecate we were not before | |
| 14:09:54 | dansmith | no, I thought it was just a matter of when we patched | |
| 14:10:22 | sean-k-mooney | perhaps but i tought it was that we did not patch before but now needed to due to the scatter gather | |
| 14:10:43 | sean-k-mooney | in anycase i jsut wnated to get your input on the default change | |
| 14:11:00 | sean-k-mooney | it feel kind of late to chagne the default in oslo | |
| 14:11:13 | sean-k-mooney | deprecating it im more ok with | |
| 14:11:31 | dansmith | well, we might have only noticed that we weren't patching when the cells stuff was added, we weren't really running in real wsgi mode much before that, | |
| 14:11:41 | dansmith | so perhaps we weren't but didn't notice in a devstack that it mattered | |
| 14:12:15 | sean-k-mooney | ya that is more or less my feels on it too | |
| 14:12:37 | dansmith | but yeah, I dunno about changing the default.. especially if it's configureable back, it doesn't seem *that* bad to me | |
| 14:13:11 | sean-k-mooney | i guess we just need to test it and flag it to ooo if we see it causing gate issue | |
| 14:13:20 | sean-k-mooney | or in our donwstream testing | |
| 14:13:34 | dansmith | aye | |
| 14:16:28 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Support SRIOV interface attach and detach https://review.opendev.org/740995 | |
| 14:17:24 | gibi | sean-k-mooney, stephenfin: I finished adding functional tests. I consider this patch ready for review ^^ | |
| 14:19:55 | sean-k-mooney | gibi: cool on downstream call but ill look after | |
| 14:20:02 | gibi | thanks | |
| 14:31:04 | gibi | stephenfin: here is a simple doc patch to light up your Friday https://review.opendev.org/#/c/744492 | |
| 14:48:31 | stephenfin | gibi: done the latter, looking at the former now | |
| 14:49:50 | openstackgerrit | Sylvain Bauza proposed openstack/nova master: WIP: Add a routed networks scheduler pre-filter https://review.opendev.org/749068 | |
| 14:49:51 | openstackgerrit | Sylvain Bauza proposed openstack/nova master: Add requested_networks field to RequestSpec object https://review.opendev.org/749977 | |
| 14:49:57 | bauzas | gibi: sean-k-mooney: ^ routed networks | |