| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-09-04 | |||
| 09:23:12 | brinzhang | very thankful | |
| 09:23:45 | gibi | please note that every project has its own numbering as they had their first release at different cycle | |
| 09:23:56 | gibi | and also in the past openstack used different tag names | |
| 09:25:51 | brinzhang | yeah, I think https://releases.openstack.org/ussuri is correctly | |
| 09:26:25 | brinzhang | just need to change the realse name | |
| 09:26:35 | brinzhang | just need to change the realse name in the url | |
| 09:43:54 | brinzhang | gibi: I find we dont necessary to checkout to the stable/ussuri, in master branch, we also can get the bug | |
| 09:44:17 | brinzhang | of course, we should point the correct branch number | |
| 09:44:55 | gibi | brinzhang: yes, git allows to query the tree without checking out anything | |
| 09:45:08 | brinzhang | \o | |
| 11:11:48 | noonedeadpunk | folks, need some help with figuring out the best way to separate hosts. Let me describe scenario | |
| 11:13:29 | noonedeadpunk | We have group of hosts that have ephemeral storages saved on NFS (/var/lib/nova/instances eventually). And we have another group of hosts with ceph. And we want all new VMs to be created on these ceph hosts, while leave old ones to be operating normally for a while | |
| 11:14:29 | noonedeadpunk | I'm aware about isolate aggregates, but then we will need to hide all current flavors, set some metadata on them, and create new set of flavors | |
| 11:14:51 | noonedeadpunk | Which will result that ppl won't be able to do retype | |
| 11:15:25 | noonedeadpunk | Another thing that come up to mind were AZ. But again, newly created VMs won't be guaranteed not to move to old AZ | |
| 11:16:15 | noonedeadpunk | Didn't look into cells yet tbh, but we eventually want to get rid of first AZ pretty soon, so don't want to overcomplicate things | |
| 11:19:15 | noonedeadpunk | Not usre if setting default_schedule_zone will set AZ for all newly created instances | |
| 11:19:53 | noonedeadpunk | Ok, found it, disregard please:) | |
| 11:19:59 | noonedeadpunk | https://docs.openstack.org/nova/latest/admin/availability-zones.html#implications-for-moving-servers | |
| 11:36:30 | lyarwood | sean-k-mooney: think I've finally reproduced the focal detach issue btw | |
| 11:36:57 | lyarwood | sean-k-mooney: had to do a complete tempest run on a small 8gb devstack instance | |
| 11:50:23 | sean-k-mooney | oh ok | |
| 11:51:04 | sean-k-mooney | any idea what is the cause | |
| 11:51:16 | sean-k-mooney | lyarwood: ^ | |
| 11:59:08 | lyarwood | sean-k-mooney: still working my way through the logs, honestly just feels like the guestOS is just too slow in detaching the device | |
| 12:03:33 | lyarwood | sean-k-mooney: I'm going to wire up the libvirt events for this and see if they show the disk eventually being detached later on or something | |
| 12:06:35 | openstackgerrit | Lee Yarwood proposed openstack/nova master: WIP/DNM libvirt: Increase device detach wait times https://review.opendev.org/749827 | |
| 12:06:36 | openstackgerrit | Lee Yarwood proposed openstack/nova master: WIP/DNM libvirt: Start emitting DeviceRemovedEvent and DeviceRemovalFailedEvent events https://review.opendev.org/749929 | |
| 12:07:21 | sean-k-mooney | lyarwood: i wonder if this is realted to the pc machine type too | |
| 12:07:38 | sean-k-mooney | lyarwood: we handel pci hotplug differently in q35 | |
| 12:07:48 | sean-k-mooney | the virtio-block interfaces we use by default | |
| 12:08:07 | lyarwood | sean-k-mooney: the machine type should be the same as earlier releases no? | |
| 12:08:07 | sean-k-mooney | all are hotpulged and unpluged when we attach and detach them | |
| 12:08:18 | sean-k-mooney | it is | |
| 12:08:49 | sean-k-mooney | but for the pc machine type we are usin ahci interupt to do the hot unplug for the detach | |
| 12:09:03 | sean-k-mooney | but we use the native pcie support in q35 | |
| 12:09:12 | sean-k-mooney | the ahci appoch is slower and more error prone | |
| 12:09:31 | sean-k-mooney | lyarwood: so im jsut aggreeing really that this coudl be a slow guest | |
| 12:09:37 | lyarwood | ah right | |
| 12:09:38 | sean-k-mooney | or at least its taking a while | |
| 12:09:46 | sean-k-mooney | q35 might mitigate that | |
| 12:09:55 | sean-k-mooney | __might__ | |
| 12:10:00 | sean-k-mooney | its really just a guess | |
| 12:10:17 | lyarwood | it could be that the newer versions of QEMU are consuming more resources on the host leading to instances being slower than previous releases | |
| 12:10:31 | sean-k-mooney | maybe | |
| 12:10:33 | lyarwood | but agreed that q35 should make this better | |
| 12:10:43 | sean-k-mooney | i wonder if the issue would go away if we used virtio-scsi too | |
| 12:11:16 | sean-k-mooney | if nothing else if we confrimed that that could be a workaround for the guest | |
| 12:11:18 | sean-k-mooney | *ci | |
| 12:11:56 | sean-k-mooney | im guessing we need a retry loop or something else to handel it more robustly however | |
| 12:12:03 | sean-k-mooney | if i can help let me know | |
| 12:15:10 | lyarwood | there is a loop at the moment that I'd like to replace with the events listed above instead eventually | |
| 12:15:24 | lyarwood | that still has corner cases but it would be better than this imho | |
| 12:16:48 | sean-k-mooney | ya the events would be better | |
| 12:52:52 | openstackgerrit | Merged openstack/nova stable/ussuri: Add a lock to prevent race during detach/attach of interface https://review.opendev.org/749033 | |
| 13:12:51 | bauzas | dansmith: when you're up, remember me, we can't just change a method name in an o.vo object ? | |
| 13:13:04 | bauzas | because of the compatibility | |
| 13:13:53 | bauzas | dansmith: like, I'd like to rename https://github.com/openstack/nova/blob/master/nova/objects/request_spec.py#L538 | |
| 13:14:13 | bauzas | to be, say, ensure_network_information(ctxt, instance) | |
| 13:14:53 | bauzas | ah no | |
| 13:14:57 | bauzas | now I remember | |
| 13:15:11 | bauzas | unless you say it's a remotable method, you can do it | |
| 13:16:00 | bauzas | like https://github.com/openstack/nova/blob/master/nova/objects/compute_node.py#L256 | |
| 13:16:25 | bauzas | kk, all good | |
| 13:28:17 | dansmith | bauzas: sure you can change a remotable one all you want :) | |
| 13:28:22 | dansmith | er a NON-remotable on | |
| 13:28:30 | bauzas | yup, now I remember | |
| 13:28:53 | bauzas | only decorated methods need to have a major version to be removed or renamed | |
| 13:29:51 | bauzas | that's what happens when you stay too much in libvirt :) | |
| 13:35:41 | dansmith | heh | |
| 13:42:12 | sean-k-mooney | dansmith: added you to this https://review.opendev.org/#/c/747395/ | |
| 13:42:31 | sean-k-mooney | dansmith: its an oslo messigning change to move the heartbeat to a real thread | |
| 13:42:42 | dansmith | oh boy | |
| 13:42:48 | sean-k-mooney | i know you had concens about that and wsgi before | |
| 13:43:06 | sean-k-mooney | mainly breaking the wsgi server thread lifecycle | |
| 13:43:50 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Add type hints to 'nova.virt.libvirt.utils' https://review.opendev.org/742865 | |
| 13:43:51 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Add support for resize and cold migration of emulated TPM files https://review.opendev.org/639934 | |
| 13:47:13 | dansmith | sean-k-mooney: yeah, I just added native threading support to glance's API so that they could do the things they need to do in wsgi mode and it seems to be fine | |
| 13:47:26 | dansmith | dug into what uwsgi expects you to do for threading more so I have a better handle on it | |
| 13:53:50 | sean-k-mooney | ok just wanted to keep you in the loop incase that could cause issues there is a FFE request on the mailing list related to it | |
| 13:58:37 | dansmith | ugh, well, *rushing* that doesn't seem like a good plan | |
| 13:59:05 | dansmith | oh, I see, | |
| 13:59:19 | dansmith | it's already there, just not defaulted | |
| 13:59:37 | dansmith | have we tried turning that on for nova with wsgi api to see if it solves our rabbit heartbeating problem? | |
| 14:00:19 | dansmith | in glance, I avoided the eventlet setup if we were in pure wsgi mode, but I think nova is always doing it right now | |
| 14:05:28 | sean-k-mooney | it is because fo the multi cell scatter gatther | |
| 14:05:46 | sean-k-mooney | before that the api did not use eventlets at all | |
| 14:05:54 | sean-k-mooney | in wsig mode at least | |
| 14:06:27 | sean-k-mooney | it id with the console_srcipt entrypoint by being in the cmd module | |
| 14:06:34 | dansmith | eh? it has always used eventlet for greenthreading right? you mean we didn't call any of the eventlet primitives ourselves before that | |
| 14:06:39 | sean-k-mooney | we monkey patch in __init__.py | |
| 14:06:45 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Support SRIOV interface attach and detach https://review.opendev.org/740995 | |
| 14:06:54 | sean-k-mooney | dansmith: it did not use anyting directly ya | |
| 14:07:19 | dansmith | sure, but otherwise we'd be limited to a single request per worker which'd be pretty limiting | |
| 14:07:21 | sean-k-mooney | but when it was not run by the console_script and was run under uswigi id was not mokeypatched at all in that config | |
| 14:07:32 | dansmith | are you sure? | |
| 14:07:37 | sean-k-mooney | pretty sure | |
| 14:07:53 | sean-k-mooney | i didnt see the monkeypatch call in that code path before | |
| 14:08:08 | dansmith | so we relied on uwsgi spawning enough worker processes to handle any blocking calls we made? that seems unlikely to me | |
| 14:08:12 | sean-k-mooney | i could have missed it but it was not obvious if it was beeing monkey patched | |