Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2022-02-11
14:49:08 kashyap gibi: So cracked the prob! It's the guest OS indeed - adding a delay helps here?
14:49:15 kashyap s/So/So you/
14:53:17 kashyap I think for now, going with the extra delay before the detach happens is fine. That saves more time here, before the big Tempest series gets merged
14:55:49 gibi kashyap: I don't have brains any more today but next week I can put up a tempest patch with some selective delays. I'm not sure how well QA will appreciate it
14:56:39 gibi also I can take a look at lyarwood's series and try to move that forward
14:56:46 gibi kashyap: thanks you for your help!
15:04:40 frickler gibi: thx for the update, do you know why this only occurs on c9s? is booting slower or did previous libvirts not care whether that release actually happens?
15:19:27 gibi frickler: I think older libvirt let nova to restart the detach process but newer libvirt simply rejectes the retry as the original detach is still ongoing
15:22:29 opendevreview Merged openstack/nova stable/victoria: Add a WA flag waiting for vif-plugged event during reboot https://review.opendev.org/c/openstack/nova/+/818559
15:39:10 sean-k-mooney gibi: older qemu did not support restart the detach but did not raise an error then qemu started enforcing it
15:40:15 gibi sean-k-mooney: yeah, sorry, s/libvirt/qemu.
15:45:10 elodilles melwitt: whenever you have time, could you review this patch? https://review.opendev.org/c/openstack/nova/+/805628
15:45:33 frickler sean-k-mooney: gibi: but then it sounds to me that the real fix would still be to make nova not retry the detach, just wait longer?
15:45:47 elodilles melwitt: i think it would reduce the number of rechecks in wallaby and victoria if it merges (and its devstack part)
15:46:56 gibi frickler: if the detach happens while the cirros is booting then the guest OS never releases the device
15:47:06 gibi so right now waiting more is not an option
15:47:15 gibi but in general I agree to remove the retry loop from nova
15:47:28 gibi as it is pointless after qemu starts rejecting the retry
15:47:49 sean-k-mooney really the jobs should wait for the instance to be pingable/sshable
15:47:53 sean-k-mooney and only detach then
15:48:11 sean-k-mooney and nova shoudl not retry if the detach fails and just have the client retry
15:48:35 sean-k-mooney clieht beign tempest or enduser if a retry is needed
15:50:04 gibi sean-k-mooney: yepp
15:50:40 gibi on the other hand if ever a detach is issue by the client right after a boot then that detach will time out in nova, but I'm not sure it ever time outs in qemu
15:51:15 gibi so in that case a client retry will not help either
15:51:22 sean-k-mooney we might be abel to expicitly cancel the job in qemu
15:51:24 gibi as qemu will say that the detach is in progress
15:51:26 sean-k-mooney when we time out
15:51:57 opendevreview Jonathan Race proposed openstack/nova master: object/notification for Adds Pick guest CPU architecture based on host arch in libvirt driver support https://review.opendev.org/c/openstack/nova/+/828369
15:51:57 opendevreview Jonathan Race proposed openstack/nova master: driver/secheduler/docs for Adds Pick guest CPU architecture based on host arch in libvirt driver support https://review.opendev.org/c/openstack/nova/+/822053
15:51:58 opendevreview Jonathan Race proposed openstack/nova master: zuul-job for Adds Pick guest CPU architecture based on host arch in libvirt driver support https://review.opendev.org/c/openstack/nova/+/828372
15:54:11 gibi sean-k-mooney: at least the doc did not mention a way to cancel via libvirt https://libvirt.org/html/libvirt-libvirt-domain.html#virDomainDetachDeviceFlags
15:55:47 sean-k-mooney we could always issue a detach again :) since that will do an abort in qemu
15:56:22 sean-k-mooney but ya we shoudl aske the libirt folks although im on PTO today so im going to drop off irc again soon
15:56:47 sean-k-mooney so kashyap maybe you could folow up and see if there is a way to abort the detach or pass a timeout to qemu via libfirt
15:57:58 gibi sean-k-mooney: if we attach the detach again qemu reject it
15:58:04 gibi sean-k-mooney: if we issue the detach again qemu reject it
15:58:29 gibi as the pervious one is still ongoing
15:59:08 sean-k-mooney yep
15:59:38 sean-k-mooney but we could catch the error
15:59:58 gibi yepp, but that does not make the device actually detached :D
16:00:01 sean-k-mooney if it say devcice not found well presumabel it finsihed before we sent the detach after the time out
16:00:27 gibi sean-k-mooney: it say detach is ongoing
16:00:44 sean-k-mooney right but the second detach will abort the detach
16:00:56 sean-k-mooney that is the new behavior in qemu
16:01:15 gibi really?
16:01:24 gibi I've only checked the first two detach
16:01:33 gibi so you say the 3rd returns device not found?
16:02:34 sean-k-mooney no
16:02:52 sean-k-mooney im saying the second detach that return "detach is ongoing" cause qemu to abort the detach
16:03:12 sean-k-mooney at lest that is what i was told was the new behavior
16:09:10 gmann gibi: ack, thanks. I will check that tempest patches.
16:10:27 gibi sean-k-mooney: qemu reject each 7 retries with the message the the unplug is in progress https://paste.opendev.org/show/bW5wXCyH5em5tNI34zwV/
16:10:52 gibi sean-k-mooney: I don't think the first detach job was abborted by the second detach
16:11:06 gibi gmann: they are WIP
16:12:09 gibi gmann: how QA would feel about a 20second sleep in the tempest volume detach code? that would be a quick fix compared to the sshable series
16:12:55 sean-k-mooney gibi: huh ok i was told it would but perhaps not.
16:13:25 gibi sean-k-mooney: if there would be a way to abort a detach then we could adapt now to it
16:13:51 gibi anyhow go enjoy your PTO, this will be an open issue on Monday too :)
16:15:15 sean-k-mooney ack im currently trying to decide if i want to use brick or paving slabs in my garden to make paths and beds
16:15:27 gibi nice problem :)
16:16:10 sean-k-mooney ya im half tempted to just go with wood chip since its eaiser but a lot less permenent and i woudl have to do it every year
16:16:43 gibi less permanent mean you can decide next year to replace it with brick or slab :)
16:17:03 sean-k-mooney hehe thats true too
16:17:35 sean-k-mooney also cheaper
16:22:10 gmann gibi: I think we can wait for sshable series as it is hitting only in cenos9-stream
16:22:18 gibi gmann: ack
16:24:52 opendevreview Merged openstack/nova stable/xena: Avoid unbound instance_uuid var during delete https://review.opendev.org/c/openstack/nova/+/816488
16:26:36 opendevreview Balazs Gibizer proposed openstack/nova stable/wallaby: Avoid unbound instance_uuid var during delete https://review.opendev.org/c/openstack/nova/+/828839
16:52:29 opendevreview Jonathan Race proposed openstack/nova master: driver/secheduler/docs for Adds Pick guest CPU architecture based on host arch in libvirt driver support https://review.opendev.org/c/openstack/nova/+/822053
16:52:29 opendevreview Jonathan Race proposed openstack/nova master: zuul-job for Adds Pick guest CPU architecture based on host arch in libvirt driver support https://review.opendev.org/c/openstack/nova/+/828372
18:49:48 melwitt elodilles: done
19:49:51 opendevreview Jonathan Race proposed openstack/nova master: object/notification for Adds Pick guest CPU architecture based on host arch in libvirt driver support https://review.opendev.org/c/openstack/nova/+/828369
19:49:51 opendevreview Jonathan Race proposed openstack/nova master: driver/secheduler/docs for Adds Pick guest CPU architecture based on host arch in libvirt driver support https://review.opendev.org/c/openstack/nova/+/822053
19:49:52 opendevreview Jonathan Race proposed openstack/nova master: zuul-job for Adds Pick guest CPU architecture based on host arch in libvirt driver support https://review.opendev.org/c/openstack/nova/+/828372
#openstack-nova - 2022-02-12
17:11:12 opendevreview Merged openstack/nova stable/train: Ensure MAC addresses characters are in the same case https://review.opendev.org/c/openstack/nova/+/817830
#openstack-nova - 2022-02-13
22:25:08 opendevreview Mauricio Faria de Oliveira proposed openstack/nova stable/victoria: libvirt: disable secure boot on non-q35 or with os secure_boot options https://review.opendev.org/c/openstack/nova/+/828978
22:27:19 opendevreview Mauricio Faria de Oliveira proposed openstack/nova stable/victoria: libvirt: disable secure boot on non-q35 or with os secure_boot options https://review.opendev.org/c/openstack/nova/+/828979
22:28:24 opendevreview Mauricio Faria de Oliveira proposed openstack/nova stable/ussuri: libvirt: disable secure boot on non-q35 or with os secure_boot options https://review.opendev.org/c/openstack/nova/+/828980
#openstack-nova - 2022-02-14
05:07:06 opendevreview Ghanshyam proposed openstack/nova master: Modify remaining APIs as per RBAC new guidelines https://review.opendev.org/c/openstack/nova/+/828994
08:36:54 kashyap gibi: Morning; reading back. Don't worry, you've done a lot more already! Thank _you_ for the quick trials and tests
09:43:11 gibi kashyap: hi!
09:45:36 gibi good luck :)
10:25:09 zigo Would anyone have any idea what's going on with https://bugs.debian.org/1005632 ?
10:56:26 gibi zigo: looking...
10:58:29 zigo gibi: It's likely caused by an update of some Python dependencies in Unstable, but I can't figure it out ... :/
11:04:09 gibi zigo: this seems relevant https://github.com/openstack/python-novaclient/blob/90525a1f5f55e206554da373dda9f26735e7c67f/novaclient/tests/unit/test_shell.py#L586-L599
11:04:21 gibi so I guess it is prettytable
11:04:56 gibi from the debian build log: prettytable==0.0.0
11:05:00 gibi that seems worng :)
11:05:28 gibi but also it says python3-prettytable all 2.5.0-1
11:12:29 gibi it can be that the test code wrongly detects the version of prettytable
11:15:01 gibi dansmith: hi! I've pulled you into a review as I have some ovo / grenade questions https://review.opendev.org/c/openstack/nova/+/828369/7#message-fc72a0dda5b368c23a91f8a6b0244e10b8d511a5
11:25:04 zigo gibi: Let me check what's the result of installing prettytable in Sid and see if the egginfo is wrong.
11:25:13 gibi zigo: ok
11:26:38 zigo Installed version really says 0.0.0 ... :/
11:27:10 zigo So probably a packaging issue (I'm not the maintainer of that one ...).
11:27:41 stephenfin zigo: whenever you see 0.0.0, that usually means pbr hasn't been able to extract the version info from either git or sdist metadata
11:28:10 stephenfin I suspect whoever is building that package _might_ be using plain old tarballs or a shallow clone as opposed to an sdist or full clone
11:28:12 bauzas gmann: when you're up, I'm looking at https://review.opendev.org/c/openstack/nova/+/764292/34

Earlier   Later