Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2021-10-27
14:21:35 bauzas gibi: sorry you're maybe right
14:21:42 gibi what I see is that in the bug they evacuate instances without restarting the compute node
14:21:57 bauzas I originally thought this was about evacuate/evacuateback/evacuate
14:22:33 bauzas adding a comment
14:23:00 gibi so far we said that you can only evacuate if you make sure that the compute is dead
14:23:52 gibi in the bug case the compute was halted / stuck, the heartbeat was missing so the service was considered down, nova allowed evacuation, then the compute recovered without the nova-compute service restarted
14:24:10 gibi nova-compute only cleans up evacuated instance during init_host but does not do it periodically
14:24:45 gibi so in this case the evacuated instance was not cleaned up on the source leading to duplicated instances causing corruption
14:25:55 gibi option a) change nova-compute to clean up evacuated instance in a periodic
14:26:37 gibi option b) change the evac API to only allow evacuation if the compute is forced down (meaning the admin mades sure the host is fenced)
14:27:14 gibi option c) declare the current bug as user error as the nova-compute was not restarted as part of the compute node recovery
14:33:29 bauzas gibi: I wrote a large comment on the bvug
14:33:51 bauzas I think CERN is triggering evacuations before verifying the host status
14:34:32 sean-k-mooney bauzas: i dont think we talked baout this either
14:34:33 bauzas as I said, I feel healthchecks can help them getting a better decision-making about whether they need to evacuate or not
14:34:49 sean-k-mooney bauzas: we talk about a related issue with allcoations tha talso impact evacuate
14:35:03 bauzas sean-k-mooney: yeah, my confusion, I originally thought it was about the back-and-forth about evacuate we discussed for pain points
14:35:05 sean-k-mooney e.g. if for any reason we oversubscie the allcoations then we cant evacuate
14:36:15 sean-k-mooney i have not read it fully but it sound like they are not properly fencing the node an ensuring the vm is not running
14:36:26 sean-k-mooney before evacuating if they are having apllciation data currption
14:41:46 bauzas that's literrally what I wrote.
14:41:58 bauzas anyway, moving to a new bug.
14:42:43 sean-k-mooney ack as i said have not finsihed reading the bug description or comments so glad we agree :)
14:51:50 kashyap frickler: Ah-ha, noted. Good news: there's already some response from two QEMU devs, with a patch in a newer version :)
14:55:27 frickler kashyap: yeah I just responded, but I didn't see the patch reference. going from 32M to 1G really sounds a bit excessive, would be good to be able to tune that
14:55:29 kashyap (Well, I don't quite think it's "good news" ...)
14:55:47 kashyap frickler: Sorry, I was referring to the commit that DanPB pointed out - https://gitlab.com/qemu-project/qemu/-/commit/600e17b26
14:56:39 frickler kashyap: ah, yes, that seems to be the patch that triggers this, I though you were referring to a fix in a recent commit
14:56:45 kashyap frickler: Yeah, that increase is a tad too much.
14:56:53 kashyap frickler: Yes, poor phrasing on my part.
14:56:59 opendevreview Balazs Gibizer proposed openstack/nova master: Remove SESSION_CONFIGURED global from DB fixture https://review.opendev.org/c/openstack/nova/+/815689
14:57:42 frickler kashyap: otoh that also is likely to explain why tests seemed to be going faster on Bullseye than on Focal
14:58:18 opendevreview Balazs Gibizer proposed openstack/nova master: Refactor Database fixture https://review.opendev.org/c/openstack/nova/+/815690
14:59:22 kashyap frickler: Interesting; what tests are going faster?
14:59:35 opendevreview Balazs Gibizer proposed openstack/nova master: Fix interference in db unit test https://review.opendev.org/c/openstack/nova/+/814735
15:00:06 gibi stephenfin: ^^ here is the removal of the global SESSION_CONFIGURED flag from the DB fixture and some extra :D
15:02:37 frickler kashyap: I didn't check in particular, but the whole tempest-full job with --serial on Debian doesn't take much longer than with the default (-c 4 I think) on Focal
15:02:54 kashyap I see.
15:05:17 gibi melwitt: thanks a lot for the help exlaning the global db transaction factory situation. I used your info to actually remove SESSION_CONFIGURED from our fixture along the the unit test fixes
15:05:21 kashyap frickler: So, it is tunable via command-line, but it's not wired up in libvirt yet, though.
15:05:57 kashyap frickler: See the option: -accel=tcg,tb-size=$value_in_MiB
15:06:10 kashyap "tb-size" in the man page
15:10:32 frickler kashyap: as long as libvirt doesn't support it, I fear that won't help much. might be good to cap it to something like 50% of the VM memory
15:11:06 kashyap frickler: Right; libvirt just didn't wire it up ... we can meanwhile do a nasty hack of uploading a QEMU binary to the CI system w/ this param tweaked
15:11:52 kashyap frickler: Do you have the appetite to file a libvirt upstream RFE? (Then I can clone it downstream, and get it triaged)
15:13:15 frickler kashyap: I think I'll do a local test with a reduced default tb-size first in order to be certain that that's the cause. but not before tomorrow
15:13:25 kashyap Right, no rush at all
15:14:49 gibi bauzas: replied in https://bugs.launchpad.net/nova/+bug/1947753 I think _destroy_evacuated_instances is not called periodically
15:16:45 kashyap frickler: So I see that someone else has raised this upstream last year: https://lists.gnu.org/archive/html/qemu-devel/2020-07/msg05235.html (TB Cache size grows out of control with qemu 5.0)
15:22:08 bauzas gibi: indeed, only when restarting
15:22:16 bauzas did I said the other way ?
15:22:57 kashyap -accel tcg,tb-size=256
15:22:57 kashyap -machine q35
15:22:57 kashyap frickler: So, this worked for me:
15:23:03 kashyap (As an example)
15:25:07 gibi bauzas: at least I understood this sentence that way "Either way, if the service continues to run, it verifies the evacuation status periodically and deletes the host."
15:27:04 gibi bauzas: btw, about https://bugs.launchpad.net/nova/+bug/1947687 I cannot formulate a logstash signature it seems that this error happens in a lot of cases when no test cases are failing so I get a lot of false positives
15:27:21 bauzas gibi: okay, then my brain fucked
15:27:33 kashyap frickler: For reference, a minimal command-line:
15:27:35 kashyap $> qemu-kvm -display none -cpu Nehalem -no-user-config \ -machine q35 \ -accel tcg,tb-size=256 \ -nodefaults -m 2048 -serial stdio \ -drive file=/export/vm1.qcow2,format=qcow2,if=virtio
15:27:48 kashyap (Ugh, line-breaks are broken, but you see what I mean)
15:28:49 bauzas gibi: ack for the logstash thing, no worries
15:34:57 frickler kashyap: thx, added a comment to the issue, seems the libvirt path is really the most promising one
15:39:22 kashyap frickler: Definitely. Please file the RFE (and post a link to me, Bz Ccs will take me slower to process) when you can
15:39:29 kashyap Thanks for the patience :)
15:49:31 melwitt bauzas: hi, could you pls take a look at these train backports when you get a chance? someone posted a comment on the top patch yesterday indicating they are awaiting merge of the fixes https://review.opendev.org/q/topic:%2522bug/1927677%2522+branch:stable/train+status:open
15:49:36 opendevreview Balazs Gibizer proposed openstack/nova stable/pike: Add a WA flag waiting for vif-plugged event during reboot https://review.opendev.org/c/openstack/nova/+/813437
15:49:50 bauzas melwitt: ack, doing it now
15:50:03 melwitt thanks!
15:51:55 bauzas melwitt: I already reviewed them but forgot to submit, my bad
15:52:05 bauzas now this is fixed.
15:59:21 melwitt bauzas: a-ha, thank you
16:13:36 stephenfin gibi: question on https://review.opendev.org/c/openstack/nova/+/815690
16:13:43 stephenfin please excuse my ignorance
16:14:18 gibi lookgin
16:23:38 gibi stephenfin: you are right something is fishy there
16:23:53 gibi I have to go back and poke that test to understand what is happening
16:32:40 opendevreview Artom Lifshitz proposed openstack/nova master: DNM:goat https://review.opendev.org/c/openstack/nova/+/815705
16:32:41 opendevreview Artom Lifshitz proposed openstack/nova master: DNM: goat3 https://review.opendev.org/c/openstack/nova/+/815707
16:32:41 opendevreview Artom Lifshitz proposed openstack/nova master: DNM: goat 2 https://review.opendev.org/c/openstack/nova/+/815706
16:34:08 gibi gmann: I did the change you requested in https://review.opendev.org/c/openstack/tempest/+/809168/comment/35477e85_10754ba5/ but I wondering why we need that indirection
16:36:01 opendevreview Artom Lifshitz proposed openstack/nova master: DNM: goat3 https://review.opendev.org/c/openstack/nova/+/815707
16:36:01 opendevreview Artom Lifshitz proposed openstack/nova master: DNM: goat 2 https://review.opendev.org/c/openstack/nova/+/815706
17:16:06 em_ are there currently issues with xena nova and (debian) cloud images? Neither my ssh keys nor the admin password seems to get applied. Any open bugs (maybe libvirt issues or kernel related?) using 5.10 debian bullseye as host, kolla xena (ubuntu/source) as libvirt
17:18:47 opendevreview Balazs Gibizer proposed openstack/nova master: Refactor Database fixture https://review.opendev.org/c/openstack/nova/+/815690
17:19:16 gibi stephenfin: you had a valid point, fixed it ^^
17:20:05 opendevreview Balazs Gibizer proposed openstack/nova master: Fix interference in db unit test https://review.opendev.org/c/openstack/nova/+/814735
17:21:00 gmann gibi: replied, basically Tempest test the services with what is configured to test instead of 'test what cloud/service APIs return'
17:22:09 gmann autodetecting service features/extensions to what to test can hide the error.
17:26:41 gibi gmann: OK, I think I got it. Does devstack needs to be changed to generate the extension name to the tempest config/
17:26:44 gibi ?
17:28:05 gmann gibi: we do that, like master test with 'All' (enable everything) and stable are pin with the extensions list at the time of stable branch is released. like this - https://review.opendev.org/c/openstack/devstack/+/811485
17:28:47 gmann for now on master we do not need to do anything in devstack side
17:30:26 gibi gmann: ack, thanks for the help and explanation
17:32:43 gmann I will review the tempest patch once gate result is finished
17:32:47 gmann thanks for update
18:03:20 opendevreview Merged openstack/nova master: Ensure MAC addresses characters are in the same case https://review.opendev.org/c/openstack/nova/+/811947
18:29:59 opendevreview Merged openstack/nova master: Fix instance's image_ref lost on failed unshelving https://review.opendev.org/c/openstack/nova/+/807551
19:14:44 Zer0Byte hey
19:14:45 Zer0Byte question

Earlier   Later