| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-10-27 | |||
| 14:36:15 | sean-k-mooney | i have not read it fully but it sound like they are not properly fencing the node an ensuring the vm is not running | |
| 14:36:26 | sean-k-mooney | before evacuating if they are having apllciation data currption | |
| 14:41:46 | bauzas | that's literrally what I wrote. | |
| 14:41:58 | bauzas | anyway, moving to a new bug. | |
| 14:42:43 | sean-k-mooney | ack as i said have not finsihed reading the bug description or comments so glad we agree :) | |
| 14:51:50 | kashyap | frickler: Ah-ha, noted. Good news: there's already some response from two QEMU devs, with a patch in a newer version :) | |
| 14:55:27 | frickler | kashyap: yeah I just responded, but I didn't see the patch reference. going from 32M to 1G really sounds a bit excessive, would be good to be able to tune that | |
| 14:55:29 | kashyap | (Well, I don't quite think it's "good news" ...) | |
| 14:55:47 | kashyap | frickler: Sorry, I was referring to the commit that DanPB pointed out - https://gitlab.com/qemu-project/qemu/-/commit/600e17b26 | |
| 14:56:39 | frickler | kashyap: ah, yes, that seems to be the patch that triggers this, I though you were referring to a fix in a recent commit | |
| 14:56:45 | kashyap | frickler: Yeah, that increase is a tad too much. | |
| 14:56:53 | kashyap | frickler: Yes, poor phrasing on my part. | |
| 14:56:59 | opendevreview | Balazs Gibizer proposed openstack/nova master: Remove SESSION_CONFIGURED global from DB fixture https://review.opendev.org/c/openstack/nova/+/815689 | |
| 14:57:42 | frickler | kashyap: otoh that also is likely to explain why tests seemed to be going faster on Bullseye than on Focal | |
| 14:58:18 | opendevreview | Balazs Gibizer proposed openstack/nova master: Refactor Database fixture https://review.opendev.org/c/openstack/nova/+/815690 | |
| 14:59:22 | kashyap | frickler: Interesting; what tests are going faster? | |
| 14:59:35 | opendevreview | Balazs Gibizer proposed openstack/nova master: Fix interference in db unit test https://review.opendev.org/c/openstack/nova/+/814735 | |
| 15:00:06 | gibi | stephenfin: ^^ here is the removal of the global SESSION_CONFIGURED flag from the DB fixture and some extra :D | |
| 15:02:37 | frickler | kashyap: I didn't check in particular, but the whole tempest-full job with --serial on Debian doesn't take much longer than with the default (-c 4 I think) on Focal | |
| 15:02:54 | kashyap | I see. | |
| 15:05:17 | gibi | melwitt: thanks a lot for the help exlaning the global db transaction factory situation. I used your info to actually remove SESSION_CONFIGURED from our fixture along the the unit test fixes | |
| 15:05:21 | kashyap | frickler: So, it is tunable via command-line, but it's not wired up in libvirt yet, though. | |
| 15:05:57 | kashyap | frickler: See the option: -accel=tcg,tb-size=$value_in_MiB | |
| 15:06:10 | kashyap | "tb-size" in the man page | |
| 15:10:32 | frickler | kashyap: as long as libvirt doesn't support it, I fear that won't help much. might be good to cap it to something like 50% of the VM memory | |
| 15:11:06 | kashyap | frickler: Right; libvirt just didn't wire it up ... we can meanwhile do a nasty hack of uploading a QEMU binary to the CI system w/ this param tweaked | |
| 15:11:52 | kashyap | frickler: Do you have the appetite to file a libvirt upstream RFE? (Then I can clone it downstream, and get it triaged) | |
| 15:13:15 | frickler | kashyap: I think I'll do a local test with a reduced default tb-size first in order to be certain that that's the cause. but not before tomorrow | |
| 15:13:25 | kashyap | Right, no rush at all | |
| 15:14:49 | gibi | bauzas: replied in https://bugs.launchpad.net/nova/+bug/1947753 I think _destroy_evacuated_instances is not called periodically | |
| 15:16:45 | kashyap | frickler: So I see that someone else has raised this upstream last year: https://lists.gnu.org/archive/html/qemu-devel/2020-07/msg05235.html (TB Cache size grows out of control with qemu 5.0) | |
| 15:22:08 | bauzas | gibi: indeed, only when restarting | |
| 15:22:16 | bauzas | did I said the other way ? | |
| 15:22:57 | kashyap | -accel tcg,tb-size=256 | |
| 15:22:57 | kashyap | -machine q35 | |
| 15:22:57 | kashyap | frickler: So, this worked for me: | |
| 15:23:03 | kashyap | (As an example) | |
| 15:25:07 | gibi | bauzas: at least I understood this sentence that way "Either way, if the service continues to run, it verifies the evacuation status periodically and deletes the host." | |
| 15:27:04 | gibi | bauzas: btw, about https://bugs.launchpad.net/nova/+bug/1947687 I cannot formulate a logstash signature it seems that this error happens in a lot of cases when no test cases are failing so I get a lot of false positives | |
| 15:27:21 | bauzas | gibi: okay, then my brain fucked | |
| 15:27:33 | kashyap | frickler: For reference, a minimal command-line: | |
| 15:27:35 | kashyap | $> qemu-kvm -display none -cpu Nehalem -no-user-config \ -machine q35 \ -accel tcg,tb-size=256 \ -nodefaults -m 2048 -serial stdio \ -drive file=/export/vm1.qcow2,format=qcow2,if=virtio | |
| 15:27:48 | kashyap | (Ugh, line-breaks are broken, but you see what I mean) | |
| 15:28:49 | bauzas | gibi: ack for the logstash thing, no worries | |
| 15:34:57 | frickler | kashyap: thx, added a comment to the issue, seems the libvirt path is really the most promising one | |
| 15:39:22 | kashyap | frickler: Definitely. Please file the RFE (and post a link to me, Bz Ccs will take me slower to process) when you can | |
| 15:39:29 | kashyap | Thanks for the patience :) | |
| 15:49:31 | melwitt | bauzas: hi, could you pls take a look at these train backports when you get a chance? someone posted a comment on the top patch yesterday indicating they are awaiting merge of the fixes https://review.opendev.org/q/topic:%2522bug/1927677%2522+branch:stable/train+status:open | |
| 15:49:36 | opendevreview | Balazs Gibizer proposed openstack/nova stable/pike: Add a WA flag waiting for vif-plugged event during reboot https://review.opendev.org/c/openstack/nova/+/813437 | |
| 15:49:50 | bauzas | melwitt: ack, doing it now | |
| 15:50:03 | melwitt | thanks! | |
| 15:51:55 | bauzas | melwitt: I already reviewed them but forgot to submit, my bad | |
| 15:52:05 | bauzas | now this is fixed. | |
| 15:59:21 | melwitt | bauzas: a-ha, thank you | |
| 16:13:36 | stephenfin | gibi: question on https://review.opendev.org/c/openstack/nova/+/815690 | |
| 16:13:43 | stephenfin | please excuse my ignorance | |
| 16:14:18 | gibi | lookgin | |
| 16:23:38 | gibi | stephenfin: you are right something is fishy there | |
| 16:23:53 | gibi | I have to go back and poke that test to understand what is happening | |
| 16:32:40 | opendevreview | Artom Lifshitz proposed openstack/nova master: DNM:goat https://review.opendev.org/c/openstack/nova/+/815705 | |
| 16:32:41 | opendevreview | Artom Lifshitz proposed openstack/nova master: DNM: goat3 https://review.opendev.org/c/openstack/nova/+/815707 | |
| 16:32:41 | opendevreview | Artom Lifshitz proposed openstack/nova master: DNM: goat 2 https://review.opendev.org/c/openstack/nova/+/815706 | |
| 16:34:08 | gibi | gmann: I did the change you requested in https://review.opendev.org/c/openstack/tempest/+/809168/comment/35477e85_10754ba5/ but I wondering why we need that indirection | |
| 16:36:01 | opendevreview | Artom Lifshitz proposed openstack/nova master: DNM: goat3 https://review.opendev.org/c/openstack/nova/+/815707 | |
| 16:36:01 | opendevreview | Artom Lifshitz proposed openstack/nova master: DNM: goat 2 https://review.opendev.org/c/openstack/nova/+/815706 | |
| 17:16:06 | em_ | are there currently issues with xena nova and (debian) cloud images? Neither my ssh keys nor the admin password seems to get applied. Any open bugs (maybe libvirt issues or kernel related?) using 5.10 debian bullseye as host, kolla xena (ubuntu/source) as libvirt | |
| 17:18:47 | opendevreview | Balazs Gibizer proposed openstack/nova master: Refactor Database fixture https://review.opendev.org/c/openstack/nova/+/815690 | |
| 17:19:16 | gibi | stephenfin: you had a valid point, fixed it ^^ | |
| 17:20:05 | opendevreview | Balazs Gibizer proposed openstack/nova master: Fix interference in db unit test https://review.opendev.org/c/openstack/nova/+/814735 | |
| 17:21:00 | gmann | gibi: replied, basically Tempest test the services with what is configured to test instead of 'test what cloud/service APIs return' | |
| 17:22:09 | gmann | autodetecting service features/extensions to what to test can hide the error. | |
| 17:26:41 | gibi | gmann: OK, I think I got it. Does devstack needs to be changed to generate the extension name to the tempest config/ | |
| 17:26:44 | gibi | ? | |
| 17:28:05 | gmann | gibi: we do that, like master test with 'All' (enable everything) and stable are pin with the extensions list at the time of stable branch is released. like this - https://review.opendev.org/c/openstack/devstack/+/811485 | |
| 17:28:47 | gmann | for now on master we do not need to do anything in devstack side | |
| 17:30:26 | gibi | gmann: ack, thanks for the help and explanation | |
| 17:32:43 | gmann | I will review the tempest patch once gate result is finished | |
| 17:32:47 | gmann | thanks for update | |
| 18:03:20 | opendevreview | Merged openstack/nova master: Ensure MAC addresses characters are in the same case https://review.opendev.org/c/openstack/nova/+/811947 | |
| 18:29:59 | opendevreview | Merged openstack/nova master: Fix instance's image_ref lost on failed unshelving https://review.opendev.org/c/openstack/nova/+/807551 | |
| 19:14:44 | Zer0Byte | hey | |
| 19:14:45 | Zer0Byte | question | |
| 19:15:02 | Zer0Byte | im using the cinder frontend option to perform QOS at the storage with the spec total_iops_sec_per_gb=3 | |
| 19:15:07 | Zer0Byte | is working great | |
| 19:15:34 | Zer0Byte | but after extend the volume is not updating the total_iops_sec property on the KVM template | |
| 19:15:38 | Zer0Byte | is that normal | |
| 19:15:39 | Zer0Byte | ? | |
| 19:37:25 | EugenMayer | Hello. Anybody else has troubles with (Xena) bootstrapping a debian 11 (generic cloud) or debian 10(openstack variant) and not able to pre-deploy a ssh-key or even a root password? Looking at the logs, it always prints that there is no suitable ssh key to deploy. Tried it with an rsa ord ed key, no hopes. Any hints? | |
| 19:39:40 | EugenMayer | The boot log looks like this: https://gist.github.com/EugenMayer/452de9229e8f47dad0fadb4f8774d482 | |
| 20:21:35 | clarkb | EugenMayer: are you booting it with the proper flag to assign a nova ssh key to the instance? | |
| 20:22:02 | clarkb | Also if cloud-init can't reach the nova metadata service this might happen. You might try using a config drive if it isn't already | |
| 23:52:08 | Zer0Byte | no one with the issue of refresh the kvm | |
| 23:52:16 | Zer0Byte | volume iops | |
| 23:52:17 | Zer0Byte | ? | |
| #openstack-nova - 2021-10-28 | |||
| 07:05:52 | frickler | Zer0Byte: very likely that's a feature that would need to get implemented, could you create a bug report on launchpad? | |
| 07:56:18 | opendevreview | Federico Ressi proposed openstack/nova master: Debug Nova APIs call failures https://review.opendev.org/c/openstack/nova/+/806683 | |
| 07:58:14 | opendevreview | Federico Ressi proposed openstack/nova master: Check Nova project changes with Tobiko scenario test cases https://review.opendev.org/c/openstack/nova/+/806853 | |
| 08:08:33 | bauzas | morning Nova | |
| 08:12:31 | gibi | bauzas: morning | |
| 08:13:54 | gibi | bauzas: tesla, pixel 6, you will be pretty up to date with tech :) | |