| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2023-01-19 | |||
| 10:27:57 | bauzas | gibi: using $ stestr run --load-list FILENAME | |
| 10:28:58 | bauzas | gibi: lemme try to do it | |
| 10:30:23 | bauzas | but after this, I'll need to go off until end of the day as I need to help my daughters due to the french strike | |
| 10:31:56 | gibi | bauzas: yepp | |
| 10:32:30 | gibi | bauzas: I tried the list of test from the failed worker from the recent gate failure and also tried the list from my past reproducers in that bug report | |
| 10:32:46 | gibi | neither triggered the problem after couple of hours of continuous run | |
| 10:33:02 | gibi | now simply run the test with --until-failure and --random | |
| 10:33:06 | gibi | but this is a long shot | |
| 10:33:44 | bauzas | gibi: ok, I'll try to run it the same | |
| 10:34:01 | bauzas | gibi: I guess you know about https://stestr.readthedocs.io/en/latest/MANUAL.html#automated-test-isolation-bisection ? | |
| 10:34:41 | gibi | i think that only works after we can reproduce the fauilure | |
| 10:35:03 | gibi | and it needs a stable reproduction not a stohastic one | |
| 10:35:35 | bauzas | gibi: I'll try to look at that | |
| 10:35:56 | bauzas | folks, as I also said downstream, I'll be off until 1630UTC | |
| 10:39:33 | bauzas | gibi: so I loaded the subunit | |
| 10:39:36 | bauzas | into stestr | |
| 10:39:49 | bauzas | (functional-py38) [sbauza@sbauza nova]$ stestr load < /tmp/zuul-logs.bNmdDA/testrepository.subunit | |
| 10:40:06 | bauzas | I should now be able to run the failing test | |
| 10:41:34 | bauzas | anyway, I need to disappear :( | |
| 10:55:22 | gokhanis | hello folks, after rebooting my compute host, I can't attach my cinder volumes to instances. Nova throws "unable to lock /var/lib/nova/mnt/dgf/volume-xx for metadata change: No locks available" Full logs are in https://paste.openstack.org/show/beicZ71J17WeNwLjghKc/ What can be reason of this problem and how can I solve this issue ? | |
| 11:06:33 | sean-k-mooney | gokhanis: what kind fo cinder backedn are you using | |
| 11:06:57 | sean-k-mooney | is this nfs based? | |
| 11:08:16 | gokhanis | sean-k-mooney, yes I am using cinder nfs driver. | |
| 11:08:22 | sean-k-mooney | this does not sound like it actuly a nova issue but rather and issue with how you have deployed it. ie. like you are hitting a ulimit or similar in the env | |
| 11:08:34 | sean-k-mooney | gokhanis: ok is it nfs3 or nfs4 | |
| 11:08:54 | sean-k-mooney | nfs3 has locking issues and we dont really support it anymore | |
| 11:09:19 | sean-k-mooney | ideally you should use nfs 4.2 or newer | |
| 11:10:42 | sean-k-mooney | gokhanis: i think your hitiing this https://bugzilla.redhat.com/show_bug.cgi?id=1547095 | |
| 11:11:47 | sean-k-mooney | gokhanis: apprently you can "fix" it by using the nolock mount option for nfsv3 | |
| 11:12:07 | sean-k-mooney | gokhanis: also relevent https://access.redhat.com/solutions/2780381 | |
| 11:12:50 | gokhanis | sean-k-mooney, I deployed my env with Openstack Ansible. I have create nfs share on zfs storage and I used it as cinder nfs backend. | |
| 11:12:56 | sean-k-mooney | gokhanis: form a downstream perspetive redhat discontinued support for nfsv3 in our porudct because of these locking issues | |
| 11:14:21 | sean-k-mooney | gokhanis: yep so on the zfs culster you need to either ensure that the zfs pool is exported as nfsv4, nfsv3 with the nfsv4 lock manager or you have to ensure the nfs share is mounted with the nolock option on the compute hosts | |
| 11:15:11 | sean-k-mooney | gokhanis: you can use https://docs.openstack.org/nova/latest/configuration/config.html#libvirt.nfs_mount_options to set teh nolock option i think | |
| 11:15:48 | sean-k-mooney | but if the fileserver supprot v4 then vers=4.2,lookupcache=pos | |
| 11:15:56 | sean-k-mooney | woudl be better then nolock | |
| 11:17:18 | sean-k-mooney | gokhanis: some more details on nolock https://github.com/oVirt/vdsm/commit/8aad4ebad10bab15c46ff743c105b6c2cba68d4e | |
| 11:17:32 | gokhanis | thanks sean-k-mooney I will check nfs side in my env and add options you have said. | |
| 12:39:22 | opendevreview | melanie witt proposed openstack/nova master: imagebackend: Add support to libvirt_info for LUKS based encryption https://review.opendev.org/c/openstack/nova/+/826755 | |
| 12:39:22 | opendevreview | melanie witt proposed openstack/nova master: libvirt: Configure and teardown ephemeral encryption secrets https://review.opendev.org/c/openstack/nova/+/826754 | |
| 12:39:23 | opendevreview | melanie witt proposed openstack/nova master: Support create with ephemeral encryption for qcow2 https://review.opendev.org/c/openstack/nova/+/870932 | |
| 12:39:23 | opendevreview | melanie witt proposed openstack/nova master: imagebackend: Cache the key manager when disk is encrypted https://review.opendev.org/c/openstack/nova/+/826756 | |
| 12:39:24 | opendevreview | melanie witt proposed openstack/nova master: Add encryption support to convert_image https://review.opendev.org/c/openstack/nova/+/870934 | |
| 12:39:24 | opendevreview | melanie witt proposed openstack/nova master: Support resize with ephemeral encryption https://review.opendev.org/c/openstack/nova/+/870933 | |
| 12:39:25 | opendevreview | melanie witt proposed openstack/nova master: Add encryption support to qemu-img rebase https://review.opendev.org/c/openstack/nova/+/870936 | |
| 12:39:25 | opendevreview | melanie witt proposed openstack/nova master: Add hw_ephemeral_encryption_secret_uuid image property https://review.opendev.org/c/openstack/nova/+/870935 | |
| 12:39:26 | opendevreview | melanie witt proposed openstack/nova master: Support snapshot with ephemeral encryption https://review.opendev.org/c/openstack/nova/+/870937 | |
| 12:39:27 | opendevreview | melanie witt proposed openstack/nova master: Update driver BDMs with ephemeral encryption image properties https://review.opendev.org/c/openstack/nova/+/870939 | |
| 12:39:27 | opendevreview | melanie witt proposed openstack/nova master: Add reset_encryption_fields() and save_all() to BlockDeviceMappingList https://review.opendev.org/c/openstack/nova/+/870938 | |
| 12:39:29 | opendevreview | melanie witt proposed openstack/nova master: libvirt: Introduce support for qcow2 with LUKS https://review.opendev.org/c/openstack/nova/+/772273 | |
| 12:44:10 | opendevreview | melanie witt proposed openstack/nova master: DNM test ephemeral encryption + resize: qcow2, raw https://review.opendev.org/c/openstack/nova/+/862416 | |
| 12:45:31 | opendevreview | Rajesh Tailor proposed openstack/nova master: Handle InstanceInvalidState exception https://review.opendev.org/c/openstack/nova/+/861738 | |
| 12:57:02 | opendevreview | Kashyap Chamarthy proposed openstack/nova master: libvirt: At start-up allow skiping compareCPU() with a workaround https://review.opendev.org/c/openstack/nova/+/870794 | |
| 12:57:16 | kashyap | gibi: --^ Fixed the test, please have a re-look if you get a chance today | |
| 13:39:55 | opendevreview | Merged openstack/nova master: Clean up after ImportModulePoisonFixture https://review.opendev.org/c/openstack/nova/+/870993 | |
| 13:43:39 | gibi | bauzas, dansmith: the dstat log in https://review.opendev.org/c/openstack/nova/+/870950 shows that without that one test case that loads the image data there is no other memory hogger test case. At the highest memory load there is still 800MB free RAM in the VM. | |
| 13:51:20 | gibi | kashyap: +2, thanks | |
| 13:51:45 | sean-k-mooney | oh something merged thats a shock | |
| 13:52:19 | sean-k-mooney | its because test only change do not run the tempest jobs | |
| 13:52:24 | sean-k-mooney | similar for docs onlychanges | |
| 13:53:26 | kashyap | gibi: Thank you; (I fixed your other points too; sure you've seen it) | |
| 13:54:01 | kashyap | gibi: Is the nit worth respinning? | |
| 13:54:44 | sean-k-mooney | gibi: can you look at https://review.opendev.org/c/openstack/nova/+/869950 | |
| 13:54:45 | kashyap | sean-k-mooney: melwitt: or anyone else: I'm chaing a second +2/+W: please have a look at this workaround - https://review.opendev.org/c/openstack/nova/+/870794 | |
| 13:55:30 | sean-k-mooney | i have it open but i need to leave in 5 mins to drop back the tempory car i have and get back before my next meeting | |
| 13:55:37 | kashyap | gibi: sean-k-mooney: On _top_ of this workaround I'll add in the "replace old API with new API" patch - https://review.opendev.org/c/openstack/nova/+/869950 -- libvirt: Replace usage of compareCPU() with compareHypervisorCPU() | |
| 13:55:42 | sean-k-mooney | so it might be when i get back | |
| 13:55:45 | kashyap | Sure. | |
| 13:56:17 | sean-k-mooney | yep i want to merge both although i also think we should simplfy it so we only do one check | |
| 13:56:22 | gibi | kashyap: that nit from me does not worth a respin | |
| 13:56:50 | sean-k-mooney | e.g. only check with model+extra_flags | |
| 13:57:08 | kashyap | Yep; ie. remove the workaround-code | |
| 13:57:17 | kashyap | gibi: Nod; thx | |
| 13:57:34 | kashyap | (Remove later, i.e. not now :)) | |
| 13:57:59 | sean-k-mooney | anyway ill be back hopefully in an hour | |
| 13:58:10 | gibi | kashyap: so https://review.opendev.org/c/openstack/nova/+/869950 will be top of https://review.opendev.org/c/openstack/nova/+/870794 ? | |
| 13:58:21 | kashyap | gibi: Right | |
| 13:58:27 | kashyap | Makes sense to you? | |
| 13:58:35 | sean-k-mooney | well the workaround right now is disabling both check right | |
| 13:58:44 | gibi | kashyap: yeah that make sens | |
| 13:58:58 | sean-k-mooney | i have not looked at that patch in a few revisions | |
| 13:58:59 | kashyap | sean-k-mooney: Yes, only if you enable the workaround, obv. | |
| 13:59:27 | kashyap | sean-k-mooney: It's dead-simple - if workaround -> just skip the first _compare_cpu() call. | |
| 13:59:53 | kashyap | sean-k-mooney: And thanks, I see you've also +2ed the API-replacement patch, but I need to rebase it on top of the workaroud patch | |
| 14:00:09 | sean-k-mooney | and we still do the second call in init wiht the flags | |
| 14:00:49 | sean-k-mooney | ok am ill leave this for now and come back to it when i have time to review both again | |
| 14:01:39 | kashyap | Yeah, the second call correctly succeeds as it also takes into account the flags | |
| 14:02:22 | sean-k-mooney | yep so as long as that is alwasy there i was ok with removing the first call | |
| 14:02:38 | sean-k-mooney | i was not ok with removeing both of them and that why i wanted the workaround | |
| 14:02:51 | sean-k-mooney | anyway got to go be back later | |
| 14:05:58 | opendevreview | Sofia Enriquez proposed openstack/nova master: Implement is_luks_inside_qcow2 funtion https://review.opendev.org/c/openstack/nova/+/854030 | |
| 14:22:26 | opendevreview | Kashyap Chamarthy proposed openstack/nova master: libvirt: Replace usage of compareCPU() with compareHypervisorCPU() https://review.opendev.org/c/openstack/nova/+/869950 | |
| 14:23:07 | kashyap | gibi: --^ (No hurry) Done, rebased it on top of the workaround. | |
| 14:33:17 | gibi | kashyap: done :) | |
| 14:34:10 | kashyap | (シ_ _)シ | |
| 14:34:16 | kashyap | (It's a Japanese bow) | |
| 14:47:56 | gibi | ahh, I see :) | |
| 15:00:36 | dansmith | gibi: nice | |
| 15:00:48 | dansmith | gibi: I'm working on the real fix, but it's giving me a hard time.. lots of layers to cut through | |
| 15:06:54 | opendevreview | Alexey Stupnikov proposed openstack/nova stable/ussuri: fixtures: Handle binding of first port https://review.opendev.org/c/openstack/nova/+/871048 | |