| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2023-01-18 | |||
| 18:36:22 | sean-k-mooney | well didnt we also have OOM issues in teh funtional tests | |
| 18:36:47 | sean-k-mooney | wll not OOM python interpreter crahses | |
| 18:38:22 | sean-k-mooney | gibi: one question however | |
| 18:39:25 | sean-k-mooney | https://review.opendev.org/c/openstack/nova/+/870993/1/nova/tests/fixtures/nova.py#1838 | |
| 18:39:41 | sean-k-mooney | shoudl we install the filter in setUp not init | |
| 18:40:02 | sean-k-mooney | i assume the orgianl intent of doign it in init was to do it only once | |
| 18:40:13 | sean-k-mooney | for the life time of the fixture and resue the fixture | |
| 18:40:21 | sean-k-mooney | which is not how we are doign it | |
| 18:40:54 | sean-k-mooney | so if we are going to clean up in cleanup should we set it up in setUp | |
| 21:01:04 | opendevreview | Merged openstack/nova master: Use new get_rpc_client API from oslo.messaging https://review.opendev.org/c/openstack/nova/+/869900 | |
| 21:01:25 | opendevreview | Merged openstack/nova master: FUP for the scheduler part of PCI in placement https://review.opendev.org/c/openstack/nova/+/862876 | |
| #openstack-nova - 2023-01-19 | |||
| 00:41:13 | opendevreview | Ghanshyam proposed openstack/nova master: Bump openstack-placement version in functional tox env https://review.opendev.org/c/openstack/nova/+/871011 | |
| 01:27:03 | gmann | gibi: bauzas: placement is released and bumping the version in tox.ini https://review.opendev.org/c/openstack/nova/+/871011 | |
| 08:33:25 | gibi | gmann: thanks | |
| 08:37:20 | bauzas | gmann: ack, looking | |
| 09:19:26 | opendevreview | Balazs Gibizer proposed openstack/nova master: Clean up after ImportModulePoisonFixture https://review.opendev.org/c/openstack/nova/+/870993 | |
| 09:19:51 | gibi | sean-k-mooney: good point about __init__ vs setUp() in the fixture, I fixed it up ^^ | |
| 09:21:07 | gibi | sean-k-mooney: we have python test executor crashes in functional but we don't have syslog saved in functional so I cannot confirm that it is due to OOM. I see python crashes locally with functional run couple of weeks back and that wasnt OOM but segfault from the interpreter | |
| 09:21:35 | zigo | https://salsa.debian.org/openstack-team/services/ceilometer-instance-poller/-/blob/debian/zed/ceilometer_instance_poller/instance_poller.py#L133 | |
| 09:21:35 | zigo | which works well, but crashes here: | |
| 09:21:35 | zigo | https://salsa.debian.org/openstack-team/services/ceilometer-instance-poller/ | |
| 09:21:35 | zigo | We wrote this: | |
| 09:21:35 | zigo | Hi there! | |
| 09:21:37 | zigo | if we're using the Ceph backend. | |
| 09:21:39 | zigo | Does anyone of you know how to set the libvirt Ceph secret so then libvirt/libguestfs knows how to use the Ceph backend? | |
| 09:37:37 | bauzas | zigo: sorry, not a expert at all on how we integrate with ceph | |
| 09:37:54 | bauzas | maybe look at how nova attaches rdb disks | |
| 09:37:59 | bauzas | rbd* | |
| 09:38:08 | zigo | Yeah, it's smoething like that! :) | |
| 09:38:30 | zigo | Got to find out how to tell libvirt/libguestfs what username and secret to use. | |
| 09:38:50 | zigo | Obviously, I've been search for a long time before asking here ... | |
| 09:40:09 | bauzas | gibi: weirdo | |
| 09:40:46 | bauzas | gibi: I find we only take 30secs for grabbing the image which sizes 85 | |
| 09:40:48 | bauzas | 2023-01-18 16:24:58.358 127779 INFO tempest.api.compute.admin.test_aaa_volume [-] DNM bauzas: Time elapsed: 31.457598802999883 2023-01-18 16:24:58.379 127779 INFO tempest.api.compute.admin.test_aaa_volume [-] DNM bauzas: Size of custom img: 85 | |
| 09:41:44 | bauzas | and in this run, it took 93 secs | |
| 09:41:56 | bauzas | have we changed the size of the default image ? | |
| 09:42:29 | bauzas | mmm, yeah | |
| 09:42:30 | bauzas | 2023-01-18 16:22:09.379365 | controller | ++ lib/tempest:configure_tempest:434 : iniset /opt/stack/tempest/etc/tempest.conf compute image_ref 7bac73d8-9596-43ae-900f-546a3d53de36 2023-01-18 16:22:09.411839 | controller | ++ lib/tempest:configure_tempest:435 : iniset /opt/stack/tempest/etc/tempest.conf compute image_ref_alt 7bac73d8-9596-43ae-900f-546a3d53de36 | |
| 09:44:51 | bauzas | oh, nevermind | |
| 09:45:00 | bauzas | we return the image_id hence the very small size | |
| 10:11:39 | opendevreview | Merged openstack/nova master: Bump openstack-placement version in functional tox env https://review.opendev.org/c/openstack/nova/+/871011 | |
| 10:19:33 | gibi | I cannot reproduce the functional failure locally with the same way I did in the past https://bugs.launchpad.net/nova/+bug/1946339 so I'm stuck a bit there | |
| 10:26:40 | bauzas | gibi: I'll try to run it | |
| 10:27:14 | bauzas | gibi: have you tried to run the functional tests by using the subunit ? | |
| 10:27:57 | bauzas | gibi: using $ stestr run --load-list FILENAME | |
| 10:28:58 | bauzas | gibi: lemme try to do it | |
| 10:30:23 | bauzas | but after this, I'll need to go off until end of the day as I need to help my daughters due to the french strike | |
| 10:31:56 | gibi | bauzas: yepp | |
| 10:32:30 | gibi | bauzas: I tried the list of test from the failed worker from the recent gate failure and also tried the list from my past reproducers in that bug report | |
| 10:32:46 | gibi | neither triggered the problem after couple of hours of continuous run | |
| 10:33:02 | gibi | now simply run the test with --until-failure and --random | |
| 10:33:06 | gibi | but this is a long shot | |
| 10:33:44 | bauzas | gibi: ok, I'll try to run it the same | |
| 10:34:01 | bauzas | gibi: I guess you know about https://stestr.readthedocs.io/en/latest/MANUAL.html#automated-test-isolation-bisection ? | |
| 10:34:41 | gibi | i think that only works after we can reproduce the fauilure | |
| 10:35:03 | gibi | and it needs a stable reproduction not a stohastic one | |
| 10:35:35 | bauzas | gibi: I'll try to look at that | |
| 10:35:56 | bauzas | folks, as I also said downstream, I'll be off until 1630UTC | |
| 10:39:33 | bauzas | gibi: so I loaded the subunit | |
| 10:39:36 | bauzas | into stestr | |
| 10:39:49 | bauzas | (functional-py38) [sbauza@sbauza nova]$ stestr load < /tmp/zuul-logs.bNmdDA/testrepository.subunit | |
| 10:40:06 | bauzas | I should now be able to run the failing test | |
| 10:41:34 | bauzas | anyway, I need to disappear :( | |
| 10:55:22 | gokhanis | hello folks, after rebooting my compute host, I can't attach my cinder volumes to instances. Nova throws "unable to lock /var/lib/nova/mnt/dgf/volume-xx for metadata change: No locks available" Full logs are in https://paste.openstack.org/show/beicZ71J17WeNwLjghKc/ What can be reason of this problem and how can I solve this issue ? | |
| 11:06:33 | sean-k-mooney | gokhanis: what kind fo cinder backedn are you using | |
| 11:06:57 | sean-k-mooney | is this nfs based? | |
| 11:08:16 | gokhanis | sean-k-mooney, yes I am using cinder nfs driver. | |
| 11:08:22 | sean-k-mooney | this does not sound like it actuly a nova issue but rather and issue with how you have deployed it. ie. like you are hitting a ulimit or similar in the env | |
| 11:08:34 | sean-k-mooney | gokhanis: ok is it nfs3 or nfs4 | |
| 11:08:54 | sean-k-mooney | nfs3 has locking issues and we dont really support it anymore | |
| 11:09:19 | sean-k-mooney | ideally you should use nfs 4.2 or newer | |
| 11:10:42 | sean-k-mooney | gokhanis: i think your hitiing this https://bugzilla.redhat.com/show_bug.cgi?id=1547095 | |
| 11:11:47 | sean-k-mooney | gokhanis: apprently you can "fix" it by using the nolock mount option for nfsv3 | |
| 11:12:07 | sean-k-mooney | gokhanis: also relevent https://access.redhat.com/solutions/2780381 | |
| 11:12:50 | gokhanis | sean-k-mooney, I deployed my env with Openstack Ansible. I have create nfs share on zfs storage and I used it as cinder nfs backend. | |
| 11:12:56 | sean-k-mooney | gokhanis: form a downstream perspetive redhat discontinued support for nfsv3 in our porudct because of these locking issues | |
| 11:14:21 | sean-k-mooney | gokhanis: yep so on the zfs culster you need to either ensure that the zfs pool is exported as nfsv4, nfsv3 with the nfsv4 lock manager or you have to ensure the nfs share is mounted with the nolock option on the compute hosts | |
| 11:15:11 | sean-k-mooney | gokhanis: you can use https://docs.openstack.org/nova/latest/configuration/config.html#libvirt.nfs_mount_options to set teh nolock option i think | |
| 11:15:48 | sean-k-mooney | but if the fileserver supprot v4 then vers=4.2,lookupcache=pos | |
| 11:15:56 | sean-k-mooney | woudl be better then nolock | |
| 11:17:18 | sean-k-mooney | gokhanis: some more details on nolock https://github.com/oVirt/vdsm/commit/8aad4ebad10bab15c46ff743c105b6c2cba68d4e | |
| 11:17:32 | gokhanis | thanks sean-k-mooney I will check nfs side in my env and add options you have said. | |
| 12:39:22 | opendevreview | melanie witt proposed openstack/nova master: imagebackend: Add support to libvirt_info for LUKS based encryption https://review.opendev.org/c/openstack/nova/+/826755 | |
| 12:39:22 | opendevreview | melanie witt proposed openstack/nova master: libvirt: Configure and teardown ephemeral encryption secrets https://review.opendev.org/c/openstack/nova/+/826754 | |
| 12:39:23 | opendevreview | melanie witt proposed openstack/nova master: Support create with ephemeral encryption for qcow2 https://review.opendev.org/c/openstack/nova/+/870932 | |
| 12:39:23 | opendevreview | melanie witt proposed openstack/nova master: imagebackend: Cache the key manager when disk is encrypted https://review.opendev.org/c/openstack/nova/+/826756 | |
| 12:39:24 | opendevreview | melanie witt proposed openstack/nova master: Add encryption support to convert_image https://review.opendev.org/c/openstack/nova/+/870934 | |
| 12:39:24 | opendevreview | melanie witt proposed openstack/nova master: Support resize with ephemeral encryption https://review.opendev.org/c/openstack/nova/+/870933 | |
| 12:39:25 | opendevreview | melanie witt proposed openstack/nova master: Add encryption support to qemu-img rebase https://review.opendev.org/c/openstack/nova/+/870936 | |
| 12:39:25 | opendevreview | melanie witt proposed openstack/nova master: Add hw_ephemeral_encryption_secret_uuid image property https://review.opendev.org/c/openstack/nova/+/870935 | |
| 12:39:26 | opendevreview | melanie witt proposed openstack/nova master: Support snapshot with ephemeral encryption https://review.opendev.org/c/openstack/nova/+/870937 | |
| 12:39:27 | opendevreview | melanie witt proposed openstack/nova master: Update driver BDMs with ephemeral encryption image properties https://review.opendev.org/c/openstack/nova/+/870939 | |
| 12:39:27 | opendevreview | melanie witt proposed openstack/nova master: Add reset_encryption_fields() and save_all() to BlockDeviceMappingList https://review.opendev.org/c/openstack/nova/+/870938 | |
| 12:39:29 | opendevreview | melanie witt proposed openstack/nova master: libvirt: Introduce support for qcow2 with LUKS https://review.opendev.org/c/openstack/nova/+/772273 | |
| 12:44:10 | opendevreview | melanie witt proposed openstack/nova master: DNM test ephemeral encryption + resize: qcow2, raw https://review.opendev.org/c/openstack/nova/+/862416 | |
| 12:45:31 | opendevreview | Rajesh Tailor proposed openstack/nova master: Handle InstanceInvalidState exception https://review.opendev.org/c/openstack/nova/+/861738 | |
| 12:57:02 | opendevreview | Kashyap Chamarthy proposed openstack/nova master: libvirt: At start-up allow skiping compareCPU() with a workaround https://review.opendev.org/c/openstack/nova/+/870794 | |
| 12:57:16 | kashyap | gibi: --^ Fixed the test, please have a re-look if you get a chance today | |
| 13:39:55 | opendevreview | Merged openstack/nova master: Clean up after ImportModulePoisonFixture https://review.opendev.org/c/openstack/nova/+/870993 | |
| 13:43:39 | gibi | bauzas, dansmith: the dstat log in https://review.opendev.org/c/openstack/nova/+/870950 shows that without that one test case that loads the image data there is no other memory hogger test case. At the highest memory load there is still 800MB free RAM in the VM. | |