| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2023-01-18 | |||
| 10:24:25 | gibi | lets keep https://review.opendev.org/c/openstack/nova/+/869900/ alone. as I want to land that regardless of our troubleshooting here. As I think some of the other problems might go away after we remove the excessive logging by that. | |
| 10:25:32 | gibi | about https://review.opendev.org/c/openstack/tempest/+/870913 why we are trying to move the test to the front? I thought we wanted to moved it later or even disable it temporarily to see if other tests are triggering the same OOM behavior and hence trying to establish a pattern causing the OOM | |
| 10:32:25 | bauzas | gibi: OK, then I'll add a DNM on nova, np | |
| 10:32:45 | bauzas | gibi: good question, I was wanting to see whether it was due to this test or not | |
| 10:33:03 | bauzas | if we call it first, and if this is due to this test, it would be killed earlier, right? | |
| 10:39:53 | opendevreview | Sylvain Bauza proposed openstack/nova master: DNM: Testing the killed test https://review.opendev.org/c/openstack/nova/+/870924 | |
| 10:44:42 | kashyap | gibi: Can you have a quick look at this workaround patch when you can (for a change all CI have passed): https://review.opendev.org/c/openstack/nova/+/870794 | |
| 10:44:47 | kashyap | (When you get a minute, that is) | |
| 10:48:15 | tobias-urdin | gibi: any possibility that we can backport this https://review.opendev.org/c/openstack/nova/+/838976 and parent reproducer patch? we are currently patching that in production as we're on newer libvirt with older nova release (xena right now, probably be yoga later this year) | |
| 11:06:50 | ierdem | Hi everyone, is there any way to boot a signed image from volume? I am testing image validation, I can create VM by using signed images on ephemeral disks but when I try boot from volume, it throws an excepiton (https://paste.openstack.org/show/blZen5ID7OIbi47TN8ib/). I am currently working on OpenStack Ussuri, and image backend is Ceph | |
| 11:15:49 | gokhanisi | hello folks, after rebooting my compute host, I can't attach my cinder volumes to instances. Nova throws "unable to lock /var/lib/nova/mnt/dgf/volume-xx for metadata change: No locks available" Full logs are in https://paste.openstack.org/show/beicZ71J17WeNwLjghKc/ What can be reason of this problem ? I am on victoria. On this compute node I am using also gpu passthrough. | |
| 11:23:15 | opendevreview | Sylvain Bauza proposed openstack/nova master: DNM: Testing the killed test https://review.opendev.org/c/openstack/nova/+/870924 | |
| 11:24:03 | gibi | tobias-urdin: regarding https://review.opendev.org/c/openstack/nova/+/838976 I think this is technically backportable but bauzas should know more about it as it is vgpu related | |
| 11:24:46 | bauzas | gibi: tobias-urdin: I already proposed the backports down to wallaby | |
| 11:25:40 | sean-k-mooney | tobias-urdin: there is a backport already | |
| 11:25:54 | sean-k-mooney | tobias-urdin: https://review.opendev.org/c/openstack/nova/+/866156 is the xena cherry pick | |
| 11:26:49 | sean-k-mooney | tobias-urdin: we needed it for wallaby for our downstream product so all the patches are up for review but we have already merged them downstream at the end of the year | |
| 11:35:48 | gibi | ahh I missed the backports as the topic was not set on them | |
| 11:39:10 | tobias-urdin | oh great, thanks! | |
| 11:39:55 | gibi | bauzas: regardin OOM I can do a parallel experiement moving the test to the latest to see if others before it trigger the OOM or not | |
| 11:40:04 | gibi | s/latest/last/ | |
| 11:40:06 | bauzas | gibi: sure, do it | |
| 11:40:10 | gibi | ack | |
| 11:40:39 | bauzas | gibi: I'm starting to look at the functest races | |
| 11:40:46 | bauzas | but I'm hungry | |
| 11:41:01 | gibi | ack. I start to get hungry too | |
| 11:41:03 | gibi | damn biology | |
| 11:47:49 | gibi | bauzas: this was the bug https://bugs.launchpad.net/nova/+bug/1946339 I referred to yesterday related to the funct test failures. It might or might not be related :/ | |
| 12:38:07 | opendevreview | melanie witt proposed openstack/nova master: imagebackend: Add support to libvirt_info for LUKS based encryption https://review.opendev.org/c/openstack/nova/+/826755 | |
| 12:38:08 | opendevreview | melanie witt proposed openstack/nova master: imagebackend: Cache the key manager when disk is encrypted https://review.opendev.org/c/openstack/nova/+/826756 | |
| 12:38:08 | opendevreview | melanie witt proposed openstack/nova master: libvirt: Introduce support for qcow2 with LUKS https://review.opendev.org/c/openstack/nova/+/772273 | |
| 12:38:09 | opendevreview | melanie witt proposed openstack/nova master: libvirt: Configure and teardown ephemeral encryption secrets https://review.opendev.org/c/openstack/nova/+/870931 | |
| 12:38:09 | opendevreview | melanie witt proposed openstack/nova master: Support create with ephemeral encryption for qcow2 https://review.opendev.org/c/openstack/nova/+/870932 | |
| 12:38:10 | opendevreview | melanie witt proposed openstack/nova master: Support resize with ephemeral encryption https://review.opendev.org/c/openstack/nova/+/870933 | |
| 12:38:10 | opendevreview | melanie witt proposed openstack/nova master: Add encryption support to convert_image https://review.opendev.org/c/openstack/nova/+/870934 | |
| 12:38:11 | opendevreview | melanie witt proposed openstack/nova master: Add hw_ephemeral_encryption_secret_uuid image property https://review.opendev.org/c/openstack/nova/+/870935 | |
| 12:38:11 | opendevreview | melanie witt proposed openstack/nova master: Add encryption support to qemu-img rebase https://review.opendev.org/c/openstack/nova/+/870936 | |
| 12:38:13 | opendevreview | melanie witt proposed openstack/nova master: Support snapshot with ephemeral encryption https://review.opendev.org/c/openstack/nova/+/870937 | |
| 12:38:13 | opendevreview | melanie witt proposed openstack/nova master: Add reset_encryption_fields() and save_all() to BlockDeviceMappingList https://review.opendev.org/c/openstack/nova/+/870938 | |
| 12:38:15 | opendevreview | melanie witt proposed openstack/nova master: Update driver BDMs with ephemeral encryption image properties https://review.opendev.org/c/openstack/nova/+/870939 | |
| 13:00:04 | opendevreview | melanie witt proposed openstack/nova master: DNM test ephemeral encryption + resize: qcow2, raw https://review.opendev.org/c/openstack/nova/+/862416 | |
| 13:03:13 | opendevreview | melanie witt proposed openstack/nova master: DNM test ephemeral encryption + resize: qcow2, raw https://review.opendev.org/c/openstack/nova/+/862416 | |
| 13:28:35 | opendevreview | Balazs Gibizer proposed openstack/nova master: DNM: Test OOM killed test https://review.opendev.org/c/openstack/nova/+/870950 | |
| 13:29:25 | opendevreview | Balazs Gibizer proposed openstack/nova master: DNM: Test OOM killed test https://review.opendev.org/c/openstack/nova/+/870950 | |
| 13:29:54 | gibi | bauzas: my trial is at https://review.opendev.org/c/openstack/tempest/+/870947 and https://review.opendev.org/c/openstack/nova/+/870950 | |
| 13:33:27 | opendevreview | Balazs Gibizer proposed openstack/nova master: DNM: Test OOM killed test https://review.opendev.org/c/openstack/nova/+/870950 | |
| 13:35:23 | kashyap | gibi: Thanks for catching my sloppiness here (I actually was rephrasing it locally). Do my replies seem reasonable to you? - https://review.opendev.org/c/openstack/nova/+/870794 | |
| 13:37:16 | bauzas | gibi: hmm, TIL about dstat and memory_tracker usage from devstack | |
| 13:41:08 | sean-k-mooney | ya they run as a service in all the devstack jobs | |
| 13:41:19 | sean-k-mooney | we used to have peak_mem_tracker too or somethign like that | |
| 13:41:31 | sean-k-mooney | dstat more or less has all the info you want/need | |
| 13:54:12 | kashyap | gibi: When you get a sec, I'm wondering what else is missing in this unit-test diff to check the API is called only once - https://paste.opendev.org/show/b7CXHOkMeuuXuzQD6QtW/ | |
| 14:15:49 | gibi | bauzas: this fresh functional failure https://7ffaea22ff93fca2f0ea-bf433abff5f8b85f7f80257b72ac6f67.ssl.cf5.rackcdn.com/869900/7/gate/nova-tox-functional-py38/3b10d8a/testr_results.html is very similar to what we discuss with melwitt in the comments of https://bugs.launchpad.net/nova/+bug/1946339 | |
| 14:16:52 | gibi | so probably eventlets are escaping the end of test case execution whete they were born and interfering with later tests | |
| 14:17:22 | sean-k-mooney | parsing that statement... | |
| 14:18:09 | sean-k-mooney | that sound kind fo familar | |
| 14:18:24 | gibi | yepp we fixed a set of those in the past but not all it seems | |
| 14:18:59 | sean-k-mooney | i tought we were explcitly shutting dow the event loop between tests gobally | |
| 14:19:09 | sean-k-mooney | as in vai a fixture | |
| 14:19:11 | gibi | is there a way to do that? | |
| 14:19:20 | sean-k-mooney | well yes | |
| 14:19:32 | sean-k-mooney | if we modify the base test case to call into eventlet | |
| 14:19:38 | sean-k-mooney | in test cleanup | |
| 14:20:53 | sean-k-mooney | i think there is a global kill but there is also a per greenthered kill | |
| 14:21:02 | sean-k-mooney | https://eventlet.net/doc/modules/greenthread.html#eventlet.greenthread.kill thats the per green tread one | |
| 14:21:29 | sean-k-mooney | we can also use waitall | |
| 14:21:31 | sean-k-mooney | https://eventlet.net/doc/modules/greenpool.html#eventlet.greenpool.GreenPool.waitall | |
| 14:22:08 | gibi | we are not pooling our eventlets | |
| 14:23:00 | gibi | and as you noted the greenthread.kill assumes we have access to the greented to kill | |
| 14:23:23 | sean-k-mooney | well we do have a greenthread pool but its provide by oslo | |
| 14:23:38 | sean-k-mooney | i was just looking at the docs to see if we have a way to list the greenthreads | |
| 14:23:45 | gibi | when we call spawn or spawn_n we are not using the greenlet from the pool | |
| 14:24:04 | sean-k-mooney | ya but i think there is a default pool that is used | |
| 14:24:10 | sean-k-mooney | i could be wrong | |
| 14:24:51 | gibi | at least I haven't came accross it when originally fixed part of this problem | |
| 14:25:18 | sean-k-mooney | i guess if there is one it does not say https://eventlet.net/doc/basic_usage.html#eventlet.spawn | |
| 14:25:40 | sean-k-mooney | is there any reason not to jsut have one gloabl pool | |
| 14:26:39 | sean-k-mooney | gibi: i was thinking of https://docs.openstack.org/nova/latest/configuration/config.html#DEFAULT.executor_thread_pool_size by the way | |
| 14:26:46 | sean-k-mooney | Size of executor thread pool when executor is threading or eventlet. | |
| 14:28:34 | gibi | as far as I see that is only used by oslo_messaging creating rpc message handler threads / eventlets. but nova uses spawn and spawn_n directly outside of oslo messaging | |
| 14:28:55 | gibi | we can try to pool them but I'm not sure both spawn and spawn_n can be pooled in the same way | |
| 14:29:05 | gibi | as they are not creating the same entity | |
| 14:29:24 | sean-k-mooney | there are spawn and spawn_n function on the pools | |
| 14:29:32 | sean-k-mooney | we might need a speerate on form the rpc one | |
| 14:29:39 | sean-k-mooney | or want a seperate one | |
| 14:29:48 | sean-k-mooney | but i think form an api point of view it shoudl be fine | |
| 14:30:07 | sean-k-mooney | https://eventlet.net/doc/modules/greenpool.html#eventlet.greenpool.GreenPool.spawn and https://eventlet.net/doc/modules/greenpool.html#eventlet.greenpool.GreenPool.spawn_n | |
| 14:30:44 | sean-k-mooney | hopefully we could just update it here https://github.com/openstack/nova/blob/master/nova/utils.py#L635-L684 | |
| 14:31:12 | sean-k-mooney | so create a module level pool and use that then in the test call waitall on the base testcase cleanup | |
| 14:32:34 | sean-k-mooney | https://github.com/openstack/nova/blob/master/nova/test.py#L150 we currently done tha a cleanup function but in setup we can also jsut add | |
| 14:32:55 | sean-k-mooney | self.addCleanup(utils.greenpool.waitall) | |
| 14:35:26 | gibi | I can try to set up a way to reproduce the issue more frequently locally and try to see if the pooling might solve it or not | |
| 14:40:36 | bauzas | sorry, was at the hairdresser | |
| 14:48:57 | opendevreview | Balazs Gibizer proposed openstack/nova master: DNM: Test OOM killed test https://review.opendev.org/c/openstack/nova/+/870950 | |
| 14:54:00 | gibi | bauzas: one result from your OOM trial is that I noticed that test in question takes a realitvely long time even if it passes tempest.api.compute.admin.test_aaa_volume.AttachSCSIVolumeTestJSON.test_attach_scsi_disk_with_config_drive [181.818420s] | |
| 14:54:27 | bauzas | gibi: yup, I've seen it | |
| 14:54:56 | bauzas | maybe we should introspect the memory size of the cached image | |
| 15:20:41 | bauzas | gibi: fwiw, since the UT ran successfully in the DNM patch, I looked at n-api log and I found we called it | |
| 15:20:53 | bauzas | gibi: while on https://834de1be955e9175dba1-6977f7378e5264bdb9ba9d1465839752.ssl.cf1.rackcdn.com/869900/6/gate/nova-ceph-multistore/f5aa5ed/controller/logs/screen-n-api.txt we were not calling it | |
| 15:21:16 | bauzas | so, I think the test was killed during the first glance call | |