| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2023-01-18 | |||
| 10:01:01 | ierdem | Hi folks, is there any way to boot a signed image from volume? I am testing image validation, I can create VM by using signed images on ephemeral disks but when I try boot from volume, it throws an excepiton (https://paste.openstack.org/show/blZen5ID7OIbi47TN8ib/). I am currently working on OpenStack Ussuri, and image backend is Ceph | |
| 10:10:11 | bauzas | gibi: so, there are many ways tricking stestr scheduling | |
| 10:10:43 | bauzas | gibi: but given we directly call tempest which eventually calls stestr, the quickiest way to change the test scheduling is about renaming the testname | |
| 10:11:09 | bauzas | https://stestr.readthedocs.io/en/latest/MANUAL.html#test-scheduling | |
| 10:11:26 | bauzas | "By default stestr schedules the tests by first checking if there is any historical timing data on any tests. It then sorts the tests by that timing data loops over the tests in order and adds one to each worker that it will launch. For tests without timing data, the same is done, except the tests are in alphabetical order instead of based on timing data. If a group regex is used the same algorithm is used with groups instead of | |
| 10:11:26 | bauzas | individual tests." | |
| 10:20:12 | bauzas | gibi: https://review.opendev.org/c/openstack/tempest/+/870913 | |
| 10:20:44 | bauzas | gibi: could I modify https://review.opendev.org/c/openstack/nova/+/869900/ to be Depending on ^ ? | |
| 10:24:25 | gibi | lets keep https://review.opendev.org/c/openstack/nova/+/869900/ alone. as I want to land that regardless of our troubleshooting here. As I think some of the other problems might go away after we remove the excessive logging by that. | |
| 10:25:32 | gibi | about https://review.opendev.org/c/openstack/tempest/+/870913 why we are trying to move the test to the front? I thought we wanted to moved it later or even disable it temporarily to see if other tests are triggering the same OOM behavior and hence trying to establish a pattern causing the OOM | |
| 10:32:25 | bauzas | gibi: OK, then I'll add a DNM on nova, np | |
| 10:32:45 | bauzas | gibi: good question, I was wanting to see whether it was due to this test or not | |
| 10:33:03 | bauzas | if we call it first, and if this is due to this test, it would be killed earlier, right? | |
| 10:39:53 | opendevreview | Sylvain Bauza proposed openstack/nova master: DNM: Testing the killed test https://review.opendev.org/c/openstack/nova/+/870924 | |
| 10:44:42 | kashyap | gibi: Can you have a quick look at this workaround patch when you can (for a change all CI have passed): https://review.opendev.org/c/openstack/nova/+/870794 | |
| 10:44:47 | kashyap | (When you get a minute, that is) | |
| 10:48:15 | tobias-urdin | gibi: any possibility that we can backport this https://review.opendev.org/c/openstack/nova/+/838976 and parent reproducer patch? we are currently patching that in production as we're on newer libvirt with older nova release (xena right now, probably be yoga later this year) | |
| 11:06:50 | ierdem | Hi everyone, is there any way to boot a signed image from volume? I am testing image validation, I can create VM by using signed images on ephemeral disks but when I try boot from volume, it throws an excepiton (https://paste.openstack.org/show/blZen5ID7OIbi47TN8ib/). I am currently working on OpenStack Ussuri, and image backend is Ceph | |
| 11:15:49 | gokhanisi | hello folks, after rebooting my compute host, I can't attach my cinder volumes to instances. Nova throws "unable to lock /var/lib/nova/mnt/dgf/volume-xx for metadata change: No locks available" Full logs are in https://paste.openstack.org/show/beicZ71J17WeNwLjghKc/ What can be reason of this problem ? I am on victoria. On this compute node I am using also gpu passthrough. | |
| 11:23:15 | opendevreview | Sylvain Bauza proposed openstack/nova master: DNM: Testing the killed test https://review.opendev.org/c/openstack/nova/+/870924 | |
| 11:24:03 | gibi | tobias-urdin: regarding https://review.opendev.org/c/openstack/nova/+/838976 I think this is technically backportable but bauzas should know more about it as it is vgpu related | |
| 11:24:46 | bauzas | gibi: tobias-urdin: I already proposed the backports down to wallaby | |
| 11:25:40 | sean-k-mooney | tobias-urdin: there is a backport already | |
| 11:25:54 | sean-k-mooney | tobias-urdin: https://review.opendev.org/c/openstack/nova/+/866156 is the xena cherry pick | |
| 11:26:49 | sean-k-mooney | tobias-urdin: we needed it for wallaby for our downstream product so all the patches are up for review but we have already merged them downstream at the end of the year | |
| 11:35:48 | gibi | ahh I missed the backports as the topic was not set on them | |
| 11:39:10 | tobias-urdin | oh great, thanks! | |
| 11:39:55 | gibi | bauzas: regardin OOM I can do a parallel experiement moving the test to the latest to see if others before it trigger the OOM or not | |
| 11:40:04 | gibi | s/latest/last/ | |
| 11:40:06 | bauzas | gibi: sure, do it | |
| 11:40:10 | gibi | ack | |
| 11:40:39 | bauzas | gibi: I'm starting to look at the functest races | |
| 11:40:46 | bauzas | but I'm hungry | |
| 11:41:01 | gibi | ack. I start to get hungry too | |
| 11:41:03 | gibi | damn biology | |
| 11:47:49 | gibi | bauzas: this was the bug https://bugs.launchpad.net/nova/+bug/1946339 I referred to yesterday related to the funct test failures. It might or might not be related :/ | |
| 12:38:07 | opendevreview | melanie witt proposed openstack/nova master: imagebackend: Add support to libvirt_info for LUKS based encryption https://review.opendev.org/c/openstack/nova/+/826755 | |
| 12:38:08 | opendevreview | melanie witt proposed openstack/nova master: imagebackend: Cache the key manager when disk is encrypted https://review.opendev.org/c/openstack/nova/+/826756 | |
| 12:38:08 | opendevreview | melanie witt proposed openstack/nova master: libvirt: Introduce support for qcow2 with LUKS https://review.opendev.org/c/openstack/nova/+/772273 | |
| 12:38:09 | opendevreview | melanie witt proposed openstack/nova master: libvirt: Configure and teardown ephemeral encryption secrets https://review.opendev.org/c/openstack/nova/+/870931 | |
| 12:38:09 | opendevreview | melanie witt proposed openstack/nova master: Support create with ephemeral encryption for qcow2 https://review.opendev.org/c/openstack/nova/+/870932 | |
| 12:38:10 | opendevreview | melanie witt proposed openstack/nova master: Support resize with ephemeral encryption https://review.opendev.org/c/openstack/nova/+/870933 | |
| 12:38:10 | opendevreview | melanie witt proposed openstack/nova master: Add encryption support to convert_image https://review.opendev.org/c/openstack/nova/+/870934 | |
| 12:38:11 | opendevreview | melanie witt proposed openstack/nova master: Add hw_ephemeral_encryption_secret_uuid image property https://review.opendev.org/c/openstack/nova/+/870935 | |
| 12:38:11 | opendevreview | melanie witt proposed openstack/nova master: Add encryption support to qemu-img rebase https://review.opendev.org/c/openstack/nova/+/870936 | |
| 12:38:13 | opendevreview | melanie witt proposed openstack/nova master: Support snapshot with ephemeral encryption https://review.opendev.org/c/openstack/nova/+/870937 | |
| 12:38:13 | opendevreview | melanie witt proposed openstack/nova master: Add reset_encryption_fields() and save_all() to BlockDeviceMappingList https://review.opendev.org/c/openstack/nova/+/870938 | |
| 12:38:15 | opendevreview | melanie witt proposed openstack/nova master: Update driver BDMs with ephemeral encryption image properties https://review.opendev.org/c/openstack/nova/+/870939 | |
| 13:00:04 | opendevreview | melanie witt proposed openstack/nova master: DNM test ephemeral encryption + resize: qcow2, raw https://review.opendev.org/c/openstack/nova/+/862416 | |
| 13:03:13 | opendevreview | melanie witt proposed openstack/nova master: DNM test ephemeral encryption + resize: qcow2, raw https://review.opendev.org/c/openstack/nova/+/862416 | |
| 13:28:35 | opendevreview | Balazs Gibizer proposed openstack/nova master: DNM: Test OOM killed test https://review.opendev.org/c/openstack/nova/+/870950 | |
| 13:29:25 | opendevreview | Balazs Gibizer proposed openstack/nova master: DNM: Test OOM killed test https://review.opendev.org/c/openstack/nova/+/870950 | |
| 13:29:54 | gibi | bauzas: my trial is at https://review.opendev.org/c/openstack/tempest/+/870947 and https://review.opendev.org/c/openstack/nova/+/870950 | |
| 13:33:27 | opendevreview | Balazs Gibizer proposed openstack/nova master: DNM: Test OOM killed test https://review.opendev.org/c/openstack/nova/+/870950 | |
| 13:35:23 | kashyap | gibi: Thanks for catching my sloppiness here (I actually was rephrasing it locally). Do my replies seem reasonable to you? - https://review.opendev.org/c/openstack/nova/+/870794 | |
| 13:37:16 | bauzas | gibi: hmm, TIL about dstat and memory_tracker usage from devstack | |
| 13:41:08 | sean-k-mooney | ya they run as a service in all the devstack jobs | |
| 13:41:19 | sean-k-mooney | we used to have peak_mem_tracker too or somethign like that | |
| 13:41:31 | sean-k-mooney | dstat more or less has all the info you want/need | |
| 13:54:12 | kashyap | gibi: When you get a sec, I'm wondering what else is missing in this unit-test diff to check the API is called only once - https://paste.opendev.org/show/b7CXHOkMeuuXuzQD6QtW/ | |
| 14:15:49 | gibi | bauzas: this fresh functional failure https://7ffaea22ff93fca2f0ea-bf433abff5f8b85f7f80257b72ac6f67.ssl.cf5.rackcdn.com/869900/7/gate/nova-tox-functional-py38/3b10d8a/testr_results.html is very similar to what we discuss with melwitt in the comments of https://bugs.launchpad.net/nova/+bug/1946339 | |
| 14:16:52 | gibi | so probably eventlets are escaping the end of test case execution whete they were born and interfering with later tests | |
| 14:17:22 | sean-k-mooney | parsing that statement... | |
| 14:18:09 | sean-k-mooney | that sound kind fo familar | |
| 14:18:24 | gibi | yepp we fixed a set of those in the past but not all it seems | |
| 14:18:59 | sean-k-mooney | i tought we were explcitly shutting dow the event loop between tests gobally | |
| 14:19:09 | sean-k-mooney | as in vai a fixture | |
| 14:19:11 | gibi | is there a way to do that? | |
| 14:19:20 | sean-k-mooney | well yes | |
| 14:19:32 | sean-k-mooney | if we modify the base test case to call into eventlet | |
| 14:19:38 | sean-k-mooney | in test cleanup | |
| 14:20:53 | sean-k-mooney | i think there is a global kill but there is also a per greenthered kill | |
| 14:21:02 | sean-k-mooney | https://eventlet.net/doc/modules/greenthread.html#eventlet.greenthread.kill thats the per green tread one | |
| 14:21:29 | sean-k-mooney | we can also use waitall | |
| 14:21:31 | sean-k-mooney | https://eventlet.net/doc/modules/greenpool.html#eventlet.greenpool.GreenPool.waitall | |
| 14:22:08 | gibi | we are not pooling our eventlets | |
| 14:23:00 | gibi | and as you noted the greenthread.kill assumes we have access to the greented to kill | |
| 14:23:23 | sean-k-mooney | well we do have a greenthread pool but its provide by oslo | |
| 14:23:38 | sean-k-mooney | i was just looking at the docs to see if we have a way to list the greenthreads | |
| 14:23:45 | gibi | when we call spawn or spawn_n we are not using the greenlet from the pool | |
| 14:24:04 | sean-k-mooney | ya but i think there is a default pool that is used | |
| 14:24:10 | sean-k-mooney | i could be wrong | |
| 14:24:51 | gibi | at least I haven't came accross it when originally fixed part of this problem | |
| 14:25:18 | sean-k-mooney | i guess if there is one it does not say https://eventlet.net/doc/basic_usage.html#eventlet.spawn | |
| 14:25:40 | sean-k-mooney | is there any reason not to jsut have one gloabl pool | |
| 14:26:39 | sean-k-mooney | gibi: i was thinking of https://docs.openstack.org/nova/latest/configuration/config.html#DEFAULT.executor_thread_pool_size by the way | |
| 14:26:46 | sean-k-mooney | Size of executor thread pool when executor is threading or eventlet. | |
| 14:28:34 | gibi | as far as I see that is only used by oslo_messaging creating rpc message handler threads / eventlets. but nova uses spawn and spawn_n directly outside of oslo messaging | |
| 14:28:55 | gibi | we can try to pool them but I'm not sure both spawn and spawn_n can be pooled in the same way | |
| 14:29:05 | gibi | as they are not creating the same entity | |
| 14:29:24 | sean-k-mooney | there are spawn and spawn_n function on the pools | |
| 14:29:32 | sean-k-mooney | we might need a speerate on form the rpc one | |
| 14:29:39 | sean-k-mooney | or want a seperate one | |
| 14:29:48 | sean-k-mooney | but i think form an api point of view it shoudl be fine | |
| 14:30:07 | sean-k-mooney | https://eventlet.net/doc/modules/greenpool.html#eventlet.greenpool.GreenPool.spawn and https://eventlet.net/doc/modules/greenpool.html#eventlet.greenpool.GreenPool.spawn_n | |
| 14:30:44 | sean-k-mooney | hopefully we could just update it here https://github.com/openstack/nova/blob/master/nova/utils.py#L635-L684 | |
| 14:31:12 | sean-k-mooney | so create a module level pool and use that then in the test call waitall on the base testcase cleanup | |
| 14:32:34 | sean-k-mooney | https://github.com/openstack/nova/blob/master/nova/test.py#L150 we currently done tha a cleanup function but in setup we can also jsut add | |
| 14:32:55 | sean-k-mooney | self.addCleanup(utils.greenpool.waitall) | |
| 14:35:26 | gibi | I can try to set up a way to reproduce the issue more frequently locally and try to see if the pooling might solve it or not | |