| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2023-01-17 | |||
| 17:53:55 | sean-k-mooney | sigh... https://review.opendev.org/c/openstack/nova/+/869900 will never merge | |
| 18:14:14 | sean-k-mooney | bauzas: summerised usecase here https://blueprints.launchpad.net/nova/+spec/default-ephemeral-format-unformated | |
| 18:14:21 | sean-k-mooney | in the whiteboard | |
| 18:14:27 | bauzas | sean-k-mooney: all good thanks | |
| 18:43:02 | dansmith | yeah looking at the latest ceph-multistore oom, one of tempest's workers is using >900MiB of ram, where the others are <10 | |
| 18:43:09 | dansmith | so something is clearly going haywire | |
| 18:43:33 | dansmith | gmann: you're aware of this right? | |
| 18:43:52 | dansmith | I assume if you knew of any recent tempest changes that could be responsible, you'd have spoken up by now :) | |
| 18:57:01 | gmann | dansmith: no, I cannot recall any relevant change happened in tempest | |
| 18:57:15 | dansmith | yeah I looked too, and nothing much lately | |
| 18:59:49 | gmann | and no stestr version change recently | |
| 19:01:33 | dansmith | I'm trying to stack for the first time this year so I can repro and I'm getting this failure to install pbr.build when it tries to install os-testr | |
| 19:01:38 | dansmith | ModuleNotFoundError: No module named 'pbr.build' | |
| 19:01:45 | dansmith | is this known? | |
| 19:01:52 | dansmith | gmann: ^ | |
| 19:03:33 | gmann | no, i did not see this before | |
| 19:04:01 | sean-k-mooney | stephenfin: fixed a pbr issues recently not sure if its releaed | |
| 19:05:22 | sean-k-mooney | this https://review.opendev.org/q/topic:pep-517 but that looks more tox 4 related | |
| 19:06:46 | dansmith | blargh | |
| 19:06:56 | dansmith | I can install os-testr without constraints, but it fails like this otherwise | |
| 19:07:08 | sean-k-mooney | wait os-testr | |
| 19:07:11 | gmann | yeah I think those were tox4 related | |
| 19:07:13 | sean-k-mooney | what is using that | |
| 19:07:22 | dansmith | sean-k-mooney: +./stack.sh:main:803 pip_install -U os-testr | |
| 19:07:24 | sean-k-mooney | we shoud not be using os-testr anywhere anymore | |
| 19:07:35 | sean-k-mooney | everthing should be using stestr | |
| 19:07:41 | gmann | that is installed successfully in gate I think that is using constraint? | |
| 19:07:59 | gmann | https://zuul.opendev.org/t/openstack/build/0fc9dc8ecbe748498c941c6f21cbf057/log/job-output.txt#4367 | |
| 19:08:23 | dansmith | yeah I dunno why I can't install it | |
| 19:08:31 | sean-k-mooney | its in uc ya https://opendev.org/openstack/requirements/src/branch/master/openstack_requirements/tests/files/upper-constraints.txt#L359 | |
| 19:08:37 | dansmith | unless it's a mirror sync thing? | |
| 19:08:57 | sean-k-mooney | https://opendev.org/openstack/devstack/src/branch/master/stack.sh#L803 | |
| 19:09:04 | sean-k-mooney | comment ou tthat line | |
| 19:09:26 | gmann | not this week but last week i stack successfully | |
| 19:09:53 | clarkb | note contraints don't pick what is installed. Only what version to install if something is to be installed | |
| 19:09:56 | dansmith | sean-k-mooney: heh, yeah, I could but.. | |
| 19:10:27 | sean-k-mooney | clarkb: yep i know just pointing out that it would be constrained if it was installed for other projects | |
| 19:10:39 | dansmith | I purged pbr and os-testr locally, which might have gotten past it | |
| 19:10:59 | sean-k-mooney | pip_install -U os-testr | |
| 19:11:04 | sean-k-mooney | so that is unconstiaed in devstack | |
| 19:11:18 | sean-k-mooney | unless pip_install in devstack add uc by default | |
| 19:12:23 | gmann | https://zuul.opendev.org/t/openstack/build/0fc9dc8ecbe748498c941c6f21cbf057/log/job-output.txt#4367 | |
| 19:12:37 | dansmith | sean-k-mooney: it does | |
| 19:12:47 | dansmith | I could install it myself without uc | |
| 19:12:58 | dansmith | but just purging those two packages locally seems to have worked | |
| 19:13:06 | dansmith | so maybe some not-so-correct version deps | |
| 19:13:28 | sean-k-mooney | what os are you using out of interest | |
| 19:13:29 | gmann | maybe | |
| 19:13:51 | dansmith | sean-k-mooney: focal | |
| 19:13:55 | sean-k-mooney | you said you had not stacked this year yet is it 20.04 | |
| 19:14:08 | sean-k-mooney | ok that should also work | |
| 19:14:21 | sean-k-mooney | well for the next 4-6 weeks | |
| 19:14:54 | gmann | focal should all work for latest master/constraints | |
| 19:15:03 | sean-k-mooney | yep | |
| 19:15:20 | sean-k-mooney | i just ment that after RC1 and we reopen master for bobcat | |
| 19:15:30 | sean-k-mooney | then it wont be in the testing runtime | |
| 19:15:35 | sean-k-mooney | nor wil python 3.8 | |
| 19:15:45 | sean-k-mooney | so it will proably be fine but it could start breaking | |
| 20:02:57 | opendevreview | Sofia Enriquez proposed openstack/nova master: Check NFS protocol https://review.opendev.org/c/openstack/nova/+/854030 | |
| 20:05:22 | opendevreview | Dan Smith proposed openstack/nova master: WIP: Detect host renames and abort startup https://review.opendev.org/c/openstack/nova/+/863920 | |
| #openstack-nova - 2023-01-18 | |||
| 10:01:01 | ierdem | Hi folks, is there any way to boot a signed image from volume? I am testing image validation, I can create VM by using signed images on ephemeral disks but when I try boot from volume, it throws an excepiton (https://paste.openstack.org/show/blZen5ID7OIbi47TN8ib/). I am currently working on OpenStack Ussuri, and image backend is Ceph | |
| 10:10:11 | bauzas | gibi: so, there are many ways tricking stestr scheduling | |
| 10:10:43 | bauzas | gibi: but given we directly call tempest which eventually calls stestr, the quickiest way to change the test scheduling is about renaming the testname | |
| 10:11:09 | bauzas | https://stestr.readthedocs.io/en/latest/MANUAL.html#test-scheduling | |
| 10:11:26 | bauzas | "By default stestr schedules the tests by first checking if there is any historical timing data on any tests. It then sorts the tests by that timing data loops over the tests in order and adds one to each worker that it will launch. For tests without timing data, the same is done, except the tests are in alphabetical order instead of based on timing data. If a group regex is used the same algorithm is used with groups instead of | |
| 10:11:26 | bauzas | individual tests." | |
| 10:20:12 | bauzas | gibi: https://review.opendev.org/c/openstack/tempest/+/870913 | |
| 10:20:44 | bauzas | gibi: could I modify https://review.opendev.org/c/openstack/nova/+/869900/ to be Depending on ^ ? | |
| 10:24:25 | gibi | lets keep https://review.opendev.org/c/openstack/nova/+/869900/ alone. as I want to land that regardless of our troubleshooting here. As I think some of the other problems might go away after we remove the excessive logging by that. | |
| 10:25:32 | gibi | about https://review.opendev.org/c/openstack/tempest/+/870913 why we are trying to move the test to the front? I thought we wanted to moved it later or even disable it temporarily to see if other tests are triggering the same OOM behavior and hence trying to establish a pattern causing the OOM | |
| 10:32:25 | bauzas | gibi: OK, then I'll add a DNM on nova, np | |
| 10:32:45 | bauzas | gibi: good question, I was wanting to see whether it was due to this test or not | |
| 10:33:03 | bauzas | if we call it first, and if this is due to this test, it would be killed earlier, right? | |
| 10:39:53 | opendevreview | Sylvain Bauza proposed openstack/nova master: DNM: Testing the killed test https://review.opendev.org/c/openstack/nova/+/870924 | |
| 10:44:42 | kashyap | gibi: Can you have a quick look at this workaround patch when you can (for a change all CI have passed): https://review.opendev.org/c/openstack/nova/+/870794 | |
| 10:44:47 | kashyap | (When you get a minute, that is) | |
| 10:48:15 | tobias-urdin | gibi: any possibility that we can backport this https://review.opendev.org/c/openstack/nova/+/838976 and parent reproducer patch? we are currently patching that in production as we're on newer libvirt with older nova release (xena right now, probably be yoga later this year) | |
| 11:06:50 | ierdem | Hi everyone, is there any way to boot a signed image from volume? I am testing image validation, I can create VM by using signed images on ephemeral disks but when I try boot from volume, it throws an excepiton (https://paste.openstack.org/show/blZen5ID7OIbi47TN8ib/). I am currently working on OpenStack Ussuri, and image backend is Ceph | |
| 11:15:49 | gokhanisi | hello folks, after rebooting my compute host, I can't attach my cinder volumes to instances. Nova throws "unable to lock /var/lib/nova/mnt/dgf/volume-xx for metadata change: No locks available" Full logs are in https://paste.openstack.org/show/beicZ71J17WeNwLjghKc/ What can be reason of this problem ? I am on victoria. On this compute node I am using also gpu passthrough. | |
| 11:23:15 | opendevreview | Sylvain Bauza proposed openstack/nova master: DNM: Testing the killed test https://review.opendev.org/c/openstack/nova/+/870924 | |
| 11:24:03 | gibi | tobias-urdin: regarding https://review.opendev.org/c/openstack/nova/+/838976 I think this is technically backportable but bauzas should know more about it as it is vgpu related | |
| 11:24:46 | bauzas | gibi: tobias-urdin: I already proposed the backports down to wallaby | |
| 11:25:40 | sean-k-mooney | tobias-urdin: there is a backport already | |
| 11:25:54 | sean-k-mooney | tobias-urdin: https://review.opendev.org/c/openstack/nova/+/866156 is the xena cherry pick | |
| 11:26:49 | sean-k-mooney | tobias-urdin: we needed it for wallaby for our downstream product so all the patches are up for review but we have already merged them downstream at the end of the year | |
| 11:35:48 | gibi | ahh I missed the backports as the topic was not set on them | |
| 11:39:10 | tobias-urdin | oh great, thanks! | |
| 11:39:55 | gibi | bauzas: regardin OOM I can do a parallel experiement moving the test to the latest to see if others before it trigger the OOM or not | |
| 11:40:04 | gibi | s/latest/last/ | |
| 11:40:06 | bauzas | gibi: sure, do it | |
| 11:40:10 | gibi | ack | |
| 11:40:39 | bauzas | gibi: I'm starting to look at the functest races | |
| 11:40:46 | bauzas | but I'm hungry | |
| 11:41:01 | gibi | ack. I start to get hungry too | |
| 11:41:03 | gibi | damn biology | |
| 11:47:49 | gibi | bauzas: this was the bug https://bugs.launchpad.net/nova/+bug/1946339 I referred to yesterday related to the funct test failures. It might or might not be related :/ | |
| 12:38:07 | opendevreview | melanie witt proposed openstack/nova master: imagebackend: Add support to libvirt_info for LUKS based encryption https://review.opendev.org/c/openstack/nova/+/826755 | |
| 12:38:08 | opendevreview | melanie witt proposed openstack/nova master: imagebackend: Cache the key manager when disk is encrypted https://review.opendev.org/c/openstack/nova/+/826756 | |
| 12:38:08 | opendevreview | melanie witt proposed openstack/nova master: libvirt: Introduce support for qcow2 with LUKS https://review.opendev.org/c/openstack/nova/+/772273 | |
| 12:38:09 | opendevreview | melanie witt proposed openstack/nova master: libvirt: Configure and teardown ephemeral encryption secrets https://review.opendev.org/c/openstack/nova/+/870931 | |