| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2023-04-25 | |||
| 16:48:59 | auniyal | sorry I didn't get, any action on above | |
| 16:49:20 | auniyal | we need to fix tempest tests ? | |
| 16:49:47 | sean-k-mooney | i think just continue to reheck it. gibi found at least on test that is not waiting for sshable | |
| 16:49:53 | bauzas | no, we have some tempest patches up | |
| 16:49:57 | sean-k-mooney | and notice other dont appear to eb waiting but i dont have the context | |
| 16:50:01 | bauzas | and yoga would benefit from those | |
| 16:50:10 | bauzas | since tempest is branchless | |
| 16:50:14 | sean-k-mooney | oh do you have a link? | |
| 16:50:16 | auniyal | ack thanks | |
| 16:51:54 | bauzas | I was referring to gibi's recent discoveries of testing gap for ssh wait | |
| 16:52:13 | bauzas | (sorry was looking at the -tc meeting) | |
| 16:52:23 | bauzas | -tc chan* | |
| 16:52:35 | bauzas | can we close this meeting now ? | |
| 16:52:38 | sean-k-mooney | its fine we can wrap this here and chat after | |
| 16:53:06 | bauzas | cool | |
| 16:53:08 | bauzas | thanks all | |
| 16:53:12 | bauzas | #endmeeting | |
| 16:53:12 | opendevmeet | Meeting ended Tue Apr 25 16:53:12 2023 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | |
| 16:53:12 | opendevmeet | Minutes: https://meetings.opendev.org/meetings/nova/2023/nova.2023-04-25-16.00.html | |
| 16:53:12 | opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/nova/2023/nova.2023-04-25-16.00.txt | |
| 16:53:12 | opendevmeet | Log: https://meetings.opendev.org/meetings/nova/2023/nova.2023-04-25-16.00.log.html | |
| 16:53:14 | sean-k-mooney | https://review.opendev.org/c/openstack/tempest/+/880891 | |
| 16:53:17 | sean-k-mooney | seams to be related | |
| 16:53:43 | elodilles | thanks o/ | |
| 17:01:21 | sean-k-mooney | so that test is wrong | |
| 17:01:40 | sean-k-mooney | we do not supprot attaching or detaching prots or volume from neuton or cinder | |
| 17:02:28 | frickler | so we run this test for 6 years, have issues with it time and again, and only now notice that it tests an unsupported scenario? cool | |
| 17:03:33 | sean-k-mooney | frickler: it has never been supported | |
| 17:03:49 | sean-k-mooney | i jsut notice this existed because gmann has a dnm patch up | |
| 17:04:23 | gmann | which one ? | |
| 17:04:28 | frickler | yes, saw the comment in the patch. also didn't want to blame anyone, just enjoying the wondrous world of openstack once again | |
| 17:04:32 | sean-k-mooney | https://github.com/openstack/tempest/blob/master/tempest/api/volume/test_volumes_actions.py#L39-L55 | |
| 17:04:34 | frickler | https://review.opendev.org/c/openstack/tempest/+/881132/3 | |
| 17:05:18 | sean-k-mooney | it kind of depend on what self.volumes_client.attach_volume actully does | |
| 17:05:22 | frickler | ah, no, the one below | |
| 17:05:22 | sean-k-mooney | if its calling nova its fine | |
| 17:05:37 | sean-k-mooney | if its using the cinder attachments api directly its not | |
| 17:07:16 | sean-k-mooney | that looks like its callining cinder https://github.com/openstack/tempest/blob/20e460dacfae6b4546903a9caaf9253330f27b5a/tempest/clients.py#L286 | |
| 17:10:02 | sean-k-mooney | frickler: this was actully added 11 years ago https://github.com/openstack/tempest/commit/a42fe441703084449107fabb15fe42938c02ba08 | |
| 17:10:15 | sean-k-mooney | that does not mean it has been correct or supported for all tha time | |
| 17:10:27 | frickler | ah, I was only looking at the current blame, which says 2017 | |
| 17:10:45 | frickler | you can see the actual API calls in https://4ae644854fb3bf106e9b-6877b85dbe482cd2daa62a6731b06023.ssl.cf1.rackcdn.com/881132/3/check/tempest-full-py3/37d3ce7/controller/logs/tempest_log.txt | |
| 17:12:11 | gmann | frickler: sean-k-mooney ohk that one. those tests are meant to be cinder standalone case and they are not valid scenario involving nova in half way | |
| 17:12:32 | sean-k-mooney | right they are fine fi you are using cinder standalone | |
| 17:12:36 | frickler | POST https://213.32.75.38/compute/v2.1/servers/2a24008b-6c93-4b83-a678-8d5b0be7b6a1/os-volume_attachments | |
| 17:12:37 | gmann | nobody since starting tested if passing nova server id in attachment via cinder will work from nova perspective or not | |
| 17:12:58 | frickler | that looks like nova being used | |
| 17:13:07 | gmann | I was testing those to remove nova involvement from those tests and nova+cinder attachment anyways are tested in many other tests | |
| 17:13:32 | sean-k-mooney | yep nova should be revmoed form them | |
| 17:13:45 | gmann | frickler: attachment is directly to cinder not via nova so nova does not know about attachment but cinder think server is attched to volume so make it in-use | |
| 17:14:08 | sean-k-mooney | marking it in use is correct | |
| 17:14:22 | sean-k-mooney | but we shoudl not see the volume attaed to the vm | |
| 17:14:34 | gmann | yeah, i mean as nova does not know about attachment, adding server_id as valid attachment is not correct. | |
| 17:14:45 | gmann | hat server_id can be invalid or can be deleted anytime | |
| 17:14:50 | gmann | without cinder knopwing | |
| 17:15:05 | gmann | sean-k-mooney: yeah, VM does not know about volume | |
| 17:15:55 | sean-k-mooney | anyway im glad you are looking at it you can ping me after the patch is out of DNM if you want me to reivew | |
| 17:16:28 | gmann | k | |
| 17:17:06 | sean-k-mooney | dansmith: you pinged my yesterday to look at a patch maybe related to the stable uuid stuff | |
| 17:17:32 | sean-k-mooney | do you rememebr what it was | |
| 17:17:35 | dansmith | sean-k-mooney: the rt stuff, but it's all blocked of course.. gibi gave it the +W so it's probably good for me to just fast-approve once the gate is unblocked | |
| 17:17:49 | sean-k-mooney | oh right ya that was it | |
| 17:18:00 | sean-k-mooney | i remeber seeing it had a +w | |
| 17:18:02 | sean-k-mooney | cool | |
| 17:18:34 | dansmith | yep, thanks | |
| 17:18:47 | dansmith | I'll definitely hit you up if I need a re-review once things get unblocked | |
| 17:18:53 | dansmith | *if* they get unblocked I should say :) | |
| 17:19:17 | frickler | the pysaml revert mergen, so I think CI should be unblocked | |
| 17:19:21 | frickler | merged even | |
| 17:19:46 | dansmith | we'll see :) | |
| 17:20:28 | sean-k-mooney | im going to go get dinner. i might be around later but im mostly done for today | |
| 17:50:23 | dansmith | mmm, ceph job appears to be failing again in a similar way.. hope we don't have more work to do | |
| 18:31:21 | bauzas | dansmith: which patch are you checking for the job runs ? | |
| 18:31:46 | bauzas | so I can try to look over it tomorrow morning | |
| 18:32:05 | dansmith | https://review.opendev.org/c/openstack/devstack-plugin-ceph/+/865315 | |
| 18:32:31 | bauzas | ack, will target it tomorrow morning | |
| 18:32:39 | dansmith | it only failed six tests this time instead of a timeout, so maybe it's better than I thought | |
| 18:32:49 | dansmith | but six is still a lot, and I haven't gone through the latest logs yet | |
| 18:33:35 | bauzas | I can try to dig into those later | |
| 18:44:26 | dansmith | the fails look all volume detach related | |
| 18:44:31 | dansmith | so perhaps it's not really a ceph problem | |
| 18:44:38 | dansmith | but it seems like a large number for a single run, so I'm not sure | |
| 20:31:17 | dansmith | melwitt: can you +W this? https://review.opendev.org/c/openstack/nova/+/881409/2 | |
| 20:38:44 | dansmith | eharney: gouthamr: Well, it installs and "works" on jammy, but something isn't happy: https://review.opendev.org/c/openstack/devstack-plugin-ceph/+/865315?tab=change-view-tab-header-zuul-results-summary | |
| 20:41:21 | dansmith | eharney: gouthamr I don't really know what I'm looking at, but I don't see any errors in the cinder or ceph stuff that I recognize, and just some "busy" messages from rbd around the failed detaches in the n-cpu log | |
| 20:42:25 | gouthamr | hey dansmith - /me is late to the party | |
| 20:43:06 | dansmith | gouthamr: we need to drop focal from the jobs and our gate has been blocked for two days because some did it early..we've reverted those things for the moment, but we need to get the ceph job working on jammy | |
| 20:43:37 | dansmith | gouthamr: the above patch unpins the jobs to let them run on jammy and they get pretty far, but some volume/ceph related failures are showing that something is not happy | |
| 20:44:31 | dansmith | gouthamr: are you the right person to get that working? | |
| 20:44:49 | melwitt | dansmith: just to confirm, you still want to remove after neutron has reverted? https://review.opendev.org/c/openstack/neutron/+/881430 | |
| 20:45:27 | dansmith | melwitt: yeah the neutron failure was a couple failures ago, and not even the only problem.. but as noted in the original patch, it was intended to only live for antelope and then be reverted, so we need to do it anyway | |
| 20:45:27 | gouthamr | dansmith: probably not; i'm not an expert on rbd or cinder... | |
| 20:45:41 | dansmith | gouthamr: oh.. who is then? | |
| 20:46:10 | gouthamr | eharney is my go to guy, probably jbernard | |
| 20:46:48 | dansmith | gouthamr: okay he said earlier today that he was not likely the guy to ask (unless I misunderstood) | |
| 20:47:42 | gouthamr | ah :) let me look at the logs and see if something pops out | |
| 20:48:23 | gouthamr | we've been burnt before by using distro packages for ceph because fixes took forever to land - so we shied away from them and looked upstream.. | |
| 20:49:08 | gouthamr | but, like you've discussed, the ceph community hasn't built jammy packages for the latest release (quincy) - they meant to, they lost people/mindshare in the recent months | |
| 20:49:20 | dansmith | gouthamr: yeah, but last I checked, there were not packages from ceph themselves for jammy | |
| 20:49:34 | dansmith | gouthamr: and the cephadm job is even more broken and marked n-v so I assume it's not healthier | |