| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2023-04-25 | |||
| 17:10:15 | sean-k-mooney | that does not mean it has been correct or supported for all tha time | |
| 17:10:27 | frickler | ah, I was only looking at the current blame, which says 2017 | |
| 17:10:45 | frickler | you can see the actual API calls in https://4ae644854fb3bf106e9b-6877b85dbe482cd2daa62a6731b06023.ssl.cf1.rackcdn.com/881132/3/check/tempest-full-py3/37d3ce7/controller/logs/tempest_log.txt | |
| 17:12:11 | gmann | frickler: sean-k-mooney ohk that one. those tests are meant to be cinder standalone case and they are not valid scenario involving nova in half way | |
| 17:12:32 | sean-k-mooney | right they are fine fi you are using cinder standalone | |
| 17:12:36 | frickler | POST https://213.32.75.38/compute/v2.1/servers/2a24008b-6c93-4b83-a678-8d5b0be7b6a1/os-volume_attachments | |
| 17:12:37 | gmann | nobody since starting tested if passing nova server id in attachment via cinder will work from nova perspective or not | |
| 17:12:58 | frickler | that looks like nova being used | |
| 17:13:07 | gmann | I was testing those to remove nova involvement from those tests and nova+cinder attachment anyways are tested in many other tests | |
| 17:13:32 | sean-k-mooney | yep nova should be revmoed form them | |
| 17:13:45 | gmann | frickler: attachment is directly to cinder not via nova so nova does not know about attachment but cinder think server is attched to volume so make it in-use | |
| 17:14:08 | sean-k-mooney | marking it in use is correct | |
| 17:14:22 | sean-k-mooney | but we shoudl not see the volume attaed to the vm | |
| 17:14:34 | gmann | yeah, i mean as nova does not know about attachment, adding server_id as valid attachment is not correct. | |
| 17:14:45 | gmann | hat server_id can be invalid or can be deleted anytime | |
| 17:14:50 | gmann | without cinder knopwing | |
| 17:15:05 | gmann | sean-k-mooney: yeah, VM does not know about volume | |
| 17:15:55 | sean-k-mooney | anyway im glad you are looking at it you can ping me after the patch is out of DNM if you want me to reivew | |
| 17:16:28 | gmann | k | |
| 17:17:06 | sean-k-mooney | dansmith: you pinged my yesterday to look at a patch maybe related to the stable uuid stuff | |
| 17:17:32 | sean-k-mooney | do you rememebr what it was | |
| 17:17:35 | dansmith | sean-k-mooney: the rt stuff, but it's all blocked of course.. gibi gave it the +W so it's probably good for me to just fast-approve once the gate is unblocked | |
| 17:17:49 | sean-k-mooney | oh right ya that was it | |
| 17:18:00 | sean-k-mooney | i remeber seeing it had a +w | |
| 17:18:02 | sean-k-mooney | cool | |
| 17:18:34 | dansmith | yep, thanks | |
| 17:18:47 | dansmith | I'll definitely hit you up if I need a re-review once things get unblocked | |
| 17:18:53 | dansmith | *if* they get unblocked I should say :) | |
| 17:19:17 | frickler | the pysaml revert mergen, so I think CI should be unblocked | |
| 17:19:21 | frickler | merged even | |
| 17:19:46 | dansmith | we'll see :) | |
| 17:20:28 | sean-k-mooney | im going to go get dinner. i might be around later but im mostly done for today | |
| 17:50:23 | dansmith | mmm, ceph job appears to be failing again in a similar way.. hope we don't have more work to do | |
| 18:31:21 | bauzas | dansmith: which patch are you checking for the job runs ? | |
| 18:31:46 | bauzas | so I can try to look over it tomorrow morning | |
| 18:32:05 | dansmith | https://review.opendev.org/c/openstack/devstack-plugin-ceph/+/865315 | |
| 18:32:31 | bauzas | ack, will target it tomorrow morning | |
| 18:32:39 | dansmith | it only failed six tests this time instead of a timeout, so maybe it's better than I thought | |
| 18:32:49 | dansmith | but six is still a lot, and I haven't gone through the latest logs yet | |
| 18:33:35 | bauzas | I can try to dig into those later | |
| 18:44:26 | dansmith | the fails look all volume detach related | |
| 18:44:31 | dansmith | so perhaps it's not really a ceph problem | |
| 18:44:38 | dansmith | but it seems like a large number for a single run, so I'm not sure | |
| 20:31:17 | dansmith | melwitt: can you +W this? https://review.opendev.org/c/openstack/nova/+/881409/2 | |
| 20:38:44 | dansmith | eharney: gouthamr: Well, it installs and "works" on jammy, but something isn't happy: https://review.opendev.org/c/openstack/devstack-plugin-ceph/+/865315?tab=change-view-tab-header-zuul-results-summary | |
| 20:41:21 | dansmith | eharney: gouthamr I don't really know what I'm looking at, but I don't see any errors in the cinder or ceph stuff that I recognize, and just some "busy" messages from rbd around the failed detaches in the n-cpu log | |
| 20:42:25 | gouthamr | hey dansmith - /me is late to the party | |
| 20:43:06 | dansmith | gouthamr: we need to drop focal from the jobs and our gate has been blocked for two days because some did it early..we've reverted those things for the moment, but we need to get the ceph job working on jammy | |
| 20:43:37 | dansmith | gouthamr: the above patch unpins the jobs to let them run on jammy and they get pretty far, but some volume/ceph related failures are showing that something is not happy | |
| 20:44:31 | dansmith | gouthamr: are you the right person to get that working? | |
| 20:44:49 | melwitt | dansmith: just to confirm, you still want to remove after neutron has reverted? https://review.opendev.org/c/openstack/neutron/+/881430 | |
| 20:45:27 | gouthamr | dansmith: probably not; i'm not an expert on rbd or cinder... | |
| 20:45:27 | dansmith | melwitt: yeah the neutron failure was a couple failures ago, and not even the only problem.. but as noted in the original patch, it was intended to only live for antelope and then be reverted, so we need to do it anyway | |
| 20:45:41 | dansmith | gouthamr: oh.. who is then? | |
| 20:46:10 | gouthamr | eharney is my go to guy, probably jbernard | |
| 20:46:48 | dansmith | gouthamr: okay he said earlier today that he was not likely the guy to ask (unless I misunderstood) | |
| 20:47:42 | gouthamr | ah :) let me look at the logs and see if something pops out | |
| 20:48:23 | gouthamr | we've been burnt before by using distro packages for ceph because fixes took forever to land - so we shied away from them and looked upstream.. | |
| 20:49:08 | gouthamr | but, like you've discussed, the ceph community hasn't built jammy packages for the latest release (quincy) - they meant to, they lost people/mindshare in the recent months | |
| 20:49:20 | dansmith | gouthamr: yeah, but last I checked, there were not packages from ceph themselves for jammy | |
| 20:49:34 | dansmith | gouthamr: and the cephadm job is even more broken and marked n-v so I assume it's not healthier | |
| 20:49:48 | gouthamr | on the manila jobs, we pivoted to use centos-stream-9 because ceph folks continue to publish packages there | |
| 20:49:54 | gouthamr | s/there/for it | |
| 20:50:29 | gouthamr | dansmith: yep; on that change, i see there's a problem with podman... | |
| 20:50:35 | dansmith | gouthamr: yeah, but stream breaks us constantly | |
| 20:50:40 | gouthamr | oh | |
| 20:50:54 | dansmith | there's very little chance we're going to be able to have this job run on stream :) | |
| 20:51:15 | dansmith | gouthamr: yeah I see the podman thing, but the job is also marked non-voting | |
| 20:51:26 | dansmith | so I assume it doesn't have a long track record of stability :) | |
| 20:52:03 | gouthamr | yes; i don't think we've run the job long enough to test for stability: https://zuul.opendev.org/t/openstack/builds?job_name=devstack-plugin-ceph-cephfs-nfs | |
| 20:52:15 | dansmith | presumably the cephadm approach means we could run on jammy but with upstream ceph fixes | |
| 20:52:26 | gouthamr | yep | |
| 20:52:37 | dansmith | gouthamr: that's the cephfs job, but I assume that's not what nova needs | |
| 20:52:57 | dansmith | I'm looking at devstack-plugin-ceph-tempest-cephadm | |
| 20:53:01 | gouthamr | yes; just pointing out that job because it uses centos-9-stream | |
| 20:53:07 | dansmith | ah okay | |
| 20:53:14 | gouthamr | ack; | |
| 20:53:32 | gouthamr | "devstack-plugin-ceph-tempest-cephadm" on focal-fossa used a third party repo to get podman | |
| 20:54:02 | dansmith | ah, and podman is in jammy itself I think right? | |
| 20:54:12 | gouthamr | by the looks of it, yes | |
| 20:54:30 | dansmith | although it seems broken :) | |
| 20:55:08 | dansmith | anyway, I thought there was also some concern that the cephadm job didn't expose the ceph config that nova needed or something like that, but I heard that like 20th hand | |
| 20:55:27 | gouthamr | shouldn't be the case | |
| 20:55:55 | dansmith | okay | |
| 20:56:23 | gouthamr | i think we tested some of this without tempest in the picture - but that job ("devstack-plugin-ceph-tempest-cephadm") has never passed; we assumed someone working on nova/cinder/glance would help looking at it at some point | |
| 20:57:32 | gouthamr | sorry this feels disjointed - conversations happened on irc and gerrit, ptg and the ML iirc.. but its time to reprise this because it's urgent.. | |
| 20:57:49 | gouthamr | that's a tangent though, let me see if i can spot an issue with the package based job you're looking to fix | |
| 20:58:03 | dansmith | ugh, never passed? that's no good.. I wonder if for the same reason the non-cephadm job is failing/ | |
| 20:58:21 | dansmith | I can try to get podman working on this to see if it's otherwise the same | |
| 20:58:26 | gouthamr | ++ | |
| 20:58:41 | dansmith | gouthamr: this is definitely disjointed, and I feel like I'm just flailing because nobody else is :/ | |
| 20:59:25 | gouthamr | you're doing godly work :D | |
| 20:59:53 | dansmith | $deitly work you mean :) | |
| 21:00:07 | dansmith | er, $deityly .. or soething | |
| 21:00:18 | gouthamr | :P | |
| 21:00:32 | dansmith | okay I just pushed something that might get podman working based on that error message, so we'll see | |
| 21:01:34 | dansmith | melwitt: thanks for the +W.. I would have just removed that neutron reference and changed to "effing everything" but didn't want to have to make another trip through the jobs, as you can probably imagine :) | |
| 21:06:04 | dansmith | gouthamr: the cephadm that has never passed.. is that always on quincy (i.e. newer than what we were running in focal) or what? | |
| 21:06:08 | dansmith | I | |
| 21:06:29 | melwitt | dansmith: understandable :) | |