| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2023-07-27 | |||
| 13:44:09 | dansmith | er | |
| 13:44:24 | dansmith | I actually don't see that on those patches.. is that in experimental or something? | |
| 13:44:40 | sean-k-mooney | no https://github.com/openstack/nova/blob/master/.zuul.yaml#L230-L244 | |
| 13:44:48 | sean-k-mooney | but it does not run on all changes | |
| 13:45:13 | sean-k-mooney | it only runs on change to libvirt driver | |
| 13:45:15 | dansmith | ah, only on libvirt I see | |
| 13:45:16 | dansmith | okay | |
| 13:45:25 | dansmith | well in that case we *definitely* shouldn't be holding up review ;P | |
| 13:45:38 | bauzas | sorry, on an internal meeting now | |
| 13:46:22 | bauzas | dansmith: yeah, I haven't asked to hold reviews :) | |
| 13:46:45 | dansmith | bauzas: sounded like you did, but okay :) | |
| 13:47:50 | bauzas | dansmith: it's the nova-lvm job, so yeah no | |
| 13:48:08 | dansmith | ack | |
| 13:48:17 | sean-k-mooney | ill bring this up as a ptg topic but we have had the vmware driver as experimatal and deprecated for almost 2 cycle now. i think in C we should consider removing it | |
| 13:49:16 | sean-k-mooney | or at least moving it out of tree | |
| 13:51:23 | bauzas | I don't personnally disagree, since Caracal will be a SLURP release | |
| 13:52:27 | sean-k-mooney | the deprecation/experimental warning went out in antelope so we ahve warned people and i dotn think we have had any ci in maybe 2 years | |
| 13:53:14 | sean-k-mooney | there is a spec propsoed to add a new nova-mksproxy intree for vmware. when we added mks supprot in liberty the proxy server was maintained out of tree | |
| 13:53:34 | sean-k-mooney | https://opendev.org/x/nova-mksproxy | |
| 13:53:55 | sean-k-mooney | but i dont really like the idea of adding a new proxy in tree only for vmware with no testing | |
| 13:54:35 | sean-k-mooney | maintianed is proably an overly stong statement as almost nothign has been done to it in 6 years | |
| 14:02:39 | bauzas | sean-k-mooney: at least I haven't heard of any operator using vmware after Train | |
| 14:27:48 | dansmith | sean-k-mooney: still failing this one: https://d5c0e4c80966c29b54ec-9ed411199ab33310a8b6bd95d117cd22.ssl.cf1.rackcdn.com/889383/4/check/nova-lvm/9890c9b/testr_results.html | |
| 14:28:39 | sean-k-mooney | ya.. just saw | |
| 14:28:56 | sean-k-mooney | any other idea on how to fix other then disable it | |
| 14:29:39 | dansmith | that's weird, because it's failing after it has already been sshable and shutoff | |
| 14:29:42 | dansmith | oh wait, lemme check something | |
| 14:30:42 | dansmith | okay, no.. I was thinking maybe validation was disabled for this job, but it's not | |
| 14:30:46 | dansmith | so yeah, pretty weird | |
| 14:31:00 | dansmith | maybe the api times out getting the console from the compute and doesn't report failure or something? | |
| 14:45:27 | dansmith | sean-k-mooney: man, I think that's a legit failure, I just don't know why | |
| 14:46:24 | dansmith | we ssh into the guest and then we poll the console log a LOT of times before we give up and there's nothing in it | |
| 14:47:10 | dansmith | AFAICT, that basically means we lost the console for that instance, maybe because we overwrote the file or it is hooked up to a different inode than what we're reading, like if we raced to create/ensure it or something | |
| 15:03:17 | bauzas | dansmith: sean-k-mooney: we should possibly skip the test until we find the root cause | |
| 15:03:37 | dansmith | bauzas: sean-k-mooney is going to try to repro locally and if not, then skip yeah | |
| 15:03:43 | bauzas | ++ | |
| 15:03:57 | frickler | if an instance is shutoff, is the console log cleared? or should the log from the last run still be shown? | |
| 15:05:05 | dansmith | frickler: the test is specifically testing that it's not cleared | |
| 15:05:30 | dansmith | however, there are circumstances where we read the log from a pty, which is supposed to be for container stuff | |
| 15:05:42 | dansmith | but if we're getting that on the lvm config because we try not to write files, then that could be a cause | |
| 15:05:56 | dansmith | in which case the test is invalid (currently) for LVM and we were just racing before to pass it | |
| 15:06:04 | dansmith | but that is what we're going to check locally | |
| 15:07:05 | sean-k-mooney | frickler: for our default config we write it to a file on disk (or rahter libvirt/qemu does) | |
| 15:07:19 | sean-k-mooney | so we normally would epxect it to still be there when its powered off | |
| 15:07:41 | sean-k-mooney | but it gets cleard via being overwtitten on the next vm start | |
| 15:09:49 | frickler | ah, o.k., I would have expected it to get removed when the instances is shut down | |
| 15:14:38 | sean-k-mooney | for the most part we just stop the domain and dont do any cleanup of files on shutdown | |
| 15:14:57 | sean-k-mooney | and since this file is created by qemu or virtlogd depening on your libvirt config | |
| 15:15:21 | sean-k-mooney | nova does not really mange it excpetion when you delete a vm | |
| 15:15:41 | sean-k-mooney | its created under the instance dir so it gets cleaned up when that is removed | |
| 15:17:26 | frickler | makes sense when you think about it, yes | |
| 15:23:29 | sean-k-mooney | the odd thing is the test has not changed in like 6 years or more | |
| 15:23:40 | sean-k-mooney | so why it started failing 3 days ago is beyond me | |
| 15:27:40 | melwitt | I was just about to ask, was this passing recently | |
| 15:28:44 | sean-k-mooney | ya opensearch says it started failing on the 24th | |
| 15:29:20 | sean-k-mooney | dan optimised the execution time by making some of the test run across multipel workers and that seams to have exposed an existing bug | |
| 15:29:57 | sean-k-mooney | wether its a bug in nova or in the test is TBD | |
| 15:30:10 | melwitt | I can try that locally and see if it shows up for me | |
| 15:30:16 | sean-k-mooney | hence why we are repoducing it locally now. im just waiting on devstack to complete | |
| 15:30:24 | sean-k-mooney | the more the merrier | |
| 15:30:39 | melwitt | lord knows I got devstacks already available | |
| 15:31:03 | sean-k-mooney | well do you have one with lvm | |
| 15:31:20 | sean-k-mooney | this is only affecting the lvm job and ther emight eb some lvm specific logic at play | |
| 15:32:06 | melwitt | hm, no. would it be equivalent at all to do it with a boot from lvm volume? | |
| 15:32:37 | sean-k-mooney | i dont think so but its worth a try i guess if you have it to hand | |
| 15:32:56 | sean-k-mooney | the test is passing we thing in other jobs but ill admit i didnt actully check if it was running in other jobs | |
| 15:33:54 | melwitt | I'll check. given that the only condition appears to be CONF.compute_feature_enabled.console_output I'd expect it is running for other jobs | |
| 15:35:48 | melwitt | yeah it runs and passes in tempest-integrated-compute, for example | |
| 15:35:51 | sean-k-mooney | ya it is | |
| 15:36:05 | sean-k-mooney | also in nova-ceph-multistore | |
| 15:41:02 | dansmith | yeah, and also note that sean-k-mooney has proposed changes to it, | |
| 15:41:07 | dansmith | which may make it even more lvm-specific | |
| 15:47:26 | sean-k-mooney | ... typo in my local.conf, take 2 i guess | |
| 15:51:56 | melwitt | well, I'm getting weird results in both the bfv and non bfv cases I get a 404 error when requesting the console log. not sure if that's just something wrong in my environment | |
| 15:52:24 | melwitt | meaning, I know it's passing in non lvm jobs so why is it doing this for me | |
| 15:53:04 | sean-k-mooney | not that could be the same issue | |
| 15:54:58 | dansmith | the lvm jobs are lvm backend right? | |
| 15:55:04 | dansmith | so really have nothing to do with a bfv instance | |
| 15:55:25 | sean-k-mooney | yes image_type=lvm | |
| 15:55:45 | melwitt | like, the code is relying on the guest xml to get the console log path, but a stopped instance won't have xml, so how is this ever working? | |
| 15:56:17 | melwitt | I must be missing something | |
| 15:56:24 | sean-k-mooney | it does | |
| 15:56:31 | sean-k-mooney | we dont undefien the xml when we power off | |
| 15:57:19 | sean-k-mooney | we delete the domain and recreate it every tiem we start a domian but power off leaves it in place | |
| 15:57:34 | melwitt | hm ok | |
| 15:57:39 | sean-k-mooney | we use that fact to make vGPU work by reading the mdev path out of the xml | |
| 15:58:48 | dansmith | yeah, it has xml when stopped | |
| 15:59:16 | dansmith | that's the "defined" vs. "active | |
| 15:59:19 | dansmith | " xml | |
| 15:59:33 | melwitt | ok, I'm not able to get the domain with 'virsh dumpxml [--inactive] <uuid>' so far I dunno how to look at it | |
| 15:59:41 | melwitt | gotcha | |
| 16:00:02 | sean-k-mooney | do virsh list --all | |
| 16:00:21 | dansmith | you need to use the id not the uuid | |
| 16:00:23 | melwitt | ok, thanks I see them | |
| 16:00:38 | melwitt | id is blank but it has "name" so I'll try that | |
| 16:00:58 | dansmith | oh for the inactive yeah | |
| 16:01:11 | sean-k-mooney | ok stacked lets boot a vm and see what happened | |
| 16:01:21 | dansmith | but uuid is just metadata for libvirt AFAIK.. I've never known to be able to use that for virsh commands | |
| 16:02:01 | melwitt | you can. I had been doing that to look at inactive domain xml | |
| 16:03:12 | melwitt | ok got it. I wasn't using sudo 🙄 | |