| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2023-07-27 | |||
| 13:35:51 | bauzas | dansmith: as we discussed with sean-k-mooney this morning, we have a job issue | |
| 13:36:37 | dansmith | I realize that, I still would like to have it ready to go, the patch below it is still waiting so +W isn't going to send it | |
| 13:37:02 | sean-k-mooney | i dont think we should slow down reviewing bacause of the job issue | |
| 13:37:15 | dansmith | agree | |
| 13:37:43 | dansmith | sean-k-mooney: I see the console_output thing, that's because it's being requested as sshable and doesn't have the validation things it needs | |
| 13:38:00 | dansmith | sean-k-mooney: not sure how that changed for that test, but I can fix it easily | |
| 13:38:31 | dansmith | sean-k-mooney: did you open a bug for it? | |
| 13:38:39 | sean-k-mooney | i fixed the validation thing | |
| 13:38:45 | sean-k-mooney | i opened a bug | |
| 13:38:52 | sean-k-mooney | and i have a ptach in ci right now | |
| 13:39:01 | sean-k-mooney | its almost completed running | |
| 13:39:02 | dansmith | link? | |
| 13:39:15 | sean-k-mooney | https://review.opendev.org/c/openstack/tempest/+/889847 | |
| 13:39:24 | sean-k-mooney | https://launchpad.net/bugs/2028851 | |
| 13:39:33 | sean-k-mooney | https://review.opendev.org/c/openstack/nova/+/889383 | |
| 13:39:44 | sean-k-mooney | i used my disk caching patch to run it with a depends on | |
| 13:39:52 | sean-k-mooney | https://zuul.openstack.org/stream/9890c9b53059410eabc0c61df0376217?logfile=console.log | |
| 13:40:02 | sean-k-mooney | that is the nova-lvm job i have it open on my other screen | |
| 13:40:03 | dansmith | oh I was looking at this: https://8d44202ac4ef26a68d85-49968699300828e6c9b78fd54dff75ef.ssl.cf1.rackcdn.com/889383/4/check/nova-lvm/368f504/testr_results.html | |
| 13:40:06 | dansmith | the in_shutoff test fail | |
| 13:40:14 | dansmith | where it claims it doesn't have validation_resourecs | |
| 13:40:36 | dansmith | sorry, no the unlimited_size one | |
| 13:40:45 | dansmith | well, both | |
| 13:40:55 | dansmith | ah, that's yours | |
| 13:40:56 | sean-k-mooney | ya in a previous revsion i enabled sshable | |
| 13:41:01 | sean-k-mooney | but not the validation resouces | |
| 13:41:07 | dansmith | ah, okay cool | |
| 13:42:15 | dansmith | okay so yep, increased load making us lose that race makes sense | |
| 13:42:37 | dansmith | it's definitely not 100% fail | |
| 13:43:07 | sean-k-mooney | its only happening in the lvm job and i think that uses lvm on a loopback file so i suspect the disk io is worse there then other jobs | |
| 13:43:13 | dansmith | because we've merged stuff in the last couple of days | |
| 13:44:09 | dansmith | er | |
| 13:44:24 | dansmith | I actually don't see that on those patches.. is that in experimental or something? | |
| 13:44:40 | sean-k-mooney | no https://github.com/openstack/nova/blob/master/.zuul.yaml#L230-L244 | |
| 13:44:48 | sean-k-mooney | but it does not run on all changes | |
| 13:45:13 | sean-k-mooney | it only runs on change to libvirt driver | |
| 13:45:15 | dansmith | ah, only on libvirt I see | |
| 13:45:16 | dansmith | okay | |
| 13:45:25 | dansmith | well in that case we *definitely* shouldn't be holding up review ;P | |
| 13:45:38 | bauzas | sorry, on an internal meeting now | |
| 13:46:22 | bauzas | dansmith: yeah, I haven't asked to hold reviews :) | |
| 13:46:45 | dansmith | bauzas: sounded like you did, but okay :) | |
| 13:47:50 | bauzas | dansmith: it's the nova-lvm job, so yeah no | |
| 13:48:08 | dansmith | ack | |
| 13:48:17 | sean-k-mooney | ill bring this up as a ptg topic but we have had the vmware driver as experimatal and deprecated for almost 2 cycle now. i think in C we should consider removing it | |
| 13:49:16 | sean-k-mooney | or at least moving it out of tree | |
| 13:51:23 | bauzas | I don't personnally disagree, since Caracal will be a SLURP release | |
| 13:52:27 | sean-k-mooney | the deprecation/experimental warning went out in antelope so we ahve warned people and i dotn think we have had any ci in maybe 2 years | |
| 13:53:14 | sean-k-mooney | there is a spec propsoed to add a new nova-mksproxy intree for vmware. when we added mks supprot in liberty the proxy server was maintained out of tree | |
| 13:53:34 | sean-k-mooney | https://opendev.org/x/nova-mksproxy | |
| 13:53:55 | sean-k-mooney | but i dont really like the idea of adding a new proxy in tree only for vmware with no testing | |
| 13:54:35 | sean-k-mooney | maintianed is proably an overly stong statement as almost nothign has been done to it in 6 years | |
| 14:02:39 | bauzas | sean-k-mooney: at least I haven't heard of any operator using vmware after Train | |
| 14:27:48 | dansmith | sean-k-mooney: still failing this one: https://d5c0e4c80966c29b54ec-9ed411199ab33310a8b6bd95d117cd22.ssl.cf1.rackcdn.com/889383/4/check/nova-lvm/9890c9b/testr_results.html | |
| 14:28:39 | sean-k-mooney | ya.. just saw | |
| 14:28:56 | sean-k-mooney | any other idea on how to fix other then disable it | |
| 14:29:39 | dansmith | that's weird, because it's failing after it has already been sshable and shutoff | |
| 14:29:42 | dansmith | oh wait, lemme check something | |
| 14:30:42 | dansmith | okay, no.. I was thinking maybe validation was disabled for this job, but it's not | |
| 14:30:46 | dansmith | so yeah, pretty weird | |
| 14:31:00 | dansmith | maybe the api times out getting the console from the compute and doesn't report failure or something? | |
| 14:45:27 | dansmith | sean-k-mooney: man, I think that's a legit failure, I just don't know why | |
| 14:46:24 | dansmith | we ssh into the guest and then we poll the console log a LOT of times before we give up and there's nothing in it | |
| 14:47:10 | dansmith | AFAICT, that basically means we lost the console for that instance, maybe because we overwrote the file or it is hooked up to a different inode than what we're reading, like if we raced to create/ensure it or something | |
| 15:03:17 | bauzas | dansmith: sean-k-mooney: we should possibly skip the test until we find the root cause | |
| 15:03:37 | dansmith | bauzas: sean-k-mooney is going to try to repro locally and if not, then skip yeah | |
| 15:03:43 | bauzas | ++ | |
| 15:03:57 | frickler | if an instance is shutoff, is the console log cleared? or should the log from the last run still be shown? | |
| 15:05:05 | dansmith | frickler: the test is specifically testing that it's not cleared | |
| 15:05:30 | dansmith | however, there are circumstances where we read the log from a pty, which is supposed to be for container stuff | |
| 15:05:42 | dansmith | but if we're getting that on the lvm config because we try not to write files, then that could be a cause | |
| 15:05:56 | dansmith | in which case the test is invalid (currently) for LVM and we were just racing before to pass it | |
| 15:06:04 | dansmith | but that is what we're going to check locally | |
| 15:07:05 | sean-k-mooney | frickler: for our default config we write it to a file on disk (or rahter libvirt/qemu does) | |
| 15:07:19 | sean-k-mooney | so we normally would epxect it to still be there when its powered off | |
| 15:07:41 | sean-k-mooney | but it gets cleard via being overwtitten on the next vm start | |
| 15:09:49 | frickler | ah, o.k., I would have expected it to get removed when the instances is shut down | |
| 15:14:38 | sean-k-mooney | for the most part we just stop the domain and dont do any cleanup of files on shutdown | |
| 15:14:57 | sean-k-mooney | and since this file is created by qemu or virtlogd depening on your libvirt config | |
| 15:15:21 | sean-k-mooney | nova does not really mange it excpetion when you delete a vm | |
| 15:15:41 | sean-k-mooney | its created under the instance dir so it gets cleaned up when that is removed | |
| 15:17:26 | frickler | makes sense when you think about it, yes | |
| 15:23:29 | sean-k-mooney | the odd thing is the test has not changed in like 6 years or more | |
| 15:23:40 | sean-k-mooney | so why it started failing 3 days ago is beyond me | |
| 15:27:40 | melwitt | I was just about to ask, was this passing recently | |
| 15:28:44 | sean-k-mooney | ya opensearch says it started failing on the 24th | |
| 15:29:20 | sean-k-mooney | dan optimised the execution time by making some of the test run across multipel workers and that seams to have exposed an existing bug | |
| 15:29:57 | sean-k-mooney | wether its a bug in nova or in the test is TBD | |
| 15:30:10 | melwitt | I can try that locally and see if it shows up for me | |
| 15:30:16 | sean-k-mooney | hence why we are repoducing it locally now. im just waiting on devstack to complete | |
| 15:30:24 | sean-k-mooney | the more the merrier | |
| 15:30:39 | melwitt | lord knows I got devstacks already available | |
| 15:31:03 | sean-k-mooney | well do you have one with lvm | |
| 15:31:20 | sean-k-mooney | this is only affecting the lvm job and ther emight eb some lvm specific logic at play | |
| 15:32:06 | melwitt | hm, no. would it be equivalent at all to do it with a boot from lvm volume? | |
| 15:32:37 | sean-k-mooney | i dont think so but its worth a try i guess if you have it to hand | |
| 15:32:56 | sean-k-mooney | the test is passing we thing in other jobs but ill admit i didnt actully check if it was running in other jobs | |
| 15:33:54 | melwitt | I'll check. given that the only condition appears to be CONF.compute_feature_enabled.console_output I'd expect it is running for other jobs | |
| 15:35:48 | melwitt | yeah it runs and passes in tempest-integrated-compute, for example | |
| 15:35:51 | sean-k-mooney | ya it is | |