| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2023-07-27 | |||
| 10:53:42 | bauzas | yeah sounds plausible | |
| 10:54:06 | sean-k-mooney | im just going to change server = self.create_test_server(wait_until='ACTIVE') | |
| 10:54:08 | sean-k-mooney | to server = self.create_test_server(wait_until='SSHABLE') | |
| 10:56:09 | bauzas | shoot | |
| 10:56:20 | bauzas | and depends-on it with a nova dnm please | |
| 10:56:31 | sean-k-mooney | yep | |
| 10:57:37 | sean-k-mooney | now i want one :) | |
| 10:57:45 | sean-k-mooney | enjoy | |
| 11:03:46 | opendevreview | sean mooney proposed openstack/nova master: use disk caching to hide slow cinder performance https://review.opendev.org/c/openstack/nova/+/889383 | |
| 11:04:40 | sean-k-mooney | ill use my diskcache mode change to test it instead ^ that way we can test both and save ci time | |
| 12:56:42 | opendevreview | alecorps proposed openstack/nova-specs master: new spec: support of vnc console for mks https://review.opendev.org/c/openstack/nova-specs/+/889857 | |
| 13:29:45 | dansmith | sean-k-mooney: if you could use my nick so I get a notify I'd appreciate it | |
| 13:29:52 | dansmith | I can't notify on "dan" because it's far too common | |
| 13:30:03 | dansmith | sean-k-mooney: so which test are you changing from active to sshable? | |
| 13:30:29 | dansmith | the split patch literally just moved code, so it shouldn't affect anything | |
| 13:30:59 | dansmith | it will increase parallelism, but we usually are running that test when the workers are fully loaded with tests, not after they've run out of work, | |
| 13:31:10 | dansmith | so I wouldn't expect it to really change the profile much | |
| 13:35:18 | dansmith | bauzas: can you please get this queued up? you've already +2d the follow-up to address your feedback: https://review.opendev.org/c/openstack/nova/+/884752/2 | |
| 13:35:30 | bauzas | dansmith: cool | |
| 13:35:37 | dansmith | I've been rechecking these for three weeks and I'm getting close, so I want to be able to land this when the time comes | |
| 13:35:51 | bauzas | dansmith: as we discussed with sean-k-mooney this morning, we have a job issue | |
| 13:36:37 | dansmith | I realize that, I still would like to have it ready to go, the patch below it is still waiting so +W isn't going to send it | |
| 13:37:02 | sean-k-mooney | i dont think we should slow down reviewing bacause of the job issue | |
| 13:37:15 | dansmith | agree | |
| 13:37:43 | dansmith | sean-k-mooney: I see the console_output thing, that's because it's being requested as sshable and doesn't have the validation things it needs | |
| 13:38:00 | dansmith | sean-k-mooney: not sure how that changed for that test, but I can fix it easily | |
| 13:38:31 | dansmith | sean-k-mooney: did you open a bug for it? | |
| 13:38:39 | sean-k-mooney | i fixed the validation thing | |
| 13:38:45 | sean-k-mooney | i opened a bug | |
| 13:38:52 | sean-k-mooney | and i have a ptach in ci right now | |
| 13:39:01 | sean-k-mooney | its almost completed running | |
| 13:39:02 | dansmith | link? | |
| 13:39:15 | sean-k-mooney | https://review.opendev.org/c/openstack/tempest/+/889847 | |
| 13:39:24 | sean-k-mooney | https://launchpad.net/bugs/2028851 | |
| 13:39:33 | sean-k-mooney | https://review.opendev.org/c/openstack/nova/+/889383 | |
| 13:39:44 | sean-k-mooney | i used my disk caching patch to run it with a depends on | |
| 13:39:52 | sean-k-mooney | https://zuul.openstack.org/stream/9890c9b53059410eabc0c61df0376217?logfile=console.log | |
| 13:40:02 | sean-k-mooney | that is the nova-lvm job i have it open on my other screen | |
| 13:40:03 | dansmith | oh I was looking at this: https://8d44202ac4ef26a68d85-49968699300828e6c9b78fd54dff75ef.ssl.cf1.rackcdn.com/889383/4/check/nova-lvm/368f504/testr_results.html | |
| 13:40:06 | dansmith | the in_shutoff test fail | |
| 13:40:14 | dansmith | where it claims it doesn't have validation_resourecs | |
| 13:40:36 | dansmith | sorry, no the unlimited_size one | |
| 13:40:45 | dansmith | well, both | |
| 13:40:55 | dansmith | ah, that's yours | |
| 13:40:56 | sean-k-mooney | ya in a previous revsion i enabled sshable | |
| 13:41:01 | sean-k-mooney | but not the validation resouces | |
| 13:41:07 | dansmith | ah, okay cool | |
| 13:42:15 | dansmith | okay so yep, increased load making us lose that race makes sense | |
| 13:42:37 | dansmith | it's definitely not 100% fail | |
| 13:43:07 | sean-k-mooney | its only happening in the lvm job and i think that uses lvm on a loopback file so i suspect the disk io is worse there then other jobs | |
| 13:43:13 | dansmith | because we've merged stuff in the last couple of days | |
| 13:44:09 | dansmith | er | |
| 13:44:24 | dansmith | I actually don't see that on those patches.. is that in experimental or something? | |
| 13:44:40 | sean-k-mooney | no https://github.com/openstack/nova/blob/master/.zuul.yaml#L230-L244 | |
| 13:44:48 | sean-k-mooney | but it does not run on all changes | |
| 13:45:13 | sean-k-mooney | it only runs on change to libvirt driver | |
| 13:45:15 | dansmith | ah, only on libvirt I see | |
| 13:45:16 | dansmith | okay | |
| 13:45:25 | dansmith | well in that case we *definitely* shouldn't be holding up review ;P | |
| 13:45:38 | bauzas | sorry, on an internal meeting now | |
| 13:46:22 | bauzas | dansmith: yeah, I haven't asked to hold reviews :) | |
| 13:46:45 | dansmith | bauzas: sounded like you did, but okay :) | |
| 13:47:50 | bauzas | dansmith: it's the nova-lvm job, so yeah no | |
| 13:48:08 | dansmith | ack | |
| 13:48:17 | sean-k-mooney | ill bring this up as a ptg topic but we have had the vmware driver as experimatal and deprecated for almost 2 cycle now. i think in C we should consider removing it | |
| 13:49:16 | sean-k-mooney | or at least moving it out of tree | |
| 13:51:23 | bauzas | I don't personnally disagree, since Caracal will be a SLURP release | |
| 13:52:27 | sean-k-mooney | the deprecation/experimental warning went out in antelope so we ahve warned people and i dotn think we have had any ci in maybe 2 years | |
| 13:53:14 | sean-k-mooney | there is a spec propsoed to add a new nova-mksproxy intree for vmware. when we added mks supprot in liberty the proxy server was maintained out of tree | |
| 13:53:34 | sean-k-mooney | https://opendev.org/x/nova-mksproxy | |
| 13:53:55 | sean-k-mooney | but i dont really like the idea of adding a new proxy in tree only for vmware with no testing | |
| 13:54:35 | sean-k-mooney | maintianed is proably an overly stong statement as almost nothign has been done to it in 6 years | |
| 14:02:39 | bauzas | sean-k-mooney: at least I haven't heard of any operator using vmware after Train | |
| 14:27:48 | dansmith | sean-k-mooney: still failing this one: https://d5c0e4c80966c29b54ec-9ed411199ab33310a8b6bd95d117cd22.ssl.cf1.rackcdn.com/889383/4/check/nova-lvm/9890c9b/testr_results.html | |
| 14:28:39 | sean-k-mooney | ya.. just saw | |
| 14:28:56 | sean-k-mooney | any other idea on how to fix other then disable it | |
| 14:29:39 | dansmith | that's weird, because it's failing after it has already been sshable and shutoff | |
| 14:29:42 | dansmith | oh wait, lemme check something | |
| 14:30:42 | dansmith | okay, no.. I was thinking maybe validation was disabled for this job, but it's not | |
| 14:30:46 | dansmith | so yeah, pretty weird | |
| 14:31:00 | dansmith | maybe the api times out getting the console from the compute and doesn't report failure or something? | |
| 14:45:27 | dansmith | sean-k-mooney: man, I think that's a legit failure, I just don't know why | |
| 14:46:24 | dansmith | we ssh into the guest and then we poll the console log a LOT of times before we give up and there's nothing in it | |
| 14:47:10 | dansmith | AFAICT, that basically means we lost the console for that instance, maybe because we overwrote the file or it is hooked up to a different inode than what we're reading, like if we raced to create/ensure it or something | |
| 15:03:17 | bauzas | dansmith: sean-k-mooney: we should possibly skip the test until we find the root cause | |
| 15:03:37 | dansmith | bauzas: sean-k-mooney is going to try to repro locally and if not, then skip yeah | |
| 15:03:43 | bauzas | ++ | |
| 15:03:57 | frickler | if an instance is shutoff, is the console log cleared? or should the log from the last run still be shown? | |
| 15:05:05 | dansmith | frickler: the test is specifically testing that it's not cleared | |
| 15:05:30 | dansmith | however, there are circumstances where we read the log from a pty, which is supposed to be for container stuff | |
| 15:05:42 | dansmith | but if we're getting that on the lvm config because we try not to write files, then that could be a cause | |
| 15:05:56 | dansmith | in which case the test is invalid (currently) for LVM and we were just racing before to pass it | |
| 15:06:04 | dansmith | but that is what we're going to check locally | |
| 15:07:05 | sean-k-mooney | frickler: for our default config we write it to a file on disk (or rahter libvirt/qemu does) | |
| 15:07:19 | sean-k-mooney | so we normally would epxect it to still be there when its powered off | |
| 15:07:41 | sean-k-mooney | but it gets cleard via being overwtitten on the next vm start | |
| 15:09:49 | frickler | ah, o.k., I would have expected it to get removed when the instances is shut down | |
| 15:14:38 | sean-k-mooney | for the most part we just stop the domain and dont do any cleanup of files on shutdown | |
| 15:14:57 | sean-k-mooney | and since this file is created by qemu or virtlogd depening on your libvirt config | |
| 15:15:21 | sean-k-mooney | nova does not really mange it excpetion when you delete a vm | |