| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2023-07-27 | |||
| 15:30:10 | melwitt | I can try that locally and see if it shows up for me | |
| 15:30:16 | sean-k-mooney | hence why we are repoducing it locally now. im just waiting on devstack to complete | |
| 15:30:24 | sean-k-mooney | the more the merrier | |
| 15:30:39 | melwitt | lord knows I got devstacks already available | |
| 15:31:03 | sean-k-mooney | well do you have one with lvm | |
| 15:31:20 | sean-k-mooney | this is only affecting the lvm job and ther emight eb some lvm specific logic at play | |
| 15:32:06 | melwitt | hm, no. would it be equivalent at all to do it with a boot from lvm volume? | |
| 15:32:37 | sean-k-mooney | i dont think so but its worth a try i guess if you have it to hand | |
| 15:32:56 | sean-k-mooney | the test is passing we thing in other jobs but ill admit i didnt actully check if it was running in other jobs | |
| 15:33:54 | melwitt | I'll check. given that the only condition appears to be CONF.compute_feature_enabled.console_output I'd expect it is running for other jobs | |
| 15:35:48 | melwitt | yeah it runs and passes in tempest-integrated-compute, for example | |
| 15:35:51 | sean-k-mooney | ya it is | |
| 15:36:05 | sean-k-mooney | also in nova-ceph-multistore | |
| 15:41:02 | dansmith | yeah, and also note that sean-k-mooney has proposed changes to it, | |
| 15:41:07 | dansmith | which may make it even more lvm-specific | |
| 15:47:26 | sean-k-mooney | ... typo in my local.conf, take 2 i guess | |
| 15:51:56 | melwitt | well, I'm getting weird results in both the bfv and non bfv cases I get a 404 error when requesting the console log. not sure if that's just something wrong in my environment | |
| 15:52:24 | melwitt | meaning, I know it's passing in non lvm jobs so why is it doing this for me | |
| 15:53:04 | sean-k-mooney | not that could be the same issue | |
| 15:54:58 | dansmith | the lvm jobs are lvm backend right? | |
| 15:55:04 | dansmith | so really have nothing to do with a bfv instance | |
| 15:55:25 | sean-k-mooney | yes image_type=lvm | |
| 15:55:45 | melwitt | like, the code is relying on the guest xml to get the console log path, but a stopped instance won't have xml, so how is this ever working? | |
| 15:56:17 | melwitt | I must be missing something | |
| 15:56:24 | sean-k-mooney | it does | |
| 15:56:31 | sean-k-mooney | we dont undefien the xml when we power off | |
| 15:57:19 | sean-k-mooney | we delete the domain and recreate it every tiem we start a domian but power off leaves it in place | |
| 15:57:34 | melwitt | hm ok | |
| 15:57:39 | sean-k-mooney | we use that fact to make vGPU work by reading the mdev path out of the xml | |
| 15:58:48 | dansmith | yeah, it has xml when stopped | |
| 15:59:16 | dansmith | that's the "defined" vs. "active | |
| 15:59:19 | dansmith | " xml | |
| 15:59:33 | melwitt | ok, I'm not able to get the domain with 'virsh dumpxml [--inactive] <uuid>' so far I dunno how to look at it | |
| 15:59:41 | melwitt | gotcha | |
| 16:00:02 | sean-k-mooney | do virsh list --all | |
| 16:00:21 | dansmith | you need to use the id not the uuid | |
| 16:00:23 | melwitt | ok, thanks I see them | |
| 16:00:38 | melwitt | id is blank but it has "name" so I'll try that | |
| 16:00:58 | dansmith | oh for the inactive yeah | |
| 16:01:11 | sean-k-mooney | ok stacked lets boot a vm and see what happened | |
| 16:01:21 | dansmith | but uuid is just metadata for libvirt AFAIK.. I've never known to be able to use that for virsh commands | |
| 16:02:01 | melwitt | you can. I had been doing that to look at inactive domain xml | |
| 16:03:12 | melwitt | ok got it. I wasn't using sudo 🙄 | |
| 16:03:12 | melwitt | ok got it. I wasn't using sudo 🙄 | |
| 16:03:15 | sean-k-mooney | ok so i have the console in horizon | |
| 16:03:31 | sean-k-mooney | ill check the host fs/xml and then stop it | |
| 16:03:35 | melwitt | uuid works with sudo too | |
| 16:03:36 | melwitt | ok | |
| 16:04:34 | sean-k-mooney | so there is a console.log in the instanace dir | |
| 16:05:15 | sean-k-mooney | and it has data since i can cat it so ill power it off now | |
| 16:05:32 | melwitt | same | |
| 16:05:34 | melwitt | same | |
| 16:05:38 | melwitt | mine has data after poweroff to | |
| 16:05:45 | melwitt | but, in my case console_type == 'pty' and so there must be something weird happening with that | |
| 16:06:06 | melwitt | the console log file path in the xml is correct and I can use it to open the log file | |
| 16:06:09 | sean-k-mooney | https://paste.opendev.org/show/bMa9I7WsUnEfsrMC6iQz/ | |
| 16:06:45 | melwitt | is the console log get API working for you? | |
| 16:07:06 | sean-k-mooney | i did it in horizon but ill check in a sec on the cli | |
| 16:07:27 | sean-k-mooney | horizon is now showing "Unable to get log for instance "ea890cd1-c3fe-4455-bcd9-b44ea830a136"." | |
| 16:08:14 | dansmith | melwitt: interesting, I started with libvirt before there were even non-active domains and definitely since before uuids were in the xml, so I guess 15-year habits die hard :) | |
| 16:08:40 | melwitt | for mine there's helpfully no log messages that say anything about why/where it's running into a problem | |
| 16:08:43 | dansmith | ahh, console type is pty, | |
| 16:08:49 | dansmith | that's what I was wondering if it was happening | |
| 16:09:03 | dansmith | the comment says it's only for container types, but it would explain why we're losing console on shutdown | |
| 16:09:28 | dansmith | so with pty, there's no storage of the console stuff once the other side goes away, like when shutdown | |
| 16:09:36 | melwitt | I feel like it has to be this failing: data = nova.privsep.libvirt.readpty(console_path) | |
| 16:10:05 | dansmith | melwitt: your console path is a file or a path to a pty? | |
| 16:10:26 | melwitt | it's a file. but it is also type pty. which I assume is not right? | |
| 16:10:36 | dansmith | idk, to be honest | |
| 16:10:55 | sean-k-mooney | on a call by the way | |
| 16:10:56 | dansmith | the code looks like nova reads directly from the pty if the type is pty, so the file wouldn't be used | |
| 16:11:21 | dansmith | ah, wait, | |
| 16:11:32 | melwitt | the console files are there and are properly with data | |
| 16:11:32 | dansmith | it looks like it reads from the pty and writes the file if type=pty | |
| 16:11:40 | dansmith | but that would only work when it's running | |
| 16:12:10 | dansmith | so I wonder if the test is now breaking because we never read the log before the instance is shutdown (which is what the test was supposed to be doing) | |
| 16:12:11 | melwitt | oh hm ok, it must be bc I queried for the console before I shut it down | |
| 16:12:21 | melwitt | I'll try not querying before | |
| 16:12:29 | dansmith | if we did once, then it would write the file and then we could read the file after it's shutdown | |
| 16:12:46 | dansmith | so maybe the test just needs to hit the console endpoint once after ssh'able and then shutdown, and verify that it still works after | |
| 16:13:09 | melwitt | yeah.. for me I still can't get it via the API despite the fact that the file is there and the file path matches what's in the xml | |
| 16:13:54 | dansmith | is the domain inactive? | |
| 16:13:58 | melwitt | yes | |
| 16:14:07 | dansmith | I would think the console element would go away partially while inactive, | |
| 16:14:10 | dansmith | but is that causing us to not read from the file? | |
| 16:14:49 | melwitt | </console> | |
| 16:14:49 | melwitt | <target type='serial' port='0'/> | |
| 16:14:49 | melwitt | <log file='/opt/stack/data/nova/instances/682931e5-4563-4955-a552-526a31c6957a/console.log' append='off'/> | |
| 16:14:49 | melwitt | <console type='pty'> | |
| 16:15:03 | melwitt | and that file is right. it just can't get it for whatever reason | |
| 16:15:07 | dansmith | yeah | |
| 16:15:11 | dansmith | lemme re-read this code, hang on | |
| 16:15:40 | dansmith | okay so if it's a pty, we read anything from it we can and append to the log, | |
| 16:16:07 | dansmith | I guess so that we're constantly pulling pty content and filling the log file, and then we just "should" be returning the log file contents for the requester | |
| 16:16:38 | dansmith | melwitt: can you just add some logs to those functions and see what's going on? | |
| 16:16:55 | melwitt | yeah | |
| 16:58:04 | sean-k-mooney | ok call done | |
| 16:58:14 | sean-k-mooney | i was going to check the cli | |
| 16:58:22 | sean-k-mooney | did ye fined anything in the intrim | |
| 16:58:39 | sean-k-mooney | the content is still there on disk for what its worth | |