| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2024-02-13 | |||
| 16:28:01 | fwiesel | At least from what I can gather, it is a rather strange one, and probably is too much for the summary in this meeting. | |
| 16:28:52 | fwiesel | I am almost of a mind to rip out the code for image upload for the one used in cinder (incidentally the one in oslo.vmware). Any strong feelings on this one? | |
| 16:28:53 | bauzas | fwiesel: from your report, it sounds to me this is a glance issue | |
| 16:28:59 | fwiesel | Well, cinder works. | |
| 16:29:09 | fwiesel | With the same glance :) | |
| 16:29:11 | bauzas | so that's a client issue | |
| 16:29:36 | fwiesel | It all runs in the same venv, so the code for all the libraries are the same. | |
| 16:31:24 | fwiesel | The debug output for the request for nova and cinder against glance is almost the same. Cinder also passes on the X-Service-Token to glance, while nova doesn't. | |
| 16:31:38 | dansmith | not sure how it's a glance thing | |
| 16:31:52 | fwiesel | Not that I believe that to be the error, just for completness. | |
| 16:32:11 | bauzas | dansmith: right my bad, it's a client thing | |
| 16:32:20 | bauzas | but the OSError sounds a permission issue | |
| 16:32:23 | fwiesel | Either way, probably not something that can be solved in a couple of minutes in this meeting. I presume | |
| 16:32:54 | fwiesel | You get an "OSError" because the client (i.e. nova) closes the connections, so glance cannot write to the socket anymore. | |
| 16:33:32 | dansmith | fwiesel: not sure that makes sense either :) | |
| 16:34:09 | fwiesel | Yeah, either way. I would suggest to leave the discussion on the details of the bug for after the meeting. | |
| 16:34:11 | dansmith | I'll have to look more at the logs after the meeting, there's not really enough meat in the bug to really see I think | |
| 16:34:17 | bauzas | I don't really know the workflow that's implied by this oslo.vmware call | |
| 16:35:07 | bauzas | oh found it | |
| 16:35:09 | bauzas | http://openstack-ci-logs.global.cloud.sap/openstack/nova/35af4b345d997b63f999a090e236d91b78ea4304/n-cpu-1.service.log | |
| 16:35:47 | bauzas | that's when getting the image from glance | |
| 16:36:27 | sean-k-mooney | right so that should be via http via the glance api | |
| 16:36:36 | sean-k-mooney | even if its on NFS as a stoage backend | |
| 16:37:06 | bauzas | I don't actually indeed see why we need to call oslo.vmware | |
| 16:37:34 | sean-k-mooney | well for the volume creation we would just ask cindeer to create the voluem from the image right | |
| 16:38:04 | fwiesel | That happens for the boot from volume case. And that works. But we also have the boot from "local" disk, and then nova needs to pull the image. | |
| 16:38:11 | sean-k-mooney | we would only need to call into oslo.vmware when asking vmware to create the instance using the cidner volume | |
| 16:38:22 | bauzas | yeah their problem is with the standard boot from image to disk | |
| 16:38:38 | sean-k-mooney | ok but we dont have an agent runing on the esxi host | |
| 16:38:46 | sean-k-mooney | or vshper node | |
| 16:39:06 | sean-k-mooney | we are asking vsphare to download it form glance right | |
| 16:39:11 | bauzas | apparently they do some caching on the downloaded image | |
| 16:39:22 | bauzas | so this is indeed not a glance communication problem | |
| 16:39:42 | dansmith | can we discuss after the meeting? | |
| 16:39:48 | bauzas | from what I can understand from the stacktrace, the image is downloaded but then you call oslo.vmware to cache that image | |
| 16:39:54 | bauzas | dansmith: good point | |
| 16:40:01 | bauzas | the next topic should be empty | |
| 16:40:27 | bauzas | fwiesel: are you done with your points ? we'll continue troubleshooting right after the meeting | |
| 16:40:47 | bauzas | apparently so | |
| 16:40:48 | fwiesel | I am done. Over to you | |
| 16:40:51 | bauzas | #topic Open discussion | |
| 16:40:55 | bauzas | . | |
| 16:41:00 | bauzas | nothing in the agenda | |
| 16:41:04 | bauzas | anything anyone ? | |
| 16:41:20 | sean-k-mooney | did you want to chat about the sate of the vgpu seriese | |
| 16:41:36 | bauzas | sean-k-mooney: well, I eventually was able to live-migrate | |
| 16:41:39 | sean-k-mooney | or wait till next week when you have done some more testing | |
| 16:41:57 | bauzas | I know now the reason why we need some very large downtime option | |
| 16:42:29 | sean-k-mooney | do wehttps://docs.openstack.org/nova/latest/configuration/config.html#libvirt.live_migration_downtime | |
| 16:42:37 | sean-k-mooney | is that enough to configure it | |
| 16:42:41 | bauzas | but again, I think I'm done, I'm currently working on providing a asciinema stuff | |
| 16:42:50 | bauzas | so people will see it | |
| 16:43:03 | bauzas | sean-k-mooney: correct, that and the two other options | |
| 16:43:09 | sean-k-mooney | ack | |
| 16:43:24 | sean-k-mooney | i just wanted to confirm if the existign config options and the code you have for review is enough | |
| 16:44:20 | sean-k-mooney | assuming yes we can proceed on gerrit | |
| 16:44:21 | bauzas | yeah, so I'll modify https://review.opendev.org/c/openstack/nova/+/904258/13/doc/source/admin/virtual-gpu.rst | |
| 16:44:42 | bauzas | to explain that people will need to use a large max downtimez | |
| 16:44:54 | sean-k-mooney | ack | |
| 16:45:07 | bauzas | anyway, I'm done now | |
| 16:45:13 | bauzas | any other things ? | |
| 16:45:41 | sean-k-mooney | nope just wanted to confim that before i review the rest of your seies | |
| 16:45:52 | bauzas | sean-k-mooney: no worries | |
| 16:45:58 | bauzas | so, thanks all | |
| 16:46:07 | fwiesel | Thanks a lot. | |
| 16:46:08 | opendevmeet | Log: https://meetings.opendev.org/meetings/nova/2024/nova.2024-02-13-16.00.log.html | |
| 16:46:08 | opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/nova/2024/nova.2024-02-13-16.00.txt | |
| 16:46:08 | opendevmeet | Minutes: https://meetings.opendev.org/meetings/nova/2024/nova.2024-02-13-16.00.html | |
| 16:46:08 | opendevmeet | Meeting ended Tue Feb 13 16:46:08 2024 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | |
| 16:46:08 | bauzas | #endmeeting | |
| 16:51:54 | sean-k-mooney | fwiesel: so it looks like the vmware direver has some image caching code in spawn https://github.com/openstack/nova/blob/master/nova/virt/vmwareapi/vmops.py#L738 is it the _fetch_image_if_missing function that is failing in hte local storage case? | |
| 16:54:04 | sean-k-mooney | fwiesel: there are some other error stoo but i noticed "RROR nova.db.main.api [None req-4114694a-df48-45f8-b7ff-32706629cf90 None None] No DB access allowed in nova-compute: File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 264," | |
| 16:54:13 | sean-k-mooney | that sames to go away however | |
| 16:54:46 | sean-k-mooney | i added the backtrace to the bug https://bugs.launchpad.net/nova/+bug/2053027/comments/2 | |
| 16:59:56 | fwiesel | sean-k-mooney: Yes that is the failing one. The invalid argument is a bit misleading, and comes from a zero byte file size. | |
| 17:00:36 | opendevreview | Merged openstack/nova stable/2023.1: testing: Reset affinity support global variables https://review.opendev.org/c/openstack/nova/+/908839 | |
| 17:01:48 | sean-k-mooney | fwiesel: how is your glanced configured is it using cinder for storage | |
| 17:03:26 | gmann | dansmith: can you re-check this https://review.opendev.org/c/openstack/nova/+/906629 | |
| 17:03:41 | gmann | dansmith: sean-k-mooney and rest of the patch in remove hyperv series | |
| 17:04:55 | sean-k-mooney | fwiesel: glance_store.default_store = file so ya thats odd it should be pulling the image form the rest api then | |
| 17:04:56 | dansmith | gmann: sorry I didn't realize I hadn't circled back on that one after you fixed the qemu thing | |
| 17:04:59 | sean-k-mooney | gmann: sure | |
| 17:05:14 | gmann | thanks | |
| 17:05:31 | fwiesel | sean-k-mooney: Yes, it is pulling it via the rest api from a local file. And the fun part is, cinder does the same and it works. | |
| 17:05:54 | dansmith | sean-k-mooney: maybe read my comment here: https://review.opendev.org/c/openstack/nova/+/906636/3/nova/tests/unit/objects/test_objects.py#b1105 | |
| 17:06:40 | sean-k-mooney | dansmith: ya i saw that. i was personally fine with leave everythign after the virt driver remove for a later release | |
| 17:07:04 | sean-k-mooney | dansmith: so with https://review.opendev.org/c/openstack/nova/+/894466/3 merged we have removed the mvp | |
| 17:07:37 | dansmith | yeah | |
| 17:07:46 | sean-k-mooney | so if you want to keep HyperVLiveMigrateData that ok | |
| 17:07:56 | dansmith | although I'd say the doc patch should be considered mandatory | |
| 17:08:03 | sean-k-mooney | we can do the RDP console changes if we want even with out the object change | |
| 17:08:18 | sean-k-mooney | oh yes but that has +2w no? | |
| 17:08:39 | sean-k-mooney | ya it has so that just need to make it way through ci | |
| 17:08:49 | dansmith | sean-k-mooney: it does now, I'm just saying if you're talking about mvp, we should include not having docs about drivers that aren't in tree :) | |
| 17:09:10 | sean-k-mooney | :) yes your right | |
| 17:11:12 | gmann | but if we keep field and remove HyperVLiveMigrateData maybe that is fine? | |
| 17:11:38 | gmann | dansmith: forgot to reply there but I checked and that is what we did for xenapi driver removal | |
| 17:12:52 | sean-k-mooney | gmann: having the opbject has little to no maintance overhead | |
| 17:13:18 | dansmith | gmann: the xenbus object is still there.. what object are you talking about? | |
| 17:13:42 | dansmith | maybe the xen live migratedata? | |