| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2024-02-13 | |||
| 16:21:51 | bauzas | unmaintained branches | |
| 16:21:53 | bauzas | * | |
| 16:22:24 | bauzas | people can fork as much as they want provided they don't name those branches "nova-something" | |
| 16:23:03 | bauzas | anyway, I'm done | |
| 16:23:07 | bauzas | moving on ? | |
| 16:23:10 | elodilles | +1 | |
| 16:23:22 | bauzas | #topic vmwareapi 3rd-party CI efforts Highlights | |
| 16:23:28 | fwiesel | #info Fixes to CI for various branches. Missing is still versions prior zed (requiring Ubuntu 20.04) | |
| 16:23:29 | bauzas | fwiesel: heya | |
| 16:23:51 | bauzas | yoga is now unmaintained :D | |
| 16:23:56 | fwiesel | So, I only tested before on master, which didn't translate well to other branches. | |
| 16:24:20 | bauzas | so IMHO you shouldn't really care on maintaining 3rd party jobs for Yoga and older branches | |
| 16:24:42 | fwiesel | Problem is xena. It is also Ubuntu 20.04. | |
| 16:24:58 | fwiesel | Ah, x<z ... | |
| 16:25:15 | fwiesel | Great, that was my implicit question. So we are fine just with zed and later? | |
| 16:25:22 | elodilles | (and Zed will move to Unmaintained in ~3 months) | |
| 16:26:02 | bauzas | fwiesel: that actually depends on what you want to test for yourselves | |
| 16:26:32 | bauzas | but if you don't really want to test Xena for your own sake, fwiw, the nova project is fine with not testing vmwareapi on that branch | |
| 16:26:44 | fwiesel | Well, we want to move away from Xena ourselves. My main reason for testing older versions was trying to bisect the bug I was mentioning. | |
| 16:26:54 | fwiesel | #info Debugging local root disk (Raised bug: https://bugs.launchpad.net/nova/+bug/2053027) | |
| 16:27:21 | fwiesel | That took most of my time. It works for us on our heavily patched xena, I hoped to establish a base-line. | |
| 16:28:01 | fwiesel | At least from what I can gather, it is a rather strange one, and probably is too much for the summary in this meeting. | |
| 16:28:52 | fwiesel | I am almost of a mind to rip out the code for image upload for the one used in cinder (incidentally the one in oslo.vmware). Any strong feelings on this one? | |
| 16:28:53 | bauzas | fwiesel: from your report, it sounds to me this is a glance issue | |
| 16:28:59 | fwiesel | Well, cinder works. | |
| 16:29:09 | fwiesel | With the same glance :) | |
| 16:29:11 | bauzas | so that's a client issue | |
| 16:29:36 | fwiesel | It all runs in the same venv, so the code for all the libraries are the same. | |
| 16:31:24 | fwiesel | The debug output for the request for nova and cinder against glance is almost the same. Cinder also passes on the X-Service-Token to glance, while nova doesn't. | |
| 16:31:38 | dansmith | not sure how it's a glance thing | |
| 16:31:52 | fwiesel | Not that I believe that to be the error, just for completness. | |
| 16:32:11 | bauzas | dansmith: right my bad, it's a client thing | |
| 16:32:20 | bauzas | but the OSError sounds a permission issue | |
| 16:32:23 | fwiesel | Either way, probably not something that can be solved in a couple of minutes in this meeting. I presume | |
| 16:32:54 | fwiesel | You get an "OSError" because the client (i.e. nova) closes the connections, so glance cannot write to the socket anymore. | |
| 16:33:32 | dansmith | fwiesel: not sure that makes sense either :) | |
| 16:34:09 | fwiesel | Yeah, either way. I would suggest to leave the discussion on the details of the bug for after the meeting. | |
| 16:34:11 | dansmith | I'll have to look more at the logs after the meeting, there's not really enough meat in the bug to really see I think | |
| 16:34:17 | bauzas | I don't really know the workflow that's implied by this oslo.vmware call | |
| 16:35:07 | bauzas | oh found it | |
| 16:35:09 | bauzas | http://openstack-ci-logs.global.cloud.sap/openstack/nova/35af4b345d997b63f999a090e236d91b78ea4304/n-cpu-1.service.log | |
| 16:35:47 | bauzas | that's when getting the image from glance | |
| 16:36:27 | sean-k-mooney | right so that should be via http via the glance api | |
| 16:36:36 | sean-k-mooney | even if its on NFS as a stoage backend | |
| 16:37:06 | bauzas | I don't actually indeed see why we need to call oslo.vmware | |
| 16:37:34 | sean-k-mooney | well for the volume creation we would just ask cindeer to create the voluem from the image right | |
| 16:38:04 | fwiesel | That happens for the boot from volume case. And that works. But we also have the boot from "local" disk, and then nova needs to pull the image. | |
| 16:38:11 | sean-k-mooney | we would only need to call into oslo.vmware when asking vmware to create the instance using the cidner volume | |
| 16:38:22 | bauzas | yeah their problem is with the standard boot from image to disk | |
| 16:38:38 | sean-k-mooney | ok but we dont have an agent runing on the esxi host | |
| 16:38:46 | sean-k-mooney | or vshper node | |
| 16:39:06 | sean-k-mooney | we are asking vsphare to download it form glance right | |
| 16:39:11 | bauzas | apparently they do some caching on the downloaded image | |
| 16:39:22 | bauzas | so this is indeed not a glance communication problem | |
| 16:39:42 | dansmith | can we discuss after the meeting? | |
| 16:39:48 | bauzas | from what I can understand from the stacktrace, the image is downloaded but then you call oslo.vmware to cache that image | |
| 16:39:54 | bauzas | dansmith: good point | |
| 16:40:01 | bauzas | the next topic should be empty | |
| 16:40:27 | bauzas | fwiesel: are you done with your points ? we'll continue troubleshooting right after the meeting | |
| 16:40:47 | bauzas | apparently so | |
| 16:40:48 | fwiesel | I am done. Over to you | |
| 16:40:51 | bauzas | #topic Open discussion | |
| 16:40:55 | bauzas | . | |
| 16:41:00 | bauzas | nothing in the agenda | |
| 16:41:04 | bauzas | anything anyone ? | |
| 16:41:20 | sean-k-mooney | did you want to chat about the sate of the vgpu seriese | |
| 16:41:36 | bauzas | sean-k-mooney: well, I eventually was able to live-migrate | |
| 16:41:39 | sean-k-mooney | or wait till next week when you have done some more testing | |
| 16:41:57 | bauzas | I know now the reason why we need some very large downtime option | |
| 16:42:29 | sean-k-mooney | do wehttps://docs.openstack.org/nova/latest/configuration/config.html#libvirt.live_migration_downtime | |
| 16:42:37 | sean-k-mooney | is that enough to configure it | |
| 16:42:41 | bauzas | but again, I think I'm done, I'm currently working on providing a asciinema stuff | |
| 16:42:50 | bauzas | so people will see it | |
| 16:43:03 | bauzas | sean-k-mooney: correct, that and the two other options | |
| 16:43:09 | sean-k-mooney | ack | |
| 16:43:24 | sean-k-mooney | i just wanted to confirm if the existign config options and the code you have for review is enough | |
| 16:44:20 | sean-k-mooney | assuming yes we can proceed on gerrit | |
| 16:44:21 | bauzas | yeah, so I'll modify https://review.opendev.org/c/openstack/nova/+/904258/13/doc/source/admin/virtual-gpu.rst | |
| 16:44:42 | bauzas | to explain that people will need to use a large max downtimez | |
| 16:44:54 | sean-k-mooney | ack | |
| 16:45:07 | bauzas | anyway, I'm done now | |
| 16:45:13 | bauzas | any other things ? | |
| 16:45:41 | sean-k-mooney | nope just wanted to confim that before i review the rest of your seies | |
| 16:45:52 | bauzas | sean-k-mooney: no worries | |
| 16:45:58 | bauzas | so, thanks all | |
| 16:46:07 | fwiesel | Thanks a lot. | |
| 16:46:08 | opendevmeet | Log: https://meetings.opendev.org/meetings/nova/2024/nova.2024-02-13-16.00.log.html | |
| 16:46:08 | opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/nova/2024/nova.2024-02-13-16.00.txt | |
| 16:46:08 | opendevmeet | Minutes: https://meetings.opendev.org/meetings/nova/2024/nova.2024-02-13-16.00.html | |
| 16:46:08 | opendevmeet | Meeting ended Tue Feb 13 16:46:08 2024 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | |
| 16:46:08 | bauzas | #endmeeting | |
| 16:51:54 | sean-k-mooney | fwiesel: so it looks like the vmware direver has some image caching code in spawn https://github.com/openstack/nova/blob/master/nova/virt/vmwareapi/vmops.py#L738 is it the _fetch_image_if_missing function that is failing in hte local storage case? | |
| 16:54:04 | sean-k-mooney | fwiesel: there are some other error stoo but i noticed "RROR nova.db.main.api [None req-4114694a-df48-45f8-b7ff-32706629cf90 None None] No DB access allowed in nova-compute: File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 264," | |
| 16:54:13 | sean-k-mooney | that sames to go away however | |
| 16:54:46 | sean-k-mooney | i added the backtrace to the bug https://bugs.launchpad.net/nova/+bug/2053027/comments/2 | |
| 16:59:56 | fwiesel | sean-k-mooney: Yes that is the failing one. The invalid argument is a bit misleading, and comes from a zero byte file size. | |
| 17:00:36 | opendevreview | Merged openstack/nova stable/2023.1: testing: Reset affinity support global variables https://review.opendev.org/c/openstack/nova/+/908839 | |
| 17:01:48 | sean-k-mooney | fwiesel: how is your glanced configured is it using cinder for storage | |
| 17:03:26 | gmann | dansmith: can you re-check this https://review.opendev.org/c/openstack/nova/+/906629 | |
| 17:03:41 | gmann | dansmith: sean-k-mooney and rest of the patch in remove hyperv series | |