| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2022-03-29 | |||
| 16:08:09 | elodilles | reminder: Victoria Extended Maintenance transition is due ~ in a month (2022-04-27) | |
| 16:08:26 | gmann | we discussed it in openinfra channel and we need these two backports #link https://review.opendev.org/c/openstack/oslo.vmware/+/835672 #link https://review.opendev.org/c/openstack/requirements/+/835671 | |
| 16:08:31 | elodilles | gmann: but that's in the same job, right? | |
| 16:08:53 | gmann | elodilles: same job , failure can be seen in https://review.opendev.org/c/openstack/nova/+/834765 | |
| 16:09:28 | gmann | suds-jurko version conflict is consistent there from oslo/vmware which was fixed in yoga | |
| 16:09:32 | elodilles | gmann: yes, thanks for that DNM patch | |
| 16:10:06 | gmann | I was trying those mirror issue and traced to this issue which actually need to be fixed | |
| 16:10:38 | elodilles | just for the record, we could unblock the gate via this patch: https://review.opendev.org/c/openstack/nova/+/834854 | |
| 16:10:43 | elodilles | (the correct link) | |
| 16:10:52 | gmann | elodilles: bauzas if we think req and oslo.vmware backport and then oslo.vmware release for stable/xena and then u-c update can take time then we can go with n-v for now and then revert | |
| 16:11:11 | bauzas | ok | |
| 16:11:12 | elodilles | gmann: ack | |
| 16:11:18 | bauzas | how can I help ? | |
| 16:11:20 | elodilles | thanks for the info | |
| 16:12:39 | elodilles | bauzas: by reviewing the n-v patch (linked above ^^^) o:) | |
| 16:12:51 | bauzas | elodilles: cool, thanks | |
| 16:13:03 | bauzas | oh done then | |
| 16:13:04 | gmann | yeah, I will add these info there and will propsoe revert once all backport and u-c are in place | |
| 16:13:31 | elodilles | thanks gmann ! | |
| 16:13:52 | elodilles | i'll also try to look at the vmware and req patches after Yoga release :) | |
| 16:14:35 | bauzas | ok, moving on ? | |
| 16:14:49 | gmann | elodilles: ack, thanks | |
| 16:14:53 | bauzas | #topic Open discussion | |
| 16:15:00 | bauzas | any item to discuss ? | |
| 16:15:07 | bauzas | or can I end the meeting ? | |
| 16:15:09 | gmann | nothing from me. | |
| 16:15:36 | bauzas | thanks all then for this quick meeting and seeing hopefully all of you next week :) | |
| 16:15:40 | opendevmeet | Log: https://meetings.opendev.org/meetings/nova/2022/nova.2022-03-29-16.00.log.html | |
| 16:15:40 | opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/nova/2022/nova.2022-03-29-16.00.txt | |
| 16:15:40 | opendevmeet | Minutes: https://meetings.opendev.org/meetings/nova/2022/nova.2022-03-29-16.00.html | |
| 16:15:40 | opendevmeet | Meeting ended Tue Mar 29 16:15:40 2022 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | |
| 16:15:40 | bauzas | #endmeeting | |
| 16:16:04 | elodilles | o/ | |
| 16:16:46 | gibi | o/ | |
| 17:31:43 | opendevreview | kiran pawar proposed openstack/nova master: VMware: Early fail spwan if memory is not multiple of 4. https://review.opendev.org/c/openstack/nova/+/835739 | |
| 20:08:39 | opendevreview | melanie witt proposed openstack/nova-specs master: Remove setup.py and setup.cfg https://review.opendev.org/c/openstack/nova-specs/+/835759 | |
| 20:08:39 | opendevreview | melanie witt proposed openstack/nova-specs master: Move implemented specs for the Yoga release https://review.opendev.org/c/openstack/nova-specs/+/835272 | |
| 20:25:27 | melwitt | bauzas: rebased your change on top of a change meant to fix the docs and pep8 jobs in nova-specs ^ | |
| 21:14:17 | melwitt | gmann: these two reviews have been sitting for awhile, they're about the perfload jobs that run on the placement repo, if you might be interested in reviewing https://review.opendev.org/q/topic:perfload-error | |
| 21:29:30 | opendevreview | melanie witt proposed openstack/nova master: Fix pre_live_migration rollback https://review.opendev.org/c/openstack/nova/+/815324 | |
| #openstack-nova - 2022-03-30 | |||
| 00:33:40 | opendevreview | Merged openstack/nova master: Fix pre_live_migration rollback https://review.opendev.org/c/openstack/nova/+/815324 | |
| 09:14:21 | opendevreview | Merged openstack/nova-specs master: Remove setup.py and setup.cfg https://review.opendev.org/c/openstack/nova-specs/+/835759 | |
| 09:31:20 | opendevreview | Merged openstack/nova-specs master: Move implemented specs for the Yoga release https://review.opendev.org/c/openstack/nova-specs/+/835272 | |
| 09:33:38 | bauzas | melwitt: thanks for the help ^ | |
| 10:13:27 | ihti[m] | Hi, we are facing a bug with volume attachments(https://bugs.launchpad.net/nova/+bug/1964576). We have a proposed fix for it. If some one has time to review the bug/fix, it would be great. Thanks! | |
| 11:45:16 | EugenMayer | sean-k-mooney i know have the issue present with glance showing a 'queed' status for an image, while nova / the instance shows 'image backup'. I cannot see any tasks use 'glance task-list' nor 'glance image-tasks <id>' | |
| 11:47:19 | EugenMayer | which nova-logs could be interesting? glance logs do not show anything interesting / error like. neither nova-api-error.log | |
| 11:48:19 | sean-k-mooney | the nova compute agent log is the only place that might have an error | |
| 11:48:51 | sean-k-mooney | but if the image is queued i think that means that nova has already finsihed uploading it | |
| 11:48:53 | opendevreview | kiran pawar proposed openstack/nova master: VMware: Early fail spawn if memory is not multiple of 4. https://review.opendev.org/c/openstack/nova/+/835739 | |
| 11:49:12 | sean-k-mooney | and glance should be processing it | |
| 11:49:30 | sean-k-mooney | have you checked the glance api host to see if it actully has the image on disk | |
| 11:50:35 | EugenMayer | sean-k-mooney so the nova compute could have logs or the glance api if the image is there, the question is, why the status is queue and how to find out why it is that way | |
| 11:52:17 | EugenMayer | there are no error logs on nova-compute.log since 30 days (nothing has logged at all, last from 23th march | |
| 11:53:02 | sean-k-mooney | so i think this si useing the glance interoperal import pipeline when the image is uploaded then queued to become active after the import pipeline has finsihed processin the image | |
| 11:53:20 | sean-k-mooney | EugenMayer: that kind of sound like the agent is hung | |
| 11:53:34 | sean-k-mooney | you shoudl at least see the periodics | |
| 11:53:41 | EugenMayer | So my image id (that is queued) is 57850bd9-dfdc-45bd-bd9f-cec297f3fdae - checking the storage folder i see images, but this one is not present | |
| 11:53:56 | sean-k-mooney | ack | |
| 11:54:19 | sean-k-mooney | dansmith: do you know where the image would be in the queued state? | |
| 11:54:38 | sean-k-mooney | it only enters that state after the upload has happend right? | |
| 11:54:46 | EugenMayer | using 'glance image-tasks <id>' does not show any tasks, neither 'glance task-list' | |
| 11:54:46 | sean-k-mooney | or am i miss rememebering that | |
| 11:56:01 | EugenMayer | interesting, all those backup tasks on compute3 are broken. Means other computes finished, just compute3 backups did not (all of them). This kind of tells that the compute is somehow flaky - but why and what | |
| 12:20:19 | EugenMayer | if you have any idea how to trace, happy to look at it. Currently not sure where to look at at all | |
| 12:23:11 | sean-k-mooney | the only thing that comes to mind is that the agent is exasuting the thread pool or has made a blocking call on the main thread that was not monkey patched by eventlets | |
| 12:23:30 | sean-k-mooney | generating a guru meditation report might shed some light on that | |
| 12:25:23 | sean-k-mooney | but that will basicaly crashdump the process so you will have to restart the agent after you do the sig_hup | |
| 12:25:32 | sean-k-mooney | actully not sig_hup | |
| 12:25:38 | sean-k-mooney | sig_usr2 | |
| 12:27:18 | EugenMayer | who holds the state in general right now? | |
| 12:31:35 | sean-k-mooney | when nova calls glance evenlet yeild form the greenthread and the state is stored in memory in the greenthread local varibles | |
| 12:32:05 | sean-k-mooney | same when we do any io like copying the image for snapshot | |
| 12:32:11 | sean-k-mooney | we yield | |
| 12:32:23 | sean-k-mooney | and when the io op complete event resumes the greenthread | |
| 12:48:09 | EugenMayer | not sure what a greenthread is. If the state is a memory state, restarting the service (what-ever that is) would reset the state for nova, right? | |
| 12:51:13 | sean-k-mooney | the threading model in nova is to use implicat coroutiens by using userspace thread | |
| 12:51:35 | sean-k-mooney | https://github.com/openstack/nova/blob/master/doc/source/reference/threading.rst | |
| 12:52:13 | sean-k-mooney | so everythime we do io eventlet yeild execution of the current function and it starts running the next greenthread | |
| 12:52:36 | sean-k-mooney | then when the io compelte the previous green trhead is added to the queue to be resumed | |
| 12:52:36 | EugenMayer | i see, this is the nova-compute process then, right? | |
| 12:52:45 | sean-k-mooney | yes | |
| 12:52:57 | sean-k-mooney | nova-compute but also conductor and schduler | |
| 12:53:30 | sean-k-mooney | technially nova-api is monkeypatch but the way its run with appache means it only process one request per worker process | |
| 12:53:45 | sean-k-mooney | because apache queues the request before it get to the api application | |
| 12:57:10 | EugenMayer | oh holy moly. | |
| 12:58:57 | EugenMayer | I mean, my day job is being a software engeneer. Yes with bigger EE software, yes with microservices, distributed and all that. But this really is very weired to me - or it is simply to complex for me to play around in the mind since i do not know any components properly and have no save-haven to return / start thinking from | |
| 12:59:03 | EugenMayer | thank you for elaborating on that | |
| 13:00:03 | EugenMayer | I left with 2 things: glance has an tasks status 'queued' of an tasks that does not exists and it is unclear where this state comes from. Second is, why my compute3 (out of 4) fails to create any backups, all others can. | |
| 13:00:43 | EugenMayer | Ah now i understand - not a task is 'queued' .. the image is queued - without any task. So it is the image state | |
| 13:01:45 | sean-k-mooney | a very long time ago around the catus release opensack moved form twisted to eventlet to remove the need for peopel to explcitly think about multithreading and concurancy most of the time. howver ther eare still case where you have to use locks ectra to ensure no data races. so for the most part eventlet simplifes the common code path when you are io bound which tends to be | |
| 13:01:47 | sean-k-mooney | the case for nova | |
| 13:02:26 | sean-k-mooney | yes the image is queue | |
| 13:02:45 | sean-k-mooney | not a task | |
| 13:05:01 | sean-k-mooney | https://docs.openstack.org/glance/latest/user/statuses.html | |
| 13:05:09 | sean-k-mooney | queued | |
| 13:05:11 | sean-k-mooney | The image identifier has been reserved for an image in the Glance registry. No image data has been uploaded to Glance and the image size was not explicitly set to zero on creation. | |
| 13:05:32 | sean-k-mooney | ok so queue means we have crerate the image but not uploaded it | |
| 13:06:22 | sean-k-mooney | which i guess make sense since nova is not currently in the image_uploading task_state | |
| 13:06:59 | sean-k-mooney | so that likely means that nova is failing to create the snapshot via libvirt/qemu | |