| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2022-03-29 | |||
| 16:14:35 | bauzas | ok, moving on ? | |
| 16:14:49 | gmann | elodilles: ack, thanks | |
| 16:14:53 | bauzas | #topic Open discussion | |
| 16:15:00 | bauzas | any item to discuss ? | |
| 16:15:07 | bauzas | or can I end the meeting ? | |
| 16:15:09 | gmann | nothing from me. | |
| 16:15:36 | bauzas | thanks all then for this quick meeting and seeing hopefully all of you next week :) | |
| 16:15:40 | opendevmeet | Log: https://meetings.opendev.org/meetings/nova/2022/nova.2022-03-29-16.00.log.html | |
| 16:15:40 | opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/nova/2022/nova.2022-03-29-16.00.txt | |
| 16:15:40 | opendevmeet | Minutes: https://meetings.opendev.org/meetings/nova/2022/nova.2022-03-29-16.00.html | |
| 16:15:40 | opendevmeet | Meeting ended Tue Mar 29 16:15:40 2022 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | |
| 16:15:40 | bauzas | #endmeeting | |
| 16:16:04 | elodilles | o/ | |
| 16:16:46 | gibi | o/ | |
| 17:31:43 | opendevreview | kiran pawar proposed openstack/nova master: VMware: Early fail spwan if memory is not multiple of 4. https://review.opendev.org/c/openstack/nova/+/835739 | |
| 20:08:39 | opendevreview | melanie witt proposed openstack/nova-specs master: Remove setup.py and setup.cfg https://review.opendev.org/c/openstack/nova-specs/+/835759 | |
| 20:08:39 | opendevreview | melanie witt proposed openstack/nova-specs master: Move implemented specs for the Yoga release https://review.opendev.org/c/openstack/nova-specs/+/835272 | |
| 20:25:27 | melwitt | bauzas: rebased your change on top of a change meant to fix the docs and pep8 jobs in nova-specs ^ | |
| 21:14:17 | melwitt | gmann: these two reviews have been sitting for awhile, they're about the perfload jobs that run on the placement repo, if you might be interested in reviewing https://review.opendev.org/q/topic:perfload-error | |
| 21:29:30 | opendevreview | melanie witt proposed openstack/nova master: Fix pre_live_migration rollback https://review.opendev.org/c/openstack/nova/+/815324 | |
| #openstack-nova - 2022-03-30 | |||
| 00:33:40 | opendevreview | Merged openstack/nova master: Fix pre_live_migration rollback https://review.opendev.org/c/openstack/nova/+/815324 | |
| 09:14:21 | opendevreview | Merged openstack/nova-specs master: Remove setup.py and setup.cfg https://review.opendev.org/c/openstack/nova-specs/+/835759 | |
| 09:31:20 | opendevreview | Merged openstack/nova-specs master: Move implemented specs for the Yoga release https://review.opendev.org/c/openstack/nova-specs/+/835272 | |
| 09:33:38 | bauzas | melwitt: thanks for the help ^ | |
| 10:13:27 | ihti[m] | Hi, we are facing a bug with volume attachments(https://bugs.launchpad.net/nova/+bug/1964576). We have a proposed fix for it. If some one has time to review the bug/fix, it would be great. Thanks! | |
| 11:45:16 | EugenMayer | sean-k-mooney i know have the issue present with glance showing a 'queed' status for an image, while nova / the instance shows 'image backup'. I cannot see any tasks use 'glance task-list' nor 'glance image-tasks <id>' | |
| 11:47:19 | EugenMayer | which nova-logs could be interesting? glance logs do not show anything interesting / error like. neither nova-api-error.log | |
| 11:48:19 | sean-k-mooney | the nova compute agent log is the only place that might have an error | |
| 11:48:51 | sean-k-mooney | but if the image is queued i think that means that nova has already finsihed uploading it | |
| 11:48:53 | opendevreview | kiran pawar proposed openstack/nova master: VMware: Early fail spawn if memory is not multiple of 4. https://review.opendev.org/c/openstack/nova/+/835739 | |
| 11:49:12 | sean-k-mooney | and glance should be processing it | |
| 11:49:30 | sean-k-mooney | have you checked the glance api host to see if it actully has the image on disk | |
| 11:50:35 | EugenMayer | sean-k-mooney so the nova compute could have logs or the glance api if the image is there, the question is, why the status is queue and how to find out why it is that way | |
| 11:52:17 | EugenMayer | there are no error logs on nova-compute.log since 30 days (nothing has logged at all, last from 23th march | |
| 11:53:02 | sean-k-mooney | so i think this si useing the glance interoperal import pipeline when the image is uploaded then queued to become active after the import pipeline has finsihed processin the image | |
| 11:53:20 | sean-k-mooney | EugenMayer: that kind of sound like the agent is hung | |
| 11:53:34 | sean-k-mooney | you shoudl at least see the periodics | |
| 11:53:41 | EugenMayer | So my image id (that is queued) is 57850bd9-dfdc-45bd-bd9f-cec297f3fdae - checking the storage folder i see images, but this one is not present | |
| 11:53:56 | sean-k-mooney | ack | |
| 11:54:19 | sean-k-mooney | dansmith: do you know where the image would be in the queued state? | |
| 11:54:38 | sean-k-mooney | it only enters that state after the upload has happend right? | |
| 11:54:46 | EugenMayer | using 'glance image-tasks <id>' does not show any tasks, neither 'glance task-list' | |
| 11:54:46 | sean-k-mooney | or am i miss rememebering that | |
| 11:56:01 | EugenMayer | interesting, all those backup tasks on compute3 are broken. Means other computes finished, just compute3 backups did not (all of them). This kind of tells that the compute is somehow flaky - but why and what | |
| 12:20:19 | EugenMayer | if you have any idea how to trace, happy to look at it. Currently not sure where to look at at all | |
| 12:23:11 | sean-k-mooney | the only thing that comes to mind is that the agent is exasuting the thread pool or has made a blocking call on the main thread that was not monkey patched by eventlets | |
| 12:23:30 | sean-k-mooney | generating a guru meditation report might shed some light on that | |
| 12:25:23 | sean-k-mooney | but that will basicaly crashdump the process so you will have to restart the agent after you do the sig_hup | |
| 12:25:32 | sean-k-mooney | actully not sig_hup | |
| 12:25:38 | sean-k-mooney | sig_usr2 | |
| 12:27:18 | EugenMayer | who holds the state in general right now? | |
| 12:31:35 | sean-k-mooney | when nova calls glance evenlet yeild form the greenthread and the state is stored in memory in the greenthread local varibles | |
| 12:32:05 | sean-k-mooney | same when we do any io like copying the image for snapshot | |
| 12:32:11 | sean-k-mooney | we yield | |
| 12:32:23 | sean-k-mooney | and when the io op complete event resumes the greenthread | |
| 12:48:09 | EugenMayer | not sure what a greenthread is. If the state is a memory state, restarting the service (what-ever that is) would reset the state for nova, right? | |
| 12:51:13 | sean-k-mooney | the threading model in nova is to use implicat coroutiens by using userspace thread | |
| 12:51:35 | sean-k-mooney | https://github.com/openstack/nova/blob/master/doc/source/reference/threading.rst | |
| 12:52:13 | sean-k-mooney | so everythime we do io eventlet yeild execution of the current function and it starts running the next greenthread | |
| 12:52:36 | sean-k-mooney | then when the io compelte the previous green trhead is added to the queue to be resumed | |
| 12:52:36 | EugenMayer | i see, this is the nova-compute process then, right? | |
| 12:52:45 | sean-k-mooney | yes | |
| 12:52:57 | sean-k-mooney | nova-compute but also conductor and schduler | |
| 12:53:30 | sean-k-mooney | technially nova-api is monkeypatch but the way its run with appache means it only process one request per worker process | |
| 12:53:45 | sean-k-mooney | because apache queues the request before it get to the api application | |
| 12:57:10 | EugenMayer | oh holy moly. | |
| 12:58:57 | EugenMayer | I mean, my day job is being a software engeneer. Yes with bigger EE software, yes with microservices, distributed and all that. But this really is very weired to me - or it is simply to complex for me to play around in the mind since i do not know any components properly and have no save-haven to return / start thinking from | |
| 12:59:03 | EugenMayer | thank you for elaborating on that | |
| 13:00:03 | EugenMayer | I left with 2 things: glance has an tasks status 'queued' of an tasks that does not exists and it is unclear where this state comes from. Second is, why my compute3 (out of 4) fails to create any backups, all others can. | |
| 13:00:43 | EugenMayer | Ah now i understand - not a task is 'queued' .. the image is queued - without any task. So it is the image state | |
| 13:01:45 | sean-k-mooney | a very long time ago around the catus release opensack moved form twisted to eventlet to remove the need for peopel to explcitly think about multithreading and concurancy most of the time. howver ther eare still case where you have to use locks ectra to ensure no data races. so for the most part eventlet simplifes the common code path when you are io bound which tends to be | |
| 13:01:47 | sean-k-mooney | the case for nova | |
| 13:02:26 | sean-k-mooney | yes the image is queue | |
| 13:02:45 | sean-k-mooney | not a task | |
| 13:05:01 | sean-k-mooney | https://docs.openstack.org/glance/latest/user/statuses.html | |
| 13:05:09 | sean-k-mooney | queued | |
| 13:05:11 | sean-k-mooney | The image identifier has been reserved for an image in the Glance registry. No image data has been uploaded to Glance and the image size was not explicitly set to zero on creation. | |
| 13:05:32 | sean-k-mooney | ok so queue means we have crerate the image but not uploaded it | |
| 13:06:22 | sean-k-mooney | which i guess make sense since nova is not currently in the image_uploading task_state | |
| 13:06:59 | sean-k-mooney | so that likely means that nova is failing to create the snapshot via libvirt/qemu | |
| 13:07:38 | EugenMayer | i see, thank you so much sean! | |
| 13:08:01 | EugenMayer | i removed the broken (queued) images now | |
| 13:08:10 | EugenMayer | reset the instances and restarted them | |
| 13:08:17 | sean-k-mooney | ack | |
| 13:08:39 | EugenMayer | i will check the logs of that compute once again and then restart it. Usually those restart fix 9/10 issues i have with openstack | |
| 13:08:51 | sean-k-mooney | what you likely shoudl do is try and find the request-id for the backup call and see if you can fined the last log for that operation | |
| 13:08:52 | EugenMayer | Which obviously is not a good sign, sure | |
| 13:08:58 | sean-k-mooney | to seee where it got stuck | |
| 13:09:37 | EugenMayer | this is SO hard to trace for me considering the amount of subsystems, proxies and systems involved | |
| 13:10:21 | EugenMayer | it usually is my bread and butter debugging those kind of things in our software stacks. But well, as i must learn, the reason i can do it there is - i know the software a lot better | |
| 13:22:25 | sean-k-mooney | ya there is a lot of context to grok | |
| 14:42:44 | viks__ | hi, with `soft-anti-affinity`, whenever i create 2 instances together via horizon, it goes in to 2 different hosts. But when i create one after the other with `soft-anti-affinity`, it goes in to the same host? is it expected? | |
| 15:01:28 | sean-k-mooney | viks__: the behavior of the second case will depend on if you waith for the first one to go active before you create the second | |
| 15:02:02 | sean-k-mooney | but yes it is expected in the second case you ahve a race to update the instance.host before the second vm is schduled | |
| 15:02:31 | sean-k-mooney | we provide no affinity garunetees in this case unless you enable the affinity upcall | |
| 15:05:00 | viks__ | sean-k-mooney: what all things i need to set for affinity upcall in the nova.conf? | |
| 15:24:13 | sean-k-mooney | the workaround config option in the compute nova.conf and api database in conductor nova.conf | |
| 15:39:29 | opendevreview | Alexey Stupnikov proposed openstack/nova stable/xena: Test aborting queued live migration https://review.opendev.org/c/openstack/nova/+/835853 | |
| 15:40:07 | opendevreview | Alexey Stupnikov proposed openstack/nova stable/xena: Add functional tests to reproduce bug #1960412 https://review.opendev.org/c/openstack/nova/+/835854 | |
| 15:41:04 | opendevreview | Alexey Stupnikov proposed openstack/nova stable/xena: Clean up when queued live migration aborted https://review.opendev.org/c/openstack/nova/+/835855 | |