| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2022-03-30 | |||
| 11:53:56 | sean-k-mooney | ack | |
| 11:54:19 | sean-k-mooney | dansmith: do you know where the image would be in the queued state? | |
| 11:54:38 | sean-k-mooney | it only enters that state after the upload has happend right? | |
| 11:54:46 | EugenMayer | using 'glance image-tasks <id>' does not show any tasks, neither 'glance task-list' | |
| 11:54:46 | sean-k-mooney | or am i miss rememebering that | |
| 11:56:01 | EugenMayer | interesting, all those backup tasks on compute3 are broken. Means other computes finished, just compute3 backups did not (all of them). This kind of tells that the compute is somehow flaky - but why and what | |
| 12:20:19 | EugenMayer | if you have any idea how to trace, happy to look at it. Currently not sure where to look at at all | |
| 12:23:11 | sean-k-mooney | the only thing that comes to mind is that the agent is exasuting the thread pool or has made a blocking call on the main thread that was not monkey patched by eventlets | |
| 12:23:30 | sean-k-mooney | generating a guru meditation report might shed some light on that | |
| 12:25:23 | sean-k-mooney | but that will basicaly crashdump the process so you will have to restart the agent after you do the sig_hup | |
| 12:25:32 | sean-k-mooney | actully not sig_hup | |
| 12:25:38 | sean-k-mooney | sig_usr2 | |
| 12:27:18 | EugenMayer | who holds the state in general right now? | |
| 12:31:35 | sean-k-mooney | when nova calls glance evenlet yeild form the greenthread and the state is stored in memory in the greenthread local varibles | |
| 12:32:05 | sean-k-mooney | same when we do any io like copying the image for snapshot | |
| 12:32:11 | sean-k-mooney | we yield | |
| 12:32:23 | sean-k-mooney | and when the io op complete event resumes the greenthread | |
| 12:48:09 | EugenMayer | not sure what a greenthread is. If the state is a memory state, restarting the service (what-ever that is) would reset the state for nova, right? | |
| 12:51:13 | sean-k-mooney | the threading model in nova is to use implicat coroutiens by using userspace thread | |
| 12:51:35 | sean-k-mooney | https://github.com/openstack/nova/blob/master/doc/source/reference/threading.rst | |
| 12:52:13 | sean-k-mooney | so everythime we do io eventlet yeild execution of the current function and it starts running the next greenthread | |
| 12:52:36 | sean-k-mooney | then when the io compelte the previous green trhead is added to the queue to be resumed | |
| 12:52:36 | EugenMayer | i see, this is the nova-compute process then, right? | |
| 12:52:45 | sean-k-mooney | yes | |
| 12:52:57 | sean-k-mooney | nova-compute but also conductor and schduler | |
| 12:53:30 | sean-k-mooney | technially nova-api is monkeypatch but the way its run with appache means it only process one request per worker process | |
| 12:53:45 | sean-k-mooney | because apache queues the request before it get to the api application | |
| 12:57:10 | EugenMayer | oh holy moly. | |
| 12:58:57 | EugenMayer | I mean, my day job is being a software engeneer. Yes with bigger EE software, yes with microservices, distributed and all that. But this really is very weired to me - or it is simply to complex for me to play around in the mind since i do not know any components properly and have no save-haven to return / start thinking from | |
| 12:59:03 | EugenMayer | thank you for elaborating on that | |
| 13:00:03 | EugenMayer | I left with 2 things: glance has an tasks status 'queued' of an tasks that does not exists and it is unclear where this state comes from. Second is, why my compute3 (out of 4) fails to create any backups, all others can. | |
| 13:00:43 | EugenMayer | Ah now i understand - not a task is 'queued' .. the image is queued - without any task. So it is the image state | |
| 13:01:45 | sean-k-mooney | a very long time ago around the catus release opensack moved form twisted to eventlet to remove the need for peopel to explcitly think about multithreading and concurancy most of the time. howver ther eare still case where you have to use locks ectra to ensure no data races. so for the most part eventlet simplifes the common code path when you are io bound which tends to be | |
| 13:01:47 | sean-k-mooney | the case for nova | |
| 13:02:26 | sean-k-mooney | yes the image is queue | |
| 13:02:45 | sean-k-mooney | not a task | |
| 13:05:01 | sean-k-mooney | https://docs.openstack.org/glance/latest/user/statuses.html | |
| 13:05:09 | sean-k-mooney | queued | |
| 13:05:11 | sean-k-mooney | The image identifier has been reserved for an image in the Glance registry. No image data has been uploaded to Glance and the image size was not explicitly set to zero on creation. | |
| 13:05:32 | sean-k-mooney | ok so queue means we have crerate the image but not uploaded it | |
| 13:06:22 | sean-k-mooney | which i guess make sense since nova is not currently in the image_uploading task_state | |
| 13:06:59 | sean-k-mooney | so that likely means that nova is failing to create the snapshot via libvirt/qemu | |
| 13:07:38 | EugenMayer | i see, thank you so much sean! | |
| 13:08:01 | EugenMayer | i removed the broken (queued) images now | |
| 13:08:10 | EugenMayer | reset the instances and restarted them | |
| 13:08:17 | sean-k-mooney | ack | |
| 13:08:39 | EugenMayer | i will check the logs of that compute once again and then restart it. Usually those restart fix 9/10 issues i have with openstack | |
| 13:08:51 | sean-k-mooney | what you likely shoudl do is try and find the request-id for the backup call and see if you can fined the last log for that operation | |
| 13:08:52 | EugenMayer | Which obviously is not a good sign, sure | |
| 13:08:58 | sean-k-mooney | to seee where it got stuck | |
| 13:09:37 | EugenMayer | this is SO hard to trace for me considering the amount of subsystems, proxies and systems involved | |
| 13:10:21 | EugenMayer | it usually is my bread and butter debugging those kind of things in our software stacks. But well, as i must learn, the reason i can do it there is - i know the software a lot better | |
| 13:22:25 | sean-k-mooney | ya there is a lot of context to grok | |
| 14:42:44 | viks__ | hi, with `soft-anti-affinity`, whenever i create 2 instances together via horizon, it goes in to 2 different hosts. But when i create one after the other with `soft-anti-affinity`, it goes in to the same host? is it expected? | |
| 15:01:28 | sean-k-mooney | viks__: the behavior of the second case will depend on if you waith for the first one to go active before you create the second | |
| 15:02:02 | sean-k-mooney | but yes it is expected in the second case you ahve a race to update the instance.host before the second vm is schduled | |
| 15:02:31 | sean-k-mooney | we provide no affinity garunetees in this case unless you enable the affinity upcall | |
| 15:05:00 | viks__ | sean-k-mooney: what all things i need to set for affinity upcall in the nova.conf? | |
| 15:24:13 | sean-k-mooney | the workaround config option in the compute nova.conf and api database in conductor nova.conf | |
| 15:39:29 | opendevreview | Alexey Stupnikov proposed openstack/nova stable/xena: Test aborting queued live migration https://review.opendev.org/c/openstack/nova/+/835853 | |
| 15:40:07 | opendevreview | Alexey Stupnikov proposed openstack/nova stable/xena: Add functional tests to reproduce bug #1960412 https://review.opendev.org/c/openstack/nova/+/835854 | |
| 15:41:04 | opendevreview | Alexey Stupnikov proposed openstack/nova stable/xena: Clean up when queued live migration aborted https://review.opendev.org/c/openstack/nova/+/835855 | |
| 15:53:11 | viks__ | sean-k-mooney: ok.. but even with `[workaround]/disable_group_policy_check_upcall = false`, i get the same behaviour | |
| 15:53:41 | sean-k-mooney | the second isntance will stilll race and get sent to the same host | |
| 15:53:48 | sean-k-mooney | but it shoudl then be rejected | |
| 15:53:54 | sean-k-mooney | and reschudled to a differnt host | |
| 15:54:01 | sean-k-mooney | form the alternate host list | |
| 17:09:04 | sean-k-mooney | noonedeadpunk: by they way https://specs.openstack.org/openstack/nova-specs/specs/ussuri/implemented/flavor-extra-spec-validators.html should have caught your typo if you use the correct microverion | |
| 17:14:56 | opendevreview | melanie witt proposed openstack/nova-specs master: Repropose spec for ephemeral storage encryption https://review.opendev.org/c/openstack/nova-specs/+/835877 | |
| 17:36:47 | opendevreview | melanie witt proposed openstack/nova-specs master: Repropose spec for ephemeral storage encryption https://review.opendev.org/c/openstack/nova-specs/+/835877 | |
| #openstack-nova - 2022-03-31 | |||
| 04:53:08 | opendevreview | melanie witt proposed openstack/nova-specs master: Make 'tox -e fast-specs' fail on sphinx warnings https://review.opendev.org/c/openstack/nova-specs/+/835930 | |
| 07:49:14 | kashyap | stephenfin: Thanks for the review here! - https://review.opendev.org/c/openstack/nova/+/762330/. Will address | |
| 08:05:58 | noonedeadpunk | sean-k-mooney: well... if openstackclient tended to use highest version available instead of lowest version supported... | |
| 08:56:57 | amoralej | hi kashyap, we need to discuss wrt https://review.rdoproject.org/r/c/openstack/nova-distgit/+/38029 it doesn't work with aarch64 as it seems there is no virtio-vga for it? | |
| 08:57:12 | kashyap | amoralej: Hi | |
| 08:58:56 | amoralej | kashyap, see https://gitlab.com/redhat/centos-stream/rpms/qemu-kvm/-/blob/c9s/qemu-kvm.spec#L582 | |
| 08:59:06 | kashyap | amoralej: Hmm, I don't top off my head, lemme check w/ AArch64 folks. | |
| 08:59:47 | amoralej | it seems it's only for x86_64 and power64 | |
| 08:59:54 | kashyap | amoralej: Isn't that line you linked to for 'power64'? | |
| 09:00:15 | kashyap | Hmm, yea; did you run into failures on AArch64 hosts? | |
| 09:01:00 | kashyap | amoralej: Maybe we can wrap the "Requires(pre):" in Nova under "%ifarch x86_64 %{power64}" | |
| 09:02:19 | amoralej | yes, but nova is noarch | |
| 09:02:58 | amoralej | and iirc %ifarch is only applied at build time, not at install time | |
| 09:03:04 | amoralej | but i need to check | |
| 09:08:51 | kashyap | amoralej: I have joined #rdo on Libera net, if you want to sort it out there. :) | |
| 09:09:07 | amoralej | kashyap, better #rdo in oftc, please :) | |
| 09:09:18 | kashyap | Ah, I didn't know it's on OFTC, joining | |
| 09:35:02 | andry_g | Hi all, we had inserted some vms into an host group for anti-affinity rules after creation, the vms were been created some months ago, we know that it's not possible by a doc but we tried to insert 2 records into db nova_api table instance_group_member. It's works fine but we have a problem if we try to do a resize of vm, we have this error: 2022-03-31 11:12:23.455 45106 ERROR oslo_messaging.rpc.server [req-be200e2d-af30-4f16-8571-f3a04533dbd6 5 | |
| 09:35:33 | andry_g | Exception during message handling: AttributeError: 'NoneType' object has no attribute 'hosts' Seems that something is missing in db. Could you give us any suggestion please? | |
| 09:38:47 | sean-k-mooney | you proably missed somethign in the requestspec in the api db | |
| 09:40:14 | sean-k-mooney | without knowing what line/file that was raised form (preferable the full traceback) i really cant say what is missing | |
| 09:43:32 | andry_g | https://paste.openstack.org/show/bmnRZsPkgpPjdIhwWUss/ | |
| 09:44:21 | sean-k-mooney | ya so its the group_info.host that is not populated when its trying to poulate teh request spec for schduling | |
| 09:44:23 | sean-k-mooney | request_spec.instance_group.hosts = list(group_info.hosts) | |
| 09:44:32 | andry_g | the record that has bee insert is insert into instance_group_member(created_at,updated_at,id,instance_uuid,group_id) values('2022-03-31 08:53:19',NULL,'138','13052619-612f-4eea-b34d-59d3d473329f','67'); | |
| 09:48:04 | andry_g | we try to fix it, I''ll let you know. Thank you | |
| 09:49:04 | sean-k-mooney | this is the code where its failing https://github.com/openstack/nova/blob/master/nova/scheduler/utils.py#L1168-L1208 | |
| 09:49:25 | sean-k-mooney | L1206 specifcally | |
| 09:50:15 | sean-k-mooney | did you update the request spec to set request_spec.instance_group | |
| 09:50:30 | sean-k-mooney | on the instance in addtion to insterting it into that db | |