Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2022-03-30
12:52:36 sean-k-mooney then when the io compelte the previous green trhead is added to the queue to be resumed
12:52:36 EugenMayer i see, this is the nova-compute process then, right?
12:52:45 sean-k-mooney yes
12:52:57 sean-k-mooney nova-compute but also conductor and schduler
12:53:30 sean-k-mooney technially nova-api is monkeypatch but the way its run with appache means it only process one request per worker process
12:53:45 sean-k-mooney because apache queues the request before it get to the api application
12:57:10 EugenMayer oh holy moly.
12:58:57 EugenMayer I mean, my day job is being a software engeneer. Yes with bigger EE software, yes with microservices, distributed and all that. But this really is very weired to me - or it is simply to complex for me to play around in the mind since i do not know any components properly and have no save-haven to return / start thinking from
12:59:03 EugenMayer thank you for elaborating on that
13:00:03 EugenMayer I left with 2 things: glance has an tasks status 'queued' of an tasks that does not exists and it is unclear where this state comes from. Second is, why my compute3 (out of 4) fails to create any backups, all others can.
13:00:43 EugenMayer Ah now i understand - not a task is 'queued' .. the image is queued - without any task. So it is the image state
13:01:45 sean-k-mooney a very long time ago around the catus release opensack moved form twisted to eventlet to remove the need for peopel to explcitly think about multithreading and concurancy most of the time. howver ther eare still case where you have to use locks ectra to ensure no data races. so for the most part eventlet simplifes the common code path when you are io bound which tends to be
13:01:47 sean-k-mooney the case for nova
13:02:26 sean-k-mooney yes the image is queue
13:02:45 sean-k-mooney not a task
13:05:01 sean-k-mooney https://docs.openstack.org/glance/latest/user/statuses.html
13:05:09 sean-k-mooney queued
13:05:11 sean-k-mooney The image identifier has been reserved for an image in the Glance registry. No image data has been uploaded to Glance and the image size was not explicitly set to zero on creation.
13:05:32 sean-k-mooney ok so queue means we have crerate the image but not uploaded it
13:06:22 sean-k-mooney which i guess make sense since nova is not currently in the image_uploading task_state
13:06:59 sean-k-mooney so that likely means that nova is failing to create the snapshot via libvirt/qemu
13:07:38 EugenMayer i see, thank you so much sean!
13:08:01 EugenMayer i removed the broken (queued) images now
13:08:10 EugenMayer reset the instances and restarted them
13:08:17 sean-k-mooney ack
13:08:39 EugenMayer i will check the logs of that compute once again and then restart it. Usually those restart fix 9/10 issues i have with openstack
13:08:51 sean-k-mooney what you likely shoudl do is try and find the request-id for the backup call and see if you can fined the last log for that operation
13:08:52 EugenMayer Which obviously is not a good sign, sure
13:08:58 sean-k-mooney to seee where it got stuck
13:09:37 EugenMayer this is SO hard to trace for me considering the amount of subsystems, proxies and systems involved
13:10:21 EugenMayer it usually is my bread and butter debugging those kind of things in our software stacks. But well, as i must learn, the reason i can do it there is - i know the software a lot better
13:22:25 sean-k-mooney ya there is a lot of context to grok
14:42:44 viks__ hi, with `soft-anti-affinity`, whenever i create 2 instances together via horizon, it goes in to 2 different hosts. But when i create one after the other with `soft-anti-affinity`, it goes in to the same host? is it expected?
15:01:28 sean-k-mooney viks__: the behavior of the second case will depend on if you waith for the first one to go active before you create the second
15:02:02 sean-k-mooney but yes it is expected in the second case you ahve a race to update the instance.host before the second vm is schduled
15:02:31 sean-k-mooney we provide no affinity garunetees in this case unless you enable the affinity upcall
15:05:00 viks__ sean-k-mooney: what all things i need to set for affinity upcall in the nova.conf?
15:24:13 sean-k-mooney the workaround config option in the compute nova.conf and api database in conductor nova.conf
15:39:29 opendevreview Alexey Stupnikov proposed openstack/nova stable/xena: Test aborting queued live migration https://review.opendev.org/c/openstack/nova/+/835853
15:40:07 opendevreview Alexey Stupnikov proposed openstack/nova stable/xena: Add functional tests to reproduce bug #1960412 https://review.opendev.org/c/openstack/nova/+/835854
15:41:04 opendevreview Alexey Stupnikov proposed openstack/nova stable/xena: Clean up when queued live migration aborted https://review.opendev.org/c/openstack/nova/+/835855
15:53:11 viks__ sean-k-mooney: ok.. but even with `[workaround]/disable_group_policy_check_upcall = false`, i get the same behaviour
15:53:41 sean-k-mooney the second isntance will stilll race and get sent to the same host
15:53:48 sean-k-mooney but it shoudl then be rejected
15:53:54 sean-k-mooney and reschudled to a differnt host
15:54:01 sean-k-mooney form the alternate host list
17:09:04 sean-k-mooney noonedeadpunk: by they way https://specs.openstack.org/openstack/nova-specs/specs/ussuri/implemented/flavor-extra-spec-validators.html should have caught your typo if you use the correct microverion
17:14:56 opendevreview melanie witt proposed openstack/nova-specs master: Repropose spec for ephemeral storage encryption https://review.opendev.org/c/openstack/nova-specs/+/835877
17:36:47 opendevreview melanie witt proposed openstack/nova-specs master: Repropose spec for ephemeral storage encryption https://review.opendev.org/c/openstack/nova-specs/+/835877
#openstack-nova - 2022-03-31
04:53:08 opendevreview melanie witt proposed openstack/nova-specs master: Make 'tox -e fast-specs' fail on sphinx warnings https://review.opendev.org/c/openstack/nova-specs/+/835930
07:49:14 kashyap stephenfin: Thanks for the review here! - https://review.opendev.org/c/openstack/nova/+/762330/. Will address
08:05:58 noonedeadpunk sean-k-mooney: well... if openstackclient tended to use highest version available instead of lowest version supported...
08:56:57 amoralej hi kashyap, we need to discuss wrt https://review.rdoproject.org/r/c/openstack/nova-distgit/+/38029 it doesn't work with aarch64 as it seems there is no virtio-vga for it?
08:57:12 kashyap amoralej: Hi
08:58:56 amoralej kashyap, see https://gitlab.com/redhat/centos-stream/rpms/qemu-kvm/-/blob/c9s/qemu-kvm.spec#L582
08:59:06 kashyap amoralej: Hmm, I don't top off my head, lemme check w/ AArch64 folks.
08:59:47 amoralej it seems it's only for x86_64 and power64
08:59:54 kashyap amoralej: Isn't that line you linked to for 'power64'?
09:00:15 kashyap Hmm, yea; did you run into failures on AArch64 hosts?
09:01:00 kashyap amoralej: Maybe we can wrap the "Requires(pre):" in Nova under "%ifarch x86_64 %{power64}"
09:02:19 amoralej yes, but nova is noarch
09:02:58 amoralej and iirc %ifarch is only applied at build time, not at install time
09:03:04 amoralej but i need to check
09:08:51 kashyap amoralej: I have joined #rdo on Libera net, if you want to sort it out there. :)
09:09:07 amoralej kashyap, better #rdo in oftc, please :)
09:09:18 kashyap Ah, I didn't know it's on OFTC, joining
09:35:02 andry_g Hi all, we had inserted some vms into an host group for anti-affinity rules after creation, the vms were been created some months ago, we know that it's not possible by a doc but we tried to insert 2 records into db nova_api table instance_group_member. It's works fine but we have a problem if we try to do a resize of vm, we have this error: 2022-03-31 11:12:23.455 45106 ERROR oslo_messaging.rpc.server [req-be200e2d-af30-4f16-8571-f3a04533dbd6 5
09:35:33 andry_g Exception during message handling: AttributeError: 'NoneType' object has no attribute 'hosts' Seems that something is missing in db. Could you give us any suggestion please?
09:38:47 sean-k-mooney you proably missed somethign in the requestspec in the api db
09:40:14 sean-k-mooney without knowing what line/file that was raised form (preferable the full traceback) i really cant say what is missing
09:43:32 andry_g https://paste.openstack.org/show/bmnRZsPkgpPjdIhwWUss/
09:44:21 sean-k-mooney ya so its the group_info.host that is not populated when its trying to poulate teh request spec for schduling
09:44:23 sean-k-mooney request_spec.instance_group.hosts = list(group_info.hosts)
09:44:32 andry_g the record that has bee insert is insert into instance_group_member(created_at,updated_at,id,instance_uuid,group_id) values('2022-03-31 08:53:19',NULL,'138','13052619-612f-4eea-b34d-59d3d473329f','67');
09:48:04 andry_g we try to fix it, I''ll let you know. Thank you
09:49:04 sean-k-mooney this is the code where its failing https://github.com/openstack/nova/blob/master/nova/scheduler/utils.py#L1168-L1208
09:49:25 sean-k-mooney L1206 specifcally
09:50:15 sean-k-mooney did you update the request spec to set request_spec.instance_group
09:50:30 sean-k-mooney on the instance in addtion to insterting it into that db
10:04:21 andry_g Thank you Sean, it's working fine!
10:04:53 sean-k-mooney cool
10:10:15 andry_g sean-k-mooney: there is another method on new release to do it without modify db?
10:10:48 sean-k-mooney no there was a feature request to add support about a year ago but it was not implemented
10:13:40 sean-k-mooney https://review.opendev.org/c/openstack/nova-specs/+/782353
10:14:27 sean-k-mooney andry_g:^ that was the most recent proposal but it did not have agreement yet on how to allow it
10:14:37 sean-k-mooney no one is currenlty working on it as far as i am aware
10:20:05 sean-k-mooney andry_g: just lookign that the comments it seams like we were convergin on extneing the move oeprations so resize, shelve, cold migrate and live migrate to allow you to migrate into a group
12:59:50 viks__ sean-k-mooney: i have tried to find the "upcall" in nova logs for investigating the anti-affinity, but did not find any even after enabling debug log.. not sure what i'm doing wrongly
13:04:16 sean-k-mooney so if the polciy is violated we shoudl raise thei exception https://opendev.org/openstack/nova/src/branch/master/nova/compute/manager.py#L1782-L1786
13:04:43 sean-k-mooney or this one if its affinity https://opendev.org/openstack/nova/src/branch/master/nova/compute/manager.py#L1796-L1799
13:05:24 sean-k-mooney you should atlest see that in the conductor log but i think you shoudl see that in the compute log too
13:06:36 sean-k-mooney it should be caut and logged here i think https://opendev.org/openstack/nova/src/branch/master/nova/compute/manager.py#L2318-L2323
13:07:06 sean-k-mooney well actully here https://opendev.org/openstack/nova/src/branch/master/nova/compute/manager.py#L2333
13:09:13 opendevreview Erlon R. Cruz proposed openstack/nova stable/yoga: Fix pre_live_migration rollback https://review.opendev.org/c/openstack/nova/+/836014
13:10:07 opendevreview Erlon R. Cruz proposed openstack/nova stable/xena: Fix pre_live_migration rollback https://review.opendev.org/c/openstack/nova/+/836015
13:10:18 opendevreview Erlon R. Cruz proposed openstack/nova stable/wallaby: Fix pre_live_migration rollback https://review.opendev.org/c/openstack/nova/+/836016
13:10:30 opendevreview Erlon R. Cruz proposed openstack/nova stable/victoria: Fix pre_live_migration rollback https://review.opendev.org/c/openstack/nova/+/836017
13:10:42 opendevreview Erlon R. Cruz proposed openstack/nova stable/ussuri: Fix pre_live_migration rollback https://review.opendev.org/c/openstack/nova/+/836018
13:14:08 viks__ sean-k-mooney: to get myself clarified, if i create 2 instance i.e. create 1st instance with soft-anti-affinity and then after that got created if i create one more with the same, it should go to the same node or different...? i'm not sure if this method is the proper way of testing the upcall..
13:14:39 viks__ also i'm using openstack-train .. is the upcall supposed to work in this release?

Earlier   Later