| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2024-09-02 | |||
| 15:44:55 | noonedeadpunk | hey folks! We've just updated one of our regions to 2024.1 and we do see a severe performance hit comparing to 2023.1. at least when trying to execute smth like `openstack server list --all-projects`. We've noticed, as we had to double timeout limit on the loadbalancer for nova-api backends | |
| 15:45:19 | noonedeadpunk | from 60s to 3m as listing takes reliably 2m19s at the moment. | |
| 15:45:34 | noonedeadpunk | do you happen to be aware of anything like that? | |
| 15:47:38 | ykarel | sean-k-mooney, ack thx will check and update that hack patch tomorrow, need to drop now | |
| 16:01:00 | sean-k-mooney | noonedeadpunk: not that im aware off, you should avoid that query but im not sure why would have regressed that | |
| 16:01:44 | sean-k-mooney | hum | |
| 16:01:53 | noonedeadpunk | we've noticed as ruinning it quite periodically for billing | |
| 16:02:07 | sean-k-mooney | noonedeadpunk: there might be 2 things in effect | |
| 16:02:09 | sean-k-mooney | https://docs.openstack.org/nova/latest/reference/api-microversion-history.html#maximum-in-2024-1-caracal | |
| 16:02:23 | sean-k-mooney | we added the pinned_avaiablity zone which need to read form the request spec | |
| 16:02:44 | sean-k-mooney | noonedeadpunk: and around that time openstack client started using the latest microversion instaead of 2.1 | |
| 16:03:03 | noonedeadpunk | well... I don't do --long | |
| 16:03:18 | sean-k-mooney | noonedeadpunk: can you try setting --os-compute-api 2.1 | |
| 16:03:54 | sean-k-mooney | if your osc is new enough to default to latest you got all the other thigns we added in the last 10 years too | |
| 16:04:06 | noonedeadpunk | frankly - I was kind thinking more in a direction for compatability choices made for sqlalchemy 2 | |
| 16:04:28 | sean-k-mooney | i dont think we are aware of any performace impacts for any of those | |
| 16:04:47 | sean-k-mooney | certenly not doubling the responce time | |
| 16:05:27 | sean-k-mooney | noonedeadpunk: can you try pinning the microversion to 2.1 | |
| 16:05:38 | noonedeadpunk | yeah, sec | |
| 16:06:10 | noonedeadpunk | ok, so `--long` takes *exact* same time as not `--long` | |
| 16:06:54 | noonedeadpunk | so I assume long just adds more response while all the same stuff is gathered | |
| 16:07:12 | sean-k-mooney | long might be clinet side only parsing | |
| 16:07:37 | sean-k-mooney | im not sure it changes the actually api request unless its changing between /servers/ and /servers/detail | |
| 16:08:15 | noonedeadpunk | `--os-compute-api 2.1` doesn't make any difference to me | |
| 16:09:21 | noonedeadpunk | yeah, seems just a client thing | |
| 16:09:22 | sean-k-mooney | ok i guess we should capture this in a bug as a possibel performance reguression between 2023.1 and 2024.1 | |
| 16:10:01 | noonedeadpunk | ok, I just wanted to check if you might be aware of that before digging deeper | |
| 16:10:59 | sean-k-mooney | unfortunetly i suspect whatever it may be does not present on devstack or other small deployments | |
| 16:11:58 | sean-k-mooney | i tought maybe it was https://docs.openstack.org/nova/latest/reference/api-microversion-history.html#id63 | |
| 16:12:33 | sean-k-mooney | since if you have down cells that might change behaivor | |
| 16:12:53 | noonedeadpunk | nah, it's a single cell and single AZ environment | |
| 16:13:10 | noonedeadpunk | as we've started upgrade from the smallest thing we had | |
| 16:13:47 | sean-k-mooney | ya so get /servers at the nova api level should be very quick | |
| 16:13:51 | sean-k-mooney | https://docs.openstack.org/api-ref/compute/#id9 | |
| 16:14:01 | sean-k-mooney | as it only retuns the minimal info in the api db | |
| 16:14:08 | sean-k-mooney | but openstack server list | |
| 16:14:13 | sean-k-mooney | is not the same thing as that | |
| 16:14:39 | sean-k-mooney | i think it does more behind the scenes | |
| 16:15:15 | sean-k-mooney | noonedeadpunk: around that time osc started replacing using python-novaclient with the sdk as well | |
| 16:15:22 | noonedeadpunk | client also does pass bunch of filters: /v2.1/servers/detail?flavor=None&image=None&name=None&project_id=None&reservation_id=None&status=None&user_id=None&deleted=False&host=None&ip=None&ip6=None&changes-since=None&changes-before=None&all_tenants=True | |
| 16:15:34 | noonedeadpunk | so I wonder if that might be client that I've upgraded | |
| 16:15:56 | sean-k-mooney | you could try doing it with nova client or installing an older osc in a venv | |
| 16:16:09 | sean-k-mooney | that woudl at least tell us if its server or client side | |
| 16:16:12 | noonedeadpunk | or just do REST call | |
| 16:16:17 | sean-k-mooney | yep or that | |
| 16:16:40 | sean-k-mooney | you can do it with curl if you have a keyston token but i always have to look up the syntax | |
| 16:19:42 | noonedeadpunk | so curl takes half of the time, as I assume osc does fetch more info to render uuids | |
| 16:20:04 | noonedeadpunk | still 1m24s while haproxy used to be set to 60s timeout | |
| 16:20:37 | noonedeadpunk | and request was `time curl -g -i --cacert "/etc/ssl/certs/ca-certificates.crt" -X GET "http://cloud.com:8774/v2.1/servers/detail?all_tenants=True" -H "Accept: application/json" -H "X-Auth-Token: ${token}" -H "X-OpenStack-Nova-API-Version: 2.92"` | |
| 16:26:44 | sean-k-mooney | if you have the request id it would be good to check how long the request taks on the nova-api side | |
| 16:27:12 | sean-k-mooney | /server/detail is not the same as /server | |
| 16:27:33 | sean-k-mooney | so if you dont want all the addtional info avoid /server/detail | |
| 16:28:26 | noonedeadpunk | well, I'm not sure about detail part, but assume it was preffered to limit amount of following requests. | |
| 16:28:40 | noonedeadpunk | but also interesting to find | |
| 16:29:12 | sean-k-mooney | well /detail is not really advisabel on large cloud if using --all-tenatns | |
| 16:29:35 | sean-k-mooney | it can inovled calls to neutron or other sirvices for info in some cases | |
| 16:29:59 | sean-k-mooney | we mostly try to cache some of that info but we dont cache all of it | |
| 16:34:27 | sean-k-mooney | i am thinking of https://bugs.launchpad.net/nova/+bug/1923560 | |
| 16:34:43 | sean-k-mooney | without https://review.opendev.org/c/openstack/nova/+/786348 we need to do a call to neutron for every server | |
| 16:34:53 | sean-k-mooney | if you use /server/detail | |
| 17:41:02 | opendevreview | sean mooney proposed openstack/nova master: enable numa live migration in the ceph job https://review.opendev.org/c/openstack/nova/+/913842 | |
| 17:41:36 | opendevreview | sean mooney proposed openstack/nova master: enable numa live migration in the ceph job https://review.opendev.org/c/openstack/nova/+/913842 | |
| 22:53:43 | opendevreview | sean mooney proposed openstack/nova master: [DNM] testing rdo experimental jobs https://review.opendev.org/c/openstack/nova/+/927762 | |
| #openstack-nova - 2024-09-03 | |||
| 00:56:17 | opendevreview | Jay Faulkner proposed openstack/nova master: [ironic] Followup fixes around flavor ids https://review.opendev.org/c/openstack/nova/+/927764 | |
| 01:39:54 | stan | Guys can I set a limit of how many VMs can be created simultaneously? Let's say if the user wants to create 20 VMs at a time, is there a mechanism to create 5vms in the first batch, and put the rest of the vm request in queue? | |
| 01:45:21 | bauzas | I officially called out the FeatureFreeze | |
| 01:45:40 | bauzas | all the 2024.2 blueprints are now either Deferred or Implemented | |
| 01:45:53 | bauzas | numbers aren't great, tbh. | |
| 01:46:47 | bauzas | https://blueprints.launchpad.net/nova/2024.2 | |
| 06:33:08 | opendevreview | Takashi Kajinami proposed openstack/nova master: Add hw_mem_encryption_model image property https://review.opendev.org/c/openstack/nova/+/927706 | |
| 06:33:09 | opendevreview | Takashi Kajinami proposed openstack/nova master: libvirt: Launch instances with SEV-ES memory encryption https://review.opendev.org/c/openstack/nova/+/926106 | |
| 07:34:25 | opendevreview | Viktor Křivák proposed openstack/nova master: Ensure claims to placement are freed after evacuate complete Improve resource_tracker delete_allocation_for_evacuated_instance to handle non-cached compute hosts Fix test to accept this change https://review.opendev.org/c/openstack/nova/+/926292 | |
| 09:05:40 | opendevreview | Takashi Kajinami proposed openstack/nova master: Add hw_mem_encryption_model image property https://review.opendev.org/c/openstack/nova/+/927706 | |
| 09:05:40 | opendevreview | Takashi Kajinami proposed openstack/nova master: libvirt: Launch instances with SEV-ES memory encryption https://review.opendev.org/c/openstack/nova/+/926106 | |
| 10:43:59 | sean-k-mooney | stan: not on a per user basis or at teh api level | |
| 10:44:08 | sean-k-mooney | stan: you can set a limit per compute node | |
| 10:44:37 | sean-k-mooney | it defaults to 10 https://docs.openstack.org/nova/latest/configuration/config.html#DEFAULT.max_concurrent_builds | |
| 10:59:37 | stan | sean-k-mooney Thanks for the information. :) | |
| 11:04:08 | stan | I assume I can make changes to this in combination with scheduler timeout to achieve our use case. | |
| 11:04:30 | sean-k-mooney | note really | |
| 11:04:52 | sean-k-mooney | the compute node config only takes effect after the instance is schduled and just delays it spawning | |
| 11:17:38 | stan | sean-k-mooney Alright. I will test this in our lab 👍 | |
| 11:30:02 | stan | sean-k-mooney So nova timeout will effectively come into play after its schedule or after the VM starts to spawn? | |
| 11:30:46 | sean-k-mooney | no | |
| 11:31:36 | sean-k-mooney | there are a few tiemouts at play. there is an api request time out for the intiall http respocne, then there is the rpc timeout between the conductor schduelr adn computes | |
| 11:32:04 | sean-k-mooney | and then there is an instance build timeout https://docs.openstack.org/nova/latest/configuration/config.html#DEFAULT.instance_build_timeout | |
| 11:32:23 | sean-k-mooney | which defaults ot 0 meaning no limit | |
| 12:16:56 | sean-k-mooney | dansmith: looks like we have another test that should be skiped without qemu-img | |
| 12:16:58 | sean-k-mooney | https://paste.opendev.org/show/bOFlCycmdpOV1shHsHQX/ | |
| 12:17:36 | sean-k-mooney | did we plan to merge your patch to use the oslo version? i lost track of where we were with that | |
| 12:29:51 | jlejeune_ | sean-k-mooney: hello, indeed, that's why I've proposed: https://review.opendev.org/c/openstack/nova/+/925208 | |
| 12:30:16 | sean-k-mooney | ah yes i tought this was familar | |
| 12:30:23 | jlejeune | sean-k-mooney: :) | |
| 12:31:12 | sean-k-mooney | i am usign distobox to create a clean dev env on my laptop to workaround some packaging issues so didnt have it installed | |
| 12:31:48 | jlejeune | like on our CI | |
| 12:32:27 | sean-k-mooney | well in the first party ci we do have qemu-img isntalled | |
| 12:32:35 | sean-k-mooney | but in general it should not be requried | |
| 12:33:22 | sean-k-mooney | https://review.opendev.org/c/openstack/nova/+/925208/comment/2b6ac8e7_286f92b1/ is still an open question that needs input form others | |
| 12:34:47 | sean-k-mooney | im ok with the stub as i dont think the existin unit test should be executing that code but we still need others to weigh in | |
| 12:35:19 | sean-k-mooney | jlejeune: to be honest i had tought we merged that already | |