| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2024-02-22 | |||
| 19:32:05 | sean-k-mooney | really | |
| 19:33:01 | sean-k-mooney | or did i forget to delete on eof the checks | |
| 19:36:03 | sean-k-mooney | so tha tot me say that we have not configure bandwith qos properly | |
| 19:36:43 | sean-k-mooney | am i need ot go have somethign to eat but ok ill just drop all the port stuff | |
| 19:36:59 | sean-k-mooney | and if that does not work we can drop all the heal code | |
| 19:39:44 | mnaser | https://bugs.launchpad.net/nova/+bug/2052915 is affecting nova right now but i have no idea where to look at the refernece | |
| 19:40:36 | melwitt | sean-k-mooney: I'm not sure .. it's failing on "bandwidth_allocations=$(echo "$allocations" | grep NET_BW_EGR_KILOBIT_PER_SEC)" | |
| 19:40:36 | sean-k-mooney | that sounds like neutron broke upgrades | |
| 19:40:59 | sean-k-mooney | yep so if bandwith qos was working that allocation should exist | |
| 19:41:50 | melwitt | the weird thing is that it's supposed to "echo "Failed to heal port allocations."" if the above returned "" but it's not echoing that | |
| 19:42:22 | sean-k-mooney | yep we have allcoations but not for that type | |
| 19:42:32 | melwitt | oh, I see. ok | |
| 19:42:58 | sean-k-mooney | so what i ment by not configre correctly is i think we may not have enabel the qos extionion in the job | |
| 19:43:08 | sean-k-mooney | or somethign liek that | |
| 19:43:37 | opendevreview | sean mooney proposed openstack/nova master: [S-RBAC] adapt nova-next for port's binding:profile field change https://review.opendev.org/c/openstack/nova/+/909859 | |
| 19:44:24 | melwitt | doesn't the fact that it used to work before the service token change mean the extension is present? | |
| 19:44:52 | sean-k-mooney | i would have to read the code but not nessiarlay | |
| 19:45:19 | melwitt | ok | |
| 19:45:23 | sean-k-mooney | the allocation is create by nova but if we dont have teh qos extion is not loaded | |
| 19:45:43 | sean-k-mooney | what will happen is neutron will not make a placement resouce request in teh port | |
| 19:46:32 | sean-k-mooney | melwitt: the hook was not asserting the bandwith resouce request existed before we healed | |
| 19:46:54 | sean-k-mooney | or that the the port has port resouce requests | |
| 19:47:57 | melwitt | 😵💫 | |
| 19:55:18 | sean-k-mooney | it does look like the qos plugin was loaded https://7afe9236b7eaaf244ae9-d98f699786987cf0f7a232f67c9b09f7.ssl.cf2.rackcdn.com/909859/2/check/nova-next/3c2c795/controller/logs/etc/neutron/neutron_conf.txt | |
| 19:56:26 | sean-k-mooney | and we see | |
| 19:56:28 | sean-k-mooney | 2024-02-22 18:53:27.154019 | controller | | resource_request | {'request_groups': [{'id': '35e2b68f-9d87-54ff-a7c1-c0772986407e', 'required': ['CUSTOM_PHYSNET_PUBLIC', 'CUSTOM_VNIC_TYPE_NORMAL'], 'resources': {'NET_BW_EGR_KILOBIT_PER_SEC': 1000, 'NET_BW_IGR_KILOBIT_PER_SEC': 1000}}], 'same_subtree': ['35e2b68f-9d87-54ff-a7c1-c0772986407e']} | | |
| 19:56:35 | sean-k-mooney | in the neutron port | |
| 19:57:00 | sean-k-mooney | so the neutron port appears to have the port resouce reqquest in it | |
| 19:57:45 | sean-k-mooney | its also in allocations we fore we delete them | |
| 19:58:17 | sean-k-mooney | melwitt: but its not after we heal the allcoations | |
| 19:58:58 | sean-k-mooney | https://paste.opendev.org/show/bltzmAmf8ZoEE2NEimRR/ | |
| 20:00:31 | sean-k-mooney | melwitt: i need to finish for today but i wonder if there is another regression here | |
| 20:01:21 | melwitt | sean-k-mooney: hm, ok. thanks for looking o/ | |
| 20:06:42 | sean-k-mooney | https://github.com/openstack/nova/blob/master/nova/cmd/manage.py#L1749 | |
| 20:06:50 | sean-k-mooney | i would expec that to print somewhere firht | |
| 20:07:57 | sean-k-mooney | or this https://github.com/openstack/nova/blob/master/nova/cmd/manage.py#L1760C1-L1785C62 | |
| 20:16:05 | sean-k-mooney | ok im actully going to leave now but this is the neutron clien twe are using in nova_manage | |
| 20:16:07 | sean-k-mooney | https://github.com/openstack/nova/blob/master/nova/network/neutron.py#L248 | |
| 20:16:18 | sean-k-mooney | we are creating it with admin=true | |
| 20:16:52 | sean-k-mooney | and this is where the service auth token shoudl be enabled | |
| 20:16:54 | sean-k-mooney | https://github.com/openstack/nova/blob/master/nova/network/neutron.py#L219-L233 | |
| 20:17:49 | sean-k-mooney | so it shoudl have an admin client with service tokens | |
| 20:18:04 | sean-k-mooney | but the nova code seams to not be healign the port so we should see why | |
| 20:20:00 | melwitt | sean-k-mooney: thanks for the guidance | |
| 21:55:59 | melwitt | looks like nova-ceph-multistore is at a 100% fail too with "Details: b'400 Bad Request\n\nThe Store URI was malformed.\n\n '" | |
| 21:58:56 | melwitt | 😩 | |
| 22:58:19 | opendevreview | melanie witt proposed openstack/nova-specs master: Update ephemeral encryption specs to reflect implementation https://review.opendev.org/c/openstack/nova-specs/+/907654 | |
| #openstack-nova - 2024-02-23 | |||
| 01:50:55 | opendevreview | melanie witt proposed openstack/nova master: Report ephemeral disk encryption in the metadata API https://review.opendev.org/c/openstack/nova/+/909945 | |
| 02:26:03 | opendevreview | melanie witt proposed openstack/nova master: Deprecate legacy ephemeral storage encryption using dm-crypt https://review.opendev.org/c/openstack/nova/+/909947 | |
| 02:54:39 | melwitt | looks like nova-ceph-multistore is fixed by the merge of https://review.opendev.org/c/openstack/requirements/+/909923 | |
| 05:31:20 | opendevreview | Merged openstack/nova master: [S-RBAC] adapt nova-next for port's binding:profile field change https://review.opendev.org/c/openstack/nova/+/909859 | |
| 05:32:12 | opendevreview | Maxim Monin proposed openstack/nova master: Server Rescue leads to Server ERROR state if original image is deleted https://review.opendev.org/c/openstack/nova/+/872385 | |
| 05:52:40 | opendevreview | melanie witt proposed openstack/nova master: Deprecate legacy ephemeral storage encryption using dm-crypt https://review.opendev.org/c/openstack/nova/+/909947 | |
| 07:42:06 | opendevreview | Maxim Monin proposed openstack/nova master: Server Rescue leads to Server ERROR state if original image is deleted https://review.opendev.org/c/openstack/nova/+/872385 | |
| 08:03:24 | opendevreview | Dr. Jens Harbott proposed openstack/nova master: DNM: Test glance new location api https://review.opendev.org/c/openstack/nova/+/891207 | |
| 09:36:51 | LarsErikP | I have discovered something I wonder if it's a bug, or if it is intended behavior. In yoga at least, I can unshelve a VM that will cause an overconsumption of the CPU quota.. | |
| 09:38:59 | LarsErikP | Created a project with 4 VCPUs and 4GB RAM. Then spawned server1 with 2 cores and 2GB RAM. Shelved it. After that i spawned server2 with 4 cores and 2GB RAM. I have now used 4 of 4 CPUs in the quota. Then I unshelved server1 without any problem, and now I have used 6 of 4 CPUs... | |
| 09:39:34 | LarsErikP | I.. did not think this should be possible, as it enables users to easily overconsume their quota.. | |
| 09:48:42 | LarsErikP | you can do the same trick with overconsuming RAM as well | |
| 09:51:06 | LarsErikP | and, fwiw, both at once is also possible | |
| 10:33:34 | bauzas | LarsErikP: most of the community today is on PTO | |
| 10:34:04 | bauzas | are you sure that your limits were for 4 VCPUs ? | |
| 10:48:34 | LarsErikP | yes | |
| 10:49:52 | LarsErikP | I filed a bug anyways :P https://bugs.launchpad.net/nova/+bug/2054797 | |
| 10:52:56 | LarsErikP | and it kind of scales.. meaning if I create a bunch of VMs, shelve all of them, create a new one consuming the entire quota; I can stil unshelve all the shelved VMs | |
| 11:06:42 | opendevreview | sean mooney proposed openstack/nova master: add pyproject.toml to support pip 23.1 https://review.opendev.org/c/openstack/nova/+/899753 | |
| 11:20:49 | opendevreview | Konrad Gube proposed openstack/nova master: Use Cinder's os-extend_volume_completion volume action. https://review.opendev.org/c/openstack/nova/+/873560 | |
| 11:55:26 | opendevreview | sean mooney proposed openstack/nova master: Revert "[S-RBAC] adapt nova-next for port's binding:profile field change" https://review.opendev.org/c/openstack/nova/+/909834 | |
| 12:47:52 | kgube | bauzas, gibi: Hi, could you have another look at this? https://review.opendev.org/c/openstack/nova/+/873560 | |
| 13:26:19 | opendevreview | Merged openstack/nova master: Follow up patch to correct a minor documentation issue https://review.opendev.org/c/openstack/nova/+/908863 | |
| 14:21:07 | opendevreview | Merged openstack/os-vif master: Fix missing or unnecessary dependencies https://review.opendev.org/c/openstack/os-vif/+/909341 | |
| 14:49:13 | u00599 | Hey, we came across a problem in our openstack environment where the metadata-service is not marking interfaces with dhcp_enabled for network config with cloud-init. We came across the following commit which wanted to solve this problem, but this doesn't work for ovn with native dhcp (no agents) as there are no dhcp-agents anymore. The concerned networks don't have a default gateway set, so the "fix" is not working for us. Anybody came | |
| 14:49:22 | u00599 | https://github.com/openstack/nova/commit/c7f572d65b57e034c1391ed49d84f6e5f1d672ad | |
| 15:17:09 | artom | So it's hw:emulator_threads_policy with the s but hw:cpu_policy without the s | |
| 15:17:15 | artom | We're just evil, aren't we | |
| 15:40:01 | opendevreview | Merged openstack/nova master: check both source and dest compute libvirt versions for mdev lv https://review.opendev.org/c/openstack/nova/+/904176 | |
| 15:40:32 | jamespage | LarsErikP: https://bugs.launchpad.net/cloud-archive/+bug/2054687 to track | |
| 15:46:03 | opendevreview | Merged openstack/nova master: Check if destination can support the src mdev types https://review.opendev.org/c/openstack/nova/+/904177 | |
| 15:46:10 | opendevreview | Merged openstack/nova master: Reserve mdevs to return to the source https://review.opendev.org/c/openstack/nova/+/904209 | |
| 15:52:24 | sean-k-mooney | u00599: ovn has its ovn metadata agent | |
| 15:53:01 | sean-k-mooney | u00599: but unless they have fixed it neutron is not waiting for it to be ready before seneding the network-vif-plugged event | |
| 15:54:12 | sean-k-mooney | so there is or was a race when using ovn | |
| 16:00:16 | sean-k-mooney | LarsErikP: this is related to counting limits in placment | |
| 16:06:21 | sean-k-mooney | https://docs.openstack.org/nova/latest/configuration/config.html#quota.count_usage_from_placement | |
| 16:06:47 | sean-k-mooney | Behavior will be different for servers in SHELVED_OFFLOADED state. A server in SHELVED_OFFLOADED state will not have placement allocations, so it will not consume quota usage for cores and ram. Note that because of this, it will be possible for a request to unshelve a server to be rejected if the user does not have enough quota available to support the cores and ram needed by | |
| 16:06:49 | sean-k-mooney | the server to be unshelved. | |
| 16:16:22 | sean-k-mooney | LarsErikP: https://review.opendev.org/c/openstack/nova/+/888096 is the yoga fix but yoga is now unmaintained https://opendev.org/openstack/nova/src/branch/unmaintained/yoga | |
| 16:17:25 | sean-k-mooney | hum although im not sure that is corect since wallay and victoria should be unmainted then too | |
| 16:17:34 | sean-k-mooney | and xena | |
| 16:17:54 | sean-k-mooney | elodilles: ^ we should look into what up there | |
| 16:18:09 | sean-k-mooney | https://opendev.org/openstack/nova/branches | |
| 16:18:42 | sean-k-mooney | i forget what we had agreed but i taught we agreed not to skip any releases | |
| 16:18:57 | sean-k-mooney | so if yoga is moved to unamined i woudl expect all older branch to be as well | |
| 16:29:31 | elodilles | sean-k-mooney: we started the transition with yoga, as that had a 'deadline' some months ago... now we'll continue with the rest (xena, wallaby and victoria), actually, i just said on the Release meeting that i'm planning to propose the transition patches on Monday (despite the busy Caracal-3 milestone busy time... :S) | |
| 16:30:20 | sean-k-mooney | elodilles: oh ok | |
| 16:30:25 | sean-k-mooney | so they are just in flight | |
| 16:30:27 | sean-k-mooney | cool | |
| 16:31:45 | elodilles | yepp-yepp | |
| 16:31:49 | elodilles | that's the plan | |
| 16:52:10 | artom | Bleargh, the CPU power management stuff is not testable for multiple compute hosts in functional tests | |