| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2024-02-22 | |||
| 19:45:23 | sean-k-mooney | the allocation is create by nova but if we dont have teh qos extion is not loaded | |
| 19:45:43 | sean-k-mooney | what will happen is neutron will not make a placement resouce request in teh port | |
| 19:46:32 | sean-k-mooney | melwitt: the hook was not asserting the bandwith resouce request existed before we healed | |
| 19:46:54 | sean-k-mooney | or that the the port has port resouce requests | |
| 19:47:57 | melwitt | 😵💫 | |
| 19:55:18 | sean-k-mooney | it does look like the qos plugin was loaded https://7afe9236b7eaaf244ae9-d98f699786987cf0f7a232f67c9b09f7.ssl.cf2.rackcdn.com/909859/2/check/nova-next/3c2c795/controller/logs/etc/neutron/neutron_conf.txt | |
| 19:56:26 | sean-k-mooney | and we see | |
| 19:56:28 | sean-k-mooney | 2024-02-22 18:53:27.154019 | controller | | resource_request | {'request_groups': [{'id': '35e2b68f-9d87-54ff-a7c1-c0772986407e', 'required': ['CUSTOM_PHYSNET_PUBLIC', 'CUSTOM_VNIC_TYPE_NORMAL'], 'resources': {'NET_BW_EGR_KILOBIT_PER_SEC': 1000, 'NET_BW_IGR_KILOBIT_PER_SEC': 1000}}], 'same_subtree': ['35e2b68f-9d87-54ff-a7c1-c0772986407e']} | | |
| 19:56:35 | sean-k-mooney | in the neutron port | |
| 19:57:00 | sean-k-mooney | so the neutron port appears to have the port resouce reqquest in it | |
| 19:57:45 | sean-k-mooney | its also in allocations we fore we delete them | |
| 19:58:17 | sean-k-mooney | melwitt: but its not after we heal the allcoations | |
| 19:58:58 | sean-k-mooney | https://paste.opendev.org/show/bltzmAmf8ZoEE2NEimRR/ | |
| 20:00:31 | sean-k-mooney | melwitt: i need to finish for today but i wonder if there is another regression here | |
| 20:01:21 | melwitt | sean-k-mooney: hm, ok. thanks for looking o/ | |
| 20:06:42 | sean-k-mooney | https://github.com/openstack/nova/blob/master/nova/cmd/manage.py#L1749 | |
| 20:06:50 | sean-k-mooney | i would expec that to print somewhere firht | |
| 20:07:57 | sean-k-mooney | or this https://github.com/openstack/nova/blob/master/nova/cmd/manage.py#L1760C1-L1785C62 | |
| 20:16:05 | sean-k-mooney | ok im actully going to leave now but this is the neutron clien twe are using in nova_manage | |
| 20:16:07 | sean-k-mooney | https://github.com/openstack/nova/blob/master/nova/network/neutron.py#L248 | |
| 20:16:18 | sean-k-mooney | we are creating it with admin=true | |
| 20:16:52 | sean-k-mooney | and this is where the service auth token shoudl be enabled | |
| 20:16:54 | sean-k-mooney | https://github.com/openstack/nova/blob/master/nova/network/neutron.py#L219-L233 | |
| 20:17:49 | sean-k-mooney | so it shoudl have an admin client with service tokens | |
| 20:18:04 | sean-k-mooney | but the nova code seams to not be healign the port so we should see why | |
| 20:20:00 | melwitt | sean-k-mooney: thanks for the guidance | |
| 21:55:59 | melwitt | looks like nova-ceph-multistore is at a 100% fail too with "Details: b'400 Bad Request\n\nThe Store URI was malformed.\n\n '" | |
| 21:58:56 | melwitt | 😩 | |
| 22:58:19 | opendevreview | melanie witt proposed openstack/nova-specs master: Update ephemeral encryption specs to reflect implementation https://review.opendev.org/c/openstack/nova-specs/+/907654 | |
| #openstack-nova - 2024-02-23 | |||
| 01:50:55 | opendevreview | melanie witt proposed openstack/nova master: Report ephemeral disk encryption in the metadata API https://review.opendev.org/c/openstack/nova/+/909945 | |
| 02:26:03 | opendevreview | melanie witt proposed openstack/nova master: Deprecate legacy ephemeral storage encryption using dm-crypt https://review.opendev.org/c/openstack/nova/+/909947 | |
| 02:54:39 | melwitt | looks like nova-ceph-multistore is fixed by the merge of https://review.opendev.org/c/openstack/requirements/+/909923 | |
| 05:31:20 | opendevreview | Merged openstack/nova master: [S-RBAC] adapt nova-next for port's binding:profile field change https://review.opendev.org/c/openstack/nova/+/909859 | |
| 05:32:12 | opendevreview | Maxim Monin proposed openstack/nova master: Server Rescue leads to Server ERROR state if original image is deleted https://review.opendev.org/c/openstack/nova/+/872385 | |
| 05:52:40 | opendevreview | melanie witt proposed openstack/nova master: Deprecate legacy ephemeral storage encryption using dm-crypt https://review.opendev.org/c/openstack/nova/+/909947 | |
| 07:42:06 | opendevreview | Maxim Monin proposed openstack/nova master: Server Rescue leads to Server ERROR state if original image is deleted https://review.opendev.org/c/openstack/nova/+/872385 | |
| 08:03:24 | opendevreview | Dr. Jens Harbott proposed openstack/nova master: DNM: Test glance new location api https://review.opendev.org/c/openstack/nova/+/891207 | |
| 09:36:51 | LarsErikP | I have discovered something I wonder if it's a bug, or if it is intended behavior. In yoga at least, I can unshelve a VM that will cause an overconsumption of the CPU quota.. | |
| 09:38:59 | LarsErikP | Created a project with 4 VCPUs and 4GB RAM. Then spawned server1 with 2 cores and 2GB RAM. Shelved it. After that i spawned server2 with 4 cores and 2GB RAM. I have now used 4 of 4 CPUs in the quota. Then I unshelved server1 without any problem, and now I have used 6 of 4 CPUs... | |
| 09:39:34 | LarsErikP | I.. did not think this should be possible, as it enables users to easily overconsume their quota.. | |
| 09:48:42 | LarsErikP | you can do the same trick with overconsuming RAM as well | |
| 09:51:06 | LarsErikP | and, fwiw, both at once is also possible | |
| 10:33:34 | bauzas | LarsErikP: most of the community today is on PTO | |
| 10:34:04 | bauzas | are you sure that your limits were for 4 VCPUs ? | |
| 10:48:34 | LarsErikP | yes | |
| 10:49:52 | LarsErikP | I filed a bug anyways :P https://bugs.launchpad.net/nova/+bug/2054797 | |
| 10:52:56 | LarsErikP | and it kind of scales.. meaning if I create a bunch of VMs, shelve all of them, create a new one consuming the entire quota; I can stil unshelve all the shelved VMs | |
| 11:06:42 | opendevreview | sean mooney proposed openstack/nova master: add pyproject.toml to support pip 23.1 https://review.opendev.org/c/openstack/nova/+/899753 | |
| 11:20:49 | opendevreview | Konrad Gube proposed openstack/nova master: Use Cinder's os-extend_volume_completion volume action. https://review.opendev.org/c/openstack/nova/+/873560 | |
| 11:55:26 | opendevreview | sean mooney proposed openstack/nova master: Revert "[S-RBAC] adapt nova-next for port's binding:profile field change" https://review.opendev.org/c/openstack/nova/+/909834 | |
| 12:47:52 | kgube | bauzas, gibi: Hi, could you have another look at this? https://review.opendev.org/c/openstack/nova/+/873560 | |
| 13:26:19 | opendevreview | Merged openstack/nova master: Follow up patch to correct a minor documentation issue https://review.opendev.org/c/openstack/nova/+/908863 | |
| 14:21:07 | opendevreview | Merged openstack/os-vif master: Fix missing or unnecessary dependencies https://review.opendev.org/c/openstack/os-vif/+/909341 | |
| 14:49:13 | u00599 | Hey, we came across a problem in our openstack environment where the metadata-service is not marking interfaces with dhcp_enabled for network config with cloud-init. We came across the following commit which wanted to solve this problem, but this doesn't work for ovn with native dhcp (no agents) as there are no dhcp-agents anymore. The concerned networks don't have a default gateway set, so the "fix" is not working for us. Anybody came | |
| 14:49:22 | u00599 | https://github.com/openstack/nova/commit/c7f572d65b57e034c1391ed49d84f6e5f1d672ad | |
| 15:17:09 | artom | So it's hw:emulator_threads_policy with the s but hw:cpu_policy without the s | |
| 15:17:15 | artom | We're just evil, aren't we | |
| 15:40:01 | opendevreview | Merged openstack/nova master: check both source and dest compute libvirt versions for mdev lv https://review.opendev.org/c/openstack/nova/+/904176 | |
| 15:40:32 | jamespage | LarsErikP: https://bugs.launchpad.net/cloud-archive/+bug/2054687 to track | |
| 15:46:03 | opendevreview | Merged openstack/nova master: Check if destination can support the src mdev types https://review.opendev.org/c/openstack/nova/+/904177 | |
| 15:46:10 | opendevreview | Merged openstack/nova master: Reserve mdevs to return to the source https://review.opendev.org/c/openstack/nova/+/904209 | |
| 15:52:24 | sean-k-mooney | u00599: ovn has its ovn metadata agent | |
| 15:53:01 | sean-k-mooney | u00599: but unless they have fixed it neutron is not waiting for it to be ready before seneding the network-vif-plugged event | |
| 15:54:12 | sean-k-mooney | so there is or was a race when using ovn | |
| 16:00:16 | sean-k-mooney | LarsErikP: this is related to counting limits in placment | |
| 16:06:21 | sean-k-mooney | https://docs.openstack.org/nova/latest/configuration/config.html#quota.count_usage_from_placement | |
| 16:06:47 | sean-k-mooney | Behavior will be different for servers in SHELVED_OFFLOADED state. A server in SHELVED_OFFLOADED state will not have placement allocations, so it will not consume quota usage for cores and ram. Note that because of this, it will be possible for a request to unshelve a server to be rejected if the user does not have enough quota available to support the cores and ram needed by | |
| 16:06:49 | sean-k-mooney | the server to be unshelved. | |
| 16:16:22 | sean-k-mooney | LarsErikP: https://review.opendev.org/c/openstack/nova/+/888096 is the yoga fix but yoga is now unmaintained https://opendev.org/openstack/nova/src/branch/unmaintained/yoga | |
| 16:17:25 | sean-k-mooney | hum although im not sure that is corect since wallay and victoria should be unmainted then too | |
| 16:17:34 | sean-k-mooney | and xena | |
| 16:17:54 | sean-k-mooney | elodilles: ^ we should look into what up there | |
| 16:18:09 | sean-k-mooney | https://opendev.org/openstack/nova/branches | |
| 16:18:42 | sean-k-mooney | i forget what we had agreed but i taught we agreed not to skip any releases | |
| 16:18:57 | sean-k-mooney | so if yoga is moved to unamined i woudl expect all older branch to be as well | |
| 16:29:31 | elodilles | sean-k-mooney: we started the transition with yoga, as that had a 'deadline' some months ago... now we'll continue with the rest (xena, wallaby and victoria), actually, i just said on the Release meeting that i'm planning to propose the transition patches on Monday (despite the busy Caracal-3 milestone busy time... :S) | |
| 16:30:20 | sean-k-mooney | elodilles: oh ok | |
| 16:30:25 | sean-k-mooney | so they are just in flight | |
| 16:30:27 | sean-k-mooney | cool | |
| 16:31:45 | elodilles | yepp-yepp | |
| 16:31:49 | elodilles | that's the plan | |
| 16:52:10 | artom | Bleargh, the CPU power management stuff is not testable for multiple compute hosts in functional tests | |
| 16:52:39 | artom | All the work is being done by module-level functions, no way to mock/patch different instances on the two compute services/libvirt drivers | |
| 16:52:50 | artom | IOW, live migration reproducer tests are impossible | |
| 16:53:55 | artom | And I feel like if I make the CPU API a class and stick in an attribute of the driver, it won't fly, because it's a change that would only exist to allow tests | |
| 16:54:26 | sean-k-mooney | artom: we could test it with multiple compute nodes in the functional tests | |
| 16:54:37 | sean-k-mooney | but we are missing test infra to do that | |
| 16:54:59 | artom | sean-k-mooney, I don't see a way to the assert the online/offline CPUs on each host | |
| 16:55:33 | artom | Because the libvirt driver imports cpu/api.py as a global libvirt_cpu, and calls that | |
| 17:03:12 | artom | Actually, what I'd really need is to somehow stub in different "instances" of the Core class for each libvirt driver | |
| 17:04:36 | artom | More like, a different SysFS fixture for each... | |
| 17:17:49 | sean-k-mooney | artom: we made live migration work in teh functrional tests with sriov | |
| 17:18:01 | sean-k-mooney | we can make it work for power management | |
| 17:18:08 | sean-k-mooney | if we put in the effort | |
| 17:18:32 | artom | Right, I'm just not sure it's possible without changing the actual Nova code | |
| 17:21:18 | sean-k-mooney | anywya im ment to be off today so im going to leave the office again | |
| 17:22:11 | artom | Yeah, I was going to say, make sure you compensate for working today by not working Monday or something :) | |
| 17:47:05 | elodilles | sean-k-mooney LarsErikP : i've cherry picked the patch to unmaintained/yoga: https://review.opendev.org/c/openstack/nova/+/910010 | |
| 17:47:59 | elodilles | gate is happy with it, though there are no devstack based jobs among the zuul jobs at all, so that should be added to the branch first | |
| 18:14:54 | opendevreview | Artom Lifshitz proposed openstack/nova master: Power on cores for isolated emulator threads https://review.opendev.org/c/openstack/nova/+/909795 | |