| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2024-02-22 | |||
| 17:12:27 | bauzas | JayF: (sorry, forgot to see your reply) sure, I'll lookup then your patch tomorrow (even if at RH, I'll work on tomorrow morning) | |
| 17:12:52 | JayF | Today and tomorrow :) So look tomorrow but don't expect until Monday | |
| 17:14:11 | opendevreview | Amit Uniyal proposed openstack/nova master: enforce remote console shutdown https://review.opendev.org/c/openstack/nova/+/901824 | |
| 17:16:34 | sean-k-mooney | melwitt: so this dates form when we first added live snapshot https://github.com/uggla/nova/commit/46de2d1e2d0abd6fdcd4da13facaf3225c721f5e | |
| 17:17:52 | melwitt | quite a bit ago :) | |
| 17:18:37 | sean-k-mooney | yep so the simpeler ways of doign this were not "bug-free" in qemu 1.3 | |
| 17:19:07 | sean-k-mooney | but we havng used that in half a decade or more | |
| 17:21:00 | melwitt | yeah, agree it would be good to clean that up | |
| 17:25:13 | sean-k-mooney | based on https://libvirt.org/kbase/domainstatecapture.html we have a few options | |
| 17:27:16 | sean-k-mooney | i would hope we could use the direct backup functionaltiy or similar | |
| 17:28:22 | sean-k-mooney | something like this https://libvirt.org/formatbackup.html#examples | |
| 17:29:51 | melwitt | ah, ok | |
| 17:30:40 | sean-k-mooney | the thing is i dont know if any of that will work with encyption | |
| 17:31:30 | sean-k-mooney | if it does not virDomainSnapshotCreateXML may or we might just need to keep what we are doing | |
| 17:31:46 | sean-k-mooney | but it wooul dbe a good conversation to have with the virt folks about | |
| 17:32:04 | sean-k-mooney | "if we were ot do it from scratch today what would you recommend" | |
| 17:32:38 | melwitt | yeah. I think they mentioned the real snapshot API would be the better way to do it (with encryption) | |
| 17:33:48 | sean-k-mooney | this one https://libvirt.org/html/libvirt-libvirt-domain-snapshot.html#virDomainSnapshotCreateXML | |
| 17:34:02 | sean-k-mooney | ya qemu can aslo do memory snapshots and other fancy thigns via that | |
| 17:34:37 | sean-k-mooney | we can pass flags=VIR_DOMAIN_SNAPSHOT_CREATE_LIVE|VIR_DOMAIN_SNAPSHOT_CREATE_DISK_ONLY | |
| 17:35:22 | sean-k-mooney | and there is VIR_DOMAIN_SNAPSHOT_CREATE_REUSE_EXT | |
| 17:35:33 | sean-k-mooney | which alluse you ot resue a precreate file i think | |
| 17:36:05 | melwitt | yeah that's right | |
| 17:39:03 | sean-k-mooney | anway that a long way to say im +1 on the bockcopy usage and we may be able to simplfy later | |
| 17:40:35 | melwitt | ack ++ :) | |
| 19:22:07 | opendevreview | Dan Smith proposed openstack/nova master: Catch ImageNotFound on snapshot failure https://review.opendev.org/c/openstack/nova/+/905316 | |
| 19:22:08 | opendevreview | Dan Smith proposed openstack/nova master: Support glance's new location API https://review.opendev.org/c/openstack/nova/+/891036 | |
| 19:22:08 | opendevreview | Dan Smith proposed openstack/nova master: DNM: Test glance new location api https://review.opendev.org/c/openstack/nova/+/891207 | |
| 19:25:20 | melwitt | sean-k-mooney: nova-next is still unhappy :( https://zuul.opendev.org/t/openstack/build/3c2c7955a4b34112a377a857623d6a73/log/job-output.txt#37662 | |
| 19:26:36 | melwitt | guess we can't check for healed port allocations | |
| 19:32:05 | sean-k-mooney | really | |
| 19:33:01 | sean-k-mooney | or did i forget to delete on eof the checks | |
| 19:36:03 | sean-k-mooney | so tha tot me say that we have not configure bandwith qos properly | |
| 19:36:43 | sean-k-mooney | am i need ot go have somethign to eat but ok ill just drop all the port stuff | |
| 19:36:59 | sean-k-mooney | and if that does not work we can drop all the heal code | |
| 19:39:44 | mnaser | https://bugs.launchpad.net/nova/+bug/2052915 is affecting nova right now but i have no idea where to look at the refernece | |
| 19:40:36 | sean-k-mooney | that sounds like neutron broke upgrades | |
| 19:40:36 | melwitt | sean-k-mooney: I'm not sure .. it's failing on "bandwidth_allocations=$(echo "$allocations" | grep NET_BW_EGR_KILOBIT_PER_SEC)" | |
| 19:40:59 | sean-k-mooney | yep so if bandwith qos was working that allocation should exist | |
| 19:41:50 | melwitt | the weird thing is that it's supposed to "echo "Failed to heal port allocations."" if the above returned "" but it's not echoing that | |
| 19:42:22 | sean-k-mooney | yep we have allcoations but not for that type | |
| 19:42:32 | melwitt | oh, I see. ok | |
| 19:42:58 | sean-k-mooney | so what i ment by not configre correctly is i think we may not have enabel the qos extionion in the job | |
| 19:43:08 | sean-k-mooney | or somethign liek that | |
| 19:43:37 | opendevreview | sean mooney proposed openstack/nova master: [S-RBAC] adapt nova-next for port's binding:profile field change https://review.opendev.org/c/openstack/nova/+/909859 | |
| 19:44:24 | melwitt | doesn't the fact that it used to work before the service token change mean the extension is present? | |
| 19:44:52 | sean-k-mooney | i would have to read the code but not nessiarlay | |
| 19:45:19 | melwitt | ok | |
| 19:45:23 | sean-k-mooney | the allocation is create by nova but if we dont have teh qos extion is not loaded | |
| 19:45:43 | sean-k-mooney | what will happen is neutron will not make a placement resouce request in teh port | |
| 19:46:32 | sean-k-mooney | melwitt: the hook was not asserting the bandwith resouce request existed before we healed | |
| 19:46:54 | sean-k-mooney | or that the the port has port resouce requests | |
| 19:47:57 | melwitt | 😵💫 | |
| 19:55:18 | sean-k-mooney | it does look like the qos plugin was loaded https://7afe9236b7eaaf244ae9-d98f699786987cf0f7a232f67c9b09f7.ssl.cf2.rackcdn.com/909859/2/check/nova-next/3c2c795/controller/logs/etc/neutron/neutron_conf.txt | |
| 19:56:26 | sean-k-mooney | and we see | |
| 19:56:28 | sean-k-mooney | 2024-02-22 18:53:27.154019 | controller | | resource_request | {'request_groups': [{'id': '35e2b68f-9d87-54ff-a7c1-c0772986407e', 'required': ['CUSTOM_PHYSNET_PUBLIC', 'CUSTOM_VNIC_TYPE_NORMAL'], 'resources': {'NET_BW_EGR_KILOBIT_PER_SEC': 1000, 'NET_BW_IGR_KILOBIT_PER_SEC': 1000}}], 'same_subtree': ['35e2b68f-9d87-54ff-a7c1-c0772986407e']} | | |
| 19:56:35 | sean-k-mooney | in the neutron port | |
| 19:57:00 | sean-k-mooney | so the neutron port appears to have the port resouce reqquest in it | |
| 19:57:45 | sean-k-mooney | its also in allocations we fore we delete them | |
| 19:58:17 | sean-k-mooney | melwitt: but its not after we heal the allcoations | |
| 19:58:58 | sean-k-mooney | https://paste.opendev.org/show/bltzmAmf8ZoEE2NEimRR/ | |
| 20:00:31 | sean-k-mooney | melwitt: i need to finish for today but i wonder if there is another regression here | |
| 20:01:21 | melwitt | sean-k-mooney: hm, ok. thanks for looking o/ | |
| 20:06:42 | sean-k-mooney | https://github.com/openstack/nova/blob/master/nova/cmd/manage.py#L1749 | |
| 20:06:50 | sean-k-mooney | i would expec that to print somewhere firht | |
| 20:07:57 | sean-k-mooney | or this https://github.com/openstack/nova/blob/master/nova/cmd/manage.py#L1760C1-L1785C62 | |
| 20:16:05 | sean-k-mooney | ok im actully going to leave now but this is the neutron clien twe are using in nova_manage | |
| 20:16:07 | sean-k-mooney | https://github.com/openstack/nova/blob/master/nova/network/neutron.py#L248 | |
| 20:16:18 | sean-k-mooney | we are creating it with admin=true | |
| 20:16:52 | sean-k-mooney | and this is where the service auth token shoudl be enabled | |
| 20:16:54 | sean-k-mooney | https://github.com/openstack/nova/blob/master/nova/network/neutron.py#L219-L233 | |
| 20:17:49 | sean-k-mooney | so it shoudl have an admin client with service tokens | |
| 20:18:04 | sean-k-mooney | but the nova code seams to not be healign the port so we should see why | |
| 20:20:00 | melwitt | sean-k-mooney: thanks for the guidance | |
| 21:55:59 | melwitt | looks like nova-ceph-multistore is at a 100% fail too with "Details: b'400 Bad Request\n\nThe Store URI was malformed.\n\n '" | |
| 21:58:56 | melwitt | 😩 | |
| 22:58:19 | opendevreview | melanie witt proposed openstack/nova-specs master: Update ephemeral encryption specs to reflect implementation https://review.opendev.org/c/openstack/nova-specs/+/907654 | |
| #openstack-nova - 2024-02-23 | |||
| 01:50:55 | opendevreview | melanie witt proposed openstack/nova master: Report ephemeral disk encryption in the metadata API https://review.opendev.org/c/openstack/nova/+/909945 | |
| 02:26:03 | opendevreview | melanie witt proposed openstack/nova master: Deprecate legacy ephemeral storage encryption using dm-crypt https://review.opendev.org/c/openstack/nova/+/909947 | |
| 02:54:39 | melwitt | looks like nova-ceph-multistore is fixed by the merge of https://review.opendev.org/c/openstack/requirements/+/909923 | |
| 05:31:20 | opendevreview | Merged openstack/nova master: [S-RBAC] adapt nova-next for port's binding:profile field change https://review.opendev.org/c/openstack/nova/+/909859 | |
| 05:32:12 | opendevreview | Maxim Monin proposed openstack/nova master: Server Rescue leads to Server ERROR state if original image is deleted https://review.opendev.org/c/openstack/nova/+/872385 | |
| 05:52:40 | opendevreview | melanie witt proposed openstack/nova master: Deprecate legacy ephemeral storage encryption using dm-crypt https://review.opendev.org/c/openstack/nova/+/909947 | |
| 07:42:06 | opendevreview | Maxim Monin proposed openstack/nova master: Server Rescue leads to Server ERROR state if original image is deleted https://review.opendev.org/c/openstack/nova/+/872385 | |
| 08:03:24 | opendevreview | Dr. Jens Harbott proposed openstack/nova master: DNM: Test glance new location api https://review.opendev.org/c/openstack/nova/+/891207 | |
| 09:36:51 | LarsErikP | I have discovered something I wonder if it's a bug, or if it is intended behavior. In yoga at least, I can unshelve a VM that will cause an overconsumption of the CPU quota.. | |
| 09:38:59 | LarsErikP | Created a project with 4 VCPUs and 4GB RAM. Then spawned server1 with 2 cores and 2GB RAM. Shelved it. After that i spawned server2 with 4 cores and 2GB RAM. I have now used 4 of 4 CPUs in the quota. Then I unshelved server1 without any problem, and now I have used 6 of 4 CPUs... | |
| 09:39:34 | LarsErikP | I.. did not think this should be possible, as it enables users to easily overconsume their quota.. | |
| 09:48:42 | LarsErikP | you can do the same trick with overconsuming RAM as well | |
| 09:51:06 | LarsErikP | and, fwiw, both at once is also possible | |
| 10:33:34 | bauzas | LarsErikP: most of the community today is on PTO | |
| 10:34:04 | bauzas | are you sure that your limits were for 4 VCPUs ? | |
| 10:48:34 | LarsErikP | yes | |
| 10:49:52 | LarsErikP | I filed a bug anyways :P https://bugs.launchpad.net/nova/+bug/2054797 | |
| 10:52:56 | LarsErikP | and it kind of scales.. meaning if I create a bunch of VMs, shelve all of them, create a new one consuming the entire quota; I can stil unshelve all the shelved VMs | |
| 11:06:42 | opendevreview | sean mooney proposed openstack/nova master: add pyproject.toml to support pip 23.1 https://review.opendev.org/c/openstack/nova/+/899753 | |
| 11:20:49 | opendevreview | Konrad Gube proposed openstack/nova master: Use Cinder's os-extend_volume_completion volume action. https://review.opendev.org/c/openstack/nova/+/873560 | |
| 11:55:26 | opendevreview | sean mooney proposed openstack/nova master: Revert "[S-RBAC] adapt nova-next for port's binding:profile field change" https://review.opendev.org/c/openstack/nova/+/909834 | |
| 12:47:52 | kgube | bauzas, gibi: Hi, could you have another look at this? https://review.opendev.org/c/openstack/nova/+/873560 | |
| 13:26:19 | opendevreview | Merged openstack/nova master: Follow up patch to correct a minor documentation issue https://review.opendev.org/c/openstack/nova/+/908863 | |