| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2023-02-16 | |||
| 16:29:05 | ralonsoh | and some missing flows for the metadata | |
| 16:29:17 | ralonsoh | Yatin found it and we are skipping those tests | |
| 16:29:38 | ralonsoh | actually we are going to test using a compiled version of OVN | |
| 16:29:41 | ralonsoh | https://review.opendev.org/c/openstack/neutron/+/874112/1 | |
| 16:30:52 | bauzas | ack ok | |
| 16:51:31 | opendevreview | Alexey Stupnikov proposed openstack/nova stable/ussuri: Test aborting queued live migration https://review.opendev.org/c/openstack/nova/+/873575 | |
| 16:51:32 | opendevreview | Alexey Stupnikov proposed openstack/nova stable/ussuri: Add functional tests to reproduce bug #1960412 https://review.opendev.org/c/openstack/nova/+/873576 | |
| 17:01:18 | opendevreview | Alexey Stupnikov proposed openstack/nova stable/ussuri: Clean up when queued live migration aborted https://review.opendev.org/c/openstack/nova/+/873577 | |
| 17:09:43 | opendevreview | Alexey Stupnikov proposed openstack/nova stable/ussuri: Clean up when queued live migration aborted https://review.opendev.org/c/openstack/nova/+/873577 | |
| 17:25:16 | fungi | ralonsoh: any idea if the fixes have been backported to v22 such that ubuntu could do an sru to patch their packages? | |
| 17:25:57 | ralonsoh | fungi, Yatin opened a bug today: https://bugs.launchpad.net/ubuntu/+source/ovn/+bug/2003056 | |
| 17:26:02 | fungi | awesome | |
| 17:26:12 | ralonsoh | (sorry, not today) | |
| 17:26:31 | fungi | i'd hate to see devstack using a bespoke ovn build long-term | |
| 17:26:42 | fungi | here's hoping they're able to patch it | |
| 17:27:15 | ralonsoh | from https://bugs.launchpad.net/ubuntu/+source/ovn/+bug/2003056/comments/4, there should be a new version now (22.04.1, instead of 22.03) | |
| 17:29:24 | bauzas | ralonsoh: I guess you don't recommend us to work on nova's zuul jobs to build ovs from source? | |
| 17:29:34 | fungi | oh, it didn't dawn on me that those might be date-based versions rather than semver, so yeah here's hoping backporting the fix in ubuntu won't be painful | |
| 17:29:43 | bauzas | s/ovs/ovn | |
| 17:29:54 | ralonsoh | bauzas, no, we should use the OS released version | |
| 17:30:01 | bauzas | cool | |
| 17:30:09 | ralonsoh | we use compiled version in Neutron for testing only | |
| 17:30:18 | bauzas | yup saw the DNM | |
| 17:30:22 | ralonsoh | (and we had problems for this, this is why we use both now) | |
| 17:30:34 | bauzas | but I was wondering how much of this was actionable on our side | |
| 17:30:47 | bauzas | I'm like done rechecking every 2 hours | |
| 17:31:52 | bauzas | any actional progress sounds to a sweet spot :) | |
| 17:31:58 | bauzas | sounds to me* | |
| 17:32:21 | fungi | oh yay, so it ended up in j-p-u yesterday and should be showing up on mirrors at any moment assuming the ubuntu autobuilders aren't clogged | |
| 17:33:14 | fungi | we may need to add that repository temporarily to the sources list in affected jobs, until it migrates into a jammy point release | |
| 17:35:17 | fungi | https://launchpad.net/ubuntu/+source/ovn/22.03.2-0ubuntu0.22.04.1 indicates the binary packages haven't built yet | |
| 17:37:03 | bauzas | I hereby declare Nova on Feature Freeze :) | |
| 17:37:27 | bauzas | (anyway, all the accepted blueprints were reviewed) | |
| 17:37:59 | bauzas | kbye ;) | |
| 17:38:31 | fungi | fnordahl: i'm a little fuzzy on ubuntu's sru flow... is it like proposed-updates in debian where it only makes it into the mainstream indices in periodic point releases and we need to put jammy-proposed in sources.list in the interim? | |
| 17:56:29 | gmann | dansmith: bauzas: ralonsoh: not read all the logs but related to cirros image there is patch up to bump it to version 0.6.1. https://review.opendev.org/c/openstack/devstack/+/859773 | |
| 17:59:10 | ralonsoh | gmann, thanks. We have seen some seg faults and kernel panics during the VM boot, using this image | |
| 17:59:40 | ralonsoh | in https://review.opendev.org/c/openstack/nova/+/873934 | |
| 19:05:01 | spatel | sean-k-mooney i saw your post about my question related ceph disaster | |
| 19:05:39 | spatel | I am trying to do rescue method and stuck here - https://ibb.co/y84BLGY | |
| 19:08:37 | sean-k-mooney | hum ok have you tried un rescuing and seing if it fixed enough for the vm ot recover its self | |
| 19:09:01 | spatel | Let me try now.. | |
| 19:10:31 | spatel | no luck - https://ibb.co/4swXjQ1 | |
| 19:11:19 | spatel | ceph status showing all PGs are clean and active - https://paste.opendev.org/show/bnG8rvXJydADTZknd2QD/ | |
| 19:11:36 | spatel | not sure why i got filesystem corruption. | |
| 19:24:53 | mnaser | is there ci jobs that are testing secure rbac across all services? | |
| 19:25:05 | mnaser | in an all zed env, enabling it for neutron seems to break new vm deployments | |
| 19:25:28 | mnaser | something along these lines: 2023-02-15 22:04:28.241 2935939 ERROR nova.compute.manager [instance: aaac6261-721a-404d-80d3-94cf25bf869b] neutronclient.common.exceptions.PortNotFoundClient: Port 485c1e6e-fdcd-45ca-ae52-c2cdeb95bfdd could not be found. | |
| 19:25:29 | sean-k-mooney | im not really sure what to do. you could try stoping the vm and mounting the volume on the host and running the filesystem recovery form there | |
| 19:25:49 | sean-k-mooney | you might be able to fix the superblock but im not sure if that will work | |
| 19:27:04 | sean-k-mooney | mnaser: i tought there was a devstack job for this yes gmann would know more | |
| 19:27:22 | mnaser | i mean enabling it in nova works fine, so nova is happy, but enabling in neutron makes it un happy | |
| 19:27:39 | mnaser | so it could be a neutron issue so i dont know if the devstack job enables it for all or just for specific services | |
| 19:27:42 | sean-k-mooney | right but i tought we had it enabled for both | |
| 19:28:02 | sean-k-mooney | ya its a good question im not sure either | |
| 19:29:30 | mnaser | https://github.com/openstack/nova/blob/master/.zuul.yaml#L678-L684 | |
| 19:29:31 | mnaser | wonder if its that | |
| 19:33:34 | sean-k-mooney | ya that should have it enabled for those 4 services | |
| 19:33:59 | sean-k-mooney | mnaser: https://zuul.openstack.org/builds?job_name=tempest-integrated-compute-enforce-scope-new-defaults&skip=0 | |
| 19:34:04 | sean-k-mooney | it looks pretty green too | |
| 19:34:50 | sean-k-mooney | welll there is at least some green menaing it should work in general but im not sure how much is covered by that | |
| 19:37:22 | mnaser | sean-k-mooney: i wonder if we are getting hit by this since its not in zed yet - https://github.com/openstack/neutron/commit/6d8ada0ac93beed05b45adb9582c3ef23bef49d2 | |
| 19:37:36 | mnaser | and the test that failed was actually as an admin | |
| 19:38:09 | sean-k-mooney | oh your trying to do this in zed | |
| 19:38:26 | sean-k-mooney | ya ok we only enabled it by defualt this cycle | |
| 19:39:36 | sean-k-mooney | mnaser: im not sure that was planned ot be backported | |
| 19:40:51 | sean-k-mooney | i would ask the neutrnon folk to backport it if you intend to enable it | |
| 19:41:06 | sean-k-mooney | its kind fo feature ish | |
| 19:41:58 | mnaser | sean-k-mooney: yeah i guess one could argue its a bug too | |
| 19:42:37 | sean-k-mooney | its because of the pivort that happend at the yoga fourm/ptg | |
| 19:42:53 | sean-k-mooney | when we deiced to revert a lot of the work and remove the use of scopes form most apis | |
| 19:43:22 | sean-k-mooney | under the orginal plan admin should not be global admin | |
| 19:43:46 | sean-k-mooney | so they adapted to that change in zed | |
| 21:51:09 | gmann | mnaser: yes, you found those. during integration testing in this cycle (tempest-full-enforce-scope-new-defaults), we found few bugs in neutron and they got fixed in master | |
| 21:51:22 | gmann | mnaser: I think slaweq was planning to backport those to stable/zed or older if needed | |
| 21:51:47 | gmann | let me find those patches | |
| 21:55:24 | gmann | basically these three bugs https://bugs.launchpad.net/neutron/+bug/1996150 https://bugs.launchpad.net/neutron/+bug/1996836 https://bugs.launchpad.net/neutron/+bug/1997089 | |
| 21:57:58 | gmann | mnaser: pinged about these in neutron channel. I was in impressions that they wee backported already | |
| 22:06:58 | gmann | mnaser: glad to know it worked fine for nova. did you enable scope and new defaults both or just new defaults ? | |
| 23:13:41 | mnaser | we enabled both gmann ! | |
| 23:17:52 | gmann | ok | |
| #openstack-nova - 2023-02-17 | |||
| 01:45:43 | opendevreview | Merged openstack/nova master: libvirt: Add configuration options to set SPICE compression settings https://review.opendev.org/c/openstack/nova/+/828675 | |
| 03:07:56 | opendevreview | melanie witt proposed openstack/nova master: doc: Add details about the behavior of server delete https://review.opendev.org/c/openstack/nova/+/874188 | |
| 07:10:13 | opendevreview | Amit Uniyal proposed openstack/nova master: Added context manager for instance lock https://review.opendev.org/c/openstack/nova/+/873648 | |
| 07:58:14 | opendevreview | Amit Uniyal proposed openstack/nova master: Added context manager for instance lock https://review.opendev.org/c/openstack/nova/+/873648 | |
| 08:32:25 | bauzas | brightning new day, and brighting new CI failure \o/ | |
| 08:46:50 | opendevreview | Amit Uniyal proposed openstack/nova master: Remove "see nova-manage.log" str from console msg https://review.opendev.org/c/openstack/nova/+/874206 | |
| 09:02:42 | opendevreview | Nobuhiro MIKI proposed openstack/nova master: libvirt: Add 'COMPUTE_ADDRESS_SPACE_*' traits support https://review.opendev.org/c/openstack/nova/+/873221 | |
| 10:18:57 | samuelkunkel[m] | Good morning, I have a question regarding AMD SEV. I am running into an issue where the the first check of hardware.get_mem_encryption_constraint is working properly (machine type correct, uefi boot check etc). further down the line nova runs libvirt_utils.get_flags_by_flavor_specs. In this function we try to fetch the ResourceRequest via scheduler_utils. At this state we only have the information about the flavor. Not the image. | |
| 10:18:57 | samuelkunkel[m] | During the the execution of this nova will run res_req._translate_memory_encryption(request_spec.flavor, image) where the image is a just a plain objects.ImageMeta(properties=objects.ImageMetaProps())... and now nova fails (its running the function now for the second time) _check_mem_encryption_uses_uefi_image as the image does not contain useful information at all and therefore we never can create an instance as nova assumes in | |
| 10:18:57 | samuelkunkel[m] | this stage we use an image without uefi boot. | |
| 10:19:06 | samuelkunkel[m] | Am I missing something here? | |
| 10:25:57 | samuelkunkel[m] | If I adjust the conditions in _check_mem_encryption_uses_uefi_image (basically if the image is just plain and does not contain any information at all) I just return (basically mocking that we use uefi here) | |
| 10:26:02 | samuelkunkel[m] | then it works properly | |
| 10:33:31 | bauzas | gibi: I may have spotted some eventlet threading issue https://paste.opendev.org/show/bFU6z82FyLG12FKMJzdN/ | |
| 10:40:57 | gibi | bauzas: ack, I have some backlog work through so I haven't looked at the result from that yet. If you can identify the leaking test based on the new log the you can run that tests multiple times in the same executor by duplicating the test case to see if it helps reproducing locally | |
| 10:41:36 | bauzas | gibi: for the moment, we don't have a lot of issues with it | |
| 10:41:49 | bauzas | gibi: so if I have time, yeah I'll try to reproduce it | |
| 10:42:23 | bauzas | gibi: (it was just for telling it for you ;) ) | |
| 10:42:41 | gibi | thanks | |