| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2023-02-16 | |||
| 16:08:47 | bauzas | ralonsoh: yup, which kind of failures do you want to see ? for the dhcp query? | |
| 16:09:04 | ralonsoh | yes and the kernel panic | |
| 16:20:57 | bauzas | ralonsoh: one for segfaults https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_da2/821228/7/check/nova-multi-cell/da2689f/job-output.txt | |
| 16:23:22 | bauzas | ralonsoh: that one for kernel panicking https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_e4a/821228/7/gate/nova-next/e4ab52f/job-output.txt | |
| 16:24:47 | bauzas | ralonsoh: that one is an interesting case where the default route is already present and metadata goes into weeds https://ae59d1e8526fa7671728-240e4b572b6f89b26c1b0e70b1c00c17.ssl.cf1.rackcdn.com/872413/3/check/nova-multi-cell/5e89e48/job-output.txt | |
| 16:27:11 | ralonsoh | I'll check this last one | |
| 16:28:05 | ralonsoh | bauzas, eh hold on, this could be a problem with the OVN version in Jammy | |
| 16:28:27 | ralonsoh | --> https://review.opendev.org/c/openstack/neutron/+/873684 | |
| 16:28:59 | ralonsoh | there is an issue with ovn v22.03.0, included in jammy | |
| 16:29:05 | ralonsoh | and some missing flows for the metadata | |
| 16:29:17 | ralonsoh | Yatin found it and we are skipping those tests | |
| 16:29:38 | ralonsoh | actually we are going to test using a compiled version of OVN | |
| 16:29:41 | ralonsoh | https://review.opendev.org/c/openstack/neutron/+/874112/1 | |
| 16:30:52 | bauzas | ack ok | |
| 16:51:31 | opendevreview | Alexey Stupnikov proposed openstack/nova stable/ussuri: Test aborting queued live migration https://review.opendev.org/c/openstack/nova/+/873575 | |
| 16:51:32 | opendevreview | Alexey Stupnikov proposed openstack/nova stable/ussuri: Add functional tests to reproduce bug #1960412 https://review.opendev.org/c/openstack/nova/+/873576 | |
| 17:01:18 | opendevreview | Alexey Stupnikov proposed openstack/nova stable/ussuri: Clean up when queued live migration aborted https://review.opendev.org/c/openstack/nova/+/873577 | |
| 17:09:43 | opendevreview | Alexey Stupnikov proposed openstack/nova stable/ussuri: Clean up when queued live migration aborted https://review.opendev.org/c/openstack/nova/+/873577 | |
| 17:25:16 | fungi | ralonsoh: any idea if the fixes have been backported to v22 such that ubuntu could do an sru to patch their packages? | |
| 17:25:57 | ralonsoh | fungi, Yatin opened a bug today: https://bugs.launchpad.net/ubuntu/+source/ovn/+bug/2003056 | |
| 17:26:02 | fungi | awesome | |
| 17:26:12 | ralonsoh | (sorry, not today) | |
| 17:26:31 | fungi | i'd hate to see devstack using a bespoke ovn build long-term | |
| 17:26:42 | fungi | here's hoping they're able to patch it | |
| 17:27:15 | ralonsoh | from https://bugs.launchpad.net/ubuntu/+source/ovn/+bug/2003056/comments/4, there should be a new version now (22.04.1, instead of 22.03) | |
| 17:29:24 | bauzas | ralonsoh: I guess you don't recommend us to work on nova's zuul jobs to build ovs from source? | |
| 17:29:34 | fungi | oh, it didn't dawn on me that those might be date-based versions rather than semver, so yeah here's hoping backporting the fix in ubuntu won't be painful | |
| 17:29:43 | bauzas | s/ovs/ovn | |
| 17:29:54 | ralonsoh | bauzas, no, we should use the OS released version | |
| 17:30:01 | bauzas | cool | |
| 17:30:09 | ralonsoh | we use compiled version in Neutron for testing only | |
| 17:30:18 | bauzas | yup saw the DNM | |
| 17:30:22 | ralonsoh | (and we had problems for this, this is why we use both now) | |
| 17:30:34 | bauzas | but I was wondering how much of this was actionable on our side | |
| 17:30:47 | bauzas | I'm like done rechecking every 2 hours | |
| 17:31:52 | bauzas | any actional progress sounds to a sweet spot :) | |
| 17:31:58 | bauzas | sounds to me* | |
| 17:32:21 | fungi | oh yay, so it ended up in j-p-u yesterday and should be showing up on mirrors at any moment assuming the ubuntu autobuilders aren't clogged | |
| 17:33:14 | fungi | we may need to add that repository temporarily to the sources list in affected jobs, until it migrates into a jammy point release | |
| 17:35:17 | fungi | https://launchpad.net/ubuntu/+source/ovn/22.03.2-0ubuntu0.22.04.1 indicates the binary packages haven't built yet | |
| 17:37:03 | bauzas | I hereby declare Nova on Feature Freeze :) | |
| 17:37:27 | bauzas | (anyway, all the accepted blueprints were reviewed) | |
| 17:37:59 | bauzas | kbye ;) | |
| 17:38:31 | fungi | fnordahl: i'm a little fuzzy on ubuntu's sru flow... is it like proposed-updates in debian where it only makes it into the mainstream indices in periodic point releases and we need to put jammy-proposed in sources.list in the interim? | |
| 17:56:29 | gmann | dansmith: bauzas: ralonsoh: not read all the logs but related to cirros image there is patch up to bump it to version 0.6.1. https://review.opendev.org/c/openstack/devstack/+/859773 | |
| 17:59:10 | ralonsoh | gmann, thanks. We have seen some seg faults and kernel panics during the VM boot, using this image | |
| 17:59:40 | ralonsoh | in https://review.opendev.org/c/openstack/nova/+/873934 | |
| 19:05:01 | spatel | sean-k-mooney i saw your post about my question related ceph disaster | |
| 19:05:39 | spatel | I am trying to do rescue method and stuck here - https://ibb.co/y84BLGY | |
| 19:08:37 | sean-k-mooney | hum ok have you tried un rescuing and seing if it fixed enough for the vm ot recover its self | |
| 19:09:01 | spatel | Let me try now.. | |
| 19:10:31 | spatel | no luck - https://ibb.co/4swXjQ1 | |
| 19:11:19 | spatel | ceph status showing all PGs are clean and active - https://paste.opendev.org/show/bnG8rvXJydADTZknd2QD/ | |
| 19:11:36 | spatel | not sure why i got filesystem corruption. | |
| 19:24:53 | mnaser | is there ci jobs that are testing secure rbac across all services? | |
| 19:25:05 | mnaser | in an all zed env, enabling it for neutron seems to break new vm deployments | |
| 19:25:28 | mnaser | something along these lines: 2023-02-15 22:04:28.241 2935939 ERROR nova.compute.manager [instance: aaac6261-721a-404d-80d3-94cf25bf869b] neutronclient.common.exceptions.PortNotFoundClient: Port 485c1e6e-fdcd-45ca-ae52-c2cdeb95bfdd could not be found. | |
| 19:25:29 | sean-k-mooney | im not really sure what to do. you could try stoping the vm and mounting the volume on the host and running the filesystem recovery form there | |
| 19:25:49 | sean-k-mooney | you might be able to fix the superblock but im not sure if that will work | |
| 19:27:04 | sean-k-mooney | mnaser: i tought there was a devstack job for this yes gmann would know more | |
| 19:27:22 | mnaser | i mean enabling it in nova works fine, so nova is happy, but enabling in neutron makes it un happy | |
| 19:27:39 | mnaser | so it could be a neutron issue so i dont know if the devstack job enables it for all or just for specific services | |
| 19:27:42 | sean-k-mooney | right but i tought we had it enabled for both | |
| 19:28:02 | sean-k-mooney | ya its a good question im not sure either | |
| 19:29:30 | mnaser | https://github.com/openstack/nova/blob/master/.zuul.yaml#L678-L684 | |
| 19:29:31 | mnaser | wonder if its that | |
| 19:33:34 | sean-k-mooney | ya that should have it enabled for those 4 services | |
| 19:33:59 | sean-k-mooney | mnaser: https://zuul.openstack.org/builds?job_name=tempest-integrated-compute-enforce-scope-new-defaults&skip=0 | |
| 19:34:04 | sean-k-mooney | it looks pretty green too | |
| 19:34:50 | sean-k-mooney | welll there is at least some green menaing it should work in general but im not sure how much is covered by that | |
| 19:37:22 | mnaser | sean-k-mooney: i wonder if we are getting hit by this since its not in zed yet - https://github.com/openstack/neutron/commit/6d8ada0ac93beed05b45adb9582c3ef23bef49d2 | |
| 19:37:36 | mnaser | and the test that failed was actually as an admin | |
| 19:38:09 | sean-k-mooney | oh your trying to do this in zed | |
| 19:38:26 | sean-k-mooney | ya ok we only enabled it by defualt this cycle | |
| 19:39:36 | sean-k-mooney | mnaser: im not sure that was planned ot be backported | |
| 19:40:51 | sean-k-mooney | i would ask the neutrnon folk to backport it if you intend to enable it | |
| 19:41:06 | sean-k-mooney | its kind fo feature ish | |
| 19:41:58 | mnaser | sean-k-mooney: yeah i guess one could argue its a bug too | |
| 19:42:37 | sean-k-mooney | its because of the pivort that happend at the yoga fourm/ptg | |
| 19:42:53 | sean-k-mooney | when we deiced to revert a lot of the work and remove the use of scopes form most apis | |
| 19:43:22 | sean-k-mooney | under the orginal plan admin should not be global admin | |
| 19:43:46 | sean-k-mooney | so they adapted to that change in zed | |
| 21:51:09 | gmann | mnaser: yes, you found those. during integration testing in this cycle (tempest-full-enforce-scope-new-defaults), we found few bugs in neutron and they got fixed in master | |
| 21:51:22 | gmann | mnaser: I think slaweq was planning to backport those to stable/zed or older if needed | |
| 21:51:47 | gmann | let me find those patches | |
| 21:55:24 | gmann | basically these three bugs https://bugs.launchpad.net/neutron/+bug/1996150 https://bugs.launchpad.net/neutron/+bug/1996836 https://bugs.launchpad.net/neutron/+bug/1997089 | |
| 21:57:58 | gmann | mnaser: pinged about these in neutron channel. I was in impressions that they wee backported already | |
| 22:06:58 | gmann | mnaser: glad to know it worked fine for nova. did you enable scope and new defaults both or just new defaults ? | |
| 23:13:41 | mnaser | we enabled both gmann ! | |
| 23:17:52 | gmann | ok | |
| #openstack-nova - 2023-02-17 | |||
| 01:45:43 | opendevreview | Merged openstack/nova master: libvirt: Add configuration options to set SPICE compression settings https://review.opendev.org/c/openstack/nova/+/828675 | |
| 03:07:56 | opendevreview | melanie witt proposed openstack/nova master: doc: Add details about the behavior of server delete https://review.opendev.org/c/openstack/nova/+/874188 | |
| 07:10:13 | opendevreview | Amit Uniyal proposed openstack/nova master: Added context manager for instance lock https://review.opendev.org/c/openstack/nova/+/873648 | |
| 07:58:14 | opendevreview | Amit Uniyal proposed openstack/nova master: Added context manager for instance lock https://review.opendev.org/c/openstack/nova/+/873648 | |
| 08:32:25 | bauzas | brightning new day, and brighting new CI failure \o/ | |
| 08:46:50 | opendevreview | Amit Uniyal proposed openstack/nova master: Remove "see nova-manage.log" str from console msg https://review.opendev.org/c/openstack/nova/+/874206 | |
| 09:02:42 | opendevreview | Nobuhiro MIKI proposed openstack/nova master: libvirt: Add 'COMPUTE_ADDRESS_SPACE_*' traits support https://review.opendev.org/c/openstack/nova/+/873221 | |
| 10:18:57 | samuelkunkel[m] | Good morning, I have a question regarding AMD SEV. I am running into an issue where the the first check of hardware.get_mem_encryption_constraint is working properly (machine type correct, uefi boot check etc). further down the line nova runs libvirt_utils.get_flags_by_flavor_specs. In this function we try to fetch the ResourceRequest via scheduler_utils. At this state we only have the information about the flavor. Not the image. | |
| 10:18:57 | samuelkunkel[m] | During the the execution of this nova will run res_req._translate_memory_encryption(request_spec.flavor, image) where the image is a just a plain objects.ImageMeta(properties=objects.ImageMetaProps())... and now nova fails (its running the function now for the second time) _check_mem_encryption_uses_uefi_image as the image does not contain useful information at all and therefore we never can create an instance as nova assumes in | |
| 10:18:57 | samuelkunkel[m] | this stage we use an image without uefi boot. | |