| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2023-02-16 | |||
| 16:04:19 | bauzas | ralonsoh: I have a few logs where the guest failed to acquire a lease, sec | |
| 16:04:36 | bauzas | and some where the guest panickjed | |
| 16:06:11 | sean-k-mooney | for what its worht the every increaseign memory requirement for cirrios was one of the reasons i looked at moving us to alpine a few years ago | |
| 16:06:20 | sean-k-mooney | longterm i still think that would be a better approch | |
| 16:06:58 | dansmith | is alpine really going to be smaller than cirros? I mean, that seems odd to me | |
| 16:07:10 | opendevreview | Alexey Stupnikov proposed openstack/nova stable/victoria: Test aborting queued live migration https://review.opendev.org/c/openstack/nova/+/845748 | |
| 16:07:11 | opendevreview | Alexey Stupnikov proposed openstack/nova stable/victoria: Add functional tests to reproduce bug #1960412 https://review.opendev.org/c/openstack/nova/+/845753 | |
| 16:07:22 | dansmith | so I think we get the automatic console stuff when we use specific waiters for servers to be available | |
| 16:07:28 | dansmith | so that test must use a different one | |
| 16:07:52 | bauzas | oh damn, internal meeting | |
| 16:08:13 | ralonsoh | bauzas, do you have some links? in any case, this is not Nova nor Neutron fault, I think | |
| 16:08:14 | opendevreview | Alexey Stupnikov proposed openstack/nova stable/victoria: Clean up when queued live migration aborted https://review.opendev.org/c/openstack/nova/+/845754 | |
| 16:08:16 | ralonsoh | at least in this case | |
| 16:08:47 | bauzas | ralonsoh: yup, which kind of failures do you want to see ? for the dhcp query? | |
| 16:09:04 | ralonsoh | yes and the kernel panic | |
| 16:20:57 | bauzas | ralonsoh: one for segfaults https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_da2/821228/7/check/nova-multi-cell/da2689f/job-output.txt | |
| 16:23:22 | bauzas | ralonsoh: that one for kernel panicking https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_e4a/821228/7/gate/nova-next/e4ab52f/job-output.txt | |
| 16:24:47 | bauzas | ralonsoh: that one is an interesting case where the default route is already present and metadata goes into weeds https://ae59d1e8526fa7671728-240e4b572b6f89b26c1b0e70b1c00c17.ssl.cf1.rackcdn.com/872413/3/check/nova-multi-cell/5e89e48/job-output.txt | |
| 16:27:11 | ralonsoh | I'll check this last one | |
| 16:28:05 | ralonsoh | bauzas, eh hold on, this could be a problem with the OVN version in Jammy | |
| 16:28:27 | ralonsoh | --> https://review.opendev.org/c/openstack/neutron/+/873684 | |
| 16:28:59 | ralonsoh | there is an issue with ovn v22.03.0, included in jammy | |
| 16:29:05 | ralonsoh | and some missing flows for the metadata | |
| 16:29:17 | ralonsoh | Yatin found it and we are skipping those tests | |
| 16:29:38 | ralonsoh | actually we are going to test using a compiled version of OVN | |
| 16:29:41 | ralonsoh | https://review.opendev.org/c/openstack/neutron/+/874112/1 | |
| 16:30:52 | bauzas | ack ok | |
| 16:51:31 | opendevreview | Alexey Stupnikov proposed openstack/nova stable/ussuri: Test aborting queued live migration https://review.opendev.org/c/openstack/nova/+/873575 | |
| 16:51:32 | opendevreview | Alexey Stupnikov proposed openstack/nova stable/ussuri: Add functional tests to reproduce bug #1960412 https://review.opendev.org/c/openstack/nova/+/873576 | |
| 17:01:18 | opendevreview | Alexey Stupnikov proposed openstack/nova stable/ussuri: Clean up when queued live migration aborted https://review.opendev.org/c/openstack/nova/+/873577 | |
| 17:09:43 | opendevreview | Alexey Stupnikov proposed openstack/nova stable/ussuri: Clean up when queued live migration aborted https://review.opendev.org/c/openstack/nova/+/873577 | |
| 17:25:16 | fungi | ralonsoh: any idea if the fixes have been backported to v22 such that ubuntu could do an sru to patch their packages? | |
| 17:25:57 | ralonsoh | fungi, Yatin opened a bug today: https://bugs.launchpad.net/ubuntu/+source/ovn/+bug/2003056 | |
| 17:26:02 | fungi | awesome | |
| 17:26:12 | ralonsoh | (sorry, not today) | |
| 17:26:31 | fungi | i'd hate to see devstack using a bespoke ovn build long-term | |
| 17:26:42 | fungi | here's hoping they're able to patch it | |
| 17:27:15 | ralonsoh | from https://bugs.launchpad.net/ubuntu/+source/ovn/+bug/2003056/comments/4, there should be a new version now (22.04.1, instead of 22.03) | |
| 17:29:24 | bauzas | ralonsoh: I guess you don't recommend us to work on nova's zuul jobs to build ovs from source? | |
| 17:29:34 | fungi | oh, it didn't dawn on me that those might be date-based versions rather than semver, so yeah here's hoping backporting the fix in ubuntu won't be painful | |
| 17:29:43 | bauzas | s/ovs/ovn | |
| 17:29:54 | ralonsoh | bauzas, no, we should use the OS released version | |
| 17:30:01 | bauzas | cool | |
| 17:30:09 | ralonsoh | we use compiled version in Neutron for testing only | |
| 17:30:18 | bauzas | yup saw the DNM | |
| 17:30:22 | ralonsoh | (and we had problems for this, this is why we use both now) | |
| 17:30:34 | bauzas | but I was wondering how much of this was actionable on our side | |
| 17:30:47 | bauzas | I'm like done rechecking every 2 hours | |
| 17:31:52 | bauzas | any actional progress sounds to a sweet spot :) | |
| 17:31:58 | bauzas | sounds to me* | |
| 17:32:21 | fungi | oh yay, so it ended up in j-p-u yesterday and should be showing up on mirrors at any moment assuming the ubuntu autobuilders aren't clogged | |
| 17:33:14 | fungi | we may need to add that repository temporarily to the sources list in affected jobs, until it migrates into a jammy point release | |
| 17:35:17 | fungi | https://launchpad.net/ubuntu/+source/ovn/22.03.2-0ubuntu0.22.04.1 indicates the binary packages haven't built yet | |
| 17:37:03 | bauzas | I hereby declare Nova on Feature Freeze :) | |
| 17:37:27 | bauzas | (anyway, all the accepted blueprints were reviewed) | |
| 17:37:59 | bauzas | kbye ;) | |
| 17:38:31 | fungi | fnordahl: i'm a little fuzzy on ubuntu's sru flow... is it like proposed-updates in debian where it only makes it into the mainstream indices in periodic point releases and we need to put jammy-proposed in sources.list in the interim? | |
| 17:56:29 | gmann | dansmith: bauzas: ralonsoh: not read all the logs but related to cirros image there is patch up to bump it to version 0.6.1. https://review.opendev.org/c/openstack/devstack/+/859773 | |
| 17:59:10 | ralonsoh | gmann, thanks. We have seen some seg faults and kernel panics during the VM boot, using this image | |
| 17:59:40 | ralonsoh | in https://review.opendev.org/c/openstack/nova/+/873934 | |
| 19:05:01 | spatel | sean-k-mooney i saw your post about my question related ceph disaster | |
| 19:05:39 | spatel | I am trying to do rescue method and stuck here - https://ibb.co/y84BLGY | |
| 19:08:37 | sean-k-mooney | hum ok have you tried un rescuing and seing if it fixed enough for the vm ot recover its self | |
| 19:09:01 | spatel | Let me try now.. | |
| 19:10:31 | spatel | no luck - https://ibb.co/4swXjQ1 | |
| 19:11:19 | spatel | ceph status showing all PGs are clean and active - https://paste.opendev.org/show/bnG8rvXJydADTZknd2QD/ | |
| 19:11:36 | spatel | not sure why i got filesystem corruption. | |
| 19:24:53 | mnaser | is there ci jobs that are testing secure rbac across all services? | |
| 19:25:05 | mnaser | in an all zed env, enabling it for neutron seems to break new vm deployments | |
| 19:25:28 | mnaser | something along these lines: 2023-02-15 22:04:28.241 2935939 ERROR nova.compute.manager [instance: aaac6261-721a-404d-80d3-94cf25bf869b] neutronclient.common.exceptions.PortNotFoundClient: Port 485c1e6e-fdcd-45ca-ae52-c2cdeb95bfdd could not be found. | |
| 19:25:29 | sean-k-mooney | im not really sure what to do. you could try stoping the vm and mounting the volume on the host and running the filesystem recovery form there | |
| 19:25:49 | sean-k-mooney | you might be able to fix the superblock but im not sure if that will work | |
| 19:27:04 | sean-k-mooney | mnaser: i tought there was a devstack job for this yes gmann would know more | |
| 19:27:22 | mnaser | i mean enabling it in nova works fine, so nova is happy, but enabling in neutron makes it un happy | |
| 19:27:39 | mnaser | so it could be a neutron issue so i dont know if the devstack job enables it for all or just for specific services | |
| 19:27:42 | sean-k-mooney | right but i tought we had it enabled for both | |
| 19:28:02 | sean-k-mooney | ya its a good question im not sure either | |
| 19:29:30 | mnaser | https://github.com/openstack/nova/blob/master/.zuul.yaml#L678-L684 | |
| 19:29:31 | mnaser | wonder if its that | |
| 19:33:34 | sean-k-mooney | ya that should have it enabled for those 4 services | |
| 19:33:59 | sean-k-mooney | mnaser: https://zuul.openstack.org/builds?job_name=tempest-integrated-compute-enforce-scope-new-defaults&skip=0 | |
| 19:34:04 | sean-k-mooney | it looks pretty green too | |
| 19:34:50 | sean-k-mooney | welll there is at least some green menaing it should work in general but im not sure how much is covered by that | |
| 19:37:22 | mnaser | sean-k-mooney: i wonder if we are getting hit by this since its not in zed yet - https://github.com/openstack/neutron/commit/6d8ada0ac93beed05b45adb9582c3ef23bef49d2 | |
| 19:37:36 | mnaser | and the test that failed was actually as an admin | |
| 19:38:09 | sean-k-mooney | oh your trying to do this in zed | |
| 19:38:26 | sean-k-mooney | ya ok we only enabled it by defualt this cycle | |
| 19:39:36 | sean-k-mooney | mnaser: im not sure that was planned ot be backported | |
| 19:40:51 | sean-k-mooney | i would ask the neutrnon folk to backport it if you intend to enable it | |
| 19:41:06 | sean-k-mooney | its kind fo feature ish | |
| 19:41:58 | mnaser | sean-k-mooney: yeah i guess one could argue its a bug too | |
| 19:42:37 | sean-k-mooney | its because of the pivort that happend at the yoga fourm/ptg | |
| 19:42:53 | sean-k-mooney | when we deiced to revert a lot of the work and remove the use of scopes form most apis | |
| 19:43:22 | sean-k-mooney | under the orginal plan admin should not be global admin | |
| 19:43:46 | sean-k-mooney | so they adapted to that change in zed | |
| 21:51:09 | gmann | mnaser: yes, you found those. during integration testing in this cycle (tempest-full-enforce-scope-new-defaults), we found few bugs in neutron and they got fixed in master | |
| 21:51:22 | gmann | mnaser: I think slaweq was planning to backport those to stable/zed or older if needed | |
| 21:51:47 | gmann | let me find those patches | |
| 21:55:24 | gmann | basically these three bugs https://bugs.launchpad.net/neutron/+bug/1996150 https://bugs.launchpad.net/neutron/+bug/1996836 https://bugs.launchpad.net/neutron/+bug/1997089 | |
| 21:57:58 | gmann | mnaser: pinged about these in neutron channel. I was in impressions that they wee backported already | |