| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2023-02-16 | |||
| 15:47:29 | bauzas | that's a good catch | |
| 15:47:35 | ralonsoh | dansmith, no, this is the VM booting | |
| 15:47:41 | bauzas | I haven't checked the flavors that were related to the cirros failures I saw | |
| 15:47:42 | ralonsoh | first log in the compute agent | |
| 15:47:54 | dansmith | flavors are 42, and 84 | |
| 15:48:21 | ralonsoh | 128 and 192 | |
| 15:48:26 | ralonsoh | I'll check neutron CI | |
| 15:49:52 | dansmith | 42 is the default, and that's 128M | |
| 15:50:19 | opendevreview | Alexey Stupnikov proposed openstack/nova stable/ussuri: Test aborting queued live migration https://review.opendev.org/c/openstack/nova/+/873575 | |
| 15:50:34 | dansmith | 84 is 192M yeah | |
| 15:51:06 | dansmith | so perhaps we're flying too close to the sun with 128M | |
| 15:51:38 | ralonsoh | dansmith, I'm checking what we are using in Neutron | |
| 15:52:54 | dansmith | AFAIK, these are devstack defaults | |
| 15:53:10 | ralonsoh | we use the default values too, 128M | |
| 15:53:14 | dansmith | bumping the flavor memory is likely to result in other instabilities, I fear | |
| 15:53:16 | ralonsoh | {'resources': {'DISK_GB': 1, 'MEMORY_MB': 128, 'VCPU': 1}} | |
| 15:53:20 | dansmith | yeah | |
| 15:54:20 | ralonsoh | if you could retrieve the console logs, that could help | |
| 15:54:32 | ralonsoh | at least to know that the VM tried to request an IP | |
| 15:54:43 | ralonsoh | and, somewhere, this request was dropped | |
| 15:55:27 | dansmith | yeah, we should chat with gmann about it. I can go look at the tempest stuff to figure out why, | |
| 15:55:32 | dansmith | but he probably knows off the top of his head | |
| 16:04:19 | bauzas | ralonsoh: I have a few logs where the guest failed to acquire a lease, sec | |
| 16:04:36 | bauzas | and some where the guest panickjed | |
| 16:06:11 | sean-k-mooney | for what its worht the every increaseign memory requirement for cirrios was one of the reasons i looked at moving us to alpine a few years ago | |
| 16:06:20 | sean-k-mooney | longterm i still think that would be a better approch | |
| 16:06:58 | dansmith | is alpine really going to be smaller than cirros? I mean, that seems odd to me | |
| 16:07:10 | opendevreview | Alexey Stupnikov proposed openstack/nova stable/victoria: Test aborting queued live migration https://review.opendev.org/c/openstack/nova/+/845748 | |
| 16:07:11 | opendevreview | Alexey Stupnikov proposed openstack/nova stable/victoria: Add functional tests to reproduce bug #1960412 https://review.opendev.org/c/openstack/nova/+/845753 | |
| 16:07:22 | dansmith | so I think we get the automatic console stuff when we use specific waiters for servers to be available | |
| 16:07:28 | dansmith | so that test must use a different one | |
| 16:07:52 | bauzas | oh damn, internal meeting | |
| 16:08:13 | ralonsoh | bauzas, do you have some links? in any case, this is not Nova nor Neutron fault, I think | |
| 16:08:14 | opendevreview | Alexey Stupnikov proposed openstack/nova stable/victoria: Clean up when queued live migration aborted https://review.opendev.org/c/openstack/nova/+/845754 | |
| 16:08:16 | ralonsoh | at least in this case | |
| 16:08:47 | bauzas | ralonsoh: yup, which kind of failures do you want to see ? for the dhcp query? | |
| 16:09:04 | ralonsoh | yes and the kernel panic | |
| 16:20:57 | bauzas | ralonsoh: one for segfaults https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_da2/821228/7/check/nova-multi-cell/da2689f/job-output.txt | |
| 16:23:22 | bauzas | ralonsoh: that one for kernel panicking https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_e4a/821228/7/gate/nova-next/e4ab52f/job-output.txt | |
| 16:24:47 | bauzas | ralonsoh: that one is an interesting case where the default route is already present and metadata goes into weeds https://ae59d1e8526fa7671728-240e4b572b6f89b26c1b0e70b1c00c17.ssl.cf1.rackcdn.com/872413/3/check/nova-multi-cell/5e89e48/job-output.txt | |
| 16:27:11 | ralonsoh | I'll check this last one | |
| 16:28:05 | ralonsoh | bauzas, eh hold on, this could be a problem with the OVN version in Jammy | |
| 16:28:27 | ralonsoh | --> https://review.opendev.org/c/openstack/neutron/+/873684 | |
| 16:28:59 | ralonsoh | there is an issue with ovn v22.03.0, included in jammy | |
| 16:29:05 | ralonsoh | and some missing flows for the metadata | |
| 16:29:17 | ralonsoh | Yatin found it and we are skipping those tests | |
| 16:29:38 | ralonsoh | actually we are going to test using a compiled version of OVN | |
| 16:29:41 | ralonsoh | https://review.opendev.org/c/openstack/neutron/+/874112/1 | |
| 16:30:52 | bauzas | ack ok | |
| 16:51:31 | opendevreview | Alexey Stupnikov proposed openstack/nova stable/ussuri: Test aborting queued live migration https://review.opendev.org/c/openstack/nova/+/873575 | |
| 16:51:32 | opendevreview | Alexey Stupnikov proposed openstack/nova stable/ussuri: Add functional tests to reproduce bug #1960412 https://review.opendev.org/c/openstack/nova/+/873576 | |
| 17:01:18 | opendevreview | Alexey Stupnikov proposed openstack/nova stable/ussuri: Clean up when queued live migration aborted https://review.opendev.org/c/openstack/nova/+/873577 | |
| 17:09:43 | opendevreview | Alexey Stupnikov proposed openstack/nova stable/ussuri: Clean up when queued live migration aborted https://review.opendev.org/c/openstack/nova/+/873577 | |
| 17:25:16 | fungi | ralonsoh: any idea if the fixes have been backported to v22 such that ubuntu could do an sru to patch their packages? | |
| 17:25:57 | ralonsoh | fungi, Yatin opened a bug today: https://bugs.launchpad.net/ubuntu/+source/ovn/+bug/2003056 | |
| 17:26:02 | fungi | awesome | |
| 17:26:12 | ralonsoh | (sorry, not today) | |
| 17:26:31 | fungi | i'd hate to see devstack using a bespoke ovn build long-term | |
| 17:26:42 | fungi | here's hoping they're able to patch it | |
| 17:27:15 | ralonsoh | from https://bugs.launchpad.net/ubuntu/+source/ovn/+bug/2003056/comments/4, there should be a new version now (22.04.1, instead of 22.03) | |
| 17:29:24 | bauzas | ralonsoh: I guess you don't recommend us to work on nova's zuul jobs to build ovs from source? | |
| 17:29:34 | fungi | oh, it didn't dawn on me that those might be date-based versions rather than semver, so yeah here's hoping backporting the fix in ubuntu won't be painful | |
| 17:29:43 | bauzas | s/ovs/ovn | |
| 17:29:54 | ralonsoh | bauzas, no, we should use the OS released version | |
| 17:30:01 | bauzas | cool | |
| 17:30:09 | ralonsoh | we use compiled version in Neutron for testing only | |
| 17:30:18 | bauzas | yup saw the DNM | |
| 17:30:22 | ralonsoh | (and we had problems for this, this is why we use both now) | |
| 17:30:34 | bauzas | but I was wondering how much of this was actionable on our side | |
| 17:30:47 | bauzas | I'm like done rechecking every 2 hours | |
| 17:31:52 | bauzas | any actional progress sounds to a sweet spot :) | |
| 17:31:58 | bauzas | sounds to me* | |
| 17:32:21 | fungi | oh yay, so it ended up in j-p-u yesterday and should be showing up on mirrors at any moment assuming the ubuntu autobuilders aren't clogged | |
| 17:33:14 | fungi | we may need to add that repository temporarily to the sources list in affected jobs, until it migrates into a jammy point release | |
| 17:35:17 | fungi | https://launchpad.net/ubuntu/+source/ovn/22.03.2-0ubuntu0.22.04.1 indicates the binary packages haven't built yet | |
| 17:37:03 | bauzas | I hereby declare Nova on Feature Freeze :) | |
| 17:37:27 | bauzas | (anyway, all the accepted blueprints were reviewed) | |
| 17:37:59 | bauzas | kbye ;) | |
| 17:38:31 | fungi | fnordahl: i'm a little fuzzy on ubuntu's sru flow... is it like proposed-updates in debian where it only makes it into the mainstream indices in periodic point releases and we need to put jammy-proposed in sources.list in the interim? | |
| 17:56:29 | gmann | dansmith: bauzas: ralonsoh: not read all the logs but related to cirros image there is patch up to bump it to version 0.6.1. https://review.opendev.org/c/openstack/devstack/+/859773 | |
| 17:59:10 | ralonsoh | gmann, thanks. We have seen some seg faults and kernel panics during the VM boot, using this image | |
| 17:59:40 | ralonsoh | in https://review.opendev.org/c/openstack/nova/+/873934 | |
| 19:05:01 | spatel | sean-k-mooney i saw your post about my question related ceph disaster | |
| 19:05:39 | spatel | I am trying to do rescue method and stuck here - https://ibb.co/y84BLGY | |
| 19:08:37 | sean-k-mooney | hum ok have you tried un rescuing and seing if it fixed enough for the vm ot recover its self | |
| 19:09:01 | spatel | Let me try now.. | |
| 19:10:31 | spatel | no luck - https://ibb.co/4swXjQ1 | |
| 19:11:19 | spatel | ceph status showing all PGs are clean and active - https://paste.opendev.org/show/bnG8rvXJydADTZknd2QD/ | |
| 19:11:36 | spatel | not sure why i got filesystem corruption. | |
| 19:24:53 | mnaser | is there ci jobs that are testing secure rbac across all services? | |
| 19:25:05 | mnaser | in an all zed env, enabling it for neutron seems to break new vm deployments | |
| 19:25:28 | mnaser | something along these lines: 2023-02-15 22:04:28.241 2935939 ERROR nova.compute.manager [instance: aaac6261-721a-404d-80d3-94cf25bf869b] neutronclient.common.exceptions.PortNotFoundClient: Port 485c1e6e-fdcd-45ca-ae52-c2cdeb95bfdd could not be found. | |
| 19:25:29 | sean-k-mooney | im not really sure what to do. you could try stoping the vm and mounting the volume on the host and running the filesystem recovery form there | |
| 19:25:49 | sean-k-mooney | you might be able to fix the superblock but im not sure if that will work | |
| 19:27:04 | sean-k-mooney | mnaser: i tought there was a devstack job for this yes gmann would know more | |
| 19:27:22 | mnaser | i mean enabling it in nova works fine, so nova is happy, but enabling in neutron makes it un happy | |
| 19:27:39 | mnaser | so it could be a neutron issue so i dont know if the devstack job enables it for all or just for specific services | |
| 19:27:42 | sean-k-mooney | right but i tought we had it enabled for both | |
| 19:28:02 | sean-k-mooney | ya its a good question im not sure either | |
| 19:29:30 | mnaser | https://github.com/openstack/nova/blob/master/.zuul.yaml#L678-L684 | |