| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2023-02-16 | |||
| 12:23:10 | bauzas | gibi: nope, we haven't merged yet the change | |
| 12:23:41 | bauzas | I haven't verified which cirros image we were having with the test, but I guess it's still 0.5 | |
| 12:24:06 | bauzas | 2023-02-15 20:08:15.854905 | controller | ++ stackrc:source:692 : DEFAULT_IMAGE_NAME=cirros-0.5.2-x86_64-disk 2023-02-15 20:08:15.856966 | controller | ++ stackrc:source:693 : DEFAULT_IMAGE_FILE_NAME=cirros-0.5.2-x86_64-disk.img | |
| 12:24:10 | bauzas | indeed | |
| 12:26:53 | sean-k-mooney | i said this a few days ago but i do think we should look at going to 0.6.x | |
| 12:27:17 | sean-k-mooney | there are a few kernel bugs in the 0.5.2 image that cause ocational kernel panics related to the apic in the guest | |
| 12:27:41 | sean-k-mooney | the 0.6.2 image is built on the ubuntu 22.04 kernel | |
| 12:29:10 | gibi | bauzas: thanks, then I guess yet another guest kernel bug | |
| 12:30:06 | bauzas | sean-k-mooney: I have a change for this | |
| 12:30:17 | bauzas | sec. | |
| 12:30:32 | bauzas | sean-k-mooney: https://review.opendev.org/c/openstack/nova/+/873934 | |
| 12:30:56 | bauzas | we could also use 0.6.2 if we want, I'm not against | |
| 12:31:08 | sean-k-mooney | i would prefer ot change it in devstack | |
| 12:31:26 | sean-k-mooney | we can do it per job | |
| 12:31:37 | sean-k-mooney | but it would be better to use the same cirror image in all jobs | |
| 12:32:00 | bauzas | sean-k-mooney: but we can test the new cirros image by nova-next and if we see it works, then we could indeed use it for all our jobs | |
| 12:32:15 | bauzas | nova-next is also there for testing new stuff | |
| 12:32:46 | bauzas | I'm just afraid of changing all our jobs by once without correctly testing first | |
| 12:33:01 | sean-k-mooney | i guess but given we have known kernel issue in the 5.2 image nad we have been trying to change it for years i would prefer to do the change in Antilepe if we can | |
| 14:02:40 | opendevreview | Amit Uniyal proposed openstack/nova master: Added a lock_unlock dcorator for instance https://review.opendev.org/c/openstack/nova/+/873648 | |
| 14:21:20 | opendevreview | Amit Uniyal proposed openstack/nova master: Added context manager for instance lock https://review.opendev.org/c/openstack/nova/+/873648 | |
| 14:42:28 | bauzas | aaaaaand now I see more and more cirros guest segfaulting... https://review.opendev.org/c/openstack/nova/+/821228/7 | |
| 14:42:38 | bauzas | https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_da2/821228/7/check/nova-multi-cell/da2689f/job-output.txt | |
| 14:44:02 | opendevreview | Takashi Natsume proposed openstack/nova master: doc: mark the max microversion for 2023.1 Antelope https://review.opendev.org/c/openstack/nova/+/874103 | |
| 14:50:30 | dansmith | bauzas: where does that scenario:dhcp_client thing come in? | |
| 14:50:56 | bauzas | dansmith: https://review.opendev.org/c/openstack/nova/+/873934 | |
| 14:51:06 | bauzas | dansmith: and https://launchpad.net/bugs/2006467 | |
| 14:51:09 | dansmith | bauzas: right, what does scenario:dhcp_client affect | |
| 14:51:39 | dansmith | does that end up with cirros behaving differently? or does it make tempest do something inside the guest? | |
| 14:51:45 | bauzas | dansmith: ralonsoh told me to use that way b/c of https://review.opendev.org/c/openstack/neutron/+/871272/1/zuul.d/tempest-multinode.yaml | |
| 14:52:02 | bauzas | and I shamelessly copy/pasteed | |
| 14:52:46 | ralonsoh | this is the dhcp client used in cirros 0.6.1 | |
| 14:53:19 | ralonsoh | more info here: https://review.opendev.org/c/openstack/neutron/+/871272 | |
| 14:53:28 | ralonsoh | (in the commit message) | |
| 14:53:55 | dansmith | ralonsoh: but if the cirros client just uses it, what does the tempest config change? | |
| 14:54:39 | ralonsoh | dansmith, https://review.opendev.org/c/openstack/tempest/+/871270/2/tempest/common/utils/linux/remote_client.py | |
| 14:54:49 | ralonsoh | we choose what is the VM dhcp client | |
| 14:55:11 | ralonsoh | --> https://review.opendev.org/c/openstack/tempest/+/871270/2/tempest/config.py | |
| 14:55:21 | dansmith | ralonsoh: that's just for a manual renew, but doesn't affect how the guest works on first boot.. is that what you mean? | |
| 14:55:50 | ralonsoh | dansmith, no, that should not affect how the VM boots | |
| 14:56:06 | ralonsoh | the OS will use the exiting dhcp client | |
| 14:56:12 | dansmith | ralonsoh: gotcha okay, so I guess most of the issues I see getting an IP seem to be related to initial boot | |
| 14:56:17 | ralonsoh | right | |
| 14:56:38 | dansmith | okay cool, just making sure I understand | |
| 15:05:14 | bauzas | ralonsoh: and to clarify, cirros-0.6.x switched its dhch client to dhcpcd ? | |
| 15:05:25 | ralonsoh | yes | |
| 15:06:20 | bauzas | ack | |
| 15:06:32 | bauzas | then I understand what I wrote, huzzah :D | |
| 15:08:03 | dansmith | ralonsoh: does neutron record an event if the IP gets actually leased? | |
| 15:08:32 | dansmith | meaning, on failure can we query to neutron to see if the guest ever pulled its IP, to distinguish between "we can't ssh to the guest because of network problems" vs. "the guest is not alive and never pulled its ip" ? | |
| 15:08:34 | ralonsoh | dansmith, let me check, maybe in the syslog | |
| 15:08:52 | ralonsoh | understood, let me check | |
| 15:10:36 | ralonsoh | dansmith, neutron builds (adds/deletes) the leases file but we don't log this event. This is done by dnsmasq, you should be able to see that in syslog | |
| 15:10:47 | bauzas | oh good point | |
| 15:10:48 | sean-k-mooney | dansmith: i dont think neutron does but dnsmacq might | |
| 15:11:02 | bauzas | I forgot to look at dnsmasq, fucking shit | |
| 15:11:18 | bauzas | my ops skills become rusty | |
| 15:11:26 | dansmith | it's too bad because it might be a nice API to be able to poke that remotely.. i.e. instead of sshing forever, have a reasonably short timeout for the is-it-leased | |
| 15:11:33 | sean-k-mooney | bauzas: is this ovn | |
| 15:11:40 | fungi | so are the cirros kernel panics similar to one another, or random excuses? | |
| 15:11:42 | sean-k-mooney | because if its ovn we are not useing dnsmasq | |
| 15:11:54 | dansmith | and for reporting on failure, so we can say what forensics have been done | |
| 15:11:56 | sean-k-mooney | this is being handeled by openflow rules added by ovn | |
| 15:12:37 | dansmith | also, all three of those failed tests are volume-related | |
| 15:12:41 | sean-k-mooney | fungi: if they are related to acpi then its a know issue withthe cirrus 5.2 kernel | |
| 15:12:50 | dansmith | so I still wouldn't write-off it being a volume problem | |
| 15:13:02 | fungi | sean-k-mooney: sounds like a good reason to switch to 0.6.1 then | |
| 15:13:35 | dansmith | fungi: that's what I said, but 0.6.1 bringing other changes could be more destabilizing | |
| 15:13:37 | sean-k-mooney | fungi: i started working on alpine based image 2 years ago after i found out that the kernel bug was fixed in a later ubuntu kernel and cirro was just not updated | |
| 15:13:40 | ralonsoh | dansmith, what is the backend? OVS or OVN? | |
| 15:13:48 | dansmith | ralonsoh: I dunno | |
| 15:13:49 | ralonsoh | is this nova-next, right? | |
| 15:13:55 | dansmith | ralonsoh: yes | |
| 15:14:00 | ralonsoh | ok, let me check | |
| 15:14:02 | fungi | dansmith: agreed, the devils you know vs the ones you don't | |
| 15:14:34 | bauzas | ralonsoh: I've seen the dhcp lease problems in nova-next yes | |
| 15:14:59 | bauzas | man, can't I provide a regex when gerrit searching with 'comment' ? | |
| 15:15:01 | dansmith | this is one with the cirros bumped: https://44f9259a9cd22acee92d-000061e1666ecf9c52f0643ab3c391ab.ssl.cf1.rackcdn.com/873934/1/check/nova-next/785cc57/testr_results.html | |
| 15:15:21 | dansmith | three tests with ssh timeouts in one job is higher than average I'd say | |
| 15:15:23 | dansmith | which makes me concerned | |
| 15:15:51 | ralonsoh | dansmith, nova-next uses OVS. About this API call, could be something to be implemented, yes | |
| 15:15:56 | ralonsoh | but we don't have it now | |
| 15:16:23 | dansmith | ralonsoh: ack, it just seems like it would be nice to have | |
| 15:16:56 | sean-k-mooney | dansmith: im not sure it woudl be easy to do in all cases | |
| 15:17:04 | sean-k-mooney | it would be ml2 driver specific | |
| 15:17:10 | dansmith | sean-k-mooney: I'm sure it wouldn't | |
| 15:17:35 | sean-k-mooney | i dont think you coudl do it with ovn currenlty | |
| 15:17:43 | sean-k-mooney | not without changes to ovn | |
| 15:18:14 | dansmith | necessity, it's the mother of.. I forget.. something.. :D | |
| 15:19:33 | bauzas | don't let me play that mother game | |
| 15:19:56 | dansmith | bauzas: you and your language lately.. should probably avoid adding "mother" to things :D | |
| 15:20:57 | bauzas | :) | |
| 15:27:35 | fungi | bauzas: supposedly you can. "regular expressions can be enabled by starting with ^" https://review.opendev.org/Documentation/user-search.html | |
| 15:28:01 | fungi | says it specifically in the entry for the message: expression | |
| 15:28:02 | bauzas | ralonsoh: I just discovered some timeout on ovs | |
| 15:28:06 | bauzas | ralonsoh: Feb 15 17:21:05.272099 np0033113580 nova-compute[73993]: DEBUG ovsdbapp.backend.ovs_idl.vlog [-] 0-ms timeout {{(pid=73993) __log_wakeup /usr/local/lib/python3.10/dist-packages/ovs/poller.py:248}} | |
| 15:28:13 | bauzas | Feb 15 17:21:05.274517 np0033113580 nova-compute[73993]: DEBUG nova.compute.manager [None req-f2621b7f-2ead-4510-9e82-aad93ba9f29d tempest-ListServersNegativeTestJSON-1193978856 tempest-ListServersNegativeTestJSON-1193978856-project] [instance: d68e1732-b508-4f6b-be25-cbae06fde7c2] Build of instance d68e1732-b508-4f6b-be25-cbae06fde7c2 was re-scheduled: Timed out waiting for a reply to message ID c6193dab09f444b796478e321d69e3a2 | |
| 15:28:13 | bauzas | {{(pid=73993) _do_build_and_run_instance /opt/stack/nova/nova/compute/manager.py:2450}} | |
| 15:28:31 | bauzas | fungi: damn shit, missed that even if I did RTFM | |
| 15:28:32 | fungi | bauzas: sorry, i misread what you said. you're searching for comment not message | |