| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2023-02-16 | |||
| 14:52:46 | ralonsoh | this is the dhcp client used in cirros 0.6.1 | |
| 14:53:19 | ralonsoh | more info here: https://review.opendev.org/c/openstack/neutron/+/871272 | |
| 14:53:28 | ralonsoh | (in the commit message) | |
| 14:53:55 | dansmith | ralonsoh: but if the cirros client just uses it, what does the tempest config change? | |
| 14:54:39 | ralonsoh | dansmith, https://review.opendev.org/c/openstack/tempest/+/871270/2/tempest/common/utils/linux/remote_client.py | |
| 14:54:49 | ralonsoh | we choose what is the VM dhcp client | |
| 14:55:11 | ralonsoh | --> https://review.opendev.org/c/openstack/tempest/+/871270/2/tempest/config.py | |
| 14:55:21 | dansmith | ralonsoh: that's just for a manual renew, but doesn't affect how the guest works on first boot.. is that what you mean? | |
| 14:55:50 | ralonsoh | dansmith, no, that should not affect how the VM boots | |
| 14:56:06 | ralonsoh | the OS will use the exiting dhcp client | |
| 14:56:12 | dansmith | ralonsoh: gotcha okay, so I guess most of the issues I see getting an IP seem to be related to initial boot | |
| 14:56:17 | ralonsoh | right | |
| 14:56:38 | dansmith | okay cool, just making sure I understand | |
| 15:05:14 | bauzas | ralonsoh: and to clarify, cirros-0.6.x switched its dhch client to dhcpcd ? | |
| 15:05:25 | ralonsoh | yes | |
| 15:06:20 | bauzas | ack | |
| 15:06:32 | bauzas | then I understand what I wrote, huzzah :D | |
| 15:08:03 | dansmith | ralonsoh: does neutron record an event if the IP gets actually leased? | |
| 15:08:32 | dansmith | meaning, on failure can we query to neutron to see if the guest ever pulled its IP, to distinguish between "we can't ssh to the guest because of network problems" vs. "the guest is not alive and never pulled its ip" ? | |
| 15:08:34 | ralonsoh | dansmith, let me check, maybe in the syslog | |
| 15:08:52 | ralonsoh | understood, let me check | |
| 15:10:36 | ralonsoh | dansmith, neutron builds (adds/deletes) the leases file but we don't log this event. This is done by dnsmasq, you should be able to see that in syslog | |
| 15:10:47 | bauzas | oh good point | |
| 15:10:48 | sean-k-mooney | dansmith: i dont think neutron does but dnsmacq might | |
| 15:11:02 | bauzas | I forgot to look at dnsmasq, fucking shit | |
| 15:11:18 | bauzas | my ops skills become rusty | |
| 15:11:26 | dansmith | it's too bad because it might be a nice API to be able to poke that remotely.. i.e. instead of sshing forever, have a reasonably short timeout for the is-it-leased | |
| 15:11:33 | sean-k-mooney | bauzas: is this ovn | |
| 15:11:40 | fungi | so are the cirros kernel panics similar to one another, or random excuses? | |
| 15:11:42 | sean-k-mooney | because if its ovn we are not useing dnsmasq | |
| 15:11:54 | dansmith | and for reporting on failure, so we can say what forensics have been done | |
| 15:11:56 | sean-k-mooney | this is being handeled by openflow rules added by ovn | |
| 15:12:37 | dansmith | also, all three of those failed tests are volume-related | |
| 15:12:41 | sean-k-mooney | fungi: if they are related to acpi then its a know issue withthe cirrus 5.2 kernel | |
| 15:12:50 | dansmith | so I still wouldn't write-off it being a volume problem | |
| 15:13:02 | fungi | sean-k-mooney: sounds like a good reason to switch to 0.6.1 then | |
| 15:13:35 | dansmith | fungi: that's what I said, but 0.6.1 bringing other changes could be more destabilizing | |
| 15:13:37 | sean-k-mooney | fungi: i started working on alpine based image 2 years ago after i found out that the kernel bug was fixed in a later ubuntu kernel and cirro was just not updated | |
| 15:13:40 | ralonsoh | dansmith, what is the backend? OVS or OVN? | |
| 15:13:48 | dansmith | ralonsoh: I dunno | |
| 15:13:49 | ralonsoh | is this nova-next, right? | |
| 15:13:55 | dansmith | ralonsoh: yes | |
| 15:14:00 | ralonsoh | ok, let me check | |
| 15:14:02 | fungi | dansmith: agreed, the devils you know vs the ones you don't | |
| 15:14:34 | bauzas | ralonsoh: I've seen the dhcp lease problems in nova-next yes | |
| 15:14:59 | bauzas | man, can't I provide a regex when gerrit searching with 'comment' ? | |
| 15:15:01 | dansmith | this is one with the cirros bumped: https://44f9259a9cd22acee92d-000061e1666ecf9c52f0643ab3c391ab.ssl.cf1.rackcdn.com/873934/1/check/nova-next/785cc57/testr_results.html | |
| 15:15:21 | dansmith | three tests with ssh timeouts in one job is higher than average I'd say | |
| 15:15:23 | dansmith | which makes me concerned | |
| 15:15:51 | ralonsoh | dansmith, nova-next uses OVS. About this API call, could be something to be implemented, yes | |
| 15:15:56 | ralonsoh | but we don't have it now | |
| 15:16:23 | dansmith | ralonsoh: ack, it just seems like it would be nice to have | |
| 15:16:56 | sean-k-mooney | dansmith: im not sure it woudl be easy to do in all cases | |
| 15:17:04 | sean-k-mooney | it would be ml2 driver specific | |
| 15:17:10 | dansmith | sean-k-mooney: I'm sure it wouldn't | |
| 15:17:35 | sean-k-mooney | i dont think you coudl do it with ovn currenlty | |
| 15:17:43 | sean-k-mooney | not without changes to ovn | |
| 15:18:14 | dansmith | necessity, it's the mother of.. I forget.. something.. :D | |
| 15:19:33 | bauzas | don't let me play that mother game | |
| 15:19:56 | dansmith | bauzas: you and your language lately.. should probably avoid adding "mother" to things :D | |
| 15:20:57 | bauzas | :) | |
| 15:27:35 | fungi | bauzas: supposedly you can. "regular expressions can be enabled by starting with ^" https://review.opendev.org/Documentation/user-search.html | |
| 15:28:01 | fungi | says it specifically in the entry for the message: expression | |
| 15:28:02 | bauzas | ralonsoh: I just discovered some timeout on ovs | |
| 15:28:06 | bauzas | ralonsoh: Feb 15 17:21:05.272099 np0033113580 nova-compute[73993]: DEBUG ovsdbapp.backend.ovs_idl.vlog [-] 0-ms timeout {{(pid=73993) __log_wakeup /usr/local/lib/python3.10/dist-packages/ovs/poller.py:248}} | |
| 15:28:13 | bauzas | {{(pid=73993) _do_build_and_run_instance /opt/stack/nova/nova/compute/manager.py:2450}} | |
| 15:28:13 | bauzas | Feb 15 17:21:05.274517 np0033113580 nova-compute[73993]: DEBUG nova.compute.manager [None req-f2621b7f-2ead-4510-9e82-aad93ba9f29d tempest-ListServersNegativeTestJSON-1193978856 tempest-ListServersNegativeTestJSON-1193978856-project] [instance: d68e1732-b508-4f6b-be25-cbae06fde7c2] Build of instance d68e1732-b508-4f6b-be25-cbae06fde7c2 was re-scheduled: Timed out waiting for a reply to message ID c6193dab09f444b796478e321d69e3a2 | |
| 15:28:31 | bauzas | fungi: damn shit, missed that even if I did RTFM | |
| 15:28:32 | fungi | bauzas: sorry, i misread what you said. you're searching for comment not message | |
| 15:28:40 | bauzas | fungi: yeah that | |
| 15:28:49 | ralonsoh | bauzas, what is this job link? | |
| 15:28:52 | fungi | the entry for comment: doesn't mention regex | |
| 15:29:06 | fungi | just says it's a string match | |
| 15:29:13 | bauzas | ralonsoh: something new https://zuul.opendev.org/t/openstack/build/76f29afe3f134f139a48b537de7029dc | |
| 15:29:14 | fungi | so you're probably right | |
| 15:29:23 | bauzas | fungi: doh | |
| 15:29:28 | fungi | sadly | |
| 15:29:35 | bauzas | ok, I was wanting to query all the recheck messages I wrote | |
| 15:29:44 | bauzas | and since I haven't followed a clear pattern... | |
| 15:30:07 | fungi | could script that through the rest api, but it would be a bit of work | |
| 15:30:19 | fungi | depends on how much you want it, i guess | |
| 15:31:28 | bauzas | well, it was just for a quick lool | |
| 15:31:35 | bauzas | look* even | |
| 15:31:37 | bauzas | nevermind | |
| 15:32:16 | ralonsoh | bauzas, I don't know where this message is coming, but this is the OVS local service | |
| 15:32:38 | bauzas | yup on n-cpu | |
| 15:32:55 | bauzas | but apparently it's enough serious to do a reschedule | |
| 15:33:19 | bauzas | ... which on an AIO doesn't help | |
| 15:33:22 | bauzas | :) | |
| 15:34:05 | ralonsoh | if not events: | |
| 15:34:05 | ralonsoh | def __log_wakeup(self, events): | |
| 15:34:05 | ralonsoh | bauzas, actually this is something normal, coming from the python ovs bindings | |
| 15:34:06 | ralonsoh | vlog.dbg("%d-ms timeout" % self.timeout) | |
| 15:34:52 | ralonsoh | https://github.com/openvswitch/ovs/blob/master/python/ovs/poller.py#L246# | |
| 15:37:00 | bauzas | hmmmm | |
| 15:37:23 | bauzas | I'm able to find another patchset that got the exact failure from the same job on the same test tempest.api.compute.servers.test_list_servers_negative.ListServersNegativeTestJSON | |
| 15:37:27 | bauzas | https://d2746e36843633ae266c-ad135f72b22a132e11a904324fbc4e60.ssl.cf1.rackcdn.com/872413/3/check/tempest-integrated-compute-enforce-scope-new-defaults/5204f72/job-output.txt | |
| 15:41:05 | ralonsoh | bauzas, I'm checking https://44f9259a9cd22acee92d-000061e1666ecf9c52f0643ab3c391ab.ssl.cf1.rackcdn.com/873934/1/check/nova-next/785cc57 | |
| 15:41:15 | ralonsoh | for example the first test case failing | |
| 15:41:24 | ralonsoh | test_attach_scsi_disk_with_config_drive | |