| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2023-02-16 | |||
| 15:08:32 | dansmith | meaning, on failure can we query to neutron to see if the guest ever pulled its IP, to distinguish between "we can't ssh to the guest because of network problems" vs. "the guest is not alive and never pulled its ip" ? | |
| 15:08:34 | ralonsoh | dansmith, let me check, maybe in the syslog | |
| 15:08:52 | ralonsoh | understood, let me check | |
| 15:10:36 | ralonsoh | dansmith, neutron builds (adds/deletes) the leases file but we don't log this event. This is done by dnsmasq, you should be able to see that in syslog | |
| 15:10:47 | bauzas | oh good point | |
| 15:10:48 | sean-k-mooney | dansmith: i dont think neutron does but dnsmacq might | |
| 15:11:02 | bauzas | I forgot to look at dnsmasq, fucking shit | |
| 15:11:18 | bauzas | my ops skills become rusty | |
| 15:11:26 | dansmith | it's too bad because it might be a nice API to be able to poke that remotely.. i.e. instead of sshing forever, have a reasonably short timeout for the is-it-leased | |
| 15:11:33 | sean-k-mooney | bauzas: is this ovn | |
| 15:11:40 | fungi | so are the cirros kernel panics similar to one another, or random excuses? | |
| 15:11:42 | sean-k-mooney | because if its ovn we are not useing dnsmasq | |
| 15:11:54 | dansmith | and for reporting on failure, so we can say what forensics have been done | |
| 15:11:56 | sean-k-mooney | this is being handeled by openflow rules added by ovn | |
| 15:12:37 | dansmith | also, all three of those failed tests are volume-related | |
| 15:12:41 | sean-k-mooney | fungi: if they are related to acpi then its a know issue withthe cirrus 5.2 kernel | |
| 15:12:50 | dansmith | so I still wouldn't write-off it being a volume problem | |
| 15:13:02 | fungi | sean-k-mooney: sounds like a good reason to switch to 0.6.1 then | |
| 15:13:35 | dansmith | fungi: that's what I said, but 0.6.1 bringing other changes could be more destabilizing | |
| 15:13:37 | sean-k-mooney | fungi: i started working on alpine based image 2 years ago after i found out that the kernel bug was fixed in a later ubuntu kernel and cirro was just not updated | |
| 15:13:40 | ralonsoh | dansmith, what is the backend? OVS or OVN? | |
| 15:13:48 | dansmith | ralonsoh: I dunno | |
| 15:13:49 | ralonsoh | is this nova-next, right? | |
| 15:13:55 | dansmith | ralonsoh: yes | |
| 15:14:00 | ralonsoh | ok, let me check | |
| 15:14:02 | fungi | dansmith: agreed, the devils you know vs the ones you don't | |
| 15:14:34 | bauzas | ralonsoh: I've seen the dhcp lease problems in nova-next yes | |
| 15:14:59 | bauzas | man, can't I provide a regex when gerrit searching with 'comment' ? | |
| 15:15:01 | dansmith | this is one with the cirros bumped: https://44f9259a9cd22acee92d-000061e1666ecf9c52f0643ab3c391ab.ssl.cf1.rackcdn.com/873934/1/check/nova-next/785cc57/testr_results.html | |
| 15:15:21 | dansmith | three tests with ssh timeouts in one job is higher than average I'd say | |
| 15:15:23 | dansmith | which makes me concerned | |
| 15:15:51 | ralonsoh | dansmith, nova-next uses OVS. About this API call, could be something to be implemented, yes | |
| 15:15:56 | ralonsoh | but we don't have it now | |
| 15:16:23 | dansmith | ralonsoh: ack, it just seems like it would be nice to have | |
| 15:16:56 | sean-k-mooney | dansmith: im not sure it woudl be easy to do in all cases | |
| 15:17:04 | sean-k-mooney | it would be ml2 driver specific | |
| 15:17:10 | dansmith | sean-k-mooney: I'm sure it wouldn't | |
| 15:17:35 | sean-k-mooney | i dont think you coudl do it with ovn currenlty | |
| 15:17:43 | sean-k-mooney | not without changes to ovn | |
| 15:18:14 | dansmith | necessity, it's the mother of.. I forget.. something.. :D | |
| 15:19:33 | bauzas | don't let me play that mother game | |
| 15:19:56 | dansmith | bauzas: you and your language lately.. should probably avoid adding "mother" to things :D | |
| 15:20:57 | bauzas | :) | |
| 15:27:35 | fungi | bauzas: supposedly you can. "regular expressions can be enabled by starting with ^" https://review.opendev.org/Documentation/user-search.html | |
| 15:28:01 | fungi | says it specifically in the entry for the message: expression | |
| 15:28:02 | bauzas | ralonsoh: I just discovered some timeout on ovs | |
| 15:28:06 | bauzas | ralonsoh: Feb 15 17:21:05.272099 np0033113580 nova-compute[73993]: DEBUG ovsdbapp.backend.ovs_idl.vlog [-] 0-ms timeout {{(pid=73993) __log_wakeup /usr/local/lib/python3.10/dist-packages/ovs/poller.py:248}} | |
| 15:28:13 | bauzas | Feb 15 17:21:05.274517 np0033113580 nova-compute[73993]: DEBUG nova.compute.manager [None req-f2621b7f-2ead-4510-9e82-aad93ba9f29d tempest-ListServersNegativeTestJSON-1193978856 tempest-ListServersNegativeTestJSON-1193978856-project] [instance: d68e1732-b508-4f6b-be25-cbae06fde7c2] Build of instance d68e1732-b508-4f6b-be25-cbae06fde7c2 was re-scheduled: Timed out waiting for a reply to message ID c6193dab09f444b796478e321d69e3a2 | |
| 15:28:13 | bauzas | {{(pid=73993) _do_build_and_run_instance /opt/stack/nova/nova/compute/manager.py:2450}} | |
| 15:28:31 | bauzas | fungi: damn shit, missed that even if I did RTFM | |
| 15:28:32 | fungi | bauzas: sorry, i misread what you said. you're searching for comment not message | |
| 15:28:40 | bauzas | fungi: yeah that | |
| 15:28:49 | ralonsoh | bauzas, what is this job link? | |
| 15:28:52 | fungi | the entry for comment: doesn't mention regex | |
| 15:29:06 | fungi | just says it's a string match | |
| 15:29:13 | bauzas | ralonsoh: something new https://zuul.opendev.org/t/openstack/build/76f29afe3f134f139a48b537de7029dc | |
| 15:29:14 | fungi | so you're probably right | |
| 15:29:23 | bauzas | fungi: doh | |
| 15:29:28 | fungi | sadly | |
| 15:29:35 | bauzas | ok, I was wanting to query all the recheck messages I wrote | |
| 15:29:44 | bauzas | and since I haven't followed a clear pattern... | |
| 15:30:07 | fungi | could script that through the rest api, but it would be a bit of work | |
| 15:30:19 | fungi | depends on how much you want it, i guess | |
| 15:31:28 | bauzas | well, it was just for a quick lool | |
| 15:31:35 | bauzas | look* even | |
| 15:31:37 | bauzas | nevermind | |
| 15:32:16 | ralonsoh | bauzas, I don't know where this message is coming, but this is the OVS local service | |
| 15:32:38 | bauzas | yup on n-cpu | |
| 15:32:55 | bauzas | but apparently it's enough serious to do a reschedule | |
| 15:33:19 | bauzas | ... which on an AIO doesn't help | |
| 15:33:22 | bauzas | :) | |
| 15:34:05 | ralonsoh | bauzas, actually this is something normal, coming from the python ovs bindings | |
| 15:34:05 | ralonsoh | def __log_wakeup(self, events): | |
| 15:34:05 | ralonsoh | if not events: | |
| 15:34:06 | ralonsoh | vlog.dbg("%d-ms timeout" % self.timeout) | |
| 15:34:52 | ralonsoh | https://github.com/openvswitch/ovs/blob/master/python/ovs/poller.py#L246# | |
| 15:37:00 | bauzas | hmmmm | |
| 15:37:23 | bauzas | I'm able to find another patchset that got the exact failure from the same job on the same test tempest.api.compute.servers.test_list_servers_negative.ListServersNegativeTestJSON | |
| 15:37:27 | bauzas | https://d2746e36843633ae266c-ad135f72b22a132e11a904324fbc4e60.ssl.cf1.rackcdn.com/872413/3/check/tempest-integrated-compute-enforce-scope-new-defaults/5204f72/job-output.txt | |
| 15:41:05 | ralonsoh | bauzas, I'm checking https://44f9259a9cd22acee92d-000061e1666ecf9c52f0643ab3c391ab.ssl.cf1.rackcdn.com/873934/1/check/nova-next/785cc57 | |
| 15:41:15 | ralonsoh | for example the first test case failing | |
| 15:41:24 | ralonsoh | test_attach_scsi_disk_with_config_drive | |
| 15:42:01 | ralonsoh | I see the DHCP agent configuring dnsmasq process, I see n-cpu creating the interface, OVS agent receiving this creation event | |
| 15:42:17 | ralonsoh | but I see nowhere in the DHCP agent logs the DHCPREQUEST for this fixed IP | |
| 15:42:43 | ralonsoh | and the tempest test doesn't retrieve the VM logs | |
| 15:43:26 | dansmith | yeah I'm not sure why we're not logging the console in that case | |
| 15:43:35 | dansmith | because we should see that, to know if it's a guest crash | |
| 15:46:15 | ralonsoh | dansmith, qq, this is about the image resources | |
| 15:46:25 | ralonsoh | Feb 15 15:24:26.291000 np0033111254 nova-compute[39923]: DEBUG nova.compute.resource_tracker [None req-a80484d7-0e98-4086-a01f-afda9d54f06b None None] Instance 749a5ad0-e7b8-47f2-8e9c-78896c92b80e actively managed on this compute host and has allocations in placement: {'resources': {'VCPU': 1, 'MEMORY_MB': 128, 'DISK_GB': 1}}. {{(pid=39923) _remove_deleted_instances_allocations /opt/stack/nova/nova/compute/resource_tra | |
| 15:46:25 | ralonsoh | cker.py:1632}} | |
| 15:46:32 | bauzas | despite, none of them show the console log | |
| 15:46:38 | ralonsoh | why are you using 128MB? | |
| 15:46:49 | ralonsoh | cirros should be using 256, right? | |
| 15:47:12 | bauzas | hmmm | |
| 15:47:14 | dansmith | ralonsoh: idk, maybe that was the result of a resize? | |
| 15:47:29 | bauzas | that's a good catch | |
| 15:47:35 | ralonsoh | dansmith, no, this is the VM booting | |
| 15:47:41 | bauzas | I haven't checked the flavors that were related to the cirros failures I saw | |
| 15:47:42 | ralonsoh | first log in the compute agent | |
| 15:47:54 | dansmith | flavors are 42, and 84 | |