Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2023-02-16
15:11:26 dansmith it's too bad because it might be a nice API to be able to poke that remotely.. i.e. instead of sshing forever, have a reasonably short timeout for the is-it-leased
15:11:33 sean-k-mooney bauzas: is this ovn
15:11:40 fungi so are the cirros kernel panics similar to one another, or random excuses?
15:11:42 sean-k-mooney because if its ovn we are not useing dnsmasq
15:11:54 dansmith and for reporting on failure, so we can say what forensics have been done
15:11:56 sean-k-mooney this is being handeled by openflow rules added by ovn
15:12:37 dansmith also, all three of those failed tests are volume-related
15:12:41 sean-k-mooney fungi: if they are related to acpi then its a know issue withthe cirrus 5.2 kernel
15:12:50 dansmith so I still wouldn't write-off it being a volume problem
15:13:02 fungi sean-k-mooney: sounds like a good reason to switch to 0.6.1 then
15:13:35 dansmith fungi: that's what I said, but 0.6.1 bringing other changes could be more destabilizing
15:13:37 sean-k-mooney fungi: i started working on alpine based image 2 years ago after i found out that the kernel bug was fixed in a later ubuntu kernel and cirro was just not updated
15:13:40 ralonsoh dansmith, what is the backend? OVS or OVN?
15:13:48 dansmith ralonsoh: I dunno
15:13:49 ralonsoh is this nova-next, right?
15:13:55 dansmith ralonsoh: yes
15:14:00 ralonsoh ok, let me check
15:14:02 fungi dansmith: agreed, the devils you know vs the ones you don't
15:14:34 bauzas ralonsoh: I've seen the dhcp lease problems in nova-next yes
15:14:59 bauzas man, can't I provide a regex when gerrit searching with 'comment' ?
15:15:01 dansmith this is one with the cirros bumped: https://44f9259a9cd22acee92d-000061e1666ecf9c52f0643ab3c391ab.ssl.cf1.rackcdn.com/873934/1/check/nova-next/785cc57/testr_results.html
15:15:21 dansmith three tests with ssh timeouts in one job is higher than average I'd say
15:15:23 dansmith which makes me concerned
15:15:51 ralonsoh dansmith, nova-next uses OVS. About this API call, could be something to be implemented, yes
15:15:56 ralonsoh but we don't have it now
15:16:23 dansmith ralonsoh: ack, it just seems like it would be nice to have
15:16:56 sean-k-mooney dansmith: im not sure it woudl be easy to do in all cases
15:17:04 sean-k-mooney it would be ml2 driver specific
15:17:10 dansmith sean-k-mooney: I'm sure it wouldn't
15:17:35 sean-k-mooney i dont think you coudl do it with ovn currenlty
15:17:43 sean-k-mooney not without changes to ovn
15:18:14 dansmith necessity, it's the mother of.. I forget.. something.. :D
15:19:33 bauzas don't let me play that mother game
15:19:56 dansmith bauzas: you and your language lately.. should probably avoid adding "mother" to things :D
15:20:57 bauzas :)
15:27:35 fungi bauzas: supposedly you can. "regular expressions can be enabled by starting with ^" https://review.opendev.org/Documentation/user-search.html
15:28:01 fungi says it specifically in the entry for the message: expression
15:28:02 bauzas ralonsoh: I just discovered some timeout on ovs
15:28:06 bauzas ralonsoh: Feb 15 17:21:05.272099 np0033113580 nova-compute[73993]: DEBUG ovsdbapp.backend.ovs_idl.vlog [-] 0-ms timeout {{(pid=73993) __log_wakeup /usr/local/lib/python3.10/dist-packages/ovs/poller.py:248}}
15:28:13 bauzas Feb 15 17:21:05.274517 np0033113580 nova-compute[73993]: DEBUG nova.compute.manager [None req-f2621b7f-2ead-4510-9e82-aad93ba9f29d tempest-ListServersNegativeTestJSON-1193978856 tempest-ListServersNegativeTestJSON-1193978856-project] [instance: d68e1732-b508-4f6b-be25-cbae06fde7c2] Build of instance d68e1732-b508-4f6b-be25-cbae06fde7c2 was re-scheduled: Timed out waiting for a reply to message ID c6193dab09f444b796478e321d69e3a2
15:28:13 bauzas {{(pid=73993) _do_build_and_run_instance /opt/stack/nova/nova/compute/manager.py:2450}}
15:28:31 bauzas fungi: damn shit, missed that even if I did RTFM
15:28:32 fungi bauzas: sorry, i misread what you said. you're searching for comment not message
15:28:40 bauzas fungi: yeah that
15:28:49 ralonsoh bauzas, what is this job link?
15:28:52 fungi the entry for comment: doesn't mention regex
15:29:06 fungi just says it's a string match
15:29:13 bauzas ralonsoh: something new https://zuul.opendev.org/t/openstack/build/76f29afe3f134f139a48b537de7029dc
15:29:14 fungi so you're probably right
15:29:23 bauzas fungi: doh
15:29:28 fungi sadly
15:29:35 bauzas ok, I was wanting to query all the recheck messages I wrote
15:29:44 bauzas and since I haven't followed a clear pattern...
15:30:07 fungi could script that through the rest api, but it would be a bit of work
15:30:19 fungi depends on how much you want it, i guess
15:31:28 bauzas well, it was just for a quick lool
15:31:35 bauzas look* even
15:31:37 bauzas nevermind
15:32:16 ralonsoh bauzas, I don't know where this message is coming, but this is the OVS local service
15:32:38 bauzas yup on n-cpu
15:32:55 bauzas but apparently it's enough serious to do a reschedule
15:33:19 bauzas ... which on an AIO doesn't help
15:33:22 bauzas :)
15:34:05 ralonsoh bauzas, actually this is something normal, coming from the python ovs bindings
15:34:05 ralonsoh def __log_wakeup(self, events):
15:34:05 ralonsoh if not events:
15:34:06 ralonsoh vlog.dbg("%d-ms timeout" % self.timeout)
15:34:52 ralonsoh https://github.com/openvswitch/ovs/blob/master/python/ovs/poller.py#L246#
15:37:00 bauzas hmmmm
15:37:23 bauzas I'm able to find another patchset that got the exact failure from the same job on the same test tempest.api.compute.servers.test_list_servers_negative.ListServersNegativeTestJSON
15:37:27 bauzas https://d2746e36843633ae266c-ad135f72b22a132e11a904324fbc4e60.ssl.cf1.rackcdn.com/872413/3/check/tempest-integrated-compute-enforce-scope-new-defaults/5204f72/job-output.txt
15:41:05 ralonsoh bauzas, I'm checking https://44f9259a9cd22acee92d-000061e1666ecf9c52f0643ab3c391ab.ssl.cf1.rackcdn.com/873934/1/check/nova-next/785cc57
15:41:15 ralonsoh for example the first test case failing
15:41:24 ralonsoh test_attach_scsi_disk_with_config_drive
15:42:01 ralonsoh I see the DHCP agent configuring dnsmasq process, I see n-cpu creating the interface, OVS agent receiving this creation event
15:42:17 ralonsoh but I see nowhere in the DHCP agent logs the DHCPREQUEST for this fixed IP
15:42:43 ralonsoh and the tempest test doesn't retrieve the VM logs
15:43:26 dansmith yeah I'm not sure why we're not logging the console in that case
15:43:35 dansmith because we should see that, to know if it's a guest crash
15:46:15 ralonsoh dansmith, qq, this is about the image resources
15:46:25 ralonsoh Feb 15 15:24:26.291000 np0033111254 nova-compute[39923]: DEBUG nova.compute.resource_tracker [None req-a80484d7-0e98-4086-a01f-afda9d54f06b None None] Instance 749a5ad0-e7b8-47f2-8e9c-78896c92b80e actively managed on this compute host and has allocations in placement: {'resources': {'VCPU': 1, 'MEMORY_MB': 128, 'DISK_GB': 1}}. {{(pid=39923) _remove_deleted_instances_allocations /opt/stack/nova/nova/compute/resource_tra
15:46:25 ralonsoh cker.py:1632}}
15:46:32 bauzas despite, none of them show the console log
15:46:38 ralonsoh why are you using 128MB?
15:46:49 ralonsoh cirros should be using 256, right?
15:47:12 bauzas hmmm
15:47:14 dansmith ralonsoh: idk, maybe that was the result of a resize?
15:47:29 bauzas that's a good catch
15:47:35 ralonsoh dansmith, no, this is the VM booting
15:47:41 bauzas I haven't checked the flavors that were related to the cirros failures I saw
15:47:42 ralonsoh first log in the compute agent
15:47:54 dansmith flavors are 42, and 84
15:48:21 ralonsoh 128 and 192
15:48:26 ralonsoh I'll check neutron CI
15:49:52 dansmith 42 is the default, and that's 128M
15:50:19 opendevreview Alexey Stupnikov proposed openstack/nova stable/ussuri: Test aborting queued live migration https://review.opendev.org/c/openstack/nova/+/873575
15:50:34 dansmith 84 is 192M yeah
15:51:06 dansmith so perhaps we're flying too close to the sun with 128M
15:51:38 ralonsoh dansmith, I'm checking what we are using in Neutron
15:52:54 dansmith AFAIK, these are devstack defaults

Earlier   Later