| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2023-08-10 | |||
| 17:10:57 | dansmith | melwitt: you've seen multiple cases of n-cpu just freezing or just the one? | |
| 17:11:36 | melwitt | my first thought was, is it possible n-cpu is being OOM killed or something but I don't see any OOM messages in syslog.txt (assuming that's where they would be) | |
| 17:13:42 | dansmith | yeah I saw no OOM in my case either (n-cpu or otherwise) | |
| 17:13:46 | dansmith | yep, syslog | |
| 17:15:36 | melwitt | I saw two so far, the other one being https://zuul.openstack.org/build/4884721dbf764b33995d71c990f65dfe/logs | |
| 17:15:57 | melwitt | (there's a third but it's in a DNM ceph job I'm using to test ephemeral encryption) | |
| 17:16:09 | dansmith | oh that one was mine, yeah | |
| 17:19:04 | dansmith | melwitt: so in that last one, the last thing nova logs is running a ceph command ... :) | |
| 17:19:20 | dansmith | oh but it returned | |
| 17:22:11 | melwitt | :) | |
| 17:22:43 | melwitt | it does seem to stop at the same place though, "Creating image(s)" | |
| 17:27:46 | dansmith | weird, I dunno what would have changed in the last few days related to this, and I haven't seen this before | |
| 17:28:22 | melwitt | same :( | |
| 17:33:36 | sean-k-mooney | f**** | |
| 17:34:08 | sean-k-mooney | DEBUG ovsdbapp.backend.ovs_idl.vlog [-] 4999-ms timeout {{(pid=1435862) __log_wakeup /opt/stack/data/venv/lib/python3.10/site-packages/ovs/poller.py:248}} | |
| 17:34:44 | sean-k-mooney | are we seeing that in ci | |
| 17:35:00 | sean-k-mooney | it looks liek the ovsdbapp thing might be regressed | |
| 17:35:10 | sean-k-mooney | im gong to check my devstack/nova logs | |
| 17:35:29 | sean-k-mooney | thats the issue where ovs blocks nova for 5 seconds at a time | |
| 17:36:21 | melwitt | actually I found one in there https://zuul.openstack.org/build/e7c306fad21e4057a3af40720049482a/log/controller/logs/screen-n-cpu.txt#39523 | |
| 17:36:23 | sean-k-mooney | this could be a regression in the devstack in a venv change | |
| 17:36:52 | sean-k-mooney | but my code in devstack to workaround this does not seam to have put the thing in the nova.conf | |
| 17:37:11 | melwitt | it's logged 15 times on that run | |
| 17:38:05 | sean-k-mooney | really..... https://github.com/openstack/devstack/commit/db3eff7dd27acdc973e8d189bda80d642be92f03 | |
| 17:39:04 | sean-k-mooney | i need to revert that | |
| 17:39:34 | sean-k-mooney | this is fix only if you use a new enough ovs | |
| 17:39:41 | sean-k-mooney | which apprenly is not in ubuntu 22.04 | |
| 17:42:17 | melwitt | the other example run has DEBUG ovsdbapp.backend.ovs_idl.vlog [-] 999-ms timeout {{(pid=98090) __log_wakeup /usr/local/lib/python3.10/dist-packages/ovs/poller.py:248}} not sure if that's bad also | |
| 17:42:35 | melwitt | it doesn't have the 4999 | |
| 17:42:50 | sean-k-mooney | yep the exact amount isnt really imporant | |
| 17:43:11 | sean-k-mooney | the timeout is around 5 seconds | |
| 17:43:36 | sean-k-mooney | but you can get reconnnect at shorter intervals depending on what else nova is doing | |
| 17:46:25 | sean-k-mooney | https://review.opendev.org/c/openstack/devstack/+/891101 will revert it | |
| 17:46:46 | melwitt | wonder why I only see it with ceph. a coincidence? | |
| 17:46:58 | sean-k-mooney | probaly im seeing it locally | |
| 17:47:12 | sean-k-mooney | i noticed it while i was looking at somethign for dansmith | |
| 17:47:35 | melwitt | oh ok | |
| 17:52:12 | sean-k-mooney | so i have a 50 second span | |
| 17:52:34 | sean-k-mooney | where the compute agent does not log anything else implying it was blocked for 50 seconds reconnecting ot ovs | |
| 17:52:58 | sean-k-mooney | the problem is i have a version of ovs that apprenly has this fix and a version of ovsdbapp that also has this fix | |
| 17:53:10 | sean-k-mooney | so either the fix does not work or it has regressed again | |
| 17:53:43 | sean-k-mooney | this woudl cause the ci jobs to fail alot but also cause issues for people in production obviously | |
| 17:56:44 | melwitt | sean-k-mooney++ | |
| 17:58:39 | sean-k-mooney | so i have a patch version os-vif, ovsdbapp and ovs python binings and its still happening meaning it looks like maybe it regressed again | |
| 18:00:49 | melwitt | 🤯 | |
| 18:00:49 | melwitt | 🤯 | |
| 18:02:16 | sean-k-mooney | or this is the problem https://github.com/openstack/os-vif/blob/master/vif_plug_ovs/ovsdb/impl_idl.py#L35 | |
| 18:02:37 | sean-k-mooney | this is the ovsdbapp fix | |
| 18:02:39 | sean-k-mooney | https://github.com/openstack/ovsdbapp/blob/master/ovsdbapp/backend/ovs_idl/connection.py#L148-L150 | |
| 18:02:56 | sean-k-mooney | in its subclass of idl.Idl | |
| 18:03:17 | sean-k-mooney | but we are doing " return idl.Idl(conn, helper)" directly usign the one form ovs | |
| 18:04:57 | sean-k-mooney | that should be using the NeutronOvsdbIdl class below i think | |
| 18:35:37 | dansmith | melwitt: oh SNAP | |
| 18:35:47 | dansmith | melwitt: I wonder if the thing with the ceph job is something I broke | |
| 18:36:01 | dansmith | melwitt: https://review.opendev.org/c/openstack/devstack/+/890439 | |
| 18:36:16 | dansmith | I adjusted the interval and the deadline so I think it *should* be okay | |
| 18:36:28 | dansmith | although that wouldn't explain nova just not logging anything after the hang | |
| 18:37:00 | melwitt | yeah.. 🤔 | |
| 18:37:00 | melwitt | yeah.. 🤔 | |
| 18:54:24 | sean-k-mooney | ok the issue i mentioned might be fixed | |
| 18:55:29 | sean-k-mooney | i configured my keepalive interval in the kernel to 10 secodn i think but ovs is expecting probes every 5 seconds | |
| 18:55:48 | melwitt | yeah, I was just looking at it again and n-cpu hadn't reported a heartbeat in 14 minutes and never came back after the hang | |
| 18:57:15 | melwitt | bizarre | |
| 19:00:13 | melwitt | libvirt log activity also stops for the most part at the same time n-cpu stops https://zuul.openstack.org/build/e7c306fad21e4057a3af40720049482a/log/controller/logs/libvirt/libvirt/libvirtd_log.txt | |
| 19:00:52 | melwitt | not sure if anything near the end of that log is unusual or problematic | |
| 21:16:31 | melwitt | finally found that on those failures, at the same time that n-cpu freezes, ceph starts logging WRN (warn) health status such as "osd.0 47 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.15524.0:32 4.1e 4:7cf325a3:::rbd_header.3c6ebb6b8706:head [watch ping cookie 140317856977664] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)" | |
| 21:16:45 | melwitt | example https://zuul.openstack.org/build/4884721dbf764b33995d71c990f65dfe/log/controller/logs/ceph/ceph/7f57d687-6678-46f4-ae09-7e6faa476d1a/ceph-osd.0_log.txt#11752 | |
| 21:17:35 | melwitt | log_channel(cluster) log [WRN] : 1 slow requests (by type [ 'delayed' : 1 ] most affected pool [ 'vms' : 1 ]) | |
| 22:50:41 | opendevreview | melanie witt proposed openstack/nova master: DNM test ephemeral encryption + resize: qcow2, raw, rbd https://review.opendev.org/c/openstack/nova/+/862416 | |
| #openstack-nova - 2023-08-11 | |||
| 08:19:13 | auniyal | Hey gibi, can you please have a look on https://review.opendev.org/c/openstack/nova/+/882920 mostly good just a backpot for yoga | |
| 09:03:06 | tobias-urdin | two small doc/log fixes when somebody has some seconds to review https://review.opendev.org/c/openstack/nova/+/880819 https://review.opendev.org/c/openstack/nova/+/873846 | |
| 09:22:33 | opendevreview | Danylo Vodopianov proposed openstack/nova master: Napatech SmartNIC support https://review.opendev.org/c/openstack/nova/+/859577 | |
| 09:24:29 | opendevreview | Danylo Vodopianov proposed openstack/os-vif master: Openvswitch driver was extended https://review.opendev.org/c/openstack/os-vif/+/859574 | |
| 09:24:44 | tobias-urdin | another one https://review.opendev.org/c/openstack/nova/+/874083 | |
| 10:20:21 | auniyal | thanks gibi | |
| 11:33:32 | opendevreview | Danylo Vodopianov proposed openstack/os-vif master: Openvswitch driver was extended https://review.opendev.org/c/openstack/os-vif/+/859574 | |
| 11:53:33 | opendevreview | Merged openstack/nova stable/yoga: Have host look for CPU controller of cgroupsv2 location. https://review.opendev.org/c/openstack/nova/+/882920 | |
| 12:54:55 | opendevreview | Viktor Křivák proposed openstack/nova-specs master: Add availability zone-aware affinity filter to the scheduler https://review.opendev.org/c/openstack/nova-specs/+/890779 | |
| 13:15:19 | opendevreview | Viktor Křivák proposed openstack/nova-specs master: Add availability zone-aware affinity filter to the scheduler https://review.opendev.org/c/openstack/nova-specs/+/890779 | |
| 14:56:23 | opendevreview | Dan Smith proposed openstack/nova master: WIP: Support glance's new location API https://review.opendev.org/c/openstack/nova/+/891036 | |
| 14:56:52 | dansmith | abhishekk: ^ untested, but that's part of your patch but with some new-or-old logic in it and less change overall as I think we can just ignore the return from that operation | |
| 14:57:10 | dansmith | we'll see how that goes, not sure I'll have a change to test locally today but we can iterate from that | |
| 14:57:25 | abhishekk | looking | |
| 15:00:18 | abhishekk | I didn't understood raise exc part at line #571 | |
| 15:03:09 | dansmith | abhishekk: oh that was supposed to end up under the badrequest thing | |
| 15:03:20 | dansmith | ah right but I need to test | |
| 15:03:45 | dansmith | so the goal is to try each of those interfaces, but if neither work, raise the error we got from the last one | |
| 15:04:23 | dansmith | so we might be able to just raise the badrequest whenever we get it, but I need to test glance's behavior with each of the new stuff missing and the old stuff disabled | |
| 15:04:49 | abhishekk | ack, so you need another plain Exception block as well | |
| 15:04:50 | abhishekk | ? | |
| 15:04:52 | dansmith | so that _reraise was supposed to be a pass and replace the raise below | |
| 15:05:06 | dansmith | no, I think we want to try both and raise the last one | |
| 15:05:34 | dansmith | and we need to catch whatever expected exceptions will come from trying the new interface if it's missing | |
| 15:05:36 | abhishekk | ack, because what I understood is lien #571 might express as raise None | |
| 15:05:40 | dansmith | should just be notfound right? | |
| 15:06:14 | dansmith | oh, two notfounds could raise none, yeah you're right | |
| 15:06:25 | dansmith | anyway, like i said, untested, just kinda what I'm thinking :) | |
| 15:06:37 | abhishekk | :D | |
| 15:07:21 | abhishekk | I am not sure client will be updated today to raise Notfound if version is less than 2.17 | |
| 15:07:43 | dansmith | if glance is not updated, the URL should generate a notfound, no? | |