| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2023-08-08 | |||
| 18:14:05 | dansmith | ack | |
| 18:14:13 | sean-k-mooney | the same seams to be true for addign caching that is about to finish | |
| 18:14:27 | sean-k-mooney | i dont really see any indeication fo increase memory pressure | |
| 18:14:27 | dansmith | the variance is much larger these days than it used to be so I don't think we can really know without enabling it on a job or two and monitor it longer term | |
| 18:14:53 | sean-k-mooney | ya | |
| 18:16:25 | sean-k-mooney | most of the jobs i looked at didnt seam to be swappign that hard and then ones that were actully seamed to have more overall free memory then i expected | |
| 18:16:54 | dansmith | the ceph job is usually pretty deep into swap, although not enough to cause problems | |
| 18:16:57 | dansmith | it rarely ooms lately | |
| 18:17:40 | sean-k-mooney | ya i looked at ceph multi store | |
| 18:17:42 | sean-k-mooney | https://zuul.opendev.org/t/openstack/build/66029a86e4bd49d0935021197991a1db/log/controller/logs/screen-memory_tracker.txt | |
| 18:17:54 | dansmith | ack | |
| 18:18:05 | sean-k-mooney | low_point: 244568 | |
| 18:18:31 | sean-k-mooney | its usinbg about half the swap with 4125888 free of 8GB | |
| 18:19:07 | sean-k-mooney | that ran for 1 hr 36 mins 12 secs | |
| 18:19:14 | sean-k-mooney | which is on the lower end of normal https://zuul.opendev.org/t/openstack/builds?job_name=nova-ceph-multistore&project=openstack%2Fnova | |
| 18:19:40 | sean-k-mooney | but the job ranges form about 1:30 to 2 hours | |
| 18:26:47 | dansmith | gmann: I've seen timeouts in pep8 and cover today: https://zuul.opendev.org/t/openstack/build/0c92c74e8734426fb3fd05ff67893f60 | |
| 18:26:56 | dansmith | I'm sure those have to be related to the node right? | |
| 18:27:31 | gmann | its ovh provider this time, not 100% sure if it is slow node | |
| 18:27:42 | dansmith | welp... | |
| #openstack-nova - 2023-08-09 | |||
| 08:40:41 | opendevreview | Takashi Kajinami proposed openstack/nova master: conf: Add four new '[libvirt] live_migration_*' options https://review.opendev.org/c/openstack/nova/+/456571 | |
| 12:20:11 | opendevreview | Felipe Reyes proposed openstack/nova master: Add RISCV64 QEMU emulation support. https://review.opendev.org/c/openstack/nova/+/889137 | |
| #openstack-nova - 2023-08-10 | |||
| 01:01:07 | opendevreview | melanie witt proposed openstack/nova master: WIP libvirt: Introduce support for rbd with LUKS https://review.opendev.org/c/openstack/nova/+/889912 | |
| 13:24:02 | opendevreview | Elod Illes proposed openstack/nova master: Fix tox docs target https://review.opendev.org/c/openstack/nova/+/891033 | |
| 14:00:13 | opendevreview | Jimmy McCrory proposed openstack/nova master: Fix typo in placement audit command argument https://review.opendev.org/c/openstack/nova/+/891035 | |
| 14:27:55 | opendevreview | Dan Smith proposed openstack/nova master: DNM: Test glance new location API https://review.opendev.org/c/openstack/nova/+/891036 | |
| 14:56:20 | kashyap | dansmith: Thank you for doing the recheck work for me! https://review.opendev.org/c/openstack/nova/+/887255 | |
| 15:28:31 | ozzzo_work | I have a couple of VMs that are half-created: https://paste.openstack.org/show/b7wqqzgumoH8tdtvmEh1/ | |
| 15:28:48 | ozzzo_work | how can I clean this up? I looked in the db and they don't exist in "instances" | |
| 15:30:40 | dansmith | ozzzo_work: check the api database's instance_build_requests table | |
| 15:31:04 | dansmith | I'm guessing you're seeing that because those request-to-build-an-instance entries are there and we never got beyond that | |
| 15:31:26 | dansmith | I'd check the logs to see if you're seeing any fireworks there, because that *should* be resolvable, AFAIK | |
| 15:31:56 | dansmith | any manual DB surgery that has been done? | |
| 15:49:37 | ozzzo_work | oic it was in the nova_cell0 table | |
| 15:50:39 | ozzzo_work | we're not using multiple cells but I guess everything must go in cell0 by default now | |
| 16:12:36 | dansmith | ozzzo_work: dead instances that fail to schedule always go into the cell0 DB | |
| 16:12:53 | dansmith | everyone is using cells, you might just only have one cell of computes :) | |
| 17:02:04 | melwitt | dansmith: have you noticed "no valid host" fails in nova-ceph-multistore and "Seems service nova-compute on host np0034917240 is down. Last heartbeat was 2023-08-10 15:59:29. Elapsed time is 847.603029" messages in the n-sch log? | |
| 17:03:00 | dansmith | I saw one example of that and it looked like rabbit crashed or something because I saw a bunch of amq errors in multiple places | |
| 17:04:06 | melwitt | hm, I hadn't noticed the rabbit correlation | |
| 17:06:34 | melwitt | the couple of examples I saw so far, the n-cpu log just stops ... early. and the nova valid host "returned 0 hosts" messages in n-sch start after the last timestamp seen in n-cpu | |
| 17:09:05 | melwitt | last log message Aug 10 16:00:00.843521 https://zuul.openstack.org/build/e7c306fad21e4057a3af40720049482a/log/controller/logs/screen-n-cpu.txt and then first "ComputeFilter returned 0 hosts" at Aug 10 16:13:36.604315 https://zuul.openstack.org/build/e7c306fad21e4057a3af40720049482a/log/controller/logs/screen-n-sch.txt#3881 | |
| 17:10:57 | dansmith | melwitt: you've seen multiple cases of n-cpu just freezing or just the one? | |
| 17:11:36 | melwitt | my first thought was, is it possible n-cpu is being OOM killed or something but I don't see any OOM messages in syslog.txt (assuming that's where they would be) | |
| 17:13:42 | dansmith | yeah I saw no OOM in my case either (n-cpu or otherwise) | |
| 17:13:46 | dansmith | yep, syslog | |
| 17:15:36 | melwitt | I saw two so far, the other one being https://zuul.openstack.org/build/4884721dbf764b33995d71c990f65dfe/logs | |
| 17:15:57 | melwitt | (there's a third but it's in a DNM ceph job I'm using to test ephemeral encryption) | |
| 17:16:09 | dansmith | oh that one was mine, yeah | |
| 17:19:04 | dansmith | melwitt: so in that last one, the last thing nova logs is running a ceph command ... :) | |
| 17:19:20 | dansmith | oh but it returned | |
| 17:22:11 | melwitt | :) | |
| 17:22:43 | melwitt | it does seem to stop at the same place though, "Creating image(s)" | |
| 17:27:46 | dansmith | weird, I dunno what would have changed in the last few days related to this, and I haven't seen this before | |
| 17:28:22 | melwitt | same :( | |
| 17:33:36 | sean-k-mooney | f**** | |
| 17:34:08 | sean-k-mooney | DEBUG ovsdbapp.backend.ovs_idl.vlog [-] 4999-ms timeout {{(pid=1435862) __log_wakeup /opt/stack/data/venv/lib/python3.10/site-packages/ovs/poller.py:248}} | |
| 17:34:44 | sean-k-mooney | are we seeing that in ci | |
| 17:35:00 | sean-k-mooney | it looks liek the ovsdbapp thing might be regressed | |
| 17:35:10 | sean-k-mooney | im gong to check my devstack/nova logs | |
| 17:35:29 | sean-k-mooney | thats the issue where ovs blocks nova for 5 seconds at a time | |
| 17:36:21 | melwitt | actually I found one in there https://zuul.openstack.org/build/e7c306fad21e4057a3af40720049482a/log/controller/logs/screen-n-cpu.txt#39523 | |
| 17:36:23 | sean-k-mooney | this could be a regression in the devstack in a venv change | |
| 17:36:52 | sean-k-mooney | but my code in devstack to workaround this does not seam to have put the thing in the nova.conf | |
| 17:37:11 | melwitt | it's logged 15 times on that run | |
| 17:38:05 | sean-k-mooney | really..... https://github.com/openstack/devstack/commit/db3eff7dd27acdc973e8d189bda80d642be92f03 | |
| 17:39:04 | sean-k-mooney | i need to revert that | |
| 17:39:34 | sean-k-mooney | this is fix only if you use a new enough ovs | |
| 17:39:41 | sean-k-mooney | which apprenly is not in ubuntu 22.04 | |
| 17:42:17 | melwitt | the other example run has DEBUG ovsdbapp.backend.ovs_idl.vlog [-] 999-ms timeout {{(pid=98090) __log_wakeup /usr/local/lib/python3.10/dist-packages/ovs/poller.py:248}} not sure if that's bad also | |
| 17:42:35 | melwitt | it doesn't have the 4999 | |
| 17:42:50 | sean-k-mooney | yep the exact amount isnt really imporant | |
| 17:43:11 | sean-k-mooney | the timeout is around 5 seconds | |
| 17:43:36 | sean-k-mooney | but you can get reconnnect at shorter intervals depending on what else nova is doing | |
| 17:46:25 | sean-k-mooney | https://review.opendev.org/c/openstack/devstack/+/891101 will revert it | |
| 17:46:46 | melwitt | wonder why I only see it with ceph. a coincidence? | |
| 17:46:58 | sean-k-mooney | probaly im seeing it locally | |
| 17:47:12 | sean-k-mooney | i noticed it while i was looking at somethign for dansmith | |
| 17:47:35 | melwitt | oh ok | |
| 17:52:12 | sean-k-mooney | so i have a 50 second span | |
| 17:52:34 | sean-k-mooney | where the compute agent does not log anything else implying it was blocked for 50 seconds reconnecting ot ovs | |
| 17:52:58 | sean-k-mooney | the problem is i have a version of ovs that apprenly has this fix and a version of ovsdbapp that also has this fix | |
| 17:53:10 | sean-k-mooney | so either the fix does not work or it has regressed again | |
| 17:53:43 | sean-k-mooney | this woudl cause the ci jobs to fail alot but also cause issues for people in production obviously | |
| 17:56:44 | melwitt | sean-k-mooney++ | |
| 17:58:39 | sean-k-mooney | so i have a patch version os-vif, ovsdbapp and ovs python binings and its still happening meaning it looks like maybe it regressed again | |
| 18:00:49 | melwitt | 🤯 | |
| 18:00:49 | melwitt | 🤯 | |
| 18:02:16 | sean-k-mooney | or this is the problem https://github.com/openstack/os-vif/blob/master/vif_plug_ovs/ovsdb/impl_idl.py#L35 | |
| 18:02:37 | sean-k-mooney | this is the ovsdbapp fix | |
| 18:02:39 | sean-k-mooney | https://github.com/openstack/ovsdbapp/blob/master/ovsdbapp/backend/ovs_idl/connection.py#L148-L150 | |
| 18:02:56 | sean-k-mooney | in its subclass of idl.Idl | |
| 18:03:17 | sean-k-mooney | but we are doing " return idl.Idl(conn, helper)" directly usign the one form ovs | |
| 18:04:57 | sean-k-mooney | that should be using the NeutronOvsdbIdl class below i think | |
| 18:35:37 | dansmith | melwitt: oh SNAP | |
| 18:35:47 | dansmith | melwitt: I wonder if the thing with the ceph job is something I broke | |
| 18:36:01 | dansmith | melwitt: https://review.opendev.org/c/openstack/devstack/+/890439 | |
| 18:36:16 | dansmith | I adjusted the interval and the deadline so I think it *should* be okay | |
| 18:36:28 | dansmith | although that wouldn't explain nova just not logging anything after the hang | |
| 18:37:00 | melwitt | yeah.. 🤔 | |