Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2023-08-07
18:01:18 sean-k-mooney dansmith: +2 i forgot about the service id fielsd in the instance that you added so thanks for checking both the compute nodes and instances
18:09:57 dansmith yep
18:10:05 dansmith thanks
#openstack-nova - 2023-08-08
00:44:06 dansmith tempest fix merged so nova patches can be rechecked now
06:07:42 opendevreview Takashi Kajinami proposed openstack/nova master: Fix missing oslo.versionedobjects library option https://review.opendev.org/c/openstack/nova/+/890718
07:23:53 frickler so this says that the workaround is to be removed when the followup blueprint is completed. the latter has happened, but the removal seems to have been missed? https://opendev.org/openstack/nova/commit/267a40663cd8d0b94bbc5ebda4ece55a45753b64#diff-efaf3dbc9a92a63307e9ffdc70a6c3ebbbd4bb7d
10:14:47 dvo-plv Hello, maybe you will have a time to make new review for our patch https://review.opendev.org/c/openstack/nova/+/876075
15:52:23 opendevreview sean mooney proposed openstack/nova master: use disk caching to hide slow cinder performance https://review.opendev.org/c/openstack/nova/+/889383
15:53:07 opendevreview sean mooney proposed openstack/nova master: use disk caching to hide slow cinder performance https://review.opendev.org/c/openstack/nova/+/889383
15:55:40 sean-k-mooney dansmith: just an fyi ^ now has a depend on against my zram change so we can see if zram + sysctl tuning + ksm + qemu disk caching will actully spead things up
16:04:53 dansmith sean-k-mooney: so my concern with that is that it will increase memory pressure
16:05:35 sean-k-mooney the disk caching in qemu
16:06:27 sean-k-mooney it may yes we could also use cache mode node instead of writeback to explcitly disabel the page cache
16:07:10 sean-k-mooney the way i have it configred it only cached block devs so cinder voluems but perhaps the caching is not needed with the vfat change
16:07:20 dansmith yeah
16:08:09 sean-k-mooney has that merged by the way
16:08:19 dansmith yes
16:08:35 sean-k-mooney cool hopefully that test will fail less
18:13:52 sean-k-mooney dansmith: the zswap change is within the noise of the ci varaiance but its either neutral or slightly better.
18:14:05 dansmith ack
18:14:13 sean-k-mooney the same seams to be true for addign caching that is about to finish
18:14:27 sean-k-mooney i dont really see any indeication fo increase memory pressure
18:14:27 dansmith the variance is much larger these days than it used to be so I don't think we can really know without enabling it on a job or two and monitor it longer term
18:14:53 sean-k-mooney ya
18:16:25 sean-k-mooney most of the jobs i looked at didnt seam to be swappign that hard and then ones that were actully seamed to have more overall free memory then i expected
18:16:54 dansmith the ceph job is usually pretty deep into swap, although not enough to cause problems
18:16:57 dansmith it rarely ooms lately
18:17:40 sean-k-mooney ya i looked at ceph multi store
18:17:42 sean-k-mooney https://zuul.opendev.org/t/openstack/build/66029a86e4bd49d0935021197991a1db/log/controller/logs/screen-memory_tracker.txt
18:17:54 dansmith ack
18:18:05 sean-k-mooney low_point: 244568
18:18:31 sean-k-mooney its usinbg about half the swap with 4125888 free of 8GB
18:19:07 sean-k-mooney that ran for 1 hr 36 mins 12 secs
18:19:14 sean-k-mooney which is on the lower end of normal https://zuul.opendev.org/t/openstack/builds?job_name=nova-ceph-multistore&project=openstack%2Fnova
18:19:40 sean-k-mooney but the job ranges form about 1:30 to 2 hours
18:26:47 dansmith gmann: I've seen timeouts in pep8 and cover today: https://zuul.opendev.org/t/openstack/build/0c92c74e8734426fb3fd05ff67893f60
18:26:56 dansmith I'm sure those have to be related to the node right?
18:27:31 gmann its ovh provider this time, not 100% sure if it is slow node
18:27:42 dansmith welp...
#openstack-nova - 2023-08-09
08:40:41 opendevreview Takashi Kajinami proposed openstack/nova master: conf: Add four new '[libvirt] live_migration_*' options https://review.opendev.org/c/openstack/nova/+/456571
12:20:11 opendevreview Felipe Reyes proposed openstack/nova master: Add RISCV64 QEMU emulation support. https://review.opendev.org/c/openstack/nova/+/889137
#openstack-nova - 2023-08-10
01:01:07 opendevreview melanie witt proposed openstack/nova master: WIP libvirt: Introduce support for rbd with LUKS https://review.opendev.org/c/openstack/nova/+/889912
13:24:02 opendevreview Elod Illes proposed openstack/nova master: Fix tox docs target https://review.opendev.org/c/openstack/nova/+/891033
14:00:13 opendevreview Jimmy McCrory proposed openstack/nova master: Fix typo in placement audit command argument https://review.opendev.org/c/openstack/nova/+/891035
14:27:55 opendevreview Dan Smith proposed openstack/nova master: DNM: Test glance new location API https://review.opendev.org/c/openstack/nova/+/891036
14:56:20 kashyap dansmith: Thank you for doing the recheck work for me! https://review.opendev.org/c/openstack/nova/+/887255
15:28:31 ozzzo_work I have a couple of VMs that are half-created: https://paste.openstack.org/show/b7wqqzgumoH8tdtvmEh1/
15:28:48 ozzzo_work how can I clean this up? I looked in the db and they don't exist in "instances"
15:30:40 dansmith ozzzo_work: check the api database's instance_build_requests table
15:31:04 dansmith I'm guessing you're seeing that because those request-to-build-an-instance entries are there and we never got beyond that
15:31:26 dansmith I'd check the logs to see if you're seeing any fireworks there, because that *should* be resolvable, AFAIK
15:31:56 dansmith any manual DB surgery that has been done?
15:49:37 ozzzo_work oic it was in the nova_cell0 table
15:50:39 ozzzo_work we're not using multiple cells but I guess everything must go in cell0 by default now
16:12:36 dansmith ozzzo_work: dead instances that fail to schedule always go into the cell0 DB
16:12:53 dansmith everyone is using cells, you might just only have one cell of computes :)
17:02:04 melwitt dansmith: have you noticed "no valid host" fails in nova-ceph-multistore and "Seems service nova-compute on host np0034917240 is down. Last heartbeat was 2023-08-10 15:59:29. Elapsed time is 847.603029" messages in the n-sch log?
17:03:00 dansmith I saw one example of that and it looked like rabbit crashed or something because I saw a bunch of amq errors in multiple places
17:04:06 melwitt hm, I hadn't noticed the rabbit correlation
17:06:34 melwitt the couple of examples I saw so far, the n-cpu log just stops ... early. and the nova valid host "returned 0 hosts" messages in n-sch start after the last timestamp seen in n-cpu
17:09:05 melwitt last log message Aug 10 16:00:00.843521 https://zuul.openstack.org/build/e7c306fad21e4057a3af40720049482a/log/controller/logs/screen-n-cpu.txt and then first "ComputeFilter returned 0 hosts" at Aug 10 16:13:36.604315 https://zuul.openstack.org/build/e7c306fad21e4057a3af40720049482a/log/controller/logs/screen-n-sch.txt#3881
17:10:57 dansmith melwitt: you've seen multiple cases of n-cpu just freezing or just the one?
17:11:36 melwitt my first thought was, is it possible n-cpu is being OOM killed or something but I don't see any OOM messages in syslog.txt (assuming that's where they would be)
17:13:42 dansmith yeah I saw no OOM in my case either (n-cpu or otherwise)
17:13:46 dansmith yep, syslog
17:15:36 melwitt I saw two so far, the other one being https://zuul.openstack.org/build/4884721dbf764b33995d71c990f65dfe/logs
17:15:57 melwitt (there's a third but it's in a DNM ceph job I'm using to test ephemeral encryption)
17:16:09 dansmith oh that one was mine, yeah
17:19:04 dansmith melwitt: so in that last one, the last thing nova logs is running a ceph command ... :)
17:19:20 dansmith oh but it returned
17:22:11 melwitt :)
17:22:43 melwitt it does seem to stop at the same place though, "Creating image(s)"
17:27:46 dansmith weird, I dunno what would have changed in the last few days related to this, and I haven't seen this before
17:28:22 melwitt same :(
17:33:36 sean-k-mooney f****
17:34:08 sean-k-mooney DEBUG ovsdbapp.backend.ovs_idl.vlog [-] 4999-ms timeout {{(pid=1435862) __log_wakeup /opt/stack/data/venv/lib/python3.10/site-packages/ovs/poller.py:248}}
17:34:44 sean-k-mooney are we seeing that in ci
17:35:00 sean-k-mooney it looks liek the ovsdbapp thing might be regressed
17:35:10 sean-k-mooney im gong to check my devstack/nova logs
17:35:29 sean-k-mooney thats the issue where ovs blocks nova for 5 seconds at a time
17:36:21 melwitt actually I found one in there https://zuul.openstack.org/build/e7c306fad21e4057a3af40720049482a/log/controller/logs/screen-n-cpu.txt#39523
17:36:23 sean-k-mooney this could be a regression in the devstack in a venv change
17:36:52 sean-k-mooney but my code in devstack to workaround this does not seam to have put the thing in the nova.conf
17:37:11 melwitt it's logged 15 times on that run
17:38:05 sean-k-mooney really..... https://github.com/openstack/devstack/commit/db3eff7dd27acdc973e8d189bda80d642be92f03
17:39:04 sean-k-mooney i need to revert that
17:39:34 sean-k-mooney this is fix only if you use a new enough ovs
17:39:41 sean-k-mooney which apprenly is not in ubuntu 22.04
17:42:17 melwitt the other example run has DEBUG ovsdbapp.backend.ovs_idl.vlog [-] 999-ms timeout {{(pid=98090) __log_wakeup /usr/local/lib/python3.10/dist-packages/ovs/poller.py:248}} not sure if that's bad also
17:42:35 melwitt it doesn't have the 4999
17:42:50 sean-k-mooney yep the exact amount isnt really imporant
17:43:11 sean-k-mooney the timeout is around 5 seconds
17:43:36 sean-k-mooney but you can get reconnnect at shorter intervals depending on what else nova is doing
17:46:25 sean-k-mooney https://review.opendev.org/c/openstack/devstack/+/891101 will revert it
17:46:46 melwitt wonder why I only see it with ceph. a coincidence?
17:46:58 sean-k-mooney probaly im seeing it locally
17:47:12 sean-k-mooney i noticed it while i was looking at somethign for dansmith
17:47:35 melwitt oh ok
17:52:12 sean-k-mooney so i have a 50 second span
17:52:34 sean-k-mooney where the compute agent does not log anything else implying it was blocked for 50 seconds reconnecting ot ovs

Earlier   Later