| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2023-08-02 | |||
| 16:09:59 | stephenfin | but I have no idea how to even start debugging it. My inclination is to disable the test temporarily but it would seem there's _something_ off in the Apache2-uWSGI-placement stack | |
| 16:17:59 | dansmith | stephenfin: I saw one case of it where it looked like apache made one query, which never made it to placement, | |
| 16:18:18 | dansmith | then apache trying to resolve the error document ended up actually making the call to placement which did succeed, | |
| 16:18:26 | dansmith | but it was too late for apache to return the result to the client | |
| 16:18:28 | dansmith | it was strange | |
| 16:19:27 | dansmith | if you can repro it locally I'm definitely less willing to blame an overworked apache | |
| 16:20:52 | stephenfin | it's strange indeed. The fact that it's reproducible locally and popping up frequently means I can't just write it off as an intermittent flake. I guess that's a good thing in the long run, since we'll have to fix it, but not helpful right now 😅 | |
| 16:20:52 | stephenfin | it's strange indeed. The fact that it's reproducible locally and popping up frequently means I can't just write it off as an intermittent flake. I guess that's a good thing in the long run, since we'll have to fix it, but not helpful right now 😅 | |
| 16:21:09 | stephenfin | but yeah, just a devstack vm running in PSI internally | |
| 16:22:51 | dansmith | ack | |
| 21:33:24 | opendevreview | Danylo Vodopianov proposed openstack/nova master: Packed virtqueue support was added. https://review.opendev.org/c/openstack/nova/+/876075 | |
| #openstack-nova - 2023-08-03 | |||
| 00:33:30 | opendevreview | Ghanshyam proposed openstack/nova master: Add service role in nova policy https://review.opendev.org/c/openstack/nova/+/864594 | |
| 04:24:47 | opendevreview | Ghanshyam proposed openstack/nova master: api-ref update: Move the service APIs in a separate section https://review.opendev.org/c/openstack/nova/+/864655 | |
| 08:45:49 | auniyal | Hello @sean-k-mooney, regarding healthcheck patch | |
| 08:45:49 | auniyal | pre_start_hook here https://review.opendev.org/c/openstack/nova/+/827844/7/nova/compute/manager.py#1759 | |
| 08:45:49 | auniyal | failing reason here: https://ce8e360d85abcdeffc92-a619ea89024f9935a8230ca8f397a8a1.ssl.cf1.rackcdn.com/827844/7/check/openstack-tox-cover/e5c590f/testr_results.html | |
| 08:47:02 | auniyal | so I can fix errors, can you please provide me some info/context, how its helpful in healthcheck | |
| 09:06:27 | auniyal | so the reason was to use service.uuid but in BaseTestCase while creating service they set the property as id, | |
| 09:17:54 | auniyal | I believe it should uuid and not id in BaseTestCase, | |
| 09:22:28 | auniyal | https://review.opendev.org/c/openstack/nova/+/827844/7/nova/manager.py | |
| 10:00:38 | opendevreview | Merged openstack/nova stable/wallaby: Fix get_segments_id with subnets without segment_id https://review.opendev.org/c/openstack/nova/+/883726 | |
| 10:06:20 | opendevreview | Amit Uniyal proposed openstack/nova master: [WIP] add healthcheck manager to manager base https://review.opendev.org/c/openstack/nova/+/827844 | |
| 10:06:21 | opendevreview | Amit Uniyal proposed openstack/nova master: [WIP] add healthcheck tracker to nova context https://review.opendev.org/c/openstack/nova/+/829468 | |
| 10:06:21 | opendevreview | Amit Uniyal proposed openstack/nova master: [WIP] add healthcheck utils and constants https://review.opendev.org/c/openstack/nova/+/829469 | |
| 10:06:22 | opendevreview | Amit Uniyal proposed openstack/nova master: add healthcheck endpoint to proxy commands https://review.opendev.org/c/openstack/nova/+/830703 | |
| 11:09:23 | dvo-plv | sean-k-mooney, Hello, I have successfully regression after my yesterday comments resolving. Maybe you will have time to verify it today ? https://review.opendev.org/c/openstack/nova/+/876075 | |
| 11:38:40 | opendevreview | Bence Romsics proposed openstack/nova master: Do not untrack resources of a server being unshelved https://review.opendev.org/c/openstack/nova/+/890406 | |
| #openstack-nova - 2023-08-04 | |||
| 20:57:51 | eandersson | Something going on with nova? been seeing a lot of | |
| 20:57:52 | eandersson | > nova-api did not start | |
| 20:59:13 | dansmith | eandersson: something's going on with everything | |
| 20:59:22 | dansmith | link me and I'll have a look | |
| #openstack-nova - 2023-08-07 | |||
| 07:27:29 | opendevreview | Amit Uniyal proposed openstack/nova master: Added context manager for instance lock https://review.opendev.org/c/openstack/nova/+/873648 | |
| 07:27:30 | opendevreview | Amit Uniyal proposed openstack/nova master: Disconnecting volume from the compute host https://review.opendev.org/c/openstack/nova/+/877446 | |
| 15:45:10 | opendevreview | Amit Uniyal proposed openstack/nova master: [WIP] add initial healthcheck support https://review.opendev.org/c/openstack/nova/+/825015 | |
| 15:45:11 | opendevreview | Amit Uniyal proposed openstack/nova master: [WIP] add healthcheck manager to manager base https://review.opendev.org/c/openstack/nova/+/827844 | |
| 15:45:11 | opendevreview | Amit Uniyal proposed openstack/nova master: [WIP] add healthcheck tracker to nova context https://review.opendev.org/c/openstack/nova/+/829468 | |
| 15:45:12 | opendevreview | Amit Uniyal proposed openstack/nova master: [WIP] add healthcheck utils and constants https://review.opendev.org/c/openstack/nova/+/829469 | |
| 15:45:12 | opendevreview | Amit Uniyal proposed openstack/nova master: add healthcheck endpoint to proxy commands https://review.opendev.org/c/openstack/nova/+/830703 | |
| 16:27:50 | dansmith | FYI, no point in rechecking right now, as there's a glance test that was added which is failing 100% of the time as far as I can tell | |
| 16:28:04 | dansmith | manifests as test_image_tasks_create() failing in the ceph job | |
| 16:36:53 | sean-k-mooney | hum do you knwo why? | |
| 16:37:30 | sean-k-mooney | i would be surpiesed if we did not have a ceph job in the tempest repo or glance project | |
| 16:38:07 | sean-k-mooney | so im kind of wondering how the new test merged. is it config specific? | |
| 16:38:52 | sean-k-mooney | oh your talking about it on #openstack-qa | |
| 17:37:47 | dansmith | sean-k-mooney: reminder that this is still waiting for your review: https://review.opendev.org/c/openstack/nova/+/889740 | |
| 17:38:03 | dansmith | no point in +Wing it now but if you +2 it I can +W it once that skip passes the gaste | |
| 17:38:06 | dansmith | or the gate even | |
| 17:39:32 | sean-k-mooney | oh ya sorry ill review it shortly | |
| 17:39:54 | sean-k-mooney | and ack ill leave the +w to your or other when the ci is unblocked | |
| 17:45:09 | dansmith | thanks | |
| 18:01:18 | sean-k-mooney | dansmith: +2 i forgot about the service id fielsd in the instance that you added so thanks for checking both the compute nodes and instances | |
| 18:09:57 | dansmith | yep | |
| 18:10:05 | dansmith | thanks | |
| #openstack-nova - 2023-08-08 | |||
| 00:44:06 | dansmith | tempest fix merged so nova patches can be rechecked now | |
| 06:07:42 | opendevreview | Takashi Kajinami proposed openstack/nova master: Fix missing oslo.versionedobjects library option https://review.opendev.org/c/openstack/nova/+/890718 | |
| 07:23:53 | frickler | so this says that the workaround is to be removed when the followup blueprint is completed. the latter has happened, but the removal seems to have been missed? https://opendev.org/openstack/nova/commit/267a40663cd8d0b94bbc5ebda4ece55a45753b64#diff-efaf3dbc9a92a63307e9ffdc70a6c3ebbbd4bb7d | |
| 10:14:47 | dvo-plv | Hello, maybe you will have a time to make new review for our patch https://review.opendev.org/c/openstack/nova/+/876075 | |
| 15:52:23 | opendevreview | sean mooney proposed openstack/nova master: use disk caching to hide slow cinder performance https://review.opendev.org/c/openstack/nova/+/889383 | |
| 15:53:07 | opendevreview | sean mooney proposed openstack/nova master: use disk caching to hide slow cinder performance https://review.opendev.org/c/openstack/nova/+/889383 | |
| 15:55:40 | sean-k-mooney | dansmith: just an fyi ^ now has a depend on against my zram change so we can see if zram + sysctl tuning + ksm + qemu disk caching will actully spead things up | |
| 16:04:53 | dansmith | sean-k-mooney: so my concern with that is that it will increase memory pressure | |
| 16:05:35 | sean-k-mooney | the disk caching in qemu | |
| 16:06:27 | sean-k-mooney | it may yes we could also use cache mode node instead of writeback to explcitly disabel the page cache | |
| 16:07:10 | sean-k-mooney | the way i have it configred it only cached block devs so cinder voluems but perhaps the caching is not needed with the vfat change | |
| 16:07:20 | dansmith | yeah | |
| 16:08:09 | sean-k-mooney | has that merged by the way | |
| 16:08:19 | dansmith | yes | |
| 16:08:35 | sean-k-mooney | cool hopefully that test will fail less | |
| 18:13:52 | sean-k-mooney | dansmith: the zswap change is within the noise of the ci varaiance but its either neutral or slightly better. | |
| 18:14:05 | dansmith | ack | |
| 18:14:13 | sean-k-mooney | the same seams to be true for addign caching that is about to finish | |
| 18:14:27 | sean-k-mooney | i dont really see any indeication fo increase memory pressure | |
| 18:14:27 | dansmith | the variance is much larger these days than it used to be so I don't think we can really know without enabling it on a job or two and monitor it longer term | |
| 18:14:53 | sean-k-mooney | ya | |
| 18:16:25 | sean-k-mooney | most of the jobs i looked at didnt seam to be swappign that hard and then ones that were actully seamed to have more overall free memory then i expected | |
| 18:16:54 | dansmith | the ceph job is usually pretty deep into swap, although not enough to cause problems | |
| 18:16:57 | dansmith | it rarely ooms lately | |
| 18:17:40 | sean-k-mooney | ya i looked at ceph multi store | |
| 18:17:42 | sean-k-mooney | https://zuul.opendev.org/t/openstack/build/66029a86e4bd49d0935021197991a1db/log/controller/logs/screen-memory_tracker.txt | |
| 18:17:54 | dansmith | ack | |
| 18:18:05 | sean-k-mooney | low_point: 244568 | |
| 18:18:31 | sean-k-mooney | its usinbg about half the swap with 4125888 free of 8GB | |
| 18:19:07 | sean-k-mooney | that ran for 1 hr 36 mins 12 secs | |
| 18:19:14 | sean-k-mooney | which is on the lower end of normal https://zuul.opendev.org/t/openstack/builds?job_name=nova-ceph-multistore&project=openstack%2Fnova | |
| 18:19:40 | sean-k-mooney | but the job ranges form about 1:30 to 2 hours | |
| 18:26:47 | dansmith | gmann: I've seen timeouts in pep8 and cover today: https://zuul.opendev.org/t/openstack/build/0c92c74e8734426fb3fd05ff67893f60 | |
| 18:26:56 | dansmith | I'm sure those have to be related to the node right? | |
| 18:27:31 | gmann | its ovh provider this time, not 100% sure if it is slow node | |
| 18:27:42 | dansmith | welp... | |
| #openstack-nova - 2023-08-09 | |||
| 08:40:41 | opendevreview | Takashi Kajinami proposed openstack/nova master: conf: Add four new '[libvirt] live_migration_*' options https://review.opendev.org/c/openstack/nova/+/456571 | |
| 12:20:11 | opendevreview | Felipe Reyes proposed openstack/nova master: Add RISCV64 QEMU emulation support. https://review.opendev.org/c/openstack/nova/+/889137 | |
| #openstack-nova - 2023-08-10 | |||
| 01:01:07 | opendevreview | melanie witt proposed openstack/nova master: WIP libvirt: Introduce support for rbd with LUKS https://review.opendev.org/c/openstack/nova/+/889912 | |
| 13:24:02 | opendevreview | Elod Illes proposed openstack/nova master: Fix tox docs target https://review.opendev.org/c/openstack/nova/+/891033 | |
| 14:00:13 | opendevreview | Jimmy McCrory proposed openstack/nova master: Fix typo in placement audit command argument https://review.opendev.org/c/openstack/nova/+/891035 | |
| 14:27:55 | opendevreview | Dan Smith proposed openstack/nova master: DNM: Test glance new location API https://review.opendev.org/c/openstack/nova/+/891036 | |
| 14:56:20 | kashyap | dansmith: Thank you for doing the recheck work for me! https://review.opendev.org/c/openstack/nova/+/887255 | |
| 15:28:31 | ozzzo_work | I have a couple of VMs that are half-created: https://paste.openstack.org/show/b7wqqzgumoH8tdtvmEh1/ | |
| 15:28:48 | ozzzo_work | how can I clean this up? I looked in the db and they don't exist in "instances" | |
| 15:30:40 | dansmith | ozzzo_work: check the api database's instance_build_requests table | |
| 15:31:04 | dansmith | I'm guessing you're seeing that because those request-to-build-an-instance entries are there and we never got beyond that | |