| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2023-08-10 | |||
| 18:36:01 | dansmith | melwitt: https://review.opendev.org/c/openstack/devstack/+/890439 | |
| 18:36:16 | dansmith | I adjusted the interval and the deadline so I think it *should* be okay | |
| 18:36:28 | dansmith | although that wouldn't explain nova just not logging anything after the hang | |
| 18:37:00 | melwitt | yeah.. 🤔 | |
| 18:37:00 | melwitt | yeah.. 🤔 | |
| 18:54:24 | sean-k-mooney | ok the issue i mentioned might be fixed | |
| 18:55:29 | sean-k-mooney | i configured my keepalive interval in the kernel to 10 secodn i think but ovs is expecting probes every 5 seconds | |
| 18:55:48 | melwitt | yeah, I was just looking at it again and n-cpu hadn't reported a heartbeat in 14 minutes and never came back after the hang | |
| 18:57:15 | melwitt | bizarre | |
| 19:00:13 | melwitt | libvirt log activity also stops for the most part at the same time n-cpu stops https://zuul.openstack.org/build/e7c306fad21e4057a3af40720049482a/log/controller/logs/libvirt/libvirt/libvirtd_log.txt | |
| 19:00:52 | melwitt | not sure if anything near the end of that log is unusual or problematic | |
| 21:16:31 | melwitt | finally found that on those failures, at the same time that n-cpu freezes, ceph starts logging WRN (warn) health status such as "osd.0 47 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.15524.0:32 4.1e 4:7cf325a3:::rbd_header.3c6ebb6b8706:head [watch ping cookie 140317856977664] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)" | |
| 21:16:45 | melwitt | example https://zuul.openstack.org/build/4884721dbf764b33995d71c990f65dfe/log/controller/logs/ceph/ceph/7f57d687-6678-46f4-ae09-7e6faa476d1a/ceph-osd.0_log.txt#11752 | |
| 21:17:35 | melwitt | log_channel(cluster) log [WRN] : 1 slow requests (by type [ 'delayed' : 1 ] most affected pool [ 'vms' : 1 ]) | |
| 22:50:41 | opendevreview | melanie witt proposed openstack/nova master: DNM test ephemeral encryption + resize: qcow2, raw, rbd https://review.opendev.org/c/openstack/nova/+/862416 | |
| #openstack-nova - 2023-08-11 | |||
| 08:19:13 | auniyal | Hey gibi, can you please have a look on https://review.opendev.org/c/openstack/nova/+/882920 mostly good just a backpot for yoga | |
| 09:03:06 | tobias-urdin | two small doc/log fixes when somebody has some seconds to review https://review.opendev.org/c/openstack/nova/+/880819 https://review.opendev.org/c/openstack/nova/+/873846 | |
| 09:22:33 | opendevreview | Danylo Vodopianov proposed openstack/nova master: Napatech SmartNIC support https://review.opendev.org/c/openstack/nova/+/859577 | |
| 09:24:29 | opendevreview | Danylo Vodopianov proposed openstack/os-vif master: Openvswitch driver was extended https://review.opendev.org/c/openstack/os-vif/+/859574 | |
| 09:24:44 | tobias-urdin | another one https://review.opendev.org/c/openstack/nova/+/874083 | |
| 10:20:21 | auniyal | thanks gibi | |
| 11:33:32 | opendevreview | Danylo Vodopianov proposed openstack/os-vif master: Openvswitch driver was extended https://review.opendev.org/c/openstack/os-vif/+/859574 | |
| 11:53:33 | opendevreview | Merged openstack/nova stable/yoga: Have host look for CPU controller of cgroupsv2 location. https://review.opendev.org/c/openstack/nova/+/882920 | |
| 12:54:55 | opendevreview | Viktor Křivák proposed openstack/nova-specs master: Add availability zone-aware affinity filter to the scheduler https://review.opendev.org/c/openstack/nova-specs/+/890779 | |
| 13:15:19 | opendevreview | Viktor Křivák proposed openstack/nova-specs master: Add availability zone-aware affinity filter to the scheduler https://review.opendev.org/c/openstack/nova-specs/+/890779 | |
| 14:56:23 | opendevreview | Dan Smith proposed openstack/nova master: WIP: Support glance's new location API https://review.opendev.org/c/openstack/nova/+/891036 | |
| 14:56:52 | dansmith | abhishekk: ^ untested, but that's part of your patch but with some new-or-old logic in it and less change overall as I think we can just ignore the return from that operation | |
| 14:57:10 | dansmith | we'll see how that goes, not sure I'll have a change to test locally today but we can iterate from that | |
| 14:57:25 | abhishekk | looking | |
| 15:00:18 | abhishekk | I didn't understood raise exc part at line #571 | |
| 15:03:09 | dansmith | abhishekk: oh that was supposed to end up under the badrequest thing | |
| 15:03:20 | dansmith | ah right but I need to test | |
| 15:03:45 | dansmith | so the goal is to try each of those interfaces, but if neither work, raise the error we got from the last one | |
| 15:04:23 | dansmith | so we might be able to just raise the badrequest whenever we get it, but I need to test glance's behavior with each of the new stuff missing and the old stuff disabled | |
| 15:04:49 | abhishekk | ack, so you need another plain Exception block as well | |
| 15:04:50 | abhishekk | ? | |
| 15:04:52 | dansmith | so that _reraise was supposed to be a pass and replace the raise below | |
| 15:05:06 | dansmith | no, I think we want to try both and raise the last one | |
| 15:05:34 | dansmith | and we need to catch whatever expected exceptions will come from trying the new interface if it's missing | |
| 15:05:36 | abhishekk | ack, because what I understood is lien #571 might express as raise None | |
| 15:05:40 | dansmith | should just be notfound right? | |
| 15:06:14 | dansmith | oh, two notfounds could raise none, yeah you're right | |
| 15:06:25 | dansmith | anyway, like i said, untested, just kinda what I'm thinking :) | |
| 15:06:37 | abhishekk | :D | |
| 15:07:21 | abhishekk | I am not sure client will be updated today to raise Notfound if version is less than 2.17 | |
| 15:07:43 | dansmith | if glance is not updated, the URL should generate a notfound, no? | |
| 15:08:02 | dansmith | you think glanceclient will translate that into something else? | |
| 15:08:42 | abhishekk | NO, as you added depends on in commit message, it will call add_image_location, right? | |
| 15:09:05 | dansmith | right, but we need to handle the case where glance is not upgraded | |
| 15:09:28 | abhishekk | yes, | |
| 15:09:45 | dansmith | so actually, let me change this so it's clearer | |
| 15:09:59 | abhishekk | that needs to be there in client patch, but since your patch has glance and client dependency it will call new API i guess | |
| 15:10:55 | opendevreview | Dan Smith proposed openstack/nova master: DNM: Test glance new location api https://review.opendev.org/c/openstack/nova/+/891207 | |
| 15:10:55 | opendevreview | Dan Smith proposed openstack/nova master: WIP: Support glance's new location API https://review.opendev.org/c/openstack/nova/+/891036 | |
| 15:11:00 | dansmith | this stack needs to work ^ | |
| 15:11:17 | dansmith | er, well, actually, | |
| 15:11:32 | dansmith | the glanceclient depends-on should go in the first patch I guess. | |
| 15:11:40 | dansmith | we can require glanceclient version, but we can't require glance version | |
| 15:11:50 | abhishekk | right | |
| 15:13:58 | opendevreview | Dan Smith proposed openstack/nova master: DNM: Test glance new location api https://review.opendev.org/c/openstack/nova/+/891207 | |
| 15:13:58 | opendevreview | Dan Smith proposed openstack/nova master: WIP: Support glance's new location API https://review.opendev.org/c/openstack/nova/+/891036 | |
| 15:14:19 | dansmith | first depends on glanceclient, second depends on glance | |
| 15:17:03 | abhishekk | This looks good now | |
| 15:17:23 | abhishekk | thank you for your time !! | |
| 15:17:40 | dansmith | ack, I'll try to poke more at it on monday | |
| 15:17:56 | abhishekk | Cool, will be around as well | |
| 17:35:52 | abhishekk | dansmith, nova-ceph-multistore passed on both the patches | |
| 17:36:55 | dansmith | yeah I saw | |
| 17:37:09 | dansmith | did you confirm in the glance log that it's hitting the new one? | |
| 17:40:51 | abhishekk | looking for it now, | |
| 17:40:58 | abhishekk | https://zuul.opendev.org/t/openstack/build/9c045d22dee34b08bb8767a1f972db09/log/job-output.txt#22303 this is the test right? | |
| 17:42:08 | dansmith | I see location_import task running so I assume that means it is | |
| 17:42:22 | dansmith | abhishekk: any server snapshot task should trigger it | |
| 17:42:38 | abhishekk | yeah | |
| 17:42:40 | abhishekk | me too | |
| 17:44:07 | abhishekk | location_import-CalculateHash | |
| 17:45:11 | abhishekk | at least some positive signs :D | |
| 17:47:06 | dansmith | yup, I definitely expected more fails for sure | |
| 17:52:41 | abhishekk | on that note, I am stepping in the weekend mood | |
| 17:53:02 | dansmith | enjoy ;) | |
| 17:53:07 | abhishekk | See you monday, happy weekend o/~ | |
| 19:42:59 | opendevreview | Ghanshyam proposed openstack/nova master: DNM: testing py3.11 job https://review.opendev.org/c/openstack/nova/+/891228 | |
| 19:43:31 | opendevreview | Ghanshyam proposed openstack/nova stable/2023.1: DNM: testing py3.11 job not running on 2023.1 template https://review.opendev.org/c/openstack/nova/+/891113 | |
| 20:43:23 | opendevreview | Ghanshyam proposed openstack/nova master: DNM: testing some zuul template config https://review.opendev.org/c/openstack/nova/+/891235 | |
| 20:43:41 | opendevreview | Ghanshyam proposed openstack/nova stable/2023.1: DNM: testing some zuul template config https://review.opendev.org/c/openstack/nova/+/891114 | |
| 20:50:07 | opendevreview | melanie witt proposed openstack/nova master: Use futurist.ThreadPoolExecutor in scatter_gather_cells https://review.opendev.org/c/openstack/nova/+/650172 | |
| 21:14:23 | opendevreview | Ghanshyam proposed openstack/nova master: DNM: testing some zuul template config https://review.opendev.org/c/openstack/nova/+/891235 | |
| 21:14:37 | opendevreview | Ghanshyam proposed openstack/nova stable/2023.1: DNM: testing some zuul template config https://review.opendev.org/c/openstack/nova/+/891114 | |
| 21:34:40 | opendevreview | Takashi Natsume proposed openstack/nova master: Update contributor guide for 2023.2 Bobcat https://review.opendev.org/c/openstack/nova/+/876447 | |
| #openstack-nova - 2023-08-12 | |||
| 00:39:42 | opendevreview | Jay Faulkner proposed openstack/nova master: DNM: Testing against oslo.messaging change https://review.opendev.org/c/openstack/nova/+/891245 | |
| 01:03:12 | opendevreview | melanie witt proposed openstack/nova master: Use OSC in run-evacuate-hook instead of novaclient https://review.opendev.org/c/openstack/nova/+/891247 | |
| 01:08:33 | melwitt | gmann, dansmith: ^ fix (hopefully) for nova-live-migration, nova-live-migration-ceph, and nova-ovs-hybrid-plug POST_FAILURE | |
| 01:12:09 | gmann | melwitt: thanks, I hope we do not need this devstack fix/workaround after your fix right? https://review.opendev.org/c/openstack/devstack/+/891230 | |
| 01:12:46 | gmann | melwitt: devstack one failed in gate so thinking if your fix solve the issue then we do not need to recheck that | |
| 01:17:49 | melwitt | gmann: oh, I had missed that patch, sorry. I only looked for nova patches 😣 yeah we shouldn't need it after my patch (but let's see if it passes 😛) but either way we can land the devstack one and if my patch works we can revert later | |
| 01:17:49 | melwitt | gmann: oh, I had missed that patch, sorry. I only looked for nova patches 😣 yeah we shouldn't need it after my patch (but let's see if it passes 😛) but either way we can land the devstack one and if my patch works we can revert later | |
| 01:27:42 | gmann | melwitt: ok, sounds good. waiting for gate result | |
| 01:43:56 | gmann | melwitt: you missed to pass server in https://review.opendev.org/c/openstack/nova/+/891247/1/roles/run-evacuate-hook/files/test_negative_evacuate.sh#b20 | |
| 01:44:10 | gmann | it is failing in live migration job https://zuul.opendev.org/t/openstack/build/aef77238f1d24785825165f8bd2ca913/console | |
| 01:44:12 | opendevreview | melanie witt proposed openstack/nova master: Use OSC in run-evacuate-hook instead of novaclient https://review.opendev.org/c/openstack/nova/+/891247 | |