| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-02-05 | |||
| 18:25:26 | dansmith | oh sorry you mean the not yet reported job? | |
| 18:25:47 | sean-k-mooney | no the zull summary tab | |
| 18:26:27 | dansmith | ack, was looking at the original zuul -1 on PS5, I see now | |
| 18:26:54 | dansmith | that report you linked is an actual fail, but unrelated to the nova-glance interaction | |
| 18:27:22 | sean-k-mooney | ya it hit two since test failure in gate and check | |
| 18:27:33 | sean-k-mooney | in different jobs that were unrelated | |
| 18:33:42 | dansmith | that fail doesn't seem to be giving it up to me, but I'd guess our http request to pull the image the test is asking for is stalled | |
| 18:34:59 | melwitt | fwiw I have reported a qemu bug recently related to block copy which is used in some volume tests https://bugs.launchpad.net/qemu/+bug/1914282 and this one presents as a volume not reaching the expected state within some timeout (usually 196s) | |
| 18:34:59 | openstack | Launchpad bug 1914282 in QEMU "block copy job sometimes hangs on the last block for minutes" [Undecided,New] | |
| 18:35:42 | melwitt | *within some timeout in our tempest tests | |
| 18:36:16 | sean-k-mooney | that looks like what happend in one fo those failures | |
| 18:36:45 | melwitt | I'm not sure exactly how often it happens because I haven't figured out a unique fingerprint to logstash for | |
| 18:37:06 | melwitt | but anecdotally I've seen it at least a few times lately | |
| 18:37:46 | melwitt | this is another volume related one that I've only seen two logstash hits for https://bugs.launchpad.net/nova/+bug/1914296 | |
| 18:37:46 | openstack | Launchpad bug 1914296 in OpenStack Compute (nova) "tests involving encrypted volumes sometimes (rarely) fail with libvirtError: [...] Volume is not in LUKS format" [Medium,Confirmed] | |
| 18:38:11 | dansmith | boom urllib.error.URLError: <urlopen error [Errno 110] ETIMEDOUT> | |
| 18:39:12 | melwitt | and then some of the ssh fails I've seen are this where instance fails to get pub key from metadata API (in cases where force_config_drive=False) https://bugs.launchpad.net/openstack-gate/+bug/1912845 | |
| 18:39:12 | openstack | Launchpad bug 1912845 in OpenStack-Gate "504 Gateway Timeout while trying to verify_metadata for a server" [Undecided,New] | |
| 18:39:30 | sean-k-mooney | the encpted volume one failed with "libvirt.libvirtError: internal error: unable to execute QEMU command 'blockdev-add': Volume is not in LUKS format" | |
| 18:39:49 | sean-k-mooney | https://zuul.opendev.org/t/openstack/build/57b34c2d89cb44dc8f716dcc4fdaf792/log/controller/logs/screen-n-cpu.txt#28853 | |
| 18:40:07 | melwitt | yeah, that's one of the 3 gate bugs I reported and linked ^ | |
| 18:40:18 | sean-k-mooney | yep | |
| 18:40:27 | melwitt | I showed that one to lyarwood and he said he's never seen it before | |
| 18:41:01 | sean-k-mooney | thats what faild in the gate queue for the multi-cell-job then the image download swalled in the check queue in the multistore jobs | |
| 18:46:15 | melwitt | oh yeah and I also reported a bug where lvcreate hangs once and then hangs on every attempt to lvcreate after that https://bugs.launchpad.net/cinder/+bug/1912031 | |
| 18:46:15 | openstack | Launchpad bug 1912031 in Cinder "lvcreate sometimes hangs and does not return, subsequent lvcreate commands hang from then on" [Medium,Triaged] | |
| 18:46:38 | melwitt | that also presents as volumes not reaching expected states within time | |
| 18:47:05 | sean-k-mooney | so safe to say there are just one or two volume issues :P | |
| 18:47:31 | melwitt | seems like some kind of internal (in lvm) locking problem where deadlock happens and then things are screwed from then on. I dunno | |
| 18:47:41 | melwitt | yeah, it's a #hotmess | |
| 18:48:43 | sean-k-mooney | ya some of thoes issues are really not cinders fault | |
| 18:49:05 | sean-k-mooney | but it is one of the places we tend to have issue with more offten just because of all the layesr invovled | |
| 18:49:11 | melwitt | yeah, it doesn't look like cinder itself but I thought they'd be most likely to know what's wrong or where to direct the bug next | |
| 18:49:38 | melwitt | or know what could possibly be done about it | |
| 20:24:20 | dansmith | sean-k-mooney: that failed multistore test run yielded a tempest fix, a serious glance bug, and a negative test in tempest to poke the bug | |
| 20:24:22 | dansmith | so... thanks :) | |
| #openstack-nova - 2021-02-06 | |||
| 00:16:48 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: WIP: libvirt: start tracking NUMACell.socket for hosts https://review.opendev.org/c/openstack/nova/+/766816 | |
| 00:16:49 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: WIP: extra specs/image pros: add `socket` PCI NUMA affinity https://review.opendev.org/c/openstack/nova/+/772748 | |
| 00:16:49 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: WIP: Add `socket` PCI NUMA affinity policy request prefilter https://review.opendev.org/c/openstack/nova/+/772749 | |
| 00:16:50 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: WIP: Track host NUMA topology in PCI manager https://review.opendev.org/c/openstack/nova/+/774149 | |
| 00:16:50 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: WIP: pci: implement the `socket` NUMA affinity policy https://review.opendev.org/c/openstack/nova/+/772779 | |
| 01:09:45 | openstackgerrit | Merged openstack/nova master: docs: Move the LibvirtDistroSupportMatrix wiki page into our docs https://review.opendev.org/c/openstack/nova/+/771981 | |
| 01:10:17 | openstackgerrit | Merged openstack/nova master: glance: Remove [glance]/allowed_direct_url_schemes https://review.opendev.org/c/openstack/nova/+/772874 | |
| 06:20:21 | openstackgerrit | Merged openstack/nova master: db: Compact Liberty database migrations https://review.opendev.org/c/openstack/nova/+/758397 | |
| 16:38:17 | openstackgerrit | Merged openstack/nova master: db: Compact Mitaka database migrations https://review.opendev.org/c/openstack/nova/+/758398 | |
| #openstack-nova - 2021-02-07 | |||
| 02:49:12 | openstackgerrit | Wenping Song proposed openstack/nova master: Replaces tenant_id with project_id from List/Show usage APIs https://review.opendev.org/c/openstack/nova/+/768509 | |
| 02:49:12 | openstackgerrit | Wenping Song proposed openstack/nova master: Replace tenants* with projects* of policies https://review.opendev.org/c/openstack/nova/+/765315 | |
| 06:37:06 | openstackgerrit | Merged openstack/nova master: Refactor ResourceRequest constructor https://review.opendev.org/c/openstack/nova/+/769720 | |
| 12:02:35 | openstackgerrit | Merged openstack/nova master: db: Compact Newton database migrations https://review.opendev.org/c/openstack/nova/+/759085 | |
| #openstack-nova - 2021-02-08 | |||
| 00:32:04 | openstackgerrit | Merged openstack/nova master: db: Compact Ocata database migrations https://review.opendev.org/c/openstack/nova/+/759086 | |
| 01:47:05 | swp20 | sean-k-mooney: hi, about the noVNC patch, i think the case in https://review.opendev.org/c/openstack/nova/+/622336/30/nova/tests/unit/api/openstack/compute/test_remote_consoles.py#375 covers what you said. | |
| 03:37:49 | openstackgerrit | MIKI Nobuhiro proposed openstack/nova master: libvirt: add IP address to libvirt guest metadata https://review.opendev.org/c/openstack/nova/+/750552 | |
| 09:38:30 | gibi | lyarwood: Hi! What are the patches we need to merge before we can drop one of th grenade job as in https://review.opendev.org/c/openstack/tempest/+/771499 ? | |
| 09:41:23 | lyarwood | gibi: https://review.opendev.org/c/openstack/nova/+/742056 - I think you're confusing this with that moves the job over to zuulv3 | |
| 09:41:33 | lyarwood | gibi: I think we can drop the standalone job ahead of that | |
| 09:41:53 | lyarwood | gibi: but should still try to push the move to zuulv3 along this cycle | |
| 09:55:51 | gibi | lyarwood: do you mean that we could merge https://review.opendev.org/c/openstack/tempest/+/771499 now without any dependency? | |
| 09:56:09 | stephenfin | dansmith: I'm seeing multiple networks found quite often also. I've tried figuring out how to fix that but my Google-fu is failing me :-\ | |
| 09:56:53 | gibi | stephenfin: I think ther is an open tempest patch for that ^^ let me find it | |
| 09:57:38 | lyarwood | yeah that should be listed on status.openstack.org iirc | |
| 09:58:35 | lyarwood | gibi: I believe so yes, the switch to zuulv3 is seperate AFAICT | |
| 09:58:57 | gibi | stephenfin: https://review.opendev.org/c/openstack/tempest/+/770169 | |
| 09:59:13 | lyarwood | stephenfin: http://status.openstack.org/elastic-recheck/#1844568 | |
| 09:59:17 | gibi | lyarwood: cool, please not that on Dan's patch | |
| 10:00:24 | lyarwood | gibi: hmm I see dansmith's note in the change, he might have reasons for waiting until we've completed the switch | |
| 10:00:49 | lyarwood | gibi: let me sync with him later today, I've likley missing some context on Friday while I was out | |
| 10:01:04 | gibi | lyarwood: cool thanks | |
| 10:05:55 | gibi | gmann: Is there anything I can help with https://review.opendev.org/c/openstack/tempest/+/770169 ? | |
| 10:15:31 | openstackgerrit | Johannes Kulik proposed openstack/nova master: Faults from cell DB missing in GET /servers/detail https://review.opendev.org/c/openstack/nova/+/699176 | |
| 10:28:17 | kashyap | gibi: This review you of CPU flags (which we'll discuss in the meeting this evening) has a couple of bugs, I'm fixing it as we speak | |
| 10:28:23 | kashyap | [https://review.opendev.org/c/openstack/nova/+/774240] | |
| 10:29:10 | kashyap | It should use lstrip() instead of index slicing; and I'm using the wrong libvirt class in libvirt/driver.py. /me goes to fiddle [after a call] | |
| 10:29:46 | gibi | kashyap: OK. which meeting this evening do you refer to ? | |
| 10:30:22 | kashyap | gibi: Err, is there Nova meeting tonight, or am I dreaming? | |
| 10:30:42 | gibi | nova meetings are every thursday | |
| 10:30:56 | kashyap | Err, right; thx | |
| 11:46:03 | openstackgerrit | Kashyap Chamarthy proposed openstack/nova master: libvirt: Allow disabling CPU flags via `cpu_model_extra_flags` https://review.opendev.org/c/openstack/nova/+/774240 | |
| 12:10:45 | gibi | bauzas: left feedback on your the routed net series | |
| 12:18:55 | openstackgerrit | Kashyap Chamarthy proposed openstack/nova master: libvirt: Allow disabling CPU flags via `cpu_model_extra_flags` https://review.opendev.org/c/openstack/nova/+/774240 | |
| 12:20:11 | kashyap | gibi: Hey, --^ fixed the problems in the above; and it works as intended | |
| 12:25:22 | kashyap | Note to self: missing doc update; and a rel note | |
| 12:29:49 | gibi | kashyap: ack, will check back | |
| 12:46:04 | bauzas | gibi: ack, will look | |
| 13:26:48 | sean-k-mooney | lyarwood: by the way you can proably drop the WIP on you machine_type patch in the next revision | |
| 13:27:15 | sean-k-mooney | ill try an review it when its up but feel free to ping me if i dont | |
| 13:28:18 | lyarwood | sean-k-mooney: yup planning to, just sorting some tests out and writing docs | |
| 13:28:57 | lyarwood | sean-k-mooney: thanks again for the review btw | |
| 13:29:36 | gibi | kashyap: I've left feedback in https://review.opendev.org/c/openstack/nova/+/774240 | |
| 13:29:47 | kashyap | gibi: Thanks; will look | |
| 13:38:28 | bauzas | gibi: shit about the 2 NICs... | |
| 13:38:34 | bauzas | :( | |
| 13:39:01 | bauzas | gibi: unfortunately, we need to express ONE Requestspec for both | |
| 13:39:26 | bauzas | context : https://review.opendev.org/c/openstack/nova/+/749068/12/nova/scheduler/request_filter.py | |
| 13:40:41 | bauzas | but we need to express aggregates requirement something like "(agg1 or agg2) AND agg3" in the same spec | |
| 13:41:51 | bauzas | so, like you said, by calling require_aggs() multiple ways | |
| 13:46:27 | gibi | bauzas: I think we have ONE request spec but we have a list of network requests | |
| 13:47:24 | gibi | and the input in the network request list | |
| 13:47:56 | gibi | and the output is in the Destination | |
| 13:48:13 | gibi | maybe we just need to move the require_aggregate into the for loop | |
| 13:48:24 | gibi | instead of having it after the for loop | |