| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-02-05 | |||
| 18:04:16 | sean-k-mooney | https://specs.openstack.org/openstack/nova-specs/specs/stein/implemented/boot-instance-specific-storage-backend.html | |
| 18:04:22 | sean-k-mooney | added in stien yes | |
| 18:05:04 | sean-k-mooney | you can manually specifcy the bdms as a json blob too right? | |
| 18:05:10 | sean-k-mooney | so you can use that to add it | |
| 18:05:47 | mnaser | neat. this cloud is running train so i can take advantage of it | |
| 18:05:54 | mnaser | awesome, thank you so much stephenfin / sean-k-mooney :) | |
| 18:06:08 | sean-k-mooney | so openstack server create --block-device-mapping {stuff} | |
| 18:08:57 | sean-k-mooney | mnaser: im not sure the current osc suport will actully work but you can defeintl do it with nova clinet until stephenfin patch lands | |
| 18:10:22 | sean-k-mooney | mnaser: https://review.opendev.org/c/openstack/python-openstackclient/+/771699 | |
| 18:13:03 | dansmith | stephenfin: I'm seeing all your db rechecks because I'm subscribed to all of them and there are a bunch | |
| 18:13:15 | dansmith | stephenfin: any hunch on what the common fails are | |
| 18:13:25 | dansmith | I feel like I'm seeing a lot of cinder fails these days | |
| 18:17:51 | sean-k-mooney | we have some ocational issue with the ssh verifciaton in some of the cinder tests | |
| 18:18:09 | sean-k-mooney | although i have not seen any one test stand out to me that much | |
| 18:18:13 | dansmith | are the ssh things specific to cinder tests? I didn't think so | |
| 18:18:33 | dansmith | lately when I need to recheck it seems like it's always some volume-related test | |
| 18:18:36 | sean-k-mooney | i have noticed it on some of the volumn ones but i think that just a seperate issue | |
| 18:18:41 | dansmith | usually related to not being able to delete a volume | |
| 18:20:53 | sean-k-mooney | https://review.opendev.org/c/openstack/nova/+/769720 just fialed with image import | |
| 18:21:06 | sean-k-mooney | in the nova-ceph-multistore job | |
| 18:21:42 | sean-k-mooney | and it previously failed in gate with tempest.scenario.test_encrypted_cinder_volumes.TestEncryptedCinderVolumes | |
| 18:22:26 | dansmith | this one is a timeout | |
| 18:23:04 | sean-k-mooney | well not a job time out though so something falied but ya | |
| 18:23:34 | dansmith | yes, job timeout | |
| 18:23:43 | dansmith | stopped in the middle of an otherwise successful-looking tempest | |
| 18:24:16 | sean-k-mooney | https://review.opendev.org/c/openstack/nova/+/769720? | |
| 18:24:29 | dansmith | nova-ceph-multistore TIMED_OUT | |
| 18:24:30 | dansmith | yeah | |
| 18:24:50 | dansmith | RUN END RESULT_TIMED_OUT: [untrusted : opendev.org/openstack/tempest/playbooks/devstack-tempest.yaml@master] | |
| 18:25:06 | sean-k-mooney | thats not what im seeing is says failure for me in the dashborad | |
| 18:25:16 | sean-k-mooney | https://zuul.opendev.org/t/openstack/build/b7ad596d805b4f4fb428a047459d7ed8 | |
| 18:25:26 | dansmith | oh sorry you mean the not yet reported job? | |
| 18:25:47 | sean-k-mooney | no the zull summary tab | |
| 18:26:27 | dansmith | ack, was looking at the original zuul -1 on PS5, I see now | |
| 18:26:54 | dansmith | that report you linked is an actual fail, but unrelated to the nova-glance interaction | |
| 18:27:22 | sean-k-mooney | ya it hit two since test failure in gate and check | |
| 18:27:33 | sean-k-mooney | in different jobs that were unrelated | |
| 18:33:42 | dansmith | that fail doesn't seem to be giving it up to me, but I'd guess our http request to pull the image the test is asking for is stalled | |
| 18:34:59 | openstack | Launchpad bug 1914282 in QEMU "block copy job sometimes hangs on the last block for minutes" [Undecided,New] | |
| 18:34:59 | melwitt | fwiw I have reported a qemu bug recently related to block copy which is used in some volume tests https://bugs.launchpad.net/qemu/+bug/1914282 and this one presents as a volume not reaching the expected state within some timeout (usually 196s) | |
| 18:35:42 | melwitt | *within some timeout in our tempest tests | |
| 18:36:16 | sean-k-mooney | that looks like what happend in one fo those failures | |
| 18:36:45 | melwitt | I'm not sure exactly how often it happens because I haven't figured out a unique fingerprint to logstash for | |
| 18:37:06 | melwitt | but anecdotally I've seen it at least a few times lately | |
| 18:37:46 | openstack | Launchpad bug 1914296 in OpenStack Compute (nova) "tests involving encrypted volumes sometimes (rarely) fail with libvirtError: [...] Volume is not in LUKS format" [Medium,Confirmed] | |
| 18:37:46 | melwitt | this is another volume related one that I've only seen two logstash hits for https://bugs.launchpad.net/nova/+bug/1914296 | |
| 18:38:11 | dansmith | boom urllib.error.URLError: <urlopen error [Errno 110] ETIMEDOUT> | |
| 18:39:12 | openstack | Launchpad bug 1912845 in OpenStack-Gate "504 Gateway Timeout while trying to verify_metadata for a server" [Undecided,New] | |
| 18:39:12 | melwitt | and then some of the ssh fails I've seen are this where instance fails to get pub key from metadata API (in cases where force_config_drive=False) https://bugs.launchpad.net/openstack-gate/+bug/1912845 | |
| 18:39:30 | sean-k-mooney | the encpted volume one failed with "libvirt.libvirtError: internal error: unable to execute QEMU command 'blockdev-add': Volume is not in LUKS format" | |
| 18:39:49 | sean-k-mooney | https://zuul.opendev.org/t/openstack/build/57b34c2d89cb44dc8f716dcc4fdaf792/log/controller/logs/screen-n-cpu.txt#28853 | |
| 18:40:07 | melwitt | yeah, that's one of the 3 gate bugs I reported and linked ^ | |
| 18:40:18 | sean-k-mooney | yep | |
| 18:40:27 | melwitt | I showed that one to lyarwood and he said he's never seen it before | |
| 18:41:01 | sean-k-mooney | thats what faild in the gate queue for the multi-cell-job then the image download swalled in the check queue in the multistore jobs | |
| 18:46:15 | openstack | Launchpad bug 1912031 in Cinder "lvcreate sometimes hangs and does not return, subsequent lvcreate commands hang from then on" [Medium,Triaged] | |
| 18:46:15 | melwitt | oh yeah and I also reported a bug where lvcreate hangs once and then hangs on every attempt to lvcreate after that https://bugs.launchpad.net/cinder/+bug/1912031 | |
| 18:46:38 | melwitt | that also presents as volumes not reaching expected states within time | |
| 18:47:05 | sean-k-mooney | so safe to say there are just one or two volume issues :P | |
| 18:47:31 | melwitt | seems like some kind of internal (in lvm) locking problem where deadlock happens and then things are screwed from then on. I dunno | |
| 18:47:41 | melwitt | yeah, it's a #hotmess | |
| 18:48:43 | sean-k-mooney | ya some of thoes issues are really not cinders fault | |
| 18:49:05 | sean-k-mooney | but it is one of the places we tend to have issue with more offten just because of all the layesr invovled | |
| 18:49:11 | melwitt | yeah, it doesn't look like cinder itself but I thought they'd be most likely to know what's wrong or where to direct the bug next | |
| 18:49:38 | melwitt | or know what could possibly be done about it | |
| 20:24:20 | dansmith | sean-k-mooney: that failed multistore test run yielded a tempest fix, a serious glance bug, and a negative test in tempest to poke the bug | |
| 20:24:22 | dansmith | so... thanks :) | |
| #openstack-nova - 2021-02-06 | |||
| 00:16:48 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: WIP: libvirt: start tracking NUMACell.socket for hosts https://review.opendev.org/c/openstack/nova/+/766816 | |
| 00:16:49 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: WIP: Add `socket` PCI NUMA affinity policy request prefilter https://review.opendev.org/c/openstack/nova/+/772749 | |
| 00:16:49 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: WIP: extra specs/image pros: add `socket` PCI NUMA affinity https://review.opendev.org/c/openstack/nova/+/772748 | |
| 00:16:50 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: WIP: pci: implement the `socket` NUMA affinity policy https://review.opendev.org/c/openstack/nova/+/772779 | |
| 00:16:50 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: WIP: Track host NUMA topology in PCI manager https://review.opendev.org/c/openstack/nova/+/774149 | |
| 01:09:45 | openstackgerrit | Merged openstack/nova master: docs: Move the LibvirtDistroSupportMatrix wiki page into our docs https://review.opendev.org/c/openstack/nova/+/771981 | |
| 01:10:17 | openstackgerrit | Merged openstack/nova master: glance: Remove [glance]/allowed_direct_url_schemes https://review.opendev.org/c/openstack/nova/+/772874 | |
| 06:20:21 | openstackgerrit | Merged openstack/nova master: db: Compact Liberty database migrations https://review.opendev.org/c/openstack/nova/+/758397 | |
| 16:38:17 | openstackgerrit | Merged openstack/nova master: db: Compact Mitaka database migrations https://review.opendev.org/c/openstack/nova/+/758398 | |
| #openstack-nova - 2021-02-07 | |||
| 02:49:12 | openstackgerrit | Wenping Song proposed openstack/nova master: Replace tenants* with projects* of policies https://review.opendev.org/c/openstack/nova/+/765315 | |
| 02:49:12 | openstackgerrit | Wenping Song proposed openstack/nova master: Replaces tenant_id with project_id from List/Show usage APIs https://review.opendev.org/c/openstack/nova/+/768509 | |
| 06:37:06 | openstackgerrit | Merged openstack/nova master: Refactor ResourceRequest constructor https://review.opendev.org/c/openstack/nova/+/769720 | |
| 12:02:35 | openstackgerrit | Merged openstack/nova master: db: Compact Newton database migrations https://review.opendev.org/c/openstack/nova/+/759085 | |
| #openstack-nova - 2021-02-08 | |||
| 00:32:04 | openstackgerrit | Merged openstack/nova master: db: Compact Ocata database migrations https://review.opendev.org/c/openstack/nova/+/759086 | |
| 01:47:05 | swp20 | sean-k-mooney: hi, about the noVNC patch, i think the case in https://review.opendev.org/c/openstack/nova/+/622336/30/nova/tests/unit/api/openstack/compute/test_remote_consoles.py#375 covers what you said. | |
| 03:37:49 | openstackgerrit | MIKI Nobuhiro proposed openstack/nova master: libvirt: add IP address to libvirt guest metadata https://review.opendev.org/c/openstack/nova/+/750552 | |
| 09:38:30 | gibi | lyarwood: Hi! What are the patches we need to merge before we can drop one of th grenade job as in https://review.opendev.org/c/openstack/tempest/+/771499 ? | |
| 09:41:23 | lyarwood | gibi: https://review.opendev.org/c/openstack/nova/+/742056 - I think you're confusing this with that moves the job over to zuulv3 | |
| 09:41:33 | lyarwood | gibi: I think we can drop the standalone job ahead of that | |
| 09:41:53 | lyarwood | gibi: but should still try to push the move to zuulv3 along this cycle | |
| 09:55:51 | gibi | lyarwood: do you mean that we could merge https://review.opendev.org/c/openstack/tempest/+/771499 now without any dependency? | |
| 09:56:09 | stephenfin | dansmith: I'm seeing multiple networks found quite often also. I've tried figuring out how to fix that but my Google-fu is failing me :-\ | |
| 09:56:53 | gibi | stephenfin: I think ther is an open tempest patch for that ^^ let me find it | |
| 09:57:38 | lyarwood | yeah that should be listed on status.openstack.org iirc | |
| 09:58:35 | lyarwood | gibi: I believe so yes, the switch to zuulv3 is seperate AFAICT | |
| 09:58:57 | gibi | stephenfin: https://review.opendev.org/c/openstack/tempest/+/770169 | |
| 09:59:13 | lyarwood | stephenfin: http://status.openstack.org/elastic-recheck/#1844568 | |
| 09:59:17 | gibi | lyarwood: cool, please not that on Dan's patch | |
| 10:00:24 | lyarwood | gibi: hmm I see dansmith's note in the change, he might have reasons for waiting until we've completed the switch | |
| 10:00:49 | lyarwood | gibi: let me sync with him later today, I've likley missing some context on Friday while I was out | |
| 10:01:04 | gibi | lyarwood: cool thanks | |
| 10:05:55 | gibi | gmann: Is there anything I can help with https://review.opendev.org/c/openstack/tempest/+/770169 ? | |
| 10:15:31 | openstackgerrit | Johannes Kulik proposed openstack/nova master: Faults from cell DB missing in GET /servers/detail https://review.opendev.org/c/openstack/nova/+/699176 | |