| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-02-05 | |||
| 15:06:58 | kashyap | Affects upstream too | |
| 15:07:36 | sean-k-mooney | to be clear this wont acutlly help improve upgrade without a vm reboot | |
| 15:07:56 | sean-k-mooney | anyway ya i should join that too | |
| 15:08:01 | sean-k-mooney | the call | |
| 15:09:20 | kashyap | sean-k-mooney: It won't; but let's not get bogged down into details, please :) | |
| 15:12:31 | kashyap | gibi: In short, that fix (which allows selectively disabling CPU flags) helps migrating instances from any Intel host that doesn't have "TSX" to a destination host that has "TSX" | |
| 15:20:39 | gibi | kashyap: lets bring it up next week on the nova meeting. I will review your patch now. If next week we see that it has a positive review and close to merge then I can accept to approve the bp late and at the same time merge the code and forget all about it | |
| 15:25:58 | kashyap | gibi: Thank you; sure. | |
| 15:31:19 | openstackgerrit | Stephen Finucane proposed openstack/nova master: policy: Copy rules before providing them to enforcer https://review.opendev.org/c/openstack/nova/+/774252 | |
| 15:31:23 | stephenfin | gmann: ^ | |
| 15:31:35 | stephenfin | Just to be safe | |
| 15:33:36 | artom | Wait, is making a field nullable in an object mean a version bump? | |
| 15:33:49 | artom | Wait, no, ignore me | |
| 15:51:15 | sean-k-mooney | gmann: is https://opendev.org/openstack/openstack-tempest-skiplist new? | |
| 15:51:41 | sean-k-mooney | i dont recall seeing this before i assuem we do not use it in nova? | |
| 15:52:48 | sean-k-mooney | ah this is a ooo thing | |
| 15:53:36 | sean-k-mooney | thats fine i was concerned that we would be skipping tests without knowing it just be cause ooo or another poejct hit an issue | |
| 15:54:04 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: WIP: libvirt: start tracking NUMACell.socket for hosts https://review.opendev.org/c/openstack/nova/+/766816 | |
| 15:54:07 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: WIP: extra specs/image pros: add `socket` PCI NUMA affinity https://review.opendev.org/c/openstack/nova/+/772748 | |
| 15:54:09 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: WIP: Add `socket` PCI NUMA affinity policy request prefilter https://review.opendev.org/c/openstack/nova/+/772749 | |
| 15:54:12 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: WIP: Track host NUMA topology in PCI manager https://review.opendev.org/c/openstack/nova/+/774149 | |
| 15:54:16 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: WIP: pci: implement the `socket` NUMA affinity policy https://review.opendev.org/c/openstack/nova/+/772779 | |
| 16:32:40 | stephenfin | gmann: I removed that legacy PolicyFixture. 482 unit test failures. Haven't run functional tests yet. This is going to take some work :-) | |
| 16:48:27 | kashyap | gibi: Thanks for the quick review. Yes, let's talk on the Nova meeting week | |
| 17:57:05 | mnaser | anyone know off the top of their head if you can specify volume type when using bfv (so nova creating the volume?) | |
| 17:59:25 | sean-k-mooney | mnaser: via the block device mappings i think that came up at some point | |
| 17:59:40 | mnaser | sean-k-mooney: yeah, i'm looking at the code to see if that is currently possible | |
| 18:00:13 | mnaser | > If you want to create a volume to a specific storage backend, you need to use an image which has cinder_img_volume_type property. In this case, a new volume will be created as storage_backend1 volume type. | |
| 18:00:45 | sean-k-mooney | mnaser: we talked about it at the ptg but i cant recally if we said yes or not | |
| 18:01:05 | sean-k-mooney | i know we were relutant to support it as we did not want to keep proxing thigns to other services | |
| 18:01:13 | sean-k-mooney | but we may have allowed this | |
| 18:01:17 | stephenfin | mnaser: yes, you can | |
| 18:02:10 | mnaser | > Microversion 2.67 adds the optional parameter ``volume_type`` to block_device_mapping_v2, which can be used to specify ``volume_type`` when creating a server. | |
| 18:02:11 | mnaser | aha! | |
| 18:03:22 | stephenfin | mnaser: If you're using novaclient, you can pass the 'volume_type' key to the '--block-device' parameter | |
| 18:03:30 | stephenfin | (of 'nova boot', of course) | |
| 18:04:16 | sean-k-mooney | https://specs.openstack.org/openstack/nova-specs/specs/stein/implemented/boot-instance-specific-storage-backend.html | |
| 18:04:22 | sean-k-mooney | added in stien yes | |
| 18:05:04 | sean-k-mooney | you can manually specifcy the bdms as a json blob too right? | |
| 18:05:10 | sean-k-mooney | so you can use that to add it | |
| 18:05:47 | mnaser | neat. this cloud is running train so i can take advantage of it | |
| 18:05:54 | mnaser | awesome, thank you so much stephenfin / sean-k-mooney :) | |
| 18:06:08 | sean-k-mooney | so openstack server create --block-device-mapping {stuff} | |
| 18:08:57 | sean-k-mooney | mnaser: im not sure the current osc suport will actully work but you can defeintl do it with nova clinet until stephenfin patch lands | |
| 18:10:22 | sean-k-mooney | mnaser: https://review.opendev.org/c/openstack/python-openstackclient/+/771699 | |
| 18:13:03 | dansmith | stephenfin: I'm seeing all your db rechecks because I'm subscribed to all of them and there are a bunch | |
| 18:13:15 | dansmith | stephenfin: any hunch on what the common fails are | |
| 18:13:25 | dansmith | I feel like I'm seeing a lot of cinder fails these days | |
| 18:17:51 | sean-k-mooney | we have some ocational issue with the ssh verifciaton in some of the cinder tests | |
| 18:18:09 | sean-k-mooney | although i have not seen any one test stand out to me that much | |
| 18:18:13 | dansmith | are the ssh things specific to cinder tests? I didn't think so | |
| 18:18:33 | dansmith | lately when I need to recheck it seems like it's always some volume-related test | |
| 18:18:36 | sean-k-mooney | i have noticed it on some of the volumn ones but i think that just a seperate issue | |
| 18:18:41 | dansmith | usually related to not being able to delete a volume | |
| 18:20:53 | sean-k-mooney | https://review.opendev.org/c/openstack/nova/+/769720 just fialed with image import | |
| 18:21:06 | sean-k-mooney | in the nova-ceph-multistore job | |
| 18:21:42 | sean-k-mooney | and it previously failed in gate with tempest.scenario.test_encrypted_cinder_volumes.TestEncryptedCinderVolumes | |
| 18:22:26 | dansmith | this one is a timeout | |
| 18:23:04 | sean-k-mooney | well not a job time out though so something falied but ya | |
| 18:23:34 | dansmith | yes, job timeout | |
| 18:23:43 | dansmith | stopped in the middle of an otherwise successful-looking tempest | |
| 18:24:16 | sean-k-mooney | https://review.opendev.org/c/openstack/nova/+/769720? | |
| 18:24:29 | dansmith | nova-ceph-multistore TIMED_OUT | |
| 18:24:30 | dansmith | yeah | |
| 18:24:50 | dansmith | RUN END RESULT_TIMED_OUT: [untrusted : opendev.org/openstack/tempest/playbooks/devstack-tempest.yaml@master] | |
| 18:25:06 | sean-k-mooney | thats not what im seeing is says failure for me in the dashborad | |
| 18:25:16 | sean-k-mooney | https://zuul.opendev.org/t/openstack/build/b7ad596d805b4f4fb428a047459d7ed8 | |
| 18:25:26 | dansmith | oh sorry you mean the not yet reported job? | |
| 18:25:47 | sean-k-mooney | no the zull summary tab | |
| 18:26:27 | dansmith | ack, was looking at the original zuul -1 on PS5, I see now | |
| 18:26:54 | dansmith | that report you linked is an actual fail, but unrelated to the nova-glance interaction | |
| 18:27:22 | sean-k-mooney | ya it hit two since test failure in gate and check | |
| 18:27:33 | sean-k-mooney | in different jobs that were unrelated | |
| 18:33:42 | dansmith | that fail doesn't seem to be giving it up to me, but I'd guess our http request to pull the image the test is asking for is stalled | |
| 18:34:59 | melwitt | fwiw I have reported a qemu bug recently related to block copy which is used in some volume tests https://bugs.launchpad.net/qemu/+bug/1914282 and this one presents as a volume not reaching the expected state within some timeout (usually 196s) | |
| 18:34:59 | openstack | Launchpad bug 1914282 in QEMU "block copy job sometimes hangs on the last block for minutes" [Undecided,New] | |
| 18:35:42 | melwitt | *within some timeout in our tempest tests | |
| 18:36:16 | sean-k-mooney | that looks like what happend in one fo those failures | |
| 18:36:45 | melwitt | I'm not sure exactly how often it happens because I haven't figured out a unique fingerprint to logstash for | |
| 18:37:06 | melwitt | but anecdotally I've seen it at least a few times lately | |
| 18:37:46 | melwitt | this is another volume related one that I've only seen two logstash hits for https://bugs.launchpad.net/nova/+bug/1914296 | |
| 18:37:46 | openstack | Launchpad bug 1914296 in OpenStack Compute (nova) "tests involving encrypted volumes sometimes (rarely) fail with libvirtError: [...] Volume is not in LUKS format" [Medium,Confirmed] | |
| 18:38:11 | dansmith | boom urllib.error.URLError: <urlopen error [Errno 110] ETIMEDOUT> | |
| 18:39:12 | melwitt | and then some of the ssh fails I've seen are this where instance fails to get pub key from metadata API (in cases where force_config_drive=False) https://bugs.launchpad.net/openstack-gate/+bug/1912845 | |
| 18:39:12 | openstack | Launchpad bug 1912845 in OpenStack-Gate "504 Gateway Timeout while trying to verify_metadata for a server" [Undecided,New] | |
| 18:39:30 | sean-k-mooney | the encpted volume one failed with "libvirt.libvirtError: internal error: unable to execute QEMU command 'blockdev-add': Volume is not in LUKS format" | |
| 18:39:49 | sean-k-mooney | https://zuul.opendev.org/t/openstack/build/57b34c2d89cb44dc8f716dcc4fdaf792/log/controller/logs/screen-n-cpu.txt#28853 | |
| 18:40:07 | melwitt | yeah, that's one of the 3 gate bugs I reported and linked ^ | |
| 18:40:18 | sean-k-mooney | yep | |
| 18:40:27 | melwitt | I showed that one to lyarwood and he said he's never seen it before | |
| 18:41:01 | sean-k-mooney | thats what faild in the gate queue for the multi-cell-job then the image download swalled in the check queue in the multistore jobs | |
| 18:46:15 | melwitt | oh yeah and I also reported a bug where lvcreate hangs once and then hangs on every attempt to lvcreate after that https://bugs.launchpad.net/cinder/+bug/1912031 | |
| 18:46:15 | openstack | Launchpad bug 1912031 in Cinder "lvcreate sometimes hangs and does not return, subsequent lvcreate commands hang from then on" [Medium,Triaged] | |
| 18:46:38 | melwitt | that also presents as volumes not reaching expected states within time | |
| 18:47:05 | sean-k-mooney | so safe to say there are just one or two volume issues :P | |
| 18:47:31 | melwitt | seems like some kind of internal (in lvm) locking problem where deadlock happens and then things are screwed from then on. I dunno | |
| 18:47:41 | melwitt | yeah, it's a #hotmess | |
| 18:48:43 | sean-k-mooney | ya some of thoes issues are really not cinders fault | |
| 18:49:05 | sean-k-mooney | but it is one of the places we tend to have issue with more offten just because of all the layesr invovled | |
| 18:49:11 | melwitt | yeah, it doesn't look like cinder itself but I thought they'd be most likely to know what's wrong or where to direct the bug next | |