| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2022-03-04 | |||
| 19:27:07 | noonedeadpunk | hey there! I know it's friday evening, but I was wondering if you see https://review.opendev.org/c/openstack/nova/+/822749 as appropriate backport to stable branches? | |
| 19:27:53 | noonedeadpunk | as powervm is really pita on older releases where you have py3.6 | |
| 19:35:32 | noonedeadpunk | or well, should likely patch u-c instead to map pypowervm back to 1.1.24 where it was at least installing | |
| 20:41:25 | elodilles | noonedeadpunk: req changes are not allowed to backport, but as I see it actually does not change them, just moves them to setup.cfg's [extra] section, so I think this is OK to backport | |
| 23:00:40 | opendevreview | Mauricio Faria de Oliveira proposed openstack/nova stable/victoria: [stable-only] libvirt: UEFI: skip OVMF_CODE.secboot.fd on pc if possible https://review.opendev.org/c/openstack/nova/+/828979 | |
| 23:01:18 | opendevreview | Mauricio Faria de Oliveira proposed openstack/nova stable/victoria: [stable-only] libvirt: UEFI: skip OVMF_CODE.secboot.fd on pc if possible https://review.opendev.org/c/openstack/nova/+/828979 | |
| 23:02:46 | opendevreview | Mauricio Faria de Oliveira proposed openstack/nova stable/ussuri: [stable-only] libvirt: UEFI: skip OVMF_CODE.secboot.fd on pc if possible https://review.opendev.org/c/openstack/nova/+/828980 | |
| #openstack-nova - 2022-03-05 | |||
| 07:25:31 | opendevreview | Merged openstack/nova master: Enable unified limits in the nova-next job https://review.opendev.org/c/openstack/nova/+/789963 | |
| 09:30:38 | opendevreview | kiran pawar proposed openstack/nova master: VMware: Split out VMwareAPISession https://review.opendev.org/c/openstack/nova/+/832156 | |
| 10:20:13 | opendevreview | kiran pawar proposed openstack/nova master: VMware: Split out VMwareAPISession https://review.opendev.org/c/openstack/nova/+/832156 | |
| 14:03:32 | opendevreview | kiran pawar proposed openstack/nova master: VMware: StableMoRefProxy for moref recovery https://review.opendev.org/c/openstack/nova/+/832164 | |
| 14:07:14 | opendevreview | kiran pawar proposed openstack/nova master: VMware: StableMoRefProxy for moref recovery https://review.opendev.org/c/openstack/nova/+/832164 | |
| 14:17:47 | opendevreview | kiran pawar proposed openstack/nova master: VMware: Split out VMwareAPISession https://review.opendev.org/c/openstack/nova/+/832156 | |
| 14:17:48 | opendevreview | kiran pawar proposed openstack/nova master: VMware: StableMoRefProxy for moref recovery https://review.opendev.org/c/openstack/nova/+/832164 | |
| 15:48:27 | opendevreview | Merged openstack/nova stable/wallaby: compute: Avoid duplicate BDMs during reserve_block_device_name https://review.opendev.org/c/openstack/nova/+/803718 | |
| #openstack-nova - 2022-03-07 | |||
| 02:05:14 | opendevreview | HYSong proposed openstack/nova master: fix local volume extend https://review.opendev.org/c/openstack/nova/+/832180 | |
| 03:29:18 | opendevreview | Xuan Yandong proposed openstack/nova master: Remove redundant symbols https://review.opendev.org/c/openstack/nova/+/832185 | |
| 09:41:36 | opendevreview | kiran pawar proposed openstack/nova master: VMware: Split out VMwareAPISession https://review.opendev.org/c/openstack/nova/+/832156 | |
| 09:41:36 | opendevreview | kiran pawar proposed openstack/nova master: VMware: StableMoRefProxy for moref recovery https://review.opendev.org/c/openstack/nova/+/832164 | |
| 10:10:47 | ignaziocassano_ | Hello, sometimes the volume retype from a netapp nfs storage to another netapp nfs storage does not work. I do not know the reason but I think something is going wrong in nova: | |
| 10:11:14 | ignaziocassano_ | File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1593, in _swap_volume\n raise exception.VolumeRebaseFailed(reason=six.text_type(exc))\n', "VolumeRebaseFailed: Volume rebase failed: Requested operation is not valid: pivot of disk 'vda' requires an active copy job\n"]: VolumeAttachmentNotFound: Volume attachment 2cd820e0-85e8-498d-a62a-800260d0cf31 could not be found | |
| 10:11:24 | ignaziocassano_ | Any helo please ? | |
| 10:11:30 | ignaziocassano_ | Any help please ? | |
| 10:41:52 | kashyap | ignaziocassano_: No direct answer, but that error (from libvirt) means: the "volume retype" (i.e volume migration) itself is not active | |
| 10:42:14 | kashyap | "active copy" == the copy that is on the NFS and is being mirrored from the NetApp storage | |
| 10:42:36 | kashyap | Also what version of OSP is this? And also mention libvirt/QEMU versions | |
| 10:45:26 | ignaziocassano_ | kashyap: I am using queens on centos 7 libvirt 4.5.0 QEMU emulator version 2.12.0 (qemu-kvm-ev-2.12.0-33.1.el7_7.4) | |
| 10:48:45 | ignaziocassano_ | Somettimes retyped volumes are corrupted and file system on instances went in read only | |
| 11:05:29 | kashyap | ignaziocassano_: The versions seems moderately old (~2017/2018); lots of storage bugs have been fixed in this area. And that corruption doesn't sound good. | |
| 11:05:47 | kashyap | I don't know if this is even reproducible consistenlty in your env. | |
| 11:05:53 | kashyap | So many variables :-( | |
| 11:15:42 | sean-k-mooney | ignaziocassano_: what version of nfs are you using | |
| 11:16:16 | sean-k-mooney | ignaziocassano_: nova recommends v4.0 as a minium preferably 4.2 | |
| 11:17:17 | ignaziocassano_ | sean-k-mooney: I do not knkow why, but the controlles mount cinder with version 4.0 while the compute nodes are using nfs vers 3 | |
| 11:17:57 | sean-k-mooney | we know that v3 has some issue with lockign that might affect data integrety | |
| 11:18:11 | sean-k-mooney | im really not sure how mixing would affect things | |
| 11:18:33 | ignaziocassano_ | ok, so I must specify nfs_mount_options in nova.conf on compute nodes | |
| 11:18:40 | sean-k-mooney | it slikely not adviasble but i suspect that this is outside the scope of nova. | |
| 11:19:12 | ignaziocassano_ | I will try it | |
| 11:19:27 | kashyap | Oh, yeah - the NFS version also plays a role. Indeed we recommend a minimum of NFS > 4.2 | |
| 11:19:29 | sean-k-mooney | https://docs.openstack.org/nova/latest/configuration/config.html#libvirt.nfs_mount_options | |
| 11:19:38 | sean-k-mooney | likely can help | |
| 11:19:44 | sean-k-mooney | but i am not that familar with it | |
| 11:19:58 | sean-k-mooney | i know downstream we have some recommended optiosn for that related to selinux | |
| 11:20:12 | sean-k-mooney | beyond that i have never really looked at what we suggest setting there | |
| 11:24:42 | ignaziocassano_ | thanks for your help | |
| 14:49:59 | dmitriis | sean-k-mooney, gibi: zuul seems to be happy about https://review.opendev.org/c/openstack/nova/+/829974 | |
| 15:01:37 | sean-k-mooney | ah thanks for the reminder | |
| 15:10:26 | opendevreview | Alexey Stupnikov proposed openstack/nova master: Add functional tests to reproduce bug #1960412 https://review.opendev.org/c/openstack/nova/+/830010 | |
| 15:36:12 | sean-k-mooney | gibi: can you spot check my expectations, when we shelve an instance would you expect use to unbinding the ports form the host | |
| 15:36:29 | sean-k-mooney | or well when we shelve offload | |
| 15:39:31 | gibi | sean-k-mooney: I think when we offload we should unbind from the host, otherwise the physical resource (i.e. pci device) is not freed. Or can we free up a PCI device without unbinding the port? | |
| 15:39:52 | sean-k-mooney | ack that is what i woudl expect too but we dont | |
| 15:40:51 | sean-k-mooney | the port shoudl still be atached to the vm but it shoudl not be bound to a host or an ml2 driver once its offloaded | |
| 15:41:25 | gibi | so we shoudl keep device_id in the port. Is that enough to keep the port reserved in neutron? | |
| 15:41:51 | sean-k-mooney | yes the device_id is enough to track ownwership | |
| 15:42:05 | sean-k-mooney | but binding:host_id shoudl be set to '' | |
| 15:42:53 | sean-k-mooney | im trying to create functional test for vdpa move operations while if figure out if i can get a 2 node deployment to test/develop move ops | |
| 15:43:09 | sean-k-mooney | but when i was writign the shelve test i noticed it was not cleared | |
| 15:43:28 | sean-k-mooney | so i dont know of this cause any bug but its not what i was expecting | |
| 15:44:04 | sean-k-mooney | gibi: we are relying on driver.cleanup to tear down the networking on the host | |
| 15:44:24 | sean-k-mooney | which it does and that also disconnects the volumes | |
| 15:44:24 | gibi | do we free the compute claim? | |
| 15:44:35 | sean-k-mooney | but we dont actully unbind the ports | |
| 15:45:07 | sean-k-mooney | its a good question i would have to look but i think so | |
| 15:45:14 | sean-k-mooney | but im not certin now | |
| 15:46:18 | gibi | if we free the claim but does not unbind the port then I think we have no resource problems it is just ugly / misleading that we keep the binding:host_id in neutron | |
| 15:46:56 | sean-k-mooney | ya | |
| 15:47:05 | sean-k-mooney | that would happen in the compute manager i guess | |
| 15:47:31 | sean-k-mooney | we call self.rt.delete_allocation_for_shelve_offloaded_instance | |
| 15:47:46 | sean-k-mooney | i would assuem that would free them | |
| 15:48:59 | sean-k-mooney | no... that is just clearing the placment allocation | |
| 15:49:14 | sean-k-mooney | we do update the resouce tracker after that however | |
| 15:58:12 | sean-k-mooney | my func test say we do not free testtools.matchers._impl.MismatchError: 4 != 3 | |
| 15:58:39 | opendevreview | Alexey Stupnikov proposed openstack/nova master: Clean up when queued live migration aborted https://review.opendev.org/c/openstack/nova/+/828570 | |
| 16:05:35 | sean-k-mooney | gibi: i might try and reporduce this in devstack and see if the same is true in realitiy | |
| 16:06:13 | gibi | maybe the periodic update_available_resources task frees it | |
| 16:06:37 | sean-k-mooney | i ran that in the func test with run_perodics | |
| 16:06:42 | sean-k-mooney | i tought it would | |
| 16:06:52 | sean-k-mooney | but apperently not | |
| 16:08:33 | sean-k-mooney | its not inconciveable this is a func test issue but its worth figureing out | |
| 16:09:33 | dansmith | what is the nova-emulation job? | |
| 16:09:48 | sean-k-mooney | it test emulating arm vms | |
| 16:10:00 | sean-k-mooney | so x86 host bootign arm vms | |
| 16:10:03 | dansmith | is it supposed to be stable? | |
| 16:10:19 | dansmith | seems like a ton of rechecking going on these days, and nova jobs have gotten pretty fat | |
| 16:10:50 | sean-k-mooney | am i dont know if that is stable | |
| 16:10:56 | sean-k-mooney | we just enabled it a few days ago | |
| 16:11:20 | dansmith | it's voting, | |
| 16:11:21 | sean-k-mooney | i did not think it was failing but it can certenly be set non-voting or moved to periodic | |
| 16:11:26 | dansmith | and I just saw a kernel panic on it | |
| 16:11:33 | dansmith | I think it's a guest kernel | |
| 16:12:00 | sean-k-mooney | https://zuul.openstack.org/builds?job_name=nova-emulation&skip=0 | |
| 16:12:06 | sean-k-mooney | it looks kind of ok | |
| 16:12:14 | sean-k-mooney | i think that is the first failure since it was merged | |
| 16:12:16 | dansmith | https://zuul.opendev.org/t/openstack/build/cb1314bff0f34bfdbb3a4f1fd5547b72 | |
| 16:12:39 | dansmith | okay, well, regardless, nova jobs are looking pretty heavy | |
| 16:12:55 | dansmith | I dunno how widely-known it is, but we're losing 30% of our CI capacity at the end of the month | |
| 16:13:43 | dansmith | so we'll probably need to be making some cuts | |
| 16:13:51 | dansmith | what's the major benefit of testing arm-on-x86? | |