| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-06-17 | |||
| 09:32:55 | elodilles | sure, looking at the patches :) | |
| 09:42:49 | gibi | lyarwood, stephenfin: yesterday infra turned off limestone due to the pip cache issue | |
| 09:43:09 | gibi | so we should not see these nonsensical req conflicts | |
| 09:43:13 | gibi | any more today | |
| 09:45:51 | yonglihe | stephenfin, that's because that patch lost the decency to second patch, fixed. | |
| 09:45:53 | lyarwood | wonderful | |
| 09:46:26 | lyarwood | gibi: https://bugs.launchpad.net/cinder/+bug/1932287 just caught this if you see any random volume creation failures today | |
| 09:47:30 | gibi | lyarwood: thanks, I haven't seen that issue yet | |
| 09:47:31 | opendevreview | Merged openstack/nova stable/rocky: Remove allocations before setting vm_status to SHELVED_OFFLOADED https://review.opendev.org/c/openstack/nova/+/771985 | |
| 09:48:35 | stephenfin | elodilles: Yeah, as lyarwood said, we need to move the cherry-pick change out of the pep8 job. I hadn't seen that failure | |
| 09:50:31 | gibi | lyarwood: with the exit code 139 lvs complains about missing devices and that I saw before | |
| 09:50:56 | lyarwood | yeah https://review.opendev.org/c/openstack/cinder/+/783660 fixed it elsewhere | |
| 09:51:01 | lyarwood | just not in this path | |
| 09:51:44 | gibi | lyarwood: cool, then we have a way forward | |
| 09:51:55 | opendevreview | Stephen Finucane proposed openstack/nova stable/ussuri: Reproduce bug 1897528 https://review.opendev.org/c/openstack/nova/+/791770 | |
| 09:51:56 | opendevreview | Stephen Finucane proposed openstack/nova stable/ussuri: Ignore PCI devices with 32bit domain https://review.opendev.org/c/openstack/nova/+/791771 | |
| 09:52:06 | stephenfin | elodilles: lyarwood: fixed the pep8 failure ^ | |
| 09:52:58 | gibi | lyarwood: I'm hitting https://bugs.launchpad.net/nova/+bug/1912310 many times now and almost always in the test_volume_backed_live_migration tempest test. Wondering if it worth to disable that test until ovsdbapp fix lands | |
| 09:53:06 | opendevreview | Stephen Finucane proposed openstack/nova stable/train: Reproduce bug 1897528 https://review.opendev.org/c/openstack/nova/+/792116 | |
| 09:53:07 | opendevreview | Stephen Finucane proposed openstack/nova stable/train: Ignore PCI devices with 32bit domain https://review.opendev.org/c/openstack/nova/+/792117 | |
| 09:53:22 | stephenfin | and the train ones are updated now too | |
| 09:57:33 | lyarwood | gibi: ack lets do it, I'll disable them now | |
| 10:04:22 | opendevreview | Lee Yarwood proposed openstack/nova master: zuul: Skip volume backed LM tests until bug #1912310 is resolved https://review.opendev.org/c/openstack/nova/+/796813 | |
| 10:04:29 | lyarwood | gibi: ^ hopefully that's enough, if it isn't then we might want to move the LM jobs to non-voting | |
| 10:11:15 | opendevreview | Stephen Finucane proposed openstack/nova master: db: Reintroduce validation of shadow table schema https://review.opendev.org/c/openstack/nova/+/796814 | |
| 10:11:39 | stephenfin | lyarwood: gibi: one final one, as requested ^ | |
| 10:12:38 | gibi | lyarwood: thanks | |
| 10:13:05 | elodilles | stephenfin: actually i was surprised that pep8 is failing in ussuri because of py27/six problem as py27 should be supported only up until train :-o | |
| 10:13:43 | stephenfin | elodilles: yeah, we simply weren't aggressive enough in dropping the no-longer relevant hacking checks | |
| 10:13:54 | elodilles | oh, i see | |
| 10:15:02 | stephenfin | I would personally like to backport the patch that dropped the check, but I don't know what you think about that. I can't imagine that would violate stable policy since it's nothing to do with production code | |
| 10:15:26 | stephenfin | (commit 9dca0d186f834c38d0d06e226b18ab3ae717c140 fwiw) | |
| 10:17:42 | lyarwood | Yup I assumed we would tbh | |
| 10:18:02 | lyarwood | no reason to leave it just on >=stable/xena | |
| 10:21:52 | elodilles | stephenfin: well, it formally violates, as it is a blueprint o:) ... anyway, I would stick to backporting only bug fixes... but... given that py27 is not supported in ussuri anymore... anyway I'm a bit unsure... o:) | |
| 10:24:12 | lyarwood | oh sorry I thought we were talking about the cherry-pick script | |
| 10:35:35 | elodilles | lyarwood: actually I've missed that discussion :X Are you planning to move out the cherry-pick-check from pep8? | |
| 10:36:19 | elodilles | lyarwood: nevermind, i'm just a bit slow today :D | |
| 10:36:31 | lyarwood | elodilles: https://review.opendev.org/c/openstack/nova/+/796626 yeah that's the idea, make it non-voting in check and only voting in the gate | |
| 10:36:47 | elodilles | lyarwood: yeah, sorry :X | |
| 10:45:11 | sean-k-mooney | lyarwood: instead of skiping the test https://review.opendev.org/c/openstack/nova/+/796813 why not just use the old driver | |
| 10:45:32 | sean-k-mooney | the stalling issue only happens if you use the native driver | |
| 10:45:44 | sean-k-mooney | the vsctl one wont have that problem | |
| 10:50:34 | lyarwood | We could but unless that's done in devstack across all jobs we end up with a mixed set of jobs | |
| 10:51:07 | sean-k-mooney | is that a bad thing | |
| 10:51:42 | sean-k-mooney | with the current patch your just reducing coverage | |
| 10:51:56 | sean-k-mooney | but the issue can still happen | |
| 10:52:21 | sean-k-mooney | im fine with makeing that change in devstack tempeorally | |
| 10:52:56 | lyarwood | kk if you could post that we can yank or revert this | |
| 10:53:40 | sean-k-mooney | i left a -1 on the patch already but would you like me to do the devstack change | |
| 10:54:12 | sean-k-mooney | jsut finishing an email but i can do it then | |
| 10:56:23 | lyarwood | sean-k-mooney: ack | |
| 10:56:30 | lyarwood | stephenfin: can you yank the +W on https://review.opendev.org/c/openstack/nova/+/796813 | |
| 10:56:45 | lyarwood | stephenfin: sean-k-mooney is going to work around this in devstack | |
| 10:56:52 | stephenfin | sure, done | |
| 10:56:55 | lyarwood | ta | |
| 11:05:05 | lyarwood | noice, the FIPS fallout doesn't look that bad | |
| 11:05:06 | lyarwood | https://6cbf38d10f57b850b36e-212ab268b5e4bbb4b3348f98a2a831ee.ssl.cf5.rackcdn.com/790519/6/check/nova-fips/914f344/testr_results.html | |
| 11:06:16 | lyarwood | paramiko as expected and a server create timeout | |
| 11:07:57 | lyarwood | and that smells like the ovs locking up issue | |
| 11:12:00 | sean-k-mooney | lyarwood: cool | |
| 11:12:34 | sean-k-mooney | by the way i have found an interestign ceph issue that well i know how to fix but dont know how to detect | |
| 11:13:00 | sean-k-mooney | lyarwood: are you familar with EC pools in ceph | |
| 11:13:57 | sean-k-mooney | i was following https://docs.ceph.com/en/latest/rbd/rbd-openstack/ to configure ceph for openstack in general and https://themeanti.me/technology/2018/08/23/ceph_erasure_openstack.html for the ec pools | |
| 11:14:13 | sean-k-mooney | and i missed a step kind of | |
| 11:14:43 | sean-k-mooney | since i have a vms pool for nova and a vms_data pool | |
| 11:15:14 | sean-k-mooney | wehn i was doing the cephx user caps configuration i need to list both the vms pool and vms_data pool | |
| 11:15:19 | sean-k-mooney | only listed vms | |
| 11:15:37 | sean-k-mooney | the result of which is that nova booted a vm and it went into the active state | |
| 11:15:48 | sean-k-mooney | but it could not reade or write its root disk | |
| 11:16:34 | sean-k-mooney | the root disk has all the data present but it was inaccessable to qemu | |
| 11:17:20 | sean-k-mooney | lyarwood: due you think that qemu might be abel to detect that and create a warnig/error or could we detech that somehow and create an error | |
| 11:17:55 | sean-k-mooney | nova is calling ceph directly to get the avaibale storage | |
| 11:18:39 | sean-k-mooney | im debating if woudl make sense for nova to try and create a volume and read form it on the host or something on startup of the agent | |
| 11:22:52 | lyarwood | I'm not sure how QEMU could catch that tbh | |
| 11:23:11 | lyarwood | tbh that smells more like a deployment tooling validation? | |
| 11:23:19 | lyarwood | I wouldn't ask Nova to check it | |
| 11:23:33 | sean-k-mooney | ok its just annoying to debug | |
| 11:23:44 | sean-k-mooney | there is no error in qemu or nova or ceph | |
| 11:24:00 | sean-k-mooney | the vm just cant find a bootable disk | |
| 11:24:26 | sean-k-mooney | lyarwood: the deployment tool im using does not technially support this anymore which is why i messed it up | |
| 11:24:36 | sean-k-mooney | kolla-ansibel now just has extrenal ceph support | |
| 11:25:00 | lyarwood | that could still be a validation for external ceph | |
| 11:25:02 | sean-k-mooney | so you predeply ceph with your favor tool and then pass it a few files like the keyrings and it does the rest | |
| 11:25:06 | lyarwood | that the keyring has r/w access | |
| 11:25:15 | sean-k-mooney | it could yes | |
| 11:25:25 | sean-k-mooney | there are post run check which i did not run | |
| 11:25:48 | sean-k-mooney | i might add one for this | |
| 11:26:31 | sean-k-mooney | i was going to try and update there docs later anyway to document some of the more advanced customisation that im doing | |
| 11:26:57 | sean-k-mooney | for example running all fo the opnestack servics on the same port but with different subdomains | |
| 11:54:54 | sean-k-mooney | lyarwood: stephenfin https://review.opendev.org/c/openstack/devstack/+/796826 | |
| 11:55:00 | sean-k-mooney | i think that will do the right thing | |
| 12:09:46 | lyarwood | LGTM but I'll wait for CI to run before I vote | |
| 12:12:26 | sean-k-mooney | i have not had time to test that so that is proably a good idea :) | |
| 12:14:19 | lyarwood | Small nit in the commit message btw, you called out the wrong bug. | |
| 12:14:37 | sean-k-mooney | oh | |
| 12:14:51 | sean-k-mooney | i can fix it but might wait for the ci to finish | |
| 12:14:56 | lyarwood | yeah no issues | |
| 12:15:12 | sean-k-mooney | ah i di | |
| 12:15:33 | sean-k-mooney | it should be https://bugs.launchpad.net/nova/+bug/1929446 | |