| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-01-25 | |||
| 14:07:18 | sean-k-mooney | i suspect that it was prviously configured in monolich mode and is not in modular mode | |
| 14:09:06 | sean-k-mooney | it looks like libvirt is planning to eventually remove the monolithic daemon | |
| 14:10:31 | sean-k-mooney | zigo: is debian sid the unstable testing branch of debian | |
| 14:10:50 | sean-k-mooney | bulleye it the latest lts right | |
| 14:11:43 | sean-k-mooney | i suspect that the libvirt version has been bumped in sid and it now default to modular which is causing your current issue | |
| 14:14:35 | zigo | sean-k-mooney: I'm trying to get OpenStack to work in Sid/Unstable because we're going to freeze Bullseye in a few days from now, yes. | |
| 14:14:53 | zigo | And lots of things are broken ... :( | |
| 14:15:03 | sean-k-mooney | ah right well it just got https://tracker.debian.org/news/1223928/libvirt-690-4-migrated-to-testing/ | |
| 14:15:18 | sean-k-mooney | although it had 6.9.0-1 in late november | |
| 14:15:36 | sean-k-mooney | zigo: is bullseye running python 3.9? | |
| 14:15:42 | zigo | Yeah. | |
| 14:15:54 | sean-k-mooney | zigo: i tested with fedroa rawhid in decmeber and we cant run master on it | |
| 14:15:59 | zigo | Eventlet and Amqp broke... | |
| 14:16:02 | sean-k-mooney | yep | |
| 14:16:16 | zigo | I think we got Eventlet back to working, but Amqp still breaks with ssl. | |
| 14:16:31 | sean-k-mooney | there was a new eventlet release on december 13th but not sure its fully working | |
| 14:16:34 | sean-k-mooney | ya | |
| 14:16:36 | zigo | TypeError: _wrap_socket() argument 'sock' must be _socket.socket, not SSLSocket | |
| 14:16:43 | zigo | I'm really not sure how to fix that one... | |
| 14:16:50 | sean-k-mooney | yep the bug is open for a long time for that | |
| 14:16:54 | sean-k-mooney | on sec | |
| 14:17:00 | zigo | Though I'm currently trying to fix the libvirt issue first. | |
| 14:17:02 | sean-k-mooney | there is an issue with dnspython | |
| 14:17:07 | zigo | Oh ... | |
| 14:17:27 | sean-k-mooney | https://github.com/eventlet/eventlet/issues/632 | |
| 14:17:31 | sean-k-mooney | https://github.com/eventlet/eventlet/issues/619 | |
| 14:17:41 | sean-k-mooney | eventlet need dnspyton<2.0.0 | |
| 14:20:11 | sean-k-mooney | https://github.com/eventlet/eventlet/pull/639 partly fixes things but not fully and its not merged | |
| 14:22:30 | zigo | sean-k-mooney: I have the patch already in the Debian package. | |
| 14:23:19 | sean-k-mooney | i see and still no joy | |
| 14:23:43 | zigo | TypeError: _wrap_socket() argument 'sock' must be _socket.socket, not SSLSocket | |
| 14:23:43 | zigo | Though do you think this one is an eventlet problem too, rather than in AMQP? | |
| 14:24:25 | sean-k-mooney | that one im not sure about | |
| 14:24:37 | sean-k-mooney | its a similar error but i have not seen it yet | |
| 14:25:03 | sean-k-mooney | if you find out i would like to know however | |
| 14:25:53 | zigo | :) | |
| 14:26:08 | zigo | One thing at a time, I guess. FIrst, that libvirt issue ... | |
| 14:26:48 | sean-k-mooney | ya i might need to flag the libvirt issue internally too. | |
| 14:27:06 | sean-k-mooney | i dont think we had planned to need to run libvirt in multiple continers | |
| 14:27:16 | sean-k-mooney | or have multiple processes in the libvirt container. | |
| 14:27:25 | sean-k-mooney | beyond virtlogd | |
| 14:32:50 | kashyap | zigo: Modular daemons in libvirt have been around for about a year-and-a-half | |
| 14:34:09 | zigo | Definitively a configuration problem, on my side. At first attempt, nova tries the correct socket path. Then I comment out the libvirtd_opts="-l" to make libvirt work propertly (since it defaults to socket activation), then systemctl start libvritd, and then nova tries the wrong socket path ... | |
| 15:18:47 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Replace blind retry with libvirt event waiting in detach https://review.opendev.org/c/openstack/nova/+/770246 | |
| 16:32:50 | openstackgerrit | sean mooney proposed openstack/nova master: libvirt: delegate ovs plug to os-vif https://review.opendev.org/c/openstack/nova/+/602432 | |
| 17:15:19 | stephenfin | If anyone wants to make themselves really miserable, take a look at https://review.opendev.org/c/openstack/placement/+/772335/ | |
| 17:15:46 | stephenfin | Somehow, config that I set for oslo.policy is being unset/overridden when I run tests in parallel | |
| 17:16:21 | stephenfin | and I haven't a clue what's bleeding across, despite spending the bulk of the day looking :) | |
| 17:16:31 | stephenfin | stupid global stuff | |
| 17:16:53 | openstackgerrit | Lance Bragstad proposed openstack/placement master: policy: Remove the deprecated 'placement' rule https://review.opendev.org/c/openstack/placement/+/772333 | |
| 17:18:31 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: libvirt: add AsyncDeviceDetachEventsHandler https://review.opendev.org/c/openstack/nova/+/772381 | |
| 17:18:31 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: libvirt: Define and emit DeviceRemovedEvent and DeviceRemovalFailedEvent https://review.opendev.org/c/openstack/nova/+/749929 | |
| 17:19:49 | lbragstad | stephenfin i'm not sure if it's related - but implementing some additional gabbi tests for inventories stumped my late last week | |
| 17:20:06 | stephenfin | lbragstad: The one and the same | |
| 17:20:28 | stephenfin | That was an attempted fix for same and it works...if you don't run things in parallel | |
| 17:20:46 | lbragstad | i was trying to implement tests for all the personas in the legacy rbac tests - per gmann's suggestion | |
| 17:20:47 | lbragstad | https://review.opendev.org/c/openstack/placement/+/760238/7 | |
| 17:21:01 | lbragstad | just like what i did here - https://review.opendev.org/c/openstack/placement/+/760240/19/placement/tests/functional/gabbits/resource-provider-legacy-rbac.yaml | |
| 17:21:53 | lbragstad | and in the process of testing inventory updates - the placement API consistently gives me a 409 | |
| 17:22:23 | lbragstad | i tried cleaning up all possible resources in the test and starting from scratch, but nothing worked | |
| 17:22:47 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: libvirt: allow querying devices from the persistent domain https://review.opendev.org/c/openstack/nova/+/772383 | |
| 17:22:49 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: libvirt: parse alias out from device config https://review.opendev.org/c/openstack/nova/+/772384 | |
| 17:22:56 | stephenfin | lbragstad: Does it work if you run a single test by itself? | |
| 17:23:46 | lbragstad | i was running all the tests within inventory-legacy-rbac.yaml since the first few tests create resource providers necessary for inventories | |
| 17:24:21 | stephenfin | (you can run it using e.g. 'tox -e functional-py38 -- placement.tests.functional.test_api.inventory-legacy-rbac') | |
| 17:24:32 | lbragstad | yeah - that's what i was doing | |
| 17:24:38 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Replace blind retry with libvirt event waiting in detach https://review.opendev.org/c/openstack/nova/+/770246 | |
| 17:24:40 | stephenfin | hmm, odd | |
| 17:24:50 | stephenfin | I'm pretty sure things aren't fully isolated at this point | |
| 17:25:05 | lbragstad | yeah, it's strange | |
| 17:27:15 | lbragstad | i was going to take another crack at it today, but i might wait for some feedback | |
| 17:28:10 | stephenfin | lbragstad: Perhaps check to see if https://review.opendev.org/c/openstack/placement/+/772335/ fixes things for you | |
| 17:28:31 | stephenfin | I suspect it might, since the point I put forward is still valid (I think) | |
| 17:28:49 | stephenfin | It's just not a perfect fix due to aforementioned bleeding of some kind of config | |
| 17:31:15 | lbragstad | stephenfin https://zuul.opendev.org/t/openstack/build/6afd68bd7a1a444a856f538ee9ef3476 is the same policy failure i got prior to your fix | |
| 17:31:54 | openstackgerrit | Lance Bragstad proposed openstack/placement master: policy: Deprecate 'admin_api' rule https://review.opendev.org/c/openstack/placement/+/772334 | |
| 17:32:05 | stephenfin | yup, but if you run the test by itself then it works | |
| 17:32:11 | openstackgerrit | Lance Bragstad proposed openstack/placement master: WIP: Add fix for fixture https://review.opendev.org/c/openstack/placement/+/772335 | |
| 17:34:59 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: libvirt: Define and emit DeviceRemovedEvent and DeviceRemovalFailedEvent https://review.opendev.org/c/openstack/nova/+/749929 | |
| 17:35:00 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: libvirt: allow querying devices from the persistent domain https://review.opendev.org/c/openstack/nova/+/772383 | |
| 17:35:00 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: libvirt: add AsyncDeviceDetachEventsHandler https://review.opendev.org/c/openstack/nova/+/772381 | |
| 17:39:41 | lbragstad | stephenfin ok - yeah | |
| 17:39:42 | lbragstad | http://paste.openstack.org/show/801955/ | |
| 17:39:43 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: libvirt: parse alias out from device config https://review.opendev.org/c/openstack/nova/+/772384 | |
| 17:39:44 | lbragstad | i see what you mean | |
| 17:41:08 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Replace blind retry with libvirt event waiting in detach https://review.opendev.org/c/openstack/nova/+/770246 | |
| 17:41:29 | stephenfin | so I guess we need to either figure out what's causing the contamination, isolate the secure-rbac stuff from other tests, or simply comment out those tests with a FIXME | |
| 17:42:21 | stephenfin | I tried using the isolation bisect feature in stestr to identify the leak but it's not a single test and stestr couldn't find anything | |
| 17:42:41 | lbragstad | i'll defer to whatever the placement and nova folks want to do | |
| 17:42:57 | lbragstad | technically, this happening for all gabbi tests, i assume | |
| 17:43:08 | stephenfin | Yes, I'd suspect so | |
| 17:43:16 | lbragstad | when they aren't run individually | |
| 17:43:31 | stephenfin | I'll discuss with gmann, gibi etc. in the AM and see what people are happy with | |
| 17:43:51 | lbragstad | ok - thanks, i'm going to push this to the back burner then and give some of the other projects a hand | |
| 17:44:01 | stephenfin | ack, makes sense | |
| 17:44:18 | lbragstad | last question - should this be a bug against placement? | |
| 17:44:40 | stephenfin | Probably, yes. Tests aren't isolated | |
| 17:44:52 | lbragstad | ok - i'll go open one | |
| 17:44:57 | stephenfin | Cool | |
| 17:50:51 | lbragstad | https://storyboard.openstack.org/#!/story/2008556 | |