| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-05-21 | |||
| 17:58:48 | openstackgerrit | Merged openstack/nova stable/wallaby: libvirt: Remove dead error handling code https://review.opendev.org/c/openstack/nova/+/788724 | |
| 17:58:48 | openstackgerrit | Merged openstack/nova stable/wallaby: libvirt: Remove dead error handling code https://review.opendev.org/c/openstack/nova/+/788724 | |
| 19:02:12 | openstackgerrit | Dmitrii Shcherbakov proposed openstack/nova-specs master: Introduce Transport Nodes https://review.opendev.org/c/openstack/nova-specs/+/787458 | |
| 19:02:12 | openstackgerrit | Dmitrii Shcherbakov proposed openstack/nova-specs master: Introduce Transport Nodes https://review.opendev.org/c/openstack/nova-specs/+/787458 | |
| #openstack-nova - 2021-05-22 | |||
| 05:30:22 | openstackgerrit | Merged openstack/nova stable/wallaby: Move instance power state check to _detach_with_retry https://review.opendev.org/c/openstack/nova/+/788725 | |
| 05:30:22 | openstackgerrit | Merged openstack/nova stable/wallaby: Move instance power state check to _detach_with_retry https://review.opendev.org/c/openstack/nova/+/788725 | |
| 17:47:20 | openstackgerrit | Merged openstack/nova stable/wallaby: Consolidate device detach error handling https://review.opendev.org/c/openstack/nova/+/788726 | |
| 17:47:20 | openstackgerrit | Merged openstack/nova stable/wallaby: Consolidate device detach error handling https://review.opendev.org/c/openstack/nova/+/788726 | |
| 20:54:06 | openstackgerrit | Merged openstack/nova stable/train: Make _rebase_with_qemu_img() generic https://review.opendev.org/c/openstack/nova/+/780782 | |
| 20:54:06 | openstackgerrit | Merged openstack/nova stable/train: Make _rebase_with_qemu_img() generic https://review.opendev.org/c/openstack/nova/+/780782 | |
| #openstack-nova - 2021-05-23 | |||
| 07:23:45 | openstackgerrit | Wenping Song proposed openstack/placement master: Remove unvalid test https://review.opendev.org/c/openstack/placement/+/792734 | |
| 11:15:03 | openstackgerrit | Merged openstack/nova stable/train: Rebase qcow2 images when unshelving an instance https://review.opendev.org/c/openstack/nova/+/785627 | |
| 14:52:23 | openstackgerrit | Merged openstack/nova stable/train: Update image_base_image_ref during rebuild. https://review.opendev.org/c/openstack/nova/+/787945 | |
| #openstack-nova - 2021-05-24 | |||
| 08:01:54 | lyarwood | \o morning | |
| 08:28:45 | lyarwood | stephenfin: would you mind hitting these again if you're around today https://review.opendev.org/q/topic:%22bug%252F1928063%22+(status:open%20OR%20status:merged) | |
| 09:03:55 | kevinz | lyarwood: stephenfin: morning! Could you help to review this when you convenient? https://review.opendev.org/c/openstack/nova/+/763928, It is about live migration support on Arm64 | |
| 10:21:18 | openstackgerrit | Sylvain Bauza proposed openstack/nova-specs master: Add generic mdevs to Nova https://review.opendev.org/c/openstack/nova-specs/+/792796 | |
| 11:41:40 | openstackgerrit | Stephen Finucane proposed openstack/nova master: tests: Move libvirt-specific fixtures https://review.opendev.org/c/openstack/nova/+/790969 | |
| 11:41:41 | openstackgerrit | Stephen Finucane proposed openstack/nova master: tests: Add os-brick fixture https://review.opendev.org/c/openstack/nova/+/790970 | |
| 11:41:41 | openstackgerrit | Stephen Finucane proposed openstack/nova master: tests: Rename 'ImageBackendFixture' to 'LibvirtImageBackendFixture' https://review.opendev.org/c/openstack/nova/+/792353 | |
| 11:41:42 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Create a fixture around fake_notifier https://review.opendev.org/c/openstack/nova/+/758446 | |
| 11:41:43 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Use NotificationFixture for legacy notifications too https://review.opendev.org/c/openstack/nova/+/758448 | |
| 11:41:44 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Test the NotificationFixture https://review.opendev.org/c/openstack/nova/+/758450 | |
| 11:41:44 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Move fake_notifier impl under NotificationFixture https://review.opendev.org/c/openstack/nova/+/758451 | |
| 11:41:45 | openstackgerrit | Stephen Finucane proposed openstack/nova master: rpc: Mark attributes as private https://review.opendev.org/c/openstack/nova/+/792803 | |
| 12:34:43 | openstackgerrit | Merged openstack/nova master: tests: Move libvirt-specific fixtures https://review.opendev.org/c/openstack/nova/+/790969 | |
| 12:35:26 | openstackgerrit | Merged openstack/nova master: tests: Add os-brick fixture https://review.opendev.org/c/openstack/nova/+/790970 | |
| 12:36:04 | sean-k-mooney | stephenfin: are you rewriting all the test fixutures again :P | |
| 12:36:35 | openstackgerrit | Merged openstack/nova master: tests: Rename 'ImageBackendFixture' to 'LibvirtImageBackendFixture' https://review.opendev.org/c/openstack/nova/+/792353 | |
| 12:36:58 | lyarwood | I didn't like backporting func tests anyway | |
| 12:37:08 | lyarwood | /s | |
| 12:37:13 | sean-k-mooney | :) | |
| 12:37:36 | sean-k-mooney | we just need to get all our customer to deploy master at all times | |
| 12:37:39 | sean-k-mooney | problem solved | |
| 12:39:44 | lyarwood | * lyarwood nods | |
| 13:48:56 | lyarwood | Random, as an admin I can't seem to list server events for a user defined instance? | |
| 13:49:03 | lyarwood | $ openstack server event list srvtree-server1-rh5da6maeidh | |
| 13:49:03 | lyarwood | No server with a name or ID of 'srvtree-server1-rh5da6maeidh' exists. | |
| 13:51:26 | stephenfin | @lyarwood If you're in a different project then that won't work | |
| 13:51:36 | lyarwood | even as the admin? | |
| 13:51:45 | lyarwood | okay weird | |
| 13:51:52 | lyarwood | I thought this worked previously | |
| 13:51:56 | stephenfin | because the name-based search is done against the instances in the project | |
| 13:52:19 | stephenfin | I doubt it. A couple of command have an '--all-projects' option just for this | |
| 13:52:51 | stephenfin | * stephenfin double checks to make sure he didn't change anything recently, just in case | |
| 13:53:20 | lyarwood | ah right I was likely using the UUID in the past | |
| 13:53:57 | stephenfin | Nope, I added missing options but it's otherwise unchanged, save for some docs, since it was added in 2017 | |
| 13:54:07 | stephenfin | I'd say so | |
| 13:56:17 | lyarwood | Yeah sorry, I've used UUIDs everywhere in the request-id presentation and that just threw me | |
| 14:35:49 | lyarwood | test_volume_backed_live_migration keeps failing on master at the moment btw | |
| 14:49:44 | lyarwood | Ah I see why, slow nodes and pre_live_migration is timing out | |
| 15:35:57 | melwitt | stephenfin: were you planning to work on trying to remove eventlet or would you mind if I took a try at it? the ptg discussion on that occurred earlier than I had come online that day | |
| 15:36:57 | stephenfin | melwitt: go for it. I am hoping to do something on it but it's a potential minefield that would benefit from many eyes | |
| 15:41:06 | melwitt | stephenfin: cool, thanks. I've gone quite down the rabbit hole related to eventlet on some downstream bugs, so I've some ideas now (fortunately or unfortunately) | |
| 16:06:51 | dansmith | melwitt: this is remove eventlet from api right? | |
| 16:07:44 | melwitt | dansmith: and potentially everything else too | |
| 16:09:12 | dansmith | melwitt: nothing else is threadsafe so I have a hard time imagining that being a thing | |
| 16:10:31 | melwitt | tl;dr is there's a bad interaction and failure mode between gevent/eventlet and pymysql and mysqlconnector wherein if a green thread is killed before a connection is cleaned up, it leaves it in an inconsistent state and the next attempt to use a connection blows up | |
| 16:11:47 | melwitt | I've spoken at length with zzzeek about this and my understanding is this can't be worked around or handled and that replacing our usage of eventlet with native threading or similar is the only way to avoid it | |
| 16:12:33 | dansmith | is there a pointer to something to read about it? | |
| 16:12:51 | melwitt | yeah, sec | |
| 16:15:01 | dansmith | api and conductor going to native threads are doable I think without too much crazy, | |
| 16:15:23 | dansmith | but I think compute will be a nightmare, but it also doesn't use the DB driver at all, so it should be immune | |
| 16:15:30 | melwitt | dansmith: this comment contains the relevant references https://bugzilla.redhat.com/show_bug.cgi?id=1927994#c45 the rest of that bug has a lot of comments, most of which are private because I'm not sure they help bring any clarity. it's been a long discussion on there | |
| 16:16:01 | melwitt | oh, the entire bug looks to be private /facepalm | |
| 16:16:14 | melwitt | https://github.com/PyMySQL/PyMySQL/issues/234 | |
| 16:16:19 | melwitt | https://github.com/sqlalchemy/sqlalchemy/issues/3258 | |
| 16:16:26 | melwitt | https://github.com/PyMySQL/PyMySQL/issues/260 | |
| 16:16:33 | melwitt | those are the references ^ | |
| 16:17:01 | dansmith | thanks, I can read it at least | |
| 16:17:16 | dansmith | so the assertion is that all openstack projects will have to undertake this conversion? | |
| 16:17:28 | dansmith | that's a pretty big deal | |
| 16:18:11 | melwitt | for whatever reason nova is the only one that seems affected by this, this error has not been found in any other service's logs in the same deployments that see it in nova often | |
| 16:19:16 | dansmith | and only nova-api? | |
| 16:19:17 | melwitt | and I think most of the appearance of the error is from back when we had the eventlet-based wsgi server for nova-api. my guess is that since moving away from that, our usage is much reduced and reduces the chances of hitting this | |
| 16:19:45 | melwitt | no I have seen traces in nova-scheduler and nova-conductor as well | |
| 16:20:23 | dansmith | okay, because nova-api already runs with a combination of native and green threads, so if it was just api, then that could be why, | |
| 16:20:45 | dansmith | but if scheduler and conductor see it as well, I'm not sure why no other service would be affected | |
| 16:22:53 | melwitt | yeah, I'm guessing it's because it becomes very rare when the only things using eventlet are the periodic tasks, timers/retries, and some bits of scatter/gather. the error shows up under high load and coexists with other various connection errors to the database | |
| 16:23:21 | melwitt | well, maybe "very rare" is not a good way to put it, "more rare" | |
| 16:25:43 | melwitt | I have searched for some way that nova does something different than any other service wrt to database access and found nothing | |
| 16:26:15 | melwitt | the only lead we have so far is that nova uses eventlet more than any other services do (afaik so far) | |
| 16:26:17 | sean-k-mooney | melwitt: i assume we have not had any downstream sqlalcamy updtes in rhel 7 | |
| 16:27:21 | sean-k-mooney | does this happen in modren openstack that was for 13 so queens | |
| 16:27:48 | melwitt | sean-k-mooney: what do you mean, like version changes? I don't think so but zzzeek would have covered that | |
| 16:28:12 | sean-k-mooney | yep or perhaps a backport that could have caused it | |
| 16:28:59 | melwitt | sean-k-mooney: to your other question, I could find no bug reports for this for newer than 13 | |
| 16:29:39 | melwitt | that could either mean it went away or that it happens rarely enough that no one has bothered reporting it. not sure what to think | |
| 16:30:09 | dansmith | melwitt: but nothing changed about conductor and scheduler that would account for them not showing the issue in later versions | |
| 16:30:37 | dansmith | meaning api going from evenlet to real wsgi is a change that could affect this, but that doesn't impact the other services | |
| 16:30:53 | melwitt | zzzeek anyway strongly recommended we stop using eventlet for its known issues with the mysql connectors | |
| 16:31:10 | dansmith | I've read the bug now, and I see that he has, | |
| 16:31:20 | dansmith | but it's not quite as simple as just turning it off | |
| 16:31:51 | melwitt | dansmith: yeah, I appreciate that. but it's hard to know if this is just not been reported or if it's really not there anymore | |
| 16:31:57 | sean-k-mooney | i wonder is this wsgi related | |
| 16:32:07 | dansmith | so much of what goes on changes if you don't have those yield points patched in | |
| 16:32:29 | dansmith | sean-k-mooney: melwitt says she has reports of it from scheduler and conductor, which is what really puzzles me | |
| 16:32:47 | melwitt | yeah, I know it's not simple as turning it off but afaict we could switch everything to native threads. I've started looking into it (just by having to explain to the customer where/when it's workaroundable and when it's not) | |
| 16:32:53 | sean-k-mooney | ok i was wondering if it was related to the issue with enabling multiple tread in the wsgi process | |
| 16:33:07 | sean-k-mooney | but if its in the schduler and conductor its not that | |