| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-05-29 | |||
| 20:04:35 | sean-k-mooney | melwitt: we we ar confident we should not disable heatbeats then we can save martin time and do? | |
| 20:04:58 | aspiers | artom: which was one of the codes in one of the Denver escape rooms XD | |
| 20:05:07 | artom | Noice. | |
| 20:05:13 | sean-k-mooney | nothing? supress log? make ideal time 5 minuets for mod_wsgi? | |
| 20:06:36 | sean-k-mooney | i think the current situation is more or less lets figure out how to fix it upstream and then backport that fix upstream and downstream | |
| 20:07:36 | melwitt | sean-k-mooney: yes, based on the fact that rabbit will not clean up stale connections in that case, I don't think we should disable it. admittedly I don't understand a ton about this, but the docs recommend against disabling unless tcp keepalives are used in their place https://www.rabbitmq.com/heartbeats.html#disabling | |
| 20:08:03 | sean-k-mooney | we do use tcp keepalives | |
| 20:08:17 | sean-k-mooney | and we set it to like 1 second intervals | |
| 20:08:58 | sean-k-mooney | triplos defalts are pretty agressive in trying to keep the connection alive | |
| 20:09:22 | melwitt | the only possible thing we could do with the logging, if oslo.messaging ppl are agreeable, would be to downgrade the "connection reset" message to WARNING and then set our log level to ERROR to hide it | |
| 20:09:47 | sean-k-mooney | melwitt: i was looking it python log filters a bit too | |
| 20:09:57 | melwitt | and then obviously we have to have threads=1 | |
| 20:10:18 | sean-k-mooney | i think we could intall a logfilter that would match on the oslo meesge and drop them too if oslo do not want to change the level | |
| 20:10:38 | mriedem | melwitt: ok, i'm torn between waffling over a reno on the issue and figuring out just the right words vs releasing what we have (which is quite a bit unreleased in stable/stein) and just letting the reno happen later if at all | |
| 20:10:40 | melwitt | oh, I see | |
| 20:10:47 | mriedem | b/c i'm not an authority to review that reno | |
| 20:11:23 | melwitt | yeah, that's fair | |
| 20:11:49 | melwitt | we don't need to do a release to get the reno out there right? it will just get published to the docs automatically | |
| 20:11:51 | sean-k-mooney | mriedem: i dont think the reno is critcal jsut nice to have and sice we are still debating what the best solution is im not sure we should hold up the release for it | |
| 20:15:48 | melwitt | sean-k-mooney: ok, well, I guess according to that doc it's maybe probably ok to disable heartbeats if we have tcp keepalives in place https://www.rabbitmq.com/heartbeats.html#tcp-keepalives I think either way we'd want thorough testing of it before we go ahead and put the change out there | |
| 20:19:26 | sean-k-mooney | melwitt: this is what triplo configures https://github.com/openstack/tripleo-heat-templates/blob/d90434187315e96d3ebc184866dcd91954c0471d/deployment/kernel/kernel-baremetal-puppet.yaml#L117-L122 | |
| 20:21:48 | sean-k-mooney | it looks like that is even more agressive then rabbitmqs suggestions https://www.rabbitmq.com/networking.html#tcp-keepalives | |
| 20:22:30 | melwitt | mriedem: ack | |
| 20:22:49 | melwitt | oops | |
| 20:22:51 | melwitt | sean-k-mooney: ack | |
| 20:22:55 | sean-k-mooney | :) | |
| 20:25:39 | melwitt | mriedem: I'm fine either way releasing the reno now vs "soon" since AFAIK it doesn't have to be connected with the release, reno gets published independently. I was mostly saying I think we should publish a known issue vs not | |
| 20:26:11 | mriedem | i removed the -WIP so the release gods can do as they please | |
| 20:26:25 | melwitt | k | |
| 20:52:10 | sean-k-mooney | fun fact. resize between flavor with different size hugepages is broken | |
| 20:53:01 | sean-k-mooney | mainly becasue the nova.virt.hardware modeuld does not know what a rebuild or resize is and we end up using the page size form the embed instance numa toplogy instead of the new flavor/image | |
| 21:14:41 | melwitt | sean-k-mooney: reading through the full comments on https://bugs.launchpad.net/nova/+bug/1829062 it doesn't look like it's the same issue as the pause/resume rabbit reconnecting thing we were talking about | |
| 21:14:43 | openstack | Launchpad bug 1829062 in StarlingX "nova placement api non-responsive due to eventlet error" [Critical,In progress] - Assigned to Gerry Kopec (gerry-kopec) | |
| 21:15:30 | mriedem | melwitt: replies to your replies in https://review.opendev.org/#/c/638073/31 but +W - some of the stuff in there could go into your FUP | |
| 21:15:31 | melwitt | I would expect their wsgi config is default threads=1 | |
| 21:16:39 | melwitt | mriedem: yaaaaaay thanks. will add to the FollowUP | |
| 21:16:53 | sean-k-mooney | melwitt: the fact that its complaining about resuming on a different thread implies they have treads!=1 | |
| 21:17:55 | sean-k-mooney | but part of the annoyance it that placmeent has never used eventlet and we only started monkeypatching it in stien after extracting placment | |
| 21:18:06 | melwitt | sean-k-mooney: o. yeah | |
| 21:18:30 | sean-k-mooney | so people running runnign placment under the non extract version are running a config that we really have not tested well at all | |
| 21:18:41 | sean-k-mooney | since the gate uses extracted placement | |
| 21:19:04 | melwitt | right | |
| 21:19:25 | melwitt | ok, I'll throw another comment onto that bug to clarify the wsgi config thing | |
| 21:19:47 | melwitt | I'm trying to write the reno also | |
| 21:19:52 | sean-k-mooney | placement is written assuming eventlet is not used so there might be dragons there wen run under eventlet in general | |
| 21:20:03 | sean-k-mooney | ack | |
| 21:20:22 | melwitt | this whole issue makes me :*****( | |
| 21:21:02 | melwitt | have two paragraphs so far | |
| 21:21:09 | sean-k-mooney | there are too many ways to fix it all of which suck for different reasons | |
| 21:25:19 | mriedem | tonyb: this is all yours https://review.opendev.org/#/c/655692/ | |
| 21:26:06 | tonyb[m] | mriedem: Okay I | |
| 21:26:14 | tonyb[m] | 'll look after the school run | |
| 21:27:03 | aspiers | efried: you still around? | |
| 21:27:10 | efried | y | |
| 21:27:22 | aspiers | turns out that placement's inventory policy hardcodes a minimum of 1, not 0, for resource class inventory :-/ | |
| 21:27:35 | aspiers | for MEM_ENCRYPTION_CONTEXT, this is clearly a problem | |
| 21:28:14 | efried | why? | |
| 21:28:23 | aspiers | there are even explicit tests for this https://opendev.org/openstack/placement/src/branch/master/placement/tests/functional/gabbits/allocations.yaml#L107 | |
| 21:28:37 | aspiers | because hosts which can't do SEV need to be inventoried as 0 units | |
| 21:29:02 | efried | the test you pointed to is for an *allocation* requesting zero units of a thing. | |
| 21:29:15 | aspiers | yes, and it expects a 400 failure | |
| 21:29:23 | efried | I don't see the issue here. | |
| 21:29:56 | aspiers | the issue is that if I lower the minimum from 1 to 0 then these tests fail | |
| 21:30:04 | efried | "lower the minimum" - of what? | |
| 21:30:06 | aspiers | in the schema | |
| 21:30:14 | efried | argh, in *what* schema? | |
| 21:30:23 | aspiers | https://opendev.org/openstack/placement/src/branch/master/placement/schemas/inventory.py#L29 | |
| 21:30:49 | aspiers | without that, MEM_ENCRYPTION_CONTEXT can't be inventoried as 0 | |
| 21:31:01 | efried | If you can't do SEV, you don't put up any inventory of MEM_ENC_CTX. If you don't want SEV, you don't include MEM_ENC_CTX in your GET /a_c request or consequently in your allocation. | |
| 21:31:06 | aspiers | maybe we can tweak the schema to treat this resource class differently | |
| 21:31:08 | efried | You never inventory a thing with 0 total | |
| 21:31:09 | efried | no | |
| 21:31:13 | efried | do not muck with the schema | |
| 21:31:13 | aspiers | ohhh I see | |
| 21:31:16 | efried | the schema is right | |
| 21:31:28 | aspiers | OK, so just don't include it in the inventory | |
| 21:31:30 | efried | right | |
| 21:31:37 | aspiers | sounds good, I'll try that. thanks! | |
| 21:31:40 | efried | ++ | |
| 21:32:01 | sean-k-mooney | aspiers: ya you only create an invetory if there is capasity to consume | |
| 21:32:16 | sean-k-mooney | so if a host does not support sev dont create an sev inventory on the RP | |
| 21:34:06 | openstackgerrit | melanie witt proposed openstack/nova stable/stein: Add reno about nova-api eventlet monkey-patching and rabbitmq https://review.opendev.org/662095 | |
| 21:35:08 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Fix hard-delete of instance with soft-deleted referential constraints https://review.opendev.org/661398 | |
| 21:35:09 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Update usage in RT.drop_move_claim during confirm resize https://review.opendev.org/641806 | |
| 21:35:09 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Robustify attachment tracking in CinderFixtureNewAttachFlow https://review.opendev.org/658904 | |
| 21:35:10 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add Migration.cross_cell_move and get_by_uuid https://review.opendev.org/614012 | |
| 21:35:10 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add InstanceAction/Event create() method https://review.opendev.org/614036 | |
| 21:35:11 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add Instance.hidden field https://review.opendev.org/631123 | |
| 21:35:11 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add TargetDBSetupTask https://review.opendev.org/627892 | |
| 21:35:12 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add CrossCellMigrationTask https://review.opendev.org/631581 | |
| 21:35:12 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Execute TargetDBSetupTask https://review.opendev.org/633853 | |
| 21:35:13 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add prep_snapshot_based_resize_at_dest compute method https://review.opendev.org/633293 | |
| 21:35:13 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add PrepResizeAtDestTask https://review.opendev.org/627890 | |
| 21:35:14 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add prep_snapshot_based_resize_at_source compute method https://review.opendev.org/634832 | |
| 21:35:14 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add nova.compute.utils.delete_image https://review.opendev.org/637605 | |
| 21:35:15 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add PrepResizeAtSourceTask https://review.opendev.org/627891 | |
| 21:35:15 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Refactor ComputeManager.remove_volume_connection https://review.opendev.org/642183 | |
| 21:35:16 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add power_on kwarg to ComputeDriver.spawn() method https://review.opendev.org/642590 | |
| 21:35:16 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add finish_snapshot_based_resize_at_dest compute method https://review.opendev.org/635080 | |
| 21:35:17 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add FinishResizeAtDestTask https://review.opendev.org/635646 | |