Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-12-05
14:47:35 sean-k-mooney ya that would proably make it more likely to fail in nova-next
14:49:57 mriedem all things i just said above
14:50:56 mriedem efried: gibi: so i think it was actually enabling tempest.scenario.test_minbw_allocation_placement.MinBwAllocationPlacementTest.test_qos_min_bw_allocation_basic in nova-next again that is making this worse
14:51:07 mriedem that test creates a public network
14:51:22 mriedem and everything that failed in this job was right around the time that test was running https://zuul.opendev.org/t/openstack/build/b7872ac3886e4cc7a060c1376a2a3395/log/job-output.txt#79124
14:51:37 mriedem which is also our spike in failures http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22Details%3A%20%7B'code'%3A%20409%2C%20'message'%3A%20'Multiple%20possible%20networks%20found%2C%20use%20a%20Network%20ID%20to%20be%20more%20specific.'%7D%5C%22%20AND%20tags%3A%5C%22console%5C%22%20AND%20voting%3A1&from=7d
14:57:43 openstackgerrit Matt Riedemann proposed openstack/nova master: Skip test_minbw_allocation_placement in nova-next job https://review.opendev.org/697491
14:57:49 mriedem gibi: efried: stephenfin: dansmith: if you wanna help the gate and love patriotism ^
14:58:20 stephenfin I hate patriotism, but I do have the feels for the gate
14:59:00 stephenfin yuuup, +2
15:01:02 stephenfin lyarwood: What was the reason for the request for configurables here? https://review.opendev.org/#/c/669674/3/nova/compute/manager.py@2613
15:01:49 stephenfin I want to -1 that patch _because_ of the seemingly extraneous knob but I could missing something
15:07:34 lyarwood stephenfin: looking
15:10:54 gibi mriedem: you are right. at the same time of the skip the test we need to keep an eye on the real fix https://review.opendev.org/#/c/682964
15:11:31 mriedem lyarwood: stephenfin: i left several comments in there too,
15:12:08 mriedem blindly retrying on *any* exception when doing something like deleting an attachment makes me nervous because if cinder returns a 404 because the original request completed and then we keep retrying, we're just going to keep retrying and getting 404s until we exhaust our retries
15:12:21 stephenfin good point
15:12:25 lyarwood stephenfin: no specific reason other than a personal dislike of hardcoded retry values tbh
15:12:27 mriedem iow, can we scope those retries to just 500 errors?
15:12:58 stephenfin and I have a personal dislike of knobs that no one ever touches
15:13:05 stephenfin Unstoppable force, meet immovable object :)
15:13:31 lyarwood Meets the guy with +2 over the guy with +1 so you win
15:13:53 lyarwood mriedem: yeah that's fair, I think a previous PS did previously and I've likely missed that cinder_exception.ClientException covers both?
15:14:48 lyarwood yeah it's the base, gah my bad.
15:15:26 mriedem right so https://github.com/openstack/python-cinderclient/blob/5.0.0/cinderclient/exceptions.py#L242 for 404 will create a NotFound(ClientException) and we'd retry on that
15:15:49 mriedem so we need to scope the retries to ClientExceptions with code==500 only imo
15:16:36 mriedem would also be nice to eventually add long_rpc_timeout to cinder for apis like this that have been known to take awhile
15:16:43 mriedem i know we've seen attachment delete timeout in the gate
15:16:57 mriedem surely red hat has some people working on cinder yeah?
15:17:37 lyarwood yup we do, I can follow up with them about that
15:17:46 mriedem https://bugs.launchpad.net/cinder/+bug/1763712 is where i suggested using long_rpc_timeout in cinder
15:17:47 openstack Launchpad bug 1763712 in Cinder "Unable to update the attachment.: MessagingTimeout" [Medium,Confirmed]
15:17:59 mriedem it shouldn't be that hard to add since the cinder rpc stuff is likely very similar to nova's
15:32:22 mriedem gibi: on https://review.opendev.org/#/c/693248/ can't we just hard-code the server created to use the net0 network - that's the one that has the port with the qos allocations
15:34:48 mriedem oh well i guess shared != external
15:35:15 mriedem so we got the shared public external network and nova puked because by default non-admins can't create servers on external networks
15:39:38 openstackgerrit Matt Riedemann proposed openstack/nova stable/rocky: Cache security group driver https://review.opendev.org/697505
15:44:16 openstackgerrit Matt Riedemann proposed openstack/nova master: Regression test for bug 1849657 https://review.opendev.org/693248
15:44:16 openstack bug 1849657 in OpenStack Compute (nova) train " allocation key is missing from the binding:profile of the neutron qos port when the server is created by a non-admin user" [Medium,Fix committed] https://launchpad.net/bugs/1849657 - Assigned to Balazs Gibizer (balazs-gibizer)
15:49:01 openstackgerrit Matt Riedemann proposed openstack/nova master: WIP: log when loading security group driver https://review.opendev.org/652783
15:53:54 efried mriedem: ack, thanks for that.
15:54:02 efried my cold has decided to kill me, so I may be sparse today.
15:54:46 mriedem RIP
15:55:37 mriedem efried: fyi i've got a thing in the nova meeting agenda for today but likely not going to be able to attend, it's just a question about if a spec is needed for that filter i'm working on
15:56:05 efried ack
15:56:17 mriedem also, the price is right is on in 4 minutes
15:56:31 mriedem good time to get on the couch with a wubby
16:00:45 efried stephenfin: https://review.opendev.org/#/c/696507/ if you want to update that commit message quick I'll fast approve, no need to rebase anything.
16:01:32 efried do we still say things like $show is on at $time? Who watches anything on a broadcast schedule?
16:02:17 stephenfin okidok
16:03:36 openstackgerrit Stephen Finucane proposed openstack/nova master: nova-net: Remove SG tests that don't apply to neutron https://review.opendev.org/696507
16:03:44 stephenfin efried: ^
16:04:18 efried stephenfin: +A, thanks for accommodating my pickiness.
16:04:29 stephenfin thanks for the review (y)
16:04:50 mordred efried: I think we maybe contracted the same plague
16:05:09 mordred mriedem: do we need an sdk release for the ironic patch we landed for you?
16:05:21 efried I suspect I actually got two things back to back. Which isn't supposed to happen.
16:07:11 mriedem mordred: at some point, but the nova blueprint that needs it isn't approved yet so it's not a rush
16:07:39 mriedem efried: i wouldn't dvr the price is right because i only watch it like twice a year
16:08:29 mriedem about the same amount of times i buy bacon - usually in the same week
16:08:49 mriedem ah the holidays, a time for bacon and day time tv game shows
16:10:59 efried wow, I buy bacon weekly, in multiple pounds.
16:12:31 mordred my biggest issue with bacon is that it's never possible to buy enough
16:13:19 mriedem no one else in my house will eat it (now that mortimer the dog is dead anyway) so i'm not making that smelly mess for myself every day
16:14:03 efried I make the whole mess at once and bag it. Bacon is a food that loses almost nothing reheated.
16:14:23 mriedem reheated in the microwave?
16:14:27 mriedem blasphemy
16:31:02 dansmith not blasphemy, plus it means bacon more often
16:31:06 dansmith BACON AT EVERY MEAL (tm)
16:32:31 efried Okay, I updated the meeting agenda. I'm going to go be miserable somewhere for a while.
16:34:59 openstackgerrit Matt Riedemann proposed openstack/nova stable/stein: Improve metadata server performance with large security groups https://review.opendev.org/695925
16:40:30 openstackgerrit Matt Riedemann proposed openstack/nova stable/rocky: Improve metadata server performance with large security groups https://review.opendev.org/697517
16:41:31 melwitt mriedem: I went out on a limb and proposed a backport to queens that landed in rocky, for cleaning up placement consumer records that no longer have allocations https://review.opendev.org/697398 I had to modify it a lot because the code was in different files in queens
16:42:46 openstackgerrit Matt Riedemann proposed openstack/nova stable/queens: Improve metadata server performance with large security groups https://review.opendev.org/697518
16:43:04 melwitt not sure whether that's going to fly but I figured I'd try because it's a pretty big unbounded table grow that otherwise has to be cleaned up manually
16:45:48 mriedem queens is EM so that's pretty low priority to address but fine to post if you're doing it for downstream anyway and if others want to use it
16:46:37 melwitt ah, I forgot it's EM. ok
16:46:49 melwitt understood
16:52:13 mriedem dansmith: how do you feel about backporting this? https://review.opendev.org/#/c/696953/
16:52:28 mriedem seems ok since it's tied to a long-standing bug but wanted a 2nd opinion
16:53:04 dansmith mriedem: yeah, I mean... as long as the old version has a sufficient oslo.messaging and the config knob, I can't really think of any reason not to do tose
16:53:05 dansmith *those
16:53:47 openstackgerrit Matt Riedemann proposed openstack/nova stable/pike: Improve metadata server performance with large security groups https://review.opendev.org/697523
16:56:25 mriedem lyarwood: i'd be +2 on this https://review.opendev.org/#/c/696953/ but i'm not sure how pedantic you want to be about using the correct source commit hash
16:57:21 mriedem given train and stein had conflicts which you've already resolved i'm assuming you don't want to redo all of that
16:59:35 mriedem nvm i just +2ed
17:01:05 mriedem dansmith: you want to get this https://review.opendev.org/#/c/696339/ and this then https://review.opendev.org/696953
17:03:22 dansmith yar
17:06:31 lyarwood mriedem: sorry was on a call, I really don't care about the source commit hash with backports given the Change IDs tbh. That said I have no idea how I ended up using the wrong one for this.
17:06:34 lyarwood mriedem: and thanks
17:33:14 openstackgerrit Matt Riedemann proposed openstack/nova stable/pike: rt: only map compute node if we created it https://review.opendev.org/676463
17:54:48 openstackgerrit Matt Riedemann proposed openstack/nova stable/ocata: Add functional regression test for bug 1849409 https://review.opendev.org/690746
17:54:48 openstack bug 1849409 in OpenStack Compute (nova) queens "openstack server list --deleted --limit -1 hangs" [High,In progress] https://launchpad.net/bugs/1849409 - Assigned to Matt Riedemann (mriedem)
17:54:48 openstackgerrit Matt Riedemann proposed openstack/nova stable/ocata: Fix listing deleted servers with a marker https://review.opendev.org/690747
17:57:21 openstackgerrit Stephen Finucane proposed openstack/nova master: functional: Make '_IntegratedTestBase' subclass 'InstanceHelperMixin' https://review.opendev.org/689182
17:57:21 openstackgerrit Stephen Finucane proposed openstack/nova master: functional: Remove 'get_invalid_image' https://review.opendev.org/697454
17:57:22 openstackgerrit Stephen Finucane proposed openstack/nova master: functional: Add unified '_build_server' helper function https://review.opendev.org/697537
17:57:22 openstackgerrit Stephen Finucane proposed openstack/nova master: functional: Add unified '_(build|create)_flavor' helper functions https://review.opendev.org/697538
17:57:23 openstackgerrit Stephen Finucane proposed openstack/nova master: functional: Add '_delete_server' to 'InstanceHelperMixin' https://review.opendev.org/697539
17:57:23 openstackgerrit Stephen Finucane proposed openstack/nova master: functional: Add '_create_server' helper https://review.opendev.org/697540

Earlier   Later