Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-12-05
14:24:29 mriedem shilpasd: comments on https://review.opendev.org/#/c/694462/
14:25:15 shilpasd mriedem: thank you for review, will work on them
14:25:36 openstackgerrit Matt Riedemann proposed openstack/nova stable/stein: Cache security group driver https://review.opendev.org/697480
14:26:37 mriedem stephenfin: trying it
14:29:24 mriedem efried: this is why that multiple possible networks found thing is intermittent https://bugs.launchpad.net/nova/+bug/1831048
14:29:24 openstack Launchpad bug 1844568 in tempest "duplicate for #1831048 [compute] "create_test_server" if networks is undefined and more than one network is present" [Medium,In progress] - Assigned to Rodolfo Alonso (rodolfo-alonso-hernandez)
14:29:36 mriedem neutron tests create a new network at the same time we're creating a server and kaboom
14:30:01 mriedem though we shouldn't be running tempest.api.network tests in nova-next, but it could fail during scenario runs or something i guess?
14:30:08 mriedem stephenfin: i didn't get that problem
14:30:12 mriedem i'm on ubuntu bionic
14:30:20 stephenfin okay, it's a Fedora issue
14:30:22 stephenfin mriedem++ thanks
14:40:04 mriedem i wonder if the nova-next job hits that more often because of how we do concurrency differently in that job from others?
14:40:27 mriedem i think tempest-full runs api tests concurrently and scenario tests serially, and i think nova-next runs them all concurretly
14:43:01 mriedem yeah so i bet a network scenario test creates a network while the compute api tests are running and it blows up
14:44:04 sean-k-mooney mriedem: for the senario test do we not run those with concurancy 1
14:45:40 sean-k-mooney but yes i can see how it could fial in if a neutron test created a network in paralle with a compute test although ideally it would do that in a different tenant so that it could not affect other tests
14:46:03 mriedem there is probably a scenario test that creates a public network
14:46:16 mriedem and yes we run all tests, api and scenario, concurrently in nova-next
14:47:17 sean-k-mooney ah ok so we dont do the two pahse execution we do in tempst full wehre it runs all test excpet the senario tests first with concurrency and then runs the senario tests serially
14:47:35 sean-k-mooney ya that would proably make it more likely to fail in nova-next
14:49:57 mriedem all things i just said above
14:50:56 mriedem efried: gibi: so i think it was actually enabling tempest.scenario.test_minbw_allocation_placement.MinBwAllocationPlacementTest.test_qos_min_bw_allocation_basic in nova-next again that is making this worse
14:51:07 mriedem that test creates a public network
14:51:22 mriedem and everything that failed in this job was right around the time that test was running https://zuul.opendev.org/t/openstack/build/b7872ac3886e4cc7a060c1376a2a3395/log/job-output.txt#79124
14:51:37 mriedem which is also our spike in failures http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22Details%3A%20%7B'code'%3A%20409%2C%20'message'%3A%20'Multiple%20possible%20networks%20found%2C%20use%20a%20Network%20ID%20to%20be%20more%20specific.'%7D%5C%22%20AND%20tags%3A%5C%22console%5C%22%20AND%20voting%3A1&from=7d
14:57:43 openstackgerrit Matt Riedemann proposed openstack/nova master: Skip test_minbw_allocation_placement in nova-next job https://review.opendev.org/697491
14:57:49 mriedem gibi: efried: stephenfin: dansmith: if you wanna help the gate and love patriotism ^
14:58:20 stephenfin I hate patriotism, but I do have the feels for the gate
14:59:00 stephenfin yuuup, +2
15:01:02 stephenfin lyarwood: What was the reason for the request for configurables here? https://review.opendev.org/#/c/669674/3/nova/compute/manager.py@2613
15:01:49 stephenfin I want to -1 that patch _because_ of the seemingly extraneous knob but I could missing something
15:07:34 lyarwood stephenfin: looking
15:10:54 gibi mriedem: you are right. at the same time of the skip the test we need to keep an eye on the real fix https://review.opendev.org/#/c/682964
15:11:31 mriedem lyarwood: stephenfin: i left several comments in there too,
15:12:08 mriedem blindly retrying on *any* exception when doing something like deleting an attachment makes me nervous because if cinder returns a 404 because the original request completed and then we keep retrying, we're just going to keep retrying and getting 404s until we exhaust our retries
15:12:21 stephenfin good point
15:12:25 lyarwood stephenfin: no specific reason other than a personal dislike of hardcoded retry values tbh
15:12:27 mriedem iow, can we scope those retries to just 500 errors?
15:12:58 stephenfin and I have a personal dislike of knobs that no one ever touches
15:13:05 stephenfin Unstoppable force, meet immovable object :)
15:13:31 lyarwood Meets the guy with +2 over the guy with +1 so you win
15:13:53 lyarwood mriedem: yeah that's fair, I think a previous PS did previously and I've likely missed that cinder_exception.ClientException covers both?
15:14:48 lyarwood yeah it's the base, gah my bad.
15:15:26 mriedem right so https://github.com/openstack/python-cinderclient/blob/5.0.0/cinderclient/exceptions.py#L242 for 404 will create a NotFound(ClientException) and we'd retry on that
15:15:49 mriedem so we need to scope the retries to ClientExceptions with code==500 only imo
15:16:36 mriedem would also be nice to eventually add long_rpc_timeout to cinder for apis like this that have been known to take awhile
15:16:43 mriedem i know we've seen attachment delete timeout in the gate
15:16:57 mriedem surely red hat has some people working on cinder yeah?
15:17:37 lyarwood yup we do, I can follow up with them about that
15:17:46 mriedem https://bugs.launchpad.net/cinder/+bug/1763712 is where i suggested using long_rpc_timeout in cinder
15:17:47 openstack Launchpad bug 1763712 in Cinder "Unable to update the attachment.: MessagingTimeout" [Medium,Confirmed]
15:17:59 mriedem it shouldn't be that hard to add since the cinder rpc stuff is likely very similar to nova's
15:32:22 mriedem gibi: on https://review.opendev.org/#/c/693248/ can't we just hard-code the server created to use the net0 network - that's the one that has the port with the qos allocations
15:34:48 mriedem oh well i guess shared != external
15:35:15 mriedem so we got the shared public external network and nova puked because by default non-admins can't create servers on external networks
15:39:38 openstackgerrit Matt Riedemann proposed openstack/nova stable/rocky: Cache security group driver https://review.opendev.org/697505
15:44:16 openstackgerrit Matt Riedemann proposed openstack/nova master: Regression test for bug 1849657 https://review.opendev.org/693248
15:44:16 openstack bug 1849657 in OpenStack Compute (nova) train " allocation key is missing from the binding:profile of the neutron qos port when the server is created by a non-admin user" [Medium,Fix committed] https://launchpad.net/bugs/1849657 - Assigned to Balazs Gibizer (balazs-gibizer)
15:49:01 openstackgerrit Matt Riedemann proposed openstack/nova master: WIP: log when loading security group driver https://review.opendev.org/652783
15:53:54 efried mriedem: ack, thanks for that.
15:54:02 efried my cold has decided to kill me, so I may be sparse today.
15:54:46 mriedem RIP
15:55:37 mriedem efried: fyi i've got a thing in the nova meeting agenda for today but likely not going to be able to attend, it's just a question about if a spec is needed for that filter i'm working on
15:56:05 efried ack
15:56:17 mriedem also, the price is right is on in 4 minutes
15:56:31 mriedem good time to get on the couch with a wubby
16:00:45 efried stephenfin: https://review.opendev.org/#/c/696507/ if you want to update that commit message quick I'll fast approve, no need to rebase anything.
16:01:32 efried do we still say things like $show is on at $time? Who watches anything on a broadcast schedule?
16:02:17 stephenfin okidok
16:03:36 openstackgerrit Stephen Finucane proposed openstack/nova master: nova-net: Remove SG tests that don't apply to neutron https://review.opendev.org/696507
16:03:44 stephenfin efried: ^
16:04:18 efried stephenfin: +A, thanks for accommodating my pickiness.
16:04:29 stephenfin thanks for the review (y)
16:04:50 mordred efried: I think we maybe contracted the same plague
16:05:09 mordred mriedem: do we need an sdk release for the ironic patch we landed for you?
16:05:21 efried I suspect I actually got two things back to back. Which isn't supposed to happen.
16:07:11 mriedem mordred: at some point, but the nova blueprint that needs it isn't approved yet so it's not a rush
16:07:39 mriedem efried: i wouldn't dvr the price is right because i only watch it like twice a year
16:08:29 mriedem about the same amount of times i buy bacon - usually in the same week
16:08:49 mriedem ah the holidays, a time for bacon and day time tv game shows
16:10:59 efried wow, I buy bacon weekly, in multiple pounds.
16:12:31 mordred my biggest issue with bacon is that it's never possible to buy enough
16:13:19 mriedem no one else in my house will eat it (now that mortimer the dog is dead anyway) so i'm not making that smelly mess for myself every day
16:14:03 efried I make the whole mess at once and bag it. Bacon is a food that loses almost nothing reheated.
16:14:23 mriedem reheated in the microwave?
16:14:27 mriedem blasphemy
16:31:02 dansmith not blasphemy, plus it means bacon more often
16:31:06 dansmith BACON AT EVERY MEAL (tm)
16:32:31 efried Okay, I updated the meeting agenda. I'm going to go be miserable somewhere for a while.
16:34:59 openstackgerrit Matt Riedemann proposed openstack/nova stable/stein: Improve metadata server performance with large security groups https://review.opendev.org/695925
16:40:30 openstackgerrit Matt Riedemann proposed openstack/nova stable/rocky: Improve metadata server performance with large security groups https://review.opendev.org/697517
16:41:31 melwitt mriedem: I went out on a limb and proposed a backport to queens that landed in rocky, for cleaning up placement consumer records that no longer have allocations https://review.opendev.org/697398 I had to modify it a lot because the code was in different files in queens
16:42:46 openstackgerrit Matt Riedemann proposed openstack/nova stable/queens: Improve metadata server performance with large security groups https://review.opendev.org/697518
16:43:04 melwitt not sure whether that's going to fly but I figured I'd try because it's a pretty big unbounded table grow that otherwise has to be cleaned up manually
16:45:48 mriedem queens is EM so that's pretty low priority to address but fine to post if you're doing it for downstream anyway and if others want to use it
16:46:37 melwitt ah, I forgot it's EM. ok
16:46:49 melwitt understood
16:52:13 mriedem dansmith: how do you feel about backporting this? https://review.opendev.org/#/c/696953/
16:52:28 mriedem seems ok since it's tied to a long-standing bug but wanted a 2nd opinion

Earlier   Later