Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-06-06
18:02:32 mriedem bugtastic
18:05:57 sean-k-mooney mriedem_lunch: the comment help but ya that looks correct
18:32:47 efried I wonder, if we made a bug whose title referenced another bug, and that bug's title referenced the first, could we make patchbot spin forever?
18:41:13 openstackgerrit Eric Fried proposed openstack/nova master: Introduces the openstacksdk to nova https://review.opendev.org/643664
18:41:14 openstackgerrit Eric Fried proposed openstack/nova master: Use OpenStack SDK for placement https://review.opendev.org/656023
18:41:35 openstackgerrit Eric Fried proposed openstack/nova master: Introduces SDK to IronicDriver and uses for node.get https://review.opendev.org/642899
18:41:44 openstackgerrit Eric Fried proposed openstack/nova master: Use SDK instead of ironicclient for node.list https://review.opendev.org/656027
18:41:52 openstackgerrit Eric Fried proposed openstack/nova master: Use SDK instead of ironicclient for validating instance and node https://review.opendev.org/656028
18:42:01 openstackgerrit Eric Fried proposed openstack/nova master: Use SDK instead of ironicclient for setting instance id https://review.opendev.org/659690
18:50:52 openstackgerrit Merged openstack/nova master: Add testing guide for down cells https://review.opendev.org/650167
20:04:45 mriedem artom: i guess you missed these comments https://review.opendev.org/#/c/644881/19//COMMIT_MSG@11
20:27:48 mriedem dansmith et al, I'm +2 on the nova/cyborg spec now https://review.opendev.org/#/c/603955/
20:49:09 mriedem melwitt: see what you think about my proposed wording in this "specify az on unshelve" spec and if you agree i'll update it https://review.opendev.org/#/c/624689/10/specs/train/approved/support-specifying-az-when-restore-shelved-server.rst@62
20:58:00 efried nova meeting in 3 minutes in #openstack-meeting
20:59:11 melwitt mriedem: ack
21:02:16 efried nova meeting now
21:07:23 mriedem no meeting?!
21:07:28 mriedem was reviewing something...
21:28:25 melwitt mriedem_afk: +1 to proposed wording
21:43:02 openstackgerrit Dustin Cowles proposed openstack/nova master: WIP: Use SDK instead of ironicclient for add/remove instance info from node https://review.opendev.org/659691
21:53:14 tonyb kashyap: I don't but it's on my list of things to get workign this year
21:53:19 tonyb kashyap: 'sup?
22:14:36 openstackgerrit Merged openstack/nova master: [Docs] Update the confusing console output https://review.opendev.org/589004
22:47:09 mnaser alright. has anyone ever ran into this? i'm convinced this feels like an oslo bug or something.
22:47:31 mnaser i've noticed that if i have a 3 node rabbitmq cluster, and the cluster loses a node and regains it, for some reason, its like nothing gets routed across rabbitmq
22:47:43 mnaser in this case, an instance gets booted and goes into SCHEDULING state and gets stuck forever
22:48:41 mnaser this is with rabbitmq 3.6.16 -- the only way i've solved it often is by deleting all the queues and restarting nova
22:48:54 mnaser the one 'odd' thing is with osa, we deploy a cluster with ssl for rabbitmq
22:49:32 mnaser i have a (non-prod) cluster stuck at this state right now and i really want to get to the bottom of what/why
22:51:01 mnaser all queues listing 0 messages.. except for notifications
22:59:40 melwitt mnaser: feel like you're going to need to get some other operator input about that one.. I haven't personally heard of it
23:00:03 melwitt imacdonn ^
23:00:06 mnaser melwitt: i've ran into it a bunch of times and had others who ran to it and the "drop all queues" thing fixed it..
23:00:26 melwitt I also asked penick to look at your messages here and let us know if he's seen it
23:01:18 mnaser melwitt: yay cool thanks, i'm going to continue digging, this is a non prod environment so i have the time to dig into it
23:01:25 mnaser usually things are under fire when i need to do this
23:01:38 melwitt yeah, that's awesome you got a repro in non-prod
23:02:14 mnaser feels like it's gonna be a really annoying one though
23:03:26 melwitt yeah. another person we could ask is kgiusti, he knows oslo.messaging
23:04:22 mnaser so, instance in scheduling state, 7 messages in a bunch of scheduler_fanout_* queues
23:04:44 mnaser but nothing happening, so i guess the conductor is doing what its suppoed to do
23:04:58 penick @mnaser we've had weird issues with rabbitmq and ssl. Mostly with performance and the queue faceplanting though. I don't think we've ever had to wipe and recreate all queues.
23:05:24 mnaser :( sads, yeah, it's pretty destructive but not sure..
23:05:28 penick Though the way we worked around the ssl issue is we switched from native SSL to using stunnel. Which is awful
23:05:38 penick something about native rabbit ssl is borked
23:05:42 mnaser oh wow, is the native rabbit ssl that bad
23:05:48 mnaser can i ask what release were you running then (or now)?
23:05:54 penick Ocata
23:05:58 penick oh, of rabbit
23:05:59 penick uh, one sec
23:06:05 mnaser thanks :>
23:06:12 mnaser there is exactly 48 scheduler_fanout_* queues
23:06:54 mnaser each controller has 16 nova-scheduler processes, which adds up to 48
23:07:08 mnaser so the theory is nova-conductor is doing its job but the scheduler just isn't picking up work off the queue..
23:07:56 penick 3.6.15
23:08:09 melwitt penick: I wondered whether it might be related to queue mirroring or non, clustered or non, how it behaves when a node leaves and rejoins
23:08:22 mnaser yeah we're at 3.6.16 so we're not far away
23:08:38 mnaser melwitt: yeah so actually i wondered if that was maybe the issue..
23:08:46 imacdonn FWIW, I don't run clustered rabbitmq - just a single instance
23:09:01 melwitt I had thought there might have been some configuration where that doesn't work well, but I can't remember anything more about it
23:09:12 mnaser imacdonn: you're living life
23:09:40 melwitt :D
23:09:44 imacdonn I think so :) My career experience with clustering amounts to "it usually causes more problems than it solves"
23:09:48 mnaser thanks penick for the insights, i guess we might be hitting the weird ssl things you might have ran into
23:10:15 mnaser i guess i can bring the log levels up on the mq stuff and see how/why its getting bound and why its not picking up any messages
23:10:44 imacdonn yeah, there's stuff you can turn on with default_log_levels
23:11:12 mnaser i mean i think the api is doing its job pushing things out to the scheduler fanout, but then things faceplant there
23:12:01 melwitt log levels in python are totally easy to control (not)
23:12:29 mnaser i'll at least shut down the other 2 schedulers so i can debug on one ost
23:16:01 mnaser oh that's interesitng
23:16:04 mnaser it actually schedules it
23:16:28 mnaser it even puts allocations in placement
23:16:34 mnaser sends notifications..
23:18:32 melwitt so, it doesn't get past the scheduler? doesn't move on back to conductor and then to compute?
23:20:04 mnaser it gets to the scheduler, then the scheduler sends something, gets a reply, and it seems to stall out there
23:22:40 mnaser i guess i need to find out what happens after an instance is scheduled, it send something back to the conductor (or it casts to that cell?)
23:22:51 penick @mnaser if it helps we don't use durable queues
23:22:56 melwitt mnaser: this is old but sounds kind of the same? https://github.com/rabbitmq/rabbitmq-server/issues/224
23:23:19 mnaser hmm, wonder what the defaults are
23:23:42 mnaser oslo_messaging_rabbit.amqp_durable_queues = False yeah same here
23:24:51 mnaser oh interesting
23:25:39 melwitt are you using mirrored queues?
23:26:41 melwitt penick doesn't use mirrored and I think that's the thing that had issues that I'm vaguely remembering. and that issue I linked from rabbitmq also seems to have to do with mirrored queues
23:26:49 mnaser i think we mirror a subset of queus in OSA
23:29:19 melwitt that issue was fixed eons ago but was about "some kind of race with mirrored, auto_delete queues"
23:29:27 mnaser https://github.com/openstack/openstack-ansible/blob/dc5729ad6f10b7e6083d60c2270f75090cf3d5f4/inventory/group_vars/all/infra.yml#L27-L31
23:29:56 melwitt I wonder if a new race could have cropped up recently
23:31:15 mnaser so question, after an instance is scheduled, does it then contact the rabbitmq cluster @ the cell?
23:33:58 melwitt we do a sync call over rpc from conductor to scheduler, get the result in conductor, then conductor async casts to compute over rpc
23:34:35 melwitt it sounds like what you're saying is scheduler does fine, creates allocs in placement etc, but then the reply never goes back to conductor over rpc
23:35:04 mnaser it looks like the reply to the conductor does get sent out.. so i need to maybe go back and check the conductor this time i guess
23:35:09 melwitt if that is the case, then it's never getting the to cell mq part
23:35:52 melwitt if the super conductor receives the reply, the next step is super conductor casts to the cell mq
23:36:05 mnaser ok, ill move away from scheduler and start looking at conductor now
23:37:12 mnaser conductor, the worlds quietest service
23:37:16 melwitt :)
23:38:12 melwitt you know.... we do cache mq connection info. so if you find it's hanging up at the step where super conductor is trying to drop the "build instance" message onto the cell mq and that cached info is no longer valid, that could be the culprit
23:38:40 melwitt but no, you said things return to normal if you kill the rabbit queues and recreate them without restarting any nova services yeah?
23:38:54 melwitt that would mean the nova cache thing doesn't matter
23:40:23 mnaser ok so schedule_and_build_instances gets to the point where its gets an asigned host

Earlier   Later