| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-06-06 | |||
| 18:02:25 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add functional recreate test for bug 1829479 and bug 1817833 https://review.opendev.org/663737 | |
| 18:02:27 | mriedem | cfriesen: sean-k-mooney: ^ | |
| 18:02:27 | openstack | bug 1829479 in OpenStack Compute (nova) "The allocation table has residual records when instance is evacuated and the source physical node is removed" [Medium,Triaged] https://launchpad.net/bugs/1829479 | |
| 18:02:28 | openstack | bug 1817833 in OpenStack Compute (nova) "Check compute_id existence when nova-compute reports info to placement" [Undecided,In progress] https://launchpad.net/bugs/1817833 - Assigned to xulei (605423512-j) | |
| 18:02:32 | mriedem | bugtastic | |
| 18:05:57 | sean-k-mooney | mriedem_lunch: the comment help but ya that looks correct | |
| 18:32:47 | efried | I wonder, if we made a bug whose title referenced another bug, and that bug's title referenced the first, could we make patchbot spin forever? | |
| 18:41:13 | openstackgerrit | Eric Fried proposed openstack/nova master: Introduces the openstacksdk to nova https://review.opendev.org/643664 | |
| 18:41:14 | openstackgerrit | Eric Fried proposed openstack/nova master: Use OpenStack SDK for placement https://review.opendev.org/656023 | |
| 18:41:35 | openstackgerrit | Eric Fried proposed openstack/nova master: Introduces SDK to IronicDriver and uses for node.get https://review.opendev.org/642899 | |
| 18:41:44 | openstackgerrit | Eric Fried proposed openstack/nova master: Use SDK instead of ironicclient for node.list https://review.opendev.org/656027 | |
| 18:41:52 | openstackgerrit | Eric Fried proposed openstack/nova master: Use SDK instead of ironicclient for validating instance and node https://review.opendev.org/656028 | |
| 18:42:01 | openstackgerrit | Eric Fried proposed openstack/nova master: Use SDK instead of ironicclient for setting instance id https://review.opendev.org/659690 | |
| 18:50:52 | openstackgerrit | Merged openstack/nova master: Add testing guide for down cells https://review.opendev.org/650167 | |
| 20:04:45 | mriedem | artom: i guess you missed these comments https://review.opendev.org/#/c/644881/19//COMMIT_MSG@11 | |
| 20:27:48 | mriedem | dansmith et al, I'm +2 on the nova/cyborg spec now https://review.opendev.org/#/c/603955/ | |
| 20:49:09 | mriedem | melwitt: see what you think about my proposed wording in this "specify az on unshelve" spec and if you agree i'll update it https://review.opendev.org/#/c/624689/10/specs/train/approved/support-specifying-az-when-restore-shelved-server.rst@62 | |
| 20:58:00 | efried | nova meeting in 3 minutes in #openstack-meeting | |
| 20:59:11 | melwitt | mriedem: ack | |
| 21:02:16 | efried | nova meeting now | |
| 21:07:23 | mriedem | no meeting?! | |
| 21:07:28 | mriedem | was reviewing something... | |
| 21:28:25 | melwitt | mriedem_afk: +1 to proposed wording | |
| 21:43:02 | openstackgerrit | Dustin Cowles proposed openstack/nova master: WIP: Use SDK instead of ironicclient for add/remove instance info from node https://review.opendev.org/659691 | |
| 21:53:14 | tonyb | kashyap: I don't but it's on my list of things to get workign this year | |
| 21:53:19 | tonyb | kashyap: 'sup? | |
| 22:14:36 | openstackgerrit | Merged openstack/nova master: [Docs] Update the confusing console output https://review.opendev.org/589004 | |
| 22:47:09 | mnaser | alright. has anyone ever ran into this? i'm convinced this feels like an oslo bug or something. | |
| 22:47:31 | mnaser | i've noticed that if i have a 3 node rabbitmq cluster, and the cluster loses a node and regains it, for some reason, its like nothing gets routed across rabbitmq | |
| 22:47:43 | mnaser | in this case, an instance gets booted and goes into SCHEDULING state and gets stuck forever | |
| 22:48:41 | mnaser | this is with rabbitmq 3.6.16 -- the only way i've solved it often is by deleting all the queues and restarting nova | |
| 22:48:54 | mnaser | the one 'odd' thing is with osa, we deploy a cluster with ssl for rabbitmq | |
| 22:49:32 | mnaser | i have a (non-prod) cluster stuck at this state right now and i really want to get to the bottom of what/why | |
| 22:51:01 | mnaser | all queues listing 0 messages.. except for notifications | |
| 22:59:40 | melwitt | mnaser: feel like you're going to need to get some other operator input about that one.. I haven't personally heard of it | |
| 23:00:03 | melwitt | imacdonn ^ | |
| 23:00:06 | mnaser | melwitt: i've ran into it a bunch of times and had others who ran to it and the "drop all queues" thing fixed it.. | |
| 23:00:26 | melwitt | I also asked penick to look at your messages here and let us know if he's seen it | |
| 23:01:18 | mnaser | melwitt: yay cool thanks, i'm going to continue digging, this is a non prod environment so i have the time to dig into it | |
| 23:01:25 | mnaser | usually things are under fire when i need to do this | |
| 23:01:38 | melwitt | yeah, that's awesome you got a repro in non-prod | |
| 23:02:14 | mnaser | feels like it's gonna be a really annoying one though | |
| 23:03:26 | melwitt | yeah. another person we could ask is kgiusti, he knows oslo.messaging | |
| 23:04:22 | mnaser | so, instance in scheduling state, 7 messages in a bunch of scheduler_fanout_* queues | |
| 23:04:44 | mnaser | but nothing happening, so i guess the conductor is doing what its suppoed to do | |
| 23:04:58 | penick | @mnaser we've had weird issues with rabbitmq and ssl. Mostly with performance and the queue faceplanting though. I don't think we've ever had to wipe and recreate all queues. | |
| 23:05:24 | mnaser | :( sads, yeah, it's pretty destructive but not sure.. | |
| 23:05:28 | penick | Though the way we worked around the ssl issue is we switched from native SSL to using stunnel. Which is awful | |
| 23:05:38 | penick | something about native rabbit ssl is borked | |
| 23:05:42 | mnaser | oh wow, is the native rabbit ssl that bad | |
| 23:05:48 | mnaser | can i ask what release were you running then (or now)? | |
| 23:05:54 | penick | Ocata | |
| 23:05:58 | penick | oh, of rabbit | |
| 23:05:59 | penick | uh, one sec | |
| 23:06:05 | mnaser | thanks :> | |
| 23:06:12 | mnaser | there is exactly 48 scheduler_fanout_* queues | |
| 23:06:54 | mnaser | each controller has 16 nova-scheduler processes, which adds up to 48 | |
| 23:07:08 | mnaser | so the theory is nova-conductor is doing its job but the scheduler just isn't picking up work off the queue.. | |
| 23:07:56 | penick | 3.6.15 | |
| 23:08:09 | melwitt | penick: I wondered whether it might be related to queue mirroring or non, clustered or non, how it behaves when a node leaves and rejoins | |
| 23:08:22 | mnaser | yeah we're at 3.6.16 so we're not far away | |
| 23:08:38 | mnaser | melwitt: yeah so actually i wondered if that was maybe the issue.. | |
| 23:08:46 | imacdonn | FWIW, I don't run clustered rabbitmq - just a single instance | |
| 23:09:01 | melwitt | I had thought there might have been some configuration where that doesn't work well, but I can't remember anything more about it | |
| 23:09:12 | mnaser | imacdonn: you're living life | |
| 23:09:40 | melwitt | :D | |
| 23:09:44 | imacdonn | I think so :) My career experience with clustering amounts to "it usually causes more problems than it solves" | |
| 23:09:48 | mnaser | thanks penick for the insights, i guess we might be hitting the weird ssl things you might have ran into | |
| 23:10:15 | mnaser | i guess i can bring the log levels up on the mq stuff and see how/why its getting bound and why its not picking up any messages | |
| 23:10:44 | imacdonn | yeah, there's stuff you can turn on with default_log_levels | |
| 23:11:12 | mnaser | i mean i think the api is doing its job pushing things out to the scheduler fanout, but then things faceplant there | |
| 23:12:01 | melwitt | log levels in python are totally easy to control (not) | |
| 23:12:29 | mnaser | i'll at least shut down the other 2 schedulers so i can debug on one ost | |
| 23:16:01 | mnaser | oh that's interesitng | |
| 23:16:04 | mnaser | it actually schedules it | |
| 23:16:28 | mnaser | it even puts allocations in placement | |
| 23:16:34 | mnaser | sends notifications.. | |
| 23:18:32 | melwitt | so, it doesn't get past the scheduler? doesn't move on back to conductor and then to compute? | |
| 23:20:04 | mnaser | it gets to the scheduler, then the scheduler sends something, gets a reply, and it seems to stall out there | |
| 23:22:40 | mnaser | i guess i need to find out what happens after an instance is scheduled, it send something back to the conductor (or it casts to that cell?) | |
| 23:22:51 | penick | @mnaser if it helps we don't use durable queues | |
| 23:22:56 | melwitt | mnaser: this is old but sounds kind of the same? https://github.com/rabbitmq/rabbitmq-server/issues/224 | |
| 23:23:19 | mnaser | hmm, wonder what the defaults are | |
| 23:23:42 | mnaser | oslo_messaging_rabbit.amqp_durable_queues = False yeah same here | |
| 23:24:51 | mnaser | oh interesting | |
| 23:25:39 | melwitt | are you using mirrored queues? | |
| 23:26:41 | melwitt | penick doesn't use mirrored and I think that's the thing that had issues that I'm vaguely remembering. and that issue I linked from rabbitmq also seems to have to do with mirrored queues | |
| 23:26:49 | mnaser | i think we mirror a subset of queus in OSA | |
| 23:29:19 | melwitt | that issue was fixed eons ago but was about "some kind of race with mirrored, auto_delete queues" | |
| 23:29:27 | mnaser | https://github.com/openstack/openstack-ansible/blob/dc5729ad6f10b7e6083d60c2270f75090cf3d5f4/inventory/group_vars/all/infra.yml#L27-L31 | |
| 23:29:56 | melwitt | I wonder if a new race could have cropped up recently | |
| 23:31:15 | mnaser | so question, after an instance is scheduled, does it then contact the rabbitmq cluster @ the cell? | |
| 23:33:58 | melwitt | we do a sync call over rpc from conductor to scheduler, get the result in conductor, then conductor async casts to compute over rpc | |
| 23:34:35 | melwitt | it sounds like what you're saying is scheduler does fine, creates allocs in placement etc, but then the reply never goes back to conductor over rpc | |
| 23:35:04 | mnaser | it looks like the reply to the conductor does get sent out.. so i need to maybe go back and check the conductor this time i guess | |
| 23:35:09 | melwitt | if that is the case, then it's never getting the to cell mq part | |
| 23:35:52 | melwitt | if the super conductor receives the reply, the next step is super conductor casts to the cell mq | |
| 23:36:05 | mnaser | ok, ill move away from scheduler and start looking at conductor now | |
| 23:37:12 | mnaser | conductor, the worlds quietest service | |
| 23:37:16 | melwitt | :) | |