| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-06-06 | |||
| 17:15:21 | mriedem | heads in the sand and whatnot | |
| 17:16:13 | melwitt | I think we switched our default network backend to ovn recently. sean-k-mooney correct me | |
| 17:16:18 | mriedem | artom: if you're cool with dropping the nova-live-migratoin rename + resize revert stuff (i still thing the rename is good at some point, just not while hurrying for a fix) i'm ok with adding a depends-on from my unskip to your fix | |
| 17:16:25 | sean-k-mooney | hehe well for 15 we were planning to test with ovn but it has other issues | |
| 17:16:39 | mriedem | ooo the plot thickens | |
| 17:16:48 | artom | mriedem, 🤝 | |
| 17:17:01 | artom | (That's a handshake shaped like a yellow heart) | |
| 17:17:10 | mriedem | ok - yeah pidgin doesn't render those | |
| 17:17:29 | melwitt | I can see it. looks cool | |
| 17:17:35 | artom | sean-k-mooney, so what's geneve? another name for OVN? | |
| 17:17:40 | sean-k-mooney | melwitt: yes but ovn has only works when we disabel waiting for the network-evnets on live migation so i think they wanted to test ml2/ovn with iptabls as a fallback plan | |
| 17:17:45 | artom | 'cuz that's the default networking thingee in our ospd custom job | |
| 17:17:59 | sean-k-mooney | geneve is a l3 tunneling protocol like vxlan | |
| 17:18:20 | sean-k-mooney | and its the protocol ovn uses for its networking by defualt | |
| 17:19:02 | sean-k-mooney | artom: you are refering to the neutron type driver which for ovn would be geneve | |
| 17:19:16 | artom | sean-k-mooney, "Choose the network variant (default for OSP15 is 'geneve') | |
| 17:19:16 | artom | For the geneve option set the NETWORK_OVN paremeter to yes." | |
| 17:19:26 | artom | (Form the Jenkins job build page) | |
| 17:19:32 | artom | So yeah, looks like OSP15 default is OVN | |
| 17:20:13 | sean-k-mooney | mriedem: but to your point yes this has always been broken in one way or another | |
| 17:21:09 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: Revert resize: wait for events according to hybrid plug https://review.opendev.org/644881 | |
| 17:21:09 | sean-k-mooney | but we i think we only started waithing in stien so its only showing up now as they are testing Stien/osp15 | |
| 17:22:08 | artom | sean-k-mooney, I guess you can abandon https://review.opendev.org/#/c/660782/ now, since we're settling on https://review.opendev.org/#/c/663405/1 | |
| 17:23:20 | sean-k-mooney | am sure i never intended it to merge anyway so if you dont need it for testing then sure ill abandon | |
| 17:24:05 | artom | We could replace it with another DNM on top of mriedem's tempest patch to run tempest-slow with hybrid plug | |
| 17:24:12 | artom | For the extra confidence | |
| 17:24:34 | artom | But we've seen it pass already, not sure how necessary a new patch would be | |
| 17:24:51 | sean-k-mooney | well i can add a depens on to matts change too but ok ill just abandon it | |
| 17:25:13 | artom | Well no you can't, because your change was in the multinode job | |
| 17:25:30 | artom | Whereas mriedem's change unksips a test in tempest-slow | |
| 17:25:53 | sean-k-mooney | yes but may change didnt diable tempest-slow | |
| 17:26:00 | sean-k-mooney | but anyway its abandoned | |
| 17:26:20 | sean-k-mooney | you know how to reporduce this if you need to so i dont need to keep a patch open | |
| 17:26:43 | artom | I need lunch | |
| 18:02:25 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add functional recreate test for bug 1829479 and bug 1817833 https://review.opendev.org/663737 | |
| 18:02:27 | openstack | bug 1829479 in OpenStack Compute (nova) "The allocation table has residual records when instance is evacuated and the source physical node is removed" [Medium,Triaged] https://launchpad.net/bugs/1829479 | |
| 18:02:27 | mriedem | cfriesen: sean-k-mooney: ^ | |
| 18:02:28 | openstack | bug 1817833 in OpenStack Compute (nova) "Check compute_id existence when nova-compute reports info to placement" [Undecided,In progress] https://launchpad.net/bugs/1817833 - Assigned to xulei (605423512-j) | |
| 18:02:32 | mriedem | bugtastic | |
| 18:05:57 | sean-k-mooney | mriedem_lunch: the comment help but ya that looks correct | |
| 18:32:47 | efried | I wonder, if we made a bug whose title referenced another bug, and that bug's title referenced the first, could we make patchbot spin forever? | |
| 18:41:13 | openstackgerrit | Eric Fried proposed openstack/nova master: Introduces the openstacksdk to nova https://review.opendev.org/643664 | |
| 18:41:14 | openstackgerrit | Eric Fried proposed openstack/nova master: Use OpenStack SDK for placement https://review.opendev.org/656023 | |
| 18:41:35 | openstackgerrit | Eric Fried proposed openstack/nova master: Introduces SDK to IronicDriver and uses for node.get https://review.opendev.org/642899 | |
| 18:41:44 | openstackgerrit | Eric Fried proposed openstack/nova master: Use SDK instead of ironicclient for node.list https://review.opendev.org/656027 | |
| 18:41:52 | openstackgerrit | Eric Fried proposed openstack/nova master: Use SDK instead of ironicclient for validating instance and node https://review.opendev.org/656028 | |
| 18:42:01 | openstackgerrit | Eric Fried proposed openstack/nova master: Use SDK instead of ironicclient for setting instance id https://review.opendev.org/659690 | |
| 18:50:52 | openstackgerrit | Merged openstack/nova master: Add testing guide for down cells https://review.opendev.org/650167 | |
| 20:04:45 | mriedem | artom: i guess you missed these comments https://review.opendev.org/#/c/644881/19//COMMIT_MSG@11 | |
| 20:27:48 | mriedem | dansmith et al, I'm +2 on the nova/cyborg spec now https://review.opendev.org/#/c/603955/ | |
| 20:49:09 | mriedem | melwitt: see what you think about my proposed wording in this "specify az on unshelve" spec and if you agree i'll update it https://review.opendev.org/#/c/624689/10/specs/train/approved/support-specifying-az-when-restore-shelved-server.rst@62 | |
| 20:58:00 | efried | nova meeting in 3 minutes in #openstack-meeting | |
| 20:59:11 | melwitt | mriedem: ack | |
| 21:02:16 | efried | nova meeting now | |
| 21:07:23 | mriedem | no meeting?! | |
| 21:07:28 | mriedem | was reviewing something... | |
| 21:28:25 | melwitt | mriedem_afk: +1 to proposed wording | |
| 21:43:02 | openstackgerrit | Dustin Cowles proposed openstack/nova master: WIP: Use SDK instead of ironicclient for add/remove instance info from node https://review.opendev.org/659691 | |
| 21:53:14 | tonyb | kashyap: I don't but it's on my list of things to get workign this year | |
| 21:53:19 | tonyb | kashyap: 'sup? | |
| 22:14:36 | openstackgerrit | Merged openstack/nova master: [Docs] Update the confusing console output https://review.opendev.org/589004 | |
| 22:47:09 | mnaser | alright. has anyone ever ran into this? i'm convinced this feels like an oslo bug or something. | |
| 22:47:31 | mnaser | i've noticed that if i have a 3 node rabbitmq cluster, and the cluster loses a node and regains it, for some reason, its like nothing gets routed across rabbitmq | |
| 22:47:43 | mnaser | in this case, an instance gets booted and goes into SCHEDULING state and gets stuck forever | |
| 22:48:41 | mnaser | this is with rabbitmq 3.6.16 -- the only way i've solved it often is by deleting all the queues and restarting nova | |
| 22:48:54 | mnaser | the one 'odd' thing is with osa, we deploy a cluster with ssl for rabbitmq | |
| 22:49:32 | mnaser | i have a (non-prod) cluster stuck at this state right now and i really want to get to the bottom of what/why | |
| 22:51:01 | mnaser | all queues listing 0 messages.. except for notifications | |
| 22:59:40 | melwitt | mnaser: feel like you're going to need to get some other operator input about that one.. I haven't personally heard of it | |
| 23:00:03 | melwitt | imacdonn ^ | |
| 23:00:06 | mnaser | melwitt: i've ran into it a bunch of times and had others who ran to it and the "drop all queues" thing fixed it.. | |
| 23:00:26 | melwitt | I also asked penick to look at your messages here and let us know if he's seen it | |
| 23:01:18 | mnaser | melwitt: yay cool thanks, i'm going to continue digging, this is a non prod environment so i have the time to dig into it | |
| 23:01:25 | mnaser | usually things are under fire when i need to do this | |
| 23:01:38 | melwitt | yeah, that's awesome you got a repro in non-prod | |
| 23:02:14 | mnaser | feels like it's gonna be a really annoying one though | |
| 23:03:26 | melwitt | yeah. another person we could ask is kgiusti, he knows oslo.messaging | |
| 23:04:22 | mnaser | so, instance in scheduling state, 7 messages in a bunch of scheduler_fanout_* queues | |
| 23:04:44 | mnaser | but nothing happening, so i guess the conductor is doing what its suppoed to do | |
| 23:04:58 | penick | @mnaser we've had weird issues with rabbitmq and ssl. Mostly with performance and the queue faceplanting though. I don't think we've ever had to wipe and recreate all queues. | |
| 23:05:24 | mnaser | :( sads, yeah, it's pretty destructive but not sure.. | |
| 23:05:28 | penick | Though the way we worked around the ssl issue is we switched from native SSL to using stunnel. Which is awful | |
| 23:05:38 | penick | something about native rabbit ssl is borked | |
| 23:05:42 | mnaser | oh wow, is the native rabbit ssl that bad | |
| 23:05:48 | mnaser | can i ask what release were you running then (or now)? | |
| 23:05:54 | penick | Ocata | |
| 23:05:58 | penick | oh, of rabbit | |
| 23:05:59 | penick | uh, one sec | |
| 23:06:05 | mnaser | thanks :> | |
| 23:06:12 | mnaser | there is exactly 48 scheduler_fanout_* queues | |
| 23:06:54 | mnaser | each controller has 16 nova-scheduler processes, which adds up to 48 | |
| 23:07:08 | mnaser | so the theory is nova-conductor is doing its job but the scheduler just isn't picking up work off the queue.. | |
| 23:07:56 | penick | 3.6.15 | |
| 23:08:09 | melwitt | penick: I wondered whether it might be related to queue mirroring or non, clustered or non, how it behaves when a node leaves and rejoins | |
| 23:08:22 | mnaser | yeah we're at 3.6.16 so we're not far away | |
| 23:08:38 | mnaser | melwitt: yeah so actually i wondered if that was maybe the issue.. | |
| 23:08:46 | imacdonn | FWIW, I don't run clustered rabbitmq - just a single instance | |
| 23:09:01 | melwitt | I had thought there might have been some configuration where that doesn't work well, but I can't remember anything more about it | |
| 23:09:12 | mnaser | imacdonn: you're living life | |
| 23:09:40 | melwitt | :D | |