| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-02-15 | |||
| 19:15:31 | dansmith | thus nova never got to delete the RPs | |
| 19:15:55 | TheJulia | I think the restart is during the ironic upgrade, checking to see when it actually occured | |
| 19:16:09 | dansmith | right, but if placement started crashing during the upgrade of packages, | |
| 19:16:21 | dansmith | which is when nova would have noticed ironic went away and tried to delete RPs or something, | |
| 19:16:24 | jroll | OH | |
| 19:16:28 | jroll | keystone is upgrading there | |
| 19:16:35 | jroll | and so placement can't validate the token | |
| 19:16:36 | dansmith | and then you restart apache.. | |
| 19:16:38 | dansmith | ohh | |
| 19:16:43 | jroll | http://logs.openstack.org/50/544750/8/check/ironic-grenade-dsvm-multinode-multitenant/5713fb8/logs/screen-placement-api.txt.gz#_Feb_15_17_58_22_463228 | |
| 19:17:02 | mriedem | melwitt: i'm likely also going to start writing a functional test for the case that we delete a build request for a bfv instance during local delete, because we aren't cleaning up volumes there either | |
| 19:17:08 | dansmith | and you get a bug, and you get a bug, and you get a bug... | |
| 19:17:21 | dansmith | jroll: good catch | |
| 19:17:28 | melwitt | mriedem: ack | |
| 19:17:50 | TheJulia | http://logs.openstack.org/50/544750/8/check/ironic-grenade-dsvm-multinode-multitenant/5713fb8/logs/grenade.sh.txt.gz#_2018-02-15_18_00_28_322 is when we restart apache | |
| 19:17:53 | dansmith | jroll: surely you're not intending to have keystone be upgradingin this task right? | |
| 19:17:59 | dansmith | s/task/job/ | |
| 19:18:30 | jroll | dansmith: I think we just haven't cared either way in the past | |
| 19:18:44 | jroll | keystone's one of those hand-wavy things to me, it's always there, don't care what version | |
| 19:18:46 | dansmith | jroll: do you care now? :P | |
| 19:18:49 | jroll | heh | |
| 19:21:37 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add admin guide doc on volume multiattach support https://review.openstack.org/544090 | |
| 19:21:48 | TheJulia | jroll: we could just skip restarting apache and fire up the local ironic-api, however then we're not forcing traffic through an older API endpoint which changes the scenario, although we would still have the rpc version pin | |
| 19:21:56 | mriedem | bauzas: since you helped review the multiattach series, can you check out ^ so we can backport that for queens RC2? | |
| 19:22:23 | jroll | TheJulia: eh, I'd rather not | |
| 19:22:28 | jroll | also these times aren't lining up :/ | |
| 19:22:32 | TheJulia | actually, other services will still likely need to restart things | |
| 19:22:44 | TheJulia | so we shouldn't try to avoid restarting apache | |
| 19:22:59 | jroll | oh it does line up, okay | |
| 19:23:16 | jroll | TheJulia: maybe we um, try to make apache not need 66 seconds to restart | |
| 19:23:18 | TheJulia | heh, 28 apache restarts in the grenade log | |
| 19:23:27 | jroll | jesus | |
| 19:23:31 | dansmith | wow | |
| 19:23:33 | dansmith | that's impressive | |
| 19:23:57 | mriedem | the edge people said openstack needed to be slimmed down | |
| 19:24:13 | TheJulia | yeah.... | |
| 19:24:15 | cfriesen | mriedem: I think you're right, I got messed up with which patch was fixing what. :) the fixes at https://review.openstack.org/#/c/528385 and https://review.openstack.org/#/c/340614/ look like they might do the trick. | |
| 19:24:31 | mriedem | cfriesen: cool | |
| 19:24:36 | mriedem | it is confusing | |
| 19:24:55 | mriedem | cfriesen: the point of creating the bdms in cell0 also was so that the local delete in the api can remove them | |
| 19:25:06 | mriedem | and thus avoid us having nova-compute, nova-api AND nova-conductor doing volume cleanup | |
| 19:25:49 | TheJulia | jroll: what makes you think 66 seconds? | |
| 19:26:27 | jroll | TheJulia: this line to the next one: http://logs.openstack.org/50/544750/8/check/ironic-grenade-dsvm-multinode-multitenant/5713fb8/logs/screen-keystone.txt.gz#_Feb_15_17_57_44_538878 | |
| 19:26:31 | jroll | it isn't 66 seconds to restart | |
| 19:26:48 | jroll | nor is it an apache thing | |
| 19:26:50 | jroll | but we shut down keystone for the entire keystone upgrade | |
| 19:26:58 | TheJulia | ahh, yeah | |
| 19:27:02 | jroll | which is silly | |
| 19:27:22 | TheJulia | Well.... are we sure it is not silly? | |
| 19:28:39 | jroll | I would think keystone should be able to do rolling upgrades very easily | |
| 19:28:47 | jroll | but they don't have the magic tag, so who knows | |
| 19:30:24 | dansmith | well, it's not really placement, it's nova-compute right? | |
| 19:30:39 | dansmith | we're probably not handling the 503 very gracefully in compute whilst trying to do things | |
| 19:33:30 | mriedem | is the 503 this? https://github.com/openstack/nova/blob/master/nova/scheduler/client/report.py#L93 | |
| 19:34:13 | dansmith | I woudn't think so, we're getting a response | |
| 19:34:46 | oomichi | hi, can someone take a look at https://review.openstack.org/#/c/532822 ? that is easy and completely the same patch is posted. so want to avoid the same thing | |
| 19:34:48 | dansmith | we just raise and kill the RT | |
| 19:37:45 | oomichi | thanks, mriedem | |
| 19:40:41 | mriedem | melwitt: i went ahead and created this for tracking the community wide goal and linked it to the other 4 bp's we've had in the past for the same thing https://blueprints.launchpad.net/nova/+spec/mox-removal | |
| 19:40:57 | openstackgerrit | Jay Pipes proposed openstack/nova-specs master: Account for host agg allocation ratio in placement https://review.openstack.org/544683 | |
| 19:41:00 | mriedem | and marked the other as done https://review.openstack.org/545100 | |
| 19:41:21 | melwitt | mriedem: great, thank you | |
| 19:41:23 | jaypipes | bauzas, cdent, edleafe, dansmith, mriedem, efried: ^^ round two on the aggregate allocation ratio one. | |
| 19:43:31 | openstackgerrit | Jay Pipes proposed openstack/nova-specs master: Account for host agg allocation ratio in placement https://review.openstack.org/544683 | |
| 19:44:09 | mriedem | melwitt: i also assume we'll want 2 blueprints for removing nova-network and cellsv1 | |
| 19:44:17 | mriedem | as those likely won't be a single change | |
| 19:44:27 | openstackgerrit | Jay Pipes proposed openstack/nova-specs master: mirror nova host aggregates to placement API https://review.openstack.org/545057 | |
| 19:44:34 | melwitt | mriedem: yeah, I don't think they would be a single change | |
| 19:44:44 | mriedem | i'm not sure on the order there though, probably nova-network first since you can't start nova-network w/o cells v1 enabled | |
| 19:45:01 | mriedem | and some people run cellsv1 with neutron | |
| 19:46:32 | openstackgerrit | Jay Pipes proposed openstack/nova-specs master: Account for host agg allocation ratio in placement https://review.openstack.org/544683 | |
| 19:50:00 | efried | jaypipes: Ya done yet? | |
| 19:58:07 | jaypipes | efried: yeah, sorry :) | |
| 20:10:43 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Convert driver supported capabilities to compute node provider traits https://review.openstack.org/538498 | |
| 20:14:03 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Convert driver supported capabilities to compute node provider traits https://review.openstack.org/538498 | |
| 20:19:19 | openstackgerrit | Jay Pipes proposed openstack/nova-specs master: mirror nova host aggregates to placement API https://review.openstack.org/545057 | |
| 20:24:08 | mnaser | mriedem: do i have to make any changes at this point for the clean up patch .. i know there's a lot of discussion but anything actionable for me right now? | |
| 20:25:11 | mriedem | mnaser: nope | |
| 20:26:18 | mnaser | jaypipes: ahah. words that once i settle down with someone + kids, time disappers | |
| 20:26:29 | mnaser | enjoying it while it lasts, ha | |
| 20:29:05 | mriedem | you can still have a family, and ignore them | |
| 20:29:17 | jaypipes | mnaser: :) | |
| 20:29:20 | mnaser | haha | |
| 20:29:56 | jaypipes | mnaser: that's awesome to hear! | |
| 20:30:28 | jroll | that is awesome, but I'm skeptical, *every* bit? | |
| 20:30:38 | jroll | :P | |
| 20:30:39 | mnaser | jaypipes: :D | |
| 20:30:55 | mnaser | jroll: it helps having super awesome understanding customers that know all about open source projects | |
| 20:31:16 | mnaser | any issues we usually work with them to iron out and work with openstack teams to roll fixes out and everyone's happy | |
| 20:31:23 | jroll | sweet :) | |
| 20:31:36 | jaypipes | mnaser: perhaps you can come visit with my internal customers ... ;) | |
| 20:32:05 | mnaser | maybe they should just use us and make your life easier </salesman> | |
| 20:32:10 | mnaser | haha | |
| 20:32:25 | jaypipes | mnaser: ++ :) | |
| 20:32:40 | jroll | jaypipes: wow, trying to drag mnaser down so quickly after mentioning the fun | |
| 20:33:18 | mnaser | i'm just a bit bummed that i don't think we'll be able to get to queens right away | |
| 20:33:32 | jaypipes | lol | |
| 20:33:33 | mnaser | mainly because we have consumers using the images v1 api | |
| 20:34:50 | openstackgerrit | Matthew Edmonds proposed openstack/nova-specs master: PowerVM Virt Integration (Rocky) https://review.openstack.org/545111 | |
| 20:36:17 | mnaser | jaypipes: will you be at the ptg? | |