Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-01-13
15:42:52 mwhahaha i had a solution for the upgrade until the requirement of compute nodes was added
15:43:01 mriedem mwhahaha: EmilienM: so can we hold on a minute,
15:43:08 mriedem i understand you're frustrated,
15:43:31 bauzas melwitt: it only helps their CI to be green
15:43:51 mriedem i need help with posting the list of issues you have in a form that doesn't require me to be chasing 3 conversations at once
15:43:54 melwitt I'm not sure it'll run correctly after that if there are no cell mappings created though (nova)
15:43:56 EmilienM it's not about a green CI
15:44:05 openstackgerrit Davanum Srinivas (dims) proposed openstack/nova: [py35] Fixes to get rally scenarios working https://review.openstack.org/419476
15:44:05 EmilienM it's about how people outside devstack deploy your software
15:44:17 mriedem mwhahaha: EmilienM: so can you guys help me out by starting a ML thread with the current known issues you have so we can work through it there?
15:44:23 EmilienM mriedem: of course
15:44:58 mriedem b/c right now i'm not keeping up with this fire hose of stuff
15:45:03 mriedem and not everyone is around right now
15:45:18 mriedem so we need to start a list of items in the ML and tackle those
15:45:31 mwhahaha yea we can move to the ML
15:45:46 bauzas EmilienM: I don't disagree, I'm just trying to reduce the criticity so we can sit down and discuss without having the tense of urgency
15:46:26 mriedem we could start an etherpad too if that is helpful
15:46:41 mriedem hell i'm pretty sure we already have an etherpad for cells v2
15:46:54 bauzas EmilienM: I'm really sorry that you folks are experiencing bugs against our recent improvements and I think your insights are definitely valid
15:47:54 mriedem i can't find the etherpad, only the wiki page
15:48:04 mriedem melwitt: do you know if we had a generic cells v2 etherpad for work items?
15:48:43 melwitt mriedem: we've had a number of etherpads, let me see
15:49:04 mriedem melwitt: yeah this is the older wiki https://wiki.openstack.org/wiki/Nova-Cells-v2
15:49:08 mriedem but not seeing anything generic there
15:49:28 EmilienM bauzas: let's get this thing fixed
15:50:05 melwitt https://etherpad.openstack.org/p/nova-multiple-cells https://etherpad.openstack.org/p/cellsV2-remaining-work-items are old ones
15:51:44 mriedem melwitt: thanks i added those to the wiki
15:54:47 mriedem trying to update the TODOs in there also
15:56:20 melwitt thanks mriedem
15:57:19 mriedem also https://review.openstack.org/#/c/420051/
15:58:13 mriedem EmilienM: melwitt: bauzas: mwhahaha: ok i've updated https://wiki.openstack.org/wiki/Nova-Cells-v2#Nova_Cells_v2 a bit - linked in those older etherpads, freshed up the TODOs, and added those bugs
15:58:18 mriedem let me know if i missed something
15:59:32 EmilienM mriedem: thanks for helping
16:00:00 EmilienM mriedem: are we sure to backport https://review.openstack.org/#/c/420051/ ?
16:00:05 EmilienM it broke us on master :/
16:01:59 mriedem EmilienM: i thought that change was made on master specifically for you guys
16:02:04 mwhahaha no
16:02:09 mwhahaha it's not for us, we were working prior to that
16:02:10 melwitt I think it was sort of a coincidence that it broke you, because simple_cell_setup has always returned 1 in the case of not finding compute nodes. the reason you got past it is because you ran map_cell0 separately before running simple_cell_setup, and the old version of simple_cell_setup returned 0 if it found cell0 i.e. it assumed it had already run in hte past
16:02:24 mwhahaha we don't run map_cell0
16:02:36 mwhahaha cause we didn't know that was a thing
16:04:28 mriedem ok i don't have the context on how that change broke you, is that in a bug somewhere?
16:04:30 openstack Launchpad bug 1656276 in tripleo "Error running nova-manage cell_v2 simple_cell_setup when configuring nova with puppet-nova" [Critical,Triaged]
16:04:30 melwitt in https://bugs.launchpad.net/nova/+bug/1656276 amoralej said you were mapping cell0 separately?
16:04:37 openstack Launchpad bug 1649341 in tripleo "Undercloud upgrade fails with "Cell mappings are not created, but required for Ocata"" [Critical,In progress] - Assigned to Alex Schultz (alex-schultz)
16:04:37 mriedem is that https://bugs.launchpad.net/tripleo/+bug/1649341 ?
16:04:55 melwitt also from the code it's the only way you could have gotten simple_cell_setup to return 0 without any compute nodes
16:05:01 melwitt unless I've missed something
16:05:12 mriedem sorry i meant https://bugs.launchpad.net/nova/+bug/1656276
16:05:48 mwhahaha so we're relying on puppet, and all we knew is that we had to run the simple setup (thats what we were told last time we came here with problems)
16:05:54 mriedem ok i've -W'd the backport until we can sort it out
16:05:55 mwhahaha so i went and implemented just the simple setup
16:06:03 melwitt mriedem: I *think* it's because they are running 'map_cell0' and then 'simple_cell_setup', before the idempotence patch it would all return 0, after the patch it returns 1 because it gets to the 'checking for hosts' part
16:06:06 mwhahaha cause i didn't know those other commands were required
16:06:36 melwitt the other commands aren't required
16:06:37 amoralej melwitt, we get cell0 by simple_cell_setup command
16:06:50 mwhahaha melwitt: evidently they are
16:06:55 mwhahaha melwitt: they weren't until that patch
16:07:03 alaski EmilienM: bauzas I've been out of discussions for a bit, but I had a PoC up at one point for computes to self register with a cell. It put an entry in the db that they could pull which told them which cell they were in, and then I was adding an HTTP API call for them to hit for registration.
16:07:21 mriedem melwitt: so you mean they got here https://review.openstack.org/#/c/420051/1/nova/cmd/manage.py@1240 ?
16:07:24 mriedem after that change i mean
16:07:52 melwitt yes
16:08:23 amoralej melwitt, we were getting return 0 in second execution
16:08:28 mwhahaha are you sure we're just not failing from line 1226
16:08:29 amoralej because we retry it
16:08:39 mriedem melwitt: and that's because they got to https://review.openstack.org/#/c/420051/1/nova/cmd/manage.py@1379 ?
16:08:45 mriedem i.e. no computes yet so it would return 1
16:08:47 mriedem well,
16:08:51 alaski bauzas: EmilienM https://review.openstack.org/#/c/369634/
16:08:54 mriedem return None from that method, and simple_cell_setup would return 1
16:08:59 mriedem which then broke tripleo, right?
16:09:25 bauzas alaski: yeah I remember now our discussions
16:09:54 alaski that was just an idea I had though, there had been no real discussion or consensus on the direction
16:10:08 bauzas yup, what I told to EmilienM and amoralej
16:10:56 bauzas EmilienM: amoralej: mwhahaha: melwitt: mriedem: sorry, was in a meeting
16:11:06 melwitt alaski: that requires that a first cell was ever created though, right? they're in a situation where they have no compute hosts and no cell mapping other than cell0
16:11:32 amoralej but, why can't create a cell without compute nodes?
16:11:47 melwitt I'm guessing to avoid creating orphaned cells
16:11:52 bauzas melwitt: we could simple setup cells with no computes, and populate the mappings when adding the nodes
16:11:58 alaski melwitt: right. that wouldn't solve their issue. I wasn't exactly clear what's being discussed now.
16:12:08 mwhahaha shouldn't the cell be created first then nodes added to it?
16:12:14 melwitt alaski: is that why we don't create cell mappings with no hosts? to avoid orphaned cells?
16:12:14 mwhahaha like it seems weird to do the other way around
16:12:16 amoralej from a functional, operational point of view, creating the cell first and registering compute nodes later make sense imo
16:12:36 amoralej in fact compute nodes should be automatically added
16:12:43 amoralej i'd say
16:13:01 melwitt they could, but they'd have to know what cell to automatically map to
16:13:06 mriedem the compute nodes self registering with a cell mapping requires an upcall from the compute to the api which we were trying to avoid,
16:13:15 mriedem but is what alaski's WIP patch was for i think
16:13:31 mwhahaha but if you have the transport url, isn't it just a message?
16:13:35 melwitt if you have several, they'd have to automatically go to some 'staging cell' before you move them to where you actually want them. I think that's why it's not straightforward to be automatic. unless you're talking single cell
16:13:40 mwhahaha i mean whats the point of adding the transport url to the cell config
16:13:47 alaski mriedem: yep, but I was going for an HTTP call which sidesteps it a bit. it's assuming there's an admin API for registering a node in a cell
16:14:17 alaski melwitt: I'm not sure there was a conscious decision to not allow mapping an empty cell, at least not that I'm aware of.
16:14:54 melwitt mwhahaha: that's for the api cell to call into a compute cell. when the cell mapping is created in the api, the compute cell's transport_url is part of the mapping so the api cell can talk to it
16:15:04 alaski melwitt: it is a bit strange to stand up a cell if there are no ready computes, but I can see why people would want that
16:15:22 melwitt alaski: okay. all the code is written that way so I think I assumed there was some consensus I missed :P
16:15:38 melwitt yeah. it only makes sense in the fresh install case
16:15:50 alaski melwitt: it's also possible I'm forgetting some discussion that happened
16:15:51 mwhahaha alaski: i would respectfully disagree because as you deploy, people don't configure comptues first

Earlier   Later