Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-01-13
15:59:32 EmilienM mriedem: thanks for helping
16:00:00 EmilienM mriedem: are we sure to backport https://review.openstack.org/#/c/420051/ ?
16:00:05 EmilienM it broke us on master :/
16:01:59 mriedem EmilienM: i thought that change was made on master specifically for you guys
16:02:04 mwhahaha no
16:02:09 mwhahaha it's not for us, we were working prior to that
16:02:10 melwitt I think it was sort of a coincidence that it broke you, because simple_cell_setup has always returned 1 in the case of not finding compute nodes. the reason you got past it is because you ran map_cell0 separately before running simple_cell_setup, and the old version of simple_cell_setup returned 0 if it found cell0 i.e. it assumed it had already run in hte past
16:02:24 mwhahaha we don't run map_cell0
16:02:36 mwhahaha cause we didn't know that was a thing
16:04:28 mriedem ok i don't have the context on how that change broke you, is that in a bug somewhere?
16:04:30 melwitt in https://bugs.launchpad.net/nova/+bug/1656276 amoralej said you were mapping cell0 separately?
16:04:30 openstack Launchpad bug 1656276 in tripleo "Error running nova-manage cell_v2 simple_cell_setup when configuring nova with puppet-nova" [Critical,Triaged]
16:04:37 mriedem is that https://bugs.launchpad.net/tripleo/+bug/1649341 ?
16:04:37 openstack Launchpad bug 1649341 in tripleo "Undercloud upgrade fails with "Cell mappings are not created, but required for Ocata"" [Critical,In progress] - Assigned to Alex Schultz (alex-schultz)
16:04:55 melwitt also from the code it's the only way you could have gotten simple_cell_setup to return 0 without any compute nodes
16:05:01 melwitt unless I've missed something
16:05:12 mriedem sorry i meant https://bugs.launchpad.net/nova/+bug/1656276
16:05:48 mwhahaha so we're relying on puppet, and all we knew is that we had to run the simple setup (thats what we were told last time we came here with problems)
16:05:54 mriedem ok i've -W'd the backport until we can sort it out
16:05:55 mwhahaha so i went and implemented just the simple setup
16:06:03 melwitt mriedem: I *think* it's because they are running 'map_cell0' and then 'simple_cell_setup', before the idempotence patch it would all return 0, after the patch it returns 1 because it gets to the 'checking for hosts' part
16:06:06 mwhahaha cause i didn't know those other commands were required
16:06:36 melwitt the other commands aren't required
16:06:37 amoralej melwitt, we get cell0 by simple_cell_setup command
16:06:50 mwhahaha melwitt: evidently they are
16:06:55 mwhahaha melwitt: they weren't until that patch
16:07:03 alaski EmilienM: bauzas I've been out of discussions for a bit, but I had a PoC up at one point for computes to self register with a cell. It put an entry in the db that they could pull which told them which cell they were in, and then I was adding an HTTP API call for them to hit for registration.
16:07:21 mriedem melwitt: so you mean they got here https://review.openstack.org/#/c/420051/1/nova/cmd/manage.py@1240 ?
16:07:24 mriedem after that change i mean
16:07:52 melwitt yes
16:08:23 amoralej melwitt, we were getting return 0 in second execution
16:08:28 mwhahaha are you sure we're just not failing from line 1226
16:08:29 amoralej because we retry it
16:08:39 mriedem melwitt: and that's because they got to https://review.openstack.org/#/c/420051/1/nova/cmd/manage.py@1379 ?
16:08:45 mriedem i.e. no computes yet so it would return 1
16:08:47 mriedem well,
16:08:51 alaski bauzas: EmilienM https://review.openstack.org/#/c/369634/
16:08:54 mriedem return None from that method, and simple_cell_setup would return 1
16:08:59 mriedem which then broke tripleo, right?
16:09:25 bauzas alaski: yeah I remember now our discussions
16:09:54 alaski that was just an idea I had though, there had been no real discussion or consensus on the direction
16:10:08 bauzas yup, what I told to EmilienM and amoralej
16:10:56 bauzas EmilienM: amoralej: mwhahaha: melwitt: mriedem: sorry, was in a meeting
16:11:06 melwitt alaski: that requires that a first cell was ever created though, right? they're in a situation where they have no compute hosts and no cell mapping other than cell0
16:11:32 amoralej but, why can't create a cell without compute nodes?
16:11:47 melwitt I'm guessing to avoid creating orphaned cells
16:11:52 bauzas melwitt: we could simple setup cells with no computes, and populate the mappings when adding the nodes
16:11:58 alaski melwitt: right. that wouldn't solve their issue. I wasn't exactly clear what's being discussed now.
16:12:08 mwhahaha shouldn't the cell be created first then nodes added to it?
16:12:14 mwhahaha like it seems weird to do the other way around
16:12:14 melwitt alaski: is that why we don't create cell mappings with no hosts? to avoid orphaned cells?
16:12:16 amoralej from a functional, operational point of view, creating the cell first and registering compute nodes later make sense imo
16:12:36 amoralej in fact compute nodes should be automatically added
16:12:43 amoralej i'd say
16:13:01 melwitt they could, but they'd have to know what cell to automatically map to
16:13:06 mriedem the compute nodes self registering with a cell mapping requires an upcall from the compute to the api which we were trying to avoid,
16:13:15 mriedem but is what alaski's WIP patch was for i think
16:13:31 mwhahaha but if you have the transport url, isn't it just a message?
16:13:35 melwitt if you have several, they'd have to automatically go to some 'staging cell' before you move them to where you actually want them. I think that's why it's not straightforward to be automatic. unless you're talking single cell
16:13:40 mwhahaha i mean whats the point of adding the transport url to the cell config
16:13:47 alaski mriedem: yep, but I was going for an HTTP call which sidesteps it a bit. it's assuming there's an admin API for registering a node in a cell
16:14:17 alaski melwitt: I'm not sure there was a conscious decision to not allow mapping an empty cell, at least not that I'm aware of.
16:14:54 melwitt mwhahaha: that's for the api cell to call into a compute cell. when the cell mapping is created in the api, the compute cell's transport_url is part of the mapping so the api cell can talk to it
16:15:04 alaski melwitt: it is a bit strange to stand up a cell if there are no ready computes, but I can see why people would want that
16:15:22 melwitt alaski: okay. all the code is written that way so I think I assumed there was some consensus I missed :P
16:15:38 melwitt yeah. it only makes sense in the fresh install case
16:15:50 alaski melwitt: it's also possible I'm forgetting some discussion that happened
16:15:51 mwhahaha alaski: i would respectfully disagree because as you deploy, people don't configure comptues first
16:16:11 mwhahaha alaski: controllers (and their config) generally are first which is why this is a problem
16:16:19 mriedem honestly i think we probably just always thought about cells v2 in an upgrade scenario, not fresh install
16:16:22 mriedem for the most part
16:16:28 alaski yeah
16:16:49 alaski for the initial deploy an empty cell makes more sense, but standing up a second cell wouldn't typically happen if there are no computes for it
16:17:07 melwitt okay. well, I saw andrey has a review up for creating an empty cell
16:17:08 mwhahaha no it still makes sense
16:17:23 mwhahaha because you're growing your cloud, so you create the cell config and populate it with the new nodes
16:17:50 alaski mwhahaha: I'm not saying it doesn't make sense, but it also makes sense to say don't stand up a new cell if there are no computes for it yet
16:18:41 cfriesen mdbooth: are you there?
16:18:53 melwitt typically I think we've thought of adding cells as something that you do while the cloud is running, not during code deploy. right?
16:19:10 alaski yes
16:19:13 melwitt it's just this initial state problem
16:20:56 mdbooth cfriesen: Yep
16:21:24 mwhahaha alaski: so i guess i'm not clear on what you mean 'to say don't stand up a new cell if there are no computes for it'
16:21:41 cfriesen re: my question at https://review.openstack.org/#/c/337160/17/nova/virt/libvirt/driver.py. I'm trying to figure out why we unconditionally call root_disk.create_snap(). Why is it needed for resize/migration to a different host?
16:21:43 mwhahaha alaski: do you mean as a recommendation to operators that you shouldn't have an empty cell running?
16:21:55 EmilienM I agree, compute nodes are usually deployed at the very end
16:22:17 mwhahaha alaski: what's the impact of the empty cell? would break something?
16:22:20 EmilienM I think part of the reason is because in the past it used to requires some API to be running
16:22:28 mdbooth cfriesen: It's used for rollback of Rbd disks.
16:22:33 alaski mwhahaha: I'm saying that as a deployer I would not add a new cell to the system unless I had computes ready to add to that cell. I'm not going to stand up a new cell that won't do anything yet
16:22:58 mdbooth cfriesen: IIRC it's a no-op for all other backends.
16:23:24 mdbooth However, cold migration/resize have a resize->confirm/rollback workflow
16:23:24 mwhahaha alaski: well if i know i'm goign to have computes, i'd want to create the cell but wait to add the computes until i get to that phase of my install
16:23:49 mwhahaha alaski: that's the thing, fuel, kolla, tripleo we all do controller configs first then add computes later because we need to have keystone and stuff running first
16:23:51 mdbooth For disk-based local storage, we copy the instance dir, whether single host or migrating host
16:24:10 mdbooth For rbd, they're the same disks, so if we didn't snapshot we wouldn't be able to rollback.
16:24:11 alaski mwhahaha: an empty cell should not break anything afaik, but I've been out of cells for a bit now so I may be mistaken.
16:24:19 cfriesen mdbooth: okay, that makes sense. missed the fact it was a no-op for most cases.
16:24:26 mwhahaha alaski: sure, just trying to understand this better
16:24:26 mriedem alaski: is your perspective on that model biased to how it was done with cells v1?

Earlier   Later