| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-01-13 | |||
| 15:39:22 | mwhahaha | it doesn't | |
| 15:39:32 | mwhahaha | the upgrade problem is something else | |
| 15:39:42 | mwhahaha | the problem is that you're adding requirements that no one know sabout | |
| 15:39:50 | mriedem | melwitt: which cells v2 upgrade check? the db schema migration? | |
| 15:39:51 | openstack | Launchpad bug 1656276 in tripleo "Error running nova-manage cell_v2 simple_cell_setup when configuring nova with puppet-nova" [Critical,Triaged] | |
| 15:39:51 | bauzas | melwitt: https://bugs.launchpad.net/nova/+bug/1656276 refers to a fresh install | |
| 15:40:06 | mwhahaha | oh yea,that's new as of today | |
| 15:40:08 | melwitt | mriedem: the thing that raises an error if there aren't two cell mappings | |
| 15:40:16 | bauzas | melwitt: and the proposed fix to amend https://github.com/openstack/puppet-nova/blob/master/manifests/db/sync_cell_v2.pp#L32 with return code 1 as valid would fix $ | |
| 15:40:22 | bauzas | would fix ^ (I mean) | |
| 15:40:40 | bauzas | what mwhahaha is discussing is another issue, related to the upgrade path | |
| 15:40:40 | mriedem | melwitt: mwhahaha: so the db schema migration runs on a fresh install by first checking to see if there are any flavors, if there aren't then it noops | |
| 15:40:43 | mwhahaha | https://bugs.launchpad.net/tripleo/+bug/1649341 was teh upgrade bug | |
| 15:40:44 | openstack | Launchpad bug 1649341 in tripleo "Undercloud upgrade fails with "Cell mappings are not created, but required for Ocata"" [Critical,In progress] - Assigned to Alex Schultz (alex-schultz) | |
| 15:40:57 | bauzas | so another bug | |
| 15:40:58 | melwitt | mriedem: the (_('Cells v2'), _check_cellsv2), in nova/cmd/status.py | |
| 15:41:05 | mriedem | melwitt: so that's different | |
| 15:41:07 | melwitt | I see | |
| 15:41:18 | mriedem | and we haven't integrated the nova-status upgrade check stuff into our CI pipeline yet | |
| 15:41:21 | mriedem | another TODO of mine | |
| 15:41:41 | openstack | Launchpad bug 1656276 in tripleo "Error running nova-manage cell_v2 simple_cell_setup when configuring nova with puppet-nova" [Critical,Triaged] | |
| 15:41:41 | mwhahaha | which the problem is to solve https://bugs.launchpad.net/tripleo/+bug/1649341 i need to know how to properly tackle https://bugs.launchpad.net/nova/+bug/1656276 | |
| 15:41:49 | mwhahaha | where i didn't until today | |
| 15:41:59 | mwhahaha | so now we're just adding complexity on complexity | |
| 15:42:31 | mwhahaha | and adding accept 1 doesn't solve the upgrade case | |
| 15:42:45 | melwitt | yeah, I don't think accepting 1 helps anything really | |
| 15:42:52 | mwhahaha | i had a solution for the upgrade until the requirement of compute nodes was added | |
| 15:43:01 | mriedem | mwhahaha: EmilienM: so can we hold on a minute, | |
| 15:43:08 | mriedem | i understand you're frustrated, | |
| 15:43:31 | bauzas | melwitt: it only helps their CI to be green | |
| 15:43:51 | mriedem | i need help with posting the list of issues you have in a form that doesn't require me to be chasing 3 conversations at once | |
| 15:43:54 | melwitt | I'm not sure it'll run correctly after that if there are no cell mappings created though (nova) | |
| 15:43:56 | EmilienM | it's not about a green CI | |
| 15:44:05 | openstackgerrit | Davanum Srinivas (dims) proposed openstack/nova: [py35] Fixes to get rally scenarios working https://review.openstack.org/419476 | |
| 15:44:05 | EmilienM | it's about how people outside devstack deploy your software | |
| 15:44:17 | mriedem | mwhahaha: EmilienM: so can you guys help me out by starting a ML thread with the current known issues you have so we can work through it there? | |
| 15:44:23 | EmilienM | mriedem: of course | |
| 15:44:58 | mriedem | b/c right now i'm not keeping up with this fire hose of stuff | |
| 15:45:03 | mriedem | and not everyone is around right now | |
| 15:45:18 | mriedem | so we need to start a list of items in the ML and tackle those | |
| 15:45:31 | mwhahaha | yea we can move to the ML | |
| 15:45:46 | bauzas | EmilienM: I don't disagree, I'm just trying to reduce the criticity so we can sit down and discuss without having the tense of urgency | |
| 15:46:26 | mriedem | we could start an etherpad too if that is helpful | |
| 15:46:41 | mriedem | hell i'm pretty sure we already have an etherpad for cells v2 | |
| 15:46:54 | bauzas | EmilienM: I'm really sorry that you folks are experiencing bugs against our recent improvements and I think your insights are definitely valid | |
| 15:47:54 | mriedem | i can't find the etherpad, only the wiki page | |
| 15:48:04 | mriedem | melwitt: do you know if we had a generic cells v2 etherpad for work items? | |
| 15:48:43 | melwitt | mriedem: we've had a number of etherpads, let me see | |
| 15:49:04 | mriedem | melwitt: yeah this is the older wiki https://wiki.openstack.org/wiki/Nova-Cells-v2 | |
| 15:49:08 | mriedem | but not seeing anything generic there | |
| 15:49:28 | EmilienM | bauzas: let's get this thing fixed | |
| 15:50:05 | melwitt | https://etherpad.openstack.org/p/nova-multiple-cells https://etherpad.openstack.org/p/cellsV2-remaining-work-items are old ones | |
| 15:51:44 | mriedem | melwitt: thanks i added those to the wiki | |
| 15:54:47 | mriedem | trying to update the TODOs in there also | |
| 15:56:20 | melwitt | thanks mriedem | |
| 15:57:19 | mriedem | also https://review.openstack.org/#/c/420051/ | |
| 15:58:13 | mriedem | EmilienM: melwitt: bauzas: mwhahaha: ok i've updated https://wiki.openstack.org/wiki/Nova-Cells-v2#Nova_Cells_v2 a bit - linked in those older etherpads, freshed up the TODOs, and added those bugs | |
| 15:58:18 | mriedem | let me know if i missed something | |
| 15:59:32 | EmilienM | mriedem: thanks for helping | |
| 16:00:00 | EmilienM | mriedem: are we sure to backport https://review.openstack.org/#/c/420051/ ? | |
| 16:00:05 | EmilienM | it broke us on master :/ | |
| 16:01:59 | mriedem | EmilienM: i thought that change was made on master specifically for you guys | |
| 16:02:04 | mwhahaha | no | |
| 16:02:09 | mwhahaha | it's not for us, we were working prior to that | |
| 16:02:10 | melwitt | I think it was sort of a coincidence that it broke you, because simple_cell_setup has always returned 1 in the case of not finding compute nodes. the reason you got past it is because you ran map_cell0 separately before running simple_cell_setup, and the old version of simple_cell_setup returned 0 if it found cell0 i.e. it assumed it had already run in hte past | |
| 16:02:24 | mwhahaha | we don't run map_cell0 | |
| 16:02:36 | mwhahaha | cause we didn't know that was a thing | |
| 16:04:28 | mriedem | ok i don't have the context on how that change broke you, is that in a bug somewhere? | |
| 16:04:30 | openstack | Launchpad bug 1656276 in tripleo "Error running nova-manage cell_v2 simple_cell_setup when configuring nova with puppet-nova" [Critical,Triaged] | |
| 16:04:30 | melwitt | in https://bugs.launchpad.net/nova/+bug/1656276 amoralej said you were mapping cell0 separately? | |
| 16:04:37 | openstack | Launchpad bug 1649341 in tripleo "Undercloud upgrade fails with "Cell mappings are not created, but required for Ocata"" [Critical,In progress] - Assigned to Alex Schultz (alex-schultz) | |
| 16:04:37 | mriedem | is that https://bugs.launchpad.net/tripleo/+bug/1649341 ? | |
| 16:04:55 | melwitt | also from the code it's the only way you could have gotten simple_cell_setup to return 0 without any compute nodes | |
| 16:05:01 | melwitt | unless I've missed something | |
| 16:05:12 | mriedem | sorry i meant https://bugs.launchpad.net/nova/+bug/1656276 | |
| 16:05:48 | mwhahaha | so we're relying on puppet, and all we knew is that we had to run the simple setup (thats what we were told last time we came here with problems) | |
| 16:05:54 | mriedem | ok i've -W'd the backport until we can sort it out | |
| 16:05:55 | mwhahaha | so i went and implemented just the simple setup | |
| 16:06:03 | melwitt | mriedem: I *think* it's because they are running 'map_cell0' and then 'simple_cell_setup', before the idempotence patch it would all return 0, after the patch it returns 1 because it gets to the 'checking for hosts' part | |
| 16:06:06 | mwhahaha | cause i didn't know those other commands were required | |
| 16:06:36 | melwitt | the other commands aren't required | |
| 16:06:37 | amoralej | melwitt, we get cell0 by simple_cell_setup command | |
| 16:06:50 | mwhahaha | melwitt: evidently they are | |
| 16:06:55 | mwhahaha | melwitt: they weren't until that patch | |
| 16:07:03 | alaski | EmilienM: bauzas I've been out of discussions for a bit, but I had a PoC up at one point for computes to self register with a cell. It put an entry in the db that they could pull which told them which cell they were in, and then I was adding an HTTP API call for them to hit for registration. | |
| 16:07:21 | mriedem | melwitt: so you mean they got here https://review.openstack.org/#/c/420051/1/nova/cmd/manage.py@1240 ? | |
| 16:07:24 | mriedem | after that change i mean | |
| 16:07:52 | melwitt | yes | |
| 16:08:23 | amoralej | melwitt, we were getting return 0 in second execution | |
| 16:08:28 | mwhahaha | are you sure we're just not failing from line 1226 | |
| 16:08:29 | amoralej | because we retry it | |
| 16:08:39 | mriedem | melwitt: and that's because they got to https://review.openstack.org/#/c/420051/1/nova/cmd/manage.py@1379 ? | |
| 16:08:45 | mriedem | i.e. no computes yet so it would return 1 | |
| 16:08:47 | mriedem | well, | |
| 16:08:51 | alaski | bauzas: EmilienM https://review.openstack.org/#/c/369634/ | |
| 16:08:54 | mriedem | return None from that method, and simple_cell_setup would return 1 | |
| 16:08:59 | mriedem | which then broke tripleo, right? | |
| 16:09:25 | bauzas | alaski: yeah I remember now our discussions | |
| 16:09:54 | alaski | that was just an idea I had though, there had been no real discussion or consensus on the direction | |
| 16:10:08 | bauzas | yup, what I told to EmilienM and amoralej | |