Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-01-13
16:18:41 cfriesen mdbooth: are you there?
16:18:53 melwitt typically I think we've thought of adding cells as something that you do while the cloud is running, not during code deploy. right?
16:19:10 alaski yes
16:19:13 melwitt it's just this initial state problem
16:20:56 mdbooth cfriesen: Yep
16:21:24 mwhahaha alaski: so i guess i'm not clear on what you mean 'to say don't stand up a new cell if there are no computes for it'
16:21:41 cfriesen re: my question at https://review.openstack.org/#/c/337160/17/nova/virt/libvirt/driver.py. I'm trying to figure out why we unconditionally call root_disk.create_snap(). Why is it needed for resize/migration to a different host?
16:21:43 mwhahaha alaski: do you mean as a recommendation to operators that you shouldn't have an empty cell running?
16:21:55 EmilienM I agree, compute nodes are usually deployed at the very end
16:22:17 mwhahaha alaski: what's the impact of the empty cell? would break something?
16:22:20 EmilienM I think part of the reason is because in the past it used to requires some API to be running
16:22:28 mdbooth cfriesen: It's used for rollback of Rbd disks.
16:22:33 alaski mwhahaha: I'm saying that as a deployer I would not add a new cell to the system unless I had computes ready to add to that cell. I'm not going to stand up a new cell that won't do anything yet
16:22:58 mdbooth cfriesen: IIRC it's a no-op for all other backends.
16:23:24 mwhahaha alaski: well if i know i'm goign to have computes, i'd want to create the cell but wait to add the computes until i get to that phase of my install
16:23:24 mdbooth However, cold migration/resize have a resize->confirm/rollback workflow
16:23:49 mwhahaha alaski: that's the thing, fuel, kolla, tripleo we all do controller configs first then add computes later because we need to have keystone and stuff running first
16:23:51 mdbooth For disk-based local storage, we copy the instance dir, whether single host or migrating host
16:24:10 mdbooth For rbd, they're the same disks, so if we didn't snapshot we wouldn't be able to rollback.
16:24:11 alaski mwhahaha: an empty cell should not break anything afaik, but I've been out of cells for a bit now so I may be mistaken.
16:24:19 cfriesen mdbooth: okay, that makes sense. missed the fact it was a no-op for most cases.
16:24:26 mriedem alaski: is your perspective on that model biased to how it was done with cells v1?
16:24:26 mwhahaha alaski: sure, just trying to understand this better
16:25:06 mriedem an empty cell shouldn't break anything, no
16:25:07 mdbooth cfriesen: It's a reasonable hook. Possible other storage could be cleverer under some circumstances, but Rbd requires it.
16:25:17 alaski mriedem: very likely, I'm irreparably damaged from v1
16:25:27 mriedem alaski: you're safe now, it's OK :)
16:25:38 openstackgerrit Sven Anderson proposed openstack/nova: Fix suggested database migration command https://review.openstack.org/420060
16:25:43 cfriesen mdbooth: agreed, it's fine. I just didn't understand what was going on.
16:26:01 mriedem ok so i've updated https://wiki.openstack.org/wiki/Nova-Cells-v2#Nova_Cells_v2 with bugs, todos and open questoins
16:26:03 mriedem *questions
16:26:27 alaski note that I'm not trying to argue for not allowing an empty cell, just explaining why that assumption was in my head when writing code which likely affected the direction
16:27:11 dtp dansmith melwitt i'm at your disposal today
16:27:35 mriedem alaski: yup
16:28:02 dims mriedem : bauzas : https://review.openstack.org/#/c/419476/ gets the full cinder/rally test working. that test does a nova boot/attach/detach/delete etc. should be easy to review as well (almost all code protected with six.PY3 checks so wont regress existing stuff too)
16:28:03 melwitt dtp: dansmith is on PTO today FYI
16:28:10 dtp ok
16:29:44 alaski o/
16:29:54 melwitt so things used to work prior to that patch because the first run of simple_cell_setup created cell0 and then failed when it found no compute nodes, then the retry found cell0 and returned 0
16:30:15 melwitt so it continued on without creating any cell mapping
16:30:54 bauzas melwitt: yup
16:33:04 dtp melwitt i'll just be reviewing your and dan's changes under your blueprints. lemme know if there's something else i should do
16:35:14 dtp if i don't respond immdediately, i'm trapped in a meeting
16:36:13 openstackgerrit Balazs Gibizer proposed openstack/nova: handle uninited fields in notification payload https://review.openstack.org/415857
16:36:44 mriedem melwitt: so the question right now is shoud this be a 1 or a 0 https://github.com/openstack/nova/blob/3ec43d81c324d8229bbb4b5db301175886a049b6/nova/cmd/manage.py#L1114
16:37:42 mriedem that's also called from the map_cell_and_hosts CLI https://github.com/openstack/nova/blob/3ec43d81c324d8229bbb4b5db301175886a049b6/nova/cmd/manage.py#L1316
16:37:56 mriedem note the comment in there about what the return codes mean (0 everything is done, 1 means you have more to do)
16:38:13 melwitt yeah. I think the 1 is supposed to be a signal that "hey, I didn't create any cell mappings other than cell0"
16:38:17 mriedem i think i took issue with that a bit during a review because i consider >0 == error
16:38:34 sdague which is the standard unixism
16:38:38 melwitt yeah
16:38:50 sdague set -e on bash, and this will blow up any script running it
16:39:22 bauzas mriedem: I think that if we wanna have simple_cell_setup idempotent, it's acceptable to have it return 0 if you have no hosts
16:39:53 mriedem right i tend to think that simple_cell_setup should not return >0 unless there is a real error
16:39:57 melwitt that's what I'm not sure of. I thought the command means "create cell mappings" and if there are no hosts, it doesn't
16:39:58 bauzas I agree
16:40:07 bauzas well, I agree with mriedem
16:40:23 mriedem map_cell_and_hosts *should* return 1 if there are no computes
16:40:33 owalsh mriedem: what's worse is that unhandled exceptions will return 1
16:40:35 mriedem because you're specifically telling it to map hosts to cell mappings and if there are no computes, it's an error
16:40:44 melwitt mriedem: okay, I see what you're getting at
16:40:46 mriedem owalsh: well it depends on what the exception is
16:40:47 bauzas mriedem: I agree (again)
16:41:13 bauzas people could put simple_cells_setup in a cron and leave it run
16:41:20 mriedem we also need to, at some point in the near future, probably do an audit of the various commands and figure out in which scenarios you'd use them
16:41:24 mriedem because we have a glut of CLIs for this right now
16:41:29 owalsh mriedem: anything uncaught that subclasses Exception it caugh in __main__
16:41:42 mriedem owalsh: i don't see a problem with that
16:42:00 bauzas mriedem: melwitt: fancy leaving me submit a patch for changing the return code to 0 ?
16:42:02 mriedem owalsh: if we can't connect to the database and it raises an exception, then i think we should return a non-0 exit code
16:42:05 dims mriedem : "IBM PowerKVM" CI uses github to pull stuff? "GIT_BASE=git://github.com" (http://dal05.objectstorage.softlayer.net/v1/AUTH_3d8e6ecb-f597-448c-8ec2-164e9f710dd6/pkvmci/nova/76/419476/11/check/tempest-dsvm-full-xenial/2649398/console.html)
16:42:18 bauzas mriedem: yup, if we can't connect, it's a very different situation
16:42:20 mriedem dims: ask mmedvede
16:42:23 bauzas something is messed up
16:42:27 owalsh mriedem: yees, but having 1 == on computes, 2 == something else doesn't communicate any info
16:42:35 owalsh s/on/no/
16:42:51 dims mmedvede : looks like "IBM PowerKVM CI" got busted when github went down
16:43:02 openstack bug 1656276 in tripleo "Error running nova-manage cell_v2 simple_cell_setup when configuring nova with puppet-nova" [Critical,Triaged] https://launchpad.net/bugs/1656276
16:43:02 mriedem bauzas: i think that's something we should at least put up a patch for yeah for bug 1656276 - make simple_cell_setup return 0 if map_cell_and_hosts returns None, and change map_cell_and_hosts (the CLI) to return 1 if there is nothing mapped
16:43:28 mriedem owalsh: i'm saying i don't like a non-0 exit code communicating non-error conditions,
16:43:29 bauzas mriedem: yup, and you reported a bug too ?
16:43:39 mriedem i.e. 1 == no computes
16:43:44 mriedem bauzas: that's bug 1656276
16:43:50 mriedem it's what tripleo is hitting
16:44:07 rfolco dims, looking
16:44:13 mmedvede dims: yeah, we had problems pulling from git.o.o at some point
16:44:17 openstack Launchpad bug 1656017 in OpenStack Compute (nova) "nova-manage cell_v2 map_cell0 always returns a non-0 exit code" [Medium,Confirmed]
16:44:17 bauzas mriedem: I was thinking of https://bugs.launchpad.net/nova/+bug/1656017
16:44:18 mmedvede that is the reason
16:44:26 mriedem bauzas: that's different
16:44:35 bauzas mriedem: but yeah, the patch I'm writing on is against 1656276
16:44:35 dims mmedvede : interesting, talked to our infra folks?
16:44:43 bauzas mriedem: ie. changing what we return
16:44:56 mriedem dtp: you could tackle bug 1656017 if you want
16:45:06 mriedem dtp: that fix should be pretty straight-forward
16:45:17 mmedvede dims: it appeared to be a problem with our internal network, so I did not discuss with infra
16:45:19 dtp ok, i'll check it out
16:45:28 dims mmedvede : ack. thanks
16:45:29 mriedem bauzas: melwitt: the only reservation i have is that simpe_cell_setup will now return 0 for an incomplete setup
16:45:41 mriedem bauzas: melwitt: and we need to understand the implications of that

Earlier   Later