Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-07-26
15:01:53 mriedem could tease out some interesting bugs we don't know about yet
15:02:16 mriedem where is CELL_INDEX?
15:02:16 dansmith well, I didn't know we had three node jobs until you told me yesterday
15:02:49 dansmith I rememberd the name wrong, but it's this: https://github.com/openstack-dev/devstack/blob/master/lib/nova#L66-L68
15:03:45 mriedem couldn't that just be: NOVA_CPU_CELL=${NOVA_CPU_CELL:-$NOVA_NUM_CELLS} ?
15:03:48 dansmith I just want to say that I wish we had taken my original suggestion to call cellsv2 something other than cells
15:03:49 mriedem are they always the same?
15:03:57 sdague ok, I'm trying to figure out a way to explain this in stackrc that isn't going to require poking people in irc.
15:04:03 dansmith no,
15:04:16 dansmith mriedem: you need it to be different on each node you want to be in a different cell
15:05:02 mriedem ah ok
15:05:04 sdague so, the fleetify patch creates a conductor per compute.
15:05:10 sdague plus the api conductor
15:05:11 mriedem conductor per node
15:05:16 dansmith condutor per cell
15:05:26 mriedem well,
15:05:31 sdague right, it's good we're all on the same page :)
15:05:32 mriedem not really
15:05:45 mriedem dansmith: today the multinode job with this change is 1 cell
15:05:46 openstackgerrit Eric Fried proposed openstack/nova master: Add requirement for os-service-types https://review.openstack.org/487471
15:06:01 mriedem there is a cell conductor and super conductor
15:06:02 dansmith mriedem: it's conductor per cell, plus one super conductor
15:06:02 mriedem yeah,
15:06:03 sdague mriedem: right, so 1 compute conductor + 1 api conductor, right ?
15:06:06 mriedem and the subnodes talk to the cell conductor
15:06:09 mriedem api talks to super conductor
15:06:28 mriedem fwiw this is why i wrote a todo to doc this in devstack later
15:06:48 mriedem because we have 3 nova confs now and multiple conductors so debugging it is a challenge at first
15:06:55 openstackgerrit OpenStack Proposal Bot proposed openstack/nova master: Updated from global requirements https://review.openstack.org/487473
15:06:57 sdague so, if we introduced this as CELLSV2_SETUP=allinone
15:06:59 dansmith mriedem: trying to help by starting that nova doc with diagrams of course
15:07:04 mriedem dansmith: yup
15:07:09 sdague CELLSV2_SETUP=condpercell
15:07:09 mriedem that's higher priority IMO anyway
15:07:38 sdague or something, then we could global switch there?
15:07:42 dansmith sdague: if it wasn't for ironic (and grenade) I'd say we don't need a flag
15:07:56 sdague dansmith: sure, ironic is always going to be broken this way?
15:08:01 dansmith sdague: and we really don't want people doing the allinone if possible
15:08:11 dansmith no
15:08:41 sdague what's the ironic path fix here?
15:08:41 dansmith ironic will stop being broken when we get the claims patch in,
15:08:56 dansmith and when we have proper reschedules (which we've punted at this point) then double fixed
15:09:28 mriedem 1. ironic sets the tiered=False flag temporarily in devstack-plugin-ironic,
15:09:30 mriedem 2. we land https://review.openstack.org/#/c/483566/
15:09:34 mriedem 3. ironic reverts #1
15:09:35 mriedem right?
15:09:45 dansmith technically yes
15:10:02 dansmith although since we punted on reschedules,
15:10:03 cdent If any core types have gaps between wtf-ing at cellsv2 and claims in the scheduler, there’s a stack of api-ref stuff for placement where at least the first few are ready to go: https://review.openstack.org/#/c/474550/
15:10:06 dansmith I think it's probably a good idea to let people setup devstack in this older configuration for the time being,
15:10:14 dansmith because otherwise they can't reschedule
15:10:31 mriedem are these ironic multi-node jobs?
15:10:38 mriedem or just ironic being multi-node by design
15:10:41 mriedem because of the 1:M thing
15:10:55 dansmith well, ironic multinode jobs, but not sure if it's multi-compute or just multi-ironic-node
15:11:18 dansmith either requires a reschedule, which is the problem
15:11:23 mriedem that's a good point about reschedules
15:12:02 dansmith mriedem: sdague: you want to hangout to hash this over?
15:12:09 dansmith sdague hasn't said anything in a few minutes, may be very confused
15:12:40 mriedem i just left a comment that it might be good to mention the reschedule thing,
15:12:45 mriedem if you're doing tiered and multiple computes
15:14:07 bauzas mriedem: 11 comments in https://review.openstack.org/#/c/485435/6 but only 2 I care
15:14:53 bauzas mriedem: tl;dr should we mention in the relnote that nova wouldn't support disabling non-compute services, and why should we having two different representative strings for a service object?
15:15:03 sdague dansmith: sorry, I was writing a patch
15:15:11 sdague I can do a hangout as well
15:15:37 sdague dansmith: I was thinking, change the devstack setup to this https://review.openstack.org/487478
15:16:05 sdague because as much as I don't like this mode switch, I like it even less if there are 2 different ways to do it
15:16:19 dansmith sdague: we have to fix grenade for that first
15:16:24 sdague dansmith: yep
15:16:29 sdague working on that patch right now
15:16:39 dansmith sdague: agreed about the two ways, I was just trying to unbreak the ironic people quickly
15:17:14 dansmith vdrok: following?
15:17:37 vdrok reading
15:18:02 dansmith vdrok: basically, sdague hates it when I play with his toys
15:19:03 mriedem here is the 2-cell goody btw https://review.openstack.org/487479
15:19:36 dansmith mriedem: that's just going to put all the nodes into the same cell again right?
15:20:02 dansmith you need to set CPU_CELL=$node_number or something
15:20:36 dansmith this will just end up with an empty cell1, which is cool, but not much different from what we're already doing, AFAICT
15:21:19 vdrok dansmith: so basically CELLSV2_SETUP=singleconductor instead of NOVA_CELLS_TIER=False?
15:21:26 dansmith vdrok: yes
15:21:27 mriedem vdrok: yes
15:21:34 mriedem dansmith: i'd have to set that from like devstack-gate right?
15:21:41 dansmith mriedem: something yeah
15:21:43 mriedem since devstack doesn't know which subnode it is
15:21:44 mriedem ok
15:21:45 dansmith right
15:21:56 openstackgerrit Jay Pipes proposed openstack/nova master: claim resources in placement API during schedule() https://review.openstack.org/483566
15:21:57 openstackgerrit Jay Pipes proposed openstack/nova master: add a retry on DBDeadlock to _set_allocations() https://review.openstack.org/487483
15:21:58 jaypipes alex_xu, cdent, mriedem, dansmith: ok, added protection against DBDeadlock (separate patch) and cleaned up the one conditional with unit tests that alex_xu pointed out. ^^
15:22:08 jaypipes and rebased to master that cdent wanted...
15:22:10 dansmith mriedem: I should have kept quiet and had you be amazed and impressed at no more failures with your patch :P
15:22:19 vdrok mriedem: dansmith hrm, so doing it in devstack/settings won't work? OK, will add to devstack-gate
15:22:24 cdent jaypipes: cool, will look asap
15:22:30 vdrok smth like https://review.openstack.org/#/c/487458/1/devstack/settings
15:24:45 mriedem jaypipes: questions in the bottom change
15:24:59 dansmith vdrok: I think what you have will work right?
15:25:10 dansmith vdrok: as long as it gets loaded before the other stuff runs
15:25:35 vdrok dansmith: I think so, yes, it should run before nova
15:25:39 mriedem what he has should work, unless we don't go with that patch and instead go with seans
15:26:32 jaypipes mriedem: answered.

Earlier   Later