Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-01-11
19:44:48 EmilienM dansmith: yeah? I just found the bug in tripleo :-P
19:45:00 EmilienM we deploy placement but in some random cases, it was not starting.
19:45:18 dansmith EmilienM: I did, specifically to make the upgrade case a little more graceful
19:45:24 dansmith EmilienM: planning to backport that to newton
19:46:26 EmilienM dansmith++
19:47:25 EmilienM dansmith: do you think https://review.openstack.org/#/c/418590/ can merge today?
19:52:16 dansmith EmilienM: you should hassle someone that can do something about it
19:52:28 dansmith EmilienM: jaypipes once was +2 on that and may grant your wish
19:56:02 EmilienM jaypipes: in case you have some time to review, please: https://review.openstack.org/#/c/418590/
20:01:18 mriedem claudiub|2: you could emit a warning in the logs...
20:02:16 mriedem claudiub|2: i'd also get it into the numa docs in the admin guide: http://docs.openstack.org/admin-guide/compute-cpu-topologies.html - so tag the patch with DocImpact and leave a comment in the commit message about updating those docs?
20:02:31 diana_clarke mriedem: I wasn't sure what to work on instead, so I am going to keep playing with microversions & osc. We'll see how it goes. https://review.openstack.org/#/c/419108/
20:03:28 mriedem dansmith: are you going to tweek the docs in a follow up to https://review.openstack.org/#/c/418590/ ?
20:03:38 mriedem i.e. http://git.openstack.org/cgit/openstack/nova/tree/doc/source/placement.rst?id=f5d7ea52c681611c2d85f344da643f9f630d1ef9#n108
20:04:30 dansmith mriedem: tbh, it's not high on my list considering time available and the rush to unfsck the current stuff
20:04:38 dansmith mriedem: if you think it's a priority for right now then I will,
20:04:49 dansmith but if not, maybe we could find someone interested in doing it?
20:05:14 diana_clarke dansmith, mriedem: I can try and do it.
20:05:41 dansmith diana_clarke: cool, thanks
20:06:18 mriedem i don't really see how spamming the logs fixes the complaints i was hearing from the red hat people yesterday,
20:06:27 mriedem but if it's something you guys have agreed it makes your lives easier somehow, then ok
20:06:40 mriedem that's all, re: why i'm not rushing it in
20:07:01 dansmith mriedem: spamming the logs doesn't fix anything for us, continuing to try does
20:07:09 openstackgerrit Dan Smith proposed openstack/nova: Make metadata server know about cell mappings https://review.openstack.org/417962
20:07:11 openstackgerrit Dan Smith proposed openstack/nova: WTFWIP: Multicell support for instance listing https://review.openstack.org/396775
20:07:11 openstackgerrit Dan Smith proposed openstack/nova: Don't bypass cellsv1 replication if cellsv2 maps are in place https://review.openstack.org/418141
20:09:05 mriedem ok so in a point release rhops rolls out creds to the newton computes and restarts nova-compute on them, right? which as long as they keep trying to connect to placement, it just happens whenever rhosp rolls out the placement service in an update for newton? i guess i just figured the restart that you need to do for the computes would come after placement is available, at which point we don't need the continual retry
20:09:52 dansmith mriedem: we can't really enforce that ordering,
20:10:06 dansmith in general, and for a couple different approaches to the upgrade
20:10:33 mriedem ok so you/we might avoid an unnecessary restart of the computes depending on the order of the upgrade
20:10:35 mriedem which is the point
20:11:10 mriedem w/o the patch, you could have deployed creds updates to the computes, restarted them, then rolled out placement, at which point you have to go back and restart all of the computes again
20:11:22 mriedem and it's that potential 2nd mass restart that is the point of the patch (trying to avoid it)
20:12:52 dansmith at least that yeah
20:13:09 mriedem ok
20:13:16 mriedem i think i can smell what you're cooking now
20:14:18 dansmith so, that and this gets us a little more nag spam in the logs to make sure that people running newton see that this is required before they have kicked off their upgradeand decided to skim the release notes once things are broken
20:14:32 dansmith which goes back to our not so awesome messaging, retroactive "oh and also..." thing in the point release, etc
20:21:40 EmilienM dansmith: look at what we have to do when we missed nova cells v2 in newton https://review.openstack.org/#/c/419121/1/elements/puppet-stack-config/puppet-stack-config.pp
20:22:31 jaypipes mriedem, claudiub|2: hyper-v osvif patch reviewed.
20:23:20 dansmith EmilienM: what about it?
20:23:44 edleafe jaypipes: got a resource provider question if you have a few minutes
20:23:49 EmilienM dansmith: I just wanted to show you the complexity of things about nova cells v2 upgrade from newton to ocata
20:23:58 dansmith EmilienM: to make me feel bad?
20:24:10 EmilienM dansmith: no :-) so you can maybe give feedback
20:24:55 EmilienM and yes, hopefully you feel very sad
20:25:16 EmilienM because we missed it in tripleo/newton, now we have to do that thing
20:25:16 edleafe jaypipes: if I have a reference to a provider, say 'rp', and call rp.set_inventory(inv), that 'rp' reference is now stale, since it now 1 generation behind the DB.
20:25:16 openstackgerrit Diana Clarke proposed openstack/nova: docs - Connect to placement service & retries https://review.openstack.org/419123
20:25:41 EmilienM jaypipes: thx for the +w :-)
20:26:05 edleafe jaypipes: my question is: should methods that change generation also update the generation field in the current object?
20:26:18 claudiub|2 jaypipes: ty, will upload a new ps shortly. :D
20:26:42 diana_clarke dansmith: It's hard to write about things you don't know much about, but there goes ;) ^^
20:26:48 dansmith diana_clarke: looks good tome
20:26:51 dansmith also to me
20:27:41 dansmith diana_clarke: mriedem will -1 one time to eff up my stats, but otherwise I think it covers all the things
20:29:30 mriedem dansmith: EmilienM: jaypipes: postmortem https://review.openstack.org/#/c/418590/ +1 fwiw with a more detailed recap of my understanding of the issue the patch is trying to avoid/fix
20:29:39 mriedem i just got distracted with PTG talk in another channel
20:30:00 dansmith mriedem: it's not dead yet.. it's like peri-mortem
20:30:17 EmilienM mriedem: ack
20:30:40 EmilienM I like the idea of https://review.openstack.org/#/c/418590/ being backported fwiw
20:30:58 EmilienM I'm pretty sure it will really help deployers
20:31:23 jaypipes edleafe: https://github.com/openstack/nova/blob/master/nova/objects/resource_provider.py#L306
20:31:56 jaypipes EmilienM: np :)
20:32:10 edleafe jaypipes: that's a static method that only updates the DB
20:32:30 mriedem EmilienM: so in retrospec, i guess we're going to need to figure out a working relationship on how to communicate to the tripleo guys that even though something is optional now, you're going to want to bake it into your tooling, because release notes aren't going to cut it
20:32:33 edleafe jaypipes: the object that called it is now stale
20:32:35 mriedem *retrospect
20:32:49 EmilienM mriedem: I do agree
20:32:58 mriedem i.e. if nova has to touch grenade for ANY reason, someone from tripleo needs to know
20:33:08 mriedem s/tripleo/any of the 40 openstack deployment projects/
20:33:12 dansmith mriedem: I was just about to say, it's in the grenade history
20:33:23 jaypipes edleafe: https://github.com/openstack/nova/blob/master/nova/objects/resource_provider.py#L372
20:33:32 jaypipes edleafe: the object that called it is self.
20:33:34 edleafe jaypipes: IOW, if you make two set_inventory calls with the same object, the second will throw a ConcurrentUpdateDetected
20:33:46 dansmith mriedem: tempted to start up a twitter account and git commit hook :)
20:33:53 mriedem so i'll add something to the nova docs in our review guide - albeit chances are low anyone will read that
20:34:00 mriedem dansmith: you or me?
20:34:05 dansmith mriedem: I'm tempted
20:34:07 jaypipes edleafe: I don't believe that is true.
20:34:08 EmilienM mriedem: we are building a grenade like job in tripleo, where for the first time we'll test in gate upgrades with complex deployments
20:34:25 EmilienM mriedem: it might be useful to have the job in experimental pipeline in nova (one day)
20:34:40 edleafe jaypipes: I'm getting it in some of my tests.
20:35:03 openstackgerrit Claudiu Belu proposed openstack/nova: Adds Hyper-V OVS ViF driver https://review.openstack.org/140045
20:35:06 edleafe jaypipes: let me see if there is somewhere else it is getting changed
20:36:47 mriedem EmilienM: well the thing is, placement and cells v2 were optional in newton,
20:37:00 mriedem so we had to explicitly enable those in specific jobs in newton to test them out
20:37:29 mriedem EmilienM: which takes the forethought to know you have to toggle those bits, we're missing that as part of our process right now for non-devstack/grenade things
20:47:17 mriedem diana_clarke: i'll approve the doc update once https://review.openstack.org/#/c/418590/ is merged
20:48:07 diana_clarke mriedem: thanks, I probably should have has a depends on
20:48:13 diana_clarke had
20:48:14 openstackgerrit Eric Fried proposed openstack/nova: Introduce base PowerVM driver https://review.openstack.org/391288
20:49:03 openstackgerrit Matt Riedemann proposed openstack/nova: docs - Connect to placement service & retries https://review.openstack.org/419123
20:49:09 mriedem diana_clarke: done
20:50:23 openstackgerrit Eric Fried proposed openstack/nova: WIP(2): Introduce base PowerVM driver (disk drv) https://review.openstack.org/409401
20:50:36 edleafe jaypipes: found it. _check_capacity_exceeded() creates new RP objects, and these are the ones whose generation gets updated instead of the originals
20:50:39 edleafe jaypipes: https://github.com/openstack/nova/blob/master/nova/objects/resource_provider.py#L1078-L1081
20:51:33 openstackgerrit Eric Fried proposed openstack/nova: WIP(3): Introduce base PowerVM driver (console) https://review.openstack.org/409402
20:51:43 dansmith diana_clarke: mriedem: thanks
20:52:44 EmilienM diana_clarke: fyi, depends-on works for inter-projects connection, but for the same project, you better to use git rebase

Earlier   Later