Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-05-03
21:41:20 mriedem is that group going to have a cell-targeted context in it?
21:41:24 dansmith no,
21:41:27 dansmith because it's local to the cell
21:41:29 dansmith if it's compute manager
21:41:34 dansmith it can't go anywhere other than the local cell
21:41:38 mriedem so it just reads the [database]/connection
21:41:41 dansmith right
21:41:41 mriedem ah right
21:42:06 melwitt the thing I'm concerned about is whether my patch will make that go to all cells, which would be wrong?
21:42:56 dansmith does compute manager use setup_instance_group?
21:42:58 dansmith I didn't think so
21:43:14 melwitt oh, no it doesn't. okay, my bad
21:43:50 mriedem setup_instance_group is only called from conductor i think, prior to sending the request spec to the scheduler to pick a host
21:44:27 dansmith you know that there's still a race though, right?
21:44:31 dansmith that's what the late-binding check is for,
21:44:39 dansmith and we can't do that from the cell anyway
21:44:57 dansmith so melwitt's patch will close the race by not just getting an empty list every time,
21:45:09 dansmith but if we're concurrently booting several instances, you can still sprinkle them among the cells
21:45:33 dansmith although, that makes me wonder...
21:45:34 mriedem right i know for the concurrent boot scenario, we're already in trouble
21:45:41 dansmith instead of doing the check late like we do now,
21:46:12 dansmith what if we create the instance mapping (which means we've destined it to a cell) and then do the affinity check right there to make sure that the group hasn't spread across cells,
21:46:29 dansmith then once we've done that, we can let the late check happen in computemanager like normal, which will only need to consider the local cell anyway
21:47:22 mriedem there is still a race there though isn't there?
21:47:30 mriedem it's the same race as for getting on different computes
21:47:59 dansmith no, because if we know everything is in the same cell, then the late check as it is today works fine yeah?
21:48:07 dansmith oh, although we can't look up the server group members there
21:48:08 dansmith that's why
21:48:34 mriedem my brain hurts
21:49:17 mriedem so to summarize, i think with the concurrent affinity group create race, before multi-cell, you could race past the scheduler and late affinity check in the compute has to fix the race,
21:49:43 mriedem with multiple cells, you could race past the scheduler and the late affinity check in compute might not fail because it can't "see" that the group has members in another cell
21:49:47 mriedem to which it's not affined
21:49:54 mriedem and you end up with an affinity group with members in multiple cells
21:49:55 dansmith no,
21:50:21 dansmith the late check is just for anti-affinity
21:50:22 dansmith right?
21:50:26 mriedem no
21:50:37 dansmith oh, I guess it does check both
21:50:42 dansmith the comment only says anti-affinity
21:51:10 dansmith mriedem: so yes, your second comment is right, except that:
21:51:23 mriedem so before multi-cell, you'd reschedule until you find the right host or fail,
21:51:24 dansmith 1. for anti-affinity it doesn't matter since anything it doesn't find is clearly not on the same host and,
21:51:37 mriedem after multi-cell, we might not even see the problem
21:51:37 dansmith 2. the problem is we can't upcall to get the members, not just that we can't see them in our db
21:51:52 dansmith right but with multi-cell #2 gets you anyway
21:51:53 mriedem agree with 1 yes, anti-affinity is totes fine
21:51:55 dansmith because of the upcall
21:52:42 mriedem if the cell isn't configured to hit the api db, then yes we can't upcall to iterate the cells to find hosts for all members in the group
21:53:08 mriedem this https://github.com/openstack/nova/blob/0ef3c685b9d2e0049f38fcf1a268870e69a5b9cf/nova/objects/instance_group.py#L473
21:53:09 mriedem yeah?
21:53:38 dansmith well,
21:53:45 dansmith we won't iterate cells anyway, but yeah
21:58:34 mriedem so the only way to really fix this is for the scheduler to be aware of the location of the group members at the time of picking a host, and that has to be global, so placement,
21:58:59 mriedem and we likely need to model affinity (distance) for the group members using....provider aggregates? i know this is a rathole,
21:59:05 mriedem i keep thinking of cells as a provider aggregate
21:59:45 dansmith well, this is why I've been punting the problem because placement affinity is better for lots of reasons
22:00:04 mriedem melwitt: anyway, i think the functional change in your patch is ok, my main issue / concern is with the test
22:00:23 melwitt mriedem: yeah, in the middle of replying now
22:00:30 mriedem melwitt: a functional test with real services would be ideal
22:00:57 melwitt well, I think we need the unit test to cover all the weird paths, unless I'm just not thinking creatively enough
22:01:32 mriedem the majority of the work in a functional test for stuff like this is in the setup,
22:01:34 melwitt I'm gonna try to take a step back and see if I can come up with a real world scenario(s) that will take all of the paths that we can verify
22:01:37 mriedem after that it's just running scenarios and asserting stuff
22:02:07 mriedem https://review.openstack.org/#/c/565886/6/nova/tests/functional/test_nova_manage.py@357 is an example with 2 working cells
22:03:39 melwitt this is weird because there are two steps: 1) look up the hosts for members of the requested group 2) look up the hosts for members of the group the instance is a member of
22:04:38 melwitt and I was trying to test that both of those queries do the multi-cell thing. but yeah, probably could set this up for a functional test. I'll try it
22:05:37 mriedem with complicated changes like this, i find it's easier to write the functional test to setup the environment like the user would run a use case
22:05:48 mriedem using the actual APIs
22:05:53 mriedem to create the groups and add members to them and such
22:06:16 mriedem otherwise it's too easy to fake things out in the db that aren't accurate
22:07:17 melwitt for whatever reason, I did not expect it would be easy to do a functional test. I agree it would be a lot better to reason about too
22:07:29 mriedem mostly just copy/paste the setup
22:07:31 mriedem pretty easy
22:08:07 mriedem the one thing with multi-cell functional and having different hosts in different cells, you'll need https://review.openstack.org/#/c/558160/
22:08:19 mriedem otherwise the computes all get created in the default cell1
22:08:44 mriedem but that's approved now so shouldn't be a problem - you just need to specify the cell you want the compute in when you create it
22:09:25 mriedem i've been told i need to go workout because i've become somewhat of a troll, so ttyl
22:09:25 melwitt I know, I tried to solve that problem with my CellDatabases patch months ago but people weren't okay with it because I did the ServiceWrapper thing
22:23:03 openstackgerrit Jay Pipes proposed openstack/nova master: process groups individually and merge candidates https://review.openstack.org/566180
22:52:34 openstackgerrit Merged openstack/nova master: Handle @safe_connect returns None side effect in _ensure_resource_provider https://review.openstack.org/566096
22:58:45 openstackgerrit Merged openstack/nova master: Fix the request context in ServiceFixture https://review.openstack.org/558160
23:07:57 openstackgerrit Arvind Nadendla proposed openstack/nova-specs master: Handle rebuild of instance with new image https://review.openstack.org/560718
23:33:50 idlemind http://paste.openstack.org/show/720326/
23:34:02 idlemind I need those database connections to be updated to .9 not .11 ... what's the best way to do that?
23:34:25 idlemind can i just drop the cells (delete) and they'll get recreated?
23:34:42 idlemind or should i update them
23:46:08 openstackgerrit Oliver Walsh proposed openstack/nova stable/pike: Handle @safe_connect returns None side effect in _ensure_resource_provider https://review.openstack.org/566164
23:48:57 openstackgerrit Arvind Nadendla proposed openstack/nova-specs master: Handle rebuild of instance with new image https://review.openstack.org/560718
#openstack-nova - 2018-05-04
00:54:00 openstackgerrit Tetsuro Nakamura proposed openstack/nova master: Add tests for alloc_cands with member_of https://review.openstack.org/561399
00:54:01 openstackgerrit Tetsuro Nakamura proposed openstack/nova master: Fix member_of with sharing providers https://review.openstack.org/561400
00:54:02 openstackgerrit Tetsuro Nakamura proposed openstack/nova master: Expand member_of functional test cases https://review.openstack.org/566011
01:05:26 mrjazzercise idlemind: you've got 2 cell1s, so probably drop the one with the wrong URL
01:05:36 mrjazzercise and update the cell0 one using nova-manage cell_v2 update_cell
01:06:05 mrjazzercise some services cache the cells so you'll have to restart some services, check the man page on update_cell
01:06:21 mrjazzercise nova-api, nova-conductor and nova-scheduler i think
02:09:08 idlemind mrjazzercise thx
04:31:51 openstackgerrit jichenjc proposed openstack/nova master: WIP:[doc]Move configuration to admin subfolder https://review.openstack.org/566212
05:43:56 openstackgerrit jichenjc proposed openstack/nova master: [doc]Move configuration to admin subfolder https://review.openstack.org/566212
05:45:35 nsingh any command or way to confirm which services running on compute nodes???
06:48:25 openstackgerrit zhangyangyang proposed openstack/nova master: Remove the function get_back_port() https://review.openstack.org/566219

Earlier   Later