Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-03-02
16:43:58 mriedem so the proper fix is probably what i described above, with creating cell1 at the same time as cell0
16:44:08 dansmith mriedem: yeah
16:44:10 mriedem and then change simple_cell_setup in the post-setup to just discover_hosts
16:44:17 mriedem or map_cell_and_hosts or whatever
16:44:51 dansmith not map_cell_and_hosts
16:44:52 mriedem i don't know if/how that messes up upgrade because we'd already have cell1
16:44:53 dansmith that creates a cell
16:44:56 dansmith just discover at the end
16:46:05 mriedem we need to not create cell1 if it already exists, which we could maybe figure out with the list_cells command
16:46:40 mriedem jaypipes: +2 on https://review.openstack.org/#/c/440678/
16:47:54 dansmith mriedem: that point where we create cell0 we're on an empty db regardless of upgrade because we're on the old side right?
16:49:13 mdbooth Is this going to be a problem in general when adding a cell? If you can add a cell and have instances scheduled to it instantly, but those instances won't be visible, isn't this something you're always going to hit when adding a cell?
16:49:14 jaypipes mriedem: cheers. the end goal here is to release os-traits lib with a git tag today so we can begin working on tests for that "sync os-traits" thing in placement-manage per bauzas and alex_xu's discussion on the traits spec.
16:50:01 mdbooth iow is it worth fixing this with something a bit more general purpose than a workaround for cell1?
16:50:32 mriedem dansmith: i think https://github.com/openstack-dev/devstack/blob/master/lib/nova#L700 is the dirty secret
16:50:40 mriedem we drop and recreate the nova_api db
16:50:51 mriedem dansmith: same for cell0
16:50:58 dansmith mriedem: eh?
16:51:03 mriedem i think from newton->ocata that wasn't a problem for cell0 because in newton we didn't put anything into cell0
16:51:15 mriedem but ocata cell0 could have stuff in it
16:51:22 dansmith init_nova only runs on the old side right?
16:51:31 mriedem idk
16:51:57 mriedem i thought it ran on both but i don't have grenade memorized
16:52:02 dansmith we blow away all the dbs there.. nothing would work if we ran it in both places
16:53:34 mriedem ok. i was thinking of this https://review.openstack.org/#/c/393441/2/projects/60_nova/from-newton/upgrade-nova@8
16:53:42 mriedem unrelated and confusing myself i suppose
16:54:15 mriedem anyway, i can work a change to do create_cell cell1 after map_cell0 and change simple_cell_setup to be discover_hosts in devstack
16:54:24 mriedem and if grenade hates it i guess we'll find out
16:55:46 openstackgerrit Lee Yarwood proposed openstack/nova master: WIP compute: Move detach logic from manager into driver BDM https://review.openstack.org/439520
16:55:47 openstackgerrit Lee Yarwood proposed openstack/nova master: compute: Only destroy BDMs after successful detach call https://review.openstack.org/440693
16:58:31 aunnam johnthetubaguy, Will you be updating the poc patch https://review.openstack.org/#/c/434842/ by removing scope and access as they are not needed right now?
16:59:12 johnthetubaguy aunnam: I was hoping you or sneti might fancy doing that :)
16:59:18 aunnam johnthetubaguy, Or you want us to update the patch?
16:59:35 aunnam johnthetubaguy, sure will do that, thanks
17:00:32 mriedem sileht: looks like my nova workaround patch fixed the gnocchi issue
17:02:20 mdbooth mriedem: re the CELLs cache, would it be feasible to manage that with memcached via oslo_cache? Could a newly created cell poke itself into the cache before allowing anything to be scheduled there?
17:02:55 mdbooth Should mean you'd have no weirdness period after creating a new cell.
17:03:13 mriedem mdbooth: at a minimum yes on oslo.cache to add a timer to it
17:03:19 mriedem mdbooth: feel free to start that
17:03:36 mriedem as for how to auto-register a new cell to invalidate the cache or put itself into it, i'm not sure how we do that right now
17:03:57 mdbooth mriedem: Well I was explicitly thinking of a distributed cache, like memcached
17:04:17 mriedem with oslo.cache the backing cache store is abstracted
17:04:20 mriedem i thought anyway
17:04:28 mdbooth I think we have the infrastructure in place for that already, but I don't know what the deployment concerns would be
17:04:40 mdbooth Yes, looks like it is
17:05:11 mriedem we already use oslo.cache in several places so it should be a common pattern
17:05:44 mriedem even with a timeout on the cache, we still have a timing issue between when a new cell is added and when the cache invalidates and gets updated
17:06:02 mriedem having said that, we don't expect new cells to be added all the time
17:06:10 mdbooth Right. Unless you've got a distributed cache, you're always going to have a broken period there.
17:06:55 mdbooth Could you put it on the deployer? Before enabling scheduling to a new cell, all n-api need to be restarted?
17:07:48 mdbooth Ugly, but deterministic.
17:08:46 mriedem i believe that's what we've said if you update db or transport url connection info in a cell mapping
17:10:49 mriedem yeah https://docs.openstack.org/developer/nova/man/nova-manage.html#nova-cells-v2
17:10:54 mriedem "NOTE: Updating the transport_url or database_connection fields on a running system will NOT result in all nodes immediately using the new values. Use caution when changing these values."
17:12:52 mdbooth mriedem: Is it possible to add a cell without allowing instances to be scheduled there, btw?
17:13:11 dansmith we are really a long way from needing to overthink this cache thing
17:13:14 mriedem mdbooth: if the cell doesn't have hostmappings
17:13:30 mriedem if it doesn't have host mappings nothing goes into it
17:13:34 dansmith you can't even have multiple cells today
17:13:36 dansmith so a static list is *fine*
17:13:40 mdbooth dansmith: Yeah, with the restarting things I was backing away from distributed caches :)
17:17:42 mriedem dansmith: so you want me to just change this to check for len 1 and just always re-query until we have more than 1? https://review.openstack.org/#/c/440652/1/nova/compute/api.py
17:18:00 dansmith mriedem: it's not what I want, but I will accept it :)
17:18:09 dansmith we need to fix devstack anyway right?
17:18:30 mriedem i think so
17:18:46 mriedem i only worry about fixing devstack and others in the wild falling into the same trap
17:21:34 dansmith mriedem: this is exactly the same as the service catalog thing to me
17:21:35 dansmith mriedem: but, like I say, whatever you want
17:25:57 mriedem dansmith: service catalog re: order of steps during deploy?
17:26:02 dansmith yeah
17:27:20 mriedem ok, so maybe i'll just do the devstack change first, hold off on the nova patch and keep it in the pocket
17:28:19 mriedem we could log a warning if there is only 1 cell cached when listing instances, saying hey dummy you started n-api before all of your cells were mapped, you need to create the cells and restart nova-api
17:30:21 dansmith yeah, totes
17:34:39 openstackgerrit Eric Fried proposed openstack/nova master: PowerVM Driver: spawn/destroy #3: TaskFlow https://review.openstack.org/438729
17:36:05 sileht mriedem: thx, should abandon mine ?
17:36:43 mriedem sileht: it's probably still a good idea to not load extras/plugins until nova is setup
17:40:05 mriedem dansmith: do you remember why for cellsv1 at the end we sync the db? https://github.com/openstack-dev/devstack/blob/master/lib/nova#L978
17:40:11 mriedem after creating cell1 and mapping the hosts to it
17:40:40 dansmith that line is for cellsv2
17:41:21 dansmith mriedem: I think some of this came from just doing cellsv2 stuff where we had bits for cellsv1 which isn't necessarily sensical
17:41:35 mriedem that's cellsv1
17:41:39 mriedem if not n-cell is above
17:41:42 mriedem n-cell is v1
17:41:52 mriedem i'm going to remove it and see what happens
17:46:41 dansmith oh sorry I see I read it wrong
17:47:04 dansmith so .. no I'm not sure
17:52:01 mriedem dansmith: let's find out https://review.openstack.org/440739
17:53:12 mriedem i'm going to lunch
18:02:54 openstackgerrit Anusha Unnam proposed openstack/nova master: POC: add descriptions to server tags https://review.openstack.org/434842
18:04:04 openstackgerrit Jay Pipes proposed openstack/nova master: placement: ironic inventory reporting https://review.openstack.org/437602
18:04:15 jaypipes cdent: ^^ fixed your requests
18:04:27 cdent rad
18:15:04 openstackgerrit Dan Smith proposed openstack/nova master: WIP: Get instance availability_zone without hitting the api db https://review.openstack.org/439754
18:15:05 openstackgerrit Dan Smith proposed openstack/nova master: Change MQ targeting to honor only what is in the context https://review.openstack.org/438023
18:15:05 openstackgerrit Dan Smith proposed openstack/nova master: Target cell in super conductor operations https://review.openstack.org/438022
18:15:06 openstackgerrit Dan Smith proposed openstack/nova master: Add cell field to Destination object https://review.openstack.org/438024
18:15:06 openstackgerrit Dan Smith proposed openstack/nova master: Make scheduler target cells to get compute node instance info https://review.openstack.org/439891
18:15:07 openstackgerrit Dan Smith proposed openstack/nova master: WIP: Make conductor ask scheduler to limit migrates to same cell https://review.openstack.org/438025
18:18:44 openstackgerrit John Garbutt proposed openstack/nova-specs master: Add spec to use cinder's new attachment API https://review.openstack.org/373203
18:48:56 dansmith cdent: edleafe since jaypipes isn't around.. do you recall why he wants ironic to be special in the situation where custom resource classes are auto-created?

Earlier   Later