Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-03-02
17:05:11 mriedem we already use oslo.cache in several places so it should be a common pattern
17:05:44 mriedem even with a timeout on the cache, we still have a timing issue between when a new cell is added and when the cache invalidates and gets updated
17:06:02 mriedem having said that, we don't expect new cells to be added all the time
17:06:10 mdbooth Right. Unless you've got a distributed cache, you're always going to have a broken period there.
17:06:55 mdbooth Could you put it on the deployer? Before enabling scheduling to a new cell, all n-api need to be restarted?
17:07:48 mdbooth Ugly, but deterministic.
17:08:46 mriedem i believe that's what we've said if you update db or transport url connection info in a cell mapping
17:10:49 mriedem yeah https://docs.openstack.org/developer/nova/man/nova-manage.html#nova-cells-v2
17:10:54 mriedem "NOTE: Updating the transport_url or database_connection fields on a running system will NOT result in all nodes immediately using the new values. Use caution when changing these values."
17:12:52 mdbooth mriedem: Is it possible to add a cell without allowing instances to be scheduled there, btw?
17:13:11 dansmith we are really a long way from needing to overthink this cache thing
17:13:14 mriedem mdbooth: if the cell doesn't have hostmappings
17:13:30 mriedem if it doesn't have host mappings nothing goes into it
17:13:34 dansmith you can't even have multiple cells today
17:13:36 dansmith so a static list is *fine*
17:13:40 mdbooth dansmith: Yeah, with the restarting things I was backing away from distributed caches :)
17:17:42 mriedem dansmith: so you want me to just change this to check for len 1 and just always re-query until we have more than 1? https://review.openstack.org/#/c/440652/1/nova/compute/api.py
17:18:00 dansmith mriedem: it's not what I want, but I will accept it :)
17:18:09 dansmith we need to fix devstack anyway right?
17:18:30 mriedem i think so
17:18:46 mriedem i only worry about fixing devstack and others in the wild falling into the same trap
17:21:34 dansmith mriedem: this is exactly the same as the service catalog thing to me
17:21:35 dansmith mriedem: but, like I say, whatever you want
17:25:57 mriedem dansmith: service catalog re: order of steps during deploy?
17:26:02 dansmith yeah
17:27:20 mriedem ok, so maybe i'll just do the devstack change first, hold off on the nova patch and keep it in the pocket
17:28:19 mriedem we could log a warning if there is only 1 cell cached when listing instances, saying hey dummy you started n-api before all of your cells were mapped, you need to create the cells and restart nova-api
17:30:21 dansmith yeah, totes
17:34:39 openstackgerrit Eric Fried proposed openstack/nova master: PowerVM Driver: spawn/destroy #3: TaskFlow https://review.openstack.org/438729
17:36:05 sileht mriedem: thx, should abandon mine ?
17:36:43 mriedem sileht: it's probably still a good idea to not load extras/plugins until nova is setup
17:40:05 mriedem dansmith: do you remember why for cellsv1 at the end we sync the db? https://github.com/openstack-dev/devstack/blob/master/lib/nova#L978
17:40:11 mriedem after creating cell1 and mapping the hosts to it
17:40:40 dansmith that line is for cellsv2
17:41:21 dansmith mriedem: I think some of this came from just doing cellsv2 stuff where we had bits for cellsv1 which isn't necessarily sensical
17:41:35 mriedem that's cellsv1
17:41:39 mriedem if not n-cell is above
17:41:42 mriedem n-cell is v1
17:41:52 mriedem i'm going to remove it and see what happens
17:46:41 dansmith oh sorry I see I read it wrong
17:47:04 dansmith so .. no I'm not sure
17:52:01 mriedem dansmith: let's find out https://review.openstack.org/440739
17:53:12 mriedem i'm going to lunch
18:02:54 openstackgerrit Anusha Unnam proposed openstack/nova master: POC: add descriptions to server tags https://review.openstack.org/434842
18:04:04 openstackgerrit Jay Pipes proposed openstack/nova master: placement: ironic inventory reporting https://review.openstack.org/437602
18:04:15 jaypipes cdent: ^^ fixed your requests
18:04:27 cdent rad
18:15:04 openstackgerrit Dan Smith proposed openstack/nova master: WIP: Get instance availability_zone without hitting the api db https://review.openstack.org/439754
18:15:05 openstackgerrit Dan Smith proposed openstack/nova master: Change MQ targeting to honor only what is in the context https://review.openstack.org/438023
18:15:05 openstackgerrit Dan Smith proposed openstack/nova master: Target cell in super conductor operations https://review.openstack.org/438022
18:15:06 openstackgerrit Dan Smith proposed openstack/nova master: Add cell field to Destination object https://review.openstack.org/438024
18:15:06 openstackgerrit Dan Smith proposed openstack/nova master: Make scheduler target cells to get compute node instance info https://review.openstack.org/439891
18:15:07 openstackgerrit Dan Smith proposed openstack/nova master: WIP: Make conductor ask scheduler to limit migrates to same cell https://review.openstack.org/438025
18:18:44 openstackgerrit John Garbutt proposed openstack/nova-specs master: Add spec to use cinder's new attachment API https://review.openstack.org/373203
18:48:56 dansmith cdent: edleafe since jaypipes isn't around.. do you recall why he wants ironic to be special in the situation where custom resource classes are auto-created?
18:49:12 dansmith I'm not sure I ever knew, so maybe recall is the wrong word
18:50:09 cdent dansmith: my recollection is thin but I think it was because we already promised we would, but that we should not extend that promise to anything else. As you've asked, "why not?" is a fair question on that.
18:50:23 dansmith hmm, okay
18:51:04 dansmith if there is a sunset date on that, then maybe I'm on board, like after ironic creates and maintains these themselves or something
18:51:07 cdent It seems to me that if we're in the context of the compute-node then we may as well auto create because we're in charge there
18:51:19 dansmith right, and virt drivers are in-tree
18:51:23 cdent and if we do it for one, may as well for anything
18:51:35 dansmith yeah
18:51:36 dansmith and you could use the same logic,
18:51:51 dansmith in that until operators are familiar with placement, we auto-create things for them from all virt drivers
18:52:28 cdent the issue with that, of course, is that whole "precedent" thing. If we start auto creating willy nilly now, then everyone will want it later or never adapt to the time when we don't
18:52:29 dansmith also, if your virt driver knows about VFs, PFs, and GPUs, but can't report them until you create a class for them or something.. I dunno, kinda feels too manual to me
18:52:40 dansmith well, I dunno
18:52:53 dansmith but I also don't really know why we wouldn't auto-create anyway, so..
18:53:03 cdent Yeah, that's the crux
18:53:05 dansmith but, sounds like I'm not missing something major, which is good
18:53:39 cdent as far as I recall, you're not, but it often feels like there are major things happening while I blink
18:54:09 dansmith well, it's enough to justify my -1 until we get jay back to discuss at least :)
18:59:05 edleafe dansmith: there was a little bit of provincialism, in that Nova resources have standard names, whereas Ironic resources all have to be "CUSTOM_blah"
18:59:41 dansmith edleafe: yeah, but any other driver could expose a custom one
18:59:56 dansmith edleafe: if we merged that code, we clearly thought it was legit, so why not auto-create it in the same place?
19:01:42 edleafe dansmith: dunno. Maybe Jay remembers why
19:07:06 mriedem1 blarg!
19:07:21 mriedem init_nova_cells is called after init_nova
19:11:56 mriedem sileht: i've updated the dependency for https://review.openstack.org/#/c/440657/ to be the devstack fix i pushed
19:12:10 mriedem it already passed in the non-cells v1 case, and i just fixed it for the cells v1 case
19:22:07 melwitt mriedem: I looked at rajesh's email earlier, he's been working on the problem too and will be proposing some changes, once we figure out which direction to go
19:29:36 cfriesen Is anyone aware of a Mitaka bug where rows in instance_system_metadata for a given instance don't get deleted when the instance gets deleted? I've got a system with 400K rows in "instance_system_metadata" where deleted_at is null, and only 40 rows in "instances" where deleted_at is null.
19:30:54 mriedem yes
19:34:26 mriedem1 cfriesen: you should have this in mitaka https://github.com/openstack/nova/commit/2c48ff44a21083340cec35242a9fc006d3cd828d
19:35:28 mriedem1 ^ was fixed in liberty
19:35:43 mriedem1 that doesn't mean you don't have a ton of old orphaned stuff before that
19:36:49 cfriesen mriedem1: my bad, this DB was actually from Kilo. I made an incorrect assumption. That's almost certainly the issue. Thanks.
19:44:35 mriedem cfriesen: you can repay me by sending sdague the nickel i owe him
19:47:46 cfriesen that'll be 6 cents Canadian.
19:48:48 sdague I'm going to have to play these exchange rate games and work my self up to 2 bits
19:55:37 _gryf dansmith, I may be wrong, but idea to have ironic custom-class to be auto created is be backward compatible
19:55:57 _gryf I mean, there is no discovery mechanism for custom resources
19:56:21 _gryf and in past releases we do support ironic resource, right?
20:06:52 openstackgerrit Matt Riedemann proposed openstack/nova master: Log a warning if there is only one cell when listing instances https://review.openstack.org/440652
20:08:18 edleafe _gryf: there weren't resources in past releases
20:09:09 _gryf edleafe, but it was tracked by resource tracker
20:09:26 melwitt mriedem: I also have reproposed the spec for using the quota-related tables in the api db here https://review.openstack.org/#/c/431748 in case you didn't know
20:09:26 _gryf maybe that was the reason
20:09:32 edleafe _gryf: yes, but that was VCPU, RAM, disk, etc.

Earlier   Later