Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-01-06
21:54:11 superdan like, finally some sane failures: http://logs.openstack.org/75/396775/23/check/gate-tempest-dsvm-cells-ubuntu-xenial/c10b6b3/logs/screen-n-api.txt.gz?level=TRACE#_2017-01-06_20_19_05_133
21:54:19 melwitt superdan: heh, yeah. I saw there's a lot of UnknownCell errors in there now but hopefully that won't be too hard to track down
21:55:05 superdan yeah, I dunno what that means, but.. it's something to chase at least
21:55:23 melwitt definitely a big improvement over crazy pants
21:55:33 superdan huh, it means instance.cell_name is not set
21:56:23 melwitt meaning instance wasn't assigned to a cell v1 cell somehow. or hasn't been scheduled yet
21:56:40 superdan yeah
21:59:39 melwitt hm, so when an instance record is written to a cell db, when it syncs to the top the cell_name is set based on the message routing path?
22:00:17 melwitt I don't think I ever knew how cell_name got set, but it looks like it happens in instance_update_at_top in cells/messaging
22:01:24 superdan that's..odd
22:01:31 superdan I guess so the cell doesn't have to know its name or something?
22:01:48 melwitt seems like it. I dunno
22:04:04 superdan man
22:04:07 melwitt superdan: I wonder if it's something like, it's reading the instance from the cell db so it doesn't have cell_name there. cell_name is only in the record at the top?
22:04:15 superdan I'm just shocked that this continues to work with none of us really knowing about it
22:04:25 superdan yeah might be
22:05:06 melwitt yeah well, we've taken care not to touch the code too much. and I broke things when I object-ified some stuff there that had to get fixed
22:07:42 openstackgerrit Dan Smith proposed openstack/nova: WTFWIP: Multicell support for instance listing https://review.openstack.org/396775
22:07:44 superdan theory-testing hack ^
22:07:51 melwitt I wonder if there are any other problems with reading instances from the compute cell db rather than the api cell db. I hope not
22:09:13 superdan well,
22:09:28 superdan it may surprise some people that don't expect connections from api to those dbs
22:09:35 superdan but that's kinda part of the migration path anyway
22:09:36 melwitt I'm not sure that will work, because I don't think cell_name is dirtied in the compute cell
22:10:05 superdan oh, because that makes an rpc call back to messaging
22:10:32 melwitt yeah, the instance.save() will call up to instance_update_at_top, and that function will set cell_name only at the top
22:11:07 melwitt https://github.com/openstack/nova/blob/master/nova/cells/messaging.py#L1009
22:11:20 superdan right but you mean call up to the regional cell service, which is across rpc
22:11:31 melwitt and that's not going to sync back down
22:11:36 superdan wait
22:11:41 superdan oh yeah
22:11:48 melwitt yeah rpc call to the api cell
22:11:57 melwitt I think
22:12:04 superdan I was thinking that was a direct call into there because the name was the same, but I jumped the api->messaging gap
22:12:35 superdan so I should be able to get it from conf actually
22:12:50 melwitt well, I mean the compute cell will never run the code that changes the cell_name I mean
22:13:13 melwitt I dunno, I'm getting confused. anyway
22:15:08 openstackgerrit Dan Smith proposed openstack/nova: WTFWIP: Multicell support for instance listing https://review.openstack.org/396775
22:15:09 superdan hack @3
22:15:11 superdan er, #2
22:19:41 melwitt superdan: I think that might not work because I think the instance.create() goes only to the api cell db, then later when instance.save() happens in the api cell, that's when it gets written to the compute cell db :\
22:21:11 melwitt as part of the sync down
22:21:53 superdan melwitt: we have to create() once in the child cell
22:21:55 superdan we can't just save
22:23:33 melwitt superdan: okay. I had thought when we were looking at replays of compute/api the instance create isn't one of them and so I went looking around for how the record ever ends up in the compute cell db
22:24:25 superdan melwitt: well, we have to call db.instance_create() at least
22:24:42 superdan but I can't imagine we don't go through the object, because we'd miss out on the other stuff we create there,
22:24:54 superdan like instance_extra and such
22:24:56 superdan but if you know other, please point to it
22:25:13 smcginnis Does Nova actual use limits? "nova limits"
22:25:40 smcginnis And if so, where are these set? I've found references that limits are set in the project.conf file, but nothing stating what needs to actually be set.
22:25:57 melwitt superdan: yeah, double checking to see if I missed something last time
22:26:14 smcginnis Cinder has absolute limits too, but it appears to be left over from the Nova split. Trying to figure out if they are actually used/working.
22:38:51 mriedem smcginnis: limits == quotas
22:38:55 mriedem os-quota-sets API
22:39:04 smcginnis mriedem: nope
22:39:14 mriedem the limits API is just the read-only API for getting the quota limits
22:39:22 smcginnis mriedem: Not from what I've seen. Though maybe that's changed in Nova?
22:39:33 smcginnis mriedem: Ah, that might explain some things. :)
22:39:36 mriedem smcginnis: well,
22:39:39 mriedem ...
22:39:40 mriedem ...
22:39:47 mriedem trying to figure out how long i want to make this
22:39:47 smcginnis mriedem: Some really misleading stuff out there.
22:39:57 mriedem quotas come from 3 different places in nova,
22:40:01 mriedem 1. config
22:40:06 smcginnis mriedem: It's late on Friday. Probably better just to tell me to step off.
22:40:11 mriedem 2. quota classes
22:40:12 melwitt superdan: this is where I don't see how instance.create() is ever called in compute cells https://github.com/openstack/nova/blob/master/nova/compute/cells_api.py#L194-L198 so ... I dunno, we'll see
22:40:20 mriedem 3. i might only be thinking of 2
22:40:43 mriedem basically if you don't have default quotas overridden in the API, then the default quotas for all projects comes from config,
22:40:56 mriedem but if you change default quota in the REST API, then it's in the nova DB
22:41:03 mriedem and we never read from config for that resource type again
22:41:11 mriedem and you can tailor default quota globally or per-tenant
22:41:17 mriedem i think that was my #4
22:41:19 mriedem *#3
22:41:31 mriedem 1. check if there is quota in the db for this project and resource, if not,
22:41:37 smcginnis mriedem: OK, that makes sense. Maybe just thrown off by having two different names and some old or incorrect comments I read.
22:41:41 smcginnis mriedem: Thanks
22:41:43 mriedem 2. check if there is default quota in the db for this resource, if not,
22:41:45 mriedem 3, check config
22:41:56 superdan melwitt: okay, but.. the only thing in db_api that will create an instance row is db.instance_create(), and I don't see us calling that from anywhere other than the object, right?
22:42:15 mriedem smcginnis: http://lists.openstack.org/pipermail/openstack-dev/2016-December/108997.html and the linked spec might help a bit
22:42:26 mriedem the spec gives more background in a distilled way on how this all works today
22:42:59 mriedem i should actually move the problem description from https://review.openstack.org/#/c/411035/2/specs/pike/approved/deprecate-os-quota-class-sets.rst into the devref
22:43:13 mriedem since it took me a day plus sanity checking from melwitt to sort it out
22:43:23 smcginnis heh, how timely.
22:43:28 melwitt superdan: that's what I wasn't sure about, I assumed db.instance_update_and_get_original can somehow create an instance row, being that I never found a place instance.create() was called
22:43:45 superdan pretty sure not.. it should fail immediately if you do
22:44:09 superdan melwitt: https://github.com/openstack/nova/blob/master/nova/db/sqlalchemy/api.py#L2696-L2697
22:44:11 superdan should fail right there
22:45:09 melwitt oh, hmm. I'm stumped then
22:45:52 superdan ah
22:46:53 superdan well,
22:46:59 superdan there is this weird automatic hijack of the compute rpc,
22:47:06 superdan which might have had it before
22:47:18 superdan where we neuter the rpc at the api level, and auto-call to the child in some cases
22:47:20 superdan but create isn't in there
22:47:24 superdan build is though
22:47:55 superdan anyway, seems like it has to be something like that

Earlier   Later