Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-01-06
20:09:21 ildikov mriedem: I fixed it locally by adding the endpoint_template to the config, but I still experience weird things
20:09:59 ildikov mriedem: oh and tnx for the heads up on this :)
20:10:22 openstackgerrit Sarafraj Singh proposed openstack/nova: Add service_token for nova-cinder interaction https://review.openstack.org/397399
20:11:09 openstackgerrit Sarafraj Singh proposed openstack/nova: Add service_token for nova-neutron interaction https://review.openstack.org/410394
20:32:50 openstackgerrit Diana Clarke proposed openstack/nova: Query parameter validation for usages API https://review.openstack.org/417500
20:36:22 openstackgerrit Diana Clarke proposed openstack/nova: Query parameter validation for usages API https://review.openstack.org/417500
20:37:11 melwitt hm, grenade jobs failing with "/opt/stack/new/grenade/projects/70_cinder/resources.sh:destroy:199 : openstack server remove floating ip cinder_server1 172.24.5.6 | get() takes exactly 1 argument (2 given)"
20:48:30 openstackgerrit Diana Clarke proposed openstack/nova: Query parameter validation for hypervisors API https://review.openstack.org/417507
20:57:13 openstackgerrit Claudiu Belu proposed openstack/nova: Adds Hyper-V OVS ViF driver https://review.openstack.org/140045
21:10:31 openstackgerrit Diana Clarke proposed openstack/nova: Query parameter validation for hypervisors API https://review.openstack.org/417507
21:36:38 openstackgerrit Diana Clarke proposed openstack/nova: Query parameter validation for usages API https://review.openstack.org/417500
21:43:34 openstackgerrit Claudiu Belu proposed openstack/nova: Hyper-V: fix live migration with CSVs https://review.openstack.org/380448
21:53:34 superdan melwitt: well, that got it back down from insane-fail to lots-of-fail
21:53:38 superdan which I call major progress
21:54:11 superdan like, finally some sane failures: http://logs.openstack.org/75/396775/23/check/gate-tempest-dsvm-cells-ubuntu-xenial/c10b6b3/logs/screen-n-api.txt.gz?level=TRACE#_2017-01-06_20_19_05_133
21:54:19 melwitt superdan: heh, yeah. I saw there's a lot of UnknownCell errors in there now but hopefully that won't be too hard to track down
21:55:05 superdan yeah, I dunno what that means, but.. it's something to chase at least
21:55:23 melwitt definitely a big improvement over crazy pants
21:55:33 superdan huh, it means instance.cell_name is not set
21:56:23 melwitt meaning instance wasn't assigned to a cell v1 cell somehow. or hasn't been scheduled yet
21:56:40 superdan yeah
21:59:39 melwitt hm, so when an instance record is written to a cell db, when it syncs to the top the cell_name is set based on the message routing path?
22:00:17 melwitt I don't think I ever knew how cell_name got set, but it looks like it happens in instance_update_at_top in cells/messaging
22:01:24 superdan that's..odd
22:01:31 superdan I guess so the cell doesn't have to know its name or something?
22:01:48 melwitt seems like it. I dunno
22:04:04 superdan man
22:04:07 melwitt superdan: I wonder if it's something like, it's reading the instance from the cell db so it doesn't have cell_name there. cell_name is only in the record at the top?
22:04:15 superdan I'm just shocked that this continues to work with none of us really knowing about it
22:04:25 superdan yeah might be
22:05:06 melwitt yeah well, we've taken care not to touch the code too much. and I broke things when I object-ified some stuff there that had to get fixed
22:07:42 openstackgerrit Dan Smith proposed openstack/nova: WTFWIP: Multicell support for instance listing https://review.openstack.org/396775
22:07:44 superdan theory-testing hack ^
22:07:51 melwitt I wonder if there are any other problems with reading instances from the compute cell db rather than the api cell db. I hope not
22:09:13 superdan well,
22:09:28 superdan it may surprise some people that don't expect connections from api to those dbs
22:09:35 superdan but that's kinda part of the migration path anyway
22:09:36 melwitt I'm not sure that will work, because I don't think cell_name is dirtied in the compute cell
22:10:05 superdan oh, because that makes an rpc call back to messaging
22:10:32 melwitt yeah, the instance.save() will call up to instance_update_at_top, and that function will set cell_name only at the top
22:11:07 melwitt https://github.com/openstack/nova/blob/master/nova/cells/messaging.py#L1009
22:11:20 superdan right but you mean call up to the regional cell service, which is across rpc
22:11:31 melwitt and that's not going to sync back down
22:11:36 superdan wait
22:11:41 superdan oh yeah
22:11:48 melwitt yeah rpc call to the api cell
22:11:57 melwitt I think
22:12:04 superdan I was thinking that was a direct call into there because the name was the same, but I jumped the api->messaging gap
22:12:35 superdan so I should be able to get it from conf actually
22:12:50 melwitt well, I mean the compute cell will never run the code that changes the cell_name I mean
22:13:13 melwitt I dunno, I'm getting confused. anyway
22:15:08 openstackgerrit Dan Smith proposed openstack/nova: WTFWIP: Multicell support for instance listing https://review.openstack.org/396775
22:15:09 superdan hack @3
22:15:11 superdan er, #2
22:19:41 melwitt superdan: I think that might not work because I think the instance.create() goes only to the api cell db, then later when instance.save() happens in the api cell, that's when it gets written to the compute cell db :\
22:21:11 melwitt as part of the sync down
22:21:53 superdan melwitt: we have to create() once in the child cell
22:21:55 superdan we can't just save
22:23:33 melwitt superdan: okay. I had thought when we were looking at replays of compute/api the instance create isn't one of them and so I went looking around for how the record ever ends up in the compute cell db
22:24:25 superdan melwitt: well, we have to call db.instance_create() at least
22:24:42 superdan but I can't imagine we don't go through the object, because we'd miss out on the other stuff we create there,
22:24:54 superdan like instance_extra and such
22:24:56 superdan but if you know other, please point to it
22:25:13 smcginnis Does Nova actual use limits? "nova limits"
22:25:40 smcginnis And if so, where are these set? I've found references that limits are set in the project.conf file, but nothing stating what needs to actually be set.
22:25:57 melwitt superdan: yeah, double checking to see if I missed something last time
22:26:14 smcginnis Cinder has absolute limits too, but it appears to be left over from the Nova split. Trying to figure out if they are actually used/working.
22:38:51 mriedem smcginnis: limits == quotas
22:38:55 mriedem os-quota-sets API
22:39:04 smcginnis mriedem: nope
22:39:14 mriedem the limits API is just the read-only API for getting the quota limits
22:39:22 smcginnis mriedem: Not from what I've seen. Though maybe that's changed in Nova?
22:39:33 smcginnis mriedem: Ah, that might explain some things. :)
22:39:36 mriedem smcginnis: well,
22:39:39 mriedem ...
22:39:40 mriedem ...
22:39:47 smcginnis mriedem: Some really misleading stuff out there.
22:39:47 mriedem trying to figure out how long i want to make this
22:39:57 mriedem quotas come from 3 different places in nova,
22:40:01 mriedem 1. config
22:40:06 smcginnis mriedem: It's late on Friday. Probably better just to tell me to step off.
22:40:11 mriedem 2. quota classes
22:40:12 melwitt superdan: this is where I don't see how instance.create() is ever called in compute cells https://github.com/openstack/nova/blob/master/nova/compute/cells_api.py#L194-L198 so ... I dunno, we'll see
22:40:20 mriedem 3. i might only be thinking of 2
22:40:43 mriedem basically if you don't have default quotas overridden in the API, then the default quotas for all projects comes from config,
22:40:56 mriedem but if you change default quota in the REST API, then it's in the nova DB
22:41:03 mriedem and we never read from config for that resource type again
22:41:11 mriedem and you can tailor default quota globally or per-tenant
22:41:17 mriedem i think that was my #4
22:41:19 mriedem *#3
22:41:31 mriedem 1. check if there is quota in the db for this project and resource, if not,
22:41:37 smcginnis mriedem: OK, that makes sense. Maybe just thrown off by having two different names and some old or incorrect comments I read.
22:41:41 smcginnis mriedem: Thanks
22:41:43 mriedem 2. check if there is default quota in the db for this resource, if not,
22:41:45 mriedem 3, check config
22:41:56 superdan melwitt: okay, but.. the only thing in db_api that will create an instance row is db.instance_create(), and I don't see us calling that from anywhere other than the object, right?
22:42:15 mriedem smcginnis: http://lists.openstack.org/pipermail/openstack-dev/2016-December/108997.html and the linked spec might help a bit
22:42:26 mriedem the spec gives more background in a distilled way on how this all works today
22:42:59 mriedem i should actually move the problem description from https://review.openstack.org/#/c/411035/2/specs/pike/approved/deprecate-os-quota-class-sets.rst into the devref
22:43:13 mriedem since it took me a day plus sanity checking from melwitt to sort it out

Earlier   Later