| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-01-06 | |||
| 20:09:59 | ildikov | mriedem: oh and tnx for the heads up on this :) | |
| 20:10:22 | openstackgerrit | Sarafraj Singh proposed openstack/nova: Add service_token for nova-cinder interaction https://review.openstack.org/397399 | |
| 20:11:09 | openstackgerrit | Sarafraj Singh proposed openstack/nova: Add service_token for nova-neutron interaction https://review.openstack.org/410394 | |
| 20:32:50 | openstackgerrit | Diana Clarke proposed openstack/nova: Query parameter validation for usages API https://review.openstack.org/417500 | |
| 20:36:22 | openstackgerrit | Diana Clarke proposed openstack/nova: Query parameter validation for usages API https://review.openstack.org/417500 | |
| 20:37:11 | melwitt | hm, grenade jobs failing with "/opt/stack/new/grenade/projects/70_cinder/resources.sh:destroy:199 : openstack server remove floating ip cinder_server1 172.24.5.6 | get() takes exactly 1 argument (2 given)" | |
| 20:48:30 | openstackgerrit | Diana Clarke proposed openstack/nova: Query parameter validation for hypervisors API https://review.openstack.org/417507 | |
| 20:57:13 | openstackgerrit | Claudiu Belu proposed openstack/nova: Adds Hyper-V OVS ViF driver https://review.openstack.org/140045 | |
| 21:10:31 | openstackgerrit | Diana Clarke proposed openstack/nova: Query parameter validation for hypervisors API https://review.openstack.org/417507 | |
| 21:36:38 | openstackgerrit | Diana Clarke proposed openstack/nova: Query parameter validation for usages API https://review.openstack.org/417500 | |
| 21:43:34 | openstackgerrit | Claudiu Belu proposed openstack/nova: Hyper-V: fix live migration with CSVs https://review.openstack.org/380448 | |
| 21:53:34 | superdan | melwitt: well, that got it back down from insane-fail to lots-of-fail | |
| 21:53:38 | superdan | which I call major progress | |
| 21:54:11 | superdan | like, finally some sane failures: http://logs.openstack.org/75/396775/23/check/gate-tempest-dsvm-cells-ubuntu-xenial/c10b6b3/logs/screen-n-api.txt.gz?level=TRACE#_2017-01-06_20_19_05_133 | |
| 21:54:19 | melwitt | superdan: heh, yeah. I saw there's a lot of UnknownCell errors in there now but hopefully that won't be too hard to track down | |
| 21:55:05 | superdan | yeah, I dunno what that means, but.. it's something to chase at least | |
| 21:55:23 | melwitt | definitely a big improvement over crazy pants | |
| 21:55:33 | superdan | huh, it means instance.cell_name is not set | |
| 21:56:23 | melwitt | meaning instance wasn't assigned to a cell v1 cell somehow. or hasn't been scheduled yet | |
| 21:56:40 | superdan | yeah | |
| 21:59:39 | melwitt | hm, so when an instance record is written to a cell db, when it syncs to the top the cell_name is set based on the message routing path? | |
| 22:00:17 | melwitt | I don't think I ever knew how cell_name got set, but it looks like it happens in instance_update_at_top in cells/messaging | |
| 22:01:24 | superdan | that's..odd | |
| 22:01:31 | superdan | I guess so the cell doesn't have to know its name or something? | |
| 22:01:48 | melwitt | seems like it. I dunno | |
| 22:04:04 | superdan | man | |
| 22:04:07 | melwitt | superdan: I wonder if it's something like, it's reading the instance from the cell db so it doesn't have cell_name there. cell_name is only in the record at the top? | |
| 22:04:15 | superdan | I'm just shocked that this continues to work with none of us really knowing about it | |
| 22:04:25 | superdan | yeah might be | |
| 22:05:06 | melwitt | yeah well, we've taken care not to touch the code too much. and I broke things when I object-ified some stuff there that had to get fixed | |
| 22:07:42 | openstackgerrit | Dan Smith proposed openstack/nova: WTFWIP: Multicell support for instance listing https://review.openstack.org/396775 | |
| 22:07:44 | superdan | theory-testing hack ^ | |
| 22:07:51 | melwitt | I wonder if there are any other problems with reading instances from the compute cell db rather than the api cell db. I hope not | |
| 22:09:13 | superdan | well, | |
| 22:09:28 | superdan | it may surprise some people that don't expect connections from api to those dbs | |
| 22:09:35 | superdan | but that's kinda part of the migration path anyway | |
| 22:09:36 | melwitt | I'm not sure that will work, because I don't think cell_name is dirtied in the compute cell | |
| 22:10:05 | superdan | oh, because that makes an rpc call back to messaging | |
| 22:10:32 | melwitt | yeah, the instance.save() will call up to instance_update_at_top, and that function will set cell_name only at the top | |
| 22:11:07 | melwitt | https://github.com/openstack/nova/blob/master/nova/cells/messaging.py#L1009 | |
| 22:11:20 | superdan | right but you mean call up to the regional cell service, which is across rpc | |
| 22:11:31 | melwitt | and that's not going to sync back down | |
| 22:11:36 | superdan | wait | |
| 22:11:41 | superdan | oh yeah | |
| 22:11:48 | melwitt | yeah rpc call to the api cell | |
| 22:11:57 | melwitt | I think | |
| 22:12:04 | superdan | I was thinking that was a direct call into there because the name was the same, but I jumped the api->messaging gap | |
| 22:12:35 | superdan | so I should be able to get it from conf actually | |
| 22:12:50 | melwitt | well, I mean the compute cell will never run the code that changes the cell_name I mean | |
| 22:13:13 | melwitt | I dunno, I'm getting confused. anyway | |
| 22:15:08 | openstackgerrit | Dan Smith proposed openstack/nova: WTFWIP: Multicell support for instance listing https://review.openstack.org/396775 | |
| 22:15:09 | superdan | hack @3 | |
| 22:15:11 | superdan | er, #2 | |
| 22:19:41 | melwitt | superdan: I think that might not work because I think the instance.create() goes only to the api cell db, then later when instance.save() happens in the api cell, that's when it gets written to the compute cell db :\ | |
| 22:21:11 | melwitt | as part of the sync down | |
| 22:21:53 | superdan | melwitt: we have to create() once in the child cell | |
| 22:21:55 | superdan | we can't just save | |
| 22:23:33 | melwitt | superdan: okay. I had thought when we were looking at replays of compute/api the instance create isn't one of them and so I went looking around for how the record ever ends up in the compute cell db | |
| 22:24:25 | superdan | melwitt: well, we have to call db.instance_create() at least | |
| 22:24:42 | superdan | but I can't imagine we don't go through the object, because we'd miss out on the other stuff we create there, | |
| 22:24:54 | superdan | like instance_extra and such | |
| 22:24:56 | superdan | but if you know other, please point to it | |
| 22:25:13 | smcginnis | Does Nova actual use limits? "nova limits" | |
| 22:25:40 | smcginnis | And if so, where are these set? I've found references that limits are set in the project.conf file, but nothing stating what needs to actually be set. | |
| 22:25:57 | melwitt | superdan: yeah, double checking to see if I missed something last time | |
| 22:26:14 | smcginnis | Cinder has absolute limits too, but it appears to be left over from the Nova split. Trying to figure out if they are actually used/working. | |
| 22:38:51 | mriedem | smcginnis: limits == quotas | |
| 22:38:55 | mriedem | os-quota-sets API | |
| 22:39:04 | smcginnis | mriedem: nope | |
| 22:39:14 | mriedem | the limits API is just the read-only API for getting the quota limits | |
| 22:39:22 | smcginnis | mriedem: Not from what I've seen. Though maybe that's changed in Nova? | |
| 22:39:33 | smcginnis | mriedem: Ah, that might explain some things. :) | |
| 22:39:36 | mriedem | smcginnis: well, | |
| 22:39:39 | mriedem | ... | |
| 22:39:40 | mriedem | ... | |
| 22:39:47 | smcginnis | mriedem: Some really misleading stuff out there. | |
| 22:39:47 | mriedem | trying to figure out how long i want to make this | |
| 22:39:57 | mriedem | quotas come from 3 different places in nova, | |
| 22:40:01 | mriedem | 1. config | |
| 22:40:06 | smcginnis | mriedem: It's late on Friday. Probably better just to tell me to step off. | |
| 22:40:11 | mriedem | 2. quota classes | |
| 22:40:12 | melwitt | superdan: this is where I don't see how instance.create() is ever called in compute cells https://github.com/openstack/nova/blob/master/nova/compute/cells_api.py#L194-L198 so ... I dunno, we'll see | |
| 22:40:20 | mriedem | 3. i might only be thinking of 2 | |
| 22:40:43 | mriedem | basically if you don't have default quotas overridden in the API, then the default quotas for all projects comes from config, | |
| 22:40:56 | mriedem | but if you change default quota in the REST API, then it's in the nova DB | |
| 22:41:03 | mriedem | and we never read from config for that resource type again | |
| 22:41:11 | mriedem | and you can tailor default quota globally or per-tenant | |
| 22:41:17 | mriedem | i think that was my #4 | |
| 22:41:19 | mriedem | *#3 | |
| 22:41:31 | mriedem | 1. check if there is quota in the db for this project and resource, if not, | |
| 22:41:37 | smcginnis | mriedem: OK, that makes sense. Maybe just thrown off by having two different names and some old or incorrect comments I read. | |
| 22:41:41 | smcginnis | mriedem: Thanks | |
| 22:41:43 | mriedem | 2. check if there is default quota in the db for this resource, if not, | |
| 22:41:45 | mriedem | 3, check config | |
| 22:41:56 | superdan | melwitt: okay, but.. the only thing in db_api that will create an instance row is db.instance_create(), and I don't see us calling that from anywhere other than the object, right? | |
| 22:42:15 | mriedem | smcginnis: http://lists.openstack.org/pipermail/openstack-dev/2016-December/108997.html and the linked spec might help a bit | |
| 22:42:26 | mriedem | the spec gives more background in a distilled way on how this all works today | |
| 22:42:59 | mriedem | i should actually move the problem description from https://review.openstack.org/#/c/411035/2/specs/pike/approved/deprecate-os-quota-class-sets.rst into the devref | |
| 22:43:13 | mriedem | since it took me a day plus sanity checking from melwitt to sort it out | |
| 22:43:23 | smcginnis | heh, how timely. | |