| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-06-14 | |||
| 18:42:45 | mnaser | so say you have 200 vms quota, 200 ports quota in neutron, you can send a request with num_instances = 200 and things can get a bit messy | |
| 18:42:52 | mriedem | i'd have to look but i think if you ask for 200 but you can only get 100, we drop the count down to 100 based on how many ports neutron will let you create | |
| 18:43:11 | mriedem | but then it's a race to the computes which actually create the ports | |
| 18:43:29 | mnaser | thing is: i'm okay with a customer having 200 vms, it's just the "create them all at once" thing that ends up hurting a bit | |
| 18:44:22 | mnaser | because then not only do their vms end up in error states but all new vms timeout in spawning until neutron-dhcp-agent catches up (because reload_allocations does 1 network at a time, one port update at a time) | |
| 18:44:23 | mriedem | yeah we don't have any cap on how many instances you can ask for in the REST API schema itself | |
| 18:45:10 | mriedem | melwitt: speaking of api quota things, and your networks quota patch, do you remember if you can even POST /os-tenant-networks in cells v1? | |
| 18:45:12 | mnaser | so the dhcp agent queue grows (and the problem gets even worse when these vms fail, all the ports that failed to get activated receive a delete request, and you just doubled the size of that queue as the dhcp agent is updated about "deleting" them) | |
| 18:45:26 | mnaser | but that's more of a neutron side of things i'm hoping to bring up | |
| 18:45:56 | mriedem | that will also cause api traffice between nova and neutron when the ports are deleted | |
| 18:46:00 | melwitt | mriedem: I think you can, it just won't propagate to child cells at all, so would not result in any instance being able to use the new network if you added one | |
| 18:46:23 | mriedem | nova deletes the port, neutron sends an api request back to nova saying something change via the os-server-external-events api, which rpc casts to the compute to do some cleanup | |
| 18:47:35 | mnaser | yeah, it just causes a lot of bad times overall | |
| 18:47:35 | mriedem | melwitt: so you can, but it doesn't work | |
| 18:47:37 | mriedem | so you can't :) | |
| 18:47:59 | mnaser | workaround would be to bring vif_plugging_timeout up but .. 300 seconds is already long enough :x | |
| 18:48:02 | mriedem | melwitt: i see we exclude tempest test test_list_show_tenant_networks from the cells v1 job | |
| 18:49:17 | mriedem | melwitt: i just got thinking about that because of my rubiks cube questions in that change about assuming single cell for getting all networks | |
| 18:49:33 | mriedem | took me to a dark place about cells v1 and cells v2 and nova-network and what won't work | |
| 18:50:08 | melwitt | mriedem: that stuff is nova-network only too, I think. the create/delete network stuff | |
| 18:51:45 | mriedem | melwitt: yes for os-tenant-networks it is | |
| 18:52:33 | melwitt | yeah | |
| 18:56:28 | mriedem | dansmith: whenever we declare support for multiple v2 cells in the release notes, we should remember to mention that excludes nova-network https://review.openstack.org/#/c/446241/20/nova/api/openstack/compute/tenant_networks.py@200 | |
| 19:34:22 | dansmith | mriedem: totes | |
| 19:59:39 | cfriesen | dansmith: this may be an issue for bauzas, but I had an entry in the DB for the "pci_requests" column in the "instance_extra" table. I then ran the online DB migrations which generated a new RequestSpec for the instance, but the generated RequestSpec for this instance has an emtpy list of pci requests and the "pci_requests" column in "instance_extra" is now empty | |
| 20:03:49 | melwitt | mriedem: replied on the networks quota patch. it seems to me that disassociate is only needed if associate could ever have happened and in the recheck_quota case, it couldn't have. do you think we should still do disassociate anyway? | |
| 20:07:51 | mriedem | melwitt: is that the POST /os-networks/{network_id}/action "associate_host" action? | |
| 20:09:41 | melwitt | lemme see ... whatever calls the nova-network manager's associate method | |
| 20:09:47 | mriedem | os-networks-associate | |
| 20:10:07 | melwitt | this is so confusing | |
| 20:10:58 | melwitt | anyway, I'm not opposed to adding the disassociate call, I just noticed I don't think it would ever do anything | |
| 20:11:06 | mriedem | no i think you're right | |
| 20:11:47 | melwitt | I'm going to have to respin anyway for the errant CellDatabases fixture change | |
| 20:12:37 | melwitt | and the cleanup() function that never gets called probably. it's just weird how all of the fixtures have it wrong | |
| 20:12:46 | melwitt | I don't know why we haven't seen any problems around that | |
| 20:14:24 | mriedem | melwitt: i replied about that too | |
| 20:14:25 | mriedem | and the other | |
| 20:14:37 | mriedem | so i guess just fix that cells db fixture thing and i'm +2 | |
| 20:15:25 | mriedem | speaking of confusing, | |
| 20:15:26 | mriedem | https://developer.openstack.org/api-ref/compute/#hypervisors-os-hypervisors | |
| 20:15:38 | melwitt | mriedem: thanks. yeah changing it to cleanUp broke stuff. gah | |
| 20:15:46 | mriedem | i'm trying to jam GET /os-hypervisors/{hypervisor_hostname_pattern}/servers and GET /os-hypervisors/{hypervisor_hostname_pattern}/search into GET /os-hypervisors now | |
| 20:15:55 | mriedem | plus paging | |
| 20:16:07 | mriedem | this is going to be bug city, and the code is nearly unreadable | |
| 20:16:39 | melwitt | good times | |
| 20:21:13 | openstackgerrit | melanie witt proposed openstack/nova master: Add FloatingIPList.get_count_by_project() https://review.openstack.org/446247 | |
| 20:21:13 | openstackgerrit | melanie witt proposed openstack/nova master: Add FixedIPList.get_count_by_project() https://review.openstack.org/446246 | |
| 20:21:14 | openstackgerrit | melanie witt proposed openstack/nova master: Add SecurityGroupList.get_counts() https://review.openstack.org/446245 | |
| 20:21:14 | openstackgerrit | melanie witt proposed openstack/nova master: Add get_count_by_vm_state() to InstanceList object https://review.openstack.org/446244 | |
| 20:21:16 | openstackgerrit | melanie witt proposed openstack/nova master: Remove 'reserved' count from used limits https://review.openstack.org/446242 | |
| 20:21:17 | openstackgerrit | melanie witt proposed openstack/nova master: Remove useless quota_usage_refresh from nova-manage https://review.openstack.org/446243 | |
| 20:21:19 | openstackgerrit | melanie witt proposed openstack/nova master: Count networks to check quota https://review.openstack.org/446241 | |
| 20:21:21 | openstackgerrit | melanie witt proposed openstack/nova master: Count instances to check quota https://review.openstack.org/416521 | |
| 20:21:23 | openstackgerrit | melanie witt proposed openstack/nova master: Count security groups to check quota https://review.openstack.org/457860 | |
| 20:21:26 | openstackgerrit | melanie witt proposed openstack/nova master: Count fixed ips to check quota https://review.openstack.org/457861 | |
| 20:21:31 | openstackgerrit | melanie witt proposed openstack/nova master: Make Quotas object favor the API database https://review.openstack.org/410945 | |
| 20:21:34 | openstackgerrit | melanie witt proposed openstack/nova master: Add online migration to move quotas to API database https://review.openstack.org/410946 | |
| 20:21:37 | openstackgerrit | melanie witt proposed openstack/nova master: Count floating ips to check quota https://review.openstack.org/457862 | |
| 20:24:51 | mriedem | oomichi: do you remember why we need to apply the validation.query_schema on all api versioned methods even if some microversions don't have query parameters? | |
| 20:25:06 | mriedem | oomichi: for example, GET /os-keypairs/{id} on 2.1 has https://github.com/openstack/nova/blob/master/nova/api/openstack/compute/keypairs.py#L205 | |
| 20:25:13 | mriedem | which is an empty schema | |
| 20:25:29 | mriedem | but then 2.10 has query parameter schema for passing a user_id https://github.com/openstack/nova/blob/master/nova/api/openstack/compute/keypairs.py#L191 | |
| 20:25:48 | mriedem | so i understand why we have that for the 2.10 method, but not for 2.1 or 2.2 | |
| 20:39:19 | mriedem | oy, i suppose a compute_nodes.hypervisor_hostname field can be an IP? | |
| 20:40:13 | mriedem | or a uuid in the case of ironic | |
| 20:40:39 | mriedem | that makes GET /os-hypervisors/{hypervisor_hostname_pattern}/search fun if you're using ironic | |
| 20:41:54 | mriedem | and this looks wrong https://developer.openstack.org/api-ref/compute/?expanded=search-hypervisor-detail#search-hypervisor | |
| 20:41:59 | mriedem | "The hypervisor host name provided by the Nova virt driver. For the Ironic driver, it is the Ironic node name." | |
| 20:42:24 | mriedem | https://github.com/openstack/nova/blob/master/nova/virt/ironic/driver.py#L332 | |
| 20:56:46 | cfriesen | does nova-conductor need to be running in order to run "nova db online_data_migrations"? | |
| 20:56:53 | melwitt | cells meeting in 4 minutes | |
| 21:18:30 | eandersson | https://review.openstack.org/#/c/390984 | |
| 21:18:30 | eandersson | Anyone that got some time to help me look into this :D | |
| 21:19:33 | eandersson | I am trying to figure out why the Server Group Tests are failing because the resource usage isn't always updated. | |
| 21:19:56 | eandersson | https://github.com/openstack/nova/blob/master/nova/compute/resource_tracker.py#L168 | |
| 21:20:27 | eandersson | Need to figure out if it is the test that is broken, or if there is a race condition happening here. | |
| 21:20:45 | eandersson | jaypipes maybe? | |
| 21:30:44 | mriedem | sdague: any idea why we'd cache compute nodes in the request here https://github.com/openstack/nova/blob/master/nova/api/openstack/compute/hypervisors.py#L110 ? | |
| 21:30:53 | mriedem | nothing is looking up a cached compute node record from what i can tell | |
| 21:30:55 | mriedem | anywhere else in the api | |
| 21:31:01 | mriedem | do we just do it for fun and games? | |
| 21:38:13 | oomichi | mriedem: I guessed that was a reason why unknown queries should be blocked, but now additionalProperties is True on https://github.com/openstack/nova/blob/master/nova/api/openstack/compute/schemas/keypairs.py#L94 | |
| 21:38:30 | mriedem | yeah i looked up the original change that added that and it wasn't clear to me | |
| 21:38:45 | oomichi | mriedem: so I can not find the reason now. alex_xu can answer anyways | |
| 21:38:55 | mriedem | yeah i have a lot of questions for alex_xu | |
| 21:39:12 | mriedem | 5:40am for him, i'm not sure why he's not up and working yet :) | |
| 21:39:13 | oomichi | oh, "a lot".. | |
| 21:39:50 | mriedem | see john asked about the same sort of thing here https://review.openstack.org/#/c/398755/18/nova/api/openstack/compute/schemas/keypairs.py | |
| 21:40:03 | mriedem | btw, i removed those decorators from the methods that didn't need them and no tests failed | |
| 21:40:08 | mriedem | so i'm not really sure why we need them | |
| 21:40:11 | mriedem | but that's a question for alex | |
| 21:40:16 | mriedem | one of many | |
| 21:40:58 | oomichi | hehe, I saw the patch on the history. and I found I +2d on that before without this question | |
| 21:41:14 | mriedem | oomichi: did you see my question to sean above about https://github.com/openstack/nova/blob/master/nova/api/openstack/compute/hypervisors.py#L110 ? | |
| 21:41:24 | mriedem | as far as i can tell, nothing uses those cached compute nodes in the request | |
| 21:41:32 | mriedem | i think the cache is more or less for extensions isn't it? | |
| 21:43:39 | melwitt | that's my understanding, that those request caches are just to avoid doing DB lookup every time for each extension that receives the request | |
| 21:44:30 | melwitt | since multiple extensions can operate on the same request | |
| 21:44:59 | mriedem | i don't see anything extending the "hypervisors" resource extension | |
| 21:45:09 | mriedem | therefore i don't think anything would be using the cache | |