Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-06-14
18:35:08 mriedem "ServerGroupAffinityFilter",
18:35:36 mriedem i think we can drop ramfilter and diskfilter from defaults now that we have placement required in ocata, but would have to check that out again
18:35:59 mnaser computefilter pretty much checks if service is alive
18:36:00 mnaser so... id imagine that's more important than figuring out if you have enough ram
18:36:22 mnaser but i guess the logic was ram will eliminate a bigger subset of possiblities rather than disabled nodes
18:36:31 mnaser but yeah, placement mix address a whole bunch of that
18:36:33 mriedem assuming there was logic
18:36:54 mriedem mnaser: anyway, let us know if you find anything
18:37:03 mriedem i always like to hear about performance issues
18:37:33 mnaser we've had some neutron ones but so far with nova, it's just trying to see how we can minimize rpc traffic so to handle big loads
18:37:51 mnaser we had the pleasure of an api request with num_instances => 200
18:38:31 mnaser nova seemed to cope well, but neutron couldn't catch up (more specifically neutron-dhcp-agent), so at some point instance port plugging started timing out (vif_plugging_timeout was getting hit)
18:40:28 mnaser which did raise the issue of.. there is no way of limiting num_instances in api requests (based on my research of the code base), so if the quota is 100 vms.. with one api request you can cause a lot of trouble (rate limiting wouldn't be a solution here considering its a single request)
18:41:23 mriedem eesh that's a good point
18:41:50 mriedem i thought there was a port quota in neutron of 50 by default?
18:42:05 mnaser right, but we naturally raise the port quota when we increase the VM quota anyways
18:42:08 mriedem the compute api checks with neutron how many ports can be created in a single multi-instance create run
18:42:18 mriedem ok
18:42:45 mnaser so say you have 200 vms quota, 200 ports quota in neutron, you can send a request with num_instances = 200 and things can get a bit messy
18:42:52 mriedem i'd have to look but i think if you ask for 200 but you can only get 100, we drop the count down to 100 based on how many ports neutron will let you create
18:43:11 mriedem but then it's a race to the computes which actually create the ports
18:43:29 mnaser thing is: i'm okay with a customer having 200 vms, it's just the "create them all at once" thing that ends up hurting a bit
18:44:22 mnaser because then not only do their vms end up in error states but all new vms timeout in spawning until neutron-dhcp-agent catches up (because reload_allocations does 1 network at a time, one port update at a time)
18:44:23 mriedem yeah we don't have any cap on how many instances you can ask for in the REST API schema itself
18:45:10 mriedem melwitt: speaking of api quota things, and your networks quota patch, do you remember if you can even POST /os-tenant-networks in cells v1?
18:45:12 mnaser so the dhcp agent queue grows (and the problem gets even worse when these vms fail, all the ports that failed to get activated receive a delete request, and you just doubled the size of that queue as the dhcp agent is updated about "deleting" them)
18:45:26 mnaser but that's more of a neutron side of things i'm hoping to bring up
18:45:56 mriedem that will also cause api traffice between nova and neutron when the ports are deleted
18:46:00 melwitt mriedem: I think you can, it just won't propagate to child cells at all, so would not result in any instance being able to use the new network if you added one
18:46:23 mriedem nova deletes the port, neutron sends an api request back to nova saying something change via the os-server-external-events api, which rpc casts to the compute to do some cleanup
18:47:35 mnaser yeah, it just causes a lot of bad times overall
18:47:35 mriedem melwitt: so you can, but it doesn't work
18:47:37 mriedem so you can't :)
18:47:59 mnaser workaround would be to bring vif_plugging_timeout up but .. 300 seconds is already long enough :x
18:48:02 mriedem melwitt: i see we exclude tempest test test_list_show_tenant_networks from the cells v1 job
18:49:17 mriedem melwitt: i just got thinking about that because of my rubiks cube questions in that change about assuming single cell for getting all networks
18:49:33 mriedem took me to a dark place about cells v1 and cells v2 and nova-network and what won't work
18:50:08 melwitt mriedem: that stuff is nova-network only too, I think. the create/delete network stuff
18:51:45 mriedem melwitt: yes for os-tenant-networks it is
18:52:33 melwitt yeah
18:56:28 mriedem dansmith: whenever we declare support for multiple v2 cells in the release notes, we should remember to mention that excludes nova-network https://review.openstack.org/#/c/446241/20/nova/api/openstack/compute/tenant_networks.py@200
19:34:22 dansmith mriedem: totes
19:59:39 cfriesen dansmith: this may be an issue for bauzas, but I had an entry in the DB for the "pci_requests" column in the "instance_extra" table. I then ran the online DB migrations which generated a new RequestSpec for the instance, but the generated RequestSpec for this instance has an emtpy list of pci requests and the "pci_requests" column in "instance_extra" is now empty
20:03:49 melwitt mriedem: replied on the networks quota patch. it seems to me that disassociate is only needed if associate could ever have happened and in the recheck_quota case, it couldn't have. do you think we should still do disassociate anyway?
20:07:51 mriedem melwitt: is that the POST /os-networks/{network_id}/action "associate_host" action?
20:09:41 melwitt lemme see ... whatever calls the nova-network manager's associate method
20:09:47 mriedem os-networks-associate
20:10:07 melwitt this is so confusing
20:10:58 melwitt anyway, I'm not opposed to adding the disassociate call, I just noticed I don't think it would ever do anything
20:11:06 mriedem no i think you're right
20:11:47 melwitt I'm going to have to respin anyway for the errant CellDatabases fixture change
20:12:37 melwitt and the cleanup() function that never gets called probably. it's just weird how all of the fixtures have it wrong
20:12:46 melwitt I don't know why we haven't seen any problems around that
20:14:24 mriedem melwitt: i replied about that too
20:14:25 mriedem and the other
20:14:37 mriedem so i guess just fix that cells db fixture thing and i'm +2
20:15:25 mriedem speaking of confusing,
20:15:26 mriedem https://developer.openstack.org/api-ref/compute/#hypervisors-os-hypervisors
20:15:38 melwitt mriedem: thanks. yeah changing it to cleanUp broke stuff. gah
20:15:46 mriedem i'm trying to jam GET /os-hypervisors/{hypervisor_hostname_pattern}/servers and GET /os-hypervisors/{hypervisor_hostname_pattern}/search into GET /os-hypervisors now
20:15:55 mriedem plus paging
20:16:07 mriedem this is going to be bug city, and the code is nearly unreadable
20:16:39 melwitt good times
20:21:13 openstackgerrit melanie witt proposed openstack/nova master: Add FloatingIPList.get_count_by_project() https://review.openstack.org/446247
20:21:13 openstackgerrit melanie witt proposed openstack/nova master: Add FixedIPList.get_count_by_project() https://review.openstack.org/446246
20:21:14 openstackgerrit melanie witt proposed openstack/nova master: Add SecurityGroupList.get_counts() https://review.openstack.org/446245
20:21:14 openstackgerrit melanie witt proposed openstack/nova master: Add get_count_by_vm_state() to InstanceList object https://review.openstack.org/446244
20:21:16 openstackgerrit melanie witt proposed openstack/nova master: Remove 'reserved' count from used limits https://review.openstack.org/446242
20:21:17 openstackgerrit melanie witt proposed openstack/nova master: Remove useless quota_usage_refresh from nova-manage https://review.openstack.org/446243
20:21:19 openstackgerrit melanie witt proposed openstack/nova master: Count networks to check quota https://review.openstack.org/446241
20:21:21 openstackgerrit melanie witt proposed openstack/nova master: Count instances to check quota https://review.openstack.org/416521
20:21:23 openstackgerrit melanie witt proposed openstack/nova master: Count security groups to check quota https://review.openstack.org/457860
20:21:26 openstackgerrit melanie witt proposed openstack/nova master: Count fixed ips to check quota https://review.openstack.org/457861
20:21:31 openstackgerrit melanie witt proposed openstack/nova master: Make Quotas object favor the API database https://review.openstack.org/410945
20:21:34 openstackgerrit melanie witt proposed openstack/nova master: Add online migration to move quotas to API database https://review.openstack.org/410946
20:21:37 openstackgerrit melanie witt proposed openstack/nova master: Count floating ips to check quota https://review.openstack.org/457862
20:24:51 mriedem oomichi: do you remember why we need to apply the validation.query_schema on all api versioned methods even if some microversions don't have query parameters?
20:25:06 mriedem oomichi: for example, GET /os-keypairs/{id} on 2.1 has https://github.com/openstack/nova/blob/master/nova/api/openstack/compute/keypairs.py#L205
20:25:13 mriedem which is an empty schema
20:25:29 mriedem but then 2.10 has query parameter schema for passing a user_id https://github.com/openstack/nova/blob/master/nova/api/openstack/compute/keypairs.py#L191
20:25:48 mriedem so i understand why we have that for the 2.10 method, but not for 2.1 or 2.2
20:39:19 mriedem oy, i suppose a compute_nodes.hypervisor_hostname field can be an IP?
20:40:13 mriedem or a uuid in the case of ironic
20:40:39 mriedem that makes GET /os-hypervisors/{hypervisor_hostname_pattern}/search fun if you're using ironic
20:41:54 mriedem and this looks wrong https://developer.openstack.org/api-ref/compute/?expanded=search-hypervisor-detail#search-hypervisor
20:41:59 mriedem "The hypervisor host name provided by the Nova virt driver. For the Ironic driver, it is the Ironic node name."
20:42:24 mriedem https://github.com/openstack/nova/blob/master/nova/virt/ironic/driver.py#L332
20:56:46 cfriesen does nova-conductor need to be running in order to run "nova db online_data_migrations"?
20:56:53 melwitt cells meeting in 4 minutes
21:18:30 eandersson https://review.openstack.org/#/c/390984
21:18:30 eandersson Anyone that got some time to help me look into this :D
21:19:33 eandersson I am trying to figure out why the Server Group Tests are failing because the resource usage isn't always updated.
21:19:56 eandersson https://github.com/openstack/nova/blob/master/nova/compute/resource_tracker.py#L168
21:20:27 eandersson Need to figure out if it is the test that is broken, or if there is a race condition happening here.
21:20:45 eandersson jaypipes maybe?
21:30:44 mriedem sdague: any idea why we'd cache compute nodes in the request here https://github.com/openstack/nova/blob/master/nova/api/openstack/compute/hypervisors.py#L110 ?
21:30:53 mriedem nothing is looking up a cached compute node record from what i can tell
21:30:55 mriedem anywhere else in the api
21:31:01 mriedem do we just do it for fun and games?
21:38:13 oomichi mriedem: I guessed that was a reason why unknown queries should be blocked, but now additionalProperties is True on https://github.com/openstack/nova/blob/master/nova/api/openstack/compute/schemas/keypairs.py#L94

Earlier   Later